2023-10-24T23:38:56.896 INFO:root:teuthology version: 0.0.1.dev201+g202b180 2023-10-24T23:38:56.897 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi --owner scheduled_yuriw@teuthology --description fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} --archive /home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357 -- /home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357/orig.config.yaml 2023-10-24T23:38:56.959 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-24T23:38:57.048 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357 branch: wip-yuri6-testing-2023-10-23-1148-reef description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} email: yweinste@redhat.com first_in_suite: false job_id: '7435357' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri6-testing-2023-10-23-1148-reef ceph: cephfs: fs: - name: a - name: b session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds - SLOW_OPS - slow request sha1: e652850897f60c32cb9cc05022cf9f93e3b00db1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: cephfs_name: a client.1: cephfs_name: b install: ceph: flavor: default sha1: e652850897f60c32cb9cc05022cf9f93e3b00db1 kclient: client.0: cephfs_name: a client.1: cephfs_name: b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2023-10-23-1148-reef sha1: e652850897f60c32cb9cc05022cf9f93e3b00db1 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 7106 sha1: e652850897f60c32cb9cc05022cf9f93e3b00db1 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri6-testing-2023-10-23-1148-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e652850897f60c32cb9cc05022cf9f93e3b00db1 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPL1FzpXP2qyGnObmjAsCdBxl4upbUplk4LyLEI0kr2ltoLMG8qjHtik+hMWxC3AgtLNcANRvfLwcYhciGTi1r8= smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDF755M2c+AY0wPO/teNzumWbZarR33MHIzud+Kqg6Eo9+1OS5/JoqKUGpHGNoV1DVCGhd2kCArBv8Hzdp+dJjU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 202b180cb047e798fb131047314a862593f45403 timestamp: 2023-10-24_00:06:34 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-24T23:38:57.048 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa; will attempt to use it 2023-10-24T23:38:57.049 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks 2023-10-24T23:38:57.049 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-24T23:38:57.051 INFO:teuthology.task.internal:Checking packages... 2023-10-24T23:38:57.074 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e652850897f60c32cb9cc05022cf9f93e3b00db1' 2023-10-24T23:38:57.075 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-24T23:38:57.075 INFO:teuthology.packaging:ref: None 2023-10-24T23:38:57.076 INFO:teuthology.packaging:tag: None 2023-10-24T23:38:57.076 INFO:teuthology.packaging:branch: wip-yuri6-testing-2023-10-23-1148-reef 2023-10-24T23:38:57.076 INFO:teuthology.packaging:sha1: e652850897f60c32cb9cc05022cf9f93e3b00db1 2023-10-24T23:38:57.077 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2023-10-23-1148-reef 2023-10-24T23:38:57.228 INFO:teuthology.task.internal:Found packages for ceph version 18.2.0-1095-ge6528508-1jammy 2023-10-24T23:38:57.229 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-24T23:38:57.236 INFO:teuthology.task.internal:no buildpackages task found 2023-10-24T23:38:57.236 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-24T23:38:57.289 INFO:teuthology.task.internal:Saving configuration 2023-10-24T23:38:57.308 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-24T23:38:57.314 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-24T23:38:57.340 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-24 23:33:08.598976', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMj6NsGHStOpalWaBtedr8WXpmKREhabQueythvRb7IW'} 2023-10-24T23:38:57.367 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-24 23:33:08.597388', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKlNIAEwaCrZix1aDbmu0121NCeI3T026sGJo9FNyMSI'} 2023-10-24T23:38:57.367 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-24T23:38:57.374 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-24T23:38:57.375 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2023-10-24T23:38:57.375 INFO:teuthology.run_tasks:Running task console_log... 2023-10-24T23:38:57.458 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faa6db54af0>, signals=[15]) 2023-10-24T23:38:57.459 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-24T23:38:57.465 INFO:teuthology.task.internal:Opening connections... 2023-10-24T23:38:57.466 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2023-10-24T23:38:57.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:38:57.548 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2023-10-24T23:38:57.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:38:57.621 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-24T23:38:57.635 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2023-10-24T23:38:57.640 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2023-10-24T23:38:57.641 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2023-10-24T23:38:57.691 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-24T23:38:57.691 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="Ubuntu" 2023-10-24T23:38:57.691 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="22.04" 2023-10-24T23:38:57.691 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-24T23:38:57.692 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_CODENAME=jammy 2023-10-24T23:38:57.692 INFO:teuthology.orchestra.run.smithi107.stdout:ID=ubuntu 2023-10-24T23:38:57.692 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE=debian 2023-10-24T23:38:57.692 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-24T23:38:57.692 INFO:teuthology.orchestra.run.smithi107.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-24T23:38:57.693 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-24T23:38:57.693 INFO:teuthology.orchestra.run.smithi107.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-24T23:38:57.693 INFO:teuthology.orchestra.run.smithi107.stdout:UBUNTU_CODENAME=jammy 2023-10-24T23:38:57.694 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2023-10-24T23:38:57.719 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2023-10-24T23:38:57.724 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2023-10-24T23:38:57.724 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2023-10-24T23:38:57.772 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-24T23:38:57.773 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="Ubuntu" 2023-10-24T23:38:57.773 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="22.04" 2023-10-24T23:38:57.773 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-24T23:38:57.773 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_CODENAME=jammy 2023-10-24T23:38:57.774 INFO:teuthology.orchestra.run.smithi110.stdout:ID=ubuntu 2023-10-24T23:38:57.774 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE=debian 2023-10-24T23:38:57.774 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-24T23:38:57.774 INFO:teuthology.orchestra.run.smithi110.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-24T23:38:57.774 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-24T23:38:57.775 INFO:teuthology.orchestra.run.smithi110.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-24T23:38:57.775 INFO:teuthology.orchestra.run.smithi110.stdout:UBUNTU_CODENAME=jammy 2023-10-24T23:38:57.775 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2023-10-24T23:38:57.802 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-24T23:38:57.812 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-24T23:38:57.837 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-24T23:38:57.838 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2023-10-24T23:38:57.841 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2023-10-24T23:38:57.845 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-24T23:38:57.863 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-24T23:38:57.863 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2023-10-24T23:38:57.886 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2023-10-24T23:38:57.894 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-24T23:38:57.948 INFO:teuthology.run_tasks:Running task kernel... 2023-10-24T23:38:57.967 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-24T23:38:57.968 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-24T23:38:57.969 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-24T23:38:57.970 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-24T23:38:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:38:57.975 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-10-24T23:38:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:38:57.976 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2023-10-24T23:38:58.021 INFO:teuthology.orchestra.run.smithi107.stdout:5.15.0-56-generic 2023-10-24T23:38:58.021 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.15.0-56-generic 2023-10-24T23:38:58.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get clean 2023-10-24T23:38:58.024 INFO:teuthology.orchestra.run.smithi110.stdout:5.15.0-56-generic 2023-10-24T23:38:58.025 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.15.0-56-generic 2023-10-24T23:38:58.025 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2023-10-24T23:38:58.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get update 2023-10-24T23:38:58.150 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2023-10-24T23:38:58.272 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-24T23:38:58.277 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-24T23:38:58.293 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-24T23:38:58.355 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-24T23:38:58.385 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-24T23:38:58.395 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-24T23:38:58.467 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-24T23:38:58.499 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [896 kB] 2023-10-24T23:38:58.513 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [515 kB] 2023-10-24T23:38:58.597 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [353 kB] 2023-10-24T23:38:58.598 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,103 kB] 2023-10-24T23:38:58.603 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [180 kB] 2023-10-24T23:38:58.606 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-24T23:38:58.607 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,016 kB] 2023-10-24T23:38:58.624 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [239 kB] 2023-10-24T23:38:58.626 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-24T23:38:58.627 INFO:teuthology.orchestra.run.smithi107.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [164 kB] 2023-10-24T23:38:58.629 INFO:teuthology.orchestra.run.smithi107.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-10-24T23:38:58.629 INFO:teuthology.orchestra.run.smithi107.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-10-24T23:38:58.630 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-24T23:38:58.630 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,036 kB] 2023-10-24T23:38:58.637 INFO:teuthology.orchestra.run.smithi107.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [793 kB] 2023-10-24T23:38:58.640 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-10-24T23:38:58.640 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [167 kB] 2023-10-24T23:38:58.642 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-10-24T23:38:58.642 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-10-24T23:38:58.646 INFO:teuthology.orchestra.run.smithi107.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-24T23:38:58.647 INFO:teuthology.orchestra.run.smithi107.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-10-24T23:38:58.648 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [995 kB] 2023-10-24T23:38:58.666 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-10-24T23:38:58.668 INFO:teuthology.orchestra.run.smithi107.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-24T23:38:58.670 INFO:teuthology.orchestra.run.smithi107.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-24T23:38:58.670 INFO:teuthology.orchestra.run.smithi107.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-24T23:38:58.670 INFO:teuthology.orchestra.run.smithi107.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-24T23:38:58.670 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-24T23:38:58.671 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-24T23:38:58.671 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-24T23:38:58.671 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-24T23:38:58.671 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-24T23:38:58.675 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-10-24T23:38:58.676 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-10-24T23:38:58.677 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-24T23:38:58.692 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-24T23:38:58.692 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-10-24T23:38:58.693 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-10-24T23:38:58.693 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-24T23:38:58.693 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-10-24T23:38:58.807 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [353 kB] 2023-10-24T23:38:58.820 INFO:teuthology.orchestra.run.smithi107.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-24T23:38:58.960 INFO:teuthology.orchestra.run.smithi107.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [515 kB] 2023-10-24T23:38:59.069 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [896 kB] 2023-10-24T23:38:59.144 INFO:teuthology.orchestra.run.smithi107.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,103 kB] 2023-10-24T23:38:59.215 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [180 kB] 2023-10-24T23:38:59.235 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-10-24T23:38:59.236 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.0 kB] 2023-10-24T23:38:59.238 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,016 kB] 2023-10-24T23:38:59.271 INFO:teuthology.orchestra.run.smithi107.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [239 kB] 2023-10-24T23:38:59.288 INFO:teuthology.orchestra.run.smithi107.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.0 kB] 2023-10-24T23:38:59.290 INFO:teuthology.orchestra.run.smithi107.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,036 kB] 2023-10-24T23:38:59.303 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [164 kB] 2023-10-24T23:38:59.313 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [536 B] 2023-10-24T23:38:59.313 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [793 kB] 2023-10-24T23:38:59.342 INFO:teuthology.orchestra.run.smithi107.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.4 kB] 2023-10-24T23:38:59.343 INFO:teuthology.orchestra.run.smithi107.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [167 kB] 2023-10-24T23:38:59.345 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [562 kB] 2023-10-24T23:38:59.350 INFO:teuthology.orchestra.run.smithi107.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [536 B] 2023-10-24T23:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [660 kB] 2023-10-24T23:38:59.527 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [145 kB] 2023-10-24T23:38:59.908 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-10-24T23:38:59.915 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-10-24T23:38:59.931 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-10-24T23:38:59.931 INFO:teuthology.orchestra.run.smithi110.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-10-24T23:38:59.934 INFO:teuthology.orchestra.run.smithi110.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-10-24T23:39:00.087 INFO:teuthology.orchestra.run.smithi107.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [995 kB] 2023-10-24T23:39:00.197 INFO:teuthology.orchestra.run.smithi107.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [218 kB] 2023-10-24T23:39:00.216 INFO:teuthology.orchestra.run.smithi107.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.0 kB] 2023-10-24T23:39:00.218 INFO:teuthology.orchestra.run.smithi107.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-10-24T23:39:00.221 INFO:teuthology.orchestra.run.smithi107.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-10-24T23:39:00.222 INFO:teuthology.orchestra.run.smithi107.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-10-24T23:39:00.223 INFO:teuthology.orchestra.run.smithi107.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-10-24T23:39:00.223 INFO:teuthology.orchestra.run.smithi107.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-10-24T23:39:00.228 INFO:teuthology.orchestra.run.smithi107.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-10-24T23:39:00.231 INFO:teuthology.orchestra.run.smithi107.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-10-24T23:39:00.396 INFO:teuthology.orchestra.run.smithi107.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-10-24T23:39:00.478 INFO:teuthology.orchestra.run.smithi107.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-10-24T23:39:00.580 INFO:teuthology.orchestra.run.smithi107.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.9 kB] 2023-10-24T23:39:00.613 INFO:teuthology.orchestra.run.smithi107.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.4 kB] 2023-10-24T23:39:00.671 INFO:teuthology.orchestra.run.smithi107.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-10-24T23:39:04.219 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 9,806 kB in 2s (5,707 kB/s) 2023-10-24T23:39:04.956 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 9,806 kB in 2s (3,976 kB/s) 2023-10-24T23:39:05.492 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-24T23:39:05.510 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-24T23:39:05.561 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-24T23:39:05.725 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2023-10-24T23:39:05.726 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2023-10-24T23:39:05.865 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2023-10-24T23:39:05.865 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-24T23:39:05.865 INFO:teuthology.orchestra.run.smithi110.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-24T23:39:05.866 INFO:teuthology.orchestra.run.smithi110.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-24T23:39:05.866 INFO:teuthology.orchestra.run.smithi110.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-24T23:39:05.866 INFO:teuthology.orchestra.run.smithi110.stdout: python2.7-minimal 2023-10-24T23:39:05.866 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-24T23:39:05.867 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2023-10-24T23:39:05.867 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-24T23:39:05.867 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-generic linux-image-5.15.0-87-generic 2023-10-24T23:39:05.868 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-5.15.0-87-generic linux-modules-extra-5.15.0-87-generic 2023-10-24T23:39:05.868 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2023-10-24T23:39:05.868 INFO:teuthology.orchestra.run.smithi110.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-24T23:39:05.869 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2023-10-24T23:39:05.869 INFO:teuthology.orchestra.run.smithi110.stdout: thermald 2023-10-24T23:39:05.914 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2023-10-24T23:39:05.914 INFO:teuthology.orchestra.run.smithi110.stdout: linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-24T23:39:05.914 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.15.0-87-generic linux-modules-5.15.0-87-generic 2023-10-24T23:39:05.915 INFO:teuthology.orchestra.run.smithi110.stdout: linux-modules-extra-5.15.0-87-generic 2023-10-24T23:39:05.915 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2023-10-24T23:39:05.916 INFO:teuthology.orchestra.run.smithi110.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-24T23:39:06.093 INFO:teuthology.orchestra.run.smithi110.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-10-24T23:39:06.094 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 113 MB of archives. 2023-10-24T23:39:06.094 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-24T23:39:06.094 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-87-generic amd64 5.15.0-87.97 [22.4 MB] 2023-10-24T23:39:06.299 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-10-24T23:39:06.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-24T23:39:06.370 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-10-24T23:39:06.535 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2023-10-24T23:39:06.536 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2023-10-24T23:39:06.676 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages were automatically installed and are no longer required: 2023-10-24T23:39:06.676 INFO:teuthology.orchestra.run.smithi107.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-24T23:39:06.676 INFO:teuthology.orchestra.run.smithi107.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-24T23:39:06.677 INFO:teuthology.orchestra.run.smithi107.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-24T23:39:06.677 INFO:teuthology.orchestra.run.smithi107.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-24T23:39:06.677 INFO:teuthology.orchestra.run.smithi107.stdout: python2.7-minimal 2023-10-24T23:39:06.678 INFO:teuthology.orchestra.run.smithi107.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-24T23:39:06.678 INFO:teuthology.orchestra.run.smithi107.stdout:The following additional packages will be installed: 2023-10-24T23:39:06.679 INFO:teuthology.orchestra.run.smithi107.stdout: linux-generic linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-24T23:39:06.679 INFO:teuthology.orchestra.run.smithi107.stdout: linux-headers-generic linux-image-5.15.0-87-generic 2023-10-24T23:39:06.679 INFO:teuthology.orchestra.run.smithi107.stdout: linux-modules-5.15.0-87-generic linux-modules-extra-5.15.0-87-generic 2023-10-24T23:39:06.679 INFO:teuthology.orchestra.run.smithi107.stdout:Suggested packages: 2023-10-24T23:39:06.680 INFO:teuthology.orchestra.run.smithi107.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-10-24T23:39:06.680 INFO:teuthology.orchestra.run.smithi107.stdout:Recommended packages: 2023-10-24T23:39:06.680 INFO:teuthology.orchestra.run.smithi107.stdout: thermald 2023-10-24T23:39:06.731 INFO:teuthology.orchestra.run.smithi107.stdout:The following NEW packages will be installed: 2023-10-24T23:39:06.731 INFO:teuthology.orchestra.run.smithi107.stdout: linux-headers-5.15.0-87 linux-headers-5.15.0-87-generic 2023-10-24T23:39:06.731 INFO:teuthology.orchestra.run.smithi107.stdout: linux-image-5.15.0-87-generic linux-modules-5.15.0-87-generic 2023-10-24T23:39:06.732 INFO:teuthology.orchestra.run.smithi107.stdout: linux-modules-extra-5.15.0-87-generic 2023-10-24T23:39:06.732 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages will be upgraded: 2023-10-24T23:39:06.733 INFO:teuthology.orchestra.run.smithi107.stdout: linux-generic linux-headers-generic linux-image-generic 2023-10-24T23:39:06.786 INFO:teuthology.orchestra.run.smithi107.stdout:3 upgraded, 5 newly installed, 0 to remove and 269 not upgraded. 2023-10-24T23:39:06.787 INFO:teuthology.orchestra.run.smithi107.stdout:Need to get 113 MB of archives. 2023-10-24T23:39:06.787 INFO:teuthology.orchestra.run.smithi107.stdout:After this operation, 582 MB of additional disk space will be used. 2023-10-24T23:39:06.787 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-87-generic amd64 5.15.0-87.97 [22.4 MB] 2023-10-24T23:39:07.235 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-87-generic amd64 5.15.0-87.97 [11.5 MB] 2023-10-24T23:39:07.338 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-87-generic amd64 5.15.0-87.97 [64.0 MB] 2023-10-24T23:39:07.632 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-87-generic amd64 5.15.0-87.97 [11.5 MB] 2023-10-24T23:39:07.974 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-87-generic amd64 5.15.0-87.97 [64.0 MB] 2023-10-24T23:39:08.238 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.87.84 [1,698 B] 2023-10-24T23:39:08.238 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.87.84 [2,482 B] 2023-10-24T23:39:08.238 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87 all 5.15.0-87.97 [12.3 MB] 2023-10-24T23:39:08.371 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87-generic amd64 5.15.0-87.97 [2,888 kB] 2023-10-24T23:39:08.396 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.87.84 [2,322 B] 2023-10-24T23:39:08.779 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 113 MB in 2s (68.2 MB/s) 2023-10-24T23:39:08.934 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-modules-5.15.0-87-generic. 2023-10-24T23:39:10.099 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.87.84 [1,698 B] 2023-10-24T23:39:10.100 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.87.84 [2,482 B] 2023-10-24T23:39:10.100 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87 all 5.15.0-87.97 [12.3 MB] 2023-10-24T23:39:10.443 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-87-generic amd64 5.15.0-87.97 [2,888 kB] 2023-10-24T23:39:10.523 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.87.84 [2,322 B] 2023-10-24T23:39:10.907 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 113 MB in 5s (24.6 MB/s) 2023-10-24T23:39:10.932 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-24T23:39:10.936 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../0-linux-modules-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-24T23:39:10.998 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:11.076 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-5.15.0-87-generic. 2023-10-24T23:39:13.077 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-10-24T23:39:13.081 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../0-linux-modules-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-24T23:39:13.149 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:13.374 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-image-5.15.0-87-generic. 2023-10-24T23:39:13.395 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../1-linux-image-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-24T23:39:13.477 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:13.918 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-87-generic. 2023-10-24T23:39:13.938 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-24T23:39:13.976 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:15.517 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-image-5.15.0-87-generic. 2023-10-24T23:39:15.539 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../1-linux-image-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-24T23:39:15.621 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:16.046 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-87-generic. 2023-10-24T23:39:16.066 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-24T23:39:16.104 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:20.359 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../3-linux-generic_5.15.0.87.84_amd64.deb ... 2023-10-24T23:39:20.486 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-24T23:39:20.802 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.87.84_amd64.deb ... 2023-10-24T23:39:20.907 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-image-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-24T23:39:21.159 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-headers-5.15.0-87. 2023-10-24T23:39:21.170 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../5-linux-headers-5.15.0-87_5.15.0-87.97_all.deb ... 2023-10-24T23:39:21.209 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-24T23:39:21.818 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../3-linux-generic_5.15.0.87.84_amd64.deb ... 2023-10-24T23:39:21.951 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-24T23:39:22.217 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.87.84_amd64.deb ... 2023-10-24T23:39:22.330 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-image-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-24T23:39:22.633 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-5.15.0-87. 2023-10-24T23:39:22.655 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../5-linux-headers-5.15.0-87_5.15.0-87.97_all.deb ... 2023-10-24T23:39:22.699 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-24T23:39:26.850 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package linux-headers-5.15.0-87-generic. 2023-10-24T23:39:26.887 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../6-linux-headers-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-24T23:39:26.925 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:27.929 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package linux-headers-5.15.0-87-generic. 2023-10-24T23:39:27.967 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../6-linux-headers-5.15.0-87-generic_5.15.0-87.97_amd64.deb ... 2023-10-24T23:39:28.004 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:29.487 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.87.84_amd64.deb ... 2023-10-24T23:39:29.609 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking linux-headers-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-24T23:39:29.959 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-24T23:39:30.081 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:30.207 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-headers-generic (5.15.0.87.84) ... 2023-10-24T23:39:30.334 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:31.203 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.87.84_amd64.deb ... 2023-10-24T23:39:31.326 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking linux-headers-generic (5.15.0.87.84) over (5.15.0.56.54) ... 2023-10-24T23:39:31.651 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-5.15.0-87 (5.15.0-87.97) ... 2023-10-24T23:39:31.753 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:31.756 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:31.857 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-headers-generic (5.15.0.87.84) ... 2023-10-24T23:39:31.976 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:33.112 INFO:teuthology.orchestra.run.smithi107.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-24T23:39:33.113 INFO:teuthology.orchestra.run.smithi107.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-24T23:39:33.113 INFO:teuthology.orchestra.run.smithi107.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-87-generic 2023-10-24T23:39:33.113 INFO:teuthology.orchestra.run.smithi107.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-87-generic 2023-10-24T23:39:33.237 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:33.388 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:34.741 INFO:teuthology.orchestra.run.smithi110.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-10-24T23:39:34.741 INFO:teuthology.orchestra.run.smithi110.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-10-24T23:39:34.741 INFO:teuthology.orchestra.run.smithi110.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-87-generic 2023-10-24T23:39:34.741 INFO:teuthology.orchestra.run.smithi110.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-87-generic 2023-10-24T23:39:34.781 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-image-generic (5.15.0.87.84) ... 2023-10-24T23:39:34.865 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-modules-extra-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:34.906 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up linux-generic (5.15.0.87.84) ... 2023-10-24T23:39:35.033 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:35.100 INFO:teuthology.orchestra.run.smithi107.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-24T23:39:35.100 INFO:teuthology.orchestra.run.smithi107.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-87-generic 2023-10-24T23:39:36.242 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-image-generic (5.15.0.87.84) ... 2023-10-24T23:39:36.376 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up linux-generic (5.15.0.87.84) ... 2023-10-24T23:39:36.502 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for linux-image-5.15.0-87-generic (5.15.0-87.97) ... 2023-10-24T23:39:36.565 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-10-24T23:39:36.565 INFO:teuthology.orchestra.run.smithi110.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-87-generic 2023-10-24T23:39:49.690 INFO:teuthology.orchestra.run.smithi107.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-24T23:39:49.691 INFO:teuthology.orchestra.run.smithi107.stdout:Sourcing file `/etc/default/grub' 2023-10-24T23:39:49.707 INFO:teuthology.orchestra.run.smithi107.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-24T23:39:49.732 INFO:teuthology.orchestra.run.smithi107.stdout:Generating grub configuration file ... 2023-10-24T23:39:49.988 INFO:teuthology.orchestra.run.smithi107.stdout:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-24T23:39:50.006 INFO:teuthology.orchestra.run.smithi107.stdout:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-24T23:39:50.285 INFO:teuthology.orchestra.run.smithi107.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-24T23:39:50.290 INFO:teuthology.orchestra.run.smithi107.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-24T23:39:50.360 INFO:teuthology.orchestra.run.smithi107.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-24T23:39:50.365 INFO:teuthology.orchestra.run.smithi107.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-24T23:39:50.432 INFO:teuthology.orchestra.run.smithi110.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-10-24T23:39:50.432 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub' 2023-10-24T23:39:50.436 INFO:teuthology.orchestra.run.smithi110.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-24T23:39:50.461 INFO:teuthology.orchestra.run.smithi110.stdout:Generating grub configuration file ... 2023-10-24T23:39:50.534 INFO:teuthology.orchestra.run.smithi107.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-24T23:39:50.534 INFO:teuthology.orchestra.run.smithi107.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-24T23:39:50.535 INFO:teuthology.orchestra.run.smithi107.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-24T23:39:50.583 INFO:teuthology.orchestra.run.smithi107.stdout:done 2023-10-24T23:39:50.817 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-24T23:39:50.839 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-24T23:39:51.029 DEBUG:teuthology.orchestra.run.smithi107:> dpkg -s linux-image-generic 2023-10-24T23:39:51.052 INFO:teuthology.orchestra.run.smithi107.stdout:Package: linux-image-generic 2023-10-24T23:39:51.052 INFO:teuthology.orchestra.run.smithi107.stdout:Status: install ok installed 2023-10-24T23:39:51.053 INFO:teuthology.orchestra.run.smithi107.stdout:Priority: optional 2023-10-24T23:39:51.053 INFO:teuthology.orchestra.run.smithi107.stdout:Section: kernel 2023-10-24T23:39:51.053 INFO:teuthology.orchestra.run.smithi107.stdout:Installed-Size: 20 2023-10-24T23:39:51.053 INFO:teuthology.orchestra.run.smithi107.stdout:Maintainer: Ubuntu Kernel Team 2023-10-24T23:39:51.053 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: amd64 2023-10-24T23:39:51.054 INFO:teuthology.orchestra.run.smithi107.stdout:Source: linux-meta 2023-10-24T23:39:51.054 INFO:teuthology.orchestra.run.smithi107.stdout:Version: 5.15.0.87.84 2023-10-24T23:39:51.054 INFO:teuthology.orchestra.run.smithi107.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-24T23:39:51.054 INFO:teuthology.orchestra.run.smithi107.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-24T23:39:51.054 INFO:teuthology.orchestra.run.smithi107.stdout:Recommends: thermald 2023-10-24T23:39:51.055 INFO:teuthology.orchestra.run.smithi107.stdout:Description: Generic Linux kernel image 2023-10-24T23:39:51.055 INFO:teuthology.orchestra.run.smithi107.stdout: This package will always depend on the latest generic kernel image 2023-10-24T23:39:51.055 INFO:teuthology.orchestra.run.smithi107.stdout: available. 2023-10-24T23:39:51.055 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-87-generic 2023-10-24T23:39:51.056 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-24T23:39:51.056 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-24T23:39:51.056 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-24T23:39:51.056 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2023-10-24T23:39:51.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get clean 2023-10-24T23:39:51.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get update 2023-10-24T23:39:51.224 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-24T23:39:51.230 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-24T23:39:51.301 INFO:teuthology.orchestra.run.smithi110.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-24T23:39:51.307 INFO:teuthology.orchestra.run.smithi110.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-24T23:39:51.342 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-24T23:39:51.342 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-24T23:39:51.342 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-24T23:39:51.361 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-24T23:39:51.475 INFO:teuthology.orchestra.run.smithi110.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-24T23:39:51.475 INFO:teuthology.orchestra.run.smithi110.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-10-24T23:39:51.475 INFO:teuthology.orchestra.run.smithi110.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-24T23:39:51.507 INFO:teuthology.orchestra.run.smithi110.stdout:done 2023-10-24T23:39:51.955 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2023-10-24T23:39:51.972 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2023-10-24T23:39:51.973 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2023-10-24T23:39:51.973 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2023-10-24T23:39:51.974 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2023-10-24T23:39:51.974 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 20 2023-10-24T23:39:51.974 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2023-10-24T23:39:51.975 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2023-10-24T23:39:51.975 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2023-10-24T23:39:51.976 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 5.15.0.87.84 2023-10-24T23:39:51.976 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-24T23:39:51.976 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-24T23:39:51.976 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2023-10-24T23:39:51.977 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2023-10-24T23:39:51.977 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2023-10-24T23:39:51.977 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2023-10-24T23:39:51.978 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-87-generic 2023-10-24T23:39:51.978 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-24T23:39:51.979 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-24T23:39:51.979 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-10-24T23:39:51.980 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2023-10-24T23:39:51.980 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get clean 2023-10-24T23:39:52.142 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2023-10-24T23:39:52.254 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-24T23:39:52.255 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-24T23:39:52.264 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-24T23:39:52.281 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-24T23:39:52.647 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-10-24T23:39:52.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-24T23:39:52.719 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-10-24T23:39:52.894 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2023-10-24T23:39:52.895 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2023-10-24T23:39:53.036 INFO:teuthology.orchestra.run.smithi107.stdout:linux-image-generic is already the newest version (5.15.0.87.84). 2023-10-24T23:39:53.037 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages were automatically installed and are no longer required: 2023-10-24T23:39:53.037 INFO:teuthology.orchestra.run.smithi107.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-24T23:39:53.037 INFO:teuthology.orchestra.run.smithi107.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-24T23:39:53.037 INFO:teuthology.orchestra.run.smithi107.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-24T23:39:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-24T23:39:53.038 INFO:teuthology.orchestra.run.smithi107.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-24T23:39:53.038 INFO:teuthology.orchestra.run.smithi107.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-24T23:39:53.073 INFO:teuthology.orchestra.run.smithi107.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-10-24T23:39:53.075 DEBUG:teuthology.orchestra.run.smithi107:> dpkg -s linux-image-generic 2023-10-24T23:39:53.131 INFO:teuthology.orchestra.run.smithi107.stdout:Package: linux-image-generic 2023-10-24T23:39:53.131 INFO:teuthology.orchestra.run.smithi107.stdout:Status: install ok installed 2023-10-24T23:39:53.131 INFO:teuthology.orchestra.run.smithi107.stdout:Priority: optional 2023-10-24T23:39:53.132 INFO:teuthology.orchestra.run.smithi107.stdout:Section: kernel 2023-10-24T23:39:53.132 INFO:teuthology.orchestra.run.smithi107.stdout:Installed-Size: 20 2023-10-24T23:39:53.132 INFO:teuthology.orchestra.run.smithi107.stdout:Maintainer: Ubuntu Kernel Team 2023-10-24T23:39:53.132 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: amd64 2023-10-24T23:39:53.132 INFO:teuthology.orchestra.run.smithi107.stdout:Source: linux-meta 2023-10-24T23:39:53.133 INFO:teuthology.orchestra.run.smithi107.stdout:Version: 5.15.0.87.84 2023-10-24T23:39:53.133 INFO:teuthology.orchestra.run.smithi107.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-24T23:39:53.133 INFO:teuthology.orchestra.run.smithi107.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-24T23:39:53.133 INFO:teuthology.orchestra.run.smithi107.stdout:Recommends: thermald 2023-10-24T23:39:53.133 INFO:teuthology.orchestra.run.smithi107.stdout:Description: Generic Linux kernel image 2023-10-24T23:39:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: This package will always depend on the latest generic kernel image 2023-10-24T23:39:53.134 INFO:teuthology.orchestra.run.smithi107.stdout: available. 2023-10-24T23:39:53.134 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-10-24T23:39:53.176 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.f4hPb3GHPD 2023-10-24T23:39:53.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub/grub.cfg /tmp/tmp.f4hPb3GHPD 2023-10-24T23:39:53.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.f4hPb3GHPD 2023-10-24T23:39:53.354 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.f4hPb3GHPD is 10KB 2023-10-24T23:39:53.367 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.f4hPb3GHPD 2023-10-24T23:39:53.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-24T23:39:53.428 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:39:53.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-24T23:39:53.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-24T23:39:53.495 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-87-generic 2023-10-24T23:39:53.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo update-grub 2023-10-24T23:39:53.542 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-24T23:39:53.559 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-10-24T23:39:53.619 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-24T23:39:53.805 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2023-10-24T23:39:53.806 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2023-10-24T23:39:53.962 INFO:teuthology.orchestra.run.smithi110.stdout:linux-image-generic is already the newest version (5.15.0.87.84). 2023-10-24T23:39:53.962 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2023-10-24T23:39:53.962 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-24T23:39:53.962 INFO:teuthology.orchestra.run.smithi110.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-24T23:39:53.963 INFO:teuthology.orchestra.run.smithi110.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-24T23:39:53.963 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-24T23:39:53.963 INFO:teuthology.orchestra.run.smithi110.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-24T23:39:53.963 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-24T23:39:53.999 INFO:teuthology.orchestra.run.smithi110.stdout:0 upgraded, 0 newly installed, 0 to remove and 269 not upgraded. 2023-10-24T23:39:54.001 DEBUG:teuthology.orchestra.run.smithi110:> dpkg -s linux-image-generic 2023-10-24T23:39:54.016 INFO:teuthology.orchestra.run.smithi110.stdout:Package: linux-image-generic 2023-10-24T23:39:54.017 INFO:teuthology.orchestra.run.smithi110.stdout:Status: install ok installed 2023-10-24T23:39:54.017 INFO:teuthology.orchestra.run.smithi110.stdout:Priority: optional 2023-10-24T23:39:54.017 INFO:teuthology.orchestra.run.smithi110.stdout:Section: kernel 2023-10-24T23:39:54.017 INFO:teuthology.orchestra.run.smithi110.stdout:Installed-Size: 20 2023-10-24T23:39:54.018 INFO:teuthology.orchestra.run.smithi110.stdout:Maintainer: Ubuntu Kernel Team 2023-10-24T23:39:54.018 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: amd64 2023-10-24T23:39:54.018 INFO:teuthology.orchestra.run.smithi110.stdout:Source: linux-meta 2023-10-24T23:39:54.018 INFO:teuthology.orchestra.run.smithi110.stdout:Version: 5.15.0.87.84 2023-10-24T23:39:54.018 INFO:teuthology.orchestra.run.smithi110.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-87), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-10-24T23:39:54.019 INFO:teuthology.orchestra.run.smithi110.stdout:Depends: linux-image-5.15.0-87-generic, linux-modules-extra-5.15.0-87-generic, linux-firmware, intel-microcode, amd64-microcode 2023-10-24T23:39:54.019 INFO:teuthology.orchestra.run.smithi110.stdout:Recommends: thermald 2023-10-24T23:39:54.019 INFO:teuthology.orchestra.run.smithi110.stdout:Description: Generic Linux kernel image 2023-10-24T23:39:54.019 INFO:teuthology.orchestra.run.smithi110.stdout: This package will always depend on the latest generic kernel image 2023-10-24T23:39:54.019 INFO:teuthology.orchestra.run.smithi110.stdout: available. 2023-10-24T23:39:54.020 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-10-24T23:39:54.064 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.ExGOXbW5Dk 2023-10-24T23:39:54.065 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ExGOXbW5Dk 2023-10-24T23:39:54.119 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.ExGOXbW5Dk 2023-10-24T23:39:54.237 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.ExGOXbW5Dk is 10KB 2023-10-24T23:39:54.250 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.ExGOXbW5Dk 2023-10-24T23:39:54.256 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-10-24T23:39:54.312 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:39:54.312 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-10-24T23:39:54.312 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-10-24T23:39:54.380 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-87-generic 2023-10-24T23:39:54.381 DEBUG:teuthology.orchestra.run.smithi110:> sudo update-grub 2023-10-24T23:39:54.688 INFO:teuthology.orchestra.run.smithi107.stderr:Sourcing file `/etc/default/grub' 2023-10-24T23:39:54.690 INFO:teuthology.orchestra.run.smithi107.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-24T23:39:54.692 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2023-10-24T23:39:54.884 INFO:teuthology.orchestra.run.smithi107.stderr:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-24T23:39:54.889 INFO:teuthology.orchestra.run.smithi107.stderr:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-24T23:39:55.129 INFO:teuthology.orchestra.run.smithi107.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-24T23:39:55.135 INFO:teuthology.orchestra.run.smithi107.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-24T23:39:55.177 INFO:teuthology.orchestra.run.smithi107.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-24T23:39:55.183 INFO:teuthology.orchestra.run.smithi107.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-24T23:39:55.290 INFO:teuthology.orchestra.run.smithi107.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-24T23:39:55.291 INFO:teuthology.orchestra.run.smithi107.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-24T23:39:55.291 INFO:teuthology.orchestra.run.smithi107.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-24T23:39:55.313 INFO:teuthology.orchestra.run.smithi107.stderr:done 2023-10-24T23:39:55.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2023-10-24T23:39:55.501 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub' 2023-10-24T23:39:55.502 INFO:teuthology.orchestra.run.smithi110.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-24T23:39:55.503 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2023-10-24T23:39:55.710 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.15.0-87-generic 2023-10-24T23:39:55.715 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.15.0-87-generic 2023-10-24T23:39:55.987 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-24T23:39:55.992 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-24T23:39:56.034 INFO:teuthology.orchestra.run.smithi110.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-24T23:39:56.039 INFO:teuthology.orchestra.run.smithi110.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-24T23:39:56.194 INFO:teuthology.orchestra.run.smithi110.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-24T23:39:56.194 INFO:teuthology.orchestra.run.smithi110.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-24T23:39:56.194 INFO:teuthology.orchestra.run.smithi110.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-24T23:39:56.205 INFO:teuthology.orchestra.run.smithi110.stderr:done 2023-10-24T23:39:56.223 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2023-10-24T23:40:25.354 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-24T23:40:25.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-10-24T23:40:25.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:40:26.227 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-10-24T23:40:26.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-24T23:40:26.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:40:43.770 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-10-24T23:40:44.794 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-10-24T23:40:52.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-10-24T23:40:52.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:40:53.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-24T23:40:53.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:40:55.834 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-10-24T23:40:56.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-10-24T23:41:07.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-10-24T23:41:07.847 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:41:08.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-24T23:41:08.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:41:26.266 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2023-10-24T23:41:27.290 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-10-24T23:41:41.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-10-24T23:41:41.279 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:41:41.667 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-10-24T23:41:42.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-24T23:41:42.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:41:42.613 DEBUG:teuthology.orchestra.run.smithi110:> true 2023-10-24T23:41:42.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-10-24T23:41:42.861 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-87-generic"... 2023-10-24T23:41:42.861 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-10-24T23:41:42.909 INFO:teuthology.orchestra.run.smithi107.stdout:5.15.0-87-generic 2023-10-24T23:41:42.909 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-87-generic vs 5.15.0-87-generic 2023-10-24T23:41:42.910 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-24T23:41:42.910 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-24T23:41:43.149 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-24T23:41:43.149 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-87-generic"... 2023-10-24T23:41:43.150 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2023-10-24T23:41:43.198 INFO:teuthology.orchestra.run.smithi110.stdout:5.15.0-87-generic 2023-10-24T23:41:43.199 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-87-generic vs 5.15.0-87-generic 2023-10-24T23:41:43.199 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-24T23:41:43.199 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-24T23:41:43.910 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-24T23:41:43.911 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-24T23:41:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2023-10-24T23:41:44.079 DEBUG:teuthology.parallel:result is None 2023-10-24T23:41:44.200 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-10-24T23:41:44.201 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-24T23:41:44.266 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2023-10-24T23:41:44.292 DEBUG:teuthology.parallel:result is None 2023-10-24T23:41:44.293 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-24T23:41:44.302 INFO:teuthology.task.internal:Creating test directory... 2023-10-24T23:41:44.303 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-24T23:41:44.305 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-24T23:41:44.311 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-24T23:41:44.330 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-24T23:41:44.378 INFO:teuthology.task.internal:Creating archive directory... 2023-10-24T23:41:44.378 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-24T23:41:44.381 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-24T23:41:44.402 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-24T23:41:44.428 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-24T23:41:44.428 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-24T23:41:44.435 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-24T23:41:44.454 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-24T23:41:44.458 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-24T23:41:44.461 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-24T23:41:44.466 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-24T23:41:44.467 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-24T23:41:44.477 INFO:teuthology.task.internal:Configuring sudo... 2023-10-24T23:41:44.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-24T23:41:44.515 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-24T23:41:44.527 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-24T23:41:44.536 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-24T23:41:44.537 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-24T23:41:44.575 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-24T23:41:44.580 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-24T23:41:44.626 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-24T23:41:44.674 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:41:44.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-24T23:41:44.745 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-24T23:41:44.752 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-24T23:41:44.799 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:41:44.799 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-24T23:41:44.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2023-10-24T23:41:44.874 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2023-10-24T23:41:44.997 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-24T23:41:45.005 INFO:teuthology.task.internal:Starting timer... 2023-10-24T23:41:45.005 INFO:teuthology.run_tasks:Running task pcp... 2023-10-24T23:41:45.031 INFO:teuthology.run_tasks:Running task selinux... 2023-10-24T23:41:45.079 DEBUG:teuthology.task.selinux:Excluding smithi107: OS 'ubuntu' does not support SELinux 2023-10-24T23:41:45.080 DEBUG:teuthology.task.selinux:Excluding smithi110: OS 'ubuntu' does not support SELinux 2023-10-24T23:41:45.080 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-24T23:41:45.081 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-24T23:41:45.081 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-24T23:41:45.081 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-24T23:41:45.130 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-24T23:41:45.131 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-24T23:41:45.147 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-24T23:41:45.148 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi110.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-24T23:49:20.371 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com')] 2023-10-24T23:49:20.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-10-24T23:49:20.373 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:49:20.453 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-10-24T23:49:20.570 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-10-24T23:49:20.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-24T23:49:20.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-24T23:49:20.647 DEBUG:teuthology.orchestra.run.smithi110:> true 2023-10-24T23:49:20.750 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-10-24T23:49:20.751 INFO:teuthology.run_tasks:Running task clock... 2023-10-24T23:49:20.762 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-24T23:49:20.763 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-24T23:49:20.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-24T23:49:20.766 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-24T23:49:20.766 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-24T23:49:20.791 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-24T23:49:20.791 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: Command line: ntpd -gq 2023-10-24T23:49:20.791 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: ---------------------------------------------------- 2023-10-24T23:49:20.792 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: ntp-4 is maintained by Network Time Foundation, 2023-10-24T23:49:20.792 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-24T23:49:20.792 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: corporation. Support and training for ntp-4 are 2023-10-24T23:49:20.792 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: available at https://www.nwtime.org/support 2023-10-24T23:49:20.792 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: ---------------------------------------------------- 2023-10-24T23:49:20.792 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: proto: precision = 0.060 usec (-24) 2023-10-24T23:49:20.793 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: basedate set to 2022-02-04 2023-10-24T23:49:20.793 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: gps base set to 2022-02-06 (week 2196) 2023-10-24T23:49:20.794 INFO:teuthology.orchestra.run.smithi107.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-24T23:49:20.794 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-24T23:49:20.794 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: restrict ::: KOD does nothing without LIMITED. 2023-10-24T23:49:20.795 INFO:teuthology.orchestra.run.smithi107.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-24T23:49:20.796 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: Listen and drop on 0 v6wildcard [::]:123 2023-10-24T23:49:20.796 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-24T23:49:20.796 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: Listen normally on 2 lo 127.0.0.1:123 2023-10-24T23:49:20.796 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: Listen normally on 3 enp3s0f1 172.21.15.107:123 2023-10-24T23:49:20.797 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: Listen normally on 4 lo [::1]:123 2023-10-24T23:49:20.797 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:47eb%5]:123 2023-10-24T23:49:20.797 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:20 ntpd[27324]: Listening on routing socket on fd #22 for interface updates 2023-10-24T23:49:20.823 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-24T23:49:20.824 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: Command line: ntpd -gq 2023-10-24T23:49:20.824 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: ---------------------------------------------------- 2023-10-24T23:49:20.824 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: ntp-4 is maintained by Network Time Foundation, 2023-10-24T23:49:20.825 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-24T23:49:20.825 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: corporation. Support and training for ntp-4 are 2023-10-24T23:49:20.825 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: available at https://www.nwtime.org/support 2023-10-24T23:49:20.825 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: ---------------------------------------------------- 2023-10-24T23:49:20.826 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: proto: precision = 0.045 usec (-24) 2023-10-24T23:49:20.826 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: basedate set to 2022-02-04 2023-10-24T23:49:20.826 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: gps base set to 2022-02-06 (week 2196) 2023-10-24T23:49:20.827 INFO:teuthology.orchestra.run.smithi110.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-24T23:49:20.827 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-24T23:49:20.827 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: restrict ::: KOD does nothing without LIMITED. 2023-10-24T23:49:20.828 INFO:teuthology.orchestra.run.smithi110.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-24T23:49:20.829 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: Listen and drop on 0 v6wildcard [::]:123 2023-10-24T23:49:20.829 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-24T23:49:20.829 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: Listen normally on 2 lo 127.0.0.1:123 2023-10-24T23:49:20.829 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: Listen normally on 3 enp3s0f1 172.21.15.110:123 2023-10-24T23:49:20.829 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: Listen normally on 4 lo [::1]:123 2023-10-24T23:49:20.830 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc35%5]:123 2023-10-24T23:49:20.830 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:20 ntpd[27377]: Listening on routing socket on fd #22 for interface updates 2023-10-24T23:49:21.792 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:21.792 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:21.792 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:21.792 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:21.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:21.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:21.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:21.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:21.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:21.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:21.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:21.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:21.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:21.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:21 ntpd[27324]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:21 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:21.824 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:21 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:21.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:21 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:21.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:21 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:21.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:21 ntpd[27377]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:21.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:21 ntpd[27377]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:23.792 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:23.792 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:23.792 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:23.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:23.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:23.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:23.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:23.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:23.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:23.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:23.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:23.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:23 ntpd[27324]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:23.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:23.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:23.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:23.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:23.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:23.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:23.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:23.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:23.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:23.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:23.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:23.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:23.828 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:23.828 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:23 ntpd[27377]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:24.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:24 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:24.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:24 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:24.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:24 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:24.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:24 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:24.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:24 ntpd[27377]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:24.828 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:24 ntpd[27377]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:25.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:25.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:25.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:25.795 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:25.797 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:25.797 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:25.797 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:25.798 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:25.799 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:25 ntpd[27324]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:25.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:25.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:25.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:25.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:25.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:25.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:25.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:25.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:25.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:25.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:25.828 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:25.828 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:25 ntpd[27377]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:26.824 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:26 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:26.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:26 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:26.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:26 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:26.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:26 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:26.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:26 ntpd[27377]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:26.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:26 ntpd[27377]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:27.792 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:27.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:27.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:27.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:27.793 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:27.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:27.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:27.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:27.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:27.794 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:27.795 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:27.795 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:27.795 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:27.795 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:27.796 INFO:teuthology.orchestra.run.smithi107.stdout:24 Oct 23:49:27 ntpd[27324]: ntpd: time slew +0.001631 s 2023-10-24T23:49:27.796 INFO:teuthology.orchestra.run.smithi107.stdout:ntpd: time slew +0.001631s 2023-10-24T23:49:27.796 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-24T23:49:27.797 INFO:teuthology.orchestra.run.smithi107.stderr:24 Oct 23:49:27 ntpd[27324]: can't open /var/log/ntpstats/loopstats.20231024: Permission denied 2023-10-24T23:49:27.824 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:27 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:27 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:27 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:27 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:27.825 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:27 ntpd[27377]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-24T23:49:27.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:27 ntpd[27377]: can't open /var/log/ntpstats/rawstats.20231024: Permission denied 2023-10-24T23:49:27.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:27 ntpd[27377]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-24T23:49:27.826 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:27 ntpd[27377]: can't open /var/log/ntpstats/peerstats.20231024: Permission denied 2023-10-24T23:49:27.826 INFO:teuthology.orchestra.run.smithi110.stdout:24 Oct 23:49:27 ntpd[27377]: ntpd: time slew -0.002284 s 2023-10-24T23:49:27.827 INFO:teuthology.orchestra.run.smithi110.stdout:ntpd: time slew -0.002284s 2023-10-24T23:49:27.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:27 ntpd[27377]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-24T23:49:27.827 INFO:teuthology.orchestra.run.smithi110.stderr:24 Oct 23:49:27 ntpd[27377]: can't open /var/log/ntpstats/loopstats.20231024: Permission denied 2023-10-24T23:49:27.913 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2023-10-24T23:49:27.913 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2023-10-24T23:49:27.913 INFO:teuthology.orchestra.run.smithi110.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-24T23:49:27.914 INFO:teuthology.orchestra.run.smithi110.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-24T23:49:27.914 INFO:teuthology.orchestra.run.smithi110.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-24T23:49:27.914 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-24T23:49:27.943 INFO:teuthology.orchestra.run.smithi107.stdout: remote refid st t when poll reach delay offset jitter 2023-10-24T23:49:27.943 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================== 2023-10-24T23:49:27.943 INFO:teuthology.orchestra.run.smithi107.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-24T23:49:27.944 INFO:teuthology.orchestra.run.smithi107.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-24T23:49:27.944 INFO:teuthology.orchestra.run.smithi107.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-24T23:49:27.944 INFO:teuthology.orchestra.run.smithi107.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-24T23:49:27.945 INFO:teuthology.run_tasks:Running task install... 2023-10-24T23:49:27.953 DEBUG:teuthology.task.install:project ceph 2023-10-24T23:49:27.954 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e652850897f60c32cb9cc05022cf9f93e3b00db1'}} 2023-10-24T23:49:27.954 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e652850897f60c32cb9cc05022cf9f93e3b00db1'} 2023-10-24T23:49:27.955 INFO:teuthology.task.install:Using flavor: default 2023-10-24T23:49:27.960 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-24T23:49:27.961 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-24T23:49:27.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-key list | grep Ceph 2023-10-24T23:49:27.962 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-key list | grep Ceph 2023-10-24T23:49:28.022 INFO:teuthology.orchestra.run.smithi110.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-24T23:49:28.052 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-24T23:49:28.053 INFO:teuthology.orchestra.run.smithi110.stdout:uid [ unknown] Ceph.com (release key) 2023-10-24T23:49:28.053 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-24T23:49:28.054 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-24T23:49:28.054 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e652850897f60c32cb9cc05022cf9f93e3b00db1 2023-10-24T23:49:28.058 INFO:teuthology.orchestra.run.smithi107.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-24T23:49:28.087 INFO:teuthology.orchestra.run.smithi107.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-24T23:49:28.087 INFO:teuthology.orchestra.run.smithi107.stdout:uid [ unknown] Ceph.com (release key) 2023-10-24T23:49:28.088 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-24T23:49:28.088 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-24T23:49:28.088 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e652850897f60c32cb9cc05022cf9f93e3b00db1 2023-10-24T23:49:28.201 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default/ 2023-10-24T23:49:28.201 INFO:teuthology.task.install.deb:Package version is 18.2.0-1095-ge6528508-1jammy 2023-10-24T23:49:28.224 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default/ 2023-10-24T23:49:28.225 INFO:teuthology.task.install.deb:Package version is 18.2.0-1095-ge6528508-1jammy 2023-10-24T23:49:28.337 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:49:28.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-24T23:49:28.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo apt-get update 2023-10-24T23:49:28.373 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:49:28.373 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-24T23:49:28.388 DEBUG:teuthology.orchestra.run.smithi110:> sudo apt-get update 2023-10-24T23:49:28.511 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-10-24T23:49:28.557 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-24T23:49:28.577 INFO:teuthology.orchestra.run.smithi107.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy InRelease 2023-10-24T23:49:28.599 INFO:teuthology.orchestra.run.smithi107.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-24T23:49:28.599 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-24T23:49:28.608 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy InRelease 2023-10-24T23:49:28.619 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-10-24T23:49:28.621 INFO:teuthology.orchestra.run.smithi107.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-24T23:49:28.631 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-24T23:49:28.643 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-10-24T23:49:28.653 INFO:teuthology.orchestra.run.smithi110.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-24T23:49:28.671 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,437 B] 2023-10-24T23:49:28.675 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-10-24T23:49:28.703 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,437 B] 2023-10-24T23:49:28.703 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-10-24T23:49:28.722 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-24T23:49:28.806 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-24T23:49:28.885 INFO:teuthology.orchestra.run.smithi110.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-24T23:49:28.930 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 366 kB in 0s (917 kB/s) 2023-10-24T23:49:29.021 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 28.5 kB in 0s (62.4 kB/s) 2023-10-24T23:49:29.808 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-10-24T23:49:29.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.0-1095-ge6528508-1jammy cephadm=18.2.0-1095-ge6528508-1jammy ceph-mds=18.2.0-1095-ge6528508-1jammy ceph-mgr=18.2.0-1095-ge6528508-1jammy ceph-common=18.2.0-1095-ge6528508-1jammy ceph-fuse=18.2.0-1095-ge6528508-1jammy ceph-test=18.2.0-1095-ge6528508-1jammy ceph-volume=18.2.0-1095-ge6528508-1jammy radosgw=18.2.0-1095-ge6528508-1jammy python3-rados=18.2.0-1095-ge6528508-1jammy python3-rgw=18.2.0-1095-ge6528508-1jammy python3-cephfs=18.2.0-1095-ge6528508-1jammy python3-rbd=18.2.0-1095-ge6528508-1jammy libcephfs2=18.2.0-1095-ge6528508-1jammy libcephfs-dev=18.2.0-1095-ge6528508-1jammy librados2=18.2.0-1095-ge6528508-1jammy librbd1=18.2.0-1095-ge6528508-1jammy rbd-fuse=18.2.0-1095-ge6528508-1jammy python3-cephfs=18.2.0-1095-ge6528508-1jammy cephfs-shell=18.2.0-1095-ge6528508-1jammy cephfs-top=18.2.0-1095-ge6528508-1jammy cephfs-mirror=18.2.0-1095-ge6528508-1jammy 2023-10-24T23:49:29.841 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-24T23:49:29.856 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.0-1095-ge6528508-1jammy cephadm=18.2.0-1095-ge6528508-1jammy ceph-mds=18.2.0-1095-ge6528508-1jammy ceph-mgr=18.2.0-1095-ge6528508-1jammy ceph-common=18.2.0-1095-ge6528508-1jammy ceph-fuse=18.2.0-1095-ge6528508-1jammy ceph-test=18.2.0-1095-ge6528508-1jammy ceph-volume=18.2.0-1095-ge6528508-1jammy radosgw=18.2.0-1095-ge6528508-1jammy python3-rados=18.2.0-1095-ge6528508-1jammy python3-rgw=18.2.0-1095-ge6528508-1jammy python3-cephfs=18.2.0-1095-ge6528508-1jammy python3-rbd=18.2.0-1095-ge6528508-1jammy libcephfs2=18.2.0-1095-ge6528508-1jammy libcephfs-dev=18.2.0-1095-ge6528508-1jammy librados2=18.2.0-1095-ge6528508-1jammy librbd1=18.2.0-1095-ge6528508-1jammy rbd-fuse=18.2.0-1095-ge6528508-1jammy python3-cephfs=18.2.0-1095-ge6528508-1jammy cephfs-shell=18.2.0-1095-ge6528508-1jammy cephfs-top=18.2.0-1095-ge6528508-1jammy cephfs-mirror=18.2.0-1095-ge6528508-1jammy 2023-10-24T23:49:29.886 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-10-24T23:49:29.908 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-24T23:49:30.073 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2023-10-24T23:49:30.073 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2023-10-24T23:49:30.079 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2023-10-24T23:49:30.080 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2023-10-24T23:49:30.222 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages were automatically installed and are no longer required: 2023-10-24T23:49:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-24T23:49:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-24T23:49:30.223 INFO:teuthology.orchestra.run.smithi107.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-24T23:49:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-24T23:49:30.224 INFO:teuthology.orchestra.run.smithi107.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-24T23:49:30.224 INFO:teuthology.orchestra.run.smithi107.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-24T23:49:30.225 INFO:teuthology.orchestra.run.smithi107.stdout:The following additional packages will be installed: 2023-10-24T23:49:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-24T23:49:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-24T23:49:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-24T23:49:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-24T23:49:30.226 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-24T23:49:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-24T23:49:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-24T23:49:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-24T23:49:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-24T23:49:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-24T23:49:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-24T23:49:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-24T23:49:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-24T23:49:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-24T23:49:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-24T23:49:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-24T23:49:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-24T23:49:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-24T23:49:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-24T23:49:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-24T23:49:30.229 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-24T23:49:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-24T23:49:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-24T23:49:30.230 INFO:teuthology.orchestra.run.smithi107.stdout:Suggested packages: 2023-10-24T23:49:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-24T23:49:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-24T23:49:30.230 INFO:teuthology.orchestra.run.smithi107.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-24T23:49:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-24T23:49:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-24T23:49:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-24T23:49:30.231 INFO:teuthology.orchestra.run.smithi107.stdout: python3-watchdog 2023-10-24T23:49:30.231 INFO:teuthology.orchestra.run.smithi107.stdout:Recommended packages: 2023-10-24T23:49:30.232 INFO:teuthology.orchestra.run.smithi107.stdout: btrfs-tools 2023-10-24T23:49:30.235 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2023-10-24T23:49:30.235 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-24T23:49:30.235 INFO:teuthology.orchestra.run.smithi110.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-24T23:49:30.235 INFO:teuthology.orchestra.run.smithi110.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-24T23:49:30.236 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-24T23:49:30.236 INFO:teuthology.orchestra.run.smithi110.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-24T23:49:30.236 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-24T23:49:30.237 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2023-10-24T23:49:30.237 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-24T23:49:30.238 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-24T23:49:30.238 INFO:teuthology.orchestra.run.smithi110.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-24T23:49:30.238 INFO:teuthology.orchestra.run.smithi110.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-24T23:49:30.238 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-24T23:49:30.239 INFO:teuthology.orchestra.run.smithi110.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-24T23:49:30.239 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-24T23:49:30.239 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-24T23:49:30.239 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-24T23:49:30.239 INFO:teuthology.orchestra.run.smithi110.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-24T23:49:30.239 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-24T23:49:30.240 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-24T23:49:30.240 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-24T23:49:30.240 INFO:teuthology.orchestra.run.smithi110.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-24T23:49:30.240 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-24T23:49:30.240 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-24T23:49:30.241 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-24T23:49:30.241 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-24T23:49:30.241 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-24T23:49:30.241 INFO:teuthology.orchestra.run.smithi110.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-24T23:49:30.241 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-24T23:49:30.241 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-24T23:49:30.242 INFO:teuthology.orchestra.run.smithi110.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-24T23:49:30.242 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2023-10-24T23:49:30.242 INFO:teuthology.orchestra.run.smithi110.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-24T23:49:30.243 INFO:teuthology.orchestra.run.smithi110.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-24T23:49:30.243 INFO:teuthology.orchestra.run.smithi110.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-24T23:49:30.243 INFO:teuthology.orchestra.run.smithi110.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-24T23:49:30.243 INFO:teuthology.orchestra.run.smithi110.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-24T23:49:30.243 INFO:teuthology.orchestra.run.smithi110.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-24T23:49:30.244 INFO:teuthology.orchestra.run.smithi110.stdout: python3-watchdog 2023-10-24T23:49:30.244 INFO:teuthology.orchestra.run.smithi110.stdout:Recommended packages: 2023-10-24T23:49:30.244 INFO:teuthology.orchestra.run.smithi110.stdout: btrfs-tools 2023-10-24T23:49:30.326 INFO:teuthology.orchestra.run.smithi107.stdout:The following NEW packages will be installed: 2023-10-24T23:49:30.327 INFO:teuthology.orchestra.run.smithi107.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-24T23:49:30.327 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-24T23:49:30.327 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-24T23:49:30.327 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-24T23:49:30.328 INFO:teuthology.orchestra.run.smithi107.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-24T23:49:30.328 INFO:teuthology.orchestra.run.smithi107.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-24T23:49:30.328 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-24T23:49:30.329 INFO:teuthology.orchestra.run.smithi107.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-24T23:49:30.329 INFO:teuthology.orchestra.run.smithi107.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-24T23:49:30.329 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-24T23:49:30.330 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-24T23:49:30.330 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-24T23:49:30.330 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-24T23:49:30.330 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-24T23:49:30.330 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-24T23:49:30.331 INFO:teuthology.orchestra.run.smithi107.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-24T23:49:30.331 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-24T23:49:30.331 INFO:teuthology.orchestra.run.smithi107.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-24T23:49:30.331 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-24T23:49:30.331 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-24T23:49:30.332 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-24T23:49:30.332 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-24T23:49:30.332 INFO:teuthology.orchestra.run.smithi107.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-24T23:49:30.332 INFO:teuthology.orchestra.run.smithi107.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-24T23:49:30.332 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-24T23:49:30.332 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-24T23:49:30.333 INFO:teuthology.orchestra.run.smithi107.stdout: socat xmlstarlet 2023-10-24T23:49:30.337 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2023-10-24T23:49:30.337 INFO:teuthology.orchestra.run.smithi110.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-24T23:49:30.337 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-24T23:49:30.338 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-24T23:49:30.339 INFO:teuthology.orchestra.run.smithi110.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-24T23:49:30.339 INFO:teuthology.orchestra.run.smithi110.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-24T23:49:30.339 INFO:teuthology.orchestra.run.smithi110.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-24T23:49:30.340 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-24T23:49:30.340 INFO:teuthology.orchestra.run.smithi110.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-24T23:49:30.340 INFO:teuthology.orchestra.run.smithi110.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-24T23:49:30.340 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-24T23:49:30.340 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-24T23:49:30.341 INFO:teuthology.orchestra.run.smithi110.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-24T23:49:30.341 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-24T23:49:30.341 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-24T23:49:30.341 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-24T23:49:30.341 INFO:teuthology.orchestra.run.smithi110.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-24T23:49:30.341 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-24T23:49:30.342 INFO:teuthology.orchestra.run.smithi110.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-24T23:49:30.342 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-24T23:49:30.342 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-24T23:49:30.342 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-24T23:49:30.342 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-24T23:49:30.343 INFO:teuthology.orchestra.run.smithi110.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-24T23:49:30.343 INFO:teuthology.orchestra.run.smithi110.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-24T23:49:30.343 INFO:teuthology.orchestra.run.smithi110.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-24T23:49:30.343 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-24T23:49:30.343 INFO:teuthology.orchestra.run.smithi110.stdout: socat xmlstarlet 2023-10-24T23:49:30.388 INFO:teuthology.orchestra.run.smithi107.stdout:0 upgraded, 115 newly installed, 0 to remove and 257 not upgraded. 2023-10-24T23:49:30.388 INFO:teuthology.orchestra.run.smithi107.stdout:Need to get 181 MB of archives. 2023-10-24T23:49:30.388 INFO:teuthology.orchestra.run.smithi107.stdout:After this operation, 812 MB of additional disk space will be used. 2023-10-24T23:49:30.388 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-24T23:49:30.395 INFO:teuthology.orchestra.run.smithi110.stdout:0 upgraded, 115 newly installed, 0 to remove and 257 not upgraded. 2023-10-24T23:49:30.395 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 181 MB of archives. 2023-10-24T23:49:30.396 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 812 MB of additional disk space will be used. 2023-10-24T23:49:30.396 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-24T23:49:30.405 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-24T23:49:30.412 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-24T23:49:30.415 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-24T23:49:30.422 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-24T23:49:30.455 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-24T23:49:30.462 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-24T23:49:30.464 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-24T23:49:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.0-1095-ge6528508-1jammy [3,417 kB] 2023-10-24T23:49:30.471 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-24T23:49:30.481 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-24T23:49:30.483 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-24T23:49:30.487 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-24T23:49:30.489 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-24T23:49:30.492 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-24T23:49:30.498 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-24T23:49:30.538 INFO:teuthology.orchestra.run.smithi107.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-24T23:49:30.542 INFO:teuthology.orchestra.run.smithi107.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-24T23:49:30.543 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-24T23:49:30.547 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-24T23:49:30.553 INFO:teuthology.orchestra.run.smithi107.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-24T23:49:30.557 INFO:teuthology.orchestra.run.smithi107.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-24T23:49:30.558 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-24T23:49:30.558 INFO:teuthology.orchestra.run.smithi107.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-24T23:49:30.560 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.0-1095-ge6528508-1jammy [3,417 kB] 2023-10-24T23:49:30.561 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-24T23:49:30.562 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-24T23:49:30.565 INFO:teuthology.orchestra.run.smithi107.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-24T23:49:30.566 INFO:teuthology.orchestra.run.smithi107.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-24T23:49:30.567 INFO:teuthology.orchestra.run.smithi107.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-24T23:49:30.568 INFO:teuthology.orchestra.run.smithi107.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-24T23:49:30.568 INFO:teuthology.orchestra.run.smithi107.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-24T23:49:30.570 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-24T23:49:30.570 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-24T23:49:30.571 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-24T23:49:30.572 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-24T23:49:30.572 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-24T23:49:30.575 INFO:teuthology.orchestra.run.smithi107.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-24T23:49:30.575 INFO:teuthology.orchestra.run.smithi107.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-24T23:49:30.580 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-24T23:49:30.580 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-24T23:49:30.592 INFO:teuthology.orchestra.run.smithi107.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-24T23:49:30.592 INFO:teuthology.orchestra.run.smithi107.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-24T23:49:30.592 INFO:teuthology.orchestra.run.smithi107.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-24T23:49:30.595 INFO:teuthology.orchestra.run.smithi107.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-24T23:49:30.595 INFO:teuthology.orchestra.run.smithi107.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-24T23:49:30.596 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-24T23:49:30.597 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-24T23:49:30.597 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-24T23:49:30.600 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-24T23:49:30.600 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-24T23:49:30.680 INFO:teuthology.orchestra.run.smithi107.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.0-1095-ge6528508-1jammy [3,104 kB] 2023-10-24T23:49:30.778 INFO:teuthology.orchestra.run.smithi107.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.0-1095-ge6528508-1jammy [670 kB] 2023-10-24T23:49:30.801 INFO:teuthology.orchestra.run.smithi107.stdout:Get:29 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.0-1095-ge6528508-1jammy [356 kB] 2023-10-24T23:49:30.805 INFO:teuthology.orchestra.run.smithi107.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-24T23:49:30.805 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-24T23:49:30.805 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-24T23:49:30.806 INFO:teuthology.orchestra.run.smithi107.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-24T23:49:30.807 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-24T23:49:30.807 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-24T23:49:30.807 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-24T23:49:30.808 INFO:teuthology.orchestra.run.smithi107.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-24T23:49:30.808 INFO:teuthology.orchestra.run.smithi107.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-24T23:49:30.809 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-24T23:49:30.809 INFO:teuthology.orchestra.run.smithi107.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-24T23:49:30.811 INFO:teuthology.orchestra.run.smithi107.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-24T23:49:30.812 INFO:teuthology.orchestra.run.smithi107.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.0-1095-ge6528508-1jammy [32.8 kB] 2023-10-24T23:49:30.813 INFO:teuthology.orchestra.run.smithi107.stdout:Get:37 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.0-1095-ge6528508-1jammy [180 kB] 2023-10-24T23:49:30.819 INFO:teuthology.orchestra.run.smithi107.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.0-1095-ge6528508-1jammy [61.5 kB] 2023-10-24T23:49:30.821 INFO:teuthology.orchestra.run.smithi107.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.0-1095-ge6528508-1jammy [330 kB] 2023-10-24T23:49:30.833 INFO:teuthology.orchestra.run.smithi107.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.0-1095-ge6528508-1jammy [5,884 kB] 2023-10-24T23:49:30.989 INFO:teuthology.orchestra.run.smithi107.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-24T23:49:30.989 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-24T23:49:30.990 INFO:teuthology.orchestra.run.smithi107.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-24T23:49:30.990 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-24T23:49:31.024 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-24T23:49:31.025 INFO:teuthology.orchestra.run.smithi107.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-24T23:49:31.025 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-24T23:49:31.026 INFO:teuthology.orchestra.run.smithi107.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-24T23:49:31.026 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-24T23:49:31.026 INFO:teuthology.orchestra.run.smithi107.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-24T23:49:31.039 INFO:teuthology.orchestra.run.smithi107.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-24T23:49:31.040 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-24T23:49:31.041 INFO:teuthology.orchestra.run.smithi107.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.0-1095-ge6528508-1jammy [112 kB] 2023-10-24T23:49:31.044 INFO:teuthology.orchestra.run.smithi107.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.0-1095-ge6528508-1jammy [442 kB] 2023-10-24T23:49:31.051 INFO:teuthology.orchestra.run.smithi107.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-24T23:49:31.052 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-24T23:49:31.054 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-24T23:49:31.055 INFO:teuthology.orchestra.run.smithi107.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-24T23:49:31.056 INFO:teuthology.orchestra.run.smithi110.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-24T23:49:31.057 INFO:teuthology.orchestra.run.smithi107.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-24T23:49:31.059 INFO:teuthology.orchestra.run.smithi110.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-24T23:49:31.059 INFO:teuthology.orchestra.run.smithi107.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.0-1095-ge6528508-1jammy [22.0 MB] 2023-10-24T23:49:31.060 INFO:teuthology.orchestra.run.smithi107.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-24T23:49:31.060 INFO:teuthology.orchestra.run.smithi107.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-24T23:49:31.061 INFO:teuthology.orchestra.run.smithi110.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-24T23:49:31.066 INFO:teuthology.orchestra.run.smithi107.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-24T23:49:31.066 INFO:teuthology.orchestra.run.smithi110.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-24T23:49:31.067 INFO:teuthology.orchestra.run.smithi107.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-24T23:49:31.068 INFO:teuthology.orchestra.run.smithi110.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-24T23:49:31.071 INFO:teuthology.orchestra.run.smithi110.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-24T23:49:31.072 INFO:teuthology.orchestra.run.smithi107.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-24T23:49:31.144 INFO:teuthology.orchestra.run.smithi110.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-24T23:49:31.144 INFO:teuthology.orchestra.run.smithi107.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-24T23:49:31.145 INFO:teuthology.orchestra.run.smithi110.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-24T23:49:31.145 INFO:teuthology.orchestra.run.smithi107.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-24T23:49:31.146 INFO:teuthology.orchestra.run.smithi107.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-24T23:49:31.147 INFO:teuthology.orchestra.run.smithi110.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-24T23:49:31.147 INFO:teuthology.orchestra.run.smithi110.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-24T23:49:31.147 INFO:teuthology.orchestra.run.smithi110.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-24T23:49:31.148 INFO:teuthology.orchestra.run.smithi107.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-24T23:49:31.148 INFO:teuthology.orchestra.run.smithi107.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-24T23:49:31.148 INFO:teuthology.orchestra.run.smithi107.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-24T23:49:31.148 INFO:teuthology.orchestra.run.smithi107.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-24T23:49:31.149 INFO:teuthology.orchestra.run.smithi107.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-24T23:49:31.150 INFO:teuthology.orchestra.run.smithi107.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-24T23:49:31.156 INFO:teuthology.orchestra.run.smithi110.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-24T23:49:31.156 INFO:teuthology.orchestra.run.smithi110.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-24T23:49:31.157 INFO:teuthology.orchestra.run.smithi110.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-24T23:49:31.159 INFO:teuthology.orchestra.run.smithi110.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-24T23:49:31.162 INFO:teuthology.orchestra.run.smithi110.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-24T23:49:31.163 INFO:teuthology.orchestra.run.smithi107.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-24T23:49:31.168 INFO:teuthology.orchestra.run.smithi107.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-24T23:49:31.176 INFO:teuthology.orchestra.run.smithi110.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-24T23:49:31.186 INFO:teuthology.orchestra.run.smithi107.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-24T23:49:31.188 INFO:teuthology.orchestra.run.smithi107.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-24T23:49:31.188 INFO:teuthology.orchestra.run.smithi107.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-24T23:49:31.194 INFO:teuthology.orchestra.run.smithi110.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-24T23:49:31.196 INFO:teuthology.orchestra.run.smithi110.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-24T23:49:31.196 INFO:teuthology.orchestra.run.smithi110.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-24T23:49:31.204 INFO:teuthology.orchestra.run.smithi107.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-24T23:49:31.212 INFO:teuthology.orchestra.run.smithi110.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-24T23:49:31.890 INFO:teuthology.orchestra.run.smithi107.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.0-1095-ge6528508-1jammy [6,838 kB] 2023-10-24T23:49:32.062 INFO:teuthology.orchestra.run.smithi107.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.0-1095-ge6528508-1jammy [237 kB] 2023-10-24T23:49:32.070 INFO:teuthology.orchestra.run.smithi107.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.0-1095-ge6528508-1jammy [130 kB] 2023-10-24T23:49:32.076 INFO:teuthology.orchestra.run.smithi107.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.0-1095-ge6528508-1jammy [1,482 kB] 2023-10-24T23:49:32.126 INFO:teuthology.orchestra.run.smithi107.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.0-1095-ge6528508-1jammy [9,875 kB] 2023-10-24T23:49:32.476 INFO:teuthology.orchestra.run.smithi107.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.0-1095-ge6528508-1jammy [31.5 MB] 2023-10-24T23:49:32.477 INFO:teuthology.orchestra.run.smithi110.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.0-1095-ge6528508-1jammy [3,104 kB] 2023-10-24T23:49:32.563 INFO:teuthology.orchestra.run.smithi107.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-24T23:49:32.564 INFO:teuthology.orchestra.run.smithi110.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-24T23:49:32.564 INFO:teuthology.orchestra.run.smithi107.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-24T23:49:32.565 INFO:teuthology.orchestra.run.smithi110.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-24T23:49:32.565 INFO:teuthology.orchestra.run.smithi110.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-24T23:49:32.566 INFO:teuthology.orchestra.run.smithi110.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-24T23:49:32.566 INFO:teuthology.orchestra.run.smithi107.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-24T23:49:32.566 INFO:teuthology.orchestra.run.smithi107.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-24T23:49:32.567 INFO:teuthology.orchestra.run.smithi107.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-24T23:49:32.567 INFO:teuthology.orchestra.run.smithi110.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-24T23:49:32.569 INFO:teuthology.orchestra.run.smithi110.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-24T23:49:32.569 INFO:teuthology.orchestra.run.smithi107.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-24T23:49:32.570 INFO:teuthology.orchestra.run.smithi110.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-24T23:49:32.570 INFO:teuthology.orchestra.run.smithi107.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-24T23:49:32.575 INFO:teuthology.orchestra.run.smithi107.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-24T23:49:32.575 INFO:teuthology.orchestra.run.smithi110.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-24T23:49:32.577 INFO:teuthology.orchestra.run.smithi107.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-24T23:49:32.578 INFO:teuthology.orchestra.run.smithi110.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-24T23:49:32.581 INFO:teuthology.orchestra.run.smithi107.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-24T23:49:32.581 INFO:teuthology.orchestra.run.smithi110.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-24T23:49:32.597 INFO:teuthology.orchestra.run.smithi107.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-24T23:49:32.597 INFO:teuthology.orchestra.run.smithi110.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-24T23:49:32.602 INFO:teuthology.orchestra.run.smithi107.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-24T23:49:32.602 INFO:teuthology.orchestra.run.smithi110.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-24T23:49:32.606 INFO:teuthology.orchestra.run.smithi107.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-24T23:49:32.606 INFO:teuthology.orchestra.run.smithi110.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-24T23:49:32.606 INFO:teuthology.orchestra.run.smithi107.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-24T23:49:32.610 INFO:teuthology.orchestra.run.smithi110.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-24T23:49:32.613 INFO:teuthology.orchestra.run.smithi107.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-24T23:49:32.613 INFO:teuthology.orchestra.run.smithi110.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-24T23:49:32.617 INFO:teuthology.orchestra.run.smithi107.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-24T23:49:32.618 INFO:teuthology.orchestra.run.smithi110.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-24T23:49:32.618 INFO:teuthology.orchestra.run.smithi107.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-24T23:49:32.619 INFO:teuthology.orchestra.run.smithi110.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-24T23:49:32.620 INFO:teuthology.orchestra.run.smithi107.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-24T23:49:32.621 INFO:teuthology.orchestra.run.smithi110.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-24T23:49:32.630 INFO:teuthology.orchestra.run.smithi107.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-24T23:49:32.631 INFO:teuthology.orchestra.run.smithi107.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-24T23:49:32.632 INFO:teuthology.orchestra.run.smithi110.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-24T23:49:32.632 INFO:teuthology.orchestra.run.smithi110.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-24T23:49:32.632 INFO:teuthology.orchestra.run.smithi107.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-24T23:49:32.633 INFO:teuthology.orchestra.run.smithi110.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-24T23:49:33.730 INFO:teuthology.orchestra.run.smithi107.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.0-1095-ge6528508-1jammy [14.1 kB] 2023-10-24T23:49:33.731 INFO:teuthology.orchestra.run.smithi107.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.0-1095-ge6528508-1jammy [824 kB] 2023-10-24T23:49:33.748 INFO:teuthology.orchestra.run.smithi107.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.0-1095-ge6528508-1jammy [2,170 kB] 2023-10-24T23:49:33.766 INFO:teuthology.orchestra.run.smithi110.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.0-1095-ge6528508-1jammy [670 kB] 2023-10-24T23:49:33.786 INFO:teuthology.orchestra.run.smithi107.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.0-1095-ge6528508-1jammy [235 kB] 2023-10-24T23:49:33.790 INFO:teuthology.orchestra.run.smithi107.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.0-1095-ge6528508-1jammy [138 kB] 2023-10-24T23:49:33.792 INFO:teuthology.orchestra.run.smithi107.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.0-1095-ge6528508-1jammy [1,906 kB] 2023-10-24T23:49:33.817 INFO:teuthology.orchestra.run.smithi107.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.0-1095-ge6528508-1jammy [8,625 kB] 2023-10-24T23:49:34.037 INFO:teuthology.orchestra.run.smithi110.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.0-1095-ge6528508-1jammy [356 kB] 2023-10-24T23:49:34.097 INFO:teuthology.orchestra.run.smithi107.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.0-1095-ge6528508-1jammy [14.2 kB] 2023-10-24T23:49:34.097 INFO:teuthology.orchestra.run.smithi107.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.0-1095-ge6528508-1jammy [46.5 MB] 2023-10-24T23:49:34.211 INFO:teuthology.orchestra.run.smithi110.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.0-1095-ge6528508-1jammy [32.8 kB] 2023-10-24T23:49:34.216 INFO:teuthology.orchestra.run.smithi110.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.0-1095-ge6528508-1jammy [180 kB] 2023-10-24T23:49:34.297 INFO:teuthology.orchestra.run.smithi110.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.0-1095-ge6528508-1jammy [61.5 kB] 2023-10-24T23:49:34.340 INFO:teuthology.orchestra.run.smithi110.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.0-1095-ge6528508-1jammy [330 kB] 2023-10-24T23:49:34.490 INFO:teuthology.orchestra.run.smithi110.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.0-1095-ge6528508-1jammy [5,884 kB] 2023-10-24T23:49:36.016 INFO:teuthology.orchestra.run.smithi107.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.0-1095-ge6528508-1jammy [120 kB] 2023-10-24T23:49:36.018 INFO:teuthology.orchestra.run.smithi107.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.2.0-1095-ge6528508-1jammy [228 kB] 2023-10-24T23:49:36.023 INFO:teuthology.orchestra.run.smithi107.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.2.0-1095-ge6528508-1jammy [34.0 kB] 2023-10-24T23:49:36.024 INFO:teuthology.orchestra.run.smithi107.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.2.0-1095-ge6528508-1jammy [23.3 kB] 2023-10-24T23:49:36.082 INFO:teuthology.orchestra.run.smithi107.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.0-1095-ge6528508-1jammy [38.3 kB] 2023-10-24T23:49:36.083 INFO:teuthology.orchestra.run.smithi107.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.0-1095-ge6528508-1jammy [11.6 MB] 2023-10-24T23:49:36.369 INFO:teuthology.orchestra.run.smithi107.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.0-1095-ge6528508-1jammy [92.7 kB] 2023-10-24T23:49:36.594 INFO:teuthology.orchestra.run.smithi110.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.0-1095-ge6528508-1jammy [112 kB] 2023-10-24T23:49:36.609 INFO:teuthology.orchestra.run.smithi110.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.0-1095-ge6528508-1jammy [442 kB] 2023-10-24T23:49:36.703 INFO:teuthology.orchestra.run.smithi110.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.0-1095-ge6528508-1jammy [22.0 MB] 2023-10-24T23:49:36.917 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 181 MB in 6s (30.0 MB/s) 2023-10-24T23:49:37.055 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-24T23:49:37.142 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165984 files and directories currently installed.) 2023-10-24T23:49:37.148 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-24T23:49:37.203 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-24T23:49:37.481 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-24T23:49:37.508 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-24T23:49:37.548 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-24T23:49:37.801 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cmd2. 2023-10-24T23:49:37.827 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-24T23:49:37.867 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-24T23:49:38.162 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-24T23:49:38.189 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-24T23:49:38.271 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-24T23:49:38.599 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-24T23:49:38.625 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-24T23:49:38.665 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-24T23:49:38.986 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-24T23:49:39.012 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-24T23:49:39.052 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-24T23:49:39.330 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-24T23:49:39.357 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-24T23:49:39.397 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-24T23:49:39.834 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-24T23:49:39.860 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-24T23:49:39.908 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:49:40.312 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-24T23:49:40.339 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-24T23:49:40.370 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:49:40.508 INFO:teuthology.orchestra.run.smithi110.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.0-1095-ge6528508-1jammy [6,838 kB] 2023-10-24T23:49:40.690 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-24T23:49:40.717 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-24T23:49:40.757 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:49:41.110 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-24T23:49:41.137 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-24T23:49:41.177 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-24T23:49:41.412 INFO:teuthology.orchestra.run.smithi110.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.0-1095-ge6528508-1jammy [237 kB] 2023-10-24T23:49:41.419 INFO:teuthology.orchestra.run.smithi110.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.0-1095-ge6528508-1jammy [130 kB] 2023-10-24T23:49:41.451 INFO:teuthology.orchestra.run.smithi110.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.0-1095-ge6528508-1jammy [1,482 kB] 2023-10-24T23:49:41.455 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package librados2. 2023-10-24T23:49:41.481 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../011-librados2_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:41.513 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librados2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:41.637 INFO:teuthology.orchestra.run.smithi110.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.0-1095-ge6528508-1jammy [9,875 kB] 2023-10-24T23:49:41.949 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package librbd1. 2023-10-24T23:49:41.976 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../012-librbd1_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:42.016 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librbd1 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:42.444 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libcephfs2. 2023-10-24T23:49:42.471 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../013-libcephfs2_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:42.510 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libcephfs2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:42.763 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rados. 2023-10-24T23:49:42.778 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../014-python3-rados_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:42.813 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rados (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:42.940 INFO:teuthology.orchestra.run.smithi110.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.0-1095-ge6528508-1jammy [31.5 MB] 2023-10-24T23:49:43.066 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-24T23:49:43.093 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:49:43.132 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-ceph-argparse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:43.385 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cephfs. 2023-10-24T23:49:43.401 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../016-python3-cephfs_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:43.443 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cephfs (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:43.788 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-24T23:49:43.815 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../017-python3-ceph-common_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:49:43.855 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-ceph-common (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:44.183 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-prettytable. 2023-10-24T23:49:44.209 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-24T23:49:44.249 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-24T23:49:44.510 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rbd. 2023-10-24T23:49:44.537 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../019-python3-rbd_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:44.577 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rbd (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:44.889 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-24T23:49:44.915 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-24T23:49:44.947 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-24T23:49:45.266 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package librgw2. 2023-10-24T23:49:45.292 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../021-librgw2_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:45.333 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking librgw2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:45.821 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rgw. 2023-10-24T23:49:45.847 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../022-python3-rgw_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:45.887 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rgw (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:46.011 INFO:teuthology.orchestra.run.smithi110.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.0-1095-ge6528508-1jammy [14.1 kB] 2023-10-24T23:49:46.012 INFO:teuthology.orchestra.run.smithi110.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.0-1095-ge6528508-1jammy [824 kB] 2023-10-24T23:49:46.028 INFO:teuthology.orchestra.run.smithi110.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.0-1095-ge6528508-1jammy [2,170 kB] 2023-10-24T23:49:46.067 INFO:teuthology.orchestra.run.smithi110.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.0-1095-ge6528508-1jammy [235 kB] 2023-10-24T23:49:46.085 INFO:teuthology.orchestra.run.smithi110.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.0-1095-ge6528508-1jammy [138 kB] 2023-10-24T23:49:46.087 INFO:teuthology.orchestra.run.smithi110.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.0-1095-ge6528508-1jammy [1,906 kB] 2023-10-24T23:49:46.174 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-24T23:49:46.195 INFO:teuthology.orchestra.run.smithi110.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.0-1095-ge6528508-1jammy [8,625 kB] 2023-10-24T23:49:46.200 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-24T23:49:46.240 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-24T23:49:46.551 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libradosstriper1. 2023-10-24T23:49:46.578 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../024-libradosstriper1_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:46.618 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libradosstriper1 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:46.633 INFO:teuthology.orchestra.run.smithi110.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.0-1095-ge6528508-1jammy [14.2 kB] 2023-10-24T23:49:46.633 INFO:teuthology.orchestra.run.smithi110.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.0-1095-ge6528508-1jammy [46.5 MB] 2023-10-24T23:49:47.038 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-common. 2023-10-24T23:49:47.065 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../025-ceph-common_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:47.105 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-common (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:48.725 INFO:teuthology.orchestra.run.smithi110.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.0-1095-ge6528508-1jammy [120 kB] 2023-10-24T23:49:48.727 INFO:teuthology.orchestra.run.smithi110.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.2.0-1095-ge6528508-1jammy [228 kB] 2023-10-24T23:49:48.728 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-base. 2023-10-24T23:49:48.732 INFO:teuthology.orchestra.run.smithi110.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.2.0-1095-ge6528508-1jammy [34.0 kB] 2023-10-24T23:49:48.733 INFO:teuthology.orchestra.run.smithi110.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.2.0-1095-ge6528508-1jammy [23.3 kB] 2023-10-24T23:49:48.734 INFO:teuthology.orchestra.run.smithi110.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.0-1095-ge6528508-1jammy [38.3 kB] 2023-10-24T23:49:48.735 INFO:teuthology.orchestra.run.smithi110.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.0-1095-ge6528508-1jammy [11.6 MB] 2023-10-24T23:49:48.755 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../026-ceph-base_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:48.803 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-base (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:49.037 INFO:teuthology.orchestra.run.smithi110.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2023-10-23-1148-reef/e652850897f60c32cb9cc05022cf9f93e3b00db1/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.0-1095-ge6528508-1jammy [92.7 kB] 2023-10-24T23:49:49.499 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-24T23:49:49.525 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-24T23:49:49.565 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-24T23:49:49.594 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 181 MB in 19s (9,677 kB/s) 2023-10-24T23:49:49.735 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-24T23:49:49.826 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cheroot. 2023-10-24T23:49:49.834 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165984 files and directories currently installed.) 2023-10-24T23:49:49.839 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-24T23:49:49.841 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-24T23:49:49.876 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-24T23:49:49.883 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-24T23:49:50.178 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-24T23:49:50.196 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-24T23:49:50.205 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-24T23:49:50.223 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-24T23:49:50.245 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-24T23:49:50.263 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-24T23:49:50.531 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cmd2. 2023-10-24T23:49:50.547 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-24T23:49:50.558 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-24T23:49:50.581 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-24T23:49:50.584 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-24T23:49:50.624 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-24T23:49:50.859 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-24T23:49:50.874 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-24T23:49:50.927 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-24T23:49:50.951 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-24T23:49:50.954 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-24T23:49:50.994 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-24T23:49:51.280 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-tempora. 2023-10-24T23:49:51.287 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-24T23:49:51.307 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-24T23:49:51.314 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-24T23:49:51.338 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-24T23:49:51.354 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-24T23:49:51.641 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-portend. 2023-10-24T23:49:51.668 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-24T23:49:51.708 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-24T23:49:51.732 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-24T23:49:51.759 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-24T23:49:51.799 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-24T23:49:52.011 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-24T23:49:52.037 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-24T23:49:52.077 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-24T23:49:52.094 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-24T23:49:52.120 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-24T23:49:52.152 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-24T23:49:52.364 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-24T23:49:52.379 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-24T23:49:52.414 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-24T23:49:52.589 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-24T23:49:52.663 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:49:52.759 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-natsort. 2023-10-24T23:49:52.786 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-24T23:49:52.826 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-24T23:49:53.058 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-24T23:49:53.085 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-24T23:49:53.125 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:49:53.138 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-logutils. 2023-10-24T23:49:53.164 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-24T23:49:53.204 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-24T23:49:53.437 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-24T23:49:53.463 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-24T23:49:53.465 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-24T23:49:53.492 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-24T23:49:53.503 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:49:53.532 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-24T23:49:53.843 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-mako. 2023-10-24T23:49:53.870 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-24T23:49:53.873 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-24T23:49:53.899 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-24T23:49:53.909 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-24T23:49:53.943 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-24T23:49:54.222 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-24T23:49:54.248 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-24T23:49:54.288 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-24T23:49:54.442 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librados2. 2023-10-24T23:49:54.469 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../011-librados2_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:54.509 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librados2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:54.591 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-24T23:49:54.618 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-24T23:49:54.657 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-24T23:49:54.927 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-webob. 2023-10-24T23:49:54.937 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librbd1. 2023-10-24T23:49:54.954 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-24T23:49:54.963 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../012-librbd1_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:54.994 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-24T23:49:55.003 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librbd1 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:55.348 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-waitress. 2023-10-24T23:49:55.357 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-24T23:49:55.411 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-24T23:49:55.468 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs2. 2023-10-24T23:49:55.491 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../013-libcephfs2_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:55.533 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:55.708 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-tempita. 2023-10-24T23:49:55.735 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-24T23:49:55.775 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-24T23:49:55.808 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rados. 2023-10-24T23:49:55.824 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../014-python3-rados_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:55.859 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rados (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:56.078 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-paste. 2023-10-24T23:49:56.104 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-24T23:49:56.144 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-24T23:49:56.153 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-24T23:49:56.179 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:49:56.220 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-argparse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:56.472 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cephfs. 2023-10-24T23:49:56.473 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-24T23:49:56.487 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../016-python3-cephfs_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:56.488 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-24T23:49:56.530 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cephfs (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:56.531 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-24T23:49:56.850 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-24T23:49:56.850 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-24T23:49:56.876 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../017-python3-ceph-common_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:49:56.877 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-24T23:49:56.917 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-ceph-common (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:56.917 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-24T23:49:57.220 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-webtest. 2023-10-24T23:49:57.245 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-prettytable. 2023-10-24T23:49:57.247 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-24T23:49:57.271 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-24T23:49:57.278 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-24T23:49:57.311 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-24T23:49:57.540 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pecan. 2023-10-24T23:49:57.547 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rbd. 2023-10-24T23:49:57.562 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../019-python3-rbd_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:57.566 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-24T23:49:57.597 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rbd (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:57.606 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-24T23:49:57.928 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-certifi. 2023-10-24T23:49:57.942 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-24T23:49:57.954 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-24T23:49:57.968 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-24T23:49:57.995 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-24T23:49:58.008 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-24T23:49:58.311 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package librgw2. 2023-10-24T23:49:58.337 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../021-librgw2_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:58.339 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-idna. 2023-10-24T23:49:58.366 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-24T23:49:58.378 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking librgw2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:58.406 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-24T23:49:58.717 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-urllib3. 2023-10-24T23:49:58.744 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-24T23:49:58.784 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-24T23:49:58.872 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rgw. 2023-10-24T23:49:58.887 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../022-python3-rgw_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:58.930 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rgw (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:59.283 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-24T23:49:59.304 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-requests. 2023-10-24T23:49:59.309 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-24T23:49:59.331 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-24T23:49:59.350 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-24T23:49:59.370 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-24T23:49:59.657 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-24T23:49:59.669 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libradosstriper1. 2023-10-24T23:49:59.684 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-24T23:49:59.696 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../024-libradosstriper1_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:49:59.723 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-24T23:49:59.747 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libradosstriper1 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:49:59.985 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-24T23:50:00.000 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:00.034 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-modules-core (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:00.056 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-common. 2023-10-24T23:50:00.082 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../025-ceph-common_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:00.114 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-common (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:00.373 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-24T23:50:00.400 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:00.440 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libsqlite3-mod-ceph (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:00.751 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-openssl. 2023-10-24T23:50:00.777 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-24T23:50:00.817 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-24T23:50:01.163 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr. 2023-10-24T23:50:01.189 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../058-ceph-mgr_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:01.229 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:01.353 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-base. 2023-10-24T23:50:01.380 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../026-ceph-base_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:01.429 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-base (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:01.557 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mon. 2023-10-24T23:50:01.584 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../059-ceph-mon_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:01.624 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mon (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:02.108 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-24T23:50:02.134 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-24T23:50:02.166 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-24T23:50:02.494 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cheroot. 2023-10-24T23:50:02.520 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-24T23:50:02.527 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-osd. 2023-10-24T23:50:02.554 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../060-ceph-osd_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:02.560 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-24T23:50:02.585 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-osd (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:02.905 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-24T23:50:02.932 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-24T23:50:02.972 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-24T23:50:03.258 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-24T23:50:03.285 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-24T23:50:03.325 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-24T23:50:03.595 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-24T23:50:03.621 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-24T23:50:03.661 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-24T23:50:04.514 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-tempora. 2023-10-24T23:50:04.541 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-24T23:50:04.581 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-24T23:50:04.850 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-portend. 2023-10-24T23:50:04.877 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-24T23:50:04.881 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph. 2023-10-24T23:50:04.907 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../061-ceph_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:04.917 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-24T23:50:04.947 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:05.211 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-24T23:50:05.238 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-24T23:50:05.275 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-fuse. 2023-10-24T23:50:05.278 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-24T23:50:05.301 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../062-ceph-fuse_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:05.342 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-fuse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:05.581 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-24T23:50:05.607 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-24T23:50:05.647 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-24T23:50:05.654 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mds. 2023-10-24T23:50:05.680 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../063-ceph-mds_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:05.712 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mds (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:05.993 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-natsort. 2023-10-24T23:50:06.020 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-24T23:50:06.060 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-24T23:50:06.099 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package cephadm. 2023-10-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../064-cephadm_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:06.165 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking cephadm (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:06.379 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-logutils. 2023-10-24T23:50:06.405 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-24T23:50:06.446 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-24T23:50:06.459 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-24T23:50:06.475 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-24T23:50:06.518 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-24T23:50:06.748 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-24T23:50:06.775 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-24T23:50:06.815 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-24T23:50:06.838 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-24T23:50:06.864 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-24T23:50:06.904 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-24T23:50:07.126 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-mako. 2023-10-24T23:50:07.152 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-24T23:50:07.192 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-24T23:50:07.504 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-24T23:50:07.531 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-24T23:50:07.571 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-24T23:50:07.840 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-24T23:50:07.867 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-24T23:50:07.907 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-24T23:50:07.918 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-babel. 2023-10-24T23:50:07.945 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-24T23:50:07.984 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-24T23:50:08.185 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webob. 2023-10-24T23:50:08.211 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-24T23:50:08.251 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-24T23:50:08.321 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-jinja2. 2023-10-24T23:50:08.348 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-24T23:50:08.387 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-24T23:50:08.596 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-waitress. 2023-10-24T23:50:08.623 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-24T23:50:08.666 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-24T23:50:08.733 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-24T23:50:08.759 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:08.799 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-cephadm (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:08.974 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-tempita. 2023-10-24T23:50:09.001 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-24T23:50:09.040 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-24T23:50:09.136 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-24T23:50:09.163 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-24T23:50:09.202 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-24T23:50:09.301 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-paste. 2023-10-24T23:50:09.317 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-24T23:50:09.352 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-24T23:50:09.663 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-24T23:50:09.690 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-24T23:50:09.697 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pluggy. 2023-10-24T23:50:09.724 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../071-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-24T23:50:09.730 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-24T23:50:09.764 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-24T23:50:09.974 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-24T23:50:10.001 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-24T23:50:10.032 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-24T23:50:10.050 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-py. 2023-10-24T23:50:10.077 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../072-python3-py_1.10.0-1_all.deb ... 2023-10-24T23:50:10.108 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-24T23:50:10.335 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-webtest. 2023-10-24T23:50:10.362 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-24T23:50:10.396 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-24T23:50:10.402 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-24T23:50:10.422 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../073-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-24T23:50:10.462 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-24T23:50:10.738 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pecan. 2023-10-24T23:50:10.765 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-24T23:50:10.766 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-toml. 2023-10-24T23:50:10.792 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../074-python3-toml_0.10.2-1_all.deb ... 2023-10-24T23:50:10.805 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-24T23:50:10.831 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-24T23:50:11.135 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pytest. 2023-10-24T23:50:11.160 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-certifi. 2023-10-24T23:50:11.161 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../075-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-24T23:50:11.186 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-24T23:50:11.201 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-24T23:50:11.226 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-24T23:50:11.546 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-idna. 2023-10-24T23:50:11.555 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-24T23:50:11.573 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-24T23:50:11.582 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2023-10-24T23:50:11.612 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-24T23:50:11.621 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-24T23:50:11.882 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-urllib3. 2023-10-24T23:50:11.909 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-24T23:50:11.925 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-routes. 2023-10-24T23:50:11.940 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-24T23:50:11.952 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-24T23:50:11.991 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-24T23:50:12.218 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-requests. 2023-10-24T23:50:12.245 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-24T23:50:12.285 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-24T23:50:12.320 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-24T23:50:12.347 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:12.395 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-dashboard (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:12.596 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-24T23:50:12.623 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-24T23:50:12.663 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-24T23:50:12.807 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-24T23:50:12.822 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-24T23:50:12.856 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-24T23:50:12.899 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-24T23:50:12.915 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:12.949 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-modules-core (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:13.329 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-24T23:50:13.356 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:13.368 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-joblib. 2023-10-24T23:50:13.395 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-24T23:50:13.396 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libsqlite3-mod-ceph (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:13.435 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-24T23:50:13.665 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-openssl. 2023-10-24T23:50:13.689 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-24T23:50:13.723 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-24T23:50:13.806 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-24T23:50:13.832 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-24T23:50:13.872 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-24T23:50:14.035 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr. 2023-10-24T23:50:14.062 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../058-ceph-mgr_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:14.126 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:14.150 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-sklearn. 2023-10-24T23:50:14.177 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-24T23:50:14.208 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-24T23:50:14.505 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mon. 2023-10-24T23:50:14.532 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../059-ceph-mon_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:14.571 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mon (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:14.977 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-24T23:50:15.005 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:15.044 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:15.433 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-osd. 2023-10-24T23:50:15.460 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../060-ceph-osd_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:15.499 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-osd (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:15.981 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-cachetools. 2023-10-24T23:50:16.008 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2023-10-24T23:50:16.047 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-24T23:50:16.325 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-24T23:50:16.352 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-24T23:50:16.391 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-24T23:50:16.720 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-24T23:50:16.736 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-24T23:50:16.795 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-24T23:50:17.090 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-rsa. 2023-10-24T23:50:17.117 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2023-10-24T23:50:17.156 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-24T23:50:17.236 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph. 2023-10-24T23:50:17.252 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../061-ceph_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:17.294 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:17.469 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-google-auth. 2023-10-24T23:50:17.496 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2023-10-24T23:50:17.535 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-24T23:50:17.614 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-fuse. 2023-10-24T23:50:17.641 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../062-ceph-fuse_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:17.697 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-fuse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:17.864 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-24T23:50:17.891 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-24T23:50:17.930 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-24T23:50:18.000 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mds. 2023-10-24T23:50:18.028 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../063-ceph-mds_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:18.067 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mds (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:18.208 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-websocket. 2023-10-24T23:50:18.235 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2023-10-24T23:50:18.274 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-24T23:50:18.412 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephadm. 2023-10-24T23:50:18.439 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../064-cephadm_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:18.487 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephadm (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:18.595 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-24T23:50:18.622 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-24T23:50:18.661 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-24T23:50:18.798 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-24T23:50:18.825 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-24T23:50:18.867 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-24T23:50:19.159 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-24T23:50:19.186 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-24T23:50:19.225 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-24T23:50:19.284 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-24T23:50:19.311 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:19.350 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-mgr-k8sevents (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:19.670 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-24T23:50:19.697 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-24T23:50:19.736 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-24T23:50:20.056 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-24T23:50:20.084 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-24T23:50:20.114 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-24T23:50:20.231 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-babel. 2023-10-24T23:50:20.258 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-24T23:50:20.298 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-24T23:50:20.400 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package jq. 2023-10-24T23:50:20.416 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-24T23:50:20.459 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-24T23:50:20.634 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-jinja2. 2023-10-24T23:50:20.661 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-24T23:50:20.701 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-24T23:50:20.712 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package socat. 2023-10-24T23:50:20.738 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-24T23:50:20.778 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-24T23:50:21.031 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-24T23:50:21.057 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:21.090 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package xmlstarlet. 2023-10-24T23:50:21.097 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-cephadm (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:21.106 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-24T23:50:21.148 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-24T23:50:21.434 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-24T23:50:21.461 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-24T23:50:21.503 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-test. 2023-10-24T23:50:21.517 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-24T23:50:21.530 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../098-ceph-test_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:21.569 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-test (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:21.820 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pluggy. 2023-10-24T23:50:21.847 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../071-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-24T23:50:21.887 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-24T23:50:22.198 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-py. 2023-10-24T23:50:22.226 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../072-python3-py_1.10.0-1_all.deb ... 2023-10-24T23:50:22.265 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-24T23:50:22.577 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-24T23:50:22.604 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../073-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-24T23:50:22.644 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-24T23:50:22.913 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-toml. 2023-10-24T23:50:22.940 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../074-python3-toml_0.10.2-1_all.deb ... 2023-10-24T23:50:22.979 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-24T23:50:23.282 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pytest. 2023-10-24T23:50:23.309 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../075-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-24T23:50:23.349 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-24T23:50:23.686 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-24T23:50:23.713 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2023-10-24T23:50:23.752 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-24T23:50:23.800 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ceph-volume. 2023-10-24T23:50:23.827 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../099-ceph-volume_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:23.866 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ceph-volume (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:24.055 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-routes. 2023-10-24T23:50:24.082 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-24T23:50:24.122 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-24T23:50:24.254 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-24T23:50:24.281 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../100-cephfs-mirror_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:24.320 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking cephfs-mirror (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:24.459 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-24T23:50:24.486 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:24.525 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-dashboard (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:24.590 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-colorama. 2023-10-24T23:50:24.605 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../101-python3-colorama_0.4.4-1_all.deb ... 2023-10-24T23:50:24.640 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-24T23:50:24.895 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-24T23:50:24.909 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package cephfs-shell. 2023-10-24T23:50:24.922 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-24T23:50:24.936 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../102-cephfs-shell_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:24.962 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-24T23:50:25.200 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking cephfs-shell (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:25.461 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package cephfs-top. 2023-10-24T23:50:25.489 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../103-cephfs-top_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:25.528 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking cephfs-top (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:25.749 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-joblib. 2023-10-24T23:50:25.776 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-24T23:50:25.789 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-24T23:50:25.815 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-24T23:50:25.816 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../104-libcephfs-dev_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:25.855 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libcephfs-dev (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:26.127 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-24T23:50:26.154 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-24T23:50:26.183 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package nvme-cli. 2023-10-24T23:50:26.185 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-24T23:50:26.210 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../105-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-24T23:50:26.249 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-24T23:50:26.480 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-sklearn. 2023-10-24T23:50:26.579 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-24T23:50:26.580 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../106-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-24T23:50:26.580 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-24T23:50:26.580 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-24T23:50:26.595 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-24T23:50:27.016 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pastescript. 2023-10-24T23:50:27.043 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2023-10-24T23:50:27.082 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-24T23:50:27.284 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-24T23:50:27.309 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:27.349 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:27.402 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-psutil. 2023-10-24T23:50:27.431 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-24T23:50:27.468 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-24T23:50:27.829 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pygments. 2023-10-24T23:50:27.857 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../109-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-24T23:50:27.904 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-24T23:50:28.318 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-cachetools. 2023-10-24T23:50:28.345 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2023-10-24T23:50:28.349 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-24T23:50:28.378 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-24T23:50:28.385 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-24T23:50:28.416 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-24T23:50:28.662 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-24T23:50:28.678 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-24T23:50:28.702 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package python3-simplejson. 2023-10-24T23:50:28.712 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-24T23:50:28.730 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-24T23:50:28.768 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-24T23:50:29.008 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-24T23:50:29.023 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-24T23:50:29.037 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-24T23:50:29.057 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-24T23:50:29.065 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-24T23:50:29.095 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-24T23:50:29.419 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-rsa. 2023-10-24T23:50:29.446 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2023-10-24T23:50:29.485 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-24T23:50:29.798 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-google-auth. 2023-10-24T23:50:29.825 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2023-10-24T23:50:29.841 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package radosgw. 2023-10-24T23:50:29.864 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-24T23:50:29.869 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../113-radosgw_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:29.908 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking radosgw (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:30.159 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-24T23:50:30.186 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-24T23:50:30.292 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-24T23:50:30.545 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-websocket. 2023-10-24T23:50:30.572 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2023-10-24T23:50:30.610 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package rbd-fuse. 2023-10-24T23:50:30.612 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-24T23:50:30.638 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../114-rbd-fuse_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:30.677 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking rbd-fuse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:30.932 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-24T23:50:30.959 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-24T23:50:30.981 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-24T23:50:30.998 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-24T23:50:31.240 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-24T23:50:31.349 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-24T23:50:31.597 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-24T23:50:31.754 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-24T23:50:31.770 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:31.812 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-mgr-k8sevents (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:31.973 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-24T23:50:32.140 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-24T23:50:32.167 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-24T23:50:32.215 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-24T23:50:32.439 INFO:teuthology.orchestra.run.smithi107.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-24T23:50:32.542 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up cephadm (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:32.551 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-24T23:50:32.579 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-24T23:50:32.618 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-24T23:50:32.845 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package jq. 2023-10-24T23:50:32.861 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-24T23:50:32.903 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-24T23:50:33.085 INFO:teuthology.orchestra.run.smithi107.stdout:Adding system user cephadm....done 2023-10-24T23:50:33.156 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package socat. 2023-10-24T23:50:33.184 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-24T23:50:33.223 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-24T23:50:33.239 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-24T23:50:33.507 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-24T23:50:33.535 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package xmlstarlet. 2023-10-24T23:50:33.562 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-24T23:50:33.601 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-24T23:50:33.742 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-24T23:50:33.868 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-24T23:50:33.955 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-test. 2023-10-24T23:50:33.971 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../098-ceph-test_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:34.013 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-test (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:34.127 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-24T23:50:34.370 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-24T23:50:34.471 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-24T23:50:34.772 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-24T23:50:35.124 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-24T23:50:35.375 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-24T23:50:35.659 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-24T23:50:35.935 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-24T23:50:36.186 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-24T23:50:36.235 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ceph-volume. 2023-10-24T23:50:36.262 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../099-ceph-volume_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:36.301 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ceph-volume (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-24T23:50:36.697 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-ceph-argparse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:36.697 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-24T23:50:36.725 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../100-cephfs-mirror_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:36.763 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephfs-mirror (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:36.964 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-24T23:50:37.042 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-colorama. 2023-10-24T23:50:37.069 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../101-python3-colorama_0.4.4-1_all.deb ... 2023-10-24T23:50:37.090 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-24T23:50:37.108 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-24T23:50:37.349 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-24T23:50:37.370 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephfs-shell. 2023-10-24T23:50:37.385 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../102-cephfs-shell_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:37.427 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephfs-shell (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:37.601 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-24T23:50:37.688 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package cephfs-top. 2023-10-24T23:50:37.716 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../103-cephfs-top_18.2.0-1095-ge6528508-1jammy_all.deb ... 2023-10-24T23:50:37.726 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-24T23:50:37.755 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking cephfs-top (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:38.008 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-24T23:50:38.024 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../104-libcephfs-dev_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:38.057 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libcephfs-dev (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:38.077 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-24T23:50:38.353 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package nvme-cli. 2023-10-24T23:50:38.380 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../105-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-24T23:50:38.411 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-24T23:50:38.463 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-24T23:50:38.722 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-24T23:50:38.782 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-24T23:50:38.798 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../106-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-24T23:50:38.840 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-24T23:50:39.032 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-24T23:50:39.260 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pastescript. 2023-10-24T23:50:39.287 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2023-10-24T23:50:39.327 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-24T23:50:39.647 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-psutil. 2023-10-24T23:50:39.674 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-24T23:50:39.692 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-24T23:50:39.713 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-24T23:50:39.951 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-24T23:50:40.060 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-24T23:50:40.066 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pygments. 2023-10-24T23:50:40.094 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../109-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-24T23:50:40.141 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-24T23:50:40.161 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-24T23:50:40.487 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-24T23:50:40.612 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-24T23:50:40.640 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../110-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-24T23:50:40.678 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-24T23:50:40.981 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package python3-simplejson. 2023-10-24T23:50:41.009 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-24T23:50:41.055 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-24T23:50:41.298 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-24T23:50:41.325 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-24T23:50:41.341 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-24T23:50:41.384 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-24T23:50:41.683 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-24T23:50:41.975 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-24T23:50:42.104 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package radosgw. 2023-10-24T23:50:42.132 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../113-radosgw_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:42.170 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking radosgw (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:42.293 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-24T23:50:42.628 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-24T23:50:42.879 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-24T23:50:42.890 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package rbd-fuse. 2023-10-24T23:50:42.906 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../114-rbd-fuse_18.2.0-1095-ge6528508-1jammy_amd64.deb ... 2023-10-24T23:50:42.949 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking rbd-fuse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:43.005 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-ceph-common (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:43.236 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-24T23:50:43.289 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-idna (3.3-1) ... 2023-10-24T23:50:43.478 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-24T23:50:43.580 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-24T23:50:43.590 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-24T23:50:43.830 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-24T23:50:44.168 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-24T23:50:44.656 INFO:teuthology.orchestra.run.smithi110.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-24T23:50:44.721 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:50:44.723 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephadm (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:44.855 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-24T23:50:45.114 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-24T23:50:45.233 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user cephadm....done 2023-10-24T23:50:45.240 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-24T23:50:45.366 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-24T23:50:45.388 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-24T23:50:45.672 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-24T23:50:45.684 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-24T23:50:45.924 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-24T23:50:45.951 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-24T23:50:46.033 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-24T23:50:46.186 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:50:46.276 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-24T23:50:46.320 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-24T23:50:46.446 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-24T23:50:46.535 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-24T23:50:46.645 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-24T23:50:46.747 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-24T23:50:46.930 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-24T23:50:46.998 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-24T23:50:47.257 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-24T23:50:47.281 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-24T23:50:47.508 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-24T23:50:47.566 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-24T23:50:47.751 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-24T23:50:47.801 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-24T23:50:48.002 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-24T23:50:48.060 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-24T23:50:48.111 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-24T23:50:48.303 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-24T23:50:48.404 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-24T23:50:48.530 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-24T23:50:48.571 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-24T23:50:48.814 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-argparse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:48.814 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-24T23:50:49.116 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-24T23:50:49.123 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-24T23:50:49.241 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-24T23:50:49.358 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-24T23:50:49.500 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-24T23:50:49.743 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-24T23:50:49.847 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-24T23:50:49.869 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-24T23:50:50.094 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-24T23:50:50.229 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-24T23:50:50.245 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-24T23:50:50.354 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-24T23:50:50.622 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-24T23:50:50.638 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-24T23:50:50.923 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-24T23:50:50.972 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-24T23:50:51.215 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:50:51.292 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-24T23:50:51.333 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-24T23:50:51.642 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-24T23:50:51.935 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-24T23:50:51.977 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-24T23:50:52.202 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-24T23:50:52.228 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-24T23:50:52.337 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-24T23:50:52.354 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-24T23:50:52.480 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-24T23:50:52.629 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-24T23:50:52.773 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-24T23:50:53.149 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-24T23:50:53.156 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-24T23:50:53.392 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-24T23:50:53.541 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-24T23:50:53.760 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-24T23:50:53.809 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-24T23:50:54.062 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-24T23:50:54.136 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-24T23:50:54.270 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librados2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:54.371 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-24T23:50:54.396 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librgw2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:54.599 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libsqlite3-mod-ceph (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:54.706 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-24T23:50:54.716 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libcephfs2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:54.826 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libradosstriper1 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:54.952 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-24T23:50:54.965 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-24T23:50:55.108 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-ceph-common (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:55.237 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up librbd1 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:55.363 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-modules-core (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:55.401 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-idna (3.3-1) ... 2023-10-24T23:50:55.489 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-fuse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:55.700 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-24T23:50:55.700 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-24T23:50:55.726 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-24T23:50:56.442 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libcephfs-dev (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:56.633 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rados (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:56.827 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up cephfs-top (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:56.932 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:50:56.953 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-24T23:50:57.067 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-24T23:50:57.326 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-24T23:50:57.453 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-24T23:50:57.578 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-24T23:50:57.896 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-24T23:50:58.156 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-24T23:50:58.202 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rbd (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:58.407 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:50:58.454 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up rbd-fuse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:58.516 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-24T23:50:58.580 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-rgw (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:58.626 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-24T23:50:58.706 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up python3-cephfs (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:58.832 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up cephfs-shell (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:58.927 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-24T23:50:58.958 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-common (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:50:59.170 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-24T23:50:59.429 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-24T23:50:59.484 INFO:teuthology.orchestra.run.smithi107.stdout:Adding group ceph....done 2023-10-24T23:50:59.664 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-24T23:50:59.852 INFO:teuthology.orchestra.run.smithi107.stdout:Adding system user ceph....done 2023-10-24T23:50:59.914 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-24T23:51:00.021 INFO:teuthology.orchestra.run.smithi107.stdout:Setting system user ceph properties....done 2023-10-24T23:51:00.033 INFO:teuthology.orchestra.run.smithi107.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-24T23:51:00.175 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-24T23:51:00.191 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-24T23:51:00.292 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-24T23:51:00.510 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-24T23:51:00.569 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-24T23:51:00.686 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-24T23:51:00.970 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-24T23:51:01.075 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up cephfs-mirror (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:01.280 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-24T23:51:01.291 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-24T23:51:01.292 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-24T23:51:01.531 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-24T23:51:01.809 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-test (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:01.915 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-24T23:51:01.935 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up radosgw (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:02.142 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-24T23:51:02.243 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-24T23:51:02.437 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-24T23:51:02.437 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-24T23:51:02.620 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-24T23:51:02.854 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-24T23:51:02.937 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-base (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:03.122 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-24T23:51:03.248 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-24T23:51:03.281 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-24T23:51:03.566 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-24T23:51:03.830 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mds (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:03.850 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-24T23:51:04.082 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-24T23:51:04.083 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-24T23:51:04.118 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-24T23:51:04.236 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-24T23:51:04.487 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-24T23:51:04.607 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:04.738 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-24T23:51:04.807 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-24T23:51:04.807 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-24T23:51:04.989 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-24T23:51:05.382 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-24T23:51:05.391 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-osd (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:05.650 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-24T23:51:05.696 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-24T23:51:05.697 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-24T23:51:05.986 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-24T23:51:06.137 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librados2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:06.209 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-k8sevents (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:06.263 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librgw2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:06.311 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-diskprediction-local (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:06.445 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mon (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:06.468 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libsqlite3-mod-ceph (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:06.594 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs2 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:06.658 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-24T23:51:06.659 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-24T23:51:06.728 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libradosstriper1 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:06.854 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-24T23:51:07.147 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up librbd1 (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:07.233 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-cephadm (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:07.273 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-modules-core (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:07.384 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:07.400 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-fuse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:07.544 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-mgr-dashboard (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:07.614 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-24T23:51:07.614 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-24T23:51:07.695 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ceph-volume (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:08.064 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-24T23:51:08.109 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libcephfs-dev (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:08.210 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rados (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:08.336 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephfs-top (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:08.462 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-24T23:51:08.706 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-24T23:51:09.161 INFO:teuthology.orchestra.run.smithi107.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-24T23:51:09.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-24T23:51:09.218 INFO:teuthology.orchestra.run.smithi107.stdout:Reading package lists... 2023-10-24T23:51:09.405 INFO:teuthology.orchestra.run.smithi107.stdout:Building dependency tree... 2023-10-24T23:51:09.405 INFO:teuthology.orchestra.run.smithi107.stdout:Reading state information... 2023-10-24T23:51:09.573 INFO:teuthology.orchestra.run.smithi107.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-24T23:51:09.573 INFO:teuthology.orchestra.run.smithi107.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-24T23:51:09.573 INFO:teuthology.orchestra.run.smithi107.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-24T23:51:09.573 INFO:teuthology.orchestra.run.smithi107.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-24T23:51:09.574 INFO:teuthology.orchestra.run.smithi107.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-24T23:51:09.574 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages were automatically installed and are no longer required: 2023-10-24T23:51:09.574 INFO:teuthology.orchestra.run.smithi107.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-24T23:51:09.574 INFO:teuthology.orchestra.run.smithi107.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-24T23:51:09.575 INFO:teuthology.orchestra.run.smithi107.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-24T23:51:09.575 INFO:teuthology.orchestra.run.smithi107.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-24T23:51:09.575 INFO:teuthology.orchestra.run.smithi107.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-24T23:51:09.575 INFO:teuthology.orchestra.run.smithi107.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-24T23:51:09.576 INFO:teuthology.orchestra.run.smithi107.stdout:The following additional packages will be installed: 2023-10-24T23:51:09.577 INFO:teuthology.orchestra.run.smithi107.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-24T23:51:09.577 INFO:teuthology.orchestra.run.smithi107.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-24T23:51:09.577 INFO:teuthology.orchestra.run.smithi107.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-24T23:51:09.577 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-24T23:51:09.578 INFO:teuthology.orchestra.run.smithi107.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-24T23:51:09.578 INFO:teuthology.orchestra.run.smithi107.stdout:Suggested packages: 2023-10-24T23:51:09.578 INFO:teuthology.orchestra.run.smithi107.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-24T23:51:09.578 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-24T23:51:09.579 INFO:teuthology.orchestra.run.smithi107.stdout: libengine-pkcs11-openssl 2023-10-24T23:51:09.613 INFO:teuthology.orchestra.run.smithi107.stdout:The following NEW packages will be installed: 2023-10-24T23:51:09.614 INFO:teuthology.orchestra.run.smithi107.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-24T23:51:09.615 INFO:teuthology.orchestra.run.smithi107.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-24T23:51:09.615 INFO:teuthology.orchestra.run.smithi107.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-24T23:51:09.615 INFO:teuthology.orchestra.run.smithi107.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-24T23:51:09.615 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-24T23:51:09.615 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-24T23:51:09.616 INFO:teuthology.orchestra.run.smithi107.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-24T23:51:09.616 INFO:teuthology.orchestra.run.smithi107.stdout:The following packages will be upgraded: 2023-10-24T23:51:09.617 INFO:teuthology.orchestra.run.smithi107.stdout: libssl3 2023-10-24T23:51:09.691 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rbd (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:09.797 INFO:teuthology.orchestra.run.smithi107.stdout:1 upgraded, 39 newly installed, 0 to remove and 256 not upgraded. 2023-10-24T23:51:09.797 INFO:teuthology.orchestra.run.smithi107.stdout:Need to get 54.6 MB of archives. 2023-10-24T23:51:09.798 INFO:teuthology.orchestra.run.smithi107.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-24T23:51:09.798 INFO:teuthology.orchestra.run.smithi107.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-24T23:51:09.817 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up rbd-fuse (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:09.935 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-rgw (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:10.053 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up python3-cephfs (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:10.179 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephfs-shell (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:10.276 INFO:teuthology.orchestra.run.smithi107.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-24T23:51:10.305 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-common (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:10.532 INFO:teuthology.orchestra.run.smithi107.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-24T23:51:10.567 INFO:teuthology.orchestra.run.smithi107.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-24T23:51:10.568 INFO:teuthology.orchestra.run.smithi107.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-24T23:51:10.574 INFO:teuthology.orchestra.run.smithi107.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-24T23:51:10.577 INFO:teuthology.orchestra.run.smithi107.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-24T23:51:10.578 INFO:teuthology.orchestra.run.smithi107.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-24T23:51:10.581 INFO:teuthology.orchestra.run.smithi107.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-24T23:51:10.583 INFO:teuthology.orchestra.run.smithi107.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-24T23:51:10.749 INFO:teuthology.orchestra.run.smithi107.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-24T23:51:10.749 INFO:teuthology.orchestra.run.smithi107.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-24T23:51:10.822 INFO:teuthology.orchestra.run.smithi110.stdout:Adding group ceph....done 2023-10-24T23:51:10.994 INFO:teuthology.orchestra.run.smithi107.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-24T23:51:11.208 INFO:teuthology.orchestra.run.smithi110.stdout:Adding system user ceph....done 2023-10-24T23:51:11.384 INFO:teuthology.orchestra.run.smithi110.stdout:Setting system user ceph properties....done 2023-10-24T23:51:11.397 INFO:teuthology.orchestra.run.smithi110.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-24T23:51:11.536 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-24T23:51:11.913 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-24T23:51:12.164 INFO:teuthology.orchestra.run.smithi107.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-24T23:51:12.169 INFO:teuthology.orchestra.run.smithi107.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-24T23:51:12.169 INFO:teuthology.orchestra.run.smithi107.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-24T23:51:12.176 INFO:teuthology.orchestra.run.smithi107.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-24T23:51:12.176 INFO:teuthology.orchestra.run.smithi107.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-24T23:51:12.187 INFO:teuthology.orchestra.run.smithi107.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-24T23:51:12.191 INFO:teuthology.orchestra.run.smithi107.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-24T23:51:12.208 INFO:teuthology.orchestra.run.smithi107.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-24T23:51:12.372 INFO:teuthology.orchestra.run.smithi107.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-24T23:51:12.422 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up cephfs-mirror (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:12.616 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-24T23:51:12.616 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-24T23:51:13.094 INFO:teuthology.orchestra.run.smithi107.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-24T23:51:13.095 INFO:teuthology.orchestra.run.smithi107.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-24T23:51:13.121 INFO:teuthology.orchestra.run.smithi107.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-24T23:51:13.123 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-test (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:13.199 INFO:teuthology.orchestra.run.smithi107.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-24T23:51:13.207 INFO:teuthology.orchestra.run.smithi107.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-24T23:51:13.208 INFO:teuthology.orchestra.run.smithi107.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-24T23:51:13.210 INFO:teuthology.orchestra.run.smithi107.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-24T23:51:13.211 INFO:teuthology.orchestra.run.smithi107.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-24T23:51:13.237 INFO:teuthology.orchestra.run.smithi107.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-24T23:51:13.241 INFO:teuthology.orchestra.run.smithi107.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-24T23:51:13.266 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up radosgw (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:13.737 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-24T23:51:13.737 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-24T23:51:13.932 INFO:teuthology.orchestra.run.smithi107.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-24T23:51:14.017 INFO:teuthology.orchestra.run.smithi107.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-24T23:51:14.018 INFO:teuthology.orchestra.run.smithi107.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-24T23:51:14.019 INFO:teuthology.orchestra.run.smithi107.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-24T23:51:14.108 INFO:teuthology.orchestra.run.smithi107.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-24T23:51:14.113 INFO:teuthology.orchestra.run.smithi107.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-24T23:51:14.218 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-base (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:14.535 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-24T23:51:14.570 INFO:teuthology.orchestra.run.smithi107.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-24T23:51:14.572 INFO:teuthology.orchestra.run.smithi107.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-24T23:51:14.969 INFO:teuthology.orchestra.run.smithi107.stdout:Preconfiguring packages ... 2023-10-24T23:51:15.045 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mds (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:15.122 INFO:teuthology.orchestra.run.smithi107.stdout:Fetched 54.6 MB in 5s (11.0 MB/s) 2023-10-24T23:51:15.261 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package flex. 2023-10-24T23:51:15.278 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-24T23:51:15.278 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-24T23:51:15.354 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172901 files and directories currently installed.) 2023-10-24T23:51:15.359 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-24T23:51:15.401 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-24T23:51:15.763 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:15.870 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-24T23:51:15.983 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-24T23:51:15.984 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-24T23:51:16.007 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-24T23:51:16.407 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-24T23:51:16.523 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-osd (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:16.854 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-24T23:51:16.855 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-24T23:51:16.916 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package bison. 2023-10-24T23:51:17.019 INFO:teuthology.orchestra.run.smithi107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172974 files and directories currently installed.) 2023-10-24T23:51:17.025 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-24T23:51:17.068 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-24T23:51:17.374 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-k8sevents (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:17.444 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-24T23:51:17.471 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-24T23:51:17.484 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-diskprediction-local (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:17.510 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-24T23:51:17.610 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mon (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:17.808 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-24T23:51:17.809 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-24T23:51:17.838 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package dump. 2023-10-24T23:51:17.865 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-24T23:51:17.905 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking dump (0.4b47-3) ... 2023-10-24T23:51:18.194 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package indent. 2023-10-24T23:51:18.221 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:18.258 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-24T23:51:18.353 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-cephadm (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:18.477 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-24T23:51:18.493 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-24T23:51:18.495 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:18.527 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-24T23:51:18.605 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-mgr-dashboard (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:18.731 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ceph-volume (18.2.0-1095-ge6528508-1jammy) ... 2023-10-24T23:51:18.855 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-24T23:51:18.883 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-24T23:51:18.922 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-24T23:51:19.075 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-24T23:51:19.182 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-24T23:51:19.198 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-24T23:51:19.240 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-24T23:51:19.571 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-24T23:51:19.598 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-24T23:51:19.637 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-24T23:51:19.906 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-24T23:51:19.922 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-24T23:51:19.964 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-24T23:51:20.184 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-24T23:51:20.208 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libjson-perl. 2023-10-24T23:51:20.225 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-24T23:51:20.267 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-24T23:51:20.554 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-24T23:51:20.581 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-24T23:51:20.612 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-24T23:51:20.637 INFO:teuthology.orchestra.run.smithi110.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-24T23:51:20.640 DEBUG:teuthology.orchestra.run.smithi110:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-24T23:51:20.741 INFO:teuthology.orchestra.run.smithi110.stdout:Reading package lists... 2023-10-24T23:51:20.927 INFO:teuthology.orchestra.run.smithi110.stdout:Building dependency tree... 2023-10-24T23:51:20.928 INFO:teuthology.orchestra.run.smithi110.stdout:Reading state information... 2023-10-24T23:51:21.084 INFO:teuthology.orchestra.run.smithi110.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-24T23:51:21.084 INFO:teuthology.orchestra.run.smithi110.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-24T23:51:21.084 INFO:teuthology.orchestra.run.smithi110.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-24T23:51:21.084 INFO:teuthology.orchestra.run.smithi110.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-24T23:51:21.085 INFO:teuthology.orchestra.run.smithi110.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-24T23:51:21.085 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages were automatically installed and are no longer required: 2023-10-24T23:51:21.085 INFO:teuthology.orchestra.run.smithi110.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-24T23:51:21.085 INFO:teuthology.orchestra.run.smithi110.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-24T23:51:21.085 INFO:teuthology.orchestra.run.smithi110.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-24T23:51:21.086 INFO:teuthology.orchestra.run.smithi110.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-24T23:51:21.086 INFO:teuthology.orchestra.run.smithi110.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-24T23:51:21.086 INFO:teuthology.orchestra.run.smithi110.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-24T23:51:21.087 INFO:teuthology.orchestra.run.smithi110.stdout:The following additional packages will be installed: 2023-10-24T23:51:21.087 INFO:teuthology.orchestra.run.smithi110.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-24T23:51:21.087 INFO:teuthology.orchestra.run.smithi110.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-24T23:51:21.088 INFO:teuthology.orchestra.run.smithi110.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-24T23:51:21.088 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-24T23:51:21.088 INFO:teuthology.orchestra.run.smithi110.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-24T23:51:21.088 INFO:teuthology.orchestra.run.smithi110.stdout:Suggested packages: 2023-10-24T23:51:21.088 INFO:teuthology.orchestra.run.smithi110.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-24T23:51:21.089 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-24T23:51:21.089 INFO:teuthology.orchestra.run.smithi110.stdout: libengine-pkcs11-openssl 2023-10-24T23:51:21.122 INFO:teuthology.orchestra.run.smithi110.stdout:The following NEW packages will be installed: 2023-10-24T23:51:21.123 INFO:teuthology.orchestra.run.smithi110.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-24T23:51:21.123 INFO:teuthology.orchestra.run.smithi110.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-24T23:51:21.124 INFO:teuthology.orchestra.run.smithi110.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-24T23:51:21.124 INFO:teuthology.orchestra.run.smithi110.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-24T23:51:21.124 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-24T23:51:21.124 INFO:teuthology.orchestra.run.smithi110.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-24T23:51:21.124 INFO:teuthology.orchestra.run.smithi110.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-24T23:51:21.125 INFO:teuthology.orchestra.run.smithi110.stdout:The following packages will be upgraded: 2023-10-24T23:51:21.125 INFO:teuthology.orchestra.run.smithi110.stdout: libssl3 2023-10-24T23:51:21.183 INFO:teuthology.orchestra.run.smithi110.stdout:1 upgraded, 39 newly installed, 0 to remove and 256 not upgraded. 2023-10-24T23:51:21.184 INFO:teuthology.orchestra.run.smithi110.stdout:Need to get 54.6 MB of archives. 2023-10-24T23:51:21.184 INFO:teuthology.orchestra.run.smithi110.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-24T23:51:21.184 INFO:teuthology.orchestra.run.smithi110.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-24T23:51:21.283 INFO:teuthology.orchestra.run.smithi110.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-24T23:51:21.345 INFO:teuthology.orchestra.run.smithi110.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-24T23:51:21.356 INFO:teuthology.orchestra.run.smithi110.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-24T23:51:21.357 INFO:teuthology.orchestra.run.smithi110.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-24T23:51:21.359 INFO:teuthology.orchestra.run.smithi110.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-24T23:51:21.360 INFO:teuthology.orchestra.run.smithi110.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-24T23:51:21.361 INFO:teuthology.orchestra.run.smithi110.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-24T23:51:21.362 INFO:teuthology.orchestra.run.smithi110.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-24T23:51:21.363 INFO:teuthology.orchestra.run.smithi110.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-24T23:51:21.363 INFO:teuthology.orchestra.run.smithi110.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-24T23:51:21.367 INFO:teuthology.orchestra.run.smithi110.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-24T23:51:21.380 INFO:teuthology.orchestra.run.smithi110.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-24T23:51:21.680 INFO:teuthology.orchestra.run.smithi110.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-24T23:51:21.727 INFO:teuthology.orchestra.run.smithi110.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-24T23:51:21.728 INFO:teuthology.orchestra.run.smithi110.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-24T23:51:21.730 INFO:teuthology.orchestra.run.smithi110.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-24T23:51:21.730 INFO:teuthology.orchestra.run.smithi110.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-24T23:51:21.735 INFO:teuthology.orchestra.run.smithi110.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-24T23:51:21.736 INFO:teuthology.orchestra.run.smithi110.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-24T23:51:21.744 INFO:teuthology.orchestra.run.smithi110.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-24T23:51:21.744 INFO:teuthology.orchestra.run.smithi110.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-24T23:51:21.766 INFO:teuthology.orchestra.run.smithi110.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-24T23:51:21.767 INFO:teuthology.orchestra.run.smithi110.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-24T23:51:21.777 INFO:teuthology.orchestra.run.smithi110.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-24T23:51:21.797 INFO:teuthology.orchestra.run.smithi110.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-24T23:51:21.800 INFO:teuthology.orchestra.run.smithi110.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-24T23:51:21.800 INFO:teuthology.orchestra.run.smithi110.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-24T23:51:21.801 INFO:teuthology.orchestra.run.smithi110.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-24T23:51:21.802 INFO:teuthology.orchestra.run.smithi110.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-24T23:51:21.812 INFO:teuthology.orchestra.run.smithi110.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-24T23:51:21.813 INFO:teuthology.orchestra.run.smithi110.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-24T23:51:22.018 INFO:teuthology.orchestra.run.smithi110.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-24T23:51:22.019 INFO:teuthology.orchestra.run.smithi110.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-24T23:51:22.019 INFO:teuthology.orchestra.run.smithi110.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-24T23:51:22.019 INFO:teuthology.orchestra.run.smithi110.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-24T23:51:22.019 INFO:teuthology.orchestra.run.smithi110.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-24T23:51:22.019 INFO:teuthology.orchestra.run.smithi110.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-24T23:51:22.030 INFO:teuthology.orchestra.run.smithi110.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-24T23:51:22.031 INFO:teuthology.orchestra.run.smithi110.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-24T23:51:22.339 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-24T23:51:22.367 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-24T23:51:22.405 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-24T23:51:22.428 INFO:teuthology.orchestra.run.smithi110.stdout:Preconfiguring packages ... 2023-10-24T23:51:22.577 INFO:teuthology.orchestra.run.smithi110.stdout:Fetched 54.6 MB in 1s (60.7 MB/s) 2023-10-24T23:51:22.667 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-24T23:51:22.682 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package flex. 2023-10-24T23:51:22.694 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-24T23:51:22.733 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-24T23:51:22.744 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172901 files and directories currently installed.) 2023-10-24T23:51:22.749 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-24T23:51:22.789 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-24T23:51:23.052 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-24T23:51:23.080 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-24T23:51:23.119 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-24T23:51:23.388 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-24T23:51:23.416 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-24T23:51:23.438 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-24T23:51:23.446 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-24T23:51:23.604 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-24T23:51:23.766 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-24T23:51:23.793 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-24T23:51:23.832 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-24T23:51:24.004 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-24T23:51:24.177 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-24T23:51:24.204 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:24.243 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:24.496 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package bison. 2023-10-24T23:51:24.563 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-24T23:51:24.590 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:24.598 INFO:teuthology.orchestra.run.smithi110.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 172974 files and directories currently installed.) 2023-10-24T23:51:24.604 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-24T23:51:24.621 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-24T23:51:24.649 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-24T23:51:24.898 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-24T23:51:24.914 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:24.957 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-24T23:51:25.057 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-24T23:51:25.085 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-24T23:51:25.124 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-24T23:51:25.201 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-24T23:51:25.217 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-24T23:51:25.259 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-24T23:51:25.418 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package dump. 2023-10-24T23:51:25.446 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-24T23:51:25.485 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking dump (0.4b47-3) ... 2023-10-24T23:51:25.763 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package indent. 2023-10-24T23:51:25.788 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-24T23:51:25.791 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:25.815 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-24T23:51:25.830 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-24T23:51:25.854 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-24T23:51:26.057 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-24T23:51:26.073 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-24T23:51:26.115 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-24T23:51:26.149 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package modemmanager. 2023-10-24T23:51:26.176 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-24T23:51:26.207 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-24T23:51:26.546 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-24T23:51:26.573 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-24T23:51:26.612 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-24T23:51:26.865 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-24T23:51:26.881 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-24T23:51:26.923 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-24T23:51:26.980 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package network-manager. 2023-10-24T23:51:27.007 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-24T23:51:27.054 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-24T23:51:27.662 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package ppp. 2023-10-24T23:51:27.684 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-24T23:51:27.689 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-24T23:51:27.712 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-24T23:51:27.751 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-24T23:51:27.972 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-24T23:51:28.003 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-24T23:51:28.019 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-24T23:51:28.062 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-24T23:51:28.283 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package pptp-linux. 2023-10-24T23:51:28.311 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-24T23:51:28.314 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libjson-perl. 2023-10-24T23:51:28.341 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-24T23:51:28.349 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-24T23:51:28.381 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-24T23:51:28.602 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-24T23:51:28.630 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-24T23:51:28.668 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-24T23:51:28.692 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-24T23:51:28.719 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-24T23:51:28.758 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-24T23:51:28.955 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-24T23:51:28.971 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-24T23:51:29.013 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-client-common (238) ... 2023-10-24T23:51:29.358 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-24T23:51:29.386 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:29.425 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:29.955 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-common. 2023-10-24T23:51:29.983 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-24T23:51:30.039 INFO:teuthology.orchestra.run.smithi107.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-24T23:51:30.070 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-common (238) ... 2023-10-24T23:51:30.403 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-24T23:51:30.430 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-24T23:51:30.469 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-24T23:51:30.537 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-14. 2023-10-24T23:51:30.565 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:30.607 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:30.755 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-24T23:51:30.771 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-24T23:51:30.813 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-24T23:51:31.166 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-24T23:51:31.194 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-24T23:51:31.232 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-24T23:51:31.552 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-24T23:51:31.580 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-24T23:51:31.610 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-24T23:51:31.831 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql. 2023-10-24T23:51:31.848 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-24T23:51:31.881 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql (14+238) ... 2023-10-24T23:51:31.913 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-24T23:51:31.940 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-24T23:51:31.979 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-24T23:51:32.108 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-client. 2023-10-24T23:51:32.126 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-24T23:51:32.166 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-client (14+238) ... 2023-10-24T23:51:32.332 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-24T23:51:32.360 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:32.398 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:32.410 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-24T23:51:32.427 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-24T23:51:32.469 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-24T23:51:32.763 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package tcl8.6. 2023-10-24T23:51:32.780 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-24T23:51:32.813 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-24T23:51:32.827 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-24T23:51:32.856 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:32.886 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-24T23:51:33.090 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package tcl. 2023-10-24T23:51:33.119 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-24T23:51:33.162 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-24T23:51:33.171 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-24T23:51:33.199 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:33.238 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-24T23:51:33.482 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-24T23:51:33.509 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-24T23:51:33.510 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package wpasupplicant. 2023-10-24T23:51:33.538 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-24T23:51:33.548 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-24T23:51:33.576 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-24T23:51:33.931 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-24T23:51:33.959 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-24T23:51:33.997 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-24T23:51:34.068 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-24T23:51:34.096 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-24T23:51:34.143 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-24T23:51:34.375 INFO:teuthology.orchestra.run.smithi107.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-24T23:51:34.403 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-24T23:51:34.450 INFO:teuthology.orchestra.run.smithi107.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-24T23:51:34.496 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package modemmanager. 2023-10-24T23:51:34.524 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-24T23:51:34.563 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-24T23:51:34.753 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-client-common (238) ... 2023-10-24T23:51:34.948 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-24T23:51:35.065 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-24T23:51:35.152 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package network-manager. 2023-10-24T23:51:35.168 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-24T23:51:35.191 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:35.209 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-24T23:51:35.317 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-24T23:51:35.443 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-24T23:51:35.569 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:35.817 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package ppp. 2023-10-24T23:51:35.845 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-24T23:51:36.033 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-24T23:51:36.108 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-24T23:51:36.142 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-24T23:51:36.455 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package pptp-linux. 2023-10-24T23:51:36.483 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-24T23:51:36.492 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-24T23:51:36.493 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-24T23:51:36.521 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-24T23:51:36.774 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-24T23:51:36.802 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-24T23:51:36.840 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-24T23:51:37.028 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-24T23:51:37.093 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-24T23:51:37.109 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-24T23:51:37.143 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking postgresql-client-common (238) ... 2023-10-24T23:51:37.154 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-24T23:51:37.280 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-24T23:51:37.406 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-24T23:51:37.447 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-24T23:51:37.474 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:37.513 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:37.532 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-24T23:51:37.652 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-24T23:51:38.077 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package postgresql-common. 2023-10-24T23:51:38.105 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-24T23:51:38.153 INFO:teuthology.orchestra.run.smithi110.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-24T23:51:38.184 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-24T23:51:38.185 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking postgresql-common (238) ... 2023-10-24T23:51:38.635 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package postgresql-14. 2023-10-24T23:51:38.662 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-24T23:51:38.704 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:38.856 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-24T23:51:38.982 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up dump (0.4b47-3) ... 2023-10-24T23:51:39.075 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-24T23:51:39.141 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-24T23:51:39.267 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-24T23:51:39.368 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-24T23:51:39.494 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-24T23:51:39.629 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-24T23:51:39.718 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-24T23:51:39.789 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-24T23:51:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-24T23:51:40.071 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package postgresql. 2023-10-24T23:51:40.088 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-24T23:51:40.091 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-24T23:51:40.121 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking postgresql (14+238) ... 2023-10-24T23:51:40.233 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-24T23:51:40.359 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-24T23:51:40.365 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package postgresql-client. 2023-10-24T23:51:40.393 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-24T23:51:40.431 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking postgresql-client (14+238) ... 2023-10-24T23:51:40.502 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-client (14+238) ... 2023-10-24T23:51:40.628 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-24T23:51:40.675 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-24T23:51:40.703 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-24T23:51:40.742 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-24T23:51:41.061 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package tcl8.6. 2023-10-24T23:51:41.089 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-24T23:51:41.127 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-24T23:51:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:51:41.157 INFO:teuthology.orchestra.run.smithi107.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-24T23:51:41.157 INFO:teuthology.orchestra.run.smithi107.stdout:which means they are currently configured by ifupdown: 2023-10-24T23:51:41.157 INFO:teuthology.orchestra.run.smithi107.stdout:- enp3s0f1 2023-10-24T23:51:41.157 INFO:teuthology.orchestra.run.smithi107.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-24T23:51:41.158 INFO:teuthology.orchestra.run.smithi107.stdout:remove their configuration from /etc/network/interfaces. 2023-10-24T23:51:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:51:41.283 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-24T23:51:41.371 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package tcl. 2023-10-24T23:51:41.388 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-24T23:51:41.426 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-24T23:51:41.656 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-24T23:51:41.766 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package wpasupplicant. 2023-10-24T23:51:41.794 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-24T23:51:41.832 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-24T23:51:42.019 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-24T23:51:42.221 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-24T23:51:42.237 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-24T23:51:42.279 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-24T23:51:42.614 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-24T23:51:42.672 INFO:teuthology.orchestra.run.smithi110.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-24T23:51:42.701 INFO:teuthology.orchestra.run.smithi110.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-24T23:51:42.732 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-24T23:51:42.748 INFO:teuthology.orchestra.run.smithi110.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-24T23:51:42.858 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-24T23:51:42.984 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-common (238) ... 2023-10-24T23:51:43.129 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up postgresql-client-common (238) ... 2023-10-24T23:51:43.339 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-24T23:51:43.474 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-24T23:51:43.600 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:43.725 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-24T23:51:43.851 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-24T23:51:43.960 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:44.387 INFO:teuthology.orchestra.run.smithi107.stdout:Adding user postgres to group ssl-cert 2023-10-24T23:51:44.421 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-24T23:51:44.500 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-24T23:51:44.622 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:51:44.623 INFO:teuthology.orchestra.run.smithi107.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-24T23:51:44.830 INFO:teuthology.orchestra.run.smithi107.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-24T23:51:44.830 INFO:teuthology.orchestra.run.smithi107.stdout:Removing obsolete dictionary files: 2023-10-24T23:51:44.879 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-24T23:51:44.879 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-24T23:51:45.420 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-24T23:51:45.421 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-24T23:51:45.546 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-24T23:51:45.697 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-24T23:51:45.810 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:45.823 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-24T23:51:45.949 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-24T23:51:46.075 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-24T23:51:46.137 INFO:teuthology.orchestra.run.smithi107.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-24T23:51:46.148 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-24T23:51:46.193 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-24T23:51:46.194 INFO:teuthology.orchestra.run.smithi107.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-24T23:51:46.194 INFO:teuthology.orchestra.run.smithi107.stdout:This user must also own the server process. 2023-10-24T23:51:46.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:51:46.194 INFO:teuthology.orchestra.run.smithi107.stdout:The database cluster will be initialized with locale "en_US". 2023-10-24T23:51:46.195 INFO:teuthology.orchestra.run.smithi107.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-24T23:51:46.195 INFO:teuthology.orchestra.run.smithi107.stdout:The default text search configuration will be set to "english". 2023-10-24T23:51:46.195 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:51:46.195 INFO:teuthology.orchestra.run.smithi107.stdout:Data page checksums are disabled. 2023-10-24T23:51:46.196 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:51:46.196 INFO:teuthology.orchestra.run.smithi107.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-24T23:51:46.196 INFO:teuthology.orchestra.run.smithi107.stdout:creating subdirectories ... ok 2023-10-24T23:51:46.197 INFO:teuthology.orchestra.run.smithi107.stdout:selecting dynamic shared memory implementation ... posix 2023-10-24T23:51:46.210 INFO:teuthology.orchestra.run.smithi107.stdout:selecting default max_connections ... 100 2023-10-24T23:51:46.229 INFO:teuthology.orchestra.run.smithi107.stdout:selecting default shared_buffers ... 128MB 2023-10-24T23:51:46.239 INFO:teuthology.orchestra.run.smithi107.stdout:selecting default time zone ... Etc/UTC 2023-10-24T23:51:46.240 INFO:teuthology.orchestra.run.smithi107.stdout:creating configuration files ... ok 2023-10-24T23:51:46.352 INFO:teuthology.orchestra.run.smithi107.stdout:running bootstrap script ... ok 2023-10-24T23:51:46.962 INFO:teuthology.orchestra.run.smithi107.stdout:performing post-bootstrap initialization ... ok 2023-10-24T23:51:47.255 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-24T23:51:47.488 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up dump (0.4b47-3) ... 2023-10-24T23:51:47.589 INFO:teuthology.orchestra.run.smithi107.stdout:syncing data to disk ... ok 2023-10-24T23:51:47.641 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-24T23:51:47.766 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-24T23:51:47.893 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-24T23:51:47.988 INFO:teuthology.orchestra.run.smithi107.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-24T23:51:48.010 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-24T23:51:48.128 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-24T23:51:48.254 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-24T23:51:48.345 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-24T23:51:48.414 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-24T23:51:48.540 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-24T23:51:48.716 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-24T23:51:48.834 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-24T23:51:48.935 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-24T23:51:49.094 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up postgresql-client (14+238) ... 2023-10-24T23:51:49.223 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-24T23:51:49.701 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:51:49.701 INFO:teuthology.orchestra.run.smithi110.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-24T23:51:49.701 INFO:teuthology.orchestra.run.smithi110.stdout:which means they are currently configured by ifupdown: 2023-10-24T23:51:49.701 INFO:teuthology.orchestra.run.smithi110.stdout:- enp3s0f1 2023-10-24T23:51:49.702 INFO:teuthology.orchestra.run.smithi110.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-24T23:51:49.702 INFO:teuthology.orchestra.run.smithi110.stdout:remove their configuration from /etc/network/interfaces. 2023-10-24T23:51:49.702 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:51:49.831 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-24T23:51:50.185 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-24T23:51:50.528 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-24T23:51:50.758 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-24T23:51:50.867 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-24T23:51:50.985 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up postgresql (14+238) ... 2023-10-24T23:51:51.083 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-24T23:51:51.111 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-24T23:51:51.210 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-24T23:51:51.254 INFO:teuthology.orchestra.run.smithi107.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-24T23:51:51.336 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-24T23:51:51.462 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up postgresql-common (238) ... 2023-10-24T23:51:51.505 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-24T23:51:51.506 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-24T23:51:52.106 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-24T23:51:52.355 INFO:teuthology.orchestra.run.smithi110.stdout:Adding user postgres to group ssl-cert 2023-10-24T23:51:52.392 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-24T23:51:52.551 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-24T23:51:52.597 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:51:52.597 INFO:teuthology.orchestra.run.smithi110.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-24T23:51:52.823 INFO:teuthology.orchestra.run.smithi110.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-24T23:51:52.823 INFO:teuthology.orchestra.run.smithi110.stdout:Removing obsolete dictionary files: 2023-10-24T23:51:53.401 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-24T23:51:53.790 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-24T23:51:53.905 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-24T23:51:54.005 INFO:teuthology.orchestra.run.smithi107.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-24T23:51:54.096 INFO:teuthology.orchestra.run.smithi110.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-24T23:51:54.112 INFO:teuthology.orchestra.run.smithi110.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-24T23:51:54.163 INFO:teuthology.orchestra.run.smithi110.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-24T23:51:54.164 INFO:teuthology.orchestra.run.smithi110.stdout:This user must also own the server process. 2023-10-24T23:51:54.164 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:51:54.164 INFO:teuthology.orchestra.run.smithi110.stdout:The database cluster will be initialized with locale "en_US". 2023-10-24T23:51:54.164 INFO:teuthology.orchestra.run.smithi110.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-24T23:51:54.164 INFO:teuthology.orchestra.run.smithi110.stdout:The default text search configuration will be set to "english". 2023-10-24T23:51:54.165 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:51:54.165 INFO:teuthology.orchestra.run.smithi110.stdout:Data page checksums are disabled. 2023-10-24T23:51:54.165 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:51:54.165 INFO:teuthology.orchestra.run.smithi110.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-24T23:51:54.166 INFO:teuthology.orchestra.run.smithi110.stdout:creating subdirectories ... ok 2023-10-24T23:51:54.166 INFO:teuthology.orchestra.run.smithi110.stdout:selecting dynamic shared memory implementation ... posix 2023-10-24T23:51:54.181 INFO:teuthology.orchestra.run.smithi110.stdout:selecting default max_connections ... 100 2023-10-24T23:51:54.202 INFO:teuthology.orchestra.run.smithi110.stdout:selecting default shared_buffers ... 128MB 2023-10-24T23:51:54.214 INFO:teuthology.orchestra.run.smithi110.stdout:selecting default time zone ... Etc/UTC 2023-10-24T23:51:54.214 INFO:teuthology.orchestra.run.smithi110.stdout:creating configuration files ... ok 2023-10-24T23:51:54.328 INFO:teuthology.orchestra.run.smithi110.stdout:running bootstrap script ... ok 2023-10-24T23:51:54.625 INFO:teuthology.orchestra.run.smithi107.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-24T23:51:54.628 DEBUG:teuthology.parallel:result is None 2023-10-24T23:51:54.961 INFO:teuthology.orchestra.run.smithi110.stdout:performing post-bootstrap initialization ... ok 2023-10-24T23:51:55.634 INFO:teuthology.orchestra.run.smithi110.stdout:syncing data to disk ... ok 2023-10-24T23:51:56.042 INFO:teuthology.orchestra.run.smithi110.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-24T23:51:58.871 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-24T23:51:58.998 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-24T23:51:59.124 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up postgresql (14+238) ... 2023-10-24T23:51:59.250 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-24T23:51:59.367 INFO:teuthology.orchestra.run.smithi110.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-24T23:51:59.605 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-24T23:51:59.605 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-24T23:52:00.187 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-24T23:52:00.489 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-24T23:52:00.657 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-24T23:52:01.659 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-24T23:52:01.743 INFO:teuthology.orchestra.run.smithi110.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-24T23:52:02.362 INFO:teuthology.orchestra.run.smithi110.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-24T23:52:02.365 DEBUG:teuthology.parallel:result is None 2023-10-24T23:52:02.366 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e652850897f60c32cb9cc05022cf9f93e3b00db1 2023-10-24T23:52:02.597 DEBUG:teuthology.orchestra.run.smithi107:> dpkg-query -W -f '${Version}' ceph 2023-10-24T23:52:02.617 INFO:teuthology.orchestra.run.smithi107.stdout:18.2.0-1095-ge6528508-1jammy 2023-10-24T23:52:02.617 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-1095-ge6528508-1jammy 2023-10-24T23:52:02.618 INFO:teuthology.task.install:The correct ceph version 18.2.0-1095-ge6528508-1jammy is installed. 2023-10-24T23:52:02.618 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e652850897f60c32cb9cc05022cf9f93e3b00db1 2023-10-24T23:52:02.834 DEBUG:teuthology.orchestra.run.smithi110:> dpkg-query -W -f '${Version}' ceph 2023-10-24T23:52:02.853 INFO:teuthology.orchestra.run.smithi110.stdout:18.2.0-1095-ge6528508-1jammy 2023-10-24T23:52:02.854 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-1095-ge6528508-1jammy 2023-10-24T23:52:02.854 INFO:teuthology.task.install:The correct ceph version 18.2.0-1095-ge6528508-1jammy is installed. 2023-10-24T23:52:02.855 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-24T23:52:02.855 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:02.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-24T23:52:02.870 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:02.870 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-24T23:52:02.908 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-24T23:52:02.909 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:02.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2023-10-24T23:52:02.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-24T23:52:02.982 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:02.983 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2023-10-24T23:52:02.996 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-24T23:52:03.051 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-24T23:52:03.051 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:03.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-24T23:52:03.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-24T23:52:03.118 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:03.118 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-24T23:52:03.131 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-24T23:52:03.187 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-24T23:52:03.187 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:03.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2023-10-24T23:52:03.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-24T23:52:03.254 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:03.255 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2023-10-24T23:52:03.267 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-24T23:52:03.323 INFO:teuthology.run_tasks:Running task ceph... 2023-10-24T23:52:03.385 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-24T23:52:03.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2023-10-24T23:52:03.388 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2023-10-24T23:52:03.400 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-24T23:52:03.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-24T23:52:03.444 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-24T23:52:03.456 INFO:tasks.ceph:Creating extra log directories... 2023-10-24T23:52:03.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-24T23:52:03.500 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-24T23:52:03.519 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-24T23:52:03.519 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-24T23:52:03.519 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-24T23:52:03.520 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:03.520 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-24T23:52:03.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-24T23:52:03.637 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:03.638 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-24T23:52:03.644 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-24T23:52:03.724 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-24T23:52:03.724 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-24T23:52:03.724 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357', 'branch': 'wip-yuri6-testing-2023-10-23-1148-reef', 'description': 'fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7435357', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri6-testing-2023-10-23-1148-reef'}, 'ceph': {'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': 'e652850897f60c32cb9cc05022cf9f93e3b00db1'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'ceph-fuse': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e652850897f60c32cb9cc05022cf9f93e3b00db1'}}, 'kclient': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri6-testing-2023-10-23-1148-reef', 'sha1': 'e652850897f60c32cb9cc05022cf9f93e3b00db1'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 7106, 'sha1': 'e652850897f60c32cb9cc05022cf9f93e3b00db1', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'wip-yuri6-testing-2023-10-23-1148-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'e652850897f60c32cb9cc05022cf9f93e3b00db1', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPL1FzpXP2qyGnObmjAsCdBxl4upbUplk4LyLEI0kr2ltoLMG8qjHtik+hMWxC3AgtLNcANRvfLwcYhciGTi1r8=', 'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDF755M2c+AY0wPO/teNzumWbZarR33MHIzud+Kqg6Eo9+1OS5/JoqKUGpHGNoV1DVCGhd2kCArBv8Hzdp+dJjU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e652850897f60c32cb9cc05022cf9f93e3b00db1'}}, {'ceph': None}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '202b180cb047e798fb131047314a862593f45403', 'timestamp': '2023-10-24_00:06:34', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-24T23:52:03.725 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-24T23:52:03.728 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-24T23:52:03.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2023-10-24T23:52:03.776 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2023-10-24T23:52:03.819 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:03.820 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2023-10-24T23:52:03.835 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-24T23:52:03.835 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2023-10-24T23:52:03.883 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-24T23:52:03.883 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-24T23:52:03.883 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 889 Links: 1 2023-10-24T23:52:03.884 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-24T23:52:03.884 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-10-24 23:45:59.799502010 +0000 2023-10-24T23:52:03.884 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-10-24 23:45:59.795502047 +0000 2023-10-24T23:52:03.884 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-10-24 23:45:59.795502047 +0000 2023-10-24T23:52:03.884 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-10-24T23:52:03.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-24T23:52:03.937 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-10-24T23:52:03.938 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-10-24T23:52:03.938 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000428935 s, 1.2 MB/s 2023-10-24T23:52:03.939 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-24T23:52:03.989 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2023-10-24T23:52:04.043 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-24T23:52:04.043 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-24T23:52:04.045 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 906 Links: 1 2023-10-24T23:52:04.045 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-24T23:52:04.045 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-10-24 23:46:00.735493650 +0000 2023-10-24T23:52:04.045 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-10-24 23:46:00.383496795 +0000 2023-10-24T23:52:04.046 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-10-24 23:46:00.383496795 +0000 2023-10-24T23:52:04.046 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-10-24T23:52:04.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-24T23:52:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-10-24T23:52:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-10-24T23:52:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000303224 s, 1.7 MB/s 2023-10-24T23:52:04.101 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-24T23:52:04.156 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2023-10-24T23:52:04.205 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-24T23:52:04.205 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-24T23:52:04.205 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 917 Links: 1 2023-10-24T23:52:04.205 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-24T23:52:04.206 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-10-24 23:46:01.315488469 +0000 2023-10-24T23:52:04.206 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-10-24 23:46:00.963491614 +0000 2023-10-24T23:52:04.206 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-10-24 23:46:00.963491614 +0000 2023-10-24T23:52:04.207 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-10-24T23:52:04.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-24T23:52:04.258 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-10-24T23:52:04.258 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-10-24T23:52:04.259 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000338813 s, 1.5 MB/s 2023-10-24T23:52:04.260 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-24T23:52:04.309 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2023-10-24T23:52:04.355 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-24T23:52:04.356 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-24T23:52:04.356 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 940 Links: 1 2023-10-24T23:52:04.356 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-24T23:52:04.356 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-10-24 23:46:01.891483325 +0000 2023-10-24T23:52:04.356 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-10-24 23:46:01.547486397 +0000 2023-10-24T23:52:04.357 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-10-24 23:46:01.547486397 +0000 2023-10-24T23:52:04.357 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-10-24T23:52:04.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-24T23:52:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-10-24T23:52:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-10-24T23:52:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000387364 s, 1.3 MB/s 2023-10-24T23:52:04.411 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-24T23:52:04.460 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-24T23:52:04.461 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:04.461 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2023-10-24T23:52:04.467 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-24T23:52:04.468 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2023-10-24T23:52:04.516 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-24T23:52:04.516 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-24T23:52:04.516 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 873 Links: 1 2023-10-24T23:52:04.517 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-24T23:52:04.517 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-10-24 23:45:58.387282170 +0000 2023-10-24T23:52:04.517 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-10-24 23:45:58.383282207 +0000 2023-10-24T23:52:04.517 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-10-24 23:45:58.383282207 +0000 2023-10-24T23:52:04.517 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-10-24T23:52:04.518 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-24T23:52:04.570 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-10-24T23:52:04.571 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-10-24T23:52:04.571 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.00031988 s, 1.6 MB/s 2023-10-24T23:52:04.573 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-24T23:52:04.622 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2023-10-24T23:52:04.668 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-24T23:52:04.669 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-24T23:52:04.669 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 891 Links: 1 2023-10-24T23:52:04.669 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-24T23:52:04.669 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-10-24 23:45:59.335273705 +0000 2023-10-24T23:52:04.670 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-10-24 23:45:58.979276883 +0000 2023-10-24T23:52:04.670 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-10-24 23:45:58.979276883 +0000 2023-10-24T23:52:04.670 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-10-24T23:52:04.670 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-24T23:52:04.722 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-10-24T23:52:04.722 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-10-24T23:52:04.723 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000302271 s, 1.7 MB/s 2023-10-24T23:52:04.723 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-24T23:52:04.774 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2023-10-24T23:52:04.820 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-24T23:52:04.820 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-24T23:52:04.820 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 909 Links: 1 2023-10-24T23:52:04.821 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-24T23:52:04.821 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-10-24 23:45:59.935268351 +0000 2023-10-24T23:52:04.821 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-10-24 23:45:59.563271673 +0000 2023-10-24T23:52:04.821 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-10-24 23:45:59.563271673 +0000 2023-10-24T23:52:04.821 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-10-24T23:52:04.822 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-24T23:52:04.874 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-10-24T23:52:04.875 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-10-24T23:52:04.875 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000304224 s, 1.7 MB/s 2023-10-24T23:52:04.876 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-24T23:52:04.925 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2023-10-24T23:52:04.972 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-24T23:52:04.972 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-24T23:52:04.972 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 926 Links: 1 2023-10-24T23:52:04.972 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-24T23:52:04.973 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-10-24 23:46:00.535262994 +0000 2023-10-24T23:52:04.973 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-10-24 23:46:00.163266316 +0000 2023-10-24T23:52:04.973 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-10-24 23:46:00.163266316 +0000 2023-10-24T23:52:04.973 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-10-24T23:52:04.974 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-24T23:52:05.026 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-10-24T23:52:05.026 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-10-24T23:52:05.026 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000324631 s, 1.6 MB/s 2023-10-24T23:52:05.027 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-24T23:52:05.077 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-24T23:52:05.077 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-24T23:52:05.078 INFO:tasks.ceph:Generating config... 2023-10-24T23:52:05.079 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-24T23:52:05.079 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2023-10-24T23:52:05.079 INFO:tasks.ceph:[client] debug client = 20 2023-10-24T23:52:05.080 INFO:tasks.ceph:[client] debug ms = 1 2023-10-24T23:52:05.080 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-24T23:52:05.080 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-24T23:52:05.081 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-24T23:52:05.081 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-24T23:52:05.081 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-24T23:52:05.081 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-10-24T23:52:05.082 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-10-24T23:52:05.082 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-10-24T23:52:05.082 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-10-24T23:52:05.082 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-24T23:52:05.083 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-24T23:52:05.083 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-24T23:52:05.083 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-24T23:52:05.083 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-10-24T23:52:05.084 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-10-24T23:52:05.084 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-10-24T23:52:05.084 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-10-24T23:52:05.084 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-10-24T23:52:05.085 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-24T23:52:05.085 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-24T23:52:05.085 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-24T23:52:05.085 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-24T23:52:05.086 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-24T23:52:05.086 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-24T23:52:05.086 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-24T23:52:05.086 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-24T23:52:05.087 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-24T23:52:05.087 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-24T23:52:05.087 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-24T23:52:05.087 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-24T23:52:05.088 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-24T23:52:05.088 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-24T23:52:05.088 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-24T23:52:05.088 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-24T23:52:05.089 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-24T23:52:05.089 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-24T23:52:05.089 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-24T23:52:05.089 INFO:tasks.ceph:[osd] filestore flush min = 0 2023-10-24T23:52:05.089 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-10-24T23:52:05.090 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-24T23:52:05.090 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-24T23:52:05.090 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-24T23:52:05.091 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-10-24T23:52:05.091 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-10-24T23:52:05.091 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-10-24T23:52:05.091 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-10-24T23:52:05.092 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-10-24T23:52:05.092 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-24T23:52:05.092 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-10-24T23:52:05.092 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-24T23:52:05.093 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-24T23:52:05.093 INFO:tasks.ceph:Setting up mon.a... 2023-10-24T23:52:05.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-24T23:52:05.128 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.keyring 2023-10-24T23:52:05.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-24T23:52:05.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-24T23:52:05.278 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.107'), ('mon.b', '172.21.15.110'), ('mon.c', '[v2:172.21.15.110:3301,v1:172.21.15.110:6790]')] 2023-10-24T23:52:05.279 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.107,172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-24T23:52:05.280 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:05.280 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-24T23:52:05.327 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.107 --add b 172.21.15.110 --addv c '[v2:172.21.15.110:3301,v1:172.21.15.110:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:05.411 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:05.411 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: generated fsid c963b8f9-0f03-467c-8a02-12930711a2e8 2023-10-24T23:52:05.412 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = pacific 2023-10-24T23:52:05.412 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 0 2023-10-24T23:52:05.412 INFO:teuthology.orchestra.run.smithi107.stdout:fsid c963b8f9-0f03-467c-8a02-12930711a2e8 2023-10-24T23:52:05.412 INFO:teuthology.orchestra.run.smithi107.stdout:last_changed 2023-10-24T23:52:05.413766+0000 2023-10-24T23:52:05.412 INFO:teuthology.orchestra.run.smithi107.stdout:created 2023-10-24T23:52:05.413766+0000 2023-10-24T23:52:05.413 INFO:teuthology.orchestra.run.smithi107.stdout:min_mon_release 16 (pacific) 2023-10-24T23:52:05.413 INFO:teuthology.orchestra.run.smithi107.stdout:election_strategy: 1 2023-10-24T23:52:05.413 INFO:teuthology.orchestra.run.smithi107.stdout:0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2023-10-24T23:52:05.413 INFO:teuthology.orchestra.run.smithi107.stdout:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-10-24T23:52:05.413 INFO:teuthology.orchestra.run.smithi107.stdout:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2023-10-24T23:52:05.414 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-24T23:52:05.415 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-24T23:52:05.464 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c963b8f9-0f03-467c-8a02-12930711a2e8... 2023-10-24T23:52:05.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-24T23:52:05.508 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-24T23:52:05.537 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2023-10-24T23:52:05.538 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2023-10-24T23:52:05.538 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-24T23:52:05.539 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = cephx 2023-10-24T23:52:05.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.539 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2023-10-24T23:52:05.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.540 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2023-10-24T23:52:05.540 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.540 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2023-10-24T23:52:05.540 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2023-10-24T23:52:05.541 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.541 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2023-10-24T23:52:05.541 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2023-10-24T23:52:05.541 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.541 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-24T23:52:05.542 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2023-10-24T23:52:05.542 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.542 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2023-10-24T23:52:05.542 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-24T23:52:05.543 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.543 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2023-10-24T23:52:05.543 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.543 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2023-10-24T23:52:05.543 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2023-10-24T23:52:05.543 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2023-10-24T23:52:05.544 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2023-10-24T23:52:05.544 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2023-10-24T23:52:05.544 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2023-10-24T23:52:05.544 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2023-10-24T23:52:05.544 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-24T23:52:05.545 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-24T23:52:05.545 INFO:teuthology.orchestra.run.smithi107.stdout: mon_allow_pool_size_one = true 2023-10-24T23:52:05.545 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.545 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-24T23:52:05.545 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.546 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2023-10-24T23:52:05.546 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.546 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2023-10-24T23:52:05.546 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.546 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2023-10-24T23:52:05.547 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2023-10-24T23:52:05.547 INFO:teuthology.orchestra.run.smithi107.stdout: mon health detail to clog = false 2023-10-24T23:52:05.547 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = "172.21.15.107,172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790]" 2023-10-24T23:52:05.547 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = c963b8f9-0f03-467c-8a02-12930711a2e8 2023-10-24T23:52:05.547 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.548 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2023-10-24T23:52:05.548 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2023-10-24T23:52:05.548 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.548 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2023-10-24T23:52:05.548 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 600 2023-10-24T23:52:05.548 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock profile = high_recovery_ops 2023-10-24T23:52:05.549 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.549 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2023-10-24T23:52:05.549 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2023-10-24T23:52:05.549 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.549 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2023-10-24T23:52:05.550 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2023-10-24T23:52:05.550 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2023-10-24T23:52:05.550 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.550 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2023-10-24T23:52:05.550 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2023-10-24T23:52:05.551 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.551 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2023-10-24T23:52:05.551 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.551 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2023-10-24T23:52:05.551 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.552 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2023-10-24T23:52:05.552 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.552 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2023-10-24T23:52:05.552 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2023-10-24T23:52:05.552 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.552 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2023-10-24T23:52:05.553 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2023-10-24T23:52:05.553 INFO:teuthology.orchestra.run.smithi107.stdout: bdev async discard = True 2023-10-24T23:52:05.553 INFO:teuthology.orchestra.run.smithi107.stdout: bdev enable discard = True 2023-10-24T23:52:05.553 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore allocator = bitmap 2023-10-24T23:52:05.553 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2023-10-24T23:52:05.554 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2023-10-24T23:52:05.554 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 1/20 2023-10-24T23:52:05.554 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 1/20 2023-10-24T23:52:05.554 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-10-24T23:52:05.554 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 20 2023-10-24T23:52:05.555 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 4/10 2023-10-24T23:52:05.555 INFO:teuthology.orchestra.run.smithi107.stdout: filestore flush min = 0 2023-10-24T23:52:05.555 INFO:teuthology.orchestra.run.smithi107.stdout: mon client directed command retry = 5 2023-10-24T23:52:05.555 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2023-10-24T23:52:05.555 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2023-10-24T23:52:05.556 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2023-10-24T23:52:05.556 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay max = 1 2023-10-24T23:52:05.556 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay probability = 0.005 2023-10-24T23:52:05.556 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay type = client mds 2023-10-24T23:52:05.556 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject socket failures = 2500 2023-10-24T23:52:05.557 INFO:teuthology.orchestra.run.smithi107.stdout: osd blocked scrub grace period = 3600 2023-10-24T23:52:05.557 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2023-10-24T23:52:05.557 INFO:teuthology.orchestra.run.smithi107.stdout: osd max markdown count = 1000 2023-10-24T23:52:05.557 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2023-10-24T23:52:05.558 INFO:teuthology.orchestra.run.smithi107.stdout: osd op complaint time = 180 2023-10-24T23:52:05.558 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.558 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2023-10-24T23:52:05.558 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-10-24T23:52:05.559 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2023-10-24T23:52:05.559 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2023-10-24T23:52:05.559 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2023-10-24T23:52:05.559 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2023-10-24T23:52:05.559 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2023-10-24T23:52:05.560 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2023-10-24T23:52:05.560 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.560 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2023-10-24T23:52:05.560 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-10-24T23:52:05.560 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2023-10-24T23:52:05.561 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 20 2023-10-24T23:52:05.561 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2023-10-24T23:52:05.561 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2023-10-24T23:52:05.561 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2023-10-24T23:52:05.561 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2023-10-24T23:52:05.561 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2023-10-24T23:52:05.562 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2023-10-24T23:52:05.562 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2023-10-24T23:52:05.562 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.562 INFO:teuthology.orchestra.run.smithi107.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-24T23:52:05.562 INFO:teuthology.orchestra.run.smithi107.stdout: auth mon ticket ttl = 660 # 11m 2023-10-24T23:52:05.563 INFO:teuthology.orchestra.run.smithi107.stdout: auth service ticket ttl = 240 # 4m 2023-10-24T23:52:05.563 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.563 INFO:teuthology.orchestra.run.smithi107.stdout: # don't complain about insecure global_id in the test suite 2023-10-24T23:52:05.563 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-24T23:52:05.563 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-24T23:52:05.564 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.564 INFO:teuthology.orchestra.run.smithi107.stdout: # 1m isn't quite enough 2023-10-24T23:52:05.564 INFO:teuthology.orchestra.run.smithi107.stdout: mon_down_mkfs_grace = 2m 2023-10-24T23:52:05.564 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.564 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_filestore_osds = false 2023-10-24T23:52:05.564 INFO:teuthology.orchestra.run.smithi107.stdout: mon op complaint time = 120 2023-10-24T23:52:05.565 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:05.565 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2023-10-24T23:52:05.565 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2023-10-24T23:52:05.565 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2023-10-24T23:52:05.565 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2023-10-24T23:52:05.566 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-24T23:52:05.566 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-24T23:52:05.566 INFO:teuthology.orchestra.run.smithi107.stdout: client mount timeout = 600 2023-10-24T23:52:05.566 INFO:teuthology.orchestra.run.smithi107.stdout: client_shutdown_timeout = 120 2023-10-24T23:52:05.566 INFO:teuthology.orchestra.run.smithi107.stdout: debug client = 20 2023-10-24T23:52:05.566 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-10-24T23:52:05.567 INFO:teuthology.orchestra.run.smithi107.stdout: rados mon op timeout = 900 2023-10-24T23:52:05.567 INFO:teuthology.orchestra.run.smithi107.stdout: rados osd op timeout = 900 2023-10-24T23:52:05.567 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2023-10-24T23:52:05.567 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.b] 2023-10-24T23:52:05.567 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.c] 2023-10-24T23:52:05.568 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.a] 2023-10-24T23:52:05.568 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.c] 2023-10-24T23:52:05.568 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.b] 2023-10-24T23:52:05.568 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.d] 2023-10-24T23:52:05.568 INFO:teuthology.orchestra.run.smithi107.stdout:[mds] 2023-10-24T23:52:05.569 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds = 20 2023-10-24T23:52:05.569 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds balancer = 20 2023-10-24T23:52:05.569 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2023-10-24T23:52:05.569 INFO:teuthology.orchestra.run.smithi107.stdout: mds bal fragment size max = 10000 2023-10-24T23:52:05.569 INFO:teuthology.orchestra.run.smithi107.stdout: mds bal merge size = 5 2023-10-24T23:52:05.570 INFO:teuthology.orchestra.run.smithi107.stdout: mds bal split bits = 3 2023-10-24T23:52:05.570 INFO:teuthology.orchestra.run.smithi107.stdout: mds bal split size = 100 2023-10-24T23:52:05.570 INFO:teuthology.orchestra.run.smithi107.stdout: mds debug frag = True 2023-10-24T23:52:05.570 INFO:teuthology.orchestra.run.smithi107.stdout: mds debug scatterstat = True 2023-10-24T23:52:05.570 INFO:teuthology.orchestra.run.smithi107.stdout: mds op complaint time = 180 2023-10-24T23:52:05.571 INFO:teuthology.orchestra.run.smithi107.stdout: mds verify scatter = True 2023-10-24T23:52:05.571 INFO:teuthology.orchestra.run.smithi107.stdout: mon client directed command retry = 5 2023-10-24T23:52:05.571 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay max = 1 2023-10-24T23:52:05.571 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay probability = 0.005 2023-10-24T23:52:05.571 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject delay type = client mds osd 2023-10-24T23:52:05.571 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject socket failures = 2500 2023-10-24T23:52:05.572 INFO:teuthology.orchestra.run.smithi107.stdout: osd op complaint time = 180 2023-10-24T23:52:05.572 INFO:teuthology.orchestra.run.smithi107.stdout: rados mon op timeout = 900 2023-10-24T23:52:05.572 INFO:teuthology.orchestra.run.smithi107.stdout: rados osd op timeout = 900 2023-10-24T23:52:05.574 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2023-10-24T23:52:05.574 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2023-10-24T23:52:05.574 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-24T23:52:05.574 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2023-10-24T23:52:05.574 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.575 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2023-10-24T23:52:05.575 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.575 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2023-10-24T23:52:05.575 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.575 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2023-10-24T23:52:05.576 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2023-10-24T23:52:05.576 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.576 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2023-10-24T23:52:05.576 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2023-10-24T23:52:05.576 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.577 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-24T23:52:05.577 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2023-10-24T23:52:05.577 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.577 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2023-10-24T23:52:05.577 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-24T23:52:05.577 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.578 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2023-10-24T23:52:05.578 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.578 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2023-10-24T23:52:05.578 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2023-10-24T23:52:05.578 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2023-10-24T23:52:05.579 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2023-10-24T23:52:05.579 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2023-10-24T23:52:05.579 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2023-10-24T23:52:05.579 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2023-10-24T23:52:05.579 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-24T23:52:05.580 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-24T23:52:05.580 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2023-10-24T23:52:05.580 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.580 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-24T23:52:05.580 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.581 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2023-10-24T23:52:05.581 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.581 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2023-10-24T23:52:05.581 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.581 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2023-10-24T23:52:05.582 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2023-10-24T23:52:05.582 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2023-10-24T23:52:05.582 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "172.21.15.107,172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790]" 2023-10-24T23:52:05.582 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = c963b8f9-0f03-467c-8a02-12930711a2e8 2023-10-24T23:52:05.582 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.582 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2023-10-24T23:52:05.583 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2023-10-24T23:52:05.583 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.583 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2023-10-24T23:52:05.583 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2023-10-24T23:52:05.583 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2023-10-24T23:52:05.584 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.584 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2023-10-24T23:52:05.584 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2023-10-24T23:52:05.584 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.584 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2023-10-24T23:52:05.585 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2023-10-24T23:52:05.585 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2023-10-24T23:52:05.585 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.585 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2023-10-24T23:52:05.585 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2023-10-24T23:52:05.585 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.586 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2023-10-24T23:52:05.586 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.586 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2023-10-24T23:52:05.586 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.586 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2023-10-24T23:52:05.587 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.587 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2023-10-24T23:52:05.587 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2023-10-24T23:52:05.587 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.587 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2023-10-24T23:52:05.588 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2023-10-24T23:52:05.588 INFO:teuthology.orchestra.run.smithi110.stdout: bdev async discard = True 2023-10-24T23:52:05.588 INFO:teuthology.orchestra.run.smithi110.stdout: bdev enable discard = True 2023-10-24T23:52:05.588 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore allocator = bitmap 2023-10-24T23:52:05.588 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2023-10-24T23:52:05.588 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2023-10-24T23:52:05.589 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 1/20 2023-10-24T23:52:05.589 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 1/20 2023-10-24T23:52:05.589 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-24T23:52:05.589 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2023-10-24T23:52:05.589 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 4/10 2023-10-24T23:52:05.590 INFO:teuthology.orchestra.run.smithi110.stdout: filestore flush min = 0 2023-10-24T23:52:05.590 INFO:teuthology.orchestra.run.smithi110.stdout: mon client directed command retry = 5 2023-10-24T23:52:05.590 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2023-10-24T23:52:05.590 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2023-10-24T23:52:05.590 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2023-10-24T23:52:05.591 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject delay max = 1 2023-10-24T23:52:05.591 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject delay probability = 0.005 2023-10-24T23:52:05.591 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject delay type = client mds 2023-10-24T23:52:05.591 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject socket failures = 2500 2023-10-24T23:52:05.591 INFO:teuthology.orchestra.run.smithi110.stdout: osd blocked scrub grace period = 3600 2023-10-24T23:52:05.591 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2023-10-24T23:52:05.592 INFO:teuthology.orchestra.run.smithi110.stdout: osd max markdown count = 1000 2023-10-24T23:52:05.592 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2023-10-24T23:52:05.592 INFO:teuthology.orchestra.run.smithi110.stdout: osd op complaint time = 180 2023-10-24T23:52:05.592 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.592 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2023-10-24T23:52:05.593 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-24T23:52:05.593 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2023-10-24T23:52:05.593 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-10-24T23:52:05.593 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-10-24T23:52:05.593 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-10-24T23:52:05.593 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-10-24T23:52:05.594 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2023-10-24T23:52:05.594 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.594 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2023-10-24T23:52:05.594 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-24T23:52:05.594 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-10-24T23:52:05.595 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2023-10-24T23:52:05.595 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-10-24T23:52:05.595 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2023-10-24T23:52:05.595 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2023-10-24T23:52:05.595 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-10-24T23:52:05.596 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2023-10-24T23:52:05.596 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2023-10-24T23:52:05.596 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-10-24T23:52:05.596 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.596 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-24T23:52:05.597 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2023-10-24T23:52:05.597 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2023-10-24T23:52:05.597 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.597 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2023-10-24T23:52:05.597 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-24T23:52:05.597 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-24T23:52:05.598 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.598 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2023-10-24T23:52:05.598 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2023-10-24T23:52:05.598 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.598 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2023-10-24T23:52:05.599 INFO:teuthology.orchestra.run.smithi110.stdout: mon op complaint time = 120 2023-10-24T23:52:05.599 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-10-24T23:52:05.599 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2023-10-24T23:52:05.599 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2023-10-24T23:52:05.599 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2023-10-24T23:52:05.599 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2023-10-24T23:52:05.600 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-24T23:52:05.600 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-24T23:52:05.600 INFO:teuthology.orchestra.run.smithi110.stdout: client mount timeout = 600 2023-10-24T23:52:05.600 INFO:teuthology.orchestra.run.smithi110.stdout: client_shutdown_timeout = 120 2023-10-24T23:52:05.600 INFO:teuthology.orchestra.run.smithi110.stdout: debug client = 20 2023-10-24T23:52:05.601 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-24T23:52:05.601 INFO:teuthology.orchestra.run.smithi110.stdout: rados mon op timeout = 900 2023-10-24T23:52:05.601 INFO:teuthology.orchestra.run.smithi110.stdout: rados osd op timeout = 900 2023-10-24T23:52:05.601 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2023-10-24T23:52:05.601 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2023-10-24T23:52:05.602 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2023-10-24T23:52:05.602 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.a] 2023-10-24T23:52:05.602 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.c] 2023-10-24T23:52:05.602 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.b] 2023-10-24T23:52:05.602 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.d] 2023-10-24T23:52:05.603 INFO:teuthology.orchestra.run.smithi110.stdout:[mds] 2023-10-24T23:52:05.603 INFO:teuthology.orchestra.run.smithi110.stdout: debug mds = 20 2023-10-24T23:52:05.603 INFO:teuthology.orchestra.run.smithi110.stdout: debug mds balancer = 20 2023-10-24T23:52:05.603 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-10-24T23:52:05.603 INFO:teuthology.orchestra.run.smithi110.stdout: mds bal fragment size max = 10000 2023-10-24T23:52:05.603 INFO:teuthology.orchestra.run.smithi110.stdout: mds bal merge size = 5 2023-10-24T23:52:05.604 INFO:teuthology.orchestra.run.smithi110.stdout: mds bal split bits = 3 2023-10-24T23:52:05.604 INFO:teuthology.orchestra.run.smithi110.stdout: mds bal split size = 100 2023-10-24T23:52:05.604 INFO:teuthology.orchestra.run.smithi110.stdout: mds debug frag = True 2023-10-24T23:52:05.604 INFO:teuthology.orchestra.run.smithi110.stdout: mds debug scatterstat = True 2023-10-24T23:52:05.604 INFO:teuthology.orchestra.run.smithi110.stdout: mds op complaint time = 180 2023-10-24T23:52:05.604 INFO:teuthology.orchestra.run.smithi110.stdout: mds verify scatter = True 2023-10-24T23:52:05.605 INFO:teuthology.orchestra.run.smithi110.stdout: mon client directed command retry = 5 2023-10-24T23:52:05.605 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject delay max = 1 2023-10-24T23:52:05.605 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject delay probability = 0.005 2023-10-24T23:52:05.605 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject delay type = client mds osd 2023-10-24T23:52:05.605 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject socket failures = 2500 2023-10-24T23:52:05.606 INFO:teuthology.orchestra.run.smithi110.stdout: osd op complaint time = 180 2023-10-24T23:52:05.606 INFO:teuthology.orchestra.run.smithi110.stdout: rados mon op timeout = 900 2023-10-24T23:52:05.606 INFO:teuthology.orchestra.run.smithi110.stdout: rados osd op timeout = 900 2023-10-24T23:52:05.606 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-24T23:52:05.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-24T23:52:05.663 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-24T23:52:05.664 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:05.664 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-24T23:52:05.670 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:05.670 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-24T23:52:05.719 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2023-10-24T23:52:05.719 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:05.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-24T23:52:05.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-24T23:52:05.783 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:05.784 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:05.831 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2023-10-24T23:52:05.831 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:05.832 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-24T23:52:05.832 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-24T23:52:05.855 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:05.855 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:05.904 INFO:tasks.ceph:Setting up mon nodes... 2023-10-24T23:52:05.905 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-24T23:52:05.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-24T23:52:05.947 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-24T23:52:05.950 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-24T23:52:05.993 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-24T23:52:06.005 INFO:tasks.ceph:Setting up mds nodes... 2023-10-24T23:52:06.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-24T23:52:06.047 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-24T23:52:06.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-24T23:52:06.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-24T23:52:06.188 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-24T23:52:06.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-24T23:52:06.245 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-24T23:52:06.289 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-24T23:52:06.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-24T23:52:06.349 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-24T23:52:06.432 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-24T23:52:06.435 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-24T23:52:06.490 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-24T23:52:06.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-24T23:52:06.526 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-24T23:52:06.536 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-24T23:52:06.571 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-24T23:52:06.581 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-24T23:52:06.582 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-24T23:52:06.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-24T23:52:06.596 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-24T23:52:06.596 INFO:tasks.ceph:role: osd.0 2023-10-24T23:52:06.596 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com 2023-10-24T23:52:06.597 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-24T23:52:06.658 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-24T23:52:06.658 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-24T23:52:06.659 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-24T23:52:06.659 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-24T23:52:06.659 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-24T23:52:06.659 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-10-24T23:52:06.660 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-24T23:52:06.660 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-24T23:52:06.660 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-24T23:52:06.660 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-24T23:52:07.110 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-10-24T23:52:07.112 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-10-24T23:52:07.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-24T23:52:07.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-24T23:52:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:52:07.230 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: /sbin/restorecon: command not found 2023-10-24T23:52:07.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-24T23:52:07.287 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-24T23:52:07.287 INFO:tasks.ceph:role: osd.1 2023-10-24T23:52:07.287 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com 2023-10-24T23:52:07.288 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-24T23:52:07.352 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-24T23:52:07.353 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-24T23:52:07.353 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-24T23:52:07.353 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-24T23:52:07.353 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-24T23:52:07.354 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-10-24T23:52:07.354 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-24T23:52:07.354 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-24T23:52:07.354 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-24T23:52:07.354 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-24T23:52:07.817 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-10-24T23:52:07.819 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-10-24T23:52:07.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-24T23:52:07.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-24T23:52:07.959 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: /sbin/restorecon: command not found 2023-10-24T23:52:07.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:52:07.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-24T23:52:08.015 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-24T23:52:08.015 INFO:tasks.ceph:role: osd.2 2023-10-24T23:52:08.015 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com 2023-10-24T23:52:08.016 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-24T23:52:08.079 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-24T23:52:08.079 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-24T23:52:08.079 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-24T23:52:08.079 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-24T23:52:08.080 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-24T23:52:08.080 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-10-24T23:52:08.080 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-24T23:52:08.080 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-24T23:52:08.080 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-24T23:52:08.080 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-24T23:52:08.558 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-10-24T23:52:08.560 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-10-24T23:52:08.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-24T23:52:08.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-24T23:52:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: /sbin/restorecon: command not found 2023-10-24T23:52:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:52:08.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-24T23:52:08.763 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-24T23:52:08.763 INFO:tasks.ceph:role: osd.3 2023-10-24T23:52:08.763 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com 2023-10-24T23:52:08.764 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-24T23:52:08.830 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-24T23:52:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-24T23:52:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-24T23:52:08.831 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-24T23:52:08.832 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-24T23:52:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2023-10-24T23:52:08.832 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-24T23:52:08.832 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-24T23:52:08.832 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-24T23:52:08.833 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-24T23:52:09.247 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2023-10-24T23:52:09.249 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2023-10-24T23:52:09.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-24T23:52:09.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-24T23:52:09.358 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: /sbin/restorecon: command not found 2023-10-24T23:52:09.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:52:09.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:09.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:09.594+0000 7fd469575640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-24T23:52:09.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:09.598+0000 7fd469575640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-24T23:52:09.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:09.598+0000 7fd469575640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-24T23:52:09.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:09.598+0000 7fd469575640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-24T23:52:09.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:09.598+0000 7fd469575640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-24T23:52:09.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:09.602+0000 7fd469575640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-24T23:52:12.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-24T23:52:12.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:12.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:12.710+0000 7f55aae04640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-24T23:52:12.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:12.710+0000 7f55aae04640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-24T23:52:12.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:12.710+0000 7f55aae04640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-24T23:52:12.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:12.710+0000 7f55aae04640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-24T23:52:12.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:12.710+0000 7f55aae04640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-24T23:52:12.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:12.710+0000 7f55aae04640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-24T23:52:15.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-24T23:52:15.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:15.854+0000 7f40509cb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-24T23:52:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:15.854+0000 7f40509cb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-24T23:52:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:15.854+0000 7f40509cb640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-24T23:52:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:15.854+0000 7f40509cb640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-24T23:52:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:15.854+0000 7f40509cb640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-24T23:52:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:15.854+0000 7f40509cb640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-24T23:52:18.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-24T23:52:18.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:19.022+0000 7f6c67534640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-24T23:52:19.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:19.022+0000 7f6c67534640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-24T23:52:19.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:19.022+0000 7f6c67534640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-24T23:52:19.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:19.022+0000 7f6c67534640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-24T23:52:19.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:19.022+0000 7f6c67534640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-24T23:52:19.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:19.026+0000 7f6c67534640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-24T23:52:22.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-24T23:52:22.104 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-24T23:52:22.118 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-24T23:52:22.118 INFO:tasks.ceph:role: osd.4 2023-10-24T23:52:22.118 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-24T23:52:22.119 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-24T23:52:22.171 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-24T23:52:22.171 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-24T23:52:22.171 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-24T23:52:22.171 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-24T23:52:22.172 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-24T23:52:22.172 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-10-24T23:52:22.172 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-24T23:52:22.172 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-24T23:52:22.172 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-24T23:52:22.173 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-24T23:52:22.195 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-10-24T23:52:22.197 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-10-24T23:52:22.197 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-24T23:52:22.256 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-24T23:52:22.307 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-10-24T23:52:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:52:22.309 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-24T23:52:22.361 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-24T23:52:22.361 INFO:tasks.ceph:role: osd.5 2023-10-24T23:52:22.362 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-24T23:52:22.362 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-24T23:52:22.415 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-24T23:52:22.416 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-24T23:52:22.416 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-24T23:52:22.416 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-24T23:52:22.416 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-24T23:52:22.416 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-10-24T23:52:22.416 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-24T23:52:22.417 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-24T23:52:22.417 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-24T23:52:22.417 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-24T23:52:22.440 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-10-24T23:52:22.442 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-10-24T23:52:22.442 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-24T23:52:22.497 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-24T23:52:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:52:22.550 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-10-24T23:52:22.550 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-24T23:52:22.603 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-24T23:52:22.604 INFO:tasks.ceph:role: osd.6 2023-10-24T23:52:22.604 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-24T23:52:22.604 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-24T23:52:22.659 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-24T23:52:22.659 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-24T23:52:22.660 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-24T23:52:22.660 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-24T23:52:22.660 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-24T23:52:22.660 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-10-24T23:52:22.660 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-24T23:52:22.661 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-24T23:52:22.661 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-24T23:52:22.661 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-24T23:52:22.683 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-10-24T23:52:22.685 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-10-24T23:52:22.685 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-24T23:52:22.748 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-24T23:52:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:52:22.801 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-10-24T23:52:22.801 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-24T23:52:22.855 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-24T23:52:22.855 INFO:tasks.ceph:role: osd.7 2023-10-24T23:52:22.855 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-24T23:52:22.855 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-24T23:52:22.911 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-24T23:52:22.912 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-24T23:52:22.912 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-24T23:52:22.912 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-24T23:52:22.912 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-24T23:52:22.913 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-10-24T23:52:22.913 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-24T23:52:22.913 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-24T23:52:22.913 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-24T23:52:22.913 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-24T23:52:22.938 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-10-24T23:52:22.940 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-10-24T23:52:22.940 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-24T23:52:23.003 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-24T23:52:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:52:23.057 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: /sbin/restorecon: command not found 2023-10-24T23:52:23.057 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:23.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:23.173+0000 7fe22c1ea640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-24T23:52:23.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:23.173+0000 7fe22c1ea640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-24T23:52:23.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:23.173+0000 7fe22c1ea640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-24T23:52:23.179 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:23.173+0000 7fe22c1ea640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-24T23:52:23.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:23.173+0000 7fe22c1ea640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-24T23:52:23.180 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:23.173+0000 7fe22c1ea640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-24T23:52:26.205 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-24T23:52:26.218 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:26.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:26.293+0000 7f60ab238640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-24T23:52:26.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:26.293+0000 7f60ab238640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-24T23:52:26.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:26.293+0000 7f60ab238640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-24T23:52:26.300 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:26.293+0000 7f60ab238640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-24T23:52:26.300 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:26.293+0000 7f60ab238640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-24T23:52:26.300 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:26.297+0000 7f60ab238640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-24T23:52:29.292 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-24T23:52:29.347 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:29.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:29.425+0000 7f0c6e25d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-24T23:52:29.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:29.429+0000 7f0c6e25d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-24T23:52:29.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:29.429+0000 7f0c6e25d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-24T23:52:29.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:29.429+0000 7f0c6e25d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-24T23:52:29.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:29.429+0000 7f0c6e25d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-24T23:52:29.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:29.429+0000 7f0c6e25d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-24T23:52:32.439 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-24T23:52:32.496 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:32.573+0000 7f389a63c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-24T23:52:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:32.573+0000 7f389a63c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-24T23:52:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:32.573+0000 7f389a63c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-24T23:52:32.576 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:32.573+0000 7f389a63c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-24T23:52:32.577 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:32.573+0000 7f389a63c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-24T23:52:32.577 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:32.573+0000 7f389a63c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-24T23:52:33.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:52:33.729 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:52:35.575 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-24T23:52:35.588 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-24T23:52:35.589 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:35.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-24T23:52:35.603 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:35.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-24T23:52:35.657 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:35.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-24T23:52:35.710 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:35.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-24T23:52:35.765 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:35.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-24T23:52:35.825 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:35.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-24T23:52:35.882 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:35.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-24T23:52:35.937 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:35.938 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-24T23:52:35.951 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:35.952 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-24T23:52:36.007 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:36.007 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-24T23:52:36.063 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:36.063 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-24T23:52:36.119 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:36.119 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-24T23:52:36.175 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:36.175 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-24T23:52:36.231 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:36.231 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-24T23:52:36.287 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:36.288 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-24T23:52:36.294 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:36.294 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-24T23:52:36.335 INFO:tasks.ceph:Adding keys to all mons... 2023-10-24T23:52:36.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-24T23:52:36.339 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-24T23:52:36.381 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.y] 2023-10-24T23:52:36.382 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQClWDhl9R+hOBAApFXUe2ARpeipxPXfhgESUQ== 2023-10-24T23:52:36.382 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.a] 2023-10-24T23:52:36.382 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCmWDhlbQH2AhAA2rFjeQzt3jmIIM2FJUZ2Wg== 2023-10-24T23:52:36.382 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.c] 2023-10-24T23:52:36.383 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCmWDhldopaCxAA6N50wG0cGXMtgv323stwJw== 2023-10-24T23:52:36.383 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2023-10-24T23:52:36.383 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCpWDhlWcDDIxAAH5RZbJOevILTg05Hhsr4ow== 2023-10-24T23:52:36.383 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2023-10-24T23:52:36.383 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCsWDhl+oekKhAAn1FcV1t15VPwOINFCXNT6A== 2023-10-24T23:52:36.384 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2023-10-24T23:52:36.384 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCvWDhlkvg9MxAAkBVp0F5U6zPfCm1YZIhvBw== 2023-10-24T23:52:36.384 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.3] 2023-10-24T23:52:36.384 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCzWDhlw4SrARAAhYdE59flRuY5G4Dc8CeUGA== 2023-10-24T23:52:36.384 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.x] 2023-10-24T23:52:36.385 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQClWDhlo+Q1OxAAySuSQNUEiQyubLz75V4vhQ== 2023-10-24T23:52:36.385 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.b] 2023-10-24T23:52:36.385 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCmWDhl3zpBERAAPZRAC1fIPFlL/ZQLELPmvg== 2023-10-24T23:52:36.385 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.d] 2023-10-24T23:52:36.385 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCmWDhl0nLFGRAAj5Mu8hTKzsBktHT7xtReGw== 2023-10-24T23:52:36.386 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.4] 2023-10-24T23:52:36.386 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQC3WDhlkWWhChAA2TmK8kjzKFmDDECutuH3TA== 2023-10-24T23:52:36.386 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.5] 2023-10-24T23:52:36.386 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQC6WDhlT7DMERAAxa7HjFUXwFSq8PlalygjNA== 2023-10-24T23:52:36.386 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.6] 2023-10-24T23:52:36.387 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQC9WDhlcICvGRAABrD2wXbA1R5mS/+OHdnXPA== 2023-10-24T23:52:36.387 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.7] 2023-10-24T23:52:36.387 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQDAWDhlwzdPIhAA94K2/DfKya7S7RyrNIpJzg== 2023-10-24T23:52:36.387 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2023-10-24T23:52:36.387 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCmWDhlpIWMHxAAks/5ubmE+B4PZhCVFhZxOA== 2023-10-24T23:52:36.387 INFO:teuthology.orchestra.run.smithi107.stdout:[client.1] 2023-10-24T23:52:36.388 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCmWDhlIdEEIhAAPfHcOi/P77+2BjICl2wEvg== 2023-10-24T23:52:36.389 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2023-10-24T23:52:36.389 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQClWDhl9R+hOBAApFXUe2ARpeipxPXfhgESUQ== 2023-10-24T23:52:36.389 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.a] 2023-10-24T23:52:36.390 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCmWDhlbQH2AhAA2rFjeQzt3jmIIM2FJUZ2Wg== 2023-10-24T23:52:36.390 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.c] 2023-10-24T23:52:36.390 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCmWDhldopaCxAA6N50wG0cGXMtgv323stwJw== 2023-10-24T23:52:36.390 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2023-10-24T23:52:36.391 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCpWDhlWcDDIxAAH5RZbJOevILTg05Hhsr4ow== 2023-10-24T23:52:36.391 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2023-10-24T23:52:36.391 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCsWDhl+oekKhAAn1FcV1t15VPwOINFCXNT6A== 2023-10-24T23:52:36.391 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2023-10-24T23:52:36.392 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCvWDhlkvg9MxAAkBVp0F5U6zPfCm1YZIhvBw== 2023-10-24T23:52:36.392 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2023-10-24T23:52:36.392 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCzWDhlw4SrARAAhYdE59flRuY5G4Dc8CeUGA== 2023-10-24T23:52:36.392 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2023-10-24T23:52:36.392 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQClWDhlo+Q1OxAAySuSQNUEiQyubLz75V4vhQ== 2023-10-24T23:52:36.393 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.b] 2023-10-24T23:52:36.393 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCmWDhl3zpBERAAPZRAC1fIPFlL/ZQLELPmvg== 2023-10-24T23:52:36.393 INFO:teuthology.orchestra.run.smithi110.stdout:[mds.d] 2023-10-24T23:52:36.393 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCmWDhl0nLFGRAAj5Mu8hTKzsBktHT7xtReGw== 2023-10-24T23:52:36.393 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2023-10-24T23:52:36.394 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC3WDhlkWWhChAA2TmK8kjzKFmDDECutuH3TA== 2023-10-24T23:52:36.394 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2023-10-24T23:52:36.394 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC6WDhlT7DMERAAxa7HjFUXwFSq8PlalygjNA== 2023-10-24T23:52:36.394 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2023-10-24T23:52:36.394 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQC9WDhlcICvGRAABrD2wXbA1R5mS/+OHdnXPA== 2023-10-24T23:52:36.395 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2023-10-24T23:52:36.395 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQDAWDhlwzdPIhAA94K2/DfKya7S7RyrNIpJzg== 2023-10-24T23:52:36.395 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2023-10-24T23:52:36.395 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCmWDhlpIWMHxAAks/5ubmE+B4PZhCVFhZxOA== 2023-10-24T23:52:36.395 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2023-10-24T23:52:36.396 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQCmWDhlIdEEIhAAPfHcOi/P77+2BjICl2wEvg== 2023-10-24T23:52:36.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-24T23:52:36.434 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-24T23:52:36.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-24T23:52:36.520 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-24T23:52:36.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-24T23:52:36.607 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-24T23:52:36.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:36.692 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:36.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:36.772 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:36.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:36.852 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:36.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:36.935 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:36.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-24T23:52:37.015 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-24T23:52:37.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-24T23:52:37.100 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-24T23:52:37.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-24T23:52:37.188 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-24T23:52:37.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:37.279 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:37.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:37.363 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:37.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:37.447 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:37.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:37.531 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-24T23:52:37.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-24T23:52:37.616 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-24T23:52:37.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-24T23:52:37.700 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-24T23:52:37.746 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-24T23:52:37.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-24T23:52:37.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-24T23:52:37.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-24T23:52:37.868 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-24T23:52:37.882 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-24T23:52:37.982 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-24T23:52:37.993 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-24T23:52:38.044 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-24T23:52:38.136 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-24T23:52:38.147 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:38.149 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-24T23:52:38.194 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-24T23:52:38.195 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-24T23:52:38.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-24T23:52:38.198 INFO:tasks.ceph.mon.a:Started 2023-10-24T23:52:38.198 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-24T23:52:38.199 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-24T23:52:38.241 INFO:tasks.ceph.mon.b:Started 2023-10-24T23:52:38.241 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-24T23:52:38.241 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-24T23:52:38.243 INFO:tasks.ceph.mon.c:Started 2023-10-24T23:52:38.244 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-24T23:52:38.244 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-24T23:52:38.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-24T23:52:38.247 INFO:tasks.ceph.mgr.y:Started 2023-10-24T23:52:38.247 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-24T23:52:38.248 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-24T23:52:38.249 INFO:tasks.ceph.mgr.x:Started 2023-10-24T23:52:38.250 INFO:tasks.ceph:Setting crush tunables to default 2023-10-24T23:52:38.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd crush tunables default 2023-10-24T23:52:38.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876bb0f640 1 Processor -- start 2023-10-24T23:52:38.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876bb0f640 1 -- start start 2023-10-24T23:52:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876bb0f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 0x7f876406c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876bb0f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 0x7f876415f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876bb0f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876406d200 con 0x7f876406c460 2023-10-24T23:52:38.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876bb0f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87640aa300 con 0x7f8764158b30 2023-10-24T23:52:38.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876bb0f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87641552a0 con 0x7f87640aae20 2023-10-24T23:52:38.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f87698ab640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f876406c460 0x7f8764158420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58488/0 (socket says 172.21.15.107:58488) 2023-10-24T23:52:38.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f87698ab640 1 -- 172.21.15.107:0/22193987 learned_addr learned my addr 172.21.15.107:0/22193987 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:52:38.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f87690aa640 1 --2- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 0x7f876406c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f87690aa640 1 -- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 msgr2=0x7f876406c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-24T23:52:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f87690aa640 1 -- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 msgr2=0x7f876406c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-24T23:52:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f87690aa640 1 --2- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 0x7f876406c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-24T23:52:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f87690aa640 1 --2- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 0x7f876406c840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-24T23:52:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876a0ac640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 0x7f876415f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:38.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876a0ac640 1 -- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 msgr2=0x7f876415f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-24T23:52:38.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876a0ac640 1 -- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 msgr2=0x7f876415f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-24T23:52:38.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876a0ac640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 0x7f876415f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-24T23:52:38.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.361+0000 7f876a0ac640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 0x7f876415f1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-24T23:52:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.565+0000 7f87690aa640 1 --2- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 0x7f876406c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.565+0000 7f876a0ac640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 0x7f876415f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.565+0000 7f87690aa640 1 -- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 msgr2=0x7f876406c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-24T23:52:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.565+0000 7f87690aa640 1 -- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 msgr2=0x7f876406c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-24T23:52:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.565+0000 7f87690aa640 1 --2- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 0x7f876406c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-24T23:52:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.565+0000 7f876a0ac640 1 -- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 msgr2=0x7f876415f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-24T23:52:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.565+0000 7f876a0ac640 1 -- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 msgr2=0x7f876415f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-24T23:52:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.565+0000 7f876a0ac640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 0x7f876415f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-24T23:52:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.565+0000 7f87690aa640 1 --2- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 0x7f876406c840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-24T23:52:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.565+0000 7f876a0ac640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 0x7f876415f1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-24T23:52:38.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670544070 0 0) 0x7f876406d200 con 0x7f876406c460 2023-10-24T23:52:38.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8750003530 con 0x7f876406c460 2023-10-24T23:52:38.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 198858190 0 0) 0x7f8750003530 con 0x7f876406c460 2023-10-24T23:52:38.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 msgr2=0x7f876415f1a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-24T23:52:38.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 0x7f876415f1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:38.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 msgr2=0x7f876406c840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-24T23:52:38.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 --2- 172.21.15.107:0/22193987 >> v2:172.21.15.107:3300/0 conn(0x7f8764158b30 0x7f876406c840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:38.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87640c1250 con 0x7f876406c460 2023-10-24T23:52:38.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f8760001800 con 0x7f876406c460 2023-10-24T23:52:38.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 >> v1:172.21.15.107:6789/0 conn(0x7f876406c460 legacy=0x7f8764158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:38.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8750003980 0x7f8750003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:38.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 0x7f87500047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:38.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8750004d10 0x7f8750005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:38.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87640aa300 con 0x7f8750004d10 2023-10-24T23:52:38.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87641552a0 con 0x7f8750003980 2023-10-24T23:52:38.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f875000bac0 con 0x7f87640aae20 2023-10-24T23:52:38.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f876a0ac640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8750004d10 0x7f8750005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:38.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87698ab640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8750003980 0x7f8750003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:38.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87690aa640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 0x7f87500047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:38.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f876a0ac640 1 -- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 msgr2=0x7f87500047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:38.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f876a0ac640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640aae20 0x7f87500047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:38.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f876a0ac640 1 -- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8750003980 msgr2=0x7f8750003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:38.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f876a0ac640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8750003980 0x7f8750003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:38.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f876a0ac640 1 -- 172.21.15.107:0/22193987 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8750003530 con 0x7f8750004d10 2023-10-24T23:52:38.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.597+0000 7f87698ab640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8750003980 0x7f8750003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:52:38.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.601+0000 7f876a0ac640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8750004d10 0x7f8750005190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f875c009fa0 tx=0x7f875c01cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9c9f2dc8ba9d500 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:38.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.601+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c0a5e80 con 0x7f8750004d10 2023-10-24T23:52:38.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.605+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c0a9070 con 0x7f8750004d10 2023-10-24T23:52:38.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.605+0000 7f87688a9640 1 -- 172.21.15.107:0/22193987 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c0af640 con 0x7f8750004d10 2023-10-24T23:52:38.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.605+0000 7f876bb0f640 1 -- 172.21.15.107:0/22193987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8750004d10 msgr2=0x7f8750005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:38.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.605+0000 7f876bb0f640 1 --2- 172.21.15.107:0/22193987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8750004d10 0x7f8750005190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f875c009fa0 tx=0x7f875c01cf70 comp rx=0 tx=0).stop 2023-10-24T23:52:38.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.605+0000 7f876a0ac640 1 -- 172.21.15.107:0/22193987 reap_dead start 2023-10-24T23:52:38.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.605+0000 7f876bb0f640 1 -- 172.21.15.107:0/22193987 shutdown_connections 2023-10-24T23:52:38.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.605+0000 7f876bb0f640 1 -- 172.21.15.107:0/22193987 >> 172.21.15.107:0/22193987 conn(0x7f87640c3140 msgr2=0x7f87640c3550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:38.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.605+0000 7f876bb0f640 1 -- 172.21.15.107:0/22193987 shutdown_connections 2023-10-24T23:52:38.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.605+0000 7f876bb0f640 1 -- 172.21.15.107:0/22193987 wait complete. 2023-10-24T23:52:38.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.605+0000 7f876bb0f640 1 Processor -- start 2023-10-24T23:52:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.605+0000 7f876bb0f640 1 -- start start 2023-10-24T23:52:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876bb0f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87641ed6a0 0x7f8764157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876bb0f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640c4ed0 0x7f8764144e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876bb0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87641578a0 0x7f87641453b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:38.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876bb0f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876414d750 con 0x7f87641578a0 2023-10-24T23:52:38.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876bb0f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876414d0d0 con 0x7f87641ed6a0 2023-10-24T23:52:38.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876bb0f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876414c910 con 0x7f87640c4ed0 2023-10-24T23:52:38.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876a0ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87641578a0 0x7f87641453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:38.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876a0ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87641578a0 0x7f87641453b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54592/0 (socket says 172.21.15.107:54592) 2023-10-24T23:52:38.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876a0ac640 1 -- 172.21.15.107:0/2023621213 learned_addr learned my addr 172.21.15.107:0/2023621213 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:38.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f87698ab640 1 --2- 172.21.15.107:0/2023621213 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87641ed6a0 0x7f8764157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:38.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f87690aa640 1 --2- 172.21.15.107:0/2023621213 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640c4ed0 0x7f8764144e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:38.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876a0ac640 1 -- 172.21.15.107:0/2023621213 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640c4ed0 msgr2=0x7f8764144e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:38.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876a0ac640 1 --2- 172.21.15.107:0/2023621213 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640c4ed0 0x7f8764144e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:38.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876a0ac640 1 -- 172.21.15.107:0/2023621213 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87641ed6a0 msgr2=0x7f8764157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:38.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876a0ac640 1 --2- 172.21.15.107:0/2023621213 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87641ed6a0 0x7f8764157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:38.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876a0ac640 1 -- 172.21.15.107:0/2023621213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87641edff0 con 0x7f87641578a0 2023-10-24T23:52:38.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f87698ab640 1 --2- 172.21.15.107:0/2023621213 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87641ed6a0 0x7f8764157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:52:38.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876a0ac640 1 --2- 172.21.15.107:0/2023621213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87641578a0 0x7f87641453b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f875c01d810 tx=0x7f8750005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:38.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f875a7fc640 1 -- 172.21.15.107:0/2023621213 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c0ba640 con 0x7f87641578a0 2023-10-24T23:52:38.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f875a7fc640 1 -- 172.21.15.107:0/2023621213 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875c0ba7a0 con 0x7f87641578a0 2023-10-24T23:52:38.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876bb0f640 1 -- 172.21.15.107:0/2023621213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f876414cbb0 con 0x7f87641578a0 2023-10-24T23:52:38.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f875a7fc640 1 -- 172.21.15.107:0/2023621213 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875c0bab10 con 0x7f87641578a0 2023-10-24T23:52:38.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876bb0f640 1 -- 172.21.15.107:0/2023621213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87640c0190 con 0x7f87641578a0 2023-10-24T23:52:38.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f875a7fc640 1 -- 172.21.15.107:0/2023621213 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f875c0c6460 con 0x7f87641578a0 2023-10-24T23:52:38.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f875a7fc640 1 -- 172.21.15.107:0/2023621213 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f875c009610 con 0x7f87641578a0 2023-10-24T23:52:38.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.609+0000 7f876bb0f640 1 -- 172.21.15.107:0/2023621213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f876414cdd0 con 0x7f87641578a0 2023-10-24T23:52:38.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.613+0000 7f875a7fc640 1 -- 172.21.15.107:0/2023621213 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f875c0ab070 con 0x7f87641578a0 2023-10-24T23:52:38.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.681+0000 7f876bb0f640 1 -- 172.21.15.107:0/2023621213 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8764073c00 con 0x7f87641578a0 2023-10-24T23:52:38.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.685+0000 7f875a7fc640 1 -- 172.21.15.107:0/2023621213 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f875c009ba0 con 0x7f87641578a0 2023-10-24T23:52:38.690 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2023-10-24T23:52:38.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.689+0000 7f876bb0f640 1 -- 172.21.15.107:0/2023621213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87641578a0 msgr2=0x7f87641453b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:38.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.689+0000 7f876bb0f640 1 --2- 172.21.15.107:0/2023621213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87641578a0 0x7f87641453b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f875c01d810 tx=0x7f8750005800 comp rx=0 tx=0).stop 2023-10-24T23:52:38.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.689+0000 7f876bb0f640 1 -- 172.21.15.107:0/2023621213 shutdown_connections 2023-10-24T23:52:38.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.689+0000 7f876bb0f640 1 --2- 172.21.15.107:0/2023621213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87641578a0 0x7f87641453b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:38.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.689+0000 7f876bb0f640 1 --2- 172.21.15.107:0/2023621213 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87640c4ed0 0x7f8764144e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:38.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.689+0000 7f876bb0f640 1 --2- 172.21.15.107:0/2023621213 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87641ed6a0 0x7f8764157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:38.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.689+0000 7f876bb0f640 1 -- 172.21.15.107:0/2023621213 >> 172.21.15.107:0/2023621213 conn(0x7f87640c3140 msgr2=0x7f87640c3550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:38.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.689+0000 7f876bb0f640 1 -- 172.21.15.107:0/2023621213 shutdown_connections 2023-10-24T23:52:38.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:38.689+0000 7f876bb0f640 1 -- 172.21.15.107:0/2023621213 wait complete. 2023-10-24T23:52:38.713 INFO:tasks.ceph:check_enable_crimson: False 2023-10-24T23:52:38.714 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-24T23:52:38.714 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:38.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-24T23:52:38.724 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:38.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-24T23:52:38.779 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:38.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-24T23:52:38.831 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:52:38.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-24T23:52:38.833 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:38.829+0000 7fd74cede180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-24T23:52:38.843 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:38.841+0000 7fc8fab5b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-24T23:52:38.853 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:38.853 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-24T23:52:38.866 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:38.867 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-24T23:52:38.922 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:38.922 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-24T23:52:38.974 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:52:38.974 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-24T23:52:39.027 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 1a640883-d481-424e-ab6c-f1e1d6a15ea5 0 2023-10-24T23:52:39.198 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.193+0000 7f86c5b82640 1 Processor -- start 2023-10-24T23:52:39.198 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.193+0000 7f86c5b82640 1 -- start start 2023-10-24T23:52:39.200 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86c5b82640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f86c014ed10 0x7f86c00c2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.200 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86c5b82640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86c0154c30 0x7f86c00a9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.200 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86c5b82640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c014f0f0 con 0x7f86c00c2db0 2023-10-24T23:52:39.201 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86c5b82640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c01524e0 con 0x7f86c0154c30 2023-10-24T23:52:39.201 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86c5b82640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c0151860 con 0x7f86c014ed10 2023-10-24T23:52:39.201 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86bf7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f86c014ed10 0x7f86c00c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.201 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86bf7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f86c014ed10 0x7f86c00c2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40884/0 (socket says 172.21.15.110:40884) 2023-10-24T23:52:39.201 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86bf7fe640 1 -- 172.21.15.110:0/198766371 learned_addr learned my addr 172.21.15.110:0/198766371 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:39.202 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86beffd640 1 --2- 172.21.15.110:0/198766371 >> v2:172.21.15.107:3300/0 conn(0x7f86c0154c30 0x7f86c00a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.202 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86bf7fe640 1 -- 172.21.15.110:0/198766371 >> v2:172.21.15.107:3300/0 conn(0x7f86c0154c30 msgr2=0x7f86c00a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:39.202 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86bf7fe640 1 --2- 172.21.15.110:0/198766371 >> v2:172.21.15.107:3300/0 conn(0x7f86c0154c30 0x7f86c00a9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.202 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86bf7fe640 1 -- 172.21.15.110:0/198766371 >> v1:172.21.15.110:6789/0 conn(0x7f86c00c2db0 legacy=0x7f86c00a9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.202 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86bf7fe640 1 -- 172.21.15.110:0/198766371 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c0151b40 con 0x7f86c014ed10 2023-10-24T23:52:39.203 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86beffd640 1 --2- 172.21.15.110:0/198766371 >> v2:172.21.15.107:3300/0 conn(0x7f86c0154c30 0x7f86c00a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:52:39.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86bf7fe640 1 --2- 172.21.15.110:0/198766371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86c014ed10 0x7f86c00c2870 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f86c0069760 tx=0x7f86b801c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f576ff5ed9173c62 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:39.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86be7fc640 1 -- 172.21.15.110:0/198766371 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b80a57d0 con 0x7f86c014ed10 2023-10-24T23:52:39.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86be7fc640 1 -- 172.21.15.110:0/198766371 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b80a5930 con 0x7f86c014ed10 2023-10-24T23:52:39.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86be7fc640 1 -- 172.21.15.110:0/198766371 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b80a5cc0 con 0x7f86c014ed10 2023-10-24T23:52:39.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86c5b82640 1 -- 172.21.15.110:0/198766371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86c014ed10 msgr2=0x7f86c00c2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86c5b82640 1 --2- 172.21.15.110:0/198766371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86c014ed10 0x7f86c00c2870 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f86c0069760 tx=0x7f86b801c910 comp rx=0 tx=0).stop 2023-10-24T23:52:39.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86c5b82640 1 -- 172.21.15.110:0/198766371 shutdown_connections 2023-10-24T23:52:39.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86c5b82640 1 --2- 172.21.15.110:0/198766371 >> v2:172.21.15.107:3300/0 conn(0x7f86c0154c30 0x7f86c00a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86c5b82640 1 --2- 172.21.15.110:0/198766371 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86c014ed10 0x7f86c00c2870 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.197+0000 7f86c5b82640 1 -- 172.21.15.110:0/198766371 >> 172.21.15.110:0/198766371 conn(0x7f86c00c3960 msgr2=0x7f86c00c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:39.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 -- 172.21.15.110:0/198766371 shutdown_connections 2023-10-24T23:52:39.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 -- 172.21.15.110:0/198766371 wait complete. 2023-10-24T23:52:39.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 Processor -- start 2023-10-24T23:52:39.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 -- start start 2023-10-24T23:52:39.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86c00c2db0 0x7f86c00c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86c014ed10 0x7f86c0150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86c0154c30 0x7f86c01eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.209 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c00c6930 con 0x7f86c00c2db0 2023-10-24T23:52:39.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c01eeb60 con 0x7f86c014ed10 2023-10-24T23:52:39.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c00c6b10 con 0x7f86c0154c30 2023-10-24T23:52:39.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86beffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86c014ed10 0x7f86c0150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86bffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86c0154c30 0x7f86c01eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.210 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86beffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86c014ed10 0x7f86c0150f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40886/0 (socket says 172.21.15.110:40886) 2023-10-24T23:52:39.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86beffd640 1 -- 172.21.15.110:0/3096968257 learned_addr learned my addr 172.21.15.110:0/3096968257 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:39.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86bf7fe640 1 --2- 172.21.15.110:0/3096968257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86c00c2db0 0x7f86c00c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86beffd640 1 -- 172.21.15.110:0/3096968257 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86c0154c30 msgr2=0x7f86c01eacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86beffd640 1 --2- 172.21.15.110:0/3096968257 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86c0154c30 0x7f86c01eacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.211 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86beffd640 1 -- 172.21.15.110:0/3096968257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86c00c2db0 msgr2=0x7f86c00c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86beffd640 1 --2- 172.21.15.110:0/3096968257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86c00c2db0 0x7f86c00c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86beffd640 1 -- 172.21.15.110:0/3096968257 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b8009b30 con 0x7f86c014ed10 2023-10-24T23:52:39.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86beffd640 1 --2- 172.21.15.110:0/3096968257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86c014ed10 0x7f86c0150f30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f86b000c7f0 tx=0x7f86b000ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:39.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86b7fff640 1 -- 172.21.15.110:0/3096968257 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0007820 con 0x7f86c014ed10 2023-10-24T23:52:39.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86b7fff640 1 -- 172.21.15.110:0/3096968257 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b0007980 con 0x7f86c014ed10 2023-10-24T23:52:39.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86b7fff640 1 -- 172.21.15.110:0/3096968257 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0007d30 con 0x7f86c014ed10 2023-10-24T23:52:39.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 -- 172.21.15.110:0/3096968257 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c01ee230 con 0x7f86c014ed10 2023-10-24T23:52:39.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 -- 172.21.15.110:0/3096968257 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c01ee7f0 con 0x7f86c014ed10 2023-10-24T23:52:39.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86b7fff640 1 -- 172.21.15.110:0/3096968257 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f86b001a660 con 0x7f86c014ed10 2023-10-24T23:52:39.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86c5b82640 1 -- 172.21.15.110:0/3096968257 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c00c1540 con 0x7f86c014ed10 2023-10-24T23:52:39.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.201+0000 7f86b7fff640 1 -- 172.21.15.110:0/3096968257 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f86b00119d0 con 0x7f86c014ed10 2023-10-24T23:52:39.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.205+0000 7f86b7fff640 1 -- 172.21.15.110:0/3096968257 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f86b0039b00 con 0x7f86c014ed10 2023-10-24T23:52:39.277 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.273+0000 7f86c5b82640 1 -- 172.21.15.110:0/3096968257 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1a640883-d481-424e-ab6c-f1e1d6a15ea5", "id": 0} v 0) v1 -- 0x7f86c004e130 con 0x7f86c014ed10 2023-10-24T23:52:39.284 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.281+0000 7f86b7fff640 1 -- 172.21.15.110:0/3096968257 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1a640883-d481-424e-ab6c-f1e1d6a15ea5", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f86b0028030 con 0x7f86c014ed10 2023-10-24T23:52:39.284 INFO:teuthology.orchestra.run.smithi110.stdout:0 2023-10-24T23:52:39.285 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.281+0000 7f86c5b82640 1 -- 172.21.15.110:0/3096968257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86c014ed10 msgr2=0x7f86c0150f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.281+0000 7f86c5b82640 1 --2- 172.21.15.110:0/3096968257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86c014ed10 0x7f86c0150f30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f86b000c7f0 tx=0x7f86b000ccb0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.281+0000 7f86c5b82640 1 -- 172.21.15.110:0/3096968257 shutdown_connections 2023-10-24T23:52:39.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.281+0000 7f86c5b82640 1 --2- 172.21.15.110:0/3096968257 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86c0154c30 0x7f86c01eacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.286 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.281+0000 7f86c5b82640 1 --2- 172.21.15.110:0/3096968257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86c014ed10 0x7f86c0150f30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.281+0000 7f86c5b82640 1 --2- 172.21.15.110:0/3096968257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86c00c2db0 0x7f86c00c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.281+0000 7f86c5b82640 1 -- 172.21.15.110:0/3096968257 >> 172.21.15.110:0/3096968257 conn(0x7f86c00c3960 msgr2=0x7f86c006a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:39.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.281+0000 7f86c5b82640 1 -- 172.21.15.110:0/3096968257 shutdown_connections 2023-10-24T23:52:39.287 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.281+0000 7f86c5b82640 1 -- 172.21.15.110:0/3096968257 wait complete. 2023-10-24T23:52:39.301 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new bcf419d2-e458-4108-8a67-e253c573ad97 1 2023-10-24T23:52:39.368 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:39.365+0000 7fc8fab5b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-24T23:52:39.400 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:39.397+0000 7fd74cede180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-24T23:52:39.417 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb2b8ad640 1 Processor -- start 2023-10-24T23:52:39.418 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb2b8ad640 1 -- start start 2023-10-24T23:52:39.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb2b8ad640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feb240abfe0 0x7feb24152670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb2b8ad640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240c3450 0x7feb24069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.419 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb2b8ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb240aa680 con 0x7feb2406a3d0 2023-10-24T23:52:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb2b8ad640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb24152c80 con 0x7feb240abfe0 2023-10-24T23:52:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb2b8ad640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb24152f60 con 0x7feb240c3450 2023-10-24T23:52:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb28e48640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240c3450 0x7feb24069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb29649640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feb240abfe0 0x7feb24152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.420 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb29649640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feb240abfe0 0x7feb24152670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40902/0 (socket says 172.21.15.110:40902) 2023-10-24T23:52:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb28e48640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240c3450 0x7feb24069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54396/0 (socket says 172.21.15.110:54396) 2023-10-24T23:52:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.413+0000 7feb29649640 1 -- 172.21.15.110:0/437307016 learned_addr learned my addr 172.21.15.110:0/437307016 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb28e48640 1 -- 172.21.15.110:0/437307016 >> v2:172.21.15.110:3300/0 conn(0x7feb240abfe0 msgr2=0x7feb24152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:39.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb28e48640 1 --2- 172.21.15.110:0/437307016 >> v2:172.21.15.110:3300/0 conn(0x7feb240abfe0 0x7feb24152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb28e48640 1 -- 172.21.15.110:0/437307016 >> v1:172.21.15.107:6789/0 conn(0x7feb2406a3d0 legacy=0x7feb24156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb28e48640 1 -- 172.21.15.110:0/437307016 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb24153240 con 0x7feb240c3450 2023-10-24T23:52:39.422 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb29649640 1 --2- 172.21.15.110:0/437307016 >> v2:172.21.15.110:3300/0 conn(0x7feb240abfe0 0x7feb24152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:52:39.423 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb28e48640 1 --2- 172.21.15.110:0/437307016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240c3450 0x7feb24069e90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb200097c0 tx=0x7feb2001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46e8db5ba7908a48 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:39.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb1bfff640 1 -- 172.21.15.110:0/437307016 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb200ab600 con 0x7feb240c3450 2023-10-24T23:52:39.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb1bfff640 1 -- 172.21.15.110:0/437307016 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb200a7050 con 0x7feb240c3450 2023-10-24T23:52:39.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb1bfff640 1 -- 172.21.15.110:0/437307016 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb200ab980 con 0x7feb240c3450 2023-10-24T23:52:39.424 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb2b8ad640 1 -- 172.21.15.110:0/437307016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240c3450 msgr2=0x7feb24069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb2b8ad640 1 --2- 172.21.15.110:0/437307016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240c3450 0x7feb24069e90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7feb200097c0 tx=0x7feb2001c3c0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb2b8ad640 1 -- 172.21.15.110:0/437307016 shutdown_connections 2023-10-24T23:52:39.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb2b8ad640 1 --2- 172.21.15.110:0/437307016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240c3450 0x7feb24069e90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb2b8ad640 1 --2- 172.21.15.110:0/437307016 >> v2:172.21.15.110:3300/0 conn(0x7feb240abfe0 0x7feb24152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.425 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb2b8ad640 1 -- 172.21.15.110:0/437307016 >> 172.21.15.110:0/437307016 conn(0x7feb240c3960 msgr2=0x7feb240c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb2b8ad640 1 -- 172.21.15.110:0/437307016 shutdown_connections 2023-10-24T23:52:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.417+0000 7feb2b8ad640 1 -- 172.21.15.110:0/437307016 wait complete. 2023-10-24T23:52:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb2b8ad640 1 Processor -- start 2023-10-24T23:52:39.426 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb2b8ad640 1 -- start start 2023-10-24T23:52:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb2b8ad640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb2406a3d0 0x7feb241ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.428 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb2b8ad640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240abfe0 0x7feb24144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb2b8ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb240c3450 0x7feb2406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb2b8ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2414d180 con 0x7feb240c3450 2023-10-24T23:52:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb2b8ad640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb241ef800 con 0x7feb2406a3d0 2023-10-24T23:52:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb2b8ad640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb240c61c0 con 0x7feb240abfe0 2023-10-24T23:52:39.429 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb28e48640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240abfe0 0x7feb24144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb29649640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb2406a3d0 0x7feb241ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb28e48640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240abfe0 0x7feb24144c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54398/0 (socket says 172.21.15.110:54398) 2023-10-24T23:52:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb28e48640 1 -- 172.21.15.110:0/2457481589 learned_addr learned my addr 172.21.15.110:0/2457481589 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:39.430 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb29e4a640 1 --2- 172.21.15.110:0/2457481589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb240c3450 0x7feb2406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb28e48640 1 -- 172.21.15.110:0/2457481589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb2406a3d0 msgr2=0x7feb241ef2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb28e48640 1 --2- 172.21.15.110:0/2457481589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb2406a3d0 0x7feb241ef2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb28e48640 1 -- 172.21.15.110:0/2457481589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb240c3450 msgr2=0x7feb2406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb28e48640 1 --2- 172.21.15.110:0/2457481589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb240c3450 0x7feb2406c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb28e48640 1 -- 172.21.15.110:0/2457481589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb140094d0 con 0x7feb240abfe0 2023-10-24T23:52:39.431 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb29649640 1 --2- 172.21.15.110:0/2457481589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb2406a3d0 0x7feb241ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:52:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb29e4a640 1 --2- 172.21.15.110:0/2457481589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb240c3450 0x7feb2406c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:52:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb28e48640 1 --2- 172.21.15.110:0/2457481589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240abfe0 0x7feb24144c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7feb2001aa70 tx=0x7feb20019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb19ffb640 1 -- 172.21.15.110:0/2457481589 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb200b4400 con 0x7feb240abfe0 2023-10-24T23:52:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb19ffb640 1 -- 172.21.15.110:0/2457481589 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb200b4560 con 0x7feb240abfe0 2023-10-24T23:52:39.432 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb19ffb640 1 -- 172.21.15.110:0/2457481589 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb200b4910 con 0x7feb240abfe0 2023-10-24T23:52:39.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb2b8ad640 1 -- 172.21.15.110:0/2457481589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb20009420 con 0x7feb240abfe0 2023-10-24T23:52:39.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb2b8ad640 1 -- 172.21.15.110:0/2457481589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb240c6780 con 0x7feb240abfe0 2023-10-24T23:52:39.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb19ffb640 1 -- 172.21.15.110:0/2457481589 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7feb200b2040 con 0x7feb240abfe0 2023-10-24T23:52:39.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb19ffb640 1 -- 172.21.15.110:0/2457481589 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7feb200c03c0 con 0x7feb240abfe0 2023-10-24T23:52:39.433 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.421+0000 7feb2b8ad640 1 -- 172.21.15.110:0/2457481589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb240ab3a0 con 0x7feb240abfe0 2023-10-24T23:52:39.434 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.425+0000 7feb19ffb640 1 -- 172.21.15.110:0/2457481589 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7feb200b0070 con 0x7feb240abfe0 2023-10-24T23:52:39.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.497+0000 7feb2b8ad640 1 -- 172.21.15.110:0/2457481589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd new", "uuid": "bcf419d2-e458-4108-8a67-e253c573ad97", "id": 1} v 0) v1 -- 0x7feb2415cdd0 con 0x7feb240abfe0 2023-10-24T23:52:39.509 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.505+0000 7feb19ffb640 1 -- 172.21.15.110:0/2457481589 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bcf419d2-e458-4108-8a67-e253c573ad97", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7feb200c1460 con 0x7feb240abfe0 2023-10-24T23:52:39.509 INFO:teuthology.orchestra.run.smithi110.stdout:1 2023-10-24T23:52:39.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.505+0000 7feb2b8ad640 1 -- 172.21.15.110:0/2457481589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240abfe0 msgr2=0x7feb24144c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.505+0000 7feb2b8ad640 1 --2- 172.21.15.110:0/2457481589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240abfe0 0x7feb24144c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7feb2001aa70 tx=0x7feb20019160 comp rx=0 tx=0).stop 2023-10-24T23:52:39.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.505+0000 7feb2b8ad640 1 -- 172.21.15.110:0/2457481589 shutdown_connections 2023-10-24T23:52:39.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.505+0000 7feb2b8ad640 1 --2- 172.21.15.110:0/2457481589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb240c3450 0x7feb2406c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.511 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.505+0000 7feb2b8ad640 1 --2- 172.21.15.110:0/2457481589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb240abfe0 0x7feb24144c40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.505+0000 7feb2b8ad640 1 --2- 172.21.15.110:0/2457481589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb2406a3d0 0x7feb241ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.505+0000 7feb2b8ad640 1 -- 172.21.15.110:0/2457481589 >> 172.21.15.110:0/2457481589 conn(0x7feb240c3960 msgr2=0x7feb24153ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:39.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.505+0000 7feb2b8ad640 1 -- 172.21.15.110:0/2457481589 shutdown_connections 2023-10-24T23:52:39.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.505+0000 7feb2b8ad640 1 -- 172.21.15.110:0/2457481589 wait complete. 2023-10-24T23:52:39.526 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 3083fe01-9371-44a6-bc38-cb318ad3d8df 2 2023-10-24T23:52:39.552 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:39.549+0000 7fc8fab5b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-24T23:52:39.601 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:39.597+0000 7fd74cede180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-24T23:52:39.642 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.637+0000 7fd30a772640 1 Processor -- start 2023-10-24T23:52:39.643 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.637+0000 7fd30a772640 1 -- start start 2023-10-24T23:52:39.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd30a772640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd304071c50 0x7fd30406d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.644 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd30a772640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3041560b0 0x7fd3041564e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd30a772640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30406f4d0 con 0x7fd3040aa200 2023-10-24T23:52:39.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd30a772640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3040a9780 con 0x7fd304071c50 2023-10-24T23:52:39.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd30a772640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30406d5a0 con 0x7fd3041560b0 2023-10-24T23:52:39.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd303fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd304071c50 0x7fd30406d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.645 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd308d0f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3041560b0 0x7fd3041564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd303fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd304071c50 0x7fd30406d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40916/0 (socket says 172.21.15.110:40916) 2023-10-24T23:52:39.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd308d0f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3041560b0 0x7fd3041564e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54406/0 (socket says 172.21.15.110:54406) 2023-10-24T23:52:39.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd303fff640 1 -- 172.21.15.110:0/1502644030 learned_addr learned my addr 172.21.15.110:0/1502644030 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:39.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd303fff640 1 -- 172.21.15.110:0/1502644030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3041560b0 msgr2=0x7fd3041564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:39.646 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd303fff640 1 --2- 172.21.15.110:0/1502644030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3041560b0 0x7fd3041564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd303fff640 1 -- 172.21.15.110:0/1502644030 >> v1:172.21.15.107:6789/0 conn(0x7fd3040aa200 legacy=0x7fd30414fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:39.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd303fff640 1 -- 172.21.15.110:0/1502644030 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd304155c40 con 0x7fd304071c50 2023-10-24T23:52:39.647 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd308d0f640 1 --2- 172.21.15.110:0/1502644030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3041560b0 0x7fd3041564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:52:39.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd303fff640 1 --2- 172.21.15.110:0/1502644030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd304071c50 0x7fd30406d060 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009870 tx=0x7fd2f401c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b2a24f1eb17e810 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:39.650 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd3037fe640 1 -- 172.21.15.110:0/1502644030 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40ab600 con 0x7fd304071c50 2023-10-24T23:52:39.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd3037fe640 1 -- 172.21.15.110:0/1502644030 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f40ab760 con 0x7fd304071c50 2023-10-24T23:52:39.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd3037fe640 1 -- 172.21.15.110:0/1502644030 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40abaf0 con 0x7fd304071c50 2023-10-24T23:52:39.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd30a772640 1 -- 172.21.15.110:0/1502644030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd304071c50 msgr2=0x7fd30406d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd30a772640 1 --2- 172.21.15.110:0/1502644030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd304071c50 0x7fd30406d060 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4009870 tx=0x7fd2f401c440 comp rx=0 tx=0).stop 2023-10-24T23:52:39.651 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd30a772640 1 -- 172.21.15.110:0/1502644030 shutdown_connections 2023-10-24T23:52:39.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd30a772640 1 --2- 172.21.15.110:0/1502644030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3041560b0 0x7fd3041564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.641+0000 7fd30a772640 1 --2- 172.21.15.110:0/1502644030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd304071c50 0x7fd30406d060 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 -- 172.21.15.110:0/1502644030 >> 172.21.15.110:0/1502644030 conn(0x7fd304070ed0 msgr2=0x7fd3040712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:39.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 -- 172.21.15.110:0/1502644030 shutdown_connections 2023-10-24T23:52:39.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 -- 172.21.15.110:0/1502644030 wait complete. 2023-10-24T23:52:39.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 Processor -- start 2023-10-24T23:52:39.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 -- start start 2023-10-24T23:52:39.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd304071c50 0x7fd3041efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3040aa200 0x7fd3040c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.653 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3041560b0 0x7fd3040c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3040c6a60 con 0x7fd3041560b0 2023-10-24T23:52:39.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd304143e80 con 0x7fd304071c50 2023-10-24T23:52:39.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3040c6d20 con 0x7fd3040aa200 2023-10-24T23:52:39.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd303fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd304071c50 0x7fd3041efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.654 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd2fbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3040aa200 0x7fd3040c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd303fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd304071c50 0x7fd3041efe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40924/0 (socket says 172.21.15.110:40924) 2023-10-24T23:52:39.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd303fff640 1 -- 172.21.15.110:0/307854498 learned_addr learned my addr 172.21.15.110:0/307854498 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:39.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd308d0f640 1 --2- 172.21.15.110:0/307854498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3041560b0 0x7fd3040c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd2fbfff640 1 -- 172.21.15.110:0/307854498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd304071c50 msgr2=0x7fd3041efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd2fbfff640 1 --2- 172.21.15.110:0/307854498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd304071c50 0x7fd3041efe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd2fbfff640 1 -- 172.21.15.110:0/307854498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3041560b0 msgr2=0x7fd3040c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd2fbfff640 1 --2- 172.21.15.110:0/307854498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3041560b0 0x7fd3040c1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd2fbfff640 1 -- 172.21.15.110:0/307854498 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc009a70 con 0x7fd3040aa200 2023-10-24T23:52:39.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd303fff640 1 --2- 172.21.15.110:0/307854498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd304071c50 0x7fd3041efe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:52:39.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd2fbfff640 1 --2- 172.21.15.110:0/307854498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3040aa200 0x7fd3040c71b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00c3e0 tx=0x7fd2ec00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:39.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd3017fa640 1 -- 172.21.15.110:0/307854498 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec011790 con 0x7fd3040aa200 2023-10-24T23:52:39.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd3017fa640 1 -- 172.21.15.110:0/307854498 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ec0118f0 con 0x7fd3040aa200 2023-10-24T23:52:39.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 -- 172.21.15.110:0/307854498 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f40094d0 con 0x7fd3040aa200 2023-10-24T23:52:39.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd3017fa640 1 -- 172.21.15.110:0/307854498 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec011ce0 con 0x7fd3040aa200 2023-10-24T23:52:39.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 -- 172.21.15.110:0/307854498 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3041440d0 con 0x7fd3040aa200 2023-10-24T23:52:39.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd3017fa640 1 -- 172.21.15.110:0/307854498 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd2ec01c760 con 0x7fd3040aa200 2023-10-24T23:52:39.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd3017fa640 1 -- 172.21.15.110:0/307854498 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fd2ec012b10 con 0x7fd3040aa200 2023-10-24T23:52:39.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.645+0000 7fd30a772640 1 -- 172.21.15.110:0/307854498 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3040c6e90 con 0x7fd3040aa200 2023-10-24T23:52:39.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.649+0000 7fd3017fa640 1 -- 172.21.15.110:0/307854498 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fd2ec011e40 con 0x7fd3040aa200 2023-10-24T23:52:39.721 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.717+0000 7fd30a772640 1 -- 172.21.15.110:0/307854498 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3083fe01-9371-44a6-bc38-cb318ad3d8df", "id": 2} v 0) v1 -- 0x7fd304066680 con 0x7fd3040aa200 2023-10-24T23:52:39.729 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.725+0000 7fd3017fa640 1 -- 172.21.15.110:0/307854498 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3083fe01-9371-44a6-bc38-cb318ad3d8df", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd2ec012df0 con 0x7fd3040aa200 2023-10-24T23:52:39.730 INFO:teuthology.orchestra.run.smithi110.stdout:2 2023-10-24T23:52:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.725+0000 7fd30a772640 1 -- 172.21.15.110:0/307854498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3040aa200 msgr2=0x7fd3040c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.725+0000 7fd30a772640 1 --2- 172.21.15.110:0/307854498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3040aa200 0x7fd3040c71b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec00c3e0 tx=0x7fd2ec00c8a0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.731 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.725+0000 7fd30a772640 1 -- 172.21.15.110:0/307854498 shutdown_connections 2023-10-24T23:52:39.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.725+0000 7fd30a772640 1 --2- 172.21.15.110:0/307854498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3041560b0 0x7fd3040c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.725+0000 7fd30a772640 1 --2- 172.21.15.110:0/307854498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3040aa200 0x7fd3040c71b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.725+0000 7fd30a772640 1 --2- 172.21.15.110:0/307854498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd304071c50 0x7fd3041efe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.725+0000 7fd30a772640 1 -- 172.21.15.110:0/307854498 >> 172.21.15.110:0/307854498 conn(0x7fd304070ed0 msgr2=0x7fd304151450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:39.732 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.729+0000 7fd30a772640 1 -- 172.21.15.110:0/307854498 shutdown_connections 2023-10-24T23:52:39.733 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.729+0000 7fd30a772640 1 -- 172.21.15.110:0/307854498 wait complete. 2023-10-24T23:52:39.745 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 5b68afcb-5b5c-4723-a177-8d01a6ab2983 3 2023-10-24T23:52:39.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e0d5a4640 1 Processor -- start 2023-10-24T23:52:39.861 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e0d5a4640 1 -- start start 2023-10-24T23:52:39.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e0d5a4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8e08071f70 0x7f8e080c7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.863 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e0d5a4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e080c71a0 con 0x7f8e08155b70 2023-10-24T23:52:39.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e0d5a4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e081569f0 con 0x7f8e0814fd10 2023-10-24T23:52:39.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e0d5a4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e080a9b60 con 0x7f8e08071f70 2023-10-24T23:52:39.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e06ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8e0814fd10 0x7f8e080a9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55002/0 (socket says 172.21.15.110:55002) 2023-10-24T23:52:39.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e06ffd640 1 -- 172.21.15.110:0/4099802298 learned_addr learned my addr 172.21.15.110:0/4099802298 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-24T23:52:39.864 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e067fc640 1 --2- 172.21.15.110:0/4099802298 >> v2:172.21.15.107:3300/0 conn(0x7f8e08071f70 0x7f8e080c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231669495 0 0) 0x7f8e081569f0 con 0x7f8e0814fd10 2023-10-24T23:52:39.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df0003530 con 0x7f8e0814fd10 2023-10-24T23:52:39.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1301622626 0 0) 0x7f8df0003530 con 0x7f8e0814fd10 2023-10-24T23:52:39.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 >> v2:172.21.15.107:3300/0 conn(0x7f8e08071f70 msgr2=0x7f8e080c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:39.865 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 --2- 172.21.15.110:0/4099802298 >> v2:172.21.15.107:3300/0 conn(0x7f8e08071f70 0x7f8e080c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 >> v1:172.21.15.107:6789/0 conn(0x7f8e08155b70 legacy=0x7f8e08156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0806d150 con 0x7f8e0814fd10 2023-10-24T23:52:39.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f8dfc001680 con 0x7f8e0814fd10 2023-10-24T23:52:39.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 >> v1:172.21.15.110:6789/0 conn(0x7f8e0814fd10 legacy=0x7f8e080a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 --2- 172.21.15.110:0/4099802298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8df00039d0 0x7f8df0003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 --2- 172.21.15.110:0/4099802298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8df00047d0 0x7f8df0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 --2- 172.21.15.110:0/4099802298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8df0005190 0x7f8df000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080a9b60 con 0x7f8df00039d0 2023-10-24T23:52:39.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df000c070 con 0x7f8df0005190 2023-10-24T23:52:39.867 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df000c230 con 0x7f8df00047d0 2023-10-24T23:52:39.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e077fe640 1 --2- 172.21.15.110:0/4099802298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8df0005190 0x7f8df000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e067fc640 1 --2- 172.21.15.110:0/4099802298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8df00047d0 0x7f8df0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e06ffd640 1 --2- 172.21.15.110:0/4099802298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8df00039d0 0x7f8df0003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e077fe640 1 -- 172.21.15.110:0/4099802298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8df00047d0 msgr2=0x7f8df0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:39.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e077fe640 1 --2- 172.21.15.110:0/4099802298 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8df00047d0 0x7f8df0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.868 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e077fe640 1 -- 172.21.15.110:0/4099802298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8df00039d0 msgr2=0x7f8df0003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:39.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e077fe640 1 --2- 172.21.15.110:0/4099802298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8df00039d0 0x7f8df0003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.869 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.857+0000 7f8e077fe640 1 -- 172.21.15.110:0/4099802298 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df0003530 con 0x7f8df0005190 2023-10-24T23:52:39.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e077fe640 1 -- 172.21.15.110:0/4099802298 reap_dead start 2023-10-24T23:52:39.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e077fe640 1 --2- 172.21.15.110:0/4099802298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8df0005190 0x7f8df000b990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8e00001e70 tx=0x7f8e0001c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deea88237378f9e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:39.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00003b80 con 0x7f8df0005190 2023-10-24T23:52:39.872 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e00003d00 con 0x7f8df0005190 2023-10-24T23:52:39.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e05ffb640 1 -- 172.21.15.110:0/4099802298 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00005260 con 0x7f8df0005190 2023-10-24T23:52:39.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/4099802298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8df0005190 msgr2=0x7f8df000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e0d5a4640 1 --2- 172.21.15.110:0/4099802298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8df0005190 0x7f8df000b990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8e00001e70 tx=0x7f8e0001c760 comp rx=0 tx=0).stop 2023-10-24T23:52:39.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/4099802298 shutdown_connections 2023-10-24T23:52:39.873 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e0d5a4640 1 --2- 172.21.15.110:0/4099802298 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8df0005190 0x7f8df000b990 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/4099802298 >> 172.21.15.110:0/4099802298 conn(0x7f8e080c8020 msgr2=0x7f8e080c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:39.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/4099802298 shutdown_connections 2023-10-24T23:52:39.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/4099802298 wait complete. 2023-10-24T23:52:39.874 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e0d5a4640 1 Processor -- start 2023-10-24T23:52:39.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.861+0000 7f8e0d5a4640 1 -- start start 2023-10-24T23:52:39.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e0d5a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8df0005190 0x7f8e0806b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e0d5a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e080ab870 0x7f8e08152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e0d5a4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e081eddf0 0x7f8e08145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:39.875 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e0d5a4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080ad900 con 0x7f8df0005190 2023-10-24T23:52:39.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e0d5a4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e081ed660 con 0x7f8e081eddf0 2023-10-24T23:52:39.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e0d5a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080abcf0 con 0x7f8e080ab870 2023-10-24T23:52:39.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e067fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e080ab870 0x7f8e08152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e067fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e080ab870 0x7f8e08152820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54436/0 (socket says 172.21.15.110:54436) 2023-10-24T23:52:39.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e077fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e081eddf0 0x7f8e08145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.876 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e067fc640 1 -- 172.21.15.110:0/582794961 learned_addr learned my addr 172.21.15.110:0/582794961 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:39.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e06ffd640 1 --2- 172.21.15.110:0/582794961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8df0005190 0x7f8e0806b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:39.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e077fe640 1 -- 172.21.15.110:0/582794961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e080ab870 msgr2=0x7f8e08152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e077fe640 1 --2- 172.21.15.110:0/582794961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e080ab870 0x7f8e08152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e077fe640 1 -- 172.21.15.110:0/582794961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8df0005190 msgr2=0x7f8e0806b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.877 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e077fe640 1 --2- 172.21.15.110:0/582794961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8df0005190 0x7f8e0806b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e077fe640 1 -- 172.21.15.110:0/582794961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0806c7f0 con 0x7f8e081eddf0 2023-10-24T23:52:39.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e067fc640 1 --2- 172.21.15.110:0/582794961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e080ab870 0x7f8e08152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:52:39.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e077fe640 1 --2- 172.21.15.110:0/582794961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e081eddf0 0x7f8e08145800 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8e08155f50 tx=0x7f8df0005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:39.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8def7fe640 1 -- 172.21.15.110:0/582794961 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000ac070 con 0x7f8e081eddf0 2023-10-24T23:52:39.878 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8def7fe640 1 -- 172.21.15.110:0/582794961 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e00003b90 con 0x7f8e081eddf0 2023-10-24T23:52:39.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/582794961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0806cad0 con 0x7f8e081eddf0 2023-10-24T23:52:39.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8def7fe640 1 -- 172.21.15.110:0/582794961 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e00007dc0 con 0x7f8e081eddf0 2023-10-24T23:52:39.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/582794961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e08153060 con 0x7f8e081eddf0 2023-10-24T23:52:39.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8def7fe640 1 -- 172.21.15.110:0/582794961 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8e00006b60 con 0x7f8e081eddf0 2023-10-24T23:52:39.879 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8def7fe640 1 -- 172.21.15.110:0/582794961 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f8e000a6460 con 0x7f8e081eddf0 2023-10-24T23:52:39.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.865+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/582794961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e081edb10 con 0x7f8e081eddf0 2023-10-24T23:52:39.880 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.869+0000 7f8def7fe640 1 -- 172.21.15.110:0/582794961 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f8e081edb10 con 0x7f8e081eddf0 2023-10-24T23:52:39.937 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.933+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/582794961 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5b68afcb-5b5c-4723-a177-8d01a6ab2983", "id": 3} v 0) v1 -- 0x7f8e0814a8b0 con 0x7f8e081eddf0 2023-10-24T23:52:39.946 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.941+0000 7f8def7fe640 1 -- 172.21.15.110:0/582794961 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5b68afcb-5b5c-4723-a177-8d01a6ab2983", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8e00001a90 con 0x7f8e081eddf0 2023-10-24T23:52:39.946 INFO:teuthology.orchestra.run.smithi110.stdout:3 2023-10-24T23:52:39.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.945+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/582794961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e081eddf0 msgr2=0x7f8e08145800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:39.947 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.945+0000 7f8e0d5a4640 1 --2- 172.21.15.110:0/582794961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e081eddf0 0x7f8e08145800 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8e08155f50 tx=0x7f8df0005800 comp rx=0 tx=0).stop 2023-10-24T23:52:39.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.945+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/582794961 shutdown_connections 2023-10-24T23:52:39.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.945+0000 7f8e0d5a4640 1 --2- 172.21.15.110:0/582794961 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8e081eddf0 0x7f8e08145800 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.945+0000 7f8e0d5a4640 1 --2- 172.21.15.110:0/582794961 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8e080ab870 0x7f8e08152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.945+0000 7f8e0d5a4640 1 --2- 172.21.15.110:0/582794961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8df0005190 0x7f8e0806b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:39.948 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.945+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/582794961 >> 172.21.15.110:0/582794961 conn(0x7f8e080c8020 msgr2=0x7f8e080c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:39.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.945+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/582794961 shutdown_connections 2023-10-24T23:52:39.949 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:39.945+0000 7f8e0d5a4640 1 -- 172.21.15.110:0/582794961 wait complete. 2023-10-24T23:52:39.962 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 906cdc85-4970-46df-8824-cbf545d165be 4 2023-10-24T23:52:40.077 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f23a4e3c640 1 Processor -- start 2023-10-24T23:52:40.078 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f23a4e3c640 1 -- start start 2023-10-24T23:52:40.079 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f23a4e3c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f23a014ed10 0x7f23a00c2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f23a4e3c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f23a00c2db0 0x7f23a00a9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f23a4e3c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a014f0f0 con 0x7f23a0154c30 2023-10-24T23:52:40.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f23a4e3c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a01524e0 con 0x7f23a00c2db0 2023-10-24T23:52:40.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f23a4e3c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a0151860 con 0x7f23a014ed10 2023-10-24T23:52:40.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f239e59c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f23a014ed10 0x7f23a00c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f239e59c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f23a014ed10 0x7f23a00c2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40934/0 (socket says 172.21.15.110:40934) 2023-10-24T23:52:40.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f239e59c640 1 -- 172.21.15.110:0/1328072224 learned_addr learned my addr 172.21.15.110:0/1328072224 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:40.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f239ed9d640 1 --2- 172.21.15.110:0/1328072224 >> v2:172.21.15.107:3300/0 conn(0x7f23a00c2db0 0x7f23a00a9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.081 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f239e59c640 1 -- 172.21.15.110:0/1328072224 >> v2:172.21.15.107:3300/0 conn(0x7f23a00c2db0 msgr2=0x7f23a00a9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:40.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f239e59c640 1 --2- 172.21.15.110:0/1328072224 >> v2:172.21.15.107:3300/0 conn(0x7f23a00c2db0 0x7f23a00a9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f239e59c640 1 -- 172.21.15.110:0/1328072224 >> v1:172.21.15.107:6789/0 conn(0x7f23a0154c30 legacy=0x7f23a00a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:40.082 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.073+0000 7f239e59c640 1 -- 172.21.15.110:0/1328072224 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a0151b40 con 0x7f23a014ed10 2023-10-24T23:52:40.085 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f239e59c640 1 --2- 172.21.15.110:0/1328072224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a014ed10 0x7f23a00c2870 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f23a0069760 tx=0x7f239401c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e40f1d2fa874ddd server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f239d59a640 1 -- 172.21.15.110:0/1328072224 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23940ab600 con 0x7f23a014ed10 2023-10-24T23:52:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f239d59a640 1 -- 172.21.15.110:0/1328072224 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23940ab760 con 0x7f23a014ed10 2023-10-24T23:52:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f239d59a640 1 -- 172.21.15.110:0/1328072224 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23940abaf0 con 0x7f23a014ed10 2023-10-24T23:52:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f23a4e3c640 1 -- 172.21.15.110:0/1328072224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a014ed10 msgr2=0x7f23a00c2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.086 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f23a4e3c640 1 --2- 172.21.15.110:0/1328072224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a014ed10 0x7f23a00c2870 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f23a0069760 tx=0x7f239401c400 comp rx=0 tx=0).stop 2023-10-24T23:52:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f23a4e3c640 1 -- 172.21.15.110:0/1328072224 shutdown_connections 2023-10-24T23:52:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f23a4e3c640 1 --2- 172.21.15.110:0/1328072224 >> v2:172.21.15.107:3300/0 conn(0x7f23a00c2db0 0x7f23a00a9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.087 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f23a4e3c640 1 --2- 172.21.15.110:0/1328072224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a014ed10 0x7f23a00c2870 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f23a4e3c640 1 -- 172.21.15.110:0/1328072224 >> 172.21.15.110:0/1328072224 conn(0x7f23a00c3960 msgr2=0x7f23a00c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:40.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f23a4e3c640 1 -- 172.21.15.110:0/1328072224 shutdown_connections 2023-10-24T23:52:40.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.077+0000 7f23a4e3c640 1 -- 172.21.15.110:0/1328072224 wait complete. 2023-10-24T23:52:40.088 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f23a4e3c640 1 Processor -- start 2023-10-24T23:52:40.089 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f23a4e3c640 1 -- start start 2023-10-24T23:52:40.089 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f23a4e3c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23a00c2db0 0x7f23a00c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.089 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f23a4e3c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a014ed10 0x7f23a015a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.089 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f23a4e3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a0154c30 0x7f23a01f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f23a4e3c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a01edd60 con 0x7f23a00c2db0 2023-10-24T23:52:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f23a4e3c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a014bed0 con 0x7f23a014ed10 2023-10-24T23:52:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f23a4e3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a014c1b0 con 0x7f23a0154c30 2023-10-24T23:52:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a0154c30 0x7f23a01f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.090 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239dd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a014ed10 0x7f23a015a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a0154c30 0x7f23a01f0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54450/0 (socket says 172.21.15.110:54450) 2023-10-24T23:52:40.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239dd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a014ed10 0x7f23a015a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40946/0 (socket says 172.21.15.110:40946) 2023-10-24T23:52:40.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239ed9d640 1 -- 172.21.15.110:0/959878757 learned_addr learned my addr 172.21.15.110:0/959878757 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:40.091 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239e59c640 1 --2- 172.21.15.110:0/959878757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23a00c2db0 0x7f23a00c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239ed9d640 1 -- 172.21.15.110:0/959878757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a014ed10 msgr2=0x7f23a015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239ed9d640 1 --2- 172.21.15.110:0/959878757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a014ed10 0x7f23a015a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239ed9d640 1 -- 172.21.15.110:0/959878757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23a00c2db0 msgr2=0x7f23a00c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.092 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239ed9d640 1 --2- 172.21.15.110:0/959878757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23a00c2db0 0x7f23a00c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239ed9d640 1 -- 172.21.15.110:0/959878757 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23940094d0 con 0x7f23a0154c30 2023-10-24T23:52:40.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f239ed9d640 1 --2- 172.21.15.110:0/959878757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a0154c30 0x7f23a01f0720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2398007e60 tx=0x7f239800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:40.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f238effd640 1 -- 172.21.15.110:0/959878757 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398014420 con 0x7f23a0154c30 2023-10-24T23:52:40.093 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f238effd640 1 -- 172.21.15.110:0/959878757 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23980145a0 con 0x7f23a0154c30 2023-10-24T23:52:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f23a4e3c640 1 -- 172.21.15.110:0/959878757 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a014adc0 con 0x7f23a0154c30 2023-10-24T23:52:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f238effd640 1 -- 172.21.15.110:0/959878757 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398014950 con 0x7f23a0154c30 2023-10-24T23:52:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f23a4e3c640 1 -- 172.21.15.110:0/959878757 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a015aa50 con 0x7f23a0154c30 2023-10-24T23:52:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f238effd640 1 -- 172.21.15.110:0/959878757 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f239801e460 con 0x7f23a0154c30 2023-10-24T23:52:40.094 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f238effd640 1 -- 172.21.15.110:0/959878757 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f239801d8d0 con 0x7f23a0154c30 2023-10-24T23:52:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.081+0000 7f23a4e3c640 1 -- 172.21.15.110:0/959878757 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a00c1540 con 0x7f23a0154c30 2023-10-24T23:52:40.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.085+0000 7f238effd640 1 -- 172.21.15.110:0/959878757 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f239801db30 con 0x7f23a0154c30 2023-10-24T23:52:40.121 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:40.117+0000 7fc8fab5b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-24T23:52:40.156 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.153+0000 7f23a4e3c640 1 -- 172.21.15.110:0/959878757 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd new", "uuid": "906cdc85-4970-46df-8824-cbf545d165be", "id": 4} v 0) v1 -- 0x7f23a0070eb0 con 0x7f23a0154c30 2023-10-24T23:52:40.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.161+0000 7f238effd640 1 -- 172.21.15.110:0/959878757 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "906cdc85-4970-46df-8824-cbf545d165be", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f239802d0b0 con 0x7f23a0154c30 2023-10-24T23:52:40.167 INFO:teuthology.orchestra.run.smithi110.stdout:4 2023-10-24T23:52:40.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.165+0000 7f23a4e3c640 1 -- 172.21.15.110:0/959878757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a0154c30 msgr2=0x7f23a01f0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.165+0000 7f23a4e3c640 1 --2- 172.21.15.110:0/959878757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a0154c30 0x7f23a01f0720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2398007e60 tx=0x7f239800e4b0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.165+0000 7f23a4e3c640 1 -- 172.21.15.110:0/959878757 shutdown_connections 2023-10-24T23:52:40.169 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.165+0000 7f23a4e3c640 1 --2- 172.21.15.110:0/959878757 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23a0154c30 0x7f23a01f0720 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.169 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.165+0000 7f23a4e3c640 1 --2- 172.21.15.110:0/959878757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23a014ed10 0x7f23a015a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.169 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.165+0000 7f23a4e3c640 1 --2- 172.21.15.110:0/959878757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23a00c2db0 0x7f23a00c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.169 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.165+0000 7f23a4e3c640 1 -- 172.21.15.110:0/959878757 >> 172.21.15.110:0/959878757 conn(0x7f23a00c3960 msgr2=0x7f23a014d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:40.169 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.165+0000 7f23a4e3c640 1 -- 172.21.15.110:0/959878757 shutdown_connections 2023-10-24T23:52:40.170 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.165+0000 7f23a4e3c640 1 -- 172.21.15.110:0/959878757 wait complete. 2023-10-24T23:52:40.175 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:40.173+0000 7fc8fab5b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-24T23:52:40.183 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new e0f462d0-c6ff-41aa-9b25-2f586762248a 5 2023-10-24T23:52:40.184 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:40.181+0000 7fd74cede180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-24T23:52:40.227 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:40.225+0000 7fc8fab5b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-24T23:52:40.238 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:40.233+0000 7fd74cede180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-24T23:52:40.292 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:40.289+0000 7fd74cede180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-24T23:52:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.289+0000 7f4211a32640 1 Processor -- start 2023-10-24T23:52:40.294 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.289+0000 7f4211a32640 1 -- start start 2023-10-24T23:52:40.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f4211a32640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f420c14ed10 0x7f420c0c2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f4211a32640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420c14f0f0 con 0x7f420c154c30 2023-10-24T23:52:40.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f4211a32640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420c151860 con 0x7f420c0c2db0 2023-10-24T23:52:40.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f4211a32640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f420c1524e0 con 0x7f420c14ed10 2023-10-24T23:52:40.296 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420b7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f420c0c2db0 0x7f420c0a9c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:55016/0 (socket says 172.21.15.110:55016) 2023-10-24T23:52:40.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420b7fe640 1 -- 172.21.15.110:0/1819598110 learned_addr learned my addr 172.21.15.110:0/1819598110 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-10-24T23:52:40.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420affd640 1 --2- 172.21.15.110:0/1819598110 >> v2:172.21.15.107:3300/0 conn(0x7f420c14ed10 0x7f420c0c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420a7fc640 1 -- 172.21.15.110:0/1819598110 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277500668 0 0) 0x7f420c151860 con 0x7f420c0c2db0 2023-10-24T23:52:40.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420a7fc640 1 -- 172.21.15.110:0/1819598110 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f8005740 con 0x7f420c0c2db0 2023-10-24T23:52:40.297 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420a7fc640 1 -- 172.21.15.110:0/1819598110 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2242929860 0 0) 0x7f41f8005740 con 0x7f420c0c2db0 2023-10-24T23:52:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420affd640 1 -- 172.21.15.110:0/1819598110 >> v1:172.21.15.110:6789/0 conn(0x7f420c0c2db0 legacy=0x7f420c0a9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420affd640 1 -- 172.21.15.110:0/1819598110 >> v1:172.21.15.107:6789/0 conn(0x7f420c154c30 legacy=0x7f420c0a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.298 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420affd640 1 -- 172.21.15.110:0/1819598110 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c151b40 con 0x7f420c14ed10 2023-10-24T23:52:40.299 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420affd640 1 --2- 172.21.15.110:0/1819598110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420c14ed10 0x7f420c0c2870 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f41fc009740 tx=0x7f41fc01c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b54f50a4006bd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:40.300 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420a7fc640 1 -- 172.21.15.110:0/1819598110 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4204001a00 con 0x7f420c14ed10 2023-10-24T23:52:40.300 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420a7fc640 1 -- 172.21.15.110:0/1819598110 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41fc0a7050 con 0x7f420c14ed10 2023-10-24T23:52:40.300 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f420a7fc640 1 -- 172.21.15.110:0/1819598110 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0ab620 con 0x7f420c14ed10 2023-10-24T23:52:40.300 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f4211a32640 1 -- 172.21.15.110:0/1819598110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420c14ed10 msgr2=0x7f420c0c2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.301 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.293+0000 7f4211a32640 1 --2- 172.21.15.110:0/1819598110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420c14ed10 0x7f420c0c2870 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f41fc009740 tx=0x7f41fc01c1f0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.301 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 -- 172.21.15.110:0/1819598110 shutdown_connections 2023-10-24T23:52:40.301 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 --2- 172.21.15.110:0/1819598110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420c14ed10 0x7f420c0c2870 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.301 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 -- 172.21.15.110:0/1819598110 >> 172.21.15.110:0/1819598110 conn(0x7f420c0c3960 msgr2=0x7f420c0c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:40.302 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 -- 172.21.15.110:0/1819598110 shutdown_connections 2023-10-24T23:52:40.302 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 -- 172.21.15.110:0/1819598110 wait complete. 2023-10-24T23:52:40.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 Processor -- start 2023-10-24T23:52:40.304 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 -- start start 2023-10-24T23:52:40.305 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f420c0c2db0 0x7f420c0c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.305 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f420c14ed10 0x7f420c1456a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.305 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420c154c30 0x7f420c153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.305 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c154a60 con 0x7f420c154c30 2023-10-24T23:52:40.305 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0c6b60 con 0x7f420c0c2db0 2023-10-24T23:52:40.306 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c1503d0 con 0x7f420c14ed10 2023-10-24T23:52:40.306 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f420affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f420c0c2db0 0x7f420c0c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.306 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f420affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f420c0c2db0 0x7f420c0c4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40962/0 (socket says 172.21.15.110:40962) 2023-10-24T23:52:40.306 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f420affd640 1 -- 172.21.15.110:0/587542602 learned_addr learned my addr 172.21.15.110:0/587542602 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:40.306 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4203fff640 1 --2- 172.21.15.110:0/587542602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f420c14ed10 0x7f420c1456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.307 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f420b7fe640 1 --2- 172.21.15.110:0/587542602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420c154c30 0x7f420c153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.307 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f420affd640 1 -- 172.21.15.110:0/587542602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f420c14ed10 msgr2=0x7f420c1456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.307 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f420affd640 1 --2- 172.21.15.110:0/587542602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f420c14ed10 0x7f420c1456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.307 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f420affd640 1 -- 172.21.15.110:0/587542602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420c154c30 msgr2=0x7f420c153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.307 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f420affd640 1 --2- 172.21.15.110:0/587542602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420c154c30 0x7f420c153420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.308 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f420affd640 1 -- 172.21.15.110:0/587542602 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc009420 con 0x7f420c0c2db0 2023-10-24T23:52:40.308 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f420affd640 1 --2- 172.21.15.110:0/587542602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f420c0c2db0 0x7f420c0c4e90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0ae040 tx=0x7f41fc01aa30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:40.308 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4208ff9640 1 -- 172.21.15.110:0/587542602 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0abd80 con 0x7f420c0c2db0 2023-10-24T23:52:40.308 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4208ff9640 1 -- 172.21.15.110:0/587542602 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41fc004020 con 0x7f420c0c2db0 2023-10-24T23:52:40.308 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4208ff9640 1 -- 172.21.15.110:0/587542602 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0043d0 con 0x7f420c0c2db0 2023-10-24T23:52:40.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 -- 172.21.15.110:0/587542602 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420c150670 con 0x7f420c0c2db0 2023-10-24T23:52:40.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 -- 172.21.15.110:0/587542602 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420c0c67f0 con 0x7f420c0c2db0 2023-10-24T23:52:40.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4208ff9640 1 -- 172.21.15.110:0/587542602 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f41fc003920 con 0x7f420c0c2db0 2023-10-24T23:52:40.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4208ff9640 1 -- 172.21.15.110:0/587542602 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f41fc0b7d90 con 0x7f420c0c2db0 2023-10-24T23:52:40.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.297+0000 7f4211a32640 1 -- 172.21.15.110:0/587542602 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420c0c6650 con 0x7f420c0c2db0 2023-10-24T23:52:40.310 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:40.301+0000 7fc8fab5b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-24T23:52:40.310 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.301+0000 7f4208ff9640 1 -- 172.21.15.110:0/587542602 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f420c0c6650 con 0x7f420c0c2db0 2023-10-24T23:52:40.355 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:40.353+0000 7fc8fab5b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-24T23:52:40.370 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:40.365+0000 7fd74cede180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-24T23:52:40.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.369+0000 7f4211a32640 1 -- 172.21.15.110:0/587542602 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e0f462d0-c6ff-41aa-9b25-2f586762248a", "id": 5} v 0) v1 -- 0x7f420c14c220 con 0x7f420c0c2db0 2023-10-24T23:52:40.383 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.377+0000 7f4208ff9640 1 -- 172.21.15.110:0/587542602 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e0f462d0-c6ff-41aa-9b25-2f586762248a", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f41fc0bb030 con 0x7f420c0c2db0 2023-10-24T23:52:40.383 INFO:teuthology.orchestra.run.smithi110.stdout:5 2023-10-24T23:52:40.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.381+0000 7f4211a32640 1 -- 172.21.15.110:0/587542602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f420c0c2db0 msgr2=0x7f420c0c4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.381+0000 7f4211a32640 1 --2- 172.21.15.110:0/587542602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f420c0c2db0 0x7f420c0c4e90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f41fc0ae040 tx=0x7f41fc01aa30 comp rx=0 tx=0).stop 2023-10-24T23:52:40.384 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.381+0000 7f4211a32640 1 -- 172.21.15.110:0/587542602 shutdown_connections 2023-10-24T23:52:40.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.381+0000 7f4211a32640 1 --2- 172.21.15.110:0/587542602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420c154c30 0x7f420c153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.381+0000 7f4211a32640 1 --2- 172.21.15.110:0/587542602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f420c14ed10 0x7f420c1456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.381+0000 7f4211a32640 1 --2- 172.21.15.110:0/587542602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f420c0c2db0 0x7f420c0c4e90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.381+0000 7f4211a32640 1 -- 172.21.15.110:0/587542602 >> 172.21.15.110:0/587542602 conn(0x7f420c0c3960 msgr2=0x7f420c071740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:40.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.381+0000 7f4211a32640 1 -- 172.21.15.110:0/587542602 shutdown_connections 2023-10-24T23:52:40.386 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.381+0000 7f4211a32640 1 -- 172.21.15.110:0/587542602 wait complete. 2023-10-24T23:52:40.399 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new a2702df0-c858-4ffd-ac62-6a15da158e5d 6 2023-10-24T23:52:40.425 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:40.421+0000 7fd74cede180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-24T23:52:40.432 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:40.429+0000 7fc8fab5b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-24T23:52:40.483 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:40.481+0000 7fc8fab5b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-24T23:52:40.507 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:40.505+0000 7fd74cede180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-24T23:52:40.518 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b9d527640 1 Processor -- start 2023-10-24T23:52:40.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b9d527640 1 -- start start 2023-10-24T23:52:40.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b9d527640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b980c2ad0 0x7f0b981587f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b9d527640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b98154b40 con 0x7f0b9806fd80 2023-10-24T23:52:40.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b9d527640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b981547f0 con 0x7f0b980aafb0 2023-10-24T23:52:40.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b9d527640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b980ab3e0 con 0x7f0b980c2ad0 2023-10-24T23:52:40.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b96ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b980c2ad0 0x7f0b981587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b96ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b980c2ad0 0x7f0b981587f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40968/0 (socket says 172.21.15.110:40968) 2023-10-24T23:52:40.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b96ffd640 1 -- 172.21.15.110:0/315063635 learned_addr learned my addr 172.21.15.110:0/315063635 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:40.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b96ffd640 1 -- 172.21.15.110:0/315063635 >> v1:172.21.15.110:6789/0 conn(0x7f0b980aafb0 legacy=0x7f0b9813d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b96ffd640 1 -- 172.21.15.110:0/315063635 >> v1:172.21.15.107:6789/0 conn(0x7f0b9806fd80 legacy=0x7f0b98154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:40.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.513+0000 7f0b96ffd640 1 -- 172.21.15.110:0/315063635 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b981556e0 con 0x7f0b980c2ad0 2023-10-24T23:52:40.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b96ffd640 1 --2- 172.21.15.110:0/315063635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b980c2ad0 0x7f0b981587f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009870 tx=0x7f0b8c01c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7e96de69cb981f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:40.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b95ffb640 1 -- 172.21.15.110:0/315063635 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0ab600 con 0x7f0b980c2ad0 2023-10-24T23:52:40.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b95ffb640 1 -- 172.21.15.110:0/315063635 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8c0a7050 con 0x7f0b980c2ad0 2023-10-24T23:52:40.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b95ffb640 1 -- 172.21.15.110:0/315063635 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0ab980 con 0x7f0b980c2ad0 2023-10-24T23:52:40.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 -- 172.21.15.110:0/315063635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b980c2ad0 msgr2=0x7f0b981587f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 --2- 172.21.15.110:0/315063635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b980c2ad0 0x7f0b981587f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009870 tx=0x7f0b8c01c3e0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 -- 172.21.15.110:0/315063635 shutdown_connections 2023-10-24T23:52:40.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 --2- 172.21.15.110:0/315063635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b980c2ad0 0x7f0b981587f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 -- 172.21.15.110:0/315063635 >> 172.21.15.110:0/315063635 conn(0x7f0b980c3960 msgr2=0x7f0b980c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:40.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 -- 172.21.15.110:0/315063635 shutdown_connections 2023-10-24T23:52:40.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 -- 172.21.15.110:0/315063635 wait complete. 2023-10-24T23:52:40.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 Processor -- start 2023-10-24T23:52:40.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 -- start start 2023-10-24T23:52:40.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b9806fd80 0x7f0b9806bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b980aafb0 0x7f0b98071b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b980c2ad0 0x7f0b9806d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9815ea40 con 0x7f0b9806fd80 2023-10-24T23:52:40.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b981f0760 con 0x7f0b980c2ad0 2023-10-24T23:52:40.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b9d527640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b98159b60 con 0x7f0b980aafb0 2023-10-24T23:52:40.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b967fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b980aafb0 0x7f0b98071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b977fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b980c2ad0 0x7f0b9806d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b967fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b980aafb0 0x7f0b98071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54462/0 (socket says 172.21.15.110:54462) 2023-10-24T23:52:40.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b967fc640 1 -- 172.21.15.110:0/2081437158 learned_addr learned my addr 172.21.15.110:0/2081437158 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:40.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b96ffd640 1 --2- 172.21.15.110:0/2081437158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b9806fd80 0x7f0b9806bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b967fc640 1 -- 172.21.15.110:0/2081437158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b980c2ad0 msgr2=0x7f0b9806d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b967fc640 1 --2- 172.21.15.110:0/2081437158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b980c2ad0 0x7f0b9806d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b967fc640 1 -- 172.21.15.110:0/2081437158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b9806fd80 msgr2=0x7f0b9806bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b967fc640 1 --2- 172.21.15.110:0/2081437158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b9806fd80 0x7f0b9806bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b967fc640 1 -- 172.21.15.110:0/2081437158 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c0094d0 con 0x7f0b980aafb0 2023-10-24T23:52:40.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b96ffd640 1 --2- 172.21.15.110:0/2081437158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b9806fd80 0x7f0b9806bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:52:40.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.517+0000 7f0b977fe640 1 --2- 172.21.15.110:0/2081437158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b980c2ad0 0x7f0b9806d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:52:40.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.521+0000 7f0b967fc640 1 --2- 172.21.15.110:0/2081437158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b980aafb0 0x7f0b98071b70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800b440 tx=0x7f0b8800b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:40.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.521+0000 7f0b7f7fe640 1 -- 172.21.15.110:0/2081437158 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88011890 con 0x7f0b980aafb0 2023-10-24T23:52:40.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.521+0000 7f0b7f7fe640 1 -- 172.21.15.110:0/2081437158 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b880119f0 con 0x7f0b980aafb0 2023-10-24T23:52:40.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.521+0000 7f0b9d527640 1 -- 172.21.15.110:0/2081437158 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b98159e40 con 0x7f0b980aafb0 2023-10-24T23:52:40.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.521+0000 7f0b7f7fe640 1 -- 172.21.15.110:0/2081437158 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88011da0 con 0x7f0b980aafb0 2023-10-24T23:52:40.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.521+0000 7f0b9d527640 1 -- 172.21.15.110:0/2081437158 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9815ec90 con 0x7f0b980aafb0 2023-10-24T23:52:40.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.521+0000 7f0b7f7fe640 1 -- 172.21.15.110:0/2081437158 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0b88004320 con 0x7f0b980aafb0 2023-10-24T23:52:40.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.521+0000 7f0b7f7fe640 1 -- 172.21.15.110:0/2081437158 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f0b8801a9f0 con 0x7f0b980aafb0 2023-10-24T23:52:40.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.521+0000 7f0b9d527640 1 -- 172.21.15.110:0/2081437158 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b981f0530 con 0x7f0b980aafb0 2023-10-24T23:52:40.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.521+0000 7f0b7f7fe640 1 -- 172.21.15.110:0/2081437158 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f0b88038b00 con 0x7f0b980aafb0 2023-10-24T23:52:40.564 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:40.561+0000 7fd74cede180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-24T23:52:40.594 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.589+0000 7f0b9d527640 1 -- 172.21.15.110:0/2081437158 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a2702df0-c858-4ffd-ac62-6a15da158e5d", "id": 6} v 0) v1 -- 0x7f0b9815a030 con 0x7f0b980aafb0 2023-10-24T23:52:40.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.597+0000 7f0b7f7fe640 1 -- 172.21.15.110:0/2081437158 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a2702df0-c858-4ffd-ac62-6a15da158e5d", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0b88027070 con 0x7f0b980aafb0 2023-10-24T23:52:40.603 INFO:teuthology.orchestra.run.smithi110.stdout:6 2023-10-24T23:52:40.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.601+0000 7f0b9d527640 1 -- 172.21.15.110:0/2081437158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b980aafb0 msgr2=0x7f0b98071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.604 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.601+0000 7f0b9d527640 1 --2- 172.21.15.110:0/2081437158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b980aafb0 0x7f0b98071b70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800b440 tx=0x7f0b8800b900 comp rx=0 tx=0).stop 2023-10-24T23:52:40.605 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.601+0000 7f0b9d527640 1 -- 172.21.15.110:0/2081437158 shutdown_connections 2023-10-24T23:52:40.605 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.601+0000 7f0b9d527640 1 --2- 172.21.15.110:0/2081437158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b980c2ad0 0x7f0b9806d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.605 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.601+0000 7f0b9d527640 1 --2- 172.21.15.110:0/2081437158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b980aafb0 0x7f0b98071b70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.605 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.601+0000 7f0b9d527640 1 --2- 172.21.15.110:0/2081437158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b9806fd80 0x7f0b9806bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.605 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.601+0000 7f0b9d527640 1 -- 172.21.15.110:0/2081437158 >> 172.21.15.110:0/2081437158 conn(0x7f0b980c3960 msgr2=0x7f0b98154fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:40.606 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.601+0000 7f0b9d527640 1 -- 172.21.15.110:0/2081437158 shutdown_connections 2023-10-24T23:52:40.606 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.601+0000 7f0b9d527640 1 -- 172.21.15.110:0/2081437158 wait complete. 2023-10-24T23:52:40.620 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new d6cdb9b9-7f96-400e-afbd-522f3d3ff856 7 2023-10-24T23:52:40.624 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:40.621+0000 7fc8fab5b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-24T23:52:40.683 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:40.681+0000 7fc8fab5b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-24T23:52:40.718 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:40.713+0000 7fd74cede180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-24T23:52:40.777 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b4c578640 1 Processor -- start 2023-10-24T23:52:40.778 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b4c578640 1 -- start start 2023-10-24T23:52:40.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b4c578640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440727d0 0x7f7b44152c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.779 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b4c578640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b440c7110 0x7f7b441531b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b4c578640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b440c1770 con 0x7f7b440abb20 2023-10-24T23:52:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b4c578640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b44152820 con 0x7f7b440c7110 2023-10-24T23:52:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b4c578640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b44073330 con 0x7f7b440727d0 2023-10-24T23:52:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b49b13640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440727d0 0x7f7b44152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.780 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b4ab15640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b440c7110 0x7f7b441531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b49b13640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440727d0 0x7f7b44152c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54478/0 (socket says 172.21.15.110:54478) 2023-10-24T23:52:40.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b4ab15640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b440c7110 0x7f7b441531b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40980/0 (socket says 172.21.15.110:40980) 2023-10-24T23:52:40.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.773+0000 7f7b49b13640 1 -- 172.21.15.110:0/2712892618 learned_addr learned my addr 172.21.15.110:0/2712892618 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:40.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b49b13640 1 -- 172.21.15.110:0/2712892618 >> v2:172.21.15.110:3300/0 conn(0x7f7b440c7110 msgr2=0x7f7b441531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:40.781 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b49b13640 1 --2- 172.21.15.110:0/2712892618 >> v2:172.21.15.110:3300/0 conn(0x7f7b440c7110 0x7f7b441531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b49b13640 1 -- 172.21.15.110:0/2712892618 >> v1:172.21.15.107:6789/0 conn(0x7f7b440abb20 legacy=0x7f7b4414fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b49b13640 1 -- 172.21.15.110:0/2712892618 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b440c5410 con 0x7f7b440727d0 2023-10-24T23:52:40.782 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b4ab15640 1 --2- 172.21.15.110:0/2712892618 >> v2:172.21.15.110:3300/0 conn(0x7f7b440c7110 0x7f7b441531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:52:40.782 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:40.777+0000 7fd74cede180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-24T23:52:40.785 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b49b13640 1 --2- 172.21.15.110:0/2712892618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440727d0 0x7f7b44152c70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009870 tx=0x7f7b3401c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=378e8900ff4fd993 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b49312640 1 -- 172.21.15.110:0/2712892618 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b340aa670 con 0x7f7b440727d0 2023-10-24T23:52:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b49312640 1 -- 172.21.15.110:0/2712892618 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b340aa7d0 con 0x7f7b440727d0 2023-10-24T23:52:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b49312640 1 -- 172.21.15.110:0/2712892618 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b340aab60 con 0x7f7b440727d0 2023-10-24T23:52:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b4c578640 1 -- 172.21.15.110:0/2712892618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440727d0 msgr2=0x7f7b44152c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.786 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b4c578640 1 --2- 172.21.15.110:0/2712892618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440727d0 0x7f7b44152c70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009870 tx=0x7f7b3401c510 comp rx=0 tx=0).stop 2023-10-24T23:52:40.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b4c578640 1 -- 172.21.15.110:0/2712892618 shutdown_connections 2023-10-24T23:52:40.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b4c578640 1 --2- 172.21.15.110:0/2712892618 >> v2:172.21.15.110:3300/0 conn(0x7f7b440c7110 0x7f7b441531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b4c578640 1 --2- 172.21.15.110:0/2712892618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440727d0 0x7f7b44152c70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b4c578640 1 -- 172.21.15.110:0/2712892618 >> 172.21.15.110:0/2712892618 conn(0x7f7b440c6620 msgr2=0x7f7b440c6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:40.787 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b4c578640 1 -- 172.21.15.110:0/2712892618 shutdown_connections 2023-10-24T23:52:40.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.777+0000 7f7b4c578640 1 -- 172.21.15.110:0/2712892618 wait complete. 2023-10-24T23:52:40.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4c578640 1 Processor -- start 2023-10-24T23:52:40.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4c578640 1 -- start start 2023-10-24T23:52:40.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4c578640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b440727d0 0x7f7b440c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.788 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4c578640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b440abb20 0x7f7b440a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4c578640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440c7110 0x7f7b441467a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:40.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4c578640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441ede00 con 0x7f7b440abb20 2023-10-24T23:52:40.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4c578640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441ecf30 con 0x7f7b440727d0 2023-10-24T23:52:40.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4c578640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b441ed1f0 con 0x7f7b440c7110 2023-10-24T23:52:40.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4ab15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440c7110 0x7f7b441467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.789 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4a314640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b440727d0 0x7f7b440c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4a314640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b440727d0 0x7f7b440c2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40986/0 (socket says 172.21.15.110:40986) 2023-10-24T23:52:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4ab15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440c7110 0x7f7b441467a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54488/0 (socket says 172.21.15.110:54488) 2023-10-24T23:52:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4a314640 1 -- 172.21.15.110:0/414907924 learned_addr learned my addr 172.21.15.110:0/414907924 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-10-24T23:52:40.790 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b49b13640 1 --2- 172.21.15.110:0/414907924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b440abb20 0x7f7b440a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4a314640 1 -- 172.21.15.110:0/414907924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440c7110 msgr2=0x7f7b441467a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4a314640 1 --2- 172.21.15.110:0/414907924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440c7110 0x7f7b441467a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4a314640 1 -- 172.21.15.110:0/414907924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b440abb20 msgr2=0x7f7b440a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4a314640 1 --2- 172.21.15.110:0/414907924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b440abb20 0x7f7b440a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4a314640 1 -- 172.21.15.110:0/414907924 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b40009420 con 0x7f7b440727d0 2023-10-24T23:52:40.791 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4ab15640 1 --2- 172.21.15.110:0/414907924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440c7110 0x7f7b441467a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:52:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b49b13640 1 --2- 172.21.15.110:0/414907924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b440abb20 0x7f7b440a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:52:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4a314640 1 --2- 172.21.15.110:0/414907924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b440727d0 0x7f7b440c2b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00c7f0 tx=0x7f7b3c00ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b3affd640 1 -- 172.21.15.110:0/414907924 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c003b80 con 0x7f7b440727d0 2023-10-24T23:52:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b3affd640 1 -- 172.21.15.110:0/414907924 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3c003ce0 con 0x7f7b440727d0 2023-10-24T23:52:40.792 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b3affd640 1 -- 172.21.15.110:0/414907924 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c00aa90 con 0x7f7b440727d0 2023-10-24T23:52:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4c578640 1 -- 172.21.15.110:0/414907924 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b340094d0 con 0x7f7b440727d0 2023-10-24T23:52:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4c578640 1 -- 172.21.15.110:0/414907924 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b441ed7b0 con 0x7f7b440727d0 2023-10-24T23:52:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b3affd640 1 -- 172.21.15.110:0/414907924 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7b3c0050d0 con 0x7f7b440727d0 2023-10-24T23:52:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b3affd640 1 -- 172.21.15.110:0/414907924 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f7b3c012510 con 0x7f7b440727d0 2023-10-24T23:52:40.793 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.781+0000 7f7b4c578640 1 -- 172.21.15.110:0/414907924 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b44067960 con 0x7f7b440727d0 2023-10-24T23:52:40.794 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.785+0000 7f7b3affd640 1 -- 172.21.15.110:0/414907924 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f7b3c005380 con 0x7f7b440727d0 2023-10-24T23:52:40.851 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.849+0000 7f7b4c578640 1 -- 172.21.15.110:0/414907924 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d6cdb9b9-7f96-400e-afbd-522f3d3ff856", "id": 7} v 0) v1 -- 0x7f7b440c3de0 con 0x7f7b440727d0 2023-10-24T23:52:40.858 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.853+0000 7f7b3affd640 1 -- 172.21.15.110:0/414907924 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d6cdb9b9-7f96-400e-afbd-522f3d3ff856", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7b3c005520 con 0x7f7b440727d0 2023-10-24T23:52:40.858 INFO:teuthology.orchestra.run.smithi110.stdout:7 2023-10-24T23:52:40.859 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.853+0000 7f7b4c578640 1 -- 172.21.15.110:0/414907924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b440727d0 msgr2=0x7f7b440c2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:40.859 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.853+0000 7f7b4c578640 1 --2- 172.21.15.110:0/414907924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b440727d0 0x7f7b440c2b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00c7f0 tx=0x7f7b3c00ccb0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.859 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.853+0000 7f7b4c578640 1 -- 172.21.15.110:0/414907924 shutdown_connections 2023-10-24T23:52:40.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.853+0000 7f7b4c578640 1 --2- 172.21.15.110:0/414907924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b440c7110 0x7f7b441467a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.853+0000 7f7b4c578640 1 --2- 172.21.15.110:0/414907924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b440abb20 0x7f7b440a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.853+0000 7f7b4c578640 1 --2- 172.21.15.110:0/414907924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b440727d0 0x7f7b440c2b80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:40.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.853+0000 7f7b4c578640 1 -- 172.21.15.110:0/414907924 >> 172.21.15.110:0/414907924 conn(0x7f7b440c6620 msgr2=0x7f7b44151350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:40.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.857+0000 7f7b4c578640 1 -- 172.21.15.110:0/414907924 shutdown_connections 2023-10-24T23:52:40.860 INFO:teuthology.orchestra.run.smithi110.stderr:2023-10-24T23:52:40.857+0000 7f7b4c578640 1 -- 172.21.15.110:0/414907924 wait complete. 2023-10-24T23:52:40.874 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-24T23:52:40.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-24T23:52:40.877 INFO:tasks.ceph.osd.0:Started 2023-10-24T23:52:40.877 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-24T23:52:40.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-24T23:52:40.880 INFO:tasks.ceph.osd.1:Started 2023-10-24T23:52:40.881 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-24T23:52:40.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-24T23:52:40.884 INFO:tasks.ceph.osd.2:Started 2023-10-24T23:52:40.884 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-24T23:52:40.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-24T23:52:40.886 INFO:tasks.ceph.osd.3:Started 2023-10-24T23:52:40.887 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-24T23:52:40.887 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-24T23:52:40.889 INFO:tasks.ceph.osd.4:Started 2023-10-24T23:52:40.889 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-24T23:52:40.890 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-24T23:52:40.892 INFO:tasks.ceph.osd.5:Started 2023-10-24T23:52:40.892 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-24T23:52:40.892 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-24T23:52:40.895 INFO:tasks.ceph.osd.6:Started 2023-10-24T23:52:40.895 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-24T23:52:40.897 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-24T23:52:40.899 INFO:tasks.ceph.osd.7:Started 2023-10-24T23:52:40.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:52:41.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.065+0000 7febfa99c640 1 Processor -- start 2023-10-24T23:52:41.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.065+0000 7febfa99c640 1 -- start start 2023-10-24T23:52:41.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.065+0000 7febfa99c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febf414c470 0x7febf414c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:41.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febfa99c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7febf4152b30 0x7febf4152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:41.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febfa99c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf4155cd0 con 0x7febf406f490 2023-10-24T23:52:41.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febfa99c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf41526e0 con 0x7febf4152b30 2023-10-24T23:52:41.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febfa99c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf40c4380 con 0x7febf414c470 2023-10-24T23:52:41.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7febf406f490 0x7febf4071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58530/0 (socket says 172.21.15.107:58530) 2023-10-24T23:52:41.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf37fe640 1 -- 172.21.15.107:0/3642141848 learned_addr learned my addr 172.21.15.107:0/3642141848 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:52:41.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf8f39640 1 --2- 172.21.15.107:0/3642141848 >> v2:172.21.15.107:3300/0 conn(0x7febf4152b30 0x7febf4152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:41.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf3fff640 1 --2- 172.21.15.107:0/3642141848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febf414c470 0x7febf414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:41.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf2ffd640 1 -- 172.21.15.107:0/3642141848 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274336018 0 0) 0x7febf4155cd0 con 0x7febf406f490 2023-10-24T23:52:41.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf2ffd640 1 -- 172.21.15.107:0/3642141848 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc003530 con 0x7febf406f490 2023-10-24T23:52:41.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf8f39640 1 -- 172.21.15.107:0/3642141848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febf414c470 msgr2=0x7febf414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:41.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf8f39640 1 --2- 172.21.15.107:0/3642141848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febf414c470 0x7febf414c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:41.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf8f39640 1 -- 172.21.15.107:0/3642141848 >> v1:172.21.15.107:6789/0 conn(0x7febf406f490 legacy=0x7febf4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:41.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf8f39640 1 -- 172.21.15.107:0/3642141848 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf40c48b0 con 0x7febf4152b30 2023-10-24T23:52:41.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf8f39640 1 --2- 172.21.15.107:0/3642141848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febf4152b30 0x7febf4152fa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7febec009d90 tx=0x7febec01c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f55341aa84b4299 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:41.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf2ffd640 1 -- 172.21.15.107:0/3642141848 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec0a57e0 con 0x7febf4152b30 2023-10-24T23:52:41.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf2ffd640 1 -- 172.21.15.107:0/3642141848 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febec0a5940 con 0x7febf4152b30 2023-10-24T23:52:41.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.069+0000 7febf2ffd640 1 -- 172.21.15.107:0/3642141848 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec0a5cd0 con 0x7febf4152b30 2023-10-24T23:52:41.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 -- 172.21.15.107:0/3642141848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febf4152b30 msgr2=0x7febf4152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:41.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 --2- 172.21.15.107:0/3642141848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febf4152b30 0x7febf4152fa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7febec009d90 tx=0x7febec01c830 comp rx=0 tx=0).stop 2023-10-24T23:52:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 -- 172.21.15.107:0/3642141848 shutdown_connections 2023-10-24T23:52:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 --2- 172.21.15.107:0/3642141848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febf4152b30 0x7febf4152fa0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 --2- 172.21.15.107:0/3642141848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febf414c470 0x7febf414c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:41.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 -- 172.21.15.107:0/3642141848 >> 172.21.15.107:0/3642141848 conn(0x7febf4070f60 msgr2=0x7febf4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:41.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 -- 172.21.15.107:0/3642141848 shutdown_connections 2023-10-24T23:52:41.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 -- 172.21.15.107:0/3642141848 wait complete. 2023-10-24T23:52:41.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 Processor -- start 2023-10-24T23:52:41.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 -- start start 2023-10-24T23:52:41.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf406f490 0x7febf40c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:41.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febf414c470 0x7febf40ab570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:41.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febf4152b30 0x7febf406b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:41.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf41556e0 con 0x7febf4152b30 2023-10-24T23:52:41.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf406aee0 con 0x7febf406f490 2023-10-24T23:52:41.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf4154a60 con 0x7febf414c470 2023-10-24T23:52:41.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf8f39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febf4152b30 0x7febf406b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:41.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf8f39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febf4152b30 0x7febf406b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54646/0 (socket says 172.21.15.107:54646) 2023-10-24T23:52:41.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf8f39640 1 -- 172.21.15.107:0/1059162900 learned_addr learned my addr 172.21.15.107:0/1059162900 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:41.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf37fe640 1 --2- 172.21.15.107:0/1059162900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febf414c470 0x7febf40ab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:41.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf3fff640 1 --2- 172.21.15.107:0/1059162900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf406f490 0x7febf40c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:41.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf8f39640 1 -- 172.21.15.107:0/1059162900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febf414c470 msgr2=0x7febf40ab570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:41.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf8f39640 1 --2- 172.21.15.107:0/1059162900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febf414c470 0x7febf40ab570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf8f39640 1 -- 172.21.15.107:0/1059162900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf406f490 msgr2=0x7febf40c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf8f39640 1 --2- 172.21.15.107:0/1059162900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf406f490 0x7febf40c1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf8f39640 1 -- 172.21.15.107:0/1059162900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec009a70 con 0x7febf4152b30 2023-10-24T23:52:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf8f39640 1 --2- 172.21.15.107:0/1059162900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febf4152b30 0x7febf406b190 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7febec0b0040 tx=0x7febec0195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf0ff9640 1 -- 172.21.15.107:0/1059162900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec0b9650 con 0x7febf4152b30 2023-10-24T23:52:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf0ff9640 1 -- 172.21.15.107:0/1059162900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febec0b97b0 con 0x7febf4152b30 2023-10-24T23:52:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 -- 172.21.15.107:0/1059162900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf4154d40 con 0x7febf4152b30 2023-10-24T23:52:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf0ff9640 1 -- 172.21.15.107:0/1059162900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febec0b9b40 con 0x7febf4152b30 2023-10-24T23:52:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febfa99c640 1 -- 172.21.15.107:0/1059162900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf4155300 con 0x7febf4152b30 2023-10-24T23:52:41.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf0ff9640 1 -- 172.21.15.107:0/1059162900 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7febec0ad440 con 0x7febf4152b30 2023-10-24T23:52:41.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.073+0000 7febf0ff9640 1 -- 172.21.15.107:0/1059162900 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7febec0c2ad0 con 0x7febf4152b30 2023-10-24T23:52:41.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.077+0000 7febfa99c640 1 -- 172.21.15.107:0/1059162900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf4155820 con 0x7febf4152b30 2023-10-24T23:52:41.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.093+0000 7febf0ff9640 1 -- 172.21.15.107:0/1059162900 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7febec0bd020 con 0x7febf4152b30 2023-10-24T23:52:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.145+0000 7febfa99c640 1 -- 172.21.15.107:0/1059162900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febf40746e0 con 0x7febf4152b30 2023-10-24T23:52:41.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.149+0000 7febf0ff9640 1 -- 172.21.15.107:0/1059162900 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7febec0b2070 con 0x7febf4152b30 2023-10-24T23:52:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:41.153 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":10,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:52:40.855520+0000","last_up_change":"0.000000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:52:41.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.149+0000 7febfa99c640 1 -- 172.21.15.107:0/1059162900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febf4152b30 msgr2=0x7febf406b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:41.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.149+0000 7febfa99c640 1 --2- 172.21.15.107:0/1059162900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febf4152b30 0x7febf406b190 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7febec0b0040 tx=0x7febec0195d0 comp rx=0 tx=0).stop 2023-10-24T23:52:41.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.149+0000 7febfa99c640 1 -- 172.21.15.107:0/1059162900 shutdown_connections 2023-10-24T23:52:41.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.149+0000 7febfa99c640 1 --2- 172.21.15.107:0/1059162900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febf4152b30 0x7febf406b190 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:41.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.149+0000 7febfa99c640 1 --2- 172.21.15.107:0/1059162900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febf414c470 0x7febf40ab570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:41.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.149+0000 7febfa99c640 1 --2- 172.21.15.107:0/1059162900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febf406f490 0x7febf40c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:41.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.149+0000 7febfa99c640 1 -- 172.21.15.107:0/1059162900 >> 172.21.15.107:0/1059162900 conn(0x7febf4070f60 msgr2=0x7febf4071340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:41.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.149+0000 7febfa99c640 1 -- 172.21.15.107:0/1059162900 shutdown_connections 2023-10-24T23:52:41.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:41.149+0000 7febfa99c640 1 -- 172.21.15.107:0/1059162900 wait complete. 2023-10-24T23:52:41.168 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-24T23:52:41.168 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-24T23:52:41.297 INFO:tasks.ceph.mgr.y.smithi107.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-24T23:52:41.297 INFO:tasks.ceph.mgr.y.smithi107.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-24T23:52:41.297 INFO:tasks.ceph.mgr.y.smithi107.stderr: from numpy import show_config as show_numpy_config 2023-10-24T23:52:41.393 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:41.389+0000 7fc8fab5b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-24T23:52:41.415 INFO:tasks.ceph.mgr.x.smithi110.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-24T23:52:41.415 INFO:tasks.ceph.mgr.x.smithi110.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-24T23:52:41.415 INFO:tasks.ceph.mgr.x.smithi110.stderr: from numpy import show_config as show_numpy_config 2023-10-24T23:52:41.502 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:41.497+0000 7fd74cede180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-24T23:52:41.507 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-10-24T23:52:41.505+0000 7ff5fbd70640 -1 Falling back to public interface 2023-10-24T23:52:41.513 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-10-24T23:52:41.509+0000 7f384b2d6640 -1 Falling back to public interface 2023-10-24T23:52:41.513 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-10-24T23:52:41.509+0000 7f597a20e640 -1 Falling back to public interface 2023-10-24T23:52:41.517 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-10-24T23:52:41.513+0000 7fa70fec5640 -1 Falling back to public interface 2023-10-24T23:52:41.523 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-10-24T23:52:41.521+0000 7fced40f6640 -1 Falling back to public interface 2023-10-24T23:52:41.524 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-10-24T23:52:41.521+0000 7fab977d6640 -1 Falling back to public interface 2023-10-24T23:52:41.524 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-10-24T23:52:41.521+0000 7f148f382640 -1 Falling back to public interface 2023-10-24T23:52:41.525 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-10-24T23:52:41.521+0000 7f267eb0c640 -1 Falling back to public interface 2023-10-24T23:52:41.572 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:41.569+0000 7fc8fab5b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-24T23:52:41.674 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:41.669+0000 7fd74cede180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-24T23:52:41.733 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:41.729+0000 7fc8fab5b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-24T23:52:41.810 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:41.809+0000 7fc8fab5b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-24T23:52:41.848 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:41.845+0000 7fd74cede180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-24T23:52:41.917 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:41.913+0000 7fd74cede180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-24T23:52:41.946 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:41.945+0000 7fc8fab5b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-24T23:52:41.974 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-24T23:52:42.051 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:42.049+0000 7fd74cede180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-24T23:52:42.082 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96ea6eb640 1 Processor -- start 2023-10-24T23:52:42.082 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96ea6eb640 1 -- start start 2023-10-24T23:52:42.084 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96ea6eb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f96e40c2ad0 0x7f96e4158930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:42.084 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96ea6eb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e414e3a0 0x7f96e414e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:42.084 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96ea6eb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e41584a0 con 0x7f96e41551b0 2023-10-24T23:52:42.085 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96ea6eb640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e40a9a60 con 0x7f96e40c2ad0 2023-10-24T23:52:42.085 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96ea6eb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e40a9780 con 0x7f96e414e3a0 2023-10-24T23:52:42.085 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96e8c88640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e414e3a0 0x7f96e414e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:42.085 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96e3fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f96e40c2ad0 0x7f96e4158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:42.085 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96e8c88640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e414e3a0 0x7f96e414e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:44112/0 (socket says 172.21.15.107:44112) 2023-10-24T23:52:42.086 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96e3fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f96e40c2ad0 0x7f96e4158930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:56532/0 (socket says 172.21.15.107:56532) 2023-10-24T23:52:42.086 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96e8c88640 1 -- 172.21.15.107:0/2862065122 learned_addr learned my addr 172.21.15.107:0/2862065122 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:42.087 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96e3fff640 1 -- 172.21.15.107:0/2862065122 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e414e3a0 msgr2=0x7f96e414e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:42.087 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96e3fff640 1 --2- 172.21.15.107:0/2862065122 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e414e3a0 0x7f96e414e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:42.087 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96e3fff640 1 -- 172.21.15.107:0/2862065122 >> v1:172.21.15.110:6789/0 conn(0x7f96e41551b0 legacy=0x7f96e40aafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:42.088 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96e3fff640 1 -- 172.21.15.107:0/2862065122 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e4157e50 con 0x7f96e40c2ad0 2023-10-24T23:52:42.088 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.081+0000 7f96e8c88640 1 --2- 172.21.15.107:0/2862065122 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e414e3a0 0x7f96e414e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:52:42.088 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96e3fff640 1 --2- 172.21.15.107:0/2862065122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96e40c2ad0 0x7f96e4158930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f96d40097c0 tx=0x7f96d401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43152159342d7a22 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:42.088 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96e2ffd640 1 -- 172.21.15.107:0/2862065122 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d40ad660 con 0x7f96e40c2ad0 2023-10-24T23:52:42.089 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96e2ffd640 1 -- 172.21.15.107:0/2862065122 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d40ad7c0 con 0x7f96e40c2ad0 2023-10-24T23:52:42.089 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96e2ffd640 1 -- 172.21.15.107:0/2862065122 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d40adb50 con 0x7f96e40c2ad0 2023-10-24T23:52:42.092 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 -- 172.21.15.107:0/2862065122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96e40c2ad0 msgr2=0x7f96e4158930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:42.092 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 --2- 172.21.15.107:0/2862065122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96e40c2ad0 0x7f96e4158930 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f96d40097c0 tx=0x7f96d401c3c0 comp rx=0 tx=0).stop 2023-10-24T23:52:42.092 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 -- 172.21.15.107:0/2862065122 shutdown_connections 2023-10-24T23:52:42.092 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 --2- 172.21.15.107:0/2862065122 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e414e3a0 0x7f96e414e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:42.093 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 --2- 172.21.15.107:0/2862065122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96e40c2ad0 0x7f96e4158930 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:42.093 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 -- 172.21.15.107:0/2862065122 >> 172.21.15.107:0/2862065122 conn(0x7f96e40c3960 msgr2=0x7f96e40c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:42.093 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 -- 172.21.15.107:0/2862065122 shutdown_connections 2023-10-24T23:52:42.093 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 -- 172.21.15.107:0/2862065122 wait complete. 2023-10-24T23:52:42.093 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 Processor -- start 2023-10-24T23:52:42.094 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 -- start start 2023-10-24T23:52:42.094 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96e40c2ad0 0x7f96e41512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:42.094 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e414e3a0 0x7f96e414e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:42.094 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e41551b0 0x7f96e406d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:42.095 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e414bed0 con 0x7f96e414e3a0 2023-10-24T23:52:42.095 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e406df10 con 0x7f96e40c2ad0 2023-10-24T23:52:42.095 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96ea6eb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e414c180 con 0x7f96e41551b0 2023-10-24T23:52:42.095 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e414e3a0 0x7f96e414e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:42.095 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e414e3a0 0x7f96e414e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54650/0 (socket says 172.21.15.107:54650) 2023-10-24T23:52:42.096 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96e37fe640 1 -- 172.21.15.107:0/1255662069 learned_addr learned my addr 172.21.15.107:0/1255662069 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:42.096 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96e3fff640 1 --2- 172.21.15.107:0/1255662069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96e40c2ad0 0x7f96e41512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:42.096 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.085+0000 7f96e8c88640 1 --2- 172.21.15.107:0/1255662069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e41551b0 0x7f96e406d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:42.096 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96e37fe640 1 -- 172.21.15.107:0/1255662069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e41551b0 msgr2=0x7f96e406d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:42.096 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96e37fe640 1 --2- 172.21.15.107:0/1255662069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e41551b0 0x7f96e406d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:42.097 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96e37fe640 1 -- 172.21.15.107:0/1255662069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96e40c2ad0 msgr2=0x7f96e41512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:42.097 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96e37fe640 1 --2- 172.21.15.107:0/1255662069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96e40c2ad0 0x7f96e41512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:42.097 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96e37fe640 1 -- 172.21.15.107:0/1255662069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc009b30 con 0x7f96e414e3a0 2023-10-24T23:52:42.097 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96e37fe640 1 --2- 172.21.15.107:0/1255662069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e414e3a0 0x7f96e414e810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f96d800c530 tx=0x7f96d800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:42.097 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96e0ff9640 1 -- 172.21.15.107:0/1255662069 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8005220 con 0x7f96e414e3a0 2023-10-24T23:52:42.098 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96e0ff9640 1 -- 172.21.15.107:0/1255662069 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d8005380 con 0x7f96e414e3a0 2023-10-24T23:52:42.098 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96e0ff9640 1 -- 172.21.15.107:0/1255662069 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8007270 con 0x7f96e414e3a0 2023-10-24T23:52:42.098 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96ea6eb640 1 -- 172.21.15.107:0/1255662069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d4009420 con 0x7f96e414e3a0 2023-10-24T23:52:42.098 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96ea6eb640 1 -- 172.21.15.107:0/1255662069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e406dc30 con 0x7f96e414e3a0 2023-10-24T23:52:42.099 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96e0ff9640 1 -- 172.21.15.107:0/1255662069 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f96d80067e0 con 0x7f96e414e3a0 2023-10-24T23:52:42.099 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96e0ff9640 1 -- 172.21.15.107:0/1255662069 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f96d801bad0 con 0x7f96e414e3a0 2023-10-24T23:52:42.099 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.089+0000 7f96ea6eb640 1 -- 172.21.15.107:0/1255662069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e41ed1c0 con 0x7f96e414e3a0 2023-10-24T23:52:42.100 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.093+0000 7f96e0ff9640 1 -- 172.21.15.107:0/1255662069 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f96e41ed1c0 con 0x7f96e414e3a0 2023-10-24T23:52:42.151 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.149+0000 7f96ea6eb640 1 -- 172.21.15.107:0/1255662069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96e4150fc0 con 0x7f96e414e3a0 2023-10-24T23:52:42.152 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.149+0000 7f96e0ff9640 1 -- 172.21.15.107:0/1255662069 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f96e4150fc0 con 0x7f96e414e3a0 2023-10-24T23:52:42.152 INFO:teuthology.misc.health.smithi107.stdout: 2023-10-24T23:52:42.152 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":10,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:52:40.855520+0000","last_up_change":"0.000000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:52:42.153 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.149+0000 7f96ea6eb640 1 -- 172.21.15.107:0/1255662069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e414e3a0 msgr2=0x7f96e414e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:42.154 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.149+0000 7f96ea6eb640 1 --2- 172.21.15.107:0/1255662069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e414e3a0 0x7f96e414e810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f96d800c530 tx=0x7f96d800c9f0 comp rx=0 tx=0).stop 2023-10-24T23:52:42.154 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.149+0000 7f96ea6eb640 1 -- 172.21.15.107:0/1255662069 shutdown_connections 2023-10-24T23:52:42.154 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.149+0000 7f96ea6eb640 1 --2- 172.21.15.107:0/1255662069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96e41551b0 0x7f96e406d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:42.154 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.149+0000 7f96ea6eb640 1 --2- 172.21.15.107:0/1255662069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e414e3a0 0x7f96e414e810 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:42.155 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.149+0000 7f96ea6eb640 1 --2- 172.21.15.107:0/1255662069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96e40c2ad0 0x7f96e41512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:42.155 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.149+0000 7f96ea6eb640 1 -- 172.21.15.107:0/1255662069 >> 172.21.15.107:0/1255662069 conn(0x7f96e40c3960 msgr2=0x7f96e41545b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:42.155 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.149+0000 7f96ea6eb640 1 -- 172.21.15.107:0/1255662069 shutdown_connections 2023-10-24T23:52:42.155 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:42.149+0000 7f96ea6eb640 1 -- 172.21.15.107:0/1255662069 wait complete. 2023-10-24T23:52:42.167 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-24T23:52:42.570 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:42.565+0000 7fc8fab5b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-24T23:52:42.645 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:42.641+0000 7fc8fab5b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-24T23:52:42.682 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:42.677+0000 7fd74cede180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-24T23:52:42.757 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:42.753+0000 7fd74cede180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-24T23:52:42.815 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:42.813+0000 7fc8fab5b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-24T23:52:42.888 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:42.885+0000 7fc8fab5b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-24T23:52:42.925 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:42.921+0000 7fd74cede180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-24T23:52:42.997 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:42.993+0000 7fd74cede180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-24T23:52:43.176 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:43.173+0000 7fc8fab5b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-24T23:52:43.234 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:43.233+0000 7fc8fab5b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-24T23:52:43.284 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:43.281+0000 7fd74cede180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-24T23:52:43.347 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-24T23:52:43.345+0000 7fd74cede180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-24T23:52:44.301 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:44.297+0000 7fc8a241a640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-24T23:52:44.302 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-24T23:52:44.297+0000 7fc8a241a640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-24T23:52:45.382 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-10-24T23:52:45.377+0000 7ff5fbd70640 -1 osd.1 0 log_to_monitors true 2023-10-24T23:52:45.388 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-10-24T23:52:45.385+0000 7fa70fec5640 -1 osd.5 0 log_to_monitors true 2023-10-24T23:52:45.418 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-10-24T23:52:45.413+0000 7f597a20e640 -1 osd.6 0 log_to_monitors true 2023-10-24T23:52:45.452 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-10-24T23:52:45.449+0000 7fab977d6640 -1 osd.7 0 log_to_monitors true 2023-10-24T23:52:45.453 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-10-24T23:52:45.449+0000 7f267eb0c640 -1 osd.3 0 log_to_monitors true 2023-10-24T23:52:45.458 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-10-24T23:52:45.453+0000 7f148f382640 -1 osd.0 0 log_to_monitors true 2023-10-24T23:52:45.482 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-10-24T23:52:45.477+0000 7fced40f6640 -1 osd.2 0 log_to_monitors true 2023-10-24T23:52:45.503 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-10-24T23:52:45.501+0000 7f384b2d6640 -1 osd.4 0 log_to_monitors true 2023-10-24T23:52:48.929 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-10-24T23:52:48.925+0000 7fab94793640 -1 osd.7 0 waiting for initial osdmap 2023-10-24T23:52:48.967 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-10-24T23:52:48.965+0000 7fab8b59d640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-24T23:52:48.969 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-10-24T23:52:48.965+0000 7f5975990640 -1 osd.6 0 waiting for initial osdmap 2023-10-24T23:52:48.974 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-24T23:52:49.007 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-10-24T23:52:49.005+0000 7f596dfd5640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-24T23:52:49.013 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-10-24T23:52:49.009+0000 7f3848293640 -1 osd.4 0 waiting for initial osdmap 2023-10-24T23:52:49.014 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-10-24T23:52:49.009+0000 7fa70ce82640 -1 osd.5 0 waiting for initial osdmap 2023-10-24T23:52:49.035 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-10-24T23:52:49.033+0000 7f383f09d640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-24T23:52:49.035 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-10-24T23:52:49.033+0000 7fa703c8c640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-24T23:52:49.171 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdedcf640 1 Processor -- start 2023-10-24T23:52:49.173 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdedcf640 1 -- start start 2023-10-24T23:52:49.174 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdedcf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3cd0006da0 0x7f3cd0007180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:49.174 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdedcf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3cd00462e0 0x7f3cd00d39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:49.174 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdedcf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd00d72a0 con 0x7f3cd00d3f10 2023-10-24T23:52:49.174 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdedcf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd00d43a0 con 0x7f3cd00462e0 2023-10-24T23:52:49.175 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdedcf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00d7560 con 0x7f3cd0006da0 2023-10-24T23:52:49.175 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdcb6b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3cd0006da0 0x7f3cd0007180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:49.175 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdcb6b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3cd0006da0 0x7f3cd0007180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:44284/0 (socket says 172.21.15.107:44284) 2023-10-24T23:52:49.175 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdcb6b640 1 -- 172.21.15.107:0/3944977282 learned_addr learned my addr 172.21.15.107:0/3944977282 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:49.175 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdcb6b640 1 -- 172.21.15.107:0/3944977282 >> v2:172.21.15.107:3300/0 conn(0x7f3cd00462e0 msgr2=0x7f3cd00d39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:49.176 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cd7fff640 1 --2- 172.21.15.107:0/3944977282 >> v2:172.21.15.107:3300/0 conn(0x7f3cd00462e0 0x7f3cd00d39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:49.176 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdcb6b640 1 --2- 172.21.15.107:0/3944977282 >> v2:172.21.15.107:3300/0 conn(0x7f3cd00462e0 0x7f3cd00d39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:49.177 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdcb6b640 1 -- 172.21.15.107:0/3944977282 >> v1:172.21.15.107:6789/0 conn(0x7f3cd00d3f10 legacy=0x7f3cd00d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:49.177 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.172+0000 7f3cdcb6b640 1 -- 172.21.15.107:0/3944977282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd0046a60 con 0x7f3cd0006da0 2023-10-24T23:52:49.177 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.176+0000 7f3cd7fff640 1 --2- 172.21.15.107:0/3944977282 >> v2:172.21.15.107:3300/0 conn(0x7f3cd00462e0 0x7f3cd00d39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:52:49.178 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.176+0000 7f3cdcb6b640 1 --2- 172.21.15.107:0/3944977282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3cd0006da0 0x7f3cd0007180 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3cd802c850 tx=0x7f3cd80995b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7047309a6274d793 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:49.178 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.176+0000 7f3cd77fe640 1 -- 172.21.15.107:0/3944977282 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd812b630 con 0x7f3cd0006da0 2023-10-24T23:52:49.178 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.176+0000 7f3cd77fe640 1 -- 172.21.15.107:0/3944977282 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd812b790 con 0x7f3cd0006da0 2023-10-24T23:52:49.179 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.176+0000 7f3cd77fe640 1 -- 172.21.15.107:0/3944977282 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd812bc00 con 0x7f3cd0006da0 2023-10-24T23:52:49.179 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.180+0000 7f3cdedcf640 1 -- 172.21.15.107:0/3944977282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3cd0006da0 msgr2=0x7f3cd0007180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:49.180 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.180+0000 7f3cdedcf640 1 --2- 172.21.15.107:0/3944977282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3cd0006da0 0x7f3cd0007180 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3cd802c850 tx=0x7f3cd80995b0 comp rx=0 tx=0).stop 2023-10-24T23:52:49.180 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.180+0000 7f3cdedcf640 1 -- 172.21.15.107:0/3944977282 shutdown_connections 2023-10-24T23:52:49.180 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.180+0000 7f3cdedcf640 1 --2- 172.21.15.107:0/3944977282 >> v2:172.21.15.107:3300/0 conn(0x7f3cd00462e0 0x7f3cd00d39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:49.180 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.180+0000 7f3cdedcf640 1 --2- 172.21.15.107:0/3944977282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3cd0006da0 0x7f3cd0007180 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:49.181 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.180+0000 7f3cdedcf640 1 -- 172.21.15.107:0/3944977282 >> 172.21.15.107:0/3944977282 conn(0x7f3cd0006020 msgr2=0x7f3cd0006430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:49.183 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.184+0000 7f3cdedcf640 1 -- 172.21.15.107:0/3944977282 shutdown_connections 2023-10-24T23:52:49.183 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.184+0000 7f3cdedcf640 1 -- 172.21.15.107:0/3944977282 wait complete. 2023-10-24T23:52:49.184 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.184+0000 7f3cdedcf640 1 Processor -- start 2023-10-24T23:52:49.186 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.184+0000 7f3cdedcf640 1 -- start start 2023-10-24T23:52:49.186 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.184+0000 7f3cdedcf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cd0006da0 0x7f3cd00cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:49.187 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.184+0000 7f3cdedcf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd00462e0 0x7f3cd00c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:49.187 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.184+0000 7f3cd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd00462e0 0x7f3cd00c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:49.187 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd00462e0 0x7f3cd00c4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36536/0 (socket says 172.21.15.107:36536) 2023-10-24T23:52:49.187 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cdedcf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3cd00d3f10 0x7f3cd00c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:49.188 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cd7fff640 1 -- 172.21.15.107:0/1700165383 learned_addr learned my addr 172.21.15.107:0/1700165383 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:49.188 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0177710 con 0x7f3cd00462e0 2023-10-24T23:52:49.188 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd00c4c80 con 0x7f3cd0006da0 2023-10-24T23:52:49.188 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd0176b20 con 0x7f3cd00d3f10 2023-10-24T23:52:49.189 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cdcb6b640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cd0006da0 0x7f3cd00cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:49.189 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cd7fff640 1 -- 172.21.15.107:0/1700165383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3cd00d3f10 msgr2=0x7f3cd00c4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:49.190 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cd7fff640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3cd00d3f10 0x7f3cd00c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:49.190 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cd7fff640 1 -- 172.21.15.107:0/1700165383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cd0006da0 msgr2=0x7f3cd00cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:49.190 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cd7fff640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cd0006da0 0x7f3cd00cccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:49.190 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cd7fff640 1 -- 172.21.15.107:0/1700165383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd804efe0 con 0x7f3cd00462e0 2023-10-24T23:52:49.190 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.188+0000 7f3cd7fff640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd00462e0 0x7f3cd00c4200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3cc400c7c0 tx=0x7f3cc400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:49.195 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.196+0000 7f3cd57fa640 1 -- 172.21.15.107:0/1700165383 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4007a30 con 0x7f3cd00462e0 2023-10-24T23:52:49.195 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.196+0000 7f3cd57fa640 1 -- 172.21.15.107:0/1700165383 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc4007b90 con 0x7f3cd00462e0 2023-10-24T23:52:49.195 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.196+0000 7f3cd57fa640 1 -- 172.21.15.107:0/1700165383 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc400ce70 con 0x7f3cd00462e0 2023-10-24T23:52:49.195 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.196+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd00d69e0 con 0x7f3cd00462e0 2023-10-24T23:52:49.196 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.196+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd01770c0 con 0x7f3cd00462e0 2023-10-24T23:52:49.196 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.196+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd01778c0 con 0x7f3cd00462e0 2023-10-24T23:52:49.197 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.196+0000 7f3cd57fa640 1 -- 172.21.15.107:0/1700165383 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 4) v1 ==== 85472+0+0 (secure 0 0 0) 0x7f3cc40113f0 con 0x7f3cd00462e0 2023-10-24T23:52:49.198 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.196+0000 7f3cd57fa640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cbc065ed0 0x7f3cbc068390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:49.198 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.196+0000 7f3cdcb6b640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cbc065ed0 0x7f3cbc068390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:49.199 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.200+0000 7f3cd57fa640 1 -- 172.21.15.107:0/1700165383 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f3cc4082c90 con 0x7f3cd00462e0 2023-10-24T23:52:49.199 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.200+0000 7f3cdcb6b640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cbc065ed0 0x7f3cbc068390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00d44e0 tx=0x7f3cd806d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:49.203 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.204+0000 7f3cd57fa640 1 -- 172.21.15.107:0/1700165383 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3cd01778c0 con 0x7f3cd00462e0 2023-10-24T23:52:49.524 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-10-24T23:52:49.524+0000 7f267a28e640 -1 osd.3 0 waiting for initial osdmap 2023-10-24T23:52:49.567 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-10-24T23:52:49.568+0000 7f26728d3640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-24T23:52:49.572 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-10-24T23:52:49.572+0000 7f148ab04640 -1 osd.0 0 waiting for initial osdmap 2023-10-24T23:52:49.607 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-10-24T23:52:49.608+0000 7f1483149640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-24T23:52:49.643 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cd0005450 con 0x7f3cd00462e0 2023-10-24T23:52:49.644 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cd57fa640 1 -- 172.21.15.107:0/1700165383 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f3cc4051290 con 0x7f3cd00462e0 2023-10-24T23:52:49.644 INFO:teuthology.misc.health.smithi107.stdout: 2023-10-24T23:52:49.644 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":12,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:52:47.311604+0000","last_up_change":"0.000000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:52:49.646 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cbc065ed0 msgr2=0x7f3cbc068390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:49.646 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cbc065ed0 0x7f3cbc068390 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00d44e0 tx=0x7f3cd806d040 comp rx=0 tx=0).stop 2023-10-24T23:52:49.646 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd00462e0 msgr2=0x7f3cd00c4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:49.647 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd00462e0 0x7f3cd00c4200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3cc400c7c0 tx=0x7f3cc400cc80 comp rx=0 tx=0).stop 2023-10-24T23:52:49.647 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 shutdown_connections 2023-10-24T23:52:49.647 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cbc065ed0 0x7f3cbc068390 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:49.647 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3cd00d3f10 0x7f3cd00c4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:49.647 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3cd00462e0 0x7f3cd00c4200 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:49.648 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 --2- 172.21.15.107:0/1700165383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3cd0006da0 0x7f3cd00cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:49.648 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 >> 172.21.15.107:0/1700165383 conn(0x7f3cd0006020 msgr2=0x7f3cd00cfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:49.648 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 shutdown_connections 2023-10-24T23:52:49.648 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:49.644+0000 7f3cdedcf640 1 -- 172.21.15.107:0/1700165383 wait complete. 2023-10-24T23:52:49.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-24T23:52:49.739 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-10-24T23:52:49.740+0000 7ff5f74f2640 -1 osd.1 0 waiting for initial osdmap 2023-10-24T23:52:49.767 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-10-24T23:52:49.768+0000 7ff5efb37640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-24T23:52:49.815 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-10-24T23:52:49.812+0000 7fcecf878640 -1 osd.2 0 waiting for initial osdmap 2023-10-24T23:52:49.839 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-10-24T23:52:49.840+0000 7fcec7ebd640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-24T23:52:56.472 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-24T23:52:56.595 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.592+0000 7f2243c96640 1 Processor -- start 2023-10-24T23:52:56.595 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.592+0000 7f2243c96640 1 -- start start 2023-10-24T23:52:56.598 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2243c96640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f223c14fc40 0x7f223c1566d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.598 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2243c96640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c0aab80 con 0x7f223c150020 2023-10-24T23:52:56.598 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2243c96640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c0a9400 con 0x7f223c0c5980 2023-10-24T23:52:56.598 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2243c96640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f223c0ab8f0 con 0x7f223c14fc40 2023-10-24T23:52:56.599 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2241231640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f223c150020 0x7f223c0a8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42972/0 (socket says 172.21.15.107:42972) 2023-10-24T23:52:56.599 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2241231640 1 -- 172.21.15.107:0/1908436574 learned_addr learned my addr 172.21.15.107:0/1908436574 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:52:56.599 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2241a32640 1 --2- 172.21.15.107:0/1908436574 >> v2:172.21.15.110:3300/0 conn(0x7f223c14fc40 0x7f223c1566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:56.599 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257689200 0 0) 0x7f223c0aab80 con 0x7f223c150020 2023-10-24T23:52:56.599 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2224003530 con 0x7f223c150020 2023-10-24T23:52:56.600 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2960614892 0 0) 0x7f2224003530 con 0x7f223c150020 2023-10-24T23:52:56.600 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 >> v2:172.21.15.110:3300/0 conn(0x7f223c14fc40 msgr2=0x7f223c1566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:56.600 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 --2- 172.21.15.107:0/1908436574 >> v2:172.21.15.110:3300/0 conn(0x7f223c14fc40 0x7f223c1566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.600 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 >> v1:172.21.15.110:6789/0 conn(0x7f223c0c5980 legacy=0x7f223c0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:56.600 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c155c30 con 0x7f223c150020 2023-10-24T23:52:56.600 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f2230001a00 con 0x7f223c150020 2023-10-24T23:52:56.601 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 >> v1:172.21.15.107:6789/0 conn(0x7f223c150020 legacy=0x7f223c0a8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:56.601 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 --2- 172.21.15.107:0/1908436574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 0x7f2224003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.601 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 --2- 172.21.15.107:0/1908436574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22240047d0 0x7f2224004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.601 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 --2- 172.21.15.107:0/1908436574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2224005190 0x7f222400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.601 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222400c000 con 0x7f2224003980 2023-10-24T23:52:56.602 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222400c170 con 0x7f22240047d0 2023-10-24T23:52:56.602 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222400c330 con 0x7f2224005190 2023-10-24T23:52:56.602 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2241a32640 1 --2- 172.21.15.107:0/1908436574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 0x7f2224003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:56.602 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2241231640 1 --2- 172.21.15.107:0/1908436574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22240047d0 0x7f2224004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:56.602 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2241a32640 1 -- 172.21.15.107:0/1908436574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2224005190 msgr2=0x7f222400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:56.603 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2241a32640 1 --2- 172.21.15.107:0/1908436574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2224005190 0x7f222400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.603 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2241a32640 1 -- 172.21.15.107:0/1908436574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22240047d0 msgr2=0x7f2224004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:56.603 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2241a32640 1 --2- 172.21.15.107:0/1908436574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22240047d0 0x7f2224004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.603 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2241a32640 1 -- 172.21.15.107:0/1908436574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2224003530 con 0x7f2224003980 2023-10-24T23:52:56.603 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.596+0000 7f2242233640 1 -- 172.21.15.107:0/1908436574 reap_dead start 2023-10-24T23:52:56.607 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2241a32640 1 --2- 172.21.15.107:0/1908436574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 0x7f2224003da0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f222c009ae0 tx=0x7f222c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e92d36858a1f37f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:56.607 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c0a5510 con 0x7f2224003980 2023-10-24T23:52:56.607 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222c0a5670 con 0x7f2224003980 2023-10-24T23:52:56.607 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2240a30640 1 -- 172.21.15.107:0/1908436574 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c0a5a20 con 0x7f2224003980 2023-10-24T23:52:56.608 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 -- 172.21.15.107:0/1908436574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 msgr2=0x7f2224003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:56.608 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 --2- 172.21.15.107:0/1908436574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 0x7f2224003da0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f222c009ae0 tx=0x7f222c01ca30 comp rx=0 tx=0).stop 2023-10-24T23:52:56.608 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 -- 172.21.15.107:0/1908436574 shutdown_connections 2023-10-24T23:52:56.608 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 --2- 172.21.15.107:0/1908436574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 0x7f2224003da0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.609 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 -- 172.21.15.107:0/1908436574 >> 172.21.15.107:0/1908436574 conn(0x7f223c0c2860 msgr2=0x7f223c0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:56.609 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 -- 172.21.15.107:0/1908436574 shutdown_connections 2023-10-24T23:52:56.609 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 -- 172.21.15.107:0/1908436574 wait complete. 2023-10-24T23:52:56.609 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 Processor -- start 2023-10-24T23:52:56.609 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 -- start start 2023-10-24T23:52:56.609 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 0x7f223c1e2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.610 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f223c06eba0 0x7f223c0bbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.610 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f223c0bc2b0 0x7f223c06b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.610 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c06fb10 con 0x7f2224003980 2023-10-24T23:52:56.610 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c0bc730 con 0x7f223c0bc2b0 2023-10-24T23:52:56.610 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f223c1f0b30 con 0x7f223c06eba0 2023-10-24T23:52:56.611 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2241a32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 0x7f223c1e2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:56.611 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2241a32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 0x7f223c1e2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58292/0 (socket says 172.21.15.107:58292) 2023-10-24T23:52:56.611 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2241a32640 1 -- 172.21.15.107:0/2259100274 learned_addr learned my addr 172.21.15.107:0/2259100274 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:56.611 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2241231640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f223c06eba0 0x7f223c0bbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:56.611 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2242233640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f223c0bc2b0 0x7f223c06b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:56.612 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2241a32640 1 -- 172.21.15.107:0/2259100274 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f223c06eba0 msgr2=0x7f223c0bbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:56.612 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2241a32640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f223c06eba0 0x7f223c0bbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.612 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2241a32640 1 -- 172.21.15.107:0/2259100274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f223c0bc2b0 msgr2=0x7f223c06b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:56.612 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2241a32640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f223c0bc2b0 0x7f223c06b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.612 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2241a32640 1 -- 172.21.15.107:0/2259100274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c1f1060 con 0x7f2224003980 2023-10-24T23:52:56.613 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2242233640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f223c0bc2b0 0x7f223c06b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:52:56.613 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2241a32640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 0x7f223c1e2280 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f222c01c860 tx=0x7f222c0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:56.613 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f223a7fc640 1 -- 172.21.15.107:0/2259100274 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c0a5510 con 0x7f2224003980 2023-10-24T23:52:56.613 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f223a7fc640 1 -- 172.21.15.107:0/2259100274 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222c004020 con 0x7f2224003980 2023-10-24T23:52:56.613 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 -- 172.21.15.107:0/2259100274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223c1f0d80 con 0x7f2224003980 2023-10-24T23:52:56.614 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f223a7fc640 1 -- 172.21.15.107:0/2259100274 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c0043d0 con 0x7f2224003980 2023-10-24T23:52:56.614 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.600+0000 7f2243c96640 1 -- 172.21.15.107:0/2259100274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223c06cb10 con 0x7f2224003980 2023-10-24T23:52:56.614 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.604+0000 7f2243c96640 1 -- 172.21.15.107:0/2259100274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223c06af40 con 0x7f2224003980 2023-10-24T23:52:56.614 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.604+0000 7f223a7fc640 1 -- 172.21.15.107:0/2259100274 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f222c0b4070 con 0x7f2224003980 2023-10-24T23:52:56.614 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.604+0000 7f223a7fc640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2228065e60 0x7f2228068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.615 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.604+0000 7f223a7fc640 1 -- 172.21.15.107:0/2259100274 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f222c11cca0 con 0x7f2224003980 2023-10-24T23:52:56.615 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.604+0000 7f2241231640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2228065e60 0x7f2228068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:56.615 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.604+0000 7f2241231640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2228065e60 0x7f2228068320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f223c1ea9f0 tx=0x7f2230009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:56.616 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.608+0000 7f223a7fc640 1 -- 172.21.15.107:0/2259100274 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f222c0ea830 con 0x7f2224003980 2023-10-24T23:52:56.764 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.764+0000 7f2243c96640 1 -- 172.21.15.107:0/2259100274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f223c04e050 con 0x7f2224003980 2023-10-24T23:52:56.765 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.764+0000 7f223a7fc640 1 -- 172.21.15.107:0/2259100274 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12430 (secure 0 0 0) 0x7f222c0b2020 con 0x7f2224003980 2023-10-24T23:52:56.766 INFO:teuthology.misc.health.smithi107.stdout: 2023-10-24T23:52:56.766 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":16,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:52:53.356087+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:52:56.767 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.764+0000 7f2243c96640 1 -- 172.21.15.107:0/2259100274 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2228065e60 msgr2=0x7f2228068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:56.768 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.764+0000 7f2243c96640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2228065e60 0x7f2228068320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f223c1ea9f0 tx=0x7f2230009040 comp rx=0 tx=0).stop 2023-10-24T23:52:56.768 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.764+0000 7f2243c96640 1 -- 172.21.15.107:0/2259100274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 msgr2=0x7f223c1e2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:56.768 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.764+0000 7f2243c96640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 0x7f223c1e2280 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f222c01c860 tx=0x7f222c0197d0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.768 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.768+0000 7f2243c96640 1 -- 172.21.15.107:0/2259100274 shutdown_connections 2023-10-24T23:52:56.769 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.768+0000 7f2243c96640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f223c0bc2b0 0x7f223c06b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.769 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.768+0000 7f2243c96640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2228065e60 0x7f2228068320 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.769 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.768+0000 7f2243c96640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f223c06eba0 0x7f223c0bbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.769 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.768+0000 7f2243c96640 1 --2- 172.21.15.107:0/2259100274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2224003980 0x7f223c1e2280 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.770 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.768+0000 7f2243c96640 1 -- 172.21.15.107:0/2259100274 >> 172.21.15.107:0/2259100274 conn(0x7f223c0c2860 msgr2=0x7f223c1553e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:56.770 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.768+0000 7f2243c96640 1 -- 172.21.15.107:0/2259100274 shutdown_connections 2023-10-24T23:52:56.770 INFO:teuthology.misc.health.smithi107.stderr:2023-10-24T23:52:56.768+0000 7f2243c96640 1 -- 172.21.15.107:0/2259100274 wait complete. 2023-10-24T23:52:56.783 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-24T23:52:56.783 INFO:tasks.ceph:Creating RBD pool 2023-10-24T23:52:56.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-24T23:52:56.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb7fbf640 1 Processor -- start 2023-10-24T23:52:56.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb7fbf640 1 -- start start 2023-10-24T23:52:56.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb7fbf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbcb014fd10 0x7fbcb00c7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb7fbf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcb0071f70 0x7fbcb00a9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb7fbf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcb00c71a0 con 0x7fbcb0155b70 2023-10-24T23:52:56.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb7fbf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcb01569f0 con 0x7fbcb014fd10 2023-10-24T23:52:56.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb7fbf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb006d150 con 0x7fbcb0071f70 2023-10-24T23:52:56.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb5d5b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbcb014fd10 0x7fbcb00c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:56.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb5d5b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbcb014fd10 0x7fbcb00c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58298/0 (socket says 172.21.15.107:58298) 2023-10-24T23:52:56.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb5d5b640 1 -- 172.21.15.107:0/828867711 learned_addr learned my addr 172.21.15.107:0/828867711 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:56.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb555a640 1 --2- 172.21.15.107:0/828867711 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcb0071f70 0x7fbcb00a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:56.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb5d5b640 1 -- 172.21.15.107:0/828867711 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcb0071f70 msgr2=0x7fbcb00a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:56.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb5d5b640 1 --2- 172.21.15.107:0/828867711 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcb0071f70 0x7fbcb00a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb5d5b640 1 -- 172.21.15.107:0/828867711 >> v1:172.21.15.110:6789/0 conn(0x7fbcb0155b70 legacy=0x7fbcb0156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:56.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.892+0000 7fbcb5d5b640 1 -- 172.21.15.107:0/828867711 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb006d430 con 0x7fbcb014fd10 2023-10-24T23:52:56.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb5d5b640 1 --2- 172.21.15.107:0/828867711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb014fd10 0x7fbcb00c7680 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009870 tx=0x7fbcac01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b25bdc82907d98d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:56.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb4d59640 1 -- 172.21.15.107:0/828867711 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac0a89b0 con 0x7fbcb014fd10 2023-10-24T23:52:56.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb4d59640 1 -- 172.21.15.107:0/828867711 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcac0a8b10 con 0x7fbcb014fd10 2023-10-24T23:52:56.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb4d59640 1 -- 172.21.15.107:0/828867711 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac0a8ea0 con 0x7fbcb014fd10 2023-10-24T23:52:56.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/828867711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb014fd10 msgr2=0x7fbcb00c7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:56.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 --2- 172.21.15.107:0/828867711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb014fd10 0x7fbcb00c7680 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009870 tx=0x7fbcac01ecf0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/828867711 shutdown_connections 2023-10-24T23:52:56.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 --2- 172.21.15.107:0/828867711 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcb0071f70 0x7fbcb00a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 --2- 172.21.15.107:0/828867711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb014fd10 0x7fbcb00c7680 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/828867711 >> 172.21.15.107:0/828867711 conn(0x7fbcb00c8020 msgr2=0x7fbcb00c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:56.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/828867711 shutdown_connections 2023-10-24T23:52:56.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/828867711 wait complete. 2023-10-24T23:52:56.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 Processor -- start 2023-10-24T23:52:56.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 -- start start 2023-10-24T23:52:56.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb0071f70 0x7fbcb01dcae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcb014fd10 0x7fbcb0154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbcb0155b70 0x7fbcb00c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb01dd3d0 con 0x7fbcb0071f70 2023-10-24T23:52:56.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb01dd020 con 0x7fbcb0155b70 2023-10-24T23:52:56.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb01dbc80 con 0x7fbcb014fd10 2023-10-24T23:52:56.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb5d5b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb0071f70 0x7fbcb01dcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:56.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb5d5b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb0071f70 0x7fbcb01dcae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58300/0 (socket says 172.21.15.107:58300) 2023-10-24T23:52:56.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb5d5b640 1 -- 172.21.15.107:0/1203447101 learned_addr learned my addr 172.21.15.107:0/1203447101 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:56.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb5d5b640 1 -- 172.21.15.107:0/1203447101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcb014fd10 msgr2=0x7fbcb0154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:56.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb5d5b640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcb014fd10 0x7fbcb0154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb5d5b640 1 -- 172.21.15.107:0/1203447101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbcb0155b70 msgr2=0x7fbcb00c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:56.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb5d5b640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbcb0155b70 0x7fbcb00c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:56.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb5d5b640 1 -- 172.21.15.107:0/1203447101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcac0094d0 con 0x7fbcb0071f70 2023-10-24T23:52:56.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb5d5b640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb0071f70 0x7fbcb01dcae0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbcac0b3040 tx=0x7fbcac01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:56.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbca67fc640 1 -- 172.21.15.107:0/1203447101 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac0a89b0 con 0x7fbcb0071f70 2023-10-24T23:52:56.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbca67fc640 1 -- 172.21.15.107:0/1203447101 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcac0bb480 con 0x7fbcb0071f70 2023-10-24T23:52:56.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/1203447101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb01de5e0 con 0x7fbcb0071f70 2023-10-24T23:52:56.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.896+0000 7fbca67fc640 1 -- 172.21.15.107:0/1203447101 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac0bb850 con 0x7fbcb0071f70 2023-10-24T23:52:56.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.900+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/1203447101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb01dbf20 con 0x7fbcb0071f70 2023-10-24T23:52:56.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.900+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/1203447101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcb006e5b0 con 0x7fbcb0071f70 2023-10-24T23:52:56.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.900+0000 7fbca67fc640 1 -- 172.21.15.107:0/1203447101 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbcac0bb9b0 con 0x7fbcb0071f70 2023-10-24T23:52:56.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.900+0000 7fbca67fc640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc94065e60 0x7fbc94068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:56.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.900+0000 7fbca67fc640 1 -- 172.21.15.107:0/1203447101 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbcac121b80 con 0x7fbcb0071f70 2023-10-24T23:52:56.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.900+0000 7fbcb555a640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc94065e60 0x7fbc94068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:56.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.900+0000 7fbcb555a640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc94065e60 0x7fbc94068320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbcb01dfd60 tx=0x7fbca0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:56.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:56.904+0000 7fbca67fc640 1 -- 172.21.15.107:0/1203447101 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbcac0ef670 con 0x7fbcb0071f70 2023-10-24T23:52:57.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.048+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/1203447101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fbcb006e5b0 con 0x7fbcb0071f70 2023-10-24T23:52:57.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbca67fc640 1 -- 172.21.15.107:0/1203447101 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7fbcac0b50a0 con 0x7fbcb0071f70 2023-10-24T23:52:57.145 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'rbd' created 2023-10-24T23:52:57.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/1203447101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc94065e60 msgr2=0x7fbc94068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:57.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc94065e60 0x7fbc94068320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbcb01dfd60 tx=0x7fbca0008040 comp rx=0 tx=0).stop 2023-10-24T23:52:57.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/1203447101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb0071f70 msgr2=0x7fbcb01dcae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:57.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb0071f70 0x7fbcb01dcae0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbcac0b3040 tx=0x7fbcac01ba90 comp rx=0 tx=0).stop 2023-10-24T23:52:57.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/1203447101 shutdown_connections 2023-10-24T23:52:57.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbcb0155b70 0x7fbcb00c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:57.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc94065e60 0x7fbc94068320 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:57.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcb014fd10 0x7fbcb0154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:57.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 --2- 172.21.15.107:0/1203447101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb0071f70 0x7fbcb01dcae0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbcac0b3040 tx=0x7fbcac01ba90 comp rx=0 tx=0).stop 2023-10-24T23:52:57.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/1203447101 >> 172.21.15.107:0/1203447101 conn(0x7fbcb00c8020 msgr2=0x7fbcb014e340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:57.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/1203447101 shutdown_connections 2023-10-24T23:52:57.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.144+0000 7fbcb7fbf640 1 -- 172.21.15.107:0/1203447101 wait complete. 2023-10-24T23:52:57.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-24T23:52:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.288+0000 7f12f86a1640 1 Processor -- start 2023-10-24T23:52:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.288+0000 7f12f86a1640 1 -- start start 2023-10-24T23:52:57.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.288+0000 7f12f86a1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f006cc30 0x7f12f015e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:57.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.288+0000 7f12f86a1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f0156880 con 0x7f12f006aac0 2023-10-24T23:52:57.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.288+0000 7f12f86a1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f0155f50 con 0x7f12f00ab2d0 2023-10-24T23:52:57.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.288+0000 7f12f86a1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0156230 con 0x7f12f006cc30 2023-10-24T23:52:57.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.288+0000 7f12f5c3c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f12f006aac0 0x7f12f00c0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42976/0 (socket says 172.21.15.107:42976) 2023-10-24T23:52:57.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.288+0000 7f12f5c3c640 1 -- 172.21.15.107:0/1021327703 learned_addr learned my addr 172.21.15.107:0/1021327703 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:52:57.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.288+0000 7f12f6c3e640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f006cc30 0x7f12f015e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:57.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.288+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670621084 0 0) 0x7f12f0156880 con 0x7f12f006aac0 2023-10-24T23:52:57.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.288+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12dc003530 con 0x7f12f006aac0 2023-10-24T23:52:57.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 180903131 0 0) 0x7f12dc003530 con 0x7f12f006aac0 2023-10-24T23:52:57.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f006cc30 msgr2=0x7f12f015e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:57.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f006cc30 0x7f12f015e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:57.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 >> v1:172.21.15.110:6789/0 conn(0x7f12f00ab2d0 legacy=0x7f12f00a4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:57.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f0156510 con 0x7f12f006aac0 2023-10-24T23:52:57.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f12e4001680 con 0x7f12f006aac0 2023-10-24T23:52:57.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 >> v1:172.21.15.107:6789/0 conn(0x7f12f006aac0 legacy=0x7f12f00c0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12dc0039b0 0x7f12dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f006cc30 0x7f12dc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12dc004d10 0x7f12dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0156230 con 0x7f12dc004d10 2023-10-24T23:52:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc00ba40 con 0x7f12dc0039b0 2023-10-24T23:52:57.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc00bc00 con 0x7f12f006cc30 2023-10-24T23:52:57.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f6c3e640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12dc004d10 0x7f12dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:57.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f5c3c640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f006cc30 0x7f12dc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:57.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f643d640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12dc0039b0 0x7f12dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:57.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f6c3e640 1 -- 172.21.15.107:0/1021327703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f006cc30 msgr2=0x7f12dc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f6c3e640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f006cc30 0x7f12dc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f6c3e640 1 -- 172.21.15.107:0/1021327703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12dc0039b0 msgr2=0x7f12dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f6c3e640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12dc0039b0 0x7f12dc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f6c3e640 1 -- 172.21.15.107:0/1021327703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc003530 con 0x7f12dc004d10 2023-10-24T23:52:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f6c3e640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12dc004d10 0x7f12dc005190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12f00c3880 tx=0x7f12e801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab40fc53c1ea677d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:57.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e80a5c10 con 0x7f12dc004d10 2023-10-24T23:52:57.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e800f040 con 0x7f12dc004d10 2023-10-24T23:52:57.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f543b640 1 -- 172.21.15.107:0/1021327703 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e80ad440 con 0x7f12dc004d10 2023-10-24T23:52:57.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f86a1640 1 -- 172.21.15.107:0/1021327703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12dc004d10 msgr2=0x7f12dc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:57.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f86a1640 1 --2- 172.21.15.107:0/1021327703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12dc004d10 0x7f12dc005190 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f12f00c3880 tx=0x7f12e801cc00 comp rx=0 tx=0).stop 2023-10-24T23:52:57.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f6c3e640 1 -- 172.21.15.107:0/1021327703 reap_dead start 2023-10-24T23:52:57.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f86a1640 1 -- 172.21.15.107:0/1021327703 shutdown_connections 2023-10-24T23:52:57.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f86a1640 1 -- 172.21.15.107:0/1021327703 >> 172.21.15.107:0/1021327703 conn(0x7f12f00c27f0 msgr2=0x7f12f00c2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:57.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f86a1640 1 -- 172.21.15.107:0/1021327703 shutdown_connections 2023-10-24T23:52:57.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.292+0000 7f12f86a1640 1 -- 172.21.15.107:0/1021327703 wait complete. 2023-10-24T23:52:57.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f86a1640 1 Processor -- start 2023-10-24T23:52:57.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f86a1640 1 -- start start 2023-10-24T23:52:57.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f86a1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12f01e7b90 0x7f12f014b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:57.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f86a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12f0152f20 0x7f12f01e7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:57.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f86a1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f0148680 0x7f12f01e84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:57.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f86a1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f01dd4e0 con 0x7f12f0152f20 2023-10-24T23:52:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f86a1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f014c220 con 0x7f12f01e7b90 2023-10-24T23:52:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f86a1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f01dc8d0 con 0x7f12f0148680 2023-10-24T23:52:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f5c3c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12f0152f20 0x7f12f01e7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f5c3c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12f0152f20 0x7f12f01e7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58316/0 (socket says 172.21.15.107:58316) 2023-10-24T23:52:57.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f5c3c640 1 -- 172.21.15.107:0/578014594 learned_addr learned my addr 172.21.15.107:0/578014594 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:57.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f6c3e640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f0148680 0x7f12f01e84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:57.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f643d640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12f01e7b90 0x7f12f014b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:57.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f5c3c640 1 -- 172.21.15.107:0/578014594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f0148680 msgr2=0x7f12f01e84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:57.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f5c3c640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f0148680 0x7f12f01e84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:57.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f5c3c640 1 -- 172.21.15.107:0/578014594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12f01e7b90 msgr2=0x7f12f014b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:57.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f5c3c640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12f01e7b90 0x7f12f014b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:57.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f5c3c640 1 -- 172.21.15.107:0/578014594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f01dcbb0 con 0x7f12f0152f20 2023-10-24T23:52:57.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f5c3c640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12f0152f20 0x7f12f01e7f70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f12e400eaf0 tx=0x7f12e400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:57.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12e2ffd640 1 -- 172.21.15.107:0/578014594 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e400c8d0 con 0x7f12f0152f20 2023-10-24T23:52:57.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12e2ffd640 1 -- 172.21.15.107:0/578014594 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e400ca30 con 0x7f12f0152f20 2023-10-24T23:52:57.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f86a1640 1 -- 172.21.15.107:0/578014594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f01dce90 con 0x7f12f0152f20 2023-10-24T23:52:57.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12e2ffd640 1 -- 172.21.15.107:0/578014594 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e400cde0 con 0x7f12f0152f20 2023-10-24T23:52:57.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f86a1640 1 -- 172.21.15.107:0/578014594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f01dd970 con 0x7f12f0152f20 2023-10-24T23:52:57.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12f86a1640 1 -- 172.21.15.107:0/578014594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f00746e0 con 0x7f12f0152f20 2023-10-24T23:52:57.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.296+0000 7f12e2ffd640 1 -- 172.21.15.107:0/578014594 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f12e401d460 con 0x7f12f0152f20 2023-10-24T23:52:57.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.300+0000 7f12e2ffd640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f12d4065e90 0x7f12d4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:57.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.300+0000 7f12e2ffd640 1 -- 172.21.15.107:0/578014594 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f12e4083560 con 0x7f12f0152f20 2023-10-24T23:52:57.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.300+0000 7f12f643d640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f12d4065e90 0x7f12d4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:57.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.300+0000 7f12f643d640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f12d4065e90 0x7f12d4068350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f12ec005510 tx=0x7f12ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:57.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.304+0000 7f12e2ffd640 1 -- 172.21.15.107:0/578014594 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f12e4050f10 con 0x7f12f0152f20 2023-10-24T23:52:57.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:57.456+0000 7f12f86a1640 1 -- 172.21.15.107:0/578014594 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f12f00bfe20 con 0x7f12f0152f20 2023-10-24T23:52:58.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.152+0000 7f12e2ffd640 1 -- 172.21.15.107:0/578014594 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f12e4054530 con 0x7f12f0152f20 2023-10-24T23:52:58.155 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-24T23:52:58.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.156+0000 7f12f86a1640 1 -- 172.21.15.107:0/578014594 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f12d4065e90 msgr2=0x7f12d4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.156+0000 7f12f86a1640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f12d4065e90 0x7f12d4068350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f12ec005510 tx=0x7f12ec009040 comp rx=0 tx=0).stop 2023-10-24T23:52:58.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.156+0000 7f12f86a1640 1 -- 172.21.15.107:0/578014594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12f0152f20 msgr2=0x7f12f01e7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.156+0000 7f12f86a1640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12f0152f20 0x7f12f01e7f70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f12e400eaf0 tx=0x7f12e400eef0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.160+0000 7f12f86a1640 1 -- 172.21.15.107:0/578014594 shutdown_connections 2023-10-24T23:52:58.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.160+0000 7f12f86a1640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f12d4065e90 0x7f12d4068350 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.160+0000 7f12f86a1640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f12f0148680 0x7f12f01e84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.160+0000 7f12f86a1640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12f0152f20 0x7f12f01e7f70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.160+0000 7f12f86a1640 1 --2- 172.21.15.107:0/578014594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f12f01e7b90 0x7f12f014b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.160+0000 7f12f86a1640 1 -- 172.21.15.107:0/578014594 >> 172.21.15.107:0/578014594 conn(0x7f12f00c27f0 msgr2=0x7f12f014a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:58.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.160+0000 7f12f86a1640 1 -- 172.21.15.107:0/578014594 shutdown_connections 2023-10-24T23:52:58.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.160+0000 7f12f86a1640 1 -- 172.21.15.107:0/578014594 wait complete. 2023-10-24T23:52:58.180 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-24T23:52:58.181 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-24T23:52:58.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-24T23:52:58.183 INFO:tasks.ceph.mds.a:Started 2023-10-24T23:52:58.184 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-24T23:52:58.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-24T23:52:58.186 INFO:tasks.ceph.mds.c:Started 2023-10-24T23:52:58.186 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-24T23:52:58.187 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-24T23:52:58.190 INFO:tasks.ceph.mds.b:Started 2023-10-24T23:52:58.190 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-24T23:52:58.191 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-24T23:52:58.193 INFO:tasks.ceph.mds.d:Started 2023-10-24T23:52:58.193 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-24T23:52:58.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:52:58.262 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-24T23:52:58.266 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-24T23:52:58.275 INFO:tasks.ceph.mds.b.smithi110.stdout:starting mds.b at 2023-10-24T23:52:58.276 INFO:tasks.ceph.mds.d.smithi110.stdout:starting mds.d at 2023-10-24T23:52:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742c34b640 1 Processor -- start 2023-10-24T23:52:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742c34b640 1 -- start start 2023-10-24T23:52:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742c34b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74240720c0 0x7f74240c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742c34b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7424157e50 0x7f74241582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742c34b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7424154860 con 0x7f742413dc20 2023-10-24T23:52:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742c34b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7424158800 con 0x7f7424157e50 2023-10-24T23:52:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742c34b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74240c1f10 con 0x7f74240720c0 2023-10-24T23:52:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742a8e8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7424157e50 0x7f74241582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742a8e8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7424157e50 0x7f74241582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58372/0 (socket says 172.21.15.107:58372) 2023-10-24T23:52:58.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742a8e8640 1 -- 172.21.15.107:0/2067571855 learned_addr learned my addr 172.21.15.107:0/2067571855 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:58.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f74298e6640 1 --2- 172.21.15.107:0/2067571855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74240720c0 0x7f74240c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742a8e8640 1 -- 172.21.15.107:0/2067571855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74240720c0 msgr2=0x7f74240c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:58.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742a8e8640 1 --2- 172.21.15.107:0/2067571855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74240720c0 0x7f74240c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742a8e8640 1 -- 172.21.15.107:0/2067571855 >> v1:172.21.15.110:6789/0 conn(0x7f742413dc20 legacy=0x7f742406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f742a8e8640 1 -- 172.21.15.107:0/2067571855 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7424154b20 con 0x7f7424157e50 2023-10-24T23:52:58.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.308+0000 7f74298e6640 1 --2- 172.21.15.107:0/2067571855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74240720c0 0x7f74240c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:52:58.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742a8e8640 1 --2- 172.21.15.107:0/2067571855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7424157e50 0x7f74241582c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f741c009e50 tx=0x7f741c01c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf7269d0e34e550 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:58.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f74290e5640 1 -- 172.21.15.107:0/2067571855 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c0a56a0 con 0x7f7424157e50 2023-10-24T23:52:58.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f74290e5640 1 -- 172.21.15.107:0/2067571855 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c0a5800 con 0x7f7424157e50 2023-10-24T23:52:58.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f74290e5640 1 -- 172.21.15.107:0/2067571855 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c0a5b90 con 0x7f7424157e50 2023-10-24T23:52:58.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 -- 172.21.15.107:0/2067571855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7424157e50 msgr2=0x7f74241582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 --2- 172.21.15.107:0/2067571855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7424157e50 0x7f74241582c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f741c009e50 tx=0x7f741c01c8f0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 -- 172.21.15.107:0/2067571855 shutdown_connections 2023-10-24T23:52:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 --2- 172.21.15.107:0/2067571855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7424157e50 0x7f74241582c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 --2- 172.21.15.107:0/2067571855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74240720c0 0x7f74240c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 -- 172.21.15.107:0/2067571855 >> 172.21.15.107:0/2067571855 conn(0x7f74240c39a0 msgr2=0x7f74240c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 -- 172.21.15.107:0/2067571855 shutdown_connections 2023-10-24T23:52:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 -- 172.21.15.107:0/2067571855 wait complete. 2023-10-24T23:52:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 Processor -- start 2023-10-24T23:52:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 -- start start 2023-10-24T23:52:58.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240720c0 0x7f74241f04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f742413dc20 0x7f742415e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7424157e50 0x7f74241397c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.312+0000 7f742c34b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742413b260 con 0x7f74240720c0 2023-10-24T23:52:58.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742c34b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74241efa90 con 0x7f7424157e50 2023-10-24T23:52:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742c34b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74241f0050 con 0x7f742413dc20 2023-10-24T23:52:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742a0e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240720c0 0x7f74241f04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742a0e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240720c0 0x7f74241f04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58380/0 (socket says 172.21.15.107:58380) 2023-10-24T23:52:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742a0e7640 1 -- 172.21.15.107:0/2464123414 learned_addr learned my addr 172.21.15.107:0/2464123414 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:58.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f74298e6640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f742413dc20 0x7f742415e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742a8e8640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7424157e50 0x7f74241397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742a0e7640 1 -- 172.21.15.107:0/2464123414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f742413dc20 msgr2=0x7f742415e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742a0e7640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f742413dc20 0x7f742415e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742a0e7640 1 -- 172.21.15.107:0/2464123414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7424157e50 msgr2=0x7f74241397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742a0e7640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7424157e50 0x7f74241397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742a0e7640 1 -- 172.21.15.107:0/2464123414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741c009b30 con 0x7f74240720c0 2023-10-24T23:52:58.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742a0e7640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240720c0 0x7f74241f04e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f741400c3f0 tx=0x7f741400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f741affd640 1 -- 172.21.15.107:0/2464123414 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74140052d0 con 0x7f74240720c0 2023-10-24T23:52:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f741affd640 1 -- 172.21.15.107:0/2464123414 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7414005430 con 0x7f74240720c0 2023-10-24T23:52:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742c34b640 1 -- 172.21.15.107:0/2464123414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742413c220 con 0x7f74240720c0 2023-10-24T23:52:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f741affd640 1 -- 172.21.15.107:0/2464123414 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414003840 con 0x7f74240720c0 2023-10-24T23:52:58.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742c34b640 1 -- 172.21.15.107:0/2464123414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74240c6b60 con 0x7f74240720c0 2023-10-24T23:52:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.316+0000 7f742c34b640 1 -- 172.21.15.107:0/2464123414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74240746e0 con 0x7f74240720c0 2023-10-24T23:52:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.320+0000 7f741affd640 1 -- 172.21.15.107:0/2464123414 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f74140039a0 con 0x7f74240720c0 2023-10-24T23:52:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.320+0000 7f741affd640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7408065e60 0x7f7408068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.320+0000 7f741affd640 1 -- 172.21.15.107:0/2464123414 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7414082940 con 0x7f74240720c0 2023-10-24T23:52:58.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.320+0000 7f74298e6640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7408065e60 0x7f7408068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.320+0000 7f74298e6640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7408065e60 0x7f7408068320 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f74241360b0 tx=0x7f7420008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:58.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.324+0000 7f741affd640 1 -- 172.21.15.107:0/2464123414 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7414050420 con 0x7f74240720c0 2023-10-24T23:52:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.472+0000 7f742c34b640 1 -- 172.21.15.107:0/2464123414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74240c1190 con 0x7f74240720c0 2023-10-24T23:52:58.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f741affd640 1 -- 172.21.15.107:0/2464123414 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+14229 (secure 0 0 0) 0x7f7414053a40 con 0x7f74240720c0 2023-10-24T23:52:58.478 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:58.478 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":18,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:52:58.140602+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:52:58.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 -- 172.21.15.107:0/2464123414 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7408065e60 msgr2=0x7f7408068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7408065e60 0x7f7408068320 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f74241360b0 tx=0x7f7420008040 comp rx=0 tx=0).stop 2023-10-24T23:52:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 -- 172.21.15.107:0/2464123414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240720c0 msgr2=0x7f74241f04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240720c0 0x7f74241f04e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f741400c3f0 tx=0x7f741400c8b0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 -- 172.21.15.107:0/2464123414 shutdown_connections 2023-10-24T23:52:58.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7424157e50 0x7f74241397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7408065e60 0x7f7408068320 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f742413dc20 0x7f742415e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 --2- 172.21.15.107:0/2464123414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74240720c0 0x7f74241f04e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 -- 172.21.15.107:0/2464123414 >> 172.21.15.107:0/2464123414 conn(0x7f74240c39a0 msgr2=0x7f742413ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:58.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 -- 172.21.15.107:0/2464123414 shutdown_connections 2023-10-24T23:52:58.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.476+0000 7f742c34b640 1 -- 172.21.15.107:0/2464123414 wait complete. 2023-10-24T23:52:58.494 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-24T23:52:58.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-24T23:52:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb971ebb640 1 Processor -- start 2023-10-24T23:52:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb971ebb640 1 -- start start 2023-10-24T23:52:58.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb971ebb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb96c0bd930 0x7fb96c1602a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb971ebb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c0c8180 con 0x7fb96c0af9b0 2023-10-24T23:52:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb971ebb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb96c155430 con 0x7fb96c06b360 2023-10-24T23:52:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb971ebb640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb96c0c5870 con 0x7fb96c0bd930 2023-10-24T23:52:58.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb96c0af9b0 0x7fb96c0c6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42986/0 (socket says 172.21.15.107:42986) 2023-10-24T23:52:58.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96b7fe640 1 -- 172.21.15.107:0/1482065330 learned_addr learned my addr 172.21.15.107:0/1482065330 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:52:58.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892214768 0 0) 0x7fb96c0c8180 con 0x7fb96c0af9b0 2023-10-24T23:52:58.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb958003530 con 0x7fb96c0af9b0 2023-10-24T23:52:58.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96bfff640 1 --2- 172.21.15.107:0/1482065330 >> v2:172.21.15.110:3300/0 conn(0x7fb96c0bd930 0x7fb96c1602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 918608417 0 0) 0x7fb958003530 con 0x7fb96c0af9b0 2023-10-24T23:52:58.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 >> v2:172.21.15.110:3300/0 conn(0x7fb96c0bd930 msgr2=0x7fb96c1602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:58.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 --2- 172.21.15.107:0/1482065330 >> v2:172.21.15.110:3300/0 conn(0x7fb96c0bd930 0x7fb96c1602a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 >> v1:172.21.15.110:6789/0 conn(0x7fb96c06b360 legacy=0x7fb96c069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:58.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c155710 con 0x7fb96c0af9b0 2023-10-24T23:52:58.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb95c001680 con 0x7fb96c0af9b0 2023-10-24T23:52:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 >> v1:172.21.15.107:6789/0 conn(0x7fb96c0af9b0 legacy=0x7fb96c0c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 --2- 172.21.15.107:0/1482065330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb958003980 0x7fb958003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 --2- 172.21.15.107:0/1482065330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9580047d0 0x7fb958004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 --2- 172.21.15.107:0/1482065330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9580052b0 0x7fb95800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c0c5870 con 0x7fb9580052b0 2023-10-24T23:52:58.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95800c160 con 0x7fb9580047d0 2023-10-24T23:52:58.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95800c320 con 0x7fb958003980 2023-10-24T23:52:58.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96bfff640 1 --2- 172.21.15.107:0/1482065330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9580052b0 0x7fb95800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96b7fe640 1 --2- 172.21.15.107:0/1482065330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb958003980 0x7fb958003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb962ffd640 1 --2- 172.21.15.107:0/1482065330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9580047d0 0x7fb958004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96bfff640 1 -- 172.21.15.107:0/1482065330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb958003980 msgr2=0x7fb958003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:58.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.604+0000 7fb96bfff640 1 --2- 172.21.15.107:0/1482065330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb958003980 0x7fb958003d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb96bfff640 1 -- 172.21.15.107:0/1482065330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9580047d0 msgr2=0x7fb958004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:58.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb96bfff640 1 --2- 172.21.15.107:0/1482065330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9580047d0 0x7fb958004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb96bfff640 1 -- 172.21.15.107:0/1482065330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb958003530 con 0x7fb9580052b0 2023-10-24T23:52:58.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb96bfff640 1 -- 172.21.15.107:0/1482065330 reap_dead start 2023-10-24T23:52:58.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb96bfff640 1 --2- 172.21.15.107:0/1482065330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9580052b0 0x7fb95800b990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb96c08c360 tx=0x7fb96401ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0e5db92c20754f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:58.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640a5d00 con 0x7fb9580052b0 2023-10-24T23:52:58.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9640a5e60 con 0x7fb9580052b0 2023-10-24T23:52:58.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb96affd640 1 -- 172.21.15.107:0/1482065330 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640ad660 con 0x7fb9580052b0 2023-10-24T23:52:58.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb971ebb640 1 -- 172.21.15.107:0/1482065330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9580052b0 msgr2=0x7fb95800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb971ebb640 1 --2- 172.21.15.107:0/1482065330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9580052b0 0x7fb95800b990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb96c08c360 tx=0x7fb96401ce00 comp rx=0 tx=0).stop 2023-10-24T23:52:58.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb971ebb640 1 -- 172.21.15.107:0/1482065330 shutdown_connections 2023-10-24T23:52:58.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb971ebb640 1 --2- 172.21.15.107:0/1482065330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9580052b0 0x7fb95800b990 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb971ebb640 1 -- 172.21.15.107:0/1482065330 >> 172.21.15.107:0/1482065330 conn(0x7fb96c071b00 msgr2=0x7fb96c0abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:58.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb971ebb640 1 -- 172.21.15.107:0/1482065330 shutdown_connections 2023-10-24T23:52:58.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb971ebb640 1 -- 172.21.15.107:0/1482065330 wait complete. 2023-10-24T23:52:58.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb971ebb640 1 Processor -- start 2023-10-24T23:52:58.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.608+0000 7fb971ebb640 1 -- start start 2023-10-24T23:52:58.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb971ebb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9580052b0 0x7fb96c0c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb971ebb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb96c156db0 0x7fb96c145320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb971ebb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb96c157230 0x7fb96c145860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb971ebb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c156510 con 0x7fb96c156db0 2023-10-24T23:52:58.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb971ebb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c155d00 con 0x7fb96c157230 2023-10-24T23:52:58.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb971ebb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c0c32c0 con 0x7fb9580052b0 2023-10-24T23:52:58.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb962ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb96c156db0 0x7fb96c145320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb962ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb96c156db0 0x7fb96c145320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58398/0 (socket says 172.21.15.107:58398) 2023-10-24T23:52:58.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb962ffd640 1 -- 172.21.15.107:0/1358680741 learned_addr learned my addr 172.21.15.107:0/1358680741 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:58.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb96b7fe640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9580052b0 0x7fb96c0c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb96bfff640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb96c157230 0x7fb96c145860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb962ffd640 1 -- 172.21.15.107:0/1358680741 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9580052b0 msgr2=0x7fb96c0c3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb962ffd640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9580052b0 0x7fb96c0c3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb962ffd640 1 -- 172.21.15.107:0/1358680741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb96c157230 msgr2=0x7fb96c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb962ffd640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb96c157230 0x7fb96c145860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb962ffd640 1 -- 172.21.15.107:0/1358680741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96c145da0 con 0x7fb96c156db0 2023-10-24T23:52:58.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb962ffd640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb96c156db0 0x7fb96c145320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb95400d450 tx=0x7fb95400d910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:58.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb968ff9640 1 -- 172.21.15.107:0/1358680741 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb954012800 con 0x7fb96c156db0 2023-10-24T23:52:58.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb968ff9640 1 -- 172.21.15.107:0/1358680741 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb954012980 con 0x7fb96c156db0 2023-10-24T23:52:58.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb968ff9640 1 -- 172.21.15.107:0/1358680741 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb954012d30 con 0x7fb96c156db0 2023-10-24T23:52:58.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb971ebb640 1 -- 172.21.15.107:0/1358680741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96c155f50 con 0x7fb96c156db0 2023-10-24T23:52:58.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb971ebb640 1 -- 172.21.15.107:0/1358680741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96c0c2f50 con 0x7fb96c156db0 2023-10-24T23:52:58.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb971ebb640 1 -- 172.21.15.107:0/1358680741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96c0c2db0 con 0x7fb96c156db0 2023-10-24T23:52:58.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb968ff9640 1 -- 172.21.15.107:0/1358680741 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb954023350 con 0x7fb96c156db0 2023-10-24T23:52:58.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb968ff9640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb94c065e60 0x7fb94c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb968ff9640 1 -- 172.21.15.107:0/1358680741 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fb9540834a0 con 0x7fb96c156db0 2023-10-24T23:52:58.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.612+0000 7fb96b7fe640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb94c065e60 0x7fb94c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.616+0000 7fb96b7fe640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb94c065e60 0x7fb94c068320 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0017b0 tx=0x7fb95c001480 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:58.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.616+0000 7fb968ff9640 1 -- 172.21.15.107:0/1358680741 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb96c0c2db0 con 0x7fb96c156db0 2023-10-24T23:52:58.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.756+0000 7fb971ebb640 1 -- 172.21.15.107:0/1358680741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb96c0c2db0 con 0x7fb96c156db0 2023-10-24T23:52:58.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb968ff9640 1 -- 172.21.15.107:0/1358680741 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb96c0c2db0 con 0x7fb96c156db0 2023-10-24T23:52:58.760 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-24T23:52:58.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 -- 172.21.15.107:0/1358680741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb94c065e60 msgr2=0x7fb94c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb94c065e60 0x7fb94c068320 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0017b0 tx=0x7fb95c001480 comp rx=0 tx=0).stop 2023-10-24T23:52:58.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 -- 172.21.15.107:0/1358680741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb96c156db0 msgr2=0x7fb96c145320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb96c156db0 0x7fb96c145320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb95400d450 tx=0x7fb95400d910 comp rx=0 tx=0).stop 2023-10-24T23:52:58.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 -- 172.21.15.107:0/1358680741 shutdown_connections 2023-10-24T23:52:58.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb96c157230 0x7fb96c145860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb96c156db0 0x7fb96c145320 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb94c065e60 0x7fb94c068320 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 --2- 172.21.15.107:0/1358680741 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9580052b0 0x7fb96c0c3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 -- 172.21.15.107:0/1358680741 >> 172.21.15.107:0/1358680741 conn(0x7fb96c071b00 msgr2=0x7fb96c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 -- 172.21.15.107:0/1358680741 shutdown_connections 2023-10-24T23:52:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.760+0000 7fb971ebb640 1 -- 172.21.15.107:0/1358680741 wait complete. 2023-10-24T23:52:58.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-24T23:52:58.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e3ea9640 1 Processor -- start 2023-10-24T23:52:58.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e3ea9640 1 -- start start 2023-10-24T23:52:58.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e3ea9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc0ab1a0 0x7f47dc0c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e3ea9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f47dc072720 0x7f47dc0bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e3ea9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc0c1ce0 con 0x7f47dc0bceb0 2023-10-24T23:52:58.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e3ea9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47dc0be1d0 con 0x7f47dc072720 2023-10-24T23:52:58.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e3ea9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc0c0880 con 0x7f47dc0ab1a0 2023-10-24T23:52:58.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e2446640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f47dc0bceb0 0x7f47dc0c4cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42988/0 (socket says 172.21.15.107:42988) 2023-10-24T23:52:58.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e2446640 1 -- 172.21.15.107:0/2294616262 learned_addr learned my addr 172.21.15.107:0/2294616262 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:52:58.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e1c45640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc0ab1a0 0x7f47dc0c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178931539 0 0) 0x7f47dc0c1ce0 con 0x7f47dc0bceb0 2023-10-24T23:52:58.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c8003530 con 0x7f47dc0bceb0 2023-10-24T23:52:58.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e1444640 1 --2- 172.21.15.107:0/2294616262 >> v2:172.21.15.110:3300/0 conn(0x7f47dc072720 0x7f47dc0bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3444061448 0 0) 0x7f47c8003530 con 0x7f47dc0bceb0 2023-10-24T23:52:58.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc0ab1a0 msgr2=0x7f47dc0c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:58.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc0ab1a0 0x7f47dc0c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 >> v2:172.21.15.110:3300/0 conn(0x7f47dc072720 msgr2=0x7f47dc0bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:58.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 --2- 172.21.15.107:0/2294616262 >> v2:172.21.15.110:3300/0 conn(0x7f47dc072720 0x7f47dc0bfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc0c1f30 con 0x7f47dc0bceb0 2023-10-24T23:52:58.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f47d4001a00 con 0x7f47dc0bceb0 2023-10-24T23:52:58.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 >> v1:172.21.15.107:6789/0 conn(0x7f47dc0bceb0 legacy=0x7f47dc0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 0x7f47c8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47c80047d0 0x7f47c8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc0ab1a0 0x7f47c8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc0c0880 con 0x7f47c8003980 2023-10-24T23:52:58.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c800b990 con 0x7f47c80047d0 2023-10-24T23:52:58.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c800bb50 con 0x7f47dc0ab1a0 2023-10-24T23:52:58.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e1c45640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 0x7f47c8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e1444640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47c80047d0 0x7f47c8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e2446640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc0ab1a0 0x7f47c8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e1c45640 1 -- 172.21.15.107:0/2294616262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc0ab1a0 msgr2=0x7f47c8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:58.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e1c45640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc0ab1a0 0x7f47c8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e1c45640 1 -- 172.21.15.107:0/2294616262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47c80047d0 msgr2=0x7f47c8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:58.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e1c45640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47c80047d0 0x7f47c8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.900+0000 7f47e1c45640 1 -- 172.21.15.107:0/2294616262 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c8003530 con 0x7f47c8003980 2023-10-24T23:52:58.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.904+0000 7f47e1c45640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 0x7f47c8003da0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f47d8009ae0 tx=0x7f47d801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790b6b95555c5e1f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:58.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.904+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d80a6510 con 0x7f47c8003980 2023-10-24T23:52:58.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.904+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d80a6670 con 0x7f47c8003980 2023-10-24T23:52:58.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.904+0000 7f47e0c43640 1 -- 172.21.15.107:0/2294616262 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d80a6a20 con 0x7f47c8003980 2023-10-24T23:52:58.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.904+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2294616262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 msgr2=0x7f47c8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.904+0000 7f47e3ea9640 1 --2- 172.21.15.107:0/2294616262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 0x7f47c8003da0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f47d8009ae0 tx=0x7f47d801ca20 comp rx=0 tx=0).stop 2023-10-24T23:52:58.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.904+0000 7f47e2446640 1 -- 172.21.15.107:0/2294616262 reap_dead start 2023-10-24T23:52:58.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.904+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2294616262 shutdown_connections 2023-10-24T23:52:58.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.904+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2294616262 >> 172.21.15.107:0/2294616262 conn(0x7f47dc0c2860 msgr2=0x7f47dc0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:58.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.904+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2294616262 shutdown_connections 2023-10-24T23:52:58.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.904+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2294616262 wait complete. 2023-10-24T23:52:58.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e3ea9640 1 Processor -- start 2023-10-24T23:52:58.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e3ea9640 1 -- start start 2023-10-24T23:52:58.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e3ea9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 0x7f47dc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e3ea9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47dc0b9670 0x7f47dc1512c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e3ea9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc1ee280 0x7f47dc14fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e3ea9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc14ea10 con 0x7f47c8003980 2023-10-24T23:52:58.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e3ea9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc14ecf0 con 0x7f47dc0b9670 2023-10-24T23:52:58.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e3ea9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc14efd0 con 0x7f47dc1ee280 2023-10-24T23:52:58.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e1c45640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 0x7f47dc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e1c45640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 0x7f47dc06a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58412/0 (socket says 172.21.15.107:58412) 2023-10-24T23:52:58.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e1c45640 1 -- 172.21.15.107:0/2972113660 learned_addr learned my addr 172.21.15.107:0/2972113660 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:58.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e1444640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47dc0b9670 0x7f47dc1512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e2446640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc1ee280 0x7f47dc14fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e1c45640 1 -- 172.21.15.107:0/2972113660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc1ee280 msgr2=0x7f47dc14fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e1c45640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc1ee280 0x7f47dc14fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e1c45640 1 -- 172.21.15.107:0/2972113660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47dc0b9670 msgr2=0x7f47dc1512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:58.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e1c45640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47dc0b9670 0x7f47dc1512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:58.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e1c45640 1 -- 172.21.15.107:0/2972113660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc14f2b0 con 0x7f47c8003980 2023-10-24T23:52:58.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e1c45640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 0x7f47dc06a840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f47d801c850 tx=0x7f47c8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:58.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47d27fc640 1 -- 172.21.15.107:0/2972113660 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d800f0c0 con 0x7f47c8003980 2023-10-24T23:52:58.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47d27fc640 1 -- 172.21.15.107:0/2972113660 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d80a67e0 con 0x7f47c8003980 2023-10-24T23:52:58.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47d27fc640 1 -- 172.21.15.107:0/2972113660 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d80a6b90 con 0x7f47c8003980 2023-10-24T23:52:58.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2972113660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc14f590 con 0x7f47c8003980 2023-10-24T23:52:58.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.908+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2972113660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc14fff0 con 0x7f47c8003980 2023-10-24T23:52:58.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.912+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2972113660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc06a620 con 0x7f47c8003980 2023-10-24T23:52:58.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.912+0000 7f47d27fc640 1 -- 172.21.15.107:0/2972113660 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f47d80a6cf0 con 0x7f47c8003980 2023-10-24T23:52:58.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.912+0000 7f47d27fc640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c0065e90 0x7f47c0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:58.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.912+0000 7f47d27fc640 1 -- 172.21.15.107:0/2972113660 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f47d80aa070 con 0x7f47c8003980 2023-10-24T23:52:58.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.912+0000 7f47e1444640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c0065e90 0x7f47c0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:58.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.912+0000 7f47e1444640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c0065e90 0x7f47c0068350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f47dc06d430 tx=0x7f47cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:58.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:58.916+0000 7f47d27fc640 1 -- 172.21.15.107:0/2972113660 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f47d80ebb80 con 0x7f47c8003980 2023-10-24T23:52:59.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.068+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2972113660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f47dc04e130 con 0x7f47c8003980 2023-10-24T23:52:59.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.068+0000 7f47d27fc640 1 -- 172.21.15.107:0/2972113660 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) v1 ==== 92+0+10 (secure 0 0 0) 0x7f47d80ef1a0 con 0x7f47c8003980 2023-10-24T23:52:59.071 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-24T23:52:59.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2972113660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c0065e90 msgr2=0x7f47c0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c0065e90 0x7f47c0068350 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f47dc06d430 tx=0x7f47cc008040 comp rx=0 tx=0).stop 2023-10-24T23:52:59.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2972113660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 msgr2=0x7f47dc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 0x7f47dc06a840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f47d801c850 tx=0x7f47c8005800 comp rx=0 tx=0).stop 2023-10-24T23:52:59.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2972113660 shutdown_connections 2023-10-24T23:52:59.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c0065e90 0x7f47c0068350 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47dc1ee280 0x7f47dc14fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47dc0b9670 0x7f47dc1512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 --2- 172.21.15.107:0/2972113660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47c8003980 0x7f47dc06a840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2972113660 >> 172.21.15.107:0/2972113660 conn(0x7f47dc0c2860 msgr2=0x7f47dc153210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2972113660 shutdown_connections 2023-10-24T23:52:59.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.072+0000 7f47e3ea9640 1 -- 172.21.15.107:0/2972113660 wait complete. 2023-10-24T23:52:59.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:52:59.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccbb6d1640 1 Processor -- start 2023-10-24T23:52:59.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccbb6d1640 1 -- start start 2023-10-24T23:52:59.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccbb6d1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fccb4156770 0x7fccb40ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccbb6d1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb40c1ca0 con 0x7fccb40c70e0 2023-10-24T23:52:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccbb6d1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb40c10c0 con 0x7fccb40be6b0 2023-10-24T23:52:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccbb6d1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb40c27e0 con 0x7fccb4156770 2023-10-24T23:52:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb8c6c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fccb40c70e0 0x7fccb40bdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43002/0 (socket says 172.21.15.107:43002) 2023-10-24T23:52:59.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb8c6c640 1 -- 172.21.15.107:0/3069908224 learned_addr learned my addr 172.21.15.107:0/3069908224 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:52:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb946d640 1 --2- 172.21.15.107:0/3069908224 >> v2:172.21.15.110:3300/0 conn(0x7fccb4156770 0x7fccb40ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662513660 0 0) 0x7fccb40c1ca0 con 0x7fccb40c70e0 2023-10-24T23:52:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc9c003530 con 0x7fccb40c70e0 2023-10-24T23:52:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2351500448 0 0) 0x7fcc9c003530 con 0x7fccb40c70e0 2023-10-24T23:52:59.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 >> v2:172.21.15.110:3300/0 conn(0x7fccb4156770 msgr2=0x7fccb40ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 --2- 172.21.15.107:0/3069908224 >> v2:172.21.15.110:3300/0 conn(0x7fccb4156770 0x7fccb40ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 >> v1:172.21.15.110:6789/0 conn(0x7fccb40be6b0 legacy=0x7fccb415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40c13a0 con 0x7fccb40c70e0 2023-10-24T23:52:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fcca8001680 con 0x7fccb40c70e0 2023-10-24T23:52:59.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 >> v1:172.21.15.107:6789/0 conn(0x7fccb40c70e0 legacy=0x7fccb40bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc9c003980 0x7fcc9c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc9c0047d0 0x7fcc9c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc9c0052b0 0x7fcc9c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c00c150 con 0x7fcc9c0052b0 2023-10-24T23:52:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c00c2c0 con 0x7fcc9c0047d0 2023-10-24T23:52:59.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c00c480 con 0x7fcc9c003980 2023-10-24T23:52:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb9c6e640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc9c0052b0 0x7fcc9c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb946d640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc9c003980 0x7fcc9c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb8c6c640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc9c0047d0 0x7fcc9c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb9c6e640 1 -- 172.21.15.107:0/3069908224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc9c003980 msgr2=0x7fcc9c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:59.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb9c6e640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc9c003980 0x7fcc9c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb9c6e640 1 -- 172.21.15.107:0/3069908224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc9c0047d0 msgr2=0x7fcc9c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb9c6e640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc9c0047d0 0x7fcc9c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb9c6e640 1 -- 172.21.15.107:0/3069908224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c003530 con 0x7fcc9c0052b0 2023-10-24T23:52:59.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.204+0000 7fccb8c6c640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc9c0047d0 0x7fcc9c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:52:59.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccb9c6e640 1 -- 172.21.15.107:0/3069908224 reap_dead start 2023-10-24T23:52:59.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccb9c6e640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc9c0052b0 0x7fcc9c00b990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fccb40beb00 tx=0x7fccac01c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e02bfdc150b9f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:59.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac003b80 con 0x7fcc9c0052b0 2023-10-24T23:52:59.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccac003ce0 con 0x7fcc9c0052b0 2023-10-24T23:52:59.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccb3fff640 1 -- 172.21.15.107:0/3069908224 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac005310 con 0x7fcc9c0052b0 2023-10-24T23:52:59.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3069908224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc9c0052b0 msgr2=0x7fcc9c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc9c0052b0 0x7fcc9c00b990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fccb40beb00 tx=0x7fccac01c6d0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3069908224 shutdown_connections 2023-10-24T23:52:59.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 --2- 172.21.15.107:0/3069908224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc9c0052b0 0x7fcc9c00b990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3069908224 >> 172.21.15.107:0/3069908224 conn(0x7fccb40c65c0 msgr2=0x7fccb40c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:59.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3069908224 shutdown_connections 2023-10-24T23:52:59.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3069908224 wait complete. 2023-10-24T23:52:59.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 Processor -- start 2023-10-24T23:52:59.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 -- start start 2023-10-24T23:52:59.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc9c0052b0 0x7fccb41edf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb40bbeb0 0x7fccb41ed4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccb40a7d60 0x7fccb41ec1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb41eed00 con 0x7fccb40bbeb0 2023-10-24T23:52:59.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb414d3f0 con 0x7fcc9c0052b0 2023-10-24T23:52:59.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.208+0000 7fccbb6d1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb41eea20 con 0x7fccb40a7d60 2023-10-24T23:52:59.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb8c6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb40bbeb0 0x7fccb41ed4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb8c6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb40bbeb0 0x7fccb41ed4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58438/0 (socket says 172.21.15.107:58438) 2023-10-24T23:52:59.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb8c6c640 1 -- 172.21.15.107:0/3315380380 learned_addr learned my addr 172.21.15.107:0/3315380380 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:59.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb9c6e640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccb40a7d60 0x7fccb41ec1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb946d640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc9c0052b0 0x7fccb41edf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb8c6c640 1 -- 172.21.15.107:0/3315380380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccb40a7d60 msgr2=0x7fccb41ec1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb8c6c640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccb40a7d60 0x7fccb41ec1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb8c6c640 1 -- 172.21.15.107:0/3315380380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc9c0052b0 msgr2=0x7fccb41edf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb8c6c640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc9c0052b0 0x7fccb41edf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb8c6c640 1 -- 172.21.15.107:0/3315380380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb414cda0 con 0x7fccb40bbeb0 2023-10-24T23:52:59.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb8c6c640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb40bbeb0 0x7fccb41ed4c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcca8001ec0 tx=0x7fcca800dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:59.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb1ffb640 1 -- 172.21.15.107:0/3315380380 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca80074e0 con 0x7fccb40bbeb0 2023-10-24T23:52:59.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb1ffb640 1 -- 172.21.15.107:0/3315380380 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca8007640 con 0x7fccb40bbeb0 2023-10-24T23:52:59.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3315380380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb414d080 con 0x7fccb40bbeb0 2023-10-24T23:52:59.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb1ffb640 1 -- 172.21.15.107:0/3315380380 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8004a10 con 0x7fccb40bbeb0 2023-10-24T23:52:59.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3315380380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb4153dc0 con 0x7fccb40bbeb0 2023-10-24T23:52:59.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3315380380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb41ebee0 con 0x7fccb40bbeb0 2023-10-24T23:52:59.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb1ffb640 1 -- 172.21.15.107:0/3315380380 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcca8004b70 con 0x7fccb40bbeb0 2023-10-24T23:52:59.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb1ffb640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcca0065e90 0x7fcca0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb1ffb640 1 -- 172.21.15.107:0/3315380380 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fcca8083040 con 0x7fccb40bbeb0 2023-10-24T23:52:59.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb946d640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcca0065e90 0x7fcca0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.212+0000 7fccb946d640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcca0065e90 0x7fcca0068350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcca40099b0 tx=0x7fcca4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:59.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.216+0000 7fccb1ffb640 1 -- 172.21.15.107:0/3315380380 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fccb41ebee0 con 0x7fccb40bbeb0 2023-10-24T23:52:59.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.408+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3315380380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccb41ebee0 con 0x7fccb40bbeb0 2023-10-24T23:52:59.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.408+0000 7fccb1ffb640 1 -- 172.21.15.107:0/3315380380 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3533 (secure 0 0 0) 0x7fccb41ebee0 con 0x7fccb40bbeb0 2023-10-24T23:52:59.410 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:59.410 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4299,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-10-24T23:52:59.411 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 3 2023-10-24T23:52:59.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3315380380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcca0065e90 msgr2=0x7fcca0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcca0065e90 0x7fcca0068350 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcca40099b0 tx=0x7fcca4008040 comp rx=0 tx=0).stop 2023-10-24T23:52:59.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3315380380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb40bbeb0 msgr2=0x7fccb41ed4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb40bbeb0 0x7fccb41ed4c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcca8001ec0 tx=0x7fcca800dc70 comp rx=0 tx=0).stop 2023-10-24T23:52:59.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3315380380 shutdown_connections 2023-10-24T23:52:59.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcca0065e90 0x7fcca0068350 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccb40a7d60 0x7fccb41ec1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb40bbeb0 0x7fccb41ed4c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 --2- 172.21.15.107:0/3315380380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc9c0052b0 0x7fccb41edf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3315380380 >> 172.21.15.107:0/3315380380 conn(0x7fccb40c65c0 msgr2=0x7fccb40c2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:59.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3315380380 shutdown_connections 2023-10-24T23:52:59.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.412+0000 7fccbb6d1640 1 -- 172.21.15.107:0/3315380380 wait complete. 2023-10-24T23:52:59.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:52:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.540+0000 7f495ede5640 1 Processor -- start 2023-10-24T23:52:59.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.540+0000 7f495ede5640 1 -- start start 2023-10-24T23:52:59.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f495814fd40 0x7f49580ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f495806ee40 0x7f49580aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495806f730 con 0x7f49580a9820 2023-10-24T23:52:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49580aa750 con 0x7f495806ee40 2023-10-24T23:52:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4958156a40 con 0x7f495814fd40 2023-10-24T23:52:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495d382640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f495806ee40 0x7f49580aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495d382640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f495806ee40 0x7f49580aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58450/0 (socket says 172.21.15.107:58450) 2023-10-24T23:52:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495d382640 1 -- 172.21.15.107:0/2126669612 learned_addr learned my addr 172.21.15.107:0/2126669612 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495cb81640 1 --2- 172.21.15.107:0/2126669612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f495814fd40 0x7f49580ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495d382640 1 -- 172.21.15.107:0/2126669612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f495814fd40 msgr2=0x7f49580ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:59.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495d382640 1 --2- 172.21.15.107:0/2126669612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f495814fd40 0x7f49580ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495d382640 1 -- 172.21.15.107:0/2126669612 >> v1:172.21.15.110:6789/0 conn(0x7f49580a9820 legacy=0x7f49580c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:59.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495d382640 1 -- 172.21.15.107:0/2126669612 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4958156710 con 0x7f495806ee40 2023-10-24T23:52:59.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495d382640 1 --2- 172.21.15.107:0/2126669612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f495806ee40 0x7f49580aa210 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4950009e50 tx=0x7f495001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f392bb727f1650 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:59.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f49577fe640 1 -- 172.21.15.107:0/2126669612 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49500a56a0 con 0x7f495806ee40 2023-10-24T23:52:59.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f49577fe640 1 -- 172.21.15.107:0/2126669612 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49500a5800 con 0x7f495806ee40 2023-10-24T23:52:59.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f49577fe640 1 -- 172.21.15.107:0/2126669612 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49500a5b90 con 0x7f495806ee40 2023-10-24T23:52:59.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 -- 172.21.15.107:0/2126669612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f495806ee40 msgr2=0x7f49580aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 --2- 172.21.15.107:0/2126669612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f495806ee40 0x7f49580aa210 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4950009e50 tx=0x7f495001c880 comp rx=0 tx=0).stop 2023-10-24T23:52:59.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 -- 172.21.15.107:0/2126669612 shutdown_connections 2023-10-24T23:52:59.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 --2- 172.21.15.107:0/2126669612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f495806ee40 0x7f49580aa210 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 --2- 172.21.15.107:0/2126669612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f495814fd40 0x7f49580ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 -- 172.21.15.107:0/2126669612 >> 172.21.15.107:0/2126669612 conn(0x7f49580c2950 msgr2=0x7f49580c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:59.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 -- 172.21.15.107:0/2126669612 shutdown_connections 2023-10-24T23:52:59.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.544+0000 7f495ede5640 1 -- 172.21.15.107:0/2126669612 wait complete. 2023-10-24T23:52:59.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495ede5640 1 Processor -- start 2023-10-24T23:52:59.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495ede5640 1 -- start start 2023-10-24T23:52:59.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495ede5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495806ee40 0x7f49581ee860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495ede5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f495814fd40 0x7f4958151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495ede5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49581446e0 0x7f4958144b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495ede5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495814f2d0 con 0x7f49581446e0 2023-10-24T23:52:59.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495ede5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49581547f0 con 0x7f495806ee40 2023-10-24T23:52:59.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495ede5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4958154230 con 0x7f495814fd40 2023-10-24T23:52:59.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495d382640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49581446e0 0x7f4958144b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495d382640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49581446e0 0x7f4958144b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58458/0 (socket says 172.21.15.107:58458) 2023-10-24T23:52:59.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495d382640 1 -- 172.21.15.107:0/189856982 learned_addr learned my addr 172.21.15.107:0/189856982 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:59.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f4957fff640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f495814fd40 0x7f4958151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495cb81640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495806ee40 0x7f49581ee860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495d382640 1 -- 172.21.15.107:0/189856982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f495814fd40 msgr2=0x7f4958151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495d382640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f495814fd40 0x7f4958151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495d382640 1 -- 172.21.15.107:0/189856982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495806ee40 msgr2=0x7f49581ee860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495d382640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495806ee40 0x7f49581ee860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495d382640 1 -- 172.21.15.107:0/189856982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4950009b30 con 0x7f49581446e0 2023-10-24T23:52:59.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495d382640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49581446e0 0x7f4958144b90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f49500af040 tx=0x7f4950019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:59.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f49557fa640 1 -- 172.21.15.107:0/189856982 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49500b8670 con 0x7f49581446e0 2023-10-24T23:52:59.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f49557fa640 1 -- 172.21.15.107:0/189856982 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49500b87d0 con 0x7f49581446e0 2023-10-24T23:52:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495ede5640 1 -- 172.21.15.107:0/189856982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4958154510 con 0x7f49581446e0 2023-10-24T23:52:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f49557fa640 1 -- 172.21.15.107:0/189856982 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49500b8b80 con 0x7f49581446e0 2023-10-24T23:52:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495ede5640 1 -- 172.21.15.107:0/189856982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49581519a0 con 0x7f49581446e0 2023-10-24T23:52:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.548+0000 7f495ede5640 1 -- 172.21.15.107:0/189856982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49580c24a0 con 0x7f49581446e0 2023-10-24T23:52:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.552+0000 7f49557fa640 1 -- 172.21.15.107:0/189856982 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f49500b8ce0 con 0x7f49581446e0 2023-10-24T23:52:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.552+0000 7f49557fa640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f493c065e60 0x7f493c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.552+0000 7f49557fa640 1 -- 172.21.15.107:0/189856982 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f495011e9e0 con 0x7f49581446e0 2023-10-24T23:52:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.552+0000 7f495cb81640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f493c065e60 0x7f493c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.552+0000 7f495cb81640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f493c065e60 0x7f493c068320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f494c00b9d0 tx=0x7f494c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:59.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.556+0000 7f49557fa640 1 -- 172.21.15.107:0/189856982 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f49500ec5e0 con 0x7f49581446e0 2023-10-24T23:52:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.704+0000 7f495ede5640 1 -- 172.21.15.107:0/189856982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4958070840 con 0x7f49581446e0 2023-10-24T23:52:59.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f49557fa640 1 -- 172.21.15.107:0/189856982 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14216 (secure 0 0 0) 0x7f49500efc00 con 0x7f49581446e0 2023-10-24T23:52:59.709 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:52:59.709 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":19,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:52:59.149291+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:52:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 -- 172.21.15.107:0/189856982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f493c065e60 msgr2=0x7f493c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f493c065e60 0x7f493c068320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f494c00b9d0 tx=0x7f494c008040 comp rx=0 tx=0).stop 2023-10-24T23:52:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 -- 172.21.15.107:0/189856982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49581446e0 msgr2=0x7f4958144b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49581446e0 0x7f4958144b90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f49500af040 tx=0x7f4950019620 comp rx=0 tx=0).stop 2023-10-24T23:52:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 -- 172.21.15.107:0/189856982 shutdown_connections 2023-10-24T23:52:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49581446e0 0x7f4958144b90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f493c065e60 0x7f493c068320 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f495814fd40 0x7f4958151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 --2- 172.21.15.107:0/189856982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f495806ee40 0x7f49581ee860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 -- 172.21.15.107:0/189856982 >> 172.21.15.107:0/189856982 conn(0x7f49580c2950 msgr2=0x7f495815c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 -- 172.21.15.107:0/189856982 shutdown_connections 2023-10-24T23:52:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.708+0000 7f495ede5640 1 -- 172.21.15.107:0/189856982 wait complete. 2023-10-24T23:52:59.725 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-24T23:52:59.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-24T23:52:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3c870640 1 Processor -- start 2023-10-24T23:52:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3c870640 1 -- start start 2023-10-24T23:52:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3c870640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fad380c1b30 0x7fad38072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3c870640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad380aa9f0 0x7fad38155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3c870640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad380befa0 con 0x7fad380a9300 2023-10-24T23:52:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3c870640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad380a9b80 con 0x7fad380c1b30 2023-10-24T23:52:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3c870640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad380aa0b0 con 0x7fad380aa9f0 2023-10-24T23:52:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fad380c1b30 0x7fad38072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad35d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fad380a9300 0x7fad380aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43006/0 (socket says 172.21.15.107:43006) 2023-10-24T23:52:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad35d9b640 1 -- 172.21.15.107:0/1080642310 learned_addr learned my addr 172.21.15.107:0/1080642310 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:52:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad36d9d640 1 --2- 172.21.15.107:0/1080642310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad380aa9f0 0x7fad38155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3659c640 1 -- 172.21.15.107:0/1080642310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad380aa9f0 msgr2=0x7fad38155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:52:59.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3559a640 1 -- 172.21.15.107:0/1080642310 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688417625 0 0) 0x7fad380befa0 con 0x7fad380a9300 2023-10-24T23:52:59.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3659c640 1 --2- 172.21.15.107:0/1080642310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad380aa9f0 0x7fad38155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3659c640 1 -- 172.21.15.107:0/1080642310 >> v1:172.21.15.107:6789/0 conn(0x7fad380a9300 legacy=0x7fad380aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.848+0000 7fad3659c640 1 -- 172.21.15.107:0/1080642310 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad380bf1f0 con 0x7fad380c1b30 2023-10-24T23:52:59.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3659c640 1 --2- 172.21.15.107:0/1080642310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad380c1b30 0x7fad38072210 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fad300097c0 tx=0x7fad3001e2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d866bea52169e69 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:59.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3559a640 1 -- 172.21.15.107:0/1080642310 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad300ad600 con 0x7fad380c1b30 2023-10-24T23:52:59.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3559a640 1 -- 172.21.15.107:0/1080642310 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad300a9050 con 0x7fad380c1b30 2023-10-24T23:52:59.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3559a640 1 -- 172.21.15.107:0/1080642310 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad300ad980 con 0x7fad380c1b30 2023-10-24T23:52:59.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 -- 172.21.15.107:0/1080642310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad380c1b30 msgr2=0x7fad38072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 --2- 172.21.15.107:0/1080642310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad380c1b30 0x7fad38072210 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fad300097c0 tx=0x7fad3001e2b0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 -- 172.21.15.107:0/1080642310 shutdown_connections 2023-10-24T23:52:59.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 --2- 172.21.15.107:0/1080642310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad380aa9f0 0x7fad38155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 --2- 172.21.15.107:0/1080642310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad380c1b30 0x7fad38072210 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 -- 172.21.15.107:0/1080642310 >> 172.21.15.107:0/1080642310 conn(0x7fad380c2860 msgr2=0x7fad380c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:52:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 -- 172.21.15.107:0/1080642310 shutdown_connections 2023-10-24T23:52:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 -- 172.21.15.107:0/1080642310 wait complete. 2023-10-24T23:52:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 Processor -- start 2023-10-24T23:52:59.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 -- start start 2023-10-24T23:52:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad380a9300 0x7fad38153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad380c1b30 0x7fad3814c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad3806ec50 0x7fad381ea090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3806a530 con 0x7fad380c1b30 2023-10-24T23:52:59.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad381530c0 con 0x7fad380a9300 2023-10-24T23:52:59.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3c870640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad381e9b80 con 0x7fad3806ec50 2023-10-24T23:52:59.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad35d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad380c1b30 0x7fad3814c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad35d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad380c1b30 0x7fad3814c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58472/0 (socket says 172.21.15.107:58472) 2023-10-24T23:52:59.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad35d9b640 1 -- 172.21.15.107:0/674147372 learned_addr learned my addr 172.21.15.107:0/674147372 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:52:59.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad3659c640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad380a9300 0x7fad38153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.852+0000 7fad36d9d640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad3806ec50 0x7fad381ea090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad35d9b640 1 -- 172.21.15.107:0/674147372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad3806ec50 msgr2=0x7fad381ea090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad35d9b640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad3806ec50 0x7fad381ea090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad35d9b640 1 -- 172.21.15.107:0/674147372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad380a9300 msgr2=0x7fad38153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:52:59.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad35d9b640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad380a9300 0x7fad38153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:52:59.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad35d9b640 1 -- 172.21.15.107:0/674147372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad30009420 con 0x7fad380c1b30 2023-10-24T23:52:59.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad35d9b640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad380c1b30 0x7fad3814c590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00c4e0 tx=0x7fad2c00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:59.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad26ffd640 1 -- 172.21.15.107:0/674147372 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c0074e0 con 0x7fad380c1b30 2023-10-24T23:52:59.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad26ffd640 1 -- 172.21.15.107:0/674147372 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c007660 con 0x7fad380c1b30 2023-10-24T23:52:59.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad3c870640 1 -- 172.21.15.107:0/674147372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad381e8c20 con 0x7fad380c1b30 2023-10-24T23:52:59.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad26ffd640 1 -- 172.21.15.107:0/674147372 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c004a30 con 0x7fad380c1b30 2023-10-24T23:52:59.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad3c870640 1 -- 172.21.15.107:0/674147372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad38152d50 con 0x7fad380c1b30 2023-10-24T23:52:59.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad3c870640 1 -- 172.21.15.107:0/674147372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3804e130 con 0x7fad380c1b30 2023-10-24T23:52:59.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad26ffd640 1 -- 172.21.15.107:0/674147372 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fad2c004b90 con 0x7fad380c1b30 2023-10-24T23:52:59.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad26ffd640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad18065e90 0x7fad18068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:52:59.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad26ffd640 1 -- 172.21.15.107:0/674147372 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fad2c082940 con 0x7fad380c1b30 2023-10-24T23:52:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad3659c640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad18065e90 0x7fad18068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:52:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.856+0000 7fad3659c640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad18065e90 0x7fad18068350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fad300af000 tx=0x7fad30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:52:59.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:52:59.860+0000 7fad26ffd640 1 -- 172.21.15.107:0/674147372 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fad2c050430 con 0x7fad380c1b30 2023-10-24T23:53:00.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.035+0000 7fad3c870640 1 -- 172.21.15.107:0/674147372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fad380a90e0 con 0x7fad380c1b30 2023-10-24T23:53:00.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.035+0000 7fad26ffd640 1 -- 172.21.15.107:0/674147372 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad2c0180a0 con 0x7fad380c1b30 2023-10-24T23:53:00.039 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-24T23:53:00.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 -- 172.21.15.107:0/674147372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad18065e90 msgr2=0x7fad18068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad18065e90 0x7fad18068350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fad300af000 tx=0x7fad30008040 comp rx=0 tx=0).stop 2023-10-24T23:53:00.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 -- 172.21.15.107:0/674147372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad380c1b30 msgr2=0x7fad3814c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad380c1b30 0x7fad3814c590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00c4e0 tx=0x7fad2c00c9a0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 -- 172.21.15.107:0/674147372 shutdown_connections 2023-10-24T23:53:00.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad18065e90 0x7fad18068350 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad3806ec50 0x7fad381ea090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad380c1b30 0x7fad3814c590 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 --2- 172.21.15.107:0/674147372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad380a9300 0x7fad38153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 -- 172.21.15.107:0/674147372 >> 172.21.15.107:0/674147372 conn(0x7fad380c2860 msgr2=0x7fad380aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:00.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 -- 172.21.15.107:0/674147372 shutdown_connections 2023-10-24T23:53:00.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.039+0000 7fad3c870640 1 -- 172.21.15.107:0/674147372 wait complete. 2023-10-24T23:53:00.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-24T23:53:00.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d122de640 1 Processor -- start 2023-10-24T23:53:00.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d122de640 1 -- start start 2023-10-24T23:53:00.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d122de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d0c1553e0 0x7f8d0c15d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d122de640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0c065930 con 0x7f8d0c0c1320 2023-10-24T23:53:00.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d122de640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0c065b80 con 0x7f8d0c06d7c0 2023-10-24T23:53:00.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d122de640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c065e60 con 0x7f8d0c1553e0 2023-10-24T23:53:00.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8d0c0c1320 0x7f8d0c154cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43014/0 (socket says 172.21.15.107:43014) 2023-10-24T23:53:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0b7fe640 1 -- 172.21.15.107:0/3091722951 learned_addr learned my addr 172.21.15.107:0/3091722951 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d1087b640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d0c1553e0 0x7f8d0c15d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710856949 0 0) 0x7f8d0c065930 con 0x7f8d0c0c1320 2023-10-24T23:53:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cf8003530 con 0x7f8d0c0c1320 2023-10-24T23:53:00.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 838953091 0 0) 0x7f8cf8003530 con 0x7f8d0c0c1320 2023-10-24T23:53:00.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d0c1553e0 msgr2=0x7f8d0c15d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:00.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d0c1553e0 0x7f8d0c15d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 >> v1:172.21.15.110:6789/0 conn(0x7f8d0c06d7c0 legacy=0x7f8d0c14e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c0c5700 con 0x7f8d0c0c1320 2023-10-24T23:53:00.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f8cf4001680 con 0x7f8d0c0c1320 2023-10-24T23:53:00.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 >> v1:172.21.15.107:6789/0 conn(0x7f8d0c0c1320 legacy=0x7f8d0c154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cf80039b0 0x7f8cf8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cf80047d0 0x7f8cf8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf8005190 0x7f8cf800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c065e60 con 0x7f8cf80047d0 2023-10-24T23:53:00.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf800bed0 con 0x7f8cf80039b0 2023-10-24T23:53:00.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf800e270 con 0x7f8cf8005190 2023-10-24T23:53:00.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0b7fe640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cf80047d0 0x7f8cf8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0bfff640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cf80039b0 0x7f8cf8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0b7fe640 1 -- 172.21.15.107:0/3091722951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf8005190 msgr2=0x7f8cf800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:00.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d1087b640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf8005190 0x7f8cf800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0b7fe640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cf8005190 0x7f8cf800b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0b7fe640 1 -- 172.21.15.107:0/3091722951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cf80039b0 msgr2=0x7f8cf8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0b7fe640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cf80039b0 0x7f8cf8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.183+0000 7f8d0b7fe640 1 -- 172.21.15.107:0/3091722951 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf8003530 con 0x7f8cf80047d0 2023-10-24T23:53:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d0b7fe640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cf80047d0 0x7f8cf8004c50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400c240 tx=0x7f8cf401cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9fc3952f4416aab server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4005ae0 con 0x7f8cf80047d0 2023-10-24T23:53:00.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf4005c60 con 0x7f8cf80047d0 2023-10-24T23:53:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d0affd640 1 -- 172.21.15.107:0/3091722951 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf40a57f0 con 0x7f8cf80047d0 2023-10-24T23:53:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 -- 172.21.15.107:0/3091722951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cf80047d0 msgr2=0x7f8cf8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 --2- 172.21.15.107:0/3091722951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cf80047d0 0x7f8cf8004c50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400c240 tx=0x7f8cf401cc10 comp rx=0 tx=0).stop 2023-10-24T23:53:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d1087b640 1 -- 172.21.15.107:0/3091722951 reap_dead start 2023-10-24T23:53:00.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 -- 172.21.15.107:0/3091722951 shutdown_connections 2023-10-24T23:53:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 -- 172.21.15.107:0/3091722951 >> 172.21.15.107:0/3091722951 conn(0x7f8d0c0c65e0 msgr2=0x7f8d0c0c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 -- 172.21.15.107:0/3091722951 shutdown_connections 2023-10-24T23:53:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 -- 172.21.15.107:0/3091722951 wait complete. 2023-10-24T23:53:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 Processor -- start 2023-10-24T23:53:00.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 -- start start 2023-10-24T23:53:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d0c06f680 0x7f8d0c06fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d0c156d10 0x7f8d0c14db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c14e0d0 0x7f8d0c1e8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c1e8090 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c157190 con 0x7f8d0c156d10 2023-10-24T23:53:00.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d122de640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c070d70 con 0x7f8d0c06f680 2023-10-24T23:53:00.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d1087b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c14e0d0 0x7f8d0c1e8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d1087b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c14e0d0 0x7f8d0c1e8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58496/0 (socket says 172.21.15.107:58496) 2023-10-24T23:53:00.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d1087b640 1 -- 172.21.15.107:0/2612360998 learned_addr learned my addr 172.21.15.107:0/2612360998 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:00.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d0b7fe640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d0c156d10 0x7f8d0c14db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d0bfff640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d0c06f680 0x7f8d0c06fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d1087b640 1 -- 172.21.15.107:0/2612360998 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d0c06f680 msgr2=0x7f8d0c06fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d1087b640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d0c06f680 0x7f8d0c06fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d1087b640 1 -- 172.21.15.107:0/2612360998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d0c156d10 msgr2=0x7f8d0c14db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d1087b640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d0c156d10 0x7f8d0c14db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d1087b640 1 -- 172.21.15.107:0/2612360998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0c14e550 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.187+0000 7f8d1087b640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c14e0d0 0x7f8d0c1e8520 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8004750 tx=0x7f8d0400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.191+0000 7f8d08ff9640 1 -- 172.21.15.107:0/2612360998 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d04017400 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.191+0000 7f8d08ff9640 1 -- 172.21.15.107:0/2612360998 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d04011040 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.191+0000 7f8d122de640 1 -- 172.21.15.107:0/2612360998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0c1ec0b0 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.191+0000 7f8d08ff9640 1 -- 172.21.15.107:0/2612360998 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d040177c0 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.191+0000 7f8d122de640 1 -- 172.21.15.107:0/2612360998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0c06dd60 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.191+0000 7f8d122de640 1 -- 172.21.15.107:0/2612360998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0c1ec560 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.191+0000 7f8d08ff9640 1 -- 172.21.15.107:0/2612360998 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8d040203b0 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.191+0000 7f8d08ff9640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cec065e90 0x7f8cec068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.191+0000 7f8d08ff9640 1 -- 172.21.15.107:0/2612360998 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8d04083610 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.191+0000 7f8d0bfff640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cec065e90 0x7f8cec068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.191+0000 7f8d0bfff640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cec065e90 0x7f8cec068350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc005490 tx=0x7f8cfc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:00.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.195+0000 7f8d08ff9640 1 -- 172.21.15.107:0/2612360998 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8d04051040 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.339+0000 7f8d122de640 1 -- 172.21.15.107:0/2612360998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f8d0c067960 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.339+0000 7f8d08ff9640 1 -- 172.21.15.107:0/2612360998 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f8d04054660 con 0x7f8d0c14e0d0 2023-10-24T23:53:00.343 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-24T23:53:00.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 -- 172.21.15.107:0/2612360998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cec065e90 msgr2=0x7f8cec068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cec065e90 0x7f8cec068350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc005490 tx=0x7f8cfc009040 comp rx=0 tx=0).stop 2023-10-24T23:53:00.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 -- 172.21.15.107:0/2612360998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c14e0d0 msgr2=0x7f8d0c1e8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c14e0d0 0x7f8d0c1e8520 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8004750 tx=0x7f8d0400f4a0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 -- 172.21.15.107:0/2612360998 shutdown_connections 2023-10-24T23:53:00.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d0c14e0d0 0x7f8d0c1e8520 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d0c156d10 0x7f8d0c14db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cec065e90 0x7f8cec068350 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 --2- 172.21.15.107:0/2612360998 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d0c06f680 0x7f8d0c06fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 -- 172.21.15.107:0/2612360998 >> 172.21.15.107:0/2612360998 conn(0x7f8d0c0c65e0 msgr2=0x7f8d0c0a7750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:00.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 -- 172.21.15.107:0/2612360998 shutdown_connections 2023-10-24T23:53:00.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.343+0000 7f8d122de640 1 -- 172.21.15.107:0/2612360998 wait complete. 2023-10-24T23:53:00.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-24T23:53:00.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.471+0000 7fbcc3a88640 1 Processor -- start 2023-10-24T23:53:00.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.471+0000 7fbcc3a88640 1 -- start start 2023-10-24T23:53:00.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc3a88640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbcbc14c470 0x7fbcbc14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc3a88640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcbc155cd0 con 0x7fbcbc06f490 2023-10-24T23:53:00.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc3a88640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcbc0c4380 con 0x7fbcbc152b30 2023-10-24T23:53:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc3a88640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcbc1526e0 con 0x7fbcbc14c470 2023-10-24T23:53:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc1023640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbcbc06f490 0x7fbcbc071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43016/0 (socket says 172.21.15.107:43016) 2023-10-24T23:53:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc1023640 1 -- 172.21.15.107:0/2007055983 learned_addr learned my addr 172.21.15.107:0/2007055983 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc1824640 1 --2- 172.21.15.107:0/2007055983 >> v2:172.21.15.110:3300/0 conn(0x7fbcbc14c470 0x7fbcbc14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074264239 0 0) 0x7fbcbc155cd0 con 0x7fbcbc06f490 2023-10-24T23:53:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca8003530 con 0x7fbcbc06f490 2023-10-24T23:53:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2432535783 0 0) 0x7fbca8003530 con 0x7fbcbc06f490 2023-10-24T23:53:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 >> v2:172.21.15.110:3300/0 conn(0x7fbcbc14c470 msgr2=0x7fbcbc14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 --2- 172.21.15.107:0/2007055983 >> v2:172.21.15.110:3300/0 conn(0x7fbcbc14c470 0x7fbcbc14c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 >> v1:172.21.15.110:6789/0 conn(0x7fbcbc152b30 legacy=0x7fbcbc152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc0c48b0 con 0x7fbcbc06f490 2023-10-24T23:53:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fbcb8001680 con 0x7fbcbc06f490 2023-10-24T23:53:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 >> v1:172.21.15.107:6789/0 conn(0x7fbcbc06f490 legacy=0x7fbcbc071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 --2- 172.21.15.107:0/2007055983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbca8003980 0x7fbca8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 --2- 172.21.15.107:0/2007055983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbca80047d0 0x7fbca8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 --2- 172.21.15.107:0/2007055983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbca80052b0 0x7fbca800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc1526e0 con 0x7fbca8003980 2023-10-24T23:53:00.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca800c180 con 0x7fbca80052b0 2023-10-24T23:53:00.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca800c340 con 0x7fbca80047d0 2023-10-24T23:53:00.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc1824640 1 --2- 172.21.15.107:0/2007055983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbca8003980 0x7fbca8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc2025640 1 --2- 172.21.15.107:0/2007055983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbca80052b0 0x7fbca800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc1023640 1 --2- 172.21.15.107:0/2007055983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbca80047d0 0x7fbca8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc1824640 1 -- 172.21.15.107:0/2007055983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbca80047d0 msgr2=0x7fbca8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:00.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc1824640 1 --2- 172.21.15.107:0/2007055983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbca80047d0 0x7fbca8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc1824640 1 -- 172.21.15.107:0/2007055983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbca80052b0 msgr2=0x7fbca800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:00.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc1824640 1 --2- 172.21.15.107:0/2007055983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbca80052b0 0x7fbca800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc2025640 1 -- 172.21.15.107:0/2007055983 reap_dead start 2023-10-24T23:53:00.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc1824640 1 -- 172.21.15.107:0/2007055983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca8003530 con 0x7fbca8003980 2023-10-24T23:53:00.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.475+0000 7fbcc1824640 1 --2- 172.21.15.107:0/2007055983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbca8003980 0x7fbca8003da0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009ab0 tx=0x7fbcac01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f428506134f3649 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:00.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac0a9410 con 0x7fbca8003980 2023-10-24T23:53:00.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcac0a9570 con 0x7fbca8003980 2023-10-24T23:53:00.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc0822640 1 -- 172.21.15.107:0/2007055983 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac0a9920 con 0x7fbca8003980 2023-10-24T23:53:00.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2007055983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbca8003980 msgr2=0x7fbca8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 --2- 172.21.15.107:0/2007055983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbca8003980 0x7fbca8003da0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbcac009ab0 tx=0x7fbcac01ca30 comp rx=0 tx=0).stop 2023-10-24T23:53:00.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2007055983 shutdown_connections 2023-10-24T23:53:00.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 --2- 172.21.15.107:0/2007055983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbca8003980 0x7fbca8003da0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2007055983 >> 172.21.15.107:0/2007055983 conn(0x7fbcbc070f60 msgr2=0x7fbcbc071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:00.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2007055983 shutdown_connections 2023-10-24T23:53:00.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2007055983 wait complete. 2023-10-24T23:53:00.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 Processor -- start 2023-10-24T23:53:00.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 -- start start 2023-10-24T23:53:00.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbca8003980 0x7fbcbc1ebd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcbc14fdd0 0x7fbcbc1eb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcbc14f5b0 0x7fbcbc14e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc1eb870 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc14cda0 con 0x7fbca8003980 2023-10-24T23:53:00.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc3a88640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc14cff0 con 0x7fbcbc14f5b0 2023-10-24T23:53:00.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc1023640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcbc14fdd0 0x7fbcbc1eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc1023640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcbc14fdd0 0x7fbcbc1eb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58510/0 (socket says 172.21.15.107:58510) 2023-10-24T23:53:00.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.479+0000 7fbcc1023640 1 -- 172.21.15.107:0/2452707466 learned_addr learned my addr 172.21.15.107:0/2452707466 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:00.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcc1023640 1 -- 172.21.15.107:0/2452707466 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcbc14f5b0 msgr2=0x7fbcbc14e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcc1023640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcbc14f5b0 0x7fbcbc14e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcc1023640 1 -- 172.21.15.107:0/2452707466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbca8003980 msgr2=0x7fbcbc1ebd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcc1023640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbca8003980 0x7fbcbc1ebd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcc1023640 1 -- 172.21.15.107:0/2452707466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc14d2d0 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcc1023640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcbc14fdd0 0x7fbcbc1eb330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800e7b0 tx=0x7fbcb800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:00.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcb27fc640 1 -- 172.21.15.107:0/2452707466 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb800c660 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcb27fc640 1 -- 172.21.15.107:0/2452707466 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb800c7e0 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2452707466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcbc14d5b0 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcb27fc640 1 -- 172.21.15.107:0/2452707466 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb800cb90 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2452707466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcbc14db70 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2452707466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcbc14dd60 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcb27fc640 1 -- 172.21.15.107:0/2452707466 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbcb800ccf0 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcb27fc640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbca0065e90 0x7fbca0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcc1824640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbca0065e90 0x7fbca0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcb27fc640 1 -- 172.21.15.107:0/2452707466 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbcb8082ff0 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.483+0000 7fbcc1824640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbca0065e90 0x7fbca0068350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbcac0ae000 tx=0x7fbcac0b3040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:00.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.487+0000 7fbcb27fc640 1 -- 172.21.15.107:0/2452707466 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbcb80b4040 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2452707466 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbcbc0746e0 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcb27fc640 1 -- 172.21.15.107:0/2452707466 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+126 (secure 0 0 0) 0x7fbcb8050930 con 0x7fbcbc14fdd0 2023-10-24T23:53:00.646 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:00.646 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-10-24T23:53:00.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-10-24T23:53:00.646 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-10-24T23:53:00.647 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-10-24T23:53:00.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-10-24T23:53:00.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-10-24T23:53:00.647 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 2, 2023-10-24T23:53:00.647 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": "rbd" 2023-10-24T23:53:00.648 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-10-24T23:53:00.648 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-10-24T23:53:00.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2452707466 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbca0065e90 msgr2=0x7fbca0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbca0065e90 0x7fbca0068350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbcac0ae000 tx=0x7fbcac0b3040 comp rx=0 tx=0).stop 2023-10-24T23:53:00.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2452707466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcbc14fdd0 msgr2=0x7fbcbc1eb330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcbc14fdd0 0x7fbcbc1eb330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbcb800e7b0 tx=0x7fbcb800ec70 comp rx=0 tx=0).stop 2023-10-24T23:53:00.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2452707466 shutdown_connections 2023-10-24T23:53:00.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbca0065e90 0x7fbca0068350 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbcbc14f5b0 0x7fbcbc14e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcbc14fdd0 0x7fbcbc1eb330 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 --2- 172.21.15.107:0/2452707466 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbca8003980 0x7fbcbc1ebd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2452707466 >> 172.21.15.107:0/2452707466 conn(0x7fbcbc070f60 msgr2=0x7fbcbc073170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:00.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2452707466 shutdown_connections 2023-10-24T23:53:00.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.643+0000 7fbcc3a88640 1 -- 172.21.15.107:0/2452707466 wait complete. 2023-10-24T23:53:00.664 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'a' 2023-10-24T23:53:00.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_metadata --pg_num_min 64 2023-10-24T23:53:00.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3ad0cc640 1 Processor -- start 2023-10-24T23:53:00.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3ad0cc640 1 -- start start 2023-10-24T23:53:00.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3ad0cc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa3a80c1800 0x7fa3a80a9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3ad0cc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3a80c1be0 0x7fa3a8155570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3ad0cc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3a8155ab0 con 0x7fa3a80a8d60 2023-10-24T23:53:00.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3ad0cc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3a80c2070 con 0x7fa3a80c1800 2023-10-24T23:53:00.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3ad0cc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a8151960 con 0x7fa3a80c1be0 2023-10-24T23:53:00.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3a6d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa3a80c1800 0x7fa3a80a9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3a659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa3a80a8d60 0x7fa3a8154e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43032/0 (socket says 172.21.15.107:43032) 2023-10-24T23:53:00.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3a659c640 1 -- 172.21.15.107:0/4163366406 learned_addr learned my addr 172.21.15.107:0/4163366406 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:00.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3a759e640 1 --2- 172.21.15.107:0/4163366406 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3a80c1be0 0x7fa3a8155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3a6d9d640 1 -- 172.21.15.107:0/4163366406 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3a80c1be0 msgr2=0x7fa3a8155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:00.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3a6d9d640 1 --2- 172.21.15.107:0/4163366406 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3a80c1be0 0x7fa3a8155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3a6d9d640 1 -- 172.21.15.107:0/4163366406 >> v1:172.21.15.107:6789/0 conn(0x7fa3a80a8d60 legacy=0x7fa3a8154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3a6d9d640 1 -- 172.21.15.107:0/4163366406 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a8151c40 con 0x7fa3a80c1800 2023-10-24T23:53:00.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.783+0000 7fa3a5d9b640 1 -- 172.21.15.107:0/4163366406 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505504239 0 0) 0x7fa3a8155ab0 con 0x7fa3a80a8d60 2023-10-24T23:53:00.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3a6d9d640 1 --2- 172.21.15.107:0/4163366406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a80c1800 0x7fa3a80a9370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa3900097c0 tx=0x7fa39001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d73d253a81cdcae server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:00.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3a5d9b640 1 -- 172.21.15.107:0/4163366406 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3900ab600 con 0x7fa3a80c1800 2023-10-24T23:53:00.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3a5d9b640 1 -- 172.21.15.107:0/4163366406 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3900a7050 con 0x7fa3a80c1800 2023-10-24T23:53:00.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3a5d9b640 1 -- 172.21.15.107:0/4163366406 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3900ab980 con 0x7fa3a80c1800 2023-10-24T23:53:00.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/4163366406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a80c1800 msgr2=0x7fa3a80a9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3ad0cc640 1 --2- 172.21.15.107:0/4163366406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a80c1800 0x7fa3a80a9370 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa3900097c0 tx=0x7fa39001c360 comp rx=0 tx=0).stop 2023-10-24T23:53:00.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/4163366406 shutdown_connections 2023-10-24T23:53:00.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3ad0cc640 1 --2- 172.21.15.107:0/4163366406 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3a80c1be0 0x7fa3a8155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3ad0cc640 1 --2- 172.21.15.107:0/4163366406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a80c1800 0x7fa3a80a9370 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/4163366406 >> 172.21.15.107:0/4163366406 conn(0x7fa3a80c27f0 msgr2=0x7fa3a80c2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:00.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/4163366406 shutdown_connections 2023-10-24T23:53:00.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/4163366406 wait complete. 2023-10-24T23:53:00.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3ad0cc640 1 Processor -- start 2023-10-24T23:53:00.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.787+0000 7fa3ad0cc640 1 -- start start 2023-10-24T23:53:00.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3ad0cc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3a80a8d60 0x7fa3a81469a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3ad0cc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3a80c1be0 0x7fa3a80c0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3ad0cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a8146ee0 0x7fa3a81ec930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3ad0cc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a806ec60 con 0x7fa3a8146ee0 2023-10-24T23:53:00.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3ad0cc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a806f220 con 0x7fa3a80c1be0 2023-10-24T23:53:00.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3ad0cc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a8144130 con 0x7fa3a80a8d60 2023-10-24T23:53:00.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a8146ee0 0x7fa3a81ec930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a8146ee0 0x7fa3a81ec930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58530/0 (socket says 172.21.15.107:58530) 2023-10-24T23:53:00.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a759e640 1 -- 172.21.15.107:0/1142294585 learned_addr learned my addr 172.21.15.107:0/1142294585 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:00.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a759e640 1 -- 172.21.15.107:0/1142294585 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3a80a8d60 msgr2=0x7fa3a81469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a6d9d640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3a80a8d60 0x7fa3a81469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a659c640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3a80c1be0 0x7fa3a80c0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a759e640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3a80a8d60 0x7fa3a81469a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a759e640 1 -- 172.21.15.107:0/1142294585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3a80c1be0 msgr2=0x7fa3a80c0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:00.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a759e640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3a80c1be0 0x7fa3a80c0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:00.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a759e640 1 -- 172.21.15.107:0/1142294585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa390009420 con 0x7fa3a8146ee0 2023-10-24T23:53:00.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a759e640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a8146ee0 0x7fa3a81ec930 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000cc60 tx=0x7fa3a000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:00.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa39f7fe640 1 -- 172.21.15.107:0/1142294585 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000ce70 con 0x7fa3a8146ee0 2023-10-24T23:53:00.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa39f7fe640 1 -- 172.21.15.107:0/1142294585 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a00039a0 con 0x7fa3a8146ee0 2023-10-24T23:53:00.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/1142294585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a81f0850 con 0x7fa3a8146ee0 2023-10-24T23:53:00.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa39f7fe640 1 -- 172.21.15.107:0/1142294585 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0003d30 con 0x7fa3a8146ee0 2023-10-24T23:53:00.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/1142294585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a806eeb0 con 0x7fa3a8146ee0 2023-10-24T23:53:00.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/1142294585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a80746e0 con 0x7fa3a8146ee0 2023-10-24T23:53:00.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa39f7fe640 1 -- 172.21.15.107:0/1142294585 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa3a0003e90 con 0x7fa3a8146ee0 2023-10-24T23:53:00.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa39f7fe640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa388065e60 0x7fa388068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:00.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa39f7fe640 1 -- 172.21.15.107:0/1142294585 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa3a0083870 con 0x7fa3a8146ee0 2023-10-24T23:53:00.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a6d9d640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa388065e60 0x7fa388068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:00.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.791+0000 7fa3a6d9d640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa388065e60 0x7fa388068320 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3900ad000 tx=0x7fa3900a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:00.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.795+0000 7fa39f7fe640 1 -- 172.21.15.107:0/1142294585 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa3a0051210 con 0x7fa3a8146ee0 2023-10-24T23:53:00.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:00.947+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/1142294585 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa3a806f360 con 0x7fa3a8146ee0 2023-10-24T23:53:01.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.295+0000 7fa39f7fe640 1 -- 172.21.15.107:0/1142294585 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64}]=0 pool 'a_metadata' created v20) v1 ==== 128+0+0 (secure 0 0 0) 0x7fa3a001d070 con 0x7fa3a8146ee0 2023-10-24T23:53:01.298 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'a_metadata' created 2023-10-24T23:53:01.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.299+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/1142294585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa388065e60 msgr2=0x7fa388068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:01.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.299+0000 7fa3ad0cc640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa388065e60 0x7fa388068320 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3900ad000 tx=0x7fa3900a6000 comp rx=0 tx=0).stop 2023-10-24T23:53:01.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.299+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/1142294585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a8146ee0 msgr2=0x7fa3a81ec930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:01.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.299+0000 7fa3ad0cc640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a8146ee0 0x7fa3a81ec930 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000cc60 tx=0x7fa3a000a5c0 comp rx=0 tx=0).stop 2023-10-24T23:53:01.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.303+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/1142294585 shutdown_connections 2023-10-24T23:53:01.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.303+0000 7fa3ad0cc640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a8146ee0 0x7fa3a81ec930 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:01.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.303+0000 7fa3ad0cc640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3a80c1be0 0x7fa3a80c0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:01.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.303+0000 7fa3ad0cc640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa388065e60 0x7fa388068320 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.303+0000 7fa3ad0cc640 1 --2- 172.21.15.107:0/1142294585 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3a80a8d60 0x7fa3a81469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.303+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/1142294585 >> 172.21.15.107:0/1142294585 conn(0x7fa3a80c27f0 msgr2=0x7fa3a815b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.303+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/1142294585 shutdown_connections 2023-10-24T23:53:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.303+0000 7fa3ad0cc640 1 -- 172.21.15.107:0/1142294585 wait complete. 2023-10-24T23:53:01.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-24T23:53:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f338a1e5640 1 Processor -- start 2023-10-24T23:53:01.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f338a1e5640 1 -- start start 2023-10-24T23:53:01.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f338a1e5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33840ab1a0 0x7f33840c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:01.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f338a1e5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3384072720 0x7f33840bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:01.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f338a1e5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33840c1ce0 con 0x7f33840bceb0 2023-10-24T23:53:01.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f338a1e5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33840be1d0 con 0x7f3384072720 2023-10-24T23:53:01.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f338a1e5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33840c0880 con 0x7f33840ab1a0 2023-10-24T23:53:01.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f33837fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33840ab1a0 0x7f33840c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:01.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f337bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3384072720 0x7f33840bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:01.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f33837fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33840ab1a0 0x7f33840c0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:37334/0 (socket says 172.21.15.107:37334) 2023-10-24T23:53:01.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f33837fe640 1 -- 172.21.15.107:0/1410045652 learned_addr learned my addr 172.21.15.107:0/1410045652 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:01.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f337bfff640 1 -- 172.21.15.107:0/1410045652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33840ab1a0 msgr2=0x7f33840c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:01.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f337bfff640 1 --2- 172.21.15.107:0/1410045652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33840ab1a0 0x7f33840c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:01.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f337bfff640 1 -- 172.21.15.107:0/1410045652 >> v1:172.21.15.110:6789/0 conn(0x7f33840bceb0 legacy=0x7f33840c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.443+0000 7f337bfff640 1 -- 172.21.15.107:0/1410045652 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33840c1f30 con 0x7f3384072720 2023-10-24T23:53:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f337bfff640 1 --2- 172.21.15.107:0/1410045652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3384072720 0x7f33840bfb30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f336c0097c0 tx=0x7f336c01c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2750cf9db7c2073 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f3382ffd640 1 -- 172.21.15.107:0/1410045652 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c0ab600 con 0x7f3384072720 2023-10-24T23:53:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f3382ffd640 1 -- 172.21.15.107:0/1410045652 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c0a7050 con 0x7f3384072720 2023-10-24T23:53:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f3382ffd640 1 -- 172.21.15.107:0/1410045652 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c0ab980 con 0x7f3384072720 2023-10-24T23:53:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 -- 172.21.15.107:0/1410045652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3384072720 msgr2=0x7f33840bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 --2- 172.21.15.107:0/1410045652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3384072720 0x7f33840bfb30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f336c0097c0 tx=0x7f336c01c330 comp rx=0 tx=0).stop 2023-10-24T23:53:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 -- 172.21.15.107:0/1410045652 shutdown_connections 2023-10-24T23:53:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 --2- 172.21.15.107:0/1410045652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3384072720 0x7f33840bfb30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 --2- 172.21.15.107:0/1410045652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33840ab1a0 0x7f33840c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:01.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 -- 172.21.15.107:0/1410045652 >> 172.21.15.107:0/1410045652 conn(0x7f33840c2860 msgr2=0x7f33840c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:01.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 -- 172.21.15.107:0/1410045652 shutdown_connections 2023-10-24T23:53:01.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 -- 172.21.15.107:0/1410045652 wait complete. 2023-10-24T23:53:01.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 Processor -- start 2023-10-24T23:53:01.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 -- start start 2023-10-24T23:53:01.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33840ab1a0 0x7f3384070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:01.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33840bceb0 0x7f33840aa300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:01.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33840aa840 0x7f338414d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:01.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3384071980 con 0x7f33840aa840 2023-10-24T23:53:01.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33840a49b0 con 0x7f33840bceb0 2023-10-24T23:53:01.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.447+0000 7f338a1e5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33840a9e70 con 0x7f33840ab1a0 2023-10-24T23:53:01.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3383fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33840aa840 0x7f338414d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:01.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3383fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33840aa840 0x7f338414d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58550/0 (socket says 172.21.15.107:58550) 2023-10-24T23:53:01.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3383fff640 1 -- 172.21.15.107:0/2239351349 learned_addr learned my addr 172.21.15.107:0/2239351349 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:01.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f33837fe640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33840ab1a0 0x7f3384070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:01.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f337bfff640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33840bceb0 0x7f33840aa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:01.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3383fff640 1 -- 172.21.15.107:0/2239351349 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33840ab1a0 msgr2=0x7f3384070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:01.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3383fff640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33840ab1a0 0x7f3384070ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3383fff640 1 -- 172.21.15.107:0/2239351349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33840bceb0 msgr2=0x7f33840aa300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3383fff640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33840bceb0 0x7f33840aa300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3383fff640 1 -- 172.21.15.107:0/2239351349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c009420 con 0x7f33840aa840 2023-10-24T23:53:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3383fff640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33840aa840 0x7f338414d580 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f337c00c730 tx=0x7f337c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3380ff9640 1 -- 172.21.15.107:0/2239351349 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c003b80 con 0x7f33840aa840 2023-10-24T23:53:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3380ff9640 1 -- 172.21.15.107:0/2239351349 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f337c003ce0 con 0x7f33840aa840 2023-10-24T23:53:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f338a1e5640 1 -- 172.21.15.107:0/2239351349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33840a46d0 con 0x7f33840aa840 2023-10-24T23:53:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3380ff9640 1 -- 172.21.15.107:0/2239351349 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337c00aa30 con 0x7f33840aa840 2023-10-24T23:53:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f338a1e5640 1 -- 172.21.15.107:0/2239351349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33840a9b00 con 0x7f33840aa840 2023-10-24T23:53:01.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f338a1e5640 1 -- 172.21.15.107:0/2239351349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33840a9fe0 con 0x7f33840aa840 2023-10-24T23:53:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3380ff9640 1 -- 172.21.15.107:0/2239351349 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f337c00ab90 con 0x7f33840aa840 2023-10-24T23:53:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3380ff9640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3364065e90 0x7f3364068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f33837fe640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3364065e90 0x7f3364068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f3380ff9640 1 -- 172.21.15.107:0/2239351349 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5575+0+0 (secure 0 0 0) 0x7f337c0833e0 con 0x7f33840aa840 2023-10-24T23:53:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.451+0000 7f33837fe640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3364065e90 0x7f3364068350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f338406fe30 tx=0x7f3374008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.455+0000 7f3380ff9640 1 -- 172.21.15.107:0/2239351349 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f337c050c20 con 0x7f33840aa840 2023-10-24T23:53:01.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:01.603+0000 7f338a1e5640 1 -- 172.21.15.107:0/2239351349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f33840beff0 con 0x7f33840aa840 2023-10-24T23:53:02.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.311+0000 7f3380ff9640 1 -- 172.21.15.107:0/2239351349 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'a_data' created v21) v1 ==== 160+0+0 (secure 0 0 0) 0x7f337c054240 con 0x7f33840aa840 2023-10-24T23:53:02.324 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'a_data' created 2023-10-24T23:53:02.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.331+0000 7f338a1e5640 1 -- 172.21.15.107:0/2239351349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3364065e90 msgr2=0x7f3364068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.331+0000 7f338a1e5640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3364065e90 0x7f3364068350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f338406fe30 tx=0x7f3374008040 comp rx=0 tx=0).stop 2023-10-24T23:53:02.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.331+0000 7f338a1e5640 1 -- 172.21.15.107:0/2239351349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33840aa840 msgr2=0x7f338414d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.331+0000 7f338a1e5640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33840aa840 0x7f338414d580 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f337c00c730 tx=0x7f337c00cbf0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.335+0000 7f338a1e5640 1 -- 172.21.15.107:0/2239351349 shutdown_connections 2023-10-24T23:53:02.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.335+0000 7f338a1e5640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33840aa840 0x7f338414d580 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.335+0000 7f338a1e5640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33840bceb0 0x7f33840aa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.335+0000 7f338a1e5640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3364065e90 0x7f3364068350 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.335+0000 7f338a1e5640 1 --2- 172.21.15.107:0/2239351349 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33840ab1a0 0x7f3384070ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.335+0000 7f338a1e5640 1 -- 172.21.15.107:0/2239351349 >> 172.21.15.107:0/2239351349 conn(0x7f33840c2860 msgr2=0x7f33840c2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:02.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.335+0000 7f338a1e5640 1 -- 172.21.15.107:0/2239351349 shutdown_connections 2023-10-24T23:53:02.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.335+0000 7f338a1e5640 1 -- 172.21.15.107:0/2239351349 wait complete. 2023-10-24T23:53:02.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new a a_metadata a_data 2023-10-24T23:53:02.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.471+0000 7f80c8e76640 1 Processor -- start 2023-10-24T23:53:02.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.471+0000 7f80c8e76640 1 -- start start 2023-10-24T23:53:02.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c8e76640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f80c40bceb0 0x7f80c40c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c8e76640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c40c1ce0 con 0x7f80c4072720 2023-10-24T23:53:02.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c8e76640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c40c0240 con 0x7f80c40ab1a0 2023-10-24T23:53:02.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c8e76640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c40be1d0 con 0x7f80c40bceb0 2023-10-24T23:53:02.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c1d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f80c4072720 0x7f80c415a650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43046/0 (socket says 172.21.15.107:43046) 2023-10-24T23:53:02.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c1d9b640 1 -- 172.21.15.107:0/3582927945 learned_addr learned my addr 172.21.15.107:0/3582927945 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:02.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11985722 0 0) 0x7f80c40c1ce0 con 0x7f80c4072720 2023-10-24T23:53:02.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80ac003530 con 0x7f80c4072720 2023-10-24T23:53:02.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c2d9d640 1 --2- 172.21.15.107:0/3582927945 >> v2:172.21.15.110:3300/0 conn(0x7f80c40bceb0 0x7f80c40c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1528660149 0 0) 0x7f80ac003530 con 0x7f80c4072720 2023-10-24T23:53:02.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 >> v2:172.21.15.110:3300/0 conn(0x7f80c40bceb0 msgr2=0x7f80c40c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:02.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 --2- 172.21.15.107:0/3582927945 >> v2:172.21.15.110:3300/0 conn(0x7f80c40bceb0 0x7f80c40c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 >> v1:172.21.15.110:6789/0 conn(0x7f80c40ab1a0 legacy=0x7f80c40bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c40c0880 con 0x7f80c4072720 2023-10-24T23:53:02.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f80b8001680 con 0x7f80c4072720 2023-10-24T23:53:02.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 >> v1:172.21.15.107:6789/0 conn(0x7f80c4072720 legacy=0x7f80c415a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80ac003980 0x7f80ac003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80ac0047d0 0x7f80ac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 0x7f80ac00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac00c150 con 0x7f80ac0052b0 2023-10-24T23:53:02.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac00c2c0 con 0x7f80ac0047d0 2023-10-24T23:53:02.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ac00c480 con 0x7f80ac003980 2023-10-24T23:53:02.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c2d9d640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 0x7f80ac00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c1d9b640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80ac0047d0 0x7f80ac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c259c640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80ac003980 0x7f80ac003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c2d9d640 1 -- 172.21.15.107:0/3582927945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80ac003980 msgr2=0x7f80ac003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:02.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c2d9d640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80ac003980 0x7f80ac003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c2d9d640 1 -- 172.21.15.107:0/3582927945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80ac0047d0 msgr2=0x7f80ac004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:02.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c2d9d640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80ac0047d0 0x7f80ac004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c2d9d640 1 -- 172.21.15.107:0/3582927945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac003530 con 0x7f80ac0052b0 2023-10-24T23:53:02.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c259c640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80ac003980 0x7f80ac003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:02.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c2d9d640 1 -- 172.21.15.107:0/3582927945 reap_dead start 2023-10-24T23:53:02.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c2d9d640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 0x7f80ac00b990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80c40abe40 tx=0x7f80bc01ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d81b27be5d3366ba server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:02.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc0a5bf0 con 0x7f80ac0052b0 2023-10-24T23:53:02.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80bc0a5d50 con 0x7f80ac0052b0 2023-10-24T23:53:02.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.475+0000 7f80c159a640 1 -- 172.21.15.107:0/3582927945 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80bc003be0 con 0x7f80ac0052b0 2023-10-24T23:53:02.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- 172.21.15.107:0/3582927945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 msgr2=0x7f80ac00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 0x7f80ac00b990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80c40abe40 tx=0x7f80bc01ce00 comp rx=0 tx=0).stop 2023-10-24T23:53:02.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- 172.21.15.107:0/3582927945 shutdown_connections 2023-10-24T23:53:02.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 --2- 172.21.15.107:0/3582927945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 0x7f80ac00b990 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- 172.21.15.107:0/3582927945 >> 172.21.15.107:0/3582927945 conn(0x7f80c40c2860 msgr2=0x7f80c40c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:02.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- 172.21.15.107:0/3582927945 shutdown_connections 2023-10-24T23:53:02.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- 172.21.15.107:0/3582927945 wait complete. 2023-10-24T23:53:02.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 Processor -- start 2023-10-24T23:53:02.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- start start 2023-10-24T23:53:02.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 0x7f80c41da530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80c41d8960 0x7f80c41d7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80c41d7c90 0x7f80c41e7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c41ec1d0 con 0x7f80ac0052b0 2023-10-24T23:53:02.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c41ebd40 con 0x7f80c41d7c90 2023-10-24T23:53:02.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c41eb410 con 0x7f80c41d8960 2023-10-24T23:53:02.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 0x7f80c41da530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 0x7f80c41da530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58566/0 (socket says 172.21.15.107:58566) 2023-10-24T23:53:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c259c640 1 -- 172.21.15.107:0/2611446266 learned_addr learned my addr 172.21.15.107:0/2611446266 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c2d9d640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80c41d7c90 0x7f80c41e7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c1d9b640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80c41d8960 0x7f80c41d7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c259c640 1 -- 172.21.15.107:0/2611446266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80c41d8960 msgr2=0x7f80c41d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c259c640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80c41d8960 0x7f80c41d7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c259c640 1 -- 172.21.15.107:0/2611446266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80c41d7c90 msgr2=0x7f80c41e7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c259c640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80c41d7c90 0x7f80c41e7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c259c640 1 -- 172.21.15.107:0/2611446266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c41eb6f0 con 0x7f80ac0052b0 2023-10-24T23:53:02.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c1d9b640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80c41d8960 0x7f80c41d7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:02.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c2d9d640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80c41d7c90 0x7f80c41e7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c259c640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 0x7f80c41da530 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f80b400d540 tx=0x7f80b400da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80b2ffd640 1 -- 172.21.15.107:0/2611446266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b4005480 con 0x7f80ac0052b0 2023-10-24T23:53:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80b2ffd640 1 -- 172.21.15.107:0/2611446266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b4003620 con 0x7f80ac0052b0 2023-10-24T23:53:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- 172.21.15.107:0/2611446266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c41eb9d0 con 0x7f80ac0052b0 2023-10-24T23:53:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80b2ffd640 1 -- 172.21.15.107:0/2611446266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b40039b0 con 0x7f80ac0052b0 2023-10-24T23:53:02.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- 172.21.15.107:0/2611446266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c41d8110 con 0x7f80ac0052b0 2023-10-24T23:53:02.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.479+0000 7f80c8e76640 1 -- 172.21.15.107:0/2611446266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c406d240 con 0x7f80ac0052b0 2023-10-24T23:53:02.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.483+0000 7f80b2ffd640 1 -- 172.21.15.107:0/2611446266 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f80b4003b10 con 0x7f80ac0052b0 2023-10-24T23:53:02.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.483+0000 7f80b2ffd640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80a4065e90 0x7f80a4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.483+0000 7f80b2ffd640 1 -- 172.21.15.107:0/2611446266 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5952+0+0 (secure 0 0 0) 0x7f80b4083690 con 0x7f80ac0052b0 2023-10-24T23:53:02.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.483+0000 7f80c1d9b640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80a4065e90 0x7f80a4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.483+0000 7f80c1d9b640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80a4065e90 0x7f80a4068350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f80c409b200 tx=0x7f80b8001440 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:02.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.487+0000 7f80b2ffd640 1 -- 172.21.15.107:0/2611446266 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f80b4050d50 con 0x7f80ac0052b0 2023-10-24T23:53:02.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.675+0000 7f80c8e76640 1 -- 172.21.15.107:0/2611446266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"} v 0) v1 -- 0x7f80c404e130 con 0x7f80ac0052b0 2023-10-24T23:53:02.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.687+0000 7f80b2ffd640 1 -- 172.21.15.107:0/2611446266 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 157+0+0 (secure 0 0 0) 0x7f80b4001240 con 0x7f80ac0052b0 2023-10-24T23:53:02.696 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 3 and data pool 4 2023-10-24T23:53:02.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.695+0000 7f80c8e76640 1 -- 172.21.15.107:0/2611446266 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80a4065e90 msgr2=0x7f80a4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.695+0000 7f80c8e76640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80a4065e90 0x7f80a4068350 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f80c409b200 tx=0x7f80b8001440 comp rx=0 tx=0).stop 2023-10-24T23:53:02.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.695+0000 7f80c8e76640 1 -- 172.21.15.107:0/2611446266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 msgr2=0x7f80c41da530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.695+0000 7f80c8e76640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 0x7f80c41da530 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f80b400d540 tx=0x7f80b400da00 comp rx=0 tx=0).stop 2023-10-24T23:53:02.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.699+0000 7f80c8e76640 1 -- 172.21.15.107:0/2611446266 shutdown_connections 2023-10-24T23:53:02.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.699+0000 7f80c8e76640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80c41d7c90 0x7f80c41e7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.699+0000 7f80c8e76640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80a4065e90 0x7f80a4068350 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.699+0000 7f80c8e76640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80c41d8960 0x7f80c41d7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.699+0000 7f80c8e76640 1 --2- 172.21.15.107:0/2611446266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80ac0052b0 0x7f80c41da530 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.699+0000 7f80c8e76640 1 -- 172.21.15.107:0/2611446266 >> 172.21.15.107:0/2611446266 conn(0x7f80c40c2860 msgr2=0x7f80c40c0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:02.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.699+0000 7f80c8e76640 1 -- 172.21.15.107:0/2611446266 shutdown_connections 2023-10-24T23:53:02.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.703+0000 7f80c8e76640 1 -- 172.21.15.107:0/2611446266 wait complete. 2023-10-24T23:53:02.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:02.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.843+0000 7f11106ed640 1 Processor -- start 2023-10-24T23:53:02.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.843+0000 7f11106ed640 1 -- start start 2023-10-24T23:53:02.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f11106ed640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1108152020 0x7f11080c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f11106ed640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11081553d0 0x7f110806ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f11106ed640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1108152400 con 0x7f11080c8640 2023-10-24T23:53:02.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f11106ed640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11081558d0 con 0x7f1108152020 2023-10-24T23:53:02.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f11106ed640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11080c6ce0 con 0x7f11081553d0 2023-10-24T23:53:02.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110dc88640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f11080c8640 0x7f110806a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39150/0 (socket says 172.21.15.107:39150) 2023-10-24T23:53:02.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110dc88640 1 -- 172.21.15.107:0/163958740 learned_addr learned my addr 172.21.15.107:0/163958740 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:02.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110ec8a640 1 --2- 172.21.15.107:0/163958740 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11081553d0 0x7f110806ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110e489640 1 --2- 172.21.15.107:0/163958740 >> v2:172.21.15.110:3300/0 conn(0x7f1108152020 0x7f11080c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203728070 0 0) 0x7f1108152400 con 0x7f11080c8640 2023-10-24T23:53:02.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f4003530 con 0x7f11080c8640 2023-10-24T23:53:02.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 627919399 0 0) 0x7f10f4003530 con 0x7f11080c8640 2023-10-24T23:53:02.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11081553d0 msgr2=0x7f110806ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:02.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 --2- 172.21.15.107:0/163958740 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11081553d0 0x7f110806ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 >> v2:172.21.15.110:3300/0 conn(0x7f1108152020 msgr2=0x7f11080c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:02.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 --2- 172.21.15.107:0/163958740 >> v2:172.21.15.110:3300/0 conn(0x7f1108152020 0x7f11080c4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080c6fc0 con 0x7f11080c8640 2023-10-24T23:53:02.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f10f8001680 con 0x7f11080c8640 2023-10-24T23:53:02.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 >> v1:172.21.15.107:6789/0 conn(0x7f11080c8640 legacy=0x7f110806a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 --2- 172.21.15.107:0/163958740 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11081553d0 0x7f10f4003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 --2- 172.21.15.107:0/163958740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10f4003f00 0x7f10f40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 --2- 172.21.15.107:0/163958740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f4004d10 0x7f10f4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11081558d0 con 0x7f10f4003f00 2023-10-24T23:53:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11080c6ce0 con 0x7f10f4004d10 2023-10-24T23:53:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f400b9e0 con 0x7f11081553d0 2023-10-24T23:53:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110dc88640 1 --2- 172.21.15.107:0/163958740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10f4003f00 0x7f10f40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110e489640 1 --2- 172.21.15.107:0/163958740 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11081553d0 0x7f10f4003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110dc88640 1 -- 172.21.15.107:0/163958740 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11081553d0 msgr2=0x7f10f4003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110dc88640 1 --2- 172.21.15.107:0/163958740 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11081553d0 0x7f10f4003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110dc88640 1 -- 172.21.15.107:0/163958740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f4004d10 msgr2=0x7f10f4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110dc88640 1 --2- 172.21.15.107:0/163958740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f4004d10 0x7f10f4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.847+0000 7f110dc88640 1 -- 172.21.15.107:0/163958740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f4003530 con 0x7f10f4003f00 2023-10-24T23:53:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110dc88640 1 --2- 172.21.15.107:0/163958740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10f4003f00 0x7f10f40047d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f10f800c290 tx=0x7f10f801d6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7ee078b38d9d8b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f80058e0 con 0x7f10f4003f00 2023-10-24T23:53:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f8005a40 con 0x7f10f4003f00 2023-10-24T23:53:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110d487640 1 -- 172.21.15.107:0/163958740 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8005df0 con 0x7f10f4003f00 2023-10-24T23:53:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 -- 172.21.15.107:0/163958740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10f4003f00 msgr2=0x7f10f40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 --2- 172.21.15.107:0/163958740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10f4003f00 0x7f10f40047d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f10f800c290 tx=0x7f10f801d6d0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110ec8a640 1 -- 172.21.15.107:0/163958740 reap_dead start 2023-10-24T23:53:02.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 -- 172.21.15.107:0/163958740 shutdown_connections 2023-10-24T23:53:02.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 -- 172.21.15.107:0/163958740 >> 172.21.15.107:0/163958740 conn(0x7f11080abd60 msgr2=0x7f11080ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:02.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 -- 172.21.15.107:0/163958740 shutdown_connections 2023-10-24T23:53:02.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 -- 172.21.15.107:0/163958740 wait complete. 2023-10-24T23:53:02.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 Processor -- start 2023-10-24T23:53:02.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 -- start start 2023-10-24T23:53:02.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f4003f00 0x7f11080c2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11080c3930 0x7f11080c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1108153d70 0x7f11081541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11080c3580 con 0x7f1108153d70 2023-10-24T23:53:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11081f1790 con 0x7f10f4003f00 2023-10-24T23:53:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1108154730 con 0x7f11080c3930 2023-10-24T23:53:02.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110ec8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1108153d70 0x7f11081541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110ec8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1108153d70 0x7f11081541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43312/0 (socket says 172.21.15.107:43312) 2023-10-24T23:53:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110ec8a640 1 -- 172.21.15.107:0/2977981516 learned_addr learned my addr 172.21.15.107:0/2977981516 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110dc88640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11080c3930 0x7f11080c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110e489640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f4003f00 0x7f11080c2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110ec8a640 1 -- 172.21.15.107:0/2977981516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11080c3930 msgr2=0x7f11080c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110ec8a640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11080c3930 0x7f11080c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110ec8a640 1 -- 172.21.15.107:0/2977981516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f4003f00 msgr2=0x7f11080c2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110ec8a640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f4003f00 0x7f11080c2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110ec8a640 1 -- 172.21.15.107:0/2977981516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080ae860 con 0x7f1108153d70 2023-10-24T23:53:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110e489640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f4003f00 0x7f11080c2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f110ec8a640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1108153d70 0x7f11081541f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f110000ee50 tx=0x7f110000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f10feffd640 1 -- 172.21.15.107:0/2977981516 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100010040 con 0x7f1108153d70 2023-10-24T23:53:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f10feffd640 1 -- 172.21.15.107:0/2977981516 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110000bbf0 con 0x7f1108153d70 2023-10-24T23:53:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 -- 172.21.15.107:0/2977981516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11080c32e0 con 0x7f1108153d70 2023-10-24T23:53:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f10feffd640 1 -- 172.21.15.107:0/2977981516 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100014400 con 0x7f1108153d70 2023-10-24T23:53:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.851+0000 7f11106ed640 1 -- 172.21.15.107:0/2977981516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11081f09d0 con 0x7f1108153d70 2023-10-24T23:53:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.855+0000 7f11106ed640 1 -- 172.21.15.107:0/2977981516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110814f090 con 0x7f1108153d70 2023-10-24T23:53:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.855+0000 7f10feffd640 1 -- 172.21.15.107:0/2977981516 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1100014560 con 0x7f1108153d70 2023-10-24T23:53:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.855+0000 7f10feffd640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f10ec065e90 0x7f10ec068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.855+0000 7f10feffd640 1 -- 172.21.15.107:0/2977981516 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f11000842a0 con 0x7f1108153d70 2023-10-24T23:53:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.855+0000 7f110e489640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f10ec065e90 0x7f10ec068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:02.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.855+0000 7f110e489640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f10ec065e90 0x7f10ec068350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f11040099a0 tx=0x7f1104008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:02.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:02.859+0000 7f10feffd640 1 -- 172.21.15.107:0/2977981516 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1100051a70 con 0x7f1108153d70 2023-10-24T23:53:03.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.015+0000 7f11106ed640 1 -- 172.21.15.107:0/2977981516 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11080746e0 con 0x7f1108153d70 2023-10-24T23:53:03.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f10feffd640 1 -- 172.21.15.107:0/2977981516 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17852 (secure 0 0 0) 0x7f1100055090 con 0x7f1108153d70 2023-10-24T23:53:03.021 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:03.021 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:02.678834+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:03.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 -- 172.21.15.107:0/2977981516 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f10ec065e90 msgr2=0x7f10ec068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f10ec065e90 0x7f10ec068350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f11040099a0 tx=0x7f1104008040 comp rx=0 tx=0).stop 2023-10-24T23:53:03.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 -- 172.21.15.107:0/2977981516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1108153d70 msgr2=0x7f11081541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1108153d70 0x7f11081541f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f110000ee50 tx=0x7f110000b480 comp rx=0 tx=0).stop 2023-10-24T23:53:03.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 -- 172.21.15.107:0/2977981516 shutdown_connections 2023-10-24T23:53:03.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1108153d70 0x7f11081541f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f10ec065e90 0x7f10ec068350 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11080c3930 0x7f11080c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 --2- 172.21.15.107:0/2977981516 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10f4003f00 0x7f11080c2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 -- 172.21.15.107:0/2977981516 >> 172.21.15.107:0/2977981516 conn(0x7f11080abd60 msgr2=0x7f110806a710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:03.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 -- 172.21.15.107:0/2977981516 shutdown_connections 2023-10-24T23:53:03.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.019+0000 7f11106ed640 1 -- 172.21.15.107:0/2977981516 wait complete. 2023-10-24T23:53:03.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:03.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.159+0000 7feff5452640 1 Processor -- start 2023-10-24T23:53:03.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.159+0000 7feff5452640 1 -- start start 2023-10-24T23:53:03.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7feff5452640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feff014b600 0x7feff0151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7feff5452640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feff006af70 0x7feff006c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7feff5452640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff00c1860 con 0x7feff006a230 2023-10-24T23:53:03.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7feff5452640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff0154d50 con 0x7feff006af70 2023-10-24T23:53:03.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7feff5452640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff01524f0 con 0x7feff014b600 2023-10-24T23:53:03.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefe67fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feff006af70 0x7feff006c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefef7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feff006a230 0x7feff01551e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39152/0 (socket says 172.21.15.107:39152) 2023-10-24T23:53:03.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefef7fe640 1 -- 172.21.15.107:0/2267733431 learned_addr learned my addr 172.21.15.107:0/2267733431 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:03.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefeeffd640 1 --2- 172.21.15.107:0/2267733431 >> v2:172.21.15.110:3300/0 conn(0x7feff014b600 0x7feff0151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefe67fc640 1 -- 172.21.15.107:0/2267733431 >> v2:172.21.15.110:3300/0 conn(0x7feff014b600 msgr2=0x7feff0151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:03.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefe67fc640 1 --2- 172.21.15.107:0/2267733431 >> v2:172.21.15.110:3300/0 conn(0x7feff014b600 0x7feff0151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefe67fc640 1 -- 172.21.15.107:0/2267733431 >> v1:172.21.15.107:6789/0 conn(0x7feff006a230 legacy=0x7feff01551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefee7fc640 1 -- 172.21.15.107:0/2267733431 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187261538 0 0) 0x7feff00c1860 con 0x7feff006a230 2023-10-24T23:53:03.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefe67fc640 1 -- 172.21.15.107:0/2267733431 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff014b9e0 con 0x7feff006af70 2023-10-24T23:53:03.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefe67fc640 1 --2- 172.21.15.107:0/2267733431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff006af70 0x7feff006c4c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feff006f860 tx=0x7fefd801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761478099b59509c server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:03.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefee7fc640 1 -- 172.21.15.107:0/2267733431 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd80ac600 con 0x7feff006af70 2023-10-24T23:53:03.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefee7fc640 1 -- 172.21.15.107:0/2267733431 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd80a8050 con 0x7feff006af70 2023-10-24T23:53:03.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.163+0000 7fefee7fc640 1 -- 172.21.15.107:0/2267733431 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd80ac980 con 0x7feff006af70 2023-10-24T23:53:03.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 -- 172.21.15.107:0/2267733431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff006af70 msgr2=0x7feff006c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 --2- 172.21.15.107:0/2267733431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff006af70 0x7feff006c4c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feff006f860 tx=0x7fefd801c3a0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 -- 172.21.15.107:0/2267733431 shutdown_connections 2023-10-24T23:53:03.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 --2- 172.21.15.107:0/2267733431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff006af70 0x7feff006c4c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 --2- 172.21.15.107:0/2267733431 >> v2:172.21.15.110:3300/0 conn(0x7feff014b600 0x7feff0151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 -- 172.21.15.107:0/2267733431 >> 172.21.15.107:0/2267733431 conn(0x7feff00c2950 msgr2=0x7feff00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:03.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 -- 172.21.15.107:0/2267733431 shutdown_connections 2023-10-24T23:53:03.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 -- 172.21.15.107:0/2267733431 wait complete. 2023-10-24T23:53:03.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 Processor -- start 2023-10-24T23:53:03.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 -- start start 2023-10-24T23:53:03.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feff006a230 0x7feff00a96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff006af70 0x7feff0150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff014b600 0x7feff0141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff00c4950 con 0x7feff006af70 2023-10-24T23:53:03.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff00c3ab0 con 0x7feff014b600 2023-10-24T23:53:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff00c4340 con 0x7feff006a230 2023-10-24T23:53:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff006af70 0x7feff0150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff006af70 0x7feff0150b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43332/0 (socket says 172.21.15.107:43332) 2023-10-24T23:53:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe67fc640 1 -- 172.21.15.107:0/487800769 learned_addr learned my addr 172.21.15.107:0/487800769 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:03.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefef7fe640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff014b600 0x7feff0141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefeeffd640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feff006a230 0x7feff00a96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe67fc640 1 -- 172.21.15.107:0/487800769 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feff006a230 msgr2=0x7feff00a96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe67fc640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feff006a230 0x7feff00a96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe67fc640 1 -- 172.21.15.107:0/487800769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff014b600 msgr2=0x7feff0141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe67fc640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff014b600 0x7feff0141b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe67fc640 1 -- 172.21.15.107:0/487800769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd8009420 con 0x7feff006af70 2023-10-24T23:53:03.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefef7fe640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff014b600 0x7feff0141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:03.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe67fc640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff006af70 0x7feff0150b90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fefd8018fb0 tx=0x7fefd801abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:03.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe7fff640 1 -- 172.21.15.107:0/487800769 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd80b7450 con 0x7feff006af70 2023-10-24T23:53:03.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe7fff640 1 -- 172.21.15.107:0/487800769 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd80b75b0 con 0x7feff006af70 2023-10-24T23:53:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 -- 172.21.15.107:0/487800769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff0150390 con 0x7feff006af70 2023-10-24T23:53:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7fefe7fff640 1 -- 172.21.15.107:0/487800769 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd80b7920 con 0x7feff006af70 2023-10-24T23:53:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.167+0000 7feff5452640 1 -- 172.21.15.107:0/487800769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff0150670 con 0x7feff006af70 2023-10-24T23:53:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.171+0000 7feff5452640 1 -- 172.21.15.107:0/487800769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff00c4b00 con 0x7feff006af70 2023-10-24T23:53:03.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.171+0000 7fefe7fff640 1 -- 172.21.15.107:0/487800769 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fefd80b3070 con 0x7feff006af70 2023-10-24T23:53:03.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.171+0000 7fefe7fff640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd4065e60 0x7fefd4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.171+0000 7fefeeffd640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd4065e60 0x7fefd4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.171+0000 7fefe7fff640 1 -- 172.21.15.107:0/487800769 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6042+0+0 (secure 0 0 0) 0x7fefd811de10 con 0x7feff006af70 2023-10-24T23:53:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.171+0000 7fefeeffd640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd4065e60 0x7fefd4068320 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009a50 tx=0x7fefe0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:03.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.175+0000 7fefe7fff640 1 -- 172.21.15.107:0/487800769 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fefd80eb500 con 0x7feff006af70 2023-10-24T23:53:03.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.331+0000 7feff5452640 1 -- 172.21.15.107:0/487800769 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feff0150890 con 0x7feff006af70 2023-10-24T23:53:03.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.335+0000 7fefe7fff640 1 -- 172.21.15.107:0/487800769 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+17852 (secure 0 0 0) 0x7fefd80eeb20 con 0x7feff006af70 2023-10-24T23:53:03.338 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:03.338 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:02.678834+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:03.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.335+0000 7feff5452640 1 -- 172.21.15.107:0/487800769 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd4065e60 msgr2=0x7fefd4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.335+0000 7feff5452640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd4065e60 0x7fefd4068320 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009a50 tx=0x7fefe0008040 comp rx=0 tx=0).stop 2023-10-24T23:53:03.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.335+0000 7feff5452640 1 -- 172.21.15.107:0/487800769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff006af70 msgr2=0x7feff0150b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.335+0000 7feff5452640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff006af70 0x7feff0150b90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fefd8018fb0 tx=0x7fefd801abe0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.339+0000 7feff5452640 1 -- 172.21.15.107:0/487800769 shutdown_connections 2023-10-24T23:53:03.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.339+0000 7feff5452640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff014b600 0x7feff0141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.339+0000 7feff5452640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff006af70 0x7feff0150b90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.339+0000 7feff5452640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd4065e60 0x7fefd4068320 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.339+0000 7feff5452640 1 --2- 172.21.15.107:0/487800769 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feff006a230 0x7feff00a96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.339+0000 7feff5452640 1 -- 172.21.15.107:0/487800769 >> 172.21.15.107:0/487800769 conn(0x7feff00c2950 msgr2=0x7feff006bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:03.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.339+0000 7feff5452640 1 -- 172.21.15.107:0/487800769 shutdown_connections 2023-10-24T23:53:03.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.339+0000 7feff5452640 1 -- 172.21.15.107:0/487800769 wait complete. 2023-10-24T23:53:03.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a standby_count_wanted 0 2023-10-24T23:53:03.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e702d3640 1 Processor -- start 2023-10-24T23:53:03.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e702d3640 1 -- start start 2023-10-24T23:53:03.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e702d3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e680c1b30 0x7f7e68072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e702d3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7e680aa9f0 0x7f7e68155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e702d3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e680befa0 con 0x7f7e680a9300 2023-10-24T23:53:03.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e702d3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e680a9b80 con 0x7f7e680aa9f0 2023-10-24T23:53:03.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e702d3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680aa0b0 con 0x7f7e680c1b30 2023-10-24T23:53:03.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e6e870640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7e680aa9f0 0x7f7e68155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e6e06f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e680c1b30 0x7f7e68072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e6e870640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7e680aa9f0 0x7f7e68155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60278/0 (socket says 172.21.15.107:60278) 2023-10-24T23:53:03.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e6e870640 1 -- 172.21.15.107:0/3460973911 learned_addr learned my addr 172.21.15.107:0/3460973911 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:03.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e6e870640 1 -- 172.21.15.107:0/3460973911 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e680c1b30 msgr2=0x7f7e68072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:03.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e6e870640 1 --2- 172.21.15.107:0/3460973911 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e680c1b30 0x7f7e68072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e6e870640 1 -- 172.21.15.107:0/3460973911 >> v1:172.21.15.110:6789/0 conn(0x7f7e680a9300 legacy=0x7f7e680aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.471+0000 7f7e6e870640 1 -- 172.21.15.107:0/3460973911 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e680bf1f0 con 0x7f7e680aa9f0 2023-10-24T23:53:03.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e6e870640 1 --2- 172.21.15.107:0/3460973911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e680aa9f0 0x7f7e68155620 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009e50 tx=0x7f7e5801c630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a62477b6afcfb5dd server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:03.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e6d06d640 1 -- 172.21.15.107:0/3460973911 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e580a7070 con 0x7f7e680aa9f0 2023-10-24T23:53:03.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e6d06d640 1 -- 172.21.15.107:0/3460973911 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e580090d0 con 0x7f7e680aa9f0 2023-10-24T23:53:03.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e6d06d640 1 -- 172.21.15.107:0/3460973911 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e580ab610 con 0x7f7e680aa9f0 2023-10-24T23:53:03.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e702d3640 1 -- 172.21.15.107:0/3460973911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e680aa9f0 msgr2=0x7f7e68155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e702d3640 1 --2- 172.21.15.107:0/3460973911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e680aa9f0 0x7f7e68155620 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009e50 tx=0x7f7e5801c630 comp rx=0 tx=0).stop 2023-10-24T23:53:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e702d3640 1 -- 172.21.15.107:0/3460973911 shutdown_connections 2023-10-24T23:53:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e702d3640 1 --2- 172.21.15.107:0/3460973911 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e680aa9f0 0x7f7e68155620 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e702d3640 1 --2- 172.21.15.107:0/3460973911 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e680c1b30 0x7f7e68072210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e702d3640 1 -- 172.21.15.107:0/3460973911 >> 172.21.15.107:0/3460973911 conn(0x7f7e680c2860 msgr2=0x7f7e680c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e702d3640 1 -- 172.21.15.107:0/3460973911 shutdown_connections 2023-10-24T23:53:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e702d3640 1 -- 172.21.15.107:0/3460973911 wait complete. 2023-10-24T23:53:03.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e702d3640 1 Processor -- start 2023-10-24T23:53:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.475+0000 7f7e702d3640 1 -- start start 2023-10-24T23:53:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e702d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e680a9300 0x7f7e680b9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e702d3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e680c1b30 0x7f7e68154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e702d3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e68071870 0x7f7e6806fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e702d3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6806eec0 con 0x7f7e680a9300 2023-10-24T23:53:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e702d3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680c4cd0 con 0x7f7e68071870 2023-10-24T23:53:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e702d3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e681f1040 con 0x7f7e680c1b30 2023-10-24T23:53:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6e06f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e680a9300 0x7f7e680b9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6e06f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e680a9300 0x7f7e680b9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43344/0 (socket says 172.21.15.107:43344) 2023-10-24T23:53:03.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6e06f640 1 -- 172.21.15.107:0/494372674 learned_addr learned my addr 172.21.15.107:0/494372674 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:03.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6d86e640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e680c1b30 0x7f7e68154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6e870640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e68071870 0x7f7e6806fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6e06f640 1 -- 172.21.15.107:0/494372674 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e680c1b30 msgr2=0x7f7e68154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6e06f640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e680c1b30 0x7f7e68154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6e06f640 1 -- 172.21.15.107:0/494372674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e68071870 msgr2=0x7f7e6806fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:03.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6e06f640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e68071870 0x7f7e6806fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:03.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6e06f640 1 -- 172.21.15.107:0/494372674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e58009b30 con 0x7f7e680a9300 2023-10-24T23:53:03.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6e870640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e68071870 0x7f7e6806fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:03.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6e06f640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e680a9300 0x7f7e680b9050 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000e870 tx=0x7f7e6000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:03.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e66ffd640 1 -- 172.21.15.107:0/494372674 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60009510 con 0x7f7e680a9300 2023-10-24T23:53:03.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e66ffd640 1 -- 172.21.15.107:0/494372674 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6000eec0 con 0x7f7e680a9300 2023-10-24T23:53:03.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e702d3640 1 -- 172.21.15.107:0/494372674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6806f110 con 0x7f7e680a9300 2023-10-24T23:53:03.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e66ffd640 1 -- 172.21.15.107:0/494372674 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6000cb40 con 0x7f7e680a9300 2023-10-24T23:53:03.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e702d3640 1 -- 172.21.15.107:0/494372674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6806f6d0 con 0x7f7e680a9300 2023-10-24T23:53:03.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e702d3640 1 -- 172.21.15.107:0/494372674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6806f890 con 0x7f7e680a9300 2023-10-24T23:53:03.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e66ffd640 1 -- 172.21.15.107:0/494372674 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7e6000cca0 con 0x7f7e680a9300 2023-10-24T23:53:03.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e66ffd640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 0x7f7e54068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:03.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e66ffd640 1 -- 172.21.15.107:0/494372674 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f7e60083940 con 0x7f7e680a9300 2023-10-24T23:53:03.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6d86e640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 0x7f7e54068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:03.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.479+0000 7f7e6d86e640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 0x7f7e54068320 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e680a4ca0 tx=0x7f7e5c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:03.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.483+0000 7f7e66ffd640 1 -- 172.21.15.107:0/494372674 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7e60050fa0 con 0x7f7e680a9300 2023-10-24T23:53:03.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:03.675+0000 7f7e702d3640 1 -- 172.21.15.107:0/494372674 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7e6804e130 con 0x7f7e680a9300 2023-10-24T23:53:03.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:53:03.799 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:53:04.285 INFO:tasks.ceph.mds.d.smithi110.stderr:2023-10-24T23:53:04.281+0000 7f3e357de640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:53:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.331+0000 7f7e66ffd640 1 -- 172.21.15.107:0/494372674 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 113+0+0 (secure 0 0 0) 0x7f7e600170b0 con 0x7f7e680a9300 2023-10-24T23:53:04.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 -- 172.21.15.107:0/494372674 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 msgr2=0x7f7e54068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:04.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 0x7f7e54068320 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e680a4ca0 tx=0x7f7e5c009040 comp rx=0 tx=0).stop 2023-10-24T23:53:04.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 -- 172.21.15.107:0/494372674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e680a9300 msgr2=0x7f7e680b9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:04.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e680a9300 0x7f7e680b9050 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000e870 tx=0x7f7e6000ec70 comp rx=0 tx=0).stop 2023-10-24T23:53:04.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 -- 172.21.15.107:0/494372674 shutdown_connections 2023-10-24T23:53:04.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e68071870 0x7f7e6806fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:04.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 0x7f7e54068320 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:04.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e680c1b30 0x7f7e68154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 --2- 172.21.15.107:0/494372674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e680a9300 0x7f7e680b9050 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 -- 172.21.15.107:0/494372674 >> 172.21.15.107:0/494372674 conn(0x7f7e680c2860 msgr2=0x7f7e680c2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 -- 172.21.15.107:0/494372674 shutdown_connections 2023-10-24T23:53:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.335+0000 7f7e702d3640 1 -- 172.21.15.107:0/494372674 wait complete. 2023-10-24T23:53:04.356 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2023-10-24T23:53:04.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false 2023-10-24T23:53:04.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8cb0b32640 1 Processor -- start 2023-10-24T23:53:04.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8cb0b32640 1 -- start start 2023-10-24T23:53:04.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8cb0b32640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8cac0bd930 0x7f8cac1602a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:04.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8cb0b32640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac0c8180 con 0x7f8cac06b360 2023-10-24T23:53:04.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8cb0b32640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cac155430 con 0x7f8cac0af9b0 2023-10-24T23:53:04.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8cb0b32640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cac0c5870 con 0x7f8cac0bd930 2023-10-24T23:53:04.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8cac06b360 0x7f8cac069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39160/0 (socket says 172.21.15.107:39160) 2023-10-24T23:53:04.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca3fff640 1 -- 172.21.15.107:0/3889400339 learned_addr learned my addr 172.21.15.107:0/3889400339 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:04.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8caad9d640 1 --2- 172.21.15.107:0/3889400339 >> v2:172.21.15.110:3300/0 conn(0x7f8cac0bd930 0x7f8cac1602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:04.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902682817 0 0) 0x7f8cac0c8180 con 0x7f8cac06b360 2023-10-24T23:53:04.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c98003530 con 0x7f8cac06b360 2023-10-24T23:53:04.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3651694313 0 0) 0x7f8c98003530 con 0x7f8cac06b360 2023-10-24T23:53:04.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 >> v2:172.21.15.110:3300/0 conn(0x7f8cac0bd930 msgr2=0x7f8cac1602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:04.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 --2- 172.21.15.107:0/3889400339 >> v2:172.21.15.110:3300/0 conn(0x7f8cac0bd930 0x7f8cac1602a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:04.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 >> v1:172.21.15.110:6789/0 conn(0x7f8cac0af9b0 legacy=0x7f8cac0c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:04.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac155710 con 0x7f8cac06b360 2023-10-24T23:53:04.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f8c94001680 con 0x7f8cac06b360 2023-10-24T23:53:04.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 >> v1:172.21.15.107:6789/0 conn(0x7f8cac06b360 legacy=0x7f8cac069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:04.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c98003980 0x7f8c98003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c980047d0 0x7f8c98004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c98005190 0x7f8c9800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac0c5870 con 0x7f8c98003980 2023-10-24T23:53:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9800c130 con 0x7f8c98005190 2023-10-24T23:53:04.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9800c2f0 con 0x7f8c980047d0 2023-10-24T23:53:04.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8caa59c640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c98003980 0x7f8c98003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:04.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8caad9d640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c98005190 0x7f8c9800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:04.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8ca3fff640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c980047d0 0x7f8c98004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:04.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8caa59c640 1 -- 172.21.15.107:0/3889400339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c980047d0 msgr2=0x7f8c98004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:04.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8caa59c640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c980047d0 0x7f8c98004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:04.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8caa59c640 1 -- 172.21.15.107:0/3889400339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c98005190 msgr2=0x7f8c9800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8caa59c640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c98005190 0x7f8c9800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8caa59c640 1 -- 172.21.15.107:0/3889400339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c98003530 con 0x7f8c98003980 2023-10-24T23:53:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8caad9d640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c98005190 0x7f8c9800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:04.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.475+0000 7f8caad9d640 1 -- 172.21.15.107:0/3889400339 reap_dead start 2023-10-24T23:53:04.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8caa59c640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c98003980 0x7f8c98003da0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c001780 tx=0x7f8c9c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb74f693ea00c670 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:04.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c0050d0 con 0x7f8c98003980 2023-10-24T23:53:04.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9c0a7050 con 0x7f8c98003980 2023-10-24T23:53:04.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8ca9d9b640 1 -- 172.21.15.107:0/3889400339 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c005490 con 0x7f8c98003980 2023-10-24T23:53:04.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 -- 172.21.15.107:0/3889400339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c98003980 msgr2=0x7f8c98003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:04.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c98003980 0x7f8c98003da0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c001780 tx=0x7f8c9c01c3b0 comp rx=0 tx=0).stop 2023-10-24T23:53:04.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 -- 172.21.15.107:0/3889400339 shutdown_connections 2023-10-24T23:53:04.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 --2- 172.21.15.107:0/3889400339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c98003980 0x7f8c98003da0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:04.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 -- 172.21.15.107:0/3889400339 >> 172.21.15.107:0/3889400339 conn(0x7f8cac071b00 msgr2=0x7f8cac0abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:04.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 -- 172.21.15.107:0/3889400339 shutdown_connections 2023-10-24T23:53:04.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 -- 172.21.15.107:0/3889400339 wait complete. 2023-10-24T23:53:04.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 Processor -- start 2023-10-24T23:53:04.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 -- start start 2023-10-24T23:53:04.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c98003980 0x7f8cac155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:04.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cac14f800 0x7f8cac0c3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:04.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cac1e2320 0x7f8cac1e27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:04.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac1564c0 con 0x7f8cac1e2320 2023-10-24T23:53:04.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac0c1ef0 con 0x7f8c98003980 2023-10-24T23:53:04.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8cb0b32640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac1ed670 con 0x7f8cac14f800 2023-10-24T23:53:04.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8caad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cac1e2320 0x7f8cac1e27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:04.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8caad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cac1e2320 0x7f8cac1e27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43352/0 (socket says 172.21.15.107:43352) 2023-10-24T23:53:04.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8caad9d640 1 -- 172.21.15.107:0/538425436 learned_addr learned my addr 172.21.15.107:0/538425436 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:04.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8ca3fff640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cac14f800 0x7f8cac0c3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:04.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8caa59c640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c98003980 0x7f8cac155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:04.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8caad9d640 1 -- 172.21.15.107:0/538425436 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cac14f800 msgr2=0x7f8cac0c3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:04.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8caad9d640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cac14f800 0x7f8cac0c3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:04.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8caad9d640 1 -- 172.21.15.107:0/538425436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c98003980 msgr2=0x7f8cac155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:04.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8caad9d640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c98003980 0x7f8cac155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:04.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.479+0000 7f8caad9d640 1 -- 172.21.15.107:0/538425436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac1ed910 con 0x7f8cac1e2320 2023-10-24T23:53:04.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8caad9d640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cac1e2320 0x7f8cac1e27a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8cac08c360 tx=0x7f8ca400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:04.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8ca2ffd640 1 -- 172.21.15.107:0/538425436 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4016420 con 0x7f8cac1e2320 2023-10-24T23:53:04.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8ca2ffd640 1 -- 172.21.15.107:0/538425436 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca4016580 con 0x7f8cac1e2320 2023-10-24T23:53:04.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8cb0b32640 1 -- 172.21.15.107:0/538425436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac0c18a0 con 0x7f8cac1e2320 2023-10-24T23:53:04.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8ca2ffd640 1 -- 172.21.15.107:0/538425436 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40168f0 con 0x7f8cac1e2320 2023-10-24T23:53:04.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8cb0b32640 1 -- 172.21.15.107:0/538425436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac0c4130 con 0x7f8cac1e2320 2023-10-24T23:53:04.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8cb0b32640 1 -- 172.21.15.107:0/538425436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cac066560 con 0x7f8cac1e2320 2023-10-24T23:53:04.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8ca2ffd640 1 -- 172.21.15.107:0/538425436 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8ca4016a50 con 0x7f8cac1e2320 2023-10-24T23:53:04.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8ca2ffd640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c8c065e60 0x7f8c8c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:04.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8ca2ffd640 1 -- 172.21.15.107:0/538425436 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f8ca4084610 con 0x7f8cac1e2320 2023-10-24T23:53:04.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8caa59c640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c8c065e60 0x7f8c8c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:04.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.483+0000 7f8caa59c640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c8c065e60 0x7f8c8c068320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c001750 tx=0x7f8c9c0011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:04.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.487+0000 7f8ca2ffd640 1 -- 172.21.15.107:0/538425436 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8ca4051d00 con 0x7f8cac1e2320 2023-10-24T23:53:04.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:04.663+0000 7f8cb0b32640 1 -- 172.21.15.107:0/538425436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8cac04e130 con 0x7f8cac1e2320 2023-10-24T23:53:05.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.335+0000 7f8ca2ffd640 1 -- 172.21.15.107:0/538425436 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 117+0+0 (secure 0 0 0) 0x7f8ca4055320 con 0x7f8cac1e2320 2023-10-24T23:53:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 -- 172.21.15.107:0/538425436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c8c065e60 msgr2=0x7f8c8c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:05.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c8c065e60 0x7f8c8c068320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c001750 tx=0x7f8c9c0011a0 comp rx=0 tx=0).stop 2023-10-24T23:53:05.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 -- 172.21.15.107:0/538425436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cac1e2320 msgr2=0x7f8cac1e27a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:05.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cac1e2320 0x7f8cac1e27a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8cac08c360 tx=0x7f8ca400f520 comp rx=0 tx=0).stop 2023-10-24T23:53:05.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 -- 172.21.15.107:0/538425436 shutdown_connections 2023-10-24T23:53:05.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cac1e2320 0x7f8cac1e27a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:05.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c8c065e60 0x7f8c8c068320 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:05.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cac14f800 0x7f8cac0c3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:05.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 --2- 172.21.15.107:0/538425436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c98003980 0x7f8cac155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:05.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 -- 172.21.15.107:0/538425436 >> 172.21.15.107:0/538425436 conn(0x7f8cac071b00 msgr2=0x7f8cac0c5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:05.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 -- 172.21.15.107:0/538425436 shutdown_connections 2023-10-24T23:53:05.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.339+0000 7f8cb0b32640 1 -- 172.21.15.107:0/538425436 wait complete. 2023-10-24T23:53:05.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a session_timeout 300 2023-10-24T23:53:05.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6ef1be3640 1 Processor -- start 2023-10-24T23:53:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6ef1be3640 1 -- start start 2023-10-24T23:53:05.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6ef1be3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6eec0ab1a0 0x7f6eec0c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:05.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6ef1be3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6eec072720 0x7f6eec0bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:05.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6ef1be3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec0c1ce0 con 0x7f6eec0bceb0 2023-10-24T23:53:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6ef1be3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eec0be1d0 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6ef1be3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eec0c0880 con 0x7f6eec072720 2023-10-24T23:53:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6eeb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6eec0ab1a0 0x7f6eec0c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6eeb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6eec0ab1a0 0x7f6eec0c0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43362/0 (socket says 172.21.15.107:43362) 2023-10-24T23:53:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/224220809 learned_addr learned my addr 172.21.15.107:0/224220809 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:05.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6eeaffd640 1 --2- 172.21.15.107:0/224220809 >> v2:172.21.15.110:3300/0 conn(0x7f6eec072720 0x7f6eec0bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:05.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/224220809 >> v2:172.21.15.110:3300/0 conn(0x7f6eec072720 msgr2=0x7f6eec0bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:05.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6eeb7fe640 1 --2- 172.21.15.107:0/224220809 >> v2:172.21.15.110:3300/0 conn(0x7f6eec072720 0x7f6eec0bfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:05.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/224220809 >> v1:172.21.15.110:6789/0 conn(0x7f6eec0bceb0 legacy=0x7f6eec0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:05.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.487+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/224220809 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec0c1f30 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6eeb7fe640 1 --2- 172.21.15.107:0/224220809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0ab1a0 0x7f6eec0c0d00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6eec071e30 tx=0x7f6ee001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d638fde318ca4d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:05.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6eea7fc640 1 -- 172.21.15.107:0/224220809 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00ab600 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6eea7fc640 1 -- 172.21.15.107:0/224220809 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee00a7050 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6eea7fc640 1 -- 172.21.15.107:0/224220809 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00ab980 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6ef1be3640 1 -- 172.21.15.107:0/224220809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0ab1a0 msgr2=0x7f6eec0c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:05.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6ef1be3640 1 --2- 172.21.15.107:0/224220809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0ab1a0 0x7f6eec0c0d00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6eec071e30 tx=0x7f6ee001c400 comp rx=0 tx=0).stop 2023-10-24T23:53:05.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6ef1be3640 1 -- 172.21.15.107:0/224220809 shutdown_connections 2023-10-24T23:53:05.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6ef1be3640 1 --2- 172.21.15.107:0/224220809 >> v2:172.21.15.110:3300/0 conn(0x7f6eec072720 0x7f6eec0bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:05.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6ef1be3640 1 --2- 172.21.15.107:0/224220809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0ab1a0 0x7f6eec0c0d00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:05.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6ef1be3640 1 -- 172.21.15.107:0/224220809 >> 172.21.15.107:0/224220809 conn(0x7f6eec0c2860 msgr2=0x7f6eec0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:05.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6ef1be3640 1 -- 172.21.15.107:0/224220809 shutdown_connections 2023-10-24T23:53:05.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6ef1be3640 1 -- 172.21.15.107:0/224220809 wait complete. 2023-10-24T23:53:05.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6ef1be3640 1 Processor -- start 2023-10-24T23:53:05.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.491+0000 7f6ef1be3640 1 -- start start 2023-10-24T23:53:05.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ef1be3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0ab1a0 0x7f6eec06b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:05.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ef1be3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6eec0bceb0 0x7f6eec155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:05.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ef1be3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6eec071960 0x7f6eec1ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:05.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ef1be3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec06a430 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ef1be3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec1eb740 con 0x7f6eec071960 2023-10-24T23:53:05.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ef1be3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec1edd50 con 0x7f6eec0bceb0 2023-10-24T23:53:05.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0ab1a0 0x7f6eec06b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:05.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0ab1a0 0x7f6eec06b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43366/0 (socket says 172.21.15.107:43366) 2023-10-24T23:53:05.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/1871898546 learned_addr learned my addr 172.21.15.107:0/1871898546 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:05.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eebfff640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6eec071960 0x7f6eec1ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:05.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeaffd640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6eec0bceb0 0x7f6eec155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:05.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/1871898546 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6eec0bceb0 msgr2=0x7f6eec155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:05.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeb7fe640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6eec0bceb0 0x7f6eec155260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:05.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/1871898546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6eec071960 msgr2=0x7f6eec1ee4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:05.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeb7fe640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6eec071960 0x7f6eec1ee4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:05.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeb7fe640 1 -- 172.21.15.107:0/1871898546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee00094d0 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eebfff640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6eec071960 0x7f6eec1ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:05.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeb7fe640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0ab1a0 0x7f6eec06b8a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6ee001aab0 tx=0x7f6ee00191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:05.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ed3fff640 1 -- 172.21.15.107:0/1871898546 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00aa7b0 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ed3fff640 1 -- 172.21.15.107:0/1871898546 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee00aa910 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ed3fff640 1 -- 172.21.15.107:0/1871898546 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00aac80 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ef1be3640 1 -- 172.21.15.107:0/1871898546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eec069e70 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ef1be3640 1 -- 172.21.15.107:0/1871898546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eec1edfa0 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ef1be3640 1 -- 172.21.15.107:0/1871898546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eec0beff0 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ed3fff640 1 -- 172.21.15.107:0/1871898546 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6ee00b2070 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ed3fff640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed8065e60 0x7f6ed8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:05.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeaffd640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed8065e60 0x7f6ed8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:05.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6ed3fff640 1 -- 172.21.15.107:0/1871898546 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f6ee011d820 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.495+0000 7f6eeaffd640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed8065e60 0x7f6ed8068320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6eec071820 tx=0x7f6edc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:05.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.499+0000 7f6ed3fff640 1 -- 172.21.15.107:0/1871898546 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6ee00eaf10 con 0x7f6eec0ab1a0 2023-10-24T23:53:05.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:05.683+0000 7f6ef1be3640 1 -- 172.21.15.107:0/1871898546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f6eec04e130 con 0x7f6eec0ab1a0 2023-10-24T23:53:06.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.347+0000 7f6ed3fff640 1 -- 172.21.15.107:0/1871898546 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 110+0+0 (secure 0 0 0) 0x7f6ee00ee530 con 0x7f6eec0ab1a0 2023-10-24T23:53:06.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 -- 172.21.15.107:0/1871898546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed8065e60 msgr2=0x7f6ed8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed8065e60 0x7f6ed8068320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6eec071820 tx=0x7f6edc008040 comp rx=0 tx=0).stop 2023-10-24T23:53:06.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 -- 172.21.15.107:0/1871898546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0ab1a0 msgr2=0x7f6eec06b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0ab1a0 0x7f6eec06b8a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6ee001aab0 tx=0x7f6ee00191a0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 -- 172.21.15.107:0/1871898546 shutdown_connections 2023-10-24T23:53:06.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6eec071960 0x7f6eec1ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed8065e60 0x7f6ed8068320 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6eec0bceb0 0x7f6eec155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 --2- 172.21.15.107:0/1871898546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eec0ab1a0 0x7f6eec06b8a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 -- 172.21.15.107:0/1871898546 >> 172.21.15.107:0/1871898546 conn(0x7f6eec0c2860 msgr2=0x7f6eec06f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:06.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 -- 172.21.15.107:0/1871898546 shutdown_connections 2023-10-24T23:53:06.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.351+0000 7f6ef1be3640 1 -- 172.21.15.107:0/1871898546 wait complete. 2023-10-24T23:53:06.374 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'a' 2023-10-24T23:53:06.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:06.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfeb857640 1 Processor -- start 2023-10-24T23:53:06.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfeb857640 1 -- start start 2023-10-24T23:53:06.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfeb857640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbfe40c8100 0x7fbfe40abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:06.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfeb857640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfe406a4b0 0x7fbfe40ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:06.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfeb857640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe4155680 con 0x7fbfe4152040 2023-10-24T23:53:06.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfeb857640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe4154a70 con 0x7fbfe40c8100 2023-10-24T23:53:06.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfeb857640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4154d50 con 0x7fbfe406a4b0 2023-10-24T23:53:06.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfe8df2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbfe40c8100 0x7fbfe40abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:06.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfe8df2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbfe40c8100 0x7fbfe40abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43380/0 (socket says 172.21.15.107:43380) 2023-10-24T23:53:06.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfe8df2640 1 -- 172.21.15.107:0/3246829321 learned_addr learned my addr 172.21.15.107:0/3246829321 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:06.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfe8df2640 1 -- 172.21.15.107:0/3246829321 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfe406a4b0 msgr2=0x7fbfe40ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:06.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfe8df2640 1 --2- 172.21.15.107:0/3246829321 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfe406a4b0 0x7fbfe40ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfe8df2640 1 -- 172.21.15.107:0/3246829321 >> v1:172.21.15.110:6789/0 conn(0x7fbfe4152040 legacy=0x7fbfe40c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:06.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfe8df2640 1 -- 172.21.15.107:0/3246829321 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe4155030 con 0x7fbfe40c8100 2023-10-24T23:53:06.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfe8df2640 1 --2- 172.21.15.107:0/3246829321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe40c8100 0x7fbfe40abd00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009870 tx=0x7fbfd401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53962b44018da9fe server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:06.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.491+0000 7fbfdbfff640 1 -- 172.21.15.107:0/3246829321 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd40ab600 con 0x7fbfe40c8100 2023-10-24T23:53:06.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfdbfff640 1 -- 172.21.15.107:0/3246829321 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd40a7050 con 0x7fbfe40c8100 2023-10-24T23:53:06.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfdbfff640 1 -- 172.21.15.107:0/3246829321 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd40ab980 con 0x7fbfe40c8100 2023-10-24T23:53:06.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 -- 172.21.15.107:0/3246829321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe40c8100 msgr2=0x7fbfe40abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 --2- 172.21.15.107:0/3246829321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe40c8100 0x7fbfe40abd00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009870 tx=0x7fbfd401c470 comp rx=0 tx=0).stop 2023-10-24T23:53:06.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 -- 172.21.15.107:0/3246829321 shutdown_connections 2023-10-24T23:53:06.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 --2- 172.21.15.107:0/3246829321 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfe406a4b0 0x7fbfe40ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 --2- 172.21.15.107:0/3246829321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe40c8100 0x7fbfe40abd00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 -- 172.21.15.107:0/3246829321 >> 172.21.15.107:0/3246829321 conn(0x7fbfe40ca200 msgr2=0x7fbfe40ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:06.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 -- 172.21.15.107:0/3246829321 shutdown_connections 2023-10-24T23:53:06.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 -- 172.21.15.107:0/3246829321 wait complete. 2023-10-24T23:53:06.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 Processor -- start 2023-10-24T23:53:06.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 -- start start 2023-10-24T23:53:06.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfe406a4b0 0x7fbfe40ab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:06.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfe40c8100 0x7fbfe41f0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:06.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe4152040 0x7fbfe41f0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:06.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41517e0 con 0x7fbfe4152040 2023-10-24T23:53:06.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41485c0 con 0x7fbfe40c8100 2023-10-24T23:53:06.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfeb857640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe41eed00 con 0x7fbfe406a4b0 2023-10-24T23:53:06.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfe9df4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe4152040 0x7fbfe41f0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:06.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfe9df4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe4152040 0x7fbfe41f0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43390/0 (socket says 172.21.15.107:43390) 2023-10-24T23:53:06.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfe9df4640 1 -- 172.21.15.107:0/1969776614 learned_addr learned my addr 172.21.15.107:0/1969776614 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:06.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfe95f3640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfe406a4b0 0x7fbfe40ab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:06.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfe8df2640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfe40c8100 0x7fbfe41f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:06.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfe9df4640 1 -- 172.21.15.107:0/1969776614 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfe406a4b0 msgr2=0x7fbfe40ab0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfe9df4640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfe406a4b0 0x7fbfe40ab0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfe9df4640 1 -- 172.21.15.107:0/1969776614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfe40c8100 msgr2=0x7fbfe41f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfe9df4640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfe40c8100 0x7fbfe41f0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfe9df4640 1 -- 172.21.15.107:0/1969776614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd40094d0 con 0x7fbfe4152040 2023-10-24T23:53:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.495+0000 7fbfe8df2640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfe40c8100 0x7fbfe41f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfe9df4640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe4152040 0x7fbfe41f0b70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00b970 tx=0x7fbfdc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:06.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfd9ffb640 1 -- 172.21.15.107:0/1969776614 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc00cb40 con 0x7fbfe4152040 2023-10-24T23:53:06.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfd9ffb640 1 -- 172.21.15.107:0/1969776614 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc00ccc0 con 0x7fbfe4152040 2023-10-24T23:53:06.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfeb857640 1 -- 172.21.15.107:0/1969776614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe41f10b0 con 0x7fbfe4152040 2023-10-24T23:53:06.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfd9ffb640 1 -- 172.21.15.107:0/1969776614 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc014680 con 0x7fbfe4152040 2023-10-24T23:53:06.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfeb857640 1 -- 172.21.15.107:0/1969776614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe41ee6b0 con 0x7fbfe4152040 2023-10-24T23:53:06.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfeb857640 1 -- 172.21.15.107:0/1969776614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe41eee70 con 0x7fbfe4152040 2023-10-24T23:53:06.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfd9ffb640 1 -- 172.21.15.107:0/1969776614 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbfdc01d3b0 con 0x7fbfe4152040 2023-10-24T23:53:06.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfd9ffb640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbfc8065e60 0x7fbfc8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:06.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfe95f3640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbfc8065e60 0x7fbfc8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:06.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfd9ffb640 1 -- 172.21.15.107:0/1969776614 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7fbfdc083770 con 0x7fbfe4152040 2023-10-24T23:53:06.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.499+0000 7fbfe95f3640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbfc8065e60 0x7fbfc8068320 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00016c0 tx=0x7fbfe0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:06.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.503+0000 7fbfd9ffb640 1 -- 172.21.15.107:0/1969776614 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbfdc050db0 con 0x7fbfe4152040 2023-10-24T23:53:06.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.683+0000 7fbfeb857640 1 -- 172.21.15.107:0/1969776614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfe40746e0 con 0x7fbfe4152040 2023-10-24T23:53:06.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfd9ffb640 1 -- 172.21.15.107:0/1969776614 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4647 (secure 0 0 0) 0x7fbfdc0543d0 con 0x7fbfe4152040 2023-10-24T23:53:06.689 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:06.689 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-24T23:53:06.689 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 8 2023-10-24T23:53:06.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 -- 172.21.15.107:0/1969776614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbfc8065e60 msgr2=0x7fbfc8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbfc8065e60 0x7fbfc8068320 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00016c0 tx=0x7fbfe0009040 comp rx=0 tx=0).stop 2023-10-24T23:53:06.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 -- 172.21.15.107:0/1969776614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe4152040 msgr2=0x7fbfe41f0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe4152040 0x7fbfe41f0b70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00b970 tx=0x7fbfdc00be30 comp rx=0 tx=0).stop 2023-10-24T23:53:06.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 -- 172.21.15.107:0/1969776614 shutdown_connections 2023-10-24T23:53:06.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbfe4152040 0x7fbfe41f0b70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbfe40c8100 0x7fbfe41f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbfc8065e60 0x7fbfc8068320 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 --2- 172.21.15.107:0/1969776614 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbfe406a4b0 0x7fbfe40ab0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 -- 172.21.15.107:0/1969776614 >> 172.21.15.107:0/1969776614 conn(0x7fbfe40ca200 msgr2=0x7fbfe414b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:06.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 -- 172.21.15.107:0/1969776614 shutdown_connections 2023-10-24T23:53:06.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.687+0000 7fbfeb857640 1 -- 172.21.15.107:0/1969776614 wait complete. 2023-10-24T23:53:06.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:06.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.815+0000 7fbd44310640 1 Processor -- start 2023-10-24T23:53:06.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.815+0000 7fbd44310640 1 -- start start 2023-10-24T23:53:06.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd44310640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd3c152060 0x7fbd3c0aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:06.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd44310640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd3c0a9aa0 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd44310640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd3c155480 con 0x7fbd3c0c2990 2023-10-24T23:53:06.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd44310640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd3c155760 con 0x7fbd3c152060 2023-10-24T23:53:06.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd420ac640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd3c152060 0x7fbd3c0aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:06.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd428ad640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbd3c0c20b0 0x7fbd3c0c1480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39170/0 (socket says 172.21.15.107:39170) 2023-10-24T23:53:06.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd428ad640 1 -- 172.21.15.107:0/1382606944 learned_addr learned my addr 172.21.15.107:0/1382606944 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:06.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd420ac640 1 -- 172.21.15.107:0/1382606944 >> v1:172.21.15.110:6789/0 conn(0x7fbd3c0c2990 legacy=0x7fbd3c0a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:06.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd420ac640 1 -- 172.21.15.107:0/1382606944 >> v1:172.21.15.107:6789/0 conn(0x7fbd3c0c20b0 legacy=0x7fbd3c0c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd420ac640 1 -- 172.21.15.107:0/1382606944 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd3c0a9450 con 0x7fbd3c152060 2023-10-24T23:53:06.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd410aa640 1 -- 172.21.15.107:0/1382606944 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102519483 0 0) 0x7fbd3c0a9aa0 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd420ac640 1 --2- 172.21.15.107:0/1382606944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3c152060 0x7fbd3c0aada0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbd380097c0 tx=0x7fbd3801c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943cef92083759cd server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:06.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd410aa640 1 -- 172.21.15.107:0/1382606944 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380ab600 con 0x7fbd3c152060 2023-10-24T23:53:06.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd410aa640 1 -- 172.21.15.107:0/1382606944 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd380a7050 con 0x7fbd3c152060 2023-10-24T23:53:06.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd410aa640 1 -- 172.21.15.107:0/1382606944 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380ab980 con 0x7fbd3c152060 2023-10-24T23:53:06.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd44310640 1 -- 172.21.15.107:0/1382606944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3c152060 msgr2=0x7fbd3c0aada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd44310640 1 --2- 172.21.15.107:0/1382606944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3c152060 0x7fbd3c0aada0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbd380097c0 tx=0x7fbd3801c2f0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd44310640 1 -- 172.21.15.107:0/1382606944 shutdown_connections 2023-10-24T23:53:06.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd44310640 1 --2- 172.21.15.107:0/1382606944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3c152060 0x7fbd3c0aada0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd44310640 1 -- 172.21.15.107:0/1382606944 >> 172.21.15.107:0/1382606944 conn(0x7fbd3c0c4a90 msgr2=0x7fbd3c0c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:06.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd44310640 1 -- 172.21.15.107:0/1382606944 shutdown_connections 2023-10-24T23:53:06.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.819+0000 7fbd44310640 1 -- 172.21.15.107:0/1382606944 wait complete. 2023-10-24T23:53:06.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd44310640 1 Processor -- start 2023-10-24T23:53:06.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd44310640 1 -- start start 2023-10-24T23:53:06.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd44310640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3c0c20b0 0x7fbd3c144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:06.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd44310640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd3c0c2990 0x7fbd3c1ed260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:06.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd44310640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd3c152060 0x7fbd3c0c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd44310640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c14ea50 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd44310640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c14f890 con 0x7fbd3c0c2990 2023-10-24T23:53:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd44310640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c153bd0 con 0x7fbd3c152060 2023-10-24T23:53:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd420ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3c0c20b0 0x7fbd3c144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:06.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd420ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3c0c20b0 0x7fbd3c144f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43408/0 (socket says 172.21.15.107:43408) 2023-10-24T23:53:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd420ac640 1 -- 172.21.15.107:0/3890672902 learned_addr learned my addr 172.21.15.107:0/3890672902 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd420ac640 1 -- 172.21.15.107:0/3890672902 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd3c152060 msgr2=0x7fbd3c0c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd420ac640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd3c152060 0x7fbd3c0c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd420ac640 1 -- 172.21.15.107:0/3890672902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd3c0c2990 msgr2=0x7fbd3c1ed260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd420ac640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd3c0c2990 0x7fbd3c1ed260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd420ac640 1 -- 172.21.15.107:0/3890672902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd38009420 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd420ac640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3c0c20b0 0x7fbd3c144f70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbd380ae040 tx=0x7fbd3801ab30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:06.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd32ffd640 1 -- 172.21.15.107:0/3890672902 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380b7650 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd32ffd640 1 -- 172.21.15.107:0/3890672902 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd380b77d0 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd44310640 1 -- 172.21.15.107:0/3890672902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd3c1ed7a0 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd32ffd640 1 -- 172.21.15.107:0/3890672902 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380b7b80 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd44310640 1 -- 172.21.15.107:0/3890672902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd3c151060 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd44310640 1 -- 172.21.15.107:0/3890672902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd3c0746e0 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.823+0000 7fbd32ffd640 1 -- 172.21.15.107:0/3890672902 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbd380b7ce0 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.827+0000 7fbd32ffd640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd20065e60 0x7fbd20068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:06.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.827+0000 7fbd418ab640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd20065e60 0x7fbd20068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:06.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.827+0000 7fbd32ffd640 1 -- 172.21.15.107:0/3890672902 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7fbd3811deb0 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.827+0000 7fbd418ab640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd20065e60 0x7fbd20068320 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c007fd0 tx=0x7fbd2c0079a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:06.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.831+0000 7fbd32ffd640 1 -- 172.21.15.107:0/3890672902 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbd380eb5a0 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.971+0000 7fbd44310640 1 -- 172.21.15.107:0/3890672902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd3c1ed7a0 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd32ffd640 1 -- 172.21.15.107:0/3890672902 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+17839 (secure 0 0 0) 0x7fbd3c1ed7a0 con 0x7fbd3c0c20b0 2023-10-24T23:53:06.978 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:06.978 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:03.684277+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:06.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 -- 172.21.15.107:0/3890672902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd20065e60 msgr2=0x7fbd20068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd20065e60 0x7fbd20068320 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c007fd0 tx=0x7fbd2c0079a0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 -- 172.21.15.107:0/3890672902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3c0c20b0 msgr2=0x7fbd3c144f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:06.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3c0c20b0 0x7fbd3c144f70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbd380ae040 tx=0x7fbd3801ab30 comp rx=0 tx=0).stop 2023-10-24T23:53:06.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 -- 172.21.15.107:0/3890672902 shutdown_connections 2023-10-24T23:53:06.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd20065e60 0x7fbd20068320 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd3c152060 0x7fbd3c0c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd3c0c2990 0x7fbd3c1ed260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 --2- 172.21.15.107:0/3890672902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3c0c20b0 0x7fbd3c144f70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:06.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 -- 172.21.15.107:0/3890672902 >> 172.21.15.107:0/3890672902 conn(0x7fbd3c0c4a90 msgr2=0x7fbd3c0ad160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:06.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 -- 172.21.15.107:0/3890672902 shutdown_connections 2023-10-24T23:53:06.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:06.975+0000 7fbd44310640 1 -- 172.21.15.107:0/3890672902 wait complete. 2023-10-24T23:53:06.996 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-24T23:53:06.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a097c2640 1 Processor -- start 2023-10-24T23:53:07.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a097c2640 1 -- start start 2023-10-24T23:53:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a097c2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6a0414fd40 0x7f6a040ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a097c2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6a040a9820 0x7f6a040aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a097c2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0406f730 con 0x7f6a0406ee40 2023-10-24T23:53:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a097c2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a040aa750 con 0x7f6a040a9820 2023-10-24T23:53:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a097c2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a04156a40 con 0x7f6a0414fd40 2023-10-24T23:53:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a027fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6a040a9820 0x7f6a040aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a037fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6a0406ee40 0x7f6a040c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39172/0 (socket says 172.21.15.107:39172) 2023-10-24T23:53:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a037fe640 1 -- 172.21.15.107:0/554698228 learned_addr learned my addr 172.21.15.107:0/554698228 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a02ffd640 1 --2- 172.21.15.107:0/554698228 >> v2:172.21.15.110:3300/0 conn(0x7f6a0414fd40 0x7f6a040ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a027fc640 1 -- 172.21.15.107:0/554698228 >> v2:172.21.15.110:3300/0 conn(0x7f6a0414fd40 msgr2=0x7f6a040ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a027fc640 1 --2- 172.21.15.107:0/554698228 >> v2:172.21.15.110:3300/0 conn(0x7f6a0414fd40 0x7f6a040ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a027fc640 1 -- 172.21.15.107:0/554698228 >> v1:172.21.15.107:6789/0 conn(0x7f6a0406ee40 legacy=0x7f6a040c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a027fc640 1 -- 172.21.15.107:0/554698228 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a04156710 con 0x7f6a040a9820 2023-10-24T23:53:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.099+0000 7f6a01ffb640 1 -- 172.21.15.107:0/554698228 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474263680 0 0) 0x7f6a0406f730 con 0x7f6a0406ee40 2023-10-24T23:53:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a027fc640 1 --2- 172.21.15.107:0/554698228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a040a9820 0x7f6a040aa210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6a040c28b0 tx=0x7f69f801c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0964bce54400c0d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a01ffb640 1 -- 172.21.15.107:0/554698228 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f80ab600 con 0x7f6a040a9820 2023-10-24T23:53:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a01ffb640 1 -- 172.21.15.107:0/554698228 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f80a7050 con 0x7f6a040a9820 2023-10-24T23:53:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a01ffb640 1 -- 172.21.15.107:0/554698228 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f80ab980 con 0x7f6a040a9820 2023-10-24T23:53:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a097c2640 1 -- 172.21.15.107:0/554698228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a040a9820 msgr2=0x7f6a040aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a097c2640 1 --2- 172.21.15.107:0/554698228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a040a9820 0x7f6a040aa210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6a040c28b0 tx=0x7f69f801c310 comp rx=0 tx=0).stop 2023-10-24T23:53:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a097c2640 1 -- 172.21.15.107:0/554698228 shutdown_connections 2023-10-24T23:53:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a097c2640 1 --2- 172.21.15.107:0/554698228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a040a9820 0x7f6a040aa210 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a097c2640 1 --2- 172.21.15.107:0/554698228 >> v2:172.21.15.110:3300/0 conn(0x7f6a0414fd40 0x7f6a040ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a097c2640 1 -- 172.21.15.107:0/554698228 >> 172.21.15.107:0/554698228 conn(0x7f6a040c2950 msgr2=0x7f6a040c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a097c2640 1 -- 172.21.15.107:0/554698228 shutdown_connections 2023-10-24T23:53:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a097c2640 1 -- 172.21.15.107:0/554698228 wait complete. 2023-10-24T23:53:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a097c2640 1 Processor -- start 2023-10-24T23:53:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.103+0000 7f6a097c2640 1 -- start start 2023-10-24T23:53:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a097c2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a0406ee40 0x7f6a041ea9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a097c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a0414fd40 0x7f6a0414bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a097c2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a04148840 0x7f6a04153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a097c2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0414d010 con 0x7f6a0414fd40 2023-10-24T23:53:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a097c2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0414f080 con 0x7f6a04148840 2023-10-24T23:53:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a097c2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a041eaf00 con 0x7f6a0406ee40 2023-10-24T23:53:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a027fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a0414fd40 0x7f6a0414bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a027fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a0414fd40 0x7f6a0414bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43420/0 (socket says 172.21.15.107:43420) 2023-10-24T23:53:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a027fc640 1 -- 172.21.15.107:0/4246209672 learned_addr learned my addr 172.21.15.107:0/4246209672 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a02ffd640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a0406ee40 0x7f6a041ea9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a037fe640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a04148840 0x7f6a04153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a027fc640 1 -- 172.21.15.107:0/4246209672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a0406ee40 msgr2=0x7f6a041ea9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a027fc640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a0406ee40 0x7f6a041ea9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a027fc640 1 -- 172.21.15.107:0/4246209672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a04148840 msgr2=0x7f6a04153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a027fc640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a04148840 0x7f6a04153b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a027fc640 1 -- 172.21.15.107:0/4246209672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f8009420 con 0x7f6a0414fd40 2023-10-24T23:53:07.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a02ffd640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a0406ee40 0x7f6a041ea9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:07.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a027fc640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a0414fd40 0x7f6a0414bf00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f69f80ae040 tx=0x7f69f80190b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:07.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f69eb7fe640 1 -- 172.21.15.107:0/4246209672 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f80b4840 con 0x7f6a0414fd40 2023-10-24T23:53:07.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f69eb7fe640 1 -- 172.21.15.107:0/4246209672 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f80b49c0 con 0x7f6a0414fd40 2023-10-24T23:53:07.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a097c2640 1 -- 172.21.15.107:0/4246209672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a04155ba0 con 0x7f6a0414fd40 2023-10-24T23:53:07.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f69eb7fe640 1 -- 172.21.15.107:0/4246209672 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f80b4d70 con 0x7f6a0414fd40 2023-10-24T23:53:07.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a097c2640 1 -- 172.21.15.107:0/4246209672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a041eeae0 con 0x7f6a0414fd40 2023-10-24T23:53:07.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a097c2640 1 -- 172.21.15.107:0/4246209672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a040c24a0 con 0x7f6a0414fd40 2023-10-24T23:53:07.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f69eb7fe640 1 -- 172.21.15.107:0/4246209672 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f69f80b2040 con 0x7f6a0414fd40 2023-10-24T23:53:07.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f69eb7fe640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69f0065e60 0x7f69f0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a02ffd640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69f0065e60 0x7f69f0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f69eb7fe640 1 -- 172.21.15.107:0/4246209672 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f69f811db40 con 0x7f6a0414fd40 2023-10-24T23:53:07.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.107+0000 7f6a02ffd640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69f0065e60 0x7f69f0068320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f69f4006fd0 tx=0x7f69f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:07.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.111+0000 7f69eb7fe640 1 -- 172.21.15.107:0/4246209672 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f69f80eb230 con 0x7f6a0414fd40 2023-10-24T23:53:07.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.251+0000 7f6a097c2640 1 -- 172.21.15.107:0/4246209672 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a04070840 con 0x7f69f0065e60 2023-10-24T23:53:07.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.271+0000 7f69eb7fe640 1 -- 172.21.15.107:0/4246209672 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+271898 (secure 0 0 0) 0x7f6a04070840 con 0x7f69f0065e60 2023-10-24T23:53:07.276 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:07.278 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:07.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 -- 172.21.15.107:0/4246209672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69f0065e60 msgr2=0x7f69f0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69f0065e60 0x7f69f0068320 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f69f4006fd0 tx=0x7f69f4008040 comp rx=0 tx=0).stop 2023-10-24T23:53:07.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 -- 172.21.15.107:0/4246209672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a0414fd40 msgr2=0x7f6a0414bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a0414fd40 0x7f6a0414bf00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f69f80ae040 tx=0x7f69f80190b0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 -- 172.21.15.107:0/4246209672 shutdown_connections 2023-10-24T23:53:07.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a04148840 0x7f6a04153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a0414fd40 0x7f6a0414bf00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69f0065e60 0x7f69f0068320 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 --2- 172.21.15.107:0/4246209672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a0406ee40 0x7f6a041ea9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 -- 172.21.15.107:0/4246209672 >> 172.21.15.107:0/4246209672 conn(0x7f6a040c2950 msgr2=0x7f6a04151ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:07.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 -- 172.21.15.107:0/4246209672 shutdown_connections 2023-10-24T23:53:07.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.275+0000 7f6a097c2640 1 -- 172.21.15.107:0/4246209672 wait complete. 2023-10-24T23:53:07.293 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-24T23:53:05.275754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":234616,"kb_used_data":2616,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754740104,"statfs":{"total":773094113280,"available":772853866496,"internally_reserved":0,"allocated":2678784,"data_stored":1661968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001273"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-24T23:52:58.159478+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:52:58.159478+0000","last_peered":"2023-10-24T23:52:58.159478+0000","last_clean":"2023-10-24T23:52:58.159478+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:52:58.159478+0000","last_undegraded":"2023-10-24T23:52:58.159478+0000","last_fullsized":"2023-10-24T23:52:58.159478+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-24T23:52:59.156580+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:52:59.156580+0000","last_peered":"2023-10-24T23:52:59.156580+0000","last_clean":"2023-10-24T23:52:59.156580+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:52:59.156580+0000","last_undegraded":"2023-10-24T23:52:59.156580+0000","last_fullsized":"2023-10-24T23:52:59.156580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-24T23:53:00.063273+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:00.063273+0000","last_peered":"2023-10-24T23:53:00.063273+0000","last_clean":"2023-10-24T23:53:00.063273+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:00.063273+0000","last_undegraded":"2023-10-24T23:53:00.063273+0000","last_fullsized":"2023-10-24T23:53:00.063273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-24T23:52:59.991804+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:52:59.991804+0000","last_peered":"2023-10-24T23:52:59.991804+0000","last_clean":"2023-10-24T23:52:59.991804+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:52:59.991804+0000","last_undegraded":"2023-10-24T23:52:59.991804+0000","last_fullsized":"2023-10-24T23:52:59.991804+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-24T23:52:59.156580+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:52:59.156580+0000","last_peered":"2023-10-24T23:52:59.156580+0000","last_clean":"2023-10-24T23:52:59.156580+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:52:59.156580+0000","last_undegraded":"2023-10-24T23:52:59.156580+0000","last_fullsized":"2023-10-24T23:52:59.156580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-24T23:53:00.062187+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:00.062187+0000","last_peered":"2023-10-24T23:53:00.062187+0000","last_clean":"2023-10-24T23:53:00.062187+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:00.062187+0000","last_undegraded":"2023-10-24T23:53:00.062187+0000","last_fullsized":"2023-10-24T23:53:00.062187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-24T23:53:00.063297+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:00.063297+0000","last_peered":"2023-10-24T23:53:00.063297+0000","last_clean":"2023-10-24T23:53:00.063297+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:00.063297+0000","last_undegraded":"2023-10-24T23:53:00.063297+0000","last_fullsized":"2023-10-24T23:53:00.063297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-24T23:53:00.062221+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:00.062221+0000","last_peered":"2023-10-24T23:53:00.062221+0000","last_clean":"2023-10-24T23:53:00.062221+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:00.062221+0000","last_undegraded":"2023-10-24T23:53:00.062221+0000","last_fullsized":"2023-10-24T23:53:00.062221+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-24T23:53:00.063558+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:00.063558+0000","last_peered":"2023-10-24T23:53:00.063558+0000","last_clean":"2023-10-24T23:53:00.063558+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:00.063558+0000","last_undegraded":"2023-10-24T23:53:00.063558+0000","last_fullsized":"2023-10-24T23:53:00.063558+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:02.290575+0000","last_change":"2023-10-24T23:53:02.290575+0000","last_active":"2023-10-24T23:53:02.290575+0000","last_peered":"2023-10-24T23:53:02.290575+0000","last_clean":"2023-10-24T23:53:02.290575+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:02.290575+0000","last_undegraded":"2023-10-24T23:53:02.290575+0000","last_fullsized":"2023-10-24T23:53:02.290575+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:01.277969+0000","last_change":"2023-10-24T23:53:01.277969+0000","last_active":"2023-10-24T23:53:01.277969+0000","last_peered":"2023-10-24T23:53:01.277969+0000","last_clean":"2023-10-24T23:53:01.277969+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:01.277969+0000","last_undegraded":"2023-10-24T23:53:01.277969+0000","last_fullsized":"2023-10-24T23:53:01.277969+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":756,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":774144,"data_stored":647648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31224,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340616,"statfs":{"total":96636764160,"available":96604790784,"internally_reserved":0,"allocated":180224,"data_stored":57280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0429999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1679999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31236,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340604,"statfs":{"total":96636764160,"available":96604778496,"internally_reserved":0,"allocated":192512,"data_stored":63028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.244}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]}]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31224,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340616,"statfs":{"total":96636764160,"available":96604790784,"internally_reserved":0,"allocated":180224,"data_stored":57280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":192512,"data_stored":63028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27128,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344712,"statfs":{"total":96636764160,"available":96608985088,"internally_reserved":0,"allocated":180224,"data_stored":57280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":192512,"data_stored":63028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2929999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27720,"kb_used_data":768,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344120,"statfs":{"total":96636764160,"available":96608378880,"internally_reserved":0,"allocated":786432,"data_stored":653396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4610000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2470000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.679}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:07.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:07.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.447+0000 7fa4064f6640 1 Processor -- start 2023-10-24T23:53:07.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.447+0000 7fa4064f6640 1 -- start start 2023-10-24T23:53:07.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa4064f6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa40014fd40 0x7fa4000ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa4064f6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40006f730 con 0x7fa40006ee40 2023-10-24T23:53:07.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa4064f6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa400156a40 con 0x7fa4000a9820 2023-10-24T23:53:07.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa4064f6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa400156710 con 0x7fa40014fd40 2023-10-24T23:53:07.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa3fffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa40014fd40 0x7fa4000ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa404a93640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa40006ee40 0x7fa40015cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39174/0 (socket says 172.21.15.107:39174) 2023-10-24T23:53:07.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa404a93640 1 -- 172.21.15.107:0/2384123592 learned_addr learned my addr 172.21.15.107:0/2384123592 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:07.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa3fffff640 1 -- 172.21.15.107:0/2384123592 >> v1:172.21.15.110:6789/0 conn(0x7fa4000a9820 legacy=0x7fa4000c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:07.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa3fffff640 1 -- 172.21.15.107:0/2384123592 >> v1:172.21.15.107:6789/0 conn(0x7fa40006ee40 legacy=0x7fa40015cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa3fffff640 1 -- 172.21.15.107:0/2384123592 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40006f980 con 0x7fa40014fd40 2023-10-24T23:53:07.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa3feffd640 1 -- 172.21.15.107:0/2384123592 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817620629 0 0) 0x7fa40006f730 con 0x7fa40006ee40 2023-10-24T23:53:07.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa3fffff640 1 --2- 172.21.15.107:0/2384123592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40014fd40 0x7fa4000ab090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00097c0 tx=0x7fa3f001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51bf7a8c1d106aeb server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:07.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa3feffd640 1 -- 172.21.15.107:0/2384123592 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f00ab600 con 0x7fa40014fd40 2023-10-24T23:53:07.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa3feffd640 1 -- 172.21.15.107:0/2384123592 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f00a7050 con 0x7fa40014fd40 2023-10-24T23:53:07.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa3feffd640 1 -- 172.21.15.107:0/2384123592 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f00ab980 con 0x7fa40014fd40 2023-10-24T23:53:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa4064f6640 1 -- 172.21.15.107:0/2384123592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40014fd40 msgr2=0x7fa4000ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa4064f6640 1 --2- 172.21.15.107:0/2384123592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40014fd40 0x7fa4000ab090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00097c0 tx=0x7fa3f001c320 comp rx=0 tx=0).stop 2023-10-24T23:53:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa4064f6640 1 -- 172.21.15.107:0/2384123592 shutdown_connections 2023-10-24T23:53:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa4064f6640 1 --2- 172.21.15.107:0/2384123592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40014fd40 0x7fa4000ab090 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa4064f6640 1 -- 172.21.15.107:0/2384123592 >> 172.21.15.107:0/2384123592 conn(0x7fa4000c2950 msgr2=0x7fa4000c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:07.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa4064f6640 1 -- 172.21.15.107:0/2384123592 shutdown_connections 2023-10-24T23:53:07.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.451+0000 7fa4064f6640 1 -- 172.21.15.107:0/2384123592 wait complete. 2023-10-24T23:53:07.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa4064f6640 1 Processor -- start 2023-10-24T23:53:07.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa4064f6640 1 -- start start 2023-10-24T23:53:07.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa4064f6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa40006ee40 0x7fa400148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa4064f6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40014fd40 0x7fa4001eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa4064f6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa4001ee5d0 0x7fa4001ebb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa4064f6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4001eeb40 con 0x7fa40014fd40 2023-10-24T23:53:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa4064f6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4001eee20 con 0x7fa40006ee40 2023-10-24T23:53:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa4064f6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4001ef100 con 0x7fa4001ee5d0 2023-10-24T23:53:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3ff7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40014fd40 0x7fa4001eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3ff7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40014fd40 0x7fa4001eb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43436/0 (socket says 172.21.15.107:43436) 2023-10-24T23:53:07.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3ff7fe640 1 -- 172.21.15.107:0/2855909988 learned_addr learned my addr 172.21.15.107:0/2855909988 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:07.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3ff7fe640 1 -- 172.21.15.107:0/2855909988 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa4001ee5d0 msgr2=0x7fa4001ebb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3ff7fe640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa4001ee5d0 0x7fa4001ebb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3ff7fe640 1 -- 172.21.15.107:0/2855909988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa40006ee40 msgr2=0x7fa400148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3ff7fe640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa40006ee40 0x7fa400148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3ff7fe640 1 -- 172.21.15.107:0/2855909988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f0009420 con 0x7fa40014fd40 2023-10-24T23:53:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3ff7fe640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40014fd40 0x7fa4001eb5f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400c490 tx=0x7fa3f4001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3fcff9640 1 -- 172.21.15.107:0/2855909988 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4003600 con 0x7fa40014fd40 2023-10-24T23:53:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3fcff9640 1 -- 172.21.15.107:0/2855909988 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f4003780 con 0x7fa40014fd40 2023-10-24T23:53:07.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa4064f6640 1 -- 172.21.15.107:0/2855909988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4001ef3e0 con 0x7fa40014fd40 2023-10-24T23:53:07.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3fcff9640 1 -- 172.21.15.107:0/2855909988 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4003b70 con 0x7fa40014fd40 2023-10-24T23:53:07.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa4064f6640 1 -- 172.21.15.107:0/2855909988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa400149240 con 0x7fa40014fd40 2023-10-24T23:53:07.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa4064f6640 1 -- 172.21.15.107:0/2855909988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4000c24a0 con 0x7fa40014fd40 2023-10-24T23:53:07.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.455+0000 7fa3fcff9640 1 -- 172.21.15.107:0/2855909988 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa3f4003cd0 con 0x7fa40014fd40 2023-10-24T23:53:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.459+0000 7fa3fcff9640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3ec065e60 0x7fa3ec068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.459+0000 7fa3fffff640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3ec065e60 0x7fa3ec068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.459+0000 7fa3fcff9640 1 -- 172.21.15.107:0/2855909988 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7fa3f4082a60 con 0x7fa40014fd40 2023-10-24T23:53:07.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.459+0000 7fa3fffff640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3ec065e60 0x7fa3ec068320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00ad000 tx=0x7fa3f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:07.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.463+0000 7fa3fcff9640 1 -- 172.21.15.107:0/2855909988 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa3f40500c0 con 0x7fa40014fd40 2023-10-24T23:53:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.603+0000 7fa4064f6640 1 -- 172.21.15.107:0/2855909988 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4000aa620 con 0x7fa3ec065e60 2023-10-24T23:53:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.611+0000 7fa3fcff9640 1 -- 172.21.15.107:0/2855909988 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286664 (secure 0 0 0) 0x7fa3f00ab6a0 con 0x7fa3ec065e60 2023-10-24T23:53:07.615 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.611+0000 7fa4064f6640 1 -- 172.21.15.107:0/2855909988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3ec065e60 msgr2=0x7fa3ec068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.611+0000 7fa4064f6640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3ec065e60 0x7fa3ec068320 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00ad000 tx=0x7fa3f0008040 comp rx=0 tx=0).stop 2023-10-24T23:53:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.615+0000 7fa4064f6640 1 -- 172.21.15.107:0/2855909988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40014fd40 msgr2=0x7fa4001eb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.615+0000 7fa4064f6640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40014fd40 0x7fa4001eb5f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400c490 tx=0x7fa3f4001e80 comp rx=0 tx=0).stop 2023-10-24T23:53:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.615+0000 7fa4064f6640 1 -- 172.21.15.107:0/2855909988 shutdown_connections 2023-10-24T23:53:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.615+0000 7fa4064f6640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3ec065e60 0x7fa3ec068320 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.615+0000 7fa4064f6640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa4001ee5d0 0x7fa4001ebb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.615+0000 7fa4064f6640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa40014fd40 0x7fa4001eb5f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.615+0000 7fa4064f6640 1 --2- 172.21.15.107:0/2855909988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa40006ee40 0x7fa400148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.615+0000 7fa4064f6640 1 -- 172.21.15.107:0/2855909988 >> 172.21.15.107:0/2855909988 conn(0x7fa4000c2950 msgr2=0x7fa4000c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.615+0000 7fa4064f6640 1 -- 172.21.15.107:0/2855909988 shutdown_connections 2023-10-24T23:53:07.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.615+0000 7fa4064f6640 1 -- 172.21.15.107:0/2855909988 wait complete. 2023-10-24T23:53:07.631 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-24T23:53:07.276470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235112,"kb_used_data":3112,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754739608,"statfs":{"total":773094113280,"available":772853358592,"internally_reserved":0,"allocated":3186688,"data_stored":1901612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.985219"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696971+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-24T23:53:03.696971+0000","last_peered":"2023-10-24T23:53:03.696971+0000","last_clean":"2023-10-24T23:53:03.696971+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-24T23:53:03.696971+0000","last_undegraded":"2023-10-24T23:53:03.696971+0000","last_fullsized":"2023-10-24T23:53:03.696971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699801+0000","last_change":"2023-10-24T23:53:02.715966+0000","last_active":"2023-10-24T23:53:03.699801+0000","last_peered":"2023-10-24T23:53:03.699801+0000","last_clean":"2023-10-24T23:53:03.699801+0000","last_became_active":"2023-10-24T23:53:02.715631+0000","last_became_peered":"2023-10-24T23:53:02.715631+0000","last_unstale":"2023-10-24T23:53:03.699801+0000","last_undegraded":"2023-10-24T23:53:03.699801+0000","last_fullsized":"2023-10-24T23:53:03.699801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:11.771864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499435+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-24T23:53:04.499435+0000","last_peered":"2023-10-24T23:53:04.499435+0000","last_clean":"2023-10-24T23:53:04.499435+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-24T23:53:04.499435+0000","last_undegraded":"2023-10-24T23:53:04.499435+0000","last_fullsized":"2023-10-24T23:53:04.499435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697967+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-24T23:53:03.697967+0000","last_peered":"2023-10-24T23:53:03.697967+0000","last_clean":"2023-10-24T23:53:03.697967+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-24T23:53:03.697967+0000","last_undegraded":"2023-10-24T23:53:03.697967+0000","last_fullsized":"2023-10-24T23:53:03.697967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497236+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-24T23:53:04.497236+0000","last_peered":"2023-10-24T23:53:04.497236+0000","last_clean":"2023-10-24T23:53:04.497236+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-24T23:53:04.497236+0000","last_undegraded":"2023-10-24T23:53:04.497236+0000","last_fullsized":"2023-10-24T23:53:04.497236+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498842+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-24T23:53:04.498842+0000","last_peered":"2023-10-24T23:53:04.498842+0000","last_clean":"2023-10-24T23:53:04.498842+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-24T23:53:04.498842+0000","last_undegraded":"2023-10-24T23:53:04.498842+0000","last_fullsized":"2023-10-24T23:53:04.498842+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.061899+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-24T23:53:04.061899+0000","last_peered":"2023-10-24T23:53:04.061899+0000","last_clean":"2023-10-24T23:53:04.061899+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-24T23:53:04.061899+0000","last_undegraded":"2023-10-24T23:53:04.061899+0000","last_fullsized":"2023-10-24T23:53:04.061899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698958+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-24T23:53:03.698958+0000","last_peered":"2023-10-24T23:53:03.698958+0000","last_clean":"2023-10-24T23:53:03.698958+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-24T23:53:03.698958+0000","last_undegraded":"2023-10-24T23:53:03.698958+0000","last_fullsized":"2023-10-24T23:53:03.698958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497251+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-24T23:53:04.497251+0000","last_peered":"2023-10-24T23:53:04.497251+0000","last_clean":"2023-10-24T23:53:04.497251+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-24T23:53:04.497251+0000","last_undegraded":"2023-10-24T23:53:04.497251+0000","last_fullsized":"2023-10-24T23:53:04.497251+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699835+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-24T23:53:03.699835+0000","last_peered":"2023-10-24T23:53:03.699835+0000","last_clean":"2023-10-24T23:53:03.699835+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-24T23:53:03.699835+0000","last_undegraded":"2023-10-24T23:53:03.699835+0000","last_fullsized":"2023-10-24T23:53:03.699835+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499695+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-24T23:53:04.499695+0000","last_peered":"2023-10-24T23:53:04.499695+0000","last_clean":"2023-10-24T23:53:04.499695+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-24T23:53:04.499695+0000","last_undegraded":"2023-10-24T23:53:04.499695+0000","last_fullsized":"2023-10-24T23:53:04.499695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696679+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-24T23:53:03.696679+0000","last_peered":"2023-10-24T23:53:03.696679+0000","last_clean":"2023-10-24T23:53:03.696679+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-24T23:53:03.696679+0000","last_undegraded":"2023-10-24T23:53:03.696679+0000","last_fullsized":"2023-10-24T23:53:03.696679+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696626+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-24T23:53:03.696626+0000","last_peered":"2023-10-24T23:53:03.696626+0000","last_clean":"2023-10-24T23:53:03.696626+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-24T23:53:03.696626+0000","last_undegraded":"2023-10-24T23:53:03.696626+0000","last_fullsized":"2023-10-24T23:53:03.696626+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699415+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-24T23:53:03.699415+0000","last_peered":"2023-10-24T23:53:03.699415+0000","last_clean":"2023-10-24T23:53:03.699415+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-24T23:53:03.699415+0000","last_undegraded":"2023-10-24T23:53:03.699415+0000","last_fullsized":"2023-10-24T23:53:03.699415+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062078+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-24T23:53:04.062078+0000","last_peered":"2023-10-24T23:53:04.062078+0000","last_clean":"2023-10-24T23:53:04.062078+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-24T23:53:04.062078+0000","last_undegraded":"2023-10-24T23:53:04.062078+0000","last_fullsized":"2023-10-24T23:53:04.062078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498748+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-24T23:53:04.498748+0000","last_peered":"2023-10-24T23:53:04.498748+0000","last_clean":"2023-10-24T23:53:04.498748+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-24T23:53:04.498748+0000","last_undegraded":"2023-10-24T23:53:04.498748+0000","last_fullsized":"2023-10-24T23:53:04.498748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497346+0000","last_change":"2023-10-24T23:53:02.728971+0000","last_active":"2023-10-24T23:53:04.497346+0000","last_peered":"2023-10-24T23:53:04.497346+0000","last_clean":"2023-10-24T23:53:04.497346+0000","last_became_active":"2023-10-24T23:53:02.728048+0000","last_became_peered":"2023-10-24T23:53:02.728048+0000","last_unstale":"2023-10-24T23:53:04.497346+0000","last_undegraded":"2023-10-24T23:53:04.497346+0000","last_fullsized":"2023-10-24T23:53:04.497346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:31:32.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499911+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-24T23:53:04.499911+0000","last_peered":"2023-10-24T23:53:04.499911+0000","last_clean":"2023-10-24T23:53:04.499911+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-24T23:53:04.499911+0000","last_undegraded":"2023-10-24T23:53:04.499911+0000","last_fullsized":"2023-10-24T23:53:04.499911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696750+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-24T23:53:03.696750+0000","last_peered":"2023-10-24T23:53:03.696750+0000","last_clean":"2023-10-24T23:53:03.696750+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-24T23:53:03.696750+0000","last_undegraded":"2023-10-24T23:53:03.696750+0000","last_fullsized":"2023-10-24T23:53:03.696750+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499118+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-24T23:53:04.499118+0000","last_peered":"2023-10-24T23:53:04.499118+0000","last_clean":"2023-10-24T23:53:04.499118+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-24T23:53:04.499118+0000","last_undegraded":"2023-10-24T23:53:04.499118+0000","last_fullsized":"2023-10-24T23:53:04.499118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062078+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-24T23:53:04.062078+0000","last_peered":"2023-10-24T23:53:04.062078+0000","last_clean":"2023-10-24T23:53:04.062078+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-24T23:53:04.062078+0000","last_undegraded":"2023-10-24T23:53:04.062078+0000","last_fullsized":"2023-10-24T23:53:04.062078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496931+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-24T23:53:04.496931+0000","last_peered":"2023-10-24T23:53:04.496931+0000","last_clean":"2023-10-24T23:53:04.496931+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-24T23:53:04.496931+0000","last_undegraded":"2023-10-24T23:53:04.496931+0000","last_fullsized":"2023-10-24T23:53:04.496931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165275+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-24T23:53:04.165275+0000","last_peered":"2023-10-24T23:53:04.165275+0000","last_clean":"2023-10-24T23:53:04.165275+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-24T23:53:04.165275+0000","last_undegraded":"2023-10-24T23:53:04.165275+0000","last_fullsized":"2023-10-24T23:53:04.165275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.061804+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-24T23:53:04.061804+0000","last_peered":"2023-10-24T23:53:04.061804+0000","last_clean":"2023-10-24T23:53:04.061804+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-24T23:53:04.061804+0000","last_undegraded":"2023-10-24T23:53:04.061804+0000","last_fullsized":"2023-10-24T23:53:04.061804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497069+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-24T23:53:04.497069+0000","last_peered":"2023-10-24T23:53:04.497069+0000","last_clean":"2023-10-24T23:53:04.497069+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-24T23:53:04.497069+0000","last_undegraded":"2023-10-24T23:53:04.497069+0000","last_fullsized":"2023-10-24T23:53:04.497069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497056+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-24T23:53:04.497056+0000","last_peered":"2023-10-24T23:53:04.497056+0000","last_clean":"2023-10-24T23:53:04.497056+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-24T23:53:04.497056+0000","last_undegraded":"2023-10-24T23:53:04.497056+0000","last_fullsized":"2023-10-24T23:53:04.497056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699631+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-24T23:53:03.699631+0000","last_peered":"2023-10-24T23:53:03.699631+0000","last_clean":"2023-10-24T23:53:03.699631+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-24T23:53:03.699631+0000","last_undegraded":"2023-10-24T23:53:03.699631+0000","last_fullsized":"2023-10-24T23:53:03.699631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497047+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-24T23:53:04.497047+0000","last_peered":"2023-10-24T23:53:04.497047+0000","last_clean":"2023-10-24T23:53:04.497047+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-24T23:53:04.497047+0000","last_undegraded":"2023-10-24T23:53:04.497047+0000","last_fullsized":"2023-10-24T23:53:04.497047+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165731+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-24T23:53:04.165731+0000","last_peered":"2023-10-24T23:53:04.165731+0000","last_clean":"2023-10-24T23:53:04.165731+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-24T23:53:04.165731+0000","last_undegraded":"2023-10-24T23:53:04.165731+0000","last_fullsized":"2023-10-24T23:53:04.165731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499350+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-24T23:53:04.499350+0000","last_peered":"2023-10-24T23:53:04.499350+0000","last_clean":"2023-10-24T23:53:04.499350+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-24T23:53:04.499350+0000","last_undegraded":"2023-10-24T23:53:04.499350+0000","last_fullsized":"2023-10-24T23:53:04.499350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165369+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-24T23:53:04.165369+0000","last_peered":"2023-10-24T23:53:04.165369+0000","last_clean":"2023-10-24T23:53:04.165369+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-24T23:53:04.165369+0000","last_undegraded":"2023-10-24T23:53:04.165369+0000","last_fullsized":"2023-10-24T23:53:04.165369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698525+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-24T23:53:03.698525+0000","last_peered":"2023-10-24T23:53:03.698525+0000","last_clean":"2023-10-24T23:53:03.698525+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-24T23:53:03.698525+0000","last_undegraded":"2023-10-24T23:53:03.698525+0000","last_fullsized":"2023-10-24T23:53:03.698525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166359+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-24T23:53:04.166359+0000","last_peered":"2023-10-24T23:53:04.166359+0000","last_clean":"2023-10-24T23:53:04.166359+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-24T23:53:04.166359+0000","last_undegraded":"2023-10-24T23:53:04.166359+0000","last_fullsized":"2023-10-24T23:53:04.166359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062264+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-24T23:53:04.062264+0000","last_peered":"2023-10-24T23:53:04.062264+0000","last_clean":"2023-10-24T23:53:04.062264+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-24T23:53:04.062264+0000","last_undegraded":"2023-10-24T23:53:04.062264+0000","last_fullsized":"2023-10-24T23:53:04.062264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699735+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-24T23:53:03.699735+0000","last_peered":"2023-10-24T23:53:03.699735+0000","last_clean":"2023-10-24T23:53:03.699735+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-24T23:53:03.699735+0000","last_undegraded":"2023-10-24T23:53:03.699735+0000","last_fullsized":"2023-10-24T23:53:03.699735+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497015+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-24T23:53:04.497015+0000","last_peered":"2023-10-24T23:53:04.497015+0000","last_clean":"2023-10-24T23:53:04.497015+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-24T23:53:04.497015+0000","last_undegraded":"2023-10-24T23:53:04.497015+0000","last_fullsized":"2023-10-24T23:53:04.497015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497582+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-24T23:53:04.497582+0000","last_peered":"2023-10-24T23:53:04.497582+0000","last_clean":"2023-10-24T23:53:04.497582+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-24T23:53:04.497582+0000","last_undegraded":"2023-10-24T23:53:04.497582+0000","last_fullsized":"2023-10-24T23:53:04.497582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062154+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-24T23:53:04.062154+0000","last_peered":"2023-10-24T23:53:04.062154+0000","last_clean":"2023-10-24T23:53:04.062154+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-24T23:53:04.062154+0000","last_undegraded":"2023-10-24T23:53:04.062154+0000","last_fullsized":"2023-10-24T23:53:04.062154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166270+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-24T23:53:04.166270+0000","last_peered":"2023-10-24T23:53:04.166270+0000","last_clean":"2023-10-24T23:53:04.166270+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-24T23:53:04.166270+0000","last_undegraded":"2023-10-24T23:53:04.166270+0000","last_fullsized":"2023-10-24T23:53:04.166270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697647+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-24T23:53:03.697647+0000","last_peered":"2023-10-24T23:53:03.697647+0000","last_clean":"2023-10-24T23:53:03.697647+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-24T23:53:03.697647+0000","last_undegraded":"2023-10-24T23:53:03.697647+0000","last_fullsized":"2023-10-24T23:53:03.697647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.063138+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-24T23:53:04.063138+0000","last_peered":"2023-10-24T23:53:04.063138+0000","last_clean":"2023-10-24T23:53:04.063138+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-24T23:53:04.063138+0000","last_undegraded":"2023-10-24T23:53:04.063138+0000","last_fullsized":"2023-10-24T23:53:04.063138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698927+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-24T23:53:03.698927+0000","last_peered":"2023-10-24T23:53:03.698927+0000","last_clean":"2023-10-24T23:53:03.698927+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-24T23:53:03.698927+0000","last_undegraded":"2023-10-24T23:53:03.698927+0000","last_fullsized":"2023-10-24T23:53:03.698927+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.063027+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-24T23:53:04.063027+0000","last_peered":"2023-10-24T23:53:04.063027+0000","last_clean":"2023-10-24T23:53:04.063027+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-24T23:53:04.063027+0000","last_undegraded":"2023-10-24T23:53:04.063027+0000","last_fullsized":"2023-10-24T23:53:04.063027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499840+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-24T23:53:04.499840+0000","last_peered":"2023-10-24T23:53:04.499840+0000","last_clean":"2023-10-24T23:53:04.499840+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-24T23:53:04.499840+0000","last_undegraded":"2023-10-24T23:53:04.499840+0000","last_fullsized":"2023-10-24T23:53:04.499840+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062733+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-24T23:53:04.062733+0000","last_peered":"2023-10-24T23:53:04.062733+0000","last_clean":"2023-10-24T23:53:04.062733+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-24T23:53:04.062733+0000","last_undegraded":"2023-10-24T23:53:04.062733+0000","last_fullsized":"2023-10-24T23:53:04.062733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697568+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-24T23:53:03.697568+0000","last_peered":"2023-10-24T23:53:03.697568+0000","last_clean":"2023-10-24T23:53:03.697568+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-24T23:53:03.697568+0000","last_undegraded":"2023-10-24T23:53:03.697568+0000","last_fullsized":"2023-10-24T23:53:03.697568+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699348+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-24T23:53:03.699348+0000","last_peered":"2023-10-24T23:53:03.699348+0000","last_clean":"2023-10-24T23:53:03.699348+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-24T23:53:03.699348+0000","last_undegraded":"2023-10-24T23:53:03.699348+0000","last_fullsized":"2023-10-24T23:53:03.699348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496747+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-24T23:53:04.496747+0000","last_peered":"2023-10-24T23:53:04.496747+0000","last_clean":"2023-10-24T23:53:04.496747+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-24T23:53:04.496747+0000","last_undegraded":"2023-10-24T23:53:04.496747+0000","last_fullsized":"2023-10-24T23:53:04.496747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165994+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-24T23:53:04.165994+0000","last_peered":"2023-10-24T23:53:04.165994+0000","last_clean":"2023-10-24T23:53:04.165994+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-24T23:53:04.165994+0000","last_undegraded":"2023-10-24T23:53:04.165994+0000","last_fullsized":"2023-10-24T23:53:04.165994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499931+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-24T23:53:04.499931+0000","last_peered":"2023-10-24T23:53:04.499931+0000","last_clean":"2023-10-24T23:53:04.499931+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-24T23:53:04.499931+0000","last_undegraded":"2023-10-24T23:53:04.499931+0000","last_fullsized":"2023-10-24T23:53:04.499931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698204+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-24T23:53:03.698204+0000","last_peered":"2023-10-24T23:53:03.698204+0000","last_clean":"2023-10-24T23:53:03.698204+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-24T23:53:03.698204+0000","last_undegraded":"2023-10-24T23:53:03.698204+0000","last_fullsized":"2023-10-24T23:53:03.698204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499791+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-24T23:53:04.499791+0000","last_peered":"2023-10-24T23:53:04.499791+0000","last_clean":"2023-10-24T23:53:04.499791+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-24T23:53:04.499791+0000","last_undegraded":"2023-10-24T23:53:04.499791+0000","last_fullsized":"2023-10-24T23:53:04.499791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499622+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-24T23:53:04.499622+0000","last_peered":"2023-10-24T23:53:04.499622+0000","last_clean":"2023-10-24T23:53:04.499622+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-24T23:53:04.499622+0000","last_undegraded":"2023-10-24T23:53:04.499622+0000","last_fullsized":"2023-10-24T23:53:04.499622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499944+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-24T23:53:04.499944+0000","last_peered":"2023-10-24T23:53:04.499944+0000","last_clean":"2023-10-24T23:53:04.499944+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-24T23:53:04.499944+0000","last_undegraded":"2023-10-24T23:53:04.499944+0000","last_fullsized":"2023-10-24T23:53:04.499944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165664+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-24T23:53:04.165664+0000","last_peered":"2023-10-24T23:53:04.165664+0000","last_clean":"2023-10-24T23:53:04.165664+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-24T23:53:04.165664+0000","last_undegraded":"2023-10-24T23:53:04.165664+0000","last_fullsized":"2023-10-24T23:53:04.165664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497591+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-24T23:53:04.497591+0000","last_peered":"2023-10-24T23:53:04.497591+0000","last_clean":"2023-10-24T23:53:04.497591+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-24T23:53:04.497591+0000","last_undegraded":"2023-10-24T23:53:04.497591+0000","last_fullsized":"2023-10-24T23:53:04.497591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.713291+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:53:02.713291+0000","last_peered":"2023-10-24T23:53:02.713291+0000","last_clean":"2023-10-24T23:53:02.713291+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:53:02.713291+0000","last_undegraded":"2023-10-24T23:53:02.713291+0000","last_fullsized":"2023-10-24T23:53:02.713291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.689415+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:53:02.689415+0000","last_peered":"2023-10-24T23:53:02.689415+0000","last_clean":"2023-10-24T23:53:02.689415+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:53:02.689415+0000","last_undegraded":"2023-10-24T23:53:02.689415+0000","last_fullsized":"2023-10-24T23:53:02.689415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165624+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-24T23:53:04.165624+0000","last_peered":"2023-10-24T23:53:04.165624+0000","last_clean":"2023-10-24T23:53:04.165624+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-24T23:53:04.165624+0000","last_undegraded":"2023-10-24T23:53:04.165624+0000","last_fullsized":"2023-10-24T23:53:04.165624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698126+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-24T23:53:03.698126+0000","last_peered":"2023-10-24T23:53:03.698126+0000","last_clean":"2023-10-24T23:53:03.698126+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-24T23:53:03.698126+0000","last_undegraded":"2023-10-24T23:53:03.698126+0000","last_fullsized":"2023-10-24T23:53:03.698126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.713207+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:02.713207+0000","last_peered":"2023-10-24T23:53:02.713207+0000","last_clean":"2023-10-24T23:53:02.713207+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:02.713207+0000","last_undegraded":"2023-10-24T23:53:02.713207+0000","last_fullsized":"2023-10-24T23:53:02.713207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496751+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-24T23:53:04.496751+0000","last_peered":"2023-10-24T23:53:04.496751+0000","last_clean":"2023-10-24T23:53:04.496751+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-24T23:53:04.496751+0000","last_undegraded":"2023-10-24T23:53:04.496751+0000","last_fullsized":"2023-10-24T23:53:04.496751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497538+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-24T23:53:04.497538+0000","last_peered":"2023-10-24T23:53:04.497538+0000","last_clean":"2023-10-24T23:53:04.497538+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-24T23:53:04.497538+0000","last_undegraded":"2023-10-24T23:53:04.497538+0000","last_fullsized":"2023-10-24T23:53:04.497538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.688268+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:53:02.688268+0000","last_peered":"2023-10-24T23:53:02.688268+0000","last_clean":"2023-10-24T23:53:02.688268+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:53:02.688268+0000","last_undegraded":"2023-10-24T23:53:02.688268+0000","last_fullsized":"2023-10-24T23:53:02.688268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496567+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-24T23:53:04.496567+0000","last_peered":"2023-10-24T23:53:04.496567+0000","last_clean":"2023-10-24T23:53:04.496567+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-24T23:53:04.496567+0000","last_undegraded":"2023-10-24T23:53:04.496567+0000","last_fullsized":"2023-10-24T23:53:04.496567+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497633+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-24T23:53:04.497633+0000","last_peered":"2023-10-24T23:53:04.497633+0000","last_clean":"2023-10-24T23:53:04.497633+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-24T23:53:04.497633+0000","last_undegraded":"2023-10-24T23:53:04.497633+0000","last_fullsized":"2023-10-24T23:53:04.497633+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.691164+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:53:02.691164+0000","last_peered":"2023-10-24T23:53:02.691164+0000","last_clean":"2023-10-24T23:53:02.691164+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:53:02.691164+0000","last_undegraded":"2023-10-24T23:53:02.691164+0000","last_fullsized":"2023-10-24T23:53:02.691164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499793+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-24T23:53:04.499793+0000","last_peered":"2023-10-24T23:53:04.499793+0000","last_clean":"2023-10-24T23:53:04.499793+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-24T23:53:04.499793+0000","last_undegraded":"2023-10-24T23:53:04.499793+0000","last_fullsized":"2023-10-24T23:53:04.499793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496956+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-24T23:53:04.496956+0000","last_peered":"2023-10-24T23:53:04.496956+0000","last_clean":"2023-10-24T23:53:04.496956+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-24T23:53:04.496956+0000","last_undegraded":"2023-10-24T23:53:04.496956+0000","last_fullsized":"2023-10-24T23:53:04.496956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699976+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-24T23:53:03.699976+0000","last_peered":"2023-10-24T23:53:03.699976+0000","last_clean":"2023-10-24T23:53:03.699976+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-24T23:53:03.699976+0000","last_undegraded":"2023-10-24T23:53:03.699976+0000","last_fullsized":"2023-10-24T23:53:03.699976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.710865+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:02.710865+0000","last_peered":"2023-10-24T23:53:02.710865+0000","last_clean":"2023-10-24T23:53:02.710865+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:02.710865+0000","last_undegraded":"2023-10-24T23:53:02.710865+0000","last_fullsized":"2023-10-24T23:53:02.710865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496893+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-24T23:53:04.496893+0000","last_peered":"2023-10-24T23:53:04.496893+0000","last_clean":"2023-10-24T23:53:04.496893+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-24T23:53:04.496893+0000","last_undegraded":"2023-10-24T23:53:04.496893+0000","last_fullsized":"2023-10-24T23:53:04.496893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497005+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-24T23:53:04.497005+0000","last_peered":"2023-10-24T23:53:04.497005+0000","last_clean":"2023-10-24T23:53:04.497005+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-24T23:53:04.497005+0000","last_undegraded":"2023-10-24T23:53:04.497005+0000","last_fullsized":"2023-10-24T23:53:04.497005+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.714178+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:02.714178+0000","last_peered":"2023-10-24T23:53:02.714178+0000","last_clean":"2023-10-24T23:53:02.714178+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:02.714178+0000","last_undegraded":"2023-10-24T23:53:02.714178+0000","last_fullsized":"2023-10-24T23:53:02.714178+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497155+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-24T23:53:04.497155+0000","last_peered":"2023-10-24T23:53:04.497155+0000","last_clean":"2023-10-24T23:53:04.497155+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-24T23:53:04.497155+0000","last_undegraded":"2023-10-24T23:53:04.497155+0000","last_fullsized":"2023-10-24T23:53:04.497155+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166479+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-24T23:53:04.166479+0000","last_peered":"2023-10-24T23:53:04.166479+0000","last_clean":"2023-10-24T23:53:04.166479+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-24T23:53:04.166479+0000","last_undegraded":"2023-10-24T23:53:04.166479+0000","last_fullsized":"2023-10-24T23:53:04.166479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.705879+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:02.705879+0000","last_peered":"2023-10-24T23:53:02.705879+0000","last_clean":"2023-10-24T23:53:02.705879+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:02.705879+0000","last_undegraded":"2023-10-24T23:53:02.705879+0000","last_fullsized":"2023-10-24T23:53:02.705879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165320+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-24T23:53:04.165320+0000","last_peered":"2023-10-24T23:53:04.165320+0000","last_clean":"2023-10-24T23:53:04.165320+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-24T23:53:04.165320+0000","last_undegraded":"2023-10-24T23:53:04.165320+0000","last_fullsized":"2023-10-24T23:53:04.165320+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497621+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-24T23:53:04.497621+0000","last_peered":"2023-10-24T23:53:04.497621+0000","last_clean":"2023-10-24T23:53:04.497621+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-24T23:53:04.497621+0000","last_undegraded":"2023-10-24T23:53:04.497621+0000","last_fullsized":"2023-10-24T23:53:04.497621+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.693689+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:02.693689+0000","last_peered":"2023-10-24T23:53:02.693689+0000","last_clean":"2023-10-24T23:53:02.693689+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:02.693689+0000","last_undegraded":"2023-10-24T23:53:02.693689+0000","last_fullsized":"2023-10-24T23:53:02.693689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499388+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-24T23:53:04.499388+0000","last_peered":"2023-10-24T23:53:04.499388+0000","last_clean":"2023-10-24T23:53:04.499388+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-24T23:53:04.499388+0000","last_undegraded":"2023-10-24T23:53:04.499388+0000","last_fullsized":"2023-10-24T23:53:04.499388+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499511+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-24T23:53:04.499511+0000","last_peered":"2023-10-24T23:53:04.499511+0000","last_clean":"2023-10-24T23:53:04.499511+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-24T23:53:04.499511+0000","last_undegraded":"2023-10-24T23:53:04.499511+0000","last_fullsized":"2023-10-24T23:53:04.499511+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499662+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-24T23:53:04.499662+0000","last_peered":"2023-10-24T23:53:04.499662+0000","last_clean":"2023-10-24T23:53:04.499662+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-24T23:53:04.499662+0000","last_undegraded":"2023-10-24T23:53:04.499662+0000","last_fullsized":"2023-10-24T23:53:04.499662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496844+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-24T23:53:04.496844+0000","last_peered":"2023-10-24T23:53:04.496844+0000","last_clean":"2023-10-24T23:53:04.496844+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-24T23:53:04.496844+0000","last_undegraded":"2023-10-24T23:53:04.496844+0000","last_fullsized":"2023-10-24T23:53:04.496844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062786+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-24T23:53:04.062786+0000","last_peered":"2023-10-24T23:53:04.062786+0000","last_clean":"2023-10-24T23:53:04.062786+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-24T23:53:04.062786+0000","last_undegraded":"2023-10-24T23:53:04.062786+0000","last_fullsized":"2023-10-24T23:53:04.062786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699929+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-24T23:53:03.699929+0000","last_peered":"2023-10-24T23:53:03.699929+0000","last_clean":"2023-10-24T23:53:03.699929+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-24T23:53:03.699929+0000","last_undegraded":"2023-10-24T23:53:03.699929+0000","last_fullsized":"2023-10-24T23:53:03.699929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165604+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-24T23:53:04.165604+0000","last_peered":"2023-10-24T23:53:04.165604+0000","last_clean":"2023-10-24T23:53:04.165604+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-24T23:53:04.165604+0000","last_undegraded":"2023-10-24T23:53:04.165604+0000","last_fullsized":"2023-10-24T23:53:04.165604+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497493+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-24T23:53:04.497493+0000","last_peered":"2023-10-24T23:53:04.497493+0000","last_clean":"2023-10-24T23:53:04.497493+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-24T23:53:04.497493+0000","last_undegraded":"2023-10-24T23:53:04.497493+0000","last_fullsized":"2023-10-24T23:53:04.497493+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165168+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-24T23:53:04.165168+0000","last_peered":"2023-10-24T23:53:04.165168+0000","last_clean":"2023-10-24T23:53:04.165168+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-24T23:53:04.165168+0000","last_undegraded":"2023-10-24T23:53:04.165168+0000","last_fullsized":"2023-10-24T23:53:04.165168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697959+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-24T23:53:03.697959+0000","last_peered":"2023-10-24T23:53:03.697959+0000","last_clean":"2023-10-24T23:53:03.697959+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-24T23:53:03.697959+0000","last_undegraded":"2023-10-24T23:53:03.697959+0000","last_fullsized":"2023-10-24T23:53:03.697959+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165441+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-24T23:53:04.165441+0000","last_peered":"2023-10-24T23:53:04.165441+0000","last_clean":"2023-10-24T23:53:04.165441+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-24T23:53:04.165441+0000","last_undegraded":"2023-10-24T23:53:04.165441+0000","last_fullsized":"2023-10-24T23:53:04.165441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497403+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-24T23:53:04.497403+0000","last_peered":"2023-10-24T23:53:04.497403+0000","last_clean":"2023-10-24T23:53:04.497403+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-24T23:53:04.497403+0000","last_undegraded":"2023-10-24T23:53:04.497403+0000","last_fullsized":"2023-10-24T23:53:04.497403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496826+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-24T23:53:04.496826+0000","last_peered":"2023-10-24T23:53:04.496826+0000","last_clean":"2023-10-24T23:53:04.496826+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-24T23:53:04.496826+0000","last_undegraded":"2023-10-24T23:53:04.496826+0000","last_fullsized":"2023-10-24T23:53:04.496826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166207+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-24T23:53:04.166207+0000","last_peered":"2023-10-24T23:53:04.166207+0000","last_clean":"2023-10-24T23:53:04.166207+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-24T23:53:04.166207+0000","last_undegraded":"2023-10-24T23:53:04.166207+0000","last_fullsized":"2023-10-24T23:53:04.166207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499601+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-24T23:53:04.499601+0000","last_peered":"2023-10-24T23:53:04.499601+0000","last_clean":"2023-10-24T23:53:04.499601+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-24T23:53:04.499601+0000","last_undegraded":"2023-10-24T23:53:04.499601+0000","last_fullsized":"2023-10-24T23:53:04.499601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696609+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-24T23:53:03.696609+0000","last_peered":"2023-10-24T23:53:03.696609+0000","last_clean":"2023-10-24T23:53:03.696609+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-24T23:53:03.696609+0000","last_undegraded":"2023-10-24T23:53:03.696609+0000","last_fullsized":"2023-10-24T23:53:03.696609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497288+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-24T23:53:04.497288+0000","last_peered":"2023-10-24T23:53:04.497288+0000","last_clean":"2023-10-24T23:53:04.497288+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-24T23:53:04.497288+0000","last_undegraded":"2023-10-24T23:53:04.497288+0000","last_fullsized":"2023-10-24T23:53:04.497288+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499513+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-24T23:53:04.499513+0000","last_peered":"2023-10-24T23:53:04.499513+0000","last_clean":"2023-10-24T23:53:04.499513+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-24T23:53:04.499513+0000","last_undegraded":"2023-10-24T23:53:04.499513+0000","last_fullsized":"2023-10-24T23:53:04.499513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699150+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-24T23:53:03.699150+0000","last_peered":"2023-10-24T23:53:03.699150+0000","last_clean":"2023-10-24T23:53:03.699150+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-24T23:53:03.699150+0000","last_undegraded":"2023-10-24T23:53:03.699150+0000","last_fullsized":"2023-10-24T23:53:03.699150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499034+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-24T23:53:04.499034+0000","last_peered":"2023-10-24T23:53:04.499034+0000","last_clean":"2023-10-24T23:53:04.499034+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-24T23:53:04.499034+0000","last_undegraded":"2023-10-24T23:53:04.499034+0000","last_fullsized":"2023-10-24T23:53:04.499034+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699826+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-24T23:53:03.699826+0000","last_peered":"2023-10-24T23:53:03.699826+0000","last_clean":"2023-10-24T23:53:03.699826+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-24T23:53:03.699826+0000","last_undegraded":"2023-10-24T23:53:03.699826+0000","last_fullsized":"2023-10-24T23:53:03.699826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166134+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-24T23:53:04.166134+0000","last_peered":"2023-10-24T23:53:04.166134+0000","last_clean":"2023-10-24T23:53:04.166134+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-24T23:53:04.166134+0000","last_undegraded":"2023-10-24T23:53:04.166134+0000","last_fullsized":"2023-10-24T23:53:04.166134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165883+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-24T23:53:04.165883+0000","last_peered":"2023-10-24T23:53:04.165883+0000","last_clean":"2023-10-24T23:53:04.165883+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-24T23:53:04.165883+0000","last_undegraded":"2023-10-24T23:53:04.165883+0000","last_fullsized":"2023-10-24T23:53:04.165883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498958+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-24T23:53:04.498958+0000","last_peered":"2023-10-24T23:53:04.498958+0000","last_clean":"2023-10-24T23:53:04.498958+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-24T23:53:04.498958+0000","last_undegraded":"2023-10-24T23:53:04.498958+0000","last_fullsized":"2023-10-24T23:53:04.498958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497423+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-24T23:53:04.497423+0000","last_peered":"2023-10-24T23:53:04.497423+0000","last_clean":"2023-10-24T23:53:04.497423+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-24T23:53:04.497423+0000","last_undegraded":"2023-10-24T23:53:04.497423+0000","last_fullsized":"2023-10-24T23:53:04.497423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":864256,"data_stored":681667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0629999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7170000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6850000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31284,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340556,"statfs":{"total":96636764160,"available":96604729344,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31292,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340548,"statfs":{"total":96636764160,"available":96604721152,"internally_reserved":0,"allocated":249856,"data_stored":90171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.107}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.177}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5389999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":262144,"data_stored":91139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.999}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":245760,"data_stored":89609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.748}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":839680,"data_stored":680469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:07.637 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-24T23:53:07.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42cad11640 1 Processor -- start 2023-10-24T23:53:07.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42cad11640 1 -- start start 2023-10-24T23:53:07.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42cad11640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f42c4158970 0x7f42c406a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42cad11640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c406cea0 con 0x7f42c4069d40 2023-10-24T23:53:07.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42cad11640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c41584e0 con 0x7f42c4065bc0 2023-10-24T23:53:07.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42cad11640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c40ac650 con 0x7f42c4158970 2023-10-24T23:53:07.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42c8aad640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f42c4069d40 0x7f42c40a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39186/0 (socket says 172.21.15.107:39186) 2023-10-24T23:53:07.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42c8aad640 1 -- 172.21.15.107:0/3600306533 learned_addr learned my addr 172.21.15.107:0/3600306533 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:07.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42c92ae640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f42c4158970 0x7f42c406a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42c37fe640 1 -- 172.21.15.107:0/3600306533 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226105687 0 0) 0x7f42c406cea0 con 0x7f42c4069d40 2023-10-24T23:53:07.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42c37fe640 1 -- 172.21.15.107:0/3600306533 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b4003530 con 0x7f42c4069d40 2023-10-24T23:53:07.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42c92ae640 1 -- 172.21.15.107:0/3600306533 >> v1:172.21.15.110:6789/0 conn(0x7f42c4065bc0 legacy=0x7f42c40aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:07.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42c92ae640 1 -- 172.21.15.107:0/3600306533 >> v1:172.21.15.107:6789/0 conn(0x7f42c4069d40 legacy=0x7f42c40a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42c92ae640 1 -- 172.21.15.107:0/3600306533 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c406d3d0 con 0x7f42c4158970 2023-10-24T23:53:07.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.795+0000 7f42c92ae640 1 --2- 172.21.15.107:0/3600306533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c4158970 0x7f42c406a120 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009d90 tx=0x7f42bc01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9b57514eaebd server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:07.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c37fe640 1 -- 172.21.15.107:0/3600306533 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0a5780 con 0x7f42c4158970 2023-10-24T23:53:07.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c37fe640 1 -- 172.21.15.107:0/3600306533 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc0a58e0 con 0x7f42c4158970 2023-10-24T23:53:07.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c37fe640 1 -- 172.21.15.107:0/3600306533 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0a5c70 con 0x7f42c4158970 2023-10-24T23:53:07.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 -- 172.21.15.107:0/3600306533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c4158970 msgr2=0x7f42c406a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 --2- 172.21.15.107:0/3600306533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c4158970 0x7f42c406a120 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009d90 tx=0x7f42bc01c8c0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 -- 172.21.15.107:0/3600306533 shutdown_connections 2023-10-24T23:53:07.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 --2- 172.21.15.107:0/3600306533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c4158970 0x7f42c406a120 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 -- 172.21.15.107:0/3600306533 >> 172.21.15.107:0/3600306533 conn(0x7f42c40c65c0 msgr2=0x7f42c40c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:07.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 -- 172.21.15.107:0/3600306533 shutdown_connections 2023-10-24T23:53:07.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 -- 172.21.15.107:0/3600306533 wait complete. 2023-10-24T23:53:07.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 Processor -- start 2023-10-24T23:53:07.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 -- start start 2023-10-24T23:53:07.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c4065bc0 0x7f42c40bcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f42c4069d40 0x7f42c41f01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42c4158970 0x7f42c40bc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c414a190 con 0x7f42c4065bc0 2023-10-24T23:53:07.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c41f0a40 con 0x7f42c4158970 2023-10-24T23:53:07.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c41f0d20 con 0x7f42c4069d40 2023-10-24T23:53:07.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c8aad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c4065bc0 0x7f42c40bcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c8aad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c4065bc0 0x7f42c40bcfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43456/0 (socket says 172.21.15.107:43456) 2023-10-24T23:53:07.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c8aad640 1 -- 172.21.15.107:0/3294815874 learned_addr learned my addr 172.21.15.107:0/3294815874 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:07.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c92ae640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42c4158970 0x7f42c40bc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c3fff640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f42c4069d40 0x7f42c41f01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c8aad640 1 -- 172.21.15.107:0/3294815874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f42c4069d40 msgr2=0x7f42c41f01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c8aad640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f42c4069d40 0x7f42c41f01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c8aad640 1 -- 172.21.15.107:0/3294815874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42c4158970 msgr2=0x7f42c40bc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c8aad640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42c4158970 0x7f42c40bc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c8aad640 1 -- 172.21.15.107:0/3294815874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc009a70 con 0x7f42c4065bc0 2023-10-24T23:53:07.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c8aad640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c4065bc0 0x7f42c40bcfe0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f42b800c4d0 tx=0x7f42b800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:07.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c17fa640 1 -- 172.21.15.107:0/3294815874 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8005cf0 con 0x7f42c4065bc0 2023-10-24T23:53:07.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c17fa640 1 -- 172.21.15.107:0/3294815874 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b8005e50 con 0x7f42c4065bc0 2023-10-24T23:53:07.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 -- 172.21.15.107:0/3294815874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c4071b10 con 0x7f42c4065bc0 2023-10-24T23:53:07.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42c17fa640 1 -- 172.21.15.107:0/3294815874 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8007270 con 0x7f42c4065bc0 2023-10-24T23:53:07.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.799+0000 7f42cad11640 1 -- 172.21.15.107:0/3294815874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c40720d0 con 0x7f42c4065bc0 2023-10-24T23:53:07.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.803+0000 7f42cad11640 1 -- 172.21.15.107:0/3294815874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c40722c0 con 0x7f42c4065bc0 2023-10-24T23:53:07.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.803+0000 7f42c17fa640 1 -- 172.21.15.107:0/3294815874 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f42b80073d0 con 0x7f42c4065bc0 2023-10-24T23:53:07.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.803+0000 7f42c17fa640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f42a8065e90 0x7f42a8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:07.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.803+0000 7f42c17fa640 1 -- 172.21.15.107:0/3294815874 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f42b8082d00 con 0x7f42c4065bc0 2023-10-24T23:53:07.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.803+0000 7f42c3fff640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f42a8065e90 0x7f42a8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:07.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.803+0000 7f42c3fff640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f42a8065e90 0x7f42a8068350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f42c414dda0 tx=0x7f42b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:07.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.807+0000 7f42c17fa640 1 -- 172.21.15.107:0/3294815874 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f42b8050400 con 0x7f42c4065bc0 2023-10-24T23:53:07.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.947+0000 7f42cad11640 1 -- 172.21.15.107:0/3294815874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42c40746e0 con 0x7f42c4065bc0 2023-10-24T23:53:07.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42c17fa640 1 -- 172.21.15.107:0/3294815874 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+17839 (secure 0 0 0) 0x7f42b8017090 con 0x7f42c4065bc0 2023-10-24T23:53:07.954 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:07.954 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:03.684277+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 -- 172.21.15.107:0/3294815874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f42a8065e90 msgr2=0x7f42a8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f42a8065e90 0x7f42a8068350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f42c414dda0 tx=0x7f42b0009040 comp rx=0 tx=0).stop 2023-10-24T23:53:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 -- 172.21.15.107:0/3294815874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c4065bc0 msgr2=0x7f42c40bcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c4065bc0 0x7f42c40bcfe0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f42b800c4d0 tx=0x7f42b800c990 comp rx=0 tx=0).stop 2023-10-24T23:53:07.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 -- 172.21.15.107:0/3294815874 shutdown_connections 2023-10-24T23:53:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42c4158970 0x7f42c40bc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f42a8065e90 0x7f42a8068350 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f42c4069d40 0x7f42c41f01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 --2- 172.21.15.107:0/3294815874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42c4065bc0 0x7f42c40bcfe0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 -- 172.21.15.107:0/3294815874 >> 172.21.15.107:0/3294815874 conn(0x7f42c40c65c0 msgr2=0x7f42c406c490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:07.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 -- 172.21.15.107:0/3294815874 shutdown_connections 2023-10-24T23:53:07.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:07.951+0000 7f42cad11640 1 -- 172.21.15.107:0/3294815874 wait complete. 2023-10-24T23:53:07.971 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-24T23:53:07.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-24T23:53:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8b4dd6640 1 Processor -- start 2023-10-24T23:53:08.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8b4dd6640 1 -- start start 2023-10-24T23:53:08.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8b4dd6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8b0072720 0x7ff8b00c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8b4dd6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8b00bceb0 0x7ff8b015c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8b4dd6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b00c1ce0 con 0x7ff8b00ab1a0 2023-10-24T23:53:08.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8b4dd6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b00be1d0 con 0x7ff8b00bceb0 2023-10-24T23:53:08.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8b4dd6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b00c0240 con 0x7ff8b0072720 2023-10-24T23:53:08.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8aed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8b00bceb0 0x7ff8b015c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8aed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8b00bceb0 0x7ff8b015c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43458/0 (socket says 172.21.15.107:43458) 2023-10-24T23:53:08.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8aed9d640 1 -- 172.21.15.107:0/2406203337 learned_addr learned my addr 172.21.15.107:0/2406203337 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:08.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8a7fff640 1 --2- 172.21.15.107:0/2406203337 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8b0072720 0x7ff8b00c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8aed9d640 1 -- 172.21.15.107:0/2406203337 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8b0072720 msgr2=0x7ff8b00c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:08.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8aed9d640 1 --2- 172.21.15.107:0/2406203337 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8b0072720 0x7ff8b00c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8aed9d640 1 -- 172.21.15.107:0/2406203337 >> v1:172.21.15.110:6789/0 conn(0x7ff8b00ab1a0 legacy=0x7ff8b00bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:08.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8aed9d640 1 -- 172.21.15.107:0/2406203337 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b00c0880 con 0x7ff8b00bceb0 2023-10-24T23:53:08.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8aed9d640 1 --2- 172.21.15.107:0/2406203337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b00bceb0 0x7ff8b015c660 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009e50 tx=0x7ff8a801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3cbc3243dc5e243 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8add9b640 1 -- 172.21.15.107:0/2406203337 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80a5700 con 0x7ff8b00bceb0 2023-10-24T23:53:08.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8add9b640 1 -- 172.21.15.107:0/2406203337 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a80a5860 con 0x7ff8b00bceb0 2023-10-24T23:53:08.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.083+0000 7ff8add9b640 1 -- 172.21.15.107:0/2406203337 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80a5bf0 con 0x7ff8b00bceb0 2023-10-24T23:53:08.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/2406203337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b00bceb0 msgr2=0x7ff8b015c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 --2- 172.21.15.107:0/2406203337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b00bceb0 0x7ff8b015c660 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009e50 tx=0x7ff8a801c880 comp rx=0 tx=0).stop 2023-10-24T23:53:08.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/2406203337 shutdown_connections 2023-10-24T23:53:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 --2- 172.21.15.107:0/2406203337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b00bceb0 0x7ff8b015c660 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 --2- 172.21.15.107:0/2406203337 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8b0072720 0x7ff8b00c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/2406203337 >> 172.21.15.107:0/2406203337 conn(0x7ff8b00c2860 msgr2=0x7ff8b00c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/2406203337 shutdown_connections 2023-10-24T23:53:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/2406203337 wait complete. 2023-10-24T23:53:08.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 Processor -- start 2023-10-24T23:53:08.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 -- start start 2023-10-24T23:53:08.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8b00ab1a0 0x7ff8b0157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8b00bceb0 0x7ff8b00a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b0154fb0 0x7ff8b00a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b0070130 con 0x7ff8b0154fb0 2023-10-24T23:53:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b0071b80 con 0x7ff8b00bceb0 2023-10-24T23:53:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b00715c0 con 0x7ff8b00ab1a0 2023-10-24T23:53:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8aed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b0154fb0 0x7ff8b00a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8aed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b0154fb0 0x7ff8b00a3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43474/0 (socket says 172.21.15.107:43474) 2023-10-24T23:53:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8aed9d640 1 -- 172.21.15.107:0/3532596035 learned_addr learned my addr 172.21.15.107:0/3532596035 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8a7fff640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8b00bceb0 0x7ff8b00a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8ae59c640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8b00ab1a0 0x7ff8b0157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8aed9d640 1 -- 172.21.15.107:0/3532596035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8b00ab1a0 msgr2=0x7ff8b0157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8aed9d640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8b00ab1a0 0x7ff8b0157bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8aed9d640 1 -- 172.21.15.107:0/3532596035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8b00bceb0 msgr2=0x7ff8b00a9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8aed9d640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8b00bceb0 0x7ff8b00a9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8aed9d640 1 -- 172.21.15.107:0/3532596035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a8009b30 con 0x7ff8b0154fb0 2023-10-24T23:53:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8aed9d640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b0154fb0 0x7ff8b00a3e50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80af040 tx=0x7ff8a8019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8a6ffd640 1 -- 172.21.15.107:0/3532596035 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80b8670 con 0x7ff8b0154fb0 2023-10-24T23:53:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8a6ffd640 1 -- 172.21.15.107:0/3532596035 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a80b87d0 con 0x7ff8b0154fb0 2023-10-24T23:53:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8a6ffd640 1 -- 172.21.15.107:0/3532596035 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a80b8b60 con 0x7ff8b0154fb0 2023-10-24T23:53:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/3532596035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b00718a0 con 0x7ff8b0154fb0 2023-10-24T23:53:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.087+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/3532596035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b006fe50 con 0x7ff8b0154fb0 2023-10-24T23:53:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.091+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/3532596035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b00beff0 con 0x7ff8b0154fb0 2023-10-24T23:53:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.091+0000 7ff8a6ffd640 1 -- 172.21.15.107:0/3532596035 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff8a80b8cc0 con 0x7ff8b0154fb0 2023-10-24T23:53:08.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.091+0000 7ff8a6ffd640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff890065e60 0x7ff890068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.091+0000 7ff8a6ffd640 1 -- 172.21.15.107:0/3532596035 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7ff8a811ed30 con 0x7ff8b0154fb0 2023-10-24T23:53:08.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.091+0000 7ff8ae59c640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff890065e60 0x7ff890068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.091+0000 7ff8ae59c640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff890065e60 0x7ff890068320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0004e30 tx=0x7ff8a0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.095+0000 7ff8a6ffd640 1 -- 172.21.15.107:0/3532596035 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff8a80ec4b0 con 0x7ff8b0154fb0 2023-10-24T23:53:08.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.247+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/3532596035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff8b004e130 con 0x7ff8b0154fb0 2023-10-24T23:53:08.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.247+0000 7ff8a6ffd640 1 -- 172.21.15.107:0/3532596035 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff8a8009510 con 0x7ff8b0154fb0 2023-10-24T23:53:08.250 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-24T23:53:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/3532596035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff890065e60 msgr2=0x7ff890068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff890065e60 0x7ff890068320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0004e30 tx=0x7ff8a0009040 comp rx=0 tx=0).stop 2023-10-24T23:53:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/3532596035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b0154fb0 msgr2=0x7ff8b00a3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b0154fb0 0x7ff8b00a3e50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80af040 tx=0x7ff8a8019620 comp rx=0 tx=0).stop 2023-10-24T23:53:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/3532596035 shutdown_connections 2023-10-24T23:53:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b0154fb0 0x7ff8b00a3e50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8b00bceb0 0x7ff8b00a9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff890065e60 0x7ff890068320 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 --2- 172.21.15.107:0/3532596035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8b00ab1a0 0x7ff8b0157bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/3532596035 >> 172.21.15.107:0/3532596035 conn(0x7ff8b00c2860 msgr2=0x7ff8b00a6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:08.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/3532596035 shutdown_connections 2023-10-24T23:53:08.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.251+0000 7ff8b4dd6640 1 -- 172.21.15.107:0/3532596035 wait complete. 2023-10-24T23:53:08.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-24T23:53:08.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.391+0000 7f2679d32640 1 Processor -- start 2023-10-24T23:53:08.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.391+0000 7f2679d32640 1 -- start start 2023-10-24T23:53:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2679d32640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f26740a9300 0x7f2674072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2679d32640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26740befa0 con 0x7f26740c1b30 2023-10-24T23:53:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2679d32640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26740a9b80 con 0x7f26740aa9f0 2023-10-24T23:53:08.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2679d32640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26740aa0b0 con 0x7f26740a9300 2023-10-24T23:53:08.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26737fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f26740c1b30 0x7f26740aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39200/0 (socket says 172.21.15.107:39200) 2023-10-24T23:53:08.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26737fe640 1 -- 172.21.15.107:0/3466601631 learned_addr learned my addr 172.21.15.107:0/3466601631 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:08.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035378864 0 0) 0x7f26740befa0 con 0x7f26740c1b30 2023-10-24T23:53:08.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2660003530 con 0x7f26740c1b30 2023-10-24T23:53:08.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2673fff640 1 --2- 172.21.15.107:0/3466601631 >> v2:172.21.15.110:3300/0 conn(0x7f26740a9300 0x7f2674072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 182088899 0 0) 0x7f2660003530 con 0x7f26740c1b30 2023-10-24T23:53:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 >> v2:172.21.15.110:3300/0 conn(0x7f26740a9300 msgr2=0x7f2674072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 --2- 172.21.15.107:0/3466601631 >> v2:172.21.15.110:3300/0 conn(0x7f26740a9300 0x7f2674072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 >> v1:172.21.15.110:6789/0 conn(0x7f26740aa9f0 legacy=0x7f2674155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:08.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26740bf1f0 con 0x7f26740c1b30 2023-10-24T23:53:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f266c001a00 con 0x7f26740c1b30 2023-10-24T23:53:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 >> v1:172.21.15.107:6789/0 conn(0x7f26740c1b30 legacy=0x7f26740aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 --2- 172.21.15.107:0/3466601631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2660003980 0x7f2660003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 --2- 172.21.15.107:0/3466601631 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26600047d0 0x7f2660004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 --2- 172.21.15.107:0/3466601631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26600052b0 0x7f266000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26740aa0b0 con 0x7f26600052b0 2023-10-24T23:53:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266000c200 con 0x7f2660003980 2023-10-24T23:53:08.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266000c3c0 con 0x7f26600047d0 2023-10-24T23:53:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2673fff640 1 --2- 172.21.15.107:0/3466601631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26600052b0 0x7f266000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26737fe640 1 --2- 172.21.15.107:0/3466601631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2660003980 0x7f2660003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2672ffd640 1 --2- 172.21.15.107:0/3466601631 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26600047d0 0x7f2660004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2673fff640 1 -- 172.21.15.107:0/3466601631 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26600047d0 msgr2=0x7f2660004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2673fff640 1 --2- 172.21.15.107:0/3466601631 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26600047d0 0x7f2660004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2673fff640 1 -- 172.21.15.107:0/3466601631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2660003980 msgr2=0x7f2660003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2673fff640 1 --2- 172.21.15.107:0/3466601631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2660003980 0x7f2660003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2673fff640 1 -- 172.21.15.107:0/3466601631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2660003530 con 0x7f26600052b0 2023-10-24T23:53:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2673fff640 1 -- 172.21.15.107:0/3466601631 reap_dead start 2023-10-24T23:53:08.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f2673fff640 1 --2- 172.21.15.107:0/3466601631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26600052b0 0x7f266000b990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f26740a9710 tx=0x7f266401c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2e045f258b3740 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2664004020 con 0x7f26600052b0 2023-10-24T23:53:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26640a7030 con 0x7f26600052b0 2023-10-24T23:53:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.395+0000 7f26727fc640 1 -- 172.21.15.107:0/3466601631 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26640043c0 con 0x7f26600052b0 2023-10-24T23:53:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 -- 172.21.15.107:0/3466601631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26600052b0 msgr2=0x7f266000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 --2- 172.21.15.107:0/3466601631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26600052b0 0x7f266000b990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f26740a9710 tx=0x7f266401c660 comp rx=0 tx=0).stop 2023-10-24T23:53:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 -- 172.21.15.107:0/3466601631 shutdown_connections 2023-10-24T23:53:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 --2- 172.21.15.107:0/3466601631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26600052b0 0x7f266000b990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 -- 172.21.15.107:0/3466601631 >> 172.21.15.107:0/3466601631 conn(0x7f26740c2860 msgr2=0x7f26740c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 -- 172.21.15.107:0/3466601631 shutdown_connections 2023-10-24T23:53:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 -- 172.21.15.107:0/3466601631 wait complete. 2023-10-24T23:53:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 Processor -- start 2023-10-24T23:53:08.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 -- start start 2023-10-24T23:53:08.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26600052b0 0x7f26740c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2674155230 0x7f267415aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26741556f0 0x7f267415af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26741612a0 con 0x7f26741556f0 2023-10-24T23:53:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26741617d0 con 0x7f2674155230 2023-10-24T23:53:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26741f0db0 con 0x7f26600052b0 2023-10-24T23:53:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2673fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26741556f0 0x7f267415af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2673fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26741556f0 0x7f267415af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43506/0 (socket says 172.21.15.107:43506) 2023-10-24T23:53:08.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2673fff640 1 -- 172.21.15.107:0/2247914419 learned_addr learned my addr 172.21.15.107:0/2247914419 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:08.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2672ffd640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2674155230 0x7f267415aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f26737fe640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26600052b0 0x7f26740c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2673fff640 1 -- 172.21.15.107:0/2247914419 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26600052b0 msgr2=0x7f26740c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2673fff640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26600052b0 0x7f26740c3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2673fff640 1 -- 172.21.15.107:0/2247914419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2674155230 msgr2=0x7f267415aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2673fff640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2674155230 0x7f267415aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2673fff640 1 -- 172.21.15.107:0/2247914419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26741614f0 con 0x7f26741556f0 2023-10-24T23:53:08.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2672ffd640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2674155230 0x7f267415aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:08.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2673fff640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26741556f0 0x7f267415af80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f266401c490 tx=0x7f2660005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f266bfff640 1 -- 172.21.15.107:0/2247914419 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2664003620 con 0x7f26741556f0 2023-10-24T23:53:08.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f266bfff640 1 -- 172.21.15.107:0/2247914419 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2664003780 con 0x7f26741556f0 2023-10-24T23:53:08.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 -- 172.21.15.107:0/2247914419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2674160310 con 0x7f26741556f0 2023-10-24T23:53:08.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f266bfff640 1 -- 172.21.15.107:0/2247914419 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2664003910 con 0x7f26741556f0 2023-10-24T23:53:08.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.399+0000 7f2679d32640 1 -- 172.21.15.107:0/2247914419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26741f0a40 con 0x7f26741556f0 2023-10-24T23:53:08.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.403+0000 7f2679d32640 1 -- 172.21.15.107:0/2247914419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2674160520 con 0x7f26741556f0 2023-10-24T23:53:08.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.403+0000 7f266bfff640 1 -- 172.21.15.107:0/2247914419 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2664003a70 con 0x7f26741556f0 2023-10-24T23:53:08.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.403+0000 7f266bfff640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2654065e90 0x7f2654068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.403+0000 7f26737fe640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2654065e90 0x7f2654068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.403+0000 7f266bfff640 1 -- 172.21.15.107:0/2247914419 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f266411d500 con 0x7f26741556f0 2023-10-24T23:53:08.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.403+0000 7f26737fe640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2654065e90 0x7f2654068350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f266c000f90 tx=0x7f266c006040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.407+0000 7f266bfff640 1 -- 172.21.15.107:0/2247914419 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f26640ee7a0 con 0x7f26741556f0 2023-10-24T23:53:08.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.559+0000 7f2679d32640 1 -- 172.21.15.107:0/2247914419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f267404e130 con 0x7f26741556f0 2023-10-24T23:53:08.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f266bfff640 1 -- 172.21.15.107:0/2247914419 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f26640ee330 con 0x7f26741556f0 2023-10-24T23:53:08.565 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-24T23:53:08.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 -- 172.21.15.107:0/2247914419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2654065e90 msgr2=0x7f2654068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2654065e90 0x7f2654068350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f266c000f90 tx=0x7f266c006040 comp rx=0 tx=0).stop 2023-10-24T23:53:08.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 -- 172.21.15.107:0/2247914419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26741556f0 msgr2=0x7f267415af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26741556f0 0x7f267415af80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f266401c490 tx=0x7f2660005800 comp rx=0 tx=0).stop 2023-10-24T23:53:08.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 -- 172.21.15.107:0/2247914419 shutdown_connections 2023-10-24T23:53:08.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26741556f0 0x7f267415af80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2674155230 0x7f267415aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2654065e90 0x7f2654068350 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 --2- 172.21.15.107:0/2247914419 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26600052b0 0x7f26740c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 -- 172.21.15.107:0/2247914419 >> 172.21.15.107:0/2247914419 conn(0x7f26740c2860 msgr2=0x7f26740aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:08.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 -- 172.21.15.107:0/2247914419 shutdown_connections 2023-10-24T23:53:08.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.563+0000 7f2679d32640 1 -- 172.21.15.107:0/2247914419 wait complete. 2023-10-24T23:53:08.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-24T23:53:08.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf5c9c7640 1 Processor -- start 2023-10-24T23:53:08.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf5c9c7640 1 -- start start 2023-10-24T23:53:08.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf5c9c7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf580bd930 0x7fdf581602a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf5c9c7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf580c8180 con 0x7fdf5806b360 2023-10-24T23:53:08.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf5c9c7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf58155430 con 0x7fdf580af9b0 2023-10-24T23:53:08.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf5c9c7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf580c5870 con 0x7fdf580bd930 2023-10-24T23:53:08.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf56d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf580bd930 0x7fdf581602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf55d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdf5806b360 0x7fdf58069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39204/0 (socket says 172.21.15.107:39204) 2023-10-24T23:53:08.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf55d9b640 1 -- 172.21.15.107:0/2857154289 learned_addr learned my addr 172.21.15.107:0/2857154289 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:08.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf56d9d640 1 -- 172.21.15.107:0/2857154289 >> v1:172.21.15.110:6789/0 conn(0x7fdf580af9b0 legacy=0x7fdf580c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:08.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf5559a640 1 -- 172.21.15.107:0/2857154289 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355557697 0 0) 0x7fdf580c8180 con 0x7fdf5806b360 2023-10-24T23:53:08.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf56d9d640 1 -- 172.21.15.107:0/2857154289 >> v1:172.21.15.107:6789/0 conn(0x7fdf5806b360 legacy=0x7fdf58069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.695+0000 7fdf56d9d640 1 -- 172.21.15.107:0/2857154289 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf58155710 con 0x7fdf580bd930 2023-10-24T23:53:08.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf56d9d640 1 --2- 172.21.15.107:0/2857154289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf580bd930 0x7fdf581602a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009d90 tx=0x7fdf4801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d371718cabe9217d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5559a640 1 -- 172.21.15.107:0/2857154289 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf480a58a0 con 0x7fdf580bd930 2023-10-24T23:53:08.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5559a640 1 -- 172.21.15.107:0/2857154289 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf480a5a00 con 0x7fdf580bd930 2023-10-24T23:53:08.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5559a640 1 -- 172.21.15.107:0/2857154289 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf480a5d90 con 0x7fdf580bd930 2023-10-24T23:53:08.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/2857154289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf580bd930 msgr2=0x7fdf581602a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 --2- 172.21.15.107:0/2857154289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf580bd930 0x7fdf581602a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009d90 tx=0x7fdf4801c850 comp rx=0 tx=0).stop 2023-10-24T23:53:08.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/2857154289 shutdown_connections 2023-10-24T23:53:08.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 --2- 172.21.15.107:0/2857154289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf580bd930 0x7fdf581602a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/2857154289 >> 172.21.15.107:0/2857154289 conn(0x7fdf58071b00 msgr2=0x7fdf580abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:08.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/2857154289 shutdown_connections 2023-10-24T23:53:08.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/2857154289 wait complete. 2023-10-24T23:53:08.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 Processor -- start 2023-10-24T23:53:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 -- start start 2023-10-24T23:53:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf5806b360 0x7fdf581edc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf580af9b0 0x7fdf580c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf580bd930 0x7fdf58149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf581f0b60 con 0x7fdf580bd930 2023-10-24T23:53:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf581ed270 con 0x7fdf580af9b0 2023-10-24T23:53:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf581ed7f0 con 0x7fdf5806b360 2023-10-24T23:53:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf56d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf580bd930 0x7fdf58149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf56d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf580bd930 0x7fdf58149a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43524/0 (socket says 172.21.15.107:43524) 2023-10-24T23:53:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf56d9d640 1 -- 172.21.15.107:0/168244166 learned_addr learned my addr 172.21.15.107:0/168244166 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:08.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5659c640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf5806b360 0x7fdf581edc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf55d9b640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf580af9b0 0x7fdf580c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf56d9d640 1 -- 172.21.15.107:0/168244166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf5806b360 msgr2=0x7fdf581edc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf56d9d640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf5806b360 0x7fdf581edc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf56d9d640 1 -- 172.21.15.107:0/168244166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf580af9b0 msgr2=0x7fdf580c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf56d9d640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf580af9b0 0x7fdf580c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf56d9d640 1 -- 172.21.15.107:0/168244166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf48009a70 con 0x7fdf580bd930 2023-10-24T23:53:08.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf56d9d640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf580bd930 0x7fdf58149a80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdf480b0040 tx=0x7fdf480195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf4effd640 1 -- 172.21.15.107:0/168244166 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf480ad850 con 0x7fdf580bd930 2023-10-24T23:53:08.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf4effd640 1 -- 172.21.15.107:0/168244166 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf480ad9b0 con 0x7fdf580bd930 2023-10-24T23:53:08.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/168244166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf581ee1c0 con 0x7fdf580bd930 2023-10-24T23:53:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf4effd640 1 -- 172.21.15.107:0/168244166 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf480add60 con 0x7fdf580bd930 2023-10-24T23:53:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.699+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/168244166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf581f0880 con 0x7fdf580bd930 2023-10-24T23:53:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.703+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/168244166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf581ed960 con 0x7fdf580bd930 2023-10-24T23:53:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.703+0000 7fdf4effd640 1 -- 172.21.15.107:0/168244166 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdf480b4050 con 0x7fdf580bd930 2023-10-24T23:53:08.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.703+0000 7fdf4effd640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf38065e30 0x7fdf380682f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.703+0000 7fdf4effd640 1 -- 172.21.15.107:0/168244166 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7fdf4811e8c0 con 0x7fdf580bd930 2023-10-24T23:53:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.703+0000 7fdf5659c640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf38065e30 0x7fdf380682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.703+0000 7fdf5659c640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf38065e30 0x7fdf380682f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdf50004e30 tx=0x7fdf50009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.707+0000 7fdf4effd640 1 -- 172.21.15.107:0/168244166 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdf480ec040 con 0x7fdf580bd930 2023-10-24T23:53:08.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/168244166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdf58066560 con 0x7fdf580bd930 2023-10-24T23:53:08.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf4effd640 1 -- 172.21.15.107:0/168244166 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v23) v1 ==== 99+0+11 (secure 0 0 0) 0x7fdf480c70a0 con 0x7fdf580bd930 2023-10-24T23:53:08.850 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:08.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/168244166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf38065e30 msgr2=0x7fdf380682f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf38065e30 0x7fdf380682f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdf50004e30 tx=0x7fdf50009040 comp rx=0 tx=0).stop 2023-10-24T23:53:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/168244166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf580bd930 msgr2=0x7fdf58149a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf580bd930 0x7fdf58149a80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdf480b0040 tx=0x7fdf480195f0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/168244166 shutdown_connections 2023-10-24T23:53:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf580bd930 0x7fdf58149a80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf580af9b0 0x7fdf580c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf38065e30 0x7fdf380682f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 --2- 172.21.15.107:0/168244166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf5806b360 0x7fdf581edc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/168244166 >> 172.21.15.107:0/168244166 conn(0x7fdf58071b00 msgr2=0x7fdf5814e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/168244166 shutdown_connections 2023-10-24T23:53:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.847+0000 7fdf5c9c7640 1 -- 172.21.15.107:0/168244166 wait complete. 2023-10-24T23:53:08.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-24T23:53:08.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.975+0000 7f4847a37640 1 Processor -- start 2023-10-24T23:53:08.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.975+0000 7f4847a37640 1 -- start start 2023-10-24T23:53:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4847a37640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f48400c1c30 0x7f48400aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4847a37640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400c4cf0 0x7f48400c5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4847a37640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48400bdad0 con 0x7f48400a50d0 2023-10-24T23:53:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4847a37640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48400b97d0 con 0x7f48400c1c30 2023-10-24T23:53:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4847a37640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48400a8fd0 con 0x7f48400c4cf0 2023-10-24T23:53:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4845fd4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400c4cf0 0x7f48400c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4844fd2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f48400c1c30 0x7f48400aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4844fd2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f48400c1c30 0x7f48400aaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60426/0 (socket says 172.21.15.107:60426) 2023-10-24T23:53:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4845fd4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400c4cf0 0x7f48400c5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:36028/0 (socket says 172.21.15.107:36028) 2023-10-24T23:53:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4844fd2640 1 -- 172.21.15.107:0/555143134 learned_addr learned my addr 172.21.15.107:0/555143134 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4844fd2640 1 -- 172.21.15.107:0/555143134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400c4cf0 msgr2=0x7f48400c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4844fd2640 1 --2- 172.21.15.107:0/555143134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400c4cf0 0x7f48400c5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4844fd2640 1 -- 172.21.15.107:0/555143134 >> v1:172.21.15.110:6789/0 conn(0x7f48400a50d0 legacy=0x7f48400bd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4844fd2640 1 -- 172.21.15.107:0/555143134 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48400a9cf0 con 0x7f48400c1c30 2023-10-24T23:53:08.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4845fd4640 1 --2- 172.21.15.107:0/555143134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400c4cf0 0x7f48400c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:53:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.979+0000 7f4844fd2640 1 --2- 172.21.15.107:0/555143134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48400c1c30 0x7f48400aaab0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f483c009870 tx=0x7f483c01c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b7064e3921e441e server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4837fff640 1 -- 172.21.15.107:0/555143134 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0a7070 con 0x7f48400c1c30 2023-10-24T23:53:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4837fff640 1 -- 172.21.15.107:0/555143134 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483c0ab430 con 0x7f48400c1c30 2023-10-24T23:53:08.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4837fff640 1 -- 172.21.15.107:0/555143134 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0ab7c0 con 0x7f48400c1c30 2023-10-24T23:53:08.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 -- 172.21.15.107:0/555143134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48400c1c30 msgr2=0x7f48400aaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 --2- 172.21.15.107:0/555143134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48400c1c30 0x7f48400aaab0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f483c009870 tx=0x7f483c01c540 comp rx=0 tx=0).stop 2023-10-24T23:53:08.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 -- 172.21.15.107:0/555143134 shutdown_connections 2023-10-24T23:53:08.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 --2- 172.21.15.107:0/555143134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400c4cf0 0x7f48400c5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 --2- 172.21.15.107:0/555143134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48400c1c30 0x7f48400aaab0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 -- 172.21.15.107:0/555143134 >> 172.21.15.107:0/555143134 conn(0x7f48400c2860 msgr2=0x7f48400c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:08.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 -- 172.21.15.107:0/555143134 shutdown_connections 2023-10-24T23:53:08.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 -- 172.21.15.107:0/555143134 wait complete. 2023-10-24T23:53:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 Processor -- start 2023-10-24T23:53:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 -- start start 2023-10-24T23:53:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400a50d0 0x7f484015fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48400c1c30 0x7f4840070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48400c4cf0 0x7f4840071420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48401f1f30 con 0x7f48400c1c30 2023-10-24T23:53:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4840071cd0 con 0x7f48400c4cf0 2023-10-24T23:53:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48401f1340 con 0x7f48400a50d0 2023-10-24T23:53:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4844fd2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48400c1c30 0x7f4840070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4844fd2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48400c1c30 0x7f4840070ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43538/0 (socket says 172.21.15.107:43538) 2023-10-24T23:53:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4844fd2640 1 -- 172.21.15.107:0/159724004 learned_addr learned my addr 172.21.15.107:0/159724004 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f48457d3640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400a50d0 0x7f484015fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4845fd4640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48400c4cf0 0x7f4840071420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4844fd2640 1 -- 172.21.15.107:0/159724004 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400a50d0 msgr2=0x7f484015fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4844fd2640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400a50d0 0x7f484015fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4844fd2640 1 -- 172.21.15.107:0/159724004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48400c4cf0 msgr2=0x7f4840071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4844fd2640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48400c4cf0 0x7f4840071420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4844fd2640 1 -- 172.21.15.107:0/159724004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4838009a70 con 0x7f48400c1c30 2023-10-24T23:53:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4845fd4640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48400c4cf0 0x7f4840071420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4844fd2640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48400c1c30 0x7f4840070ee0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f483c01a170 tx=0x7f483c0192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4835ffb640 1 -- 172.21.15.107:0/159724004 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0ba070 con 0x7f48400c1c30 2023-10-24T23:53:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4835ffb640 1 -- 172.21.15.107:0/159724004 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483c0b4420 con 0x7f48400c1c30 2023-10-24T23:53:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 -- 172.21.15.107:0/159724004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c0094d0 con 0x7f48400c1c30 2023-10-24T23:53:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4835ffb640 1 -- 172.21.15.107:0/159724004 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0b47d0 con 0x7f48400c1c30 2023-10-24T23:53:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.983+0000 7f4847a37640 1 -- 172.21.15.107:0/159724004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48401f18e0 con 0x7f48400c1c30 2023-10-24T23:53:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.987+0000 7f4847a37640 1 -- 172.21.15.107:0/159724004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48400a76d0 con 0x7f48400c1c30 2023-10-24T23:53:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.987+0000 7f4835ffb640 1 -- 172.21.15.107:0/159724004 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f483c0b4930 con 0x7f48400c1c30 2023-10-24T23:53:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.987+0000 7f4835ffb640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4824065e60 0x7f4824068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.987+0000 7f48457d3640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4824065e60 0x7f4824068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.987+0000 7f4835ffb640 1 -- 172.21.15.107:0/159724004 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f483c11d1a0 con 0x7f48400c1c30 2023-10-24T23:53:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.987+0000 7f48457d3640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4824065e60 0x7f4824068320 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4830000f70 tx=0x7f4830009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:08.991+0000 7f4835ffb640 1 -- 172.21.15.107:0/159724004 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f483c0ea890 con 0x7f48400c1c30 2023-10-24T23:53:09.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 -- 172.21.15.107:0/159724004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f48400bcf50 con 0x7f48400c1c30 2023-10-24T23:53:09.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4835ffb640 1 -- 172.21.15.107:0/159724004 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v23) v1 ==== 95+0+11 (secure 0 0 0) 0x7f483c0edeb0 con 0x7f48400c1c30 2023-10-24T23:53:09.141 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:09.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 -- 172.21.15.107:0/159724004 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4824065e60 msgr2=0x7f4824068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4824065e60 0x7f4824068320 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4830000f70 tx=0x7f4830009040 comp rx=0 tx=0).stop 2023-10-24T23:53:09.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 -- 172.21.15.107:0/159724004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48400c1c30 msgr2=0x7f4840070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48400c1c30 0x7f4840070ee0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f483c01a170 tx=0x7f483c0192e0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 -- 172.21.15.107:0/159724004 shutdown_connections 2023-10-24T23:53:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f48400c4cf0 0x7f4840071420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f48400c1c30 0x7f4840070ee0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4824065e60 0x7f4824068320 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 --2- 172.21.15.107:0/159724004 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f48400a50d0 0x7f484015fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 -- 172.21.15.107:0/159724004 >> 172.21.15.107:0/159724004 conn(0x7f48400c2860 msgr2=0x7f48400c2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:09.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 -- 172.21.15.107:0/159724004 shutdown_connections 2023-10-24T23:53:09.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.139+0000 7f4847a37640 1 -- 172.21.15.107:0/159724004 wait complete. 2023-10-24T23:53:09.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-24T23:53:09.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.275+0000 7fe90b314640 1 Processor -- start 2023-10-24T23:53:09.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.275+0000 7fe90b314640 1 -- start start 2023-10-24T23:53:09.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe90b314640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9040a9820 0x7fe9040ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:09.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe90b314640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90406ee40 0x7fe9040aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:09.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe90b314640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90406f730 con 0x7fe90414fd40 2023-10-24T23:53:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe90b314640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9040aa750 con 0x7fe9040a9820 2023-10-24T23:53:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe90b314640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe904156a40 con 0x7fe90406ee40 2023-10-24T23:53:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe9098b1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90406ee40 0x7fe9040aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe9088af640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9040a9820 0x7fe9040ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe9098b1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90406ee40 0x7fe9040aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:36058/0 (socket says 172.21.15.107:36058) 2023-10-24T23:53:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe9098b1640 1 -- 172.21.15.107:0/2149959638 learned_addr learned my addr 172.21.15.107:0/2149959638 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe9098b1640 1 -- 172.21.15.107:0/2149959638 >> v2:172.21.15.110:3300/0 conn(0x7fe9040a9820 msgr2=0x7fe9040ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe9098b1640 1 --2- 172.21.15.107:0/2149959638 >> v2:172.21.15.110:3300/0 conn(0x7fe9040a9820 0x7fe9040ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe9098b1640 1 -- 172.21.15.107:0/2149959638 >> v1:172.21.15.110:6789/0 conn(0x7fe90414fd40 legacy=0x7fe9040c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe9098b1640 1 -- 172.21.15.107:0/2149959638 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe904156710 con 0x7fe90406ee40 2023-10-24T23:53:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.279+0000 7fe9098b1640 1 --2- 172.21.15.107:0/2149959638 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90406ee40 0x7fe9040aa210 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009e50 tx=0x7fe8fc01c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ac9cdd642ac09b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe8fbfff640 1 -- 172.21.15.107:0/2149959638 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc0a5720 con 0x7fe90406ee40 2023-10-24T23:53:09.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe8fbfff640 1 -- 172.21.15.107:0/2149959638 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8fc0a5880 con 0x7fe90406ee40 2023-10-24T23:53:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe8fbfff640 1 -- 172.21.15.107:0/2149959638 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc0a5c10 con 0x7fe90406ee40 2023-10-24T23:53:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 -- 172.21.15.107:0/2149959638 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90406ee40 msgr2=0x7fe9040aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 --2- 172.21.15.107:0/2149959638 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90406ee40 0x7fe9040aa210 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009e50 tx=0x7fe8fc01c8c0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 -- 172.21.15.107:0/2149959638 shutdown_connections 2023-10-24T23:53:09.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 --2- 172.21.15.107:0/2149959638 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90406ee40 0x7fe9040aa210 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 --2- 172.21.15.107:0/2149959638 >> v2:172.21.15.110:3300/0 conn(0x7fe9040a9820 0x7fe9040ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 -- 172.21.15.107:0/2149959638 >> 172.21.15.107:0/2149959638 conn(0x7fe9040c2950 msgr2=0x7fe9040c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 -- 172.21.15.107:0/2149959638 shutdown_connections 2023-10-24T23:53:09.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 -- 172.21.15.107:0/2149959638 wait complete. 2023-10-24T23:53:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 Processor -- start 2023-10-24T23:53:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 -- start start 2023-10-24T23:53:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90406ee40 0x7fe9041eb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:09.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90414fd40 0x7fe904146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9040c56a0 0x7fe9040bf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90415b600 con 0x7fe9040c56a0 2023-10-24T23:53:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90415a080 con 0x7fe90406ee40 2023-10-24T23:53:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe90b314640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe904159750 con 0x7fe90414fd40 2023-10-24T23:53:09.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe9098b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9040c56a0 0x7fe9040bf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:09.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe9098b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9040c56a0 0x7fe9040bf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43546/0 (socket says 172.21.15.107:43546) 2023-10-24T23:53:09.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe9098b1640 1 -- 172.21.15.107:0/52587118 learned_addr learned my addr 172.21.15.107:0/52587118 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:09.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe9088af640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90414fd40 0x7fe904146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:09.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe9090b0640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90406ee40 0x7fe9041eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:09.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe9098b1640 1 -- 172.21.15.107:0/52587118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90414fd40 msgr2=0x7fe904146fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe9098b1640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90414fd40 0x7fe904146fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe9098b1640 1 -- 172.21.15.107:0/52587118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90406ee40 msgr2=0x7fe9041eb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe9098b1640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90406ee40 0x7fe9041eb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.283+0000 7fe9098b1640 1 -- 172.21.15.107:0/52587118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc009b30 con 0x7fe9040c56a0 2023-10-24T23:53:09.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe9098b1640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9040c56a0 0x7fe9040bf980 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc0af040 tx=0x7fe8fc019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:09.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe8f9ffb640 1 -- 172.21.15.107:0/52587118 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc0b8670 con 0x7fe9040c56a0 2023-10-24T23:53:09.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe8f9ffb640 1 -- 172.21.15.107:0/52587118 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8fc0b87d0 con 0x7fe9040c56a0 2023-10-24T23:53:09.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe8f9ffb640 1 -- 172.21.15.107:0/52587118 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc0b8ba0 con 0x7fe9040c56a0 2023-10-24T23:53:09.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe90b314640 1 -- 172.21.15.107:0/52587118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe904159a30 con 0x7fe9040c56a0 2023-10-24T23:53:09.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe90b314640 1 -- 172.21.15.107:0/52587118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90414f440 con 0x7fe9040c56a0 2023-10-24T23:53:09.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe90b314640 1 -- 172.21.15.107:0/52587118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9040c24a0 con 0x7fe9040c56a0 2023-10-24T23:53:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe8f9ffb640 1 -- 172.21.15.107:0/52587118 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe8fc0b8d00 con 0x7fe9040c56a0 2023-10-24T23:53:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe8f9ffb640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8e8065e90 0x7fe8e8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe8f9ffb640 1 -- 172.21.15.107:0/52587118 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7fe8fc11ede0 con 0x7fe9040c56a0 2023-10-24T23:53:09.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe9090b0640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8e8065e90 0x7fe8e8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.287+0000 7fe9090b0640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8e8065e90 0x7fe8e8068350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe9000015b0 tx=0x7fe900009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:09.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.291+0000 7fe8f9ffb640 1 -- 172.21.15.107:0/52587118 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe8fc0ec5f0 con 0x7fe9040c56a0 2023-10-24T23:53:09.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe90b314640 1 -- 172.21.15.107:0/52587118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe904070840 con 0x7fe9040c56a0 2023-10-24T23:53:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe8f9ffb640 1 -- 172.21.15.107:0/52587118 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v23) v1 ==== 84+0+258 (secure 0 0 0) 0x7fe8fc0efc10 con 0x7fe9040c56a0 2023-10-24T23:53:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:09.443 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2023-10-24T23:53:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-10-24T23:53:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2023-10-24T23:53:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2023-10-24T23:53:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-10-24T23:53:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-10-24T23:53:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 2, 2023-10-24T23:53:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": "rbd" 2023-10-24T23:53:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-10-24T23:53:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-10-24T23:53:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 3, 2023-10-24T23:53:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": "a_metadata" 2023-10-24T23:53:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-10-24T23:53:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2023-10-24T23:53:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 4, 2023-10-24T23:53:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": "a_data" 2023-10-24T23:53:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-10-24T23:53:09.446 INFO:teuthology.orchestra.run.smithi107.stdout:] 2023-10-24T23:53:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe90b314640 1 -- 172.21.15.107:0/52587118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8e8065e90 msgr2=0x7fe8e8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe90b314640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8e8065e90 0x7fe8e8068350 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe9000015b0 tx=0x7fe900009040 comp rx=0 tx=0).stop 2023-10-24T23:53:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe90b314640 1 -- 172.21.15.107:0/52587118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9040c56a0 msgr2=0x7fe9040bf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe90b314640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9040c56a0 0x7fe9040bf980 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc0af040 tx=0x7fe8fc019660 comp rx=0 tx=0).stop 2023-10-24T23:53:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe90b314640 1 -- 172.21.15.107:0/52587118 shutdown_connections 2023-10-24T23:53:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe90b314640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9040c56a0 0x7fe9040bf980 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe90b314640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8e8065e90 0x7fe8e8068350 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe90b314640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe90414fd40 0x7fe904146fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe90b314640 1 --2- 172.21.15.107:0/52587118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe90406ee40 0x7fe9041eb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.439+0000 7fe90b314640 1 -- 172.21.15.107:0/52587118 >> 172.21.15.107:0/52587118 conn(0x7fe9040c2950 msgr2=0x7fe9040c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.443+0000 7fe90b314640 1 -- 172.21.15.107:0/52587118 shutdown_connections 2023-10-24T23:53:09.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.443+0000 7fe90b314640 1 -- 172.21.15.107:0/52587118 wait complete. 2023-10-24T23:53:09.460 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'b' 2023-10-24T23:53:09.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_metadata --pg_num_min 64 2023-10-24T23:53:09.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fcfd71640 1 Processor -- start 2023-10-24T23:53:09.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fcfd71640 1 -- start start 2023-10-24T23:53:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fcfd71640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fc8155310 0x7f1fc8158150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fcfd71640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc8155820 con 0x7f1fc80c8100 2023-10-24T23:53:09.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fcfd71640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc80c6cb0 con 0x7f1fc8151dd0 2023-10-24T23:53:09.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fcfd71640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc80c6f90 con 0x7f1fc8155310 2023-10-24T23:53:09.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fce30e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1fc80c8100 0x7f1fc8158690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39218/0 (socket says 172.21.15.107:39218) 2023-10-24T23:53:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fce30e640 1 -- 172.21.15.107:0/1780227762 learned_addr learned my addr 172.21.15.107:0/1780227762 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fcd30c640 1 --2- 172.21.15.107:0/1780227762 >> v2:172.21.15.107:3300/0 conn(0x7f1fc8155310 0x7f1fc8158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:09.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fccb0b640 1 -- 172.21.15.107:0/1780227762 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421777858 0 0) 0x7f1fc8155820 con 0x7f1fc80c8100 2023-10-24T23:53:09.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fccb0b640 1 -- 172.21.15.107:0/1780227762 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb4003530 con 0x7f1fc80c8100 2023-10-24T23:53:09.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fcd30c640 1 -- 172.21.15.107:0/1780227762 >> v1:172.21.15.110:6789/0 conn(0x7f1fc8151dd0 legacy=0x7f1fc80c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fcd30c640 1 -- 172.21.15.107:0/1780227762 >> v1:172.21.15.107:6789/0 conn(0x7f1fc80c8100 legacy=0x7f1fc8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.575+0000 7f1fcd30c640 1 -- 172.21.15.107:0/1780227762 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc80c7270 con 0x7f1fc8155310 2023-10-24T23:53:09.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcd30c640 1 --2- 172.21.15.107:0/1780227762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc8155310 0x7f1fc8158150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40097c0 tx=0x7f1fc401ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80d09325f7d50ac server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:09.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fccb0b640 1 -- 172.21.15.107:0/1780227762 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc40ab610 con 0x7f1fc8155310 2023-10-24T23:53:09.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fccb0b640 1 -- 172.21.15.107:0/1780227762 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc40ab770 con 0x7f1fc8155310 2023-10-24T23:53:09.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fccb0b640 1 -- 172.21.15.107:0/1780227762 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc40abb00 con 0x7f1fc8155310 2023-10-24T23:53:09.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 -- 172.21.15.107:0/1780227762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc8155310 msgr2=0x7f1fc8158150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 --2- 172.21.15.107:0/1780227762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc8155310 0x7f1fc8158150 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40097c0 tx=0x7f1fc401ce50 comp rx=0 tx=0).stop 2023-10-24T23:53:09.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 -- 172.21.15.107:0/1780227762 shutdown_connections 2023-10-24T23:53:09.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 --2- 172.21.15.107:0/1780227762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc8155310 0x7f1fc8158150 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 -- 172.21.15.107:0/1780227762 >> 172.21.15.107:0/1780227762 conn(0x7f1fc80abdb0 msgr2=0x7f1fc80ac1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:09.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 -- 172.21.15.107:0/1780227762 shutdown_connections 2023-10-24T23:53:09.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 -- 172.21.15.107:0/1780227762 wait complete. 2023-10-24T23:53:09.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 Processor -- start 2023-10-24T23:53:09.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 -- start start 2023-10-24T23:53:09.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc80c8100 0x7f1fc81509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:09.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc8151dd0 0x7f1fc814cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:09.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc8155310 0x7f1fc814d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:09.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc8152a10 con 0x7f1fc8155310 2023-10-24T23:53:09.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc81f0500 con 0x7f1fc80c8100 2023-10-24T23:53:09.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcfd71640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc8156e40 con 0x7f1fc8151dd0 2023-10-24T23:53:09.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fce30e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc8155310 0x7f1fc814d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:09.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fce30e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc8155310 0x7f1fc814d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43562/0 (socket says 172.21.15.107:43562) 2023-10-24T23:53:09.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fce30e640 1 -- 172.21.15.107:0/756930218 learned_addr learned my addr 172.21.15.107:0/756930218 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:09.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcdb0d640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc80c8100 0x7f1fc81509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:09.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fce30e640 1 -- 172.21.15.107:0/756930218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc8151dd0 msgr2=0x7f1fc814cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcd30c640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc8151dd0 0x7f1fc814cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:09.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fce30e640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc8151dd0 0x7f1fc814cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fce30e640 1 -- 172.21.15.107:0/756930218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc80c8100 msgr2=0x7f1fc81509e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fce30e640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc80c8100 0x7f1fc81509e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fce30e640 1 -- 172.21.15.107:0/756930218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4009420 con 0x7f1fc8155310 2023-10-24T23:53:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fcdb0d640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc80c8100 0x7f1fc81509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fce30e640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc8155310 0x7f1fc814d020 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000cb40 tx=0x7f1fc000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:09.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.579+0000 7f1fba7fc640 1 -- 172.21.15.107:0/756930218 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc000aca0 con 0x7f1fc8155310 2023-10-24T23:53:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.583+0000 7f1fba7fc640 1 -- 172.21.15.107:0/756930218 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc000ae00 con 0x7f1fc8155310 2023-10-24T23:53:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.583+0000 7f1fcfd71640 1 -- 172.21.15.107:0/756930218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc81f09f0 con 0x7f1fc8155310 2023-10-24T23:53:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.583+0000 7f1fba7fc640 1 -- 172.21.15.107:0/756930218 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0012620 con 0x7f1fc8155310 2023-10-24T23:53:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.583+0000 7f1fcfd71640 1 -- 172.21.15.107:0/756930218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc81f0710 con 0x7f1fc8155310 2023-10-24T23:53:09.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.583+0000 7f1fcfd71640 1 -- 172.21.15.107:0/756930218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc80746e0 con 0x7f1fc8155310 2023-10-24T23:53:09.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.583+0000 7f1fba7fc640 1 -- 172.21.15.107:0/756930218 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1fc0007070 con 0x7f1fc8155310 2023-10-24T23:53:09.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.583+0000 7f1fba7fc640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 0x7f1fac068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:09.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.583+0000 7f1fba7fc640 1 -- 172.21.15.107:0/756930218 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6042+0+0 (secure 0 0 0) 0x7f1fc0083800 con 0x7f1fc8155310 2023-10-24T23:53:09.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.583+0000 7f1fcdb0d640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 0x7f1fac068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:09.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.583+0000 7f1fcdb0d640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 0x7f1fac068320 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1fc81512a0 tx=0x7f1fbc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:09.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.587+0000 7f1fba7fc640 1 -- 172.21.15.107:0/756930218 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1fc0050ef0 con 0x7f1fc8155310 2023-10-24T23:53:09.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:09.743+0000 7f1fcfd71640 1 -- 172.21.15.107:0/756930218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1fc8152e00 con 0x7f1fc8155310 2023-10-24T23:53:10.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.387+0000 7f1fba7fc640 1 -- 172.21.15.107:0/756930218 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64}]=0 pool 'b_metadata' created v24) v1 ==== 128+0+0 (secure 0 0 0) 0x7f1fc0054510 con 0x7f1fc8155310 2023-10-24T23:53:10.392 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'b_metadata' created 2023-10-24T23:53:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.391+0000 7f1fcfd71640 1 -- 172.21.15.107:0/756930218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 msgr2=0x7f1fac068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.391+0000 7f1fcfd71640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 0x7f1fac068320 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1fc81512a0 tx=0x7f1fbc009040 comp rx=0 tx=0).stop 2023-10-24T23:53:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.391+0000 7f1fcfd71640 1 -- 172.21.15.107:0/756930218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc8155310 msgr2=0x7f1fc814d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:10.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.391+0000 7f1fcfd71640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc8155310 0x7f1fc814d020 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000cb40 tx=0x7f1fc000a060 comp rx=0 tx=0).stop 2023-10-24T23:53:10.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.391+0000 7f1fcfd71640 1 -- 172.21.15.107:0/756930218 shutdown_connections 2023-10-24T23:53:10.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.391+0000 7f1fcfd71640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc8155310 0x7f1fc814d020 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:10.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.391+0000 7f1fcfd71640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 0x7f1fac068320 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:10.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.391+0000 7f1fcfd71640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc8151dd0 0x7f1fc814cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:10.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.391+0000 7f1fcfd71640 1 --2- 172.21.15.107:0/756930218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc80c8100 0x7f1fc81509e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:10.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.391+0000 7f1fcfd71640 1 -- 172.21.15.107:0/756930218 >> 172.21.15.107:0/756930218 conn(0x7f1fc80abdb0 msgr2=0x7f1fc80c4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:10.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.395+0000 7f1fcfd71640 1 -- 172.21.15.107:0/756930218 shutdown_connections 2023-10-24T23:53:10.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.395+0000 7f1fcfd71640 1 -- 172.21.15.107:0/756930218 wait complete. 2023-10-24T23:53:10.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-24T23:53:10.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04c338640 1 Processor -- start 2023-10-24T23:53:10.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04c338640 1 -- start start 2023-10-24T23:53:10.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04c338640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff04406b360 0x7ff044069d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:10.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04c338640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff0440bd930 0x7ff04406a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:10.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04c338640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04406a7c0 con 0x7ff0440af9b0 2023-10-24T23:53:10.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04c338640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0440c5870 con 0x7ff0440bd930 2023-10-24T23:53:10.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04c338640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440c8180 con 0x7ff04406b360 2023-10-24T23:53:10.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04a8d5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff0440bd930 0x7ff04406a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:10.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff0498d3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff04406b360 0x7ff044069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:10.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04a8d5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff0440bd930 0x7ff04406a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60472/0 (socket says 172.21.15.107:60472) 2023-10-24T23:53:10.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04a8d5640 1 -- 172.21.15.107:0/3595543110 learned_addr learned my addr 172.21.15.107:0/3595543110 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:10.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04a8d5640 1 -- 172.21.15.107:0/3595543110 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff04406b360 msgr2=0x7ff044069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:10.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04a8d5640 1 --2- 172.21.15.107:0/3595543110 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff04406b360 0x7ff044069d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:10.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04a8d5640 1 -- 172.21.15.107:0/3595543110 >> v1:172.21.15.110:6789/0 conn(0x7ff0440af9b0 legacy=0x7ff0440c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:10.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.551+0000 7ff04a8d5640 1 -- 172.21.15.107:0/3595543110 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff044155430 con 0x7ff0440bd930 2023-10-24T23:53:10.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff04a8d5640 1 --2- 172.21.15.107:0/3595543110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0440bd930 0x7ff04406a280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff034009870 tx=0x7ff03401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93ae7d0cfa6fee3 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:10.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff0490d2640 1 -- 172.21.15.107:0/3595543110 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0340ab680 con 0x7ff0440bd930 2023-10-24T23:53:10.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff0490d2640 1 -- 172.21.15.107:0/3595543110 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0340ab7e0 con 0x7ff0440bd930 2023-10-24T23:53:10.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff0490d2640 1 -- 172.21.15.107:0/3595543110 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0340abb70 con 0x7ff0440bd930 2023-10-24T23:53:10.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff04c338640 1 -- 172.21.15.107:0/3595543110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0440bd930 msgr2=0x7ff04406a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:10.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff04c338640 1 --2- 172.21.15.107:0/3595543110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0440bd930 0x7ff04406a280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff034009870 tx=0x7ff03401c480 comp rx=0 tx=0).stop 2023-10-24T23:53:10.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff04c338640 1 -- 172.21.15.107:0/3595543110 shutdown_connections 2023-10-24T23:53:10.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff04c338640 1 --2- 172.21.15.107:0/3595543110 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0440bd930 0x7ff04406a280 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:10.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff04c338640 1 --2- 172.21.15.107:0/3595543110 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff04406b360 0x7ff044069d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:10.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff04c338640 1 -- 172.21.15.107:0/3595543110 >> 172.21.15.107:0/3595543110 conn(0x7ff044071b00 msgr2=0x7ff0440abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:10.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff04c338640 1 -- 172.21.15.107:0/3595543110 shutdown_connections 2023-10-24T23:53:10.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.555+0000 7ff04c338640 1 -- 172.21.15.107:0/3595543110 wait complete. 2023-10-24T23:53:10.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04c338640 1 Processor -- start 2023-10-24T23:53:10.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04c338640 1 -- start start 2023-10-24T23:53:10.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04c338640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff04406b360 0x7ff044151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:10.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04c338640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0440af9b0 0x7ff0440c1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:10.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04c338640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0440bd930 0x7ff0440c1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:10.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04c338640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440ac3d0 con 0x7ff04406b360 2023-10-24T23:53:10.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04c338640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440ab280 con 0x7ff0440bd930 2023-10-24T23:53:10.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04c338640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0440abea0 con 0x7ff0440af9b0 2023-10-24T23:53:10.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04a0d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff04406b360 0x7ff044151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:10.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04a0d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff04406b360 0x7ff044151890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43578/0 (socket says 172.21.15.107:43578) 2023-10-24T23:53:10.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04a0d4640 1 -- 172.21.15.107:0/3931754433 learned_addr learned my addr 172.21.15.107:0/3931754433 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:10.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff0498d3640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0440af9b0 0x7ff0440c1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:10.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04a8d5640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0440bd930 0x7ff0440c1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:10.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04a0d4640 1 -- 172.21.15.107:0/3931754433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0440af9b0 msgr2=0x7ff0440c1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04a0d4640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0440af9b0 0x7ff0440c1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04a0d4640 1 -- 172.21.15.107:0/3931754433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0440bd930 msgr2=0x7ff0440c1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04a0d4640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0440bd930 0x7ff0440c1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04a0d4640 1 -- 172.21.15.107:0/3931754433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0340094d0 con 0x7ff04406b360 2023-10-24T23:53:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04a8d5640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0440bd930 0x7ff0440c1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:10.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04a0d4640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff04406b360 0x7ff044151890 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00c7f0 tx=0x7ff03c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff03affd640 1 -- 172.21.15.107:0/3931754433 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c003b80 con 0x7ff04406b360 2023-10-24T23:53:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff03affd640 1 -- 172.21.15.107:0/3931754433 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03c003ce0 con 0x7ff04406b360 2023-10-24T23:53:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04c338640 1 -- 172.21.15.107:0/3931754433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0440ab4d0 con 0x7ff04406b360 2023-10-24T23:53:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff03affd640 1 -- 172.21.15.107:0/3931754433 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c00aa90 con 0x7ff04406b360 2023-10-24T23:53:10.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04c338640 1 -- 172.21.15.107:0/3931754433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0440aa870 con 0x7ff04406b360 2023-10-24T23:53:10.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff04c338640 1 -- 172.21.15.107:0/3931754433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0440aa6d0 con 0x7ff04406b360 2023-10-24T23:53:10.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.559+0000 7ff03affd640 1 -- 172.21.15.107:0/3931754433 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff03c00abf0 con 0x7ff04406b360 2023-10-24T23:53:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.563+0000 7ff03affd640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff028065e90 0x7ff028068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.563+0000 7ff03affd640 1 -- 172.21.15.107:0/3931754433 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6407+0+0 (secure 0 0 0) 0x7ff03c083850 con 0x7ff04406b360 2023-10-24T23:53:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.563+0000 7ff0498d3640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff028065e90 0x7ff028068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.563+0000 7ff0498d3640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff028065e90 0x7ff028068350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff0441542f0 tx=0x7ff040008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:10.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.567+0000 7ff03affd640 1 -- 172.21.15.107:0/3931754433 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff0440aa6d0 con 0x7ff04406b360 2023-10-24T23:53:10.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:10.727+0000 7ff04c338640 1 -- 172.21.15.107:0/3931754433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff0440aa6d0 con 0x7ff04406b360 2023-10-24T23:53:11.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.403+0000 7ff03affd640 1 -- 172.21.15.107:0/3931754433 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'b_data' created v25) v1 ==== 160+0+0 (secure 0 0 0) 0x7ff0440aa6d0 con 0x7ff04406b360 2023-10-24T23:53:11.406 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'b_data' created 2023-10-24T23:53:11.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.411+0000 7ff04c338640 1 -- 172.21.15.107:0/3931754433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff028065e90 msgr2=0x7ff028068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.411+0000 7ff04c338640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff028065e90 0x7ff028068350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff0441542f0 tx=0x7ff040008040 comp rx=0 tx=0).stop 2023-10-24T23:53:11.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.411+0000 7ff04c338640 1 -- 172.21.15.107:0/3931754433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff04406b360 msgr2=0x7ff044151890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.411+0000 7ff04c338640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff04406b360 0x7ff044151890 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00c7f0 tx=0x7ff03c00ccb0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.411+0000 7ff04c338640 1 -- 172.21.15.107:0/3931754433 shutdown_connections 2023-10-24T23:53:11.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.411+0000 7ff04c338640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0440bd930 0x7ff0440c1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.411+0000 7ff04c338640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff028065e90 0x7ff028068350 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.411+0000 7ff04c338640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0440af9b0 0x7ff0440c1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.411+0000 7ff04c338640 1 --2- 172.21.15.107:0/3931754433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff04406b360 0x7ff044151890 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.411+0000 7ff04c338640 1 -- 172.21.15.107:0/3931754433 >> 172.21.15.107:0/3931754433 conn(0x7ff044071b00 msgr2=0x7ff0440adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:11.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.415+0000 7ff04c338640 1 -- 172.21.15.107:0/3931754433 shutdown_connections 2023-10-24T23:53:11.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.415+0000 7ff04c338640 1 -- 172.21.15.107:0/3931754433 wait complete. 2023-10-24T23:53:11.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new b b_metadata b_data 2023-10-24T23:53:11.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.567+0000 7ff314022640 1 Processor -- start 2023-10-24T23:53:11.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.567+0000 7ff314022640 1 -- start start 2023-10-24T23:53:11.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff314022640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff30c0c19c0 0x7ff30c0c1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff314022640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30c06ca00 con 0x7ff30c14aca0 2023-10-24T23:53:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff314022640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30c06cce0 con 0x7ff30c06c460 2023-10-24T23:53:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff314022640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff30c06cfc0 con 0x7ff30c0c19c0 2023-10-24T23:53:11.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff311dbe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff30c14aca0 0x7ff30c151760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39232/0 (socket says 172.21.15.107:39232) 2023-10-24T23:53:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff311dbe640 1 -- 172.21.15.107:0/308720023 learned_addr learned my addr 172.21.15.107:0/308720023 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364716449 0 0) 0x7ff30c06ca00 con 0x7ff30c14aca0 2023-10-24T23:53:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f8003530 con 0x7ff30c14aca0 2023-10-24T23:53:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff3125bf640 1 --2- 172.21.15.107:0/308720023 >> v2:172.21.15.110:3300/0 conn(0x7ff30c0c19c0 0x7ff30c0c1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3200852857 0 0) 0x7ff2f8003530 con 0x7ff30c14aca0 2023-10-24T23:53:11.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 >> v2:172.21.15.110:3300/0 conn(0x7ff30c0c19c0 msgr2=0x7ff30c0c1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:11.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 --2- 172.21.15.107:0/308720023 >> v2:172.21.15.110:3300/0 conn(0x7ff30c0c19c0 0x7ff30c0c1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 >> v1:172.21.15.110:6789/0 conn(0x7ff30c06c460 legacy=0x7ff30c0c26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c0c11c0 con 0x7ff30c14aca0 2023-10-24T23:53:11.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7ff308001800 con 0x7ff30c14aca0 2023-10-24T23:53:11.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 >> v1:172.21.15.107:6789/0 conn(0x7ff30c14aca0 legacy=0x7ff30c151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 --2- 172.21.15.107:0/308720023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2f8003980 0x7ff2f8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 --2- 172.21.15.107:0/308720023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2f80047d0 0x7ff2f8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 --2- 172.21.15.107:0/308720023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2f8005190 0x7ff2f800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c06cfc0 con 0x7ff2f8005190 2023-10-24T23:53:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f800c030 con 0x7ff2f8003980 2023-10-24T23:53:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f800d300 con 0x7ff2f80047d0 2023-10-24T23:53:11.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff3125bf640 1 --2- 172.21.15.107:0/308720023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2f8005190 0x7ff2f800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff311dbe640 1 --2- 172.21.15.107:0/308720023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2f8003980 0x7ff2f8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff3115bd640 1 --2- 172.21.15.107:0/308720023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2f80047d0 0x7ff2f8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff3125bf640 1 -- 172.21.15.107:0/308720023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2f80047d0 msgr2=0x7ff2f8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:11.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff3125bf640 1 --2- 172.21.15.107:0/308720023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2f80047d0 0x7ff2f8004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff3125bf640 1 -- 172.21.15.107:0/308720023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2f8003980 msgr2=0x7ff2f8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:11.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff3125bf640 1 --2- 172.21.15.107:0/308720023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2f8003980 0x7ff2f8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff3125bf640 1 -- 172.21.15.107:0/308720023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f8003530 con 0x7ff2f8005190 2023-10-24T23:53:11.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff3125bf640 1 -- 172.21.15.107:0/308720023 reap_dead start 2023-10-24T23:53:11.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff3125bf640 1 --2- 172.21.15.107:0/308720023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2f8005190 0x7ff2f800b990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff304009fc0 tx=0x7ff30401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b89ffeb7408347 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:11.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3040a5cb0 con 0x7ff2f8005190 2023-10-24T23:53:11.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30400f040 con 0x7ff2f8005190 2023-10-24T23:53:11.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.571+0000 7ff310dbc640 1 -- 172.21.15.107:0/308720023 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff304003be0 con 0x7ff2f8005190 2023-10-24T23:53:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 -- 172.21.15.107:0/308720023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2f8005190 msgr2=0x7ff2f800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 --2- 172.21.15.107:0/308720023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2f8005190 0x7ff2f800b990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff304009fc0 tx=0x7ff30401cca0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 -- 172.21.15.107:0/308720023 shutdown_connections 2023-10-24T23:53:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 --2- 172.21.15.107:0/308720023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2f8005190 0x7ff2f800b990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 -- 172.21.15.107:0/308720023 >> 172.21.15.107:0/308720023 conn(0x7ff30c0c3140 msgr2=0x7ff30c0c3550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:11.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 -- 172.21.15.107:0/308720023 shutdown_connections 2023-10-24T23:53:11.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 -- 172.21.15.107:0/308720023 wait complete. 2023-10-24T23:53:11.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 Processor -- start 2023-10-24T23:53:11.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 -- start start 2023-10-24T23:53:11.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2f8005190 0x7ff30c148300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0c6170 0x7ff30c142990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff30c150a60 0x7ff30c142ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c0c4800 con 0x7ff30c0c6170 2023-10-24T23:53:11.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c1e9410 con 0x7ff2f8005190 2023-10-24T23:53:11.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c1e9940 con 0x7ff30c150a60 2023-10-24T23:53:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3115bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0c6170 0x7ff30c142990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3115bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0c6170 0x7ff30c142990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43590/0 (socket says 172.21.15.107:43590) 2023-10-24T23:53:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3115bd640 1 -- 172.21.15.107:0/3487736539 learned_addr learned my addr 172.21.15.107:0/3487736539 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3125bf640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff30c150a60 0x7ff30c142ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff311dbe640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2f8005190 0x7ff30c148300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3115bd640 1 -- 172.21.15.107:0/3487736539 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff30c150a60 msgr2=0x7ff30c142ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3115bd640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff30c150a60 0x7ff30c142ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3115bd640 1 -- 172.21.15.107:0/3487736539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2f8005190 msgr2=0x7ff30c148300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3115bd640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2f8005190 0x7ff30c148300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3115bd640 1 -- 172.21.15.107:0/3487736539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c071ed0 con 0x7ff30c0c6170 2023-10-24T23:53:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3125bf640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff30c150a60 0x7ff30c142ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff311dbe640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2f8005190 0x7ff30c148300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3115bd640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0c6170 0x7ff30c142990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc00d570 tx=0x7ff2fc00da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3027fc640 1 -- 172.21.15.107:0/3487736539 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc005480 con 0x7ff30c0c6170 2023-10-24T23:53:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3027fc640 1 -- 172.21.15.107:0/3487736539 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2fc003620 con 0x7ff30c0c6170 2023-10-24T23:53:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 -- 172.21.15.107:0/3487736539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30c1e9660 con 0x7ff30c0c6170 2023-10-24T23:53:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff3027fc640 1 -- 172.21.15.107:0/3487736539 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc003990 con 0x7ff30c0c6170 2023-10-24T23:53:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.575+0000 7ff314022640 1 -- 172.21.15.107:0/3487736539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30c143410 con 0x7ff30c0c6170 2023-10-24T23:53:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.579+0000 7ff314022640 1 -- 172.21.15.107:0/3487736539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30c073c00 con 0x7ff30c0c6170 2023-10-24T23:53:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.579+0000 7ff3027fc640 1 -- 172.21.15.107:0/3487736539 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff2fc003af0 con 0x7ff30c0c6170 2023-10-24T23:53:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.579+0000 7ff3027fc640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2f0065e80 0x7ff2f0068340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.579+0000 7ff3027fc640 1 -- 172.21.15.107:0/3487736539 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6784+0+0 (secure 0 0 0) 0x7ff2fc083860 con 0x7ff30c0c6170 2023-10-24T23:53:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.579+0000 7ff311dbe640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2f0065e80 0x7ff2f0068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.579+0000 7ff311dbe640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2f0065e80 0x7ff2f0068340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff308001390 tx=0x7ff308009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:11.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.583+0000 7ff3027fc640 1 -- 172.21.15.107:0/3487736539 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff2fc050c70 con 0x7ff30c0c6170 2023-10-24T23:53:11.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.771+0000 7ff314022640 1 -- 172.21.15.107:0/3487736539 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"} v 0) v1 -- 0x7ff30c04e130 con 0x7ff30c0c6170 2023-10-24T23:53:11.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.783+0000 7ff3027fc640 1 -- 172.21.15.107:0/3487736539 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"}]=0 new fs with metadata pool 5 and data pool 6 v9) v1 ==== 157+0+0 (secure 0 0 0) 0x7ff2fc054290 con 0x7ff30c0c6170 2023-10-24T23:53:11.785 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 5 and data pool 6 2023-10-24T23:53:11.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 -- 172.21.15.107:0/3487736539 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2f0065e80 msgr2=0x7ff2f0068340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2f0065e80 0x7ff2f0068340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff308001390 tx=0x7ff308009040 comp rx=0 tx=0).stop 2023-10-24T23:53:11.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 -- 172.21.15.107:0/3487736539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0c6170 msgr2=0x7ff30c142990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0c6170 0x7ff30c142990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc00d570 tx=0x7ff2fc00da30 comp rx=0 tx=0).stop 2023-10-24T23:53:11.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 -- 172.21.15.107:0/3487736539 shutdown_connections 2023-10-24T23:53:11.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2f0065e80 0x7ff2f0068340 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff30c150a60 0x7ff30c142ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff30c0c6170 0x7ff30c142990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 --2- 172.21.15.107:0/3487736539 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2f8005190 0x7ff30c148300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 -- 172.21.15.107:0/3487736539 >> 172.21.15.107:0/3487736539 conn(0x7ff30c0c3140 msgr2=0x7ff30c0aaf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:11.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 -- 172.21.15.107:0/3487736539 shutdown_connections 2023-10-24T23:53:11.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.787+0000 7ff314022640 1 -- 172.21.15.107:0/3487736539 wait complete. 2023-10-24T23:53:11.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:11.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.931+0000 7f51ad742640 1 Processor -- start 2023-10-24T23:53:11.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.931+0000 7f51ad742640 1 -- start start 2023-10-24T23:53:11.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.931+0000 7f51ad742640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f51a814b600 0x7f51a8151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.931+0000 7f51ad742640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f51a806af70 0x7f51a806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.931+0000 7f51ad742640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51a80c1860 con 0x7f51a806a230 2023-10-24T23:53:11.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.931+0000 7f51ad742640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a8154d50 con 0x7f51a806af70 2023-10-24T23:53:11.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.931+0000 7f51ad742640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51a81524f0 con 0x7f51a814b600 2023-10-24T23:53:11.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f519ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f51a806af70 0x7f51a806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f519ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f51a806af70 0x7f51a806c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43600/0 (socket says 172.21.15.107:43600) 2023-10-24T23:53:11.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f519ffff640 1 -- 172.21.15.107:0/3850771882 learned_addr learned my addr 172.21.15.107:0/3850771882 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51a6ffd640 1 --2- 172.21.15.107:0/3850771882 >> v2:172.21.15.110:3300/0 conn(0x7f51a814b600 0x7f51a8151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f519ffff640 1 -- 172.21.15.107:0/3850771882 >> v2:172.21.15.110:3300/0 conn(0x7f51a814b600 msgr2=0x7f51a8151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f519ffff640 1 --2- 172.21.15.107:0/3850771882 >> v2:172.21.15.110:3300/0 conn(0x7f51a814b600 0x7f51a8151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f519ffff640 1 -- 172.21.15.107:0/3850771882 >> v1:172.21.15.110:6789/0 conn(0x7f51a806a230 legacy=0x7f51a81551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f519ffff640 1 -- 172.21.15.107:0/3850771882 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51a814b9e0 con 0x7f51a806af70 2023-10-24T23:53:11.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f519ffff640 1 --2- 172.21.15.107:0/3850771882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51a806af70 0x7f51a806c4c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f51a806f860 tx=0x7f519001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17959027da007d68 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:11.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51a67fc640 1 -- 172.21.15.107:0/3850771882 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51900ab640 con 0x7f51a806af70 2023-10-24T23:53:11.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51a67fc640 1 -- 172.21.15.107:0/3850771882 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51900a7070 con 0x7f51a806af70 2023-10-24T23:53:11.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51a67fc640 1 -- 172.21.15.107:0/3850771882 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51900ab9c0 con 0x7f51a806af70 2023-10-24T23:53:11.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51ad742640 1 -- 172.21.15.107:0/3850771882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51a806af70 msgr2=0x7f51a806c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51ad742640 1 --2- 172.21.15.107:0/3850771882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51a806af70 0x7f51a806c4c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f51a806f860 tx=0x7f519001c440 comp rx=0 tx=0).stop 2023-10-24T23:53:11.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51ad742640 1 -- 172.21.15.107:0/3850771882 shutdown_connections 2023-10-24T23:53:11.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51ad742640 1 --2- 172.21.15.107:0/3850771882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51a806af70 0x7f51a806c4c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51ad742640 1 --2- 172.21.15.107:0/3850771882 >> v2:172.21.15.110:3300/0 conn(0x7f51a814b600 0x7f51a8151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51ad742640 1 -- 172.21.15.107:0/3850771882 >> 172.21.15.107:0/3850771882 conn(0x7f51a80c2950 msgr2=0x7f51a80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:11.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51ad742640 1 -- 172.21.15.107:0/3850771882 shutdown_connections 2023-10-24T23:53:11.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.935+0000 7f51ad742640 1 -- 172.21.15.107:0/3850771882 wait complete. 2023-10-24T23:53:11.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51ad742640 1 Processor -- start 2023-10-24T23:53:11.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51ad742640 1 -- start start 2023-10-24T23:53:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51ad742640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51a806a230 0x7f51a815a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51ad742640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f51a806af70 0x7f51a8071850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51ad742640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51a814b600 0x7f51a8071d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51ad742640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a814f430 con 0x7f51a814b600 2023-10-24T23:53:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51ad742640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a80722d0 con 0x7f51a806a230 2023-10-24T23:53:11.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51ad742640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51a815e060 con 0x7f51a806af70 2023-10-24T23:53:11.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51a77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51a814b600 0x7f51a8071d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51a77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51a814b600 0x7f51a8071d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43606/0 (socket says 172.21.15.107:43606) 2023-10-24T23:53:11.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51a77fe640 1 -- 172.21.15.107:0/2286109177 learned_addr learned my addr 172.21.15.107:0/2286109177 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:11.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f519ffff640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f51a806af70 0x7f51a8071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51a6ffd640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51a806a230 0x7f51a815a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51a77fe640 1 -- 172.21.15.107:0/2286109177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f51a806af70 msgr2=0x7f51a8071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51a77fe640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f51a806af70 0x7f51a8071850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51a77fe640 1 -- 172.21.15.107:0/2286109177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51a806a230 msgr2=0x7f51a815a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:11.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51a77fe640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51a806a230 0x7f51a815a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:11.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51a77fe640 1 -- 172.21.15.107:0/2286109177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51900094d0 con 0x7f51a814b600 2023-10-24T23:53:11.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51a6ffd640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51a806a230 0x7f51a815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:11.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51a77fe640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51a814b600 0x7f51a8071d90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f51a000c7d0 tx=0x7f51a000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:11.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f519f7fe640 1 -- 172.21.15.107:0/2286109177 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a0003b80 con 0x7f51a814b600 2023-10-24T23:53:11.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f519f7fe640 1 -- 172.21.15.107:0/2286109177 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51a0003ce0 con 0x7f51a814b600 2023-10-24T23:53:11.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51ad742640 1 -- 172.21.15.107:0/2286109177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a815e620 con 0x7f51a814b600 2023-10-24T23:53:11.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f519f7fe640 1 -- 172.21.15.107:0/2286109177 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a000aa30 con 0x7f51a814b600 2023-10-24T23:53:11.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51ad742640 1 -- 172.21.15.107:0/2286109177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a80c5a10 con 0x7f51a814b600 2023-10-24T23:53:11.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.939+0000 7f51ad742640 1 -- 172.21.15.107:0/2286109177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a815e7b0 con 0x7f51a814b600 2023-10-24T23:53:11.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.943+0000 7f519f7fe640 1 -- 172.21.15.107:0/2286109177 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f51a000ab90 con 0x7f51a814b600 2023-10-24T23:53:11.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.943+0000 7f519f7fe640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5188065e60 0x7f5188068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:11.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.943+0000 7f519f7fe640 1 -- 172.21.15.107:0/2286109177 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f51a0083910 con 0x7f51a814b600 2023-10-24T23:53:11.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.943+0000 7f51a6ffd640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5188065e60 0x7f5188068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:11.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.943+0000 7f51a6ffd640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5188065e60 0x7f5188068320 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51980097a0 tx=0x7f5198008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:11.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:11.947+0000 7f519f7fe640 1 -- 172.21.15.107:0/2286109177 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f51a0050cc0 con 0x7f51a814b600 2023-10-24T23:53:12.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.103+0000 7f51ad742640 1 -- 172.21.15.107:0/2286109177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51a80746e0 con 0x7f51a814b600 2023-10-24T23:53:12.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f519f7fe640 1 -- 172.21.15.107:0/2286109177 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+21544 (secure 0 0 0) 0x7f51a00542e0 con 0x7f51a814b600 2023-10-24T23:53:12.110 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:12.110 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":26,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:11.777031+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:12.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 -- 172.21.15.107:0/2286109177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5188065e60 msgr2=0x7f5188068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5188065e60 0x7f5188068320 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f51980097a0 tx=0x7f5198008040 comp rx=0 tx=0).stop 2023-10-24T23:53:12.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 -- 172.21.15.107:0/2286109177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51a814b600 msgr2=0x7f51a8071d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51a814b600 0x7f51a8071d90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f51a000c7d0 tx=0x7f51a000cc90 comp rx=0 tx=0).stop 2023-10-24T23:53:12.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 -- 172.21.15.107:0/2286109177 shutdown_connections 2023-10-24T23:53:12.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51a814b600 0x7f51a8071d90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5188065e60 0x7f5188068320 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f51a806af70 0x7f51a8071850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 --2- 172.21.15.107:0/2286109177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f51a806a230 0x7f51a815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 -- 172.21.15.107:0/2286109177 >> 172.21.15.107:0/2286109177 conn(0x7f51a80c2950 msgr2=0x7f51a8150d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:12.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 -- 172.21.15.107:0/2286109177 shutdown_connections 2023-10-24T23:53:12.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.107+0000 7f51ad742640 1 -- 172.21.15.107:0/2286109177 wait complete. 2023-10-24T23:53:12.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:12.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f6477c04640 1 Processor -- start 2023-10-24T23:53:12.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f6477c04640 1 -- start start 2023-10-24T23:53:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f6477c04640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6470071220 0x7f647006bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f6477c04640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64700c24c0 0x7f647006a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f6477c04640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6470150bc0 con 0x7f6470151960 2023-10-24T23:53:12.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f6477c04640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647006c7f0 con 0x7f64700c24c0 2023-10-24T23:53:12.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f6477c04640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647006c510 con 0x7f6470071220 2023-10-24T23:53:12.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f64761a1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64700c24c0 0x7f647006a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f64761a1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64700c24c0 0x7f647006a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43620/0 (socket says 172.21.15.107:43620) 2023-10-24T23:53:12.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f64761a1640 1 -- 172.21.15.107:0/3845631890 learned_addr learned my addr 172.21.15.107:0/3845631890 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:12.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f64759a0640 1 --2- 172.21.15.107:0/3845631890 >> v2:172.21.15.110:3300/0 conn(0x7f6470071220 0x7f647006bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f64761a1640 1 -- 172.21.15.107:0/3845631890 >> v2:172.21.15.110:3300/0 conn(0x7f6470071220 msgr2=0x7f647006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:12.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f64761a1640 1 --2- 172.21.15.107:0/3845631890 >> v2:172.21.15.110:3300/0 conn(0x7f6470071220 0x7f647006bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f64761a1640 1 -- 172.21.15.107:0/3845631890 >> v1:172.21.15.110:6789/0 conn(0x7f6470151960 legacy=0x7f6470071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f64761a1640 1 -- 172.21.15.107:0/3845631890 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6470150e80 con 0x7f64700c24c0 2023-10-24T23:53:12.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.243+0000 7f64759a0640 1 --2- 172.21.15.107:0/3845631890 >> v2:172.21.15.110:3300/0 conn(0x7f6470071220 0x7f647006bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:12.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f64761a1640 1 --2- 172.21.15.107:0/3845631890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64700c24c0 0x7f647006a9c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f647006d0a0 tx=0x7f646c01d6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8b04062d44400 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:12.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647499e640 1 -- 172.21.15.107:0/3845631890 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c0aa400 con 0x7f64700c24c0 2023-10-24T23:53:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647499e640 1 -- 172.21.15.107:0/3845631890 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c0aa560 con 0x7f64700c24c0 2023-10-24T23:53:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647499e640 1 -- 172.21.15.107:0/3845631890 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c0aa8f0 con 0x7f64700c24c0 2023-10-24T23:53:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 -- 172.21.15.107:0/3845631890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64700c24c0 msgr2=0x7f647006a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 --2- 172.21.15.107:0/3845631890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64700c24c0 0x7f647006a9c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f647006d0a0 tx=0x7f646c01d6a0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 -- 172.21.15.107:0/3845631890 shutdown_connections 2023-10-24T23:53:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 --2- 172.21.15.107:0/3845631890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64700c24c0 0x7f647006a9c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 --2- 172.21.15.107:0/3845631890 >> v2:172.21.15.110:3300/0 conn(0x7f6470071220 0x7f647006bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 -- 172.21.15.107:0/3845631890 >> 172.21.15.107:0/3845631890 conn(0x7f64700c2950 msgr2=0x7f64700c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 -- 172.21.15.107:0/3845631890 shutdown_connections 2023-10-24T23:53:12.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 -- 172.21.15.107:0/3845631890 wait complete. 2023-10-24T23:53:12.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 Processor -- start 2023-10-24T23:53:12.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 -- start start 2023-10-24T23:53:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6470071220 0x7f6470145f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64700c24c0 0x7f6470143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6470151960 0x7f64700bf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64701edb20 con 0x7f64700c24c0 2023-10-24T23:53:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64701492e0 con 0x7f6470071220 2023-10-24T23:53:12.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64700c58a0 con 0x7f6470151960 2023-10-24T23:53:12.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647519f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64700c24c0 0x7f6470143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647519f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64700c24c0 0x7f6470143210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43632/0 (socket says 172.21.15.107:43632) 2023-10-24T23:53:12.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647519f640 1 -- 172.21.15.107:0/2190709514 learned_addr learned my addr 172.21.15.107:0/2190709514 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:12.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f64761a1640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6470151960 0x7f64700bf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f64759a0640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6470071220 0x7f6470145f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647519f640 1 -- 172.21.15.107:0/2190709514 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6470151960 msgr2=0x7f64700bf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647519f640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6470151960 0x7f64700bf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647519f640 1 -- 172.21.15.107:0/2190709514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6470071220 msgr2=0x7f6470145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647519f640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6470071220 0x7f6470145f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647519f640 1 -- 172.21.15.107:0/2190709514 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c0094d0 con 0x7f64700c24c0 2023-10-24T23:53:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f64759a0640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6470071220 0x7f6470145f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f647519f640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64700c24c0 0x7f6470143210 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f646400c500 tx=0x7f646400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f64627fc640 1 -- 172.21.15.107:0/2190709514 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464005220 con 0x7f64700c24c0 2023-10-24T23:53:12.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f64627fc640 1 -- 172.21.15.107:0/2190709514 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6464005380 con 0x7f64700c24c0 2023-10-24T23:53:12.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 -- 172.21.15.107:0/2190709514 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6470149580 con 0x7f64700c24c0 2023-10-24T23:53:12.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f64627fc640 1 -- 172.21.15.107:0/2190709514 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464007270 con 0x7f64700c24c0 2023-10-24T23:53:12.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.247+0000 7f6477c04640 1 -- 172.21.15.107:0/2190709514 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6470143750 con 0x7f64700c24c0 2023-10-24T23:53:12.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.251+0000 7f6477c04640 1 -- 172.21.15.107:0/2190709514 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64701497a0 con 0x7f64700c24c0 2023-10-24T23:53:12.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.251+0000 7f64627fc640 1 -- 172.21.15.107:0/2190709514 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f64640073d0 con 0x7f64700c24c0 2023-10-24T23:53:12.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.251+0000 7f64627fc640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6454065e90 0x7f6454068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.251+0000 7f64627fc640 1 -- 172.21.15.107:0/2190709514 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f64640841b0 con 0x7f64700c24c0 2023-10-24T23:53:12.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.251+0000 7f64759a0640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6454065e90 0x7f6454068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.251+0000 7f64759a0640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6454065e90 0x7f6454068350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f64680094c0 tx=0x7f646800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:12.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.255+0000 7f64627fc640 1 -- 172.21.15.107:0/2190709514 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6464051560 con 0x7f64700c24c0 2023-10-24T23:53:12.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.403+0000 7f6477c04640 1 -- 172.21.15.107:0/2190709514 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64700746e0 con 0x7f64700c24c0 2023-10-24T23:53:12.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f64627fc640 1 -- 172.21.15.107:0/2190709514 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+21544 (secure 0 0 0) 0x7f6464054b80 con 0x7f64700c24c0 2023-10-24T23:53:12.410 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:12.410 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":26,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:11.777031+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:12.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 -- 172.21.15.107:0/2190709514 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6454065e90 msgr2=0x7f6454068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6454065e90 0x7f6454068350 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f64680094c0 tx=0x7f646800b040 comp rx=0 tx=0).stop 2023-10-24T23:53:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 -- 172.21.15.107:0/2190709514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64700c24c0 msgr2=0x7f6470143210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64700c24c0 0x7f6470143210 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f646400c500 tx=0x7f646400c9c0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 -- 172.21.15.107:0/2190709514 shutdown_connections 2023-10-24T23:53:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6454065e90 0x7f6454068350 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6470151960 0x7f64700bf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64700c24c0 0x7f6470143210 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 --2- 172.21.15.107:0/2190709514 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6470071220 0x7f6470145f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 -- 172.21.15.107:0/2190709514 >> 172.21.15.107:0/2190709514 conn(0x7f64700c2950 msgr2=0x7f647014ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:12.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 -- 172.21.15.107:0/2190709514 shutdown_connections 2023-10-24T23:53:12.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.407+0000 7f6477c04640 1 -- 172.21.15.107:0/2190709514 wait complete. 2023-10-24T23:53:12.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b standby_count_wanted 0 2023-10-24T23:53:12.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.531+0000 7f3f1c5f7640 1 Processor -- start 2023-10-24T23:53:12.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.531+0000 7f3f1c5f7640 1 -- start start 2023-10-24T23:53:12.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f1c5f7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3f140ab1a0 0x7f3f140c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f1c5f7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f14072720 0x7f3f140bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f1c5f7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f140c1ce0 con 0x7f3f140bceb0 2023-10-24T23:53:12.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f1c5f7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f140be1d0 con 0x7f3f140ab1a0 2023-10-24T23:53:12.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f1c5f7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140c0880 con 0x7f3f14072720 2023-10-24T23:53:12.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f19b92640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f14072720 0x7f3f140bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f1a393640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3f140ab1a0 0x7f3f140c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f1ab94640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3f140bceb0 0x7f3f140c4cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:39820/0 (socket says 172.21.15.107:39820) 2023-10-24T23:53:12.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f1ab94640 1 -- 172.21.15.107:0/1708826234 learned_addr learned my addr 172.21.15.107:0/1708826234 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:12.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f19b92640 1 -- 172.21.15.107:0/1708826234 >> v2:172.21.15.110:3300/0 conn(0x7f3f140ab1a0 msgr2=0x7f3f140c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f19b92640 1 --2- 172.21.15.107:0/1708826234 >> v2:172.21.15.110:3300/0 conn(0x7f3f140ab1a0 0x7f3f140c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f19b92640 1 -- 172.21.15.107:0/1708826234 >> v1:172.21.15.110:6789/0 conn(0x7f3f140bceb0 legacy=0x7f3f140c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f19b92640 1 -- 172.21.15.107:0/1708826234 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f140c1f30 con 0x7f3f14072720 2023-10-24T23:53:12.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.535+0000 7f3f1a393640 1 --2- 172.21.15.107:0/1708826234 >> v2:172.21.15.110:3300/0 conn(0x7f3f140ab1a0 0x7f3f140c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:53:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19b92640 1 --2- 172.21.15.107:0/1708826234 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f14072720 0x7f3f140bfb30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3f040097c0 tx=0x7f3f0401c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65abea805cb2f988 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19391640 1 -- 172.21.15.107:0/1708826234 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f040ab600 con 0x7f3f14072720 2023-10-24T23:53:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19391640 1 -- 172.21.15.107:0/1708826234 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f040a7050 con 0x7f3f14072720 2023-10-24T23:53:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19391640 1 -- 172.21.15.107:0/1708826234 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f040ab980 con 0x7f3f14072720 2023-10-24T23:53:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/1708826234 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f14072720 msgr2=0x7f3f140bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 --2- 172.21.15.107:0/1708826234 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f14072720 0x7f3f140bfb30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3f040097c0 tx=0x7f3f0401c330 comp rx=0 tx=0).stop 2023-10-24T23:53:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/1708826234 shutdown_connections 2023-10-24T23:53:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 --2- 172.21.15.107:0/1708826234 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f14072720 0x7f3f140bfb30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 --2- 172.21.15.107:0/1708826234 >> v2:172.21.15.110:3300/0 conn(0x7f3f140ab1a0 0x7f3f140c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/1708826234 >> 172.21.15.107:0/1708826234 conn(0x7f3f140c2860 msgr2=0x7f3f140c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/1708826234 shutdown_connections 2023-10-24T23:53:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/1708826234 wait complete. 2023-10-24T23:53:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 Processor -- start 2023-10-24T23:53:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 -- start start 2023-10-24T23:53:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f140ab1a0 0x7f3f140c3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f140bceb0 0x7f3f14152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f140a5300 0x7f3f14153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1406a390 con 0x7f3f140bceb0 2023-10-24T23:53:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1406b5a0 con 0x7f3f140ab1a0 2023-10-24T23:53:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1c5f7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f141ee610 con 0x7f3f140a5300 2023-10-24T23:53:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19b92640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f140bceb0 0x7f3f14152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19b92640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f140bceb0 0x7f3f14152e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50244/0 (socket says 172.21.15.107:50244) 2023-10-24T23:53:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19b92640 1 -- 172.21.15.107:0/3973015007 learned_addr learned my addr 172.21.15.107:0/3973015007 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1a393640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f140ab1a0 0x7f3f140c3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1ab94640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f140a5300 0x7f3f14153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19b92640 1 -- 172.21.15.107:0/3973015007 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f140a5300 msgr2=0x7f3f14153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19b92640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f140a5300 0x7f3f14153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19b92640 1 -- 172.21.15.107:0/3973015007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f140ab1a0 msgr2=0x7f3f140c3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19b92640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f140ab1a0 0x7f3f140c3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f19b92640 1 -- 172.21.15.107:0/3973015007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f10009420 con 0x7f3f140bceb0 2023-10-24T23:53:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1ab94640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f140a5300 0x7f3f14153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.539+0000 7f3f1a393640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f140ab1a0 0x7f3f140c3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f19b92640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f140bceb0 0x7f3f14152e00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3f0401a9e0 tx=0x7f3f040190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f0affd640 1 -- 172.21.15.107:0/3973015007 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f04006e20 con 0x7f3f140bceb0 2023-10-24T23:53:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f0affd640 1 -- 172.21.15.107:0/3973015007 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f040b4400 con 0x7f3f140bceb0 2023-10-24T23:53:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/3973015007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f04009420 con 0x7f3f140bceb0 2023-10-24T23:53:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f0affd640 1 -- 172.21.15.107:0/3973015007 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f040b47b0 con 0x7f3f140bceb0 2023-10-24T23:53:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/3973015007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1406bb20 con 0x7f3f140bceb0 2023-10-24T23:53:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/3973015007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f140beff0 con 0x7f3f140bceb0 2023-10-24T23:53:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f0affd640 1 -- 172.21.15.107:0/3973015007 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3f040b2040 con 0x7f3f140bceb0 2023-10-24T23:53:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f0affd640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3ef8065e90 0x7f3ef8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f0affd640 1 -- 172.21.15.107:0/3973015007 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f3f0411da80 con 0x7f3f140bceb0 2023-10-24T23:53:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f1a393640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3ef8065e90 0x7f3ef8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.543+0000 7f3f1a393640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3ef8065e90 0x7f3ef8068350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009d40 tx=0x7f3f10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.547+0000 7f3f0affd640 1 -- 172.21.15.107:0/3973015007 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3f040eada0 con 0x7f3f140bceb0 2023-10-24T23:53:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.731+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/3973015007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3f1404e130 con 0x7f3f140bceb0 2023-10-24T23:53:12.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.783+0000 7f3f0affd640 1 -- 172.21.15.107:0/3973015007 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"}]=0 v10) v1 ==== 113+0+0 (secure 0 0 0) 0x7f3f040ee3c0 con 0x7f3f140bceb0 2023-10-24T23:53:12.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.783+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/3973015007 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3ef8065e90 msgr2=0x7f3ef8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.783+0000 7f3f1c5f7640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3ef8065e90 0x7f3ef8068350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3f10009d40 tx=0x7f3f10008040 comp rx=0 tx=0).stop 2023-10-24T23:53:12.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.783+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/3973015007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f140bceb0 msgr2=0x7f3f14152e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.783+0000 7f3f1c5f7640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f140bceb0 0x7f3f14152e00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3f0401a9e0 tx=0x7f3f040190d0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.783+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/3973015007 shutdown_connections 2023-10-24T23:53:12.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.783+0000 7f3f1c5f7640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3ef8065e90 0x7f3ef8068350 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.783+0000 7f3f1c5f7640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3f140a5300 0x7f3f14153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.783+0000 7f3f1c5f7640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3f140bceb0 0x7f3f14152e00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.783+0000 7f3f1c5f7640 1 --2- 172.21.15.107:0/3973015007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3f140ab1a0 0x7f3f140c3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.783+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/3973015007 >> 172.21.15.107:0/3973015007 conn(0x7f3f140c2860 msgr2=0x7f3f14159d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:12.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.787+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/3973015007 shutdown_connections 2023-10-24T23:53:12.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.787+0000 7f3f1c5f7640 1 -- 172.21.15.107:0/3973015007 wait complete. 2023-10-24T23:53:12.808 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2023-10-24T23:53:12.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b allow_standby_replay false 2023-10-24T23:53:12.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba8dd3640 1 Processor -- start 2023-10-24T23:53:12.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba8dd3640 1 -- start start 2023-10-24T23:53:12.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba8dd3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba406b360 0x7fdba4069d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba8dd3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40c8180 con 0x7fdba40bd930 2023-10-24T23:53:12.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba8dd3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba4155430 con 0x7fdba40af9b0 2023-10-24T23:53:12.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba8dd3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40c5870 con 0x7fdba406b360 2023-10-24T23:53:12.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba2d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdba40bd930 0x7fdba415f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55194/0 (socket says 172.21.15.107:55194) 2023-10-24T23:53:12.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba2d9d640 1 -- 172.21.15.107:0/2243217045 learned_addr learned my addr 172.21.15.107:0/2243217045 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:12.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba1d9b640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba406b360 0x7fdba4069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605956254 0 0) 0x7fdba40c8180 con 0x7fdba40bd930 2023-10-24T23:53:12.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb90003530 con 0x7fdba40bd930 2023-10-24T23:53:12.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 268145341 0 0) 0x7fdb90003530 con 0x7fdba40bd930 2023-10-24T23:53:12.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba406b360 msgr2=0x7fdba4069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:12.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba406b360 0x7fdba4069d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 >> v1:172.21.15.110:6789/0 conn(0x7fdba40af9b0 legacy=0x7fdba40c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba4155710 con 0x7fdba40bd930 2023-10-24T23:53:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fdb9c001a00 con 0x7fdba40bd930 2023-10-24T23:53:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 >> v1:172.21.15.107:6789/0 conn(0x7fdba40bd930 legacy=0x7fdba415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba406b360 0x7fdb900039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 0x7fdb900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb90004d10 0x7fdb90005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40c5870 con 0x7fdb90003f30 2023-10-24T23:53:12.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9000bb20 con 0x7fdb90004d10 2023-10-24T23:53:12.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9000bce0 con 0x7fdba406b360 2023-10-24T23:53:12.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba1d9b640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 0x7fdb900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba259c640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba406b360 0x7fdb900039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba2d9d640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb90004d10 0x7fdb90005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba1d9b640 1 -- 172.21.15.107:0/2243217045 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba406b360 msgr2=0x7fdb900039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:12.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba1d9b640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba406b360 0x7fdb900039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba1d9b640 1 -- 172.21.15.107:0/2243217045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb90004d10 msgr2=0x7fdb90005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:12.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba1d9b640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb90004d10 0x7fdb90005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.927+0000 7fdba1d9b640 1 -- 172.21.15.107:0/2243217045 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb90003530 con 0x7fdb90003f30 2023-10-24T23:53:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba1d9b640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 0x7fdb900047d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c009ae0 tx=0x7fdb8c01ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f8f0920d104e890 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:12.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c0a5580 con 0x7fdb90003f30 2023-10-24T23:53:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c0a56e0 con 0x7fdb90003f30 2023-10-24T23:53:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba159a640 1 -- 172.21.15.107:0/2243217045 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c0a5a90 con 0x7fdb90003f30 2023-10-24T23:53:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 -- 172.21.15.107:0/2243217045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 msgr2=0x7fdb900047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 --2- 172.21.15.107:0/2243217045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 0x7fdb900047d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c009ae0 tx=0x7fdb8c01ca20 comp rx=0 tx=0).stop 2023-10-24T23:53:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba2d9d640 1 -- 172.21.15.107:0/2243217045 reap_dead start 2023-10-24T23:53:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 -- 172.21.15.107:0/2243217045 shutdown_connections 2023-10-24T23:53:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 -- 172.21.15.107:0/2243217045 >> 172.21.15.107:0/2243217045 conn(0x7fdba4071b00 msgr2=0x7fdba40abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 -- 172.21.15.107:0/2243217045 shutdown_connections 2023-10-24T23:53:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 -- 172.21.15.107:0/2243217045 wait complete. 2023-10-24T23:53:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 Processor -- start 2023-10-24T23:53:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 -- start start 2023-10-24T23:53:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 0x7fdba41f0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba41f0ca0 0x7fdba41e4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdba41e1f90 0x7fdba41e4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40c1450 con 0x7fdb90003f30 2023-10-24T23:53:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba41f1440 con 0x7fdba41e1f90 2023-10-24T23:53:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba8dd3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba40abf00 con 0x7fdba41f0ca0 2023-10-24T23:53:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 0x7fdba41f0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 0x7fdba41f0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50250/0 (socket says 172.21.15.107:50250) 2023-10-24T23:53:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba259c640 1 -- 172.21.15.107:0/3761055833 learned_addr learned my addr 172.21.15.107:0/3761055833 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba259c640 1 -- 172.21.15.107:0/3761055833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba41f0ca0 msgr2=0x7fdba41e4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba259c640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba41f0ca0 0x7fdba41e4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba259c640 1 -- 172.21.15.107:0/3761055833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdba41e1f90 msgr2=0x7fdba41e4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.931+0000 7fdba2d9d640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdba41e1f90 0x7fdba41e4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdba259c640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdba41e1f90 0x7fdba41e4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdba259c640 1 -- 172.21.15.107:0/3761055833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba40ac1e0 con 0x7fdb90003f30 2023-10-24T23:53:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdba259c640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 0x7fdba41f0760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400e550 tx=0x7fdb9400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdb9affd640 1 -- 172.21.15.107:0/3761055833 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb940052e0 con 0x7fdb90003f30 2023-10-24T23:53:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdb9affd640 1 -- 172.21.15.107:0/3761055833 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb94005440 con 0x7fdb90003f30 2023-10-24T23:53:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdba8dd3640 1 -- 172.21.15.107:0/3761055833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba41f1160 con 0x7fdb90003f30 2023-10-24T23:53:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdb9affd640 1 -- 172.21.15.107:0/3761055833 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94003820 con 0x7fdb90003f30 2023-10-24T23:53:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdba8dd3640 1 -- 172.21.15.107:0/3761055833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba40aa6d0 con 0x7fdb90003f30 2023-10-24T23:53:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdba8dd3640 1 -- 172.21.15.107:0/3761055833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba40aa8c0 con 0x7fdb90003f30 2023-10-24T23:53:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdb9affd640 1 -- 172.21.15.107:0/3761055833 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdb94003980 con 0x7fdb90003f30 2023-10-24T23:53:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdb9affd640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb84065e60 0x7fdb84068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:12.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdba1d9b640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb84065e60 0x7fdb84068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:12.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdb9affd640 1 -- 172.21.15.107:0/3761055833 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdb940838e0 con 0x7fdb90003f30 2023-10-24T23:53:12.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.935+0000 7fdba1d9b640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb84065e60 0x7fdb84068320 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c009ae0 tx=0x7fdb8c004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:12.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:12.939+0000 7fdb9affd640 1 -- 172.21.15.107:0/3761055833 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdb94050c90 con 0x7fdb90003f30 2023-10-24T23:53:13.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.123+0000 7fdba8dd3640 1 -- 172.21.15.107:0/3761055833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fdba4066560 con 0x7fdb90003f30 2023-10-24T23:53:13.278 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-24T23:53:13.275+0000 7ffa27194640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:53:13.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.799+0000 7fdb9affd640 1 -- 172.21.15.107:0/3761055833 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"}]=0 v11) v1 ==== 117+0+0 (secure 0 0 0) 0x7fdb940542b0 con 0x7fdb90003f30 2023-10-24T23:53:13.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 -- 172.21.15.107:0/3761055833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb84065e60 msgr2=0x7fdb84068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:13.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb84065e60 0x7fdb84068320 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c009ae0 tx=0x7fdb8c004740 comp rx=0 tx=0).stop 2023-10-24T23:53:13.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 -- 172.21.15.107:0/3761055833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 msgr2=0x7fdba41f0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:13.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 0x7fdba41f0760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400e550 tx=0x7fdb9400ea10 comp rx=0 tx=0).stop 2023-10-24T23:53:13.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 -- 172.21.15.107:0/3761055833 shutdown_connections 2023-10-24T23:53:13.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdba41e1f90 0x7fdba41e4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:13.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb84065e60 0x7fdb84068320 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:13.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba41f0ca0 0x7fdba41e4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:13.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 --2- 172.21.15.107:0/3761055833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb90003f30 0x7fdba41f0760 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:13.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 -- 172.21.15.107:0/3761055833 >> 172.21.15.107:0/3761055833 conn(0x7fdba4071b00 msgr2=0x7fdba40c93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:13.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 -- 172.21.15.107:0/3761055833 shutdown_connections 2023-10-24T23:53:13.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.803+0000 7fdba8dd3640 1 -- 172.21.15.107:0/3761055833 wait complete. 2023-10-24T23:53:13.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b session_timeout 300 2023-10-24T23:53:13.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.951+0000 7fa6240a3640 1 Processor -- start 2023-10-24T23:53:13.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.951+0000 7fa6240a3640 1 -- start start 2023-10-24T23:53:13.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa6240a3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa61c0c1b30 0x7fa61c072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:13.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa6240a3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c0befa0 con 0x7fa61c0a9300 2023-10-24T23:53:13.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa6240a3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c0a9b80 con 0x7fa61c0aa9f0 2023-10-24T23:53:13.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa6240a3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61c0aa0b0 con 0x7fa61c0c1b30 2023-10-24T23:53:13.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa62163e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa61c0a9300 0x7fa61c0aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55210/0 (socket says 172.21.15.107:55210) 2023-10-24T23:53:13.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa62163e640 1 -- 172.21.15.107:0/364568291 learned_addr learned my addr 172.21.15.107:0/364568291 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:13.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa621e3f640 1 --2- 172.21.15.107:0/364568291 >> v2:172.21.15.110:3300/0 conn(0x7fa61c0c1b30 0x7fa61c072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:13.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550117532 0 0) 0x7fa61c0befa0 con 0x7fa61c0a9300 2023-10-24T23:53:13.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa608003530 con 0x7fa61c0a9300 2023-10-24T23:53:13.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2115605922 0 0) 0x7fa608003530 con 0x7fa61c0a9300 2023-10-24T23:53:13.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 >> v2:172.21.15.110:3300/0 conn(0x7fa61c0c1b30 msgr2=0x7fa61c072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:13.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 --2- 172.21.15.107:0/364568291 >> v2:172.21.15.110:3300/0 conn(0x7fa61c0c1b30 0x7fa61c072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:13.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 >> v1:172.21.15.110:6789/0 conn(0x7fa61c0aa9f0 legacy=0x7fa61c155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:13.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c0bf1f0 con 0x7fa61c0a9300 2023-10-24T23:53:13.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fa610001680 con 0x7fa61c0a9300 2023-10-24T23:53:13.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 >> v1:172.21.15.107:6789/0 conn(0x7fa61c0a9300 legacy=0x7fa61c0aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:13.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 --2- 172.21.15.107:0/364568291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa608003980 0x7fa608003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:13.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 --2- 172.21.15.107:0/364568291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6080047d0 0x7fa608004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:13.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 --2- 172.21.15.107:0/364568291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6080052b0 0x7fa60800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:13.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0aa0b0 con 0x7fa6080047d0 2023-10-24T23:53:13.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60800c180 con 0x7fa6080052b0 2023-10-24T23:53:13.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60800c340 con 0x7fa608003980 2023-10-24T23:53:13.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa62163e640 1 --2- 172.21.15.107:0/364568291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6080047d0 0x7fa608004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:13.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa621e3f640 1 --2- 172.21.15.107:0/364568291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa608003980 0x7fa608003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:13.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa62163e640 1 -- 172.21.15.107:0/364568291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa608003980 msgr2=0x7fa608003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:13.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa62163e640 1 --2- 172.21.15.107:0/364568291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa608003980 0x7fa608003da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:13.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa62163e640 1 -- 172.21.15.107:0/364568291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6080052b0 msgr2=0x7fa60800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:13.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa62163e640 1 --2- 172.21.15.107:0/364568291 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6080052b0 0x7fa60800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:13.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa62163e640 1 -- 172.21.15.107:0/364568291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa608003530 con 0x7fa6080047d0 2023-10-24T23:53:13.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.955+0000 7fa622640640 1 -- 172.21.15.107:0/364568291 reap_dead start 2023-10-24T23:53:13.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa62163e640 1 --2- 172.21.15.107:0/364568291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6080047d0 0x7fa608004c50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0a9710 tx=0x7fa61001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162b02cd6ade504b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:13.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610005ae0 con 0x7fa6080047d0 2023-10-24T23:53:13.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa610005c60 con 0x7fa6080047d0 2023-10-24T23:53:13.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa620e3d640 1 -- 172.21.15.107:0/364568291 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa610007070 con 0x7fa6080047d0 2023-10-24T23:53:13.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 -- 172.21.15.107:0/364568291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6080047d0 msgr2=0x7fa608004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:13.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 --2- 172.21.15.107:0/364568291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6080047d0 0x7fa608004c50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0a9710 tx=0x7fa61001cbe0 comp rx=0 tx=0).stop 2023-10-24T23:53:13.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 -- 172.21.15.107:0/364568291 shutdown_connections 2023-10-24T23:53:13.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 --2- 172.21.15.107:0/364568291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6080047d0 0x7fa608004c50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:13.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 -- 172.21.15.107:0/364568291 >> 172.21.15.107:0/364568291 conn(0x7fa61c0c2860 msgr2=0x7fa61c0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:13.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 -- 172.21.15.107:0/364568291 shutdown_connections 2023-10-24T23:53:13.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 -- 172.21.15.107:0/364568291 wait complete. 2023-10-24T23:53:13.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 Processor -- start 2023-10-24T23:53:13.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 -- start start 2023-10-24T23:53:13.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6080047d0 0x7fa61c06c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:13.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa61c0a6600 0x7fa61c1dd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:13.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa61c1e30d0 0x7fa61c1ddf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:13.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c154e20 con 0x7fa61c0a6600 2023-10-24T23:53:13.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c06cea0 con 0x7fa61c1e30d0 2023-10-24T23:53:13.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c1dd560 con 0x7fa6080047d0 2023-10-24T23:53:13.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa62163e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa61c0a6600 0x7fa61c1dd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:13.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa62163e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa61c0a6600 0x7fa61c1dd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50280/0 (socket says 172.21.15.107:50280) 2023-10-24T23:53:13.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa62163e640 1 -- 172.21.15.107:0/2830752196 learned_addr learned my addr 172.21.15.107:0/2830752196 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:13.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa621e3f640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6080047d0 0x7fa61c06c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:13.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa622640640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa61c1e30d0 0x7fa61c1ddf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:13.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa62163e640 1 -- 172.21.15.107:0/2830752196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6080047d0 msgr2=0x7fa61c06c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:13.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa62163e640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6080047d0 0x7fa61c06c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:13.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa62163e640 1 -- 172.21.15.107:0/2830752196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa61c1e30d0 msgr2=0x7fa61c1ddf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:13.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa62163e640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa61c1e30d0 0x7fa61c1ddf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:13.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa62163e640 1 -- 172.21.15.107:0/2830752196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c154700 con 0x7fa61c0a6600 2023-10-24T23:53:13.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa62163e640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa61c0a6600 0x7fa61c1dd9f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa610001940 tx=0x7fa608005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:13.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa60e7fc640 1 -- 172.21.15.107:0/2830752196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6100b4070 con 0x7fa61c0a6600 2023-10-24T23:53:13.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa60e7fc640 1 -- 172.21.15.107:0/2830752196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa610000fa0 con 0x7fa61c0a6600 2023-10-24T23:53:13.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 -- 172.21.15.107:0/2830752196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61c06d0b0 con 0x7fa61c0a6600 2023-10-24T23:53:13.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa60e7fc640 1 -- 172.21.15.107:0/2830752196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6100047f0 con 0x7fa61c0a6600 2023-10-24T23:53:13.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.959+0000 7fa6240a3640 1 -- 172.21.15.107:0/2830752196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61c1549e0 con 0x7fa61c0a6600 2023-10-24T23:53:13.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.963+0000 7fa6240a3640 1 -- 172.21.15.107:0/2830752196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa61c1dd710 con 0x7fa61c0a6600 2023-10-24T23:53:13.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.963+0000 7fa60e7fc640 1 -- 172.21.15.107:0/2830752196 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa610004950 con 0x7fa61c0a6600 2023-10-24T23:53:13.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.963+0000 7fa60e7fc640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa600065e60 0x7fa600068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:13.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.963+0000 7fa60e7fc640 1 -- 172.21.15.107:0/2830752196 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fa61011c9c0 con 0x7fa61c0a6600 2023-10-24T23:53:13.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.963+0000 7fa621e3f640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa600065e60 0x7fa600068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:13.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.963+0000 7fa621e3f640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa600065e60 0x7fa600068320 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0c2cd0 tx=0x7fa618008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:13.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:13.967+0000 7fa60e7fc640 1 -- 172.21.15.107:0/2830752196 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa61c1dd710 con 0x7fa61c0a6600 2023-10-24T23:53:14.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.151+0000 7fa6240a3640 1 -- 172.21.15.107:0/2830752196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fa61c04e130 con 0x7fa61c0a6600 2023-10-24T23:53:14.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.803+0000 7fa60e7fc640 1 -- 172.21.15.107:0/2830752196 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"}]=0 v12) v1 ==== 110+0+0 (secure 0 0 0) 0x7fa6100aa020 con 0x7fa61c0a6600 2023-10-24T23:53:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 -- 172.21.15.107:0/2830752196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa600065e60 msgr2=0x7fa600068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa600065e60 0x7fa600068320 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0c2cd0 tx=0x7fa618008040 comp rx=0 tx=0).stop 2023-10-24T23:53:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 -- 172.21.15.107:0/2830752196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa61c0a6600 msgr2=0x7fa61c1dd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa61c0a6600 0x7fa61c1dd9f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa610001940 tx=0x7fa608005800 comp rx=0 tx=0).stop 2023-10-24T23:53:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 -- 172.21.15.107:0/2830752196 shutdown_connections 2023-10-24T23:53:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa61c1e30d0 0x7fa61c1ddf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa61c0a6600 0x7fa61c1dd9f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa600065e60 0x7fa600068320 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 --2- 172.21.15.107:0/2830752196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6080047d0 0x7fa61c06c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 -- 172.21.15.107:0/2830752196 >> 172.21.15.107:0/2830752196 conn(0x7fa61c0c2860 msgr2=0x7fa61c0a70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 -- 172.21.15.107:0/2830752196 shutdown_connections 2023-10-24T23:53:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.807+0000 7fa6240a3640 1 -- 172.21.15.107:0/2830752196 wait complete. 2023-10-24T23:53:14.832 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'b' 2023-10-24T23:53:14.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae2cc7b640 1 Processor -- start 2023-10-24T23:53:14.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae2cc7b640 1 -- start start 2023-10-24T23:53:14.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae2cc7b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae2806ee40 0x7fae280ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:14.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae2cc7b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2806f730 con 0x7fae280a9820 2023-10-24T23:53:14.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae2cc7b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae28156a40 con 0x7fae2814fd40 2023-10-24T23:53:14.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae2cc7b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae28156710 con 0x7fae2806ee40 2023-10-24T23:53:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae25d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fae280a9820 0x7fae2815af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55222/0 (socket says 172.21.15.107:55222) 2023-10-24T23:53:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae25d9b640 1 -- 172.21.15.107:0/3080408634 learned_addr learned my addr 172.21.15.107:0/3080408634 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae26d9d640 1 --2- 172.21.15.107:0/3080408634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae2806ee40 0x7fae280ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae2559a640 1 -- 172.21.15.107:0/3080408634 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164806317 0 0) 0x7fae2806f730 con 0x7fae280a9820 2023-10-24T23:53:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae2559a640 1 -- 172.21.15.107:0/3080408634 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae10003530 con 0x7fae280a9820 2023-10-24T23:53:14.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae26d9d640 1 -- 172.21.15.107:0/3080408634 >> v1:172.21.15.110:6789/0 conn(0x7fae2814fd40 legacy=0x7fae280c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae26d9d640 1 -- 172.21.15.107:0/3080408634 >> v1:172.21.15.107:6789/0 conn(0x7fae280a9820 legacy=0x7fae2815af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae2559a640 1 -- 172.21.15.107:0/3080408634 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3846452985 0 0) 0x7fae10003530 con 0x7fae280a9820 2023-10-24T23:53:14.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.955+0000 7fae26d9d640 1 -- 172.21.15.107:0/3080408634 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2806f980 con 0x7fae2806ee40 2023-10-24T23:53:14.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae26d9d640 1 --2- 172.21.15.107:0/3080408634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae2806ee40 0x7fae280ab090 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fae20009dc0 tx=0x7fae2001c770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ada2c14d79f94833 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:14.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae2559a640 1 -- 172.21.15.107:0/3080408634 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c001680 con 0x7fae2806ee40 2023-10-24T23:53:14.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae2559a640 1 -- 172.21.15.107:0/3080408634 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae200a64e0 con 0x7fae2806ee40 2023-10-24T23:53:14.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae2559a640 1 -- 172.21.15.107:0/3080408634 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae200a6850 con 0x7fae2806ee40 2023-10-24T23:53:14.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae2cc7b640 1 -- 172.21.15.107:0/3080408634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae2806ee40 msgr2=0x7fae280ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:14.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae2cc7b640 1 --2- 172.21.15.107:0/3080408634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae2806ee40 0x7fae280ab090 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fae20009dc0 tx=0x7fae2001c770 comp rx=0 tx=0).stop 2023-10-24T23:53:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae2cc7b640 1 -- 172.21.15.107:0/3080408634 shutdown_connections 2023-10-24T23:53:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae2cc7b640 1 --2- 172.21.15.107:0/3080408634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae2806ee40 0x7fae280ab090 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae2cc7b640 1 -- 172.21.15.107:0/3080408634 >> 172.21.15.107:0/3080408634 conn(0x7fae280c2950 msgr2=0x7fae280c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:14.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae2cc7b640 1 -- 172.21.15.107:0/3080408634 shutdown_connections 2023-10-24T23:53:14.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae2cc7b640 1 -- 172.21.15.107:0/3080408634 wait complete. 2023-10-24T23:53:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.959+0000 7fae2cc7b640 1 Processor -- start 2023-10-24T23:53:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2cc7b640 1 -- start start 2023-10-24T23:53:14.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2cc7b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae2806ee40 0x7fae280bae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2cc7b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae2814fd40 0x7fae280c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2cc7b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae281eb2e0 0x7fae28149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2cc7b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae280bb620 con 0x7fae2806ee40 2023-10-24T23:53:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2cc7b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae28149630 con 0x7fae2814fd40 2023-10-24T23:53:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2cc7b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae2814a750 con 0x7fae281eb2e0 2023-10-24T23:53:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae2806ee40 0x7fae280bae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:14.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae2806ee40 0x7fae280bae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50282/0 (socket says 172.21.15.107:50282) 2023-10-24T23:53:14.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2659c640 1 -- 172.21.15.107:0/2403283792 learned_addr learned my addr 172.21.15.107:0/2403283792 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:14.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2659c640 1 -- 172.21.15.107:0/2403283792 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae281eb2e0 msgr2=0x7fae28149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:14.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2659c640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae281eb2e0 0x7fae28149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:14.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2659c640 1 -- 172.21.15.107:0/2403283792 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae2814fd40 msgr2=0x7fae280c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:14.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2659c640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae2814fd40 0x7fae280c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:14.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2659c640 1 -- 172.21.15.107:0/2403283792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae20009a70 con 0x7fae2806ee40 2023-10-24T23:53:14.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2659c640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae2806ee40 0x7fae280bae70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fae1800c3f0 tx=0x7fae1800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:14.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae16ffd640 1 -- 172.21.15.107:0/2403283792 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae180052d0 con 0x7fae2806ee40 2023-10-24T23:53:14.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae16ffd640 1 -- 172.21.15.107:0/2403283792 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae18005430 con 0x7fae2806ee40 2023-10-24T23:53:14.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2cc7b640 1 -- 172.21.15.107:0/2403283792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae281eb830 con 0x7fae2806ee40 2023-10-24T23:53:14.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae16ffd640 1 -- 172.21.15.107:0/2403283792 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae18003840 con 0x7fae2806ee40 2023-10-24T23:53:14.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2cc7b640 1 -- 172.21.15.107:0/2403283792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae281ebdf0 con 0x7fae2806ee40 2023-10-24T23:53:14.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae2cc7b640 1 -- 172.21.15.107:0/2403283792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae280c24a0 con 0x7fae2806ee40 2023-10-24T23:53:14.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae16ffd640 1 -- 172.21.15.107:0/2403283792 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fae180039a0 con 0x7fae2806ee40 2023-10-24T23:53:14.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae16ffd640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae08065e90 0x7fae08068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:14.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae16ffd640 1 -- 172.21.15.107:0/2403283792 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fae18083070 con 0x7fae2806ee40 2023-10-24T23:53:14.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae25d9b640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae08065e90 0x7fae08068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:14.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.963+0000 7fae25d9b640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae08065e90 0x7fae08068350 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fae1c000c10 tx=0x7fae1c0011b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:14.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:14.967+0000 7fae16ffd640 1 -- 172.21.15.107:0/2403283792 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fae18050420 con 0x7fae2806ee40 2023-10-24T23:53:15.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.143+0000 7fae2cc7b640 1 -- 172.21.15.107:0/2403283792 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae280746e0 con 0x7fae2806ee40 2023-10-24T23:53:15.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.143+0000 7fae16ffd640 1 -- 172.21.15.107:0/2403283792 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7fae18053a40 con 0x7fae2806ee40 2023-10-24T23:53:15.147 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:15.147 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:53:15.148 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:53:15.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 -- 172.21.15.107:0/2403283792 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae08065e90 msgr2=0x7fae08068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae08065e90 0x7fae08068350 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fae1c000c10 tx=0x7fae1c0011b0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 -- 172.21.15.107:0/2403283792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae2806ee40 msgr2=0x7fae280bae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae2806ee40 0x7fae280bae70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fae1800c3f0 tx=0x7fae1800c8b0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 -- 172.21.15.107:0/2403283792 shutdown_connections 2023-10-24T23:53:15.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae08065e90 0x7fae08068350 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae281eb2e0 0x7fae28149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae2814fd40 0x7fae280c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 --2- 172.21.15.107:0/2403283792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae2806ee40 0x7fae280bae70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 -- 172.21.15.107:0/2403283792 >> 172.21.15.107:0/2403283792 conn(0x7fae280c2950 msgr2=0x7fae2814eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:15.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 -- 172.21.15.107:0/2403283792 shutdown_connections 2023-10-24T23:53:15.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.147+0000 7fae2cc7b640 1 -- 172.21.15.107:0/2403283792 wait complete. 2023-10-24T23:53:15.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.267+0000 7f3003bb0640 1 Processor -- start 2023-10-24T23:53:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.267+0000 7f3003bb0640 1 -- start start 2023-10-24T23:53:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f3003bb0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2ffc14fd40 0x7f2ffc0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f3003bb0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc06f730 con 0x7f2ffc0a9820 2023-10-24T23:53:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f3003bb0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc156a40 con 0x7f2ffc06ee40 2023-10-24T23:53:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f3003bb0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ffc156710 con 0x7f2ffc14fd40 2023-10-24T23:53:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300114b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2ffc0a9820 0x7f2ffc0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55236/0 (socket says 172.21.15.107:55236) 2023-10-24T23:53:15.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300114b640 1 -- 172.21.15.107:0/4273808801 learned_addr learned my addr 172.21.15.107:0/4273808801 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:15.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231227899 0 0) 0x7f2ffc06f730 con 0x7f2ffc0a9820 2023-10-24T23:53:15.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe8003530 con 0x7f2ffc0a9820 2023-10-24T23:53:15.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300194c640 1 --2- 172.21.15.107:0/4273808801 >> v2:172.21.15.110:3300/0 conn(0x7f2ffc14fd40 0x7f2ffc0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 44548801 0 0) 0x7f2fe8003530 con 0x7f2ffc0a9820 2023-10-24T23:53:15.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 >> v2:172.21.15.110:3300/0 conn(0x7f2ffc14fd40 msgr2=0x7f2ffc0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:15.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 --2- 172.21.15.107:0/4273808801 >> v2:172.21.15.110:3300/0 conn(0x7f2ffc14fd40 0x7f2ffc0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 >> v1:172.21.15.110:6789/0 conn(0x7f2ffc06ee40 legacy=0x7f2ffc15cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc06f980 con 0x7f2ffc0a9820 2023-10-24T23:53:15.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f2fec001680 con 0x7f2ffc0a9820 2023-10-24T23:53:15.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 >> v1:172.21.15.107:6789/0 conn(0x7f2ffc0a9820 legacy=0x7f2ffc0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fe8003980 0x7f2fe8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fe80047d0 0x7f2fe8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fe8005390 0x7f2fe800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc156710 con 0x7f2fe8005390 2023-10-24T23:53:15.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe800c220 con 0x7f2fe80047d0 2023-10-24T23:53:15.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe800c3e0 con 0x7f2fe8003980 2023-10-24T23:53:15.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300214d640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fe8005390 0x7f2fe800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300114b640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fe80047d0 0x7f2fe8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300194c640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fe8003980 0x7f2fe8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300214d640 1 -- 172.21.15.107:0/4273808801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fe8003980 msgr2=0x7f2fe8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:15.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300214d640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fe8003980 0x7f2fe8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300214d640 1 -- 172.21.15.107:0/4273808801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fe80047d0 msgr2=0x7f2fe8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:15.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300214d640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fe80047d0 0x7f2fe8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300214d640 1 -- 172.21.15.107:0/4273808801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe8003530 con 0x7f2fe8005390 2023-10-24T23:53:15.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300194c640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fe8003980 0x7f2fe8003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:15.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300114b640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fe80047d0 0x7f2fe8004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:15.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300214d640 1 -- 172.21.15.107:0/4273808801 reap_dead start 2023-10-24T23:53:15.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300214d640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fe8005390 0x7f2fe800b990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4001e40 tx=0x7f2ff401c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63333fa366627c1d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:15.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4003b80 con 0x7f2fe8005390 2023-10-24T23:53:15.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff4003ce0 con 0x7f2fe8005390 2023-10-24T23:53:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f300094a640 1 -- 172.21.15.107:0/4273808801 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4005310 con 0x7f2fe8005390 2023-10-24T23:53:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f3003bb0640 1 -- 172.21.15.107:0/4273808801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fe8005390 msgr2=0x7f2fe800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f3003bb0640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fe8005390 0x7f2fe800b990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4001e40 tx=0x7f2ff401c740 comp rx=0 tx=0).stop 2023-10-24T23:53:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f3003bb0640 1 -- 172.21.15.107:0/4273808801 shutdown_connections 2023-10-24T23:53:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f3003bb0640 1 --2- 172.21.15.107:0/4273808801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fe8005390 0x7f2fe800b990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.271+0000 7f3003bb0640 1 -- 172.21.15.107:0/4273808801 >> 172.21.15.107:0/4273808801 conn(0x7f2ffc0c2950 msgr2=0x7f2ffc0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 -- 172.21.15.107:0/4273808801 shutdown_connections 2023-10-24T23:53:15.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 -- 172.21.15.107:0/4273808801 wait complete. 2023-10-24T23:53:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 Processor -- start 2023-10-24T23:53:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 -- start start 2023-10-24T23:53:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fe8005390 0x7f2ffc147d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2ffc141b70 0x7f2ffc14eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ffc141060 0x7f2ffc1414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc155ba0 con 0x7f2ffc141060 2023-10-24T23:53:15.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc145aa0 con 0x7f2fe8005390 2023-10-24T23:53:15.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc145d80 con 0x7f2ffc141b70 2023-10-24T23:53:15.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f300214d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ffc141060 0x7f2ffc1414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f300214d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ffc141060 0x7f2ffc1414e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50306/0 (socket says 172.21.15.107:50306) 2023-10-24T23:53:15.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f300214d640 1 -- 172.21.15.107:0/1697234478 learned_addr learned my addr 172.21.15.107:0/1697234478 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:15.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f300194c640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fe8005390 0x7f2ffc147d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f300114b640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2ffc141b70 0x7f2ffc14eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f300214d640 1 -- 172.21.15.107:0/1697234478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2ffc141b70 msgr2=0x7f2ffc14eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f300214d640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2ffc141b70 0x7f2ffc14eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f300214d640 1 -- 172.21.15.107:0/1697234478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fe8005390 msgr2=0x7f2ffc147d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f300214d640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fe8005390 0x7f2ffc147d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f300214d640 1 -- 172.21.15.107:0/1697234478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc0c5730 con 0x7f2ffc141060 2023-10-24T23:53:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f300214d640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ffc141060 0x7f2ffc1414e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2ff401cfe0 tx=0x7f2fe8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f2ff27fc640 1 -- 172.21.15.107:0/1697234478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff40012e0 con 0x7f2ffc141060 2023-10-24T23:53:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f2ff27fc640 1 -- 172.21.15.107:0/1697234478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff4003980 con 0x7f2ffc141060 2023-10-24T23:53:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 -- 172.21.15.107:0/1697234478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc0c5a10 con 0x7f2ffc141060 2023-10-24T23:53:15.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f2ff27fc640 1 -- 172.21.15.107:0/1697234478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4006da0 con 0x7f2ffc141060 2023-10-24T23:53:15.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 -- 172.21.15.107:0/1697234478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc14f020 con 0x7f2ffc141060 2023-10-24T23:53:15.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.275+0000 7f3003bb0640 1 -- 172.21.15.107:0/1697234478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ffc0c24a0 con 0x7f2ffc141060 2023-10-24T23:53:15.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.279+0000 7f2ff27fc640 1 -- 172.21.15.107:0/1697234478 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2ff40a7070 con 0x7f2ffc141060 2023-10-24T23:53:15.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.279+0000 7f2ff27fc640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fe0065e90 0x7f2fe0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.279+0000 7f300194c640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fe0065e90 0x7f2fe0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.279+0000 7f2ff27fc640 1 -- 172.21.15.107:0/1697234478 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f2ff40b0070 con 0x7f2ffc141060 2023-10-24T23:53:15.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.279+0000 7f300194c640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fe0065e90 0x7f2fe0068350 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800c890 tx=0x7f2ff8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:15.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.283+0000 7f2ff27fc640 1 -- 172.21.15.107:0/1697234478 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2ff40ea1e0 con 0x7f2ffc141060 2023-10-24T23:53:15.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.427+0000 7f3003bb0640 1 -- 172.21.15.107:0/1697234478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ffc0c5590 con 0x7f2ffc141060 2023-10-24T23:53:15.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f2ff27fc640 1 -- 172.21.15.107:0/1697234478 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21531 (secure 0 0 0) 0x7f2ffc0c5590 con 0x7f2ffc141060 2023-10-24T23:53:15.434 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:15.434 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:12.781093+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:15.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 -- 172.21.15.107:0/1697234478 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fe0065e90 msgr2=0x7f2fe0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fe0065e90 0x7f2fe0068350 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800c890 tx=0x7f2ff8008040 comp rx=0 tx=0).stop 2023-10-24T23:53:15.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 -- 172.21.15.107:0/1697234478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ffc141060 msgr2=0x7f2ffc1414e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ffc141060 0x7f2ffc1414e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2ff401cfe0 tx=0x7f2fe8005800 comp rx=0 tx=0).stop 2023-10-24T23:53:15.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 -- 172.21.15.107:0/1697234478 shutdown_connections 2023-10-24T23:53:15.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ffc141060 0x7f2ffc1414e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fe0065e90 0x7f2fe0068350 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2ffc141b70 0x7f2ffc14eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 --2- 172.21.15.107:0/1697234478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fe8005390 0x7f2ffc147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 -- 172.21.15.107:0/1697234478 >> 172.21.15.107:0/1697234478 conn(0x7f2ffc0c2950 msgr2=0x7f2ffc1517c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:15.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 -- 172.21.15.107:0/1697234478 shutdown_connections 2023-10-24T23:53:15.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.431+0000 7f3003bb0640 1 -- 172.21.15.107:0/1697234478 wait complete. 2023-10-24T23:53:15.451 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-24T23:53:15.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:15.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.567+0000 7f6474e5d640 1 Processor -- start 2023-10-24T23:53:15.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.567+0000 7f6474e5d640 1 -- start start 2023-10-24T23:53:15.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f6474e5d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6470152040 0x7f64700c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f6474e5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64700c8100 0x7f64700abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f6474e5d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6470155680 con 0x7f647006a4b0 2023-10-24T23:53:15.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f6474e5d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6470154a70 con 0x7f6470152040 2023-10-24T23:53:15.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f6474e5d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6470154d50 con 0x7f64700c8100 2023-10-24T23:53:15.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f647006a4b0 0x7f64700ac240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55238/0 (socket says 172.21.15.107:55238) 2023-10-24T23:53:15.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646ed9d640 1 -- 172.21.15.107:0/93838257 learned_addr learned my addr 172.21.15.107:0/93838257 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:15.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646dd9b640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64700c8100 0x7f64700abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646e59c640 1 --2- 172.21.15.107:0/93838257 >> v2:172.21.15.110:3300/0 conn(0x7f6470152040 0x7f64700c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009090944 0 0) 0x7f6470155680 con 0x7f647006a4b0 2023-10-24T23:53:15.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6458003530 con 0x7f647006a4b0 2023-10-24T23:53:15.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1028981834 0 0) 0x7f6458003530 con 0x7f647006a4b0 2023-10-24T23:53:15.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64700c8100 msgr2=0x7f64700abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:15.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64700c8100 0x7f64700abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 >> v2:172.21.15.110:3300/0 conn(0x7f6470152040 msgr2=0x7f64700c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 --2- 172.21.15.107:0/93838257 >> v2:172.21.15.110:3300/0 conn(0x7f6470152040 0x7f64700c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6470155030 con 0x7f647006a4b0 2023-10-24T23:53:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6468001a00 con 0x7f647006a4b0 2023-10-24T23:53:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 >> v1:172.21.15.107:6789/0 conn(0x7f647006a4b0 legacy=0x7f64700ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6458003980 0x7f6458003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64700c8100 0x7f64580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6458004d10 0x7f6458005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6470154a70 con 0x7f6458003980 2023-10-24T23:53:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6470154d50 con 0x7f6458004d10 2023-10-24T23:53:15.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645800b990 con 0x7f64700c8100 2023-10-24T23:53:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646e59c640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6458003980 0x7f6458003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646dd9b640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64700c8100 0x7f64580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646e59c640 1 -- 172.21.15.107:0/93838257 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64700c8100 msgr2=0x7f64580047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646ed9d640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6458004d10 0x7f6458005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646e59c640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64700c8100 0x7f64580047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646e59c640 1 -- 172.21.15.107:0/93838257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6458004d10 msgr2=0x7f6458005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:15.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646e59c640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6458004d10 0x7f6458005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.571+0000 7f646e59c640 1 -- 172.21.15.107:0/93838257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6458003530 con 0x7f6458003980 2023-10-24T23:53:15.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646e59c640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6458003980 0x7f6458003da0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6460009ae0 tx=0x7f646001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac2ac937a992014 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:15.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64600a5570 con 0x7f6458003980 2023-10-24T23:53:15.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64600a56d0 con 0x7f6458003980 2023-10-24T23:53:15.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646d59a640 1 -- 172.21.15.107:0/93838257 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64600a5a80 con 0x7f6458003980 2023-10-24T23:53:15.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 -- 172.21.15.107:0/93838257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6458003980 msgr2=0x7f6458003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 --2- 172.21.15.107:0/93838257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6458003980 0x7f6458003da0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6460009ae0 tx=0x7f646001ca90 comp rx=0 tx=0).stop 2023-10-24T23:53:15.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646ed9d640 1 -- 172.21.15.107:0/93838257 reap_dead start 2023-10-24T23:53:15.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 -- 172.21.15.107:0/93838257 shutdown_connections 2023-10-24T23:53:15.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 -- 172.21.15.107:0/93838257 >> 172.21.15.107:0/93838257 conn(0x7f64700ca200 msgr2=0x7f64700ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:15.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 -- 172.21.15.107:0/93838257 shutdown_connections 2023-10-24T23:53:15.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 -- 172.21.15.107:0/93838257 wait complete. 2023-10-24T23:53:15.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 Processor -- start 2023-10-24T23:53:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 -- start start 2023-10-24T23:53:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6458003980 0x7f64700be370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6470150ba0 0x7f647006ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64701ecc70 0x7f64700c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64700c26a0 con 0x7f64701ecc70 2023-10-24T23:53:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64701edba0 con 0x7f6470150ba0 2023-10-24T23:53:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64701ed5e0 con 0x7f6458003980 2023-10-24T23:53:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64701ecc70 0x7f64700c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64701ecc70 0x7f64700c2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50332/0 (socket says 172.21.15.107:50332) 2023-10-24T23:53:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646ed9d640 1 -- 172.21.15.107:0/3033741375 learned_addr learned my addr 172.21.15.107:0/3033741375 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646dd9b640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6470150ba0 0x7f647006ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646e59c640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6458003980 0x7f64700be370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646ed9d640 1 -- 172.21.15.107:0/3033741375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6458003980 msgr2=0x7f64700be370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646ed9d640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6458003980 0x7f64700be370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646ed9d640 1 -- 172.21.15.107:0/3033741375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6470150ba0 msgr2=0x7f647006ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646ed9d640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6470150ba0 0x7f647006ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646ed9d640 1 -- 172.21.15.107:0/3033741375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64701ed8c0 con 0x7f64701ecc70 2023-10-24T23:53:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f646ed9d640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64701ecc70 0x7f64700c2b30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6468000f90 tx=0x7f646800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f645effd640 1 -- 172.21.15.107:0/3033741375 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646800bb80 con 0x7f64701ecc70 2023-10-24T23:53:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f645effd640 1 -- 172.21.15.107:0/3033741375 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646800bce0 con 0x7f64701ecc70 2023-10-24T23:53:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 -- 172.21.15.107:0/3033741375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64700c2330 con 0x7f64701ecc70 2023-10-24T23:53:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f645effd640 1 -- 172.21.15.107:0/3033741375 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468007290 con 0x7f64701ecc70 2023-10-24T23:53:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.575+0000 7f6474e5d640 1 -- 172.21.15.107:0/3033741375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64700c3070 con 0x7f64701ecc70 2023-10-24T23:53:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.579+0000 7f6474e5d640 1 -- 172.21.15.107:0/3033741375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64700746e0 con 0x7f64701ecc70 2023-10-24T23:53:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.579+0000 7f645effd640 1 -- 172.21.15.107:0/3033741375 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f64680073f0 con 0x7f64701ecc70 2023-10-24T23:53:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.579+0000 7f645effd640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6450065e60 0x7f6450068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.579+0000 7f645effd640 1 -- 172.21.15.107:0/3033741375 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f6468083dc0 con 0x7f64701ecc70 2023-10-24T23:53:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.579+0000 7f646e59c640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6450065e60 0x7f6450068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.579+0000 7f646e59c640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6450065e60 0x7f6450068320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f64600af000 tx=0x7f64600046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.583+0000 7f645effd640 1 -- 172.21.15.107:0/3033741375 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f64680510e0 con 0x7f64701ecc70 2023-10-24T23:53:15.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.735+0000 7f6474e5d640 1 -- 172.21.15.107:0/3033741375 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64700c32f0 con 0x7f6450065e60 2023-10-24T23:53:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.747+0000 7f645effd640 1 -- 172.21.15.107:0/3033741375 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+510297 (secure 0 0 0) 0x7f64700c32f0 con 0x7f6450065e60 2023-10-24T23:53:15.753 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:15.758 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:15.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 -- 172.21.15.107:0/3033741375 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6450065e60 msgr2=0x7f6450068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6450065e60 0x7f6450068320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f64600af000 tx=0x7f64600046d0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 -- 172.21.15.107:0/3033741375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64701ecc70 msgr2=0x7f64700c2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64701ecc70 0x7f64700c2b30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6468000f90 tx=0x7f646800b4a0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 -- 172.21.15.107:0/3033741375 shutdown_connections 2023-10-24T23:53:15.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64701ecc70 0x7f64700c2b30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6470150ba0 0x7f647006ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6450065e60 0x7f6450068320 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 --2- 172.21.15.107:0/3033741375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6458003980 0x7f64700be370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 -- 172.21.15.107:0/3033741375 >> 172.21.15.107:0/3033741375 conn(0x7f64700ca200 msgr2=0x7f64700c88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:15.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 -- 172.21.15.107:0/3033741375 shutdown_connections 2023-10-24T23:53:15.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.755+0000 7f6474e5d640 1 -- 172.21.15.107:0/3033741375 wait complete. 2023-10-24T23:53:15.774 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-10-24T23:53:15.279214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235112,"kb_used_data":3112,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754739608,"statfs":{"total":773094113280,"available":772853358592,"internally_reserved":0,"allocated":3186688,"data_stored":1901612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.892239"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497423+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-24T23:53:04.497423+0000","last_peered":"2023-10-24T23:53:04.497423+0000","last_clean":"2023-10-24T23:53:04.497423+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-24T23:53:04.497423+0000","last_undegraded":"2023-10-24T23:53:04.497423+0000","last_fullsized":"2023-10-24T23:53:04.497423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498958+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-24T23:53:04.498958+0000","last_peered":"2023-10-24T23:53:04.498958+0000","last_clean":"2023-10-24T23:53:04.498958+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-24T23:53:04.498958+0000","last_undegraded":"2023-10-24T23:53:04.498958+0000","last_fullsized":"2023-10-24T23:53:04.498958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165883+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-24T23:53:04.165883+0000","last_peered":"2023-10-24T23:53:04.165883+0000","last_clean":"2023-10-24T23:53:04.165883+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-24T23:53:04.165883+0000","last_undegraded":"2023-10-24T23:53:04.165883+0000","last_fullsized":"2023-10-24T23:53:04.165883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166134+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-24T23:53:04.166134+0000","last_peered":"2023-10-24T23:53:04.166134+0000","last_clean":"2023-10-24T23:53:04.166134+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-24T23:53:04.166134+0000","last_undegraded":"2023-10-24T23:53:04.166134+0000","last_fullsized":"2023-10-24T23:53:04.166134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699826+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-24T23:53:03.699826+0000","last_peered":"2023-10-24T23:53:03.699826+0000","last_clean":"2023-10-24T23:53:03.699826+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-24T23:53:03.699826+0000","last_undegraded":"2023-10-24T23:53:03.699826+0000","last_fullsized":"2023-10-24T23:53:03.699826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499034+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-24T23:53:04.499034+0000","last_peered":"2023-10-24T23:53:04.499034+0000","last_clean":"2023-10-24T23:53:04.499034+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-24T23:53:04.499034+0000","last_undegraded":"2023-10-24T23:53:04.499034+0000","last_fullsized":"2023-10-24T23:53:04.499034+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699150+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-24T23:53:03.699150+0000","last_peered":"2023-10-24T23:53:03.699150+0000","last_clean":"2023-10-24T23:53:03.699150+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-24T23:53:03.699150+0000","last_undegraded":"2023-10-24T23:53:03.699150+0000","last_fullsized":"2023-10-24T23:53:03.699150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499513+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-24T23:53:04.499513+0000","last_peered":"2023-10-24T23:53:04.499513+0000","last_clean":"2023-10-24T23:53:04.499513+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-24T23:53:04.499513+0000","last_undegraded":"2023-10-24T23:53:04.499513+0000","last_fullsized":"2023-10-24T23:53:04.499513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497288+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-24T23:53:04.497288+0000","last_peered":"2023-10-24T23:53:04.497288+0000","last_clean":"2023-10-24T23:53:04.497288+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-24T23:53:04.497288+0000","last_undegraded":"2023-10-24T23:53:04.497288+0000","last_fullsized":"2023-10-24T23:53:04.497288+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696609+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-24T23:53:03.696609+0000","last_peered":"2023-10-24T23:53:03.696609+0000","last_clean":"2023-10-24T23:53:03.696609+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-24T23:53:03.696609+0000","last_undegraded":"2023-10-24T23:53:03.696609+0000","last_fullsized":"2023-10-24T23:53:03.696609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499601+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-24T23:53:04.499601+0000","last_peered":"2023-10-24T23:53:04.499601+0000","last_clean":"2023-10-24T23:53:04.499601+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-24T23:53:04.499601+0000","last_undegraded":"2023-10-24T23:53:04.499601+0000","last_fullsized":"2023-10-24T23:53:04.499601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166207+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-24T23:53:04.166207+0000","last_peered":"2023-10-24T23:53:04.166207+0000","last_clean":"2023-10-24T23:53:04.166207+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-24T23:53:04.166207+0000","last_undegraded":"2023-10-24T23:53:04.166207+0000","last_fullsized":"2023-10-24T23:53:04.166207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496826+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-24T23:53:04.496826+0000","last_peered":"2023-10-24T23:53:04.496826+0000","last_clean":"2023-10-24T23:53:04.496826+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-24T23:53:04.496826+0000","last_undegraded":"2023-10-24T23:53:04.496826+0000","last_fullsized":"2023-10-24T23:53:04.496826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497403+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-24T23:53:04.497403+0000","last_peered":"2023-10-24T23:53:04.497403+0000","last_clean":"2023-10-24T23:53:04.497403+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-24T23:53:04.497403+0000","last_undegraded":"2023-10-24T23:53:04.497403+0000","last_fullsized":"2023-10-24T23:53:04.497403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165441+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-24T23:53:04.165441+0000","last_peered":"2023-10-24T23:53:04.165441+0000","last_clean":"2023-10-24T23:53:04.165441+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-24T23:53:04.165441+0000","last_undegraded":"2023-10-24T23:53:04.165441+0000","last_fullsized":"2023-10-24T23:53:04.165441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697959+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-24T23:53:03.697959+0000","last_peered":"2023-10-24T23:53:03.697959+0000","last_clean":"2023-10-24T23:53:03.697959+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-24T23:53:03.697959+0000","last_undegraded":"2023-10-24T23:53:03.697959+0000","last_fullsized":"2023-10-24T23:53:03.697959+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165168+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-24T23:53:04.165168+0000","last_peered":"2023-10-24T23:53:04.165168+0000","last_clean":"2023-10-24T23:53:04.165168+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-24T23:53:04.165168+0000","last_undegraded":"2023-10-24T23:53:04.165168+0000","last_fullsized":"2023-10-24T23:53:04.165168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497493+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-24T23:53:04.497493+0000","last_peered":"2023-10-24T23:53:04.497493+0000","last_clean":"2023-10-24T23:53:04.497493+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-24T23:53:04.497493+0000","last_undegraded":"2023-10-24T23:53:04.497493+0000","last_fullsized":"2023-10-24T23:53:04.497493+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165604+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-24T23:53:04.165604+0000","last_peered":"2023-10-24T23:53:04.165604+0000","last_clean":"2023-10-24T23:53:04.165604+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-24T23:53:04.165604+0000","last_undegraded":"2023-10-24T23:53:04.165604+0000","last_fullsized":"2023-10-24T23:53:04.165604+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699929+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-24T23:53:03.699929+0000","last_peered":"2023-10-24T23:53:03.699929+0000","last_clean":"2023-10-24T23:53:03.699929+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-24T23:53:03.699929+0000","last_undegraded":"2023-10-24T23:53:03.699929+0000","last_fullsized":"2023-10-24T23:53:03.699929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062786+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-24T23:53:04.062786+0000","last_peered":"2023-10-24T23:53:04.062786+0000","last_clean":"2023-10-24T23:53:04.062786+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-24T23:53:04.062786+0000","last_undegraded":"2023-10-24T23:53:04.062786+0000","last_fullsized":"2023-10-24T23:53:04.062786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496844+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-24T23:53:04.496844+0000","last_peered":"2023-10-24T23:53:04.496844+0000","last_clean":"2023-10-24T23:53:04.496844+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-24T23:53:04.496844+0000","last_undegraded":"2023-10-24T23:53:04.496844+0000","last_fullsized":"2023-10-24T23:53:04.496844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499662+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-24T23:53:04.499662+0000","last_peered":"2023-10-24T23:53:04.499662+0000","last_clean":"2023-10-24T23:53:04.499662+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-24T23:53:04.499662+0000","last_undegraded":"2023-10-24T23:53:04.499662+0000","last_fullsized":"2023-10-24T23:53:04.499662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499511+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-24T23:53:04.499511+0000","last_peered":"2023-10-24T23:53:04.499511+0000","last_clean":"2023-10-24T23:53:04.499511+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-24T23:53:04.499511+0000","last_undegraded":"2023-10-24T23:53:04.499511+0000","last_fullsized":"2023-10-24T23:53:04.499511+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499388+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-24T23:53:04.499388+0000","last_peered":"2023-10-24T23:53:04.499388+0000","last_clean":"2023-10-24T23:53:04.499388+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-24T23:53:04.499388+0000","last_undegraded":"2023-10-24T23:53:04.499388+0000","last_fullsized":"2023-10-24T23:53:04.499388+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.693689+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:02.693689+0000","last_peered":"2023-10-24T23:53:02.693689+0000","last_clean":"2023-10-24T23:53:02.693689+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:02.693689+0000","last_undegraded":"2023-10-24T23:53:02.693689+0000","last_fullsized":"2023-10-24T23:53:02.693689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:05.341603+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-24T23:53:05.341603+0000","last_peered":"2023-10-24T23:53:05.341603+0000","last_clean":"2023-10-24T23:53:05.341603+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-24T23:53:05.341603+0000","last_undegraded":"2023-10-24T23:53:05.341603+0000","last_fullsized":"2023-10-24T23:53:05.341603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165320+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-24T23:53:04.165320+0000","last_peered":"2023-10-24T23:53:04.165320+0000","last_clean":"2023-10-24T23:53:04.165320+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-24T23:53:04.165320+0000","last_undegraded":"2023-10-24T23:53:04.165320+0000","last_fullsized":"2023-10-24T23:53:04.165320+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.705879+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:02.705879+0000","last_peered":"2023-10-24T23:53:02.705879+0000","last_clean":"2023-10-24T23:53:02.705879+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:02.705879+0000","last_undegraded":"2023-10-24T23:53:02.705879+0000","last_fullsized":"2023-10-24T23:53:02.705879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166479+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-24T23:53:04.166479+0000","last_peered":"2023-10-24T23:53:04.166479+0000","last_clean":"2023-10-24T23:53:04.166479+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-24T23:53:04.166479+0000","last_undegraded":"2023-10-24T23:53:04.166479+0000","last_fullsized":"2023-10-24T23:53:04.166479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497155+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-24T23:53:04.497155+0000","last_peered":"2023-10-24T23:53:04.497155+0000","last_clean":"2023-10-24T23:53:04.497155+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-24T23:53:04.497155+0000","last_undegraded":"2023-10-24T23:53:04.497155+0000","last_fullsized":"2023-10-24T23:53:04.497155+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.714178+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:02.714178+0000","last_peered":"2023-10-24T23:53:02.714178+0000","last_clean":"2023-10-24T23:53:02.714178+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:02.714178+0000","last_undegraded":"2023-10-24T23:53:02.714178+0000","last_fullsized":"2023-10-24T23:53:02.714178+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497005+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-24T23:53:04.497005+0000","last_peered":"2023-10-24T23:53:04.497005+0000","last_clean":"2023-10-24T23:53:04.497005+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-24T23:53:04.497005+0000","last_undegraded":"2023-10-24T23:53:04.497005+0000","last_fullsized":"2023-10-24T23:53:04.497005+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496893+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-24T23:53:04.496893+0000","last_peered":"2023-10-24T23:53:04.496893+0000","last_clean":"2023-10-24T23:53:04.496893+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-24T23:53:04.496893+0000","last_undegraded":"2023-10-24T23:53:04.496893+0000","last_fullsized":"2023-10-24T23:53:04.496893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.710865+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:02.710865+0000","last_peered":"2023-10-24T23:53:02.710865+0000","last_clean":"2023-10-24T23:53:02.710865+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:02.710865+0000","last_undegraded":"2023-10-24T23:53:02.710865+0000","last_fullsized":"2023-10-24T23:53:02.710865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699976+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-24T23:53:03.699976+0000","last_peered":"2023-10-24T23:53:03.699976+0000","last_clean":"2023-10-24T23:53:03.699976+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-24T23:53:03.699976+0000","last_undegraded":"2023-10-24T23:53:03.699976+0000","last_fullsized":"2023-10-24T23:53:03.699976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496956+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-24T23:53:04.496956+0000","last_peered":"2023-10-24T23:53:04.496956+0000","last_clean":"2023-10-24T23:53:04.496956+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-24T23:53:04.496956+0000","last_undegraded":"2023-10-24T23:53:04.496956+0000","last_fullsized":"2023-10-24T23:53:04.496956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499793+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-24T23:53:04.499793+0000","last_peered":"2023-10-24T23:53:04.499793+0000","last_clean":"2023-10-24T23:53:04.499793+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-24T23:53:04.499793+0000","last_undegraded":"2023-10-24T23:53:04.499793+0000","last_fullsized":"2023-10-24T23:53:04.499793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.691164+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:53:02.691164+0000","last_peered":"2023-10-24T23:53:02.691164+0000","last_clean":"2023-10-24T23:53:02.691164+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:53:02.691164+0000","last_undegraded":"2023-10-24T23:53:02.691164+0000","last_fullsized":"2023-10-24T23:53:02.691164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497633+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-24T23:53:04.497633+0000","last_peered":"2023-10-24T23:53:04.497633+0000","last_clean":"2023-10-24T23:53:04.497633+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-24T23:53:04.497633+0000","last_undegraded":"2023-10-24T23:53:04.497633+0000","last_fullsized":"2023-10-24T23:53:04.497633+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496567+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-24T23:53:04.496567+0000","last_peered":"2023-10-24T23:53:04.496567+0000","last_clean":"2023-10-24T23:53:04.496567+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-24T23:53:04.496567+0000","last_undegraded":"2023-10-24T23:53:04.496567+0000","last_fullsized":"2023-10-24T23:53:04.496567+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.688268+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:53:02.688268+0000","last_peered":"2023-10-24T23:53:02.688268+0000","last_clean":"2023-10-24T23:53:02.688268+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:53:02.688268+0000","last_undegraded":"2023-10-24T23:53:02.688268+0000","last_fullsized":"2023-10-24T23:53:02.688268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497538+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-24T23:53:04.497538+0000","last_peered":"2023-10-24T23:53:04.497538+0000","last_clean":"2023-10-24T23:53:04.497538+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-24T23:53:04.497538+0000","last_undegraded":"2023-10-24T23:53:04.497538+0000","last_fullsized":"2023-10-24T23:53:04.497538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496751+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-24T23:53:04.496751+0000","last_peered":"2023-10-24T23:53:04.496751+0000","last_clean":"2023-10-24T23:53:04.496751+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-24T23:53:04.496751+0000","last_undegraded":"2023-10-24T23:53:04.496751+0000","last_fullsized":"2023-10-24T23:53:04.496751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.713207+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:02.713207+0000","last_peered":"2023-10-24T23:53:02.713207+0000","last_clean":"2023-10-24T23:53:02.713207+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:02.713207+0000","last_undegraded":"2023-10-24T23:53:02.713207+0000","last_fullsized":"2023-10-24T23:53:02.713207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698126+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-24T23:53:03.698126+0000","last_peered":"2023-10-24T23:53:03.698126+0000","last_clean":"2023-10-24T23:53:03.698126+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-24T23:53:03.698126+0000","last_undegraded":"2023-10-24T23:53:03.698126+0000","last_fullsized":"2023-10-24T23:53:03.698126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165624+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-24T23:53:04.165624+0000","last_peered":"2023-10-24T23:53:04.165624+0000","last_clean":"2023-10-24T23:53:04.165624+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-24T23:53:04.165624+0000","last_undegraded":"2023-10-24T23:53:04.165624+0000","last_fullsized":"2023-10-24T23:53:04.165624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.689415+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:53:02.689415+0000","last_peered":"2023-10-24T23:53:02.689415+0000","last_clean":"2023-10-24T23:53:02.689415+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:53:02.689415+0000","last_undegraded":"2023-10-24T23:53:02.689415+0000","last_fullsized":"2023-10-24T23:53:02.689415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.713291+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:53:02.713291+0000","last_peered":"2023-10-24T23:53:02.713291+0000","last_clean":"2023-10-24T23:53:02.713291+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:53:02.713291+0000","last_undegraded":"2023-10-24T23:53:02.713291+0000","last_fullsized":"2023-10-24T23:53:02.713291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497591+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-24T23:53:04.497591+0000","last_peered":"2023-10-24T23:53:04.497591+0000","last_clean":"2023-10-24T23:53:04.497591+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-24T23:53:04.497591+0000","last_undegraded":"2023-10-24T23:53:04.497591+0000","last_fullsized":"2023-10-24T23:53:04.497591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165664+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-24T23:53:04.165664+0000","last_peered":"2023-10-24T23:53:04.165664+0000","last_clean":"2023-10-24T23:53:04.165664+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-24T23:53:04.165664+0000","last_undegraded":"2023-10-24T23:53:04.165664+0000","last_fullsized":"2023-10-24T23:53:04.165664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499944+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-24T23:53:04.499944+0000","last_peered":"2023-10-24T23:53:04.499944+0000","last_clean":"2023-10-24T23:53:04.499944+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-24T23:53:04.499944+0000","last_undegraded":"2023-10-24T23:53:04.499944+0000","last_fullsized":"2023-10-24T23:53:04.499944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499622+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-24T23:53:04.499622+0000","last_peered":"2023-10-24T23:53:04.499622+0000","last_clean":"2023-10-24T23:53:04.499622+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-24T23:53:04.499622+0000","last_undegraded":"2023-10-24T23:53:04.499622+0000","last_fullsized":"2023-10-24T23:53:04.499622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499791+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-24T23:53:04.499791+0000","last_peered":"2023-10-24T23:53:04.499791+0000","last_clean":"2023-10-24T23:53:04.499791+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-24T23:53:04.499791+0000","last_undegraded":"2023-10-24T23:53:04.499791+0000","last_fullsized":"2023-10-24T23:53:04.499791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698204+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-24T23:53:03.698204+0000","last_peered":"2023-10-24T23:53:03.698204+0000","last_clean":"2023-10-24T23:53:03.698204+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-24T23:53:03.698204+0000","last_undegraded":"2023-10-24T23:53:03.698204+0000","last_fullsized":"2023-10-24T23:53:03.698204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499931+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-24T23:53:04.499931+0000","last_peered":"2023-10-24T23:53:04.499931+0000","last_clean":"2023-10-24T23:53:04.499931+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-24T23:53:04.499931+0000","last_undegraded":"2023-10-24T23:53:04.499931+0000","last_fullsized":"2023-10-24T23:53:04.499931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165994+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-24T23:53:04.165994+0000","last_peered":"2023-10-24T23:53:04.165994+0000","last_clean":"2023-10-24T23:53:04.165994+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-24T23:53:04.165994+0000","last_undegraded":"2023-10-24T23:53:04.165994+0000","last_fullsized":"2023-10-24T23:53:04.165994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496747+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-24T23:53:04.496747+0000","last_peered":"2023-10-24T23:53:04.496747+0000","last_clean":"2023-10-24T23:53:04.496747+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-24T23:53:04.496747+0000","last_undegraded":"2023-10-24T23:53:04.496747+0000","last_fullsized":"2023-10-24T23:53:04.496747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699348+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-24T23:53:03.699348+0000","last_peered":"2023-10-24T23:53:03.699348+0000","last_clean":"2023-10-24T23:53:03.699348+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-24T23:53:03.699348+0000","last_undegraded":"2023-10-24T23:53:03.699348+0000","last_fullsized":"2023-10-24T23:53:03.699348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697568+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-24T23:53:03.697568+0000","last_peered":"2023-10-24T23:53:03.697568+0000","last_clean":"2023-10-24T23:53:03.697568+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-24T23:53:03.697568+0000","last_undegraded":"2023-10-24T23:53:03.697568+0000","last_fullsized":"2023-10-24T23:53:03.697568+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062733+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-24T23:53:04.062733+0000","last_peered":"2023-10-24T23:53:04.062733+0000","last_clean":"2023-10-24T23:53:04.062733+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-24T23:53:04.062733+0000","last_undegraded":"2023-10-24T23:53:04.062733+0000","last_fullsized":"2023-10-24T23:53:04.062733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499840+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-24T23:53:04.499840+0000","last_peered":"2023-10-24T23:53:04.499840+0000","last_clean":"2023-10-24T23:53:04.499840+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-24T23:53:04.499840+0000","last_undegraded":"2023-10-24T23:53:04.499840+0000","last_fullsized":"2023-10-24T23:53:04.499840+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.063027+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-24T23:53:04.063027+0000","last_peered":"2023-10-24T23:53:04.063027+0000","last_clean":"2023-10-24T23:53:04.063027+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-24T23:53:04.063027+0000","last_undegraded":"2023-10-24T23:53:04.063027+0000","last_fullsized":"2023-10-24T23:53:04.063027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698927+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-24T23:53:03.698927+0000","last_peered":"2023-10-24T23:53:03.698927+0000","last_clean":"2023-10-24T23:53:03.698927+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-24T23:53:03.698927+0000","last_undegraded":"2023-10-24T23:53:03.698927+0000","last_fullsized":"2023-10-24T23:53:03.698927+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.063138+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-24T23:53:04.063138+0000","last_peered":"2023-10-24T23:53:04.063138+0000","last_clean":"2023-10-24T23:53:04.063138+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-24T23:53:04.063138+0000","last_undegraded":"2023-10-24T23:53:04.063138+0000","last_fullsized":"2023-10-24T23:53:04.063138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697647+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-24T23:53:03.697647+0000","last_peered":"2023-10-24T23:53:03.697647+0000","last_clean":"2023-10-24T23:53:03.697647+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-24T23:53:03.697647+0000","last_undegraded":"2023-10-24T23:53:03.697647+0000","last_fullsized":"2023-10-24T23:53:03.697647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166270+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-24T23:53:04.166270+0000","last_peered":"2023-10-24T23:53:04.166270+0000","last_clean":"2023-10-24T23:53:04.166270+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-24T23:53:04.166270+0000","last_undegraded":"2023-10-24T23:53:04.166270+0000","last_fullsized":"2023-10-24T23:53:04.166270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062154+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-24T23:53:04.062154+0000","last_peered":"2023-10-24T23:53:04.062154+0000","last_clean":"2023-10-24T23:53:04.062154+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-24T23:53:04.062154+0000","last_undegraded":"2023-10-24T23:53:04.062154+0000","last_fullsized":"2023-10-24T23:53:04.062154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497582+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-24T23:53:04.497582+0000","last_peered":"2023-10-24T23:53:04.497582+0000","last_clean":"2023-10-24T23:53:04.497582+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-24T23:53:04.497582+0000","last_undegraded":"2023-10-24T23:53:04.497582+0000","last_fullsized":"2023-10-24T23:53:04.497582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497015+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-24T23:53:04.497015+0000","last_peered":"2023-10-24T23:53:04.497015+0000","last_clean":"2023-10-24T23:53:04.497015+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-24T23:53:04.497015+0000","last_undegraded":"2023-10-24T23:53:04.497015+0000","last_fullsized":"2023-10-24T23:53:04.497015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699735+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-24T23:53:03.699735+0000","last_peered":"2023-10-24T23:53:03.699735+0000","last_clean":"2023-10-24T23:53:03.699735+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-24T23:53:03.699735+0000","last_undegraded":"2023-10-24T23:53:03.699735+0000","last_fullsized":"2023-10-24T23:53:03.699735+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062264+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-24T23:53:04.062264+0000","last_peered":"2023-10-24T23:53:04.062264+0000","last_clean":"2023-10-24T23:53:04.062264+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-24T23:53:04.062264+0000","last_undegraded":"2023-10-24T23:53:04.062264+0000","last_fullsized":"2023-10-24T23:53:04.062264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166359+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-24T23:53:04.166359+0000","last_peered":"2023-10-24T23:53:04.166359+0000","last_clean":"2023-10-24T23:53:04.166359+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-24T23:53:04.166359+0000","last_undegraded":"2023-10-24T23:53:04.166359+0000","last_fullsized":"2023-10-24T23:53:04.166359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698525+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-24T23:53:03.698525+0000","last_peered":"2023-10-24T23:53:03.698525+0000","last_clean":"2023-10-24T23:53:03.698525+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-24T23:53:03.698525+0000","last_undegraded":"2023-10-24T23:53:03.698525+0000","last_fullsized":"2023-10-24T23:53:03.698525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165369+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-24T23:53:04.165369+0000","last_peered":"2023-10-24T23:53:04.165369+0000","last_clean":"2023-10-24T23:53:04.165369+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-24T23:53:04.165369+0000","last_undegraded":"2023-10-24T23:53:04.165369+0000","last_fullsized":"2023-10-24T23:53:04.165369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499350+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-24T23:53:04.499350+0000","last_peered":"2023-10-24T23:53:04.499350+0000","last_clean":"2023-10-24T23:53:04.499350+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-24T23:53:04.499350+0000","last_undegraded":"2023-10-24T23:53:04.499350+0000","last_fullsized":"2023-10-24T23:53:04.499350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165731+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-24T23:53:04.165731+0000","last_peered":"2023-10-24T23:53:04.165731+0000","last_clean":"2023-10-24T23:53:04.165731+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-24T23:53:04.165731+0000","last_undegraded":"2023-10-24T23:53:04.165731+0000","last_fullsized":"2023-10-24T23:53:04.165731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497047+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-24T23:53:04.497047+0000","last_peered":"2023-10-24T23:53:04.497047+0000","last_clean":"2023-10-24T23:53:04.497047+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-24T23:53:04.497047+0000","last_undegraded":"2023-10-24T23:53:04.497047+0000","last_fullsized":"2023-10-24T23:53:04.497047+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699631+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-24T23:53:03.699631+0000","last_peered":"2023-10-24T23:53:03.699631+0000","last_clean":"2023-10-24T23:53:03.699631+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-24T23:53:03.699631+0000","last_undegraded":"2023-10-24T23:53:03.699631+0000","last_fullsized":"2023-10-24T23:53:03.699631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497056+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-24T23:53:04.497056+0000","last_peered":"2023-10-24T23:53:04.497056+0000","last_clean":"2023-10-24T23:53:04.497056+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-24T23:53:04.497056+0000","last_undegraded":"2023-10-24T23:53:04.497056+0000","last_fullsized":"2023-10-24T23:53:04.497056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497069+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-24T23:53:04.497069+0000","last_peered":"2023-10-24T23:53:04.497069+0000","last_clean":"2023-10-24T23:53:04.497069+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-24T23:53:04.497069+0000","last_undegraded":"2023-10-24T23:53:04.497069+0000","last_fullsized":"2023-10-24T23:53:04.497069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.061804+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-24T23:53:04.061804+0000","last_peered":"2023-10-24T23:53:04.061804+0000","last_clean":"2023-10-24T23:53:04.061804+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-24T23:53:04.061804+0000","last_undegraded":"2023-10-24T23:53:04.061804+0000","last_fullsized":"2023-10-24T23:53:04.061804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165275+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-24T23:53:04.165275+0000","last_peered":"2023-10-24T23:53:04.165275+0000","last_clean":"2023-10-24T23:53:04.165275+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-24T23:53:04.165275+0000","last_undegraded":"2023-10-24T23:53:04.165275+0000","last_fullsized":"2023-10-24T23:53:04.165275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496931+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-24T23:53:04.496931+0000","last_peered":"2023-10-24T23:53:04.496931+0000","last_clean":"2023-10-24T23:53:04.496931+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-24T23:53:04.496931+0000","last_undegraded":"2023-10-24T23:53:04.496931+0000","last_fullsized":"2023-10-24T23:53:04.496931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062078+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-24T23:53:04.062078+0000","last_peered":"2023-10-24T23:53:04.062078+0000","last_clean":"2023-10-24T23:53:04.062078+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-24T23:53:04.062078+0000","last_undegraded":"2023-10-24T23:53:04.062078+0000","last_fullsized":"2023-10-24T23:53:04.062078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499118+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-24T23:53:04.499118+0000","last_peered":"2023-10-24T23:53:04.499118+0000","last_clean":"2023-10-24T23:53:04.499118+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-24T23:53:04.499118+0000","last_undegraded":"2023-10-24T23:53:04.499118+0000","last_fullsized":"2023-10-24T23:53:04.499118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696750+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-24T23:53:03.696750+0000","last_peered":"2023-10-24T23:53:03.696750+0000","last_clean":"2023-10-24T23:53:03.696750+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-24T23:53:03.696750+0000","last_undegraded":"2023-10-24T23:53:03.696750+0000","last_fullsized":"2023-10-24T23:53:03.696750+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499911+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-24T23:53:04.499911+0000","last_peered":"2023-10-24T23:53:04.499911+0000","last_clean":"2023-10-24T23:53:04.499911+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-24T23:53:04.499911+0000","last_undegraded":"2023-10-24T23:53:04.499911+0000","last_fullsized":"2023-10-24T23:53:04.499911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497346+0000","last_change":"2023-10-24T23:53:02.728971+0000","last_active":"2023-10-24T23:53:04.497346+0000","last_peered":"2023-10-24T23:53:04.497346+0000","last_clean":"2023-10-24T23:53:04.497346+0000","last_became_active":"2023-10-24T23:53:02.728048+0000","last_became_peered":"2023-10-24T23:53:02.728048+0000","last_unstale":"2023-10-24T23:53:04.497346+0000","last_undegraded":"2023-10-24T23:53:04.497346+0000","last_fullsized":"2023-10-24T23:53:04.497346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:31:32.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498748+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-24T23:53:04.498748+0000","last_peered":"2023-10-24T23:53:04.498748+0000","last_clean":"2023-10-24T23:53:04.498748+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-24T23:53:04.498748+0000","last_undegraded":"2023-10-24T23:53:04.498748+0000","last_fullsized":"2023-10-24T23:53:04.498748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062078+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-24T23:53:04.062078+0000","last_peered":"2023-10-24T23:53:04.062078+0000","last_clean":"2023-10-24T23:53:04.062078+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-24T23:53:04.062078+0000","last_undegraded":"2023-10-24T23:53:04.062078+0000","last_fullsized":"2023-10-24T23:53:04.062078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699415+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-24T23:53:03.699415+0000","last_peered":"2023-10-24T23:53:03.699415+0000","last_clean":"2023-10-24T23:53:03.699415+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-24T23:53:03.699415+0000","last_undegraded":"2023-10-24T23:53:03.699415+0000","last_fullsized":"2023-10-24T23:53:03.699415+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696626+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-24T23:53:03.696626+0000","last_peered":"2023-10-24T23:53:03.696626+0000","last_clean":"2023-10-24T23:53:03.696626+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-24T23:53:03.696626+0000","last_undegraded":"2023-10-24T23:53:03.696626+0000","last_fullsized":"2023-10-24T23:53:03.696626+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696679+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-24T23:53:03.696679+0000","last_peered":"2023-10-24T23:53:03.696679+0000","last_clean":"2023-10-24T23:53:03.696679+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-24T23:53:03.696679+0000","last_undegraded":"2023-10-24T23:53:03.696679+0000","last_fullsized":"2023-10-24T23:53:03.696679+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499695+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-24T23:53:04.499695+0000","last_peered":"2023-10-24T23:53:04.499695+0000","last_clean":"2023-10-24T23:53:04.499695+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-24T23:53:04.499695+0000","last_undegraded":"2023-10-24T23:53:04.499695+0000","last_fullsized":"2023-10-24T23:53:04.499695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699835+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-24T23:53:03.699835+0000","last_peered":"2023-10-24T23:53:03.699835+0000","last_clean":"2023-10-24T23:53:03.699835+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-24T23:53:03.699835+0000","last_undegraded":"2023-10-24T23:53:03.699835+0000","last_fullsized":"2023-10-24T23:53:03.699835+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497251+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-24T23:53:04.497251+0000","last_peered":"2023-10-24T23:53:04.497251+0000","last_clean":"2023-10-24T23:53:04.497251+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-24T23:53:04.497251+0000","last_undegraded":"2023-10-24T23:53:04.497251+0000","last_fullsized":"2023-10-24T23:53:04.497251+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698958+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-24T23:53:03.698958+0000","last_peered":"2023-10-24T23:53:03.698958+0000","last_clean":"2023-10-24T23:53:03.698958+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-24T23:53:03.698958+0000","last_undegraded":"2023-10-24T23:53:03.698958+0000","last_fullsized":"2023-10-24T23:53:03.698958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.061899+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-24T23:53:04.061899+0000","last_peered":"2023-10-24T23:53:04.061899+0000","last_clean":"2023-10-24T23:53:04.061899+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-24T23:53:04.061899+0000","last_undegraded":"2023-10-24T23:53:04.061899+0000","last_fullsized":"2023-10-24T23:53:04.061899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498842+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-24T23:53:04.498842+0000","last_peered":"2023-10-24T23:53:04.498842+0000","last_clean":"2023-10-24T23:53:04.498842+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-24T23:53:04.498842+0000","last_undegraded":"2023-10-24T23:53:04.498842+0000","last_fullsized":"2023-10-24T23:53:04.498842+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497236+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-24T23:53:04.497236+0000","last_peered":"2023-10-24T23:53:04.497236+0000","last_clean":"2023-10-24T23:53:04.497236+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-24T23:53:04.497236+0000","last_undegraded":"2023-10-24T23:53:04.497236+0000","last_fullsized":"2023-10-24T23:53:04.497236+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697967+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-24T23:53:03.697967+0000","last_peered":"2023-10-24T23:53:03.697967+0000","last_clean":"2023-10-24T23:53:03.697967+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-24T23:53:03.697967+0000","last_undegraded":"2023-10-24T23:53:03.697967+0000","last_fullsized":"2023-10-24T23:53:03.697967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499435+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-24T23:53:04.499435+0000","last_peered":"2023-10-24T23:53:04.499435+0000","last_clean":"2023-10-24T23:53:04.499435+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-24T23:53:04.499435+0000","last_undegraded":"2023-10-24T23:53:04.499435+0000","last_fullsized":"2023-10-24T23:53:04.499435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699801+0000","last_change":"2023-10-24T23:53:02.715966+0000","last_active":"2023-10-24T23:53:03.699801+0000","last_peered":"2023-10-24T23:53:03.699801+0000","last_clean":"2023-10-24T23:53:03.699801+0000","last_became_active":"2023-10-24T23:53:02.715631+0000","last_became_peered":"2023-10-24T23:53:02.715631+0000","last_unstale":"2023-10-24T23:53:03.699801+0000","last_undegraded":"2023-10-24T23:53:03.699801+0000","last_fullsized":"2023-10-24T23:53:03.699801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:11.771864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696971+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-24T23:53:03.696971+0000","last_peered":"2023-10-24T23:53:03.696971+0000","last_clean":"2023-10-24T23:53:03.696971+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-24T23:53:03.696971+0000","last_undegraded":"2023-10-24T23:53:03.696971+0000","last_fullsized":"2023-10-24T23:53:03.696971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":864256,"data_stored":681667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31284,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340556,"statfs":{"total":96636764160,"available":96604729344,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31292,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340548,"statfs":{"total":96636764160,"available":96604721152,"internally_reserved":0,"allocated":249856,"data_stored":90171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":262144,"data_stored":91139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":245760,"data_stored":89609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":839680,"data_stored":680469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:15.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:15.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.927+0000 7f53ff686640 1 Processor -- start 2023-10-24T23:53:15.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.927+0000 7f53ff686640 1 -- start start 2023-10-24T23:53:15.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53ff686640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53f806a230 0x7f53f81551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53ff686640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f80c1860 con 0x7f53f806af70 2023-10-24T23:53:15.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53ff686640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f8154d50 con 0x7f53f814b600 2023-10-24T23:53:15.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53ff686640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f814b9e0 con 0x7f53f806a230 2023-10-24T23:53:15.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53fdc23640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53f806a230 0x7f53f81551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53fcc21640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f53f806af70 0x7f53f806c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55240/0 (socket says 172.21.15.107:55240) 2023-10-24T23:53:15.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53fdc23640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f53f806a230 0x7f53f81551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50334/0 (socket says 172.21.15.107:50334) 2023-10-24T23:53:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53fcc21640 1 -- 172.21.15.107:0/845616465 learned_addr learned my addr 172.21.15.107:0/845616465 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53fdc23640 1 -- 172.21.15.107:0/845616465 >> v1:172.21.15.110:6789/0 conn(0x7f53f814b600 legacy=0x7f53f8151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53fdc23640 1 -- 172.21.15.107:0/845616465 >> v1:172.21.15.107:6789/0 conn(0x7f53f806af70 legacy=0x7f53f806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53fdc23640 1 -- 172.21.15.107:0/845616465 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f80c1ab0 con 0x7f53f806a230 2023-10-24T23:53:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53effff640 1 -- 172.21.15.107:0/845616465 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623512565 0 0) 0x7f53f80c1860 con 0x7f53f806af70 2023-10-24T23:53:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53fdc23640 1 --2- 172.21.15.107:0/845616465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f806a230 0x7f53f81551e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009d90 tx=0x7f53f001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3daab50e2fada server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53effff640 1 -- 172.21.15.107:0/845616465 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f00a5820 con 0x7f53f806a230 2023-10-24T23:53:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53effff640 1 -- 172.21.15.107:0/845616465 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f00a5980 con 0x7f53f806a230 2023-10-24T23:53:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53effff640 1 -- 172.21.15.107:0/845616465 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f00a5d10 con 0x7f53f806a230 2023-10-24T23:53:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53ff686640 1 -- 172.21.15.107:0/845616465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f806a230 msgr2=0x7f53f81551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53ff686640 1 --2- 172.21.15.107:0/845616465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f806a230 0x7f53f81551e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009d90 tx=0x7f53f001c850 comp rx=0 tx=0).stop 2023-10-24T23:53:15.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53ff686640 1 -- 172.21.15.107:0/845616465 shutdown_connections 2023-10-24T23:53:15.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53ff686640 1 --2- 172.21.15.107:0/845616465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f806a230 0x7f53f81551e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53ff686640 1 -- 172.21.15.107:0/845616465 >> 172.21.15.107:0/845616465 conn(0x7f53f80c2950 msgr2=0x7f53f80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:15.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53ff686640 1 -- 172.21.15.107:0/845616465 shutdown_connections 2023-10-24T23:53:15.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.931+0000 7f53ff686640 1 -- 172.21.15.107:0/845616465 wait complete. 2023-10-24T23:53:15.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53ff686640 1 Processor -- start 2023-10-24T23:53:15.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53ff686640 1 -- start start 2023-10-24T23:53:15.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53ff686640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53f806a230 0x7f53f81482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53ff686640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53f806af70 0x7f53f80be4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53ff686640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f814b600 0x7f53f80bea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53ff686640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f80bd7c0 con 0x7f53f814b600 2023-10-24T23:53:15.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53ff686640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f81ecdc0 con 0x7f53f806a230 2023-10-24T23:53:15.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53ff686640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f81ed0a0 con 0x7f53f806af70 2023-10-24T23:53:15.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53fdc23640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f814b600 0x7f53f80bea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53fdc23640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f814b600 0x7f53f80bea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50340/0 (socket says 172.21.15.107:50340) 2023-10-24T23:53:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53fdc23640 1 -- 172.21.15.107:0/1576563779 learned_addr learned my addr 172.21.15.107:0/1576563779 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53fd422640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53f806a230 0x7f53f81482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53fcc21640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53f806af70 0x7f53f80be4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53fdc23640 1 -- 172.21.15.107:0/1576563779 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53f806af70 msgr2=0x7f53f80be4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53fdc23640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53f806af70 0x7f53f80be4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53fdc23640 1 -- 172.21.15.107:0/1576563779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53f806a230 msgr2=0x7f53f81482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53fdc23640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53f806a230 0x7f53f81482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53fdc23640 1 -- 172.21.15.107:0/1576563779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f0009a70 con 0x7f53f814b600 2023-10-24T23:53:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53fdc23640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f814b600 0x7f53f80bea30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f53f00b0040 tx=0x7f53f00195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53edffb640 1 -- 172.21.15.107:0/1576563779 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f00ad850 con 0x7f53f814b600 2023-10-24T23:53:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53edffb640 1 -- 172.21.15.107:0/1576563779 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f00ad9b0 con 0x7f53f814b600 2023-10-24T23:53:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53edffb640 1 -- 172.21.15.107:0/1576563779 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f00adda0 con 0x7f53f814b600 2023-10-24T23:53:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53ff686640 1 -- 172.21.15.107:0/1576563779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f81ed380 con 0x7f53f814b600 2023-10-24T23:53:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.935+0000 7f53ff686640 1 -- 172.21.15.107:0/1576563779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f8148800 con 0x7f53f814b600 2023-10-24T23:53:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.939+0000 7f53ff686640 1 -- 172.21.15.107:0/1576563779 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f80be210 con 0x7f53f814b600 2023-10-24T23:53:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.939+0000 7f53edffb640 1 -- 172.21.15.107:0/1576563779 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f53f00b4050 con 0x7f53f814b600 2023-10-24T23:53:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.939+0000 7f53edffb640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53dc065e60 0x7f53dc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.939+0000 7f53edffb640 1 -- 172.21.15.107:0/1576563779 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f53f011ea20 con 0x7f53f814b600 2023-10-24T23:53:15.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.939+0000 7f53fd422640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53dc065e60 0x7f53dc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.939+0000 7f53fd422640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53dc065e60 0x7f53dc068320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53f40098e0 tx=0x7f53f4009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:15.943+0000 7f53edffb640 1 -- 172.21.15.107:0/1576563779 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f53f80be210 con 0x7f53f814b600 2023-10-24T23:53:16.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.099+0000 7f53ff686640 1 -- 172.21.15.107:0/1576563779 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f53f815f6e0 con 0x7f53dc065e60 2023-10-24T23:53:16.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.114+0000 7f53edffb640 1 -- 172.21.15.107:0/1576563779 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+510297 (secure 0 0 0) 0x7f53f4009530 con 0x7f53dc065e60 2023-10-24T23:53:16.123 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:16.126 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 -- 172.21.15.107:0/1576563779 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53dc065e60 msgr2=0x7f53dc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53dc065e60 0x7f53dc068320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53f40098e0 tx=0x7f53f4009060 comp rx=0 tx=0).stop 2023-10-24T23:53:16.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 -- 172.21.15.107:0/1576563779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f814b600 msgr2=0x7f53f80bea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f814b600 0x7f53f80bea30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f53f00b0040 tx=0x7f53f00195f0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 -- 172.21.15.107:0/1576563779 shutdown_connections 2023-10-24T23:53:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53f814b600 0x7f53f80bea30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53dc065e60 0x7f53dc068320 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53f806af70 0x7f53f80be4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 --2- 172.21.15.107:0/1576563779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53f806a230 0x7f53f81482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 -- 172.21.15.107:0/1576563779 >> 172.21.15.107:0/1576563779 conn(0x7f53f80c2950 msgr2=0x7f53f80aacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 -- 172.21.15.107:0/1576563779 shutdown_connections 2023-10-24T23:53:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.118+0000 7f53ff686640 1 -- 172.21.15.107:0/1576563779 wait complete. 2023-10-24T23:53:16.140 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-10-24T23:53:15.279214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235112,"kb_used_data":3112,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754739608,"statfs":{"total":773094113280,"available":772853358592,"internally_reserved":0,"allocated":3186688,"data_stored":1901612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.892239"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497423+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-24T23:53:04.497423+0000","last_peered":"2023-10-24T23:53:04.497423+0000","last_clean":"2023-10-24T23:53:04.497423+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-24T23:53:04.497423+0000","last_undegraded":"2023-10-24T23:53:04.497423+0000","last_fullsized":"2023-10-24T23:53:04.497423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498958+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-24T23:53:04.498958+0000","last_peered":"2023-10-24T23:53:04.498958+0000","last_clean":"2023-10-24T23:53:04.498958+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-24T23:53:04.498958+0000","last_undegraded":"2023-10-24T23:53:04.498958+0000","last_fullsized":"2023-10-24T23:53:04.498958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165883+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-24T23:53:04.165883+0000","last_peered":"2023-10-24T23:53:04.165883+0000","last_clean":"2023-10-24T23:53:04.165883+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-24T23:53:04.165883+0000","last_undegraded":"2023-10-24T23:53:04.165883+0000","last_fullsized":"2023-10-24T23:53:04.165883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166134+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-24T23:53:04.166134+0000","last_peered":"2023-10-24T23:53:04.166134+0000","last_clean":"2023-10-24T23:53:04.166134+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-24T23:53:04.166134+0000","last_undegraded":"2023-10-24T23:53:04.166134+0000","last_fullsized":"2023-10-24T23:53:04.166134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699826+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-24T23:53:03.699826+0000","last_peered":"2023-10-24T23:53:03.699826+0000","last_clean":"2023-10-24T23:53:03.699826+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-24T23:53:03.699826+0000","last_undegraded":"2023-10-24T23:53:03.699826+0000","last_fullsized":"2023-10-24T23:53:03.699826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499034+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-24T23:53:04.499034+0000","last_peered":"2023-10-24T23:53:04.499034+0000","last_clean":"2023-10-24T23:53:04.499034+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-24T23:53:04.499034+0000","last_undegraded":"2023-10-24T23:53:04.499034+0000","last_fullsized":"2023-10-24T23:53:04.499034+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699150+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-24T23:53:03.699150+0000","last_peered":"2023-10-24T23:53:03.699150+0000","last_clean":"2023-10-24T23:53:03.699150+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-24T23:53:03.699150+0000","last_undegraded":"2023-10-24T23:53:03.699150+0000","last_fullsized":"2023-10-24T23:53:03.699150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499513+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-24T23:53:04.499513+0000","last_peered":"2023-10-24T23:53:04.499513+0000","last_clean":"2023-10-24T23:53:04.499513+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-24T23:53:04.499513+0000","last_undegraded":"2023-10-24T23:53:04.499513+0000","last_fullsized":"2023-10-24T23:53:04.499513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497288+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-24T23:53:04.497288+0000","last_peered":"2023-10-24T23:53:04.497288+0000","last_clean":"2023-10-24T23:53:04.497288+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-24T23:53:04.497288+0000","last_undegraded":"2023-10-24T23:53:04.497288+0000","last_fullsized":"2023-10-24T23:53:04.497288+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696609+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-24T23:53:03.696609+0000","last_peered":"2023-10-24T23:53:03.696609+0000","last_clean":"2023-10-24T23:53:03.696609+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-24T23:53:03.696609+0000","last_undegraded":"2023-10-24T23:53:03.696609+0000","last_fullsized":"2023-10-24T23:53:03.696609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499601+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-24T23:53:04.499601+0000","last_peered":"2023-10-24T23:53:04.499601+0000","last_clean":"2023-10-24T23:53:04.499601+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-24T23:53:04.499601+0000","last_undegraded":"2023-10-24T23:53:04.499601+0000","last_fullsized":"2023-10-24T23:53:04.499601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166207+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-24T23:53:04.166207+0000","last_peered":"2023-10-24T23:53:04.166207+0000","last_clean":"2023-10-24T23:53:04.166207+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-24T23:53:04.166207+0000","last_undegraded":"2023-10-24T23:53:04.166207+0000","last_fullsized":"2023-10-24T23:53:04.166207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496826+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-24T23:53:04.496826+0000","last_peered":"2023-10-24T23:53:04.496826+0000","last_clean":"2023-10-24T23:53:04.496826+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-24T23:53:04.496826+0000","last_undegraded":"2023-10-24T23:53:04.496826+0000","last_fullsized":"2023-10-24T23:53:04.496826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497403+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-24T23:53:04.497403+0000","last_peered":"2023-10-24T23:53:04.497403+0000","last_clean":"2023-10-24T23:53:04.497403+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-24T23:53:04.497403+0000","last_undegraded":"2023-10-24T23:53:04.497403+0000","last_fullsized":"2023-10-24T23:53:04.497403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165441+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-24T23:53:04.165441+0000","last_peered":"2023-10-24T23:53:04.165441+0000","last_clean":"2023-10-24T23:53:04.165441+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-24T23:53:04.165441+0000","last_undegraded":"2023-10-24T23:53:04.165441+0000","last_fullsized":"2023-10-24T23:53:04.165441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697959+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-24T23:53:03.697959+0000","last_peered":"2023-10-24T23:53:03.697959+0000","last_clean":"2023-10-24T23:53:03.697959+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-24T23:53:03.697959+0000","last_undegraded":"2023-10-24T23:53:03.697959+0000","last_fullsized":"2023-10-24T23:53:03.697959+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165168+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-24T23:53:04.165168+0000","last_peered":"2023-10-24T23:53:04.165168+0000","last_clean":"2023-10-24T23:53:04.165168+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-24T23:53:04.165168+0000","last_undegraded":"2023-10-24T23:53:04.165168+0000","last_fullsized":"2023-10-24T23:53:04.165168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497493+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-24T23:53:04.497493+0000","last_peered":"2023-10-24T23:53:04.497493+0000","last_clean":"2023-10-24T23:53:04.497493+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-24T23:53:04.497493+0000","last_undegraded":"2023-10-24T23:53:04.497493+0000","last_fullsized":"2023-10-24T23:53:04.497493+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165604+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-24T23:53:04.165604+0000","last_peered":"2023-10-24T23:53:04.165604+0000","last_clean":"2023-10-24T23:53:04.165604+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-24T23:53:04.165604+0000","last_undegraded":"2023-10-24T23:53:04.165604+0000","last_fullsized":"2023-10-24T23:53:04.165604+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699929+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-24T23:53:03.699929+0000","last_peered":"2023-10-24T23:53:03.699929+0000","last_clean":"2023-10-24T23:53:03.699929+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-24T23:53:03.699929+0000","last_undegraded":"2023-10-24T23:53:03.699929+0000","last_fullsized":"2023-10-24T23:53:03.699929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062786+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-24T23:53:04.062786+0000","last_peered":"2023-10-24T23:53:04.062786+0000","last_clean":"2023-10-24T23:53:04.062786+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-24T23:53:04.062786+0000","last_undegraded":"2023-10-24T23:53:04.062786+0000","last_fullsized":"2023-10-24T23:53:04.062786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496844+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-24T23:53:04.496844+0000","last_peered":"2023-10-24T23:53:04.496844+0000","last_clean":"2023-10-24T23:53:04.496844+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-24T23:53:04.496844+0000","last_undegraded":"2023-10-24T23:53:04.496844+0000","last_fullsized":"2023-10-24T23:53:04.496844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499662+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-24T23:53:04.499662+0000","last_peered":"2023-10-24T23:53:04.499662+0000","last_clean":"2023-10-24T23:53:04.499662+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-24T23:53:04.499662+0000","last_undegraded":"2023-10-24T23:53:04.499662+0000","last_fullsized":"2023-10-24T23:53:04.499662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499511+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-24T23:53:04.499511+0000","last_peered":"2023-10-24T23:53:04.499511+0000","last_clean":"2023-10-24T23:53:04.499511+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-24T23:53:04.499511+0000","last_undegraded":"2023-10-24T23:53:04.499511+0000","last_fullsized":"2023-10-24T23:53:04.499511+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499388+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-24T23:53:04.499388+0000","last_peered":"2023-10-24T23:53:04.499388+0000","last_clean":"2023-10-24T23:53:04.499388+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-24T23:53:04.499388+0000","last_undegraded":"2023-10-24T23:53:04.499388+0000","last_fullsized":"2023-10-24T23:53:04.499388+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.693689+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:02.693689+0000","last_peered":"2023-10-24T23:53:02.693689+0000","last_clean":"2023-10-24T23:53:02.693689+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:02.693689+0000","last_undegraded":"2023-10-24T23:53:02.693689+0000","last_fullsized":"2023-10-24T23:53:02.693689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:05.341603+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-24T23:53:05.341603+0000","last_peered":"2023-10-24T23:53:05.341603+0000","last_clean":"2023-10-24T23:53:05.341603+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-24T23:53:05.341603+0000","last_undegraded":"2023-10-24T23:53:05.341603+0000","last_fullsized":"2023-10-24T23:53:05.341603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165320+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-24T23:53:04.165320+0000","last_peered":"2023-10-24T23:53:04.165320+0000","last_clean":"2023-10-24T23:53:04.165320+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-24T23:53:04.165320+0000","last_undegraded":"2023-10-24T23:53:04.165320+0000","last_fullsized":"2023-10-24T23:53:04.165320+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.705879+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:02.705879+0000","last_peered":"2023-10-24T23:53:02.705879+0000","last_clean":"2023-10-24T23:53:02.705879+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:02.705879+0000","last_undegraded":"2023-10-24T23:53:02.705879+0000","last_fullsized":"2023-10-24T23:53:02.705879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166479+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-24T23:53:04.166479+0000","last_peered":"2023-10-24T23:53:04.166479+0000","last_clean":"2023-10-24T23:53:04.166479+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-24T23:53:04.166479+0000","last_undegraded":"2023-10-24T23:53:04.166479+0000","last_fullsized":"2023-10-24T23:53:04.166479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497155+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-24T23:53:04.497155+0000","last_peered":"2023-10-24T23:53:04.497155+0000","last_clean":"2023-10-24T23:53:04.497155+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-24T23:53:04.497155+0000","last_undegraded":"2023-10-24T23:53:04.497155+0000","last_fullsized":"2023-10-24T23:53:04.497155+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.714178+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:02.714178+0000","last_peered":"2023-10-24T23:53:02.714178+0000","last_clean":"2023-10-24T23:53:02.714178+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:02.714178+0000","last_undegraded":"2023-10-24T23:53:02.714178+0000","last_fullsized":"2023-10-24T23:53:02.714178+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497005+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-24T23:53:04.497005+0000","last_peered":"2023-10-24T23:53:04.497005+0000","last_clean":"2023-10-24T23:53:04.497005+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-24T23:53:04.497005+0000","last_undegraded":"2023-10-24T23:53:04.497005+0000","last_fullsized":"2023-10-24T23:53:04.497005+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496893+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-24T23:53:04.496893+0000","last_peered":"2023-10-24T23:53:04.496893+0000","last_clean":"2023-10-24T23:53:04.496893+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-24T23:53:04.496893+0000","last_undegraded":"2023-10-24T23:53:04.496893+0000","last_fullsized":"2023-10-24T23:53:04.496893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.710865+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:02.710865+0000","last_peered":"2023-10-24T23:53:02.710865+0000","last_clean":"2023-10-24T23:53:02.710865+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:02.710865+0000","last_undegraded":"2023-10-24T23:53:02.710865+0000","last_fullsized":"2023-10-24T23:53:02.710865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699976+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-24T23:53:03.699976+0000","last_peered":"2023-10-24T23:53:03.699976+0000","last_clean":"2023-10-24T23:53:03.699976+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-24T23:53:03.699976+0000","last_undegraded":"2023-10-24T23:53:03.699976+0000","last_fullsized":"2023-10-24T23:53:03.699976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496956+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-24T23:53:04.496956+0000","last_peered":"2023-10-24T23:53:04.496956+0000","last_clean":"2023-10-24T23:53:04.496956+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-24T23:53:04.496956+0000","last_undegraded":"2023-10-24T23:53:04.496956+0000","last_fullsized":"2023-10-24T23:53:04.496956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499793+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-24T23:53:04.499793+0000","last_peered":"2023-10-24T23:53:04.499793+0000","last_clean":"2023-10-24T23:53:04.499793+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-24T23:53:04.499793+0000","last_undegraded":"2023-10-24T23:53:04.499793+0000","last_fullsized":"2023-10-24T23:53:04.499793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.691164+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:53:02.691164+0000","last_peered":"2023-10-24T23:53:02.691164+0000","last_clean":"2023-10-24T23:53:02.691164+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:53:02.691164+0000","last_undegraded":"2023-10-24T23:53:02.691164+0000","last_fullsized":"2023-10-24T23:53:02.691164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497633+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-24T23:53:04.497633+0000","last_peered":"2023-10-24T23:53:04.497633+0000","last_clean":"2023-10-24T23:53:04.497633+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-24T23:53:04.497633+0000","last_undegraded":"2023-10-24T23:53:04.497633+0000","last_fullsized":"2023-10-24T23:53:04.497633+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496567+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-24T23:53:04.496567+0000","last_peered":"2023-10-24T23:53:04.496567+0000","last_clean":"2023-10-24T23:53:04.496567+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-24T23:53:04.496567+0000","last_undegraded":"2023-10-24T23:53:04.496567+0000","last_fullsized":"2023-10-24T23:53:04.496567+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.688268+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:53:02.688268+0000","last_peered":"2023-10-24T23:53:02.688268+0000","last_clean":"2023-10-24T23:53:02.688268+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:53:02.688268+0000","last_undegraded":"2023-10-24T23:53:02.688268+0000","last_fullsized":"2023-10-24T23:53:02.688268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497538+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-24T23:53:04.497538+0000","last_peered":"2023-10-24T23:53:04.497538+0000","last_clean":"2023-10-24T23:53:04.497538+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-24T23:53:04.497538+0000","last_undegraded":"2023-10-24T23:53:04.497538+0000","last_fullsized":"2023-10-24T23:53:04.497538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496751+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-24T23:53:04.496751+0000","last_peered":"2023-10-24T23:53:04.496751+0000","last_clean":"2023-10-24T23:53:04.496751+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-24T23:53:04.496751+0000","last_undegraded":"2023-10-24T23:53:04.496751+0000","last_fullsized":"2023-10-24T23:53:04.496751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.713207+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:02.713207+0000","last_peered":"2023-10-24T23:53:02.713207+0000","last_clean":"2023-10-24T23:53:02.713207+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:02.713207+0000","last_undegraded":"2023-10-24T23:53:02.713207+0000","last_fullsized":"2023-10-24T23:53:02.713207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698126+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-24T23:53:03.698126+0000","last_peered":"2023-10-24T23:53:03.698126+0000","last_clean":"2023-10-24T23:53:03.698126+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-24T23:53:03.698126+0000","last_undegraded":"2023-10-24T23:53:03.698126+0000","last_fullsized":"2023-10-24T23:53:03.698126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165624+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-24T23:53:04.165624+0000","last_peered":"2023-10-24T23:53:04.165624+0000","last_clean":"2023-10-24T23:53:04.165624+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-24T23:53:04.165624+0000","last_undegraded":"2023-10-24T23:53:04.165624+0000","last_fullsized":"2023-10-24T23:53:04.165624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.689415+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:53:02.689415+0000","last_peered":"2023-10-24T23:53:02.689415+0000","last_clean":"2023-10-24T23:53:02.689415+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:53:02.689415+0000","last_undegraded":"2023-10-24T23:53:02.689415+0000","last_fullsized":"2023-10-24T23:53:02.689415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.713291+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:53:02.713291+0000","last_peered":"2023-10-24T23:53:02.713291+0000","last_clean":"2023-10-24T23:53:02.713291+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:53:02.713291+0000","last_undegraded":"2023-10-24T23:53:02.713291+0000","last_fullsized":"2023-10-24T23:53:02.713291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497591+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-24T23:53:04.497591+0000","last_peered":"2023-10-24T23:53:04.497591+0000","last_clean":"2023-10-24T23:53:04.497591+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-24T23:53:04.497591+0000","last_undegraded":"2023-10-24T23:53:04.497591+0000","last_fullsized":"2023-10-24T23:53:04.497591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165664+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-24T23:53:04.165664+0000","last_peered":"2023-10-24T23:53:04.165664+0000","last_clean":"2023-10-24T23:53:04.165664+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-24T23:53:04.165664+0000","last_undegraded":"2023-10-24T23:53:04.165664+0000","last_fullsized":"2023-10-24T23:53:04.165664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499944+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-24T23:53:04.499944+0000","last_peered":"2023-10-24T23:53:04.499944+0000","last_clean":"2023-10-24T23:53:04.499944+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-24T23:53:04.499944+0000","last_undegraded":"2023-10-24T23:53:04.499944+0000","last_fullsized":"2023-10-24T23:53:04.499944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499622+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-24T23:53:04.499622+0000","last_peered":"2023-10-24T23:53:04.499622+0000","last_clean":"2023-10-24T23:53:04.499622+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-24T23:53:04.499622+0000","last_undegraded":"2023-10-24T23:53:04.499622+0000","last_fullsized":"2023-10-24T23:53:04.499622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499791+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-24T23:53:04.499791+0000","last_peered":"2023-10-24T23:53:04.499791+0000","last_clean":"2023-10-24T23:53:04.499791+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-24T23:53:04.499791+0000","last_undegraded":"2023-10-24T23:53:04.499791+0000","last_fullsized":"2023-10-24T23:53:04.499791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698204+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-24T23:53:03.698204+0000","last_peered":"2023-10-24T23:53:03.698204+0000","last_clean":"2023-10-24T23:53:03.698204+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-24T23:53:03.698204+0000","last_undegraded":"2023-10-24T23:53:03.698204+0000","last_fullsized":"2023-10-24T23:53:03.698204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499931+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-24T23:53:04.499931+0000","last_peered":"2023-10-24T23:53:04.499931+0000","last_clean":"2023-10-24T23:53:04.499931+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-24T23:53:04.499931+0000","last_undegraded":"2023-10-24T23:53:04.499931+0000","last_fullsized":"2023-10-24T23:53:04.499931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165994+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-24T23:53:04.165994+0000","last_peered":"2023-10-24T23:53:04.165994+0000","last_clean":"2023-10-24T23:53:04.165994+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-24T23:53:04.165994+0000","last_undegraded":"2023-10-24T23:53:04.165994+0000","last_fullsized":"2023-10-24T23:53:04.165994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496747+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-24T23:53:04.496747+0000","last_peered":"2023-10-24T23:53:04.496747+0000","last_clean":"2023-10-24T23:53:04.496747+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-24T23:53:04.496747+0000","last_undegraded":"2023-10-24T23:53:04.496747+0000","last_fullsized":"2023-10-24T23:53:04.496747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699348+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-24T23:53:03.699348+0000","last_peered":"2023-10-24T23:53:03.699348+0000","last_clean":"2023-10-24T23:53:03.699348+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-24T23:53:03.699348+0000","last_undegraded":"2023-10-24T23:53:03.699348+0000","last_fullsized":"2023-10-24T23:53:03.699348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697568+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-24T23:53:03.697568+0000","last_peered":"2023-10-24T23:53:03.697568+0000","last_clean":"2023-10-24T23:53:03.697568+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-24T23:53:03.697568+0000","last_undegraded":"2023-10-24T23:53:03.697568+0000","last_fullsized":"2023-10-24T23:53:03.697568+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062733+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-24T23:53:04.062733+0000","last_peered":"2023-10-24T23:53:04.062733+0000","last_clean":"2023-10-24T23:53:04.062733+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-24T23:53:04.062733+0000","last_undegraded":"2023-10-24T23:53:04.062733+0000","last_fullsized":"2023-10-24T23:53:04.062733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499840+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-24T23:53:04.499840+0000","last_peered":"2023-10-24T23:53:04.499840+0000","last_clean":"2023-10-24T23:53:04.499840+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-24T23:53:04.499840+0000","last_undegraded":"2023-10-24T23:53:04.499840+0000","last_fullsized":"2023-10-24T23:53:04.499840+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.063027+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-24T23:53:04.063027+0000","last_peered":"2023-10-24T23:53:04.063027+0000","last_clean":"2023-10-24T23:53:04.063027+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-24T23:53:04.063027+0000","last_undegraded":"2023-10-24T23:53:04.063027+0000","last_fullsized":"2023-10-24T23:53:04.063027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698927+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-24T23:53:03.698927+0000","last_peered":"2023-10-24T23:53:03.698927+0000","last_clean":"2023-10-24T23:53:03.698927+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-24T23:53:03.698927+0000","last_undegraded":"2023-10-24T23:53:03.698927+0000","last_fullsized":"2023-10-24T23:53:03.698927+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.063138+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-24T23:53:04.063138+0000","last_peered":"2023-10-24T23:53:04.063138+0000","last_clean":"2023-10-24T23:53:04.063138+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-24T23:53:04.063138+0000","last_undegraded":"2023-10-24T23:53:04.063138+0000","last_fullsized":"2023-10-24T23:53:04.063138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697647+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-24T23:53:03.697647+0000","last_peered":"2023-10-24T23:53:03.697647+0000","last_clean":"2023-10-24T23:53:03.697647+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-24T23:53:03.697647+0000","last_undegraded":"2023-10-24T23:53:03.697647+0000","last_fullsized":"2023-10-24T23:53:03.697647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166270+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-24T23:53:04.166270+0000","last_peered":"2023-10-24T23:53:04.166270+0000","last_clean":"2023-10-24T23:53:04.166270+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-24T23:53:04.166270+0000","last_undegraded":"2023-10-24T23:53:04.166270+0000","last_fullsized":"2023-10-24T23:53:04.166270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062154+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-24T23:53:04.062154+0000","last_peered":"2023-10-24T23:53:04.062154+0000","last_clean":"2023-10-24T23:53:04.062154+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-24T23:53:04.062154+0000","last_undegraded":"2023-10-24T23:53:04.062154+0000","last_fullsized":"2023-10-24T23:53:04.062154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497582+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-24T23:53:04.497582+0000","last_peered":"2023-10-24T23:53:04.497582+0000","last_clean":"2023-10-24T23:53:04.497582+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-24T23:53:04.497582+0000","last_undegraded":"2023-10-24T23:53:04.497582+0000","last_fullsized":"2023-10-24T23:53:04.497582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497015+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-24T23:53:04.497015+0000","last_peered":"2023-10-24T23:53:04.497015+0000","last_clean":"2023-10-24T23:53:04.497015+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-24T23:53:04.497015+0000","last_undegraded":"2023-10-24T23:53:04.497015+0000","last_fullsized":"2023-10-24T23:53:04.497015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699735+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-24T23:53:03.699735+0000","last_peered":"2023-10-24T23:53:03.699735+0000","last_clean":"2023-10-24T23:53:03.699735+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-24T23:53:03.699735+0000","last_undegraded":"2023-10-24T23:53:03.699735+0000","last_fullsized":"2023-10-24T23:53:03.699735+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062264+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-24T23:53:04.062264+0000","last_peered":"2023-10-24T23:53:04.062264+0000","last_clean":"2023-10-24T23:53:04.062264+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-24T23:53:04.062264+0000","last_undegraded":"2023-10-24T23:53:04.062264+0000","last_fullsized":"2023-10-24T23:53:04.062264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166359+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-24T23:53:04.166359+0000","last_peered":"2023-10-24T23:53:04.166359+0000","last_clean":"2023-10-24T23:53:04.166359+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-24T23:53:04.166359+0000","last_undegraded":"2023-10-24T23:53:04.166359+0000","last_fullsized":"2023-10-24T23:53:04.166359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698525+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-24T23:53:03.698525+0000","last_peered":"2023-10-24T23:53:03.698525+0000","last_clean":"2023-10-24T23:53:03.698525+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-24T23:53:03.698525+0000","last_undegraded":"2023-10-24T23:53:03.698525+0000","last_fullsized":"2023-10-24T23:53:03.698525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165369+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-24T23:53:04.165369+0000","last_peered":"2023-10-24T23:53:04.165369+0000","last_clean":"2023-10-24T23:53:04.165369+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-24T23:53:04.165369+0000","last_undegraded":"2023-10-24T23:53:04.165369+0000","last_fullsized":"2023-10-24T23:53:04.165369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499350+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-24T23:53:04.499350+0000","last_peered":"2023-10-24T23:53:04.499350+0000","last_clean":"2023-10-24T23:53:04.499350+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-24T23:53:04.499350+0000","last_undegraded":"2023-10-24T23:53:04.499350+0000","last_fullsized":"2023-10-24T23:53:04.499350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165731+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-24T23:53:04.165731+0000","last_peered":"2023-10-24T23:53:04.165731+0000","last_clean":"2023-10-24T23:53:04.165731+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-24T23:53:04.165731+0000","last_undegraded":"2023-10-24T23:53:04.165731+0000","last_fullsized":"2023-10-24T23:53:04.165731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497047+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-24T23:53:04.497047+0000","last_peered":"2023-10-24T23:53:04.497047+0000","last_clean":"2023-10-24T23:53:04.497047+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-24T23:53:04.497047+0000","last_undegraded":"2023-10-24T23:53:04.497047+0000","last_fullsized":"2023-10-24T23:53:04.497047+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699631+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-24T23:53:03.699631+0000","last_peered":"2023-10-24T23:53:03.699631+0000","last_clean":"2023-10-24T23:53:03.699631+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-24T23:53:03.699631+0000","last_undegraded":"2023-10-24T23:53:03.699631+0000","last_fullsized":"2023-10-24T23:53:03.699631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497056+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-24T23:53:04.497056+0000","last_peered":"2023-10-24T23:53:04.497056+0000","last_clean":"2023-10-24T23:53:04.497056+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-24T23:53:04.497056+0000","last_undegraded":"2023-10-24T23:53:04.497056+0000","last_fullsized":"2023-10-24T23:53:04.497056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497069+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-24T23:53:04.497069+0000","last_peered":"2023-10-24T23:53:04.497069+0000","last_clean":"2023-10-24T23:53:04.497069+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-24T23:53:04.497069+0000","last_undegraded":"2023-10-24T23:53:04.497069+0000","last_fullsized":"2023-10-24T23:53:04.497069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.061804+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-24T23:53:04.061804+0000","last_peered":"2023-10-24T23:53:04.061804+0000","last_clean":"2023-10-24T23:53:04.061804+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-24T23:53:04.061804+0000","last_undegraded":"2023-10-24T23:53:04.061804+0000","last_fullsized":"2023-10-24T23:53:04.061804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165275+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-24T23:53:04.165275+0000","last_peered":"2023-10-24T23:53:04.165275+0000","last_clean":"2023-10-24T23:53:04.165275+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-24T23:53:04.165275+0000","last_undegraded":"2023-10-24T23:53:04.165275+0000","last_fullsized":"2023-10-24T23:53:04.165275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496931+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-24T23:53:04.496931+0000","last_peered":"2023-10-24T23:53:04.496931+0000","last_clean":"2023-10-24T23:53:04.496931+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-24T23:53:04.496931+0000","last_undegraded":"2023-10-24T23:53:04.496931+0000","last_fullsized":"2023-10-24T23:53:04.496931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062078+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-24T23:53:04.062078+0000","last_peered":"2023-10-24T23:53:04.062078+0000","last_clean":"2023-10-24T23:53:04.062078+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-24T23:53:04.062078+0000","last_undegraded":"2023-10-24T23:53:04.062078+0000","last_fullsized":"2023-10-24T23:53:04.062078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499118+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-24T23:53:04.499118+0000","last_peered":"2023-10-24T23:53:04.499118+0000","last_clean":"2023-10-24T23:53:04.499118+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-24T23:53:04.499118+0000","last_undegraded":"2023-10-24T23:53:04.499118+0000","last_fullsized":"2023-10-24T23:53:04.499118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696750+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-24T23:53:03.696750+0000","last_peered":"2023-10-24T23:53:03.696750+0000","last_clean":"2023-10-24T23:53:03.696750+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-24T23:53:03.696750+0000","last_undegraded":"2023-10-24T23:53:03.696750+0000","last_fullsized":"2023-10-24T23:53:03.696750+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499911+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-24T23:53:04.499911+0000","last_peered":"2023-10-24T23:53:04.499911+0000","last_clean":"2023-10-24T23:53:04.499911+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-24T23:53:04.499911+0000","last_undegraded":"2023-10-24T23:53:04.499911+0000","last_fullsized":"2023-10-24T23:53:04.499911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497346+0000","last_change":"2023-10-24T23:53:02.728971+0000","last_active":"2023-10-24T23:53:04.497346+0000","last_peered":"2023-10-24T23:53:04.497346+0000","last_clean":"2023-10-24T23:53:04.497346+0000","last_became_active":"2023-10-24T23:53:02.728048+0000","last_became_peered":"2023-10-24T23:53:02.728048+0000","last_unstale":"2023-10-24T23:53:04.497346+0000","last_undegraded":"2023-10-24T23:53:04.497346+0000","last_fullsized":"2023-10-24T23:53:04.497346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:31:32.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498748+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-24T23:53:04.498748+0000","last_peered":"2023-10-24T23:53:04.498748+0000","last_clean":"2023-10-24T23:53:04.498748+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-24T23:53:04.498748+0000","last_undegraded":"2023-10-24T23:53:04.498748+0000","last_fullsized":"2023-10-24T23:53:04.498748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062078+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-24T23:53:04.062078+0000","last_peered":"2023-10-24T23:53:04.062078+0000","last_clean":"2023-10-24T23:53:04.062078+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-24T23:53:04.062078+0000","last_undegraded":"2023-10-24T23:53:04.062078+0000","last_fullsized":"2023-10-24T23:53:04.062078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699415+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-24T23:53:03.699415+0000","last_peered":"2023-10-24T23:53:03.699415+0000","last_clean":"2023-10-24T23:53:03.699415+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-24T23:53:03.699415+0000","last_undegraded":"2023-10-24T23:53:03.699415+0000","last_fullsized":"2023-10-24T23:53:03.699415+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696626+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-24T23:53:03.696626+0000","last_peered":"2023-10-24T23:53:03.696626+0000","last_clean":"2023-10-24T23:53:03.696626+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-24T23:53:03.696626+0000","last_undegraded":"2023-10-24T23:53:03.696626+0000","last_fullsized":"2023-10-24T23:53:03.696626+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696679+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-24T23:53:03.696679+0000","last_peered":"2023-10-24T23:53:03.696679+0000","last_clean":"2023-10-24T23:53:03.696679+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-24T23:53:03.696679+0000","last_undegraded":"2023-10-24T23:53:03.696679+0000","last_fullsized":"2023-10-24T23:53:03.696679+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499695+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-24T23:53:04.499695+0000","last_peered":"2023-10-24T23:53:04.499695+0000","last_clean":"2023-10-24T23:53:04.499695+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-24T23:53:04.499695+0000","last_undegraded":"2023-10-24T23:53:04.499695+0000","last_fullsized":"2023-10-24T23:53:04.499695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699835+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-24T23:53:03.699835+0000","last_peered":"2023-10-24T23:53:03.699835+0000","last_clean":"2023-10-24T23:53:03.699835+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-24T23:53:03.699835+0000","last_undegraded":"2023-10-24T23:53:03.699835+0000","last_fullsized":"2023-10-24T23:53:03.699835+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497251+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-24T23:53:04.497251+0000","last_peered":"2023-10-24T23:53:04.497251+0000","last_clean":"2023-10-24T23:53:04.497251+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-24T23:53:04.497251+0000","last_undegraded":"2023-10-24T23:53:04.497251+0000","last_fullsized":"2023-10-24T23:53:04.497251+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698958+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-24T23:53:03.698958+0000","last_peered":"2023-10-24T23:53:03.698958+0000","last_clean":"2023-10-24T23:53:03.698958+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-24T23:53:03.698958+0000","last_undegraded":"2023-10-24T23:53:03.698958+0000","last_fullsized":"2023-10-24T23:53:03.698958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.061899+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-24T23:53:04.061899+0000","last_peered":"2023-10-24T23:53:04.061899+0000","last_clean":"2023-10-24T23:53:04.061899+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-24T23:53:04.061899+0000","last_undegraded":"2023-10-24T23:53:04.061899+0000","last_fullsized":"2023-10-24T23:53:04.061899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498842+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-24T23:53:04.498842+0000","last_peered":"2023-10-24T23:53:04.498842+0000","last_clean":"2023-10-24T23:53:04.498842+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-24T23:53:04.498842+0000","last_undegraded":"2023-10-24T23:53:04.498842+0000","last_fullsized":"2023-10-24T23:53:04.498842+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497236+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-24T23:53:04.497236+0000","last_peered":"2023-10-24T23:53:04.497236+0000","last_clean":"2023-10-24T23:53:04.497236+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-24T23:53:04.497236+0000","last_undegraded":"2023-10-24T23:53:04.497236+0000","last_fullsized":"2023-10-24T23:53:04.497236+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697967+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-24T23:53:03.697967+0000","last_peered":"2023-10-24T23:53:03.697967+0000","last_clean":"2023-10-24T23:53:03.697967+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-24T23:53:03.697967+0000","last_undegraded":"2023-10-24T23:53:03.697967+0000","last_fullsized":"2023-10-24T23:53:03.697967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499435+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-24T23:53:04.499435+0000","last_peered":"2023-10-24T23:53:04.499435+0000","last_clean":"2023-10-24T23:53:04.499435+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-24T23:53:04.499435+0000","last_undegraded":"2023-10-24T23:53:04.499435+0000","last_fullsized":"2023-10-24T23:53:04.499435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699801+0000","last_change":"2023-10-24T23:53:02.715966+0000","last_active":"2023-10-24T23:53:03.699801+0000","last_peered":"2023-10-24T23:53:03.699801+0000","last_clean":"2023-10-24T23:53:03.699801+0000","last_became_active":"2023-10-24T23:53:02.715631+0000","last_became_peered":"2023-10-24T23:53:02.715631+0000","last_unstale":"2023-10-24T23:53:03.699801+0000","last_undegraded":"2023-10-24T23:53:03.699801+0000","last_fullsized":"2023-10-24T23:53:03.699801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:11.771864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696971+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-24T23:53:03.696971+0000","last_peered":"2023-10-24T23:53:03.696971+0000","last_clean":"2023-10-24T23:53:03.696971+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-24T23:53:03.696971+0000","last_undegraded":"2023-10-24T23:53:03.696971+0000","last_fullsized":"2023-10-24T23:53:03.696971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":864256,"data_stored":681667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31284,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340556,"statfs":{"total":96636764160,"available":96604729344,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31292,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340548,"statfs":{"total":96636764160,"available":96604721152,"internally_reserved":0,"allocated":249856,"data_stored":90171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":262144,"data_stored":91139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":245760,"data_stored":89609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":839680,"data_stored":680469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:16.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:16.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.302+0000 7f5b833a2640 1 Processor -- start 2023-10-24T23:53:16.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.302+0000 7f5b833a2640 1 -- start start 2023-10-24T23:53:16.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b833a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b7c151960 0x7f5b7c071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b833a2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5b7c0c24c0 0x7f5b7c06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b833a2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c150bc0 con 0x7f5b7c071220 2023-10-24T23:53:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b833a2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b7c06c7f0 con 0x7f5b7c0c24c0 2023-10-24T23:53:16.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b833a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c06c510 con 0x7f5b7c151960 2023-10-24T23:53:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b8093d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b7c151960 0x7f5b7c071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b8193f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5b7c0c24c0 0x7f5b7c06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b8093d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b7c151960 0x7f5b7c071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:51158/0 (socket says 172.21.15.107:51158) 2023-10-24T23:53:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b8093d640 1 -- 172.21.15.107:0/580137854 learned_addr learned my addr 172.21.15.107:0/580137854 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:16.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b8193f640 1 -- 172.21.15.107:0/580137854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b7c151960 msgr2=0x7f5b7c071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:16.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b8193f640 1 --2- 172.21.15.107:0/580137854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b7c151960 0x7f5b7c071a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b8193f640 1 -- 172.21.15.107:0/580137854 >> v1:172.21.15.110:6789/0 conn(0x7f5b7c071220 legacy=0x7f5b7c06bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b8193f640 1 -- 172.21.15.107:0/580137854 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c150e80 con 0x7f5b7c0c24c0 2023-10-24T23:53:16.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b8193f640 1 --2- 172.21.15.107:0/580137854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b7c0c24c0 0x7f5b7c06a9c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009d90 tx=0x7f5b7401c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84bfc1bc7f93c9c server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:16.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b73fff640 1 -- 172.21.15.107:0/580137854 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b740a5870 con 0x7f5b7c0c24c0 2023-10-24T23:53:16.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b73fff640 1 -- 172.21.15.107:0/580137854 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b740a59d0 con 0x7f5b7c0c24c0 2023-10-24T23:53:16.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.306+0000 7f5b73fff640 1 -- 172.21.15.107:0/580137854 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b740a5d60 con 0x7f5b7c0c24c0 2023-10-24T23:53:16.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 -- 172.21.15.107:0/580137854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b7c0c24c0 msgr2=0x7f5b7c06a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 --2- 172.21.15.107:0/580137854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b7c0c24c0 0x7f5b7c06a9c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009d90 tx=0x7f5b7401c8c0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 -- 172.21.15.107:0/580137854 shutdown_connections 2023-10-24T23:53:16.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 --2- 172.21.15.107:0/580137854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b7c0c24c0 0x7f5b7c06a9c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 --2- 172.21.15.107:0/580137854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b7c151960 0x7f5b7c071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 -- 172.21.15.107:0/580137854 >> 172.21.15.107:0/580137854 conn(0x7f5b7c0c2950 msgr2=0x7f5b7c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:16.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 -- 172.21.15.107:0/580137854 shutdown_connections 2023-10-24T23:53:16.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 -- 172.21.15.107:0/580137854 wait complete. 2023-10-24T23:53:16.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 Processor -- start 2023-10-24T23:53:16.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 -- start start 2023-10-24T23:53:16.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b7c071220 0x7f5b7c150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b7c0c24c0 0x7f5b7c14b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b7c151960 0x7f5b7c14b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c1eebd0 con 0x7f5b7c071220 2023-10-24T23:53:16.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c1edfc0 con 0x7f5b7c0c24c0 2023-10-24T23:53:16.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c1ee2a0 con 0x7f5b7c151960 2023-10-24T23:53:16.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b8113e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b7c071220 0x7f5b7c150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b8113e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b7c071220 0x7f5b7c150040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50344/0 (socket says 172.21.15.107:50344) 2023-10-24T23:53:16.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b8113e640 1 -- 172.21.15.107:0/571952123 learned_addr learned my addr 172.21.15.107:0/571952123 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:16.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b8193f640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b7c151960 0x7f5b7c14b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b8093d640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b7c0c24c0 0x7f5b7c14b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b8113e640 1 -- 172.21.15.107:0/571952123 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b7c151960 msgr2=0x7f5b7c14b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b8113e640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b7c151960 0x7f5b7c14b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b8113e640 1 -- 172.21.15.107:0/571952123 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b7c0c24c0 msgr2=0x7f5b7c14b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b8113e640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b7c0c24c0 0x7f5b7c14b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b8113e640 1 -- 172.21.15.107:0/571952123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b74009a70 con 0x7f5b7c071220 2023-10-24T23:53:16.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b8113e640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b7c071220 0x7f5b7c150040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800c500 tx=0x7f5b7800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:16.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b71ffb640 1 -- 172.21.15.107:0/571952123 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78005220 con 0x7f5b7c071220 2023-10-24T23:53:16.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b71ffb640 1 -- 172.21.15.107:0/571952123 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b78005380 con 0x7f5b7c071220 2023-10-24T23:53:16.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 -- 172.21.15.107:0/571952123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7c1ee580 con 0x7f5b7c071220 2023-10-24T23:53:16.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b71ffb640 1 -- 172.21.15.107:0/571952123 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78007270 con 0x7f5b7c071220 2023-10-24T23:53:16.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.310+0000 7f5b833a2640 1 -- 172.21.15.107:0/571952123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c150580 con 0x7f5b7c071220 2023-10-24T23:53:16.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.314+0000 7f5b833a2640 1 -- 172.21.15.107:0/571952123 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7c0746e0 con 0x7f5b7c071220 2023-10-24T23:53:16.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.314+0000 7f5b71ffb640 1 -- 172.21.15.107:0/571952123 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5b780073d0 con 0x7f5b7c071220 2023-10-24T23:53:16.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.314+0000 7f5b71ffb640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b60065e60 0x7f5b60068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.314+0000 7f5b8093d640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b60065e60 0x7f5b60068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.314+0000 7f5b71ffb640 1 -- 172.21.15.107:0/571952123 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f5b78083370 con 0x7f5b7c071220 2023-10-24T23:53:16.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.314+0000 7f5b8093d640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b60065e60 0x7f5b60068320 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c15c880 tx=0x7f5b6c009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:16.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.318+0000 7f5b71ffb640 1 -- 172.21.15.107:0/571952123 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5b78050750 con 0x7f5b7c071220 2023-10-24T23:53:16.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.466+0000 7f5b833a2640 1 -- 172.21.15.107:0/571952123 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b7c1ee410 con 0x7f5b60065e60 2023-10-24T23:53:16.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.486+0000 7f5b71ffb640 1 -- 172.21.15.107:0/571952123 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+510297 (secure 0 0 0) 0x7f5b7c1ee410 con 0x7f5b60065e60 2023-10-24T23:53:16.493 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:16.497 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:16.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 -- 172.21.15.107:0/571952123 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b60065e60 msgr2=0x7f5b60068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b60065e60 0x7f5b60068320 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c15c880 tx=0x7f5b6c009240 comp rx=0 tx=0).stop 2023-10-24T23:53:16.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 -- 172.21.15.107:0/571952123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b7c071220 msgr2=0x7f5b7c150040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b7c071220 0x7f5b7c150040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800c500 tx=0x7f5b7800c9c0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 -- 172.21.15.107:0/571952123 shutdown_connections 2023-10-24T23:53:16.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b60065e60 0x7f5b60068320 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b7c151960 0x7f5b7c14b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b7c0c24c0 0x7f5b7c14b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 --2- 172.21.15.107:0/571952123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b7c071220 0x7f5b7c150040 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 -- 172.21.15.107:0/571952123 >> 172.21.15.107:0/571952123 conn(0x7f5b7c0c2950 msgr2=0x7f5b7c069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:16.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 -- 172.21.15.107:0/571952123 shutdown_connections 2023-10-24T23:53:16.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.490+0000 7f5b833a2640 1 -- 172.21.15.107:0/571952123 wait complete. 2023-10-24T23:53:16.509 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-10-24T23:53:15.279214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235112,"kb_used_data":3112,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754739608,"statfs":{"total":773094113280,"available":772853358592,"internally_reserved":0,"allocated":3186688,"data_stored":1901612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.892239"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497423+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-24T23:53:04.497423+0000","last_peered":"2023-10-24T23:53:04.497423+0000","last_clean":"2023-10-24T23:53:04.497423+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-24T23:53:04.497423+0000","last_undegraded":"2023-10-24T23:53:04.497423+0000","last_fullsized":"2023-10-24T23:53:04.497423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498958+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-24T23:53:04.498958+0000","last_peered":"2023-10-24T23:53:04.498958+0000","last_clean":"2023-10-24T23:53:04.498958+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-24T23:53:04.498958+0000","last_undegraded":"2023-10-24T23:53:04.498958+0000","last_fullsized":"2023-10-24T23:53:04.498958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165883+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-24T23:53:04.165883+0000","last_peered":"2023-10-24T23:53:04.165883+0000","last_clean":"2023-10-24T23:53:04.165883+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-24T23:53:04.165883+0000","last_undegraded":"2023-10-24T23:53:04.165883+0000","last_fullsized":"2023-10-24T23:53:04.165883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166134+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-24T23:53:04.166134+0000","last_peered":"2023-10-24T23:53:04.166134+0000","last_clean":"2023-10-24T23:53:04.166134+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-24T23:53:04.166134+0000","last_undegraded":"2023-10-24T23:53:04.166134+0000","last_fullsized":"2023-10-24T23:53:04.166134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699826+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-24T23:53:03.699826+0000","last_peered":"2023-10-24T23:53:03.699826+0000","last_clean":"2023-10-24T23:53:03.699826+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-24T23:53:03.699826+0000","last_undegraded":"2023-10-24T23:53:03.699826+0000","last_fullsized":"2023-10-24T23:53:03.699826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499034+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-24T23:53:04.499034+0000","last_peered":"2023-10-24T23:53:04.499034+0000","last_clean":"2023-10-24T23:53:04.499034+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-24T23:53:04.499034+0000","last_undegraded":"2023-10-24T23:53:04.499034+0000","last_fullsized":"2023-10-24T23:53:04.499034+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699150+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-24T23:53:03.699150+0000","last_peered":"2023-10-24T23:53:03.699150+0000","last_clean":"2023-10-24T23:53:03.699150+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-24T23:53:03.699150+0000","last_undegraded":"2023-10-24T23:53:03.699150+0000","last_fullsized":"2023-10-24T23:53:03.699150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499513+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-24T23:53:04.499513+0000","last_peered":"2023-10-24T23:53:04.499513+0000","last_clean":"2023-10-24T23:53:04.499513+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-24T23:53:04.499513+0000","last_undegraded":"2023-10-24T23:53:04.499513+0000","last_fullsized":"2023-10-24T23:53:04.499513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497288+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-24T23:53:04.497288+0000","last_peered":"2023-10-24T23:53:04.497288+0000","last_clean":"2023-10-24T23:53:04.497288+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-24T23:53:04.497288+0000","last_undegraded":"2023-10-24T23:53:04.497288+0000","last_fullsized":"2023-10-24T23:53:04.497288+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696609+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-24T23:53:03.696609+0000","last_peered":"2023-10-24T23:53:03.696609+0000","last_clean":"2023-10-24T23:53:03.696609+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-24T23:53:03.696609+0000","last_undegraded":"2023-10-24T23:53:03.696609+0000","last_fullsized":"2023-10-24T23:53:03.696609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499601+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-24T23:53:04.499601+0000","last_peered":"2023-10-24T23:53:04.499601+0000","last_clean":"2023-10-24T23:53:04.499601+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-24T23:53:04.499601+0000","last_undegraded":"2023-10-24T23:53:04.499601+0000","last_fullsized":"2023-10-24T23:53:04.499601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166207+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-24T23:53:04.166207+0000","last_peered":"2023-10-24T23:53:04.166207+0000","last_clean":"2023-10-24T23:53:04.166207+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-24T23:53:04.166207+0000","last_undegraded":"2023-10-24T23:53:04.166207+0000","last_fullsized":"2023-10-24T23:53:04.166207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496826+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-24T23:53:04.496826+0000","last_peered":"2023-10-24T23:53:04.496826+0000","last_clean":"2023-10-24T23:53:04.496826+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-24T23:53:04.496826+0000","last_undegraded":"2023-10-24T23:53:04.496826+0000","last_fullsized":"2023-10-24T23:53:04.496826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497403+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-24T23:53:04.497403+0000","last_peered":"2023-10-24T23:53:04.497403+0000","last_clean":"2023-10-24T23:53:04.497403+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-24T23:53:04.497403+0000","last_undegraded":"2023-10-24T23:53:04.497403+0000","last_fullsized":"2023-10-24T23:53:04.497403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165441+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-24T23:53:04.165441+0000","last_peered":"2023-10-24T23:53:04.165441+0000","last_clean":"2023-10-24T23:53:04.165441+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-24T23:53:04.165441+0000","last_undegraded":"2023-10-24T23:53:04.165441+0000","last_fullsized":"2023-10-24T23:53:04.165441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697959+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-24T23:53:03.697959+0000","last_peered":"2023-10-24T23:53:03.697959+0000","last_clean":"2023-10-24T23:53:03.697959+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-24T23:53:03.697959+0000","last_undegraded":"2023-10-24T23:53:03.697959+0000","last_fullsized":"2023-10-24T23:53:03.697959+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165168+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-24T23:53:04.165168+0000","last_peered":"2023-10-24T23:53:04.165168+0000","last_clean":"2023-10-24T23:53:04.165168+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-24T23:53:04.165168+0000","last_undegraded":"2023-10-24T23:53:04.165168+0000","last_fullsized":"2023-10-24T23:53:04.165168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497493+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-24T23:53:04.497493+0000","last_peered":"2023-10-24T23:53:04.497493+0000","last_clean":"2023-10-24T23:53:04.497493+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-24T23:53:04.497493+0000","last_undegraded":"2023-10-24T23:53:04.497493+0000","last_fullsized":"2023-10-24T23:53:04.497493+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165604+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-24T23:53:04.165604+0000","last_peered":"2023-10-24T23:53:04.165604+0000","last_clean":"2023-10-24T23:53:04.165604+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-24T23:53:04.165604+0000","last_undegraded":"2023-10-24T23:53:04.165604+0000","last_fullsized":"2023-10-24T23:53:04.165604+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699929+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-24T23:53:03.699929+0000","last_peered":"2023-10-24T23:53:03.699929+0000","last_clean":"2023-10-24T23:53:03.699929+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-24T23:53:03.699929+0000","last_undegraded":"2023-10-24T23:53:03.699929+0000","last_fullsized":"2023-10-24T23:53:03.699929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062786+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-24T23:53:04.062786+0000","last_peered":"2023-10-24T23:53:04.062786+0000","last_clean":"2023-10-24T23:53:04.062786+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-24T23:53:04.062786+0000","last_undegraded":"2023-10-24T23:53:04.062786+0000","last_fullsized":"2023-10-24T23:53:04.062786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496844+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-24T23:53:04.496844+0000","last_peered":"2023-10-24T23:53:04.496844+0000","last_clean":"2023-10-24T23:53:04.496844+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-24T23:53:04.496844+0000","last_undegraded":"2023-10-24T23:53:04.496844+0000","last_fullsized":"2023-10-24T23:53:04.496844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499662+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-24T23:53:04.499662+0000","last_peered":"2023-10-24T23:53:04.499662+0000","last_clean":"2023-10-24T23:53:04.499662+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-24T23:53:04.499662+0000","last_undegraded":"2023-10-24T23:53:04.499662+0000","last_fullsized":"2023-10-24T23:53:04.499662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499511+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-24T23:53:04.499511+0000","last_peered":"2023-10-24T23:53:04.499511+0000","last_clean":"2023-10-24T23:53:04.499511+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-24T23:53:04.499511+0000","last_undegraded":"2023-10-24T23:53:04.499511+0000","last_fullsized":"2023-10-24T23:53:04.499511+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499388+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-24T23:53:04.499388+0000","last_peered":"2023-10-24T23:53:04.499388+0000","last_clean":"2023-10-24T23:53:04.499388+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-24T23:53:04.499388+0000","last_undegraded":"2023-10-24T23:53:04.499388+0000","last_fullsized":"2023-10-24T23:53:04.499388+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.693689+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:02.693689+0000","last_peered":"2023-10-24T23:53:02.693689+0000","last_clean":"2023-10-24T23:53:02.693689+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:02.693689+0000","last_undegraded":"2023-10-24T23:53:02.693689+0000","last_fullsized":"2023-10-24T23:53:02.693689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:05.341603+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-24T23:53:05.341603+0000","last_peered":"2023-10-24T23:53:05.341603+0000","last_clean":"2023-10-24T23:53:05.341603+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-24T23:53:05.341603+0000","last_undegraded":"2023-10-24T23:53:05.341603+0000","last_fullsized":"2023-10-24T23:53:05.341603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165320+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-24T23:53:04.165320+0000","last_peered":"2023-10-24T23:53:04.165320+0000","last_clean":"2023-10-24T23:53:04.165320+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-24T23:53:04.165320+0000","last_undegraded":"2023-10-24T23:53:04.165320+0000","last_fullsized":"2023-10-24T23:53:04.165320+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.705879+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:02.705879+0000","last_peered":"2023-10-24T23:53:02.705879+0000","last_clean":"2023-10-24T23:53:02.705879+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:02.705879+0000","last_undegraded":"2023-10-24T23:53:02.705879+0000","last_fullsized":"2023-10-24T23:53:02.705879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166479+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-24T23:53:04.166479+0000","last_peered":"2023-10-24T23:53:04.166479+0000","last_clean":"2023-10-24T23:53:04.166479+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-24T23:53:04.166479+0000","last_undegraded":"2023-10-24T23:53:04.166479+0000","last_fullsized":"2023-10-24T23:53:04.166479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497155+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-24T23:53:04.497155+0000","last_peered":"2023-10-24T23:53:04.497155+0000","last_clean":"2023-10-24T23:53:04.497155+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-24T23:53:04.497155+0000","last_undegraded":"2023-10-24T23:53:04.497155+0000","last_fullsized":"2023-10-24T23:53:04.497155+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.714178+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:02.714178+0000","last_peered":"2023-10-24T23:53:02.714178+0000","last_clean":"2023-10-24T23:53:02.714178+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:02.714178+0000","last_undegraded":"2023-10-24T23:53:02.714178+0000","last_fullsized":"2023-10-24T23:53:02.714178+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497005+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-24T23:53:04.497005+0000","last_peered":"2023-10-24T23:53:04.497005+0000","last_clean":"2023-10-24T23:53:04.497005+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-24T23:53:04.497005+0000","last_undegraded":"2023-10-24T23:53:04.497005+0000","last_fullsized":"2023-10-24T23:53:04.497005+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496893+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-24T23:53:04.496893+0000","last_peered":"2023-10-24T23:53:04.496893+0000","last_clean":"2023-10-24T23:53:04.496893+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-24T23:53:04.496893+0000","last_undegraded":"2023-10-24T23:53:04.496893+0000","last_fullsized":"2023-10-24T23:53:04.496893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.710865+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:02.710865+0000","last_peered":"2023-10-24T23:53:02.710865+0000","last_clean":"2023-10-24T23:53:02.710865+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:02.710865+0000","last_undegraded":"2023-10-24T23:53:02.710865+0000","last_fullsized":"2023-10-24T23:53:02.710865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699976+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-24T23:53:03.699976+0000","last_peered":"2023-10-24T23:53:03.699976+0000","last_clean":"2023-10-24T23:53:03.699976+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-24T23:53:03.699976+0000","last_undegraded":"2023-10-24T23:53:03.699976+0000","last_fullsized":"2023-10-24T23:53:03.699976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496956+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-24T23:53:04.496956+0000","last_peered":"2023-10-24T23:53:04.496956+0000","last_clean":"2023-10-24T23:53:04.496956+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-24T23:53:04.496956+0000","last_undegraded":"2023-10-24T23:53:04.496956+0000","last_fullsized":"2023-10-24T23:53:04.496956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499793+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-24T23:53:04.499793+0000","last_peered":"2023-10-24T23:53:04.499793+0000","last_clean":"2023-10-24T23:53:04.499793+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-24T23:53:04.499793+0000","last_undegraded":"2023-10-24T23:53:04.499793+0000","last_fullsized":"2023-10-24T23:53:04.499793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.691164+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:53:02.691164+0000","last_peered":"2023-10-24T23:53:02.691164+0000","last_clean":"2023-10-24T23:53:02.691164+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:53:02.691164+0000","last_undegraded":"2023-10-24T23:53:02.691164+0000","last_fullsized":"2023-10-24T23:53:02.691164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497633+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-24T23:53:04.497633+0000","last_peered":"2023-10-24T23:53:04.497633+0000","last_clean":"2023-10-24T23:53:04.497633+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-24T23:53:04.497633+0000","last_undegraded":"2023-10-24T23:53:04.497633+0000","last_fullsized":"2023-10-24T23:53:04.497633+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496567+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-24T23:53:04.496567+0000","last_peered":"2023-10-24T23:53:04.496567+0000","last_clean":"2023-10-24T23:53:04.496567+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-24T23:53:04.496567+0000","last_undegraded":"2023-10-24T23:53:04.496567+0000","last_fullsized":"2023-10-24T23:53:04.496567+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.688268+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:53:02.688268+0000","last_peered":"2023-10-24T23:53:02.688268+0000","last_clean":"2023-10-24T23:53:02.688268+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:53:02.688268+0000","last_undegraded":"2023-10-24T23:53:02.688268+0000","last_fullsized":"2023-10-24T23:53:02.688268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497538+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-24T23:53:04.497538+0000","last_peered":"2023-10-24T23:53:04.497538+0000","last_clean":"2023-10-24T23:53:04.497538+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-24T23:53:04.497538+0000","last_undegraded":"2023-10-24T23:53:04.497538+0000","last_fullsized":"2023-10-24T23:53:04.497538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496751+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-24T23:53:04.496751+0000","last_peered":"2023-10-24T23:53:04.496751+0000","last_clean":"2023-10-24T23:53:04.496751+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-24T23:53:04.496751+0000","last_undegraded":"2023-10-24T23:53:04.496751+0000","last_fullsized":"2023-10-24T23:53:04.496751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.713207+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:02.713207+0000","last_peered":"2023-10-24T23:53:02.713207+0000","last_clean":"2023-10-24T23:53:02.713207+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:02.713207+0000","last_undegraded":"2023-10-24T23:53:02.713207+0000","last_fullsized":"2023-10-24T23:53:02.713207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698126+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-24T23:53:03.698126+0000","last_peered":"2023-10-24T23:53:03.698126+0000","last_clean":"2023-10-24T23:53:03.698126+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-24T23:53:03.698126+0000","last_undegraded":"2023-10-24T23:53:03.698126+0000","last_fullsized":"2023-10-24T23:53:03.698126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165624+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-24T23:53:04.165624+0000","last_peered":"2023-10-24T23:53:04.165624+0000","last_clean":"2023-10-24T23:53:04.165624+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-24T23:53:04.165624+0000","last_undegraded":"2023-10-24T23:53:04.165624+0000","last_fullsized":"2023-10-24T23:53:04.165624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.689415+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:53:02.689415+0000","last_peered":"2023-10-24T23:53:02.689415+0000","last_clean":"2023-10-24T23:53:02.689415+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:53:02.689415+0000","last_undegraded":"2023-10-24T23:53:02.689415+0000","last_fullsized":"2023-10-24T23:53:02.689415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.713291+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:53:02.713291+0000","last_peered":"2023-10-24T23:53:02.713291+0000","last_clean":"2023-10-24T23:53:02.713291+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:53:02.713291+0000","last_undegraded":"2023-10-24T23:53:02.713291+0000","last_fullsized":"2023-10-24T23:53:02.713291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497591+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-24T23:53:04.497591+0000","last_peered":"2023-10-24T23:53:04.497591+0000","last_clean":"2023-10-24T23:53:04.497591+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-24T23:53:04.497591+0000","last_undegraded":"2023-10-24T23:53:04.497591+0000","last_fullsized":"2023-10-24T23:53:04.497591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165664+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-24T23:53:04.165664+0000","last_peered":"2023-10-24T23:53:04.165664+0000","last_clean":"2023-10-24T23:53:04.165664+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-24T23:53:04.165664+0000","last_undegraded":"2023-10-24T23:53:04.165664+0000","last_fullsized":"2023-10-24T23:53:04.165664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499944+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-24T23:53:04.499944+0000","last_peered":"2023-10-24T23:53:04.499944+0000","last_clean":"2023-10-24T23:53:04.499944+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-24T23:53:04.499944+0000","last_undegraded":"2023-10-24T23:53:04.499944+0000","last_fullsized":"2023-10-24T23:53:04.499944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499622+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-24T23:53:04.499622+0000","last_peered":"2023-10-24T23:53:04.499622+0000","last_clean":"2023-10-24T23:53:04.499622+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-24T23:53:04.499622+0000","last_undegraded":"2023-10-24T23:53:04.499622+0000","last_fullsized":"2023-10-24T23:53:04.499622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499791+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-24T23:53:04.499791+0000","last_peered":"2023-10-24T23:53:04.499791+0000","last_clean":"2023-10-24T23:53:04.499791+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-24T23:53:04.499791+0000","last_undegraded":"2023-10-24T23:53:04.499791+0000","last_fullsized":"2023-10-24T23:53:04.499791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698204+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-24T23:53:03.698204+0000","last_peered":"2023-10-24T23:53:03.698204+0000","last_clean":"2023-10-24T23:53:03.698204+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-24T23:53:03.698204+0000","last_undegraded":"2023-10-24T23:53:03.698204+0000","last_fullsized":"2023-10-24T23:53:03.698204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499931+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-24T23:53:04.499931+0000","last_peered":"2023-10-24T23:53:04.499931+0000","last_clean":"2023-10-24T23:53:04.499931+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-24T23:53:04.499931+0000","last_undegraded":"2023-10-24T23:53:04.499931+0000","last_fullsized":"2023-10-24T23:53:04.499931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165994+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-24T23:53:04.165994+0000","last_peered":"2023-10-24T23:53:04.165994+0000","last_clean":"2023-10-24T23:53:04.165994+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-24T23:53:04.165994+0000","last_undegraded":"2023-10-24T23:53:04.165994+0000","last_fullsized":"2023-10-24T23:53:04.165994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496747+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-24T23:53:04.496747+0000","last_peered":"2023-10-24T23:53:04.496747+0000","last_clean":"2023-10-24T23:53:04.496747+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-24T23:53:04.496747+0000","last_undegraded":"2023-10-24T23:53:04.496747+0000","last_fullsized":"2023-10-24T23:53:04.496747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699348+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-24T23:53:03.699348+0000","last_peered":"2023-10-24T23:53:03.699348+0000","last_clean":"2023-10-24T23:53:03.699348+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-24T23:53:03.699348+0000","last_undegraded":"2023-10-24T23:53:03.699348+0000","last_fullsized":"2023-10-24T23:53:03.699348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697568+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-24T23:53:03.697568+0000","last_peered":"2023-10-24T23:53:03.697568+0000","last_clean":"2023-10-24T23:53:03.697568+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-24T23:53:03.697568+0000","last_undegraded":"2023-10-24T23:53:03.697568+0000","last_fullsized":"2023-10-24T23:53:03.697568+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062733+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-24T23:53:04.062733+0000","last_peered":"2023-10-24T23:53:04.062733+0000","last_clean":"2023-10-24T23:53:04.062733+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-24T23:53:04.062733+0000","last_undegraded":"2023-10-24T23:53:04.062733+0000","last_fullsized":"2023-10-24T23:53:04.062733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499840+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-24T23:53:04.499840+0000","last_peered":"2023-10-24T23:53:04.499840+0000","last_clean":"2023-10-24T23:53:04.499840+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-24T23:53:04.499840+0000","last_undegraded":"2023-10-24T23:53:04.499840+0000","last_fullsized":"2023-10-24T23:53:04.499840+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.063027+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-24T23:53:04.063027+0000","last_peered":"2023-10-24T23:53:04.063027+0000","last_clean":"2023-10-24T23:53:04.063027+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-24T23:53:04.063027+0000","last_undegraded":"2023-10-24T23:53:04.063027+0000","last_fullsized":"2023-10-24T23:53:04.063027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698927+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-24T23:53:03.698927+0000","last_peered":"2023-10-24T23:53:03.698927+0000","last_clean":"2023-10-24T23:53:03.698927+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-24T23:53:03.698927+0000","last_undegraded":"2023-10-24T23:53:03.698927+0000","last_fullsized":"2023-10-24T23:53:03.698927+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.063138+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-24T23:53:04.063138+0000","last_peered":"2023-10-24T23:53:04.063138+0000","last_clean":"2023-10-24T23:53:04.063138+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-24T23:53:04.063138+0000","last_undegraded":"2023-10-24T23:53:04.063138+0000","last_fullsized":"2023-10-24T23:53:04.063138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697647+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-24T23:53:03.697647+0000","last_peered":"2023-10-24T23:53:03.697647+0000","last_clean":"2023-10-24T23:53:03.697647+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-24T23:53:03.697647+0000","last_undegraded":"2023-10-24T23:53:03.697647+0000","last_fullsized":"2023-10-24T23:53:03.697647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166270+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-24T23:53:04.166270+0000","last_peered":"2023-10-24T23:53:04.166270+0000","last_clean":"2023-10-24T23:53:04.166270+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-24T23:53:04.166270+0000","last_undegraded":"2023-10-24T23:53:04.166270+0000","last_fullsized":"2023-10-24T23:53:04.166270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062154+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-24T23:53:04.062154+0000","last_peered":"2023-10-24T23:53:04.062154+0000","last_clean":"2023-10-24T23:53:04.062154+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-24T23:53:04.062154+0000","last_undegraded":"2023-10-24T23:53:04.062154+0000","last_fullsized":"2023-10-24T23:53:04.062154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497582+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-24T23:53:04.497582+0000","last_peered":"2023-10-24T23:53:04.497582+0000","last_clean":"2023-10-24T23:53:04.497582+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-24T23:53:04.497582+0000","last_undegraded":"2023-10-24T23:53:04.497582+0000","last_fullsized":"2023-10-24T23:53:04.497582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497015+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-24T23:53:04.497015+0000","last_peered":"2023-10-24T23:53:04.497015+0000","last_clean":"2023-10-24T23:53:04.497015+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-24T23:53:04.497015+0000","last_undegraded":"2023-10-24T23:53:04.497015+0000","last_fullsized":"2023-10-24T23:53:04.497015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699735+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-24T23:53:03.699735+0000","last_peered":"2023-10-24T23:53:03.699735+0000","last_clean":"2023-10-24T23:53:03.699735+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-24T23:53:03.699735+0000","last_undegraded":"2023-10-24T23:53:03.699735+0000","last_fullsized":"2023-10-24T23:53:03.699735+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062264+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-24T23:53:04.062264+0000","last_peered":"2023-10-24T23:53:04.062264+0000","last_clean":"2023-10-24T23:53:04.062264+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-24T23:53:04.062264+0000","last_undegraded":"2023-10-24T23:53:04.062264+0000","last_fullsized":"2023-10-24T23:53:04.062264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166359+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-24T23:53:04.166359+0000","last_peered":"2023-10-24T23:53:04.166359+0000","last_clean":"2023-10-24T23:53:04.166359+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-24T23:53:04.166359+0000","last_undegraded":"2023-10-24T23:53:04.166359+0000","last_fullsized":"2023-10-24T23:53:04.166359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698525+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-24T23:53:03.698525+0000","last_peered":"2023-10-24T23:53:03.698525+0000","last_clean":"2023-10-24T23:53:03.698525+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-24T23:53:03.698525+0000","last_undegraded":"2023-10-24T23:53:03.698525+0000","last_fullsized":"2023-10-24T23:53:03.698525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165369+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-24T23:53:04.165369+0000","last_peered":"2023-10-24T23:53:04.165369+0000","last_clean":"2023-10-24T23:53:04.165369+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-24T23:53:04.165369+0000","last_undegraded":"2023-10-24T23:53:04.165369+0000","last_fullsized":"2023-10-24T23:53:04.165369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499350+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-24T23:53:04.499350+0000","last_peered":"2023-10-24T23:53:04.499350+0000","last_clean":"2023-10-24T23:53:04.499350+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-24T23:53:04.499350+0000","last_undegraded":"2023-10-24T23:53:04.499350+0000","last_fullsized":"2023-10-24T23:53:04.499350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165731+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-24T23:53:04.165731+0000","last_peered":"2023-10-24T23:53:04.165731+0000","last_clean":"2023-10-24T23:53:04.165731+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-24T23:53:04.165731+0000","last_undegraded":"2023-10-24T23:53:04.165731+0000","last_fullsized":"2023-10-24T23:53:04.165731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497047+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-24T23:53:04.497047+0000","last_peered":"2023-10-24T23:53:04.497047+0000","last_clean":"2023-10-24T23:53:04.497047+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-24T23:53:04.497047+0000","last_undegraded":"2023-10-24T23:53:04.497047+0000","last_fullsized":"2023-10-24T23:53:04.497047+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699631+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-24T23:53:03.699631+0000","last_peered":"2023-10-24T23:53:03.699631+0000","last_clean":"2023-10-24T23:53:03.699631+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-24T23:53:03.699631+0000","last_undegraded":"2023-10-24T23:53:03.699631+0000","last_fullsized":"2023-10-24T23:53:03.699631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497056+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-24T23:53:04.497056+0000","last_peered":"2023-10-24T23:53:04.497056+0000","last_clean":"2023-10-24T23:53:04.497056+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-24T23:53:04.497056+0000","last_undegraded":"2023-10-24T23:53:04.497056+0000","last_fullsized":"2023-10-24T23:53:04.497056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497069+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-24T23:53:04.497069+0000","last_peered":"2023-10-24T23:53:04.497069+0000","last_clean":"2023-10-24T23:53:04.497069+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-24T23:53:04.497069+0000","last_undegraded":"2023-10-24T23:53:04.497069+0000","last_fullsized":"2023-10-24T23:53:04.497069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.061804+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-24T23:53:04.061804+0000","last_peered":"2023-10-24T23:53:04.061804+0000","last_clean":"2023-10-24T23:53:04.061804+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-24T23:53:04.061804+0000","last_undegraded":"2023-10-24T23:53:04.061804+0000","last_fullsized":"2023-10-24T23:53:04.061804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165275+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-24T23:53:04.165275+0000","last_peered":"2023-10-24T23:53:04.165275+0000","last_clean":"2023-10-24T23:53:04.165275+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-24T23:53:04.165275+0000","last_undegraded":"2023-10-24T23:53:04.165275+0000","last_fullsized":"2023-10-24T23:53:04.165275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496931+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-24T23:53:04.496931+0000","last_peered":"2023-10-24T23:53:04.496931+0000","last_clean":"2023-10-24T23:53:04.496931+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-24T23:53:04.496931+0000","last_undegraded":"2023-10-24T23:53:04.496931+0000","last_fullsized":"2023-10-24T23:53:04.496931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062078+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-24T23:53:04.062078+0000","last_peered":"2023-10-24T23:53:04.062078+0000","last_clean":"2023-10-24T23:53:04.062078+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-24T23:53:04.062078+0000","last_undegraded":"2023-10-24T23:53:04.062078+0000","last_fullsized":"2023-10-24T23:53:04.062078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499118+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-24T23:53:04.499118+0000","last_peered":"2023-10-24T23:53:04.499118+0000","last_clean":"2023-10-24T23:53:04.499118+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-24T23:53:04.499118+0000","last_undegraded":"2023-10-24T23:53:04.499118+0000","last_fullsized":"2023-10-24T23:53:04.499118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696750+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-24T23:53:03.696750+0000","last_peered":"2023-10-24T23:53:03.696750+0000","last_clean":"2023-10-24T23:53:03.696750+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-24T23:53:03.696750+0000","last_undegraded":"2023-10-24T23:53:03.696750+0000","last_fullsized":"2023-10-24T23:53:03.696750+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499911+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-24T23:53:04.499911+0000","last_peered":"2023-10-24T23:53:04.499911+0000","last_clean":"2023-10-24T23:53:04.499911+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-24T23:53:04.499911+0000","last_undegraded":"2023-10-24T23:53:04.499911+0000","last_fullsized":"2023-10-24T23:53:04.499911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497346+0000","last_change":"2023-10-24T23:53:02.728971+0000","last_active":"2023-10-24T23:53:04.497346+0000","last_peered":"2023-10-24T23:53:04.497346+0000","last_clean":"2023-10-24T23:53:04.497346+0000","last_became_active":"2023-10-24T23:53:02.728048+0000","last_became_peered":"2023-10-24T23:53:02.728048+0000","last_unstale":"2023-10-24T23:53:04.497346+0000","last_undegraded":"2023-10-24T23:53:04.497346+0000","last_fullsized":"2023-10-24T23:53:04.497346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:31:32.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498748+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-24T23:53:04.498748+0000","last_peered":"2023-10-24T23:53:04.498748+0000","last_clean":"2023-10-24T23:53:04.498748+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-24T23:53:04.498748+0000","last_undegraded":"2023-10-24T23:53:04.498748+0000","last_fullsized":"2023-10-24T23:53:04.498748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062078+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-24T23:53:04.062078+0000","last_peered":"2023-10-24T23:53:04.062078+0000","last_clean":"2023-10-24T23:53:04.062078+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-24T23:53:04.062078+0000","last_undegraded":"2023-10-24T23:53:04.062078+0000","last_fullsized":"2023-10-24T23:53:04.062078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699415+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-24T23:53:03.699415+0000","last_peered":"2023-10-24T23:53:03.699415+0000","last_clean":"2023-10-24T23:53:03.699415+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-24T23:53:03.699415+0000","last_undegraded":"2023-10-24T23:53:03.699415+0000","last_fullsized":"2023-10-24T23:53:03.699415+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696626+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-24T23:53:03.696626+0000","last_peered":"2023-10-24T23:53:03.696626+0000","last_clean":"2023-10-24T23:53:03.696626+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-24T23:53:03.696626+0000","last_undegraded":"2023-10-24T23:53:03.696626+0000","last_fullsized":"2023-10-24T23:53:03.696626+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696679+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-24T23:53:03.696679+0000","last_peered":"2023-10-24T23:53:03.696679+0000","last_clean":"2023-10-24T23:53:03.696679+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-24T23:53:03.696679+0000","last_undegraded":"2023-10-24T23:53:03.696679+0000","last_fullsized":"2023-10-24T23:53:03.696679+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499695+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-24T23:53:04.499695+0000","last_peered":"2023-10-24T23:53:04.499695+0000","last_clean":"2023-10-24T23:53:04.499695+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-24T23:53:04.499695+0000","last_undegraded":"2023-10-24T23:53:04.499695+0000","last_fullsized":"2023-10-24T23:53:04.499695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699835+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-24T23:53:03.699835+0000","last_peered":"2023-10-24T23:53:03.699835+0000","last_clean":"2023-10-24T23:53:03.699835+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-24T23:53:03.699835+0000","last_undegraded":"2023-10-24T23:53:03.699835+0000","last_fullsized":"2023-10-24T23:53:03.699835+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497251+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-24T23:53:04.497251+0000","last_peered":"2023-10-24T23:53:04.497251+0000","last_clean":"2023-10-24T23:53:04.497251+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-24T23:53:04.497251+0000","last_undegraded":"2023-10-24T23:53:04.497251+0000","last_fullsized":"2023-10-24T23:53:04.497251+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698958+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-24T23:53:03.698958+0000","last_peered":"2023-10-24T23:53:03.698958+0000","last_clean":"2023-10-24T23:53:03.698958+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-24T23:53:03.698958+0000","last_undegraded":"2023-10-24T23:53:03.698958+0000","last_fullsized":"2023-10-24T23:53:03.698958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.061899+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-24T23:53:04.061899+0000","last_peered":"2023-10-24T23:53:04.061899+0000","last_clean":"2023-10-24T23:53:04.061899+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-24T23:53:04.061899+0000","last_undegraded":"2023-10-24T23:53:04.061899+0000","last_fullsized":"2023-10-24T23:53:04.061899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498842+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-24T23:53:04.498842+0000","last_peered":"2023-10-24T23:53:04.498842+0000","last_clean":"2023-10-24T23:53:04.498842+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-24T23:53:04.498842+0000","last_undegraded":"2023-10-24T23:53:04.498842+0000","last_fullsized":"2023-10-24T23:53:04.498842+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497236+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-24T23:53:04.497236+0000","last_peered":"2023-10-24T23:53:04.497236+0000","last_clean":"2023-10-24T23:53:04.497236+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-24T23:53:04.497236+0000","last_undegraded":"2023-10-24T23:53:04.497236+0000","last_fullsized":"2023-10-24T23:53:04.497236+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697967+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-24T23:53:03.697967+0000","last_peered":"2023-10-24T23:53:03.697967+0000","last_clean":"2023-10-24T23:53:03.697967+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-24T23:53:03.697967+0000","last_undegraded":"2023-10-24T23:53:03.697967+0000","last_fullsized":"2023-10-24T23:53:03.697967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499435+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-24T23:53:04.499435+0000","last_peered":"2023-10-24T23:53:04.499435+0000","last_clean":"2023-10-24T23:53:04.499435+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-24T23:53:04.499435+0000","last_undegraded":"2023-10-24T23:53:04.499435+0000","last_fullsized":"2023-10-24T23:53:04.499435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699801+0000","last_change":"2023-10-24T23:53:02.715966+0000","last_active":"2023-10-24T23:53:03.699801+0000","last_peered":"2023-10-24T23:53:03.699801+0000","last_clean":"2023-10-24T23:53:03.699801+0000","last_became_active":"2023-10-24T23:53:02.715631+0000","last_became_peered":"2023-10-24T23:53:02.715631+0000","last_unstale":"2023-10-24T23:53:03.699801+0000","last_undegraded":"2023-10-24T23:53:03.699801+0000","last_fullsized":"2023-10-24T23:53:03.699801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:11.771864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696971+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-24T23:53:03.696971+0000","last_peered":"2023-10-24T23:53:03.696971+0000","last_clean":"2023-10-24T23:53:03.696971+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-24T23:53:03.696971+0000","last_undegraded":"2023-10-24T23:53:03.696971+0000","last_fullsized":"2023-10-24T23:53:03.696971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":864256,"data_stored":681667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31284,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340556,"statfs":{"total":96636764160,"available":96604729344,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31292,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340548,"statfs":{"total":96636764160,"available":96604721152,"internally_reserved":0,"allocated":249856,"data_stored":90171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":262144,"data_stored":91139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":245760,"data_stored":89609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":839680,"data_stored":680469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:16.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2023-10-24T23:53:16.519 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.519 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2023-10-24T23:53:16.520 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.520 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2023-10-24T23:53:16.520 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.521 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2023-10-24T23:53:16.521 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.521 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2023-10-24T23:53:16.522 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.522 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2023-10-24T23:53:16.522 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.523 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2023-10-24T23:53:16.523 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.523 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2023-10-24T23:53:16.524 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.524 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2023-10-24T23:53:16.524 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2023-10-24T23:53:16.525 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.525 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2023-10-24T23:53:16.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.526 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2023-10-24T23:53:16.526 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2023-10-24T23:53:16.527 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.527 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2023-10-24T23:53:16.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.528 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2023-10-24T23:53:16.528 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2023-10-24T23:53:16.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.529 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2023-10-24T23:53:16.529 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2023-10-24T23:53:16.530 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.530 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2023-10-24T23:53:16.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.531 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2023-10-24T23:53:16.531 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2023-10-24T23:53:16.532 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.532 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2023-10-24T23:53:16.533 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.533 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2023-10-24T23:53:16.533 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.534 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2023-10-24T23:53:16.534 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.534 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2023-10-24T23:53:16.535 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.535 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2023-10-24T23:53:16.535 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.536 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2023-10-24T23:53:16.536 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.536 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2023-10-24T23:53:16.537 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.537 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2023-10-24T23:53:16.537 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.538 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2023-10-24T23:53:16.538 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.538 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2023-10-24T23:53:16.539 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.539 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2023-10-24T23:53:16.539 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.540 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2023-10-24T23:53:16.540 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.540 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2023-10-24T23:53:16.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.541 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2023-10-24T23:53:16.541 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2023-10-24T23:53:16.542 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.542 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2023-10-24T23:53:16.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.543 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2023-10-24T23:53:16.543 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2023-10-24T23:53:16.544 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.544 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2023-10-24T23:53:16.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.545 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2023-10-24T23:53:16.545 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2023-10-24T23:53:16.546 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.546 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2023-10-24T23:53:16.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.547 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2023-10-24T23:53:16.547 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2023-10-24T23:53:16.548 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.548 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2023-10-24T23:53:16.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.549 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2023-10-24T23:53:16.549 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2023-10-24T23:53:16.550 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.550 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2023-10-24T23:53:16.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.551 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2023-10-24T23:53:16.551 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2023-10-24T23:53:16.552 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.552 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2023-10-24T23:53:16.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.553 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2023-10-24T23:53:16.553 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2023-10-24T23:53:16.554 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.554 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2023-10-24T23:53:16.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.555 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2023-10-24T23:53:16.555 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2023-10-24T23:53:16.556 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.556 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2023-10-24T23:53:16.557 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.557 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2023-10-24T23:53:16.557 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.558 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2023-10-24T23:53:16.558 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.558 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2023-10-24T23:53:16.559 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.559 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2023-10-24T23:53:16.559 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.560 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2023-10-24T23:53:16.560 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.560 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2023-10-24T23:53:16.561 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:10.374501+0000', 'last_change': '2023-10-24T23:53:10.374501+0000', 'last_active': '2023-10-24T23:53:10.374501+0000', 'last_peered': '2023-10-24T23:53:10.374501+0000', 'last_clean': '2023-10-24T23:53:10.374501+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:10.374501+0000', 'last_undegraded': '2023-10-24T23:53:10.374501+0000', 'last_fullsized': '2023-10-24T23:53:10.374501+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:10.374501+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.561 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2023-10-24T23:53:16.561 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.562 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2023-10-24T23:53:16.562 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.562 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2023-10-24T23:53:16.562 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.563 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2023-10-24T23:53:16.563 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.563 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2023-10-24T23:53:16.564 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.564 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2023-10-24T23:53:16.564 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.565 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2023-10-24T23:53:16.565 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.565 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2023-10-24T23:53:16.566 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.566 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2023-10-24T23:53:16.566 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.567 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2023-10-24T23:53:16.567 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.567 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2023-10-24T23:53:16.568 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.568 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2023-10-24T23:53:16.568 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.569 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2023-10-24T23:53:16.569 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.569 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2023-10-24T23:53:16.570 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.570 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2023-10-24T23:53:16.570 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.571 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2023-10-24T23:53:16.571 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.571 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2023-10-24T23:53:16.572 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.572 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2023-10-24T23:53:16.572 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.573 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2023-10-24T23:53:16.573 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.573 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2023-10-24T23:53:16.574 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.574 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2023-10-24T23:53:16.574 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.575 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2023-10-24T23:53:16.575 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.575 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2023-10-24T23:53:16.576 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.576 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2023-10-24T23:53:16.576 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2023-10-24T23:53:16.577 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.577 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2023-10-24T23:53:16.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.578 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2023-10-24T23:53:16.578 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2023-10-24T23:53:16.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2023-10-24T23:53:16.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2023-10-24T23:53:16.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2023-10-24T23:53:16.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2023-10-24T23:53:16.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2023-10-24T23:53:11.386581+0000', 'last_change': '2023-10-24T23:53:11.386581+0000', 'last_active': '2023-10-24T23:53:11.386581+0000', 'last_peered': '2023-10-24T23:53:11.386581+0000', 'last_clean': '2023-10-24T23:53:11.386581+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-24T23:53:11.386581+0000', 'last_undegraded': '2023-10-24T23:53:11.386581+0000', 'last_fullsized': '2023-10-24T23:53:11.386581+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'last_clean_scrub_stamp': '2023-10-24T23:53:11.386581+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2023-10-24T23:53:16.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-24T23:53:16.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.698+0000 7efe8921f640 1 Processor -- start 2023-10-24T23:53:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.698+0000 7efe8921f640 1 -- start start 2023-10-24T23:53:16.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.698+0000 7efe8921f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efe84152040 0x7efe840c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.698+0000 7efe8921f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe840c8100 0x7efe840abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.698+0000 7efe8921f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe84155680 con 0x7efe8406a4b0 2023-10-24T23:53:16.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.698+0000 7efe8921f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe84154a70 con 0x7efe840c8100 2023-10-24T23:53:16.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.698+0000 7efe8921f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe84154d50 con 0x7efe84152040 2023-10-24T23:53:16.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.698+0000 7efe8259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe840c8100 0x7efe840abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.698+0000 7efe8259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efe840c8100 0x7efe840abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50356/0 (socket says 172.21.15.107:50356) 2023-10-24T23:53:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.698+0000 7efe8259c640 1 -- 172.21.15.107:0/2696541338 learned_addr learned my addr 172.21.15.107:0/2696541338 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe82d9d640 1 --2- 172.21.15.107:0/2696541338 >> v2:172.21.15.110:3300/0 conn(0x7efe84152040 0x7efe840c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8259c640 1 -- 172.21.15.107:0/2696541338 >> v2:172.21.15.110:3300/0 conn(0x7efe84152040 msgr2=0x7efe840c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8259c640 1 --2- 172.21.15.107:0/2696541338 >> v2:172.21.15.110:3300/0 conn(0x7efe84152040 0x7efe840c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8259c640 1 -- 172.21.15.107:0/2696541338 >> v1:172.21.15.110:6789/0 conn(0x7efe8406a4b0 legacy=0x7efe840ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8259c640 1 -- 172.21.15.107:0/2696541338 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe84155030 con 0x7efe840c8100 2023-10-24T23:53:16.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8259c640 1 --2- 172.21.15.107:0/2696541338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe840c8100 0x7efe840abd00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efe84067ba0 tx=0x7efe7401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85df95519c0c32ec server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:16.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe81d9b640 1 -- 172.21.15.107:0/2696541338 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe740ad680 con 0x7efe840c8100 2023-10-24T23:53:16.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe81d9b640 1 -- 172.21.15.107:0/2696541338 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe740ad7e0 con 0x7efe840c8100 2023-10-24T23:53:16.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe81d9b640 1 -- 172.21.15.107:0/2696541338 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe740adb70 con 0x7efe840c8100 2023-10-24T23:53:16.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8921f640 1 -- 172.21.15.107:0/2696541338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe840c8100 msgr2=0x7efe840abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8921f640 1 --2- 172.21.15.107:0/2696541338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe840c8100 0x7efe840abd00 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efe84067ba0 tx=0x7efe7401c470 comp rx=0 tx=0).stop 2023-10-24T23:53:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8921f640 1 -- 172.21.15.107:0/2696541338 shutdown_connections 2023-10-24T23:53:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8921f640 1 --2- 172.21.15.107:0/2696541338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe840c8100 0x7efe840abd00 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8921f640 1 --2- 172.21.15.107:0/2696541338 >> v2:172.21.15.110:3300/0 conn(0x7efe84152040 0x7efe840c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8921f640 1 -- 172.21.15.107:0/2696541338 >> 172.21.15.107:0/2696541338 conn(0x7efe840ca200 msgr2=0x7efe840ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8921f640 1 -- 172.21.15.107:0/2696541338 shutdown_connections 2023-10-24T23:53:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.702+0000 7efe8921f640 1 -- 172.21.15.107:0/2696541338 wait complete. 2023-10-24T23:53:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8921f640 1 Processor -- start 2023-10-24T23:53:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8921f640 1 -- start start 2023-10-24T23:53:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8921f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efe8406a4b0 0x7efe840a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8921f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe840c8100 0x7efe840aaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8921f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe84152040 0x7efe841477a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8921f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe840ab4f0 con 0x7efe840c8100 2023-10-24T23:53:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8921f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8414bb80 con 0x7efe84152040 2023-10-24T23:53:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8921f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe840aa2c0 con 0x7efe8406a4b0 2023-10-24T23:53:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe840c8100 0x7efe840aaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe840c8100 0x7efe840aaa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50366/0 (socket says 172.21.15.107:50366) 2023-10-24T23:53:16.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8259c640 1 -- 172.21.15.107:0/2528954416 learned_addr learned my addr 172.21.15.107:0/2528954416 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:16.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe82d9d640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efe8406a4b0 0x7efe840a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8359e640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe84152040 0x7efe841477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8259c640 1 -- 172.21.15.107:0/2528954416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efe8406a4b0 msgr2=0x7efe840a9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8259c640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efe8406a4b0 0x7efe840a9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8259c640 1 -- 172.21.15.107:0/2528954416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe84152040 msgr2=0x7efe841477a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8259c640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe84152040 0x7efe841477a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8259c640 1 -- 172.21.15.107:0/2528954416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe740094d0 con 0x7efe840c8100 2023-10-24T23:53:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8259c640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe840c8100 0x7efe840aaa00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efe74019080 tx=0x7efe7401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe6b7fe640 1 -- 172.21.15.107:0/2528954416 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe740ade20 con 0x7efe840c8100 2023-10-24T23:53:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe6b7fe640 1 -- 172.21.15.107:0/2528954416 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe740b8450 con 0x7efe840c8100 2023-10-24T23:53:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8921f640 1 -- 172.21.15.107:0/2528954416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe84149580 con 0x7efe840c8100 2023-10-24T23:53:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe6b7fe640 1 -- 172.21.15.107:0/2528954416 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe740b87c0 con 0x7efe840c8100 2023-10-24T23:53:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8921f640 1 -- 172.21.15.107:0/2528954416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe84149b40 con 0x7efe840c8100 2023-10-24T23:53:16.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe8921f640 1 -- 172.21.15.107:0/2528954416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8414bcf0 con 0x7efe840c8100 2023-10-24T23:53:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe6b7fe640 1 -- 172.21.15.107:0/2528954416 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7efe740b4070 con 0x7efe840c8100 2023-10-24T23:53:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe6b7fe640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efe70065e90 0x7efe70068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe6b7fe640 1 -- 172.21.15.107:0/2528954416 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7efe7411dea0 con 0x7efe840c8100 2023-10-24T23:53:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.706+0000 7efe82d9d640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efe70065e90 0x7efe70068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.710+0000 7efe82d9d640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efe70065e90 0x7efe70068350 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efe780096f0 tx=0x7efe78008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:16.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.710+0000 7efe6b7fe640 1 -- 172.21.15.107:0/2528954416 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7efe8414bcf0 con 0x7efe840c8100 2023-10-24T23:53:16.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.906+0000 7efe8921f640 1 -- 172.21.15.107:0/2528954416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7efe8414bcf0 con 0x7efe840c8100 2023-10-24T23:53:16.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe6b7fe640 1 -- 172.21.15.107:0/2528954416 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1126 (secure 0 0 0) 0x7efe8414bcf0 con 0x7efe840c8100 2023-10-24T23:53:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:16.913 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":38,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":27,"num_osds":8,"num_up_osds":8,"osd_up_since":1698191569,"num_in_osds":8,"osd_in_since":1698191560,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":105},{"state_name":"unknown","count":96}],"num_pgs":201,"num_pools":6,"num_objects":24,"data_bytes":592730,"bytes_used":240754688,"bytes_avail":772853358592,"bytes_total":773094113280,"unknown_pgs_ratio":0.47761192917823792},"fsmap":{"epoch":12,"id":1,"up":1,"in":1,"max":1,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":4293},{"filesystem_id":1,"rank":0,"name":"d","status":"up:active","gid":4299}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2023-10-24T23:52:38.580619+0000","services":{}},"progress_events":{}} 2023-10-24T23:53:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 -- 172.21.15.107:0/2528954416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efe70065e90 msgr2=0x7efe70068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efe70065e90 0x7efe70068350 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efe780096f0 tx=0x7efe78008040 comp rx=0 tx=0).stop 2023-10-24T23:53:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 -- 172.21.15.107:0/2528954416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe840c8100 msgr2=0x7efe840aaa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe840c8100 0x7efe840aaa00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efe74019080 tx=0x7efe7401acb0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 -- 172.21.15.107:0/2528954416 shutdown_connections 2023-10-24T23:53:16.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efe84152040 0x7efe841477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efe840c8100 0x7efe840aaa00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efe70065e90 0x7efe70068350 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 --2- 172.21.15.107:0/2528954416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efe8406a4b0 0x7efe840a9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:16.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 -- 172.21.15.107:0/2528954416 >> 172.21.15.107:0/2528954416 conn(0x7efe840ca200 msgr2=0x7efe840ae3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:16.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 -- 172.21.15.107:0/2528954416 shutdown_connections 2023-10-24T23:53:16.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:16.910+0000 7efe8921f640 1 -- 172.21.15.107:0/2528954416 wait complete. 2023-10-24T23:53:16.930 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-24T23:53:16.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.038+0000 7f26bc1ac640 1 Processor -- start 2023-10-24T23:53:17.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.038+0000 7f26bc1ac640 1 -- start start 2023-10-24T23:53:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26bc1ac640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f26b414fd40 0x7f26b40ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26bc1ac640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b40a9820 0x7f26b40aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26bc1ac640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b406f730 con 0x7f26b406ee40 2023-10-24T23:53:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26bc1ac640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b40aa750 con 0x7f26b414fd40 2023-10-24T23:53:17.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26bc1ac640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4156a40 con 0x7f26b40a9820 2023-10-24T23:53:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b9f48640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f26b414fd40 0x7f26b40ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b9f48640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f26b414fd40 0x7f26b40ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50374/0 (socket says 172.21.15.107:50374) 2023-10-24T23:53:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b9f48640 1 -- 172.21.15.107:0/2912976488 learned_addr learned my addr 172.21.15.107:0/2912976488 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b9747640 1 --2- 172.21.15.107:0/2912976488 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b40a9820 0x7f26b40aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:17.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b9f48640 1 -- 172.21.15.107:0/2912976488 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b40a9820 msgr2=0x7f26b40aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:17.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b9f48640 1 --2- 172.21.15.107:0/2912976488 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b40a9820 0x7f26b40aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:17.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b9f48640 1 -- 172.21.15.107:0/2912976488 >> v1:172.21.15.110:6789/0 conn(0x7f26b406ee40 legacy=0x7f26b40c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:17.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b9f48640 1 -- 172.21.15.107:0/2912976488 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b4156710 con 0x7f26b414fd40 2023-10-24T23:53:17.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b9747640 1 --2- 172.21.15.107:0/2912976488 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b40a9820 0x7f26b40aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:17.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b9f48640 1 --2- 172.21.15.107:0/2912976488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26b414fd40 0x7f26b40ab090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009870 tx=0x7f26b001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b36ddb951541cd1 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:17.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b8f46640 1 -- 172.21.15.107:0/2912976488 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b00ac640 con 0x7f26b414fd40 2023-10-24T23:53:17.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b8f46640 1 -- 172.21.15.107:0/2912976488 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26b00ac7a0 con 0x7f26b414fd40 2023-10-24T23:53:17.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.042+0000 7f26b8f46640 1 -- 172.21.15.107:0/2912976488 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b00acb30 con 0x7f26b414fd40 2023-10-24T23:53:17.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2912976488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26b414fd40 msgr2=0x7f26b40ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:17.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 --2- 172.21.15.107:0/2912976488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26b414fd40 0x7f26b40ab090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f26b0009870 tx=0x7f26b001c3b0 comp rx=0 tx=0).stop 2023-10-24T23:53:17.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2912976488 shutdown_connections 2023-10-24T23:53:17.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 --2- 172.21.15.107:0/2912976488 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b40a9820 0x7f26b40aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:17.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 --2- 172.21.15.107:0/2912976488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26b414fd40 0x7f26b40ab090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:17.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2912976488 >> 172.21.15.107:0/2912976488 conn(0x7f26b40c2950 msgr2=0x7f26b40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:17.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2912976488 shutdown_connections 2023-10-24T23:53:17.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2912976488 wait complete. 2023-10-24T23:53:17.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 Processor -- start 2023-10-24T23:53:17.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 -- start start 2023-10-24T23:53:17.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b406ee40 0x7f26b41549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26b414fd40 0x7f26b4144670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26b41474b0 0x7f26b414e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b41539d0 con 0x7f26b414fd40 2023-10-24T23:53:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4153c20 con 0x7f26b41474b0 2023-10-24T23:53:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4153f00 con 0x7f26b406ee40 2023-10-24T23:53:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26b9747640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26b414fd40 0x7f26b4144670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26b9747640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26b414fd40 0x7f26b4144670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50382/0 (socket says 172.21.15.107:50382) 2023-10-24T23:53:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26b9747640 1 -- 172.21.15.107:0/2346831188 learned_addr learned my addr 172.21.15.107:0/2346831188 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26b9f48640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b406ee40 0x7f26b41549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26ba749640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26b41474b0 0x7f26b414e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:17.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26b9747640 1 -- 172.21.15.107:0/2346831188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b406ee40 msgr2=0x7f26b41549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:17.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26b9747640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b406ee40 0x7f26b41549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:17.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26b9747640 1 -- 172.21.15.107:0/2346831188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26b41474b0 msgr2=0x7f26b414e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:17.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26b9747640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26b41474b0 0x7f26b414e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:17.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26b9747640 1 -- 172.21.15.107:0/2346831188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b00094d0 con 0x7f26b414fd40 2023-10-24T23:53:17.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26b9f48640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b406ee40 0x7f26b41549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:17.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26ba749640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26b41474b0 0x7f26b414e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:17.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26b9747640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26b414fd40 0x7f26b4144670 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f26a400c7c0 tx=0x7f26a400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:17.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26aa7fc640 1 -- 172.21.15.107:0/2346831188 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4007a30 con 0x7f26b414fd40 2023-10-24T23:53:17.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26aa7fc640 1 -- 172.21.15.107:0/2346831188 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a4007b90 con 0x7f26b414fd40 2023-10-24T23:53:17.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2346831188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b41541e0 con 0x7f26b414fd40 2023-10-24T23:53:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26aa7fc640 1 -- 172.21.15.107:0/2346831188 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a400ce70 con 0x7f26b414fd40 2023-10-24T23:53:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.046+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2346831188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b414d940 con 0x7f26b414fd40 2023-10-24T23:53:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.050+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2346831188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b40c24a0 con 0x7f26b414fd40 2023-10-24T23:53:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.050+0000 7f26aa7fc640 1 -- 172.21.15.107:0/2346831188 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f26a40113f0 con 0x7f26b414fd40 2023-10-24T23:53:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.050+0000 7f26aa7fc640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2698065e60 0x7f2698068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.050+0000 7f26aa7fc640 1 -- 172.21.15.107:0/2346831188 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f26a4083f60 con 0x7f26b414fd40 2023-10-24T23:53:17.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.050+0000 7f26b9f48640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2698065e60 0x7f2698068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:17.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.050+0000 7f26b9f48640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2698065e60 0x7f2698068320 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f26b00ae000 tx=0x7f26b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:17.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.054+0000 7f26aa7fc640 1 -- 172.21.15.107:0/2346831188 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f26a4051310 con 0x7f26b414fd40 2023-10-24T23:53:17.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.194+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2346831188 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f26b4153d90 con 0x7f2698065e60 2023-10-24T23:53:17.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.230+0000 7f26aa7fc640 1 -- 172.21.15.107:0/2346831188 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+510297 (secure 0 0 0) 0x7f26b4153d90 con 0x7f2698065e60 2023-10-24T23:53:17.238 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:17.242 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:17.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2346831188 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2698065e60 msgr2=0x7f2698068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:17.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2698065e60 0x7f2698068320 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f26b00ae000 tx=0x7f26b0008040 comp rx=0 tx=0).stop 2023-10-24T23:53:17.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2346831188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26b414fd40 msgr2=0x7f26b4144670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:17.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26b414fd40 0x7f26b4144670 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f26a400c7c0 tx=0x7f26a400cc80 comp rx=0 tx=0).stop 2023-10-24T23:53:17.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2346831188 shutdown_connections 2023-10-24T23:53:17.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26b41474b0 0x7f26b414e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:17.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26b414fd40 0x7f26b4144670 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:17.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2698065e60 0x7f2698068320 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:17.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 --2- 172.21.15.107:0/2346831188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26b406ee40 0x7f26b41549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:17.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2346831188 >> 172.21.15.107:0/2346831188 conn(0x7f26b40c2950 msgr2=0x7f26b40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:17.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2346831188 shutdown_connections 2023-10-24T23:53:17.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:17.234+0000 7f26bc1ac640 1 -- 172.21.15.107:0/2346831188 wait complete. 2023-10-24T23:53:17.257 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2023-10-24T23:53:15.279214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235112,"kb_used_data":3112,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754739608,"statfs":{"total":773094113280,"available":772853358592,"internally_reserved":0,"allocated":3186688,"data_stored":1901612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.892239"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497423+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-24T23:53:04.497423+0000","last_peered":"2023-10-24T23:53:04.497423+0000","last_clean":"2023-10-24T23:53:04.497423+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-24T23:53:04.497423+0000","last_undegraded":"2023-10-24T23:53:04.497423+0000","last_fullsized":"2023-10-24T23:53:04.497423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498958+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-24T23:53:04.498958+0000","last_peered":"2023-10-24T23:53:04.498958+0000","last_clean":"2023-10-24T23:53:04.498958+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-24T23:53:04.498958+0000","last_undegraded":"2023-10-24T23:53:04.498958+0000","last_fullsized":"2023-10-24T23:53:04.498958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165883+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-24T23:53:04.165883+0000","last_peered":"2023-10-24T23:53:04.165883+0000","last_clean":"2023-10-24T23:53:04.165883+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-24T23:53:04.165883+0000","last_undegraded":"2023-10-24T23:53:04.165883+0000","last_fullsized":"2023-10-24T23:53:04.165883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166134+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-24T23:53:04.166134+0000","last_peered":"2023-10-24T23:53:04.166134+0000","last_clean":"2023-10-24T23:53:04.166134+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-24T23:53:04.166134+0000","last_undegraded":"2023-10-24T23:53:04.166134+0000","last_fullsized":"2023-10-24T23:53:04.166134+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699826+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-24T23:53:03.699826+0000","last_peered":"2023-10-24T23:53:03.699826+0000","last_clean":"2023-10-24T23:53:03.699826+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-24T23:53:03.699826+0000","last_undegraded":"2023-10-24T23:53:03.699826+0000","last_fullsized":"2023-10-24T23:53:03.699826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499034+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-24T23:53:04.499034+0000","last_peered":"2023-10-24T23:53:04.499034+0000","last_clean":"2023-10-24T23:53:04.499034+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-24T23:53:04.499034+0000","last_undegraded":"2023-10-24T23:53:04.499034+0000","last_fullsized":"2023-10-24T23:53:04.499034+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699150+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-24T23:53:03.699150+0000","last_peered":"2023-10-24T23:53:03.699150+0000","last_clean":"2023-10-24T23:53:03.699150+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-24T23:53:03.699150+0000","last_undegraded":"2023-10-24T23:53:03.699150+0000","last_fullsized":"2023-10-24T23:53:03.699150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499513+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-24T23:53:04.499513+0000","last_peered":"2023-10-24T23:53:04.499513+0000","last_clean":"2023-10-24T23:53:04.499513+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-24T23:53:04.499513+0000","last_undegraded":"2023-10-24T23:53:04.499513+0000","last_fullsized":"2023-10-24T23:53:04.499513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497288+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-24T23:53:04.497288+0000","last_peered":"2023-10-24T23:53:04.497288+0000","last_clean":"2023-10-24T23:53:04.497288+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-24T23:53:04.497288+0000","last_undegraded":"2023-10-24T23:53:04.497288+0000","last_fullsized":"2023-10-24T23:53:04.497288+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696609+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-24T23:53:03.696609+0000","last_peered":"2023-10-24T23:53:03.696609+0000","last_clean":"2023-10-24T23:53:03.696609+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-24T23:53:03.696609+0000","last_undegraded":"2023-10-24T23:53:03.696609+0000","last_fullsized":"2023-10-24T23:53:03.696609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499601+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-24T23:53:04.499601+0000","last_peered":"2023-10-24T23:53:04.499601+0000","last_clean":"2023-10-24T23:53:04.499601+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-24T23:53:04.499601+0000","last_undegraded":"2023-10-24T23:53:04.499601+0000","last_fullsized":"2023-10-24T23:53:04.499601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166207+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-24T23:53:04.166207+0000","last_peered":"2023-10-24T23:53:04.166207+0000","last_clean":"2023-10-24T23:53:04.166207+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-24T23:53:04.166207+0000","last_undegraded":"2023-10-24T23:53:04.166207+0000","last_fullsized":"2023-10-24T23:53:04.166207+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496826+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-24T23:53:04.496826+0000","last_peered":"2023-10-24T23:53:04.496826+0000","last_clean":"2023-10-24T23:53:04.496826+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-24T23:53:04.496826+0000","last_undegraded":"2023-10-24T23:53:04.496826+0000","last_fullsized":"2023-10-24T23:53:04.496826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497403+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-24T23:53:04.497403+0000","last_peered":"2023-10-24T23:53:04.497403+0000","last_clean":"2023-10-24T23:53:04.497403+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-24T23:53:04.497403+0000","last_undegraded":"2023-10-24T23:53:04.497403+0000","last_fullsized":"2023-10-24T23:53:04.497403+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165441+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-24T23:53:04.165441+0000","last_peered":"2023-10-24T23:53:04.165441+0000","last_clean":"2023-10-24T23:53:04.165441+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-24T23:53:04.165441+0000","last_undegraded":"2023-10-24T23:53:04.165441+0000","last_fullsized":"2023-10-24T23:53:04.165441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697959+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-24T23:53:03.697959+0000","last_peered":"2023-10-24T23:53:03.697959+0000","last_clean":"2023-10-24T23:53:03.697959+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-24T23:53:03.697959+0000","last_undegraded":"2023-10-24T23:53:03.697959+0000","last_fullsized":"2023-10-24T23:53:03.697959+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165168+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-24T23:53:04.165168+0000","last_peered":"2023-10-24T23:53:04.165168+0000","last_clean":"2023-10-24T23:53:04.165168+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-24T23:53:04.165168+0000","last_undegraded":"2023-10-24T23:53:04.165168+0000","last_fullsized":"2023-10-24T23:53:04.165168+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497493+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-24T23:53:04.497493+0000","last_peered":"2023-10-24T23:53:04.497493+0000","last_clean":"2023-10-24T23:53:04.497493+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-24T23:53:04.497493+0000","last_undegraded":"2023-10-24T23:53:04.497493+0000","last_fullsized":"2023-10-24T23:53:04.497493+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165604+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-24T23:53:04.165604+0000","last_peered":"2023-10-24T23:53:04.165604+0000","last_clean":"2023-10-24T23:53:04.165604+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-24T23:53:04.165604+0000","last_undegraded":"2023-10-24T23:53:04.165604+0000","last_fullsized":"2023-10-24T23:53:04.165604+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699929+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-24T23:53:03.699929+0000","last_peered":"2023-10-24T23:53:03.699929+0000","last_clean":"2023-10-24T23:53:03.699929+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-24T23:53:03.699929+0000","last_undegraded":"2023-10-24T23:53:03.699929+0000","last_fullsized":"2023-10-24T23:53:03.699929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062786+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-24T23:53:04.062786+0000","last_peered":"2023-10-24T23:53:04.062786+0000","last_clean":"2023-10-24T23:53:04.062786+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-24T23:53:04.062786+0000","last_undegraded":"2023-10-24T23:53:04.062786+0000","last_fullsized":"2023-10-24T23:53:04.062786+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496844+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-24T23:53:04.496844+0000","last_peered":"2023-10-24T23:53:04.496844+0000","last_clean":"2023-10-24T23:53:04.496844+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-24T23:53:04.496844+0000","last_undegraded":"2023-10-24T23:53:04.496844+0000","last_fullsized":"2023-10-24T23:53:04.496844+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499662+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-24T23:53:04.499662+0000","last_peered":"2023-10-24T23:53:04.499662+0000","last_clean":"2023-10-24T23:53:04.499662+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-24T23:53:04.499662+0000","last_undegraded":"2023-10-24T23:53:04.499662+0000","last_fullsized":"2023-10-24T23:53:04.499662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499511+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-24T23:53:04.499511+0000","last_peered":"2023-10-24T23:53:04.499511+0000","last_clean":"2023-10-24T23:53:04.499511+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-24T23:53:04.499511+0000","last_undegraded":"2023-10-24T23:53:04.499511+0000","last_fullsized":"2023-10-24T23:53:04.499511+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499388+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-24T23:53:04.499388+0000","last_peered":"2023-10-24T23:53:04.499388+0000","last_clean":"2023-10-24T23:53:04.499388+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-24T23:53:04.499388+0000","last_undegraded":"2023-10-24T23:53:04.499388+0000","last_fullsized":"2023-10-24T23:53:04.499388+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.693689+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:02.693689+0000","last_peered":"2023-10-24T23:53:02.693689+0000","last_clean":"2023-10-24T23:53:02.693689+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:02.693689+0000","last_undegraded":"2023-10-24T23:53:02.693689+0000","last_fullsized":"2023-10-24T23:53:02.693689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:05.341603+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-24T23:53:05.341603+0000","last_peered":"2023-10-24T23:53:05.341603+0000","last_clean":"2023-10-24T23:53:05.341603+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-24T23:53:05.341603+0000","last_undegraded":"2023-10-24T23:53:05.341603+0000","last_fullsized":"2023-10-24T23:53:05.341603+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165320+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-24T23:53:04.165320+0000","last_peered":"2023-10-24T23:53:04.165320+0000","last_clean":"2023-10-24T23:53:04.165320+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-24T23:53:04.165320+0000","last_undegraded":"2023-10-24T23:53:04.165320+0000","last_fullsized":"2023-10-24T23:53:04.165320+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.705879+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:02.705879+0000","last_peered":"2023-10-24T23:53:02.705879+0000","last_clean":"2023-10-24T23:53:02.705879+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:02.705879+0000","last_undegraded":"2023-10-24T23:53:02.705879+0000","last_fullsized":"2023-10-24T23:53:02.705879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166479+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-24T23:53:04.166479+0000","last_peered":"2023-10-24T23:53:04.166479+0000","last_clean":"2023-10-24T23:53:04.166479+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-24T23:53:04.166479+0000","last_undegraded":"2023-10-24T23:53:04.166479+0000","last_fullsized":"2023-10-24T23:53:04.166479+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497155+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-24T23:53:04.497155+0000","last_peered":"2023-10-24T23:53:04.497155+0000","last_clean":"2023-10-24T23:53:04.497155+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-24T23:53:04.497155+0000","last_undegraded":"2023-10-24T23:53:04.497155+0000","last_fullsized":"2023-10-24T23:53:04.497155+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.714178+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:02.714178+0000","last_peered":"2023-10-24T23:53:02.714178+0000","last_clean":"2023-10-24T23:53:02.714178+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:02.714178+0000","last_undegraded":"2023-10-24T23:53:02.714178+0000","last_fullsized":"2023-10-24T23:53:02.714178+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497005+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-24T23:53:04.497005+0000","last_peered":"2023-10-24T23:53:04.497005+0000","last_clean":"2023-10-24T23:53:04.497005+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-24T23:53:04.497005+0000","last_undegraded":"2023-10-24T23:53:04.497005+0000","last_fullsized":"2023-10-24T23:53:04.497005+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496893+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-24T23:53:04.496893+0000","last_peered":"2023-10-24T23:53:04.496893+0000","last_clean":"2023-10-24T23:53:04.496893+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-24T23:53:04.496893+0000","last_undegraded":"2023-10-24T23:53:04.496893+0000","last_fullsized":"2023-10-24T23:53:04.496893+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.710865+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:02.710865+0000","last_peered":"2023-10-24T23:53:02.710865+0000","last_clean":"2023-10-24T23:53:02.710865+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:02.710865+0000","last_undegraded":"2023-10-24T23:53:02.710865+0000","last_fullsized":"2023-10-24T23:53:02.710865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699976+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-24T23:53:03.699976+0000","last_peered":"2023-10-24T23:53:03.699976+0000","last_clean":"2023-10-24T23:53:03.699976+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-24T23:53:03.699976+0000","last_undegraded":"2023-10-24T23:53:03.699976+0000","last_fullsized":"2023-10-24T23:53:03.699976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496956+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-24T23:53:04.496956+0000","last_peered":"2023-10-24T23:53:04.496956+0000","last_clean":"2023-10-24T23:53:04.496956+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-24T23:53:04.496956+0000","last_undegraded":"2023-10-24T23:53:04.496956+0000","last_fullsized":"2023-10-24T23:53:04.496956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499793+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-24T23:53:04.499793+0000","last_peered":"2023-10-24T23:53:04.499793+0000","last_clean":"2023-10-24T23:53:04.499793+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-24T23:53:04.499793+0000","last_undegraded":"2023-10-24T23:53:04.499793+0000","last_fullsized":"2023-10-24T23:53:04.499793+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.691164+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:53:02.691164+0000","last_peered":"2023-10-24T23:53:02.691164+0000","last_clean":"2023-10-24T23:53:02.691164+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:53:02.691164+0000","last_undegraded":"2023-10-24T23:53:02.691164+0000","last_fullsized":"2023-10-24T23:53:02.691164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497633+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-24T23:53:04.497633+0000","last_peered":"2023-10-24T23:53:04.497633+0000","last_clean":"2023-10-24T23:53:04.497633+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-24T23:53:04.497633+0000","last_undegraded":"2023-10-24T23:53:04.497633+0000","last_fullsized":"2023-10-24T23:53:04.497633+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496567+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-24T23:53:04.496567+0000","last_peered":"2023-10-24T23:53:04.496567+0000","last_clean":"2023-10-24T23:53:04.496567+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-24T23:53:04.496567+0000","last_undegraded":"2023-10-24T23:53:04.496567+0000","last_fullsized":"2023-10-24T23:53:04.496567+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.688268+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:53:02.688268+0000","last_peered":"2023-10-24T23:53:02.688268+0000","last_clean":"2023-10-24T23:53:02.688268+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:53:02.688268+0000","last_undegraded":"2023-10-24T23:53:02.688268+0000","last_fullsized":"2023-10-24T23:53:02.688268+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497538+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-24T23:53:04.497538+0000","last_peered":"2023-10-24T23:53:04.497538+0000","last_clean":"2023-10-24T23:53:04.497538+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-24T23:53:04.497538+0000","last_undegraded":"2023-10-24T23:53:04.497538+0000","last_fullsized":"2023-10-24T23:53:04.497538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496751+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-24T23:53:04.496751+0000","last_peered":"2023-10-24T23:53:04.496751+0000","last_clean":"2023-10-24T23:53:04.496751+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-24T23:53:04.496751+0000","last_undegraded":"2023-10-24T23:53:04.496751+0000","last_fullsized":"2023-10-24T23:53:04.496751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.713207+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:02.713207+0000","last_peered":"2023-10-24T23:53:02.713207+0000","last_clean":"2023-10-24T23:53:02.713207+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:02.713207+0000","last_undegraded":"2023-10-24T23:53:02.713207+0000","last_fullsized":"2023-10-24T23:53:02.713207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698126+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-24T23:53:03.698126+0000","last_peered":"2023-10-24T23:53:03.698126+0000","last_clean":"2023-10-24T23:53:03.698126+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-24T23:53:03.698126+0000","last_undegraded":"2023-10-24T23:53:03.698126+0000","last_fullsized":"2023-10-24T23:53:03.698126+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165624+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-24T23:53:04.165624+0000","last_peered":"2023-10-24T23:53:04.165624+0000","last_clean":"2023-10-24T23:53:04.165624+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-24T23:53:04.165624+0000","last_undegraded":"2023-10-24T23:53:04.165624+0000","last_fullsized":"2023-10-24T23:53:04.165624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.689415+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:53:02.689415+0000","last_peered":"2023-10-24T23:53:02.689415+0000","last_clean":"2023-10-24T23:53:02.689415+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:53:02.689415+0000","last_undegraded":"2023-10-24T23:53:02.689415+0000","last_fullsized":"2023-10-24T23:53:02.689415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":167,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-24T23:53:02.713291+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:53:02.713291+0000","last_peered":"2023-10-24T23:53:02.713291+0000","last_clean":"2023-10-24T23:53:02.713291+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:53:02.713291+0000","last_undegraded":"2023-10-24T23:53:02.713291+0000","last_fullsized":"2023-10-24T23:53:02.713291+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497591+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-24T23:53:04.497591+0000","last_peered":"2023-10-24T23:53:04.497591+0000","last_clean":"2023-10-24T23:53:04.497591+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-24T23:53:04.497591+0000","last_undegraded":"2023-10-24T23:53:04.497591+0000","last_fullsized":"2023-10-24T23:53:04.497591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165664+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-24T23:53:04.165664+0000","last_peered":"2023-10-24T23:53:04.165664+0000","last_clean":"2023-10-24T23:53:04.165664+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-24T23:53:04.165664+0000","last_undegraded":"2023-10-24T23:53:04.165664+0000","last_fullsized":"2023-10-24T23:53:04.165664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499944+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-24T23:53:04.499944+0000","last_peered":"2023-10-24T23:53:04.499944+0000","last_clean":"2023-10-24T23:53:04.499944+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-24T23:53:04.499944+0000","last_undegraded":"2023-10-24T23:53:04.499944+0000","last_fullsized":"2023-10-24T23:53:04.499944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499622+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-24T23:53:04.499622+0000","last_peered":"2023-10-24T23:53:04.499622+0000","last_clean":"2023-10-24T23:53:04.499622+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-24T23:53:04.499622+0000","last_undegraded":"2023-10-24T23:53:04.499622+0000","last_fullsized":"2023-10-24T23:53:04.499622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499791+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-24T23:53:04.499791+0000","last_peered":"2023-10-24T23:53:04.499791+0000","last_clean":"2023-10-24T23:53:04.499791+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-24T23:53:04.499791+0000","last_undegraded":"2023-10-24T23:53:04.499791+0000","last_fullsized":"2023-10-24T23:53:04.499791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698204+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-24T23:53:03.698204+0000","last_peered":"2023-10-24T23:53:03.698204+0000","last_clean":"2023-10-24T23:53:03.698204+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-24T23:53:03.698204+0000","last_undegraded":"2023-10-24T23:53:03.698204+0000","last_fullsized":"2023-10-24T23:53:03.698204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499931+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-24T23:53:04.499931+0000","last_peered":"2023-10-24T23:53:04.499931+0000","last_clean":"2023-10-24T23:53:04.499931+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-24T23:53:04.499931+0000","last_undegraded":"2023-10-24T23:53:04.499931+0000","last_fullsized":"2023-10-24T23:53:04.499931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165994+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-24T23:53:04.165994+0000","last_peered":"2023-10-24T23:53:04.165994+0000","last_clean":"2023-10-24T23:53:04.165994+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-24T23:53:04.165994+0000","last_undegraded":"2023-10-24T23:53:04.165994+0000","last_fullsized":"2023-10-24T23:53:04.165994+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496747+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-24T23:53:04.496747+0000","last_peered":"2023-10-24T23:53:04.496747+0000","last_clean":"2023-10-24T23:53:04.496747+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-24T23:53:04.496747+0000","last_undegraded":"2023-10-24T23:53:04.496747+0000","last_fullsized":"2023-10-24T23:53:04.496747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699348+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-24T23:53:03.699348+0000","last_peered":"2023-10-24T23:53:03.699348+0000","last_clean":"2023-10-24T23:53:03.699348+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-24T23:53:03.699348+0000","last_undegraded":"2023-10-24T23:53:03.699348+0000","last_fullsized":"2023-10-24T23:53:03.699348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697568+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-24T23:53:03.697568+0000","last_peered":"2023-10-24T23:53:03.697568+0000","last_clean":"2023-10-24T23:53:03.697568+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-24T23:53:03.697568+0000","last_undegraded":"2023-10-24T23:53:03.697568+0000","last_fullsized":"2023-10-24T23:53:03.697568+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062733+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-24T23:53:04.062733+0000","last_peered":"2023-10-24T23:53:04.062733+0000","last_clean":"2023-10-24T23:53:04.062733+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-24T23:53:04.062733+0000","last_undegraded":"2023-10-24T23:53:04.062733+0000","last_fullsized":"2023-10-24T23:53:04.062733+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499840+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-24T23:53:04.499840+0000","last_peered":"2023-10-24T23:53:04.499840+0000","last_clean":"2023-10-24T23:53:04.499840+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-24T23:53:04.499840+0000","last_undegraded":"2023-10-24T23:53:04.499840+0000","last_fullsized":"2023-10-24T23:53:04.499840+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.063027+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-24T23:53:04.063027+0000","last_peered":"2023-10-24T23:53:04.063027+0000","last_clean":"2023-10-24T23:53:04.063027+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-24T23:53:04.063027+0000","last_undegraded":"2023-10-24T23:53:04.063027+0000","last_fullsized":"2023-10-24T23:53:04.063027+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698927+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-24T23:53:03.698927+0000","last_peered":"2023-10-24T23:53:03.698927+0000","last_clean":"2023-10-24T23:53:03.698927+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-24T23:53:03.698927+0000","last_undegraded":"2023-10-24T23:53:03.698927+0000","last_fullsized":"2023-10-24T23:53:03.698927+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.063138+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-24T23:53:04.063138+0000","last_peered":"2023-10-24T23:53:04.063138+0000","last_clean":"2023-10-24T23:53:04.063138+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-24T23:53:04.063138+0000","last_undegraded":"2023-10-24T23:53:04.063138+0000","last_fullsized":"2023-10-24T23:53:04.063138+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":19,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697647+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-24T23:53:03.697647+0000","last_peered":"2023-10-24T23:53:03.697647+0000","last_clean":"2023-10-24T23:53:03.697647+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-24T23:53:03.697647+0000","last_undegraded":"2023-10-24T23:53:03.697647+0000","last_fullsized":"2023-10-24T23:53:03.697647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166270+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-24T23:53:04.166270+0000","last_peered":"2023-10-24T23:53:04.166270+0000","last_clean":"2023-10-24T23:53:04.166270+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-24T23:53:04.166270+0000","last_undegraded":"2023-10-24T23:53:04.166270+0000","last_fullsized":"2023-10-24T23:53:04.166270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062154+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-24T23:53:04.062154+0000","last_peered":"2023-10-24T23:53:04.062154+0000","last_clean":"2023-10-24T23:53:04.062154+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-24T23:53:04.062154+0000","last_undegraded":"2023-10-24T23:53:04.062154+0000","last_fullsized":"2023-10-24T23:53:04.062154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497582+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-24T23:53:04.497582+0000","last_peered":"2023-10-24T23:53:04.497582+0000","last_clean":"2023-10-24T23:53:04.497582+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-24T23:53:04.497582+0000","last_undegraded":"2023-10-24T23:53:04.497582+0000","last_fullsized":"2023-10-24T23:53:04.497582+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497015+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-24T23:53:04.497015+0000","last_peered":"2023-10-24T23:53:04.497015+0000","last_clean":"2023-10-24T23:53:04.497015+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-24T23:53:04.497015+0000","last_undegraded":"2023-10-24T23:53:04.497015+0000","last_fullsized":"2023-10-24T23:53:04.497015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699735+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-24T23:53:03.699735+0000","last_peered":"2023-10-24T23:53:03.699735+0000","last_clean":"2023-10-24T23:53:03.699735+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-24T23:53:03.699735+0000","last_undegraded":"2023-10-24T23:53:03.699735+0000","last_fullsized":"2023-10-24T23:53:03.699735+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":13,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062264+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-24T23:53:04.062264+0000","last_peered":"2023-10-24T23:53:04.062264+0000","last_clean":"2023-10-24T23:53:04.062264+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-24T23:53:04.062264+0000","last_undegraded":"2023-10-24T23:53:04.062264+0000","last_fullsized":"2023-10-24T23:53:04.062264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.166359+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-24T23:53:04.166359+0000","last_peered":"2023-10-24T23:53:04.166359+0000","last_clean":"2023-10-24T23:53:04.166359+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-24T23:53:04.166359+0000","last_undegraded":"2023-10-24T23:53:04.166359+0000","last_fullsized":"2023-10-24T23:53:04.166359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:10.374501+0000","last_change":"2023-10-24T23:53:10.374501+0000","last_active":"2023-10-24T23:53:10.374501+0000","last_peered":"2023-10-24T23:53:10.374501+0000","last_clean":"2023-10-24T23:53:10.374501+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:10.374501+0000","last_undegraded":"2023-10-24T23:53:10.374501+0000","last_fullsized":"2023-10-24T23:53:10.374501+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698525+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-24T23:53:03.698525+0000","last_peered":"2023-10-24T23:53:03.698525+0000","last_clean":"2023-10-24T23:53:03.698525+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-24T23:53:03.698525+0000","last_undegraded":"2023-10-24T23:53:03.698525+0000","last_fullsized":"2023-10-24T23:53:03.698525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165369+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-24T23:53:04.165369+0000","last_peered":"2023-10-24T23:53:04.165369+0000","last_clean":"2023-10-24T23:53:04.165369+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-24T23:53:04.165369+0000","last_undegraded":"2023-10-24T23:53:04.165369+0000","last_fullsized":"2023-10-24T23:53:04.165369+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499350+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-24T23:53:04.499350+0000","last_peered":"2023-10-24T23:53:04.499350+0000","last_clean":"2023-10-24T23:53:04.499350+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-24T23:53:04.499350+0000","last_undegraded":"2023-10-24T23:53:04.499350+0000","last_fullsized":"2023-10-24T23:53:04.499350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165731+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-24T23:53:04.165731+0000","last_peered":"2023-10-24T23:53:04.165731+0000","last_clean":"2023-10-24T23:53:04.165731+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-24T23:53:04.165731+0000","last_undegraded":"2023-10-24T23:53:04.165731+0000","last_fullsized":"2023-10-24T23:53:04.165731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497047+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-24T23:53:04.497047+0000","last_peered":"2023-10-24T23:53:04.497047+0000","last_clean":"2023-10-24T23:53:04.497047+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-24T23:53:04.497047+0000","last_undegraded":"2023-10-24T23:53:04.497047+0000","last_fullsized":"2023-10-24T23:53:04.497047+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699631+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-24T23:53:03.699631+0000","last_peered":"2023-10-24T23:53:03.699631+0000","last_clean":"2023-10-24T23:53:03.699631+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-24T23:53:03.699631+0000","last_undegraded":"2023-10-24T23:53:03.699631+0000","last_fullsized":"2023-10-24T23:53:03.699631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497056+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-24T23:53:04.497056+0000","last_peered":"2023-10-24T23:53:04.497056+0000","last_clean":"2023-10-24T23:53:04.497056+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-24T23:53:04.497056+0000","last_undegraded":"2023-10-24T23:53:04.497056+0000","last_fullsized":"2023-10-24T23:53:04.497056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497069+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-24T23:53:04.497069+0000","last_peered":"2023-10-24T23:53:04.497069+0000","last_clean":"2023-10-24T23:53:04.497069+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-24T23:53:04.497069+0000","last_undegraded":"2023-10-24T23:53:04.497069+0000","last_fullsized":"2023-10-24T23:53:04.497069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.061804+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-24T23:53:04.061804+0000","last_peered":"2023-10-24T23:53:04.061804+0000","last_clean":"2023-10-24T23:53:04.061804+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-24T23:53:04.061804+0000","last_undegraded":"2023-10-24T23:53:04.061804+0000","last_fullsized":"2023-10-24T23:53:04.061804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.165275+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-24T23:53:04.165275+0000","last_peered":"2023-10-24T23:53:04.165275+0000","last_clean":"2023-10-24T23:53:04.165275+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-24T23:53:04.165275+0000","last_undegraded":"2023-10-24T23:53:04.165275+0000","last_fullsized":"2023-10-24T23:53:04.165275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.496931+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-24T23:53:04.496931+0000","last_peered":"2023-10-24T23:53:04.496931+0000","last_clean":"2023-10-24T23:53:04.496931+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-24T23:53:04.496931+0000","last_undegraded":"2023-10-24T23:53:04.496931+0000","last_fullsized":"2023-10-24T23:53:04.496931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062078+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-24T23:53:04.062078+0000","last_peered":"2023-10-24T23:53:04.062078+0000","last_clean":"2023-10-24T23:53:04.062078+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-24T23:53:04.062078+0000","last_undegraded":"2023-10-24T23:53:04.062078+0000","last_fullsized":"2023-10-24T23:53:04.062078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499118+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-24T23:53:04.499118+0000","last_peered":"2023-10-24T23:53:04.499118+0000","last_clean":"2023-10-24T23:53:04.499118+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-24T23:53:04.499118+0000","last_undegraded":"2023-10-24T23:53:04.499118+0000","last_fullsized":"2023-10-24T23:53:04.499118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696750+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-24T23:53:03.696750+0000","last_peered":"2023-10-24T23:53:03.696750+0000","last_clean":"2023-10-24T23:53:03.696750+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-24T23:53:03.696750+0000","last_undegraded":"2023-10-24T23:53:03.696750+0000","last_fullsized":"2023-10-24T23:53:03.696750+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499911+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-24T23:53:04.499911+0000","last_peered":"2023-10-24T23:53:04.499911+0000","last_clean":"2023-10-24T23:53:04.499911+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-24T23:53:04.499911+0000","last_undegraded":"2023-10-24T23:53:04.499911+0000","last_fullsized":"2023-10-24T23:53:04.499911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497346+0000","last_change":"2023-10-24T23:53:02.728971+0000","last_active":"2023-10-24T23:53:04.497346+0000","last_peered":"2023-10-24T23:53:04.497346+0000","last_clean":"2023-10-24T23:53:04.497346+0000","last_became_active":"2023-10-24T23:53:02.728048+0000","last_became_peered":"2023-10-24T23:53:02.728048+0000","last_unstale":"2023-10-24T23:53:04.497346+0000","last_undegraded":"2023-10-24T23:53:04.497346+0000","last_fullsized":"2023-10-24T23:53:04.497346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:31:32.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498748+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-24T23:53:04.498748+0000","last_peered":"2023-10-24T23:53:04.498748+0000","last_clean":"2023-10-24T23:53:04.498748+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-24T23:53:04.498748+0000","last_undegraded":"2023-10-24T23:53:04.498748+0000","last_fullsized":"2023-10-24T23:53:04.498748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.062078+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-24T23:53:04.062078+0000","last_peered":"2023-10-24T23:53:04.062078+0000","last_clean":"2023-10-24T23:53:04.062078+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-24T23:53:04.062078+0000","last_undegraded":"2023-10-24T23:53:04.062078+0000","last_fullsized":"2023-10-24T23:53:04.062078+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699415+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-24T23:53:03.699415+0000","last_peered":"2023-10-24T23:53:03.699415+0000","last_clean":"2023-10-24T23:53:03.699415+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-24T23:53:03.699415+0000","last_undegraded":"2023-10-24T23:53:03.699415+0000","last_fullsized":"2023-10-24T23:53:03.699415+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696626+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-24T23:53:03.696626+0000","last_peered":"2023-10-24T23:53:03.696626+0000","last_clean":"2023-10-24T23:53:03.696626+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-24T23:53:03.696626+0000","last_undegraded":"2023-10-24T23:53:03.696626+0000","last_fullsized":"2023-10-24T23:53:03.696626+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696679+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-24T23:53:03.696679+0000","last_peered":"2023-10-24T23:53:03.696679+0000","last_clean":"2023-10-24T23:53:03.696679+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-24T23:53:03.696679+0000","last_undegraded":"2023-10-24T23:53:03.696679+0000","last_fullsized":"2023-10-24T23:53:03.696679+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499695+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-24T23:53:04.499695+0000","last_peered":"2023-10-24T23:53:04.499695+0000","last_clean":"2023-10-24T23:53:04.499695+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-24T23:53:04.499695+0000","last_undegraded":"2023-10-24T23:53:04.499695+0000","last_fullsized":"2023-10-24T23:53:04.499695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699835+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-24T23:53:03.699835+0000","last_peered":"2023-10-24T23:53:03.699835+0000","last_clean":"2023-10-24T23:53:03.699835+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-24T23:53:03.699835+0000","last_undegraded":"2023-10-24T23:53:03.699835+0000","last_fullsized":"2023-10-24T23:53:03.699835+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497251+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-24T23:53:04.497251+0000","last_peered":"2023-10-24T23:53:04.497251+0000","last_clean":"2023-10-24T23:53:04.497251+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-24T23:53:04.497251+0000","last_undegraded":"2023-10-24T23:53:04.497251+0000","last_fullsized":"2023-10-24T23:53:04.497251+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.698958+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-24T23:53:03.698958+0000","last_peered":"2023-10-24T23:53:03.698958+0000","last_clean":"2023-10-24T23:53:03.698958+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-24T23:53:03.698958+0000","last_undegraded":"2023-10-24T23:53:03.698958+0000","last_fullsized":"2023-10-24T23:53:03.698958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.061899+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-24T23:53:04.061899+0000","last_peered":"2023-10-24T23:53:04.061899+0000","last_clean":"2023-10-24T23:53:04.061899+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-24T23:53:04.061899+0000","last_undegraded":"2023-10-24T23:53:04.061899+0000","last_fullsized":"2023-10-24T23:53:04.061899+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.498842+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-24T23:53:04.498842+0000","last_peered":"2023-10-24T23:53:04.498842+0000","last_clean":"2023-10-24T23:53:04.498842+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-24T23:53:04.498842+0000","last_undegraded":"2023-10-24T23:53:04.498842+0000","last_fullsized":"2023-10-24T23:53:04.498842+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.497236+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-24T23:53:04.497236+0000","last_peered":"2023-10-24T23:53:04.497236+0000","last_clean":"2023-10-24T23:53:04.497236+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-24T23:53:04.497236+0000","last_undegraded":"2023-10-24T23:53:04.497236+0000","last_fullsized":"2023-10-24T23:53:04.497236+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.697967+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-24T23:53:03.697967+0000","last_peered":"2023-10-24T23:53:03.697967+0000","last_clean":"2023-10-24T23:53:03.697967+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-24T23:53:03.697967+0000","last_undegraded":"2023-10-24T23:53:03.697967+0000","last_fullsized":"2023-10-24T23:53:03.697967+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:04.499435+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-24T23:53:04.499435+0000","last_peered":"2023-10-24T23:53:04.499435+0000","last_clean":"2023-10-24T23:53:04.499435+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-24T23:53:04.499435+0000","last_undegraded":"2023-10-24T23:53:04.499435+0000","last_fullsized":"2023-10-24T23:53:04.499435+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.699801+0000","last_change":"2023-10-24T23:53:02.715966+0000","last_active":"2023-10-24T23:53:03.699801+0000","last_peered":"2023-10-24T23:53:03.699801+0000","last_clean":"2023-10-24T23:53:03.699801+0000","last_became_active":"2023-10-24T23:53:02.715631+0000","last_became_peered":"2023-10-24T23:53:02.715631+0000","last_unstale":"2023-10-24T23:53:03.699801+0000","last_undegraded":"2023-10-24T23:53:03.699801+0000","last_fullsized":"2023-10-24T23:53:03.699801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:11.771864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-24T23:53:11.386581+0000","last_change":"2023-10-24T23:53:11.386581+0000","last_active":"2023-10-24T23:53:11.386581+0000","last_peered":"2023-10-24T23:53:11.386581+0000","last_clean":"2023-10-24T23:53:11.386581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-24T23:53:11.386581+0000","last_undegraded":"2023-10-24T23:53:11.386581+0000","last_fullsized":"2023-10-24T23:53:11.386581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":10,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-24T23:53:03.696971+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-24T23:53:03.696971+0000","last_peered":"2023-10-24T23:53:03.696971+0000","last_clean":"2023-10-24T23:53:03.696971+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-24T23:53:03.696971+0000","last_undegraded":"2023-10-24T23:53:03.696971+0000","last_fullsized":"2023-10-24T23:53:03.696971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":864256,"data_stored":681667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31284,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340556,"statfs":{"total":96636764160,"available":96604729344,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0780000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.105}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31292,"kb_used_data":244,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340548,"statfs":{"total":96636764160,"available":96604721152,"internally_reserved":0,"allocated":249856,"data_stored":90171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0149999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":262144,"data_stored":91139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.073}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0329999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":240,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":245760,"data_stored":89609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":236,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":241664,"data_stored":89519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":839680,"data_stored":680469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:20.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:20.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.394+0000 7fb752118640 1 Processor -- start 2023-10-24T23:53:20.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.394+0000 7fb752118640 1 -- start start 2023-10-24T23:53:20.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb752118640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c14b600 0x7fb74c151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb752118640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74c0c1860 con 0x7fb74c06af70 2023-10-24T23:53:20.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb752118640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74c154d50 con 0x7fb74c06a230 2023-10-24T23:53:20.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb752118640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c1524f0 con 0x7fb74c14b600 2023-10-24T23:53:20.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb74c06af70 0x7fb74c06c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55246/0 (socket says 172.21.15.107:55246) 2023-10-24T23:53:20.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74affd640 1 -- 172.21.15.107:0/3151522235 learned_addr learned my addr 172.21.15.107:0/3151522235 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:20.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74b7fe640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c14b600 0x7fb74c151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648830078 0 0) 0x7fb74c0c1860 con 0x7fb74c06af70 2023-10-24T23:53:20.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb734003530 con 0x7fb74c06af70 2023-10-24T23:53:20.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3369851782 0 0) 0x7fb734003530 con 0x7fb74c06af70 2023-10-24T23:53:20.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c14b600 msgr2=0x7fb74c151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:20.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c14b600 0x7fb74c151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 >> v1:172.21.15.110:6789/0 conn(0x7fb74c06a230 legacy=0x7fb74c1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c14b9e0 con 0x7fb74c06af70 2023-10-24T23:53:20.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb740001680 con 0x7fb74c06af70 2023-10-24T23:53:20.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 >> v1:172.21.15.107:6789/0 conn(0x7fb74c06af70 legacy=0x7fb74c06c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 0x7fb734003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c14b600 0x7fb7340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb734004d10 0x7fb734005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c1524f0 con 0x7fb7340039b0 2023-10-24T23:53:20.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73400bac0 con 0x7fb734004d10 2023-10-24T23:53:20.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73400bc80 con 0x7fb74c14b600 2023-10-24T23:53:20.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74b7fe640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 0x7fb734003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74affd640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c14b600 0x7fb7340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74bfff640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb734004d10 0x7fb734005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74b7fe640 1 -- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c14b600 msgr2=0x7fb7340047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:20.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74b7fe640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c14b600 0x7fb7340047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74b7fe640 1 -- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb734004d10 msgr2=0x7fb734005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:20.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74b7fe640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb734004d10 0x7fb734005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74b7fe640 1 -- 172.21.15.107:0/3151522235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb734003530 con 0x7fb7340039b0 2023-10-24T23:53:20.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.398+0000 7fb74bfff640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb734004d10 0x7fb734005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:20.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74b7fe640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 0x7fb734003dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009fc0 tx=0x7fb73c01cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653b692129f9f69f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:20.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0a5a70 con 0x7fb7340039b0 2023-10-24T23:53:20.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c00f040 con 0x7fb7340039b0 2023-10-24T23:53:20.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74a7fc640 1 -- 172.21.15.107:0/3151522235 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c0a5e10 con 0x7fb7340039b0 2023-10-24T23:53:20.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 -- 172.21.15.107:0/3151522235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 msgr2=0x7fb734003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 --2- 172.21.15.107:0/3151522235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 0x7fb734003dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009fc0 tx=0x7fb73c01cb70 comp rx=0 tx=0).stop 2023-10-24T23:53:20.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74bfff640 1 -- 172.21.15.107:0/3151522235 reap_dead start 2023-10-24T23:53:20.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 -- 172.21.15.107:0/3151522235 shutdown_connections 2023-10-24T23:53:20.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 -- 172.21.15.107:0/3151522235 >> 172.21.15.107:0/3151522235 conn(0x7fb74c0c2950 msgr2=0x7fb74c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:20.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 -- 172.21.15.107:0/3151522235 shutdown_connections 2023-10-24T23:53:20.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 -- 172.21.15.107:0/3151522235 wait complete. 2023-10-24T23:53:20.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 Processor -- start 2023-10-24T23:53:20.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 -- start start 2023-10-24T23:53:20.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 0x7fb74c1529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c153de0 0x7fb74c0bf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c143660 0x7fb74c143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c144020 con 0x7fb7340039b0 2023-10-24T23:53:20.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c1ed470 con 0x7fb74c153de0 2023-10-24T23:53:20.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c1ecb60 con 0x7fb74c143660 2023-10-24T23:53:20.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 0x7fb74c1529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 0x7fb74c1529b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50408/0 (socket says 172.21.15.107:50408) 2023-10-24T23:53:20.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74b7fe640 1 -- 172.21.15.107:0/581486136 learned_addr learned my addr 172.21.15.107:0/581486136 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:20.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74affd640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c153de0 0x7fb74c0bf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74bfff640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c143660 0x7fb74c143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74b7fe640 1 -- 172.21.15.107:0/581486136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c143660 msgr2=0x7fb74c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74b7fe640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c143660 0x7fb74c143ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74b7fe640 1 -- 172.21.15.107:0/581486136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c153de0 msgr2=0x7fb74c0bf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74b7fe640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c153de0 0x7fb74c0bf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74b7fe640 1 -- 172.21.15.107:0/581486136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0bffc0 con 0x7fb7340039b0 2023-10-24T23:53:20.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74affd640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c153de0 0x7fb74c0bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:20.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb74b7fe640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 0x7fb74c1529b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb73c01c5e0 tx=0x7fb73c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:20.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb733fff640 1 -- 172.21.15.107:0/581486136 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c003ba0 con 0x7fb7340039b0 2023-10-24T23:53:20.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb733fff640 1 -- 172.21.15.107:0/581486136 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c003d00 con 0x7fb7340039b0 2023-10-24T23:53:20.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb733fff640 1 -- 172.21.15.107:0/581486136 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c004240 con 0x7fb7340039b0 2023-10-24T23:53:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 -- 172.21.15.107:0/581486136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74c1ece20 con 0x7fb7340039b0 2023-10-24T23:53:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.402+0000 7fb752118640 1 -- 172.21.15.107:0/581486136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74c0bfa20 con 0x7fb7340039b0 2023-10-24T23:53:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.406+0000 7fb752118640 1 -- 172.21.15.107:0/581486136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74c151360 con 0x7fb7340039b0 2023-10-24T23:53:20.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.406+0000 7fb733fff640 1 -- 172.21.15.107:0/581486136 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb73c0b4040 con 0x7fb7340039b0 2023-10-24T23:53:20.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.406+0000 7fb733fff640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb738065e90 0x7fb738068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.406+0000 7fb733fff640 1 -- 172.21.15.107:0/581486136 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fb73c11e150 con 0x7fb7340039b0 2023-10-24T23:53:20.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.406+0000 7fb74affd640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb738065e90 0x7fb738068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.406+0000 7fb74affd640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb738065e90 0x7fb738068350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb740001ec0 tx=0x7fb740000ee0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:20.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.410+0000 7fb733fff640 1 -- 172.21.15.107:0/581486136 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb73c0eb620 con 0x7fb7340039b0 2023-10-24T23:53:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.562+0000 7fb752118640 1 -- 172.21.15.107:0/581486136 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb74c1ed2d0 con 0x7fb738065e90 2023-10-24T23:53:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.586+0000 7fb733fff640 1 -- 172.21.15.107:0/581486136 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525160 (secure 0 0 0) 0x7fb740001610 con 0x7fb738065e90 2023-10-24T23:53:20.595 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:20.600 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:20.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 -- 172.21.15.107:0/581486136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb738065e90 msgr2=0x7fb738068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb738065e90 0x7fb738068350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb740001ec0 tx=0x7fb740000ee0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 -- 172.21.15.107:0/581486136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 msgr2=0x7fb74c1529b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 0x7fb74c1529b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb73c01c5e0 tx=0x7fb73c009750 comp rx=0 tx=0).stop 2023-10-24T23:53:20.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 -- 172.21.15.107:0/581486136 shutdown_connections 2023-10-24T23:53:20.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb738065e90 0x7fb738068350 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c143660 0x7fb74c143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c153de0 0x7fb74c0bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 --2- 172.21.15.107:0/581486136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7340039b0 0x7fb74c1529b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 -- 172.21.15.107:0/581486136 >> 172.21.15.107:0/581486136 conn(0x7fb74c0c2950 msgr2=0x7fb74c150c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 -- 172.21.15.107:0/581486136 shutdown_connections 2023-10-24T23:53:20.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.590+0000 7fb752118640 1 -- 172.21.15.107:0/581486136 wait complete. 2023-10-24T23:53:20.614 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-24T23:53:19.280369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595092,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":378,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235560,"kb_used_data":3560,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754739160,"statfs":{"total":773094113280,"available":772852899840,"internally_reserved":0,"allocated":3645440,"data_stored":2144952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":61,"apply_latency_ms":61,"commit_latency_ns":61000000,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.880298"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603561+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-24T23:53:13.603561+0000","last_peered":"2023-10-24T23:53:13.603561+0000","last_clean":"2023-10-24T23:53:13.603561+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-24T23:53:13.603561+0000","last_undegraded":"2023-10-24T23:53:13.603561+0000","last_fullsized":"2023-10-24T23:53:13.603561+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:51:20.137796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794961+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-24T23:53:12.794961+0000","last_peered":"2023-10-24T23:53:12.794961+0000","last_clean":"2023-10-24T23:53:12.794961+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-24T23:53:12.794961+0000","last_undegraded":"2023-10-24T23:53:12.794961+0000","last_fullsized":"2023-10-24T23:53:12.794961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:00:20.101379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788258+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-24T23:53:11.788258+0000","last_peered":"2023-10-24T23:53:11.788258+0000","last_clean":"2023-10-24T23:53:11.788258+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-24T23:53:11.788258+0000","last_undegraded":"2023-10-24T23:53:11.788258+0000","last_fullsized":"2023-10-24T23:53:11.788258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791737+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-24T23:53:11.791737+0000","last_peered":"2023-10-24T23:53:11.791737+0000","last_clean":"2023-10-24T23:53:11.791737+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-24T23:53:11.791737+0000","last_undegraded":"2023-10-24T23:53:11.791737+0000","last_fullsized":"2023-10-24T23:53:11.791737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794565+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-24T23:53:12.794565+0000","last_peered":"2023-10-24T23:53:12.794565+0000","last_clean":"2023-10-24T23:53:12.794565+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-24T23:53:12.794565+0000","last_undegraded":"2023-10-24T23:53:12.794565+0000","last_fullsized":"2023-10-24T23:53:12.794565+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:59.368156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.601719+0000","last_change":"2023-10-24T23:53:11.444089+0000","last_active":"2023-10-24T23:53:13.601719+0000","last_peered":"2023-10-24T23:53:13.601719+0000","last_clean":"2023-10-24T23:53:13.601719+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-24T23:53:13.601719+0000","last_undegraded":"2023-10-24T23:53:13.601719+0000","last_fullsized":"2023-10-24T23:53:13.601719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:54:06.875882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807520+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-24T23:53:11.807520+0000","last_peered":"2023-10-24T23:53:11.807520+0000","last_clean":"2023-10-24T23:53:11.807520+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-24T23:53:11.807520+0000","last_undegraded":"2023-10-24T23:53:11.807520+0000","last_fullsized":"2023-10-24T23:53:11.807520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806881+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-24T23:53:11.806881+0000","last_peered":"2023-10-24T23:53:11.806881+0000","last_clean":"2023-10-24T23:53:11.806881+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-24T23:53:11.806881+0000","last_undegraded":"2023-10-24T23:53:11.806881+0000","last_fullsized":"2023-10-24T23:53:11.806881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602008+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-24T23:53:13.602008+0000","last_peered":"2023-10-24T23:53:13.602008+0000","last_clean":"2023-10-24T23:53:13.602008+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-24T23:53:13.602008+0000","last_undegraded":"2023-10-24T23:53:13.602008+0000","last_fullsized":"2023-10-24T23:53:13.602008+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:38:56.643025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101201+0000","last_change":"2023-10-24T23:53:11.438681+0000","last_active":"2023-10-24T23:53:13.101201+0000","last_peered":"2023-10-24T23:53:13.101201+0000","last_clean":"2023-10-24T23:53:13.101201+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-24T23:53:13.101201+0000","last_undegraded":"2023-10-24T23:53:13.101201+0000","last_fullsized":"2023-10-24T23:53:13.101201+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:50.139697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832718+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-24T23:53:11.832718+0000","last_peered":"2023-10-24T23:53:11.832718+0000","last_clean":"2023-10-24T23:53:11.832718+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-24T23:53:11.832718+0000","last_undegraded":"2023-10-24T23:53:11.832718+0000","last_fullsized":"2023-10-24T23:53:11.832718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.810798+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-24T23:53:11.810798+0000","last_peered":"2023-10-24T23:53:11.810798+0000","last_clean":"2023-10-24T23:53:11.810798+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-24T23:53:11.810798+0000","last_undegraded":"2023-10-24T23:53:11.810798+0000","last_fullsized":"2023-10-24T23:53:11.810798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101349+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-24T23:53:13.101349+0000","last_peered":"2023-10-24T23:53:13.101349+0000","last_clean":"2023-10-24T23:53:13.101349+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-24T23:53:13.101349+0000","last_undegraded":"2023-10-24T23:53:13.101349+0000","last_fullsized":"2023-10-24T23:53:13.101349+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:22.974333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796171+0000","last_change":"2023-10-24T23:53:11.454537+0000","last_active":"2023-10-24T23:53:12.796171+0000","last_peered":"2023-10-24T23:53:12.796171+0000","last_clean":"2023-10-24T23:53:12.796171+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-24T23:53:12.796171+0000","last_undegraded":"2023-10-24T23:53:12.796171+0000","last_fullsized":"2023-10-24T23:53:12.796171+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:14:06.792651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.830575+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-24T23:53:11.830575+0000","last_peered":"2023-10-24T23:53:11.830575+0000","last_clean":"2023-10-24T23:53:11.830575+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-24T23:53:11.830575+0000","last_undegraded":"2023-10-24T23:53:11.830575+0000","last_fullsized":"2023-10-24T23:53:11.830575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800231+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-24T23:53:11.800231+0000","last_peered":"2023-10-24T23:53:11.800231+0000","last_clean":"2023-10-24T23:53:11.800231+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-24T23:53:11.800231+0000","last_undegraded":"2023-10-24T23:53:11.800231+0000","last_fullsized":"2023-10-24T23:53:11.800231+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605269+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-24T23:53:13.605269+0000","last_peered":"2023-10-24T23:53:13.605269+0000","last_clean":"2023-10-24T23:53:13.605269+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-24T23:53:13.605269+0000","last_undegraded":"2023-10-24T23:53:13.605269+0000","last_fullsized":"2023-10-24T23:53:13.605269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:35:39.879563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795373+0000","last_change":"2023-10-24T23:53:11.444138+0000","last_active":"2023-10-24T23:53:12.795373+0000","last_peered":"2023-10-24T23:53:12.795373+0000","last_clean":"2023-10-24T23:53:12.795373+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-24T23:53:12.795373+0000","last_undegraded":"2023-10-24T23:53:12.795373+0000","last_fullsized":"2023-10-24T23:53:12.795373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:27:38.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788658+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-24T23:53:11.788658+0000","last_peered":"2023-10-24T23:53:11.788658+0000","last_clean":"2023-10-24T23:53:11.788658+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-24T23:53:11.788658+0000","last_undegraded":"2023-10-24T23:53:11.788658+0000","last_fullsized":"2023-10-24T23:53:11.788658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.797645+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-24T23:53:11.797645+0000","last_peered":"2023-10-24T23:53:11.797645+0000","last_clean":"2023-10-24T23:53:11.797645+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-24T23:53:11.797645+0000","last_undegraded":"2023-10-24T23:53:11.797645+0000","last_fullsized":"2023-10-24T23:53:11.797645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795697+0000","last_change":"2023-10-24T23:53:11.832809+0000","last_active":"2023-10-24T23:53:12.795697+0000","last_peered":"2023-10-24T23:53:12.795697+0000","last_clean":"2023-10-24T23:53:12.795697+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-24T23:53:12.795697+0000","last_undegraded":"2023-10-24T23:53:12.795697+0000","last_fullsized":"2023-10-24T23:53:12.795697+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:01.151315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604963+0000","last_change":"2023-10-24T23:53:11.442035+0000","last_active":"2023-10-24T23:53:13.604963+0000","last_peered":"2023-10-24T23:53:13.604963+0000","last_clean":"2023-10-24T23:53:13.604963+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-24T23:53:13.604963+0000","last_undegraded":"2023-10-24T23:53:13.604963+0000","last_fullsized":"2023-10-24T23:53:13.604963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:13.717844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.811329+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-24T23:53:11.811329+0000","last_peered":"2023-10-24T23:53:11.811329+0000","last_clean":"2023-10-24T23:53:11.811329+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-24T23:53:11.811329+0000","last_undegraded":"2023-10-24T23:53:11.811329+0000","last_fullsized":"2023-10-24T23:53:11.811329+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807632+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-24T23:53:11.807632+0000","last_peered":"2023-10-24T23:53:11.807632+0000","last_clean":"2023-10-24T23:53:11.807632+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-24T23:53:11.807632+0000","last_undegraded":"2023-10-24T23:53:11.807632+0000","last_fullsized":"2023-10-24T23:53:11.807632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797540+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-24T23:53:12.797540+0000","last_peered":"2023-10-24T23:53:12.797540+0000","last_clean":"2023-10-24T23:53:12.797540+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-24T23:53:12.797540+0000","last_undegraded":"2023-10-24T23:53:12.797540+0000","last_fullsized":"2023-10-24T23:53:12.797540+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:26.012711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604200+0000","last_change":"2023-10-24T23:53:11.448032+0000","last_active":"2023-10-24T23:53:13.604200+0000","last_peered":"2023-10-24T23:53:13.604200+0000","last_clean":"2023-10-24T23:53:13.604200+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-24T23:53:13.604200+0000","last_undegraded":"2023-10-24T23:53:13.604200+0000","last_fullsized":"2023-10-24T23:53:13.604200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:50:32.490810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791720+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-24T23:53:11.791720+0000","last_peered":"2023-10-24T23:53:11.791720+0000","last_clean":"2023-10-24T23:53:11.791720+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-24T23:53:11.791720+0000","last_undegraded":"2023-10-24T23:53:11.791720+0000","last_fullsized":"2023-10-24T23:53:11.791720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788475+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-24T23:53:11.788475+0000","last_peered":"2023-10-24T23:53:11.788475+0000","last_clean":"2023-10-24T23:53:11.788475+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-24T23:53:11.788475+0000","last_undegraded":"2023-10-24T23:53:11.788475+0000","last_fullsized":"2023-10-24T23:53:11.788475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795316+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-24T23:53:12.795316+0000","last_peered":"2023-10-24T23:53:12.795316+0000","last_clean":"2023-10-24T23:53:12.795316+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-24T23:53:12.795316+0000","last_undegraded":"2023-10-24T23:53:12.795316+0000","last_fullsized":"2023-10-24T23:53:12.795316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:10.820967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603323+0000","last_change":"2023-10-24T23:53:11.446925+0000","last_active":"2023-10-24T23:53:13.603323+0000","last_peered":"2023-10-24T23:53:13.603323+0000","last_clean":"2023-10-24T23:53:13.603323+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-24T23:53:13.603323+0000","last_undegraded":"2023-10-24T23:53:13.603323+0000","last_fullsized":"2023-10-24T23:53:13.603323+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:59:52.485469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804040+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-24T23:53:11.804040+0000","last_peered":"2023-10-24T23:53:11.804040+0000","last_clean":"2023-10-24T23:53:11.804040+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-24T23:53:11.804040+0000","last_undegraded":"2023-10-24T23:53:11.804040+0000","last_fullsized":"2023-10-24T23:53:11.804040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791850+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-24T23:53:11.791850+0000","last_peered":"2023-10-24T23:53:11.791850+0000","last_clean":"2023-10-24T23:53:11.791850+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-24T23:53:11.791850+0000","last_undegraded":"2023-10-24T23:53:11.791850+0000","last_fullsized":"2023-10-24T23:53:11.791850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795971+0000","last_change":"2023-10-24T23:53:11.849056+0000","last_active":"2023-10-24T23:53:12.795971+0000","last_peered":"2023-10-24T23:53:12.795971+0000","last_clean":"2023-10-24T23:53:12.795971+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-24T23:53:12.795971+0000","last_undegraded":"2023-10-24T23:53:12.795971+0000","last_fullsized":"2023-10-24T23:53:12.795971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:08.932942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803866+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-24T23:53:11.803866+0000","last_peered":"2023-10-24T23:53:11.803866+0000","last_clean":"2023-10-24T23:53:11.803866+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-24T23:53:11.803866+0000","last_undegraded":"2023-10-24T23:53:11.803866+0000","last_fullsized":"2023-10-24T23:53:11.803866+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788258+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-24T23:53:11.788258+0000","last_peered":"2023-10-24T23:53:11.788258+0000","last_clean":"2023-10-24T23:53:11.788258+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-24T23:53:11.788258+0000","last_undegraded":"2023-10-24T23:53:11.788258+0000","last_fullsized":"2023-10-24T23:53:11.788258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602180+0000","last_change":"2023-10-24T23:53:11.447167+0000","last_active":"2023-10-24T23:53:13.602180+0000","last_peered":"2023-10-24T23:53:13.602180+0000","last_clean":"2023-10-24T23:53:13.602180+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-24T23:53:13.602180+0000","last_undegraded":"2023-10-24T23:53:13.602180+0000","last_fullsized":"2023-10-24T23:53:13.602180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:06:25.006728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794920+0000","last_change":"2023-10-24T23:53:11.813704+0000","last_active":"2023-10-24T23:53:12.794920+0000","last_peered":"2023-10-24T23:53:12.794920+0000","last_clean":"2023-10-24T23:53:12.794920+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-24T23:53:12.794920+0000","last_undegraded":"2023-10-24T23:53:12.794920+0000","last_fullsized":"2023-10-24T23:53:12.794920+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:39:09.996039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804467+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-24T23:53:11.804467+0000","last_peered":"2023-10-24T23:53:11.804467+0000","last_clean":"2023-10-24T23:53:11.804467+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-24T23:53:11.804467+0000","last_undegraded":"2023-10-24T23:53:11.804467+0000","last_fullsized":"2023-10-24T23:53:11.804467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838755+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-24T23:53:11.838755+0000","last_peered":"2023-10-24T23:53:11.838755+0000","last_clean":"2023-10-24T23:53:11.838755+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-24T23:53:11.838755+0000","last_undegraded":"2023-10-24T23:53:11.838755+0000","last_fullsized":"2023-10-24T23:53:11.838755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795718+0000","last_change":"2023-10-24T23:53:11.450974+0000","last_active":"2023-10-24T23:53:12.795718+0000","last_peered":"2023-10-24T23:53:12.795718+0000","last_clean":"2023-10-24T23:53:12.795718+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-24T23:53:12.795718+0000","last_undegraded":"2023-10-24T23:53:12.795718+0000","last_fullsized":"2023-10-24T23:53:12.795718+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:51:05.443664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602086+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-24T23:53:13.602086+0000","last_peered":"2023-10-24T23:53:13.602086+0000","last_clean":"2023-10-24T23:53:13.602086+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-24T23:53:13.602086+0000","last_undegraded":"2023-10-24T23:53:13.602086+0000","last_fullsized":"2023-10-24T23:53:13.602086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:37:30.278659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825877+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-24T23:53:11.825877+0000","last_peered":"2023-10-24T23:53:11.825877+0000","last_clean":"2023-10-24T23:53:11.825877+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-24T23:53:11.825877+0000","last_undegraded":"2023-10-24T23:53:11.825877+0000","last_fullsized":"2023-10-24T23:53:11.825877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791237+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-24T23:53:11.791237+0000","last_peered":"2023-10-24T23:53:11.791237+0000","last_clean":"2023-10-24T23:53:11.791237+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-24T23:53:11.791237+0000","last_undegraded":"2023-10-24T23:53:11.791237+0000","last_fullsized":"2023-10-24T23:53:11.791237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794303+0000","last_change":"2023-10-24T23:53:11.429694+0000","last_active":"2023-10-24T23:53:12.794303+0000","last_peered":"2023-10-24T23:53:12.794303+0000","last_clean":"2023-10-24T23:53:12.794303+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-24T23:53:12.794303+0000","last_undegraded":"2023-10-24T23:53:12.794303+0000","last_fullsized":"2023-10-24T23:53:12.794303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:15.964994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793903+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-24T23:53:12.793903+0000","last_peered":"2023-10-24T23:53:12.793903+0000","last_clean":"2023-10-24T23:53:12.793903+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-24T23:53:12.793903+0000","last_undegraded":"2023-10-24T23:53:12.793903+0000","last_fullsized":"2023-10-24T23:53:12.793903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:38:18.678087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803275+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-24T23:53:11.803275+0000","last_peered":"2023-10-24T23:53:11.803275+0000","last_clean":"2023-10-24T23:53:11.803275+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-24T23:53:11.803275+0000","last_undegraded":"2023-10-24T23:53:11.803275+0000","last_fullsized":"2023-10-24T23:53:11.803275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800342+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-24T23:53:11.800342+0000","last_peered":"2023-10-24T23:53:11.800342+0000","last_clean":"2023-10-24T23:53:11.800342+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-24T23:53:11.800342+0000","last_undegraded":"2023-10-24T23:53:11.800342+0000","last_fullsized":"2023-10-24T23:53:11.800342+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793830+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-24T23:53:12.793830+0000","last_peered":"2023-10-24T23:53:12.793830+0000","last_clean":"2023-10-24T23:53:12.793830+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-24T23:53:12.793830+0000","last_undegraded":"2023-10-24T23:53:12.793830+0000","last_fullsized":"2023-10-24T23:53:12.793830+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:30.163913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797383+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-24T23:53:12.797383+0000","last_peered":"2023-10-24T23:53:12.797383+0000","last_clean":"2023-10-24T23:53:12.797383+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-24T23:53:12.797383+0000","last_undegraded":"2023-10-24T23:53:12.797383+0000","last_fullsized":"2023-10-24T23:53:12.797383+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:46.000330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.802095+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-24T23:53:11.802095+0000","last_peered":"2023-10-24T23:53:11.802095+0000","last_clean":"2023-10-24T23:53:11.802095+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-24T23:53:11.802095+0000","last_undegraded":"2023-10-24T23:53:11.802095+0000","last_fullsized":"2023-10-24T23:53:11.802095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.792050+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:11.792050+0000","last_peered":"2023-10-24T23:53:11.792050+0000","last_clean":"2023-10-24T23:53:11.792050+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:11.792050+0000","last_undegraded":"2023-10-24T23:53:11.792050+0000","last_fullsized":"2023-10-24T23:53:11.792050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788646+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-24T23:53:11.788646+0000","last_peered":"2023-10-24T23:53:11.788646+0000","last_clean":"2023-10-24T23:53:11.788646+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-24T23:53:11.788646+0000","last_undegraded":"2023-10-24T23:53:11.788646+0000","last_fullsized":"2023-10-24T23:53:11.788646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793988+0000","last_change":"2023-10-24T23:53:11.433306+0000","last_active":"2023-10-24T23:53:12.793988+0000","last_peered":"2023-10-24T23:53:12.793988+0000","last_clean":"2023-10-24T23:53:12.793988+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-24T23:53:12.793988+0000","last_undegraded":"2023-10-24T23:53:12.793988+0000","last_fullsized":"2023-10-24T23:53:12.793988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:44:38.728345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604210+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-24T23:53:13.604210+0000","last_peered":"2023-10-24T23:53:13.604210+0000","last_clean":"2023-10-24T23:53:13.604210+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-24T23:53:13.604210+0000","last_undegraded":"2023-10-24T23:53:13.604210+0000","last_fullsized":"2023-10-24T23:53:13.604210+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:30:24.805861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803946+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-24T23:53:11.803946+0000","last_peered":"2023-10-24T23:53:11.803946+0000","last_clean":"2023-10-24T23:53:11.803946+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-24T23:53:11.803946+0000","last_undegraded":"2023-10-24T23:53:11.803946+0000","last_fullsized":"2023-10-24T23:53:11.803946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790715+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:11.790715+0000","last_peered":"2023-10-24T23:53:11.790715+0000","last_clean":"2023-10-24T23:53:11.790715+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:11.790715+0000","last_undegraded":"2023-10-24T23:53:11.790715+0000","last_fullsized":"2023-10-24T23:53:11.790715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807502+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-24T23:53:11.807502+0000","last_peered":"2023-10-24T23:53:11.807502+0000","last_clean":"2023-10-24T23:53:11.807502+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-24T23:53:11.807502+0000","last_undegraded":"2023-10-24T23:53:11.807502+0000","last_fullsized":"2023-10-24T23:53:11.807502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602440+0000","last_change":"2023-10-24T23:53:11.441983+0000","last_active":"2023-10-24T23:53:13.602440+0000","last_peered":"2023-10-24T23:53:13.602440+0000","last_clean":"2023-10-24T23:53:13.602440+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-24T23:53:13.602440+0000","last_undegraded":"2023-10-24T23:53:13.602440+0000","last_fullsized":"2023-10-24T23:53:13.602440+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:30.643019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794864+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-24T23:53:12.794864+0000","last_peered":"2023-10-24T23:53:12.794864+0000","last_clean":"2023-10-24T23:53:12.794864+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-24T23:53:12.794864+0000","last_undegraded":"2023-10-24T23:53:12.794864+0000","last_fullsized":"2023-10-24T23:53:12.794864+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:50:53.467058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788379+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-24T23:53:11.788379+0000","last_peered":"2023-10-24T23:53:11.788379+0000","last_clean":"2023-10-24T23:53:11.788379+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-24T23:53:11.788379+0000","last_undegraded":"2023-10-24T23:53:11.788379+0000","last_fullsized":"2023-10-24T23:53:11.788379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.826147+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:11.826147+0000","last_peered":"2023-10-24T23:53:11.826147+0000","last_clean":"2023-10-24T23:53:11.826147+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:11.826147+0000","last_undegraded":"2023-10-24T23:53:11.826147+0000","last_fullsized":"2023-10-24T23:53:11.826147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790306+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-24T23:53:11.790306+0000","last_peered":"2023-10-24T23:53:11.790306+0000","last_clean":"2023-10-24T23:53:11.790306+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-24T23:53:11.790306+0000","last_undegraded":"2023-10-24T23:53:11.790306+0000","last_fullsized":"2023-10-24T23:53:11.790306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602338+0000","last_change":"2023-10-24T23:53:11.426287+0000","last_active":"2023-10-24T23:53:13.602338+0000","last_peered":"2023-10-24T23:53:13.602338+0000","last_clean":"2023-10-24T23:53:13.602338+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-24T23:53:13.602338+0000","last_undegraded":"2023-10-24T23:53:13.602338+0000","last_fullsized":"2023-10-24T23:53:13.602338+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:02:39.399866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602429+0000","last_change":"2023-10-24T23:53:11.811139+0000","last_active":"2023-10-24T23:53:13.602429+0000","last_peered":"2023-10-24T23:53:13.602429+0000","last_clean":"2023-10-24T23:53:13.602429+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-24T23:53:13.602429+0000","last_undegraded":"2023-10-24T23:53:13.602429+0000","last_fullsized":"2023-10-24T23:53:13.602429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:05.507633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791317+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-24T23:53:11.791317+0000","last_peered":"2023-10-24T23:53:11.791317+0000","last_clean":"2023-10-24T23:53:11.791317+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-24T23:53:11.791317+0000","last_undegraded":"2023-10-24T23:53:11.791317+0000","last_fullsized":"2023-10-24T23:53:11.791317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790574+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:11.790574+0000","last_peered":"2023-10-24T23:53:11.790574+0000","last_clean":"2023-10-24T23:53:11.790574+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:11.790574+0000","last_undegraded":"2023-10-24T23:53:11.790574+0000","last_fullsized":"2023-10-24T23:53:11.790574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832916+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-24T23:53:11.832916+0000","last_peered":"2023-10-24T23:53:11.832916+0000","last_clean":"2023-10-24T23:53:11.832916+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-24T23:53:11.832916+0000","last_undegraded":"2023-10-24T23:53:11.832916+0000","last_fullsized":"2023-10-24T23:53:11.832916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602790+0000","last_change":"2023-10-24T23:53:11.426199+0000","last_active":"2023-10-24T23:53:13.602790+0000","last_peered":"2023-10-24T23:53:13.602790+0000","last_clean":"2023-10-24T23:53:13.602790+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-24T23:53:13.602790+0000","last_undegraded":"2023-10-24T23:53:13.602790+0000","last_fullsized":"2023-10-24T23:53:13.602790+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:57:51.018816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794361+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-24T23:53:12.794361+0000","last_peered":"2023-10-24T23:53:12.794361+0000","last_clean":"2023-10-24T23:53:12.794361+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-24T23:53:12.794361+0000","last_undegraded":"2023-10-24T23:53:12.794361+0000","last_fullsized":"2023-10-24T23:53:12.794361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:20:16.596993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790230+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-24T23:53:11.790230+0000","last_peered":"2023-10-24T23:53:11.790230+0000","last_clean":"2023-10-24T23:53:11.790230+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-24T23:53:11.790230+0000","last_undegraded":"2023-10-24T23:53:11.790230+0000","last_fullsized":"2023-10-24T23:53:11.790230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800496+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-24T23:53:11.800496+0000","last_peered":"2023-10-24T23:53:11.800496+0000","last_clean":"2023-10-24T23:53:11.800496+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-24T23:53:11.800496+0000","last_undegraded":"2023-10-24T23:53:11.800496+0000","last_fullsized":"2023-10-24T23:53:11.800496+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828620+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:53:11.828620+0000","last_peered":"2023-10-24T23:53:11.828620+0000","last_clean":"2023-10-24T23:53:11.828620+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:53:11.828620+0000","last_undegraded":"2023-10-24T23:53:11.828620+0000","last_fullsized":"2023-10-24T23:53:11.828620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603076+0000","last_change":"2023-10-24T23:53:11.432666+0000","last_active":"2023-10-24T23:53:13.603076+0000","last_peered":"2023-10-24T23:53:13.603076+0000","last_clean":"2023-10-24T23:53:13.603076+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-24T23:53:13.603076+0000","last_undegraded":"2023-10-24T23:53:13.603076+0000","last_fullsized":"2023-10-24T23:53:13.603076+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:17.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797676+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-24T23:53:12.797676+0000","last_peered":"2023-10-24T23:53:12.797676+0000","last_clean":"2023-10-24T23:53:12.797676+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-24T23:53:12.797676+0000","last_undegraded":"2023-10-24T23:53:12.797676+0000","last_fullsized":"2023-10-24T23:53:12.797676+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:42.131555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789311+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-24T23:53:11.789311+0000","last_peered":"2023-10-24T23:53:11.789311+0000","last_clean":"2023-10-24T23:53:11.789311+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-24T23:53:11.789311+0000","last_undegraded":"2023-10-24T23:53:11.789311+0000","last_fullsized":"2023-10-24T23:53:11.789311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789825+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-24T23:53:11.789825+0000","last_peered":"2023-10-24T23:53:11.789825+0000","last_clean":"2023-10-24T23:53:11.789825+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-24T23:53:11.789825+0000","last_undegraded":"2023-10-24T23:53:11.789825+0000","last_fullsized":"2023-10-24T23:53:11.789825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.795048+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:53:11.795048+0000","last_peered":"2023-10-24T23:53:11.795048+0000","last_clean":"2023-10-24T23:53:11.795048+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:53:11.795048+0000","last_undegraded":"2023-10-24T23:53:11.795048+0000","last_fullsized":"2023-10-24T23:53:11.795048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"27'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.279161+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-24T23:53:13.279161+0000","last_peered":"2023-10-24T23:53:13.279161+0000","last_clean":"2023-10-24T23:53:13.279161+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-24T23:53:13.279161+0000","last_undegraded":"2023-10-24T23:53:13.279161+0000","last_fullsized":"2023-10-24T23:53:13.279161+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:55:47.548432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795507+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-24T23:53:12.795507+0000","last_peered":"2023-10-24T23:53:12.795507+0000","last_clean":"2023-10-24T23:53:12.795507+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-24T23:53:12.795507+0000","last_undegraded":"2023-10-24T23:53:12.795507+0000","last_fullsized":"2023-10-24T23:53:12.795507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:11:14.431984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788576+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-24T23:53:11.788576+0000","last_peered":"2023-10-24T23:53:11.788576+0000","last_clean":"2023-10-24T23:53:11.788576+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-24T23:53:11.788576+0000","last_undegraded":"2023-10-24T23:53:11.788576+0000","last_fullsized":"2023-10-24T23:53:11.788576+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789662+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-24T23:53:11.789662+0000","last_peered":"2023-10-24T23:53:11.789662+0000","last_clean":"2023-10-24T23:53:11.789662+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-24T23:53:11.789662+0000","last_undegraded":"2023-10-24T23:53:11.789662+0000","last_fullsized":"2023-10-24T23:53:11.789662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828080+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:11.828080+0000","last_peered":"2023-10-24T23:53:11.828080+0000","last_clean":"2023-10-24T23:53:11.828080+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:11.828080+0000","last_undegraded":"2023-10-24T23:53:11.828080+0000","last_fullsized":"2023-10-24T23:53:11.828080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102659+0000","last_change":"2023-10-24T23:53:11.436603+0000","last_active":"2023-10-24T23:53:13.102659+0000","last_peered":"2023-10-24T23:53:13.102659+0000","last_clean":"2023-10-24T23:53:13.102659+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-24T23:53:13.102659+0000","last_undegraded":"2023-10-24T23:53:13.102659+0000","last_fullsized":"2023-10-24T23:53:13.102659+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:09:05.948928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102438+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-24T23:53:13.102438+0000","last_peered":"2023-10-24T23:53:13.102438+0000","last_clean":"2023-10-24T23:53:13.102438+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-24T23:53:13.102438+0000","last_undegraded":"2023-10-24T23:53:13.102438+0000","last_fullsized":"2023-10-24T23:53:13.102438+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:01:59.639080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793650+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-24T23:53:11.793650+0000","last_peered":"2023-10-24T23:53:11.793650+0000","last_clean":"2023-10-24T23:53:11.793650+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-24T23:53:11.793650+0000","last_undegraded":"2023-10-24T23:53:11.793650+0000","last_fullsized":"2023-10-24T23:53:11.793650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807120+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-24T23:53:11.807120+0000","last_peered":"2023-10-24T23:53:11.807120+0000","last_clean":"2023-10-24T23:53:11.807120+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-24T23:53:11.807120+0000","last_undegraded":"2023-10-24T23:53:11.807120+0000","last_fullsized":"2023-10-24T23:53:11.807120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825313+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:53:11.825313+0000","last_peered":"2023-10-24T23:53:11.825313+0000","last_clean":"2023-10-24T23:53:11.825313+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:53:11.825313+0000","last_undegraded":"2023-10-24T23:53:11.825313+0000","last_fullsized":"2023-10-24T23:53:11.825313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.834350+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:53:11.834350+0000","last_peered":"2023-10-24T23:53:11.834350+0000","last_clean":"2023-10-24T23:53:11.834350+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:53:11.834350+0000","last_undegraded":"2023-10-24T23:53:11.834350+0000","last_fullsized":"2023-10-24T23:53:11.834350+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797622+0000","last_change":"2023-10-24T23:53:11.451073+0000","last_active":"2023-10-24T23:53:12.797622+0000","last_peered":"2023-10-24T23:53:12.797622+0000","last_clean":"2023-10-24T23:53:12.797622+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-24T23:53:12.797622+0000","last_undegraded":"2023-10-24T23:53:12.797622+0000","last_fullsized":"2023-10-24T23:53:12.797622+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:32.324710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101638+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-24T23:53:13.101638+0000","last_peered":"2023-10-24T23:53:13.101638+0000","last_clean":"2023-10-24T23:53:13.101638+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-24T23:53:13.101638+0000","last_undegraded":"2023-10-24T23:53:13.101638+0000","last_fullsized":"2023-10-24T23:53:13.101638+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:04.125835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789210+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-24T23:53:11.789210+0000","last_peered":"2023-10-24T23:53:11.789210+0000","last_clean":"2023-10-24T23:53:11.789210+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-24T23:53:11.789210+0000","last_undegraded":"2023-10-24T23:53:11.789210+0000","last_fullsized":"2023-10-24T23:53:11.789210+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804119+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-24T23:53:11.804119+0000","last_peered":"2023-10-24T23:53:11.804119+0000","last_clean":"2023-10-24T23:53:11.804119+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-24T23:53:11.804119+0000","last_undegraded":"2023-10-24T23:53:11.804119+0000","last_fullsized":"2023-10-24T23:53:11.804119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605414+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-24T23:53:13.605414+0000","last_peered":"2023-10-24T23:53:13.605414+0000","last_clean":"2023-10-24T23:53:13.605414+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-24T23:53:13.605414+0000","last_undegraded":"2023-10-24T23:53:13.605414+0000","last_fullsized":"2023-10-24T23:53:13.605414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:06:53.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794699+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-24T23:53:12.794699+0000","last_peered":"2023-10-24T23:53:12.794699+0000","last_clean":"2023-10-24T23:53:12.794699+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-24T23:53:12.794699+0000","last_undegraded":"2023-10-24T23:53:12.794699+0000","last_fullsized":"2023-10-24T23:53:12.794699+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:41.905772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804055+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-24T23:53:11.804055+0000","last_peered":"2023-10-24T23:53:11.804055+0000","last_clean":"2023-10-24T23:53:11.804055+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-24T23:53:11.804055+0000","last_undegraded":"2023-10-24T23:53:11.804055+0000","last_fullsized":"2023-10-24T23:53:11.804055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.802309+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-24T23:53:11.802309+0000","last_peered":"2023-10-24T23:53:11.802309+0000","last_clean":"2023-10-24T23:53:11.802309+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-24T23:53:11.802309+0000","last_undegraded":"2023-10-24T23:53:11.802309+0000","last_fullsized":"2023-10-24T23:53:11.802309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102244+0000","last_change":"2023-10-24T23:53:11.458463+0000","last_active":"2023-10-24T23:53:13.102244+0000","last_peered":"2023-10-24T23:53:13.102244+0000","last_clean":"2023-10-24T23:53:13.102244+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-24T23:53:13.102244+0000","last_undegraded":"2023-10-24T23:53:13.102244+0000","last_fullsized":"2023-10-24T23:53:13.102244+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:11:45.040156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275727+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-24T23:53:13.275727+0000","last_peered":"2023-10-24T23:53:13.275727+0000","last_clean":"2023-10-24T23:53:13.275727+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-24T23:53:13.275727+0000","last_undegraded":"2023-10-24T23:53:13.275727+0000","last_fullsized":"2023-10-24T23:53:13.275727+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:51.707178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800564+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-24T23:53:11.800564+0000","last_peered":"2023-10-24T23:53:11.800564+0000","last_clean":"2023-10-24T23:53:11.800564+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-24T23:53:11.800564+0000","last_undegraded":"2023-10-24T23:53:11.800564+0000","last_fullsized":"2023-10-24T23:53:11.800564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790875+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-24T23:53:11.790875+0000","last_peered":"2023-10-24T23:53:11.790875+0000","last_clean":"2023-10-24T23:53:11.790875+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-24T23:53:11.790875+0000","last_undegraded":"2023-10-24T23:53:11.790875+0000","last_fullsized":"2023-10-24T23:53:11.790875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795610+0000","last_change":"2023-10-24T23:53:11.448675+0000","last_active":"2023-10-24T23:53:12.795610+0000","last_peered":"2023-10-24T23:53:12.795610+0000","last_clean":"2023-10-24T23:53:12.795610+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-24T23:53:12.795610+0000","last_undegraded":"2023-10-24T23:53:12.795610+0000","last_fullsized":"2023-10-24T23:53:12.795610+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:01.502343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605168+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-24T23:53:13.605168+0000","last_peered":"2023-10-24T23:53:13.605168+0000","last_clean":"2023-10-24T23:53:13.605168+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-24T23:53:13.605168+0000","last_undegraded":"2023-10-24T23:53:13.605168+0000","last_fullsized":"2023-10-24T23:53:13.605168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:54.478897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793341+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-24T23:53:11.793341+0000","last_peered":"2023-10-24T23:53:11.793341+0000","last_clean":"2023-10-24T23:53:11.793341+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-24T23:53:11.793341+0000","last_undegraded":"2023-10-24T23:53:11.793341+0000","last_fullsized":"2023-10-24T23:53:11.793341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807904+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-24T23:53:11.807904+0000","last_peered":"2023-10-24T23:53:11.807904+0000","last_clean":"2023-10-24T23:53:11.807904+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-24T23:53:11.807904+0000","last_undegraded":"2023-10-24T23:53:11.807904+0000","last_fullsized":"2023-10-24T23:53:11.807904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275531+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-24T23:53:13.275531+0000","last_peered":"2023-10-24T23:53:13.275531+0000","last_clean":"2023-10-24T23:53:13.275531+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-24T23:53:13.275531+0000","last_undegraded":"2023-10-24T23:53:13.275531+0000","last_fullsized":"2023-10-24T23:53:13.275531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:58.545886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602449+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-24T23:53:13.602449+0000","last_peered":"2023-10-24T23:53:13.602449+0000","last_clean":"2023-10-24T23:53:13.602449+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-24T23:53:13.602449+0000","last_undegraded":"2023-10-24T23:53:13.602449+0000","last_fullsized":"2023-10-24T23:53:13.602449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:05.595887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788253+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-24T23:53:11.788253+0000","last_peered":"2023-10-24T23:53:11.788253+0000","last_clean":"2023-10-24T23:53:11.788253+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-24T23:53:11.788253+0000","last_undegraded":"2023-10-24T23:53:11.788253+0000","last_fullsized":"2023-10-24T23:53:11.788253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.830849+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-24T23:53:11.830849+0000","last_peered":"2023-10-24T23:53:11.830849+0000","last_clean":"2023-10-24T23:53:11.830849+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-24T23:53:11.830849+0000","last_undegraded":"2023-10-24T23:53:11.830849+0000","last_fullsized":"2023-10-24T23:53:11.830849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275166+0000","last_change":"2023-10-24T23:53:11.454658+0000","last_active":"2023-10-24T23:53:13.275166+0000","last_peered":"2023-10-24T23:53:13.275166+0000","last_clean":"2023-10-24T23:53:13.275166+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-24T23:53:13.275166+0000","last_undegraded":"2023-10-24T23:53:13.275166+0000","last_fullsized":"2023-10-24T23:53:13.275166+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:43:43.869069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796376+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-24T23:53:12.796376+0000","last_peered":"2023-10-24T23:53:12.796376+0000","last_clean":"2023-10-24T23:53:12.796376+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-24T23:53:12.796376+0000","last_undegraded":"2023-10-24T23:53:12.796376+0000","last_fullsized":"2023-10-24T23:53:12.796376+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:09:40.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790517+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-24T23:53:11.790517+0000","last_peered":"2023-10-24T23:53:11.790517+0000","last_clean":"2023-10-24T23:53:11.790517+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-24T23:53:11.790517+0000","last_undegraded":"2023-10-24T23:53:11.790517+0000","last_fullsized":"2023-10-24T23:53:11.790517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825937+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-24T23:53:11.825937+0000","last_peered":"2023-10-24T23:53:11.825937+0000","last_clean":"2023-10-24T23:53:11.825937+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-24T23:53:11.825937+0000","last_undegraded":"2023-10-24T23:53:11.825937+0000","last_fullsized":"2023-10-24T23:53:11.825937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275899+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-24T23:53:13.275899+0000","last_peered":"2023-10-24T23:53:13.275899+0000","last_clean":"2023-10-24T23:53:13.275899+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-24T23:53:13.275899+0000","last_undegraded":"2023-10-24T23:53:13.275899+0000","last_fullsized":"2023-10-24T23:53:13.275899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:02.591627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794746+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-24T23:53:12.794746+0000","last_peered":"2023-10-24T23:53:12.794746+0000","last_clean":"2023-10-24T23:53:12.794746+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-24T23:53:12.794746+0000","last_undegraded":"2023-10-24T23:53:12.794746+0000","last_fullsized":"2023-10-24T23:53:12.794746+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:43.015050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.801731+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-24T23:53:11.801731+0000","last_peered":"2023-10-24T23:53:11.801731+0000","last_clean":"2023-10-24T23:53:11.801731+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-24T23:53:11.801731+0000","last_undegraded":"2023-10-24T23:53:11.801731+0000","last_fullsized":"2023-10-24T23:53:11.801731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825206+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-24T23:53:11.825206+0000","last_peered":"2023-10-24T23:53:11.825206+0000","last_clean":"2023-10-24T23:53:11.825206+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-24T23:53:11.825206+0000","last_undegraded":"2023-10-24T23:53:11.825206+0000","last_fullsized":"2023-10-24T23:53:11.825206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.274975+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-24T23:53:13.274975+0000","last_peered":"2023-10-24T23:53:13.274975+0000","last_clean":"2023-10-24T23:53:13.274975+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-24T23:53:13.274975+0000","last_undegraded":"2023-10-24T23:53:13.274975+0000","last_fullsized":"2023-10-24T23:53:13.274975+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:59.068359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.100994+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-24T23:53:13.100994+0000","last_peered":"2023-10-24T23:53:13.100994+0000","last_clean":"2023-10-24T23:53:13.100994+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-24T23:53:13.100994+0000","last_undegraded":"2023-10-24T23:53:13.100994+0000","last_fullsized":"2023-10-24T23:53:13.100994+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:13:03.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806468+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-24T23:53:11.806468+0000","last_peered":"2023-10-24T23:53:11.806468+0000","last_clean":"2023-10-24T23:53:11.806468+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-24T23:53:11.806468+0000","last_undegraded":"2023-10-24T23:53:11.806468+0000","last_fullsized":"2023-10-24T23:53:11.806468+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.827865+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-24T23:53:11.827865+0000","last_peered":"2023-10-24T23:53:11.827865+0000","last_clean":"2023-10-24T23:53:11.827865+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-24T23:53:11.827865+0000","last_undegraded":"2023-10-24T23:53:11.827865+0000","last_fullsized":"2023-10-24T23:53:11.827865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796200+0000","last_change":"2023-10-24T23:53:11.451013+0000","last_active":"2023-10-24T23:53:12.796200+0000","last_peered":"2023-10-24T23:53:12.796200+0000","last_clean":"2023-10-24T23:53:12.796200+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-24T23:53:12.796200+0000","last_undegraded":"2023-10-24T23:53:12.796200+0000","last_fullsized":"2023-10-24T23:53:12.796200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:52:22.503023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797682+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-24T23:53:12.797682+0000","last_peered":"2023-10-24T23:53:12.797682+0000","last_clean":"2023-10-24T23:53:12.797682+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-24T23:53:12.797682+0000","last_undegraded":"2023-10-24T23:53:12.797682+0000","last_fullsized":"2023-10-24T23:53:12.797682+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:05:30.149693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102137+0000","last_change":"2023-10-24T23:53:11.434538+0000","last_active":"2023-10-24T23:53:13.102137+0000","last_peered":"2023-10-24T23:53:13.102137+0000","last_clean":"2023-10-24T23:53:13.102137+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-24T23:53:13.102137+0000","last_undegraded":"2023-10-24T23:53:13.102137+0000","last_fullsized":"2023-10-24T23:53:13.102137+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:55.179253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789405+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-24T23:53:11.789405+0000","last_peered":"2023-10-24T23:53:11.789405+0000","last_clean":"2023-10-24T23:53:11.789405+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-24T23:53:11.789405+0000","last_undegraded":"2023-10-24T23:53:11.789405+0000","last_fullsized":"2023-10-24T23:53:11.789405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807405+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-24T23:53:11.807405+0000","last_peered":"2023-10-24T23:53:11.807405+0000","last_clean":"2023-10-24T23:53:11.807405+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-24T23:53:11.807405+0000","last_undegraded":"2023-10-24T23:53:11.807405+0000","last_fullsized":"2023-10-24T23:53:11.807405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275539+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-24T23:53:13.275539+0000","last_peered":"2023-10-24T23:53:13.275539+0000","last_clean":"2023-10-24T23:53:13.275539+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-24T23:53:13.275539+0000","last_undegraded":"2023-10-24T23:53:13.275539+0000","last_fullsized":"2023-10-24T23:53:13.275539+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:33:08.373963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795605+0000","last_change":"2023-10-24T23:53:11.433387+0000","last_active":"2023-10-24T23:53:12.795605+0000","last_peered":"2023-10-24T23:53:12.795605+0000","last_clean":"2023-10-24T23:53:12.795605+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-24T23:53:12.795605+0000","last_undegraded":"2023-10-24T23:53:12.795605+0000","last_fullsized":"2023-10-24T23:53:12.795605+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:29.148450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.824306+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-24T23:53:11.824306+0000","last_peered":"2023-10-24T23:53:11.824306+0000","last_clean":"2023-10-24T23:53:11.824306+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-24T23:53:11.824306+0000","last_undegraded":"2023-10-24T23:53:11.824306+0000","last_fullsized":"2023-10-24T23:53:11.824306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788623+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-24T23:53:11.788623+0000","last_peered":"2023-10-24T23:53:11.788623+0000","last_clean":"2023-10-24T23:53:11.788623+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-24T23:53:11.788623+0000","last_undegraded":"2023-10-24T23:53:11.788623+0000","last_fullsized":"2023-10-24T23:53:11.788623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605588+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-24T23:53:13.605588+0000","last_peered":"2023-10-24T23:53:13.605588+0000","last_clean":"2023-10-24T23:53:13.605588+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-24T23:53:13.605588+0000","last_undegraded":"2023-10-24T23:53:13.605588+0000","last_fullsized":"2023-10-24T23:53:13.605588+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:33:06.172412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788704+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-24T23:53:11.788704+0000","last_peered":"2023-10-24T23:53:11.788704+0000","last_clean":"2023-10-24T23:53:11.788704+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-24T23:53:11.788704+0000","last_undegraded":"2023-10-24T23:53:11.788704+0000","last_fullsized":"2023-10-24T23:53:11.788704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832501+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-24T23:53:11.832501+0000","last_peered":"2023-10-24T23:53:11.832501+0000","last_clean":"2023-10-24T23:53:11.832501+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-24T23:53:11.832501+0000","last_undegraded":"2023-10-24T23:53:11.832501+0000","last_fullsized":"2023-10-24T23:53:11.832501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"26'6","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794898+0000","last_change":"2023-10-24T23:53:11.446979+0000","last_active":"2023-10-24T23:53:12.794898+0000","last_peered":"2023-10-24T23:53:12.794898+0000","last_clean":"2023-10-24T23:53:12.794898+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-24T23:53:12.794898+0000","last_undegraded":"2023-10-24T23:53:12.794898+0000","last_fullsized":"2023-10-24T23:53:12.794898+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:54:52.843631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603260+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-24T23:53:13.603260+0000","last_peered":"2023-10-24T23:53:13.603260+0000","last_clean":"2023-10-24T23:53:13.603260+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-24T23:53:13.603260+0000","last_undegraded":"2023-10-24T23:53:13.603260+0000","last_fullsized":"2023-10-24T23:53:13.603260+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:18:48.679682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.816486+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-24T23:53:11.816486+0000","last_peered":"2023-10-24T23:53:11.816486+0000","last_clean":"2023-10-24T23:53:11.816486+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-24T23:53:11.816486+0000","last_undegraded":"2023-10-24T23:53:11.816486+0000","last_fullsized":"2023-10-24T23:53:11.816486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807790+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-24T23:53:11.807790+0000","last_peered":"2023-10-24T23:53:11.807790+0000","last_clean":"2023-10-24T23:53:11.807790+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-24T23:53:11.807790+0000","last_undegraded":"2023-10-24T23:53:11.807790+0000","last_fullsized":"2023-10-24T23:53:11.807790+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101724+0000","last_change":"2023-10-24T23:53:11.436964+0000","last_active":"2023-10-24T23:53:13.101724+0000","last_peered":"2023-10-24T23:53:13.101724+0000","last_clean":"2023-10-24T23:53:13.101724+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-24T23:53:13.101724+0000","last_undegraded":"2023-10-24T23:53:13.101724+0000","last_fullsized":"2023-10-24T23:53:13.101724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:04:15.366966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602222+0000","last_change":"2023-10-24T23:53:11.814304+0000","last_active":"2023-10-24T23:53:13.602222+0000","last_peered":"2023-10-24T23:53:13.602222+0000","last_clean":"2023-10-24T23:53:13.602222+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-24T23:53:13.602222+0000","last_undegraded":"2023-10-24T23:53:13.602222+0000","last_fullsized":"2023-10-24T23:53:13.602222+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:16.635178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789162+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-24T23:53:11.789162+0000","last_peered":"2023-10-24T23:53:11.789162+0000","last_clean":"2023-10-24T23:53:11.789162+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-24T23:53:11.789162+0000","last_undegraded":"2023-10-24T23:53:11.789162+0000","last_fullsized":"2023-10-24T23:53:11.789162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603105+0000","last_change":"2023-10-24T23:53:11.807096+0000","last_active":"2023-10-24T23:53:13.603105+0000","last_peered":"2023-10-24T23:53:13.603105+0000","last_clean":"2023-10-24T23:53:13.603105+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-24T23:53:13.603105+0000","last_undegraded":"2023-10-24T23:53:13.603105+0000","last_fullsized":"2023-10-24T23:53:13.603105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:25.144578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804445+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-24T23:53:11.804445+0000","last_peered":"2023-10-24T23:53:11.804445+0000","last_clean":"2023-10-24T23:53:11.804445+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-24T23:53:11.804445+0000","last_undegraded":"2023-10-24T23:53:11.804445+0000","last_fullsized":"2023-10-24T23:53:11.804445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793680+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-24T23:53:12.793680+0000","last_peered":"2023-10-24T23:53:12.793680+0000","last_clean":"2023-10-24T23:53:12.793680+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-24T23:53:12.793680+0000","last_undegraded":"2023-10-24T23:53:12.793680+0000","last_fullsized":"2023-10-24T23:53:12.793680+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:38.946585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.811071+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-24T23:53:11.811071+0000","last_peered":"2023-10-24T23:53:11.811071+0000","last_clean":"2023-10-24T23:53:11.811071+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-24T23:53:11.811071+0000","last_undegraded":"2023-10-24T23:53:11.811071+0000","last_fullsized":"2023-10-24T23:53:11.811071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797036+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-24T23:53:12.797036+0000","last_peered":"2023-10-24T23:53:12.797036+0000","last_clean":"2023-10-24T23:53:12.797036+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-24T23:53:12.797036+0000","last_undegraded":"2023-10-24T23:53:12.797036+0000","last_fullsized":"2023-10-24T23:53:12.797036+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:48:24.911279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807673+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-24T23:53:11.807673+0000","last_peered":"2023-10-24T23:53:11.807673+0000","last_clean":"2023-10-24T23:53:11.807673+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-24T23:53:11.807673+0000","last_undegraded":"2023-10-24T23:53:11.807673+0000","last_fullsized":"2023-10-24T23:53:11.807673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795515+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-24T23:53:12.795515+0000","last_peered":"2023-10-24T23:53:12.795515+0000","last_clean":"2023-10-24T23:53:12.795515+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-24T23:53:12.795515+0000","last_undegraded":"2023-10-24T23:53:12.795515+0000","last_fullsized":"2023-10-24T23:53:12.795515+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:21.165755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790400+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-24T23:53:11.790400+0000","last_peered":"2023-10-24T23:53:11.790400+0000","last_clean":"2023-10-24T23:53:11.790400+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-24T23:53:11.790400+0000","last_undegraded":"2023-10-24T23:53:11.790400+0000","last_fullsized":"2023-10-24T23:53:11.790400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604073+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-24T23:53:13.604073+0000","last_peered":"2023-10-24T23:53:13.604073+0000","last_clean":"2023-10-24T23:53:13.604073+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-24T23:53:13.604073+0000","last_undegraded":"2023-10-24T23:53:13.604073+0000","last_fullsized":"2023-10-24T23:53:13.604073+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832305+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-24T23:53:11.832305+0000","last_peered":"2023-10-24T23:53:11.832305+0000","last_clean":"2023-10-24T23:53:11.832305+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-24T23:53:11.832305+0000","last_undegraded":"2023-10-24T23:53:11.832305+0000","last_fullsized":"2023-10-24T23:53:11.832305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101339+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-24T23:53:13.101339+0000","last_peered":"2023-10-24T23:53:13.101339+0000","last_clean":"2023-10-24T23:53:13.101339+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-24T23:53:13.101339+0000","last_undegraded":"2023-10-24T23:53:13.101339+0000","last_fullsized":"2023-10-24T23:53:13.101339+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:54.422365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790575+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-24T23:53:11.790575+0000","last_peered":"2023-10-24T23:53:11.790575+0000","last_clean":"2023-10-24T23:53:11.790575+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-24T23:53:11.790575+0000","last_undegraded":"2023-10-24T23:53:11.790575+0000","last_fullsized":"2023-10-24T23:53:11.790575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603531+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-24T23:53:13.603531+0000","last_peered":"2023-10-24T23:53:13.603531+0000","last_clean":"2023-10-24T23:53:13.603531+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-24T23:53:13.603531+0000","last_undegraded":"2023-10-24T23:53:13.603531+0000","last_fullsized":"2023-10-24T23:53:13.603531+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:01:11.838035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789166+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-24T23:53:11.789166+0000","last_peered":"2023-10-24T23:53:11.789166+0000","last_clean":"2023-10-24T23:53:11.789166+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-24T23:53:11.789166+0000","last_undegraded":"2023-10-24T23:53:11.789166+0000","last_fullsized":"2023-10-24T23:53:11.789166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603142+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-24T23:53:13.603142+0000","last_peered":"2023-10-24T23:53:13.603142+0000","last_clean":"2023-10-24T23:53:13.603142+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-24T23:53:13.603142+0000","last_undegraded":"2023-10-24T23:53:13.603142+0000","last_fullsized":"2023-10-24T23:53:13.603142+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:09:25.424344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.827005+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-24T23:53:11.827005+0000","last_peered":"2023-10-24T23:53:11.827005+0000","last_clean":"2023-10-24T23:53:11.827005+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-24T23:53:11.827005+0000","last_undegraded":"2023-10-24T23:53:11.827005+0000","last_fullsized":"2023-10-24T23:53:11.827005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101152+0000","last_change":"2023-10-24T23:53:11.824274+0000","last_active":"2023-10-24T23:53:13.101152+0000","last_peered":"2023-10-24T23:53:13.101152+0000","last_clean":"2023-10-24T23:53:13.101152+0000","last_became_active":"2023-10-24T23:53:11.823586+0000","last_became_peered":"2023-10-24T23:53:11.823586+0000","last_unstale":"2023-10-24T23:53:13.101152+0000","last_undegraded":"2023-10-24T23:53:13.101152+0000","last_fullsized":"2023-10-24T23:53:13.101152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:47:36.970336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804379+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-24T23:53:11.804379+0000","last_peered":"2023-10-24T23:53:11.804379+0000","last_clean":"2023-10-24T23:53:11.804379+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-24T23:53:11.804379+0000","last_undegraded":"2023-10-24T23:53:11.804379+0000","last_fullsized":"2023-10-24T23:53:11.804379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.274967+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-24T23:53:13.274967+0000","last_peered":"2023-10-24T23:53:13.274967+0000","last_clean":"2023-10-24T23:53:13.274967+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-24T23:53:13.274967+0000","last_undegraded":"2023-10-24T23:53:13.274967+0000","last_fullsized":"2023-10-24T23:53:13.274967+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:10:20.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788652+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-24T23:53:11.788652+0000","last_peered":"2023-10-24T23:53:11.788652+0000","last_clean":"2023-10-24T23:53:11.788652+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-24T23:53:11.788652+0000","last_undegraded":"2023-10-24T23:53:11.788652+0000","last_fullsized":"2023-10-24T23:53:11.788652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796443+0000","last_change":"2023-10-24T23:53:11.860133+0000","last_active":"2023-10-24T23:53:12.796443+0000","last_peered":"2023-10-24T23:53:12.796443+0000","last_clean":"2023-10-24T23:53:12.796443+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-24T23:53:12.796443+0000","last_undegraded":"2023-10-24T23:53:12.796443+0000","last_fullsized":"2023-10-24T23:53:12.796443+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:43.051362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.816202+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-24T23:53:11.816202+0000","last_peered":"2023-10-24T23:53:11.816202+0000","last_clean":"2023-10-24T23:53:11.816202+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-24T23:53:11.816202+0000","last_undegraded":"2023-10-24T23:53:11.816202+0000","last_fullsized":"2023-10-24T23:53:11.816202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603745+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-24T23:53:13.603745+0000","last_peered":"2023-10-24T23:53:13.603745+0000","last_clean":"2023-10-24T23:53:13.603745+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-24T23:53:13.603745+0000","last_undegraded":"2023-10-24T23:53:13.603745+0000","last_fullsized":"2023-10-24T23:53:13.603745+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:39:14.947372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793137+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-24T23:53:11.793137+0000","last_peered":"2023-10-24T23:53:11.793137+0000","last_clean":"2023-10-24T23:53:11.793137+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-24T23:53:11.793137+0000","last_undegraded":"2023-10-24T23:53:11.793137+0000","last_fullsized":"2023-10-24T23:53:11.793137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.798110+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-24T23:53:12.798110+0000","last_peered":"2023-10-24T23:53:12.798110+0000","last_clean":"2023-10-24T23:53:12.798110+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-24T23:53:12.798110+0000","last_undegraded":"2023-10-24T23:53:12.798110+0000","last_fullsized":"2023-10-24T23:53:12.798110+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:37.413259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.796910+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-24T23:53:11.796910+0000","last_peered":"2023-10-24T23:53:11.796910+0000","last_clean":"2023-10-24T23:53:11.796910+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-24T23:53:11.796910+0000","last_undegraded":"2023-10-24T23:53:11.796910+0000","last_fullsized":"2023-10-24T23:53:11.796910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603512+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-24T23:53:13.603512+0000","last_peered":"2023-10-24T23:53:13.603512+0000","last_clean":"2023-10-24T23:53:13.603512+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-24T23:53:13.603512+0000","last_undegraded":"2023-10-24T23:53:13.603512+0000","last_fullsized":"2023-10-24T23:53:13.603512+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:31.344724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803849+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-24T23:53:11.803849+0000","last_peered":"2023-10-24T23:53:11.803849+0000","last_clean":"2023-10-24T23:53:11.803849+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-24T23:53:11.803849+0000","last_undegraded":"2023-10-24T23:53:11.803849+0000","last_fullsized":"2023-10-24T23:53:11.803849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275469+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-24T23:53:13.275469+0000","last_peered":"2023-10-24T23:53:13.275469+0000","last_clean":"2023-10-24T23:53:13.275469+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-24T23:53:13.275469+0000","last_undegraded":"2023-10-24T23:53:13.275469+0000","last_fullsized":"2023-10-24T23:53:13.275469+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788160+0000","last_change":"2023-10-24T23:53:02.728971+0000","last_active":"2023-10-24T23:53:11.788160+0000","last_peered":"2023-10-24T23:53:11.788160+0000","last_clean":"2023-10-24T23:53:11.788160+0000","last_became_active":"2023-10-24T23:53:02.728048+0000","last_became_peered":"2023-10-24T23:53:02.728048+0000","last_unstale":"2023-10-24T23:53:11.788160+0000","last_undegraded":"2023-10-24T23:53:11.788160+0000","last_fullsized":"2023-10-24T23:53:11.788160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:31:32.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.601984+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-24T23:53:13.601984+0000","last_peered":"2023-10-24T23:53:13.601984+0000","last_clean":"2023-10-24T23:53:13.601984+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-24T23:53:13.601984+0000","last_undegraded":"2023-10-24T23:53:13.601984+0000","last_fullsized":"2023-10-24T23:53:13.601984+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:32:05.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791536+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-24T23:53:11.791536+0000","last_peered":"2023-10-24T23:53:11.791536+0000","last_clean":"2023-10-24T23:53:11.791536+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-24T23:53:11.791536+0000","last_undegraded":"2023-10-24T23:53:11.791536+0000","last_fullsized":"2023-10-24T23:53:11.791536+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101460+0000","last_change":"2023-10-24T23:53:11.847814+0000","last_active":"2023-10-24T23:53:13.101460+0000","last_peered":"2023-10-24T23:53:13.101460+0000","last_clean":"2023-10-24T23:53:13.101460+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-24T23:53:13.101460+0000","last_undegraded":"2023-10-24T23:53:13.101460+0000","last_fullsized":"2023-10-24T23:53:13.101460+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:59.075858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825376+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-24T23:53:11.825376+0000","last_peered":"2023-10-24T23:53:11.825376+0000","last_clean":"2023-10-24T23:53:11.825376+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-24T23:53:11.825376+0000","last_undegraded":"2023-10-24T23:53:11.825376+0000","last_fullsized":"2023-10-24T23:53:11.825376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796122+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-24T23:53:12.796122+0000","last_peered":"2023-10-24T23:53:12.796122+0000","last_clean":"2023-10-24T23:53:12.796122+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-24T23:53:12.796122+0000","last_undegraded":"2023-10-24T23:53:12.796122+0000","last_fullsized":"2023-10-24T23:53:12.796122+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:25:12.525273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828503+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-24T23:53:11.828503+0000","last_peered":"2023-10-24T23:53:11.828503+0000","last_clean":"2023-10-24T23:53:11.828503+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-24T23:53:11.828503+0000","last_undegraded":"2023-10-24T23:53:11.828503+0000","last_fullsized":"2023-10-24T23:53:11.828503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101011+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-24T23:53:13.101011+0000","last_peered":"2023-10-24T23:53:13.101011+0000","last_clean":"2023-10-24T23:53:13.101011+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-24T23:53:13.101011+0000","last_undegraded":"2023-10-24T23:53:13.101011+0000","last_fullsized":"2023-10-24T23:53:13.101011+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:12:21.026996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.795075+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-24T23:53:11.795075+0000","last_peered":"2023-10-24T23:53:11.795075+0000","last_clean":"2023-10-24T23:53:11.795075+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-24T23:53:11.795075+0000","last_undegraded":"2023-10-24T23:53:11.795075+0000","last_fullsized":"2023-10-24T23:53:11.795075+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101949+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-24T23:53:13.101949+0000","last_peered":"2023-10-24T23:53:13.101949+0000","last_clean":"2023-10-24T23:53:13.101949+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-24T23:53:13.101949+0000","last_undegraded":"2023-10-24T23:53:13.101949+0000","last_fullsized":"2023-10-24T23:53:13.101949+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:07.909021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.797593+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-24T23:53:11.797593+0000","last_peered":"2023-10-24T23:53:11.797593+0000","last_clean":"2023-10-24T23:53:11.797593+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-24T23:53:11.797593+0000","last_undegraded":"2023-10-24T23:53:11.797593+0000","last_fullsized":"2023-10-24T23:53:11.797593+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603906+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-24T23:53:13.603906+0000","last_peered":"2023-10-24T23:53:13.603906+0000","last_clean":"2023-10-24T23:53:13.603906+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-24T23:53:13.603906+0000","last_undegraded":"2023-10-24T23:53:13.603906+0000","last_fullsized":"2023-10-24T23:53:13.603906+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:48.217889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800508+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-24T23:53:11.800508+0000","last_peered":"2023-10-24T23:53:11.800508+0000","last_clean":"2023-10-24T23:53:11.800508+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-24T23:53:11.800508+0000","last_undegraded":"2023-10-24T23:53:11.800508+0000","last_fullsized":"2023-10-24T23:53:11.800508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602960+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-24T23:53:13.602960+0000","last_peered":"2023-10-24T23:53:13.602960+0000","last_clean":"2023-10-24T23:53:13.602960+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-24T23:53:13.602960+0000","last_undegraded":"2023-10-24T23:53:13.602960+0000","last_fullsized":"2023-10-24T23:53:13.602960+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:10:18.973606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838707+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-24T23:53:11.838707+0000","last_peered":"2023-10-24T23:53:11.838707+0000","last_clean":"2023-10-24T23:53:11.838707+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-24T23:53:11.838707+0000","last_undegraded":"2023-10-24T23:53:11.838707+0000","last_fullsized":"2023-10-24T23:53:11.838707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101457+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-24T23:53:13.101457+0000","last_peered":"2023-10-24T23:53:13.101457+0000","last_clean":"2023-10-24T23:53:13.101457+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-24T23:53:13.101457+0000","last_undegraded":"2023-10-24T23:53:13.101457+0000","last_fullsized":"2023-10-24T23:53:13.101457+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:24:45.536141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788141+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-24T23:53:11.788141+0000","last_peered":"2023-10-24T23:53:11.788141+0000","last_clean":"2023-10-24T23:53:11.788141+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-24T23:53:11.788141+0000","last_undegraded":"2023-10-24T23:53:11.788141+0000","last_fullsized":"2023-10-24T23:53:11.788141+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793724+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-24T23:53:12.793724+0000","last_peered":"2023-10-24T23:53:12.793724+0000","last_clean":"2023-10-24T23:53:12.793724+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-24T23:53:12.793724+0000","last_undegraded":"2023-10-24T23:53:12.793724+0000","last_fullsized":"2023-10-24T23:53:12.793724+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:06.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806250+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-24T23:53:11.806250+0000","last_peered":"2023-10-24T23:53:11.806250+0000","last_clean":"2023-10-24T23:53:11.806250+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-24T23:53:11.806250+0000","last_undegraded":"2023-10-24T23:53:11.806250+0000","last_fullsized":"2023-10-24T23:53:11.806250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275114+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-24T23:53:13.275114+0000","last_peered":"2023-10-24T23:53:13.275114+0000","last_clean":"2023-10-24T23:53:13.275114+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-24T23:53:13.275114+0000","last_undegraded":"2023-10-24T23:53:13.275114+0000","last_fullsized":"2023-10-24T23:53:13.275114+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:56:02.798009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.823970+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-24T23:53:11.823970+0000","last_peered":"2023-10-24T23:53:11.823970+0000","last_clean":"2023-10-24T23:53:11.823970+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-24T23:53:11.823970+0000","last_undegraded":"2023-10-24T23:53:11.823970+0000","last_fullsized":"2023-10-24T23:53:11.823970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793903+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-24T23:53:12.793903+0000","last_peered":"2023-10-24T23:53:12.793903+0000","last_clean":"2023-10-24T23:53:12.793903+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-24T23:53:12.793903+0000","last_undegraded":"2023-10-24T23:53:12.793903+0000","last_fullsized":"2023-10-24T23:53:12.793903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:40.168016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.809345+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-24T23:53:11.809345+0000","last_peered":"2023-10-24T23:53:11.809345+0000","last_clean":"2023-10-24T23:53:11.809345+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-24T23:53:11.809345+0000","last_undegraded":"2023-10-24T23:53:11.809345+0000","last_fullsized":"2023-10-24T23:53:11.809345+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.100799+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-24T23:53:13.100799+0000","last_peered":"2023-10-24T23:53:13.100799+0000","last_clean":"2023-10-24T23:53:13.100799+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-24T23:53:13.100799+0000","last_undegraded":"2023-10-24T23:53:13.100799+0000","last_fullsized":"2023-10-24T23:53:13.100799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:39.720375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788501+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-24T23:53:11.788501+0000","last_peered":"2023-10-24T23:53:11.788501+0000","last_clean":"2023-10-24T23:53:11.788501+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-24T23:53:11.788501+0000","last_undegraded":"2023-10-24T23:53:11.788501+0000","last_fullsized":"2023-10-24T23:53:11.788501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794282+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-24T23:53:12.794282+0000","last_peered":"2023-10-24T23:53:12.794282+0000","last_clean":"2023-10-24T23:53:12.794282+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-24T23:53:12.794282+0000","last_undegraded":"2023-10-24T23:53:12.794282+0000","last_fullsized":"2023-10-24T23:53:12.794282+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:51:40.960630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790034+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-24T23:53:11.790034+0000","last_peered":"2023-10-24T23:53:11.790034+0000","last_clean":"2023-10-24T23:53:11.790034+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-24T23:53:11.790034+0000","last_undegraded":"2023-10-24T23:53:11.790034+0000","last_fullsized":"2023-10-24T23:53:11.790034+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795555+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-24T23:53:12.795555+0000","last_peered":"2023-10-24T23:53:12.795555+0000","last_clean":"2023-10-24T23:53:12.795555+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-24T23:53:12.795555+0000","last_undegraded":"2023-10-24T23:53:12.795555+0000","last_fullsized":"2023-10-24T23:53:12.795555+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:22:42.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793089+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-24T23:53:11.793089+0000","last_peered":"2023-10-24T23:53:11.793089+0000","last_clean":"2023-10-24T23:53:11.793089+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-24T23:53:11.793089+0000","last_undegraded":"2023-10-24T23:53:11.793089+0000","last_fullsized":"2023-10-24T23:53:11.793089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275507+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-24T23:53:13.275507+0000","last_peered":"2023-10-24T23:53:13.275507+0000","last_clean":"2023-10-24T23:53:13.275507+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-24T23:53:13.275507+0000","last_undegraded":"2023-10-24T23:53:13.275507+0000","last_fullsized":"2023-10-24T23:53:13.275507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:47.658717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838499+0000","last_change":"2023-10-24T23:53:02.715966+0000","last_active":"2023-10-24T23:53:11.838499+0000","last_peered":"2023-10-24T23:53:11.838499+0000","last_clean":"2023-10-24T23:53:11.838499+0000","last_became_active":"2023-10-24T23:53:02.715631+0000","last_became_peered":"2023-10-24T23:53:02.715631+0000","last_unstale":"2023-10-24T23:53:11.838499+0000","last_undegraded":"2023-10-24T23:53:11.838499+0000","last_fullsized":"2023-10-24T23:53:11.838499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:11.771864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603292+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-24T23:53:13.603292+0000","last_peered":"2023-10-24T23:53:13.603292+0000","last_clean":"2023-10-24T23:53:13.603292+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-24T23:53:13.603292+0000","last_undegraded":"2023-10-24T23:53:13.603292+0000","last_fullsized":"2023-10-24T23:53:13.603292+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:50.994593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.796890+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-24T23:53:11.796890+0000","last_peered":"2023-10-24T23:53:11.796890+0000","last_clean":"2023-10-24T23:53:11.796890+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-24T23:53:11.796890+0000","last_undegraded":"2023-10-24T23:53:11.796890+0000","last_fullsized":"2023-10-24T23:53:11.796890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":929792,"data_stored":713128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0820000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":294912,"data_stored":119928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0580000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":307200,"data_stored":120110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.292}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":319488,"data_stored":121570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0760000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2809999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1180000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":121034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0880000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1040000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":892928,"data_stored":710330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:20.623 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-24T23:53:20.624 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-24T23:53:20.624 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-24T23:53:20.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-24T23:53:20.625 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-24T23:53:20.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea8fbc640 1 Processor -- start 2023-10-24T23:53:20.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea8fbc640 1 -- start start 2023-10-24T23:53:20.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea8fbc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcea40a9820 0x7fcea40ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea8fbc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea406f730 con 0x7fcea414fd40 2023-10-24T23:53:20.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea8fbc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea4156a40 con 0x7fcea406ee40 2023-10-24T23:53:20.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea8fbc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea4156710 con 0x7fcea40a9820 2023-10-24T23:53:20.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fce9bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcea40a9820 0x7fcea40ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcea414fd40 0x7fcea40c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55256/0 (socket says 172.21.15.107:55256) 2023-10-24T23:53:20.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea259c640 1 -- 172.21.15.107:0/2618110368 learned_addr learned my addr 172.21.15.107:0/2618110368 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:20.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fce9bfff640 1 -- 172.21.15.107:0/2618110368 >> v1:172.21.15.110:6789/0 conn(0x7fcea406ee40 legacy=0x7fcea415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:20.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fce9bfff640 1 -- 172.21.15.107:0/2618110368 >> v1:172.21.15.107:6789/0 conn(0x7fcea414fd40 legacy=0x7fcea40c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fce9bfff640 1 -- 172.21.15.107:0/2618110368 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea406f980 con 0x7fcea40a9820 2023-10-24T23:53:20.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea1d9b640 1 -- 172.21.15.107:0/2618110368 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000429328 0 0) 0x7fcea406f730 con 0x7fcea414fd40 2023-10-24T23:53:20.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fce9bfff640 1 --2- 172.21.15.107:0/2618110368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea40a9820 0x7fcea40ab090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0097c0 tx=0x7fce8c01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c220ba145cb15d65 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:20.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea1d9b640 1 -- 172.21.15.107:0/2618110368 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c0ab600 con 0x7fcea40a9820 2023-10-24T23:53:20.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea1d9b640 1 -- 172.21.15.107:0/2618110368 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c0a7050 con 0x7fcea40a9820 2023-10-24T23:53:20.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.778+0000 7fcea1d9b640 1 -- 172.21.15.107:0/2618110368 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c0ab980 con 0x7fcea40a9820 2023-10-24T23:53:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 -- 172.21.15.107:0/2618110368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea40a9820 msgr2=0x7fcea40ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 --2- 172.21.15.107:0/2618110368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea40a9820 0x7fcea40ab090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0097c0 tx=0x7fce8c01c320 comp rx=0 tx=0).stop 2023-10-24T23:53:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 -- 172.21.15.107:0/2618110368 shutdown_connections 2023-10-24T23:53:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 --2- 172.21.15.107:0/2618110368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea40a9820 0x7fcea40ab090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 -- 172.21.15.107:0/2618110368 >> 172.21.15.107:0/2618110368 conn(0x7fcea40c2950 msgr2=0x7fcea40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:20.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 -- 172.21.15.107:0/2618110368 shutdown_connections 2023-10-24T23:53:20.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 -- 172.21.15.107:0/2618110368 wait complete. 2023-10-24T23:53:20.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 Processor -- start 2023-10-24T23:53:20.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 -- start start 2023-10-24T23:53:20.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcea406ee40 0x7fcea41ed260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcea414fd40 0x7fcea4148880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea414e850 0x7fcea40bf840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea41ed7a0 con 0x7fcea414e850 2023-10-24T23:53:20.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea406bd70 con 0x7fcea414fd40 2023-10-24T23:53:20.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40bf3b0 con 0x7fcea406ee40 2023-10-24T23:53:20.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea414e850 0x7fcea40bf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea414e850 0x7fcea40bf840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50428/0 (socket says 172.21.15.107:50428) 2023-10-24T23:53:20.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea2d9d640 1 -- 172.21.15.107:0/1906924780 learned_addr learned my addr 172.21.15.107:0/1906924780 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:20.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fce9bfff640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcea414fd40 0x7fcea4148880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea259c640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcea406ee40 0x7fcea41ed260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea2d9d640 1 -- 172.21.15.107:0/1906924780 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcea406ee40 msgr2=0x7fcea41ed260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea2d9d640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcea406ee40 0x7fcea41ed260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea2d9d640 1 -- 172.21.15.107:0/1906924780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcea414fd40 msgr2=0x7fcea4148880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea2d9d640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcea414fd40 0x7fcea4148880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea2d9d640 1 -- 172.21.15.107:0/1906924780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8c009420 con 0x7fcea414e850 2023-10-24T23:53:20.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea259c640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcea406ee40 0x7fcea41ed260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:20.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea2d9d640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea414e850 0x7fcea40bf840 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00c780 tx=0x7fce9c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:20.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fce9affd640 1 -- 172.21.15.107:0/1906924780 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c00a9f0 con 0x7fcea414e850 2023-10-24T23:53:20.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fce9affd640 1 -- 172.21.15.107:0/1906924780 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c00ce90 con 0x7fcea414e850 2023-10-24T23:53:20.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 -- 172.21.15.107:0/1906924780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea4066fb0 con 0x7fcea414e850 2023-10-24T23:53:20.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fce9affd640 1 -- 172.21.15.107:0/1906924780 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c00ad90 con 0x7fcea414e850 2023-10-24T23:53:20.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.782+0000 7fcea8fbc640 1 -- 172.21.15.107:0/1906924780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea406b7b0 con 0x7fcea414e850 2023-10-24T23:53:20.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.786+0000 7fcea8fbc640 1 -- 172.21.15.107:0/1906924780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea40c24a0 con 0x7fcea414e850 2023-10-24T23:53:20.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.786+0000 7fce9affd640 1 -- 172.21.15.107:0/1906924780 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fce9c012420 con 0x7fcea414e850 2023-10-24T23:53:20.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.786+0000 7fce9affd640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce84065e60 0x7fce84068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:20.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.786+0000 7fcea259c640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce84065e60 0x7fce84068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:20.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.786+0000 7fce9affd640 1 -- 172.21.15.107:0/1906924780 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fce9c083c60 con 0x7fcea414e850 2023-10-24T23:53:20.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.786+0000 7fcea259c640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce84065e60 0x7fce84068320 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fce94000f70 tx=0x7fce94009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:20.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.790+0000 7fce9affd640 1 -- 172.21.15.107:0/1906924780 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fce9c051010 con 0x7fcea414e850 2023-10-24T23:53:20.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.974+0000 7fcea8fbc640 1 -- 172.21.15.107:0/1906924780 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fcea40bf520 con 0x7fcea414e850 2023-10-24T23:53:20.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.982+0000 7fce9affd640 1 -- 172.21.15.107:0/1906924780 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+163890 (secure 0 0 0) 0x7fce9c016020 con 0x7fcea414e850 2023-10-24T23:53:20.986 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 -- 172.21.15.107:0/1906924780 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce84065e60 msgr2=0x7fce84068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce84065e60 0x7fce84068320 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fce94000f70 tx=0x7fce94009040 comp rx=0 tx=0).stop 2023-10-24T23:53:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 -- 172.21.15.107:0/1906924780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea414e850 msgr2=0x7fcea40bf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:20.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea414e850 0x7fcea40bf840 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00c780 tx=0x7fce9c00cc40 comp rx=0 tx=0).stop 2023-10-24T23:53:20.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 -- 172.21.15.107:0/1906924780 shutdown_connections 2023-10-24T23:53:20.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcea414e850 0x7fcea40bf840 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcea414fd40 0x7fcea4148880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce84065e60 0x7fce84068320 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 --2- 172.21.15.107:0/1906924780 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcea406ee40 0x7fcea41ed260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:20.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 -- 172.21.15.107:0/1906924780 >> 172.21.15.107:0/1906924780 conn(0x7fcea40c2950 msgr2=0x7fcea40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:20.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 -- 172.21.15.107:0/1906924780 shutdown_connections 2023-10-24T23:53:20.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:20.986+0000 7fcea8fbc640 1 -- 172.21.15.107:0/1906924780 wait complete. 2023-10-24T23:53:21.005 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":5,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":37466},{"type":"v1","addr":"172.21.15.107:6833","nonce":37466}]},"active_addr":"172.21.15.107:6833/37466","active_change":"2023-10-24T23:52:43.258660+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4107,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2334453794}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3879975022}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2967572014}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":1992740500}]}]} 2023-10-24T23:53:21.009 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-24T23:53:21.010 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-24T23:53:21.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:21.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.122+0000 7f0c5df01640 1 Processor -- start 2023-10-24T23:53:21.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.122+0000 7f0c5df01640 1 -- start start 2023-10-24T23:53:21.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c5df01640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0c5806ee40 0x7f0c580ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c5df01640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5806f730 con 0x7f0c580a9820 2023-10-24T23:53:21.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c5df01640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c58156a40 con 0x7f0c5814fd40 2023-10-24T23:53:21.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c5df01640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c58156710 con 0x7f0c5806ee40 2023-10-24T23:53:21.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c57fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0c5806ee40 0x7f0c580ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c56ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0c580a9820 0x7f0c5815af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55268/0 (socket says 172.21.15.107:55268) 2023-10-24T23:53:21.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c56ffd640 1 -- 172.21.15.107:0/2670636748 learned_addr learned my addr 172.21.15.107:0/2670636748 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:21.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c57fff640 1 -- 172.21.15.107:0/2670636748 >> v1:172.21.15.110:6789/0 conn(0x7f0c5814fd40 legacy=0x7f0c580c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c57fff640 1 -- 172.21.15.107:0/2670636748 >> v1:172.21.15.107:6789/0 conn(0x7f0c580a9820 legacy=0x7f0c5815af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c57fff640 1 -- 172.21.15.107:0/2670636748 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5806f980 con 0x7f0c5806ee40 2023-10-24T23:53:21.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c57fff640 1 --2- 172.21.15.107:0/2670636748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5806ee40 0x7f0c580ab090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009d90 tx=0x7f0c5001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387eb3112a4274cf server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c567fc640 1 -- 172.21.15.107:0/2670636748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c500a57d0 con 0x7f0c5806ee40 2023-10-24T23:53:21.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c567fc640 1 -- 172.21.15.107:0/2670636748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c500a5930 con 0x7f0c5806ee40 2023-10-24T23:53:21.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c567fc640 1 -- 172.21.15.107:0/2670636748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c500a5cc0 con 0x7f0c5806ee40 2023-10-24T23:53:21.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c5df01640 1 -- 172.21.15.107:0/2670636748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5806ee40 msgr2=0x7f0c580ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c5df01640 1 --2- 172.21.15.107:0/2670636748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5806ee40 0x7f0c580ab090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009d90 tx=0x7f0c5001c7f0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c5df01640 1 -- 172.21.15.107:0/2670636748 shutdown_connections 2023-10-24T23:53:21.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c5df01640 1 --2- 172.21.15.107:0/2670636748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5806ee40 0x7f0c580ab090 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c5df01640 1 -- 172.21.15.107:0/2670636748 >> 172.21.15.107:0/2670636748 conn(0x7f0c580c2950 msgr2=0x7f0c580c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.126+0000 7f0c5df01640 1 -- 172.21.15.107:0/2670636748 shutdown_connections 2023-10-24T23:53:21.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 -- 172.21.15.107:0/2670636748 wait complete. 2023-10-24T23:53:21.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 Processor -- start 2023-10-24T23:53:21.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 -- start start 2023-10-24T23:53:21.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5806ee40 0x7f0c580c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0c5814fd40 0x7f0c580c4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c5813e350 0x7f0c5814e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5815e440 con 0x7f0c5806ee40 2023-10-24T23:53:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c581efc60 con 0x7f0c5813e350 2023-10-24T23:53:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c580c3ee0 con 0x7f0c5814fd40 2023-10-24T23:53:21.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5806ee40 0x7f0c580c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5806ee40 0x7f0c580c4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50446/0 (socket says 172.21.15.107:50446) 2023-10-24T23:53:21.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c577fe640 1 -- 172.21.15.107:0/320401353 learned_addr learned my addr 172.21.15.107:0/320401353 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c577fe640 1 -- 172.21.15.107:0/320401353 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0c5814fd40 msgr2=0x7f0c580c4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c577fe640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0c5814fd40 0x7f0c580c4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c577fe640 1 -- 172.21.15.107:0/320401353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c5813e350 msgr2=0x7f0c5814e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c577fe640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c5813e350 0x7f0c5814e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c577fe640 1 -- 172.21.15.107:0/320401353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c50009a70 con 0x7f0c5806ee40 2023-10-24T23:53:21.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c577fe640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5806ee40 0x7f0c580c4fe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800b470 tx=0x7f0c4800b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c3ffff640 1 -- 172.21.15.107:0/320401353 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48004020 con 0x7f0c5806ee40 2023-10-24T23:53:21.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c3ffff640 1 -- 172.21.15.107:0/320401353 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c48004180 con 0x7f0c5806ee40 2023-10-24T23:53:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 -- 172.21.15.107:0/320401353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c581ef6e0 con 0x7f0c5806ee40 2023-10-24T23:53:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c3ffff640 1 -- 172.21.15.107:0/320401353 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48011690 con 0x7f0c5806ee40 2023-10-24T23:53:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 -- 172.21.15.107:0/320401353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c580c3b70 con 0x7f0c5806ee40 2023-10-24T23:53:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.130+0000 7f0c5df01640 1 -- 172.21.15.107:0/320401353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c580c24a0 con 0x7f0c5806ee40 2023-10-24T23:53:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.134+0000 7f0c3ffff640 1 -- 172.21.15.107:0/320401353 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0c480117f0 con 0x7f0c5806ee40 2023-10-24T23:53:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.134+0000 7f0c3ffff640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c44065e90 0x7f0c44068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.134+0000 7f0c56ffd640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c44065e90 0x7f0c44068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.134+0000 7f0c3ffff640 1 -- 172.21.15.107:0/320401353 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f0c48083ab0 con 0x7f0c5806ee40 2023-10-24T23:53:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.134+0000 7f0c56ffd640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c44065e90 0x7f0c44068350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0c5813eb00 tx=0x7f0c4c007a50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.138+0000 7f0c3ffff640 1 -- 172.21.15.107:0/320401353 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0c480548f0 con 0x7f0c5806ee40 2023-10-24T23:53:21.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.290+0000 7f0c5df01640 1 -- 172.21.15.107:0/320401353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c580746e0 con 0x7f0c5806ee40 2023-10-24T23:53:21.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.290+0000 7f0c3ffff640 1 -- 172.21.15.107:0/320401353 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21531 (secure 0 0 0) 0x7f0c48015090 con 0x7f0c5806ee40 2023-10-24T23:53:21.297 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:21.297 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:12.781093+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:21.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 -- 172.21.15.107:0/320401353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c44065e90 msgr2=0x7f0c44068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c44065e90 0x7f0c44068350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0c5813eb00 tx=0x7f0c4c007a50 comp rx=0 tx=0).stop 2023-10-24T23:53:21.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 -- 172.21.15.107:0/320401353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5806ee40 msgr2=0x7f0c580c4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5806ee40 0x7f0c580c4fe0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800b470 tx=0x7f0c4800b930 comp rx=0 tx=0).stop 2023-10-24T23:53:21.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 -- 172.21.15.107:0/320401353 shutdown_connections 2023-10-24T23:53:21.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c5813e350 0x7f0c5814e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c44065e90 0x7f0c44068350 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0c5814fd40 0x7f0c580c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 --2- 172.21.15.107:0/320401353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c5806ee40 0x7f0c580c4fe0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 -- 172.21.15.107:0/320401353 >> 172.21.15.107:0/320401353 conn(0x7f0c580c2950 msgr2=0x7f0c580c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 -- 172.21.15.107:0/320401353 shutdown_connections 2023-10-24T23:53:21.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.294+0000 7f0c5df01640 1 -- 172.21.15.107:0/320401353 wait complete. 2023-10-24T23:53:21.316 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-24T23:53:21.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:21.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b66a82640 1 Processor -- start 2023-10-24T23:53:21.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b66a82640 1 -- start start 2023-10-24T23:53:21.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b66a82640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006af70 0x7f0b6006c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b66a82640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b6006a230 0x7f0b601551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b66a82640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600c1860 con 0x7f0b6014b600 2023-10-24T23:53:21.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b66a82640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b60154d50 con 0x7f0b6006a230 2023-10-24T23:53:21.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b66a82640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6014b9e0 con 0x7f0b6006af70 2023-10-24T23:53:21.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b6501f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b6006a230 0x7f0b601551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b5ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006af70 0x7f0b6006c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b6501f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b6006a230 0x7f0b601551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54554/0 (socket says 172.21.15.107:54554) 2023-10-24T23:53:21.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b5ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006af70 0x7f0b6006c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:51256/0 (socket says 172.21.15.107:51256) 2023-10-24T23:53:21.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b6481e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0b6014b600 0x7f0b60151fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:39936/0 (socket says 172.21.15.107:39936) 2023-10-24T23:53:21.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b6501f640 1 -- 172.21.15.107:0/689127415 learned_addr learned my addr 172.21.15.107:0/689127415 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b5ffff640 1 -- 172.21.15.107:0/689127415 >> v2:172.21.15.110:3300/0 conn(0x7f0b6006a230 msgr2=0x7f0b601551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b5ffff640 1 --2- 172.21.15.107:0/689127415 >> v2:172.21.15.110:3300/0 conn(0x7f0b6006a230 0x7f0b601551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b5ffff640 1 -- 172.21.15.107:0/689127415 >> v1:172.21.15.110:6789/0 conn(0x7f0b6014b600 legacy=0x7f0b60151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b5ffff640 1 -- 172.21.15.107:0/689127415 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b600c1ab0 con 0x7f0b6006af70 2023-10-24T23:53:21.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.430+0000 7f0b6501f640 1 --2- 172.21.15.107:0/689127415 >> v2:172.21.15.110:3300/0 conn(0x7f0b6006a230 0x7f0b601551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:53:21.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b5ffff640 1 --2- 172.21.15.107:0/689127415 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006af70 0x7f0b6006c4c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0b540097c0 tx=0x7f0b5401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad27fd3f842971a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b5f7fe640 1 -- 172.21.15.107:0/689127415 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b540ac620 con 0x7f0b6006af70 2023-10-24T23:53:21.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b5f7fe640 1 -- 172.21.15.107:0/689127415 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b540ac780 con 0x7f0b6006af70 2023-10-24T23:53:21.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b5f7fe640 1 -- 172.21.15.107:0/689127415 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b540acb10 con 0x7f0b6006af70 2023-10-24T23:53:21.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b66a82640 1 -- 172.21.15.107:0/689127415 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006af70 msgr2=0x7f0b6006c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b66a82640 1 --2- 172.21.15.107:0/689127415 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006af70 0x7f0b6006c4c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0b540097c0 tx=0x7f0b5401c3c0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b66a82640 1 -- 172.21.15.107:0/689127415 shutdown_connections 2023-10-24T23:53:21.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b66a82640 1 --2- 172.21.15.107:0/689127415 >> v2:172.21.15.110:3300/0 conn(0x7f0b6006a230 0x7f0b601551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b66a82640 1 --2- 172.21.15.107:0/689127415 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006af70 0x7f0b6006c4c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b66a82640 1 -- 172.21.15.107:0/689127415 >> 172.21.15.107:0/689127415 conn(0x7f0b600c2950 msgr2=0x7f0b600c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b66a82640 1 -- 172.21.15.107:0/689127415 shutdown_connections 2023-10-24T23:53:21.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.434+0000 7f0b66a82640 1 -- 172.21.15.107:0/689127415 wait complete. 2023-10-24T23:53:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b66a82640 1 Processor -- start 2023-10-24T23:53:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b66a82640 1 -- start start 2023-10-24T23:53:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b66a82640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006a230 0x7f0b600bf660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b66a82640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b6006af70 0x7f0b600a96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b66a82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b6014b600 0x7f0b601f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b66a82640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6014d6e0 con 0x7f0b6014b600 2023-10-24T23:53:21.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b66a82640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6014ebd0 con 0x7f0b6006af70 2023-10-24T23:53:21.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b66a82640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6014e8f0 con 0x7f0b6006a230 2023-10-24T23:53:21.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b6501f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b6014b600 0x7f0b601f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b6501f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b6014b600 0x7f0b601f0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50462/0 (socket says 172.21.15.107:50462) 2023-10-24T23:53:21.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b6501f640 1 -- 172.21.15.107:0/1334953551 learned_addr learned my addr 172.21.15.107:0/1334953551 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b6481e640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006a230 0x7f0b600bf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b5ffff640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b6006af70 0x7f0b600a96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b6501f640 1 -- 172.21.15.107:0/1334953551 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006a230 msgr2=0x7f0b600bf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b6501f640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006a230 0x7f0b600bf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b6501f640 1 -- 172.21.15.107:0/1334953551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b6006af70 msgr2=0x7f0b600a96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b6501f640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b6006af70 0x7f0b600a96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b6501f640 1 -- 172.21.15.107:0/1334953551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b58009b30 con 0x7f0b6014b600 2023-10-24T23:53:21.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b6501f640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b6014b600 0x7f0b601f0720 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800ef90 tx=0x7f0b5800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b5d7fa640 1 -- 172.21.15.107:0/1334953551 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58018070 con 0x7f0b6014b600 2023-10-24T23:53:21.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b66a82640 1 -- 172.21.15.107:0/1334953551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b54009420 con 0x7f0b6014b600 2023-10-24T23:53:21.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b5d7fa640 1 -- 172.21.15.107:0/1334953551 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5800f040 con 0x7f0b6014b600 2023-10-24T23:53:21.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b5d7fa640 1 -- 172.21.15.107:0/1334953551 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5800ce20 con 0x7f0b6014b600 2023-10-24T23:53:21.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b66a82640 1 -- 172.21.15.107:0/1334953551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b601edfb0 con 0x7f0b6014b600 2023-10-24T23:53:21.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.438+0000 7f0b66a82640 1 -- 172.21.15.107:0/1334953551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b6014ab50 con 0x7f0b6014b600 2023-10-24T23:53:21.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.442+0000 7f0b5d7fa640 1 -- 172.21.15.107:0/1334953551 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0b580039a0 con 0x7f0b6014b600 2023-10-24T23:53:21.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.442+0000 7f0b5d7fa640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b4c065e60 0x7f0b4c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.442+0000 7f0b5d7fa640 1 -- 172.21.15.107:0/1334953551 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f0b58084760 con 0x7f0b6014b600 2023-10-24T23:53:21.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.442+0000 7f0b6481e640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b4c065e60 0x7f0b4c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.442+0000 7f0b6481e640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b4c065e60 0x7f0b4c068320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b50000f70 tx=0x7f0b50007cf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.446+0000 7f0b5d7fa640 1 -- 172.21.15.107:0/1334953551 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0b58051b10 con 0x7f0b6014b600 2023-10-24T23:53:21.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.586+0000 7f0b66a82640 1 -- 172.21.15.107:0/1334953551 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b600746e0 con 0x7f0b6014b600 2023-10-24T23:53:21.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.590+0000 7f0b5d7fa640 1 -- 172.21.15.107:0/1334953551 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21531 (secure 0 0 0) 0x7f0b58055130 con 0x7f0b6014b600 2023-10-24T23:53:21.596 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:21.597 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:12.781093+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:21.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 -- 172.21.15.107:0/1334953551 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b4c065e60 msgr2=0x7f0b4c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b4c065e60 0x7f0b4c068320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b50000f70 tx=0x7f0b50007cf0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 -- 172.21.15.107:0/1334953551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b6014b600 msgr2=0x7f0b601f0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b6014b600 0x7f0b601f0720 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0b5800ef90 tx=0x7f0b5800c490 comp rx=0 tx=0).stop 2023-10-24T23:53:21.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 -- 172.21.15.107:0/1334953551 shutdown_connections 2023-10-24T23:53:21.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b6014b600 0x7f0b601f0720 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b6006af70 0x7f0b600a96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b4c065e60 0x7f0b4c068320 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 --2- 172.21.15.107:0/1334953551 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b6006a230 0x7f0b600bf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 -- 172.21.15.107:0/1334953551 >> 172.21.15.107:0/1334953551 conn(0x7f0b600c2950 msgr2=0x7f0b60071c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 -- 172.21.15.107:0/1334953551 shutdown_connections 2023-10-24T23:53:21.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.594+0000 7f0b66a82640 1 -- 172.21.15.107:0/1334953551 wait complete. 2023-10-24T23:53:21.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-24T23:53:21.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-24T23:53:21.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-24T23:53:21.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-24T23:53:21.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-24T23:53:21.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-24T23:53:21.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-24T23:53:21.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-24T23:53:21.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.786+0000 7fc48469e640 1 Processor -- start 2023-10-24T23:53:21.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.786+0000 7fc48469e640 1 -- start start 2023-10-24T23:53:21.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.786+0000 7fc9ac498640 1 Processor -- start 2023-10-24T23:53:21.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.786+0000 7fc48469e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc47c0c1b40 0x7fc47c0a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.786+0000 7fc48469e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc47c0b9470 0x7fc47c154610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.786+0000 7fc48469e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47c06a750 con 0x7fc47c0c4f50 2023-10-24T23:53:21.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.786+0000 7fc48469e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc47c06a9a0 con 0x7fc47c0b9470 2023-10-24T23:53:21.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.786+0000 7fc48469e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc47c06ac80 con 0x7fc47c0c1b40 2023-10-24T23:53:21.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.786+0000 7f1eea163640 1 Processor -- start 2023-10-24T23:53:21.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.786+0000 7fc9ac498640 1 -- start start 2023-10-24T23:53:21.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc482c3b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc47c0b9470 0x7fc47c154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc482c3b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc47c0b9470 0x7fc47c154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50470/0 (socket says 172.21.15.107:50470) 2023-10-24T23:53:21.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc482c3b640 1 -- 172.21.15.107:0/230709187 learned_addr learned my addr 172.21.15.107:0/230709187 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc481c39640 1 --2- 172.21.15.107:0/230709187 >> v2:172.21.15.110:3300/0 conn(0x7fc47c0c1b40 0x7fc47c0a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9ac498640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc9a40c2370 0x7fc9a40bceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9ac498640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a4072660 con 0x7fc9a40c1870 2023-10-24T23:53:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9ac498640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a406a880 con 0x7fc9a406b040 2023-10-24T23:53:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9ac498640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a408bab0 con 0x7fc9a40c2370 2023-10-24T23:53:21.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9aaa35640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc9a406b040 0x7fc9a40a4a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:39952/0 (socket says 172.21.15.107:39952) 2023-10-24T23:53:21.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9aaa35640 1 -- 172.21.15.107:0/2824289561 learned_addr learned my addr 172.21.15.107:0/2824289561 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:21.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1eea163640 1 -- start start 2023-10-24T23:53:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1eea163640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1ee40c1b40 0x7f1ee406b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1eea163640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1ee40b9470 0x7f1ee40a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1eea163640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee406a750 con 0x7f1ee40c3380 2023-10-24T23:53:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1eea163640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee406bca0 con 0x7f1ee40b9470 2023-10-24T23:53:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1eea163640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ee406a9a0 con 0x7f1ee40c1b40 2023-10-24T23:53:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9aa234640 1 --2- 172.21.15.107:0/2824289561 >> v2:172.21.15.107:3300/0 conn(0x7fc9a40c2370 0x7fc9a40bceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557662577 0 0) 0x7fc9a406a880 con 0x7fc9a406b040 2023-10-24T23:53:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc990003530 con 0x7fc9a406b040 2023-10-24T23:53:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8f5444640 1 Processor -- start 2023-10-24T23:53:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1ee37fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1ee40c1b40 0x7f1ee406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1ee37fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1ee40c1b40 0x7f1ee406b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54578/0 (socket says 172.21.15.107:54578) 2023-10-24T23:53:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1ee37fe640 1 -- 172.21.15.107:0/1194317713 learned_addr learned my addr 172.21.15.107:0/1194317713 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1ee2ffd640 1 --1- 172.21.15.107:0/1194317713 >> v1:172.21.15.110:6789/0 conn(0x7f1ee40c3380 0x7f1ee40c0ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:39964/0 (socket says 172.21.15.107:39964) 2023-10-24T23:53:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8f5444640 1 -- start start 2023-10-24T23:53:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040329300 0 0) 0x7fc9a4072660 con 0x7fc9a40c1870 2023-10-24T23:53:21.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a406a880 con 0x7fc9a40c1870 2023-10-24T23:53:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1ee3fff640 1 --2- 172.21.15.107:0/1194317713 >> v2:172.21.15.107:3300/0 conn(0x7f1ee40b9470 0x7f1ee40a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1ee37fe640 1 -- 172.21.15.107:0/1194317713 >> v2:172.21.15.107:3300/0 conn(0x7f1ee40b9470 msgr2=0x7f1ee40a91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f95e3fff640 1 Processor -- start 2023-10-24T23:53:21.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f95e3fff640 1 -- start start 2023-10-24T23:53:21.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3042586613 0 0) 0x7fc990003530 con 0x7fc9a406b040 2023-10-24T23:53:21.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 >> v2:172.21.15.107:3300/0 conn(0x7fc9a40c2370 msgr2=0x7fc9a40bceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 --2- 172.21.15.107:0/2824289561 >> v2:172.21.15.107:3300/0 conn(0x7fc9a40c2370 0x7fc9a40bceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 >> v1:172.21.15.107:6789/0 conn(0x7fc9a40c1870 legacy=0x7fc9a40c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a406ab20 con 0x7fc9a406b040 2023-10-24T23:53:21.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fc9a0001680 con 0x7fc9a406b040 2023-10-24T23:53:21.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 >> v1:172.21.15.110:6789/0 conn(0x7fc9a406b040 legacy=0x7fc9a40a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 --2- 172.21.15.107:0/2824289561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc990003b10 0x7fc990003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 --2- 172.21.15.107:0/2824289561 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9900047d0 0x7fc990004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 --2- 172.21.15.107:0/2824289561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc990005190 0x7fc9900141a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a408bab0 con 0x7fc990005190 2023-10-24T23:53:21.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9900147f0 con 0x7fc990003b10 2023-10-24T23:53:21.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9900149b0 con 0x7fc9900047d0 2023-10-24T23:53:21.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9a9a33640 1 --2- 172.21.15.107:0/2824289561 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9900047d0 0x7fc990004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9aa234640 1 --2- 172.21.15.107:0/2824289561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc990003b10 0x7fc990003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9aaa35640 1 --2- 172.21.15.107:0/2824289561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc990005190 0x7fc9900141a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8f5444640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb8f00c4d70 0x7fb8f01530f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8f5444640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8f006ccd0 0x7fb8f0071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8f5444640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f00c58a0 con 0x7fb8f00a5f30 2023-10-24T23:53:21.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8f5444640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f0153630 con 0x7fb8f006ccd0 2023-10-24T23:53:21.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8f5444640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f00a6310 con 0x7fb8f00c4d70 2023-10-24T23:53:21.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1ee37fe640 1 --2- 172.21.15.107:0/1194317713 >> v2:172.21.15.107:3300/0 conn(0x7f1ee40b9470 0x7f1ee40a91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1ee37fe640 1 -- 172.21.15.107:0/1194317713 >> v1:172.21.15.110:6789/0 conn(0x7f1ee40c3380 legacy=0x7f1ee40c0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1ee37fe640 1 -- 172.21.15.107:0/1194317713 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee406ac80 con 0x7f1ee40c1b40 2023-10-24T23:53:21.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7f1ee3fff640 1 --2- 172.21.15.107:0/1194317713 >> v2:172.21.15.107:3300/0 conn(0x7f1ee40b9470 0x7f1ee40a91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:21.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8f4c43640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8f006ccd0 0x7fb8f0071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8f4c43640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8f006ccd0 0x7fb8f0071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50510/0 (socket says 172.21.15.107:50510) 2023-10-24T23:53:21.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8f4c43640 1 -- 172.21.15.107:0/859842608 learned_addr learned my addr 172.21.15.107:0/859842608 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8ef7fe640 1 --2- 172.21.15.107:0/859842608 >> v2:172.21.15.110:3300/0 conn(0x7fb8f00c4d70 0x7fb8f01530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9aa234640 1 -- 172.21.15.107:0/2824289561 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9900047d0 msgr2=0x7fc990004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9aa234640 1 --2- 172.21.15.107:0/2824289561 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9900047d0 0x7fc990004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9aa234640 1 -- 172.21.15.107:0/2824289561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc990005190 msgr2=0x7fc9900141a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9aa234640 1 --2- 172.21.15.107:0/2824289561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc990005190 0x7fc9900141a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9aa234640 1 -- 172.21.15.107:0/2824289561 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a406ab20 con 0x7fc990003b10 2023-10-24T23:53:21.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc9aaa35640 1 -- 172.21.15.107:0/2824289561 reap_dead start 2023-10-24T23:53:21.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8ef7fe640 1 -- 172.21.15.107:0/859842608 >> v2:172.21.15.107:3300/0 conn(0x7fb8f006ccd0 msgr2=0x7fb8f0071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8ef7fe640 1 --2- 172.21.15.107:0/859842608 >> v2:172.21.15.107:3300/0 conn(0x7fb8f006ccd0 0x7fb8f0071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8ef7fe640 1 -- 172.21.15.107:0/859842608 >> v1:172.21.15.107:6789/0 conn(0x7fb8f00a5f30 legacy=0x7fb8f0156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fb8ef7fe640 1 -- 172.21.15.107:0/859842608 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0152630 con 0x7fb8f00c4d70 2023-10-24T23:53:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc482c3b640 1 -- 172.21.15.107:0/230709187 >> v2:172.21.15.110:3300/0 conn(0x7fc47c0c1b40 msgr2=0x7fc47c0a91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc482c3b640 1 --2- 172.21.15.107:0/230709187 >> v2:172.21.15.110:3300/0 conn(0x7fc47c0c1b40 0x7fc47c0a91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc482c3b640 1 -- 172.21.15.107:0/230709187 >> v1:172.21.15.110:6789/0 conn(0x7fc47c0c4f50 legacy=0x7fc47c06b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.790+0000 7fc482c3b640 1 -- 172.21.15.107:0/230709187 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c06af60 con 0x7fc47c0b9470 2023-10-24T23:53:21.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f6975213640 1 Processor -- start 2023-10-24T23:53:21.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc482c3b640 1 --2- 172.21.15.107:0/230709187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc47c0b9470 0x7fc47c154610 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc47c069870 tx=0x7fc47801eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259d4f3adf04804a server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f6975213640 1 -- start start 2023-10-24T23:53:21.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc481438640 1 -- 172.21.15.107:0/230709187 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4780a7c80 con 0x7fc47c0b9470 2023-10-24T23:53:21.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc481438640 1 -- 172.21.15.107:0/230709187 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4780ab070 con 0x7fc47c0b9470 2023-10-24T23:53:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc481438640 1 -- 172.21.15.107:0/230709187 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4780b1400 con 0x7fc47c0b9470 2023-10-24T23:53:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1ee37fe640 1 --2- 172.21.15.107:0/1194317713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ee40c1b40 0x7f1ee406b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ee406e0c0 tx=0x7f1ed401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=872f1cac2f4ca3ed server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e3fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f95dc0d4200 0x7f95dc0dac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc48469e640 1 -- 172.21.15.107:0/230709187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc47c0b9470 msgr2=0x7fc47c154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc48469e640 1 --2- 172.21.15.107:0/230709187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc47c0b9470 0x7fc47c154610 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc47c069870 tx=0x7fc47801eca0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f6975213640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f69680d3a60 0x7f69680d7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f6975213640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69680da110 0x7f69680465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f95dc043240 0x7f95dc007180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95dc0076c0 0x7f95dc00e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e3fff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95dc0db280 con 0x7f95dc043240 2023-10-24T23:53:21.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e3fff640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95dc00e170 con 0x7f95dc0d4200 2023-10-24T23:53:21.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e3fff640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc04a6d0 con 0x7f95dc0076c0 2023-10-24T23:53:21.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc9aa234640 1 --2- 172.21.15.107:0/2824289561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc990003b10 0x7fc990003ef0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009fc0 tx=0x7fc99c01cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e3b98be9c660001 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f6975213640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6968046b30 0x7f69680d6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e27fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f95dc043240 0x7f95dc007180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e27fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f95dc043240 0x7f95dc007180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50516/0 (socket says 172.21.15.107:50516) 2023-10-24T23:53:21.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e27fc640 1 -- 172.21.15.107:0/2527680052 learned_addr learned my addr 172.21.15.107:0/2527680052 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f6975213640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696802e750 con 0x7f69680d3a60 2023-10-24T23:53:21.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1ee27fc640 1 -- 172.21.15.107:0/1194317713 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed40ab600 con 0x7f1ee40c1b40 2023-10-24T23:53:21.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e2ffd640 1 --2- 172.21.15.107:0/2527680052 >> v2:172.21.15.110:3300/0 conn(0x7f95dc0d4200 0x7f95dc0dac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc48469e640 1 -- 172.21.15.107:0/230709187 shutdown_connections 2023-10-24T23:53:21.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc48469e640 1 --2- 172.21.15.107:0/230709187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc47c0b9470 0x7fc47c154610 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc48469e640 1 --2- 172.21.15.107:0/230709187 >> v2:172.21.15.110:3300/0 conn(0x7fc47c0c1b40 0x7fc47c0a91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1ee27fc640 1 -- 172.21.15.107:0/1194317713 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed40a7050 con 0x7f1ee40c1b40 2023-10-24T23:53:21.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1ee27fc640 1 -- 172.21.15.107:0/1194317713 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed40ab980 con 0x7f1ee40c1b40 2023-10-24T23:53:21.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e37fe640 1 --2- 172.21.15.107:0/2527680052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95dc0076c0 0x7f95dc00e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc48469e640 1 -- 172.21.15.107:0/230709187 >> 172.21.15.107:0/230709187 conn(0x7fc47c0c2860 msgr2=0x7fc47c0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0a5c70 con 0x7fc990003b10 2023-10-24T23:53:21.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c00f040 con 0x7fc990003b10 2023-10-24T23:53:21.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc9a9232640 1 -- 172.21.15.107:0/2824289561 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0ad440 con 0x7fc990003b10 2023-10-24T23:53:21.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc48469e640 1 -- 172.21.15.107:0/230709187 shutdown_connections 2023-10-24T23:53:21.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fd0c9d82640 1 Processor -- start 2023-10-24T23:53:21.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fd0c9d82640 1 -- start start 2023-10-24T23:53:21.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc9ac498640 1 -- 172.21.15.107:0/2824289561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc990003b10 msgr2=0x7fc990003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fc9ac498640 1 --2- 172.21.15.107:0/2824289561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc990003b10 0x7fc990003ef0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009fc0 tx=0x7fc99c01cc80 comp rx=0 tx=0).stop 2023-10-24T23:53:21.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fb8ef7fe640 1 --2- 172.21.15.107:0/859842608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f00c4d70 0x7fb8f01530f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb8f00c6d20 tx=0x7fb8e001c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3353b4aaabd9bc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f6975213640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696802e560 con 0x7f6968046b30 2023-10-24T23:53:21.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f6975213640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69680d3e40 con 0x7f69680da110 2023-10-24T23:53:21.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f696e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69680da110 0x7f69680465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f696e59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69680da110 0x7f69680465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:51292/0 (socket says 172.21.15.107:51292) 2023-10-24T23:53:21.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f696e59c640 1 -- 172.21.15.107:0/3109370451 learned_addr learned my addr 172.21.15.107:0/3109370451 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f696ed9d640 1 --2- 172.21.15.107:0/3109370451 >> v2:172.21.15.107:3300/0 conn(0x7f69680d3a60 0x7f69680d7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1eea163640 1 -- 172.21.15.107:0/1194317713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ee40c1b40 msgr2=0x7f1ee406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1eea163640 1 --2- 172.21.15.107:0/1194317713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ee40c1b40 0x7f1ee406b760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1ee406e0c0 tx=0x7f1ed401c450 comp rx=0 tx=0).stop 2023-10-24T23:53:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1eea163640 1 -- 172.21.15.107:0/1194317713 shutdown_connections 2023-10-24T23:53:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1eea163640 1 --2- 172.21.15.107:0/1194317713 >> v2:172.21.15.107:3300/0 conn(0x7f1ee40b9470 0x7f1ee40a91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1eea163640 1 --2- 172.21.15.107:0/1194317713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ee40c1b40 0x7f1ee406b760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1eea163640 1 -- 172.21.15.107:0/1194317713 >> 172.21.15.107:0/1194317713 conn(0x7f1ee40c2860 msgr2=0x7f1ee40c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1eea163640 1 -- 172.21.15.107:0/1194317713 shutdown_connections 2023-10-24T23:53:21.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f1eea163640 1 -- 172.21.15.107:0/1194317713 wait complete. 2023-10-24T23:53:21.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fd0c9d82640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd0c40a5be0 0x7fd0c415c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fd0c9d82640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c40c0f60 0x7fd0c415c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fd0c9d82640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c40c2d80 con 0x7fd0c4065b50 2023-10-24T23:53:21.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fd0c9d82640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c415ce70 con 0x7fd0c40a5be0 2023-10-24T23:53:21.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7fd0c9d82640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c40c27c0 con 0x7fd0c40c0f60 2023-10-24T23:53:21.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc48469e640 1 -- 172.21.15.107:0/230709187 wait complete. 2023-10-24T23:53:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e27fc640 1 -- 172.21.15.107:0/2527680052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95dc0076c0 msgr2=0x7f95dc00e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e27fc640 1 --2- 172.21.15.107:0/2527680052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95dc0076c0 0x7f95dc00e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e27fc640 1 -- 172.21.15.107:0/2527680052 >> v2:172.21.15.110:3300/0 conn(0x7f95dc0d4200 msgr2=0x7f95dc0dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 -- 172.21.15.107:0/2824289561 shutdown_connections 2023-10-24T23:53:21.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 --2- 172.21.15.107:0/2824289561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc990003b10 0x7fc990003ef0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 -- 172.21.15.107:0/2824289561 >> 172.21.15.107:0/2824289561 conn(0x7fc9a40c2860 msgr2=0x7fc9a40c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e27fc640 1 --2- 172.21.15.107:0/2527680052 >> v2:172.21.15.110:3300/0 conn(0x7f95dc0d4200 0x7f95dc0dac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f95e27fc640 1 -- 172.21.15.107:0/2527680052 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95dc04a970 con 0x7f95dc043240 2023-10-24T23:53:21.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fd0c2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c40c0f60 0x7fd0c415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fd0c2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c40c0f60 0x7fd0c415c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:51296/0 (socket says 172.21.15.107:51296) 2023-10-24T23:53:21.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fd0c2ffd640 1 -- 172.21.15.107:0/3002780338 learned_addr learned my addr 172.21.15.107:0/3002780338 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 -- 172.21.15.107:0/2824289561 shutdown_connections 2023-10-24T23:53:21.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fd0c37fe640 1 --2- 172.21.15.107:0/3002780338 >> v2:172.21.15.107:3300/0 conn(0x7fd0c40a5be0 0x7fd0c415c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 -- 172.21.15.107:0/2824289561 wait complete. 2023-10-24T23:53:21.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f696ed9d640 1 -- 172.21.15.107:0/3109370451 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69680da110 msgr2=0x7f69680465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f696ed9d640 1 --2- 172.21.15.107:0/3109370451 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69680da110 0x7f69680465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f696ed9d640 1 -- 172.21.15.107:0/3109370451 >> v2:172.21.15.110:3300/0 conn(0x7f6968046b30 msgr2=0x7f69680d6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f696f59e640 1 --2- 172.21.15.107:0/3109370451 >> v2:172.21.15.110:3300/0 conn(0x7f6968046b30 0x7f69680d6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f696ed9d640 1 --2- 172.21.15.107:0/3109370451 >> v2:172.21.15.110:3300/0 conn(0x7f6968046b30 0x7f69680d6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.794+0000 7f696ed9d640 1 -- 172.21.15.107:0/3109370451 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6968007210 con 0x7f69680d3a60 2023-10-24T23:53:21.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fb8eeffd640 1 -- 172.21.15.107:0/859842608 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e00ab620 con 0x7fb8f00c4d70 2023-10-24T23:53:21.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f1eea163640 1 Processor -- start 2023-10-24T23:53:21.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc48469e640 1 Processor -- start 2023-10-24T23:53:21.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc48469e640 1 -- start start 2023-10-24T23:53:21.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 Processor -- start 2023-10-24T23:53:21.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fb8eeffd640 1 -- 172.21.15.107:0/859842608 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e00ab780 con 0x7fb8f00c4d70 2023-10-24T23:53:21.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fb8eeffd640 1 -- 172.21.15.107:0/859842608 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e00abb10 con 0x7fb8f00c4d70 2023-10-24T23:53:21.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fb8f5444640 1 -- 172.21.15.107:0/859842608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f00c4d70 msgr2=0x7fb8f01530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fb8f5444640 1 --2- 172.21.15.107:0/859842608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f00c4d70 0x7fb8f01530f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb8f00c6d20 tx=0x7fb8e001c420 comp rx=0 tx=0).stop 2023-10-24T23:53:21.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc48469e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc47c0b9470 0x7fc47c0bfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc48469e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc47c0c1b40 0x7fc47c0705d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc48469e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc47c0c4f50 0x7fc47c14dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc48469e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c0be880 con 0x7fc47c0c1b40 2023-10-24T23:53:21.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc48469e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c153140 con 0x7fc47c0b9470 2023-10-24T23:53:21.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc48469e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc47c071900 con 0x7fc47c0c4f50 2023-10-24T23:53:21.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f95e27fc640 1 --2- 172.21.15.107:0/2527680052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95dc043240 0x7f95dc007180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f95dc011a80 tx=0x7f95d801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f164283a878d5a27 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc481c39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc47c0c1b40 0x7fc47c0705d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc48243a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc47c0b9470 0x7fc47c0bfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc481c39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc47c0c1b40 0x7fc47c0705d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50530/0 (socket says 172.21.15.107:50530) 2023-10-24T23:53:21.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc481c39640 1 -- 172.21.15.107:0/3862168941 learned_addr learned my addr 172.21.15.107:0/3862168941 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc482c3b640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc47c0c4f50 0x7fc47c14dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 -- start start 2023-10-24T23:53:21.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc990003b10 0x7fc9a406ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9a4070ea0 0x7fc9a40b9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a4071300 0x7fc9a40bc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a406f580 con 0x7fc9a4071300 2023-10-24T23:53:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a40c4cf0 con 0x7fc990003b10 2023-10-24T23:53:21.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9ac498640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a40b93e0 con 0x7fc9a4070ea0 2023-10-24T23:53:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9a9a33640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9a4070ea0 0x7fc9a40b9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fd0c2ffd640 1 -- 172.21.15.107:0/3002780338 >> v2:172.21.15.107:3300/0 conn(0x7fd0c40a5be0 msgr2=0x7fd0c415c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fd0c2ffd640 1 --2- 172.21.15.107:0/3002780338 >> v2:172.21.15.107:3300/0 conn(0x7fd0c40a5be0 0x7fd0c415c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fd0c2ffd640 1 -- 172.21.15.107:0/3002780338 >> v1:172.21.15.107:6789/0 conn(0x7fd0c4065b50 legacy=0x7fd0c406d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fd0c2ffd640 1 -- 172.21.15.107:0/3002780338 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c40c2aa0 con 0x7fd0c40c0f60 2023-10-24T23:53:21.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f696ed9d640 1 --2- 172.21.15.107:0/3109370451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69680d3a60 0x7f69680d7810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6968009610 tx=0x7f696001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2660b4029411fae3 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9a9a33640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9a4070ea0 0x7fc9a40b9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:51308/0 (socket says 172.21.15.107:51308) 2023-10-24T23:53:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9a9a33640 1 -- 172.21.15.107:0/3902407032 learned_addr learned my addr 172.21.15.107:0/3902407032 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9aa234640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc990003b10 0x7fc9a406ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f95e1ffb640 1 -- 172.21.15.107:0/2527680052 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95dc00f350 con 0x7f95dc043240 2023-10-24T23:53:21.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f95e1ffb640 1 -- 172.21.15.107:0/2527680052 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d80a7050 con 0x7f95dc043240 2023-10-24T23:53:21.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f95e1ffb640 1 -- 172.21.15.107:0/2527680052 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d80ab820 con 0x7f95dc043240 2023-10-24T23:53:21.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc481c39640 1 -- 172.21.15.107:0/3862168941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc47c0c4f50 msgr2=0x7fc47c14dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc481c39640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc47c0c4f50 0x7fc47c14dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc481c39640 1 -- 172.21.15.107:0/3862168941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc47c0b9470 msgr2=0x7fc47c0bfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc481c39640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc47c0b9470 0x7fc47c0bfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc481c39640 1 -- 172.21.15.107:0/3862168941 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47800d050 con 0x7fc47c0c1b40 2023-10-24T23:53:21.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9aaa35640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a4071300 0x7fc9a40bc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f1eea163640 1 -- start start 2023-10-24T23:53:21.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f95e3fff640 1 -- 172.21.15.107:0/2527680052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95dc043240 msgr2=0x7f95dc007180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f95e3fff640 1 --2- 172.21.15.107:0/2527680052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95dc043240 0x7f95dc007180 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f95dc011a80 tx=0x7f95d801c3d0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fb8f5444640 1 -- 172.21.15.107:0/859842608 shutdown_connections 2023-10-24T23:53:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fb8f5444640 1 --2- 172.21.15.107:0/859842608 >> v2:172.21.15.107:3300/0 conn(0x7fb8f006ccd0 0x7fb8f0071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fb8f5444640 1 --2- 172.21.15.107:0/859842608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f00c4d70 0x7fb8f01530f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fb8f5444640 1 -- 172.21.15.107:0/859842608 >> 172.21.15.107:0/859842608 conn(0x7fb8f00c6dc0 msgr2=0x7fb8f00c71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f95e3fff640 1 -- 172.21.15.107:0/2527680052 shutdown_connections 2023-10-24T23:53:21.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f95e3fff640 1 --2- 172.21.15.107:0/2527680052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95dc0076c0 0x7f95dc00e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f95e3fff640 1 --2- 172.21.15.107:0/2527680052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95dc043240 0x7f95dc007180 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f95dc011a80 tx=0x7f95d801c3d0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f95e3fff640 1 --2- 172.21.15.107:0/2527680052 >> v2:172.21.15.110:3300/0 conn(0x7f95dc0d4200 0x7f95dc0dac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7f95e3fff640 1 -- 172.21.15.107:0/2527680052 >> 172.21.15.107:0/2527680052 conn(0x7f95dc04c630 msgr2=0x7f95dc04ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.798+0000 7fc9a9a33640 1 -- 172.21.15.107:0/3902407032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc990003b10 msgr2=0x7fc9a406ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc9a9a33640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc990003b10 0x7fc9a406ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc9a9a33640 1 -- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a4071300 msgr2=0x7fc9a40bc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc9a9a33640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a4071300 0x7fc9a40bc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc9a9a33640 1 -- 172.21.15.107:0/3902407032 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4072430 con 0x7fc9a4070ea0 2023-10-24T23:53:21.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c2ffd640 1 --2- 172.21.15.107:0/3002780338 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c40c0f60 0x7fd0c415c800 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009870 tx=0x7fd0b401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=755562119bdf0e28 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f5444640 1 -- 172.21.15.107:0/859842608 shutdown_connections 2023-10-24T23:53:21.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1eea163640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee40b9470 0x7f1ee40bb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1eea163640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ee40c3380 0x7f1ee4071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1eea163640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ee41f1d80 0x7f1ee406f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1eea163640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee41f2280 con 0x7f1ee40b9470 2023-10-24T23:53:21.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1eea163640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee4070370 con 0x7f1ee40c3380 2023-10-24T23:53:21.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1eea163640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee4070650 con 0x7f1ee41f1d80 2023-10-24T23:53:21.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc481c39640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc47c0c1b40 0x7fc47c0705d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc46c009f40 tx=0x7fc46c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc9a9a33640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9a4070ea0 0x7fc9a40b9c50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc994012820 tx=0x7fc994012c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc99affd640 1 -- 172.21.15.107:0/3902407032 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9940054e0 con 0x7fc9a4070ea0 2023-10-24T23:53:21.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc9ac498640 1 -- 172.21.15.107:0/3902407032 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a406f790 con 0x7fc9a4070ea0 2023-10-24T23:53:21.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc9ac498640 1 -- 172.21.15.107:0/3902407032 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a40ba190 con 0x7fc9a4070ea0 2023-10-24T23:53:21.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc99affd640 1 -- 172.21.15.107:0/3902407032 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc994012e70 con 0x7fc9a4070ea0 2023-10-24T23:53:21.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc99affd640 1 -- 172.21.15.107:0/3902407032 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994010990 con 0x7fc9a4070ea0 2023-10-24T23:53:21.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc472ffd640 1 -- 172.21.15.107:0/3862168941 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c009510 con 0x7fc47c0c1b40 2023-10-24T23:53:21.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc472ffd640 1 -- 172.21.15.107:0/3862168941 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46c00eec0 con 0x7fc47c0c1b40 2023-10-24T23:53:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc472ffd640 1 -- 172.21.15.107:0/3862168941 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46c00cb80 con 0x7fc47c0c1b40 2023-10-24T23:53:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c27fc640 1 -- 172.21.15.107:0/3002780338 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b40ad680 con 0x7fd0c40c0f60 2023-10-24T23:53:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c27fc640 1 -- 172.21.15.107:0/3002780338 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b40ad7e0 con 0x7fd0c40c0f60 2023-10-24T23:53:21.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c27fc640 1 -- 172.21.15.107:0/3002780338 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b40adb70 con 0x7fd0c40c0f60 2023-10-24T23:53:21.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee40b9470 0x7f1ee40bb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee40b9470 0x7f1ee40bb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50560/0 (socket says 172.21.15.107:50560) 2023-10-24T23:53:21.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee37fe640 1 -- 172.21.15.107:0/49266883 learned_addr learned my addr 172.21.15.107:0/49266883 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee2ffd640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ee40c3380 0x7f1ee4071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee37fe640 1 -- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ee41f1d80 msgr2=0x7f1ee406f270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-24T23:53:21.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f696dd9b640 1 -- 172.21.15.107:0/3109370451 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69600ad680 con 0x7f69680d3a60 2023-10-24T23:53:21.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f696dd9b640 1 -- 172.21.15.107:0/3109370451 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69600ad7e0 con 0x7f69680d3a60 2023-10-24T23:53:21.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f696dd9b640 1 -- 172.21.15.107:0/3109370451 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69600adb50 con 0x7f69680d3a60 2023-10-24T23:53:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f6975213640 1 -- 172.21.15.107:0/3109370451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69680d3a60 msgr2=0x7f69680d7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f6975213640 1 --2- 172.21.15.107:0/3109370451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69680d3a60 0x7f69680d7810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6968009610 tx=0x7f696001c3c0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f95e3fff640 1 -- 172.21.15.107:0/2527680052 shutdown_connections 2023-10-24T23:53:21.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f5444640 1 -- 172.21.15.107:0/859842608 wait complete. 2023-10-24T23:53:21.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f5444640 1 Processor -- start 2023-10-24T23:53:21.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc9ac498640 1 -- 172.21.15.107:0/3902407032 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc98c000fd0 con 0x7fc9a4070ea0 2023-10-24T23:53:21.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f5444640 1 -- start start 2023-10-24T23:53:21.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f5444640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8f006ccd0 0x7fb8f00bf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f5444640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f00a5f30 0x7fb8f00a8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f5444640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00c4d70 0x7fb8f00a8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f5444640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00c0220 con 0x7fb8f00c4d70 2023-10-24T23:53:21.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f5444640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f00be0c0 con 0x7fb8f00a5f30 2023-10-24T23:53:21.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f5444640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f01562b0 con 0x7fb8f006ccd0 2023-10-24T23:53:21.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc48469e640 1 -- 172.21.15.107:0/3862168941 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc47c070b10 con 0x7fc47c0c1b40 2023-10-24T23:53:21.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc48469e640 1 -- 172.21.15.107:0/3862168941 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc47c153390 con 0x7fc47c0c1b40 2023-10-24T23:53:21.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fc48469e640 1 -- 172.21.15.107:0/3862168941 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc464000fd0 con 0x7fc47c0c1b40 2023-10-24T23:53:21.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c9d82640 1 -- 172.21.15.107:0/3002780338 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c40c0f60 msgr2=0x7fd0c415c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c9d82640 1 --2- 172.21.15.107:0/3002780338 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c40c0f60 0x7fd0c415c800 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009870 tx=0x7fd0b401c400 comp rx=0 tx=0).stop 2023-10-24T23:53:21.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c9d82640 1 -- 172.21.15.107:0/3002780338 shutdown_connections 2023-10-24T23:53:21.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c9d82640 1 --2- 172.21.15.107:0/3002780338 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c40c0f60 0x7fd0c415c800 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c9d82640 1 --2- 172.21.15.107:0/3002780338 >> v2:172.21.15.107:3300/0 conn(0x7fd0c40a5be0 0x7fd0c415c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c9d82640 1 -- 172.21.15.107:0/3002780338 >> 172.21.15.107:0/3002780338 conn(0x7fd0c40c65e0 msgr2=0x7fd0c40c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c9d82640 1 -- 172.21.15.107:0/3002780338 shutdown_connections 2023-10-24T23:53:21.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f6975213640 1 -- 172.21.15.107:0/3109370451 shutdown_connections 2023-10-24T23:53:21.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f6975213640 1 --2- 172.21.15.107:0/3109370451 >> v2:172.21.15.110:3300/0 conn(0x7f6968046b30 0x7f69680d6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f6975213640 1 --2- 172.21.15.107:0/3109370451 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69680da110 0x7f69680465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f6975213640 1 --2- 172.21.15.107:0/3109370451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69680d3a60 0x7f69680d7810 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f6975213640 1 -- 172.21.15.107:0/3109370451 >> 172.21.15.107:0/3109370451 conn(0x7f6968006330 msgr2=0x7f6968006740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f6975213640 1 -- 172.21.15.107:0/3109370451 shutdown_connections 2023-10-24T23:53:21.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee3fff640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ee41f1d80 0x7f1ee406f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee37fe640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ee41f1d80 0x7f1ee406f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee37fe640 1 -- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ee40c3380 msgr2=0x7f1ee4071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee37fe640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ee40c3380 0x7f1ee4071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee37fe640 1 -- 172.21.15.107:0/49266883 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed40094d0 con 0x7f1ee40b9470 2023-10-24T23:53:21.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f95e3fff640 1 -- 172.21.15.107:0/2527680052 wait complete. 2023-10-24T23:53:21.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f298359e640 1 Processor -- start 2023-10-24T23:53:21.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8effff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8f006ccd0 0x7fb8f00bf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f4c43640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00c4d70 0x7fb8f00a8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f4c43640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00c4d70 0x7fb8f00a8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50574/0 (socket says 172.21.15.107:50574) 2023-10-24T23:53:21.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f4c43640 1 -- 172.21.15.107:0/2837310646 learned_addr learned my addr 172.21.15.107:0/2837310646 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f4c43640 1 -- 172.21.15.107:0/2837310646 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8f006ccd0 msgr2=0x7fb8f00bf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f4c43640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8f006ccd0 0x7fb8f00bf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f4c43640 1 -- 172.21.15.107:0/2837310646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f00a5f30 msgr2=0x7fb8f00a8730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-24T23:53:21.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f4c43640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8f00a5f30 0x7fb8f00a8730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f4c43640 1 -- 172.21.15.107:0/2837310646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e0009420 con 0x7fb8f00c4d70 2023-10-24T23:53:21.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fb8f4c43640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00c4d70 0x7fb8f00a8c70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800efc0 tx=0x7fb8e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee37fe640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee40b9470 0x7f1ee40bb200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1ed40ae040 tx=0x7f1ed40191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ecbfff640 1 -- 172.21.15.107:0/49266883 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed40b4640 con 0x7f1ee40b9470 2023-10-24T23:53:21.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ecbfff640 1 -- 172.21.15.107:0/49266883 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed40b47c0 con 0x7f1ee40b9470 2023-10-24T23:53:21.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1eea163640 1 -- 172.21.15.107:0/49266883 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee4070930 con 0x7f1ee40b9470 2023-10-24T23:53:21.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ecbfff640 1 -- 172.21.15.107:0/49266883 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed40b4b70 con 0x7f1ee40b9470 2023-10-24T23:53:21.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1eea163640 1 -- 172.21.15.107:0/49266883 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee40bb740 con 0x7f1ee40b9470 2023-10-24T23:53:21.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c9d82640 1 -- 172.21.15.107:0/3002780338 wait complete. 2023-10-24T23:53:21.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f6975213640 1 -- 172.21.15.107:0/3109370451 wait complete. 2023-10-24T23:53:21.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c9d82640 1 Processor -- start 2023-10-24T23:53:21.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f6975213640 1 Processor -- start 2023-10-24T23:53:21.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8ecff9640 1 -- 172.21.15.107:0/2837310646 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800cd20 con 0x7fb8f00c4d70 2023-10-24T23:53:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8ecff9640 1 -- 172.21.15.107:0/2837310646 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e800f040 con 0x7fb8f00c4d70 2023-10-24T23:53:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8ecff9640 1 -- 172.21.15.107:0/2837310646 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8003be0 con 0x7fb8f00c4d70 2023-10-24T23:53:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7fd0c9d82640 1 -- start start 2023-10-24T23:53:21.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c9d82640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c4065b50 0x7fd0c40aa770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c9d82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c40a5be0 0x7fd0c40aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c9d82640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0c40c0f60 0x7fd0c41f9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f1ee2ffd640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ee40c3380 0x7f1ee4071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:53:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c9d82640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c406d560 con 0x7fd0c40a5be0 2023-10-24T23:53:21.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f95e3fff640 1 Processor -- start 2023-10-24T23:53:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.802+0000 7f298359e640 1 -- start start 2023-10-24T23:53:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298359e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f297c03f900 0x7f297c03fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298359e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 0x7f297c0dea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298359e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f297c04b440 con 0x7f297c009140 2023-10-24T23:53:21.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298359e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f297c040220 con 0x7f297c03f900 2023-10-24T23:53:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298359e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297c0de6c0 con 0x7f297c00c1d0 2023-10-24T23:53:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298259c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f297c03f900 0x7f297c03fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298259c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f297c03f900 0x7f297c03fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54674/0 (socket says 172.21.15.107:54674) 2023-10-24T23:53:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298259c640 1 -- 172.21.15.107:0/2293017225 learned_addr learned my addr 172.21.15.107:0/2293017225 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f2981d9b640 1 --2- 172.21.15.107:0/2293017225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 0x7f297c0dea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8f5444640 1 -- 172.21.15.107:0/2837310646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f00a91b0 con 0x7fb8f00c4d70 2023-10-24T23:53:21.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8f5444640 1 -- 172.21.15.107:0/2837310646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f00be5f0 con 0x7fb8f00c4d70 2023-10-24T23:53:21.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc472ffd640 1 -- 172.21.15.107:0/3862168941 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc46c00cce0 con 0x7fc47c0c1b40 2023-10-24T23:53:21.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc472ffd640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc460065e60 0x7fc460068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc472ffd640 1 -- 172.21.15.107:0/3862168941 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fc46c087950 con 0x7fc47c0c1b40 2023-10-24T23:53:21.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc472ffd640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] conn(0x7fc460070da0 0x7fc4600731c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc472ffd640 1 -- 172.21.15.107:0/3862168941 --> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc460073870 con 0x7fc460070da0 2023-10-24T23:53:21.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc472ffd640 1 -- 172.21.15.107:0/3862168941 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc46c087d00 con 0x7fc47c0c1b40 2023-10-24T23:53:21.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc48243a640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc460065e60 0x7fc460068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc482c3b640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] conn(0x7fc460070da0 0x7fc4600731c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c9d82640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c4157190 con 0x7fd0c40c0f60 2023-10-24T23:53:21.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c9d82640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c41596d0 con 0x7fd0c4065b50 2023-10-24T23:53:21.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0c40c0f60 0x7fd0c41f9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0c40c0f60 0x7fd0c41f9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54690/0 (socket says 172.21.15.107:54690) 2023-10-24T23:53:21.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c3fff640 1 -- 172.21.15.107:0/995878872 learned_addr learned my addr 172.21.15.107:0/995878872 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f6975213640 1 -- start start 2023-10-24T23:53:21.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f6975213640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6968046b30 0x7f696804ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f6975213640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69680d3a60 0x7f6968178640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f6975213640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69680da110 0x7f6968178b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f6975213640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69680d5d10 con 0x7f69680d3a60 2023-10-24T23:53:21.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f6975213640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69680d4c60 con 0x7f69680da110 2023-10-24T23:53:21.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f6975213640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69680d4f40 con 0x7f6968046b30 2023-10-24T23:53:21.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696f59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69680da110 0x7f6968178b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696f59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69680da110 0x7f6968178b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54692/0 (socket says 172.21.15.107:54692) 2023-10-24T23:53:21.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696f59e640 1 -- 172.21.15.107:0/2399793531 learned_addr learned my addr 172.21.15.107:0/2399793531 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f1eea163640 1 -- 172.21.15.107:0/49266883 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1ee40a9730 con 0x7f1ee40b9470 2023-10-24T23:53:21.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c37fe640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c4065b50 0x7fd0c40aa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696ed9d640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6968046b30 0x7f696804ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c3fff640 1 -- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c4065b50 msgr2=0x7fd0c40aa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c3fff640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c4065b50 0x7fd0c40aa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c3fff640 1 -- 172.21.15.107:0/995878872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c40a5be0 msgr2=0x7fd0c40aacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c3fff640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0c40a5be0 0x7fd0c40aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c3fff640 1 -- 172.21.15.107:0/995878872 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b40094d0 con 0x7fd0c40c0f60 2023-10-24T23:53:21.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8f5444640 1 -- 172.21.15.107:0/2837310646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb8d0000fd0 con 0x7fb8f00c4d70 2023-10-24T23:53:21.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c37fe640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0c4065b50 0x7fd0c40aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:21.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc99affd640 1 -- 172.21.15.107:0/3902407032 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc994010af0 con 0x7fc9a4070ea0 2023-10-24T23:53:21.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc99affd640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc988065e60 0x7fc988068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc99affd640 1 -- 172.21.15.107:0/3902407032 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fc994087730 con 0x7fc9a4070ea0 2023-10-24T23:53:21.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc99affd640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] conn(0x7fc988070d20 0x7fc988073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc99affd640 1 -- 172.21.15.107:0/3902407032 --> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9880737f0 con 0x7fc988070d20 2023-10-24T23:53:21.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc99affd640 1 -- 172.21.15.107:0/3902407032 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc994087ae0 con 0x7fc9a4070ea0 2023-10-24T23:53:21.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696e59c640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69680d3a60 0x7f6968178640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696f59e640 1 -- 172.21.15.107:0/2399793531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6968046b30 msgr2=0x7f696804ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696f59e640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6968046b30 0x7f696804ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696f59e640 1 -- 172.21.15.107:0/2399793531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69680d3a60 msgr2=0x7f6968178640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696f59e640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69680d3a60 0x7f6968178640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc9aaa35640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] conn(0x7fc988070d20 0x7fc988073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696f59e640 1 -- 172.21.15.107:0/2399793531 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6960009420 con 0x7f69680da110 2023-10-24T23:53:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696e59c640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69680d3a60 0x7f6968178640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c3fff640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0c40c0f60 0x7fd0c41f9e10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00b990 tx=0x7fd0bc00be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f95e3fff640 1 -- start start 2023-10-24T23:53:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8ecff9640 1 -- 172.21.15.107:0/2837310646 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb8e8003d40 con 0x7fb8f00c4d70 2023-10-24T23:53:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8ecff9640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8dc065e60 0x7fb8dc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc9aa234640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc988065e60 0x7fc988068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f696f59e640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69680da110 0x7f6968178b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f697002c850 tx=0x7f6970067a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0abfff640 1 -- 172.21.15.107:0/995878872 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc00cba0 con 0x7fd0c40c0f60 2023-10-24T23:53:21.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8ecff9640 1 -- 172.21.15.107:0/2837310646 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fb8e8084180 con 0x7fb8f00c4d70 2023-10-24T23:53:21.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8effff640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8dc065e60 0x7fb8dc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8ecff9640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] conn(0x7fb8dc070d20 0x7fb8dc073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8ecff9640 1 -- 172.21.15.107:0/2837310646 --> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8dc0737f0 con 0x7fb8dc070d20 2023-10-24T23:53:21.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8ecff9640 1 -- 172.21.15.107:0/2837310646 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8e8088170 con 0x7fb8f00c4d70 2023-10-24T23:53:21.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc9aaa35640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] conn(0x7fc988070d20 0x7fc988073140 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c9d82640 1 -- 172.21.15.107:0/995878872 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c41599b0 con 0x7fd0c40c0f60 2023-10-24T23:53:21.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fd0c9d82640 1 -- 172.21.15.107:0/995878872 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c4156e20 con 0x7fd0c40c0f60 2023-10-24T23:53:21.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f95e3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95dc0076c0 0x7f95dc032850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f95e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95dc0d4200 0x7f95dc0d6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f95e3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95dc034450 0x7f95dc17bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f95e3fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc17f590 con 0x7f95dc0d4200 2023-10-24T23:53:21.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f95e3fff640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc033e30 con 0x7f95dc0076c0 2023-10-24T23:53:21.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f95e3fff640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95dc0460c0 con 0x7f95dc034450 2023-10-24T23:53:21.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95dc0d4200 0x7f95dc0d6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95dc0d4200 0x7f95dc0d6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50602/0 (socket says 172.21.15.107:50602) 2023-10-24T23:53:21.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e27fc640 1 -- 172.21.15.107:0/1366911094 learned_addr learned my addr 172.21.15.107:0/1366911094 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e37fe640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95dc034450 0x7f95dc17bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7fc9aa234640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc988065e60 0x7fc988068320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0af000 tx=0x7fc99c004ac0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fb8effff640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8dc065e60 0x7fb8dc068320 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4001600 tx=0x7fb8e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f69577fe640 1 -- 172.21.15.107:0/2399793531 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6970069990 con 0x7f69680da110 2023-10-24T23:53:21.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f6975213640 1 -- 172.21.15.107:0/2399793531 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69680d59a0 con 0x7f69680da110 2023-10-24T23:53:21.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f6975213640 1 -- 172.21.15.107:0/2399793531 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696804b5b0 con 0x7f69680da110 2023-10-24T23:53:21.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f69577fe640 1 -- 172.21.15.107:0/2399793531 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6970071050 con 0x7f69680da110 2023-10-24T23:53:21.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e27fc640 1 -- 172.21.15.107:0/1366911094 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95dc034450 msgr2=0x7f95dc17bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e27fc640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f95dc034450 0x7f95dc17bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e27fc640 1 -- 172.21.15.107:0/1366911094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95dc0076c0 msgr2=0x7f95dc032850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e27fc640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f95dc0076c0 0x7f95dc032850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e27fc640 1 -- 172.21.15.107:0/1366911094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d80094d0 con 0x7f95dc0d4200 2023-10-24T23:53:21.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e27fc640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95dc0d4200 0x7f95dc0d6910 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f95d801a000 tx=0x7f95d8019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7fc99affd640 1 -- 172.21.15.107:0/3902407032 <== osd.1 v2:172.21.15.107:6800/37537 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc9880737f0 con 0x7fc988070d20 2023-10-24T23:53:21.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298259c640 1 -- 172.21.15.107:0/2293017225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 msgr2=0x7f297c0dea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298259c640 1 --2- 172.21.15.107:0/2293017225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 0x7f297c0dea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298259c640 1 -- 172.21.15.107:0/2293017225 >> v1:172.21.15.107:6789/0 conn(0x7f297c009140 legacy=0x7f297c0def70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:21.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7f298259c640 1 -- 172.21.15.107:0/2293017225 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c04a830 con 0x7f297c03f900 2023-10-24T23:53:21.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7fd0abfff640 1 -- 172.21.15.107:0/995878872 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc00cd00 con 0x7fd0c40c0f60 2023-10-24T23:53:21.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7fd0abfff640 1 -- 172.21.15.107:0/995878872 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc014680 con 0x7fd0c40c0f60 2023-10-24T23:53:21.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7fd0c9d82640 1 -- 172.21.15.107:0/995878872 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd0c40c44e0 con 0x7fd0c40c0f60 2023-10-24T23:53:21.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f2981d9b640 1 --2- 172.21.15.107:0/2293017225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 0x7f297c0dea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:21.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95cb7fe640 1 -- 172.21.15.107:0/1366911094 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d80abe60 con 0x7f95dc0d4200 2023-10-24T23:53:21.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e3fff640 1 -- 172.21.15.107:0/1366911094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95dc046360 con 0x7f95dc0d4200 2023-10-24T23:53:21.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e3fff640 1 -- 172.21.15.107:0/1366911094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95dc006bc0 con 0x7f95dc0d4200 2023-10-24T23:53:21.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95cb7fe640 1 -- 172.21.15.107:0/1366911094 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d8004030 con 0x7f95dc0d4200 2023-10-24T23:53:21.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95cb7fe640 1 -- 172.21.15.107:0/1366911094 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d80043a0 con 0x7f95dc0d4200 2023-10-24T23:53:21.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f69577fe640 1 -- 172.21.15.107:0/2399793531 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69700725d0 con 0x7f69680da110 2023-10-24T23:53:21.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.806+0000 7fc48243a640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc460065e60 0x7fc460068320 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc474007fd0 tx=0x7fc474009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7fb8ef7fe640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] conn(0x7fb8dc070d20 0x7fb8dc073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f6975213640 1 -- 172.21.15.107:0/2399793531 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6950000fd0 con 0x7f69680da110 2023-10-24T23:53:21.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95cb7fe640 1 -- 172.21.15.107:0/1366911094 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f95d8003620 con 0x7f95dc0d4200 2023-10-24T23:53:21.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f298259c640 1 --2- 172.21.15.107:0/2293017225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f297c03f900 0x7f297c03fce0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2978009870 tx=0x7f297801c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fcaed75882e7590 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7fb8ef7fe640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] conn(0x7fb8dc070d20 0x7fb8dc073140 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f1ecbfff640 1 -- 172.21.15.107:0/49266883 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1ed40b2070 con 0x7f1ee40b9470 2023-10-24T23:53:21.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f1ecbfff640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ed0065e60 0x7f1ed0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f1ecbfff640 1 -- 172.21.15.107:0/49266883 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f1ed40a4e40 con 0x7f1ee40b9470 2023-10-24T23:53:21.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f1ecbfff640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] conn(0x7f1ed0070d20 0x7f1ed0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f1ecbfff640 1 -- 172.21.15.107:0/49266883 --> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ed00737f0 con 0x7f1ed0070d20 2023-10-24T23:53:21.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f1ecbfff640 1 -- 172.21.15.107:0/49266883 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ed411e0b0 con 0x7f1ee40b9470 2023-10-24T23:53:21.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f1ee2ffd640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ed0065e60 0x7f1ed0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f1ee3fff640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] conn(0x7f1ed0070d20 0x7f1ed0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7fd0abfff640 1 -- 172.21.15.107:0/995878872 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd0bc01d3b0 con 0x7fd0c40c0f60 2023-10-24T23:53:21.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7fb8ecff9640 1 -- 172.21.15.107:0/2837310646 <== osd.3 v2:172.21.15.107:6812/37538 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb8dc0737f0 con 0x7fb8dc070d20 2023-10-24T23:53:21.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f1ee2ffd640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ed0065e60 0x7f1ed0068320 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40c30f0 tx=0x7f1edc009690 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95cb7fe640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f95d0065e90 0x7f95d0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e3fff640 1 -- 172.21.15.107:0/1366911094 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f95c4000fd0 con 0x7f95dc0d4200 2023-10-24T23:53:21.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7f95e2ffd640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f95d0065e90 0x7f95d0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7fd0abfff640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd0b0065f30 0x7fd0b00683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.810+0000 7fd0abfff640 1 -- 172.21.15.107:0/995878872 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fd0bc0841e0 con 0x7fd0c40c0f60 2023-10-24T23:53:21.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7fd0abfff640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] conn(0x7fd0b0070e70 0x7fd0b0073290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7fd0abfff640 1 -- 172.21.15.107:0/995878872 --> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0b0073940 con 0x7fd0b0070e70 2023-10-24T23:53:21.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7fc482c3b640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] conn(0x7fc460070da0 0x7fc4600731c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7fd0abfff640 1 -- 172.21.15.107:0/995878872 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0bc084590 con 0x7fd0c40c0f60 2023-10-24T23:53:21.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f69577fe640 1 -- 172.21.15.107:0/2399793531 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6970091070 con 0x7f69680da110 2023-10-24T23:53:21.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7fd0c37fe640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd0b0065f30 0x7fd0b00683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f69577fe640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f695c065eb0 0x7f695c068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f95e2ffd640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f95d0065e90 0x7f95d0068350 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f95d40099c0 tx=0x7f95d4009480 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7fd0c2ffd640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] conn(0x7fd0b0070e70 0x7fd0b0073290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f69577fe640 1 -- 172.21.15.107:0/2399793531 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f69700fe010 con 0x7f69680da110 2023-10-24T23:53:21.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f69577fe640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] conn(0x7f695c070c80 0x7f695c0730a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f69577fe640 1 -- 172.21.15.107:0/2399793531 --> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f695c073750 con 0x7f695c070c80 2023-10-24T23:53:21.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f69577fe640 1 -- 172.21.15.107:0/2399793531 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69700fe380 con 0x7f69680da110 2023-10-24T23:53:21.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f95cb7fe640 1 -- 172.21.15.107:0/1366911094 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f95d811dc50 con 0x7f95dc0d4200 2023-10-24T23:53:21.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f95cb7fe640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] conn(0x7f95d0070cd0 0x7f95d00730f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f95cb7fe640 1 -- 172.21.15.107:0/1366911094 --> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95d00737a0 con 0x7f95d0070cd0 2023-10-24T23:53:21.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f95cb7fe640 1 -- 172.21.15.107:0/1366911094 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95d8120310 con 0x7f95dc0d4200 2023-10-24T23:53:21.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7fd0c37fe640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd0b0065f30 0x7fd0b00683f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8009860 tx=0x7fd0b8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7fd0c2ffd640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] conn(0x7fd0b0070e70 0x7fd0b0073290 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f696ed9d640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f695c065eb0 0x7f695c068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f696e59c640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] conn(0x7f695c070c80 0x7f695c0730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f95e37fe640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] conn(0x7f95d0070cd0 0x7f95d00730f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f696ed9d640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f695c065eb0 0x7f695c068370 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f69600af000 tx=0x7f69600a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f696e59c640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] conn(0x7f695c070c80 0x7f695c0730a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f298159a640 1 -- 172.21.15.107:0/2293017225 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29780ad640 con 0x7f297c03f900 2023-10-24T23:53:21.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f298159a640 1 -- 172.21.15.107:0/2293017225 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29780ad7a0 con 0x7f297c03f900 2023-10-24T23:53:21.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f298159a640 1 -- 172.21.15.107:0/2293017225 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29780adb70 con 0x7f297c03f900 2023-10-24T23:53:21.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f95e37fe640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] conn(0x7f95d0070cd0 0x7f95d00730f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7fc472ffd640 1 -- 172.21.15.107:0/3862168941 <== osd.4 v2:172.21.15.110:6801/37764 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc460073870 con 0x7fc460070da0 2023-10-24T23:53:21.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f1ee3fff640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] conn(0x7f1ed0070d20 0x7f1ed0073140 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f298359e640 1 -- 172.21.15.107:0/2293017225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f297c03f900 msgr2=0x7f297c03fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f298359e640 1 --2- 172.21.15.107:0/2293017225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f297c03f900 0x7f297c03fce0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2978009870 tx=0x7f297801c3d0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f95cb7fe640 1 -- 172.21.15.107:0/1366911094 <== osd.0 v2:172.21.15.107:6810/37540 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f95d00737a0 con 0x7f95d0070cd0 2023-10-24T23:53:21.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f298359e640 1 -- 172.21.15.107:0/2293017225 shutdown_connections 2023-10-24T23:53:21.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f298359e640 1 --2- 172.21.15.107:0/2293017225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 0x7f297c0dea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f298359e640 1 --2- 172.21.15.107:0/2293017225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f297c03f900 0x7f297c03fce0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f298359e640 1 -- 172.21.15.107:0/2293017225 >> 172.21.15.107:0/2293017225 conn(0x7f297c007c10 msgr2=0x7f297c008020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f298359e640 1 -- 172.21.15.107:0/2293017225 shutdown_connections 2023-10-24T23:53:21.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.814+0000 7f298359e640 1 -- 172.21.15.107:0/2293017225 wait complete. 2023-10-24T23:53:21.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f69577fe640 1 -- 172.21.15.107:0/2399793531 <== osd.6 v2:172.21.15.110:6800/37765 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f695c073750 con 0x7f695c070c80 2023-10-24T23:53:21.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f1ecbfff640 1 -- 172.21.15.107:0/49266883 <== osd.7 v2:172.21.15.110:6824/37766 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1ed00737f0 con 0x7f1ed0070d20 2023-10-24T23:53:21.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f298359e640 1 Processor -- start 2023-10-24T23:53:21.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f298359e640 1 -- start start 2023-10-24T23:53:21.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f298359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f297c009140 0x7f297c0cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f298359e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 0x7f297c0dc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f298359e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f297c03f900 0x7f297c031fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f298359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c0dda50 con 0x7f297c009140 2023-10-24T23:53:21.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f298359e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c0dc8e0 con 0x7f297c03f900 2023-10-24T23:53:21.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f298359e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297c0dc550 con 0x7f297c00c1d0 2023-10-24T23:53:21.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2981d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 0x7f297c0dc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2981d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 0x7f297c0dc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:51382/0 (socket says 172.21.15.107:51382) 2023-10-24T23:53:21.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2981d9b640 1 -- 172.21.15.107:0/3095230121 learned_addr learned my addr 172.21.15.107:0/3095230121 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:21.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2981d9b640 1 -- 172.21.15.107:0/3095230121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f297c03f900 msgr2=0x7f297c031fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-24T23:53:21.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2981d9b640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f297c03f900 0x7f297c031fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2981d9b640 1 -- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f297c009140 msgr2=0x7f297c0cbf30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-24T23:53:21.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2981d9b640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f297c009140 0x7f297c0cbf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2981d9b640 1 -- 172.21.15.107:0/3095230121 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29780094d0 con 0x7f297c00c1d0 2023-10-24T23:53:21.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2981d9b640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 0x7f297c0dc010 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f297400c7d0 tx=0x7f297400cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2972ffd640 1 -- 172.21.15.107:0/3095230121 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2974007a30 con 0x7f297c00c1d0 2023-10-24T23:53:21.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2972ffd640 1 -- 172.21.15.107:0/3095230121 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2974007b90 con 0x7f297c00c1d0 2023-10-24T23:53:21.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.818+0000 7f2972ffd640 1 -- 172.21.15.107:0/3095230121 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29740113f0 con 0x7f297c00c1d0 2023-10-24T23:53:21.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.822+0000 7f298359e640 1 -- 172.21.15.107:0/3095230121 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297c0dd080 con 0x7f297c00c1d0 2023-10-24T23:53:21.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.822+0000 7f298359e640 1 -- 172.21.15.107:0/3095230121 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f297c0dcda0 con 0x7f297c00c1d0 2023-10-24T23:53:21.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.822+0000 7f2972ffd640 1 -- 172.21.15.107:0/3095230121 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2974011550 con 0x7f297c00c1d0 2023-10-24T23:53:21.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.822+0000 7f2972ffd640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2964087e70 0x7f296408a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.822+0000 7f2972ffd640 1 -- 172.21.15.107:0/3095230121 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f2974083ce0 con 0x7f297c00c1d0 2023-10-24T23:53:21.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.822+0000 7f298259c640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2964087e70 0x7f296408a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.826+0000 7fc9ac498640 1 -- 172.21.15.107:0/3902407032 --> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc98c002cc0 con 0x7fc988070d20 2023-10-24T23:53:21.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.826+0000 7f298259c640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2964087e70 0x7f296408a330 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f29780af000 tx=0x7f29780a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.826+0000 7fc99affd640 1 -- 172.21.15.107:0/3902407032 <== osd.1 v2:172.21.15.107:6800/37537 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc98c002cc0 con 0x7fc988070d20 2023-10-24T23:53:21.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.826+0000 7f298359e640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] conn(0x7f29680015f0 0x7f2968003ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:21.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.826+0000 7f298359e640 1 -- 172.21.15.107:0/3095230121 --> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2968006b50 con 0x7f29680015f0 2023-10-24T23:53:21.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.826+0000 7f2982d9d640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] conn(0x7f29680015f0 0x7f2968003ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:21.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.826+0000 7f2982d9d640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] conn(0x7f29680015f0 0x7f2968003ab0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:21.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.826+0000 7f2972ffd640 1 -- 172.21.15.107:0/3095230121 <== osd.2 v2:172.21.15.107:6808/37539 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f2968006b50 con 0x7f29680015f0 2023-10-24T23:53:21.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fd0abfff640 1 -- 172.21.15.107:0/995878872 <== osd.5 v2:172.21.15.110:6816/37763 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd0b0073940 con 0x7fd0b0070e70 2023-10-24T23:53:21.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f5444640 1 -- 172.21.15.107:0/2837310646 --> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb8d0002cc0 con 0x7fb8dc070d20 2023-10-24T23:53:21.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8ecff9640 1 -- 172.21.15.107:0/2837310646 <== osd.3 v2:172.21.15.107:6812/37538 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb8d0002cc0 con 0x7fb8dc070d20 2023-10-24T23:53:21.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f5444640 1 -- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] conn(0x7fb8dc070d20 msgr2=0x7fb8dc073140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f5444640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] conn(0x7fb8dc070d20 0x7fb8dc073140 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f5444640 1 -- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8dc065e60 msgr2=0x7fb8dc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f5444640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8dc065e60 0x7fb8dc068320 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4001600 tx=0x7fb8e4009040 comp rx=0 tx=0).stop 2023-10-24T23:53:21.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f5444640 1 -- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00c4d70 msgr2=0x7fb8f00a8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f5444640 1 --2- 172.21.15.107:0/2837310646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8f00c4d70 0x7fb8f00a8c70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800efc0 tx=0x7fb8e800c490 comp rx=0 tx=0).stop 2023-10-24T23:53:21.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f4c43640 1 -- 172.21.15.107:0/2837310646 reap_dead start 2023-10-24T23:53:21.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f5444640 1 -- 172.21.15.107:0/2837310646 shutdown_connections 2023-10-24T23:53:21.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f5444640 1 -- 172.21.15.107:0/2837310646 >> 172.21.15.107:0/2837310646 conn(0x7fb8f00c6dc0 msgr2=0x7fb8f0154d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f5444640 1 -- 172.21.15.107:0/2837310646 shutdown_connections 2023-10-24T23:53:21.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.830+0000 7fb8f5444640 1 -- 172.21.15.107:0/2837310646 wait complete. 2023-10-24T23:53:21.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc9ac498640 1 -- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] conn(0x7fc988070d20 msgr2=0x7fc988073140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc9ac498640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] conn(0x7fc988070d20 0x7fc988073140 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc9ac498640 1 -- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc988065e60 msgr2=0x7fc988068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc9ac498640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc988065e60 0x7fc988068320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0af000 tx=0x7fc99c004ac0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc9ac498640 1 -- 172.21.15.107:0/3902407032 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9a4070ea0 msgr2=0x7fc9a40b9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc9ac498640 1 --2- 172.21.15.107:0/3902407032 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9a4070ea0 0x7fc9a40b9c50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc994012820 tx=0x7fc994012c20 comp rx=0 tx=0).stop 2023-10-24T23:53:21.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc9aaa35640 1 -- 172.21.15.107:0/3902407032 reap_dead start 2023-10-24T23:53:21.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc9ac498640 1 -- 172.21.15.107:0/3902407032 shutdown_connections 2023-10-24T23:53:21.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc9ac498640 1 -- 172.21.15.107:0/3902407032 >> 172.21.15.107:0/3902407032 conn(0x7fc9a40c2860 msgr2=0x7fc9a40beb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc9ac498640 1 -- 172.21.15.107:0/3902407032 shutdown_connections 2023-10-24T23:53:21.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc9ac498640 1 -- 172.21.15.107:0/3902407032 wait complete. 2023-10-24T23:53:21.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.838+0000 7fc48469e640 1 -- 172.21.15.107:0/3862168941 --> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc464002cc0 con 0x7fc460070da0 2023-10-24T23:53:21.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7f95e3fff640 1 -- 172.21.15.107:0/1366911094 --> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f95c4002cc0 con 0x7f95d0070cd0 2023-10-24T23:53:21.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7f95cb7fe640 1 -- 172.21.15.107:0/1366911094 <== osd.0 v2:172.21.15.107:6810/37540 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f95c4002cc0 con 0x7f95d0070cd0 2023-10-24T23:53:21.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc472ffd640 1 -- 172.21.15.107:0/3862168941 <== osd.4 v2:172.21.15.110:6801/37764 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc464002cc0 con 0x7fc460070da0 2023-10-24T23:53:21.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7f95e3fff640 1 -- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] conn(0x7f95d0070cd0 msgr2=0x7f95d00730f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc48469e640 1 -- 172.21.15.107:0/3862168941 >> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] conn(0x7fc460070da0 msgr2=0x7fc4600731c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc48469e640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] conn(0x7fc460070da0 0x7fc4600731c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc48469e640 1 -- 172.21.15.107:0/3862168941 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc460065e60 msgr2=0x7fc460068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc48469e640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc460065e60 0x7fc460068320 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc474007fd0 tx=0x7fc474009040 comp rx=0 tx=0).stop 2023-10-24T23:53:21.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc48469e640 1 -- 172.21.15.107:0/3862168941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc47c0c1b40 msgr2=0x7fc47c0705d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc48469e640 1 --2- 172.21.15.107:0/3862168941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc47c0c1b40 0x7fc47c0705d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc46c009f40 tx=0x7fc46c00ecd0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc482c3b640 1 -- 172.21.15.107:0/3862168941 reap_dead start 2023-10-24T23:53:21.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc48469e640 1 -- 172.21.15.107:0/3862168941 shutdown_connections 2023-10-24T23:53:21.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc48469e640 1 -- 172.21.15.107:0/3862168941 >> 172.21.15.107:0/3862168941 conn(0x7fc47c0c2860 msgr2=0x7fc47c0c45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc48469e640 1 -- 172.21.15.107:0/3862168941 shutdown_connections 2023-10-24T23:53:21.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7fc48469e640 1 -- 172.21.15.107:0/3862168941 wait complete. 2023-10-24T23:53:21.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7f95e3fff640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] conn(0x7f95d0070cd0 0x7f95d00730f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7f95e3fff640 1 -- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f95d0065e90 msgr2=0x7f95d0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7f95e3fff640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f95d0065e90 0x7f95d0068350 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f95d40099c0 tx=0x7f95d4009480 comp rx=0 tx=0).stop 2023-10-24T23:53:21.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.842+0000 7f95e3fff640 1 -- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95dc0d4200 msgr2=0x7f95dc0d6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f6975213640 1 -- 172.21.15.107:0/2399793531 --> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6950002cc0 con 0x7f695c070c80 2023-10-24T23:53:21.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f1eea163640 1 -- 172.21.15.107:0/49266883 --> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1ee40a9730 con 0x7f1ed0070d20 2023-10-24T23:53:21.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f95e3fff640 1 --2- 172.21.15.107:0/1366911094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95dc0d4200 0x7f95dc0d6910 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f95d801a000 tx=0x7f95d8019170 comp rx=0 tx=0).stop 2023-10-24T23:53:21.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f95e37fe640 1 -- 172.21.15.107:0/1366911094 reap_dead start 2023-10-24T23:53:21.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f95e3fff640 1 -- 172.21.15.107:0/1366911094 shutdown_connections 2023-10-24T23:53:21.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f95e3fff640 1 -- 172.21.15.107:0/1366911094 >> 172.21.15.107:0/1366911094 conn(0x7f95dc04c630 msgr2=0x7f95dc0dcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f298359e640 1 -- 172.21.15.107:0/3095230121 --> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2968005bd0 con 0x7f29680015f0 2023-10-24T23:53:21.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f95e3fff640 1 -- 172.21.15.107:0/1366911094 shutdown_connections 2023-10-24T23:53:21.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f69577fe640 1 -- 172.21.15.107:0/2399793531 <== osd.6 v2:172.21.15.110:6800/37765 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6950002cc0 con 0x7f695c070c80 2023-10-24T23:53:21.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f95e3fff640 1 -- 172.21.15.107:0/1366911094 wait complete. 2023-10-24T23:53:21.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f2972ffd640 1 -- 172.21.15.107:0/3095230121 <== osd.2 v2:172.21.15.107:6808/37539 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2968005bd0 con 0x7f29680015f0 2023-10-24T23:53:21.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f1ecbfff640 1 -- 172.21.15.107:0/49266883 <== osd.7 v2:172.21.15.110:6824/37766 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1ed800c040 con 0x7f1ed0070d20 2023-10-24T23:53:21.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f6975213640 1 -- 172.21.15.107:0/2399793531 >> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] conn(0x7f695c070c80 msgr2=0x7f695c0730a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.846+0000 7f6975213640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] conn(0x7f695c070c80 0x7f695c0730a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f2970ff9640 1 -- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] conn(0x7f29680015f0 msgr2=0x7f2968003ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f2970ff9640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] conn(0x7f29680015f0 0x7f2968003ab0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f2970ff9640 1 -- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2964087e70 msgr2=0x7f296408a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f2970ff9640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2964087e70 0x7f296408a330 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f29780af000 tx=0x7f29780a7070 comp rx=0 tx=0).stop 2023-10-24T23:53:21.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f2970ff9640 1 -- 172.21.15.107:0/3095230121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 msgr2=0x7f297c0dc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f2970ff9640 1 --2- 172.21.15.107:0/3095230121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f297c00c1d0 0x7f297c0dc010 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f297400c7d0 tx=0x7f297400cc90 comp rx=0 tx=0).stop 2023-10-24T23:53:21.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f2982d9d640 1 -- 172.21.15.107:0/3095230121 reap_dead start 2023-10-24T23:53:21.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f2970ff9640 1 -- 172.21.15.107:0/3095230121 shutdown_connections 2023-10-24T23:53:21.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f2970ff9640 1 -- 172.21.15.107:0/3095230121 >> 172.21.15.107:0/3095230121 conn(0x7f297c007c10 msgr2=0x7f297c00aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f1eea163640 1 -- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] conn(0x7f1ed0070d20 msgr2=0x7f1ed0073140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f1eea163640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] conn(0x7f1ed0070d20 0x7f1ed0073140 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f2970ff9640 1 -- 172.21.15.107:0/3095230121 shutdown_connections 2023-10-24T23:53:21.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f1eea163640 1 -- 172.21.15.107:0/49266883 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ed0065e60 msgr2=0x7f1ed0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f1eea163640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ed0065e60 0x7f1ed0068320 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1ee40c30f0 tx=0x7f1edc009690 comp rx=0 tx=0).stop 2023-10-24T23:53:21.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f2970ff9640 1 -- 172.21.15.107:0/3095230121 wait complete. 2023-10-24T23:53:21.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f6975213640 1 -- 172.21.15.107:0/2399793531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f695c065eb0 msgr2=0x7f695c068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f1eea163640 1 -- 172.21.15.107:0/49266883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee40b9470 msgr2=0x7f1ee40bb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f1eea163640 1 --2- 172.21.15.107:0/49266883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ee40b9470 0x7f1ee40bb200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1ed40ae040 tx=0x7f1ed40191f0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f1ee3fff640 1 -- 172.21.15.107:0/49266883 reap_dead start 2023-10-24T23:53:21.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f1eea163640 1 -- 172.21.15.107:0/49266883 shutdown_connections 2023-10-24T23:53:21.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f1eea163640 1 -- 172.21.15.107:0/49266883 >> 172.21.15.107:0/49266883 conn(0x7f1ee40c2860 msgr2=0x7f1ee4154f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f1eea163640 1 -- 172.21.15.107:0/49266883 shutdown_connections 2023-10-24T23:53:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f6975213640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f695c065eb0 0x7f695c068370 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f69600af000 tx=0x7f69600a7000 comp rx=0 tx=0).stop 2023-10-24T23:53:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f6975213640 1 -- 172.21.15.107:0/2399793531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69680da110 msgr2=0x7f6968178b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f1eea163640 1 -- 172.21.15.107:0/49266883 wait complete. 2023-10-24T23:53:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f6975213640 1 --2- 172.21.15.107:0/2399793531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69680da110 0x7f6968178b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f697002c850 tx=0x7f6970067a00 comp rx=0 tx=0).stop 2023-10-24T23:53:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f696f59e640 1 -- 172.21.15.107:0/2399793531 reap_dead start 2023-10-24T23:53:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f6975213640 1 -- 172.21.15.107:0/2399793531 shutdown_connections 2023-10-24T23:53:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f6975213640 1 -- 172.21.15.107:0/2399793531 >> 172.21.15.107:0/2399793531 conn(0x7f6968006330 msgr2=0x7f69680dff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f6975213640 1 -- 172.21.15.107:0/2399793531 shutdown_connections 2023-10-24T23:53:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.850+0000 7f6975213640 1 -- 172.21.15.107:0/2399793531 wait complete. 2023-10-24T23:53:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.854+0000 7fd0c9d82640 1 -- 172.21.15.107:0/995878872 --> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd0c40c44e0 con 0x7fd0b0070e70 2023-10-24T23:53:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0abfff640 1 -- 172.21.15.107:0/995878872 <== osd.5 v2:172.21.15.110:6816/37763 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd0b40ad5a0 con 0x7fd0b0070e70 2023-10-24T23:53:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0c9d82640 1 -- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] conn(0x7fd0b0070e70 msgr2=0x7fd0b0073290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0c9d82640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] conn(0x7fd0b0070e70 0x7fd0b0073290 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:21.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0c9d82640 1 -- 172.21.15.107:0/995878872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd0b0065f30 msgr2=0x7fd0b00683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0c9d82640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd0b0065f30 0x7fd0b00683f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8009860 tx=0x7fd0b8008040 comp rx=0 tx=0).stop 2023-10-24T23:53:21.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0c9d82640 1 -- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0c40c0f60 msgr2=0x7fd0c41f9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:21.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0c9d82640 1 --2- 172.21.15.107:0/995878872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0c40c0f60 0x7fd0c41f9e10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00b990 tx=0x7fd0bc00be50 comp rx=0 tx=0).stop 2023-10-24T23:53:21.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0c3fff640 1 -- 172.21.15.107:0/995878872 reap_dead start 2023-10-24T23:53:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0c9d82640 1 -- 172.21.15.107:0/995878872 shutdown_connections 2023-10-24T23:53:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0c9d82640 1 -- 172.21.15.107:0/995878872 >> 172.21.15.107:0/995878872 conn(0x7fd0c40c65e0 msgr2=0x7fd0c40c3880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0c9d82640 1 -- 172.21.15.107:0/995878872 shutdown_connections 2023-10-24T23:53:21.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:21.858+0000 7fd0c9d82640 1 -- 172.21.15.107:0/995878872 wait complete. 2023-10-24T23:53:21.980 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:21.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-24T23:53:21.981 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:21.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-24T23:53:21.983 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-10-24T23:53:21.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-24T23:53:21.984 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-10-24T23:53:21.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-24T23:53:21.985 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:21.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-24T23:53:21.986 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:21.987 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:21.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-24T23:53:21.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-24T23:53:21.989 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:21.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-24T23:53:22.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fe1cecfe640 1 Processor -- start 2023-10-24T23:53:22.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fe1cecfe640 1 -- start start 2023-10-24T23:53:22.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb0766c4640 1 Processor -- start 2023-10-24T23:53:22.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb0766c4640 1 -- start start 2023-10-24T23:53:22.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f796759e640 1 Processor -- start 2023-10-24T23:53:22.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f257a489640 1 Processor -- start 2023-10-24T23:53:22.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fe1cecfe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe1c004b280 0x7fe1c00df210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fe1cecfe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe1c0007000 0x7fe1c0007480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fe1cecfe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c00df750 con 0x7fe1c000a770 2023-10-24T23:53:22.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fe1cecfe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c0046520 con 0x7fe1c004b280 2023-10-24T23:53:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fe1cecfe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c00481f0 con 0x7fe1c0007000 2023-10-24T23:53:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f796759e640 1 -- start start 2023-10-24T23:53:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f374b257640 1 Processor -- start 2023-10-24T23:53:22.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fe1ce4fd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe1c000a770 0x7fe1c000ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55320/0 (socket says 172.21.15.107:55320) 2023-10-24T23:53:22.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fe1ce4fd640 1 -- 172.21.15.107:0/2259887481 learned_addr learned my addr 172.21.15.107:0/2259887481 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:22.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fe1cd4fb640 1 --2- 172.21.15.107:0/2259887481 >> v2:172.21.15.110:3300/0 conn(0x7fe1c0007000 0x7fe1c0007480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f796759e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f79680abdb0 0x7f79681551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f796759e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79680c7f50 0x7f796814e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f796759e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79681556e0 con 0x7f796814e8e0 2023-10-24T23:53:22.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f796759e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7968071e70 con 0x7f79680abdb0 2023-10-24T23:53:22.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f796759e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7968072110 con 0x7f79680c7f50 2023-10-24T23:53:22.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb0766c4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb0680d7cc0 0x7fb0680d49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb0766c4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb068007000 0x7fb068007480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb0766c4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb06804a1d0 0x7fb06804a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb0766c4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06802e8c0 con 0x7fb06804a1d0 2023-10-24T23:53:22.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb0766c4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06804ab20 con 0x7fb068007000 2023-10-24T23:53:22.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb0766c4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06804b6b0 con 0x7fb0680d7cc0 2023-10-24T23:53:22.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb074ec1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb068007000 0x7fb068007480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb075ec3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb06804a1d0 0x7fb06804a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb074ec1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb068007000 0x7fb068007480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54732/0 (socket says 172.21.15.107:54732) 2023-10-24T23:53:22.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7fb074ec1640 1 -- 172.21.15.107:0/357993955 learned_addr learned my addr 172.21.15.107:0/357993955 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fe1cdcfc640 1 --2- 172.21.15.107:0/2259887481 >> v2:172.21.15.107:3300/0 conn(0x7fe1c004b280 0x7fe1c00df210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f257a489640 1 -- start start 2023-10-24T23:53:22.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f257a489640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f256c04b170 0x7f256c0df100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f257a489640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f256c006ef0 0x7f256c007370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f257a489640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f256c00a660 0x7f256c00aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f257a489640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256c046410 con 0x7f256c006ef0 2023-10-24T23:53:22.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f257a489640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256c0480e0 con 0x7f256c00a660 2023-10-24T23:53:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f257a489640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256c0df640 con 0x7f256c04b170 2023-10-24T23:53:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fe1cdcfc640 1 -- 172.21.15.107:0/2259887481 >> v2:172.21.15.110:3300/0 conn(0x7fe1c0007000 msgr2=0x7fe1c0007480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f374b257640 1 -- start start 2023-10-24T23:53:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f374b257640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f37440aaa30 0x7f37440aae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f796659c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f79680abdb0 0x7f79681551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f795ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79680c7f50 0x7f796814e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f795ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79680c7f50 0x7f796814e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:51398/0 (socket says 172.21.15.107:51398) 2023-10-24T23:53:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f795ffff640 1 -- 172.21.15.107:0/832990890 learned_addr learned my addr 172.21.15.107:0/832990890 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f374b257640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37440a9060 0x7f37440c1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f374b257640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37440721e0 con 0x7f37440a7280 2023-10-24T23:53:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f374b257640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37440724c0 con 0x7f37440aaa30 2023-10-24T23:53:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f374b257640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3744151430 con 0x7f37440a9060 2023-10-24T23:53:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f2578a26640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f256c00a660 0x7f256c00aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f2578a26640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f256c00a660 0x7f256c00aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54746/0 (socket says 172.21.15.107:54746) 2023-10-24T23:53:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f2578a26640 1 -- 172.21.15.107:0/658387120 learned_addr learned my addr 172.21.15.107:0/658387120 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f2573fff640 1 --2- 172.21.15.107:0/658387120 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f256c04b170 0x7f256c0df100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f3743fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37440a9060 0x7f37440c1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f5974fac640 1 Processor -- start 2023-10-24T23:53:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.194+0000 7f5974fac640 1 -- start start 2023-10-24T23:53:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f5974fac640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 0x7f59680f94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f5974fac640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f59680fc200 0x7f5968102970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f5974fac640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5968006ed0 con 0x7f59680fbde0 2023-10-24T23:53:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f5974fac640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596802e410 con 0x7f59680fc200 2023-10-24T23:53:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f5974fac640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596802e220 con 0x7f59680f5730 2023-10-24T23:53:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37497f4640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f37440a7280 0x7f37440bb520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55334/0 (socket says 172.21.15.107:55334) 2023-10-24T23:53:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37497f4640 1 -- 172.21.15.107:0/2003818824 learned_addr learned my addr 172.21.15.107:0/2003818824 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fb075ec3640 1 -- 172.21.15.107:0/357993955 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb0680d7cc0 msgr2=0x7fb0680d49c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-24T23:53:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fb075ec3640 1 --2- 172.21.15.107:0/357993955 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb0680d7cc0 0x7fb0680d49c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fb075ec3640 1 -- 172.21.15.107:0/357993955 >> v2:172.21.15.110:3300/0 conn(0x7fb068007000 msgr2=0x7fb068007480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fb075ec3640 1 --2- 172.21.15.107:0/357993955 >> v2:172.21.15.110:3300/0 conn(0x7fb068007000 0x7fb068007480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fb075ec3640 1 -- 172.21.15.107:0/357993955 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb068049d40 con 0x7fb06804a1d0 2023-10-24T23:53:22.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f596f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 0x7f59680f94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f25737fe640 1 --2- 172.21.15.107:0/658387120 >> v2:172.21.15.107:3300/0 conn(0x7f256c006ef0 0x7f256c007370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f596f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 0x7f59680f94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:51426/0 (socket says 172.21.15.107:51426) 2023-10-24T23:53:22.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f596f7fe640 1 -- 172.21.15.107:0/422801772 learned_addr learned my addr 172.21.15.107:0/422801772 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f795ffff640 1 -- 172.21.15.107:0/832990890 >> v2:172.21.15.110:3300/0 conn(0x7f79680abdb0 msgr2=0x7f79681551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f795ffff640 1 --2- 172.21.15.107:0/832990890 >> v2:172.21.15.110:3300/0 conn(0x7f79680abdb0 0x7f79681551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f795ffff640 1 -- 172.21.15.107:0/832990890 >> v1:172.21.15.110:6789/0 conn(0x7f796814e8e0 legacy=0x7f7968157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f795ffff640 1 -- 172.21.15.107:0/832990890 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79680c4a70 con 0x7f79680c7f50 2023-10-24T23:53:22.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777363895 0 0) 0x7f37440721e0 con 0x7f37440a7280 2023-10-24T23:53:22.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3730003530 con 0x7f37440a7280 2023-10-24T23:53:22.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f25737fe640 1 -- 172.21.15.107:0/658387120 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f256c04b170 msgr2=0x7f256c0df100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f25737fe640 1 --2- 172.21.15.107:0/658387120 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f256c04b170 0x7f256c0df100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f25737fe640 1 -- 172.21.15.107:0/658387120 >> v2:172.21.15.110:3300/0 conn(0x7f256c00a660 msgr2=0x7f256c00aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f25737fe640 1 --2- 172.21.15.107:0/658387120 >> v2:172.21.15.110:3300/0 conn(0x7f256c00a660 0x7f256c00aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f25737fe640 1 -- 172.21.15.107:0/658387120 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256c00a1d0 con 0x7f256c006ef0 2023-10-24T23:53:22.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f2578a26640 1 --2- 172.21.15.107:0/658387120 >> v2:172.21.15.110:3300/0 conn(0x7f256c00a660 0x7f256c00aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:53:22.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f3748ff3640 1 --2- 172.21.15.107:0/2003818824 >> v2:172.21.15.110:3300/0 conn(0x7f37440aaa30 0x7f37440aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 34321517 0 0) 0x7f3730003530 con 0x7f37440a7280 2023-10-24T23:53:22.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37440a9060 msgr2=0x7f37440c1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 --2- 172.21.15.107:0/2003818824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37440a9060 0x7f37440c1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f596ffff640 1 --2- 172.21.15.107:0/422801772 >> v2:172.21.15.110:3300/0 conn(0x7f59680fc200 0x7f5968102970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f596f7fe640 1 -- 172.21.15.107:0/422801772 >> v2:172.21.15.110:3300/0 conn(0x7f59680fc200 msgr2=0x7f5968102970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f596f7fe640 1 --2- 172.21.15.107:0/422801772 >> v2:172.21.15.110:3300/0 conn(0x7f59680fc200 0x7f5968102970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f596f7fe640 1 -- 172.21.15.107:0/422801772 >> v1:172.21.15.110:6789/0 conn(0x7f59680fbde0 legacy=0x7f59680462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fe1cdcfc640 1 --2- 172.21.15.107:0/2259887481 >> v2:172.21.15.110:3300/0 conn(0x7fe1c0007000 0x7fe1c0007480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fe1cdcfc640 1 -- 172.21.15.107:0/2259887481 >> v1:172.21.15.107:6789/0 conn(0x7fe1c000a770 legacy=0x7fe1c000ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fe1cdcfc640 1 -- 172.21.15.107:0/2259887481 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c000a2e0 con 0x7fe1c004b280 2023-10-24T23:53:22.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fe1cd4fb640 1 --2- 172.21.15.107:0/2259887481 >> v2:172.21.15.110:3300/0 conn(0x7fe1c0007000 0x7fe1c0007480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:53:22.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fb075ec3640 1 --2- 172.21.15.107:0/357993955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb06804a1d0 0x7fb06804a5e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb06802b1f0 tx=0x7fb070096cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f72c6b37309e57 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 >> v2:172.21.15.110:3300/0 conn(0x7f37440aaa30 msgr2=0x7f37440aae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 --2- 172.21.15.107:0/2003818824 >> v2:172.21.15.110:3300/0 conn(0x7f37440aaa30 0x7f37440aae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3744151620 con 0x7f37440a7280 2023-10-24T23:53:22.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f3734001a00 con 0x7f37440a7280 2023-10-24T23:53:22.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 >> v1:172.21.15.107:6789/0 conn(0x7f37440a7280 legacy=0x7f37440bb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 --2- 172.21.15.107:0/2003818824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3730003980 0x7f3730003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 --2- 172.21.15.107:0/2003818824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37300047d0 0x7f3730004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 --2- 172.21.15.107:0/2003818824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3730005150 0x7f3730014590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37440724c0 con 0x7f3730005150 2023-10-24T23:53:22.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3730004630 con 0x7f3730003980 2023-10-24T23:53:22.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3730009a00 con 0x7f37300047d0 2023-10-24T23:53:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37497f4640 1 --2- 172.21.15.107:0/2003818824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3730005150 0x7f3730014590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f3743fff640 1 --2- 172.21.15.107:0/2003818824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37300047d0 0x7f3730004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37497f4640 1 -- 172.21.15.107:0/2003818824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37300047d0 msgr2=0x7f3730004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37497f4640 1 --2- 172.21.15.107:0/2003818824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37300047d0 0x7f3730004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37497f4640 1 -- 172.21.15.107:0/2003818824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3730003980 msgr2=0x7f3730003da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-24T23:53:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37497f4640 1 --2- 172.21.15.107:0/2003818824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3730003980 0x7f3730003da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f37497f4640 1 -- 172.21.15.107:0/2003818824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3730003530 con 0x7f3730005150 2023-10-24T23:53:22.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fe1cdcfc640 1 --2- 172.21.15.107:0/2259887481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1c004b280 0x7fe1c00df210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0007aa0 tx=0x7fe1c401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37aa343bafee89be server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f596f7fe640 1 -- 172.21.15.107:0/422801772 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59680071b0 con 0x7f59680f5730 2023-10-24T23:53:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7f596ffff640 1 --2- 172.21.15.107:0/422801772 >> v2:172.21.15.110:3300/0 conn(0x7f59680fc200 0x7f5968102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:22.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fb067fff640 1 -- 172.21.15.107:0/357993955 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07011f970 con 0x7fb06804a1d0 2023-10-24T23:53:22.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fb067fff640 1 -- 172.21.15.107:0/357993955 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb07011fb10 con 0x7fb06804a1d0 2023-10-24T23:53:22.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fb067fff640 1 -- 172.21.15.107:0/357993955 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070127440 con 0x7fb06804a1d0 2023-10-24T23:53:22.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.198+0000 7fbb271fb640 1 Processor -- start 2023-10-24T23:53:22.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb271fb640 1 -- start start 2023-10-24T23:53:22.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f3748ff3640 1 --2- 172.21.15.107:0/2003818824 >> v2:172.21.15.110:3300/0 conn(0x7f37440aaa30 0x7f37440aae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:22.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 -- 172.21.15.107:0/357993955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb06804a1d0 msgr2=0x7fb06804a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 --2- 172.21.15.107:0/357993955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb06804a1d0 0x7fb06804a5e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb06802b1f0 tx=0x7fb070096cd0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f795ffff640 1 --2- 172.21.15.107:0/832990890 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79680c7f50 0x7f796814e3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7950009870 tx=0x7f795001c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d362efcebb5ce38 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 -- 172.21.15.107:0/357993955 shutdown_connections 2023-10-24T23:53:22.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 --2- 172.21.15.107:0/357993955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb06804a1d0 0x7fb06804a5e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 --2- 172.21.15.107:0/357993955 >> v2:172.21.15.110:3300/0 conn(0x7fb068007000 0x7fb068007480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 --2- 172.21.15.107:0/357993955 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb0680d7cc0 0x7fb0680d49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 -- 172.21.15.107:0/357993955 >> 172.21.15.107:0/357993955 conn(0x7fb0680483b0 msgr2=0x7fb0680487c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fe1cccfa640 1 -- 172.21.15.107:0/2259887481 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c40ab600 con 0x7fe1c004b280 2023-10-24T23:53:22.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fe1cccfa640 1 -- 172.21.15.107:0/2259887481 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c40a7050 con 0x7fe1c004b280 2023-10-24T23:53:22.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fe1cccfa640 1 -- 172.21.15.107:0/2259887481 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c40ab9c0 con 0x7fe1c004b280 2023-10-24T23:53:22.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 -- 172.21.15.107:0/357993955 shutdown_connections 2023-10-24T23:53:22.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb271fb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb200ab050 0x7fbb200bb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb271fb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbb2006d340 0x7fbb200c0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb271fb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb200bf840 con 0x7fbb200bb970 2023-10-24T23:53:22.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb271fb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb200bc070 con 0x7fbb2006d340 2023-10-24T23:53:22.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb271fb640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb200bf560 con 0x7fbb200ab050 2023-10-24T23:53:22.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f7965d9b640 1 -- 172.21.15.107:0/832990890 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79500ac640 con 0x7f79680c7f50 2023-10-24T23:53:22.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f7965d9b640 1 -- 172.21.15.107:0/832990890 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79500ac7a0 con 0x7f79680c7f50 2023-10-24T23:53:22.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f7965d9b640 1 -- 172.21.15.107:0/832990890 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79500acb30 con 0x7f79680c7f50 2023-10-24T23:53:22.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb24f97640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb200ab050 0x7fbb200bb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb24f97640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb200ab050 0x7fbb200bb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54766/0 (socket says 172.21.15.107:54766) 2023-10-24T23:53:22.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb24f97640 1 -- 172.21.15.107:0/59210284 learned_addr learned my addr 172.21.15.107:0/59210284 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb1ffff640 1 --2- 172.21.15.107:0/59210284 >> v2:172.21.15.107:3300/0 conn(0x7fbb2006d340 0x7fbb200c0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f596f7fe640 1 --2- 172.21.15.107:0/422801772 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 0x7f59680f94e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f597002c850 tx=0x7f5970086fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3c8d2a79bdb501c server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fe1cecfe640 1 -- 172.21.15.107:0/2259887481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1c004b280 msgr2=0x7fe1c00df210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fe1cecfe640 1 --2- 172.21.15.107:0/2259887481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1c004b280 0x7fe1c00df210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0007aa0 tx=0x7fe1c401c3f0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb1ffff640 1 -- 172.21.15.107:0/59210284 >> v2:172.21.15.110:3300/0 conn(0x7fbb200ab050 msgr2=0x7fbb200bb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb1ffff640 1 --2- 172.21.15.107:0/59210284 >> v2:172.21.15.110:3300/0 conn(0x7fbb200ab050 0x7fbb200bb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb1ffff640 1 -- 172.21.15.107:0/59210284 >> v1:172.21.15.110:6789/0 conn(0x7fbb200bb970 legacy=0x7fbb200c0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb1ffff640 1 -- 172.21.15.107:0/59210284 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb200c1ec0 con 0x7fbb2006d340 2023-10-24T23:53:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fbb24f97640 1 --2- 172.21.15.107:0/59210284 >> v2:172.21.15.110:3300/0 conn(0x7fbb200ab050 0x7fbb200bb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:53:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f596e7fc640 1 -- 172.21.15.107:0/422801772 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597010fea0 con 0x7f59680f5730 2023-10-24T23:53:22.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f596e7fc640 1 -- 172.21.15.107:0/422801772 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5970117440 con 0x7f59680f5730 2023-10-24T23:53:22.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 -- 172.21.15.107:0/357993955 wait complete. 2023-10-24T23:53:22.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 Processor -- start 2023-10-24T23:53:22.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fe1cecfe640 1 -- 172.21.15.107:0/2259887481 shutdown_connections 2023-10-24T23:53:22.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fe1cecfe640 1 --2- 172.21.15.107:0/2259887481 >> v2:172.21.15.110:3300/0 conn(0x7fe1c0007000 0x7fe1c0007480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fe1cecfe640 1 --2- 172.21.15.107:0/2259887481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1c004b280 0x7fe1c00df210 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fe1cecfe640 1 -- 172.21.15.107:0/2259887481 >> 172.21.15.107:0/2259887481 conn(0x7fe1c00483b0 msgr2=0x7fe1c00487c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fe1cecfe640 1 -- 172.21.15.107:0/2259887481 shutdown_connections 2023-10-24T23:53:22.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f37497f4640 1 --2- 172.21.15.107:0/2003818824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3730005150 0x7f3730014590 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f373401b8b0 tx=0x7f3734030b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2d456e9b9ef020 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fe1cecfe640 1 -- 172.21.15.107:0/2259887481 wait complete. 2023-10-24T23:53:22.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f796759e640 1 -- 172.21.15.107:0/832990890 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79680c7f50 msgr2=0x7f796814e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f796759e640 1 --2- 172.21.15.107:0/832990890 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79680c7f50 0x7f796814e3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7950009870 tx=0x7f795001c3b0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f796759e640 1 -- 172.21.15.107:0/832990890 shutdown_connections 2023-10-24T23:53:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f796759e640 1 --2- 172.21.15.107:0/832990890 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79680c7f50 0x7f796814e3a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f796759e640 1 --2- 172.21.15.107:0/832990890 >> v2:172.21.15.110:3300/0 conn(0x7f79680abdb0 0x7f79681551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f796759e640 1 -- 172.21.15.107:0/832990890 >> 172.21.15.107:0/832990890 conn(0x7f79680ca050 msgr2=0x7f79680ca460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f796759e640 1 -- 172.21.15.107:0/832990890 shutdown_connections 2023-10-24T23:53:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f596e7fc640 1 -- 172.21.15.107:0/422801772 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970117870 con 0x7f59680f5730 2023-10-24T23:53:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f5974fac640 1 -- 172.21.15.107:0/422801772 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 msgr2=0x7f59680f94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f5974fac640 1 --2- 172.21.15.107:0/422801772 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 0x7f59680f94e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f597002c850 tx=0x7f5970086fe0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 -- start start 2023-10-24T23:53:22.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb068007000 0x7fb068042a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0766c4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb06804a1d0 0x7fb0680dc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f796759e640 1 -- 172.21.15.107:0/832990890 wait complete. 2023-10-24T23:53:22.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0756c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb068007000 0x7fb068042a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7fb0756c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb068007000 0x7fb068042a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50634/0 (socket says 172.21.15.107:50634) 2023-10-24T23:53:22.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0756c2640 1 -- 172.21.15.107:0/1908640600 learned_addr learned my addr 172.21.15.107:0/1908640600 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37340ba860 con 0x7f3730005150 2023-10-24T23:53:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37340ba9c0 con 0x7f3730005150 2023-10-24T23:53:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f37437fe640 1 -- 172.21.15.107:0/2003818824 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37340bad70 con 0x7f3730005150 2023-10-24T23:53:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f374b257640 1 -- 172.21.15.107:0/2003818824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3730005150 msgr2=0x7f3730014590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f374b257640 1 --2- 172.21.15.107:0/2003818824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3730005150 0x7f3730014590 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f373401b8b0 tx=0x7f3734030b50 comp rx=0 tx=0).stop 2023-10-24T23:53:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f37497f4640 1 -- 172.21.15.107:0/2003818824 reap_dead start 2023-10-24T23:53:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 -- 172.21.15.107:0/422801772 shutdown_connections 2023-10-24T23:53:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 --2- 172.21.15.107:0/422801772 >> v2:172.21.15.110:3300/0 conn(0x7f59680fc200 0x7f5968102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 --2- 172.21.15.107:0/422801772 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 0x7f59680f94e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f374b257640 1 -- 172.21.15.107:0/2003818824 shutdown_connections 2023-10-24T23:53:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.202+0000 7f374b257640 1 -- 172.21.15.107:0/2003818824 >> 172.21.15.107:0/2003818824 conn(0x7f37440a5750 msgr2=0x7f37440a5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 -- 172.21.15.107:0/422801772 >> 172.21.15.107:0/422801772 conn(0x7f5968005ff0 msgr2=0x7f5968006400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cecfe640 1 Processor -- start 2023-10-24T23:53:22.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 -- 172.21.15.107:0/422801772 shutdown_connections 2023-10-24T23:53:22.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 -- 172.21.15.107:0/422801772 wait complete. 2023-10-24T23:53:22.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb074ec1640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb06804a1d0 0x7fb0680dc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f796759e640 1 Processor -- start 2023-10-24T23:53:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fbb1ffff640 1 --2- 172.21.15.107:0/59210284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb2006d340 0x7fbb200c0510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbb200676e0 tx=0x7fbb1801c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bb79524f480641 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f796759e640 1 -- start start 2023-10-24T23:53:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f374b257640 1 -- 172.21.15.107:0/2003818824 shutdown_connections 2023-10-24T23:53:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f25737fe640 1 --2- 172.21.15.107:0/658387120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256c006ef0 0x7f256c007370 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f256c02d2e0 tx=0x7f256001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b089f7eea1c378b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fbb1f7fe640 1 -- 172.21.15.107:0/59210284 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb180a6710 con 0x7fbb2006d340 2023-10-24T23:53:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fbb1f7fe640 1 -- 172.21.15.107:0/59210284 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb180a6870 con 0x7fbb2006d340 2023-10-24T23:53:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fbb1f7fe640 1 -- 172.21.15.107:0/59210284 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb180a6c00 con 0x7fbb2006d340 2023-10-24T23:53:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0766c4640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0680d7cc0 0x7fb0680e42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cecfe640 1 -- start start 2023-10-24T23:53:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cecfe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1c0007000 0x7fe1c00d9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cecfe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1c000a770 0x7fe1c017a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f2572ffd640 1 -- 172.21.15.107:0/658387120 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25600ab600 con 0x7f256c006ef0 2023-10-24T23:53:22.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f2572ffd640 1 -- 172.21.15.107:0/658387120 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25600a7050 con 0x7f256c006ef0 2023-10-24T23:53:22.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f2572ffd640 1 -- 172.21.15.107:0/658387120 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25600ab980 con 0x7f256c006ef0 2023-10-24T23:53:22.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cecfe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1c004b280 0x7fe1c017a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cecfe640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c00d52b0 con 0x7fe1c004b280 2023-10-24T23:53:22.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cecfe640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c01777b0 con 0x7fe1c000a770 2023-10-24T23:53:22.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cecfe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0176e80 con 0x7fe1c0007000 2023-10-24T23:53:22.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f796759e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79680abdb0 0x7f79681fcc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f796759e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79680c7f50 0x7f79680af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f796759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f796814e8e0 0x7f7968149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f796759e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79680beef0 con 0x7f796814e8e0 2023-10-24T23:53:22.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f796759e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79680bfc60 con 0x7f79680c7f50 2023-10-24T23:53:22.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f796759e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79680bff30 con 0x7f79680abdb0 2023-10-24T23:53:22.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f7966d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f796814e8e0 0x7f7968149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f7966d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f796814e8e0 0x7f7968149570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50652/0 (socket says 172.21.15.107:50652) 2023-10-24T23:53:22.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f7966d9d640 1 -- 172.21.15.107:0/2674697265 learned_addr learned my addr 172.21.15.107:0/2674697265 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f795ffff640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79680c7f50 0x7f79680af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1ce4fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1c004b280 0x7fe1c017a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1ce4fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1c004b280 0x7fe1c017a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50638/0 (socket says 172.21.15.107:50638) 2023-10-24T23:53:22.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1ce4fd640 1 -- 172.21.15.107:0/264586092 learned_addr learned my addr 172.21.15.107:0/264586092 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cd4fb640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1c000a770 0x7fe1c017a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cdcfc640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1c0007000 0x7fe1c00d9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1ce4fd640 1 -- 172.21.15.107:0/264586092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1c0007000 msgr2=0x7fe1c00d9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1ce4fd640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1c0007000 0x7fe1c00d9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1ce4fd640 1 -- 172.21.15.107:0/264586092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1c000a770 msgr2=0x7fe1c017a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1ce4fd640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1c000a770 0x7fe1c017a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1ce4fd640 1 -- 172.21.15.107:0/264586092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c804e5c0 con 0x7fe1c004b280 2023-10-24T23:53:22.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0766c4640 1 -- 172.21.15.107:0/1908640600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680d70c0 con 0x7fb068007000 2023-10-24T23:53:22.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0766c4640 1 -- 172.21.15.107:0/1908640600 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680c8ce0 con 0x7fb0680d7cc0 2023-10-24T23:53:22.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0766c4640 1 -- 172.21.15.107:0/1908640600 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680da070 con 0x7fb06804a1d0 2023-10-24T23:53:22.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb075ec3640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0680d7cc0 0x7fb0680e42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0756c2640 1 -- 172.21.15.107:0/1908640600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb06804a1d0 msgr2=0x7fb0680dc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0756c2640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb06804a1d0 0x7fb0680dc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0756c2640 1 -- 172.21.15.107:0/1908640600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0680d7cc0 msgr2=0x7fb0680e42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0756c2640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0680d7cc0 0x7fb0680e42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0756c2640 1 -- 172.21.15.107:0/1908640600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb07004e5c0 con 0x7fb068007000 2023-10-24T23:53:22.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f374b257640 1 -- 172.21.15.107:0/2003818824 wait complete. 2023-10-24T23:53:22.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f374b257640 1 Processor -- start 2023-10-24T23:53:22.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 Processor -- start 2023-10-24T23:53:22.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 -- start start 2023-10-24T23:53:22.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f7966d9d640 1 -- 172.21.15.107:0/2674697265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79680abdb0 msgr2=0x7f79681fcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f7966d9d640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79680abdb0 0x7f79681fcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f7966d9d640 1 -- 172.21.15.107:0/2674697265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79680c7f50 msgr2=0x7f79680af4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f7966d9d640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79680c7f50 0x7f79680af4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f7966d9d640 1 -- 172.21.15.107:0/2674697265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79500094d0 con 0x7f796814e8e0 2023-10-24T23:53:22.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f257a489640 1 -- 172.21.15.107:0/658387120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256c006ef0 msgr2=0x7f256c007370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f257a489640 1 --2- 172.21.15.107:0/658387120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256c006ef0 0x7f256c007370 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f256c02d2e0 tx=0x7f256001c3d0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f257a489640 1 -- 172.21.15.107:0/658387120 shutdown_connections 2023-10-24T23:53:22.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f257a489640 1 --2- 172.21.15.107:0/658387120 >> v2:172.21.15.110:3300/0 conn(0x7f256c00a660 0x7f256c00aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f257a489640 1 --2- 172.21.15.107:0/658387120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256c006ef0 0x7f256c007370 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f257a489640 1 --2- 172.21.15.107:0/658387120 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f256c04b170 0x7f256c0df100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f257a489640 1 -- 172.21.15.107:0/658387120 >> 172.21.15.107:0/658387120 conn(0x7f256c0482a0 msgr2=0x7f256c0486b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0756c2640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb068007000 0x7fb068042a00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb06c00b3f0 tx=0x7fb06c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 0x7f59680ea9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59680fbde0 0x7f5968195140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f7966d9d640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f796814e8e0 0x7f7968149570 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f796000c780 tx=0x7f796000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59680fc200 0x7f5968195680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59680f7060 con 0x7f59680fc200 2023-10-24T23:53:22.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59680ef810 con 0x7f59680fbde0 2023-10-24T23:53:22.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f5974fac640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59680f6850 con 0x7f59680f5730 2023-10-24T23:53:22.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb065ffb640 1 -- 172.21.15.107:0/1908640600 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c011830 con 0x7fb068007000 2023-10-24T23:53:22.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f374b257640 1 -- start start 2023-10-24T23:53:22.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f374b257640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3730005150 0x7f374414cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f374b257640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37440be890 0x7f37440bed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f596f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 0x7f59680ea9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f374b257640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37441e6610 0x7f37441e6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f374b257640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37440bf250 con 0x7f37441e6610 2023-10-24T23:53:22.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f374b257640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3744152930 con 0x7f37440be890 2023-10-24T23:53:22.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f374b257640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3744152300 con 0x7f3730005150 2023-10-24T23:53:22.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1ce4fd640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1c004b280 0x7fe1c017a5e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe1b800c370 tx=0x7fe1b800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb065ffb640 1 -- 172.21.15.107:0/1908640600 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06c0119b0 con 0x7fb068007000 2023-10-24T23:53:22.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb065ffb640 1 -- 172.21.15.107:0/1908640600 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c011d60 con 0x7fb068007000 2023-10-24T23:53:22.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f37497f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37441e6610 0x7f37441e6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f3748ff3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3730005150 0x7f374414cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f37497f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37441e6610 0x7f37441e6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50674/0 (socket says 172.21.15.107:50674) 2023-10-24T23:53:22.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f37497f4640 1 -- 172.21.15.107:0/1707062136 learned_addr learned my addr 172.21.15.107:0/1707062136 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f596ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59680fc200 0x7f5968195680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f596f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 0x7f59680ea9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:51450/0 (socket says 172.21.15.107:51450) 2023-10-24T23:53:22.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f596f7fe640 1 -- 172.21.15.107:0/1251271118 learned_addr learned my addr 172.21.15.107:0/1251271118 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1be7fc640 1 -- 172.21.15.107:0/264586092 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8005cf0 con 0x7fe1c004b280 2023-10-24T23:53:22.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f596effd640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59680fbde0 0x7f5968195140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1be7fc640 1 -- 172.21.15.107:0/264586092 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b8005e50 con 0x7fe1c004b280 2023-10-24T23:53:22.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1be7fc640 1 -- 172.21.15.107:0/264586092 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b80072d0 con 0x7fe1c004b280 2023-10-24T23:53:22.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f257a489640 1 -- 172.21.15.107:0/658387120 shutdown_connections 2023-10-24T23:53:22.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f3743fff640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37440be890 0x7f37440bed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cecfe640 1 -- 172.21.15.107:0/264586092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c4009420 con 0x7fe1c004b280 2023-10-24T23:53:22.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0766c4640 1 -- 172.21.15.107:0/1908640600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0680d6de0 con 0x7fb068007000 2023-10-24T23:53:22.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fe1cecfe640 1 -- 172.21.15.107:0/264586092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c0177440 con 0x7fe1c004b280 2023-10-24T23:53:22.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7fb0766c4640 1 -- 172.21.15.107:0/1908640600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0680dd0f0 con 0x7fb068007000 2023-10-24T23:53:22.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f257a489640 1 -- 172.21.15.107:0/658387120 wait complete. 2023-10-24T23:53:22.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f3748ff3640 1 -- 172.21.15.107:0/1707062136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37440be890 msgr2=0x7f37440bed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f3748ff3640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37440be890 0x7f37440bed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f3748ff3640 1 -- 172.21.15.107:0/1707062136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37441e6610 msgr2=0x7f37441e6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.206+0000 7f3748ff3640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37441e6610 0x7f37441e6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f3748ff3640 1 -- 172.21.15.107:0/1707062136 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3744154a20 con 0x7f3730005150 2023-10-24T23:53:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f3748ff3640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3730005150 0x7f374414cce0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f373c00d960 tx=0x7f373c00de20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f596effd640 1 -- 172.21.15.107:0/1251271118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 msgr2=0x7f59680ea9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f596effd640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 0x7f59680ea9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f596effd640 1 -- 172.21.15.107:0/1251271118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59680fc200 msgr2=0x7f5968195680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f596effd640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59680fc200 0x7f5968195680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f596effd640 1 -- 172.21.15.107:0/1251271118 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597004e5c0 con 0x7f59680fbde0 2023-10-24T23:53:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 -- 172.21.15.107:0/59210284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb2006d340 msgr2=0x7fbb200c0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 --2- 172.21.15.107:0/59210284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb2006d340 0x7fbb200c0510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbb200676e0 tx=0x7fbb1801c7c0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f37417fa640 1 -- 172.21.15.107:0/1707062136 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c00ab80 con 0x7f3730005150 2023-10-24T23:53:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f37417fa640 1 -- 172.21.15.107:0/1707062136 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373c00ace0 con 0x7f3730005150 2023-10-24T23:53:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f37417fa640 1 -- 172.21.15.107:0/1707062136 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c010670 con 0x7f3730005150 2023-10-24T23:53:22.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 -- 172.21.15.107:0/59210284 shutdown_connections 2023-10-24T23:53:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 --2- 172.21.15.107:0/59210284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb2006d340 0x7fbb200c0510 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 --2- 172.21.15.107:0/59210284 >> v2:172.21.15.110:3300/0 conn(0x7fbb200ab050 0x7fbb200bb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 -- 172.21.15.107:0/59210284 >> 172.21.15.107:0/59210284 conn(0x7fbb200a5750 msgr2=0x7fbb200a5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f596effd640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59680fbde0 0x7f5968195140 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f596400c3e0 tx=0x7f596400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 -- 172.21.15.107:0/59210284 shutdown_connections 2023-10-24T23:53:22.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 -- 172.21.15.107:0/59210284 wait complete. 2023-10-24T23:53:22.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f595bfff640 1 -- 172.21.15.107:0/1251271118 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964011790 con 0x7f59680fbde0 2023-10-24T23:53:22.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f595bfff640 1 -- 172.21.15.107:0/1251271118 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59640118f0 con 0x7f59680fbde0 2023-10-24T23:53:22.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f5974fac640 1 -- 172.21.15.107:0/1251271118 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59680f6b30 con 0x7f59680fbde0 2023-10-24T23:53:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f595bfff640 1 -- 172.21.15.107:0/1251271118 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964011ce0 con 0x7f59680fbde0 2023-10-24T23:53:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f257a489640 1 Processor -- start 2023-10-24T23:53:22.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f5974fac640 1 -- 172.21.15.107:0/1251271118 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59680f6200 con 0x7f59680fbde0 2023-10-24T23:53:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 Processor -- start 2023-10-24T23:53:22.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 -- start start 2023-10-24T23:53:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f374b257640 1 -- 172.21.15.107:0/1707062136 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3744154d00 con 0x7f3730005150 2023-10-24T23:53:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f374b257640 1 -- 172.21.15.107:0/1707062136 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3744153af0 con 0x7f3730005150 2023-10-24T23:53:22.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fe1be7fc640 1 -- 172.21.15.107:0/264586092 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe1b8007430 con 0x7fe1c004b280 2023-10-24T23:53:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fe1cecfe640 1 -- 172.21.15.107:0/264586092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c000b270 con 0x7fe1c004b280 2023-10-24T23:53:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fe1be7fc640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1ac065f60 0x7fe1ac068420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fe1be7fc640 1 -- 172.21.15.107:0/264586092 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fe1b8082e20 con 0x7fe1c004b280 2023-10-24T23:53:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb200ab050 0x7fbb201ecbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fbb271fb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb200bb970 0x7fbb2006c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb271fb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb20156f30 0x7fbb2015ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb24f97640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb200ab050 0x7fbb201ecbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb24f97640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb200ab050 0x7fbb201ecbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54820/0 (socket says 172.21.15.107:54820) 2023-10-24T23:53:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb24f97640 1 -- 172.21.15.107:0/1933385386 learned_addr learned my addr 172.21.15.107:0/1933385386 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb1ffff640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb200bb970 0x7fbb2006c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb25798640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb20156f30 0x7fbb2015ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2006f790 con 0x7fbb20156f30 2023-10-24T23:53:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb201ee030 con 0x7fbb200ab050 2023-10-24T23:53:22.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb201ed1a0 con 0x7fbb200bb970 2023-10-24T23:53:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f087ac5c640 1 Processor -- start 2023-10-24T23:53:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f087ac5c640 1 -- start start 2023-10-24T23:53:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f257a489640 1 -- start start 2023-10-24T23:53:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f257a489640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f256c006ef0 0x7f256c0cef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f257a489640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256c00a660 0x7f256c044910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f257a489640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f256c04b170 0x7f256c044e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f257a489640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c040100 con 0x7f256c00a660 2023-10-24T23:53:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f257a489640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c0d80c0 con 0x7f256c006ef0 2023-10-24T23:53:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f257a489640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f256c0cc430 con 0x7f256c04b170 2023-10-24T23:53:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fb065ffb640 1 -- 172.21.15.107:0/1908640600 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb06c01a400 con 0x7fb068007000 2023-10-24T23:53:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fb0537fe640 1 -- 172.21.15.107:0/1908640600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06802ed40 con 0x7fb068007000 2023-10-24T23:53:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fb065ffb640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb054065e60 0x7fb054068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fb065ffb640 1 -- 172.21.15.107:0/1908640600 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fb06c083bc0 con 0x7fb068007000 2023-10-24T23:53:22.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7fb074ec1640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb054065e60 0x7fb054068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f5974fac640 1 -- 172.21.15.107:0/1251271118 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5950005260 con 0x7f59680fbde0 2023-10-24T23:53:22.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f595bfff640 1 -- 172.21.15.107:0/1251271118 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f596401c460 con 0x7f59680fbde0 2023-10-24T23:53:22.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f2573fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f256c006ef0 0x7f256c0cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f2578a26640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f256c04b170 0x7f256c044e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f2573fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f256c006ef0 0x7f256c0cef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54836/0 (socket says 172.21.15.107:54836) 2023-10-24T23:53:22.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f2573fff640 1 -- 172.21.15.107:0/1438479120 learned_addr learned my addr 172.21.15.107:0/1438479120 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fe1cdcfc640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1ac065f60 0x7fe1ac068420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f795effd640 1 -- 172.21.15.107:0/2674697265 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7960004b70 con 0x7f796814e8e0 2023-10-24T23:53:22.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f595bfff640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5954065e90 0x7f5954068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f25737fe640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256c00a660 0x7f256c044910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f087ac5c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0874156580 0x7f08740c7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f795effd640 1 -- 172.21.15.107:0/2674697265 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796000ce90 con 0x7f796814e8e0 2023-10-24T23:53:22.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f796759e640 1 -- 172.21.15.107:0/2674697265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79681fd720 con 0x7f796814e8e0 2023-10-24T23:53:22.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f087ac5c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08740c6030 0x7f08740c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f595bfff640 1 -- 172.21.15.107:0/1251271118 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f5964083810 con 0x7f59680fbde0 2023-10-24T23:53:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f087ac5c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f08740c33b0 0x7f0874071930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f596f7fe640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5954065e90 0x7f5954068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f087ac5c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0874070b90 con 0x7f08740c33b0 2023-10-24T23:53:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f08789f8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0874156580 0x7f08740c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f08789f8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0874156580 0x7f08740c7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54842/0 (socket says 172.21.15.107:54842) 2023-10-24T23:53:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f08789f8640 1 -- 172.21.15.107:0/883994946 learned_addr learned my addr 172.21.15.107:0/883994946 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f0873fff640 1 --2- 172.21.15.107:0/883994946 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08740c6030 0x7f08740c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f087ac5c640 1 -- 172.21.15.107:0/883994946 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0874156960 con 0x7f0874156580 2023-10-24T23:53:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f087ac5c640 1 -- 172.21.15.107:0/883994946 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08740c7970 con 0x7f08740c6030 2023-10-24T23:53:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f2573fff640 1 -- 172.21.15.107:0/1438479120 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f256c04b170 msgr2=0x7f256c044e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f2573fff640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f256c04b170 0x7f256c044e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f2573fff640 1 -- 172.21.15.107:0/1438479120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256c00a660 msgr2=0x7f256c044910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f2573fff640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256c00a660 0x7f256c044910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f2573fff640 1 -- 172.21.15.107:0/1438479120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257404e5c0 con 0x7f256c006ef0 2023-10-24T23:53:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fb074ec1640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb054065e60 0x7fb054068320 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb068005b70 tx=0x7fb060009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f596f7fe640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5954065e90 0x7f5954068350 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f597004eea0 tx=0x7f5970111070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f2573fff640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f256c006ef0 0x7f256c0cef20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f256800e930 tx=0x7f256800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb24f97640 1 -- 172.21.15.107:0/1933385386 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb200bb970 msgr2=0x7fbb2006c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb24f97640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb200bb970 0x7fbb2006c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb24f97640 1 -- 172.21.15.107:0/1933385386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb20156f30 msgr2=0x7fbb2015ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb24f97640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb20156f30 0x7fbb2015ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb24f97640 1 -- 172.21.15.107:0/1933385386 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb100094d0 con 0x7fbb200ab050 2023-10-24T23:53:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f795effd640 1 -- 172.21.15.107:0/2674697265 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796000a7f0 con 0x7f796814e8e0 2023-10-24T23:53:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f796759e640 1 -- 172.21.15.107:0/2674697265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79681545e0 con 0x7f796814e8e0 2023-10-24T23:53:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f796759e640 1 -- 172.21.15.107:0/2674697265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796804e050 con 0x7f796814e8e0 2023-10-24T23:53:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fe1cdcfc640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1ac065f60 0x7fe1ac068420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40ad000 tx=0x7fe1c4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.210+0000 7f374b257640 1 -- 172.21.15.107:0/1707062136 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37440c1590 con 0x7f3730005150 2023-10-24T23:53:22.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f37417fa640 1 -- 172.21.15.107:0/1707062136 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f373c0107d0 con 0x7f3730005150 2023-10-24T23:53:22.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7fbb24f97640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb200ab050 0x7fbb201ecbc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000e9d0 tx=0x7fbb1000ee90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f08789f8640 1 -- 172.21.15.107:0/883994946 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08740c6030 msgr2=0x7f08740c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f08789f8640 1 --2- 172.21.15.107:0/883994946 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08740c6030 0x7f08740c2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f08789f8640 1 -- 172.21.15.107:0/883994946 >> v2:172.21.15.107:3300/0 conn(0x7f08740c33b0 msgr2=0x7f0874071930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f08789f8640 1 --2- 172.21.15.107:0/883994946 >> v2:172.21.15.107:3300/0 conn(0x7f08740c33b0 0x7f0874071930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f08789f8640 1 -- 172.21.15.107:0/883994946 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0874070e50 con 0x7f0874156580 2023-10-24T23:53:22.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f37417fa640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3724065e90 0x7f3724068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f37417fa640 1 -- 172.21.15.107:0/1707062136 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f373c0843d0 con 0x7f3730005150 2023-10-24T23:53:22.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f3743fff640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3724065e90 0x7f3724068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.214+0000 7f3743fff640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3724065e90 0x7f3724068350 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f37440bf600 tx=0x7f372c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.218+0000 7f08789f8640 1 --2- 172.21.15.107:0/883994946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0874156580 0x7f08740c7390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f08740673e0 tx=0x7f08680208e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e01444fea52de50d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.218+0000 7fb065ffb640 1 -- 172.21.15.107:0/1908640600 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb06c050f00 con 0x7fb068007000 2023-10-24T23:53:22.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.222+0000 7fe1be7fc640 1 -- 172.21.15.107:0/264586092 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe1c000b270 con 0x7fe1c004b280 2023-10-24T23:53:22.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f795effd640 1 -- 172.21.15.107:0/2674697265 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7960006b60 con 0x7f796814e8e0 2023-10-24T23:53:22.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f795effd640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7948065f30 0x7f79480683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f795effd640 1 -- 172.21.15.107:0/2674697265 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f79600848f0 con 0x7f796814e8e0 2023-10-24T23:53:22.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f796659c640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7948065f30 0x7f79480683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f796659c640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7948065f30 0x7f79480683f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7958006aa0 tx=0x7f7958006a30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f37417fa640 1 -- 172.21.15.107:0/1707062136 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f373c051760 con 0x7f3730005150 2023-10-24T23:53:22.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f2570ff9640 1 -- 172.21.15.107:0/1438479120 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568009690 con 0x7f256c006ef0 2023-10-24T23:53:22.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f257a489640 1 -- 172.21.15.107:0/1438479120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2560009420 con 0x7f256c006ef0 2023-10-24T23:53:22.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f595bfff640 1 -- 172.21.15.107:0/1251271118 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5964050c70 con 0x7f59680fbde0 2023-10-24T23:53:22.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f2570ff9640 1 -- 172.21.15.107:0/1438479120 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256800ca00 con 0x7f256c006ef0 2023-10-24T23:53:22.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f257a489640 1 -- 172.21.15.107:0/1438479120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256c040500 con 0x7f256c006ef0 2023-10-24T23:53:22.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7fbb1d7fa640 1 -- 172.21.15.107:0/1933385386 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1000ca60 con 0x7fbb200ab050 2023-10-24T23:53:22.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7f2570ff9640 1 -- 172.21.15.107:0/1438479120 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256800cd70 con 0x7f256c006ef0 2023-10-24T23:53:22.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb18009a70 con 0x7fbb200ab050 2023-10-24T23:53:22.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb201ed760 con 0x7fbb200ab050 2023-10-24T23:53:22.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.226+0000 7fbb1d7fa640 1 -- 172.21.15.107:0/1933385386 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1000cbc0 con 0x7fbb200ab050 2023-10-24T23:53:22.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7fbb1d7fa640 1 -- 172.21.15.107:0/1933385386 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb10013470 con 0x7fbb200ab050 2023-10-24T23:53:22.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f257a489640 1 -- 172.21.15.107:0/1438479120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2554005260 con 0x7f256c006ef0 2023-10-24T23:53:22.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb00005260 con 0x7fbb200ab050 2023-10-24T23:53:22.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f795effd640 1 -- 172.21.15.107:0/2674697265 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7960051bf0 con 0x7f796814e8e0 2023-10-24T23:53:22.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f08737fe640 1 -- 172.21.15.107:0/883994946 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08680a97e0 con 0x7f0874156580 2023-10-24T23:53:22.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f08737fe640 1 -- 172.21.15.107:0/883994946 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08680a9940 con 0x7f0874156580 2023-10-24T23:53:22.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f08737fe640 1 -- 172.21.15.107:0/883994946 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08680a9cd0 con 0x7f0874156580 2023-10-24T23:53:22.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f2570ff9640 1 -- 172.21.15.107:0/1438479120 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f25680059a0 con 0x7f256c006ef0 2023-10-24T23:53:22.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f2570ff9640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2558065e60 0x7f2558068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f2570ff9640 1 -- 172.21.15.107:0/1438479120 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f2568084160 con 0x7f256c006ef0 2023-10-24T23:53:22.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f25737fe640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2558065e60 0x7f2558068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7fbb1d7fa640 1 -- 172.21.15.107:0/1933385386 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbb100135d0 con 0x7fbb200ab050 2023-10-24T23:53:22.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f25737fe640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2558065e60 0x7f2558068320 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25600ad000 tx=0x7f25600a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f087ac5c640 1 -- 172.21.15.107:0/883994946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0874156580 msgr2=0x7f08740c7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f087ac5c640 1 --2- 172.21.15.107:0/883994946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0874156580 0x7f08740c7390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f08740673e0 tx=0x7f08680208e0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f087ac5c640 1 -- 172.21.15.107:0/883994946 shutdown_connections 2023-10-24T23:53:22.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f087ac5c640 1 --2- 172.21.15.107:0/883994946 >> v2:172.21.15.107:3300/0 conn(0x7f08740c33b0 0x7f0874071930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f087ac5c640 1 --2- 172.21.15.107:0/883994946 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08740c6030 0x7f08740c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f087ac5c640 1 --2- 172.21.15.107:0/883994946 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0874156580 0x7f08740c7390 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f087ac5c640 1 -- 172.21.15.107:0/883994946 >> 172.21.15.107:0/883994946 conn(0x7f08740c7c70 msgr2=0x7f08740c8080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f087ac5c640 1 -- 172.21.15.107:0/883994946 shutdown_connections 2023-10-24T23:53:22.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7fbb1d7fa640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb0c065f60 0x7fbb0c068420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7fbb1d7fa640 1 -- 172.21.15.107:0/1933385386 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fbb100844d0 con 0x7fbb200ab050 2023-10-24T23:53:22.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7fbb1ffff640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb0c065f60 0x7fbb0c068420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7fbb1ffff640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb0c065f60 0x7fbb0c068420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb180af000 tx=0x7fbb180a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.230+0000 7f087ac5c640 1 -- 172.21.15.107:0/883994946 wait complete. 2023-10-24T23:53:22.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f087ac5c640 1 Processor -- start 2023-10-24T23:53:22.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f087ac5c640 1 -- start start 2023-10-24T23:53:22.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f087ac5c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08740c33b0 0x7f08740bfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f087ac5c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08740c6030 0x7f087406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f087ac5c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0874156580 0x7f08741fc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f08791f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0874156580 0x7f08741fc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f0873fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08740c6030 0x7f087406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f08791f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0874156580 0x7f08741fc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50710/0 (socket says 172.21.15.107:50710) 2023-10-24T23:53:22.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f0873fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08740c6030 0x7f087406ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:51508/0 (socket says 172.21.15.107:51508) 2023-10-24T23:53:22.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f08789f8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08740c33b0 0x7f08740bfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f087ac5c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08740a7040 con 0x7f0874156580 2023-10-24T23:53:22.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f087ac5c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087406e810 con 0x7f08740c33b0 2023-10-24T23:53:22.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f087ac5c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08740c11d0 con 0x7f08740c6030 2023-10-24T23:53:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f08791f9640 1 -- 172.21.15.107:0/3129912463 learned_addr learned my addr 172.21.15.107:0/3129912463 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f08791f9640 1 -- 172.21.15.107:0/3129912463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08740c6030 msgr2=0x7f087406ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f08791f9640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08740c6030 0x7f087406ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f08791f9640 1 -- 172.21.15.107:0/3129912463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08740c33b0 msgr2=0x7f08740bfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f08791f9640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08740c33b0 0x7f08740bfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f08791f9640 1 -- 172.21.15.107:0/3129912463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08680094d0 con 0x7f0874156580 2023-10-24T23:53:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.234+0000 7f08791f9640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0874156580 0x7f08741fc420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f086c00be30 tx=0x7f086c00bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.238+0000 7f08717fa640 1 -- 172.21.15.107:0/3129912463 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c00ce10 con 0x7f0874156580 2023-10-24T23:53:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.238+0000 7f08717fa640 1 -- 172.21.15.107:0/3129912463 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086c014440 con 0x7f0874156580 2023-10-24T23:53:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.238+0000 7f087ac5c640 1 -- 172.21.15.107:0/3129912463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08740c18d0 con 0x7f0874156580 2023-10-24T23:53:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.238+0000 7f087ac5c640 1 -- 172.21.15.107:0/3129912463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08740a6d60 con 0x7f0874156580 2023-10-24T23:53:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.238+0000 7f08717fa640 1 -- 172.21.15.107:0/3129912463 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c0147f0 con 0x7f0874156580 2023-10-24T23:53:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.238+0000 7f087ac5c640 1 -- 172.21.15.107:0/3129912463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087406e9b0 con 0x7f0874156580 2023-10-24T23:53:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.238+0000 7f08717fa640 1 -- 172.21.15.107:0/3129912463 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f086c014950 con 0x7f0874156580 2023-10-24T23:53:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.238+0000 7f08717fa640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0858065e60 0x7f0858068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.238+0000 7f08717fa640 1 -- 172.21.15.107:0/3129912463 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f086c083ee0 con 0x7f0874156580 2023-10-24T23:53:22.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.242+0000 7f08789f8640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0858065e60 0x7f0858068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:22.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.242+0000 7f08789f8640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0858065e60 0x7f0858068320 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f08680b3000 tx=0x7f08680ab070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:22.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.242+0000 7f2570ff9640 1 -- 172.21.15.107:0/1438479120 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2568051510 con 0x7f256c006ef0 2023-10-24T23:53:22.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.250+0000 7f08717fa640 1 -- 172.21.15.107:0/3129912463 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f086c0b3040 con 0x7f0874156580 2023-10-24T23:53:22.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.254+0000 7fbb1d7fa640 1 -- 172.21.15.107:0/1933385386 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbb10051910 con 0x7fbb200ab050 2023-10-24T23:53:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.514+0000 7f5974fac640 1 -- 172.21.15.107:0/1251271118 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f59500057e0 con 0x7f59680fbde0 2023-10-24T23:53:22.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.514+0000 7f595bfff640 1 -- 172.21.15.107:0/1251271118 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5964054290 con 0x7f59680fbde0 2023-10-24T23:53:22.520 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-10-24T23:53:22.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.514+0000 7f5959ffb640 1 -- 172.21.15.107:0/1251271118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5954065e90 msgr2=0x7f5954068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.518+0000 7f5959ffb640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5954065e90 0x7f5954068350 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f597004eea0 tx=0x7f5970111070 comp rx=0 tx=0).stop 2023-10-24T23:53:22.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.518+0000 7f5959ffb640 1 -- 172.21.15.107:0/1251271118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59680fbde0 msgr2=0x7f5968195140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.518+0000 7f5959ffb640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59680fbde0 0x7f5968195140 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f596400c3e0 tx=0x7f596400c8a0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.518+0000 7f5959ffb640 1 -- 172.21.15.107:0/1251271118 shutdown_connections 2023-10-24T23:53:22.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.518+0000 7f5959ffb640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59680fc200 0x7f5968195680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.518+0000 7f5959ffb640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59680fbde0 0x7f5968195140 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.518+0000 7f5959ffb640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5954065e90 0x7f5954068350 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.518+0000 7f5959ffb640 1 --2- 172.21.15.107:0/1251271118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59680f5730 0x7f59680ea9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.518+0000 7f5959ffb640 1 -- 172.21.15.107:0/1251271118 >> 172.21.15.107:0/1251271118 conn(0x7f5968005ff0 msgr2=0x7f59680f04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.518+0000 7f5959ffb640 1 -- 172.21.15.107:0/1251271118 shutdown_connections 2023-10-24T23:53:22.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.518+0000 7f5959ffb640 1 -- 172.21.15.107:0/1251271118 wait complete. 2023-10-24T23:53:22.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.534+0000 7fb0537fe640 1 -- 172.21.15.107:0/1908640600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb0680d9ed0 con 0x7fb068007000 2023-10-24T23:53:22.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.534+0000 7fb065ffb640 1 -- 172.21.15.107:0/1908640600 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb0680d9ed0 con 0x7fb068007000 2023-10-24T23:53:22.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.534+0000 7fe1cecfe640 1 -- 172.21.15.107:0/264586092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe1c00df750 con 0x7fe1c004b280 2023-10-24T23:53:22.539 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-10-24T23:53:22.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.534+0000 7fe1be7fc640 1 -- 172.21.15.107:0/264586092 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe1b8017020 con 0x7fe1c004b280 2023-10-24T23:53:22.540 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-10-24T23:53:22.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fb0537fe640 1 -- 172.21.15.107:0/1908640600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb054065e60 msgr2=0x7fb054068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fb0537fe640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb054065e60 0x7fb054068320 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb068005b70 tx=0x7fb060009240 comp rx=0 tx=0).stop 2023-10-24T23:53:22.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fb0537fe640 1 -- 172.21.15.107:0/1908640600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb068007000 msgr2=0x7fb068042a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fb0537fe640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb068007000 0x7fb068042a00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb06c00b3f0 tx=0x7fb06c00b8b0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 -- 172.21.15.107:0/264586092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1ac065f60 msgr2=0x7fe1ac068420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1ac065f60 0x7fe1ac068420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe1c40ad000 tx=0x7fe1c4008040 comp rx=0 tx=0).stop 2023-10-24T23:53:22.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 -- 172.21.15.107:0/264586092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1c004b280 msgr2=0x7fe1c017a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1c004b280 0x7fe1c017a5e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe1b800c370 tx=0x7fe1b800c830 comp rx=0 tx=0).stop 2023-10-24T23:53:22.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 -- 172.21.15.107:0/264586092 shutdown_connections 2023-10-24T23:53:22.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1c004b280 0x7fe1c017a5e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1c000a770 0x7fe1c017a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1ac065f60 0x7fe1ac068420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 --2- 172.21.15.107:0/264586092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1c0007000 0x7fe1c00d9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 -- 172.21.15.107:0/264586092 >> 172.21.15.107:0/264586092 conn(0x7fe1c00483b0 msgr2=0x7fe1c002b280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 -- 172.21.15.107:0/264586092 shutdown_connections 2023-10-24T23:53:22.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.538+0000 7fe1cecfe640 1 -- 172.21.15.107:0/264586092 wait complete. 2023-10-24T23:53:22.546 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.5 2023-10-24T23:53:22.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7fb0537fe640 1 -- 172.21.15.107:0/1908640600 shutdown_connections 2023-10-24T23:53:22.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7fb0537fe640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0680d7cc0 0x7fb0680e42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7fb0537fe640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb054065e60 0x7fb054068320 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7fb0537fe640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb06804a1d0 0x7fb0680dc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7fb0537fe640 1 --2- 172.21.15.107:0/1908640600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb068007000 0x7fb068042a00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7fb0537fe640 1 -- 172.21.15.107:0/1908640600 >> 172.21.15.107:0/1908640600 conn(0x7fb0680483b0 msgr2=0x7fb0680d37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7fb0537fe640 1 -- 172.21.15.107:0/1908640600 shutdown_connections 2023-10-24T23:53:22.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7fb0537fe640 1 -- 172.21.15.107:0/1908640600 wait complete. 2023-10-24T23:53:22.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7f796759e640 1 -- 172.21.15.107:0/2674697265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f79680c0070 con 0x7f796814e8e0 2023-10-24T23:53:22.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7f795effd640 1 -- 172.21.15.107:0/2674697265 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7960055210 con 0x7f796814e8e0 2023-10-24T23:53:22.551 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-10-24T23:53:22.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7f374b257640 1 -- 172.21.15.107:0/1707062136 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f374404e050 con 0x7f3730005150 2023-10-24T23:53:22.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.542+0000 7f37417fa640 1 -- 172.21.15.107:0/1707062136 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f373c054d80 con 0x7f3730005150 2023-10-24T23:53:22.553 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-10-24T23:53:22.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 -- 172.21.15.107:0/2674697265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7948065f30 msgr2=0x7f79480683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7948065f30 0x7f79480683f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7958006aa0 tx=0x7f7958006a30 comp rx=0 tx=0).stop 2023-10-24T23:53:22.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 -- 172.21.15.107:0/2674697265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f796814e8e0 msgr2=0x7f7968149570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f796814e8e0 0x7f7968149570 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f796000c780 tx=0x7f796000cc40 comp rx=0 tx=0).stop 2023-10-24T23:53:22.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 -- 172.21.15.107:0/2674697265 shutdown_connections 2023-10-24T23:53:22.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f796814e8e0 0x7f7968149570 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79680c7f50 0x7f79680af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7948065f30 0x7f79480683f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 --2- 172.21.15.107:0/2674697265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79680abdb0 0x7f79681fcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 -- 172.21.15.107:0/2674697265 >> 172.21.15.107:0/2674697265 conn(0x7f79680ca050 msgr2=0x7f79680c6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 -- 172.21.15.107:0/2674697265 shutdown_connections 2023-10-24T23:53:22.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f796759e640 1 -- 172.21.15.107:0/2674697265 wait complete. 2023-10-24T23:53:22.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 -- 172.21.15.107:0/1707062136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3724065e90 msgr2=0x7f3724068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3724065e90 0x7f3724068350 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f37440bf600 tx=0x7f372c008040 comp rx=0 tx=0).stop 2023-10-24T23:53:22.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 -- 172.21.15.107:0/1707062136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3730005150 msgr2=0x7f374414cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3730005150 0x7f374414cce0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f373c00d960 tx=0x7f373c00de20 comp rx=0 tx=0).stop 2023-10-24T23:53:22.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 -- 172.21.15.107:0/1707062136 shutdown_connections 2023-10-24T23:53:22.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37441e6610 0x7f37441e6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37440be890 0x7f37440bed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3724065e90 0x7f3724068350 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 --2- 172.21.15.107:0/1707062136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3730005150 0x7f374414cce0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 -- 172.21.15.107:0/1707062136 >> 172.21.15.107:0/1707062136 conn(0x7f37440a5750 msgr2=0x7f37440c1790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 -- 172.21.15.107:0/1707062136 shutdown_connections 2023-10-24T23:53:22.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.546+0000 7f374b257640 1 -- 172.21.15.107:0/1707062136 wait complete. 2023-10-24T23:53:22.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.554+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fbb000057e0 con 0x7fbb200ab050 2023-10-24T23:53:22.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.554+0000 7fbb1d7fa640 1 -- 172.21.15.107:0/1933385386 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbb10017020 con 0x7fbb200ab050 2023-10-24T23:53:22.560 INFO:teuthology.orchestra.run.smithi107.stdout:55834574855 2023-10-24T23:53:22.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.554+0000 7f257a489640 1 -- 172.21.15.107:0/1438479120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f25540057e0 con 0x7f256c006ef0 2023-10-24T23:53:22.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7f2570ff9640 1 -- 172.21.15.107:0/1438479120 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2568021020 con 0x7f256c006ef0 2023-10-24T23:53:22.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb0c065f60 msgr2=0x7fbb0c068420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb0c065f60 0x7fbb0c068420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb180af000 tx=0x7fbb180a7000 comp rx=0 tx=0).stop 2023-10-24T23:53:22.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb200ab050 msgr2=0x7fbb201ecbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb200ab050 0x7fbb201ecbc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000e9d0 tx=0x7fbb1000ee90 comp rx=0 tx=0).stop 2023-10-24T23:53:22.562 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-10-24T23:53:22.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 shutdown_connections 2023-10-24T23:53:22.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb20156f30 0x7fbb2015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb0c065f60 0x7fbb0c068420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb200bb970 0x7fbb2006c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 --2- 172.21.15.107:0/1933385386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb200ab050 0x7fbb201ecbc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 >> 172.21.15.107:0/1933385386 conn(0x7fbb200a5750 msgr2=0x7fbb200c5130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 shutdown_connections 2023-10-24T23:53:22.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7fbb271fb640 1 -- 172.21.15.107:0/1933385386 wait complete. 2023-10-24T23:53:22.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7f087ac5c640 1 -- 172.21.15.107:0/3129912463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f087404e050 con 0x7f0874156580 2023-10-24T23:53:22.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7f08717fa640 1 -- 172.21.15.107:0/3129912463 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f087406e9b0 con 0x7f0874156580 2023-10-24T23:53:22.566 INFO:teuthology.orchestra.run.smithi107.stdout:55834574855 2023-10-24T23:53:22.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7f257a489640 1 -- 172.21.15.107:0/1438479120 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2558065e60 msgr2=0x7f2558068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7f257a489640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2558065e60 0x7f2558068320 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25600ad000 tx=0x7f25600a6000 comp rx=0 tx=0).stop 2023-10-24T23:53:22.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7f257a489640 1 -- 172.21.15.107:0/1438479120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f256c006ef0 msgr2=0x7f256c0cef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7f257a489640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f256c006ef0 0x7f256c0cef20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f256800e930 tx=0x7f256800edf0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7f257a489640 1 -- 172.21.15.107:0/1438479120 shutdown_connections 2023-10-24T23:53:22.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7f257a489640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2558065e60 0x7f2558068320 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.558+0000 7f257a489640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f256c04b170 0x7f256c044e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f257a489640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256c00a660 0x7f256c044910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f257a489640 1 --2- 172.21.15.107:0/1438479120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f256c006ef0 0x7f256c0cef20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f257a489640 1 -- 172.21.15.107:0/1438479120 >> 172.21.15.107:0/1438479120 conn(0x7f256c0482a0 msgr2=0x7f256c004990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f257a489640 1 -- 172.21.15.107:0/1438479120 shutdown_connections 2023-10-24T23:53:22.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f257a489640 1 -- 172.21.15.107:0/1438479120 wait complete. 2023-10-24T23:53:22.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 -- 172.21.15.107:0/3129912463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0858065e60 msgr2=0x7f0858068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0858065e60 0x7f0858068320 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f08680b3000 tx=0x7f08680ab070 comp rx=0 tx=0).stop 2023-10-24T23:53:22.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 -- 172.21.15.107:0/3129912463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0874156580 msgr2=0x7f08741fc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:22.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0874156580 0x7f08741fc420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f086c00be30 tx=0x7f086c00bf20 comp rx=0 tx=0).stop 2023-10-24T23:53:22.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 -- 172.21.15.107:0/3129912463 shutdown_connections 2023-10-24T23:53:22.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0874156580 0x7f08741fc420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0858065e60 0x7f0858068320 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08740c6030 0x7f087406ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 --2- 172.21.15.107:0/3129912463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08740c33b0 0x7f08740bfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:22.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 -- 172.21.15.107:0/3129912463 >> 172.21.15.107:0/3129912463 conn(0x7f08740c7c70 msgr2=0x7f08740a9050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:22.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 -- 172.21.15.107:0/3129912463 shutdown_connections 2023-10-24T23:53:22.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:22.562+0000 7f087ac5c640 1 -- 172.21.15.107:0/3129912463 wait complete. 2023-10-24T23:53:22.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.3 2023-10-24T23:53:22.578 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.7 2023-10-24T23:53:22.586 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2023-10-24T23:53:22.587 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.1 2023-10-24T23:53:22.588 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.4 2023-10-24T23:53:22.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.0 2023-10-24T23:53:22.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2023-10-24T23:53:23.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-24T23:53:23.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-24T23:53:23.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-24T23:53:23.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-24T23:53:23.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-24T23:53:23.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-24T23:53:23.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-24T23:53:23.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-24T23:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.706+0000 7fd64616b640 1 Processor -- start 2023-10-24T23:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.706+0000 7fd64616b640 1 -- start start 2023-10-24T23:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.706+0000 7fd64616b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd640159de0 0x7fd6400a8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.706+0000 7fd64616b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd640072020 0x7fd6400bbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.706+0000 7fd64616b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6400bae30 con 0x7fd64015c890 2023-10-24T23:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.706+0000 7fd64616b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6400ab830 con 0x7fd640159de0 2023-10-24T23:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.706+0000 7fd64616b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6400ab500 con 0x7fd640072020 2023-10-24T23:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.706+0000 7fd63ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd640072020 0x7fd6400bbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.706+0000 7fd63ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd640072020 0x7fd6400bbb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:49184/0 (socket says 172.21.15.107:49184) 2023-10-24T23:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.710+0000 7fd63ffff640 1 -- 172.21.15.107:0/1864458874 learned_addr learned my addr 172.21.15.107:0/1864458874 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.710+0000 7fd63effd640 1 --2- 172.21.15.107:0/1864458874 >> v2:172.21.15.107:3300/0 conn(0x7fd640159de0 0x7fd6400a8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.710+0000 7fd63ffff640 1 -- 172.21.15.107:0/1864458874 >> v2:172.21.15.107:3300/0 conn(0x7fd640159de0 msgr2=0x7fd6400a8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.710+0000 7fd63ffff640 1 --2- 172.21.15.107:0/1864458874 >> v2:172.21.15.107:3300/0 conn(0x7fd640159de0 0x7fd6400a8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.710+0000 7fd63ffff640 1 -- 172.21.15.107:0/1864458874 >> v1:172.21.15.110:6789/0 conn(0x7fd64015c890 legacy=0x7fd6401596d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-24T23:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.710+0000 7fd63ffff640 1 -- 172.21.15.107:0/1864458874 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64015d1e0 con 0x7fd640072020 2023-10-24T23:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.710+0000 7fd63effd640 1 --2- 172.21.15.107:0/1864458874 >> v2:172.21.15.107:3300/0 conn(0x7fd640159de0 0x7fd6400a8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.714+0000 7fd63ffff640 1 --2- 172.21.15.107:0/1864458874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd640072020 0x7fd6400bbb00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd6400662a0 tx=0x7fd63801c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd4789a6722160af server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.714+0000 7fd63e7fc640 1 -- 172.21.15.107:0/1864458874 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6380a56f0 con 0x7fd640072020 2023-10-24T23:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.714+0000 7fd63e7fc640 1 -- 172.21.15.107:0/1864458874 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6380a5850 con 0x7fd640072020 2023-10-24T23:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.714+0000 7fd64616b640 1 -- 172.21.15.107:0/1864458874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd640072020 msgr2=0x7fd6400bbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.714+0000 7fd64616b640 1 --2- 172.21.15.107:0/1864458874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd640072020 0x7fd6400bbb00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd6400662a0 tx=0x7fd63801c920 comp rx=0 tx=0).stop 2023-10-24T23:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.714+0000 7fd64616b640 1 -- 172.21.15.107:0/1864458874 shutdown_connections 2023-10-24T23:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.714+0000 7fd64616b640 1 --2- 172.21.15.107:0/1864458874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd640072020 0x7fd6400bbb00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.714+0000 7fd64616b640 1 --2- 172.21.15.107:0/1864458874 >> v2:172.21.15.107:3300/0 conn(0x7fd640159de0 0x7fd6400a8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.714+0000 7fd64616b640 1 -- 172.21.15.107:0/1864458874 >> 172.21.15.107:0/1864458874 conn(0x7fd6400a5750 msgr2=0x7fd6400a5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.714+0000 7fd64616b640 1 -- 172.21.15.107:0/1864458874 shutdown_connections 2023-10-24T23:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.714+0000 7fd64616b640 1 -- 172.21.15.107:0/1864458874 wait complete. 2023-10-24T23:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd64616b640 1 Processor -- start 2023-10-24T23:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd64616b640 1 -- start start 2023-10-24T23:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd64616b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd640072020 0x7fd640154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd64616b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd640159de0 0x7fd6400c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd64616b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd64015c890 0x7fd6400707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd64616b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64006ed90 con 0x7fd640159de0 2023-10-24T23:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd64616b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64006eaf0 con 0x7fd640072020 2023-10-24T23:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd64616b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64006fcd0 con 0x7fd64015c890 2023-10-24T23:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd63ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd64015c890 0x7fd6400707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd63ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd64015c890 0x7fd6400707b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:54520/0 (socket says 172.21.15.107:54520) 2023-10-24T23:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd63ffff640 1 -- 172.21.15.107:0/971851334 learned_addr learned my addr 172.21.15.107:0/971851334 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd63effd640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd640159de0 0x7fd6400c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd63ffff640 1 -- 172.21.15.107:0/971851334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd640072020 msgr2=0x7fd640154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd63ffff640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd640072020 0x7fd640154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd63ffff640 1 -- 172.21.15.107:0/971851334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd640159de0 msgr2=0x7fd6400c4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd63ffff640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd640159de0 0x7fd6400c4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd63ffff640 1 -- 172.21.15.107:0/971851334 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd638009b30 con 0x7fd64015c890 2023-10-24T23:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.718+0000 7fd63effd640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd640159de0 0x7fd6400c4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.722+0000 7fd63ffff640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd64015c890 0x7fd6400707b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd6380af040 tx=0x7fd6380196c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.722+0000 7fd62ffff640 1 -- 172.21.15.107:0/971851334 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6380a56f0 con 0x7fd64015c890 2023-10-24T23:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.722+0000 7fd64616b640 1 -- 172.21.15.107:0/971851334 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64006ffb0 con 0x7fd64015c890 2023-10-24T23:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.722+0000 7fd64616b640 1 -- 172.21.15.107:0/971851334 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6400c52b0 con 0x7fd64015c890 2023-10-24T23:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.722+0000 7fd62ffff640 1 -- 172.21.15.107:0/971851334 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd638007de0 con 0x7fd64015c890 2023-10-24T23:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.722+0000 7fd62ffff640 1 -- 172.21.15.107:0/971851334 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6380b6480 con 0x7fd64015c890 2023-10-24T23:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.722+0000 7fd62ffff640 1 -- 172.21.15.107:0/971851334 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd6380b65e0 con 0x7fd64015c890 2023-10-24T23:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.726+0000 7fd62ffff640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd624065e30 0x7fd6240682f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.726+0000 7fd62ffff640 1 -- 172.21.15.107:0/971851334 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fd63811ed40 con 0x7fd64015c890 2023-10-24T23:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.726+0000 7fd63f7fe640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd624065e30 0x7fd6240682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.726+0000 7fd64616b640 1 -- 172.21.15.107:0/971851334 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61c005260 con 0x7fd64015c890 2023-10-24T23:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.726+0000 7fd63f7fe640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd624065e30 0x7fd6240682f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd640157500 tx=0x7fd634006bb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.738+0000 7fd62ffff640 1 -- 172.21.15.107:0/971851334 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd6380ec050 con 0x7fd64015c890 2023-10-24T23:53:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.750+0000 7f2e8ca62640 1 Processor -- start 2023-10-24T23:53:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.750+0000 7f2e8ca62640 1 -- start start 2023-10-24T23:53:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e8ca62640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2e880c2860 0x7f2e8814dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e8ca62640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2e880c20d0 0x7f2e8806b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e8ca62640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e880aaac0 con 0x7f2e880a83a0 2023-10-24T23:53:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e8ca62640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e880c12d0 con 0x7f2e880c20d0 2023-10-24T23:53:23.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e8ca62640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e880c0630 con 0x7f2e880c2860 2023-10-24T23:53:23.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e87fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2e880a83a0 0x7f2e8806bdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38176/0 (socket says 172.21.15.107:38176) 2023-10-24T23:53:23.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e86ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2e880c20d0 0x7f2e8806b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e86ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2e880c20d0 0x7f2e8806b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33118/0 (socket says 172.21.15.107:33118) 2023-10-24T23:53:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e86ffd640 1 -- 172.21.15.107:0/3934556881 learned_addr learned my addr 172.21.15.107:0/3934556881 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e877fe640 1 --2- 172.21.15.107:0/3934556881 >> v2:172.21.15.110:3300/0 conn(0x7f2e880c2860 0x7f2e8814dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e86ffd640 1 -- 172.21.15.107:0/3934556881 >> v2:172.21.15.110:3300/0 conn(0x7f2e880c2860 msgr2=0x7f2e8814dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:23.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e86ffd640 1 --2- 172.21.15.107:0/3934556881 >> v2:172.21.15.110:3300/0 conn(0x7f2e880c2860 0x7f2e8814dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e86ffd640 1 -- 172.21.15.107:0/3934556881 >> v1:172.21.15.107:6789/0 conn(0x7f2e880a83a0 legacy=0x7f2e8806bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.754+0000 7f2e86ffd640 1 -- 172.21.15.107:0/3934556881 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e880c3d10 con 0x7f2e880c20d0 2023-10-24T23:53:23.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.758+0000 7f2e86ffd640 1 --2- 172.21.15.107:0/3934556881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e880c20d0 0x7f2e8806b8b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2e88069550 tx=0x7f2e8001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d94e15e1c83cb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.758+0000 7f2e867fc640 1 -- 172.21.15.107:0/3934556881 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e800a5860 con 0x7f2e880c20d0 2023-10-24T23:53:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.758+0000 7f2e867fc640 1 -- 172.21.15.107:0/3934556881 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e800a59c0 con 0x7f2e880c20d0 2023-10-24T23:53:23.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.758+0000 7f2e867fc640 1 -- 172.21.15.107:0/3934556881 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e800a5d50 con 0x7f2e880c20d0 2023-10-24T23:53:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.758+0000 7f2e8ca62640 1 -- 172.21.15.107:0/3934556881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e880c20d0 msgr2=0x7f2e8806b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.758+0000 7f2e8ca62640 1 --2- 172.21.15.107:0/3934556881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e880c20d0 0x7f2e8806b8b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2e88069550 tx=0x7f2e8001c9a0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 -- 172.21.15.107:0/3934556881 shutdown_connections 2023-10-24T23:53:23.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 --2- 172.21.15.107:0/3934556881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e880c20d0 0x7f2e8806b8b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 --2- 172.21.15.107:0/3934556881 >> v2:172.21.15.110:3300/0 conn(0x7f2e880c2860 0x7f2e8814dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 -- 172.21.15.107:0/3934556881 >> 172.21.15.107:0/3934556881 conn(0x7f2e880a7880 msgr2=0x7f2e880a7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:23.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 -- 172.21.15.107:0/3934556881 shutdown_connections 2023-10-24T23:53:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 -- 172.21.15.107:0/3934556881 wait complete. 2023-10-24T23:53:23.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 Processor -- start 2023-10-24T23:53:23.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 -- start start 2023-10-24T23:53:23.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e880a83a0 0x7f2e88154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e880c20d0 0x7f2e880a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e880c2860 0x7f2e88152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e880a4550 con 0x7f2e880a83a0 2023-10-24T23:53:23.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e881f4a60 con 0x7f2e880c20d0 2023-10-24T23:53:23.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e8ca62640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e88154c90 con 0x7f2e880c2860 2023-10-24T23:53:23.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e880a83a0 0x7f2e88154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e880a83a0 0x7f2e88154590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33132/0 (socket says 172.21.15.107:33132) 2023-10-24T23:53:23.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e877fe640 1 -- 172.21.15.107:0/968582699 learned_addr learned my addr 172.21.15.107:0/968582699 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e86ffd640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e880c20d0 0x7f2e880a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.762+0000 7f2e87fff640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e880c2860 0x7f2e88152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e86ffd640 1 -- 172.21.15.107:0/968582699 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e880c2860 msgr2=0x7f2e88152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e86ffd640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e880c2860 0x7f2e88152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e86ffd640 1 -- 172.21.15.107:0/968582699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e880a83a0 msgr2=0x7f2e88154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e86ffd640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e880a83a0 0x7f2e88154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e86ffd640 1 -- 172.21.15.107:0/968582699 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e80009b30 con 0x7f2e880c20d0 2023-10-24T23:53:23.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e87fff640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e880c2860 0x7f2e88152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:23.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e86ffd640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e880c20d0 0x7f2e880a6420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e800ab000 tx=0x7f2e80019740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e77fff640 1 -- 172.21.15.107:0/968582699 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e800b8470 con 0x7f2e880c20d0 2023-10-24T23:53:23.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e8ca62640 1 -- 172.21.15.107:0/968582699 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e881f4cb0 con 0x7f2e880c20d0 2023-10-24T23:53:23.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e8ca62640 1 -- 172.21.15.107:0/968582699 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e881f5270 con 0x7f2e880c20d0 2023-10-24T23:53:23.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e77fff640 1 -- 172.21.15.107:0/968582699 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e800b85d0 con 0x7f2e880c20d0 2023-10-24T23:53:23.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7febae5b7640 1 Processor -- start 2023-10-24T23:53:23.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e8ca62640 1 -- 172.21.15.107:0/968582699 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e880aafb0 con 0x7f2e880c20d0 2023-10-24T23:53:23.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7f2e77fff640 1 -- 172.21.15.107:0/968582699 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e800b8980 con 0x7f2e880c20d0 2023-10-24T23:53:23.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7febae5b7640 1 -- start start 2023-10-24T23:53:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7febae5b7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feba808ba10 0x7feba80c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7f2e77fff640 1 -- 172.21.15.107:0/968582699 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2e800b8ae0 con 0x7f2e880c20d0 2023-10-24T23:53:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7f2e77fff640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2e70065f70 0x7f2e70068430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7f2e77fff640 1 -- 172.21.15.107:0/968582699 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f2e8011f190 con 0x7f2e880c20d0 2023-10-24T23:53:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7f2e877fe640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2e70065f70 0x7f2e70068430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7f2e877fe640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2e70065f70 0x7f2e70068430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009b80 tx=0x7f2e78009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7febae5b7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feba80a4ec0 0x7feba80c2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7febae5b7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba80c0790 con 0x7feba80c25a0 2023-10-24T23:53:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7febae5b7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba80c0e10 con 0x7feba80a4ec0 2023-10-24T23:53:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7febae5b7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba80c2ec0 con 0x7feba808ba10 2023-10-24T23:53:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7febaddb6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feba80a4ec0 0x7feba80c2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7febaddb6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feba80a4ec0 0x7feba80c2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33134/0 (socket says 172.21.15.107:33134) 2023-10-24T23:53:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7febaddb6640 1 -- 172.21.15.107:0/4252150267 learned_addr learned my addr 172.21.15.107:0/4252150267 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.766+0000 7febacdb4640 1 --2- 172.21.15.107:0/4252150267 >> v2:172.21.15.110:3300/0 conn(0x7feba808ba10 0x7feba80c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7febaddb6640 1 -- 172.21.15.107:0/4252150267 >> v2:172.21.15.110:3300/0 conn(0x7feba808ba10 msgr2=0x7feba80c1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:23.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7febaddb6640 1 --2- 172.21.15.107:0/4252150267 >> v2:172.21.15.110:3300/0 conn(0x7feba808ba10 0x7feba80c1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7febaddb6640 1 -- 172.21.15.107:0/4252150267 >> v1:172.21.15.110:6789/0 conn(0x7feba80c25a0 legacy=0x7feba806cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:23.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7febaddb6640 1 -- 172.21.15.107:0/4252150267 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba80c59f0 con 0x7feba80a4ec0 2023-10-24T23:53:23.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7f85b960c640 1 Processor -- start 2023-10-24T23:53:23.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7f85b960c640 1 -- start start 2023-10-24T23:53:23.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7febaddb6640 1 --2- 172.21.15.107:0/4252150267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba80a4ec0 0x7feba80c2980 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feba806e370 tx=0x7feba0020d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9becd6ed449face server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7feb9ffff640 1 -- 172.21.15.107:0/4252150267 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba00a9a10 con 0x7feba80a4ec0 2023-10-24T23:53:23.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7feb9ffff640 1 -- 172.21.15.107:0/4252150267 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba00a9b70 con 0x7feba80a4ec0 2023-10-24T23:53:23.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7feb9ffff640 1 -- 172.21.15.107:0/4252150267 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba00b13f0 con 0x7feba80a4ec0 2023-10-24T23:53:23.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7f5dedc3f640 1 Processor -- start 2023-10-24T23:53:23.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.770+0000 7f5dedc3f640 1 -- start start 2023-10-24T23:53:23.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b960c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85ac0462e0 0x7f85ac0d82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b960c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f85ac006b40 0x7f85ac006fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b960c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac046a60 con 0x7f85ac00b1a0 2023-10-24T23:53:23.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b960c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ac046d40 con 0x7f85ac006b40 2023-10-24T23:53:23.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b960c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac009880 con 0x7f85ac0462e0 2023-10-24T23:53:23.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 -- 172.21.15.107:0/4252150267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba80a4ec0 msgr2=0x7feba80c2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 --2- 172.21.15.107:0/4252150267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba80a4ec0 0x7feba80c2980 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feba806e370 tx=0x7feba0020d40 comp rx=0 tx=0).stop 2023-10-24T23:53:23.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 -- 172.21.15.107:0/4252150267 shutdown_connections 2023-10-24T23:53:23.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 --2- 172.21.15.107:0/4252150267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba80a4ec0 0x7feba80c2980 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 --2- 172.21.15.107:0/4252150267 >> v2:172.21.15.110:3300/0 conn(0x7feba808ba10 0x7feba80c1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 -- 172.21.15.107:0/4252150267 >> 172.21.15.107:0/4252150267 conn(0x7feba80a7880 msgr2=0x7feba80a7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:23.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b27fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85ac0462e0 0x7f85ac0d82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5dedc3f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5de80c4f50 0x7f5de80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5dedc3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5de80c2370 0x7f5de80a5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5dedc3f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5de806b220 con 0x7f5de80c3380 2023-10-24T23:53:23.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5dedc3f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de80c1af0 con 0x7f5de80c2370 2023-10-24T23:53:23.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5dedc3f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5de806a700 con 0x7f5de80c4f50 2023-10-24T23:53:23.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b27fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85ac0462e0 0x7f85ac0d82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:54538/0 (socket says 172.21.15.107:54538) 2023-10-24T23:53:23.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b27fc640 1 -- 172.21.15.107:0/3379441647 learned_addr learned my addr 172.21.15.107:0/3379441647 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5de6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5de80c3380 0x7f5de80bceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38184/0 (socket says 172.21.15.107:38184) 2023-10-24T23:53:23.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5de6ffd640 1 -- 172.21.15.107:0/3557209506 learned_addr learned my addr 172.21.15.107:0/3557209506 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:23.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 -- 172.21.15.107:0/4252150267 shutdown_connections 2023-10-24T23:53:23.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 -- 172.21.15.107:0/4252150267 wait complete. 2023-10-24T23:53:23.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5de7fff640 1 --2- 172.21.15.107:0/3557209506 >> v2:172.21.15.107:3300/0 conn(0x7f5de80c2370 0x7f5de80a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b37fe640 1 --2- 172.21.15.107:0/3379441647 >> v2:172.21.15.110:3300/0 conn(0x7f85ac006b40 0x7f85ac006fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5de77fe640 1 --2- 172.21.15.107:0/3557209506 >> v2:172.21.15.110:3300/0 conn(0x7f5de80c4f50 0x7f5de80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b27fc640 1 -- 172.21.15.107:0/3379441647 >> v2:172.21.15.110:3300/0 conn(0x7f85ac006b40 msgr2=0x7f85ac006fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:23.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b27fc640 1 --2- 172.21.15.107:0/3379441647 >> v2:172.21.15.110:3300/0 conn(0x7f85ac006b40 0x7f85ac006fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b27fc640 1 -- 172.21.15.107:0/3379441647 >> v1:172.21.15.110:6789/0 conn(0x7f85ac00b1a0 legacy=0x7f85ac0d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:23.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f85b27fc640 1 -- 172.21.15.107:0/3379441647 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac0d87f0 con 0x7f85ac0462e0 2023-10-24T23:53:23.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5de77fe640 1 -- 172.21.15.107:0/3557209506 >> v2:172.21.15.107:3300/0 conn(0x7f5de80c2370 msgr2=0x7f5de80a5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:23.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5de77fe640 1 --2- 172.21.15.107:0/3557209506 >> v2:172.21.15.107:3300/0 conn(0x7f5de80c2370 0x7f5de80a5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5de77fe640 1 -- 172.21.15.107:0/3557209506 >> v1:172.21.15.107:6789/0 conn(0x7f5de80c3380 legacy=0x7f5de80bceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:23.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7f5de77fe640 1 -- 172.21.15.107:0/3557209506 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de806a8f0 con 0x7f5de80c4f50 2023-10-24T23:53:23.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 Processor -- start 2023-10-24T23:53:23.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 -- start start 2023-10-24T23:53:23.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feba808ba10 0x7feba8087c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feba80a4ec0 0x7feba815f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba80c25a0 0x7feba815ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba8160d80 con 0x7feba80c25a0 2023-10-24T23:53:23.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba806cdd0 con 0x7feba80a4ec0 2023-10-24T23:53:23.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febae5b7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba815f4e0 con 0x7feba808ba10 2023-10-24T23:53:23.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febaddb6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba80c25a0 0x7feba815ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.774+0000 7febaddb6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba80c25a0 0x7feba815ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33144/0 (socket says 172.21.15.107:33144) 2023-10-24T23:53:23.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febaddb6640 1 -- 172.21.15.107:0/443612190 learned_addr learned my addr 172.21.15.107:0/443612190 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febad5b5640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feba808ba10 0x7feba8087c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febacdb4640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feba80a4ec0 0x7feba815f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febaddb6640 1 -- 172.21.15.107:0/443612190 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feba808ba10 msgr2=0x7feba8087c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febaddb6640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feba808ba10 0x7feba8087c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febaddb6640 1 -- 172.21.15.107:0/443612190 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feba80a4ec0 msgr2=0x7feba815f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febaddb6640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feba80a4ec0 0x7feba815f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febaddb6640 1 -- 172.21.15.107:0/443612190 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba000d040 con 0x7feba80c25a0 2023-10-24T23:53:23.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7f5de77fe640 1 --2- 172.21.15.107:0/3557209506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5de80c4f50 0x7f5de80a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5de806e8e0 tx=0x7f5de001c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7e1c6cfb002b1ef server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febaddb6640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba80c25a0 0x7feba815ff20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7feba00b3040 tx=0x7feba001dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7f85b27fc640 1 --2- 172.21.15.107:0/3379441647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85ac0462e0 0x7f85ac0d82b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f85a4009870 tx=0x7f85a401d730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60aba207c5885812 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7feb9dffb640 1 -- 172.21.15.107:0/443612190 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba00a9a10 con 0x7feba80c25a0 2023-10-24T23:53:23.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7feb9dffb640 1 -- 172.21.15.107:0/443612190 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba0007de0 con 0x7feba80c25a0 2023-10-24T23:53:23.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febae5b7640 1 -- 172.21.15.107:0/443612190 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba81fa790 con 0x7feba80c25a0 2023-10-24T23:53:23.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febae5b7640 1 -- 172.21.15.107:0/443612190 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba81612b0 con 0x7feba80c25a0 2023-10-24T23:53:23.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7f85b1ffb640 1 -- 172.21.15.107:0/3379441647 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a40aa5f0 con 0x7f85ac0462e0 2023-10-24T23:53:23.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7f85b1ffb640 1 -- 172.21.15.107:0/3379441647 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a40aa750 con 0x7f85ac0462e0 2023-10-24T23:53:23.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7f85b1ffb640 1 -- 172.21.15.107:0/3379441647 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a40aaae0 con 0x7f85ac0462e0 2023-10-24T23:53:23.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7f85b960c640 1 -- 172.21.15.107:0/3379441647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85ac0462e0 msgr2=0x7f85ac0d82b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7f85b960c640 1 --2- 172.21.15.107:0/3379441647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85ac0462e0 0x7f85ac0d82b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f85a4009870 tx=0x7f85a401d730 comp rx=0 tx=0).stop 2023-10-24T23:53:23.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7febae5b7640 1 -- 172.21.15.107:0/443612190 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb88005260 con 0x7feba80c25a0 2023-10-24T23:53:23.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7feb9dffb640 1 -- 172.21.15.107:0/443612190 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba00bc690 con 0x7feba80c25a0 2023-10-24T23:53:23.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7f04c5cdd640 1 Processor -- start 2023-10-24T23:53:23.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7f04c5cdd640 1 -- start start 2023-10-24T23:53:23.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.778+0000 7feb9dffb640 1 -- 172.21.15.107:0/443612190 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7feba00bc7f0 con 0x7feba80c25a0 2023-10-24T23:53:23.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7feb9dffb640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb8c065eb0 0x7feb8c068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7feb9dffb640 1 -- 172.21.15.107:0/443612190 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7feba0123ab0 con 0x7feba80c25a0 2023-10-24T23:53:23.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7febad5b5640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb8c065eb0 0x7feb8c068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 -- 172.21.15.107:0/3379441647 shutdown_connections 2023-10-24T23:53:23.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 --2- 172.21.15.107:0/3379441647 >> v2:172.21.15.110:3300/0 conn(0x7f85ac006b40 0x7f85ac006fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 --2- 172.21.15.107:0/3379441647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85ac0462e0 0x7f85ac0d82b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 -- 172.21.15.107:0/3379441647 >> 172.21.15.107:0/3379441647 conn(0x7f85ac006020 msgr2=0x7f85ac006430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:23.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 -- 172.21.15.107:0/3379441647 shutdown_connections 2023-10-24T23:53:23.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04c5cdd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04c0152120 0x7f04c0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 -- 172.21.15.107:0/3379441647 wait complete. 2023-10-24T23:53:23.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7febad5b5640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb8c065eb0 0x7feb8c068370 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7feb98004e30 tx=0x7feb98009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04c5cdd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f04c00c7890 0x7f04c00c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04bf7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04c0152120 0x7f04c0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04bf7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f04c0152120 0x7f04c0158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33150/0 (socket says 172.21.15.107:33150) 2023-10-24T23:53:23.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04bf7fe640 1 -- 172.21.15.107:0/3026521405 learned_addr learned my addr 172.21.15.107:0/3026521405 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04c5cdd640 1 -- 172.21.15.107:0/3026521405 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c0158650 con 0x7f04c00c8350 2023-10-24T23:53:23.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04c5cdd640 1 -- 172.21.15.107:0/3026521405 --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c0154b20 con 0x7f04c0152120 2023-10-24T23:53:23.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04c5cdd640 1 -- 172.21.15.107:0/3026521405 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c0155a10 con 0x7f04c00c7890 2023-10-24T23:53:23.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04bffff640 1 --2- 172.21.15.107:0/3026521405 >> v2:172.21.15.110:3300/0 conn(0x7f04c00c7890 0x7f04c00c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04bf7fe640 1 -- 172.21.15.107:0/3026521405 >> v2:172.21.15.110:3300/0 conn(0x7f04c00c7890 msgr2=0x7f04c00c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:23.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04bf7fe640 1 --2- 172.21.15.107:0/3026521405 >> v2:172.21.15.110:3300/0 conn(0x7f04c00c7890 0x7f04c00c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04bf7fe640 1 -- 172.21.15.107:0/3026521405 >> v1:172.21.15.110:6789/0 conn(0x7f04c00c8350 legacy=0x7f04c0155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f04bf7fe640 1 -- 172.21.15.107:0/3026521405 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c01588e0 con 0x7f04c0152120 2023-10-24T23:53:23.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f2e77fff640 1 -- 172.21.15.107:0/968582699 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2e800ec520 con 0x7f2e880c20d0 2023-10-24T23:53:23.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 Processor -- start 2023-10-24T23:53:23.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 -- start start 2023-10-24T23:53:23.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85ac006b40 0x7f85ac16f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85ac00b1a0 0x7f85ac0d57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85ac0462e0 0x7f85ac0cab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac16f850 con 0x7f85ac0462e0 2023-10-24T23:53:23.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0d0b30 con 0x7f85ac006b40 2023-10-24T23:53:23.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b960c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0d0e40 con 0x7f85ac00b1a0 2023-10-24T23:53:23.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85ac0462e0 0x7f85ac0cab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85ac0462e0 0x7f85ac0cab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33156/0 (socket says 172.21.15.107:33156) 2023-10-24T23:53:23.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b37fe640 1 -- 172.21.15.107:0/2548352092 learned_addr learned my addr 172.21.15.107:0/2548352092 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b2ffd640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85ac006b40 0x7f85ac16f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b37fe640 1 -- 172.21.15.107:0/2548352092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85ac00b1a0 msgr2=0x7f85ac0d57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b37fe640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85ac00b1a0 0x7f85ac0d57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b37fe640 1 -- 172.21.15.107:0/2548352092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85ac006b40 msgr2=0x7f85ac16f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b37fe640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85ac006b40 0x7f85ac16f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.782+0000 7f85b37fe640 1 -- 172.21.15.107:0/2548352092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a40094d0 con 0x7f85ac0462e0 2023-10-24T23:53:23.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f85b37fe640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85ac0462e0 0x7f85ac0cab60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f85b407a6e0 tx=0x7f85b404ea80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f04bf7fe640 1 --2- 172.21.15.107:0/3026521405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c0152120 0x7f04c0158110 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f04c00bd960 tx=0x7f04b401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766c024a0066491b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f5de67fc640 1 -- 172.21.15.107:0/3557209506 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de00a57f0 con 0x7f5de80c4f50 2023-10-24T23:53:23.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f5de67fc640 1 -- 172.21.15.107:0/3557209506 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de00a5950 con 0x7f5de80c4f50 2023-10-24T23:53:23.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f5de67fc640 1 -- 172.21.15.107:0/3557209506 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de00a5ce0 con 0x7f5de80c4f50 2023-10-24T23:53:23.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f5dedc3f640 1 -- 172.21.15.107:0/3557209506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5de80c4f50 msgr2=0x7f5de80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f5dedc3f640 1 --2- 172.21.15.107:0/3557209506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5de80c4f50 0x7f5de80a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5de806e8e0 tx=0x7f5de001c840 comp rx=0 tx=0).stop 2023-10-24T23:53:23.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f5dedc3f640 1 -- 172.21.15.107:0/3557209506 shutdown_connections 2023-10-24T23:53:23.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f5dedc3f640 1 --2- 172.21.15.107:0/3557209506 >> v2:172.21.15.107:3300/0 conn(0x7f5de80c2370 0x7f5de80a5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f5dedc3f640 1 --2- 172.21.15.107:0/3557209506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5de80c4f50 0x7f5de80a4d50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f5dedc3f640 1 -- 172.21.15.107:0/3557209506 >> 172.21.15.107:0/3557209506 conn(0x7f5de80c2860 msgr2=0x7f5de80c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:23.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f5dedc3f640 1 -- 172.21.15.107:0/3557209506 shutdown_connections 2023-10-24T23:53:23.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f5dedc3f640 1 -- 172.21.15.107:0/3557209506 wait complete. 2023-10-24T23:53:23.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f04be7fc640 1 -- 172.21.15.107:0/3026521405 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40ab600 con 0x7f04c0152120 2023-10-24T23:53:23.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.786+0000 7f04be7fc640 1 -- 172.21.15.107:0/3026521405 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b40a7050 con 0x7f04c0152120 2023-10-24T23:53:23.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f859b7fe640 1 -- 172.21.15.107:0/2548352092 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40721d0 con 0x7f85ac0462e0 2023-10-24T23:53:23.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f04be7fc640 1 -- 172.21.15.107:0/3026521405 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40ab9c0 con 0x7f04c0152120 2023-10-24T23:53:23.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f04c5cdd640 1 -- 172.21.15.107:0/3026521405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c0152120 msgr2=0x7f04c0158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f04c5cdd640 1 --2- 172.21.15.107:0/3026521405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c0152120 0x7f04c0158110 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f04c00bd960 tx=0x7f04b401c490 comp rx=0 tx=0).stop 2023-10-24T23:53:23.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7feb9dffb640 1 -- 172.21.15.107:0/443612190 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7feba00f0fe0 con 0x7feba80c25a0 2023-10-24T23:53:23.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dedc3f640 1 Processor -- start 2023-10-24T23:53:23.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dedc3f640 1 -- start start 2023-10-24T23:53:23.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f85b960c640 1 -- 172.21.15.107:0/2548352092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac02b020 con 0x7f85ac0462e0 2023-10-24T23:53:23.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f85b960c640 1 -- 172.21.15.107:0/2548352092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ac02c8f0 con 0x7f85ac0462e0 2023-10-24T23:53:23.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f04c5cdd640 1 -- 172.21.15.107:0/3026521405 shutdown_connections 2023-10-24T23:53:23.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f04c5cdd640 1 --2- 172.21.15.107:0/3026521405 >> v2:172.21.15.110:3300/0 conn(0x7f04c00c7890 0x7f04c00c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f04c5cdd640 1 --2- 172.21.15.107:0/3026521405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c0152120 0x7f04c0158110 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f04c5cdd640 1 -- 172.21.15.107:0/3026521405 >> 172.21.15.107:0/3026521405 conn(0x7f04c00abdc0 msgr2=0x7f04c00ac1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:23.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dedc3f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5de80c2370 0x7f5de80ab2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dedc3f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5de80c3380 0x7f5de81f5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dedc3f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5de80c4f50 0x7f5de80a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dedc3f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de81f5ba0 con 0x7f5de80c2370 2023-10-24T23:53:23.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dedc3f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de806ec10 con 0x7f5de80c4f50 2023-10-24T23:53:23.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dedc3f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5de81f16c0 con 0x7f5de80c3380 2023-10-24T23:53:23.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5de80c2370 0x7f5de80ab2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5de80c2370 0x7f5de80ab2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33166/0 (socket says 172.21.15.107:33166) 2023-10-24T23:53:23.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de77fe640 1 -- 172.21.15.107:0/3886141136 learned_addr learned my addr 172.21.15.107:0/3886141136 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de6ffd640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5de80c3380 0x7f5de81f5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de7fff640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5de80c4f50 0x7f5de80a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de77fe640 1 -- 172.21.15.107:0/3886141136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5de80c3380 msgr2=0x7f5de81f5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de77fe640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5de80c3380 0x7f5de81f5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de77fe640 1 -- 172.21.15.107:0/3886141136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5de80c4f50 msgr2=0x7f5de80a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de77fe640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5de80c4f50 0x7f5de80a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de77fe640 1 -- 172.21.15.107:0/3886141136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de0009a70 con 0x7f5de80c2370 2023-10-24T23:53:23.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de7fff640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5de80c4f50 0x7f5de80a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:23.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f85b960c640 1 -- 172.21.15.107:0/2548352092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ac0054d0 con 0x7f85ac0462e0 2023-10-24T23:53:23.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5de77fe640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5de80c2370 0x7f5de80ab2e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5de0019450 tx=0x7f5de00195e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dcffff640 1 -- 172.21.15.107:0/3886141136 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de00ad640 con 0x7f5de80c2370 2023-10-24T23:53:23.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dcffff640 1 -- 172.21.15.107:0/3886141136 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de00ad7a0 con 0x7f5de80c2370 2023-10-24T23:53:23.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dcffff640 1 -- 172.21.15.107:0/3886141136 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de00adb30 con 0x7f5de80c2370 2023-10-24T23:53:23.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dedc3f640 1 -- 172.21.15.107:0/3886141136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de806eeb0 con 0x7f5de80c2370 2023-10-24T23:53:23.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.790+0000 7f5dedc3f640 1 -- 172.21.15.107:0/3886141136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de80b96a0 con 0x7f5de80c2370 2023-10-24T23:53:23.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.794+0000 7f5dcdffb640 1 -- 172.21.15.107:0/3886141136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc0005260 con 0x7f5de80c2370 2023-10-24T23:53:23.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.794+0000 7f859b7fe640 1 -- 172.21.15.107:0/2548352092 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b4072330 con 0x7f85ac0462e0 2023-10-24T23:53:23.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.794+0000 7f859b7fe640 1 -- 172.21.15.107:0/2548352092 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b40726e0 con 0x7f85ac0462e0 2023-10-24T23:53:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.794+0000 7f859b7fe640 1 -- 172.21.15.107:0/2548352092 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f85b4080070 con 0x7f85ac0462e0 2023-10-24T23:53:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.794+0000 7f859b7fe640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f859c068320 0x7f859c06a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.794+0000 7f859b7fe640 1 -- 172.21.15.107:0/2548352092 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f85b410ebf0 con 0x7f85ac0462e0 2023-10-24T23:53:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.794+0000 7f85b2ffd640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f859c068320 0x7f859c06a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.794+0000 7f5dcffff640 1 -- 172.21.15.107:0/3886141136 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5de00b4070 con 0x7f5de80c2370 2023-10-24T23:53:23.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.798+0000 7f5dcffff640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5dd4065e60 0x7f5dd4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.798+0000 7f85b2ffd640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f859c068320 0x7f859c06a7e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f85a8004e30 tx=0x7f85a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.798+0000 7f5dcffff640 1 -- 172.21.15.107:0/3886141136 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f5de011eb00 con 0x7f5de80c2370 2023-10-24T23:53:23.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.798+0000 7f5de6ffd640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5dd4065e60 0x7f5dd4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.798+0000 7f5de6ffd640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5dd4065e60 0x7f5dd4068320 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8007fd0 tx=0x7f5dd8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2ffe62640 1 Processor -- start 2023-10-24T23:53:23.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2ffe62640 1 -- start start 2023-10-24T23:53:23.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2ffe62640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2f8155ca0 0x7fd2f80c1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2ffe62640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2f806c3a0 0x7fd2f806c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7f859b7fe640 1 -- 172.21.15.107:0/2548352092 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f85b40dbf80 con 0x7f85ac0462e0 2023-10-24T23:53:23.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2ffe62640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f806a970 con 0x7fd2f814fec0 2023-10-24T23:53:23.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2fd3fd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2f8155ca0 0x7fd2f80c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2ffe62640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f80a9550 con 0x7fd2f8155ca0 2023-10-24T23:53:23.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2fd3fd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd2f8155ca0 0x7fd2f80c1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33176/0 (socket says 172.21.15.107:33176) 2023-10-24T23:53:23.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2fd3fd640 1 -- 172.21.15.107:0/2781530310 learned_addr learned my addr 172.21.15.107:0/2781530310 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2ffe62640 1 -- 172.21.15.107:0/2781530310 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80c2c50 con 0x7fd2f806c3a0 2023-10-24T23:53:23.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2fe3ff640 1 --2- 172.21.15.107:0/2781530310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2f806c3a0 0x7fd2f806c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04c5cdd640 1 -- 172.21.15.107:0/3026521405 shutdown_connections 2023-10-24T23:53:23.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04c5cdd640 1 -- 172.21.15.107:0/3026521405 wait complete. 2023-10-24T23:53:23.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2fd3fd640 1 -- 172.21.15.107:0/2781530310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2f806c3a0 msgr2=0x7fd2f806c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:23.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2fd3fd640 1 --2- 172.21.15.107:0/2781530310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2f806c3a0 0x7fd2f806c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2fd3fd640 1 -- 172.21.15.107:0/2781530310 >> v1:172.21.15.107:6789/0 conn(0x7fd2f814fec0 legacy=0x7fd2f8069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:23.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.802+0000 7fd2fd3fd640 1 -- 172.21.15.107:0/2781530310 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80c2040 con 0x7fd2f8155ca0 2023-10-24T23:53:23.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04c5cdd640 1 Processor -- start 2023-10-24T23:53:23.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04c5cdd640 1 -- start start 2023-10-24T23:53:23.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7fd2fd3fd640 1 --2- 172.21.15.107:0/2781530310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f8155ca0 0x7fd2f80c1940 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009870 tx=0x7fd2e801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5c56f35f31202c server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04c5cdd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04c00c7890 0x7f04c00a9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04c5cdd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04c00c8350 0x7f04c0150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04c5cdd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c0152120 0x7f04c0142f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04c5cdd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00aba50 con 0x7f04c0152120 2023-10-24T23:53:23.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04c5cdd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c01edb30 con 0x7f04c00c8350 2023-10-24T23:53:23.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04c5cdd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c01eddd0 con 0x7f04c00c7890 2023-10-24T23:53:23.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c0152120 0x7f04c0142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c0152120 0x7f04c0142f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33186/0 (socket says 172.21.15.107:33186) 2023-10-24T23:53:23.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04bffff640 1 -- 172.21.15.107:0/2192816630 learned_addr learned my addr 172.21.15.107:0/2192816630 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04beffd640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04c00c8350 0x7f04c0150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04bf7fe640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04c00c7890 0x7f04c00a9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04bffff640 1 -- 172.21.15.107:0/2192816630 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04c00c7890 msgr2=0x7f04c00a9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04bffff640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04c00c7890 0x7f04c00a9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04bffff640 1 -- 172.21.15.107:0/2192816630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04c00c8350 msgr2=0x7f04c0150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04bffff640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04c00c8350 0x7f04c0150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.806+0000 7f04bffff640 1 -- 172.21.15.107:0/2192816630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b40094d0 con 0x7f04c0152120 2023-10-24T23:53:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.810+0000 7f04bffff640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c0152120 0x7f04c0142f80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f04b800efc0 tx=0x7f04b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2fcbfc640 1 -- 172.21.15.107:0/2781530310 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80ac600 con 0x7fd2f8155ca0 2023-10-24T23:53:23.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7f5dcffff640 1 -- 172.21.15.107:0/3886141136 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5de00ebe20 con 0x7f5de80c2370 2023-10-24T23:53:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7f04a7fff640 1 -- 172.21.15.107:0/2192816630 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b8014480 con 0x7f04c0152120 2023-10-24T23:53:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7f04c5cdd640 1 -- 172.21.15.107:0/2192816630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c00ab400 con 0x7f04c0152120 2023-10-24T23:53:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7f04c5cdd640 1 -- 172.21.15.107:0/2192816630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c01512d0 con 0x7f04c0152120 2023-10-24T23:53:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7f04a7fff640 1 -- 172.21.15.107:0/2192816630 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b800f040 con 0x7f04c0152120 2023-10-24T23:53:23.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2fcbfc640 1 -- 172.21.15.107:0/2781530310 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e80a8050 con 0x7fd2f8155ca0 2023-10-24T23:53:23.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2fcbfc640 1 -- 172.21.15.107:0/2781530310 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80ac980 con 0x7fd2f8155ca0 2023-10-24T23:53:23.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 -- 172.21.15.107:0/2781530310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f8155ca0 msgr2=0x7fd2f80c1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 --2- 172.21.15.107:0/2781530310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f8155ca0 0x7fd2f80c1940 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009870 tx=0x7fd2e801c3e0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 -- 172.21.15.107:0/2781530310 shutdown_connections 2023-10-24T23:53:23.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 --2- 172.21.15.107:0/2781530310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2f806c3a0 0x7fd2f806c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 --2- 172.21.15.107:0/2781530310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f8155ca0 0x7fd2f80c1940 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 -- 172.21.15.107:0/2781530310 >> 172.21.15.107:0/2781530310 conn(0x7fd2f80c3a40 msgr2=0x7fd2f80c3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:23.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 -- 172.21.15.107:0/2781530310 shutdown_connections 2023-10-24T23:53:23.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7f04a7fff640 1 -- 172.21.15.107:0/2192816630 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b80147e0 con 0x7f04c0152120 2023-10-24T23:53:23.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 -- 172.21.15.107:0/2781530310 wait complete. 2023-10-24T23:53:23.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 Processor -- start 2023-10-24T23:53:23.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 -- start start 2023-10-24T23:53:23.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2f806c3a0 0x7fd2f8152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f814fec0 0x7fd2f8135250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2f8155ca0 0x7fd2f81f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f8139bd0 con 0x7fd2f814fec0 2023-10-24T23:53:23.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f81f4e40 con 0x7fd2f8155ca0 2023-10-24T23:53:23.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.814+0000 7fd2ffe62640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f81f50e0 con 0x7fd2f806c3a0 2023-10-24T23:53:23.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2fdbfe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2f806c3a0 0x7fd2f8152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2fe3ff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2f8155ca0 0x7fd2f81f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2fe3ff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2f8155ca0 0x7fd2f81f18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:49302/0 (socket says 172.21.15.107:49302) 2023-10-24T23:53:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2fe3ff640 1 -- 172.21.15.107:0/3326814504 learned_addr learned my addr 172.21.15.107:0/3326814504 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2fe3ff640 1 -- 172.21.15.107:0/3326814504 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2f806c3a0 msgr2=0x7fd2f8152800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2fe3ff640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2f806c3a0 0x7fd2f8152800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2fe3ff640 1 -- 172.21.15.107:0/3326814504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f814fec0 msgr2=0x7fd2f8135250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-24T23:53:23.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2fe3ff640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f814fec0 0x7fd2f8135250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2fe3ff640 1 -- 172.21.15.107:0/3326814504 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80094d0 con 0x7fd2f8155ca0 2023-10-24T23:53:23.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2fe3ff640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2f8155ca0 0x7fd2f81f18d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000efc0 tx=0x7fd2f000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2ee7fc640 1 -- 172.21.15.107:0/3326814504 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0009ba0 con 0x7fd2f8155ca0 2023-10-24T23:53:23.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2ee7fc640 1 -- 172.21.15.107:0/3326814504 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f000f040 con 0x7fd2f8155ca0 2023-10-24T23:53:23.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2ee7fc640 1 -- 172.21.15.107:0/3326814504 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00146c0 con 0x7fd2f8155ca0 2023-10-24T23:53:23.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7f04a7fff640 1 -- 172.21.15.107:0/2192816630 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f04b8014940 con 0x7f04c0152120 2023-10-24T23:53:23.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7f04a7fff640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04ac065e60 0x7f04ac068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7f04a7fff640 1 -- 172.21.15.107:0/2192816630 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f04b8084990 con 0x7f04c0152120 2023-10-24T23:53:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7f04bf7fe640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04ac065e60 0x7f04ac068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2ffe62640 1 -- 172.21.15.107:0/3326814504 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f81f53c0 con 0x7fd2f8155ca0 2023-10-24T23:53:23.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2ffe62640 1 -- 172.21.15.107:0/3326814504 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f81f5980 con 0x7fd2f8155ca0 2023-10-24T23:53:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7f04bf7fe640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04ac065e60 0x7f04ac068320 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f04b40ad000 tx=0x7f04b40a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.818+0000 7fd2ffe62640 1 -- 172.21.15.107:0/3326814504 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f81f5b70 con 0x7fd2f8155ca0 2023-10-24T23:53:23.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.822+0000 7fd2ee7fc640 1 -- 172.21.15.107:0/3326814504 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd2f0014820 con 0x7fd2f8155ca0 2023-10-24T23:53:23.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.822+0000 7fd2ee7fc640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2dc065e60 0x7fd2dc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.822+0000 7fd2ee7fc640 1 -- 172.21.15.107:0/3326814504 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fd2f00848d0 con 0x7fd2f8155ca0 2023-10-24T23:53:23.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.822+0000 7fd2fdbfe640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2dc065e60 0x7fd2dc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.822+0000 7fd2fdbfe640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2dc065e60 0x7fd2dc068320 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4007fd0 tx=0x7fd2f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.826+0000 7f04c5cdd640 1 -- 172.21.15.107:0/2192816630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a0005260 con 0x7f04c0152120 2023-10-24T23:53:23.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.826+0000 7fb4de9e6640 1 Processor -- start 2023-10-24T23:53:23.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.826+0000 7fb4de9e6640 1 -- start start 2023-10-24T23:53:23.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.826+0000 7fb4de9e6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb4d00d3b70 0x7fb4d00d7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.826+0000 7fb4de1e5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb4d00da670 0x7fb4d00e0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38212/0 (socket says 172.21.15.107:38212) 2023-10-24T23:53:23.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.826+0000 7fb4de1e5640 1 -- 172.21.15.107:0/1156713054 learned_addr learned my addr 172.21.15.107:0/1156713054 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:23.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.826+0000 7fb4dd9e4640 1 --2- 172.21.15.107:0/1156713054 >> v2:172.21.15.110:3300/0 conn(0x7fb4d00d3b70 0x7fb4d00d7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.826+0000 7fb4de9e6640 1 -- 172.21.15.107:0/1156713054 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d0007320 con 0x7fb4d00da670 2023-10-24T23:53:23.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.826+0000 7fb4de9e6640 1 -- 172.21.15.107:0/1156713054 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d0007600 con 0x7fb4d00da220 2023-10-24T23:53:23.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.826+0000 7fb4de9e6640 1 -- 172.21.15.107:0/1156713054 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d002e860 con 0x7fb4d00d3b70 2023-10-24T23:53:23.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.830+0000 7fb4dd9e4640 1 -- 172.21.15.107:0/1156713054 >> v1:172.21.15.110:6789/0 conn(0x7fb4d00da220 legacy=0x7fb4d0046700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.830+0000 7fb4dd9e4640 1 -- 172.21.15.107:0/1156713054 >> v1:172.21.15.107:6789/0 conn(0x7fb4d00da670 legacy=0x7fb4d00e0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.830+0000 7fb4dd9e4640 1 -- 172.21.15.107:0/1156713054 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d00d9a20 con 0x7fb4d00d3b70 2023-10-24T23:53:23.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.830+0000 7fb4dd9e4640 1 --2- 172.21.15.107:0/1156713054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4d00d3b70 0x7fb4d00d7920 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400a960 tx=0x7fb4d401e8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adb7a3792fbeeb5b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.830+0000 7fb4dc9e2640 1 -- 172.21.15.107:0/1156713054 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d40a77a0 con 0x7fb4d00d3b70 2023-10-24T23:53:23.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.830+0000 7fb4dc9e2640 1 -- 172.21.15.107:0/1156713054 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d40a7900 con 0x7fb4d00d3b70 2023-10-24T23:53:23.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.830+0000 7fb4dc9e2640 1 -- 172.21.15.107:0/1156713054 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d40a7c90 con 0x7fb4d00d3b70 2023-10-24T23:53:23.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.834+0000 7fd2ee7fc640 1 -- 172.21.15.107:0/3326814504 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd2f0051d60 con 0x7fd2f8155ca0 2023-10-24T23:53:23.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.834+0000 7fb4de9e6640 1 -- 172.21.15.107:0/1156713054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4d00d3b70 msgr2=0x7fb4d00d7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.834+0000 7fb4de9e6640 1 --2- 172.21.15.107:0/1156713054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4d00d3b70 0x7fb4d00d7920 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400a960 tx=0x7fb4d401e8d0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.834+0000 7fb4de9e6640 1 -- 172.21.15.107:0/1156713054 shutdown_connections 2023-10-24T23:53:23.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.834+0000 7fb4de9e6640 1 --2- 172.21.15.107:0/1156713054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4d00d3b70 0x7fb4d00d7920 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.834+0000 7fb4de9e6640 1 -- 172.21.15.107:0/1156713054 >> 172.21.15.107:0/1156713054 conn(0x7fb4d0006440 msgr2=0x7fb4d0006850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:23.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4de9e6640 1 -- 172.21.15.107:0/1156713054 shutdown_connections 2023-10-24T23:53:23.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4de9e6640 1 -- 172.21.15.107:0/1156713054 wait complete. 2023-10-24T23:53:23.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4de9e6640 1 Processor -- start 2023-10-24T23:53:23.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4de9e6640 1 -- start start 2023-10-24T23:53:23.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4de9e6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4d00d3b70 0x7fb4d00c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4de9e6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d00da220 0x7fb4d00cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4dd1e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d00da220 0x7fb4d00cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4dd1e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d00da220 0x7fb4d00cd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33190/0 (socket says 172.21.15.107:33190) 2023-10-24T23:53:23.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4dd1e3640 1 -- 172.21.15.107:0/3265873884 learned_addr learned my addr 172.21.15.107:0/3265873884 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:23.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4de9e6640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4d00da670 0x7fb4d004b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4dd9e4640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4d00d3b70 0x7fb4d00c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d00d2540 con 0x7fb4d00da220 2023-10-24T23:53:23.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d016f7b0 con 0x7fb4d00da670 2023-10-24T23:53:23.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d00cccb0 con 0x7fb4d00d3b70 2023-10-24T23:53:23.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4dd1e3640 1 -- 172.21.15.107:0/3265873884 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4d00d3b70 msgr2=0x7fb4d00c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:23.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4dd1e3640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4d00d3b70 0x7fb4d00c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4dd1e3640 1 -- 172.21.15.107:0/3265873884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4d00da670 msgr2=0x7fb4d004b340 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-24T23:53:23.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4dd1e3640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4d00da670 0x7fb4d004b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:23.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4dd1e3640 1 -- 172.21.15.107:0/3265873884 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d400a5c0 con 0x7fb4d00da220 2023-10-24T23:53:23.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4dd9e4640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4d00d3b70 0x7fb4d00c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:53:23.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4dd1e3640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d00da220 0x7fb4d00cd790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800c500 tx=0x7fb4c800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.838+0000 7f04a7fff640 1 -- 172.21.15.107:0/2192816630 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f04b8051c20 con 0x7f04c0152120 2023-10-24T23:53:23.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4ce7fc640 1 -- 172.21.15.107:0/3265873884 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8005220 con 0x7fb4d00da220 2023-10-24T23:53:23.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d00ccf90 con 0x7fb4d00da220 2023-10-24T23:53:23.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d016f4a0 con 0x7fb4d00da220 2023-10-24T23:53:23.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4ce7fc640 1 -- 172.21.15.107:0/3265873884 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c80011d0 con 0x7fb4d00da220 2023-10-24T23:53:23.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.842+0000 7fb4ce7fc640 1 -- 172.21.15.107:0/3265873884 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8007030 con 0x7fb4d00da220 2023-10-24T23:53:23.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.846+0000 7fb4ce7fc640 1 -- 172.21.15.107:0/3265873884 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb4c8007250 con 0x7fb4d00da220 2023-10-24T23:53:23.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.846+0000 7fb4ce7fc640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4bc065e60 0x7fb4bc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:23.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.846+0000 7fb4dd9e4640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4bc065e60 0x7fb4bc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:23.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.846+0000 7fb4ce7fc640 1 -- 172.21.15.107:0/3265873884 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fb4c8054b10 con 0x7fb4d00da220 2023-10-24T23:53:23.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.846+0000 7fb4dd9e4640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4bc065e60 0x7fb4bc068320 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb4d40b1000 tx=0x7fb4d40a9070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:23.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.854+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c0005260 con 0x7fb4d00da220 2023-10-24T23:53:23.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:23.862+0000 7fb4ce7fc640 1 -- 172.21.15.107:0/3265873884 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb4c80541e0 con 0x7fb4d00da220 2023-10-24T23:53:24.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.062+0000 7fd64616b640 1 -- 172.21.15.107:0/971851334 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd61c005510 con 0x7fd64015c890 2023-10-24T23:53:24.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.062+0000 7fd62ffff640 1 -- 172.21.15.107:0/971851334 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd6380ef670 con 0x7fd64015c890 2023-10-24T23:53:24.069 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 -- 172.21.15.107:0/971851334 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd624065e30 msgr2=0x7fd6240682f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd624065e30 0x7fd6240682f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd640157500 tx=0x7fd634006bb0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 -- 172.21.15.107:0/971851334 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd64015c890 msgr2=0x7fd6400707b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd64015c890 0x7fd6400707b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd6380af040 tx=0x7fd6380196c0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 -- 172.21.15.107:0/971851334 shutdown_connections 2023-10-24T23:53:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd624065e30 0x7fd6240682f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd64015c890 0x7fd6400707b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd640159de0 0x7fd6400c4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 --2- 172.21.15.107:0/971851334 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd640072020 0x7fd640154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 -- 172.21.15.107:0/971851334 >> 172.21.15.107:0/971851334 conn(0x7fd6400a5750 msgr2=0x7fd640157c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 -- 172.21.15.107:0/971851334 shutdown_connections 2023-10-24T23:53:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.066+0000 7fd62dffb640 1 -- 172.21.15.107:0/971851334 wait complete. 2023-10-24T23:53:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.082+0000 7f2e8ca62640 1 -- 172.21.15.107:0/968582699 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2e88151490 con 0x7f2e880c20d0 2023-10-24T23:53:24.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.082+0000 7f2e77fff640 1 -- 172.21.15.107:0/968582699 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e88151490 con 0x7f2e880c20d0 2023-10-24T23:53:24.088 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 -- 172.21.15.107:0/968582699 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2e70065f70 msgr2=0x7f2e70068430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2e70065f70 0x7f2e70068430 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009b80 tx=0x7f2e78009190 comp rx=0 tx=0).stop 2023-10-24T23:53:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 -- 172.21.15.107:0/968582699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e880c20d0 msgr2=0x7f2e880a6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e880c20d0 0x7f2e880a6420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e800ab000 tx=0x7f2e80019740 comp rx=0 tx=0).stop 2023-10-24T23:53:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 -- 172.21.15.107:0/968582699 shutdown_connections 2023-10-24T23:53:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2e70065f70 0x7f2e70068430 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e880c2860 0x7f2e88152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e880c20d0 0x7f2e880a6420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 --2- 172.21.15.107:0/968582699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e880a83a0 0x7f2e88154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 -- 172.21.15.107:0/968582699 >> 172.21.15.107:0/968582699 conn(0x7f2e880a7880 msgr2=0x7f2e880a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.094 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2023-10-24T23:53:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 -- 172.21.15.107:0/968582699 shutdown_connections 2023-10-24T23:53:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.086+0000 7f2e8ca62640 1 -- 172.21.15.107:0/968582699 wait complete. 2023-10-24T23:53:24.095 DEBUG:teuthology.parallel:result is None 2023-10-24T23:53:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.094+0000 7febae5b7640 1 -- 172.21.15.107:0/443612190 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7feb880057e0 con 0x7feba80c25a0 2023-10-24T23:53:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.094+0000 7feb9dffb640 1 -- 172.21.15.107:0/443612190 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feba00f4600 con 0x7feba80c25a0 2023-10-24T23:53:24.099 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:24.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.094+0000 7febae5b7640 1 -- 172.21.15.107:0/443612190 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb8c065eb0 msgr2=0x7feb8c068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.094+0000 7febae5b7640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb8c065eb0 0x7feb8c068370 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7feb98004e30 tx=0x7feb98009040 comp rx=0 tx=0).stop 2023-10-24T23:53:24.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.098+0000 7febae5b7640 1 -- 172.21.15.107:0/443612190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba80c25a0 msgr2=0x7feba815ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.098+0000 7febae5b7640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba80c25a0 0x7feba815ff20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7feba00b3040 tx=0x7feba001dae0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.098+0000 7febae5b7640 1 -- 172.21.15.107:0/443612190 shutdown_connections 2023-10-24T23:53:24.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.098+0000 7febae5b7640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba80c25a0 0x7feba815ff20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.098+0000 7febae5b7640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feba80a4ec0 0x7feba815f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.098+0000 7febae5b7640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb8c065eb0 0x7feb8c068370 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.098+0000 7febae5b7640 1 --2- 172.21.15.107:0/443612190 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feba808ba10 0x7feba8087c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.098+0000 7febae5b7640 1 -- 172.21.15.107:0/443612190 >> 172.21.15.107:0/443612190 conn(0x7feba80a7880 msgr2=0x7feba806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.098+0000 7febae5b7640 1 -- 172.21.15.107:0/443612190 shutdown_connections 2023-10-24T23:53:24.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.098+0000 7febae5b7640 1 -- 172.21.15.107:0/443612190 wait complete. 2023-10-24T23:53:24.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.106+0000 7f85b960c640 1 -- 172.21.15.107:0/2548352092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f85ac0008e0 con 0x7f85ac0462e0 2023-10-24T23:53:24.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.106+0000 7f859b7fe640 1 -- 172.21.15.107:0/2548352092 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f85b40df5a0 con 0x7f85ac0462e0 2023-10-24T23:53:24.110 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-10-24T23:53:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 -- 172.21.15.107:0/2548352092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f859c068320 msgr2=0x7f859c06a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f859c068320 0x7f859c06a7e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f85a8004e30 tx=0x7f85a8009040 comp rx=0 tx=0).stop 2023-10-24T23:53:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 -- 172.21.15.107:0/2548352092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85ac0462e0 msgr2=0x7f85ac0cab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85ac0462e0 0x7f85ac0cab60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f85b407a6e0 tx=0x7f85b404ea80 comp rx=0 tx=0).stop 2023-10-24T23:53:24.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 -- 172.21.15.107:0/2548352092 shutdown_connections 2023-10-24T23:53:24.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85ac0462e0 0x7f85ac0cab60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f859c068320 0x7f859c06a7e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85ac00b1a0 0x7f85ac0d57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 --2- 172.21.15.107:0/2548352092 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85ac006b40 0x7f85ac16f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 -- 172.21.15.107:0/2548352092 >> 172.21.15.107:0/2548352092 conn(0x7f85ac006020 msgr2=0x7f85ac006400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 -- 172.21.15.107:0/2548352092 shutdown_connections 2023-10-24T23:53:24.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f85b960c640 1 -- 172.21.15.107:0/2548352092 wait complete. 2023-10-24T23:53:24.117 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2023-10-24T23:53:24.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.110+0000 7f5dcdffb640 1 -- 172.21.15.107:0/3886141136 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f5dc00057e0 con 0x7f5de80c2370 2023-10-24T23:53:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcffff640 1 -- 172.21.15.107:0/3886141136 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5de00ef440 con 0x7f5de80c2370 2023-10-24T23:53:24.118 DEBUG:teuthology.parallel:result is None 2023-10-24T23:53:24.118 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 -- 172.21.15.107:0/3886141136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5dd4065e60 msgr2=0x7f5dd4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5dd4065e60 0x7f5dd4068320 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8007fd0 tx=0x7f5dd8009040 comp rx=0 tx=0).stop 2023-10-24T23:53:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 -- 172.21.15.107:0/3886141136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5de80c2370 msgr2=0x7f5de80ab2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5de80c2370 0x7f5de80ab2e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5de0019450 tx=0x7f5de00195e0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 -- 172.21.15.107:0/3886141136 shutdown_connections 2023-10-24T23:53:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5de80c4f50 0x7f5de80a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5dd4065e60 0x7f5dd4068320 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5de80c3380 0x7f5de81f5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 --2- 172.21.15.107:0/3886141136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5de80c2370 0x7f5de80ab2e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 -- 172.21.15.107:0/3886141136 >> 172.21.15.107:0/3886141136 conn(0x7f5de80c2860 msgr2=0x7f5de80bceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 -- 172.21.15.107:0/3886141136 shutdown_connections 2023-10-24T23:53:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.114+0000 7f5dcdffb640 1 -- 172.21.15.107:0/3886141136 wait complete. 2023-10-24T23:53:24.123 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2023-10-24T23:53:24.123 DEBUG:teuthology.parallel:result is None 2023-10-24T23:53:24.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.126+0000 7fd2ffe62640 1 -- 172.21.15.107:0/3326814504 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd2f806be60 con 0x7fd2f8155ca0 2023-10-24T23:53:24.133 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2023-10-24T23:53:24.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ee7fc640 1 -- 172.21.15.107:0/3326814504 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd2f0055380 con 0x7fd2f8155ca0 2023-10-24T23:53:24.134 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:24.134 DEBUG:teuthology.parallel:result is None 2023-10-24T23:53:24.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 -- 172.21.15.107:0/3326814504 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2dc065e60 msgr2=0x7fd2dc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2dc065e60 0x7fd2dc068320 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4007fd0 tx=0x7fd2f4009040 comp rx=0 tx=0).stop 2023-10-24T23:53:24.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 -- 172.21.15.107:0/3326814504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2f8155ca0 msgr2=0x7fd2f81f18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2f8155ca0 0x7fd2f81f18d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000efc0 tx=0x7fd2f000c490 comp rx=0 tx=0).stop 2023-10-24T23:53:24.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 -- 172.21.15.107:0/3326814504 shutdown_connections 2023-10-24T23:53:24.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2f8155ca0 0x7fd2f81f18d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2f814fec0 0x7fd2f8135250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2dc065e60 0x7fd2dc068320 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 --2- 172.21.15.107:0/3326814504 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2f806c3a0 0x7fd2f8152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 -- 172.21.15.107:0/3326814504 >> 172.21.15.107:0/3326814504 conn(0x7fd2f80c3a40 msgr2=0x7fd2f80aa3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 -- 172.21.15.107:0/3326814504 shutdown_connections 2023-10-24T23:53:24.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.130+0000 7fd2ffe62640 1 -- 172.21.15.107:0/3326814504 wait complete. 2023-10-24T23:53:24.138 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2023-10-24T23:53:24.138 DEBUG:teuthology.parallel:result is None 2023-10-24T23:53:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 -- 172.21.15.107:0/2192816630 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f04a0005510 con 0x7f04c0152120 2023-10-24T23:53:24.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04a7fff640 1 -- 172.21.15.107:0/2192816630 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f04b8055240 con 0x7f04c0152120 2023-10-24T23:53:24.149 INFO:teuthology.orchestra.run.smithi107.stdout:55834574857 2023-10-24T23:53:24.150 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2023-10-24T23:53:24.150 DEBUG:teuthology.parallel:result is None 2023-10-24T23:53:24.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 -- 172.21.15.107:0/2192816630 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04ac065e60 msgr2=0x7f04ac068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04ac065e60 0x7f04ac068320 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f04b40ad000 tx=0x7f04b40a6000 comp rx=0 tx=0).stop 2023-10-24T23:53:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 -- 172.21.15.107:0/2192816630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c0152120 msgr2=0x7f04c0142f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c0152120 0x7f04c0142f80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f04b800efc0 tx=0x7f04b800c490 comp rx=0 tx=0).stop 2023-10-24T23:53:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 -- 172.21.15.107:0/2192816630 shutdown_connections 2023-10-24T23:53:24.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04c0152120 0x7f04c0142f80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04c00c8350 0x7f04c0150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04ac065e60 0x7f04ac068320 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 --2- 172.21.15.107:0/2192816630 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04c00c7890 0x7f04c00a9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 -- 172.21.15.107:0/2192816630 >> 172.21.15.107:0/2192816630 conn(0x7f04c00abdc0 msgr2=0x7f04c0155300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 -- 172.21.15.107:0/2192816630 shutdown_connections 2023-10-24T23:53:24.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.146+0000 7f04c5cdd640 1 -- 172.21.15.107:0/2192816630 wait complete. 2023-10-24T23:53:24.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb4c00050d0 con 0x7fb4d00da220 2023-10-24T23:53:24.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4ce7fc640 1 -- 172.21.15.107:0/3265873884 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb4c8007500 con 0x7fb4d00da220 2023-10-24T23:53:24.162 INFO:teuthology.orchestra.run.smithi107.stdout:55834574856 2023-10-24T23:53:24.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4bc065e60 msgr2=0x7fb4bc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4bc065e60 0x7fb4bc068320 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb4d40b1000 tx=0x7fb4d40a9070 comp rx=0 tx=0).stop 2023-10-24T23:53:24.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d00da220 msgr2=0x7fb4d00cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d00da220 0x7fb4d00cd790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800c500 tx=0x7fb4c800c9c0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 shutdown_connections 2023-10-24T23:53:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4d00da670 0x7fb4d004b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4d00da220 0x7fb4d00cd790 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4bc065e60 0x7fb4bc068320 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 --2- 172.21.15.107:0/3265873884 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4d00d3b70 0x7fb4d00c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 >> 172.21.15.107:0/3265873884 conn(0x7fb4d0006440 msgr2=0x7fb4d00d7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 shutdown_connections 2023-10-24T23:53:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.158+0000 7fb4de9e6640 1 -- 172.21.15.107:0/3265873884 wait complete. 2023-10-24T23:53:24.167 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2023-10-24T23:53:24.167 DEBUG:teuthology.parallel:result is None 2023-10-24T23:53:24.178 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2023-10-24T23:53:24.179 DEBUG:teuthology.parallel:result is None 2023-10-24T23:53:24.179 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-24T23:53:24.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:24.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5ff85640 1 Processor -- start 2023-10-24T23:53:24.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5ff85640 1 -- start start 2023-10-24T23:53:24.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5ff85640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d5806f490 0x7f6d5814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5ff85640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6d58152b30 0x7f6d58152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5ff85640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d58155cd0 con 0x7f6d5814c470 2023-10-24T23:53:24.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5ff85640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d581526e0 con 0x7f6d58152b30 2023-10-24T23:53:24.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5ff85640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d580c4380 con 0x7f6d5806f490 2023-10-24T23:53:24.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5dd21640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6d5814c470 0x7f6d58071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38222/0 (socket says 172.21.15.107:38222) 2023-10-24T23:53:24.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5dd21640 1 -- 172.21.15.107:0/1272280972 learned_addr learned my addr 172.21.15.107:0/1272280972 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:24.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5e522640 1 --2- 172.21.15.107:0/1272280972 >> v2:172.21.15.107:3300/0 conn(0x7f6d58152b30 0x7f6d58152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5d520640 1 --2- 172.21.15.107:0/1272280972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d5806f490 0x7f6d5814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5e522640 1 -- 172.21.15.107:0/1272280972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d5806f490 msgr2=0x7f6d5814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:24.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5e522640 1 --2- 172.21.15.107:0/1272280972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d5806f490 0x7f6d5814c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5e522640 1 -- 172.21.15.107:0/1272280972 >> v1:172.21.15.107:6789/0 conn(0x7f6d5814c470 legacy=0x7f6d58071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5e522640 1 -- 172.21.15.107:0/1272280972 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d580c48b0 con 0x7f6d58152b30 2023-10-24T23:53:24.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5cd1f640 1 -- 172.21.15.107:0/1272280972 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080750202 0 0) 0x7f6d58155cd0 con 0x7f6d5814c470 2023-10-24T23:53:24.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.322+0000 7f6d5d520640 1 --2- 172.21.15.107:0/1272280972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d5806f490 0x7f6d5814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:24.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5e522640 1 --2- 172.21.15.107:0/1272280972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d58152b30 0x7f6d58152fa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009d90 tx=0x7f6d4801c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=543067ec9b2d6cd1 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:24.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5cd1f640 1 -- 172.21.15.107:0/1272280972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d480a6700 con 0x7f6d58152b30 2023-10-24T23:53:24.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5cd1f640 1 -- 172.21.15.107:0/1272280972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d480a6860 con 0x7f6d58152b30 2023-10-24T23:53:24.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5cd1f640 1 -- 172.21.15.107:0/1272280972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d480a6bf0 con 0x7f6d58152b30 2023-10-24T23:53:24.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 -- 172.21.15.107:0/1272280972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d58152b30 msgr2=0x7f6d58152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 --2- 172.21.15.107:0/1272280972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d58152b30 0x7f6d58152fa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009d90 tx=0x7f6d4801c830 comp rx=0 tx=0).stop 2023-10-24T23:53:24.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 -- 172.21.15.107:0/1272280972 shutdown_connections 2023-10-24T23:53:24.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 --2- 172.21.15.107:0/1272280972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d58152b30 0x7f6d58152fa0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 --2- 172.21.15.107:0/1272280972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d5806f490 0x7f6d5814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 -- 172.21.15.107:0/1272280972 >> 172.21.15.107:0/1272280972 conn(0x7f6d58070f60 msgr2=0x7f6d58071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 -- 172.21.15.107:0/1272280972 shutdown_connections 2023-10-24T23:53:24.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 -- 172.21.15.107:0/1272280972 wait complete. 2023-10-24T23:53:24.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 Processor -- start 2023-10-24T23:53:24.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 -- start start 2023-10-24T23:53:24.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d5806f490 0x7f6d58144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d5814c470 0x7f6d581497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d58152b30 0x7f6d58142ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d581ed990 con 0x7f6d58152b30 2023-10-24T23:53:24.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d58148f10 con 0x7f6d5806f490 2023-10-24T23:53:24.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5814b7a0 con 0x7f6d5814c470 2023-10-24T23:53:24.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5e522640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d58152b30 0x7f6d58142ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5e522640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d58152b30 0x7f6d58142ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33210/0 (socket says 172.21.15.107:33210) 2023-10-24T23:53:24.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5e522640 1 -- 172.21.15.107:0/282914191 learned_addr learned my addr 172.21.15.107:0/282914191 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:24.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5d520640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d5814c470 0x7f6d581497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5e522640 1 -- 172.21.15.107:0/282914191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d5814c470 msgr2=0x7f6d581497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5e522640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d5814c470 0x7f6d581497e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5e522640 1 -- 172.21.15.107:0/282914191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d5806f490 msgr2=0x7f6d58144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5dd21640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d5806f490 0x7f6d58144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5e522640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d5806f490 0x7f6d58144220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5e522640 1 -- 172.21.15.107:0/282914191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d48009a70 con 0x7f6d58152b30 2023-10-24T23:53:24.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5e522640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d58152b30 0x7f6d58142ee0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6d4801aee0 tx=0x7f6d480195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:24.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d567fc640 1 -- 172.21.15.107:0/282914191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d480ba880 con 0x7f6d58152b30 2023-10-24T23:53:24.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d567fc640 1 -- 172.21.15.107:0/282914191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d480ba9e0 con 0x7f6d58152b30 2023-10-24T23:53:24.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 -- 172.21.15.107:0/282914191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5814ba80 con 0x7f6d58152b30 2023-10-24T23:53:24.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d567fc640 1 -- 172.21.15.107:0/282914191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d480bad50 con 0x7f6d58152b30 2023-10-24T23:53:24.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.326+0000 7f6d5ff85640 1 -- 172.21.15.107:0/282914191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5814c040 con 0x7f6d58152b30 2023-10-24T23:53:24.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.330+0000 7f6d5ff85640 1 -- 172.21.15.107:0/282914191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d58143be0 con 0x7f6d58152b30 2023-10-24T23:53:24.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.330+0000 7f6d567fc640 1 -- 172.21.15.107:0/282914191 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6d480c5460 con 0x7f6d58152b30 2023-10-24T23:53:24.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.330+0000 7f6d567fc640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6d44065e60 0x7f6d44068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.330+0000 7f6d567fc640 1 -- 172.21.15.107:0/282914191 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f6d480aa070 con 0x7f6d58152b30 2023-10-24T23:53:24.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.330+0000 7f6d5dd21640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6d44065e60 0x7f6d44068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.330+0000 7f6d5dd21640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6d44065e60 0x7f6d44068320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6d50000f70 tx=0x7f6d50009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:24.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.334+0000 7f6d567fc640 1 -- 172.21.15.107:0/282914191 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6d480eca20 con 0x7f6d58152b30 2023-10-24T23:53:24.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.474+0000 7f6d5ff85640 1 -- 172.21.15.107:0/282914191 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d580666c0 con 0x7f6d44065e60 2023-10-24T23:53:24.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.510+0000 7f6d567fc640 1 -- 172.21.15.107:0/282914191 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525074 (secure 0 0 0) 0x7f6d50001c80 con 0x7f6d44065e60 2023-10-24T23:53:24.519 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:24.523 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 -- 172.21.15.107:0/282914191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6d44065e60 msgr2=0x7f6d44068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6d44065e60 0x7f6d44068320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6d50000f70 tx=0x7f6d50009040 comp rx=0 tx=0).stop 2023-10-24T23:53:24.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 -- 172.21.15.107:0/282914191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d58152b30 msgr2=0x7f6d58142ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d58152b30 0x7f6d58142ee0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6d4801aee0 tx=0x7f6d480195d0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 -- 172.21.15.107:0/282914191 shutdown_connections 2023-10-24T23:53:24.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d58152b30 0x7f6d58142ee0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6d44065e60 0x7f6d44068320 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d5814c470 0x7f6d581497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 --2- 172.21.15.107:0/282914191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6d5806f490 0x7f6d58144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 -- 172.21.15.107:0/282914191 >> 172.21.15.107:0/282914191 conn(0x7f6d58070f60 msgr2=0x7f6d58152f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 -- 172.21.15.107:0/282914191 shutdown_connections 2023-10-24T23:53:24.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.514+0000 7f6d5ff85640 1 -- 172.21.15.107:0/282914191 wait complete. 2023-10-24T23:53:24.539 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-24T23:53:23.282371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595092,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235560,"kb_used_data":3560,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754739160,"statfs":{"total":773094113280,"available":772852899840,"internally_reserved":0,"allocated":3645440,"data_stored":2144952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.491437"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603561+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-24T23:53:13.603561+0000","last_peered":"2023-10-24T23:53:13.603561+0000","last_clean":"2023-10-24T23:53:13.603561+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-24T23:53:13.603561+0000","last_undegraded":"2023-10-24T23:53:13.603561+0000","last_fullsized":"2023-10-24T23:53:13.603561+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:51:20.137796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794961+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-24T23:53:12.794961+0000","last_peered":"2023-10-24T23:53:12.794961+0000","last_clean":"2023-10-24T23:53:12.794961+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-24T23:53:12.794961+0000","last_undegraded":"2023-10-24T23:53:12.794961+0000","last_fullsized":"2023-10-24T23:53:12.794961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:00:20.101379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788258+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-24T23:53:11.788258+0000","last_peered":"2023-10-24T23:53:11.788258+0000","last_clean":"2023-10-24T23:53:11.788258+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-24T23:53:11.788258+0000","last_undegraded":"2023-10-24T23:53:11.788258+0000","last_fullsized":"2023-10-24T23:53:11.788258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791737+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-24T23:53:11.791737+0000","last_peered":"2023-10-24T23:53:11.791737+0000","last_clean":"2023-10-24T23:53:11.791737+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-24T23:53:11.791737+0000","last_undegraded":"2023-10-24T23:53:11.791737+0000","last_fullsized":"2023-10-24T23:53:11.791737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794565+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-24T23:53:12.794565+0000","last_peered":"2023-10-24T23:53:12.794565+0000","last_clean":"2023-10-24T23:53:12.794565+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-24T23:53:12.794565+0000","last_undegraded":"2023-10-24T23:53:12.794565+0000","last_fullsized":"2023-10-24T23:53:12.794565+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:59.368156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.601719+0000","last_change":"2023-10-24T23:53:11.444089+0000","last_active":"2023-10-24T23:53:13.601719+0000","last_peered":"2023-10-24T23:53:13.601719+0000","last_clean":"2023-10-24T23:53:13.601719+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-24T23:53:13.601719+0000","last_undegraded":"2023-10-24T23:53:13.601719+0000","last_fullsized":"2023-10-24T23:53:13.601719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:54:06.875882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807520+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-24T23:53:11.807520+0000","last_peered":"2023-10-24T23:53:11.807520+0000","last_clean":"2023-10-24T23:53:11.807520+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-24T23:53:11.807520+0000","last_undegraded":"2023-10-24T23:53:11.807520+0000","last_fullsized":"2023-10-24T23:53:11.807520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806881+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-24T23:53:11.806881+0000","last_peered":"2023-10-24T23:53:11.806881+0000","last_clean":"2023-10-24T23:53:11.806881+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-24T23:53:11.806881+0000","last_undegraded":"2023-10-24T23:53:11.806881+0000","last_fullsized":"2023-10-24T23:53:11.806881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602008+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-24T23:53:13.602008+0000","last_peered":"2023-10-24T23:53:13.602008+0000","last_clean":"2023-10-24T23:53:13.602008+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-24T23:53:13.602008+0000","last_undegraded":"2023-10-24T23:53:13.602008+0000","last_fullsized":"2023-10-24T23:53:13.602008+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:38:56.643025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101201+0000","last_change":"2023-10-24T23:53:11.438681+0000","last_active":"2023-10-24T23:53:13.101201+0000","last_peered":"2023-10-24T23:53:13.101201+0000","last_clean":"2023-10-24T23:53:13.101201+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-24T23:53:13.101201+0000","last_undegraded":"2023-10-24T23:53:13.101201+0000","last_fullsized":"2023-10-24T23:53:13.101201+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:50.139697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832718+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-24T23:53:11.832718+0000","last_peered":"2023-10-24T23:53:11.832718+0000","last_clean":"2023-10-24T23:53:11.832718+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-24T23:53:11.832718+0000","last_undegraded":"2023-10-24T23:53:11.832718+0000","last_fullsized":"2023-10-24T23:53:11.832718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.810798+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-24T23:53:11.810798+0000","last_peered":"2023-10-24T23:53:11.810798+0000","last_clean":"2023-10-24T23:53:11.810798+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-24T23:53:11.810798+0000","last_undegraded":"2023-10-24T23:53:11.810798+0000","last_fullsized":"2023-10-24T23:53:11.810798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101349+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-24T23:53:13.101349+0000","last_peered":"2023-10-24T23:53:13.101349+0000","last_clean":"2023-10-24T23:53:13.101349+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-24T23:53:13.101349+0000","last_undegraded":"2023-10-24T23:53:13.101349+0000","last_fullsized":"2023-10-24T23:53:13.101349+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:22.974333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796171+0000","last_change":"2023-10-24T23:53:11.454537+0000","last_active":"2023-10-24T23:53:12.796171+0000","last_peered":"2023-10-24T23:53:12.796171+0000","last_clean":"2023-10-24T23:53:12.796171+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-24T23:53:12.796171+0000","last_undegraded":"2023-10-24T23:53:12.796171+0000","last_fullsized":"2023-10-24T23:53:12.796171+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:14:06.792651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.830575+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-24T23:53:11.830575+0000","last_peered":"2023-10-24T23:53:11.830575+0000","last_clean":"2023-10-24T23:53:11.830575+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-24T23:53:11.830575+0000","last_undegraded":"2023-10-24T23:53:11.830575+0000","last_fullsized":"2023-10-24T23:53:11.830575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800231+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-24T23:53:11.800231+0000","last_peered":"2023-10-24T23:53:11.800231+0000","last_clean":"2023-10-24T23:53:11.800231+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-24T23:53:11.800231+0000","last_undegraded":"2023-10-24T23:53:11.800231+0000","last_fullsized":"2023-10-24T23:53:11.800231+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605269+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-24T23:53:13.605269+0000","last_peered":"2023-10-24T23:53:13.605269+0000","last_clean":"2023-10-24T23:53:13.605269+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-24T23:53:13.605269+0000","last_undegraded":"2023-10-24T23:53:13.605269+0000","last_fullsized":"2023-10-24T23:53:13.605269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:35:39.879563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795373+0000","last_change":"2023-10-24T23:53:11.444138+0000","last_active":"2023-10-24T23:53:12.795373+0000","last_peered":"2023-10-24T23:53:12.795373+0000","last_clean":"2023-10-24T23:53:12.795373+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-24T23:53:12.795373+0000","last_undegraded":"2023-10-24T23:53:12.795373+0000","last_fullsized":"2023-10-24T23:53:12.795373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:27:38.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788658+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-24T23:53:11.788658+0000","last_peered":"2023-10-24T23:53:11.788658+0000","last_clean":"2023-10-24T23:53:11.788658+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-24T23:53:11.788658+0000","last_undegraded":"2023-10-24T23:53:11.788658+0000","last_fullsized":"2023-10-24T23:53:11.788658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.797645+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-24T23:53:11.797645+0000","last_peered":"2023-10-24T23:53:11.797645+0000","last_clean":"2023-10-24T23:53:11.797645+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-24T23:53:11.797645+0000","last_undegraded":"2023-10-24T23:53:11.797645+0000","last_fullsized":"2023-10-24T23:53:11.797645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795697+0000","last_change":"2023-10-24T23:53:11.832809+0000","last_active":"2023-10-24T23:53:12.795697+0000","last_peered":"2023-10-24T23:53:12.795697+0000","last_clean":"2023-10-24T23:53:12.795697+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-24T23:53:12.795697+0000","last_undegraded":"2023-10-24T23:53:12.795697+0000","last_fullsized":"2023-10-24T23:53:12.795697+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:01.151315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604963+0000","last_change":"2023-10-24T23:53:11.442035+0000","last_active":"2023-10-24T23:53:13.604963+0000","last_peered":"2023-10-24T23:53:13.604963+0000","last_clean":"2023-10-24T23:53:13.604963+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-24T23:53:13.604963+0000","last_undegraded":"2023-10-24T23:53:13.604963+0000","last_fullsized":"2023-10-24T23:53:13.604963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:13.717844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.811329+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-24T23:53:11.811329+0000","last_peered":"2023-10-24T23:53:11.811329+0000","last_clean":"2023-10-24T23:53:11.811329+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-24T23:53:11.811329+0000","last_undegraded":"2023-10-24T23:53:11.811329+0000","last_fullsized":"2023-10-24T23:53:11.811329+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807632+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-24T23:53:11.807632+0000","last_peered":"2023-10-24T23:53:11.807632+0000","last_clean":"2023-10-24T23:53:11.807632+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-24T23:53:11.807632+0000","last_undegraded":"2023-10-24T23:53:11.807632+0000","last_fullsized":"2023-10-24T23:53:11.807632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797540+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-24T23:53:12.797540+0000","last_peered":"2023-10-24T23:53:12.797540+0000","last_clean":"2023-10-24T23:53:12.797540+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-24T23:53:12.797540+0000","last_undegraded":"2023-10-24T23:53:12.797540+0000","last_fullsized":"2023-10-24T23:53:12.797540+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:26.012711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604200+0000","last_change":"2023-10-24T23:53:11.448032+0000","last_active":"2023-10-24T23:53:13.604200+0000","last_peered":"2023-10-24T23:53:13.604200+0000","last_clean":"2023-10-24T23:53:13.604200+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-24T23:53:13.604200+0000","last_undegraded":"2023-10-24T23:53:13.604200+0000","last_fullsized":"2023-10-24T23:53:13.604200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:50:32.490810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791720+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-24T23:53:11.791720+0000","last_peered":"2023-10-24T23:53:11.791720+0000","last_clean":"2023-10-24T23:53:11.791720+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-24T23:53:11.791720+0000","last_undegraded":"2023-10-24T23:53:11.791720+0000","last_fullsized":"2023-10-24T23:53:11.791720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788475+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-24T23:53:11.788475+0000","last_peered":"2023-10-24T23:53:11.788475+0000","last_clean":"2023-10-24T23:53:11.788475+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-24T23:53:11.788475+0000","last_undegraded":"2023-10-24T23:53:11.788475+0000","last_fullsized":"2023-10-24T23:53:11.788475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795316+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-24T23:53:12.795316+0000","last_peered":"2023-10-24T23:53:12.795316+0000","last_clean":"2023-10-24T23:53:12.795316+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-24T23:53:12.795316+0000","last_undegraded":"2023-10-24T23:53:12.795316+0000","last_fullsized":"2023-10-24T23:53:12.795316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:10.820967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603323+0000","last_change":"2023-10-24T23:53:11.446925+0000","last_active":"2023-10-24T23:53:13.603323+0000","last_peered":"2023-10-24T23:53:13.603323+0000","last_clean":"2023-10-24T23:53:13.603323+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-24T23:53:13.603323+0000","last_undegraded":"2023-10-24T23:53:13.603323+0000","last_fullsized":"2023-10-24T23:53:13.603323+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:59:52.485469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804040+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-24T23:53:11.804040+0000","last_peered":"2023-10-24T23:53:11.804040+0000","last_clean":"2023-10-24T23:53:11.804040+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-24T23:53:11.804040+0000","last_undegraded":"2023-10-24T23:53:11.804040+0000","last_fullsized":"2023-10-24T23:53:11.804040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791850+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-24T23:53:11.791850+0000","last_peered":"2023-10-24T23:53:11.791850+0000","last_clean":"2023-10-24T23:53:11.791850+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-24T23:53:11.791850+0000","last_undegraded":"2023-10-24T23:53:11.791850+0000","last_fullsized":"2023-10-24T23:53:11.791850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795971+0000","last_change":"2023-10-24T23:53:11.849056+0000","last_active":"2023-10-24T23:53:12.795971+0000","last_peered":"2023-10-24T23:53:12.795971+0000","last_clean":"2023-10-24T23:53:12.795971+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-24T23:53:12.795971+0000","last_undegraded":"2023-10-24T23:53:12.795971+0000","last_fullsized":"2023-10-24T23:53:12.795971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:08.932942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803866+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-24T23:53:11.803866+0000","last_peered":"2023-10-24T23:53:11.803866+0000","last_clean":"2023-10-24T23:53:11.803866+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-24T23:53:11.803866+0000","last_undegraded":"2023-10-24T23:53:11.803866+0000","last_fullsized":"2023-10-24T23:53:11.803866+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788258+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-24T23:53:11.788258+0000","last_peered":"2023-10-24T23:53:11.788258+0000","last_clean":"2023-10-24T23:53:11.788258+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-24T23:53:11.788258+0000","last_undegraded":"2023-10-24T23:53:11.788258+0000","last_fullsized":"2023-10-24T23:53:11.788258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602180+0000","last_change":"2023-10-24T23:53:11.447167+0000","last_active":"2023-10-24T23:53:13.602180+0000","last_peered":"2023-10-24T23:53:13.602180+0000","last_clean":"2023-10-24T23:53:13.602180+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-24T23:53:13.602180+0000","last_undegraded":"2023-10-24T23:53:13.602180+0000","last_fullsized":"2023-10-24T23:53:13.602180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:06:25.006728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794920+0000","last_change":"2023-10-24T23:53:11.813704+0000","last_active":"2023-10-24T23:53:12.794920+0000","last_peered":"2023-10-24T23:53:12.794920+0000","last_clean":"2023-10-24T23:53:12.794920+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-24T23:53:12.794920+0000","last_undegraded":"2023-10-24T23:53:12.794920+0000","last_fullsized":"2023-10-24T23:53:12.794920+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:39:09.996039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804467+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-24T23:53:11.804467+0000","last_peered":"2023-10-24T23:53:11.804467+0000","last_clean":"2023-10-24T23:53:11.804467+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-24T23:53:11.804467+0000","last_undegraded":"2023-10-24T23:53:11.804467+0000","last_fullsized":"2023-10-24T23:53:11.804467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838755+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-24T23:53:11.838755+0000","last_peered":"2023-10-24T23:53:11.838755+0000","last_clean":"2023-10-24T23:53:11.838755+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-24T23:53:11.838755+0000","last_undegraded":"2023-10-24T23:53:11.838755+0000","last_fullsized":"2023-10-24T23:53:11.838755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795718+0000","last_change":"2023-10-24T23:53:11.450974+0000","last_active":"2023-10-24T23:53:12.795718+0000","last_peered":"2023-10-24T23:53:12.795718+0000","last_clean":"2023-10-24T23:53:12.795718+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-24T23:53:12.795718+0000","last_undegraded":"2023-10-24T23:53:12.795718+0000","last_fullsized":"2023-10-24T23:53:12.795718+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:51:05.443664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602086+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-24T23:53:13.602086+0000","last_peered":"2023-10-24T23:53:13.602086+0000","last_clean":"2023-10-24T23:53:13.602086+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-24T23:53:13.602086+0000","last_undegraded":"2023-10-24T23:53:13.602086+0000","last_fullsized":"2023-10-24T23:53:13.602086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:37:30.278659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825877+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-24T23:53:11.825877+0000","last_peered":"2023-10-24T23:53:11.825877+0000","last_clean":"2023-10-24T23:53:11.825877+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-24T23:53:11.825877+0000","last_undegraded":"2023-10-24T23:53:11.825877+0000","last_fullsized":"2023-10-24T23:53:11.825877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791237+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-24T23:53:11.791237+0000","last_peered":"2023-10-24T23:53:11.791237+0000","last_clean":"2023-10-24T23:53:11.791237+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-24T23:53:11.791237+0000","last_undegraded":"2023-10-24T23:53:11.791237+0000","last_fullsized":"2023-10-24T23:53:11.791237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794303+0000","last_change":"2023-10-24T23:53:11.429694+0000","last_active":"2023-10-24T23:53:12.794303+0000","last_peered":"2023-10-24T23:53:12.794303+0000","last_clean":"2023-10-24T23:53:12.794303+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-24T23:53:12.794303+0000","last_undegraded":"2023-10-24T23:53:12.794303+0000","last_fullsized":"2023-10-24T23:53:12.794303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:15.964994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793903+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-24T23:53:12.793903+0000","last_peered":"2023-10-24T23:53:12.793903+0000","last_clean":"2023-10-24T23:53:12.793903+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-24T23:53:12.793903+0000","last_undegraded":"2023-10-24T23:53:12.793903+0000","last_fullsized":"2023-10-24T23:53:12.793903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:38:18.678087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803275+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-24T23:53:11.803275+0000","last_peered":"2023-10-24T23:53:11.803275+0000","last_clean":"2023-10-24T23:53:11.803275+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-24T23:53:11.803275+0000","last_undegraded":"2023-10-24T23:53:11.803275+0000","last_fullsized":"2023-10-24T23:53:11.803275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800342+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-24T23:53:11.800342+0000","last_peered":"2023-10-24T23:53:11.800342+0000","last_clean":"2023-10-24T23:53:11.800342+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-24T23:53:11.800342+0000","last_undegraded":"2023-10-24T23:53:11.800342+0000","last_fullsized":"2023-10-24T23:53:11.800342+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793830+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-24T23:53:12.793830+0000","last_peered":"2023-10-24T23:53:12.793830+0000","last_clean":"2023-10-24T23:53:12.793830+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-24T23:53:12.793830+0000","last_undegraded":"2023-10-24T23:53:12.793830+0000","last_fullsized":"2023-10-24T23:53:12.793830+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:30.163913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797383+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-24T23:53:12.797383+0000","last_peered":"2023-10-24T23:53:12.797383+0000","last_clean":"2023-10-24T23:53:12.797383+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-24T23:53:12.797383+0000","last_undegraded":"2023-10-24T23:53:12.797383+0000","last_fullsized":"2023-10-24T23:53:12.797383+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:46.000330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.802095+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-24T23:53:11.802095+0000","last_peered":"2023-10-24T23:53:11.802095+0000","last_clean":"2023-10-24T23:53:11.802095+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-24T23:53:11.802095+0000","last_undegraded":"2023-10-24T23:53:11.802095+0000","last_fullsized":"2023-10-24T23:53:11.802095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.792050+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:11.792050+0000","last_peered":"2023-10-24T23:53:11.792050+0000","last_clean":"2023-10-24T23:53:11.792050+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:11.792050+0000","last_undegraded":"2023-10-24T23:53:11.792050+0000","last_fullsized":"2023-10-24T23:53:11.792050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788646+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-24T23:53:11.788646+0000","last_peered":"2023-10-24T23:53:11.788646+0000","last_clean":"2023-10-24T23:53:11.788646+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-24T23:53:11.788646+0000","last_undegraded":"2023-10-24T23:53:11.788646+0000","last_fullsized":"2023-10-24T23:53:11.788646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793988+0000","last_change":"2023-10-24T23:53:11.433306+0000","last_active":"2023-10-24T23:53:12.793988+0000","last_peered":"2023-10-24T23:53:12.793988+0000","last_clean":"2023-10-24T23:53:12.793988+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-24T23:53:12.793988+0000","last_undegraded":"2023-10-24T23:53:12.793988+0000","last_fullsized":"2023-10-24T23:53:12.793988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:44:38.728345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604210+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-24T23:53:13.604210+0000","last_peered":"2023-10-24T23:53:13.604210+0000","last_clean":"2023-10-24T23:53:13.604210+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-24T23:53:13.604210+0000","last_undegraded":"2023-10-24T23:53:13.604210+0000","last_fullsized":"2023-10-24T23:53:13.604210+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:30:24.805861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803946+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-24T23:53:11.803946+0000","last_peered":"2023-10-24T23:53:11.803946+0000","last_clean":"2023-10-24T23:53:11.803946+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-24T23:53:11.803946+0000","last_undegraded":"2023-10-24T23:53:11.803946+0000","last_fullsized":"2023-10-24T23:53:11.803946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790715+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:11.790715+0000","last_peered":"2023-10-24T23:53:11.790715+0000","last_clean":"2023-10-24T23:53:11.790715+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:11.790715+0000","last_undegraded":"2023-10-24T23:53:11.790715+0000","last_fullsized":"2023-10-24T23:53:11.790715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807502+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-24T23:53:11.807502+0000","last_peered":"2023-10-24T23:53:11.807502+0000","last_clean":"2023-10-24T23:53:11.807502+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-24T23:53:11.807502+0000","last_undegraded":"2023-10-24T23:53:11.807502+0000","last_fullsized":"2023-10-24T23:53:11.807502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602440+0000","last_change":"2023-10-24T23:53:11.441983+0000","last_active":"2023-10-24T23:53:13.602440+0000","last_peered":"2023-10-24T23:53:13.602440+0000","last_clean":"2023-10-24T23:53:13.602440+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-24T23:53:13.602440+0000","last_undegraded":"2023-10-24T23:53:13.602440+0000","last_fullsized":"2023-10-24T23:53:13.602440+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:30.643019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794864+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-24T23:53:12.794864+0000","last_peered":"2023-10-24T23:53:12.794864+0000","last_clean":"2023-10-24T23:53:12.794864+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-24T23:53:12.794864+0000","last_undegraded":"2023-10-24T23:53:12.794864+0000","last_fullsized":"2023-10-24T23:53:12.794864+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:50:53.467058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788379+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-24T23:53:11.788379+0000","last_peered":"2023-10-24T23:53:11.788379+0000","last_clean":"2023-10-24T23:53:11.788379+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-24T23:53:11.788379+0000","last_undegraded":"2023-10-24T23:53:11.788379+0000","last_fullsized":"2023-10-24T23:53:11.788379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.826147+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:11.826147+0000","last_peered":"2023-10-24T23:53:11.826147+0000","last_clean":"2023-10-24T23:53:11.826147+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:11.826147+0000","last_undegraded":"2023-10-24T23:53:11.826147+0000","last_fullsized":"2023-10-24T23:53:11.826147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790306+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-24T23:53:11.790306+0000","last_peered":"2023-10-24T23:53:11.790306+0000","last_clean":"2023-10-24T23:53:11.790306+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-24T23:53:11.790306+0000","last_undegraded":"2023-10-24T23:53:11.790306+0000","last_fullsized":"2023-10-24T23:53:11.790306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602338+0000","last_change":"2023-10-24T23:53:11.426287+0000","last_active":"2023-10-24T23:53:13.602338+0000","last_peered":"2023-10-24T23:53:13.602338+0000","last_clean":"2023-10-24T23:53:13.602338+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-24T23:53:13.602338+0000","last_undegraded":"2023-10-24T23:53:13.602338+0000","last_fullsized":"2023-10-24T23:53:13.602338+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:02:39.399866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602429+0000","last_change":"2023-10-24T23:53:11.811139+0000","last_active":"2023-10-24T23:53:13.602429+0000","last_peered":"2023-10-24T23:53:13.602429+0000","last_clean":"2023-10-24T23:53:13.602429+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-24T23:53:13.602429+0000","last_undegraded":"2023-10-24T23:53:13.602429+0000","last_fullsized":"2023-10-24T23:53:13.602429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:05.507633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791317+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-24T23:53:11.791317+0000","last_peered":"2023-10-24T23:53:11.791317+0000","last_clean":"2023-10-24T23:53:11.791317+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-24T23:53:11.791317+0000","last_undegraded":"2023-10-24T23:53:11.791317+0000","last_fullsized":"2023-10-24T23:53:11.791317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790574+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:11.790574+0000","last_peered":"2023-10-24T23:53:11.790574+0000","last_clean":"2023-10-24T23:53:11.790574+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:11.790574+0000","last_undegraded":"2023-10-24T23:53:11.790574+0000","last_fullsized":"2023-10-24T23:53:11.790574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832916+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-24T23:53:11.832916+0000","last_peered":"2023-10-24T23:53:11.832916+0000","last_clean":"2023-10-24T23:53:11.832916+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-24T23:53:11.832916+0000","last_undegraded":"2023-10-24T23:53:11.832916+0000","last_fullsized":"2023-10-24T23:53:11.832916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602790+0000","last_change":"2023-10-24T23:53:11.426199+0000","last_active":"2023-10-24T23:53:13.602790+0000","last_peered":"2023-10-24T23:53:13.602790+0000","last_clean":"2023-10-24T23:53:13.602790+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-24T23:53:13.602790+0000","last_undegraded":"2023-10-24T23:53:13.602790+0000","last_fullsized":"2023-10-24T23:53:13.602790+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:57:51.018816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794361+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-24T23:53:12.794361+0000","last_peered":"2023-10-24T23:53:12.794361+0000","last_clean":"2023-10-24T23:53:12.794361+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-24T23:53:12.794361+0000","last_undegraded":"2023-10-24T23:53:12.794361+0000","last_fullsized":"2023-10-24T23:53:12.794361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:20:16.596993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790230+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-24T23:53:11.790230+0000","last_peered":"2023-10-24T23:53:11.790230+0000","last_clean":"2023-10-24T23:53:11.790230+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-24T23:53:11.790230+0000","last_undegraded":"2023-10-24T23:53:11.790230+0000","last_fullsized":"2023-10-24T23:53:11.790230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800496+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-24T23:53:11.800496+0000","last_peered":"2023-10-24T23:53:11.800496+0000","last_clean":"2023-10-24T23:53:11.800496+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-24T23:53:11.800496+0000","last_undegraded":"2023-10-24T23:53:11.800496+0000","last_fullsized":"2023-10-24T23:53:11.800496+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828620+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:53:11.828620+0000","last_peered":"2023-10-24T23:53:11.828620+0000","last_clean":"2023-10-24T23:53:11.828620+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:53:11.828620+0000","last_undegraded":"2023-10-24T23:53:11.828620+0000","last_fullsized":"2023-10-24T23:53:11.828620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603076+0000","last_change":"2023-10-24T23:53:11.432666+0000","last_active":"2023-10-24T23:53:13.603076+0000","last_peered":"2023-10-24T23:53:13.603076+0000","last_clean":"2023-10-24T23:53:13.603076+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-24T23:53:13.603076+0000","last_undegraded":"2023-10-24T23:53:13.603076+0000","last_fullsized":"2023-10-24T23:53:13.603076+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:17.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797676+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-24T23:53:12.797676+0000","last_peered":"2023-10-24T23:53:12.797676+0000","last_clean":"2023-10-24T23:53:12.797676+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-24T23:53:12.797676+0000","last_undegraded":"2023-10-24T23:53:12.797676+0000","last_fullsized":"2023-10-24T23:53:12.797676+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:42.131555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789311+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-24T23:53:11.789311+0000","last_peered":"2023-10-24T23:53:11.789311+0000","last_clean":"2023-10-24T23:53:11.789311+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-24T23:53:11.789311+0000","last_undegraded":"2023-10-24T23:53:11.789311+0000","last_fullsized":"2023-10-24T23:53:11.789311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789825+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-24T23:53:11.789825+0000","last_peered":"2023-10-24T23:53:11.789825+0000","last_clean":"2023-10-24T23:53:11.789825+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-24T23:53:11.789825+0000","last_undegraded":"2023-10-24T23:53:11.789825+0000","last_fullsized":"2023-10-24T23:53:11.789825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.795048+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:53:11.795048+0000","last_peered":"2023-10-24T23:53:11.795048+0000","last_clean":"2023-10-24T23:53:11.795048+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:53:11.795048+0000","last_undegraded":"2023-10-24T23:53:11.795048+0000","last_fullsized":"2023-10-24T23:53:11.795048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"27'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.279161+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-24T23:53:13.279161+0000","last_peered":"2023-10-24T23:53:13.279161+0000","last_clean":"2023-10-24T23:53:13.279161+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-24T23:53:13.279161+0000","last_undegraded":"2023-10-24T23:53:13.279161+0000","last_fullsized":"2023-10-24T23:53:13.279161+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:55:47.548432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795507+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-24T23:53:12.795507+0000","last_peered":"2023-10-24T23:53:12.795507+0000","last_clean":"2023-10-24T23:53:12.795507+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-24T23:53:12.795507+0000","last_undegraded":"2023-10-24T23:53:12.795507+0000","last_fullsized":"2023-10-24T23:53:12.795507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:11:14.431984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788576+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-24T23:53:11.788576+0000","last_peered":"2023-10-24T23:53:11.788576+0000","last_clean":"2023-10-24T23:53:11.788576+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-24T23:53:11.788576+0000","last_undegraded":"2023-10-24T23:53:11.788576+0000","last_fullsized":"2023-10-24T23:53:11.788576+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789662+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-24T23:53:11.789662+0000","last_peered":"2023-10-24T23:53:11.789662+0000","last_clean":"2023-10-24T23:53:11.789662+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-24T23:53:11.789662+0000","last_undegraded":"2023-10-24T23:53:11.789662+0000","last_fullsized":"2023-10-24T23:53:11.789662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828080+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:11.828080+0000","last_peered":"2023-10-24T23:53:11.828080+0000","last_clean":"2023-10-24T23:53:11.828080+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:11.828080+0000","last_undegraded":"2023-10-24T23:53:11.828080+0000","last_fullsized":"2023-10-24T23:53:11.828080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102659+0000","last_change":"2023-10-24T23:53:11.436603+0000","last_active":"2023-10-24T23:53:13.102659+0000","last_peered":"2023-10-24T23:53:13.102659+0000","last_clean":"2023-10-24T23:53:13.102659+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-24T23:53:13.102659+0000","last_undegraded":"2023-10-24T23:53:13.102659+0000","last_fullsized":"2023-10-24T23:53:13.102659+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:09:05.948928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102438+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-24T23:53:13.102438+0000","last_peered":"2023-10-24T23:53:13.102438+0000","last_clean":"2023-10-24T23:53:13.102438+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-24T23:53:13.102438+0000","last_undegraded":"2023-10-24T23:53:13.102438+0000","last_fullsized":"2023-10-24T23:53:13.102438+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:01:59.639080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793650+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-24T23:53:11.793650+0000","last_peered":"2023-10-24T23:53:11.793650+0000","last_clean":"2023-10-24T23:53:11.793650+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-24T23:53:11.793650+0000","last_undegraded":"2023-10-24T23:53:11.793650+0000","last_fullsized":"2023-10-24T23:53:11.793650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807120+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-24T23:53:11.807120+0000","last_peered":"2023-10-24T23:53:11.807120+0000","last_clean":"2023-10-24T23:53:11.807120+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-24T23:53:11.807120+0000","last_undegraded":"2023-10-24T23:53:11.807120+0000","last_fullsized":"2023-10-24T23:53:11.807120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825313+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:53:11.825313+0000","last_peered":"2023-10-24T23:53:11.825313+0000","last_clean":"2023-10-24T23:53:11.825313+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:53:11.825313+0000","last_undegraded":"2023-10-24T23:53:11.825313+0000","last_fullsized":"2023-10-24T23:53:11.825313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.834350+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:53:11.834350+0000","last_peered":"2023-10-24T23:53:11.834350+0000","last_clean":"2023-10-24T23:53:11.834350+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:53:11.834350+0000","last_undegraded":"2023-10-24T23:53:11.834350+0000","last_fullsized":"2023-10-24T23:53:11.834350+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797622+0000","last_change":"2023-10-24T23:53:11.451073+0000","last_active":"2023-10-24T23:53:12.797622+0000","last_peered":"2023-10-24T23:53:12.797622+0000","last_clean":"2023-10-24T23:53:12.797622+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-24T23:53:12.797622+0000","last_undegraded":"2023-10-24T23:53:12.797622+0000","last_fullsized":"2023-10-24T23:53:12.797622+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:32.324710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101638+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-24T23:53:13.101638+0000","last_peered":"2023-10-24T23:53:13.101638+0000","last_clean":"2023-10-24T23:53:13.101638+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-24T23:53:13.101638+0000","last_undegraded":"2023-10-24T23:53:13.101638+0000","last_fullsized":"2023-10-24T23:53:13.101638+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:04.125835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789210+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-24T23:53:11.789210+0000","last_peered":"2023-10-24T23:53:11.789210+0000","last_clean":"2023-10-24T23:53:11.789210+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-24T23:53:11.789210+0000","last_undegraded":"2023-10-24T23:53:11.789210+0000","last_fullsized":"2023-10-24T23:53:11.789210+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804119+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-24T23:53:11.804119+0000","last_peered":"2023-10-24T23:53:11.804119+0000","last_clean":"2023-10-24T23:53:11.804119+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-24T23:53:11.804119+0000","last_undegraded":"2023-10-24T23:53:11.804119+0000","last_fullsized":"2023-10-24T23:53:11.804119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605414+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-24T23:53:13.605414+0000","last_peered":"2023-10-24T23:53:13.605414+0000","last_clean":"2023-10-24T23:53:13.605414+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-24T23:53:13.605414+0000","last_undegraded":"2023-10-24T23:53:13.605414+0000","last_fullsized":"2023-10-24T23:53:13.605414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:06:53.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794699+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-24T23:53:12.794699+0000","last_peered":"2023-10-24T23:53:12.794699+0000","last_clean":"2023-10-24T23:53:12.794699+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-24T23:53:12.794699+0000","last_undegraded":"2023-10-24T23:53:12.794699+0000","last_fullsized":"2023-10-24T23:53:12.794699+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:41.905772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804055+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-24T23:53:11.804055+0000","last_peered":"2023-10-24T23:53:11.804055+0000","last_clean":"2023-10-24T23:53:11.804055+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-24T23:53:11.804055+0000","last_undegraded":"2023-10-24T23:53:11.804055+0000","last_fullsized":"2023-10-24T23:53:11.804055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.802309+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-24T23:53:11.802309+0000","last_peered":"2023-10-24T23:53:11.802309+0000","last_clean":"2023-10-24T23:53:11.802309+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-24T23:53:11.802309+0000","last_undegraded":"2023-10-24T23:53:11.802309+0000","last_fullsized":"2023-10-24T23:53:11.802309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102244+0000","last_change":"2023-10-24T23:53:11.458463+0000","last_active":"2023-10-24T23:53:13.102244+0000","last_peered":"2023-10-24T23:53:13.102244+0000","last_clean":"2023-10-24T23:53:13.102244+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-24T23:53:13.102244+0000","last_undegraded":"2023-10-24T23:53:13.102244+0000","last_fullsized":"2023-10-24T23:53:13.102244+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:11:45.040156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275727+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-24T23:53:13.275727+0000","last_peered":"2023-10-24T23:53:13.275727+0000","last_clean":"2023-10-24T23:53:13.275727+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-24T23:53:13.275727+0000","last_undegraded":"2023-10-24T23:53:13.275727+0000","last_fullsized":"2023-10-24T23:53:13.275727+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:51.707178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800564+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-24T23:53:11.800564+0000","last_peered":"2023-10-24T23:53:11.800564+0000","last_clean":"2023-10-24T23:53:11.800564+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-24T23:53:11.800564+0000","last_undegraded":"2023-10-24T23:53:11.800564+0000","last_fullsized":"2023-10-24T23:53:11.800564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790875+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-24T23:53:11.790875+0000","last_peered":"2023-10-24T23:53:11.790875+0000","last_clean":"2023-10-24T23:53:11.790875+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-24T23:53:11.790875+0000","last_undegraded":"2023-10-24T23:53:11.790875+0000","last_fullsized":"2023-10-24T23:53:11.790875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795610+0000","last_change":"2023-10-24T23:53:11.448675+0000","last_active":"2023-10-24T23:53:12.795610+0000","last_peered":"2023-10-24T23:53:12.795610+0000","last_clean":"2023-10-24T23:53:12.795610+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-24T23:53:12.795610+0000","last_undegraded":"2023-10-24T23:53:12.795610+0000","last_fullsized":"2023-10-24T23:53:12.795610+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:01.502343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605168+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-24T23:53:13.605168+0000","last_peered":"2023-10-24T23:53:13.605168+0000","last_clean":"2023-10-24T23:53:13.605168+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-24T23:53:13.605168+0000","last_undegraded":"2023-10-24T23:53:13.605168+0000","last_fullsized":"2023-10-24T23:53:13.605168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:54.478897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793341+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-24T23:53:11.793341+0000","last_peered":"2023-10-24T23:53:11.793341+0000","last_clean":"2023-10-24T23:53:11.793341+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-24T23:53:11.793341+0000","last_undegraded":"2023-10-24T23:53:11.793341+0000","last_fullsized":"2023-10-24T23:53:11.793341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807904+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-24T23:53:11.807904+0000","last_peered":"2023-10-24T23:53:11.807904+0000","last_clean":"2023-10-24T23:53:11.807904+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-24T23:53:11.807904+0000","last_undegraded":"2023-10-24T23:53:11.807904+0000","last_fullsized":"2023-10-24T23:53:11.807904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275531+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-24T23:53:13.275531+0000","last_peered":"2023-10-24T23:53:13.275531+0000","last_clean":"2023-10-24T23:53:13.275531+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-24T23:53:13.275531+0000","last_undegraded":"2023-10-24T23:53:13.275531+0000","last_fullsized":"2023-10-24T23:53:13.275531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:58.545886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602449+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-24T23:53:13.602449+0000","last_peered":"2023-10-24T23:53:13.602449+0000","last_clean":"2023-10-24T23:53:13.602449+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-24T23:53:13.602449+0000","last_undegraded":"2023-10-24T23:53:13.602449+0000","last_fullsized":"2023-10-24T23:53:13.602449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:05.595887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788253+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-24T23:53:11.788253+0000","last_peered":"2023-10-24T23:53:11.788253+0000","last_clean":"2023-10-24T23:53:11.788253+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-24T23:53:11.788253+0000","last_undegraded":"2023-10-24T23:53:11.788253+0000","last_fullsized":"2023-10-24T23:53:11.788253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.830849+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-24T23:53:11.830849+0000","last_peered":"2023-10-24T23:53:11.830849+0000","last_clean":"2023-10-24T23:53:11.830849+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-24T23:53:11.830849+0000","last_undegraded":"2023-10-24T23:53:11.830849+0000","last_fullsized":"2023-10-24T23:53:11.830849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275166+0000","last_change":"2023-10-24T23:53:11.454658+0000","last_active":"2023-10-24T23:53:13.275166+0000","last_peered":"2023-10-24T23:53:13.275166+0000","last_clean":"2023-10-24T23:53:13.275166+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-24T23:53:13.275166+0000","last_undegraded":"2023-10-24T23:53:13.275166+0000","last_fullsized":"2023-10-24T23:53:13.275166+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:43:43.869069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796376+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-24T23:53:12.796376+0000","last_peered":"2023-10-24T23:53:12.796376+0000","last_clean":"2023-10-24T23:53:12.796376+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-24T23:53:12.796376+0000","last_undegraded":"2023-10-24T23:53:12.796376+0000","last_fullsized":"2023-10-24T23:53:12.796376+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:09:40.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790517+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-24T23:53:11.790517+0000","last_peered":"2023-10-24T23:53:11.790517+0000","last_clean":"2023-10-24T23:53:11.790517+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-24T23:53:11.790517+0000","last_undegraded":"2023-10-24T23:53:11.790517+0000","last_fullsized":"2023-10-24T23:53:11.790517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825937+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-24T23:53:11.825937+0000","last_peered":"2023-10-24T23:53:11.825937+0000","last_clean":"2023-10-24T23:53:11.825937+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-24T23:53:11.825937+0000","last_undegraded":"2023-10-24T23:53:11.825937+0000","last_fullsized":"2023-10-24T23:53:11.825937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275899+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-24T23:53:13.275899+0000","last_peered":"2023-10-24T23:53:13.275899+0000","last_clean":"2023-10-24T23:53:13.275899+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-24T23:53:13.275899+0000","last_undegraded":"2023-10-24T23:53:13.275899+0000","last_fullsized":"2023-10-24T23:53:13.275899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:02.591627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794746+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-24T23:53:12.794746+0000","last_peered":"2023-10-24T23:53:12.794746+0000","last_clean":"2023-10-24T23:53:12.794746+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-24T23:53:12.794746+0000","last_undegraded":"2023-10-24T23:53:12.794746+0000","last_fullsized":"2023-10-24T23:53:12.794746+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:43.015050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.801731+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-24T23:53:11.801731+0000","last_peered":"2023-10-24T23:53:11.801731+0000","last_clean":"2023-10-24T23:53:11.801731+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-24T23:53:11.801731+0000","last_undegraded":"2023-10-24T23:53:11.801731+0000","last_fullsized":"2023-10-24T23:53:11.801731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825206+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-24T23:53:11.825206+0000","last_peered":"2023-10-24T23:53:11.825206+0000","last_clean":"2023-10-24T23:53:11.825206+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-24T23:53:11.825206+0000","last_undegraded":"2023-10-24T23:53:11.825206+0000","last_fullsized":"2023-10-24T23:53:11.825206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.274975+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-24T23:53:13.274975+0000","last_peered":"2023-10-24T23:53:13.274975+0000","last_clean":"2023-10-24T23:53:13.274975+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-24T23:53:13.274975+0000","last_undegraded":"2023-10-24T23:53:13.274975+0000","last_fullsized":"2023-10-24T23:53:13.274975+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:59.068359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.100994+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-24T23:53:13.100994+0000","last_peered":"2023-10-24T23:53:13.100994+0000","last_clean":"2023-10-24T23:53:13.100994+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-24T23:53:13.100994+0000","last_undegraded":"2023-10-24T23:53:13.100994+0000","last_fullsized":"2023-10-24T23:53:13.100994+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:13:03.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806468+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-24T23:53:11.806468+0000","last_peered":"2023-10-24T23:53:11.806468+0000","last_clean":"2023-10-24T23:53:11.806468+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-24T23:53:11.806468+0000","last_undegraded":"2023-10-24T23:53:11.806468+0000","last_fullsized":"2023-10-24T23:53:11.806468+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.827865+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-24T23:53:11.827865+0000","last_peered":"2023-10-24T23:53:11.827865+0000","last_clean":"2023-10-24T23:53:11.827865+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-24T23:53:11.827865+0000","last_undegraded":"2023-10-24T23:53:11.827865+0000","last_fullsized":"2023-10-24T23:53:11.827865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796200+0000","last_change":"2023-10-24T23:53:11.451013+0000","last_active":"2023-10-24T23:53:12.796200+0000","last_peered":"2023-10-24T23:53:12.796200+0000","last_clean":"2023-10-24T23:53:12.796200+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-24T23:53:12.796200+0000","last_undegraded":"2023-10-24T23:53:12.796200+0000","last_fullsized":"2023-10-24T23:53:12.796200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:52:22.503023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797682+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-24T23:53:12.797682+0000","last_peered":"2023-10-24T23:53:12.797682+0000","last_clean":"2023-10-24T23:53:12.797682+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-24T23:53:12.797682+0000","last_undegraded":"2023-10-24T23:53:12.797682+0000","last_fullsized":"2023-10-24T23:53:12.797682+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:05:30.149693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102137+0000","last_change":"2023-10-24T23:53:11.434538+0000","last_active":"2023-10-24T23:53:13.102137+0000","last_peered":"2023-10-24T23:53:13.102137+0000","last_clean":"2023-10-24T23:53:13.102137+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-24T23:53:13.102137+0000","last_undegraded":"2023-10-24T23:53:13.102137+0000","last_fullsized":"2023-10-24T23:53:13.102137+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:55.179253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789405+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-24T23:53:11.789405+0000","last_peered":"2023-10-24T23:53:11.789405+0000","last_clean":"2023-10-24T23:53:11.789405+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-24T23:53:11.789405+0000","last_undegraded":"2023-10-24T23:53:11.789405+0000","last_fullsized":"2023-10-24T23:53:11.789405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807405+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-24T23:53:11.807405+0000","last_peered":"2023-10-24T23:53:11.807405+0000","last_clean":"2023-10-24T23:53:11.807405+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-24T23:53:11.807405+0000","last_undegraded":"2023-10-24T23:53:11.807405+0000","last_fullsized":"2023-10-24T23:53:11.807405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275539+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-24T23:53:13.275539+0000","last_peered":"2023-10-24T23:53:13.275539+0000","last_clean":"2023-10-24T23:53:13.275539+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-24T23:53:13.275539+0000","last_undegraded":"2023-10-24T23:53:13.275539+0000","last_fullsized":"2023-10-24T23:53:13.275539+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:33:08.373963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795605+0000","last_change":"2023-10-24T23:53:11.433387+0000","last_active":"2023-10-24T23:53:12.795605+0000","last_peered":"2023-10-24T23:53:12.795605+0000","last_clean":"2023-10-24T23:53:12.795605+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-24T23:53:12.795605+0000","last_undegraded":"2023-10-24T23:53:12.795605+0000","last_fullsized":"2023-10-24T23:53:12.795605+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:29.148450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.824306+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-24T23:53:11.824306+0000","last_peered":"2023-10-24T23:53:11.824306+0000","last_clean":"2023-10-24T23:53:11.824306+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-24T23:53:11.824306+0000","last_undegraded":"2023-10-24T23:53:11.824306+0000","last_fullsized":"2023-10-24T23:53:11.824306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788623+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-24T23:53:11.788623+0000","last_peered":"2023-10-24T23:53:11.788623+0000","last_clean":"2023-10-24T23:53:11.788623+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-24T23:53:11.788623+0000","last_undegraded":"2023-10-24T23:53:11.788623+0000","last_fullsized":"2023-10-24T23:53:11.788623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605588+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-24T23:53:13.605588+0000","last_peered":"2023-10-24T23:53:13.605588+0000","last_clean":"2023-10-24T23:53:13.605588+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-24T23:53:13.605588+0000","last_undegraded":"2023-10-24T23:53:13.605588+0000","last_fullsized":"2023-10-24T23:53:13.605588+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:33:06.172412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788704+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-24T23:53:11.788704+0000","last_peered":"2023-10-24T23:53:11.788704+0000","last_clean":"2023-10-24T23:53:11.788704+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-24T23:53:11.788704+0000","last_undegraded":"2023-10-24T23:53:11.788704+0000","last_fullsized":"2023-10-24T23:53:11.788704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832501+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-24T23:53:11.832501+0000","last_peered":"2023-10-24T23:53:11.832501+0000","last_clean":"2023-10-24T23:53:11.832501+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-24T23:53:11.832501+0000","last_undegraded":"2023-10-24T23:53:11.832501+0000","last_fullsized":"2023-10-24T23:53:11.832501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"26'6","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794898+0000","last_change":"2023-10-24T23:53:11.446979+0000","last_active":"2023-10-24T23:53:12.794898+0000","last_peered":"2023-10-24T23:53:12.794898+0000","last_clean":"2023-10-24T23:53:12.794898+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-24T23:53:12.794898+0000","last_undegraded":"2023-10-24T23:53:12.794898+0000","last_fullsized":"2023-10-24T23:53:12.794898+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:54:52.843631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603260+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-24T23:53:13.603260+0000","last_peered":"2023-10-24T23:53:13.603260+0000","last_clean":"2023-10-24T23:53:13.603260+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-24T23:53:13.603260+0000","last_undegraded":"2023-10-24T23:53:13.603260+0000","last_fullsized":"2023-10-24T23:53:13.603260+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:18:48.679682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.816486+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-24T23:53:11.816486+0000","last_peered":"2023-10-24T23:53:11.816486+0000","last_clean":"2023-10-24T23:53:11.816486+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-24T23:53:11.816486+0000","last_undegraded":"2023-10-24T23:53:11.816486+0000","last_fullsized":"2023-10-24T23:53:11.816486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807790+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-24T23:53:11.807790+0000","last_peered":"2023-10-24T23:53:11.807790+0000","last_clean":"2023-10-24T23:53:11.807790+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-24T23:53:11.807790+0000","last_undegraded":"2023-10-24T23:53:11.807790+0000","last_fullsized":"2023-10-24T23:53:11.807790+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101724+0000","last_change":"2023-10-24T23:53:11.436964+0000","last_active":"2023-10-24T23:53:13.101724+0000","last_peered":"2023-10-24T23:53:13.101724+0000","last_clean":"2023-10-24T23:53:13.101724+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-24T23:53:13.101724+0000","last_undegraded":"2023-10-24T23:53:13.101724+0000","last_fullsized":"2023-10-24T23:53:13.101724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:04:15.366966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602222+0000","last_change":"2023-10-24T23:53:11.814304+0000","last_active":"2023-10-24T23:53:13.602222+0000","last_peered":"2023-10-24T23:53:13.602222+0000","last_clean":"2023-10-24T23:53:13.602222+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-24T23:53:13.602222+0000","last_undegraded":"2023-10-24T23:53:13.602222+0000","last_fullsized":"2023-10-24T23:53:13.602222+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:16.635178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789162+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-24T23:53:11.789162+0000","last_peered":"2023-10-24T23:53:11.789162+0000","last_clean":"2023-10-24T23:53:11.789162+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-24T23:53:11.789162+0000","last_undegraded":"2023-10-24T23:53:11.789162+0000","last_fullsized":"2023-10-24T23:53:11.789162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603105+0000","last_change":"2023-10-24T23:53:11.807096+0000","last_active":"2023-10-24T23:53:13.603105+0000","last_peered":"2023-10-24T23:53:13.603105+0000","last_clean":"2023-10-24T23:53:13.603105+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-24T23:53:13.603105+0000","last_undegraded":"2023-10-24T23:53:13.603105+0000","last_fullsized":"2023-10-24T23:53:13.603105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:25.144578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804445+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-24T23:53:11.804445+0000","last_peered":"2023-10-24T23:53:11.804445+0000","last_clean":"2023-10-24T23:53:11.804445+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-24T23:53:11.804445+0000","last_undegraded":"2023-10-24T23:53:11.804445+0000","last_fullsized":"2023-10-24T23:53:11.804445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793680+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-24T23:53:12.793680+0000","last_peered":"2023-10-24T23:53:12.793680+0000","last_clean":"2023-10-24T23:53:12.793680+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-24T23:53:12.793680+0000","last_undegraded":"2023-10-24T23:53:12.793680+0000","last_fullsized":"2023-10-24T23:53:12.793680+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:38.946585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.811071+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-24T23:53:11.811071+0000","last_peered":"2023-10-24T23:53:11.811071+0000","last_clean":"2023-10-24T23:53:11.811071+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-24T23:53:11.811071+0000","last_undegraded":"2023-10-24T23:53:11.811071+0000","last_fullsized":"2023-10-24T23:53:11.811071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797036+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-24T23:53:12.797036+0000","last_peered":"2023-10-24T23:53:12.797036+0000","last_clean":"2023-10-24T23:53:12.797036+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-24T23:53:12.797036+0000","last_undegraded":"2023-10-24T23:53:12.797036+0000","last_fullsized":"2023-10-24T23:53:12.797036+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:48:24.911279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807673+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-24T23:53:11.807673+0000","last_peered":"2023-10-24T23:53:11.807673+0000","last_clean":"2023-10-24T23:53:11.807673+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-24T23:53:11.807673+0000","last_undegraded":"2023-10-24T23:53:11.807673+0000","last_fullsized":"2023-10-24T23:53:11.807673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795515+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-24T23:53:12.795515+0000","last_peered":"2023-10-24T23:53:12.795515+0000","last_clean":"2023-10-24T23:53:12.795515+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-24T23:53:12.795515+0000","last_undegraded":"2023-10-24T23:53:12.795515+0000","last_fullsized":"2023-10-24T23:53:12.795515+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:21.165755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790400+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-24T23:53:11.790400+0000","last_peered":"2023-10-24T23:53:11.790400+0000","last_clean":"2023-10-24T23:53:11.790400+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-24T23:53:11.790400+0000","last_undegraded":"2023-10-24T23:53:11.790400+0000","last_fullsized":"2023-10-24T23:53:11.790400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604073+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-24T23:53:13.604073+0000","last_peered":"2023-10-24T23:53:13.604073+0000","last_clean":"2023-10-24T23:53:13.604073+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-24T23:53:13.604073+0000","last_undegraded":"2023-10-24T23:53:13.604073+0000","last_fullsized":"2023-10-24T23:53:13.604073+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832305+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-24T23:53:11.832305+0000","last_peered":"2023-10-24T23:53:11.832305+0000","last_clean":"2023-10-24T23:53:11.832305+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-24T23:53:11.832305+0000","last_undegraded":"2023-10-24T23:53:11.832305+0000","last_fullsized":"2023-10-24T23:53:11.832305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101339+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-24T23:53:13.101339+0000","last_peered":"2023-10-24T23:53:13.101339+0000","last_clean":"2023-10-24T23:53:13.101339+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-24T23:53:13.101339+0000","last_undegraded":"2023-10-24T23:53:13.101339+0000","last_fullsized":"2023-10-24T23:53:13.101339+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:54.422365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790575+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-24T23:53:11.790575+0000","last_peered":"2023-10-24T23:53:11.790575+0000","last_clean":"2023-10-24T23:53:11.790575+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-24T23:53:11.790575+0000","last_undegraded":"2023-10-24T23:53:11.790575+0000","last_fullsized":"2023-10-24T23:53:11.790575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603531+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-24T23:53:13.603531+0000","last_peered":"2023-10-24T23:53:13.603531+0000","last_clean":"2023-10-24T23:53:13.603531+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-24T23:53:13.603531+0000","last_undegraded":"2023-10-24T23:53:13.603531+0000","last_fullsized":"2023-10-24T23:53:13.603531+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:01:11.838035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789166+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-24T23:53:11.789166+0000","last_peered":"2023-10-24T23:53:11.789166+0000","last_clean":"2023-10-24T23:53:11.789166+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-24T23:53:11.789166+0000","last_undegraded":"2023-10-24T23:53:11.789166+0000","last_fullsized":"2023-10-24T23:53:11.789166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603142+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-24T23:53:13.603142+0000","last_peered":"2023-10-24T23:53:13.603142+0000","last_clean":"2023-10-24T23:53:13.603142+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-24T23:53:13.603142+0000","last_undegraded":"2023-10-24T23:53:13.603142+0000","last_fullsized":"2023-10-24T23:53:13.603142+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:09:25.424344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.827005+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-24T23:53:11.827005+0000","last_peered":"2023-10-24T23:53:11.827005+0000","last_clean":"2023-10-24T23:53:11.827005+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-24T23:53:11.827005+0000","last_undegraded":"2023-10-24T23:53:11.827005+0000","last_fullsized":"2023-10-24T23:53:11.827005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101152+0000","last_change":"2023-10-24T23:53:11.824274+0000","last_active":"2023-10-24T23:53:13.101152+0000","last_peered":"2023-10-24T23:53:13.101152+0000","last_clean":"2023-10-24T23:53:13.101152+0000","last_became_active":"2023-10-24T23:53:11.823586+0000","last_became_peered":"2023-10-24T23:53:11.823586+0000","last_unstale":"2023-10-24T23:53:13.101152+0000","last_undegraded":"2023-10-24T23:53:13.101152+0000","last_fullsized":"2023-10-24T23:53:13.101152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:47:36.970336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804379+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-24T23:53:11.804379+0000","last_peered":"2023-10-24T23:53:11.804379+0000","last_clean":"2023-10-24T23:53:11.804379+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-24T23:53:11.804379+0000","last_undegraded":"2023-10-24T23:53:11.804379+0000","last_fullsized":"2023-10-24T23:53:11.804379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.274967+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-24T23:53:13.274967+0000","last_peered":"2023-10-24T23:53:13.274967+0000","last_clean":"2023-10-24T23:53:13.274967+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-24T23:53:13.274967+0000","last_undegraded":"2023-10-24T23:53:13.274967+0000","last_fullsized":"2023-10-24T23:53:13.274967+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:10:20.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788652+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-24T23:53:11.788652+0000","last_peered":"2023-10-24T23:53:11.788652+0000","last_clean":"2023-10-24T23:53:11.788652+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-24T23:53:11.788652+0000","last_undegraded":"2023-10-24T23:53:11.788652+0000","last_fullsized":"2023-10-24T23:53:11.788652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796443+0000","last_change":"2023-10-24T23:53:11.860133+0000","last_active":"2023-10-24T23:53:12.796443+0000","last_peered":"2023-10-24T23:53:12.796443+0000","last_clean":"2023-10-24T23:53:12.796443+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-24T23:53:12.796443+0000","last_undegraded":"2023-10-24T23:53:12.796443+0000","last_fullsized":"2023-10-24T23:53:12.796443+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:43.051362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.816202+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-24T23:53:11.816202+0000","last_peered":"2023-10-24T23:53:11.816202+0000","last_clean":"2023-10-24T23:53:11.816202+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-24T23:53:11.816202+0000","last_undegraded":"2023-10-24T23:53:11.816202+0000","last_fullsized":"2023-10-24T23:53:11.816202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603745+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-24T23:53:13.603745+0000","last_peered":"2023-10-24T23:53:13.603745+0000","last_clean":"2023-10-24T23:53:13.603745+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-24T23:53:13.603745+0000","last_undegraded":"2023-10-24T23:53:13.603745+0000","last_fullsized":"2023-10-24T23:53:13.603745+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:39:14.947372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793137+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-24T23:53:11.793137+0000","last_peered":"2023-10-24T23:53:11.793137+0000","last_clean":"2023-10-24T23:53:11.793137+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-24T23:53:11.793137+0000","last_undegraded":"2023-10-24T23:53:11.793137+0000","last_fullsized":"2023-10-24T23:53:11.793137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.798110+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-24T23:53:12.798110+0000","last_peered":"2023-10-24T23:53:12.798110+0000","last_clean":"2023-10-24T23:53:12.798110+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-24T23:53:12.798110+0000","last_undegraded":"2023-10-24T23:53:12.798110+0000","last_fullsized":"2023-10-24T23:53:12.798110+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:37.413259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.796910+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-24T23:53:11.796910+0000","last_peered":"2023-10-24T23:53:11.796910+0000","last_clean":"2023-10-24T23:53:11.796910+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-24T23:53:11.796910+0000","last_undegraded":"2023-10-24T23:53:11.796910+0000","last_fullsized":"2023-10-24T23:53:11.796910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603512+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-24T23:53:13.603512+0000","last_peered":"2023-10-24T23:53:13.603512+0000","last_clean":"2023-10-24T23:53:13.603512+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-24T23:53:13.603512+0000","last_undegraded":"2023-10-24T23:53:13.603512+0000","last_fullsized":"2023-10-24T23:53:13.603512+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:31.344724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803849+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-24T23:53:11.803849+0000","last_peered":"2023-10-24T23:53:11.803849+0000","last_clean":"2023-10-24T23:53:11.803849+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-24T23:53:11.803849+0000","last_undegraded":"2023-10-24T23:53:11.803849+0000","last_fullsized":"2023-10-24T23:53:11.803849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275469+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-24T23:53:13.275469+0000","last_peered":"2023-10-24T23:53:13.275469+0000","last_clean":"2023-10-24T23:53:13.275469+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-24T23:53:13.275469+0000","last_undegraded":"2023-10-24T23:53:13.275469+0000","last_fullsized":"2023-10-24T23:53:13.275469+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788160+0000","last_change":"2023-10-24T23:53:02.728971+0000","last_active":"2023-10-24T23:53:11.788160+0000","last_peered":"2023-10-24T23:53:11.788160+0000","last_clean":"2023-10-24T23:53:11.788160+0000","last_became_active":"2023-10-24T23:53:02.728048+0000","last_became_peered":"2023-10-24T23:53:02.728048+0000","last_unstale":"2023-10-24T23:53:11.788160+0000","last_undegraded":"2023-10-24T23:53:11.788160+0000","last_fullsized":"2023-10-24T23:53:11.788160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:31:32.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.601984+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-24T23:53:13.601984+0000","last_peered":"2023-10-24T23:53:13.601984+0000","last_clean":"2023-10-24T23:53:13.601984+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-24T23:53:13.601984+0000","last_undegraded":"2023-10-24T23:53:13.601984+0000","last_fullsized":"2023-10-24T23:53:13.601984+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:32:05.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791536+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-24T23:53:11.791536+0000","last_peered":"2023-10-24T23:53:11.791536+0000","last_clean":"2023-10-24T23:53:11.791536+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-24T23:53:11.791536+0000","last_undegraded":"2023-10-24T23:53:11.791536+0000","last_fullsized":"2023-10-24T23:53:11.791536+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101460+0000","last_change":"2023-10-24T23:53:11.847814+0000","last_active":"2023-10-24T23:53:13.101460+0000","last_peered":"2023-10-24T23:53:13.101460+0000","last_clean":"2023-10-24T23:53:13.101460+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-24T23:53:13.101460+0000","last_undegraded":"2023-10-24T23:53:13.101460+0000","last_fullsized":"2023-10-24T23:53:13.101460+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:59.075858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825376+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-24T23:53:11.825376+0000","last_peered":"2023-10-24T23:53:11.825376+0000","last_clean":"2023-10-24T23:53:11.825376+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-24T23:53:11.825376+0000","last_undegraded":"2023-10-24T23:53:11.825376+0000","last_fullsized":"2023-10-24T23:53:11.825376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796122+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-24T23:53:12.796122+0000","last_peered":"2023-10-24T23:53:12.796122+0000","last_clean":"2023-10-24T23:53:12.796122+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-24T23:53:12.796122+0000","last_undegraded":"2023-10-24T23:53:12.796122+0000","last_fullsized":"2023-10-24T23:53:12.796122+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:25:12.525273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828503+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-24T23:53:11.828503+0000","last_peered":"2023-10-24T23:53:11.828503+0000","last_clean":"2023-10-24T23:53:11.828503+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-24T23:53:11.828503+0000","last_undegraded":"2023-10-24T23:53:11.828503+0000","last_fullsized":"2023-10-24T23:53:11.828503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101011+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-24T23:53:13.101011+0000","last_peered":"2023-10-24T23:53:13.101011+0000","last_clean":"2023-10-24T23:53:13.101011+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-24T23:53:13.101011+0000","last_undegraded":"2023-10-24T23:53:13.101011+0000","last_fullsized":"2023-10-24T23:53:13.101011+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:12:21.026996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.795075+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-24T23:53:11.795075+0000","last_peered":"2023-10-24T23:53:11.795075+0000","last_clean":"2023-10-24T23:53:11.795075+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-24T23:53:11.795075+0000","last_undegraded":"2023-10-24T23:53:11.795075+0000","last_fullsized":"2023-10-24T23:53:11.795075+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101949+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-24T23:53:13.101949+0000","last_peered":"2023-10-24T23:53:13.101949+0000","last_clean":"2023-10-24T23:53:13.101949+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-24T23:53:13.101949+0000","last_undegraded":"2023-10-24T23:53:13.101949+0000","last_fullsized":"2023-10-24T23:53:13.101949+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:07.909021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.797593+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-24T23:53:11.797593+0000","last_peered":"2023-10-24T23:53:11.797593+0000","last_clean":"2023-10-24T23:53:11.797593+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-24T23:53:11.797593+0000","last_undegraded":"2023-10-24T23:53:11.797593+0000","last_fullsized":"2023-10-24T23:53:11.797593+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603906+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-24T23:53:13.603906+0000","last_peered":"2023-10-24T23:53:13.603906+0000","last_clean":"2023-10-24T23:53:13.603906+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-24T23:53:13.603906+0000","last_undegraded":"2023-10-24T23:53:13.603906+0000","last_fullsized":"2023-10-24T23:53:13.603906+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:48.217889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800508+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-24T23:53:11.800508+0000","last_peered":"2023-10-24T23:53:11.800508+0000","last_clean":"2023-10-24T23:53:11.800508+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-24T23:53:11.800508+0000","last_undegraded":"2023-10-24T23:53:11.800508+0000","last_fullsized":"2023-10-24T23:53:11.800508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602960+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-24T23:53:13.602960+0000","last_peered":"2023-10-24T23:53:13.602960+0000","last_clean":"2023-10-24T23:53:13.602960+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-24T23:53:13.602960+0000","last_undegraded":"2023-10-24T23:53:13.602960+0000","last_fullsized":"2023-10-24T23:53:13.602960+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:10:18.973606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838707+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-24T23:53:11.838707+0000","last_peered":"2023-10-24T23:53:11.838707+0000","last_clean":"2023-10-24T23:53:11.838707+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-24T23:53:11.838707+0000","last_undegraded":"2023-10-24T23:53:11.838707+0000","last_fullsized":"2023-10-24T23:53:11.838707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101457+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-24T23:53:13.101457+0000","last_peered":"2023-10-24T23:53:13.101457+0000","last_clean":"2023-10-24T23:53:13.101457+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-24T23:53:13.101457+0000","last_undegraded":"2023-10-24T23:53:13.101457+0000","last_fullsized":"2023-10-24T23:53:13.101457+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:24:45.536141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788141+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-24T23:53:11.788141+0000","last_peered":"2023-10-24T23:53:11.788141+0000","last_clean":"2023-10-24T23:53:11.788141+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-24T23:53:11.788141+0000","last_undegraded":"2023-10-24T23:53:11.788141+0000","last_fullsized":"2023-10-24T23:53:11.788141+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793724+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-24T23:53:12.793724+0000","last_peered":"2023-10-24T23:53:12.793724+0000","last_clean":"2023-10-24T23:53:12.793724+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-24T23:53:12.793724+0000","last_undegraded":"2023-10-24T23:53:12.793724+0000","last_fullsized":"2023-10-24T23:53:12.793724+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:06.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806250+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-24T23:53:11.806250+0000","last_peered":"2023-10-24T23:53:11.806250+0000","last_clean":"2023-10-24T23:53:11.806250+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-24T23:53:11.806250+0000","last_undegraded":"2023-10-24T23:53:11.806250+0000","last_fullsized":"2023-10-24T23:53:11.806250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275114+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-24T23:53:13.275114+0000","last_peered":"2023-10-24T23:53:13.275114+0000","last_clean":"2023-10-24T23:53:13.275114+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-24T23:53:13.275114+0000","last_undegraded":"2023-10-24T23:53:13.275114+0000","last_fullsized":"2023-10-24T23:53:13.275114+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:56:02.798009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.823970+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-24T23:53:11.823970+0000","last_peered":"2023-10-24T23:53:11.823970+0000","last_clean":"2023-10-24T23:53:11.823970+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-24T23:53:11.823970+0000","last_undegraded":"2023-10-24T23:53:11.823970+0000","last_fullsized":"2023-10-24T23:53:11.823970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793903+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-24T23:53:12.793903+0000","last_peered":"2023-10-24T23:53:12.793903+0000","last_clean":"2023-10-24T23:53:12.793903+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-24T23:53:12.793903+0000","last_undegraded":"2023-10-24T23:53:12.793903+0000","last_fullsized":"2023-10-24T23:53:12.793903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:40.168016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.809345+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-24T23:53:11.809345+0000","last_peered":"2023-10-24T23:53:11.809345+0000","last_clean":"2023-10-24T23:53:11.809345+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-24T23:53:11.809345+0000","last_undegraded":"2023-10-24T23:53:11.809345+0000","last_fullsized":"2023-10-24T23:53:11.809345+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.100799+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-24T23:53:13.100799+0000","last_peered":"2023-10-24T23:53:13.100799+0000","last_clean":"2023-10-24T23:53:13.100799+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-24T23:53:13.100799+0000","last_undegraded":"2023-10-24T23:53:13.100799+0000","last_fullsized":"2023-10-24T23:53:13.100799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:39.720375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788501+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-24T23:53:11.788501+0000","last_peered":"2023-10-24T23:53:11.788501+0000","last_clean":"2023-10-24T23:53:11.788501+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-24T23:53:11.788501+0000","last_undegraded":"2023-10-24T23:53:11.788501+0000","last_fullsized":"2023-10-24T23:53:11.788501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794282+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-24T23:53:12.794282+0000","last_peered":"2023-10-24T23:53:12.794282+0000","last_clean":"2023-10-24T23:53:12.794282+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-24T23:53:12.794282+0000","last_undegraded":"2023-10-24T23:53:12.794282+0000","last_fullsized":"2023-10-24T23:53:12.794282+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:51:40.960630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790034+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-24T23:53:11.790034+0000","last_peered":"2023-10-24T23:53:11.790034+0000","last_clean":"2023-10-24T23:53:11.790034+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-24T23:53:11.790034+0000","last_undegraded":"2023-10-24T23:53:11.790034+0000","last_fullsized":"2023-10-24T23:53:11.790034+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795555+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-24T23:53:12.795555+0000","last_peered":"2023-10-24T23:53:12.795555+0000","last_clean":"2023-10-24T23:53:12.795555+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-24T23:53:12.795555+0000","last_undegraded":"2023-10-24T23:53:12.795555+0000","last_fullsized":"2023-10-24T23:53:12.795555+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:22:42.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793089+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-24T23:53:11.793089+0000","last_peered":"2023-10-24T23:53:11.793089+0000","last_clean":"2023-10-24T23:53:11.793089+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-24T23:53:11.793089+0000","last_undegraded":"2023-10-24T23:53:11.793089+0000","last_fullsized":"2023-10-24T23:53:11.793089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275507+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-24T23:53:13.275507+0000","last_peered":"2023-10-24T23:53:13.275507+0000","last_clean":"2023-10-24T23:53:13.275507+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-24T23:53:13.275507+0000","last_undegraded":"2023-10-24T23:53:13.275507+0000","last_fullsized":"2023-10-24T23:53:13.275507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:47.658717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838499+0000","last_change":"2023-10-24T23:53:02.715966+0000","last_active":"2023-10-24T23:53:11.838499+0000","last_peered":"2023-10-24T23:53:11.838499+0000","last_clean":"2023-10-24T23:53:11.838499+0000","last_became_active":"2023-10-24T23:53:02.715631+0000","last_became_peered":"2023-10-24T23:53:02.715631+0000","last_unstale":"2023-10-24T23:53:11.838499+0000","last_undegraded":"2023-10-24T23:53:11.838499+0000","last_fullsized":"2023-10-24T23:53:11.838499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:11.771864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603292+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-24T23:53:13.603292+0000","last_peered":"2023-10-24T23:53:13.603292+0000","last_clean":"2023-10-24T23:53:13.603292+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-24T23:53:13.603292+0000","last_undegraded":"2023-10-24T23:53:13.603292+0000","last_fullsized":"2023-10-24T23:53:13.603292+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:50.994593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.796890+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-24T23:53:11.796890+0000","last_peered":"2023-10-24T23:53:11.796890+0000","last_clean":"2023-10-24T23:53:11.796890+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-24T23:53:11.796890+0000","last_undegraded":"2023-10-24T23:53:11.796890+0000","last_fullsized":"2023-10-24T23:53:11.796890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":929792,"data_stored":713128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":294912,"data_stored":119928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":307200,"data_stored":120110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":319488,"data_stored":121570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":121034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":892928,"data_stored":710330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:24.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:24.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.694+0000 7fcc6e7b3640 1 Processor -- start 2023-10-24T23:53:24.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.694+0000 7fcc6e7b3640 1 -- start start 2023-10-24T23:53:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6e7b3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc6814b600 0x7fcc68151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6e7b3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680c1860 con 0x7fcc6806a230 2023-10-24T23:53:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6e7b3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc68154d50 con 0x7fcc6806af70 2023-10-24T23:53:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6e7b3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681524f0 con 0x7fcc6814b600 2023-10-24T23:53:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6cd50640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcc6806a230 0x7fcc681551e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38230/0 (socket says 172.21.15.107:38230) 2023-10-24T23:53:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6cd50640 1 -- 172.21.15.107:0/2530827701 learned_addr learned my addr 172.21.15.107:0/2530827701 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc67fff640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc6814b600 0x7fcc68151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758241513 0 0) 0x7fcc680c1860 con 0x7fcc6806a230 2023-10-24T23:53:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc54003530 con 0x7fcc6806a230 2023-10-24T23:53:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1447996802 0 0) 0x7fcc54003530 con 0x7fcc6806a230 2023-10-24T23:53:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc6814b600 msgr2=0x7fcc68151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc6814b600 0x7fcc68151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 >> v1:172.21.15.110:6789/0 conn(0x7fcc6806af70 legacy=0x7fcc6806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6814b9e0 con 0x7fcc6806a230 2023-10-24T23:53:24.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fcc60001a00 con 0x7fcc6806a230 2023-10-24T23:53:24.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 >> v1:172.21.15.107:6789/0 conn(0x7fcc6806a230 legacy=0x7fcc681551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc540039b0 0x7fcc54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc6814b600 0x7fcc540047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc54004d10 0x7fcc54005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681524f0 con 0x7fcc54004d10 2023-10-24T23:53:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5400bac0 con 0x7fcc540039b0 2023-10-24T23:53:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5400bc80 con 0x7fcc6814b600 2023-10-24T23:53:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6cd50640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc54004d10 0x7fcc54005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc5ffff640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc6814b600 0x7fcc540047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc67fff640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc540039b0 0x7fcc54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6cd50640 1 -- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc6814b600 msgr2=0x7fcc540047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6cd50640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc6814b600 0x7fcc540047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6cd50640 1 -- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc540039b0 msgr2=0x7fcc54003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6cd50640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc540039b0 0x7fcc54003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6cd50640 1 -- 172.21.15.107:0/2530827701 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc54003530 con 0x7fcc54004d10 2023-10-24T23:53:24.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc5ffff640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc6814b600 0x7fcc540047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc6cd50640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc54004d10 0x7fcc54005190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcc6806a650 tx=0x7fcc6001fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d19b521111fe648 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc600aa670 con 0x7fcc54004d10 2023-10-24T23:53:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc600aa7d0 con 0x7fcc54004d10 2023-10-24T23:53:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.698+0000 7fcc677fe640 1 -- 172.21.15.107:0/2530827701 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc600aab80 con 0x7fcc54004d10 2023-10-24T23:53:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/2530827701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc54004d10 msgr2=0x7fcc54005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 --2- 172.21.15.107:0/2530827701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc54004d10 0x7fcc54005190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcc6806a650 tx=0x7fcc6001fa40 comp rx=0 tx=0).stop 2023-10-24T23:53:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6cd50640 1 -- 172.21.15.107:0/2530827701 reap_dead start 2023-10-24T23:53:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/2530827701 shutdown_connections 2023-10-24T23:53:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/2530827701 >> 172.21.15.107:0/2530827701 conn(0x7fcc680c2950 msgr2=0x7fcc680c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/2530827701 shutdown_connections 2023-10-24T23:53:24.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/2530827701 wait complete. 2023-10-24T23:53:24.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 Processor -- start 2023-10-24T23:53:24.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- start start 2023-10-24T23:53:24.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc54004d10 0x7fcc68153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc680bf5f0 0x7fcc680bfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc68152920 0x7fcc68152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68070c10 con 0x7fcc680bf5f0 2023-10-24T23:53:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680713e0 con 0x7fcc54004d10 2023-10-24T23:53:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc680bffb0 con 0x7fcc68152920 2023-10-24T23:53:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc5ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc680bf5f0 0x7fcc680bfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc5ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc680bf5f0 0x7fcc680bfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33218/0 (socket says 172.21.15.107:33218) 2023-10-24T23:53:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc5ffff640 1 -- 172.21.15.107:0/359497902 learned_addr learned my addr 172.21.15.107:0/359497902 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc67fff640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc54004d10 0x7fcc68153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6cd50640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc68152920 0x7fcc68152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc5ffff640 1 -- 172.21.15.107:0/359497902 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc68152920 msgr2=0x7fcc68152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc5ffff640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc68152920 0x7fcc68152db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc5ffff640 1 -- 172.21.15.107:0/359497902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc54004d10 msgr2=0x7fcc68153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc5ffff640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc54004d10 0x7fcc68153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc5ffff640 1 -- 172.21.15.107:0/359497902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc68071a30 con 0x7fcc680bf5f0 2023-10-24T23:53:24.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc67fff640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc54004d10 0x7fcc68153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:24.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc5ffff640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc680bf5f0 0x7fcc680bfa70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000d7a0 tx=0x7fcc5000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:24.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc657fa640 1 -- 172.21.15.107:0/359497902 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50017070 con 0x7fcc680bf5f0 2023-10-24T23:53:24.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc657fa640 1 -- 172.21.15.107:0/359497902 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc50005070 con 0x7fcc680bf5f0 2023-10-24T23:53:24.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/359497902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc68071d10 con 0x7fcc680bf5f0 2023-10-24T23:53:24.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc657fa640 1 -- 172.21.15.107:0/359497902 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50005420 con 0x7fcc680bf5f0 2023-10-24T23:53:24.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/359497902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc68071100 con 0x7fcc680bf5f0 2023-10-24T23:53:24.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.702+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/359497902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc680746e0 con 0x7fcc680bf5f0 2023-10-24T23:53:24.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.706+0000 7fcc657fa640 1 -- 172.21.15.107:0/359497902 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcc50007030 con 0x7fcc680bf5f0 2023-10-24T23:53:24.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.706+0000 7fcc657fa640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc48065e60 0x7fcc48068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:24.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.706+0000 7fcc657fa640 1 -- 172.21.15.107:0/359497902 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fcc500833a0 con 0x7fcc680bf5f0 2023-10-24T23:53:24.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.706+0000 7fcc67fff640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc48065e60 0x7fcc48068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.706+0000 7fcc67fff640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc48065e60 0x7fcc48068320 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcc680c0400 tx=0x7fcc58008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.710+0000 7fcc657fa640 1 -- 172.21.15.107:0/359497902 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fcc50050750 con 0x7fcc680bf5f0 2023-10-24T23:53:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.846+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/359497902 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcc680c0120 con 0x7fcc48065e60 2023-10-24T23:53:24.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.862+0000 7fcc657fa640 1 -- 172.21.15.107:0/359497902 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525074 (secure 0 0 0) 0x7fcc680c0120 con 0x7fcc48065e60 2023-10-24T23:53:24.868 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:24.874 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:24.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/359497902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc48065e60 msgr2=0x7fcc48068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc48065e60 0x7fcc48068320 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcc680c0400 tx=0x7fcc58008040 comp rx=0 tx=0).stop 2023-10-24T23:53:24.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/359497902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc680bf5f0 msgr2=0x7fcc680bfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:24.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc680bf5f0 0x7fcc680bfa70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000d7a0 tx=0x7fcc5000dc60 comp rx=0 tx=0).stop 2023-10-24T23:53:24.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/359497902 shutdown_connections 2023-10-24T23:53:24.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc48065e60 0x7fcc48068320 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc68152920 0x7fcc68152db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc680bf5f0 0x7fcc680bfa70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 --2- 172.21.15.107:0/359497902 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc54004d10 0x7fcc68153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:24.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/359497902 >> 172.21.15.107:0/359497902 conn(0x7fcc680c2950 msgr2=0x7fcc6814ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:24.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/359497902 shutdown_connections 2023-10-24T23:53:24.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:24.866+0000 7fcc6e7b3640 1 -- 172.21.15.107:0/359497902 wait complete. 2023-10-24T23:53:24.888 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2023-10-24T23:53:23.282371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595092,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235560,"kb_used_data":3560,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754739160,"statfs":{"total":773094113280,"available":772852899840,"internally_reserved":0,"allocated":3645440,"data_stored":2144952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.491437"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603561+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-24T23:53:13.603561+0000","last_peered":"2023-10-24T23:53:13.603561+0000","last_clean":"2023-10-24T23:53:13.603561+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-24T23:53:13.603561+0000","last_undegraded":"2023-10-24T23:53:13.603561+0000","last_fullsized":"2023-10-24T23:53:13.603561+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:51:20.137796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794961+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-24T23:53:12.794961+0000","last_peered":"2023-10-24T23:53:12.794961+0000","last_clean":"2023-10-24T23:53:12.794961+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-24T23:53:12.794961+0000","last_undegraded":"2023-10-24T23:53:12.794961+0000","last_fullsized":"2023-10-24T23:53:12.794961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:00:20.101379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788258+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-24T23:53:11.788258+0000","last_peered":"2023-10-24T23:53:11.788258+0000","last_clean":"2023-10-24T23:53:11.788258+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-24T23:53:11.788258+0000","last_undegraded":"2023-10-24T23:53:11.788258+0000","last_fullsized":"2023-10-24T23:53:11.788258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791737+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-24T23:53:11.791737+0000","last_peered":"2023-10-24T23:53:11.791737+0000","last_clean":"2023-10-24T23:53:11.791737+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-24T23:53:11.791737+0000","last_undegraded":"2023-10-24T23:53:11.791737+0000","last_fullsized":"2023-10-24T23:53:11.791737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794565+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-24T23:53:12.794565+0000","last_peered":"2023-10-24T23:53:12.794565+0000","last_clean":"2023-10-24T23:53:12.794565+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-24T23:53:12.794565+0000","last_undegraded":"2023-10-24T23:53:12.794565+0000","last_fullsized":"2023-10-24T23:53:12.794565+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:59.368156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.601719+0000","last_change":"2023-10-24T23:53:11.444089+0000","last_active":"2023-10-24T23:53:13.601719+0000","last_peered":"2023-10-24T23:53:13.601719+0000","last_clean":"2023-10-24T23:53:13.601719+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-24T23:53:13.601719+0000","last_undegraded":"2023-10-24T23:53:13.601719+0000","last_fullsized":"2023-10-24T23:53:13.601719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:54:06.875882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807520+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-24T23:53:11.807520+0000","last_peered":"2023-10-24T23:53:11.807520+0000","last_clean":"2023-10-24T23:53:11.807520+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-24T23:53:11.807520+0000","last_undegraded":"2023-10-24T23:53:11.807520+0000","last_fullsized":"2023-10-24T23:53:11.807520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806881+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-24T23:53:11.806881+0000","last_peered":"2023-10-24T23:53:11.806881+0000","last_clean":"2023-10-24T23:53:11.806881+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-24T23:53:11.806881+0000","last_undegraded":"2023-10-24T23:53:11.806881+0000","last_fullsized":"2023-10-24T23:53:11.806881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602008+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-24T23:53:13.602008+0000","last_peered":"2023-10-24T23:53:13.602008+0000","last_clean":"2023-10-24T23:53:13.602008+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-24T23:53:13.602008+0000","last_undegraded":"2023-10-24T23:53:13.602008+0000","last_fullsized":"2023-10-24T23:53:13.602008+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:38:56.643025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101201+0000","last_change":"2023-10-24T23:53:11.438681+0000","last_active":"2023-10-24T23:53:13.101201+0000","last_peered":"2023-10-24T23:53:13.101201+0000","last_clean":"2023-10-24T23:53:13.101201+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-24T23:53:13.101201+0000","last_undegraded":"2023-10-24T23:53:13.101201+0000","last_fullsized":"2023-10-24T23:53:13.101201+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:50.139697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832718+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-24T23:53:11.832718+0000","last_peered":"2023-10-24T23:53:11.832718+0000","last_clean":"2023-10-24T23:53:11.832718+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-24T23:53:11.832718+0000","last_undegraded":"2023-10-24T23:53:11.832718+0000","last_fullsized":"2023-10-24T23:53:11.832718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.810798+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-24T23:53:11.810798+0000","last_peered":"2023-10-24T23:53:11.810798+0000","last_clean":"2023-10-24T23:53:11.810798+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-24T23:53:11.810798+0000","last_undegraded":"2023-10-24T23:53:11.810798+0000","last_fullsized":"2023-10-24T23:53:11.810798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101349+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-24T23:53:13.101349+0000","last_peered":"2023-10-24T23:53:13.101349+0000","last_clean":"2023-10-24T23:53:13.101349+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-24T23:53:13.101349+0000","last_undegraded":"2023-10-24T23:53:13.101349+0000","last_fullsized":"2023-10-24T23:53:13.101349+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:22.974333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796171+0000","last_change":"2023-10-24T23:53:11.454537+0000","last_active":"2023-10-24T23:53:12.796171+0000","last_peered":"2023-10-24T23:53:12.796171+0000","last_clean":"2023-10-24T23:53:12.796171+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-24T23:53:12.796171+0000","last_undegraded":"2023-10-24T23:53:12.796171+0000","last_fullsized":"2023-10-24T23:53:12.796171+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:14:06.792651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.830575+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-24T23:53:11.830575+0000","last_peered":"2023-10-24T23:53:11.830575+0000","last_clean":"2023-10-24T23:53:11.830575+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-24T23:53:11.830575+0000","last_undegraded":"2023-10-24T23:53:11.830575+0000","last_fullsized":"2023-10-24T23:53:11.830575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800231+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-24T23:53:11.800231+0000","last_peered":"2023-10-24T23:53:11.800231+0000","last_clean":"2023-10-24T23:53:11.800231+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-24T23:53:11.800231+0000","last_undegraded":"2023-10-24T23:53:11.800231+0000","last_fullsized":"2023-10-24T23:53:11.800231+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605269+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-24T23:53:13.605269+0000","last_peered":"2023-10-24T23:53:13.605269+0000","last_clean":"2023-10-24T23:53:13.605269+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-24T23:53:13.605269+0000","last_undegraded":"2023-10-24T23:53:13.605269+0000","last_fullsized":"2023-10-24T23:53:13.605269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:35:39.879563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795373+0000","last_change":"2023-10-24T23:53:11.444138+0000","last_active":"2023-10-24T23:53:12.795373+0000","last_peered":"2023-10-24T23:53:12.795373+0000","last_clean":"2023-10-24T23:53:12.795373+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-24T23:53:12.795373+0000","last_undegraded":"2023-10-24T23:53:12.795373+0000","last_fullsized":"2023-10-24T23:53:12.795373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:27:38.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788658+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-24T23:53:11.788658+0000","last_peered":"2023-10-24T23:53:11.788658+0000","last_clean":"2023-10-24T23:53:11.788658+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-24T23:53:11.788658+0000","last_undegraded":"2023-10-24T23:53:11.788658+0000","last_fullsized":"2023-10-24T23:53:11.788658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.797645+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-24T23:53:11.797645+0000","last_peered":"2023-10-24T23:53:11.797645+0000","last_clean":"2023-10-24T23:53:11.797645+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-24T23:53:11.797645+0000","last_undegraded":"2023-10-24T23:53:11.797645+0000","last_fullsized":"2023-10-24T23:53:11.797645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795697+0000","last_change":"2023-10-24T23:53:11.832809+0000","last_active":"2023-10-24T23:53:12.795697+0000","last_peered":"2023-10-24T23:53:12.795697+0000","last_clean":"2023-10-24T23:53:12.795697+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-24T23:53:12.795697+0000","last_undegraded":"2023-10-24T23:53:12.795697+0000","last_fullsized":"2023-10-24T23:53:12.795697+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:01.151315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604963+0000","last_change":"2023-10-24T23:53:11.442035+0000","last_active":"2023-10-24T23:53:13.604963+0000","last_peered":"2023-10-24T23:53:13.604963+0000","last_clean":"2023-10-24T23:53:13.604963+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-24T23:53:13.604963+0000","last_undegraded":"2023-10-24T23:53:13.604963+0000","last_fullsized":"2023-10-24T23:53:13.604963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:13.717844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.811329+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-24T23:53:11.811329+0000","last_peered":"2023-10-24T23:53:11.811329+0000","last_clean":"2023-10-24T23:53:11.811329+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-24T23:53:11.811329+0000","last_undegraded":"2023-10-24T23:53:11.811329+0000","last_fullsized":"2023-10-24T23:53:11.811329+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807632+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-24T23:53:11.807632+0000","last_peered":"2023-10-24T23:53:11.807632+0000","last_clean":"2023-10-24T23:53:11.807632+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-24T23:53:11.807632+0000","last_undegraded":"2023-10-24T23:53:11.807632+0000","last_fullsized":"2023-10-24T23:53:11.807632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797540+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-24T23:53:12.797540+0000","last_peered":"2023-10-24T23:53:12.797540+0000","last_clean":"2023-10-24T23:53:12.797540+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-24T23:53:12.797540+0000","last_undegraded":"2023-10-24T23:53:12.797540+0000","last_fullsized":"2023-10-24T23:53:12.797540+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:26.012711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604200+0000","last_change":"2023-10-24T23:53:11.448032+0000","last_active":"2023-10-24T23:53:13.604200+0000","last_peered":"2023-10-24T23:53:13.604200+0000","last_clean":"2023-10-24T23:53:13.604200+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-24T23:53:13.604200+0000","last_undegraded":"2023-10-24T23:53:13.604200+0000","last_fullsized":"2023-10-24T23:53:13.604200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:50:32.490810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791720+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-24T23:53:11.791720+0000","last_peered":"2023-10-24T23:53:11.791720+0000","last_clean":"2023-10-24T23:53:11.791720+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-24T23:53:11.791720+0000","last_undegraded":"2023-10-24T23:53:11.791720+0000","last_fullsized":"2023-10-24T23:53:11.791720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788475+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-24T23:53:11.788475+0000","last_peered":"2023-10-24T23:53:11.788475+0000","last_clean":"2023-10-24T23:53:11.788475+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-24T23:53:11.788475+0000","last_undegraded":"2023-10-24T23:53:11.788475+0000","last_fullsized":"2023-10-24T23:53:11.788475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795316+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-24T23:53:12.795316+0000","last_peered":"2023-10-24T23:53:12.795316+0000","last_clean":"2023-10-24T23:53:12.795316+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-24T23:53:12.795316+0000","last_undegraded":"2023-10-24T23:53:12.795316+0000","last_fullsized":"2023-10-24T23:53:12.795316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:10.820967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603323+0000","last_change":"2023-10-24T23:53:11.446925+0000","last_active":"2023-10-24T23:53:13.603323+0000","last_peered":"2023-10-24T23:53:13.603323+0000","last_clean":"2023-10-24T23:53:13.603323+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-24T23:53:13.603323+0000","last_undegraded":"2023-10-24T23:53:13.603323+0000","last_fullsized":"2023-10-24T23:53:13.603323+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:59:52.485469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804040+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-24T23:53:11.804040+0000","last_peered":"2023-10-24T23:53:11.804040+0000","last_clean":"2023-10-24T23:53:11.804040+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-24T23:53:11.804040+0000","last_undegraded":"2023-10-24T23:53:11.804040+0000","last_fullsized":"2023-10-24T23:53:11.804040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791850+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-24T23:53:11.791850+0000","last_peered":"2023-10-24T23:53:11.791850+0000","last_clean":"2023-10-24T23:53:11.791850+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-24T23:53:11.791850+0000","last_undegraded":"2023-10-24T23:53:11.791850+0000","last_fullsized":"2023-10-24T23:53:11.791850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795971+0000","last_change":"2023-10-24T23:53:11.849056+0000","last_active":"2023-10-24T23:53:12.795971+0000","last_peered":"2023-10-24T23:53:12.795971+0000","last_clean":"2023-10-24T23:53:12.795971+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-24T23:53:12.795971+0000","last_undegraded":"2023-10-24T23:53:12.795971+0000","last_fullsized":"2023-10-24T23:53:12.795971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:08.932942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803866+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-24T23:53:11.803866+0000","last_peered":"2023-10-24T23:53:11.803866+0000","last_clean":"2023-10-24T23:53:11.803866+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-24T23:53:11.803866+0000","last_undegraded":"2023-10-24T23:53:11.803866+0000","last_fullsized":"2023-10-24T23:53:11.803866+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788258+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-24T23:53:11.788258+0000","last_peered":"2023-10-24T23:53:11.788258+0000","last_clean":"2023-10-24T23:53:11.788258+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-24T23:53:11.788258+0000","last_undegraded":"2023-10-24T23:53:11.788258+0000","last_fullsized":"2023-10-24T23:53:11.788258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602180+0000","last_change":"2023-10-24T23:53:11.447167+0000","last_active":"2023-10-24T23:53:13.602180+0000","last_peered":"2023-10-24T23:53:13.602180+0000","last_clean":"2023-10-24T23:53:13.602180+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-24T23:53:13.602180+0000","last_undegraded":"2023-10-24T23:53:13.602180+0000","last_fullsized":"2023-10-24T23:53:13.602180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:06:25.006728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794920+0000","last_change":"2023-10-24T23:53:11.813704+0000","last_active":"2023-10-24T23:53:12.794920+0000","last_peered":"2023-10-24T23:53:12.794920+0000","last_clean":"2023-10-24T23:53:12.794920+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-24T23:53:12.794920+0000","last_undegraded":"2023-10-24T23:53:12.794920+0000","last_fullsized":"2023-10-24T23:53:12.794920+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:39:09.996039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804467+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-24T23:53:11.804467+0000","last_peered":"2023-10-24T23:53:11.804467+0000","last_clean":"2023-10-24T23:53:11.804467+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-24T23:53:11.804467+0000","last_undegraded":"2023-10-24T23:53:11.804467+0000","last_fullsized":"2023-10-24T23:53:11.804467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838755+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-24T23:53:11.838755+0000","last_peered":"2023-10-24T23:53:11.838755+0000","last_clean":"2023-10-24T23:53:11.838755+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-24T23:53:11.838755+0000","last_undegraded":"2023-10-24T23:53:11.838755+0000","last_fullsized":"2023-10-24T23:53:11.838755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795718+0000","last_change":"2023-10-24T23:53:11.450974+0000","last_active":"2023-10-24T23:53:12.795718+0000","last_peered":"2023-10-24T23:53:12.795718+0000","last_clean":"2023-10-24T23:53:12.795718+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-24T23:53:12.795718+0000","last_undegraded":"2023-10-24T23:53:12.795718+0000","last_fullsized":"2023-10-24T23:53:12.795718+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:51:05.443664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602086+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-24T23:53:13.602086+0000","last_peered":"2023-10-24T23:53:13.602086+0000","last_clean":"2023-10-24T23:53:13.602086+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-24T23:53:13.602086+0000","last_undegraded":"2023-10-24T23:53:13.602086+0000","last_fullsized":"2023-10-24T23:53:13.602086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:37:30.278659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825877+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-24T23:53:11.825877+0000","last_peered":"2023-10-24T23:53:11.825877+0000","last_clean":"2023-10-24T23:53:11.825877+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-24T23:53:11.825877+0000","last_undegraded":"2023-10-24T23:53:11.825877+0000","last_fullsized":"2023-10-24T23:53:11.825877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791237+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-24T23:53:11.791237+0000","last_peered":"2023-10-24T23:53:11.791237+0000","last_clean":"2023-10-24T23:53:11.791237+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-24T23:53:11.791237+0000","last_undegraded":"2023-10-24T23:53:11.791237+0000","last_fullsized":"2023-10-24T23:53:11.791237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794303+0000","last_change":"2023-10-24T23:53:11.429694+0000","last_active":"2023-10-24T23:53:12.794303+0000","last_peered":"2023-10-24T23:53:12.794303+0000","last_clean":"2023-10-24T23:53:12.794303+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-24T23:53:12.794303+0000","last_undegraded":"2023-10-24T23:53:12.794303+0000","last_fullsized":"2023-10-24T23:53:12.794303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:15.964994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793903+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-24T23:53:12.793903+0000","last_peered":"2023-10-24T23:53:12.793903+0000","last_clean":"2023-10-24T23:53:12.793903+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-24T23:53:12.793903+0000","last_undegraded":"2023-10-24T23:53:12.793903+0000","last_fullsized":"2023-10-24T23:53:12.793903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:38:18.678087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"22'6","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803275+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-24T23:53:11.803275+0000","last_peered":"2023-10-24T23:53:11.803275+0000","last_clean":"2023-10-24T23:53:11.803275+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-24T23:53:11.803275+0000","last_undegraded":"2023-10-24T23:53:11.803275+0000","last_fullsized":"2023-10-24T23:53:11.803275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800342+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-24T23:53:11.800342+0000","last_peered":"2023-10-24T23:53:11.800342+0000","last_clean":"2023-10-24T23:53:11.800342+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-24T23:53:11.800342+0000","last_undegraded":"2023-10-24T23:53:11.800342+0000","last_fullsized":"2023-10-24T23:53:11.800342+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793830+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-24T23:53:12.793830+0000","last_peered":"2023-10-24T23:53:12.793830+0000","last_clean":"2023-10-24T23:53:12.793830+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-24T23:53:12.793830+0000","last_undegraded":"2023-10-24T23:53:12.793830+0000","last_fullsized":"2023-10-24T23:53:12.793830+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:30.163913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797383+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-24T23:53:12.797383+0000","last_peered":"2023-10-24T23:53:12.797383+0000","last_clean":"2023-10-24T23:53:12.797383+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-24T23:53:12.797383+0000","last_undegraded":"2023-10-24T23:53:12.797383+0000","last_fullsized":"2023-10-24T23:53:12.797383+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:46.000330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.802095+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-24T23:53:11.802095+0000","last_peered":"2023-10-24T23:53:11.802095+0000","last_clean":"2023-10-24T23:53:11.802095+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-24T23:53:11.802095+0000","last_undegraded":"2023-10-24T23:53:11.802095+0000","last_fullsized":"2023-10-24T23:53:11.802095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.792050+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:11.792050+0000","last_peered":"2023-10-24T23:53:11.792050+0000","last_clean":"2023-10-24T23:53:11.792050+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:11.792050+0000","last_undegraded":"2023-10-24T23:53:11.792050+0000","last_fullsized":"2023-10-24T23:53:11.792050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788646+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-24T23:53:11.788646+0000","last_peered":"2023-10-24T23:53:11.788646+0000","last_clean":"2023-10-24T23:53:11.788646+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-24T23:53:11.788646+0000","last_undegraded":"2023-10-24T23:53:11.788646+0000","last_fullsized":"2023-10-24T23:53:11.788646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793988+0000","last_change":"2023-10-24T23:53:11.433306+0000","last_active":"2023-10-24T23:53:12.793988+0000","last_peered":"2023-10-24T23:53:12.793988+0000","last_clean":"2023-10-24T23:53:12.793988+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-24T23:53:12.793988+0000","last_undegraded":"2023-10-24T23:53:12.793988+0000","last_fullsized":"2023-10-24T23:53:12.793988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:44:38.728345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604210+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-24T23:53:13.604210+0000","last_peered":"2023-10-24T23:53:13.604210+0000","last_clean":"2023-10-24T23:53:13.604210+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-24T23:53:13.604210+0000","last_undegraded":"2023-10-24T23:53:13.604210+0000","last_fullsized":"2023-10-24T23:53:13.604210+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:30:24.805861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803946+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-24T23:53:11.803946+0000","last_peered":"2023-10-24T23:53:11.803946+0000","last_clean":"2023-10-24T23:53:11.803946+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-24T23:53:11.803946+0000","last_undegraded":"2023-10-24T23:53:11.803946+0000","last_fullsized":"2023-10-24T23:53:11.803946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790715+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:11.790715+0000","last_peered":"2023-10-24T23:53:11.790715+0000","last_clean":"2023-10-24T23:53:11.790715+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:11.790715+0000","last_undegraded":"2023-10-24T23:53:11.790715+0000","last_fullsized":"2023-10-24T23:53:11.790715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807502+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-24T23:53:11.807502+0000","last_peered":"2023-10-24T23:53:11.807502+0000","last_clean":"2023-10-24T23:53:11.807502+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-24T23:53:11.807502+0000","last_undegraded":"2023-10-24T23:53:11.807502+0000","last_fullsized":"2023-10-24T23:53:11.807502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602440+0000","last_change":"2023-10-24T23:53:11.441983+0000","last_active":"2023-10-24T23:53:13.602440+0000","last_peered":"2023-10-24T23:53:13.602440+0000","last_clean":"2023-10-24T23:53:13.602440+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-24T23:53:13.602440+0000","last_undegraded":"2023-10-24T23:53:13.602440+0000","last_fullsized":"2023-10-24T23:53:13.602440+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:30.643019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794864+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-24T23:53:12.794864+0000","last_peered":"2023-10-24T23:53:12.794864+0000","last_clean":"2023-10-24T23:53:12.794864+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-24T23:53:12.794864+0000","last_undegraded":"2023-10-24T23:53:12.794864+0000","last_fullsized":"2023-10-24T23:53:12.794864+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:50:53.467058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788379+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-24T23:53:11.788379+0000","last_peered":"2023-10-24T23:53:11.788379+0000","last_clean":"2023-10-24T23:53:11.788379+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-24T23:53:11.788379+0000","last_undegraded":"2023-10-24T23:53:11.788379+0000","last_fullsized":"2023-10-24T23:53:11.788379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.826147+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:11.826147+0000","last_peered":"2023-10-24T23:53:11.826147+0000","last_clean":"2023-10-24T23:53:11.826147+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:11.826147+0000","last_undegraded":"2023-10-24T23:53:11.826147+0000","last_fullsized":"2023-10-24T23:53:11.826147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790306+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-24T23:53:11.790306+0000","last_peered":"2023-10-24T23:53:11.790306+0000","last_clean":"2023-10-24T23:53:11.790306+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-24T23:53:11.790306+0000","last_undegraded":"2023-10-24T23:53:11.790306+0000","last_fullsized":"2023-10-24T23:53:11.790306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602338+0000","last_change":"2023-10-24T23:53:11.426287+0000","last_active":"2023-10-24T23:53:13.602338+0000","last_peered":"2023-10-24T23:53:13.602338+0000","last_clean":"2023-10-24T23:53:13.602338+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-24T23:53:13.602338+0000","last_undegraded":"2023-10-24T23:53:13.602338+0000","last_fullsized":"2023-10-24T23:53:13.602338+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:02:39.399866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602429+0000","last_change":"2023-10-24T23:53:11.811139+0000","last_active":"2023-10-24T23:53:13.602429+0000","last_peered":"2023-10-24T23:53:13.602429+0000","last_clean":"2023-10-24T23:53:13.602429+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-24T23:53:13.602429+0000","last_undegraded":"2023-10-24T23:53:13.602429+0000","last_fullsized":"2023-10-24T23:53:13.602429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:05.507633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791317+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-24T23:53:11.791317+0000","last_peered":"2023-10-24T23:53:11.791317+0000","last_clean":"2023-10-24T23:53:11.791317+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-24T23:53:11.791317+0000","last_undegraded":"2023-10-24T23:53:11.791317+0000","last_fullsized":"2023-10-24T23:53:11.791317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790574+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:11.790574+0000","last_peered":"2023-10-24T23:53:11.790574+0000","last_clean":"2023-10-24T23:53:11.790574+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:11.790574+0000","last_undegraded":"2023-10-24T23:53:11.790574+0000","last_fullsized":"2023-10-24T23:53:11.790574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832916+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-24T23:53:11.832916+0000","last_peered":"2023-10-24T23:53:11.832916+0000","last_clean":"2023-10-24T23:53:11.832916+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-24T23:53:11.832916+0000","last_undegraded":"2023-10-24T23:53:11.832916+0000","last_fullsized":"2023-10-24T23:53:11.832916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602790+0000","last_change":"2023-10-24T23:53:11.426199+0000","last_active":"2023-10-24T23:53:13.602790+0000","last_peered":"2023-10-24T23:53:13.602790+0000","last_clean":"2023-10-24T23:53:13.602790+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-24T23:53:13.602790+0000","last_undegraded":"2023-10-24T23:53:13.602790+0000","last_fullsized":"2023-10-24T23:53:13.602790+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:57:51.018816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794361+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-24T23:53:12.794361+0000","last_peered":"2023-10-24T23:53:12.794361+0000","last_clean":"2023-10-24T23:53:12.794361+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-24T23:53:12.794361+0000","last_undegraded":"2023-10-24T23:53:12.794361+0000","last_fullsized":"2023-10-24T23:53:12.794361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:20:16.596993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790230+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-24T23:53:11.790230+0000","last_peered":"2023-10-24T23:53:11.790230+0000","last_clean":"2023-10-24T23:53:11.790230+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-24T23:53:11.790230+0000","last_undegraded":"2023-10-24T23:53:11.790230+0000","last_fullsized":"2023-10-24T23:53:11.790230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800496+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-24T23:53:11.800496+0000","last_peered":"2023-10-24T23:53:11.800496+0000","last_clean":"2023-10-24T23:53:11.800496+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-24T23:53:11.800496+0000","last_undegraded":"2023-10-24T23:53:11.800496+0000","last_fullsized":"2023-10-24T23:53:11.800496+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828620+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:53:11.828620+0000","last_peered":"2023-10-24T23:53:11.828620+0000","last_clean":"2023-10-24T23:53:11.828620+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:53:11.828620+0000","last_undegraded":"2023-10-24T23:53:11.828620+0000","last_fullsized":"2023-10-24T23:53:11.828620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603076+0000","last_change":"2023-10-24T23:53:11.432666+0000","last_active":"2023-10-24T23:53:13.603076+0000","last_peered":"2023-10-24T23:53:13.603076+0000","last_clean":"2023-10-24T23:53:13.603076+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-24T23:53:13.603076+0000","last_undegraded":"2023-10-24T23:53:13.603076+0000","last_fullsized":"2023-10-24T23:53:13.603076+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:17.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797676+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-24T23:53:12.797676+0000","last_peered":"2023-10-24T23:53:12.797676+0000","last_clean":"2023-10-24T23:53:12.797676+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-24T23:53:12.797676+0000","last_undegraded":"2023-10-24T23:53:12.797676+0000","last_fullsized":"2023-10-24T23:53:12.797676+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:42.131555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789311+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-24T23:53:11.789311+0000","last_peered":"2023-10-24T23:53:11.789311+0000","last_clean":"2023-10-24T23:53:11.789311+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-24T23:53:11.789311+0000","last_undegraded":"2023-10-24T23:53:11.789311+0000","last_fullsized":"2023-10-24T23:53:11.789311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789825+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-24T23:53:11.789825+0000","last_peered":"2023-10-24T23:53:11.789825+0000","last_clean":"2023-10-24T23:53:11.789825+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-24T23:53:11.789825+0000","last_undegraded":"2023-10-24T23:53:11.789825+0000","last_fullsized":"2023-10-24T23:53:11.789825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.795048+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:53:11.795048+0000","last_peered":"2023-10-24T23:53:11.795048+0000","last_clean":"2023-10-24T23:53:11.795048+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:53:11.795048+0000","last_undegraded":"2023-10-24T23:53:11.795048+0000","last_fullsized":"2023-10-24T23:53:11.795048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"27'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.279161+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-24T23:53:13.279161+0000","last_peered":"2023-10-24T23:53:13.279161+0000","last_clean":"2023-10-24T23:53:13.279161+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-24T23:53:13.279161+0000","last_undegraded":"2023-10-24T23:53:13.279161+0000","last_fullsized":"2023-10-24T23:53:13.279161+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:55:47.548432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795507+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-24T23:53:12.795507+0000","last_peered":"2023-10-24T23:53:12.795507+0000","last_clean":"2023-10-24T23:53:12.795507+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-24T23:53:12.795507+0000","last_undegraded":"2023-10-24T23:53:12.795507+0000","last_fullsized":"2023-10-24T23:53:12.795507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:11:14.431984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788576+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-24T23:53:11.788576+0000","last_peered":"2023-10-24T23:53:11.788576+0000","last_clean":"2023-10-24T23:53:11.788576+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-24T23:53:11.788576+0000","last_undegraded":"2023-10-24T23:53:11.788576+0000","last_fullsized":"2023-10-24T23:53:11.788576+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789662+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-24T23:53:11.789662+0000","last_peered":"2023-10-24T23:53:11.789662+0000","last_clean":"2023-10-24T23:53:11.789662+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-24T23:53:11.789662+0000","last_undegraded":"2023-10-24T23:53:11.789662+0000","last_fullsized":"2023-10-24T23:53:11.789662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828080+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:11.828080+0000","last_peered":"2023-10-24T23:53:11.828080+0000","last_clean":"2023-10-24T23:53:11.828080+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:11.828080+0000","last_undegraded":"2023-10-24T23:53:11.828080+0000","last_fullsized":"2023-10-24T23:53:11.828080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102659+0000","last_change":"2023-10-24T23:53:11.436603+0000","last_active":"2023-10-24T23:53:13.102659+0000","last_peered":"2023-10-24T23:53:13.102659+0000","last_clean":"2023-10-24T23:53:13.102659+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-24T23:53:13.102659+0000","last_undegraded":"2023-10-24T23:53:13.102659+0000","last_fullsized":"2023-10-24T23:53:13.102659+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:09:05.948928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102438+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-24T23:53:13.102438+0000","last_peered":"2023-10-24T23:53:13.102438+0000","last_clean":"2023-10-24T23:53:13.102438+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-24T23:53:13.102438+0000","last_undegraded":"2023-10-24T23:53:13.102438+0000","last_fullsized":"2023-10-24T23:53:13.102438+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:01:59.639080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793650+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-24T23:53:11.793650+0000","last_peered":"2023-10-24T23:53:11.793650+0000","last_clean":"2023-10-24T23:53:11.793650+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-24T23:53:11.793650+0000","last_undegraded":"2023-10-24T23:53:11.793650+0000","last_fullsized":"2023-10-24T23:53:11.793650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807120+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-24T23:53:11.807120+0000","last_peered":"2023-10-24T23:53:11.807120+0000","last_clean":"2023-10-24T23:53:11.807120+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-24T23:53:11.807120+0000","last_undegraded":"2023-10-24T23:53:11.807120+0000","last_fullsized":"2023-10-24T23:53:11.807120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825313+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:53:11.825313+0000","last_peered":"2023-10-24T23:53:11.825313+0000","last_clean":"2023-10-24T23:53:11.825313+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:53:11.825313+0000","last_undegraded":"2023-10-24T23:53:11.825313+0000","last_fullsized":"2023-10-24T23:53:11.825313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.834350+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:53:11.834350+0000","last_peered":"2023-10-24T23:53:11.834350+0000","last_clean":"2023-10-24T23:53:11.834350+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:53:11.834350+0000","last_undegraded":"2023-10-24T23:53:11.834350+0000","last_fullsized":"2023-10-24T23:53:11.834350+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797622+0000","last_change":"2023-10-24T23:53:11.451073+0000","last_active":"2023-10-24T23:53:12.797622+0000","last_peered":"2023-10-24T23:53:12.797622+0000","last_clean":"2023-10-24T23:53:12.797622+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-24T23:53:12.797622+0000","last_undegraded":"2023-10-24T23:53:12.797622+0000","last_fullsized":"2023-10-24T23:53:12.797622+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:32.324710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101638+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-24T23:53:13.101638+0000","last_peered":"2023-10-24T23:53:13.101638+0000","last_clean":"2023-10-24T23:53:13.101638+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-24T23:53:13.101638+0000","last_undegraded":"2023-10-24T23:53:13.101638+0000","last_fullsized":"2023-10-24T23:53:13.101638+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:04.125835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789210+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-24T23:53:11.789210+0000","last_peered":"2023-10-24T23:53:11.789210+0000","last_clean":"2023-10-24T23:53:11.789210+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-24T23:53:11.789210+0000","last_undegraded":"2023-10-24T23:53:11.789210+0000","last_fullsized":"2023-10-24T23:53:11.789210+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804119+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-24T23:53:11.804119+0000","last_peered":"2023-10-24T23:53:11.804119+0000","last_clean":"2023-10-24T23:53:11.804119+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-24T23:53:11.804119+0000","last_undegraded":"2023-10-24T23:53:11.804119+0000","last_fullsized":"2023-10-24T23:53:11.804119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605414+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-24T23:53:13.605414+0000","last_peered":"2023-10-24T23:53:13.605414+0000","last_clean":"2023-10-24T23:53:13.605414+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-24T23:53:13.605414+0000","last_undegraded":"2023-10-24T23:53:13.605414+0000","last_fullsized":"2023-10-24T23:53:13.605414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:06:53.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794699+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-24T23:53:12.794699+0000","last_peered":"2023-10-24T23:53:12.794699+0000","last_clean":"2023-10-24T23:53:12.794699+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-24T23:53:12.794699+0000","last_undegraded":"2023-10-24T23:53:12.794699+0000","last_fullsized":"2023-10-24T23:53:12.794699+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:41.905772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804055+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-24T23:53:11.804055+0000","last_peered":"2023-10-24T23:53:11.804055+0000","last_clean":"2023-10-24T23:53:11.804055+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-24T23:53:11.804055+0000","last_undegraded":"2023-10-24T23:53:11.804055+0000","last_fullsized":"2023-10-24T23:53:11.804055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.802309+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-24T23:53:11.802309+0000","last_peered":"2023-10-24T23:53:11.802309+0000","last_clean":"2023-10-24T23:53:11.802309+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-24T23:53:11.802309+0000","last_undegraded":"2023-10-24T23:53:11.802309+0000","last_fullsized":"2023-10-24T23:53:11.802309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102244+0000","last_change":"2023-10-24T23:53:11.458463+0000","last_active":"2023-10-24T23:53:13.102244+0000","last_peered":"2023-10-24T23:53:13.102244+0000","last_clean":"2023-10-24T23:53:13.102244+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-24T23:53:13.102244+0000","last_undegraded":"2023-10-24T23:53:13.102244+0000","last_fullsized":"2023-10-24T23:53:13.102244+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:11:45.040156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275727+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-24T23:53:13.275727+0000","last_peered":"2023-10-24T23:53:13.275727+0000","last_clean":"2023-10-24T23:53:13.275727+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-24T23:53:13.275727+0000","last_undegraded":"2023-10-24T23:53:13.275727+0000","last_fullsized":"2023-10-24T23:53:13.275727+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:51.707178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800564+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-24T23:53:11.800564+0000","last_peered":"2023-10-24T23:53:11.800564+0000","last_clean":"2023-10-24T23:53:11.800564+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-24T23:53:11.800564+0000","last_undegraded":"2023-10-24T23:53:11.800564+0000","last_fullsized":"2023-10-24T23:53:11.800564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790875+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-24T23:53:11.790875+0000","last_peered":"2023-10-24T23:53:11.790875+0000","last_clean":"2023-10-24T23:53:11.790875+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-24T23:53:11.790875+0000","last_undegraded":"2023-10-24T23:53:11.790875+0000","last_fullsized":"2023-10-24T23:53:11.790875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795610+0000","last_change":"2023-10-24T23:53:11.448675+0000","last_active":"2023-10-24T23:53:12.795610+0000","last_peered":"2023-10-24T23:53:12.795610+0000","last_clean":"2023-10-24T23:53:12.795610+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-24T23:53:12.795610+0000","last_undegraded":"2023-10-24T23:53:12.795610+0000","last_fullsized":"2023-10-24T23:53:12.795610+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:01.502343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605168+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-24T23:53:13.605168+0000","last_peered":"2023-10-24T23:53:13.605168+0000","last_clean":"2023-10-24T23:53:13.605168+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-24T23:53:13.605168+0000","last_undegraded":"2023-10-24T23:53:13.605168+0000","last_fullsized":"2023-10-24T23:53:13.605168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:54.478897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793341+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-24T23:53:11.793341+0000","last_peered":"2023-10-24T23:53:11.793341+0000","last_clean":"2023-10-24T23:53:11.793341+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-24T23:53:11.793341+0000","last_undegraded":"2023-10-24T23:53:11.793341+0000","last_fullsized":"2023-10-24T23:53:11.793341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807904+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-24T23:53:11.807904+0000","last_peered":"2023-10-24T23:53:11.807904+0000","last_clean":"2023-10-24T23:53:11.807904+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-24T23:53:11.807904+0000","last_undegraded":"2023-10-24T23:53:11.807904+0000","last_fullsized":"2023-10-24T23:53:11.807904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275531+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-24T23:53:13.275531+0000","last_peered":"2023-10-24T23:53:13.275531+0000","last_clean":"2023-10-24T23:53:13.275531+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-24T23:53:13.275531+0000","last_undegraded":"2023-10-24T23:53:13.275531+0000","last_fullsized":"2023-10-24T23:53:13.275531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:58.545886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602449+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-24T23:53:13.602449+0000","last_peered":"2023-10-24T23:53:13.602449+0000","last_clean":"2023-10-24T23:53:13.602449+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-24T23:53:13.602449+0000","last_undegraded":"2023-10-24T23:53:13.602449+0000","last_fullsized":"2023-10-24T23:53:13.602449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:05.595887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788253+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-24T23:53:11.788253+0000","last_peered":"2023-10-24T23:53:11.788253+0000","last_clean":"2023-10-24T23:53:11.788253+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-24T23:53:11.788253+0000","last_undegraded":"2023-10-24T23:53:11.788253+0000","last_fullsized":"2023-10-24T23:53:11.788253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.830849+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-24T23:53:11.830849+0000","last_peered":"2023-10-24T23:53:11.830849+0000","last_clean":"2023-10-24T23:53:11.830849+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-24T23:53:11.830849+0000","last_undegraded":"2023-10-24T23:53:11.830849+0000","last_fullsized":"2023-10-24T23:53:11.830849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275166+0000","last_change":"2023-10-24T23:53:11.454658+0000","last_active":"2023-10-24T23:53:13.275166+0000","last_peered":"2023-10-24T23:53:13.275166+0000","last_clean":"2023-10-24T23:53:13.275166+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-24T23:53:13.275166+0000","last_undegraded":"2023-10-24T23:53:13.275166+0000","last_fullsized":"2023-10-24T23:53:13.275166+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:43:43.869069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796376+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-24T23:53:12.796376+0000","last_peered":"2023-10-24T23:53:12.796376+0000","last_clean":"2023-10-24T23:53:12.796376+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-24T23:53:12.796376+0000","last_undegraded":"2023-10-24T23:53:12.796376+0000","last_fullsized":"2023-10-24T23:53:12.796376+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:09:40.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790517+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-24T23:53:11.790517+0000","last_peered":"2023-10-24T23:53:11.790517+0000","last_clean":"2023-10-24T23:53:11.790517+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-24T23:53:11.790517+0000","last_undegraded":"2023-10-24T23:53:11.790517+0000","last_fullsized":"2023-10-24T23:53:11.790517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825937+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-24T23:53:11.825937+0000","last_peered":"2023-10-24T23:53:11.825937+0000","last_clean":"2023-10-24T23:53:11.825937+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-24T23:53:11.825937+0000","last_undegraded":"2023-10-24T23:53:11.825937+0000","last_fullsized":"2023-10-24T23:53:11.825937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275899+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-24T23:53:13.275899+0000","last_peered":"2023-10-24T23:53:13.275899+0000","last_clean":"2023-10-24T23:53:13.275899+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-24T23:53:13.275899+0000","last_undegraded":"2023-10-24T23:53:13.275899+0000","last_fullsized":"2023-10-24T23:53:13.275899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:02.591627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794746+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-24T23:53:12.794746+0000","last_peered":"2023-10-24T23:53:12.794746+0000","last_clean":"2023-10-24T23:53:12.794746+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-24T23:53:12.794746+0000","last_undegraded":"2023-10-24T23:53:12.794746+0000","last_fullsized":"2023-10-24T23:53:12.794746+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:43.015050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.801731+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-24T23:53:11.801731+0000","last_peered":"2023-10-24T23:53:11.801731+0000","last_clean":"2023-10-24T23:53:11.801731+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-24T23:53:11.801731+0000","last_undegraded":"2023-10-24T23:53:11.801731+0000","last_fullsized":"2023-10-24T23:53:11.801731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825206+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-24T23:53:11.825206+0000","last_peered":"2023-10-24T23:53:11.825206+0000","last_clean":"2023-10-24T23:53:11.825206+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-24T23:53:11.825206+0000","last_undegraded":"2023-10-24T23:53:11.825206+0000","last_fullsized":"2023-10-24T23:53:11.825206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.274975+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-24T23:53:13.274975+0000","last_peered":"2023-10-24T23:53:13.274975+0000","last_clean":"2023-10-24T23:53:13.274975+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-24T23:53:13.274975+0000","last_undegraded":"2023-10-24T23:53:13.274975+0000","last_fullsized":"2023-10-24T23:53:13.274975+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:59.068359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.100994+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-24T23:53:13.100994+0000","last_peered":"2023-10-24T23:53:13.100994+0000","last_clean":"2023-10-24T23:53:13.100994+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-24T23:53:13.100994+0000","last_undegraded":"2023-10-24T23:53:13.100994+0000","last_fullsized":"2023-10-24T23:53:13.100994+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:13:03.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806468+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-24T23:53:11.806468+0000","last_peered":"2023-10-24T23:53:11.806468+0000","last_clean":"2023-10-24T23:53:11.806468+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-24T23:53:11.806468+0000","last_undegraded":"2023-10-24T23:53:11.806468+0000","last_fullsized":"2023-10-24T23:53:11.806468+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.827865+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-24T23:53:11.827865+0000","last_peered":"2023-10-24T23:53:11.827865+0000","last_clean":"2023-10-24T23:53:11.827865+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-24T23:53:11.827865+0000","last_undegraded":"2023-10-24T23:53:11.827865+0000","last_fullsized":"2023-10-24T23:53:11.827865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796200+0000","last_change":"2023-10-24T23:53:11.451013+0000","last_active":"2023-10-24T23:53:12.796200+0000","last_peered":"2023-10-24T23:53:12.796200+0000","last_clean":"2023-10-24T23:53:12.796200+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-24T23:53:12.796200+0000","last_undegraded":"2023-10-24T23:53:12.796200+0000","last_fullsized":"2023-10-24T23:53:12.796200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:52:22.503023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797682+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-24T23:53:12.797682+0000","last_peered":"2023-10-24T23:53:12.797682+0000","last_clean":"2023-10-24T23:53:12.797682+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-24T23:53:12.797682+0000","last_undegraded":"2023-10-24T23:53:12.797682+0000","last_fullsized":"2023-10-24T23:53:12.797682+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:05:30.149693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102137+0000","last_change":"2023-10-24T23:53:11.434538+0000","last_active":"2023-10-24T23:53:13.102137+0000","last_peered":"2023-10-24T23:53:13.102137+0000","last_clean":"2023-10-24T23:53:13.102137+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-24T23:53:13.102137+0000","last_undegraded":"2023-10-24T23:53:13.102137+0000","last_fullsized":"2023-10-24T23:53:13.102137+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:55.179253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789405+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-24T23:53:11.789405+0000","last_peered":"2023-10-24T23:53:11.789405+0000","last_clean":"2023-10-24T23:53:11.789405+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-24T23:53:11.789405+0000","last_undegraded":"2023-10-24T23:53:11.789405+0000","last_fullsized":"2023-10-24T23:53:11.789405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807405+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-24T23:53:11.807405+0000","last_peered":"2023-10-24T23:53:11.807405+0000","last_clean":"2023-10-24T23:53:11.807405+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-24T23:53:11.807405+0000","last_undegraded":"2023-10-24T23:53:11.807405+0000","last_fullsized":"2023-10-24T23:53:11.807405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275539+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-24T23:53:13.275539+0000","last_peered":"2023-10-24T23:53:13.275539+0000","last_clean":"2023-10-24T23:53:13.275539+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-24T23:53:13.275539+0000","last_undegraded":"2023-10-24T23:53:13.275539+0000","last_fullsized":"2023-10-24T23:53:13.275539+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:33:08.373963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795605+0000","last_change":"2023-10-24T23:53:11.433387+0000","last_active":"2023-10-24T23:53:12.795605+0000","last_peered":"2023-10-24T23:53:12.795605+0000","last_clean":"2023-10-24T23:53:12.795605+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-24T23:53:12.795605+0000","last_undegraded":"2023-10-24T23:53:12.795605+0000","last_fullsized":"2023-10-24T23:53:12.795605+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:29.148450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.824306+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-24T23:53:11.824306+0000","last_peered":"2023-10-24T23:53:11.824306+0000","last_clean":"2023-10-24T23:53:11.824306+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-24T23:53:11.824306+0000","last_undegraded":"2023-10-24T23:53:11.824306+0000","last_fullsized":"2023-10-24T23:53:11.824306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788623+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-24T23:53:11.788623+0000","last_peered":"2023-10-24T23:53:11.788623+0000","last_clean":"2023-10-24T23:53:11.788623+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-24T23:53:11.788623+0000","last_undegraded":"2023-10-24T23:53:11.788623+0000","last_fullsized":"2023-10-24T23:53:11.788623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605588+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-24T23:53:13.605588+0000","last_peered":"2023-10-24T23:53:13.605588+0000","last_clean":"2023-10-24T23:53:13.605588+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-24T23:53:13.605588+0000","last_undegraded":"2023-10-24T23:53:13.605588+0000","last_fullsized":"2023-10-24T23:53:13.605588+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:33:06.172412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788704+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-24T23:53:11.788704+0000","last_peered":"2023-10-24T23:53:11.788704+0000","last_clean":"2023-10-24T23:53:11.788704+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-24T23:53:11.788704+0000","last_undegraded":"2023-10-24T23:53:11.788704+0000","last_fullsized":"2023-10-24T23:53:11.788704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832501+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-24T23:53:11.832501+0000","last_peered":"2023-10-24T23:53:11.832501+0000","last_clean":"2023-10-24T23:53:11.832501+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-24T23:53:11.832501+0000","last_undegraded":"2023-10-24T23:53:11.832501+0000","last_fullsized":"2023-10-24T23:53:11.832501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"26'6","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794898+0000","last_change":"2023-10-24T23:53:11.446979+0000","last_active":"2023-10-24T23:53:12.794898+0000","last_peered":"2023-10-24T23:53:12.794898+0000","last_clean":"2023-10-24T23:53:12.794898+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-24T23:53:12.794898+0000","last_undegraded":"2023-10-24T23:53:12.794898+0000","last_fullsized":"2023-10-24T23:53:12.794898+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:54:52.843631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603260+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-24T23:53:13.603260+0000","last_peered":"2023-10-24T23:53:13.603260+0000","last_clean":"2023-10-24T23:53:13.603260+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-24T23:53:13.603260+0000","last_undegraded":"2023-10-24T23:53:13.603260+0000","last_fullsized":"2023-10-24T23:53:13.603260+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:18:48.679682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.816486+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-24T23:53:11.816486+0000","last_peered":"2023-10-24T23:53:11.816486+0000","last_clean":"2023-10-24T23:53:11.816486+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-24T23:53:11.816486+0000","last_undegraded":"2023-10-24T23:53:11.816486+0000","last_fullsized":"2023-10-24T23:53:11.816486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807790+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-24T23:53:11.807790+0000","last_peered":"2023-10-24T23:53:11.807790+0000","last_clean":"2023-10-24T23:53:11.807790+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-24T23:53:11.807790+0000","last_undegraded":"2023-10-24T23:53:11.807790+0000","last_fullsized":"2023-10-24T23:53:11.807790+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101724+0000","last_change":"2023-10-24T23:53:11.436964+0000","last_active":"2023-10-24T23:53:13.101724+0000","last_peered":"2023-10-24T23:53:13.101724+0000","last_clean":"2023-10-24T23:53:13.101724+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-24T23:53:13.101724+0000","last_undegraded":"2023-10-24T23:53:13.101724+0000","last_fullsized":"2023-10-24T23:53:13.101724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:04:15.366966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602222+0000","last_change":"2023-10-24T23:53:11.814304+0000","last_active":"2023-10-24T23:53:13.602222+0000","last_peered":"2023-10-24T23:53:13.602222+0000","last_clean":"2023-10-24T23:53:13.602222+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-24T23:53:13.602222+0000","last_undegraded":"2023-10-24T23:53:13.602222+0000","last_fullsized":"2023-10-24T23:53:13.602222+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:16.635178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789162+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-24T23:53:11.789162+0000","last_peered":"2023-10-24T23:53:11.789162+0000","last_clean":"2023-10-24T23:53:11.789162+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-24T23:53:11.789162+0000","last_undegraded":"2023-10-24T23:53:11.789162+0000","last_fullsized":"2023-10-24T23:53:11.789162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603105+0000","last_change":"2023-10-24T23:53:11.807096+0000","last_active":"2023-10-24T23:53:13.603105+0000","last_peered":"2023-10-24T23:53:13.603105+0000","last_clean":"2023-10-24T23:53:13.603105+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-24T23:53:13.603105+0000","last_undegraded":"2023-10-24T23:53:13.603105+0000","last_fullsized":"2023-10-24T23:53:13.603105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:25.144578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804445+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-24T23:53:11.804445+0000","last_peered":"2023-10-24T23:53:11.804445+0000","last_clean":"2023-10-24T23:53:11.804445+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-24T23:53:11.804445+0000","last_undegraded":"2023-10-24T23:53:11.804445+0000","last_fullsized":"2023-10-24T23:53:11.804445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793680+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-24T23:53:12.793680+0000","last_peered":"2023-10-24T23:53:12.793680+0000","last_clean":"2023-10-24T23:53:12.793680+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-24T23:53:12.793680+0000","last_undegraded":"2023-10-24T23:53:12.793680+0000","last_fullsized":"2023-10-24T23:53:12.793680+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:38.946585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.811071+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-24T23:53:11.811071+0000","last_peered":"2023-10-24T23:53:11.811071+0000","last_clean":"2023-10-24T23:53:11.811071+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-24T23:53:11.811071+0000","last_undegraded":"2023-10-24T23:53:11.811071+0000","last_fullsized":"2023-10-24T23:53:11.811071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797036+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-24T23:53:12.797036+0000","last_peered":"2023-10-24T23:53:12.797036+0000","last_clean":"2023-10-24T23:53:12.797036+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-24T23:53:12.797036+0000","last_undegraded":"2023-10-24T23:53:12.797036+0000","last_fullsized":"2023-10-24T23:53:12.797036+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:48:24.911279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807673+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-24T23:53:11.807673+0000","last_peered":"2023-10-24T23:53:11.807673+0000","last_clean":"2023-10-24T23:53:11.807673+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-24T23:53:11.807673+0000","last_undegraded":"2023-10-24T23:53:11.807673+0000","last_fullsized":"2023-10-24T23:53:11.807673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795515+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-24T23:53:12.795515+0000","last_peered":"2023-10-24T23:53:12.795515+0000","last_clean":"2023-10-24T23:53:12.795515+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-24T23:53:12.795515+0000","last_undegraded":"2023-10-24T23:53:12.795515+0000","last_fullsized":"2023-10-24T23:53:12.795515+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:21.165755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790400+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-24T23:53:11.790400+0000","last_peered":"2023-10-24T23:53:11.790400+0000","last_clean":"2023-10-24T23:53:11.790400+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-24T23:53:11.790400+0000","last_undegraded":"2023-10-24T23:53:11.790400+0000","last_fullsized":"2023-10-24T23:53:11.790400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604073+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-24T23:53:13.604073+0000","last_peered":"2023-10-24T23:53:13.604073+0000","last_clean":"2023-10-24T23:53:13.604073+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-24T23:53:13.604073+0000","last_undegraded":"2023-10-24T23:53:13.604073+0000","last_fullsized":"2023-10-24T23:53:13.604073+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832305+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-24T23:53:11.832305+0000","last_peered":"2023-10-24T23:53:11.832305+0000","last_clean":"2023-10-24T23:53:11.832305+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-24T23:53:11.832305+0000","last_undegraded":"2023-10-24T23:53:11.832305+0000","last_fullsized":"2023-10-24T23:53:11.832305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101339+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-24T23:53:13.101339+0000","last_peered":"2023-10-24T23:53:13.101339+0000","last_clean":"2023-10-24T23:53:13.101339+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-24T23:53:13.101339+0000","last_undegraded":"2023-10-24T23:53:13.101339+0000","last_fullsized":"2023-10-24T23:53:13.101339+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:54.422365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790575+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-24T23:53:11.790575+0000","last_peered":"2023-10-24T23:53:11.790575+0000","last_clean":"2023-10-24T23:53:11.790575+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-24T23:53:11.790575+0000","last_undegraded":"2023-10-24T23:53:11.790575+0000","last_fullsized":"2023-10-24T23:53:11.790575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603531+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-24T23:53:13.603531+0000","last_peered":"2023-10-24T23:53:13.603531+0000","last_clean":"2023-10-24T23:53:13.603531+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-24T23:53:13.603531+0000","last_undegraded":"2023-10-24T23:53:13.603531+0000","last_fullsized":"2023-10-24T23:53:13.603531+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:01:11.838035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789166+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-24T23:53:11.789166+0000","last_peered":"2023-10-24T23:53:11.789166+0000","last_clean":"2023-10-24T23:53:11.789166+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-24T23:53:11.789166+0000","last_undegraded":"2023-10-24T23:53:11.789166+0000","last_fullsized":"2023-10-24T23:53:11.789166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603142+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-24T23:53:13.603142+0000","last_peered":"2023-10-24T23:53:13.603142+0000","last_clean":"2023-10-24T23:53:13.603142+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-24T23:53:13.603142+0000","last_undegraded":"2023-10-24T23:53:13.603142+0000","last_fullsized":"2023-10-24T23:53:13.603142+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:09:25.424344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.827005+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-24T23:53:11.827005+0000","last_peered":"2023-10-24T23:53:11.827005+0000","last_clean":"2023-10-24T23:53:11.827005+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-24T23:53:11.827005+0000","last_undegraded":"2023-10-24T23:53:11.827005+0000","last_fullsized":"2023-10-24T23:53:11.827005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101152+0000","last_change":"2023-10-24T23:53:11.824274+0000","last_active":"2023-10-24T23:53:13.101152+0000","last_peered":"2023-10-24T23:53:13.101152+0000","last_clean":"2023-10-24T23:53:13.101152+0000","last_became_active":"2023-10-24T23:53:11.823586+0000","last_became_peered":"2023-10-24T23:53:11.823586+0000","last_unstale":"2023-10-24T23:53:13.101152+0000","last_undegraded":"2023-10-24T23:53:13.101152+0000","last_fullsized":"2023-10-24T23:53:13.101152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:47:36.970336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804379+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-24T23:53:11.804379+0000","last_peered":"2023-10-24T23:53:11.804379+0000","last_clean":"2023-10-24T23:53:11.804379+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-24T23:53:11.804379+0000","last_undegraded":"2023-10-24T23:53:11.804379+0000","last_fullsized":"2023-10-24T23:53:11.804379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.274967+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-24T23:53:13.274967+0000","last_peered":"2023-10-24T23:53:13.274967+0000","last_clean":"2023-10-24T23:53:13.274967+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-24T23:53:13.274967+0000","last_undegraded":"2023-10-24T23:53:13.274967+0000","last_fullsized":"2023-10-24T23:53:13.274967+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:10:20.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788652+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-24T23:53:11.788652+0000","last_peered":"2023-10-24T23:53:11.788652+0000","last_clean":"2023-10-24T23:53:11.788652+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-24T23:53:11.788652+0000","last_undegraded":"2023-10-24T23:53:11.788652+0000","last_fullsized":"2023-10-24T23:53:11.788652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796443+0000","last_change":"2023-10-24T23:53:11.860133+0000","last_active":"2023-10-24T23:53:12.796443+0000","last_peered":"2023-10-24T23:53:12.796443+0000","last_clean":"2023-10-24T23:53:12.796443+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-24T23:53:12.796443+0000","last_undegraded":"2023-10-24T23:53:12.796443+0000","last_fullsized":"2023-10-24T23:53:12.796443+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:43.051362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.816202+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-24T23:53:11.816202+0000","last_peered":"2023-10-24T23:53:11.816202+0000","last_clean":"2023-10-24T23:53:11.816202+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-24T23:53:11.816202+0000","last_undegraded":"2023-10-24T23:53:11.816202+0000","last_fullsized":"2023-10-24T23:53:11.816202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603745+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-24T23:53:13.603745+0000","last_peered":"2023-10-24T23:53:13.603745+0000","last_clean":"2023-10-24T23:53:13.603745+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-24T23:53:13.603745+0000","last_undegraded":"2023-10-24T23:53:13.603745+0000","last_fullsized":"2023-10-24T23:53:13.603745+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:39:14.947372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793137+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-24T23:53:11.793137+0000","last_peered":"2023-10-24T23:53:11.793137+0000","last_clean":"2023-10-24T23:53:11.793137+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-24T23:53:11.793137+0000","last_undegraded":"2023-10-24T23:53:11.793137+0000","last_fullsized":"2023-10-24T23:53:11.793137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.798110+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-24T23:53:12.798110+0000","last_peered":"2023-10-24T23:53:12.798110+0000","last_clean":"2023-10-24T23:53:12.798110+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-24T23:53:12.798110+0000","last_undegraded":"2023-10-24T23:53:12.798110+0000","last_fullsized":"2023-10-24T23:53:12.798110+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:37.413259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.796910+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-24T23:53:11.796910+0000","last_peered":"2023-10-24T23:53:11.796910+0000","last_clean":"2023-10-24T23:53:11.796910+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-24T23:53:11.796910+0000","last_undegraded":"2023-10-24T23:53:11.796910+0000","last_fullsized":"2023-10-24T23:53:11.796910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603512+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-24T23:53:13.603512+0000","last_peered":"2023-10-24T23:53:13.603512+0000","last_clean":"2023-10-24T23:53:13.603512+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-24T23:53:13.603512+0000","last_undegraded":"2023-10-24T23:53:13.603512+0000","last_fullsized":"2023-10-24T23:53:13.603512+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:31.344724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803849+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-24T23:53:11.803849+0000","last_peered":"2023-10-24T23:53:11.803849+0000","last_clean":"2023-10-24T23:53:11.803849+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-24T23:53:11.803849+0000","last_undegraded":"2023-10-24T23:53:11.803849+0000","last_fullsized":"2023-10-24T23:53:11.803849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275469+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-24T23:53:13.275469+0000","last_peered":"2023-10-24T23:53:13.275469+0000","last_clean":"2023-10-24T23:53:13.275469+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-24T23:53:13.275469+0000","last_undegraded":"2023-10-24T23:53:13.275469+0000","last_fullsized":"2023-10-24T23:53:13.275469+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788160+0000","last_change":"2023-10-24T23:53:02.728971+0000","last_active":"2023-10-24T23:53:11.788160+0000","last_peered":"2023-10-24T23:53:11.788160+0000","last_clean":"2023-10-24T23:53:11.788160+0000","last_became_active":"2023-10-24T23:53:02.728048+0000","last_became_peered":"2023-10-24T23:53:02.728048+0000","last_unstale":"2023-10-24T23:53:11.788160+0000","last_undegraded":"2023-10-24T23:53:11.788160+0000","last_fullsized":"2023-10-24T23:53:11.788160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:31:32.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.601984+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-24T23:53:13.601984+0000","last_peered":"2023-10-24T23:53:13.601984+0000","last_clean":"2023-10-24T23:53:13.601984+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-24T23:53:13.601984+0000","last_undegraded":"2023-10-24T23:53:13.601984+0000","last_fullsized":"2023-10-24T23:53:13.601984+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:32:05.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791536+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-24T23:53:11.791536+0000","last_peered":"2023-10-24T23:53:11.791536+0000","last_clean":"2023-10-24T23:53:11.791536+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-24T23:53:11.791536+0000","last_undegraded":"2023-10-24T23:53:11.791536+0000","last_fullsized":"2023-10-24T23:53:11.791536+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101460+0000","last_change":"2023-10-24T23:53:11.847814+0000","last_active":"2023-10-24T23:53:13.101460+0000","last_peered":"2023-10-24T23:53:13.101460+0000","last_clean":"2023-10-24T23:53:13.101460+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-24T23:53:13.101460+0000","last_undegraded":"2023-10-24T23:53:13.101460+0000","last_fullsized":"2023-10-24T23:53:13.101460+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:59.075858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825376+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-24T23:53:11.825376+0000","last_peered":"2023-10-24T23:53:11.825376+0000","last_clean":"2023-10-24T23:53:11.825376+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-24T23:53:11.825376+0000","last_undegraded":"2023-10-24T23:53:11.825376+0000","last_fullsized":"2023-10-24T23:53:11.825376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796122+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-24T23:53:12.796122+0000","last_peered":"2023-10-24T23:53:12.796122+0000","last_clean":"2023-10-24T23:53:12.796122+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-24T23:53:12.796122+0000","last_undegraded":"2023-10-24T23:53:12.796122+0000","last_fullsized":"2023-10-24T23:53:12.796122+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:25:12.525273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828503+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-24T23:53:11.828503+0000","last_peered":"2023-10-24T23:53:11.828503+0000","last_clean":"2023-10-24T23:53:11.828503+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-24T23:53:11.828503+0000","last_undegraded":"2023-10-24T23:53:11.828503+0000","last_fullsized":"2023-10-24T23:53:11.828503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101011+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-24T23:53:13.101011+0000","last_peered":"2023-10-24T23:53:13.101011+0000","last_clean":"2023-10-24T23:53:13.101011+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-24T23:53:13.101011+0000","last_undegraded":"2023-10-24T23:53:13.101011+0000","last_fullsized":"2023-10-24T23:53:13.101011+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:12:21.026996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.795075+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-24T23:53:11.795075+0000","last_peered":"2023-10-24T23:53:11.795075+0000","last_clean":"2023-10-24T23:53:11.795075+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-24T23:53:11.795075+0000","last_undegraded":"2023-10-24T23:53:11.795075+0000","last_fullsized":"2023-10-24T23:53:11.795075+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101949+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-24T23:53:13.101949+0000","last_peered":"2023-10-24T23:53:13.101949+0000","last_clean":"2023-10-24T23:53:13.101949+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-24T23:53:13.101949+0000","last_undegraded":"2023-10-24T23:53:13.101949+0000","last_fullsized":"2023-10-24T23:53:13.101949+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:07.909021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.797593+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-24T23:53:11.797593+0000","last_peered":"2023-10-24T23:53:11.797593+0000","last_clean":"2023-10-24T23:53:11.797593+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-24T23:53:11.797593+0000","last_undegraded":"2023-10-24T23:53:11.797593+0000","last_fullsized":"2023-10-24T23:53:11.797593+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603906+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-24T23:53:13.603906+0000","last_peered":"2023-10-24T23:53:13.603906+0000","last_clean":"2023-10-24T23:53:13.603906+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-24T23:53:13.603906+0000","last_undegraded":"2023-10-24T23:53:13.603906+0000","last_fullsized":"2023-10-24T23:53:13.603906+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:48.217889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800508+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-24T23:53:11.800508+0000","last_peered":"2023-10-24T23:53:11.800508+0000","last_clean":"2023-10-24T23:53:11.800508+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-24T23:53:11.800508+0000","last_undegraded":"2023-10-24T23:53:11.800508+0000","last_fullsized":"2023-10-24T23:53:11.800508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602960+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-24T23:53:13.602960+0000","last_peered":"2023-10-24T23:53:13.602960+0000","last_clean":"2023-10-24T23:53:13.602960+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-24T23:53:13.602960+0000","last_undegraded":"2023-10-24T23:53:13.602960+0000","last_fullsized":"2023-10-24T23:53:13.602960+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:10:18.973606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838707+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-24T23:53:11.838707+0000","last_peered":"2023-10-24T23:53:11.838707+0000","last_clean":"2023-10-24T23:53:11.838707+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-24T23:53:11.838707+0000","last_undegraded":"2023-10-24T23:53:11.838707+0000","last_fullsized":"2023-10-24T23:53:11.838707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101457+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-24T23:53:13.101457+0000","last_peered":"2023-10-24T23:53:13.101457+0000","last_clean":"2023-10-24T23:53:13.101457+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-24T23:53:13.101457+0000","last_undegraded":"2023-10-24T23:53:13.101457+0000","last_fullsized":"2023-10-24T23:53:13.101457+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:24:45.536141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788141+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-24T23:53:11.788141+0000","last_peered":"2023-10-24T23:53:11.788141+0000","last_clean":"2023-10-24T23:53:11.788141+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-24T23:53:11.788141+0000","last_undegraded":"2023-10-24T23:53:11.788141+0000","last_fullsized":"2023-10-24T23:53:11.788141+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793724+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-24T23:53:12.793724+0000","last_peered":"2023-10-24T23:53:12.793724+0000","last_clean":"2023-10-24T23:53:12.793724+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-24T23:53:12.793724+0000","last_undegraded":"2023-10-24T23:53:12.793724+0000","last_fullsized":"2023-10-24T23:53:12.793724+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:06.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806250+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-24T23:53:11.806250+0000","last_peered":"2023-10-24T23:53:11.806250+0000","last_clean":"2023-10-24T23:53:11.806250+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-24T23:53:11.806250+0000","last_undegraded":"2023-10-24T23:53:11.806250+0000","last_fullsized":"2023-10-24T23:53:11.806250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275114+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-24T23:53:13.275114+0000","last_peered":"2023-10-24T23:53:13.275114+0000","last_clean":"2023-10-24T23:53:13.275114+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-24T23:53:13.275114+0000","last_undegraded":"2023-10-24T23:53:13.275114+0000","last_fullsized":"2023-10-24T23:53:13.275114+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:56:02.798009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.823970+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-24T23:53:11.823970+0000","last_peered":"2023-10-24T23:53:11.823970+0000","last_clean":"2023-10-24T23:53:11.823970+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-24T23:53:11.823970+0000","last_undegraded":"2023-10-24T23:53:11.823970+0000","last_fullsized":"2023-10-24T23:53:11.823970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793903+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-24T23:53:12.793903+0000","last_peered":"2023-10-24T23:53:12.793903+0000","last_clean":"2023-10-24T23:53:12.793903+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-24T23:53:12.793903+0000","last_undegraded":"2023-10-24T23:53:12.793903+0000","last_fullsized":"2023-10-24T23:53:12.793903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:40.168016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.809345+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-24T23:53:11.809345+0000","last_peered":"2023-10-24T23:53:11.809345+0000","last_clean":"2023-10-24T23:53:11.809345+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-24T23:53:11.809345+0000","last_undegraded":"2023-10-24T23:53:11.809345+0000","last_fullsized":"2023-10-24T23:53:11.809345+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.100799+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-24T23:53:13.100799+0000","last_peered":"2023-10-24T23:53:13.100799+0000","last_clean":"2023-10-24T23:53:13.100799+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-24T23:53:13.100799+0000","last_undegraded":"2023-10-24T23:53:13.100799+0000","last_fullsized":"2023-10-24T23:53:13.100799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:39.720375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788501+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-24T23:53:11.788501+0000","last_peered":"2023-10-24T23:53:11.788501+0000","last_clean":"2023-10-24T23:53:11.788501+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-24T23:53:11.788501+0000","last_undegraded":"2023-10-24T23:53:11.788501+0000","last_fullsized":"2023-10-24T23:53:11.788501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794282+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-24T23:53:12.794282+0000","last_peered":"2023-10-24T23:53:12.794282+0000","last_clean":"2023-10-24T23:53:12.794282+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-24T23:53:12.794282+0000","last_undegraded":"2023-10-24T23:53:12.794282+0000","last_fullsized":"2023-10-24T23:53:12.794282+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:51:40.960630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790034+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-24T23:53:11.790034+0000","last_peered":"2023-10-24T23:53:11.790034+0000","last_clean":"2023-10-24T23:53:11.790034+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-24T23:53:11.790034+0000","last_undegraded":"2023-10-24T23:53:11.790034+0000","last_fullsized":"2023-10-24T23:53:11.790034+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795555+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-24T23:53:12.795555+0000","last_peered":"2023-10-24T23:53:12.795555+0000","last_clean":"2023-10-24T23:53:12.795555+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-24T23:53:12.795555+0000","last_undegraded":"2023-10-24T23:53:12.795555+0000","last_fullsized":"2023-10-24T23:53:12.795555+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:22:42.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793089+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-24T23:53:11.793089+0000","last_peered":"2023-10-24T23:53:11.793089+0000","last_clean":"2023-10-24T23:53:11.793089+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-24T23:53:11.793089+0000","last_undegraded":"2023-10-24T23:53:11.793089+0000","last_fullsized":"2023-10-24T23:53:11.793089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275507+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-24T23:53:13.275507+0000","last_peered":"2023-10-24T23:53:13.275507+0000","last_clean":"2023-10-24T23:53:13.275507+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-24T23:53:13.275507+0000","last_undegraded":"2023-10-24T23:53:13.275507+0000","last_fullsized":"2023-10-24T23:53:13.275507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:47.658717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838499+0000","last_change":"2023-10-24T23:53:02.715966+0000","last_active":"2023-10-24T23:53:11.838499+0000","last_peered":"2023-10-24T23:53:11.838499+0000","last_clean":"2023-10-24T23:53:11.838499+0000","last_became_active":"2023-10-24T23:53:02.715631+0000","last_became_peered":"2023-10-24T23:53:02.715631+0000","last_unstale":"2023-10-24T23:53:11.838499+0000","last_undegraded":"2023-10-24T23:53:11.838499+0000","last_fullsized":"2023-10-24T23:53:11.838499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:11.771864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603292+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-24T23:53:13.603292+0000","last_peered":"2023-10-24T23:53:13.603292+0000","last_clean":"2023-10-24T23:53:13.603292+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-24T23:53:13.603292+0000","last_undegraded":"2023-10-24T23:53:13.603292+0000","last_fullsized":"2023-10-24T23:53:13.603292+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:50.994593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.796890+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-24T23:53:11.796890+0000","last_peered":"2023-10-24T23:53:11.796890+0000","last_clean":"2023-10-24T23:53:11.796890+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-24T23:53:11.796890+0000","last_undegraded":"2023-10-24T23:53:11.796890+0000","last_fullsized":"2023-10-24T23:53:11.796890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":929792,"data_stored":713128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":294912,"data_stored":119928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0549999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94499999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":307200,"data_stored":120110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":319488,"data_stored":121570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":121034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":892928,"data_stored":710330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0209999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:24.897 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-24T23:53:24.898 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-24T23:53:24.898 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-24T23:53:24.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-24T23:53:25.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.038+0000 7f7e7a912640 1 Processor -- start 2023-10-24T23:53:25.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.038+0000 7f7e7a912640 1 -- start start 2023-10-24T23:53:25.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e7a912640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74151960 0x7f7e74071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e7a912640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e74150bc0 con 0x7f7e74071220 2023-10-24T23:53:25.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e7a912640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7406c510 con 0x7f7e740c24c0 2023-10-24T23:53:25.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e7a912640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7406c7f0 con 0x7f7e74151960 2023-10-24T23:53:25.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e73fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7e74071220 0x7f7e7406bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38242/0 (socket says 172.21.15.107:38242) 2023-10-24T23:53:25.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e73fff640 1 -- 172.21.15.107:0/2648893717 learned_addr learned my addr 172.21.15.107:0/2648893717 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246625246 0 0) 0x7f7e74150bc0 con 0x7f7e74071220 2023-10-24T23:53:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e6b7fe640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74151960 0x7f7e74071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e60003530 con 0x7f7e74071220 2023-10-24T23:53:25.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1496236066 0 0) 0x7f7e60003530 con 0x7f7e74071220 2023-10-24T23:53:25.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74151960 msgr2=0x7f7e74071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:25.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74151960 0x7f7e74071a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 >> v1:172.21.15.110:6789/0 conn(0x7f7e740c24c0 legacy=0x7f7e7406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e74150e80 con 0x7f7e74071220 2023-10-24T23:53:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f7e64001680 con 0x7f7e74071220 2023-10-24T23:53:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 >> v1:172.21.15.107:6789/0 conn(0x7f7e74071220 legacy=0x7f7e7406bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74151960 0x7f7e600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 0x7f7e600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e60004d10 0x7f7e60005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7406c7f0 con 0x7f7e60003fe0 2023-10-24T23:53:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6000bbd0 con 0x7f7e60004d10 2023-10-24T23:53:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6000bd90 con 0x7f7e74151960 2023-10-24T23:53:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e6b7fe640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 0x7f7e600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e73fff640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74151960 0x7f7e600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e78eaf640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e60004d10 0x7f7e60005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e6b7fe640 1 -- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74151960 msgr2=0x7f7e600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e6b7fe640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74151960 0x7f7e600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e6b7fe640 1 -- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e60004d10 msgr2=0x7f7e60005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:25.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e6b7fe640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e60004d10 0x7f7e60005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e6b7fe640 1 -- 172.21.15.107:0/2648893717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e60003530 con 0x7f7e60003fe0 2023-10-24T23:53:25.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e78eaf640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e60004d10 0x7f7e60005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:25.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e6b7fe640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 0x7f7e600047d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00ba30 tx=0x7f7e5c01c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3def44a3577d4cb server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c0ae680 con 0x7f7e60003fe0 2023-10-24T23:53:25.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5c0ae7e0 con 0x7f7e60003fe0 2023-10-24T23:53:25.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.042+0000 7f7e737fe640 1 -- 172.21.15.107:0/2648893717 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c0aeb90 con 0x7f7e60003fe0 2023-10-24T23:53:25.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 -- 172.21.15.107:0/2648893717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 msgr2=0x7f7e600047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 --2- 172.21.15.107:0/2648893717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 0x7f7e600047d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00ba30 tx=0x7f7e5c01c680 comp rx=0 tx=0).stop 2023-10-24T23:53:25.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e78eaf640 1 -- 172.21.15.107:0/2648893717 reap_dead start 2023-10-24T23:53:25.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 -- 172.21.15.107:0/2648893717 shutdown_connections 2023-10-24T23:53:25.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 -- 172.21.15.107:0/2648893717 >> 172.21.15.107:0/2648893717 conn(0x7f7e740c2950 msgr2=0x7f7e740c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:25.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 -- 172.21.15.107:0/2648893717 shutdown_connections 2023-10-24T23:53:25.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 -- 172.21.15.107:0/2648893717 wait complete. 2023-10-24T23:53:25.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 Processor -- start 2023-10-24T23:53:25.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 -- start start 2023-10-24T23:53:25.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 0x7f7e74155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e7413a1f0 0x7f7e741429b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74142ef0 0x7f7e741eddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e74155d10 con 0x7f7e60003fe0 2023-10-24T23:53:25.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e741ea850 con 0x7f7e7413a1f0 2023-10-24T23:53:25.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7413faa0 con 0x7f7e74142ef0 2023-10-24T23:53:25.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 0x7f7e74155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e73fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 0x7f7e74155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33236/0 (socket says 172.21.15.107:33236) 2023-10-24T23:53:25.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e73fff640 1 -- 172.21.15.107:0/1801271633 learned_addr learned my addr 172.21.15.107:0/1801271633 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:25.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e6b7fe640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e7413a1f0 0x7f7e741429b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e73fff640 1 -- 172.21.15.107:0/1801271633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74142ef0 msgr2=0x7f7e741eddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e78eaf640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74142ef0 0x7f7e741eddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e73fff640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74142ef0 0x7f7e741eddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e73fff640 1 -- 172.21.15.107:0/1801271633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e7413a1f0 msgr2=0x7f7e741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e73fff640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e7413a1f0 0x7f7e741429b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e73fff640 1 -- 172.21.15.107:0/1801271633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e741eeb30 con 0x7f7e60003fe0 2023-10-24T23:53:25.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e6b7fe640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e7413a1f0 0x7f7e741429b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:25.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e78eaf640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74142ef0 0x7f7e741eddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:25.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e73fff640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 0x7f7e74155510 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e840 tx=0x7f7e6400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e717fa640 1 -- 172.21.15.107:0/1801271633 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400c870 con 0x7f7e60003fe0 2023-10-24T23:53:25.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e717fa640 1 -- 172.21.15.107:0/1801271633 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6400c9d0 con 0x7f7e60003fe0 2023-10-24T23:53:25.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 -- 172.21.15.107:0/1801271633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e74156e00 con 0x7f7e60003fe0 2023-10-24T23:53:25.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e717fa640 1 -- 172.21.15.107:0/1801271633 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400cd40 con 0x7f7e60003fe0 2023-10-24T23:53:25.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.046+0000 7f7e7a912640 1 -- 172.21.15.107:0/1801271633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e741573c0 con 0x7f7e60003fe0 2023-10-24T23:53:25.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.050+0000 7f7e7a912640 1 -- 172.21.15.107:0/1801271633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e740746e0 con 0x7f7e60003fe0 2023-10-24T23:53:25.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.050+0000 7f7e717fa640 1 -- 172.21.15.107:0/1801271633 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7e6401d460 con 0x7f7e60003fe0 2023-10-24T23:53:25.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.050+0000 7f7e717fa640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 0x7f7e54068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.050+0000 7f7e717fa640 1 -- 172.21.15.107:0/1801271633 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f7e64083b10 con 0x7f7e60003fe0 2023-10-24T23:53:25.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.050+0000 7f7e6b7fe640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 0x7f7e54068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.050+0000 7f7e6b7fe640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 0x7f7e54068320 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0b0000 tx=0x7f7e5c0046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.054+0000 7f7e717fa640 1 -- 172.21.15.107:0/1801271633 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7e64050ec0 con 0x7f7e60003fe0 2023-10-24T23:53:25.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 -- 172.21.15.107:0/1801271633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7e740c22a0 con 0x7f7e60003fe0 2023-10-24T23:53:25.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e717fa640 1 -- 172.21.15.107:0/1801271633 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f7e640544e0 con 0x7f7e60003fe0 2023-10-24T23:53:25.230 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:25.230 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-24T23:53:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 -- 172.21.15.107:0/1801271633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 msgr2=0x7f7e54068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 0x7f7e54068320 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0b0000 tx=0x7f7e5c0046d0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 -- 172.21.15.107:0/1801271633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 msgr2=0x7f7e74155510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 0x7f7e74155510 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400e840 tx=0x7f7e6400ed00 comp rx=0 tx=0).stop 2023-10-24T23:53:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 -- 172.21.15.107:0/1801271633 shutdown_connections 2023-10-24T23:53:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7e54065e60 0x7f7e54068320 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7e74142ef0 0x7f7e741eddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7e7413a1f0 0x7f7e741429b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 --2- 172.21.15.107:0/1801271633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e60003fe0 0x7f7e74155510 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 -- 172.21.15.107:0/1801271633 >> 172.21.15.107:0/1801271633 conn(0x7f7e740c2950 msgr2=0x7f7e7406a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 -- 172.21.15.107:0/1801271633 shutdown_connections 2023-10-24T23:53:25.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.226+0000 7f7e7a912640 1 -- 172.21.15.107:0/1801271633 wait complete. 2023-10-24T23:53:25.247 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-24T23:53:25.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23e600640 1 Processor -- start 2023-10-24T23:53:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23e600640 1 -- start start 2023-10-24T23:53:25.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23e600640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe23814c470 0x7fe23814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23e600640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe238152b30 0x7fe238152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23e600640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe238155cd0 con 0x7fe23806f490 2023-10-24T23:53:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23e600640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2381526e0 con 0x7fe238152b30 2023-10-24T23:53:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23e600640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2380c4380 con 0x7fe23814c470 2023-10-24T23:53:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23cb9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe238152b30 0x7fe238152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23cb9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe238152b30 0x7fe238152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33242/0 (socket says 172.21.15.107:33242) 2023-10-24T23:53:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23cb9d640 1 -- 172.21.15.107:0/1286052027 learned_addr learned my addr 172.21.15.107:0/1286052027 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe237fff640 1 --2- 172.21.15.107:0/1286052027 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe23814c470 0x7fe23814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23cb9d640 1 -- 172.21.15.107:0/1286052027 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe23814c470 msgr2=0x7fe23814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:25.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23cb9d640 1 --2- 172.21.15.107:0/1286052027 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe23814c470 0x7fe23814c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23cb9d640 1 -- 172.21.15.107:0/1286052027 >> v1:172.21.15.110:6789/0 conn(0x7fe23806f490 legacy=0x7fe238071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:25.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23cb9d640 1 -- 172.21.15.107:0/1286052027 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2380c48b0 con 0x7fe238152b30 2023-10-24T23:53:25.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe23cb9d640 1 --2- 172.21.15.107:0/1286052027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe238152b30 0x7fe238152fa0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe230009e50 tx=0x7fe23001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9ef45ba8ffef9 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe236ffd640 1 -- 172.21.15.107:0/1286052027 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2300a5620 con 0x7fe238152b30 2023-10-24T23:53:25.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe236ffd640 1 -- 172.21.15.107:0/1286052027 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2300a5780 con 0x7fe238152b30 2023-10-24T23:53:25.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.358+0000 7fe236ffd640 1 -- 172.21.15.107:0/1286052027 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2300a5b10 con 0x7fe238152b30 2023-10-24T23:53:25.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 -- 172.21.15.107:0/1286052027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe238152b30 msgr2=0x7fe238152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 --2- 172.21.15.107:0/1286052027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe238152b30 0x7fe238152fa0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe230009e50 tx=0x7fe23001c8b0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 -- 172.21.15.107:0/1286052027 shutdown_connections 2023-10-24T23:53:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 --2- 172.21.15.107:0/1286052027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe238152b30 0x7fe238152fa0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 --2- 172.21.15.107:0/1286052027 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe23814c470 0x7fe23814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 -- 172.21.15.107:0/1286052027 >> 172.21.15.107:0/1286052027 conn(0x7fe238070f60 msgr2=0x7fe238071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 -- 172.21.15.107:0/1286052027 shutdown_connections 2023-10-24T23:53:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 -- 172.21.15.107:0/1286052027 wait complete. 2023-10-24T23:53:25.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 Processor -- start 2023-10-24T23:53:25.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 -- start start 2023-10-24T23:53:25.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe23806f490 0x7fe2380c3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe23814c470 0x7fe23814ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe238152b30 0x7fe23814f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe23814d2a0 con 0x7fe23806f490 2023-10-24T23:53:25.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2380aacd0 con 0x7fe238152b30 2023-10-24T23:53:25.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe23814ca90 con 0x7fe23814c470 2023-10-24T23:53:25.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe237fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe23806f490 0x7fe2380c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe237fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe23806f490 0x7fe2380c3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33244/0 (socket says 172.21.15.107:33244) 2023-10-24T23:53:25.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe237fff640 1 -- 172.21.15.107:0/4148236868 learned_addr learned my addr 172.21.15.107:0/4148236868 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:25.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe2377fe640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe23814c470 0x7fe23814ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23cb9d640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe238152b30 0x7fe23814f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe237fff640 1 -- 172.21.15.107:0/4148236868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe23814c470 msgr2=0x7fe23814ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe237fff640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe23814c470 0x7fe23814ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe237fff640 1 -- 172.21.15.107:0/4148236868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe238152b30 msgr2=0x7fe23814f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe237fff640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe238152b30 0x7fe23814f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe237fff640 1 -- 172.21.15.107:0/4148236868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe230009b30 con 0x7fe23806f490 2023-10-24T23:53:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe237fff640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe23806f490 0x7fe2380c3060 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe22800d800 tx=0x7fe22800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe234ff9640 1 -- 172.21.15.107:0/4148236868 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22800b6c0 con 0x7fe23806f490 2023-10-24T23:53:25.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe234ff9640 1 -- 172.21.15.107:0/4148236868 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22800b820 con 0x7fe23806f490 2023-10-24T23:53:25.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe234ff9640 1 -- 172.21.15.107:0/4148236868 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22800bb90 con 0x7fe23806f490 2023-10-24T23:53:25.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 -- 172.21.15.107:0/4148236868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2381ee870 con 0x7fe23806f490 2023-10-24T23:53:25.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.362+0000 7fe23e600640 1 -- 172.21.15.107:0/4148236868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe23814cfc0 con 0x7fe23806f490 2023-10-24T23:53:25.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.366+0000 7fe234ff9640 1 -- 172.21.15.107:0/4148236868 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe22800bcf0 con 0x7fe23806f490 2023-10-24T23:53:25.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.366+0000 7fe23e600640 1 -- 172.21.15.107:0/4148236868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2380746e0 con 0x7fe23806f490 2023-10-24T23:53:25.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.366+0000 7fe234ff9640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe224065e60 0x7fe224068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.366+0000 7fe234ff9640 1 -- 172.21.15.107:0/4148236868 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fe228084000 con 0x7fe23806f490 2023-10-24T23:53:25.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.366+0000 7fe2377fe640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe224065e60 0x7fe224068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.366+0000 7fe2377fe640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe224065e60 0x7fe224068320 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe238151160 tx=0x7fe22c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.370+0000 7fe234ff9640 1 -- 172.21.15.107:0/4148236868 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe2280513b0 con 0x7fe23806f490 2023-10-24T23:53:25.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.510+0000 7fe23e600640 1 -- 172.21.15.107:0/4148236868 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2380ac270 con 0x7fe23806f490 2023-10-24T23:53:25.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe234ff9640 1 -- 172.21.15.107:0/4148236868 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21531 (secure 0 0 0) 0x7fe2280549d0 con 0x7fe23806f490 2023-10-24T23:53:25.518 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:25.519 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:12.781093+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:25.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 -- 172.21.15.107:0/4148236868 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe224065e60 msgr2=0x7fe224068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe224065e60 0x7fe224068320 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe238151160 tx=0x7fe22c009040 comp rx=0 tx=0).stop 2023-10-24T23:53:25.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 -- 172.21.15.107:0/4148236868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe23806f490 msgr2=0x7fe2380c3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe23806f490 0x7fe2380c3060 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe22800d800 tx=0x7fe22800dcc0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 -- 172.21.15.107:0/4148236868 shutdown_connections 2023-10-24T23:53:25.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe238152b30 0x7fe23814f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe224065e60 0x7fe224068320 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe23814c470 0x7fe23814ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 --2- 172.21.15.107:0/4148236868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe23806f490 0x7fe2380c3060 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 -- 172.21.15.107:0/4148236868 >> 172.21.15.107:0/4148236868 conn(0x7fe238070f60 msgr2=0x7fe238152f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:25.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 -- 172.21.15.107:0/4148236868 shutdown_connections 2023-10-24T23:53:25.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.514+0000 7fe23e600640 1 -- 172.21.15.107:0/4148236868 wait complete. 2023-10-24T23:53:25.535 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-10-24T23:53:09.786122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-10-24T23:53:10.766645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-24T23:53:25.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-24T23:53:25.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.642+0000 7f37bb65d640 1 Processor -- start 2023-10-24T23:53:25.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.642+0000 7f37bb65d640 1 -- start start 2023-10-24T23:53:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.642+0000 7f37bb65d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37b40ab1a0 0x7f37b40c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.642+0000 7f37bb65d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37b4072720 0x7f37b40bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.642+0000 7f37bb65d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b40c1ce0 con 0x7f37b40bceb0 2023-10-24T23:53:25.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.642+0000 7f37bb65d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37b40be1d0 con 0x7f37b40ab1a0 2023-10-24T23:53:25.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.642+0000 7f37bb65d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40c0880 con 0x7f37b4072720 2023-10-24T23:53:25.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.642+0000 7f37b93f9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37b40ab1a0 0x7f37b40c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.642+0000 7f37b93f9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37b40ab1a0 0x7f37b40c0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33256/0 (socket says 172.21.15.107:33256) 2023-10-24T23:53:25.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.642+0000 7f37b93f9640 1 -- 172.21.15.107:0/1559990380 learned_addr learned my addr 172.21.15.107:0/1559990380 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:25.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37b93f9640 1 -- 172.21.15.107:0/1559990380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37b4072720 msgr2=0x7f37b40bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:25.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37b93f9640 1 --2- 172.21.15.107:0/1559990380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37b4072720 0x7f37b40bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37b93f9640 1 -- 172.21.15.107:0/1559990380 >> v1:172.21.15.110:6789/0 conn(0x7f37b40bceb0 legacy=0x7f37b40c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:25.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37b93f9640 1 -- 172.21.15.107:0/1559990380 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b40c1f30 con 0x7f37b40ab1a0 2023-10-24T23:53:25.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37b93f9640 1 --2- 172.21.15.107:0/1559990380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b40ab1a0 0x7f37b40c0d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009870 tx=0x7f37b001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604fd4d530fe2f9b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37abfff640 1 -- 172.21.15.107:0/1559990380 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b00ab600 con 0x7f37b40ab1a0 2023-10-24T23:53:25.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37abfff640 1 -- 172.21.15.107:0/1559990380 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b00a7050 con 0x7f37b40ab1a0 2023-10-24T23:53:25.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37abfff640 1 -- 172.21.15.107:0/1559990380 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b00ab980 con 0x7f37b40ab1a0 2023-10-24T23:53:25.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37bb65d640 1 -- 172.21.15.107:0/1559990380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b40ab1a0 msgr2=0x7f37b40c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37bb65d640 1 --2- 172.21.15.107:0/1559990380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b40ab1a0 0x7f37b40c0d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009870 tx=0x7f37b001c3d0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37bb65d640 1 -- 172.21.15.107:0/1559990380 shutdown_connections 2023-10-24T23:53:25.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37bb65d640 1 --2- 172.21.15.107:0/1559990380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37b4072720 0x7f37b40bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37bb65d640 1 --2- 172.21.15.107:0/1559990380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b40ab1a0 0x7f37b40c0d00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37bb65d640 1 -- 172.21.15.107:0/1559990380 >> 172.21.15.107:0/1559990380 conn(0x7f37b40c2860 msgr2=0x7f37b40c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37bb65d640 1 -- 172.21.15.107:0/1559990380 shutdown_connections 2023-10-24T23:53:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.646+0000 7f37bb65d640 1 -- 172.21.15.107:0/1559990380 wait complete. 2023-10-24T23:53:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37bb65d640 1 Processor -- start 2023-10-24T23:53:25.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37bb65d640 1 -- start start 2023-10-24T23:53:25.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37bb65d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b40ab1a0 0x7f37b40a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37bb65d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37b40bceb0 0x7f37b4158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37bb65d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37b41f1060 0x7f37b41eceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37bb65d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40a7710 con 0x7f37b40ab1a0 2023-10-24T23:53:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37bb65d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37b40a71e0 con 0x7f37b41f1060 2023-10-24T23:53:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37bb65d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37b41f05c0 con 0x7f37b40bceb0 2023-10-24T23:53:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b93f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b40ab1a0 0x7f37b40a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b93f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b40ab1a0 0x7f37b40a9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33268/0 (socket says 172.21.15.107:33268) 2023-10-24T23:53:25.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b93f9640 1 -- 172.21.15.107:0/1308661176 learned_addr learned my addr 172.21.15.107:0/1308661176 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b93f9640 1 -- 172.21.15.107:0/1308661176 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37b40bceb0 msgr2=0x7f37b4158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b8bf8640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37b40bceb0 0x7f37b4158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b9bfa640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37b41f1060 0x7f37b41eceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b93f9640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37b40bceb0 0x7f37b4158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b93f9640 1 -- 172.21.15.107:0/1308661176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37b41f1060 msgr2=0x7f37b41eceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b93f9640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37b41f1060 0x7f37b41eceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b93f9640 1 -- 172.21.15.107:0/1308661176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b00094d0 con 0x7f37b40ab1a0 2023-10-24T23:53:25.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b93f9640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b40ab1a0 0x7f37b40a9dd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f37b00ae040 tx=0x7f37b001ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37a9ffb640 1 -- 172.21.15.107:0/1308661176 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b00b6620 con 0x7f37b40ab1a0 2023-10-24T23:53:25.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37a9ffb640 1 -- 172.21.15.107:0/1308661176 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b00b6780 con 0x7f37b40ab1a0 2023-10-24T23:53:25.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37bb65d640 1 -- 172.21.15.107:0/1308661176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b40a7430 con 0x7f37b40ab1a0 2023-10-24T23:53:25.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37a9ffb640 1 -- 172.21.15.107:0/1308661176 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b00b6b30 con 0x7f37b40ab1a0 2023-10-24T23:53:25.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37bb65d640 1 -- 172.21.15.107:0/1308661176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b41f0b40 con 0x7f37b40ab1a0 2023-10-24T23:53:25.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37bb65d640 1 -- 172.21.15.107:0/1308661176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b41f0420 con 0x7f37b40ab1a0 2023-10-24T23:53:25.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37a9ffb640 1 -- 172.21.15.107:0/1308661176 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f37b00b2040 con 0x7f37b40ab1a0 2023-10-24T23:53:25.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37a9ffb640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3798065e60 0x7f3798068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37a9ffb640 1 -- 172.21.15.107:0/1308661176 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f37b011e100 con 0x7f37b40ab1a0 2023-10-24T23:53:25.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.650+0000 7f37b8bf8640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3798065e60 0x7f3798068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.654+0000 7f37b8bf8640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3798065e60 0x7f3798068320 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f37b406cd40 tx=0x7f37a4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.654+0000 7f37a9ffb640 1 -- 172.21.15.107:0/1308661176 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f37b41f0420 con 0x7f37b40ab1a0 2023-10-24T23:53:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 -- 172.21.15.107:0/1308661176 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37b41f0420 con 0x7f37b40ab1a0 2023-10-24T23:53:25.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37a9ffb640 1 -- 172.21.15.107:0/1308661176 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37b41f0420 con 0x7f37b40ab1a0 2023-10-24T23:53:25.806 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-24T23:53:25.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 -- 172.21.15.107:0/1308661176 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3798065e60 msgr2=0x7f3798068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3798065e60 0x7f3798068320 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f37b406cd40 tx=0x7f37a4008040 comp rx=0 tx=0).stop 2023-10-24T23:53:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 -- 172.21.15.107:0/1308661176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b40ab1a0 msgr2=0x7f37b40a9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b40ab1a0 0x7f37b40a9dd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f37b00ae040 tx=0x7f37b001ac10 comp rx=0 tx=0).stop 2023-10-24T23:53:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 -- 172.21.15.107:0/1308661176 shutdown_connections 2023-10-24T23:53:25.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37b41f1060 0x7f37b41eceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3798065e60 0x7f3798068320 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37b40bceb0 0x7f37b4158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 --2- 172.21.15.107:0/1308661176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37b40ab1a0 0x7f37b40a9dd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 -- 172.21.15.107:0/1308661176 >> 172.21.15.107:0/1308661176 conn(0x7f37b40c2860 msgr2=0x7f37b406eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 -- 172.21.15.107:0/1308661176 shutdown_connections 2023-10-24T23:53:25.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.802+0000 7f37bb65d640 1 -- 172.21.15.107:0/1308661176 wait complete. 2023-10-24T23:53:25.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-24T23:53:25.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.942+0000 7f59ce6e0640 1 Processor -- start 2023-10-24T23:53:25.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.942+0000 7f59ce6e0640 1 -- start start 2023-10-24T23:53:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59ce6e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f59c81585d0 0x7f59c81601d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59ce6e0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c80be970 con 0x7f59c80c1fb0 2023-10-24T23:53:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59ce6e0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c80bec50 con 0x7f59c80bf0c0 2023-10-24T23:53:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59ce6e0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c8155360 con 0x7f59c81585d0 2023-10-24T23:53:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59c7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f59c80c1fb0 0x7f59c80719d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38244/0 (socket says 172.21.15.107:38244) 2023-10-24T23:53:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59c7fff640 1 -- 172.21.15.107:0/365182170 learned_addr learned my addr 172.21.15.107:0/365182170 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59ccc7d640 1 --2- 172.21.15.107:0/365182170 >> v2:172.21.15.107:3300/0 conn(0x7f59c81585d0 0x7f59c81601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59ccc7d640 1 -- 172.21.15.107:0/365182170 >> v1:172.21.15.110:6789/0 conn(0x7f59c80bf0c0 legacy=0x7f59c8157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:25.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59ccc7d640 1 -- 172.21.15.107:0/365182170 >> v1:172.21.15.107:6789/0 conn(0x7f59c80c1fb0 legacy=0x7f59c80719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59ccc7d640 1 -- 172.21.15.107:0/365182170 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c8066760 con 0x7f59c81585d0 2023-10-24T23:53:25.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59ccc7d640 1 --2- 172.21.15.107:0/365182170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59c81585d0 0x7f59c81601d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009d90 tx=0x7f59c001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ceb080fc14c72d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59c6ffd640 1 -- 172.21.15.107:0/365182170 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c00a58c0 con 0x7f59c81585d0 2023-10-24T23:53:25.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59c6ffd640 1 -- 172.21.15.107:0/365182170 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c00a5a20 con 0x7f59c81585d0 2023-10-24T23:53:25.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.946+0000 7f59c6ffd640 1 -- 172.21.15.107:0/365182170 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c00a5db0 con 0x7f59c81585d0 2023-10-24T23:53:25.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- 172.21.15.107:0/365182170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59c81585d0 msgr2=0x7f59c81601d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 --2- 172.21.15.107:0/365182170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59c81585d0 0x7f59c81601d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009d90 tx=0x7f59c001c910 comp rx=0 tx=0).stop 2023-10-24T23:53:25.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- 172.21.15.107:0/365182170 shutdown_connections 2023-10-24T23:53:25.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 --2- 172.21.15.107:0/365182170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59c81585d0 0x7f59c81601d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- 172.21.15.107:0/365182170 >> 172.21.15.107:0/365182170 conn(0x7f59c8070eb0 msgr2=0x7f59c80712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:25.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- 172.21.15.107:0/365182170 shutdown_connections 2023-10-24T23:53:25.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- 172.21.15.107:0/365182170 wait complete. 2023-10-24T23:53:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 Processor -- start 2023-10-24T23:53:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- start start 2023-10-24T23:53:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c80bf0c0 0x7f59c80bd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59c80c1fb0 0x7f59c814aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59c81585d0 0x7f59c8146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c806d850 con 0x7f59c81585d0 2023-10-24T23:53:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c814a7a0 con 0x7f59c80c1fb0 2023-10-24T23:53:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c8151910 con 0x7f59c80bf0c0 2023-10-24T23:53:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ccc7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59c81585d0 0x7f59c8146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ccc7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59c81585d0 0x7f59c8146aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33284/0 (socket says 172.21.15.107:33284) 2023-10-24T23:53:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ccc7d640 1 -- 172.21.15.107:0/3006181216 learned_addr learned my addr 172.21.15.107:0/3006181216 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ccc7d640 1 -- 172.21.15.107:0/3006181216 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c80bf0c0 msgr2=0x7f59c80bd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ccc7d640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c80bf0c0 0x7f59c80bd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ccc7d640 1 -- 172.21.15.107:0/3006181216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59c80c1fb0 msgr2=0x7f59c814aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59c77fe640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59c80c1fb0 0x7f59c814aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ccc7d640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59c80c1fb0 0x7f59c814aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ccc7d640 1 -- 172.21.15.107:0/3006181216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c0009a70 con 0x7f59c81585d0 2023-10-24T23:53:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ccc7d640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59c81585d0 0x7f59c8146aa0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f59c00b0040 tx=0x7f59c00196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59c4ff9640 1 -- 172.21.15.107:0/3006181216 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c00ad9d0 con 0x7f59c81585d0 2023-10-24T23:53:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59c4ff9640 1 -- 172.21.15.107:0/3006181216 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c00adb30 con 0x7f59c81585d0 2023-10-24T23:53:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- 172.21.15.107:0/3006181216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c8147b20 con 0x7f59c81585d0 2023-10-24T23:53:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59c4ff9640 1 -- 172.21.15.107:0/3006181216 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c00b9450 con 0x7f59c81585d0 2023-10-24T23:53:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- 172.21.15.107:0/3006181216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c814a1e0 con 0x7f59c81585d0 2023-10-24T23:53:25.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.950+0000 7f59ce6e0640 1 -- 172.21.15.107:0/3006181216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c804e130 con 0x7f59c81585d0 2023-10-24T23:53:25.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.954+0000 7f59c4ff9640 1 -- 172.21.15.107:0/3006181216 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f59c00b95b0 con 0x7f59c81585d0 2023-10-24T23:53:25.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.954+0000 7f59c4ff9640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f59b4065e60 0x7f59b4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:25.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.954+0000 7f59c7fff640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f59b4065e60 0x7f59b4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:25.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.954+0000 7f59c4ff9640 1 -- 172.21.15.107:0/3006181216 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f59c011f160 con 0x7f59c81585d0 2023-10-24T23:53:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.954+0000 7f59c7fff640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f59b4065e60 0x7f59b4068320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f59c80be5e0 tx=0x7f59b8007a50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:25.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:25.958+0000 7f59c4ff9640 1 -- 172.21.15.107:0/3006181216 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f59c00ec510 con 0x7f59c81585d0 2023-10-24T23:53:26.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.098+0000 7f59ce6e0640 1 -- 172.21.15.107:0/3006181216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f59c8154a70 con 0x7f59c81585d0 2023-10-24T23:53:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.098+0000 7f59c4ff9640 1 -- 172.21.15.107:0/3006181216 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f59c00efb30 con 0x7f59c81585d0 2023-10-24T23:53:26.104 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-24T23:53:26.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 -- 172.21.15.107:0/3006181216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f59b4065e60 msgr2=0x7f59b4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f59b4065e60 0x7f59b4068320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f59c80be5e0 tx=0x7f59b8007a50 comp rx=0 tx=0).stop 2023-10-24T23:53:26.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 -- 172.21.15.107:0/3006181216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59c81585d0 msgr2=0x7f59c8146aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59c81585d0 0x7f59c8146aa0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f59c00b0040 tx=0x7f59c00196b0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 -- 172.21.15.107:0/3006181216 shutdown_connections 2023-10-24T23:53:26.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59c81585d0 0x7f59c8146aa0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59c80c1fb0 0x7f59c814aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f59b4065e60 0x7f59b4068320 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 --2- 172.21.15.107:0/3006181216 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59c80bf0c0 0x7f59c80bd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 -- 172.21.15.107:0/3006181216 >> 172.21.15.107:0/3006181216 conn(0x7f59c8070eb0 msgr2=0x7f59c80c7760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:26.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 -- 172.21.15.107:0/3006181216 shutdown_connections 2023-10-24T23:53:26.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.102+0000 7f59ce6e0640 1 -- 172.21.15.107:0/3006181216 wait complete. 2023-10-24T23:53:26.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-24T23:53:26.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd818da3640 1 Processor -- start 2023-10-24T23:53:26.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd818da3640 1 -- start start 2023-10-24T23:53:26.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd818da3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8140af9b0 0x7fd8140c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd818da3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd8140bd930 0x7fd81415f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd818da3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8140c8180 con 0x7fd81406b360 2023-10-24T23:53:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd818da3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8140c5870 con 0x7fd8140bd930 2023-10-24T23:53:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd818da3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd814155430 con 0x7fd8140af9b0 2023-10-24T23:53:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd812d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd8140bd930 0x7fd81415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd811d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd81406b360 0x7fd814069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38260/0 (socket says 172.21.15.107:38260) 2023-10-24T23:53:26.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd811d9b640 1 -- 172.21.15.107:0/647999132 learned_addr learned my addr 172.21.15.107:0/647999132 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd81259c640 1 --2- 172.21.15.107:0/647999132 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8140af9b0 0x7fd8140c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd812d9d640 1 -- 172.21.15.107:0/647999132 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8140af9b0 msgr2=0x7fd8140c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:26.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd812d9d640 1 --2- 172.21.15.107:0/647999132 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8140af9b0 0x7fd8140c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd812d9d640 1 -- 172.21.15.107:0/647999132 >> v1:172.21.15.107:6789/0 conn(0x7fd81406b360 legacy=0x7fd814069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd81159a640 1 -- 172.21.15.107:0/647999132 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417268722 0 0) 0x7fd8140c8180 con 0x7fd81406b360 2023-10-24T23:53:26.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd812d9d640 1 -- 172.21.15.107:0/647999132 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd814155710 con 0x7fd8140bd930 2023-10-24T23:53:26.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.242+0000 7fd81259c640 1 --2- 172.21.15.107:0/647999132 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8140af9b0 0x7fd8140c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:26.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd812d9d640 1 --2- 172.21.15.107:0/647999132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8140bd930 0x7fd81415f290 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009d90 tx=0x7fd80c01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c4814f82f95813 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:26.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd81159a640 1 -- 172.21.15.107:0/647999132 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c0a58e0 con 0x7fd8140bd930 2023-10-24T23:53:26.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd81159a640 1 -- 172.21.15.107:0/647999132 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80c0a5a40 con 0x7fd8140bd930 2023-10-24T23:53:26.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd81159a640 1 -- 172.21.15.107:0/647999132 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c0a5dd0 con 0x7fd8140bd930 2023-10-24T23:53:26.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 -- 172.21.15.107:0/647999132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8140bd930 msgr2=0x7fd81415f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 --2- 172.21.15.107:0/647999132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8140bd930 0x7fd81415f290 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009d90 tx=0x7fd80c01c890 comp rx=0 tx=0).stop 2023-10-24T23:53:26.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 -- 172.21.15.107:0/647999132 shutdown_connections 2023-10-24T23:53:26.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 --2- 172.21.15.107:0/647999132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8140bd930 0x7fd81415f290 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 --2- 172.21.15.107:0/647999132 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8140af9b0 0x7fd8140c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 -- 172.21.15.107:0/647999132 >> 172.21.15.107:0/647999132 conn(0x7fd814071b00 msgr2=0x7fd8140abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:26.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 -- 172.21.15.107:0/647999132 shutdown_connections 2023-10-24T23:53:26.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 -- 172.21.15.107:0/647999132 wait complete. 2023-10-24T23:53:26.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 Processor -- start 2023-10-24T23:53:26.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 -- start start 2023-10-24T23:53:26.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd81406b360 0x7fd814153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8140af9b0 0x7fd8140c1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8140bd930 0x7fd814147700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8140c2a70 con 0x7fd8140af9b0 2023-10-24T23:53:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8141f0770 con 0x7fd8140bd930 2023-10-24T23:53:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8141f0a10 con 0x7fd81406b360 2023-10-24T23:53:26.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd811d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8140af9b0 0x7fd8140c1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd811d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8140af9b0 0x7fd8140c1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33310/0 (socket says 172.21.15.107:33310) 2023-10-24T23:53:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd811d9b640 1 -- 172.21.15.107:0/2023750498 learned_addr learned my addr 172.21.15.107:0/2023750498 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd812d9d640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8140bd930 0x7fd814147700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd811d9b640 1 -- 172.21.15.107:0/2023750498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd81406b360 msgr2=0x7fd814153d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd811d9b640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd81406b360 0x7fd814153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd811d9b640 1 -- 172.21.15.107:0/2023750498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8140bd930 msgr2=0x7fd814147700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd811d9b640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8140bd930 0x7fd814147700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd811d9b640 1 -- 172.21.15.107:0/2023750498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c009a70 con 0x7fd8140af9b0 2023-10-24T23:53:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd811d9b640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8140af9b0 0x7fd8140c1350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd80400c4a0 tx=0x7fd80400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd802ffd640 1 -- 172.21.15.107:0/2023750498 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8040074e0 con 0x7fd8140af9b0 2023-10-24T23:53:26.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd802ffd640 1 -- 172.21.15.107:0/2023750498 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd804007640 con 0x7fd8140af9b0 2023-10-24T23:53:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 -- 172.21.15.107:0/2023750498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8141f0cf0 con 0x7fd8140af9b0 2023-10-24T23:53:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd802ffd640 1 -- 172.21.15.107:0/2023750498 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd804004a30 con 0x7fd8140af9b0 2023-10-24T23:53:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.246+0000 7fd818da3640 1 -- 172.21.15.107:0/2023750498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8141542a0 con 0x7fd8140af9b0 2023-10-24T23:53:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.250+0000 7fd818da3640 1 -- 172.21.15.107:0/2023750498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8141f1210 con 0x7fd8140af9b0 2023-10-24T23:53:26.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.250+0000 7fd802ffd640 1 -- 172.21.15.107:0/2023750498 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd804004b90 con 0x7fd8140af9b0 2023-10-24T23:53:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.250+0000 7fd802ffd640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd7f4065e60 0x7fd7f4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.250+0000 7fd802ffd640 1 -- 172.21.15.107:0/2023750498 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fd804083020 con 0x7fd8140af9b0 2023-10-24T23:53:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.250+0000 7fd81259c640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd7f4065e60 0x7fd7f4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.250+0000 7fd81259c640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd7f4065e60 0x7fd7f4068320 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd80800a8d0 tx=0x7fd808008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:26.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.254+0000 7fd802ffd640 1 -- 172.21.15.107:0/2023750498 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd8141f1210 con 0x7fd8140af9b0 2023-10-24T23:53:26.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.410+0000 7fd818da3640 1 -- 172.21.15.107:0/2023750498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd8141f1210 con 0x7fd8140af9b0 2023-10-24T23:53:26.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd802ffd640 1 -- 172.21.15.107:0/2023750498 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7fd804018020 con 0x7fd8140af9b0 2023-10-24T23:53:26.417 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:26.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 -- 172.21.15.107:0/2023750498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd7f4065e60 msgr2=0x7fd7f4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd7f4065e60 0x7fd7f4068320 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd80800a8d0 tx=0x7fd808008040 comp rx=0 tx=0).stop 2023-10-24T23:53:26.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 -- 172.21.15.107:0/2023750498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8140af9b0 msgr2=0x7fd8140c1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8140af9b0 0x7fd8140c1350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd80400c4a0 tx=0x7fd80400c960 comp rx=0 tx=0).stop 2023-10-24T23:53:26.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 -- 172.21.15.107:0/2023750498 shutdown_connections 2023-10-24T23:53:26.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8140bd930 0x7fd814147700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8140af9b0 0x7fd8140c1350 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd7f4065e60 0x7fd7f4068320 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 --2- 172.21.15.107:0/2023750498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd81406b360 0x7fd814153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 -- 172.21.15.107:0/2023750498 >> 172.21.15.107:0/2023750498 conn(0x7fd814071b00 msgr2=0x7fd8140c8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:26.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 -- 172.21.15.107:0/2023750498 shutdown_connections 2023-10-24T23:53:26.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.414+0000 7fd818da3640 1 -- 172.21.15.107:0/2023750498 wait complete. 2023-10-24T23:53:26.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-24T23:53:26.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb446419640 1 Processor -- start 2023-10-24T23:53:26.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb446419640 1 -- start start 2023-10-24T23:53:26.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb446419640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb4400a9300 0x7fb440072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb446419640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4400befa0 con 0x7fb4400aa9f0 2023-10-24T23:53:26.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb446419640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4400a9b80 con 0x7fb4400c1b30 2023-10-24T23:53:26.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb446419640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4400aa0b0 con 0x7fb4400a9300 2023-10-24T23:53:26.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb4449b6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb4400a9300 0x7fb440072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb4449b6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb4400a9300 0x7fb440072210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33316/0 (socket says 172.21.15.107:33316) 2023-10-24T23:53:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb4449b6640 1 -- 172.21.15.107:0/3749247063 learned_addr learned my addr 172.21.15.107:0/3749247063 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb4449b6640 1 -- 172.21.15.107:0/3749247063 >> v1:172.21.15.110:6789/0 conn(0x7fb4400c1b30 legacy=0x7fb4400aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb4449b6640 1 -- 172.21.15.107:0/3749247063 >> v1:172.21.15.107:6789/0 conn(0x7fb4400aa9f0 legacy=0x7fb440155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.550+0000 7fb4449b6640 1 -- 172.21.15.107:0/3749247063 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4400bf1f0 con 0x7fb4400a9300 2023-10-24T23:53:26.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb4449b6640 1 --2- 172.21.15.107:0/3749247063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4400a9300 0x7fb440072210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb438009e50 tx=0x7fb43801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d115a3cc848c08 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:26.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb43f7fe640 1 -- 172.21.15.107:0/3749247063 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4380a8430 con 0x7fb4400a9300 2023-10-24T23:53:26.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb43f7fe640 1 -- 172.21.15.107:0/3749247063 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4380a8590 con 0x7fb4400a9300 2023-10-24T23:53:26.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb43f7fe640 1 -- 172.21.15.107:0/3749247063 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4380a8920 con 0x7fb4400a9300 2023-10-24T23:53:26.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 -- 172.21.15.107:0/3749247063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4400a9300 msgr2=0x7fb440072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 --2- 172.21.15.107:0/3749247063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4400a9300 0x7fb440072210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb438009e50 tx=0x7fb43801c8f0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 -- 172.21.15.107:0/3749247063 shutdown_connections 2023-10-24T23:53:26.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 --2- 172.21.15.107:0/3749247063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4400a9300 0x7fb440072210 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 -- 172.21.15.107:0/3749247063 >> 172.21.15.107:0/3749247063 conn(0x7fb4400c2860 msgr2=0x7fb4400c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:26.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 -- 172.21.15.107:0/3749247063 shutdown_connections 2023-10-24T23:53:26.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 -- 172.21.15.107:0/3749247063 wait complete. 2023-10-24T23:53:26.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 Processor -- start 2023-10-24T23:53:26.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 -- start start 2023-10-24T23:53:26.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4400a9300 0x7fb4401ed780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4400c1b30 0x7fb4400706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4400b9910 0x7fb440070bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4401527c0 con 0x7fb4400b9910 2023-10-24T23:53:26.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb440152cf0 con 0x7fb4400a9300 2023-10-24T23:53:26.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb446419640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb440071250 con 0x7fb4400c1b30 2023-10-24T23:53:26.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb4449b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4400b9910 0x7fb440070bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb4449b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4400b9910 0x7fb440070bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33332/0 (socket says 172.21.15.107:33332) 2023-10-24T23:53:26.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb4449b6640 1 -- 172.21.15.107:0/3779745339 learned_addr learned my addr 172.21.15.107:0/3779745339 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:26.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb437fff640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4400c1b30 0x7fb4400706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb43ffff640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4400a9300 0x7fb4401ed780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb4449b6640 1 -- 172.21.15.107:0/3779745339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4400c1b30 msgr2=0x7fb4400706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb4449b6640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4400c1b30 0x7fb4400706b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb4449b6640 1 -- 172.21.15.107:0/3779745339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4400a9300 msgr2=0x7fb4401ed780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb4449b6640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4400a9300 0x7fb4401ed780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb4449b6640 1 -- 172.21.15.107:0/3779745339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb438009b30 con 0x7fb4400b9910 2023-10-24T23:53:26.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.554+0000 7fb4449b6640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4400b9910 0x7fb440070bf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb43801a520 tx=0x7fb438019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:26.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.558+0000 7fb43d7fa640 1 -- 172.21.15.107:0/3779745339 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4380a8430 con 0x7fb4400b9910 2023-10-24T23:53:26.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.558+0000 7fb43d7fa640 1 -- 172.21.15.107:0/3779745339 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4380bd670 con 0x7fb4400b9910 2023-10-24T23:53:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.558+0000 7fb446419640 1 -- 172.21.15.107:0/3779745339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb440152a10 con 0x7fb4400b9910 2023-10-24T23:53:26.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.558+0000 7fb43d7fa640 1 -- 172.21.15.107:0/3779745339 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4380bda20 con 0x7fb4400b9910 2023-10-24T23:53:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.558+0000 7fb446419640 1 -- 172.21.15.107:0/3779745339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44006ef80 con 0x7fb4400b9910 2023-10-24T23:53:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.558+0000 7fb446419640 1 -- 172.21.15.107:0/3779745339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44006f140 con 0x7fb4400b9910 2023-10-24T23:53:26.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.558+0000 7fb43d7fa640 1 -- 172.21.15.107:0/3779745339 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb4380b1450 con 0x7fb4400b9910 2023-10-24T23:53:26.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.558+0000 7fb43d7fa640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb420065e90 0x7fb420068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.558+0000 7fb43ffff640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb420065e90 0x7fb420068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.558+0000 7fb43d7fa640 1 -- 172.21.15.107:0/3779745339 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fb43811eaa0 con 0x7fb4400b9910 2023-10-24T23:53:26.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.558+0000 7fb43ffff640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb420065e90 0x7fb420068350 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb430000f70 tx=0x7fb430009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:26.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.562+0000 7fb43d7fa640 1 -- 172.21.15.107:0/3779745339 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb4380ebf70 con 0x7fb4400b9910 2023-10-24T23:53:26.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.706+0000 7fb446419640 1 -- 172.21.15.107:0/3779745339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7fb44004e130 con 0x7fb4400b9910 2023-10-24T23:53:26.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.706+0000 7fb43d7fa640 1 -- 172.21.15.107:0/3779745339 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7fb438009510 con 0x7fb4400b9910 2023-10-24T23:53:26.713 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 -- 172.21.15.107:0/3779745339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb420065e90 msgr2=0x7fb420068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb420065e90 0x7fb420068350 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb430000f70 tx=0x7fb430009040 comp rx=0 tx=0).stop 2023-10-24T23:53:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 -- 172.21.15.107:0/3779745339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4400b9910 msgr2=0x7fb440070bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4400b9910 0x7fb440070bf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb43801a520 tx=0x7fb438019690 comp rx=0 tx=0).stop 2023-10-24T23:53:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 -- 172.21.15.107:0/3779745339 shutdown_connections 2023-10-24T23:53:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4400b9910 0x7fb440070bf0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb420065e90 0x7fb420068350 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4400c1b30 0x7fb4400706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 --2- 172.21.15.107:0/3779745339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4400a9300 0x7fb4401ed780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 -- 172.21.15.107:0/3779745339 >> 172.21.15.107:0/3779745339 conn(0x7fb4400c2860 msgr2=0x7fb4400aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 -- 172.21.15.107:0/3779745339 shutdown_connections 2023-10-24T23:53:26.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.710+0000 7fb446419640 1 -- 172.21.15.107:0/3779745339 wait complete. 2023-10-24T23:53:26.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-10-24T23:53:26.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf7b853640 1 Processor -- start 2023-10-24T23:53:26.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf7b853640 1 -- start start 2023-10-24T23:53:26.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf7b853640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf740c7680 0x7fdf74156280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf7b853640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf741567c0 con 0x7fdf7414fd10 2023-10-24T23:53:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf7b853640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf740c71a0 con 0x7fdf74071f70 2023-10-24T23:53:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf7b853640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf740a9b60 con 0x7fdf740c7680 2023-10-24T23:53:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf79df0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf740c7680 0x7fdf74156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf795ef640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdf7414fd10 0x7fdf740a9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38276/0 (socket says 172.21.15.107:38276) 2023-10-24T23:53:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf795ef640 1 -- 172.21.15.107:0/1991571282 learned_addr learned my addr 172.21.15.107:0/1991571282 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:26.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf79df0640 1 -- 172.21.15.107:0/1991571282 >> v1:172.21.15.110:6789/0 conn(0x7fdf74071f70 legacy=0x7fdf74155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:26.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf79df0640 1 -- 172.21.15.107:0/1991571282 >> v1:172.21.15.107:6789/0 conn(0x7fdf7414fd10 legacy=0x7fdf740a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.846+0000 7fdf79df0640 1 -- 172.21.15.107:0/1991571282 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf741569f0 con 0x7fdf740c7680 2023-10-24T23:53:26.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf79df0640 1 --2- 172.21.15.107:0/1991571282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf740c7680 0x7fdf74156280 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdf700097c0 tx=0x7fdf7001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74f20fc5ac43c3e server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:26.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf6bfff640 1 -- 172.21.15.107:0/1991571282 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf700ab600 con 0x7fdf740c7680 2023-10-24T23:53:26.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf6bfff640 1 -- 172.21.15.107:0/1991571282 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf700a7050 con 0x7fdf740c7680 2023-10-24T23:53:26.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf6bfff640 1 -- 172.21.15.107:0/1991571282 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf700ab980 con 0x7fdf740c7680 2023-10-24T23:53:26.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf7b853640 1 -- 172.21.15.107:0/1991571282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf740c7680 msgr2=0x7fdf74156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf7b853640 1 --2- 172.21.15.107:0/1991571282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf740c7680 0x7fdf74156280 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdf700097c0 tx=0x7fdf7001c360 comp rx=0 tx=0).stop 2023-10-24T23:53:26.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf7b853640 1 -- 172.21.15.107:0/1991571282 shutdown_connections 2023-10-24T23:53:26.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf7b853640 1 --2- 172.21.15.107:0/1991571282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf740c7680 0x7fdf74156280 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf7b853640 1 -- 172.21.15.107:0/1991571282 >> 172.21.15.107:0/1991571282 conn(0x7fdf740c8020 msgr2=0x7fdf740c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:26.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf7b853640 1 -- 172.21.15.107:0/1991571282 shutdown_connections 2023-10-24T23:53:26.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf7b853640 1 -- 172.21.15.107:0/1991571282 wait complete. 2023-10-24T23:53:26.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf7b853640 1 Processor -- start 2023-10-24T23:53:26.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.850+0000 7fdf7b853640 1 -- start start 2023-10-24T23:53:26.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf7b853640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf74071f70 0x7fdf741de1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf7b853640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf740c7680 0x7fdf740ab0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf7b853640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf7414fd10 0x7fdf741e9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf7b853640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf741dd140 con 0x7fdf74071f70 2023-10-24T23:53:26.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf7b853640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf741dd420 con 0x7fdf740c7680 2023-10-24T23:53:26.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf7b853640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf741dd700 con 0x7fdf7414fd10 2023-10-24T23:53:26.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf795ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf74071f70 0x7fdf741de1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf795ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf74071f70 0x7fdf741de1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33350/0 (socket says 172.21.15.107:33350) 2023-10-24T23:53:26.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf795ef640 1 -- 172.21.15.107:0/3503642914 learned_addr learned my addr 172.21.15.107:0/3503642914 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:26.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf78dee640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf740c7680 0x7fdf740ab0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf79df0640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf7414fd10 0x7fdf741e9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf795ef640 1 -- 172.21.15.107:0/3503642914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf7414fd10 msgr2=0x7fdf741e9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf795ef640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf7414fd10 0x7fdf741e9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf795ef640 1 -- 172.21.15.107:0/3503642914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf740c7680 msgr2=0x7fdf740ab0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:26.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf795ef640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf740c7680 0x7fdf740ab0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:26.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf795ef640 1 -- 172.21.15.107:0/3503642914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf70009420 con 0x7fdf74071f70 2023-10-24T23:53:26.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf795ef640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf74071f70 0x7fdf741de1e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c00e570 tx=0x7fdf6c00ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:26.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf69ffb640 1 -- 172.21.15.107:0/3503642914 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c012870 con 0x7fdf74071f70 2023-10-24T23:53:26.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf69ffb640 1 -- 172.21.15.107:0/3503642914 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c0129d0 con 0x7fdf74071f70 2023-10-24T23:53:26.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf7b853640 1 -- 172.21.15.107:0/3503642914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf741dd9e0 con 0x7fdf74071f70 2023-10-24T23:53:26.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf69ffb640 1 -- 172.21.15.107:0/3503642914 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c012d60 con 0x7fdf74071f70 2023-10-24T23:53:26.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf7b853640 1 -- 172.21.15.107:0/3503642914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf740ab610 con 0x7fdf74071f70 2023-10-24T23:53:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf7b853640 1 -- 172.21.15.107:0/3503642914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf741ddf00 con 0x7fdf74071f70 2023-10-24T23:53:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf69ffb640 1 -- 172.21.15.107:0/3503642914 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdf6c009070 con 0x7fdf74071f70 2023-10-24T23:53:26.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf69ffb640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf58065e60 0x7fdf58068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:26.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf69ffb640 1 -- 172.21.15.107:0/3503642914 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdf6c086b20 con 0x7fdf74071f70 2023-10-24T23:53:26.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf78dee640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf58065e60 0x7fdf58068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:26.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.854+0000 7fdf78dee640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf58065e60 0x7fdf58068320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdf7414f3c0 tx=0x7fdf64009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:26.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:26.858+0000 7fdf69ffb640 1 -- 172.21.15.107:0/3503642914 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdf741ddf00 con 0x7fdf74071f70 2023-10-24T23:53:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.010+0000 7fdf7b853640 1 -- 172.21.15.107:0/3503642914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdf741ddf00 con 0x7fdf74071f70 2023-10-24T23:53:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf69ffb640 1 -- 172.21.15.107:0/3503642914 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7fdf741ddf00 con 0x7fdf74071f70 2023-10-24T23:53:27.017 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:27.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 -- 172.21.15.107:0/3503642914 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf58065e60 msgr2=0x7fdf58068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf58065e60 0x7fdf58068320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdf7414f3c0 tx=0x7fdf64009040 comp rx=0 tx=0).stop 2023-10-24T23:53:27.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 -- 172.21.15.107:0/3503642914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf74071f70 msgr2=0x7fdf741de1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf74071f70 0x7fdf741de1e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c00e570 tx=0x7fdf6c00ea30 comp rx=0 tx=0).stop 2023-10-24T23:53:27.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 -- 172.21.15.107:0/3503642914 shutdown_connections 2023-10-24T23:53:27.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf58065e60 0x7fdf58068320 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf7414fd10 0x7fdf741e9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf740c7680 0x7fdf740ab0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 --2- 172.21.15.107:0/3503642914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf74071f70 0x7fdf741de1e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 -- 172.21.15.107:0/3503642914 >> 172.21.15.107:0/3503642914 conn(0x7fdf740c8020 msgr2=0x7fdf7414e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:27.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 -- 172.21.15.107:0/3503642914 shutdown_connections 2023-10-24T23:53:27.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.014+0000 7fdf7b853640 1 -- 172.21.15.107:0/3503642914 wait complete. 2023-10-24T23:53:27.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-10-24T23:53:27.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb792af6640 1 Processor -- start 2023-10-24T23:53:27.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb792af6640 1 -- start start 2023-10-24T23:53:27.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb792af6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb78c0a50d0 0x7fb78c0aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb792af6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c0c1c30 0x7fb78c0bd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb792af6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c0a9cf0 con 0x7fb78c0bd900 2023-10-24T23:53:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb792af6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c0b97d0 con 0x7fb78c0a50d0 2023-10-24T23:53:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb792af6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0a8fd0 con 0x7fb78c0c1c30 2023-10-24T23:53:27.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb790892640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb78c0a50d0 0x7fb78c0aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb790892640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb78c0a50d0 0x7fb78c0aaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33366/0 (socket says 172.21.15.107:33366) 2023-10-24T23:53:27.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb790892640 1 -- 172.21.15.107:0/2308262038 learned_addr learned my addr 172.21.15.107:0/2308262038 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:27.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb78bfff640 1 --2- 172.21.15.107:0/2308262038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c0c1c30 0x7fb78c0bd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb790892640 1 -- 172.21.15.107:0/2308262038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c0c1c30 msgr2=0x7fb78c0bd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:27.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb790892640 1 --2- 172.21.15.107:0/2308262038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c0c1c30 0x7fb78c0bd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb790892640 1 -- 172.21.15.107:0/2308262038 >> v1:172.21.15.110:6789/0 conn(0x7fb78c0bd900 legacy=0x7fb78c0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:27.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb790892640 1 -- 172.21.15.107:0/2308262038 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c0a9fd0 con 0x7fb78c0a50d0 2023-10-24T23:53:27.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb790892640 1 --2- 172.21.15.107:0/2308262038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c0a50d0 0x7fb78c0aaab0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb780009870 tx=0x7fb78001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf4f325b73e2e1f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:27.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb78b7fe640 1 -- 172.21.15.107:0/2308262038 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7800ac410 con 0x7fb78c0a50d0 2023-10-24T23:53:27.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb78b7fe640 1 -- 172.21.15.107:0/2308262038 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7800ac570 con 0x7fb78c0a50d0 2023-10-24T23:53:27.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.154+0000 7fb78b7fe640 1 -- 172.21.15.107:0/2308262038 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7800ac900 con 0x7fb78c0a50d0 2023-10-24T23:53:27.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- 172.21.15.107:0/2308262038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c0a50d0 msgr2=0x7fb78c0aaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 --2- 172.21.15.107:0/2308262038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c0a50d0 0x7fb78c0aaab0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb780009870 tx=0x7fb78001c500 comp rx=0 tx=0).stop 2023-10-24T23:53:27.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- 172.21.15.107:0/2308262038 shutdown_connections 2023-10-24T23:53:27.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 --2- 172.21.15.107:0/2308262038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c0c1c30 0x7fb78c0bd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 --2- 172.21.15.107:0/2308262038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c0a50d0 0x7fb78c0aaab0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- 172.21.15.107:0/2308262038 >> 172.21.15.107:0/2308262038 conn(0x7fb78c0c2860 msgr2=0x7fb78c0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:27.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- 172.21.15.107:0/2308262038 shutdown_connections 2023-10-24T23:53:27.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- 172.21.15.107:0/2308262038 wait complete. 2023-10-24T23:53:27.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 Processor -- start 2023-10-24T23:53:27.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- start start 2023-10-24T23:53:27.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb78c0a50d0 0x7fb78c1f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c0bd900 0x7fb78c161b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c0c1c30 0x7fb78c1f45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0717f0 con 0x7fb78c0c1c30 2023-10-24T23:53:27.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c161700 con 0x7fb78c0a50d0 2023-10-24T23:53:27.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c06ed10 con 0x7fb78c0bd900 2023-10-24T23:53:27.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb791093640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c0c1c30 0x7fb78c1f45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb791093640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c0c1c30 0x7fb78c1f45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33370/0 (socket says 172.21.15.107:33370) 2023-10-24T23:53:27.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb791093640 1 -- 172.21.15.107:0/3054459173 learned_addr learned my addr 172.21.15.107:0/3054459173 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:27.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb78bfff640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c0bd900 0x7fb78c161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb790892640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb78c0a50d0 0x7fb78c1f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb791093640 1 -- 172.21.15.107:0/3054459173 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c0bd900 msgr2=0x7fb78c161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb791093640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c0bd900 0x7fb78c161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb791093640 1 -- 172.21.15.107:0/3054459173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb78c0a50d0 msgr2=0x7fb78c1f2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb791093640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb78c0a50d0 0x7fb78c1f2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb791093640 1 -- 172.21.15.107:0/3054459173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7800094d0 con 0x7fb78c0c1c30 2023-10-24T23:53:27.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb791093640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c0c1c30 0x7fb78c1f45c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb78400c780 tx=0x7fb78400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:27.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb7897fa640 1 -- 172.21.15.107:0/3054459173 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78400aa50 con 0x7fb78c0c1c30 2023-10-24T23:53:27.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb7897fa640 1 -- 172.21.15.107:0/3054459173 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78400abf0 con 0x7fb78c0c1c30 2023-10-24T23:53:27.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- 172.21.15.107:0/3054459173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c161390 con 0x7fb78c0c1c30 2023-10-24T23:53:27.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb7897fa640 1 -- 172.21.15.107:0/3054459173 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb78400ce30 con 0x7fb78c0c1c30 2023-10-24T23:53:27.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- 172.21.15.107:0/3054459173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c1f50b0 con 0x7fb78c0c1c30 2023-10-24T23:53:27.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.158+0000 7fb792af6640 1 -- 172.21.15.107:0/3054459173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c0bcf50 con 0x7fb78c0c1c30 2023-10-24T23:53:27.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.162+0000 7fb7897fa640 1 -- 172.21.15.107:0/3054459173 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb784012420 con 0x7fb78c0c1c30 2023-10-24T23:53:27.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.162+0000 7fb7897fa640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e60 0x7fb778068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.162+0000 7fb7897fa640 1 -- 172.21.15.107:0/3054459173 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fb784083cd0 con 0x7fb78c0c1c30 2023-10-24T23:53:27.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.162+0000 7fb790892640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e60 0x7fb778068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.162+0000 7fb790892640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e60 0x7fb778068320 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb7800ae000 tx=0x7fb780008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:27.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.166+0000 7fb7897fa640 1 -- 172.21.15.107:0/3054459173 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb784050fe0 con 0x7fb78c0c1c30 2023-10-24T23:53:27.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.318+0000 7fb792af6640 1 -- 172.21.15.107:0/3054459173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7fb78c04e130 con 0x7fb78c0c1c30 2023-10-24T23:53:27.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.318+0000 7fb7897fa640 1 -- 172.21.15.107:0/3054459173 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7fb784003eb0 con 0x7fb78c0c1c30 2023-10-24T23:53:27.323 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:27.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 -- 172.21.15.107:0/3054459173 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e60 msgr2=0x7fb778068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e60 0x7fb778068320 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb7800ae000 tx=0x7fb780008040 comp rx=0 tx=0).stop 2023-10-24T23:53:27.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 -- 172.21.15.107:0/3054459173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c0c1c30 msgr2=0x7fb78c1f45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c0c1c30 0x7fb78c1f45c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb78400c780 tx=0x7fb78400cc40 comp rx=0 tx=0).stop 2023-10-24T23:53:27.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 -- 172.21.15.107:0/3054459173 shutdown_connections 2023-10-24T23:53:27.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c0c1c30 0x7fb78c1f45c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e60 0x7fb778068320 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c0bd900 0x7fb78c161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 --2- 172.21.15.107:0/3054459173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb78c0a50d0 0x7fb78c1f2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 -- 172.21.15.107:0/3054459173 >> 172.21.15.107:0/3054459173 conn(0x7fb78c0c2860 msgr2=0x7fb78c15db60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:27.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 -- 172.21.15.107:0/3054459173 shutdown_connections 2023-10-24T23:53:27.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.322+0000 7fb792af6640 1 -- 172.21.15.107:0/3054459173 wait complete. 2023-10-24T23:53:27.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:27.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.450+0000 7ffa72337640 1 Processor -- start 2023-10-24T23:53:27.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa72337640 1 -- start start 2023-10-24T23:53:27.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa72337640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c06db10 0x7ffa6c158030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa72337640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffa6c0c1970 0x7ffa6c158570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa72337640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c06ce40 con 0x7ffa6c1520c0 2023-10-24T23:53:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa72337640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa6c06d090 con 0x7ffa6c0c1970 2023-10-24T23:53:27.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa72337640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c06d370 con 0x7ffa6c06db10 2023-10-24T23:53:27.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa63fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c06db10 0x7ffa6c158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa708d4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffa6c0c1970 0x7ffa6c158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa63fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c06db10 0x7ffa6c158030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:54776/0 (socket says 172.21.15.107:54776) 2023-10-24T23:53:27.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa63fff640 1 -- 172.21.15.107:0/1421148280 learned_addr learned my addr 172.21.15.107:0/1421148280 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:27.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa6bfff640 1 --1- 172.21.15.107:0/1421148280 >> v1:172.21.15.110:6789/0 conn(0x7ffa6c1520c0 0x7ffa6c069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:48048/0 (socket says 172.21.15.107:48048) 2023-10-24T23:53:27.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa63fff640 1 -- 172.21.15.107:0/1421148280 >> v2:172.21.15.110:3300/0 conn(0x7ffa6c0c1970 msgr2=0x7ffa6c158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:27.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa63fff640 1 --2- 172.21.15.107:0/1421148280 >> v2:172.21.15.110:3300/0 conn(0x7ffa6c0c1970 0x7ffa6c158570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa63fff640 1 -- 172.21.15.107:0/1421148280 >> v1:172.21.15.110:6789/0 conn(0x7ffa6c1520c0 legacy=0x7ffa6c069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.454+0000 7ffa63fff640 1 -- 172.21.15.107:0/1421148280 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c0c0cc0 con 0x7ffa6c06db10 2023-10-24T23:53:27.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa63fff640 1 --2- 172.21.15.107:0/1421148280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c06db10 0x7ffa6c158030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffa54009870 tx=0x7ffa5401ce50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd93823848495d7b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:27.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa6b7fe640 1 -- 172.21.15.107:0/1421148280 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa540ab610 con 0x7ffa6c06db10 2023-10-24T23:53:27.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa6b7fe640 1 -- 172.21.15.107:0/1421148280 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa540ab770 con 0x7ffa6c06db10 2023-10-24T23:53:27.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa6b7fe640 1 -- 172.21.15.107:0/1421148280 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa540abb00 con 0x7ffa6c06db10 2023-10-24T23:53:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 -- 172.21.15.107:0/1421148280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c06db10 msgr2=0x7ffa6c158030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 --2- 172.21.15.107:0/1421148280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c06db10 0x7ffa6c158030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffa54009870 tx=0x7ffa5401ce50 comp rx=0 tx=0).stop 2023-10-24T23:53:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 -- 172.21.15.107:0/1421148280 shutdown_connections 2023-10-24T23:53:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 --2- 172.21.15.107:0/1421148280 >> v2:172.21.15.110:3300/0 conn(0x7ffa6c0c1970 0x7ffa6c158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 --2- 172.21.15.107:0/1421148280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c06db10 0x7ffa6c158030 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 -- 172.21.15.107:0/1421148280 >> 172.21.15.107:0/1421148280 conn(0x7ffa6c0c7130 msgr2=0x7ffa6c0c7540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 -- 172.21.15.107:0/1421148280 shutdown_connections 2023-10-24T23:53:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 -- 172.21.15.107:0/1421148280 wait complete. 2023-10-24T23:53:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 Processor -- start 2023-10-24T23:53:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 -- start start 2023-10-24T23:53:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa6c0c1970 0x7ffa6c14ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6c1520c0 0x7ffa6c150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c148240 0x7ffa6c0c5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c14c1a0 con 0x7ffa6c1520c0 2023-10-24T23:53:27.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c14e6d0 con 0x7ffa6c0c1970 2023-10-24T23:53:27.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa72337640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c14f390 con 0x7ffa6c148240 2023-10-24T23:53:27.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa63fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6c1520c0 0x7ffa6c150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa63fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6c1520c0 0x7ffa6c150dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33376/0 (socket says 172.21.15.107:33376) 2023-10-24T23:53:27.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa63fff640 1 -- 172.21.15.107:0/945459907 learned_addr learned my addr 172.21.15.107:0/945459907 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:27.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa6bfff640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa6c0c1970 0x7ffa6c14ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa708d4640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c148240 0x7ffa6c0c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa63fff640 1 -- 172.21.15.107:0/945459907 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c148240 msgr2=0x7ffa6c0c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa63fff640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c148240 0x7ffa6c0c5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa63fff640 1 -- 172.21.15.107:0/945459907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa6c0c1970 msgr2=0x7ffa6c14ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa63fff640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa6c0c1970 0x7ffa6c14ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa63fff640 1 -- 172.21.15.107:0/945459907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa540094d0 con 0x7ffa6c1520c0 2023-10-24T23:53:27.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa708d4640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c148240 0x7ffa6c0c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:27.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.458+0000 7ffa63fff640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6c1520c0 0x7ffa6c150dd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffa5401cc90 tx=0x7ffa5401ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:27.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.462+0000 7ffa697fa640 1 -- 172.21.15.107:0/945459907 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa540be5f0 con 0x7ffa6c1520c0 2023-10-24T23:53:27.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.462+0000 7ffa697fa640 1 -- 172.21.15.107:0/945459907 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa540be750 con 0x7ffa6c1520c0 2023-10-24T23:53:27.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.462+0000 7ffa72337640 1 -- 172.21.15.107:0/945459907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6c14dda0 con 0x7ffa6c1520c0 2023-10-24T23:53:27.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.462+0000 7ffa697fa640 1 -- 172.21.15.107:0/945459907 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa540beb00 con 0x7ffa6c1520c0 2023-10-24T23:53:27.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.462+0000 7ffa72337640 1 -- 172.21.15.107:0/945459907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6c14e360 con 0x7ffa6c1520c0 2023-10-24T23:53:27.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.462+0000 7ffa72337640 1 -- 172.21.15.107:0/945459907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6c14e810 con 0x7ffa6c1520c0 2023-10-24T23:53:27.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.462+0000 7ffa697fa640 1 -- 172.21.15.107:0/945459907 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ffa540b43f0 con 0x7ffa6c1520c0 2023-10-24T23:53:27.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.462+0000 7ffa697fa640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa4c065e90 0x7ffa4c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.462+0000 7ffa6bfff640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa4c065e90 0x7ffa4c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.462+0000 7ffa697fa640 1 -- 172.21.15.107:0/945459907 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7ffa5411ea50 con 0x7ffa6c1520c0 2023-10-24T23:53:27.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.462+0000 7ffa6bfff640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa4c065e90 0x7ffa4c068350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c000f70 tx=0x7ffa5c007cf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:27.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.466+0000 7ffa697fa640 1 -- 172.21.15.107:0/945459907 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ffa6c14e810 con 0x7ffa6c1520c0 2023-10-24T23:53:27.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.650+0000 7ffa72337640 1 -- 172.21.15.107:0/945459907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa6c0c57e0 con 0x7ffa6c1520c0 2023-10-24T23:53:27.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa697fa640 1 -- 172.21.15.107:0/945459907 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7ffa540ef890 con 0x7ffa6c1520c0 2023-10-24T23:53:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:27.657 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:53:27.658 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:53:27.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 -- 172.21.15.107:0/945459907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa4c065e90 msgr2=0x7ffa4c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa4c065e90 0x7ffa4c068350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c000f70 tx=0x7ffa5c007cf0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 -- 172.21.15.107:0/945459907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6c1520c0 msgr2=0x7ffa6c150dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6c1520c0 0x7ffa6c150dd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ffa5401cc90 tx=0x7ffa5401ac30 comp rx=0 tx=0).stop 2023-10-24T23:53:27.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 -- 172.21.15.107:0/945459907 shutdown_connections 2023-10-24T23:53:27.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa4c065e90 0x7ffa4c068350 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa6c148240 0x7ffa6c0c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6c1520c0 0x7ffa6c150dd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 --2- 172.21.15.107:0/945459907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa6c0c1970 0x7ffa6c14ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 -- 172.21.15.107:0/945459907 >> 172.21.15.107:0/945459907 conn(0x7ffa6c0c7130 msgr2=0x7ffa6c0c2050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:27.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 -- 172.21.15.107:0/945459907 shutdown_connections 2023-10-24T23:53:27.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.654+0000 7ffa72337640 1 -- 172.21.15.107:0/945459907 wait complete. 2023-10-24T23:53:27.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:27.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.786+0000 7fddb2df3640 1 Processor -- start 2023-10-24T23:53:27.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.786+0000 7fddb2df3640 1 -- start start 2023-10-24T23:53:27.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb2df3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fddac0a9820 0x7fddac0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb2df3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fddac06ee40 0x7fddac0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb2df3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddac06f730 con 0x7fddac14fd40 2023-10-24T23:53:27.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb2df3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddac0aa750 con 0x7fddac06ee40 2023-10-24T23:53:27.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb2df3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddac156a40 con 0x7fddac0a9820 2023-10-24T23:53:27.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb1390640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fddac06ee40 0x7fddac0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb1390640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fddac06ee40 0x7fddac0aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33382/0 (socket says 172.21.15.107:33382) 2023-10-24T23:53:27.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb1390640 1 -- 172.21.15.107:0/3104326947 learned_addr learned my addr 172.21.15.107:0/3104326947 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:27.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddabfff640 1 --2- 172.21.15.107:0/3104326947 >> v2:172.21.15.110:3300/0 conn(0x7fddac0a9820 0x7fddac0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb1390640 1 -- 172.21.15.107:0/3104326947 >> v2:172.21.15.110:3300/0 conn(0x7fddac0a9820 msgr2=0x7fddac0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:27.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb1390640 1 --2- 172.21.15.107:0/3104326947 >> v2:172.21.15.110:3300/0 conn(0x7fddac0a9820 0x7fddac0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb1390640 1 -- 172.21.15.107:0/3104326947 >> v1:172.21.15.110:6789/0 conn(0x7fddac14fd40 legacy=0x7fddac0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:27.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb1390640 1 -- 172.21.15.107:0/3104326947 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac156710 con 0x7fddac06ee40 2023-10-24T23:53:27.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddb1390640 1 --2- 172.21.15.107:0/3104326947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac06ee40 0x7fddac0aa210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fddac06d2f0 tx=0x7fdda401c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e493d325b7e4178 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:27.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddab7fe640 1 -- 172.21.15.107:0/3104326947 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda40a56a0 con 0x7fddac06ee40 2023-10-24T23:53:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddab7fe640 1 -- 172.21.15.107:0/3104326947 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda40a5800 con 0x7fddac06ee40 2023-10-24T23:53:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.790+0000 7fddab7fe640 1 -- 172.21.15.107:0/3104326947 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda40a5b90 con 0x7fddac06ee40 2023-10-24T23:53:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- 172.21.15.107:0/3104326947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac06ee40 msgr2=0x7fddac0aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 --2- 172.21.15.107:0/3104326947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac06ee40 0x7fddac0aa210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fddac06d2f0 tx=0x7fdda401c840 comp rx=0 tx=0).stop 2023-10-24T23:53:27.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- 172.21.15.107:0/3104326947 shutdown_connections 2023-10-24T23:53:27.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 --2- 172.21.15.107:0/3104326947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac06ee40 0x7fddac0aa210 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 --2- 172.21.15.107:0/3104326947 >> v2:172.21.15.110:3300/0 conn(0x7fddac0a9820 0x7fddac0ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- 172.21.15.107:0/3104326947 >> 172.21.15.107:0/3104326947 conn(0x7fddac0c2950 msgr2=0x7fddac0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:27.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- 172.21.15.107:0/3104326947 shutdown_connections 2023-10-24T23:53:27.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- 172.21.15.107:0/3104326947 wait complete. 2023-10-24T23:53:27.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 Processor -- start 2023-10-24T23:53:27.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- start start 2023-10-24T23:53:27.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac06ee40 0x7fddac153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fddac14fd40 0x7fddac144db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddac147230 0x7fddac1e4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddac146ee0 con 0x7fddac06ee40 2023-10-24T23:53:27.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddac1452f0 con 0x7fddac147230 2023-10-24T23:53:27.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddac1e4490 con 0x7fddac14fd40 2023-10-24T23:53:27.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb0b8f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac06ee40 0x7fddac153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb0b8f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac06ee40 0x7fddac153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33392/0 (socket says 172.21.15.107:33392) 2023-10-24T23:53:27.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb0b8f640 1 -- 172.21.15.107:0/3597035741 learned_addr learned my addr 172.21.15.107:0/3597035741 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:27.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb1390640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddac147230 0x7fddac1e4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddabfff640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fddac14fd40 0x7fddac144db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb0b8f640 1 -- 172.21.15.107:0/3597035741 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fddac14fd40 msgr2=0x7fddac144db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb0b8f640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fddac14fd40 0x7fddac144db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb0b8f640 1 -- 172.21.15.107:0/3597035741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddac147230 msgr2=0x7fddac1e4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb0b8f640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddac147230 0x7fddac1e4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb0b8f640 1 -- 172.21.15.107:0/3597035741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda4009b30 con 0x7fddac06ee40 2023-10-24T23:53:27.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb0b8f640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac06ee40 0x7fddac153910 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00c450 tx=0x7fdd9c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:27.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fdda97fa640 1 -- 172.21.15.107:0/3597035741 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c011790 con 0x7fddac06ee40 2023-10-24T23:53:27.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fdda97fa640 1 -- 172.21.15.107:0/3597035741 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd9c011910 con 0x7fddac06ee40 2023-10-24T23:53:27.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- 172.21.15.107:0/3597035741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddac1e4a50 con 0x7fddac06ee40 2023-10-24T23:53:27.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fdda97fa640 1 -- 172.21.15.107:0/3597035741 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c011cc0 con 0x7fddac06ee40 2023-10-24T23:53:27.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- 172.21.15.107:0/3597035741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddac1ebc20 con 0x7fddac06ee40 2023-10-24T23:53:27.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.794+0000 7fddb2df3640 1 -- 172.21.15.107:0/3597035741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddac0c24a0 con 0x7fddac06ee40 2023-10-24T23:53:27.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.798+0000 7fdda97fa640 1 -- 172.21.15.107:0/3597035741 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdd9c01c460 con 0x7fddac06ee40 2023-10-24T23:53:27.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.798+0000 7fdda97fa640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdd98065e60 0x7fdd98068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:27.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.798+0000 7fdda97fa640 1 -- 172.21.15.107:0/3597035741 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdd9c0839e0 con 0x7fddac06ee40 2023-10-24T23:53:27.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.798+0000 7fddabfff640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdd98065e60 0x7fdd98068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:27.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.798+0000 7fddabfff640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdd98065e60 0x7fdd98068320 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fddac151570 tx=0x7fdda0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:27.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.802+0000 7fdda97fa640 1 -- 172.21.15.107:0/3597035741 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdd9c050d90 con 0x7fddac06ee40 2023-10-24T23:53:27.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.946+0000 7fddb2df3640 1 -- 172.21.15.107:0/3597035741 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddac070840 con 0x7fddac06ee40 2023-10-24T23:53:27.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.950+0000 7fdda97fa640 1 -- 172.21.15.107:0/3597035741 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21531 (secure 0 0 0) 0x7fdd9c0543b0 con 0x7fddac06ee40 2023-10-24T23:53:27.957 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:27.957 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:12.781093+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:27.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 -- 172.21.15.107:0/3597035741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdd98065e60 msgr2=0x7fdd98068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdd98065e60 0x7fdd98068320 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fddac151570 tx=0x7fdda0008040 comp rx=0 tx=0).stop 2023-10-24T23:53:27.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 -- 172.21.15.107:0/3597035741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac06ee40 msgr2=0x7fddac153910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:27.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac06ee40 0x7fddac153910 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00c450 tx=0x7fdd9c00c910 comp rx=0 tx=0).stop 2023-10-24T23:53:27.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 -- 172.21.15.107:0/3597035741 shutdown_connections 2023-10-24T23:53:27.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fddac147230 0x7fddac1e4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdd98065e60 0x7fdd98068320 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fddac14fd40 0x7fddac144db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 --2- 172.21.15.107:0/3597035741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddac06ee40 0x7fddac153910 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:27.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 -- 172.21.15.107:0/3597035741 >> 172.21.15.107:0/3597035741 conn(0x7fddac0c2950 msgr2=0x7fddac0c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:27.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 -- 172.21.15.107:0/3597035741 shutdown_connections 2023-10-24T23:53:27.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:27.954+0000 7fddb2df3640 1 -- 172.21.15.107:0/3597035741 wait complete. 2023-10-24T23:53:27.974 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-10-24T23:53:09.786122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-10-24T23:53:10.766645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-24T23:53:27.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-24T23:53:28.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f88583d0640 1 Processor -- start 2023-10-24T23:53:28.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f88583d0640 1 -- start start 2023-10-24T23:53:28.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f88583d0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88500c2000 0x7f88500ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f88583d0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88500c4030 0x7f885015f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f88583d0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8850071de0 con 0x7f88500adda0 2023-10-24T23:53:28.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f88583d0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88500bd250 con 0x7f88500c4030 2023-10-24T23:53:28.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f88583d0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88500bcf00 con 0x7f88500c2000 2023-10-24T23:53:28.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f885696d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f88500c4030 0x7f885015f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f885596b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f88500adda0 0x7f88500c3920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38282/0 (socket says 172.21.15.107:38282) 2023-10-24T23:53:28.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f885596b640 1 -- 172.21.15.107:0/3171344888 learned_addr learned my addr 172.21.15.107:0/3171344888 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:28.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f885616c640 1 --2- 172.21.15.107:0/3171344888 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88500c2000 0x7f88500ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f885696d640 1 -- 172.21.15.107:0/3171344888 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88500c2000 msgr2=0x7f88500ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:28.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f885696d640 1 --2- 172.21.15.107:0/3171344888 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88500c2000 0x7f88500ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f885696d640 1 -- 172.21.15.107:0/3171344888 >> v1:172.21.15.107:6789/0 conn(0x7f88500adda0 legacy=0x7f88500c3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f885696d640 1 -- 172.21.15.107:0/3171344888 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885006a740 con 0x7f88500c4030 2023-10-24T23:53:28.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.090+0000 7f885696d640 1 --2- 172.21.15.107:0/3171344888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88500c4030 0x7f885015f080 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8848009d90 tx=0x7f884801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59b218d410590d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885516a640 1 -- 172.21.15.107:0/3171344888 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88480a59e0 con 0x7f88500c4030 2023-10-24T23:53:28.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885516a640 1 -- 172.21.15.107:0/3171344888 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88480a5b40 con 0x7f88500c4030 2023-10-24T23:53:28.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885516a640 1 -- 172.21.15.107:0/3171344888 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88480ad440 con 0x7f88500c4030 2023-10-24T23:53:28.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 -- 172.21.15.107:0/3171344888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88500c4030 msgr2=0x7f885015f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 --2- 172.21.15.107:0/3171344888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88500c4030 0x7f885015f080 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8848009d90 tx=0x7f884801c920 comp rx=0 tx=0).stop 2023-10-24T23:53:28.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 -- 172.21.15.107:0/3171344888 shutdown_connections 2023-10-24T23:53:28.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 --2- 172.21.15.107:0/3171344888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88500c4030 0x7f885015f080 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 --2- 172.21.15.107:0/3171344888 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88500c2000 0x7f88500ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 -- 172.21.15.107:0/3171344888 >> 172.21.15.107:0/3171344888 conn(0x7f8850070ef0 msgr2=0x7f8850071300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:28.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 -- 172.21.15.107:0/3171344888 shutdown_connections 2023-10-24T23:53:28.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 -- 172.21.15.107:0/3171344888 wait complete. 2023-10-24T23:53:28.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 Processor -- start 2023-10-24T23:53:28.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 -- start start 2023-10-24T23:53:28.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88500adda0 0x7f88501f0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88500c2000 0x7f885013d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88500c4030 0x7f885013de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88501409f0 con 0x7f88500c4030 2023-10-24T23:53:28.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8850158b00 con 0x7f88500c2000 2023-10-24T23:53:28.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88500c3860 con 0x7f88500adda0 2023-10-24T23:53:28.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885696d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88500c4030 0x7f885013de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885696d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88500c4030 0x7f885013de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33408/0 (socket says 172.21.15.107:33408) 2023-10-24T23:53:28.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885696d640 1 -- 172.21.15.107:0/666443756 learned_addr learned my addr 172.21.15.107:0/666443756 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:28.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885616c640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88500adda0 0x7f88501f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885696d640 1 -- 172.21.15.107:0/666443756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88500adda0 msgr2=0x7f88501f0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885696d640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88500adda0 0x7f88501f0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885596b640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88500c2000 0x7f885013d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885696d640 1 -- 172.21.15.107:0/666443756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88500c2000 msgr2=0x7f885013d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885696d640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88500c2000 0x7f885013d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885696d640 1 -- 172.21.15.107:0/666443756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8848009a70 con 0x7f88500c4030 2023-10-24T23:53:28.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885616c640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88500adda0 0x7f88501f0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:28.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f885696d640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88500c4030 0x7f885013de60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f88480b0040 tx=0x7f88480196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f8846ffd640 1 -- 172.21.15.107:0/666443756 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88480ade20 con 0x7f88500c4030 2023-10-24T23:53:28.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f8846ffd640 1 -- 172.21.15.107:0/666443756 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8848009450 con 0x7f88500c4030 2023-10-24T23:53:28.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 -- 172.21.15.107:0/666443756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8850140c00 con 0x7f88500c4030 2023-10-24T23:53:28.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f8846ffd640 1 -- 172.21.15.107:0/666443756 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88480b9690 con 0x7f88500c4030 2023-10-24T23:53:28.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.094+0000 7f88583d0640 1 -- 172.21.15.107:0/666443756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88501f0b60 con 0x7f88500c4030 2023-10-24T23:53:28.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.098+0000 7f88583d0640 1 -- 172.21.15.107:0/666443756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88501f0d80 con 0x7f88500c4030 2023-10-24T23:53:28.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.098+0000 7f8846ffd640 1 -- 172.21.15.107:0/666443756 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f88480039a0 con 0x7f88500c4030 2023-10-24T23:53:28.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.098+0000 7f8846ffd640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8834065e90 0x7f8834068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.098+0000 7f8846ffd640 1 -- 172.21.15.107:0/666443756 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f884811eef0 con 0x7f88500c4030 2023-10-24T23:53:28.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.098+0000 7f885616c640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8834065e90 0x7f8834068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.098+0000 7f885616c640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8834065e90 0x7f8834068350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f884c0097a0 tx=0x7f884c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.102+0000 7f8846ffd640 1 -- 172.21.15.107:0/666443756 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f88480ec2a0 con 0x7f88500c4030 2023-10-24T23:53:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.250+0000 7f88583d0640 1 -- 172.21.15.107:0/666443756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8850073c30 con 0x7f88500c4030 2023-10-24T23:53:28.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.250+0000 7f8846ffd640 1 -- 172.21.15.107:0/666443756 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f88480b20a0 con 0x7f88500c4030 2023-10-24T23:53:28.256 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-24T23:53:28.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 -- 172.21.15.107:0/666443756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8834065e90 msgr2=0x7f8834068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8834065e90 0x7f8834068350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f884c0097a0 tx=0x7f884c008040 comp rx=0 tx=0).stop 2023-10-24T23:53:28.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 -- 172.21.15.107:0/666443756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88500c4030 msgr2=0x7f885013de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88500c4030 0x7f885013de60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f88480b0040 tx=0x7f88480196c0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 -- 172.21.15.107:0/666443756 shutdown_connections 2023-10-24T23:53:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88500c4030 0x7f885013de60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88500c2000 0x7f885013d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8834065e90 0x7f8834068350 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 --2- 172.21.15.107:0/666443756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f88500adda0 0x7f88501f0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 -- 172.21.15.107:0/666443756 >> 172.21.15.107:0/666443756 conn(0x7f8850070ef0 msgr2=0x7f88501477b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 -- 172.21.15.107:0/666443756 shutdown_connections 2023-10-24T23:53:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.254+0000 7f88583d0640 1 -- 172.21.15.107:0/666443756 wait complete. 2023-10-24T23:53:28.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-24T23:53:28.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82f2233640 1 Processor -- start 2023-10-24T23:53:28.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82f2233640 1 -- start start 2023-10-24T23:53:28.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82f2233640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82ec0c1b30 0x7f82ec072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82f2233640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0aa9f0 0x7f82ec155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82f2233640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ec0befa0 con 0x7f82ec0a9300 2023-10-24T23:53:28.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82f2233640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ec0a9b80 con 0x7f82ec0c1b30 2023-10-24T23:53:28.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82f2233640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0aa0b0 con 0x7f82ec0aa9f0 2023-10-24T23:53:28.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82eb7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82ec0c1b30 0x7f82ec072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82ebfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0aa9f0 0x7f82ec155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82eaffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f82ec0a9300 0x7f82ec0aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:48062/0 (socket says 172.21.15.107:48062) 2023-10-24T23:53:28.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82ebfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0aa9f0 0x7f82ec155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:54814/0 (socket says 172.21.15.107:54814) 2023-10-24T23:53:28.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82eaffd640 1 -- 172.21.15.107:0/2920042193 learned_addr learned my addr 172.21.15.107:0/2920042193 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:28.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82eb7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82ec0c1b30 0x7f82ec072210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:49498/0 (socket says 172.21.15.107:49498) 2023-10-24T23:53:28.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82eb7fe640 1 -- 172.21.15.107:0/2920042193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0aa9f0 msgr2=0x7f82ec155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82eb7fe640 1 --2- 172.21.15.107:0/2920042193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0aa9f0 0x7f82ec155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82eb7fe640 1 -- 172.21.15.107:0/2920042193 >> v1:172.21.15.110:6789/0 conn(0x7f82ec0a9300 legacy=0x7f82ec0aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.382+0000 7f82eb7fe640 1 -- 172.21.15.107:0/2920042193 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec0bf1f0 con 0x7f82ec0c1b30 2023-10-24T23:53:28.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82eb7fe640 1 --2- 172.21.15.107:0/2920042193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec0c1b30 0x7f82ec072210 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0097c0 tx=0x7f82dc01c0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bf98c53e8f23cb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82ea7fc640 1 -- 172.21.15.107:0/2920042193 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0a4d40 con 0x7f82ec0c1b30 2023-10-24T23:53:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82ea7fc640 1 -- 172.21.15.107:0/2920042193 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82dc0ac440 con 0x7f82ec0c1b30 2023-10-24T23:53:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82ea7fc640 1 -- 172.21.15.107:0/2920042193 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0a4ea0 con 0x7f82ec0c1b30 2023-10-24T23:53:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82f2233640 1 -- 172.21.15.107:0/2920042193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec0c1b30 msgr2=0x7f82ec072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82f2233640 1 --2- 172.21.15.107:0/2920042193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec0c1b30 0x7f82ec072210 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0097c0 tx=0x7f82dc01c0a0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82f2233640 1 -- 172.21.15.107:0/2920042193 shutdown_connections 2023-10-24T23:53:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82f2233640 1 --2- 172.21.15.107:0/2920042193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0aa9f0 0x7f82ec155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82f2233640 1 --2- 172.21.15.107:0/2920042193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec0c1b30 0x7f82ec072210 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82f2233640 1 -- 172.21.15.107:0/2920042193 >> 172.21.15.107:0/2920042193 conn(0x7f82ec0c2860 msgr2=0x7f82ec0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:28.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82f2233640 1 -- 172.21.15.107:0/2920042193 shutdown_connections 2023-10-24T23:53:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82f2233640 1 -- 172.21.15.107:0/2920042193 wait complete. 2023-10-24T23:53:28.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82f2233640 1 Processor -- start 2023-10-24T23:53:28.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.386+0000 7f82f2233640 1 -- start start 2023-10-24T23:53:28.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82f2233640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0a9300 0x7f82ec0bbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82f2233640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec0c1b30 0x7f82ec1542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82f2233640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec1ec120 0x7f82ec1e90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82f2233640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec0bac00 con 0x7f82ec0c1b30 2023-10-24T23:53:28.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82f2233640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec1ec620 con 0x7f82ec1ec120 2023-10-24T23:53:28.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82f2233640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec1ec900 con 0x7f82ec0a9300 2023-10-24T23:53:28.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec0c1b30 0x7f82ec1542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec0c1b30 0x7f82ec1542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33424/0 (socket says 172.21.15.107:33424) 2023-10-24T23:53:28.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eaffd640 1 -- 172.21.15.107:0/3813113084 learned_addr learned my addr 172.21.15.107:0/3813113084 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82ebfff640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec1ec120 0x7f82ec1e90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eb7fe640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0a9300 0x7f82ec0bbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eaffd640 1 -- 172.21.15.107:0/3813113084 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0a9300 msgr2=0x7f82ec0bbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eaffd640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0a9300 0x7f82ec0bbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eaffd640 1 -- 172.21.15.107:0/3813113084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec1ec120 msgr2=0x7f82ec1e90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eaffd640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec1ec120 0x7f82ec1e90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eaffd640 1 -- 172.21.15.107:0/3813113084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc009420 con 0x7f82ec0c1b30 2023-10-24T23:53:28.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eb7fe640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0a9300 0x7f82ec0bbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:28.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eaffd640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec0c1b30 0x7f82ec1542e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c430 tx=0x7f82e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82d3fff640 1 -- 172.21.15.107:0/3813113084 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0005cf0 con 0x7f82ec0c1b30 2023-10-24T23:53:28.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82d3fff640 1 -- 172.21.15.107:0/3813113084 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e0005e50 con 0x7f82ec0c1b30 2023-10-24T23:53:28.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82f2233640 1 -- 172.21.15.107:0/3813113084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec1ecbe0 con 0x7f82ec0c1b30 2023-10-24T23:53:28.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82d3fff640 1 -- 172.21.15.107:0/3813113084 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00072f0 con 0x7f82ec0c1b30 2023-10-24T23:53:28.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82f2233640 1 -- 172.21.15.107:0/3813113084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec06ec50 con 0x7f82ec0c1b30 2023-10-24T23:53:28.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82f2233640 1 -- 172.21.15.107:0/3813113084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec1e8dd0 con 0x7f82ec0c1b30 2023-10-24T23:53:28.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82d3fff640 1 -- 172.21.15.107:0/3813113084 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f82e0007450 con 0x7f82ec0c1b30 2023-10-24T23:53:28.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82d3fff640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f82d8065e90 0x7f82d8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82d3fff640 1 -- 172.21.15.107:0/3813113084 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f82e0083080 con 0x7f82ec0c1b30 2023-10-24T23:53:28.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eb7fe640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f82d8065e90 0x7f82d8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.390+0000 7f82eb7fe640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f82d8065e90 0x7f82d8068350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0ae000 tx=0x7f82dc0a6070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.394+0000 7f82d3fff640 1 -- 172.21.15.107:0/3813113084 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f82ec1e8dd0 con 0x7f82ec0c1b30 2023-10-24T23:53:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 -- 172.21.15.107:0/3813113084 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f82ec1e8dd0 con 0x7f82ec0c1b30 2023-10-24T23:53:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82d3fff640 1 -- 172.21.15.107:0/3813113084 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f82e0050450 con 0x7f82ec0c1b30 2023-10-24T23:53:28.547 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-24T23:53:28.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 -- 172.21.15.107:0/3813113084 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f82d8065e90 msgr2=0x7f82d8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f82d8065e90 0x7f82d8068350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0ae000 tx=0x7f82dc0a6070 comp rx=0 tx=0).stop 2023-10-24T23:53:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 -- 172.21.15.107:0/3813113084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec0c1b30 msgr2=0x7f82ec1542e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec0c1b30 0x7f82ec1542e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c430 tx=0x7f82e000c8f0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 -- 172.21.15.107:0/3813113084 shutdown_connections 2023-10-24T23:53:28.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82ec1ec120 0x7f82ec1e90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82ec0c1b30 0x7f82ec1542e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f82d8065e90 0x7f82d8068350 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 --2- 172.21.15.107:0/3813113084 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82ec0a9300 0x7f82ec0bbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 -- 172.21.15.107:0/3813113084 >> 172.21.15.107:0/3813113084 conn(0x7f82ec0c2860 msgr2=0x7f82ec0aade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:28.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 -- 172.21.15.107:0/3813113084 shutdown_connections 2023-10-24T23:53:28.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.542+0000 7f82f2233640 1 -- 172.21.15.107:0/3813113084 wait complete. 2023-10-24T23:53:28.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-24T23:53:28.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.670+0000 7f0587abc640 1 Processor -- start 2023-10-24T23:53:28.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.670+0000 7f0587abc640 1 -- start start 2023-10-24T23:53:28.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0587abc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0580072720 0x7f05800c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0587abc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05800bceb0 0x7f058015c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0587abc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05800c1ce0 con 0x7f05800ab1a0 2023-10-24T23:53:28.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0587abc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05800be1d0 con 0x7f0580072720 2023-10-24T23:53:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0587abc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05800c0240 con 0x7f05800bceb0 2023-10-24T23:53:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0585057640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0580072720 0x7f05800c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0585858640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f05800ab1a0 0x7f05800bfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:48072/0 (socket says 172.21.15.107:48072) 2023-10-24T23:53:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0585858640 1 -- 172.21.15.107:0/3086474935 learned_addr learned my addr 172.21.15.107:0/3086474935 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:28.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0586059640 1 --2- 172.21.15.107:0/3086474935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05800bceb0 0x7f058015c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0585057640 1 -- 172.21.15.107:0/3086474935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05800bceb0 msgr2=0x7f058015c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:28.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0585057640 1 --2- 172.21.15.107:0/3086474935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05800bceb0 0x7f058015c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0585057640 1 -- 172.21.15.107:0/3086474935 >> v1:172.21.15.110:6789/0 conn(0x7f05800ab1a0 legacy=0x7f05800bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0585057640 1 -- 172.21.15.107:0/3086474935 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05800c0880 con 0x7f0580072720 2023-10-24T23:53:28.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.674+0000 7f0585057640 1 --2- 172.21.15.107:0/3086474935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0580072720 0x7f05800c0d00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f05700097c0 tx=0x7f057001c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce3fa0eb2dc028fa server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0584856640 1 -- 172.21.15.107:0/3086474935 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05700ab600 con 0x7f0580072720 2023-10-24T23:53:28.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0584856640 1 -- 172.21.15.107:0/3086474935 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05700a7050 con 0x7f0580072720 2023-10-24T23:53:28.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0584856640 1 -- 172.21.15.107:0/3086474935 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05700ab980 con 0x7f0580072720 2023-10-24T23:53:28.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 -- 172.21.15.107:0/3086474935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0580072720 msgr2=0x7f05800c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 --2- 172.21.15.107:0/3086474935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0580072720 0x7f05800c0d00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f05700097c0 tx=0x7f057001c330 comp rx=0 tx=0).stop 2023-10-24T23:53:28.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 -- 172.21.15.107:0/3086474935 shutdown_connections 2023-10-24T23:53:28.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 --2- 172.21.15.107:0/3086474935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05800bceb0 0x7f058015c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 --2- 172.21.15.107:0/3086474935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0580072720 0x7f05800c0d00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 -- 172.21.15.107:0/3086474935 >> 172.21.15.107:0/3086474935 conn(0x7f05800c2860 msgr2=0x7f05800c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:28.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 -- 172.21.15.107:0/3086474935 shutdown_connections 2023-10-24T23:53:28.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 -- 172.21.15.107:0/3086474935 wait complete. 2023-10-24T23:53:28.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 Processor -- start 2023-10-24T23:53:28.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 -- start start 2023-10-24T23:53:28.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05800ab1a0 0x7f0580154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05800bceb0 0x7f05801ee250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580071790 0x7f058006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0580155370 con 0x7f0580071790 2023-10-24T23:53:28.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058014caa0 con 0x7f05800ab1a0 2023-10-24T23:53:28.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058014c7c0 con 0x7f05800bceb0 2023-10-24T23:53:28.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0586059640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580071790 0x7f058006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0586059640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580071790 0x7f058006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33430/0 (socket says 172.21.15.107:33430) 2023-10-24T23:53:28.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0586059640 1 -- 172.21.15.107:0/2997695390 learned_addr learned my addr 172.21.15.107:0/2997695390 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:28.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0585057640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05800bceb0 0x7f05801ee250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0585858640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05800ab1a0 0x7f0580154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0586059640 1 -- 172.21.15.107:0/2997695390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05800bceb0 msgr2=0x7f05801ee250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0586059640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05800bceb0 0x7f05801ee250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0586059640 1 -- 172.21.15.107:0/2997695390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05800ab1a0 msgr2=0x7f0580154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0586059640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05800ab1a0 0x7f0580154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0586059640 1 -- 172.21.15.107:0/2997695390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0570009420 con 0x7f0580071790 2023-10-24T23:53:28.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0585057640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05800bceb0 0x7f05801ee250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:28.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0586059640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580071790 0x7f058006fb80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f057800efc0 tx=0x7f057800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f05767fc640 1 -- 172.21.15.107:0/2997695390 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0578009ba0 con 0x7f0580071790 2023-10-24T23:53:28.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f05767fc640 1 -- 172.21.15.107:0/2997695390 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057800f040 con 0x7f0580071790 2023-10-24T23:53:28.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 -- 172.21.15.107:0/2997695390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05801eda50 con 0x7f0580071790 2023-10-24T23:53:28.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f05767fc640 1 -- 172.21.15.107:0/2997695390 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05780146c0 con 0x7f0580071790 2023-10-24T23:53:28.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.678+0000 7f0587abc640 1 -- 172.21.15.107:0/2997695390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0580154c40 con 0x7f0580071790 2023-10-24T23:53:28.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.682+0000 7f0587abc640 1 -- 172.21.15.107:0/2997695390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05800beff0 con 0x7f0580071790 2023-10-24T23:53:28.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.682+0000 7f05767fc640 1 -- 172.21.15.107:0/2997695390 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0578014820 con 0x7f0580071790 2023-10-24T23:53:28.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.682+0000 7f05767fc640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e60 0x7f0564068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.682+0000 7f0585858640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e60 0x7f0564068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.682+0000 7f05767fc640 1 -- 172.21.15.107:0/2997695390 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f05780845e0 con 0x7f0580071790 2023-10-24T23:53:28.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.682+0000 7f0585858640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e60 0x7f0564068320 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0580152e40 tx=0x7f057c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.686+0000 7f05767fc640 1 -- 172.21.15.107:0/2997695390 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0578051990 con 0x7f0580071790 2023-10-24T23:53:28.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.834+0000 7f0587abc640 1 -- 172.21.15.107:0/2997695390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f05801edf00 con 0x7f0580071790 2023-10-24T23:53:28.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.834+0000 7f05767fc640 1 -- 172.21.15.107:0/2997695390 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7f0578054fb0 con 0x7f0580071790 2023-10-24T23:53:28.842 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:28.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 -- 172.21.15.107:0/2997695390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e60 msgr2=0x7f0564068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e60 0x7f0564068320 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0580152e40 tx=0x7f057c009040 comp rx=0 tx=0).stop 2023-10-24T23:53:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 -- 172.21.15.107:0/2997695390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580071790 msgr2=0x7f058006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580071790 0x7f058006fb80 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f057800efc0 tx=0x7f057800c490 comp rx=0 tx=0).stop 2023-10-24T23:53:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 -- 172.21.15.107:0/2997695390 shutdown_connections 2023-10-24T23:53:28.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0580071790 0x7f058006fb80 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e60 0x7f0564068320 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05800bceb0 0x7f05801ee250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 --2- 172.21.15.107:0/2997695390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05800ab1a0 0x7f0580154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 -- 172.21.15.107:0/2997695390 >> 172.21.15.107:0/2997695390 conn(0x7f05800c2860 msgr2=0x7f058015c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:28.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 -- 172.21.15.107:0/2997695390 shutdown_connections 2023-10-24T23:53:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.838+0000 7f0587abc640 1 -- 172.21.15.107:0/2997695390 wait complete. 2023-10-24T23:53:28.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-24T23:53:28.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.962+0000 7f9fe2057640 1 Processor -- start 2023-10-24T23:53:28.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fe2057640 1 -- start start 2023-10-24T23:53:28.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fe2057640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9fdc0aa9f0 0x7f9fdc072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fe2057640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9fdc0a9300 0x7f9fdc155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fe2057640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fdc0befa0 con 0x7f9fdc0c1b30 2023-10-24T23:53:28.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fe2057640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fdc0a9b80 con 0x7f9fdc0a9300 2023-10-24T23:53:28.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fe2057640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc0aa0b0 con 0x7f9fdc0aa9f0 2023-10-24T23:53:28.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fdbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9fdc0a9300 0x7f9fdc155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fdbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9fdc0a9300 0x7f9fdc155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33438/0 (socket says 172.21.15.107:33438) 2023-10-24T23:53:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fdb7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9fdc0c1b30 0x7f9fdc0aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:48076/0 (socket says 172.21.15.107:48076) 2023-10-24T23:53:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fdbfff640 1 -- 172.21.15.107:0/1953254203 learned_addr learned my addr 172.21.15.107:0/1953254203 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fdaffd640 1 --2- 172.21.15.107:0/1953254203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9fdc0aa9f0 0x7f9fdc072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fdbfff640 1 -- 172.21.15.107:0/1953254203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9fdc0aa9f0 msgr2=0x7f9fdc072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:28.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fdbfff640 1 --2- 172.21.15.107:0/1953254203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9fdc0aa9f0 0x7f9fdc072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fdbfff640 1 -- 172.21.15.107:0/1953254203 >> v1:172.21.15.110:6789/0 conn(0x7f9fdc0c1b30 legacy=0x7f9fdc0aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fdbfff640 1 -- 172.21.15.107:0/1953254203 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc0bf1f0 con 0x7f9fdc0a9300 2023-10-24T23:53:28.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fdbfff640 1 --2- 172.21.15.107:0/1953254203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0a9300 0x7f9fdc155620 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009870 tx=0x7f9fd401c160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23dc25c384f69d18 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fda7fc640 1 -- 172.21.15.107:0/1953254203 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40a4e00 con 0x7f9fdc0a9300 2023-10-24T23:53:28.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fda7fc640 1 -- 172.21.15.107:0/1953254203 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd40ac440 con 0x7f9fdc0a9300 2023-10-24T23:53:28.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.966+0000 7f9fda7fc640 1 -- 172.21.15.107:0/1953254203 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40ac7d0 con 0x7f9fdc0a9300 2023-10-24T23:53:28.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- 172.21.15.107:0/1953254203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0a9300 msgr2=0x7f9fdc155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 --2- 172.21.15.107:0/1953254203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0a9300 0x7f9fdc155620 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009870 tx=0x7f9fd401c160 comp rx=0 tx=0).stop 2023-10-24T23:53:28.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- 172.21.15.107:0/1953254203 shutdown_connections 2023-10-24T23:53:28.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 --2- 172.21.15.107:0/1953254203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0a9300 0x7f9fdc155620 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 --2- 172.21.15.107:0/1953254203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9fdc0aa9f0 0x7f9fdc072210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- 172.21.15.107:0/1953254203 >> 172.21.15.107:0/1953254203 conn(0x7f9fdc0c2860 msgr2=0x7f9fdc0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:28.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- 172.21.15.107:0/1953254203 shutdown_connections 2023-10-24T23:53:28.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- 172.21.15.107:0/1953254203 wait complete. 2023-10-24T23:53:28.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 Processor -- start 2023-10-24T23:53:28.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- start start 2023-10-24T23:53:28.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9fdc0a9300 0x7f9fdc06c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0c1b30 0x7f9fdc1ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9fdc150b70 0x7f9fdc0c0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc06fee0 con 0x7f9fdc0c1b30 2023-10-24T23:53:28.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc06bc90 con 0x7f9fdc0a9300 2023-10-24T23:53:28.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc06f9b0 con 0x7f9fdc150b70 2023-10-24T23:53:28.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fdaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0c1b30 0x7f9fdc1ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fdaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0c1b30 0x7f9fdc1ee8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33452/0 (socket says 172.21.15.107:33452) 2023-10-24T23:53:28.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fdaffd640 1 -- 172.21.15.107:0/112824518 learned_addr learned my addr 172.21.15.107:0/112824518 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:28.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fdaffd640 1 -- 172.21.15.107:0/112824518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9fdc150b70 msgr2=0x7f9fdc0c0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fdb7fe640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9fdc0a9300 0x7f9fdc06c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fdaffd640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9fdc150b70 0x7f9fdc0c0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fdaffd640 1 -- 172.21.15.107:0/112824518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9fdc0a9300 msgr2=0x7f9fdc06c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:28.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fdaffd640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9fdc0a9300 0x7f9fdc06c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:28.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fdaffd640 1 -- 172.21.15.107:0/112824518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd40094d0 con 0x7f9fdc0c1b30 2023-10-24T23:53:28.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fdaffd640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0c1b30 0x7f9fdc1ee8c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400efc0 tx=0x7f9fc400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fd3fff640 1 -- 172.21.15.107:0/112824518 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4014480 con 0x7f9fdc0c1b30 2023-10-24T23:53:28.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fd3fff640 1 -- 172.21.15.107:0/112824518 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc400f040 con 0x7f9fdc0c1b30 2023-10-24T23:53:28.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- 172.21.15.107:0/112824518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fdc06c230 con 0x7f9fdc0c1b30 2023-10-24T23:53:28.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fd3fff640 1 -- 172.21.15.107:0/112824518 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4014800 con 0x7f9fdc0c1b30 2023-10-24T23:53:28.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- 172.21.15.107:0/112824518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fdc06fc00 con 0x7f9fdc0c1b30 2023-10-24T23:53:28.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.970+0000 7f9fe2057640 1 -- 172.21.15.107:0/112824518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fdc06c3c0 con 0x7f9fdc0c1b30 2023-10-24T23:53:28.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.974+0000 7f9fd3fff640 1 -- 172.21.15.107:0/112824518 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9fc4014960 con 0x7f9fdc0c1b30 2023-10-24T23:53:28.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.974+0000 7f9fd3fff640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9fbc065e60 0x7f9fbc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:28.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.974+0000 7f9fd3fff640 1 -- 172.21.15.107:0/112824518 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f9fc4084710 con 0x7f9fdc0c1b30 2023-10-24T23:53:28.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.974+0000 7f9fdb7fe640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9fbc065e60 0x7f9fbc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:28.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.974+0000 7f9fdb7fe640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9fbc065e60 0x7f9fbc068320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc000f70 tx=0x7f9fcc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:28.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:28.978+0000 7f9fd3fff640 1 -- 172.21.15.107:0/112824518 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9fc4051aa0 con 0x7f9fdc0c1b30 2023-10-24T23:53:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.118+0000 7f9fe2057640 1 -- 172.21.15.107:0/112824518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f9fdc0aa600 con 0x7f9fdc0c1b30 2023-10-24T23:53:29.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fd3fff640 1 -- 172.21.15.107:0/112824518 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7f9fc40550c0 con 0x7f9fdc0c1b30 2023-10-24T23:53:29.125 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:29.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 -- 172.21.15.107:0/112824518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9fbc065e60 msgr2=0x7f9fbc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9fbc065e60 0x7f9fbc068320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc000f70 tx=0x7f9fcc009040 comp rx=0 tx=0).stop 2023-10-24T23:53:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 -- 172.21.15.107:0/112824518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0c1b30 msgr2=0x7f9fdc1ee8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0c1b30 0x7f9fdc1ee8c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400efc0 tx=0x7f9fc400c490 comp rx=0 tx=0).stop 2023-10-24T23:53:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 -- 172.21.15.107:0/112824518 shutdown_connections 2023-10-24T23:53:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9fbc065e60 0x7f9fbc068320 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9fdc150b70 0x7f9fdc0c0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fdc0c1b30 0x7f9fdc1ee8c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 --2- 172.21.15.107:0/112824518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9fdc0a9300 0x7f9fdc06c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 -- 172.21.15.107:0/112824518 >> 172.21.15.107:0/112824518 conn(0x7f9fdc0c2860 msgr2=0x7f9fdc0c4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 -- 172.21.15.107:0/112824518 shutdown_connections 2023-10-24T23:53:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.122+0000 7f9fe2057640 1 -- 172.21.15.107:0/112824518 wait complete. 2023-10-24T23:53:29.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-10-24T23:53:29.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.254+0000 7f0a9eccb640 1 Processor -- start 2023-10-24T23:53:29.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.254+0000 7f0a9eccb640 1 -- start start 2023-10-24T23:53:29.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0a980ab1a0 0x7f0a980c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0a98072720 0x7f0a980bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a980c1ce0 con 0x7f0a980bceb0 2023-10-24T23:53:29.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a980be1d0 con 0x7f0a980ab1a0 2023-10-24T23:53:29.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a980c0880 con 0x7f0a98072720 2023-10-24T23:53:29.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9ca67640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0a980ab1a0 0x7f0a980c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9d268640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0a980bceb0 0x7f0a980c4cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38292/0 (socket says 172.21.15.107:38292) 2023-10-24T23:53:29.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9d268640 1 -- 172.21.15.107:0/551119702 learned_addr learned my addr 172.21.15.107:0/551119702 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:29.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9ca67640 1 -- 172.21.15.107:0/551119702 >> v2:172.21.15.110:3300/0 conn(0x7f0a98072720 msgr2=0x7f0a980bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:29.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9ca67640 1 --2- 172.21.15.107:0/551119702 >> v2:172.21.15.110:3300/0 conn(0x7f0a98072720 0x7f0a980bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9ca67640 1 -- 172.21.15.107:0/551119702 >> v1:172.21.15.107:6789/0 conn(0x7f0a980bceb0 legacy=0x7f0a980c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a977fe640 1 -- 172.21.15.107:0/551119702 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192039649 0 0) 0x7f0a980c1ce0 con 0x7f0a980bceb0 2023-10-24T23:53:29.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9ca67640 1 -- 172.21.15.107:0/551119702 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a980c1f30 con 0x7f0a980ab1a0 2023-10-24T23:53:29.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9ca67640 1 --2- 172.21.15.107:0/551119702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980ab1a0 0x7f0a980c0d00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a98071e30 tx=0x7f0a9001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddff09a32daae7f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:29.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a977fe640 1 -- 172.21.15.107:0/551119702 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900a5870 con 0x7f0a980ab1a0 2023-10-24T23:53:29.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a977fe640 1 -- 172.21.15.107:0/551119702 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a900a59d0 con 0x7f0a980ab1a0 2023-10-24T23:53:29.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a977fe640 1 -- 172.21.15.107:0/551119702 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900a5d60 con 0x7f0a980ab1a0 2023-10-24T23:53:29.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 -- 172.21.15.107:0/551119702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980ab1a0 msgr2=0x7f0a980c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 --2- 172.21.15.107:0/551119702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980ab1a0 0x7f0a980c0d00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a98071e30 tx=0x7f0a9001c870 comp rx=0 tx=0).stop 2023-10-24T23:53:29.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 -- 172.21.15.107:0/551119702 shutdown_connections 2023-10-24T23:53:29.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 --2- 172.21.15.107:0/551119702 >> v2:172.21.15.110:3300/0 conn(0x7f0a98072720 0x7f0a980bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 --2- 172.21.15.107:0/551119702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980ab1a0 0x7f0a980c0d00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 -- 172.21.15.107:0/551119702 >> 172.21.15.107:0/551119702 conn(0x7f0a980c2860 msgr2=0x7f0a980c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:29.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 -- 172.21.15.107:0/551119702 shutdown_connections 2023-10-24T23:53:29.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.258+0000 7f0a9eccb640 1 -- 172.21.15.107:0/551119702 wait complete. 2023-10-24T23:53:29.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9eccb640 1 Processor -- start 2023-10-24T23:53:29.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9eccb640 1 -- start start 2023-10-24T23:53:29.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9eccb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a980ab1a0 0x7f0a981edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9eccb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a980bceb0 0x7f0a98151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9eccb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980a5930 0x7f0a981eb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9eccb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98150080 con 0x7f0a980a5930 2023-10-24T23:53:29.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9eccb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a981ee580 con 0x7f0a980bceb0 2023-10-24T23:53:29.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9eccb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a981ee860 con 0x7f0a980ab1a0 2023-10-24T23:53:29.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9d268640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980a5930 0x7f0a981eb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9d268640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980a5930 0x7f0a981eb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33466/0 (socket says 172.21.15.107:33466) 2023-10-24T23:53:29.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9d268640 1 -- 172.21.15.107:0/975219067 learned_addr learned my addr 172.21.15.107:0/975219067 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:29.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a97fff640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a980bceb0 0x7f0a98151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9ca67640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a980ab1a0 0x7f0a981edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9d268640 1 -- 172.21.15.107:0/975219067 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a980ab1a0 msgr2=0x7f0a981edfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9d268640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a980ab1a0 0x7f0a981edfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9d268640 1 -- 172.21.15.107:0/975219067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a980bceb0 msgr2=0x7f0a98151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9d268640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a980bceb0 0x7f0a98151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9d268640 1 -- 172.21.15.107:0/975219067 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a90009a70 con 0x7f0a980a5930 2023-10-24T23:53:29.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a97fff640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a980bceb0 0x7f0a98151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:29.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9ca67640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a980ab1a0 0x7f0a981edfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:29.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9d268640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980a5930 0x7f0a981eb060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800c390 tx=0x7f0a8800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:29.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a957fa640 1 -- 172.21.15.107:0/975219067 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a880074e0 con 0x7f0a980a5930 2023-10-24T23:53:29.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a957fa640 1 -- 172.21.15.107:0/975219067 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a88007640 con 0x7f0a980a5930 2023-10-24T23:53:29.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9eccb640 1 -- 172.21.15.107:0/975219067 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a981eeb40 con 0x7f0a980a5930 2023-10-24T23:53:29.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a957fa640 1 -- 172.21.15.107:0/975219067 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88004a10 con 0x7f0a980a5930 2023-10-24T23:53:29.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9eccb640 1 -- 172.21.15.107:0/975219067 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a981eee20 con 0x7f0a980a5930 2023-10-24T23:53:29.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a9eccb640 1 -- 172.21.15.107:0/975219067 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a981ead80 con 0x7f0a980a5930 2023-10-24T23:53:29.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.262+0000 7f0a957fa640 1 -- 172.21.15.107:0/975219067 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0a88004b70 con 0x7f0a980a5930 2023-10-24T23:53:29.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.266+0000 7f0a957fa640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a84065e90 0x7f0a84068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.266+0000 7f0a957fa640 1 -- 172.21.15.107:0/975219067 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f0a880838b0 con 0x7f0a980a5930 2023-10-24T23:53:29.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.266+0000 7f0a9ca67640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a84065e90 0x7f0a84068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.266+0000 7f0a9ca67640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a84065e90 0x7f0a84068350 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0a900af000 tx=0x7f0a900a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:29.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.270+0000 7f0a957fa640 1 -- 172.21.15.107:0/975219067 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0a981ead80 con 0x7f0a980a5930 2023-10-24T23:53:29.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.414+0000 7f0a9eccb640 1 -- 172.21.15.107:0/975219067 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0a981ead80 con 0x7f0a980a5930 2023-10-24T23:53:29.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.414+0000 7f0a957fa640 1 -- 172.21.15.107:0/975219067 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7f0a981ead80 con 0x7f0a980a5930 2023-10-24T23:53:29.420 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:29.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 -- 172.21.15.107:0/975219067 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a84065e90 msgr2=0x7f0a84068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a84065e90 0x7f0a84068350 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0a900af000 tx=0x7f0a900a7070 comp rx=0 tx=0).stop 2023-10-24T23:53:29.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 -- 172.21.15.107:0/975219067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980a5930 msgr2=0x7f0a981eb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980a5930 0x7f0a981eb060 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800c390 tx=0x7f0a8800c850 comp rx=0 tx=0).stop 2023-10-24T23:53:29.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 -- 172.21.15.107:0/975219067 shutdown_connections 2023-10-24T23:53:29.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980a5930 0x7f0a981eb060 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a980bceb0 0x7f0a98151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a84065e90 0x7f0a84068350 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 --2- 172.21.15.107:0/975219067 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a980ab1a0 0x7f0a981edfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 -- 172.21.15.107:0/975219067 >> 172.21.15.107:0/975219067 conn(0x7f0a980c2860 msgr2=0x7f0a981592f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:29.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 -- 172.21.15.107:0/975219067 shutdown_connections 2023-10-24T23:53:29.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.418+0000 7f0a9eccb640 1 -- 172.21.15.107:0/975219067 wait complete. 2023-10-24T23:53:29.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-10-24T23:53:29.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.550+0000 7fa9e255e640 1 Processor -- start 2023-10-24T23:53:29.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.550+0000 7fa9e255e640 1 -- start start 2023-10-24T23:53:29.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9e255e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9dc0ab1a0 0x7fa9dc0c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9e255e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa9dc0bceb0 0x7fa9dc15c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9e255e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc0c1ce0 con 0x7fa9dc072720 2023-10-24T23:53:29.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9e255e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc0be1d0 con 0x7fa9dc0bceb0 2023-10-24T23:53:29.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9e255e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0c0240 con 0x7fa9dc0ab1a0 2023-10-24T23:53:29.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9e0afb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa9dc0bceb0 0x7fa9dc15c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9dbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9dc0ab1a0 0x7fa9dc0c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9db7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa9dc072720 0x7fa9dc0bfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:48084/0 (socket says 172.21.15.107:48084) 2023-10-24T23:53:29.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9db7fe640 1 -- 172.21.15.107:0/1982419809 learned_addr learned my addr 172.21.15.107:0/1982419809 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:29.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9e0afb640 1 -- 172.21.15.107:0/1982419809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9dc0ab1a0 msgr2=0x7fa9dc0c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:29.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9e0afb640 1 --2- 172.21.15.107:0/1982419809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9dc0ab1a0 0x7fa9dc0c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9e0afb640 1 -- 172.21.15.107:0/1982419809 >> v1:172.21.15.110:6789/0 conn(0x7fa9dc072720 legacy=0x7fa9dc0bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9e0afb640 1 -- 172.21.15.107:0/1982419809 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc0c0880 con 0x7fa9dc0bceb0 2023-10-24T23:53:29.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9dbfff640 1 --2- 172.21.15.107:0/1982419809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9dc0ab1a0 0x7fa9dc0c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:53:29.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9e0afb640 1 --2- 172.21.15.107:0/1982419809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9dc0bceb0 0x7fa9dc15c660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0097c0 tx=0x7fa9cc01c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49a9818d884c2d8f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:29.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9daffd640 1 -- 172.21.15.107:0/1982419809 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc0ab600 con 0x7fa9dc0bceb0 2023-10-24T23:53:29.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9daffd640 1 -- 172.21.15.107:0/1982419809 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc0a7050 con 0x7fa9dc0bceb0 2023-10-24T23:53:29.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.554+0000 7fa9daffd640 1 -- 172.21.15.107:0/1982419809 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc0ab980 con 0x7fa9dc0bceb0 2023-10-24T23:53:29.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 -- 172.21.15.107:0/1982419809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9dc0bceb0 msgr2=0x7fa9dc15c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 --2- 172.21.15.107:0/1982419809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9dc0bceb0 0x7fa9dc15c660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0097c0 tx=0x7fa9cc01c330 comp rx=0 tx=0).stop 2023-10-24T23:53:29.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 -- 172.21.15.107:0/1982419809 shutdown_connections 2023-10-24T23:53:29.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 --2- 172.21.15.107:0/1982419809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9dc0bceb0 0x7fa9dc15c660 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 --2- 172.21.15.107:0/1982419809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9dc0ab1a0 0x7fa9dc0c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 -- 172.21.15.107:0/1982419809 >> 172.21.15.107:0/1982419809 conn(0x7fa9dc0c2860 msgr2=0x7fa9dc0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:29.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 -- 172.21.15.107:0/1982419809 shutdown_connections 2023-10-24T23:53:29.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 -- 172.21.15.107:0/1982419809 wait complete. 2023-10-24T23:53:29.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 Processor -- start 2023-10-24T23:53:29.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 -- start start 2023-10-24T23:53:29.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9dc0ab1a0 0x7fa9dc15a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9dc0bceb0 0x7fa9dc1ee580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc0a7140 0x7fa9dc1eaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1eeac0 con 0x7fa9dc0a7140 2023-10-24T23:53:29.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1eaa80 con 0x7fa9dc0ab1a0 2023-10-24T23:53:29.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc070db0 con 0x7fa9dc0bceb0 2023-10-24T23:53:29.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e0afb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc0a7140 0x7fa9dc1eaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e0afb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc0a7140 0x7fa9dc1eaf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33474/0 (socket says 172.21.15.107:33474) 2023-10-24T23:53:29.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e0afb640 1 -- 172.21.15.107:0/2134707382 learned_addr learned my addr 172.21.15.107:0/2134707382 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:29.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9db7fe640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9dc0bceb0 0x7fa9dc1ee580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9dbfff640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9dc0ab1a0 0x7fa9dc15a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e0afb640 1 -- 172.21.15.107:0/2134707382 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9dc0bceb0 msgr2=0x7fa9dc1ee580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e0afb640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9dc0bceb0 0x7fa9dc1ee580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e0afb640 1 -- 172.21.15.107:0/2134707382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9dc0ab1a0 msgr2=0x7fa9dc15a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e0afb640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9dc0ab1a0 0x7fa9dc15a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e0afb640 1 -- 172.21.15.107:0/2134707382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d4009a70 con 0x7fa9dc0a7140 2023-10-24T23:53:29.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e0afb640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc0a7140 0x7fa9dc1eaf10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0097c0 tx=0x7fa9cc0190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:29.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9d8ff9640 1 -- 172.21.15.107:0/2134707382 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc006e20 con 0x7fa9dc0a7140 2023-10-24T23:53:29.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9d8ff9640 1 -- 172.21.15.107:0/2134707382 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc0b4400 con 0x7fa9dc0a7140 2023-10-24T23:53:29.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 -- 172.21.15.107:0/2134707382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9cc009420 con 0x7fa9dc0a7140 2023-10-24T23:53:29.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9d8ff9640 1 -- 172.21.15.107:0/2134707382 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc0b47b0 con 0x7fa9dc0a7140 2023-10-24T23:53:29.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.558+0000 7fa9e255e640 1 -- 172.21.15.107:0/2134707382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc071370 con 0x7fa9dc0a7140 2023-10-24T23:53:29.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.562+0000 7fa9e255e640 1 -- 172.21.15.107:0/2134707382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc0beff0 con 0x7fa9dc0a7140 2023-10-24T23:53:29.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.562+0000 7fa9d8ff9640 1 -- 172.21.15.107:0/2134707382 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa9cc0b2040 con 0x7fa9dc0a7140 2023-10-24T23:53:29.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.562+0000 7fa9d8ff9640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa9c8065e90 0x7fa9c8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.562+0000 7fa9d8ff9640 1 -- 172.21.15.107:0/2134707382 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fa9cc11dad0 con 0x7fa9dc0a7140 2023-10-24T23:53:29.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.562+0000 7fa9dbfff640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa9c8065e90 0x7fa9c8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.562+0000 7fa9dbfff640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa9c8065e90 0x7fa9c8068350 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4007600 tx=0x7fa9d400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:29.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.566+0000 7fa9d8ff9640 1 -- 172.21.15.107:0/2134707382 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa9cc0eadf0 con 0x7fa9dc0a7140 2023-10-24T23:53:29.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.706+0000 7fa9e255e640 1 -- 172.21.15.107:0/2134707382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7fa9dc1eabc0 con 0x7fa9dc0a7140 2023-10-24T23:53:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.706+0000 7fa9d8ff9640 1 -- 172.21.15.107:0/2134707382 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7fa9cc0ee410 con 0x7fa9dc0a7140 2023-10-24T23:53:29.714 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 -- 172.21.15.107:0/2134707382 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa9c8065e90 msgr2=0x7fa9c8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa9c8065e90 0x7fa9c8068350 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4007600 tx=0x7fa9d400b040 comp rx=0 tx=0).stop 2023-10-24T23:53:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 -- 172.21.15.107:0/2134707382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc0a7140 msgr2=0x7fa9dc1eaf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc0a7140 0x7fa9dc1eaf10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0097c0 tx=0x7fa9cc0190d0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 -- 172.21.15.107:0/2134707382 shutdown_connections 2023-10-24T23:53:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc0a7140 0x7fa9dc1eaf10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa9c8065e90 0x7fa9c8068350 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9dc0bceb0 0x7fa9dc1ee580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 --2- 172.21.15.107:0/2134707382 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9dc0ab1a0 0x7fa9dc15a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 -- 172.21.15.107:0/2134707382 >> 172.21.15.107:0/2134707382 conn(0x7fa9dc0c2860 msgr2=0x7fa9dc152800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:29.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 -- 172.21.15.107:0/2134707382 shutdown_connections 2023-10-24T23:53:29.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.710+0000 7fa9e255e640 1 -- 172.21.15.107:0/2134707382 wait complete. 2023-10-24T23:53:29.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:29.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.842+0000 7fd5acf00640 1 Processor -- start 2023-10-24T23:53:29.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.842+0000 7fd5acf00640 1 -- start start 2023-10-24T23:53:29.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5a80a9820 0x7fd5a80ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5a806ee40 0x7fd5a80aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a806f730 con 0x7fd5a814fd40 2023-10-24T23:53:29.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a80aa750 con 0x7fd5a80a9820 2023-10-24T23:53:29.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8156a40 con 0x7fd5a806ee40 2023-10-24T23:53:29.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd5a814fd40 0x7fd5a80c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38302/0 (socket says 172.21.15.107:38302) 2023-10-24T23:53:29.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a5d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5a80a9820 0x7fd5a80ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a659c640 1 -- 172.21.15.107:0/856906434 learned_addr learned my addr 172.21.15.107:0/856906434 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:29.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a5d9b640 1 -- 172.21.15.107:0/856906434 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5a806ee40 msgr2=0x7fd5a80aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:29.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a5d9b640 1 --2- 172.21.15.107:0/856906434 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5a806ee40 0x7fd5a80aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a5d9b640 1 -- 172.21.15.107:0/856906434 >> v1:172.21.15.107:6789/0 conn(0x7fd5a814fd40 legacy=0x7fd5a80c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a5d9b640 1 -- 172.21.15.107:0/856906434 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a8156710 con 0x7fd5a80a9820 2023-10-24T23:53:29.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a559a640 1 -- 172.21.15.107:0/856906434 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671449009 0 0) 0x7fd5a806f730 con 0x7fd5a814fd40 2023-10-24T23:53:29.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a5d9b640 1 --2- 172.21.15.107:0/856906434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a80a9820 0x7fd5a80ab090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0097c0 tx=0x7fd59c01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30cf25001c1dda server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:29.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a559a640 1 -- 172.21.15.107:0/856906434 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c0ab600 con 0x7fd5a80a9820 2023-10-24T23:53:29.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a559a640 1 -- 172.21.15.107:0/856906434 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59c0a7050 con 0x7fd5a80a9820 2023-10-24T23:53:29.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5a559a640 1 -- 172.21.15.107:0/856906434 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c0ab980 con 0x7fd5a80a9820 2023-10-24T23:53:29.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 -- 172.21.15.107:0/856906434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a80a9820 msgr2=0x7fd5a80ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 --2- 172.21.15.107:0/856906434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a80a9820 0x7fd5a80ab090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0097c0 tx=0x7fd59c01c330 comp rx=0 tx=0).stop 2023-10-24T23:53:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 -- 172.21.15.107:0/856906434 shutdown_connections 2023-10-24T23:53:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 --2- 172.21.15.107:0/856906434 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5a806ee40 0x7fd5a80aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 --2- 172.21.15.107:0/856906434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a80a9820 0x7fd5a80ab090 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 -- 172.21.15.107:0/856906434 >> 172.21.15.107:0/856906434 conn(0x7fd5a80c2950 msgr2=0x7fd5a80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:29.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 -- 172.21.15.107:0/856906434 shutdown_connections 2023-10-24T23:53:29.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.846+0000 7fd5acf00640 1 -- 172.21.15.107:0/856906434 wait complete. 2023-10-24T23:53:29.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5acf00640 1 Processor -- start 2023-10-24T23:53:29.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5acf00640 1 -- start start 2023-10-24T23:53:29.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5acf00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5a806ee40 0x7fd5a80bba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5acf00640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5a814fd40 0x7fd5a80bc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5acf00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a814ced0 0x7fd5a80c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5acf00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a814cd20 con 0x7fd5a814ced0 2023-10-24T23:53:29.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5acf00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a8155c00 con 0x7fd5a806ee40 2023-10-24T23:53:29.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5acf00640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a80bb540 con 0x7fd5a814fd40 2023-10-24T23:53:29.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a814ced0 0x7fd5a80c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a814ced0 0x7fd5a80c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33488/0 (socket says 172.21.15.107:33488) 2023-10-24T23:53:29.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a6d9d640 1 -- 172.21.15.107:0/1789976273 learned_addr learned my addr 172.21.15.107:0/1789976273 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:29.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a5d9b640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5a814fd40 0x7fd5a80bc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a659c640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5a806ee40 0x7fd5a80bba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a6d9d640 1 -- 172.21.15.107:0/1789976273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5a814fd40 msgr2=0x7fd5a80bc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a6d9d640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5a814fd40 0x7fd5a80bc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a6d9d640 1 -- 172.21.15.107:0/1789976273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5a806ee40 msgr2=0x7fd5a80bba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:29.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a6d9d640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5a806ee40 0x7fd5a80bba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:29.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a6d9d640 1 -- 172.21.15.107:0/1789976273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c009420 con 0x7fd5a814ced0 2023-10-24T23:53:29.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a659c640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5a806ee40 0x7fd5a80bba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:29.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5a6d9d640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a814ced0 0x7fd5a80c5400 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000b9e0 tx=0x7fd5a000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:29.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd596ffd640 1 -- 172.21.15.107:0/1789976273 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a000cb20 con 0x7fd5a814ced0 2023-10-24T23:53:29.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd596ffd640 1 -- 172.21.15.107:0/1789976273 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a000cc80 con 0x7fd5a814ced0 2023-10-24T23:53:29.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5acf00640 1 -- 172.21.15.107:0/1789976273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a814ca40 con 0x7fd5a814ced0 2023-10-24T23:53:29.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd596ffd640 1 -- 172.21.15.107:0/1789976273 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0014680 con 0x7fd5a814ced0 2023-10-24T23:53:29.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.850+0000 7fd5acf00640 1 -- 172.21.15.107:0/1789976273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a80bced0 con 0x7fd5a814ced0 2023-10-24T23:53:29.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.854+0000 7fd5acf00640 1 -- 172.21.15.107:0/1789976273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a80c24a0 con 0x7fd5a814ced0 2023-10-24T23:53:29.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.854+0000 7fd596ffd640 1 -- 172.21.15.107:0/1789976273 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd5a001d3b0 con 0x7fd5a814ced0 2023-10-24T23:53:29.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.854+0000 7fd596ffd640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd588065e60 0x7fd588068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:29.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.854+0000 7fd596ffd640 1 -- 172.21.15.107:0/1789976273 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fd5a0083a30 con 0x7fd5a814ced0 2023-10-24T23:53:29.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.854+0000 7fd5a659c640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd588065e60 0x7fd588068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:29.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.854+0000 7fd5a659c640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd588065e60 0x7fd588068320 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd598007fd0 tx=0x7fd598007c20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:29.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:29.858+0000 7fd596ffd640 1 -- 172.21.15.107:0/1789976273 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd5a0050de0 con 0x7fd5a814ced0 2023-10-24T23:53:30.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.042+0000 7fd5acf00640 1 -- 172.21.15.107:0/1789976273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5a80bb3a0 con 0x7fd5a814ced0 2023-10-24T23:53:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.042+0000 7fd596ffd640 1 -- 172.21.15.107:0/1789976273 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7fd5a80bb3a0 con 0x7fd5a814ced0 2023-10-24T23:53:30.047 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:30.048 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:53:30.048 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:53:30.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 -- 172.21.15.107:0/1789976273 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd588065e60 msgr2=0x7fd588068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd588065e60 0x7fd588068320 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd598007fd0 tx=0x7fd598007c20 comp rx=0 tx=0).stop 2023-10-24T23:53:30.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 -- 172.21.15.107:0/1789976273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a814ced0 msgr2=0x7fd5a80c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a814ced0 0x7fd5a80c5400 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000b9e0 tx=0x7fd5a000bea0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 -- 172.21.15.107:0/1789976273 shutdown_connections 2023-10-24T23:53:30.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5a814ced0 0x7fd5a80c5400 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd588065e60 0x7fd588068320 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5a814fd40 0x7fd5a80bc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 --2- 172.21.15.107:0/1789976273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5a806ee40 0x7fd5a80bba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 -- 172.21.15.107:0/1789976273 >> 172.21.15.107:0/1789976273 conn(0x7fd5a80c2950 msgr2=0x7fd5a8151e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:30.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 -- 172.21.15.107:0/1789976273 shutdown_connections 2023-10-24T23:53:30.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.046+0000 7fd5acf00640 1 -- 172.21.15.107:0/1789976273 wait complete. 2023-10-24T23:53:30.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:30.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.174+0000 7f09035e9640 1 Processor -- start 2023-10-24T23:53:30.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.174+0000 7f09035e9640 1 -- start start 2023-10-24T23:53:30.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f09035e9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08fc071220 0x7f08fc06bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f09035e9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08fc150bc0 con 0x7f08fc151960 2023-10-24T23:53:30.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f09035e9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08fc06c510 con 0x7f08fc0c24c0 2023-10-24T23:53:30.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f09035e9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08fc06c7f0 con 0x7f08fc071220 2023-10-24T23:53:30.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0900b84640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f08fc151960 0x7f08fc071a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38310/0 (socket says 172.21.15.107:38310) 2023-10-24T23:53:30.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0900b84640 1 -- 172.21.15.107:0/2174385943 learned_addr learned my addr 172.21.15.107:0/2174385943 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:30.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0901385640 1 --2- 172.21.15.107:0/2174385943 >> v2:172.21.15.110:3300/0 conn(0x7f08fc071220 0x7f08fc06bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160972939 0 0) 0x7f08fc150bc0 con 0x7f08fc151960 2023-10-24T23:53:30.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08e8004640 con 0x7f08fc151960 2023-10-24T23:53:30.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1770580388 0 0) 0x7f08e8004640 con 0x7f08fc151960 2023-10-24T23:53:30.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 >> v2:172.21.15.110:3300/0 conn(0x7f08fc071220 msgr2=0x7f08fc06bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:30.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 --2- 172.21.15.107:0/2174385943 >> v2:172.21.15.110:3300/0 conn(0x7f08fc071220 0x7f08fc06bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 >> v1:172.21.15.110:6789/0 conn(0x7f08fc0c24c0 legacy=0x7f08fc06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc150e80 con 0x7f08fc151960 2023-10-24T23:53:30.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f08ec001680 con 0x7f08fc151960 2023-10-24T23:53:30.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 >> v1:172.21.15.107:6789/0 conn(0x7f08fc151960 legacy=0x7f08fc071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 --2- 172.21.15.107:0/2174385943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08e8004a90 0x7f08e8004eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 --2- 172.21.15.107:0/2174385943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08e80058e0 0x7f08e8005d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 --2- 172.21.15.107:0/2174385943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08e80062a0 0x7f08e800caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc06c7f0 con 0x7f08e80058e0 2023-10-24T23:53:30.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e800d240 con 0x7f08e80062a0 2023-10-24T23:53:30.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e800d400 con 0x7f08e8004a90 2023-10-24T23:53:30.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0900b84640 1 --2- 172.21.15.107:0/2174385943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08e80058e0 0x7f08e8005d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0901b86640 1 --2- 172.21.15.107:0/2174385943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08e80062a0 0x7f08e800caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0901385640 1 --2- 172.21.15.107:0/2174385943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08e8004a90 0x7f08e8004eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0900b84640 1 -- 172.21.15.107:0/2174385943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08e8004a90 msgr2=0x7f08e8004eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:30.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0900b84640 1 --2- 172.21.15.107:0/2174385943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08e8004a90 0x7f08e8004eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0900b84640 1 -- 172.21.15.107:0/2174385943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08e80062a0 msgr2=0x7f08e800caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:30.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0900b84640 1 --2- 172.21.15.107:0/2174385943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08e80062a0 0x7f08e800caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0900b84640 1 -- 172.21.15.107:0/2174385943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e8004640 con 0x7f08e80058e0 2023-10-24T23:53:30.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0901b86640 1 -- 172.21.15.107:0/2174385943 reap_dead start 2023-10-24T23:53:30.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.178+0000 7f0900b84640 1 --2- 172.21.15.107:0/2174385943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08e80058e0 0x7f08e8005d60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f08ec00c240 tx=0x7f08ec01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=222e9facbd3013b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:30.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec005ae0 con 0x7f08e80058e0 2023-10-24T23:53:30.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ec005c40 con 0x7f08e80058e0 2023-10-24T23:53:30.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f08f3fff640 1 -- 172.21.15.107:0/2174385943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec0a59d0 con 0x7f08e80058e0 2023-10-24T23:53:30.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 -- 172.21.15.107:0/2174385943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08e80058e0 msgr2=0x7f08e8005d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 --2- 172.21.15.107:0/2174385943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08e80058e0 0x7f08e8005d60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f08ec00c240 tx=0x7f08ec01cc40 comp rx=0 tx=0).stop 2023-10-24T23:53:30.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 -- 172.21.15.107:0/2174385943 shutdown_connections 2023-10-24T23:53:30.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 --2- 172.21.15.107:0/2174385943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08e80058e0 0x7f08e8005d60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 -- 172.21.15.107:0/2174385943 >> 172.21.15.107:0/2174385943 conn(0x7f08fc0c2950 msgr2=0x7f08fc0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:30.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 -- 172.21.15.107:0/2174385943 shutdown_connections 2023-10-24T23:53:30.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 -- 172.21.15.107:0/2174385943 wait complete. 2023-10-24T23:53:30.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 Processor -- start 2023-10-24T23:53:30.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 -- start start 2023-10-24T23:53:30.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08e80058e0 0x7f08fc0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08fc149220 0x7f08fc148490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc1489d0 0x7f08fc14ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc150510 con 0x7f08fc1489d0 2023-10-24T23:53:30.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc150ad0 con 0x7f08fc149220 2023-10-24T23:53:30.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc148e50 con 0x7f08e80058e0 2023-10-24T23:53:30.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0901b86640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc1489d0 0x7f08fc14ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0901b86640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc1489d0 0x7f08fc14ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33508/0 (socket says 172.21.15.107:33508) 2023-10-24T23:53:30.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0901b86640 1 -- 172.21.15.107:0/3753004484 learned_addr learned my addr 172.21.15.107:0/3753004484 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:30.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0900b84640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08fc149220 0x7f08fc148490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0901385640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08e80058e0 0x7f08fc0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0901b86640 1 -- 172.21.15.107:0/3753004484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08e80058e0 msgr2=0x7f08fc0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0901b86640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08e80058e0 0x7f08fc0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0901b86640 1 -- 172.21.15.107:0/3753004484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08fc149220 msgr2=0x7f08fc148490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0901b86640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08fc149220 0x7f08fc148490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0901b86640 1 -- 172.21.15.107:0/3753004484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc1f1730 con 0x7f08fc1489d0 2023-10-24T23:53:30.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0901385640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08e80058e0 0x7f08fc0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:30.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f0901b86640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc1489d0 0x7f08fc14ec40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f08fc06d320 tx=0x7f08f400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:30.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f08f1ffb640 1 -- 172.21.15.107:0/3753004484 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4006e30 con 0x7f08fc1489d0 2023-10-24T23:53:30.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f08f1ffb640 1 -- 172.21.15.107:0/3753004484 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f4003980 con 0x7f08fc1489d0 2023-10-24T23:53:30.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 -- 172.21.15.107:0/3753004484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08fc1f1a10 con 0x7f08fc1489d0 2023-10-24T23:53:30.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f08f1ffb640 1 -- 172.21.15.107:0/3753004484 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4003d30 con 0x7f08fc1489d0 2023-10-24T23:53:30.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.182+0000 7f09035e9640 1 -- 172.21.15.107:0/3753004484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08fc0c5720 con 0x7f08fc1489d0 2023-10-24T23:53:30.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.186+0000 7f09035e9640 1 -- 172.21.15.107:0/3753004484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08fc14ea00 con 0x7f08fc1489d0 2023-10-24T23:53:30.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.186+0000 7f08f1ffb640 1 -- 172.21.15.107:0/3753004484 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f08f4005110 con 0x7f08fc1489d0 2023-10-24T23:53:30.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.186+0000 7f08f1ffb640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f08e0065e60 0x7f08e0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.186+0000 7f08f1ffb640 1 -- 172.21.15.107:0/3753004484 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f08f4083220 con 0x7f08fc1489d0 2023-10-24T23:53:30.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.186+0000 7f0901385640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f08e0065e60 0x7f08e0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.186+0000 7f0901385640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f08e0065e60 0x7f08e0068320 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009b70 tx=0x7f08f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:30.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.190+0000 7f08f1ffb640 1 -- 172.21.15.107:0/3753004484 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f08f4050680 con 0x7f08fc1489d0 2023-10-24T23:53:30.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.342+0000 7f09035e9640 1 -- 172.21.15.107:0/3753004484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08fc0746e0 con 0x7f08fc1489d0 2023-10-24T23:53:30.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.346+0000 7f08f1ffb640 1 -- 172.21.15.107:0/3753004484 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21531 (secure 0 0 0) 0x7f08f4053ca0 con 0x7f08fc1489d0 2023-10-24T23:53:30.352 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:30.352 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:12.781093+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:30.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 -- 172.21.15.107:0/3753004484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f08e0065e60 msgr2=0x7f08e0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f08e0065e60 0x7f08e0068320 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009b70 tx=0x7f08f8008040 comp rx=0 tx=0).stop 2023-10-24T23:53:30.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 -- 172.21.15.107:0/3753004484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc1489d0 msgr2=0x7f08fc14ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc1489d0 0x7f08fc14ec40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f08fc06d320 tx=0x7f08f400e960 comp rx=0 tx=0).stop 2023-10-24T23:53:30.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 -- 172.21.15.107:0/3753004484 shutdown_connections 2023-10-24T23:53:30.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc1489d0 0x7f08fc14ec40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08fc149220 0x7f08fc148490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f08e0065e60 0x7f08e0068320 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 --2- 172.21.15.107:0/3753004484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08e80058e0 0x7f08fc0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 -- 172.21.15.107:0/3753004484 >> 172.21.15.107:0/3753004484 conn(0x7f08fc0c2950 msgr2=0x7f08fc0700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:30.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 -- 172.21.15.107:0/3753004484 shutdown_connections 2023-10-24T23:53:30.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.350+0000 7f09035e9640 1 -- 172.21.15.107:0/3753004484 wait complete. 2023-10-24T23:53:30.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:30.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.470+0000 7f0b7e49c640 1 Processor -- start 2023-10-24T23:53:30.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.470+0000 7f0b7e49c640 1 -- start start 2023-10-24T23:53:30.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.470+0000 7f0b7e49c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b78152040 0x7f0b780c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.470+0000 7f0b7e49c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0b780c8100 0x7f0b780abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.470+0000 7f0b7e49c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b7806a4b0 0x7f0b780ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.470+0000 7f0b7e49c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b78155680 con 0x7f0b78152040 2023-10-24T23:53:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.470+0000 7f0b7e49c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b78154a70 con 0x7f0b780c8100 2023-10-24T23:53:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.470+0000 7f0b7e49c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b78154d50 con 0x7f0b7806a4b0 2023-10-24T23:53:30.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b77fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b78152040 0x7f0b780c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b77fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0b78152040 0x7f0b780c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33518/0 (socket says 172.21.15.107:33518) 2023-10-24T23:53:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b77fff640 1 -- 172.21.15.107:0/3280379977 learned_addr learned my addr 172.21.15.107:0/3280379977 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b7ca39640 1 --2- 172.21.15.107:0/3280379977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b7806a4b0 0x7f0b780ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b777fe640 1 --2- 172.21.15.107:0/3280379977 >> v2:172.21.15.110:3300/0 conn(0x7f0b780c8100 0x7f0b780abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b77fff640 1 -- 172.21.15.107:0/3280379977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b7806a4b0 msgr2=0x7f0b780ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:30.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b77fff640 1 --2- 172.21.15.107:0/3280379977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b7806a4b0 0x7f0b780ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b77fff640 1 -- 172.21.15.107:0/3280379977 >> v2:172.21.15.110:3300/0 conn(0x7f0b780c8100 msgr2=0x7f0b780abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:30.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b77fff640 1 --2- 172.21.15.107:0/3280379977 >> v2:172.21.15.110:3300/0 conn(0x7f0b780c8100 0x7f0b780abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b77fff640 1 -- 172.21.15.107:0/3280379977 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78155030 con 0x7f0b78152040 2023-10-24T23:53:30.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b77fff640 1 --2- 172.21.15.107:0/3280379977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b78152040 0x7f0b780c4b80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0b780c4500 tx=0x7f0b6801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2c627a6539e8f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:30.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b76ffd640 1 -- 172.21.15.107:0/3280379977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b680ab600 con 0x7f0b78152040 2023-10-24T23:53:30.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b76ffd640 1 -- 172.21.15.107:0/3280379977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b680a7050 con 0x7f0b78152040 2023-10-24T23:53:30.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b76ffd640 1 -- 172.21.15.107:0/3280379977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b680ab980 con 0x7f0b78152040 2023-10-24T23:53:30.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b7e49c640 1 -- 172.21.15.107:0/3280379977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b78152040 msgr2=0x7f0b780c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b7e49c640 1 --2- 172.21.15.107:0/3280379977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b78152040 0x7f0b780c4b80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0b780c4500 tx=0x7f0b6801c470 comp rx=0 tx=0).stop 2023-10-24T23:53:30.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b7e49c640 1 -- 172.21.15.107:0/3280379977 shutdown_connections 2023-10-24T23:53:30.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b7e49c640 1 --2- 172.21.15.107:0/3280379977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b7806a4b0 0x7f0b780ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b7e49c640 1 --2- 172.21.15.107:0/3280379977 >> v2:172.21.15.110:3300/0 conn(0x7f0b780c8100 0x7f0b780abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b7e49c640 1 --2- 172.21.15.107:0/3280379977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b78152040 0x7f0b780c4b80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b7e49c640 1 -- 172.21.15.107:0/3280379977 >> 172.21.15.107:0/3280379977 conn(0x7f0b780ca200 msgr2=0x7f0b780ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:30.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b7e49c640 1 -- 172.21.15.107:0/3280379977 shutdown_connections 2023-10-24T23:53:30.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.474+0000 7f0b7e49c640 1 -- 172.21.15.107:0/3280379977 wait complete. 2023-10-24T23:53:30.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7e49c640 1 Processor -- start 2023-10-24T23:53:30.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7e49c640 1 -- start start 2023-10-24T23:53:30.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7e49c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b7806a4b0 0x7f0b780aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7e49c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b780c8100 0x7f0b78156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7e49c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b78152040 0x7f0b781573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7e49c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780c1490 con 0x7f0b78152040 2023-10-24T23:53:30.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7e49c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780aa430 con 0x7f0b7806a4b0 2023-10-24T23:53:30.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7e49c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780aa1c0 con 0x7f0b780c8100 2023-10-24T23:53:30.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7ca39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b78152040 0x7f0b781573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7ca39640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b78152040 0x7f0b781573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33530/0 (socket says 172.21.15.107:33530) 2023-10-24T23:53:30.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7ca39640 1 -- 172.21.15.107:0/807361138 learned_addr learned my addr 172.21.15.107:0/807361138 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:30.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b777fe640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b780c8100 0x7f0b78156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b77fff640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b7806a4b0 0x7f0b780aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7ca39640 1 -- 172.21.15.107:0/807361138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b780c8100 msgr2=0x7f0b78156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7ca39640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b780c8100 0x7f0b78156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7ca39640 1 -- 172.21.15.107:0/807361138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b7806a4b0 msgr2=0x7f0b780aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7ca39640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b7806a4b0 0x7f0b780aa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7ca39640 1 -- 172.21.15.107:0/807361138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b680094d0 con 0x7f0b78152040 2023-10-24T23:53:30.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7ca39640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b78152040 0x7f0b781573e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0b70007d50 tx=0x7f0b7000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b74ff9640 1 -- 172.21.15.107:0/807361138 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70014420 con 0x7f0b78152040 2023-10-24T23:53:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b74ff9640 1 -- 172.21.15.107:0/807361138 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b700145a0 con 0x7f0b78152040 2023-10-24T23:53:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7e49c640 1 -- 172.21.15.107:0/807361138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b781563c0 con 0x7f0b78152040 2023-10-24T23:53:30.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b74ff9640 1 -- 172.21.15.107:0/807361138 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70014950 con 0x7f0b78152040 2023-10-24T23:53:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7e49c640 1 -- 172.21.15.107:0/807361138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b78156980 con 0x7f0b78152040 2023-10-24T23:53:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.478+0000 7f0b7e49c640 1 -- 172.21.15.107:0/807361138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b78153940 con 0x7f0b78152040 2023-10-24T23:53:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.482+0000 7f0b74ff9640 1 -- 172.21.15.107:0/807361138 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0b70014ab0 con 0x7f0b78152040 2023-10-24T23:53:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.482+0000 7f0b74ff9640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b58065e90 0x7f0b58068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.482+0000 7f0b74ff9640 1 -- 172.21.15.107:0/807361138 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f0b70084340 con 0x7f0b78152040 2023-10-24T23:53:30.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.482+0000 7f0b77fff640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b58065e90 0x7f0b58068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:30.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.482+0000 7f0b77fff640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b58065e90 0x7f0b58068350 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0b78067ba0 tx=0x7f0b68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:30.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.486+0000 7f0b74ff9640 1 -- 172.21.15.107:0/807361138 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0b700516f0 con 0x7f0b78152040 2023-10-24T23:53:30.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.658+0000 7f0b7e49c640 1 -- 172.21.15.107:0/807361138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b780c4ef0 con 0x7f0b78152040 2023-10-24T23:53:30.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.658+0000 7f0b74ff9640 1 -- 172.21.15.107:0/807361138 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f0b70054d10 con 0x7f0b78152040 2023-10-24T23:53:30.664 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:30.664 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:53:30.664 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:53:30.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 -- 172.21.15.107:0/807361138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b58065e90 msgr2=0x7f0b58068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b58065e90 0x7f0b58068350 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0b78067ba0 tx=0x7f0b68009040 comp rx=0 tx=0).stop 2023-10-24T23:53:30.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 -- 172.21.15.107:0/807361138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b78152040 msgr2=0x7f0b781573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:30.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b78152040 0x7f0b781573e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0b70007d50 tx=0x7f0b7000e4b0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 -- 172.21.15.107:0/807361138 shutdown_connections 2023-10-24T23:53:30.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b78152040 0x7f0b781573e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0b58065e90 0x7f0b58068350 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0b780c8100 0x7f0b78156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 --2- 172.21.15.107:0/807361138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b7806a4b0 0x7f0b780aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:30.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 -- 172.21.15.107:0/807361138 >> 172.21.15.107:0/807361138 conn(0x7f0b780ca200 msgr2=0x7f0b78153230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:30.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 -- 172.21.15.107:0/807361138 shutdown_connections 2023-10-24T23:53:30.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:30.662+0000 7f0b7e49c640 1 -- 172.21.15.107:0/807361138 wait complete. 2023-10-24T23:53:31.084 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-24T23:53:02.679785+0000', 'modified': '2023-10-24T23:53:06.342509+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4299}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4299': {'gid': 4299, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.110:6835/2480841919', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.110:6833', 'nonce': 2480841919}, {'type': 'v1', 'addr': '172.21.15.110:6835', 'nonce': 2480841919}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'a', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-24T23:53:31.084 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-24T23:53:31.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-24T23:53:31.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4781200640 1 Processor -- start 2023-10-24T23:53:31.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4781200640 1 -- start start 2023-10-24T23:53:31.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4781200640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f477c0c1b30 0x7f477c072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4781200640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f477c0aa9f0 0x7f477c155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4781200640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477c0befa0 con 0x7f477c0a9300 2023-10-24T23:53:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4781200640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477c0a9b80 con 0x7f477c0c1b30 2023-10-24T23:53:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4781200640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477c0aa0b0 con 0x7f477c0aa9f0 2023-10-24T23:53:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477a59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f477c0a9300 0x7f477c0aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38318/0 (socket says 172.21.15.107:38318) 2023-10-24T23:53:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477a59c640 1 -- 172.21.15.107:0/64904472 learned_addr learned my addr 172.21.15.107:0/64904472 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:31.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477ad9d640 1 --2- 172.21.15.107:0/64904472 >> v2:172.21.15.110:3300/0 conn(0x7f477c0c1b30 0x7f477c072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477b59e640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f477c0aa9f0 0x7f477c155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211517317 0 0) 0x7f477c0befa0 con 0x7f477c0a9300 2023-10-24T23:53:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4764005740 con 0x7f477c0a9300 2023-10-24T23:53:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3534642705 0 0) 0x7f4764005740 con 0x7f477c0a9300 2023-10-24T23:53:31.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f477c0aa9f0 msgr2=0x7f477c155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f477c0aa9f0 0x7f477c155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 >> v2:172.21.15.110:3300/0 conn(0x7f477c0c1b30 msgr2=0x7f477c072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 --2- 172.21.15.107:0/64904472 >> v2:172.21.15.110:3300/0 conn(0x7f477c0c1b30 0x7f477c072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c0bf1f0 con 0x7f477c0a9300 2023-10-24T23:53:31.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f476c001680 con 0x7f477c0a9300 2023-10-24T23:53:31.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 >> v1:172.21.15.107:6789/0 conn(0x7f477c0a9300 legacy=0x7f477c0aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4764005b90 0x7f4764005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47640064f0 0x7f47640069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47640070d0 0x7f476400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c0a9b80 con 0x7f47640070d0 2023-10-24T23:53:31.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c0aa0b0 con 0x7f47640064f0 2023-10-24T23:53:31.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f476400e0e0 con 0x7f4764005b90 2023-10-24T23:53:31.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477b59e640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47640070d0 0x7f476400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477a59c640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47640064f0 0x7f47640069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477ad9d640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4764005b90 0x7f4764005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477b59e640 1 -- 172.21.15.107:0/64904472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4764005b90 msgr2=0x7f4764005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477b59e640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4764005b90 0x7f4764005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477b59e640 1 -- 172.21.15.107:0/64904472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47640064f0 msgr2=0x7f47640069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477b59e640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47640064f0 0x7f47640069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.206+0000 7f477b59e640 1 -- 172.21.15.107:0/64904472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4764005740 con 0x7f47640070d0 2023-10-24T23:53:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f477b59e640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47640070d0 0x7f476400dba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f477c06ec20 tx=0x7f477401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36b862b62c0d6e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740a58e0 con 0x7f47640070d0 2023-10-24T23:53:31.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477400f040 con 0x7f47640070d0 2023-10-24T23:53:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f4779d9b640 1 -- 172.21.15.107:0/64904472 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740a5c80 con 0x7f47640070d0 2023-10-24T23:53:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f4781200640 1 -- 172.21.15.107:0/64904472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47640070d0 msgr2=0x7f476400dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f4781200640 1 --2- 172.21.15.107:0/64904472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47640070d0 0x7f476400dba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f477c06ec20 tx=0x7f477401cc00 comp rx=0 tx=0).stop 2023-10-24T23:53:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f477b59e640 1 -- 172.21.15.107:0/64904472 reap_dead start 2023-10-24T23:53:31.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f4781200640 1 -- 172.21.15.107:0/64904472 shutdown_connections 2023-10-24T23:53:31.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f4781200640 1 -- 172.21.15.107:0/64904472 >> 172.21.15.107:0/64904472 conn(0x7f477c0c2860 msgr2=0x7f477c0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:31.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f4781200640 1 -- 172.21.15.107:0/64904472 shutdown_connections 2023-10-24T23:53:31.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f4781200640 1 -- 172.21.15.107:0/64904472 wait complete. 2023-10-24T23:53:31.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f4781200640 1 Processor -- start 2023-10-24T23:53:31.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.210+0000 7f4781200640 1 -- start start 2023-10-24T23:53:31.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f4781200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f477c154a80 0x7f477c1ec5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f4781200640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f477c1e8c90 0x7f477c1e9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f4781200640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f477c070df0 0x7f477c0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f4781200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c1e9650 con 0x7f477c154a80 2023-10-24T23:53:31.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f4781200640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477c06a050 con 0x7f477c070df0 2023-10-24T23:53:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f4781200640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477c06a330 con 0x7f477c1e8c90 2023-10-24T23:53:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f477c154a80 0x7f477c1ec5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f477c154a80 0x7f477c1ec5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33548/0 (socket says 172.21.15.107:33548) 2023-10-24T23:53:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477ad9d640 1 -- 172.21.15.107:0/3601352878 learned_addr learned my addr 172.21.15.107:0/3601352878 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:31.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477a59c640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f477c1e8c90 0x7f477c1e9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477b59e640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f477c070df0 0x7f477c0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477ad9d640 1 -- 172.21.15.107:0/3601352878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f477c1e8c90 msgr2=0x7f477c1e9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477ad9d640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f477c1e8c90 0x7f477c1e9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477ad9d640 1 -- 172.21.15.107:0/3601352878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f477c070df0 msgr2=0x7f477c0712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477ad9d640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f477c070df0 0x7f477c0712a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477ad9d640 1 -- 172.21.15.107:0/3601352878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477c06a610 con 0x7f477c154a80 2023-10-24T23:53:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477ad9d640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f477c154a80 0x7f477c1ec5e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f47700115e0 tx=0x7f4770011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f47637fe640 1 -- 172.21.15.107:0/3601352878 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477000f8f0 con 0x7f477c154a80 2023-10-24T23:53:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f47637fe640 1 -- 172.21.15.107:0/3601352878 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477000fa50 con 0x7f477c154a80 2023-10-24T23:53:31.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f4781200640 1 -- 172.21.15.107:0/3601352878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477c1e8770 con 0x7f477c154a80 2023-10-24T23:53:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f47637fe640 1 -- 172.21.15.107:0/3601352878 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477000fdc0 con 0x7f477c154a80 2023-10-24T23:53:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f4781200640 1 -- 172.21.15.107:0/3601352878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477c1e9ad0 con 0x7f477c154a80 2023-10-24T23:53:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f47637fe640 1 -- 172.21.15.107:0/3601352878 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4770015460 con 0x7f477c154a80 2023-10-24T23:53:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f47637fe640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4768065e90 0x7f4768068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f47637fe640 1 -- 172.21.15.107:0/3601352878 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f47700843a0 con 0x7f477c154a80 2023-10-24T23:53:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477a59c640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4768065e90 0x7f4768068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f477a59c640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4768065e90 0x7f4768068350 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f476c004850 tx=0x7f476c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f4784328000 1 Processor -- start 2023-10-24T23:53:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.214+0000 7f4784328000 1 -- start start 2023-10-24T23:53:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd1b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1b750 0x55c55cd1bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 0x55c55cd65060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55cbe5fc0 con 0x55c55cd1ae30 2023-10-24T23:53:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55cb741a0 con 0x55c55cd1b750 2023-10-24T23:53:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c55cb71770 con 0x55c55cd1c070 2023-10-24T23:53:31.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd1b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd1b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33552/0 (socket says 172.21.15.107:33552) 2023-10-24T23:53:31.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 -- 172.21.15.107:0/1904618774 learned_addr learned my addr 172.21.15.107:0/1904618774 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:31.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 -- 172.21.15.107:0/1904618774 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 msgr2=0x55c55cd65060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 --2- 172.21.15.107:0/1904618774 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 0x55c55cd65060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 -- 172.21.15.107:0/1904618774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1b750 msgr2=0x55c55cd1bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 --2- 172.21.15.107:0/1904618774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1b750 0x55c55cd1bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 -- 172.21.15.107:0/1904618774 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c55c869890 con 0x55c55cd1ae30 2023-10-24T23:53:31.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 --2- 172.21.15.107:0/1904618774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd1b210 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f47700824f0 tx=0x7f4770054e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa19dea9e86e16d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4760ff9640 1 -- 172.21.15.107:0/1904618774 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4770059240 con 0x55c55cd1ae30 2023-10-24T23:53:31.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4760ff9640 1 -- 172.21.15.107:0/1904618774 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477005d3f0 con 0x55c55cd1ae30 2023-10-24T23:53:31.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4760ff9640 1 -- 172.21.15.107:0/1904618774 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47700840b0 con 0x55c55cd1ae30 2023-10-24T23:53:31.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- 172.21.15.107:0/1904618774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 msgr2=0x55c55cd1b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 --2- 172.21.15.107:0/1904618774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd1b210 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f47700824f0 tx=0x7f4770054e80 comp rx=0 tx=0).stop 2023-10-24T23:53:31.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- 172.21.15.107:0/1904618774 shutdown_connections 2023-10-24T23:53:31.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 --2- 172.21.15.107:0/1904618774 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 0x55c55cd65060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 --2- 172.21.15.107:0/1904618774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1b750 0x55c55cd1bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 --2- 172.21.15.107:0/1904618774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd1b210 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- 172.21.15.107:0/1904618774 >> 172.21.15.107:0/1904618774 conn(0x55c55cd58dc0 msgr2=0x55c55cd5b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:31.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- 172.21.15.107:0/1904618774 shutdown_connections 2023-10-24T23:53:31.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- 172.21.15.107:0/1904618774 wait complete. 2023-10-24T23:53:31.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- start start 2023-10-24T23:53:31.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 Processor -- start 2023-10-24T23:53:31.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd23930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1b750 0x55c55cd690c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 0x55c55cd6b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55cb6d060 con 0x55c55cd1ae30 2023-10-24T23:53:31.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55cb0c290 con 0x55c55cd1b750 2023-10-24T23:53:31.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c55cb2a6f0 con 0x55c55cd1c070 2023-10-24T23:53:31.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd23930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 10 client.0 _ll_register_callbacks cb 0x55c55cd091d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-24T23:53:31.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd23930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33566/0 (socket says 172.21.15.107:33566) 2023-10-24T23:53:31.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 -- 172.21.15.107:0/1832549251 learned_addr learned my addr 172.21.15.107:0/1832549251 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:31.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 -- 172.21.15.107:0/1832549251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 msgr2=0x55c55cd6b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 --2- 172.21.15.107:0/1832549251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 0x55c55cd6b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 -- 172.21.15.107:0/1832549251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1b750 msgr2=0x55c55cd690c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 --2- 172.21.15.107:0/1832549251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1b750 0x55c55cd690c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 -- 172.21.15.107:0/1832549251 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477c1e9ad0 con 0x55c55cd1ae30 2023-10-24T23:53:31.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f477ad9d640 1 --2- 172.21.15.107:0/1832549251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd23930 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f477001c000 tx=0x7f4770084040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5128e897e4da5dc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4759ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-10-24T23:53:31.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4759ffb640 1 -- 172.21.15.107:0/1832549251 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4770073200 con 0x55c55cd1ae30 2023-10-24T23:53:31.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4759ffb640 1 -- 172.21.15.107:0/1832549251 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f4770068e40 con 0x55c55cd1ae30 2023-10-24T23:53:31.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.218+0000 7f4784328000 1 -- 172.21.15.107:0/1832549251 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c55cb6b9b0 con 0x55c55cd1ae30 2023-10-24T23:53:31.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4759ffb640 1 -- 172.21.15.107:0/1832549251 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f477005d3f0 con 0x55c55cd1ae30 2023-10-24T23:53:31.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 10 client.4851 fetch_fsmap learned FSMap version 12 2023-10-24T23:53:31.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 -- 172.21.15.107:0/1832549251 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55c55cb321e0 con 0x55c55cd1ae30 2023-10-24T23:53:31.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4759ffb640 1 -- 172.21.15.107:0/1832549251 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c55cb6b9b0 con 0x55c55cd1ae30 2023-10-24T23:53:31.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4759ffb640 1 -- 172.21.15.107:0/1832549251 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 12) v1 ==== 3652+0+0 (secure 0 0 0) 0x7f477001b020 con 0x55c55cd1ae30 2023-10-24T23:53:31.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 10 client.4851 fetch_fsmap finished waiting for FSMap version 12 2023-10-24T23:53:31.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 10 client.4851 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-24T23:53:31.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 20 client.4851 populate_metadata read hostname 'smithi107' 2023-10-24T23:53:31.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 --2- 172.21.15.107:0/1832549251 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd6e320 0x55c55cd70710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 4 client.4851 mds_command: new command op to 4299 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-24T23:53:31.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 -- 172.21.15.107:0/1832549251 --> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c55cb9c4d0 con 0x55c55cd6e320 2023-10-24T23:53:31.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f477a59c640 1 --2- 172.21.15.107:0/1832549251 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd6e320 0x55c55cd70710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4759ffb640 10 client.4851 ms_handle_connect on v2:172.21.15.110:6833/2480841919 2023-10-24T23:53:31.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f477a59c640 1 --2- 172.21.15.107:0/1832549251 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd6e320 0x55c55cd70710 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2071b1518ed5d77 server_cookie=6f4abc2b69e6617a in_seq=0 out_seq=0 2023-10-24T23:53:31.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4759ffb640 1 -- 172.21.15.107:0/1832549251 <== mds.0 v2:172.21.15.110:6833/2480841919 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55c55cb9c4d0 con 0x55c55cd6e320 2023-10-24T23:53:31.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4759ffb640 10 client.4851 handle_command_reply: tid=0 2023-10-24T23:53:31.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 client.4851 shutdown 2023-10-24T23:53:31.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 -- 172.21.15.107:0/1832549251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 msgr2=0x55c55cd23930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 --2- 172.21.15.107:0/1832549251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd23930 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f477001c000 tx=0x7f4770084040 comp rx=0 tx=0).stop 2023-10-24T23:53:31.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 -- 172.21.15.107:0/1832549251 shutdown_connections 2023-10-24T23:53:31.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 --2- 172.21.15.107:0/1832549251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 0x55c55cd6b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 --2- 172.21.15.107:0/1832549251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1b750 0x55c55cd690c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 --2- 172.21.15.107:0/1832549251 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd6e320 0x55c55cd70710 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 --2- 172.21.15.107:0/1832549251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd23930 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4759ffb640 0 client.4851 ms_handle_reset on v2:172.21.15.110:6833/2480841919 2023-10-24T23:53:31.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 -- 172.21.15.107:0/1832549251 >> 172.21.15.107:0/1832549251 conn(0x55c55cd58dc0 msgr2=0x55c55cd1a520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:31.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 -- 172.21.15.107:0/1832549251 shutdown_connections 2023-10-24T23:53:31.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 1 -- 172.21.15.107:0/1832549251 wait complete. 2023-10-24T23:53:31.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.222+0000 7f4784328000 20 client.4851 trim_cache size 0 max 16384 2023-10-24T23:53:31.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f4784328000 1 Processor -- start 2023-10-24T23:53:31.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f4784328000 1 -- start start 2023-10-24T23:53:31.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f4784328000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd69e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f4784328000 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1b750 0x55c55cd6a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f4784328000 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1c070 0x55c55cd6a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f4784328000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55c82a940 con 0x55c55cd1ae30 2023-10-24T23:53:31.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f4784328000 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55cb6d910 con 0x55c55cd1c070 2023-10-24T23:53:31.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f4784328000 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c55cb3bec0 con 0x55c55cd1b750 2023-10-24T23:53:31.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f477ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd69e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f477ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd69e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33576/0 (socket says 172.21.15.107:33576) 2023-10-24T23:53:31.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f477ad9d640 1 -- 172.21.15.107:0/1986790261 learned_addr learned my addr 172.21.15.107:0/1986790261 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:31.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f477ad9d640 1 -- 172.21.15.107:0/1986790261 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1b750 msgr2=0x55c55cd6a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f477ad9d640 1 --2- 172.21.15.107:0/1986790261 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1b750 0x55c55cd6a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f477ad9d640 1 -- 172.21.15.107:0/1986790261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1c070 msgr2=0x55c55cd6a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f477ad9d640 1 --2- 172.21.15.107:0/1986790261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1c070 0x55c55cd6a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f477ad9d640 1 -- 172.21.15.107:0/1986790261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c55cb18330 con 0x55c55cd1ae30 2023-10-24T23:53:31.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f477ad9d640 1 --2- 172.21.15.107:0/1986790261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd69e60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f47700822f0 tx=0x7f4770080000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6a27d23f119eb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f47617fa640 1 -- 172.21.15.107:0/1986790261 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4770073200 con 0x55c55cd1ae30 2023-10-24T23:53:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f47617fa640 1 -- 172.21.15.107:0/1986790261 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477005d3f0 con 0x55c55cd1ae30 2023-10-24T23:53:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.238+0000 7f47617fa640 1 -- 172.21.15.107:0/1986790261 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477005cf60 con 0x55c55cd1ae30 2023-10-24T23:53:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- 172.21.15.107:0/1986790261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 msgr2=0x55c55cd69e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 --2- 172.21.15.107:0/1986790261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd69e60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f47700822f0 tx=0x7f4770080000 comp rx=0 tx=0).stop 2023-10-24T23:53:31.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- 172.21.15.107:0/1986790261 shutdown_connections 2023-10-24T23:53:31.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 --2- 172.21.15.107:0/1986790261 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1c070 0x55c55cd6a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 --2- 172.21.15.107:0/1986790261 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1b750 0x55c55cd6a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 --2- 172.21.15.107:0/1986790261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1ae30 0x55c55cd69e60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- 172.21.15.107:0/1986790261 >> 172.21.15.107:0/1986790261 conn(0x55c55cd58dc0 msgr2=0x55c55cd19590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:31.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- 172.21.15.107:0/1986790261 shutdown_connections 2023-10-24T23:53:31.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- 172.21.15.107:0/1986790261 wait complete. 2023-10-24T23:53:31.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- start start 2023-10-24T23:53:31.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 Processor -- start 2023-10-24T23:53:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1ae30 0x55c55cd71940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1b750 0x55c55cd7f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 0x55c55cd7fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55cb6e5f0 con 0x55c55cd1b750 2023-10-24T23:53:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c55cbb0560 con 0x55c55cd1ae30 2023-10-24T23:53:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c55cb126f0 con 0x55c55cd1c070 2023-10-24T23:53:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 10 client.0 _ll_register_callbacks cb 0x55c55cc0c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-24T23:53:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1b750 0x55c55cd7f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1b750 0x55c55cd7f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33588/0 (socket says 172.21.15.107:33588) 2023-10-24T23:53:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477a59c640 1 -- 172.21.15.107:0/3786501636 learned_addr learned my addr 172.21.15.107:0/3786501636 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477a59c640 1 -- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 msgr2=0x55c55cd7fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477a59c640 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 0x55c55cd7fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477a59c640 1 -- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1ae30 msgr2=0x55c55cd71940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477a59c640 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1ae30 0x55c55cd71940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477a59c640 1 -- 172.21.15.107:0/3786501636 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c55cb321e0 con 0x55c55cd1b750 2023-10-24T23:53:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477a59c640 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1b750 0x55c55cd7f800 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f476c010be0 tx=0x7f476c0157e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d032c9a22bf6a94 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4759ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-10-24T23:53:31.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4759ffb640 1 -- 172.21.15.107:0/3786501636 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476c010e90 con 0x55c55cd1b750 2023-10-24T23:53:31.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- 172.21.15.107:0/3786501636 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c55cb3cb90 con 0x55c55cd1b750 2023-10-24T23:53:31.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4759ffb640 1 -- 172.21.15.107:0/3786501636 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f476c0227a0 con 0x55c55cd1b750 2023-10-24T23:53:31.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4759ffb640 1 -- 172.21.15.107:0/3786501636 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f476c022960 con 0x55c55cd1b750 2023-10-24T23:53:31.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 10 client.4857 fetch_fsmap learned FSMap version 12 2023-10-24T23:53:31.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- 172.21.15.107:0/3786501636 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55c55c869cc0 con 0x55c55cd1b750 2023-10-24T23:53:31.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4759ffb640 1 -- 172.21.15.107:0/3786501636 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c55cb3cb90 con 0x55c55cd1b750 2023-10-24T23:53:31.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4759ffb640 1 -- 172.21.15.107:0/3786501636 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 12) v1 ==== 3652+0+0 (secure 0 0 0) 0x7f476c021040 con 0x55c55cd1b750 2023-10-24T23:53:31.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 10 client.4857 fetch_fsmap finished waiting for FSMap version 12 2023-10-24T23:53:31.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 10 client.4857 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-24T23:53:31.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 20 client.4857 populate_metadata read hostname 'smithi107' 2023-10-24T23:53:31.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd80280 0x55c55cdc2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 4 client.4857 mds_command: new command op to 4299 tid=0[{"prefix": "status"}] 2023-10-24T23:53:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f4784328000 1 -- 172.21.15.107:0/3786501636 --> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c55cb660e0 con 0x55c55cd80280 2023-10-24T23:53:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477ad9d640 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd80280 0x55c55cdc2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477ad9d640 1 -- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd80280 msgr2=0x55c55cdc2bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2023-10-24T23:53:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477ad9d640 1 -- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd80280 msgr2=0x55c55cdc2bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-24T23:53:31.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477ad9d640 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd80280 0x55c55cdc2bf0 crc :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-24T23:53:31.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.242+0000 7f477ad9d640 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd80280 0x55c55cdc2bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-24T23:53:31.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f477ad9d640 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd80280 0x55c55cdc2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f477ad9d640 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd80280 0x55c55cdc2bf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0c1883d60ac40e6 server_cookie=e5c22b3c4643f127 in_seq=0 out_seq=0 2023-10-24T23:53:31.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4759ffb640 10 client.4857 ms_handle_connect on v2:172.21.15.110:6833/2480841919 2023-10-24T23:53:31.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4759ffb640 1 -- 172.21.15.107:0/3786501636 <== mds.0 v2:172.21.15.110:6833/2480841919 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55c55cb660e0 con 0x55c55cd80280 2023-10-24T23:53:31.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4759ffb640 10 client.4857 handle_command_reply: tid=0 2023-10-24T23:53:31.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 1 client.4857 shutdown 2023-10-24T23:53:31.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 1 -- 172.21.15.107:0/3786501636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1b750 msgr2=0x55c55cd7f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1b750 0x55c55cd7f800 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f476c010be0 tx=0x7f476c0157e0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 1 -- 172.21.15.107:0/3786501636 shutdown_connections 2023-10-24T23:53:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55c55cd1c070 0x55c55cd7fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:6833/2480841919,v1:172.21.15.110:6835/2480841919] conn(0x55c55cd80280 0x55c55cdc2bf0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c55cd1b750 0x55c55cd7f800 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 1 --2- 172.21.15.107:0/3786501636 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55c55cd1ae30 0x55c55cd71940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4759ffb640 0 client.4857 ms_handle_reset on v2:172.21.15.110:6833/2480841919 2023-10-24T23:53:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 1 -- 172.21.15.107:0/3786501636 >> 172.21.15.107:0/3786501636 conn(0x55c55cd58dc0 msgr2=0x55c55cd19e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 1 -- 172.21.15.107:0/3786501636 shutdown_connections 2023-10-24T23:53:31.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 1 -- 172.21.15.107:0/3786501636 wait complete. 2023-10-24T23:53:31.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4784328000 20 client.4857 trim_cache size 0 max 16384 2023-10-24T23:53:31.455 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-10-24T23:53:31.455 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "c963b8f9-0f03-467c-8a02-12930711a2e8", 2023-10-24T23:53:31.455 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-10-24T23:53:31.456 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 4299, 2023-10-24T23:53:31.456 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-10-24T23:53:31.456 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-10-24T23:53:31.456 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "a", 2023-10-24T23:53:31.456 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 28.766743887000001, 2023-10-24T23:53:31.456 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 8, 2023-10-24T23:53:31.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 27, 2023-10-24T23:53:31.457 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 23, 2023-10-24T23:53:31.457 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 33.173343207999999 2023-10-24T23:53:31.457 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-10-24T23:53:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 -- 172.21.15.107:0/3601352878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4768065e90 msgr2=0x7f4768068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4768065e90 0x7f4768068350 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f476c004850 tx=0x7f476c009040 comp rx=0 tx=0).stop 2023-10-24T23:53:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 -- 172.21.15.107:0/3601352878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f477c154a80 msgr2=0x7f477c1ec5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f477c154a80 0x7f477c1ec5e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f47700115e0 tx=0x7f4770011aa0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 -- 172.21.15.107:0/3601352878 shutdown_connections 2023-10-24T23:53:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f477c070df0 0x7f477c0712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4768065e90 0x7f4768068350 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f477c1e8c90 0x7f477c1e9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 --2- 172.21.15.107:0/3601352878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f477c154a80 0x7f477c1ec5e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 -- 172.21.15.107:0/3601352878 >> 172.21.15.107:0/3601352878 conn(0x7f477c0c2860 msgr2=0x7f477c0bb920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 -- 172.21.15.107:0/3601352878 shutdown_connections 2023-10-24T23:53:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.446+0000 7f4781200640 1 -- 172.21.15.107:0/3601352878 wait complete. 2023-10-24T23:53:31.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:31.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.602+0000 7fb10843f640 1 Processor -- start 2023-10-24T23:53:31.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.602+0000 7fb10843f640 1 -- start start 2023-10-24T23:53:31.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb10843f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb10014fd40 0x7fb1000ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb10843f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10006f730 con 0x7fb1000a9820 2023-10-24T23:53:31.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb10843f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb100156a40 con 0x7fb10006ee40 2023-10-24T23:53:31.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb10843f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb100156710 con 0x7fb10014fd40 2023-10-24T23:53:31.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1059da640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb1000a9820 0x7fb1000c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38322/0 (socket says 172.21.15.107:38322) 2023-10-24T23:53:31.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1059da640 1 -- 172.21.15.107:0/874548177 learned_addr learned my addr 172.21.15.107:0/874548177 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:31.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1061db640 1 --2- 172.21.15.107:0/874548177 >> v2:172.21.15.110:3300/0 conn(0x7fb10014fd40 0x7fb1000ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870386602 0 0) 0x7fb10006f730 con 0x7fb1000a9820 2023-10-24T23:53:31.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ec003530 con 0x7fb1000a9820 2023-10-24T23:53:31.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2586751259 0 0) 0x7fb0ec003530 con 0x7fb1000a9820 2023-10-24T23:53:31.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 >> v2:172.21.15.110:3300/0 conn(0x7fb10014fd40 msgr2=0x7fb1000ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 --2- 172.21.15.107:0/874548177 >> v2:172.21.15.110:3300/0 conn(0x7fb10014fd40 0x7fb1000ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 >> v1:172.21.15.110:6789/0 conn(0x7fb10006ee40 legacy=0x7fb10015cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10006f980 con 0x7fb1000a9820 2023-10-24T23:53:31.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb0fc001680 con 0x7fb1000a9820 2023-10-24T23:53:31.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 >> v1:172.21.15.107:6789/0 conn(0x7fb1000a9820 legacy=0x7fb1000c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 --2- 172.21.15.107:0/874548177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb0ec003980 0x7fb0ec003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 --2- 172.21.15.107:0/874548177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0ec0047d0 0x7fb0ec004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 --2- 172.21.15.107:0/874548177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0ec0052b0 0x7fb0ec00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb100156710 con 0x7fb0ec0047d0 2023-10-24T23:53:31.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec00c180 con 0x7fb0ec0052b0 2023-10-24T23:53:31.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec00c340 con 0x7fb0ec003980 2023-10-24T23:53:31.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1059da640 1 --2- 172.21.15.107:0/874548177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0ec0047d0 0x7fb0ec004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1059da640 1 -- 172.21.15.107:0/874548177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb0ec003980 msgr2=0x7fb0ec003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1059da640 1 --2- 172.21.15.107:0/874548177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb0ec003980 0x7fb0ec003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1069dc640 1 --2- 172.21.15.107:0/874548177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0ec0052b0 0x7fb0ec00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1059da640 1 -- 172.21.15.107:0/874548177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0ec0052b0 msgr2=0x7fb0ec00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1059da640 1 --2- 172.21.15.107:0/874548177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0ec0052b0 0x7fb0ec00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1059da640 1 -- 172.21.15.107:0/874548177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec003530 con 0x7fb0ec0047d0 2023-10-24T23:53:31.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1069dc640 1 --2- 172.21.15.107:0/874548177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0ec0052b0 0x7fb0ec00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:31.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.606+0000 7fb1069dc640 1 -- 172.21.15.107:0/874548177 reap_dead start 2023-10-24T23:53:31.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1059da640 1 --2- 172.21.15.107:0/874548177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0ec0047d0 0x7fb0ec004c50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00c210 tx=0x7fb0fc01cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80789c715aed5afc server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc005ae0 con 0x7fb0ec0047d0 2023-10-24T23:53:31.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc005c60 con 0x7fb0ec0047d0 2023-10-24T23:53:31.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1051d9640 1 -- 172.21.15.107:0/874548177 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0a5820 con 0x7fb0ec0047d0 2023-10-24T23:53:31.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 -- 172.21.15.107:0/874548177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0ec0047d0 msgr2=0x7fb0ec004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 --2- 172.21.15.107:0/874548177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0ec0047d0 0x7fb0ec004c50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00c210 tx=0x7fb0fc01cc20 comp rx=0 tx=0).stop 2023-10-24T23:53:31.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 -- 172.21.15.107:0/874548177 shutdown_connections 2023-10-24T23:53:31.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 --2- 172.21.15.107:0/874548177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0ec0047d0 0x7fb0ec004c50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 -- 172.21.15.107:0/874548177 >> 172.21.15.107:0/874548177 conn(0x7fb1000c2950 msgr2=0x7fb1000c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:31.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 -- 172.21.15.107:0/874548177 shutdown_connections 2023-10-24T23:53:31.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 -- 172.21.15.107:0/874548177 wait complete. 2023-10-24T23:53:31.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 Processor -- start 2023-10-24T23:53:31.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 -- start start 2023-10-24T23:53:31.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0ec0047d0 0x7fb1001ec8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1001df060 0x7fb1001de860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb10014b700 0x7fb10013d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1000bb280 con 0x7fb1001df060 2023-10-24T23:53:31.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1001ebef0 con 0x7fb0ec0047d0 2023-10-24T23:53:31.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1001ec420 con 0x7fb10014b700 2023-10-24T23:53:31.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1059da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1001df060 0x7fb1001de860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1059da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1001df060 0x7fb1001de860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33610/0 (socket says 172.21.15.107:33610) 2023-10-24T23:53:31.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1059da640 1 -- 172.21.15.107:0/3175461386 learned_addr learned my addr 172.21.15.107:0/3175461386 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:31.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1061db640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0ec0047d0 0x7fb1001ec8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1069dc640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb10014b700 0x7fb10013d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1059da640 1 -- 172.21.15.107:0/3175461386 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb10014b700 msgr2=0x7fb10013d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1059da640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb10014b700 0x7fb10013d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1059da640 1 -- 172.21.15.107:0/3175461386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0ec0047d0 msgr2=0x7fb1001ec8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1059da640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0ec0047d0 0x7fb1001ec8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1059da640 1 -- 172.21.15.107:0/3175461386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1001ece20 con 0x7fb1001df060 2023-10-24T23:53:31.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb1059da640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1001df060 0x7fb1001de860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc01ca50 tx=0x7fb0ec005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb0f6ffd640 1 -- 172.21.15.107:0/3175461386 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0b4070 con 0x7fb1001df060 2023-10-24T23:53:31.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb0f6ffd640 1 -- 172.21.15.107:0/3175461386 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc001480 con 0x7fb1001df060 2023-10-24T23:53:31.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 -- 172.21.15.107:0/3175461386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1001ec140 con 0x7fb1001df060 2023-10-24T23:53:31.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb0f6ffd640 1 -- 172.21.15.107:0/3175461386 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0a5a80 con 0x7fb1001df060 2023-10-24T23:53:31.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.610+0000 7fb10843f640 1 -- 172.21.15.107:0/3175461386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1001de060 con 0x7fb1001df060 2023-10-24T23:53:31.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.614+0000 7fb10843f640 1 -- 172.21.15.107:0/3175461386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1000c24a0 con 0x7fb1001df060 2023-10-24T23:53:31.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.614+0000 7fb0f6ffd640 1 -- 172.21.15.107:0/3175461386 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb0fc0a5be0 con 0x7fb1001df060 2023-10-24T23:53:31.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.614+0000 7fb0f6ffd640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0e4065e90 0x7fb0e4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.614+0000 7fb0f6ffd640 1 -- 172.21.15.107:0/3175461386 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fb0fc11d650 con 0x7fb1001df060 2023-10-24T23:53:31.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.614+0000 7fb1061db640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0e4065e90 0x7fb0e4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.614+0000 7fb1061db640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0e4065e90 0x7fb0e4068350 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb10014f1b0 tx=0x7fb0f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.618+0000 7fb0f6ffd640 1 -- 172.21.15.107:0/3175461386 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb0fc0eaa00 con 0x7fb1001df060 2023-10-24T23:53:31.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.762+0000 7fb10843f640 1 -- 172.21.15.107:0/3175461386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1000746e0 con 0x7fb1001df060 2023-10-24T23:53:31.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.762+0000 7fb0f6ffd640 1 -- 172.21.15.107:0/3175461386 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21531 (secure 0 0 0) 0x7fb0fc0ee020 con 0x7fb1001df060 2023-10-24T23:53:31.768 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:31.768 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:12.781093+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 -- 172.21.15.107:0/3175461386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0e4065e90 msgr2=0x7fb0e4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0e4065e90 0x7fb0e4068350 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb10014f1b0 tx=0x7fb0f0008040 comp rx=0 tx=0).stop 2023-10-24T23:53:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 -- 172.21.15.107:0/3175461386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1001df060 msgr2=0x7fb1001de860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1001df060 0x7fb1001de860 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc01ca50 tx=0x7fb0ec005800 comp rx=0 tx=0).stop 2023-10-24T23:53:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 -- 172.21.15.107:0/3175461386 shutdown_connections 2023-10-24T23:53:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0e4065e90 0x7fb0e4068350 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb10014b700 0x7fb10013d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1001df060 0x7fb1001de860 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 --2- 172.21.15.107:0/3175461386 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0ec0047d0 0x7fb1001ec8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 -- 172.21.15.107:0/3175461386 >> 172.21.15.107:0/3175461386 conn(0x7fb1000c2950 msgr2=0x7fb10014d720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 -- 172.21.15.107:0/3175461386 shutdown_connections 2023-10-24T23:53:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.766+0000 7fb10843f640 1 -- 172.21.15.107:0/3175461386 wait complete. 2023-10-24T23:53:31.786 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-10-24T23:53:09.786122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-10-24T23:53:10.766645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-24T23:53:31.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-24T23:53:31.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f01a5c1d640 1 Processor -- start 2023-10-24T23:53:31.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f01a5c1d640 1 -- start start 2023-10-24T23:53:31.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f01a5c1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f01a00af9b0 0x7f01a00c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f01a5c1d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a00c8180 con 0x7f01a00bd930 2023-10-24T23:53:31.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f01a5c1d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a0155430 con 0x7f01a006b360 2023-10-24T23:53:31.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f01a5c1d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a00c5870 con 0x7f01a00af9b0 2023-10-24T23:53:31.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f019f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f01a00af9b0 0x7f01a00c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f019f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f01a00af9b0 0x7f01a00c6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33618/0 (socket says 172.21.15.107:33618) 2023-10-24T23:53:31.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f019f7fe640 1 -- 172.21.15.107:0/2397849268 learned_addr learned my addr 172.21.15.107:0/2397849268 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:31.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f019ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f01a00bd930 0x7f01a015f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38334/0 (socket says 172.21.15.107:38334) 2023-10-24T23:53:31.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f019f7fe640 1 -- 172.21.15.107:0/2397849268 >> v1:172.21.15.110:6789/0 conn(0x7f01a006b360 legacy=0x7f01a0069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:31.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f019f7fe640 1 -- 172.21.15.107:0/2397849268 >> v1:172.21.15.107:6789/0 conn(0x7f01a00bd930 legacy=0x7f01a015f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f019e7fc640 1 -- 172.21.15.107:0/2397849268 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432117585 0 0) 0x7f01a00c8180 con 0x7f01a00bd930 2023-10-24T23:53:31.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.902+0000 7f019f7fe640 1 -- 172.21.15.107:0/2397849268 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a0155710 con 0x7f01a00af9b0 2023-10-24T23:53:31.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f019f7fe640 1 --2- 172.21.15.107:0/2397849268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a00af9b0 0x7f01a00c6c90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0190009870 tx=0x7f019001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb3042286de9137 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f019e7fc640 1 -- 172.21.15.107:0/2397849268 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01900ac630 con 0x7f01a00af9b0 2023-10-24T23:53:31.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f019e7fc640 1 -- 172.21.15.107:0/2397849268 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01900ac790 con 0x7f01a00af9b0 2023-10-24T23:53:31.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f019e7fc640 1 -- 172.21.15.107:0/2397849268 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01900acb20 con 0x7f01a00af9b0 2023-10-24T23:53:31.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f01a5c1d640 1 -- 172.21.15.107:0/2397849268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a00af9b0 msgr2=0x7f01a00c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f01a5c1d640 1 --2- 172.21.15.107:0/2397849268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a00af9b0 0x7f01a00c6c90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0190009870 tx=0x7f019001c440 comp rx=0 tx=0).stop 2023-10-24T23:53:31.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f01a5c1d640 1 -- 172.21.15.107:0/2397849268 shutdown_connections 2023-10-24T23:53:31.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f01a5c1d640 1 --2- 172.21.15.107:0/2397849268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a00af9b0 0x7f01a00c6c90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f01a5c1d640 1 -- 172.21.15.107:0/2397849268 >> 172.21.15.107:0/2397849268 conn(0x7f01a0071b00 msgr2=0x7f01a00abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:31.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f01a5c1d640 1 -- 172.21.15.107:0/2397849268 shutdown_connections 2023-10-24T23:53:31.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f01a5c1d640 1 -- 172.21.15.107:0/2397849268 wait complete. 2023-10-24T23:53:31.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f01a5c1d640 1 Processor -- start 2023-10-24T23:53:31.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.906+0000 7f01a5c1d640 1 -- start start 2023-10-24T23:53:31.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f01a5c1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01a006b360 0x7f01a00c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f01a5c1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01a00af9b0 0x7f01a00c3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f01a5c1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a00bd930 0x7f01a01f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f01a5c1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00c22e0 con 0x7f01a00bd930 2023-10-24T23:53:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f01a5c1d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a0153a40 con 0x7f01a00af9b0 2023-10-24T23:53:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f01a5c1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00c2530 con 0x7f01a006b360 2023-10-24T23:53:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a00bd930 0x7f01a01f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a00bd930 0x7f01a01f0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33622/0 (socket says 172.21.15.107:33622) 2023-10-24T23:53:31.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019ffff640 1 -- 172.21.15.107:0/4113461566 learned_addr learned my addr 172.21.15.107:0/4113461566 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:31.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019ffff640 1 -- 172.21.15.107:0/4113461566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01a006b360 msgr2=0x7f01a00c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019f7fe640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01a006b360 0x7f01a00c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019effd640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01a00af9b0 0x7f01a00c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019ffff640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01a006b360 0x7f01a00c87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019ffff640 1 -- 172.21.15.107:0/4113461566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01a00af9b0 msgr2=0x7f01a00c3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:31.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019ffff640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01a00af9b0 0x7f01a00c3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:31.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019ffff640 1 -- 172.21.15.107:0/4113461566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01900094d0 con 0x7f01a00bd930 2023-10-24T23:53:31.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019effd640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01a00af9b0 0x7f01a00c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:31.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019ffff640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a00bd930 0x7f01a01f0c50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f019800c750 tx=0x7f019800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f0187fff640 1 -- 172.21.15.107:0/4113461566 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019800ce60 con 0x7f01a00bd930 2023-10-24T23:53:31.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f0187fff640 1 -- 172.21.15.107:0/4113461566 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019800a640 con 0x7f01a00bd930 2023-10-24T23:53:31.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f01a5c1d640 1 -- 172.21.15.107:0/4113461566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a00c2810 con 0x7f01a00bd930 2023-10-24T23:53:31.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f0187fff640 1 -- 172.21.15.107:0/4113461566 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019800a9b0 con 0x7f01a00bd930 2023-10-24T23:53:31.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f01a5c1d640 1 -- 172.21.15.107:0/4113461566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a00c8cf0 con 0x7f01a00bd930 2023-10-24T23:53:31.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f01a5c1d640 1 -- 172.21.15.107:0/4113461566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a01f0970 con 0x7f01a00bd930 2023-10-24T23:53:31.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f0187fff640 1 -- 172.21.15.107:0/4113461566 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f019800ab10 con 0x7f01a00bd930 2023-10-24T23:53:31.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f0187fff640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f018c065e90 0x7f018c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:31.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f0187fff640 1 -- 172.21.15.107:0/4113461566 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f0198082f70 con 0x7f01a00bd930 2023-10-24T23:53:31.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019f7fe640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f018c065e90 0x7f018c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:31.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.910+0000 7f019f7fe640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f018c065e90 0x7f018c068350 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f01900ae000 tx=0x7f0190008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:31.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:31.914+0000 7f0187fff640 1 -- 172.21.15.107:0/4113461566 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f01980b4040 con 0x7f01a00bd930 2023-10-24T23:53:32.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 -- 172.21.15.107:0/4113461566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f01a0066560 con 0x7f01a00bd930 2023-10-24T23:53:32.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f0187fff640 1 -- 172.21.15.107:0/4113461566 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f01980180a0 con 0x7f01a00bd930 2023-10-24T23:53:32.074 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-24T23:53:32.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 -- 172.21.15.107:0/4113461566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f018c065e90 msgr2=0x7f018c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f018c065e90 0x7f018c068350 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f01900ae000 tx=0x7f0190008040 comp rx=0 tx=0).stop 2023-10-24T23:53:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 -- 172.21.15.107:0/4113461566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a00bd930 msgr2=0x7f01a01f0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a00bd930 0x7f01a01f0c50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f019800c750 tx=0x7f019800cc10 comp rx=0 tx=0).stop 2023-10-24T23:53:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 -- 172.21.15.107:0/4113461566 shutdown_connections 2023-10-24T23:53:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a00bd930 0x7f01a01f0c50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01a00af9b0 0x7f01a00c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f018c065e90 0x7f018c068350 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 --2- 172.21.15.107:0/4113461566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01a006b360 0x7f01a00c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 -- 172.21.15.107:0/4113461566 >> 172.21.15.107:0/4113461566 conn(0x7f01a0071b00 msgr2=0x7f01a00c9710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 -- 172.21.15.107:0/4113461566 shutdown_connections 2023-10-24T23:53:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.070+0000 7f01a5c1d640 1 -- 172.21.15.107:0/4113461566 wait complete. 2023-10-24T23:53:32.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-24T23:53:32.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.210+0000 7f83ed869640 1 Processor -- start 2023-10-24T23:53:32.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.210+0000 7f83ed869640 1 -- start start 2023-10-24T23:53:32.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.210+0000 7f83ed869640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f83e80ab1a0 0x7f83e80c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.210+0000 7f83ed869640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f83e80bceb0 0x7f83e815c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.210+0000 7f83ed869640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e80c1ce0 con 0x7f83e8072720 2023-10-24T23:53:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.210+0000 7f83ed869640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e80be1d0 con 0x7f83e80bceb0 2023-10-24T23:53:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.210+0000 7f83ed869640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e80c0240 con 0x7f83e80ab1a0 2023-10-24T23:53:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.210+0000 7f83e77fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f83e80bceb0 0x7f83e815c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.210+0000 7f83e6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f83e80ab1a0 0x7f83e80c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.210+0000 7f83e67fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f83e8072720 0x7f83e80bfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:48112/0 (socket says 172.21.15.107:48112) 2023-10-24T23:53:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.210+0000 7f83e67fc640 1 -- 172.21.15.107:0/2635310271 learned_addr learned my addr 172.21.15.107:0/2635310271 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:32.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.214+0000 7f83e77fe640 1 -- 172.21.15.107:0/2635310271 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f83e80ab1a0 msgr2=0x7f83e80c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.214+0000 7f83e77fe640 1 --2- 172.21.15.107:0/2635310271 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f83e80ab1a0 0x7f83e80c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.214+0000 7f83e77fe640 1 -- 172.21.15.107:0/2635310271 >> v1:172.21.15.110:6789/0 conn(0x7f83e8072720 legacy=0x7f83e80bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.214+0000 7f83e77fe640 1 -- 172.21.15.107:0/2635310271 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e80c0880 con 0x7f83e80bceb0 2023-10-24T23:53:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.214+0000 7f83e5ffb640 1 -- 172.21.15.107:0/2635310271 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246569381 0 0) 0x7f83e80c1ce0 con 0x7f83e8072720 2023-10-24T23:53:32.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.214+0000 7f83e6ffd640 1 --2- 172.21.15.107:0/2635310271 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f83e80ab1a0 0x7f83e80c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:53:32.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e77fe640 1 --2- 172.21.15.107:0/2635310271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83e80bceb0 0x7f83e815c660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f83d80097c0 tx=0x7f83d801c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b3318efe289591c server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e5ffb640 1 -- 172.21.15.107:0/2635310271 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d80ab600 con 0x7f83e80bceb0 2023-10-24T23:53:32.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e5ffb640 1 -- 172.21.15.107:0/2635310271 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d80a7050 con 0x7f83e80bceb0 2023-10-24T23:53:32.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e5ffb640 1 -- 172.21.15.107:0/2635310271 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d80ab980 con 0x7f83e80bceb0 2023-10-24T23:53:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 -- 172.21.15.107:0/2635310271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83e80bceb0 msgr2=0x7f83e815c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 --2- 172.21.15.107:0/2635310271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83e80bceb0 0x7f83e815c660 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f83d80097c0 tx=0x7f83d801c330 comp rx=0 tx=0).stop 2023-10-24T23:53:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 -- 172.21.15.107:0/2635310271 shutdown_connections 2023-10-24T23:53:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 --2- 172.21.15.107:0/2635310271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83e80bceb0 0x7f83e815c660 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 --2- 172.21.15.107:0/2635310271 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f83e80ab1a0 0x7f83e80c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 -- 172.21.15.107:0/2635310271 >> 172.21.15.107:0/2635310271 conn(0x7f83e80c2860 msgr2=0x7f83e80c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:32.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 -- 172.21.15.107:0/2635310271 shutdown_connections 2023-10-24T23:53:32.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 -- 172.21.15.107:0/2635310271 wait complete. 2023-10-24T23:53:32.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 Processor -- start 2023-10-24T23:53:32.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 -- start start 2023-10-24T23:53:32.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83e80ab1a0 0x7f83e814c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f83e80bceb0 0x7f83e80a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83e8153fc0 0x7f83e80a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8148a20 con 0x7f83e8153fc0 2023-10-24T23:53:32.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8153190 con 0x7f83e80ab1a0 2023-10-24T23:53:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83ed869640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e81533e0 con 0x7f83e80bceb0 2023-10-24T23:53:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83e8153fc0 0x7f83e80a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83e8153fc0 0x7f83e80a5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33628/0 (socket says 172.21.15.107:33628) 2023-10-24T23:53:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e77fe640 1 -- 172.21.15.107:0/2868117881 learned_addr learned my addr 172.21.15.107:0/2868117881 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:32.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e6ffd640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83e80ab1a0 0x7f83e814c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e67fc640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f83e80bceb0 0x7f83e80a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e77fe640 1 -- 172.21.15.107:0/2868117881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f83e80bceb0 msgr2=0x7f83e80a64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e77fe640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f83e80bceb0 0x7f83e80a64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e77fe640 1 -- 172.21.15.107:0/2868117881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83e80ab1a0 msgr2=0x7f83e814c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e77fe640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83e80ab1a0 0x7f83e814c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.218+0000 7f83e77fe640 1 -- 172.21.15.107:0/2868117881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e0009420 con 0x7f83e8153fc0 2023-10-24T23:53:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83e77fe640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83e8153fc0 0x7f83e80a5500 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f83d80097c0 tx=0x7f83d80190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83cffff640 1 -- 172.21.15.107:0/2868117881 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8004030 con 0x7f83e8153fc0 2023-10-24T23:53:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83cffff640 1 -- 172.21.15.107:0/2868117881 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d8004190 con 0x7f83e8153fc0 2023-10-24T23:53:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83ed869640 1 -- 172.21.15.107:0/2868117881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d8009420 con 0x7f83e8153fc0 2023-10-24T23:53:32.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83cffff640 1 -- 172.21.15.107:0/2868117881 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8004500 con 0x7f83e8153fc0 2023-10-24T23:53:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83ed869640 1 -- 172.21.15.107:0/2868117881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e81537c0 con 0x7f83e8153fc0 2023-10-24T23:53:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83ed869640 1 -- 172.21.15.107:0/2868117881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e80beff0 con 0x7f83e8153fc0 2023-10-24T23:53:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83cffff640 1 -- 172.21.15.107:0/2868117881 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f83d80b2040 con 0x7f83e8153fc0 2023-10-24T23:53:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83cffff640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83d4065e60 0x7f83d4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83cffff640 1 -- 172.21.15.107:0/2868117881 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f83d811dc40 con 0x7f83e8153fc0 2023-10-24T23:53:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83e6ffd640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83d4065e60 0x7f83d4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.222+0000 7f83e6ffd640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83d4065e60 0x7f83d4068320 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f83e0009c80 tx=0x7f83e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.226+0000 7f83cffff640 1 -- 172.21.15.107:0/2868117881 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f83d80eaff0 con 0x7f83e8153fc0 2023-10-24T23:53:32.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.378+0000 7f83ed869640 1 -- 172.21.15.107:0/2868117881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f83e80bcc20 con 0x7f83e8153fc0 2023-10-24T23:53:32.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.378+0000 7f83cffff640 1 -- 172.21.15.107:0/2868117881 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f83d80bc350 con 0x7f83e8153fc0 2023-10-24T23:53:32.385 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-24T23:53:32.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 -- 172.21.15.107:0/2868117881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83d4065e60 msgr2=0x7f83d4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83d4065e60 0x7f83d4068320 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f83e0009c80 tx=0x7f83e0008040 comp rx=0 tx=0).stop 2023-10-24T23:53:32.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 -- 172.21.15.107:0/2868117881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83e8153fc0 msgr2=0x7f83e80a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83e8153fc0 0x7f83e80a5500 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f83d80097c0 tx=0x7f83d80190d0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 -- 172.21.15.107:0/2868117881 shutdown_connections 2023-10-24T23:53:32.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83e8153fc0 0x7f83e80a5500 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83d4065e60 0x7f83d4068320 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f83e80bceb0 0x7f83e80a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 --2- 172.21.15.107:0/2868117881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f83e80ab1a0 0x7f83e814c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 -- 172.21.15.107:0/2868117881 >> 172.21.15.107:0/2868117881 conn(0x7f83e80c2860 msgr2=0x7f83e806eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:32.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 -- 172.21.15.107:0/2868117881 shutdown_connections 2023-10-24T23:53:32.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.382+0000 7f83ed869640 1 -- 172.21.15.107:0/2868117881 wait complete. 2023-10-24T23:53:32.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-24T23:53:32.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.522+0000 7fc138fc5640 1 Processor -- start 2023-10-24T23:53:32.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.522+0000 7fc138fc5640 1 -- start start 2023-10-24T23:53:32.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc138fc5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1340aa9f0 0x7fc134072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc138fc5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1340befa0 con 0x7fc1340c1b30 2023-10-24T23:53:32.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc138fc5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1340a9b80 con 0x7fc1340a9300 2023-10-24T23:53:32.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc138fc5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1340aa0b0 con 0x7fc1340aa9f0 2023-10-24T23:53:32.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc1340c1b30 0x7fc1340aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38342/0 (socket says 172.21.15.107:38342) 2023-10-24T23:53:32.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13259c640 1 -- 172.21.15.107:0/396564080 learned_addr learned my addr 172.21.15.107:0/396564080 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:32.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc131d9b640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1340aa9f0 0x7fc134072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502367002 0 0) 0x7fc1340befa0 con 0x7fc1340c1b30 2023-10-24T23:53:32.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc11c003530 con 0x7fc1340c1b30 2023-10-24T23:53:32.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1006291628 0 0) 0x7fc11c003530 con 0x7fc1340c1b30 2023-10-24T23:53:32.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1340aa9f0 msgr2=0x7fc134072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:32.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1340aa9f0 0x7fc134072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 >> v1:172.21.15.110:6789/0 conn(0x7fc1340a9300 legacy=0x7fc134155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1340bf1f0 con 0x7fc1340c1b30 2023-10-24T23:53:32.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fc124001680 con 0x7fc1340c1b30 2023-10-24T23:53:32.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 >> v1:172.21.15.107:6789/0 conn(0x7fc1340c1b30 legacy=0x7fc1340aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0039b0 0x7fc11c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c0047d0 0x7fc11c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c005150 0x7fc11c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1340aa0b0 con 0x7fc11c0039b0 2023-10-24T23:53:32.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c00bf80 con 0x7fc11c005150 2023-10-24T23:53:32.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c00c110 con 0x7fc11c0047d0 2023-10-24T23:53:32.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13259c640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0039b0 0x7fc11c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc131d9b640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c0047d0 0x7fc11c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc132d9d640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c005150 0x7fc11c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13259c640 1 -- 172.21.15.107:0/396564080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c0047d0 msgr2=0x7fc11c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:32.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13259c640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c0047d0 0x7fc11c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13259c640 1 -- 172.21.15.107:0/396564080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c005150 msgr2=0x7fc11c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:32.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13259c640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c005150 0x7fc11c00b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13259c640 1 -- 172.21.15.107:0/396564080 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c003530 con 0x7fc11c0039b0 2023-10-24T23:53:32.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13259c640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0039b0 0x7fc11c003dd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c270 tx=0x7fc12401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40a3a6766daf059 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:32.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124005ae0 con 0x7fc11c0039b0 2023-10-24T23:53:32.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.526+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc124005c40 con 0x7fc11c0039b0 2023-10-24T23:53:32.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc13159a640 1 -- 172.21.15.107:0/396564080 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124007070 con 0x7fc11c0039b0 2023-10-24T23:53:32.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 -- 172.21.15.107:0/396564080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0039b0 msgr2=0x7fc11c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 --2- 172.21.15.107:0/396564080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0039b0 0x7fc11c003dd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c270 tx=0x7fc12401cc40 comp rx=0 tx=0).stop 2023-10-24T23:53:32.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc132d9d640 1 -- 172.21.15.107:0/396564080 reap_dead start 2023-10-24T23:53:32.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 -- 172.21.15.107:0/396564080 shutdown_connections 2023-10-24T23:53:32.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 -- 172.21.15.107:0/396564080 >> 172.21.15.107:0/396564080 conn(0x7fc1340c2860 msgr2=0x7fc1340c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:32.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 -- 172.21.15.107:0/396564080 shutdown_connections 2023-10-24T23:53:32.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 -- 172.21.15.107:0/396564080 wait complete. 2023-10-24T23:53:32.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 Processor -- start 2023-10-24T23:53:32.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 -- start start 2023-10-24T23:53:32.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c0039b0 0x7fc13414b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1341503c0 0x7fc134150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc13406ce90 0x7fc1340ba330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1340bc2e0 con 0x7fc1341503c0 2023-10-24T23:53:32.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1341f0f10 con 0x7fc11c0039b0 2023-10-24T23:53:32.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13414bd20 con 0x7fc13406ce90 2023-10-24T23:53:32.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc131d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1341503c0 0x7fc134150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc131d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1341503c0 0x7fc134150dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56738/0 (socket says 172.21.15.107:56738) 2023-10-24T23:53:32.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc131d9b640 1 -- 172.21.15.107:0/3567722655 learned_addr learned my addr 172.21.15.107:0/3567722655 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:32.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc131d9b640 1 -- 172.21.15.107:0/3567722655 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc13406ce90 msgr2=0x7fc1340ba330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc132d9d640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc13406ce90 0x7fc1340ba330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc13259c640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c0039b0 0x7fc13414b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc131d9b640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc13406ce90 0x7fc1340ba330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc131d9b640 1 -- 172.21.15.107:0/3567722655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c0039b0 msgr2=0x7fc13414b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc131d9b640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c0039b0 0x7fc13414b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc131d9b640 1 -- 172.21.15.107:0/3567722655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1341f1400 con 0x7fc1341503c0 2023-10-24T23:53:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc131d9b640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1341503c0 0x7fc134150dc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc1280115e0 tx=0x7fc128011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc122ffd640 1 -- 172.21.15.107:0/3567722655 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128015870 con 0x7fc1341503c0 2023-10-24T23:53:32.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc122ffd640 1 -- 172.21.15.107:0/3567722655 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1280159d0 con 0x7fc1341503c0 2023-10-24T23:53:32.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 -- 172.21.15.107:0/3567722655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13414c030 con 0x7fc1341503c0 2023-10-24T23:53:32.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc122ffd640 1 -- 172.21.15.107:0/3567722655 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128015d40 con 0x7fc1341503c0 2023-10-24T23:53:32.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.530+0000 7fc138fc5640 1 -- 172.21.15.107:0/3567722655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1340b9e10 con 0x7fc1341503c0 2023-10-24T23:53:32.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.534+0000 7fc138fc5640 1 -- 172.21.15.107:0/3567722655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1340b9ac0 con 0x7fc1341503c0 2023-10-24T23:53:32.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.534+0000 7fc122ffd640 1 -- 172.21.15.107:0/3567722655 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc12800f480 con 0x7fc1341503c0 2023-10-24T23:53:32.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.534+0000 7fc122ffd640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc114065e90 0x7fc114068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.534+0000 7fc122ffd640 1 -- 172.21.15.107:0/3567722655 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fc128084240 con 0x7fc1341503c0 2023-10-24T23:53:32.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.534+0000 7fc13259c640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc114065e90 0x7fc114068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.534+0000 7fc13259c640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc114065e90 0x7fc114068350 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc1240ac000 tx=0x7fc124009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:32.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.538+0000 7fc122ffd640 1 -- 172.21.15.107:0/3567722655 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc1280515f0 con 0x7fc1341503c0 2023-10-24T23:53:32.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc138fc5640 1 -- 172.21.15.107:0/3567722655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc13404e130 con 0x7fc1341503c0 2023-10-24T23:53:32.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc122ffd640 1 -- 172.21.15.107:0/3567722655 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7fc128054c10 con 0x7fc1341503c0 2023-10-24T23:53:32.699 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:32.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc138fc5640 1 -- 172.21.15.107:0/3567722655 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc114065e90 msgr2=0x7fc114068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc138fc5640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc114065e90 0x7fc114068350 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc1240ac000 tx=0x7fc124009040 comp rx=0 tx=0).stop 2023-10-24T23:53:32.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc138fc5640 1 -- 172.21.15.107:0/3567722655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1341503c0 msgr2=0x7fc134150dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc138fc5640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1341503c0 0x7fc134150dc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc1280115e0 tx=0x7fc128011aa0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc138fc5640 1 -- 172.21.15.107:0/3567722655 shutdown_connections 2023-10-24T23:53:32.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc138fc5640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc114065e90 0x7fc114068350 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc138fc5640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc13406ce90 0x7fc1340ba330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc138fc5640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1341503c0 0x7fc134150dc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc138fc5640 1 --2- 172.21.15.107:0/3567722655 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c0039b0 0x7fc13414b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.694+0000 7fc138fc5640 1 -- 172.21.15.107:0/3567722655 >> 172.21.15.107:0/3567722655 conn(0x7fc1340c2860 msgr2=0x7fc1340c43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:32.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.698+0000 7fc138fc5640 1 -- 172.21.15.107:0/3567722655 shutdown_connections 2023-10-24T23:53:32.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.698+0000 7fc138fc5640 1 -- 172.21.15.107:0/3567722655 wait complete. 2023-10-24T23:53:32.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-24T23:53:32.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f38931a1640 1 Processor -- start 2023-10-24T23:53:32.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f38931a1640 1 -- start start 2023-10-24T23:53:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f38931a1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f388c071ee0 0x7f388c0a9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f38931a1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c0c8dd0 con 0x7f388c1521e0 2023-10-24T23:53:32.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f38931a1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c06ed10 con 0x7f388c06e360 2023-10-24T23:53:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f38931a1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388c0c2430 con 0x7f388c071ee0 2023-10-24T23:53:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f3890f3d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f388c1521e0 0x7f388c1554f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60190/0 (socket says 172.21.15.107:60190) 2023-10-24T23:53:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f3890f3d640 1 -- 172.21.15.107:0/276758931 learned_addr learned my addr 172.21.15.107:0/276758931 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f388bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f388c071ee0 0x7f388c0a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f388b7fe640 1 -- 172.21.15.107:0/276758931 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496920463 0 0) 0x7f387c0014a0 con 0x7f388c1521e0 2023-10-24T23:53:32.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f388b7fe640 1 -- 172.21.15.107:0/276758931 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3878003530 con 0x7f388c1521e0 2023-10-24T23:53:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f388bfff640 1 -- 172.21.15.107:0/276758931 >> v1:172.21.15.110:6789/0 conn(0x7f388c06e360 legacy=0x7f388c0c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f388bfff640 1 -- 172.21.15.107:0/276758931 >> v1:172.21.15.107:6789/0 conn(0x7f388c1521e0 legacy=0x7f388c1554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.834+0000 7f388bfff640 1 -- 172.21.15.107:0/276758931 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c0a9990 con 0x7f388c071ee0 2023-10-24T23:53:32.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388bfff640 1 --2- 172.21.15.107:0/276758931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f388c071ee0 0x7f388c0a9450 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f38740097c0 tx=0x7f387401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b33eb512ea2d10 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388b7fe640 1 -- 172.21.15.107:0/276758931 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38740ab600 con 0x7f388c071ee0 2023-10-24T23:53:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388b7fe640 1 -- 172.21.15.107:0/276758931 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38740a7050 con 0x7f388c071ee0 2023-10-24T23:53:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388b7fe640 1 -- 172.21.15.107:0/276758931 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38740ab980 con 0x7f388c071ee0 2023-10-24T23:53:32.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 -- 172.21.15.107:0/276758931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f388c071ee0 msgr2=0x7f388c0a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 --2- 172.21.15.107:0/276758931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f388c071ee0 0x7f388c0a9450 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f38740097c0 tx=0x7f387401c3b0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 -- 172.21.15.107:0/276758931 shutdown_connections 2023-10-24T23:53:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 --2- 172.21.15.107:0/276758931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f388c071ee0 0x7f388c0a9450 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 -- 172.21.15.107:0/276758931 >> 172.21.15.107:0/276758931 conn(0x7f388c0c8020 msgr2=0x7f388c0c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:32.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 -- 172.21.15.107:0/276758931 shutdown_connections 2023-10-24T23:53:32.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 -- 172.21.15.107:0/276758931 wait complete. 2023-10-24T23:53:32.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 Processor -- start 2023-10-24T23:53:32.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 -- start start 2023-10-24T23:53:32.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f388c06e360 0x7f388c153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f388c071ee0 0x7f388c06c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f388c1521e0 0x7f388c0c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c0c7340 con 0x7f388c071ee0 2023-10-24T23:53:32.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c0c2bd0 con 0x7f388c1521e0 2023-10-24T23:53:32.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f38931a1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388c0c6ff0 con 0x7f388c06e360 2023-10-24T23:53:32.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f388c071ee0 0x7f388c06c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f388c071ee0 0x7f388c06c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56766/0 (socket says 172.21.15.107:56766) 2023-10-24T23:53:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388bfff640 1 -- 172.21.15.107:0/871317410 learned_addr learned my addr 172.21.15.107:0/871317410 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388bfff640 1 -- 172.21.15.107:0/871317410 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f388c06e360 msgr2=0x7f388c153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388bfff640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f388c06e360 0x7f388c153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388bfff640 1 -- 172.21.15.107:0/871317410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f388c1521e0 msgr2=0x7f388c0c40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f389173e640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f388c1521e0 0x7f388c0c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388bfff640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f388c1521e0 0x7f388c0c40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.838+0000 7f388bfff640 1 -- 172.21.15.107:0/871317410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3874009420 con 0x7f388c071ee0 2023-10-24T23:53:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f388bfff640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f388c071ee0 0x7f388c06c180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3874018fc0 tx=0x7f3874019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f38897fa640 1 -- 172.21.15.107:0/871317410 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38740b4410 con 0x7f388c071ee0 2023-10-24T23:53:32.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f38897fa640 1 -- 172.21.15.107:0/871317410 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38740b4570 con 0x7f388c071ee0 2023-10-24T23:53:32.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f38931a1640 1 -- 172.21.15.107:0/871317410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388c1446b0 con 0x7f388c071ee0 2023-10-24T23:53:32.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f38897fa640 1 -- 172.21.15.107:0/871317410 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38740b4920 con 0x7f388c071ee0 2023-10-24T23:53:32.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f38931a1640 1 -- 172.21.15.107:0/871317410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388c06c6c0 con 0x7f388c071ee0 2023-10-24T23:53:32.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f38931a1640 1 -- 172.21.15.107:0/871317410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388c0c7af0 con 0x7f388c071ee0 2023-10-24T23:53:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f38897fa640 1 -- 172.21.15.107:0/871317410 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f38740b2040 con 0x7f388c071ee0 2023-10-24T23:53:32.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f38897fa640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f386c065e90 0x7f386c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:32.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f38897fa640 1 -- 172.21.15.107:0/871317410 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f38740a4e40 con 0x7f388c071ee0 2023-10-24T23:53:32.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f3890f3d640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f386c065e90 0x7f386c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:32.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.842+0000 7f3890f3d640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f386c065e90 0x7f386c068350 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f387c005c10 tx=0x7f387c0014e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:32.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.846+0000 7f38897fa640 1 -- 172.21.15.107:0/871317410 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f38740eaea0 con 0x7f388c071ee0 2023-10-24T23:53:32.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 -- 172.21.15.107:0/871317410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f388c154cc0 con 0x7f388c071ee0 2023-10-24T23:53:32.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38897fa640 1 -- 172.21.15.107:0/871317410 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7f38740b00a0 con 0x7f388c071ee0 2023-10-24T23:53:32.994 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:32.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 -- 172.21.15.107:0/871317410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f386c065e90 msgr2=0x7f386c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f386c065e90 0x7f386c068350 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f387c005c10 tx=0x7f387c0014e0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 -- 172.21.15.107:0/871317410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f388c071ee0 msgr2=0x7f388c06c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:32.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f388c071ee0 0x7f388c06c180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3874018fc0 tx=0x7f3874019150 comp rx=0 tx=0).stop 2023-10-24T23:53:32.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 -- 172.21.15.107:0/871317410 shutdown_connections 2023-10-24T23:53:32.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f388c1521e0 0x7f388c0c40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f388c071ee0 0x7f388c06c180 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f386c065e90 0x7f386c068350 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 --2- 172.21.15.107:0/871317410 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f388c06e360 0x7f388c153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:32.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 -- 172.21.15.107:0/871317410 >> 172.21.15.107:0/871317410 conn(0x7f388c0c8020 msgr2=0x7f388c1508d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:32.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 -- 172.21.15.107:0/871317410 shutdown_connections 2023-10-24T23:53:32.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:32.990+0000 7f38931a1640 1 -- 172.21.15.107:0/871317410 wait complete. 2023-10-24T23:53:33.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-10-24T23:53:33.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0c27f3640 1 Processor -- start 2023-10-24T23:53:33.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0c27f3640 1 -- start start 2023-10-24T23:53:33.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0c27f3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa0bc0adea0 0x7fa0bc152580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0c27f3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa0bc0beba0 0x7fa0bc06f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0c27f3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0bc0c1d60 con 0x7fa0bc06d9c0 2023-10-24T23:53:33.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0c27f3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0bc0bd630 con 0x7fa0bc0beba0 2023-10-24T23:53:33.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0c27f3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0bc152ac0 con 0x7fa0bc0adea0 2023-10-24T23:53:33.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0bb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa0bc0beba0 0x7fa0bc06f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0bb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa0bc0beba0 0x7fa0bc06f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56770/0 (socket says 172.21.15.107:56770) 2023-10-24T23:53:33.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0bb7fe640 1 -- 172.21.15.107:0/1419342021 learned_addr learned my addr 172.21.15.107:0/1419342021 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:33.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0bb7fe640 1 -- 172.21.15.107:0/1419342021 >> v2:172.21.15.110:3300/0 conn(0x7fa0bc0adea0 msgr2=0x7fa0bc152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:33.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0bb7fe640 1 --2- 172.21.15.107:0/1419342021 >> v2:172.21.15.110:3300/0 conn(0x7fa0bc0adea0 0x7fa0bc152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0bb7fe640 1 -- 172.21.15.107:0/1419342021 >> v1:172.21.15.110:6789/0 conn(0x7fa0bc06d9c0 legacy=0x7fa0bc071a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:33.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.122+0000 7fa0bb7fe640 1 -- 172.21.15.107:0/1419342021 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc0bdb60 con 0x7fa0bc0beba0 2023-10-24T23:53:33.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0bb7fe640 1 --2- 172.21.15.107:0/1419342021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0bc0beba0 0x7fa0bc06f280 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0708e0 tx=0x7fa0a401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c912f772d58df8 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:33.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0baffd640 1 -- 172.21.15.107:0/1419342021 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a40ab600 con 0x7fa0bc0beba0 2023-10-24T23:53:33.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0baffd640 1 -- 172.21.15.107:0/1419342021 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a40ab760 con 0x7fa0bc0beba0 2023-10-24T23:53:33.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0baffd640 1 -- 172.21.15.107:0/1419342021 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a40abaf0 con 0x7fa0bc0beba0 2023-10-24T23:53:33.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 -- 172.21.15.107:0/1419342021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0bc0beba0 msgr2=0x7fa0bc06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 --2- 172.21.15.107:0/1419342021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0bc0beba0 0x7fa0bc06f280 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc0708e0 tx=0x7fa0a401c400 comp rx=0 tx=0).stop 2023-10-24T23:53:33.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 -- 172.21.15.107:0/1419342021 shutdown_connections 2023-10-24T23:53:33.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 --2- 172.21.15.107:0/1419342021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0bc0beba0 0x7fa0bc06f280 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 --2- 172.21.15.107:0/1419342021 >> v2:172.21.15.110:3300/0 conn(0x7fa0bc0adea0 0x7fa0bc152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 -- 172.21.15.107:0/1419342021 >> 172.21.15.107:0/1419342021 conn(0x7fa0bc070ef0 msgr2=0x7fa0bc071300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:33.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 -- 172.21.15.107:0/1419342021 shutdown_connections 2023-10-24T23:53:33.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 -- 172.21.15.107:0/1419342021 wait complete. 2023-10-24T23:53:33.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 Processor -- start 2023-10-24T23:53:33.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 -- start start 2023-10-24T23:53:33.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0bc06d9c0 0x7fa0bc146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0bc0adea0 0x7fa0bc1ecc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0bc0beba0 0x7fa0bc15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc0c63f0 con 0x7fa0bc0adea0 2023-10-24T23:53:33.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc0c7e30 con 0x7fa0bc0beba0 2023-10-24T23:53:33.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc146d80 con 0x7fa0bc06d9c0 2023-10-24T23:53:33.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0bc0adea0 0x7fa0bc1ecc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0bc0adea0 0x7fa0bc1ecc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56784/0 (socket says 172.21.15.107:56784) 2023-10-24T23:53:33.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0bb7fe640 1 -- 172.21.15.107:0/3326500890 learned_addr learned my addr 172.21.15.107:0/3326500890 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:33.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0bb7fe640 1 -- 172.21.15.107:0/3326500890 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0bc06d9c0 msgr2=0x7fa0bc146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0bb7fe640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0bc06d9c0 0x7fa0bc146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0bb7fe640 1 -- 172.21.15.107:0/3326500890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0bc0beba0 msgr2=0x7fa0bc15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0bb7fe640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0bc0beba0 0x7fa0bc15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0bb7fe640 1 -- 172.21.15.107:0/3326500890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a40094d0 con 0x7fa0bc0adea0 2023-10-24T23:53:33.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0bb7fe640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0bc0adea0 0x7fa0bc1ecc80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40ad040 tx=0x7fa0a401ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:33.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0b8ff9640 1 -- 172.21.15.107:0/3326500890 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a40b4800 con 0x7fa0bc0adea0 2023-10-24T23:53:33.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0b8ff9640 1 -- 172.21.15.107:0/3326500890 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a40b4980 con 0x7fa0bc0adea0 2023-10-24T23:53:33.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 -- 172.21.15.107:0/3326500890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc147060 con 0x7fa0bc0adea0 2023-10-24T23:53:33.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0b8ff9640 1 -- 172.21.15.107:0/3326500890 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a40b4d30 con 0x7fa0bc0adea0 2023-10-24T23:53:33.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.126+0000 7fa0c27f3640 1 -- 172.21.15.107:0/3326500890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc1f01f0 con 0x7fa0bc0adea0 2023-10-24T23:53:33.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.130+0000 7fa0c27f3640 1 -- 172.21.15.107:0/3326500890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0bc073c30 con 0x7fa0bc0adea0 2023-10-24T23:53:33.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.130+0000 7fa0b8ff9640 1 -- 172.21.15.107:0/3326500890 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa0a40c7440 con 0x7fa0bc0adea0 2023-10-24T23:53:33.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.130+0000 7fa0b8ff9640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa09c065e60 0x7fa09c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.130+0000 7fa0b8ff9640 1 -- 172.21.15.107:0/3326500890 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fa0a411e670 con 0x7fa0bc0adea0 2023-10-24T23:53:33.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.130+0000 7fa0bbfff640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa09c065e60 0x7fa09c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.130+0000 7fa0bbfff640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa09c065e60 0x7fa09c068320 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00ab90 tx=0x7fa0ac009250 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:33.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.134+0000 7fa0b8ff9640 1 -- 172.21.15.107:0/3326500890 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa0a40eba20 con 0x7fa0bc0adea0 2023-10-24T23:53:33.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.286+0000 7fa0c27f3640 1 -- 172.21.15.107:0/3326500890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa0bc04e130 con 0x7fa0bc0adea0 2023-10-24T23:53:33.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.286+0000 7fa0b8ff9640 1 -- 172.21.15.107:0/3326500890 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7fa0a40ef040 con 0x7fa0bc0adea0 2023-10-24T23:53:33.291 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:33.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 -- 172.21.15.107:0/3326500890 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa09c065e60 msgr2=0x7fa09c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa09c065e60 0x7fa09c068320 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00ab90 tx=0x7fa0ac009250 comp rx=0 tx=0).stop 2023-10-24T23:53:33.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 -- 172.21.15.107:0/3326500890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0bc0adea0 msgr2=0x7fa0bc1ecc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0bc0adea0 0x7fa0bc1ecc80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40ad040 tx=0x7fa0a401ac40 comp rx=0 tx=0).stop 2023-10-24T23:53:33.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 -- 172.21.15.107:0/3326500890 shutdown_connections 2023-10-24T23:53:33.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0bc0beba0 0x7fa0bc15b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0bc0adea0 0x7fa0bc1ecc80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa09c065e60 0x7fa09c068320 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 --2- 172.21.15.107:0/3326500890 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0bc06d9c0 0x7fa0bc146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 -- 172.21.15.107:0/3326500890 >> 172.21.15.107:0/3326500890 conn(0x7fa0bc070ef0 msgr2=0x7fa0bc0c3990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:33.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 -- 172.21.15.107:0/3326500890 shutdown_connections 2023-10-24T23:53:33.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.290+0000 7fa0c27f3640 1 -- 172.21.15.107:0/3326500890 wait complete. 2023-10-24T23:53:33.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-10-24T23:53:33.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841bef4640 1 Processor -- start 2023-10-24T23:53:33.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841bef4640 1 -- start start 2023-10-24T23:53:33.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841bef4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f84140ab1a0 0x7f84140c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841bef4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414072720 0x7f84140bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841bef4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84140c1ce0 con 0x7f84140bceb0 2023-10-24T23:53:33.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841bef4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84140be1d0 con 0x7f84140ab1a0 2023-10-24T23:53:33.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841bef4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84140c0880 con 0x7f8414072720 2023-10-24T23:53:33.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841948f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414072720 0x7f84140bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f8419c90640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f84140ab1a0 0x7f84140c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841948f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414072720 0x7f84140bfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:41894/0 (socket says 172.21.15.107:41894) 2023-10-24T23:53:33.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841948f640 1 -- 172.21.15.107:0/4065957594 learned_addr learned my addr 172.21.15.107:0/4065957594 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:33.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841948f640 1 -- 172.21.15.107:0/4065957594 >> v2:172.21.15.110:3300/0 conn(0x7f84140ab1a0 msgr2=0x7f84140c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:33.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841948f640 1 --2- 172.21.15.107:0/4065957594 >> v2:172.21.15.110:3300/0 conn(0x7f84140ab1a0 0x7f84140c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841948f640 1 -- 172.21.15.107:0/4065957594 >> v1:172.21.15.110:6789/0 conn(0x7f84140bceb0 legacy=0x7f84140c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.422+0000 7f841948f640 1 -- 172.21.15.107:0/4065957594 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84140c1f30 con 0x7f8414072720 2023-10-24T23:53:33.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841948f640 1 --2- 172.21.15.107:0/4065957594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414072720 0x7f84140bfb30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8404009870 tx=0x7f840401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af5acf95287c0121 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:33.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f8418c8e640 1 -- 172.21.15.107:0/4065957594 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040ab600 con 0x7f8414072720 2023-10-24T23:53:33.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f8418c8e640 1 -- 172.21.15.107:0/4065957594 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84040a7050 con 0x7f8414072720 2023-10-24T23:53:33.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f8418c8e640 1 -- 172.21.15.107:0/4065957594 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040ab980 con 0x7f8414072720 2023-10-24T23:53:33.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 -- 172.21.15.107:0/4065957594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414072720 msgr2=0x7f84140bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 --2- 172.21.15.107:0/4065957594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414072720 0x7f84140bfb30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8404009870 tx=0x7f840401c3f0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 -- 172.21.15.107:0/4065957594 shutdown_connections 2023-10-24T23:53:33.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 --2- 172.21.15.107:0/4065957594 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414072720 0x7f84140bfb30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 --2- 172.21.15.107:0/4065957594 >> v2:172.21.15.110:3300/0 conn(0x7f84140ab1a0 0x7f84140c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 -- 172.21.15.107:0/4065957594 >> 172.21.15.107:0/4065957594 conn(0x7f84140c2860 msgr2=0x7f84140c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:33.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 -- 172.21.15.107:0/4065957594 shutdown_connections 2023-10-24T23:53:33.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 -- 172.21.15.107:0/4065957594 wait complete. 2023-10-24T23:53:33.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 Processor -- start 2023-10-24T23:53:33.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 -- start start 2023-10-24T23:53:33.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140ab1a0 0x7f84140a4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84140bceb0 0x7f84140a7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84140a8d30 0x7f8414158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84140c4bc0 con 0x7f84140bceb0 2023-10-24T23:53:33.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84140a4080 con 0x7f84140ab1a0 2023-10-24T23:53:33.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841bef4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84141f0890 con 0x7f84140a8d30 2023-10-24T23:53:33.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841948f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84140bceb0 0x7f84140a7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841948f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84140bceb0 0x7f84140a7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56786/0 (socket says 172.21.15.107:56786) 2023-10-24T23:53:33.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841948f640 1 -- 172.21.15.107:0/3390653635 learned_addr learned my addr 172.21.15.107:0/3390653635 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:33.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f8419c90640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140ab1a0 0x7f84140a4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841a491640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84140a8d30 0x7f8414158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841948f640 1 -- 172.21.15.107:0/3390653635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84140a8d30 msgr2=0x7f8414158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841948f640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84140a8d30 0x7f8414158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841948f640 1 -- 172.21.15.107:0/3390653635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140ab1a0 msgr2=0x7f84140a4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841948f640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140ab1a0 0x7f84140a4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841948f640 1 -- 172.21.15.107:0/3390653635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84040094d0 con 0x7f84140bceb0 2023-10-24T23:53:33.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f8419c90640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140ab1a0 0x7f84140a4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:33.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.426+0000 7f841948f640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84140bceb0 0x7f84140a7050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f84040ae040 tx=0x7f8404019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:33.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.430+0000 7f840a7fc640 1 -- 172.21.15.107:0/3390653635 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040b4440 con 0x7f84140bceb0 2023-10-24T23:53:33.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.430+0000 7f840a7fc640 1 -- 172.21.15.107:0/3390653635 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84040b45a0 con 0x7f84140bceb0 2023-10-24T23:53:33.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.430+0000 7f841bef4640 1 -- 172.21.15.107:0/3390653635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84140c4ea0 con 0x7f84140bceb0 2023-10-24T23:53:33.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.430+0000 7f840a7fc640 1 -- 172.21.15.107:0/3390653635 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040b4950 con 0x7f84140bceb0 2023-10-24T23:53:33.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.430+0000 7f841bef4640 1 -- 172.21.15.107:0/3390653635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84140a7590 con 0x7f84140bceb0 2023-10-24T23:53:33.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.430+0000 7f841bef4640 1 -- 172.21.15.107:0/3390653635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84140beff0 con 0x7f84140bceb0 2023-10-24T23:53:33.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.430+0000 7f840a7fc640 1 -- 172.21.15.107:0/3390653635 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f84040b2070 con 0x7f84140bceb0 2023-10-24T23:53:33.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.430+0000 7f840a7fc640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f8065e60 0x7f83f8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.430+0000 7f840a7fc640 1 -- 172.21.15.107:0/3390653635 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f840411da10 con 0x7f84140bceb0 2023-10-24T23:53:33.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.430+0000 7f8419c90640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f8065e60 0x7f83f8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.430+0000 7f8419c90640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f8065e60 0x7f83f8068320 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8410009ae0 tx=0x7f8410008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:33.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.434+0000 7f840a7fc640 1 -- 172.21.15.107:0/3390653635 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f84040ead30 con 0x7f84140bceb0 2023-10-24T23:53:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.574+0000 7f841bef4640 1 -- 172.21.15.107:0/3390653635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7f841404e130 con 0x7f84140bceb0 2023-10-24T23:53:33.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.574+0000 7f840a7fc640 1 -- 172.21.15.107:0/3390653635 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7f84040ee350 con 0x7f84140bceb0 2023-10-24T23:53:33.580 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 -- 172.21.15.107:0/3390653635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f8065e60 msgr2=0x7f83f8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f8065e60 0x7f83f8068320 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8410009ae0 tx=0x7f8410008040 comp rx=0 tx=0).stop 2023-10-24T23:53:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 -- 172.21.15.107:0/3390653635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84140bceb0 msgr2=0x7f84140a7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84140bceb0 0x7f84140a7050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f84040ae040 tx=0x7f8404019190 comp rx=0 tx=0).stop 2023-10-24T23:53:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 -- 172.21.15.107:0/3390653635 shutdown_connections 2023-10-24T23:53:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f8065e60 0x7f83f8068320 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84140a8d30 0x7f8414158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84140bceb0 0x7f84140a7050 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 --2- 172.21.15.107:0/3390653635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140ab1a0 0x7f84140a4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 -- 172.21.15.107:0/3390653635 >> 172.21.15.107:0/3390653635 conn(0x7f84140c2860 msgr2=0x7f84140c3990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 -- 172.21.15.107:0/3390653635 shutdown_connections 2023-10-24T23:53:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.578+0000 7f841bef4640 1 -- 172.21.15.107:0/3390653635 wait complete. 2023-10-24T23:53:33.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:33.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.710+0000 7f17e8825640 1 Processor -- start 2023-10-24T23:53:33.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.710+0000 7f17e8825640 1 -- start start 2023-10-24T23:53:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e8825640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f17e40c2990 0x7f17e40aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e8825640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f17e40c20b0 0x7f17e40c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e8825640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e40a9aa0 con 0x7f17e4152060 2023-10-24T23:53:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e8825640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e4155480 con 0x7f17e40c20b0 2023-10-24T23:53:33.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e8825640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e4155760 con 0x7f17e40c2990 2023-10-24T23:53:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f17e4152060 0x7f17e40a9f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60206/0 (socket says 172.21.15.107:60206) 2023-10-24T23:53:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e2d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f17e40c20b0 0x7f17e40c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e259c640 1 -- 172.21.15.107:0/3993371633 learned_addr learned my addr 172.21.15.107:0/3993371633 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e1d9b640 1 --2- 172.21.15.107:0/3993371633 >> v2:172.21.15.110:3300/0 conn(0x7f17e40c2990 0x7f17e40aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e2d9d640 1 -- 172.21.15.107:0/3993371633 >> v2:172.21.15.110:3300/0 conn(0x7f17e40c2990 msgr2=0x7f17e40aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e2d9d640 1 --2- 172.21.15.107:0/3993371633 >> v2:172.21.15.110:3300/0 conn(0x7f17e40c2990 0x7f17e40aada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e2d9d640 1 -- 172.21.15.107:0/3993371633 >> v1:172.21.15.107:6789/0 conn(0x7f17e4152060 legacy=0x7f17e40a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e159a640 1 -- 172.21.15.107:0/3993371633 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377538678 0 0) 0x7f17e40a9aa0 con 0x7f17e4152060 2023-10-24T23:53:33.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e2d9d640 1 -- 172.21.15.107:0/3993371633 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e40a9450 con 0x7f17e40c20b0 2023-10-24T23:53:33.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e2d9d640 1 --2- 172.21.15.107:0/3993371633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e40c20b0 0x7f17e40c1480 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f17e4155a40 tx=0x7f17dc01c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4b03ee1ab2f09d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:33.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e159a640 1 -- 172.21.15.107:0/3993371633 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc0a5840 con 0x7f17e40c20b0 2023-10-24T23:53:33.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e159a640 1 -- 172.21.15.107:0/3993371633 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17dc0a59a0 con 0x7f17e40c20b0 2023-10-24T23:53:33.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.714+0000 7f17e159a640 1 -- 172.21.15.107:0/3993371633 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc0a5d30 con 0x7f17e40c20b0 2023-10-24T23:53:33.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 -- 172.21.15.107:0/3993371633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e40c20b0 msgr2=0x7f17e40c1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 --2- 172.21.15.107:0/3993371633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e40c20b0 0x7f17e40c1480 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f17e4155a40 tx=0x7f17dc01c7c0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 -- 172.21.15.107:0/3993371633 shutdown_connections 2023-10-24T23:53:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 --2- 172.21.15.107:0/3993371633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e40c20b0 0x7f17e40c1480 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 --2- 172.21.15.107:0/3993371633 >> v2:172.21.15.110:3300/0 conn(0x7f17e40c2990 0x7f17e40aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 -- 172.21.15.107:0/3993371633 >> 172.21.15.107:0/3993371633 conn(0x7f17e40c4a90 msgr2=0x7f17e40c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:33.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 -- 172.21.15.107:0/3993371633 shutdown_connections 2023-10-24T23:53:33.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 -- 172.21.15.107:0/3993371633 wait complete. 2023-10-24T23:53:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 Processor -- start 2023-10-24T23:53:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 -- start start 2023-10-24T23:53:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17e40c20b0 0x7f17e41f0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e40c2990 0x7f17e41f05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17e4152060 0x7f17e4150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e414b140 con 0x7f17e40c2990 2023-10-24T23:53:33.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e414bc30 con 0x7f17e40c20b0 2023-10-24T23:53:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e41501c0 con 0x7f17e4152060 2023-10-24T23:53:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e40c2990 0x7f17e41f05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e40c2990 0x7f17e41f05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56804/0 (socket says 172.21.15.107:56804) 2023-10-24T23:53:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e1d9b640 1 -- 172.21.15.107:0/136131467 learned_addr learned my addr 172.21.15.107:0/136131467 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:33.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e259c640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17e40c20b0 0x7f17e41f0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e2d9d640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17e4152060 0x7f17e4150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e1d9b640 1 -- 172.21.15.107:0/136131467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17e4152060 msgr2=0x7f17e4150650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e1d9b640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17e4152060 0x7f17e4150650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e1d9b640 1 -- 172.21.15.107:0/136131467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17e40c20b0 msgr2=0x7f17e41f0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e1d9b640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17e40c20b0 0x7f17e41f0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e1d9b640 1 -- 172.21.15.107:0/136131467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc009a70 con 0x7f17e40c2990 2023-10-24T23:53:33.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e1d9b640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e40c2990 0x7f17e41f05a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f17d800d800 tx=0x7f17d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:33.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17d2ffd640 1 -- 172.21.15.107:0/136131467 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d8018070 con 0x7f17e40c2990 2023-10-24T23:53:33.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17d2ffd640 1 -- 172.21.15.107:0/136131467 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d8004020 con 0x7f17e40c2990 2023-10-24T23:53:33.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17d2ffd640 1 -- 172.21.15.107:0/136131467 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d80041d0 con 0x7f17e40c2990 2023-10-24T23:53:33.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 -- 172.21.15.107:0/136131467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e41f0ae0 con 0x7f17e40c2990 2023-10-24T23:53:33.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.718+0000 7f17e8825640 1 -- 172.21.15.107:0/136131467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e414b390 con 0x7f17e40c2990 2023-10-24T23:53:33.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.722+0000 7f17e8825640 1 -- 172.21.15.107:0/136131467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e40746e0 con 0x7f17e40c2990 2023-10-24T23:53:33.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.722+0000 7f17d2ffd640 1 -- 172.21.15.107:0/136131467 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f17d8004330 con 0x7f17e40c2990 2023-10-24T23:53:33.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.722+0000 7f17d2ffd640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17c4065e60 0x7f17c4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:33.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.722+0000 7f17e259c640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17c4065e60 0x7f17c4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.722+0000 7f17d2ffd640 1 -- 172.21.15.107:0/136131467 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f17d8083a60 con 0x7f17e40c2990 2023-10-24T23:53:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.722+0000 7f17e259c640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17c4065e60 0x7f17c4068320 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f17d4000f70 tx=0x7f17d4007c10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:33.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.726+0000 7f17d2ffd640 1 -- 172.21.15.107:0/136131467 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f17d8050d80 con 0x7f17e40c2990 2023-10-24T23:53:33.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:53:33.848 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:53:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.914+0000 7f17e8825640 1 -- 172.21.15.107:0/136131467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17e40c1e90 con 0x7f17e40c2990 2023-10-24T23:53:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.914+0000 7f17d2ffd640 1 -- 172.21.15.107:0/136131467 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f17d80543a0 con 0x7f17e40c2990 2023-10-24T23:53:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:33.921 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:53:33.921 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:53:33.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 -- 172.21.15.107:0/136131467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17c4065e60 msgr2=0x7f17c4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17c4065e60 0x7f17c4068320 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f17d4000f70 tx=0x7f17d4007c10 comp rx=0 tx=0).stop 2023-10-24T23:53:33.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 -- 172.21.15.107:0/136131467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e40c2990 msgr2=0x7f17e41f05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:33.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e40c2990 0x7f17e41f05a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f17d800d800 tx=0x7f17d800dcc0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 -- 172.21.15.107:0/136131467 shutdown_connections 2023-10-24T23:53:33.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17c4065e60 0x7f17c4068320 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17e4152060 0x7f17e4150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e40c2990 0x7f17e41f05a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 --2- 172.21.15.107:0/136131467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17e40c20b0 0x7f17e41f0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 -- 172.21.15.107:0/136131467 >> 172.21.15.107:0/136131467 conn(0x7f17e40c4a90 msgr2=0x7f17e40c2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:33.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 -- 172.21.15.107:0/136131467 shutdown_connections 2023-10-24T23:53:33.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:33.918+0000 7f17e8825640 1 -- 172.21.15.107:0/136131467 wait complete. 2023-10-24T23:53:33.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:34.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.086+0000 7ff596e18640 1 Processor -- start 2023-10-24T23:53:34.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.086+0000 7ff596e18640 1 -- start start 2023-10-24T23:53:34.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff59006a780 0x7ff590154d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5900c24b0 0x7ff590155280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5901557c0 con 0x7ff590151d60 2023-10-24T23:53:34.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5900aadc0 con 0x7ff5900c24b0 2023-10-24T23:53:34.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59006afe0 con 0x7ff59006a780 2023-10-24T23:53:34.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff5953b5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5900c24b0 0x7ff590155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:34.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff5953b5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5900c24b0 0x7ff590155280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56816/0 (socket says 172.21.15.107:56816) 2023-10-24T23:53:34.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff5953b5640 1 -- 172.21.15.107:0/2322987347 learned_addr learned my addr 172.21.15.107:0/2322987347 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff5953b5640 1 -- 172.21.15.107:0/2322987347 >> v2:172.21.15.110:3300/0 conn(0x7ff59006a780 msgr2=0x7ff590154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff5953b5640 1 --2- 172.21.15.107:0/2322987347 >> v2:172.21.15.110:3300/0 conn(0x7ff59006a780 0x7ff590154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff5953b5640 1 -- 172.21.15.107:0/2322987347 >> v1:172.21.15.110:6789/0 conn(0x7ff590151d60 legacy=0x7ff5900a9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:34.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff5953b5640 1 -- 172.21.15.107:0/2322987347 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590154840 con 0x7ff5900c24b0 2023-10-24T23:53:34.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff5953b5640 1 --2- 172.21.15.107:0/2322987347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5900c24b0 0x7ff590155280 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff5900aac50 tx=0x7ff58801c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa7edeeb54d8d37 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:34.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff58f7fe640 1 -- 172.21.15.107:0/2322987347 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5880a5650 con 0x7ff5900c24b0 2023-10-24T23:53:34.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff58f7fe640 1 -- 172.21.15.107:0/2322987347 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5880a57b0 con 0x7ff5900c24b0 2023-10-24T23:53:34.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff58f7fe640 1 -- 172.21.15.107:0/2322987347 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5880a5b40 con 0x7ff5900c24b0 2023-10-24T23:53:34.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 -- 172.21.15.107:0/2322987347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5900c24b0 msgr2=0x7ff590155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 --2- 172.21.15.107:0/2322987347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5900c24b0 0x7ff590155280 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff5900aac50 tx=0x7ff58801c990 comp rx=0 tx=0).stop 2023-10-24T23:53:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 -- 172.21.15.107:0/2322987347 shutdown_connections 2023-10-24T23:53:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 --2- 172.21.15.107:0/2322987347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5900c24b0 0x7ff590155280 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 --2- 172.21.15.107:0/2322987347 >> v2:172.21.15.110:3300/0 conn(0x7ff59006a780 0x7ff590154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 -- 172.21.15.107:0/2322987347 >> 172.21.15.107:0/2322987347 conn(0x7ff5900c2950 msgr2=0x7ff5900c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:34.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 -- 172.21.15.107:0/2322987347 shutdown_connections 2023-10-24T23:53:34.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.090+0000 7ff596e18640 1 -- 172.21.15.107:0/2322987347 wait complete. 2023-10-24T23:53:34.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff596e18640 1 Processor -- start 2023-10-24T23:53:34.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff596e18640 1 -- start start 2023-10-24T23:53:34.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff596e18640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5900c24b0 0x7ff5900c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff596e18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff590151d60 0x7ff5900c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff596e18640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff59006f6e0 0x7ff590146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff596e18640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59006fec0 con 0x7ff590151d60 2023-10-24T23:53:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff596e18640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59006fbe0 con 0x7ff5900c24b0 2023-10-24T23:53:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff596e18640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5901f0330 con 0x7ff59006f6e0 2023-10-24T23:53:34.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff590151d60 0x7ff5900c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:34.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff590151d60 0x7ff5900c4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56820/0 (socket says 172.21.15.107:56820) 2023-10-24T23:53:34.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58ffff640 1 -- 172.21.15.107:0/2735271000 learned_addr learned my addr 172.21.15.107:0/2735271000 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:34.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58ffff640 1 -- 172.21.15.107:0/2735271000 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff59006f6e0 msgr2=0x7ff590146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58ffff640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff59006f6e0 0x7ff590146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58ffff640 1 -- 172.21.15.107:0/2735271000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5900c24b0 msgr2=0x7ff5900c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff594bb4640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5900c24b0 0x7ff5900c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58ffff640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5900c24b0 0x7ff5900c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58ffff640 1 -- 172.21.15.107:0/2735271000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff588009b30 con 0x7ff590151d60 2023-10-24T23:53:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58ffff640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff590151d60 0x7ff5900c4ad0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff58400b550 tx=0x7ff58400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58d7fa640 1 -- 172.21.15.107:0/2735271000 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584010870 con 0x7ff590151d60 2023-10-24T23:53:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58d7fa640 1 -- 172.21.15.107:0/2735271000 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5840109f0 con 0x7ff590151d60 2023-10-24T23:53:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff58d7fa640 1 -- 172.21.15.107:0/2735271000 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584010da0 con 0x7ff590151d60 2023-10-24T23:53:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff596e18640 1 -- 172.21.15.107:0/2735271000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5901f0610 con 0x7ff590151d60 2023-10-24T23:53:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff596e18640 1 -- 172.21.15.107:0/2735271000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5900c44e0 con 0x7ff590151d60 2023-10-24T23:53:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.094+0000 7ff596e18640 1 -- 172.21.15.107:0/2735271000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5901f0190 con 0x7ff590151d60 2023-10-24T23:53:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.098+0000 7ff58d7fa640 1 -- 172.21.15.107:0/2735271000 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff58400f3c0 con 0x7ff590151d60 2023-10-24T23:53:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.098+0000 7ff58d7fa640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff57c065e60 0x7ff57c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.098+0000 7ff58d7fa640 1 -- 172.21.15.107:0/2735271000 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7ff584083ac0 con 0x7ff590151d60 2023-10-24T23:53:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.098+0000 7ff594bb4640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff57c065e60 0x7ff57c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.098+0000 7ff594bb4640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff57c065e60 0x7ff57c068320 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff58000a790 tx=0x7ff580008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.102+0000 7ff58d7fa640 1 -- 172.21.15.107:0/2735271000 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff5901f0190 con 0x7ff590151d60 2023-10-24T23:53:34.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.242+0000 7ff596e18640 1 -- 172.21.15.107:0/2735271000 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5901f0190 con 0x7ff590151d60 2023-10-24T23:53:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.242+0000 7ff58d7fa640 1 -- 172.21.15.107:0/2735271000 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21531 (secure 0 0 0) 0x7ff5901f0190 con 0x7ff590151d60 2023-10-24T23:53:34.249 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:34.249 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:12.781093+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 -- 172.21.15.107:0/2735271000 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff57c065e60 msgr2=0x7ff57c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff57c065e60 0x7ff57c068320 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff58000a790 tx=0x7ff580008040 comp rx=0 tx=0).stop 2023-10-24T23:53:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 -- 172.21.15.107:0/2735271000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff590151d60 msgr2=0x7ff5900c4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff590151d60 0x7ff5900c4ad0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff58400b550 tx=0x7ff58400ba10 comp rx=0 tx=0).stop 2023-10-24T23:53:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 -- 172.21.15.107:0/2735271000 shutdown_connections 2023-10-24T23:53:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff57c065e60 0x7ff57c068320 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff59006f6e0 0x7ff590146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff590151d60 0x7ff5900c4ad0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 --2- 172.21.15.107:0/2735271000 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5900c24b0 0x7ff5900c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 -- 172.21.15.107:0/2735271000 >> 172.21.15.107:0/2735271000 conn(0x7ff5900c2950 msgr2=0x7ff5900c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 -- 172.21.15.107:0/2735271000 shutdown_connections 2023-10-24T23:53:34.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.246+0000 7ff596e18640 1 -- 172.21.15.107:0/2735271000 wait complete. 2023-10-24T23:53:34.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:34.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.374+0000 7f820b907640 1 Processor -- start 2023-10-24T23:53:34.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.374+0000 7f820b907640 1 -- start start 2023-10-24T23:53:34.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f820b907640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f820414b600 0x7f8204151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f820b907640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82040c1860 con 0x7f820406af70 2023-10-24T23:53:34.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f820b907640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8204154d50 con 0x7f820406a230 2023-10-24T23:53:34.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f820b907640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82041524f0 con 0x7f820414b600 2023-10-24T23:53:34.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8208ea2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f820406af70 0x7f820406c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60214/0 (socket says 172.21.15.107:60214) 2023-10-24T23:53:34.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8208ea2640 1 -- 172.21.15.107:0/2734368153 learned_addr learned my addr 172.21.15.107:0/2734368153 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:34.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152156378 0 0) 0x7f82040c1860 con 0x7f820406af70 2023-10-24T23:53:34.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81f0003530 con 0x7f820406af70 2023-10-24T23:53:34.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f82096a3640 1 --2- 172.21.15.107:0/2734368153 >> v2:172.21.15.110:3300/0 conn(0x7f820414b600 0x7f8204151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:34.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4253247739 0 0) 0x7f81f0003530 con 0x7f820406af70 2023-10-24T23:53:34.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 >> v2:172.21.15.110:3300/0 conn(0x7f820414b600 msgr2=0x7f8204151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 --2- 172.21.15.107:0/2734368153 >> v2:172.21.15.110:3300/0 conn(0x7f820414b600 0x7f8204151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 >> v1:172.21.15.110:6789/0 conn(0x7f820406a230 legacy=0x7f82041551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820414b9e0 con 0x7f820406af70 2023-10-24T23:53:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f81f4001680 con 0x7f820406af70 2023-10-24T23:53:34.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 >> v1:172.21.15.107:6789/0 conn(0x7f820406af70 legacy=0x7f820406c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81f0003980 0x7f81f0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 0x7f81f0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81f0005190 0x7f81f000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82041524f0 con 0x7f81f00047d0 2023-10-24T23:53:34.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f000c130 con 0x7f81f0003980 2023-10-24T23:53:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f000c2f0 con 0x7f81f0005190 2023-10-24T23:53:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8208ea2640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 0x7f81f0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f82096a3640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81f0003980 0x7f81f0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:34.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8209ea4640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81f0005190 0x7f81f000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8208ea2640 1 -- 172.21.15.107:0/2734368153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81f0005190 msgr2=0x7f81f000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8208ea2640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81f0005190 0x7f81f000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8208ea2640 1 -- 172.21.15.107:0/2734368153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81f0003980 msgr2=0x7f81f0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8208ea2640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81f0003980 0x7f81f0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8208ea2640 1 -- 172.21.15.107:0/2734368153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f0003530 con 0x7f81f00047d0 2023-10-24T23:53:34.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8209ea4640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81f0005190 0x7f81f000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:34.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8209ea4640 1 -- 172.21.15.107:0/2734368153 reap_dead start 2023-10-24T23:53:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f8208ea2640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 0x7f81f0004c50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f81f400c210 tx=0x7f81f401cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3af57588a6c72f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f40058e0 con 0x7f81f00047d0 2023-10-24T23:53:34.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f4005a40 con 0x7f81f00047d0 2023-10-24T23:53:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.378+0000 7f81fbfff640 1 -- 172.21.15.107:0/2734368153 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f4005e70 con 0x7f81f00047d0 2023-10-24T23:53:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- 172.21.15.107:0/2734368153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 msgr2=0x7f81f0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 0x7f81f0004c50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f81f400c210 tx=0x7f81f401cbf0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- 172.21.15.107:0/2734368153 shutdown_connections 2023-10-24T23:53:34.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 --2- 172.21.15.107:0/2734368153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 0x7f81f0004c50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- 172.21.15.107:0/2734368153 >> 172.21.15.107:0/2734368153 conn(0x7f82040c2950 msgr2=0x7f82040c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:34.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- 172.21.15.107:0/2734368153 shutdown_connections 2023-10-24T23:53:34.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- 172.21.15.107:0/2734368153 wait complete. 2023-10-24T23:53:34.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 Processor -- start 2023-10-24T23:53:34.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- start start 2023-10-24T23:53:34.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 0x7f820414e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8204150ea0 0x7f82041489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82041ee0a0 0x7f82041ee520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820414da10 con 0x7f81f00047d0 2023-10-24T23:53:34.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820414ed50 con 0x7f8204150ea0 2023-10-24T23:53:34.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82041ea5a0 con 0x7f82041ee0a0 2023-10-24T23:53:34.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f82096a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 0x7f820414e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:34.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f82096a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 0x7f820414e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56824/0 (socket says 172.21.15.107:56824) 2023-10-24T23:53:34.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f82096a3640 1 -- 172.21.15.107:0/1580960864 learned_addr learned my addr 172.21.15.107:0/1580960864 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:34.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f8208ea2640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8204150ea0 0x7f82041489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f82096a3640 1 -- 172.21.15.107:0/1580960864 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82041ee0a0 msgr2=0x7f82041ee520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f82096a3640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82041ee0a0 0x7f82041ee520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f82096a3640 1 -- 172.21.15.107:0/1580960864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8204150ea0 msgr2=0x7f82041489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f82096a3640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8204150ea0 0x7f82041489c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f82096a3640 1 -- 172.21.15.107:0/1580960864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82041edb50 con 0x7f81f00047d0 2023-10-24T23:53:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f82096a3640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 0x7f820414e810 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8200011660 tx=0x7f8200011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f81f9ffb640 1 -- 172.21.15.107:0/1580960864 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820000f960 con 0x7f81f00047d0 2023-10-24T23:53:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f81f9ffb640 1 -- 172.21.15.107:0/1580960864 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f820000fac0 con 0x7f81f00047d0 2023-10-24T23:53:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f81f9ffb640 1 -- 172.21.15.107:0/1580960864 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820000fe30 con 0x7f81f00047d0 2023-10-24T23:53:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- 172.21.15.107:0/1580960864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820414fc50 con 0x7f81f00047d0 2023-10-24T23:53:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- 172.21.15.107:0/1580960864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8204150180 con 0x7f81f00047d0 2023-10-24T23:53:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.382+0000 7f820b907640 1 -- 172.21.15.107:0/1580960864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82040746e0 con 0x7f81f00047d0 2023-10-24T23:53:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.386+0000 7f81f9ffb640 1 -- 172.21.15.107:0/1580960864 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8200015460 con 0x7f81f00047d0 2023-10-24T23:53:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.386+0000 7f81f9ffb640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f81e8065e90 0x7f81e8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.386+0000 7f81f9ffb640 1 -- 172.21.15.107:0/1580960864 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f82000840f0 con 0x7f81f00047d0 2023-10-24T23:53:34.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.386+0000 7f8208ea2640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f81e8065e90 0x7f81e8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:34.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.386+0000 7f8208ea2640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f81e8065e90 0x7f81e8068350 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f81f40ab000 tx=0x7f81f40a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:34.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.390+0000 7f81f9ffb640 1 -- 172.21.15.107:0/1580960864 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8200051370 con 0x7f81f00047d0 2023-10-24T23:53:34.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.562+0000 7f820b907640 1 -- 172.21.15.107:0/1580960864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8204069fa0 con 0x7f81f00047d0 2023-10-24T23:53:34.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.562+0000 7f81f9ffb640 1 -- 172.21.15.107:0/1580960864 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f8200054990 con 0x7f81f00047d0 2023-10-24T23:53:34.567 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:34.568 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:53:34.568 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:53:34.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 -- 172.21.15.107:0/1580960864 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f81e8065e90 msgr2=0x7f81e8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f81e8065e90 0x7f81e8068350 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f81f40ab000 tx=0x7f81f40a7070 comp rx=0 tx=0).stop 2023-10-24T23:53:34.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 -- 172.21.15.107:0/1580960864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 msgr2=0x7f820414e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:34.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 0x7f820414e810 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8200011660 tx=0x7f8200011b20 comp rx=0 tx=0).stop 2023-10-24T23:53:34.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 -- 172.21.15.107:0/1580960864 shutdown_connections 2023-10-24T23:53:34.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f81e8065e90 0x7f81e8068350 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82041ee0a0 0x7f82041ee520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8204150ea0 0x7f82041489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 --2- 172.21.15.107:0/1580960864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81f00047d0 0x7f820414e810 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:34.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 -- 172.21.15.107:0/1580960864 >> 172.21.15.107:0/1580960864 conn(0x7f82040c2950 msgr2=0x7f8204149930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:34.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 -- 172.21.15.107:0/1580960864 shutdown_connections 2023-10-24T23:53:34.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:34.566+0000 7f820b907640 1 -- 172.21.15.107:0/1580960864 wait complete. 2023-10-24T23:53:34.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-24T23:53:11.777452+0000', 'modified': '2023-10-24T23:53:14.800231+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'a', 'rank': 0, 'incarnation': 9, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.107:6837/727042497', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 727042497}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 727042497}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'b', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-24T23:53:34.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-24T23:53:34.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-24T23:53:35.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecfb57640 1 Processor -- start 2023-10-24T23:53:35.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecfb57640 1 -- start start 2023-10-24T23:53:35.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecfb57640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6ec806f380 0x7f6ec806ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecfb57640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6ec81566d0 0x7f6ec81635f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecfb57640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ec806d610 con 0x7f6ec815c260 2023-10-24T23:53:35.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecfb57640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec806c5f0 con 0x7f6ec806f380 2023-10-24T23:53:35.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecfb57640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec806dbd0 con 0x7f6ec81566d0 2023-10-24T23:53:35.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecd8f3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6ec806f380 0x7f6ec806ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecd8f3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6ec806f380 0x7f6ec806ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56838/0 (socket says 172.21.15.107:56838) 2023-10-24T23:53:35.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/253326204 learned_addr learned my addr 172.21.15.107:0/253326204 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:35.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ece0f4640 1 --2- 172.21.15.107:0/253326204 >> v2:172.21.15.110:3300/0 conn(0x7f6ec81566d0 0x7f6ec81635f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/253326204 >> v2:172.21.15.110:3300/0 conn(0x7f6ec81566d0 msgr2=0x7f6ec81635f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/253326204 >> v2:172.21.15.110:3300/0 conn(0x7f6ec81566d0 0x7f6ec81635f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/253326204 >> v1:172.21.15.110:6789/0 conn(0x7f6ec815c260 legacy=0x7f6ec8155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.118+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/253326204 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec806d8f0 con 0x7f6ec806f380 2023-10-24T23:53:35.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/253326204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ec806f380 0x7f6ec806ca10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6ec806e820 tx=0x7f6ec001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93bf1bfd80c3ec2 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:35.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecc8f1640 1 -- 172.21.15.107:0/253326204 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec00a5640 con 0x7f6ec806f380 2023-10-24T23:53:35.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecc8f1640 1 -- 172.21.15.107:0/253326204 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec00a57a0 con 0x7f6ec806f380 2023-10-24T23:53:35.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecc8f1640 1 -- 172.21.15.107:0/253326204 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec00a5b30 con 0x7f6ec806f380 2023-10-24T23:53:35.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 -- 172.21.15.107:0/253326204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ec806f380 msgr2=0x7f6ec806ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 --2- 172.21.15.107:0/253326204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ec806f380 0x7f6ec806ca10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6ec806e820 tx=0x7f6ec001c870 comp rx=0 tx=0).stop 2023-10-24T23:53:35.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 -- 172.21.15.107:0/253326204 shutdown_connections 2023-10-24T23:53:35.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 --2- 172.21.15.107:0/253326204 >> v2:172.21.15.110:3300/0 conn(0x7f6ec81566d0 0x7f6ec81635f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 --2- 172.21.15.107:0/253326204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ec806f380 0x7f6ec806ca10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 -- 172.21.15.107:0/253326204 >> 172.21.15.107:0/253326204 conn(0x7f6ec8070ed0 msgr2=0x7f6ec80712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:35.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 -- 172.21.15.107:0/253326204 shutdown_connections 2023-10-24T23:53:35.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 -- 172.21.15.107:0/253326204 wait complete. 2023-10-24T23:53:35.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 Processor -- start 2023-10-24T23:53:35.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 -- start start 2023-10-24T23:53:35.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ec806f380 0x7f6ec81521f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ec81566d0 0x7f6ec814d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ec815c260 0x7f6ec814d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80ab8f0 con 0x7f6ec81566d0 2023-10-24T23:53:35.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec8152730 con 0x7f6ec815c260 2023-10-24T23:53:35.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecfb57640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80ac830 con 0x7f6ec806f380 2023-10-24T23:53:35.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecd0f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ec81566d0 0x7f6ec814d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecd0f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ec81566d0 0x7f6ec814d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56842/0 (socket says 172.21.15.107:56842) 2023-10-24T23:53:35.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecd0f2640 1 -- 172.21.15.107:0/367210103 learned_addr learned my addr 172.21.15.107:0/367210103 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:35.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ec806f380 0x7f6ec81521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ece0f4640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ec815c260 0x7f6ec814d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecd0f2640 1 -- 172.21.15.107:0/367210103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ec806f380 msgr2=0x7f6ec81521f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ec806f380 0x7f6ec81521f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecd0f2640 1 -- 172.21.15.107:0/367210103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ec815c260 msgr2=0x7f6ec814d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ec815c260 0x7f6ec814d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.122+0000 7f6ecd0f2640 1 -- 172.21.15.107:0/367210103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec0009b30 con 0x7f6ec81566d0 2023-10-24T23:53:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ec81566d0 0x7f6ec814d0c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800b410 tx=0x7f6eb800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ebe7fc640 1 -- 172.21.15.107:0/367210103 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8004020 con 0x7f6ec81566d0 2023-10-24T23:53:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ebe7fc640 1 -- 172.21.15.107:0/367210103 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb8004180 con 0x7f6ec81566d0 2023-10-24T23:53:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ebe7fc640 1 -- 172.21.15.107:0/367210103 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8011630 con 0x7f6ec81566d0 2023-10-24T23:53:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ecfb57640 1 -- 172.21.15.107:0/367210103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec80acb10 con 0x7f6ec81566d0 2023-10-24T23:53:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ecfb57640 1 -- 172.21.15.107:0/367210103 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec80ad0d0 con 0x7f6ec81566d0 2023-10-24T23:53:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ebe7fc640 1 -- 172.21.15.107:0/367210103 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6eb8011790 con 0x7f6ec81566d0 2023-10-24T23:53:35.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ebe7fc640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6eac065e60 0x7f6eac068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ebe7fc640 1 -- 172.21.15.107:0/367210103 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f6eb8083880 con 0x7f6ec81566d0 2023-10-24T23:53:35.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6eac065e60 0x7f6eac068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6eac065e60 0x7f6eac068320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00ae000 tx=0x7f6ec00a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:35.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ed2c7f000 1 Processor -- start 2023-10-24T23:53:35.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ed2c7f000 1 -- start start 2023-10-24T23:53:35.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.126+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e032e30 0x55e42e033210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e033750 0x55e42e033b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e07d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e42defdfc0 con 0x55e42e034070 2023-10-24T23:53:35.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e42de8c1a0 con 0x55e42e032e30 2023-10-24T23:53:35.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e42de89770 con 0x55e42e033750 2023-10-24T23:53:35.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e07d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e07d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56854/0 (socket says 172.21.15.107:56854) 2023-10-24T23:53:35.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 -- 172.21.15.107:0/3659255375 learned_addr learned my addr 172.21.15.107:0/3659255375 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:35.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/3659255375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e033750 0x55e42e033b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/3659255375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e032e30 0x55e42e033210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 -- 172.21.15.107:0/3659255375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e033750 msgr2=0x55e42e033b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 --2- 172.21.15.107:0/3659255375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e033750 0x55e42e033b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 -- 172.21.15.107:0/3659255375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e032e30 msgr2=0x55e42e033210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 --2- 172.21.15.107:0/3659255375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e032e30 0x55e42e033210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 -- 172.21.15.107:0/3659255375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e42db81890 con 0x55e42e034070 2023-10-24T23:53:35.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/3659255375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e033750 0x55e42e033b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:35.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 --2- 172.21.15.107:0/3659255375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e07d060 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400b8c0 tx=0x7f6ec400bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c14dec80c097ebd server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:35.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6eab7fe640 1 -- 172.21.15.107:0/3659255375 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4004440 con 0x55e42e034070 2023-10-24T23:53:35.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6eab7fe640 1 -- 172.21.15.107:0/3659255375 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec400c040 con 0x55e42e034070 2023-10-24T23:53:35.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6eab7fe640 1 -- 172.21.15.107:0/3659255375 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec400d980 con 0x55e42e034070 2023-10-24T23:53:35.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3659255375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 msgr2=0x55e42e07d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/3659255375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e07d060 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400b8c0 tx=0x7f6ec400bd80 comp rx=0 tx=0).stop 2023-10-24T23:53:35.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3659255375 shutdown_connections 2023-10-24T23:53:35.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/3659255375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e07d060 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/3659255375 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e033750 0x55e42e033b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/3659255375 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e032e30 0x55e42e033210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3659255375 >> 172.21.15.107:0/3659255375 conn(0x55e42e070dc0 msgr2=0x55e42e0731b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:35.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3659255375 shutdown_connections 2023-10-24T23:53:35.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3659255375 wait complete. 2023-10-24T23:53:35.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- start start 2023-10-24T23:53:35.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 Processor -- start 2023-10-24T23:53:35.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e032e30 0x55e42e039980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e033750 0x55e42e039ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e03a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e42de85060 con 0x55e42e034070 2023-10-24T23:53:35.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e42de24290 con 0x55e42e032e30 2023-10-24T23:53:35.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e42de426f0 con 0x55e42e033750 2023-10-24T23:53:35.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 10 client.0 _ll_register_callbacks cb 0x55e42e0211d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-24T23:53:35.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e03a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e03a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56864/0 (socket says 172.21.15.107:56864) 2023-10-24T23:53:35.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 -- 172.21.15.107:0/3288005736 learned_addr learned my addr 172.21.15.107:0/3288005736 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:35.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e033750 0x55e42e039ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e032e30 0x55e42e039980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 -- 172.21.15.107:0/3288005736 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e033750 msgr2=0x55e42e039ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e033750 0x55e42e039ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 -- 172.21.15.107:0/3288005736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e032e30 msgr2=0x55e42e039980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e032e30 0x55e42e039980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 -- 172.21.15.107:0/3288005736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec400b520 con 0x55e42e034070 2023-10-24T23:53:35.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ece0f4640 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e03a400 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400bdc0 tx=0x7f6ec4004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e2ff6c50541e15 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:35.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ea8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-10-24T23:53:35.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/3288005736 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4003620 con 0x55e42e034070 2023-10-24T23:53:35.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3288005736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e42de839b0 con 0x55e42e034070 2023-10-24T23:53:35.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/3288005736 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f6ec4003780 con 0x55e42e034070 2023-10-24T23:53:35.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/3288005736 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ec4003940 con 0x55e42e034070 2023-10-24T23:53:35.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 10 client.4932 fetch_fsmap learned FSMap version 12 2023-10-24T23:53:35.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.130+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3288005736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55e42de4a1e0 con 0x55e42e034070 2023-10-24T23:53:35.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/3288005736 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e42de839b0 con 0x55e42e034070 2023-10-24T23:53:35.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/3288005736 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 12) v1 ==== 3652+0+0 (secure 0 0 0) 0x7f6ec4003aa0 con 0x55e42e034070 2023-10-24T23:53:35.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 10 client.4932 fetch_fsmap finished waiting for FSMap version 12 2023-10-24T23:53:35.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 10 client.4932 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-24T23:53:35.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 20 client.4932 populate_metadata read hostname 'smithi107' 2023-10-24T23:53:35.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0863d0 0x55e42e0887c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 4 client.4932 mds_command: new command op to 4293 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-24T23:53:35.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3288005736 --> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e42deb44d0 con 0x55e42e0863d0 2023-10-24T23:53:35.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0863d0 0x55e42e0887c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0863d0 0x55e42e0887c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35c52e54f1b59e06 server_cookie=94e9bceea744209 in_seq=0 out_seq=0 2023-10-24T23:53:35.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ea8ff9640 10 client.4932 ms_handle_connect on v2:172.21.15.107:6836/727042497 2023-10-24T23:53:35.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/3288005736 <== mds.0 v2:172.21.15.107:6836/727042497 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55e42deb44d0 con 0x55e42e0863d0 2023-10-24T23:53:35.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ea8ff9640 10 client.4932 handle_command_reply: tid=0 2023-10-24T23:53:35.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 client.4932 shutdown 2023-10-24T23:53:35.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3288005736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 msgr2=0x55e42e03a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e03a400 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400bdc0 tx=0x7f6ec4004740 comp rx=0 tx=0).stop 2023-10-24T23:53:35.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3288005736 shutdown_connections 2023-10-24T23:53:35.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0863d0 0x55e42e0887c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e034070 0x55e42e03a400 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e033750 0x55e42e039ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/3288005736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e032e30 0x55e42e039980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ea8ff9640 0 client.4932 ms_handle_reset on v2:172.21.15.107:6836/727042497 2023-10-24T23:53:35.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3288005736 >> 172.21.15.107:0/3288005736 conn(0x55e42e070dc0 msgr2=0x55e42e032520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3288005736 shutdown_connections 2023-10-24T23:53:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/3288005736 wait complete. 2023-10-24T23:53:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.134+0000 7f6ed2c7f000 20 client.4932 trim_cache size 0 max 16384 2023-10-24T23:53:35.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 Processor -- start 2023-10-24T23:53:35.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- start start 2023-10-24T23:53:35.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0897d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e033750 0x55e42e093890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e034070 0x55e42e093dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e42db42940 con 0x55e42e032e30 2023-10-24T23:53:35.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e42de52a00 con 0x55e42e033750 2023-10-24T23:53:35.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e42de8d430 con 0x55e42e034070 2023-10-24T23:53:35.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ecd8f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0897d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ecd8f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0897d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56874/0 (socket says 172.21.15.107:56874) 2023-10-24T23:53:35.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/507694824 learned_addr learned my addr 172.21.15.107:0/507694824 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:35.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/507694824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e034070 msgr2=0x55e42e093dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/507694824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e034070 0x55e42e093dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/507694824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e033750 msgr2=0x55e42e093890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/507694824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e033750 0x55e42e093890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/507694824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e42de30330 con 0x55e42e032e30 2023-10-24T23:53:35.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/507694824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0897d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00af040 tx=0x7f6ec0004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754b72758362f706 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:35.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6eabfff640 1 -- 172.21.15.107:0/507694824 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec00c8170 con 0x55e42e032e30 2023-10-24T23:53:35.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6eabfff640 1 -- 172.21.15.107:0/507694824 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec00bf040 con 0x55e42e032e30 2023-10-24T23:53:35.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6eabfff640 1 -- 172.21.15.107:0/507694824 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec00c88f0 con 0x55e42e032e30 2023-10-24T23:53:35.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/507694824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 msgr2=0x55e42e0897d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/507694824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0897d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00af040 tx=0x7f6ec0004ac0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/507694824 shutdown_connections 2023-10-24T23:53:35.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/507694824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e034070 0x55e42e093dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/507694824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e033750 0x55e42e093890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/507694824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0897d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/507694824 >> 172.21.15.107:0/507694824 conn(0x55e42e070dc0 msgr2=0x55e42e031e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:35.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/507694824 shutdown_connections 2023-10-24T23:53:35.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/507694824 wait complete. 2023-10-24T23:53:35.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- start start 2023-10-24T23:53:35.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 Processor -- start 2023-10-24T23:53:35.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0d2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e033750 0x55e42e0cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e034070 0x55e42e0cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e42de865f0 con 0x55e42e032e30 2023-10-24T23:53:35.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e42de346a0 con 0x55e42e033750 2023-10-24T23:53:35.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.150+0000 7f6ed2c7f000 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e42df30020 con 0x55e42e034070 2023-10-24T23:53:35.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ed2c7f000 10 client.0 _ll_register_callbacks cb 0x55e42df24470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-24T23:53:35.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd8f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0d2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd8f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0d2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56876/0 (socket says 172.21.15.107:56876) 2023-10-24T23:53:35.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/2668036909 learned_addr learned my addr 172.21.15.107:0/2668036909 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:35.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e033750 0x55e42e0cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ece0f4640 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e034070 0x55e42e0cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/2668036909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e034070 msgr2=0x55e42e0cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e034070 0x55e42e0cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/2668036909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e033750 msgr2=0x55e42e0cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e033750 0x55e42e0cf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd8f3640 1 -- 172.21.15.107:0/2668036909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec806d8f0 con 0x55e42e032e30 2023-10-24T23:53:35.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd8f3640 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0d2c00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0004b00 tx=0x7f6ec001b0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea66cf1a5df1af server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:35.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ea8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2023-10-24T23:53:35.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/2668036909 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec00c82f0 con 0x55e42e032e30 2023-10-24T23:53:35.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/2668036909 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f6ec00c88f0 con 0x55e42e032e30 2023-10-24T23:53:35.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/2668036909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e42de54b90 con 0x55e42e032e30 2023-10-24T23:53:35.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/2668036909 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ec00c8450 con 0x55e42e032e30 2023-10-24T23:53:35.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ed2c7f000 10 client.4938 fetch_fsmap learned FSMap version 12 2023-10-24T23:53:35.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/2668036909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55e42db81cc0 con 0x55e42e032e30 2023-10-24T23:53:35.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/2668036909 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e42de54b90 con 0x55e42e032e30 2023-10-24T23:53:35.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/2668036909 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 12) v1 ==== 3652+0+0 (secure 0 0 0) 0x7f6ec00b5040 con 0x55e42e032e30 2023-10-24T23:53:35.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ed2c7f000 10 client.4938 fetch_fsmap finished waiting for FSMap version 12 2023-10-24T23:53:35.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ed2c7f000 10 client.4938 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-24T23:53:35.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ed2c7f000 20 client.4938 populate_metadata read hostname 'smithi107' 2023-10-24T23:53:35.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0d0710 0x55e42e0d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ed2c7f000 4 client.4938 mds_command: new command op to 4293 tid=0[{"prefix": "status"}] 2023-10-24T23:53:35.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/2668036909 --> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e42de7e0e0 con 0x55e42e0d0710 2023-10-24T23:53:35.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0d0710 0x55e42e0d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd0f2640 1 -- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0d0710 msgr2=0x55e42e0d0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2023-10-24T23:53:35.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd0f2640 1 -- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0d0710 msgr2=0x55e42e0d0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-24T23:53:35.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0d0710 0x55e42e0d0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-24T23:53:35.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.154+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0d0710 0x55e42e0d0af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-24T23:53:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0d0710 0x55e42e0d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecd0f2640 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0d0710 0x55e42e0d0af0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfa85828cbc9ecc2 server_cookie=2cae4cb21ba1f295 in_seq=0 out_seq=0 2023-10-24T23:53:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ea8ff9640 10 client.4938 ms_handle_connect on v2:172.21.15.107:6836/727042497 2023-10-24T23:53:35.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ea8ff9640 1 -- 172.21.15.107:0/2668036909 <== mds.0 v2:172.21.15.107:6836/727042497 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55e42de7e0e0 con 0x55e42e0d0710 2023-10-24T23:53:35.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ea8ff9640 10 client.4938 handle_command_reply: tid=0 2023-10-24T23:53:35.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 1 client.4938 shutdown 2023-10-24T23:53:35.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 msgr2=0x55e42e0d2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0d2c00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0004b00 tx=0x7f6ec001b0b0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/2668036909 shutdown_connections 2023-10-24T23:53:35.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:6836/727042497,v1:172.21.15.107:6837/727042497] conn(0x55e42e0d0710 0x55e42e0d0af0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55e42e034070 0x55e42e0cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55e42e033750 0x55e42e0cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 1 --2- 172.21.15.107:0/2668036909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e42e032e30 0x55e42e0d2c00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ea8ff9640 0 client.4938 ms_handle_reset on v2:172.21.15.107:6836/727042497 2023-10-24T23:53:35.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/2668036909 >> 172.21.15.107:0/2668036909 conn(0x55e42e070dc0 msgr2=0x55e42e089e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:35.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/2668036909 shutdown_connections 2023-10-24T23:53:35.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 1 -- 172.21.15.107:0/2668036909 wait complete. 2023-10-24T23:53:35.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ed2c7f000 20 client.4938 trim_cache size 0 max 16384 2023-10-24T23:53:35.364 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-10-24T23:53:35.364 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "c963b8f9-0f03-467c-8a02-12930711a2e8", 2023-10-24T23:53:35.365 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2023-10-24T23:53:35.365 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 4293, 2023-10-24T23:53:35.365 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2023-10-24T23:53:35.365 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2023-10-24T23:53:35.365 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "b", 2023-10-24T23:53:35.365 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 23.579278424000002, 2023-10-24T23:53:35.366 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 12, 2023-10-24T23:53:35.366 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 27, 2023-10-24T23:53:35.366 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 27, 2023-10-24T23:53:35.366 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 37.091419234999996 2023-10-24T23:53:35.366 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-10-24T23:53:35.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 -- 172.21.15.107:0/367210103 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6eac065e60 msgr2=0x7f6eac068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6eac065e60 0x7f6eac068320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00ae000 tx=0x7f6ec00a7070 comp rx=0 tx=0).stop 2023-10-24T23:53:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 -- 172.21.15.107:0/367210103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ec81566d0 msgr2=0x7f6ec814d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ec81566d0 0x7f6ec814d0c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800b410 tx=0x7f6eb800b8d0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 -- 172.21.15.107:0/367210103 shutdown_connections 2023-10-24T23:53:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ec815c260 0x7f6ec814d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ec81566d0 0x7f6ec814d0c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6eac065e60 0x7f6eac068320 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 --2- 172.21.15.107:0/367210103 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ec806f380 0x7f6ec81521f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 -- 172.21.15.107:0/367210103 >> 172.21.15.107:0/367210103 conn(0x7f6ec8070ed0 msgr2=0x7f6ec80c5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 -- 172.21.15.107:0/367210103 shutdown_connections 2023-10-24T23:53:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.358+0000 7f6ecfb57640 1 -- 172.21.15.107:0/367210103 wait complete. 2023-10-24T23:53:35.384 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-10-24T23:53:35.406 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-10-24T23:53:35.407 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2023-10-24T23:53:35.407 INFO:tasks.ceph_fuse:client.0 config is {'cephfs_name': 'a'} 2023-10-24T23:53:35.407 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-24T23:53:35.408 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-24T23:53:35.408 INFO:tasks.ceph_fuse:client.1 config is {'cephfs_name': 'b'} 2023-10-24T23:53:35.409 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-24T23:53:35.409 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-24T23:53:35.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:35.409 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list 2023-10-24T23:53:35.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:35.417 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link delete ceph-brx 2023-10-24T23:53:35.476 INFO:teuthology.orchestra.run.smithi110.stderr:Cannot find device "ceph-brx" 2023-10-24T23:53:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:53:35.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:35.478 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-10-24T23:53:35.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:35.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link delete ceph-brx 2023-10-24T23:53:35.545 INFO:teuthology.orchestra.run.smithi107.stderr:Cannot find device "ceph-brx" 2023-10-24T23:53:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-24T23:53:35.547 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-10-24T23:53:35.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-24T23:53:35.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.714+0000 7f262d09a640 1 Processor -- start 2023-10-24T23:53:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.714+0000 7f262d09a640 1 -- start start 2023-10-24T23:53:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f262d09a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f262814b600 0x7f2628151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f262d09a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f262806a230 0x7f26281551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f262d09a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26280c1860 con 0x7f262806af70 2023-10-24T23:53:35.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f262d09a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2628154d50 con 0x7f262814b600 2023-10-24T23:53:35.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f262d09a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26281524f0 con 0x7f262806a230 2023-10-24T23:53:35.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f2626d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f262814b600 0x7f2628151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f2626d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f262814b600 0x7f2628151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56886/0 (socket says 172.21.15.107:56886) 2023-10-24T23:53:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f2626d9d640 1 -- 172.21.15.107:0/441005284 learned_addr learned my addr 172.21.15.107:0/441005284 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f262759e640 1 --2- 172.21.15.107:0/441005284 >> v2:172.21.15.110:3300/0 conn(0x7f262806a230 0x7f26281551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f2626d9d640 1 -- 172.21.15.107:0/441005284 >> v2:172.21.15.110:3300/0 conn(0x7f262806a230 msgr2=0x7f26281551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f2626d9d640 1 --2- 172.21.15.107:0/441005284 >> v2:172.21.15.110:3300/0 conn(0x7f262806a230 0x7f26281551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f2626d9d640 1 -- 172.21.15.107:0/441005284 >> v1:172.21.15.110:6789/0 conn(0x7f262806af70 legacy=0x7f262806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f2626d9d640 1 -- 172.21.15.107:0/441005284 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262814b9e0 con 0x7f262814b600 2023-10-24T23:53:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f2626d9d640 1 --2- 172.21.15.107:0/441005284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262814b600 0x7f2628151fb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2628151530 tx=0x7f261801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8519b1c81f690fe2 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f2625d9b640 1 -- 172.21.15.107:0/441005284 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26180ac640 con 0x7f262814b600 2023-10-24T23:53:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f2625d9b640 1 -- 172.21.15.107:0/441005284 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26180ac7a0 con 0x7f262814b600 2023-10-24T23:53:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.718+0000 7f2625d9b640 1 -- 172.21.15.107:0/441005284 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26180acb30 con 0x7f262814b600 2023-10-24T23:53:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- 172.21.15.107:0/441005284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262814b600 msgr2=0x7f2628151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 --2- 172.21.15.107:0/441005284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262814b600 0x7f2628151fb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2628151530 tx=0x7f261801c400 comp rx=0 tx=0).stop 2023-10-24T23:53:35.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- 172.21.15.107:0/441005284 shutdown_connections 2023-10-24T23:53:35.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 --2- 172.21.15.107:0/441005284 >> v2:172.21.15.110:3300/0 conn(0x7f262806a230 0x7f26281551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 --2- 172.21.15.107:0/441005284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262814b600 0x7f2628151fb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- 172.21.15.107:0/441005284 >> 172.21.15.107:0/441005284 conn(0x7f26280c2950 msgr2=0x7f26280c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- 172.21.15.107:0/441005284 shutdown_connections 2023-10-24T23:53:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- 172.21.15.107:0/441005284 wait complete. 2023-10-24T23:53:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 Processor -- start 2023-10-24T23:53:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- start start 2023-10-24T23:53:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262806a230 0x7f262815a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262806af70 0x7f2628152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f262814b600 0x7f2628153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2628070b10 con 0x7f262806af70 2023-10-24T23:53:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2628071520 con 0x7f262806a230 2023-10-24T23:53:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2628145800 con 0x7f262814b600 2023-10-24T23:53:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262806af70 0x7f2628152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262806af70 0x7f2628152ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56896/0 (socket says 172.21.15.107:56896) 2023-10-24T23:53:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262659c640 1 -- 172.21.15.107:0/4189583424 learned_addr learned my addr 172.21.15.107:0/4189583424 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f2626d9d640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262806a230 0x7f262815a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262759e640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f262814b600 0x7f2628153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262659c640 1 -- 172.21.15.107:0/4189583424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f262814b600 msgr2=0x7f2628153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262659c640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f262814b600 0x7f2628153530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262659c640 1 -- 172.21.15.107:0/4189583424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262806a230 msgr2=0x7f262815a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262659c640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262806a230 0x7f262815a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262659c640 1 -- 172.21.15.107:0/4189583424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26180094d0 con 0x7f262806af70 2023-10-24T23:53:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f2626d9d640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262806a230 0x7f262815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262659c640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262806af70 0x7f2628152ff0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f261c00c410 tx=0x7f261c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f260f7fe640 1 -- 172.21.15.107:0/4189583424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c003600 con 0x7f262806af70 2023-10-24T23:53:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f260f7fe640 1 -- 172.21.15.107:0/4189583424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261c003780 con 0x7f262806af70 2023-10-24T23:53:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- 172.21.15.107:0/4189583424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2628071090 con 0x7f262806af70 2023-10-24T23:53:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f260f7fe640 1 -- 172.21.15.107:0/4189583424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c003b30 con 0x7f262806af70 2023-10-24T23:53:35.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- 172.21.15.107:0/4189583424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2628071f10 con 0x7f262806af70 2023-10-24T23:53:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.722+0000 7f262d09a640 1 -- 172.21.15.107:0/4189583424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2628071220 con 0x7f262806af70 2023-10-24T23:53:35.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.726+0000 7f260f7fe640 1 -- 172.21.15.107:0/4189583424 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f261c003c90 con 0x7f262806af70 2023-10-24T23:53:35.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.726+0000 7f260f7fe640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2614065e90 0x7f2614068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:35.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.726+0000 7f260f7fe640 1 -- 172.21.15.107:0/4189583424 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f261c082fc0 con 0x7f262806af70 2023-10-24T23:53:35.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.726+0000 7f2626d9d640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2614065e90 0x7f2614068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:35.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.726+0000 7f2626d9d640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2614065e90 0x7f2614068350 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f26180ae000 tx=0x7f2618008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:35.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.730+0000 7f260f7fe640 1 -- 172.21.15.107:0/4189583424 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f261c0b4040 con 0x7f262806af70 2023-10-24T23:53:35.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.922+0000 7f262d09a640 1 -- 172.21.15.107:0/4189583424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f26280746e0 con 0x7f262806af70 2023-10-24T23:53:35.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.922+0000 7f260f7fe640 1 -- 172.21.15.107:0/4189583424 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v12) v1 ==== 53+0+116 (secure 0 0 0) 0x7f261c017070 con 0x7f262806af70 2023-10-24T23:53:35.929 INFO:teuthology.orchestra.run.smithi107.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2023-10-24T23:53:35.929 INFO:teuthology.orchestra.run.smithi107.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2023-10-24T23:53:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 -- 172.21.15.107:0/4189583424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2614065e90 msgr2=0x7f2614068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2614065e90 0x7f2614068350 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f26180ae000 tx=0x7f2618008040 comp rx=0 tx=0).stop 2023-10-24T23:53:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 -- 172.21.15.107:0/4189583424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262806af70 msgr2=0x7f2628152ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:35.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262806af70 0x7f2628152ff0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f261c00c410 tx=0x7f261c00c8d0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 -- 172.21.15.107:0/4189583424 shutdown_connections 2023-10-24T23:53:35.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2614065e90 0x7f2614068350 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f262814b600 0x7f2628153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f262806af70 0x7f2628152ff0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 --2- 172.21.15.107:0/4189583424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f262806a230 0x7f262815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:35.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 -- 172.21.15.107:0/4189583424 >> 172.21.15.107:0/4189583424 conn(0x7f26280c2950 msgr2=0x7f26280c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:35.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 -- 172.21.15.107:0/4189583424 shutdown_connections 2023-10-24T23:53:35.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:35.926+0000 7f262d09a640 1 -- 172.21.15.107:0/4189583424 wait complete. 2023-10-24T23:53:35.947 INFO:tasks.cephfs.mount:Mounting Ceph FS a; just confirmed its presence on cluster 2023-10-24T23:53:35.948 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-24T23:53:35.948 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2023-10-24T23:53:35.948 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-24T23:53:35.948 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-24T23:53:35.948 INFO:tasks.cephfs.mount:self.cephfs_name = a 2023-10-24T23:53:35.949 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-24T23:53:35.949 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-24T23:53:35.949 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-24T23:53:35.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:35.950 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2023-10-24T23:53:35.956 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-24T23:53:35.956 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-24T23:53:35.956 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2023-10-24T23:53:35.957 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-10-24T23:53:35.957 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2023-10-24T23:53:35.957 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-10-24T23:53:35.957 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-24T23:53:35.957 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2023-10-24T23:53:35.958 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2023-10-24T23:53:35.958 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-24T23:53:35.958 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2023-10-24T23:53:35.958 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2023-10-24T23:53:35.958 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-24T23:53:35.959 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2023-10-24T23:53:35.959 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-24T23:53:35.959 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2023-10-24T23:53:35.959 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-24T23:53:35.959 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 42483sec preferred_lft 42483sec 2023-10-24T23:53:35.960 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2023-10-24T23:53:35.960 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-10-24T23:53:35.960 INFO:teuthology.orchestra.run.smithi107.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-24T23:53:35.960 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 02:42:fe:53:8d:77 brd ff:ff:ff:ff:ff:ff 2023-10-24T23:53:35.960 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-24T23:53:35.961 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2023-10-24T23:53:35.961 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-24T23:53:35.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:35.962 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-10-24T23:53:35.962 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-10-24T23:53:35.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add name ceph-brx type bridge 2023-10-24T23:53:35.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr flush dev ceph-brx 2023-10-24T23:53:35.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set ceph-brx up 2023-10-24T23:53:35.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-24T23:53:35.964 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-10-24T23:53:36.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:36.031 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-24T23:53:36.081 INFO:teuthology.orchestra.run.smithi107.stdout:1 2023-10-24T23:53:36.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:36.083 DEBUG:teuthology.orchestra.run.smithi107:> route 2023-10-24T23:53:36.152 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2023-10-24T23:53:36.152 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-24T23:53:36.153 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-24T23:53:36.153 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-24T23:53:36.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-24T23:53:36.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-24T23:53:36.154 INFO:teuthology.orchestra.run.smithi107.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-24T23:53:36.154 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-24T23:53:36.154 INFO:teuthology.orchestra.run.smithi107.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-24T23:53:36.154 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-24T23:53:36.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:36.155 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-10-24T23:53:36.155 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-10-24T23:53:36.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-24T23:53:36.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-24T23:53:36.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-24T23:53:36.156 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-10-24T23:53:36.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:36.232 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-10-24T23:53:36.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:36.283 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list-id 2023-10-24T23:53:36.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:36.330 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-10-24T23:53:36.331 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-10-24T23:53:36.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-24T23:53:36.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-24T23:53:36.331 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-10-24T23:53:36.399 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-24T23:53:36.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:36.400 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-10-24T23:53:36.400 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-10-24T23:53:36.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-24T23:53:36.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-24T23:53:36.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-24T23:53:36.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-24T23:53:36.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-24T23:53:36.402 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-10-24T23:53:36.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:36.502 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec bash -c ' 2023-10-24T23:53:36.502 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2023-10-24T23:53:36.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2023-10-24T23:53:36.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set dev brx.0 master ceph-brx 2023-10-24T23:53:36.504 DEBUG:teuthology.orchestra.run.smithi107:> ') 2023-10-24T23:53:36.523 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'cephfs_name': 'a'} 2023-10-24T23:53:36.524 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-24T23:53:36.524 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-24T23:53:36.573 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-24T23:53:36.573 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-24T23:53:36.574 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-24T23:53:36.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2023-10-24T23:53:36.678 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-24T23:53:36.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-24T23:53:36.731 INFO:teuthology.orchestra.run.smithi107.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-24T23:53:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-24T23:53:36.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:36.733 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2023-10-24T23:53:36.781 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-24T23:53:36.782 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_fs=a) 2023-10-24T23:53:36.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2023-10-24T23:53:36.839 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-24T23:53:36.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-24T23:53:36.892 INFO:teuthology.orchestra.run.smithi107.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-24T23:53:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-24T23:53:36.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:36.894 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2023-10-24T23:53:36.914 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:2023-10-24T23:53:36.910+0000 7f46c3294580 -1 init, newargv = 0x5558bc9dc5e0 newargc=15 2023-10-24T23:53:36.915 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:ceph-fuse[46369]: starting ceph client 2023-10-24T23:53:36.923 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:ceph-fuse[46369]: starting fuse 2023-10-24T23:53:38.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2023-10-24T23:53:38.036 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-24T23:53:38.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-24T23:53:38.089 INFO:teuthology.orchestra.run.smithi107.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-24T23:53:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-24T23:53:38.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:38.091 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2023-10-24T23:53:38.138 INFO:teuthology.orchestra.run.smithi107.stdout:45 2023-10-24T23:53:38.138 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-10-24T23:53:38.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-24T23:53:38.139 DEBUG:teuthology.orchestra.run.smithi107:> import glob 2023-10-24T23:53:38.139 DEBUG:teuthology.orchestra.run.smithi107:> import re 2023-10-24T23:53:38.140 DEBUG:teuthology.orchestra.run.smithi107:> import os 2023-10-24T23:53:38.140 DEBUG:teuthology.orchestra.run.smithi107:> import subprocess 2023-10-24T23:53:38.140 DEBUG:teuthology.orchestra.run.smithi107:> 2023-10-24T23:53:38.141 DEBUG:teuthology.orchestra.run.smithi107:> def _find_admin_socket(client_name): 2023-10-24T23:53:38.141 DEBUG:teuthology.orchestra.run.smithi107:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-10-24T23:53:38.141 DEBUG:teuthology.orchestra.run.smithi107:> files = glob.glob(asok_path) 2023-10-24T23:53:38.142 DEBUG:teuthology.orchestra.run.smithi107:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-10-24T23:53:38.142 DEBUG:teuthology.orchestra.run.smithi107:> 2023-10-24T23:53:38.142 DEBUG:teuthology.orchestra.run.smithi107:> # Given a non-glob path, it better be there 2023-10-24T23:53:38.142 DEBUG:teuthology.orchestra.run.smithi107:> if "*" not in asok_path: 2023-10-24T23:53:38.143 DEBUG:teuthology.orchestra.run.smithi107:> assert(len(files) == 1) 2023-10-24T23:53:38.143 DEBUG:teuthology.orchestra.run.smithi107:> return files[0] 2023-10-24T23:53:38.143 DEBUG:teuthology.orchestra.run.smithi107:> 2023-10-24T23:53:38.144 DEBUG:teuthology.orchestra.run.smithi107:> for f in files: 2023-10-24T23:53:38.144 DEBUG:teuthology.orchestra.run.smithi107:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-24T23:53:38.144 DEBUG:teuthology.orchestra.run.smithi107:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-24T23:53:38.145 DEBUG:teuthology.orchestra.run.smithi107:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-24T23:53:38.145 DEBUG:teuthology.orchestra.run.smithi107:> contents = proc_f.read() 2023-10-24T23:53:38.145 DEBUG:teuthology.orchestra.run.smithi107:> if mountpoint in contents: 2023-10-24T23:53:38.146 DEBUG:teuthology.orchestra.run.smithi107:> return f 2023-10-24T23:53:38.146 DEBUG:teuthology.orchestra.run.smithi107:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-24T23:53:38.146 DEBUG:teuthology.orchestra.run.smithi107:> 2023-10-24T23:53:38.147 DEBUG:teuthology.orchestra.run.smithi107:> print(_find_admin_socket("client.0")) 2023-10-24T23:53:38.147 DEBUG:teuthology.orchestra.run.smithi107:> ' 2023-10-24T23:53:38.258 INFO:teuthology.orchestra.run.smithi107.stdout:/var/run/ceph/ceph-client.0.46369.asok 2023-10-24T23:53:38.442 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46369.asok 2023-10-24T23:53:38.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:38.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46369.asok status 2023-10-24T23:53:38.597 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-10-24T23:53:38.598 INFO:teuthology.orchestra.run.smithi107.stdout: "metadata": { 2023-10-24T23:53:38.598 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_sha1": "e652850897f60c32cb9cc05022cf9f93e3b00db1", 2023-10-24T23:53:38.598 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 18.2.0-1095-ge6528508 (e652850897f60c32cb9cc05022cf9f93e3b00db1) reef (stable)", 2023-10-24T23:53:38.598 INFO:teuthology.orchestra.run.smithi107.stdout: "entity_id": "0", 2023-10-24T23:53:38.598 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": "smithi107", 2023-10-24T23:53:38.598 INFO:teuthology.orchestra.run.smithi107.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-10-24T23:53:38.599 INFO:teuthology.orchestra.run.smithi107.stdout: "pid": "46369", 2023-10-24T23:53:38.599 INFO:teuthology.orchestra.run.smithi107.stdout: "root": "/" 2023-10-24T23:53:38.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-10-24T23:53:38.599 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_count": 0, 2023-10-24T23:53:38.599 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_pinned_count": 0, 2023-10-24T23:53:38.600 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 4953, 2023-10-24T23:53:38.600 INFO:teuthology.orchestra.run.smithi107.stdout: "inst": { 2023-10-24T23:53:38.600 INFO:teuthology.orchestra.run.smithi107.stdout: "name": { 2023-10-24T23:53:38.600 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client", 2023-10-24T23:53:38.600 INFO:teuthology.orchestra.run.smithi107.stdout: "num": 4953 2023-10-24T23:53:38.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-10-24T23:53:38.601 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2023-10-24T23:53:38.601 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2023-10-24T23:53:38.601 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2023-10-24T23:53:38.601 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 582933152 2023-10-24T23:53:38.601 INFO:teuthology.orchestra.run.smithi107.stdout: } 2023-10-24T23:53:38.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-10-24T23:53:38.602 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2023-10-24T23:53:38.602 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2023-10-24T23:53:38.602 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2023-10-24T23:53:38.602 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 582933152 2023-10-24T23:53:38.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2023-10-24T23:53:38.602 INFO:teuthology.orchestra.run.smithi107.stdout: "inst_str": "client.4953 192.168.0.1:0/582933152", 2023-10-24T23:53:38.603 INFO:teuthology.orchestra.run.smithi107.stdout: "addr_str": "192.168.0.1:0/582933152", 2023-10-24T23:53:38.603 INFO:teuthology.orchestra.run.smithi107.stdout: "inode_count": 1, 2023-10-24T23:53:38.603 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_epoch": 8, 2023-10-24T23:53:38.603 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch": 27, 2023-10-24T23:53:38.603 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch_barrier": 0, 2023-10-24T23:53:38.604 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklisted": false, 2023-10-24T23:53:38.604 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "a" 2023-10-24T23:53:38.604 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-10-24T23:53:38.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-24T23:53:38.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa07de92640 1 Processor -- start 2023-10-24T23:53:38.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa07de92640 1 -- start start 2023-10-24T23:53:38.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa07de92640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa07814fd40 0x7fa0780ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:38.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa07de92640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0780a9820 0x7fa0780aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:38.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa07de92640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07806f730 con 0x7fa07806ee40 2023-10-24T23:53:38.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa07de92640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0780aa750 con 0x7fa07814fd40 2023-10-24T23:53:38.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa07de92640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa078156a40 con 0x7fa0780a9820 2023-10-24T23:53:38.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa0777fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa07814fd40 0x7fa0780ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:38.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa077fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa07806ee40 0x7fa0780c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60220/0 (socket says 172.21.15.107:60220) 2023-10-24T23:53:38.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa077fff640 1 -- 172.21.15.107:0/4236445695 learned_addr learned my addr 172.21.15.107:0/4236445695 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:38.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa076ffd640 1 --2- 172.21.15.107:0/4236445695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0780a9820 0x7fa0780aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:38.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa0777fe640 1 -- 172.21.15.107:0/4236445695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0780a9820 msgr2=0x7fa0780aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:38.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa0777fe640 1 --2- 172.21.15.107:0/4236445695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0780a9820 0x7fa0780aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:38.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa0777fe640 1 -- 172.21.15.107:0/4236445695 >> v1:172.21.15.107:6789/0 conn(0x7fa07806ee40 legacy=0x7fa0780c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:38.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa0767fc640 1 -- 172.21.15.107:0/4236445695 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490176164 0 0) 0x7fa07806f730 con 0x7fa07806ee40 2023-10-24T23:53:38.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.730+0000 7fa0777fe640 1 -- 172.21.15.107:0/4236445695 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa078156710 con 0x7fa07814fd40 2023-10-24T23:53:38.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa0777fe640 1 --2- 172.21.15.107:0/4236445695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa07814fd40 0x7fa0780ab090 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa0680097c0 tx=0x7fa06801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab65b19495470f6d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:38.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa0767fc640 1 -- 172.21.15.107:0/4236445695 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0680ab600 con 0x7fa07814fd40 2023-10-24T23:53:38.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa0767fc640 1 -- 172.21.15.107:0/4236445695 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0680a7050 con 0x7fa07814fd40 2023-10-24T23:53:38.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa0767fc640 1 -- 172.21.15.107:0/4236445695 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0680ab980 con 0x7fa07814fd40 2023-10-24T23:53:38.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa07de92640 1 -- 172.21.15.107:0/4236445695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa07814fd40 msgr2=0x7fa0780ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:38.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa07de92640 1 --2- 172.21.15.107:0/4236445695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa07814fd40 0x7fa0780ab090 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa0680097c0 tx=0x7fa06801c330 comp rx=0 tx=0).stop 2023-10-24T23:53:38.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa07de92640 1 -- 172.21.15.107:0/4236445695 shutdown_connections 2023-10-24T23:53:38.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa07de92640 1 --2- 172.21.15.107:0/4236445695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0780a9820 0x7fa0780aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:38.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa07de92640 1 --2- 172.21.15.107:0/4236445695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa07814fd40 0x7fa0780ab090 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:38.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa07de92640 1 -- 172.21.15.107:0/4236445695 >> 172.21.15.107:0/4236445695 conn(0x7fa0780c2950 msgr2=0x7fa0780c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:38.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa07de92640 1 -- 172.21.15.107:0/4236445695 shutdown_connections 2023-10-24T23:53:38.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa07de92640 1 -- 172.21.15.107:0/4236445695 wait complete. 2023-10-24T23:53:38.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa07de92640 1 Processor -- start 2023-10-24T23:53:38.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.734+0000 7fa07de92640 1 -- start start 2023-10-24T23:53:38.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa07de92640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa07806ee40 0x7fa0781559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:38.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa07de92640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa07814fd40 0x7fa07814bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:38.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa07de92640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa078155060 0x7fa078148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:38.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa07de92640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa078067210 con 0x7fa07806ee40 2023-10-24T23:53:38.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa07de92640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa078155f00 con 0x7fa07814fd40 2023-10-24T23:53:38.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa07de92640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa078156130 con 0x7fa078155060 2023-10-24T23:53:38.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa0777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa07806ee40 0x7fa0781559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:38.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa0777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa07806ee40 0x7fa0781559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56912/0 (socket says 172.21.15.107:56912) 2023-10-24T23:53:38.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa0777fe640 1 -- 172.21.15.107:0/2668358627 learned_addr learned my addr 172.21.15.107:0/2668358627 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:38.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa0777fe640 1 -- 172.21.15.107:0/2668358627 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa078155060 msgr2=0x7fa078148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:38.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa0777fe640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa078155060 0x7fa078148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:38.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa0777fe640 1 -- 172.21.15.107:0/2668358627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa07814fd40 msgr2=0x7fa07814bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:38.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa0777fe640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa07814fd40 0x7fa07814bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:38.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa0777fe640 1 -- 172.21.15.107:0/2668358627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa068009420 con 0x7fa07806ee40 2023-10-24T23:53:38.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa0777fe640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa07806ee40 0x7fa0781559c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa068019f60 tx=0x7fa0680190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:38.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa05ffff640 1 -- 172.21.15.107:0/2668358627 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0680b4610 con 0x7fa07806ee40 2023-10-24T23:53:38.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa05ffff640 1 -- 172.21.15.107:0/2668358627 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0680abe90 con 0x7fa07806ee40 2023-10-24T23:53:38.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa07de92640 1 -- 172.21.15.107:0/2668358627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0781ed910 con 0x7fa07806ee40 2023-10-24T23:53:38.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa05ffff640 1 -- 172.21.15.107:0/2668358627 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0680b49d0 con 0x7fa07806ee40 2023-10-24T23:53:38.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa07de92640 1 -- 172.21.15.107:0/2668358627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0781ecfe0 con 0x7fa07806ee40 2023-10-24T23:53:38.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa07de92640 1 -- 172.21.15.107:0/2668358627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0780c24a0 con 0x7fa07806ee40 2023-10-24T23:53:38.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa05ffff640 1 -- 172.21.15.107:0/2668358627 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa0680b2040 con 0x7fa07806ee40 2023-10-24T23:53:38.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa05ffff640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa064065e60 0x7fa064068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:38.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa05ffff640 1 -- 172.21.15.107:0/2668358627 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fa06811de30 con 0x7fa07806ee40 2023-10-24T23:53:38.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa076ffd640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa064065e60 0x7fa064068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:38.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.738+0000 7fa076ffd640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa064065e60 0x7fa064068320 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa07814c9f0 tx=0x7fa06c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:38.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.742+0000 7fa05ffff640 1 -- 172.21.15.107:0/2668358627 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa0680eb150 con 0x7fa07806ee40 2023-10-24T23:53:38.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.926+0000 7fa07de92640 1 -- 172.21.15.107:0/2668358627 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa078070840 con 0x7fa07806ee40 2023-10-24T23:53:38.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa05ffff640 1 -- 172.21.15.107:0/2668358627 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v12) v1 ==== 53+0+116 (secure 0 0 0) 0x7fa0680b0070 con 0x7fa07806ee40 2023-10-24T23:53:38.933 INFO:teuthology.orchestra.run.smithi107.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2023-10-24T23:53:38.934 INFO:teuthology.orchestra.run.smithi107.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2023-10-24T23:53:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 -- 172.21.15.107:0/2668358627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa064065e60 msgr2=0x7fa064068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa064065e60 0x7fa064068320 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa07814c9f0 tx=0x7fa06c008040 comp rx=0 tx=0).stop 2023-10-24T23:53:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 -- 172.21.15.107:0/2668358627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa07806ee40 msgr2=0x7fa0781559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:38.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa07806ee40 0x7fa0781559c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa068019f60 tx=0x7fa0680190d0 comp rx=0 tx=0).stop 2023-10-24T23:53:38.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 -- 172.21.15.107:0/2668358627 shutdown_connections 2023-10-24T23:53:38.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa064065e60 0x7fa064068320 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:38.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa078155060 0x7fa078148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:38.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa07814fd40 0x7fa07814bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:38.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 --2- 172.21.15.107:0/2668358627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa07806ee40 0x7fa0781559c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:38.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 -- 172.21.15.107:0/2668358627 >> 172.21.15.107:0/2668358627 conn(0x7fa0780c2950 msgr2=0x7fa07815c440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:38.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 -- 172.21.15.107:0/2668358627 shutdown_connections 2023-10-24T23:53:38.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:38.930+0000 7fa07de92640 1 -- 172.21.15.107:0/2668358627 wait complete. 2023-10-24T23:53:38.950 INFO:tasks.cephfs.mount:Mounting Ceph FS b; just confirmed its presence on cluster 2023-10-24T23:53:38.950 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-24T23:53:38.950 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi110.front.sepia.ceph.com 2023-10-24T23:53:38.950 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-24T23:53:38.951 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-24T23:53:38.951 INFO:tasks.cephfs.mount:self.cephfs_name = b 2023-10-24T23:53:38.951 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-24T23:53:38.952 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-24T23:53:38.952 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-24T23:53:38.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:38.953 DEBUG:teuthology.orchestra.run.smithi110:> ip addr 2023-10-24T23:53:38.960 INFO:teuthology.orchestra.run.smithi110.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-24T23:53:38.960 INFO:teuthology.orchestra.run.smithi110.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-24T23:53:38.960 INFO:teuthology.orchestra.run.smithi110.stdout: inet 127.0.0.1/8 scope host lo 2023-10-24T23:53:38.961 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-24T23:53:38.961 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 ::1/128 scope host 2023-10-24T23:53:38.961 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-24T23:53:38.961 INFO:teuthology.orchestra.run.smithi110.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-24T23:53:38.961 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:14 brd ff:ff:ff:ff:ff:ff 2023-10-24T23:53:38.962 INFO:teuthology.orchestra.run.smithi110.stdout: altname enp7s0f0 2023-10-24T23:53:38.962 INFO:teuthology.orchestra.run.smithi110.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-24T23:53:38.962 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:d8:db:15 brd ff:ff:ff:ff:ff:ff 2023-10-24T23:53:38.962 INFO:teuthology.orchestra.run.smithi110.stdout: altname enp7s0f1 2023-10-24T23:53:38.963 INFO:teuthology.orchestra.run.smithi110.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-24T23:53:38.963 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:34 brd ff:ff:ff:ff:ff:ff 2023-10-24T23:53:38.963 INFO:teuthology.orchestra.run.smithi110.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-24T23:53:38.963 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 0c:c4:7a:8f:cc:35 brd ff:ff:ff:ff:ff:ff 2023-10-24T23:53:38.963 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.21.15.110/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-24T23:53:38.963 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft 42470sec preferred_lft 42470sec 2023-10-24T23:53:38.964 INFO:teuthology.orchestra.run.smithi110.stdout: inet6 fe80::ec4:7aff:fe8f:cc35/64 scope link 2023-10-24T23:53:38.964 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-24T23:53:38.964 INFO:teuthology.orchestra.run.smithi110.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-24T23:53:38.964 INFO:teuthology.orchestra.run.smithi110.stdout: link/ether 02:42:7c:9c:0a:50 brd ff:ff:ff:ff:ff:ff 2023-10-24T23:53:38.964 INFO:teuthology.orchestra.run.smithi110.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-24T23:53:38.965 INFO:teuthology.orchestra.run.smithi110.stdout: valid_lft forever preferred_lft forever 2023-10-24T23:53:38.965 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-24T23:53:38.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:38.965 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-24T23:53:38.966 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-24T23:53:38.966 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link add name ceph-brx type bridge 2023-10-24T23:53:38.967 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip addr flush dev ceph-brx 2023-10-24T23:53:38.967 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set ceph-brx up 2023-10-24T23:53:38.968 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-24T23:53:38.968 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-24T23:53:39.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:39.042 DEBUG:teuthology.orchestra.run.smithi110:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-24T23:53:39.092 INFO:teuthology.orchestra.run.smithi110.stdout:1 2023-10-24T23:53:39.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:39.094 DEBUG:teuthology.orchestra.run.smithi110:> route 2023-10-24T23:53:39.271 INFO:teuthology.orchestra.run.smithi110.stdout:Kernel IP routing table 2023-10-24T23:53:39.272 INFO:teuthology.orchestra.run.smithi110.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-24T23:53:39.272 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-24T23:53:39.272 INFO:teuthology.orchestra.run.smithi110.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-24T23:53:39.272 INFO:teuthology.orchestra.run.smithi110.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-24T23:53:39.273 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-24T23:53:39.273 INFO:teuthology.orchestra.run.smithi110.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-24T23:53:39.273 INFO:teuthology.orchestra.run.smithi110.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-24T23:53:39.273 INFO:teuthology.orchestra.run.smithi110.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-24T23:53:39.273 INFO:teuthology.orchestra.run.smithi110.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-24T23:53:39.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:39.274 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-24T23:53:39.274 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-24T23:53:39.275 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-24T23:53:39.275 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-24T23:53:39.275 DEBUG:teuthology.orchestra.run.smithi110:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-24T23:53:39.276 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-24T23:53:39.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:39.310 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list 2023-10-24T23:53:39.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:39.357 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list-id 2023-10-24T23:53:39.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:39.408 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-24T23:53:39.408 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-24T23:53:39.409 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-24T23:53:39.409 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-24T23:53:39.409 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-24T23:53:39.477 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-24T23:53:39.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:39.478 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-24T23:53:39.478 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-24T23:53:39.478 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-24T23:53:39.479 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-24T23:53:39.479 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-24T23:53:39.479 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-24T23:53:39.479 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-24T23:53:39.480 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-24T23:53:39.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:39.608 DEBUG:teuthology.orchestra.run.smithi110:> (cd / && exec bash -c ' 2023-10-24T23:53:39.608 DEBUG:teuthology.orchestra.run.smithi110:> set -e 2023-10-24T23:53:39.608 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set brx.0 up 2023-10-24T23:53:39.609 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link set dev brx.0 master ceph-brx 2023-10-24T23:53:39.609 DEBUG:teuthology.orchestra.run.smithi110:> ') 2023-10-24T23:53:39.627 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {'cephfs_name': 'b'} 2023-10-24T23:53:39.627 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-24T23:53:39.628 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-24T23:53:39.674 INFO:teuthology.orchestra.run.smithi110.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-24T23:53:39.675 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-24T23:53:39.675 DEBUG:teuthology.orchestra.run.smithi110:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-24T23:53:39.722 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-24T23:53:39.777 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-24T23:53:39.777 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-24T23:53:39.830 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-24T23:53:39.831 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-24T23:53:39.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:39.832 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-24T23:53:39.879 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-24T23:53:39.880 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_fs=b) 2023-10-24T23:53:39.924 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-24T23:53:39.939 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-24T23:53:39.939 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-24T23:53:39.994 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-24T23:53:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-24T23:53:39.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:39.997 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-24T23:53:40.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi110.stderr:2023-10-24T23:53:40.012+0000 7fa08681d580 -1 init, newargv = 0x564a0bfb4820 newargc=15 2023-10-24T23:53:40.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi110.stderr:ceph-fuse[42968]: starting ceph client 2023-10-24T23:53:40.024 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi110.stderr:ceph-fuse[42968]: starting fuse 2023-10-24T23:53:41.123 DEBUG:teuthology.orchestra.run.smithi110:> sudo modprobe fuse 2023-10-24T23:53:41.140 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-24T23:53:41.141 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-24T23:53:41.195 INFO:teuthology.orchestra.run.smithi110.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-24T23:53:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-24T23:53:41.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:41.197 DEBUG:teuthology.orchestra.run.smithi110:> ls /sys/fs/fuse/connections 2023-10-24T23:53:41.244 INFO:teuthology.orchestra.run.smithi110.stdout:45 2023-10-24T23:53:41.244 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2023-10-24T23:53:41.245 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-24T23:53:41.245 DEBUG:teuthology.orchestra.run.smithi110:> import glob 2023-10-24T23:53:41.245 DEBUG:teuthology.orchestra.run.smithi110:> import re 2023-10-24T23:53:41.246 DEBUG:teuthology.orchestra.run.smithi110:> import os 2023-10-24T23:53:41.246 DEBUG:teuthology.orchestra.run.smithi110:> import subprocess 2023-10-24T23:53:41.246 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-24T23:53:41.247 DEBUG:teuthology.orchestra.run.smithi110:> def _find_admin_socket(client_name): 2023-10-24T23:53:41.247 DEBUG:teuthology.orchestra.run.smithi110:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-10-24T23:53:41.247 DEBUG:teuthology.orchestra.run.smithi110:> files = glob.glob(asok_path) 2023-10-24T23:53:41.247 DEBUG:teuthology.orchestra.run.smithi110:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-10-24T23:53:41.248 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-24T23:53:41.248 DEBUG:teuthology.orchestra.run.smithi110:> # Given a non-glob path, it better be there 2023-10-24T23:53:41.249 DEBUG:teuthology.orchestra.run.smithi110:> if "*" not in asok_path: 2023-10-24T23:53:41.249 DEBUG:teuthology.orchestra.run.smithi110:> assert(len(files) == 1) 2023-10-24T23:53:41.249 DEBUG:teuthology.orchestra.run.smithi110:> return files[0] 2023-10-24T23:53:41.250 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-24T23:53:41.250 DEBUG:teuthology.orchestra.run.smithi110:> for f in files: 2023-10-24T23:53:41.250 DEBUG:teuthology.orchestra.run.smithi110:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-24T23:53:41.251 DEBUG:teuthology.orchestra.run.smithi110:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-24T23:53:41.251 DEBUG:teuthology.orchestra.run.smithi110:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-24T23:53:41.251 DEBUG:teuthology.orchestra.run.smithi110:> contents = proc_f.read() 2023-10-24T23:53:41.252 DEBUG:teuthology.orchestra.run.smithi110:> if mountpoint in contents: 2023-10-24T23:53:41.252 DEBUG:teuthology.orchestra.run.smithi110:> return f 2023-10-24T23:53:41.252 DEBUG:teuthology.orchestra.run.smithi110:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-24T23:53:41.252 DEBUG:teuthology.orchestra.run.smithi110:> 2023-10-24T23:53:41.253 DEBUG:teuthology.orchestra.run.smithi110:> print(_find_admin_socket("client.1")) 2023-10-24T23:53:41.253 DEBUG:teuthology.orchestra.run.smithi110:> ' 2023-10-24T23:53:41.366 INFO:teuthology.orchestra.run.smithi110.stdout:/var/run/ceph/ceph-client.1.42968.asok 2023-10-24T23:53:41.548 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42968.asok 2023-10-24T23:53:41.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:41.549 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42968.asok status 2023-10-24T23:53:41.701 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2023-10-24T23:53:41.702 INFO:teuthology.orchestra.run.smithi110.stdout: "metadata": { 2023-10-24T23:53:41.702 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_sha1": "e652850897f60c32cb9cc05022cf9f93e3b00db1", 2023-10-24T23:53:41.702 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 18.2.0-1095-ge6528508 (e652850897f60c32cb9cc05022cf9f93e3b00db1) reef (stable)", 2023-10-24T23:53:41.702 INFO:teuthology.orchestra.run.smithi110.stdout: "entity_id": "1", 2023-10-24T23:53:41.703 INFO:teuthology.orchestra.run.smithi110.stdout: "hostname": "smithi110", 2023-10-24T23:53:41.703 INFO:teuthology.orchestra.run.smithi110.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-10-24T23:53:41.703 INFO:teuthology.orchestra.run.smithi110.stdout: "pid": "42968", 2023-10-24T23:53:41.703 INFO:teuthology.orchestra.run.smithi110.stdout: "root": "/" 2023-10-24T23:53:41.703 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-24T23:53:41.703 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_count": 0, 2023-10-24T23:53:41.704 INFO:teuthology.orchestra.run.smithi110.stdout: "dentry_pinned_count": 0, 2023-10-24T23:53:41.704 INFO:teuthology.orchestra.run.smithi110.stdout: "id": 4556, 2023-10-24T23:53:41.704 INFO:teuthology.orchestra.run.smithi110.stdout: "inst": { 2023-10-24T23:53:41.704 INFO:teuthology.orchestra.run.smithi110.stdout: "name": { 2023-10-24T23:53:41.704 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "client", 2023-10-24T23:53:41.705 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 4556 2023-10-24T23:53:41.705 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-24T23:53:41.705 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-24T23:53:41.705 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-24T23:53:41.705 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.1:0", 2023-10-24T23:53:41.706 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 1969380428 2023-10-24T23:53:41.706 INFO:teuthology.orchestra.run.smithi110.stdout: } 2023-10-24T23:53:41.706 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-24T23:53:41.706 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": { 2023-10-24T23:53:41.706 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2023-10-24T23:53:41.706 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "192.168.0.1:0", 2023-10-24T23:53:41.707 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 1969380428 2023-10-24T23:53:41.707 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2023-10-24T23:53:41.707 INFO:teuthology.orchestra.run.smithi110.stdout: "inst_str": "client.4556 192.168.0.1:0/1969380428", 2023-10-24T23:53:41.707 INFO:teuthology.orchestra.run.smithi110.stdout: "addr_str": "192.168.0.1:0/1969380428", 2023-10-24T23:53:41.707 INFO:teuthology.orchestra.run.smithi110.stdout: "inode_count": 1, 2023-10-24T23:53:41.708 INFO:teuthology.orchestra.run.smithi110.stdout: "mds_epoch": 12, 2023-10-24T23:53:41.708 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch": 27, 2023-10-24T23:53:41.708 INFO:teuthology.orchestra.run.smithi110.stdout: "osd_epoch_barrier": 0, 2023-10-24T23:53:41.708 INFO:teuthology.orchestra.run.smithi110.stdout: "blocklisted": false, 2023-10-24T23:53:41.708 INFO:teuthology.orchestra.run.smithi110.stdout: "fs_name": "b" 2023-10-24T23:53:41.708 INFO:teuthology.orchestra.run.smithi110.stdout:} 2023-10-24T23:53:41.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:41.715 DEBUG:teuthology.orchestra.run.smithi107:> stat --file-system '--printf=%T 2023-10-24T23:53:41.715 DEBUG:teuthology.orchestra.run.smithi107:> ' -- /home/ubuntu/cephtest/mnt.0 2023-10-24T23:53:41.723 INFO:teuthology.orchestra.run.smithi107.stdout:fuseblk 2023-10-24T23:53:41.723 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-10-24T23:53:41.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:41.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-24T23:53:41.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:41.780 DEBUG:teuthology.orchestra.run.smithi110:> stat --file-system '--printf=%T 2023-10-24T23:53:41.781 DEBUG:teuthology.orchestra.run.smithi110:> ' -- /home/ubuntu/cephtest/mnt.1 2023-10-24T23:53:41.787 INFO:teuthology.orchestra.run.smithi110.stdout:fuseblk 2023-10-24T23:53:41.787 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-10-24T23:53:41.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-24T23:53:41.788 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-24T23:53:41.845 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-10-24T23:53:41.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8c2c0640 1 Processor -- start 2023-10-24T23:53:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8c2c0640 1 -- start start 2023-10-24T23:53:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8c2c0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c8414b600 0x7f1c84151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8c2c0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c840c1860 con 0x7f1c8406a230 2023-10-24T23:53:41.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8c2c0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c84154d50 con 0x7f1c8406af70 2023-10-24T23:53:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8c2c0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c841524f0 con 0x7f1c8414b600 2023-10-24T23:53:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8a85d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1c8406a230 0x7f1c841551e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60230/0 (socket says 172.21.15.107:60230) 2023-10-24T23:53:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8a85d640 1 -- 172.21.15.107:0/1523547047 learned_addr learned my addr 172.21.15.107:0/1523547047 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8a05c640 1 --2- 172.21.15.107:0/1523547047 >> v2:172.21.15.110:3300/0 conn(0x7f1c8414b600 0x7f1c84151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562523295 0 0) 0x7f1c840c1860 con 0x7f1c8406a230 2023-10-24T23:53:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c70003530 con 0x7f1c8406a230 2023-10-24T23:53:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2711322994 0 0) 0x7f1c70003530 con 0x7f1c8406a230 2023-10-24T23:53:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 >> v2:172.21.15.110:3300/0 conn(0x7f1c8414b600 msgr2=0x7f1c84151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 --2- 172.21.15.107:0/1523547047 >> v2:172.21.15.110:3300/0 conn(0x7f1c8414b600 0x7f1c84151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 >> v1:172.21.15.110:6789/0 conn(0x7f1c8406af70 legacy=0x7f1c8406c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8414b9e0 con 0x7f1c8406a230 2023-10-24T23:53:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f1c7c001a00 con 0x7f1c8406a230 2023-10-24T23:53:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 >> v1:172.21.15.107:6789/0 conn(0x7f1c8406a230 legacy=0x7f1c841551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 --2- 172.21.15.107:0/1523547047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c70003980 0x7f1c70003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 --2- 172.21.15.107:0/1523547047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c700047d0 0x7f1c70004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 --2- 172.21.15.107:0/1523547047 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c700052b0 0x7f1c7000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:41.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c841524f0 con 0x7f1c70003980 2023-10-24T23:53:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7000c180 con 0x7f1c700047d0 2023-10-24T23:53:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7000c340 con 0x7f1c700052b0 2023-10-24T23:53:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8a05c640 1 --2- 172.21.15.107:0/1523547047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c70003980 0x7f1c70003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8985b640 1 --2- 172.21.15.107:0/1523547047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c700047d0 0x7f1c70004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8a05c640 1 -- 172.21.15.107:0/1523547047 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c700052b0 msgr2=0x7f1c7000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8a05c640 1 --2- 172.21.15.107:0/1523547047 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c700052b0 0x7f1c7000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8a05c640 1 -- 172.21.15.107:0/1523547047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c700047d0 msgr2=0x7f1c70004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8a05c640 1 --2- 172.21.15.107:0/1523547047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c700047d0 0x7f1c70004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8a05c640 1 -- 172.21.15.107:0/1523547047 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c70003530 con 0x7f1c70003980 2023-10-24T23:53:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8a85d640 1 -- 172.21.15.107:0/1523547047 reap_dead start 2023-10-24T23:53:41.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.982+0000 7f1c8985b640 1 --2- 172.21.15.107:0/1523547047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c700047d0 0x7f1c70004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8a05c640 1 --2- 172.21.15.107:0/1523547047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c70003980 0x7f1c70003da0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000ba00 tx=0x7f1c8001c610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e27618a9e4f652 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c800ad410 con 0x7f1c70003980 2023-10-24T23:53:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c800a7070 con 0x7f1c70003980 2023-10-24T23:53:41.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8905a640 1 -- 172.21.15.107:0/1523547047 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c800ad7b0 con 0x7f1c70003980 2023-10-24T23:53:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/1523547047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c70003980 msgr2=0x7f1c70003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 --2- 172.21.15.107:0/1523547047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c70003980 0x7f1c70003da0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000ba00 tx=0x7f1c8001c610 comp rx=0 tx=0).stop 2023-10-24T23:53:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/1523547047 shutdown_connections 2023-10-24T23:53:41.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 --2- 172.21.15.107:0/1523547047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c70003980 0x7f1c70003da0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/1523547047 >> 172.21.15.107:0/1523547047 conn(0x7f1c840c2950 msgr2=0x7f1c840c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/1523547047 shutdown_connections 2023-10-24T23:53:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/1523547047 wait complete. 2023-10-24T23:53:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 Processor -- start 2023-10-24T23:53:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 -- start start 2023-10-24T23:53:41.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c70003980 0x7f1c84153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c84071970 0x7f1c84142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c840c5030 0x7f1c840c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c840721a0 con 0x7f1c840c5030 2023-10-24T23:53:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c841ec840 con 0x7f1c84071970 2023-10-24T23:53:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8c2c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c841ed810 con 0x7f1c70003980 2023-10-24T23:53:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8a85d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c840c5030 0x7f1c840c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8a85d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c840c5030 0x7f1c840c54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56928/0 (socket says 172.21.15.107:56928) 2023-10-24T23:53:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8a85d640 1 -- 172.21.15.107:0/3804639253 learned_addr learned my addr 172.21.15.107:0/3804639253 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.986+0000 7f1c8985b640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c84071970 0x7f1c84142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8a05c640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c70003980 0x7f1c84153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8a85d640 1 -- 172.21.15.107:0/3804639253 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c70003980 msgr2=0x7f1c84153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8a85d640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c70003980 0x7f1c84153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8a85d640 1 -- 172.21.15.107:0/3804639253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c84071970 msgr2=0x7f1c84142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8a85d640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c84071970 0x7f1c84142ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8a85d640 1 -- 172.21.15.107:0/3804639253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c841eca50 con 0x7f1c840c5030 2023-10-24T23:53:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8985b640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c84071970 0x7f1c84142ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8a85d640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c840c5030 0x7f1c840c54b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0012f0 tx=0x7f1c7c00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c7affd640 1 -- 172.21.15.107:0/3804639253 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00a650 con 0x7f1c840c5030 2023-10-24T23:53:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c7affd640 1 -- 172.21.15.107:0/3804639253 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7c00a7b0 con 0x7f1c840c5030 2023-10-24T23:53:42.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/3804639253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c841ecd30 con 0x7f1c840c5030 2023-10-24T23:53:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c7affd640 1 -- 172.21.15.107:0/3804639253 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00ab60 con 0x7f1c840c5030 2023-10-24T23:53:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/3804639253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c841ed2f0 con 0x7f1c840c5030 2023-10-24T23:53:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/3804639253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c841ed510 con 0x7f1c840c5030 2023-10-24T23:53:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c7affd640 1 -- 172.21.15.107:0/3804639253 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1c7c00acc0 con 0x7f1c840c5030 2023-10-24T23:53:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c7affd640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c68065e90 0x7f1c68068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8a05c640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c68065e90 0x7f1c68068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c7affd640 1 -- 172.21.15.107:0/3804639253 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7f1c7c0838d0 con 0x7f1c840c5030 2023-10-24T23:53:42.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.990+0000 7f1c8a05c640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c68065e90 0x7f1c68068350 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1c800af000 tx=0x7f1c800046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:42.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:41.994+0000 7f1c7affd640 1 -- 172.21.15.107:0/3804639253 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1c7c050c80 con 0x7f1c840c5030 2023-10-24T23:53:42.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/3804639253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c840746e0 con 0x7f1c840c5030 2023-10-24T23:53:42.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c7affd640 1 -- 172.21.15.107:0/3804639253 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+21531 (secure 0 0 0) 0x7f1c7c0542a0 con 0x7f1c840c5030 2023-10-24T23:53:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:42.155 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:12.781093+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:42.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/3804639253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c68065e90 msgr2=0x7f1c68068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c68065e90 0x7f1c68068350 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1c800af000 tx=0x7f1c800046d0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/3804639253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c840c5030 msgr2=0x7f1c840c54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c840c5030 0x7f1c840c54b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0012f0 tx=0x7f1c7c00cf00 comp rx=0 tx=0).stop 2023-10-24T23:53:42.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/3804639253 shutdown_connections 2023-10-24T23:53:42.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c840c5030 0x7f1c840c54b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c84071970 0x7f1c84142ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c68065e90 0x7f1c68068350 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 --2- 172.21.15.107:0/3804639253 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c70003980 0x7f1c84153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/3804639253 >> 172.21.15.107:0/3804639253 conn(0x7f1c840c2950 msgr2=0x7f1c8406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:42.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/3804639253 shutdown_connections 2023-10-24T23:53:42.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.150+0000 7f1c8c2c0640 1 -- 172.21.15.107:0/3804639253 wait complete. 2023-10-24T23:53:42.172 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-10-24T23:53:09.786122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-10-24T23:53:10.766645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-24T23:53:42.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-24T23:53:42.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fded0014640 1 Processor -- start 2023-10-24T23:53:42.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fded0014640 1 -- start start 2023-10-24T23:53:42.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fded0014640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdec80ab1a0 0x7fdec80c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fded0014640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdec8072720 0x7fdec80bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fded0014640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdec80bceb0 0x7fdec80c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fded0014640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec80be1d0 con 0x7fdec80ab1a0 2023-10-24T23:53:42.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fded0014640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec80c1ce0 con 0x7fdec80bceb0 2023-10-24T23:53:42.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fded0014640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80c0880 con 0x7fdec8072720 2023-10-24T23:53:42.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fdecddb0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdec80ab1a0 0x7fdec80c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fdecddb0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdec80ab1a0 0x7fdec80c0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56944/0 (socket says 172.21.15.107:56944) 2023-10-24T23:53:42.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fdecddb0640 1 -- 172.21.15.107:0/3955026965 learned_addr learned my addr 172.21.15.107:0/3955026965 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:42.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fdecddb0640 1 -- 172.21.15.107:0/3955026965 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdec8072720 msgr2=0x7fdec80bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:42.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fdecddb0640 1 --2- 172.21.15.107:0/3955026965 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdec8072720 0x7fdec80bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fdecddb0640 1 -- 172.21.15.107:0/3955026965 >> v2:172.21.15.110:3300/0 conn(0x7fdec80bceb0 msgr2=0x7fdec80c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:42.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fdece5b1640 1 --2- 172.21.15.107:0/3955026965 >> v2:172.21.15.110:3300/0 conn(0x7fdec80bceb0 0x7fdec80c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fdecddb0640 1 --2- 172.21.15.107:0/3955026965 >> v2:172.21.15.110:3300/0 conn(0x7fdec80bceb0 0x7fdec80c4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fdecddb0640 1 -- 172.21.15.107:0/3955026965 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec80c1f30 con 0x7fdec80ab1a0 2023-10-24T23:53:42.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.294+0000 7fdecddb0640 1 --2- 172.21.15.107:0/3955026965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec80ab1a0 0x7fdec80c0d00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdec80c1c30 tx=0x7fdec401c110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6ea0c95442fc28 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:42.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdeccdae640 1 -- 172.21.15.107:0/3955026965 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec40ab600 con 0x7fdec80ab1a0 2023-10-24T23:53:42.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdeccdae640 1 -- 172.21.15.107:0/3955026965 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec40a7050 con 0x7fdec80ab1a0 2023-10-24T23:53:42.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdeccdae640 1 -- 172.21.15.107:0/3955026965 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec40ab980 con 0x7fdec80ab1a0 2023-10-24T23:53:42.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 -- 172.21.15.107:0/3955026965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec80ab1a0 msgr2=0x7fdec80c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 --2- 172.21.15.107:0/3955026965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec80ab1a0 0x7fdec80c0d00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdec80c1c30 tx=0x7fdec401c110 comp rx=0 tx=0).stop 2023-10-24T23:53:42.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 -- 172.21.15.107:0/3955026965 shutdown_connections 2023-10-24T23:53:42.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 --2- 172.21.15.107:0/3955026965 >> v2:172.21.15.110:3300/0 conn(0x7fdec80bceb0 0x7fdec80c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 --2- 172.21.15.107:0/3955026965 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdec8072720 0x7fdec80bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 --2- 172.21.15.107:0/3955026965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec80ab1a0 0x7fdec80c0d00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 -- 172.21.15.107:0/3955026965 >> 172.21.15.107:0/3955026965 conn(0x7fdec80c2860 msgr2=0x7fdec80c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 -- 172.21.15.107:0/3955026965 shutdown_connections 2023-10-24T23:53:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 -- 172.21.15.107:0/3955026965 wait complete. 2023-10-24T23:53:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 Processor -- start 2023-10-24T23:53:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 -- start start 2023-10-24T23:53:42.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdec80ab1a0 0x7fdec8155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdec80bceb0 0x7fdec8151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec81531f0 0x7fdec806bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec806a580 con 0x7fdec81531f0 2023-10-24T23:53:42.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec806ce30 con 0x7fdec80bceb0 2023-10-24T23:53:42.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec806cb50 con 0x7fdec80ab1a0 2023-10-24T23:53:42.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdece5b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec81531f0 0x7fdec806bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdece5b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec81531f0 0x7fdec806bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56956/0 (socket says 172.21.15.107:56956) 2023-10-24T23:53:42.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdece5b1640 1 -- 172.21.15.107:0/1337487574 learned_addr learned my addr 172.21.15.107:0/1337487574 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:42.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdece5b1640 1 -- 172.21.15.107:0/1337487574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdec80ab1a0 msgr2=0x7fdec8155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdecddb0640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdec80ab1a0 0x7fdec8155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdece5b1640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdec80ab1a0 0x7fdec8155150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdece5b1640 1 -- 172.21.15.107:0/1337487574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdec80bceb0 msgr2=0x7fdec8151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdecd5af640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdec80bceb0 0x7fdec8151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdece5b1640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdec80bceb0 0x7fdec8151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdece5b1640 1 -- 172.21.15.107:0/1337487574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec40094d0 con 0x7fdec81531f0 2023-10-24T23:53:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdece5b1640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec81531f0 0x7fdec806bf10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdec000cc30 tx=0x7fdec000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdebe7fc640 1 -- 172.21.15.107:0/1337487574 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec000ce40 con 0x7fdec81531f0 2023-10-24T23:53:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdebe7fc640 1 -- 172.21.15.107:0/1337487574 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec00039a0 con 0x7fdec81531f0 2023-10-24T23:53:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 -- 172.21.15.107:0/1337487574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec81eb400 con 0x7fdec81531f0 2023-10-24T23:53:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fdebe7fc640 1 -- 172.21.15.107:0/1337487574 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0003d30 con 0x7fdec81531f0 2023-10-24T23:53:42.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.298+0000 7fded0014640 1 -- 172.21.15.107:0/1337487574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec806a1e0 con 0x7fdec81531f0 2023-10-24T23:53:42.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.302+0000 7fded0014640 1 -- 172.21.15.107:0/1337487574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec80a70e0 con 0x7fdec81531f0 2023-10-24T23:53:42.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.302+0000 7fdebe7fc640 1 -- 172.21.15.107:0/1337487574 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdec0003e90 con 0x7fdec81531f0 2023-10-24T23:53:42.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.302+0000 7fdebe7fc640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdeac065e90 0x7fdeac068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.302+0000 7fdebe7fc640 1 -- 172.21.15.107:0/1337487574 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fdec0083e90 con 0x7fdec81531f0 2023-10-24T23:53:42.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.302+0000 7fdecddb0640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdeac065e90 0x7fdeac068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.302+0000 7fdecddb0640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdeac065e90 0x7fdeac068350 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdec40ad000 tx=0x7fdec4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:42.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.306+0000 7fdebe7fc640 1 -- 172.21.15.107:0/1337487574 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdec80a70e0 con 0x7fdec81531f0 2023-10-24T23:53:42.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 -- 172.21.15.107:0/1337487574 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdec80a70e0 con 0x7fdec81531f0 2023-10-24T23:53:42.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fdebe7fc640 1 -- 172.21.15.107:0/1337487574 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdec80a70e0 con 0x7fdec81531f0 2023-10-24T23:53:42.466 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-24T23:53:42.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 -- 172.21.15.107:0/1337487574 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdeac065e90 msgr2=0x7fdeac068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdeac065e90 0x7fdeac068350 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdec40ad000 tx=0x7fdec4008040 comp rx=0 tx=0).stop 2023-10-24T23:53:42.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 -- 172.21.15.107:0/1337487574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec81531f0 msgr2=0x7fdec806bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec81531f0 0x7fdec806bf10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdec000cc30 tx=0x7fdec000a5c0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 -- 172.21.15.107:0/1337487574 shutdown_connections 2023-10-24T23:53:42.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdec81531f0 0x7fdec806bf10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdec80bceb0 0x7fdec8151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdeac065e90 0x7fdeac068350 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 --2- 172.21.15.107:0/1337487574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdec80ab1a0 0x7fdec8155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 -- 172.21.15.107:0/1337487574 >> 172.21.15.107:0/1337487574 conn(0x7fdec80c2860 msgr2=0x7fdec80c4240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:42.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 -- 172.21.15.107:0/1337487574 shutdown_connections 2023-10-24T23:53:42.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.462+0000 7fded0014640 1 -- 172.21.15.107:0/1337487574 wait complete. 2023-10-24T23:53:42.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-24T23:53:42.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad2cb6c640 1 Processor -- start 2023-10-24T23:53:42.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad2cb6c640 1 -- start start 2023-10-24T23:53:42.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad2cb6c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad280c1b30 0x7fad28072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad2cb6c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fad280a9300 0x7fad280aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad2cb6c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fad280aa820 0x7fad280c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad2cb6c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad280befa0 con 0x7fad280aa820 2023-10-24T23:53:42.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad2cb6c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad280a9b80 con 0x7fad280a9300 2023-10-24T23:53:42.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad2cb6c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad280aa0b0 con 0x7fad280c1b30 2023-10-24T23:53:42.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad26d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fad280aa820 0x7fad280c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad26d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fad280aa820 0x7fad280c4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36788/0 (socket says 172.21.15.107:36788) 2023-10-24T23:53:42.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad26d9d640 1 -- 172.21.15.107:0/1050640453 learned_addr learned my addr 172.21.15.107:0/1050640453 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:42.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad2659c640 1 --2- 172.21.15.107:0/1050640453 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad280c1b30 0x7fad28072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad1dd9b640 1 --2- 172.21.15.107:0/1050640453 >> v2:172.21.15.110:3300/0 conn(0x7fad280a9300 0x7fad280aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad26d9d640 1 -- 172.21.15.107:0/1050640453 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad280c1b30 msgr2=0x7fad28072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:42.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad26d9d640 1 --2- 172.21.15.107:0/1050640453 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad280c1b30 0x7fad28072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad26d9d640 1 -- 172.21.15.107:0/1050640453 >> v2:172.21.15.110:3300/0 conn(0x7fad280a9300 msgr2=0x7fad280aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:42.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad26d9d640 1 --2- 172.21.15.107:0/1050640453 >> v2:172.21.15.110:3300/0 conn(0x7fad280a9300 0x7fad280aa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad26d9d640 1 -- 172.21.15.107:0/1050640453 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad280bf1f0 con 0x7fad280aa820 2023-10-24T23:53:42.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.606+0000 7fad1dd9b640 1 --2- 172.21.15.107:0/1050640453 >> v2:172.21.15.110:3300/0 conn(0x7fad280a9300 0x7fad280aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:42.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad26d9d640 1 --2- 172.21.15.107:0/1050640453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad280aa820 0x7fad280c4cf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fad2806e520 tx=0x7fad2001c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f93bd25ba80879f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:42.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad25d9b640 1 -- 172.21.15.107:0/1050640453 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad200ab400 con 0x7fad280aa820 2023-10-24T23:53:42.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad25d9b640 1 -- 172.21.15.107:0/1050640453 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad200a7050 con 0x7fad280aa820 2023-10-24T23:53:42.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad25d9b640 1 -- 172.21.15.107:0/1050640453 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad200ab780 con 0x7fad280aa820 2023-10-24T23:53:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1050640453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad280aa820 msgr2=0x7fad280c4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad2cb6c640 1 --2- 172.21.15.107:0/1050640453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad280aa820 0x7fad280c4cf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fad2806e520 tx=0x7fad2001c5f0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1050640453 shutdown_connections 2023-10-24T23:53:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad2cb6c640 1 --2- 172.21.15.107:0/1050640453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad280aa820 0x7fad280c4cf0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad2cb6c640 1 --2- 172.21.15.107:0/1050640453 >> v2:172.21.15.110:3300/0 conn(0x7fad280a9300 0x7fad280aa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad2cb6c640 1 --2- 172.21.15.107:0/1050640453 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad280c1b30 0x7fad28072210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1050640453 >> 172.21.15.107:0/1050640453 conn(0x7fad280c2860 msgr2=0x7fad280c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:42.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1050640453 shutdown_connections 2023-10-24T23:53:42.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.610+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1050640453 wait complete. 2023-10-24T23:53:42.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2cb6c640 1 Processor -- start 2023-10-24T23:53:42.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2cb6c640 1 -- start start 2023-10-24T23:53:42.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2cb6c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad280a9300 0x7fad280bb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2cb6c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad280aa820 0x7fad28159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2cb6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad280c1b30 0x7fad2815a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2cb6c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2806fcb0 con 0x7fad280c1b30 2023-10-24T23:53:42.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2cb6c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad281565f0 con 0x7fad280aa820 2023-10-24T23:53:42.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2cb6c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2815bf70 con 0x7fad280a9300 2023-10-24T23:53:42.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad26d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad280c1b30 0x7fad2815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad26d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad280c1b30 0x7fad2815a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36800/0 (socket says 172.21.15.107:36800) 2023-10-24T23:53:42.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad26d9d640 1 -- 172.21.15.107:0/1001948730 learned_addr learned my addr 172.21.15.107:0/1001948730 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:42.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad1dd9b640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad280aa820 0x7fad28159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2659c640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad280a9300 0x7fad280bb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad26d9d640 1 -- 172.21.15.107:0/1001948730 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad280a9300 msgr2=0x7fad280bb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad26d9d640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad280a9300 0x7fad280bb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad26d9d640 1 -- 172.21.15.107:0/1001948730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad280aa820 msgr2=0x7fad28159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad26d9d640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad280aa820 0x7fad28159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad26d9d640 1 -- 172.21.15.107:0/1001948730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad20009b30 con 0x7fad280c1b30 2023-10-24T23:53:42.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad1dd9b640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad280aa820 0x7fad28159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:42.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2659c640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad280a9300 0x7fad280bb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:42.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad26d9d640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad280c1b30 0x7fad2815a520 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fad2001aca0 tx=0x7fad20019390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:42.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad1f7fe640 1 -- 172.21.15.107:0/1001948730 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad200abdc0 con 0x7fad280c1b30 2023-10-24T23:53:42.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad1f7fe640 1 -- 172.21.15.107:0/1001948730 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad20009510 con 0x7fad280c1b30 2023-10-24T23:53:42.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1001948730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad2815c250 con 0x7fad280c1b30 2023-10-24T23:53:42.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad1f7fe640 1 -- 172.21.15.107:0/1001948730 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad200b6660 con 0x7fad280c1b30 2023-10-24T23:53:42.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1001948730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2806f940 con 0x7fad280c1b30 2023-10-24T23:53:42.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1001948730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2806fe60 con 0x7fad280c1b30 2023-10-24T23:53:42.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad1f7fe640 1 -- 172.21.15.107:0/1001948730 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fad200b2040 con 0x7fad280c1b30 2023-10-24T23:53:42.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad1f7fe640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad08065e60 0x7fad08068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad2659c640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad08065e60 0x7fad08068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.614+0000 7fad1f7fe640 1 -- 172.21.15.107:0/1001948730 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fad2011e0d0 con 0x7fad280c1b30 2023-10-24T23:53:42.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.618+0000 7fad2659c640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad08065e60 0x7fad08068320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad180097a0 tx=0x7fad18008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:42.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.618+0000 7fad1f7fe640 1 -- 172.21.15.107:0/1001948730 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fad2806fe60 con 0x7fad280c1b30 2023-10-24T23:53:42.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1001948730 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fad2806fdf0 con 0x7fad280c1b30 2023-10-24T23:53:42.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad1f7fe640 1 -- 172.21.15.107:0/1001948730 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7fad200eb510 con 0x7fad280c1b30 2023-10-24T23:53:42.778 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-24T23:53:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1001948730 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad08065e60 msgr2=0x7fad08068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad08065e60 0x7fad08068320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fad180097a0 tx=0x7fad18008040 comp rx=0 tx=0).stop 2023-10-24T23:53:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1001948730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad280c1b30 msgr2=0x7fad2815a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad280c1b30 0x7fad2815a520 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fad2001aca0 tx=0x7fad20019390 comp rx=0 tx=0).stop 2023-10-24T23:53:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1001948730 shutdown_connections 2023-10-24T23:53:42.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad280c1b30 0x7fad2815a520 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad280aa820 0x7fad28159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fad08065e60 0x7fad08068320 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 --2- 172.21.15.107:0/1001948730 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad280a9300 0x7fad280bb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1001948730 >> 172.21.15.107:0/1001948730 conn(0x7fad280c2860 msgr2=0x7fad280c2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:42.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1001948730 shutdown_connections 2023-10-24T23:53:42.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.774+0000 7fad2cb6c640 1 -- 172.21.15.107:0/1001948730 wait complete. 2023-10-24T23:53:42.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-24T23:53:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.898+0000 7fd40bee9640 1 Processor -- start 2023-10-24T23:53:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.898+0000 7fd40bee9640 1 -- start start 2023-10-24T23:53:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd40bee9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd40406b360 0x7fd404069d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd40bee9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4040c8180 con 0x7fd4040bd930 2023-10-24T23:53:42.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd40bee9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd404155430 con 0x7fd4040af9b0 2023-10-24T23:53:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd40bee9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4040c5870 con 0x7fd40406b360 2023-10-24T23:53:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd409484640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd40406b360 0x7fd404069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd40a486640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd4040bd930 0x7fd40415f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45052/0 (socket says 172.21.15.107:45052) 2023-10-24T23:53:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd40a486640 1 -- 172.21.15.107:0/3883521307 learned_addr learned my addr 172.21.15.107:0/3883521307 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:42.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd409484640 1 -- 172.21.15.107:0/3883521307 >> v1:172.21.15.110:6789/0 conn(0x7fd4040af9b0 legacy=0x7fd4040c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd409484640 1 -- 172.21.15.107:0/3883521307 >> v1:172.21.15.107:6789/0 conn(0x7fd4040bd930 legacy=0x7fd40415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd409484640 1 -- 172.21.15.107:0/3883521307 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd404155710 con 0x7fd40406b360 2023-10-24T23:53:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd408c83640 1 -- 172.21.15.107:0/3883521307 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286294224 0 0) 0x7fd4040c8180 con 0x7fd4040bd930 2023-10-24T23:53:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd409484640 1 --2- 172.21.15.107:0/3883521307 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40406b360 0x7fd404069d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40097c0 tx=0x7fd3f401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9582d7cf2c78c90 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd408c83640 1 -- 172.21.15.107:0/3883521307 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f40ac640 con 0x7fd40406b360 2023-10-24T23:53:42.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd408c83640 1 -- 172.21.15.107:0/3883521307 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f40ac7a0 con 0x7fd40406b360 2023-10-24T23:53:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd408c83640 1 -- 172.21.15.107:0/3883521307 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f40acb30 con 0x7fd40406b360 2023-10-24T23:53:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd40bee9640 1 -- 172.21.15.107:0/3883521307 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40406b360 msgr2=0x7fd404069d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd40bee9640 1 --2- 172.21.15.107:0/3883521307 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40406b360 0x7fd404069d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40097c0 tx=0x7fd3f401c380 comp rx=0 tx=0).stop 2023-10-24T23:53:42.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd40bee9640 1 -- 172.21.15.107:0/3883521307 shutdown_connections 2023-10-24T23:53:42.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd40bee9640 1 --2- 172.21.15.107:0/3883521307 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40406b360 0x7fd404069d40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.902+0000 7fd40bee9640 1 -- 172.21.15.107:0/3883521307 >> 172.21.15.107:0/3883521307 conn(0x7fd404071b00 msgr2=0x7fd4040abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:42.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 -- 172.21.15.107:0/3883521307 shutdown_connections 2023-10-24T23:53:42.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 -- 172.21.15.107:0/3883521307 wait complete. 2023-10-24T23:53:42.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 Processor -- start 2023-10-24T23:53:42.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 -- start start 2023-10-24T23:53:42.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40406b360 0x7fd40414fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4040af9b0 0x7fd4040abaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4040bd930 0x7fd404144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4041507d0 con 0x7fd40406b360 2023-10-24T23:53:42.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4040c3d20 con 0x7fd4040af9b0 2023-10-24T23:53:42.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4040aaec0 con 0x7fd4040bd930 2023-10-24T23:53:42.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd409c85640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40406b360 0x7fd40414fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd409c85640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40406b360 0x7fd40414fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36824/0 (socket says 172.21.15.107:36824) 2023-10-24T23:53:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd409c85640 1 -- 172.21.15.107:0/1368829522 learned_addr learned my addr 172.21.15.107:0/1368829522 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd409c85640 1 -- 172.21.15.107:0/1368829522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4040bd930 msgr2=0x7fd404144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd409c85640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4040bd930 0x7fd404144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd409c85640 1 -- 172.21.15.107:0/1368829522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4040af9b0 msgr2=0x7fd4040abaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:42.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd409c85640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4040af9b0 0x7fd4040abaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:42.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd409c85640 1 -- 172.21.15.107:0/1368829522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f4009420 con 0x7fd40406b360 2023-10-24T23:53:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd409c85640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40406b360 0x7fd40414fb50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd40000c450 tx=0x7fd40000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd3fa7fc640 1 -- 172.21.15.107:0/1368829522 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400012830 con 0x7fd40406b360 2023-10-24T23:53:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd3fa7fc640 1 -- 172.21.15.107:0/1368829522 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4000129b0 con 0x7fd40406b360 2023-10-24T23:53:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 -- 172.21.15.107:0/1368829522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd404148930 con 0x7fd40406b360 2023-10-24T23:53:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd3fa7fc640 1 -- 172.21.15.107:0/1368829522 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400012d60 con 0x7fd40406b360 2023-10-24T23:53:42.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 -- 172.21.15.107:0/1368829522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd404151ae0 con 0x7fd40406b360 2023-10-24T23:53:42.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd40bee9640 1 -- 172.21.15.107:0/1368829522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd404151920 con 0x7fd40406b360 2023-10-24T23:53:42.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.906+0000 7fd3fa7fc640 1 -- 172.21.15.107:0/1368829522 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd40001c460 con 0x7fd40406b360 2023-10-24T23:53:42.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.910+0000 7fd3fa7fc640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3e8065e60 0x7fd3e8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:42.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.910+0000 7fd3fa7fc640 1 -- 172.21.15.107:0/1368829522 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fd400083a60 con 0x7fd40406b360 2023-10-24T23:53:42.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.910+0000 7fd409484640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3e8065e60 0x7fd3e8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:42.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.910+0000 7fd409484640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3e8065e60 0x7fd3e8068320 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40ae000 tx=0x7fd3f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:42.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:42.914+0000 7fd3fa7fc640 1 -- 172.21.15.107:0/1368829522 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd400050d80 con 0x7fd40406b360 2023-10-24T23:53:43.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.050+0000 7fd40bee9640 1 -- 172.21.15.107:0/1368829522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd404066560 con 0x7fd40406b360 2023-10-24T23:53:43.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd3fa7fc640 1 -- 172.21.15.107:0/1368829522 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) v1 ==== 99+0+11 (secure 0 0 0) 0x7fd4000543a0 con 0x7fd40406b360 2023-10-24T23:53:43.058 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:43.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 -- 172.21.15.107:0/1368829522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3e8065e60 msgr2=0x7fd3e8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3e8065e60 0x7fd3e8068320 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40ae000 tx=0x7fd3f4008040 comp rx=0 tx=0).stop 2023-10-24T23:53:43.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 -- 172.21.15.107:0/1368829522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40406b360 msgr2=0x7fd40414fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40406b360 0x7fd40414fb50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd40000c450 tx=0x7fd40000c910 comp rx=0 tx=0).stop 2023-10-24T23:53:43.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 -- 172.21.15.107:0/1368829522 shutdown_connections 2023-10-24T23:53:43.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3e8065e60 0x7fd3e8068320 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4040bd930 0x7fd404144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4040af9b0 0x7fd4040abaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 --2- 172.21.15.107:0/1368829522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd40406b360 0x7fd40414fb50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 -- 172.21.15.107:0/1368829522 >> 172.21.15.107:0/1368829522 conn(0x7fd404071b00 msgr2=0x7fd4040acf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:43.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 -- 172.21.15.107:0/1368829522 shutdown_connections 2023-10-24T23:53:43.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.054+0000 7fd40bee9640 1 -- 172.21.15.107:0/1368829522 wait complete. 2023-10-24T23:53:43.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-24T23:53:43.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4bb971640 1 Processor -- start 2023-10-24T23:53:43.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4bb971640 1 -- start start 2023-10-24T23:53:43.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4bb971640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b40bceb0 0x7fb4b40c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4bb971640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b40c1ce0 con 0x7fb4b4072720 2023-10-24T23:53:43.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4bb971640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b40c0240 con 0x7fb4b40ab1a0 2023-10-24T23:53:43.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4bb971640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40be1d0 con 0x7fb4b40bceb0 2023-10-24T23:53:43.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4b8f0c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb4b4072720 0x7fb4b415a650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45060/0 (socket says 172.21.15.107:45060) 2023-10-24T23:53:43.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4b8f0c640 1 -- 172.21.15.107:0/977975180 learned_addr learned my addr 172.21.15.107:0/977975180 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:43.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4b9f0e640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b40bceb0 0x7fb4b40c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366861735 0 0) 0x7fb4b40c1ce0 con 0x7fb4b4072720 2023-10-24T23:53:43.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a0003530 con 0x7fb4b4072720 2023-10-24T23:53:43.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2481551582 0 0) 0x7fb4a0003530 con 0x7fb4b4072720 2023-10-24T23:53:43.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b40bceb0 msgr2=0x7fb4b40c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:43.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b40bceb0 0x7fb4b40c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 >> v1:172.21.15.110:6789/0 conn(0x7fb4b40ab1a0 legacy=0x7fb4b40bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b40c0880 con 0x7fb4b4072720 2023-10-24T23:53:43.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb4a4001680 con 0x7fb4b4072720 2023-10-24T23:53:43.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 >> v1:172.21.15.107:6789/0 conn(0x7fb4b4072720 legacy=0x7fb4b415a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 0x7fb4a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b40bceb0 0x7fb4a00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a0004d10 0x7fb4a0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a000bb20 con 0x7fb4a00039b0 2023-10-24T23:53:43.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a000bc90 con 0x7fb4a0004d10 2023-10-24T23:53:43.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a000be50 con 0x7fb4b40bceb0 2023-10-24T23:53:43.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4b970d640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 0x7fb4a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4b9f0e640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a0004d10 0x7fb4a0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4b970d640 1 -- 172.21.15.107:0/977975180 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b40bceb0 msgr2=0x7fb4a00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:43.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4b970d640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b40bceb0 0x7fb4a00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4b970d640 1 -- 172.21.15.107:0/977975180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a0004d10 msgr2=0x7fb4a0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:43.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4b970d640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a0004d10 0x7fb4a0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4b970d640 1 -- 172.21.15.107:0/977975180 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a0003530 con 0x7fb4a00039b0 2023-10-24T23:53:43.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.182+0000 7fb4b9f0e640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4a0004d10 0x7fb4a0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:43.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b970d640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 0x7fb4a0003dd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0001780 tx=0x7fb4b001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd29b1e6441d28f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:43.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b00052d0 con 0x7fb4a00039b0 2023-10-24T23:53:43.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b00a7050 con 0x7fb4a00039b0 2023-10-24T23:53:43.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4abfff640 1 -- 172.21.15.107:0/977975180 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0003600 con 0x7fb4a00039b0 2023-10-24T23:53:43.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 -- 172.21.15.107:0/977975180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 msgr2=0x7fb4a0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 --2- 172.21.15.107:0/977975180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 0x7fb4a0003dd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0001780 tx=0x7fb4b001c3d0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b9f0e640 1 -- 172.21.15.107:0/977975180 reap_dead start 2023-10-24T23:53:43.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 -- 172.21.15.107:0/977975180 shutdown_connections 2023-10-24T23:53:43.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 -- 172.21.15.107:0/977975180 >> 172.21.15.107:0/977975180 conn(0x7fb4b40c2860 msgr2=0x7fb4b40c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:43.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 -- 172.21.15.107:0/977975180 shutdown_connections 2023-10-24T23:53:43.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 -- 172.21.15.107:0/977975180 wait complete. 2023-10-24T23:53:43.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 Processor -- start 2023-10-24T23:53:43.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 -- start start 2023-10-24T23:53:43.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 0x7fb4b40c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b414ba40 0x7fb4b414bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4b40a62f0 0x7fb4b40a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40aa960 con 0x7fb4a00039b0 2023-10-24T23:53:43.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b40a5300 con 0x7fb4b40a62f0 2023-10-24T23:53:43.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b414c400 con 0x7fb4b414ba40 2023-10-24T23:53:43.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b970d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 0x7fb4b40c4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b970d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 0x7fb4b40c4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36854/0 (socket says 172.21.15.107:36854) 2023-10-24T23:53:43.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b970d640 1 -- 172.21.15.107:0/487565328 learned_addr learned my addr 172.21.15.107:0/487565328 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:43.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b9f0e640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4b40a62f0 0x7fb4b40a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b8f0c640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b414ba40 0x7fb4b414bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b970d640 1 -- 172.21.15.107:0/487565328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b414ba40 msgr2=0x7fb4b414bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b970d640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b414ba40 0x7fb4b414bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b970d640 1 -- 172.21.15.107:0/487565328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4b40a62f0 msgr2=0x7fb4b40a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b970d640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4b40a62f0 0x7fb4b40a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b970d640 1 -- 172.21.15.107:0/487565328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b40a5df0 con 0x7fb4a00039b0 2023-10-24T23:53:43.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4b970d640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 0x7fb4b40c4c40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0005040 tx=0x7fb4a0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:43.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4a9ffb640 1 -- 172.21.15.107:0/487565328 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0007040 con 0x7fb4a00039b0 2023-10-24T23:53:43.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4a9ffb640 1 -- 172.21.15.107:0/487565328 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b00a4e90 con 0x7fb4a00039b0 2023-10-24T23:53:43.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4bb971640 1 -- 172.21.15.107:0/487565328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b406f840 con 0x7fb4a00039b0 2023-10-24T23:53:43.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.186+0000 7fb4a9ffb640 1 -- 172.21.15.107:0/487565328 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0007420 con 0x7fb4a00039b0 2023-10-24T23:53:43.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.190+0000 7fb4bb971640 1 -- 172.21.15.107:0/487565328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b40a5830 con 0x7fb4a00039b0 2023-10-24T23:53:43.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.190+0000 7fb4bb971640 1 -- 172.21.15.107:0/487565328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b40a74d0 con 0x7fb4a00039b0 2023-10-24T23:53:43.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.190+0000 7fb4a9ffb640 1 -- 172.21.15.107:0/487565328 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb4b0007580 con 0x7fb4a00039b0 2023-10-24T23:53:43.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.190+0000 7fb4a9ffb640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb498065e60 0x7fb498068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.190+0000 7fb4a9ffb640 1 -- 172.21.15.107:0/487565328 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6874+0+0 (secure 0 0 0) 0x7fb4b011dd20 con 0x7fb4a00039b0 2023-10-24T23:53:43.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.190+0000 7fb4b8f0c640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb498065e60 0x7fb498068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.190+0000 7fb4b8f0c640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb498065e60 0x7fb498068320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40aaf10 tx=0x7fb4a40014e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:43.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.194+0000 7fb4a9ffb640 1 -- 172.21.15.107:0/487565328 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb4b00eeb60 con 0x7fb4a00039b0 2023-10-24T23:53:43.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 -- 172.21.15.107:0/487565328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7fb4b404e130 con 0x7fb4a00039b0 2023-10-24T23:53:43.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4a9ffb640 1 -- 172.21.15.107:0/487565328 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) v1 ==== 95+0+11 (secure 0 0 0) 0x7fb4b00ee6f0 con 0x7fb4a00039b0 2023-10-24T23:53:43.354 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:43.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 -- 172.21.15.107:0/487565328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb498065e60 msgr2=0x7fb498068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb498065e60 0x7fb498068320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40aaf10 tx=0x7fb4a40014e0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 -- 172.21.15.107:0/487565328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 msgr2=0x7fb4b40c4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 0x7fb4b40c4c40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0005040 tx=0x7fb4a0005800 comp rx=0 tx=0).stop 2023-10-24T23:53:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 -- 172.21.15.107:0/487565328 shutdown_connections 2023-10-24T23:53:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4b40a62f0 0x7fb4b40a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb498065e60 0x7fb498068320 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4b414ba40 0x7fb4b414bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 --2- 172.21.15.107:0/487565328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4a00039b0 0x7fb4b40c4c40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 -- 172.21.15.107:0/487565328 >> 172.21.15.107:0/487565328 conn(0x7fb4b40c2860 msgr2=0x7fb4b40715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 -- 172.21.15.107:0/487565328 shutdown_connections 2023-10-24T23:53:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.350+0000 7fb4bb971640 1 -- 172.21.15.107:0/487565328 wait complete. 2023-10-24T23:53:43.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-10-24T23:53:43.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.490+0000 7f8cdc1b0640 1 Processor -- start 2023-10-24T23:53:43.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.490+0000 7f8cdc1b0640 1 -- start start 2023-10-24T23:53:43.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.490+0000 7f8cdc1b0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8cd40af9b0 0x7f8cd40c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.490+0000 7f8cdc1b0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cd40bd930 0x7f8cd415f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.490+0000 7f8cdc1b0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd40c8180 con 0x7f8cd406b360 2023-10-24T23:53:43.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.490+0000 7f8cdc1b0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd40c5870 con 0x7f8cd40af9b0 2023-10-24T23:53:43.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.490+0000 7f8cdc1b0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd4155430 con 0x7f8cd40bd930 2023-10-24T23:53:43.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cd9f4c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8cd40af9b0 0x7f8cd40c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cd9f4c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8cd40af9b0 0x7f8cd40c6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36864/0 (socket says 172.21.15.107:36864) 2023-10-24T23:53:43.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cd9f4c640 1 -- 172.21.15.107:0/833930318 learned_addr learned my addr 172.21.15.107:0/833930318 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:43.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cd9f4c640 1 -- 172.21.15.107:0/833930318 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cd40bd930 msgr2=0x7f8cd415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:43.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cda74d640 1 --2- 172.21.15.107:0/833930318 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cd40bd930 0x7f8cd415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cd9f4c640 1 --2- 172.21.15.107:0/833930318 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cd40bd930 0x7f8cd415f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cd9f4c640 1 -- 172.21.15.107:0/833930318 >> v1:172.21.15.110:6789/0 conn(0x7f8cd406b360 legacy=0x7f8cd4069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cd9f4c640 1 -- 172.21.15.107:0/833930318 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd4155710 con 0x7f8cd40af9b0 2023-10-24T23:53:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cd9f4c640 1 --2- 172.21.15.107:0/833930318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd40af9b0 0x7f8cd40c6c90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0009870 tx=0x7f8cd001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990dd832d3a50048 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cd8f4a640 1 -- 172.21.15.107:0/833930318 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd00ad640 con 0x7f8cd40af9b0 2023-10-24T23:53:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cd8f4a640 1 -- 172.21.15.107:0/833930318 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd00ad7a0 con 0x7f8cd40af9b0 2023-10-24T23:53:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cd8f4a640 1 -- 172.21.15.107:0/833930318 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd00adb30 con 0x7f8cd40af9b0 2023-10-24T23:53:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/833930318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd40af9b0 msgr2=0x7f8cd40c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cdc1b0640 1 --2- 172.21.15.107:0/833930318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd40af9b0 0x7f8cd40c6c90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0009870 tx=0x7f8cd001c440 comp rx=0 tx=0).stop 2023-10-24T23:53:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/833930318 shutdown_connections 2023-10-24T23:53:43.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cdc1b0640 1 --2- 172.21.15.107:0/833930318 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cd40bd930 0x7f8cd415f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cdc1b0640 1 --2- 172.21.15.107:0/833930318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd40af9b0 0x7f8cd40c6c90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/833930318 >> 172.21.15.107:0/833930318 conn(0x7f8cd4071b00 msgr2=0x7f8cd40abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/833930318 shutdown_connections 2023-10-24T23:53:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.494+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/833930318 wait complete. 2023-10-24T23:53:43.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cdc1b0640 1 Processor -- start 2023-10-24T23:53:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cdc1b0640 1 -- start start 2023-10-24T23:53:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cdc1b0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cd406b360 0x7f8cd4149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cdc1b0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cd40af9b0 0x7f8cd414a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cdc1b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd40bd930 0x7f8cd41504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cdc1b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd41ee640 con 0x7f8cd40bd930 2023-10-24T23:53:43.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cdc1b0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd41eda30 con 0x7f8cd406b360 2023-10-24T23:53:43.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cdc1b0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd41edd10 con 0x7f8cd40af9b0 2023-10-24T23:53:43.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cda74d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd40bd930 0x7f8cd41504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cda74d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd40bd930 0x7f8cd41504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36866/0 (socket says 172.21.15.107:36866) 2023-10-24T23:53:43.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cda74d640 1 -- 172.21.15.107:0/2382388030 learned_addr learned my addr 172.21.15.107:0/2382388030 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:43.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cd9f4c640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cd406b360 0x7f8cd4149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cd974b640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cd40af9b0 0x7f8cd414a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cda74d640 1 -- 172.21.15.107:0/2382388030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cd40af9b0 msgr2=0x7f8cd414a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cda74d640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cd40af9b0 0x7f8cd414a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cda74d640 1 -- 172.21.15.107:0/2382388030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cd406b360 msgr2=0x7f8cd4149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cda74d640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cd406b360 0x7f8cd4149b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cda74d640 1 -- 172.21.15.107:0/2382388030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd00094d0 con 0x7f8cd40bd930 2023-10-24T23:53:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cda74d640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd40bd930 0x7f8cd41504a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00cc80 tx=0x7f8ccc00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cca7fc640 1 -- 172.21.15.107:0/2382388030 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc00ce30 con 0x7f8cd40bd930 2023-10-24T23:53:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cca7fc640 1 -- 172.21.15.107:0/2382388030 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc0039a0 con 0x7f8cd40bd930 2023-10-24T23:53:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/2382388030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd41edff0 con 0x7f8cd40bd930 2023-10-24T23:53:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cca7fc640 1 -- 172.21.15.107:0/2382388030 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc003d90 con 0x7f8cd40bd930 2023-10-24T23:53:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/2382388030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd4152550 con 0x7f8cd40bd930 2023-10-24T23:53:43.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/2382388030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd41ee7b0 con 0x7f8cd40bd930 2023-10-24T23:53:43.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cca7fc640 1 -- 172.21.15.107:0/2382388030 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8ccc003ef0 con 0x7f8cd40bd930 2023-10-24T23:53:43.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cca7fc640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cb8065e60 0x7f8cb8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cd9f4c640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cb8065e60 0x7f8cb8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.498+0000 7f8cca7fc640 1 -- 172.21.15.107:0/2382388030 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f8ccc084200 con 0x7f8cd40bd930 2023-10-24T23:53:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.502+0000 7f8cd9f4c640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cb8065e60 0x7f8cb8068320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00af000 tx=0x7f8cd00a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.502+0000 7f8cca7fc640 1 -- 172.21.15.107:0/2382388030 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8ccc051550 con 0x7f8cd40bd930 2023-10-24T23:53:43.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/2382388030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8cd4066560 con 0x7f8cd40bd930 2023-10-24T23:53:43.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cca7fc640 1 -- 172.21.15.107:0/2382388030 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v28) v1 ==== 99+0+11 (secure 0 0 0) 0x7f8ccc01d020 con 0x7f8cd40bd930 2023-10-24T23:53:43.662 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:43.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/2382388030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cb8065e60 msgr2=0x7f8cb8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cb8065e60 0x7f8cb8068320 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00af000 tx=0x7f8cd00a7070 comp rx=0 tx=0).stop 2023-10-24T23:53:43.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/2382388030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd40bd930 msgr2=0x7f8cd41504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd40bd930 0x7f8cd41504a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00cc80 tx=0x7f8ccc00a5c0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/2382388030 shutdown_connections 2023-10-24T23:53:43.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8cd40bd930 0x7f8cd41504a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8cb8065e60 0x7f8cb8068320 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8cd40af9b0 0x7f8cd414a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 --2- 172.21.15.107:0/2382388030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8cd406b360 0x7f8cd4149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/2382388030 >> 172.21.15.107:0/2382388030 conn(0x7f8cd4071b00 msgr2=0x7f8cd414df20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:43.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/2382388030 shutdown_connections 2023-10-24T23:53:43.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.658+0000 7f8cdc1b0640 1 -- 172.21.15.107:0/2382388030 wait complete. 2023-10-24T23:53:43.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-10-24T23:53:43.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.798+0000 7fab75532640 1 Processor -- start 2023-10-24T23:53:43.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.798+0000 7fab75532640 1 -- start start 2023-10-24T23:53:43.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.798+0000 7fab75532640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fab700ab1a0 0x7fab700c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.798+0000 7fab75532640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fab70072720 0x7fab700bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.798+0000 7fab75532640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab700c1ce0 con 0x7fab700bceb0 2023-10-24T23:53:43.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.798+0000 7fab75532640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab700be1d0 con 0x7fab700ab1a0 2023-10-24T23:53:43.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.798+0000 7fab75532640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab700c0880 con 0x7fab70072720 2023-10-24T23:53:43.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fab700ab1a0 0x7fab700c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fab700bceb0 0x7fab700c4cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45064/0 (socket says 172.21.15.107:45064) 2023-10-24T23:53:43.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6f7fe640 1 -- 172.21.15.107:0/698368425 learned_addr learned my addr 172.21.15.107:0/698368425 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:43.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6effd640 1 -- 172.21.15.107:0/698368425 >> v2:172.21.15.110:3300/0 conn(0x7fab70072720 msgr2=0x7fab700bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:43.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6effd640 1 --2- 172.21.15.107:0/698368425 >> v2:172.21.15.110:3300/0 conn(0x7fab70072720 0x7fab700bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6effd640 1 -- 172.21.15.107:0/698368425 >> v1:172.21.15.107:6789/0 conn(0x7fab700bceb0 legacy=0x7fab700c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6effd640 1 -- 172.21.15.107:0/698368425 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab700c1f30 con 0x7fab700ab1a0 2023-10-24T23:53:43.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6e7fc640 1 -- 172.21.15.107:0/698368425 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517556530 0 0) 0x7fab700c1ce0 con 0x7fab700bceb0 2023-10-24T23:53:43.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6effd640 1 --2- 172.21.15.107:0/698368425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab700ab1a0 0x7fab700c0d00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fab70071e30 tx=0x7fab6001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31342037e5b93fa server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:43.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6e7fc640 1 -- 172.21.15.107:0/698368425 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab600ab600 con 0x7fab700ab1a0 2023-10-24T23:53:43.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6e7fc640 1 -- 172.21.15.107:0/698368425 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab600a7050 con 0x7fab700ab1a0 2023-10-24T23:53:43.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab6e7fc640 1 -- 172.21.15.107:0/698368425 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab600ab980 con 0x7fab700ab1a0 2023-10-24T23:53:43.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab75532640 1 -- 172.21.15.107:0/698368425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab700ab1a0 msgr2=0x7fab700c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab75532640 1 --2- 172.21.15.107:0/698368425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab700ab1a0 0x7fab700c0d00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fab70071e30 tx=0x7fab6001c3c0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab75532640 1 -- 172.21.15.107:0/698368425 shutdown_connections 2023-10-24T23:53:43.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab75532640 1 --2- 172.21.15.107:0/698368425 >> v2:172.21.15.110:3300/0 conn(0x7fab70072720 0x7fab700bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab75532640 1 --2- 172.21.15.107:0/698368425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab700ab1a0 0x7fab700c0d00 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab75532640 1 -- 172.21.15.107:0/698368425 >> 172.21.15.107:0/698368425 conn(0x7fab700c2860 msgr2=0x7fab700c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:43.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab75532640 1 -- 172.21.15.107:0/698368425 shutdown_connections 2023-10-24T23:53:43.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.802+0000 7fab75532640 1 -- 172.21.15.107:0/698368425 wait complete. 2023-10-24T23:53:43.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab75532640 1 Processor -- start 2023-10-24T23:53:43.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab75532640 1 -- start start 2023-10-24T23:53:43.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab75532640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab700ab1a0 0x7fab701edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab75532640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab700bceb0 0x7fab70151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab75532640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab700a5930 0x7fab701eb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab75532640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab70150080 con 0x7fab700a5930 2023-10-24T23:53:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab75532640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab701ee580 con 0x7fab700ab1a0 2023-10-24T23:53:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab75532640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab701ee860 con 0x7fab700bceb0 2023-10-24T23:53:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab6f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab700a5930 0x7fab701eb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab6f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab700a5930 0x7fab701eb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36882/0 (socket says 172.21.15.107:36882) 2023-10-24T23:53:43.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab6f7fe640 1 -- 172.21.15.107:0/963969044 learned_addr learned my addr 172.21.15.107:0/963969044 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:43.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab6effd640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab700ab1a0 0x7fab701edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab667fc640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab700bceb0 0x7fab70151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab6f7fe640 1 -- 172.21.15.107:0/963969044 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab700bceb0 msgr2=0x7fab70151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab6f7fe640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab700bceb0 0x7fab70151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab6f7fe640 1 -- 172.21.15.107:0/963969044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab700ab1a0 msgr2=0x7fab701edfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab6f7fe640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab700ab1a0 0x7fab701edfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab6f7fe640 1 -- 172.21.15.107:0/963969044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60009420 con 0x7fab700a5930 2023-10-24T23:53:43.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab6effd640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab700ab1a0 0x7fab701edfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:43.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab667fc640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab700bceb0 0x7fab70151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:43.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab6f7fe640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab700a5930 0x7fab701eb060 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fab6800c750 tx=0x7fab6800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:43.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab67fff640 1 -- 172.21.15.107:0/963969044 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6800ce60 con 0x7fab700a5930 2023-10-24T23:53:43.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab67fff640 1 -- 172.21.15.107:0/963969044 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6800a640 con 0x7fab700a5930 2023-10-24T23:53:43.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab67fff640 1 -- 172.21.15.107:0/963969044 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6800a9b0 con 0x7fab700a5930 2023-10-24T23:53:43.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab75532640 1 -- 172.21.15.107:0/963969044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab701eeb40 con 0x7fab700a5930 2023-10-24T23:53:43.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab75532640 1 -- 172.21.15.107:0/963969044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab701eee20 con 0x7fab700a5930 2023-10-24T23:53:43.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab75532640 1 -- 172.21.15.107:0/963969044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab701ead80 con 0x7fab700a5930 2023-10-24T23:53:43.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.806+0000 7fab67fff640 1 -- 172.21.15.107:0/963969044 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fab6800ab10 con 0x7fab700a5930 2023-10-24T23:53:43.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.810+0000 7fab67fff640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab50065e90 0x7fab50068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:43.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.810+0000 7fab6effd640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab50065e90 0x7fab50068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:43.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.810+0000 7fab67fff640 1 -- 172.21.15.107:0/963969044 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6961+0+0 (secure 0 0 0) 0x7fab68083090 con 0x7fab700a5930 2023-10-24T23:53:43.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.810+0000 7fab6effd640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab50065e90 0x7fab50068350 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fab600ad000 tx=0x7fab60008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:43.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.814+0000 7fab67fff640 1 -- 172.21.15.107:0/963969044 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fab701ead80 con 0x7fab700a5930 2023-10-24T23:53:43.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 -- 172.21.15.107:0/963969044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7fab701ead80 con 0x7fab700a5930 2023-10-24T23:53:43.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab67fff640 1 -- 172.21.15.107:0/963969044 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v28) v1 ==== 95+0+11 (secure 0 0 0) 0x7fab701ead80 con 0x7fab700a5930 2023-10-24T23:53:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:43.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 -- 172.21.15.107:0/963969044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab50065e90 msgr2=0x7fab50068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab50065e90 0x7fab50068350 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fab600ad000 tx=0x7fab60008040 comp rx=0 tx=0).stop 2023-10-24T23:53:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 -- 172.21.15.107:0/963969044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab700a5930 msgr2=0x7fab701eb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab700a5930 0x7fab701eb060 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fab6800c750 tx=0x7fab6800cc10 comp rx=0 tx=0).stop 2023-10-24T23:53:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 -- 172.21.15.107:0/963969044 shutdown_connections 2023-10-24T23:53:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab700a5930 0x7fab701eb060 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab50065e90 0x7fab50068350 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab700bceb0 0x7fab70151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 --2- 172.21.15.107:0/963969044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab700ab1a0 0x7fab701edfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 -- 172.21.15.107:0/963969044 >> 172.21.15.107:0/963969044 conn(0x7fab700c2860 msgr2=0x7fab701592f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 -- 172.21.15.107:0/963969044 shutdown_connections 2023-10-24T23:53:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:43.970+0000 7fab75532640 1 -- 172.21.15.107:0/963969044 wait complete. 2023-10-24T23:53:43.992 INFO:tasks.mds_thrash:mds thrasher using random seed: 1698191623 2023-10-24T23:53:43.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:44.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f0667125640 1 Processor -- start 2023-10-24T23:53:44.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f0667125640 1 -- start start 2023-10-24T23:53:44.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f0667125640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0660152040 0x7f06600c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f0667125640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f066006a4b0 0x7f06600ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f0667125640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0660155680 con 0x7f06600c8100 2023-10-24T23:53:44.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f0667125640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0660154a70 con 0x7f066006a4b0 2023-10-24T23:53:44.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f0667125640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0660154d50 con 0x7f0660152040 2023-10-24T23:53:44.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f06656c2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f066006a4b0 0x7f06600ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f065ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f06600c8100 0x7f06600abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45068/0 (socket says 172.21.15.107:45068) 2023-10-24T23:53:44.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f065ffff640 1 -- 172.21.15.107:0/2595878745 learned_addr learned my addr 172.21.15.107:0/2595878745 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:44.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f0664ec1640 1 --2- 172.21.15.107:0/2595878745 >> v2:172.21.15.110:3300/0 conn(0x7f0660152040 0x7f06600c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f06656c2640 1 -- 172.21.15.107:0/2595878745 >> v2:172.21.15.110:3300/0 conn(0x7f0660152040 msgr2=0x7f06600c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:44.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f06656c2640 1 --2- 172.21.15.107:0/2595878745 >> v2:172.21.15.110:3300/0 conn(0x7f0660152040 0x7f06600c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f06656c2640 1 -- 172.21.15.107:0/2595878745 >> v1:172.21.15.107:6789/0 conn(0x7f06600c8100 legacy=0x7f06600abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f06656c2640 1 -- 172.21.15.107:0/2595878745 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0660155030 con 0x7f066006a4b0 2023-10-24T23:53:44.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f065f7fe640 1 -- 172.21.15.107:0/2595878745 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473144941 0 0) 0x7f0660155680 con 0x7f06600c8100 2023-10-24T23:53:44.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.110+0000 7f06656c2640 1 --2- 172.21.15.107:0/2595878745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f066006a4b0 0x7f06600ac410 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0660067ba0 tx=0x7f065801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21ef6b1f5337b6a server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:44.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f065f7fe640 1 -- 172.21.15.107:0/2595878745 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06580a5850 con 0x7f066006a4b0 2023-10-24T23:53:44.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f065f7fe640 1 -- 172.21.15.107:0/2595878745 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06580a59b0 con 0x7f066006a4b0 2023-10-24T23:53:44.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f065f7fe640 1 -- 172.21.15.107:0/2595878745 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06580a5d40 con 0x7f066006a4b0 2023-10-24T23:53:44.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 -- 172.21.15.107:0/2595878745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f066006a4b0 msgr2=0x7f06600ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 --2- 172.21.15.107:0/2595878745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f066006a4b0 0x7f06600ac410 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0660067ba0 tx=0x7f065801c880 comp rx=0 tx=0).stop 2023-10-24T23:53:44.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 -- 172.21.15.107:0/2595878745 shutdown_connections 2023-10-24T23:53:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 --2- 172.21.15.107:0/2595878745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f066006a4b0 0x7f06600ac410 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 --2- 172.21.15.107:0/2595878745 >> v2:172.21.15.110:3300/0 conn(0x7f0660152040 0x7f06600c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 -- 172.21.15.107:0/2595878745 >> 172.21.15.107:0/2595878745 conn(0x7f06600ca200 msgr2=0x7f06600ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 -- 172.21.15.107:0/2595878745 shutdown_connections 2023-10-24T23:53:44.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 -- 172.21.15.107:0/2595878745 wait complete. 2023-10-24T23:53:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 Processor -- start 2023-10-24T23:53:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 -- start start 2023-10-24T23:53:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f066006a4b0 0x7f0660151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f06600c8100 0x7f0660147680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0660152040 0x7f0660147bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06600aa160 con 0x7f0660152040 2023-10-24T23:53:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06601f0a30 con 0x7f066006a4b0 2023-10-24T23:53:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06600ae1b0 con 0x7f06600c8100 2023-10-24T23:53:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f06656c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0660152040 0x7f0660147bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f06656c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0660152040 0x7f0660147bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36896/0 (socket says 172.21.15.107:36896) 2023-10-24T23:53:44.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f06656c2640 1 -- 172.21.15.107:0/3162260330 learned_addr learned my addr 172.21.15.107:0/3162260330 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:44.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0664ec1640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f066006a4b0 0x7f0660151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f065ffff640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f06600c8100 0x7f0660147680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f06656c2640 1 -- 172.21.15.107:0/3162260330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f06600c8100 msgr2=0x7f0660147680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f06656c2640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f06600c8100 0x7f0660147680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f06656c2640 1 -- 172.21.15.107:0/3162260330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f066006a4b0 msgr2=0x7f0660151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f06656c2640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f066006a4b0 0x7f0660151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f06656c2640 1 -- 172.21.15.107:0/3162260330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0658009a70 con 0x7f0660152040 2023-10-24T23:53:44.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f06656c2640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0660152040 0x7f0660147bc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f065801cd60 tx=0x7f0658019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:44.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f065d7fa640 1 -- 172.21.15.107:0/3162260330 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06580ad850 con 0x7f0660152040 2023-10-24T23:53:44.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f065d7fa640 1 -- 172.21.15.107:0/3162260330 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06580ad9b0 con 0x7f0660152040 2023-10-24T23:53:44.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 -- 172.21.15.107:0/3162260330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06600ae490 con 0x7f0660152040 2023-10-24T23:53:44.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f065d7fa640 1 -- 172.21.15.107:0/3162260330 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06580add20 con 0x7f0660152040 2023-10-24T23:53:44.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.114+0000 7f0667125640 1 -- 172.21.15.107:0/3162260330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0660151d60 con 0x7f0660152040 2023-10-24T23:53:44.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.118+0000 7f0667125640 1 -- 172.21.15.107:0/3162260330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06600aa2d0 con 0x7f0660152040 2023-10-24T23:53:44.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.118+0000 7f065d7fa640 1 -- 172.21.15.107:0/3162260330 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f06580b4070 con 0x7f0660152040 2023-10-24T23:53:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.118+0000 7f065d7fa640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0644065e60 0x7f0644068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.118+0000 7f0664ec1640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0644065e60 0x7f0644068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.118+0000 7f065d7fa640 1 -- 172.21.15.107:0/3162260330 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f065811e780 con 0x7f0660152040 2023-10-24T23:53:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.118+0000 7f0664ec1640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0644065e60 0x7f0644068320 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f065400b800 tx=0x7f0654008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:44.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.122+0000 7f065d7fa640 1 -- 172.21.15.107:0/3162260330 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f06600aa2d0 con 0x7f0660152040 2023-10-24T23:53:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.277+0000 7f0667125640 1 -- 172.21.15.107:0/3162260330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06600aa2d0 con 0x7f0660152040 2023-10-24T23:53:44.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.277+0000 7f065d7fa640 1 -- 172.21.15.107:0/3162260330 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+21618 (secure 0 0 0) 0x7f06600aa2d0 con 0x7f0660152040 2023-10-24T23:53:44.284 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:44.284 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:43.452264+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2f","mappings":[{"from":3,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":2}]},{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:44.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 -- 172.21.15.107:0/3162260330 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0644065e60 msgr2=0x7f0644068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0644065e60 0x7f0644068320 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f065400b800 tx=0x7f0654008040 comp rx=0 tx=0).stop 2023-10-24T23:53:44.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 -- 172.21.15.107:0/3162260330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0660152040 msgr2=0x7f0660147bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0660152040 0x7f0660147bc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f065801cd60 tx=0x7f0658019620 comp rx=0 tx=0).stop 2023-10-24T23:53:44.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 -- 172.21.15.107:0/3162260330 shutdown_connections 2023-10-24T23:53:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0660152040 0x7f0660147bc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0644065e60 0x7f0644068320 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f06600c8100 0x7f0660147680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 --2- 172.21.15.107:0/3162260330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f066006a4b0 0x7f0660151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 -- 172.21.15.107:0/3162260330 >> 172.21.15.107:0/3162260330 conn(0x7f06600ca200 msgr2=0x7f066014a580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:44.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 -- 172.21.15.107:0/3162260330 shutdown_connections 2023-10-24T23:53:44.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.281+0000 7f0667125640 1 -- 172.21.15.107:0/3162260330 wait complete. 2023-10-24T23:53:44.303 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-10-24T23:53:09.786122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-10-24T23:53:10.766645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-24T23:53:44.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-24T23:53:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc552e67640 1 Processor -- start 2023-10-24T23:53:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc552e67640 1 -- start start 2023-10-24T23:53:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc552e67640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc54c0ab1a0 0x7fc54c0c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc552e67640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c0c1ce0 con 0x7fc54c072720 2023-10-24T23:53:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc552e67640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c0c0240 con 0x7fc54c0bceb0 2023-10-24T23:53:44.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc552e67640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc54c0be1d0 con 0x7fc54c0ab1a0 2023-10-24T23:53:44.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc54bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc54c072720 0x7fc54c0bfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45080/0 (socket says 172.21.15.107:45080) 2023-10-24T23:53:44.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc54bfff640 1 -- 172.21.15.107:0/4091274072 learned_addr learned my addr 172.21.15.107:0/4091274072 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:44.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc550c03640 1 --2- 172.21.15.107:0/4091274072 >> v2:172.21.15.107:3300/0 conn(0x7fc54c0ab1a0 0x7fc54c0c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc54b7fe640 1 -- 172.21.15.107:0/4091274072 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684798098 0 0) 0x7fc54c0c1ce0 con 0x7fc54c072720 2023-10-24T23:53:44.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc54b7fe640 1 -- 172.21.15.107:0/4091274072 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc534003530 con 0x7fc54c072720 2023-10-24T23:53:44.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc550c03640 1 -- 172.21.15.107:0/4091274072 >> v1:172.21.15.110:6789/0 conn(0x7fc54c0bceb0 legacy=0x7fc54c15c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:44.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc550c03640 1 -- 172.21.15.107:0/4091274072 >> v1:172.21.15.107:6789/0 conn(0x7fc54c072720 legacy=0x7fc54c0bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.417+0000 7fc550c03640 1 -- 172.21.15.107:0/4091274072 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c0c0880 con 0x7fc54c0ab1a0 2023-10-24T23:53:44.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc550c03640 1 --2- 172.21.15.107:0/4091274072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0ab1a0 0x7fc54c0c0d00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc5400097c0 tx=0x7fc54001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80352adfc4511228 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:44.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc54b7fe640 1 -- 172.21.15.107:0/4091274072 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5400ad680 con 0x7fc54c0ab1a0 2023-10-24T23:53:44.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc54b7fe640 1 -- 172.21.15.107:0/4091274072 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5400ad7e0 con 0x7fc54c0ab1a0 2023-10-24T23:53:44.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc54b7fe640 1 -- 172.21.15.107:0/4091274072 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5400adb50 con 0x7fc54c0ab1a0 2023-10-24T23:53:44.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 -- 172.21.15.107:0/4091274072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0ab1a0 msgr2=0x7fc54c0c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 --2- 172.21.15.107:0/4091274072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0ab1a0 0x7fc54c0c0d00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc5400097c0 tx=0x7fc54001c380 comp rx=0 tx=0).stop 2023-10-24T23:53:44.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 -- 172.21.15.107:0/4091274072 shutdown_connections 2023-10-24T23:53:44.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 --2- 172.21.15.107:0/4091274072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0ab1a0 0x7fc54c0c0d00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 -- 172.21.15.107:0/4091274072 >> 172.21.15.107:0/4091274072 conn(0x7fc54c0c2860 msgr2=0x7fc54c0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:44.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 -- 172.21.15.107:0/4091274072 shutdown_connections 2023-10-24T23:53:44.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 -- 172.21.15.107:0/4091274072 wait complete. 2023-10-24T23:53:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 Processor -- start 2023-10-24T23:53:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 -- start start 2023-10-24T23:53:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc54c0ab1a0 0x7fc54c154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc54c0bceb0 0x7fc54c1eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c069d40 0x7fc54c1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c1eed40 con 0x7fc54c069d40 2023-10-24T23:53:44.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c1ea840 con 0x7fc54c0ab1a0 2023-10-24T23:53:44.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc552e67640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c0c4cd0 con 0x7fc54c0bceb0 2023-10-24T23:53:44.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc551404640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c069d40 0x7fc54c1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc551404640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c069d40 0x7fc54c1521b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36914/0 (socket says 172.21.15.107:36914) 2023-10-24T23:53:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.421+0000 7fc551404640 1 -- 172.21.15.107:0/107561078 learned_addr learned my addr 172.21.15.107:0/107561078 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc550c03640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc54c0ab1a0 0x7fc54c154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc54bfff640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc54c0bceb0 0x7fc54c1eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc551404640 1 -- 172.21.15.107:0/107561078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc54c0bceb0 msgr2=0x7fc54c1eaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc551404640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc54c0bceb0 0x7fc54c1eaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc551404640 1 -- 172.21.15.107:0/107561078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc54c0ab1a0 msgr2=0x7fc54c154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc551404640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc54c0ab1a0 0x7fc54c154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc551404640 1 -- 172.21.15.107:0/107561078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc540009420 con 0x7fc54c069d40 2023-10-24T23:53:44.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc54bfff640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc54c0bceb0 0x7fc54c1eaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:44.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc551404640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c069d40 0x7fc54c1521b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc54400b9e0 tx=0x7fc54400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:44.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc5497fa640 1 -- 172.21.15.107:0/107561078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc544003b80 con 0x7fc54c069d40 2023-10-24T23:53:44.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc5497fa640 1 -- 172.21.15.107:0/107561078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc544003ce0 con 0x7fc54c069d40 2023-10-24T23:53:44.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc552e67640 1 -- 172.21.15.107:0/107561078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54c06a730 con 0x7fc54c069d40 2023-10-24T23:53:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc5497fa640 1 -- 172.21.15.107:0/107561078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54400cb00 con 0x7fc54c069d40 2023-10-24T23:53:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc552e67640 1 -- 172.21.15.107:0/107561078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54c06a980 con 0x7fc54c069d40 2023-10-24T23:53:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc552e67640 1 -- 172.21.15.107:0/107561078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54c0a76d0 con 0x7fc54c069d40 2023-10-24T23:53:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc5497fa640 1 -- 172.21.15.107:0/107561078 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc54400cc60 con 0x7fc54c069d40 2023-10-24T23:53:44.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc5497fa640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc538065e90 0x7fc538068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc5497fa640 1 -- 172.21.15.107:0/107561078 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6961+0+0 (secure 0 0 0) 0x7fc544083800 con 0x7fc54c069d40 2023-10-24T23:53:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc550c03640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc538065e90 0x7fc538068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.425+0000 7fc550c03640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc538065e90 0x7fc538068350 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc5400af000 tx=0x7fc5400a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:44.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.429+0000 7fc5497fa640 1 -- 172.21.15.107:0/107561078 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc544050b50 con 0x7fc54c069d40 2023-10-24T23:53:44.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.589+0000 7fc552e67640 1 -- 172.21.15.107:0/107561078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc54c04e130 con 0x7fc54c069d40 2023-10-24T23:53:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.589+0000 7fc5497fa640 1 -- 172.21.15.107:0/107561078 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc544054170 con 0x7fc54c069d40 2023-10-24T23:53:44.596 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-24T23:53:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 -- 172.21.15.107:0/107561078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc538065e90 msgr2=0x7fc538068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc538065e90 0x7fc538068350 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc5400af000 tx=0x7fc5400a7000 comp rx=0 tx=0).stop 2023-10-24T23:53:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 -- 172.21.15.107:0/107561078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c069d40 msgr2=0x7fc54c1521b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c069d40 0x7fc54c1521b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc54400b9e0 tx=0x7fc54400bea0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 -- 172.21.15.107:0/107561078 shutdown_connections 2023-10-24T23:53:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c069d40 0x7fc54c1521b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc538065e90 0x7fc538068350 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc54c0bceb0 0x7fc54c1eaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 --2- 172.21.15.107:0/107561078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc54c0ab1a0 0x7fc54c154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 -- 172.21.15.107:0/107561078 >> 172.21.15.107:0/107561078 conn(0x7fc54c0c2860 msgr2=0x7fc54c155220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 -- 172.21.15.107:0/107561078 shutdown_connections 2023-10-24T23:53:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.593+0000 7fc552e67640 1 -- 172.21.15.107:0/107561078 wait complete. 2023-10-24T23:53:44.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-24T23:53:44.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c6a63640 1 Processor -- start 2023-10-24T23:53:44.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c6a63640 1 -- start start 2023-10-24T23:53:44.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c6a63640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c00c3920 0x7f54c00c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c6a63640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f54c00c42a0 0x7f54c015f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c6a63640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c0071de0 con 0x7f54c00c2000 2023-10-24T23:53:44.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c6a63640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c00bd250 con 0x7f54c00c42a0 2023-10-24T23:53:44.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c6a63640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c00bcf00 con 0x7f54c00c3920 2023-10-24T23:53:44.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c5000640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f54c00c42a0 0x7f54c015f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54bffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f54c00c2000 0x7f54c00ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45090/0 (socket says 172.21.15.107:45090) 2023-10-24T23:53:44.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54bffff640 1 -- 172.21.15.107:0/1983122921 learned_addr learned my addr 172.21.15.107:0/1983122921 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:44.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54bf7fe640 1 --2- 172.21.15.107:0/1983122921 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c00c3920 0x7f54c00c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c5000640 1 -- 172.21.15.107:0/1983122921 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c00c3920 msgr2=0x7f54c00c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:44.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c5000640 1 --2- 172.21.15.107:0/1983122921 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c00c3920 0x7f54c00c3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c5000640 1 -- 172.21.15.107:0/1983122921 >> v1:172.21.15.107:6789/0 conn(0x7f54c00c2000 legacy=0x7f54c00ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54beffd640 1 -- 172.21.15.107:0/1983122921 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082854795 0 0) 0x7f54c0071de0 con 0x7f54c00c2000 2023-10-24T23:53:44.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.737+0000 7f54c5000640 1 -- 172.21.15.107:0/1983122921 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c006a740 con 0x7f54c00c42a0 2023-10-24T23:53:44.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c5000640 1 --2- 172.21.15.107:0/1983122921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c00c42a0 0x7f54c015f080 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009d90 tx=0x7f54b801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108e2cf30a19a9e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:44.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54beffd640 1 -- 172.21.15.107:0/1983122921 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b80a59e0 con 0x7f54c00c42a0 2023-10-24T23:53:44.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54beffd640 1 -- 172.21.15.107:0/1983122921 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b80a5b40 con 0x7f54c00c42a0 2023-10-24T23:53:44.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54beffd640 1 -- 172.21.15.107:0/1983122921 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b80ad440 con 0x7f54c00c42a0 2023-10-24T23:53:44.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 -- 172.21.15.107:0/1983122921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c00c42a0 msgr2=0x7f54c015f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 --2- 172.21.15.107:0/1983122921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c00c42a0 0x7f54c015f080 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009d90 tx=0x7f54b801c920 comp rx=0 tx=0).stop 2023-10-24T23:53:44.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 -- 172.21.15.107:0/1983122921 shutdown_connections 2023-10-24T23:53:44.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 --2- 172.21.15.107:0/1983122921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c00c42a0 0x7f54c015f080 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 --2- 172.21.15.107:0/1983122921 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c00c3920 0x7f54c00c3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 -- 172.21.15.107:0/1983122921 >> 172.21.15.107:0/1983122921 conn(0x7f54c0070ef0 msgr2=0x7f54c0071300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:44.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 -- 172.21.15.107:0/1983122921 shutdown_connections 2023-10-24T23:53:44.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 -- 172.21.15.107:0/1983122921 wait complete. 2023-10-24T23:53:44.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 Processor -- start 2023-10-24T23:53:44.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 -- start start 2023-10-24T23:53:44.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c00c2000 0x7f54c01ecb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c00c3920 0x7f54c01ef740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c00c42a0 0x7f54c00c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c0144bc0 con 0x7f54c00c2000 2023-10-24T23:53:44.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c01585e0 con 0x7f54c00c42a0 2023-10-24T23:53:44.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c6a63640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c01edfc0 con 0x7f54c00c3920 2023-10-24T23:53:44.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c00c2000 0x7f54c01ecb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c00c2000 0x7f54c01ecb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36926/0 (socket says 172.21.15.107:36926) 2023-10-24T23:53:44.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54bffff640 1 -- 172.21.15.107:0/206342943 learned_addr learned my addr 172.21.15.107:0/206342943 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:44.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54bf7fe640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c00c3920 0x7f54c01ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54bffff640 1 -- 172.21.15.107:0/206342943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c00c3920 msgr2=0x7f54c01ef740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54bffff640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c00c3920 0x7f54c01ef740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54c5000640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c00c42a0 0x7f54c00c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54bffff640 1 -- 172.21.15.107:0/206342943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c00c42a0 msgr2=0x7f54c00c5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54bffff640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c00c42a0 0x7f54c00c5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54bffff640 1 -- 172.21.15.107:0/206342943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b8009a70 con 0x7f54c00c2000 2023-10-24T23:53:44.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54bf7fe640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c00c3920 0x7f54c01ef740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:44.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.741+0000 7f54bffff640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c00c2000 0x7f54c01ecb50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f54b000c4b0 tx=0x7f54b000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:44.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.745+0000 7f54bcff9640 1 -- 172.21.15.107:0/206342943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b00074e0 con 0x7f54c00c2000 2023-10-24T23:53:44.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.745+0000 7f54bcff9640 1 -- 172.21.15.107:0/206342943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b0007660 con 0x7f54c00c2000 2023-10-24T23:53:44.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.745+0000 7f54c6a63640 1 -- 172.21.15.107:0/206342943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c01ed1a0 con 0x7f54c00c2000 2023-10-24T23:53:44.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.745+0000 7f54bcff9640 1 -- 172.21.15.107:0/206342943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0004a30 con 0x7f54c00c2000 2023-10-24T23:53:44.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.745+0000 7f54c6a63640 1 -- 172.21.15.107:0/206342943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c01ed760 con 0x7f54c00c2000 2023-10-24T23:53:44.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.745+0000 7f54c6a63640 1 -- 172.21.15.107:0/206342943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c0073c30 con 0x7f54c00c2000 2023-10-24T23:53:44.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.745+0000 7f54bcff9640 1 -- 172.21.15.107:0/206342943 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f54b0004b90 con 0x7f54c00c2000 2023-10-24T23:53:44.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.745+0000 7f54bcff9640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54ac065e90 0x7f54ac068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:44.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.745+0000 7f54bcff9640 1 -- 172.21.15.107:0/206342943 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f54b0082c20 con 0x7f54c00c2000 2023-10-24T23:53:44.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.745+0000 7f54bf7fe640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54ac065e90 0x7f54ac068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:44.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.745+0000 7f54bf7fe640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54ac065e90 0x7f54ac068350 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f54c015f590 tx=0x7f54b4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:44.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.749+0000 7f54bcff9640 1 -- 172.21.15.107:0/206342943 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f54b004ff70 con 0x7f54c00c2000 2023-10-24T23:53:44.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.893+0000 7f54c6a63640 1 -- 172.21.15.107:0/206342943 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f54c004e130 con 0x7f54c00c2000 2023-10-24T23:53:44.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54bcff9640 1 -- 172.21.15.107:0/206342943 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f54b0018020 con 0x7f54c00c2000 2023-10-24T23:53:44.901 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-24T23:53:44.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 -- 172.21.15.107:0/206342943 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54ac065e90 msgr2=0x7f54ac068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54ac065e90 0x7f54ac068350 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f54c015f590 tx=0x7f54b4008040 comp rx=0 tx=0).stop 2023-10-24T23:53:44.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 -- 172.21.15.107:0/206342943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c00c2000 msgr2=0x7f54c01ecb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:44.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c00c2000 0x7f54c01ecb50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f54b000c4b0 tx=0x7f54b000c970 comp rx=0 tx=0).stop 2023-10-24T23:53:44.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 -- 172.21.15.107:0/206342943 shutdown_connections 2023-10-24T23:53:44.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c00c42a0 0x7f54c00c5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54ac065e90 0x7f54ac068350 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c00c3920 0x7f54c01ef740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 --2- 172.21.15.107:0/206342943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c00c2000 0x7f54c01ecb50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:44.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 -- 172.21.15.107:0/206342943 >> 172.21.15.107:0/206342943 conn(0x7f54c0070ef0 msgr2=0x7f54c00ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:44.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 -- 172.21.15.107:0/206342943 shutdown_connections 2023-10-24T23:53:44.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:44.897+0000 7f54c6a63640 1 -- 172.21.15.107:0/206342943 wait complete. 2023-10-24T23:53:44.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-24T23:53:45.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff9559b0640 1 Processor -- start 2023-10-24T23:53:45.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff9559b0640 1 -- start start 2023-10-24T23:53:45.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff9559b0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9500aa9f0 0x7ff950072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff9559b0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9500befa0 con 0x7ff9500a9300 2023-10-24T23:53:45.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff9559b0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9500a9b80 con 0x7ff9500c1b30 2023-10-24T23:53:45.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff9559b0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9500aa0b0 con 0x7ff9500aa9f0 2023-10-24T23:53:45.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff94e7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9500aa9f0 0x7ff950072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff94f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff9500a9300 0x7ff950155620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45104/0 (socket says 172.21.15.107:45104) 2023-10-24T23:53:45.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff94e7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9500aa9f0 0x7ff950072210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36936/0 (socket says 172.21.15.107:36936) 2023-10-24T23:53:45.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff94f7fe640 1 -- 172.21.15.107:0/3254273521 learned_addr learned my addr 172.21.15.107:0/3254273521 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:45.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff94e7fc640 1 -- 172.21.15.107:0/3254273521 >> v1:172.21.15.110:6789/0 conn(0x7ff9500c1b30 legacy=0x7ff9500aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:45.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff94e7fc640 1 -- 172.21.15.107:0/3254273521 >> v1:172.21.15.107:6789/0 conn(0x7ff9500a9300 legacy=0x7ff950155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.025+0000 7ff94e7fc640 1 -- 172.21.15.107:0/3254273521 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9500bf1f0 con 0x7ff9500aa9f0 2023-10-24T23:53:45.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94e7fc640 1 --2- 172.21.15.107:0/3254273521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500aa9f0 0x7ff950072210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff9440097c0 tx=0x7ff94401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e607db20036a7557 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94dffb640 1 -- 172.21.15.107:0/3254273521 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440ac440 con 0x7ff9500aa9f0 2023-10-24T23:53:45.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94dffb640 1 -- 172.21.15.107:0/3254273521 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9440ac5a0 con 0x7ff9500aa9f0 2023-10-24T23:53:45.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94dffb640 1 -- 172.21.15.107:0/3254273521 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440ac930 con 0x7ff9500aa9f0 2023-10-24T23:53:45.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 -- 172.21.15.107:0/3254273521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500aa9f0 msgr2=0x7ff950072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 --2- 172.21.15.107:0/3254273521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500aa9f0 0x7ff950072210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff9440097c0 tx=0x7ff94401c320 comp rx=0 tx=0).stop 2023-10-24T23:53:45.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 -- 172.21.15.107:0/3254273521 shutdown_connections 2023-10-24T23:53:45.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 --2- 172.21.15.107:0/3254273521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500aa9f0 0x7ff950072210 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 -- 172.21.15.107:0/3254273521 >> 172.21.15.107:0/3254273521 conn(0x7ff9500c2860 msgr2=0x7ff9500c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:45.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 -- 172.21.15.107:0/3254273521 shutdown_connections 2023-10-24T23:53:45.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 -- 172.21.15.107:0/3254273521 wait complete. 2023-10-24T23:53:45.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 Processor -- start 2023-10-24T23:53:45.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 -- start start 2023-10-24T23:53:45.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9500a9300 0x7ff950071e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500c1b30 0x7ff95014ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9500c1460 0x7ff95014f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95006cff0 con 0x7ff9500c1b30 2023-10-24T23:53:45.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9501eb920 con 0x7ff9500c1460 2023-10-24T23:53:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95014d730 con 0x7ff9500a9300 2023-10-24T23:53:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500c1b30 0x7ff95014ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500c1b30 0x7ff95014ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36944/0 (socket says 172.21.15.107:36944) 2023-10-24T23:53:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94e7fc640 1 -- 172.21.15.107:0/2942352031 learned_addr learned my addr 172.21.15.107:0/2942352031 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94e7fc640 1 -- 172.21.15.107:0/2942352031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9500a9300 msgr2=0x7ff950071e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94e7fc640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9500a9300 0x7ff950071e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94e7fc640 1 -- 172.21.15.107:0/2942352031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9500c1460 msgr2=0x7ff95014f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94e7fc640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9500c1460 0x7ff95014f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94e7fc640 1 -- 172.21.15.107:0/2942352031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff944009420 con 0x7ff9500c1b30 2023-10-24T23:53:45.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff94e7fc640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500c1b30 0x7ff95014ee00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff944018f30 tx=0x7ff9440190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff937fff640 1 -- 172.21.15.107:0/2942352031 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440b5820 con 0x7ff9500c1b30 2023-10-24T23:53:45.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff937fff640 1 -- 172.21.15.107:0/2942352031 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9440b59a0 con 0x7ff9500c1b30 2023-10-24T23:53:45.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 -- 172.21.15.107:0/2942352031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9500b92d0 con 0x7ff9500c1b30 2023-10-24T23:53:45.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff937fff640 1 -- 172.21.15.107:0/2942352031 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440b5d50 con 0x7ff9500c1b30 2023-10-24T23:53:45.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.029+0000 7ff9559b0640 1 -- 172.21.15.107:0/2942352031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff950150b40 con 0x7ff9500c1b30 2023-10-24T23:53:45.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.033+0000 7ff9559b0640 1 -- 172.21.15.107:0/2942352031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9500a97e0 con 0x7ff9500c1b30 2023-10-24T23:53:45.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.033+0000 7ff937fff640 1 -- 172.21.15.107:0/2942352031 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff9440b3040 con 0x7ff9500c1b30 2023-10-24T23:53:45.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.033+0000 7ff937fff640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff93c065e90 0x7ff93c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.033+0000 7ff937fff640 1 -- 172.21.15.107:0/2942352031 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7ff94411dee0 con 0x7ff9500c1b30 2023-10-24T23:53:45.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.033+0000 7ff94effd640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff93c065e90 0x7ff93c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.033+0000 7ff94effd640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff93c065e90 0x7ff93c068350 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff9500a7690 tx=0x7ff9400079c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.037+0000 7ff937fff640 1 -- 172.21.15.107:0/2942352031 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff9440eb2b0 con 0x7ff9500c1b30 2023-10-24T23:53:45.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 -- 172.21.15.107:0/2942352031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9501eba90 con 0x7ff9500c1b30 2023-10-24T23:53:45.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff937fff640 1 -- 172.21.15.107:0/2942352031 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v29) v1 ==== 99+0+11 (secure 0 0 0) 0x7ff9440c3020 con 0x7ff9500c1b30 2023-10-24T23:53:45.183 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:45.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 -- 172.21.15.107:0/2942352031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff93c065e90 msgr2=0x7ff93c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff93c065e90 0x7ff93c068350 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff9500a7690 tx=0x7ff9400079c0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 -- 172.21.15.107:0/2942352031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500c1b30 msgr2=0x7ff95014ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500c1b30 0x7ff95014ee00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff944018f30 tx=0x7ff9440190c0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 -- 172.21.15.107:0/2942352031 shutdown_connections 2023-10-24T23:53:45.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9500c1460 0x7ff95014f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9500c1b30 0x7ff95014ee00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff93c065e90 0x7ff93c068350 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 --2- 172.21.15.107:0/2942352031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9500a9300 0x7ff950071e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 -- 172.21.15.107:0/2942352031 >> 172.21.15.107:0/2942352031 conn(0x7ff9500c2860 msgr2=0x7ff9500c46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:45.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 -- 172.21.15.107:0/2942352031 shutdown_connections 2023-10-24T23:53:45.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.177+0000 7ff9559b0640 1 -- 172.21.15.107:0/2942352031 wait complete. 2023-10-24T23:53:45.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-24T23:53:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d6a7b640 1 Processor -- start 2023-10-24T23:53:45.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d6a7b640 1 -- start start 2023-10-24T23:53:45.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d6a7b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00bceb0 0x7fb2d00c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d6a7b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00c1ce0 con 0x7fb2d00ab1a0 2023-10-24T23:53:45.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d6a7b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00c0240 con 0x7fb2d0072720 2023-10-24T23:53:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d6a7b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00be1d0 con 0x7fb2d00bceb0 2023-10-24T23:53:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d4817640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb2d00ab1a0 0x7fb2d00bfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45112/0 (socket says 172.21.15.107:45112) 2023-10-24T23:53:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d4817640 1 -- 172.21.15.107:0/3328809113 learned_addr learned my addr 172.21.15.107:0/3328809113 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d5018640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00bceb0 0x7fb2d00c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208826378 0 0) 0x7fb2d00c1ce0 con 0x7fb2d00ab1a0 2023-10-24T23:53:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b8003530 con 0x7fb2d00ab1a0 2023-10-24T23:53:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 753774088 0 0) 0x7fb2b8003530 con 0x7fb2d00ab1a0 2023-10-24T23:53:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00bceb0 msgr2=0x7fb2d00c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:45.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00bceb0 0x7fb2d00c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 >> v1:172.21.15.110:6789/0 conn(0x7fb2d0072720 legacy=0x7fb2d015a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d00c0880 con 0x7fb2d00ab1a0 2023-10-24T23:53:45.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb2c0001680 con 0x7fb2d00ab1a0 2023-10-24T23:53:45.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d5018640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00bceb0 0x7fb2d00c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:45.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 >> v1:172.21.15.107:6789/0 conn(0x7fb2d00ab1a0 legacy=0x7fb2d00bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b80039b0 0x7fb2b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00bceb0 0x7fb2b80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2b8004d10 0x7fb2b8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b800bb90 con 0x7fb2b80039b0 2023-10-24T23:53:45.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b800bd00 con 0x7fb2b8004d10 2023-10-24T23:53:45.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b800bec0 con 0x7fb2d00bceb0 2023-10-24T23:53:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d4817640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b80039b0 0x7fb2b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2cffff640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00bceb0 0x7fb2b80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d5018640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2b8004d10 0x7fb2b8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d4817640 1 -- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00bceb0 msgr2=0x7fb2b80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d4817640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00bceb0 0x7fb2b80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d4817640 1 -- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2b8004d10 msgr2=0x7fb2b8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d4817640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2b8004d10 0x7fb2b8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.321+0000 7fb2d4817640 1 -- 172.21.15.107:0/3328809113 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b8003530 con 0x7fb2b80039b0 2023-10-24T23:53:45.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2d4817640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b80039b0 0x7fb2b8003dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000c240 tx=0x7fb2c001cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4a0a928ed0b9a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0005ae0 con 0x7fb2b80039b0 2023-10-24T23:53:45.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c0005c40 con 0x7fb2b80039b0 2023-10-24T23:53:45.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2cf7fe640 1 -- 172.21.15.107:0/3328809113 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c00a5800 con 0x7fb2b80039b0 2023-10-24T23:53:45.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3328809113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b80039b0 msgr2=0x7fb2b8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2d6a7b640 1 --2- 172.21.15.107:0/3328809113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2b80039b0 0x7fb2b8003dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000c240 tx=0x7fb2c001cc10 comp rx=0 tx=0).stop 2023-10-24T23:53:45.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2d5018640 1 -- 172.21.15.107:0/3328809113 reap_dead start 2023-10-24T23:53:45.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3328809113 shutdown_connections 2023-10-24T23:53:45.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3328809113 >> 172.21.15.107:0/3328809113 conn(0x7fb2d00c2860 msgr2=0x7fb2d00c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:45.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3328809113 shutdown_connections 2023-10-24T23:53:45.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3328809113 wait complete. 2023-10-24T23:53:45.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2d6a7b640 1 Processor -- start 2023-10-24T23:53:45.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.325+0000 7fb2d6a7b640 1 -- start start 2023-10-24T23:53:45.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d6a7b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2d00a3500 0x7fb2d01eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d6a7b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2d0069d40 0x7fb2d006a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d6a7b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00a5a70 0x7fb2d00a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d6a7b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00a5300 con 0x7fb2d0069d40 2023-10-24T23:53:45.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d6a7b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d01ef060 con 0x7fb2d00a3500 2023-10-24T23:53:45.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d6a7b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00a4280 con 0x7fb2d00a5a70 2023-10-24T23:53:45.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2d0069d40 0x7fb2d006a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2d0069d40 0x7fb2d006a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36956/0 (socket says 172.21.15.107:36956) 2023-10-24T23:53:45.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cffff640 1 -- 172.21.15.107:0/3957634397 learned_addr learned my addr 172.21.15.107:0/3957634397 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:45.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d5018640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00a5a70 0x7fb2d00a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d4817640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2d00a3500 0x7fb2d01eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cffff640 1 -- 172.21.15.107:0/3957634397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00a5a70 msgr2=0x7fb2d00a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cffff640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00a5a70 0x7fb2d00a5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cffff640 1 -- 172.21.15.107:0/3957634397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2d00a3500 msgr2=0x7fb2d01eeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cffff640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2d00a3500 0x7fb2d01eeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cffff640 1 -- 172.21.15.107:0/3957634397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d00a4560 con 0x7fb2d0069d40 2023-10-24T23:53:45.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d4817640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2d00a3500 0x7fb2d01eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:45.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cffff640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2d0069d40 0x7fb2d006a1c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400e550 tx=0x7fb2c400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cd7fa640 1 -- 172.21.15.107:0/3957634397 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c40052e0 con 0x7fb2d0069d40 2023-10-24T23:53:45.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cd7fa640 1 -- 172.21.15.107:0/3957634397 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c4005440 con 0x7fb2d0069d40 2023-10-24T23:53:45.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3957634397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d00a4840 con 0x7fb2d0069d40 2023-10-24T23:53:45.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cd7fa640 1 -- 172.21.15.107:0/3957634397 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4003820 con 0x7fb2d0069d40 2023-10-24T23:53:45.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3957634397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d00c4e30 con 0x7fb2d0069d40 2023-10-24T23:53:45.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3957634397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d004e130 con 0x7fb2d0069d40 2023-10-24T23:53:45.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cd7fa640 1 -- 172.21.15.107:0/3957634397 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb2c4003980 con 0x7fb2d0069d40 2023-10-24T23:53:45.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cd7fa640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb2bc065e90 0x7fb2bc068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2cd7fa640 1 -- 172.21.15.107:0/3957634397 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7fb2c40839b0 con 0x7fb2d0069d40 2023-10-24T23:53:45.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d4817640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb2bc065e90 0x7fb2bc068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.329+0000 7fb2d4817640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb2bc065e90 0x7fb2bc068350 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00ab000 tx=0x7fb2c00b1040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.333+0000 7fb2cd7fa640 1 -- 172.21.15.107:0/3957634397 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb2c4050c80 con 0x7fb2d0069d40 2023-10-24T23:53:45.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.489+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3957634397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7fb2d00a49d0 con 0x7fb2d0069d40 2023-10-24T23:53:45.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.489+0000 7fb2cd7fa640 1 -- 172.21.15.107:0/3957634397 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v29) v1 ==== 95+0+11 (secure 0 0 0) 0x7fb2c40542a0 con 0x7fb2d0069d40 2023-10-24T23:53:45.495 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:45.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3957634397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb2bc065e90 msgr2=0x7fb2bc068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb2bc065e90 0x7fb2bc068350 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00ab000 tx=0x7fb2c00b1040 comp rx=0 tx=0).stop 2023-10-24T23:53:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3957634397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2d0069d40 msgr2=0x7fb2d006a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2d0069d40 0x7fb2d006a1c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400e550 tx=0x7fb2c400ea10 comp rx=0 tx=0).stop 2023-10-24T23:53:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3957634397 shutdown_connections 2023-10-24T23:53:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb2bc065e90 0x7fb2bc068350 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2d00a5a70 0x7fb2d00a5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2d0069d40 0x7fb2d006a1c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 --2- 172.21.15.107:0/3957634397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2d00a3500 0x7fb2d01eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3957634397 >> 172.21.15.107:0/3957634397 conn(0x7fb2d00c2860 msgr2=0x7fb2d006f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:45.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3957634397 shutdown_connections 2023-10-24T23:53:45.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.493+0000 7fb2d6a7b640 1 -- 172.21.15.107:0/3957634397 wait complete. 2023-10-24T23:53:45.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-10-24T23:53:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee33798640 1 Processor -- start 2023-10-24T23:53:45.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee33798640 1 -- start start 2023-10-24T23:53:45.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee33798640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee2c0abf00 0x7fee2c0ab080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee33798640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee2c1551e0 0x7fee2c0c2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee33798640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2c0c37b0 con 0x7fee2c06fd80 2023-10-24T23:53:45.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee33798640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee2c1580b0 con 0x7fee2c1551e0 2023-10-24T23:53:45.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee33798640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee2c155670 con 0x7fee2c0abf00 2023-10-24T23:53:45.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee31d35640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee2c1551e0 0x7fee2c0c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee30d33640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fee2c06fd80 0x7fee2c158540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45124/0 (socket says 172.21.15.107:45124) 2023-10-24T23:53:45.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee30d33640 1 -- 172.21.15.107:0/3370732902 learned_addr learned my addr 172.21.15.107:0/3370732902 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:45.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee31534640 1 --2- 172.21.15.107:0/3370732902 >> v2:172.21.15.110:3300/0 conn(0x7fee2c0abf00 0x7fee2c0ab080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee31d35640 1 -- 172.21.15.107:0/3370732902 >> v2:172.21.15.110:3300/0 conn(0x7fee2c0abf00 msgr2=0x7fee2c0ab080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:45.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee31d35640 1 --2- 172.21.15.107:0/3370732902 >> v2:172.21.15.110:3300/0 conn(0x7fee2c0abf00 0x7fee2c0ab080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee31d35640 1 -- 172.21.15.107:0/3370732902 >> v1:172.21.15.107:6789/0 conn(0x7fee2c06fd80 legacy=0x7fee2c158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee23fff640 1 -- 172.21.15.107:0/3370732902 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542071706 0 0) 0x7fee2c0c37b0 con 0x7fee2c06fd80 2023-10-24T23:53:45.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee31d35640 1 -- 172.21.15.107:0/3370732902 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c0a9450 con 0x7fee2c1551e0 2023-10-24T23:53:45.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.625+0000 7fee31534640 1 --2- 172.21.15.107:0/3370732902 >> v2:172.21.15.110:3300/0 conn(0x7fee2c0abf00 0x7fee2c0ab080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:45.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee31d35640 1 --2- 172.21.15.107:0/3370732902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2c1551e0 0x7fee2c0c2870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fee2c06f790 tx=0x7fee2401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77e0ebdebfe35eb server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee23fff640 1 -- 172.21.15.107:0/3370732902 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee240a59d0 con 0x7fee2c1551e0 2023-10-24T23:53:45.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee23fff640 1 -- 172.21.15.107:0/3370732902 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee240a5b30 con 0x7fee2c1551e0 2023-10-24T23:53:45.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee23fff640 1 -- 172.21.15.107:0/3370732902 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee240ad440 con 0x7fee2c1551e0 2023-10-24T23:53:45.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 -- 172.21.15.107:0/3370732902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2c1551e0 msgr2=0x7fee2c0c2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 --2- 172.21.15.107:0/3370732902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2c1551e0 0x7fee2c0c2870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fee2c06f790 tx=0x7fee2401c910 comp rx=0 tx=0).stop 2023-10-24T23:53:45.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 -- 172.21.15.107:0/3370732902 shutdown_connections 2023-10-24T23:53:45.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 --2- 172.21.15.107:0/3370732902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2c1551e0 0x7fee2c0c2870 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 --2- 172.21.15.107:0/3370732902 >> v2:172.21.15.110:3300/0 conn(0x7fee2c0abf00 0x7fee2c0ab080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 -- 172.21.15.107:0/3370732902 >> 172.21.15.107:0/3370732902 conn(0x7fee2c0c3960 msgr2=0x7fee2c0c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:45.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 -- 172.21.15.107:0/3370732902 shutdown_connections 2023-10-24T23:53:45.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 -- 172.21.15.107:0/3370732902 wait complete. 2023-10-24T23:53:45.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 Processor -- start 2023-10-24T23:53:45.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 -- start start 2023-10-24T23:53:45.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee2c06fd80 0x7fee2c14c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2c0abf00 0x7fee2c142d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2c1551e0 0x7fee2c1432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c153350 con 0x7fee2c1551e0 2023-10-24T23:53:45.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c145e00 con 0x7fee2c0abf00 2023-10-24T23:53:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee33798640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c152c50 con 0x7fee2c06fd80 2023-10-24T23:53:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee31d35640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2c1551e0 0x7fee2c1432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee31d35640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2c1551e0 0x7fee2c1432c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36970/0 (socket says 172.21.15.107:36970) 2023-10-24T23:53:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee31d35640 1 -- 172.21.15.107:0/295173388 learned_addr learned my addr 172.21.15.107:0/295173388 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee30d33640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2c0abf00 0x7fee2c142d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee31d35640 1 -- 172.21.15.107:0/295173388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee2c06fd80 msgr2=0x7fee2c14c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee31d35640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee2c06fd80 0x7fee2c14c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee31d35640 1 -- 172.21.15.107:0/295173388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2c0abf00 msgr2=0x7fee2c142d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee31d35640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2c0abf00 0x7fee2c142d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee31d35640 1 -- 172.21.15.107:0/295173388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee24009a70 con 0x7fee2c1551e0 2023-10-24T23:53:45.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.629+0000 7fee30d33640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2c0abf00 0x7fee2c142d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:45.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee31d35640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2c1551e0 0x7fee2c1432c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fee240b0040 tx=0x7fee240196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee21ffb640 1 -- 172.21.15.107:0/295173388 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee240b9650 con 0x7fee2c1551e0 2023-10-24T23:53:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee21ffb640 1 -- 172.21.15.107:0/295173388 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee240b97b0 con 0x7fee2c1551e0 2023-10-24T23:53:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee33798640 1 -- 172.21.15.107:0/295173388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2c146050 con 0x7fee2c1551e0 2023-10-24T23:53:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee21ffb640 1 -- 172.21.15.107:0/295173388 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee240b9b60 con 0x7fee2c1551e0 2023-10-24T23:53:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee33798640 1 -- 172.21.15.107:0/295173388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2c1528e0 con 0x7fee2c1551e0 2023-10-24T23:53:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee33798640 1 -- 172.21.15.107:0/295173388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2c06df90 con 0x7fee2c1551e0 2023-10-24T23:53:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee21ffb640 1 -- 172.21.15.107:0/295173388 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fee240b9cc0 con 0x7fee2c1551e0 2023-10-24T23:53:45.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee21ffb640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee10065e60 0x7fee10068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee31534640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee10065e60 0x7fee10068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee21ffb640 1 -- 172.21.15.107:0/295173388 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7fee2411f660 con 0x7fee2c1551e0 2023-10-24T23:53:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.633+0000 7fee31534640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee10065e60 0x7fee10068320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fee2800a8d0 tx=0x7fee28008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.637+0000 7fee21ffb640 1 -- 172.21.15.107:0/295173388 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fee240eca70 con 0x7fee2c1551e0 2023-10-24T23:53:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.777+0000 7fee33798640 1 -- 172.21.15.107:0/295173388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7fee2c14d5a0 con 0x7fee2c1551e0 2023-10-24T23:53:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.777+0000 7fee21ffb640 1 -- 172.21.15.107:0/295173388 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v29) v1 ==== 99+0+11 (secure 0 0 0) 0x7fee240f0090 con 0x7fee2c1551e0 2023-10-24T23:53:45.785 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:45.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 -- 172.21.15.107:0/295173388 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee10065e60 msgr2=0x7fee10068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee10065e60 0x7fee10068320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fee2800a8d0 tx=0x7fee28008040 comp rx=0 tx=0).stop 2023-10-24T23:53:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 -- 172.21.15.107:0/295173388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2c1551e0 msgr2=0x7fee2c1432c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2c1551e0 0x7fee2c1432c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fee240b0040 tx=0x7fee240196b0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 -- 172.21.15.107:0/295173388 shutdown_connections 2023-10-24T23:53:45.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2c1551e0 0x7fee2c1432c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2c0abf00 0x7fee2c142d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee10065e60 0x7fee10068320 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 --2- 172.21.15.107:0/295173388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee2c06fd80 0x7fee2c14c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 -- 172.21.15.107:0/295173388 >> 172.21.15.107:0/295173388 conn(0x7fee2c0c3960 msgr2=0x7fee2c071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:45.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 -- 172.21.15.107:0/295173388 shutdown_connections 2023-10-24T23:53:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.781+0000 7fee33798640 1 -- 172.21.15.107:0/295173388 wait complete. 2023-10-24T23:53:45.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-10-24T23:53:45.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f268e9da640 1 Processor -- start 2023-10-24T23:53:45.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f268e9da640 1 -- start start 2023-10-24T23:53:45.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f268e9da640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f26880c2000 0x7f26880ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f268e9da640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26880c4030 0x7f268815f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f268e9da640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2688071de0 con 0x7f26880adda0 2023-10-24T23:53:45.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f268e9da640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26880bd250 con 0x7f26880c2000 2023-10-24T23:53:45.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f268e9da640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26880bcf00 con 0x7f26880c4030 2023-10-24T23:53:45.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f26877fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f26880adda0 0x7f26880c3920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45132/0 (socket says 172.21.15.107:45132) 2023-10-24T23:53:45.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f26877fe640 1 -- 172.21.15.107:0/2517552914 learned_addr learned my addr 172.21.15.107:0/2517552914 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:45.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f268cf77640 1 --2- 172.21.15.107:0/2517552914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26880c4030 0x7f268815f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2687fff640 1 --2- 172.21.15.107:0/2517552914 >> v2:172.21.15.110:3300/0 conn(0x7f26880c2000 0x7f26880ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847072708 0 0) 0x7f2688071de0 con 0x7f26880adda0 2023-10-24T23:53:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2670003530 con 0x7f26880adda0 2023-10-24T23:53:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3346503330 0 0) 0x7f2670003530 con 0x7f26880adda0 2023-10-24T23:53:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26880c4030 msgr2=0x7f268815f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 --2- 172.21.15.107:0/2517552914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26880c4030 0x7f268815f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 >> v2:172.21.15.110:3300/0 conn(0x7f26880c2000 msgr2=0x7f26880ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 --2- 172.21.15.107:0/2517552914 >> v2:172.21.15.110:3300/0 conn(0x7f26880c2000 0x7f26880ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268806a740 con 0x7f26880adda0 2023-10-24T23:53:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f267c001680 con 0x7f26880adda0 2023-10-24T23:53:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 >> v1:172.21.15.107:6789/0 conn(0x7f26880adda0 legacy=0x7f26880c3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 --2- 172.21.15.107:0/2517552914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2670003980 0x7f2670003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 --2- 172.21.15.107:0/2517552914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26700047d0 0x7f2670004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 --2- 172.21.15.107:0/2517552914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26880c4030 0x7f2670005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26880bd250 con 0x7f2670003980 2023-10-24T23:53:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26880bcf00 con 0x7f26700047d0 2023-10-24T23:53:45.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f267000b990 con 0x7f26880c4030 2023-10-24T23:53:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2687fff640 1 --2- 172.21.15.107:0/2517552914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2670003980 0x7f2670003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f26877fe640 1 --2- 172.21.15.107:0/2517552914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26700047d0 0x7f2670004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2687fff640 1 -- 172.21.15.107:0/2517552914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26880c4030 msgr2=0x7f2670005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2687fff640 1 --2- 172.21.15.107:0/2517552914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26880c4030 0x7f2670005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2687fff640 1 -- 172.21.15.107:0/2517552914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26700047d0 msgr2=0x7f2670004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:45.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2687fff640 1 --2- 172.21.15.107:0/2517552914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26700047d0 0x7f2670004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.909+0000 7f2687fff640 1 -- 172.21.15.107:0/2517552914 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2670003530 con 0x7f2670003980 2023-10-24T23:53:45.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f2687fff640 1 --2- 172.21.15.107:0/2517552914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2670003980 0x7f2670003da0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2678009ae0 tx=0x7f267801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be26d83057de1fc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26780a5550 con 0x7f2670003980 2023-10-24T23:53:45.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26780a56b0 con 0x7f2670003980 2023-10-24T23:53:45.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f2686ffd640 1 -- 172.21.15.107:0/2517552914 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26780a5a60 con 0x7f2670003980 2023-10-24T23:53:45.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 -- 172.21.15.107:0/2517552914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2670003980 msgr2=0x7f2670003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 --2- 172.21.15.107:0/2517552914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2670003980 0x7f2670003da0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2678009ae0 tx=0x7f267801ca60 comp rx=0 tx=0).stop 2023-10-24T23:53:45.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268cf77640 1 -- 172.21.15.107:0/2517552914 reap_dead start 2023-10-24T23:53:45.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 -- 172.21.15.107:0/2517552914 shutdown_connections 2023-10-24T23:53:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 -- 172.21.15.107:0/2517552914 >> 172.21.15.107:0/2517552914 conn(0x7f2688070ef0 msgr2=0x7f2688071300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 -- 172.21.15.107:0/2517552914 shutdown_connections 2023-10-24T23:53:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 -- 172.21.15.107:0/2517552914 wait complete. 2023-10-24T23:53:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 Processor -- start 2023-10-24T23:53:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 -- start start 2023-10-24T23:53:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2670003980 0x7f2688148840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26880c64e0 0x7f26881ecc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26881ed190 0x7f2688147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2688158b60 con 0x7f26880c64e0 2023-10-24T23:53:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26881ed5f0 con 0x7f2670003980 2023-10-24T23:53:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268e9da640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2688148d80 con 0x7f26881ed190 2023-10-24T23:53:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f26877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26880c64e0 0x7f26881ecc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f26877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26880c64e0 0x7f26881ecc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36982/0 (socket says 172.21.15.107:36982) 2023-10-24T23:53:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f26877fe640 1 -- 172.21.15.107:0/2512473323 learned_addr learned my addr 172.21.15.107:0/2512473323 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268cf77640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26881ed190 0x7f2688147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f26877fe640 1 -- 172.21.15.107:0/2512473323 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26881ed190 msgr2=0x7f2688147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f2687fff640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2670003980 0x7f2688148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f26877fe640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26881ed190 0x7f2688147790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f26877fe640 1 -- 172.21.15.107:0/2512473323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2670003980 msgr2=0x7f2688148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:45.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f26877fe640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2670003980 0x7f2688148840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:45.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f26877fe640 1 -- 172.21.15.107:0/2512473323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2688155e70 con 0x7f26880c64e0 2023-10-24T23:53:45.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f268cf77640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26881ed190 0x7f2688147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:45.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.913+0000 7f2687fff640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2670003980 0x7f2688148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:45.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f26877fe640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26880c64e0 0x7f26881ecc50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f267c00e7c0 tx=0x7f267c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f2684ff9640 1 -- 172.21.15.107:0/2512473323 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c00eed0 con 0x7f26880c64e0 2023-10-24T23:53:45.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f2684ff9640 1 -- 172.21.15.107:0/2512473323 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267c00c870 con 0x7f26880c64e0 2023-10-24T23:53:45.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f2684ff9640 1 -- 172.21.15.107:0/2512473323 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c00cc20 con 0x7f26880c64e0 2023-10-24T23:53:45.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f268e9da640 1 -- 172.21.15.107:0/2512473323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2688148560 con 0x7f26880c64e0 2023-10-24T23:53:45.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f268e9da640 1 -- 172.21.15.107:0/2512473323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26881ec730 con 0x7f26880c64e0 2023-10-24T23:53:45.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f268e9da640 1 -- 172.21.15.107:0/2512473323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2688073c30 con 0x7f26880c64e0 2023-10-24T23:53:45.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f2684ff9640 1 -- 172.21.15.107:0/2512473323 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f267c00cd80 con 0x7f26880c64e0 2023-10-24T23:53:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f2684ff9640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2674065e90 0x7f2674068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f2687fff640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2674065e90 0x7f2674068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f2684ff9640 1 -- 172.21.15.107:0/2512473323 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f267c083ac0 con 0x7f26880c64e0 2023-10-24T23:53:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.917+0000 7f2687fff640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2674065e90 0x7f2674068350 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f26780af000 tx=0x7f26780046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:45.921+0000 7f2684ff9640 1 -- 172.21.15.107:0/2512473323 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f267c054810 con 0x7f26880c64e0 2023-10-24T23:53:46.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 -- 172.21.15.107:0/2512473323 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7f268804e130 con 0x7f26880c64e0 2023-10-24T23:53:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f2684ff9640 1 -- 172.21.15.107:0/2512473323 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v29) v1 ==== 95+0+11 (secure 0 0 0) 0x7f267c083e60 con 0x7f26880c64e0 2023-10-24T23:53:46.083 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 -- 172.21.15.107:0/2512473323 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2674065e90 msgr2=0x7f2674068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2674065e90 0x7f2674068350 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f26780af000 tx=0x7f26780046d0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 -- 172.21.15.107:0/2512473323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26880c64e0 msgr2=0x7f26881ecc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26880c64e0 0x7f26881ecc50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f267c00e7c0 tx=0x7f267c00ec80 comp rx=0 tx=0).stop 2023-10-24T23:53:46.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 -- 172.21.15.107:0/2512473323 shutdown_connections 2023-10-24T23:53:46.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2674065e90 0x7f2674068350 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26881ed190 0x7f2688147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26880c64e0 0x7f26881ecc50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 --2- 172.21.15.107:0/2512473323 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2670003980 0x7f2688148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 -- 172.21.15.107:0/2512473323 >> 172.21.15.107:0/2512473323 conn(0x7f2688070ef0 msgr2=0x7f26880c71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:46.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 -- 172.21.15.107:0/2512473323 shutdown_connections 2023-10-24T23:53:46.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.077+0000 7f268e9da640 1 -- 172.21.15.107:0/2512473323 wait complete. 2023-10-24T23:53:46.101 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-10-24T23:53:46.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:46.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.209+0000 7f53463a4640 1 Processor -- start 2023-10-24T23:53:46.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.209+0000 7f53463a4640 1 -- start start 2023-10-24T23:53:46.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f53463a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5340152040 0x7f53400c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f53463a4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5340155680 con 0x7f53400c8100 2023-10-24T23:53:46.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f53463a4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5340154a70 con 0x7f534006a4b0 2023-10-24T23:53:46.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f53463a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5340154d50 con 0x7f5340152040 2023-10-24T23:53:46.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f53400c8100 0x7f53400abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45144/0 (socket says 172.21.15.107:45144) 2023-10-24T23:53:46.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533f7fe640 1 -- 172.21.15.107:0/465253860 learned_addr learned my addr 172.21.15.107:0/465253860 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:46.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533ffff640 1 --2- 172.21.15.107:0/465253860 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5340152040 0x7f53400c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093308191 0 0) 0x7f5340155680 con 0x7f53400c8100 2023-10-24T23:53:46.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f532c003530 con 0x7f53400c8100 2023-10-24T23:53:46.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2245543829 0 0) 0x7f532c003530 con 0x7f53400c8100 2023-10-24T23:53:46.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5340152040 msgr2=0x7f53400c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:46.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 --2- 172.21.15.107:0/465253860 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5340152040 0x7f53400c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 >> v1:172.21.15.110:6789/0 conn(0x7f534006a4b0 legacy=0x7f53400ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5340155030 con 0x7f53400c8100 2023-10-24T23:53:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5328001680 con 0x7f53400c8100 2023-10-24T23:53:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 >> v1:172.21.15.107:6789/0 conn(0x7f53400c8100 legacy=0x7f53400abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 --2- 172.21.15.107:0/465253860 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5340152040 0x7f532c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 --2- 172.21.15.107:0/465253860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c003f30 0x7f532c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 --2- 172.21.15.107:0/465253860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532c004d10 0x7f532c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5340154d50 con 0x7f532c004d10 2023-10-24T23:53:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532c00bbd0 con 0x7f532c003f30 2023-10-24T23:53:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532c00bd90 con 0x7f5340152040 2023-10-24T23:53:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f5344941640 1 --2- 172.21.15.107:0/465253860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532c004d10 0x7f532c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f533f7fe640 1 --2- 172.21.15.107:0/465253860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c003f30 0x7f532c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f5344941640 1 -- 172.21.15.107:0/465253860 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5340152040 msgr2=0x7f532c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f5344941640 1 --2- 172.21.15.107:0/465253860 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5340152040 0x7f532c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f5344941640 1 -- 172.21.15.107:0/465253860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c003f30 msgr2=0x7f532c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f5344941640 1 --2- 172.21.15.107:0/465253860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f532c003f30 0x7f532c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.213+0000 7f5344941640 1 -- 172.21.15.107:0/465253860 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c003530 con 0x7f532c004d10 2023-10-24T23:53:46.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f5344941640 1 --2- 172.21.15.107:0/465253860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532c004d10 0x7f532c005190 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5330001e70 tx=0x7f533001c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c176b01dac2f25 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:46.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330003980 con 0x7f532c004d10 2023-10-24T23:53:46.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5330003ae0 con 0x7f532c004d10 2023-10-24T23:53:46.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f533effd640 1 -- 172.21.15.107:0/465253860 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330003e90 con 0x7f532c004d10 2023-10-24T23:53:46.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 -- 172.21.15.107:0/465253860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532c004d10 msgr2=0x7f532c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 --2- 172.21.15.107:0/465253860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f532c004d10 0x7f532c005190 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5330001e70 tx=0x7f533001c760 comp rx=0 tx=0).stop 2023-10-24T23:53:46.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f5344941640 1 -- 172.21.15.107:0/465253860 reap_dead start 2023-10-24T23:53:46.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 -- 172.21.15.107:0/465253860 shutdown_connections 2023-10-24T23:53:46.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 -- 172.21.15.107:0/465253860 >> 172.21.15.107:0/465253860 conn(0x7f53400ca200 msgr2=0x7f53400ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:46.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 -- 172.21.15.107:0/465253860 shutdown_connections 2023-10-24T23:53:46.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 -- 172.21.15.107:0/465253860 wait complete. 2023-10-24T23:53:46.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 Processor -- start 2023-10-24T23:53:46.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 -- start start 2023-10-24T23:53:46.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f532c004d10 0x7f53401f01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53401589e0 0x7f53400be870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53401f06e0 0x7f53400bedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53400be470 con 0x7f53401f06e0 2023-10-24T23:53:46.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53401582d0 con 0x7f53401589e0 2023-10-24T23:53:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53401f0c30 con 0x7f532c004d10 2023-10-24T23:53:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f5344941640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53401f06e0 0x7f53400bedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f5344941640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53401f06e0 0x7f53400bedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36998/0 (socket says 172.21.15.107:36998) 2023-10-24T23:53:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f5344941640 1 -- 172.21.15.107:0/593934364 learned_addr learned my addr 172.21.15.107:0/593934364 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f533ffff640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f532c004d10 0x7f53401f01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f533f7fe640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53401589e0 0x7f53400be870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f5344941640 1 -- 172.21.15.107:0/593934364 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f532c004d10 msgr2=0x7f53401f01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f5344941640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f532c004d10 0x7f53401f01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f5344941640 1 -- 172.21.15.107:0/593934364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53401589e0 msgr2=0x7f53400be870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f5344941640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53401589e0 0x7f53400be870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f5344941640 1 -- 172.21.15.107:0/593934364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5340157510 con 0x7f53401f06e0 2023-10-24T23:53:46.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f5344941640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53401f06e0 0x7f53400bedb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5330098060 tx=0x7f532c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:46.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f533cff9640 1 -- 172.21.15.107:0/593934364 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5330006d60 con 0x7f53401f06e0 2023-10-24T23:53:46.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f533cff9640 1 -- 172.21.15.107:0/593934364 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5330005030 con 0x7f53401f06e0 2023-10-24T23:53:46.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 -- 172.21.15.107:0/593934364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53401577f0 con 0x7f53401f06e0 2023-10-24T23:53:46.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f533cff9640 1 -- 172.21.15.107:0/593934364 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53300053a0 con 0x7f53401f06e0 2023-10-24T23:53:46.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.217+0000 7f53463a4640 1 -- 172.21.15.107:0/593934364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5340157db0 con 0x7f53401f06e0 2023-10-24T23:53:46.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.221+0000 7f53463a4640 1 -- 172.21.15.107:0/593934364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53400746e0 con 0x7f53401f06e0 2023-10-24T23:53:46.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.221+0000 7f533cff9640 1 -- 172.21.15.107:0/593934364 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5330005500 con 0x7f53401f06e0 2023-10-24T23:53:46.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.221+0000 7f533cff9640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e90 0x7f5320068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.221+0000 7f533cff9640 1 -- 172.21.15.107:0/593934364 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f533011e3a0 con 0x7f53401f06e0 2023-10-24T23:53:46.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.221+0000 7f533ffff640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e90 0x7f5320068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.221+0000 7f533ffff640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e90 0x7f5320068350 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f53380047c0 tx=0x7f5338008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:46.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.225+0000 7f533cff9640 1 -- 172.21.15.107:0/593934364 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f53300eb780 con 0x7f53401f06e0 2023-10-24T23:53:46.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.409+0000 7f53463a4640 1 -- 172.21.15.107:0/593934364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53400be470 con 0x7f53401f06e0 2023-10-24T23:53:46.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.409+0000 7f533cff9640 1 -- 172.21.15.107:0/593934364 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f53300eeda0 con 0x7f53401f06e0 2023-10-24T23:53:46.415 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:46.415 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:53:46.416 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:53:46.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.409+0000 7f53463a4640 1 -- 172.21.15.107:0/593934364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e90 msgr2=0x7f5320068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.409+0000 7f53463a4640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e90 0x7f5320068350 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f53380047c0 tx=0x7f5338008040 comp rx=0 tx=0).stop 2023-10-24T23:53:46.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.413+0000 7f53463a4640 1 -- 172.21.15.107:0/593934364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53401f06e0 msgr2=0x7f53400bedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.413+0000 7f53463a4640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53401f06e0 0x7f53400bedb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5330098060 tx=0x7f532c005800 comp rx=0 tx=0).stop 2023-10-24T23:53:46.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.413+0000 7f53463a4640 1 -- 172.21.15.107:0/593934364 shutdown_connections 2023-10-24T23:53:46.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.413+0000 7f53463a4640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53401f06e0 0x7f53400bedb0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.413+0000 7f53463a4640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53401589e0 0x7f53400be870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.413+0000 7f53463a4640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e90 0x7f5320068350 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.413+0000 7f53463a4640 1 --2- 172.21.15.107:0/593934364 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f532c004d10 0x7f53401f01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.413+0000 7f53463a4640 1 -- 172.21.15.107:0/593934364 >> 172.21.15.107:0/593934364 conn(0x7f53400ca200 msgr2=0x7f53401516e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:46.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.413+0000 7f53463a4640 1 -- 172.21.15.107:0/593934364 shutdown_connections 2023-10-24T23:53:46.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.413+0000 7f53463a4640 1 -- 172.21.15.107:0/593934364 wait complete. 2023-10-24T23:53:46.433 INFO:tasks.mds_thrash:Ready to start thrashing 2023-10-24T23:53:46.433 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-10-24T23:53:46.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:46.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.537+0000 7ff9d952e640 1 Processor -- start 2023-10-24T23:53:46.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.537+0000 7ff9d952e640 1 -- start start 2023-10-24T23:53:46.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9d40aa330 0x7ff9d406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff9d40a9450 0x7ff9d4071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d4152d40 con 0x7ff9d413bb00 2023-10-24T23:53:46.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d4156100 con 0x7ff9d40aa330 2023-10-24T23:53:46.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d41527c0 con 0x7ff9d40a9450 2023-10-24T23:53:46.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9cbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9d40aa330 0x7ff9d406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9cbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9d40aa330 0x7ff9d406e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37012/0 (socket says 172.21.15.107:37012) 2023-10-24T23:53:46.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9cbfff640 1 -- 172.21.15.107:0/1236665571 learned_addr learned my addr 172.21.15.107:0/1236665571 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:46.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9cbfff640 1 -- 172.21.15.107:0/1236665571 >> v2:172.21.15.110:3300/0 conn(0x7ff9d40a9450 msgr2=0x7ff9d4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:46.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9cbfff640 1 --2- 172.21.15.107:0/1236665571 >> v2:172.21.15.110:3300/0 conn(0x7ff9d40a9450 0x7ff9d4071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9cbfff640 1 -- 172.21.15.107:0/1236665571 >> v1:172.21.15.110:6789/0 conn(0x7ff9d413bb00 legacy=0x7ff9d41531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:46.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9cbfff640 1 -- 172.21.15.107:0/1236665571 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d406d660 con 0x7ff9d40aa330 2023-10-24T23:53:46.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9cbfff640 1 --2- 172.21.15.107:0/1236665571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9d40aa330 0x7ff9d406e250 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009870 tx=0x7ff9bc01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6717413d331cc49f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:46.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d27fc640 1 -- 172.21.15.107:0/1236665571 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc0ab600 con 0x7ff9d40aa330 2023-10-24T23:53:46.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d27fc640 1 -- 172.21.15.107:0/1236665571 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9bc0a7050 con 0x7ff9d40aa330 2023-10-24T23:53:46.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d27fc640 1 -- 172.21.15.107:0/1236665571 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc0ab980 con 0x7ff9d40aa330 2023-10-24T23:53:46.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 -- 172.21.15.107:0/1236665571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9d40aa330 msgr2=0x7ff9d406e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 --2- 172.21.15.107:0/1236665571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9d40aa330 0x7ff9d406e250 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009870 tx=0x7ff9bc01c3a0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 -- 172.21.15.107:0/1236665571 shutdown_connections 2023-10-24T23:53:46.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 --2- 172.21.15.107:0/1236665571 >> v2:172.21.15.110:3300/0 conn(0x7ff9d40a9450 0x7ff9d4071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 --2- 172.21.15.107:0/1236665571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9d40aa330 0x7ff9d406e250 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 -- 172.21.15.107:0/1236665571 >> 172.21.15.107:0/1236665571 conn(0x7ff9d4070f60 msgr2=0x7ff9d4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:46.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 -- 172.21.15.107:0/1236665571 shutdown_connections 2023-10-24T23:53:46.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.541+0000 7ff9d952e640 1 -- 172.21.15.107:0/1236665571 wait complete. 2023-10-24T23:53:46.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d952e640 1 Processor -- start 2023-10-24T23:53:46.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d952e640 1 -- start start 2023-10-24T23:53:46.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d952e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9d40a9450 0x7ff9d4133d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d952e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d40aa330 0x7ff9d4134b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d952e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9d413bb00 0x7ff9d4135500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d952e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d40c7b90 con 0x7ff9d413bb00 2023-10-24T23:53:46.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d952e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d4133300 con 0x7ff9d40aa330 2023-10-24T23:53:46.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d952e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d4134260 con 0x7ff9d40a9450 2023-10-24T23:53:46.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9d413bb00 0x7ff9d4135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9d413bb00 0x7ff9d4135500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37022/0 (socket says 172.21.15.107:37022) 2023-10-24T23:53:46.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d37fe640 1 -- 172.21.15.107:0/3581593396 learned_addr learned my addr 172.21.15.107:0/3581593396 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9cbfff640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d40aa330 0x7ff9d4134b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d37fe640 1 -- 172.21.15.107:0/3581593396 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9d40a9450 msgr2=0x7ff9d4133d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d2ffd640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9d40a9450 0x7ff9d4133d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d37fe640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9d40a9450 0x7ff9d4133d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d37fe640 1 -- 172.21.15.107:0/3581593396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d40aa330 msgr2=0x7ff9d4134b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d37fe640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d40aa330 0x7ff9d4134b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d37fe640 1 -- 172.21.15.107:0/3581593396 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9bc0094d0 con 0x7ff9d413bb00 2023-10-24T23:53:46.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9cbfff640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d40aa330 0x7ff9d4134b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:46.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d37fe640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9d413bb00 0x7ff9d4135500 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00b9a0 tx=0x7ff9cc00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:46.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9cb7fe640 1 -- 172.21.15.107:0/3581593396 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc00cae0 con 0x7ff9d413bb00 2023-10-24T23:53:46.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9cb7fe640 1 -- 172.21.15.107:0/3581593396 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9cc00cc60 con 0x7ff9d413bb00 2023-10-24T23:53:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9cb7fe640 1 -- 172.21.15.107:0/3581593396 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc014440 con 0x7ff9d413bb00 2023-10-24T23:53:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d952e640 1 -- 172.21.15.107:0/3581593396 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d4130d30 con 0x7ff9d413bb00 2023-10-24T23:53:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d952e640 1 -- 172.21.15.107:0/3581593396 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d4133540 con 0x7ff9d413bb00 2023-10-24T23:53:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d952e640 1 -- 172.21.15.107:0/3581593396 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d40ae030 con 0x7ff9d413bb00 2023-10-24T23:53:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9cb7fe640 1 -- 172.21.15.107:0/3581593396 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff9cc01d3b0 con 0x7ff9d413bb00 2023-10-24T23:53:46.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9cb7fe640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9b4065e60 0x7ff9b4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9cb7fe640 1 -- 172.21.15.107:0/3581593396 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7ff9cc083a90 con 0x7ff9d413bb00 2023-10-24T23:53:46.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.545+0000 7ff9d2ffd640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9b4065e60 0x7ff9b4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.549+0000 7ff9d2ffd640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9b4065e60 0x7ff9b4068320 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400a790 tx=0x7ff9c4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:46.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.553+0000 7ff9cb7fe640 1 -- 172.21.15.107:0/3581593396 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff9cc050d50 con 0x7ff9d413bb00 2023-10-24T23:53:46.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.701+0000 7ff9d952e640 1 -- 172.21.15.107:0/3581593396 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff9d40746e0 con 0x7ff9b4065e60 2023-10-24T23:53:46.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.741+0000 7ff9cb7fe640 1 -- 172.21.15.107:0/3581593396 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525098 (secure 0 0 0) 0x7ff9d40746e0 con 0x7ff9b4065e60 2023-10-24T23:53:46.751 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:46.755 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:46.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 -- 172.21.15.107:0/3581593396 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9b4065e60 msgr2=0x7ff9b4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9b4065e60 0x7ff9b4068320 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400a790 tx=0x7ff9c4008040 comp rx=0 tx=0).stop 2023-10-24T23:53:46.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 -- 172.21.15.107:0/3581593396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9d413bb00 msgr2=0x7ff9d4135500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9d413bb00 0x7ff9d4135500 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00b9a0 tx=0x7ff9cc00be60 comp rx=0 tx=0).stop 2023-10-24T23:53:46.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 -- 172.21.15.107:0/3581593396 shutdown_connections 2023-10-24T23:53:46.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9d413bb00 0x7ff9d4135500 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9d40aa330 0x7ff9d4134b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9b4065e60 0x7ff9b4068320 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 --2- 172.21.15.107:0/3581593396 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9d40a9450 0x7ff9d4133d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 -- 172.21.15.107:0/3581593396 >> 172.21.15.107:0/3581593396 conn(0x7ff9d4070f60 msgr2=0x7ff9d4073340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:46.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 -- 172.21.15.107:0/3581593396 shutdown_connections 2023-10-24T23:53:46.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.745+0000 7ff9d952e640 1 -- 172.21.15.107:0/3581593396 wait complete. 2023-10-24T23:53:46.768 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2023-10-24T23:53:45.291077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595978,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":253,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235560,"kb_used_data":3560,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754739160,"statfs":{"total":773094113280,"available":772852899840,"internally_reserved":0,"allocated":3645440,"data_stored":2146281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":886,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003731"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603561+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-24T23:53:13.603561+0000","last_peered":"2023-10-24T23:53:13.603561+0000","last_clean":"2023-10-24T23:53:13.603561+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-24T23:53:13.603561+0000","last_undegraded":"2023-10-24T23:53:13.603561+0000","last_fullsized":"2023-10-24T23:53:13.603561+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:51:20.137796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794961+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-24T23:53:12.794961+0000","last_peered":"2023-10-24T23:53:12.794961+0000","last_clean":"2023-10-24T23:53:12.794961+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-24T23:53:12.794961+0000","last_undegraded":"2023-10-24T23:53:12.794961+0000","last_fullsized":"2023-10-24T23:53:12.794961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:00:20.101379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788258+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-24T23:53:11.788258+0000","last_peered":"2023-10-24T23:53:11.788258+0000","last_clean":"2023-10-24T23:53:11.788258+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-24T23:53:11.788258+0000","last_undegraded":"2023-10-24T23:53:11.788258+0000","last_fullsized":"2023-10-24T23:53:11.788258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791737+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-24T23:53:11.791737+0000","last_peered":"2023-10-24T23:53:11.791737+0000","last_clean":"2023-10-24T23:53:11.791737+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-24T23:53:11.791737+0000","last_undegraded":"2023-10-24T23:53:11.791737+0000","last_fullsized":"2023-10-24T23:53:11.791737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794565+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-24T23:53:12.794565+0000","last_peered":"2023-10-24T23:53:12.794565+0000","last_clean":"2023-10-24T23:53:12.794565+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-24T23:53:12.794565+0000","last_undegraded":"2023-10-24T23:53:12.794565+0000","last_fullsized":"2023-10-24T23:53:12.794565+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:59.368156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.601719+0000","last_change":"2023-10-24T23:53:11.444089+0000","last_active":"2023-10-24T23:53:13.601719+0000","last_peered":"2023-10-24T23:53:13.601719+0000","last_clean":"2023-10-24T23:53:13.601719+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-24T23:53:13.601719+0000","last_undegraded":"2023-10-24T23:53:13.601719+0000","last_fullsized":"2023-10-24T23:53:13.601719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:54:06.875882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807520+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-24T23:53:11.807520+0000","last_peered":"2023-10-24T23:53:11.807520+0000","last_clean":"2023-10-24T23:53:11.807520+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-24T23:53:11.807520+0000","last_undegraded":"2023-10-24T23:53:11.807520+0000","last_fullsized":"2023-10-24T23:53:11.807520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806881+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-24T23:53:11.806881+0000","last_peered":"2023-10-24T23:53:11.806881+0000","last_clean":"2023-10-24T23:53:11.806881+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-24T23:53:11.806881+0000","last_undegraded":"2023-10-24T23:53:11.806881+0000","last_fullsized":"2023-10-24T23:53:11.806881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602008+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-24T23:53:13.602008+0000","last_peered":"2023-10-24T23:53:13.602008+0000","last_clean":"2023-10-24T23:53:13.602008+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-24T23:53:13.602008+0000","last_undegraded":"2023-10-24T23:53:13.602008+0000","last_fullsized":"2023-10-24T23:53:13.602008+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:38:56.643025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101201+0000","last_change":"2023-10-24T23:53:11.438681+0000","last_active":"2023-10-24T23:53:13.101201+0000","last_peered":"2023-10-24T23:53:13.101201+0000","last_clean":"2023-10-24T23:53:13.101201+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-24T23:53:13.101201+0000","last_undegraded":"2023-10-24T23:53:13.101201+0000","last_fullsized":"2023-10-24T23:53:13.101201+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:50.139697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832718+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-24T23:53:11.832718+0000","last_peered":"2023-10-24T23:53:11.832718+0000","last_clean":"2023-10-24T23:53:11.832718+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-24T23:53:11.832718+0000","last_undegraded":"2023-10-24T23:53:11.832718+0000","last_fullsized":"2023-10-24T23:53:11.832718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.810798+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-24T23:53:11.810798+0000","last_peered":"2023-10-24T23:53:11.810798+0000","last_clean":"2023-10-24T23:53:11.810798+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-24T23:53:11.810798+0000","last_undegraded":"2023-10-24T23:53:11.810798+0000","last_fullsized":"2023-10-24T23:53:11.810798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101349+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-24T23:53:13.101349+0000","last_peered":"2023-10-24T23:53:13.101349+0000","last_clean":"2023-10-24T23:53:13.101349+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-24T23:53:13.101349+0000","last_undegraded":"2023-10-24T23:53:13.101349+0000","last_fullsized":"2023-10-24T23:53:13.101349+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:22.974333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796171+0000","last_change":"2023-10-24T23:53:11.454537+0000","last_active":"2023-10-24T23:53:12.796171+0000","last_peered":"2023-10-24T23:53:12.796171+0000","last_clean":"2023-10-24T23:53:12.796171+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-24T23:53:12.796171+0000","last_undegraded":"2023-10-24T23:53:12.796171+0000","last_fullsized":"2023-10-24T23:53:12.796171+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:14:06.792651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.830575+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-24T23:53:11.830575+0000","last_peered":"2023-10-24T23:53:11.830575+0000","last_clean":"2023-10-24T23:53:11.830575+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-24T23:53:11.830575+0000","last_undegraded":"2023-10-24T23:53:11.830575+0000","last_fullsized":"2023-10-24T23:53:11.830575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800231+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-24T23:53:11.800231+0000","last_peered":"2023-10-24T23:53:11.800231+0000","last_clean":"2023-10-24T23:53:11.800231+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-24T23:53:11.800231+0000","last_undegraded":"2023-10-24T23:53:11.800231+0000","last_fullsized":"2023-10-24T23:53:11.800231+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605269+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-24T23:53:13.605269+0000","last_peered":"2023-10-24T23:53:13.605269+0000","last_clean":"2023-10-24T23:53:13.605269+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-24T23:53:13.605269+0000","last_undegraded":"2023-10-24T23:53:13.605269+0000","last_fullsized":"2023-10-24T23:53:13.605269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:35:39.879563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795373+0000","last_change":"2023-10-24T23:53:11.444138+0000","last_active":"2023-10-24T23:53:12.795373+0000","last_peered":"2023-10-24T23:53:12.795373+0000","last_clean":"2023-10-24T23:53:12.795373+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-24T23:53:12.795373+0000","last_undegraded":"2023-10-24T23:53:12.795373+0000","last_fullsized":"2023-10-24T23:53:12.795373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:27:38.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788658+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-24T23:53:11.788658+0000","last_peered":"2023-10-24T23:53:11.788658+0000","last_clean":"2023-10-24T23:53:11.788658+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-24T23:53:11.788658+0000","last_undegraded":"2023-10-24T23:53:11.788658+0000","last_fullsized":"2023-10-24T23:53:11.788658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.797645+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-24T23:53:11.797645+0000","last_peered":"2023-10-24T23:53:11.797645+0000","last_clean":"2023-10-24T23:53:11.797645+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-24T23:53:11.797645+0000","last_undegraded":"2023-10-24T23:53:11.797645+0000","last_fullsized":"2023-10-24T23:53:11.797645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795697+0000","last_change":"2023-10-24T23:53:11.832809+0000","last_active":"2023-10-24T23:53:12.795697+0000","last_peered":"2023-10-24T23:53:12.795697+0000","last_clean":"2023-10-24T23:53:12.795697+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-24T23:53:12.795697+0000","last_undegraded":"2023-10-24T23:53:12.795697+0000","last_fullsized":"2023-10-24T23:53:12.795697+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:01.151315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604963+0000","last_change":"2023-10-24T23:53:11.442035+0000","last_active":"2023-10-24T23:53:13.604963+0000","last_peered":"2023-10-24T23:53:13.604963+0000","last_clean":"2023-10-24T23:53:13.604963+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-24T23:53:13.604963+0000","last_undegraded":"2023-10-24T23:53:13.604963+0000","last_fullsized":"2023-10-24T23:53:13.604963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:13.717844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.811329+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-24T23:53:11.811329+0000","last_peered":"2023-10-24T23:53:11.811329+0000","last_clean":"2023-10-24T23:53:11.811329+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-24T23:53:11.811329+0000","last_undegraded":"2023-10-24T23:53:11.811329+0000","last_fullsized":"2023-10-24T23:53:11.811329+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807632+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-24T23:53:11.807632+0000","last_peered":"2023-10-24T23:53:11.807632+0000","last_clean":"2023-10-24T23:53:11.807632+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-24T23:53:11.807632+0000","last_undegraded":"2023-10-24T23:53:11.807632+0000","last_fullsized":"2023-10-24T23:53:11.807632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797540+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-24T23:53:12.797540+0000","last_peered":"2023-10-24T23:53:12.797540+0000","last_clean":"2023-10-24T23:53:12.797540+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-24T23:53:12.797540+0000","last_undegraded":"2023-10-24T23:53:12.797540+0000","last_fullsized":"2023-10-24T23:53:12.797540+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:26.012711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604200+0000","last_change":"2023-10-24T23:53:11.448032+0000","last_active":"2023-10-24T23:53:13.604200+0000","last_peered":"2023-10-24T23:53:13.604200+0000","last_clean":"2023-10-24T23:53:13.604200+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-24T23:53:13.604200+0000","last_undegraded":"2023-10-24T23:53:13.604200+0000","last_fullsized":"2023-10-24T23:53:13.604200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:50:32.490810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791720+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-24T23:53:11.791720+0000","last_peered":"2023-10-24T23:53:11.791720+0000","last_clean":"2023-10-24T23:53:11.791720+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-24T23:53:11.791720+0000","last_undegraded":"2023-10-24T23:53:11.791720+0000","last_fullsized":"2023-10-24T23:53:11.791720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788475+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-24T23:53:11.788475+0000","last_peered":"2023-10-24T23:53:11.788475+0000","last_clean":"2023-10-24T23:53:11.788475+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-24T23:53:11.788475+0000","last_undegraded":"2023-10-24T23:53:11.788475+0000","last_fullsized":"2023-10-24T23:53:11.788475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795316+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-24T23:53:12.795316+0000","last_peered":"2023-10-24T23:53:12.795316+0000","last_clean":"2023-10-24T23:53:12.795316+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-24T23:53:12.795316+0000","last_undegraded":"2023-10-24T23:53:12.795316+0000","last_fullsized":"2023-10-24T23:53:12.795316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:10.820967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603323+0000","last_change":"2023-10-24T23:53:11.446925+0000","last_active":"2023-10-24T23:53:13.603323+0000","last_peered":"2023-10-24T23:53:13.603323+0000","last_clean":"2023-10-24T23:53:13.603323+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-24T23:53:13.603323+0000","last_undegraded":"2023-10-24T23:53:13.603323+0000","last_fullsized":"2023-10-24T23:53:13.603323+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:59:52.485469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804040+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-24T23:53:11.804040+0000","last_peered":"2023-10-24T23:53:11.804040+0000","last_clean":"2023-10-24T23:53:11.804040+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-24T23:53:11.804040+0000","last_undegraded":"2023-10-24T23:53:11.804040+0000","last_fullsized":"2023-10-24T23:53:11.804040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791850+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-24T23:53:11.791850+0000","last_peered":"2023-10-24T23:53:11.791850+0000","last_clean":"2023-10-24T23:53:11.791850+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-24T23:53:11.791850+0000","last_undegraded":"2023-10-24T23:53:11.791850+0000","last_fullsized":"2023-10-24T23:53:11.791850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795971+0000","last_change":"2023-10-24T23:53:11.849056+0000","last_active":"2023-10-24T23:53:12.795971+0000","last_peered":"2023-10-24T23:53:12.795971+0000","last_clean":"2023-10-24T23:53:12.795971+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-24T23:53:12.795971+0000","last_undegraded":"2023-10-24T23:53:12.795971+0000","last_fullsized":"2023-10-24T23:53:12.795971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:08.932942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803866+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-24T23:53:11.803866+0000","last_peered":"2023-10-24T23:53:11.803866+0000","last_clean":"2023-10-24T23:53:11.803866+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-24T23:53:11.803866+0000","last_undegraded":"2023-10-24T23:53:11.803866+0000","last_fullsized":"2023-10-24T23:53:11.803866+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788258+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-24T23:53:11.788258+0000","last_peered":"2023-10-24T23:53:11.788258+0000","last_clean":"2023-10-24T23:53:11.788258+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-24T23:53:11.788258+0000","last_undegraded":"2023-10-24T23:53:11.788258+0000","last_fullsized":"2023-10-24T23:53:11.788258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602180+0000","last_change":"2023-10-24T23:53:11.447167+0000","last_active":"2023-10-24T23:53:13.602180+0000","last_peered":"2023-10-24T23:53:13.602180+0000","last_clean":"2023-10-24T23:53:13.602180+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-24T23:53:13.602180+0000","last_undegraded":"2023-10-24T23:53:13.602180+0000","last_fullsized":"2023-10-24T23:53:13.602180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:06:25.006728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794920+0000","last_change":"2023-10-24T23:53:11.813704+0000","last_active":"2023-10-24T23:53:12.794920+0000","last_peered":"2023-10-24T23:53:12.794920+0000","last_clean":"2023-10-24T23:53:12.794920+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-24T23:53:12.794920+0000","last_undegraded":"2023-10-24T23:53:12.794920+0000","last_fullsized":"2023-10-24T23:53:12.794920+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:39:09.996039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804467+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-24T23:53:11.804467+0000","last_peered":"2023-10-24T23:53:11.804467+0000","last_clean":"2023-10-24T23:53:11.804467+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-24T23:53:11.804467+0000","last_undegraded":"2023-10-24T23:53:11.804467+0000","last_fullsized":"2023-10-24T23:53:11.804467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838755+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-24T23:53:11.838755+0000","last_peered":"2023-10-24T23:53:11.838755+0000","last_clean":"2023-10-24T23:53:11.838755+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-24T23:53:11.838755+0000","last_undegraded":"2023-10-24T23:53:11.838755+0000","last_fullsized":"2023-10-24T23:53:11.838755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795718+0000","last_change":"2023-10-24T23:53:11.450974+0000","last_active":"2023-10-24T23:53:12.795718+0000","last_peered":"2023-10-24T23:53:12.795718+0000","last_clean":"2023-10-24T23:53:12.795718+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-24T23:53:12.795718+0000","last_undegraded":"2023-10-24T23:53:12.795718+0000","last_fullsized":"2023-10-24T23:53:12.795718+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:51:05.443664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602086+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-24T23:53:13.602086+0000","last_peered":"2023-10-24T23:53:13.602086+0000","last_clean":"2023-10-24T23:53:13.602086+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-24T23:53:13.602086+0000","last_undegraded":"2023-10-24T23:53:13.602086+0000","last_fullsized":"2023-10-24T23:53:13.602086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:37:30.278659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825877+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-24T23:53:11.825877+0000","last_peered":"2023-10-24T23:53:11.825877+0000","last_clean":"2023-10-24T23:53:11.825877+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-24T23:53:11.825877+0000","last_undegraded":"2023-10-24T23:53:11.825877+0000","last_fullsized":"2023-10-24T23:53:11.825877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791237+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-24T23:53:11.791237+0000","last_peered":"2023-10-24T23:53:11.791237+0000","last_clean":"2023-10-24T23:53:11.791237+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-24T23:53:11.791237+0000","last_undegraded":"2023-10-24T23:53:11.791237+0000","last_fullsized":"2023-10-24T23:53:11.791237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794303+0000","last_change":"2023-10-24T23:53:11.429694+0000","last_active":"2023-10-24T23:53:12.794303+0000","last_peered":"2023-10-24T23:53:12.794303+0000","last_clean":"2023-10-24T23:53:12.794303+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-24T23:53:12.794303+0000","last_undegraded":"2023-10-24T23:53:12.794303+0000","last_fullsized":"2023-10-24T23:53:12.794303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:15.964994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793903+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-24T23:53:12.793903+0000","last_peered":"2023-10-24T23:53:12.793903+0000","last_clean":"2023-10-24T23:53:12.793903+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-24T23:53:12.793903+0000","last_undegraded":"2023-10-24T23:53:12.793903+0000","last_fullsized":"2023-10-24T23:53:12.793903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:38:18.678087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"27'8","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:36.922446+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-24T23:53:36.922446+0000","last_peered":"2023-10-24T23:53:36.922446+0000","last_clean":"2023-10-24T23:53:36.922446+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-24T23:53:36.922446+0000","last_undegraded":"2023-10-24T23:53:36.922446+0000","last_fullsized":"2023-10-24T23:53:36.922446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2007,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800342+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-24T23:53:11.800342+0000","last_peered":"2023-10-24T23:53:11.800342+0000","last_clean":"2023-10-24T23:53:11.800342+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-24T23:53:11.800342+0000","last_undegraded":"2023-10-24T23:53:11.800342+0000","last_fullsized":"2023-10-24T23:53:11.800342+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793830+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-24T23:53:12.793830+0000","last_peered":"2023-10-24T23:53:12.793830+0000","last_clean":"2023-10-24T23:53:12.793830+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-24T23:53:12.793830+0000","last_undegraded":"2023-10-24T23:53:12.793830+0000","last_fullsized":"2023-10-24T23:53:12.793830+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:30.163913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797383+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-24T23:53:12.797383+0000","last_peered":"2023-10-24T23:53:12.797383+0000","last_clean":"2023-10-24T23:53:12.797383+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-24T23:53:12.797383+0000","last_undegraded":"2023-10-24T23:53:12.797383+0000","last_fullsized":"2023-10-24T23:53:12.797383+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:46.000330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.802095+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-24T23:53:11.802095+0000","last_peered":"2023-10-24T23:53:11.802095+0000","last_clean":"2023-10-24T23:53:11.802095+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-24T23:53:11.802095+0000","last_undegraded":"2023-10-24T23:53:11.802095+0000","last_fullsized":"2023-10-24T23:53:11.802095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.792050+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:11.792050+0000","last_peered":"2023-10-24T23:53:11.792050+0000","last_clean":"2023-10-24T23:53:11.792050+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:11.792050+0000","last_undegraded":"2023-10-24T23:53:11.792050+0000","last_fullsized":"2023-10-24T23:53:11.792050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788646+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-24T23:53:11.788646+0000","last_peered":"2023-10-24T23:53:11.788646+0000","last_clean":"2023-10-24T23:53:11.788646+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-24T23:53:11.788646+0000","last_undegraded":"2023-10-24T23:53:11.788646+0000","last_fullsized":"2023-10-24T23:53:11.788646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793988+0000","last_change":"2023-10-24T23:53:11.433306+0000","last_active":"2023-10-24T23:53:12.793988+0000","last_peered":"2023-10-24T23:53:12.793988+0000","last_clean":"2023-10-24T23:53:12.793988+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-24T23:53:12.793988+0000","last_undegraded":"2023-10-24T23:53:12.793988+0000","last_fullsized":"2023-10-24T23:53:12.793988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:44:38.728345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604210+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-24T23:53:13.604210+0000","last_peered":"2023-10-24T23:53:13.604210+0000","last_clean":"2023-10-24T23:53:13.604210+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-24T23:53:13.604210+0000","last_undegraded":"2023-10-24T23:53:13.604210+0000","last_fullsized":"2023-10-24T23:53:13.604210+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:30:24.805861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803946+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-24T23:53:11.803946+0000","last_peered":"2023-10-24T23:53:11.803946+0000","last_clean":"2023-10-24T23:53:11.803946+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-24T23:53:11.803946+0000","last_undegraded":"2023-10-24T23:53:11.803946+0000","last_fullsized":"2023-10-24T23:53:11.803946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790715+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:11.790715+0000","last_peered":"2023-10-24T23:53:11.790715+0000","last_clean":"2023-10-24T23:53:11.790715+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:11.790715+0000","last_undegraded":"2023-10-24T23:53:11.790715+0000","last_fullsized":"2023-10-24T23:53:11.790715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807502+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-24T23:53:11.807502+0000","last_peered":"2023-10-24T23:53:11.807502+0000","last_clean":"2023-10-24T23:53:11.807502+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-24T23:53:11.807502+0000","last_undegraded":"2023-10-24T23:53:11.807502+0000","last_fullsized":"2023-10-24T23:53:11.807502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602440+0000","last_change":"2023-10-24T23:53:11.441983+0000","last_active":"2023-10-24T23:53:13.602440+0000","last_peered":"2023-10-24T23:53:13.602440+0000","last_clean":"2023-10-24T23:53:13.602440+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-24T23:53:13.602440+0000","last_undegraded":"2023-10-24T23:53:13.602440+0000","last_fullsized":"2023-10-24T23:53:13.602440+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:30.643019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794864+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-24T23:53:12.794864+0000","last_peered":"2023-10-24T23:53:12.794864+0000","last_clean":"2023-10-24T23:53:12.794864+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-24T23:53:12.794864+0000","last_undegraded":"2023-10-24T23:53:12.794864+0000","last_fullsized":"2023-10-24T23:53:12.794864+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:50:53.467058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788379+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-24T23:53:11.788379+0000","last_peered":"2023-10-24T23:53:11.788379+0000","last_clean":"2023-10-24T23:53:11.788379+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-24T23:53:11.788379+0000","last_undegraded":"2023-10-24T23:53:11.788379+0000","last_fullsized":"2023-10-24T23:53:11.788379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.826147+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:11.826147+0000","last_peered":"2023-10-24T23:53:11.826147+0000","last_clean":"2023-10-24T23:53:11.826147+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:11.826147+0000","last_undegraded":"2023-10-24T23:53:11.826147+0000","last_fullsized":"2023-10-24T23:53:11.826147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790306+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-24T23:53:11.790306+0000","last_peered":"2023-10-24T23:53:11.790306+0000","last_clean":"2023-10-24T23:53:11.790306+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-24T23:53:11.790306+0000","last_undegraded":"2023-10-24T23:53:11.790306+0000","last_fullsized":"2023-10-24T23:53:11.790306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602338+0000","last_change":"2023-10-24T23:53:11.426287+0000","last_active":"2023-10-24T23:53:13.602338+0000","last_peered":"2023-10-24T23:53:13.602338+0000","last_clean":"2023-10-24T23:53:13.602338+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-24T23:53:13.602338+0000","last_undegraded":"2023-10-24T23:53:13.602338+0000","last_fullsized":"2023-10-24T23:53:13.602338+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:02:39.399866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602429+0000","last_change":"2023-10-24T23:53:11.811139+0000","last_active":"2023-10-24T23:53:13.602429+0000","last_peered":"2023-10-24T23:53:13.602429+0000","last_clean":"2023-10-24T23:53:13.602429+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-24T23:53:13.602429+0000","last_undegraded":"2023-10-24T23:53:13.602429+0000","last_fullsized":"2023-10-24T23:53:13.602429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:05.507633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791317+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-24T23:53:11.791317+0000","last_peered":"2023-10-24T23:53:11.791317+0000","last_clean":"2023-10-24T23:53:11.791317+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-24T23:53:11.791317+0000","last_undegraded":"2023-10-24T23:53:11.791317+0000","last_fullsized":"2023-10-24T23:53:11.791317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790574+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:11.790574+0000","last_peered":"2023-10-24T23:53:11.790574+0000","last_clean":"2023-10-24T23:53:11.790574+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:11.790574+0000","last_undegraded":"2023-10-24T23:53:11.790574+0000","last_fullsized":"2023-10-24T23:53:11.790574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832916+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-24T23:53:11.832916+0000","last_peered":"2023-10-24T23:53:11.832916+0000","last_clean":"2023-10-24T23:53:11.832916+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-24T23:53:11.832916+0000","last_undegraded":"2023-10-24T23:53:11.832916+0000","last_fullsized":"2023-10-24T23:53:11.832916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602790+0000","last_change":"2023-10-24T23:53:11.426199+0000","last_active":"2023-10-24T23:53:13.602790+0000","last_peered":"2023-10-24T23:53:13.602790+0000","last_clean":"2023-10-24T23:53:13.602790+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-24T23:53:13.602790+0000","last_undegraded":"2023-10-24T23:53:13.602790+0000","last_fullsized":"2023-10-24T23:53:13.602790+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:57:51.018816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794361+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-24T23:53:12.794361+0000","last_peered":"2023-10-24T23:53:12.794361+0000","last_clean":"2023-10-24T23:53:12.794361+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-24T23:53:12.794361+0000","last_undegraded":"2023-10-24T23:53:12.794361+0000","last_fullsized":"2023-10-24T23:53:12.794361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:20:16.596993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790230+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-24T23:53:11.790230+0000","last_peered":"2023-10-24T23:53:11.790230+0000","last_clean":"2023-10-24T23:53:11.790230+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-24T23:53:11.790230+0000","last_undegraded":"2023-10-24T23:53:11.790230+0000","last_fullsized":"2023-10-24T23:53:11.790230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800496+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-24T23:53:11.800496+0000","last_peered":"2023-10-24T23:53:11.800496+0000","last_clean":"2023-10-24T23:53:11.800496+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-24T23:53:11.800496+0000","last_undegraded":"2023-10-24T23:53:11.800496+0000","last_fullsized":"2023-10-24T23:53:11.800496+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828620+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:53:11.828620+0000","last_peered":"2023-10-24T23:53:11.828620+0000","last_clean":"2023-10-24T23:53:11.828620+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:53:11.828620+0000","last_undegraded":"2023-10-24T23:53:11.828620+0000","last_fullsized":"2023-10-24T23:53:11.828620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603076+0000","last_change":"2023-10-24T23:53:11.432666+0000","last_active":"2023-10-24T23:53:13.603076+0000","last_peered":"2023-10-24T23:53:13.603076+0000","last_clean":"2023-10-24T23:53:13.603076+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-24T23:53:13.603076+0000","last_undegraded":"2023-10-24T23:53:13.603076+0000","last_fullsized":"2023-10-24T23:53:13.603076+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:17.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797676+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-24T23:53:12.797676+0000","last_peered":"2023-10-24T23:53:12.797676+0000","last_clean":"2023-10-24T23:53:12.797676+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-24T23:53:12.797676+0000","last_undegraded":"2023-10-24T23:53:12.797676+0000","last_fullsized":"2023-10-24T23:53:12.797676+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:42.131555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789311+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-24T23:53:11.789311+0000","last_peered":"2023-10-24T23:53:11.789311+0000","last_clean":"2023-10-24T23:53:11.789311+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-24T23:53:11.789311+0000","last_undegraded":"2023-10-24T23:53:11.789311+0000","last_fullsized":"2023-10-24T23:53:11.789311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789825+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-24T23:53:11.789825+0000","last_peered":"2023-10-24T23:53:11.789825+0000","last_clean":"2023-10-24T23:53:11.789825+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-24T23:53:11.789825+0000","last_undegraded":"2023-10-24T23:53:11.789825+0000","last_fullsized":"2023-10-24T23:53:11.789825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.795048+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:53:11.795048+0000","last_peered":"2023-10-24T23:53:11.795048+0000","last_clean":"2023-10-24T23:53:11.795048+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:53:11.795048+0000","last_undegraded":"2023-10-24T23:53:11.795048+0000","last_fullsized":"2023-10-24T23:53:11.795048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"27'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.279161+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-24T23:53:13.279161+0000","last_peered":"2023-10-24T23:53:13.279161+0000","last_clean":"2023-10-24T23:53:13.279161+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-24T23:53:13.279161+0000","last_undegraded":"2023-10-24T23:53:13.279161+0000","last_fullsized":"2023-10-24T23:53:13.279161+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:55:47.548432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795507+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-24T23:53:12.795507+0000","last_peered":"2023-10-24T23:53:12.795507+0000","last_clean":"2023-10-24T23:53:12.795507+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-24T23:53:12.795507+0000","last_undegraded":"2023-10-24T23:53:12.795507+0000","last_fullsized":"2023-10-24T23:53:12.795507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:11:14.431984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788576+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-24T23:53:11.788576+0000","last_peered":"2023-10-24T23:53:11.788576+0000","last_clean":"2023-10-24T23:53:11.788576+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-24T23:53:11.788576+0000","last_undegraded":"2023-10-24T23:53:11.788576+0000","last_fullsized":"2023-10-24T23:53:11.788576+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789662+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-24T23:53:11.789662+0000","last_peered":"2023-10-24T23:53:11.789662+0000","last_clean":"2023-10-24T23:53:11.789662+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-24T23:53:11.789662+0000","last_undegraded":"2023-10-24T23:53:11.789662+0000","last_fullsized":"2023-10-24T23:53:11.789662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828080+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:11.828080+0000","last_peered":"2023-10-24T23:53:11.828080+0000","last_clean":"2023-10-24T23:53:11.828080+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:11.828080+0000","last_undegraded":"2023-10-24T23:53:11.828080+0000","last_fullsized":"2023-10-24T23:53:11.828080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102659+0000","last_change":"2023-10-24T23:53:11.436603+0000","last_active":"2023-10-24T23:53:13.102659+0000","last_peered":"2023-10-24T23:53:13.102659+0000","last_clean":"2023-10-24T23:53:13.102659+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-24T23:53:13.102659+0000","last_undegraded":"2023-10-24T23:53:13.102659+0000","last_fullsized":"2023-10-24T23:53:13.102659+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:09:05.948928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102438+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-24T23:53:13.102438+0000","last_peered":"2023-10-24T23:53:13.102438+0000","last_clean":"2023-10-24T23:53:13.102438+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-24T23:53:13.102438+0000","last_undegraded":"2023-10-24T23:53:13.102438+0000","last_fullsized":"2023-10-24T23:53:13.102438+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:01:59.639080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793650+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-24T23:53:11.793650+0000","last_peered":"2023-10-24T23:53:11.793650+0000","last_clean":"2023-10-24T23:53:11.793650+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-24T23:53:11.793650+0000","last_undegraded":"2023-10-24T23:53:11.793650+0000","last_fullsized":"2023-10-24T23:53:11.793650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807120+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-24T23:53:11.807120+0000","last_peered":"2023-10-24T23:53:11.807120+0000","last_clean":"2023-10-24T23:53:11.807120+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-24T23:53:11.807120+0000","last_undegraded":"2023-10-24T23:53:11.807120+0000","last_fullsized":"2023-10-24T23:53:11.807120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825313+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:53:11.825313+0000","last_peered":"2023-10-24T23:53:11.825313+0000","last_clean":"2023-10-24T23:53:11.825313+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:53:11.825313+0000","last_undegraded":"2023-10-24T23:53:11.825313+0000","last_fullsized":"2023-10-24T23:53:11.825313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.834350+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:53:11.834350+0000","last_peered":"2023-10-24T23:53:11.834350+0000","last_clean":"2023-10-24T23:53:11.834350+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:53:11.834350+0000","last_undegraded":"2023-10-24T23:53:11.834350+0000","last_fullsized":"2023-10-24T23:53:11.834350+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797622+0000","last_change":"2023-10-24T23:53:11.451073+0000","last_active":"2023-10-24T23:53:12.797622+0000","last_peered":"2023-10-24T23:53:12.797622+0000","last_clean":"2023-10-24T23:53:12.797622+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-24T23:53:12.797622+0000","last_undegraded":"2023-10-24T23:53:12.797622+0000","last_fullsized":"2023-10-24T23:53:12.797622+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:32.324710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101638+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-24T23:53:13.101638+0000","last_peered":"2023-10-24T23:53:13.101638+0000","last_clean":"2023-10-24T23:53:13.101638+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-24T23:53:13.101638+0000","last_undegraded":"2023-10-24T23:53:13.101638+0000","last_fullsized":"2023-10-24T23:53:13.101638+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:04.125835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789210+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-24T23:53:11.789210+0000","last_peered":"2023-10-24T23:53:11.789210+0000","last_clean":"2023-10-24T23:53:11.789210+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-24T23:53:11.789210+0000","last_undegraded":"2023-10-24T23:53:11.789210+0000","last_fullsized":"2023-10-24T23:53:11.789210+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804119+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-24T23:53:11.804119+0000","last_peered":"2023-10-24T23:53:11.804119+0000","last_clean":"2023-10-24T23:53:11.804119+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-24T23:53:11.804119+0000","last_undegraded":"2023-10-24T23:53:11.804119+0000","last_fullsized":"2023-10-24T23:53:11.804119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605414+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-24T23:53:13.605414+0000","last_peered":"2023-10-24T23:53:13.605414+0000","last_clean":"2023-10-24T23:53:13.605414+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-24T23:53:13.605414+0000","last_undegraded":"2023-10-24T23:53:13.605414+0000","last_fullsized":"2023-10-24T23:53:13.605414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:06:53.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794699+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-24T23:53:12.794699+0000","last_peered":"2023-10-24T23:53:12.794699+0000","last_clean":"2023-10-24T23:53:12.794699+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-24T23:53:12.794699+0000","last_undegraded":"2023-10-24T23:53:12.794699+0000","last_fullsized":"2023-10-24T23:53:12.794699+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:41.905772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804055+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-24T23:53:11.804055+0000","last_peered":"2023-10-24T23:53:11.804055+0000","last_clean":"2023-10-24T23:53:11.804055+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-24T23:53:11.804055+0000","last_undegraded":"2023-10-24T23:53:11.804055+0000","last_fullsized":"2023-10-24T23:53:11.804055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.802309+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-24T23:53:11.802309+0000","last_peered":"2023-10-24T23:53:11.802309+0000","last_clean":"2023-10-24T23:53:11.802309+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-24T23:53:11.802309+0000","last_undegraded":"2023-10-24T23:53:11.802309+0000","last_fullsized":"2023-10-24T23:53:11.802309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102244+0000","last_change":"2023-10-24T23:53:11.458463+0000","last_active":"2023-10-24T23:53:13.102244+0000","last_peered":"2023-10-24T23:53:13.102244+0000","last_clean":"2023-10-24T23:53:13.102244+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-24T23:53:13.102244+0000","last_undegraded":"2023-10-24T23:53:13.102244+0000","last_fullsized":"2023-10-24T23:53:13.102244+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:11:45.040156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275727+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-24T23:53:13.275727+0000","last_peered":"2023-10-24T23:53:13.275727+0000","last_clean":"2023-10-24T23:53:13.275727+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-24T23:53:13.275727+0000","last_undegraded":"2023-10-24T23:53:13.275727+0000","last_fullsized":"2023-10-24T23:53:13.275727+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:51.707178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800564+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-24T23:53:11.800564+0000","last_peered":"2023-10-24T23:53:11.800564+0000","last_clean":"2023-10-24T23:53:11.800564+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-24T23:53:11.800564+0000","last_undegraded":"2023-10-24T23:53:11.800564+0000","last_fullsized":"2023-10-24T23:53:11.800564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790875+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-24T23:53:11.790875+0000","last_peered":"2023-10-24T23:53:11.790875+0000","last_clean":"2023-10-24T23:53:11.790875+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-24T23:53:11.790875+0000","last_undegraded":"2023-10-24T23:53:11.790875+0000","last_fullsized":"2023-10-24T23:53:11.790875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795610+0000","last_change":"2023-10-24T23:53:11.448675+0000","last_active":"2023-10-24T23:53:12.795610+0000","last_peered":"2023-10-24T23:53:12.795610+0000","last_clean":"2023-10-24T23:53:12.795610+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-24T23:53:12.795610+0000","last_undegraded":"2023-10-24T23:53:12.795610+0000","last_fullsized":"2023-10-24T23:53:12.795610+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:01.502343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605168+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-24T23:53:13.605168+0000","last_peered":"2023-10-24T23:53:13.605168+0000","last_clean":"2023-10-24T23:53:13.605168+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-24T23:53:13.605168+0000","last_undegraded":"2023-10-24T23:53:13.605168+0000","last_fullsized":"2023-10-24T23:53:13.605168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:54.478897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793341+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-24T23:53:11.793341+0000","last_peered":"2023-10-24T23:53:11.793341+0000","last_clean":"2023-10-24T23:53:11.793341+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-24T23:53:11.793341+0000","last_undegraded":"2023-10-24T23:53:11.793341+0000","last_fullsized":"2023-10-24T23:53:11.793341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807904+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-24T23:53:11.807904+0000","last_peered":"2023-10-24T23:53:11.807904+0000","last_clean":"2023-10-24T23:53:11.807904+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-24T23:53:11.807904+0000","last_undegraded":"2023-10-24T23:53:11.807904+0000","last_fullsized":"2023-10-24T23:53:11.807904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275531+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-24T23:53:13.275531+0000","last_peered":"2023-10-24T23:53:13.275531+0000","last_clean":"2023-10-24T23:53:13.275531+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-24T23:53:13.275531+0000","last_undegraded":"2023-10-24T23:53:13.275531+0000","last_fullsized":"2023-10-24T23:53:13.275531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:58.545886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602449+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-24T23:53:13.602449+0000","last_peered":"2023-10-24T23:53:13.602449+0000","last_clean":"2023-10-24T23:53:13.602449+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-24T23:53:13.602449+0000","last_undegraded":"2023-10-24T23:53:13.602449+0000","last_fullsized":"2023-10-24T23:53:13.602449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:05.595887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788253+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-24T23:53:11.788253+0000","last_peered":"2023-10-24T23:53:11.788253+0000","last_clean":"2023-10-24T23:53:11.788253+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-24T23:53:11.788253+0000","last_undegraded":"2023-10-24T23:53:11.788253+0000","last_fullsized":"2023-10-24T23:53:11.788253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.830849+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-24T23:53:11.830849+0000","last_peered":"2023-10-24T23:53:11.830849+0000","last_clean":"2023-10-24T23:53:11.830849+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-24T23:53:11.830849+0000","last_undegraded":"2023-10-24T23:53:11.830849+0000","last_fullsized":"2023-10-24T23:53:11.830849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275166+0000","last_change":"2023-10-24T23:53:11.454658+0000","last_active":"2023-10-24T23:53:13.275166+0000","last_peered":"2023-10-24T23:53:13.275166+0000","last_clean":"2023-10-24T23:53:13.275166+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-24T23:53:13.275166+0000","last_undegraded":"2023-10-24T23:53:13.275166+0000","last_fullsized":"2023-10-24T23:53:13.275166+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:43:43.869069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796376+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-24T23:53:12.796376+0000","last_peered":"2023-10-24T23:53:12.796376+0000","last_clean":"2023-10-24T23:53:12.796376+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-24T23:53:12.796376+0000","last_undegraded":"2023-10-24T23:53:12.796376+0000","last_fullsized":"2023-10-24T23:53:12.796376+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:09:40.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790517+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-24T23:53:11.790517+0000","last_peered":"2023-10-24T23:53:11.790517+0000","last_clean":"2023-10-24T23:53:11.790517+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-24T23:53:11.790517+0000","last_undegraded":"2023-10-24T23:53:11.790517+0000","last_fullsized":"2023-10-24T23:53:11.790517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825937+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-24T23:53:11.825937+0000","last_peered":"2023-10-24T23:53:11.825937+0000","last_clean":"2023-10-24T23:53:11.825937+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-24T23:53:11.825937+0000","last_undegraded":"2023-10-24T23:53:11.825937+0000","last_fullsized":"2023-10-24T23:53:11.825937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275899+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-24T23:53:13.275899+0000","last_peered":"2023-10-24T23:53:13.275899+0000","last_clean":"2023-10-24T23:53:13.275899+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-24T23:53:13.275899+0000","last_undegraded":"2023-10-24T23:53:13.275899+0000","last_fullsized":"2023-10-24T23:53:13.275899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:02.591627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794746+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-24T23:53:12.794746+0000","last_peered":"2023-10-24T23:53:12.794746+0000","last_clean":"2023-10-24T23:53:12.794746+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-24T23:53:12.794746+0000","last_undegraded":"2023-10-24T23:53:12.794746+0000","last_fullsized":"2023-10-24T23:53:12.794746+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:43.015050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.801731+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-24T23:53:11.801731+0000","last_peered":"2023-10-24T23:53:11.801731+0000","last_clean":"2023-10-24T23:53:11.801731+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-24T23:53:11.801731+0000","last_undegraded":"2023-10-24T23:53:11.801731+0000","last_fullsized":"2023-10-24T23:53:11.801731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825206+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-24T23:53:11.825206+0000","last_peered":"2023-10-24T23:53:11.825206+0000","last_clean":"2023-10-24T23:53:11.825206+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-24T23:53:11.825206+0000","last_undegraded":"2023-10-24T23:53:11.825206+0000","last_fullsized":"2023-10-24T23:53:11.825206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.274975+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-24T23:53:13.274975+0000","last_peered":"2023-10-24T23:53:13.274975+0000","last_clean":"2023-10-24T23:53:13.274975+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-24T23:53:13.274975+0000","last_undegraded":"2023-10-24T23:53:13.274975+0000","last_fullsized":"2023-10-24T23:53:13.274975+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:59.068359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.100994+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-24T23:53:13.100994+0000","last_peered":"2023-10-24T23:53:13.100994+0000","last_clean":"2023-10-24T23:53:13.100994+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-24T23:53:13.100994+0000","last_undegraded":"2023-10-24T23:53:13.100994+0000","last_fullsized":"2023-10-24T23:53:13.100994+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:13:03.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806468+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-24T23:53:11.806468+0000","last_peered":"2023-10-24T23:53:11.806468+0000","last_clean":"2023-10-24T23:53:11.806468+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-24T23:53:11.806468+0000","last_undegraded":"2023-10-24T23:53:11.806468+0000","last_fullsized":"2023-10-24T23:53:11.806468+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.827865+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-24T23:53:11.827865+0000","last_peered":"2023-10-24T23:53:11.827865+0000","last_clean":"2023-10-24T23:53:11.827865+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-24T23:53:11.827865+0000","last_undegraded":"2023-10-24T23:53:11.827865+0000","last_fullsized":"2023-10-24T23:53:11.827865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796200+0000","last_change":"2023-10-24T23:53:11.451013+0000","last_active":"2023-10-24T23:53:12.796200+0000","last_peered":"2023-10-24T23:53:12.796200+0000","last_clean":"2023-10-24T23:53:12.796200+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-24T23:53:12.796200+0000","last_undegraded":"2023-10-24T23:53:12.796200+0000","last_fullsized":"2023-10-24T23:53:12.796200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:52:22.503023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797682+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-24T23:53:12.797682+0000","last_peered":"2023-10-24T23:53:12.797682+0000","last_clean":"2023-10-24T23:53:12.797682+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-24T23:53:12.797682+0000","last_undegraded":"2023-10-24T23:53:12.797682+0000","last_fullsized":"2023-10-24T23:53:12.797682+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:05:30.149693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102137+0000","last_change":"2023-10-24T23:53:11.434538+0000","last_active":"2023-10-24T23:53:13.102137+0000","last_peered":"2023-10-24T23:53:13.102137+0000","last_clean":"2023-10-24T23:53:13.102137+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-24T23:53:13.102137+0000","last_undegraded":"2023-10-24T23:53:13.102137+0000","last_fullsized":"2023-10-24T23:53:13.102137+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:55.179253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789405+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-24T23:53:11.789405+0000","last_peered":"2023-10-24T23:53:11.789405+0000","last_clean":"2023-10-24T23:53:11.789405+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-24T23:53:11.789405+0000","last_undegraded":"2023-10-24T23:53:11.789405+0000","last_fullsized":"2023-10-24T23:53:11.789405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807405+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-24T23:53:11.807405+0000","last_peered":"2023-10-24T23:53:11.807405+0000","last_clean":"2023-10-24T23:53:11.807405+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-24T23:53:11.807405+0000","last_undegraded":"2023-10-24T23:53:11.807405+0000","last_fullsized":"2023-10-24T23:53:11.807405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275539+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-24T23:53:13.275539+0000","last_peered":"2023-10-24T23:53:13.275539+0000","last_clean":"2023-10-24T23:53:13.275539+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-24T23:53:13.275539+0000","last_undegraded":"2023-10-24T23:53:13.275539+0000","last_fullsized":"2023-10-24T23:53:13.275539+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:33:08.373963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795605+0000","last_change":"2023-10-24T23:53:11.433387+0000","last_active":"2023-10-24T23:53:12.795605+0000","last_peered":"2023-10-24T23:53:12.795605+0000","last_clean":"2023-10-24T23:53:12.795605+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-24T23:53:12.795605+0000","last_undegraded":"2023-10-24T23:53:12.795605+0000","last_fullsized":"2023-10-24T23:53:12.795605+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:29.148450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.824306+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-24T23:53:11.824306+0000","last_peered":"2023-10-24T23:53:11.824306+0000","last_clean":"2023-10-24T23:53:11.824306+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-24T23:53:11.824306+0000","last_undegraded":"2023-10-24T23:53:11.824306+0000","last_fullsized":"2023-10-24T23:53:11.824306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788623+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-24T23:53:11.788623+0000","last_peered":"2023-10-24T23:53:11.788623+0000","last_clean":"2023-10-24T23:53:11.788623+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-24T23:53:11.788623+0000","last_undegraded":"2023-10-24T23:53:11.788623+0000","last_fullsized":"2023-10-24T23:53:11.788623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605588+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-24T23:53:13.605588+0000","last_peered":"2023-10-24T23:53:13.605588+0000","last_clean":"2023-10-24T23:53:13.605588+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-24T23:53:13.605588+0000","last_undegraded":"2023-10-24T23:53:13.605588+0000","last_fullsized":"2023-10-24T23:53:13.605588+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:33:06.172412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788704+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-24T23:53:11.788704+0000","last_peered":"2023-10-24T23:53:11.788704+0000","last_clean":"2023-10-24T23:53:11.788704+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-24T23:53:11.788704+0000","last_undegraded":"2023-10-24T23:53:11.788704+0000","last_fullsized":"2023-10-24T23:53:11.788704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832501+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-24T23:53:11.832501+0000","last_peered":"2023-10-24T23:53:11.832501+0000","last_clean":"2023-10-24T23:53:11.832501+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-24T23:53:11.832501+0000","last_undegraded":"2023-10-24T23:53:11.832501+0000","last_fullsized":"2023-10-24T23:53:11.832501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"27'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:40.023078+0000","last_change":"2023-10-24T23:53:11.446979+0000","last_active":"2023-10-24T23:53:40.023078+0000","last_peered":"2023-10-24T23:53:40.023078+0000","last_clean":"2023-10-24T23:53:40.023078+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-24T23:53:40.023078+0000","last_undegraded":"2023-10-24T23:53:40.023078+0000","last_fullsized":"2023-10-24T23:53:40.023078+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:54:52.843631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2007,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603260+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-24T23:53:13.603260+0000","last_peered":"2023-10-24T23:53:13.603260+0000","last_clean":"2023-10-24T23:53:13.603260+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-24T23:53:13.603260+0000","last_undegraded":"2023-10-24T23:53:13.603260+0000","last_fullsized":"2023-10-24T23:53:13.603260+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:18:48.679682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.816486+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-24T23:53:11.816486+0000","last_peered":"2023-10-24T23:53:11.816486+0000","last_clean":"2023-10-24T23:53:11.816486+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-24T23:53:11.816486+0000","last_undegraded":"2023-10-24T23:53:11.816486+0000","last_fullsized":"2023-10-24T23:53:11.816486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807790+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-24T23:53:11.807790+0000","last_peered":"2023-10-24T23:53:11.807790+0000","last_clean":"2023-10-24T23:53:11.807790+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-24T23:53:11.807790+0000","last_undegraded":"2023-10-24T23:53:11.807790+0000","last_fullsized":"2023-10-24T23:53:11.807790+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101724+0000","last_change":"2023-10-24T23:53:11.436964+0000","last_active":"2023-10-24T23:53:13.101724+0000","last_peered":"2023-10-24T23:53:13.101724+0000","last_clean":"2023-10-24T23:53:13.101724+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-24T23:53:13.101724+0000","last_undegraded":"2023-10-24T23:53:13.101724+0000","last_fullsized":"2023-10-24T23:53:13.101724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:04:15.366966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602222+0000","last_change":"2023-10-24T23:53:11.814304+0000","last_active":"2023-10-24T23:53:13.602222+0000","last_peered":"2023-10-24T23:53:13.602222+0000","last_clean":"2023-10-24T23:53:13.602222+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-24T23:53:13.602222+0000","last_undegraded":"2023-10-24T23:53:13.602222+0000","last_fullsized":"2023-10-24T23:53:13.602222+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:16.635178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789162+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-24T23:53:11.789162+0000","last_peered":"2023-10-24T23:53:11.789162+0000","last_clean":"2023-10-24T23:53:11.789162+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-24T23:53:11.789162+0000","last_undegraded":"2023-10-24T23:53:11.789162+0000","last_fullsized":"2023-10-24T23:53:11.789162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603105+0000","last_change":"2023-10-24T23:53:11.807096+0000","last_active":"2023-10-24T23:53:13.603105+0000","last_peered":"2023-10-24T23:53:13.603105+0000","last_clean":"2023-10-24T23:53:13.603105+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-24T23:53:13.603105+0000","last_undegraded":"2023-10-24T23:53:13.603105+0000","last_fullsized":"2023-10-24T23:53:13.603105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:25.144578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804445+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-24T23:53:11.804445+0000","last_peered":"2023-10-24T23:53:11.804445+0000","last_clean":"2023-10-24T23:53:11.804445+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-24T23:53:11.804445+0000","last_undegraded":"2023-10-24T23:53:11.804445+0000","last_fullsized":"2023-10-24T23:53:11.804445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793680+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-24T23:53:12.793680+0000","last_peered":"2023-10-24T23:53:12.793680+0000","last_clean":"2023-10-24T23:53:12.793680+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-24T23:53:12.793680+0000","last_undegraded":"2023-10-24T23:53:12.793680+0000","last_fullsized":"2023-10-24T23:53:12.793680+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:38.946585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.811071+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-24T23:53:11.811071+0000","last_peered":"2023-10-24T23:53:11.811071+0000","last_clean":"2023-10-24T23:53:11.811071+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-24T23:53:11.811071+0000","last_undegraded":"2023-10-24T23:53:11.811071+0000","last_fullsized":"2023-10-24T23:53:11.811071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797036+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-24T23:53:12.797036+0000","last_peered":"2023-10-24T23:53:12.797036+0000","last_clean":"2023-10-24T23:53:12.797036+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-24T23:53:12.797036+0000","last_undegraded":"2023-10-24T23:53:12.797036+0000","last_fullsized":"2023-10-24T23:53:12.797036+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:48:24.911279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807673+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-24T23:53:11.807673+0000","last_peered":"2023-10-24T23:53:11.807673+0000","last_clean":"2023-10-24T23:53:11.807673+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-24T23:53:11.807673+0000","last_undegraded":"2023-10-24T23:53:11.807673+0000","last_fullsized":"2023-10-24T23:53:11.807673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795515+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-24T23:53:12.795515+0000","last_peered":"2023-10-24T23:53:12.795515+0000","last_clean":"2023-10-24T23:53:12.795515+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-24T23:53:12.795515+0000","last_undegraded":"2023-10-24T23:53:12.795515+0000","last_fullsized":"2023-10-24T23:53:12.795515+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:21.165755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790400+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-24T23:53:11.790400+0000","last_peered":"2023-10-24T23:53:11.790400+0000","last_clean":"2023-10-24T23:53:11.790400+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-24T23:53:11.790400+0000","last_undegraded":"2023-10-24T23:53:11.790400+0000","last_fullsized":"2023-10-24T23:53:11.790400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604073+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-24T23:53:13.604073+0000","last_peered":"2023-10-24T23:53:13.604073+0000","last_clean":"2023-10-24T23:53:13.604073+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-24T23:53:13.604073+0000","last_undegraded":"2023-10-24T23:53:13.604073+0000","last_fullsized":"2023-10-24T23:53:13.604073+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832305+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-24T23:53:11.832305+0000","last_peered":"2023-10-24T23:53:11.832305+0000","last_clean":"2023-10-24T23:53:11.832305+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-24T23:53:11.832305+0000","last_undegraded":"2023-10-24T23:53:11.832305+0000","last_fullsized":"2023-10-24T23:53:11.832305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101339+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-24T23:53:13.101339+0000","last_peered":"2023-10-24T23:53:13.101339+0000","last_clean":"2023-10-24T23:53:13.101339+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-24T23:53:13.101339+0000","last_undegraded":"2023-10-24T23:53:13.101339+0000","last_fullsized":"2023-10-24T23:53:13.101339+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:54.422365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790575+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-24T23:53:11.790575+0000","last_peered":"2023-10-24T23:53:11.790575+0000","last_clean":"2023-10-24T23:53:11.790575+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-24T23:53:11.790575+0000","last_undegraded":"2023-10-24T23:53:11.790575+0000","last_fullsized":"2023-10-24T23:53:11.790575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603531+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-24T23:53:13.603531+0000","last_peered":"2023-10-24T23:53:13.603531+0000","last_clean":"2023-10-24T23:53:13.603531+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-24T23:53:13.603531+0000","last_undegraded":"2023-10-24T23:53:13.603531+0000","last_fullsized":"2023-10-24T23:53:13.603531+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:01:11.838035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789166+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-24T23:53:11.789166+0000","last_peered":"2023-10-24T23:53:11.789166+0000","last_clean":"2023-10-24T23:53:11.789166+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-24T23:53:11.789166+0000","last_undegraded":"2023-10-24T23:53:11.789166+0000","last_fullsized":"2023-10-24T23:53:11.789166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603142+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-24T23:53:13.603142+0000","last_peered":"2023-10-24T23:53:13.603142+0000","last_clean":"2023-10-24T23:53:13.603142+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-24T23:53:13.603142+0000","last_undegraded":"2023-10-24T23:53:13.603142+0000","last_fullsized":"2023-10-24T23:53:13.603142+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:09:25.424344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.827005+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-24T23:53:11.827005+0000","last_peered":"2023-10-24T23:53:11.827005+0000","last_clean":"2023-10-24T23:53:11.827005+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-24T23:53:11.827005+0000","last_undegraded":"2023-10-24T23:53:11.827005+0000","last_fullsized":"2023-10-24T23:53:11.827005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101152+0000","last_change":"2023-10-24T23:53:11.824274+0000","last_active":"2023-10-24T23:53:13.101152+0000","last_peered":"2023-10-24T23:53:13.101152+0000","last_clean":"2023-10-24T23:53:13.101152+0000","last_became_active":"2023-10-24T23:53:11.823586+0000","last_became_peered":"2023-10-24T23:53:11.823586+0000","last_unstale":"2023-10-24T23:53:13.101152+0000","last_undegraded":"2023-10-24T23:53:13.101152+0000","last_fullsized":"2023-10-24T23:53:13.101152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:47:36.970336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804379+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-24T23:53:11.804379+0000","last_peered":"2023-10-24T23:53:11.804379+0000","last_clean":"2023-10-24T23:53:11.804379+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-24T23:53:11.804379+0000","last_undegraded":"2023-10-24T23:53:11.804379+0000","last_fullsized":"2023-10-24T23:53:11.804379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.274967+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-24T23:53:13.274967+0000","last_peered":"2023-10-24T23:53:13.274967+0000","last_clean":"2023-10-24T23:53:13.274967+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-24T23:53:13.274967+0000","last_undegraded":"2023-10-24T23:53:13.274967+0000","last_fullsized":"2023-10-24T23:53:13.274967+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:10:20.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788652+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-24T23:53:11.788652+0000","last_peered":"2023-10-24T23:53:11.788652+0000","last_clean":"2023-10-24T23:53:11.788652+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-24T23:53:11.788652+0000","last_undegraded":"2023-10-24T23:53:11.788652+0000","last_fullsized":"2023-10-24T23:53:11.788652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796443+0000","last_change":"2023-10-24T23:53:11.860133+0000","last_active":"2023-10-24T23:53:12.796443+0000","last_peered":"2023-10-24T23:53:12.796443+0000","last_clean":"2023-10-24T23:53:12.796443+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-24T23:53:12.796443+0000","last_undegraded":"2023-10-24T23:53:12.796443+0000","last_fullsized":"2023-10-24T23:53:12.796443+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:43.051362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.816202+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-24T23:53:11.816202+0000","last_peered":"2023-10-24T23:53:11.816202+0000","last_clean":"2023-10-24T23:53:11.816202+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-24T23:53:11.816202+0000","last_undegraded":"2023-10-24T23:53:11.816202+0000","last_fullsized":"2023-10-24T23:53:11.816202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603745+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-24T23:53:13.603745+0000","last_peered":"2023-10-24T23:53:13.603745+0000","last_clean":"2023-10-24T23:53:13.603745+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-24T23:53:13.603745+0000","last_undegraded":"2023-10-24T23:53:13.603745+0000","last_fullsized":"2023-10-24T23:53:13.603745+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:39:14.947372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793137+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-24T23:53:11.793137+0000","last_peered":"2023-10-24T23:53:11.793137+0000","last_clean":"2023-10-24T23:53:11.793137+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-24T23:53:11.793137+0000","last_undegraded":"2023-10-24T23:53:11.793137+0000","last_fullsized":"2023-10-24T23:53:11.793137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.798110+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-24T23:53:12.798110+0000","last_peered":"2023-10-24T23:53:12.798110+0000","last_clean":"2023-10-24T23:53:12.798110+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-24T23:53:12.798110+0000","last_undegraded":"2023-10-24T23:53:12.798110+0000","last_fullsized":"2023-10-24T23:53:12.798110+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:37.413259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.796910+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-24T23:53:11.796910+0000","last_peered":"2023-10-24T23:53:11.796910+0000","last_clean":"2023-10-24T23:53:11.796910+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-24T23:53:11.796910+0000","last_undegraded":"2023-10-24T23:53:11.796910+0000","last_fullsized":"2023-10-24T23:53:11.796910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603512+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-24T23:53:13.603512+0000","last_peered":"2023-10-24T23:53:13.603512+0000","last_clean":"2023-10-24T23:53:13.603512+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-24T23:53:13.603512+0000","last_undegraded":"2023-10-24T23:53:13.603512+0000","last_fullsized":"2023-10-24T23:53:13.603512+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:31.344724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803849+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-24T23:53:11.803849+0000","last_peered":"2023-10-24T23:53:11.803849+0000","last_clean":"2023-10-24T23:53:11.803849+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-24T23:53:11.803849+0000","last_undegraded":"2023-10-24T23:53:11.803849+0000","last_fullsized":"2023-10-24T23:53:11.803849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275469+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-24T23:53:13.275469+0000","last_peered":"2023-10-24T23:53:13.275469+0000","last_clean":"2023-10-24T23:53:13.275469+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-24T23:53:13.275469+0000","last_undegraded":"2023-10-24T23:53:13.275469+0000","last_fullsized":"2023-10-24T23:53:13.275469+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788160+0000","last_change":"2023-10-24T23:53:02.728971+0000","last_active":"2023-10-24T23:53:11.788160+0000","last_peered":"2023-10-24T23:53:11.788160+0000","last_clean":"2023-10-24T23:53:11.788160+0000","last_became_active":"2023-10-24T23:53:02.728048+0000","last_became_peered":"2023-10-24T23:53:02.728048+0000","last_unstale":"2023-10-24T23:53:11.788160+0000","last_undegraded":"2023-10-24T23:53:11.788160+0000","last_fullsized":"2023-10-24T23:53:11.788160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:31:32.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.601984+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-24T23:53:13.601984+0000","last_peered":"2023-10-24T23:53:13.601984+0000","last_clean":"2023-10-24T23:53:13.601984+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-24T23:53:13.601984+0000","last_undegraded":"2023-10-24T23:53:13.601984+0000","last_fullsized":"2023-10-24T23:53:13.601984+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:32:05.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791536+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-24T23:53:11.791536+0000","last_peered":"2023-10-24T23:53:11.791536+0000","last_clean":"2023-10-24T23:53:11.791536+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-24T23:53:11.791536+0000","last_undegraded":"2023-10-24T23:53:11.791536+0000","last_fullsized":"2023-10-24T23:53:11.791536+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101460+0000","last_change":"2023-10-24T23:53:11.847814+0000","last_active":"2023-10-24T23:53:13.101460+0000","last_peered":"2023-10-24T23:53:13.101460+0000","last_clean":"2023-10-24T23:53:13.101460+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-24T23:53:13.101460+0000","last_undegraded":"2023-10-24T23:53:13.101460+0000","last_fullsized":"2023-10-24T23:53:13.101460+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:59.075858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825376+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-24T23:53:11.825376+0000","last_peered":"2023-10-24T23:53:11.825376+0000","last_clean":"2023-10-24T23:53:11.825376+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-24T23:53:11.825376+0000","last_undegraded":"2023-10-24T23:53:11.825376+0000","last_fullsized":"2023-10-24T23:53:11.825376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796122+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-24T23:53:12.796122+0000","last_peered":"2023-10-24T23:53:12.796122+0000","last_clean":"2023-10-24T23:53:12.796122+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-24T23:53:12.796122+0000","last_undegraded":"2023-10-24T23:53:12.796122+0000","last_fullsized":"2023-10-24T23:53:12.796122+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:25:12.525273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828503+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-24T23:53:11.828503+0000","last_peered":"2023-10-24T23:53:11.828503+0000","last_clean":"2023-10-24T23:53:11.828503+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-24T23:53:11.828503+0000","last_undegraded":"2023-10-24T23:53:11.828503+0000","last_fullsized":"2023-10-24T23:53:11.828503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101011+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-24T23:53:13.101011+0000","last_peered":"2023-10-24T23:53:13.101011+0000","last_clean":"2023-10-24T23:53:13.101011+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-24T23:53:13.101011+0000","last_undegraded":"2023-10-24T23:53:13.101011+0000","last_fullsized":"2023-10-24T23:53:13.101011+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:12:21.026996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.795075+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-24T23:53:11.795075+0000","last_peered":"2023-10-24T23:53:11.795075+0000","last_clean":"2023-10-24T23:53:11.795075+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-24T23:53:11.795075+0000","last_undegraded":"2023-10-24T23:53:11.795075+0000","last_fullsized":"2023-10-24T23:53:11.795075+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101949+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-24T23:53:13.101949+0000","last_peered":"2023-10-24T23:53:13.101949+0000","last_clean":"2023-10-24T23:53:13.101949+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-24T23:53:13.101949+0000","last_undegraded":"2023-10-24T23:53:13.101949+0000","last_fullsized":"2023-10-24T23:53:13.101949+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:07.909021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.797593+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-24T23:53:11.797593+0000","last_peered":"2023-10-24T23:53:11.797593+0000","last_clean":"2023-10-24T23:53:11.797593+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-24T23:53:11.797593+0000","last_undegraded":"2023-10-24T23:53:11.797593+0000","last_fullsized":"2023-10-24T23:53:11.797593+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603906+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-24T23:53:13.603906+0000","last_peered":"2023-10-24T23:53:13.603906+0000","last_clean":"2023-10-24T23:53:13.603906+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-24T23:53:13.603906+0000","last_undegraded":"2023-10-24T23:53:13.603906+0000","last_fullsized":"2023-10-24T23:53:13.603906+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:48.217889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800508+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-24T23:53:11.800508+0000","last_peered":"2023-10-24T23:53:11.800508+0000","last_clean":"2023-10-24T23:53:11.800508+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-24T23:53:11.800508+0000","last_undegraded":"2023-10-24T23:53:11.800508+0000","last_fullsized":"2023-10-24T23:53:11.800508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602960+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-24T23:53:13.602960+0000","last_peered":"2023-10-24T23:53:13.602960+0000","last_clean":"2023-10-24T23:53:13.602960+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-24T23:53:13.602960+0000","last_undegraded":"2023-10-24T23:53:13.602960+0000","last_fullsized":"2023-10-24T23:53:13.602960+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:10:18.973606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838707+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-24T23:53:11.838707+0000","last_peered":"2023-10-24T23:53:11.838707+0000","last_clean":"2023-10-24T23:53:11.838707+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-24T23:53:11.838707+0000","last_undegraded":"2023-10-24T23:53:11.838707+0000","last_fullsized":"2023-10-24T23:53:11.838707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101457+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-24T23:53:13.101457+0000","last_peered":"2023-10-24T23:53:13.101457+0000","last_clean":"2023-10-24T23:53:13.101457+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-24T23:53:13.101457+0000","last_undegraded":"2023-10-24T23:53:13.101457+0000","last_fullsized":"2023-10-24T23:53:13.101457+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:24:45.536141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788141+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-24T23:53:11.788141+0000","last_peered":"2023-10-24T23:53:11.788141+0000","last_clean":"2023-10-24T23:53:11.788141+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-24T23:53:11.788141+0000","last_undegraded":"2023-10-24T23:53:11.788141+0000","last_fullsized":"2023-10-24T23:53:11.788141+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793724+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-24T23:53:12.793724+0000","last_peered":"2023-10-24T23:53:12.793724+0000","last_clean":"2023-10-24T23:53:12.793724+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-24T23:53:12.793724+0000","last_undegraded":"2023-10-24T23:53:12.793724+0000","last_fullsized":"2023-10-24T23:53:12.793724+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:06.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806250+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-24T23:53:11.806250+0000","last_peered":"2023-10-24T23:53:11.806250+0000","last_clean":"2023-10-24T23:53:11.806250+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-24T23:53:11.806250+0000","last_undegraded":"2023-10-24T23:53:11.806250+0000","last_fullsized":"2023-10-24T23:53:11.806250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275114+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-24T23:53:13.275114+0000","last_peered":"2023-10-24T23:53:13.275114+0000","last_clean":"2023-10-24T23:53:13.275114+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-24T23:53:13.275114+0000","last_undegraded":"2023-10-24T23:53:13.275114+0000","last_fullsized":"2023-10-24T23:53:13.275114+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:56:02.798009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.823970+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-24T23:53:11.823970+0000","last_peered":"2023-10-24T23:53:11.823970+0000","last_clean":"2023-10-24T23:53:11.823970+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-24T23:53:11.823970+0000","last_undegraded":"2023-10-24T23:53:11.823970+0000","last_fullsized":"2023-10-24T23:53:11.823970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793903+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-24T23:53:12.793903+0000","last_peered":"2023-10-24T23:53:12.793903+0000","last_clean":"2023-10-24T23:53:12.793903+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-24T23:53:12.793903+0000","last_undegraded":"2023-10-24T23:53:12.793903+0000","last_fullsized":"2023-10-24T23:53:12.793903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:40.168016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.809345+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-24T23:53:11.809345+0000","last_peered":"2023-10-24T23:53:11.809345+0000","last_clean":"2023-10-24T23:53:11.809345+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-24T23:53:11.809345+0000","last_undegraded":"2023-10-24T23:53:11.809345+0000","last_fullsized":"2023-10-24T23:53:11.809345+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.100799+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-24T23:53:13.100799+0000","last_peered":"2023-10-24T23:53:13.100799+0000","last_clean":"2023-10-24T23:53:13.100799+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-24T23:53:13.100799+0000","last_undegraded":"2023-10-24T23:53:13.100799+0000","last_fullsized":"2023-10-24T23:53:13.100799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:39.720375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788501+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-24T23:53:11.788501+0000","last_peered":"2023-10-24T23:53:11.788501+0000","last_clean":"2023-10-24T23:53:11.788501+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-24T23:53:11.788501+0000","last_undegraded":"2023-10-24T23:53:11.788501+0000","last_fullsized":"2023-10-24T23:53:11.788501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794282+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-24T23:53:12.794282+0000","last_peered":"2023-10-24T23:53:12.794282+0000","last_clean":"2023-10-24T23:53:12.794282+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-24T23:53:12.794282+0000","last_undegraded":"2023-10-24T23:53:12.794282+0000","last_fullsized":"2023-10-24T23:53:12.794282+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:51:40.960630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790034+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-24T23:53:11.790034+0000","last_peered":"2023-10-24T23:53:11.790034+0000","last_clean":"2023-10-24T23:53:11.790034+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-24T23:53:11.790034+0000","last_undegraded":"2023-10-24T23:53:11.790034+0000","last_fullsized":"2023-10-24T23:53:11.790034+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795555+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-24T23:53:12.795555+0000","last_peered":"2023-10-24T23:53:12.795555+0000","last_clean":"2023-10-24T23:53:12.795555+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-24T23:53:12.795555+0000","last_undegraded":"2023-10-24T23:53:12.795555+0000","last_fullsized":"2023-10-24T23:53:12.795555+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:22:42.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793089+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-24T23:53:11.793089+0000","last_peered":"2023-10-24T23:53:11.793089+0000","last_clean":"2023-10-24T23:53:11.793089+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-24T23:53:11.793089+0000","last_undegraded":"2023-10-24T23:53:11.793089+0000","last_fullsized":"2023-10-24T23:53:11.793089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275507+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-24T23:53:13.275507+0000","last_peered":"2023-10-24T23:53:13.275507+0000","last_clean":"2023-10-24T23:53:13.275507+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-24T23:53:13.275507+0000","last_undegraded":"2023-10-24T23:53:13.275507+0000","last_fullsized":"2023-10-24T23:53:13.275507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:47.658717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838499+0000","last_change":"2023-10-24T23:53:02.715966+0000","last_active":"2023-10-24T23:53:11.838499+0000","last_peered":"2023-10-24T23:53:11.838499+0000","last_clean":"2023-10-24T23:53:11.838499+0000","last_became_active":"2023-10-24T23:53:02.715631+0000","last_became_peered":"2023-10-24T23:53:02.715631+0000","last_unstale":"2023-10-24T23:53:11.838499+0000","last_undegraded":"2023-10-24T23:53:11.838499+0000","last_fullsized":"2023-10-24T23:53:11.838499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:11.771864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603292+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-24T23:53:13.603292+0000","last_peered":"2023-10-24T23:53:13.603292+0000","last_clean":"2023-10-24T23:53:13.603292+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-24T23:53:13.603292+0000","last_undegraded":"2023-10-24T23:53:13.603292+0000","last_fullsized":"2023-10-24T23:53:13.603292+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:50.994593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.796890+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-24T23:53:11.796890+0000","last_peered":"2023-10-24T23:53:11.796890+0000","last_clean":"2023-10-24T23:53:11.796890+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-24T23:53:11.796890+0000","last_undegraded":"2023-10-24T23:53:11.796890+0000","last_fullsized":"2023-10-24T23:53:11.796890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2805,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2805,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":929792,"data_stored":713571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":294912,"data_stored":119928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]}]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":307200,"data_stored":120110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":319488,"data_stored":122013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.49}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":121477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4690000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.663}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":892928,"data_stored":710330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:46.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:46.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.913+0000 7f177d25c640 1 Processor -- start 2023-10-24T23:53:46.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.913+0000 7f177d25c640 1 -- start start 2023-10-24T23:53:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f177d25c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f17780acb10 0x7f17780a9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f177d25c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f177806a270 con 0x7f17780ab1d0 2023-10-24T23:53:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f177d25c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f177806a550 con 0x7f177814fd20 2023-10-24T23:53:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f177d25c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f177806aea0 con 0x7f17780acb10 2023-10-24T23:53:46.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f177759e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f17780ab1d0 0x7f17780ab660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45148/0 (socket says 172.21.15.107:45148) 2023-10-24T23:53:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f177759e640 1 -- 172.21.15.107:0/978638542 learned_addr learned my addr 172.21.15.107:0/978638542 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f177659c640 1 --2- 172.21.15.107:0/978638542 >> v2:172.21.15.110:3300/0 conn(0x7f17780acb10 0x7f17780a9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308310893 0 0) 0x7f177806a270 con 0x7f17780ab1d0 2023-10-24T23:53:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1760003530 con 0x7f17780ab1d0 2023-10-24T23:53:46.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 844617464 0 0) 0x7f1760003530 con 0x7f17780ab1d0 2023-10-24T23:53:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 >> v2:172.21.15.110:3300/0 conn(0x7f17780acb10 msgr2=0x7f17780a9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 --2- 172.21.15.107:0/978638542 >> v2:172.21.15.110:3300/0 conn(0x7f17780acb10 0x7f17780a9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 >> v1:172.21.15.110:6789/0 conn(0x7f177814fd20 legacy=0x7f17780a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780acff0 con 0x7f17780ab1d0 2023-10-24T23:53:46.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f1770001a00 con 0x7f17780ab1d0 2023-10-24T23:53:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 >> v1:172.21.15.107:6789/0 conn(0x7f17780ab1d0 legacy=0x7f17780ab660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1760003980 0x7f1760003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17600047d0 0x7f1760004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17600052b0 0x7f176000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177806aea0 con 0x7f1760003980 2023-10-24T23:53:46.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176000c180 con 0x7f17600047d0 2023-10-24T23:53:46.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176000c340 con 0x7f17600052b0 2023-10-24T23:53:46.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1776d9d640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1760003980 0x7f1760003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f177759e640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17600052b0 0x7f176000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f177659c640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17600047d0 0x7f1760004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1776d9d640 1 -- 172.21.15.107:0/978638542 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17600052b0 msgr2=0x7f176000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1776d9d640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17600052b0 0x7f176000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1776d9d640 1 -- 172.21.15.107:0/978638542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17600047d0 msgr2=0x7f1760004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:46.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1776d9d640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17600047d0 0x7f1760004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f1776d9d640 1 -- 172.21.15.107:0/978638542 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1760003530 con 0x7f1760003980 2023-10-24T23:53:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f177759e640 1 -- 172.21.15.107:0/978638542 reap_dead start 2023-10-24T23:53:46.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.917+0000 7f177659c640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17600047d0 0x7f1760004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f1776d9d640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1760003980 0x7f1760003da0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1768001750 tx=0x7f176801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378b286343e55389 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:46.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17680052d0 con 0x7f1760003980 2023-10-24T23:53:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17680a7050 con 0x7f1760003980 2023-10-24T23:53:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f1775d9b640 1 -- 172.21.15.107:0/978638542 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1768003600 con 0x7f1760003980 2023-10-24T23:53:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 -- 172.21.15.107:0/978638542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1760003980 msgr2=0x7f1760003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1760003980 0x7f1760003da0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1768001750 tx=0x7f176801c370 comp rx=0 tx=0).stop 2023-10-24T23:53:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 -- 172.21.15.107:0/978638542 shutdown_connections 2023-10-24T23:53:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 --2- 172.21.15.107:0/978638542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1760003980 0x7f1760003da0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 -- 172.21.15.107:0/978638542 >> 172.21.15.107:0/978638542 conn(0x7f17780c4e40 msgr2=0x7f17780c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 -- 172.21.15.107:0/978638542 shutdown_connections 2023-10-24T23:53:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 -- 172.21.15.107:0/978638542 wait complete. 2023-10-24T23:53:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 Processor -- start 2023-10-24T23:53:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 -- start start 2023-10-24T23:53:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1760003980 0x7f177814d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17780c0db0 0x7f177814a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17780bbf60 0x7f177814abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780bf560 con 0x7f17780bbf60 2023-10-24T23:53:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17781eb2c0 con 0x7f1760003980 2023-10-24T23:53:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17781eb7f0 con 0x7f17780c0db0 2023-10-24T23:53:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17780bbf60 0x7f177814abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17780bbf60 0x7f177814abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37046/0 (socket says 172.21.15.107:37046) 2023-10-24T23:53:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177759e640 1 -- 172.21.15.107:0/3331069910 learned_addr learned my addr 172.21.15.107:0/3331069910 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177659c640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17780c0db0 0x7f177814a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f1776d9d640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1760003980 0x7f177814d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177759e640 1 -- 172.21.15.107:0/3331069910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17780c0db0 msgr2=0x7f177814a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177759e640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17780c0db0 0x7f177814a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177759e640 1 -- 172.21.15.107:0/3331069910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1760003980 msgr2=0x7f177814d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177759e640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1760003980 0x7f177814d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177759e640 1 -- 172.21.15.107:0/3331069910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1778153330 con 0x7f17780bbf60 2023-10-24T23:53:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177759e640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17780bbf60 0x7f177814abf0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1770007c50 tx=0x7f177000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:46.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f175f7fe640 1 -- 172.21.15.107:0/3331069910 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177000deb0 con 0x7f17780bbf60 2023-10-24T23:53:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f175f7fe640 1 -- 172.21.15.107:0/3331069910 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177000bb80 con 0x7f17780bbf60 2023-10-24T23:53:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 -- 172.21.15.107:0/3331069910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17781eb510 con 0x7f17780bbf60 2023-10-24T23:53:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f175f7fe640 1 -- 172.21.15.107:0/3331069910 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1770007110 con 0x7f17780bbf60 2023-10-24T23:53:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.921+0000 7f177d25c640 1 -- 172.21.15.107:0/3331069910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177814dd60 con 0x7f17780bbf60 2023-10-24T23:53:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.925+0000 7f177d25c640 1 -- 172.21.15.107:0/3331069910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17780be6a0 con 0x7f17780bbf60 2023-10-24T23:53:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.925+0000 7f175f7fe640 1 -- 172.21.15.107:0/3331069910 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1770007270 con 0x7f17780bbf60 2023-10-24T23:53:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.925+0000 7f175f7fe640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1764065e60 0x7f1764068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.925+0000 7f175f7fe640 1 -- 172.21.15.107:0/3331069910 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f1770083f60 con 0x7f17780bbf60 2023-10-24T23:53:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.925+0000 7f1776d9d640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1764065e60 0x7f1764068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:46.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.925+0000 7f1776d9d640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1764065e60 0x7f1764068320 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f17600057b0 tx=0x7f17680011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:46.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:46.929+0000 7f175f7fe640 1 -- 172.21.15.107:0/3331069910 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f17700512b0 con 0x7f17780bbf60 2023-10-24T23:53:47.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.069+0000 7f177d25c640 1 -- 172.21.15.107:0/3331069910 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1778150100 con 0x7f1764065e60 2023-10-24T23:53:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.081+0000 7f175f7fe640 1 -- 172.21.15.107:0/3331069910 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525098 (secure 0 0 0) 0x7f17680bb090 con 0x7f1764065e60 2023-10-24T23:53:47.091 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:47.095 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:47.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 -- 172.21.15.107:0/3331069910 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1764065e60 msgr2=0x7f1764068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1764065e60 0x7f1764068320 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f17600057b0 tx=0x7f17680011a0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 -- 172.21.15.107:0/3331069910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17780bbf60 msgr2=0x7f177814abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17780bbf60 0x7f177814abf0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1770007c50 tx=0x7f177000b4a0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 -- 172.21.15.107:0/3331069910 shutdown_connections 2023-10-24T23:53:47.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17780bbf60 0x7f177814abf0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1764065e60 0x7f1764068320 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17780c0db0 0x7f177814a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 --2- 172.21.15.107:0/3331069910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1760003980 0x7f177814d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 -- 172.21.15.107:0/3331069910 >> 172.21.15.107:0/3331069910 conn(0x7f17780c4e40 msgr2=0x7f17780c28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:47.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 -- 172.21.15.107:0/3331069910 shutdown_connections 2023-10-24T23:53:47.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.085+0000 7f177d25c640 1 -- 172.21.15.107:0/3331069910 wait complete. 2023-10-24T23:53:47.108 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2023-10-24T23:53:45.291077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595978,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":253,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235560,"kb_used_data":3560,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754739160,"statfs":{"total":773094113280,"available":772852899840,"internally_reserved":0,"allocated":3645440,"data_stored":2146281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":886,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003731"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603561+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-24T23:53:13.603561+0000","last_peered":"2023-10-24T23:53:13.603561+0000","last_clean":"2023-10-24T23:53:13.603561+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-24T23:53:13.603561+0000","last_undegraded":"2023-10-24T23:53:13.603561+0000","last_fullsized":"2023-10-24T23:53:13.603561+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:51:20.137796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794961+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-24T23:53:12.794961+0000","last_peered":"2023-10-24T23:53:12.794961+0000","last_clean":"2023-10-24T23:53:12.794961+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-24T23:53:12.794961+0000","last_undegraded":"2023-10-24T23:53:12.794961+0000","last_fullsized":"2023-10-24T23:53:12.794961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:00:20.101379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788258+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-24T23:53:11.788258+0000","last_peered":"2023-10-24T23:53:11.788258+0000","last_clean":"2023-10-24T23:53:11.788258+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-24T23:53:11.788258+0000","last_undegraded":"2023-10-24T23:53:11.788258+0000","last_fullsized":"2023-10-24T23:53:11.788258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791737+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-24T23:53:11.791737+0000","last_peered":"2023-10-24T23:53:11.791737+0000","last_clean":"2023-10-24T23:53:11.791737+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-24T23:53:11.791737+0000","last_undegraded":"2023-10-24T23:53:11.791737+0000","last_fullsized":"2023-10-24T23:53:11.791737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794565+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-24T23:53:12.794565+0000","last_peered":"2023-10-24T23:53:12.794565+0000","last_clean":"2023-10-24T23:53:12.794565+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-24T23:53:12.794565+0000","last_undegraded":"2023-10-24T23:53:12.794565+0000","last_fullsized":"2023-10-24T23:53:12.794565+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:59.368156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.601719+0000","last_change":"2023-10-24T23:53:11.444089+0000","last_active":"2023-10-24T23:53:13.601719+0000","last_peered":"2023-10-24T23:53:13.601719+0000","last_clean":"2023-10-24T23:53:13.601719+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-24T23:53:13.601719+0000","last_undegraded":"2023-10-24T23:53:13.601719+0000","last_fullsized":"2023-10-24T23:53:13.601719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:54:06.875882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807520+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-24T23:53:11.807520+0000","last_peered":"2023-10-24T23:53:11.807520+0000","last_clean":"2023-10-24T23:53:11.807520+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-24T23:53:11.807520+0000","last_undegraded":"2023-10-24T23:53:11.807520+0000","last_fullsized":"2023-10-24T23:53:11.807520+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806881+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-24T23:53:11.806881+0000","last_peered":"2023-10-24T23:53:11.806881+0000","last_clean":"2023-10-24T23:53:11.806881+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-24T23:53:11.806881+0000","last_undegraded":"2023-10-24T23:53:11.806881+0000","last_fullsized":"2023-10-24T23:53:11.806881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602008+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-24T23:53:13.602008+0000","last_peered":"2023-10-24T23:53:13.602008+0000","last_clean":"2023-10-24T23:53:13.602008+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-24T23:53:13.602008+0000","last_undegraded":"2023-10-24T23:53:13.602008+0000","last_fullsized":"2023-10-24T23:53:13.602008+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:38:56.643025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101201+0000","last_change":"2023-10-24T23:53:11.438681+0000","last_active":"2023-10-24T23:53:13.101201+0000","last_peered":"2023-10-24T23:53:13.101201+0000","last_clean":"2023-10-24T23:53:13.101201+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-24T23:53:13.101201+0000","last_undegraded":"2023-10-24T23:53:13.101201+0000","last_fullsized":"2023-10-24T23:53:13.101201+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:50.139697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832718+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-24T23:53:11.832718+0000","last_peered":"2023-10-24T23:53:11.832718+0000","last_clean":"2023-10-24T23:53:11.832718+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-24T23:53:11.832718+0000","last_undegraded":"2023-10-24T23:53:11.832718+0000","last_fullsized":"2023-10-24T23:53:11.832718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.810798+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-24T23:53:11.810798+0000","last_peered":"2023-10-24T23:53:11.810798+0000","last_clean":"2023-10-24T23:53:11.810798+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-24T23:53:11.810798+0000","last_undegraded":"2023-10-24T23:53:11.810798+0000","last_fullsized":"2023-10-24T23:53:11.810798+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101349+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-24T23:53:13.101349+0000","last_peered":"2023-10-24T23:53:13.101349+0000","last_clean":"2023-10-24T23:53:13.101349+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-24T23:53:13.101349+0000","last_undegraded":"2023-10-24T23:53:13.101349+0000","last_fullsized":"2023-10-24T23:53:13.101349+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:22.974333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796171+0000","last_change":"2023-10-24T23:53:11.454537+0000","last_active":"2023-10-24T23:53:12.796171+0000","last_peered":"2023-10-24T23:53:12.796171+0000","last_clean":"2023-10-24T23:53:12.796171+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-24T23:53:12.796171+0000","last_undegraded":"2023-10-24T23:53:12.796171+0000","last_fullsized":"2023-10-24T23:53:12.796171+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:14:06.792651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.830575+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-24T23:53:11.830575+0000","last_peered":"2023-10-24T23:53:11.830575+0000","last_clean":"2023-10-24T23:53:11.830575+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-24T23:53:11.830575+0000","last_undegraded":"2023-10-24T23:53:11.830575+0000","last_fullsized":"2023-10-24T23:53:11.830575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800231+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-24T23:53:11.800231+0000","last_peered":"2023-10-24T23:53:11.800231+0000","last_clean":"2023-10-24T23:53:11.800231+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-24T23:53:11.800231+0000","last_undegraded":"2023-10-24T23:53:11.800231+0000","last_fullsized":"2023-10-24T23:53:11.800231+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605269+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-24T23:53:13.605269+0000","last_peered":"2023-10-24T23:53:13.605269+0000","last_clean":"2023-10-24T23:53:13.605269+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-24T23:53:13.605269+0000","last_undegraded":"2023-10-24T23:53:13.605269+0000","last_fullsized":"2023-10-24T23:53:13.605269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:35:39.879563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795373+0000","last_change":"2023-10-24T23:53:11.444138+0000","last_active":"2023-10-24T23:53:12.795373+0000","last_peered":"2023-10-24T23:53:12.795373+0000","last_clean":"2023-10-24T23:53:12.795373+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-24T23:53:12.795373+0000","last_undegraded":"2023-10-24T23:53:12.795373+0000","last_fullsized":"2023-10-24T23:53:12.795373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:27:38.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788658+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-24T23:53:11.788658+0000","last_peered":"2023-10-24T23:53:11.788658+0000","last_clean":"2023-10-24T23:53:11.788658+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-24T23:53:11.788658+0000","last_undegraded":"2023-10-24T23:53:11.788658+0000","last_fullsized":"2023-10-24T23:53:11.788658+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.797645+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-24T23:53:11.797645+0000","last_peered":"2023-10-24T23:53:11.797645+0000","last_clean":"2023-10-24T23:53:11.797645+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-24T23:53:11.797645+0000","last_undegraded":"2023-10-24T23:53:11.797645+0000","last_fullsized":"2023-10-24T23:53:11.797645+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795697+0000","last_change":"2023-10-24T23:53:11.832809+0000","last_active":"2023-10-24T23:53:12.795697+0000","last_peered":"2023-10-24T23:53:12.795697+0000","last_clean":"2023-10-24T23:53:12.795697+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-24T23:53:12.795697+0000","last_undegraded":"2023-10-24T23:53:12.795697+0000","last_fullsized":"2023-10-24T23:53:12.795697+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:01.151315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604963+0000","last_change":"2023-10-24T23:53:11.442035+0000","last_active":"2023-10-24T23:53:13.604963+0000","last_peered":"2023-10-24T23:53:13.604963+0000","last_clean":"2023-10-24T23:53:13.604963+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-24T23:53:13.604963+0000","last_undegraded":"2023-10-24T23:53:13.604963+0000","last_fullsized":"2023-10-24T23:53:13.604963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:13.717844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.811329+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-24T23:53:11.811329+0000","last_peered":"2023-10-24T23:53:11.811329+0000","last_clean":"2023-10-24T23:53:11.811329+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-24T23:53:11.811329+0000","last_undegraded":"2023-10-24T23:53:11.811329+0000","last_fullsized":"2023-10-24T23:53:11.811329+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807632+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-24T23:53:11.807632+0000","last_peered":"2023-10-24T23:53:11.807632+0000","last_clean":"2023-10-24T23:53:11.807632+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-24T23:53:11.807632+0000","last_undegraded":"2023-10-24T23:53:11.807632+0000","last_fullsized":"2023-10-24T23:53:11.807632+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797540+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-24T23:53:12.797540+0000","last_peered":"2023-10-24T23:53:12.797540+0000","last_clean":"2023-10-24T23:53:12.797540+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-24T23:53:12.797540+0000","last_undegraded":"2023-10-24T23:53:12.797540+0000","last_fullsized":"2023-10-24T23:53:12.797540+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:26.012711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604200+0000","last_change":"2023-10-24T23:53:11.448032+0000","last_active":"2023-10-24T23:53:13.604200+0000","last_peered":"2023-10-24T23:53:13.604200+0000","last_clean":"2023-10-24T23:53:13.604200+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-24T23:53:13.604200+0000","last_undegraded":"2023-10-24T23:53:13.604200+0000","last_fullsized":"2023-10-24T23:53:13.604200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:50:32.490810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791720+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-24T23:53:11.791720+0000","last_peered":"2023-10-24T23:53:11.791720+0000","last_clean":"2023-10-24T23:53:11.791720+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-24T23:53:11.791720+0000","last_undegraded":"2023-10-24T23:53:11.791720+0000","last_fullsized":"2023-10-24T23:53:11.791720+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788475+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-24T23:53:11.788475+0000","last_peered":"2023-10-24T23:53:11.788475+0000","last_clean":"2023-10-24T23:53:11.788475+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-24T23:53:11.788475+0000","last_undegraded":"2023-10-24T23:53:11.788475+0000","last_fullsized":"2023-10-24T23:53:11.788475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795316+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-24T23:53:12.795316+0000","last_peered":"2023-10-24T23:53:12.795316+0000","last_clean":"2023-10-24T23:53:12.795316+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-24T23:53:12.795316+0000","last_undegraded":"2023-10-24T23:53:12.795316+0000","last_fullsized":"2023-10-24T23:53:12.795316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:10.820967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603323+0000","last_change":"2023-10-24T23:53:11.446925+0000","last_active":"2023-10-24T23:53:13.603323+0000","last_peered":"2023-10-24T23:53:13.603323+0000","last_clean":"2023-10-24T23:53:13.603323+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-24T23:53:13.603323+0000","last_undegraded":"2023-10-24T23:53:13.603323+0000","last_fullsized":"2023-10-24T23:53:13.603323+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:59:52.485469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804040+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-24T23:53:11.804040+0000","last_peered":"2023-10-24T23:53:11.804040+0000","last_clean":"2023-10-24T23:53:11.804040+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-24T23:53:11.804040+0000","last_undegraded":"2023-10-24T23:53:11.804040+0000","last_fullsized":"2023-10-24T23:53:11.804040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791850+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-24T23:53:11.791850+0000","last_peered":"2023-10-24T23:53:11.791850+0000","last_clean":"2023-10-24T23:53:11.791850+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-24T23:53:11.791850+0000","last_undegraded":"2023-10-24T23:53:11.791850+0000","last_fullsized":"2023-10-24T23:53:11.791850+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795971+0000","last_change":"2023-10-24T23:53:11.849056+0000","last_active":"2023-10-24T23:53:12.795971+0000","last_peered":"2023-10-24T23:53:12.795971+0000","last_clean":"2023-10-24T23:53:12.795971+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-24T23:53:12.795971+0000","last_undegraded":"2023-10-24T23:53:12.795971+0000","last_fullsized":"2023-10-24T23:53:12.795971+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:08.932942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803866+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-24T23:53:11.803866+0000","last_peered":"2023-10-24T23:53:11.803866+0000","last_clean":"2023-10-24T23:53:11.803866+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-24T23:53:11.803866+0000","last_undegraded":"2023-10-24T23:53:11.803866+0000","last_fullsized":"2023-10-24T23:53:11.803866+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788258+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-24T23:53:11.788258+0000","last_peered":"2023-10-24T23:53:11.788258+0000","last_clean":"2023-10-24T23:53:11.788258+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-24T23:53:11.788258+0000","last_undegraded":"2023-10-24T23:53:11.788258+0000","last_fullsized":"2023-10-24T23:53:11.788258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602180+0000","last_change":"2023-10-24T23:53:11.447167+0000","last_active":"2023-10-24T23:53:13.602180+0000","last_peered":"2023-10-24T23:53:13.602180+0000","last_clean":"2023-10-24T23:53:13.602180+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-24T23:53:13.602180+0000","last_undegraded":"2023-10-24T23:53:13.602180+0000","last_fullsized":"2023-10-24T23:53:13.602180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:06:25.006728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794920+0000","last_change":"2023-10-24T23:53:11.813704+0000","last_active":"2023-10-24T23:53:12.794920+0000","last_peered":"2023-10-24T23:53:12.794920+0000","last_clean":"2023-10-24T23:53:12.794920+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-24T23:53:12.794920+0000","last_undegraded":"2023-10-24T23:53:12.794920+0000","last_fullsized":"2023-10-24T23:53:12.794920+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:39:09.996039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804467+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-24T23:53:11.804467+0000","last_peered":"2023-10-24T23:53:11.804467+0000","last_clean":"2023-10-24T23:53:11.804467+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-24T23:53:11.804467+0000","last_undegraded":"2023-10-24T23:53:11.804467+0000","last_fullsized":"2023-10-24T23:53:11.804467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838755+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-24T23:53:11.838755+0000","last_peered":"2023-10-24T23:53:11.838755+0000","last_clean":"2023-10-24T23:53:11.838755+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-24T23:53:11.838755+0000","last_undegraded":"2023-10-24T23:53:11.838755+0000","last_fullsized":"2023-10-24T23:53:11.838755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795718+0000","last_change":"2023-10-24T23:53:11.450974+0000","last_active":"2023-10-24T23:53:12.795718+0000","last_peered":"2023-10-24T23:53:12.795718+0000","last_clean":"2023-10-24T23:53:12.795718+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-24T23:53:12.795718+0000","last_undegraded":"2023-10-24T23:53:12.795718+0000","last_fullsized":"2023-10-24T23:53:12.795718+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:51:05.443664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602086+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-24T23:53:13.602086+0000","last_peered":"2023-10-24T23:53:13.602086+0000","last_clean":"2023-10-24T23:53:13.602086+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-24T23:53:13.602086+0000","last_undegraded":"2023-10-24T23:53:13.602086+0000","last_fullsized":"2023-10-24T23:53:13.602086+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:37:30.278659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825877+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-24T23:53:11.825877+0000","last_peered":"2023-10-24T23:53:11.825877+0000","last_clean":"2023-10-24T23:53:11.825877+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-24T23:53:11.825877+0000","last_undegraded":"2023-10-24T23:53:11.825877+0000","last_fullsized":"2023-10-24T23:53:11.825877+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791237+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-24T23:53:11.791237+0000","last_peered":"2023-10-24T23:53:11.791237+0000","last_clean":"2023-10-24T23:53:11.791237+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-24T23:53:11.791237+0000","last_undegraded":"2023-10-24T23:53:11.791237+0000","last_fullsized":"2023-10-24T23:53:11.791237+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794303+0000","last_change":"2023-10-24T23:53:11.429694+0000","last_active":"2023-10-24T23:53:12.794303+0000","last_peered":"2023-10-24T23:53:12.794303+0000","last_clean":"2023-10-24T23:53:12.794303+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-24T23:53:12.794303+0000","last_undegraded":"2023-10-24T23:53:12.794303+0000","last_fullsized":"2023-10-24T23:53:12.794303+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:15.964994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793903+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-24T23:53:12.793903+0000","last_peered":"2023-10-24T23:53:12.793903+0000","last_clean":"2023-10-24T23:53:12.793903+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-24T23:53:12.793903+0000","last_undegraded":"2023-10-24T23:53:12.793903+0000","last_fullsized":"2023-10-24T23:53:12.793903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:38:18.678087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"27'8","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:36.922446+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-24T23:53:36.922446+0000","last_peered":"2023-10-24T23:53:36.922446+0000","last_clean":"2023-10-24T23:53:36.922446+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-24T23:53:36.922446+0000","last_undegraded":"2023-10-24T23:53:36.922446+0000","last_fullsized":"2023-10-24T23:53:36.922446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2007,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800342+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-24T23:53:11.800342+0000","last_peered":"2023-10-24T23:53:11.800342+0000","last_clean":"2023-10-24T23:53:11.800342+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-24T23:53:11.800342+0000","last_undegraded":"2023-10-24T23:53:11.800342+0000","last_fullsized":"2023-10-24T23:53:11.800342+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793830+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-24T23:53:12.793830+0000","last_peered":"2023-10-24T23:53:12.793830+0000","last_clean":"2023-10-24T23:53:12.793830+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-24T23:53:12.793830+0000","last_undegraded":"2023-10-24T23:53:12.793830+0000","last_fullsized":"2023-10-24T23:53:12.793830+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:30.163913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797383+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-24T23:53:12.797383+0000","last_peered":"2023-10-24T23:53:12.797383+0000","last_clean":"2023-10-24T23:53:12.797383+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-24T23:53:12.797383+0000","last_undegraded":"2023-10-24T23:53:12.797383+0000","last_fullsized":"2023-10-24T23:53:12.797383+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:46.000330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.802095+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-24T23:53:11.802095+0000","last_peered":"2023-10-24T23:53:11.802095+0000","last_clean":"2023-10-24T23:53:11.802095+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-24T23:53:11.802095+0000","last_undegraded":"2023-10-24T23:53:11.802095+0000","last_fullsized":"2023-10-24T23:53:11.802095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.792050+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:11.792050+0000","last_peered":"2023-10-24T23:53:11.792050+0000","last_clean":"2023-10-24T23:53:11.792050+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:11.792050+0000","last_undegraded":"2023-10-24T23:53:11.792050+0000","last_fullsized":"2023-10-24T23:53:11.792050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788646+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-24T23:53:11.788646+0000","last_peered":"2023-10-24T23:53:11.788646+0000","last_clean":"2023-10-24T23:53:11.788646+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-24T23:53:11.788646+0000","last_undegraded":"2023-10-24T23:53:11.788646+0000","last_fullsized":"2023-10-24T23:53:11.788646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793988+0000","last_change":"2023-10-24T23:53:11.433306+0000","last_active":"2023-10-24T23:53:12.793988+0000","last_peered":"2023-10-24T23:53:12.793988+0000","last_clean":"2023-10-24T23:53:12.793988+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-24T23:53:12.793988+0000","last_undegraded":"2023-10-24T23:53:12.793988+0000","last_fullsized":"2023-10-24T23:53:12.793988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:44:38.728345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604210+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-24T23:53:13.604210+0000","last_peered":"2023-10-24T23:53:13.604210+0000","last_clean":"2023-10-24T23:53:13.604210+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-24T23:53:13.604210+0000","last_undegraded":"2023-10-24T23:53:13.604210+0000","last_fullsized":"2023-10-24T23:53:13.604210+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:30:24.805861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803946+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-24T23:53:11.803946+0000","last_peered":"2023-10-24T23:53:11.803946+0000","last_clean":"2023-10-24T23:53:11.803946+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-24T23:53:11.803946+0000","last_undegraded":"2023-10-24T23:53:11.803946+0000","last_fullsized":"2023-10-24T23:53:11.803946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790715+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:11.790715+0000","last_peered":"2023-10-24T23:53:11.790715+0000","last_clean":"2023-10-24T23:53:11.790715+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:11.790715+0000","last_undegraded":"2023-10-24T23:53:11.790715+0000","last_fullsized":"2023-10-24T23:53:11.790715+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807502+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-24T23:53:11.807502+0000","last_peered":"2023-10-24T23:53:11.807502+0000","last_clean":"2023-10-24T23:53:11.807502+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-24T23:53:11.807502+0000","last_undegraded":"2023-10-24T23:53:11.807502+0000","last_fullsized":"2023-10-24T23:53:11.807502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602440+0000","last_change":"2023-10-24T23:53:11.441983+0000","last_active":"2023-10-24T23:53:13.602440+0000","last_peered":"2023-10-24T23:53:13.602440+0000","last_clean":"2023-10-24T23:53:13.602440+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-24T23:53:13.602440+0000","last_undegraded":"2023-10-24T23:53:13.602440+0000","last_fullsized":"2023-10-24T23:53:13.602440+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:30.643019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794864+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-24T23:53:12.794864+0000","last_peered":"2023-10-24T23:53:12.794864+0000","last_clean":"2023-10-24T23:53:12.794864+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-24T23:53:12.794864+0000","last_undegraded":"2023-10-24T23:53:12.794864+0000","last_fullsized":"2023-10-24T23:53:12.794864+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:50:53.467058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788379+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-24T23:53:11.788379+0000","last_peered":"2023-10-24T23:53:11.788379+0000","last_clean":"2023-10-24T23:53:11.788379+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-24T23:53:11.788379+0000","last_undegraded":"2023-10-24T23:53:11.788379+0000","last_fullsized":"2023-10-24T23:53:11.788379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.826147+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:11.826147+0000","last_peered":"2023-10-24T23:53:11.826147+0000","last_clean":"2023-10-24T23:53:11.826147+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:11.826147+0000","last_undegraded":"2023-10-24T23:53:11.826147+0000","last_fullsized":"2023-10-24T23:53:11.826147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790306+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-24T23:53:11.790306+0000","last_peered":"2023-10-24T23:53:11.790306+0000","last_clean":"2023-10-24T23:53:11.790306+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-24T23:53:11.790306+0000","last_undegraded":"2023-10-24T23:53:11.790306+0000","last_fullsized":"2023-10-24T23:53:11.790306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602338+0000","last_change":"2023-10-24T23:53:11.426287+0000","last_active":"2023-10-24T23:53:13.602338+0000","last_peered":"2023-10-24T23:53:13.602338+0000","last_clean":"2023-10-24T23:53:13.602338+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-24T23:53:13.602338+0000","last_undegraded":"2023-10-24T23:53:13.602338+0000","last_fullsized":"2023-10-24T23:53:13.602338+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:02:39.399866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602429+0000","last_change":"2023-10-24T23:53:11.811139+0000","last_active":"2023-10-24T23:53:13.602429+0000","last_peered":"2023-10-24T23:53:13.602429+0000","last_clean":"2023-10-24T23:53:13.602429+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-24T23:53:13.602429+0000","last_undegraded":"2023-10-24T23:53:13.602429+0000","last_fullsized":"2023-10-24T23:53:13.602429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:05.507633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791317+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-24T23:53:11.791317+0000","last_peered":"2023-10-24T23:53:11.791317+0000","last_clean":"2023-10-24T23:53:11.791317+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-24T23:53:11.791317+0000","last_undegraded":"2023-10-24T23:53:11.791317+0000","last_fullsized":"2023-10-24T23:53:11.791317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790574+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:11.790574+0000","last_peered":"2023-10-24T23:53:11.790574+0000","last_clean":"2023-10-24T23:53:11.790574+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:11.790574+0000","last_undegraded":"2023-10-24T23:53:11.790574+0000","last_fullsized":"2023-10-24T23:53:11.790574+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832916+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-24T23:53:11.832916+0000","last_peered":"2023-10-24T23:53:11.832916+0000","last_clean":"2023-10-24T23:53:11.832916+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-24T23:53:11.832916+0000","last_undegraded":"2023-10-24T23:53:11.832916+0000","last_fullsized":"2023-10-24T23:53:11.832916+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602790+0000","last_change":"2023-10-24T23:53:11.426199+0000","last_active":"2023-10-24T23:53:13.602790+0000","last_peered":"2023-10-24T23:53:13.602790+0000","last_clean":"2023-10-24T23:53:13.602790+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-24T23:53:13.602790+0000","last_undegraded":"2023-10-24T23:53:13.602790+0000","last_fullsized":"2023-10-24T23:53:13.602790+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:57:51.018816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794361+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-24T23:53:12.794361+0000","last_peered":"2023-10-24T23:53:12.794361+0000","last_clean":"2023-10-24T23:53:12.794361+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-24T23:53:12.794361+0000","last_undegraded":"2023-10-24T23:53:12.794361+0000","last_fullsized":"2023-10-24T23:53:12.794361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:20:16.596993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790230+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-24T23:53:11.790230+0000","last_peered":"2023-10-24T23:53:11.790230+0000","last_clean":"2023-10-24T23:53:11.790230+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-24T23:53:11.790230+0000","last_undegraded":"2023-10-24T23:53:11.790230+0000","last_fullsized":"2023-10-24T23:53:11.790230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800496+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-24T23:53:11.800496+0000","last_peered":"2023-10-24T23:53:11.800496+0000","last_clean":"2023-10-24T23:53:11.800496+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-24T23:53:11.800496+0000","last_undegraded":"2023-10-24T23:53:11.800496+0000","last_fullsized":"2023-10-24T23:53:11.800496+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828620+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:53:11.828620+0000","last_peered":"2023-10-24T23:53:11.828620+0000","last_clean":"2023-10-24T23:53:11.828620+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:53:11.828620+0000","last_undegraded":"2023-10-24T23:53:11.828620+0000","last_fullsized":"2023-10-24T23:53:11.828620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603076+0000","last_change":"2023-10-24T23:53:11.432666+0000","last_active":"2023-10-24T23:53:13.603076+0000","last_peered":"2023-10-24T23:53:13.603076+0000","last_clean":"2023-10-24T23:53:13.603076+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-24T23:53:13.603076+0000","last_undegraded":"2023-10-24T23:53:13.603076+0000","last_fullsized":"2023-10-24T23:53:13.603076+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:17.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797676+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-24T23:53:12.797676+0000","last_peered":"2023-10-24T23:53:12.797676+0000","last_clean":"2023-10-24T23:53:12.797676+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-24T23:53:12.797676+0000","last_undegraded":"2023-10-24T23:53:12.797676+0000","last_fullsized":"2023-10-24T23:53:12.797676+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:42.131555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789311+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-24T23:53:11.789311+0000","last_peered":"2023-10-24T23:53:11.789311+0000","last_clean":"2023-10-24T23:53:11.789311+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-24T23:53:11.789311+0000","last_undegraded":"2023-10-24T23:53:11.789311+0000","last_fullsized":"2023-10-24T23:53:11.789311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789825+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-24T23:53:11.789825+0000","last_peered":"2023-10-24T23:53:11.789825+0000","last_clean":"2023-10-24T23:53:11.789825+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-24T23:53:11.789825+0000","last_undegraded":"2023-10-24T23:53:11.789825+0000","last_fullsized":"2023-10-24T23:53:11.789825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.795048+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:53:11.795048+0000","last_peered":"2023-10-24T23:53:11.795048+0000","last_clean":"2023-10-24T23:53:11.795048+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:53:11.795048+0000","last_undegraded":"2023-10-24T23:53:11.795048+0000","last_fullsized":"2023-10-24T23:53:11.795048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"27'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.279161+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-24T23:53:13.279161+0000","last_peered":"2023-10-24T23:53:13.279161+0000","last_clean":"2023-10-24T23:53:13.279161+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-24T23:53:13.279161+0000","last_undegraded":"2023-10-24T23:53:13.279161+0000","last_fullsized":"2023-10-24T23:53:13.279161+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:55:47.548432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795507+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-24T23:53:12.795507+0000","last_peered":"2023-10-24T23:53:12.795507+0000","last_clean":"2023-10-24T23:53:12.795507+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-24T23:53:12.795507+0000","last_undegraded":"2023-10-24T23:53:12.795507+0000","last_fullsized":"2023-10-24T23:53:12.795507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:11:14.431984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788576+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-24T23:53:11.788576+0000","last_peered":"2023-10-24T23:53:11.788576+0000","last_clean":"2023-10-24T23:53:11.788576+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-24T23:53:11.788576+0000","last_undegraded":"2023-10-24T23:53:11.788576+0000","last_fullsized":"2023-10-24T23:53:11.788576+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789662+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-24T23:53:11.789662+0000","last_peered":"2023-10-24T23:53:11.789662+0000","last_clean":"2023-10-24T23:53:11.789662+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-24T23:53:11.789662+0000","last_undegraded":"2023-10-24T23:53:11.789662+0000","last_fullsized":"2023-10-24T23:53:11.789662+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828080+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:11.828080+0000","last_peered":"2023-10-24T23:53:11.828080+0000","last_clean":"2023-10-24T23:53:11.828080+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:11.828080+0000","last_undegraded":"2023-10-24T23:53:11.828080+0000","last_fullsized":"2023-10-24T23:53:11.828080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102659+0000","last_change":"2023-10-24T23:53:11.436603+0000","last_active":"2023-10-24T23:53:13.102659+0000","last_peered":"2023-10-24T23:53:13.102659+0000","last_clean":"2023-10-24T23:53:13.102659+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-24T23:53:13.102659+0000","last_undegraded":"2023-10-24T23:53:13.102659+0000","last_fullsized":"2023-10-24T23:53:13.102659+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:09:05.948928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102438+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-24T23:53:13.102438+0000","last_peered":"2023-10-24T23:53:13.102438+0000","last_clean":"2023-10-24T23:53:13.102438+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-24T23:53:13.102438+0000","last_undegraded":"2023-10-24T23:53:13.102438+0000","last_fullsized":"2023-10-24T23:53:13.102438+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:01:59.639080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793650+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-24T23:53:11.793650+0000","last_peered":"2023-10-24T23:53:11.793650+0000","last_clean":"2023-10-24T23:53:11.793650+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-24T23:53:11.793650+0000","last_undegraded":"2023-10-24T23:53:11.793650+0000","last_fullsized":"2023-10-24T23:53:11.793650+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807120+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-24T23:53:11.807120+0000","last_peered":"2023-10-24T23:53:11.807120+0000","last_clean":"2023-10-24T23:53:11.807120+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-24T23:53:11.807120+0000","last_undegraded":"2023-10-24T23:53:11.807120+0000","last_fullsized":"2023-10-24T23:53:11.807120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825313+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:53:11.825313+0000","last_peered":"2023-10-24T23:53:11.825313+0000","last_clean":"2023-10-24T23:53:11.825313+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:53:11.825313+0000","last_undegraded":"2023-10-24T23:53:11.825313+0000","last_fullsized":"2023-10-24T23:53:11.825313+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":171,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.834350+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:53:11.834350+0000","last_peered":"2023-10-24T23:53:11.834350+0000","last_clean":"2023-10-24T23:53:11.834350+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:53:11.834350+0000","last_undegraded":"2023-10-24T23:53:11.834350+0000","last_fullsized":"2023-10-24T23:53:11.834350+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797622+0000","last_change":"2023-10-24T23:53:11.451073+0000","last_active":"2023-10-24T23:53:12.797622+0000","last_peered":"2023-10-24T23:53:12.797622+0000","last_clean":"2023-10-24T23:53:12.797622+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-24T23:53:12.797622+0000","last_undegraded":"2023-10-24T23:53:12.797622+0000","last_fullsized":"2023-10-24T23:53:12.797622+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:32.324710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101638+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-24T23:53:13.101638+0000","last_peered":"2023-10-24T23:53:13.101638+0000","last_clean":"2023-10-24T23:53:13.101638+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-24T23:53:13.101638+0000","last_undegraded":"2023-10-24T23:53:13.101638+0000","last_fullsized":"2023-10-24T23:53:13.101638+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:04.125835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789210+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-24T23:53:11.789210+0000","last_peered":"2023-10-24T23:53:11.789210+0000","last_clean":"2023-10-24T23:53:11.789210+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-24T23:53:11.789210+0000","last_undegraded":"2023-10-24T23:53:11.789210+0000","last_fullsized":"2023-10-24T23:53:11.789210+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804119+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-24T23:53:11.804119+0000","last_peered":"2023-10-24T23:53:11.804119+0000","last_clean":"2023-10-24T23:53:11.804119+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-24T23:53:11.804119+0000","last_undegraded":"2023-10-24T23:53:11.804119+0000","last_fullsized":"2023-10-24T23:53:11.804119+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605414+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-24T23:53:13.605414+0000","last_peered":"2023-10-24T23:53:13.605414+0000","last_clean":"2023-10-24T23:53:13.605414+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-24T23:53:13.605414+0000","last_undegraded":"2023-10-24T23:53:13.605414+0000","last_fullsized":"2023-10-24T23:53:13.605414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:06:53.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794699+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-24T23:53:12.794699+0000","last_peered":"2023-10-24T23:53:12.794699+0000","last_clean":"2023-10-24T23:53:12.794699+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-24T23:53:12.794699+0000","last_undegraded":"2023-10-24T23:53:12.794699+0000","last_fullsized":"2023-10-24T23:53:12.794699+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:41.905772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804055+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-24T23:53:11.804055+0000","last_peered":"2023-10-24T23:53:11.804055+0000","last_clean":"2023-10-24T23:53:11.804055+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-24T23:53:11.804055+0000","last_undegraded":"2023-10-24T23:53:11.804055+0000","last_fullsized":"2023-10-24T23:53:11.804055+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.802309+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-24T23:53:11.802309+0000","last_peered":"2023-10-24T23:53:11.802309+0000","last_clean":"2023-10-24T23:53:11.802309+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-24T23:53:11.802309+0000","last_undegraded":"2023-10-24T23:53:11.802309+0000","last_fullsized":"2023-10-24T23:53:11.802309+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102244+0000","last_change":"2023-10-24T23:53:11.458463+0000","last_active":"2023-10-24T23:53:13.102244+0000","last_peered":"2023-10-24T23:53:13.102244+0000","last_clean":"2023-10-24T23:53:13.102244+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-24T23:53:13.102244+0000","last_undegraded":"2023-10-24T23:53:13.102244+0000","last_fullsized":"2023-10-24T23:53:13.102244+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:11:45.040156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275727+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-24T23:53:13.275727+0000","last_peered":"2023-10-24T23:53:13.275727+0000","last_clean":"2023-10-24T23:53:13.275727+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-24T23:53:13.275727+0000","last_undegraded":"2023-10-24T23:53:13.275727+0000","last_fullsized":"2023-10-24T23:53:13.275727+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:51.707178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800564+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-24T23:53:11.800564+0000","last_peered":"2023-10-24T23:53:11.800564+0000","last_clean":"2023-10-24T23:53:11.800564+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-24T23:53:11.800564+0000","last_undegraded":"2023-10-24T23:53:11.800564+0000","last_fullsized":"2023-10-24T23:53:11.800564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790875+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-24T23:53:11.790875+0000","last_peered":"2023-10-24T23:53:11.790875+0000","last_clean":"2023-10-24T23:53:11.790875+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-24T23:53:11.790875+0000","last_undegraded":"2023-10-24T23:53:11.790875+0000","last_fullsized":"2023-10-24T23:53:11.790875+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795610+0000","last_change":"2023-10-24T23:53:11.448675+0000","last_active":"2023-10-24T23:53:12.795610+0000","last_peered":"2023-10-24T23:53:12.795610+0000","last_clean":"2023-10-24T23:53:12.795610+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-24T23:53:12.795610+0000","last_undegraded":"2023-10-24T23:53:12.795610+0000","last_fullsized":"2023-10-24T23:53:12.795610+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:01.502343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605168+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-24T23:53:13.605168+0000","last_peered":"2023-10-24T23:53:13.605168+0000","last_clean":"2023-10-24T23:53:13.605168+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-24T23:53:13.605168+0000","last_undegraded":"2023-10-24T23:53:13.605168+0000","last_fullsized":"2023-10-24T23:53:13.605168+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:54.478897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793341+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-24T23:53:11.793341+0000","last_peered":"2023-10-24T23:53:11.793341+0000","last_clean":"2023-10-24T23:53:11.793341+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-24T23:53:11.793341+0000","last_undegraded":"2023-10-24T23:53:11.793341+0000","last_fullsized":"2023-10-24T23:53:11.793341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807904+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-24T23:53:11.807904+0000","last_peered":"2023-10-24T23:53:11.807904+0000","last_clean":"2023-10-24T23:53:11.807904+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-24T23:53:11.807904+0000","last_undegraded":"2023-10-24T23:53:11.807904+0000","last_fullsized":"2023-10-24T23:53:11.807904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275531+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-24T23:53:13.275531+0000","last_peered":"2023-10-24T23:53:13.275531+0000","last_clean":"2023-10-24T23:53:13.275531+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-24T23:53:13.275531+0000","last_undegraded":"2023-10-24T23:53:13.275531+0000","last_fullsized":"2023-10-24T23:53:13.275531+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:58.545886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602449+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-24T23:53:13.602449+0000","last_peered":"2023-10-24T23:53:13.602449+0000","last_clean":"2023-10-24T23:53:13.602449+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-24T23:53:13.602449+0000","last_undegraded":"2023-10-24T23:53:13.602449+0000","last_fullsized":"2023-10-24T23:53:13.602449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:05.595887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788253+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-24T23:53:11.788253+0000","last_peered":"2023-10-24T23:53:11.788253+0000","last_clean":"2023-10-24T23:53:11.788253+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-24T23:53:11.788253+0000","last_undegraded":"2023-10-24T23:53:11.788253+0000","last_fullsized":"2023-10-24T23:53:11.788253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.830849+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-24T23:53:11.830849+0000","last_peered":"2023-10-24T23:53:11.830849+0000","last_clean":"2023-10-24T23:53:11.830849+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-24T23:53:11.830849+0000","last_undegraded":"2023-10-24T23:53:11.830849+0000","last_fullsized":"2023-10-24T23:53:11.830849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275166+0000","last_change":"2023-10-24T23:53:11.454658+0000","last_active":"2023-10-24T23:53:13.275166+0000","last_peered":"2023-10-24T23:53:13.275166+0000","last_clean":"2023-10-24T23:53:13.275166+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-24T23:53:13.275166+0000","last_undegraded":"2023-10-24T23:53:13.275166+0000","last_fullsized":"2023-10-24T23:53:13.275166+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:43:43.869069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796376+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-24T23:53:12.796376+0000","last_peered":"2023-10-24T23:53:12.796376+0000","last_clean":"2023-10-24T23:53:12.796376+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-24T23:53:12.796376+0000","last_undegraded":"2023-10-24T23:53:12.796376+0000","last_fullsized":"2023-10-24T23:53:12.796376+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:09:40.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790517+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-24T23:53:11.790517+0000","last_peered":"2023-10-24T23:53:11.790517+0000","last_clean":"2023-10-24T23:53:11.790517+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-24T23:53:11.790517+0000","last_undegraded":"2023-10-24T23:53:11.790517+0000","last_fullsized":"2023-10-24T23:53:11.790517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825937+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-24T23:53:11.825937+0000","last_peered":"2023-10-24T23:53:11.825937+0000","last_clean":"2023-10-24T23:53:11.825937+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-24T23:53:11.825937+0000","last_undegraded":"2023-10-24T23:53:11.825937+0000","last_fullsized":"2023-10-24T23:53:11.825937+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":13,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275899+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-24T23:53:13.275899+0000","last_peered":"2023-10-24T23:53:13.275899+0000","last_clean":"2023-10-24T23:53:13.275899+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-24T23:53:13.275899+0000","last_undegraded":"2023-10-24T23:53:13.275899+0000","last_fullsized":"2023-10-24T23:53:13.275899+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:02.591627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794746+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-24T23:53:12.794746+0000","last_peered":"2023-10-24T23:53:12.794746+0000","last_clean":"2023-10-24T23:53:12.794746+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-24T23:53:12.794746+0000","last_undegraded":"2023-10-24T23:53:12.794746+0000","last_fullsized":"2023-10-24T23:53:12.794746+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:43.015050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.801731+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-24T23:53:11.801731+0000","last_peered":"2023-10-24T23:53:11.801731+0000","last_clean":"2023-10-24T23:53:11.801731+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-24T23:53:11.801731+0000","last_undegraded":"2023-10-24T23:53:11.801731+0000","last_fullsized":"2023-10-24T23:53:11.801731+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825206+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-24T23:53:11.825206+0000","last_peered":"2023-10-24T23:53:11.825206+0000","last_clean":"2023-10-24T23:53:11.825206+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-24T23:53:11.825206+0000","last_undegraded":"2023-10-24T23:53:11.825206+0000","last_fullsized":"2023-10-24T23:53:11.825206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.274975+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-24T23:53:13.274975+0000","last_peered":"2023-10-24T23:53:13.274975+0000","last_clean":"2023-10-24T23:53:13.274975+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-24T23:53:13.274975+0000","last_undegraded":"2023-10-24T23:53:13.274975+0000","last_fullsized":"2023-10-24T23:53:13.274975+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:59.068359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.100994+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-24T23:53:13.100994+0000","last_peered":"2023-10-24T23:53:13.100994+0000","last_clean":"2023-10-24T23:53:13.100994+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-24T23:53:13.100994+0000","last_undegraded":"2023-10-24T23:53:13.100994+0000","last_fullsized":"2023-10-24T23:53:13.100994+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:13:03.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806468+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-24T23:53:11.806468+0000","last_peered":"2023-10-24T23:53:11.806468+0000","last_clean":"2023-10-24T23:53:11.806468+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-24T23:53:11.806468+0000","last_undegraded":"2023-10-24T23:53:11.806468+0000","last_fullsized":"2023-10-24T23:53:11.806468+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":16,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.827865+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-24T23:53:11.827865+0000","last_peered":"2023-10-24T23:53:11.827865+0000","last_clean":"2023-10-24T23:53:11.827865+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-24T23:53:11.827865+0000","last_undegraded":"2023-10-24T23:53:11.827865+0000","last_fullsized":"2023-10-24T23:53:11.827865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796200+0000","last_change":"2023-10-24T23:53:11.451013+0000","last_active":"2023-10-24T23:53:12.796200+0000","last_peered":"2023-10-24T23:53:12.796200+0000","last_clean":"2023-10-24T23:53:12.796200+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-24T23:53:12.796200+0000","last_undegraded":"2023-10-24T23:53:12.796200+0000","last_fullsized":"2023-10-24T23:53:12.796200+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:52:22.503023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797682+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-24T23:53:12.797682+0000","last_peered":"2023-10-24T23:53:12.797682+0000","last_clean":"2023-10-24T23:53:12.797682+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-24T23:53:12.797682+0000","last_undegraded":"2023-10-24T23:53:12.797682+0000","last_fullsized":"2023-10-24T23:53:12.797682+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:05:30.149693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.102137+0000","last_change":"2023-10-24T23:53:11.434538+0000","last_active":"2023-10-24T23:53:13.102137+0000","last_peered":"2023-10-24T23:53:13.102137+0000","last_clean":"2023-10-24T23:53:13.102137+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-24T23:53:13.102137+0000","last_undegraded":"2023-10-24T23:53:13.102137+0000","last_fullsized":"2023-10-24T23:53:13.102137+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:55.179253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789405+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-24T23:53:11.789405+0000","last_peered":"2023-10-24T23:53:11.789405+0000","last_clean":"2023-10-24T23:53:11.789405+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-24T23:53:11.789405+0000","last_undegraded":"2023-10-24T23:53:11.789405+0000","last_fullsized":"2023-10-24T23:53:11.789405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807405+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-24T23:53:11.807405+0000","last_peered":"2023-10-24T23:53:11.807405+0000","last_clean":"2023-10-24T23:53:11.807405+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-24T23:53:11.807405+0000","last_undegraded":"2023-10-24T23:53:11.807405+0000","last_fullsized":"2023-10-24T23:53:11.807405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275539+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-24T23:53:13.275539+0000","last_peered":"2023-10-24T23:53:13.275539+0000","last_clean":"2023-10-24T23:53:13.275539+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-24T23:53:13.275539+0000","last_undegraded":"2023-10-24T23:53:13.275539+0000","last_fullsized":"2023-10-24T23:53:13.275539+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:33:08.373963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795605+0000","last_change":"2023-10-24T23:53:11.433387+0000","last_active":"2023-10-24T23:53:12.795605+0000","last_peered":"2023-10-24T23:53:12.795605+0000","last_clean":"2023-10-24T23:53:12.795605+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-24T23:53:12.795605+0000","last_undegraded":"2023-10-24T23:53:12.795605+0000","last_fullsized":"2023-10-24T23:53:12.795605+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:29.148450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.824306+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-24T23:53:11.824306+0000","last_peered":"2023-10-24T23:53:11.824306+0000","last_clean":"2023-10-24T23:53:11.824306+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-24T23:53:11.824306+0000","last_undegraded":"2023-10-24T23:53:11.824306+0000","last_fullsized":"2023-10-24T23:53:11.824306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788623+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-24T23:53:11.788623+0000","last_peered":"2023-10-24T23:53:11.788623+0000","last_clean":"2023-10-24T23:53:11.788623+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-24T23:53:11.788623+0000","last_undegraded":"2023-10-24T23:53:11.788623+0000","last_fullsized":"2023-10-24T23:53:11.788623+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.605588+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-24T23:53:13.605588+0000","last_peered":"2023-10-24T23:53:13.605588+0000","last_clean":"2023-10-24T23:53:13.605588+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-24T23:53:13.605588+0000","last_undegraded":"2023-10-24T23:53:13.605588+0000","last_fullsized":"2023-10-24T23:53:13.605588+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:33:06.172412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788704+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-24T23:53:11.788704+0000","last_peered":"2023-10-24T23:53:11.788704+0000","last_clean":"2023-10-24T23:53:11.788704+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-24T23:53:11.788704+0000","last_undegraded":"2023-10-24T23:53:11.788704+0000","last_fullsized":"2023-10-24T23:53:11.788704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832501+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-24T23:53:11.832501+0000","last_peered":"2023-10-24T23:53:11.832501+0000","last_clean":"2023-10-24T23:53:11.832501+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-24T23:53:11.832501+0000","last_undegraded":"2023-10-24T23:53:11.832501+0000","last_fullsized":"2023-10-24T23:53:11.832501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"27'8","reported_seq":19,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:40.023078+0000","last_change":"2023-10-24T23:53:11.446979+0000","last_active":"2023-10-24T23:53:40.023078+0000","last_peered":"2023-10-24T23:53:40.023078+0000","last_clean":"2023-10-24T23:53:40.023078+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-24T23:53:40.023078+0000","last_undegraded":"2023-10-24T23:53:40.023078+0000","last_fullsized":"2023-10-24T23:53:40.023078+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:54:52.843631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2007,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603260+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-24T23:53:13.603260+0000","last_peered":"2023-10-24T23:53:13.603260+0000","last_clean":"2023-10-24T23:53:13.603260+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-24T23:53:13.603260+0000","last_undegraded":"2023-10-24T23:53:13.603260+0000","last_fullsized":"2023-10-24T23:53:13.603260+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:18:48.679682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.816486+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-24T23:53:11.816486+0000","last_peered":"2023-10-24T23:53:11.816486+0000","last_clean":"2023-10-24T23:53:11.816486+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-24T23:53:11.816486+0000","last_undegraded":"2023-10-24T23:53:11.816486+0000","last_fullsized":"2023-10-24T23:53:11.816486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807790+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-24T23:53:11.807790+0000","last_peered":"2023-10-24T23:53:11.807790+0000","last_clean":"2023-10-24T23:53:11.807790+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-24T23:53:11.807790+0000","last_undegraded":"2023-10-24T23:53:11.807790+0000","last_fullsized":"2023-10-24T23:53:11.807790+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101724+0000","last_change":"2023-10-24T23:53:11.436964+0000","last_active":"2023-10-24T23:53:13.101724+0000","last_peered":"2023-10-24T23:53:13.101724+0000","last_clean":"2023-10-24T23:53:13.101724+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-24T23:53:13.101724+0000","last_undegraded":"2023-10-24T23:53:13.101724+0000","last_fullsized":"2023-10-24T23:53:13.101724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:04:15.366966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602222+0000","last_change":"2023-10-24T23:53:11.814304+0000","last_active":"2023-10-24T23:53:13.602222+0000","last_peered":"2023-10-24T23:53:13.602222+0000","last_clean":"2023-10-24T23:53:13.602222+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-24T23:53:13.602222+0000","last_undegraded":"2023-10-24T23:53:13.602222+0000","last_fullsized":"2023-10-24T23:53:13.602222+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:16.635178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789162+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-24T23:53:11.789162+0000","last_peered":"2023-10-24T23:53:11.789162+0000","last_clean":"2023-10-24T23:53:11.789162+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-24T23:53:11.789162+0000","last_undegraded":"2023-10-24T23:53:11.789162+0000","last_fullsized":"2023-10-24T23:53:11.789162+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603105+0000","last_change":"2023-10-24T23:53:11.807096+0000","last_active":"2023-10-24T23:53:13.603105+0000","last_peered":"2023-10-24T23:53:13.603105+0000","last_clean":"2023-10-24T23:53:13.603105+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-24T23:53:13.603105+0000","last_undegraded":"2023-10-24T23:53:13.603105+0000","last_fullsized":"2023-10-24T23:53:13.603105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:25.144578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804445+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-24T23:53:11.804445+0000","last_peered":"2023-10-24T23:53:11.804445+0000","last_clean":"2023-10-24T23:53:11.804445+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-24T23:53:11.804445+0000","last_undegraded":"2023-10-24T23:53:11.804445+0000","last_fullsized":"2023-10-24T23:53:11.804445+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793680+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-24T23:53:12.793680+0000","last_peered":"2023-10-24T23:53:12.793680+0000","last_clean":"2023-10-24T23:53:12.793680+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-24T23:53:12.793680+0000","last_undegraded":"2023-10-24T23:53:12.793680+0000","last_fullsized":"2023-10-24T23:53:12.793680+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:38.946585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.811071+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-24T23:53:11.811071+0000","last_peered":"2023-10-24T23:53:11.811071+0000","last_clean":"2023-10-24T23:53:11.811071+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-24T23:53:11.811071+0000","last_undegraded":"2023-10-24T23:53:11.811071+0000","last_fullsized":"2023-10-24T23:53:11.811071+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.797036+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-24T23:53:12.797036+0000","last_peered":"2023-10-24T23:53:12.797036+0000","last_clean":"2023-10-24T23:53:12.797036+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-24T23:53:12.797036+0000","last_undegraded":"2023-10-24T23:53:12.797036+0000","last_fullsized":"2023-10-24T23:53:12.797036+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:48:24.911279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.807673+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-24T23:53:11.807673+0000","last_peered":"2023-10-24T23:53:11.807673+0000","last_clean":"2023-10-24T23:53:11.807673+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-24T23:53:11.807673+0000","last_undegraded":"2023-10-24T23:53:11.807673+0000","last_fullsized":"2023-10-24T23:53:11.807673+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795515+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-24T23:53:12.795515+0000","last_peered":"2023-10-24T23:53:12.795515+0000","last_clean":"2023-10-24T23:53:12.795515+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-24T23:53:12.795515+0000","last_undegraded":"2023-10-24T23:53:12.795515+0000","last_fullsized":"2023-10-24T23:53:12.795515+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:21.165755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790400+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-24T23:53:11.790400+0000","last_peered":"2023-10-24T23:53:11.790400+0000","last_clean":"2023-10-24T23:53:11.790400+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-24T23:53:11.790400+0000","last_undegraded":"2023-10-24T23:53:11.790400+0000","last_fullsized":"2023-10-24T23:53:11.790400+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.604073+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-24T23:53:13.604073+0000","last_peered":"2023-10-24T23:53:13.604073+0000","last_clean":"2023-10-24T23:53:13.604073+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-24T23:53:13.604073+0000","last_undegraded":"2023-10-24T23:53:13.604073+0000","last_fullsized":"2023-10-24T23:53:13.604073+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.832305+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-24T23:53:11.832305+0000","last_peered":"2023-10-24T23:53:11.832305+0000","last_clean":"2023-10-24T23:53:11.832305+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-24T23:53:11.832305+0000","last_undegraded":"2023-10-24T23:53:11.832305+0000","last_fullsized":"2023-10-24T23:53:11.832305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101339+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-24T23:53:13.101339+0000","last_peered":"2023-10-24T23:53:13.101339+0000","last_clean":"2023-10-24T23:53:13.101339+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-24T23:53:13.101339+0000","last_undegraded":"2023-10-24T23:53:13.101339+0000","last_fullsized":"2023-10-24T23:53:13.101339+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:54.422365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790575+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-24T23:53:11.790575+0000","last_peered":"2023-10-24T23:53:11.790575+0000","last_clean":"2023-10-24T23:53:11.790575+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-24T23:53:11.790575+0000","last_undegraded":"2023-10-24T23:53:11.790575+0000","last_fullsized":"2023-10-24T23:53:11.790575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603531+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-24T23:53:13.603531+0000","last_peered":"2023-10-24T23:53:13.603531+0000","last_clean":"2023-10-24T23:53:13.603531+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-24T23:53:13.603531+0000","last_undegraded":"2023-10-24T23:53:13.603531+0000","last_fullsized":"2023-10-24T23:53:13.603531+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:01:11.838035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.789166+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-24T23:53:11.789166+0000","last_peered":"2023-10-24T23:53:11.789166+0000","last_clean":"2023-10-24T23:53:11.789166+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-24T23:53:11.789166+0000","last_undegraded":"2023-10-24T23:53:11.789166+0000","last_fullsized":"2023-10-24T23:53:11.789166+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603142+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-24T23:53:13.603142+0000","last_peered":"2023-10-24T23:53:13.603142+0000","last_clean":"2023-10-24T23:53:13.603142+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-24T23:53:13.603142+0000","last_undegraded":"2023-10-24T23:53:13.603142+0000","last_fullsized":"2023-10-24T23:53:13.603142+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:09:25.424344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.827005+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-24T23:53:11.827005+0000","last_peered":"2023-10-24T23:53:11.827005+0000","last_clean":"2023-10-24T23:53:11.827005+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-24T23:53:11.827005+0000","last_undegraded":"2023-10-24T23:53:11.827005+0000","last_fullsized":"2023-10-24T23:53:11.827005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101152+0000","last_change":"2023-10-24T23:53:11.824274+0000","last_active":"2023-10-24T23:53:13.101152+0000","last_peered":"2023-10-24T23:53:13.101152+0000","last_clean":"2023-10-24T23:53:13.101152+0000","last_became_active":"2023-10-24T23:53:11.823586+0000","last_became_peered":"2023-10-24T23:53:11.823586+0000","last_unstale":"2023-10-24T23:53:13.101152+0000","last_undegraded":"2023-10-24T23:53:13.101152+0000","last_fullsized":"2023-10-24T23:53:13.101152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:47:36.970336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.804379+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-24T23:53:11.804379+0000","last_peered":"2023-10-24T23:53:11.804379+0000","last_clean":"2023-10-24T23:53:11.804379+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-24T23:53:11.804379+0000","last_undegraded":"2023-10-24T23:53:11.804379+0000","last_fullsized":"2023-10-24T23:53:11.804379+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.274967+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-24T23:53:13.274967+0000","last_peered":"2023-10-24T23:53:13.274967+0000","last_clean":"2023-10-24T23:53:13.274967+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-24T23:53:13.274967+0000","last_undegraded":"2023-10-24T23:53:13.274967+0000","last_fullsized":"2023-10-24T23:53:13.274967+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:10:20.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788652+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-24T23:53:11.788652+0000","last_peered":"2023-10-24T23:53:11.788652+0000","last_clean":"2023-10-24T23:53:11.788652+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-24T23:53:11.788652+0000","last_undegraded":"2023-10-24T23:53:11.788652+0000","last_fullsized":"2023-10-24T23:53:11.788652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796443+0000","last_change":"2023-10-24T23:53:11.860133+0000","last_active":"2023-10-24T23:53:12.796443+0000","last_peered":"2023-10-24T23:53:12.796443+0000","last_clean":"2023-10-24T23:53:12.796443+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-24T23:53:12.796443+0000","last_undegraded":"2023-10-24T23:53:12.796443+0000","last_fullsized":"2023-10-24T23:53:12.796443+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:43.051362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.816202+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-24T23:53:11.816202+0000","last_peered":"2023-10-24T23:53:11.816202+0000","last_clean":"2023-10-24T23:53:11.816202+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-24T23:53:11.816202+0000","last_undegraded":"2023-10-24T23:53:11.816202+0000","last_fullsized":"2023-10-24T23:53:11.816202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603745+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-24T23:53:13.603745+0000","last_peered":"2023-10-24T23:53:13.603745+0000","last_clean":"2023-10-24T23:53:13.603745+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-24T23:53:13.603745+0000","last_undegraded":"2023-10-24T23:53:13.603745+0000","last_fullsized":"2023-10-24T23:53:13.603745+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:39:14.947372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793137+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-24T23:53:11.793137+0000","last_peered":"2023-10-24T23:53:11.793137+0000","last_clean":"2023-10-24T23:53:11.793137+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-24T23:53:11.793137+0000","last_undegraded":"2023-10-24T23:53:11.793137+0000","last_fullsized":"2023-10-24T23:53:11.793137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.798110+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-24T23:53:12.798110+0000","last_peered":"2023-10-24T23:53:12.798110+0000","last_clean":"2023-10-24T23:53:12.798110+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-24T23:53:12.798110+0000","last_undegraded":"2023-10-24T23:53:12.798110+0000","last_fullsized":"2023-10-24T23:53:12.798110+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:37.413259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.796910+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-24T23:53:11.796910+0000","last_peered":"2023-10-24T23:53:11.796910+0000","last_clean":"2023-10-24T23:53:11.796910+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-24T23:53:11.796910+0000","last_undegraded":"2023-10-24T23:53:11.796910+0000","last_fullsized":"2023-10-24T23:53:11.796910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603512+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-24T23:53:13.603512+0000","last_peered":"2023-10-24T23:53:13.603512+0000","last_clean":"2023-10-24T23:53:13.603512+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-24T23:53:13.603512+0000","last_undegraded":"2023-10-24T23:53:13.603512+0000","last_fullsized":"2023-10-24T23:53:13.603512+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:31.344724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.803849+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-24T23:53:11.803849+0000","last_peered":"2023-10-24T23:53:11.803849+0000","last_clean":"2023-10-24T23:53:11.803849+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-24T23:53:11.803849+0000","last_undegraded":"2023-10-24T23:53:11.803849+0000","last_fullsized":"2023-10-24T23:53:11.803849+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275469+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-24T23:53:13.275469+0000","last_peered":"2023-10-24T23:53:13.275469+0000","last_clean":"2023-10-24T23:53:13.275469+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-24T23:53:13.275469+0000","last_undegraded":"2023-10-24T23:53:13.275469+0000","last_fullsized":"2023-10-24T23:53:13.275469+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788160+0000","last_change":"2023-10-24T23:53:02.728971+0000","last_active":"2023-10-24T23:53:11.788160+0000","last_peered":"2023-10-24T23:53:11.788160+0000","last_clean":"2023-10-24T23:53:11.788160+0000","last_became_active":"2023-10-24T23:53:02.728048+0000","last_became_peered":"2023-10-24T23:53:02.728048+0000","last_unstale":"2023-10-24T23:53:11.788160+0000","last_undegraded":"2023-10-24T23:53:11.788160+0000","last_fullsized":"2023-10-24T23:53:11.788160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:31:32.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.601984+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-24T23:53:13.601984+0000","last_peered":"2023-10-24T23:53:13.601984+0000","last_clean":"2023-10-24T23:53:13.601984+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-24T23:53:13.601984+0000","last_undegraded":"2023-10-24T23:53:13.601984+0000","last_fullsized":"2023-10-24T23:53:13.601984+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:32:05.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.791536+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-24T23:53:11.791536+0000","last_peered":"2023-10-24T23:53:11.791536+0000","last_clean":"2023-10-24T23:53:11.791536+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-24T23:53:11.791536+0000","last_undegraded":"2023-10-24T23:53:11.791536+0000","last_fullsized":"2023-10-24T23:53:11.791536+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101460+0000","last_change":"2023-10-24T23:53:11.847814+0000","last_active":"2023-10-24T23:53:13.101460+0000","last_peered":"2023-10-24T23:53:13.101460+0000","last_clean":"2023-10-24T23:53:13.101460+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-24T23:53:13.101460+0000","last_undegraded":"2023-10-24T23:53:13.101460+0000","last_fullsized":"2023-10-24T23:53:13.101460+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:59.075858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.825376+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-24T23:53:11.825376+0000","last_peered":"2023-10-24T23:53:11.825376+0000","last_clean":"2023-10-24T23:53:11.825376+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-24T23:53:11.825376+0000","last_undegraded":"2023-10-24T23:53:11.825376+0000","last_fullsized":"2023-10-24T23:53:11.825376+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.796122+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-24T23:53:12.796122+0000","last_peered":"2023-10-24T23:53:12.796122+0000","last_clean":"2023-10-24T23:53:12.796122+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-24T23:53:12.796122+0000","last_undegraded":"2023-10-24T23:53:12.796122+0000","last_fullsized":"2023-10-24T23:53:12.796122+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:25:12.525273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.828503+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-24T23:53:11.828503+0000","last_peered":"2023-10-24T23:53:11.828503+0000","last_clean":"2023-10-24T23:53:11.828503+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-24T23:53:11.828503+0000","last_undegraded":"2023-10-24T23:53:11.828503+0000","last_fullsized":"2023-10-24T23:53:11.828503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101011+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-24T23:53:13.101011+0000","last_peered":"2023-10-24T23:53:13.101011+0000","last_clean":"2023-10-24T23:53:13.101011+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-24T23:53:13.101011+0000","last_undegraded":"2023-10-24T23:53:13.101011+0000","last_fullsized":"2023-10-24T23:53:13.101011+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:12:21.026996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.795075+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-24T23:53:11.795075+0000","last_peered":"2023-10-24T23:53:11.795075+0000","last_clean":"2023-10-24T23:53:11.795075+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-24T23:53:11.795075+0000","last_undegraded":"2023-10-24T23:53:11.795075+0000","last_fullsized":"2023-10-24T23:53:11.795075+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101949+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-24T23:53:13.101949+0000","last_peered":"2023-10-24T23:53:13.101949+0000","last_clean":"2023-10-24T23:53:13.101949+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-24T23:53:13.101949+0000","last_undegraded":"2023-10-24T23:53:13.101949+0000","last_fullsized":"2023-10-24T23:53:13.101949+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:07.909021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.797593+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-24T23:53:11.797593+0000","last_peered":"2023-10-24T23:53:11.797593+0000","last_clean":"2023-10-24T23:53:11.797593+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-24T23:53:11.797593+0000","last_undegraded":"2023-10-24T23:53:11.797593+0000","last_fullsized":"2023-10-24T23:53:11.797593+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603906+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-24T23:53:13.603906+0000","last_peered":"2023-10-24T23:53:13.603906+0000","last_clean":"2023-10-24T23:53:13.603906+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-24T23:53:13.603906+0000","last_undegraded":"2023-10-24T23:53:13.603906+0000","last_fullsized":"2023-10-24T23:53:13.603906+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:48.217889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.800508+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-24T23:53:11.800508+0000","last_peered":"2023-10-24T23:53:11.800508+0000","last_clean":"2023-10-24T23:53:11.800508+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-24T23:53:11.800508+0000","last_undegraded":"2023-10-24T23:53:11.800508+0000","last_fullsized":"2023-10-24T23:53:11.800508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.602960+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-24T23:53:13.602960+0000","last_peered":"2023-10-24T23:53:13.602960+0000","last_clean":"2023-10-24T23:53:13.602960+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-24T23:53:13.602960+0000","last_undegraded":"2023-10-24T23:53:13.602960+0000","last_fullsized":"2023-10-24T23:53:13.602960+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:10:18.973606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838707+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-24T23:53:11.838707+0000","last_peered":"2023-10-24T23:53:11.838707+0000","last_clean":"2023-10-24T23:53:11.838707+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-24T23:53:11.838707+0000","last_undegraded":"2023-10-24T23:53:11.838707+0000","last_fullsized":"2023-10-24T23:53:11.838707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.101457+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-24T23:53:13.101457+0000","last_peered":"2023-10-24T23:53:13.101457+0000","last_clean":"2023-10-24T23:53:13.101457+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-24T23:53:13.101457+0000","last_undegraded":"2023-10-24T23:53:13.101457+0000","last_fullsized":"2023-10-24T23:53:13.101457+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:24:45.536141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788141+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-24T23:53:11.788141+0000","last_peered":"2023-10-24T23:53:11.788141+0000","last_clean":"2023-10-24T23:53:11.788141+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-24T23:53:11.788141+0000","last_undegraded":"2023-10-24T23:53:11.788141+0000","last_fullsized":"2023-10-24T23:53:11.788141+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793724+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-24T23:53:12.793724+0000","last_peered":"2023-10-24T23:53:12.793724+0000","last_clean":"2023-10-24T23:53:12.793724+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-24T23:53:12.793724+0000","last_undegraded":"2023-10-24T23:53:12.793724+0000","last_fullsized":"2023-10-24T23:53:12.793724+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:06.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.806250+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-24T23:53:11.806250+0000","last_peered":"2023-10-24T23:53:11.806250+0000","last_clean":"2023-10-24T23:53:11.806250+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-24T23:53:11.806250+0000","last_undegraded":"2023-10-24T23:53:11.806250+0000","last_fullsized":"2023-10-24T23:53:11.806250+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275114+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-24T23:53:13.275114+0000","last_peered":"2023-10-24T23:53:13.275114+0000","last_clean":"2023-10-24T23:53:13.275114+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-24T23:53:13.275114+0000","last_undegraded":"2023-10-24T23:53:13.275114+0000","last_fullsized":"2023-10-24T23:53:13.275114+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:56:02.798009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.823970+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-24T23:53:11.823970+0000","last_peered":"2023-10-24T23:53:11.823970+0000","last_clean":"2023-10-24T23:53:11.823970+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-24T23:53:11.823970+0000","last_undegraded":"2023-10-24T23:53:11.823970+0000","last_fullsized":"2023-10-24T23:53:11.823970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.793903+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-24T23:53:12.793903+0000","last_peered":"2023-10-24T23:53:12.793903+0000","last_clean":"2023-10-24T23:53:12.793903+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-24T23:53:12.793903+0000","last_undegraded":"2023-10-24T23:53:12.793903+0000","last_fullsized":"2023-10-24T23:53:12.793903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:40.168016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.809345+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-24T23:53:11.809345+0000","last_peered":"2023-10-24T23:53:11.809345+0000","last_clean":"2023-10-24T23:53:11.809345+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-24T23:53:11.809345+0000","last_undegraded":"2023-10-24T23:53:11.809345+0000","last_fullsized":"2023-10-24T23:53:11.809345+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.100799+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-24T23:53:13.100799+0000","last_peered":"2023-10-24T23:53:13.100799+0000","last_clean":"2023-10-24T23:53:13.100799+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-24T23:53:13.100799+0000","last_undegraded":"2023-10-24T23:53:13.100799+0000","last_fullsized":"2023-10-24T23:53:13.100799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:39.720375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.788501+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-24T23:53:11.788501+0000","last_peered":"2023-10-24T23:53:11.788501+0000","last_clean":"2023-10-24T23:53:11.788501+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-24T23:53:11.788501+0000","last_undegraded":"2023-10-24T23:53:11.788501+0000","last_fullsized":"2023-10-24T23:53:11.788501+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.794282+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-24T23:53:12.794282+0000","last_peered":"2023-10-24T23:53:12.794282+0000","last_clean":"2023-10-24T23:53:12.794282+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-24T23:53:12.794282+0000","last_undegraded":"2023-10-24T23:53:12.794282+0000","last_fullsized":"2023-10-24T23:53:12.794282+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:51:40.960630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.790034+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-24T23:53:11.790034+0000","last_peered":"2023-10-24T23:53:11.790034+0000","last_clean":"2023-10-24T23:53:11.790034+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-24T23:53:11.790034+0000","last_undegraded":"2023-10-24T23:53:11.790034+0000","last_fullsized":"2023-10-24T23:53:11.790034+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:12.795555+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-24T23:53:12.795555+0000","last_peered":"2023-10-24T23:53:12.795555+0000","last_clean":"2023-10-24T23:53:12.795555+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-24T23:53:12.795555+0000","last_undegraded":"2023-10-24T23:53:12.795555+0000","last_fullsized":"2023-10-24T23:53:12.795555+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:22:42.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.793089+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-24T23:53:11.793089+0000","last_peered":"2023-10-24T23:53:11.793089+0000","last_clean":"2023-10-24T23:53:11.793089+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-24T23:53:11.793089+0000","last_undegraded":"2023-10-24T23:53:11.793089+0000","last_fullsized":"2023-10-24T23:53:11.793089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.275507+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-24T23:53:13.275507+0000","last_peered":"2023-10-24T23:53:13.275507+0000","last_clean":"2023-10-24T23:53:13.275507+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-24T23:53:13.275507+0000","last_undegraded":"2023-10-24T23:53:13.275507+0000","last_fullsized":"2023-10-24T23:53:13.275507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:47.658717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.838499+0000","last_change":"2023-10-24T23:53:02.715966+0000","last_active":"2023-10-24T23:53:11.838499+0000","last_peered":"2023-10-24T23:53:11.838499+0000","last_clean":"2023-10-24T23:53:11.838499+0000","last_became_active":"2023-10-24T23:53:02.715631+0000","last_became_peered":"2023-10-24T23:53:02.715631+0000","last_unstale":"2023-10-24T23:53:11.838499+0000","last_undegraded":"2023-10-24T23:53:11.838499+0000","last_fullsized":"2023-10-24T23:53:11.838499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:11.771864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":27,"state":"active+clean","last_fresh":"2023-10-24T23:53:13.603292+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-24T23:53:13.603292+0000","last_peered":"2023-10-24T23:53:13.603292+0000","last_clean":"2023-10-24T23:53:13.603292+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-24T23:53:13.603292+0000","last_undegraded":"2023-10-24T23:53:13.603292+0000","last_fullsized":"2023-10-24T23:53:13.603292+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:50.994593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-24T23:53:11.796890+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-24T23:53:11.796890+0000","last_peered":"2023-10-24T23:53:11.796890+0000","last_clean":"2023-10-24T23:53:11.796890+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-24T23:53:11.796890+0000","last_undegraded":"2023-10-24T23:53:11.796890+0000","last_fullsized":"2023-10-24T23:53:11.796890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2805,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2805,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":929792,"data_stored":713571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":294912,"data_stored":119928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]}]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":307200,"data_stored":120110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71499999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1870000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":319488,"data_stored":122013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.49}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":304,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":311296,"data_stored":121477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2649999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4690000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.663}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574861,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":294912,"data_stored":119436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.143}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]}]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":892928,"data_stored":710330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:47.117 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-10-24T23:53:47.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-24T23:53:47.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.257+0000 7f2a2e846640 1 Processor -- start 2023-10-24T23:53:47.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.257+0000 7f2a2e846640 1 -- start start 2023-10-24T23:53:47.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a280a9820 0x7f2a280ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2a2806ee40 0x7f2a280aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a2806f730 con 0x7f2a2814fd40 2023-10-24T23:53:47.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a280aa750 con 0x7f2a2806ee40 2023-10-24T23:53:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a28156a40 con 0x7f2a280a9820 2023-10-24T23:53:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a27fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2a2814fd40 0x7f2a280c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45158/0 (socket says 172.21.15.107:45158) 2023-10-24T23:53:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a27fff640 1 -- 172.21.15.107:0/1121085106 learned_addr learned my addr 172.21.15.107:0/1121085106 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2cde3640 1 --2- 172.21.15.107:0/1121085106 >> v2:172.21.15.107:3300/0 conn(0x7f2a2806ee40 0x7f2a280aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a277fe640 1 --2- 172.21.15.107:0/1121085106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a280a9820 0x7f2a280ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2cde3640 1 -- 172.21.15.107:0/1121085106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a280a9820 msgr2=0x7f2a280ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:47.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2cde3640 1 --2- 172.21.15.107:0/1121085106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a280a9820 0x7f2a280ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2cde3640 1 -- 172.21.15.107:0/1121085106 >> v1:172.21.15.107:6789/0 conn(0x7f2a2814fd40 legacy=0x7f2a280c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2cde3640 1 -- 172.21.15.107:0/1121085106 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a28156710 con 0x7f2a2806ee40 2023-10-24T23:53:47.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a26ffd640 1 -- 172.21.15.107:0/1121085106 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616206607 0 0) 0x7f2a2806f730 con 0x7f2a2814fd40 2023-10-24T23:53:47.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2cde3640 1 --2- 172.21.15.107:0/1121085106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a2806ee40 0x7f2a280aa210 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009d90 tx=0x7f2a2001c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e293b23712ffc01 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:47.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a26ffd640 1 -- 172.21.15.107:0/1121085106 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a200a5860 con 0x7f2a2806ee40 2023-10-24T23:53:47.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a26ffd640 1 -- 172.21.15.107:0/1121085106 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a200a59c0 con 0x7f2a2806ee40 2023-10-24T23:53:47.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a26ffd640 1 -- 172.21.15.107:0/1121085106 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a200a5d50 con 0x7f2a2806ee40 2023-10-24T23:53:47.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 -- 172.21.15.107:0/1121085106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a2806ee40 msgr2=0x7f2a280aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 --2- 172.21.15.107:0/1121085106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a2806ee40 0x7f2a280aa210 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009d90 tx=0x7f2a2001c800 comp rx=0 tx=0).stop 2023-10-24T23:53:47.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 -- 172.21.15.107:0/1121085106 shutdown_connections 2023-10-24T23:53:47.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 --2- 172.21.15.107:0/1121085106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a2806ee40 0x7f2a280aa210 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 --2- 172.21.15.107:0/1121085106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a280a9820 0x7f2a280ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 -- 172.21.15.107:0/1121085106 >> 172.21.15.107:0/1121085106 conn(0x7f2a280c2950 msgr2=0x7f2a280c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:47.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 -- 172.21.15.107:0/1121085106 shutdown_connections 2023-10-24T23:53:47.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.261+0000 7f2a2e846640 1 -- 172.21.15.107:0/1121085106 wait complete. 2023-10-24T23:53:47.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a2e846640 1 Processor -- start 2023-10-24T23:53:47.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a2e846640 1 -- start start 2023-10-24T23:53:47.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a2e846640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a2806ee40 0x7f2a2815b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a2e846640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a2814fd40 0x7f2a28153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a2e846640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a28153ea0 0x7f2a28143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a2e846640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2814d530 con 0x7f2a2806ee40 2023-10-24T23:53:47.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a2e846640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a28144850 con 0x7f2a28153ea0 2023-10-24T23:53:47.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a2e846640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a28155ac0 con 0x7f2a2814fd40 2023-10-24T23:53:47.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a27fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a2806ee40 0x7f2a2815b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a27fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a2806ee40 0x7f2a2815b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37058/0 (socket says 172.21.15.107:37058) 2023-10-24T23:53:47.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a27fff640 1 -- 172.21.15.107:0/4137632076 learned_addr learned my addr 172.21.15.107:0/4137632076 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:47.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a27fff640 1 -- 172.21.15.107:0/4137632076 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a2814fd40 msgr2=0x7f2a28153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a27fff640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a2814fd40 0x7f2a28153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a27fff640 1 -- 172.21.15.107:0/4137632076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a28153ea0 msgr2=0x7f2a28143970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a27fff640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a28153ea0 0x7f2a28143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a27fff640 1 -- 172.21.15.107:0/4137632076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a20009a70 con 0x7f2a2806ee40 2023-10-24T23:53:47.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a27fff640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a2806ee40 0x7f2a2815b5a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800c4b0 tx=0x7f2a1800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:47.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a24ff9640 1 -- 172.21.15.107:0/4137632076 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a180074e0 con 0x7f2a2806ee40 2023-10-24T23:53:47.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a24ff9640 1 -- 172.21.15.107:0/4137632076 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a18007660 con 0x7f2a2806ee40 2023-10-24T23:53:47.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a2e846640 1 -- 172.21.15.107:0/4137632076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a28144570 con 0x7f2a2806ee40 2023-10-24T23:53:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a24ff9640 1 -- 172.21.15.107:0/4137632076 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18004a30 con 0x7f2a2806ee40 2023-10-24T23:53:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a2e846640 1 -- 172.21.15.107:0/4137632076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a28154fb0 con 0x7f2a2806ee40 2023-10-24T23:53:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a2e846640 1 -- 172.21.15.107:0/4137632076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a280c24a0 con 0x7f2a2806ee40 2023-10-24T23:53:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.265+0000 7f2a24ff9640 1 -- 172.21.15.107:0/4137632076 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2a18004b90 con 0x7f2a2806ee40 2023-10-24T23:53:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.269+0000 7f2a24ff9640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a14065e90 0x7f2a14068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.269+0000 7f2a24ff9640 1 -- 172.21.15.107:0/4137632076 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f2a18082ff0 con 0x7f2a2806ee40 2023-10-24T23:53:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.269+0000 7f2a277fe640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a14065e90 0x7f2a14068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.269+0000 7f2a277fe640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a14065e90 0x7f2a14068350 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2a2815be10 tx=0x7f2a1c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.273+0000 7f2a24ff9640 1 -- 172.21.15.107:0/4137632076 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2a18050340 con 0x7f2a2806ee40 2023-10-24T23:53:47.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.421+0000 7f2a2e846640 1 -- 172.21.15.107:0/4137632076 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2a2806fc60 con 0x7f2a14065e90 2023-10-24T23:53:47.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.433+0000 7f2a24ff9640 1 -- 172.21.15.107:0/4137632076 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525126 (secure 0 0 0) 0x7f2a1c009e40 con 0x7f2a14065e90 2023-10-24T23:53:47.445 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:47.449 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-24T23:53:47.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 -- 172.21.15.107:0/4137632076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a14065e90 msgr2=0x7f2a14068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a14065e90 0x7f2a14068350 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2a2815be10 tx=0x7f2a1c008040 comp rx=0 tx=0).stop 2023-10-24T23:53:47.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 -- 172.21.15.107:0/4137632076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a2806ee40 msgr2=0x7f2a2815b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a2806ee40 0x7f2a2815b5a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800c4b0 tx=0x7f2a1800c970 comp rx=0 tx=0).stop 2023-10-24T23:53:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 -- 172.21.15.107:0/4137632076 shutdown_connections 2023-10-24T23:53:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a28153ea0 0x7f2a28143970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a14065e90 0x7f2a14068350 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a2814fd40 0x7f2a28153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 --2- 172.21.15.107:0/4137632076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a2806ee40 0x7f2a2815b5a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 -- 172.21.15.107:0/4137632076 >> 172.21.15.107:0/4137632076 conn(0x7f2a280c2950 msgr2=0x7f2a28071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:47.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 -- 172.21.15.107:0/4137632076 shutdown_connections 2023-10-24T23:53:47.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.441+0000 7f2a2e846640 1 -- 172.21.15.107:0/4137632076 wait complete. 2023-10-24T23:53:47.463 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2023-10-24T23:53:47.291526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597608,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":255,"num_write_kb":2450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":401,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235716,"kb_used_data":3716,"kb_used_omap":0,"kb_used_meta":231936,"kb_avail":754739004,"statfs":{"total":773094113280,"available":772852740096,"internally_reserved":0,"allocated":3805184,"data_stored":2244288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":237502464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2516,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002826"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.321114+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-24T23:53:44.321114+0000","last_peered":"2023-10-24T23:53:44.321114+0000","last_clean":"2023-10-24T23:53:44.321114+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-24T23:53:44.321114+0000","last_undegraded":"2023-10-24T23:53:44.321114+0000","last_fullsized":"2023-10-24T23:53:44.321114+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:51:20.137796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.474861+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-24T23:53:43.474861+0000","last_peered":"2023-10-24T23:53:43.474861+0000","last_clean":"2023-10-24T23:53:43.474861+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-24T23:53:43.474861+0000","last_undegraded":"2023-10-24T23:53:43.474861+0000","last_fullsized":"2023-10-24T23:53:43.474861+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:00:20.101379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"22'4","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469185+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-24T23:53:43.469185+0000","last_peered":"2023-10-24T23:53:43.469185+0000","last_clean":"2023-10-24T23:53:43.469185+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-24T23:53:43.469185+0000","last_undegraded":"2023-10-24T23:53:43.469185+0000","last_fullsized":"2023-10-24T23:53:43.469185+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.321847+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-24T23:53:44.321847+0000","last_peered":"2023-10-24T23:53:44.321847+0000","last_clean":"2023-10-24T23:53:44.321847+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-24T23:53:44.321847+0000","last_undegraded":"2023-10-24T23:53:44.321847+0000","last_fullsized":"2023-10-24T23:53:44.321847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468422+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-24T23:53:43.468422+0000","last_peered":"2023-10-24T23:53:43.468422+0000","last_clean":"2023-10-24T23:53:43.468422+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-24T23:53:43.468422+0000","last_undegraded":"2023-10-24T23:53:43.468422+0000","last_fullsized":"2023-10-24T23:53:43.468422+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:59.368156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.321404+0000","last_change":"2023-10-24T23:53:11.444089+0000","last_active":"2023-10-24T23:53:44.321404+0000","last_peered":"2023-10-24T23:53:44.321404+0000","last_clean":"2023-10-24T23:53:44.321404+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-24T23:53:44.321404+0000","last_undegraded":"2023-10-24T23:53:44.321404+0000","last_fullsized":"2023-10-24T23:53:44.321404+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:54:06.875882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"22'1","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469131+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-24T23:53:43.469131+0000","last_peered":"2023-10-24T23:53:43.469131+0000","last_clean":"2023-10-24T23:53:43.469131+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-24T23:53:43.469131+0000","last_undegraded":"2023-10-24T23:53:43.469131+0000","last_fullsized":"2023-10-24T23:53:43.469131+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468942+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-24T23:53:43.468942+0000","last_peered":"2023-10-24T23:53:43.468942+0000","last_clean":"2023-10-24T23:53:43.468942+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-24T23:53:43.468942+0000","last_undegraded":"2023-10-24T23:53:43.468942+0000","last_fullsized":"2023-10-24T23:53:43.468942+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.467782+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-24T23:53:43.467782+0000","last_peered":"2023-10-24T23:53:43.467782+0000","last_clean":"2023-10-24T23:53:43.467782+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-24T23:53:43.467782+0000","last_undegraded":"2023-10-24T23:53:43.467782+0000","last_fullsized":"2023-10-24T23:53:43.467782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:38:56.643025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"26'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470182+0000","last_change":"2023-10-24T23:53:11.438681+0000","last_active":"2023-10-24T23:53:43.470182+0000","last_peered":"2023-10-24T23:53:43.470182+0000","last_clean":"2023-10-24T23:53:43.470182+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-24T23:53:43.470182+0000","last_undegraded":"2023-10-24T23:53:43.470182+0000","last_fullsized":"2023-10-24T23:53:43.470182+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:50.139697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"22'2","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.472904+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-24T23:53:43.472904+0000","last_peered":"2023-10-24T23:53:43.472904+0000","last_clean":"2023-10-24T23:53:43.472904+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-24T23:53:43.472904+0000","last_undegraded":"2023-10-24T23:53:43.472904+0000","last_fullsized":"2023-10-24T23:53:43.472904+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322691+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-24T23:53:44.322691+0000","last_peered":"2023-10-24T23:53:44.322691+0000","last_clean":"2023-10-24T23:53:44.322691+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-24T23:53:44.322691+0000","last_undegraded":"2023-10-24T23:53:44.322691+0000","last_fullsized":"2023-10-24T23:53:44.322691+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469672+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-24T23:53:43.469672+0000","last_peered":"2023-10-24T23:53:43.469672+0000","last_clean":"2023-10-24T23:53:43.469672+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-24T23:53:43.469672+0000","last_undegraded":"2023-10-24T23:53:43.469672+0000","last_fullsized":"2023-10-24T23:53:43.469672+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:22.974333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"26'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408174+0000","last_change":"2023-10-24T23:53:11.454537+0000","last_active":"2023-10-24T23:53:44.408174+0000","last_peered":"2023-10-24T23:53:44.408174+0000","last_clean":"2023-10-24T23:53:44.408174+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-24T23:53:44.408174+0000","last_undegraded":"2023-10-24T23:53:44.408174+0000","last_fullsized":"2023-10-24T23:53:44.408174+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:14:06.792651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"22'2","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470708+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-24T23:53:43.470708+0000","last_peered":"2023-10-24T23:53:43.470708+0000","last_clean":"2023-10-24T23:53:43.470708+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-24T23:53:43.470708+0000","last_undegraded":"2023-10-24T23:53:43.470708+0000","last_fullsized":"2023-10-24T23:53:43.470708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322561+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-24T23:53:44.322561+0000","last_peered":"2023-10-24T23:53:44.322561+0000","last_clean":"2023-10-24T23:53:44.322561+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-24T23:53:44.322561+0000","last_undegraded":"2023-10-24T23:53:44.322561+0000","last_fullsized":"2023-10-24T23:53:44.322561+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.323187+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-24T23:53:44.323187+0000","last_peered":"2023-10-24T23:53:44.323187+0000","last_clean":"2023-10-24T23:53:44.323187+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-24T23:53:44.323187+0000","last_undegraded":"2023-10-24T23:53:44.323187+0000","last_fullsized":"2023-10-24T23:53:44.323187+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:35:39.879563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"26'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475380+0000","last_change":"2023-10-24T23:53:11.444138+0000","last_active":"2023-10-24T23:53:43.475380+0000","last_peered":"2023-10-24T23:53:43.475380+0000","last_clean":"2023-10-24T23:53:43.475380+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-24T23:53:43.475380+0000","last_undegraded":"2023-10-24T23:53:43.475380+0000","last_fullsized":"2023-10-24T23:53:43.475380+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:27:38.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"22'1","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470470+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-24T23:53:43.470470+0000","last_peered":"2023-10-24T23:53:43.470470+0000","last_clean":"2023-10-24T23:53:43.470470+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-24T23:53:43.470470+0000","last_undegraded":"2023-10-24T23:53:43.470470+0000","last_fullsized":"2023-10-24T23:53:43.470470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473559+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-24T23:53:43.473559+0000","last_peered":"2023-10-24T23:53:43.473559+0000","last_clean":"2023-10-24T23:53:43.473559+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-24T23:53:43.473559+0000","last_undegraded":"2023-10-24T23:53:43.473559+0000","last_fullsized":"2023-10-24T23:53:43.473559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469486+0000","last_change":"2023-10-24T23:53:11.832809+0000","last_active":"2023-10-24T23:53:43.469486+0000","last_peered":"2023-10-24T23:53:43.469486+0000","last_clean":"2023-10-24T23:53:43.469486+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-24T23:53:43.469486+0000","last_undegraded":"2023-10-24T23:53:43.469486+0000","last_fullsized":"2023-10-24T23:53:43.469486+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:01.151315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"26'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322736+0000","last_change":"2023-10-24T23:53:11.442035+0000","last_active":"2023-10-24T23:53:44.322736+0000","last_peered":"2023-10-24T23:53:44.322736+0000","last_clean":"2023-10-24T23:53:44.322736+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-24T23:53:44.322736+0000","last_undegraded":"2023-10-24T23:53:44.322736+0000","last_fullsized":"2023-10-24T23:53:44.322736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:13.717844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"22'1","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322921+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-24T23:53:44.322921+0000","last_peered":"2023-10-24T23:53:44.322921+0000","last_clean":"2023-10-24T23:53:44.322921+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-24T23:53:44.322921+0000","last_undegraded":"2023-10-24T23:53:44.322921+0000","last_fullsized":"2023-10-24T23:53:44.322921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468808+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-24T23:53:43.468808+0000","last_peered":"2023-10-24T23:53:43.468808+0000","last_clean":"2023-10-24T23:53:43.468808+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-24T23:53:43.468808+0000","last_undegraded":"2023-10-24T23:53:43.468808+0000","last_fullsized":"2023-10-24T23:53:43.468808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.407116+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-24T23:53:44.407116+0000","last_peered":"2023-10-24T23:53:44.407116+0000","last_clean":"2023-10-24T23:53:44.407116+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-24T23:53:44.407116+0000","last_undegraded":"2023-10-24T23:53:44.407116+0000","last_fullsized":"2023-10-24T23:53:44.407116+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:26.012711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"26'4","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.241814+0000","last_change":"2023-10-24T23:53:11.448032+0000","last_active":"2023-10-24T23:53:44.241814+0000","last_peered":"2023-10-24T23:53:44.241814+0000","last_clean":"2023-10-24T23:53:44.241814+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-24T23:53:44.241814+0000","last_undegraded":"2023-10-24T23:53:44.241814+0000","last_fullsized":"2023-10-24T23:53:44.241814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:50:32.490810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.474722+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-24T23:53:43.474722+0000","last_peered":"2023-10-24T23:53:43.474722+0000","last_clean":"2023-10-24T23:53:43.474722+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-24T23:53:43.474722+0000","last_undegraded":"2023-10-24T23:53:43.474722+0000","last_fullsized":"2023-10-24T23:53:43.474722+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469859+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-24T23:53:43.469859+0000","last_peered":"2023-10-24T23:53:43.469859+0000","last_clean":"2023-10-24T23:53:43.469859+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-24T23:53:43.469859+0000","last_undegraded":"2023-10-24T23:53:43.469859+0000","last_fullsized":"2023-10-24T23:53:43.469859+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475187+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-24T23:53:43.475187+0000","last_peered":"2023-10-24T23:53:43.475187+0000","last_clean":"2023-10-24T23:53:43.475187+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-24T23:53:43.475187+0000","last_undegraded":"2023-10-24T23:53:43.475187+0000","last_fullsized":"2023-10-24T23:53:43.475187+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:10.820967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"26'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469812+0000","last_change":"2023-10-24T23:53:11.446925+0000","last_active":"2023-10-24T23:53:43.469812+0000","last_peered":"2023-10-24T23:53:43.469812+0000","last_clean":"2023-10-24T23:53:43.469812+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-24T23:53:43.469812+0000","last_undegraded":"2023-10-24T23:53:43.469812+0000","last_fullsized":"2023-10-24T23:53:43.469812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:59:52.485469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469483+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-24T23:53:43.469483+0000","last_peered":"2023-10-24T23:53:43.469483+0000","last_clean":"2023-10-24T23:53:43.469483+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-24T23:53:43.469483+0000","last_undegraded":"2023-10-24T23:53:43.469483+0000","last_fullsized":"2023-10-24T23:53:43.469483+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.241927+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-24T23:53:44.241927+0000","last_peered":"2023-10-24T23:53:44.241927+0000","last_clean":"2023-10-24T23:53:44.241927+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-24T23:53:44.241927+0000","last_undegraded":"2023-10-24T23:53:44.241927+0000","last_fullsized":"2023-10-24T23:53:44.241927+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.407076+0000","last_change":"2023-10-24T23:53:11.849056+0000","last_active":"2023-10-24T23:53:44.407076+0000","last_peered":"2023-10-24T23:53:44.407076+0000","last_clean":"2023-10-24T23:53:44.407076+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-24T23:53:44.407076+0000","last_undegraded":"2023-10-24T23:53:44.407076+0000","last_fullsized":"2023-10-24T23:53:44.407076+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:08.932942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469336+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-24T23:53:43.469336+0000","last_peered":"2023-10-24T23:53:43.469336+0000","last_clean":"2023-10-24T23:53:43.469336+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-24T23:53:43.469336+0000","last_undegraded":"2023-10-24T23:53:43.469336+0000","last_fullsized":"2023-10-24T23:53:43.469336+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469316+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-24T23:53:43.469316+0000","last_peered":"2023-10-24T23:53:43.469316+0000","last_clean":"2023-10-24T23:53:43.469316+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-24T23:53:43.469316+0000","last_undegraded":"2023-10-24T23:53:43.469316+0000","last_fullsized":"2023-10-24T23:53:43.469316+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.240420+0000","last_change":"2023-10-24T23:53:11.447167+0000","last_active":"2023-10-24T23:53:44.240420+0000","last_peered":"2023-10-24T23:53:44.240420+0000","last_clean":"2023-10-24T23:53:44.240420+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-24T23:53:44.240420+0000","last_undegraded":"2023-10-24T23:53:44.240420+0000","last_fullsized":"2023-10-24T23:53:44.240420+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:06:25.006728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468721+0000","last_change":"2023-10-24T23:53:11.813704+0000","last_active":"2023-10-24T23:53:43.468721+0000","last_peered":"2023-10-24T23:53:43.468721+0000","last_clean":"2023-10-24T23:53:43.468721+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-24T23:53:43.468721+0000","last_undegraded":"2023-10-24T23:53:43.468721+0000","last_fullsized":"2023-10-24T23:53:43.468721+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:39:09.996039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468818+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-24T23:53:43.468818+0000","last_peered":"2023-10-24T23:53:43.468818+0000","last_clean":"2023-10-24T23:53:43.468818+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-24T23:53:43.468818+0000","last_undegraded":"2023-10-24T23:53:43.468818+0000","last_fullsized":"2023-10-24T23:53:43.468818+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473269+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-24T23:53:43.473269+0000","last_peered":"2023-10-24T23:53:43.473269+0000","last_clean":"2023-10-24T23:53:43.473269+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-24T23:53:43.473269+0000","last_undegraded":"2023-10-24T23:53:43.473269+0000","last_fullsized":"2023-10-24T23:53:43.473269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"26'8","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.407753+0000","last_change":"2023-10-24T23:53:11.450974+0000","last_active":"2023-10-24T23:53:44.407753+0000","last_peered":"2023-10-24T23:53:44.407753+0000","last_clean":"2023-10-24T23:53:44.407753+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-24T23:53:44.407753+0000","last_undegraded":"2023-10-24T23:53:44.407753+0000","last_fullsized":"2023-10-24T23:53:44.407753+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:51:05.443664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.467782+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-24T23:53:43.467782+0000","last_peered":"2023-10-24T23:53:43.467782+0000","last_clean":"2023-10-24T23:53:43.467782+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-24T23:53:43.467782+0000","last_undegraded":"2023-10-24T23:53:43.467782+0000","last_fullsized":"2023-10-24T23:53:43.467782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:37:30.278659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"22'3","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.409368+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-24T23:53:44.409368+0000","last_peered":"2023-10-24T23:53:44.409368+0000","last_clean":"2023-10-24T23:53:44.409368+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-24T23:53:44.409368+0000","last_undegraded":"2023-10-24T23:53:44.409368+0000","last_fullsized":"2023-10-24T23:53:44.409368+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.474778+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-24T23:53:43.474778+0000","last_peered":"2023-10-24T23:53:43.474778+0000","last_clean":"2023-10-24T23:53:43.474778+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-24T23:53:43.474778+0000","last_undegraded":"2023-10-24T23:53:43.474778+0000","last_fullsized":"2023-10-24T23:53:43.474778+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"26'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468202+0000","last_change":"2023-10-24T23:53:11.429694+0000","last_active":"2023-10-24T23:53:43.468202+0000","last_peered":"2023-10-24T23:53:43.468202+0000","last_clean":"2023-10-24T23:53:43.468202+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-24T23:53:43.468202+0000","last_undegraded":"2023-10-24T23:53:43.468202+0000","last_fullsized":"2023-10-24T23:53:43.468202+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:15.964994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468499+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-24T23:53:43.468499+0000","last_peered":"2023-10-24T23:53:43.468499+0000","last_clean":"2023-10-24T23:53:43.468499+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-24T23:53:43.468499+0000","last_undegraded":"2023-10-24T23:53:43.468499+0000","last_fullsized":"2023-10-24T23:53:43.468499+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:38:18.678087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"27'9","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.323333+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-24T23:53:44.323333+0000","last_peered":"2023-10-24T23:53:44.323333+0000","last_clean":"2023-10-24T23:53:44.323333+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-24T23:53:44.323333+0000","last_undegraded":"2023-10-24T23:53:44.323333+0000","last_fullsized":"2023-10-24T23:53:44.323333+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322589+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-24T23:53:44.322589+0000","last_peered":"2023-10-24T23:53:44.322589+0000","last_clean":"2023-10-24T23:53:44.322589+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-24T23:53:44.322589+0000","last_undegraded":"2023-10-24T23:53:44.322589+0000","last_fullsized":"2023-10-24T23:53:44.322589+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.474544+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-24T23:53:43.474544+0000","last_peered":"2023-10-24T23:53:43.474544+0000","last_clean":"2023-10-24T23:53:43.474544+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-24T23:53:43.474544+0000","last_undegraded":"2023-10-24T23:53:43.474544+0000","last_fullsized":"2023-10-24T23:53:43.474544+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:30.163913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408236+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-24T23:53:44.408236+0000","last_peered":"2023-10-24T23:53:44.408236+0000","last_clean":"2023-10-24T23:53:44.408236+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-24T23:53:44.408236+0000","last_undegraded":"2023-10-24T23:53:44.408236+0000","last_fullsized":"2023-10-24T23:53:44.408236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:46.000330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322345+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-24T23:53:44.322345+0000","last_peered":"2023-10-24T23:53:44.322345+0000","last_clean":"2023-10-24T23:53:44.322345+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-24T23:53:44.322345+0000","last_undegraded":"2023-10-24T23:53:44.322345+0000","last_fullsized":"2023-10-24T23:53:44.322345+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.242048+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-24T23:53:44.242048+0000","last_peered":"2023-10-24T23:53:44.242048+0000","last_clean":"2023-10-24T23:53:44.242048+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-24T23:53:44.242048+0000","last_undegraded":"2023-10-24T23:53:44.242048+0000","last_fullsized":"2023-10-24T23:53:44.242048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'1","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469992+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-24T23:53:43.469992+0000","last_peered":"2023-10-24T23:53:43.469992+0000","last_clean":"2023-10-24T23:53:43.469992+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-24T23:53:43.469992+0000","last_undegraded":"2023-10-24T23:53:43.469992+0000","last_fullsized":"2023-10-24T23:53:43.469992+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469215+0000","last_change":"2023-10-24T23:53:11.433306+0000","last_active":"2023-10-24T23:53:43.469215+0000","last_peered":"2023-10-24T23:53:43.469215+0000","last_clean":"2023-10-24T23:53:43.469215+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-24T23:53:43.469215+0000","last_undegraded":"2023-10-24T23:53:43.469215+0000","last_fullsized":"2023-10-24T23:53:43.469215+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:44:38.728345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.321091+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-24T23:53:44.321091+0000","last_peered":"2023-10-24T23:53:44.321091+0000","last_clean":"2023-10-24T23:53:44.321091+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-24T23:53:44.321091+0000","last_undegraded":"2023-10-24T23:53:44.321091+0000","last_fullsized":"2023-10-24T23:53:44.321091+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:30:24.805861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469411+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-24T23:53:43.469411+0000","last_peered":"2023-10-24T23:53:43.469411+0000","last_clean":"2023-10-24T23:53:43.469411+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-24T23:53:43.469411+0000","last_undegraded":"2023-10-24T23:53:43.469411+0000","last_fullsized":"2023-10-24T23:53:43.469411+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475303+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-24T23:53:43.475303+0000","last_peered":"2023-10-24T23:53:43.475303+0000","last_clean":"2023-10-24T23:53:43.475303+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-24T23:53:43.475303+0000","last_undegraded":"2023-10-24T23:53:43.475303+0000","last_fullsized":"2023-10-24T23:53:43.475303+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'2","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470072+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-24T23:53:43.470072+0000","last_peered":"2023-10-24T23:53:43.470072+0000","last_clean":"2023-10-24T23:53:43.470072+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-24T23:53:43.470072+0000","last_undegraded":"2023-10-24T23:53:43.470072+0000","last_fullsized":"2023-10-24T23:53:43.470072+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.321885+0000","last_change":"2023-10-24T23:53:11.441983+0000","last_active":"2023-10-24T23:53:44.321885+0000","last_peered":"2023-10-24T23:53:44.321885+0000","last_clean":"2023-10-24T23:53:44.321885+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-24T23:53:44.321885+0000","last_undegraded":"2023-10-24T23:53:44.321885+0000","last_fullsized":"2023-10-24T23:53:44.321885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:30.643019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.474396+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-24T23:53:43.474396+0000","last_peered":"2023-10-24T23:53:43.474396+0000","last_clean":"2023-10-24T23:53:43.474396+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-24T23:53:43.474396+0000","last_undegraded":"2023-10-24T23:53:43.474396+0000","last_fullsized":"2023-10-24T23:53:43.474396+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:50:53.467058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470392+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-24T23:53:43.470392+0000","last_peered":"2023-10-24T23:53:43.470392+0000","last_clean":"2023-10-24T23:53:43.470392+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-24T23:53:43.470392+0000","last_undegraded":"2023-10-24T23:53:43.470392+0000","last_fullsized":"2023-10-24T23:53:43.470392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.409607+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-24T23:53:44.409607+0000","last_peered":"2023-10-24T23:53:44.409607+0000","last_clean":"2023-10-24T23:53:44.409607+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-24T23:53:44.409607+0000","last_undegraded":"2023-10-24T23:53:44.409607+0000","last_fullsized":"2023-10-24T23:53:44.409607+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475164+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-24T23:53:43.475164+0000","last_peered":"2023-10-24T23:53:43.475164+0000","last_clean":"2023-10-24T23:53:43.475164+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-24T23:53:43.475164+0000","last_undegraded":"2023-10-24T23:53:43.475164+0000","last_fullsized":"2023-10-24T23:53:43.475164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.240654+0000","last_change":"2023-10-24T23:53:11.426287+0000","last_active":"2023-10-24T23:53:44.240654+0000","last_peered":"2023-10-24T23:53:44.240654+0000","last_clean":"2023-10-24T23:53:44.240654+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-24T23:53:44.240654+0000","last_undegraded":"2023-10-24T23:53:44.240654+0000","last_fullsized":"2023-10-24T23:53:44.240654+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:02:39.399866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.467978+0000","last_change":"2023-10-24T23:53:11.811139+0000","last_active":"2023-10-24T23:53:43.467978+0000","last_peered":"2023-10-24T23:53:43.467978+0000","last_clean":"2023-10-24T23:53:43.467978+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-24T23:53:43.467978+0000","last_undegraded":"2023-10-24T23:53:43.467978+0000","last_fullsized":"2023-10-24T23:53:43.467978+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:05.507633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.474959+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-24T23:53:43.474959+0000","last_peered":"2023-10-24T23:53:43.474959+0000","last_clean":"2023-10-24T23:53:43.474959+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-24T23:53:43.474959+0000","last_undegraded":"2023-10-24T23:53:43.474959+0000","last_fullsized":"2023-10-24T23:53:43.474959+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475288+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-24T23:53:43.475288+0000","last_peered":"2023-10-24T23:53:43.475288+0000","last_clean":"2023-10-24T23:53:43.475288+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-24T23:53:43.475288+0000","last_undegraded":"2023-10-24T23:53:43.475288+0000","last_fullsized":"2023-10-24T23:53:43.475288+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473129+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-24T23:53:43.473129+0000","last_peered":"2023-10-24T23:53:43.473129+0000","last_clean":"2023-10-24T23:53:43.473129+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-24T23:53:43.473129+0000","last_undegraded":"2023-10-24T23:53:43.473129+0000","last_fullsized":"2023-10-24T23:53:43.473129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470225+0000","last_change":"2023-10-24T23:53:11.426199+0000","last_active":"2023-10-24T23:53:43.470225+0000","last_peered":"2023-10-24T23:53:43.470225+0000","last_clean":"2023-10-24T23:53:43.470225+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-24T23:53:43.470225+0000","last_undegraded":"2023-10-24T23:53:43.470225+0000","last_fullsized":"2023-10-24T23:53:43.470225+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:57:51.018816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468969+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-24T23:53:43.468969+0000","last_peered":"2023-10-24T23:53:43.468969+0000","last_clean":"2023-10-24T23:53:43.468969+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-24T23:53:43.468969+0000","last_undegraded":"2023-10-24T23:53:43.468969+0000","last_fullsized":"2023-10-24T23:53:43.468969+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:20:16.596993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475094+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-24T23:53:43.475094+0000","last_peered":"2023-10-24T23:53:43.475094+0000","last_clean":"2023-10-24T23:53:43.475094+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-24T23:53:43.475094+0000","last_undegraded":"2023-10-24T23:53:43.475094+0000","last_fullsized":"2023-10-24T23:53:43.475094+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322780+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-24T23:53:44.322780+0000","last_peered":"2023-10-24T23:53:44.322780+0000","last_clean":"2023-10-24T23:53:44.322780+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-24T23:53:44.322780+0000","last_undegraded":"2023-10-24T23:53:44.322780+0000","last_fullsized":"2023-10-24T23:53:44.322780+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470200+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-24T23:53:43.470200+0000","last_peered":"2023-10-24T23:53:43.470200+0000","last_clean":"2023-10-24T23:53:43.470200+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-24T23:53:43.470200+0000","last_undegraded":"2023-10-24T23:53:43.470200+0000","last_fullsized":"2023-10-24T23:53:43.470200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469640+0000","last_change":"2023-10-24T23:53:11.432666+0000","last_active":"2023-10-24T23:53:43.469640+0000","last_peered":"2023-10-24T23:53:43.469640+0000","last_clean":"2023-10-24T23:53:43.469640+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-24T23:53:43.469640+0000","last_undegraded":"2023-10-24T23:53:43.469640+0000","last_fullsized":"2023-10-24T23:53:43.469640+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:17.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.409012+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-24T23:53:44.409012+0000","last_peered":"2023-10-24T23:53:44.409012+0000","last_clean":"2023-10-24T23:53:44.409012+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-24T23:53:44.409012+0000","last_undegraded":"2023-10-24T23:53:44.409012+0000","last_fullsized":"2023-10-24T23:53:44.409012+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:42.131555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475350+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-24T23:53:43.475350+0000","last_peered":"2023-10-24T23:53:43.475350+0000","last_clean":"2023-10-24T23:53:43.475350+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-24T23:53:43.475350+0000","last_undegraded":"2023-10-24T23:53:43.475350+0000","last_fullsized":"2023-10-24T23:53:43.475350+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.474507+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-24T23:53:43.474507+0000","last_peered":"2023-10-24T23:53:43.474507+0000","last_clean":"2023-10-24T23:53:43.474507+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-24T23:53:43.474507+0000","last_undegraded":"2023-10-24T23:53:43.474507+0000","last_fullsized":"2023-10-24T23:53:43.474507+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473971+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-24T23:53:43.473971+0000","last_peered":"2023-10-24T23:53:43.473971+0000","last_clean":"2023-10-24T23:53:43.473971+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-24T23:53:43.473971+0000","last_undegraded":"2023-10-24T23:53:43.473971+0000","last_fullsized":"2023-10-24T23:53:43.473971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"27'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473249+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-24T23:53:43.473249+0000","last_peered":"2023-10-24T23:53:43.473249+0000","last_clean":"2023-10-24T23:53:43.473249+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-24T23:53:43.473249+0000","last_undegraded":"2023-10-24T23:53:43.473249+0000","last_fullsized":"2023-10-24T23:53:43.473249+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:55:47.548432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.474903+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-24T23:53:43.474903+0000","last_peered":"2023-10-24T23:53:43.474903+0000","last_clean":"2023-10-24T23:53:43.474903+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-24T23:53:43.474903+0000","last_undegraded":"2023-10-24T23:53:43.474903+0000","last_fullsized":"2023-10-24T23:53:43.474903+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:11:14.431984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469919+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-24T23:53:43.469919+0000","last_peered":"2023-10-24T23:53:43.469919+0000","last_clean":"2023-10-24T23:53:43.469919+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-24T23:53:43.469919+0000","last_undegraded":"2023-10-24T23:53:43.469919+0000","last_fullsized":"2023-10-24T23:53:43.469919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468308+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-24T23:53:43.468308+0000","last_peered":"2023-10-24T23:53:43.468308+0000","last_clean":"2023-10-24T23:53:43.468308+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-24T23:53:43.468308+0000","last_undegraded":"2023-10-24T23:53:43.468308+0000","last_fullsized":"2023-10-24T23:53:43.468308+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408089+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-24T23:53:44.408089+0000","last_peered":"2023-10-24T23:53:44.408089+0000","last_clean":"2023-10-24T23:53:44.408089+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-24T23:53:44.408089+0000","last_undegraded":"2023-10-24T23:53:44.408089+0000","last_fullsized":"2023-10-24T23:53:44.408089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"26'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.472309+0000","last_change":"2023-10-24T23:53:11.436603+0000","last_active":"2023-10-24T23:53:43.472309+0000","last_peered":"2023-10-24T23:53:43.472309+0000","last_clean":"2023-10-24T23:53:43.472309+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-24T23:53:43.472309+0000","last_undegraded":"2023-10-24T23:53:43.472309+0000","last_fullsized":"2023-10-24T23:53:43.472309+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:09:05.948928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.471813+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-24T23:53:43.471813+0000","last_peered":"2023-10-24T23:53:43.471813+0000","last_clean":"2023-10-24T23:53:43.471813+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-24T23:53:43.471813+0000","last_undegraded":"2023-10-24T23:53:43.471813+0000","last_fullsized":"2023-10-24T23:53:43.471813+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:01:59.639080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.241619+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-24T23:53:44.241619+0000","last_peered":"2023-10-24T23:53:44.241619+0000","last_clean":"2023-10-24T23:53:44.241619+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-24T23:53:44.241619+0000","last_undegraded":"2023-10-24T23:53:44.241619+0000","last_fullsized":"2023-10-24T23:53:44.241619+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468405+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-24T23:53:43.468405+0000","last_peered":"2023-10-24T23:53:43.468405+0000","last_clean":"2023-10-24T23:53:43.468405+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-24T23:53:43.468405+0000","last_undegraded":"2023-10-24T23:53:43.468405+0000","last_fullsized":"2023-10-24T23:53:43.468405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470728+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-24T23:53:43.470728+0000","last_peered":"2023-10-24T23:53:43.470728+0000","last_clean":"2023-10-24T23:53:43.470728+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-24T23:53:43.470728+0000","last_undegraded":"2023-10-24T23:53:43.470728+0000","last_fullsized":"2023-10-24T23:53:43.470728+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":173,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408245+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-24T23:53:44.408245+0000","last_peered":"2023-10-24T23:53:44.408245+0000","last_clean":"2023-10-24T23:53:44.408245+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-24T23:53:44.408245+0000","last_undegraded":"2023-10-24T23:53:44.408245+0000","last_fullsized":"2023-10-24T23:53:44.408245+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408794+0000","last_change":"2023-10-24T23:53:11.451073+0000","last_active":"2023-10-24T23:53:44.408794+0000","last_peered":"2023-10-24T23:53:44.408794+0000","last_clean":"2023-10-24T23:53:44.408794+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-24T23:53:44.408794+0000","last_undegraded":"2023-10-24T23:53:44.408794+0000","last_fullsized":"2023-10-24T23:53:44.408794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:32.324710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.471664+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-24T23:53:43.471664+0000","last_peered":"2023-10-24T23:53:43.471664+0000","last_clean":"2023-10-24T23:53:43.471664+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-24T23:53:43.471664+0000","last_undegraded":"2023-10-24T23:53:43.471664+0000","last_fullsized":"2023-10-24T23:53:43.471664+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:04.125835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475358+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-24T23:53:43.475358+0000","last_peered":"2023-10-24T23:53:43.475358+0000","last_clean":"2023-10-24T23:53:43.475358+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-24T23:53:43.475358+0000","last_undegraded":"2023-10-24T23:53:43.475358+0000","last_fullsized":"2023-10-24T23:53:43.475358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469554+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-24T23:53:43.469554+0000","last_peered":"2023-10-24T23:53:43.469554+0000","last_clean":"2023-10-24T23:53:43.469554+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-24T23:53:43.469554+0000","last_undegraded":"2023-10-24T23:53:43.469554+0000","last_fullsized":"2023-10-24T23:53:43.469554+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"26'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.323285+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-24T23:53:44.323285+0000","last_peered":"2023-10-24T23:53:44.323285+0000","last_clean":"2023-10-24T23:53:44.323285+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-24T23:53:44.323285+0000","last_undegraded":"2023-10-24T23:53:44.323285+0000","last_fullsized":"2023-10-24T23:53:44.323285+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:06:53.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469256+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-24T23:53:43.469256+0000","last_peered":"2023-10-24T23:53:43.469256+0000","last_clean":"2023-10-24T23:53:43.469256+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-24T23:53:43.469256+0000","last_undegraded":"2023-10-24T23:53:43.469256+0000","last_fullsized":"2023-10-24T23:53:43.469256+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:41.905772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.323519+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-24T23:53:44.323519+0000","last_peered":"2023-10-24T23:53:44.323519+0000","last_clean":"2023-10-24T23:53:44.323519+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-24T23:53:44.323519+0000","last_undegraded":"2023-10-24T23:53:44.323519+0000","last_fullsized":"2023-10-24T23:53:44.323519+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322461+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-24T23:53:44.322461+0000","last_peered":"2023-10-24T23:53:44.322461+0000","last_clean":"2023-10-24T23:53:44.322461+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-24T23:53:44.322461+0000","last_undegraded":"2023-10-24T23:53:44.322461+0000","last_fullsized":"2023-10-24T23:53:44.322461+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"26'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.471455+0000","last_change":"2023-10-24T23:53:11.458463+0000","last_active":"2023-10-24T23:53:43.471455+0000","last_peered":"2023-10-24T23:53:43.471455+0000","last_clean":"2023-10-24T23:53:43.471455+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-24T23:53:43.471455+0000","last_undegraded":"2023-10-24T23:53:43.471455+0000","last_fullsized":"2023-10-24T23:53:43.471455+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:11:45.040156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473806+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-24T23:53:43.473806+0000","last_peered":"2023-10-24T23:53:43.473806+0000","last_clean":"2023-10-24T23:53:43.473806+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-24T23:53:43.473806+0000","last_undegraded":"2023-10-24T23:53:43.473806+0000","last_fullsized":"2023-10-24T23:53:43.473806+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:51.707178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.323639+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-24T23:53:44.323639+0000","last_peered":"2023-10-24T23:53:44.323639+0000","last_clean":"2023-10-24T23:53:44.323639+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-24T23:53:44.323639+0000","last_undegraded":"2023-10-24T23:53:44.323639+0000","last_fullsized":"2023-10-24T23:53:44.323639+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.241122+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-24T23:53:44.241122+0000","last_peered":"2023-10-24T23:53:44.241122+0000","last_clean":"2023-10-24T23:53:44.241122+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-24T23:53:44.241122+0000","last_undegraded":"2023-10-24T23:53:44.241122+0000","last_fullsized":"2023-10-24T23:53:44.241122+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475216+0000","last_change":"2023-10-24T23:53:11.448675+0000","last_active":"2023-10-24T23:53:43.475216+0000","last_peered":"2023-10-24T23:53:43.475216+0000","last_clean":"2023-10-24T23:53:43.475216+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-24T23:53:43.475216+0000","last_undegraded":"2023-10-24T23:53:43.475216+0000","last_fullsized":"2023-10-24T23:53:43.475216+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:01.502343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.321369+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-24T23:53:44.321369+0000","last_peered":"2023-10-24T23:53:44.321369+0000","last_clean":"2023-10-24T23:53:44.321369+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-24T23:53:44.321369+0000","last_undegraded":"2023-10-24T23:53:44.321369+0000","last_fullsized":"2023-10-24T23:53:44.321369+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:54.478897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.323137+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-24T23:53:44.323137+0000","last_peered":"2023-10-24T23:53:44.323137+0000","last_clean":"2023-10-24T23:53:44.323137+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-24T23:53:44.323137+0000","last_undegraded":"2023-10-24T23:53:44.323137+0000","last_fullsized":"2023-10-24T23:53:44.323137+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"22'2","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468873+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-24T23:53:43.468873+0000","last_peered":"2023-10-24T23:53:43.468873+0000","last_clean":"2023-10-24T23:53:43.468873+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-24T23:53:43.468873+0000","last_undegraded":"2023-10-24T23:53:43.468873+0000","last_fullsized":"2023-10-24T23:53:43.468873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"26'1","reported_seq":14,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473961+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-24T23:53:43.473961+0000","last_peered":"2023-10-24T23:53:43.473961+0000","last_clean":"2023-10-24T23:53:43.473961+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-24T23:53:43.473961+0000","last_undegraded":"2023-10-24T23:53:43.473961+0000","last_fullsized":"2023-10-24T23:53:43.473961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:58.545886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468236+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-24T23:53:43.468236+0000","last_peered":"2023-10-24T23:53:43.468236+0000","last_clean":"2023-10-24T23:53:43.468236+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-24T23:53:43.468236+0000","last_undegraded":"2023-10-24T23:53:43.468236+0000","last_fullsized":"2023-10-24T23:53:43.468236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:05.595887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470329+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-24T23:53:43.470329+0000","last_peered":"2023-10-24T23:53:43.470329+0000","last_clean":"2023-10-24T23:53:43.470329+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-24T23:53:43.470329+0000","last_undegraded":"2023-10-24T23:53:43.470329+0000","last_fullsized":"2023-10-24T23:53:43.470329+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470865+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-24T23:53:43.470865+0000","last_peered":"2023-10-24T23:53:43.470865+0000","last_clean":"2023-10-24T23:53:43.470865+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-24T23:53:43.470865+0000","last_undegraded":"2023-10-24T23:53:43.470865+0000","last_fullsized":"2023-10-24T23:53:43.470865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473115+0000","last_change":"2023-10-24T23:53:11.454658+0000","last_active":"2023-10-24T23:53:43.473115+0000","last_peered":"2023-10-24T23:53:43.473115+0000","last_clean":"2023-10-24T23:53:43.473115+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-24T23:53:43.473115+0000","last_undegraded":"2023-10-24T23:53:43.473115+0000","last_fullsized":"2023-10-24T23:53:43.473115+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:43:43.869069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.407112+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-24T23:53:44.407112+0000","last_peered":"2023-10-24T23:53:44.407112+0000","last_clean":"2023-10-24T23:53:44.407112+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-24T23:53:44.407112+0000","last_undegraded":"2023-10-24T23:53:44.407112+0000","last_fullsized":"2023-10-24T23:53:44.407112+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:09:40.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.240892+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-24T23:53:44.240892+0000","last_peered":"2023-10-24T23:53:44.240892+0000","last_clean":"2023-10-24T23:53:44.240892+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-24T23:53:44.240892+0000","last_undegraded":"2023-10-24T23:53:44.240892+0000","last_fullsized":"2023-10-24T23:53:44.240892+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"22'1","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.409440+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-24T23:53:44.409440+0000","last_peered":"2023-10-24T23:53:44.409440+0000","last_clean":"2023-10-24T23:53:44.409440+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-24T23:53:44.409440+0000","last_undegraded":"2023-10-24T23:53:44.409440+0000","last_fullsized":"2023-10-24T23:53:44.409440+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"26'2","reported_seq":15,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473264+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-24T23:53:43.473264+0000","last_peered":"2023-10-24T23:53:43.473264+0000","last_clean":"2023-10-24T23:53:43.473264+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-24T23:53:43.473264+0000","last_undegraded":"2023-10-24T23:53:43.473264+0000","last_fullsized":"2023-10-24T23:53:43.473264+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:02.591627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468560+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-24T23:53:43.468560+0000","last_peered":"2023-10-24T23:53:43.468560+0000","last_clean":"2023-10-24T23:53:43.468560+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-24T23:53:43.468560+0000","last_undegraded":"2023-10-24T23:53:43.468560+0000","last_fullsized":"2023-10-24T23:53:43.468560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:43.015050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322072+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-24T23:53:44.322072+0000","last_peered":"2023-10-24T23:53:44.322072+0000","last_clean":"2023-10-24T23:53:44.322072+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-24T23:53:44.322072+0000","last_undegraded":"2023-10-24T23:53:44.322072+0000","last_fullsized":"2023-10-24T23:53:44.322072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"22'2","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.409469+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-24T23:53:44.409469+0000","last_peered":"2023-10-24T23:53:44.409469+0000","last_clean":"2023-10-24T23:53:44.409469+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-24T23:53:44.409469+0000","last_undegraded":"2023-10-24T23:53:44.409469+0000","last_fullsized":"2023-10-24T23:53:44.409469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473790+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-24T23:53:43.473790+0000","last_peered":"2023-10-24T23:53:43.473790+0000","last_clean":"2023-10-24T23:53:43.473790+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-24T23:53:43.473790+0000","last_undegraded":"2023-10-24T23:53:43.473790+0000","last_fullsized":"2023-10-24T23:53:43.473790+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:59.068359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469651+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-24T23:53:43.469651+0000","last_peered":"2023-10-24T23:53:43.469651+0000","last_clean":"2023-10-24T23:53:43.469651+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-24T23:53:43.469651+0000","last_undegraded":"2023-10-24T23:53:43.469651+0000","last_fullsized":"2023-10-24T23:53:43.469651+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:13:03.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469785+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-24T23:53:43.469785+0000","last_peered":"2023-10-24T23:53:43.469785+0000","last_clean":"2023-10-24T23:53:43.469785+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-24T23:53:43.469785+0000","last_undegraded":"2023-10-24T23:53:43.469785+0000","last_fullsized":"2023-10-24T23:53:43.469785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"22'1","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.409610+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-24T23:53:44.409610+0000","last_peered":"2023-10-24T23:53:44.409610+0000","last_clean":"2023-10-24T23:53:44.409610+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-24T23:53:44.409610+0000","last_undegraded":"2023-10-24T23:53:44.409610+0000","last_fullsized":"2023-10-24T23:53:44.409610+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408018+0000","last_change":"2023-10-24T23:53:11.451013+0000","last_active":"2023-10-24T23:53:44.408018+0000","last_peered":"2023-10-24T23:53:44.408018+0000","last_clean":"2023-10-24T23:53:44.408018+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-24T23:53:44.408018+0000","last_undegraded":"2023-10-24T23:53:44.408018+0000","last_fullsized":"2023-10-24T23:53:44.408018+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:52:22.503023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408982+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-24T23:53:44.408982+0000","last_peered":"2023-10-24T23:53:44.408982+0000","last_clean":"2023-10-24T23:53:44.408982+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-24T23:53:44.408982+0000","last_undegraded":"2023-10-24T23:53:44.408982+0000","last_fullsized":"2023-10-24T23:53:44.408982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:05:30.149693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.471629+0000","last_change":"2023-10-24T23:53:11.434538+0000","last_active":"2023-10-24T23:53:43.471629+0000","last_peered":"2023-10-24T23:53:43.471629+0000","last_clean":"2023-10-24T23:53:43.471629+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-24T23:53:43.471629+0000","last_undegraded":"2023-10-24T23:53:43.471629+0000","last_fullsized":"2023-10-24T23:53:43.471629+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:55.179253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"22'8","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.240696+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-24T23:53:44.240696+0000","last_peered":"2023-10-24T23:53:44.240696+0000","last_clean":"2023-10-24T23:53:44.240696+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-24T23:53:44.240696+0000","last_undegraded":"2023-10-24T23:53:44.240696+0000","last_fullsized":"2023-10-24T23:53:44.240696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469691+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-24T23:53:43.469691+0000","last_peered":"2023-10-24T23:53:43.469691+0000","last_clean":"2023-10-24T23:53:43.469691+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-24T23:53:43.469691+0000","last_undegraded":"2023-10-24T23:53:43.469691+0000","last_fullsized":"2023-10-24T23:53:43.469691+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.474187+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-24T23:53:43.474187+0000","last_peered":"2023-10-24T23:53:43.474187+0000","last_clean":"2023-10-24T23:53:43.474187+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-24T23:53:43.474187+0000","last_undegraded":"2023-10-24T23:53:43.474187+0000","last_fullsized":"2023-10-24T23:53:43.474187+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:33:08.373963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469343+0000","last_change":"2023-10-24T23:53:11.433387+0000","last_active":"2023-10-24T23:53:43.469343+0000","last_peered":"2023-10-24T23:53:43.469343+0000","last_clean":"2023-10-24T23:53:43.469343+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-24T23:53:43.469343+0000","last_undegraded":"2023-10-24T23:53:43.469343+0000","last_fullsized":"2023-10-24T23:53:43.469343+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:29.148450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408599+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-24T23:53:44.408599+0000","last_peered":"2023-10-24T23:53:44.408599+0000","last_clean":"2023-10-24T23:53:44.408599+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-24T23:53:44.408599+0000","last_undegraded":"2023-10-24T23:53:44.408599+0000","last_fullsized":"2023-10-24T23:53:44.408599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469953+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-24T23:53:43.469953+0000","last_peered":"2023-10-24T23:53:43.469953+0000","last_clean":"2023-10-24T23:53:43.469953+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-24T23:53:43.469953+0000","last_undegraded":"2023-10-24T23:53:43.469953+0000","last_fullsized":"2023-10-24T23:53:43.469953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.321418+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-24T23:53:44.321418+0000","last_peered":"2023-10-24T23:53:44.321418+0000","last_clean":"2023-10-24T23:53:44.321418+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-24T23:53:44.321418+0000","last_undegraded":"2023-10-24T23:53:44.321418+0000","last_fullsized":"2023-10-24T23:53:44.321418+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:33:06.172412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469590+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-24T23:53:43.469590+0000","last_peered":"2023-10-24T23:53:43.469590+0000","last_clean":"2023-10-24T23:53:43.469590+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-24T23:53:43.469590+0000","last_undegraded":"2023-10-24T23:53:43.469590+0000","last_fullsized":"2023-10-24T23:53:43.469590+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.472684+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-24T23:53:43.472684+0000","last_peered":"2023-10-24T23:53:43.472684+0000","last_clean":"2023-10-24T23:53:43.472684+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-24T23:53:43.472684+0000","last_undegraded":"2023-10-24T23:53:43.472684+0000","last_fullsized":"2023-10-24T23:53:43.472684+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"27'9","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469458+0000","last_change":"2023-10-24T23:53:11.446979+0000","last_active":"2023-10-24T23:53:43.469458+0000","last_peered":"2023-10-24T23:53:43.469458+0000","last_clean":"2023-10-24T23:53:43.469458+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-24T23:53:43.469458+0000","last_undegraded":"2023-10-24T23:53:43.469458+0000","last_fullsized":"2023-10-24T23:53:43.469458+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:54:52.843631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2822,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469771+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-24T23:53:43.469771+0000","last_peered":"2023-10-24T23:53:43.469771+0000","last_clean":"2023-10-24T23:53:43.469771+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-24T23:53:43.469771+0000","last_undegraded":"2023-10-24T23:53:43.469771+0000","last_fullsized":"2023-10-24T23:53:43.469771+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:18:48.679682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"22'2","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408694+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-24T23:53:44.408694+0000","last_peered":"2023-10-24T23:53:44.408694+0000","last_clean":"2023-10-24T23:53:44.408694+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-24T23:53:44.408694+0000","last_undegraded":"2023-10-24T23:53:44.408694+0000","last_fullsized":"2023-10-24T23:53:44.408694+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469766+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-24T23:53:43.469766+0000","last_peered":"2023-10-24T23:53:43.469766+0000","last_clean":"2023-10-24T23:53:43.469766+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-24T23:53:43.469766+0000","last_undegraded":"2023-10-24T23:53:43.469766+0000","last_fullsized":"2023-10-24T23:53:43.469766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"26'3","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.471836+0000","last_change":"2023-10-24T23:53:11.436964+0000","last_active":"2023-10-24T23:53:43.471836+0000","last_peered":"2023-10-24T23:53:43.471836+0000","last_clean":"2023-10-24T23:53:43.471836+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-24T23:53:43.471836+0000","last_undegraded":"2023-10-24T23:53:43.471836+0000","last_fullsized":"2023-10-24T23:53:43.471836+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:04:15.366966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.240055+0000","last_change":"2023-10-24T23:53:11.814304+0000","last_active":"2023-10-24T23:53:44.240055+0000","last_peered":"2023-10-24T23:53:44.240055+0000","last_clean":"2023-10-24T23:53:44.240055+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-24T23:53:44.240055+0000","last_undegraded":"2023-10-24T23:53:44.240055+0000","last_fullsized":"2023-10-24T23:53:44.240055+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:16.635178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.240557+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-24T23:53:44.240557+0000","last_peered":"2023-10-24T23:53:44.240557+0000","last_clean":"2023-10-24T23:53:44.240557+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-24T23:53:44.240557+0000","last_undegraded":"2023-10-24T23:53:44.240557+0000","last_fullsized":"2023-10-24T23:53:44.240557+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.240037+0000","last_change":"2023-10-24T23:53:11.807096+0000","last_active":"2023-10-24T23:53:44.240037+0000","last_peered":"2023-10-24T23:53:44.240037+0000","last_clean":"2023-10-24T23:53:44.240037+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-24T23:53:44.240037+0000","last_undegraded":"2023-10-24T23:53:44.240037+0000","last_fullsized":"2023-10-24T23:53:44.240037+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:25.144578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468748+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-24T23:53:43.468748+0000","last_peered":"2023-10-24T23:53:43.468748+0000","last_clean":"2023-10-24T23:53:43.468748+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-24T23:53:43.468748+0000","last_undegraded":"2023-10-24T23:53:43.468748+0000","last_fullsized":"2023-10-24T23:53:43.468748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.474291+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-24T23:53:43.474291+0000","last_peered":"2023-10-24T23:53:43.474291+0000","last_clean":"2023-10-24T23:53:43.474291+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-24T23:53:43.474291+0000","last_undegraded":"2023-10-24T23:53:43.474291+0000","last_fullsized":"2023-10-24T23:53:43.474291+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:38.946585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322792+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-24T23:53:44.322792+0000","last_peered":"2023-10-24T23:53:44.322792+0000","last_clean":"2023-10-24T23:53:44.322792+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-24T23:53:44.322792+0000","last_undegraded":"2023-10-24T23:53:44.322792+0000","last_fullsized":"2023-10-24T23:53:44.322792+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.407742+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-24T23:53:44.407742+0000","last_peered":"2023-10-24T23:53:44.407742+0000","last_clean":"2023-10-24T23:53:44.407742+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-24T23:53:44.407742+0000","last_undegraded":"2023-10-24T23:53:44.407742+0000","last_fullsized":"2023-10-24T23:53:44.407742+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:48:24.911279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468644+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-24T23:53:43.468644+0000","last_peered":"2023-10-24T23:53:43.468644+0000","last_clean":"2023-10-24T23:53:43.468644+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-24T23:53:43.468644+0000","last_undegraded":"2023-10-24T23:53:43.468644+0000","last_fullsized":"2023-10-24T23:53:43.468644+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475409+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-24T23:53:43.475409+0000","last_peered":"2023-10-24T23:53:43.475409+0000","last_clean":"2023-10-24T23:53:43.475409+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-24T23:53:43.475409+0000","last_undegraded":"2023-10-24T23:53:43.475409+0000","last_fullsized":"2023-10-24T23:53:43.475409+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:21.165755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475265+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-24T23:53:43.475265+0000","last_peered":"2023-10-24T23:53:43.475265+0000","last_clean":"2023-10-24T23:53:43.475265+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-24T23:53:43.475265+0000","last_undegraded":"2023-10-24T23:53:43.475265+0000","last_fullsized":"2023-10-24T23:53:43.475265+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.240349+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-24T23:53:44.240349+0000","last_peered":"2023-10-24T23:53:44.240349+0000","last_clean":"2023-10-24T23:53:44.240349+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-24T23:53:44.240349+0000","last_undegraded":"2023-10-24T23:53:44.240349+0000","last_fullsized":"2023-10-24T23:53:44.240349+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.472473+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-24T23:53:43.472473+0000","last_peered":"2023-10-24T23:53:43.472473+0000","last_clean":"2023-10-24T23:53:43.472473+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-24T23:53:43.472473+0000","last_undegraded":"2023-10-24T23:53:43.472473+0000","last_fullsized":"2023-10-24T23:53:43.472473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470625+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-24T23:53:43.470625+0000","last_peered":"2023-10-24T23:53:43.470625+0000","last_clean":"2023-10-24T23:53:43.470625+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-24T23:53:43.470625+0000","last_undegraded":"2023-10-24T23:53:43.470625+0000","last_fullsized":"2023-10-24T23:53:43.470625+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:54.422365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475180+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-24T23:53:43.475180+0000","last_peered":"2023-10-24T23:53:43.475180+0000","last_clean":"2023-10-24T23:53:43.475180+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-24T23:53:43.475180+0000","last_undegraded":"2023-10-24T23:53:43.475180+0000","last_fullsized":"2023-10-24T23:53:43.475180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.240083+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-24T23:53:44.240083+0000","last_peered":"2023-10-24T23:53:44.240083+0000","last_clean":"2023-10-24T23:53:44.240083+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-24T23:53:44.240083+0000","last_undegraded":"2023-10-24T23:53:44.240083+0000","last_fullsized":"2023-10-24T23:53:44.240083+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:01:11.838035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.475200+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-24T23:53:43.475200+0000","last_peered":"2023-10-24T23:53:43.475200+0000","last_clean":"2023-10-24T23:53:43.475200+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-24T23:53:43.475200+0000","last_undegraded":"2023-10-24T23:53:43.475200+0000","last_fullsized":"2023-10-24T23:53:43.475200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.321331+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-24T23:53:44.321331+0000","last_peered":"2023-10-24T23:53:44.321331+0000","last_clean":"2023-10-24T23:53:44.321331+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-24T23:53:44.321331+0000","last_undegraded":"2023-10-24T23:53:44.321331+0000","last_fullsized":"2023-10-24T23:53:44.321331+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:09:25.424344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408039+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-24T23:53:44.408039+0000","last_peered":"2023-10-24T23:53:44.408039+0000","last_clean":"2023-10-24T23:53:44.408039+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-24T23:53:44.408039+0000","last_undegraded":"2023-10-24T23:53:44.408039+0000","last_fullsized":"2023-10-24T23:53:44.408039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.474862+0000","last_change":"2023-10-24T23:53:44.474203+0000","last_active":"2023-10-24T23:53:44.474862+0000","last_peered":"2023-10-24T23:53:44.474862+0000","last_clean":"2023-10-24T23:53:44.474862+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-24T23:53:44.474862+0000","last_undegraded":"2023-10-24T23:53:44.474862+0000","last_fullsized":"2023-10-24T23:53:44.474862+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:16:50.292761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468626+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-24T23:53:43.468626+0000","last_peered":"2023-10-24T23:53:43.468626+0000","last_clean":"2023-10-24T23:53:43.468626+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-24T23:53:43.468626+0000","last_undegraded":"2023-10-24T23:53:43.468626+0000","last_fullsized":"2023-10-24T23:53:43.468626+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473054+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-24T23:53:43.473054+0000","last_peered":"2023-10-24T23:53:43.473054+0000","last_clean":"2023-10-24T23:53:43.473054+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-24T23:53:43.473054+0000","last_undegraded":"2023-10-24T23:53:43.473054+0000","last_fullsized":"2023-10-24T23:53:43.473054+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:10:20.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469521+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-24T23:53:43.469521+0000","last_peered":"2023-10-24T23:53:43.469521+0000","last_clean":"2023-10-24T23:53:43.469521+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-24T23:53:43.469521+0000","last_undegraded":"2023-10-24T23:53:43.469521+0000","last_fullsized":"2023-10-24T23:53:43.469521+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.407372+0000","last_change":"2023-10-24T23:53:11.860133+0000","last_active":"2023-10-24T23:53:44.407372+0000","last_peered":"2023-10-24T23:53:44.407372+0000","last_clean":"2023-10-24T23:53:44.407372+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-24T23:53:44.407372+0000","last_undegraded":"2023-10-24T23:53:44.407372+0000","last_fullsized":"2023-10-24T23:53:44.407372+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:43.051362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408566+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-24T23:53:44.408566+0000","last_peered":"2023-10-24T23:53:44.408566+0000","last_clean":"2023-10-24T23:53:44.408566+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-24T23:53:44.408566+0000","last_undegraded":"2023-10-24T23:53:44.408566+0000","last_fullsized":"2023-10-24T23:53:44.408566+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.241583+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-24T23:53:44.241583+0000","last_peered":"2023-10-24T23:53:44.241583+0000","last_clean":"2023-10-24T23:53:44.241583+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-24T23:53:44.241583+0000","last_undegraded":"2023-10-24T23:53:44.241583+0000","last_fullsized":"2023-10-24T23:53:44.241583+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:39:14.947372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322969+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-24T23:53:44.322969+0000","last_peered":"2023-10-24T23:53:44.322969+0000","last_clean":"2023-10-24T23:53:44.322969+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-24T23:53:44.322969+0000","last_undegraded":"2023-10-24T23:53:44.322969+0000","last_fullsized":"2023-10-24T23:53:44.322969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.407249+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-24T23:53:44.407249+0000","last_peered":"2023-10-24T23:53:44.407249+0000","last_clean":"2023-10-24T23:53:44.407249+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-24T23:53:44.407249+0000","last_undegraded":"2023-10-24T23:53:44.407249+0000","last_fullsized":"2023-10-24T23:53:44.407249+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:37.413259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473386+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-24T23:53:43.473386+0000","last_peered":"2023-10-24T23:53:43.473386+0000","last_clean":"2023-10-24T23:53:43.473386+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-24T23:53:43.473386+0000","last_undegraded":"2023-10-24T23:53:43.473386+0000","last_fullsized":"2023-10-24T23:53:43.473386+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.241261+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-24T23:53:44.241261+0000","last_peered":"2023-10-24T23:53:44.241261+0000","last_clean":"2023-10-24T23:53:44.241261+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-24T23:53:44.241261+0000","last_undegraded":"2023-10-24T23:53:44.241261+0000","last_fullsized":"2023-10-24T23:53:44.241261+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:31.344724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.323512+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-24T23:53:44.323512+0000","last_peered":"2023-10-24T23:53:44.323512+0000","last_clean":"2023-10-24T23:53:44.323512+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-24T23:53:44.323512+0000","last_undegraded":"2023-10-24T23:53:44.323512+0000","last_fullsized":"2023-10-24T23:53:44.323512+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473144+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-24T23:53:43.473144+0000","last_peered":"2023-10-24T23:53:43.473144+0000","last_clean":"2023-10-24T23:53:43.473144+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-24T23:53:43.473144+0000","last_undegraded":"2023-10-24T23:53:43.473144+0000","last_fullsized":"2023-10-24T23:53:43.473144+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.629959+0000","last_change":"2023-10-24T23:53:44.629959+0000","last_active":"2023-10-24T23:53:44.629959+0000","last_peered":"2023-10-24T23:53:44.629959+0000","last_clean":"2023-10-24T23:53:44.629959+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-24T23:53:44.629959+0000","last_undegraded":"2023-10-24T23:53:44.629959+0000","last_fullsized":"2023-10-24T23:53:44.629959+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:59:36.474350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.321998+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-24T23:53:44.321998+0000","last_peered":"2023-10-24T23:53:44.321998+0000","last_clean":"2023-10-24T23:53:44.321998+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-24T23:53:44.321998+0000","last_undegraded":"2023-10-24T23:53:44.321998+0000","last_fullsized":"2023-10-24T23:53:44.321998+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:32:05.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.321754+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-24T23:53:44.321754+0000","last_peered":"2023-10-24T23:53:44.321754+0000","last_clean":"2023-10-24T23:53:44.321754+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-24T23:53:44.321754+0000","last_undegraded":"2023-10-24T23:53:44.321754+0000","last_fullsized":"2023-10-24T23:53:44.321754+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470522+0000","last_change":"2023-10-24T23:53:11.847814+0000","last_active":"2023-10-24T23:53:43.470522+0000","last_peered":"2023-10-24T23:53:43.470522+0000","last_clean":"2023-10-24T23:53:43.470522+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-24T23:53:43.470522+0000","last_undegraded":"2023-10-24T23:53:43.470522+0000","last_fullsized":"2023-10-24T23:53:43.470522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:59.075858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408487+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-24T23:53:44.408487+0000","last_peered":"2023-10-24T23:53:44.408487+0000","last_clean":"2023-10-24T23:53:44.408487+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-24T23:53:44.408487+0000","last_undegraded":"2023-10-24T23:53:44.408487+0000","last_fullsized":"2023-10-24T23:53:44.408487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.407113+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-24T23:53:44.407113+0000","last_peered":"2023-10-24T23:53:44.407113+0000","last_clean":"2023-10-24T23:53:44.407113+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-24T23:53:44.407113+0000","last_undegraded":"2023-10-24T23:53:44.407113+0000","last_fullsized":"2023-10-24T23:53:44.407113+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:25:12.525273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470120+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-24T23:53:43.470120+0000","last_peered":"2023-10-24T23:53:43.470120+0000","last_clean":"2023-10-24T23:53:43.470120+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-24T23:53:43.470120+0000","last_undegraded":"2023-10-24T23:53:43.470120+0000","last_fullsized":"2023-10-24T23:53:43.470120+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469616+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-24T23:53:43.469616+0000","last_peered":"2023-10-24T23:53:43.469616+0000","last_clean":"2023-10-24T23:53:43.469616+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-24T23:53:43.469616+0000","last_undegraded":"2023-10-24T23:53:43.469616+0000","last_fullsized":"2023-10-24T23:53:43.469616+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:12:21.026996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473218+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-24T23:53:43.473218+0000","last_peered":"2023-10-24T23:53:43.473218+0000","last_clean":"2023-10-24T23:53:43.473218+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-24T23:53:43.473218+0000","last_undegraded":"2023-10-24T23:53:43.473218+0000","last_fullsized":"2023-10-24T23:53:43.473218+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.471253+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-24T23:53:43.471253+0000","last_peered":"2023-10-24T23:53:43.471253+0000","last_clean":"2023-10-24T23:53:43.471253+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-24T23:53:43.471253+0000","last_undegraded":"2023-10-24T23:53:43.471253+0000","last_fullsized":"2023-10-24T23:53:43.471253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:07.909021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473418+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-24T23:53:43.473418+0000","last_peered":"2023-10-24T23:53:43.473418+0000","last_clean":"2023-10-24T23:53:43.473418+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-24T23:53:43.473418+0000","last_undegraded":"2023-10-24T23:53:43.473418+0000","last_fullsized":"2023-10-24T23:53:43.473418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468627+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-24T23:53:43.468627+0000","last_peered":"2023-10-24T23:53:43.468627+0000","last_clean":"2023-10-24T23:53:43.468627+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-24T23:53:43.468627+0000","last_undegraded":"2023-10-24T23:53:43.468627+0000","last_fullsized":"2023-10-24T23:53:43.468627+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:48.217889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.323603+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-24T23:53:44.323603+0000","last_peered":"2023-10-24T23:53:44.323603+0000","last_clean":"2023-10-24T23:53:44.323603+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-24T23:53:44.323603+0000","last_undegraded":"2023-10-24T23:53:44.323603+0000","last_fullsized":"2023-10-24T23:53:44.323603+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468396+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-24T23:53:43.468396+0000","last_peered":"2023-10-24T23:53:43.468396+0000","last_clean":"2023-10-24T23:53:43.468396+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-24T23:53:43.468396+0000","last_undegraded":"2023-10-24T23:53:43.468396+0000","last_fullsized":"2023-10-24T23:53:43.468396+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:10:18.973606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473053+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-24T23:53:43.473053+0000","last_peered":"2023-10-24T23:53:43.473053+0000","last_clean":"2023-10-24T23:53:43.473053+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-24T23:53:43.473053+0000","last_undegraded":"2023-10-24T23:53:43.473053+0000","last_fullsized":"2023-10-24T23:53:43.473053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.471442+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-24T23:53:43.471442+0000","last_peered":"2023-10-24T23:53:43.471442+0000","last_clean":"2023-10-24T23:53:43.471442+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-24T23:53:43.471442+0000","last_undegraded":"2023-10-24T23:53:43.471442+0000","last_fullsized":"2023-10-24T23:53:43.471442+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:24:45.536141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468695+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-24T23:53:43.468695+0000","last_peered":"2023-10-24T23:53:43.468695+0000","last_clean":"2023-10-24T23:53:43.468695+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-24T23:53:43.468695+0000","last_undegraded":"2023-10-24T23:53:43.468695+0000","last_fullsized":"2023-10-24T23:53:43.468695+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.474548+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-24T23:53:43.474548+0000","last_peered":"2023-10-24T23:53:43.474548+0000","last_clean":"2023-10-24T23:53:43.474548+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-24T23:53:43.474548+0000","last_undegraded":"2023-10-24T23:53:43.474548+0000","last_fullsized":"2023-10-24T23:53:43.474548+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:06.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469626+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-24T23:53:43.469626+0000","last_peered":"2023-10-24T23:53:43.469626+0000","last_clean":"2023-10-24T23:53:43.469626+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-24T23:53:43.469626+0000","last_undegraded":"2023-10-24T23:53:43.469626+0000","last_fullsized":"2023-10-24T23:53:43.469626+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473147+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-24T23:53:43.473147+0000","last_peered":"2023-10-24T23:53:43.473147+0000","last_clean":"2023-10-24T23:53:43.473147+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-24T23:53:43.473147+0000","last_undegraded":"2023-10-24T23:53:43.473147+0000","last_fullsized":"2023-10-24T23:53:43.473147+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:56:02.798009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.408314+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-24T23:53:44.408314+0000","last_peered":"2023-10-24T23:53:44.408314+0000","last_clean":"2023-10-24T23:53:44.408314+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-24T23:53:44.408314+0000","last_undegraded":"2023-10-24T23:53:44.408314+0000","last_fullsized":"2023-10-24T23:53:44.408314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469141+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-24T23:53:43.469141+0000","last_peered":"2023-10-24T23:53:43.469141+0000","last_clean":"2023-10-24T23:53:43.469141+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-24T23:53:43.469141+0000","last_undegraded":"2023-10-24T23:53:43.469141+0000","last_fullsized":"2023-10-24T23:53:43.469141+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:40.168016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.322556+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-24T23:53:44.322556+0000","last_peered":"2023-10-24T23:53:44.322556+0000","last_clean":"2023-10-24T23:53:44.322556+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-24T23:53:44.322556+0000","last_undegraded":"2023-10-24T23:53:44.322556+0000","last_fullsized":"2023-10-24T23:53:44.322556+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469454+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-24T23:53:43.469454+0000","last_peered":"2023-10-24T23:53:43.469454+0000","last_clean":"2023-10-24T23:53:43.469454+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-24T23:53:43.469454+0000","last_undegraded":"2023-10-24T23:53:43.469454+0000","last_fullsized":"2023-10-24T23:53:43.469454+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:39.720375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.470430+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-24T23:53:43.470430+0000","last_peered":"2023-10-24T23:53:43.470430+0000","last_clean":"2023-10-24T23:53:43.470430+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-24T23:53:43.470430+0000","last_undegraded":"2023-10-24T23:53:43.470430+0000","last_fullsized":"2023-10-24T23:53:43.470430+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.468926+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-24T23:53:43.468926+0000","last_peered":"2023-10-24T23:53:43.468926+0000","last_clean":"2023-10-24T23:53:43.468926+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-24T23:53:43.468926+0000","last_undegraded":"2023-10-24T23:53:43.468926+0000","last_fullsized":"2023-10-24T23:53:43.468926+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:51:40.960630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.241408+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-24T23:53:44.241408+0000","last_peered":"2023-10-24T23:53:44.241408+0000","last_clean":"2023-10-24T23:53:44.241408+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-24T23:53:44.241408+0000","last_undegraded":"2023-10-24T23:53:44.241408+0000","last_fullsized":"2023-10-24T23:53:44.241408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469257+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-24T23:53:43.469257+0000","last_peered":"2023-10-24T23:53:43.469257+0000","last_clean":"2023-10-24T23:53:43.469257+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-24T23:53:43.469257+0000","last_undegraded":"2023-10-24T23:53:43.469257+0000","last_fullsized":"2023-10-24T23:53:43.469257+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:22:42.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.323367+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-24T23:53:44.323367+0000","last_peered":"2023-10-24T23:53:44.323367+0000","last_clean":"2023-10-24T23:53:44.323367+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-24T23:53:44.323367+0000","last_undegraded":"2023-10-24T23:53:44.323367+0000","last_fullsized":"2023-10-24T23:53:44.323367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473239+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-24T23:53:43.473239+0000","last_peered":"2023-10-24T23:53:43.473239+0000","last_clean":"2023-10-24T23:53:43.473239+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-24T23:53:43.473239+0000","last_undegraded":"2023-10-24T23:53:43.473239+0000","last_fullsized":"2023-10-24T23:53:43.473239+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:47.658717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-24T23:53:44.630867+0000","last_change":"2023-10-24T23:53:44.630867+0000","last_active":"2023-10-24T23:53:44.630867+0000","last_peered":"2023-10-24T23:53:44.630867+0000","last_clean":"2023-10-24T23:53:44.630867+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-24T23:53:44.630867+0000","last_undegraded":"2023-10-24T23:53:44.630867+0000","last_fullsized":"2023-10-24T23:53:44.630867+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:49.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.469702+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-24T23:53:43.469702+0000","last_peered":"2023-10-24T23:53:43.469702+0000","last_clean":"2023-10-24T23:53:43.469702+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-24T23:53:43.469702+0000","last_undegraded":"2023-10-24T23:53:43.469702+0000","last_fullsized":"2023-10-24T23:53:43.469702+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:50.994593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"0'0","reported_seq":16,"reported_epoch":28,"state":"active+clean","last_fresh":"2023-10-24T23:53:43.473824+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-24T23:53:43.473824+0000","last_peered":"2023-10-24T23:53:43.473824+0000","last_clean":"2023-10-24T23:53:43.473824+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-24T23:53:43.473824+0000","last_undegraded":"2023-10-24T23:53:43.473824+0000","last_fullsized":"2023-10-24T23:53:43.473824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":3620,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3620,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574862,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":942080,"data_stored":722892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574862,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31360,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340480,"statfs":{"total":96636764160,"available":96604651520,"internally_reserved":0,"allocated":319488,"data_stored":134440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]}]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":331776,"data_stored":134622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]}]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":324,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":331776,"data_stored":130076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81699999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.252}]}]},{"osd":3,"up_from":13,"seq":55834574862,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":335872,"data_stored":136804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":300,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":307200,"data_stored":126664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":312,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":319488,"data_stored":133948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95299999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27848,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":94343992,"statfs":{"total":96636764160,"available":96608247808,"internally_reserved":0,"allocated":917504,"data_stored":724842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":3038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-24T23:53:47.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:47.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.621+0000 7ff092093640 1 Processor -- start 2023-10-24T23:53:47.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.621+0000 7ff092093640 1 -- start start 2023-10-24T23:53:47.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff092093640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff08c152020 0x7ff08c0c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff092093640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c152400 con 0x7ff08c0c8640 2023-10-24T23:53:47.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff092093640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c1558d0 con 0x7ff08c1553d0 2023-10-24T23:53:47.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff092093640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08c0c6ce0 con 0x7ff08c152020 2023-10-24T23:53:47.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff082ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff08c0c8640 0x7ff08c06a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45172/0 (socket says 172.21.15.107:45172) 2023-10-24T23:53:47.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff082ffd640 1 -- 172.21.15.107:0/3395618078 learned_addr learned my addr 172.21.15.107:0/3395618078 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:47.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420621458 0 0) 0x7ff08c152400 con 0x7ff08c0c8640 2023-10-24T23:53:47.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff078003530 con 0x7ff08c0c8640 2023-10-24T23:53:47.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08b7fe640 1 --2- 172.21.15.107:0/3395618078 >> v2:172.21.15.110:3300/0 conn(0x7ff08c152020 0x7ff08c0c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3236327586 0 0) 0x7ff078003530 con 0x7ff08c0c8640 2023-10-24T23:53:47.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 >> v2:172.21.15.110:3300/0 conn(0x7ff08c152020 msgr2=0x7ff08c0c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:47.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 --2- 172.21.15.107:0/3395618078 >> v2:172.21.15.110:3300/0 conn(0x7ff08c152020 0x7ff08c0c4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 >> v1:172.21.15.110:6789/0 conn(0x7ff08c1553d0 legacy=0x7ff08c06ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:47.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c0c6fc0 con 0x7ff08c0c8640 2023-10-24T23:53:47.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7ff0740016a0 con 0x7ff08c0c8640 2023-10-24T23:53:47.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08b7fe640 1 --2- 172.21.15.107:0/3395618078 >> v2:172.21.15.110:3300/0 conn(0x7ff08c152020 0x7ff08c0c4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:47.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 >> v1:172.21.15.107:6789/0 conn(0x7ff08c0c8640 legacy=0x7ff08c06a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff078003980 0x7ff078003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0780047d0 0x7ff078004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 0x7ff07800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0c6ce0 con 0x7ff078005390 2023-10-24T23:53:47.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07800c230 con 0x7ff0780047d0 2023-10-24T23:53:47.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff07800c3f0 con 0x7ff078003980 2023-10-24T23:53:47.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08bfff640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 0x7ff07800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff082ffd640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0780047d0 0x7ff078004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08b7fe640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff078003980 0x7ff078003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08bfff640 1 -- 172.21.15.107:0/3395618078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff078003980 msgr2=0x7ff078003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:47.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08bfff640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff078003980 0x7ff078003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08bfff640 1 -- 172.21.15.107:0/3395618078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0780047d0 msgr2=0x7ff078004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:47.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08bfff640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0780047d0 0x7ff078004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08bfff640 1 -- 172.21.15.107:0/3395618078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff078003530 con 0x7ff078005390 2023-10-24T23:53:47.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff082ffd640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0780047d0 0x7ff078004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:47.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08bfff640 1 -- 172.21.15.107:0/3395618078 reap_dead start 2023-10-24T23:53:47.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08bfff640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 0x7ff07800b990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff08c155800 tx=0x7ff08401c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f90b3d6cbe0d64 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0840a9660 con 0x7ff078005390 2023-10-24T23:53:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0840a97c0 con 0x7ff078005390 2023-10-24T23:53:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.625+0000 7ff08affd640 1 -- 172.21.15.107:0/3395618078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0840a9b70 con 0x7ff078005390 2023-10-24T23:53:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- 172.21.15.107:0/3395618078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 msgr2=0x7ff07800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 0x7ff07800b990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff08c155800 tx=0x7ff08401c660 comp rx=0 tx=0).stop 2023-10-24T23:53:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- 172.21.15.107:0/3395618078 shutdown_connections 2023-10-24T23:53:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 --2- 172.21.15.107:0/3395618078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 0x7ff07800b990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- 172.21.15.107:0/3395618078 >> 172.21.15.107:0/3395618078 conn(0x7ff08c0abd60 msgr2=0x7ff08c0ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- 172.21.15.107:0/3395618078 shutdown_connections 2023-10-24T23:53:47.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- 172.21.15.107:0/3395618078 wait complete. 2023-10-24T23:53:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 Processor -- start 2023-10-24T23:53:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- start start 2023-10-24T23:53:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 0x7ff08c152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff08c157a30 0x7ff08c0ad850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff08c157390 0x7ff08c0add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c158590 con 0x7ff078005390 2023-10-24T23:53:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0acfc0 con 0x7ff08c157390 2023-10-24T23:53:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0ad260 con 0x7ff08c157a30 2023-10-24T23:53:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff08b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 0x7ff08c152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff08b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 0x7ff08c152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37080/0 (socket says 172.21.15.107:37080) 2023-10-24T23:53:47.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff08b7fe640 1 -- 172.21.15.107:0/1217217204 learned_addr learned my addr 172.21.15.107:0/1217217204 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:47.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff08b7fe640 1 -- 172.21.15.107:0/1217217204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff08c157a30 msgr2=0x7ff08c0ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff08b7fe640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff08c157a30 0x7ff08c0ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff08b7fe640 1 -- 172.21.15.107:0/1217217204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff08c157390 msgr2=0x7ff08c0add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff08b7fe640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff08c157390 0x7ff08c0add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff08b7fe640 1 -- 172.21.15.107:0/1217217204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c1587e0 con 0x7ff078005390 2023-10-24T23:53:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff08b7fe640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 0x7ff08c152770 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff07c00e870 tx=0x7ff07c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff088ff9640 1 -- 172.21.15.107:0/1217217204 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c00ee60 con 0x7ff078005390 2023-10-24T23:53:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff088ff9640 1 -- 172.21.15.107:0/1217217204 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c00a920 con 0x7ff078005390 2023-10-24T23:53:47.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- 172.21.15.107:0/1217217204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c158ac0 con 0x7ff078005390 2023-10-24T23:53:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff088ff9640 1 -- 172.21.15.107:0/1217217204 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c00acd0 con 0x7ff078005390 2023-10-24T23:53:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- 172.21.15.107:0/1217217204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c0c3e80 con 0x7ff078005390 2023-10-24T23:53:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.629+0000 7ff092093640 1 -- 172.21.15.107:0/1217217204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08c152e80 con 0x7ff078005390 2023-10-24T23:53:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.633+0000 7ff088ff9640 1 -- 172.21.15.107:0/1217217204 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff07c015480 con 0x7ff078005390 2023-10-24T23:53:47.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.633+0000 7ff088ff9640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff06c065e60 0x7ff06c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.633+0000 7ff088ff9640 1 -- 172.21.15.107:0/1217217204 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7ff07c0846f0 con 0x7ff078005390 2023-10-24T23:53:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.633+0000 7ff082ffd640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff06c065e60 0x7ff06c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.633+0000 7ff082ffd640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff06c065e60 0x7ff06c068320 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff074004880 tx=0x7ff074000ee0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.637+0000 7ff088ff9640 1 -- 172.21.15.107:0/1217217204 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff07c051a40 con 0x7ff078005390 2023-10-24T23:53:47.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.785+0000 7ff092093640 1 -- 172.21.15.107:0/1217217204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff08c0746e0 con 0x7ff078005390 2023-10-24T23:53:47.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.789+0000 7ff088ff9640 1 -- 172.21.15.107:0/1217217204 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+21618 (secure 0 0 0) 0x7ff07c055060 con 0x7ff078005390 2023-10-24T23:53:47.795 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:47.796 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:44.462555+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2f","mappings":[{"from":3,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":2}]},{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:47.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.789+0000 7ff092093640 1 -- 172.21.15.107:0/1217217204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff06c065e60 msgr2=0x7ff06c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.789+0000 7ff092093640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff06c065e60 0x7ff06c068320 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff074004880 tx=0x7ff074000ee0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.789+0000 7ff092093640 1 -- 172.21.15.107:0/1217217204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 msgr2=0x7ff08c152770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.789+0000 7ff092093640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 0x7ff08c152770 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff07c00e870 tx=0x7ff07c00ed30 comp rx=0 tx=0).stop 2023-10-24T23:53:47.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.793+0000 7ff092093640 1 -- 172.21.15.107:0/1217217204 shutdown_connections 2023-10-24T23:53:47.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.793+0000 7ff092093640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff08c157390 0x7ff08c0add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.793+0000 7ff092093640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff06c065e60 0x7ff06c068320 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.793+0000 7ff092093640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff08c157a30 0x7ff08c0ad850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.793+0000 7ff092093640 1 --2- 172.21.15.107:0/1217217204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff078005390 0x7ff08c152770 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.793+0000 7ff092093640 1 -- 172.21.15.107:0/1217217204 >> 172.21.15.107:0/1217217204 conn(0x7ff08c0abd60 msgr2=0x7ff08c0c9350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:47.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.793+0000 7ff092093640 1 -- 172.21.15.107:0/1217217204 shutdown_connections 2023-10-24T23:53:47.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.793+0000 7ff092093640 1 -- 172.21.15.107:0/1217217204 wait complete. 2023-10-24T23:53:47.812 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-10-24T23:53:09.786122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-10-24T23:53:10.766645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-24T23:53:47.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-24T23:53:47.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.929+0000 7f0bd847e640 1 Processor -- start 2023-10-24T23:53:47.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.929+0000 7f0bd847e640 1 -- start start 2023-10-24T23:53:47.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd847e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0bd00a50d0 0x7f0bd00aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd847e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00bdad0 con 0x7f0bd00c4cf0 2023-10-24T23:53:47.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd847e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00a9cf0 con 0x7f0bd00c1c30 2023-10-24T23:53:47.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd847e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd00b97d0 con 0x7f0bd00a50d0 2023-10-24T23:53:47.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd621a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0bd00a50d0 0x7f0bd00aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd6a1b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0bd00c4cf0 0x7f0bd01636d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45180/0 (socket says 172.21.15.107:45180) 2023-10-24T23:53:47.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd6a1b640 1 -- 172.21.15.107:0/3336918029 learned_addr learned my addr 172.21.15.107:0/3336918029 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:47.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd621a640 1 -- 172.21.15.107:0/3336918029 >> v1:172.21.15.110:6789/0 conn(0x7f0bd00c1c30 legacy=0x7f0bd00bd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd621a640 1 -- 172.21.15.107:0/3336918029 >> v1:172.21.15.107:6789/0 conn(0x7f0bd00c4cf0 legacy=0x7f0bd01636d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd5218640 1 -- 172.21.15.107:0/3336918029 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760502086 0 0) 0x7f0bd00bdad0 con 0x7f0bd00c4cf0 2023-10-24T23:53:47.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd621a640 1 -- 172.21.15.107:0/3336918029 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd00a9fd0 con 0x7f0bd00a50d0 2023-10-24T23:53:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd621a640 1 --2- 172.21.15.107:0/3336918029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd00a50d0 0x7f0bd00aaab0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc0097c0 tx=0x7f0bcc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce13a0b99d05157 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd5218640 1 -- 172.21.15.107:0/3336918029 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc0ab600 con 0x7f0bd00a50d0 2023-10-24T23:53:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd5218640 1 -- 172.21.15.107:0/3336918029 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc0a7050 con 0x7f0bd00a50d0 2023-10-24T23:53:47.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.933+0000 7f0bd5218640 1 -- 172.21.15.107:0/3336918029 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc0ab980 con 0x7f0bd00a50d0 2023-10-24T23:53:47.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 -- 172.21.15.107:0/3336918029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd00a50d0 msgr2=0x7f0bd00aaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 --2- 172.21.15.107:0/3336918029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd00a50d0 0x7f0bd00aaab0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc0097c0 tx=0x7f0bcc01c480 comp rx=0 tx=0).stop 2023-10-24T23:53:47.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 -- 172.21.15.107:0/3336918029 shutdown_connections 2023-10-24T23:53:47.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 --2- 172.21.15.107:0/3336918029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd00a50d0 0x7f0bd00aaab0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 -- 172.21.15.107:0/3336918029 >> 172.21.15.107:0/3336918029 conn(0x7f0bd00c2860 msgr2=0x7f0bd00c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:47.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 -- 172.21.15.107:0/3336918029 shutdown_connections 2023-10-24T23:53:47.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 -- 172.21.15.107:0/3336918029 wait complete. 2023-10-24T23:53:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 Processor -- start 2023-10-24T23:53:47.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 -- start start 2023-10-24T23:53:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd00a50d0 0x7f0bd00c0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0bd00c1c30 0x7f0bd0154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0bd00c4cf0 0x7f0bd0154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd00a8fd0 con 0x7f0bd00a50d0 2023-10-24T23:53:47.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd0154240 con 0x7f0bd00c1c30 2023-10-24T23:53:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd015bd20 con 0x7f0bd00c4cf0 2023-10-24T23:53:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd621a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd00a50d0 0x7f0bd00c0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd621a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd00a50d0 0x7f0bd00c0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37088/0 (socket says 172.21.15.107:37088) 2023-10-24T23:53:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd621a640 1 -- 172.21.15.107:0/4144105690 learned_addr learned my addr 172.21.15.107:0/4144105690 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:47.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd6a1b640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0bd00c4cf0 0x7f0bd0154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd5a19640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0bd00c1c30 0x7f0bd0154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd621a640 1 -- 172.21.15.107:0/4144105690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0bd00c4cf0 msgr2=0x7f0bd0154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd621a640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0bd00c4cf0 0x7f0bd0154ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd621a640 1 -- 172.21.15.107:0/4144105690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0bd00c1c30 msgr2=0x7f0bd0154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:47.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd621a640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0bd00c1c30 0x7f0bd0154660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:47.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd621a640 1 -- 172.21.15.107:0/4144105690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc009420 con 0x7f0bd00a50d0 2023-10-24T23:53:47.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd621a640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd00a50d0 0x7f0bd00c0e60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc019090 tx=0x7f0bcc01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:47.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bc6ffd640 1 -- 172.21.15.107:0/4144105690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc0b7650 con 0x7f0bd00a50d0 2023-10-24T23:53:47.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bc6ffd640 1 -- 172.21.15.107:0/4144105690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc0abe90 con 0x7f0bd00a50d0 2023-10-24T23:53:47.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bc6ffd640 1 -- 172.21.15.107:0/4144105690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc0b7a10 con 0x7f0bd00a50d0 2023-10-24T23:53:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 -- 172.21.15.107:0/4144105690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd00b8e80 con 0x7f0bd00a50d0 2023-10-24T23:53:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.937+0000 7f0bd847e640 1 -- 172.21.15.107:0/4144105690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd015ccd0 con 0x7f0bd00a50d0 2023-10-24T23:53:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.941+0000 7f0bd847e640 1 -- 172.21.15.107:0/4144105690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd00bcf50 con 0x7f0bd00a50d0 2023-10-24T23:53:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.941+0000 7f0bc6ffd640 1 -- 172.21.15.107:0/4144105690 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0bcc0b7b70 con 0x7f0bd00a50d0 2023-10-24T23:53:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.941+0000 7f0bc6ffd640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0bb4065e60 0x7f0bb4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.941+0000 7f0bc6ffd640 1 -- 172.21.15.107:0/4144105690 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f0bcc11e4f0 con 0x7f0bd00a50d0 2023-10-24T23:53:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.941+0000 7f0bd5a19640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0bb4065e60 0x7f0bb4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.941+0000 7f0bd5a19640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0bb4065e60 0x7f0bb4068320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0000f70 tx=0x7f0bc0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:47.945+0000 7f0bc6ffd640 1 -- 172.21.15.107:0/4144105690 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0bcc0eb960 con 0x7f0bd00a50d0 2023-10-24T23:53:48.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.085+0000 7f0bd847e640 1 -- 172.21.15.107:0/4144105690 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0bd004e130 con 0x7f0bd00a50d0 2023-10-24T23:53:48.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bc6ffd640 1 -- 172.21.15.107:0/4144105690 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0bcc0eef80 con 0x7f0bd00a50d0 2023-10-24T23:53:48.093 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-24T23:53:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 -- 172.21.15.107:0/4144105690 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0bb4065e60 msgr2=0x7f0bb4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0bb4065e60 0x7f0bb4068320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0000f70 tx=0x7f0bc0009040 comp rx=0 tx=0).stop 2023-10-24T23:53:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 -- 172.21.15.107:0/4144105690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd00a50d0 msgr2=0x7f0bd00c0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd00a50d0 0x7f0bd00c0e60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc019090 tx=0x7f0bcc01acc0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 -- 172.21.15.107:0/4144105690 shutdown_connections 2023-10-24T23:53:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0bb4065e60 0x7f0bb4068320 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0bd00c4cf0 0x7f0bd0154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0bd00c1c30 0x7f0bd0154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 --2- 172.21.15.107:0/4144105690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0bd00a50d0 0x7f0bd00c0e60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 -- 172.21.15.107:0/4144105690 >> 172.21.15.107:0/4144105690 conn(0x7f0bd00c2860 msgr2=0x7f0bd0159740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:48.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 -- 172.21.15.107:0/4144105690 shutdown_connections 2023-10-24T23:53:48.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.089+0000 7f0bd847e640 1 -- 172.21.15.107:0/4144105690 wait complete. 2023-10-24T23:53:48.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-24T23:53:48.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc20e627640 1 Processor -- start 2023-10-24T23:53:48.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc20e627640 1 -- start start 2023-10-24T23:53:48.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc20e627640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc208155cc0 0x7fc20806d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc20e627640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc208150360 con 0x7fc2080c7590 2023-10-24T23:53:48.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc20e627640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2080c7e20 con 0x7fc208071ee0 2023-10-24T23:53:48.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc20e627640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc208156c90 con 0x7fc208155cc0 2023-10-24T23:53:48.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc207fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc208155cc0 0x7fc20806d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc20cbc4640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc2080c7590 0x7fc2080a9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45188/0 (socket says 172.21.15.107:45188) 2023-10-24T23:53:48.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc20cbc4640 1 -- 172.21.15.107:0/3201966991 learned_addr learned my addr 172.21.15.107:0/3201966991 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:48.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc207fff640 1 -- 172.21.15.107:0/3201966991 >> v1:172.21.15.110:6789/0 conn(0x7fc208071ee0 legacy=0x7fc2081560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:48.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc207fff640 1 -- 172.21.15.107:0/3201966991 >> v1:172.21.15.107:6789/0 conn(0x7fc2080c7590 legacy=0x7fc2080a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc206ffd640 1 -- 172.21.15.107:0/3201966991 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971476188 0 0) 0x7fc208150360 con 0x7fc2080c7590 2023-10-24T23:53:48.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.229+0000 7fc207fff640 1 -- 172.21.15.107:0/3201966991 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20806ec00 con 0x7fc208155cc0 2023-10-24T23:53:48.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.233+0000 7fc207fff640 1 --2- 172.21.15.107:0/3201966991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc208155cc0 0x7fc20806d0d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80097c0 tx=0x7fc1f801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ecd27f9a2d7824 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:48.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.233+0000 7fc206ffd640 1 -- 172.21.15.107:0/3201966991 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f80ab600 con 0x7fc208155cc0 2023-10-24T23:53:48.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.233+0000 7fc206ffd640 1 -- 172.21.15.107:0/3201966991 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f80a7050 con 0x7fc208155cc0 2023-10-24T23:53:48.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.233+0000 7fc206ffd640 1 -- 172.21.15.107:0/3201966991 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f80ab980 con 0x7fc208155cc0 2023-10-24T23:53:48.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.233+0000 7fc20e627640 1 -- 172.21.15.107:0/3201966991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc208155cc0 msgr2=0x7fc20806d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.233+0000 7fc20e627640 1 --2- 172.21.15.107:0/3201966991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc208155cc0 0x7fc20806d0d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80097c0 tx=0x7fc1f801c360 comp rx=0 tx=0).stop 2023-10-24T23:53:48.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.233+0000 7fc20e627640 1 -- 172.21.15.107:0/3201966991 shutdown_connections 2023-10-24T23:53:48.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.233+0000 7fc20e627640 1 --2- 172.21.15.107:0/3201966991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc208155cc0 0x7fc20806d0d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.233+0000 7fc20e627640 1 -- 172.21.15.107:0/3201966991 >> 172.21.15.107:0/3201966991 conn(0x7fc2080c8020 msgr2=0x7fc2080c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:48.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.233+0000 7fc20e627640 1 -- 172.21.15.107:0/3201966991 shutdown_connections 2023-10-24T23:53:48.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.233+0000 7fc20e627640 1 -- 172.21.15.107:0/3201966991 wait complete. 2023-10-24T23:53:48.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20e627640 1 Processor -- start 2023-10-24T23:53:48.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20e627640 1 -- start start 2023-10-24T23:53:48.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20e627640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc208071ee0 0x7fc208142a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20e627640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2080c7590 0x7fc20813d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20e627640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc208155cc0 0x7fc20814e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20e627640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20814fa90 con 0x7fc208155cc0 2023-10-24T23:53:48.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20e627640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2080c33b0 con 0x7fc2080c7590 2023-10-24T23:53:48.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20e627640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2080c2d80 con 0x7fc208071ee0 2023-10-24T23:53:48.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20cbc4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc208155cc0 0x7fc20814e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20cbc4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc208155cc0 0x7fc20814e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37104/0 (socket says 172.21.15.107:37104) 2023-10-24T23:53:48.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20cbc4640 1 -- 172.21.15.107:0/1343465972 learned_addr learned my addr 172.21.15.107:0/1343465972 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:48.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc207fff640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc208071ee0 0x7fc208142a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc2077fe640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2080c7590 0x7fc20813d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20cbc4640 1 -- 172.21.15.107:0/1343465972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc208071ee0 msgr2=0x7fc208142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20cbc4640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc208071ee0 0x7fc208142a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20cbc4640 1 -- 172.21.15.107:0/1343465972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2080c7590 msgr2=0x7fc20813d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20cbc4640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2080c7590 0x7fc20813d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20cbc4640 1 -- 172.21.15.107:0/1343465972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f8009420 con 0x7fc208155cc0 2023-10-24T23:53:48.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20cbc4640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc208155cc0 0x7fc20814e9d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc20000c7b0 tx=0x7fc20000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:48.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc204ff9640 1 -- 172.21.15.107:0/1343465972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20000a5e0 con 0x7fc208155cc0 2023-10-24T23:53:48.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc204ff9640 1 -- 172.21.15.107:0/1343465972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20000a760 con 0x7fc208155cc0 2023-10-24T23:53:48.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20e627640 1 -- 172.21.15.107:0/1343465972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2080c41c0 con 0x7fc208155cc0 2023-10-24T23:53:48.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc204ff9640 1 -- 172.21.15.107:0/1343465972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20000ab10 con 0x7fc208155cc0 2023-10-24T23:53:48.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20e627640 1 -- 172.21.15.107:0/1343465972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2081ebb30 con 0x7fc208155cc0 2023-10-24T23:53:48.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.237+0000 7fc20e627640 1 -- 172.21.15.107:0/1343465972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2080c43c0 con 0x7fc208155cc0 2023-10-24T23:53:48.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.241+0000 7fc204ff9640 1 -- 172.21.15.107:0/1343465972 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc20000ac70 con 0x7fc208155cc0 2023-10-24T23:53:48.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.241+0000 7fc204ff9640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1f4065e60 0x7fc1f4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.241+0000 7fc204ff9640 1 -- 172.21.15.107:0/1343465972 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7fc200082ec0 con 0x7fc208155cc0 2023-10-24T23:53:48.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.241+0000 7fc207fff640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1f4065e60 0x7fc1f4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.241+0000 7fc207fff640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1f4065e60 0x7fc1f4068320 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80ad000 tx=0x7fc1f80a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:48.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.245+0000 7fc204ff9640 1 -- 172.21.15.107:0/1343465972 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc2080c43c0 con 0x7fc208155cc0 2023-10-24T23:53:48.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 -- 172.21.15.107:0/1343465972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc2080c43c0 con 0x7fc208155cc0 2023-10-24T23:53:48.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc204ff9640 1 -- 172.21.15.107:0/1343465972 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc2080c43c0 con 0x7fc208155cc0 2023-10-24T23:53:48.394 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-24T23:53:48.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 -- 172.21.15.107:0/1343465972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1f4065e60 msgr2=0x7fc1f4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1f4065e60 0x7fc1f4068320 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80ad000 tx=0x7fc1f80a6000 comp rx=0 tx=0).stop 2023-10-24T23:53:48.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 -- 172.21.15.107:0/1343465972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc208155cc0 msgr2=0x7fc20814e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc208155cc0 0x7fc20814e9d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc20000c7b0 tx=0x7fc20000cc70 comp rx=0 tx=0).stop 2023-10-24T23:53:48.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 -- 172.21.15.107:0/1343465972 shutdown_connections 2023-10-24T23:53:48.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc208155cc0 0x7fc20814e9d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2080c7590 0x7fc20813d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1f4065e60 0x7fc1f4068320 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 --2- 172.21.15.107:0/1343465972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc208071ee0 0x7fc208142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 -- 172.21.15.107:0/1343465972 >> 172.21.15.107:0/1343465972 conn(0x7fc2080c8020 msgr2=0x7fc20806d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:48.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 -- 172.21.15.107:0/1343465972 shutdown_connections 2023-10-24T23:53:48.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.389+0000 7fc20e627640 1 -- 172.21.15.107:0/1343465972 wait complete. 2023-10-24T23:53:48.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-24T23:53:48.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.529+0000 7f44a26e8640 1 Processor -- start 2023-10-24T23:53:48.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.529+0000 7f44a26e8640 1 -- start start 2023-10-24T23:53:48.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f44a26e8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f449c0ab1a0 0x7f449c0c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f44a26e8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f449c072720 0x7f449c0bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f44a26e8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c0c1ce0 con 0x7f449c0bceb0 2023-10-24T23:53:48.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f44a26e8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449c0be1d0 con 0x7f449c072720 2023-10-24T23:53:48.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f44a26e8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449c0c0880 con 0x7f449c0ab1a0 2023-10-24T23:53:48.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f449b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f449c072720 0x7f449c0bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f449bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f449c0ab1a0 0x7f449c0c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f449b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f449c072720 0x7f449c0bfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:53956/0 (socket says 172.21.15.107:53956) 2023-10-24T23:53:48.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f449b7fe640 1 -- 172.21.15.107:0/1764932492 learned_addr learned my addr 172.21.15.107:0/1764932492 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:48.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f44a0c85640 1 --1- 172.21.15.107:0/1764932492 >> v1:172.21.15.110:6789/0 conn(0x7f449c0bceb0 0x7f449c0c4cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:46466/0 (socket says 172.21.15.107:46466) 2023-10-24T23:53:48.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f449bfff640 1 -- 172.21.15.107:0/1764932492 >> v2:172.21.15.110:3300/0 conn(0x7f449c072720 msgr2=0x7f449c0bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:48.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f449bfff640 1 --2- 172.21.15.107:0/1764932492 >> v2:172.21.15.110:3300/0 conn(0x7f449c072720 0x7f449c0bfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f449bfff640 1 -- 172.21.15.107:0/1764932492 >> v1:172.21.15.110:6789/0 conn(0x7f449c0bceb0 legacy=0x7f449c0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f449bfff640 1 -- 172.21.15.107:0/1764932492 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c0c1f30 con 0x7f449c0ab1a0 2023-10-24T23:53:48.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f449b7fe640 1 --2- 172.21.15.107:0/1764932492 >> v2:172.21.15.110:3300/0 conn(0x7f449c072720 0x7f449c0bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:53:48.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f449bfff640 1 --2- 172.21.15.107:0/1764932492 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f449c0ab1a0 0x7f449c0c0d00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f44840097c0 tx=0x7f448401c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=345ee95db4db4831 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:48.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.533+0000 7f449affd640 1 -- 172.21.15.107:0/1764932492 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44840ab600 con 0x7f449c0ab1a0 2023-10-24T23:53:48.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f449affd640 1 -- 172.21.15.107:0/1764932492 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44840a7050 con 0x7f449c0ab1a0 2023-10-24T23:53:48.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f449affd640 1 -- 172.21.15.107:0/1764932492 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44840ab980 con 0x7f449c0ab1a0 2023-10-24T23:53:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 -- 172.21.15.107:0/1764932492 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f449c0ab1a0 msgr2=0x7f449c0c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 --2- 172.21.15.107:0/1764932492 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f449c0ab1a0 0x7f449c0c0d00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f44840097c0 tx=0x7f448401c330 comp rx=0 tx=0).stop 2023-10-24T23:53:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 -- 172.21.15.107:0/1764932492 shutdown_connections 2023-10-24T23:53:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 --2- 172.21.15.107:0/1764932492 >> v2:172.21.15.110:3300/0 conn(0x7f449c072720 0x7f449c0bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 --2- 172.21.15.107:0/1764932492 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f449c0ab1a0 0x7f449c0c0d00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 -- 172.21.15.107:0/1764932492 >> 172.21.15.107:0/1764932492 conn(0x7f449c0c2860 msgr2=0x7f449c0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:48.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 -- 172.21.15.107:0/1764932492 shutdown_connections 2023-10-24T23:53:48.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 -- 172.21.15.107:0/1764932492 wait complete. 2023-10-24T23:53:48.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 Processor -- start 2023-10-24T23:53:48.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 -- start start 2023-10-24T23:53:48.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f449c0ab1a0 0x7f449c154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f449c0bceb0 0x7f449c15c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f449c15c5a0 0x7f449c06c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c06c0b0 con 0x7f449c15c5a0 2023-10-24T23:53:48.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c06bc20 con 0x7f449c0bceb0 2023-10-24T23:53:48.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449c0a71f0 con 0x7f449c0ab1a0 2023-10-24T23:53:48.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a0c85640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f449c15c5a0 0x7f449c06c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a0c85640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f449c15c5a0 0x7f449c06c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37116/0 (socket says 172.21.15.107:37116) 2023-10-24T23:53:48.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a0c85640 1 -- 172.21.15.107:0/3632830225 learned_addr learned my addr 172.21.15.107:0/3632830225 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:48.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f449b7fe640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f449c0bceb0 0x7f449c15c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f449bfff640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f449c0ab1a0 0x7f449c154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a0c85640 1 -- 172.21.15.107:0/3632830225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f449c0ab1a0 msgr2=0x7f449c154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a0c85640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f449c0ab1a0 0x7f449c154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a0c85640 1 -- 172.21.15.107:0/3632830225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f449c0bceb0 msgr2=0x7f449c15c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a0c85640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f449c0bceb0 0x7f449c15c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a0c85640 1 -- 172.21.15.107:0/3632830225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c0094d0 con 0x7f449c15c5a0 2023-10-24T23:53:48.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a0c85640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f449c15c5a0 0x7f449c06c440 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f449400c700 tx=0x7f449400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:48.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f4498ff9640 1 -- 172.21.15.107:0/3632830225 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449400ce10 con 0x7f449c15c5a0 2023-10-24T23:53:48.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f4498ff9640 1 -- 172.21.15.107:0/3632830225 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449400a640 con 0x7f449c15c5a0 2023-10-24T23:53:48.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 -- 172.21.15.107:0/3632830225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4484009420 con 0x7f449c15c5a0 2023-10-24T23:53:48.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f4498ff9640 1 -- 172.21.15.107:0/3632830225 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449400a9b0 con 0x7f449c15c5a0 2023-10-24T23:53:48.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.537+0000 7f44a26e8640 1 -- 172.21.15.107:0/3632830225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449c155550 con 0x7f449c15c5a0 2023-10-24T23:53:48.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.541+0000 7f44a26e8640 1 -- 172.21.15.107:0/3632830225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449c06bdb0 con 0x7f449c15c5a0 2023-10-24T23:53:48.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.541+0000 7f4498ff9640 1 -- 172.21.15.107:0/3632830225 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f449400ab10 con 0x7f449c15c5a0 2023-10-24T23:53:48.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.541+0000 7f4498ff9640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f447c065e90 0x7f447c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.541+0000 7f4498ff9640 1 -- 172.21.15.107:0/3632830225 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f4494083300 con 0x7f449c15c5a0 2023-10-24T23:53:48.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.541+0000 7f449bfff640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f447c065e90 0x7f447c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.541+0000 7f449bfff640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f447c065e90 0x7f447c068350 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f44840ad000 tx=0x7f44840a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:48.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.545+0000 7f4498ff9640 1 -- 172.21.15.107:0/3632830225 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4494054100 con 0x7f449c15c5a0 2023-10-24T23:53:48.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.697+0000 7f44a26e8640 1 -- 172.21.15.107:0/3632830225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f449c0beff0 con 0x7f449c15c5a0 2023-10-24T23:53:48.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.697+0000 7f4498ff9640 1 -- 172.21.15.107:0/3632830225 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v29) v1 ==== 99+0+11 (secure 0 0 0) 0x7f4494053c90 con 0x7f449c15c5a0 2023-10-24T23:53:48.703 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:48.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 -- 172.21.15.107:0/3632830225 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f447c065e90 msgr2=0x7f447c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f447c065e90 0x7f447c068350 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f44840ad000 tx=0x7f44840a6000 comp rx=0 tx=0).stop 2023-10-24T23:53:48.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 -- 172.21.15.107:0/3632830225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f449c15c5a0 msgr2=0x7f449c06c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f449c15c5a0 0x7f449c06c440 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f449400c700 tx=0x7f449400cbc0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 -- 172.21.15.107:0/3632830225 shutdown_connections 2023-10-24T23:53:48.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f449c15c5a0 0x7f449c06c440 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f449c0bceb0 0x7f449c15c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f447c065e90 0x7f447c068350 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 --2- 172.21.15.107:0/3632830225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f449c0ab1a0 0x7f449c154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 -- 172.21.15.107:0/3632830225 >> 172.21.15.107:0/3632830225 conn(0x7f449c0c2860 msgr2=0x7f449c159c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:48.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 -- 172.21.15.107:0/3632830225 shutdown_connections 2023-10-24T23:53:48.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.701+0000 7f44a26e8640 1 -- 172.21.15.107:0/3632830225 wait complete. 2023-10-24T23:53:48.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-24T23:53:48.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f234a850640 1 Processor -- start 2023-10-24T23:53:48.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f234a850640 1 -- start start 2023-10-24T23:53:48.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f234a850640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23440af9b0 0x7f23440c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f234a850640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23440c8180 con 0x7f234406b360 2023-10-24T23:53:48.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f234a850640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2344155430 con 0x7f23440bd930 2023-10-24T23:53:48.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f234a850640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23440c5870 con 0x7f23440af9b0 2023-10-24T23:53:48.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f23437fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f234406b360 0x7f2344069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45198/0 (socket says 172.21.15.107:45198) 2023-10-24T23:53:48.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f23437fe640 1 -- 172.21.15.107:0/74821280 learned_addr learned my addr 172.21.15.107:0/74821280 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:48.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2343fff640 1 --2- 172.21.15.107:0/74821280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23440af9b0 0x7f23440c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377412402 0 0) 0x7f23440c8180 con 0x7f234406b360 2023-10-24T23:53:48.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f232c003530 con 0x7f234406b360 2023-10-24T23:53:48.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1559171050 0 0) 0x7f232c003530 con 0x7f234406b360 2023-10-24T23:53:48.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23440af9b0 msgr2=0x7f23440c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:48.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 --2- 172.21.15.107:0/74821280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23440af9b0 0x7f23440c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 >> v1:172.21.15.110:6789/0 conn(0x7f23440bd930 legacy=0x7f234415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2344155710 con 0x7f234406b360 2023-10-24T23:53:48.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f2338001680 con 0x7f234406b360 2023-10-24T23:53:48.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 >> v1:172.21.15.107:6789/0 conn(0x7f234406b360 legacy=0x7f2344069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 --2- 172.21.15.107:0/74821280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f232c0039b0 0x7f232c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 --2- 172.21.15.107:0/74821280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f232c0047d0 0x7f232c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 --2- 172.21.15.107:0/74821280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23440af9b0 0x7f232c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23440c5870 con 0x7f232c0039b0 2023-10-24T23:53:48.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c00bac0 con 0x7f232c0047d0 2023-10-24T23:53:48.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232c00bc80 con 0x7f23440af9b0 2023-10-24T23:53:48.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2343fff640 1 --2- 172.21.15.107:0/74821280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f232c0039b0 0x7f232c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f23437fe640 1 --2- 172.21.15.107:0/74821280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f232c0047d0 0x7f232c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2343fff640 1 -- 172.21.15.107:0/74821280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23440af9b0 msgr2=0x7f232c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:48.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2343fff640 1 --2- 172.21.15.107:0/74821280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23440af9b0 0x7f232c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2343fff640 1 -- 172.21.15.107:0/74821280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f232c0047d0 msgr2=0x7f232c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:48.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2343fff640 1 --2- 172.21.15.107:0/74821280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f232c0047d0 0x7f232c004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.833+0000 7f2343fff640 1 -- 172.21.15.107:0/74821280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c003530 con 0x7f232c0039b0 2023-10-24T23:53:48.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f2343fff640 1 --2- 172.21.15.107:0/74821280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f232c0039b0 0x7f232c003dd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2334009ae0 tx=0x7f233401ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ef7e165cc05a77 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:48.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23340a55c0 con 0x7f232c0039b0 2023-10-24T23:53:48.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23340a5720 con 0x7f232c0039b0 2023-10-24T23:53:48.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f2342ffd640 1 -- 172.21.15.107:0/74821280 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23340a58d0 con 0x7f232c0039b0 2023-10-24T23:53:48.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 -- 172.21.15.107:0/74821280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f232c0039b0 msgr2=0x7f232c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 --2- 172.21.15.107:0/74821280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f232c0039b0 0x7f232c003dd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2334009ae0 tx=0x7f233401ca60 comp rx=0 tx=0).stop 2023-10-24T23:53:48.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f2348ded640 1 -- 172.21.15.107:0/74821280 reap_dead start 2023-10-24T23:53:48.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 -- 172.21.15.107:0/74821280 shutdown_connections 2023-10-24T23:53:48.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 -- 172.21.15.107:0/74821280 >> 172.21.15.107:0/74821280 conn(0x7f2344071b00 msgr2=0x7f23440abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:48.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 -- 172.21.15.107:0/74821280 shutdown_connections 2023-10-24T23:53:48.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 -- 172.21.15.107:0/74821280 wait complete. 2023-10-24T23:53:48.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 Processor -- start 2023-10-24T23:53:48.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 -- start start 2023-10-24T23:53:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f232c0039b0 0x7f23440c1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23440c3060 0x7f23441587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23440ab0a0 0x7f234414c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2344156570 con 0x7f23440ab0a0 2023-10-24T23:53:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23440c1890 con 0x7f23440c3060 2023-10-24T23:53:48.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f234a850640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23441582d0 con 0x7f232c0039b0 2023-10-24T23:53:48.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.837+0000 7f2348ded640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23440ab0a0 0x7f234414c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2348ded640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23440ab0a0 0x7f234414c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37136/0 (socket says 172.21.15.107:37136) 2023-10-24T23:53:48.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2348ded640 1 -- 172.21.15.107:0/400507994 learned_addr learned my addr 172.21.15.107:0/400507994 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:48.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2343fff640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f232c0039b0 0x7f23440c1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f23437fe640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23440c3060 0x7f23441587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2348ded640 1 -- 172.21.15.107:0/400507994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f232c0039b0 msgr2=0x7f23440c1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2348ded640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f232c0039b0 0x7f23440c1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2348ded640 1 -- 172.21.15.107:0/400507994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23440c3060 msgr2=0x7f23441587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2348ded640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23440c3060 0x7f23441587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2348ded640 1 -- 172.21.15.107:0/400507994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2344155fb0 con 0x7f23440ab0a0 2023-10-24T23:53:48.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2348ded640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23440ab0a0 0x7f234414c540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f232c004750 tx=0x7f233c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:48.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2340ff9640 1 -- 172.21.15.107:0/400507994 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c005d50 con 0x7f23440ab0a0 2023-10-24T23:53:48.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2340ff9640 1 -- 172.21.15.107:0/400507994 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233c00ced0 con 0x7f23440ab0a0 2023-10-24T23:53:48.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2340ff9640 1 -- 172.21.15.107:0/400507994 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c00a850 con 0x7f23440ab0a0 2023-10-24T23:53:48.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f234a850640 1 -- 172.21.15.107:0/400507994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2344156290 con 0x7f23440ab0a0 2023-10-24T23:53:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f234a850640 1 -- 172.21.15.107:0/400507994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2344156af0 con 0x7f23440ab0a0 2023-10-24T23:53:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f234a850640 1 -- 172.21.15.107:0/400507994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2344158130 con 0x7f23440ab0a0 2023-10-24T23:53:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2340ff9640 1 -- 172.21.15.107:0/400507994 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f233c00a9b0 con 0x7f23440ab0a0 2023-10-24T23:53:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2340ff9640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2330065e60 0x7f2330068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2340ff9640 1 -- 172.21.15.107:0/400507994 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f233c082d60 con 0x7f23440ab0a0 2023-10-24T23:53:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2343fff640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2330065e60 0x7f2330068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:48.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.841+0000 7f2343fff640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2330065e60 0x7f2330068320 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f23340af000 tx=0x7f23340046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:48.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.845+0000 7f2340ff9640 1 -- 172.21.15.107:0/400507994 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2344158130 con 0x7f23440ab0a0 2023-10-24T23:53:48.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 -- 172.21.15.107:0/400507994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f2344158130 con 0x7f23440ab0a0 2023-10-24T23:53:48.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f2340ff9640 1 -- 172.21.15.107:0/400507994 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v29) v1 ==== 95+0+11 (secure 0 0 0) 0x7f2344158130 con 0x7f23440ab0a0 2023-10-24T23:53:48.994 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:48.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 -- 172.21.15.107:0/400507994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2330065e60 msgr2=0x7f2330068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2330065e60 0x7f2330068320 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f23340af000 tx=0x7f23340046d0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 -- 172.21.15.107:0/400507994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23440ab0a0 msgr2=0x7f234414c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:48.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23440ab0a0 0x7f234414c540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f232c004750 tx=0x7f233c00cb90 comp rx=0 tx=0).stop 2023-10-24T23:53:48.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 -- 172.21.15.107:0/400507994 shutdown_connections 2023-10-24T23:53:48.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23440ab0a0 0x7f234414c540 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23440c3060 0x7f23441587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2330065e60 0x7f2330068320 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 --2- 172.21.15.107:0/400507994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f232c0039b0 0x7f23440c1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:48.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 -- 172.21.15.107:0/400507994 >> 172.21.15.107:0/400507994 conn(0x7f2344071b00 msgr2=0x7f234414fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:48.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 -- 172.21.15.107:0/400507994 shutdown_connections 2023-10-24T23:53:48.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:48.989+0000 7f234a850640 1 -- 172.21.15.107:0/400507994 wait complete. 2023-10-24T23:53:49.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-10-24T23:53:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.125+0000 7f3a0d150640 1 Processor -- start 2023-10-24T23:53:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.125+0000 7f3a0d150640 1 -- start start 2023-10-24T23:53:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0d150640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3a08072720 0x7f3a080c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0d150640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a080bceb0 0x7f3a0815c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0d150640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a080c1ce0 con 0x7f3a080ab1a0 2023-10-24T23:53:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0d150640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a080be1d0 con 0x7f3a08072720 2023-10-24T23:53:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0d150640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a080c0240 con 0x7f3a080bceb0 2023-10-24T23:53:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3a08072720 0x7f3a080c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3a08072720 0x7f3a080c0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37138/0 (socket says 172.21.15.107:37138) 2023-10-24T23:53:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0659c640 1 -- 172.21.15.107:0/1774605501 learned_addr learned my addr 172.21.15.107:0/1774605501 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0759e640 1 --2- 172.21.15.107:0/1774605501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a080bceb0 0x7f3a0815c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0659c640 1 -- 172.21.15.107:0/1774605501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a080bceb0 msgr2=0x7f3a0815c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0659c640 1 --2- 172.21.15.107:0/1774605501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a080bceb0 0x7f3a0815c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0659c640 1 -- 172.21.15.107:0/1774605501 >> v1:172.21.15.110:6789/0 conn(0x7f3a080ab1a0 legacy=0x7f3a080bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:49.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0659c640 1 -- 172.21.15.107:0/1774605501 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a080c0880 con 0x7f3a08072720 2023-10-24T23:53:49.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a0659c640 1 --2- 172.21.15.107:0/1774605501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a08072720 0x7f3a080c0d00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009870 tx=0x7f39f001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9498112ec4f77dd server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:49.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a05d9b640 1 -- 172.21.15.107:0/1774605501 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f00ab600 con 0x7f3a08072720 2023-10-24T23:53:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a05d9b640 1 -- 172.21.15.107:0/1774605501 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f00a7050 con 0x7f3a08072720 2023-10-24T23:53:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.129+0000 7f3a05d9b640 1 -- 172.21.15.107:0/1774605501 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f00ab980 con 0x7f3a08072720 2023-10-24T23:53:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- 172.21.15.107:0/1774605501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a08072720 msgr2=0x7f3a080c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 --2- 172.21.15.107:0/1774605501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a08072720 0x7f3a080c0d00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f39f0009870 tx=0x7f39f001c3b0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- 172.21.15.107:0/1774605501 shutdown_connections 2023-10-24T23:53:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 --2- 172.21.15.107:0/1774605501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a080bceb0 0x7f3a0815c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 --2- 172.21.15.107:0/1774605501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a08072720 0x7f3a080c0d00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- 172.21.15.107:0/1774605501 >> 172.21.15.107:0/1774605501 conn(0x7f3a080c2860 msgr2=0x7f3a080c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- 172.21.15.107:0/1774605501 shutdown_connections 2023-10-24T23:53:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- 172.21.15.107:0/1774605501 wait complete. 2023-10-24T23:53:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 Processor -- start 2023-10-24T23:53:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- start start 2023-10-24T23:53:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a080ab1a0 0x7f3a08070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a080bceb0 0x7f3a080c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a080bc490 0x7f3a080c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081edbe0 con 0x7f3a080bceb0 2023-10-24T23:53:49.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081524c0 con 0x7f3a080ab1a0 2023-10-24T23:53:49.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081ee6f0 con 0x7f3a080bc490 2023-10-24T23:53:49.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a080bceb0 0x7f3a080c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a080bceb0 0x7f3a080c4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37152/0 (socket says 172.21.15.107:37152) 2023-10-24T23:53:49.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0659c640 1 -- 172.21.15.107:0/2234510033 learned_addr learned my addr 172.21.15.107:0/2234510033 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:49.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0759e640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a080bc490 0x7f3a080c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a06d9d640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a080ab1a0 0x7f3a08070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0659c640 1 -- 172.21.15.107:0/2234510033 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a080bc490 msgr2=0x7f3a080c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0659c640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a080bc490 0x7f3a080c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0659c640 1 -- 172.21.15.107:0/2234510033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a080ab1a0 msgr2=0x7f3a08070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0659c640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a080ab1a0 0x7f3a08070d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0659c640 1 -- 172.21.15.107:0/2234510033 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f00094d0 con 0x7f3a080bceb0 2023-10-24T23:53:49.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a06d9d640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a080ab1a0 0x7f3a08070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:49.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0659c640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a080bceb0 0x7f3a080c4910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f39f00ae040 tx=0x7f39f001abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:49.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f39ff7fe640 1 -- 172.21.15.107:0/2234510033 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f00b6620 con 0x7f3a080bceb0 2023-10-24T23:53:49.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f39ff7fe640 1 -- 172.21.15.107:0/2234510033 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f00abe90 con 0x7f3a080bceb0 2023-10-24T23:53:49.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- 172.21.15.107:0/2234510033 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a081eddc0 con 0x7f3a080bceb0 2023-10-24T23:53:49.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f39ff7fe640 1 -- 172.21.15.107:0/2234510033 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39f00b69e0 con 0x7f3a080bceb0 2023-10-24T23:53:49.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- 172.21.15.107:0/2234510033 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a081ee380 con 0x7f3a080bceb0 2023-10-24T23:53:49.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.133+0000 7f3a0d150640 1 -- 172.21.15.107:0/2234510033 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a080beff0 con 0x7f3a080bceb0 2023-10-24T23:53:49.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.137+0000 7f39ff7fe640 1 -- 172.21.15.107:0/2234510033 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f39f00b2040 con 0x7f3a080bceb0 2023-10-24T23:53:49.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.137+0000 7f39ff7fe640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39e8065e60 0x7f39e8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.137+0000 7f39ff7fe640 1 -- 172.21.15.107:0/2234510033 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f39f011e070 con 0x7f3a080bceb0 2023-10-24T23:53:49.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.137+0000 7f3a06d9d640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39e8065e60 0x7f39e8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.137+0000 7f3a06d9d640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39e8065e60 0x7f39e8068320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f39f8007fd0 tx=0x7f39f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:49.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.141+0000 7f39ff7fe640 1 -- 172.21.15.107:0/2234510033 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f39f00eb3c0 con 0x7f3a080bceb0 2023-10-24T23:53:49.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.293+0000 7f3a0d150640 1 -- 172.21.15.107:0/2234510033 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3a0804e130 con 0x7f3a080bceb0 2023-10-24T23:53:49.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.293+0000 7f39ff7fe640 1 -- 172.21.15.107:0/2234510033 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v29) v1 ==== 99+0+11 (secure 0 0 0) 0x7f39f00ee9e0 con 0x7f3a080bceb0 2023-10-24T23:53:49.301 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:49.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 -- 172.21.15.107:0/2234510033 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39e8065e60 msgr2=0x7f39e8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39e8065e60 0x7f39e8068320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f39f8007fd0 tx=0x7f39f8009040 comp rx=0 tx=0).stop 2023-10-24T23:53:49.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 -- 172.21.15.107:0/2234510033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a080bceb0 msgr2=0x7f3a080c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a080bceb0 0x7f3a080c4910 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f39f00ae040 tx=0x7f39f001abf0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 -- 172.21.15.107:0/2234510033 shutdown_connections 2023-10-24T23:53:49.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39e8065e60 0x7f39e8068320 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a080bc490 0x7f3a080c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a080bceb0 0x7f3a080c4910 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 --2- 172.21.15.107:0/2234510033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a080ab1a0 0x7f3a08070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 -- 172.21.15.107:0/2234510033 >> 172.21.15.107:0/2234510033 conn(0x7f3a080c2860 msgr2=0x7f3a080a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:49.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 -- 172.21.15.107:0/2234510033 shutdown_connections 2023-10-24T23:53:49.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.297+0000 7f3a0d150640 1 -- 172.21.15.107:0/2234510033 wait complete. 2023-10-24T23:53:49.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-10-24T23:53:49.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4d7045640 1 Processor -- start 2023-10-24T23:53:49.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4d7045640 1 -- start start 2023-10-24T23:53:49.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4d7045640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00ab1a0 0x7ff4d00c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4d7045640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4d0072720 0x7ff4d00bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4d7045640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d00c1ce0 con 0x7ff4d00bceb0 2023-10-24T23:53:49.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4d7045640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d00be1d0 con 0x7ff4d0072720 2023-10-24T23:53:49.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4d7045640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d00c0880 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4d0072720 0x7ff4d00bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4cffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4d0072720 0x7ff4d00bfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37162/0 (socket says 172.21.15.107:37162) 2023-10-24T23:53:49.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4cffff640 1 -- 172.21.15.107:0/3247547517 learned_addr learned my addr 172.21.15.107:0/3247547517 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:49.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4d4de1640 1 --2- 172.21.15.107:0/3247547517 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00ab1a0 0x7ff4d00c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4cffff640 1 -- 172.21.15.107:0/3247547517 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00ab1a0 msgr2=0x7ff4d00c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:49.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4cffff640 1 --2- 172.21.15.107:0/3247547517 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00ab1a0 0x7ff4d00c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4cffff640 1 -- 172.21.15.107:0/3247547517 >> v1:172.21.15.110:6789/0 conn(0x7ff4d00bceb0 legacy=0x7ff4d00c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:49.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4cffff640 1 -- 172.21.15.107:0/3247547517 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d00c1f30 con 0x7ff4d0072720 2023-10-24T23:53:49.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.437+0000 7ff4d4de1640 1 --2- 172.21.15.107:0/3247547517 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00ab1a0 0x7ff4d00c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:49.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4cffff640 1 --2- 172.21.15.107:0/3247547517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d0072720 0x7ff4d00bfb30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009870 tx=0x7ff4c401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3140d1e0e82383dc server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:49.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4cf7fe640 1 -- 172.21.15.107:0/3247547517 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c40ab600 con 0x7ff4d0072720 2023-10-24T23:53:49.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4cf7fe640 1 -- 172.21.15.107:0/3247547517 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c40a7050 con 0x7ff4d0072720 2023-10-24T23:53:49.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4cf7fe640 1 -- 172.21.15.107:0/3247547517 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c40ab980 con 0x7ff4d0072720 2023-10-24T23:53:49.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 -- 172.21.15.107:0/3247547517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d0072720 msgr2=0x7ff4d00bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 --2- 172.21.15.107:0/3247547517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d0072720 0x7ff4d00bfb30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009870 tx=0x7ff4c401c3b0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 -- 172.21.15.107:0/3247547517 shutdown_connections 2023-10-24T23:53:49.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 --2- 172.21.15.107:0/3247547517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d0072720 0x7ff4d00bfb30 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 --2- 172.21.15.107:0/3247547517 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00ab1a0 0x7ff4d00c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 -- 172.21.15.107:0/3247547517 >> 172.21.15.107:0/3247547517 conn(0x7ff4d00c2860 msgr2=0x7ff4d00c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:49.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 -- 172.21.15.107:0/3247547517 shutdown_connections 2023-10-24T23:53:49.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 -- 172.21.15.107:0/3247547517 wait complete. 2023-10-24T23:53:49.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 Processor -- start 2023-10-24T23:53:49.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 -- start start 2023-10-24T23:53:49.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d00ab1a0 0x7ff4d01f0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00bceb0 0x7ff4d015cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4d0155850 0x7ff4d0155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d01f10b0 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d006a230 con 0x7ff4d0155850 2023-10-24T23:53:49.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d7045640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d006ec10 con 0x7ff4d00bceb0 2023-10-24T23:53:49.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d4de1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d00ab1a0 0x7ff4d01f0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d4de1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d00ab1a0 0x7ff4d01f0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37170/0 (socket says 172.21.15.107:37170) 2023-10-24T23:53:49.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d4de1640 1 -- 172.21.15.107:0/1718494227 learned_addr learned my addr 172.21.15.107:0/1718494227 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:49.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4cffff640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00bceb0 0x7ff4d015cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d55e2640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4d0155850 0x7ff4d0155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d4de1640 1 -- 172.21.15.107:0/1718494227 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00bceb0 msgr2=0x7ff4d015cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d4de1640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00bceb0 0x7ff4d015cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d4de1640 1 -- 172.21.15.107:0/1718494227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4d0155850 msgr2=0x7ff4d0155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d4de1640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4d0155850 0x7ff4d0155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d4de1640 1 -- 172.21.15.107:0/1718494227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c40094d0 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4cffff640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00bceb0 0x7ff4d015cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:49.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d55e2640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4d0155850 0x7ff4d0155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:49.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.441+0000 7ff4d4de1640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d00ab1a0 0x7ff4d01f0b70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c7c0 tx=0x7ff4bc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:49.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.445+0000 7ff4cd7fa640 1 -- 172.21.15.107:0/1718494227 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc007a30 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.445+0000 7ff4cd7fa640 1 -- 172.21.15.107:0/1718494227 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc007b90 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.445+0000 7ff4d7045640 1 -- 172.21.15.107:0/1718494227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d006eef0 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.445+0000 7ff4cd7fa640 1 -- 172.21.15.107:0/1718494227 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc00ce70 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.445+0000 7ff4d7045640 1 -- 172.21.15.107:0/1718494227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d006b3c0 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.445+0000 7ff4d7045640 1 -- 172.21.15.107:0/1718494227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d006f130 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.445+0000 7ff4cd7fa640 1 -- 172.21.15.107:0/1718494227 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff4bc0113f0 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.445+0000 7ff4cd7fa640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4b4065e60 0x7ff4b4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.445+0000 7ff4cd7fa640 1 -- 172.21.15.107:0/1718494227 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7ff4bc083fc0 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.445+0000 7ff4cffff640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4b4065e60 0x7ff4b4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.445+0000 7ff4cffff640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4b4065e60 0x7ff4b4068320 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40ad000 tx=0x7ff4c40a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:49.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.449+0000 7ff4cd7fa640 1 -- 172.21.15.107:0/1718494227 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff4bc0b5040 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.597+0000 7ff4d7045640 1 -- 172.21.15.107:0/1718494227 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7ff4d00beff0 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.597+0000 7ff4cd7fa640 1 -- 172.21.15.107:0/1718494227 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v29) v1 ==== 95+0+11 (secure 0 0 0) 0x7ff4d006f130 con 0x7ff4d00ab1a0 2023-10-24T23:53:49.604 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:49.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 -- 172.21.15.107:0/1718494227 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4b4065e60 msgr2=0x7ff4b4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4b4065e60 0x7ff4b4068320 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40ad000 tx=0x7ff4c40a6000 comp rx=0 tx=0).stop 2023-10-24T23:53:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 -- 172.21.15.107:0/1718494227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d00ab1a0 msgr2=0x7ff4d01f0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d00ab1a0 0x7ff4d01f0b70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c7c0 tx=0x7ff4bc00cc80 comp rx=0 tx=0).stop 2023-10-24T23:53:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 -- 172.21.15.107:0/1718494227 shutdown_connections 2023-10-24T23:53:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4d0155850 0x7ff4d0155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4b4065e60 0x7ff4b4068320 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4d00bceb0 0x7ff4d015cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 --2- 172.21.15.107:0/1718494227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4d00ab1a0 0x7ff4d01f0b70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 -- 172.21.15.107:0/1718494227 >> 172.21.15.107:0/1718494227 conn(0x7ff4d00c2860 msgr2=0x7ff4d00a62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:49.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 -- 172.21.15.107:0/1718494227 shutdown_connections 2023-10-24T23:53:49.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.601+0000 7ff4d7045640 1 -- 172.21.15.107:0/1718494227 wait complete. 2023-10-24T23:53:49.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:49.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ad1893640 1 Processor -- start 2023-10-24T23:53:49.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ad1893640 1 -- start start 2023-10-24T23:53:49.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ad1893640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2acc151960 0x7f2acc071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ad1893640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2acc0c24c0 0x7f2acc06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ad1893640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2acc150bc0 con 0x7f2acc071220 2023-10-24T23:53:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ad1893640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2acc06c7f0 con 0x7f2acc151960 2023-10-24T23:53:49.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ad1893640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc06c510 con 0x7f2acc0c24c0 2023-10-24T23:53:49.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ac27fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2acc151960 0x7f2acc071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2acaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2acc071220 0x7f2acc06bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45206/0 (socket says 172.21.15.107:45206) 2023-10-24T23:53:49.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2acaffd640 1 -- 172.21.15.107:0/338027230 learned_addr learned my addr 172.21.15.107:0/338027230 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:49.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ac27fc640 1 -- 172.21.15.107:0/338027230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2acc0c24c0 msgr2=0x7f2acc06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:49.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2acb7fe640 1 --2- 172.21.15.107:0/338027230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2acc0c24c0 0x7f2acc06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ac27fc640 1 --2- 172.21.15.107:0/338027230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2acc0c24c0 0x7f2acc06a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2aca7fc640 1 -- 172.21.15.107:0/338027230 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944289237 0 0) 0x7f2acc150bc0 con 0x7f2acc071220 2023-10-24T23:53:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ac27fc640 1 -- 172.21.15.107:0/338027230 >> v1:172.21.15.107:6789/0 conn(0x7f2acc071220 legacy=0x7f2acc06bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2ac27fc640 1 -- 172.21.15.107:0/338027230 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2acc150e80 con 0x7f2acc151960 2023-10-24T23:53:49.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.725+0000 7f2acb7fe640 1 --2- 172.21.15.107:0/338027230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2acc0c24c0 0x7f2acc06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:49.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2ac27fc640 1 --2- 172.21.15.107:0/338027230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2acc151960 0x7f2acc071a90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40097c0 tx=0x7f2ab401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c8a2d319526581 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:49.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2aca7fc640 1 -- 172.21.15.107:0/338027230 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab40ab600 con 0x7f2acc151960 2023-10-24T23:53:49.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2aca7fc640 1 -- 172.21.15.107:0/338027230 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab40a7050 con 0x7f2acc151960 2023-10-24T23:53:49.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2aca7fc640 1 -- 172.21.15.107:0/338027230 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab40ab980 con 0x7f2acc151960 2023-10-24T23:53:49.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2ad1893640 1 -- 172.21.15.107:0/338027230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2acc151960 msgr2=0x7f2acc071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2ad1893640 1 --2- 172.21.15.107:0/338027230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2acc151960 0x7f2acc071a90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40097c0 tx=0x7f2ab401c3f0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2ad1893640 1 -- 172.21.15.107:0/338027230 shutdown_connections 2023-10-24T23:53:49.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2ad1893640 1 --2- 172.21.15.107:0/338027230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2acc0c24c0 0x7f2acc06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2ad1893640 1 --2- 172.21.15.107:0/338027230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2acc151960 0x7f2acc071a90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2ad1893640 1 -- 172.21.15.107:0/338027230 >> 172.21.15.107:0/338027230 conn(0x7f2acc0c2950 msgr2=0x7f2acc0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:49.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2ad1893640 1 -- 172.21.15.107:0/338027230 shutdown_connections 2023-10-24T23:53:49.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2ad1893640 1 -- 172.21.15.107:0/338027230 wait complete. 2023-10-24T23:53:49.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2ad1893640 1 Processor -- start 2023-10-24T23:53:49.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.729+0000 7f2ad1893640 1 -- start start 2023-10-24T23:53:49.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ad1893640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2acc071220 0x7f2acc1483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ad1893640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2acc0c24c0 0x7f2acc151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ad1893640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2acc151960 0x7f2acc152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ad1893640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc152f70 con 0x7f2acc0c24c0 2023-10-24T23:53:49.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ad1893640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc142dd0 con 0x7f2acc151960 2023-10-24T23:53:49.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ad1893640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2acc154890 con 0x7f2acc071220 2023-10-24T23:53:49.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2acc0c24c0 0x7f2acc151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2acc0c24c0 0x7f2acc151f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37184/0 (socket says 172.21.15.107:37184) 2023-10-24T23:53:49.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac27fc640 1 -- 172.21.15.107:0/455213740 learned_addr learned my addr 172.21.15.107:0/455213740 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:49.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2acaffd640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2acc071220 0x7f2acc1483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac27fc640 1 -- 172.21.15.107:0/455213740 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2acc071220 msgr2=0x7f2acc1483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac27fc640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2acc071220 0x7f2acc1483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac27fc640 1 -- 172.21.15.107:0/455213740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2acc151960 msgr2=0x7f2acc152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac27fc640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2acc151960 0x7f2acc152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac27fc640 1 -- 172.21.15.107:0/455213740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4009420 con 0x7f2acc0c24c0 2023-10-24T23:53:49.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac27fc640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2acc0c24c0 0x7f2acc151f00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40097c0 tx=0x7f2ab4019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:49.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac8ff9640 1 -- 172.21.15.107:0/455213740 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab40b4400 con 0x7f2acc0c24c0 2023-10-24T23:53:49.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac8ff9640 1 -- 172.21.15.107:0/455213740 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab40b4560 con 0x7f2acc0c24c0 2023-10-24T23:53:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ad1893640 1 -- 172.21.15.107:0/455213740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2acc1425c0 con 0x7f2acc0c24c0 2023-10-24T23:53:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac8ff9640 1 -- 172.21.15.107:0/455213740 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab40b4910 con 0x7f2acc0c24c0 2023-10-24T23:53:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ad1893640 1 -- 172.21.15.107:0/455213740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc142810 con 0x7f2acc0c24c0 2023-10-24T23:53:49.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ad1893640 1 -- 172.21.15.107:0/455213740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2acc1543c0 con 0x7f2acc0c24c0 2023-10-24T23:53:49.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac8ff9640 1 -- 172.21.15.107:0/455213740 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2ab40b2040 con 0x7f2acc0c24c0 2023-10-24T23:53:49.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac8ff9640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2aac065e60 0x7f2aac068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:49.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2ac8ff9640 1 -- 172.21.15.107:0/455213740 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f2ab411db90 con 0x7f2acc0c24c0 2023-10-24T23:53:49.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2acaffd640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2aac065e60 0x7f2aac068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:49.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.733+0000 7f2acaffd640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2aac065e60 0x7f2aac068320 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2abc000f70 tx=0x7f2abc007b60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:49.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.737+0000 7f2ac8ff9640 1 -- 172.21.15.107:0/455213740 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2acc1543c0 con 0x7f2acc0c24c0 2023-10-24T23:53:49.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.921+0000 7f2ad1893640 1 -- 172.21.15.107:0/455213740 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2acc154a40 con 0x7f2acc0c24c0 2023-10-24T23:53:49.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.921+0000 7f2ac8ff9640 1 -- 172.21.15.107:0/455213740 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f2acc154a40 con 0x7f2acc0c24c0 2023-10-24T23:53:49.927 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:49.928 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:53:49.928 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:53:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 -- 172.21.15.107:0/455213740 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2aac065e60 msgr2=0x7f2aac068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2aac065e60 0x7f2aac068320 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2abc000f70 tx=0x7f2abc007b60 comp rx=0 tx=0).stop 2023-10-24T23:53:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 -- 172.21.15.107:0/455213740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2acc0c24c0 msgr2=0x7f2acc151f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2acc0c24c0 0x7f2acc151f00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40097c0 tx=0x7f2ab4019190 comp rx=0 tx=0).stop 2023-10-24T23:53:49.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 -- 172.21.15.107:0/455213740 shutdown_connections 2023-10-24T23:53:49.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2acc151960 0x7f2acc152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2acc0c24c0 0x7f2acc151f00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2aac065e60 0x7f2aac068320 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 --2- 172.21.15.107:0/455213740 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2acc071220 0x7f2acc1483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 -- 172.21.15.107:0/455213740 >> 172.21.15.107:0/455213740 conn(0x7f2acc0c2950 msgr2=0x7f2acc15bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 -- 172.21.15.107:0/455213740 shutdown_connections 2023-10-24T23:53:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:49.925+0000 7f2ad1893640 1 -- 172.21.15.107:0/455213740 wait complete. 2023-10-24T23:53:49.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:50.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.053+0000 7fcce7f06640 1 Processor -- start 2023-10-24T23:53:50.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.053+0000 7fcce7f06640 1 -- start start 2023-10-24T23:53:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.053+0000 7fcce7f06640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcce006ec90 0x7fcce014c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.053+0000 7fcce7f06640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce014c9e0 0x7fcce00c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.053+0000 7fcce7f06640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce00c59d0 con 0x7fcce0155ea0 2023-10-24T23:53:50.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.053+0000 7fcce7f06640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce00c2990 con 0x7fcce006ec90 2023-10-24T23:53:50.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.053+0000 7fcce7f06640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00c2ec0 con 0x7fcce014c9e0 2023-10-24T23:53:50.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce5ca2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcce0155ea0 0x7fcce0156280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45212/0 (socket says 172.21.15.107:45212) 2023-10-24T23:53:50.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce5ca2640 1 -- 172.21.15.107:0/4283551979 learned_addr learned my addr 172.21.15.107:0/4283551979 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:50.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce54a1640 1 --2- 172.21.15.107:0/4283551979 >> v2:172.21.15.110:3300/0 conn(0x7fcce006ec90 0x7fcce014c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce64a3640 1 --2- 172.21.15.107:0/4283551979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce014c9e0 0x7fcce00c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260565271 0 0) 0x7fcce00c59d0 con 0x7fcce0155ea0 2023-10-24T23:53:50.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcccc003530 con 0x7fcce0155ea0 2023-10-24T23:53:50.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3825110829 0 0) 0x7fcccc003530 con 0x7fcce0155ea0 2023-10-24T23:53:50.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce014c9e0 msgr2=0x7fcce00c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:50.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 --2- 172.21.15.107:0/4283551979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce014c9e0 0x7fcce00c3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 >> v2:172.21.15.110:3300/0 conn(0x7fcce006ec90 msgr2=0x7fcce014c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:50.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 --2- 172.21.15.107:0/4283551979 >> v2:172.21.15.110:3300/0 conn(0x7fcce006ec90 0x7fcce014c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce006d500 con 0x7fcce0155ea0 2023-10-24T23:53:50.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fccdc0016a0 con 0x7fcce0155ea0 2023-10-24T23:53:50.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 >> v1:172.21.15.107:6789/0 conn(0x7fcce0155ea0 legacy=0x7fcce0156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 --2- 172.21.15.107:0/4283551979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcccc003980 0x7fcccc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 --2- 172.21.15.107:0/4283551979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce014c9e0 0x7fcccc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 --2- 172.21.15.107:0/4283551979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcccc004d10 0x7fcccc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00c2990 con 0x7fcccc003980 2023-10-24T23:53:50.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce00c2ec0 con 0x7fcccc004d10 2023-10-24T23:53:50.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc00ba10 con 0x7fcce014c9e0 2023-10-24T23:53:50.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce5ca2640 1 --2- 172.21.15.107:0/4283551979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcccc003980 0x7fcccc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce5ca2640 1 -- 172.21.15.107:0/4283551979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce014c9e0 msgr2=0x7fcccc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:50.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce5ca2640 1 --2- 172.21.15.107:0/4283551979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce014c9e0 0x7fcccc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce5ca2640 1 -- 172.21.15.107:0/4283551979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcccc004d10 msgr2=0x7fcccc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:50.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce5ca2640 1 --2- 172.21.15.107:0/4283551979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcccc004d10 0x7fcccc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce5ca2640 1 -- 172.21.15.107:0/4283551979 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc003530 con 0x7fcccc003980 2023-10-24T23:53:50.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce5ca2640 1 --2- 172.21.15.107:0/4283551979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcccc003980 0x7fcccc003da0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcce0072600 tx=0x7fccdc01cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df26de6b65260bd server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc0058e0 con 0x7fcccc003980 2023-10-24T23:53:50.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc005a40 con 0x7fcccc003980 2023-10-24T23:53:50.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce4ca0640 1 -- 172.21.15.107:0/4283551979 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc005df0 con 0x7fcccc003980 2023-10-24T23:53:50.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce7f06640 1 -- 172.21.15.107:0/4283551979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcccc003980 msgr2=0x7fcccc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce7f06640 1 --2- 172.21.15.107:0/4283551979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcccc003980 0x7fcccc003da0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcce0072600 tx=0x7fccdc01cc20 comp rx=0 tx=0).stop 2023-10-24T23:53:50.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce64a3640 1 -- 172.21.15.107:0/4283551979 reap_dead start 2023-10-24T23:53:50.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce7f06640 1 -- 172.21.15.107:0/4283551979 shutdown_connections 2023-10-24T23:53:50.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.057+0000 7fcce7f06640 1 -- 172.21.15.107:0/4283551979 >> 172.21.15.107:0/4283551979 conn(0x7fcce00c4a90 msgr2=0x7fcce00c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:50.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 -- 172.21.15.107:0/4283551979 shutdown_connections 2023-10-24T23:53:50.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 -- 172.21.15.107:0/4283551979 wait complete. 2023-10-24T23:53:50.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 Processor -- start 2023-10-24T23:53:50.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 -- start start 2023-10-24T23:53:50.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcccc003980 0x7fcce014ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce006ca20 0x7fcce0143a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce01474c0 0x7fcce0143f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce0144750 con 0x7fcce01474c0 2023-10-24T23:53:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce01449a0 con 0x7fcccc003980 2023-10-24T23:53:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce0144c80 con 0x7fcce006ca20 2023-10-24T23:53:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce64a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce01474c0 0x7fcce0143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce64a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce01474c0 0x7fcce0143f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37200/0 (socket says 172.21.15.107:37200) 2023-10-24T23:53:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce64a3640 1 -- 172.21.15.107:0/3079254564 learned_addr learned my addr 172.21.15.107:0/3079254564 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce54a1640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce006ca20 0x7fcce0143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce5ca2640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcccc003980 0x7fcce014ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce64a3640 1 -- 172.21.15.107:0/3079254564 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce006ca20 msgr2=0x7fcce0143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce64a3640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce006ca20 0x7fcce0143a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce64a3640 1 -- 172.21.15.107:0/3079254564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcccc003980 msgr2=0x7fcce014ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce64a3640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcccc003980 0x7fcce014ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce64a3640 1 -- 172.21.15.107:0/3079254564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce0144f60 con 0x7fcce01474c0 2023-10-24T23:53:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce64a3640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce01474c0 0x7fcce0143f60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcce006e3d0 tx=0x7fccd800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fccd67fc640 1 -- 172.21.15.107:0/3079254564 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd8010040 con 0x7fcce01474c0 2023-10-24T23:53:50.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fccd67fc640 1 -- 172.21.15.107:0/3079254564 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd800bad0 con 0x7fcce01474c0 2023-10-24T23:53:50.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 -- 172.21.15.107:0/3079254564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce0145240 con 0x7fcce01474c0 2023-10-24T23:53:50.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fccd67fc640 1 -- 172.21.15.107:0/3079254564 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd800be80 con 0x7fcce01474c0 2023-10-24T23:53:50.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 -- 172.21.15.107:0/3079254564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce00c7220 con 0x7fcce01474c0 2023-10-24T23:53:50.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.061+0000 7fcce7f06640 1 -- 172.21.15.107:0/3079254564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce0145430 con 0x7fcce01474c0 2023-10-24T23:53:50.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.065+0000 7fccd67fc640 1 -- 172.21.15.107:0/3079254564 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fccd801d400 con 0x7fcce01474c0 2023-10-24T23:53:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.065+0000 7fccd67fc640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fccc4065e90 0x7fccc4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.065+0000 7fccd67fc640 1 -- 172.21.15.107:0/3079254564 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7fccd80840f0 con 0x7fcce01474c0 2023-10-24T23:53:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.065+0000 7fcce5ca2640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fccc4065e90 0x7fccc4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.065+0000 7fcce5ca2640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fccc4065e90 0x7fccc4068350 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fccdc0aa000 tx=0x7fccdc001a30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.069+0000 7fccd67fc640 1 -- 172.21.15.107:0/3079254564 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fccd8051440 con 0x7fcce01474c0 2023-10-24T23:53:50.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.213+0000 7fcce7f06640 1 -- 172.21.15.107:0/3079254564 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcce00746e0 con 0x7fcce01474c0 2023-10-24T23:53:50.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.213+0000 7fccd67fc640 1 -- 172.21.15.107:0/3079254564 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+21618 (secure 0 0 0) 0x7fccd8054a60 con 0x7fcce01474c0 2023-10-24T23:53:50.221 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:50.221 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:44.462555+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2f","mappings":[{"from":3,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":2}]},{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:50.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 -- 172.21.15.107:0/3079254564 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fccc4065e90 msgr2=0x7fccc4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fccc4065e90 0x7fccc4068350 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fccdc0aa000 tx=0x7fccdc001a30 comp rx=0 tx=0).stop 2023-10-24T23:53:50.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 -- 172.21.15.107:0/3079254564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce01474c0 msgr2=0x7fcce0143f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce01474c0 0x7fcce0143f60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcce006e3d0 tx=0x7fccd800eea0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 -- 172.21.15.107:0/3079254564 shutdown_connections 2023-10-24T23:53:50.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce01474c0 0x7fcce0143f60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fccc4065e90 0x7fccc4068350 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce006ca20 0x7fcce0143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 --2- 172.21.15.107:0/3079254564 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcccc003980 0x7fcce014ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 -- 172.21.15.107:0/3079254564 >> 172.21.15.107:0/3079254564 conn(0x7fcce00c4a90 msgr2=0x7fcce00ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:50.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 -- 172.21.15.107:0/3079254564 shutdown_connections 2023-10-24T23:53:50.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.217+0000 7fcce7f06640 1 -- 172.21.15.107:0/3079254564 wait complete. 2023-10-24T23:53:50.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:50.239 INFO:tasks.mds_thrash.fs.[a]:starting mds_do_thrash for fs a 2023-10-24T23:53:50.240 INFO:tasks.mds_thrash.fs.[a]:waiting for 110 secs before thrashing 2023-10-24T23:53:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.349+0000 7f9d89424640 1 Processor -- start 2023-10-24T23:53:50.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.349+0000 7f9d89424640 1 -- start start 2023-10-24T23:53:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d89424640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d84155370 0x7f9d840ac450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d89424640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d84155870 con 0x7f9d84151f00 2023-10-24T23:53:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d89424640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d841522e0 con 0x7f9d840c1080 2023-10-24T23:53:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d89424640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d840c02c0 con 0x7f9d84155370 2023-10-24T23:53:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d7bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d84155370 0x7f9d840ac450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d7bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d84155370 0x7f9d840ac450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37208/0 (socket says 172.21.15.107:37208) 2023-10-24T23:53:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d7bfff640 1 -- 172.21.15.107:0/90920025 learned_addr learned my addr 172.21.15.107:0/90920025 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d82ffd640 1 --1- 172.21.15.107:0/90920025 >> v1:172.21.15.107:6789/0 conn(0x7f9d84151f00 0x7f9d84070e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45222/0 (socket says 172.21.15.107:45222) 2023-10-24T23:53:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d7bfff640 1 -- 172.21.15.107:0/90920025 >> v1:172.21.15.110:6789/0 conn(0x7f9d840c1080 legacy=0x7f9d840ac990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d7bfff640 1 -- 172.21.15.107:0/90920025 >> v1:172.21.15.107:6789/0 conn(0x7f9d84151f00 legacy=0x7f9d84070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d7bfff640 1 -- 172.21.15.107:0/90920025 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d840c05a0 con 0x7f9d84155370 2023-10-24T23:53:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d827fc640 1 -- 172.21.15.107:0/90920025 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941894221 0 0) 0x7f9d84155870 con 0x7f9d84151f00 2023-10-24T23:53:50.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.353+0000 7f9d7bfff640 1 --2- 172.21.15.107:0/90920025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d84155370 0x7f9d840ac450 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009870 tx=0x7f9d6c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab776978ab7fe367 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d827fc640 1 -- 172.21.15.107:0/90920025 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0aa660 con 0x7f9d84155370 2023-10-24T23:53:50.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d827fc640 1 -- 172.21.15.107:0/90920025 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6c0aa7c0 con 0x7f9d84155370 2023-10-24T23:53:50.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d827fc640 1 -- 172.21.15.107:0/90920025 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0aab50 con 0x7f9d84155370 2023-10-24T23:53:50.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 -- 172.21.15.107:0/90920025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d84155370 msgr2=0x7f9d840ac450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 --2- 172.21.15.107:0/90920025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d84155370 0x7f9d840ac450 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009870 tx=0x7f9d6c01c500 comp rx=0 tx=0).stop 2023-10-24T23:53:50.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 -- 172.21.15.107:0/90920025 shutdown_connections 2023-10-24T23:53:50.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 --2- 172.21.15.107:0/90920025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d84155370 0x7f9d840ac450 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 -- 172.21.15.107:0/90920025 >> 172.21.15.107:0/90920025 conn(0x7f9d840c5cd0 msgr2=0x7f9d840c60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:50.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 -- 172.21.15.107:0/90920025 shutdown_connections 2023-10-24T23:53:50.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 -- 172.21.15.107:0/90920025 wait complete. 2023-10-24T23:53:50.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 Processor -- start 2023-10-24T23:53:50.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 -- start start 2023-10-24T23:53:50.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d840c1080 0x7f9d8414a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d84151f00 0x7f9d8414f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d84155370 0x7f9d841eff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8414ada0 con 0x7f9d84155370 2023-10-24T23:53:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d841f07e0 con 0x7f9d84151f00 2023-10-24T23:53:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8406e250 con 0x7f9d840c1080 2023-10-24T23:53:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d84155370 0x7f9d841eff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d84155370 0x7f9d841eff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37212/0 (socket says 172.21.15.107:37212) 2023-10-24T23:53:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d837fe640 1 -- 172.21.15.107:0/441589144 learned_addr learned my addr 172.21.15.107:0/441589144 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d82ffd640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d840c1080 0x7f9d8414a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d7bfff640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d84151f00 0x7f9d8414f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d837fe640 1 -- 172.21.15.107:0/441589144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d840c1080 msgr2=0x7f9d8414a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d837fe640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d840c1080 0x7f9d8414a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d837fe640 1 -- 172.21.15.107:0/441589144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d84151f00 msgr2=0x7f9d8414f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d837fe640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d84151f00 0x7f9d8414f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d837fe640 1 -- 172.21.15.107:0/441589144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c0094d0 con 0x7f9d84155370 2023-10-24T23:53:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d837fe640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d84155370 0x7f9d841eff00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c780 tx=0x7f9d7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d7b7fe640 1 -- 172.21.15.107:0/441589144 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c004b70 con 0x7f9d84155370 2023-10-24T23:53:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d7b7fe640 1 -- 172.21.15.107:0/441589144 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7c00ce90 con 0x7f9d84155370 2023-10-24T23:53:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 -- 172.21.15.107:0/441589144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8406e530 con 0x7f9d84155370 2023-10-24T23:53:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d7b7fe640 1 -- 172.21.15.107:0/441589144 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c00a850 con 0x7f9d84155370 2023-10-24T23:53:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.357+0000 7f9d89424640 1 -- 172.21.15.107:0/441589144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d841f0ac0 con 0x7f9d84155370 2023-10-24T23:53:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.361+0000 7f9d89424640 1 -- 172.21.15.107:0/441589144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d840746e0 con 0x7f9d84155370 2023-10-24T23:53:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.361+0000 7f9d7b7fe640 1 -- 172.21.15.107:0/441589144 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9d7c00a9b0 con 0x7f9d84155370 2023-10-24T23:53:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.361+0000 7f9d7b7fe640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d64065e60 0x7f9d64068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.361+0000 7f9d7b7fe640 1 -- 172.21.15.107:0/441589144 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f9d7c083d00 con 0x7f9d84155370 2023-10-24T23:53:50.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.361+0000 7f9d82ffd640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d64065e60 0x7f9d64068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.361+0000 7f9d82ffd640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d64065e60 0x7f9d64068320 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9d74000f70 tx=0x7f9d74007b80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.365+0000 7f9d7b7fe640 1 -- 172.21.15.107:0/441589144 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9d7c051050 con 0x7f9d84155370 2023-10-24T23:53:50.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.509+0000 7f9d89424640 1 -- 172.21.15.107:0/441589144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d840721d0 con 0x7f9d84155370 2023-10-24T23:53:50.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.509+0000 7f9d7b7fe640 1 -- 172.21.15.107:0/441589144 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+21618 (secure 0 0 0) 0x7f9d7c054670 con 0x7f9d84155370 2023-10-24T23:53:50.516 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:50.516 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:44.462555+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2f","mappings":[{"from":3,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":2}]},{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:50.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 -- 172.21.15.107:0/441589144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d64065e60 msgr2=0x7f9d64068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d64065e60 0x7f9d64068320 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9d74000f70 tx=0x7f9d74007b80 comp rx=0 tx=0).stop 2023-10-24T23:53:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 -- 172.21.15.107:0/441589144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d84155370 msgr2=0x7f9d841eff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d84155370 0x7f9d841eff00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c780 tx=0x7f9d7c00cc40 comp rx=0 tx=0).stop 2023-10-24T23:53:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 -- 172.21.15.107:0/441589144 shutdown_connections 2023-10-24T23:53:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d84155370 0x7f9d841eff00 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d84151f00 0x7f9d8414f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d64065e60 0x7f9d64068320 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 --2- 172.21.15.107:0/441589144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d840c1080 0x7f9d8414a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 -- 172.21.15.107:0/441589144 >> 172.21.15.107:0/441589144 conn(0x7f9d840c5cd0 msgr2=0x7f9d840c60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 -- 172.21.15.107:0/441589144 shutdown_connections 2023-10-24T23:53:50.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.513+0000 7f9d89424640 1 -- 172.21.15.107:0/441589144 wait complete. 2023-10-24T23:53:50.534 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-10-24T23:53:09.786122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-10-24T23:53:10.766645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-24T23:53:50.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-24T23:53:50.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f3904909640 1 Processor -- start 2023-10-24T23:53:50.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f3904909640 1 -- start start 2023-10-24T23:53:50.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f3904909640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f39000c4bb0 0x7f3900155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f3904909640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f39000aada0 0x7f3900155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f3904909640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39001526d0 con 0x7f390014c3f0 2023-10-24T23:53:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f3904909640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39000c58d0 con 0x7f39000aada0 2023-10-24T23:53:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f3904909640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39001524c0 con 0x7f39000c4bb0 2023-10-24T23:53:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f38fed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f390014c3f0 0x7f39000a4dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45232/0 (socket says 172.21.15.107:45232) 2023-10-24T23:53:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f38fed9d640 1 -- 172.21.15.107:0/2050588943 learned_addr learned my addr 172.21.15.107:0/2050588943 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f38ff59e640 1 --2- 172.21.15.107:0/2050588943 >> v2:172.21.15.107:3300/0 conn(0x7f39000aada0 0x7f3900155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f38fdd9b640 1 -- 172.21.15.107:0/2050588943 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865953304 0 0) 0x7f39001526d0 con 0x7f390014c3f0 2023-10-24T23:53:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f38fdd9b640 1 -- 172.21.15.107:0/2050588943 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e8003530 con 0x7f390014c3f0 2023-10-24T23:53:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f38fe59c640 1 --2- 172.21.15.107:0/2050588943 >> v2:172.21.15.110:3300/0 conn(0x7f39000c4bb0 0x7f3900155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f38ff59e640 1 -- 172.21.15.107:0/2050588943 >> v2:172.21.15.110:3300/0 conn(0x7f39000c4bb0 msgr2=0x7f3900155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f38ff59e640 1 --2- 172.21.15.107:0/2050588943 >> v2:172.21.15.110:3300/0 conn(0x7f39000c4bb0 0x7f3900155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f38ff59e640 1 -- 172.21.15.107:0/2050588943 >> v1:172.21.15.107:6789/0 conn(0x7f390014c3f0 legacy=0x7f39000a4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.641+0000 7f38ff59e640 1 -- 172.21.15.107:0/2050588943 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39001529b0 con 0x7f39000aada0 2023-10-24T23:53:50.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38ff59e640 1 --2- 172.21.15.107:0/2050588943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39000aada0 0x7f3900155ff0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f39000c4f90 tx=0x7f38f801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8290300f0647e2ce server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fdd9b640 1 -- 172.21.15.107:0/2050588943 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f80a5830 con 0x7f39000aada0 2023-10-24T23:53:50.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fdd9b640 1 -- 172.21.15.107:0/2050588943 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f80a5990 con 0x7f39000aada0 2023-10-24T23:53:50.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fdd9b640 1 -- 172.21.15.107:0/2050588943 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f80a5d00 con 0x7f39000aada0 2023-10-24T23:53:50.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 -- 172.21.15.107:0/2050588943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39000aada0 msgr2=0x7f3900155ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 --2- 172.21.15.107:0/2050588943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39000aada0 0x7f3900155ff0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f39000c4f90 tx=0x7f38f801c880 comp rx=0 tx=0).stop 2023-10-24T23:53:50.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 -- 172.21.15.107:0/2050588943 shutdown_connections 2023-10-24T23:53:50.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 --2- 172.21.15.107:0/2050588943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39000aada0 0x7f3900155ff0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 --2- 172.21.15.107:0/2050588943 >> v2:172.21.15.110:3300/0 conn(0x7f39000c4bb0 0x7f3900155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 -- 172.21.15.107:0/2050588943 >> 172.21.15.107:0/2050588943 conn(0x7f39000a7880 msgr2=0x7f39000a7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:50.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 -- 172.21.15.107:0/2050588943 shutdown_connections 2023-10-24T23:53:50.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 -- 172.21.15.107:0/2050588943 wait complete. 2023-10-24T23:53:50.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 Processor -- start 2023-10-24T23:53:50.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 -- start start 2023-10-24T23:53:50.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39000aada0 0x7f39001565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39000c4bb0 0x7f3900149350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f390014c3f0 0x7f3900143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39000c34b0 con 0x7f39000aada0 2023-10-24T23:53:50.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001edc50 con 0x7f39000c4bb0 2023-10-24T23:53:50.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39001544b0 con 0x7f390014c3f0 2023-10-24T23:53:50.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39000aada0 0x7f39001565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39000aada0 0x7f39001565b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37232/0 (socket says 172.21.15.107:37232) 2023-10-24T23:53:50.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fed9d640 1 -- 172.21.15.107:0/974693409 learned_addr learned my addr 172.21.15.107:0/974693409 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:50.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fed9d640 1 -- 172.21.15.107:0/974693409 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f390014c3f0 msgr2=0x7f3900143f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fed9d640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f390014c3f0 0x7f3900143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fed9d640 1 -- 172.21.15.107:0/974693409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39000c4bb0 msgr2=0x7f3900149350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fe59c640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39000c4bb0 0x7f3900149350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fed9d640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39000c4bb0 0x7f3900149350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fed9d640 1 -- 172.21.15.107:0/974693409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f8009a70 con 0x7f39000aada0 2023-10-24T23:53:50.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38fed9d640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39000aada0 0x7f39001565b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f38f000c450 tx=0x7f38f000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38e77fe640 1 -- 172.21.15.107:0/974693409 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0005cf0 con 0x7f39000aada0 2023-10-24T23:53:50.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38e77fe640 1 -- 172.21.15.107:0/974693409 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f0005e50 con 0x7f39000aada0 2023-10-24T23:53:50.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 -- 172.21.15.107:0/974693409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3900153d60 con 0x7f39000aada0 2023-10-24T23:53:50.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f38e77fe640 1 -- 172.21.15.107:0/974693409 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0007290 con 0x7f39000aada0 2023-10-24T23:53:50.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.645+0000 7f3904909640 1 -- 172.21.15.107:0/974693409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39000c2f90 con 0x7f39000aada0 2023-10-24T23:53:50.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.649+0000 7f3904909640 1 -- 172.21.15.107:0/974693409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390004e050 con 0x7f39000aada0 2023-10-24T23:53:50.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.649+0000 7f38e77fe640 1 -- 172.21.15.107:0/974693409 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f38f00073f0 con 0x7f39000aada0 2023-10-24T23:53:50.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.649+0000 7f38e77fe640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ec065e60 0x7f38ec068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.649+0000 7f38fe59c640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ec065e60 0x7f38ec068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.649+0000 7f38e77fe640 1 -- 172.21.15.107:0/974693409 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f38f0083100 con 0x7f39000aada0 2023-10-24T23:53:50.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.649+0000 7f38fe59c640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ec065e60 0x7f38ec068320 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f39000a81f0 tx=0x7f38f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.653+0000 7f38e77fe640 1 -- 172.21.15.107:0/974693409 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f38f0050450 con 0x7f39000aada0 2023-10-24T23:53:50.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.809+0000 7f3904909640 1 -- 172.21.15.107:0/974693409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f39000c5030 con 0x7f39000aada0 2023-10-24T23:53:50.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.809+0000 7f38e77fe640 1 -- 172.21.15.107:0/974693409 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38f0017020 con 0x7f39000aada0 2023-10-24T23:53:50.815 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-24T23:53:50.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.809+0000 7f3904909640 1 -- 172.21.15.107:0/974693409 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ec065e60 msgr2=0x7f38ec068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.809+0000 7f3904909640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ec065e60 0x7f38ec068320 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f39000a81f0 tx=0x7f38f4008040 comp rx=0 tx=0).stop 2023-10-24T23:53:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.809+0000 7f3904909640 1 -- 172.21.15.107:0/974693409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39000aada0 msgr2=0x7f39001565b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.809+0000 7f3904909640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39000aada0 0x7f39001565b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f38f000c450 tx=0x7f38f000c910 comp rx=0 tx=0).stop 2023-10-24T23:53:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.813+0000 7f3904909640 1 -- 172.21.15.107:0/974693409 shutdown_connections 2023-10-24T23:53:50.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.813+0000 7f3904909640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ec065e60 0x7f38ec068320 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.813+0000 7f3904909640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f390014c3f0 0x7f3900143f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.813+0000 7f3904909640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39000c4bb0 0x7f3900149350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.813+0000 7f3904909640 1 --2- 172.21.15.107:0/974693409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39000aada0 0x7f39001565b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.813+0000 7f3904909640 1 -- 172.21.15.107:0/974693409 >> 172.21.15.107:0/974693409 conn(0x7f39000a7880 msgr2=0x7f39000c4120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:50.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.813+0000 7f3904909640 1 -- 172.21.15.107:0/974693409 shutdown_connections 2023-10-24T23:53:50.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.813+0000 7f3904909640 1 -- 172.21.15.107:0/974693409 wait complete. 2023-10-24T23:53:50.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-24T23:53:50.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.941+0000 7ff38511c640 1 Processor -- start 2023-10-24T23:53:50.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.941+0000 7ff38511c640 1 -- start start 2023-10-24T23:53:50.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff38511c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3800ab1a0 0x7ff3800c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff38511c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff380072720 0x7ff3800bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff38511c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3800c1ce0 con 0x7ff3800bceb0 2023-10-24T23:53:50.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff38511c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3800be1d0 con 0x7ff380072720 2023-10-24T23:53:50.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff38511c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800c0880 con 0x7ff3800ab1a0 2023-10-24T23:53:50.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff37ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3800ab1a0 0x7ff3800c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff37e59c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff380072720 0x7ff3800bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff37f59e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff3800bceb0 0x7ff3800c4cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:46526/0 (socket says 172.21.15.107:46526) 2023-10-24T23:53:50.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff37f59e640 1 -- 172.21.15.107:0/4110123389 learned_addr learned my addr 172.21.15.107:0/4110123389 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:50.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff37e59c640 1 -- 172.21.15.107:0/4110123389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3800ab1a0 msgr2=0x7ff3800c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:50.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff37e59c640 1 --2- 172.21.15.107:0/4110123389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3800ab1a0 0x7ff3800c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff37e59c640 1 -- 172.21.15.107:0/4110123389 >> v1:172.21.15.110:6789/0 conn(0x7ff3800bceb0 legacy=0x7ff3800c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff37e59c640 1 -- 172.21.15.107:0/4110123389 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3800c1f30 con 0x7ff380072720 2023-10-24T23:53:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.945+0000 7ff37e59c640 1 --2- 172.21.15.107:0/4110123389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff380072720 0x7ff3800bfb30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff3700097c0 tx=0x7ff37001c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=676857a1c902ab7f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37dd9b640 1 -- 172.21.15.107:0/4110123389 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700ab600 con 0x7ff380072720 2023-10-24T23:53:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37dd9b640 1 -- 172.21.15.107:0/4110123389 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3700a7050 con 0x7ff380072720 2023-10-24T23:53:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37dd9b640 1 -- 172.21.15.107:0/4110123389 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700ab980 con 0x7ff380072720 2023-10-24T23:53:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 -- 172.21.15.107:0/4110123389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff380072720 msgr2=0x7ff3800bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 --2- 172.21.15.107:0/4110123389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff380072720 0x7ff3800bfb30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff3700097c0 tx=0x7ff37001c330 comp rx=0 tx=0).stop 2023-10-24T23:53:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 -- 172.21.15.107:0/4110123389 shutdown_connections 2023-10-24T23:53:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 --2- 172.21.15.107:0/4110123389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff380072720 0x7ff3800bfb30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 --2- 172.21.15.107:0/4110123389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3800ab1a0 0x7ff3800c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 -- 172.21.15.107:0/4110123389 >> 172.21.15.107:0/4110123389 conn(0x7ff3800c2860 msgr2=0x7ff3800c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 -- 172.21.15.107:0/4110123389 shutdown_connections 2023-10-24T23:53:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 -- 172.21.15.107:0/4110123389 wait complete. 2023-10-24T23:53:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 Processor -- start 2023-10-24T23:53:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 -- start start 2023-10-24T23:53:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3800ab1a0 0x7ff380155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3800bceb0 0x7ff3800a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3800aaaf0 0x7ff38014da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3800a4630 con 0x7ff3800bceb0 2023-10-24T23:53:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff380154940 con 0x7ff3800aaaf0 2023-10-24T23:53:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff380154440 con 0x7ff3800ab1a0 2023-10-24T23:53:50.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3800bceb0 0x7ff3800a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3800bceb0 0x7ff3800a8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37242/0 (socket says 172.21.15.107:37242) 2023-10-24T23:53:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37e59c640 1 -- 172.21.15.107:0/60585621 learned_addr learned my addr 172.21.15.107:0/60585621 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37ed9d640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3800ab1a0 0x7ff380155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37e59c640 1 -- 172.21.15.107:0/60585621 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3800ab1a0 msgr2=0x7ff380155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37e59c640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3800ab1a0 0x7ff380155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37f59e640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3800aaaf0 0x7ff38014da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37e59c640 1 -- 172.21.15.107:0/60585621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3800aaaf0 msgr2=0x7ff38014da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37e59c640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3800aaaf0 0x7ff38014da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37e59c640 1 -- 172.21.15.107:0/60585621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff370009420 con 0x7ff3800bceb0 2023-10-24T23:53:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37ed9d640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3800ab1a0 0x7ff380155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff37e59c640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3800bceb0 0x7ff3800a8d30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff3700ae040 tx=0x7ff3700190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff3677fe640 1 -- 172.21.15.107:0/60585621 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700b4430 con 0x7ff3800bceb0 2023-10-24T23:53:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff3677fe640 1 -- 172.21.15.107:0/60585621 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3700b4590 con 0x7ff3800bceb0 2023-10-24T23:53:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 -- 172.21.15.107:0/60585621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3800a48a0 con 0x7ff3800bceb0 2023-10-24T23:53:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff3677fe640 1 -- 172.21.15.107:0/60585621 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3700b4940 con 0x7ff3800bceb0 2023-10-24T23:53:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.949+0000 7ff38511c640 1 -- 172.21.15.107:0/60585621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff380157190 con 0x7ff3800bceb0 2023-10-24T23:53:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.953+0000 7ff38511c640 1 -- 172.21.15.107:0/60585621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3800beff0 con 0x7ff3800bceb0 2023-10-24T23:53:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.953+0000 7ff3677fe640 1 -- 172.21.15.107:0/60585621 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff3700b2040 con 0x7ff3800bceb0 2023-10-24T23:53:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.953+0000 7ff3677fe640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff36c065e60 0x7ff36c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.953+0000 7ff3677fe640 1 -- 172.21.15.107:0/60585621 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7ff37011da50 con 0x7ff3800bceb0 2023-10-24T23:53:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.953+0000 7ff37ed9d640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff36c065e60 0x7ff36c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.953+0000 7ff37ed9d640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff36c065e60 0x7ff36c068320 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff374009ae0 tx=0x7ff374008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:50.957+0000 7ff3677fe640 1 -- 172.21.15.107:0/60585621 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff3700eada0 con 0x7ff3800bceb0 2023-10-24T23:53:51.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.097+0000 7ff38511c640 1 -- 172.21.15.107:0/60585621 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7ff38004e130 con 0x7ff3800bceb0 2023-10-24T23:53:51.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.097+0000 7ff3677fe640 1 -- 172.21.15.107:0/60585621 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff3700ee3c0 con 0x7ff3800bceb0 2023-10-24T23:53:51.104 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-24T23:53:51.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 -- 172.21.15.107:0/60585621 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff36c065e60 msgr2=0x7ff36c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff36c065e60 0x7ff36c068320 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff374009ae0 tx=0x7ff374008040 comp rx=0 tx=0).stop 2023-10-24T23:53:51.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 -- 172.21.15.107:0/60585621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3800bceb0 msgr2=0x7ff3800a8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3800bceb0 0x7ff3800a8d30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff3700ae040 tx=0x7ff3700190d0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 -- 172.21.15.107:0/60585621 shutdown_connections 2023-10-24T23:53:51.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3800aaaf0 0x7ff38014da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3800bceb0 0x7ff3800a8d30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff36c065e60 0x7ff36c068320 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 --2- 172.21.15.107:0/60585621 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3800ab1a0 0x7ff380155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 -- 172.21.15.107:0/60585621 >> 172.21.15.107:0/60585621 conn(0x7ff3800c2860 msgr2=0x7ff380159960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 -- 172.21.15.107:0/60585621 shutdown_connections 2023-10-24T23:53:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.101+0000 7ff38511c640 1 -- 172.21.15.107:0/60585621 wait complete. 2023-10-24T23:53:51.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-24T23:53:51.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.233+0000 7fc0dfa5a640 1 Processor -- start 2023-10-24T23:53:51.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.233+0000 7fc0dfa5a640 1 -- start start 2023-10-24T23:53:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dfa5a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d80af9b0 0x7fc0d80c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dfa5a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0d806b360 0x7fc0d8069d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dfa5a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc0d80bd930 0x7fc0d806a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dfa5a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d80c5870 con 0x7fc0d806b360 2023-10-24T23:53:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dfa5a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d806a7c0 con 0x7fc0d80bd930 2023-10-24T23:53:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dfa5a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80c8180 con 0x7fc0d80af9b0 2023-10-24T23:53:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dd7f6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d80af9b0 0x7fc0d80c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dcff5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0d806b360 0x7fc0d8069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dcff5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0d806b360 0x7fc0d8069d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37254/0 (socket says 172.21.15.107:37254) 2023-10-24T23:53:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0ddff7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc0d80bd930 0x7fc0d806a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dcff5640 1 -- 172.21.15.107:0/4131883790 learned_addr learned my addr 172.21.15.107:0/4131883790 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:51.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dcff5640 1 -- 172.21.15.107:0/4131883790 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d80af9b0 msgr2=0x7fc0d80c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:51.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dcff5640 1 --2- 172.21.15.107:0/4131883790 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d80af9b0 0x7fc0d80c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dcff5640 1 -- 172.21.15.107:0/4131883790 >> v2:172.21.15.110:3300/0 conn(0x7fc0d80bd930 msgr2=0x7fc0d806a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:51.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dcff5640 1 --2- 172.21.15.107:0/4131883790 >> v2:172.21.15.110:3300/0 conn(0x7fc0d80bd930 0x7fc0d806a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dcff5640 1 -- 172.21.15.107:0/4131883790 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d8155430 con 0x7fc0d806b360 2023-10-24T23:53:51.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0dcff5640 1 --2- 172.21.15.107:0/4131883790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d806b360 0x7fc0d8069d40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80acba0 tx=0x7fc0c801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d167908287894f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:51.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0cffff640 1 -- 172.21.15.107:0/4131883790 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80ab600 con 0x7fc0d806b360 2023-10-24T23:53:51.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0cffff640 1 -- 172.21.15.107:0/4131883790 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c80ab760 con 0x7fc0d806b360 2023-10-24T23:53:51.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.237+0000 7fc0cffff640 1 -- 172.21.15.107:0/4131883790 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80abaf0 con 0x7fc0d806b360 2023-10-24T23:53:51.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/4131883790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d806b360 msgr2=0x7fc0d8069d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 --2- 172.21.15.107:0/4131883790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d806b360 0x7fc0d8069d40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80acba0 tx=0x7fc0c801c440 comp rx=0 tx=0).stop 2023-10-24T23:53:51.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/4131883790 shutdown_connections 2023-10-24T23:53:51.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 --2- 172.21.15.107:0/4131883790 >> v2:172.21.15.110:3300/0 conn(0x7fc0d80bd930 0x7fc0d806a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 --2- 172.21.15.107:0/4131883790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d806b360 0x7fc0d8069d40 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 --2- 172.21.15.107:0/4131883790 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d80af9b0 0x7fc0d80c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/4131883790 >> 172.21.15.107:0/4131883790 conn(0x7fc0d8071b00 msgr2=0x7fc0d80abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:51.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/4131883790 shutdown_connections 2023-10-24T23:53:51.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/4131883790 wait complete. 2023-10-24T23:53:51.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 Processor -- start 2023-10-24T23:53:51.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- start start 2023-10-24T23:53:51.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d806b360 0x7fc0d80c31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80af9b0 0x7fc0d80c3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80bd930 0x7fc0d80ad950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d814d300 con 0x7fc0d80bd930 2023-10-24T23:53:51.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d814d550 con 0x7fc0d80af9b0 2023-10-24T23:53:51.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d80c13f0 con 0x7fc0d806b360 2023-10-24T23:53:51.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0ddff7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80bd930 0x7fc0d80ad950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0ddff7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80bd930 0x7fc0d80ad950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37270/0 (socket says 172.21.15.107:37270) 2023-10-24T23:53:51.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0ddff7640 1 -- 172.21.15.107:0/174601552 learned_addr learned my addr 172.21.15.107:0/174601552 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:51.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dcff5640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80af9b0 0x7fc0d80c3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dd7f6640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d806b360 0x7fc0d80c31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0ddff7640 1 -- 172.21.15.107:0/174601552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d806b360 msgr2=0x7fc0d80c31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0ddff7640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d806b360 0x7fc0d80c31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0ddff7640 1 -- 172.21.15.107:0/174601552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80af9b0 msgr2=0x7fc0d80c3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0ddff7640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80af9b0 0x7fc0d80c3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0ddff7640 1 -- 172.21.15.107:0/174601552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c80094d0 con 0x7fc0d80bd930 2023-10-24T23:53:51.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dd7f6640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d806b360 0x7fc0d80c31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:51.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dcff5640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80af9b0 0x7fc0d80c3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:51.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0ddff7640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80bd930 0x7fc0d80ad950 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000efc0 tx=0x7fc0d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:51.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0cdffb640 1 -- 172.21.15.107:0/174601552 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0009b20 con 0x7fc0d80bd930 2023-10-24T23:53:51.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0cdffb640 1 -- 172.21.15.107:0/174601552 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d000f040 con 0x7fc0d80bd930 2023-10-24T23:53:51.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/174601552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d814d830 con 0x7fc0d80bd930 2023-10-24T23:53:51.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0cdffb640 1 -- 172.21.15.107:0/174601552 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d00146c0 con 0x7fc0d80bd930 2023-10-24T23:53:51.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/174601552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d815e980 con 0x7fc0d80bd930 2023-10-24T23:53:51.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.241+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/174601552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d80c1910 con 0x7fc0d80bd930 2023-10-24T23:53:51.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.245+0000 7fc0cdffb640 1 -- 172.21.15.107:0/174601552 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc0d0014820 con 0x7fc0d80bd930 2023-10-24T23:53:51.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.245+0000 7fc0cdffb640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0bc065e60 0x7fc0bc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.245+0000 7fc0dd7f6640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0bc065e60 0x7fc0bc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.245+0000 7fc0cdffb640 1 -- 172.21.15.107:0/174601552 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7fc0d0084970 con 0x7fc0d80bd930 2023-10-24T23:53:51.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.245+0000 7fc0dd7f6640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0bc065e60 0x7fc0bc068320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009a60 tx=0x7fc0d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:51.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.249+0000 7fc0cdffb640 1 -- 172.21.15.107:0/174601552 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc0d80c1910 con 0x7fc0d80bd930 2023-10-24T23:53:51.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/174601552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc0d80c1af0 con 0x7fc0d80bd930 2023-10-24T23:53:51.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0cdffb640 1 -- 172.21.15.107:0/174601552 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v29) v1 ==== 99+0+11 (secure 0 0 0) 0x7fc0d80c1af0 con 0x7fc0d80bd930 2023-10-24T23:53:51.411 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/174601552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0bc065e60 msgr2=0x7fc0bc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0dfa5a640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0bc065e60 0x7fc0bc068320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009a60 tx=0x7fc0d4008040 comp rx=0 tx=0).stop 2023-10-24T23:53:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/174601552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80bd930 msgr2=0x7fc0d80ad950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0dfa5a640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80bd930 0x7fc0d80ad950 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000efc0 tx=0x7fc0d000c490 comp rx=0 tx=0).stop 2023-10-24T23:53:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/174601552 shutdown_connections 2023-10-24T23:53:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0dfa5a640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80bd930 0x7fc0d80ad950 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0dfa5a640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80af9b0 0x7fc0d80c3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0dfa5a640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0bc065e60 0x7fc0bc068320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0dfa5a640 1 --2- 172.21.15.107:0/174601552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d806b360 0x7fc0d80c31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.405+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/174601552 >> 172.21.15.107:0/174601552 conn(0x7fc0d8071b00 msgr2=0x7fc0d8151570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:51.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.409+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/174601552 shutdown_connections 2023-10-24T23:53:51.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.409+0000 7fc0dfa5a640 1 -- 172.21.15.107:0/174601552 wait complete. 2023-10-24T23:53:51.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-24T23:53:51.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc3c45d640 1 Processor -- start 2023-10-24T23:53:51.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc3c45d640 1 -- start start 2023-10-24T23:53:51.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc3c45d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc340c1b30 0x7efc34072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc3c45d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efc340a9300 0x7efc340aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc3c45d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efc340aa820 0x7efc340c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc3c45d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc340befa0 con 0x7efc340a9300 2023-10-24T23:53:51.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc3c45d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc340a9b80 con 0x7efc340aa820 2023-10-24T23:53:51.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc3c45d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc340aa0b0 con 0x7efc340c1b30 2023-10-24T23:53:51.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc399f8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efc340a9300 0x7efc340aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc399f8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efc340a9300 0x7efc340aa2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37274/0 (socket says 172.21.15.107:37274) 2023-10-24T23:53:51.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc399f8640 1 -- 172.21.15.107:0/604945962 learned_addr learned my addr 172.21.15.107:0/604945962 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:51.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc3a1f9640 1 --2- 172.21.15.107:0/604945962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc340c1b30 0x7efc34072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc399f8640 1 -- 172.21.15.107:0/604945962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc340c1b30 msgr2=0x7efc34072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:51.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc3a9fa640 1 --2- 172.21.15.107:0/604945962 >> v2:172.21.15.110:3300/0 conn(0x7efc340aa820 0x7efc340c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc399f8640 1 --2- 172.21.15.107:0/604945962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc340c1b30 0x7efc34072210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc399f8640 1 -- 172.21.15.107:0/604945962 >> v2:172.21.15.110:3300/0 conn(0x7efc340aa820 msgr2=0x7efc340c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:51.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc399f8640 1 --2- 172.21.15.107:0/604945962 >> v2:172.21.15.110:3300/0 conn(0x7efc340aa820 0x7efc340c4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.545+0000 7efc399f8640 1 -- 172.21.15.107:0/604945962 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc340bf1f0 con 0x7efc340a9300 2023-10-24T23:53:51.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.549+0000 7efc399f8640 1 --2- 172.21.15.107:0/604945962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc340a9300 0x7efc340aa2e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7efc340abf30 tx=0x7efc2401c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c013ec0ab78ef00c server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:51.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.549+0000 7efc391f7640 1 -- 172.21.15.107:0/604945962 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc240a4c70 con 0x7efc340a9300 2023-10-24T23:53:51.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.549+0000 7efc391f7640 1 -- 172.21.15.107:0/604945962 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc240a4dd0 con 0x7efc340a9300 2023-10-24T23:53:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.549+0000 7efc391f7640 1 -- 172.21.15.107:0/604945962 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc240ac660 con 0x7efc340a9300 2023-10-24T23:53:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.549+0000 7efc3c45d640 1 -- 172.21.15.107:0/604945962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc340a9300 msgr2=0x7efc340aa2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.549+0000 7efc3c45d640 1 --2- 172.21.15.107:0/604945962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc340a9300 0x7efc340aa2e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7efc340abf30 tx=0x7efc2401c120 comp rx=0 tx=0).stop 2023-10-24T23:53:51.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.549+0000 7efc3c45d640 1 -- 172.21.15.107:0/604945962 shutdown_connections 2023-10-24T23:53:51.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.549+0000 7efc3c45d640 1 --2- 172.21.15.107:0/604945962 >> v2:172.21.15.110:3300/0 conn(0x7efc340aa820 0x7efc340c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.549+0000 7efc3c45d640 1 --2- 172.21.15.107:0/604945962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc340a9300 0x7efc340aa2e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.549+0000 7efc3c45d640 1 --2- 172.21.15.107:0/604945962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc340c1b30 0x7efc34072210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.549+0000 7efc3c45d640 1 -- 172.21.15.107:0/604945962 >> 172.21.15.107:0/604945962 conn(0x7efc340c2860 msgr2=0x7efc340c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:51.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 -- 172.21.15.107:0/604945962 shutdown_connections 2023-10-24T23:53:51.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 -- 172.21.15.107:0/604945962 wait complete. 2023-10-24T23:53:51.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 Processor -- start 2023-10-24T23:53:51.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 -- start start 2023-10-24T23:53:51.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc340a9300 0x7efc340704d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc340aa820 0x7efc3406ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc340c1b30 0x7efc341edb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc341542d0 con 0x7efc340c1b30 2023-10-24T23:53:51.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc341ee080 con 0x7efc340a9300 2023-10-24T23:53:51.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc341559c0 con 0x7efc340aa820 2023-10-24T23:53:51.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3a9fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc340c1b30 0x7efc341edb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3a9fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc340c1b30 0x7efc341edb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37276/0 (socket says 172.21.15.107:37276) 2023-10-24T23:53:51.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3a9fa640 1 -- 172.21.15.107:0/2134517994 learned_addr learned my addr 172.21.15.107:0/2134517994 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:51.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3a1f9640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc340a9300 0x7efc340704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc399f8640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc340aa820 0x7efc3406ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3a9fa640 1 -- 172.21.15.107:0/2134517994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc340aa820 msgr2=0x7efc3406ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3a9fa640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc340aa820 0x7efc3406ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3a9fa640 1 -- 172.21.15.107:0/2134517994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc340a9300 msgr2=0x7efc340704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3a9fa640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc340a9300 0x7efc340704d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3a9fa640 1 -- 172.21.15.107:0/2134517994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc240094d0 con 0x7efc340c1b30 2023-10-24T23:53:51.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc399f8640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc340aa820 0x7efc3406ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:51.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3a9fa640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc340c1b30 0x7efc341edb40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efc2c00ccf0 tx=0x7efc2c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:51.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc2affd640 1 -- 172.21.15.107:0/2134517994 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c003ba0 con 0x7efc340c1b30 2023-10-24T23:53:51.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc2affd640 1 -- 172.21.15.107:0/2134517994 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2c003d00 con 0x7efc340c1b30 2023-10-24T23:53:51.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc2affd640 1 -- 172.21.15.107:0/2134517994 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c004240 con 0x7efc340c1b30 2023-10-24T23:53:51.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 -- 172.21.15.107:0/2134517994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc34155ca0 con 0x7efc340c1b30 2023-10-24T23:53:51.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 -- 172.21.15.107:0/2134517994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc341544e0 con 0x7efc340c1b30 2023-10-24T23:53:51.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.553+0000 7efc3c45d640 1 -- 172.21.15.107:0/2134517994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3404e130 con 0x7efc340c1b30 2023-10-24T23:53:51.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.557+0000 7efc2affd640 1 -- 172.21.15.107:0/2134517994 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7efc2c0043a0 con 0x7efc340c1b30 2023-10-24T23:53:51.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.557+0000 7efc2affd640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efc18065e60 0x7efc18068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.557+0000 7efc2affd640 1 -- 172.21.15.107:0/2134517994 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7efc2c012070 con 0x7efc340c1b30 2023-10-24T23:53:51.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.557+0000 7efc3a1f9640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efc18065e60 0x7efc18068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.557+0000 7efc3a1f9640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efc18065e60 0x7efc18068320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efc34156f00 tx=0x7efc30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:51.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.561+0000 7efc2affd640 1 -- 172.21.15.107:0/2134517994 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7efc2c051850 con 0x7efc340c1b30 2023-10-24T23:53:51.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.713+0000 7efc3c45d640 1 -- 172.21.15.107:0/2134517994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7efc340a90e0 con 0x7efc340c1b30 2023-10-24T23:53:51.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.713+0000 7efc2affd640 1 -- 172.21.15.107:0/2134517994 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v29) v1 ==== 95+0+11 (secure 0 0 0) 0x7efc2c054e70 con 0x7efc340c1b30 2023-10-24T23:53:51.720 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:51.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 -- 172.21.15.107:0/2134517994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efc18065e60 msgr2=0x7efc18068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efc18065e60 0x7efc18068320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efc34156f00 tx=0x7efc30008040 comp rx=0 tx=0).stop 2023-10-24T23:53:51.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 -- 172.21.15.107:0/2134517994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc340c1b30 msgr2=0x7efc341edb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc340c1b30 0x7efc341edb40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efc2c00ccf0 tx=0x7efc2c00a660 comp rx=0 tx=0).stop 2023-10-24T23:53:51.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 -- 172.21.15.107:0/2134517994 shutdown_connections 2023-10-24T23:53:51.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc340c1b30 0x7efc341edb40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efc18065e60 0x7efc18068320 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efc340aa820 0x7efc3406ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 --2- 172.21.15.107:0/2134517994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efc340a9300 0x7efc340704d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 -- 172.21.15.107:0/2134517994 >> 172.21.15.107:0/2134517994 conn(0x7efc340c2860 msgr2=0x7efc340a7000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:51.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 -- 172.21.15.107:0/2134517994 shutdown_connections 2023-10-24T23:53:51.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.717+0000 7efc3c45d640 1 -- 172.21.15.107:0/2134517994 wait complete. 2023-10-24T23:53:51.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-10-24T23:53:51.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c879c6640 1 Processor -- start 2023-10-24T23:53:51.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c879c6640 1 -- start start 2023-10-24T23:53:51.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c879c6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9c800c1b30 0x7f9c80072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c879c6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c800aa9f0 0x7f9c80155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c879c6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c800befa0 con 0x7f9c800a9300 2023-10-24T23:53:51.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c879c6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c800a9b80 con 0x7f9c800c1b30 2023-10-24T23:53:51.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c879c6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c800aa0b0 con 0x7f9c800aa9f0 2023-10-24T23:53:51.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c85762640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9c800c1b30 0x7f9c80072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c85762640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9c800c1b30 0x7f9c80072210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37282/0 (socket says 172.21.15.107:37282) 2023-10-24T23:53:51.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c85762640 1 -- 172.21.15.107:0/2739101387 learned_addr learned my addr 172.21.15.107:0/2739101387 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:51.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c85f63640 1 --2- 172.21.15.107:0/2739101387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c800aa9f0 0x7f9c80155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c85762640 1 -- 172.21.15.107:0/2739101387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c800aa9f0 msgr2=0x7f9c80155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:51.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c85762640 1 --2- 172.21.15.107:0/2739101387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c800aa9f0 0x7f9c80155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c85762640 1 -- 172.21.15.107:0/2739101387 >> v1:172.21.15.110:6789/0 conn(0x7f9c800a9300 legacy=0x7f9c800aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:51.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c85762640 1 -- 172.21.15.107:0/2739101387 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c800bf1f0 con 0x7f9c800c1b30 2023-10-24T23:53:51.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.861+0000 7f9c85f63640 1 --2- 172.21.15.107:0/2739101387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c800aa9f0 0x7f9c80155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:51.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c85762640 1 --2- 172.21.15.107:0/2739101387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c800c1b30 0x7f9c80072210 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c009870 tx=0x7f9c7c01c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379288330c0e340b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:51.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c77fff640 1 -- 172.21.15.107:0/2739101387 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0a4d80 con 0x7f9c800c1b30 2023-10-24T23:53:51.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c77fff640 1 -- 172.21.15.107:0/2739101387 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c0ac440 con 0x7f9c800c1b30 2023-10-24T23:53:51.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c77fff640 1 -- 172.21.15.107:0/2739101387 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0ac7d0 con 0x7f9c800c1b30 2023-10-24T23:53:51.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c879c6640 1 -- 172.21.15.107:0/2739101387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c800c1b30 msgr2=0x7f9c80072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c879c6640 1 --2- 172.21.15.107:0/2739101387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c800c1b30 0x7f9c80072210 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c009870 tx=0x7f9c7c01c120 comp rx=0 tx=0).stop 2023-10-24T23:53:51.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c879c6640 1 -- 172.21.15.107:0/2739101387 shutdown_connections 2023-10-24T23:53:51.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c879c6640 1 --2- 172.21.15.107:0/2739101387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c800aa9f0 0x7f9c80155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c879c6640 1 --2- 172.21.15.107:0/2739101387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c800c1b30 0x7f9c80072210 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c879c6640 1 -- 172.21.15.107:0/2739101387 >> 172.21.15.107:0/2739101387 conn(0x7f9c800c2860 msgr2=0x7f9c800c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:51.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c879c6640 1 -- 172.21.15.107:0/2739101387 shutdown_connections 2023-10-24T23:53:51.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c879c6640 1 -- 172.21.15.107:0/2739101387 wait complete. 2023-10-24T23:53:51.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.865+0000 7f9c879c6640 1 Processor -- start 2023-10-24T23:53:51.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c879c6640 1 -- start start 2023-10-24T23:53:51.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c879c6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c800a9300 0x7f9c801ee880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c879c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c800c1b30 0x7f9c8006bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c879c6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c801eedc0 0x7f9c8006c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c879c6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c800c3e00 con 0x7f9c800c1b30 2023-10-24T23:53:51.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c879c6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c801ef5f0 con 0x7f9c800a9300 2023-10-24T23:53:51.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c879c6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8006b8f0 con 0x7f9c801eedc0 2023-10-24T23:53:51.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c84f61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c800c1b30 0x7f9c8006bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c84f61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c800c1b30 0x7f9c8006bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37292/0 (socket says 172.21.15.107:37292) 2023-10-24T23:53:51.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c84f61640 1 -- 172.21.15.107:0/2204555416 learned_addr learned my addr 172.21.15.107:0/2204555416 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:51.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c85762640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c800a9300 0x7f9c801ee880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c85f63640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c801eedc0 0x7f9c8006c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c84f61640 1 -- 172.21.15.107:0/2204555416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c801eedc0 msgr2=0x7f9c8006c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c84f61640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c801eedc0 0x7f9c8006c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c84f61640 1 -- 172.21.15.107:0/2204555416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c800a9300 msgr2=0x7f9c801ee880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:51.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c84f61640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c800a9300 0x7f9c801ee880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:51.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c84f61640 1 -- 172.21.15.107:0/2204555416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c0094d0 con 0x7f9c800c1b30 2023-10-24T23:53:51.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c84f61640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c800c1b30 0x7f9c8006bd80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000c410 tx=0x7f9c7000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:51.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c75ffb640 1 -- 172.21.15.107:0/2204555416 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c70011790 con 0x7f9c800c1b30 2023-10-24T23:53:51.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c75ffb640 1 -- 172.21.15.107:0/2204555416 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c700118f0 con 0x7f9c800c1b30 2023-10-24T23:53:51.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c879c6640 1 -- 172.21.15.107:0/2204555416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c80069ee0 con 0x7f9c800c1b30 2023-10-24T23:53:51.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c75ffb640 1 -- 172.21.15.107:0/2204555416 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c70011ca0 con 0x7f9c800c1b30 2023-10-24T23:53:51.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c879c6640 1 -- 172.21.15.107:0/2204555416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c800b95a0 con 0x7f9c800c1b30 2023-10-24T23:53:51.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c879c6640 1 -- 172.21.15.107:0/2204555416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c8006ba60 con 0x7f9c800c1b30 2023-10-24T23:53:51.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c75ffb640 1 -- 172.21.15.107:0/2204555416 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9c7001c460 con 0x7f9c800c1b30 2023-10-24T23:53:51.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c75ffb640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9c64065e60 0x7f9c64068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:51.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c75ffb640 1 -- 172.21.15.107:0/2204555416 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f9c70083890 con 0x7f9c800c1b30 2023-10-24T23:53:51.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c85762640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9c64065e60 0x7f9c64068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:51.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.869+0000 7f9c85762640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9c64065e60 0x7f9c64068320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c0ae000 tx=0x7f9c7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:51.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:51.873+0000 7f9c75ffb640 1 -- 172.21.15.107:0/2204555416 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9c70050bc0 con 0x7f9c800c1b30 2023-10-24T23:53:52.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.037+0000 7f9c879c6640 1 -- 172.21.15.107:0/2204555416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9c80071d60 con 0x7f9c800c1b30 2023-10-24T23:53:52.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.037+0000 7f9c75ffb640 1 -- 172.21.15.107:0/2204555416 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v29) v1 ==== 99+0+11 (secure 0 0 0) 0x7f9c700541e0 con 0x7f9c800c1b30 2023-10-24T23:53:52.044 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-24T23:53:52.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 -- 172.21.15.107:0/2204555416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9c64065e60 msgr2=0x7f9c64068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9c64065e60 0x7f9c64068320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c0ae000 tx=0x7f9c7c008040 comp rx=0 tx=0).stop 2023-10-24T23:53:52.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 -- 172.21.15.107:0/2204555416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c800c1b30 msgr2=0x7f9c8006bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c800c1b30 0x7f9c8006bd80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000c410 tx=0x7f9c7000c8d0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 -- 172.21.15.107:0/2204555416 shutdown_connections 2023-10-24T23:53:52.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9c64065e60 0x7f9c64068320 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c801eedc0 0x7f9c8006c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c800c1b30 0x7f9c8006bd80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 --2- 172.21.15.107:0/2204555416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c800a9300 0x7f9c801ee880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 -- 172.21.15.107:0/2204555416 >> 172.21.15.107:0/2204555416 conn(0x7f9c800c2860 msgr2=0x7f9c800a5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 -- 172.21.15.107:0/2204555416 shutdown_connections 2023-10-24T23:53:52.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.041+0000 7f9c879c6640 1 -- 172.21.15.107:0/2204555416 wait complete. 2023-10-24T23:53:52.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-10-24T23:53:52.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd9d02640 1 Processor -- start 2023-10-24T23:53:52.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd9d02640 1 -- start start 2023-10-24T23:53:52.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd9d02640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fecd40ab1a0 0x7fecd40c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd9d02640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fecd4072720 0x7fecd40bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd9d02640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd40c1ce0 con 0x7fecd40bceb0 2023-10-24T23:53:52.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd9d02640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd40be1d0 con 0x7fecd4072720 2023-10-24T23:53:52.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd9d02640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd40c0880 con 0x7fecd40ab1a0 2023-10-24T23:53:52.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fecd40bceb0 0x7fecd40c4cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45246/0 (socket says 172.21.15.107:45246) 2023-10-24T23:53:52.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fecd4072720 0x7fecd40bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd3fff640 1 -- 172.21.15.107:0/43631040 learned_addr learned my addr 172.21.15.107:0/43631040 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:52.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd37fe640 1 --2- 172.21.15.107:0/43631040 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fecd40ab1a0 0x7fecd40c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd27fc640 1 -- 172.21.15.107:0/43631040 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942495369 0 0) 0x7fecd40c1ce0 con 0x7fecd40bceb0 2023-10-24T23:53:52.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd27fc640 1 -- 172.21.15.107:0/43631040 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecbc003530 con 0x7fecd40bceb0 2023-10-24T23:53:52.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd2ffd640 1 -- 172.21.15.107:0/43631040 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fecd40ab1a0 msgr2=0x7fecd40c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:52.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd2ffd640 1 --2- 172.21.15.107:0/43631040 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fecd40ab1a0 0x7fecd40c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd2ffd640 1 -- 172.21.15.107:0/43631040 >> v1:172.21.15.107:6789/0 conn(0x7fecd40bceb0 legacy=0x7fecd40c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.181+0000 7fecd2ffd640 1 -- 172.21.15.107:0/43631040 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd40c1f30 con 0x7fecd4072720 2023-10-24T23:53:52.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd2ffd640 1 --2- 172.21.15.107:0/43631040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecd4072720 0x7fecd40bfb30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fecc80097c0 tx=0x7fecc801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befd73f3e79c31a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:52.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd27fc640 1 -- 172.21.15.107:0/43631040 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc80ab600 con 0x7fecd4072720 2023-10-24T23:53:52.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd27fc640 1 -- 172.21.15.107:0/43631040 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc80a7050 con 0x7fecd4072720 2023-10-24T23:53:52.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd27fc640 1 -- 172.21.15.107:0/43631040 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc80ab980 con 0x7fecd4072720 2023-10-24T23:53:52.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd9d02640 1 -- 172.21.15.107:0/43631040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecd4072720 msgr2=0x7fecd40bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd9d02640 1 --2- 172.21.15.107:0/43631040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecd4072720 0x7fecd40bfb30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fecc80097c0 tx=0x7fecc801c330 comp rx=0 tx=0).stop 2023-10-24T23:53:52.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd9d02640 1 -- 172.21.15.107:0/43631040 shutdown_connections 2023-10-24T23:53:52.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd9d02640 1 --2- 172.21.15.107:0/43631040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecd4072720 0x7fecd40bfb30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd9d02640 1 --2- 172.21.15.107:0/43631040 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fecd40ab1a0 0x7fecd40c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd9d02640 1 -- 172.21.15.107:0/43631040 >> 172.21.15.107:0/43631040 conn(0x7fecd40c2860 msgr2=0x7fecd40c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:52.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd9d02640 1 -- 172.21.15.107:0/43631040 shutdown_connections 2023-10-24T23:53:52.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.185+0000 7fecd9d02640 1 -- 172.21.15.107:0/43631040 wait complete. 2023-10-24T23:53:52.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd9d02640 1 Processor -- start 2023-10-24T23:53:52.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd9d02640 1 -- start start 2023-10-24T23:53:52.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd9d02640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fecd40ab1a0 0x7fecd40c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd9d02640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecd40bceb0 0x7fecd40b9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd9d02640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecd40c4700 0x7fecd40ba260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd9d02640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd40aac10 con 0x7fecd40c4700 2023-10-24T23:53:52.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd9d02640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd4156a20 con 0x7fecd40bceb0 2023-10-24T23:53:52.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd9d02640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd4155970 con 0x7fecd40ab1a0 2023-10-24T23:53:52.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecd40c4700 0x7fecd40ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecd40c4700 0x7fecd40ba260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37302/0 (socket says 172.21.15.107:37302) 2023-10-24T23:53:52.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd3fff640 1 -- 172.21.15.107:0/2607120729 learned_addr learned my addr 172.21.15.107:0/2607120729 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:52.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd2ffd640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecd40bceb0 0x7fecd40b9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd3fff640 1 -- 172.21.15.107:0/2607120729 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fecd40ab1a0 msgr2=0x7fecd40c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd37fe640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fecd40ab1a0 0x7fecd40c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd3fff640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fecd40ab1a0 0x7fecd40c41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd3fff640 1 -- 172.21.15.107:0/2607120729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecd40bceb0 msgr2=0x7fecd40b9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd3fff640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecd40bceb0 0x7fecd40b9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd3fff640 1 -- 172.21.15.107:0/2607120729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc8009420 con 0x7fecd40c4700 2023-10-24T23:53:52.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd2ffd640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecd40bceb0 0x7fecd40b9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:52.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd3fff640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecd40c4700 0x7fecd40ba260 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7feccc00cad0 tx=0x7feccc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:52.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecbbfff640 1 -- 172.21.15.107:0/2607120729 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc00ac30 con 0x7fecd40c4700 2023-10-24T23:53:52.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecbbfff640 1 -- 172.21.15.107:0/2607120729 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feccc00ad90 con 0x7fecd40c4700 2023-10-24T23:53:52.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd9d02640 1 -- 172.21.15.107:0/2607120729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd4155640 con 0x7fecd40c4700 2023-10-24T23:53:52.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecbbfff640 1 -- 172.21.15.107:0/2607120729 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc012620 con 0x7fecd40c4700 2023-10-24T23:53:52.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd9d02640 1 -- 172.21.15.107:0/2607120729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd4156740 con 0x7fecd40c4700 2023-10-24T23:53:52.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd9d02640 1 -- 172.21.15.107:0/2607120729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd41554a0 con 0x7fecd40c4700 2023-10-24T23:53:52.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecbbfff640 1 -- 172.21.15.107:0/2607120729 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7feccc007070 con 0x7fecd40c4700 2023-10-24T23:53:52.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecbbfff640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fecc0065e60 0x7fecc0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecbbfff640 1 -- 172.21.15.107:0/2607120729 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7feccc083b20 con 0x7fecd40c4700 2023-10-24T23:53:52.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.189+0000 7fecd37fe640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fecc0065e60 0x7fecc0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.193+0000 7fecd37fe640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fecc0065e60 0x7fecc0068320 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fecc4009790 tx=0x7fecc4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:52.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.193+0000 7fecbbfff640 1 -- 172.21.15.107:0/2607120729 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fecd41554a0 con 0x7fecd40c4700 2023-10-24T23:53:52.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.337+0000 7fecd9d02640 1 -- 172.21.15.107:0/2607120729 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7fecd41554a0 con 0x7fecd40c4700 2023-10-24T23:53:52.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.337+0000 7fecbbfff640 1 -- 172.21.15.107:0/2607120729 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v29) v1 ==== 95+0+11 (secure 0 0 0) 0x7fecd41554a0 con 0x7fecd40c4700 2023-10-24T23:53:52.345 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-24T23:53:52.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 -- 172.21.15.107:0/2607120729 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fecc0065e60 msgr2=0x7fecc0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fecc0065e60 0x7fecc0068320 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fecc4009790 tx=0x7fecc4008040 comp rx=0 tx=0).stop 2023-10-24T23:53:52.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 -- 172.21.15.107:0/2607120729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecd40c4700 msgr2=0x7fecd40ba260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecd40c4700 0x7fecd40ba260 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7feccc00cad0 tx=0x7feccc00cf90 comp rx=0 tx=0).stop 2023-10-24T23:53:52.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 -- 172.21.15.107:0/2607120729 shutdown_connections 2023-10-24T23:53:52.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecd40c4700 0x7fecd40ba260 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fecd40bceb0 0x7fecd40b9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fecc0065e60 0x7fecc0068320 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 --2- 172.21.15.107:0/2607120729 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fecd40ab1a0 0x7fecd40c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 -- 172.21.15.107:0/2607120729 >> 172.21.15.107:0/2607120729 conn(0x7fecd40c2860 msgr2=0x7fecd40c2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:52.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 -- 172.21.15.107:0/2607120729 shutdown_connections 2023-10-24T23:53:52.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.341+0000 7fecd9d02640 1 -- 172.21.15.107:0/2607120729 wait complete. 2023-10-24T23:53:52.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:52.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.461+0000 7f0fccd8f640 1 Processor -- start 2023-10-24T23:53:52.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.461+0000 7f0fccd8f640 1 -- start start 2023-10-24T23:53:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fccd8f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fc806af70 0x7f0fc806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fccd8f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0fc806a230 0x7f0fc81551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fccd8f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc80c1860 con 0x7f0fc814b600 2023-10-24T23:53:52.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fccd8f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc8154d50 con 0x7f0fc806a230 2023-10-24T23:53:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fccd8f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc814b9e0 con 0x7f0fc806af70 2023-10-24T23:53:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fc6d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0fc806a230 0x7f0fc81551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fbdd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fc806af70 0x7f0fc806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fc659c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0fc814b600 0x7f0fc8151fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:46546/0 (socket says 172.21.15.107:46546) 2023-10-24T23:53:52.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fc6d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0fc806a230 0x7f0fc81551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54130/0 (socket says 172.21.15.107:54130) 2023-10-24T23:53:52.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fc659c640 1 -- 172.21.15.107:0/603323245 learned_addr learned my addr 172.21.15.107:0/603323245 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:52.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fc6d9d640 1 -- 172.21.15.107:0/603323245 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fc806af70 msgr2=0x7f0fc806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:52.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fc6d9d640 1 --2- 172.21.15.107:0/603323245 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fc806af70 0x7f0fc806c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fc6d9d640 1 -- 172.21.15.107:0/603323245 >> v1:172.21.15.110:6789/0 conn(0x7f0fc814b600 legacy=0x7f0fc8151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fc6d9d640 1 -- 172.21.15.107:0/603323245 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc80c1ab0 con 0x7f0fc806a230 2023-10-24T23:53:52.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fc6d9d640 1 --2- 172.21.15.107:0/603323245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fc806a230 0x7f0fc81551e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009d90 tx=0x7f0fc001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c843ab74845f2fb server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:52.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.465+0000 7f0fc5d9b640 1 -- 172.21.15.107:0/603323245 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00a5950 con 0x7f0fc806a230 2023-10-24T23:53:52.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fc5d9b640 1 -- 172.21.15.107:0/603323245 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc00a5ab0 con 0x7f0fc806a230 2023-10-24T23:53:52.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fc5d9b640 1 -- 172.21.15.107:0/603323245 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00a5e40 con 0x7f0fc806a230 2023-10-24T23:53:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- 172.21.15.107:0/603323245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fc806a230 msgr2=0x7f0fc81551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 --2- 172.21.15.107:0/603323245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fc806a230 0x7f0fc81551e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009d90 tx=0x7f0fc001c890 comp rx=0 tx=0).stop 2023-10-24T23:53:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- 172.21.15.107:0/603323245 shutdown_connections 2023-10-24T23:53:52.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 --2- 172.21.15.107:0/603323245 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fc806a230 0x7f0fc81551e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 --2- 172.21.15.107:0/603323245 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fc806af70 0x7f0fc806c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- 172.21.15.107:0/603323245 >> 172.21.15.107:0/603323245 conn(0x7f0fc80c2950 msgr2=0x7f0fc80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- 172.21.15.107:0/603323245 shutdown_connections 2023-10-24T23:53:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- 172.21.15.107:0/603323245 wait complete. 2023-10-24T23:53:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 Processor -- start 2023-10-24T23:53:52.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- start start 2023-10-24T23:53:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fc806a230 0x7f0fc8144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc806af70 0x7f0fc80bf790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fc814b600 0x7f0fc815a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc8159e50 con 0x7f0fc806af70 2023-10-24T23:53:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc80bfcd0 con 0x7f0fc814b600 2023-10-24T23:53:52.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc81433a0 con 0x7f0fc806a230 2023-10-24T23:53:52.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbdd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc806af70 0x7f0fc80bf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbdd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc806af70 0x7f0fc80bf790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37306/0 (socket says 172.21.15.107:37306) 2023-10-24T23:53:52.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbdd9b640 1 -- 172.21.15.107:0/2465206781 learned_addr learned my addr 172.21.15.107:0/2465206781 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:52.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fc659c640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fc806a230 0x7f0fc8144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fc6d9d640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fc814b600 0x7f0fc815a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbdd9b640 1 -- 172.21.15.107:0/2465206781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fc806a230 msgr2=0x7f0fc8144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbdd9b640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fc806a230 0x7f0fc8144a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbdd9b640 1 -- 172.21.15.107:0/2465206781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fc814b600 msgr2=0x7f0fc815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbdd9b640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fc814b600 0x7f0fc815a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbdd9b640 1 -- 172.21.15.107:0/2465206781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc0009a70 con 0x7f0fc806af70 2023-10-24T23:53:52.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbdd9b640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc806af70 0x7f0fc80bf790 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000e900 tx=0x7f0fb000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:52.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbf7fe640 1 -- 172.21.15.107:0/2465206781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0009690 con 0x7f0fc806af70 2023-10-24T23:53:52.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- 172.21.15.107:0/2465206781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc814ab10 con 0x7f0fc806af70 2023-10-24T23:53:52.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbf7fe640 1 -- 172.21.15.107:0/2465206781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb000ca00 con 0x7f0fc806af70 2023-10-24T23:53:52.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fbf7fe640 1 -- 172.21.15.107:0/2465206781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb000cdb0 con 0x7f0fc806af70 2023-10-24T23:53:52.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- 172.21.15.107:0/2465206781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc8147060 con 0x7f0fc806af70 2023-10-24T23:53:52.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.469+0000 7f0fccd8f640 1 -- 172.21.15.107:0/2465206781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc80c05c0 con 0x7f0fc806af70 2023-10-24T23:53:52.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.473+0000 7f0fbf7fe640 1 -- 172.21.15.107:0/2465206781 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0fb0013470 con 0x7f0fc806af70 2023-10-24T23:53:52.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.473+0000 7f0fbf7fe640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0fa8065e60 0x7f0fa8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.473+0000 7f0fbf7fe640 1 -- 172.21.15.107:0/2465206781 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f0fb0083e00 con 0x7f0fc806af70 2023-10-24T23:53:52.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.473+0000 7f0fc659c640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0fa8065e60 0x7f0fa8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.473+0000 7f0fc659c640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0fa8065e60 0x7f0fa8068320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0fc806bd40 tx=0x7f0fb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:52.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.477+0000 7f0fbf7fe640 1 -- 172.21.15.107:0/2465206781 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0fb00510d0 con 0x7f0fc806af70 2023-10-24T23:53:52.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 -- 172.21.15.107:0/2465206781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fc80746e0 con 0x7f0fc806af70 2023-10-24T23:53:52.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fbf7fe640 1 -- 172.21.15.107:0/2465206781 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f0fb00546f0 con 0x7f0fc806af70 2023-10-24T23:53:52.667 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:52.667 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:53:52.667 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:53:52.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 -- 172.21.15.107:0/2465206781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0fa8065e60 msgr2=0x7f0fa8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0fa8065e60 0x7f0fa8068320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0fc806bd40 tx=0x7f0fb8009040 comp rx=0 tx=0).stop 2023-10-24T23:53:52.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 -- 172.21.15.107:0/2465206781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc806af70 msgr2=0x7f0fc80bf790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc806af70 0x7f0fc80bf790 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000e900 tx=0x7f0fb000edc0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 -- 172.21.15.107:0/2465206781 shutdown_connections 2023-10-24T23:53:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fc814b600 0x7f0fc815a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fc806af70 0x7f0fc80bf790 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0fa8065e60 0x7f0fa8068320 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 --2- 172.21.15.107:0/2465206781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0fc806a230 0x7f0fc8144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 -- 172.21.15.107:0/2465206781 >> 172.21.15.107:0/2465206781 conn(0x7f0fc80c2950 msgr2=0x7f0fc81484e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:52.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 -- 172.21.15.107:0/2465206781 shutdown_connections 2023-10-24T23:53:52.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.661+0000 7f0fccd8f640 1 -- 172.21.15.107:0/2465206781 wait complete. 2023-10-24T23:53:52.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-24T23:53:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.793+0000 7ff8d5128640 1 Processor -- start 2023-10-24T23:53:52.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.793+0000 7ff8d5128640 1 -- start start 2023-10-24T23:53:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.793+0000 7ff8d5128640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d014b600 0x7ff8d0151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.793+0000 7ff8d5128640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff8d006a230 0x7ff8d01551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.793+0000 7ff8d5128640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d00c1860 con 0x7ff8d006af70 2023-10-24T23:53:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.793+0000 7ff8d5128640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d0154d50 con 0x7ff8d006a230 2023-10-24T23:53:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.793+0000 7ff8d5128640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01524f0 con 0x7ff8d014b600 2023-10-24T23:53:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.793+0000 7ff8ce59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff8d006af70 0x7ff8d006c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42074/0 (socket says 172.21.15.107:42074) 2023-10-24T23:53:52.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.793+0000 7ff8ce59c640 1 -- 172.21.15.107:0/3283636193 learned_addr learned my addr 172.21.15.107:0/3283636193 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52738060 0 0) 0x7ff8d00c1860 con 0x7ff8d006af70 2023-10-24T23:53:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8ced9d640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d014b600 0x7ff8d0151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cf59e640 1 --2- 172.21.15.107:0/3283636193 >> v2:172.21.15.110:3300/0 conn(0x7ff8d006a230 0x7ff8d01551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b8005740 con 0x7ff8d006af70 2023-10-24T23:53:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 320900579 0 0) 0x7ff8b8005740 con 0x7ff8d006af70 2023-10-24T23:53:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d014b600 msgr2=0x7ff8d0151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d014b600 0x7ff8d0151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 >> v2:172.21.15.110:3300/0 conn(0x7ff8d006a230 msgr2=0x7ff8d01551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 --2- 172.21.15.107:0/3283636193 >> v2:172.21.15.110:3300/0 conn(0x7ff8d006a230 0x7ff8d01551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d014b9e0 con 0x7ff8d006af70 2023-10-24T23:53:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7ff8c4001680 con 0x7ff8d006af70 2023-10-24T23:53:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8ced9d640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d014b600 0x7ff8d0151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 >> v1:172.21.15.107:6789/0 conn(0x7ff8d006af70 legacy=0x7ff8d006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8b8005b90 0x7ff8b8005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b80069e0 0x7ff8b8006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d014b600 0x7ff8b80073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d0154d50 con 0x7ff8b80069e0 2023-10-24T23:53:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01524f0 con 0x7ff8b8005b90 2023-10-24T23:53:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b800dcd0 con 0x7ff8d014b600 2023-10-24T23:53:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cf59e640 1 --2- 172.21.15.107:0/3283636193 >> v2:172.21.15.110:3300/0 conn(0x7ff8d006a230 0x7ff8d01551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:52.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8ce59c640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b80069e0 0x7ff8b8006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8ce59c640 1 -- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d014b600 msgr2=0x7ff8b80073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cf59e640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d014b600 0x7ff8b80073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8ced9d640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8b8005b90 0x7ff8b8005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8ce59c640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d014b600 0x7ff8b80073a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8ce59c640 1 -- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8b8005b90 msgr2=0x7ff8b8005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:52.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8ce59c640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8b8005b90 0x7ff8b8005fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8ce59c640 1 -- 172.21.15.107:0/3283636193 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b8005740 con 0x7ff8b80069e0 2023-10-24T23:53:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8ce59c640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b80069e0 0x7ff8b8006e60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff8d006b390 tx=0x7ff8c401cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda283f8da8656de server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c40058e0 con 0x7ff8b80069e0 2023-10-24T23:53:52.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c4005a60 con 0x7ff8b80069e0 2023-10-24T23:53:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cdd9b640 1 -- 172.21.15.107:0/3283636193 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4005e10 con 0x7ff8b80069e0 2023-10-24T23:53:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8d5128640 1 -- 172.21.15.107:0/3283636193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b80069e0 msgr2=0x7ff8b8006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8d5128640 1 --2- 172.21.15.107:0/3283636193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8b80069e0 0x7ff8b8006e60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff8d006b390 tx=0x7ff8c401cb90 comp rx=0 tx=0).stop 2023-10-24T23:53:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8cf59e640 1 -- 172.21.15.107:0/3283636193 reap_dead start 2023-10-24T23:53:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8d5128640 1 -- 172.21.15.107:0/3283636193 shutdown_connections 2023-10-24T23:53:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8d5128640 1 -- 172.21.15.107:0/3283636193 >> 172.21.15.107:0/3283636193 conn(0x7ff8d00c2950 msgr2=0x7ff8d00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8d5128640 1 -- 172.21.15.107:0/3283636193 shutdown_connections 2023-10-24T23:53:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.797+0000 7ff8d5128640 1 -- 172.21.15.107:0/3283636193 wait complete. 2023-10-24T23:53:52.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8d5128640 1 Processor -- start 2023-10-24T23:53:52.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8d5128640 1 -- start start 2023-10-24T23:53:52.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8d5128640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8d01e0390 0x7ff8d01f0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8d5128640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d01e0770 0x7ff8d0144450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8d5128640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8d0144990 0x7ff8d00c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8d5128640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d00c4c20 con 0x7ff8d01e0390 2023-10-24T23:53:52.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8d5128640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d01f10e0 con 0x7ff8d0144990 2023-10-24T23:53:52.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8d5128640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d014a230 con 0x7ff8d01e0770 2023-10-24T23:53:52.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8ced9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8d01e0390 0x7ff8d01f0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8ced9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8d01e0390 0x7ff8d01f0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45364/0 (socket says 172.21.15.107:45364) 2023-10-24T23:53:52.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8ced9d640 1 -- 172.21.15.107:0/106097244 learned_addr learned my addr 172.21.15.107:0/106097244 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:52.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8ce59c640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d01e0770 0x7ff8d0144450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8cf59e640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8d0144990 0x7ff8d00c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8ced9d640 1 -- 172.21.15.107:0/106097244 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d01e0770 msgr2=0x7ff8d0144450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8ced9d640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d01e0770 0x7ff8d0144450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8ced9d640 1 -- 172.21.15.107:0/106097244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8d0144990 msgr2=0x7ff8d00c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8ced9d640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8d0144990 0x7ff8d00c5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8ced9d640 1 -- 172.21.15.107:0/106097244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d014a510 con 0x7ff8d01e0390 2023-10-24T23:53:52.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8ced9d640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8d01e0390 0x7ff8d01f0ba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000f510 tx=0x7ff8c000f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:52.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8b77fe640 1 -- 172.21.15.107:0/106097244 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0014890 con 0x7ff8d01e0390 2023-10-24T23:53:52.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8b77fe640 1 -- 172.21.15.107:0/106097244 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c000c030 con 0x7ff8d01e0390 2023-10-24T23:53:52.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8d5128640 1 -- 172.21.15.107:0/106097244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d014a7f0 con 0x7ff8d01e0390 2023-10-24T23:53:52.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8b77fe640 1 -- 172.21.15.107:0/106097244 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0014c30 con 0x7ff8d01e0390 2023-10-24T23:53:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8d5128640 1 -- 172.21.15.107:0/106097244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d0144e10 con 0x7ff8d01e0390 2023-10-24T23:53:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.801+0000 7ff8d5128640 1 -- 172.21.15.107:0/106097244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d014aca0 con 0x7ff8d01e0390 2023-10-24T23:53:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.805+0000 7ff8b77fe640 1 -- 172.21.15.107:0/106097244 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff8c00133e0 con 0x7ff8d01e0390 2023-10-24T23:53:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.805+0000 7ff8b77fe640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8bc065e90 0x7ff8bc068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.805+0000 7ff8b77fe640 1 -- 172.21.15.107:0/106097244 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7ff8c0083c80 con 0x7ff8d01e0390 2023-10-24T23:53:52.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.805+0000 7ff8ce59c640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8bc065e90 0x7ff8bc068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.805+0000 7ff8ce59c640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8bc065e90 0x7ff8bc068350 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40aa000 tx=0x7ff8d006f430 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:52.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.809+0000 7ff8b77fe640 1 -- 172.21.15.107:0/106097244 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff8c0050fd0 con 0x7ff8d01e0390 2023-10-24T23:53:52.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.953+0000 7ff8d5128640 1 -- 172.21.15.107:0/106097244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8d00746e0 con 0x7ff8d01e0390 2023-10-24T23:53:52.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8b77fe640 1 -- 172.21.15.107:0/106097244 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+21618 (secure 0 0 0) 0x7ff8c0022090 con 0x7ff8d01e0390 2023-10-24T23:53:52.963 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:52.963 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-24T23:53:44.462555+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2f","mappings":[{"from":3,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":2}]},{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-24T23:53:52.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 -- 172.21.15.107:0/106097244 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8bc065e90 msgr2=0x7ff8bc068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8bc065e90 0x7ff8bc068350 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40aa000 tx=0x7ff8d006f430 comp rx=0 tx=0).stop 2023-10-24T23:53:52.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 -- 172.21.15.107:0/106097244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8d01e0390 msgr2=0x7ff8d01f0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:52.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8d01e0390 0x7ff8d01f0ba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000f510 tx=0x7ff8c000f9d0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 -- 172.21.15.107:0/106097244 shutdown_connections 2023-10-24T23:53:52.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8d0144990 0x7ff8d00c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8bc065e90 0x7ff8bc068350 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8d01e0770 0x7ff8d0144450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 --2- 172.21.15.107:0/106097244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8d01e0390 0x7ff8d01f0ba0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:52.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 -- 172.21.15.107:0/106097244 >> 172.21.15.107:0/106097244 conn(0x7ff8d00c2950 msgr2=0x7ff8d006c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:52.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 -- 172.21.15.107:0/106097244 shutdown_connections 2023-10-24T23:53:52.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:52.957+0000 7ff8d5128640 1 -- 172.21.15.107:0/106097244 wait complete. 2023-10-24T23:53:52.980 DEBUG:tasks.mds_thrash:Yielding 2023-10-24T23:53:52.980 INFO:teuthology.run_tasks:Running task workunit... 2023-10-24T23:53:52.993 INFO:tasks.workunit:Pulling workunits from ref e652850897f60c32cb9cc05022cf9f93e3b00db1 2023-10-24T23:53:52.994 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-24T23:53:52.994 INFO:tasks.workunit:timeout=3h 2023-10-24T23:53:52.994 INFO:tasks.workunit:cleanup=True 2023-10-24T23:53:52.995 DEBUG:teuthology.orchestra.run.smithi107:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-24T23:53:52.995 INFO:tasks.mds_thrash.fs.[b]:starting mds_do_thrash for fs b 2023-10-24T23:53:52.995 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before thrashing 2023-10-24T23:53:53.002 INFO:teuthology.orchestra.run.smithi107.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-10-24T23:53:53.002 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2023-10-24T23:53:53.002 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 2dh/45d Inode: 1 Links: 2 2023-10-24T23:53:53.002 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-24T23:53:53.003 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-10-24T23:53:53.003 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-10-24 23:53:02.685223286 +0000 2023-10-24T23:53:53.003 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-10-24 23:53:41.777334722 +0000 2023-10-24T23:53:53.003 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-10-24T23:53:53.003 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-10-24T23:53:53.004 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-10-24T23:53:53.067 DEBUG:teuthology.orchestra.run.smithi110:> stat -- /home/ubuntu/cephtest/mnt.1 2023-10-24T23:53:53.076 INFO:teuthology.orchestra.run.smithi110.stdout: File: /home/ubuntu/cephtest/mnt.1 2023-10-24T23:53:53.076 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2023-10-24T23:53:53.076 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 2dh/45d Inode: 1 Links: 2 2023-10-24T23:53:53.076 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-24T23:53:53.077 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-10-24T23:53:53.077 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-10-24 23:53:11.781607342 +0000 2023-10-24T23:53:53.077 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-10-24 23:53:41.842529183 +0000 2023-10-24T23:53:53.077 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-10-24T23:53:53.078 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2023-10-24T23:53:53.078 DEBUG:teuthology.orchestra.run.smithi110:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2023-10-24T23:53:53.140 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e652850897f60c32cb9cc05022cf9f93e3b00db1 2023-10-24T23:53:53.141 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout e652850897f60c32cb9cc05022cf9f93e3b00db1 2023-10-24T23:53:53.158 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-24T23:53:53.188 INFO:tasks.workunit.client.1.smithi110.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2023-10-24T23:53:58.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:59.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.101+0000 7f258454c640 1 Processor -- start 2023-10-24T23:53:59.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.101+0000 7f258454c640 1 -- start start 2023-10-24T23:53:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c155c20 0x7f257c156000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f257c0a9450 0x7f257c0c30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f257c14cd80 con 0x7f257c0720f0 2023-10-24T23:53:59.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f257c06f080 con 0x7f257c0a9450 2023-10-24T23:53:59.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0aa740 con 0x7f257c155c20 2023-10-24T23:53:59.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f2582ae9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f257c0a9450 0x7f257c0c30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:59.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f2582ae9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f257c0a9450 0x7f257c0c30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45366/0 (socket says 172.21.15.107:45366) 2023-10-24T23:53:59.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f2582ae9640 1 -- 172.21.15.107:0/1855481809 learned_addr learned my addr 172.21.15.107:0/1855481809 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:59.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f25822e8640 1 --2- 172.21.15.107:0/1855481809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c155c20 0x7f257c156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:59.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f2582ae9640 1 -- 172.21.15.107:0/1855481809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c155c20 msgr2=0x7f257c156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:59.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f2582ae9640 1 --2- 172.21.15.107:0/1855481809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c155c20 0x7f257c156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f2582ae9640 1 -- 172.21.15.107:0/1855481809 >> v1:172.21.15.110:6789/0 conn(0x7f257c0720f0 legacy=0x7f257c0c29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:59.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f2582ae9640 1 -- 172.21.15.107:0/1855481809 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c06e350 con 0x7f257c0a9450 2023-10-24T23:53:59.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f25822e8640 1 --2- 172.21.15.107:0/1855481809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c155c20 0x7f257c156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:59.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f2582ae9640 1 --2- 172.21.15.107:0/1855481809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f257c0a9450 0x7f257c0c30d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2574009e50 tx=0x7f257401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f265f5c041ca22 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:59.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f25812e6640 1 -- 172.21.15.107:0/1855481809 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25740a5790 con 0x7f257c0a9450 2023-10-24T23:53:59.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f25812e6640 1 -- 172.21.15.107:0/1855481809 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25740a58f0 con 0x7f257c0a9450 2023-10-24T23:53:59.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f25812e6640 1 -- 172.21.15.107:0/1855481809 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25740a5c80 con 0x7f257c0a9450 2023-10-24T23:53:59.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 -- 172.21.15.107:0/1855481809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f257c0a9450 msgr2=0x7f257c0c30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:59.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 --2- 172.21.15.107:0/1855481809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f257c0a9450 0x7f257c0c30d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2574009e50 tx=0x7f257401c910 comp rx=0 tx=0).stop 2023-10-24T23:53:59.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 -- 172.21.15.107:0/1855481809 shutdown_connections 2023-10-24T23:53:59.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 --2- 172.21.15.107:0/1855481809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f257c0a9450 0x7f257c0c30d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 --2- 172.21.15.107:0/1855481809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c155c20 0x7f257c156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 -- 172.21.15.107:0/1855481809 >> 172.21.15.107:0/1855481809 conn(0x7f257c065930 msgr2=0x7f257c065d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:59.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 -- 172.21.15.107:0/1855481809 shutdown_connections 2023-10-24T23:53:59.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.105+0000 7f258454c640 1 -- 172.21.15.107:0/1855481809 wait complete. 2023-10-24T23:53:59.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f258454c640 1 Processor -- start 2023-10-24T23:53:59.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f258454c640 1 -- start start 2023-10-24T23:53:59.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f258454c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c0720f0 0x7f257c14bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:59.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f258454c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f257c0a9450 0x7f257c151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:59.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f258454c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f257c155c20 0x7f257c1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:59.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f258454c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c0aae00 con 0x7f257c0a9450 2023-10-24T23:53:59.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f258454c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257c151770 con 0x7f257c155c20 2023-10-24T23:53:59.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f258454c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257c1ef660 con 0x7f257c0720f0 2023-10-24T23:53:59.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2581ae7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f257c0a9450 0x7f257c151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:59.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2581ae7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f257c0a9450 0x7f257c151c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45370/0 (socket says 172.21.15.107:45370) 2023-10-24T23:53:59.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2581ae7640 1 -- 172.21.15.107:0/3447264144 learned_addr learned my addr 172.21.15.107:0/3447264144 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:59.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f25822e8640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c0720f0 0x7f257c14bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:59.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2582ae9640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f257c155c20 0x7f257c1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:59.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2581ae7640 1 -- 172.21.15.107:0/3447264144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c0720f0 msgr2=0x7f257c14bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:59.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2581ae7640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c0720f0 0x7f257c14bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2581ae7640 1 -- 172.21.15.107:0/3447264144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f257c155c20 msgr2=0x7f257c1521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:59.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2581ae7640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f257c155c20 0x7f257c1521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2581ae7640 1 -- 172.21.15.107:0/3447264144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2574009b30 con 0x7f257c0a9450 2023-10-24T23:53:59.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2582ae9640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f257c155c20 0x7f257c1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:53:59.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f25822e8640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c0720f0 0x7f257c14bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:53:59.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2581ae7640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f257c0a9450 0x7f257c151c70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f256c00b3e0 tx=0x7f256c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:59.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2572ffd640 1 -- 172.21.15.107:0/3447264144 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c011830 con 0x7f257c0a9450 2023-10-24T23:53:59.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2572ffd640 1 -- 172.21.15.107:0/3447264144 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c011990 con 0x7f257c0a9450 2023-10-24T23:53:59.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f258454c640 1 -- 172.21.15.107:0/3447264144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f257c0ab0c0 con 0x7f257c0a9450 2023-10-24T23:53:59.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2572ffd640 1 -- 172.21.15.107:0/3447264144 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c011d80 con 0x7f257c0a9450 2023-10-24T23:53:59.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f258454c640 1 -- 172.21.15.107:0/3447264144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257c14c300 con 0x7f257c0a9450 2023-10-24T23:53:59.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f258454c640 1 -- 172.21.15.107:0/3447264144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f257c06d3f0 con 0x7f257c0a9450 2023-10-24T23:53:59.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.109+0000 7f2572ffd640 1 -- 172.21.15.107:0/3447264144 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f256c01a400 con 0x7f257c0a9450 2023-10-24T23:53:59.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.113+0000 7f2572ffd640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2560065e60 0x7f2560068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:59.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.113+0000 7f2572ffd640 1 -- 172.21.15.107:0/3447264144 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f256c083970 con 0x7f257c0a9450 2023-10-24T23:53:59.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.113+0000 7f25822e8640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2560065e60 0x7f2560068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:59.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.113+0000 7f25822e8640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2560065e60 0x7f2560068320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2578009970 tx=0x7f2578008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:59.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.117+0000 7f2572ffd640 1 -- 172.21.15.107:0/3447264144 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f256c050cc0 con 0x7f257c0a9450 2023-10-24T23:53:59.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 -- 172.21.15.107:0/3447264144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f257c0746e0 con 0x7f257c0a9450 2023-10-24T23:53:59.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f2572ffd640 1 -- 172.21.15.107:0/3447264144 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f256c0542e0 con 0x7f257c0a9450 2023-10-24T23:53:59.307 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:59.307 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:53:59.307 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:53:59.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 -- 172.21.15.107:0/3447264144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2560065e60 msgr2=0x7f2560068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:59.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2560065e60 0x7f2560068320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2578009970 tx=0x7f2578008040 comp rx=0 tx=0).stop 2023-10-24T23:53:59.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 -- 172.21.15.107:0/3447264144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f257c0a9450 msgr2=0x7f257c151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:59.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f257c0a9450 0x7f257c151c70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f256c00b3e0 tx=0x7f256c00b8a0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 -- 172.21.15.107:0/3447264144 shutdown_connections 2023-10-24T23:53:59.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f257c155c20 0x7f257c1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f257c0a9450 0x7f257c151c70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2560065e60 0x7f2560068320 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 --2- 172.21.15.107:0/3447264144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f257c0720f0 0x7f257c14bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 -- 172.21.15.107:0/3447264144 >> 172.21.15.107:0/3447264144 conn(0x7f257c065930 msgr2=0x7f257c154f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:59.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 -- 172.21.15.107:0/3447264144 shutdown_connections 2023-10-24T23:53:59.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.301+0000 7f258454c640 1 -- 172.21.15.107:0/3447264144 wait complete. 2023-10-24T23:53:59.325 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2023-10-24T23:53:59.325 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-24T23:53:59.326 INFO:teuthology.orchestra.run:waiting for 300 2023-10-24T23:53:59.335 INFO:tasks.ceph.mds.a:Stopped 2023-10-24T23:53:59.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:53:59.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.481+0000 7f3804858640 1 Processor -- start 2023-10-24T23:53:59.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.481+0000 7f3804858640 1 -- start start 2023-10-24T23:53:59.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.485+0000 7f3804858640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f380014eb70 0x7f38000c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:59.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.485+0000 7f3804858640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38000c22f0 con 0x7f380006df30 2023-10-24T23:53:59.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.485+0000 7f3804858640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38000c25d0 con 0x7f3800070b00 2023-10-24T23:53:59.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.485+0000 7f3804858640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3800071700 con 0x7f380014eb70 2023-10-24T23:53:59.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.485+0000 7f37fed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f380006df30 0x7f380015be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42084/0 (socket says 172.21.15.107:42084) 2023-10-24T23:53:59.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.485+0000 7f37fed9d640 1 -- 172.21.15.107:0/3608003311 learned_addr learned my addr 172.21.15.107:0/3608003311 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:53:59.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.485+0000 7f37fe59c640 1 --2- 172.21.15.107:0/3608003311 >> v2:172.21.15.107:3300/0 conn(0x7f380014eb70 0x7f38000c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:59.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.485+0000 7f37fe59c640 1 -- 172.21.15.107:0/3608003311 >> v1:172.21.15.110:6789/0 conn(0x7f3800070b00 legacy=0x7f38001514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:53:59.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.485+0000 7f37fe59c640 1 -- 172.21.15.107:0/3608003311 >> v1:172.21.15.107:6789/0 conn(0x7f380006df30 legacy=0x7f380015be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:59.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.485+0000 7f37fe59c640 1 -- 172.21.15.107:0/3608003311 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38000c2be0 con 0x7f380014eb70 2023-10-24T23:53:59.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.485+0000 7f37fe59c640 1 --2- 172.21.15.107:0/3608003311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380014eb70 0x7f38000c33e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f37f00097c0 tx=0x7f37f001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e53627def8da25 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:59.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f37fd59a640 1 -- 172.21.15.107:0/3608003311 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f0004230 con 0x7f380014eb70 2023-10-24T23:53:59.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f37fd59a640 1 -- 172.21.15.107:0/3608003311 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f00a7050 con 0x7f380014eb70 2023-10-24T23:53:59.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f37fd59a640 1 -- 172.21.15.107:0/3608003311 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f0003620 con 0x7f380014eb70 2023-10-24T23:53:59.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f3804858640 1 -- 172.21.15.107:0/3608003311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380014eb70 msgr2=0x7f38000c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:59.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f3804858640 1 --2- 172.21.15.107:0/3608003311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380014eb70 0x7f38000c33e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f37f00097c0 tx=0x7f37f001c440 comp rx=0 tx=0).stop 2023-10-24T23:53:59.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f3804858640 1 -- 172.21.15.107:0/3608003311 shutdown_connections 2023-10-24T23:53:59.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f3804858640 1 --2- 172.21.15.107:0/3608003311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380014eb70 0x7f38000c33e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f3804858640 1 -- 172.21.15.107:0/3608003311 >> 172.21.15.107:0/3608003311 conn(0x7f38000c39a0 msgr2=0x7f38000c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:59.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f3804858640 1 -- 172.21.15.107:0/3608003311 shutdown_connections 2023-10-24T23:53:59.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f3804858640 1 -- 172.21.15.107:0/3608003311 wait complete. 2023-10-24T23:53:59.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f3804858640 1 Processor -- start 2023-10-24T23:53:59.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.489+0000 7f3804858640 1 -- start start 2023-10-24T23:53:59.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f3804858640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006df30 0x7f38000ab330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:59.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f3804858640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3800070b00 0x7f38001ead30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:59.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f3804858640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f380014eb70 0x7f38001eeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:59.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f3804858640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38000ab870 con 0x7f380006df30 2023-10-24T23:53:59.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f3804858640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38000a9ef0 con 0x7f380014eb70 2023-10-24T23:53:59.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f3804858640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38001eb270 con 0x7f3800070b00 2023-10-24T23:53:59.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006df30 0x7f38000ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:59.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006df30 0x7f38000ab330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45386/0 (socket says 172.21.15.107:45386) 2023-10-24T23:53:59.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fe59c640 1 -- 172.21.15.107:0/572262267 learned_addr learned my addr 172.21.15.107:0/572262267 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:53:59.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fdd9b640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3800070b00 0x7f38001ead30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:59.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fe59c640 1 -- 172.21.15.107:0/572262267 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3800070b00 msgr2=0x7f38001ead30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:59.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fe59c640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3800070b00 0x7f38001ead30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fe59c640 1 -- 172.21.15.107:0/572262267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f380014eb70 msgr2=0x7f38001eeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:59.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fed9d640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f380014eb70 0x7f38001eeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:59.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fe59c640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f380014eb70 0x7f38001eeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fe59c640 1 -- 172.21.15.107:0/572262267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f0009420 con 0x7f380006df30 2023-10-24T23:53:59.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fe59c640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006df30 0x7f38000ab330 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f37f0019050 tx=0x7f37f00191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:59.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37eeffd640 1 -- 172.21.15.107:0/572262267 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f0003c70 con 0x7f380006df30 2023-10-24T23:53:59.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37eeffd640 1 -- 172.21.15.107:0/572262267 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f0003df0 con 0x7f380006df30 2023-10-24T23:53:59.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f3804858640 1 -- 172.21.15.107:0/572262267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38001ee3a0 con 0x7f380006df30 2023-10-24T23:53:59.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37eeffd640 1 -- 172.21.15.107:0/572262267 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f00ab680 con 0x7f380006df30 2023-10-24T23:53:59.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f3804858640 1 -- 172.21.15.107:0/572262267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38000abd30 con 0x7f380006df30 2023-10-24T23:53:59.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f3804858640 1 -- 172.21.15.107:0/572262267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38000746e0 con 0x7f380006df30 2023-10-24T23:53:59.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37eeffd640 1 -- 172.21.15.107:0/572262267 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f37f00b2040 con 0x7f380006df30 2023-10-24T23:53:59.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37eeffd640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37e0065e60 0x7f37e0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:53:59.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37eeffd640 1 -- 172.21.15.107:0/572262267 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f37f011dc00 con 0x7f380006df30 2023-10-24T23:53:59.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fdd9b640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37e0065e60 0x7f37e0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:53:59.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.493+0000 7f37fdd9b640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37e0065e60 0x7f37e0068320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f38001ebec0 tx=0x7f37f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:53:59.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.497+0000 7f37eeffd640 1 -- 172.21.15.107:0/572262267 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f37f0124080 con 0x7f380006df30 2023-10-24T23:53:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.697+0000 7f3804858640 1 -- 172.21.15.107:0/572262267 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38000c1ed0 con 0x7f380006df30 2023-10-24T23:53:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f37eeffd640 1 -- 172.21.15.107:0/572262267 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f37f00b9090 con 0x7f380006df30 2023-10-24T23:53:59.706 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:53:59.706 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:53:59.707 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:53:59.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 -- 172.21.15.107:0/572262267 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37e0065e60 msgr2=0x7f37e0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:59.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37e0065e60 0x7f37e0068320 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f38001ebec0 tx=0x7f37f4009040 comp rx=0 tx=0).stop 2023-10-24T23:53:59.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 -- 172.21.15.107:0/572262267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006df30 msgr2=0x7f38000ab330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:53:59.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006df30 0x7f38000ab330 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f37f0019050 tx=0x7f37f00191e0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 -- 172.21.15.107:0/572262267 shutdown_connections 2023-10-24T23:53:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f380014eb70 0x7f38001eeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37e0065e60 0x7f37e0068320 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3800070b00 0x7f38001ead30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 --2- 172.21.15.107:0/572262267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f380006df30 0x7f38000ab330 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:53:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 -- 172.21.15.107:0/572262267 >> 172.21.15.107:0/572262267 conn(0x7f38000c39a0 msgr2=0x7f38000c4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:53:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 -- 172.21.15.107:0/572262267 shutdown_connections 2023-10-24T23:53:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:53:59.701+0000 7f3804858640 1 -- 172.21.15.107:0/572262267 wait complete. 2023-10-24T23:53:59.724 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:54:01.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:54:01.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.857+0000 7fba1d8b9640 1 Processor -- start 2023-10-24T23:54:01.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.857+0000 7fba1d8b9640 1 -- start start 2023-10-24T23:54:01.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba1d8b9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fba180c4e20 0x7fba180c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:01.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba1d8b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fba180c5ba0 0x7fba180c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:01.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba1d8b9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba18155f20 con 0x7fba1814fbd0 2023-10-24T23:54:01.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba1d8b9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba180c6d40 con 0x7fba180c5ba0 2023-10-24T23:54:01.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba1d8b9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba180c2320 con 0x7fba180c4e20 2023-10-24T23:54:01.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba16ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fba1814fbd0 0x7fba1806df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42100/0 (socket says 172.21.15.107:42100) 2023-10-24T23:54:01.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba16ffd640 1 -- 172.21.15.107:0/3412505426 learned_addr learned my addr 172.21.15.107:0/3412505426 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:54:01.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba177fe640 1 --2- 172.21.15.107:0/3412505426 >> v2:172.21.15.107:3300/0 conn(0x7fba180c5ba0 0x7fba180c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:01.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba167fc640 1 --2- 172.21.15.107:0/3412505426 >> v2:172.21.15.110:3300/0 conn(0x7fba180c4e20 0x7fba180c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:01.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba177fe640 1 -- 172.21.15.107:0/3412505426 >> v2:172.21.15.110:3300/0 conn(0x7fba180c4e20 msgr2=0x7fba180c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:01.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba177fe640 1 --2- 172.21.15.107:0/3412505426 >> v2:172.21.15.110:3300/0 conn(0x7fba180c4e20 0x7fba180c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:01.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba177fe640 1 -- 172.21.15.107:0/3412505426 >> v1:172.21.15.107:6789/0 conn(0x7fba1814fbd0 legacy=0x7fba1806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:01.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.861+0000 7fba177fe640 1 -- 172.21.15.107:0/3412505426 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba18153560 con 0x7fba180c5ba0 2023-10-24T23:54:01.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba177fe640 1 --2- 172.21.15.107:0/3412505426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba180c5ba0 0x7fba180c1de0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fba180c56b0 tx=0x7fba1001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f708f592fe450506 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:01.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba15ffb640 1 -- 172.21.15.107:0/3412505426 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba100a5830 con 0x7fba180c5ba0 2023-10-24T23:54:01.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba15ffb640 1 -- 172.21.15.107:0/3412505426 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba100a5990 con 0x7fba180c5ba0 2023-10-24T23:54:01.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba15ffb640 1 -- 172.21.15.107:0/3412505426 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba100a5d20 con 0x7fba180c5ba0 2023-10-24T23:54:01.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3412505426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba180c5ba0 msgr2=0x7fba180c1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:01.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 --2- 172.21.15.107:0/3412505426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba180c5ba0 0x7fba180c1de0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fba180c56b0 tx=0x7fba1001c880 comp rx=0 tx=0).stop 2023-10-24T23:54:01.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3412505426 shutdown_connections 2023-10-24T23:54:01.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 --2- 172.21.15.107:0/3412505426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba180c5ba0 0x7fba180c1de0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:01.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 --2- 172.21.15.107:0/3412505426 >> v2:172.21.15.110:3300/0 conn(0x7fba180c4e20 0x7fba180c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:01.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3412505426 >> 172.21.15.107:0/3412505426 conn(0x7fba180c39a0 msgr2=0x7fba180c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:01.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3412505426 shutdown_connections 2023-10-24T23:54:01.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3412505426 wait complete. 2023-10-24T23:54:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 Processor -- start 2023-10-24T23:54:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 -- start start 2023-10-24T23:54:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba180c4e20 0x7fba1815b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba180c5ba0 0x7fba18142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba1814fbd0 0x7fba18142d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba180706b0 con 0x7fba180c5ba0 2023-10-24T23:54:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba18143580 con 0x7fba180c4e20 2023-10-24T23:54:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba18070060 con 0x7fba1814fbd0 2023-10-24T23:54:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba167fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba180c5ba0 0x7fba18142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba167fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba180c5ba0 0x7fba18142800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45402/0 (socket says 172.21.15.107:45402) 2023-10-24T23:54:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba167fc640 1 -- 172.21.15.107:0/3946570840 learned_addr learned my addr 172.21.15.107:0/3946570840 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba177fe640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba1814fbd0 0x7fba18142d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba16ffd640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba180c4e20 0x7fba1815b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba167fc640 1 -- 172.21.15.107:0/3946570840 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba1814fbd0 msgr2=0x7fba18142d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba167fc640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba1814fbd0 0x7fba18142d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba167fc640 1 -- 172.21.15.107:0/3946570840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba180c4e20 msgr2=0x7fba1815b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba167fc640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba180c4e20 0x7fba1815b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba167fc640 1 -- 172.21.15.107:0/3946570840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba10009a70 con 0x7fba180c5ba0 2023-10-24T23:54:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba167fc640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba180c5ba0 0x7fba18142800 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00e930 tx=0x7fba0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fb9fffff640 1 -- 172.21.15.107:0/3946570840 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c009690 con 0x7fba180c5ba0 2023-10-24T23:54:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fb9fffff640 1 -- 172.21.15.107:0/3946570840 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c00ca00 con 0x7fba180c5ba0 2023-10-24T23:54:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3946570840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba18070340 con 0x7fba180c5ba0 2023-10-24T23:54:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fb9fffff640 1 -- 172.21.15.107:0/3946570840 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c00cd70 con 0x7fba180c5ba0 2023-10-24T23:54:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.865+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3946570840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1815c080 con 0x7fba180c5ba0 2023-10-24T23:54:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.869+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3946570840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba18070820 con 0x7fba180c5ba0 2023-10-24T23:54:01.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.869+0000 7fb9fffff640 1 -- 172.21.15.107:0/3946570840 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fba0c013470 con 0x7fba180c5ba0 2023-10-24T23:54:01.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.869+0000 7fb9fffff640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba04065e90 0x7fba04068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:01.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.869+0000 7fb9fffff640 1 -- 172.21.15.107:0/3946570840 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7fba0c083dc0 con 0x7fba180c5ba0 2023-10-24T23:54:01.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.869+0000 7fba16ffd640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba04065e90 0x7fba04068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:01.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.869+0000 7fba16ffd640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba04065e90 0x7fba04068350 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fba08005890 tx=0x7fba08009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:01.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:01.873+0000 7fb9fffff640 1 -- 172.21.15.107:0/3946570840 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fba0c0511d0 con 0x7fba180c5ba0 2023-10-24T23:54:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.073+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3946570840 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba180746e0 con 0x7fba180c5ba0 2023-10-24T23:54:02.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.073+0000 7fb9fffff640 1 -- 172.21.15.107:0/3946570840 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7fba0c017090 con 0x7fba180c5ba0 2023-10-24T23:54:02.081 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:54:02.081 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:54:02.082 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:54:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3946570840 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba04065e90 msgr2=0x7fba04068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba04065e90 0x7fba04068350 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fba08005890 tx=0x7fba08009040 comp rx=0 tx=0).stop 2023-10-24T23:54:02.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3946570840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba180c5ba0 msgr2=0x7fba18142800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba180c5ba0 0x7fba18142800 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00e930 tx=0x7fba0c00edf0 comp rx=0 tx=0).stop 2023-10-24T23:54:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3946570840 shutdown_connections 2023-10-24T23:54:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba04065e90 0x7fba04068350 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba1814fbd0 0x7fba18142d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba180c5ba0 0x7fba18142800 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 --2- 172.21.15.107:0/3946570840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba180c4e20 0x7fba1815b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3946570840 >> 172.21.15.107:0/3946570840 conn(0x7fba180c39a0 msgr2=0x7fba181492e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3946570840 shutdown_connections 2023-10-24T23:54:02.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:02.077+0000 7fba1d8b9640 1 -- 172.21.15.107:0/3946570840 wait complete. 2023-10-24T23:54:02.100 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:54:03.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:54:03.963 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:54:04.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:54:04.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d93a2b640 1 Processor -- start 2023-10-24T23:54:04.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d93a2b640 1 -- start start 2023-10-24T23:54:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d93a2b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0d8c0aa330 0x7f0d8c06e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:04.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d93a2b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0d8c0a9450 0x7f0d8c071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:04.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d93a2b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c152d40 con 0x7f0d8c13bb00 2023-10-24T23:54:04.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d93a2b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d8c156100 con 0x7f0d8c0aa330 2023-10-24T23:54:04.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d93a2b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c1527c0 con 0x7f0d8c0a9450 2023-10-24T23:54:04.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d90fc6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0d8c0aa330 0x7f0d8c06e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:04.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d90fc6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0d8c0aa330 0x7f0d8c06e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52638/0 (socket says 172.21.15.107:52638) 2023-10-24T23:54:04.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d90fc6640 1 -- 172.21.15.107:0/2825974837 learned_addr learned my addr 172.21.15.107:0/2825974837 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:04.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d91fc8640 1 --2- 172.21.15.107:0/2825974837 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0d8c0a9450 0x7f0d8c071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:04.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d90fc6640 1 -- 172.21.15.107:0/2825974837 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0d8c0a9450 msgr2=0x7f0d8c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:04.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d90fc6640 1 --2- 172.21.15.107:0/2825974837 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0d8c0a9450 0x7f0d8c071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:04.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d90fc6640 1 -- 172.21.15.107:0/2825974837 >> v1:172.21.15.110:6789/0 conn(0x7f0d8c13bb00 legacy=0x7f0d8c1531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:04.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.241+0000 7f0d90fc6640 1 -- 172.21.15.107:0/2825974837 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c06d660 con 0x7f0d8c0aa330 2023-10-24T23:54:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d90fc6640 1 --2- 172.21.15.107:0/2825974837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d8c0aa330 0x7f0d8c06e250 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c009870 tx=0x7f0d7c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919f30578dc320b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d83fff640 1 -- 172.21.15.107:0/2825974837 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c0ac640 con 0x7f0d8c0aa330 2023-10-24T23:54:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d83fff640 1 -- 172.21.15.107:0/2825974837 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7c0ac7a0 con 0x7f0d8c0aa330 2023-10-24T23:54:04.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d83fff640 1 -- 172.21.15.107:0/2825974837 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c0acb30 con 0x7f0d8c0aa330 2023-10-24T23:54:04.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2825974837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d8c0aa330 msgr2=0x7f0d8c06e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:04.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d93a2b640 1 --2- 172.21.15.107:0/2825974837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d8c0aa330 0x7f0d8c06e250 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c009870 tx=0x7f0d7c01c420 comp rx=0 tx=0).stop 2023-10-24T23:54:04.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2825974837 shutdown_connections 2023-10-24T23:54:04.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d93a2b640 1 --2- 172.21.15.107:0/2825974837 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0d8c0a9450 0x7f0d8c071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:04.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d93a2b640 1 --2- 172.21.15.107:0/2825974837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d8c0aa330 0x7f0d8c06e250 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2825974837 >> 172.21.15.107:0/2825974837 conn(0x7f0d8c070f60 msgr2=0x7f0d8c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2825974837 shutdown_connections 2023-10-24T23:54:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.245+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2825974837 wait complete. 2023-10-24T23:54:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d93a2b640 1 Processor -- start 2023-10-24T23:54:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d93a2b640 1 -- start start 2023-10-24T23:54:04.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d93a2b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0d8c0a9450 0x7f0d8c0c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:04.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d93a2b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d8c0aa330 0x7f0d8c1eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:04.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d93a2b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d8c13bb00 0x7f0d8c0c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:04.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d93a2b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c0c61d0 con 0x7f0d8c13bb00 2023-10-24T23:54:04.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d93a2b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c1ebd00 con 0x7f0d8c0aa330 2023-10-24T23:54:04.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d93a2b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8c0c3270 con 0x7f0d8c0a9450 2023-10-24T23:54:04.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d91fc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d8c13bb00 0x7f0d8c0c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:04.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d91fc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d8c13bb00 0x7f0d8c0c75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52652/0 (socket says 172.21.15.107:52652) 2023-10-24T23:54:04.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d91fc8640 1 -- 172.21.15.107:0/2060780165 learned_addr learned my addr 172.21.15.107:0/2060780165 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:04.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d917c7640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0d8c0a9450 0x7f0d8c0c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:04.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d90fc6640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d8c0aa330 0x7f0d8c1eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d91fc8640 1 -- 172.21.15.107:0/2060780165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0d8c0a9450 msgr2=0x7f0d8c0c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d91fc8640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0d8c0a9450 0x7f0d8c0c2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d91fc8640 1 -- 172.21.15.107:0/2060780165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d8c0aa330 msgr2=0x7f0d8c1eb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d91fc8640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d8c0aa330 0x7f0d8c1eb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d91fc8640 1 -- 172.21.15.107:0/2060780165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c0094d0 con 0x7f0d8c13bb00 2023-10-24T23:54:04.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d91fc8640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d8c13bb00 0x7f0d8c0c75e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0d84007e40 tx=0x7f0d8400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:04.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d81ffb640 1 -- 172.21.15.107:0/2060780165 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d84014620 con 0x7f0d8c13bb00 2023-10-24T23:54:04.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d81ffb640 1 -- 172.21.15.107:0/2060780165 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d840147a0 con 0x7f0d8c13bb00 2023-10-24T23:54:04.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2060780165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8c0c3550 con 0x7f0d8c13bb00 2023-10-24T23:54:04.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d81ffb640 1 -- 172.21.15.107:0/2060780165 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d84014b50 con 0x7f0d8c13bb00 2023-10-24T23:54:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2060780165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8c0c3830 con 0x7f0d8c13bb00 2023-10-24T23:54:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.249+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2060780165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8c0c6380 con 0x7f0d8c13bb00 2023-10-24T23:54:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.253+0000 7f0d81ffb640 1 -- 172.21.15.107:0/2060780165 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0d84014cb0 con 0x7f0d8c13bb00 2023-10-24T23:54:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.253+0000 7f0d81ffb640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0d70065e90 0x7f0d70068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:04.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.253+0000 7f0d81ffb640 1 -- 172.21.15.107:0/2060780165 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f0d84084300 con 0x7f0d8c13bb00 2023-10-24T23:54:04.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.253+0000 7f0d917c7640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0d70065e90 0x7f0d70068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:04.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.253+0000 7f0d917c7640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0d70065e90 0x7f0d70068350 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0d88007fd0 tx=0x7f0d88009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:04.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.257+0000 7f0d81ffb640 1 -- 172.21.15.107:0/2060780165 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0d84051700 con 0x7f0d8c13bb00 2023-10-24T23:54:04.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.457+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2060780165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d8c0746e0 con 0x7f0d8c13bb00 2023-10-24T23:54:04.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d81ffb640 1 -- 172.21.15.107:0/2060780165 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f0d84054d20 con 0x7f0d8c13bb00 2023-10-24T23:54:04.462 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:54:04.463 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:54:04.463 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:54:04.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2060780165 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0d70065e90 msgr2=0x7f0d70068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:04.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0d70065e90 0x7f0d70068350 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0d88007fd0 tx=0x7f0d88009040 comp rx=0 tx=0).stop 2023-10-24T23:54:04.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2060780165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d8c13bb00 msgr2=0x7f0d8c0c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:04.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d8c13bb00 0x7f0d8c0c75e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0d84007e40 tx=0x7f0d8400e4b0 comp rx=0 tx=0).stop 2023-10-24T23:54:04.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2060780165 shutdown_connections 2023-10-24T23:54:04.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d8c13bb00 0x7f0d8c0c75e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:04.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0d8c0aa330 0x7f0d8c1eb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:04.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0d70065e90 0x7f0d70068350 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:04.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 --2- 172.21.15.107:0/2060780165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0d8c0a9450 0x7f0d8c0c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:04.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2060780165 >> 172.21.15.107:0/2060780165 conn(0x7f0d8c070f60 msgr2=0x7f0d8c1505f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:04.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2060780165 shutdown_connections 2023-10-24T23:54:04.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:04.461+0000 7f0d93a2b640 1 -- 172.21.15.107:0/2060780165 wait complete. 2023-10-24T23:54:04.481 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:54:06.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:54:06.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.608+0000 7f39825ea640 1 Processor -- start 2023-10-24T23:54:06.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.608+0000 7f39825ea640 1 -- start start 2023-10-24T23:54:06.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.608+0000 7f39825ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f397c0adb80 0x7f397c158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:06.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.608+0000 7f39825ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f397c06e800 0x7f397c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:06.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.608+0000 7f39825ea640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f397c152420 con 0x7f397c152040 2023-10-24T23:54:06.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.608+0000 7f39825ea640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f397c1556e0 con 0x7f397c0adb80 2023-10-24T23:54:06.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.608+0000 7f39825ea640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397c154d40 con 0x7f397c06e800 2023-10-24T23:54:06.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f397c152040 0x7f397c071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54026/0 (socket says 172.21.15.107:54026) 2023-10-24T23:54:06.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397bfff640 1 -- 172.21.15.107:0/3440602458 learned_addr learned my addr 172.21.15.107:0/3440602458 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:54:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f3973fff640 1 --2- 172.21.15.107:0/3440602458 >> v2:172.21.15.110:3300/0 conn(0x7f397c0adb80 0x7f397c158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f3980b87640 1 --2- 172.21.15.107:0/3440602458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f397c06e800 0x7f397c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279256518 0 0) 0x7f397c152420 con 0x7f397c152040 2023-10-24T23:54:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3968003530 con 0x7f397c152040 2023-10-24T23:54:06.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2354109287 0 0) 0x7f3968003530 con 0x7f397c152040 2023-10-24T23:54:06.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f397c06e800 msgr2=0x7f397c06ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:06.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 --2- 172.21.15.107:0/3440602458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f397c06e800 0x7f397c06ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:06.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 >> v2:172.21.15.110:3300/0 conn(0x7f397c0adb80 msgr2=0x7f397c158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:06.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 --2- 172.21.15.107:0/3440602458 >> v2:172.21.15.110:3300/0 conn(0x7f397c0adb80 0x7f397c158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:06.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c155020 con 0x7f397c152040 2023-10-24T23:54:06.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f396c001680 con 0x7f397c152040 2023-10-24T23:54:06.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f3973fff640 1 --2- 172.21.15.107:0/3440602458 >> v2:172.21.15.110:3300/0 conn(0x7f397c0adb80 0x7f397c158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:54:06.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 >> v1:172.21.15.107:6789/0 conn(0x7f397c152040 legacy=0x7f397c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:06.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 --2- 172.21.15.107:0/3440602458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3968003980 0x7f3968003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 --2- 172.21.15.107:0/3440602458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39680047d0 0x7f3968004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 --2- 172.21.15.107:0/3440602458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f397c06e800 0x7f3968005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c1556e0 con 0x7f3968003980 2023-10-24T23:54:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c154d40 con 0x7f39680047d0 2023-10-24T23:54:06.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396800b9c0 con 0x7f397c06e800 2023-10-24T23:54:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397bfff640 1 --2- 172.21.15.107:0/3440602458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3968003980 0x7f3968003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f3980b87640 1 --2- 172.21.15.107:0/3440602458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f397c06e800 0x7f3968005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397bfff640 1 -- 172.21.15.107:0/3440602458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f397c06e800 msgr2=0x7f3968005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397bfff640 1 --2- 172.21.15.107:0/3440602458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f397c06e800 0x7f3968005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:06.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397bfff640 1 -- 172.21.15.107:0/3440602458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39680047d0 msgr2=0x7f3968004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:06.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397bfff640 1 --2- 172.21.15.107:0/3440602458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39680047d0 0x7f3968004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:06.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397bfff640 1 -- 172.21.15.107:0/3440602458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3968003530 con 0x7f3968003980 2023-10-24T23:54:06.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397bfff640 1 --2- 172.21.15.107:0/3440602458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3968003980 0x7f3968003da0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f396c00c210 tx=0x7f396c01cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ed6812bee4b52 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:06.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c0058e0 con 0x7f3968003980 2023-10-24T23:54:06.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396c005a40 con 0x7f3968003980 2023-10-24T23:54:06.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.612+0000 7f397b7fe640 1 -- 172.21.15.107:0/3440602458 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396c005dd0 con 0x7f3968003980 2023-10-24T23:54:06.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 -- 172.21.15.107:0/3440602458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3968003980 msgr2=0x7f3968003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:06.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 --2- 172.21.15.107:0/3440602458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3968003980 0x7f3968003da0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f396c00c210 tx=0x7f396c01cc00 comp rx=0 tx=0).stop 2023-10-24T23:54:06.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f3980b87640 1 -- 172.21.15.107:0/3440602458 reap_dead start 2023-10-24T23:54:06.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 -- 172.21.15.107:0/3440602458 shutdown_connections 2023-10-24T23:54:06.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 -- 172.21.15.107:0/3440602458 >> 172.21.15.107:0/3440602458 conn(0x7f397c070f60 msgr2=0x7f397c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:06.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 -- 172.21.15.107:0/3440602458 shutdown_connections 2023-10-24T23:54:06.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 -- 172.21.15.107:0/3440602458 wait complete. 2023-10-24T23:54:06.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 Processor -- start 2023-10-24T23:54:06.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 -- start start 2023-10-24T23:54:06.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3968003980 0x7f397c0c2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:06.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f397c159260 0x7f397c14a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:06.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397c0c3650 0x7f397c14ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:06.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c15a250 con 0x7f397c0c3650 2023-10-24T23:54:06.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c152860 con 0x7f397c159260 2023-10-24T23:54:06.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f397c14b190 con 0x7f3968003980 2023-10-24T23:54:06.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f3980b87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397c0c3650 0x7f397c14ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:06.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f3980b87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397c0c3650 0x7f397c14ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52666/0 (socket says 172.21.15.107:52666) 2023-10-24T23:54:06.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f3980b87640 1 -- 172.21.15.107:0/1144342552 learned_addr learned my addr 172.21.15.107:0/1144342552 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:06.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f3973fff640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f397c159260 0x7f397c14a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f397bfff640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3968003980 0x7f397c0c2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f3980b87640 1 -- 172.21.15.107:0/1144342552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3968003980 msgr2=0x7f397c0c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f3980b87640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3968003980 0x7f397c0c2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f3980b87640 1 -- 172.21.15.107:0/1144342552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f397c159260 msgr2=0x7f397c14a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f3980b87640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f397c159260 0x7f397c14a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f3980b87640 1 -- 172.21.15.107:0/1144342552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f397c15aa90 con 0x7f397c0c3650 2023-10-24T23:54:06.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f3980b87640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397c0c3650 0x7f397c14ac50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f397c070f30 tx=0x7f397400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:06.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39797fa640 1 -- 172.21.15.107:0/1144342552 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974017400 con 0x7f397c0c3650 2023-10-24T23:54:06.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39797fa640 1 -- 172.21.15.107:0/1144342552 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3974011040 con 0x7f397c0c3650 2023-10-24T23:54:06.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 -- 172.21.15.107:0/1144342552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c156040 con 0x7f397c0c3650 2023-10-24T23:54:06.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39797fa640 1 -- 172.21.15.107:0/1144342552 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39740177c0 con 0x7f397c0c3650 2023-10-24T23:54:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.616+0000 7f39825ea640 1 -- 172.21.15.107:0/1144342552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c154390 con 0x7f397c0c3650 2023-10-24T23:54:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.620+0000 7f39825ea640 1 -- 172.21.15.107:0/1144342552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c152a10 con 0x7f397c0c3650 2023-10-24T23:54:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.620+0000 7f39797fa640 1 -- 172.21.15.107:0/1144342552 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3974021440 con 0x7f397c0c3650 2023-10-24T23:54:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.620+0000 7f39797fa640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f395c065e60 0x7f395c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:06.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.620+0000 7f39797fa640 1 -- 172.21.15.107:0/1144342552 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f3974083dd0 con 0x7f397c0c3650 2023-10-24T23:54:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.620+0000 7f397bfff640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f395c065e60 0x7f395c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.620+0000 7f397bfff640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f395c065e60 0x7f395c068320 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f396c0aa000 tx=0x7f396c001a10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:06.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.624+0000 7f39797fa640 1 -- 172.21.15.107:0/1144342552 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f397c152a10 con 0x7f397c0c3650 2023-10-24T23:54:06.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.836+0000 7f39825ea640 1 -- 172.21.15.107:0/1144342552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f397c152a10 con 0x7f397c0c3650 2023-10-24T23:54:06.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.836+0000 7f39797fa640 1 -- 172.21.15.107:0/1144342552 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f397c152a10 con 0x7f397c0c3650 2023-10-24T23:54:06.840 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:54:06.840 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:54:06.840 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:54:06.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 -- 172.21.15.107:0/1144342552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f395c065e60 msgr2=0x7f395c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:06.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f395c065e60 0x7f395c068320 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f396c0aa000 tx=0x7f396c001a10 comp rx=0 tx=0).stop 2023-10-24T23:54:06.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 -- 172.21.15.107:0/1144342552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397c0c3650 msgr2=0x7f397c14ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:06.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397c0c3650 0x7f397c14ac50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f397c070f30 tx=0x7f397400f4a0 comp rx=0 tx=0).stop 2023-10-24T23:54:06.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 -- 172.21.15.107:0/1144342552 shutdown_connections 2023-10-24T23:54:06.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f397c0c3650 0x7f397c14ac50 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:06.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f397c159260 0x7f397c14a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:06.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f395c065e60 0x7f395c068320 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:06.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 --2- 172.21.15.107:0/1144342552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3968003980 0x7f397c0c2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:06.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 -- 172.21.15.107:0/1144342552 >> 172.21.15.107:0/1144342552 conn(0x7f397c070f60 msgr2=0x7f397c0c7570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:06.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 -- 172.21.15.107:0/1144342552 shutdown_connections 2023-10-24T23:54:06.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:06.840+0000 7f39825ea640 1 -- 172.21.15.107:0/1144342552 wait complete. 2023-10-24T23:54:06.859 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:54:08.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:54:09.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.000+0000 7fb4c6809640 1 Processor -- start 2023-10-24T23:54:09.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.000+0000 7fb4c6809640 1 -- start start 2023-10-24T23:54:09.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4c6809640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb4c014c470 0x7fb4c014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:09.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4c6809640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c0155cd0 con 0x7fb4c006f490 2023-10-24T23:54:09.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4c6809640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c00c4380 con 0x7fb4c0152b30 2023-10-24T23:54:09.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4c6809640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c01526e0 con 0x7fb4c014c470 2023-10-24T23:54:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4bffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb4c014c470 0x7fb4c014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4bffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb4c014c470 0x7fb4c014c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52676/0 (socket says 172.21.15.107:52676) 2023-10-24T23:54:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4bffff640 1 -- 172.21.15.107:0/89512041 learned_addr learned my addr 172.21.15.107:0/89512041 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4bffff640 1 -- 172.21.15.107:0/89512041 >> v1:172.21.15.110:6789/0 conn(0x7fb4c0152b30 legacy=0x7fb4c0152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:09.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4bffff640 1 -- 172.21.15.107:0/89512041 >> v1:172.21.15.107:6789/0 conn(0x7fb4c006f490 legacy=0x7fb4c0071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:09.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4bffff640 1 -- 172.21.15.107:0/89512041 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c00c48b0 con 0x7fb4c014c470 2023-10-24T23:54:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4bffff640 1 --2- 172.21.15.107:0/89512041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4c014c470 0x7fb4c014c850 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009870 tx=0x7fb4b001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92baffda1afce511 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4beffd640 1 -- 172.21.15.107:0/89512041 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b00ab600 con 0x7fb4c014c470 2023-10-24T23:54:09.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4beffd640 1 -- 172.21.15.107:0/89512041 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b00a7050 con 0x7fb4c014c470 2023-10-24T23:54:09.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.004+0000 7fb4beffd640 1 -- 172.21.15.107:0/89512041 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b00ab980 con 0x7fb4c014c470 2023-10-24T23:54:09.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 -- 172.21.15.107:0/89512041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4c014c470 msgr2=0x7fb4c014c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:09.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 --2- 172.21.15.107:0/89512041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4c014c470 0x7fb4c014c850 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb4b0009870 tx=0x7fb4b001c3e0 comp rx=0 tx=0).stop 2023-10-24T23:54:09.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 -- 172.21.15.107:0/89512041 shutdown_connections 2023-10-24T23:54:09.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 --2- 172.21.15.107:0/89512041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4c014c470 0x7fb4c014c850 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 -- 172.21.15.107:0/89512041 >> 172.21.15.107:0/89512041 conn(0x7fb4c0070f60 msgr2=0x7fb4c0071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 -- 172.21.15.107:0/89512041 shutdown_connections 2023-10-24T23:54:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 -- 172.21.15.107:0/89512041 wait complete. 2023-10-24T23:54:09.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 Processor -- start 2023-10-24T23:54:09.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 -- start start 2023-10-24T23:54:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4c006f490 0x7fb4c014b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4c014c470 0x7fb4c00ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4c0152b30 0x7fb4c00ac980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c00abfb0 con 0x7fb4c0152b30 2023-10-24T23:54:09.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c00ab710 con 0x7fb4c014c470 2023-10-24T23:54:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c014da40 con 0x7fb4c006f490 2023-10-24T23:54:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c4da6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4c0152b30 0x7fb4c00ac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c4da6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4c0152b30 0x7fb4c00ac980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52686/0 (socket says 172.21.15.107:52686) 2023-10-24T23:54:09.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c4da6640 1 -- 172.21.15.107:0/3773710391 learned_addr learned my addr 172.21.15.107:0/3773710391 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4bffff640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4c006f490 0x7fb4c014b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4bf7fe640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4c014c470 0x7fb4c00ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c4da6640 1 -- 172.21.15.107:0/3773710391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4c006f490 msgr2=0x7fb4c014b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c4da6640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4c006f490 0x7fb4c014b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:09.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c4da6640 1 -- 172.21.15.107:0/3773710391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4c014c470 msgr2=0x7fb4c00ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c4da6640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4c014c470 0x7fb4c00ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c4da6640 1 -- 172.21.15.107:0/3773710391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b00094d0 con 0x7fb4c0152b30 2023-10-24T23:54:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4bf7fe640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4c014c470 0x7fb4c00ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:54:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c4da6640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4c0152b30 0x7fb4c00ac980 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800b970 tx=0x7fb4b800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4bcff9640 1 -- 172.21.15.107:0/3773710391 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b800cb40 con 0x7fb4c0152b30 2023-10-24T23:54:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4bcff9640 1 -- 172.21.15.107:0/3773710391 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b800cca0 con 0x7fb4c0152b30 2023-10-24T23:54:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 -- 172.21.15.107:0/3773710391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c00ab960 con 0x7fb4c0152b30 2023-10-24T23:54:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4bcff9640 1 -- 172.21.15.107:0/3773710391 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8014680 con 0x7fb4c0152b30 2023-10-24T23:54:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.008+0000 7fb4c6809640 1 -- 172.21.15.107:0/3773710391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c014be90 con 0x7fb4c0152b30 2023-10-24T23:54:09.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.012+0000 7fb4c6809640 1 -- 172.21.15.107:0/3773710391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c00ac160 con 0x7fb4c0152b30 2023-10-24T23:54:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.012+0000 7fb4bcff9640 1 -- 172.21.15.107:0/3773710391 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb4b801d3b0 con 0x7fb4c0152b30 2023-10-24T23:54:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.012+0000 7fb4bcff9640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4ac065e90 0x7fb4ac068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.012+0000 7fb4bffff640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4ac065e90 0x7fb4ac068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.012+0000 7fb4bcff9640 1 -- 172.21.15.107:0/3773710391 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7fb4b8083840 con 0x7fb4c0152b30 2023-10-24T23:54:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.012+0000 7fb4bffff640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4ac065e90 0x7fb4ac068350 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00ad000 tx=0x7fb4b00a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.016+0000 7fb4bcff9640 1 -- 172.21.15.107:0/3773710391 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb4c00ac160 con 0x7fb4c0152b30 2023-10-24T23:54:09.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.256+0000 7fb4c6809640 1 -- 172.21.15.107:0/3773710391 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4c00ac160 con 0x7fb4c0152b30 2023-10-24T23:54:09.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.256+0000 7fb4bcff9640 1 -- 172.21.15.107:0/3773710391 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7fb4c00ac160 con 0x7fb4c0152b30 2023-10-24T23:54:09.260 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:54:09.260 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:54:09.261 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:54:09.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 -- 172.21.15.107:0/3773710391 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4ac065e90 msgr2=0x7fb4ac068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:09.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4ac065e90 0x7fb4ac068350 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb4b00ad000 tx=0x7fb4b00a6000 comp rx=0 tx=0).stop 2023-10-24T23:54:09.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 -- 172.21.15.107:0/3773710391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4c0152b30 msgr2=0x7fb4c00ac980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:09.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4c0152b30 0x7fb4c00ac980 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800b970 tx=0x7fb4b800be30 comp rx=0 tx=0).stop 2023-10-24T23:54:09.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 -- 172.21.15.107:0/3773710391 shutdown_connections 2023-10-24T23:54:09.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4c0152b30 0x7fb4c00ac980 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:09.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4c014c470 0x7fb4c00ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:09.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4ac065e90 0x7fb4ac068350 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:09.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 --2- 172.21.15.107:0/3773710391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4c006f490 0x7fb4c014b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:09.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 -- 172.21.15.107:0/3773710391 >> 172.21.15.107:0/3773710391 conn(0x7fb4c0070f60 msgr2=0x7fb4c0154d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:09.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 -- 172.21.15.107:0/3773710391 shutdown_connections 2023-10-24T23:54:09.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:09.260+0000 7fb4c6809640 1 -- 172.21.15.107:0/3773710391 wait complete. 2023-10-24T23:54:09.280 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:54:11.281 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:54:11.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.408+0000 7f4875861640 1 Processor -- start 2023-10-24T23:54:11.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.408+0000 7f4875861640 1 -- start start 2023-10-24T23:54:11.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f4875861640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48700c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:11.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f4875861640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48700c22f0 con 0x7f487014eb70 2023-10-24T23:54:11.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f4875861640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48700c25d0 con 0x7f487006df30 2023-10-24T23:54:11.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f4875861640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4870071700 con 0x7f4870070b00 2023-10-24T23:54:11.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486effd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f487014eb70 0x7f48701514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54046/0 (socket says 172.21.15.107:54046) 2023-10-24T23:54:11.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486effd640 1 -- 172.21.15.107:0/2044322656 learned_addr learned my addr 172.21.15.107:0/2044322656 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:54:11.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486e7fc640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48700c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:11.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199914954 0 0) 0x7f48700c22f0 con 0x7f487014eb70 2023-10-24T23:54:11.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4858003530 con 0x7f487014eb70 2023-10-24T23:54:11.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2717035596 0 0) 0x7f4858003530 con 0x7f487014eb70 2023-10-24T23:54:11.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 msgr2=0x7f48700c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:11.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48700c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:11.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 >> v1:172.21.15.110:6789/0 conn(0x7f487006df30 legacy=0x7f487015be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:11.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700c2be0 con 0x7f487014eb70 2023-10-24T23:54:11.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f4860001680 con 0x7f487014eb70 2023-10-24T23:54:11.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 >> v1:172.21.15.107:6789/0 conn(0x7f487014eb70 legacy=0x7f48701514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:11.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48580039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:11.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4858003fc0 0x7f48580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:11.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4858004d10 0x7f4858005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:11.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485800bb20 con 0x7f4858004d10 2023-10-24T23:54:11.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485800bc60 con 0x7f4858003fc0 2023-10-24T23:54:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485800e030 con 0x7f4870070b00 2023-10-24T23:54:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486f7fe640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4858004d10 0x7f4858005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486effd640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48580039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:11.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486e7fc640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4858003fc0 0x7f48580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486f7fe640 1 -- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 msgr2=0x7f48580039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486f7fe640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48580039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486f7fe640 1 -- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4858003fc0 msgr2=0x7f48580047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:11.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486f7fe640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4858003fc0 0x7f48580047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486f7fe640 1 -- 172.21.15.107:0/2044322656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4858003530 con 0x7f4858004d10 2023-10-24T23:54:11.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.412+0000 7f486e7fc640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4858003fc0 0x7f48580047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:54:11.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.416+0000 7f486f7fe640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4858004d10 0x7f4858005190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4868007fd0 tx=0x7f486801c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0d365e2aa523ba server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:11.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.416+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48680a9660 con 0x7f4858004d10 2023-10-24T23:54:11.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.416+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48680a97e0 con 0x7f4858004d10 2023-10-24T23:54:11.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.416+0000 7f486dffb640 1 -- 172.21.15.107:0/2044322656 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48680a9b90 con 0x7f4858004d10 2023-10-24T23:54:11.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.416+0000 7f4875861640 1 -- 172.21.15.107:0/2044322656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4858004d10 msgr2=0x7f4858005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:11.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.416+0000 7f4875861640 1 --2- 172.21.15.107:0/2044322656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4858004d10 0x7f4858005190 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4868007fd0 tx=0x7f486801c660 comp rx=0 tx=0).stop 2023-10-24T23:54:11.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.416+0000 7f486f7fe640 1 -- 172.21.15.107:0/2044322656 reap_dead start 2023-10-24T23:54:11.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.416+0000 7f4875861640 1 -- 172.21.15.107:0/2044322656 shutdown_connections 2023-10-24T23:54:11.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.416+0000 7f4875861640 1 -- 172.21.15.107:0/2044322656 >> 172.21.15.107:0/2044322656 conn(0x7f48700c39a0 msgr2=0x7f48700c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:11.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.416+0000 7f4875861640 1 -- 172.21.15.107:0/2044322656 shutdown_connections 2023-10-24T23:54:11.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.416+0000 7f4875861640 1 -- 172.21.15.107:0/2044322656 wait complete. 2023-10-24T23:54:11.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4875861640 1 Processor -- start 2023-10-24T23:54:11.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4875861640 1 -- start start 2023-10-24T23:54:11.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4875861640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487013ede0 0x7f48701e95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:11.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4875861640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f487006c630 0x7f48701e6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:11.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4875861640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870069d40 0x7f4870141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:11.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487013ede0 0x7f48701e95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:11.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487013ede0 0x7f48701e95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52702/0 (socket says 172.21.15.107:52702) 2023-10-24T23:54:11.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486effd640 1 -- 172.21.15.107:0/3915735107 learned_addr learned my addr 172.21.15.107:0/3915735107 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:11.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487006aab0 con 0x7f487013ede0 2023-10-24T23:54:11.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48701e9140 con 0x7f487006c630 2023-10-24T23:54:11.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487006a2b0 con 0x7f4870069d40 2023-10-24T23:54:11.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486e7fc640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f487006c630 0x7f48701e6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:11.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486f7fe640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870069d40 0x7f4870141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:11.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486effd640 1 -- 172.21.15.107:0/3915735107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870069d40 msgr2=0x7f4870141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:11.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486effd640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870069d40 0x7f4870141ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:11.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486effd640 1 -- 172.21.15.107:0/3915735107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f487006c630 msgr2=0x7f48701e6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:11.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486effd640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f487006c630 0x7f48701e6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:11.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486effd640 1 -- 172.21.15.107:0/3915735107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48701e8af0 con 0x7f487013ede0 2023-10-24T23:54:11.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486f7fe640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870069d40 0x7f4870141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:54:11.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f486effd640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487013ede0 0x7f48701e95d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f486000ead0 tx=0x7f486000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:11.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4857fff640 1 -- 172.21.15.107:0/3915735107 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486000c8d0 con 0x7f487013ede0 2023-10-24T23:54:11.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4857fff640 1 -- 172.21.15.107:0/3915735107 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486000ca30 con 0x7f487013ede0 2023-10-24T23:54:11.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4857fff640 1 -- 172.21.15.107:0/3915735107 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486000cde0 con 0x7f487013ede0 2023-10-24T23:54:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48701e8dd0 con 0x7f487013ede0 2023-10-24T23:54:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.420+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487014d630 con 0x7f487013ede0 2023-10-24T23:54:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.424+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48700746e0 con 0x7f487013ede0 2023-10-24T23:54:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.424+0000 7f4857fff640 1 -- 172.21.15.107:0/3915735107 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f486001d460 con 0x7f487013ede0 2023-10-24T23:54:11.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.424+0000 7f4857fff640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f485c065e60 0x7f485c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.424+0000 7f486e7fc640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f485c065e60 0x7f485c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.424+0000 7f4857fff640 1 -- 172.21.15.107:0/3915735107 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6961+0+0 (secure 0 0 0) 0x7f4860083b00 con 0x7f487013ede0 2023-10-24T23:54:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.424+0000 7f486e7fc640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f485c065e60 0x7f485c068320 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4870142c50 tx=0x7f4864008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.432+0000 7f4857fff640 1 -- 172.21.15.107:0/3915735107 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4860050e50 con 0x7f487013ede0 2023-10-24T23:54:11.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48700c1a50 con 0x7f487013ede0 2023-10-24T23:54:11.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4857fff640 1 -- 172.21.15.107:0/3915735107 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+5765 (secure 0 0 0) 0x7f4860054470 con 0x7f487013ede0 2023-10-24T23:54:11.636 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:54:11.636 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4290,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:53:14.800231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"a","rank":0,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6837/727042497","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":727042497},{"type":"v1","addr":"172.21.15.107:6837","nonce":727042497}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:54:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 12 2023-10-24T23:54:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f485c065e60 msgr2=0x7f485c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f485c065e60 0x7f485c068320 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4870142c50 tx=0x7f4864008040 comp rx=0 tx=0).stop 2023-10-24T23:54:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487013ede0 msgr2=0x7f48701e95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487013ede0 0x7f48701e95d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f486000ead0 tx=0x7f486000ef90 comp rx=0 tx=0).stop 2023-10-24T23:54:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 shutdown_connections 2023-10-24T23:54:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f485c065e60 0x7f485c068320 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:11.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870069d40 0x7f4870141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:11.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f487006c630 0x7f48701e6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:11.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 --2- 172.21.15.107:0/3915735107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487013ede0 0x7f48701e95d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:11.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 >> 172.21.15.107:0/3915735107 conn(0x7f48700c39a0 msgr2=0x7f48700c3230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 shutdown_connections 2023-10-24T23:54:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:11.632+0000 7f4875861640 1 -- 172.21.15.107:0/3915735107 wait complete. 2023-10-24T23:54:11.653 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:54:13.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:54:13.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38adc04640 1 Processor -- start 2023-10-24T23:54:13.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38adc04640 1 -- start start 2023-10-24T23:54:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38adc04640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f38a80c33e0 0x7f38a815ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38adc04640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a80c22f0 con 0x7f38a814eb70 2023-10-24T23:54:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38adc04640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a80c25d0 con 0x7f38a8070b00 2023-10-24T23:54:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38adc04640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a8071700 con 0x7f38a80c33e0 2023-10-24T23:54:13.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f38a814eb70 0x7f38a81514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38866/0 (socket says 172.21.15.107:38866) 2023-10-24T23:54:13.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a77fe640 1 -- 172.21.15.107:0/2412558860 learned_addr learned my addr 172.21.15.107:0/2412558860 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:54:13.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a7fff640 1 --2- 172.21.15.107:0/2412558860 >> v2:172.21.15.110:3300/0 conn(0x7f38a80c33e0 0x7f38a815ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:13.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557963329 0 0) 0x7f38a80c22f0 con 0x7f38a814eb70 2023-10-24T23:54:13.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3894003530 con 0x7f38a814eb70 2023-10-24T23:54:13.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 132434204 0 0) 0x7f3894003530 con 0x7f38a814eb70 2023-10-24T23:54:13.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 >> v2:172.21.15.110:3300/0 conn(0x7f38a80c33e0 msgr2=0x7f38a815ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:13.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a6ffd640 1 --2- 172.21.15.107:0/2412558860 >> v2:172.21.15.110:3300/0 conn(0x7f38a80c33e0 0x7f38a815ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:13.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 >> v1:172.21.15.110:6789/0 conn(0x7f38a8070b00 legacy=0x7f38a806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:13.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a80c2be0 con 0x7f38a814eb70 2023-10-24T23:54:13.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f3898001680 con 0x7f38a814eb70 2023-10-24T23:54:13.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 >> v1:172.21.15.107:6789/0 conn(0x7f38a814eb70 legacy=0x7f38a81514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:13.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a6ffd640 1 --2- 172.21.15.107:0/2412558860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3894003980 0x7f3894003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:13.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.808+0000 7f38a6ffd640 1 --2- 172.21.15.107:0/2412558860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38940047d0 0x7f3894004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:13.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a6ffd640 1 --2- 172.21.15.107:0/2412558860 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3894005190 0x7f389400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:13.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389400c000 con 0x7f3894003980 2023-10-24T23:54:13.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389400c170 con 0x7f38940047d0 2023-10-24T23:54:13.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389400c330 con 0x7f3894005190 2023-10-24T23:54:13.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a77fe640 1 --2- 172.21.15.107:0/2412558860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3894003980 0x7f3894003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:13.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f389ffff640 1 --2- 172.21.15.107:0/2412558860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38940047d0 0x7f3894004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:13.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a77fe640 1 -- 172.21.15.107:0/2412558860 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3894005190 msgr2=0x7f389400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:13.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a77fe640 1 --2- 172.21.15.107:0/2412558860 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3894005190 0x7f389400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:13.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a77fe640 1 -- 172.21.15.107:0/2412558860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38940047d0 msgr2=0x7f3894004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:13.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a77fe640 1 --2- 172.21.15.107:0/2412558860 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38940047d0 0x7f3894004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:13.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a77fe640 1 -- 172.21.15.107:0/2412558860 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3894003530 con 0x7f3894003980 2023-10-24T23:54:13.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a7fff640 1 -- 172.21.15.107:0/2412558860 reap_dead start 2023-10-24T23:54:13.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a77fe640 1 --2- 172.21.15.107:0/2412558860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3894003980 0x7f3894003da0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f389800c240 tx=0x7f389801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb86edc640c45eb server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:13.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a8151cf0 con 0x7f3894003980 2023-10-24T23:54:13.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3898005ae0 con 0x7f3894003980 2023-10-24T23:54:13.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.812+0000 7f38a6ffd640 1 -- 172.21.15.107:0/2412558860 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38980a57f0 con 0x7f3894003980 2023-10-24T23:54:13.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 -- 172.21.15.107:0/2412558860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3894003980 msgr2=0x7f3894003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:13.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 --2- 172.21.15.107:0/2412558860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3894003980 0x7f3894003da0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f389800c240 tx=0x7f389801cc00 comp rx=0 tx=0).stop 2023-10-24T23:54:13.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 -- 172.21.15.107:0/2412558860 shutdown_connections 2023-10-24T23:54:13.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 --2- 172.21.15.107:0/2412558860 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3894003980 0x7f3894003da0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:13.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 -- 172.21.15.107:0/2412558860 >> 172.21.15.107:0/2412558860 conn(0x7f38a80c39a0 msgr2=0x7f38a80c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:13.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 -- 172.21.15.107:0/2412558860 shutdown_connections 2023-10-24T23:54:13.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 -- 172.21.15.107:0/2412558860 wait complete. 2023-10-24T23:54:13.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 Processor -- start 2023-10-24T23:54:13.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 -- start start 2023-10-24T23:54:13.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3894003980 0x7f38a814f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:13.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38a8148260 0x7f38a813d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:13.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.816+0000 7f38adc04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38a8069d40 0x7f38a806a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:13.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38adc04640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a81506e0 con 0x7f38a8069d40 2023-10-24T23:54:13.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38adc04640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a814fc00 con 0x7f3894003980 2023-10-24T23:54:13.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38adc04640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80c0650 con 0x7f38a8148260 2023-10-24T23:54:13.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38a8069d40 0x7f38a806a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:13.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38a8069d40 0x7f38a806a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40084/0 (socket says 172.21.15.107:40084) 2023-10-24T23:54:13.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a7fff640 1 -- 172.21.15.107:0/2557975426 learned_addr learned my addr 172.21.15.107:0/2557975426 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:13.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a77fe640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3894003980 0x7f38a814f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:13.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f389ffff640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38a8148260 0x7f38a813d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:13.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a7fff640 1 -- 172.21.15.107:0/2557975426 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38a8148260 msgr2=0x7f38a813d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:13.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a7fff640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38a8148260 0x7f38a813d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:13.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a7fff640 1 -- 172.21.15.107:0/2557975426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3894003980 msgr2=0x7f38a814f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:13.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a7fff640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3894003980 0x7f38a814f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:13.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a7fff640 1 -- 172.21.15.107:0/2557975426 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a8147480 con 0x7f38a8069d40 2023-10-24T23:54:13.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a77fe640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3894003980 0x7f38a814f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:54:13.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f389ffff640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38a8148260 0x7f38a813d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:54:13.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a7fff640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38a8069d40 0x7f38a806a1c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f38a80c69c0 tx=0x7f38a000ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:13.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a4ff9640 1 -- 172.21.15.107:0/2557975426 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0010040 con 0x7f38a8069d40 2023-10-24T23:54:13.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38adc04640 1 -- 172.21.15.107:0/2557975426 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a8147720 con 0x7f38a8069d40 2023-10-24T23:54:13.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a4ff9640 1 -- 172.21.15.107:0/2557975426 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a000bb70 con 0x7f38a8069d40 2023-10-24T23:54:13.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38a4ff9640 1 -- 172.21.15.107:0/2557975426 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0014400 con 0x7f38a8069d40 2023-10-24T23:54:13.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38adc04640 1 -- 172.21.15.107:0/2557975426 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a813cd00 con 0x7f38a8069d40 2023-10-24T23:54:13.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.820+0000 7f38adc04640 1 -- 172.21.15.107:0/2557975426 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a80746e0 con 0x7f38a8069d40 2023-10-24T23:54:13.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.824+0000 7f38a4ff9640 1 -- 172.21.15.107:0/2557975426 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f38a0014560 con 0x7f38a8069d40 2023-10-24T23:54:13.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.824+0000 7f38a4ff9640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3888065e90 0x7f3888068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:13.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.824+0000 7f38a4ff9640 1 -- 172.21.15.107:0/2557975426 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7f38a0084970 con 0x7f38a8069d40 2023-10-24T23:54:13.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.824+0000 7f38a77fe640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3888065e90 0x7f3888068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:13.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.824+0000 7f38a77fe640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3888065e90 0x7f3888068350 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f38980ab000 tx=0x7f38980a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:13.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:13.828+0000 7f38a4ff9640 1 -- 172.21.15.107:0/2557975426 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f38a0051bc0 con 0x7f38a8069d40 2023-10-24T23:54:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 -- 172.21.15.107:0/2557975426 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38a80c1a50 con 0x7f38a8069d40 2023-10-24T23:54:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38a4ff9640 1 -- 172.21.15.107:0/2557975426 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5011 (secure 0 0 0) 0x7f38a00551e0 con 0x7f38a8069d40 2023-10-24T23:54:14.195 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:54:14.195 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:13.341435+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:54:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 13 2023-10-24T23:54:14.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 -- 172.21.15.107:0/2557975426 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3888065e90 msgr2=0x7f3888068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:14.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3888065e90 0x7f3888068350 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f38980ab000 tx=0x7f38980a7070 comp rx=0 tx=0).stop 2023-10-24T23:54:14.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 -- 172.21.15.107:0/2557975426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38a8069d40 msgr2=0x7f38a806a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:14.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38a8069d40 0x7f38a806a1c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f38a80c69c0 tx=0x7f38a000ef30 comp rx=0 tx=0).stop 2023-10-24T23:54:14.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 -- 172.21.15.107:0/2557975426 shutdown_connections 2023-10-24T23:54:14.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38a8069d40 0x7f38a806a1c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3888065e90 0x7f3888068350 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38a8148260 0x7f38a813d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 --2- 172.21.15.107:0/2557975426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3894003980 0x7f38a814f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 -- 172.21.15.107:0/2557975426 >> 172.21.15.107:0/2557975426 conn(0x7f38a80c39a0 msgr2=0x7f38a8141620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:14.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 -- 172.21.15.107:0/2557975426 shutdown_connections 2023-10-24T23:54:14.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.192+0000 7f38adc04640 1 -- 172.21.15.107:0/2557975426 wait complete. 2023-10-24T23:54:14.217 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2023-10-24T23:54:14.218 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-24T23:54:14.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:54:14.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.320+0000 7f7b458a2640 1 Processor -- start 2023-10-24T23:54:14.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.320+0000 7f7b458a2640 1 -- start start 2023-10-24T23:54:14.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.320+0000 7f7b458a2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b400720c0 0x7f7b400c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:14.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.320+0000 7f7b458a2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b40154860 con 0x7f7b4013dc20 2023-10-24T23:54:14.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.320+0000 7f7b458a2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b400c1f10 con 0x7f7b40157e50 2023-10-24T23:54:14.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.320+0000 7f7b3e7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b400720c0 0x7f7b400c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:14.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.320+0000 7f7b3e7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b400720c0 0x7f7b400c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40092/0 (socket says 172.21.15.107:40092) 2023-10-24T23:54:14.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.320+0000 7f7b3e7fc640 1 -- 172.21.15.107:0/671602496 learned_addr learned my addr 172.21.15.107:0/671602496 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:14.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.320+0000 7f7b458a2640 1 -- 172.21.15.107:0/671602496 --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b40154b20 con 0x7f7b400720c0 2023-10-24T23:54:14.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.324+0000 7f7b3e7fc640 1 -- 172.21.15.107:0/671602496 >> v1:172.21.15.110:6789/0 conn(0x7f7b40157e50 legacy=0x7f7b4015f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:14.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.324+0000 7f7b3e7fc640 1 -- 172.21.15.107:0/671602496 >> v1:172.21.15.107:6789/0 conn(0x7f7b4013dc20 legacy=0x7f7b4006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:14.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.324+0000 7f7b3e7fc640 1 -- 172.21.15.107:0/671602496 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b40154e00 con 0x7f7b400720c0 2023-10-24T23:54:14.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.324+0000 7f7b3e7fc640 1 --2- 172.21.15.107:0/671602496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400720c0 0x7f7b400c18a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009870 tx=0x7f7b3401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01bbadb98a59b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:14.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.324+0000 7f7b3dffb640 1 -- 172.21.15.107:0/671602496 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b340ab600 con 0x7f7b400720c0 2023-10-24T23:54:14.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.324+0000 7f7b3dffb640 1 -- 172.21.15.107:0/671602496 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b340a7050 con 0x7f7b400720c0 2023-10-24T23:54:14.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.324+0000 7f7b3dffb640 1 -- 172.21.15.107:0/671602496 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b340ab980 con 0x7f7b400720c0 2023-10-24T23:54:14.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.324+0000 7f7b458a2640 1 -- 172.21.15.107:0/671602496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400720c0 msgr2=0x7f7b400c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:14.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.324+0000 7f7b458a2640 1 --2- 172.21.15.107:0/671602496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400720c0 0x7f7b400c18a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009870 tx=0x7f7b3401c3e0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 -- 172.21.15.107:0/671602496 shutdown_connections 2023-10-24T23:54:14.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 --2- 172.21.15.107:0/671602496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400720c0 0x7f7b400c18a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 -- 172.21.15.107:0/671602496 >> 172.21.15.107:0/671602496 conn(0x7f7b400c39a0 msgr2=0x7f7b400c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:14.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 -- 172.21.15.107:0/671602496 shutdown_connections 2023-10-24T23:54:14.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 -- 172.21.15.107:0/671602496 wait complete. 2023-10-24T23:54:14.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 Processor -- start 2023-10-24T23:54:14.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 -- start start 2023-10-24T23:54:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400720c0 0x7f7b40137280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b4013dc20 0x7f7b401ed390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b40157e50 0x7f7b401ed8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400720c0 0x7f7b40137280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4013a9b0 con 0x7f7b400720c0 2023-10-24T23:54:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b401f1310 con 0x7f7b40157e50 2023-10-24T23:54:14.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b401f1840 con 0x7f7b4013dc20 2023-10-24T23:54:14.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400720c0 0x7f7b40137280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40102/0 (socket says 172.21.15.107:40102) 2023-10-24T23:54:14.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b3effd640 1 -- 172.21.15.107:0/1334582055 learned_addr learned my addr 172.21.15.107:0/1334582055 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:14.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b3e7fc640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b4013dc20 0x7f7b401ed390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:14.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b3f7fe640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b40157e50 0x7f7b401ed8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:14.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b3effd640 1 -- 172.21.15.107:0/1334582055 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b4013dc20 msgr2=0x7f7b401ed390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:14.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b3effd640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b4013dc20 0x7f7b401ed390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b3effd640 1 -- 172.21.15.107:0/1334582055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b40157e50 msgr2=0x7f7b401ed8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:14.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b3effd640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b40157e50 0x7f7b401ed8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b3effd640 1 -- 172.21.15.107:0/1334582055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b340094d0 con 0x7f7b400720c0 2023-10-24T23:54:14.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b3effd640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400720c0 0x7f7b40137280 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000c530 tx=0x7f7b3000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:14.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b27fff640 1 -- 172.21.15.107:0/1334582055 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30005220 con 0x7f7b400720c0 2023-10-24T23:54:14.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b27fff640 1 -- 172.21.15.107:0/1334582055 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b30005380 con 0x7f7b400720c0 2023-10-24T23:54:14.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b27fff640 1 -- 172.21.15.107:0/1334582055 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30007270 con 0x7f7b400720c0 2023-10-24T23:54:14.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 -- 172.21.15.107:0/1334582055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b401f1560 con 0x7f7b400720c0 2023-10-24T23:54:14.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.328+0000 7f7b458a2640 1 -- 172.21.15.107:0/1334582055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4013b260 con 0x7f7b400720c0 2023-10-24T23:54:14.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.332+0000 7f7b458a2640 1 -- 172.21.15.107:0/1334582055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b400746e0 con 0x7f7b400720c0 2023-10-24T23:54:14.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.332+0000 7f7b27fff640 1 -- 172.21.15.107:0/1334582055 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7b300073d0 con 0x7f7b400720c0 2023-10-24T23:54:14.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.332+0000 7f7b27fff640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b2c065e60 0x7f7b2c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:14.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.332+0000 7f7b3e7fc640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b2c065e60 0x7f7b2c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:14.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.332+0000 7f7b3e7fc640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b2c065e60 0x7f7b2c068320 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7b340ad000 tx=0x7f7b34008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:14.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.340+0000 7f7b27fff640 1 -- 172.21.15.107:0/1334582055 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7f7b30007680 con 0x7f7b400720c0 2023-10-24T23:54:14.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.340+0000 7f7b27fff640 1 -- 172.21.15.107:0/1334582055 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7b300514f0 con 0x7f7b400720c0 2023-10-24T23:54:14.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.524+0000 7f7b458a2640 1 -- 172.21.15.107:0/1334582055 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b400c1190 con 0x7f7b400720c0 2023-10-24T23:54:14.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.524+0000 7f7b27fff640 1 -- 172.21.15.107:0/1334582055 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f7b30054b10 con 0x7f7b400720c0 2023-10-24T23:54:14.529 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:54:14.529 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:14.354837+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:54:14.529 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 14 2023-10-24T23:54:14.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 -- 172.21.15.107:0/1334582055 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b2c065e60 msgr2=0x7f7b2c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:14.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b2c065e60 0x7f7b2c068320 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7b340ad000 tx=0x7f7b34008040 comp rx=0 tx=0).stop 2023-10-24T23:54:14.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 -- 172.21.15.107:0/1334582055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400720c0 msgr2=0x7f7b40137280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:14.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400720c0 0x7f7b40137280 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000c530 tx=0x7f7b3000c9f0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 -- 172.21.15.107:0/1334582055 shutdown_connections 2023-10-24T23:54:14.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b40157e50 0x7f7b401ed8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b2c065e60 0x7f7b2c068320 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b4013dc20 0x7f7b401ed390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 --2- 172.21.15.107:0/1334582055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b400720c0 0x7f7b40137280 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:14.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 -- 172.21.15.107:0/1334582055 >> 172.21.15.107:0/1334582055 conn(0x7f7b400c39a0 msgr2=0x7f7b400c05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:14.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 -- 172.21.15.107:0/1334582055 shutdown_connections 2023-10-24T23:54:14.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:14.528+0000 7f7b458a2640 1 -- 172.21.15.107:0/1334582055 wait complete. 2023-10-24T23:54:14.546 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 14, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 8, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4299": { "addr": "172.21.15.110:6835/2480841919", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2480841919, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2480841919, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4299, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:53:06.342509+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4299 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 14, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4290": { "addr": "172.21.15.107:6835/1220982369", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 1220982369, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 1220982369, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4290, "incarnation": 13, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-24T23:54:14.354837+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4290 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/3721101226", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 3721101226, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 3721101226, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4232, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-24T23:54:16.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:54:16.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.700+0000 7f7948b00640 1 Processor -- start 2023-10-24T23:54:16.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.700+0000 7f7948b00640 1 -- start start 2023-10-24T23:54:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7948b00640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7944155c20 0x7f7944156000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7948b00640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f79440720f0 0x7f79440c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7948b00640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794414cd80 con 0x7f79440a9450 2023-10-24T23:54:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7948b00640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f794406f080 con 0x7f79440720f0 2023-10-24T23:54:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7948b00640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79440aa740 con 0x7f7944155c20 2023-10-24T23:54:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7941d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f79440720f0 0x7f79440c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7941d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f79440720f0 0x7f79440c29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40108/0 (socket says 172.21.15.107:40108) 2023-10-24T23:54:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7941d9b640 1 -- 172.21.15.107:0/3385524260 learned_addr learned my addr 172.21.15.107:0/3385524260 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f794259c640 1 --2- 172.21.15.107:0/3385524260 >> v2:172.21.15.110:3300/0 conn(0x7f7944155c20 0x7f7944156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7941d9b640 1 -- 172.21.15.107:0/3385524260 >> v2:172.21.15.110:3300/0 conn(0x7f7944155c20 msgr2=0x7f7944156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:54:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7941d9b640 1 --2- 172.21.15.107:0/3385524260 >> v2:172.21.15.110:3300/0 conn(0x7f7944155c20 0x7f7944156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:16.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7941d9b640 1 -- 172.21.15.107:0/3385524260 >> v1:172.21.15.110:6789/0 conn(0x7f79440a9450 legacy=0x7f79440c2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:16.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7941d9b640 1 -- 172.21.15.107:0/3385524260 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794406e350 con 0x7f79440720f0 2023-10-24T23:54:16.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.704+0000 7f7941d9b640 1 --2- 172.21.15.107:0/3385524260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79440720f0 0x7f79440c29c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f79440668c0 tx=0x7f793801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a660bfe19a478dd server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:16.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f794159a640 1 -- 172.21.15.107:0/3385524260 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79380ab640 con 0x7f79440720f0 2023-10-24T23:54:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f794159a640 1 -- 172.21.15.107:0/3385524260 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79380a7070 con 0x7f79440720f0 2023-10-24T23:54:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f794159a640 1 -- 172.21.15.107:0/3385524260 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79380ab9c0 con 0x7f79440720f0 2023-10-24T23:54:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f7948b00640 1 -- 172.21.15.107:0/3385524260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79440720f0 msgr2=0x7f79440c29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f7948b00640 1 --2- 172.21.15.107:0/3385524260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79440720f0 0x7f79440c29c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f79440668c0 tx=0x7f793801c440 comp rx=0 tx=0).stop 2023-10-24T23:54:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f7948b00640 1 -- 172.21.15.107:0/3385524260 shutdown_connections 2023-10-24T23:54:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f7948b00640 1 --2- 172.21.15.107:0/3385524260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79440720f0 0x7f79440c29c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f7948b00640 1 --2- 172.21.15.107:0/3385524260 >> v2:172.21.15.110:3300/0 conn(0x7f7944155c20 0x7f7944156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f7948b00640 1 -- 172.21.15.107:0/3385524260 >> 172.21.15.107:0/3385524260 conn(0x7f7944065930 msgr2=0x7f7944065d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f7948b00640 1 -- 172.21.15.107:0/3385524260 shutdown_connections 2023-10-24T23:54:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f7948b00640 1 -- 172.21.15.107:0/3385524260 wait complete. 2023-10-24T23:54:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f7948b00640 1 Processor -- start 2023-10-24T23:54:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.708+0000 7f7948b00640 1 -- start start 2023-10-24T23:54:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7948b00640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79440a9450 0x7f794414b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7948b00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7944155c20 0x7f79440c7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7948b00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79440c76c0 0x7f794413c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f794259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79440a9450 0x7f794414b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7942d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79440c76c0 0x7f794413c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:16.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f794259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79440a9450 0x7f794414b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:40470/0 (socket says 172.21.15.107:40470) 2023-10-24T23:54:16.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f794259c640 1 -- 172.21.15.107:0/545418231 learned_addr learned my addr 172.21.15.107:0/545418231 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:16.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7948b00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79440c7b50 con 0x7f79440c76c0 2023-10-24T23:54:16.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79441eee60 con 0x7f7944155c20 2023-10-24T23:54:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794415cc20 con 0x7f79440a9450 2023-10-24T23:54:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7941d9b640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7944155c20 0x7f79440c7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f794259c640 1 -- 172.21.15.107:0/545418231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7944155c20 msgr2=0x7f79440c7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f794259c640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7944155c20 0x7f79440c7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f794259c640 1 -- 172.21.15.107:0/545418231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79440c76c0 msgr2=0x7f794413c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f794259c640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79440c76c0 0x7f794413c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:16.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f794259c640 1 -- 172.21.15.107:0/545418231 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79380094d0 con 0x7f79440a9450 2023-10-24T23:54:16.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f794259c640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79440a9450 0x7f794414b160 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f793400e930 tx=0x7f793400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:16.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7932ffd640 1 -- 172.21.15.107:0/545418231 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934009690 con 0x7f79440a9450 2023-10-24T23:54:16.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7932ffd640 1 -- 172.21.15.107:0/545418231 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793400ca00 con 0x7f79440a9450 2023-10-24T23:54:16.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79441ee290 con 0x7f79440a9450 2023-10-24T23:54:16.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7932ffd640 1 -- 172.21.15.107:0/545418231 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793400cd70 con 0x7f79440a9450 2023-10-24T23:54:16.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79441ee810 con 0x7f79440a9450 2023-10-24T23:54:16.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.712+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79440746e0 con 0x7f79440a9450 2023-10-24T23:54:16.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.716+0000 7f7932ffd640 1 -- 172.21.15.107:0/545418231 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7934013470 con 0x7f79440a9450 2023-10-24T23:54:16.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.716+0000 7f7932ffd640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7924065e60 0x7f7924068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:16.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.716+0000 7f7932ffd640 1 -- 172.21.15.107:0/545418231 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7f79340841d0 con 0x7f79440a9450 2023-10-24T23:54:16.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.716+0000 7f7941d9b640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7924065e60 0x7f7924068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:16.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.716+0000 7f7941d9b640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7924065e60 0x7f7924068320 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79380af000 tx=0x7f79380a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:16.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:16.728+0000 7f7932ffd640 1 -- 172.21.15.107:0/545418231 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7934051530 con 0x7f79440a9450 2023-10-24T23:54:17.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.088+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79440a9aa0 con 0x7f79440a9450 2023-10-24T23:54:17.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.088+0000 7f7932ffd640 1 -- 172.21.15.107:0/545418231 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5012 (secure 0 0 0) 0x7f7934054b50 con 0x7f79440a9450 2023-10-24T23:54:17.093 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:54:17.093 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:16.358716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:54:17.094 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 16 2023-10-24T23:54:17.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7924065e60 msgr2=0x7f7924068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:17.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7924065e60 0x7f7924068320 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79380af000 tx=0x7f79380a7000 comp rx=0 tx=0).stop 2023-10-24T23:54:17.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79440a9450 msgr2=0x7f794414b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79440a9450 0x7f794414b160 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f793400e930 tx=0x7f793400edf0 comp rx=0 tx=0).stop 2023-10-24T23:54:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 shutdown_connections 2023-10-24T23:54:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79440c76c0 0x7f794413c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7944155c20 0x7f79440c7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:17.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7924065e60 0x7f7924068320 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 --2- 172.21.15.107:0/545418231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79440a9450 0x7f794414b160 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 >> 172.21.15.107:0/545418231 conn(0x7f7944065930 msgr2=0x7f79441430f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:17.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 shutdown_connections 2023-10-24T23:54:17.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:17.092+0000 7f7948b00640 1 -- 172.21.15.107:0/545418231 wait complete. 2023-10-24T23:54:17.115 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=4293 2023-10-24T23:54:17.115 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.a 2023-10-24T23:54:19.288 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-24T23:54:19.284+0000 7f273368c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:54:20.116 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2023-10-24T23:54:20.117 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-24T23:54:20.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-24T23:54:20.119 INFO:tasks.ceph.mds.a:Started 2023-10-24T23:54:20.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:54:20.199 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-24T23:54:20.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.272+0000 7fbedbd34640 1 Processor -- start 2023-10-24T23:54:20.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.272+0000 7fbedbd34640 1 -- start start 2023-10-24T23:54:20.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.272+0000 7fbedbd34640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbed41553d0 0x7fbed406ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:20.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.272+0000 7fbedbd34640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed4152400 con 0x7fbed4152020 2023-10-24T23:54:20.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.272+0000 7fbedbd34640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed41558d0 con 0x7fbed40c8640 2023-10-24T23:54:20.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.272+0000 7fbedbd34640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed40c6ce0 con 0x7fbed41553d0 2023-10-24T23:54:20.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.272+0000 7fbed9ad0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbed4152020 0x7fbed40c4bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38898/0 (socket says 172.21.15.107:38898) 2023-10-24T23:54:20.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.272+0000 7fbed9ad0640 1 -- 172.21.15.107:0/3231978639 learned_addr learned my addr 172.21.15.107:0/3231978639 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:54:20.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.272+0000 7fbeda2d1640 1 --2- 172.21.15.107:0/3231978639 >> v2:172.21.15.110:3300/0 conn(0x7fbed41553d0 0x7fbed406ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:20.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.272+0000 7fbed8ace640 1 -- 172.21.15.107:0/3231978639 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586914320 0 0) 0x7fbed4152400 con 0x7fbed4152020 2023-10-24T23:54:20.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.272+0000 7fbed8ace640 1 -- 172.21.15.107:0/3231978639 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbec0003530 con 0x7fbed4152020 2023-10-24T23:54:20.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.276+0000 7fbed8ace640 1 -- 172.21.15.107:0/3231978639 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504509226 0 0) 0x7fbed41558d0 con 0x7fbed40c8640 2023-10-24T23:54:20.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.276+0000 7fbed8ace640 1 -- 172.21.15.107:0/3231978639 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed4152400 con 0x7fbed40c8640 2023-10-24T23:54:20.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.276+0000 7fbeda2d1640 1 -- 172.21.15.107:0/3231978639 >> v1:172.21.15.110:6789/0 conn(0x7fbed40c8640 legacy=0x7fbed406a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:20.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.276+0000 7fbeda2d1640 1 -- 172.21.15.107:0/3231978639 >> v1:172.21.15.107:6789/0 conn(0x7fbed4152020 legacy=0x7fbed40c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:20.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.276+0000 7fbeda2d1640 1 -- 172.21.15.107:0/3231978639 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed40c6fc0 con 0x7fbed41553d0 2023-10-24T23:54:20.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.276+0000 7fbeda2d1640 1 --2- 172.21.15.107:0/3231978639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed41553d0 0x7fbed406ae20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009d90 tx=0x7fbecc01c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fef5ea2c09ee4e07 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:20.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.276+0000 7fbed8ace640 1 -- 172.21.15.107:0/3231978639 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc0a5970 con 0x7fbed41553d0 2023-10-24T23:54:20.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.276+0000 7fbed8ace640 1 -- 172.21.15.107:0/3231978639 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbecc0a5ad0 con 0x7fbed41553d0 2023-10-24T23:54:20.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.276+0000 7fbed8ace640 1 -- 172.21.15.107:0/3231978639 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc0a5e60 con 0x7fbed41553d0 2023-10-24T23:54:20.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 -- 172.21.15.107:0/3231978639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed41553d0 msgr2=0x7fbed406ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:20.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 --2- 172.21.15.107:0/3231978639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed41553d0 0x7fbed406ae20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009d90 tx=0x7fbecc01c890 comp rx=0 tx=0).stop 2023-10-24T23:54:20.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 -- 172.21.15.107:0/3231978639 shutdown_connections 2023-10-24T23:54:20.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 --2- 172.21.15.107:0/3231978639 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed41553d0 0x7fbed406ae20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:20.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 -- 172.21.15.107:0/3231978639 >> 172.21.15.107:0/3231978639 conn(0x7fbed40abd60 msgr2=0x7fbed40ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:20.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 -- 172.21.15.107:0/3231978639 shutdown_connections 2023-10-24T23:54:20.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 -- 172.21.15.107:0/3231978639 wait complete. 2023-10-24T23:54:20.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 Processor -- start 2023-10-24T23:54:20.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 -- start start 2023-10-24T23:54:20.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40c8640 0x7fbed414a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:20.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbed4152020 0x7fbed4147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:20.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed41553d0 0x7fbed4147850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:20.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed41edb60 con 0x7fbed4152020 2023-10-24T23:54:20.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed4155b90 con 0x7fbed41553d0 2023-10-24T23:54:20.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbedbd34640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed4155e70 con 0x7fbed40c8640 2023-10-24T23:54:20.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.280+0000 7fbed92cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbed4152020 0x7fbed4147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:20.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbed9ad0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40c8640 0x7fbed414a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:20.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbed92cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbed4152020 0x7fbed4147310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40142/0 (socket says 172.21.15.107:40142) 2023-10-24T23:54:20.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbed92cf640 1 -- 172.21.15.107:0/3397843159 learned_addr learned my addr 172.21.15.107:0/3397843159 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:54:20.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbeda2d1640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed41553d0 0x7fbed4147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:20.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbed9ad0640 1 -- 172.21.15.107:0/3397843159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed41553d0 msgr2=0x7fbed4147850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:20.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbed9ad0640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed41553d0 0x7fbed4147850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:20.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbed9ad0640 1 -- 172.21.15.107:0/3397843159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbed4152020 msgr2=0x7fbed4147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:20.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbed9ad0640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbed4152020 0x7fbed4147310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:20.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbed9ad0640 1 -- 172.21.15.107:0/3397843159 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc009a70 con 0x7fbed40c8640 2023-10-24T23:54:20.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbed9ad0640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40c8640 0x7fbed414a040 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbed000c400 tx=0x7fbed000c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:20.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbeca7fc640 1 -- 172.21.15.107:0/3397843159 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed0005cf0 con 0x7fbed40c8640 2023-10-24T23:54:20.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbeca7fc640 1 -- 172.21.15.107:0/3397843159 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed0005e50 con 0x7fbed40c8640 2023-10-24T23:54:20.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbedbd34640 1 -- 172.21.15.107:0/3397843159 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed4157850 con 0x7fbed40c8640 2023-10-24T23:54:20.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbeca7fc640 1 -- 172.21.15.107:0/3397843159 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed00072d0 con 0x7fbed40c8640 2023-10-24T23:54:20.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbedbd34640 1 -- 172.21.15.107:0/3397843159 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed4147d90 con 0x7fbed40c8640 2023-10-24T23:54:20.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbedbd34640 1 -- 172.21.15.107:0/3397843159 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed4157a70 con 0x7fbed40c8640 2023-10-24T23:54:20.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbeca7fc640 1 -- 172.21.15.107:0/3397843159 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbed0007450 con 0x7fbed40c8640 2023-10-24T23:54:20.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.284+0000 7fbeca7fc640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbeb8065e60 0x7fbeb8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:54:20.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.288+0000 7fbed92cf640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbeb8065e60 0x7fbeb8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:54:20.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.288+0000 7fbeca7fc640 1 -- 172.21.15.107:0/3397843159 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7fbed0083230 con 0x7fbed40c8640 2023-10-24T23:54:20.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.288+0000 7fbed92cf640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbeb8065e60 0x7fbeb8068320 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbed406aff0 tx=0x7fbec4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:54:20.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.296+0000 7fbeca7fc640 1 -- 172.21.15.107:0/3397843159 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbed0050570 con 0x7fbed40c8640 2023-10-24T23:54:20.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.656+0000 7fbedbd34640 1 -- 172.21.15.107:0/3397843159 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbed40746e0 con 0x7fbed40c8640 2023-10-24T23:54:20.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.656+0000 7fbeca7fc640 1 -- 172.21.15.107:0/3397843159 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fbed0053b90 con 0x7fbed40c8640 2023-10-24T23:54:20.662 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:54:20.662 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:16.358716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:54:20.663 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-10-24T23:54:20.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 -- 172.21.15.107:0/3397843159 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbeb8065e60 msgr2=0x7fbeb8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:20.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbeb8065e60 0x7fbeb8068320 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbed406aff0 tx=0x7fbec4009040 comp rx=0 tx=0).stop 2023-10-24T23:54:20.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 -- 172.21.15.107:0/3397843159 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40c8640 msgr2=0x7fbed414a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:54:20.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40c8640 0x7fbed414a040 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbed000c400 tx=0x7fbed000c8c0 comp rx=0 tx=0).stop 2023-10-24T23:54:20.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 -- 172.21.15.107:0/3397843159 shutdown_connections 2023-10-24T23:54:20.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbed41553d0 0x7fbed4147850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:20.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbed4152020 0x7fbed4147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:20.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbeb8065e60 0x7fbeb8068320 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:20.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 --2- 172.21.15.107:0/3397843159 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbed40c8640 0x7fbed414a040 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:54:20.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 -- 172.21.15.107:0/3397843159 >> 172.21.15.107:0/3397843159 conn(0x7fbed40abd60 msgr2=0x7fbed414b720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:54:20.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 -- 172.21.15.107:0/3397843159 shutdown_connections 2023-10-24T23:54:20.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:54:20.660+0000 7fbedbd34640 1 -- 172.21.15.107:0/3397843159 wait complete. 2023-10-24T23:54:20.685 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2023-10-24T23:54:20.685 INFO:tasks.mds_thrash.fs.[b]:waiting for 44 secs before thrashing 2023-10-24T23:54:34.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:54:34.015 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:54:40.425 INFO:tasks.workunit.client.0.smithi107.stderr:Note: switching to 'e652850897f60c32cb9cc05022cf9f93e3b00db1'. 2023-10-24T23:54:40.426 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-10-24T23:54:40.426 INFO:tasks.workunit.client.0.smithi107.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-24T23:54:40.427 INFO:tasks.workunit.client.0.smithi107.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-24T23:54:40.427 INFO:tasks.workunit.client.0.smithi107.stderr:state without impacting any branches by switching back to a branch. 2023-10-24T23:54:40.428 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-10-24T23:54:40.428 INFO:tasks.workunit.client.0.smithi107.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-24T23:54:40.428 INFO:tasks.workunit.client.0.smithi107.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-24T23:54:40.428 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-10-24T23:54:40.429 INFO:tasks.workunit.client.0.smithi107.stderr: git switch -c 2023-10-24T23:54:40.429 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-10-24T23:54:40.429 INFO:tasks.workunit.client.0.smithi107.stderr:Or undo this operation with: 2023-10-24T23:54:40.430 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-10-24T23:54:40.430 INFO:tasks.workunit.client.0.smithi107.stderr: git switch - 2023-10-24T23:54:40.430 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-10-24T23:54:40.430 INFO:tasks.workunit.client.0.smithi107.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-24T23:54:40.430 INFO:tasks.workunit.client.0.smithi107.stderr: 2023-10-24T23:54:40.431 INFO:tasks.workunit.client.0.smithi107.stderr:HEAD is now at e652850897f Merge branch 'wip-63282-reef' of https://github.com/vshankar/ceph into wip-yuri6-testing-2023-10-23-1148-reef 2023-10-24T23:54:40.432 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-24T23:54:40.438 INFO:tasks.workunit.client.0.smithi107.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-24T23:54:40.442 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-24T23:54:40.442 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-24T23:54:41.195 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-24T23:54:41.281 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-24T23:54:41.338 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-24T23:54:41.339 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-24T23:54:41.339 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-24T23:54:41.386 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-24T23:54:41.390 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-24T23:54:41.390 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-24T23:54:41.440 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2023-10-24T23:54:41.442 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2023-10-24T23:54:41.442 DEBUG:teuthology.orchestra.run.smithi107:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e652850897f60c32cb9cc05022cf9f93e3b00db1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2023-10-24T23:54:41.563 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into 'ffsb'... 2023-10-24T23:54:42.744 INFO:tasks.workunit.client.1.smithi110.stderr:Note: switching to 'e652850897f60c32cb9cc05022cf9f93e3b00db1'. 2023-10-24T23:54:42.744 INFO:tasks.workunit.client.1.smithi110.stderr: 2023-10-24T23:54:42.744 INFO:tasks.workunit.client.1.smithi110.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-24T23:54:42.745 INFO:tasks.workunit.client.1.smithi110.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-24T23:54:42.745 INFO:tasks.workunit.client.1.smithi110.stderr:state without impacting any branches by switching back to a branch. 2023-10-24T23:54:42.745 INFO:tasks.workunit.client.1.smithi110.stderr: 2023-10-24T23:54:42.745 INFO:tasks.workunit.client.1.smithi110.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-24T23:54:42.746 INFO:tasks.workunit.client.1.smithi110.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-24T23:54:42.746 INFO:tasks.workunit.client.1.smithi110.stderr: 2023-10-24T23:54:42.746 INFO:tasks.workunit.client.1.smithi110.stderr: git switch -c 2023-10-24T23:54:42.746 INFO:tasks.workunit.client.1.smithi110.stderr: 2023-10-24T23:54:42.746 INFO:tasks.workunit.client.1.smithi110.stderr:Or undo this operation with: 2023-10-24T23:54:42.747 INFO:tasks.workunit.client.1.smithi110.stderr: 2023-10-24T23:54:42.747 INFO:tasks.workunit.client.1.smithi110.stderr: git switch - 2023-10-24T23:54:42.747 INFO:tasks.workunit.client.1.smithi110.stderr: 2023-10-24T23:54:42.747 INFO:tasks.workunit.client.1.smithi110.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-24T23:54:42.747 INFO:tasks.workunit.client.1.smithi110.stderr: 2023-10-24T23:54:42.748 INFO:tasks.workunit.client.1.smithi110.stderr:HEAD is now at e652850897f Merge branch 'wip-63282-reef' of https://github.com/vshankar/ceph into wip-yuri6-testing-2023-10-23-1148-reef 2023-10-24T23:54:42.749 DEBUG:teuthology.orchestra.run.smithi110:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2023-10-24T23:54:42.755 INFO:tasks.workunit.client.1.smithi110.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-24T23:54:42.758 INFO:tasks.workunit.client.1.smithi110.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-10-24T23:54:42.758 INFO:tasks.workunit.client.1.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-24T23:54:43.537 INFO:tasks.workunit.client.1.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-24T23:54:43.619 INFO:tasks.workunit.client.1.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-24T23:54:43.676 INFO:tasks.workunit.client.1.smithi110.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-10-24T23:54:43.677 INFO:tasks.workunit.client.1.smithi110.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-10-24T23:54:43.677 INFO:tasks.workunit.client.1.smithi110.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-24T23:54:43.721 INFO:tasks.workunit.client.1.smithi110.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-10-24T23:54:43.724 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-24T23:54:43.724 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2023-10-24T23:54:43.771 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.1... 2023-10-24T23:54:43.772 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2023-10-24T23:54:43.772 DEBUG:teuthology.orchestra.run.smithi110:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e652850897f60c32cb9cc05022cf9f93e3b00db1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh 2023-10-24T23:54:43.845 INFO:tasks.workunit.client.1.smithi110.stderr:Cloning into 'ffsb'... 2023-10-24T23:54:47.030 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2023-10-24T23:54:48.050 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether build environment is sane... yes 2023-10-24T23:54:48.073 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2023-10-24T23:54:48.074 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gawk... gawk 2023-10-24T23:54:48.870 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether make sets $(MAKE)... yes 2023-10-24T23:54:48.929 INFO:tasks.workunit.client.0.smithi107.stdout:checking build system type... x86_64-unknown-linux-gnu 2023-10-24T23:54:48.929 INFO:tasks.workunit.client.0.smithi107.stdout:checking host system type... x86_64-unknown-linux-gnu 2023-10-24T23:54:48.964 INFO:tasks.workunit.client.0.smithi107.stdout:checking for style of include used by make... GNU 2023-10-24T23:54:48.971 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... gcc 2023-10-24T23:54:49.093 INFO:tasks.workunit.client.0.smithi107.stdout:checking for C compiler default output file name... a.out 2023-10-24T23:54:49.097 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the C compiler works... yes 2023-10-24T23:54:49.102 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are cross compiling... no 2023-10-24T23:54:49.174 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of executables... 2023-10-24T23:54:49.234 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of object files... o 2023-10-24T23:54:49.286 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are using the GNU C compiler... yes 2023-10-24T23:54:49.339 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... yes 2023-10-24T23:54:49.452 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to accept ISO C89... none needed 2023-10-24T23:54:49.852 INFO:tasks.workunit.client.1.smithi110.stderr:Updating files: 45% (27/59) Updating files: 47% (28/59) Updating files: 49% (29/59) Updating files: 50% (30/59) Updating files: 52% (31/59) Updating files: 54% (32/59) Updating files: 55% (33/59) Updating files: 57% (34/59) Updating files: 59% (35/59) Updating files: 61% (36/59) Updating files: 62% (37/59) Updating files: 64% (38/59) Updating files: 66% (39/59) Updating files: 67% (40/59) Updating files: 69% (41/59) Updating files: 71% (42/59) Updating files: 72% (43/59) Updating files: 74% (44/59) Updating files: 76% (45/59) Updating files: 77% (46/59) Updating files: 79% (47/59) Updating files: 81% (48/59) Updating files: 83% (49/59) Updating files: 84% (50/59) Updating files: 86% (51/59) Updating files: 88% (52/59) Updating files: 89% (53/59) Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2023-10-24T23:54:50.260 INFO:tasks.workunit.client.1.smithi110.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2023-10-24T23:54:50.377 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... gcc3 2023-10-24T23:54:50.493 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to run the C preprocessor... gcc -E 2023-10-24T23:54:50.577 INFO:tasks.workunit.client.0.smithi107.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2023-10-24T23:54:50.579 INFO:tasks.workunit.client.0.smithi107.stdout:checking for egrep... /usr/bin/grep -E 2023-10-24T23:54:50.596 INFO:tasks.workunit.client.0.smithi107.stdout:checking for AIX... no 2023-10-24T23:54:50.673 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... (cached) gcc 2023-10-24T23:54:50.697 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are using the GNU C compiler... (cached) yes 2023-10-24T23:54:50.699 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... (cached) yes 2023-10-24T23:54:50.700 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2023-10-24T23:54:50.701 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... (cached) gcc3 2023-10-24T23:54:50.828 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lm... yes 2023-10-24T23:54:50.937 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lpthread... yes 2023-10-24T23:54:51.208 INFO:tasks.workunit.client.0.smithi107.stdout:checking for ANSI C header files... yes 2023-10-24T23:54:51.275 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether build environment is sane... yes 2023-10-24T23:54:51.296 INFO:tasks.workunit.client.1.smithi110.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2023-10-24T23:54:51.298 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gawk... gawk 2023-10-24T23:54:51.338 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether make sets $(MAKE)... yes 2023-10-24T23:54:51.398 INFO:tasks.workunit.client.1.smithi110.stdout:checking build system type... x86_64-unknown-linux-gnu 2023-10-24T23:54:51.399 INFO:tasks.workunit.client.1.smithi110.stdout:checking host system type... x86_64-unknown-linux-gnu 2023-10-24T23:54:51.434 INFO:tasks.workunit.client.1.smithi110.stdout:checking for style of include used by make... GNU 2023-10-24T23:54:51.443 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gcc... gcc 2023-10-24T23:54:51.565 INFO:tasks.workunit.client.1.smithi110.stdout:checking for C compiler default output file name... a.out 2023-10-24T23:54:51.568 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether the C compiler works... yes 2023-10-24T23:54:51.573 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether we are cross compiling... no 2023-10-24T23:54:51.645 INFO:tasks.workunit.client.1.smithi110.stdout:checking for suffix of executables... 2023-10-24T23:54:51.706 INFO:tasks.workunit.client.1.smithi110.stdout:checking for suffix of object files... o 2023-10-24T23:54:51.761 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether we are using the GNU C compiler... yes 2023-10-24T23:54:51.825 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether gcc accepts -g... yes 2023-10-24T23:54:51.956 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gcc option to accept ISO C89... none needed 2023-10-24T23:54:52.205 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2023-10-24T23:54:52.250 INFO:tasks.workunit.client.1.smithi110.stdout:checking dependency style of gcc... gcc3 2023-10-24T23:54:52.291 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/types.h... yes 2023-10-24T23:54:52.365 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/stat.h... yes 2023-10-24T23:54:52.444 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdlib.h... yes 2023-10-24T23:54:52.554 INFO:tasks.workunit.client.0.smithi107.stdout:checking for string.h... yes 2023-10-24T23:54:52.662 INFO:tasks.workunit.client.0.smithi107.stdout:checking for memory.h... yes 2023-10-24T23:54:52.999 INFO:tasks.workunit.client.1.smithi110.stdout:checking how to run the C preprocessor... gcc -E 2023-10-24T23:54:53.105 INFO:tasks.workunit.client.1.smithi110.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2023-10-24T23:54:53.107 INFO:tasks.workunit.client.1.smithi110.stdout:checking for egrep... /usr/bin/grep -E 2023-10-24T23:54:53.125 INFO:tasks.workunit.client.1.smithi110.stdout:checking for AIX... no 2023-10-24T23:54:53.181 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gcc... (cached) gcc 2023-10-24T23:54:53.198 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether we are using the GNU C compiler... (cached) yes 2023-10-24T23:54:53.199 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether gcc accepts -g... (cached) yes 2023-10-24T23:54:53.200 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2023-10-24T23:54:53.200 INFO:tasks.workunit.client.1.smithi110.stdout:checking dependency style of gcc... (cached) gcc3 2023-10-24T23:54:53.338 INFO:tasks.workunit.client.1.smithi110.stdout:checking for main in -lm... yes 2023-10-24T23:54:53.410 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strings.h... yes 2023-10-24T23:54:53.433 INFO:tasks.workunit.client.1.smithi110.stdout:checking for main in -lpthread... yes 2023-10-24T23:54:53.486 INFO:tasks.workunit.client.0.smithi107.stdout:checking for inttypes.h... yes 2023-10-24T23:54:53.557 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... yes 2023-10-24T23:54:53.657 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... yes 2023-10-24T23:54:53.783 INFO:tasks.workunit.client.0.smithi107.stdout:checking pthread.h usability... yes 2023-10-24T23:54:53.829 INFO:tasks.workunit.client.0.smithi107.stdout:checking pthread.h presence... yes 2023-10-24T23:54:53.830 INFO:tasks.workunit.client.0.smithi107.stdout:checking for pthread.h... yes 2023-10-24T23:54:53.963 INFO:tasks.workunit.client.0.smithi107.stdout:checking fcntl.h usability... yes 2023-10-24T23:54:54.003 INFO:tasks.workunit.client.0.smithi107.stdout:checking fcntl.h presence... yes 2023-10-24T23:54:54.004 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fcntl.h... yes 2023-10-24T23:54:54.127 INFO:tasks.workunit.client.0.smithi107.stdout:checking limits.h usability... yes 2023-10-24T23:54:54.166 INFO:tasks.workunit.client.0.smithi107.stdout:checking limits.h presence... yes 2023-10-24T23:54:54.167 INFO:tasks.workunit.client.0.smithi107.stdout:checking for limits.h... yes 2023-10-24T23:54:54.207 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... (cached) yes 2023-10-24T23:54:54.292 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/time.h usability... yes 2023-10-24T23:54:54.331 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/time.h presence... yes 2023-10-24T23:54:54.332 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/time.h... yes 2023-10-24T23:54:54.375 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... (cached) yes 2023-10-24T23:54:54.455 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/vfs.h usability... yes 2023-10-24T23:54:54.482 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/vfs.h presence... yes 2023-10-24T23:54:54.483 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/vfs.h... yes 2023-10-24T23:54:54.538 INFO:tasks.workunit.client.1.smithi110.stdout:checking for ANSI C header files... yes 2023-10-24T23:54:54.575 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/limits.h usability... no 2023-10-24T23:54:54.603 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/limits.h presence... no 2023-10-24T23:54:54.603 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/limits.h... no 2023-10-24T23:54:54.620 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2023-10-24T23:54:54.646 INFO:tasks.workunit.client.0.smithi107.stdout:checking for an ANSI C-conforming const... yes 2023-10-24T23:54:54.684 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/types.h... yes 2023-10-24T23:54:54.689 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether time.h and sys/time.h may both be included... yes 2023-10-24T23:54:54.692 INFO:tasks.workunit.client.0.smithi107.stdout:checking for function prototypes... yes 2023-10-24T23:54:54.751 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/stat.h... yes 2023-10-24T23:54:54.824 INFO:tasks.workunit.client.1.smithi110.stdout:checking for stdlib.h... yes 2023-10-24T23:54:54.846 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether setvbuf arguments are reversed... no 2023-10-24T23:54:54.905 INFO:tasks.workunit.client.1.smithi110.stdout:checking for string.h... yes 2023-10-24T23:54:54.934 INFO:tasks.workunit.client.0.smithi107.stdout:checking for system... yes 2023-10-24T23:54:54.981 INFO:tasks.workunit.client.1.smithi110.stdout:checking for memory.h... yes 2023-10-24T23:54:55.026 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gettimeofday... yes 2023-10-24T23:54:55.051 INFO:tasks.workunit.client.1.smithi110.stdout:checking for strings.h... yes 2023-10-24T23:54:55.111 INFO:tasks.workunit.client.0.smithi107.stdout:checking for mkdir... yes 2023-10-24T23:54:55.127 INFO:tasks.workunit.client.1.smithi110.stdout:checking for inttypes.h... yes 2023-10-24T23:54:55.198 INFO:tasks.workunit.client.1.smithi110.stdout:checking for stdint.h... yes 2023-10-24T23:54:55.198 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strerror... yes 2023-10-24T23:54:55.280 INFO:tasks.workunit.client.1.smithi110.stdout:checking for unistd.h... yes 2023-10-24T23:54:55.368 INFO:tasks.workunit.client.1.smithi110.stdout:checking pthread.h usability... yes 2023-10-24T23:54:55.397 INFO:tasks.workunit.client.1.smithi110.stdout:checking pthread.h presence... yes 2023-10-24T23:54:55.398 INFO:tasks.workunit.client.1.smithi110.stdout:checking for pthread.h... yes 2023-10-24T23:54:55.486 INFO:tasks.workunit.client.1.smithi110.stdout:checking fcntl.h usability... yes 2023-10-24T23:54:55.520 INFO:tasks.workunit.client.1.smithi110.stdout:checking fcntl.h presence... yes 2023-10-24T23:54:55.521 INFO:tasks.workunit.client.1.smithi110.stdout:checking for fcntl.h... yes 2023-10-24T23:54:55.615 INFO:tasks.workunit.client.1.smithi110.stdout:checking limits.h usability... yes 2023-10-24T23:54:55.642 INFO:tasks.workunit.client.1.smithi110.stdout:checking limits.h presence... yes 2023-10-24T23:54:55.643 INFO:tasks.workunit.client.1.smithi110.stdout:checking for limits.h... yes 2023-10-24T23:54:55.666 INFO:tasks.workunit.client.0.smithi107.stdout:checking for open64... yes 2023-10-24T23:54:55.669 INFO:tasks.workunit.client.1.smithi110.stdout:checking for stdint.h... (cached) yes 2023-10-24T23:54:55.728 INFO:tasks.workunit.client.1.smithi110.stdout:checking sys/time.h usability... yes 2023-10-24T23:54:55.753 INFO:tasks.workunit.client.1.smithi110.stdout:checking sys/time.h presence... yes 2023-10-24T23:54:55.754 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/time.h... yes 2023-10-24T23:54:55.757 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stat64... yes 2023-10-24T23:54:55.776 INFO:tasks.workunit.client.1.smithi110.stdout:checking for unistd.h... (cached) yes 2023-10-24T23:54:55.844 INFO:tasks.workunit.client.1.smithi110.stdout:checking sys/vfs.h usability... yes 2023-10-24T23:54:55.870 INFO:tasks.workunit.client.1.smithi110.stdout:checking sys/vfs.h presence... yes 2023-10-24T23:54:55.870 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/vfs.h... yes 2023-10-24T23:54:55.977 INFO:tasks.workunit.client.1.smithi110.stdout:checking sys/limits.h usability... no 2023-10-24T23:54:56.008 INFO:tasks.workunit.client.1.smithi110.stdout:checking sys/limits.h presence... no 2023-10-24T23:54:56.008 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/limits.h... no 2023-10-24T23:54:56.048 INFO:tasks.workunit.client.1.smithi110.stdout:checking for an ANSI C-conforming const... yes 2023-10-24T23:54:56.091 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether time.h and sys/time.h may both be included... yes 2023-10-24T23:54:56.094 INFO:tasks.workunit.client.1.smithi110.stdout:checking for function prototypes... yes 2023-10-24T23:54:56.135 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether setvbuf arguments are reversed... no 2023-10-24T23:54:56.235 INFO:tasks.workunit.client.1.smithi110.stdout:checking for system... yes 2023-10-24T23:54:56.586 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gettimeofday... yes 2023-10-24T23:54:56.794 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fseeko64... yes 2023-10-24T23:54:56.955 INFO:tasks.workunit.client.0.smithi107.stdout:checking for lrand48_r... yes 2023-10-24T23:54:57.070 INFO:tasks.workunit.client.0.smithi107.stdout:checking for srand48_r... yes 2023-10-24T23:54:57.079 INFO:tasks.workunit.client.1.smithi110.stdout:checking for mkdir... yes 2023-10-24T23:54:57.116 INFO:tasks.workunit.client.0.smithi107.stdout:configure: creating ./config.status 2023-10-24T23:54:57.187 INFO:tasks.workunit.client.1.smithi110.stdout:checking for strerror... yes 2023-10-24T23:54:57.308 INFO:tasks.workunit.client.1.smithi110.stdout:checking for open64... yes 2023-10-24T23:54:57.449 INFO:tasks.workunit.client.1.smithi110.stdout:checking for stat64... yes 2023-10-24T23:54:58.214 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating Makefile 2023-10-24T23:54:58.241 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating config.h 2023-10-24T23:54:58.272 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing depfiles commands 2023-10-24T23:54:58.314 INFO:tasks.workunit.client.1.smithi110.stdout:checking for fseeko64... yes 2023-10-24T23:54:58.367 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing default commands 2023-10-24T23:54:58.479 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2023-10-24T23:54:58.487 INFO:tasks.workunit.client.0.smithi107.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2023-10-24T23:54:58.488 INFO:tasks.workunit.client.0.smithi107.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2023-10-24T23:54:58.488 INFO:tasks.workunit.client.0.smithi107.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2023-10-24T23:54:58.489 INFO:tasks.workunit.client.0.smithi107.stderr: to install the `Automake' and `Perl' packages. Grab them from 2023-10-24T23:54:58.489 INFO:tasks.workunit.client.0.smithi107.stderr: any GNU archive site. 2023-10-24T23:54:58.494 INFO:tasks.workunit.client.0.smithi107.stdout: cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2023-10-24T23:54:58.500 INFO:tasks.workunit.client.0.smithi107.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2023-10-24T23:54:58.501 INFO:tasks.workunit.client.0.smithi107.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2023-10-24T23:54:58.501 INFO:tasks.workunit.client.0.smithi107.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2023-10-24T23:54:58.501 INFO:tasks.workunit.client.0.smithi107.stderr: You might want to install the `Automake' and `Perl' packages. 2023-10-24T23:54:58.502 INFO:tasks.workunit.client.0.smithi107.stderr: Grab them from any GNU archive site. 2023-10-24T23:54:59.438 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2023-10-24T23:54:59.611 INFO:tasks.workunit.client.0.smithi107.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2023-10-24T23:54:59.612 INFO:tasks.workunit.client.0.smithi107.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2023-10-24T23:54:59.612 INFO:tasks.workunit.client.0.smithi107.stderr:If you have problems, you may need to regenerate the build system entirely. 2023-10-24T23:54:59.612 INFO:tasks.workunit.client.0.smithi107.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2023-10-24T23:54:59.724 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:4: warning: The macro `AC_FOREACH' is obsolete. 2023-10-24T23:54:59.724 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:4: You should run autoupdate. 2023-10-24T23:54:59.724 INFO:tasks.workunit.client.0.smithi107.stderr:./lib/autoconf/general.m4:191: AC_FOREACH is expanded from... 2023-10-24T23:54:59.725 INFO:tasks.workunit.client.0.smithi107.stderr:aclocal.m4:684: _AM_SET_OPTIONS is expanded from... 2023-10-24T23:54:59.725 INFO:tasks.workunit.client.0.smithi107.stderr:aclocal.m4:410: AM_INIT_AUTOMAKE is expanded from... 2023-10-24T23:54:59.725 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:4: the top level 2023-10-24T23:54:59.725 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:6: warning: The macro `AM_CONFIG_HEADER' is obsolete. 2023-10-24T23:54:59.725 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:6: You should run autoupdate. 2023-10-24T23:54:59.726 INFO:tasks.workunit.client.0.smithi107.stderr:aclocal.m4:383: AM_CONFIG_HEADER is expanded from... 2023-10-24T23:54:59.726 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:6: the top level 2023-10-24T23:54:59.726 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:10: warning: The macro `AC_AIX' is obsolete. 2023-10-24T23:54:59.726 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:10: You should run autoupdate. 2023-10-24T23:54:59.726 INFO:tasks.workunit.client.0.smithi107.stderr:./lib/autoconf/specific.m4:540: AC_AIX is expanded from... 2023-10-24T23:54:59.727 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:10: the top level 2023-10-24T23:54:59.727 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:36: warning: The macro `AC_HEADER_STDC' is obsolete. 2023-10-24T23:54:59.727 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:36: You should run autoupdate. 2023-10-24T23:54:59.727 INFO:tasks.workunit.client.0.smithi107.stderr:./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... 2023-10-24T23:54:59.727 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:36: the top level 2023-10-24T23:54:59.728 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:42: warning: The macro `AC_HEADER_TIME' is obsolete. 2023-10-24T23:54:59.728 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:42: You should run autoupdate. 2023-10-24T23:54:59.728 INFO:tasks.workunit.client.0.smithi107.stderr:./lib/autoconf/headers.m4:743: AC_HEADER_TIME is expanded from... 2023-10-24T23:54:59.728 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:42: the top level 2023-10-24T23:54:59.728 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:45: warning: The macro `AC_FUNC_SETVBUF_REVERSED' is obsolete. Remove it and all references to SETVBUF_REVERSED. 2023-10-24T23:54:59.729 INFO:tasks.workunit.client.0.smithi107.stderr:./lib/autoconf/functions.m4:1766: AC_FUNC_SETVBUF_REVERSED is expanded from... 2023-10-24T23:54:59.729 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:45: the top level 2023-10-24T23:54:59.729 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:50: warning: AC_OUTPUT should be used without arguments. 2023-10-24T23:54:59.729 INFO:tasks.workunit.client.0.smithi107.stderr:configure.in:50: You should run autoupdate. 2023-10-24T23:54:59.758 INFO:tasks.workunit.client.1.smithi110.stdout:checking for lrand48_r... yes 2023-10-24T23:54:59.818 INFO:tasks.workunit.client.0.smithi107.stdout:/bin/bash ./config.status --recheck 2023-10-24T23:54:59.875 INFO:tasks.workunit.client.1.smithi110.stdout:checking for srand48_r... yes 2023-10-24T23:54:59.898 INFO:tasks.workunit.client.0.smithi107.stdout:running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 2023-10-24T23:54:59.912 INFO:tasks.workunit.client.1.smithi110.stdout:configure: creating ./config.status 2023-10-24T23:55:00.061 INFO:tasks.workunit.client.1.smithi110.stdout:config.status: creating Makefile 2023-10-24T23:55:00.084 INFO:tasks.workunit.client.1.smithi110.stdout:config.status: creating config.h 2023-10-24T23:55:00.115 INFO:tasks.workunit.client.1.smithi110.stdout:config.status: executing depfiles commands 2023-10-24T23:55:00.210 INFO:tasks.workunit.client.1.smithi110.stdout:config.status: executing default commands 2023-10-24T23:55:00.664 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2023-10-24T23:55:00.670 INFO:tasks.workunit.client.1.smithi110.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run aclocal-1.10 2023-10-24T23:55:00.680 INFO:tasks.workunit.client.1.smithi110.stderr:/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2023-10-24T23:55:00.682 INFO:tasks.workunit.client.1.smithi110.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2023-10-24T23:55:00.682 INFO:tasks.workunit.client.1.smithi110.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2023-10-24T23:55:00.682 INFO:tasks.workunit.client.1.smithi110.stderr: to install the `Automake' and `Perl' packages. Grab them from 2023-10-24T23:55:00.682 INFO:tasks.workunit.client.1.smithi110.stderr: any GNU archive site. 2023-10-24T23:55:00.688 INFO:tasks.workunit.client.1.smithi110.stdout: cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run automake-1.10 --gnu 2023-10-24T23:55:00.694 INFO:tasks.workunit.client.1.smithi110.stderr:/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing: line 46: automake-1.10: command not found 2023-10-24T23:55:00.696 INFO:tasks.workunit.client.1.smithi110.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2023-10-24T23:55:00.696 INFO:tasks.workunit.client.1.smithi110.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2023-10-24T23:55:00.696 INFO:tasks.workunit.client.1.smithi110.stderr: You might want to install the `Automake' and `Perl' packages. 2023-10-24T23:55:00.696 INFO:tasks.workunit.client.1.smithi110.stderr: Grab them from any GNU archive site. 2023-10-24T23:55:01.681 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether build environment is sane... yes 2023-10-24T23:55:01.695 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a race-free mkdir -p... /usr/bin/mkdir -p 2023-10-24T23:55:01.696 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gawk... gawk 2023-10-24T23:55:01.734 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether make sets $(MAKE)... yes 2023-10-24T23:55:01.782 INFO:tasks.workunit.client.0.smithi107.stdout:checking build system type... x86_64-unknown-linux-gnu 2023-10-24T23:55:01.782 INFO:tasks.workunit.client.0.smithi107.stdout:checking host system type... x86_64-unknown-linux-gnu 2023-10-24T23:55:01.814 INFO:tasks.workunit.client.0.smithi107.stdout:checking for style of include used by make... GNU 2023-10-24T23:55:01.822 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... gcc 2023-10-24T23:55:02.015 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the C compiler works... yes 2023-10-24T23:55:02.016 INFO:tasks.workunit.client.0.smithi107.stdout:checking for C compiler default output file name... a.out 2023-10-24T23:55:02.082 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of executables... 2023-10-24T23:55:02.110 INFO:tasks.workunit.client.1.smithi110.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run autoconf 2023-10-24T23:55:02.139 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are cross compiling... no 2023-10-24T23:55:02.379 INFO:tasks.workunit.client.1.smithi110.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2023-10-24T23:55:02.380 INFO:tasks.workunit.client.1.smithi110.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2023-10-24T23:55:02.380 INFO:tasks.workunit.client.1.smithi110.stderr:If you have problems, you may need to regenerate the build system entirely. 2023-10-24T23:55:02.380 INFO:tasks.workunit.client.1.smithi110.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2023-10-24T23:55:02.505 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:4: warning: The macro `AC_FOREACH' is obsolete. 2023-10-24T23:55:02.505 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:4: You should run autoupdate. 2023-10-24T23:55:02.506 INFO:tasks.workunit.client.1.smithi110.stderr:./lib/autoconf/general.m4:191: AC_FOREACH is expanded from... 2023-10-24T23:55:02.506 INFO:tasks.workunit.client.1.smithi110.stderr:aclocal.m4:684: _AM_SET_OPTIONS is expanded from... 2023-10-24T23:55:02.506 INFO:tasks.workunit.client.1.smithi110.stderr:aclocal.m4:410: AM_INIT_AUTOMAKE is expanded from... 2023-10-24T23:55:02.506 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:4: the top level 2023-10-24T23:55:02.506 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:6: warning: The macro `AM_CONFIG_HEADER' is obsolete. 2023-10-24T23:55:02.507 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:6: You should run autoupdate. 2023-10-24T23:55:02.507 INFO:tasks.workunit.client.1.smithi110.stderr:aclocal.m4:383: AM_CONFIG_HEADER is expanded from... 2023-10-24T23:55:02.507 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:6: the top level 2023-10-24T23:55:02.507 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:10: warning: The macro `AC_AIX' is obsolete. 2023-10-24T23:55:02.507 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:10: You should run autoupdate. 2023-10-24T23:55:02.508 INFO:tasks.workunit.client.1.smithi110.stderr:./lib/autoconf/specific.m4:540: AC_AIX is expanded from... 2023-10-24T23:55:02.508 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:10: the top level 2023-10-24T23:55:02.508 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:36: warning: The macro `AC_HEADER_STDC' is obsolete. 2023-10-24T23:55:02.508 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:36: You should run autoupdate. 2023-10-24T23:55:02.508 INFO:tasks.workunit.client.1.smithi110.stderr:./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... 2023-10-24T23:55:02.508 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:36: the top level 2023-10-24T23:55:02.509 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:42: warning: The macro `AC_HEADER_TIME' is obsolete. 2023-10-24T23:55:02.509 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:42: You should run autoupdate. 2023-10-24T23:55:02.509 INFO:tasks.workunit.client.1.smithi110.stderr:./lib/autoconf/headers.m4:743: AC_HEADER_TIME is expanded from... 2023-10-24T23:55:02.509 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:42: the top level 2023-10-24T23:55:02.509 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:45: warning: The macro `AC_FUNC_SETVBUF_REVERSED' is obsolete. Remove it and all references to SETVBUF_REVERSED. 2023-10-24T23:55:02.510 INFO:tasks.workunit.client.1.smithi110.stderr:./lib/autoconf/functions.m4:1766: AC_FUNC_SETVBUF_REVERSED is expanded from... 2023-10-24T23:55:02.510 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:45: the top level 2023-10-24T23:55:02.510 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:50: warning: AC_OUTPUT should be used without arguments. 2023-10-24T23:55:02.510 INFO:tasks.workunit.client.1.smithi110.stderr:configure.in:50: You should run autoupdate. 2023-10-24T23:55:02.534 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of object files... o 2023-10-24T23:55:02.570 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the compiler supports GNU C... yes 2023-10-24T23:55:02.581 INFO:tasks.workunit.client.1.smithi110.stdout:/bin/bash ./config.status --recheck 2023-10-24T23:55:02.607 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... yes 2023-10-24T23:55:02.690 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to enable C11 features... none needed 2023-10-24T23:55:02.756 INFO:tasks.workunit.client.1.smithi110.stdout:running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 2023-10-24T23:55:02.979 INFO:tasks.workunit.client.1.smithi110.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2023-10-24T23:55:03.265 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... gcc3 2023-10-24T23:55:03.319 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdio.h... yes 2023-10-24T23:55:03.390 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdlib.h... yes 2023-10-24T23:55:03.454 INFO:tasks.workunit.client.0.smithi107.stdout:checking for string.h... yes 2023-10-24T23:55:03.506 INFO:tasks.workunit.client.0.smithi107.stdout:checking for inttypes.h... yes 2023-10-24T23:55:03.558 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... yes 2023-10-24T23:55:03.614 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strings.h... yes 2023-10-24T23:55:03.679 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/stat.h... yes 2023-10-24T23:55:03.732 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/types.h... yes 2023-10-24T23:55:03.795 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... yes 2023-10-24T23:55:03.872 INFO:tasks.workunit.client.0.smithi107.stdout:checking for wchar.h... yes 2023-10-24T23:55:03.956 INFO:tasks.workunit.client.0.smithi107.stdout:checking for minix/config.h... no 2023-10-24T23:55:03.993 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether build environment is sane... yes 2023-10-24T23:55:04.006 INFO:tasks.workunit.client.1.smithi110.stdout:checking for a race-free mkdir -p... /usr/bin/mkdir -p 2023-10-24T23:55:04.007 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gawk... gawk 2023-10-24T23:55:04.012 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/time.h... yes 2023-10-24T23:55:04.033 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether make sets $(MAKE)... yes 2023-10-24T23:55:04.077 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2023-10-24T23:55:04.084 INFO:tasks.workunit.client.1.smithi110.stdout:checking build system type... x86_64-unknown-linux-gnu 2023-10-24T23:55:04.085 INFO:tasks.workunit.client.1.smithi110.stdout:checking host system type... x86_64-unknown-linux-gnu 2023-10-24T23:55:04.106 INFO:tasks.workunit.client.1.smithi110.stdout:checking for style of include used by make... GNU 2023-10-24T23:55:04.112 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gcc... gcc 2023-10-24T23:55:04.122 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether _XOPEN_SOURCE should be defined... no 2023-10-24T23:55:04.144 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... (cached) gcc 2023-10-24T23:55:04.215 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the compiler supports GNU C... (cached) yes 2023-10-24T23:55:04.215 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... (cached) yes 2023-10-24T23:55:04.217 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to enable C11 features... (cached) none needed 2023-10-24T23:55:04.217 INFO:tasks.workunit.client.0.smithi107.stdout:checking dependency style of gcc... (cached) gcc3 2023-10-24T23:55:04.270 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether the C compiler works... yes 2023-10-24T23:55:04.270 INFO:tasks.workunit.client.1.smithi110.stdout:checking for C compiler default output file name... a.out 2023-10-24T23:55:04.279 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lm... yes 2023-10-24T23:55:04.335 INFO:tasks.workunit.client.1.smithi110.stdout:checking for suffix of executables... 2023-10-24T23:55:04.343 INFO:tasks.workunit.client.0.smithi107.stdout:checking for main in -lpthread... yes 2023-10-24T23:55:04.352 INFO:tasks.workunit.client.0.smithi107.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2023-10-24T23:55:04.353 INFO:tasks.workunit.client.0.smithi107.stdout:checking for egrep... /usr/bin/grep -E 2023-10-24T23:55:04.395 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2023-10-24T23:55:04.396 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether we are cross compiling... no 2023-10-24T23:55:04.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:55:04.419 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:55:04.448 INFO:tasks.workunit.client.1.smithi110.stdout:checking for suffix of object files... o 2023-10-24T23:55:04.451 INFO:tasks.workunit.client.0.smithi107.stdout:checking for pthread.h... yes 2023-10-24T23:55:04.480 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether the compiler supports GNU C... yes 2023-10-24T23:55:04.502 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fcntl.h... yes 2023-10-24T23:55:04.510 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether gcc accepts -g... yes 2023-10-24T23:55:04.550 INFO:tasks.workunit.client.0.smithi107.stdout:checking for limits.h... yes 2023-10-24T23:55:04.551 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... (cached) yes 2023-10-24T23:55:04.552 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/time.h... (cached) yes 2023-10-24T23:55:04.553 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... (cached) yes 2023-10-24T23:55:04.597 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/vfs.h... yes 2023-10-24T23:55:04.601 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gcc option to enable C11 features... none needed 2023-10-24T23:55:04.662 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/limits.h... no 2023-10-24T23:55:04.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:04.702 INFO:tasks.workunit.client.0.smithi107.stdout:checking for an ANSI C-conforming const... yes 2023-10-24T23:55:04.781 INFO:tasks.workunit.client.0.smithi107.stdout:checking for system... yes 2023-10-24T23:55:04.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe372ab4640 1 Processor -- start 2023-10-24T23:55:04.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe372ab4640 1 -- start start 2023-10-24T23:55:04.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe372ab4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe36c06f490 0x7fe36c14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:04.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe372ab4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe36c152b30 0x7fe36c152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:04.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe372ab4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c155cd0 con 0x7fe36c14c470 2023-10-24T23:55:04.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe372ab4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36c1526e0 con 0x7fe36c152b30 2023-10-24T23:55:04.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe372ab4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0c4380 con 0x7fe36c06f490 2023-10-24T23:55:04.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe370850640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe36c14c470 0x7fe36c071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36980/0 (socket says 172.21.15.107:36980) 2023-10-24T23:55:04.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe370850640 1 -- 172.21.15.107:0/1479475670 learned_addr learned my addr 172.21.15.107:0/1479475670 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:04.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36bfff640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe36c06f490 0x7fe36c14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:04.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe371051640 1 --2- 172.21.15.107:0/1479475670 >> v2:172.21.15.110:3300/0 conn(0x7fe36c152b30 0x7fe36c152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:04.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939147429 0 0) 0x7fe36c155cd0 con 0x7fe36c14c470 2023-10-24T23:55:04.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe35c003530 con 0x7fe36c14c470 2023-10-24T23:55:04.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3195145946 0 0) 0x7fe35c003530 con 0x7fe36c14c470 2023-10-24T23:55:04.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe36c06f490 msgr2=0x7fe36c14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:04.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe36c06f490 0x7fe36c14c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:04.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 >> v2:172.21.15.110:3300/0 conn(0x7fe36c152b30 msgr2=0x7fe36c152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:04.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 --2- 172.21.15.107:0/1479475670 >> v2:172.21.15.110:3300/0 conn(0x7fe36c152b30 0x7fe36c152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:04.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0c48b0 con 0x7fe36c14c470 2023-10-24T23:55:04.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fe358001680 con 0x7fe36c14c470 2023-10-24T23:55:04.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 >> v1:172.21.15.107:6789/0 conn(0x7fe36c14c470 legacy=0x7fe36c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:04.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe35c003980 0x7fe35c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:04.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 0x7fe35c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:04.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe35c005190 0x7fe35c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:04.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c1526e0 con 0x7fe35c0047d0 2023-10-24T23:55:04.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0c4380 con 0x7fe35c003980 2023-10-24T23:55:04.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c00bfb0 con 0x7fe35c005190 2023-10-24T23:55:04.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36bfff640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe36c06f490 0x7fe36c14c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:55:04.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe371051640 1 --2- 172.21.15.107:0/1479475670 >> v2:172.21.15.110:3300/0 conn(0x7fe36c152b30 0x7fe36c152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:55:04.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36bfff640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 0x7fe35c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:04.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe370850640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe35c003980 0x7fe35c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:04.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe371051640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe35c005190 0x7fe35c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:04.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36bfff640 1 -- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe35c005190 msgr2=0x7fe35c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:04.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36bfff640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe35c005190 0x7fe35c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:04.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36bfff640 1 -- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe35c003980 msgr2=0x7fe35c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:04.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36bfff640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe35c003980 0x7fe35c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:04.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe36bfff640 1 -- 172.21.15.107:0/1479475670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c003530 con 0x7fe35c0047d0 2023-10-24T23:55:04.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.807+0000 7fe371051640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe35c005190 0x7fe35c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:55:04.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe36bfff640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 0x7fe35c004c50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe360009b20 tx=0x7fe36001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced5523972c095c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:04.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe360004030 con 0x7fe35c0047d0 2023-10-24T23:55:04.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe360004190 con 0x7fe35c0047d0 2023-10-24T23:55:04.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe36b7fe640 1 -- 172.21.15.107:0/1479475670 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe360004520 con 0x7fe35c0047d0 2023-10-24T23:55:04.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe372ab4640 1 -- 172.21.15.107:0/1479475670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 msgr2=0x7fe35c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:04.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe372ab4640 1 --2- 172.21.15.107:0/1479475670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 0x7fe35c004c50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe360009b20 tx=0x7fe36001ca90 comp rx=0 tx=0).stop 2023-10-24T23:55:04.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe371051640 1 -- 172.21.15.107:0/1479475670 reap_dead start 2023-10-24T23:55:04.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe372ab4640 1 -- 172.21.15.107:0/1479475670 shutdown_connections 2023-10-24T23:55:04.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe372ab4640 1 -- 172.21.15.107:0/1479475670 >> 172.21.15.107:0/1479475670 conn(0x7fe36c070f60 msgr2=0x7fe36c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe372ab4640 1 -- 172.21.15.107:0/1479475670 shutdown_connections 2023-10-24T23:55:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe372ab4640 1 -- 172.21.15.107:0/1479475670 wait complete. 2023-10-24T23:55:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe372ab4640 1 Processor -- start 2023-10-24T23:55:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.811+0000 7fe372ab4640 1 -- start start 2023-10-24T23:55:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe372ab4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 0x7fe36c1587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe372ab4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe36c0c1c10 0x7fe36c1e9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe372ab4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe36c06ae50 0x7fe36c1e9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe372ab4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0ab6a0 con 0x7fe35c0047d0 2023-10-24T23:55:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe372ab4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c155f20 con 0x7fe36c0c1c10 2023-10-24T23:55:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe372ab4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c0acad0 con 0x7fe36c06ae50 2023-10-24T23:55:04.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe370850640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 0x7fe36c1587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:04.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe370850640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 0x7fe36c1587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40920/0 (socket says 172.21.15.107:40920) 2023-10-24T23:55:04.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe370850640 1 -- 172.21.15.107:0/2144056511 learned_addr learned my addr 172.21.15.107:0/2144056511 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:04.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe371051640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe36c06ae50 0x7fe36c1e9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:04.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe36bfff640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe36c0c1c10 0x7fe36c1e9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:04.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe370850640 1 -- 172.21.15.107:0/2144056511 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe36c06ae50 msgr2=0x7fe36c1e9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:04.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe370850640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe36c06ae50 0x7fe36c1e9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:04.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe370850640 1 -- 172.21.15.107:0/2144056511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe36c0c1c10 msgr2=0x7fe36c1e9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:04.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe370850640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe36c0c1c10 0x7fe36c1e9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:04.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe370850640 1 -- 172.21.15.107:0/2144056511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c1efc00 con 0x7fe35c0047d0 2023-10-24T23:55:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe370850640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 0x7fe36c1587a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe358004f80 tx=0x7fe3580074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe3697fa640 1 -- 172.21.15.107:0/2144056511 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358007740 con 0x7fe35c0047d0 2023-10-24T23:55:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe3697fa640 1 -- 172.21.15.107:0/2144056511 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35800f960 con 0x7fe35c0047d0 2023-10-24T23:55:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe372ab4640 1 -- 172.21.15.107:0/2144056511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c1efee0 con 0x7fe35c0047d0 2023-10-24T23:55:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe3697fa640 1 -- 172.21.15.107:0/2144056511 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35800fd10 con 0x7fe35c0047d0 2023-10-24T23:55:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe372ab4640 1 -- 172.21.15.107:0/2144056511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c1f04a0 con 0x7fe35c0047d0 2023-10-24T23:55:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe372ab4640 1 -- 172.21.15.107:0/2144056511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c1ec6e0 con 0x7fe35c0047d0 2023-10-24T23:55:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe3697fa640 1 -- 172.21.15.107:0/2144056511 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe35801d460 con 0x7fe35c0047d0 2023-10-24T23:55:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe3697fa640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe350065e90 0x7fe350068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe3697fa640 1 -- 172.21.15.107:0/2144056511 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7fe358083ac0 con 0x7fe35c0047d0 2023-10-24T23:55:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.815+0000 7fe36bfff640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe350065e90 0x7fe350068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:04.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.819+0000 7fe36bfff640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe350065e90 0x7fe350068350 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe3600aa000 tx=0x7fe3600046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:04.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:04.823+0000 7fe3697fa640 1 -- 172.21.15.107:0/2144056511 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe358085e60 con 0x7fe35c0047d0 2023-10-24T23:55:04.856 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gettimeofday... yes 2023-10-24T23:55:04.868 INFO:tasks.workunit.client.1.smithi110.stdout:checking dependency style of gcc... gcc3 2023-10-24T23:55:04.904 INFO:tasks.workunit.client.1.smithi110.stdout:checking for stdio.h... yes 2023-10-24T23:55:04.930 INFO:tasks.workunit.client.0.smithi107.stdout:checking for mkdir... yes 2023-10-24T23:55:04.954 INFO:tasks.workunit.client.1.smithi110.stdout:checking for stdlib.h... yes 2023-10-24T23:55:04.997 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strerror... yes 2023-10-24T23:55:05.001 INFO:tasks.workunit.client.1.smithi110.stdout:checking for string.h... yes 2023-10-24T23:55:05.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.039+0000 7fe372ab4640 1 -- 172.21.15.107:0/2144056511 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe36c06d440 con 0x7fe35c0047d0 2023-10-24T23:55:05.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe3697fa640 1 -- 172.21.15.107:0/2144056511 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fe358050dd0 con 0x7fe35c0047d0 2023-10-24T23:55:05.047 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:05.047 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:16.358716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:05.047 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-10-24T23:55:05.048 INFO:tasks.workunit.client.1.smithi110.stdout:checking for inttypes.h... yes 2023-10-24T23:55:05.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 -- 172.21.15.107:0/2144056511 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe350065e90 msgr2=0x7fe350068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:05.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe350065e90 0x7fe350068350 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe3600aa000 tx=0x7fe3600046d0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 -- 172.21.15.107:0/2144056511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 msgr2=0x7fe36c1587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:05.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 0x7fe36c1587a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe358004f80 tx=0x7fe3580074f0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 -- 172.21.15.107:0/2144056511 shutdown_connections 2023-10-24T23:55:05.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe350065e90 0x7fe350068350 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe36c06ae50 0x7fe36c1e9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe36c0c1c10 0x7fe36c1e9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 --2- 172.21.15.107:0/2144056511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c0047d0 0x7fe36c1587a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 -- 172.21.15.107:0/2144056511 >> 172.21.15.107:0/2144056511 conn(0x7fe36c070f60 msgr2=0x7fe36c0c20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:05.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 -- 172.21.15.107:0/2144056511 shutdown_connections 2023-10-24T23:55:05.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.043+0000 7fe372ab4640 1 -- 172.21.15.107:0/2144056511 wait complete. 2023-10-24T23:55:05.067 INFO:tasks.workunit.client.0.smithi107.stdout:checking for open64... yes 2023-10-24T23:55:05.068 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2023-10-24T23:55:05.068 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-24T23:55:05.068 INFO:teuthology.orchestra.run:waiting for 300 2023-10-24T23:55:05.095 INFO:tasks.workunit.client.1.smithi110.stdout:checking for stdint.h... yes 2023-10-24T23:55:05.124 INFO:tasks.ceph.mds.c:Stopped 2023-10-24T23:55:05.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:05.131 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stat64... yes 2023-10-24T23:55:05.200 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fseeko64... yes 2023-10-24T23:55:05.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5daffe1640 1 Processor -- start 2023-10-24T23:55:05.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5daffe1640 1 -- start start 2023-10-24T23:55:05.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5daffe1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5da813dc20 0x7f5da80c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:05.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5daffe1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5da8157e50 0x7f5da81582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:05.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5daffe1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da8154860 con 0x7f5da80720c0 2023-10-24T23:55:05.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5daffe1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da8158800 con 0x7f5da8157e50 2023-10-24T23:55:05.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5daffe1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da80c1f10 con 0x7f5da813dc20 2023-10-24T23:55:05.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dad57c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5da80720c0 0x7f5da806df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36994/0 (socket says 172.21.15.107:36994) 2023-10-24T23:55:05.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dad57c640 1 -- 172.21.15.107:0/2323906205 learned_addr learned my addr 172.21.15.107:0/2323906205 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:05.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682146983 0 0) 0x7f5da8154860 con 0x7f5da80720c0 2023-10-24T23:55:05.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d94003530 con 0x7f5da80720c0 2023-10-24T23:55:05.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dae57e640 1 --2- 172.21.15.107:0/2323906205 >> v2:172.21.15.110:3300/0 conn(0x7f5da8157e50 0x7f5da81582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:05.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dadd7d640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5da813dc20 0x7f5da80c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:05.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3712431170 0 0) 0x7f5d94003530 con 0x7f5da80720c0 2023-10-24T23:55:05.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5da813dc20 msgr2=0x7f5da80c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:05.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5da813dc20 0x7f5da80c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 >> v2:172.21.15.110:3300/0 conn(0x7f5da8157e50 msgr2=0x7f5da81582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:05.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 --2- 172.21.15.107:0/2323906205 >> v2:172.21.15.110:3300/0 conn(0x7f5da8157e50 0x7f5da81582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da8154b20 con 0x7f5da80720c0 2023-10-24T23:55:05.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5d98001680 con 0x7f5da80720c0 2023-10-24T23:55:05.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dae57e640 1 --2- 172.21.15.107:0/2323906205 >> v2:172.21.15.110:3300/0 conn(0x7f5da8157e50 0x7f5da81582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:55:05.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 >> v1:172.21.15.107:6789/0 conn(0x7f5da80720c0 legacy=0x7f5da806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:05.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d94003980 0x7f5d94003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:05.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5da813dc20 0x7f5d940047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:05.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d94004d10 0x7f5d94005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:05.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da80c1f10 con 0x7f5d94003980 2023-10-24T23:55:05.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9400bb20 con 0x7f5d94004d10 2023-10-24T23:55:05.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.235+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9400bce0 con 0x7f5da813dc20 2023-10-24T23:55:05.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dadd7d640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d94003980 0x7f5d94003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:05.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dad57c640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5da813dc20 0x7f5d940047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:05.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dae57e640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d94004d10 0x7f5d94005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dadd7d640 1 -- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5da813dc20 msgr2=0x7f5d940047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dadd7d640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5da813dc20 0x7f5d940047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dadd7d640 1 -- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d94004d10 msgr2=0x7f5d94005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dadd7d640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d94004d10 0x7f5d94005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dadd7d640 1 -- 172.21.15.107:0/2323906205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d94003530 con 0x7f5d94003980 2023-10-24T23:55:05.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dae57e640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d94004d10 0x7f5d94005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dadd7d640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d94003980 0x7f5d94003da0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5da400ba30 tx=0x7f5da401c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab90989695e3dc2d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da40ae410 con 0x7f5d94003980 2023-10-24T23:55:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da40a8030 con 0x7f5d94003980 2023-10-24T23:55:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dacd7b640 1 -- 172.21.15.107:0/2323906205 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da40ae7b0 con 0x7f5d94003980 2023-10-24T23:55:05.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5daffe1640 1 -- 172.21.15.107:0/2323906205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d94003980 msgr2=0x7f5d94003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:05.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5daffe1640 1 --2- 172.21.15.107:0/2323906205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d94003980 0x7f5d94003da0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5da400ba30 tx=0x7f5da401c680 comp rx=0 tx=0).stop 2023-10-24T23:55:05.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5dae57e640 1 -- 172.21.15.107:0/2323906205 reap_dead start 2023-10-24T23:55:05.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5daffe1640 1 -- 172.21.15.107:0/2323906205 shutdown_connections 2023-10-24T23:55:05.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5daffe1640 1 -- 172.21.15.107:0/2323906205 >> 172.21.15.107:0/2323906205 conn(0x7f5da80c39a0 msgr2=0x7f5da80c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:05.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5daffe1640 1 -- 172.21.15.107:0/2323906205 shutdown_connections 2023-10-24T23:55:05.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.239+0000 7f5daffe1640 1 -- 172.21.15.107:0/2323906205 wait complete. 2023-10-24T23:55:05.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5daffe1640 1 Processor -- start 2023-10-24T23:55:05.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5daffe1640 1 -- start start 2023-10-24T23:55:05.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5daffe1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d94003980 0x7f5da81537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5daffe1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5da806c630 0x7f5da813a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5daffe1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da813ac30 0x7f5da81f0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5daffe1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da80aa6b0 con 0x7f5da813ac30 2023-10-24T23:55:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5daffe1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da81f0000 con 0x7f5da806c630 2023-10-24T23:55:05.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5daffe1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da813b0e0 con 0x7f5d94003980 2023-10-24T23:55:05.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dae57e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da813ac30 0x7f5da81f0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:05.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dae57e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da813ac30 0x7f5da81f0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40944/0 (socket says 172.21.15.107:40944) 2023-10-24T23:55:05.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dae57e640 1 -- 172.21.15.107:0/1054037991 learned_addr learned my addr 172.21.15.107:0/1054037991 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:05.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dadd7d640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d94003980 0x7f5da81537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:05.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dad57c640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5da806c630 0x7f5da813a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dae57e640 1 -- 172.21.15.107:0/1054037991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d94003980 msgr2=0x7f5da81537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dae57e640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d94003980 0x7f5da81537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dae57e640 1 -- 172.21.15.107:0/1054037991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5da806c630 msgr2=0x7f5da813a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dae57e640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5da806c630 0x7f5da813a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dae57e640 1 -- 172.21.15.107:0/1054037991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da81534e0 con 0x7f5da813ac30 2023-10-24T23:55:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dadd7d640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d94003980 0x7f5da81537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:55:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5dae57e640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da813ac30 0x7f5da81f0490 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5da80c05a0 tx=0x7f5da0007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5d9e7fc640 1 -- 172.21.15.107:0/1054037991 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da00048a0 con 0x7f5da813ac30 2023-10-24T23:55:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5d9e7fc640 1 -- 172.21.15.107:0/1054037991 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da0014400 con 0x7f5da813ac30 2023-10-24T23:55:05.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5daffe1640 1 -- 172.21.15.107:0/1054037991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da80a97e0 con 0x7f5da813ac30 2023-10-24T23:55:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5d9e7fc640 1 -- 172.21.15.107:0/1054037991 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da0014790 con 0x7f5da813ac30 2023-10-24T23:55:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.243+0000 7f5daffe1640 1 -- 172.21.15.107:0/1054037991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da81eca90 con 0x7f5da813ac30 2023-10-24T23:55:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.247+0000 7f5daffe1640 1 -- 172.21.15.107:0/1054037991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da80a9c90 con 0x7f5da813ac30 2023-10-24T23:55:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.247+0000 7f5d9e7fc640 1 -- 172.21.15.107:0/1054037991 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5da001d3b0 con 0x7f5da813ac30 2023-10-24T23:55:05.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.247+0000 7f5d9e7fc640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5d8c065e90 0x7f5d8c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.247+0000 7f5dadd7d640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5d8c065e90 0x7f5d8c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.247+0000 7f5d9e7fc640 1 -- 172.21.15.107:0/1054037991 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7f5da00837d0 con 0x7f5da813ac30 2023-10-24T23:55:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.247+0000 7f5dadd7d640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5d8c065e90 0x7f5d8c068350 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5da40b0000 tx=0x7f5da40046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:05.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.251+0000 7f5d9e7fc640 1 -- 172.21.15.107:0/1054037991 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5da0050ad0 con 0x7f5da813ac30 2023-10-24T23:55:05.277 INFO:tasks.workunit.client.0.smithi107.stdout:checking for lrand48_r... yes 2023-10-24T23:55:05.354 INFO:tasks.workunit.client.0.smithi107.stdout:checking for srand48_r... yes 2023-10-24T23:55:05.377 INFO:tasks.workunit.client.0.smithi107.stdout:configure: creating ./config.status 2023-10-24T23:55:05.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.451+0000 7f5daffe1640 1 -- 172.21.15.107:0/1054037991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5da81f0170 con 0x7f5da813ac30 2023-10-24T23:55:05.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.451+0000 7f5d9e7fc640 1 -- 172.21.15.107:0/1054037991 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f5da00540f0 con 0x7f5da813ac30 2023-10-24T23:55:05.458 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:05.458 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:16.358716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-10-24T23:55:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 -- 172.21.15.107:0/1054037991 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5d8c065e90 msgr2=0x7f5d8c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5d8c065e90 0x7f5d8c068350 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5da40b0000 tx=0x7f5da40046d0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 -- 172.21.15.107:0/1054037991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da813ac30 msgr2=0x7f5da81f0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da813ac30 0x7f5da81f0490 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5da80c05a0 tx=0x7f5da0007700 comp rx=0 tx=0).stop 2023-10-24T23:55:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 -- 172.21.15.107:0/1054037991 shutdown_connections 2023-10-24T23:55:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5da813ac30 0x7f5da81f0490 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5da806c630 0x7f5da813a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5d8c065e90 0x7f5d8c068350 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 --2- 172.21.15.107:0/1054037991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d94003980 0x7f5da81537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 -- 172.21.15.107:0/1054037991 >> 172.21.15.107:0/1054037991 conn(0x7f5da80c39a0 msgr2=0x7f5da813be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:05.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 -- 172.21.15.107:0/1054037991 shutdown_connections 2023-10-24T23:55:05.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:05.455+0000 7f5daffe1640 1 -- 172.21.15.107:0/1054037991 wait complete. 2023-10-24T23:55:05.476 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:55:05.514 INFO:tasks.workunit.client.0.smithi107.stdout: /bin/bash ./config.status 2023-10-24T23:55:05.598 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating Makefile 2023-10-24T23:55:05.618 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating config.h 2023-10-24T23:55:05.659 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing depfiles commands 2023-10-24T23:55:05.677 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: executing default commands 2023-10-24T23:55:05.778 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2023-10-24T23:55:05.834 INFO:tasks.workunit.client.0.smithi107.stderr:autoheader: warning: autoconf input should be named 'configure.ac', not 'configure.in' 2023-10-24T23:55:05.914 INFO:tasks.workunit.client.0.smithi107.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2023-10-24T23:55:05.915 INFO:tasks.workunit.client.0.smithi107.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2023-10-24T23:55:05.915 INFO:tasks.workunit.client.0.smithi107.stderr:If you have problems, you may need to regenerate the build system entirely. 2023-10-24T23:55:05.915 INFO:tasks.workunit.client.0.smithi107.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2023-10-24T23:55:06.067 INFO:tasks.workunit.client.0.smithi107.stdout:rm -f stamp-h1 2023-10-24T23:55:06.074 INFO:tasks.workunit.client.0.smithi107.stdout:touch config.h.in 2023-10-24T23:55:06.081 INFO:tasks.workunit.client.0.smithi107.stdout:cd . && /bin/bash ./config.status config.h 2023-10-24T23:55:06.160 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating config.h 2023-10-24T23:55:06.215 INFO:tasks.workunit.client.0.smithi107.stdout:make all-am 2023-10-24T23:55:06.515 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2023-10-24T23:55:06.516 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2023-10-24T23:55:06.779 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2023-10-24T23:55:06.792 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2023-10-24T23:55:06.935 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2023-10-24T23:55:06.948 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2023-10-24T23:55:07.113 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/main.Tpo .deps/main.Po 2023-10-24T23:55:07.127 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2023-10-24T23:55:07.325 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2023-10-24T23:55:07.341 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2023-10-24T23:55:07.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:07.542 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2023-10-24T23:55:07.556 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2023-10-24T23:55:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.591+0000 7fc941ee8640 1 Processor -- start 2023-10-24T23:55:07.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.591+0000 7fc941ee8640 1 -- start start 2023-10-24T23:55:07.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.591+0000 7fc941ee8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc93c0c4e20 0x7fc93c0c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:07.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.591+0000 7fc941ee8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc93c0c5ba0 0x7fc93c0c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:07.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.591+0000 7fc941ee8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc93c155f20 con 0x7fc93c14fbd0 2023-10-24T23:55:07.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.591+0000 7fc941ee8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93c0c6d40 con 0x7fc93c0c4e20 2023-10-24T23:55:07.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.591+0000 7fc941ee8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93c0c2320 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.591+0000 7fc93b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc93c14fbd0 0x7fc93c06df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36998/0 (socket says 172.21.15.107:36998) 2023-10-24T23:55:07.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.591+0000 7fc93b7fe640 1 -- 172.21.15.107:0/1786579035 learned_addr learned my addr 172.21.15.107:0/1786579035 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:07.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.591+0000 7fc932ffd640 1 --2- 172.21.15.107:0/1786579035 >> v2:172.21.15.107:3300/0 conn(0x7fc93c0c4e20 0x7fc93c0c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:07.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc93bfff640 1 --2- 172.21.15.107:0/1786579035 >> v2:172.21.15.110:3300/0 conn(0x7fc93c0c5ba0 0x7fc93c0c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:07.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc93affd640 1 -- 172.21.15.107:0/1786579035 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549704947 0 0) 0x7fc93c155f20 con 0x7fc93c14fbd0 2023-10-24T23:55:07.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc93affd640 1 -- 172.21.15.107:0/1786579035 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc928003530 con 0x7fc93c14fbd0 2023-10-24T23:55:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc932ffd640 1 -- 172.21.15.107:0/1786579035 >> v2:172.21.15.110:3300/0 conn(0x7fc93c0c5ba0 msgr2=0x7fc93c0c1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc932ffd640 1 --2- 172.21.15.107:0/1786579035 >> v2:172.21.15.110:3300/0 conn(0x7fc93c0c5ba0 0x7fc93c0c1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc932ffd640 1 -- 172.21.15.107:0/1786579035 >> v1:172.21.15.107:6789/0 conn(0x7fc93c14fbd0 legacy=0x7fc93c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:07.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc932ffd640 1 -- 172.21.15.107:0/1786579035 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c153560 con 0x7fc93c0c4e20 2023-10-24T23:55:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc932ffd640 1 --2- 172.21.15.107:0/1786579035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc93c0c4e20 0x7fc93c0c18a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0c4990 tx=0x7fc92401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f94f77bbbee85b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc93affd640 1 -- 172.21.15.107:0/1786579035 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9240ab600 con 0x7fc93c0c4e20 2023-10-24T23:55:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc93affd640 1 -- 172.21.15.107:0/1786579035 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9240a7050 con 0x7fc93c0c4e20 2023-10-24T23:55:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc93affd640 1 -- 172.21.15.107:0/1786579035 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9240ab980 con 0x7fc93c0c4e20 2023-10-24T23:55:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc941ee8640 1 -- 172.21.15.107:0/1786579035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc93c0c4e20 msgr2=0x7fc93c0c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc941ee8640 1 --2- 172.21.15.107:0/1786579035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc93c0c4e20 0x7fc93c0c18a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0c4990 tx=0x7fc92401c3d0 comp rx=0 tx=0).stop 2023-10-24T23:55:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc941ee8640 1 -- 172.21.15.107:0/1786579035 shutdown_connections 2023-10-24T23:55:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc941ee8640 1 --2- 172.21.15.107:0/1786579035 >> v2:172.21.15.110:3300/0 conn(0x7fc93c0c5ba0 0x7fc93c0c1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc941ee8640 1 --2- 172.21.15.107:0/1786579035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc93c0c4e20 0x7fc93c0c18a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc941ee8640 1 -- 172.21.15.107:0/1786579035 >> 172.21.15.107:0/1786579035 conn(0x7fc93c0c39a0 msgr2=0x7fc93c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc941ee8640 1 -- 172.21.15.107:0/1786579035 shutdown_connections 2023-10-24T23:55:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.595+0000 7fc941ee8640 1 -- 172.21.15.107:0/1786579035 wait complete. 2023-10-24T23:55:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc941ee8640 1 Processor -- start 2023-10-24T23:55:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc941ee8640 1 -- start start 2023-10-24T23:55:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc941ee8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc93c0c4e20 0x7fc93c154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc941ee8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc93c0c5ba0 0x7fc93c1480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc941ee8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc93c14fbd0 0x7fc93c150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc941ee8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c151960 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc941ee8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c147bb0 con 0x7fc93c0c4e20 2023-10-24T23:55:07.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc941ee8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c147600 con 0x7fc93c14fbd0 2023-10-24T23:55:07.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc932ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc93c0c5ba0 0x7fc93c1480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:07.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc932ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc93c0c5ba0 0x7fc93c1480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40964/0 (socket says 172.21.15.107:40964) 2023-10-24T23:55:07.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc932ffd640 1 -- 172.21.15.107:0/2316768260 learned_addr learned my addr 172.21.15.107:0/2316768260 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:07.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc93b7fe640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc93c0c4e20 0x7fc93c154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc93bfff640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc93c14fbd0 0x7fc93c150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc932ffd640 1 -- 172.21.15.107:0/2316768260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc93c14fbd0 msgr2=0x7fc93c150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc932ffd640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc93c14fbd0 0x7fc93c150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc932ffd640 1 -- 172.21.15.107:0/2316768260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc93c0c4e20 msgr2=0x7fc93c154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc932ffd640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc93c0c4e20 0x7fc93c154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc932ffd640 1 -- 172.21.15.107:0/2316768260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc924009420 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc93bfff640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc93c14fbd0 0x7fc93c150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc932ffd640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc93c0c5ba0 0x7fc93c1480f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc9240ae040 tx=0x7fc924019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc938ff9640 1 -- 172.21.15.107:0/2316768260 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9240b4410 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc938ff9640 1 -- 172.21.15.107:0/2316768260 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9240b4570 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc941ee8640 1 -- 172.21.15.107:0/2316768260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93c1478e0 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc938ff9640 1 -- 172.21.15.107:0/2316768260 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9240b4920 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc941ee8640 1 -- 172.21.15.107:0/2316768260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93c151ea0 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.599+0000 7fc941ee8640 1 -- 172.21.15.107:0/2316768260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93c0746e0 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.603+0000 7fc938ff9640 1 -- 172.21.15.107:0/2316768260 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc9240b2040 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.603+0000 7fc938ff9640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc91c065e60 0x7fc91c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.603+0000 7fc93b7fe640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc91c065e60 0x7fc91c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.603+0000 7fc938ff9640 1 -- 172.21.15.107:0/2316768260 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7fc92411dbb0 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.603+0000 7fc93b7fe640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc91c065e60 0x7fc91c068320 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc92c0011f0 tx=0x7fc92c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.607+0000 7fc938ff9640 1 -- 172.21.15.107:0/2316768260 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc9240eaeb0 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.661 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2023-10-24T23:55:07.672 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2023-10-24T23:55:07.735 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2023-10-24T23:55:07.748 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2023-10-24T23:55:07.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.819+0000 7fc941ee8640 1 -- 172.21.15.107:0/2316768260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc93c0c5980 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.819+0000 7fc938ff9640 1 -- 172.21.15.107:0/2316768260 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fc9240ee4d0 con 0x7fc93c0c5ba0 2023-10-24T23:55:07.826 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:07.826 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:16.358716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:07.827 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-10-24T23:55:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 -- 172.21.15.107:0/2316768260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc91c065e60 msgr2=0x7fc91c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc91c065e60 0x7fc91c068320 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc92c0011f0 tx=0x7fc92c009040 comp rx=0 tx=0).stop 2023-10-24T23:55:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 -- 172.21.15.107:0/2316768260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc93c0c5ba0 msgr2=0x7fc93c1480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:07.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc93c0c5ba0 0x7fc93c1480f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc9240ae040 tx=0x7fc924019170 comp rx=0 tx=0).stop 2023-10-24T23:55:07.831 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2023-10-24T23:55:07.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 -- 172.21.15.107:0/2316768260 shutdown_connections 2023-10-24T23:55:07.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc91c065e60 0x7fc91c068320 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:07.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc93c14fbd0 0x7fc93c150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:07.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc93c0c5ba0 0x7fc93c1480f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:07.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 --2- 172.21.15.107:0/2316768260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc93c0c4e20 0x7fc93c154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:07.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 -- 172.21.15.107:0/2316768260 >> 172.21.15.107:0/2316768260 conn(0x7fc93c0c39a0 msgr2=0x7fc93c0c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:07.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 -- 172.21.15.107:0/2316768260 shutdown_connections 2023-10-24T23:55:07.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:07.827+0000 7fc941ee8640 1 -- 172.21.15.107:0/2316768260 wait complete. 2023-10-24T23:55:07.842 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2023-10-24T23:55:07.850 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:55:08.086 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2023-10-24T23:55:08.098 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2023-10-24T23:55:08.288 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2023-10-24T23:55:08.300 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2023-10-24T23:55:08.559 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2023-10-24T23:55:08.571 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2023-10-24T23:55:08.690 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2023-10-24T23:55:08.704 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2023-10-24T23:55:08.778 INFO:tasks.workunit.client.0.smithi107.stderr:ffsb_op.c: In function 'print_op_results': 2023-10-24T23:55:08.778 INFO:tasks.workunit.client.0.smithi107.stderr:ffsb_op.c:83:37: warning: embedded '\0' in format [-Wformat-contains-nul] 2023-10-24T23:55:08.778 INFO:tasks.workunit.client.0.smithi107.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2023-10-24T23:55:08.779 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~ 2023-10-24T23:55:08.779 INFO:tasks.workunit.client.0.smithi107.stderr:ffsb_op.c:86:33: warning: embedded '\0' in format [-Wformat-contains-nul] 2023-10-24T23:55:08.779 INFO:tasks.workunit.client.0.smithi107.stderr: 86 | sprintf(buf, "NA\0"); 2023-10-24T23:55:08.779 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~ 2023-10-24T23:55:08.799 INFO:tasks.workunit.client.0.smithi107.stderr:ffsb_op.c: In function 'print_results': 2023-10-24T23:55:08.799 INFO:tasks.workunit.client.0.smithi107.stderr:ffsb_op.c:83:33: warning: '/sec' directive writing 4 bytes into a region of size between 1 and 256 [-Wformat-overflow=] 2023-10-24T23:55:08.799 INFO:tasks.workunit.client.0.smithi107.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2023-10-24T23:55:08.799 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~ 2023-10-24T23:55:08.800 INFO:tasks.workunit.client.0.smithi107.stderr:In file included from /usr/include/stdio.h:894, 2023-10-24T23:55:08.800 INFO:tasks.workunit.client.0.smithi107.stderr: from ffsb_op.c:20: 2023-10-24T23:55:08.800 INFO:tasks.workunit.client.0.smithi107.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 5 and 260 bytes into a destination of size 256 2023-10-24T23:55:08.800 INFO:tasks.workunit.client.0.smithi107.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, 2023-10-24T23:55:08.801 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:08.801 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | __glibc_objsize (__s), __fmt, 2023-10-24T23:55:08.801 INFO:tasks.workunit.client.0.smithi107.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:08.801 INFO:tasks.workunit.client.0.smithi107.stderr: 40 | __va_arg_pack ()); 2023-10-24T23:55:08.802 INFO:tasks.workunit.client.0.smithi107.stderr: | ~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:08.855 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2023-10-24T23:55:08.870 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2023-10-24T23:55:09.066 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/util.Tpo .deps/util.Po 2023-10-24T23:55:09.078 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2023-10-24T23:55:09.160 INFO:tasks.workunit.client.0.smithi107.stderr:In file included from parser.h:22, 2023-10-24T23:55:09.160 INFO:tasks.workunit.client.0.smithi107.stderr: from parser.c:25: 2023-10-24T23:55:09.161 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c: In function 'init_filesys': 2023-10-24T23:55:09.163 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:09.163 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:09.164 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-10-24T23:55:09.164 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:09.164 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:09.165 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-10-24T23:55:09.165 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:09.166 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:09.166 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:09.166 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:09.167 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-10-24T23:55:09.167 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:09.168 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:09.168 INFO:tasks.workunit.client.0.smithi107.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2023-10-24T23:55:09.168 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:09.169 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:09.169 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:09.169 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-10-24T23:55:09.170 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:09.171 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:09.171 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-10-24T23:55:09.171 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:09.172 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:09.172 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:09.173 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:09.173 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-10-24T23:55:09.173 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:09.174 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:09.174 INFO:tasks.workunit.client.0.smithi107.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2023-10-24T23:55:09.174 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:09.175 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:09.175 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:09.176 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-10-24T23:55:09.176 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:09.176 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:09.177 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-10-24T23:55:09.177 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:09.177 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:09.178 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:09.178 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:09.179 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-10-24T23:55:09.179 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:09.179 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:09.180 INFO:tasks.workunit.client.0.smithi107.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2023-10-24T23:55:09.180 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:09.181 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:09.181 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:09.181 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-10-24T23:55:09.182 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:09.182 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:09.182 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-10-24T23:55:09.183 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:09.183 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:09.184 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:09.184 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:09.184 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-10-24T23:55:09.185 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:09.185 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:09.186 INFO:tasks.workunit.client.0.smithi107.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2023-10-24T23:55:09.186 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:09.186 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c: In function 'init_tg_stats': 2023-10-24T23:55:09.187 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:09.187 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:09.187 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-10-24T23:55:09.188 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:09.188 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:09.189 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-10-24T23:55:09.189 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:09.189 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:09.190 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:09.190 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:09.191 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-10-24T23:55:09.191 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:09.191 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:09.192 INFO:tasks.workunit.client.0.smithi107.stderr: 971 | list_for_each_entry(tmp_list, 2023-10-24T23:55:09.192 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:09.193 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:09.193 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:09.193 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-10-24T23:55:09.194 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:09.194 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:09.194 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-10-24T23:55:09.195 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:09.195 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:09.196 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:09.196 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:09.196 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-10-24T23:55:09.197 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:09.197 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:09.198 INFO:tasks.workunit.client.0.smithi107.stderr: 971 | list_for_each_entry(tmp_list, 2023-10-24T23:55:09.198 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:09.198 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:09.199 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:09.199 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-10-24T23:55:09.200 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:09.200 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:09.200 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-10-24T23:55:09.201 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:09.201 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:09.201 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:09.202 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:09.202 INFO:tasks.workunit.client.0.smithi107.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-10-24T23:55:09.203 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:09.203 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:09.203 INFO:tasks.workunit.client.0.smithi107.stderr: 980 | list_for_each_entry(tmp_list, 2023-10-24T23:55:09.204 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:09.204 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:09.204 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:09.205 INFO:tasks.workunit.client.0.smithi107.stderr: | ^ 2023-10-24T23:55:09.205 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:09.205 INFO:tasks.workunit.client.0.smithi107.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:09.206 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~ 2023-10-24T23:55:09.206 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:09.206 INFO:tasks.workunit.client.0.smithi107.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:09.207 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:09.207 INFO:tasks.workunit.client.0.smithi107.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:09.207 INFO:tasks.workunit.client.0.smithi107.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-10-24T23:55:09.207 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:09.208 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:09.208 INFO:tasks.workunit.client.0.smithi107.stderr: 980 | list_for_each_entry(tmp_list, 2023-10-24T23:55:09.208 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:09.230 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c: In function 'get_optstr': 2023-10-24T23:55:09.231 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:206:23: warning: 'strnlen' specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2023-10-24T23:55:09.231 INFO:tasks.workunit.client.0.smithi107.stderr: 206 | len = strnlen(temp, 4095) + 1; 2023-10-24T23:55:09.231 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:09.231 INFO:tasks.workunit.client.0.smithi107.stderr:parser.c:200:14: note: source object declared here 2023-10-24T23:55:09.232 INFO:tasks.workunit.client.0.smithi107.stderr: 200 | char temp[BUFSIZE]; 2023-10-24T23:55:09.232 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~ 2023-10-24T23:55:09.248 INFO:tasks.workunit.client.0.smithi107.stderr:In file included from /usr/include/string.h:535, 2023-10-24T23:55:09.249 INFO:tasks.workunit.client.0.smithi107.stderr: from parser.c:19: 2023-10-24T23:55:09.249 INFO:tasks.workunit.client.0.smithi107.stderr:In function 'memcpy', 2023-10-24T23:55:09.249 INFO:tasks.workunit.client.0.smithi107.stderr: inlined from 'size64_convert' at parser.c:119:2: 2023-10-24T23:55:09.249 INFO:tasks.workunit.client.0.smithi107.stderr:/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10: warning: '__builtin_memcpy' reading 3 bytes from a region of size 2 [-Wstringop-overread] 2023-10-24T23:55:09.249 INFO:tasks.workunit.client.0.smithi107.stderr: 29 | return __builtin___memcpy_chk (__dest, __src, __len, 2023-10-24T23:55:09.250 INFO:tasks.workunit.client.0.smithi107.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:09.250 INFO:tasks.workunit.client.0.smithi107.stderr: 30 | __glibc_objsize0 (__dest)); 2023-10-24T23:55:09.250 INFO:tasks.workunit.client.0.smithi107.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:09.586 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2023-10-24T23:55:09.598 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2023-10-24T23:55:09.727 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2023-10-24T23:55:09.739 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2023-10-24T23:55:09.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:09.882 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2023-10-24T23:55:09.892 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2023-10-24T23:55:09.946 INFO:tasks.workunit.client.0.smithi107.stdout:mv -f .deps/list.Tpo .deps/list.Po 2023-10-24T23:55:09.959 INFO:tasks.workunit.client.0.smithi107.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2023-10-24T23:55:09.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f21110ef640 1 Processor -- start 2023-10-24T23:55:09.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f21110ef640 1 -- start start 2023-10-24T23:55:09.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f21110ef640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f210c0720c0 0x7f210c0c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:09.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f21110ef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f210c157e50 0x7f210c1582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:09.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f21110ef640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210c154860 con 0x7f210c13dc20 2023-10-24T23:55:09.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f21110ef640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210c158800 con 0x7f210c157e50 2023-10-24T23:55:09.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f21110ef640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210c0c1f10 con 0x7f210c0720c0 2023-10-24T23:55:09.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f210ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f210c13dc20 0x7f210c06df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37008/0 (socket says 172.21.15.107:37008) 2023-10-24T23:55:09.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f210b59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f210c157e50 0x7f210c1582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:09.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f210ad9d640 1 -- 172.21.15.107:0/2748817865 learned_addr learned my addr 172.21.15.107:0/2748817865 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:09.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f2103fff640 1 --2- 172.21.15.107:0/2748817865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f210c0720c0 0x7f210c0c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:09.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f210b59e640 1 -- 172.21.15.107:0/2748817865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f210c0720c0 msgr2=0x7f210c0c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:09.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f210b59e640 1 --2- 172.21.15.107:0/2748817865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f210c0720c0 0x7f210c0c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:09.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f210b59e640 1 -- 172.21.15.107:0/2748817865 >> v1:172.21.15.107:6789/0 conn(0x7f210c13dc20 legacy=0x7f210c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:09.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f210b59e640 1 -- 172.21.15.107:0/2748817865 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210c154b20 con 0x7f210c157e50 2023-10-24T23:55:09.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.975+0000 7f210a59c640 1 -- 172.21.15.107:0/2748817865 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861260902 0 0) 0x7f210c154860 con 0x7f210c13dc20 2023-10-24T23:55:09.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f210b59e640 1 --2- 172.21.15.107:0/2748817865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f210c157e50 0x7f210c1582c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2104009d90 tx=0x7f210401c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3278dd44f02afc04 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:09.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f210a59c640 1 -- 172.21.15.107:0/2748817865 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21040a57e0 con 0x7f210c157e50 2023-10-24T23:55:09.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f210a59c640 1 -- 172.21.15.107:0/2748817865 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21040a5940 con 0x7f210c157e50 2023-10-24T23:55:09.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f210a59c640 1 -- 172.21.15.107:0/2748817865 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21040a5cd0 con 0x7f210c157e50 2023-10-24T23:55:09.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f21110ef640 1 -- 172.21.15.107:0/2748817865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f210c157e50 msgr2=0x7f210c1582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:09.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f21110ef640 1 --2- 172.21.15.107:0/2748817865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f210c157e50 0x7f210c1582c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2104009d90 tx=0x7f210401c830 comp rx=0 tx=0).stop 2023-10-24T23:55:09.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f21110ef640 1 -- 172.21.15.107:0/2748817865 shutdown_connections 2023-10-24T23:55:09.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f21110ef640 1 --2- 172.21.15.107:0/2748817865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f210c157e50 0x7f210c1582c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:09.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f21110ef640 1 --2- 172.21.15.107:0/2748817865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f210c0720c0 0x7f210c0c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:09.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f21110ef640 1 -- 172.21.15.107:0/2748817865 >> 172.21.15.107:0/2748817865 conn(0x7f210c0c39a0 msgr2=0x7f210c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:09.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f21110ef640 1 -- 172.21.15.107:0/2748817865 shutdown_connections 2023-10-24T23:55:09.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f21110ef640 1 -- 172.21.15.107:0/2748817865 wait complete. 2023-10-24T23:55:09.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.979+0000 7f21110ef640 1 Processor -- start 2023-10-24T23:55:09.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21110ef640 1 -- start start 2023-10-24T23:55:09.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21110ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f210c0720c0 0x7f210c13b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:09.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21110ef640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f210c13dc20 0x7f210c153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:09.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21110ef640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f210c157e50 0x7f210c1ec8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:09.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21110ef640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c1f0860 con 0x7f210c0720c0 2023-10-24T23:55:09.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21110ef640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210c1f0d90 con 0x7f210c13dc20 2023-10-24T23:55:09.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21110ef640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210c157100 con 0x7f210c157e50 2023-10-24T23:55:09.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f210ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f210c0720c0 0x7f210c13b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:09.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f210ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f210c0720c0 0x7f210c13b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40982/0 (socket says 172.21.15.107:40982) 2023-10-24T23:55:09.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f210ad9d640 1 -- 172.21.15.107:0/11073328 learned_addr learned my addr 172.21.15.107:0/11073328 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:09.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f210b59e640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f210c157e50 0x7f210c1ec8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:09.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f2103fff640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f210c13dc20 0x7f210c153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:09.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f210ad9d640 1 -- 172.21.15.107:0/11073328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f210c157e50 msgr2=0x7f210c1ec8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:09.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f210ad9d640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f210c157e50 0x7f210c1ec8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:09.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f210ad9d640 1 -- 172.21.15.107:0/11073328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f210c13dc20 msgr2=0x7f210c153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:09.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f210ad9d640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f210c13dc20 0x7f210c153720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:09.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f210ad9d640 1 -- 172.21.15.107:0/11073328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2104009a70 con 0x7f210c0720c0 2023-10-24T23:55:09.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f210b59e640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f210c157e50 0x7f210c1ec8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:09.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f210ad9d640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f210c0720c0 0x7f210c13b740 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00c480 tx=0x7f20fc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:09.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21037fe640 1 -- 172.21.15.107:0/11073328 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc0074e0 con 0x7f210c0720c0 2023-10-24T23:55:09.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21037fe640 1 -- 172.21.15.107:0/11073328 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc007660 con 0x7f210c0720c0 2023-10-24T23:55:09.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21037fe640 1 -- 172.21.15.107:0/11073328 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc004a30 con 0x7f210c0720c0 2023-10-24T23:55:09.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21110ef640 1 -- 172.21.15.107:0/11073328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f210c1503f0 con 0x7f210c0720c0 2023-10-24T23:55:09.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21110ef640 1 -- 172.21.15.107:0/11073328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210c1f0ab0 con 0x7f210c0720c0 2023-10-24T23:55:09.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21110ef640 1 -- 172.21.15.107:0/11073328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210c0746e0 con 0x7f210c0720c0 2023-10-24T23:55:09.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21037fe640 1 -- 172.21.15.107:0/11073328 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f20fc004b90 con 0x7f210c0720c0 2023-10-24T23:55:09.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21037fe640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f20ec065e60 0x7f20ec068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:09.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f21037fe640 1 -- 172.21.15.107:0/11073328 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7f20fc082de0 con 0x7f210c0720c0 2023-10-24T23:55:09.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.983+0000 7f2103fff640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f20ec065e60 0x7f20ec068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:09.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.987+0000 7f2103fff640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f20ec065e60 0x7f20ec068320 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f210c06c500 tx=0x7f20f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:09.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:09.991+0000 7f21037fe640 1 -- 172.21.15.107:0/11073328 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f20fc053b70 con 0x7f210c0720c0 2023-10-24T23:55:10.100 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2023-10-24T23:55:10.108 INFO:tasks.workunit.client.0.smithi107.stdout:FFSB version 6.0-RC2 started 2023-10-24T23:55:10.109 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-24T23:55:10.112 INFO:tasks.workunit.client.0.smithi107.stdout:benchmark time = 300 2023-10-24T23:55:10.112 INFO:tasks.workunit.client.0.smithi107.stdout:ThreadGroup 0 2023-10-24T23:55:10.112 INFO:tasks.workunit.client.0.smithi107.stdout:================ 2023-10-24T23:55:10.113 INFO:tasks.workunit.client.0.smithi107.stdout: num_threads = 32 2023-10-24T23:55:10.113 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-24T23:55:10.113 INFO:tasks.workunit.client.0.smithi107.stdout: read_random = off 2023-10-24T23:55:10.113 INFO:tasks.workunit.client.0.smithi107.stdout: read_size = 0 (0B) 2023-10-24T23:55:10.113 INFO:tasks.workunit.client.0.smithi107.stdout: read_blocksize = 0 (0B) 2023-10-24T23:55:10.114 INFO:tasks.workunit.client.0.smithi107.stdout: read_skip = off 2023-10-24T23:55:10.114 INFO:tasks.workunit.client.0.smithi107.stdout: read_skipsize = 0 (0B) 2023-10-24T23:55:10.114 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-24T23:55:10.114 INFO:tasks.workunit.client.0.smithi107.stdout: write_random = on 2023-10-24T23:55:10.114 INFO:tasks.workunit.client.0.smithi107.stdout: write_size = 5242880 (5MB) 2023-10-24T23:55:10.115 INFO:tasks.workunit.client.0.smithi107.stdout: fsync_file = 0 2023-10-24T23:55:10.115 INFO:tasks.workunit.client.0.smithi107.stdout: write_blocksize = 4096 (4KB) 2023-10-24T23:55:10.115 INFO:tasks.workunit.client.0.smithi107.stdout: wait time = 0 2023-10-24T23:55:10.115 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-24T23:55:10.115 INFO:tasks.workunit.client.0.smithi107.stdout: op weights 2023-10-24T23:55:10.115 INFO:tasks.workunit.client.0.smithi107.stdout: read = 0 (0.00%) 2023-10-24T23:55:10.116 INFO:tasks.workunit.client.0.smithi107.stdout: readall = 0 (0.00%) 2023-10-24T23:55:10.116 INFO:tasks.workunit.client.0.smithi107.stdout: write = 1 (100.00%) 2023-10-24T23:55:10.116 INFO:tasks.workunit.client.0.smithi107.stdout: create = 0 (0.00%) 2023-10-24T23:55:10.116 INFO:tasks.workunit.client.0.smithi107.stdout: append = 0 (0.00%) 2023-10-24T23:55:10.116 INFO:tasks.workunit.client.0.smithi107.stdout: delete = 0 (0.00%) 2023-10-24T23:55:10.117 INFO:tasks.workunit.client.0.smithi107.stdout: metaop = 0 (0.00%) 2023-10-24T23:55:10.117 INFO:tasks.workunit.client.0.smithi107.stdout: createdir = 0 (0.00%) 2023-10-24T23:55:10.117 INFO:tasks.workunit.client.0.smithi107.stdout: stat = 0 (0.00%) 2023-10-24T23:55:10.117 INFO:tasks.workunit.client.0.smithi107.stdout: writeall = 0 (0.00%) 2023-10-24T23:55:10.117 INFO:tasks.workunit.client.0.smithi107.stdout: writeall_fsync = 0 (0.00%) 2023-10-24T23:55:10.118 INFO:tasks.workunit.client.0.smithi107.stdout: open_close = 0 (0.00%) 2023-10-24T23:55:10.118 INFO:tasks.workunit.client.0.smithi107.stdout: write_fsync = 0 (0.00%) 2023-10-24T23:55:10.118 INFO:tasks.workunit.client.0.smithi107.stdout: create_fsync = 0 (0.00%) 2023-10-24T23:55:10.118 INFO:tasks.workunit.client.0.smithi107.stdout: append_fsync = 0 (0.00%) 2023-10-24T23:55:10.118 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-24T23:55:10.118 INFO:tasks.workunit.client.0.smithi107.stdout:FileSystem . 2023-10-24T23:55:10.119 INFO:tasks.workunit.client.0.smithi107.stdout:========== 2023-10-24T23:55:10.119 INFO:tasks.workunit.client.0.smithi107.stdout: num_dirs = 0 2023-10-24T23:55:10.119 INFO:tasks.workunit.client.0.smithi107.stdout: starting files = 128 2023-10-24T23:55:10.119 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-24T23:55:10.119 INFO:tasks.workunit.client.0.smithi107.stdout: min file size = 104857600 (100MB) 2023-10-24T23:55:10.120 INFO:tasks.workunit.client.0.smithi107.stdout: max file size = 104857600 (100MB) 2023-10-24T23:55:10.120 INFO:tasks.workunit.client.0.smithi107.stdout: directio = off 2023-10-24T23:55:10.120 INFO:tasks.workunit.client.0.smithi107.stdout: alignedio = on 2023-10-24T23:55:10.120 INFO:tasks.workunit.client.0.smithi107.stdout: bufferedio = off 2023-10-24T23:55:10.120 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-24T23:55:10.120 INFO:tasks.workunit.client.0.smithi107.stdout: aging is off 2023-10-24T23:55:10.121 INFO:tasks.workunit.client.0.smithi107.stdout: current utilization = 0.00% 2023-10-24T23:55:10.121 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-24T23:55:10.121 INFO:tasks.workunit.client.0.smithi107.stdout:checking existing fs: . 2023-10-24T23:55:10.121 INFO:tasks.workunit.client.0.smithi107.stdout:recreating new fileset 2023-10-24T23:55:10.121 INFO:tasks.workunit.client.0.smithi107.stderr:opendir: No such file or directory 2023-10-24T23:55:10.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.223+0000 7f21110ef640 1 -- 172.21.15.107:0/11073328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f210c06e240 con 0x7f210c0720c0 2023-10-24T23:55:10.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.223+0000 7f21037fe640 1 -- 172.21.15.107:0/11073328 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f20fc018020 con 0x7f210c0720c0 2023-10-24T23:55:10.230 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:10.230 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:16.358716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:10.230 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-10-24T23:55:10.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 -- 172.21.15.107:0/11073328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f20ec065e60 msgr2=0x7f20ec068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:10.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f20ec065e60 0x7f20ec068320 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f210c06c500 tx=0x7f20f4008040 comp rx=0 tx=0).stop 2023-10-24T23:55:10.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 -- 172.21.15.107:0/11073328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f210c0720c0 msgr2=0x7f210c13b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:10.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f210c0720c0 0x7f210c13b740 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00c480 tx=0x7f20fc00c940 comp rx=0 tx=0).stop 2023-10-24T23:55:10.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 -- 172.21.15.107:0/11073328 shutdown_connections 2023-10-24T23:55:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f20ec065e60 0x7f20ec068320 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f210c157e50 0x7f210c1ec8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f210c13dc20 0x7f210c153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 --2- 172.21.15.107:0/11073328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f210c0720c0 0x7f210c13b740 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 -- 172.21.15.107:0/11073328 >> 172.21.15.107:0/11073328 conn(0x7f210c0c39a0 msgr2=0x7f210c15ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:10.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 -- 172.21.15.107:0/11073328 shutdown_connections 2023-10-24T23:55:10.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:10.227+0000 7f21110ef640 1 -- 172.21.15.107:0/11073328 wait complete. 2023-10-24T23:55:10.250 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:55:12.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:12.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.399+0000 7f260e74e640 1 Processor -- start 2023-10-24T23:55:12.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.399+0000 7f260e74e640 1 -- start start 2023-10-24T23:55:12.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f260814fc30 0x7f2608153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:12.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f26080c5c20 0x7f260806df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:12.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2608072330 con 0x7f260806e470 2023-10-24T23:55:12.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2608155ca0 con 0x7f26080c5c20 2023-10-24T23:55:12.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2608155f80 con 0x7f260814fc30 2023-10-24T23:55:12.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f26077fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f26080c5c20 0x7f260806df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:12.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260cceb640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f260806e470 0x7f26080c18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37018/0 (socket says 172.21.15.107:37018) 2023-10-24T23:55:12.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260cceb640 1 -- 172.21.15.107:0/3681148987 learned_addr learned my addr 172.21.15.107:0/3681148987 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:12.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f2607fff640 1 --2- 172.21.15.107:0/3681148987 >> v2:172.21.15.110:3300/0 conn(0x7f260814fc30 0x7f2608153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:12.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f26077fe640 1 -- 172.21.15.107:0/3681148987 >> v2:172.21.15.110:3300/0 conn(0x7f260814fc30 msgr2=0x7f2608153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:12.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f26077fe640 1 --2- 172.21.15.107:0/3681148987 >> v2:172.21.15.110:3300/0 conn(0x7f260814fc30 0x7f2608153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:12.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f26077fe640 1 -- 172.21.15.107:0/3681148987 >> v1:172.21.15.107:6789/0 conn(0x7f260806e470 legacy=0x7f26080c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:12.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f26077fe640 1 -- 172.21.15.107:0/3681148987 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2608156260 con 0x7f26080c5c20 2023-10-24T23:55:12.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f2607fff640 1 --2- 172.21.15.107:0/3681148987 >> v2:172.21.15.110:3300/0 conn(0x7f260814fc30 0x7f2608153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:12.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f26077fe640 1 --2- 172.21.15.107:0/3681148987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26080c5c20 0x7f260806df30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f26080c5060 tx=0x7f25fc01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae45ffb0b215fa78 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:12.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f2606ffd640 1 -- 172.21.15.107:0/3681148987 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc0ac600 con 0x7f26080c5c20 2023-10-24T23:55:12.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f2606ffd640 1 -- 172.21.15.107:0/3681148987 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25fc0a8050 con 0x7f26080c5c20 2023-10-24T23:55:12.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f2606ffd640 1 -- 172.21.15.107:0/3681148987 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc0ac980 con 0x7f26080c5c20 2023-10-24T23:55:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 -- 172.21.15.107:0/3681148987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26080c5c20 msgr2=0x7f260806df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 --2- 172.21.15.107:0/3681148987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26080c5c20 0x7f260806df30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f26080c5060 tx=0x7f25fc01c3a0 comp rx=0 tx=0).stop 2023-10-24T23:55:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 -- 172.21.15.107:0/3681148987 shutdown_connections 2023-10-24T23:55:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 --2- 172.21.15.107:0/3681148987 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26080c5c20 0x7f260806df30 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 --2- 172.21.15.107:0/3681148987 >> v2:172.21.15.110:3300/0 conn(0x7f260814fc30 0x7f2608153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 -- 172.21.15.107:0/3681148987 >> 172.21.15.107:0/3681148987 conn(0x7f26080c39a0 msgr2=0x7f26080c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 -- 172.21.15.107:0/3681148987 shutdown_connections 2023-10-24T23:55:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.403+0000 7f260e74e640 1 -- 172.21.15.107:0/3681148987 wait complete. 2023-10-24T23:55:12.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260e74e640 1 Processor -- start 2023-10-24T23:55:12.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260e74e640 1 -- start start 2023-10-24T23:55:12.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260e74e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f260806e470 0x7f260815d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:12.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260e74e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26080c5c20 0x7f2608069fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:12.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260e74e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260814fc30 0x7f260815c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:12.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260e74e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26081edb20 con 0x7f26080c5c20 2023-10-24T23:55:12.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260e74e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26081ee920 con 0x7f260814fc30 2023-10-24T23:55:12.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260e74e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f260815dd80 con 0x7f260806e470 2023-10-24T23:55:12.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f26077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26080c5c20 0x7f2608069fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:12.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f26077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26080c5c20 0x7f2608069fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40992/0 (socket says 172.21.15.107:40992) 2023-10-24T23:55:12.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f26077fe640 1 -- 172.21.15.107:0/3571554691 learned_addr learned my addr 172.21.15.107:0/3571554691 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:12.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f2607fff640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f260806e470 0x7f260815d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:12.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260cceb640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260814fc30 0x7f260815c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:12.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f26077fe640 1 -- 172.21.15.107:0/3571554691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f260806e470 msgr2=0x7f260815d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:12.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f26077fe640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f260806e470 0x7f260815d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:12.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f26077fe640 1 -- 172.21.15.107:0/3571554691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260814fc30 msgr2=0x7f260815c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:12.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f26077fe640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260814fc30 0x7f260815c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:12.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f26077fe640 1 -- 172.21.15.107:0/3571554691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc009420 con 0x7f26080c5c20 2023-10-24T23:55:12.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f2607fff640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f260806e470 0x7f260815d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:12.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260cceb640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260814fc30 0x7f260815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:12.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f26077fe640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26080c5c20 0x7f2608069fa0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f25fc019fd0 tx=0x7f25fc01abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:12.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f2604ff9640 1 -- 172.21.15.107:0/3571554691 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc0b5830 con 0x7f26080c5c20 2023-10-24T23:55:12.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f2604ff9640 1 -- 172.21.15.107:0/3571554691 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25fc0b5990 con 0x7f26080c5c20 2023-10-24T23:55:12.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260e74e640 1 -- 172.21.15.107:0/3571554691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26081edd10 con 0x7f26080c5c20 2023-10-24T23:55:12.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f2604ff9640 1 -- 172.21.15.107:0/3571554691 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc0b5d00 con 0x7f26080c5c20 2023-10-24T23:55:12.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.407+0000 7f260e74e640 1 -- 172.21.15.107:0/3571554691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26081ee2d0 con 0x7f26080c5c20 2023-10-24T23:55:12.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.411+0000 7f260e74e640 1 -- 172.21.15.107:0/3571554691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26081eea60 con 0x7f26080c5c20 2023-10-24T23:55:12.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.411+0000 7f2604ff9640 1 -- 172.21.15.107:0/3571554691 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f25fc0b3070 con 0x7f26080c5c20 2023-10-24T23:55:12.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.411+0000 7f2604ff9640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25f4065e60 0x7f25f4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:12.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.411+0000 7f2604ff9640 1 -- 172.21.15.107:0/3571554691 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7f25fc11e370 con 0x7f26080c5c20 2023-10-24T23:55:12.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.411+0000 7f2607fff640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25f4065e60 0x7f25f4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:12.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.411+0000 7f2607fff640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25f4065e60 0x7f25f4068320 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f25f8008890 tx=0x7f25f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:12.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.415+0000 7f2604ff9640 1 -- 172.21.15.107:0/3571554691 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f26081eea60 con 0x7f26080c5c20 2023-10-24T23:55:12.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.679+0000 7f260e74e640 1 -- 172.21.15.107:0/3571554691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26081eea60 con 0x7f26080c5c20 2023-10-24T23:55:12.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.679+0000 7f2604ff9640 1 -- 172.21.15.107:0/3571554691 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f26081eea60 con 0x7f26080c5c20 2023-10-24T23:55:12.685 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:12.685 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:16.358716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-10-24T23:55:12.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 -- 172.21.15.107:0/3571554691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25f4065e60 msgr2=0x7f25f4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:12.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25f4065e60 0x7f25f4068320 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f25f8008890 tx=0x7f25f8008040 comp rx=0 tx=0).stop 2023-10-24T23:55:12.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 -- 172.21.15.107:0/3571554691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26080c5c20 msgr2=0x7f2608069fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:12.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26080c5c20 0x7f2608069fa0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f25fc019fd0 tx=0x7f25fc01abe0 comp rx=0 tx=0).stop 2023-10-24T23:55:12.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 -- 172.21.15.107:0/3571554691 shutdown_connections 2023-10-24T23:55:12.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f260814fc30 0x7f260815c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:12.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26080c5c20 0x7f2608069fa0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:12.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25f4065e60 0x7f25f4068320 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:12.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 --2- 172.21.15.107:0/3571554691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f260806e470 0x7f260815d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:12.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 -- 172.21.15.107:0/3571554691 >> 172.21.15.107:0/3571554691 conn(0x7f26080c39a0 msgr2=0x7f260806c200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:12.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 -- 172.21.15.107:0/3571554691 shutdown_connections 2023-10-24T23:55:12.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:12.683+0000 7f260e74e640 1 -- 172.21.15.107:0/3571554691 wait complete. 2023-10-24T23:55:12.706 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:55:14.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:14.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.843+0000 7fb504181640 1 Processor -- start 2023-10-24T23:55:14.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.843+0000 7fb504181640 1 -- start start 2023-10-24T23:55:14.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb504181640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb4fc152040 0x7fb4fc0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:14.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb504181640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc155680 con 0x7fb4fc0c8100 2023-10-24T23:55:14.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb504181640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4fc154a70 con 0x7fb4fc06a4b0 2023-10-24T23:55:14.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb504181640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4fc154d50 con 0x7fb4fc152040 2023-10-24T23:55:14.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb501f1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb4fc152040 0x7fb4fc0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:14.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb501f1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb4fc152040 0x7fb4fc0c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49690/0 (socket says 172.21.15.107:49690) 2023-10-24T23:55:14.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb501f1d640 1 -- 172.21.15.107:0/2518979939 learned_addr learned my addr 172.21.15.107:0/2518979939 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:14.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb50171c640 1 --1- 172.21.15.107:0/2518979939 >> v1:172.21.15.107:6789/0 conn(0x7fb4fc0c8100 0x7fb4fc0abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49028/0 (socket says 172.21.15.107:49028) 2023-10-24T23:55:14.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb501f1d640 1 -- 172.21.15.107:0/2518979939 >> v1:172.21.15.110:6789/0 conn(0x7fb4fc06a4b0 legacy=0x7fb4fc0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:14.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb501f1d640 1 -- 172.21.15.107:0/2518979939 >> v1:172.21.15.107:6789/0 conn(0x7fb4fc0c8100 legacy=0x7fb4fc0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:14.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb501f1d640 1 -- 172.21.15.107:0/2518979939 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc155030 con 0x7fb4fc152040 2023-10-24T23:55:14.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb500f1b640 1 -- 172.21.15.107:0/2518979939 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805875846 0 0) 0x7fb4fc155680 con 0x7fb4fc0c8100 2023-10-24T23:55:14.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb501f1d640 1 --2- 172.21.15.107:0/2518979939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4fc152040 0x7fb4fc0c4b80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009870 tx=0x7fb4f801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c9ce06a4c690dc server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:14.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb500f1b640 1 -- 172.21.15.107:0/2518979939 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f80ab600 con 0x7fb4fc152040 2023-10-24T23:55:14.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb500f1b640 1 -- 172.21.15.107:0/2518979939 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f80a7050 con 0x7fb4fc152040 2023-10-24T23:55:14.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb500f1b640 1 -- 172.21.15.107:0/2518979939 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f80ab980 con 0x7fb4fc152040 2023-10-24T23:55:14.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb504181640 1 -- 172.21.15.107:0/2518979939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4fc152040 msgr2=0x7fb4fc0c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:14.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb504181640 1 --2- 172.21.15.107:0/2518979939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4fc152040 0x7fb4fc0c4b80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8009870 tx=0x7fb4f801c3e0 comp rx=0 tx=0).stop 2023-10-24T23:55:14.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb504181640 1 -- 172.21.15.107:0/2518979939 shutdown_connections 2023-10-24T23:55:14.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb504181640 1 --2- 172.21.15.107:0/2518979939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4fc152040 0x7fb4fc0c4b80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:14.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.847+0000 7fb504181640 1 -- 172.21.15.107:0/2518979939 >> 172.21.15.107:0/2518979939 conn(0x7fb4fc0ca200 msgr2=0x7fb4fc0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:14.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 -- 172.21.15.107:0/2518979939 shutdown_connections 2023-10-24T23:55:14.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 -- 172.21.15.107:0/2518979939 wait complete. 2023-10-24T23:55:14.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 Processor -- start 2023-10-24T23:55:14.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 -- start start 2023-10-24T23:55:14.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4fc06a4b0 0x7fb4fc0ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:14.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4fc0c8100 0x7fb4fc0ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:14.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4fc152040 0x7fb4fc148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:14.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c2880 con 0x7fb4fc06a4b0 2023-10-24T23:55:14.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c40b0 con 0x7fb4fc152040 2023-10-24T23:55:14.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4fc0c3da0 con 0x7fb4fc0c8100 2023-10-24T23:55:14.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb501f1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4fc06a4b0 0x7fb4fc0ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:14.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb501f1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4fc06a4b0 0x7fb4fc0ab130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49704/0 (socket says 172.21.15.107:49704) 2023-10-24T23:55:14.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb501f1d640 1 -- 172.21.15.107:0/309461367 learned_addr learned my addr 172.21.15.107:0/309461367 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:14.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb501f1d640 1 -- 172.21.15.107:0/309461367 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4fc0c8100 msgr2=0x7fb4fc0ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:14.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb50171c640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4fc0c8100 0x7fb4fc0ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:14.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb50271e640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4fc152040 0x7fb4fc148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:14.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb501f1d640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4fc0c8100 0x7fb4fc0ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:14.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb501f1d640 1 -- 172.21.15.107:0/309461367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4fc152040 msgr2=0x7fb4fc148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:14.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb501f1d640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4fc152040 0x7fb4fc148640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:14.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb501f1d640 1 -- 172.21.15.107:0/309461367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f80094d0 con 0x7fb4fc06a4b0 2023-10-24T23:55:14.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb50171c640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4fc0c8100 0x7fb4fc0ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:14.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb501f1d640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4fc06a4b0 0x7fb4fc0ab130 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8018ff0 tx=0x7fb4f801ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:14.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb4f27fc640 1 -- 172.21.15.107:0/309461367 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f80b6620 con 0x7fb4fc06a4b0 2023-10-24T23:55:14.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb4f27fc640 1 -- 172.21.15.107:0/309461367 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f80b67a0 con 0x7fb4fc06a4b0 2023-10-24T23:55:14.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 -- 172.21.15.107:0/309461367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4fc146c80 con 0x7fb4fc06a4b0 2023-10-24T23:55:14.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb4f27fc640 1 -- 172.21.15.107:0/309461367 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f80b6b50 con 0x7fb4fc06a4b0 2023-10-24T23:55:14.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.851+0000 7fb504181640 1 -- 172.21.15.107:0/309461367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4fc0c25a0 con 0x7fb4fc06a4b0 2023-10-24T23:55:14.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.855+0000 7fb504181640 1 -- 172.21.15.107:0/309461367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4fc0c41f0 con 0x7fb4fc06a4b0 2023-10-24T23:55:14.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.855+0000 7fb4f27fc640 1 -- 172.21.15.107:0/309461367 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb4f80b2040 con 0x7fb4fc06a4b0 2023-10-24T23:55:14.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.855+0000 7fb4f27fc640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4e0065e60 0x7fb4e0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:14.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.855+0000 7fb4f27fc640 1 -- 172.21.15.107:0/309461367 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7fb4f811e0c0 con 0x7fb4fc06a4b0 2023-10-24T23:55:14.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.855+0000 7fb50171c640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4e0065e60 0x7fb4e0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:14.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.855+0000 7fb50171c640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4e0065e60 0x7fb4e0068320 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0c8d40 tx=0x7fb4ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:14.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:14.859+0000 7fb4f27fc640 1 -- 172.21.15.107:0/309461367 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb4fc0c41f0 con 0x7fb4fc06a4b0 2023-10-24T23:55:15.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.099+0000 7fb504181640 1 -- 172.21.15.107:0/309461367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4fc0c41f0 con 0x7fb4fc06a4b0 2023-10-24T23:55:15.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.099+0000 7fb4f27fc640 1 -- 172.21.15.107:0/309461367 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fb4fc0c41f0 con 0x7fb4fc06a4b0 2023-10-24T23:55:15.104 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:15.104 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:16.358716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:15.104 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-10-24T23:55:15.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.099+0000 7fb504181640 1 -- 172.21.15.107:0/309461367 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4e0065e60 msgr2=0x7fb4e0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:15.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.099+0000 7fb504181640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4e0065e60 0x7fb4e0068320 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc0c8d40 tx=0x7fb4ec009040 comp rx=0 tx=0).stop 2023-10-24T23:55:15.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.099+0000 7fb504181640 1 -- 172.21.15.107:0/309461367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4fc06a4b0 msgr2=0x7fb4fc0ab130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:15.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.099+0000 7fb504181640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4fc06a4b0 0x7fb4fc0ab130 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8018ff0 tx=0x7fb4f801ac20 comp rx=0 tx=0).stop 2023-10-24T23:55:15.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.103+0000 7fb504181640 1 -- 172.21.15.107:0/309461367 shutdown_connections 2023-10-24T23:55:15.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.103+0000 7fb504181640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4fc152040 0x7fb4fc148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:15.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.103+0000 7fb504181640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb4e0065e60 0x7fb4e0068320 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:15.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.103+0000 7fb504181640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb4fc0c8100 0x7fb4fc0ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:15.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.103+0000 7fb504181640 1 --2- 172.21.15.107:0/309461367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb4fc06a4b0 0x7fb4fc0ab130 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:15.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.103+0000 7fb504181640 1 -- 172.21.15.107:0/309461367 >> 172.21.15.107:0/309461367 conn(0x7fb4fc0ca200 msgr2=0x7fb4fc0c9920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:15.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.103+0000 7fb504181640 1 -- 172.21.15.107:0/309461367 shutdown_connections 2023-10-24T23:55:15.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:15.103+0000 7fb504181640 1 -- 172.21.15.107:0/309461367 wait complete. 2023-10-24T23:55:15.122 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:55:17.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:17.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.251+0000 7f1ca2184640 1 Processor -- start 2023-10-24T23:55:17.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.251+0000 7f1ca2184640 1 -- start start 2023-10-24T23:55:17.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1ca2184640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c9c0c2790 0x7f1c9c151480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:17.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1ca2184640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c9c0c44c0 con 0x7f1c9c06fde0 2023-10-24T23:55:17.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1ca2184640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c9c0c38b0 con 0x7f1c9c071770 2023-10-24T23:55:17.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1ca2184640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9c071b50 con 0x7f1c9c0c2790 2023-10-24T23:55:17.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c93fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c9c0c2790 0x7f1c9c151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:17.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c9bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1c9c06fde0 0x7f1c9c070250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49044/0 (socket says 172.21.15.107:49044) 2023-10-24T23:55:17.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c93fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c9c0c2790 0x7f1c9c151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49706/0 (socket says 172.21.15.107:49706) 2023-10-24T23:55:17.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c9bfff640 1 -- 172.21.15.107:0/3650144751 learned_addr learned my addr 172.21.15.107:0/3650144751 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:17.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c93fff640 1 -- 172.21.15.107:0/3650144751 >> v1:172.21.15.110:6789/0 conn(0x7f1c9c071770 legacy=0x7f1c9c06cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:17.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c93fff640 1 -- 172.21.15.107:0/3650144751 >> v1:172.21.15.107:6789/0 conn(0x7f1c9c06fde0 legacy=0x7f1c9c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:17.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c9affd640 1 -- 172.21.15.107:0/3650144751 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014953569 0 0) 0x7f1c9c0c44c0 con 0x7f1c9c06fde0 2023-10-24T23:55:17.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c93fff640 1 -- 172.21.15.107:0/3650144751 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9c0c3b90 con 0x7f1c9c0c2790 2023-10-24T23:55:17.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c93fff640 1 --2- 172.21.15.107:0/3650144751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9c0c2790 0x7f1c9c151480 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1c840097c0 tx=0x7f1c8401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8b048f19851de3 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:17.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c9affd640 1 -- 172.21.15.107:0/3650144751 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c840ab600 con 0x7f1c9c0c2790 2023-10-24T23:55:17.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c9affd640 1 -- 172.21.15.107:0/3650144751 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c840a7050 con 0x7f1c9c0c2790 2023-10-24T23:55:17.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1c9affd640 1 -- 172.21.15.107:0/3650144751 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c840ab980 con 0x7f1c9c0c2790 2023-10-24T23:55:17.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1ca2184640 1 -- 172.21.15.107:0/3650144751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9c0c2790 msgr2=0x7f1c9c151480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:17.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1ca2184640 1 --2- 172.21.15.107:0/3650144751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9c0c2790 0x7f1c9c151480 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f1c840097c0 tx=0x7f1c8401c340 comp rx=0 tx=0).stop 2023-10-24T23:55:17.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1ca2184640 1 -- 172.21.15.107:0/3650144751 shutdown_connections 2023-10-24T23:55:17.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1ca2184640 1 --2- 172.21.15.107:0/3650144751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9c0c2790 0x7f1c9c151480 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:17.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1ca2184640 1 -- 172.21.15.107:0/3650144751 >> 172.21.15.107:0/3650144751 conn(0x7f1c9c0c6ed0 msgr2=0x7f1c9c0c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:17.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1ca2184640 1 -- 172.21.15.107:0/3650144751 shutdown_connections 2023-10-24T23:55:17.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.255+0000 7f1ca2184640 1 -- 172.21.15.107:0/3650144751 wait complete. 2023-10-24T23:55:17.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1ca2184640 1 Processor -- start 2023-10-24T23:55:17.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1ca2184640 1 -- start start 2023-10-24T23:55:17.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1ca2184640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c9c06fde0 0x7f1c9c0a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:17.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1ca2184640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9c071770 0x7f1c9c0c0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:17.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1ca2184640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c9c0c2790 0x7f1c9c13efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:17.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1ca2184640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c1ec640 con 0x7f1c9c071770 2023-10-24T23:55:17.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1ca2184640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c1ed420 con 0x7f1c9c0c2790 2023-10-24T23:55:17.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1ca2184640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c1ec850 con 0x7f1c9c06fde0 2023-10-24T23:55:17.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c93fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9c071770 0x7f1c9c0c0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:17.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c93fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9c071770 0x7f1c9c0c0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49710/0 (socket says 172.21.15.107:49710) 2023-10-24T23:55:17.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c93fff640 1 -- 172.21.15.107:0/2848005246 learned_addr learned my addr 172.21.15.107:0/2848005246 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:17.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c9b7fe640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c9c06fde0 0x7f1c9c0a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:17.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c93fff640 1 -- 172.21.15.107:0/2848005246 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c9c06fde0 msgr2=0x7f1c9c0a9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:17.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c93fff640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c9c06fde0 0x7f1c9c0a9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:17.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c93fff640 1 -- 172.21.15.107:0/2848005246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c9c0c2790 msgr2=0x7f1c9c13efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:17.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c9bfff640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c9c0c2790 0x7f1c9c13efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:17.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c93fff640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c9c0c2790 0x7f1c9c13efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:17.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c93fff640 1 -- 172.21.15.107:0/2848005246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c84009420 con 0x7f1c9c071770 2023-10-24T23:55:17.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c9b7fe640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c9c06fde0 0x7f1c9c0a9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:17.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c93fff640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9c071770 0x7f1c9c0c0560 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1c84018f50 tx=0x7f1c8401ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:17.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c98ff9640 1 -- 172.21.15.107:0/2848005246 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c840b7650 con 0x7f1c9c071770 2023-10-24T23:55:17.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c98ff9640 1 -- 172.21.15.107:0/2848005246 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c840b77b0 con 0x7f1c9c071770 2023-10-24T23:55:17.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1ca2184640 1 -- 172.21.15.107:0/2848005246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c9c1ecaf0 con 0x7f1c9c071770 2023-10-24T23:55:17.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c98ff9640 1 -- 172.21.15.107:0/2848005246 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c840b7b60 con 0x7f1c9c071770 2023-10-24T23:55:17.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1ca2184640 1 -- 172.21.15.107:0/2848005246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9c1ed0b0 con 0x7f1c9c071770 2023-10-24T23:55:17.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1ca2184640 1 -- 172.21.15.107:0/2848005246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9c1ed590 con 0x7f1c9c071770 2023-10-24T23:55:17.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.259+0000 7f1c98ff9640 1 -- 172.21.15.107:0/2848005246 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1c840b7cc0 con 0x7f1c9c071770 2023-10-24T23:55:17.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.263+0000 7f1c98ff9640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c7c065e90 0x7f1c7c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:17.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.263+0000 7f1c98ff9640 1 -- 172.21.15.107:0/2848005246 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7f1c8411e4a0 con 0x7f1c9c071770 2023-10-24T23:55:17.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.263+0000 7f1c9b7fe640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c7c065e90 0x7f1c7c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:17.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.263+0000 7f1c9b7fe640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c7c065e90 0x7f1c7c068350 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0016e0 tx=0x7f1c8c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:17.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.267+0000 7f1c98ff9640 1 -- 172.21.15.107:0/2848005246 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1c840eb8c0 con 0x7f1c9c071770 2023-10-24T23:55:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 -- 172.21.15.107:0/2848005246 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c9c0746e0 con 0x7f1c9c071770 2023-10-24T23:55:17.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1c98ff9640 1 -- 172.21.15.107:0/2848005246 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f1c840eeee0 con 0x7f1c9c071770 2023-10-24T23:55:17.483 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:17.484 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:16.358716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:17.484 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-10-24T23:55:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 -- 172.21.15.107:0/2848005246 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c7c065e90 msgr2=0x7f1c7c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:17.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c7c065e90 0x7f1c7c068350 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0016e0 tx=0x7f1c8c009040 comp rx=0 tx=0).stop 2023-10-24T23:55:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 -- 172.21.15.107:0/2848005246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9c071770 msgr2=0x7f1c9c0c0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9c071770 0x7f1c9c0c0560 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1c84018f50 tx=0x7f1c8401ab80 comp rx=0 tx=0).stop 2023-10-24T23:55:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 -- 172.21.15.107:0/2848005246 shutdown_connections 2023-10-24T23:55:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c9c0c2790 0x7f1c9c13efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c9c071770 0x7f1c9c0c0560 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c7c065e90 0x7f1c7c068350 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 --2- 172.21.15.107:0/2848005246 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c9c06fde0 0x7f1c9c0a9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 -- 172.21.15.107:0/2848005246 >> 172.21.15.107:0/2848005246 conn(0x7f1c9c0c6ed0 msgr2=0x7f1c9c0c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:17.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 -- 172.21.15.107:0/2848005246 shutdown_connections 2023-10-24T23:55:17.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:17.479+0000 7f1ca2184640 1 -- 172.21.15.107:0/2848005246 wait complete. 2023-10-24T23:55:17.501 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:55:19.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:19.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.651+0000 7fba1addf640 1 Processor -- start 2023-10-24T23:55:19.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.651+0000 7fba1addf640 1 -- start start 2023-10-24T23:55:19.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba1addf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fba140c33e0 0x7fba1415ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:19.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba1addf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba140c22f0 con 0x7fba14070b00 2023-10-24T23:55:19.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba1addf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba140c25d0 con 0x7fba1414eb70 2023-10-24T23:55:19.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba1addf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba14071700 con 0x7fba140c33e0 2023-10-24T23:55:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba1937c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fba140c33e0 0x7fba1415ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba13fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fba14070b00 0x7fba1406df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49050/0 (socket says 172.21.15.107:49050) 2023-10-24T23:55:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba13fff640 1 -- 172.21.15.107:0/15435680 learned_addr learned my addr 172.21.15.107:0/15435680 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba1937c640 1 -- 172.21.15.107:0/15435680 >> v1:172.21.15.110:6789/0 conn(0x7fba1414eb70 legacy=0x7fba141514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba1937c640 1 -- 172.21.15.107:0/15435680 >> v1:172.21.15.107:6789/0 conn(0x7fba14070b00 legacy=0x7fba1406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba1937c640 1 -- 172.21.15.107:0/15435680 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba140c2be0 con 0x7fba140c33e0 2023-10-24T23:55:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba1937c640 1 --2- 172.21.15.107:0/15435680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba140c33e0 0x7fba1415ce70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009d90 tx=0x7fba0c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7f1410380153bd server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba137fe640 1 -- 172.21.15.107:0/15435680 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c0a58e0 con 0x7fba140c33e0 2023-10-24T23:55:19.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba137fe640 1 -- 172.21.15.107:0/15435680 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0c0a5a40 con 0x7fba140c33e0 2023-10-24T23:55:19.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.655+0000 7fba137fe640 1 -- 172.21.15.107:0/15435680 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c0a5dd0 con 0x7fba140c33e0 2023-10-24T23:55:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 -- 172.21.15.107:0/15435680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba140c33e0 msgr2=0x7fba1415ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 --2- 172.21.15.107:0/15435680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba140c33e0 0x7fba1415ce70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009d90 tx=0x7fba0c01c910 comp rx=0 tx=0).stop 2023-10-24T23:55:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 -- 172.21.15.107:0/15435680 shutdown_connections 2023-10-24T23:55:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 --2- 172.21.15.107:0/15435680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba140c33e0 0x7fba1415ce70 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 -- 172.21.15.107:0/15435680 >> 172.21.15.107:0/15435680 conn(0x7fba140c39a0 msgr2=0x7fba140c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 -- 172.21.15.107:0/15435680 shutdown_connections 2023-10-24T23:55:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 -- 172.21.15.107:0/15435680 wait complete. 2023-10-24T23:55:19.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 Processor -- start 2023-10-24T23:55:19.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 -- start start 2023-10-24T23:55:19.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba14070b00 0x7fba141473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba140c33e0 0x7fba1414ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba1414eb70 0x7fba1414b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba141378b0 con 0x7fba14070b00 2023-10-24T23:55:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1414b7b0 con 0x7fba1414eb70 2023-10-24T23:55:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba141eb660 con 0x7fba140c33e0 2023-10-24T23:55:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba18b7b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba14070b00 0x7fba141473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba18b7b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba14070b00 0x7fba141473b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49724/0 (socket says 172.21.15.107:49724) 2023-10-24T23:55:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba18b7b640 1 -- 172.21.15.107:0/2441324368 learned_addr learned my addr 172.21.15.107:0/2441324368 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba13fff640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba140c33e0 0x7fba1414ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1937c640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba1414eb70 0x7fba1414b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba18b7b640 1 -- 172.21.15.107:0/2441324368 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba140c33e0 msgr2=0x7fba1414ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:19.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba18b7b640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba140c33e0 0x7fba1414ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:19.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba18b7b640 1 -- 172.21.15.107:0/2441324368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba1414eb70 msgr2=0x7fba1414b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:19.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba18b7b640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba1414eb70 0x7fba1414b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:19.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba18b7b640 1 -- 172.21.15.107:0/2441324368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0c009a70 con 0x7fba14070b00 2023-10-24T23:55:19.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba18b7b640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba14070b00 0x7fba141473b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fba0800c450 tx=0x7fba0800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:19.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba117fa640 1 -- 172.21.15.107:0/2441324368 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08012830 con 0x7fba14070b00 2023-10-24T23:55:19.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba117fa640 1 -- 172.21.15.107:0/2441324368 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba080129b0 con 0x7fba14070b00 2023-10-24T23:55:19.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba1addf640 1 -- 172.21.15.107:0/2441324368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba141eb940 con 0x7fba14070b00 2023-10-24T23:55:19.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.659+0000 7fba117fa640 1 -- 172.21.15.107:0/2441324368 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08012d60 con 0x7fba14070b00 2023-10-24T23:55:19.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.663+0000 7fba1addf640 1 -- 172.21.15.107:0/2441324368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1406e2f0 con 0x7fba14070b00 2023-10-24T23:55:19.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.663+0000 7fba1addf640 1 -- 172.21.15.107:0/2441324368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba140746e0 con 0x7fba14070b00 2023-10-24T23:55:19.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.663+0000 7fba117fa640 1 -- 172.21.15.107:0/2441324368 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fba0801c460 con 0x7fba14070b00 2023-10-24T23:55:19.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.663+0000 7fba117fa640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb9f8065e60 0x7fb9f8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:19.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.663+0000 7fba117fa640 1 -- 172.21.15.107:0/2441324368 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7fba080844e0 con 0x7fba14070b00 2023-10-24T23:55:19.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.663+0000 7fba13fff640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb9f8065e60 0x7fb9f8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:19.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.663+0000 7fba13fff640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb9f8065e60 0x7fb9f8068320 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fba14149710 tx=0x7fba00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:19.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.667+0000 7fba117fa640 1 -- 172.21.15.107:0/2441324368 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fba080518c0 con 0x7fba14070b00 2023-10-24T23:55:19.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.887+0000 7fba1addf640 1 -- 172.21.15.107:0/2441324368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba140c1a50 con 0x7fba14070b00 2023-10-24T23:55:19.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba117fa640 1 -- 172.21.15.107:0/2441324368 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fba08054ee0 con 0x7fba14070b00 2023-10-24T23:55:19.895 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:19.895 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:16.358716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:19.896 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-10-24T23:55:19.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 -- 172.21.15.107:0/2441324368 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb9f8065e60 msgr2=0x7fb9f8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:19.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb9f8065e60 0x7fb9f8068320 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fba14149710 tx=0x7fba00009040 comp rx=0 tx=0).stop 2023-10-24T23:55:19.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 -- 172.21.15.107:0/2441324368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba14070b00 msgr2=0x7fba141473b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba14070b00 0x7fba141473b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fba0800c450 tx=0x7fba0800c910 comp rx=0 tx=0).stop 2023-10-24T23:55:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 -- 172.21.15.107:0/2441324368 shutdown_connections 2023-10-24T23:55:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba1414eb70 0x7fba1414b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb9f8065e60 0x7fb9f8068320 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba140c33e0 0x7fba1414ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 --2- 172.21.15.107:0/2441324368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba14070b00 0x7fba141473b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 -- 172.21.15.107:0/2441324368 >> 172.21.15.107:0/2441324368 conn(0x7fba140c39a0 msgr2=0x7fba1406a750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 -- 172.21.15.107:0/2441324368 shutdown_connections 2023-10-24T23:55:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:19.891+0000 7fba1addf640 1 -- 172.21.15.107:0/2441324368 wait complete. 2023-10-24T23:55:19.915 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:55:21.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:22.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.059+0000 7f8135c79640 1 Processor -- start 2023-10-24T23:55:22.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.059+0000 7f8135c79640 1 -- start start 2023-10-24T23:55:22.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f8135c79640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f813014c470 0x7f813014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:22.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f8135c79640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f813006f490 0x7f8130071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:22.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f8135c79640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8130155cd0 con 0x7f8130071fc0 2023-10-24T23:55:22.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f8135c79640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81301526e0 con 0x7f813014c470 2023-10-24T23:55:22.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f8135c79640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81300c4380 con 0x7f813006f490 2023-10-24T23:55:22.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8130071fc0 0x7f8130152b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49062/0 (socket says 172.21.15.107:49062) 2023-10-24T23:55:22.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812ffff640 1 -- 172.21.15.107:0/1531189816 learned_addr learned my addr 172.21.15.107:0/1531189816 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:22.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812effd640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f813006f490 0x7f8130071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:22.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812f7fe640 1 --2- 172.21.15.107:0/1531189816 >> v2:172.21.15.110:3300/0 conn(0x7f813014c470 0x7f813014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:22.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568060208 0 0) 0x7f8130155cd0 con 0x7f8130071fc0 2023-10-24T23:55:22.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8118003530 con 0x7f8130071fc0 2023-10-24T23:55:22.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194489737 0 0) 0x7f8118003530 con 0x7f8130071fc0 2023-10-24T23:55:22.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f813006f490 msgr2=0x7f8130071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:22.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f813006f490 0x7f8130071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:22.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 >> v2:172.21.15.110:3300/0 conn(0x7f813014c470 msgr2=0x7f813014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:22.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 --2- 172.21.15.107:0/1531189816 >> v2:172.21.15.110:3300/0 conn(0x7f813014c470 0x7f813014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:22.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81300c48b0 con 0x7f8130071fc0 2023-10-24T23:55:22.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f8128001a00 con 0x7f8130071fc0 2023-10-24T23:55:22.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 >> v1:172.21.15.107:6789/0 conn(0x7f8130071fc0 legacy=0x7f8130152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:22.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8118003980 0x7f8118003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:22.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f813006f490 0x7f81180047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:22.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8118004d10 0x7f8118005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:22.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81301526e0 con 0x7f8118003980 2023-10-24T23:55:22.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81300c4380 con 0x7f8118004d10 2023-10-24T23:55:22.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811800ba40 con 0x7f813006f490 2023-10-24T23:55:22.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812f7fe640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8118003980 0x7f8118003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:22.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812effd640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f813006f490 0x7f81180047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:22.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812f7fe640 1 -- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f813006f490 msgr2=0x7f81180047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:22.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812ffff640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8118004d10 0x7f8118005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:22.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812f7fe640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f813006f490 0x7f81180047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:22.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812f7fe640 1 -- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8118004d10 msgr2=0x7f8118005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:22.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812f7fe640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8118004d10 0x7f8118005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:22.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812f7fe640 1 -- 172.21.15.107:0/1531189816 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8118003530 con 0x7f8118003980 2023-10-24T23:55:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.063+0000 7f812effd640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f813006f490 0x7f81180047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:22.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f812f7fe640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8118003980 0x7f8118003da0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8120009ae0 tx=0x7f812001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599e1405cbf23e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:22.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81200a6470 con 0x7f8118003980 2023-10-24T23:55:22.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81200a65d0 con 0x7f8118003980 2023-10-24T23:55:22.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f812e7fc640 1 -- 172.21.15.107:0/1531189816 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81200a6980 con 0x7f8118003980 2023-10-24T23:55:22.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f8135c79640 1 -- 172.21.15.107:0/1531189816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8118003980 msgr2=0x7f8118003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:22.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f8135c79640 1 --2- 172.21.15.107:0/1531189816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8118003980 0x7f8118003da0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8120009ae0 tx=0x7f812001ca90 comp rx=0 tx=0).stop 2023-10-24T23:55:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f812ffff640 1 -- 172.21.15.107:0/1531189816 reap_dead start 2023-10-24T23:55:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f8135c79640 1 -- 172.21.15.107:0/1531189816 shutdown_connections 2023-10-24T23:55:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f8135c79640 1 -- 172.21.15.107:0/1531189816 >> 172.21.15.107:0/1531189816 conn(0x7f8130070f60 msgr2=0x7f8130071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f8135c79640 1 -- 172.21.15.107:0/1531189816 shutdown_connections 2023-10-24T23:55:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f8135c79640 1 -- 172.21.15.107:0/1531189816 wait complete. 2023-10-24T23:55:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f8135c79640 1 Processor -- start 2023-10-24T23:55:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.067+0000 7f8135c79640 1 -- start start 2023-10-24T23:55:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8135c79640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81301496a0 0x7f8130149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8135c79640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f813006b1f0 0x7f81300c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8135c79640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81300c6460 0x7f81300c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8135c79640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813006c3e0 con 0x7f81300c6460 2023-10-24T23:55:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8135c79640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8130156e20 con 0x7f813006b1f0 2023-10-24T23:55:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8135c79640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813006b850 con 0x7f81301496a0 2023-10-24T23:55:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81300c6460 0x7f81300c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81300c6460 0x7f81300c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49752/0 (socket says 172.21.15.107:49752) 2023-10-24T23:55:22.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812ffff640 1 -- 172.21.15.107:0/1158740053 learned_addr learned my addr 172.21.15.107:0/1158740053 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:22.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812f7fe640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81301496a0 0x7f8130149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:22.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812effd640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f813006b1f0 0x7f81300c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:22.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812ffff640 1 -- 172.21.15.107:0/1158740053 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81301496a0 msgr2=0x7f8130149d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812ffff640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81301496a0 0x7f8130149d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812ffff640 1 -- 172.21.15.107:0/1158740053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f813006b1f0 msgr2=0x7f81300c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812ffff640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f813006b1f0 0x7f81300c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812ffff640 1 -- 172.21.15.107:0/1158740053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813006bb30 con 0x7f81300c6460 2023-10-24T23:55:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812ffff640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81300c6460 0x7f81300c6fc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f81280014b0 tx=0x7f812800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8117fff640 1 -- 172.21.15.107:0/1158740053 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812800bd80 con 0x7f81300c6460 2023-10-24T23:55:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8117fff640 1 -- 172.21.15.107:0/1158740053 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81280070d0 con 0x7f81300c6460 2023-10-24T23:55:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8135c79640 1 -- 172.21.15.107:0/1158740053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81301ed780 con 0x7f81300c6460 2023-10-24T23:55:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8117fff640 1 -- 172.21.15.107:0/1158740053 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8128007480 con 0x7f81300c6460 2023-10-24T23:55:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8135c79640 1 -- 172.21.15.107:0/1158740053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81300c18b0 con 0x7f81300c6460 2023-10-24T23:55:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8135c79640 1 -- 172.21.15.107:0/1158740053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8110005260 con 0x7f81300c6460 2023-10-24T23:55:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8117fff640 1 -- 172.21.15.107:0/1158740053 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f81280075e0 con 0x7f81300c6460 2023-10-24T23:55:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8117fff640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f811c065e90 0x7f811c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f8117fff640 1 -- 172.21.15.107:0/1158740053 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 7047+0+0 (secure 0 0 0) 0x7f8128083b30 con 0x7f81300c6460 2023-10-24T23:55:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.071+0000 7f812f7fe640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f811c065e90 0x7f811c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.075+0000 7f812f7fe640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f811c065e90 0x7f811c068350 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f81200ad000 tx=0x7f81200b3040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.079+0000 7f8117fff640 1 -- 172.21.15.107:0/1158740053 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8128050ee0 con 0x7f81300c6460 2023-10-24T23:55:22.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.303+0000 7f8135c79640 1 -- 172.21.15.107:0/1158740053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81100050d0 con 0x7f81300c6460 2023-10-24T23:55:22.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.303+0000 7f8117fff640 1 -- 172.21.15.107:0/1158740053 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f8128019020 con 0x7f81300c6460 2023-10-24T23:55:22.309 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:22.309 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5292,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:54:16.358716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/1220982369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1220982369},{"type":"v1","addr":"172.21.15.107:6835","nonce":1220982369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:22.310 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 17 2023-10-24T23:55:22.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 -- 172.21.15.107:0/1158740053 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f811c065e90 msgr2=0x7f811c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f811c065e90 0x7f811c068350 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f81200ad000 tx=0x7f81200b3040 comp rx=0 tx=0).stop 2023-10-24T23:55:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 -- 172.21.15.107:0/1158740053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81300c6460 msgr2=0x7f81300c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81300c6460 0x7f81300c6fc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f81280014b0 tx=0x7f812800b4a0 comp rx=0 tx=0).stop 2023-10-24T23:55:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 -- 172.21.15.107:0/1158740053 shutdown_connections 2023-10-24T23:55:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81300c6460 0x7f81300c6fc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f813006b1f0 0x7f81300c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f811c065e90 0x7f811c068350 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 --2- 172.21.15.107:0/1158740053 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81301496a0 0x7f8130149d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 -- 172.21.15.107:0/1158740053 >> 172.21.15.107:0/1158740053 conn(0x7f8130070f60 msgr2=0x7f813014b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 -- 172.21.15.107:0/1158740053 shutdown_connections 2023-10-24T23:55:22.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:22.307+0000 7f8135c79640 1 -- 172.21.15.107:0/1158740053 wait complete. 2023-10-24T23:55:22.327 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:55:24.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:24.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa90bc8640 1 Processor -- start 2023-10-24T23:55:24.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa90bc8640 1 -- start start 2023-10-24T23:55:24.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa90bc8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faa8c14eb70 0x7faa8c0c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa90bc8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c070b00 0x7faa8c1514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa90bc8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faa8c151a20 0x7faa8c06df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa90bc8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c071700 con 0x7faa8c151a20 2023-10-24T23:55:24.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa90bc8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c0c22f0 con 0x7faa8c14eb70 2023-10-24T23:55:24.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa90bc8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0c25d0 con 0x7faa8c070b00 2023-10-24T23:55:24.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa8ad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faa8c151a20 0x7faa8c06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa8ad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faa8c151a20 0x7faa8c06df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41714/0 (socket says 172.21.15.107:41714) 2023-10-24T23:55:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa8ad9d640 1 -- 172.21.15.107:0/471919086 learned_addr learned my addr 172.21.15.107:0/471919086 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa89d9b640 1 --2- 172.21.15.107:0/471919086 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c070b00 0x7faa8c1514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa8a59c640 1 --2- 172.21.15.107:0/471919086 >> v2:172.21.15.110:3300/0 conn(0x7faa8c14eb70 0x7faa8c0c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa8ad9d640 1 -- 172.21.15.107:0/471919086 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c070b00 msgr2=0x7faa8c1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:24.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa8ad9d640 1 --2- 172.21.15.107:0/471919086 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c070b00 0x7faa8c1514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa8ad9d640 1 -- 172.21.15.107:0/471919086 >> v2:172.21.15.110:3300/0 conn(0x7faa8c14eb70 msgr2=0x7faa8c0c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:24.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa8ad9d640 1 --2- 172.21.15.107:0/471919086 >> v2:172.21.15.110:3300/0 conn(0x7faa8c14eb70 0x7faa8c0c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.451+0000 7faa8ad9d640 1 -- 172.21.15.107:0/471919086 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c0c2be0 con 0x7faa8c151a20 2023-10-24T23:55:24.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa8ad9d640 1 --2- 172.21.15.107:0/471919086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c151a20 0x7faa8c06df30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0c6730 tx=0x7faa8401ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303e1270b5619acd server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa8959a640 1 -- 172.21.15.107:0/471919086 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8400b2d0 con 0x7faa8c151a20 2023-10-24T23:55:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa8959a640 1 -- 172.21.15.107:0/471919086 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8400b430 con 0x7faa8c151a20 2023-10-24T23:55:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa8959a640 1 -- 172.21.15.107:0/471919086 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa840a7c20 con 0x7faa8c151a20 2023-10-24T23:55:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa90bc8640 1 -- 172.21.15.107:0/471919086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c151a20 msgr2=0x7faa8c06df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:24.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa90bc8640 1 --2- 172.21.15.107:0/471919086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c151a20 0x7faa8c06df30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0c6730 tx=0x7faa8401ebd0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa90bc8640 1 -- 172.21.15.107:0/471919086 shutdown_connections 2023-10-24T23:55:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa90bc8640 1 --2- 172.21.15.107:0/471919086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c151a20 0x7faa8c06df30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa90bc8640 1 --2- 172.21.15.107:0/471919086 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c070b00 0x7faa8c1514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa90bc8640 1 --2- 172.21.15.107:0/471919086 >> v2:172.21.15.110:3300/0 conn(0x7faa8c14eb70 0x7faa8c0c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa90bc8640 1 -- 172.21.15.107:0/471919086 >> 172.21.15.107:0/471919086 conn(0x7faa8c0c39a0 msgr2=0x7faa8c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa90bc8640 1 -- 172.21.15.107:0/471919086 shutdown_connections 2023-10-24T23:55:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa90bc8640 1 -- 172.21.15.107:0/471919086 wait complete. 2023-10-24T23:55:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa90bc8640 1 Processor -- start 2023-10-24T23:55:24.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.455+0000 7faa90bc8640 1 -- start start 2023-10-24T23:55:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa90bc8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c070b00 0x7faa8c06c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa90bc8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa8c14eb70 0x7faa8c150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa90bc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c151a20 0x7faa8c142290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa90bc8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c06cd20 con 0x7faa8c151a20 2023-10-24T23:55:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa90bc8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0abf50 con 0x7faa8c14eb70 2023-10-24T23:55:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa90bc8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c06b940 con 0x7faa8c070b00 2023-10-24T23:55:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa8ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c151a20 0x7faa8c142290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa8ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c151a20 0x7faa8c142290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41716/0 (socket says 172.21.15.107:41716) 2023-10-24T23:55:24.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa8ad9d640 1 -- 172.21.15.107:0/1107682710 learned_addr learned my addr 172.21.15.107:0/1107682710 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa89d9b640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa8c14eb70 0x7faa8c150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa8a59c640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c070b00 0x7faa8c06c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa8ad9d640 1 -- 172.21.15.107:0/1107682710 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c070b00 msgr2=0x7faa8c06c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:24.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa8ad9d640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c070b00 0x7faa8c06c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa8ad9d640 1 -- 172.21.15.107:0/1107682710 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa8c14eb70 msgr2=0x7faa8c150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa8ad9d640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa8c14eb70 0x7faa8c150cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa8ad9d640 1 -- 172.21.15.107:0/1107682710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8400bd40 con 0x7faa8c151a20 2023-10-24T23:55:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa8ad9d640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c151a20 0x7faa8c142290 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7faa840b1040 tx=0x7faa8401b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa7affd640 1 -- 172.21.15.107:0/1107682710 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84009a10 con 0x7faa8c151a20 2023-10-24T23:55:24.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa7affd640 1 -- 172.21.15.107:0/1107682710 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa84009b70 con 0x7faa8c151a20 2023-10-24T23:55:24.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa7affd640 1 -- 172.21.15.107:0/1107682710 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa840ba470 con 0x7faa8c151a20 2023-10-24T23:55:24.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa90bc8640 1 -- 172.21.15.107:0/1107682710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c06bbe0 con 0x7faa8c151a20 2023-10-24T23:55:24.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa90bc8640 1 -- 172.21.15.107:0/1107682710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c06c1a0 con 0x7faa8c151a20 2023-10-24T23:55:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa90bc8640 1 -- 172.21.15.107:0/1107682710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c0a9450 con 0x7faa8c151a20 2023-10-24T23:55:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa7affd640 1 -- 172.21.15.107:0/1107682710 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7faa840ba5d0 con 0x7faa8c151a20 2023-10-24T23:55:24.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa7affd640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa6c065e60 0x7faa6c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa7affd640 1 -- 172.21.15.107:0/1107682710 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7faa841216e0 con 0x7faa8c151a20 2023-10-24T23:55:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.459+0000 7faa8a59c640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa6c065e60 0x7faa6c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.463+0000 7faa8a59c640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa6c065e60 0x7faa6c068320 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faa7c00b800 tx=0x7faa7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:24.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.463+0000 7faa7affd640 1 -- 172.21.15.107:0/1107682710 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7faa840eea70 con 0x7faa8c151a20 2023-10-24T23:55:24.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.695+0000 7faa90bc8640 1 -- 172.21.15.107:0/1107682710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa8c0a9450 con 0x7faa8c151a20 2023-10-24T23:55:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.695+0000 7faa7affd640 1 -- 172.21.15.107:0/1107682710 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5012 (secure 0 0 0) 0x7faa8c0a9450 con 0x7faa8c151a20 2023-10-24T23:55:24.701 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:24.702 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:24.554797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 19 2023-10-24T23:55:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 -- 172.21.15.107:0/1107682710 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa6c065e60 msgr2=0x7faa6c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa6c065e60 0x7faa6c068320 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7faa7c00b800 tx=0x7faa7c008040 comp rx=0 tx=0).stop 2023-10-24T23:55:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 -- 172.21.15.107:0/1107682710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c151a20 msgr2=0x7faa8c142290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c151a20 0x7faa8c142290 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7faa840b1040 tx=0x7faa8401b970 comp rx=0 tx=0).stop 2023-10-24T23:55:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 -- 172.21.15.107:0/1107682710 shutdown_connections 2023-10-24T23:55:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c151a20 0x7faa8c142290 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa8c14eb70 0x7faa8c150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa6c065e60 0x7faa6c068320 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 --2- 172.21.15.107:0/1107682710 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c070b00 0x7faa8c06c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 -- 172.21.15.107:0/1107682710 >> 172.21.15.107:0/1107682710 conn(0x7faa8c0c39a0 msgr2=0x7faa8c1493b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:24.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 -- 172.21.15.107:0/1107682710 shutdown_connections 2023-10-24T23:55:24.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.699+0000 7faa90bc8640 1 -- 172.21.15.107:0/1107682710 wait complete. 2023-10-24T23:55:24.724 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2023-10-24T23:55:24.724 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-24T23:55:24.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:24.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989c633640 1 Processor -- start 2023-10-24T23:55:24.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989c633640 1 -- start start 2023-10-24T23:55:24.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989c633640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9894071770 0x7f9894151480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989c633640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f989406fde0 0x7f9894070250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989c633640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98940c44c0 con 0x7f98940c2790 2023-10-24T23:55:24.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989c633640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9894071b50 con 0x7f9894071770 2023-10-24T23:55:24.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989c633640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98940c38b0 con 0x7f989406fde0 2023-10-24T23:55:24.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989a3cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9894071770 0x7f9894151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989a3cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9894071770 0x7f9894151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41730/0 (socket says 172.21.15.107:41730) 2023-10-24T23:55:24.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989a3cf640 1 -- 172.21.15.107:0/3654281027 learned_addr learned my addr 172.21.15.107:0/3654281027 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:24.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989a3cf640 1 -- 172.21.15.107:0/3654281027 >> v2:172.21.15.110:3300/0 conn(0x7f989406fde0 msgr2=0x7f9894070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:24.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989a3cf640 1 --2- 172.21.15.107:0/3654281027 >> v2:172.21.15.110:3300/0 conn(0x7f989406fde0 0x7f9894070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989a3cf640 1 -- 172.21.15.107:0/3654281027 >> v1:172.21.15.110:6789/0 conn(0x7f98940c2790 legacy=0x7f989406cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:24.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.835+0000 7f989a3cf640 1 -- 172.21.15.107:0/3654281027 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98940c3b90 con 0x7f9894071770 2023-10-24T23:55:24.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f989a3cf640 1 --2- 172.21.15.107:0/3654281027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9894071770 0x7f9894151480 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f989406df20 tx=0x7f989001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e4a5e068fd4256 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:24.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f98993cd640 1 -- 172.21.15.107:0/3654281027 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98900ab600 con 0x7f9894071770 2023-10-24T23:55:24.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f98993cd640 1 -- 172.21.15.107:0/3654281027 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98900a7050 con 0x7f9894071770 2023-10-24T23:55:24.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f98993cd640 1 -- 172.21.15.107:0/3654281027 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98900ab980 con 0x7f9894071770 2023-10-24T23:55:24.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f989c633640 1 -- 172.21.15.107:0/3654281027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9894071770 msgr2=0x7f9894151480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:24.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f989c633640 1 --2- 172.21.15.107:0/3654281027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9894071770 0x7f9894151480 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f989406df20 tx=0x7f989001c3a0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f989c633640 1 -- 172.21.15.107:0/3654281027 shutdown_connections 2023-10-24T23:55:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f989c633640 1 --2- 172.21.15.107:0/3654281027 >> v2:172.21.15.110:3300/0 conn(0x7f989406fde0 0x7f9894070250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f989c633640 1 --2- 172.21.15.107:0/3654281027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9894071770 0x7f9894151480 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f989c633640 1 -- 172.21.15.107:0/3654281027 >> 172.21.15.107:0/3654281027 conn(0x7f98940c6ed0 msgr2=0x7f98940c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f989c633640 1 -- 172.21.15.107:0/3654281027 shutdown_connections 2023-10-24T23:55:24.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.839+0000 7f989c633640 1 -- 172.21.15.107:0/3654281027 wait complete. 2023-10-24T23:55:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989c633640 1 Processor -- start 2023-10-24T23:55:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989c633640 1 -- start start 2023-10-24T23:55:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989c633640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f989406fde0 0x7f98941eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989c633640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98940c2790 0x7f98941412a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989c633640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9894141d20 0x7f989414ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989c633640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9894151c80 con 0x7f98940c2790 2023-10-24T23:55:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989c633640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98941eb670 con 0x7f989406fde0 2023-10-24T23:55:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989c633640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98941ea540 con 0x7f9894141d20 2023-10-24T23:55:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989a3cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f989406fde0 0x7f98941eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989abd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9894141d20 0x7f989414ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989a3cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f989406fde0 0x7f98941eb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:38974/0 (socket says 172.21.15.107:38974) 2023-10-24T23:55:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989a3cf640 1 -- 172.21.15.107:0/1043405122 learned_addr learned my addr 172.21.15.107:0/1043405122 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f9899bce640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98940c2790 0x7f98941412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f9899bce640 1 -- 172.21.15.107:0/1043405122 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9894141d20 msgr2=0x7f989414ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f9899bce640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9894141d20 0x7f989414ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f9899bce640 1 -- 172.21.15.107:0/1043405122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f989406fde0 msgr2=0x7f98941eb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f9899bce640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f989406fde0 0x7f98941eb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f9899bce640 1 -- 172.21.15.107:0/1043405122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98900094d0 con 0x7f98940c2790 2023-10-24T23:55:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f9899bce640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98940c2790 0x7f98941412a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f988400b410 tx=0x7f988400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:24.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f988affd640 1 -- 172.21.15.107:0/1043405122 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9884011830 con 0x7f98940c2790 2023-10-24T23:55:24.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f988affd640 1 -- 172.21.15.107:0/1043405122 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98840119b0 con 0x7f98940c2790 2023-10-24T23:55:24.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f988affd640 1 -- 172.21.15.107:0/1043405122 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9884011d60 con 0x7f98940c2790 2023-10-24T23:55:24.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989c633640 1 -- 172.21.15.107:0/1043405122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9894148640 con 0x7f98940c2790 2023-10-24T23:55:24.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989c633640 1 -- 172.21.15.107:0/1043405122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9894148c00 con 0x7f98940c2790 2023-10-24T23:55:24.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.843+0000 7f989c633640 1 -- 172.21.15.107:0/1043405122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98940746e0 con 0x7f98940c2790 2023-10-24T23:55:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.847+0000 7f988affd640 1 -- 172.21.15.107:0/1043405122 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f988401a400 con 0x7f98940c2790 2023-10-24T23:55:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.847+0000 7f988affd640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9878065e60 0x7f9878068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.847+0000 7f988affd640 1 -- 172.21.15.107:0/1043405122 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f9884083b90 con 0x7f98940c2790 2023-10-24T23:55:24.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.847+0000 7f989a3cf640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9878065e60 0x7f9878068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:24.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.847+0000 7f989a3cf640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9878065e60 0x7f9878068320 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f989001c3e0 tx=0x7f98900b4040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:24.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:24.851+0000 7f988affd640 1 -- 172.21.15.107:0/1043405122 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9884050ef0 con 0x7f98940c2790 2023-10-24T23:55:25.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.071+0000 7f989c633640 1 -- 172.21.15.107:0/1043405122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f989415bbe0 con 0x7f98940c2790 2023-10-24T23:55:25.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.071+0000 7f988affd640 1 -- 172.21.15.107:0/1043405122 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+5012 (secure 0 0 0) 0x7f9884054510 con 0x7f98940c2790 2023-10-24T23:55:25.078 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:25.078 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:24.554797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:25.079 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 19 2023-10-24T23:55:25.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 -- 172.21.15.107:0/1043405122 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9878065e60 msgr2=0x7f9878068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:25.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9878065e60 0x7f9878068320 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f989001c3e0 tx=0x7f98900b4040 comp rx=0 tx=0).stop 2023-10-24T23:55:25.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 -- 172.21.15.107:0/1043405122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98940c2790 msgr2=0x7f98941412a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:25.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98940c2790 0x7f98941412a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f988400b410 tx=0x7f988400b8d0 comp rx=0 tx=0).stop 2023-10-24T23:55:25.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 -- 172.21.15.107:0/1043405122 shutdown_connections 2023-10-24T23:55:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9878065e60 0x7f9878068320 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9894141d20 0x7f989414ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98940c2790 0x7f98941412a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:25.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 --2- 172.21.15.107:0/1043405122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f989406fde0 0x7f98941eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:25.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 -- 172.21.15.107:0/1043405122 >> 172.21.15.107:0/1043405122 conn(0x7f98940c6ed0 msgr2=0x7f98940a9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:25.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 -- 172.21.15.107:0/1043405122 shutdown_connections 2023-10-24T23:55:25.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:25.075+0000 7f989c633640 1 -- 172.21.15.107:0/1043405122 wait complete. 2023-10-24T23:55:25.095 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 19, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 8, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4299": { "addr": "172.21.15.110:6835/2480841919", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2480841919, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2480841919, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4299, "incarnation": 4, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:53:06.342509+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4299 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5292": { "addr": "172.21.15.107:6837/838886697", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 838886697, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 838886697, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 18, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-24T23:55:24.554797+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5292 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/3721101226", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 3721101226, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 3721101226, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4232, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-24T23:55:27.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:27.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f8b3c640 1 Processor -- start 2023-10-24T23:55:27.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f8b3c640 1 -- start start 2023-10-24T23:55:27.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f8b3c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff6f4152040 0x7ff6f40c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:27.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f8b3c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f4155680 con 0x7ff6f40c8100 2023-10-24T23:55:27.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f8b3c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f4154a70 con 0x7ff6f406a4b0 2023-10-24T23:55:27.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f8b3c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f4154d50 con 0x7ff6f4152040 2023-10-24T23:55:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6ebfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff6f40c8100 0x7ff6f40abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51856/0 (socket says 172.21.15.107:51856) 2023-10-24T23:55:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6ebfff640 1 -- 172.21.15.107:0/1512470863 learned_addr learned my addr 172.21.15.107:0/1512470863 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f259c640 1 --2- 172.21.15.107:0/1512470863 >> v2:172.21.15.110:3300/0 conn(0x7ff6f4152040 0x7ff6f40c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977461226 0 0) 0x7ff6f4155680 con 0x7ff6f40c8100 2023-10-24T23:55:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e0003530 con 0x7ff6f40c8100 2023-10-24T23:55:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 859319642 0 0) 0x7ff6e0003530 con 0x7ff6f40c8100 2023-10-24T23:55:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 >> v2:172.21.15.110:3300/0 conn(0x7ff6f4152040 msgr2=0x7ff6f40c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 --2- 172.21.15.107:0/1512470863 >> v2:172.21.15.110:3300/0 conn(0x7ff6f4152040 0x7ff6f40c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 >> v1:172.21.15.110:6789/0 conn(0x7ff6f406a4b0 legacy=0x7ff6f40ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f4155030 con 0x7ff6f40c8100 2023-10-24T23:55:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7ff6dc001680 con 0x7ff6f40c8100 2023-10-24T23:55:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 >> v1:172.21.15.107:6789/0 conn(0x7ff6f40c8100 legacy=0x7ff6f40abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 --2- 172.21.15.107:0/1512470863 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6e0003980 0x7ff6e0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 --2- 172.21.15.107:0/1512470863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 0x7ff6e0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 --2- 172.21.15.107:0/1512470863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6e00052b0 0x7ff6e000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f4154d50 con 0x7ff6e00047d0 2023-10-24T23:55:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e000c180 con 0x7ff6e00052b0 2023-10-24T23:55:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e000c340 con 0x7ff6e0003980 2023-10-24T23:55:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6ebfff640 1 --2- 172.21.15.107:0/1512470863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 0x7ff6e0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:27.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f2d9d640 1 --2- 172.21.15.107:0/1512470863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6e00052b0 0x7ff6e000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:27.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f259c640 1 --2- 172.21.15.107:0/1512470863 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6e0003980 0x7ff6e0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:27.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6ebfff640 1 -- 172.21.15.107:0/1512470863 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6e0003980 msgr2=0x7ff6e0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:27.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6ebfff640 1 --2- 172.21.15.107:0/1512470863 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6e0003980 0x7ff6e0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:27.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6ebfff640 1 -- 172.21.15.107:0/1512470863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6e00052b0 msgr2=0x7ff6e000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:27.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6ebfff640 1 --2- 172.21.15.107:0/1512470863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6e00052b0 0x7ff6e000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:27.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6ebfff640 1 -- 172.21.15.107:0/1512470863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e0003530 con 0x7ff6e00047d0 2023-10-24T23:55:27.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.219+0000 7ff6f2d9d640 1 -- 172.21.15.107:0/1512470863 reap_dead start 2023-10-24T23:55:27.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.223+0000 7ff6ebfff640 1 --2- 172.21.15.107:0/1512470863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 0x7ff6e0004c50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40c8560 tx=0x7ff6dc01cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b3cb4c850447f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:27.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.223+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc0058e0 con 0x7ff6e00047d0 2023-10-24T23:55:27.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.223+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc005a60 con 0x7ff6e00047d0 2023-10-24T23:55:27.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.223+0000 7ff6f1d9b640 1 -- 172.21.15.107:0/1512470863 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc005e10 con 0x7ff6e00047d0 2023-10-24T23:55:27.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.223+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1512470863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 msgr2=0x7ff6e0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:27.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.223+0000 7ff6f8b3c640 1 --2- 172.21.15.107:0/1512470863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 0x7ff6e0004c50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40c8560 tx=0x7ff6dc01cbe0 comp rx=0 tx=0).stop 2023-10-24T23:55:27.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.223+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1512470863 shutdown_connections 2023-10-24T23:55:27.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.223+0000 7ff6f8b3c640 1 --2- 172.21.15.107:0/1512470863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 0x7ff6e0004c50 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:27.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.223+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1512470863 >> 172.21.15.107:0/1512470863 conn(0x7ff6f40ca200 msgr2=0x7ff6f40ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:27.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.223+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1512470863 shutdown_connections 2023-10-24T23:55:27.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.223+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1512470863 wait complete. 2023-10-24T23:55:27.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f8b3c640 1 Processor -- start 2023-10-24T23:55:27.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f8b3c640 1 -- start start 2023-10-24T23:55:27.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f8b3c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 0x7ff6f40c3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:27.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f8b3c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6f414ab80 0x7ff6f41e0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:27.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f8b3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6f41e1120 0x7ff6f414b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:27.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f8b3c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40ca0a0 con 0x7ff6e00047d0 2023-10-24T23:55:27.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f8b3c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f41ed3d0 con 0x7ff6f414ab80 2023-10-24T23:55:27.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f8b3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f41ed950 con 0x7ff6f41e1120 2023-10-24T23:55:27.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 0x7ff6f40c3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:27.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 0x7ff6f40c3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41750/0 (socket says 172.21.15.107:41750) 2023-10-24T23:55:27.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f259c640 1 -- 172.21.15.107:0/1032722733 learned_addr learned my addr 172.21.15.107:0/1032722733 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:27.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f259c640 1 -- 172.21.15.107:0/1032722733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6f41e1120 msgr2=0x7ff6f414b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:27.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f259c640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6f41e1120 0x7ff6f414b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:27.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f259c640 1 -- 172.21.15.107:0/1032722733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6f414ab80 msgr2=0x7ff6f41e0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:27.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f259c640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6f414ab80 0x7ff6f41e0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:27.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f259c640 1 -- 172.21.15.107:0/1032722733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f41ed670 con 0x7ff6e00047d0 2023-10-24T23:55:27.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f259c640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 0x7ff6f40c3940 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4011660 tx=0x7ff6e4011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:27.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6eaffd640 1 -- 172.21.15.107:0/1032722733 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e400f940 con 0x7ff6e00047d0 2023-10-24T23:55:27.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6eaffd640 1 -- 172.21.15.107:0/1032722733 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e400faa0 con 0x7ff6e00047d0 2023-10-24T23:55:27.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1032722733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f41e0100 con 0x7ff6e00047d0 2023-10-24T23:55:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6eaffd640 1 -- 172.21.15.107:0/1032722733 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e400fe10 con 0x7ff6e00047d0 2023-10-24T23:55:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1032722733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f41e06c0 con 0x7ff6e00047d0 2023-10-24T23:55:27.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1032722733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f40746e0 con 0x7ff6e00047d0 2023-10-24T23:55:27.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6eaffd640 1 -- 172.21.15.107:0/1032722733 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff6e4015460 con 0x7ff6e00047d0 2023-10-24T23:55:27.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6eaffd640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff6d4065e60 0x7ff6d4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:27.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6eaffd640 1 -- 172.21.15.107:0/1032722733 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7ff6e4084460 con 0x7ff6e00047d0 2023-10-24T23:55:27.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.227+0000 7ff6ebfff640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff6d4065e60 0x7ff6d4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:27.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.231+0000 7ff6ebfff640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff6d4065e60 0x7ff6d4068320 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0ab000 tx=0x7ff6dc001a10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:27.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.235+0000 7ff6eaffd640 1 -- 172.21.15.107:0/1032722733 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff6e40517f0 con 0x7ff6e00047d0 2023-10-24T23:55:27.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.451+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1032722733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f40c1000 con 0x7ff6e00047d0 2023-10-24T23:55:27.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.451+0000 7ff6eaffd640 1 -- 172.21.15.107:0/1032722733 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5015 (secure 0 0 0) 0x7ff6e4054e10 con 0x7ff6e00047d0 2023-10-24T23:55:27.457 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:27.457 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:26.574615+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:clientreplay","state_seq":19,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:27.457 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 21 2023-10-24T23:55:27.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1032722733 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff6d4065e60 msgr2=0x7ff6d4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:27.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff6d4065e60 0x7ff6d4068320 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0ab000 tx=0x7ff6dc001a10 comp rx=0 tx=0).stop 2023-10-24T23:55:27.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1032722733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 msgr2=0x7ff6f40c3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:27.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 0x7ff6f40c3940 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4011660 tx=0x7ff6e4011b20 comp rx=0 tx=0).stop 2023-10-24T23:55:27.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1032722733 shutdown_connections 2023-10-24T23:55:27.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff6d4065e60 0x7ff6d4068320 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:27.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6f41e1120 0x7ff6f414b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:27.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6f414ab80 0x7ff6f41e0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:27.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 --2- 172.21.15.107:0/1032722733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6e00047d0 0x7ff6f40c3940 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:27.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1032722733 >> 172.21.15.107:0/1032722733 conn(0x7ff6f40ca200 msgr2=0x7ff6f414dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:27.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1032722733 shutdown_connections 2023-10-24T23:55:27.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:27.455+0000 7ff6f8b3c640 1 -- 172.21.15.107:0/1032722733 wait complete. 2023-10-24T23:55:27.478 INFO:tasks.mds_thrash.fs.[b]:no change 2023-10-24T23:55:28.091 INFO:tasks.workunit.client.1.smithi110.stdout:checking for strings.h... yes 2023-10-24T23:55:28.156 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/stat.h... yes 2023-10-24T23:55:28.211 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/types.h... yes 2023-10-24T23:55:28.268 INFO:tasks.workunit.client.1.smithi110.stdout:checking for unistd.h... yes 2023-10-24T23:55:28.346 INFO:tasks.workunit.client.1.smithi110.stdout:checking for wchar.h... yes 2023-10-24T23:55:28.414 INFO:tasks.workunit.client.1.smithi110.stdout:checking for minix/config.h... no 2023-10-24T23:55:28.465 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/time.h... yes 2023-10-24T23:55:28.526 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2023-10-24T23:55:28.570 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether _XOPEN_SOURCE should be defined... no 2023-10-24T23:55:28.593 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gcc... (cached) gcc 2023-10-24T23:55:28.661 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether the compiler supports GNU C... (cached) yes 2023-10-24T23:55:28.662 INFO:tasks.workunit.client.1.smithi110.stdout:checking whether gcc accepts -g... (cached) yes 2023-10-24T23:55:28.662 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gcc option to enable C11 features... (cached) none needed 2023-10-24T23:55:28.663 INFO:tasks.workunit.client.1.smithi110.stdout:checking dependency style of gcc... (cached) gcc3 2023-10-24T23:55:28.743 INFO:tasks.workunit.client.1.smithi110.stdout:checking for main in -lm... yes 2023-10-24T23:55:28.822 INFO:tasks.workunit.client.1.smithi110.stdout:checking for main in -lpthread... yes 2023-10-24T23:55:28.828 INFO:tasks.workunit.client.1.smithi110.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2023-10-24T23:55:28.830 INFO:tasks.workunit.client.1.smithi110.stdout:checking for egrep... /usr/bin/grep -E 2023-10-24T23:55:28.880 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2023-10-24T23:55:28.940 INFO:tasks.workunit.client.1.smithi110.stdout:checking for pthread.h... yes 2023-10-24T23:55:28.995 INFO:tasks.workunit.client.1.smithi110.stdout:checking for fcntl.h... yes 2023-10-24T23:55:29.051 INFO:tasks.workunit.client.1.smithi110.stdout:checking for limits.h... yes 2023-10-24T23:55:29.054 INFO:tasks.workunit.client.1.smithi110.stdout:checking for stdint.h... (cached) yes 2023-10-24T23:55:29.055 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/time.h... (cached) yes 2023-10-24T23:55:29.059 INFO:tasks.workunit.client.1.smithi110.stdout:checking for unistd.h... (cached) yes 2023-10-24T23:55:29.108 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/vfs.h... yes 2023-10-24T23:55:29.171 INFO:tasks.workunit.client.1.smithi110.stdout:checking for sys/limits.h... no 2023-10-24T23:55:29.204 INFO:tasks.workunit.client.1.smithi110.stdout:checking for an ANSI C-conforming const... yes 2023-10-24T23:55:29.281 INFO:tasks.workunit.client.1.smithi110.stdout:checking for system... yes 2023-10-24T23:55:29.351 INFO:tasks.workunit.client.1.smithi110.stdout:checking for gettimeofday... yes 2023-10-24T23:55:29.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:29.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d83e54640 1 Processor -- start 2023-10-24T23:55:29.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d83e54640 1 -- start start 2023-10-24T23:55:29.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d83e54640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d7c0c8100 0x7f8d7c0abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:29.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d83e54640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8d7c06a4b0 0x7f8d7c0ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:29.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d83e54640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c155680 con 0x7f8d7c152040 2023-10-24T23:55:29.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d83e54640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7c154a70 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d83e54640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c154d50 con 0x7f8d7c0c8100 2023-10-24T23:55:29.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d823f1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8d7c06a4b0 0x7f8d7c0ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:29.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d823f1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8d7c06a4b0 0x7f8d7c0ac240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41758/0 (socket says 172.21.15.107:41758) 2023-10-24T23:55:29.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d823f1640 1 -- 172.21.15.107:0/232323081 learned_addr learned my addr 172.21.15.107:0/232323081 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:29.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d813ef640 1 --2- 172.21.15.107:0/232323081 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d7c0c8100 0x7f8d7c0abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:29.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d823f1640 1 -- 172.21.15.107:0/232323081 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d7c0c8100 msgr2=0x7f8d7c0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:29.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d823f1640 1 --2- 172.21.15.107:0/232323081 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d7c0c8100 0x7f8d7c0abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:29.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d823f1640 1 -- 172.21.15.107:0/232323081 >> v1:172.21.15.107:6789/0 conn(0x7f8d7c152040 legacy=0x7f8d7c0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:29.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d823f1640 1 -- 172.21.15.107:0/232323081 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c155030 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.595+0000 7f8d80bee640 1 -- 172.21.15.107:0/232323081 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539631689 0 0) 0x7f8d7c155680 con 0x7f8d7c152040 2023-10-24T23:55:29.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d823f1640 1 --2- 172.21.15.107:0/232323081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d7c06a4b0 0x7f8d7c0ac240 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8d74009e50 tx=0x7f8d7401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f530e8d0e7b687 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:29.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d80bee640 1 -- 172.21.15.107:0/232323081 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d740a5660 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d80bee640 1 -- 172.21.15.107:0/232323081 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d740a57c0 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d80bee640 1 -- 172.21.15.107:0/232323081 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d740a5b50 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 -- 172.21.15.107:0/232323081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d7c06a4b0 msgr2=0x7f8d7c0ac240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:29.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 --2- 172.21.15.107:0/232323081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d7c06a4b0 0x7f8d7c0ac240 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8d74009e50 tx=0x7f8d7401c9a0 comp rx=0 tx=0).stop 2023-10-24T23:55:29.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 -- 172.21.15.107:0/232323081 shutdown_connections 2023-10-24T23:55:29.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 --2- 172.21.15.107:0/232323081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d7c06a4b0 0x7f8d7c0ac240 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:29.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 --2- 172.21.15.107:0/232323081 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d7c0c8100 0x7f8d7c0abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:29.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 -- 172.21.15.107:0/232323081 >> 172.21.15.107:0/232323081 conn(0x7f8d7c0ca200 msgr2=0x7f8d7c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:29.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 -- 172.21.15.107:0/232323081 shutdown_connections 2023-10-24T23:55:29.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 -- 172.21.15.107:0/232323081 wait complete. 2023-10-24T23:55:29.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 Processor -- start 2023-10-24T23:55:29.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 -- start start 2023-10-24T23:55:29.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d7c06a4b0 0x7f8d7c0c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:29.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d7c0c8100 0x7f8d7c14ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:29.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d7c152040 0x7f8d7c1f08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:29.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c1ee070 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c0c35e0 con 0x7f8d7c152040 2023-10-24T23:55:29.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d83e54640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c148130 con 0x7f8d7c0c8100 2023-10-24T23:55:29.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d81bf0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d7c06a4b0 0x7f8d7c0c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:29.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d81bf0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d7c06a4b0 0x7f8d7c0c30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41760/0 (socket says 172.21.15.107:41760) 2023-10-24T23:55:29.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d81bf0640 1 -- 172.21.15.107:0/1919578279 learned_addr learned my addr 172.21.15.107:0/1919578279 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:29.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d813ef640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d7c0c8100 0x7f8d7c14ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:29.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d823f1640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d7c152040 0x7f8d7c1f08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:29.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d81bf0640 1 -- 172.21.15.107:0/1919578279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d7c0c8100 msgr2=0x7f8d7c14ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:29.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d81bf0640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d7c0c8100 0x7f8d7c14ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:29.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d81bf0640 1 -- 172.21.15.107:0/1919578279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d7c152040 msgr2=0x7f8d7c1f08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:29.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d81bf0640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d7c152040 0x7f8d7c1f08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:29.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d81bf0640 1 -- 172.21.15.107:0/1919578279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d74009b30 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d81bf0640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d7c06a4b0 0x7f8d7c0c30a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800c4d0 tx=0x7f8d7800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:29.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.599+0000 7f8d727fc640 1 -- 172.21.15.107:0/1919578279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78005480 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.603+0000 7f8d727fc640 1 -- 172.21.15.107:0/1919578279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d78003600 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.603+0000 7f8d83e54640 1 -- 172.21.15.107:0/1919578279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7c1ed130 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.603+0000 7f8d727fc640 1 -- 172.21.15.107:0/1919578279 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d780039b0 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.603+0000 7f8d83e54640 1 -- 172.21.15.107:0/1919578279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7c1ed6d0 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.603+0000 7f8d83e54640 1 -- 172.21.15.107:0/1919578279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7c1f0700 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.603+0000 7f8d727fc640 1 -- 172.21.15.107:0/1919578279 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8d78003b10 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.603+0000 7f8d727fc640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8d60065e90 0x7f8d60068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:29.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.603+0000 7f8d727fc640 1 -- 172.21.15.107:0/1919578279 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f8d78083260 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.603+0000 7f8d813ef640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8d60065e90 0x7f8d60068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:29.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.603+0000 7f8d813ef640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8d60065e90 0x7f8d60068350 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c149610 tx=0x7f8d6c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:29.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.607+0000 7f8d727fc640 1 -- 172.21.15.107:0/1919578279 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8d78050530 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.803+0000 7f8d83e54640 1 -- 172.21.15.107:0/1919578279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d7c0746e0 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.803+0000 7f8d727fc640 1 -- 172.21.15.107:0/1919578279 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5009 (secure 0 0 0) 0x7f8d78053b50 con 0x7f8d7c06a4b0 2023-10-24T23:55:29.810 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:29.810 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 22 2023-10-24T23:55:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 -- 172.21.15.107:0/1919578279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8d60065e90 msgr2=0x7f8d60068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8d60065e90 0x7f8d60068350 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c149610 tx=0x7f8d6c008040 comp rx=0 tx=0).stop 2023-10-24T23:55:29.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 -- 172.21.15.107:0/1919578279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d7c06a4b0 msgr2=0x7f8d7c0c30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:29.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d7c06a4b0 0x7f8d7c0c30a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800c4d0 tx=0x7f8d7800c990 comp rx=0 tx=0).stop 2023-10-24T23:55:29.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 -- 172.21.15.107:0/1919578279 shutdown_connections 2023-10-24T23:55:29.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d7c152040 0x7f8d7c1f08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:29.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8d60065e90 0x7f8d60068350 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:29.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d7c0c8100 0x7f8d7c14ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:29.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 --2- 172.21.15.107:0/1919578279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d7c06a4b0 0x7f8d7c0c30a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:29.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 -- 172.21.15.107:0/1919578279 >> 172.21.15.107:0/1919578279 conn(0x7f8d7c0ca200 msgr2=0x7f8d7c0c91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:29.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 -- 172.21.15.107:0/1919578279 shutdown_connections 2023-10-24T23:55:29.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:29.807+0000 7f8d83e54640 1 -- 172.21.15.107:0/1919578279 wait complete. 2023-10-24T23:55:29.830 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=4290 2023-10-24T23:55:29.830 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.c 2023-10-24T23:55:29.941 INFO:tasks.workunit.client.1.smithi110.stdout:checking for mkdir... yes 2023-10-24T23:55:30.043 INFO:tasks.workunit.client.1.smithi110.stdout:checking for strerror... yes 2023-10-24T23:55:30.124 INFO:tasks.workunit.client.1.smithi110.stdout:checking for open64... yes 2023-10-24T23:55:30.199 INFO:tasks.workunit.client.1.smithi110.stdout:checking for stat64... yes 2023-10-24T23:55:30.274 INFO:tasks.workunit.client.1.smithi110.stdout:checking for fseeko64... yes 2023-10-24T23:55:30.352 INFO:tasks.workunit.client.1.smithi110.stdout:checking for lrand48_r... yes 2023-10-24T23:55:30.650 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-24T23:55:30.647+0000 7faaa581f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:55:30.956 INFO:tasks.workunit.client.1.smithi110.stdout:checking for srand48_r... yes 2023-10-24T23:55:30.978 INFO:tasks.workunit.client.1.smithi110.stdout:configure: creating ./config.status 2023-10-24T23:55:31.094 INFO:tasks.workunit.client.1.smithi110.stdout: /bin/bash ./config.status 2023-10-24T23:55:31.160 INFO:tasks.workunit.client.1.smithi110.stdout:config.status: creating Makefile 2023-10-24T23:55:31.179 INFO:tasks.workunit.client.1.smithi110.stdout:config.status: creating config.h 2023-10-24T23:55:31.210 INFO:tasks.workunit.client.1.smithi110.stdout:config.status: executing depfiles commands 2023-10-24T23:55:31.239 INFO:tasks.workunit.client.1.smithi110.stdout:config.status: executing default commands 2023-10-24T23:55:31.823 INFO:tasks.workunit.client.1.smithi110.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run autoheader 2023-10-24T23:55:31.906 INFO:tasks.workunit.client.1.smithi110.stderr:autoheader: warning: autoconf input should be named 'configure.ac', not 'configure.in' 2023-10-24T23:55:32.004 INFO:tasks.workunit.client.1.smithi110.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2023-10-24T23:55:32.005 INFO:tasks.workunit.client.1.smithi110.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2023-10-24T23:55:32.005 INFO:tasks.workunit.client.1.smithi110.stderr:If you have problems, you may need to regenerate the build system entirely. 2023-10-24T23:55:32.005 INFO:tasks.workunit.client.1.smithi110.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2023-10-24T23:55:32.155 INFO:tasks.workunit.client.1.smithi110.stdout:rm -f stamp-h1 2023-10-24T23:55:32.162 INFO:tasks.workunit.client.1.smithi110.stdout:touch config.h.in 2023-10-24T23:55:32.170 INFO:tasks.workunit.client.1.smithi110.stdout:cd . && /bin/bash ./config.status config.h 2023-10-24T23:55:32.243 INFO:tasks.workunit.client.1.smithi110.stdout:config.status: creating config.h 2023-10-24T23:55:32.287 INFO:tasks.workunit.client.1.smithi110.stdout:make all-am 2023-10-24T23:55:32.317 INFO:tasks.workunit.client.1.smithi110.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb' 2023-10-24T23:55:32.318 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2023-10-24T23:55:32.560 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2023-10-24T23:55:32.571 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2023-10-24T23:55:33.831 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2023-10-24T23:55:33.831 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-24T23:55:33.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-24T23:55:33.834 INFO:tasks.ceph.mds.c:Started 2023-10-24T23:55:33.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:33.912 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-24T23:55:33.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1c0193d640 1 Processor -- start 2023-10-24T23:55:33.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1c0193d640 1 -- start start 2023-10-24T23:55:33.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1c0193d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1bfc14c850 0x7f1bfc153240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:33.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1c0193d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bfc155cd0 con 0x7f1bfc14c470 2023-10-24T23:55:33.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1c0193d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bfc0c4380 con 0x7f1bfc06f490 2023-10-24T23:55:33.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1c0193d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bfc14cc60 con 0x7f1bfc14c850 2023-10-24T23:55:33.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1bfc14c470 0x7f1bfc071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35478/0 (socket says 172.21.15.107:35478) 2023-10-24T23:55:33.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfaffd640 1 -- 172.21.15.107:0/2241076565 learned_addr learned my addr 172.21.15.107:0/2241076565 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:33.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfb7fe640 1 --2- 172.21.15.107:0/2241076565 >> v2:172.21.15.110:3300/0 conn(0x7f1bfc14c850 0x7f1bfc153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:33.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397118762 0 0) 0x7f1bfc155cd0 con 0x7f1bfc14c470 2023-10-24T23:55:33.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be8003530 con 0x7f1bfc14c470 2023-10-24T23:55:33.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1678027246 0 0) 0x7f1be8003530 con 0x7f1bfc14c470 2023-10-24T23:55:33.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 >> v2:172.21.15.110:3300/0 conn(0x7f1bfc14c850 msgr2=0x7f1bfc153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:33.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 --2- 172.21.15.107:0/2241076565 >> v2:172.21.15.110:3300/0 conn(0x7f1bfc14c850 0x7f1bfc153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:33.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 >> v1:172.21.15.110:6789/0 conn(0x7f1bfc06f490 legacy=0x7f1bfc152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:33.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc0c48b0 con 0x7f1bfc14c470 2023-10-24T23:55:33.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f1bec001680 con 0x7f1bfc14c470 2023-10-24T23:55:33.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 >> v1:172.21.15.107:6789/0 conn(0x7f1bfc14c470 legacy=0x7f1bfc071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:33.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 --2- 172.21.15.107:0/2241076565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1be8003980 0x7f1be8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:33.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 --2- 172.21.15.107:0/2241076565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1be80047d0 0x7f1be8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:33.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 --2- 172.21.15.107:0/2241076565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be80052b0 0x7f1be800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:33.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc14cc60 con 0x7f1be80052b0 2023-10-24T23:55:33.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be800c180 con 0x7f1be8003980 2023-10-24T23:55:33.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be800c340 con 0x7f1be80047d0 2023-10-24T23:55:33.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfb7fe640 1 --2- 172.21.15.107:0/2241076565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be80052b0 0x7f1be800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfb7fe640 1 -- 172.21.15.107:0/2241076565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1be80047d0 msgr2=0x7f1be8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfb7fe640 1 --2- 172.21.15.107:0/2241076565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1be80047d0 0x7f1be8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfb7fe640 1 -- 172.21.15.107:0/2241076565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1be8003980 msgr2=0x7f1be8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfb7fe640 1 --2- 172.21.15.107:0/2241076565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1be8003980 0x7f1be8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.983+0000 7f1bfb7fe640 1 -- 172.21.15.107:0/2241076565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be8003530 con 0x7f1be80052b0 2023-10-24T23:55:33.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1bfb7fe640 1 -- 172.21.15.107:0/2241076565 reap_dead start 2023-10-24T23:55:33.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1bfb7fe640 1 --2- 172.21.15.107:0/2241076565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be80052b0 0x7f1be800b990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009f90 tx=0x7f1bf401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=468d694c41c3a2ef server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:33.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4003ba0 con 0x7f1be80052b0 2023-10-24T23:55:33.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf400f040 con 0x7f1be80052b0 2023-10-24T23:55:33.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1bfa7fc640 1 -- 172.21.15.107:0/2241076565 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40a5850 con 0x7f1be80052b0 2023-10-24T23:55:33.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1c0193d640 1 -- 172.21.15.107:0/2241076565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be80052b0 msgr2=0x7f1be800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:33.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1c0193d640 1 --2- 172.21.15.107:0/2241076565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be80052b0 0x7f1be800b990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009f90 tx=0x7f1bf401cc60 comp rx=0 tx=0).stop 2023-10-24T23:55:33.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1c0193d640 1 -- 172.21.15.107:0/2241076565 shutdown_connections 2023-10-24T23:55:33.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1c0193d640 1 --2- 172.21.15.107:0/2241076565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1be80052b0 0x7f1be800b990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:33.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1c0193d640 1 -- 172.21.15.107:0/2241076565 >> 172.21.15.107:0/2241076565 conn(0x7f1bfc070f60 msgr2=0x7f1bfc071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:33.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1c0193d640 1 -- 172.21.15.107:0/2241076565 shutdown_connections 2023-10-24T23:55:34.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.987+0000 7f1c0193d640 1 -- 172.21.15.107:0/2241076565 wait complete. 2023-10-24T23:55:34.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1c0193d640 1 Processor -- start 2023-10-24T23:55:34.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1c0193d640 1 -- start start 2023-10-24T23:55:34.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1c0193d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1be80052b0 0x7f1bfc1ed8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:34.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1c0193d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bfc0ab5d0 0x7f1bfc06e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:34.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1c0193d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bfc1551c0 0x7f1bfc142660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:34.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1c0193d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc154da0 con 0x7f1bfc1551c0 2023-10-24T23:55:34.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1c0193d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc14fc70 con 0x7f1bfc0ab5d0 2023-10-24T23:55:34.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1c0193d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc143b10 con 0x7f1be80052b0 2023-10-24T23:55:34.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bfb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bfc1551c0 0x7f1bfc142660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:34.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bfb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bfc1551c0 0x7f1bfc142660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56334/0 (socket says 172.21.15.107:56334) 2023-10-24T23:55:34.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bfb7fe640 1 -- 172.21.15.107:0/3637294756 learned_addr learned my addr 172.21.15.107:0/3637294756 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:34.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bfaffd640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1be80052b0 0x7f1bfc1ed8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:34.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bf27fc640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bfc0ab5d0 0x7f1bfc06e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:34.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bfb7fe640 1 -- 172.21.15.107:0/3637294756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1be80052b0 msgr2=0x7f1bfc1ed8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:34.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bfb7fe640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1be80052b0 0x7f1bfc1ed8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:34.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bfb7fe640 1 -- 172.21.15.107:0/3637294756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bfc0ab5d0 msgr2=0x7f1bfc06e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:34.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bfb7fe640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bfc0ab5d0 0x7f1bfc06e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:34.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bfb7fe640 1 -- 172.21.15.107:0/3637294756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc0ac830 con 0x7f1bfc1551c0 2023-10-24T23:55:34.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bf27fc640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bfc0ab5d0 0x7f1bfc06e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:34.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bfb7fe640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bfc1551c0 0x7f1bfc142660 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1bf401c6d0 tx=0x7f1bf4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:34.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bf8ff9640 1 -- 172.21.15.107:0/3637294756 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40a5b10 con 0x7f1bfc1551c0 2023-10-24T23:55:34.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bf8ff9640 1 -- 172.21.15.107:0/3637294756 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf40a5c70 con 0x7f1bfc1551c0 2023-10-24T23:55:34.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1bf8ff9640 1 -- 172.21.15.107:0/3637294756 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4004020 con 0x7f1bfc1551c0 2023-10-24T23:55:34.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1c0193d640 1 -- 172.21.15.107:0/3637294756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc14f6b0 con 0x7f1bfc1551c0 2023-10-24T23:55:34.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1c0193d640 1 -- 172.21.15.107:0/3637294756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc151b90 con 0x7f1bfc1551c0 2023-10-24T23:55:34.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.991+0000 7f1c0193d640 1 -- 172.21.15.107:0/3637294756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc154b00 con 0x7f1bfc1551c0 2023-10-24T23:55:34.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.995+0000 7f1bf8ff9640 1 -- 172.21.15.107:0/3637294756 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1bf4004180 con 0x7f1bfc1551c0 2023-10-24T23:55:34.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.995+0000 7f1bf8ff9640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1bdc065e60 0x7f1bdc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:34.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.995+0000 7f1bf8ff9640 1 -- 172.21.15.107:0/3637294756 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f1bf411ee50 con 0x7f1bfc1551c0 2023-10-24T23:55:34.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.995+0000 7f1bfaffd640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1bdc065e60 0x7f1bdc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:34.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.995+0000 7f1bfaffd640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1bdc065e60 0x7f1bdc068320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1bec001830 tx=0x7f1bec001600 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:34.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:33.999+0000 7f1bf8ff9640 1 -- 172.21.15.107:0/3637294756 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1bf40ec300 con 0x7f1bfc1551c0 2023-10-24T23:55:34.090 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2023-10-24T23:55:34.102 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2023-10-24T23:55:34.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 -- 172.21.15.107:0/3637294756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bfc154ee0 con 0x7f1bfc1551c0 2023-10-24T23:55:34.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1bf8ff9640 1 -- 172.21.15.107:0/3637294756 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+5009 (secure 0 0 0) 0x7f1bfc154ee0 con 0x7f1bfc1551c0 2023-10-24T23:55:34.241 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:34.241 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:34.241 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 22 2023-10-24T23:55:34.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 -- 172.21.15.107:0/3637294756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1bdc065e60 msgr2=0x7f1bdc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:34.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1bdc065e60 0x7f1bdc068320 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1bec001830 tx=0x7f1bec001600 comp rx=0 tx=0).stop 2023-10-24T23:55:34.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 -- 172.21.15.107:0/3637294756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bfc1551c0 msgr2=0x7f1bfc142660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:34.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bfc1551c0 0x7f1bfc142660 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1bf401c6d0 tx=0x7f1bf4009750 comp rx=0 tx=0).stop 2023-10-24T23:55:34.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 -- 172.21.15.107:0/3637294756 shutdown_connections 2023-10-24T23:55:34.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bfc1551c0 0x7f1bfc142660 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:34.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1bfc0ab5d0 0x7f1bfc06e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:34.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1bdc065e60 0x7f1bdc068320 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:34.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 --2- 172.21.15.107:0/3637294756 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1be80052b0 0x7f1bfc1ed8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:34.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 -- 172.21.15.107:0/3637294756 >> 172.21.15.107:0/3637294756 conn(0x7f1bfc070f60 msgr2=0x7f1bfc0c3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:34.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 -- 172.21.15.107:0/3637294756 shutdown_connections 2023-10-24T23:55:34.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:34.235+0000 7f1c0193d640 1 -- 172.21.15.107:0/3637294756 wait complete. 2023-10-24T23:55:34.260 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-24T23:55:34.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:55:34.467 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:55:36.026 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/main.Tpo .deps/main.Po 2023-10-24T23:55:36.039 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2023-10-24T23:55:36.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cc1474640 1 Processor -- start 2023-10-24T23:55:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cc1474640 1 -- start start 2023-10-24T23:55:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cc1474640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4cbc0c8640 0x7f4cbc06a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cc1474640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cbc152400 con 0x7f4cbc152020 2023-10-24T23:55:36.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cc1474640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cbc1558d0 con 0x7f4cbc1553d0 2023-10-24T23:55:36.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cc1474640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cbc0c6ce0 con 0x7f4cbc0c8640 2023-10-24T23:55:36.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cbaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4cbc152020 0x7f4cbc0c4bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35494/0 (socket says 172.21.15.107:35494) 2023-10-24T23:55:36.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cbaffd640 1 -- 172.21.15.107:0/2577353660 learned_addr learned my addr 172.21.15.107:0/2577353660 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973609907 0 0) 0x7f4cbc152400 con 0x7f4cbc152020 2023-10-24T23:55:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cba7fc640 1 --2- 172.21.15.107:0/2577353660 >> v2:172.21.15.110:3300/0 conn(0x7f4cbc0c8640 0x7f4cbc06a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca4003530 con 0x7f4cbc152020 2023-10-24T23:55:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1066411232 0 0) 0x7f4ca4003530 con 0x7f4cbc152020 2023-10-24T23:55:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 >> v2:172.21.15.110:3300/0 conn(0x7f4cbc0c8640 msgr2=0x7f4cbc06a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 --2- 172.21.15.107:0/2577353660 >> v2:172.21.15.110:3300/0 conn(0x7f4cbc0c8640 0x7f4cbc06a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:36.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 >> v1:172.21.15.110:6789/0 conn(0x7f4cbc1553d0 legacy=0x7f4cbc06ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:36.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc0c6fc0 con 0x7f4cbc152020 2023-10-24T23:55:36.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f4cac001680 con 0x7f4cbc152020 2023-10-24T23:55:36.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 >> v1:172.21.15.107:6789/0 conn(0x7f4cbc152020 legacy=0x7f4cbc0c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:36.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 --2- 172.21.15.107:0/2577353660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca4003980 0x7f4ca4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:36.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 --2- 172.21.15.107:0/2577353660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca40047d0 0x7f4ca4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:36.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 --2- 172.21.15.107:0/2577353660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca4005190 0x7f4ca400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:36.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0c6ce0 con 0x7f4ca4005190 2023-10-24T23:55:36.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca400c130 con 0x7f4ca40047d0 2023-10-24T23:55:36.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca400c2f0 con 0x7f4ca4003980 2023-10-24T23:55:36.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cbb7fe640 1 --2- 172.21.15.107:0/2577353660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca4005190 0x7f4ca400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:36.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.387+0000 7f4cba7fc640 1 --2- 172.21.15.107:0/2577353660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca40047d0 0x7f4ca4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:36.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cbaffd640 1 --2- 172.21.15.107:0/2577353660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca4003980 0x7f4ca4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:36.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cbb7fe640 1 -- 172.21.15.107:0/2577353660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca4003980 msgr2=0x7f4ca4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cbb7fe640 1 --2- 172.21.15.107:0/2577353660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca4003980 0x7f4ca4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cbb7fe640 1 -- 172.21.15.107:0/2577353660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca40047d0 msgr2=0x7f4ca4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cbb7fe640 1 --2- 172.21.15.107:0/2577353660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca40047d0 0x7f4ca4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:36.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cbb7fe640 1 -- 172.21.15.107:0/2577353660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca4003530 con 0x7f4ca4005190 2023-10-24T23:55:36.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cbb7fe640 1 -- 172.21.15.107:0/2577353660 reap_dead start 2023-10-24T23:55:36.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cbb7fe640 1 --2- 172.21.15.107:0/2577353660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca4005190 0x7f4ca400b990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4001e40 tx=0x7f4cb401c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f499301f17eb92fe server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:36.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4003b80 con 0x7f4ca4005190 2023-10-24T23:55:36.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb4003ce0 con 0x7f4ca4005190 2023-10-24T23:55:36.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cb9ffb640 1 -- 172.21.15.107:0/2577353660 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb40052f0 con 0x7f4ca4005190 2023-10-24T23:55:36.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cc1474640 1 -- 172.21.15.107:0/2577353660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca4005190 msgr2=0x7f4ca400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:36.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cc1474640 1 --2- 172.21.15.107:0/2577353660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca4005190 0x7f4ca400b990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4001e40 tx=0x7f4cb401c740 comp rx=0 tx=0).stop 2023-10-24T23:55:36.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cc1474640 1 -- 172.21.15.107:0/2577353660 shutdown_connections 2023-10-24T23:55:36.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cc1474640 1 --2- 172.21.15.107:0/2577353660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca4005190 0x7f4ca400b990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:36.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cc1474640 1 -- 172.21.15.107:0/2577353660 >> 172.21.15.107:0/2577353660 conn(0x7f4cbc0abd60 msgr2=0x7f4cbc0ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:36.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cc1474640 1 -- 172.21.15.107:0/2577353660 shutdown_connections 2023-10-24T23:55:36.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.391+0000 7f4cc1474640 1 -- 172.21.15.107:0/2577353660 wait complete. 2023-10-24T23:55:36.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cc1474640 1 Processor -- start 2023-10-24T23:55:36.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cc1474640 1 -- start start 2023-10-24T23:55:36.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cc1474640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca4005190 0x7f4cbc151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:36.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cc1474640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc0bf790 0x7f4cbc14b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:36.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cc1474640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cbc0bfc10 0x7f4cbc14b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:36.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cc1474640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0c3810 con 0x7f4cbc0bf790 2023-10-24T23:55:36.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cc1474640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc1ea9d0 con 0x7f4ca4005190 2023-10-24T23:55:36.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cc1474640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc14beb0 con 0x7f4cbc0bfc10 2023-10-24T23:55:36.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cba7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc0bf790 0x7f4cbc14b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:36.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cba7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc0bf790 0x7f4cbc14b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56358/0 (socket says 172.21.15.107:56358) 2023-10-24T23:55:36.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cba7fc640 1 -- 172.21.15.107:0/1352926619 learned_addr learned my addr 172.21.15.107:0/1352926619 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:36.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cbb7fe640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cbc0bfc10 0x7f4cbc14b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:36.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cbaffd640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca4005190 0x7f4cbc151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:36.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cba7fc640 1 -- 172.21.15.107:0/1352926619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cbc0bfc10 msgr2=0x7f4cbc14b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:36.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cba7fc640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cbc0bfc10 0x7f4cbc14b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:36.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cba7fc640 1 -- 172.21.15.107:0/1352926619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca4005190 msgr2=0x7f4cbc151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:36.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cba7fc640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca4005190 0x7f4cbc151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:36.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cba7fc640 1 -- 172.21.15.107:0/1352926619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc1e9e50 con 0x7f4cbc0bf790 2023-10-24T23:55:36.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cba7fc640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc0bf790 0x7f4cbc14b430 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00115f0 tx=0x7f4cb0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:36.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4ca37fe640 1 -- 172.21.15.107:0/1352926619 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0004220 con 0x7f4cbc0bf790 2023-10-24T23:55:36.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4ca37fe640 1 -- 172.21.15.107:0/1352926619 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb0004380 con 0x7f4cbc0bf790 2023-10-24T23:55:36.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cc1474640 1 -- 172.21.15.107:0/1352926619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cbc1ea130 con 0x7f4cbc0bf790 2023-10-24T23:55:36.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4ca37fe640 1 -- 172.21.15.107:0/1352926619 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0015870 con 0x7f4cbc0bf790 2023-10-24T23:55:36.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.395+0000 7f4cc1474640 1 -- 172.21.15.107:0/1352926619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc1ea6f0 con 0x7f4cbc0bf790 2023-10-24T23:55:36.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.399+0000 7f4cc1474640 1 -- 172.21.15.107:0/1352926619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cbc0746e0 con 0x7f4cbc0bf790 2023-10-24T23:55:36.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.399+0000 7f4ca37fe640 1 -- 172.21.15.107:0/1352926619 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4cb00159d0 con 0x7f4cbc0bf790 2023-10-24T23:55:36.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.399+0000 7f4ca37fe640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca8065e90 0x7f4ca8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:36.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.399+0000 7f4ca37fe640 1 -- 172.21.15.107:0/1352926619 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f4cb0084360 con 0x7f4cbc0bf790 2023-10-24T23:55:36.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.399+0000 7f4cbaffd640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca8065e90 0x7f4ca8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:36.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.399+0000 7f4cbaffd640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca8065e90 0x7f4ca8068350 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4cac001060 tx=0x7f4cac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:36.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.403+0000 7f4ca37fe640 1 -- 172.21.15.107:0/1352926619 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4cb00516f0 con 0x7f4cbc0bf790 2023-10-24T23:55:36.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.655+0000 7f4cc1474640 1 -- 172.21.15.107:0/1352926619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cbc0c7800 con 0x7f4cbc0bf790 2023-10-24T23:55:36.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.655+0000 7f4ca37fe640 1 -- 172.21.15.107:0/1352926619 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f4cb0054d10 con 0x7f4cbc0bf790 2023-10-24T23:55:36.662 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:36.662 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:36.662 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-10-24T23:55:36.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 -- 172.21.15.107:0/1352926619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca8065e90 msgr2=0x7f4ca8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:36.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca8065e90 0x7f4ca8068350 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4cac001060 tx=0x7f4cac009040 comp rx=0 tx=0).stop 2023-10-24T23:55:36.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 -- 172.21.15.107:0/1352926619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc0bf790 msgr2=0x7f4cbc14b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:36.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc0bf790 0x7f4cbc14b430 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00115f0 tx=0x7f4cb0011ab0 comp rx=0 tx=0).stop 2023-10-24T23:55:36.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 -- 172.21.15.107:0/1352926619 shutdown_connections 2023-10-24T23:55:36.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca8065e90 0x7f4ca8068350 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:36.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cbc0bfc10 0x7f4cbc14b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:36.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc0bf790 0x7f4cbc14b430 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:36.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 --2- 172.21.15.107:0/1352926619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca4005190 0x7f4cbc151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:36.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 -- 172.21.15.107:0/1352926619 >> 172.21.15.107:0/1352926619 conn(0x7f4cbc0abd60 msgr2=0x7f4cbc14ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:36.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 -- 172.21.15.107:0/1352926619 shutdown_connections 2023-10-24T23:55:36.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:36.659+0000 7f4cc1474640 1 -- 172.21.15.107:0/1352926619 wait complete. 2023-10-24T23:55:36.682 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2023-10-24T23:55:36.683 INFO:tasks.mds_thrash.fs.[b]:waiting for 25 secs before thrashing 2023-10-24T23:55:37.094 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2023-10-24T23:55:37.111 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2023-10-24T23:55:38.310 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2023-10-24T23:55:38.323 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2023-10-24T23:55:38.555 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2023-10-24T23:55:38.568 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2023-10-24T23:55:38.712 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2023-10-24T23:55:38.724 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2023-10-24T23:55:38.880 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2023-10-24T23:55:38.891 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2023-10-24T23:55:39.218 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2023-10-24T23:55:39.229 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2023-10-24T23:55:39.533 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2023-10-24T23:55:39.546 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2023-10-24T23:55:40.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:40.317 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2023-10-24T23:55:40.327 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2023-10-24T23:55:40.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f894730e640 1 Processor -- start 2023-10-24T23:55:40.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f894730e640 1 -- start start 2023-10-24T23:55:40.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f894730e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89400ac9d0 0x7f8940158890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:40.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f894730e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8940154a10 0x7f8940154e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:40.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f894730e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89400ad0d0 con 0x7f8940152160 2023-10-24T23:55:40.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f894730e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f894006fe90 con 0x7f8940154a10 2023-10-24T23:55:40.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f894730e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8940158dd0 con 0x7f89400ac9d0 2023-10-24T23:55:40.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f89458ab640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8940154a10 0x7f8940154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:40.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f89448a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89400ac9d0 0x7f8940158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:40.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f89450aa640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8940152160 0x7f894006cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:35092/0 (socket says 172.21.15.107:35092) 2023-10-24T23:55:40.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f89450aa640 1 -- 172.21.15.107:0/1280972200 learned_addr learned my addr 172.21.15.107:0/1280972200 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:40.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f89448a9640 1 -- 172.21.15.107:0/1280972200 >> v2:172.21.15.110:3300/0 conn(0x7f8940154a10 msgr2=0x7f8940154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:40.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f89448a9640 1 --2- 172.21.15.107:0/1280972200 >> v2:172.21.15.110:3300/0 conn(0x7f8940154a10 0x7f8940154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f89448a9640 1 -- 172.21.15.107:0/1280972200 >> v1:172.21.15.110:6789/0 conn(0x7f8940152160 legacy=0x7f894006cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:40.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f89448a9640 1 -- 172.21.15.107:0/1280972200 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89400c0300 con 0x7f89400ac9d0 2023-10-24T23:55:40.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.363+0000 7f89458ab640 1 --2- 172.21.15.107:0/1280972200 >> v2:172.21.15.110:3300/0 conn(0x7f8940154a10 0x7f8940154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:55:40.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f89448a9640 1 --2- 172.21.15.107:0/1280972200 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89400ac9d0 0x7f8940158890 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f89300097c0 tx=0x7f893001c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c70ebe8e15c91bb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:40.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f8937fff640 1 -- 172.21.15.107:0/1280972200 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89300ab600 con 0x7f89400ac9d0 2023-10-24T23:55:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f8937fff640 1 -- 172.21.15.107:0/1280972200 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89300a7050 con 0x7f89400ac9d0 2023-10-24T23:55:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f8937fff640 1 -- 172.21.15.107:0/1280972200 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89300ab980 con 0x7f89400ac9d0 2023-10-24T23:55:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f894730e640 1 -- 172.21.15.107:0/1280972200 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89400ac9d0 msgr2=0x7f8940158890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f894730e640 1 --2- 172.21.15.107:0/1280972200 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89400ac9d0 0x7f8940158890 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f89300097c0 tx=0x7f893001c330 comp rx=0 tx=0).stop 2023-10-24T23:55:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f894730e640 1 -- 172.21.15.107:0/1280972200 shutdown_connections 2023-10-24T23:55:40.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f894730e640 1 --2- 172.21.15.107:0/1280972200 >> v2:172.21.15.110:3300/0 conn(0x7f8940154a10 0x7f8940154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f894730e640 1 --2- 172.21.15.107:0/1280972200 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89400ac9d0 0x7f8940158890 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f894730e640 1 -- 172.21.15.107:0/1280972200 >> 172.21.15.107:0/1280972200 conn(0x7f89400c6ed0 msgr2=0x7f89400c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:40.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f894730e640 1 -- 172.21.15.107:0/1280972200 shutdown_connections 2023-10-24T23:55:40.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f894730e640 1 -- 172.21.15.107:0/1280972200 wait complete. 2023-10-24T23:55:40.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f894730e640 1 Processor -- start 2023-10-24T23:55:40.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.367+0000 7f894730e640 1 -- start start 2023-10-24T23:55:40.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f894730e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89400ac9d0 0x7f89400c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f894730e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8940152160 0x7f894014d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f894730e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8940154a10 0x7f8940147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f894730e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894014fb50 con 0x7f89400ac9d0 2023-10-24T23:55:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f894730e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8940152780 con 0x7f8940154a10 2023-10-24T23:55:40.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f894730e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89401547f0 con 0x7f8940152160 2023-10-24T23:55:40.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89450aa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89400ac9d0 0x7f89400c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:40.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89450aa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89400ac9d0 0x7f89400c5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56370/0 (socket says 172.21.15.107:56370) 2023-10-24T23:55:40.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89450aa640 1 -- 172.21.15.107:0/318418612 learned_addr learned my addr 172.21.15.107:0/318418612 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:40.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89448a9640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8940152160 0x7f894014d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:40.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89458ab640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8940154a10 0x7f8940147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89450aa640 1 -- 172.21.15.107:0/318418612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8940152160 msgr2=0x7f894014d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89450aa640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8940152160 0x7f894014d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89450aa640 1 -- 172.21.15.107:0/318418612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8940154a10 msgr2=0x7f8940147a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89450aa640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8940154a10 0x7f8940147a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89450aa640 1 -- 172.21.15.107:0/318418612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8938009420 con 0x7f89400ac9d0 2023-10-24T23:55:40.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89450aa640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89400ac9d0 0x7f89400c5200 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f893c00cbc0 tx=0x7f893c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:40.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f8935ffb640 1 -- 172.21.15.107:0/318418612 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c00abe0 con 0x7f89400ac9d0 2023-10-24T23:55:40.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f8935ffb640 1 -- 172.21.15.107:0/318418612 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c00ad40 con 0x7f89400ac9d0 2023-10-24T23:55:40.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f894730e640 1 -- 172.21.15.107:0/318418612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8930009420 con 0x7f89400ac9d0 2023-10-24T23:55:40.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f8935ffb640 1 -- 172.21.15.107:0/318418612 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c012600 con 0x7f89400ac9d0 2023-10-24T23:55:40.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f894730e640 1 -- 172.21.15.107:0/318418612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89401542e0 con 0x7f89400ac9d0 2023-10-24T23:55:40.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f894730e640 1 -- 172.21.15.107:0/318418612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894014dc50 con 0x7f89400ac9d0 2023-10-24T23:55:40.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f8935ffb640 1 -- 172.21.15.107:0/318418612 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f893c007070 con 0x7f89400ac9d0 2023-10-24T23:55:40.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f8935ffb640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8924065e90 0x7f8924068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:40.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f89448a9640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8924065e90 0x7f8924068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.371+0000 7f8935ffb640 1 -- 172.21.15.107:0/318418612 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f893c083790 con 0x7f89400ac9d0 2023-10-24T23:55:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.375+0000 7f89448a9640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8924065e90 0x7f8924068350 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f89300ad000 tx=0x7f8930008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.379+0000 7f8935ffb640 1 -- 172.21.15.107:0/318418612 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f893c050af0 con 0x7f89400ac9d0 2023-10-24T23:55:40.520 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2023-10-24T23:55:40.534 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2023-10-24T23:55:40.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 -- 172.21.15.107:0/318418612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89400ad0d0 con 0x7f89400ac9d0 2023-10-24T23:55:40.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f8935ffb640 1 -- 172.21.15.107:0/318418612 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f89400ad0d0 con 0x7f89400ac9d0 2023-10-24T23:55:40.635 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:40.635 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:40.636 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-10-24T23:55:40.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 -- 172.21.15.107:0/318418612 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8924065e90 msgr2=0x7f8924068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:40.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8924065e90 0x7f8924068350 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f89300ad000 tx=0x7f8930008040 comp rx=0 tx=0).stop 2023-10-24T23:55:40.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 -- 172.21.15.107:0/318418612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89400ac9d0 msgr2=0x7f89400c5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:40.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89400ac9d0 0x7f89400c5200 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f893c00cbc0 tx=0x7f893c00a060 comp rx=0 tx=0).stop 2023-10-24T23:55:40.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 -- 172.21.15.107:0/318418612 shutdown_connections 2023-10-24T23:55:40.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8940154a10 0x7f8940147a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8924065e90 0x7f8924068350 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8940152160 0x7f894014d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 --2- 172.21.15.107:0/318418612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89400ac9d0 0x7f89400c5200 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 -- 172.21.15.107:0/318418612 >> 172.21.15.107:0/318418612 conn(0x7f89400c6ed0 msgr2=0x7f89400c72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:40.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 -- 172.21.15.107:0/318418612 shutdown_connections 2023-10-24T23:55:40.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.631+0000 7f894730e640 1 -- 172.21.15.107:0/318418612 wait complete. 2023-10-24T23:55:40.655 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2023-10-24T23:55:40.656 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-24T23:55:40.656 INFO:teuthology.orchestra.run:waiting for 300 2023-10-24T23:55:40.675 INFO:tasks.ceph.mds.d:Stopped 2023-10-24T23:55:40.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:40.686 INFO:tasks.workunit.client.1.smithi110.stderr:ffsb_op.c: In function 'print_op_results': 2023-10-24T23:55:40.687 INFO:tasks.workunit.client.1.smithi110.stderr:ffsb_op.c:83:37: warning: embedded '\0' in format [-Wformat-contains-nul] 2023-10-24T23:55:40.687 INFO:tasks.workunit.client.1.smithi110.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2023-10-24T23:55:40.687 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~ 2023-10-24T23:55:40.687 INFO:tasks.workunit.client.1.smithi110.stderr:ffsb_op.c:86:33: warning: embedded '\0' in format [-Wformat-contains-nul] 2023-10-24T23:55:40.687 INFO:tasks.workunit.client.1.smithi110.stderr: 86 | sprintf(buf, "NA\0"); 2023-10-24T23:55:40.688 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~ 2023-10-24T23:55:40.707 INFO:tasks.workunit.client.1.smithi110.stderr:ffsb_op.c: In function 'print_results': 2023-10-24T23:55:40.708 INFO:tasks.workunit.client.1.smithi110.stderr:ffsb_op.c:83:33: warning: '/sec' directive writing 4 bytes into a region of size between 1 and 256 [-Wformat-overflow=] 2023-10-24T23:55:40.708 INFO:tasks.workunit.client.1.smithi110.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2023-10-24T23:55:40.708 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~ 2023-10-24T23:55:40.708 INFO:tasks.workunit.client.1.smithi110.stderr:In file included from /usr/include/stdio.h:894, 2023-10-24T23:55:40.709 INFO:tasks.workunit.client.1.smithi110.stderr: from ffsb_op.c:20: 2023-10-24T23:55:40.709 INFO:tasks.workunit.client.1.smithi110.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 5 and 260 bytes into a destination of size 256 2023-10-24T23:55:40.709 INFO:tasks.workunit.client.1.smithi110.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, 2023-10-24T23:55:40.709 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:40.709 INFO:tasks.workunit.client.1.smithi110.stderr: 39 | __glibc_objsize (__s), __fmt, 2023-10-24T23:55:40.710 INFO:tasks.workunit.client.1.smithi110.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:40.710 INFO:tasks.workunit.client.1.smithi110.stderr: 40 | __va_arg_pack ()); 2023-10-24T23:55:40.710 INFO:tasks.workunit.client.1.smithi110.stderr: | ~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:40.766 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2023-10-24T23:55:40.777 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2023-10-24T23:55:40.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca9a89a640 1 Processor -- start 2023-10-24T23:55:40.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca9a89a640 1 -- start start 2023-10-24T23:55:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca9a89a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca9413bb00 0x7fca941531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca9a89a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca94152d40 con 0x7fca940a9450 2023-10-24T23:55:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca9a89a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9406d660 con 0x7fca940aa330 2023-10-24T23:55:40.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca9a89a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca94156100 con 0x7fca9413bb00 2023-10-24T23:55:40.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca98e37640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fca940a9450 0x7fca94071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35500/0 (socket says 172.21.15.107:35500) 2023-10-24T23:55:40.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca98e37640 1 -- 172.21.15.107:0/2325513075 learned_addr learned my addr 172.21.15.107:0/2325513075 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:40.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca93fff640 1 --2- 172.21.15.107:0/2325513075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca9413bb00 0x7fca941531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:40.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca937fe640 1 -- 172.21.15.107:0/2325513075 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718989043 0 0) 0x7fca94152d40 con 0x7fca940a9450 2023-10-24T23:55:40.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca937fe640 1 -- 172.21.15.107:0/2325513075 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca80003530 con 0x7fca940a9450 2023-10-24T23:55:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca937fe640 1 -- 172.21.15.107:0/2325513075 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942772180 0 0) 0x7fca9406d660 con 0x7fca940aa330 2023-10-24T23:55:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca937fe640 1 -- 172.21.15.107:0/2325513075 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca94152d40 con 0x7fca940aa330 2023-10-24T23:55:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca93fff640 1 -- 172.21.15.107:0/2325513075 >> v1:172.21.15.110:6789/0 conn(0x7fca940aa330 legacy=0x7fca9406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca93fff640 1 -- 172.21.15.107:0/2325513075 >> v1:172.21.15.107:6789/0 conn(0x7fca940a9450 legacy=0x7fca94071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:40.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.783+0000 7fca93fff640 1 -- 172.21.15.107:0/2325513075 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca94155e60 con 0x7fca9413bb00 2023-10-24T23:55:40.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca93fff640 1 --2- 172.21.15.107:0/2325513075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca9413bb00 0x7fca941531d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fca84009820 tx=0x7fca84020790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c17c3d7e817db8f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:40.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca937fe640 1 -- 172.21.15.107:0/2325513075 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca840aa450 con 0x7fca9413bb00 2023-10-24T23:55:40.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca937fe640 1 -- 172.21.15.107:0/2325513075 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca840aa5b0 con 0x7fca9413bb00 2023-10-24T23:55:40.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca937fe640 1 -- 172.21.15.107:0/2325513075 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca840aa920 con 0x7fca9413bb00 2023-10-24T23:55:40.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca9a89a640 1 -- 172.21.15.107:0/2325513075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca9413bb00 msgr2=0x7fca941531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:40.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca9a89a640 1 --2- 172.21.15.107:0/2325513075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca9413bb00 0x7fca941531d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fca84009820 tx=0x7fca84020790 comp rx=0 tx=0).stop 2023-10-24T23:55:40.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca9a89a640 1 -- 172.21.15.107:0/2325513075 shutdown_connections 2023-10-24T23:55:40.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca9a89a640 1 --2- 172.21.15.107:0/2325513075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca9413bb00 0x7fca941531d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca9a89a640 1 -- 172.21.15.107:0/2325513075 >> 172.21.15.107:0/2325513075 conn(0x7fca94070f60 msgr2=0x7fca94071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:40.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca9a89a640 1 -- 172.21.15.107:0/2325513075 shutdown_connections 2023-10-24T23:55:40.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca9a89a640 1 -- 172.21.15.107:0/2325513075 wait complete. 2023-10-24T23:55:40.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca9a89a640 1 Processor -- start 2023-10-24T23:55:40.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.787+0000 7fca9a89a640 1 -- start start 2023-10-24T23:55:40.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca9a89a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca940a9450 0x7fca940c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:40.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca9a89a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca940aa330 0x7fca940c2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:40.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca9a89a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca9413bb00 0x7fca94131880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:40.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca9a89a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca940c7e20 con 0x7fca9413bb00 2023-10-24T23:55:40.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca9a89a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca940c3370 con 0x7fca940a9450 2023-10-24T23:55:40.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca9a89a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca941ec850 con 0x7fca940aa330 2023-10-24T23:55:40.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca98e37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca9413bb00 0x7fca94131880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:40.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca98e37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca9413bb00 0x7fca94131880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56384/0 (socket says 172.21.15.107:56384) 2023-10-24T23:55:40.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca98e37640 1 -- 172.21.15.107:0/3753982344 learned_addr learned my addr 172.21.15.107:0/3753982344 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:40.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca93fff640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca940a9450 0x7fca940c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:40.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca8bfff640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca940aa330 0x7fca940c2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca98e37640 1 -- 172.21.15.107:0/3753982344 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca940aa330 msgr2=0x7fca940c2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca98e37640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca940aa330 0x7fca940c2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca98e37640 1 -- 172.21.15.107:0/3753982344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca940a9450 msgr2=0x7fca940c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca98e37640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca940a9450 0x7fca940c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:40.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca98e37640 1 -- 172.21.15.107:0/3753982344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca84009420 con 0x7fca9413bb00 2023-10-24T23:55:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca98e37640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca9413bb00 0x7fca94131880 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00cad0 tx=0x7fca8c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca917fa640 1 -- 172.21.15.107:0/3753982344 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c00ac30 con 0x7fca9413bb00 2023-10-24T23:55:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca917fa640 1 -- 172.21.15.107:0/3753982344 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c00adb0 con 0x7fca9413bb00 2023-10-24T23:55:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca9a89a640 1 -- 172.21.15.107:0/3753982344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca941ec520 con 0x7fca9413bb00 2023-10-24T23:55:40.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca917fa640 1 -- 172.21.15.107:0/3753982344 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c012640 con 0x7fca9413bb00 2023-10-24T23:55:40.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca9a89a640 1 -- 172.21.15.107:0/3753982344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca9414ff20 con 0x7fca9413bb00 2023-10-24T23:55:40.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca9a89a640 1 -- 172.21.15.107:0/3753982344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca940c4890 con 0x7fca9413bb00 2023-10-24T23:55:40.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca917fa640 1 -- 172.21.15.107:0/3753982344 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fca8c007070 con 0x7fca9413bb00 2023-10-24T23:55:40.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca917fa640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fca74065e60 0x7fca74068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:40.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca917fa640 1 -- 172.21.15.107:0/3753982344 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7fca8c083a80 con 0x7fca9413bb00 2023-10-24T23:55:40.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.791+0000 7fca93fff640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fca74065e60 0x7fca74068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:40.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.795+0000 7fca93fff640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fca74065e60 0x7fca74068320 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fca840b3000 tx=0x7fca840ab000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:40.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:40.799+0000 7fca917fa640 1 -- 172.21.15.107:0/3753982344 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fca940c4890 con 0x7fca9413bb00 2023-10-24T23:55:41.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.031+0000 7fca9a89a640 1 -- 172.21.15.107:0/3753982344 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca941ec990 con 0x7fca9413bb00 2023-10-24T23:55:41.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca917fa640 1 -- 172.21.15.107:0/3753982344 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fca941ec990 con 0x7fca9413bb00 2023-10-24T23:55:41.038 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:41.039 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:41.039 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-10-24T23:55:41.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 -- 172.21.15.107:0/3753982344 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fca74065e60 msgr2=0x7fca74068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:41.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fca74065e60 0x7fca74068320 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fca840b3000 tx=0x7fca840ab000 comp rx=0 tx=0).stop 2023-10-24T23:55:41.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 -- 172.21.15.107:0/3753982344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca9413bb00 msgr2=0x7fca94131880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:41.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca9413bb00 0x7fca94131880 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00cad0 tx=0x7fca8c00cf90 comp rx=0 tx=0).stop 2023-10-24T23:55:41.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 -- 172.21.15.107:0/3753982344 shutdown_connections 2023-10-24T23:55:41.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca9413bb00 0x7fca94131880 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:41.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fca74065e60 0x7fca74068320 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:41.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca940aa330 0x7fca940c2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:41.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 --2- 172.21.15.107:0/3753982344 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca940a9450 0x7fca940c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:41.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 -- 172.21.15.107:0/3753982344 >> 172.21.15.107:0/3753982344 conn(0x7fca94070f60 msgr2=0x7fca9406df40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:41.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 -- 172.21.15.107:0/3753982344 shutdown_connections 2023-10-24T23:55:41.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:41.035+0000 7fca9a89a640 1 -- 172.21.15.107:0/3753982344 wait complete. 2023-10-24T23:55:41.057 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:55:42.266 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/util.Tpo .deps/util.Po 2023-10-24T23:55:42.281 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2023-10-24T23:55:42.503 INFO:tasks.workunit.client.1.smithi110.stderr:In file included from parser.h:22, 2023-10-24T23:55:42.504 INFO:tasks.workunit.client.1.smithi110.stderr: from parser.c:25: 2023-10-24T23:55:42.504 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c: In function 'init_filesys': 2023-10-24T23:55:42.505 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:42.506 INFO:tasks.workunit.client.1.smithi110.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:42.506 INFO:tasks.workunit.client.1.smithi110.stderr: | ^ 2023-10-24T23:55:42.506 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:42.506 INFO:tasks.workunit.client.1.smithi110.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:42.509 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~ 2023-10-24T23:55:42.509 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:42.509 INFO:tasks.workunit.client.1.smithi110.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:42.510 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:42.510 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:42.510 INFO:tasks.workunit.client.1.smithi110.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-10-24T23:55:42.510 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:42.510 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:42.511 INFO:tasks.workunit.client.1.smithi110.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2023-10-24T23:55:42.511 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:42.512 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:42.512 INFO:tasks.workunit.client.1.smithi110.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:42.513 INFO:tasks.workunit.client.1.smithi110.stderr: | ^ 2023-10-24T23:55:42.513 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:42.513 INFO:tasks.workunit.client.1.smithi110.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:42.513 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~ 2023-10-24T23:55:42.513 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:42.514 INFO:tasks.workunit.client.1.smithi110.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:42.514 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:42.514 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:42.514 INFO:tasks.workunit.client.1.smithi110.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-10-24T23:55:42.514 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:42.515 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:42.515 INFO:tasks.workunit.client.1.smithi110.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2023-10-24T23:55:42.515 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:42.515 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:42.515 INFO:tasks.workunit.client.1.smithi110.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:42.516 INFO:tasks.workunit.client.1.smithi110.stderr: | ^ 2023-10-24T23:55:42.516 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:42.516 INFO:tasks.workunit.client.1.smithi110.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:42.516 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~ 2023-10-24T23:55:42.516 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:42.517 INFO:tasks.workunit.client.1.smithi110.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:42.517 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:42.517 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:42.517 INFO:tasks.workunit.client.1.smithi110.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-10-24T23:55:42.517 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:42.518 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:42.518 INFO:tasks.workunit.client.1.smithi110.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2023-10-24T23:55:42.518 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:42.518 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:42.518 INFO:tasks.workunit.client.1.smithi110.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:42.519 INFO:tasks.workunit.client.1.smithi110.stderr: | ^ 2023-10-24T23:55:42.519 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:42.519 INFO:tasks.workunit.client.1.smithi110.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:42.519 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~ 2023-10-24T23:55:42.519 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:42.519 INFO:tasks.workunit.client.1.smithi110.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:42.520 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:42.520 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:42.520 INFO:tasks.workunit.client.1.smithi110.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-10-24T23:55:42.520 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:42.520 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:42.521 INFO:tasks.workunit.client.1.smithi110.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2023-10-24T23:55:42.521 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:42.521 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c: In function 'init_tg_stats': 2023-10-24T23:55:42.521 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:42.521 INFO:tasks.workunit.client.1.smithi110.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:42.522 INFO:tasks.workunit.client.1.smithi110.stderr: | ^ 2023-10-24T23:55:42.522 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:42.522 INFO:tasks.workunit.client.1.smithi110.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:42.522 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~ 2023-10-24T23:55:42.522 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:42.523 INFO:tasks.workunit.client.1.smithi110.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:42.523 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:42.523 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:42.523 INFO:tasks.workunit.client.1.smithi110.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-10-24T23:55:42.523 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:42.524 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:42.524 INFO:tasks.workunit.client.1.smithi110.stderr: 971 | list_for_each_entry(tmp_list, 2023-10-24T23:55:42.524 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:42.524 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:42.524 INFO:tasks.workunit.client.1.smithi110.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:42.525 INFO:tasks.workunit.client.1.smithi110.stderr: | ^ 2023-10-24T23:55:42.525 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:42.525 INFO:tasks.workunit.client.1.smithi110.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:42.525 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~ 2023-10-24T23:55:42.525 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:42.526 INFO:tasks.workunit.client.1.smithi110.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:42.526 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:42.526 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:42.526 INFO:tasks.workunit.client.1.smithi110.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-10-24T23:55:42.526 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:42.527 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:42.527 INFO:tasks.workunit.client.1.smithi110.stderr: 971 | list_for_each_entry(tmp_list, 2023-10-24T23:55:42.527 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:42.527 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:42.527 INFO:tasks.workunit.client.1.smithi110.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:42.528 INFO:tasks.workunit.client.1.smithi110.stderr: | ^ 2023-10-24T23:55:42.528 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:42.528 INFO:tasks.workunit.client.1.smithi110.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:42.528 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~ 2023-10-24T23:55:42.528 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:42.529 INFO:tasks.workunit.client.1.smithi110.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:42.529 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:42.529 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:42.529 INFO:tasks.workunit.client.1.smithi110.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2023-10-24T23:55:42.529 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:42.529 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:42.530 INFO:tasks.workunit.client.1.smithi110.stderr: 980 | list_for_each_entry(tmp_list, 2023-10-24T23:55:42.530 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:42.530 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2023-10-24T23:55:42.530 INFO:tasks.workunit.client.1.smithi110.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2023-10-24T23:55:42.530 INFO:tasks.workunit.client.1.smithi110.stderr: | ^ 2023-10-24T23:55:42.531 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2023-10-24T23:55:42.531 INFO:tasks.workunit.client.1.smithi110.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2023-10-24T23:55:42.531 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~ 2023-10-24T23:55:42.531 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2023-10-24T23:55:42.531 INFO:tasks.workunit.client.1.smithi110.stderr: 36 | container_of(ptr, type, member) 2023-10-24T23:55:42.532 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~ 2023-10-24T23:55:42.532 INFO:tasks.workunit.client.1.smithi110.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2023-10-24T23:55:42.532 INFO:tasks.workunit.client.1.smithi110.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2023-10-24T23:55:42.532 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~ 2023-10-24T23:55:42.532 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2023-10-24T23:55:42.533 INFO:tasks.workunit.client.1.smithi110.stderr: 980 | list_for_each_entry(tmp_list, 2023-10-24T23:55:42.533 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:42.603 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c: In function 'get_optstr': 2023-10-24T23:55:42.603 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c:206:23: warning: 'strnlen' specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2023-10-24T23:55:42.603 INFO:tasks.workunit.client.1.smithi110.stderr: 206 | len = strnlen(temp, 4095) + 1; 2023-10-24T23:55:42.603 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:42.603 INFO:tasks.workunit.client.1.smithi110.stderr:parser.c:200:14: note: source object declared here 2023-10-24T23:55:42.604 INFO:tasks.workunit.client.1.smithi110.stderr: 200 | char temp[BUFSIZE]; 2023-10-24T23:55:42.604 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~ 2023-10-24T23:55:42.621 INFO:tasks.workunit.client.1.smithi110.stderr:In file included from /usr/include/string.h:535, 2023-10-24T23:55:42.621 INFO:tasks.workunit.client.1.smithi110.stderr: from parser.c:19: 2023-10-24T23:55:42.622 INFO:tasks.workunit.client.1.smithi110.stderr:In function 'memcpy', 2023-10-24T23:55:42.622 INFO:tasks.workunit.client.1.smithi110.stderr: inlined from 'size64_convert' at parser.c:119:2: 2023-10-24T23:55:42.622 INFO:tasks.workunit.client.1.smithi110.stderr:/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10: warning: '__builtin_memcpy' reading 3 bytes from a region of size 2 [-Wstringop-overread] 2023-10-24T23:55:42.622 INFO:tasks.workunit.client.1.smithi110.stderr: 29 | return __builtin___memcpy_chk (__dest, __src, __len, 2023-10-24T23:55:42.622 INFO:tasks.workunit.client.1.smithi110.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:42.623 INFO:tasks.workunit.client.1.smithi110.stderr: 30 | __glibc_objsize0 (__dest)); 2023-10-24T23:55:42.623 INFO:tasks.workunit.client.1.smithi110.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 2023-10-24T23:55:42.971 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2023-10-24T23:55:42.986 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2023-10-24T23:55:43.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:43.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.187+0000 7f2a23c32640 1 Processor -- start 2023-10-24T23:55:43.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.187+0000 7f2a23c32640 1 -- start start 2023-10-24T23:55:43.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a23c32640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a1c071220 0x7f2a1c06bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:43.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a23c32640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2a1c151960 0x7f2a1c071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:43.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a23c32640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c150bc0 con 0x7f2a1c0c24c0 2023-10-24T23:55:43.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a23c32640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1c06c7f0 con 0x7f2a1c151960 2023-10-24T23:55:43.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a23c32640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c06c510 con 0x7f2a1c071220 2023-10-24T23:55:43.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a211cd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2a1c151960 0x7f2a1c071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:43.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a211cd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2a1c151960 0x7f2a1c071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38876/0 (socket says 172.21.15.107:38876) 2023-10-24T23:55:43.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a211cd640 1 -- 172.21.15.107:0/1366253866 learned_addr learned my addr 172.21.15.107:0/1366253866 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:43.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a219ce640 1 --2- 172.21.15.107:0/1366253866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a1c071220 0x7f2a1c06bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:43.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a211cd640 1 -- 172.21.15.107:0/1366253866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a1c071220 msgr2=0x7f2a1c06bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:43.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a211cd640 1 --2- 172.21.15.107:0/1366253866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a1c071220 0x7f2a1c06bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:43.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a211cd640 1 -- 172.21.15.107:0/1366253866 >> v1:172.21.15.110:6789/0 conn(0x7f2a1c0c24c0 legacy=0x7f2a1c06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:43.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a211cd640 1 -- 172.21.15.107:0/1366253866 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c150e80 con 0x7f2a1c151960 2023-10-24T23:55:43.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a211cd640 1 --2- 172.21.15.107:0/1366253866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a1c151960 0x7f2a1c071a90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009870 tx=0x7f2a1801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2146413891e3fb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:43.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a209cc640 1 -- 172.21.15.107:0/1366253866 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a180ab600 con 0x7f2a1c151960 2023-10-24T23:55:43.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a209cc640 1 -- 172.21.15.107:0/1366253866 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a180a7050 con 0x7f2a1c151960 2023-10-24T23:55:43.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.191+0000 7f2a209cc640 1 -- 172.21.15.107:0/1366253866 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a180ab980 con 0x7f2a1c151960 2023-10-24T23:55:43.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 -- 172.21.15.107:0/1366253866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a1c151960 msgr2=0x7f2a1c071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:43.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 --2- 172.21.15.107:0/1366253866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a1c151960 0x7f2a1c071a90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009870 tx=0x7f2a1801c470 comp rx=0 tx=0).stop 2023-10-24T23:55:43.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 -- 172.21.15.107:0/1366253866 shutdown_connections 2023-10-24T23:55:43.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 --2- 172.21.15.107:0/1366253866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a1c151960 0x7f2a1c071a90 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:43.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 --2- 172.21.15.107:0/1366253866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a1c071220 0x7f2a1c06bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:43.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 -- 172.21.15.107:0/1366253866 >> 172.21.15.107:0/1366253866 conn(0x7f2a1c0c2950 msgr2=0x7f2a1c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:43.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 -- 172.21.15.107:0/1366253866 shutdown_connections 2023-10-24T23:55:43.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 -- 172.21.15.107:0/1366253866 wait complete. 2023-10-24T23:55:43.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 Processor -- start 2023-10-24T23:55:43.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 -- start start 2023-10-24T23:55:43.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a1c071220 0x7f2a1c154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:43.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a1c0c24c0 0x7f2a1c1ecc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:43.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a1c151960 0x7f2a1c069fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:43.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c1ed170 con 0x7f2a1c151960 2023-10-24T23:55:43.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c0c4640 con 0x7f2a1c0c24c0 2023-10-24T23:55:43.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c154cb0 con 0x7f2a1c071220 2023-10-24T23:55:43.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a221cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a1c151960 0x7f2a1c069fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:43.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a221cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a1c151960 0x7f2a1c069fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38880/0 (socket says 172.21.15.107:38880) 2023-10-24T23:55:43.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a221cf640 1 -- 172.21.15.107:0/2864259380 learned_addr learned my addr 172.21.15.107:0/2864259380 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:43.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a211cd640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a1c0c24c0 0x7f2a1c1ecc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:43.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a219ce640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a1c071220 0x7f2a1c154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:43.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a221cf640 1 -- 172.21.15.107:0/2864259380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a1c071220 msgr2=0x7f2a1c154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:43.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a221cf640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a1c071220 0x7f2a1c154660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:43.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a221cf640 1 -- 172.21.15.107:0/2864259380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a1c0c24c0 msgr2=0x7f2a1c1ecc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:43.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a221cf640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a1c0c24c0 0x7f2a1c1ecc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:43.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a221cf640 1 -- 172.21.15.107:0/2864259380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a180094d0 con 0x7f2a1c151960 2023-10-24T23:55:43.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a219ce640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a1c071220 0x7f2a1c154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:43.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a221cf640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a1c151960 0x7f2a1c069fa0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400c780 tx=0x7f2a1400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:43.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a127fc640 1 -- 172.21.15.107:0/2864259380 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14004b70 con 0x7f2a1c151960 2023-10-24T23:55:43.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a127fc640 1 -- 172.21.15.107:0/2864259380 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1400ce90 con 0x7f2a1c151960 2023-10-24T23:55:43.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 -- 172.21.15.107:0/2864259380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1c0c4900 con 0x7f2a1c151960 2023-10-24T23:55:43.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a127fc640 1 -- 172.21.15.107:0/2864259380 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1400a850 con 0x7f2a1c151960 2023-10-24T23:55:43.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.195+0000 7f2a23c32640 1 -- 172.21.15.107:0/2864259380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c0c4f30 con 0x7f2a1c151960 2023-10-24T23:55:43.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.199+0000 7f2a127fc640 1 -- 172.21.15.107:0/2864259380 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2a1400a9b0 con 0x7f2a1c151960 2023-10-24T23:55:43.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.199+0000 7f2a23c32640 1 -- 172.21.15.107:0/2864259380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a1c154030 con 0x7f2a1c151960 2023-10-24T23:55:43.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.199+0000 7f2a127fc640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a00065e60 0x7f2a00068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:43.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.199+0000 7f2a219ce640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a00065e60 0x7f2a00068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:43.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.199+0000 7f2a127fc640 1 -- 172.21.15.107:0/2864259380 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f2a14083da0 con 0x7f2a1c151960 2023-10-24T23:55:43.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.199+0000 7f2a219ce640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a00065e60 0x7f2a00068320 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c006fd0 tx=0x7f2a0c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:43.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.203+0000 7f2a127fc640 1 -- 172.21.15.107:0/2864259380 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2a1c154030 con 0x7f2a1c151960 2023-10-24T23:55:43.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.391+0000 7f2a23c32640 1 -- 172.21.15.107:0/2864259380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a1c154030 con 0x7f2a1c151960 2023-10-24T23:55:43.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a127fc640 1 -- 172.21.15.107:0/2864259380 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f2a1c154030 con 0x7f2a1c151960 2023-10-24T23:55:43.398 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:43.399 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:43.399 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-10-24T23:55:43.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 -- 172.21.15.107:0/2864259380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a00065e60 msgr2=0x7f2a00068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:43.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a00065e60 0x7f2a00068320 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c006fd0 tx=0x7f2a0c008040 comp rx=0 tx=0).stop 2023-10-24T23:55:43.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 -- 172.21.15.107:0/2864259380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a1c151960 msgr2=0x7f2a1c069fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:43.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a1c151960 0x7f2a1c069fa0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400c780 tx=0x7f2a1400cc40 comp rx=0 tx=0).stop 2023-10-24T23:55:43.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 -- 172.21.15.107:0/2864259380 shutdown_connections 2023-10-24T23:55:43.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a1c151960 0x7f2a1c069fa0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:43.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a1c0c24c0 0x7f2a1c1ecc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:43.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a00065e60 0x7f2a00068320 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:43.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 --2- 172.21.15.107:0/2864259380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a1c071220 0x7f2a1c154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:43.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 -- 172.21.15.107:0/2864259380 >> 172.21.15.107:0/2864259380 conn(0x7f2a1c0c2950 msgr2=0x7f2a1c06a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:43.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 -- 172.21.15.107:0/2864259380 shutdown_connections 2023-10-24T23:55:43.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:43.395+0000 7f2a23c32640 1 -- 172.21.15.107:0/2864259380 wait complete. 2023-10-24T23:55:43.417 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:55:43.723 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2023-10-24T23:55:43.740 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2023-10-24T23:55:44.066 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2023-10-24T23:55:44.086 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2023-10-24T23:55:44.229 INFO:tasks.workunit.client.1.smithi110.stdout:mv -f .deps/list.Tpo .deps/list.Po 2023-10-24T23:55:44.251 INFO:tasks.workunit.client.1.smithi110.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2023-10-24T23:55:45.255 INFO:tasks.workunit.client.1.smithi110.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb' 2023-10-24T23:55:45.270 INFO:tasks.workunit.client.1.smithi110.stdout:FFSB version 6.0-RC2 started 2023-10-24T23:55:45.270 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-24T23:55:45.274 INFO:tasks.workunit.client.1.smithi110.stdout:benchmark time = 300 2023-10-24T23:55:45.274 INFO:tasks.workunit.client.1.smithi110.stdout:ThreadGroup 0 2023-10-24T23:55:45.274 INFO:tasks.workunit.client.1.smithi110.stdout:================ 2023-10-24T23:55:45.274 INFO:tasks.workunit.client.1.smithi110.stdout: num_threads = 32 2023-10-24T23:55:45.275 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-24T23:55:45.275 INFO:tasks.workunit.client.1.smithi110.stdout: read_random = off 2023-10-24T23:55:45.275 INFO:tasks.workunit.client.1.smithi110.stdout: read_size = 0 (0B) 2023-10-24T23:55:45.275 INFO:tasks.workunit.client.1.smithi110.stdout: read_blocksize = 0 (0B) 2023-10-24T23:55:45.275 INFO:tasks.workunit.client.1.smithi110.stdout: read_skip = off 2023-10-24T23:55:45.276 INFO:tasks.workunit.client.1.smithi110.stdout: read_skipsize = 0 (0B) 2023-10-24T23:55:45.276 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-24T23:55:45.276 INFO:tasks.workunit.client.1.smithi110.stdout: write_random = on 2023-10-24T23:55:45.276 INFO:tasks.workunit.client.1.smithi110.stdout: write_size = 5242880 (5MB) 2023-10-24T23:55:45.276 INFO:tasks.workunit.client.1.smithi110.stdout: fsync_file = 0 2023-10-24T23:55:45.276 INFO:tasks.workunit.client.1.smithi110.stdout: write_blocksize = 4096 (4KB) 2023-10-24T23:55:45.277 INFO:tasks.workunit.client.1.smithi110.stdout: wait time = 0 2023-10-24T23:55:45.277 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-24T23:55:45.277 INFO:tasks.workunit.client.1.smithi110.stdout: op weights 2023-10-24T23:55:45.277 INFO:tasks.workunit.client.1.smithi110.stdout: read = 0 (0.00%) 2023-10-24T23:55:45.277 INFO:tasks.workunit.client.1.smithi110.stdout: readall = 0 (0.00%) 2023-10-24T23:55:45.278 INFO:tasks.workunit.client.1.smithi110.stdout: write = 1 (100.00%) 2023-10-24T23:55:45.278 INFO:tasks.workunit.client.1.smithi110.stdout: create = 0 (0.00%) 2023-10-24T23:55:45.278 INFO:tasks.workunit.client.1.smithi110.stdout: append = 0 (0.00%) 2023-10-24T23:55:45.278 INFO:tasks.workunit.client.1.smithi110.stdout: delete = 0 (0.00%) 2023-10-24T23:55:45.278 INFO:tasks.workunit.client.1.smithi110.stdout: metaop = 0 (0.00%) 2023-10-24T23:55:45.278 INFO:tasks.workunit.client.1.smithi110.stdout: createdir = 0 (0.00%) 2023-10-24T23:55:45.279 INFO:tasks.workunit.client.1.smithi110.stdout: stat = 0 (0.00%) 2023-10-24T23:55:45.279 INFO:tasks.workunit.client.1.smithi110.stdout: writeall = 0 (0.00%) 2023-10-24T23:55:45.279 INFO:tasks.workunit.client.1.smithi110.stdout: writeall_fsync = 0 (0.00%) 2023-10-24T23:55:45.279 INFO:tasks.workunit.client.1.smithi110.stdout: open_close = 0 (0.00%) 2023-10-24T23:55:45.279 INFO:tasks.workunit.client.1.smithi110.stdout: write_fsync = 0 (0.00%) 2023-10-24T23:55:45.280 INFO:tasks.workunit.client.1.smithi110.stdout: create_fsync = 0 (0.00%) 2023-10-24T23:55:45.280 INFO:tasks.workunit.client.1.smithi110.stdout: append_fsync = 0 (0.00%) 2023-10-24T23:55:45.280 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-24T23:55:45.280 INFO:tasks.workunit.client.1.smithi110.stdout:FileSystem . 2023-10-24T23:55:45.280 INFO:tasks.workunit.client.1.smithi110.stdout:========== 2023-10-24T23:55:45.280 INFO:tasks.workunit.client.1.smithi110.stdout: num_dirs = 0 2023-10-24T23:55:45.281 INFO:tasks.workunit.client.1.smithi110.stdout: starting files = 128 2023-10-24T23:55:45.281 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-24T23:55:45.281 INFO:tasks.workunit.client.1.smithi110.stdout: min file size = 104857600 (100MB) 2023-10-24T23:55:45.281 INFO:tasks.workunit.client.1.smithi110.stdout: max file size = 104857600 (100MB) 2023-10-24T23:55:45.281 INFO:tasks.workunit.client.1.smithi110.stdout: directio = off 2023-10-24T23:55:45.282 INFO:tasks.workunit.client.1.smithi110.stdout: alignedio = on 2023-10-24T23:55:45.282 INFO:tasks.workunit.client.1.smithi110.stdout: bufferedio = off 2023-10-24T23:55:45.282 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-24T23:55:45.282 INFO:tasks.workunit.client.1.smithi110.stdout: aging is off 2023-10-24T23:55:45.283 INFO:tasks.workunit.client.1.smithi110.stdout: current utilization = 0.00% 2023-10-24T23:55:45.283 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-24T23:55:45.283 INFO:tasks.workunit.client.1.smithi110.stdout:checking existing fs: . 2023-10-24T23:55:45.283 INFO:tasks.workunit.client.1.smithi110.stdout:recreating new fileset 2023-10-24T23:55:45.284 INFO:tasks.workunit.client.1.smithi110.stderr:opendir: No such file or directory 2023-10-24T23:55:45.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:45.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e8b7af640 1 Processor -- start 2023-10-24T23:55:45.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e8b7af640 1 -- start start 2023-10-24T23:55:45.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e8b7af640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0e8406af70 0x7f0e8406c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:45.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e8b7af640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e840c1860 con 0x7f0e8414b600 2023-10-24T23:55:45.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e8b7af640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e84154d50 con 0x7f0e8406a230 2023-10-24T23:55:45.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e8b7af640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8414b9e0 con 0x7f0e8406af70 2023-10-24T23:55:45.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e8954b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0e8414b600 0x7f0e84151fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56242/0 (socket says 172.21.15.107:56242) 2023-10-24T23:55:45.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e8954b640 1 -- 172.21.15.107:0/1727045383 learned_addr learned my addr 172.21.15.107:0/1727045383 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:45.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e88d4a640 1 --2- 172.21.15.107:0/1727045383 >> v2:172.21.15.110:3300/0 conn(0x7f0e8406af70 0x7f0e8406c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:45.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259129829 0 0) 0x7f0e840c1860 con 0x7f0e8414b600 2023-10-24T23:55:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e70003530 con 0x7f0e8414b600 2023-10-24T23:55:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3609481749 0 0) 0x7f0e70003530 con 0x7f0e8414b600 2023-10-24T23:55:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 >> v2:172.21.15.110:3300/0 conn(0x7f0e8406af70 msgr2=0x7f0e8406c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 --2- 172.21.15.107:0/1727045383 >> v2:172.21.15.110:3300/0 conn(0x7f0e8406af70 0x7f0e8406c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:45.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 >> v1:172.21.15.110:6789/0 conn(0x7f0e8406a230 legacy=0x7f0e841551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e840c1ab0 con 0x7f0e8414b600 2023-10-24T23:55:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f0e80001680 con 0x7f0e8414b600 2023-10-24T23:55:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 >> v1:172.21.15.107:6789/0 conn(0x7f0e8414b600 legacy=0x7f0e84151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e70003980 0x7f0e70003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:45.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e700047d0 0x7f0e70004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e70005190 0x7f0e7000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8414b9e0 con 0x7f0e70003980 2023-10-24T23:55:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7000c130 con 0x7f0e700047d0 2023-10-24T23:55:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7000c2f0 con 0x7f0e70005190 2023-10-24T23:55:45.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.547+0000 7f0e8954b640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e70003980 0x7f0e70003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e88d4a640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e700047d0 0x7f0e70004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e89d4c640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e70005190 0x7f0e7000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8954b640 1 -- 172.21.15.107:0/1727045383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e70005190 msgr2=0x7f0e7000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8954b640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e70005190 0x7f0e7000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:45.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8954b640 1 -- 172.21.15.107:0/1727045383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e700047d0 msgr2=0x7f0e70004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:45.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8954b640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e700047d0 0x7f0e70004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:45.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8954b640 1 -- 172.21.15.107:0/1727045383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e70003530 con 0x7f0e70003980 2023-10-24T23:55:45.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e89d4c640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e70005190 0x7f0e7000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:45.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e89d4c640 1 -- 172.21.15.107:0/1727045383 reap_dead start 2023-10-24T23:55:45.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8954b640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e70003980 0x7f0e70003da0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0e8000c240 tx=0x7f0e8001cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69bb690876ed6aa8 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:45.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e800058e0 con 0x7f0e70003980 2023-10-24T23:55:45.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e80005a40 con 0x7f0e70003980 2023-10-24T23:55:45.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e7bfff640 1 -- 172.21.15.107:0/1727045383 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e80005e70 con 0x7f0e70003980 2023-10-24T23:55:45.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8b7af640 1 -- 172.21.15.107:0/1727045383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e70003980 msgr2=0x7f0e70003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:45.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8b7af640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e70003980 0x7f0e70003da0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0e8000c240 tx=0x7f0e8001cc10 comp rx=0 tx=0).stop 2023-10-24T23:55:45.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8b7af640 1 -- 172.21.15.107:0/1727045383 shutdown_connections 2023-10-24T23:55:45.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8b7af640 1 --2- 172.21.15.107:0/1727045383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e70003980 0x7f0e70003da0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:45.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8b7af640 1 -- 172.21.15.107:0/1727045383 >> 172.21.15.107:0/1727045383 conn(0x7f0e840c2950 msgr2=0x7f0e840c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:45.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8b7af640 1 -- 172.21.15.107:0/1727045383 shutdown_connections 2023-10-24T23:55:45.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.551+0000 7f0e8b7af640 1 -- 172.21.15.107:0/1727045383 wait complete. 2023-10-24T23:55:45.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8b7af640 1 Processor -- start 2023-10-24T23:55:45.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8b7af640 1 -- start start 2023-10-24T23:55:45.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8b7af640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e70003980 0x7f0e8414f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:45.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8b7af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e8413fbc0 0x7f0e84150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:45.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8b7af640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e84136690 0x7f0e84136b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:45.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8b7af640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e840710d0 con 0x7f0e8413fbc0 2023-10-24T23:55:45.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8b7af640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e84151360 con 0x7f0e84136690 2023-10-24T23:55:45.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8b7af640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8414f8e0 con 0x7f0e70003980 2023-10-24T23:55:45.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e88d4a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e8413fbc0 0x7f0e84150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:45.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e88d4a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e8413fbc0 0x7f0e84150390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38902/0 (socket says 172.21.15.107:38902) 2023-10-24T23:55:45.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e88d4a640 1 -- 172.21.15.107:0/3245076157 learned_addr learned my addr 172.21.15.107:0/3245076157 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:45.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8954b640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e70003980 0x7f0e8414f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:45.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e89d4c640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e84136690 0x7f0e84136b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:45.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e88d4a640 1 -- 172.21.15.107:0/3245076157 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e70003980 msgr2=0x7f0e8414f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:45.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e88d4a640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e70003980 0x7f0e8414f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:45.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e88d4a640 1 -- 172.21.15.107:0/3245076157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e84136690 msgr2=0x7f0e84136b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:45.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e88d4a640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e84136690 0x7f0e84136b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:45.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e88d4a640 1 -- 172.21.15.107:0/3245076157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e84152540 con 0x7f0e8413fbc0 2023-10-24T23:55:45.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e88d4a640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e8413fbc0 0x7f0e84150390 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0e740119e0 tx=0x7f0e74011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:45.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e79ffb640 1 -- 172.21.15.107:0/3245076157 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74004240 con 0x7f0e8413fbc0 2023-10-24T23:55:45.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e79ffb640 1 -- 172.21.15.107:0/3245076157 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e740043a0 con 0x7f0e8413fbc0 2023-10-24T23:55:45.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8b7af640 1 -- 172.21.15.107:0/3245076157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8413bcb0 con 0x7f0e8413fbc0 2023-10-24T23:55:45.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e79ffb640 1 -- 172.21.15.107:0/3245076157 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7400fb30 con 0x7f0e8413fbc0 2023-10-24T23:55:45.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8b7af640 1 -- 172.21.15.107:0/3245076157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e84070d60 con 0x7f0e8413fbc0 2023-10-24T23:55:45.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e8b7af640 1 -- 172.21.15.107:0/3245076157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e84071280 con 0x7f0e8413fbc0 2023-10-24T23:55:45.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.555+0000 7f0e79ffb640 1 -- 172.21.15.107:0/3245076157 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0e7400fc90 con 0x7f0e8413fbc0 2023-10-24T23:55:45.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.559+0000 7f0e79ffb640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e68065e90 0x7f0e68068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:45.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.559+0000 7f0e79ffb640 1 -- 172.21.15.107:0/3245076157 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f0e740841d0 con 0x7f0e8413fbc0 2023-10-24T23:55:45.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.559+0000 7f0e8954b640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e68065e90 0x7f0e68068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:45.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.559+0000 7f0e8954b640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e68065e90 0x7f0e68068350 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0e800ab000 tx=0x7f0e800a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:45.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.563+0000 7f0e79ffb640 1 -- 172.21.15.107:0/3245076157 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0e84071280 con 0x7f0e8413fbc0 2023-10-24T23:55:45.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.755+0000 7f0e8b7af640 1 -- 172.21.15.107:0/3245076157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e84071280 con 0x7f0e8413fbc0 2023-10-24T23:55:45.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.755+0000 7f0e79ffb640 1 -- 172.21.15.107:0/3245076157 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f0e84071280 con 0x7f0e8413fbc0 2023-10-24T23:55:45.760 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:45.760 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:45.761 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-10-24T23:55:45.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 -- 172.21.15.107:0/3245076157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e68065e90 msgr2=0x7f0e68068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:45.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e68065e90 0x7f0e68068350 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0e800ab000 tx=0x7f0e800a7070 comp rx=0 tx=0).stop 2023-10-24T23:55:45.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 -- 172.21.15.107:0/3245076157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e8413fbc0 msgr2=0x7f0e84150390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:45.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e8413fbc0 0x7f0e84150390 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0e740119e0 tx=0x7f0e74011ea0 comp rx=0 tx=0).stop 2023-10-24T23:55:45.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 -- 172.21.15.107:0/3245076157 shutdown_connections 2023-10-24T23:55:45.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e84136690 0x7f0e84136b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:45.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e8413fbc0 0x7f0e84150390 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:45.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e68065e90 0x7f0e68068350 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:45.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 --2- 172.21.15.107:0/3245076157 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e70003980 0x7f0e8414f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:45.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 -- 172.21.15.107:0/3245076157 >> 172.21.15.107:0/3245076157 conn(0x7f0e840c2950 msgr2=0x7f0e84149610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:45.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 -- 172.21.15.107:0/3245076157 shutdown_connections 2023-10-24T23:55:45.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:45.759+0000 7f0e8b7af640 1 -- 172.21.15.107:0/3245076157 wait complete. 2023-10-24T23:55:45.779 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:55:47.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:47.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.903+0000 7f7f24892640 1 Processor -- start 2023-10-24T23:55:47.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.903+0000 7f7f24892640 1 -- start start 2023-10-24T23:55:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f24892640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7f20150930 0x7f7f200a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:47.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f24892640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f201510c0 0x7f7f20151550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f24892640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f200a9b90 con 0x7f7f200c1570 2023-10-24T23:55:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f24892640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f200c1390 con 0x7f7f20150930 2023-10-24T23:55:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f24892640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f20154dd0 con 0x7f7f201510c0 2023-10-24T23:55:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f1e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7f20150930 0x7f7f200a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f1e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7f20150930 0x7f7f200a94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38910/0 (socket says 172.21.15.107:38910) 2023-10-24T23:55:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f1e59c640 1 -- 172.21.15.107:0/1727669012 learned_addr learned my addr 172.21.15.107:0/1727669012 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f1ed9d640 1 --2- 172.21.15.107:0/1727669012 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f201510c0 0x7f7f20151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f1e59c640 1 -- 172.21.15.107:0/1727669012 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f201510c0 msgr2=0x7f7f20151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f1e59c640 1 --2- 172.21.15.107:0/1727669012 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f201510c0 0x7f7f20151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:47.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f1e59c640 1 -- 172.21.15.107:0/1727669012 >> v1:172.21.15.110:6789/0 conn(0x7f7f200c1570 legacy=0x7f7f20069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:47.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f1e59c640 1 -- 172.21.15.107:0/1727669012 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f201550b0 con 0x7f7f20150930 2023-10-24T23:55:47.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f1e59c640 1 --2- 172.21.15.107:0/1727669012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f20150930 0x7f7f200a94b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7f18009e50 tx=0x7f7f1801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc8157c3ed8c1f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:47.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.907+0000 7f7f1d59a640 1 -- 172.21.15.107:0/1727669012 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f180a5540 con 0x7f7f20150930 2023-10-24T23:55:47.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1d59a640 1 -- 172.21.15.107:0/1727669012 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f180a56a0 con 0x7f7f20150930 2023-10-24T23:55:47.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1d59a640 1 -- 172.21.15.107:0/1727669012 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f180a5a30 con 0x7f7f20150930 2023-10-24T23:55:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 -- 172.21.15.107:0/1727669012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f20150930 msgr2=0x7f7f200a94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 --2- 172.21.15.107:0/1727669012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f20150930 0x7f7f200a94b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7f18009e50 tx=0x7f7f1801c940 comp rx=0 tx=0).stop 2023-10-24T23:55:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 -- 172.21.15.107:0/1727669012 shutdown_connections 2023-10-24T23:55:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 --2- 172.21.15.107:0/1727669012 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f201510c0 0x7f7f20151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 --2- 172.21.15.107:0/1727669012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f20150930 0x7f7f200a94b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 -- 172.21.15.107:0/1727669012 >> 172.21.15.107:0/1727669012 conn(0x7f7f200c65c0 msgr2=0x7f7f200c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 -- 172.21.15.107:0/1727669012 shutdown_connections 2023-10-24T23:55:47.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 -- 172.21.15.107:0/1727669012 wait complete. 2023-10-24T23:55:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 Processor -- start 2023-10-24T23:55:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 -- start start 2023-10-24T23:55:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f200c1570 0x7f7f201ed960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:47.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f20150930 0x7f7f200bf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:47.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f201510c0 0x7f7f200bfe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:47.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f200bdcb0 con 0x7f7f20150930 2023-10-24T23:55:47.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2014f360 con 0x7f7f200c1570 2023-10-24T23:55:47.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f200a7e00 con 0x7f7f201510c0 2023-10-24T23:55:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f20150930 0x7f7f200bf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f20150930 0x7f7f200bf8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38916/0 (socket says 172.21.15.107:38916) 2023-10-24T23:55:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1dd9b640 1 -- 172.21.15.107:0/2511261962 learned_addr learned my addr 172.21.15.107:0/2511261962 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1e59c640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f200c1570 0x7f7f201ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:47.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1ed9d640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f201510c0 0x7f7f200bfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:47.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1dd9b640 1 -- 172.21.15.107:0/2511261962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f201510c0 msgr2=0x7f7f200bfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:47.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1dd9b640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f201510c0 0x7f7f200bfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:47.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1dd9b640 1 -- 172.21.15.107:0/2511261962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f200c1570 msgr2=0x7f7f201ed960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:47.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1dd9b640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f200c1570 0x7f7f201ed960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:47.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1dd9b640 1 -- 172.21.15.107:0/2511261962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f18009b30 con 0x7f7f20150930 2023-10-24T23:55:47.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1ed9d640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f201510c0 0x7f7f200bfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:47.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f1dd9b640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f20150930 0x7f7f200bf8d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400e760 tx=0x7f7f1400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:47.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f0effd640 1 -- 172.21.15.107:0/2511261962 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14004260 con 0x7f7f20150930 2023-10-24T23:55:47.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f0effd640 1 -- 172.21.15.107:0/2511261962 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f140043e0 con 0x7f7f20150930 2023-10-24T23:55:47.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f0effd640 1 -- 172.21.15.107:0/2511261962 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f140116b0 con 0x7f7f20150930 2023-10-24T23:55:47.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.911+0000 7f7f24892640 1 -- 172.21.15.107:0/2511261962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2014b230 con 0x7f7f20150930 2023-10-24T23:55:47.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.915+0000 7f7f24892640 1 -- 172.21.15.107:0/2511261962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2014aee0 con 0x7f7f20150930 2023-10-24T23:55:47.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.915+0000 7f7f24892640 1 -- 172.21.15.107:0/2511261962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f201ee5a0 con 0x7f7f20150930 2023-10-24T23:55:47.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.915+0000 7f7f0effd640 1 -- 172.21.15.107:0/2511261962 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7f14011810 con 0x7f7f20150930 2023-10-24T23:55:47.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.915+0000 7f7f0effd640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f00065e60 0x7f7f00068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:47.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.915+0000 7f7f1e59c640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f00065e60 0x7f7f00068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:47.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.915+0000 7f7f0effd640 1 -- 172.21.15.107:0/2511261962 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f7f14086270 con 0x7f7f20150930 2023-10-24T23:55:47.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.915+0000 7f7f1e59c640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f00065e60 0x7f7f00068320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7f180ae000 tx=0x7f7f180a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:47.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:47.919+0000 7f7f0effd640 1 -- 172.21.15.107:0/2511261962 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7f201ee5a0 con 0x7f7f20150930 2023-10-24T23:55:48.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.107+0000 7f7f24892640 1 -- 172.21.15.107:0/2511261962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f201ee5a0 con 0x7f7f20150930 2023-10-24T23:55:48.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f0effd640 1 -- 172.21.15.107:0/2511261962 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f7f201ee5a0 con 0x7f7f20150930 2023-10-24T23:55:48.114 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:48.115 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:48.115 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-10-24T23:55:48.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 -- 172.21.15.107:0/2511261962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f00065e60 msgr2=0x7f7f00068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:48.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f00065e60 0x7f7f00068320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7f180ae000 tx=0x7f7f180a7070 comp rx=0 tx=0).stop 2023-10-24T23:55:48.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 -- 172.21.15.107:0/2511261962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f20150930 msgr2=0x7f7f200bf8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:48.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f20150930 0x7f7f200bf8d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400e760 tx=0x7f7f1400ec20 comp rx=0 tx=0).stop 2023-10-24T23:55:48.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 -- 172.21.15.107:0/2511261962 shutdown_connections 2023-10-24T23:55:48.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f00065e60 0x7f7f00068320 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:48.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f201510c0 0x7f7f200bfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:48.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f20150930 0x7f7f200bf8d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:48.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 --2- 172.21.15.107:0/2511261962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f200c1570 0x7f7f201ed960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:48.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 -- 172.21.15.107:0/2511261962 >> 172.21.15.107:0/2511261962 conn(0x7f7f200c65c0 msgr2=0x7f7f200c43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:48.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 -- 172.21.15.107:0/2511261962 shutdown_connections 2023-10-24T23:55:48.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:48.111+0000 7f7f24892640 1 -- 172.21.15.107:0/2511261962 wait complete. 2023-10-24T23:55:48.133 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:55:50.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:50.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f985aa5d640 1 Processor -- start 2023-10-24T23:55:50.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f985aa5d640 1 -- start start 2023-10-24T23:55:50.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f985aa5d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9854071220 0x7f985406bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f985aa5d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9854150bc0 con 0x7f98540c24c0 2023-10-24T23:55:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f985aa5d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985406c510 con 0x7f9854151960 2023-10-24T23:55:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f985aa5d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f985406c7f0 con 0x7f9854071220 2023-10-24T23:55:50.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f9858ffa640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f98540c24c0 0x7f985406a9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56244/0 (socket says 172.21.15.107:56244) 2023-10-24T23:55:50.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f9858ffa640 1 -- 172.21.15.107:0/2492043730 learned_addr learned my addr 172.21.15.107:0/2492043730 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:50.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f9853fff640 1 --2- 172.21.15.107:0/2492043730 >> v2:172.21.15.110:3300/0 conn(0x7f9854071220 0x7f985406bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:50.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f9852ffd640 1 -- 172.21.15.107:0/2492043730 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052144130 0 0) 0x7f9854150bc0 con 0x7f98540c24c0 2023-10-24T23:55:50.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f9852ffd640 1 -- 172.21.15.107:0/2492043730 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f983c003530 con 0x7f98540c24c0 2023-10-24T23:55:50.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f9853fff640 1 -- 172.21.15.107:0/2492043730 >> v1:172.21.15.110:6789/0 conn(0x7f9854151960 legacy=0x7f9854071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:50.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f9853fff640 1 -- 172.21.15.107:0/2492043730 >> v1:172.21.15.107:6789/0 conn(0x7f98540c24c0 legacy=0x7f985406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:50.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f9852ffd640 1 -- 172.21.15.107:0/2492043730 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1446701232 0 0) 0x7f983c003530 con 0x7f98540c24c0 2023-10-24T23:55:50.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.267+0000 7f9853fff640 1 -- 172.21.15.107:0/2492043730 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9854150e80 con 0x7f9854071220 2023-10-24T23:55:50.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9853fff640 1 --2- 172.21.15.107:0/2492043730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9854071220 0x7f985406bce0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f98440097c0 tx=0x7f984401c310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fea8ed250e3294b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:50.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9852ffd640 1 -- 172.21.15.107:0/2492043730 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c001a00 con 0x7f9854071220 2023-10-24T23:55:50.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9852ffd640 1 -- 172.21.15.107:0/2492043730 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98440a7050 con 0x7f9854071220 2023-10-24T23:55:50.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9852ffd640 1 -- 172.21.15.107:0/2492043730 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98440ab620 con 0x7f9854071220 2023-10-24T23:55:50.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 -- 172.21.15.107:0/2492043730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9854071220 msgr2=0x7f985406bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:50.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 --2- 172.21.15.107:0/2492043730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9854071220 0x7f985406bce0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f98440097c0 tx=0x7f984401c310 comp rx=0 tx=0).stop 2023-10-24T23:55:50.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 -- 172.21.15.107:0/2492043730 shutdown_connections 2023-10-24T23:55:50.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 --2- 172.21.15.107:0/2492043730 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9854071220 0x7f985406bce0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:50.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 -- 172.21.15.107:0/2492043730 >> 172.21.15.107:0/2492043730 conn(0x7f98540c2950 msgr2=0x7f98540c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 -- 172.21.15.107:0/2492043730 shutdown_connections 2023-10-24T23:55:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 -- 172.21.15.107:0/2492043730 wait complete. 2023-10-24T23:55:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 Processor -- start 2023-10-24T23:55:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 -- start start 2023-10-24T23:55:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9854071220 0x7f9854141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:50.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98540c24c0 0x7f98541435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9854151960 0x7f9854143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9854143180 con 0x7f9854151960 2023-10-24T23:55:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9854144020 con 0x7f98540c24c0 2023-10-24T23:55:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9854144230 con 0x7f9854071220 2023-10-24T23:55:50.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9858ffa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9854151960 0x7f9854143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:50.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9858ffa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9854151960 0x7f9854143ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38924/0 (socket says 172.21.15.107:38924) 2023-10-24T23:55:50.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9858ffa640 1 -- 172.21.15.107:0/2618005581 learned_addr learned my addr 172.21.15.107:0/2618005581 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:50.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f98537fe640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98540c24c0 0x7f98541435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:50.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9853fff640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9854071220 0x7f9854141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:50.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9858ffa640 1 -- 172.21.15.107:0/2618005581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9854071220 msgr2=0x7f9854141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:50.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9858ffa640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9854071220 0x7f9854141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:50.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9858ffa640 1 -- 172.21.15.107:0/2618005581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98540c24c0 msgr2=0x7f98541435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:50.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9858ffa640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98540c24c0 0x7f98541435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:50.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9858ffa640 1 -- 172.21.15.107:0/2618005581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9844009420 con 0x7f9854151960 2023-10-24T23:55:50.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9853fff640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9854071220 0x7f9854141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:55:50.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9858ffa640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9854151960 0x7f9854143ae0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f984c0018a0 tx=0x7f984c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:50.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9850ff9640 1 -- 172.21.15.107:0/2618005581 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c012400 con 0x7f9854151960 2023-10-24T23:55:50.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9850ff9640 1 -- 172.21.15.107:0/2618005581 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f984c012560 con 0x7f9854151960 2023-10-24T23:55:50.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f985aa5d640 1 -- 172.21.15.107:0/2618005581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9854140580 con 0x7f9854151960 2023-10-24T23:55:50.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.271+0000 7f9850ff9640 1 -- 172.21.15.107:0/2618005581 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984c012910 con 0x7f9854151960 2023-10-24T23:55:50.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.275+0000 7f985aa5d640 1 -- 172.21.15.107:0/2618005581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9854140840 con 0x7f9854151960 2023-10-24T23:55:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.275+0000 7f985aa5d640 1 -- 172.21.15.107:0/2618005581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98541432c0 con 0x7f9854151960 2023-10-24T23:55:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.275+0000 7f9850ff9640 1 -- 172.21.15.107:0/2618005581 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f984c012a70 con 0x7f9854151960 2023-10-24T23:55:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.275+0000 7f9850ff9640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9840065e60 0x7f9840068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.275+0000 7f9850ff9640 1 -- 172.21.15.107:0/2618005581 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f984c084310 con 0x7f9854151960 2023-10-24T23:55:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.275+0000 7f9853fff640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9840065e60 0x7f9840068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.275+0000 7f9853fff640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9840065e60 0x7f9840068320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f98440ad000 tx=0x7f98440a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.279+0000 7f9850ff9640 1 -- 172.21.15.107:0/2618005581 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f98541432c0 con 0x7f9854151960 2023-10-24T23:55:50.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 -- 172.21.15.107:0/2618005581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98541432c0 con 0x7f9854151960 2023-10-24T23:55:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f9850ff9640 1 -- 172.21.15.107:0/2618005581 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f98541432c0 con 0x7f9854151960 2023-10-24T23:55:50.479 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:50.480 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:50.480 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-10-24T23:55:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 -- 172.21.15.107:0/2618005581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9840065e60 msgr2=0x7f9840068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9840065e60 0x7f9840068320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f98440ad000 tx=0x7f98440a6000 comp rx=0 tx=0).stop 2023-10-24T23:55:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 -- 172.21.15.107:0/2618005581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9854151960 msgr2=0x7f9854143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9854151960 0x7f9854143ae0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f984c0018a0 tx=0x7f984c00a430 comp rx=0 tx=0).stop 2023-10-24T23:55:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 -- 172.21.15.107:0/2618005581 shutdown_connections 2023-10-24T23:55:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9854151960 0x7f9854143ae0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98540c24c0 0x7f98541435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9840065e60 0x7f9840068320 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:50.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 --2- 172.21.15.107:0/2618005581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9854071220 0x7f9854141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:50.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 -- 172.21.15.107:0/2618005581 >> 172.21.15.107:0/2618005581 conn(0x7f98540c2950 msgr2=0x7f985406b160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:50.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 -- 172.21.15.107:0/2618005581 shutdown_connections 2023-10-24T23:55:50.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:50.475+0000 7f985aa5d640 1 -- 172.21.15.107:0/2618005581 wait complete. 2023-10-24T23:55:50.498 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:55:52.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:52.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.623+0000 7f637d444640 1 Processor -- start 2023-10-24T23:55:52.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.623+0000 7f637d444640 1 -- start start 2023-10-24T23:55:52.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f637d444640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f637814b600 0x7f6378151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:52.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f637d444640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f637806af70 0x7f637806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:52.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f637d444640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63780c1860 con 0x7f637806a230 2023-10-24T23:55:52.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f637d444640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6378154d50 con 0x7f637806af70 2023-10-24T23:55:52.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f637d444640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63781524f0 con 0x7f637814b600 2023-10-24T23:55:52.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f63767fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f637806af70 0x7f637806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:52.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f63777fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f637806a230 0x7f63781551e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47228/0 (socket says 172.21.15.107:47228) 2023-10-24T23:55:52.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f63777fe640 1 -- 172.21.15.107:0/4057911553 learned_addr learned my addr 172.21.15.107:0/4057911553 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:52.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f6376ffd640 1 --2- 172.21.15.107:0/4057911553 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f637814b600 0x7f6378151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:52.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f63767fc640 1 -- 172.21.15.107:0/4057911553 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f637814b600 msgr2=0x7f6378151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:52.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f63767fc640 1 --2- 172.21.15.107:0/4057911553 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f637814b600 0x7f6378151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:52.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f63767fc640 1 -- 172.21.15.107:0/4057911553 >> v1:172.21.15.107:6789/0 conn(0x7f637806a230 legacy=0x7f63781551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:52.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f63767fc640 1 -- 172.21.15.107:0/4057911553 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637814b9e0 con 0x7f637806af70 2023-10-24T23:55:52.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f6375ffb640 1 -- 172.21.15.107:0/4057911553 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775607653 0 0) 0x7f63780c1860 con 0x7f637806a230 2023-10-24T23:55:52.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f63767fc640 1 --2- 172.21.15.107:0/4057911553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637806af70 0x7f637806c4c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f636c0097c0 tx=0x7f636c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eddeeb73d4f031aa server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:52.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f6375ffb640 1 -- 172.21.15.107:0/4057911553 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c0ac620 con 0x7f637806af70 2023-10-24T23:55:52.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f6375ffb640 1 -- 172.21.15.107:0/4057911553 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c0ac780 con 0x7f637806af70 2023-10-24T23:55:52.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.627+0000 7f6375ffb640 1 -- 172.21.15.107:0/4057911553 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c0acb10 con 0x7f637806af70 2023-10-24T23:55:52.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 -- 172.21.15.107:0/4057911553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637806af70 msgr2=0x7f637806c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:52.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 --2- 172.21.15.107:0/4057911553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637806af70 0x7f637806c4c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f636c0097c0 tx=0x7f636c01c3c0 comp rx=0 tx=0).stop 2023-10-24T23:55:52.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 -- 172.21.15.107:0/4057911553 shutdown_connections 2023-10-24T23:55:52.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 --2- 172.21.15.107:0/4057911553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637806af70 0x7f637806c4c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:52.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 --2- 172.21.15.107:0/4057911553 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f637814b600 0x7f6378151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:52.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 -- 172.21.15.107:0/4057911553 >> 172.21.15.107:0/4057911553 conn(0x7f63780c2950 msgr2=0x7f63780c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:52.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 -- 172.21.15.107:0/4057911553 shutdown_connections 2023-10-24T23:55:52.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 -- 172.21.15.107:0/4057911553 wait complete. 2023-10-24T23:55:52.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 Processor -- start 2023-10-24T23:55:52.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 -- start start 2023-10-24T23:55:52.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f637806a230 0x7f63781ec6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:52.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637806af70 0x7f63781f0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:52.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f637814b600 0x7f637815a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:52.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63781efc10 con 0x7f637806af70 2023-10-24T23:55:52.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63781efef0 con 0x7f637806a230 2023-10-24T23:55:52.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63780c5ba0 con 0x7f637814b600 2023-10-24T23:55:52.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f63767fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637806af70 0x7f63781f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:52.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f63767fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637806af70 0x7f63781f0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46792/0 (socket says 172.21.15.107:46792) 2023-10-24T23:55:52.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f63767fc640 1 -- 172.21.15.107:0/235005191 learned_addr learned my addr 172.21.15.107:0/235005191 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:52.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f63777fe640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f637814b600 0x7f637815a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:52.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f6376ffd640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f637806a230 0x7f63781ec6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:52.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f63767fc640 1 -- 172.21.15.107:0/235005191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f637814b600 msgr2=0x7f637815a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:52.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f63767fc640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f637814b600 0x7f637815a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:52.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f63767fc640 1 -- 172.21.15.107:0/235005191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f637806a230 msgr2=0x7f63781ec6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:52.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f63767fc640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f637806a230 0x7f63781ec6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f63767fc640 1 -- 172.21.15.107:0/235005191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c009420 con 0x7f637806af70 2023-10-24T23:55:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f6376ffd640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f637806a230 0x7f63781ec6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f63767fc640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637806af70 0x7f63781f0380 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f636c0af040 tx=0x7f636c019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f635f7fe640 1 -- 172.21.15.107:0/235005191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c0b5410 con 0x7f637806af70 2023-10-24T23:55:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f635f7fe640 1 -- 172.21.15.107:0/235005191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c0b5590 con 0x7f637806af70 2023-10-24T23:55:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 -- 172.21.15.107:0/235005191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63780c4970 con 0x7f637806af70 2023-10-24T23:55:52.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f635f7fe640 1 -- 172.21.15.107:0/235005191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c0b5940 con 0x7f637806af70 2023-10-24T23:55:52.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.631+0000 7f637d444640 1 -- 172.21.15.107:0/235005191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6378146380 con 0x7f637806af70 2023-10-24T23:55:52.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.635+0000 7f637d444640 1 -- 172.21.15.107:0/235005191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63780746e0 con 0x7f637806af70 2023-10-24T23:55:52.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.635+0000 7f635f7fe640 1 -- 172.21.15.107:0/235005191 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f636c0b3040 con 0x7f637806af70 2023-10-24T23:55:52.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.635+0000 7f635f7fe640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6364065e60 0x7f6364068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:52.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.635+0000 7f635f7fe640 1 -- 172.21.15.107:0/235005191 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f636c11dcd0 con 0x7f637806af70 2023-10-24T23:55:52.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.635+0000 7f6376ffd640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6364065e60 0x7f6364068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:52.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.635+0000 7f6376ffd640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6364065e60 0x7f6364068320 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f637815ba10 tx=0x7f6368008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:52.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.639+0000 7f635f7fe640 1 -- 172.21.15.107:0/235005191 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f636c0eaf80 con 0x7f637806af70 2023-10-24T23:55:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.831+0000 7f637d444640 1 -- 172.21.15.107:0/235005191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6378069fa0 con 0x7f637806af70 2023-10-24T23:55:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.831+0000 7f635f7fe640 1 -- 172.21.15.107:0/235005191 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f636c0b1090 con 0x7f637806af70 2023-10-24T23:55:52.836 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:52.836 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-10-24T23:55:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 -- 172.21.15.107:0/235005191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6364065e60 msgr2=0x7f6364068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6364065e60 0x7f6364068320 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f637815ba10 tx=0x7f6368008040 comp rx=0 tx=0).stop 2023-10-24T23:55:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 -- 172.21.15.107:0/235005191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637806af70 msgr2=0x7f63781f0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637806af70 0x7f63781f0380 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f636c0af040 tx=0x7f636c019160 comp rx=0 tx=0).stop 2023-10-24T23:55:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 -- 172.21.15.107:0/235005191 shutdown_connections 2023-10-24T23:55:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6364065e60 0x7f6364068320 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f637814b600 0x7f637815a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f637806af70 0x7f63781f0380 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 --2- 172.21.15.107:0/235005191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f637806a230 0x7f63781ec6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 -- 172.21.15.107:0/235005191 >> 172.21.15.107:0/235005191 conn(0x7f63780c2950 msgr2=0x7f637814a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 -- 172.21.15.107:0/235005191 shutdown_connections 2023-10-24T23:55:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:52.835+0000 7f637d444640 1 -- 172.21.15.107:0/235005191 wait complete. 2023-10-24T23:55:52.855 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:55:54.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:54.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.979+0000 7f04d85c9640 1 Processor -- start 2023-10-24T23:55:54.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.979+0000 7f04d85c9640 1 -- start start 2023-10-24T23:55:54.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d85c9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f04d006ee40 0x7f04d00ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:54.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d85c9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d006f730 con 0x7f04d00a9820 2023-10-24T23:55:54.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d85c9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d0156a40 con 0x7f04d014fd40 2023-10-24T23:55:54.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d85c9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d0156710 con 0x7f04d006ee40 2023-10-24T23:55:54.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5b64640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f04d00a9820 0x7f04d015af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47236/0 (socket says 172.21.15.107:47236) 2023-10-24T23:55:54.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5b64640 1 -- 172.21.15.107:0/1962615986 learned_addr learned my addr 172.21.15.107:0/1962615986 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:55:54.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d6b66640 1 --2- 172.21.15.107:0/1962615986 >> v2:172.21.15.110:3300/0 conn(0x7f04d006ee40 0x7f04d00ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:54.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966771379 0 0) 0x7f04d006f730 con 0x7f04d00a9820 2023-10-24T23:55:54.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04bc003530 con 0x7f04d00a9820 2023-10-24T23:55:54.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1444736560 0 0) 0x7f04bc003530 con 0x7f04d00a9820 2023-10-24T23:55:54.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 >> v2:172.21.15.110:3300/0 conn(0x7f04d006ee40 msgr2=0x7f04d00ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:54.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 --2- 172.21.15.107:0/1962615986 >> v2:172.21.15.110:3300/0 conn(0x7f04d006ee40 0x7f04d00ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:54.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 >> v1:172.21.15.110:6789/0 conn(0x7f04d014fd40 legacy=0x7f04d00c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:54.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d006f980 con 0x7f04d00a9820 2023-10-24T23:55:54.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f04c0001680 con 0x7f04d00a9820 2023-10-24T23:55:54.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 >> v1:172.21.15.107:6789/0 conn(0x7f04d00a9820 legacy=0x7f04d015af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:54.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04bc003980 0x7f04bc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:54.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04bc0047d0 0x7f04bc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:54.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04bc0052b0 0x7f04bc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:54.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d0156710 con 0x7f04bc0052b0 2023-10-24T23:55:54.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc00c180 con 0x7f04bc003980 2023-10-24T23:55:54.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc00c340 con 0x7f04bc0047d0 2023-10-24T23:55:54.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d6b66640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04bc0052b0 0x7f04bc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:54.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d6365640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04bc003980 0x7f04bc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:54.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d5b64640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04bc0047d0 0x7f04bc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:54.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d6b66640 1 -- 172.21.15.107:0/1962615986 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04bc0047d0 msgr2=0x7f04bc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:54.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d6b66640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04bc0047d0 0x7f04bc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:54.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d6b66640 1 -- 172.21.15.107:0/1962615986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04bc003980 msgr2=0x7f04bc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:54.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d6b66640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04bc003980 0x7f04bc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:54.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d6b66640 1 -- 172.21.15.107:0/1962615986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04bc003530 con 0x7f04bc0052b0 2023-10-24T23:55:54.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.983+0000 7f04d6365640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04bc003980 0x7f04bc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:55:55.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d6b66640 1 -- 172.21.15.107:0/1962615986 reap_dead start 2023-10-24T23:55:55.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d6b66640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04bc0052b0 0x7f04bc00b990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f04d00725f0 tx=0x7f04c801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce808fcf7ac5f06 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c8004020 con 0x7f04bc0052b0 2023-10-24T23:55:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c8004180 con 0x7f04bc0052b0 2023-10-24T23:55:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d5363640 1 -- 172.21.15.107:0/1962615986 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c8003620 con 0x7f04bc0052b0 2023-10-24T23:55:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 -- 172.21.15.107:0/1962615986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04bc0052b0 msgr2=0x7f04bc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:55.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04bc0052b0 0x7f04bc00b990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f04d00725f0 tx=0x7f04c801ca30 comp rx=0 tx=0).stop 2023-10-24T23:55:55.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 -- 172.21.15.107:0/1962615986 shutdown_connections 2023-10-24T23:55:55.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 --2- 172.21.15.107:0/1962615986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04bc0052b0 0x7f04bc00b990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:55.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 -- 172.21.15.107:0/1962615986 >> 172.21.15.107:0/1962615986 conn(0x7f04d00c2950 msgr2=0x7f04d00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:55.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 -- 172.21.15.107:0/1962615986 shutdown_connections 2023-10-24T23:55:55.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 -- 172.21.15.107:0/1962615986 wait complete. 2023-10-24T23:55:55.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 Processor -- start 2023-10-24T23:55:55.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 -- start start 2023-10-24T23:55:55.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04bc0052b0 0x7f04d014e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:55.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04d00c00d0 0x7f04d01ee950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04d00c3e00 0x7f04d01eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d01eb3e0 con 0x7f04d00c3e00 2023-10-24T23:55:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04d00c3a50 con 0x7f04bc0052b0 2023-10-24T23:55:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d85c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d01eb660 con 0x7f04d00c00d0 2023-10-24T23:55:55.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d6b66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04d00c3e00 0x7f04d01eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:55.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d6b66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04d00c3e00 0x7f04d01eee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46810/0 (socket says 172.21.15.107:46810) 2023-10-24T23:55:55.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d6b66640 1 -- 172.21.15.107:0/499955502 learned_addr learned my addr 172.21.15.107:0/499955502 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:55.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d6365640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04bc0052b0 0x7f04d014e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:55.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d5b64640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04d00c00d0 0x7f04d01ee950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:55.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d6b66640 1 -- 172.21.15.107:0/499955502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04d00c00d0 msgr2=0x7f04d01ee950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d6b66640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04d00c00d0 0x7f04d01ee950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d6b66640 1 -- 172.21.15.107:0/499955502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04bc0052b0 msgr2=0x7f04d014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d6b66640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04bc0052b0 0x7f04d014e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.987+0000 7f04d6b66640 1 -- 172.21.15.107:0/499955502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d01eb940 con 0x7f04d00c3e00 2023-10-24T23:55:55.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04d6b66640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04d00c3e00 0x7f04d01eee90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f04c801d2d0 tx=0x7f04bc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:55.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04c6ffd640 1 -- 172.21.15.107:0/499955502 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c8004020 con 0x7f04d00c3e00 2023-10-24T23:55:55.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04c6ffd640 1 -- 172.21.15.107:0/499955502 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c8003c80 con 0x7f04d00c3e00 2023-10-24T23:55:55.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04d85c9640 1 -- 172.21.15.107:0/499955502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d01ebc20 con 0x7f04d00c3e00 2023-10-24T23:55:55.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04c6ffd640 1 -- 172.21.15.107:0/499955502 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04c80042f0 con 0x7f04d00c3e00 2023-10-24T23:55:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04d85c9640 1 -- 172.21.15.107:0/499955502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d014e7c0 con 0x7f04d00c3e00 2023-10-24T23:55:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04d85c9640 1 -- 172.21.15.107:0/499955502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d00c24a0 con 0x7f04d00c3e00 2023-10-24T23:55:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04c6ffd640 1 -- 172.21.15.107:0/499955502 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f04c80a5520 con 0x7f04d00c3e00 2023-10-24T23:55:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04c6ffd640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04b4065e60 0x7f04b4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04c6ffd640 1 -- 172.21.15.107:0/499955502 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7f04c811d080 con 0x7f04d00c3e00 2023-10-24T23:55:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04d6365640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04b4065e60 0x7f04b4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.991+0000 7f04d6365640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04b4065e60 0x7f04b4068320 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0012f0 tx=0x7f04cc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:55.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:54.995+0000 7f04c6ffd640 1 -- 172.21.15.107:0/499955502 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f04c80ea330 con 0x7f04d00c3e00 2023-10-24T23:55:55.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.183+0000 7f04d85c9640 1 -- 172.21.15.107:0/499955502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04d00746e0 con 0x7f04d00c3e00 2023-10-24T23:55:55.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04c6ffd640 1 -- 172.21.15.107:0/499955502 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7f04c80ed950 con 0x7f04d00c3e00 2023-10-24T23:55:55.190 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:55.190 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:55.191 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-10-24T23:55:55.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 -- 172.21.15.107:0/499955502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04b4065e60 msgr2=0x7f04b4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:55.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04b4065e60 0x7f04b4068320 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f04cc0012f0 tx=0x7f04cc009040 comp rx=0 tx=0).stop 2023-10-24T23:55:55.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 -- 172.21.15.107:0/499955502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04d00c3e00 msgr2=0x7f04d01eee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:55.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04d00c3e00 0x7f04d01eee90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f04c801d2d0 tx=0x7f04bc005800 comp rx=0 tx=0).stop 2023-10-24T23:55:55.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 -- 172.21.15.107:0/499955502 shutdown_connections 2023-10-24T23:55:55.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04d00c3e00 0x7f04d01eee90 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:55.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f04b4065e60 0x7f04b4068320 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:55.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04d00c00d0 0x7f04d01ee950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:55.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 --2- 172.21.15.107:0/499955502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04bc0052b0 0x7f04d014e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:55.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 -- 172.21.15.107:0/499955502 >> 172.21.15.107:0/499955502 conn(0x7f04d00c2950 msgr2=0x7f04d0145980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:55.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 -- 172.21.15.107:0/499955502 shutdown_connections 2023-10-24T23:55:55.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:55.187+0000 7f04d85c9640 1 -- 172.21.15.107:0/499955502 wait complete. 2023-10-24T23:55:55.209 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:55:57.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:57.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.335+0000 7fd5c1e6d640 1 Processor -- start 2023-10-24T23:55:57.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5c1e6d640 1 -- start start 2023-10-24T23:55:57.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5c1e6d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd5bc152060 0x7fd5bc0aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:57.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5c1e6d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5bc0c2990 0x7fd5bc0a9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:57.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5c1e6d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5bc0aa470 0x7fd5bc0c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:57.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5c1e6d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5bc0a9aa0 con 0x7fd5bc0aa470 2023-10-24T23:55:57.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5c1e6d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5bc155480 con 0x7fd5bc152060 2023-10-24T23:55:57.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5c1e6d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc155760 con 0x7fd5bc0c2990 2023-10-24T23:55:57.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5bbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5bc0aa470 0x7fd5bc0c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:57.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5bbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5bc0aa470 0x7fd5bc0c1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46824/0 (socket says 172.21.15.107:46824) 2023-10-24T23:55:57.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5bbfff640 1 -- 172.21.15.107:0/3544211628 learned_addr learned my addr 172.21.15.107:0/3544211628 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:57.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5bb7fe640 1 --2- 172.21.15.107:0/3544211628 >> v2:172.21.15.110:3300/0 conn(0x7fd5bc152060 0x7fd5bc0aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:57.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5b2ffd640 1 --2- 172.21.15.107:0/3544211628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5bc0c2990 0x7fd5bc0a9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:57.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5bbfff640 1 -- 172.21.15.107:0/3544211628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5bc0c2990 msgr2=0x7fd5bc0a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:57.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5bbfff640 1 --2- 172.21.15.107:0/3544211628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5bc0c2990 0x7fd5bc0a9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:57.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5bbfff640 1 -- 172.21.15.107:0/3544211628 >> v2:172.21.15.110:3300/0 conn(0x7fd5bc152060 msgr2=0x7fd5bc0aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:57.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5bbfff640 1 --2- 172.21.15.107:0/3544211628 >> v2:172.21.15.110:3300/0 conn(0x7fd5bc152060 0x7fd5bc0aada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:57.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5bbfff640 1 -- 172.21.15.107:0/3544211628 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5bc0a9450 con 0x7fd5bc0aa470 2023-10-24T23:55:57.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5bb7fe640 1 --2- 172.21.15.107:0/3544211628 >> v2:172.21.15.110:3300/0 conn(0x7fd5bc152060 0x7fd5bc0aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:55:57.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.339+0000 7fd5b2ffd640 1 --2- 172.21.15.107:0/3544211628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5bc0c2990 0x7fd5bc0a9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:55:57.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5bbfff640 1 --2- 172.21.15.107:0/3544211628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5bc0aa470 0x7fd5bc0c1480 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc073d70 tx=0x7fd5b401c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e6e479d3a8cd48 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:57.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5baffd640 1 -- 172.21.15.107:0/3544211628 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40a56c0 con 0x7fd5bc0aa470 2023-10-24T23:55:57.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5baffd640 1 -- 172.21.15.107:0/3544211628 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b40a5820 con 0x7fd5bc0aa470 2023-10-24T23:55:57.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5baffd640 1 -- 172.21.15.107:0/3544211628 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40a5bb0 con 0x7fd5bc0aa470 2023-10-24T23:55:57.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3544211628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5bc0aa470 msgr2=0x7fd5bc0c1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:57.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 --2- 172.21.15.107:0/3544211628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5bc0aa470 0x7fd5bc0c1480 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc073d70 tx=0x7fd5b401c880 comp rx=0 tx=0).stop 2023-10-24T23:55:57.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3544211628 shutdown_connections 2023-10-24T23:55:57.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 --2- 172.21.15.107:0/3544211628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5bc0aa470 0x7fd5bc0c1480 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:57.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 --2- 172.21.15.107:0/3544211628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5bc0c2990 0x7fd5bc0a9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:57.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 --2- 172.21.15.107:0/3544211628 >> v2:172.21.15.110:3300/0 conn(0x7fd5bc152060 0x7fd5bc0aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:57.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3544211628 >> 172.21.15.107:0/3544211628 conn(0x7fd5bc0c4a90 msgr2=0x7fd5bc0c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:57.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3544211628 shutdown_connections 2023-10-24T23:55:57.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3544211628 wait complete. 2023-10-24T23:55:57.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 Processor -- start 2023-10-24T23:55:57.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 -- start start 2023-10-24T23:55:57.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5bc0aa470 0x7fd5bc148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:57.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc0c2990 0x7fd5bc0c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:57.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5bc152060 0x7fd5bc0aa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:57.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc148fc0 con 0x7fd5bc152060 2023-10-24T23:55:57.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc145a50 con 0x7fd5bc0c2990 2023-10-24T23:55:57.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5c1e6d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc151880 con 0x7fd5bc0aa470 2023-10-24T23:55:57.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.343+0000 7fd5bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5bc152060 0x7fd5bc0aa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:57.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5bc152060 0x7fd5bc0aa930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46828/0 (socket says 172.21.15.107:46828) 2023-10-24T23:55:57.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5bbfff640 1 -- 172.21.15.107:0/3368738948 learned_addr learned my addr 172.21.15.107:0/3368738948 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:57.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5bb7fe640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5bc0aa470 0x7fd5bc148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:57.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5b2ffd640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc0c2990 0x7fd5bc0c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:57.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5bbfff640 1 -- 172.21.15.107:0/3368738948 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5bc0aa470 msgr2=0x7fd5bc148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:57.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5bbfff640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5bc0aa470 0x7fd5bc148a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:57.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5bbfff640 1 -- 172.21.15.107:0/3368738948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc0c2990 msgr2=0x7fd5bc0c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:57.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5bbfff640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc0c2990 0x7fd5bc0c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:57.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5bbfff640 1 -- 172.21.15.107:0/3368738948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4009b30 con 0x7fd5bc152060 2023-10-24T23:55:57.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5b2ffd640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc0c2990 0x7fd5bc0c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:55:57.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5bbfff640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5bc152060 0x7fd5bc0aa930 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd5b401af30 tx=0x7fd5b4019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:57.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5b8ff9640 1 -- 172.21.15.107:0/3368738948 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40b8670 con 0x7fd5bc152060 2023-10-24T23:55:57.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5b8ff9640 1 -- 172.21.15.107:0/3368738948 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b40b87f0 con 0x7fd5bc152060 2023-10-24T23:55:57.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5b8ff9640 1 -- 172.21.15.107:0/3368738948 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40b8ba0 con 0x7fd5bc152060 2023-10-24T23:55:57.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3368738948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5bc1494b0 con 0x7fd5bc152060 2023-10-24T23:55:57.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3368738948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5bc149a70 con 0x7fd5bc152060 2023-10-24T23:55:57.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3368738948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5bc0746e0 con 0x7fd5bc152060 2023-10-24T23:55:57.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5b8ff9640 1 -- 172.21.15.107:0/3368738948 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd5b40b8d00 con 0x7fd5bc152060 2023-10-24T23:55:57.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5b8ff9640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd59c065e60 0x7fd59c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:57.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5bb7fe640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd59c065e60 0x7fd59c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:57.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5b8ff9640 1 -- 172.21.15.107:0/3368738948 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 7133+0+0 (secure 0 0 0) 0x7fd5b411f140 con 0x7fd5bc152060 2023-10-24T23:55:57.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.347+0000 7fd5bb7fe640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd59c065e60 0x7fd59c068320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00ba10 tx=0x7fd5ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:57.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.351+0000 7fd5b8ff9640 1 -- 172.21.15.107:0/3368738948 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd5b40ec510 con 0x7fd5bc152060 2023-10-24T23:55:57.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.543+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3368738948 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5bc0c1e90 con 0x7fd5bc152060 2023-10-24T23:55:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.543+0000 7fd5b8ff9640 1 -- 172.21.15.107:0/3368738948 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5769 (secure 0 0 0) 0x7fd5b4009510 con 0x7fd5bc152060 2023-10-24T23:55:57.549 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:57.549 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:53:06.342509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.110:6835/2480841919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2480841919},{"type":"v1","addr":"172.21.15.110:6835","nonce":2480841919}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:57.549 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 23 2023-10-24T23:55:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3368738948 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd59c065e60 msgr2=0x7fd59c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:57.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd59c065e60 0x7fd59c068320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00ba10 tx=0x7fd5ac008040 comp rx=0 tx=0).stop 2023-10-24T23:55:57.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3368738948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5bc152060 msgr2=0x7fd5bc0aa930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:57.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5bc152060 0x7fd5bc0aa930 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd5b401af30 tx=0x7fd5b4019620 comp rx=0 tx=0).stop 2023-10-24T23:55:57.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3368738948 shutdown_connections 2023-10-24T23:55:57.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5bc152060 0x7fd5bc0aa930 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:57.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5bc0c2990 0x7fd5bc0c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:57.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd59c065e60 0x7fd59c068320 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:57.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 --2- 172.21.15.107:0/3368738948 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5bc0aa470 0x7fd5bc148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:57.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3368738948 >> 172.21.15.107:0/3368738948 conn(0x7fd5bc0c4a90 msgr2=0x7fd5bc0c2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:57.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3368738948 shutdown_connections 2023-10-24T23:55:57.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:57.547+0000 7fd5c1e6d640 1 -- 172.21.15.107:0/3368738948 wait complete. 2023-10-24T23:55:57.568 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:55:59.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:55:59.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9db69b6640 1 Processor -- start 2023-10-24T23:55:59.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9db69b6640 1 -- start start 2023-10-24T23:55:59.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9db69b6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9db014b600 0x7f9db0151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:59.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9db69b6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9db006af70 0x7f9db006c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:59.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9db69b6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db00c1860 con 0x7f9db006a230 2023-10-24T23:55:59.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9db69b6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db0154d50 con 0x7f9db006af70 2023-10-24T23:55:59.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9db69b6640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db01524f0 con 0x7f9db014b600 2023-10-24T23:55:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9da7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9db006af70 0x7f9db006c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9da7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9db006af70 0x7f9db006c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46832/0 (socket says 172.21.15.107:46832) 2023-10-24T23:55:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9da7fff640 1 -- 172.21.15.107:0/195868937 learned_addr learned my addr 172.21.15.107:0/195868937 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9daffff640 1 --2- 172.21.15.107:0/195868937 >> v2:172.21.15.110:3300/0 conn(0x7f9db014b600 0x7f9db0151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9da7fff640 1 -- 172.21.15.107:0/195868937 >> v2:172.21.15.110:3300/0 conn(0x7f9db014b600 msgr2=0x7f9db0151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9da7fff640 1 --2- 172.21.15.107:0/195868937 >> v2:172.21.15.110:3300/0 conn(0x7f9db014b600 0x7f9db0151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9da7fff640 1 -- 172.21.15.107:0/195868937 >> v1:172.21.15.110:6789/0 conn(0x7f9db006a230 legacy=0x7f9db01551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:55:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.695+0000 7f9da7fff640 1 -- 172.21.15.107:0/195868937 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db014b9e0 con 0x7f9db006af70 2023-10-24T23:55:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9da7fff640 1 --2- 172.21.15.107:0/195868937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db006af70 0x7f9db006c4c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9db006f860 tx=0x7f9d9801ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e3b47f113073e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:59.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9daf7fe640 1 -- 172.21.15.107:0/195868937 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d980a7aa0 con 0x7f9db006af70 2023-10-24T23:55:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9daf7fe640 1 -- 172.21.15.107:0/195868937 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d980a7c00 con 0x7f9db006af70 2023-10-24T23:55:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9daf7fe640 1 -- 172.21.15.107:0/195868937 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d980af440 con 0x7f9db006af70 2023-10-24T23:55:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 -- 172.21.15.107:0/195868937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db006af70 msgr2=0x7f9db006c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 --2- 172.21.15.107:0/195868937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db006af70 0x7f9db006c4c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9db006f860 tx=0x7f9d9801ead0 comp rx=0 tx=0).stop 2023-10-24T23:55:59.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 -- 172.21.15.107:0/195868937 shutdown_connections 2023-10-24T23:55:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 --2- 172.21.15.107:0/195868937 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db006af70 0x7f9db006c4c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 --2- 172.21.15.107:0/195868937 >> v2:172.21.15.110:3300/0 conn(0x7f9db014b600 0x7f9db0151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 -- 172.21.15.107:0/195868937 >> 172.21.15.107:0/195868937 conn(0x7f9db00c2950 msgr2=0x7f9db00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 -- 172.21.15.107:0/195868937 shutdown_connections 2023-10-24T23:55:59.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 -- 172.21.15.107:0/195868937 wait complete. 2023-10-24T23:55:59.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 Processor -- start 2023-10-24T23:55:59.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 -- start start 2023-10-24T23:55:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9db006a230 0x7f9db00c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db006af70 0x7f9db01500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db014b600 0x7f9db0153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db01eca60 con 0x7f9db006af70 2023-10-24T23:55:59.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db01ecf90 con 0x7f9db014b600 2023-10-24T23:55:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db69b6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db01eccb0 con 0x7f9db006a230 2023-10-24T23:55:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9da7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db006af70 0x7f9db01500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9da7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db006af70 0x7f9db01500a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46836/0 (socket says 172.21.15.107:46836) 2023-10-24T23:55:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9da7fff640 1 -- 172.21.15.107:0/3637473185 learned_addr learned my addr 172.21.15.107:0/3637473185 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:55:59.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9daffff640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9db006a230 0x7f9db00c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9db4f53640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db014b600 0x7f9db0153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9da7fff640 1 -- 172.21.15.107:0/3637473185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9db006a230 msgr2=0x7f9db00c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9da7fff640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9db006a230 0x7f9db00c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9da7fff640 1 -- 172.21.15.107:0/3637473185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db014b600 msgr2=0x7f9db0153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9da7fff640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db014b600 0x7f9db0153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9da7fff640 1 -- 172.21.15.107:0/3637473185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9800b770 con 0x7f9db006af70 2023-10-24T23:55:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.699+0000 7f9da7fff640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db006af70 0x7f9db01500a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9d9801c700 tx=0x7f9d9801b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.703+0000 7f9dad7fa640 1 -- 172.21.15.107:0/3637473185 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d98006ee0 con 0x7f9db006af70 2023-10-24T23:55:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.703+0000 7f9dad7fa640 1 -- 172.21.15.107:0/3637473185 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d980ba640 con 0x7f9db006af70 2023-10-24T23:55:59.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.703+0000 7f9db69b6640 1 -- 172.21.15.107:0/3637473185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db0152990 con 0x7f9db006af70 2023-10-24T23:55:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.703+0000 7f9dad7fa640 1 -- 172.21.15.107:0/3637473185 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d980ba9f0 con 0x7f9db006af70 2023-10-24T23:55:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.703+0000 7f9db69b6640 1 -- 172.21.15.107:0/3637473185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db01505e0 con 0x7f9db006af70 2023-10-24T23:55:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.703+0000 7f9db69b6640 1 -- 172.21.15.107:0/3637473185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db00746e0 con 0x7f9db006af70 2023-10-24T23:55:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.703+0000 7f9dad7fa640 1 -- 172.21.15.107:0/3637473185 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9d980b6040 con 0x7f9db006af70 2023-10-24T23:55:59.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.703+0000 7f9dad7fa640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d90065e60 0x7f9d90068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:55:59.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.703+0000 7f9dad7fa640 1 -- 172.21.15.107:0/3637473185 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7f9d98121270 con 0x7f9db006af70 2023-10-24T23:55:59.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.703+0000 7f9daffff640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d90065e60 0x7f9d90068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:55:59.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.703+0000 7f9daffff640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d90065e60 0x7f9d90068320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9da000b930 tx=0x7f9da0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:55:59.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.707+0000 7f9dad7fa640 1 -- 172.21.15.107:0/3637473185 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9d980ee3a0 con 0x7f9db006af70 2023-10-24T23:55:59.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.899+0000 7f9db69b6640 1 -- 172.21.15.107:0/3637473185 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9db0069fa0 con 0x7f9db006af70 2023-10-24T23:55:59.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.899+0000 7f9dad7fa640 1 -- 172.21.15.107:0/3637473185 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f9d980f19c0 con 0x7f9db006af70 2023-10-24T23:55:59.904 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:55:59.904 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:55:59.437197+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:55:59.905 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 25 2023-10-24T23:55:59.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 -- 172.21.15.107:0/3637473185 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d90065e60 msgr2=0x7f9d90068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:59.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d90065e60 0x7f9d90068320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9da000b930 tx=0x7f9da0008040 comp rx=0 tx=0).stop 2023-10-24T23:55:59.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 -- 172.21.15.107:0/3637473185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db006af70 msgr2=0x7f9db01500a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:55:59.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db006af70 0x7f9db01500a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9d9801c700 tx=0x7f9d9801b870 comp rx=0 tx=0).stop 2023-10-24T23:55:59.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 -- 172.21.15.107:0/3637473185 shutdown_connections 2023-10-24T23:55:59.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9db014b600 0x7f9db0153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:59.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9db006af70 0x7f9db01500a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:59.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d90065e60 0x7f9d90068320 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:59.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 --2- 172.21.15.107:0/3637473185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9db006a230 0x7f9db00c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:55:59.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 -- 172.21.15.107:0/3637473185 >> 172.21.15.107:0/3637473185 conn(0x7f9db00c2950 msgr2=0x7f9db0147e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:55:59.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 -- 172.21.15.107:0/3637473185 shutdown_connections 2023-10-24T23:55:59.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:55:59.903+0000 7f9db69b6640 1 -- 172.21.15.107:0/3637473185 wait complete. 2023-10-24T23:55:59.923 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2023-10-24T23:55:59.923 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-24T23:55:59.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:00.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.039+0000 7fc27792a640 1 Processor -- start 2023-10-24T23:56:00.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.039+0000 7fc27792a640 1 -- start start 2023-10-24T23:56:00.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.039+0000 7fc27792a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc270156770 0x7fc2700ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:00.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc27792a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2700c1ca0 con 0x7fc2700be6b0 2023-10-24T23:56:00.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc27792a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2700c10c0 con 0x7fc2700c70e0 2023-10-24T23:56:00.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc27792a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2700c27e0 con 0x7fc270156770 2023-10-24T23:56:00.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc275ec7640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc2700be6b0 0x7fc27015d130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47238/0 (socket says 172.21.15.107:47238) 2023-10-24T23:56:00.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc275ec7640 1 -- 172.21.15.107:0/1982411771 learned_addr learned my addr 172.21.15.107:0/1982411771 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:56:00.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc2756c6640 1 --2- 172.21.15.107:0/1982411771 >> v2:172.21.15.110:3300/0 conn(0x7fc270156770 0x7fc2700ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:00.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546934086 0 0) 0x7fc2700c1ca0 con 0x7fc2700be6b0 2023-10-24T23:56:00.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc258003530 con 0x7fc2700be6b0 2023-10-24T23:56:00.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 647882063 0 0) 0x7fc258003530 con 0x7fc2700be6b0 2023-10-24T23:56:00.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 >> v2:172.21.15.110:3300/0 conn(0x7fc270156770 msgr2=0x7fc2700ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:00.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 --2- 172.21.15.107:0/1982411771 >> v2:172.21.15.110:3300/0 conn(0x7fc270156770 0x7fc2700ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:00.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 >> v1:172.21.15.110:6789/0 conn(0x7fc2700c70e0 legacy=0x7fc2700bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:00.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2700c13a0 con 0x7fc2700be6b0 2023-10-24T23:56:00.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fc260001680 con 0x7fc2700be6b0 2023-10-24T23:56:00.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 >> v1:172.21.15.107:6789/0 conn(0x7fc2700be6b0 legacy=0x7fc27015d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:00.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 --2- 172.21.15.107:0/1982411771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc258003980 0x7fc258003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:00.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 --2- 172.21.15.107:0/1982411771 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2580047d0 0x7fc258004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:00.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 --2- 172.21.15.107:0/1982411771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2580052b0 0x7fc25800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:00.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25800c150 con 0x7fc258003980 2023-10-24T23:56:00.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25800c2c0 con 0x7fc2580052b0 2023-10-24T23:56:00.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc25800c480 con 0x7fc2580047d0 2023-10-24T23:56:00.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc2756c6640 1 --2- 172.21.15.107:0/1982411771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc258003980 0x7fc258003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:00.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc274ec5640 1 --2- 172.21.15.107:0/1982411771 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2580047d0 0x7fc258004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:00.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc275ec7640 1 --2- 172.21.15.107:0/1982411771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2580052b0 0x7fc25800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:00.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc2756c6640 1 -- 172.21.15.107:0/1982411771 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2580047d0 msgr2=0x7fc258004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:00.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc2756c6640 1 --2- 172.21.15.107:0/1982411771 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2580047d0 0x7fc258004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:00.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc2756c6640 1 -- 172.21.15.107:0/1982411771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2580052b0 msgr2=0x7fc25800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:00.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc2756c6640 1 --2- 172.21.15.107:0/1982411771 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2580052b0 0x7fc25800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:00.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc2756c6640 1 -- 172.21.15.107:0/1982411771 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc258003530 con 0x7fc258003980 2023-10-24T23:56:00.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc275ec7640 1 -- 172.21.15.107:0/1982411771 reap_dead start 2023-10-24T23:56:00.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.043+0000 7fc2756c6640 1 --2- 172.21.15.107:0/1982411771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc258003980 0x7fc258003da0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc26c009f90 tx=0x7fc26c01cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9977548a6d0d94 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:00.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c0a5b20 con 0x7fc258003980 2023-10-24T23:56:00.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26c00f040 con 0x7fc258003980 2023-10-24T23:56:00.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc267fff640 1 -- 172.21.15.107:0/1982411771 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c0ad440 con 0x7fc258003980 2023-10-24T23:56:00.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 -- 172.21.15.107:0/1982411771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc258003980 msgr2=0x7fc258003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:00.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 --2- 172.21.15.107:0/1982411771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc258003980 0x7fc258003da0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc26c009f90 tx=0x7fc26c01cbb0 comp rx=0 tx=0).stop 2023-10-24T23:56:00.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 -- 172.21.15.107:0/1982411771 shutdown_connections 2023-10-24T23:56:00.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 --2- 172.21.15.107:0/1982411771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc258003980 0x7fc258003da0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:00.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 -- 172.21.15.107:0/1982411771 >> 172.21.15.107:0/1982411771 conn(0x7fc2700c65c0 msgr2=0x7fc2700c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:00.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 -- 172.21.15.107:0/1982411771 shutdown_connections 2023-10-24T23:56:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 -- 172.21.15.107:0/1982411771 wait complete. 2023-10-24T23:56:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 Processor -- start 2023-10-24T23:56:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 -- start start 2023-10-24T23:56:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc258003980 0x7fc2701ee050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc270151b70 0x7fc2701ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:00.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc270152ee0 0x7fc270150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:00.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2701eeb70 con 0x7fc270152ee0 2023-10-24T23:56:00.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2700a7870 con 0x7fc258003980 2023-10-24T23:56:00.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc27792a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2700a7b50 con 0x7fc270151b70 2023-10-24T23:56:00.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc275ec7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc270152ee0 0x7fc270150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:00.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc275ec7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc270152ee0 0x7fc270150fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46852/0 (socket says 172.21.15.107:46852) 2023-10-24T23:56:00.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc275ec7640 1 -- 172.21.15.107:0/3469478020 learned_addr learned my addr 172.21.15.107:0/3469478020 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:00.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc274ec5640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc270151b70 0x7fc2701ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:00.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc2756c6640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc258003980 0x7fc2701ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:00.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc275ec7640 1 -- 172.21.15.107:0/3469478020 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc270151b70 msgr2=0x7fc2701ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc275ec7640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc270151b70 0x7fc2701ee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc275ec7640 1 -- 172.21.15.107:0/3469478020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc258003980 msgr2=0x7fc2701ee050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc275ec7640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc258003980 0x7fc2701ee050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc275ec7640 1 -- 172.21.15.107:0/3469478020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2700a7e30 con 0x7fc270152ee0 2023-10-24T23:56:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc274ec5640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc270151b70 0x7fc2701ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:56:00.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc275ec7640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc270152ee0 0x7fc270150fe0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc27006e350 tx=0x7fc260004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:00.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.047+0000 7fc265ffb640 1 -- 172.21.15.107:0/3469478020 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26000f600 con 0x7fc270152ee0 2023-10-24T23:56:00.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.051+0000 7fc265ffb640 1 -- 172.21.15.107:0/3469478020 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26000f780 con 0x7fc270152ee0 2023-10-24T23:56:00.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.051+0000 7fc265ffb640 1 -- 172.21.15.107:0/3469478020 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26000fb30 con 0x7fc270152ee0 2023-10-24T23:56:00.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.051+0000 7fc27792a640 1 -- 172.21.15.107:0/3469478020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc27009dd90 con 0x7fc270152ee0 2023-10-24T23:56:00.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.051+0000 7fc27792a640 1 -- 172.21.15.107:0/3469478020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2701516f0 con 0x7fc270152ee0 2023-10-24T23:56:00.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.051+0000 7fc27792a640 1 -- 172.21.15.107:0/3469478020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2700746e0 con 0x7fc270152ee0 2023-10-24T23:56:00.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.051+0000 7fc265ffb640 1 -- 172.21.15.107:0/3469478020 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc26000fc90 con 0x7fc270152ee0 2023-10-24T23:56:00.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.051+0000 7fc265ffb640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc25c065e60 0x7fc25c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:00.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.051+0000 7fc265ffb640 1 -- 172.21.15.107:0/3469478020 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7fc260083a10 con 0x7fc270152ee0 2023-10-24T23:56:00.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.051+0000 7fc2756c6640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc25c065e60 0x7fc25c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.051+0000 7fc2756c6640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc25c065e60 0x7fc25c068320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc26c0af000 tx=0x7fc26c0a7040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.055+0000 7fc265ffb640 1 -- 172.21.15.107:0/3469478020 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc260050c60 con 0x7fc270152ee0 2023-10-24T23:56:00.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.251+0000 7fc27792a640 1 -- 172.21.15.107:0/3469478020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc270065a00 con 0x7fc270152ee0 2023-10-24T23:56:00.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc265ffb640 1 -- 172.21.15.107:0/3469478020 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5015 (secure 0 0 0) 0x7fc260054280 con 0x7fc270152ee0 2023-10-24T23:56:00.258 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:00.258 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:55:59.437197+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:00.259 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 25 2023-10-24T23:56:00.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 -- 172.21.15.107:0/3469478020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc25c065e60 msgr2=0x7fc25c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:00.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc25c065e60 0x7fc25c068320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc26c0af000 tx=0x7fc26c0a7040 comp rx=0 tx=0).stop 2023-10-24T23:56:00.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 -- 172.21.15.107:0/3469478020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc270152ee0 msgr2=0x7fc270150fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc270152ee0 0x7fc270150fe0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc27006e350 tx=0x7fc260004f90 comp rx=0 tx=0).stop 2023-10-24T23:56:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 -- 172.21.15.107:0/3469478020 shutdown_connections 2023-10-24T23:56:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc270152ee0 0x7fc270150fe0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc25c065e60 0x7fc25c068320 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc270151b70 0x7fc2701ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 --2- 172.21.15.107:0/3469478020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc258003980 0x7fc2701ee050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 -- 172.21.15.107:0/3469478020 >> 172.21.15.107:0/3469478020 conn(0x7fc2700c65c0 msgr2=0x7fc270154bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 -- 172.21.15.107:0/3469478020 shutdown_connections 2023-10-24T23:56:00.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:00.255+0000 7fc27792a640 1 -- 172.21.15.107:0/3469478020 wait complete. 2023-10-24T23:56:00.278 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5403": { "addr": "172.21.15.107:6835/1004926771", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 1004926771, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 1004926771, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5403, "incarnation": 24, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:55:59.437197+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5403 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5292": { "addr": "172.21.15.107:6837/838886697", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 838886697, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 838886697, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5292, "incarnation": 18, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-24T23:55:27.587712+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5292 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/3721101226", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 3721101226, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 3721101226, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4232, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-24T23:56:01.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:01.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76d2698640 1 Processor -- start 2023-10-24T23:56:01.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76d2698640 1 -- start start 2023-10-24T23:56:01.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76d2698640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f76cc152040 0x7f76cc158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:01.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76d2698640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc152420 con 0x7f76cc06e800 2023-10-24T23:56:01.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76d2698640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76cc154d40 con 0x7f76cc0adb80 2023-10-24T23:56:01.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76d2698640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76cc1556e0 con 0x7f76cc152040 2023-10-24T23:56:01.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76d0c35640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f76cc06e800 0x7f76cc06ec90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47250/0 (socket says 172.21.15.107:47250) 2023-10-24T23:56:01.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76d0c35640 1 -- 172.21.15.107:0/3440117037 learned_addr learned my addr 172.21.15.107:0/3440117037 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:56:01.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cbfff640 1 --2- 172.21.15.107:0/3440117037 >> v2:172.21.15.110:3300/0 conn(0x7f76cc152040 0x7f76cc158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:01.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476423868 0 0) 0x7f76cc152420 con 0x7f76cc06e800 2023-10-24T23:56:01.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b8003530 con 0x7f76cc06e800 2023-10-24T23:56:01.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2407312624 0 0) 0x7f76b8003530 con 0x7f76cc06e800 2023-10-24T23:56:01.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 >> v2:172.21.15.110:3300/0 conn(0x7f76cc152040 msgr2=0x7f76cc158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:01.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 --2- 172.21.15.107:0/3440117037 >> v2:172.21.15.110:3300/0 conn(0x7f76cc152040 0x7f76cc158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:01.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 >> v1:172.21.15.110:6789/0 conn(0x7f76cc0adb80 legacy=0x7f76cc071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:01.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc155020 con 0x7f76cc06e800 2023-10-24T23:56:01.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f76c4001a00 con 0x7f76cc06e800 2023-10-24T23:56:01.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 >> v1:172.21.15.107:6789/0 conn(0x7f76cc06e800 legacy=0x7f76cc06ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:01.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 --2- 172.21.15.107:0/3440117037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b8003980 0x7f76b8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:01.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 --2- 172.21.15.107:0/3440117037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b80047d0 0x7f76b8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:01.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 --2- 172.21.15.107:0/3440117037 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76b8005190 0x7f76b800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:01.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc1556e0 con 0x7f76b80047d0 2023-10-24T23:56:01.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b800c130 con 0x7f76b8003980 2023-10-24T23:56:01.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b800c2f0 con 0x7f76b8005190 2023-10-24T23:56:01.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76c37fe640 1 --2- 172.21.15.107:0/3440117037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b80047d0 0x7f76b8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:01.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76cbfff640 1 --2- 172.21.15.107:0/3440117037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b8003980 0x7f76b8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:01.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76d0c35640 1 --2- 172.21.15.107:0/3440117037 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76b8005190 0x7f76b800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:01.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76c37fe640 1 -- 172.21.15.107:0/3440117037 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76b8005190 msgr2=0x7f76b800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:01.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76c37fe640 1 --2- 172.21.15.107:0/3440117037 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76b8005190 0x7f76b800b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:01.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76c37fe640 1 -- 172.21.15.107:0/3440117037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b8003980 msgr2=0x7f76b8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:01.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76c37fe640 1 --2- 172.21.15.107:0/3440117037 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b8003980 0x7f76b8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:01.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76c37fe640 1 -- 172.21.15.107:0/3440117037 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b8003530 con 0x7f76b80047d0 2023-10-24T23:56:01.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.815+0000 7f76d0c35640 1 -- 172.21.15.107:0/3440117037 reap_dead start 2023-10-24T23:56:01.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76c37fe640 1 --2- 172.21.15.107:0/3440117037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b80047d0 0x7f76b8004c50 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f76b4001780 tx=0x7f76b401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e949016bcad82d00 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:01.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b40050d0 con 0x7f76b80047d0 2023-10-24T23:56:01.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b40a7050 con 0x7f76b80047d0 2023-10-24T23:56:01.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76cb7fe640 1 -- 172.21.15.107:0/3440117037 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b4005490 con 0x7f76b80047d0 2023-10-24T23:56:01.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76d2698640 1 -- 172.21.15.107:0/3440117037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b80047d0 msgr2=0x7f76b8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:01.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76d2698640 1 --2- 172.21.15.107:0/3440117037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b80047d0 0x7f76b8004c50 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f76b4001780 tx=0x7f76b401c3b0 comp rx=0 tx=0).stop 2023-10-24T23:56:01.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76d2698640 1 -- 172.21.15.107:0/3440117037 shutdown_connections 2023-10-24T23:56:01.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76d2698640 1 --2- 172.21.15.107:0/3440117037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76b80047d0 0x7f76b8004c50 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76d2698640 1 -- 172.21.15.107:0/3440117037 >> 172.21.15.107:0/3440117037 conn(0x7f76cc070f60 msgr2=0x7f76cc071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76d2698640 1 -- 172.21.15.107:0/3440117037 shutdown_connections 2023-10-24T23:56:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76d2698640 1 -- 172.21.15.107:0/3440117037 wait complete. 2023-10-24T23:56:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76d2698640 1 Processor -- start 2023-10-24T23:56:01.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.819+0000 7f76d2698640 1 -- start start 2023-10-24T23:56:01.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76d2698640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b80047d0 0x7f76cc156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:01.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76d2698640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76cc155d20 0x7f76cc0c6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:01.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76d2698640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76cc0c75a0 0x7f76cc14d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:01.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76d2698640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc0c5ec0 con 0x7f76cc155d20 2023-10-24T23:56:01.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76d2698640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc156c90 con 0x7f76b80047d0 2023-10-24T23:56:01.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76d2698640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76cc1ed920 con 0x7f76cc0c75a0 2023-10-24T23:56:01.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76cc155d20 0x7f76cc0c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:01.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76cc155d20 0x7f76cc0c6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46864/0 (socket says 172.21.15.107:46864) 2023-10-24T23:56:01.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c37fe640 1 -- 172.21.15.107:0/185038938 learned_addr learned my addr 172.21.15.107:0/185038938 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:01.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76cbfff640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b80047d0 0x7f76cc156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:01.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76d0c35640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76cc0c75a0 0x7f76cc14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:01.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c37fe640 1 -- 172.21.15.107:0/185038938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76cc0c75a0 msgr2=0x7f76cc14d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:01.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c37fe640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76cc0c75a0 0x7f76cc14d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:01.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c37fe640 1 -- 172.21.15.107:0/185038938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b80047d0 msgr2=0x7f76cc156750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:01.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c37fe640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b80047d0 0x7f76cc156750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:01.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c37fe640 1 -- 172.21.15.107:0/185038938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc1edbc0 con 0x7f76cc155d20 2023-10-24T23:56:01.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c37fe640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76cc155d20 0x7f76cc0c6910 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f76b401c1e0 tx=0x7f76b8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:01.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c97fa640 1 -- 172.21.15.107:0/185038938 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b40072f0 con 0x7f76cc155d20 2023-10-24T23:56:01.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c97fa640 1 -- 172.21.15.107:0/185038938 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b4007450 con 0x7f76cc155d20 2023-10-24T23:56:01.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76d2698640 1 -- 172.21.15.107:0/185038938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76cc0c6110 con 0x7f76cc155d20 2023-10-24T23:56:01.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c97fa640 1 -- 172.21.15.107:0/185038938 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b40067e0 con 0x7f76cc155d20 2023-10-24T23:56:01.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76d2698640 1 -- 172.21.15.107:0/185038938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76cc0c6e50 con 0x7f76cc155d20 2023-10-24T23:56:01.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76d2698640 1 -- 172.21.15.107:0/185038938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc0746e0 con 0x7f76cc155d20 2023-10-24T23:56:01.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.823+0000 7f76c97fa640 1 -- 172.21.15.107:0/185038938 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f76b4006940 con 0x7f76cc155d20 2023-10-24T23:56:01.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.827+0000 7f76c97fa640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76ac065e90 0x7f76ac068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:01.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.827+0000 7f76c97fa640 1 -- 172.21.15.107:0/185038938 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7f76b411ddc0 con 0x7f76cc155d20 2023-10-24T23:56:01.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.827+0000 7f76cbfff640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76ac065e90 0x7f76ac068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:01.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.827+0000 7f76cbfff640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76ac065e90 0x7f76ac068350 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f76cc1e2d90 tx=0x7f76bc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:01.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:01.831+0000 7f76c97fa640 1 -- 172.21.15.107:0/185038938 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f76b40eb010 con 0x7f76cc155d20 2023-10-24T23:56:02.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 -- 172.21.15.107:0/185038938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76cc0ad960 con 0x7f76cc155d20 2023-10-24T23:56:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76c97fa640 1 -- 172.21.15.107:0/185038938 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f76b40ee630 con 0x7f76cc155d20 2023-10-24T23:56:02.091 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:02.091 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:02.092 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2023-10-24T23:56:02.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 -- 172.21.15.107:0/185038938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76ac065e90 msgr2=0x7f76ac068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76ac065e90 0x7f76ac068350 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f76cc1e2d90 tx=0x7f76bc008040 comp rx=0 tx=0).stop 2023-10-24T23:56:02.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 -- 172.21.15.107:0/185038938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76cc155d20 msgr2=0x7f76cc0c6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76cc155d20 0x7f76cc0c6910 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f76b401c1e0 tx=0x7f76b8005800 comp rx=0 tx=0).stop 2023-10-24T23:56:02.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 -- 172.21.15.107:0/185038938 shutdown_connections 2023-10-24T23:56:02.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76ac065e90 0x7f76ac068350 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76cc0c75a0 0x7f76cc14d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76cc155d20 0x7f76cc0c6910 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 --2- 172.21.15.107:0/185038938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76b80047d0 0x7f76cc156750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 -- 172.21.15.107:0/185038938 >> 172.21.15.107:0/185038938 conn(0x7f76cc070f60 msgr2=0x7f76cc14ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:02.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 -- 172.21.15.107:0/185038938 shutdown_connections 2023-10-24T23:56:02.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.087+0000 7f76d2698640 1 -- 172.21.15.107:0/185038938 wait complete. 2023-10-24T23:56:02.115 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2023-10-24T23:56:02.116 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-24T23:56:02.116 INFO:teuthology.orchestra.run:waiting for 300 2023-10-24T23:56:02.176 INFO:tasks.ceph.mds.a:Stopped 2023-10-24T23:56:02.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:02.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.291+0000 7fef4da7a640 1 Processor -- start 2023-10-24T23:56:02.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.291+0000 7fef4da7a640 1 -- start start 2023-10-24T23:56:02.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.291+0000 7fef4da7a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef4813dc20 0x7fef480c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.291+0000 7fef4da7a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fef480720c0 0x7fef4806df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.291+0000 7fef4da7a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef48154860 con 0x7fef4806e470 2023-10-24T23:56:02.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.291+0000 7fef4da7a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef480c1f10 con 0x7fef480720c0 2023-10-24T23:56:02.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.291+0000 7fef4da7a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef48154b20 con 0x7fef4813dc20 2023-10-24T23:56:02.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.295+0000 7fef477fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef4813dc20 0x7fef480c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.295+0000 7fef46ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fef480720c0 0x7fef4806df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.295+0000 7fef46ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fef480720c0 0x7fef4806df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35854/0 (socket says 172.21.15.107:35854) 2023-10-24T23:56:02.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.295+0000 7fef46ffd640 1 -- 172.21.15.107:0/1070664383 learned_addr learned my addr 172.21.15.107:0/1070664383 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:02.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.295+0000 7fef46ffd640 1 -- 172.21.15.107:0/1070664383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef4813dc20 msgr2=0x7fef480c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:02.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.295+0000 7fef46ffd640 1 --2- 172.21.15.107:0/1070664383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef4813dc20 0x7fef480c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.295+0000 7fef46ffd640 1 -- 172.21.15.107:0/1070664383 >> v1:172.21.15.110:6789/0 conn(0x7fef4806e470 legacy=0x7fef48157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.295+0000 7fef46ffd640 1 -- 172.21.15.107:0/1070664383 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef48154e00 con 0x7fef480720c0 2023-10-24T23:56:02.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.295+0000 7fef46ffd640 1 --2- 172.21.15.107:0/1070664383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef480720c0 0x7fef4806df30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009870 tx=0x7fef3c01c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b82710173892f46 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:02.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef467fc640 1 -- 172.21.15.107:0/1070664383 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c0ab600 con 0x7fef480720c0 2023-10-24T23:56:02.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef467fc640 1 -- 172.21.15.107:0/1070664383 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3c0a7050 con 0x7fef480720c0 2023-10-24T23:56:02.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef467fc640 1 -- 172.21.15.107:0/1070664383 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3c0ab980 con 0x7fef480720c0 2023-10-24T23:56:02.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 -- 172.21.15.107:0/1070664383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef480720c0 msgr2=0x7fef4806df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 --2- 172.21.15.107:0/1070664383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef480720c0 0x7fef4806df30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fef3c009870 tx=0x7fef3c01c420 comp rx=0 tx=0).stop 2023-10-24T23:56:02.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 -- 172.21.15.107:0/1070664383 shutdown_connections 2023-10-24T23:56:02.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 --2- 172.21.15.107:0/1070664383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef480720c0 0x7fef4806df30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 --2- 172.21.15.107:0/1070664383 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef4813dc20 0x7fef480c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 -- 172.21.15.107:0/1070664383 >> 172.21.15.107:0/1070664383 conn(0x7fef480c39a0 msgr2=0x7fef480c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:02.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 -- 172.21.15.107:0/1070664383 shutdown_connections 2023-10-24T23:56:02.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 -- 172.21.15.107:0/1070664383 wait complete. 2023-10-24T23:56:02.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 Processor -- start 2023-10-24T23:56:02.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 -- start start 2023-10-24T23:56:02.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef4806e470 0x7fef4815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef480720c0 0x7fef48135ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef4813dc20 0x7fef480c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef480ab3a0 con 0x7fef4813dc20 2023-10-24T23:56:02.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4813bb10 con 0x7fef480720c0 2023-10-24T23:56:02.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef4da7a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef481360e0 con 0x7fef4806e470 2023-10-24T23:56:02.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef47fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef4813dc20 0x7fef480c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef47fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef4813dc20 0x7fef480c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46872/0 (socket says 172.21.15.107:46872) 2023-10-24T23:56:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef47fff640 1 -- 172.21.15.107:0/2134056374 learned_addr learned my addr 172.21.15.107:0/2134056374 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef477fe640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef4806e470 0x7fef4815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.299+0000 7fef46ffd640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef480720c0 0x7fef48135ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef47fff640 1 -- 172.21.15.107:0/2134056374 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef4806e470 msgr2=0x7fef4815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef47fff640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef4806e470 0x7fef4815e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef47fff640 1 -- 172.21.15.107:0/2134056374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef480720c0 msgr2=0x7fef48135ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef47fff640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef480720c0 0x7fef48135ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef47fff640 1 -- 172.21.15.107:0/2134056374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c0094d0 con 0x7fef4813dc20 2023-10-24T23:56:02.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef47fff640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef4813dc20 0x7fef480c63b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fef4000c730 tx=0x7fef4000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:02.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef2ffff640 1 -- 172.21.15.107:0/2134056374 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef40003b80 con 0x7fef4813dc20 2023-10-24T23:56:02.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef2ffff640 1 -- 172.21.15.107:0/2134056374 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef40003ce0 con 0x7fef4813dc20 2023-10-24T23:56:02.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef4da7a640 1 -- 172.21.15.107:0/2134056374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4813bdb0 con 0x7fef4813dc20 2023-10-24T23:56:02.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef2ffff640 1 -- 172.21.15.107:0/2134056374 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4000aa30 con 0x7fef4813dc20 2023-10-24T23:56:02.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef4da7a640 1 -- 172.21.15.107:0/2134056374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4813c370 con 0x7fef4813dc20 2023-10-24T23:56:02.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef4da7a640 1 -- 172.21.15.107:0/2134056374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4813c560 con 0x7fef4813dc20 2023-10-24T23:56:02.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef2ffff640 1 -- 172.21.15.107:0/2134056374 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fef4000ab90 con 0x7fef4813dc20 2023-10-24T23:56:02.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef2ffff640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef34065e60 0x7fef34068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef2ffff640 1 -- 172.21.15.107:0/2134056374 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7fef40083890 con 0x7fef4813dc20 2023-10-24T23:56:02.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef477fe640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef34065e60 0x7fef34068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.303+0000 7fef477fe640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef34065e60 0x7fef34068320 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fef48154520 tx=0x7fef38009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:02.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.311+0000 7fef2ffff640 1 -- 172.21.15.107:0/2134056374 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fef40050b90 con 0x7fef4813dc20 2023-10-24T23:56:02.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9fe2de640 1 Processor -- start 2023-10-24T23:56:02.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9fe2de640 1 -- start start 2023-10-24T23:56:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9fe2de640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc9f80c70e0 0x7fc9f80bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9fe2de640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f80c1ca0 con 0x7fc9f8156770 2023-10-24T23:56:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9fe2de640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f80c10c0 con 0x7fc9f80c2130 2023-10-24T23:56:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9fe2de640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f80c27e0 con 0x7fc9f80c70e0 2023-10-24T23:56:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc9f8156770 0x7fc9f80ab7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47254/0 (socket says 172.21.15.107:47254) 2023-10-24T23:56:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f7fff640 1 -- 172.21.15.107:0/4262054463 learned_addr learned my addr 172.21.15.107:0/4262054463 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:56:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f77fe640 1 --2- 172.21.15.107:0/4262054463 >> v2:172.21.15.110:3300/0 conn(0x7fc9f80c70e0 0x7fc9f80bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167519910 0 0) 0x7fc9f80c1ca0 con 0x7fc9f8156770 2023-10-24T23:56:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e0003530 con 0x7fc9f8156770 2023-10-24T23:56:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1217171837 0 0) 0x7fc9e0003530 con 0x7fc9f8156770 2023-10-24T23:56:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 >> v2:172.21.15.110:3300/0 conn(0x7fc9f80c70e0 msgr2=0x7fc9f80bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 --2- 172.21.15.107:0/4262054463 >> v2:172.21.15.110:3300/0 conn(0x7fc9f80c70e0 0x7fc9f80bdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 >> v1:172.21.15.110:6789/0 conn(0x7fc9f80c2130 legacy=0x7fc9f815d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f80c13a0 con 0x7fc9f8156770 2023-10-24T23:56:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fc9e8001680 con 0x7fc9f8156770 2023-10-24T23:56:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 >> v1:172.21.15.107:6789/0 conn(0x7fc9f8156770 legacy=0x7fc9f80ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 --2- 172.21.15.107:0/4262054463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9e0003980 0x7fc9e0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 --2- 172.21.15.107:0/4262054463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 0x7fc9e0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 --2- 172.21.15.107:0/4262054463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9e00052b0 0x7fc9e000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e000c150 con 0x7fc9e00047d0 2023-10-24T23:56:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e000c2c0 con 0x7fc9e0003980 2023-10-24T23:56:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e000c480 con 0x7fc9e00052b0 2023-10-24T23:56:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f77fe640 1 --2- 172.21.15.107:0/4262054463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 0x7fc9e0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9fc87b640 1 --2- 172.21.15.107:0/4262054463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9e00052b0 0x7fc9e000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f7fff640 1 --2- 172.21.15.107:0/4262054463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9e0003980 0x7fc9e0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f77fe640 1 -- 172.21.15.107:0/4262054463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9e00052b0 msgr2=0x7fc9e000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f77fe640 1 --2- 172.21.15.107:0/4262054463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9e00052b0 0x7fc9e000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f77fe640 1 -- 172.21.15.107:0/4262054463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9e0003980 msgr2=0x7fc9e0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:02.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f77fe640 1 --2- 172.21.15.107:0/4262054463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9e0003980 0x7fc9e0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9fc87b640 1 -- 172.21.15.107:0/4262054463 reap_dead start 2023-10-24T23:56:02.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.395+0000 7fc9f77fe640 1 -- 172.21.15.107:0/4262054463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e0003530 con 0x7fc9e00047d0 2023-10-24T23:56:02.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.399+0000 7fc9f77fe640 1 --2- 172.21.15.107:0/4262054463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 0x7fc9e0004c50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec009ab0 tx=0x7fc9ec01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26eb3deda0b11a5f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:02.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.399+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec0a5590 con 0x7fc9e00047d0 2023-10-24T23:56:02.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.399+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ec0a56f0 con 0x7fc9e00047d0 2023-10-24T23:56:02.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.399+0000 7fc9f6ffd640 1 -- 172.21.15.107:0/4262054463 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec0a58e0 con 0x7fc9e00047d0 2023-10-24T23:56:02.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.399+0000 7fc9fe2de640 1 -- 172.21.15.107:0/4262054463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 msgr2=0x7fc9e0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.399+0000 7fc9fe2de640 1 --2- 172.21.15.107:0/4262054463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 0x7fc9e0004c50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec009ab0 tx=0x7fc9ec01ca00 comp rx=0 tx=0).stop 2023-10-24T23:56:02.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.399+0000 7fc9fe2de640 1 -- 172.21.15.107:0/4262054463 shutdown_connections 2023-10-24T23:56:02.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.399+0000 7fc9fe2de640 1 --2- 172.21.15.107:0/4262054463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 0x7fc9e0004c50 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.399+0000 7fc9fe2de640 1 -- 172.21.15.107:0/4262054463 >> 172.21.15.107:0/4262054463 conn(0x7fc9f80c65c0 msgr2=0x7fc9f80c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:02.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.399+0000 7fc9fe2de640 1 -- 172.21.15.107:0/4262054463 shutdown_connections 2023-10-24T23:56:02.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.399+0000 7fc9fe2de640 1 -- 172.21.15.107:0/4262054463 wait complete. 2023-10-24T23:56:02.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fe2de640 1 Processor -- start 2023-10-24T23:56:02.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fe2de640 1 -- start start 2023-10-24T23:56:02.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fe2de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 0x7fc9f81eb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fe2de640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9f8151b50 0x7fc9f8150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fe2de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9f81eac10 0x7fc9f80bf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fe2de640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f81519e0 con 0x7fc9e00047d0 2023-10-24T23:56:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fe2de640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80a0300 con 0x7fc9f8151b50 2023-10-24T23:56:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fe2de640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f81ea450 con 0x7fc9f81eac10 2023-10-24T23:56:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 0x7fc9f81eb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 0x7fc9f81eb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46886/0 (socket says 172.21.15.107:46886) 2023-10-24T23:56:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f7fff640 1 -- 172.21.15.107:0/3137641887 learned_addr learned my addr 172.21.15.107:0/3137641887 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f77fe640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9f8151b50 0x7fc9f8150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fc87b640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9f81eac10 0x7fc9f80bf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f7fff640 1 -- 172.21.15.107:0/3137641887 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9f81eac10 msgr2=0x7fc9f80bf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f7fff640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9f81eac10 0x7fc9f80bf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f7fff640 1 -- 172.21.15.107:0/3137641887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9f8151b50 msgr2=0x7fc9f8150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f7fff640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9f8151b50 0x7fc9f8150db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f7fff640 1 -- 172.21.15.107:0/3137641887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f81ee460 con 0x7fc9e00047d0 2023-10-24T23:56:02.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fc87b640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9f81eac10 0x7fc9f80bf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:56:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f7fff640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 0x7fc9f81eb7b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800e7b0 tx=0x7fc9e800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f4ff9640 1 -- 172.21.15.107:0/3137641887 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e800ee60 con 0x7fc9e00047d0 2023-10-24T23:56:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f4ff9640 1 -- 172.21.15.107:0/3137641887 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9e800c870 con 0x7fc9e00047d0 2023-10-24T23:56:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fe2de640 1 -- 172.21.15.107:0/3137641887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f81ea6f0 con 0x7fc9e00047d0 2023-10-24T23:56:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f4ff9640 1 -- 172.21.15.107:0/3137641887 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e800cc00 con 0x7fc9e00047d0 2023-10-24T23:56:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fe2de640 1 -- 172.21.15.107:0/3137641887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f81512f0 con 0x7fc9e00047d0 2023-10-24T23:56:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9fe2de640 1 -- 172.21.15.107:0/3137641887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f81ea930 con 0x7fc9e00047d0 2023-10-24T23:56:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.403+0000 7fc9f4ff9640 1 -- 172.21.15.107:0/3137641887 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc9e8013420 con 0x7fc9e00047d0 2023-10-24T23:56:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.407+0000 7fc9f4ff9640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9e4065e90 0x7fc9e4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.407+0000 7fc9f4ff9640 1 -- 172.21.15.107:0/3137641887 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7fc9e80836d0 con 0x7fc9e00047d0 2023-10-24T23:56:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.407+0000 7fc9f77fe640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9e4065e90 0x7fc9e4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.407+0000 7fc9f77fe640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9e4065e90 0x7fc9e4068350 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0af000 tx=0x7fc9ec0046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.411+0000 7fc9f4ff9640 1 -- 172.21.15.107:0/3137641887 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc9f81ea930 con 0x7fc9e00047d0 2023-10-24T23:56:02.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.555+0000 7fef4da7a640 1 -- 172.21.15.107:0/2134056374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef480746e0 con 0x7fef4813dc20 2023-10-24T23:56:02.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.555+0000 7fef2ffff640 1 -- 172.21.15.107:0/2134056374 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5013 (secure 0 0 0) 0x7fef400541b0 con 0x7fef4813dc20 2023-10-24T23:56:02.560 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:02.560 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:02.561 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2023-10-24T23:56:02.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 -- 172.21.15.107:0/2134056374 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef34065e60 msgr2=0x7fef34068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef34065e60 0x7fef34068320 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fef48154520 tx=0x7fef38009190 comp rx=0 tx=0).stop 2023-10-24T23:56:02.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 -- 172.21.15.107:0/2134056374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef4813dc20 msgr2=0x7fef480c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef4813dc20 0x7fef480c63b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fef4000c730 tx=0x7fef4000cbf0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 -- 172.21.15.107:0/2134056374 shutdown_connections 2023-10-24T23:56:02.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef4813dc20 0x7fef480c63b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef480720c0 0x7fef48135ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef34065e60 0x7fef34068320 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 --2- 172.21.15.107:0/2134056374 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef4806e470 0x7fef4815e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 -- 172.21.15.107:0/2134056374 >> 172.21.15.107:0/2134056374 conn(0x7fef480c39a0 msgr2=0x7fef48070770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:02.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 -- 172.21.15.107:0/2134056374 shutdown_connections 2023-10-24T23:56:02.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.559+0000 7fef4da7a640 1 -- 172.21.15.107:0/2134056374 wait complete. 2023-10-24T23:56:02.582 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:56:02.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.635+0000 7fc9fe2de640 1 -- 172.21.15.107:0/3137641887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9f81ea930 con 0x7fc9e00047d0 2023-10-24T23:56:02.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.635+0000 7fc9f4ff9640 1 -- 172.21.15.107:0/3137641887 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5013 (secure 0 0 0) 0x7fc9f81ea930 con 0x7fc9e00047d0 2023-10-24T23:56:02.640 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:02.640 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:02.641 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2023-10-24T23:56:02.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 -- 172.21.15.107:0/3137641887 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9e4065e90 msgr2=0x7fc9e4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9e4065e90 0x7fc9e4068350 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0af000 tx=0x7fc9ec0046d0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 -- 172.21.15.107:0/3137641887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 msgr2=0x7fc9f81eb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:02.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 0x7fc9f81eb7b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800e7b0 tx=0x7fc9e800ec70 comp rx=0 tx=0).stop 2023-10-24T23:56:02.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 -- 172.21.15.107:0/3137641887 shutdown_connections 2023-10-24T23:56:02.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9e4065e90 0x7fc9e4068350 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9f81eac10 0x7fc9f80bf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9f8151b50 0x7fc9f8150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 --2- 172.21.15.107:0/3137641887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9e00047d0 0x7fc9f81eb7b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:02.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 -- 172.21.15.107:0/3137641887 >> 172.21.15.107:0/3137641887 conn(0x7fc9f80c65c0 msgr2=0x7fc9f80c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:02.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 -- 172.21.15.107:0/3137641887 shutdown_connections 2023-10-24T23:56:02.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:02.639+0000 7fc9fe2de640 1 -- 172.21.15.107:0/3137641887 wait complete. 2023-10-24T23:56:02.657 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=4299 2023-10-24T23:56:02.658 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.d 2023-10-24T23:56:04.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:04.610 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-24T23:56:04.606+0000 7f6348a3c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:56:04.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b2871a640 1 Processor -- start 2023-10-24T23:56:04.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b2871a640 1 -- start start 2023-10-24T23:56:04.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b2871a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b20152020 0x7f4b200c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b2871a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4b200c8640 0x7f4b2006a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b2871a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b20152400 con 0x7f4b201553d0 2023-10-24T23:56:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b2871a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b201558d0 con 0x7f4b200c8640 2023-10-24T23:56:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b2871a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200c6ce0 con 0x7f4b20152020 2023-10-24T23:56:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b25cb5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4b200c8640 0x7f4b2006a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b25cb5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4b200c8640 0x7f4b2006a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34114/0 (socket says 172.21.15.107:34114) 2023-10-24T23:56:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b25cb5640 1 -- 172.21.15.107:0/514543723 learned_addr learned my addr 172.21.15.107:0/514543723 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b264b6640 1 --2- 172.21.15.107:0/514543723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b20152020 0x7f4b200c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b25cb5640 1 -- 172.21.15.107:0/514543723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b20152020 msgr2=0x7f4b200c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b25cb5640 1 --2- 172.21.15.107:0/514543723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b20152020 0x7f4b200c4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b25cb5640 1 -- 172.21.15.107:0/514543723 >> v1:172.21.15.110:6789/0 conn(0x7f4b201553d0 legacy=0x7f4b2006ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.726+0000 7f4b25cb5640 1 -- 172.21.15.107:0/514543723 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b200c6fc0 con 0x7f4b200c8640 2023-10-24T23:56:04.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b25cb5640 1 --2- 172.21.15.107:0/514543723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b200c8640 0x7f4b2006a710 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4b10009870 tx=0x7f4b1001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb728bf5ccd7c97 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:04.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b254b4640 1 -- 172.21.15.107:0/514543723 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b100ab620 con 0x7f4b200c8640 2023-10-24T23:56:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b254b4640 1 -- 172.21.15.107:0/514543723 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b100ab780 con 0x7f4b200c8640 2023-10-24T23:56:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b254b4640 1 -- 172.21.15.107:0/514543723 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b100abb10 con 0x7f4b200c8640 2023-10-24T23:56:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 -- 172.21.15.107:0/514543723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b200c8640 msgr2=0x7f4b2006a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 --2- 172.21.15.107:0/514543723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b200c8640 0x7f4b2006a710 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4b10009870 tx=0x7f4b1001c440 comp rx=0 tx=0).stop 2023-10-24T23:56:04.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 -- 172.21.15.107:0/514543723 shutdown_connections 2023-10-24T23:56:04.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 --2- 172.21.15.107:0/514543723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b200c8640 0x7f4b2006a710 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:04.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 --2- 172.21.15.107:0/514543723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b20152020 0x7f4b200c4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:04.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 -- 172.21.15.107:0/514543723 >> 172.21.15.107:0/514543723 conn(0x7f4b200abd60 msgr2=0x7f4b200ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:04.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 -- 172.21.15.107:0/514543723 shutdown_connections 2023-10-24T23:56:04.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 -- 172.21.15.107:0/514543723 wait complete. 2023-10-24T23:56:04.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 Processor -- start 2023-10-24T23:56:04.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 -- start start 2023-10-24T23:56:04.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b200c8640 0x7f4b200aad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:04.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b20152020 0x7f4b2014a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:04.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b201553d0 0x7f4b200a9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:04.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200ab2b0 con 0x7f4b20152020 2023-10-24T23:56:04.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b201ed990 con 0x7f4b200c8640 2023-10-24T23:56:04.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b2871a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200abad0 con 0x7f4b201553d0 2023-10-24T23:56:04.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b25cb5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b20152020 0x7f4b2014a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:04.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b25cb5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b20152020 0x7f4b2014a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34124/0 (socket says 172.21.15.107:34124) 2023-10-24T23:56:04.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.730+0000 7f4b25cb5640 1 -- 172.21.15.107:0/2195528064 learned_addr learned my addr 172.21.15.107:0/2195528064 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:04.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b264b6640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b200c8640 0x7f4b200aad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:04.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b26cb7640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b201553d0 0x7f4b200a9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:04.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b25cb5640 1 -- 172.21.15.107:0/2195528064 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b201553d0 msgr2=0x7f4b200a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:04.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b25cb5640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b201553d0 0x7f4b200a9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:04.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b25cb5640 1 -- 172.21.15.107:0/2195528064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b200c8640 msgr2=0x7f4b200aad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:04.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b25cb5640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b200c8640 0x7f4b200aad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:04.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b25cb5640 1 -- 172.21.15.107:0/2195528064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b100094d0 con 0x7f4b20152020 2023-10-24T23:56:04.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b25cb5640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b20152020 0x7f4b2014a990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4b100ad040 tx=0x7f4b1001ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:04.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b16ffd640 1 -- 172.21.15.107:0/2195528064 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b100be3f0 con 0x7f4b20152020 2023-10-24T23:56:04.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b16ffd640 1 -- 172.21.15.107:0/2195528064 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b100be570 con 0x7f4b20152020 2023-10-24T23:56:04.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b2871a640 1 -- 172.21.15.107:0/2195528064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b200ab4e0 con 0x7f4b20152020 2023-10-24T23:56:04.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b16ffd640 1 -- 172.21.15.107:0/2195528064 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b100be920 con 0x7f4b20152020 2023-10-24T23:56:04.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b2871a640 1 -- 172.21.15.107:0/2195528064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b201ed340 con 0x7f4b20152020 2023-10-24T23:56:04.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b2871a640 1 -- 172.21.15.107:0/2195528064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b200746e0 con 0x7f4b20152020 2023-10-24T23:56:04.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b16ffd640 1 -- 172.21.15.107:0/2195528064 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4b100c7440 con 0x7f4b20152020 2023-10-24T23:56:04.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b16ffd640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4b04065e60 0x7f4b04068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:04.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b16ffd640 1 -- 172.21.15.107:0/2195528064 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7f4b1011e4d0 con 0x7f4b20152020 2023-10-24T23:56:04.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b264b6640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4b04065e60 0x7f4b04068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:04.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.734+0000 7f4b264b6640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4b04065e60 0x7f4b04068320 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c003fd0 tx=0x7f4b1c0039a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:04.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.738+0000 7f4b16ffd640 1 -- 172.21.15.107:0/2195528064 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4b100eb720 con 0x7f4b20152020 2023-10-24T23:56:04.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:56:04.852 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:56:04.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.978+0000 7f4b2871a640 1 -- 172.21.15.107:0/2195528064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b200c7800 con 0x7f4b20152020 2023-10-24T23:56:04.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.978+0000 7f4b16ffd640 1 -- 172.21.15.107:0/2195528064 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f4b100eed40 con 0x7f4b20152020 2023-10-24T23:56:04.985 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:04.985 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:04.985 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2023-10-24T23:56:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 -- 172.21.15.107:0/2195528064 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4b04065e60 msgr2=0x7f4b04068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4b04065e60 0x7f4b04068320 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c003fd0 tx=0x7f4b1c0039a0 comp rx=0 tx=0).stop 2023-10-24T23:56:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 -- 172.21.15.107:0/2195528064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b20152020 msgr2=0x7f4b2014a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b20152020 0x7f4b2014a990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4b100ad040 tx=0x7f4b1001ac80 comp rx=0 tx=0).stop 2023-10-24T23:56:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 -- 172.21.15.107:0/2195528064 shutdown_connections 2023-10-24T23:56:04.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4b04065e60 0x7f4b04068320 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:04.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4b201553d0 0x7f4b200a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:04.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b20152020 0x7f4b2014a990 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:04.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 --2- 172.21.15.107:0/2195528064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4b200c8640 0x7f4b200aad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:04.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 -- 172.21.15.107:0/2195528064 >> 172.21.15.107:0/2195528064 conn(0x7f4b200abd60 msgr2=0x7f4b201512d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:04.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 -- 172.21.15.107:0/2195528064 shutdown_connections 2023-10-24T23:56:04.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:04.982+0000 7f4b2871a640 1 -- 172.21.15.107:0/2195528064 wait complete. 2023-10-24T23:56:05.004 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:56:07.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:07.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.138+0000 7f3bac012640 1 Processor -- start 2023-10-24T23:56:07.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.138+0000 7f3bac012640 1 -- start start 2023-10-24T23:56:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.138+0000 7f3bac012640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3ba4153190 0x7f3ba40a9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.138+0000 7f3bac012640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba40ab190 0x7f3ba40ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.138+0000 7f3bac012640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba406a6b0 con 0x7f3ba414fc20 2023-10-24T23:56:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.138+0000 7f3bac012640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba40ad0a0 con 0x7f3ba4153190 2023-10-24T23:56:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.138+0000 7f3bac012640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba406a380 con 0x7f3ba40ab190 2023-10-24T23:56:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.138+0000 7f3baa5af640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba40ab190 0x7f3ba40ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.138+0000 7f3baa5af640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba40ab190 0x7f3ba40ab5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:49194/0 (socket says 172.21.15.107:49194) 2023-10-24T23:56:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.138+0000 7f3ba95ad640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3ba4153190 0x7f3ba40a9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.138+0000 7f3baa5af640 1 -- 172.21.15.107:0/1115206900 learned_addr learned my addr 172.21.15.107:0/1115206900 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3baa5af640 1 -- 172.21.15.107:0/1115206900 >> v2:172.21.15.110:3300/0 conn(0x7f3ba4153190 msgr2=0x7f3ba40a9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3baa5af640 1 --2- 172.21.15.107:0/1115206900 >> v2:172.21.15.110:3300/0 conn(0x7f3ba4153190 0x7f3ba40a9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:07.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3baa5af640 1 -- 172.21.15.107:0/1115206900 >> v1:172.21.15.110:6789/0 conn(0x7f3ba414fc20 legacy=0x7f3ba40a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:07.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3baa5af640 1 -- 172.21.15.107:0/1115206900 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba40aa0a0 con 0x7f3ba40ab190 2023-10-24T23:56:07.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3baa5af640 1 --2- 172.21.15.107:0/1115206900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba40ab190 0x7f3ba40ab5e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009e50 tx=0x7f3b9c01c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4d89e9f3fcc915d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:07.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3ba8dac640 1 -- 172.21.15.107:0/1115206900 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c0a5750 con 0x7f3ba40ab190 2023-10-24T23:56:07.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3ba8dac640 1 -- 172.21.15.107:0/1115206900 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9c0a58b0 con 0x7f3ba40ab190 2023-10-24T23:56:07.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3ba8dac640 1 -- 172.21.15.107:0/1115206900 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c0a5c40 con 0x7f3ba40ab190 2023-10-24T23:56:07.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3bac012640 1 -- 172.21.15.107:0/1115206900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba40ab190 msgr2=0x7f3ba40ab5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:07.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3bac012640 1 --2- 172.21.15.107:0/1115206900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba40ab190 0x7f3ba40ab5e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009e50 tx=0x7f3b9c01c980 comp rx=0 tx=0).stop 2023-10-24T23:56:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3bac012640 1 -- 172.21.15.107:0/1115206900 shutdown_connections 2023-10-24T23:56:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3bac012640 1 --2- 172.21.15.107:0/1115206900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba40ab190 0x7f3ba40ab5e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3bac012640 1 --2- 172.21.15.107:0/1115206900 >> v2:172.21.15.110:3300/0 conn(0x7f3ba4153190 0x7f3ba40a9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3bac012640 1 -- 172.21.15.107:0/1115206900 >> 172.21.15.107:0/1115206900 conn(0x7f3ba40c4e40 msgr2=0x7f3ba40c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3bac012640 1 -- 172.21.15.107:0/1115206900 shutdown_connections 2023-10-24T23:56:07.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.142+0000 7f3bac012640 1 -- 172.21.15.107:0/1115206900 wait complete. 2023-10-24T23:56:07.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3bac012640 1 Processor -- start 2023-10-24T23:56:07.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3bac012640 1 -- start start 2023-10-24T23:56:07.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3bac012640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba40ab190 0x7f3ba4145570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:07.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3bac012640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba414fc20 0x7f3ba414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:07.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3bac012640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba4153190 0x7f3ba4070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:07.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3bac012640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba4154020 con 0x7f3ba40ab190 2023-10-24T23:56:07.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3bac012640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba4145ab0 con 0x7f3ba414fc20 2023-10-24T23:56:07.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3bac012640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba41521b0 con 0x7f3ba4153190 2023-10-24T23:56:07.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3ba9dae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba40ab190 0x7f3ba4145570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:07.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3ba9dae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba40ab190 0x7f3ba4145570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34130/0 (socket says 172.21.15.107:34130) 2023-10-24T23:56:07.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3ba9dae640 1 -- 172.21.15.107:0/2551589183 learned_addr learned my addr 172.21.15.107:0/2551589183 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:07.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3baa5af640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba4153190 0x7f3ba4070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:07.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3ba95ad640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba414fc20 0x7f3ba414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:07.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3ba9dae640 1 -- 172.21.15.107:0/2551589183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba4153190 msgr2=0x7f3ba4070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:07.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3ba9dae640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba4153190 0x7f3ba4070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:07.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3ba9dae640 1 -- 172.21.15.107:0/2551589183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba414fc20 msgr2=0x7f3ba414a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:07.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3ba9dae640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba414fc20 0x7f3ba414a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:07.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3ba9dae640 1 -- 172.21.15.107:0/2551589183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c009b30 con 0x7f3ba40ab190 2023-10-24T23:56:07.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3ba9dae640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba40ab190 0x7f3ba4145570 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000c420 tx=0x7f3ba000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3b9a7fc640 1 -- 172.21.15.107:0/2551589183 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba00052d0 con 0x7f3ba40ab190 2023-10-24T23:56:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3b9a7fc640 1 -- 172.21.15.107:0/2551589183 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba0005430 con 0x7f3ba40ab190 2023-10-24T23:56:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3bac012640 1 -- 172.21.15.107:0/2551589183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba41edbb0 con 0x7f3ba40ab190 2023-10-24T23:56:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3b9a7fc640 1 -- 172.21.15.107:0/2551589183 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0007260 con 0x7f3ba40ab190 2023-10-24T23:56:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3bac012640 1 -- 172.21.15.107:0/2551589183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba406ff40 con 0x7f3ba40ab190 2023-10-24T23:56:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.146+0000 7f3bac012640 1 -- 172.21.15.107:0/2551589183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba4073a00 con 0x7f3ba40ab190 2023-10-24T23:56:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.150+0000 7f3b9a7fc640 1 -- 172.21.15.107:0/2551589183 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3ba00073c0 con 0x7f3ba40ab190 2023-10-24T23:56:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.150+0000 7f3b9a7fc640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3b88065e90 0x7f3b88068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.150+0000 7f3b9a7fc640 1 -- 172.21.15.107:0/2551589183 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7f3ba0082f20 con 0x7f3ba40ab190 2023-10-24T23:56:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.150+0000 7f3ba95ad640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3b88065e90 0x7f3b88068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.150+0000 7f3ba95ad640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3b88065e90 0x7f3b88068350 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3ba406dc60 tx=0x7f3b94008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.154+0000 7f3b9a7fc640 1 -- 172.21.15.107:0/2551589183 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3ba0050220 con 0x7f3ba40ab190 2023-10-24T23:56:07.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.362+0000 7f3bac012640 1 -- 172.21.15.107:0/2551589183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ba40746e0 con 0x7f3ba40ab190 2023-10-24T23:56:07.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.362+0000 7f3b9a7fc640 1 -- 172.21.15.107:0/2551589183 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f3ba0053840 con 0x7f3ba40ab190 2023-10-24T23:56:07.368 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:07.368 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:07.368 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2023-10-24T23:56:07.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 -- 172.21.15.107:0/2551589183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3b88065e90 msgr2=0x7f3b88068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:07.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3b88065e90 0x7f3b88068350 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3ba406dc60 tx=0x7f3b94008040 comp rx=0 tx=0).stop 2023-10-24T23:56:07.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 -- 172.21.15.107:0/2551589183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba40ab190 msgr2=0x7f3ba4145570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba40ab190 0x7f3ba4145570 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3ba000c420 tx=0x7f3ba000c8e0 comp rx=0 tx=0).stop 2023-10-24T23:56:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 -- 172.21.15.107:0/2551589183 shutdown_connections 2023-10-24T23:56:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3b88065e90 0x7f3b88068350 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ba4153190 0x7f3ba4070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ba414fc20 0x7f3ba414a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 --2- 172.21.15.107:0/2551589183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ba40ab190 0x7f3ba4145570 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 -- 172.21.15.107:0/2551589183 >> 172.21.15.107:0/2551589183 conn(0x7f3ba40c4e40 msgr2=0x7f3ba40c60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 -- 172.21.15.107:0/2551589183 shutdown_connections 2023-10-24T23:56:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:07.366+0000 7f3bac012640 1 -- 172.21.15.107:0/2551589183 wait complete. 2023-10-24T23:56:07.390 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:56:09.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:09.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.546+0000 7f21c6b53640 1 Processor -- start 2023-10-24T23:56:09.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.546+0000 7f21c6b53640 1 -- start start 2023-10-24T23:56:09.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.546+0000 7f21c6b53640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21c00c20b0 0x7f21c00aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:09.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.546+0000 7f21c6b53640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c00a9aa0 con 0x7f21c0152060 2023-10-24T23:56:09.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.546+0000 7f21c6b53640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c0155480 con 0x7f21c00c2990 2023-10-24T23:56:09.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.546+0000 7f21c6b53640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c0155760 con 0x7f21c00c20b0 2023-10-24T23:56:09.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c50f0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21c00c20b0 0x7f21c00aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c50f0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f21c00c20b0 0x7f21c00aada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34138/0 (socket says 172.21.15.107:34138) 2023-10-24T23:56:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c50f0640 1 -- 172.21.15.107:0/444192260 learned_addr learned my addr 172.21.15.107:0/444192260 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c50f0640 1 -- 172.21.15.107:0/444192260 >> v1:172.21.15.110:6789/0 conn(0x7f21c00c2990 legacy=0x7f21c00c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c50f0640 1 -- 172.21.15.107:0/444192260 >> v1:172.21.15.107:6789/0 conn(0x7f21c0152060 legacy=0x7f21c00a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c50f0640 1 -- 172.21.15.107:0/444192260 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c00a9450 con 0x7f21c00c20b0 2023-10-24T23:56:09.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c50f0640 1 --2- 172.21.15.107:0/444192260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21c00c20b0 0x7f21c00aada0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f21b8009e50 tx=0x7f21b801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e771d02426ab05 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:09.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21bf7fe640 1 -- 172.21.15.107:0/444192260 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b80a5700 con 0x7f21c00c20b0 2023-10-24T23:56:09.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21bf7fe640 1 -- 172.21.15.107:0/444192260 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b80a5860 con 0x7f21c00c20b0 2023-10-24T23:56:09.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21bf7fe640 1 -- 172.21.15.107:0/444192260 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b80a5bf0 con 0x7f21c00c20b0 2023-10-24T23:56:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c6b53640 1 -- 172.21.15.107:0/444192260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21c00c20b0 msgr2=0x7f21c00aada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c6b53640 1 --2- 172.21.15.107:0/444192260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21c00c20b0 0x7f21c00aada0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f21b8009e50 tx=0x7f21b801c880 comp rx=0 tx=0).stop 2023-10-24T23:56:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c6b53640 1 -- 172.21.15.107:0/444192260 shutdown_connections 2023-10-24T23:56:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c6b53640 1 --2- 172.21.15.107:0/444192260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21c00c20b0 0x7f21c00aada0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c6b53640 1 -- 172.21.15.107:0/444192260 >> 172.21.15.107:0/444192260 conn(0x7f21c00c4a90 msgr2=0x7f21c00c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c6b53640 1 -- 172.21.15.107:0/444192260 shutdown_connections 2023-10-24T23:56:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.550+0000 7f21c6b53640 1 -- 172.21.15.107:0/444192260 wait complete. 2023-10-24T23:56:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c6b53640 1 Processor -- start 2023-10-24T23:56:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c6b53640 1 -- start start 2023-10-24T23:56:09.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c6b53640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f21c00c20b0 0x7f21c0141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:09.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c6b53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21c00c2990 0x7f21c0147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c6b53640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21c0152060 0x7f21c00c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c6b53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c01f0840 con 0x7f21c00c2990 2023-10-24T23:56:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c6b53640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c0141650 con 0x7f21c0152060 2023-10-24T23:56:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c6b53640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c01efec0 con 0x7f21c00c20b0 2023-10-24T23:56:09.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21c00c2990 0x7f21c0147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21c00c2990 0x7f21c0147f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34142/0 (socket says 172.21.15.107:34142) 2023-10-24T23:56:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bffff640 1 -- 172.21.15.107:0/533777335 learned_addr learned my addr 172.21.15.107:0/533777335 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c48ef640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f21c00c20b0 0x7f21c0141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c50f0640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21c0152060 0x7f21c00c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:09.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bffff640 1 -- 172.21.15.107:0/533777335 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f21c00c20b0 msgr2=0x7f21c0141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bffff640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f21c00c20b0 0x7f21c0141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bffff640 1 -- 172.21.15.107:0/533777335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21c0152060 msgr2=0x7f21c00c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bffff640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21c0152060 0x7f21c00c5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bffff640 1 -- 172.21.15.107:0/533777335 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b8009b30 con 0x7f21c00c2990 2023-10-24T23:56:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bffff640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21c00c2990 0x7f21c0147f40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00c4c0 tx=0x7f21ac001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bd7fa640 1 -- 172.21.15.107:0/533777335 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac005480 con 0x7f21c00c2990 2023-10-24T23:56:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bd7fa640 1 -- 172.21.15.107:0/533777335 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ac003600 con 0x7f21c00c2990 2023-10-24T23:56:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21bd7fa640 1 -- 172.21.15.107:0/533777335 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac003990 con 0x7f21c00c2990 2023-10-24T23:56:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c6b53640 1 -- 172.21.15.107:0/533777335 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c00c7ca0 con 0x7f21c00c2990 2023-10-24T23:56:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c6b53640 1 -- 172.21.15.107:0/533777335 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c01f0460 con 0x7f21c00c2990 2023-10-24T23:56:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.554+0000 7f21c6b53640 1 -- 172.21.15.107:0/533777335 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c00746e0 con 0x7f21c00c2990 2023-10-24T23:56:09.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.558+0000 7f21bd7fa640 1 -- 172.21.15.107:0/533777335 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f21ac003af0 con 0x7f21c00c2990 2023-10-24T23:56:09.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.558+0000 7f21bd7fa640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f21a4065e60 0x7f21a4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:09.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.558+0000 7f21c48ef640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f21a4065e60 0x7f21a4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:09.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.558+0000 7f21bd7fa640 1 -- 172.21.15.107:0/533777335 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7f21ac082ce0 con 0x7f21c00c2990 2023-10-24T23:56:09.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.558+0000 7f21c48ef640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f21a4065e60 0x7f21a4068320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f21b4000f70 tx=0x7f21b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:09.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.562+0000 7f21bd7fa640 1 -- 172.21.15.107:0/533777335 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f21ac04ff30 con 0x7f21c00c2990 2023-10-24T23:56:09.659 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2023-10-24T23:56:09.660 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-24T23:56:09.660 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-24T23:56:09.663 INFO:tasks.ceph.mds.d:Started 2023-10-24T23:56:09.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:09.737 INFO:tasks.ceph.mds.d.smithi110.stdout:starting mds.d at 2023-10-24T23:56:09.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.766+0000 7f21c6b53640 1 -- 172.21.15.107:0/533777335 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21c0145310 con 0x7f21c00c2990 2023-10-24T23:56:09.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.766+0000 7f21bd7fa640 1 -- 172.21.15.107:0/533777335 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f21ac017090 con 0x7f21c00c2990 2023-10-24T23:56:09.773 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:09.773 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:09.773 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2023-10-24T23:56:09.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 -- 172.21.15.107:0/533777335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f21a4065e60 msgr2=0x7f21a4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:09.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f21a4065e60 0x7f21a4068320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f21b4000f70 tx=0x7f21b4009040 comp rx=0 tx=0).stop 2023-10-24T23:56:09.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 -- 172.21.15.107:0/533777335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21c00c2990 msgr2=0x7f21c0147f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:09.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21c00c2990 0x7f21c0147f40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00c4c0 tx=0x7f21ac001e80 comp rx=0 tx=0).stop 2023-10-24T23:56:09.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 -- 172.21.15.107:0/533777335 shutdown_connections 2023-10-24T23:56:09.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21c0152060 0x7f21c00c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21c00c2990 0x7f21c0147f40 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f21a4065e60 0x7f21a4068320 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 --2- 172.21.15.107:0/533777335 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f21c00c20b0 0x7f21c0141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 -- 172.21.15.107:0/533777335 >> 172.21.15.107:0/533777335 conn(0x7f21c00c4a90 msgr2=0x7f21c00c1480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 -- 172.21.15.107:0/533777335 shutdown_connections 2023-10-24T23:56:09.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.770+0000 7f21c6b53640 1 -- 172.21.15.107:0/533777335 wait complete. 2023-10-24T23:56:09.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.778+0000 7effdf322640 1 Processor -- start 2023-10-24T23:56:09.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.778+0000 7effdf322640 1 -- start start 2023-10-24T23:56:09.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdf322640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effd8070b00 0x7effd80c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:09.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdf322640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7effd806df30 0x7effd806e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:09.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdf322640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd80c22f0 con 0x7effd814eb70 2023-10-24T23:56:09.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdf322640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd8071700 con 0x7effd806df30 2023-10-24T23:56:09.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdf322640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd80c25d0 con 0x7effd8070b00 2023-10-24T23:56:09.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdd8bf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7effd806df30 0x7effd806e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:09.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdc8bd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effd8070b00 0x7effd80c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:09.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdd0be640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7effd814eb70 0x7effd81514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:39014/0 (socket says 172.21.15.107:39014) 2023-10-24T23:56:09.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdd0be640 1 -- 172.21.15.107:0/3989462336 learned_addr learned my addr 172.21.15.107:0/3989462336 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:56:09.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdd8bf640 1 -- 172.21.15.107:0/3989462336 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effd8070b00 msgr2=0x7effd80c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:09.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdd8bf640 1 --2- 172.21.15.107:0/3989462336 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effd8070b00 0x7effd80c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdd8bf640 1 -- 172.21.15.107:0/3989462336 >> v1:172.21.15.110:6789/0 conn(0x7effd814eb70 legacy=0x7effd81514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:09.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdd8bf640 1 -- 172.21.15.107:0/3989462336 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd80c2be0 con 0x7effd806df30 2023-10-24T23:56:09.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effdd8bf640 1 --2- 172.21.15.107:0/3989462336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd806df30 0x7effd806e360 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7effd0009d90 tx=0x7effd001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a457ac025eb7ac server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:09.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effcffff640 1 -- 172.21.15.107:0/3989462336 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd00a5a30 con 0x7effd806df30 2023-10-24T23:56:09.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effcffff640 1 -- 172.21.15.107:0/3989462336 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd00a5b90 con 0x7effd806df30 2023-10-24T23:56:09.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.782+0000 7effcffff640 1 -- 172.21.15.107:0/3989462336 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd00ad440 con 0x7effd806df30 2023-10-24T23:56:09.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 -- 172.21.15.107:0/3989462336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd806df30 msgr2=0x7effd806e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:09.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 --2- 172.21.15.107:0/3989462336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd806df30 0x7effd806e360 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7effd0009d90 tx=0x7effd001c950 comp rx=0 tx=0).stop 2023-10-24T23:56:09.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 -- 172.21.15.107:0/3989462336 shutdown_connections 2023-10-24T23:56:09.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 --2- 172.21.15.107:0/3989462336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd806df30 0x7effd806e360 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 --2- 172.21.15.107:0/3989462336 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effd8070b00 0x7effd80c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 -- 172.21.15.107:0/3989462336 >> 172.21.15.107:0/3989462336 conn(0x7effd80c39a0 msgr2=0x7effd80c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:09.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 -- 172.21.15.107:0/3989462336 shutdown_connections 2023-10-24T23:56:09.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 -- 172.21.15.107:0/3989462336 wait complete. 2023-10-24T23:56:09.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 Processor -- start 2023-10-24T23:56:09.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 -- start start 2023-10-24T23:56:09.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effd806df30 0x7effd8149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:09.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd8070b00 0x7effd813f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:09.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effd814eb70 0x7effd814d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:09.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd8149350 con 0x7effd8070b00 2023-10-24T23:56:09.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd814b7d0 con 0x7effd806df30 2023-10-24T23:56:09.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd8149070 con 0x7effd814eb70 2023-10-24T23:56:09.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdc8bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd8070b00 0x7effd813f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:09.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdd0be640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effd806df30 0x7effd8149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:09.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdc8bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd8070b00 0x7effd813f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34160/0 (socket says 172.21.15.107:34160) 2023-10-24T23:56:09.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdc8bd640 1 -- 172.21.15.107:0/3245736731 learned_addr learned my addr 172.21.15.107:0/3245736731 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:09.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdd8bf640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effd814eb70 0x7effd814d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:09.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdc8bd640 1 -- 172.21.15.107:0/3245736731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effd814eb70 msgr2=0x7effd814d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:09.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdc8bd640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effd814eb70 0x7effd814d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdc8bd640 1 -- 172.21.15.107:0/3245736731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effd806df30 msgr2=0x7effd8149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:09.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdc8bd640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effd806df30 0x7effd8149f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:09.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdc8bd640 1 -- 172.21.15.107:0/3245736731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd0009a70 con 0x7effd8070b00 2023-10-24T23:56:09.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdc8bd640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd8070b00 0x7effd813f510 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7effc800e870 tx=0x7effc800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:09.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effcdffb640 1 -- 172.21.15.107:0/3245736731 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8009510 con 0x7effd8070b00 2023-10-24T23:56:09.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effcdffb640 1 -- 172.21.15.107:0/3245736731 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc800eec0 con 0x7effd8070b00 2023-10-24T23:56:09.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 -- 172.21.15.107:0/3245736731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd814ba00 con 0x7effd8070b00 2023-10-24T23:56:09.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effcdffb640 1 -- 172.21.15.107:0/3245736731 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc800cb80 con 0x7effd8070b00 2023-10-24T23:56:09.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.786+0000 7effdf322640 1 -- 172.21.15.107:0/3245736731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd8149a50 con 0x7effd8070b00 2023-10-24T23:56:09.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.790+0000 7effdf322640 1 -- 172.21.15.107:0/3245736731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd80746e0 con 0x7effd8070b00 2023-10-24T23:56:09.801 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:56:09.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.790+0000 7effcdffb640 1 -- 172.21.15.107:0/3245736731 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7effc800cce0 con 0x7effd8070b00 2023-10-24T23:56:09.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.790+0000 7effcdffb640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7effbc065e60 0x7effbc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:09.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.790+0000 7effdd0be640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7effbc065e60 0x7effbc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:09.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.790+0000 7effcdffb640 1 -- 172.21.15.107:0/3245736731 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7effc8083cc0 con 0x7effd8070b00 2023-10-24T23:56:09.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.790+0000 7effdd0be640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7effbc065e60 0x7effbc068320 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7effd4005890 tx=0x7effd4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:09.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:09.794+0000 7effcdffb640 1 -- 172.21.15.107:0/3245736731 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7effc8050fc0 con 0x7effd8070b00 2023-10-24T23:56:10.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 -- 172.21.15.107:0/3245736731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effd80c1a50 con 0x7effd8070b00 2023-10-24T23:56:10.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effcdffb640 1 -- 172.21.15.107:0/3245736731 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5013 (secure 0 0 0) 0x7effc8017090 con 0x7effd8070b00 2023-10-24T23:56:10.031 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:10.031 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 27 2023-10-24T23:56:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 -- 172.21.15.107:0/3245736731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7effbc065e60 msgr2=0x7effbc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7effbc065e60 0x7effbc068320 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7effd4005890 tx=0x7effd4009040 comp rx=0 tx=0).stop 2023-10-24T23:56:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 -- 172.21.15.107:0/3245736731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd8070b00 msgr2=0x7effd813f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:10.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd8070b00 0x7effd813f510 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7effc800e870 tx=0x7effc800ec70 comp rx=0 tx=0).stop 2023-10-24T23:56:10.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 -- 172.21.15.107:0/3245736731 shutdown_connections 2023-10-24T23:56:10.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7effbc065e60 0x7effbc068320 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:10.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7effd814eb70 0x7effd814d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:10.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effd8070b00 0x7effd813f510 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 --2- 172.21.15.107:0/3245736731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7effd806df30 0x7effd8149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 -- 172.21.15.107:0/3245736731 >> 172.21.15.107:0/3245736731 conn(0x7effd80c39a0 msgr2=0x7effd806bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 -- 172.21.15.107:0/3245736731 shutdown_connections 2023-10-24T23:56:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:10.026+0000 7effdf322640 1 -- 172.21.15.107:0/3245736731 wait complete. 2023-10-24T23:56:10.054 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-24T23:56:11.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:11.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.926+0000 7fcd4e74b640 1 Processor -- start 2023-10-24T23:56:11.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4e74b640 1 -- start start 2023-10-24T23:56:11.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4e74b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcd4814eb70 0x7fcd480c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:11.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4e74b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd480c22f0 con 0x7fcd4806df30 2023-10-24T23:56:11.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4e74b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd480c25d0 con 0x7fcd48070b00 2023-10-24T23:56:11.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4e74b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd48071700 con 0x7fcd4814eb70 2023-10-24T23:56:11.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4cce8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcd4806df30 0x7fcd4815be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53604/0 (socket says 172.21.15.107:53604) 2023-10-24T23:56:11.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4cce8640 1 -- 172.21.15.107:0/1273857077 learned_addr learned my addr 172.21.15.107:0/1273857077 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:56:11.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd47fff640 1 --2- 172.21.15.107:0/1273857077 >> v2:172.21.15.110:3300/0 conn(0x7fcd4814eb70 0x7fcd480c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:11.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335103766 0 0) 0x7fcd480c22f0 con 0x7fcd4806df30 2023-10-24T23:56:11.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd30003530 con 0x7fcd4806df30 2023-10-24T23:56:11.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3694709097 0 0) 0x7fcd30003530 con 0x7fcd4806df30 2023-10-24T23:56:11.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 >> v2:172.21.15.110:3300/0 conn(0x7fcd4814eb70 msgr2=0x7fcd480c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:11.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 --2- 172.21.15.107:0/1273857077 >> v2:172.21.15.110:3300/0 conn(0x7fcd4814eb70 0x7fcd480c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:11.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 >> v1:172.21.15.110:6789/0 conn(0x7fcd48070b00 legacy=0x7fcd481514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:11.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd480c2be0 con 0x7fcd4806df30 2023-10-24T23:56:11.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fcd40001a00 con 0x7fcd4806df30 2023-10-24T23:56:11.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 >> v1:172.21.15.107:6789/0 conn(0x7fcd4806df30 legacy=0x7fcd4815be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:11.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 --2- 172.21.15.107:0/1273857077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd30003980 0x7fcd30003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:11.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 --2- 172.21.15.107:0/1273857077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd300047d0 0x7fcd30004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:11.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 --2- 172.21.15.107:0/1273857077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd30005190 0x7fcd3000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:11.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3000e210 con 0x7fcd30005190 2023-10-24T23:56:11.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3000e380 con 0x7fcd300047d0 2023-10-24T23:56:11.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3000c050 con 0x7fcd30003980 2023-10-24T23:56:11.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4cce8640 1 --2- 172.21.15.107:0/1273857077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd30005190 0x7fcd3000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:11.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd477fe640 1 --2- 172.21.15.107:0/1273857077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd300047d0 0x7fcd30004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:11.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd47fff640 1 --2- 172.21.15.107:0/1273857077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd30003980 0x7fcd30003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4cce8640 1 -- 172.21.15.107:0/1273857077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd30003980 msgr2=0x7fcd30003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4cce8640 1 --2- 172.21.15.107:0/1273857077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd30003980 0x7fcd30003da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4cce8640 1 -- 172.21.15.107:0/1273857077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd300047d0 msgr2=0x7fcd30004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4cce8640 1 --2- 172.21.15.107:0/1273857077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd300047d0 0x7fcd30004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:11.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.930+0000 7fcd4cce8640 1 -- 172.21.15.107:0/1273857077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd30003530 con 0x7fcd30005190 2023-10-24T23:56:11.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd4cce8640 1 -- 172.21.15.107:0/1273857077 reap_dead start 2023-10-24T23:56:11.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd4cce8640 1 --2- 172.21.15.107:0/1273857077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd30005190 0x7fcd3000b990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000a830 tx=0x7fcd4001d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4329e593db90a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:11.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd400072d0 con 0x7fcd30005190 2023-10-24T23:56:11.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd400a8070 con 0x7fcd30005190 2023-10-24T23:56:11.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd46ffd640 1 -- 172.21.15.107:0/1273857077 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd400076b0 con 0x7fcd30005190 2023-10-24T23:56:11.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1273857077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd30005190 msgr2=0x7fcd3000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:11.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd4e74b640 1 --2- 172.21.15.107:0/1273857077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd30005190 0x7fcd3000b990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000a830 tx=0x7fcd4001d340 comp rx=0 tx=0).stop 2023-10-24T23:56:11.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1273857077 shutdown_connections 2023-10-24T23:56:11.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd4e74b640 1 --2- 172.21.15.107:0/1273857077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd30005190 0x7fcd3000b990 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:11.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1273857077 >> 172.21.15.107:0/1273857077 conn(0x7fcd480c39a0 msgr2=0x7fcd480c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:11.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1273857077 shutdown_connections 2023-10-24T23:56:11.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1273857077 wait complete. 2023-10-24T23:56:11.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd4e74b640 1 Processor -- start 2023-10-24T23:56:11.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.934+0000 7fcd4e74b640 1 -- start start 2023-10-24T23:56:11.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4e74b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd30005190 0x7fcd48140850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:11.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4e74b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd48069d40 0x7fcd481e8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:11.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4e74b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd481e9430 0x7fcd48149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:11.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4e74b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd48142460 con 0x7fcd481e9430 2023-10-24T23:56:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4e74b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4806a510 con 0x7fcd30005190 2023-10-24T23:56:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4e74b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd481384b0 con 0x7fcd48069d40 2023-10-24T23:56:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4cce8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd481e9430 0x7fcd48149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4cce8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd481e9430 0x7fcd48149590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34192/0 (socket says 172.21.15.107:34192) 2023-10-24T23:56:11.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4cce8640 1 -- 172.21.15.107:0/1209841444 learned_addr learned my addr 172.21.15.107:0/1209841444 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:11.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd477fe640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd48069d40 0x7fcd481e8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:11.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd47fff640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd30005190 0x7fcd48140850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:11.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4cce8640 1 -- 172.21.15.107:0/1209841444 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd48069d40 msgr2=0x7fcd481e8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:11.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4cce8640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd48069d40 0x7fcd481e8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:11.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4cce8640 1 -- 172.21.15.107:0/1209841444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd30005190 msgr2=0x7fcd48140850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:11.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4cce8640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd30005190 0x7fcd48140850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:11.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4cce8640 1 -- 172.21.15.107:0/1209841444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd481555c0 con 0x7fcd481e9430 2023-10-24T23:56:11.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4cce8640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd481e9430 0x7fcd48149590 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcd40007040 tx=0x7fcd30005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:11.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd44ff9640 1 -- 172.21.15.107:0/1209841444 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40001a00 con 0x7fcd481e9430 2023-10-24T23:56:11.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd44ff9640 1 -- 172.21.15.107:0/1209841444 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd40007450 con 0x7fcd481e9430 2023-10-24T23:56:11.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd44ff9640 1 -- 172.21.15.107:0/1209841444 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd400077e0 con 0x7fcd481e9430 2023-10-24T23:56:11.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1209841444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd48141f30 con 0x7fcd481e9430 2023-10-24T23:56:11.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1209841444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd481e8a60 con 0x7fcd481e9430 2023-10-24T23:56:11.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1209841444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd481e8bf0 con 0x7fcd481e9430 2023-10-24T23:56:11.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.938+0000 7fcd44ff9640 1 -- 172.21.15.107:0/1209841444 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcd400aa070 con 0x7fcd481e9430 2023-10-24T23:56:11.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.942+0000 7fcd44ff9640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd34065e50 0x7fcd34068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:11.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.942+0000 7fcd47fff640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd34065e50 0x7fcd34068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:11.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.942+0000 7fcd44ff9640 1 -- 172.21.15.107:0/1209841444 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7fcd4011d450 con 0x7fcd481e9430 2023-10-24T23:56:11.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.942+0000 7fcd47fff640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd34065e50 0x7fcd34068310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c009a30 tx=0x7fcd3c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:11.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:11.946+0000 7fcd44ff9640 1 -- 172.21.15.107:0/1209841444 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fcd400ea6a0 con 0x7fcd481e9430 2023-10-24T23:56:12.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:12.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1209841444 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd480746e0 con 0x7fcd481e9430 2023-10-24T23:56:12.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd44ff9640 1 -- 172.21.15.107:0/1209841444 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fcd400edcc0 con 0x7fcd481e9430 2023-10-24T23:56:12.167 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:12.167 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:12.168 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2023-10-24T23:56:12.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1209841444 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd34065e50 msgr2=0x7fcd34068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:12.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd34065e50 0x7fcd34068310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c009a30 tx=0x7fcd3c008040 comp rx=0 tx=0).stop 2023-10-24T23:56:12.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1209841444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd481e9430 msgr2=0x7fcd48149590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:12.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd481e9430 0x7fcd48149590 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcd40007040 tx=0x7fcd30005800 comp rx=0 tx=0).stop 2023-10-24T23:56:12.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1209841444 shutdown_connections 2023-10-24T23:56:12.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd481e9430 0x7fcd48149590 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd34065e50 0x7fcd34068310 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd48069d40 0x7fcd481e8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 --2- 172.21.15.107:0/1209841444 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd30005190 0x7fcd48140850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1209841444 >> 172.21.15.107:0/1209841444 conn(0x7fcd480c39a0 msgr2=0x7fcd48142e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:12.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1209841444 shutdown_connections 2023-10-24T23:56:12.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.162+0000 7fcd4e74b640 1 -- 172.21.15.107:0/1209841444 wait complete. 2023-10-24T23:56:12.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce8a8b0640 1 Processor -- start 2023-10-24T23:56:12.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce8a8b0640 1 -- start start 2023-10-24T23:56:12.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce8a8b0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce84069d40 0x7fce84158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:12.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce8a8b0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce8406a120 0x7fce840aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:12.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce8a8b0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8406cea0 con 0x7fce84065bc0 2023-10-24T23:56:12.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce8a8b0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce840ac650 con 0x7fce8406a120 2023-10-24T23:56:12.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce8a8b0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce840a9e00 con 0x7fce84069d40 2023-10-24T23:56:12.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce88e4d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce8406a120 0x7fce840aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce83fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce84069d40 0x7fce84158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce88e4d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce8406a120 0x7fce840aaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54680/0 (socket says 172.21.15.107:54680) 2023-10-24T23:56:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce88e4d640 1 -- 172.21.15.107:0/3103450102 learned_addr learned my addr 172.21.15.107:0/3103450102 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce88e4d640 1 -- 172.21.15.107:0/3103450102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce84069d40 msgr2=0x7fce84158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce88e4d640 1 --2- 172.21.15.107:0/3103450102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce84069d40 0x7fce84158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce88e4d640 1 -- 172.21.15.107:0/3103450102 >> v1:172.21.15.110:6789/0 conn(0x7fce84065bc0 legacy=0x7fce840a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:12.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.178+0000 7fce88e4d640 1 -- 172.21.15.107:0/3103450102 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce841584e0 con 0x7fce8406a120 2023-10-24T23:56:12.188 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:56:12.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.182+0000 7fce88e4d640 1 --2- 172.21.15.107:0/3103450102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce8406a120 0x7fce840aaef0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fce7c009e50 tx=0x7fce7c01c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac175c09b2dd7d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:12.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.182+0000 7fce82ffd640 1 -- 172.21.15.107:0/3103450102 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c0a5640 con 0x7fce8406a120 2023-10-24T23:56:12.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.182+0000 7fce82ffd640 1 -- 172.21.15.107:0/3103450102 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c0a57a0 con 0x7fce8406a120 2023-10-24T23:56:12.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.182+0000 7fce82ffd640 1 -- 172.21.15.107:0/3103450102 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c0a5b30 con 0x7fce8406a120 2023-10-24T23:56:12.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.182+0000 7fce8a8b0640 1 -- 172.21.15.107:0/3103450102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce8406a120 msgr2=0x7fce840aaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:12.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.182+0000 7fce8a8b0640 1 --2- 172.21.15.107:0/3103450102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce8406a120 0x7fce840aaef0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fce7c009e50 tx=0x7fce7c01c980 comp rx=0 tx=0).stop 2023-10-24T23:56:12.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.182+0000 7fce8a8b0640 1 -- 172.21.15.107:0/3103450102 shutdown_connections 2023-10-24T23:56:12.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.182+0000 7fce8a8b0640 1 --2- 172.21.15.107:0/3103450102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce8406a120 0x7fce840aaef0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.182+0000 7fce8a8b0640 1 --2- 172.21.15.107:0/3103450102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce84069d40 0x7fce84158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.182+0000 7fce8a8b0640 1 -- 172.21.15.107:0/3103450102 >> 172.21.15.107:0/3103450102 conn(0x7fce840c65c0 msgr2=0x7fce840c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 -- 172.21.15.107:0/3103450102 shutdown_connections 2023-10-24T23:56:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 -- 172.21.15.107:0/3103450102 wait complete. 2023-10-24T23:56:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 Processor -- start 2023-10-24T23:56:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 -- start start 2023-10-24T23:56:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce84065bc0 0x7fce8414d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce84069d40 0x7fce840bcba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:12.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8406a120 0x7fce841f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:12.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8414def0 con 0x7fce8406a120 2023-10-24T23:56:12.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce840ba8e0 con 0x7fce84069d40 2023-10-24T23:56:12.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8414dc50 con 0x7fce84065bc0 2023-10-24T23:56:12.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce88e4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8406a120 0x7fce841f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce88e4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8406a120 0x7fce841f0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34208/0 (socket says 172.21.15.107:34208) 2023-10-24T23:56:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce88e4d640 1 -- 172.21.15.107:0/4184761614 learned_addr learned my addr 172.21.15.107:0/4184761614 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce83fff640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce84065bc0 0x7fce8414d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce837fe640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce84069d40 0x7fce840bcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce88e4d640 1 -- 172.21.15.107:0/4184761614 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce84065bc0 msgr2=0x7fce8414d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce88e4d640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce84065bc0 0x7fce8414d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce88e4d640 1 -- 172.21.15.107:0/4184761614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce84069d40 msgr2=0x7fce840bcba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce88e4d640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce84069d40 0x7fce840bcba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce88e4d640 1 -- 172.21.15.107:0/4184761614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7c009b30 con 0x7fce8406a120 2023-10-24T23:56:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce88e4d640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8406a120 0x7fce841f0cb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fce7c009e50 tx=0x7fce7c019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce80ff9640 1 -- 172.21.15.107:0/4184761614 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c0b8670 con 0x7fce8406a120 2023-10-24T23:56:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce80ff9640 1 -- 172.21.15.107:0/4184761614 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c0b87d0 con 0x7fce8406a120 2023-10-24T23:56:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 -- 172.21.15.107:0/4184761614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce840ba290 con 0x7fce8406a120 2023-10-24T23:56:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce80ff9640 1 -- 172.21.15.107:0/4184761614 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c0b8b80 con 0x7fce8406a120 2023-10-24T23:56:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.186+0000 7fce8a8b0640 1 -- 172.21.15.107:0/4184761614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8414d760 con 0x7fce8406a120 2023-10-24T23:56:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.190+0000 7fce8a8b0640 1 -- 172.21.15.107:0/4184761614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce84072780 con 0x7fce8406a120 2023-10-24T23:56:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.190+0000 7fce80ff9640 1 -- 172.21.15.107:0/4184761614 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fce7c0b8ce0 con 0x7fce8406a120 2023-10-24T23:56:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.190+0000 7fce80ff9640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce70065e90 0x7fce70068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:12.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.190+0000 7fce83fff640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce70065e90 0x7fce70068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:12.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.190+0000 7fce80ff9640 1 -- 172.21.15.107:0/4184761614 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7fce7c11edb0 con 0x7fce8406a120 2023-10-24T23:56:12.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.190+0000 7fce83fff640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce70065e90 0x7fce70068350 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fce74009af0 tx=0x7fce74008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:12.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.194+0000 7fce80ff9640 1 -- 172.21.15.107:0/4184761614 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fce7c0ec000 con 0x7fce8406a120 2023-10-24T23:56:12.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.434+0000 7fce8a8b0640 1 -- 172.21.15.107:0/4184761614 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce840bc8c0 con 0x7fce8406a120 2023-10-24T23:56:12.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce80ff9640 1 -- 172.21.15.107:0/4184761614 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fce840bc8c0 con 0x7fce8406a120 2023-10-24T23:56:12.442 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:12.442 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:12.443 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2023-10-24T23:56:12.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 -- 172.21.15.107:0/4184761614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce70065e90 msgr2=0x7fce70068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:12.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce70065e90 0x7fce70068350 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fce74009af0 tx=0x7fce74008040 comp rx=0 tx=0).stop 2023-10-24T23:56:12.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 -- 172.21.15.107:0/4184761614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8406a120 msgr2=0x7fce841f0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:12.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8406a120 0x7fce841f0cb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fce7c009e50 tx=0x7fce7c019720 comp rx=0 tx=0).stop 2023-10-24T23:56:12.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 -- 172.21.15.107:0/4184761614 shutdown_connections 2023-10-24T23:56:12.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8406a120 0x7fce841f0cb0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce84069d40 0x7fce840bcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce70065e90 0x7fce70068350 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 --2- 172.21.15.107:0/4184761614 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce84065bc0 0x7fce8414d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 -- 172.21.15.107:0/4184761614 >> 172.21.15.107:0/4184761614 conn(0x7fce840c65c0 msgr2=0x7fce84156bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 -- 172.21.15.107:0/4184761614 shutdown_connections 2023-10-24T23:56:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:12.438+0000 7fce8a8b0640 1 -- 172.21.15.107:0/4184761614 wait complete. 2023-10-24T23:56:12.461 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2023-10-24T23:56:12.462 INFO:tasks.mds_thrash.fs.[a]:waiting for 62 secs before thrashing 2023-10-24T23:56:14.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:14.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.322+0000 7f225f467640 1 Processor -- start 2023-10-24T23:56:14.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.322+0000 7f225f467640 1 -- start start 2023-10-24T23:56:14.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.322+0000 7f225f467640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f225806a770 0x7f2258152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:14.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.322+0000 7f225f467640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f225806d2d0 0x7f2258154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:14.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.322+0000 7f225f467640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225806d760 con 0x7f225806cef0 2023-10-24T23:56:14.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.322+0000 7f225f467640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22580a9e60 con 0x7f225806d2d0 2023-10-24T23:56:14.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.322+0000 7f225f467640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22580a99d0 con 0x7f225806a770 2023-10-24T23:56:14.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.322+0000 7f225da04640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f225806d2d0 0x7f2258154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:14.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.322+0000 7f225da04640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f225806d2d0 0x7f2258154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50216/0 (socket says 172.21.15.107:50216) 2023-10-24T23:56:14.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.322+0000 7f225da04640 1 -- 172.21.15.107:0/3733795748 learned_addr learned my addr 172.21.15.107:0/3733795748 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:14.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225ca02640 1 --2- 172.21.15.107:0/3733795748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f225806a770 0x7f2258152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:14.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225da04640 1 -- 172.21.15.107:0/3733795748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f225806a770 msgr2=0x7f2258152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:14.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225da04640 1 --2- 172.21.15.107:0/3733795748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f225806a770 0x7f2258152fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:14.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225da04640 1 -- 172.21.15.107:0/3733795748 >> v1:172.21.15.110:6789/0 conn(0x7f225806cef0 legacy=0x7f2258154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:14.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225da04640 1 -- 172.21.15.107:0/3733795748 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22580a9450 con 0x7f225806d2d0 2023-10-24T23:56:14.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225da04640 1 --2- 172.21.15.107:0/3733795748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f225806d2d0 0x7f2258154a60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2250009e50 tx=0x7f225001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa3f87ccd8f964e server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:14.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f224ffff640 1 -- 172.21.15.107:0/3733795748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22500a5620 con 0x7f225806d2d0 2023-10-24T23:56:14.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f224ffff640 1 -- 172.21.15.107:0/3733795748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22500a5780 con 0x7f225806d2d0 2023-10-24T23:56:14.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f224ffff640 1 -- 172.21.15.107:0/3733795748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22500a5b10 con 0x7f225806d2d0 2023-10-24T23:56:14.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225f467640 1 -- 172.21.15.107:0/3733795748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f225806d2d0 msgr2=0x7f2258154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225f467640 1 --2- 172.21.15.107:0/3733795748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f225806d2d0 0x7f2258154a60 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2250009e50 tx=0x7f225001c8b0 comp rx=0 tx=0).stop 2023-10-24T23:56:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225f467640 1 -- 172.21.15.107:0/3733795748 shutdown_connections 2023-10-24T23:56:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225f467640 1 --2- 172.21.15.107:0/3733795748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f225806d2d0 0x7f2258154a60 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225f467640 1 --2- 172.21.15.107:0/3733795748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f225806a770 0x7f2258152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225f467640 1 -- 172.21.15.107:0/3733795748 >> 172.21.15.107:0/3733795748 conn(0x7f22580c6ed0 msgr2=0x7f22580c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:14.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225f467640 1 -- 172.21.15.107:0/3733795748 shutdown_connections 2023-10-24T23:56:14.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.326+0000 7f225f467640 1 -- 172.21.15.107:0/3733795748 wait complete. 2023-10-24T23:56:14.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225f467640 1 Processor -- start 2023-10-24T23:56:14.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225f467640 1 -- start start 2023-10-24T23:56:14.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225f467640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f225806a770 0x7f2258152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:14.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225f467640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f225806cef0 0x7f2258148940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:14.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225f467640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f225806d2d0 0x7f22580c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:14.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225f467640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2258157540 con 0x7f225806a770 2023-10-24T23:56:14.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225f467640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2258151d10 con 0x7f225806cef0 2023-10-24T23:56:14.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225f467640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22581eaf30 con 0x7f225806d2d0 2023-10-24T23:56:14.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225d203640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f225806a770 0x7f2258152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:14.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225d203640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f225806a770 0x7f2258152120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50222/0 (socket says 172.21.15.107:50222) 2023-10-24T23:56:14.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225d203640 1 -- 172.21.15.107:0/4281853253 learned_addr learned my addr 172.21.15.107:0/4281853253 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:14.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225da04640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f225806d2d0 0x7f22580c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:14.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225ca02640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f225806cef0 0x7f2258148940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:14.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225d203640 1 -- 172.21.15.107:0/4281853253 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f225806d2d0 msgr2=0x7f22580c3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:14.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225d203640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f225806d2d0 0x7f22580c3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:14.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225d203640 1 -- 172.21.15.107:0/4281853253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f225806cef0 msgr2=0x7f2258148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:14.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225d203640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f225806cef0 0x7f2258148940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:14.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225d203640 1 -- 172.21.15.107:0/4281853253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2250009b30 con 0x7f225806a770 2023-10-24T23:56:14.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225da04640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f225806d2d0 0x7f22580c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:56:14.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225d203640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f225806a770 0x7f2258152120 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f225400c450 tx=0x7f225400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:14.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f224dffb640 1 -- 172.21.15.107:0/4281853253 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254012830 con 0x7f225806a770 2023-10-24T23:56:14.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f224dffb640 1 -- 172.21.15.107:0/4281853253 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22540129b0 con 0x7f225806a770 2023-10-24T23:56:14.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225f467640 1 -- 172.21.15.107:0/4281853253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22581eb210 con 0x7f225806a770 2023-10-24T23:56:14.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f224dffb640 1 -- 172.21.15.107:0/4281853253 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254012d60 con 0x7f225806a770 2023-10-24T23:56:14.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.330+0000 7f225f467640 1 -- 172.21.15.107:0/4281853253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22581519a0 con 0x7f225806a770 2023-10-24T23:56:14.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.334+0000 7f225f467640 1 -- 172.21.15.107:0/4281853253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2258154f60 con 0x7f225806a770 2023-10-24T23:56:14.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.334+0000 7f224dffb640 1 -- 172.21.15.107:0/4281853253 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f225401c460 con 0x7f225806a770 2023-10-24T23:56:14.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.334+0000 7f224dffb640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f223c065e60 0x7f223c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:14.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.334+0000 7f224dffb640 1 -- 172.21.15.107:0/4281853253 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7f2254084520 con 0x7f225806a770 2023-10-24T23:56:14.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.334+0000 7f225ca02640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f223c065e60 0x7f223c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:14.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.334+0000 7f225ca02640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f223c065e60 0x7f223c068320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2258152660 tx=0x7f2248008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:14.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.338+0000 7f224dffb640 1 -- 172.21.15.107:0/4281853253 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2254051850 con 0x7f225806a770 2023-10-24T23:56:14.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.570+0000 7f225f467640 1 -- 172.21.15.107:0/4281853253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22580746e0 con 0x7f225806a770 2023-10-24T23:56:14.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.570+0000 7f224dffb640 1 -- 172.21.15.107:0/4281853253 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5773 (secure 0 0 0) 0x7f2254054e70 con 0x7f225806a770 2023-10-24T23:56:14.577 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:14.578 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:14.578 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2023-10-24T23:56:14.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 -- 172.21.15.107:0/4281853253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f223c065e60 msgr2=0x7f223c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:14.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f223c065e60 0x7f223c068320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2258152660 tx=0x7f2248008040 comp rx=0 tx=0).stop 2023-10-24T23:56:14.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 -- 172.21.15.107:0/4281853253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f225806a770 msgr2=0x7f2258152120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:14.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f225806a770 0x7f2258152120 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f225400c450 tx=0x7f225400c910 comp rx=0 tx=0).stop 2023-10-24T23:56:14.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 -- 172.21.15.107:0/4281853253 shutdown_connections 2023-10-24T23:56:14.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f223c065e60 0x7f223c068320 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:14.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f225806d2d0 0x7f22580c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:14.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f225806cef0 0x7f2258148940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:14.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 --2- 172.21.15.107:0/4281853253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f225806a770 0x7f2258152120 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:14.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 -- 172.21.15.107:0/4281853253 >> 172.21.15.107:0/4281853253 conn(0x7f22580c6ed0 msgr2=0x7f225806da30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:14.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 -- 172.21.15.107:0/4281853253 shutdown_connections 2023-10-24T23:56:14.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:14.574+0000 7f225f467640 1 -- 172.21.15.107:0/4281853253 wait complete. 2023-10-24T23:56:14.602 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:56:16.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:16.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.734+0000 7fb3a01c1640 1 Processor -- start 2023-10-24T23:56:16.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.734+0000 7fb3a01c1640 1 -- start start 2023-10-24T23:56:16.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.734+0000 7fb3a01c1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb39814eb70 0x7fb3980c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:16.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.734+0000 7fb3a01c1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb39806df30 0x7fb39806e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:16.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.734+0000 7fb3a01c1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3980c22f0 con 0x7fb398070b00 2023-10-24T23:56:16.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.734+0000 7fb3a01c1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb398071700 con 0x7fb39806df30 2023-10-24T23:56:16.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.734+0000 7fb3a01c1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3980c25d0 con 0x7fb39814eb70 2023-10-24T23:56:16.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.734+0000 7fb39d75c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb398070b00 0x7fb3981514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45922/0 (socket says 172.21.15.107:45922) 2023-10-24T23:56:16.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.734+0000 7fb39d75c640 1 -- 172.21.15.107:0/2854103196 learned_addr learned my addr 172.21.15.107:0/2854103196 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:56:16.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39df5d640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb39814eb70 0x7fb3980c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:16.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39e75e640 1 --2- 172.21.15.107:0/2854103196 >> v2:172.21.15.110:3300/0 conn(0x7fb39806df30 0x7fb39806e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:16.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169132619 0 0) 0x7fb3980c22f0 con 0x7fb398070b00 2023-10-24T23:56:16.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb384003530 con 0x7fb398070b00 2023-10-24T23:56:16.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1505353203 0 0) 0x7fb384003530 con 0x7fb398070b00 2023-10-24T23:56:16.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb39814eb70 msgr2=0x7fb3980c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:16.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb39814eb70 0x7fb3980c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:16.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 >> v2:172.21.15.110:3300/0 conn(0x7fb39806df30 msgr2=0x7fb39806e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:16.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 --2- 172.21.15.107:0/2854103196 >> v2:172.21.15.110:3300/0 conn(0x7fb39806df30 0x7fb39806e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:16.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3980c2be0 con 0x7fb398070b00 2023-10-24T23:56:16.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb388001680 con 0x7fb398070b00 2023-10-24T23:56:16.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 >> v1:172.21.15.107:6789/0 conn(0x7fb398070b00 legacy=0x7fb3981514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:16.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb384003980 0x7fb384003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:16.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 0x7fb384004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:16.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb39814eb70 0x7fb384005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:16.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3980c25d0 con 0x7fb3840047d0 2023-10-24T23:56:16.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38400b990 con 0x7fb384003980 2023-10-24T23:56:16.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38400bad0 con 0x7fb39814eb70 2023-10-24T23:56:16.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39d75c640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 0x7fb384004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:16.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39df5d640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb384003980 0x7fb384003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39e75e640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb39814eb70 0x7fb384005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39d75c640 1 -- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb39814eb70 msgr2=0x7fb384005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39d75c640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb39814eb70 0x7fb384005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:16.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39d75c640 1 -- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb384003980 msgr2=0x7fb384003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:16.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39d75c640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb384003980 0x7fb384003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:16.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39d75c640 1 -- 172.21.15.107:0/2854103196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb384003530 con 0x7fb3840047d0 2023-10-24T23:56:16.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39df5d640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb384003980 0x7fb384003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:56:16.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39e75e640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb39814eb70 0x7fb384005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:56:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39d75c640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 0x7fb384004c50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb38800c160 tx=0x7fb38801f290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5a838f0d7d3837 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3880a8c10 con 0x7fb3840047d0 2023-10-24T23:56:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3880a8d70 con 0x7fb3840047d0 2023-10-24T23:56:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb39cf5b640 1 -- 172.21.15.107:0/2854103196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb388005b20 con 0x7fb3840047d0 2023-10-24T23:56:16.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb3a01c1640 1 -- 172.21.15.107:0/2854103196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 msgr2=0x7fb384004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:16.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.738+0000 7fb3a01c1640 1 --2- 172.21.15.107:0/2854103196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 0x7fb384004c50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb38800c160 tx=0x7fb38801f290 comp rx=0 tx=0).stop 2023-10-24T23:56:16.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39e75e640 1 -- 172.21.15.107:0/2854103196 reap_dead start 2023-10-24T23:56:16.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 -- 172.21.15.107:0/2854103196 shutdown_connections 2023-10-24T23:56:16.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 -- 172.21.15.107:0/2854103196 >> 172.21.15.107:0/2854103196 conn(0x7fb3980c39a0 msgr2=0x7fb3980c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:16.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 -- 172.21.15.107:0/2854103196 shutdown_connections 2023-10-24T23:56:16.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 -- 172.21.15.107:0/2854103196 wait complete. 2023-10-24T23:56:16.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 Processor -- start 2023-10-24T23:56:16.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 -- start start 2023-10-24T23:56:16.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 0x7fb3981e9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:16.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3980c1200 0x7fb398070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:16.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3981e60b0 0x7fb3980c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:16.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3981e98c0 con 0x7fb3840047d0 2023-10-24T23:56:16.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb398155910 con 0x7fb3981e60b0 2023-10-24T23:56:16.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb398155630 con 0x7fb3980c1200 2023-10-24T23:56:16.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39df5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 0x7fb3981e9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:16.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39df5d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 0x7fb3981e9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50248/0 (socket says 172.21.15.107:50248) 2023-10-24T23:56:16.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39df5d640 1 -- 172.21.15.107:0/1459849787 learned_addr learned my addr 172.21.15.107:0/1459849787 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:16.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39e75e640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3981e60b0 0x7fb3980c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:16.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39d75c640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3980c1200 0x7fb398070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:16.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39df5d640 1 -- 172.21.15.107:0/1459849787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3980c1200 msgr2=0x7fb398070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:16.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39df5d640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3980c1200 0x7fb398070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:16.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39df5d640 1 -- 172.21.15.107:0/1459849787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3981e60b0 msgr2=0x7fb3980c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:16.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39df5d640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3981e60b0 0x7fb3980c4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:16.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39df5d640 1 -- 172.21.15.107:0/1459849787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3981e55d0 con 0x7fb3840047d0 2023-10-24T23:56:16.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb39df5d640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 0x7fb3981e9380 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb3940108d0 tx=0x7fb394010d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:16.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb38e7fc640 1 -- 172.21.15.107:0/1459849787 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394004220 con 0x7fb3840047d0 2023-10-24T23:56:16.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb38e7fc640 1 -- 172.21.15.107:0/1459849787 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb394004380 con 0x7fb3840047d0 2023-10-24T23:56:16.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 -- 172.21.15.107:0/1459849787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3981e58b0 con 0x7fb3840047d0 2023-10-24T23:56:16.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb38e7fc640 1 -- 172.21.15.107:0/1459849787 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39400eb40 con 0x7fb3840047d0 2023-10-24T23:56:16.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.742+0000 7fb3a01c1640 1 -- 172.21.15.107:0/1459849787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb398070680 con 0x7fb3840047d0 2023-10-24T23:56:16.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.746+0000 7fb3a01c1640 1 -- 172.21.15.107:0/1459849787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3981e5430 con 0x7fb3840047d0 2023-10-24T23:56:16.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.746+0000 7fb38e7fc640 1 -- 172.21.15.107:0/1459849787 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb39400eca0 con 0x7fb3840047d0 2023-10-24T23:56:16.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.746+0000 7fb38e7fc640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb37c065e60 0x7fb37c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:16.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.746+0000 7fb38e7fc640 1 -- 172.21.15.107:0/1459849787 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 7219+0+0 (secure 0 0 0) 0x7fb394083b40 con 0x7fb3840047d0 2023-10-24T23:56:16.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.746+0000 7fb39d75c640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb37c065e60 0x7fb37c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:16.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.746+0000 7fb39d75c640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb37c065e60 0x7fb37c068320 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb388001060 tx=0x7fb388001940 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:16.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:16.750+0000 7fb38e7fc640 1 -- 172.21.15.107:0/1459849787 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb3981e5430 con 0x7fb3840047d0 2023-10-24T23:56:17.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.002+0000 7fb3a01c1640 1 -- 172.21.15.107:0/1459849787 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3981e5430 con 0x7fb3840047d0 2023-10-24T23:56:17.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.002+0000 7fb38e7fc640 1 -- 172.21.15.107:0/1459849787 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fb3981e5430 con 0x7fb3840047d0 2023-10-24T23:56:17.008 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:17.008 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4799,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:55:27.587712+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5292},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5292":{"gid":5292,"name":"a","rank":0,"incarnation":18,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/838886697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":838886697},{"type":"v1","addr":"172.21.15.107:6837","nonce":838886697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:17.009 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2023-10-24T23:56:17.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 -- 172.21.15.107:0/1459849787 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb37c065e60 msgr2=0x7fb37c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:17.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb37c065e60 0x7fb37c068320 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb388001060 tx=0x7fb388001940 comp rx=0 tx=0).stop 2023-10-24T23:56:17.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 -- 172.21.15.107:0/1459849787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 msgr2=0x7fb3981e9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:17.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 0x7fb3981e9380 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb3940108d0 tx=0x7fb394010d90 comp rx=0 tx=0).stop 2023-10-24T23:56:17.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 -- 172.21.15.107:0/1459849787 shutdown_connections 2023-10-24T23:56:17.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3981e60b0 0x7fb3980c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:17.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb37c065e60 0x7fb37c068320 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:17.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3980c1200 0x7fb398070140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:17.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 --2- 172.21.15.107:0/1459849787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3840047d0 0x7fb3981e9380 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:17.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 -- 172.21.15.107:0/1459849787 >> 172.21.15.107:0/1459849787 conn(0x7fb3980c39a0 msgr2=0x7fb3980c3280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:17.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 -- 172.21.15.107:0/1459849787 shutdown_connections 2023-10-24T23:56:17.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:17.006+0000 7fb3a01c1640 1 -- 172.21.15.107:0/1459849787 wait complete. 2023-10-24T23:56:17.034 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:56:19.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:19.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb5dfca640 1 Processor -- start 2023-10-24T23:56:19.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb5dfca640 1 -- start start 2023-10-24T23:56:19.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb5dfca640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdb5814eb70 0x7fdb580c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb5dfca640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb5806df30 0x7fdb5806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb5dfca640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb580c22f0 con 0x7fdb58070b00 2023-10-24T23:56:19.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb5dfca640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb58071700 con 0x7fdb5814eb70 2023-10-24T23:56:19.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb5dfca640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb580c25d0 con 0x7fdb5806df30 2023-10-24T23:56:19.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb56ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdb58070b00 0x7fdb581514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45936/0 (socket says 172.21.15.107:45936) 2023-10-24T23:56:19.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb56ffd640 1 -- 172.21.15.107:0/559701744 learned_addr learned my addr 172.21.15.107:0/559701744 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:56:19.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb577fe640 1 --2- 172.21.15.107:0/559701744 >> v2:172.21.15.110:3300/0 conn(0x7fdb5814eb70 0x7fdb580c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105511314 0 0) 0x7fdb580c22f0 con 0x7fdb58070b00 2023-10-24T23:56:19.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb40003530 con 0x7fdb58070b00 2023-10-24T23:56:19.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.150+0000 7fdb57fff640 1 --2- 172.21.15.107:0/559701744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb5806df30 0x7fdb5806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3233137281 0 0) 0x7fdb40003530 con 0x7fdb58070b00 2023-10-24T23:56:19.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb5806df30 msgr2=0x7fdb5806e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:19.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 --2- 172.21.15.107:0/559701744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb5806df30 0x7fdb5806e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 >> v2:172.21.15.110:3300/0 conn(0x7fdb5814eb70 msgr2=0x7fdb580c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:19.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 --2- 172.21.15.107:0/559701744 >> v2:172.21.15.110:3300/0 conn(0x7fdb5814eb70 0x7fdb580c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb580c2be0 con 0x7fdb58070b00 2023-10-24T23:56:19.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fdb4c0016a0 con 0x7fdb58070b00 2023-10-24T23:56:19.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 >> v1:172.21.15.107:6789/0 conn(0x7fdb58070b00 legacy=0x7fdb581514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:19.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 --2- 172.21.15.107:0/559701744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb40003980 0x7fdb40003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 --2- 172.21.15.107:0/559701744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb5806df30 0x7fdb400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 --2- 172.21.15.107:0/559701744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb40004d10 0x7fdb40005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb580c25d0 con 0x7fdb40004d10 2023-10-24T23:56:19.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4000b990 con 0x7fdb40003980 2023-10-24T23:56:19.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4000bad0 con 0x7fdb5806df30 2023-10-24T23:56:19.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb57fff640 1 --2- 172.21.15.107:0/559701744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb40004d10 0x7fdb40005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb577fe640 1 --2- 172.21.15.107:0/559701744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb40003980 0x7fdb40003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb57fff640 1 -- 172.21.15.107:0/559701744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb5806df30 msgr2=0x7fdb400047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:19.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb57fff640 1 --2- 172.21.15.107:0/559701744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb5806df30 0x7fdb400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb57fff640 1 -- 172.21.15.107:0/559701744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb40003980 msgr2=0x7fdb40003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:19.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb57fff640 1 --2- 172.21.15.107:0/559701744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb40003980 0x7fdb40003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb57fff640 1 -- 172.21.15.107:0/559701744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb40003530 con 0x7fdb40004d10 2023-10-24T23:56:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb57fff640 1 --2- 172.21.15.107:0/559701744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb40004d10 0x7fdb40005190 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000e170 tx=0x7fdb5001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6afaeed154f9645 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb500a5cb0 con 0x7fdb40004d10 2023-10-24T23:56:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5000f040 con 0x7fdb40004d10 2023-10-24T23:56:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.154+0000 7fdb567fc640 1 -- 172.21.15.107:0/559701744 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb500ad680 con 0x7fdb40004d10 2023-10-24T23:56:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 -- 172.21.15.107:0/559701744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb40004d10 msgr2=0x7fdb40005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 --2- 172.21.15.107:0/559701744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb40004d10 0x7fdb40005190 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000e170 tx=0x7fdb5001cca0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb57fff640 1 -- 172.21.15.107:0/559701744 reap_dead start 2023-10-24T23:56:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 -- 172.21.15.107:0/559701744 shutdown_connections 2023-10-24T23:56:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 -- 172.21.15.107:0/559701744 >> 172.21.15.107:0/559701744 conn(0x7fdb580c39a0 msgr2=0x7fdb580c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 -- 172.21.15.107:0/559701744 shutdown_connections 2023-10-24T23:56:19.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 -- 172.21.15.107:0/559701744 wait complete. 2023-10-24T23:56:19.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 Processor -- start 2023-10-24T23:56:19.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 -- start start 2023-10-24T23:56:19.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb40004d10 0x7fdb581e8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb580a9cd0 0x7fdb580aa530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb580ac0d0 0x7fdb581e5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb580ab2c0 con 0x7fdb580a9cd0 2023-10-24T23:56:19.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb580a9ab0 con 0x7fdb40004d10 2023-10-24T23:56:19.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb581e58d0 con 0x7fdb580ac0d0 2023-10-24T23:56:19.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb56ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb580a9cd0 0x7fdb580aa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb56ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb580a9cd0 0x7fdb580aa530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50272/0 (socket says 172.21.15.107:50272) 2023-10-24T23:56:19.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb56ffd640 1 -- 172.21.15.107:0/1686575296 learned_addr learned my addr 172.21.15.107:0/1686575296 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:19.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb57fff640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb580ac0d0 0x7fdb581e5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb577fe640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb40004d10 0x7fdb581e8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb56ffd640 1 -- 172.21.15.107:0/1686575296 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb580ac0d0 msgr2=0x7fdb581e5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:19.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb56ffd640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb580ac0d0 0x7fdb581e5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb56ffd640 1 -- 172.21.15.107:0/1686575296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb40004d10 msgr2=0x7fdb581e8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:19.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb56ffd640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb40004d10 0x7fdb581e8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb56ffd640 1 -- 172.21.15.107:0/1686575296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb581e8e70 con 0x7fdb580a9cd0 2023-10-24T23:56:19.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb56ffd640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb580a9cd0 0x7fdb580aa530 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00c730 tx=0x7fdb4c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:19.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb3ffff640 1 -- 172.21.15.107:0/1686575296 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0072d0 con 0x7fdb580a9cd0 2023-10-24T23:56:19.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb3ffff640 1 -- 172.21.15.107:0/1686575296 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c007450 con 0x7fdb580a9cd0 2023-10-24T23:56:19.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 -- 172.21.15.107:0/1686575296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb580a94f0 con 0x7fdb580a9cd0 2023-10-24T23:56:19.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb3ffff640 1 -- 172.21.15.107:0/1686575296 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0047f0 con 0x7fdb580a9cd0 2023-10-24T23:56:19.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.158+0000 7fdb5dfca640 1 -- 172.21.15.107:0/1686575296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb581e5560 con 0x7fdb580a9cd0 2023-10-24T23:56:19.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.162+0000 7fdb5dfca640 1 -- 172.21.15.107:0/1686575296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb581e5720 con 0x7fdb580a9cd0 2023-10-24T23:56:19.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.162+0000 7fdb3ffff640 1 -- 172.21.15.107:0/1686575296 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdb4c004b60 con 0x7fdb580a9cd0 2023-10-24T23:56:19.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.162+0000 7fdb3ffff640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb44065e20 0x7fdb440682e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.162+0000 7fdb3ffff640 1 -- 172.21.15.107:0/1686575296 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7fdb4c083560 con 0x7fdb580a9cd0 2023-10-24T23:56:19.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.162+0000 7fdb577fe640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb44065e20 0x7fdb440682e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.162+0000 7fdb577fe640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb44065e20 0x7fdb440682e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009ab0 tx=0x7fdb48008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:19.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.170+0000 7fdb3ffff640 1 -- 172.21.15.107:0/1686575296 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdb4c0506d0 con 0x7fdb580a9cd0 2023-10-24T23:56:19.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.378+0000 7fdb5dfca640 1 -- 172.21.15.107:0/1686575296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb581e53c0 con 0x7fdb580a9cd0 2023-10-24T23:56:19.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.378+0000 7fdb3ffff640 1 -- 172.21.15.107:0/1686575296 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+5015 (secure 0 0 0) 0x7fdb581e53c0 con 0x7fdb580a9cd0 2023-10-24T23:56:19.384 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:19.384 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:18.382669+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:19.385 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2023-10-24T23:56:19.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 -- 172.21.15.107:0/1686575296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb44065e20 msgr2=0x7fdb440682e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:19.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb44065e20 0x7fdb440682e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009ab0 tx=0x7fdb48008040 comp rx=0 tx=0).stop 2023-10-24T23:56:19.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 -- 172.21.15.107:0/1686575296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb580a9cd0 msgr2=0x7fdb580aa530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:19.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb580a9cd0 0x7fdb580aa530 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00c730 tx=0x7fdb4c00cbf0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 -- 172.21.15.107:0/1686575296 shutdown_connections 2023-10-24T23:56:19.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb44065e20 0x7fdb440682e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb580ac0d0 0x7fdb581e5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb580a9cd0 0x7fdb580aa530 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 --2- 172.21.15.107:0/1686575296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb40004d10 0x7fdb581e8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 -- 172.21.15.107:0/1686575296 >> 172.21.15.107:0/1686575296 conn(0x7fdb580c39a0 msgr2=0x7fdb5814af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:19.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 -- 172.21.15.107:0/1686575296 shutdown_connections 2023-10-24T23:56:19.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.382+0000 7fdb5dfca640 1 -- 172.21.15.107:0/1686575296 wait complete. 2023-10-24T23:56:19.403 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2023-10-24T23:56:19.404 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-24T23:56:19.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:19.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e310e1640 1 Processor -- start 2023-10-24T23:56:19.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e310e1640 1 -- start start 2023-10-24T23:56:19.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e310e1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e2c152040 0x7f5e2c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e310e1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5e2c06a4b0 0x7f5e2c0ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e310e1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2c155680 con 0x7f5e2c0c8100 2023-10-24T23:56:19.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e310e1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c154a70 con 0x7f5e2c06a4b0 2023-10-24T23:56:19.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e310e1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e2c154d50 con 0x7f5e2c152040 2023-10-24T23:56:19.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e2b59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5e2c06a4b0 0x7f5e2c0ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e2b59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5e2c06a4b0 0x7f5e2c0ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50286/0 (socket says 172.21.15.107:50286) 2023-10-24T23:56:19.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e2b59e640 1 -- 172.21.15.107:0/1710440887 learned_addr learned my addr 172.21.15.107:0/1710440887 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:19.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e2ad9d640 1 --2- 172.21.15.107:0/1710440887 >> v2:172.21.15.110:3300/0 conn(0x7f5e2c152040 0x7f5e2c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e2b59e640 1 -- 172.21.15.107:0/1710440887 >> v2:172.21.15.110:3300/0 conn(0x7f5e2c152040 msgr2=0x7f5e2c0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e2b59e640 1 --2- 172.21.15.107:0/1710440887 >> v2:172.21.15.110:3300/0 conn(0x7f5e2c152040 0x7f5e2c0c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e2b59e640 1 -- 172.21.15.107:0/1710440887 >> v1:172.21.15.110:6789/0 conn(0x7f5e2c0c8100 legacy=0x7f5e2c0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:19.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.526+0000 7f5e2b59e640 1 -- 172.21.15.107:0/1710440887 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c155030 con 0x7f5e2c06a4b0 2023-10-24T23:56:19.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e2b59e640 1 --2- 172.21.15.107:0/1710440887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e2c06a4b0 0x7f5e2c0ac410 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c067ba0 tx=0x7f5e2401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238c82432d469821 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e2a59c640 1 -- 172.21.15.107:0/1710440887 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e240a56d0 con 0x7f5e2c06a4b0 2023-10-24T23:56:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e2a59c640 1 -- 172.21.15.107:0/1710440887 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e240a5830 con 0x7f5e2c06a4b0 2023-10-24T23:56:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e2a59c640 1 -- 172.21.15.107:0/1710440887 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e240a5bc0 con 0x7f5e2c06a4b0 2023-10-24T23:56:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e310e1640 1 -- 172.21.15.107:0/1710440887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e2c06a4b0 msgr2=0x7f5e2c0ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:19.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e310e1640 1 --2- 172.21.15.107:0/1710440887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e2c06a4b0 0x7f5e2c0ac410 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c067ba0 tx=0x7f5e2401c900 comp rx=0 tx=0).stop 2023-10-24T23:56:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e310e1640 1 -- 172.21.15.107:0/1710440887 shutdown_connections 2023-10-24T23:56:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e310e1640 1 --2- 172.21.15.107:0/1710440887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e2c06a4b0 0x7f5e2c0ac410 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e310e1640 1 --2- 172.21.15.107:0/1710440887 >> v2:172.21.15.110:3300/0 conn(0x7f5e2c152040 0x7f5e2c0c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e310e1640 1 -- 172.21.15.107:0/1710440887 >> 172.21.15.107:0/1710440887 conn(0x7f5e2c0ca200 msgr2=0x7f5e2c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e310e1640 1 -- 172.21.15.107:0/1710440887 shutdown_connections 2023-10-24T23:56:19.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e310e1640 1 -- 172.21.15.107:0/1710440887 wait complete. 2023-10-24T23:56:19.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e310e1640 1 Processor -- start 2023-10-24T23:56:19.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.530+0000 7f5e310e1640 1 -- start start 2023-10-24T23:56:19.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e310e1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e2c06a4b0 0x7f5e2c1ec8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e310e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e2c0c8100 0x7f5e2c14aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e310e1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e2c152040 0x7f5e2c1f0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e310e1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c1f0260 con 0x7f5e2c0c8100 2023-10-24T23:56:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e310e1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c0ae5b0 con 0x7f5e2c152040 2023-10-24T23:56:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e310e1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c14f5e0 con 0x7f5e2c06a4b0 2023-10-24T23:56:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e23fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e2c0c8100 0x7f5e2c14aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e23fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e2c0c8100 0x7f5e2c14aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50302/0 (socket says 172.21.15.107:50302) 2023-10-24T23:56:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e23fff640 1 -- 172.21.15.107:0/489532142 learned_addr learned my addr 172.21.15.107:0/489532142 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e2b59e640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e2c152040 0x7f5e2c1f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e2ad9d640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e2c06a4b0 0x7f5e2c1ec8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e23fff640 1 -- 172.21.15.107:0/489532142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e2c06a4b0 msgr2=0x7f5e2c1ec8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e23fff640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e2c06a4b0 0x7f5e2c1ec8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e23fff640 1 -- 172.21.15.107:0/489532142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e2c152040 msgr2=0x7f5e2c1f0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:19.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e23fff640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e2c152040 0x7f5e2c1f0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e23fff640 1 -- 172.21.15.107:0/489532142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e24009b30 con 0x7f5e2c0c8100 2023-10-24T23:56:19.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e23fff640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e2c0c8100 0x7f5e2c14aa10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400b3e0 tx=0x7f5e1400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:19.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e237fe640 1 -- 172.21.15.107:0/489532142 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e14011830 con 0x7f5e2c0c8100 2023-10-24T23:56:19.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e237fe640 1 -- 172.21.15.107:0/489532142 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e14011990 con 0x7f5e2c0c8100 2023-10-24T23:56:19.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e237fe640 1 -- 172.21.15.107:0/489532142 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e14011d60 con 0x7f5e2c0c8100 2023-10-24T23:56:19.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e310e1640 1 -- 172.21.15.107:0/489532142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2c1efef0 con 0x7f5e2c0c8100 2023-10-24T23:56:19.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e310e1640 1 -- 172.21.15.107:0/489532142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e2c14af50 con 0x7f5e2c0c8100 2023-10-24T23:56:19.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e310e1640 1 -- 172.21.15.107:0/489532142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e2c0746e0 con 0x7f5e2c0c8100 2023-10-24T23:56:19.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e237fe640 1 -- 172.21.15.107:0/489532142 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5e1401a400 con 0x7f5e2c0c8100 2023-10-24T23:56:19.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e237fe640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e0c065e60 0x7f5e0c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:19.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e237fe640 1 -- 172.21.15.107:0/489532142 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f5e14083e20 con 0x7f5e2c0c8100 2023-10-24T23:56:19.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.534+0000 7f5e2ad9d640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e0c065e60 0x7f5e0c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:19.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.538+0000 7f5e2ad9d640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e0c065e60 0x7f5e0c068320 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00b930 tx=0x7f5e1c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:19.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.542+0000 7f5e237fe640 1 -- 172.21.15.107:0/489532142 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5e14051020 con 0x7f5e2c0c8100 2023-10-24T23:56:19.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.742+0000 7f5e310e1640 1 -- 172.21.15.107:0/489532142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e2c14ddd0 con 0x7f5e2c0c8100 2023-10-24T23:56:19.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.742+0000 7f5e237fe640 1 -- 172.21.15.107:0/489532142 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+5018 (secure 0 0 0) 0x7f5e14054640 con 0x7f5e2c0c8100 2023-10-24T23:56:19.750 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:19.750 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:19.565268+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:19.750 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 30 2023-10-24T23:56:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 -- 172.21.15.107:0/489532142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e0c065e60 msgr2=0x7f5e0c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e0c065e60 0x7f5e0c068320 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00b930 tx=0x7f5e1c008040 comp rx=0 tx=0).stop 2023-10-24T23:56:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 -- 172.21.15.107:0/489532142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e2c0c8100 msgr2=0x7f5e2c14aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e2c0c8100 0x7f5e2c14aa10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5e1400b3e0 tx=0x7f5e1400b8a0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 -- 172.21.15.107:0/489532142 shutdown_connections 2023-10-24T23:56:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e2c152040 0x7f5e2c1f0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e2c0c8100 0x7f5e2c14aa10 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e0c065e60 0x7f5e0c068320 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 --2- 172.21.15.107:0/489532142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e2c06a4b0 0x7f5e2c1ec8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 -- 172.21.15.107:0/489532142 >> 172.21.15.107:0/489532142 conn(0x7f5e2c0ca200 msgr2=0x7f5e2c0ad3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 -- 172.21.15.107:0/489532142 shutdown_connections 2023-10-24T23:56:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:19.746+0000 7f5e310e1640 1 -- 172.21.15.107:0/489532142 wait complete. 2023-10-24T23:56:19.771 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 30, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5403": { "addr": "172.21.15.107:6835/1004926771", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 1004926771, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 1004926771, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5403, "incarnation": 24, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:56:01.459282+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5403 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4799": { "addr": "172.21.15.110:6835/1031240534", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 1031240534, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 1031240534, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4799, "incarnation": 29, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-24T23:56:19.565268+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4799 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/3721101226", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 3721101226, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 3721101226, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4232, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-24T23:56:21.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:21.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.906+0000 7fa7f1677640 1 Processor -- start 2023-10-24T23:56:21.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.906+0000 7fa7f1677640 1 -- start start 2023-10-24T23:56:21.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7f1677640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec152020 0x7fa7ec0c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:21.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7f1677640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa7ec0c8640 0x7fa7ec06a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:21.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7f1677640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ec152400 con 0x7fa7ec1553d0 2023-10-24T23:56:21.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7f1677640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7ec1558d0 con 0x7fa7ec0c8640 2023-10-24T23:56:21.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7f1677640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec0c6ce0 con 0x7fa7ec152020 2023-10-24T23:56:21.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7e3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa7ec0c8640 0x7fa7ec06a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:21.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7e3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa7ec0c8640 0x7fa7ec06a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50308/0 (socket says 172.21.15.107:50308) 2023-10-24T23:56:21.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7e3fff640 1 -- 172.21.15.107:0/1372055870 learned_addr learned my addr 172.21.15.107:0/1372055870 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:21.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7eaffd640 1 --2- 172.21.15.107:0/1372055870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec152020 0x7fa7ec0c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:21.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7e3fff640 1 -- 172.21.15.107:0/1372055870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec152020 msgr2=0x7fa7ec0c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:21.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7e3fff640 1 --2- 172.21.15.107:0/1372055870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec152020 0x7fa7ec0c4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:21.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7e3fff640 1 -- 172.21.15.107:0/1372055870 >> v1:172.21.15.110:6789/0 conn(0x7fa7ec1553d0 legacy=0x7fa7ec06ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:21.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7e3fff640 1 -- 172.21.15.107:0/1372055870 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ec0c6fc0 con 0x7fa7ec0c8640 2023-10-24T23:56:21.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7e3fff640 1 --2- 172.21.15.107:0/1372055870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7ec0c8640 0x7fa7ec06a710 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009870 tx=0x7fa7d401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a4a2d7deef0926 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:21.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7ea7fc640 1 -- 172.21.15.107:0/1372055870 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40ab620 con 0x7fa7ec0c8640 2023-10-24T23:56:21.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7ea7fc640 1 -- 172.21.15.107:0/1372055870 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d40ab780 con 0x7fa7ec0c8640 2023-10-24T23:56:21.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7ea7fc640 1 -- 172.21.15.107:0/1372055870 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40abb10 con 0x7fa7ec0c8640 2023-10-24T23:56:21.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7f1677640 1 -- 172.21.15.107:0/1372055870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7ec0c8640 msgr2=0x7fa7ec06a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:21.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.910+0000 7fa7f1677640 1 --2- 172.21.15.107:0/1372055870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7ec0c8640 0x7fa7ec06a710 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009870 tx=0x7fa7d401c440 comp rx=0 tx=0).stop 2023-10-24T23:56:21.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 -- 172.21.15.107:0/1372055870 shutdown_connections 2023-10-24T23:56:21.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 --2- 172.21.15.107:0/1372055870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7ec0c8640 0x7fa7ec06a710 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:21.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 --2- 172.21.15.107:0/1372055870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec152020 0x7fa7ec0c4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:21.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 -- 172.21.15.107:0/1372055870 >> 172.21.15.107:0/1372055870 conn(0x7fa7ec0abd60 msgr2=0x7fa7ec0ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:21.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 -- 172.21.15.107:0/1372055870 shutdown_connections 2023-10-24T23:56:21.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 -- 172.21.15.107:0/1372055870 wait complete. 2023-10-24T23:56:21.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 Processor -- start 2023-10-24T23:56:21.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 -- start start 2023-10-24T23:56:21.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec0c8640 0x7fa7ec0aad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:21.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7ec152020 0x7fa7ec14a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:21.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7ec1553d0 0x7fa7ec0a9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:21.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec0ab2b0 con 0x7fa7ec152020 2023-10-24T23:56:21.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec1ed990 con 0x7fa7ec1553d0 2023-10-24T23:56:21.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec0abad0 con 0x7fa7ec0c8640 2023-10-24T23:56:21.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7ec152020 0x7fa7ec14a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:21.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7ec152020 0x7fa7ec14a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50314/0 (socket says 172.21.15.107:50314) 2023-10-24T23:56:21.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e3fff640 1 -- 172.21.15.107:0/975216697 learned_addr learned my addr 172.21.15.107:0/975216697 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:21.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7eb7fe640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7ec1553d0 0x7fa7ec0a9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:21.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7eaffd640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec0c8640 0x7fa7ec0aad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:21.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e3fff640 1 -- 172.21.15.107:0/975216697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec0c8640 msgr2=0x7fa7ec0aad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:21.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e3fff640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec0c8640 0x7fa7ec0aad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:21.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e3fff640 1 -- 172.21.15.107:0/975216697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7ec1553d0 msgr2=0x7fa7ec0a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:21.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e3fff640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7ec1553d0 0x7fa7ec0a9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:21.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e3fff640 1 -- 172.21.15.107:0/975216697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d40094d0 con 0x7fa7ec152020 2023-10-24T23:56:21.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e3fff640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7ec152020 0x7fa7ec14a990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40ad040 tx=0x7fa7d401ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:21.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e37fe640 1 -- 172.21.15.107:0/975216697 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40be3f0 con 0x7fa7ec152020 2023-10-24T23:56:21.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e37fe640 1 -- 172.21.15.107:0/975216697 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d40be570 con 0x7fa7ec152020 2023-10-24T23:56:21.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 -- 172.21.15.107:0/975216697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ec0ab4e0 con 0x7fa7ec152020 2023-10-24T23:56:21.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7e37fe640 1 -- 172.21.15.107:0/975216697 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40be920 con 0x7fa7ec152020 2023-10-24T23:56:21.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.914+0000 7fa7f1677640 1 -- 172.21.15.107:0/975216697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ec1ed340 con 0x7fa7ec152020 2023-10-24T23:56:21.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.918+0000 7fa7f1677640 1 -- 172.21.15.107:0/975216697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ec0746e0 con 0x7fa7ec152020 2023-10-24T23:56:21.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.918+0000 7fa7e37fe640 1 -- 172.21.15.107:0/975216697 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa7d40c7440 con 0x7fa7ec152020 2023-10-24T23:56:21.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.918+0000 7fa7e37fe640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7cc065e60 0x7fa7cc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:21.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.918+0000 7fa7e37fe640 1 -- 172.21.15.107:0/975216697 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7fa7d411e4d0 con 0x7fa7ec152020 2023-10-24T23:56:21.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.918+0000 7fa7eaffd640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7cc065e60 0x7fa7cc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:21.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.918+0000 7fa7eaffd640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7cc065e60 0x7fa7cc068320 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec158dd0 tx=0x7fa7dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:21.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:21.922+0000 7fa7e37fe640 1 -- 172.21.15.107:0/975216697 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa7d40eb6d0 con 0x7fa7ec152020 2023-10-24T23:56:22.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.134+0000 7fa7f1677640 1 -- 172.21.15.107:0/975216697 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7ec0c7800 con 0x7fa7ec152020 2023-10-24T23:56:22.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.134+0000 7fa7e37fe640 1 -- 172.21.15.107:0/975216697 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5015 (secure 0 0 0) 0x7fa7d40eecf0 con 0x7fa7ec152020 2023-10-24T23:56:22.140 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:22.140 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:22.141 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-10-24T23:56:22.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 -- 172.21.15.107:0/975216697 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7cc065e60 msgr2=0x7fa7cc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:22.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7cc065e60 0x7fa7cc068320 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec158dd0 tx=0x7fa7dc008040 comp rx=0 tx=0).stop 2023-10-24T23:56:22.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 -- 172.21.15.107:0/975216697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7ec152020 msgr2=0x7fa7ec14a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:22.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7ec152020 0x7fa7ec14a990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa7d40ad040 tx=0x7fa7d401ac80 comp rx=0 tx=0).stop 2023-10-24T23:56:22.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 -- 172.21.15.107:0/975216697 shutdown_connections 2023-10-24T23:56:22.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7ec1553d0 0x7fa7ec0a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:22.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7ec152020 0x7fa7ec14a990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:22.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7cc065e60 0x7fa7cc068320 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:22.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 --2- 172.21.15.107:0/975216697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7ec0c8640 0x7fa7ec0aad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:22.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 -- 172.21.15.107:0/975216697 >> 172.21.15.107:0/975216697 conn(0x7fa7ec0abd60 msgr2=0x7fa7ec1512d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:22.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 -- 172.21.15.107:0/975216697 shutdown_connections 2023-10-24T23:56:22.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:22.138+0000 7fa7f1677640 1 -- 172.21.15.107:0/975216697 wait complete. 2023-10-24T23:56:22.158 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=5292 2023-10-24T23:56:22.158 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.a 2023-10-24T23:56:25.526 INFO:tasks.ceph.mds.d.smithi110.stderr:2023-10-24T23:56:25.521+0000 7fae79939640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:56:28.160 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2023-10-24T23:56:28.160 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-24T23:56:28.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-24T23:56:28.163 INFO:tasks.ceph.mds.a:Started 2023-10-24T23:56:28.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:28.246 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-24T23:56:28.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08cba0640 1 Processor -- start 2023-10-24T23:56:28.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08cba0640 1 -- start start 2023-10-24T23:56:28.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08cba0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe088152040 0x7fe088158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:28.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08cba0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe088152420 con 0x7fe0880adb80 2023-10-24T23:56:28.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08cba0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe088154d40 con 0x7fe08806e800 2023-10-24T23:56:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08cba0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0881556e0 con 0x7fe088152040 2023-10-24T23:56:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe085d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe0880adb80 0x7fe088071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53936/0 (socket says 172.21.15.107:53936) 2023-10-24T23:56:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe085d9b640 1 -- 172.21.15.107:0/3236824273 learned_addr learned my addr 172.21.15.107:0/3236824273 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:56:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08659c640 1 --2- 172.21.15.107:0/3236824273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe088152040 0x7fe088158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749132964 0 0) 0x7fe088152420 con 0x7fe0880adb80 2023-10-24T23:56:28.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe070003530 con 0x7fe0880adb80 2023-10-24T23:56:28.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3604074641 0 0) 0x7fe070003530 con 0x7fe0880adb80 2023-10-24T23:56:28.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe088152040 msgr2=0x7fe088158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:28.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 --2- 172.21.15.107:0/3236824273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe088152040 0x7fe088158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:28.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 >> v1:172.21.15.110:6789/0 conn(0x7fe08806e800 legacy=0x7fe08806ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:28.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe088155020 con 0x7fe0880adb80 2023-10-24T23:56:28.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fe07c001680 con 0x7fe0880adb80 2023-10-24T23:56:28.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 >> v1:172.21.15.107:6789/0 conn(0x7fe0880adb80 legacy=0x7fe088071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:28.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 --2- 172.21.15.107:0/3236824273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe088152040 0x7fe0700039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:28.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 --2- 172.21.15.107:0/3236824273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe070003f30 0x7fe0700047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:28.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.298+0000 7fe08559a640 1 --2- 172.21.15.107:0/3236824273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe070004d10 0x7fe070005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:28.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0881556e0 con 0x7fe070003f30 2023-10-24T23:56:28.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07000bb20 con 0x7fe070004d10 2023-10-24T23:56:28.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07000bce0 con 0x7fe088152040 2023-10-24T23:56:28.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe085d9b640 1 --2- 172.21.15.107:0/3236824273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe070003f30 0x7fe0700047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:28.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe08659c640 1 --2- 172.21.15.107:0/3236824273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe088152040 0x7fe0700039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:28.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe085d9b640 1 -- 172.21.15.107:0/3236824273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe088152040 msgr2=0x7fe0700039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:28.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe085d9b640 1 --2- 172.21.15.107:0/3236824273 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe088152040 0x7fe0700039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:28.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe085d9b640 1 -- 172.21.15.107:0/3236824273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe070004d10 msgr2=0x7fe070005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:28.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe085d9b640 1 --2- 172.21.15.107:0/3236824273 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe070004d10 0x7fe070005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:28.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe085d9b640 1 -- 172.21.15.107:0/3236824273 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe070003530 con 0x7fe070003f30 2023-10-24T23:56:28.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe085d9b640 1 --2- 172.21.15.107:0/3236824273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe070003f30 0x7fe0700047d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00c210 tx=0x7fe07c01cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b765feba34dec74 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c0058e0 con 0x7fe070003f30 2023-10-24T23:56:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe07c005a60 con 0x7fe070003f30 2023-10-24T23:56:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe08559a640 1 -- 172.21.15.107:0/3236824273 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c005e10 con 0x7fe070003f30 2023-10-24T23:56:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe08cba0640 1 -- 172.21.15.107:0/3236824273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe070003f30 msgr2=0x7fe0700047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe08cba0640 1 --2- 172.21.15.107:0/3236824273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe070003f30 0x7fe0700047d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00c210 tx=0x7fe07c01cbd0 comp rx=0 tx=0).stop 2023-10-24T23:56:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe086d9d640 1 -- 172.21.15.107:0/3236824273 reap_dead start 2023-10-24T23:56:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe08cba0640 1 -- 172.21.15.107:0/3236824273 shutdown_connections 2023-10-24T23:56:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.302+0000 7fe08cba0640 1 -- 172.21.15.107:0/3236824273 >> 172.21.15.107:0/3236824273 conn(0x7fe088070f60 msgr2=0x7fe088071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 -- 172.21.15.107:0/3236824273 shutdown_connections 2023-10-24T23:56:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 -- 172.21.15.107:0/3236824273 wait complete. 2023-10-24T23:56:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 Processor -- start 2023-10-24T23:56:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 -- start start 2023-10-24T23:56:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe08814adc0 0x7fe0881ecb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0880c7a10 0x7fe0881ed080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe088147770 0x7fe088147bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0881571a0 con 0x7fe088147770 2023-10-24T23:56:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0881f00b0 con 0x7fe08814adc0 2023-10-24T23:56:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08814dfd0 con 0x7fe0880c7a10 2023-10-24T23:56:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe086d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe088147770 0x7fe088147bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe086d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe088147770 0x7fe088147bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34904/0 (socket says 172.21.15.107:34904) 2023-10-24T23:56:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe086d9d640 1 -- 172.21.15.107:0/3005686698 learned_addr learned my addr 172.21.15.107:0/3005686698 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:28.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08659c640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe08814adc0 0x7fe0881ecb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:28.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe085d9b640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0880c7a10 0x7fe0881ed080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:28.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe086d9d640 1 -- 172.21.15.107:0/3005686698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0880c7a10 msgr2=0x7fe0881ed080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:28.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe086d9d640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0880c7a10 0x7fe0881ed080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:28.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe086d9d640 1 -- 172.21.15.107:0/3005686698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe08814adc0 msgr2=0x7fe0881ecb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe086d9d640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe08814adc0 0x7fe0881ecb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe086d9d640 1 -- 172.21.15.107:0/3005686698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08814e2b0 con 0x7fe088147770 2023-10-24T23:56:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe086d9d640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe088147770 0x7fe088147bf0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe088066a60 tx=0x7fe08000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe076ffd640 1 -- 172.21.15.107:0/3005686698 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080005d50 con 0x7fe088147770 2023-10-24T23:56:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe076ffd640 1 -- 172.21.15.107:0/3005686698 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08000ced0 con 0x7fe088147770 2023-10-24T23:56:28.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe076ffd640 1 -- 172.21.15.107:0/3005686698 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08000a820 con 0x7fe088147770 2023-10-24T23:56:28.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 -- 172.21.15.107:0/3005686698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0881f0300 con 0x7fe088147770 2023-10-24T23:56:28.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.306+0000 7fe08cba0640 1 -- 172.21.15.107:0/3005686698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0881f08c0 con 0x7fe088147770 2023-10-24T23:56:28.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.310+0000 7fe08cba0640 1 -- 172.21.15.107:0/3005686698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0880746e0 con 0x7fe088147770 2023-10-24T23:56:28.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.310+0000 7fe076ffd640 1 -- 172.21.15.107:0/3005686698 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe08000a980 con 0x7fe088147770 2023-10-24T23:56:28.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.310+0000 7fe076ffd640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe068065e60 0x7fe068068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:28.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.310+0000 7fe076ffd640 1 -- 172.21.15.107:0/3005686698 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7fe080082be0 con 0x7fe088147770 2023-10-24T23:56:28.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.310+0000 7fe08659c640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe068065e60 0x7fe068068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:28.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.310+0000 7fe08659c640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe068065e60 0x7fe068068320 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe078009a30 tx=0x7fe078008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:28.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.318+0000 7fe076ffd640 1 -- 172.21.15.107:0/3005686698 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe08004fd50 con 0x7fe088147770 2023-10-24T23:56:28.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.530+0000 7fe08cba0640 1 -- 172.21.15.107:0/3005686698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0880ad960 con 0x7fe088147770 2023-10-24T23:56:28.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.530+0000 7fe076ffd640 1 -- 172.21.15.107:0/3005686698 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+5015 (secure 0 0 0) 0x7fe080053370 con 0x7fe088147770 2023-10-24T23:56:28.537 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:28.537 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:28.537 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 32 2023-10-24T23:56:28.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 -- 172.21.15.107:0/3005686698 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe068065e60 msgr2=0x7fe068068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:28.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe068065e60 0x7fe068068320 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe078009a30 tx=0x7fe078008040 comp rx=0 tx=0).stop 2023-10-24T23:56:28.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 -- 172.21.15.107:0/3005686698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe088147770 msgr2=0x7fe088147bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:28.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe088147770 0x7fe088147bf0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe088066a60 tx=0x7fe08000cb90 comp rx=0 tx=0).stop 2023-10-24T23:56:28.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 -- 172.21.15.107:0/3005686698 shutdown_connections 2023-10-24T23:56:28.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe088147770 0x7fe088147bf0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:28.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe068065e60 0x7fe068068320 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:28.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0880c7a10 0x7fe0881ed080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:28.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 --2- 172.21.15.107:0/3005686698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe08814adc0 0x7fe0881ecb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:28.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 -- 172.21.15.107:0/3005686698 >> 172.21.15.107:0/3005686698 conn(0x7fe088070f60 msgr2=0x7fe08814b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 -- 172.21.15.107:0/3005686698 shutdown_connections 2023-10-24T23:56:28.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:28.534+0000 7fe08cba0640 1 -- 172.21.15.107:0/3005686698 wait complete. 2023-10-24T23:56:28.557 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-24T23:56:30.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:56:30.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a43731640 1 Processor -- start 2023-10-24T23:56:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a43731640 1 -- start start 2023-10-24T23:56:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a43731640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7a3c14c470 0x7f7a3c14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a43731640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3c06f490 0x7f7a3c071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a43731640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c155cd0 con 0x7f7a3c071fc0 2023-10-24T23:56:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a43731640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3c1526e0 con 0x7f7a3c14c470 2023-10-24T23:56:30.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a43731640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c0c4380 con 0x7f7a3c06f490 2023-10-24T23:56:30.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a414cd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7a3c14c470 0x7f7a3c14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:30.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a414cd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7a3c14c470 0x7f7a3c14c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34912/0 (socket says 172.21.15.107:34912) 2023-10-24T23:56:30.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a414cd640 1 -- 172.21.15.107:0/22187196 learned_addr learned my addr 172.21.15.107:0/22187196 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:30.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a40ccc640 1 --2- 172.21.15.107:0/22187196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3c06f490 0x7f7a3c071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:30.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a414cd640 1 -- 172.21.15.107:0/22187196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3c06f490 msgr2=0x7f7a3c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a414cd640 1 --2- 172.21.15.107:0/22187196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3c06f490 0x7f7a3c071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a414cd640 1 -- 172.21.15.107:0/22187196 >> v1:172.21.15.110:6789/0 conn(0x7f7a3c071fc0 legacy=0x7f7a3c152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:56:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.698+0000 7f7a414cd640 1 -- 172.21.15.107:0/22187196 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c0c48b0 con 0x7f7a3c14c470 2023-10-24T23:56:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a414cd640 1 --2- 172.21.15.107:0/22187196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3c14c470 0x7f7a3c14c850 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009870 tx=0x7f7a3801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec7ed697c47d5303 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a33fff640 1 -- 172.21.15.107:0/22187196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a380ab600 con 0x7f7a3c14c470 2023-10-24T23:56:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a33fff640 1 -- 172.21.15.107:0/22187196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a380a7050 con 0x7f7a3c14c470 2023-10-24T23:56:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a33fff640 1 -- 172.21.15.107:0/22187196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a380ab980 con 0x7f7a3c14c470 2023-10-24T23:56:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a43731640 1 -- 172.21.15.107:0/22187196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3c14c470 msgr2=0x7f7a3c14c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a43731640 1 --2- 172.21.15.107:0/22187196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3c14c470 0x7f7a3c14c850 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009870 tx=0x7f7a3801c3e0 comp rx=0 tx=0).stop 2023-10-24T23:56:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a43731640 1 -- 172.21.15.107:0/22187196 shutdown_connections 2023-10-24T23:56:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a43731640 1 --2- 172.21.15.107:0/22187196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3c06f490 0x7f7a3c071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a43731640 1 --2- 172.21.15.107:0/22187196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3c14c470 0x7f7a3c14c850 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a43731640 1 -- 172.21.15.107:0/22187196 >> 172.21.15.107:0/22187196 conn(0x7f7a3c070f60 msgr2=0x7f7a3c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a43731640 1 -- 172.21.15.107:0/22187196 shutdown_connections 2023-10-24T23:56:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a43731640 1 -- 172.21.15.107:0/22187196 wait complete. 2023-10-24T23:56:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a43731640 1 Processor -- start 2023-10-24T23:56:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.702+0000 7f7a43731640 1 -- start start 2023-10-24T23:56:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a43731640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3c06f490 0x7f7a3c0c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a43731640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a3c071fc0 0x7f7a3c138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a43731640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3c14c470 0x7f7a3c138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a43731640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c139300 con 0x7f7a3c14c470 2023-10-24T23:56:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a43731640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c13ccf0 con 0x7f7a3c071fc0 2023-10-24T23:56:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a43731640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c13ca80 con 0x7f7a3c06f490 2023-10-24T23:56:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a41cce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3c14c470 0x7f7a3c138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a41cce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3c14c470 0x7f7a3c138dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34920/0 (socket says 172.21.15.107:34920) 2023-10-24T23:56:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a41cce640 1 -- 172.21.15.107:0/3846508204 learned_addr learned my addr 172.21.15.107:0/3846508204 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:56:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a40ccc640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a3c071fc0 0x7f7a3c138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a414cd640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3c06f490 0x7f7a3c0c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a41cce640 1 -- 172.21.15.107:0/3846508204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3c06f490 msgr2=0x7f7a3c0c76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a41cce640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3c06f490 0x7f7a3c0c76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:30.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a41cce640 1 -- 172.21.15.107:0/3846508204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a3c071fc0 msgr2=0x7f7a3c138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:30.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a41cce640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a3c071fc0 0x7f7a3c138880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:30.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a41cce640 1 -- 172.21.15.107:0/3846508204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a380094d0 con 0x7f7a3c14c470 2023-10-24T23:56:30.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a41cce640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3c14c470 0x7f7a3c138dc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400c780 tx=0x7f7a3400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:30.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a31ffb640 1 -- 172.21.15.107:0/3846508204 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a34004b70 con 0x7f7a3c14c470 2023-10-24T23:56:30.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a31ffb640 1 -- 172.21.15.107:0/3846508204 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3400ce90 con 0x7f7a3c14c470 2023-10-24T23:56:30.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a43731640 1 -- 172.21.15.107:0/3846508204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3c147870 con 0x7f7a3c14c470 2023-10-24T23:56:30.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a31ffb640 1 -- 172.21.15.107:0/3846508204 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3400a850 con 0x7f7a3c14c470 2023-10-24T23:56:30.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a43731640 1 -- 172.21.15.107:0/3846508204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c0c6cf0 con 0x7f7a3c14c470 2023-10-24T23:56:30.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a43731640 1 -- 172.21.15.107:0/3846508204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3c147d90 con 0x7f7a3c14c470 2023-10-24T23:56:30.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a31ffb640 1 -- 172.21.15.107:0/3846508204 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7a3400a9b0 con 0x7f7a3c14c470 2023-10-24T23:56:30.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a31ffb640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a20065e60 0x7f7a20068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:56:30.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.706+0000 7f7a31ffb640 1 -- 172.21.15.107:0/3846508204 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f7a34083e30 con 0x7f7a3c14c470 2023-10-24T23:56:30.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.710+0000 7f7a414cd640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a20065e60 0x7f7a20068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:56:30.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.710+0000 7f7a414cd640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a20065e60 0x7f7a20068320 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7a380ad000 tx=0x7f7a380a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:56:30.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.714+0000 7f7a31ffb640 1 -- 172.21.15.107:0/3846508204 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7a3c147d90 con 0x7f7a3c14c470 2023-10-24T23:56:30.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.930+0000 7f7a43731640 1 -- 172.21.15.107:0/3846508204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a3c147d90 con 0x7f7a3c14c470 2023-10-24T23:56:30.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.930+0000 7f7a31ffb640 1 -- 172.21.15.107:0/3846508204 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f7a3c147d90 con 0x7f7a3c14c470 2023-10-24T23:56:30.937 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:56:30.937 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:56:30.938 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:56:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 -- 172.21.15.107:0/3846508204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a20065e60 msgr2=0x7f7a20068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a20065e60 0x7f7a20068320 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7a380ad000 tx=0x7f7a380a6000 comp rx=0 tx=0).stop 2023-10-24T23:56:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 -- 172.21.15.107:0/3846508204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3c14c470 msgr2=0x7f7a3c138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:56:30.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3c14c470 0x7f7a3c138dc0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400c780 tx=0x7f7a3400cc40 comp rx=0 tx=0).stop 2023-10-24T23:56:30.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 -- 172.21.15.107:0/3846508204 shutdown_connections 2023-10-24T23:56:30.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3c14c470 0x7f7a3c138dc0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:30.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a3c071fc0 0x7f7a3c138880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:30.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a20065e60 0x7f7a20068320 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:30.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 --2- 172.21.15.107:0/3846508204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3c06f490 0x7f7a3c0c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:56:30.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 -- 172.21.15.107:0/3846508204 >> 172.21.15.107:0/3846508204 conn(0x7f7a3c070f60 msgr2=0x7f7a3c071340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:56:30.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 -- 172.21.15.107:0/3846508204 shutdown_connections 2023-10-24T23:56:30.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:56:30.934+0000 7f7a43731640 1 -- 172.21.15.107:0/3846508204 wait complete. 2023-10-24T23:56:30.959 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2023-10-24T23:56:30.960 INFO:tasks.mds_thrash.fs.[b]:waiting for 34 secs before thrashing 2023-10-24T23:56:34.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:56:34.900 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:57:04.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:05.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:57:05.048 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:57:05.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.094+0000 7f605b0d6640 1 Processor -- start 2023-10-24T23:57:05.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f605b0d6640 1 -- start start 2023-10-24T23:57:05.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f605b0d6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6054069d40 0x7f6054158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f605b0d6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f605406a120 0x7f60540aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f605b0d6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605406cea0 con 0x7f6054065bc0 2023-10-24T23:57:05.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f605b0d6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60540ac650 con 0x7f6054069d40 2023-10-24T23:57:05.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f605b0d6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60540a9e00 con 0x7f605406a120 2023-10-24T23:57:05.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f6058e72640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6054069d40 0x7f6054158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f6058e72640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6054069d40 0x7f6054158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51698/0 (socket says 172.21.15.107:51698) 2023-10-24T23:57:05.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f6058e72640 1 -- 172.21.15.107:0/1698891443 learned_addr learned my addr 172.21.15.107:0/1698891443 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:05.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f6059673640 1 --2- 172.21.15.107:0/1698891443 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f605406a120 0x7f60540aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f6058e72640 1 -- 172.21.15.107:0/1698891443 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f605406a120 msgr2=0x7f60540aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:05.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f6058e72640 1 --2- 172.21.15.107:0/1698891443 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f605406a120 0x7f60540aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f6058e72640 1 -- 172.21.15.107:0/1698891443 >> v1:172.21.15.110:6789/0 conn(0x7f6054065bc0 legacy=0x7f60540a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f6058e72640 1 -- 172.21.15.107:0/1698891443 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60541584e0 con 0x7f6054069d40 2023-10-24T23:57:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f6058e72640 1 --2- 172.21.15.107:0/1698891443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6054069d40 0x7f6054158970 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6048009870 tx=0x7f604801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79958d10ac968ee server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f60537fe640 1 -- 172.21.15.107:0/1698891443 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480ac620 con 0x7f6054069d40 2023-10-24T23:57:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f60537fe640 1 -- 172.21.15.107:0/1698891443 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60480ac780 con 0x7f6054069d40 2023-10-24T23:57:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.098+0000 7f60537fe640 1 -- 172.21.15.107:0/1698891443 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480acb10 con 0x7f6054069d40 2023-10-24T23:57:05.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 -- 172.21.15.107:0/1698891443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6054069d40 msgr2=0x7f6054158970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 --2- 172.21.15.107:0/1698891443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6054069d40 0x7f6054158970 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6048009870 tx=0x7f604801c4b0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 -- 172.21.15.107:0/1698891443 shutdown_connections 2023-10-24T23:57:05.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 --2- 172.21.15.107:0/1698891443 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f605406a120 0x7f60540aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 --2- 172.21.15.107:0/1698891443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6054069d40 0x7f6054158970 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 -- 172.21.15.107:0/1698891443 >> 172.21.15.107:0/1698891443 conn(0x7f60540c65c0 msgr2=0x7f60540c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:05.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 -- 172.21.15.107:0/1698891443 shutdown_connections 2023-10-24T23:57:05.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 -- 172.21.15.107:0/1698891443 wait complete. 2023-10-24T23:57:05.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 Processor -- start 2023-10-24T23:57:05.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 -- start start 2023-10-24T23:57:05.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6054065bc0 0x7f6054071410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6054069d40 0x7f6054152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f605406a120 0x7f60541534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605414b330 con 0x7f6054065bc0 2023-10-24T23:57:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60540ba0f0 con 0x7f6054069d40 2023-10-24T23:57:05.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60540ba620 con 0x7f605406a120 2023-10-24T23:57:05.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f6058e72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6054065bc0 0x7f6054071410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f6058e72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6054065bc0 0x7f6054071410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51700/0 (socket says 172.21.15.107:51700) 2023-10-24T23:57:05.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f6058e72640 1 -- 172.21.15.107:0/1923251924 learned_addr learned my addr 172.21.15.107:0/1923251924 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:05.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f6059673640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f605406a120 0x7f60541534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f6053fff640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6054069d40 0x7f6054152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f6058e72640 1 -- 172.21.15.107:0/1923251924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f605406a120 msgr2=0x7f60541534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f6058e72640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f605406a120 0x7f60541534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f6058e72640 1 -- 172.21.15.107:0/1923251924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6054069d40 msgr2=0x7f6054152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f6058e72640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6054069d40 0x7f6054152660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f6058e72640 1 -- 172.21.15.107:0/1923251924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60480094d0 con 0x7f6054065bc0 2023-10-24T23:57:05.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f6058e72640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6054065bc0 0x7f6054071410 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f604801ab60 tx=0x7f6048019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:05.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f60517fa640 1 -- 172.21.15.107:0/1923251924 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480b5630 con 0x7f6054065bc0 2023-10-24T23:57:05.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f60517fa640 1 -- 172.21.15.107:0/1923251924 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60480b5790 con 0x7f6054065bc0 2023-10-24T23:57:05.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 -- 172.21.15.107:0/1923251924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60540c4160 con 0x7f6054065bc0 2023-10-24T23:57:05.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f60517fa640 1 -- 172.21.15.107:0/1923251924 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480b5b40 con 0x7f6054065bc0 2023-10-24T23:57:05.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.102+0000 7f605b0d6640 1 -- 172.21.15.107:0/1923251924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60540ba340 con 0x7f6054065bc0 2023-10-24T23:57:05.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.106+0000 7f605b0d6640 1 -- 172.21.15.107:0/1923251924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60540746e0 con 0x7f6054065bc0 2023-10-24T23:57:05.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.106+0000 7f60517fa640 1 -- 172.21.15.107:0/1923251924 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f60480b3040 con 0x7f6054065bc0 2023-10-24T23:57:05.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.106+0000 7f60517fa640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6038065e90 0x7f6038068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.106+0000 7f60517fa640 1 -- 172.21.15.107:0/1923251924 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f604811e520 con 0x7f6054065bc0 2023-10-24T23:57:05.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.106+0000 7f6053fff640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6038065e90 0x7f6038068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.106+0000 7f6053fff640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6038065e90 0x7f6038068350 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f605414dcd0 tx=0x7f6040009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:05.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.114+0000 7f60517fa640 1 -- 172.21.15.107:0/1923251924 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f60480eb820 con 0x7f6054065bc0 2023-10-24T23:57:05.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.322+0000 7f605b0d6640 1 -- 172.21.15.107:0/1923251924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f605415eef0 con 0x7f6054065bc0 2023-10-24T23:57:05.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.322+0000 7f60517fa640 1 -- 172.21.15.107:0/1923251924 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f60480eee40 con 0x7f6054065bc0 2023-10-24T23:57:05.329 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:05.329 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:05.330 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:05.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 -- 172.21.15.107:0/1923251924 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6038065e90 msgr2=0x7f6038068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6038065e90 0x7f6038068350 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f605414dcd0 tx=0x7f6040009040 comp rx=0 tx=0).stop 2023-10-24T23:57:05.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 -- 172.21.15.107:0/1923251924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6054065bc0 msgr2=0x7f6054071410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6054065bc0 0x7f6054071410 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f604801ab60 tx=0x7f6048019250 comp rx=0 tx=0).stop 2023-10-24T23:57:05.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 -- 172.21.15.107:0/1923251924 shutdown_connections 2023-10-24T23:57:05.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6038065e90 0x7f6038068350 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f605406a120 0x7f60541534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6054069d40 0x7f6054152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 --2- 172.21.15.107:0/1923251924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6054065bc0 0x7f6054071410 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 -- 172.21.15.107:0/1923251924 >> 172.21.15.107:0/1923251924 conn(0x7f60540c65c0 msgr2=0x7f605414df80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:05.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 -- 172.21.15.107:0/1923251924 shutdown_connections 2023-10-24T23:57:05.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.326+0000 7f605b0d6640 1 -- 172.21.15.107:0/1923251924 wait complete. 2023-10-24T23:57:05.348 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2023-10-24T23:57:05.348 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-24T23:57:05.349 INFO:teuthology.orchestra.run:waiting for 300 2023-10-24T23:57:05.368 INFO:tasks.ceph.mds.d:Stopped 2023-10-24T23:57:05.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:05.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.506+0000 7f436b59e640 1 Processor -- start 2023-10-24T23:57:05.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.506+0000 7f436b59e640 1 -- start start 2023-10-24T23:57:05.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436b59e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f436c0abb50 0x7f436c0ac000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436b59e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c157e20 0x7f436c1582d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436b59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436c14eaa0 con 0x7f436c14e6c0 2023-10-24T23:57:05.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436b59e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f436c0c5b80 con 0x7f436c0abb50 2023-10-24T23:57:05.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436b59e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436c072130 con 0x7f436c157e20 2023-10-24T23:57:05.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436a59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f436c14e6c0 0x7f436c155040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33602/0 (socket says 172.21.15.107:33602) 2023-10-24T23:57:05.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436a59c640 1 -- 172.21.15.107:0/2599626473 learned_addr learned my addr 172.21.15.107:0/2599626473 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:05.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436ad9d640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c157e20 0x7f436c1582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f4369d9b640 1 --2- 172.21.15.107:0/2599626473 >> v2:172.21.15.110:3300/0 conn(0x7f436c0abb50 0x7f436c0ac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663087869 0 0) 0x7f436c14eaa0 con 0x7f436c14e6c0 2023-10-24T23:57:05.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4354003530 con 0x7f436c14e6c0 2023-10-24T23:57:05.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3811851693 0 0) 0x7f4354003530 con 0x7f436c14e6c0 2023-10-24T23:57:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c157e20 msgr2=0x7f436c1582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c157e20 0x7f436c1582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 >> v2:172.21.15.110:3300/0 conn(0x7f436c0abb50 msgr2=0x7f436c0ac000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 --2- 172.21.15.107:0/2599626473 >> v2:172.21.15.110:3300/0 conn(0x7f436c0abb50 0x7f436c0ac000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c072410 con 0x7f436c14e6c0 2023-10-24T23:57:05.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f435c001680 con 0x7f436c14e6c0 2023-10-24T23:57:05.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 >> v1:172.21.15.107:6789/0 conn(0x7f436c14e6c0 legacy=0x7f436c155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4354003980 0x7f4354003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540047d0 0x7f4354004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c157e20 0x7f4354005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c072130 con 0x7f43540047d0 2023-10-24T23:57:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f435400b990 con 0x7f4354003980 2023-10-24T23:57:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435400bb00 con 0x7f436c157e20 2023-10-24T23:57:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f4369d9b640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540047d0 0x7f4354004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436a59c640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4354003980 0x7f4354003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436ad9d640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c157e20 0x7f4354005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f4369d9b640 1 -- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c157e20 msgr2=0x7f4354005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:05.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f4369d9b640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c157e20 0x7f4354005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f4369d9b640 1 -- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4354003980 msgr2=0x7f4354003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:05.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f4369d9b640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4354003980 0x7f4354003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f4369d9b640 1 -- 172.21.15.107:0/2599626473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4354003530 con 0x7f43540047d0 2023-10-24T23:57:05.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.510+0000 7f436a59c640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4354003980 0x7f4354003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:57:05.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f4369d9b640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540047d0 0x7f4354004c50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4360009ae0 tx=0x7f436001ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce7473d34517cc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:05.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43600a5550 con 0x7f43540047d0 2023-10-24T23:57:05.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43600a56b0 con 0x7f43540047d0 2023-10-24T23:57:05.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436959a640 1 -- 172.21.15.107:0/2599626473 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43600a5a60 con 0x7f43540047d0 2023-10-24T23:57:05.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 -- 172.21.15.107:0/2599626473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540047d0 msgr2=0x7f4354004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 --2- 172.21.15.107:0/2599626473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43540047d0 0x7f4354004c50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4360009ae0 tx=0x7f436001ca00 comp rx=0 tx=0).stop 2023-10-24T23:57:05.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436ad9d640 1 -- 172.21.15.107:0/2599626473 reap_dead start 2023-10-24T23:57:05.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 -- 172.21.15.107:0/2599626473 shutdown_connections 2023-10-24T23:57:05.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 -- 172.21.15.107:0/2599626473 >> 172.21.15.107:0/2599626473 conn(0x7f436c0ca050 msgr2=0x7f436c0ca460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:05.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 -- 172.21.15.107:0/2599626473 shutdown_connections 2023-10-24T23:57:05.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 -- 172.21.15.107:0/2599626473 wait complete. 2023-10-24T23:57:05.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 Processor -- start 2023-10-24T23:57:05.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 -- start start 2023-10-24T23:57:05.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f436c1495c0 0x7f436c1499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c1ec550 0x7f436c1efea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f436c153bf0 0x7f436c1f03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c1577f0 con 0x7f436c153bf0 2023-10-24T23:57:05.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c14f030 con 0x7f436c1495c0 2023-10-24T23:57:05.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436b59e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436c1eca10 con 0x7f436c1ec550 2023-10-24T23:57:05.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f436c153bf0 0x7f436c1f03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f436c153bf0 0x7f436c1f03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51724/0 (socket says 172.21.15.107:51724) 2023-10-24T23:57:05.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436ad9d640 1 -- 172.21.15.107:0/3068035997 learned_addr learned my addr 172.21.15.107:0/3068035997 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:05.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f436a59c640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f436c1495c0 0x7f436c1499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.514+0000 7f4369d9b640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c1ec550 0x7f436c1efea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f436ad9d640 1 -- 172.21.15.107:0/3068035997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c1ec550 msgr2=0x7f436c1efea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f436ad9d640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c1ec550 0x7f436c1efea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f436ad9d640 1 -- 172.21.15.107:0/3068035997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f436c1495c0 msgr2=0x7f436c1499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f436ad9d640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f436c1495c0 0x7f436c1499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f436ad9d640 1 -- 172.21.15.107:0/3068035997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c1eccf0 con 0x7f436c153bf0 2023-10-24T23:57:05.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f436ad9d640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f436c153bf0 0x7f436c1f03e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f436c06b330 tx=0x7f43640115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:05.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f435affd640 1 -- 172.21.15.107:0/3068035997 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436400e070 con 0x7f436c153bf0 2023-10-24T23:57:05.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f435affd640 1 -- 172.21.15.107:0/3068035997 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4364011e10 con 0x7f436c153bf0 2023-10-24T23:57:05.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f436b59e640 1 -- 172.21.15.107:0/3068035997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436c1ecfd0 con 0x7f436c153bf0 2023-10-24T23:57:05.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f435affd640 1 -- 172.21.15.107:0/3068035997 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43640039a0 con 0x7f436c153bf0 2023-10-24T23:57:05.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f436b59e640 1 -- 172.21.15.107:0/3068035997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436c1f0920 con 0x7f436c153bf0 2023-10-24T23:57:05.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f436b59e640 1 -- 172.21.15.107:0/3068035997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436c0c9e50 con 0x7f436c153bf0 2023-10-24T23:57:05.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f435affd640 1 -- 172.21.15.107:0/3068035997 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4364003b00 con 0x7f436c153bf0 2023-10-24T23:57:05.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f435affd640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f434c065e60 0x7f434c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:05.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f435affd640 1 -- 172.21.15.107:0/3068035997 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f4364084ee0 con 0x7f436c153bf0 2023-10-24T23:57:05.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f436a59c640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f434c065e60 0x7f434c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:05.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.518+0000 7f436a59c640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f434c065e60 0x7f434c068320 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f435c000f70 tx=0x7f435c007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:05.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.526+0000 7f435affd640 1 -- 172.21.15.107:0/3068035997 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4364052270 con 0x7f436c153bf0 2023-10-24T23:57:05.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.802+0000 7f436b59e640 1 -- 172.21.15.107:0/3068035997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f436c1efb00 con 0x7f436c153bf0 2023-10-24T23:57:05.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.802+0000 7f435affd640 1 -- 172.21.15.107:0/3068035997 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f4364055890 con 0x7f436c153bf0 2023-10-24T23:57:05.809 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:05.809 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:05.810 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:05.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 -- 172.21.15.107:0/3068035997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f434c065e60 msgr2=0x7f434c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f434c065e60 0x7f434c068320 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f435c000f70 tx=0x7f435c007040 comp rx=0 tx=0).stop 2023-10-24T23:57:05.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 -- 172.21.15.107:0/3068035997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f436c153bf0 msgr2=0x7f436c1f03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:05.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f436c153bf0 0x7f436c1f03e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f436c06b330 tx=0x7f43640115a0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 -- 172.21.15.107:0/3068035997 shutdown_connections 2023-10-24T23:57:05.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f436c153bf0 0x7f436c1f03e0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f434c065e60 0x7f434c068320 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f436c1ec550 0x7f436c1efea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 --2- 172.21.15.107:0/3068035997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f436c1495c0 0x7f436c1499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:05.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 -- 172.21.15.107:0/3068035997 >> 172.21.15.107:0/3068035997 conn(0x7f436c0ca050 msgr2=0x7f436c156580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:05.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 -- 172.21.15.107:0/3068035997 shutdown_connections 2023-10-24T23:57:05.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:05.806+0000 7f436b59e640 1 -- 172.21.15.107:0/3068035997 wait complete. 2023-10-24T23:57:05.828 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:57:07.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:07.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.970+0000 7fb5ebf50640 1 Processor -- start 2023-10-24T23:57:07.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.970+0000 7fb5ebf50640 1 -- start start 2023-10-24T23:57:07.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ebf50640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb5e40aa330 0x7fb5e406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:07.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ebf50640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb5e40a9450 0x7fb5e4071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:07.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ebf50640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e4152d40 con 0x7fb5e413bb00 2023-10-24T23:57:07.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ebf50640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e4156100 con 0x7fb5e40a9450 2023-10-24T23:57:07.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ebf50640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e41527c0 con 0x7fb5e40aa330 2023-10-24T23:57:07.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ea4ed640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb5e40a9450 0x7fb5e4071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:07.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ea4ed640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb5e40a9450 0x7fb5e4071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51734/0 (socket says 172.21.15.107:51734) 2023-10-24T23:57:07.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ea4ed640 1 -- 172.21.15.107:0/182375972 learned_addr learned my addr 172.21.15.107:0/182375972 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:07.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5e94eb640 1 --2- 172.21.15.107:0/182375972 >> v2:172.21.15.110:3300/0 conn(0x7fb5e40aa330 0x7fb5e406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:07.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ea4ed640 1 -- 172.21.15.107:0/182375972 >> v2:172.21.15.110:3300/0 conn(0x7fb5e40aa330 msgr2=0x7fb5e406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:07.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ea4ed640 1 --2- 172.21.15.107:0/182375972 >> v2:172.21.15.110:3300/0 conn(0x7fb5e40aa330 0x7fb5e406e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:07.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ea4ed640 1 -- 172.21.15.107:0/182375972 >> v1:172.21.15.110:6789/0 conn(0x7fb5e413bb00 legacy=0x7fb5e41531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:07.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ea4ed640 1 -- 172.21.15.107:0/182375972 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e406d660 con 0x7fb5e40a9450 2023-10-24T23:57:07.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5e94eb640 1 --2- 172.21.15.107:0/182375972 >> v2:172.21.15.110:3300/0 conn(0x7fb5e40aa330 0x7fb5e406e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5ea4ed640 1 --2- 172.21.15.107:0/182375972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5e40a9450 0x7fb5e4071a80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009870 tx=0x7fb5e001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c507fcbc0ac6ecb server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5e8cea640 1 -- 172.21.15.107:0/182375972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e00ab600 con 0x7fb5e40a9450 2023-10-24T23:57:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5e8cea640 1 -- 172.21.15.107:0/182375972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e00a7050 con 0x7fb5e40a9450 2023-10-24T23:57:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.974+0000 7fb5e8cea640 1 -- 172.21.15.107:0/182375972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e00ab980 con 0x7fb5e40a9450 2023-10-24T23:57:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 -- 172.21.15.107:0/182375972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5e40a9450 msgr2=0x7fb5e4071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 --2- 172.21.15.107:0/182375972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5e40a9450 0x7fb5e4071a80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009870 tx=0x7fb5e001c3a0 comp rx=0 tx=0).stop 2023-10-24T23:57:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 -- 172.21.15.107:0/182375972 shutdown_connections 2023-10-24T23:57:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 --2- 172.21.15.107:0/182375972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5e40a9450 0x7fb5e4071a80 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 --2- 172.21.15.107:0/182375972 >> v2:172.21.15.110:3300/0 conn(0x7fb5e40aa330 0x7fb5e406e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 -- 172.21.15.107:0/182375972 >> 172.21.15.107:0/182375972 conn(0x7fb5e4070f60 msgr2=0x7fb5e4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 -- 172.21.15.107:0/182375972 shutdown_connections 2023-10-24T23:57:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 -- 172.21.15.107:0/182375972 wait complete. 2023-10-24T23:57:07.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 Processor -- start 2023-10-24T23:57:07.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 -- start start 2023-10-24T23:57:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5e40a9450 0x7fb5e40c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5e40aa330 0x7fb5e41338c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5e413bb00 0x7fb5e4131290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e4151920 con 0x7fb5e40a9450 2023-10-24T23:57:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40c5dc0 con 0x7fb5e40aa330 2023-10-24T23:57:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e4154ee0 con 0x7fb5e413bb00 2023-10-24T23:57:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5e9cec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5e40a9450 0x7fb5e40c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5e9cec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5e40a9450 0x7fb5e40c6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51742/0 (socket says 172.21.15.107:51742) 2023-10-24T23:57:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5e9cec640 1 -- 172.21.15.107:0/1133096422 learned_addr learned my addr 172.21.15.107:0/1133096422 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:07.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ea4ed640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5e413bb00 0x7fb5e4131290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:07.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5e94eb640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5e40aa330 0x7fb5e41338c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:07.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5e9cec640 1 -- 172.21.15.107:0/1133096422 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5e413bb00 msgr2=0x7fb5e4131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:07.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5e9cec640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5e413bb00 0x7fb5e4131290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:07.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5e9cec640 1 -- 172.21.15.107:0/1133096422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5e40aa330 msgr2=0x7fb5e41338c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:07.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5e9cec640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5e40aa330 0x7fb5e41338c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:07.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5e9cec640 1 -- 172.21.15.107:0/1133096422 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00094d0 con 0x7fb5e40a9450 2023-10-24T23:57:07.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5e9cec640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5e40a9450 0x7fb5e40c6af0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00ba50 tx=0x7fb5dc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:07.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5da7fc640 1 -- 172.21.15.107:0/1133096422 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc00cc10 con 0x7fb5e40a9450 2023-10-24T23:57:07.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5da7fc640 1 -- 172.21.15.107:0/1133096422 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5dc00cd70 con 0x7fb5e40a9450 2023-10-24T23:57:07.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 -- 172.21.15.107:0/1133096422 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e40c6010 con 0x7fb5e40a9450 2023-10-24T23:57:07.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5da7fc640 1 -- 172.21.15.107:0/1133096422 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc014680 con 0x7fb5e40a9450 2023-10-24T23:57:07.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.978+0000 7fb5ebf50640 1 -- 172.21.15.107:0/1133096422 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e40c65d0 con 0x7fb5e40a9450 2023-10-24T23:57:07.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.982+0000 7fb5ebf50640 1 -- 172.21.15.107:0/1133096422 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e40c67c0 con 0x7fb5e40a9450 2023-10-24T23:57:07.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.982+0000 7fb5da7fc640 1 -- 172.21.15.107:0/1133096422 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb5dc007370 con 0x7fb5e40a9450 2023-10-24T23:57:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.982+0000 7fb5da7fc640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5c8065e90 0x7fb5c8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.982+0000 7fb5da7fc640 1 -- 172.21.15.107:0/1133096422 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7fb5dc083810 con 0x7fb5e40a9450 2023-10-24T23:57:07.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.982+0000 7fb5e94eb640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5c8065e90 0x7fb5c8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.982+0000 7fb5e94eb640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5c8065e90 0x7fb5c8068350 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4006fd0 tx=0x7fb5d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:07.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:07.986+0000 7fb5da7fc640 1 -- 172.21.15.107:0/1133096422 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb5dc050a10 con 0x7fb5e40a9450 2023-10-24T23:57:08.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.186+0000 7fb5ebf50640 1 -- 172.21.15.107:0/1133096422 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5e41529a0 con 0x7fb5e40a9450 2023-10-24T23:57:08.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5da7fc640 1 -- 172.21.15.107:0/1133096422 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7fb5dc054030 con 0x7fb5e40a9450 2023-10-24T23:57:08.193 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:08.194 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:08.194 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:08.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 -- 172.21.15.107:0/1133096422 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5c8065e90 msgr2=0x7fb5c8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:08.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5c8065e90 0x7fb5c8068350 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4006fd0 tx=0x7fb5d4008040 comp rx=0 tx=0).stop 2023-10-24T23:57:08.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 -- 172.21.15.107:0/1133096422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5e40a9450 msgr2=0x7fb5e40c6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:08.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5e40a9450 0x7fb5e40c6af0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00ba50 tx=0x7fb5dc00bf10 comp rx=0 tx=0).stop 2023-10-24T23:57:08.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 -- 172.21.15.107:0/1133096422 shutdown_connections 2023-10-24T23:57:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5c8065e90 0x7fb5c8068350 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5e413bb00 0x7fb5e4131290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5e40aa330 0x7fb5e41338c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 --2- 172.21.15.107:0/1133096422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5e40a9450 0x7fb5e40c6af0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:08.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 -- 172.21.15.107:0/1133096422 >> 172.21.15.107:0/1133096422 conn(0x7fb5e4070f60 msgr2=0x7fb5e4071340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 -- 172.21.15.107:0/1133096422 shutdown_connections 2023-10-24T23:57:08.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:08.190+0000 7fb5ebf50640 1 -- 172.21.15.107:0/1133096422 wait complete. 2023-10-24T23:57:08.216 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:57:10.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:10.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f402ce0d640 1 Processor -- start 2023-10-24T23:57:10.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f402ce0d640 1 -- start start 2023-10-24T23:57:10.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f402ce0d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f40280c8100 0x7f40280abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:10.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f402ce0d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f402806a4b0 0x7f40280ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:10.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f402ce0d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4028155680 con 0x7f4028152040 2023-10-24T23:57:10.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f402ce0d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4028154a70 con 0x7f40280c8100 2023-10-24T23:57:10.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f402ce0d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4028154d50 con 0x7f402806a4b0 2023-10-24T23:57:10.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f4025d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f40280c8100 0x7f40280abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:10.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f402659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4028152040 0x7f40280c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33612/0 (socket says 172.21.15.107:33612) 2023-10-24T23:57:10.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f4025d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f40280c8100 0x7f40280abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51748/0 (socket says 172.21.15.107:51748) 2023-10-24T23:57:10.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f402659c640 1 -- 172.21.15.107:0/3768083324 learned_addr learned my addr 172.21.15.107:0/3768083324 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:10.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f4026d9d640 1 --2- 172.21.15.107:0/3768083324 >> v2:172.21.15.110:3300/0 conn(0x7f402806a4b0 0x7f40280ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:10.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f4025d9b640 1 -- 172.21.15.107:0/3768083324 >> v2:172.21.15.110:3300/0 conn(0x7f402806a4b0 msgr2=0x7f40280ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:10.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f4025d9b640 1 --2- 172.21.15.107:0/3768083324 >> v2:172.21.15.110:3300/0 conn(0x7f402806a4b0 0x7f40280ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:10.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f4025d9b640 1 -- 172.21.15.107:0/3768083324 >> v1:172.21.15.107:6789/0 conn(0x7f4028152040 legacy=0x7f40280c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:10.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f4025d9b640 1 -- 172.21.15.107:0/3768083324 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4028155030 con 0x7f40280c8100 2023-10-24T23:57:10.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f402559a640 1 -- 172.21.15.107:0/3768083324 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592943578 0 0) 0x7f4028155680 con 0x7f4028152040 2023-10-24T23:57:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.354+0000 7f4025d9b640 1 --2- 172.21.15.107:0/3768083324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40280c8100 0x7f40280abd00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f402806b590 tx=0x7f401c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bffac0aacf21d06 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:10.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402559a640 1 -- 172.21.15.107:0/3768083324 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c0ab600 con 0x7f40280c8100 2023-10-24T23:57:10.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402559a640 1 -- 172.21.15.107:0/3768083324 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f401c0a7050 con 0x7f40280c8100 2023-10-24T23:57:10.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402559a640 1 -- 172.21.15.107:0/3768083324 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401c0ab980 con 0x7f40280c8100 2023-10-24T23:57:10.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 -- 172.21.15.107:0/3768083324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40280c8100 msgr2=0x7f40280abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:10.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 --2- 172.21.15.107:0/3768083324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40280c8100 0x7f40280abd00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f402806b590 tx=0x7f401c01c3d0 comp rx=0 tx=0).stop 2023-10-24T23:57:10.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 -- 172.21.15.107:0/3768083324 shutdown_connections 2023-10-24T23:57:10.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 --2- 172.21.15.107:0/3768083324 >> v2:172.21.15.110:3300/0 conn(0x7f402806a4b0 0x7f40280ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:10.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 --2- 172.21.15.107:0/3768083324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40280c8100 0x7f40280abd00 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:10.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 -- 172.21.15.107:0/3768083324 >> 172.21.15.107:0/3768083324 conn(0x7f40280ca200 msgr2=0x7f40280ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:10.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 -- 172.21.15.107:0/3768083324 shutdown_connections 2023-10-24T23:57:10.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 -- 172.21.15.107:0/3768083324 wait complete. 2023-10-24T23:57:10.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 Processor -- start 2023-10-24T23:57:10.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 -- start start 2023-10-24T23:57:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f402806a4b0 0x7f4028146f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:10.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40280c8100 0x7f4028149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:10.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4028152040 0x7f402814f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:10.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4028151000 con 0x7f402806a4b0 2023-10-24T23:57:10.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40280bd1b0 con 0x7f4028152040 2023-10-24T23:57:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402ce0d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40280c0850 con 0x7f40280c8100 2023-10-24T23:57:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f402806a4b0 0x7f4028146f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f402806a4b0 0x7f4028146f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51752/0 (socket says 172.21.15.107:51752) 2023-10-24T23:57:10.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402659c640 1 -- 172.21.15.107:0/3864428020 learned_addr learned my addr 172.21.15.107:0/3864428020 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:10.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f4025d9b640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40280c8100 0x7f4028149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:10.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402659c640 1 -- 172.21.15.107:0/3864428020 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40280c8100 msgr2=0x7f4028149570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:10.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f4026d9d640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4028152040 0x7f402814f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:10.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402659c640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40280c8100 0x7f4028149570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402659c640 1 -- 172.21.15.107:0/3864428020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4028152040 msgr2=0x7f402814f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:10.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402659c640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4028152040 0x7f402814f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f402659c640 1 -- 172.21.15.107:0/3864428020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c009420 con 0x7f402806a4b0 2023-10-24T23:57:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.358+0000 7f4025d9b640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40280c8100 0x7f4028149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:10.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f402659c640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f402806a4b0 0x7f4028146f90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f40180011f0 tx=0x7f401800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f4016ffd640 1 -- 172.21.15.107:0/3864428020 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40180074e0 con 0x7f402806a4b0 2023-10-24T23:57:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f4016ffd640 1 -- 172.21.15.107:0/3864428020 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4018007640 con 0x7f402806a4b0 2023-10-24T23:57:10.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f402ce0d640 1 -- 172.21.15.107:0/3864428020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40281ed600 con 0x7f402806a4b0 2023-10-24T23:57:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f4016ffd640 1 -- 172.21.15.107:0/3864428020 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4018004a10 con 0x7f402806a4b0 2023-10-24T23:57:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f402ce0d640 1 -- 172.21.15.107:0/3864428020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40281ed8e0 con 0x7f402806a4b0 2023-10-24T23:57:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f402ce0d640 1 -- 172.21.15.107:0/3864428020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40280746e0 con 0x7f402806a4b0 2023-10-24T23:57:10.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f4016ffd640 1 -- 172.21.15.107:0/3864428020 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4018004b70 con 0x7f402806a4b0 2023-10-24T23:57:10.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f4016ffd640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4008065e60 0x7f4008068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:10.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f4016ffd640 1 -- 172.21.15.107:0/3864428020 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f4018083240 con 0x7f402806a4b0 2023-10-24T23:57:10.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f4025d9b640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4008065e60 0x7f4008068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:10.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.362+0000 7f4025d9b640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4008065e60 0x7f4008068320 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f401c0ad000 tx=0x7f401c0a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:10.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.370+0000 7f4016ffd640 1 -- 172.21.15.107:0/3864428020 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4018050440 con 0x7f402806a4b0 2023-10-24T23:57:10.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.606+0000 7f402ce0d640 1 -- 172.21.15.107:0/3864428020 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f402814edd0 con 0x7f402806a4b0 2023-10-24T23:57:10.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f4016ffd640 1 -- 172.21.15.107:0/3864428020 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f4018053a60 con 0x7f402806a4b0 2023-10-24T23:57:10.613 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:10.614 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:10.614 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:10.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 -- 172.21.15.107:0/3864428020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4008065e60 msgr2=0x7f4008068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:10.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4008065e60 0x7f4008068320 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f401c0ad000 tx=0x7f401c0a6000 comp rx=0 tx=0).stop 2023-10-24T23:57:10.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 -- 172.21.15.107:0/3864428020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f402806a4b0 msgr2=0x7f4028146f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:10.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f402806a4b0 0x7f4028146f90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f40180011f0 tx=0x7f401800c9b0 comp rx=0 tx=0).stop 2023-10-24T23:57:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 -- 172.21.15.107:0/3864428020 shutdown_connections 2023-10-24T23:57:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4028152040 0x7f402814f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4008065e60 0x7f4008068320 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40280c8100 0x7f4028149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:10.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 --2- 172.21.15.107:0/3864428020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f402806a4b0 0x7f4028146f90 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:10.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 -- 172.21.15.107:0/3864428020 >> 172.21.15.107:0/3864428020 conn(0x7f40280ca200 msgr2=0x7f4028153f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:10.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 -- 172.21.15.107:0/3864428020 shutdown_connections 2023-10-24T23:57:10.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:10.610+0000 7f402ce0d640 1 -- 172.21.15.107:0/3864428020 wait complete. 2023-10-24T23:57:10.637 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:57:12.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:12.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.774+0000 7f3e5f551640 1 Processor -- start 2023-10-24T23:57:12.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.774+0000 7f3e5f551640 1 -- start start 2023-10-24T23:57:12.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5f551640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e5813dc20 0x7f3e580c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:12.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5f551640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e58157e50 0x7f3e581582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:12.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5f551640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e58154860 con 0x7f3e580720c0 2023-10-24T23:57:12.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5f551640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e58158800 con 0x7f3e58157e50 2023-10-24T23:57:12.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5f551640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580c1f10 con 0x7f3e5813dc20 2023-10-24T23:57:12.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5caec640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3e580720c0 0x7f3e5806df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52046/0 (socket says 172.21.15.107:52046) 2023-10-24T23:57:12.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5caec640 1 -- 172.21.15.107:0/3992096088 learned_addr learned my addr 172.21.15.107:0/3992096088 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:12.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5daee640 1 --2- 172.21.15.107:0/3992096088 >> v2:172.21.15.107:3300/0 conn(0x7f3e58157e50 0x7f3e581582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:12.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5d2ed640 1 --2- 172.21.15.107:0/3992096088 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e5813dc20 0x7f3e580c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:12.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5daee640 1 -- 172.21.15.107:0/3992096088 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e5813dc20 msgr2=0x7f3e580c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:12.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5daee640 1 --2- 172.21.15.107:0/3992096088 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e5813dc20 0x7f3e580c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:12.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5daee640 1 -- 172.21.15.107:0/3992096088 >> v1:172.21.15.107:6789/0 conn(0x7f3e580720c0 legacy=0x7f3e5806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:12.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e57fff640 1 -- 172.21.15.107:0/3992096088 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573708323 0 0) 0x7f3e58154860 con 0x7f3e580720c0 2023-10-24T23:57:12.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5daee640 1 -- 172.21.15.107:0/3992096088 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e58154b20 con 0x7f3e58157e50 2023-10-24T23:57:12.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e5daee640 1 --2- 172.21.15.107:0/3992096088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e58157e50 0x7f3e581582c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3e500097c0 tx=0x7f3e5001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dba02d8673b54e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:12.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e57fff640 1 -- 172.21.15.107:0/3992096088 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e500ab600 con 0x7f3e58157e50 2023-10-24T23:57:12.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e57fff640 1 -- 172.21.15.107:0/3992096088 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e500a7050 con 0x7f3e58157e50 2023-10-24T23:57:12.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.778+0000 7f3e57fff640 1 -- 172.21.15.107:0/3992096088 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e500ab980 con 0x7f3e58157e50 2023-10-24T23:57:12.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 -- 172.21.15.107:0/3992096088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e58157e50 msgr2=0x7f3e581582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:12.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 --2- 172.21.15.107:0/3992096088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e58157e50 0x7f3e581582c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f3e500097c0 tx=0x7f3e5001c360 comp rx=0 tx=0).stop 2023-10-24T23:57:12.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 -- 172.21.15.107:0/3992096088 shutdown_connections 2023-10-24T23:57:12.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 --2- 172.21.15.107:0/3992096088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e58157e50 0x7f3e581582c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:12.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 --2- 172.21.15.107:0/3992096088 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e5813dc20 0x7f3e580c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:12.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 -- 172.21.15.107:0/3992096088 >> 172.21.15.107:0/3992096088 conn(0x7f3e580c39a0 msgr2=0x7f3e580c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:12.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 -- 172.21.15.107:0/3992096088 shutdown_connections 2023-10-24T23:57:12.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 -- 172.21.15.107:0/3992096088 wait complete. 2023-10-24T23:57:12.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 Processor -- start 2023-10-24T23:57:12.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 -- start start 2023-10-24T23:57:12.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e580720c0 0x7f3e581efb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:12.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e5813dc20 0x7f3e581edd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:12.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e58157e50 0x7f3e581ee250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:12.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581ed880 con 0x7f3e580720c0 2023-10-24T23:57:12.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580a94f0 con 0x7f3e58157e50 2023-10-24T23:57:12.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e581f04c0 con 0x7f3e5813dc20 2023-10-24T23:57:12.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5d2ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e580720c0 0x7f3e581efb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:12.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5d2ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e580720c0 0x7f3e581efb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46860/0 (socket says 172.21.15.107:46860) 2023-10-24T23:57:12.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5d2ed640 1 -- 172.21.15.107:0/1159546155 learned_addr learned my addr 172.21.15.107:0/1159546155 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:12.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5caec640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e5813dc20 0x7f3e581edd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:12.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5daee640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e58157e50 0x7f3e581ee250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:12.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5d2ed640 1 -- 172.21.15.107:0/1159546155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e5813dc20 msgr2=0x7f3e581edd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:12.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5d2ed640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e5813dc20 0x7f3e581edd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5d2ed640 1 -- 172.21.15.107:0/1159546155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e58157e50 msgr2=0x7f3e581ee250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5d2ed640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e58157e50 0x7f3e581ee250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5d2ed640 1 -- 172.21.15.107:0/1159546155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e50009420 con 0x7f3e580720c0 2023-10-24T23:57:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5daee640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e58157e50 0x7f3e581ee250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:12.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5d2ed640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e580720c0 0x7f3e581efb70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3e48007ef0 tx=0x7f3e4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e55ffb640 1 -- 172.21.15.107:0/1159546155 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48014620 con 0x7f3e580720c0 2023-10-24T23:57:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e55ffb640 1 -- 172.21.15.107:0/1159546155 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e48014780 con 0x7f3e580720c0 2023-10-24T23:57:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 -- 172.21.15.107:0/1159546155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e580a9790 con 0x7f3e580720c0 2023-10-24T23:57:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e55ffb640 1 -- 172.21.15.107:0/1159546155 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48014b30 con 0x7f3e580720c0 2023-10-24T23:57:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.782+0000 7f3e5f551640 1 -- 172.21.15.107:0/1159546155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e58157b60 con 0x7f3e580720c0 2023-10-24T23:57:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.786+0000 7f3e5f551640 1 -- 172.21.15.107:0/1159546155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e580746e0 con 0x7f3e580720c0 2023-10-24T23:57:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.786+0000 7f3e55ffb640 1 -- 172.21.15.107:0/1159546155 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3e48014c90 con 0x7f3e580720c0 2023-10-24T23:57:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.786+0000 7f3e55ffb640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e44065e60 0x7f3e44068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.786+0000 7f3e55ffb640 1 -- 172.21.15.107:0/1159546155 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f3e48084620 con 0x7f3e580720c0 2023-10-24T23:57:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.786+0000 7f3e5caec640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e44065e60 0x7f3e44068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:12.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.786+0000 7f3e5caec640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e44065e60 0x7f3e44068320 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3e5813c910 tx=0x7f3e4c007c10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:12.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:12.790+0000 7f3e55ffb640 1 -- 172.21.15.107:0/1159546155 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3e48051820 con 0x7f3e580720c0 2023-10-24T23:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 -- 172.21.15.107:0/1159546155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e580c1190 con 0x7f3e580720c0 2023-10-24T23:57:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e55ffb640 1 -- 172.21.15.107:0/1159546155 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f3e48054e40 con 0x7f3e580720c0 2023-10-24T23:57:13.027 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:13.027 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:13.028 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 -- 172.21.15.107:0/1159546155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e44065e60 msgr2=0x7f3e44068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e44065e60 0x7f3e44068320 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3e5813c910 tx=0x7f3e4c007c10 comp rx=0 tx=0).stop 2023-10-24T23:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 -- 172.21.15.107:0/1159546155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e580720c0 msgr2=0x7f3e581efb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:13.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e580720c0 0x7f3e581efb70 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3e48007ef0 tx=0x7f3e4800e4b0 comp rx=0 tx=0).stop 2023-10-24T23:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 -- 172.21.15.107:0/1159546155 shutdown_connections 2023-10-24T23:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e58157e50 0x7f3e581ee250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e44065e60 0x7f3e44068320 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e5813dc20 0x7f3e581edd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:13.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 --2- 172.21.15.107:0/1159546155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e580720c0 0x7f3e581efb70 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 -- 172.21.15.107:0/1159546155 >> 172.21.15.107:0/1159546155 conn(0x7f3e580c39a0 msgr2=0x7f3e5813cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 -- 172.21.15.107:0/1159546155 shutdown_connections 2023-10-24T23:57:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:13.022+0000 7f3e5f551640 1 -- 172.21.15.107:0/1159546155 wait complete. 2023-10-24T23:57:13.045 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:57:14.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:14.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.590+0000 7f55184e0640 1 Processor -- start 2023-10-24T23:57:14.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.590+0000 7f55184e0640 1 -- start start 2023-10-24T23:57:14.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f55184e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5510151fb0 0x7f5510155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:14.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f55184e0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5510158bf0 0x7f55100c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:14.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f55184e0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5510152390 con 0x7f55100c82e0 2023-10-24T23:57:14.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f55184e0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55100c5630 con 0x7f5510151fb0 2023-10-24T23:57:14.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f55184e0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5510158760 con 0x7f5510158bf0 2023-10-24T23:57:14.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f551627c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5510151fb0 0x7f5510155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:14.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f551627c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5510151fb0 0x7f5510155560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46864/0 (socket says 172.21.15.107:46864) 2023-10-24T23:57:14.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f551627c640 1 -- 172.21.15.107:0/663659527 learned_addr learned my addr 172.21.15.107:0/663659527 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:14.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f5515a7b640 1 --2- 172.21.15.107:0/663659527 >> v2:172.21.15.110:3300/0 conn(0x7f5510158bf0 0x7f55100c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:14.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f551627c640 1 -- 172.21.15.107:0/663659527 >> v2:172.21.15.110:3300/0 conn(0x7f5510158bf0 msgr2=0x7f55100c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:14.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f551627c640 1 --2- 172.21.15.107:0/663659527 >> v2:172.21.15.110:3300/0 conn(0x7f5510158bf0 0x7f55100c4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:14.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f551627c640 1 -- 172.21.15.107:0/663659527 >> v1:172.21.15.110:6789/0 conn(0x7f55100c82e0 legacy=0x7f55100c6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:14.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f551627c640 1 -- 172.21.15.107:0/663659527 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55100c5080 con 0x7f5510151fb0 2023-10-24T23:57:14.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.594+0000 7f551627c640 1 --2- 172.21.15.107:0/663659527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5510151fb0 0x7f5510155560 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5510155aa0 tx=0x7f550c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa33e716e27f9aa server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f551527a640 1 -- 172.21.15.107:0/663659527 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c0ac640 con 0x7f5510151fb0 2023-10-24T23:57:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f551527a640 1 -- 172.21.15.107:0/663659527 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550c0ac7a0 con 0x7f5510151fb0 2023-10-24T23:57:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f551527a640 1 -- 172.21.15.107:0/663659527 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c0acb30 con 0x7f5510151fb0 2023-10-24T23:57:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 -- 172.21.15.107:0/663659527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5510151fb0 msgr2=0x7f5510155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 --2- 172.21.15.107:0/663659527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5510151fb0 0x7f5510155560 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5510155aa0 tx=0x7f550c01c400 comp rx=0 tx=0).stop 2023-10-24T23:57:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 -- 172.21.15.107:0/663659527 shutdown_connections 2023-10-24T23:57:14.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 --2- 172.21.15.107:0/663659527 >> v2:172.21.15.110:3300/0 conn(0x7f5510158bf0 0x7f55100c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:14.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 --2- 172.21.15.107:0/663659527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5510151fb0 0x7f5510155560 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:14.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 -- 172.21.15.107:0/663659527 >> 172.21.15.107:0/663659527 conn(0x7f55100abcf0 msgr2=0x7f55100ac100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:14.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 -- 172.21.15.107:0/663659527 shutdown_connections 2023-10-24T23:57:14.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 -- 172.21.15.107:0/663659527 wait complete. 2023-10-24T23:57:14.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 Processor -- start 2023-10-24T23:57:14.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 -- start start 2023-10-24T23:57:14.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f55100c82e0 0x7f551014a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:14.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5510151fb0 0x7f5510152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:14.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5510158bf0 0x7f551014b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:14.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55100be880 con 0x7f5510158bf0 2023-10-24T23:57:14.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55100be530 con 0x7f5510151fb0 2023-10-24T23:57:14.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f55184e0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55100bdc90 con 0x7f55100c82e0 2023-10-24T23:57:14.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f551627c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f55100c82e0 0x7f551014a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:14.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f5515a7b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5510151fb0 0x7f5510152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:14.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f551627c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f55100c82e0 0x7f551014a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:37188/0 (socket says 172.21.15.107:37188) 2023-10-24T23:57:14.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f5515a7b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5510151fb0 0x7f5510152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:50996/0 (socket says 172.21.15.107:50996) 2023-10-24T23:57:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.598+0000 7f551627c640 1 -- 172.21.15.107:0/253877124 learned_addr learned my addr 172.21.15.107:0/253877124 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f551627c640 1 -- 172.21.15.107:0/253877124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5510151fb0 msgr2=0x7f5510152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f551627c640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5510151fb0 0x7f5510152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f551627c640 1 -- 172.21.15.107:0/253877124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5510158bf0 msgr2=0x7f551014b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f5516a7d640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5510158bf0 0x7f551014b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:14.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f551627c640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5510158bf0 0x7f551014b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:14.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f551627c640 1 -- 172.21.15.107:0/253877124 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c0094d0 con 0x7f55100c82e0 2023-10-24T23:57:14.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f5516a7d640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5510158bf0 0x7f551014b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:14.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f551627c640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f55100c82e0 0x7f551014a1a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f550c0af040 tx=0x7f550c01ac40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:14.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f5506ffd640 1 -- 172.21.15.107:0/253877124 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c0b7830 con 0x7f55100c82e0 2023-10-24T23:57:14.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f5506ffd640 1 -- 172.21.15.107:0/253877124 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550c0b7990 con 0x7f55100c82e0 2023-10-24T23:57:14.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f5506ffd640 1 -- 172.21.15.107:0/253877124 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c0b7d40 con 0x7f55100c82e0 2023-10-24T23:57:14.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f55184e0640 1 -- 172.21.15.107:0/253877124 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55100c0780 con 0x7f55100c82e0 2023-10-24T23:57:14.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f55184e0640 1 -- 172.21.15.107:0/253877124 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55100be1c0 con 0x7f55100c82e0 2023-10-24T23:57:14.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f55184e0640 1 -- 172.21.15.107:0/253877124 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55100746e0 con 0x7f55100c82e0 2023-10-24T23:57:14.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f5506ffd640 1 -- 172.21.15.107:0/253877124 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f550c0b3040 con 0x7f55100c82e0 2023-10-24T23:57:14.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f5506ffd640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54f4065e60 0x7f54f4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:14.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.602+0000 7f5506ffd640 1 -- 172.21.15.107:0/253877124 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f550c11e6e0 con 0x7f55100c82e0 2023-10-24T23:57:14.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.606+0000 7f5515a7b640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54f4065e60 0x7f54f4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:14.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.606+0000 7f5515a7b640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54f4065e60 0x7f54f4068320 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f55101f1b20 tx=0x7f5500008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:14.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.614+0000 7f5506ffd640 1 -- 172.21.15.107:0/253877124 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f550c0eb970 con 0x7f55100c82e0 2023-10-24T23:57:14.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.918+0000 7f55184e0640 1 -- 172.21.15.107:0/253877124 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5510152390 con 0x7f55100c82e0 2023-10-24T23:57:14.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f5506ffd640 1 -- 172.21.15.107:0/253877124 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f5510152390 con 0x7f55100c82e0 2023-10-24T23:57:14.926 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:14.926 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:14.927 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:14.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 -- 172.21.15.107:0/253877124 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54f4065e60 msgr2=0x7f54f4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:14.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54f4065e60 0x7f54f4068320 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f55101f1b20 tx=0x7f5500008040 comp rx=0 tx=0).stop 2023-10-24T23:57:14.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 -- 172.21.15.107:0/253877124 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f55100c82e0 msgr2=0x7f551014a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:14.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f55100c82e0 0x7f551014a1a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f550c0af040 tx=0x7f550c01ac40 comp rx=0 tx=0).stop 2023-10-24T23:57:14.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 -- 172.21.15.107:0/253877124 shutdown_connections 2023-10-24T23:57:14.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5510158bf0 0x7f551014b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:14.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5510151fb0 0x7f5510152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:14.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54f4065e60 0x7f54f4068320 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:14.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 --2- 172.21.15.107:0/253877124 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f55100c82e0 0x7f551014a1a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:14.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 -- 172.21.15.107:0/253877124 >> 172.21.15.107:0/253877124 conn(0x7f55100abcf0 msgr2=0x7f5510150740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:14.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 -- 172.21.15.107:0/253877124 shutdown_connections 2023-10-24T23:57:14.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:14.922+0000 7f55184e0640 1 -- 172.21.15.107:0/253877124 wait complete. 2023-10-24T23:57:14.949 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2023-10-24T23:57:14.949 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-24T23:57:14.949 INFO:teuthology.orchestra.run:waiting for 300 2023-10-24T23:57:14.960 INFO:tasks.ceph.mds.c:Stopped 2023-10-24T23:57:14.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:15.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:15.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9ccba66640 1 Processor -- start 2023-10-24T23:57:15.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9ccba66640 1 -- start start 2023-10-24T23:57:15.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9ccba66640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9cc413bb00 0x7f9cc41531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9ccba66640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40aa330 0x7f9cc406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9ccba66640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc4152d40 con 0x7f9cc40a9450 2023-10-24T23:57:15.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9ccba66640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc4156100 con 0x7f9cc413bb00 2023-10-24T23:57:15.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9ccba66640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc41527c0 con 0x7f9cc40aa330 2023-10-24T23:57:15.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cca003640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9cc40a9450 0x7f9cc4071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52058/0 (socket says 172.21.15.107:52058) 2023-10-24T23:57:15.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cca003640 1 -- 172.21.15.107:0/38739374 learned_addr learned my addr 172.21.15.107:0/38739374 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:15.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc9001640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40aa330 0x7f9cc406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc9802640 1 --2- 172.21.15.107:0/38739374 >> v2:172.21.15.110:3300/0 conn(0x7f9cc413bb00 0x7f9cc41531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754833210 0 0) 0x7f9cc4152d40 con 0x7f9cc40a9450 2023-10-24T23:57:15.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cb0003530 con 0x7f9cc40a9450 2023-10-24T23:57:15.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2237778097 0 0) 0x7f9cb0003530 con 0x7f9cc40a9450 2023-10-24T23:57:15.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40aa330 msgr2=0x7f9cc406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:15.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40aa330 0x7f9cc406e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 >> v2:172.21.15.110:3300/0 conn(0x7f9cc413bb00 msgr2=0x7f9cc41531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:15.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 --2- 172.21.15.107:0/38739374 >> v2:172.21.15.110:3300/0 conn(0x7f9cc413bb00 0x7f9cc41531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc406d660 con 0x7f9cc40a9450 2023-10-24T23:57:15.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f9cbc001a00 con 0x7f9cc40a9450 2023-10-24T23:57:15.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 >> v1:172.21.15.107:6789/0 conn(0x7f9cc40a9450 legacy=0x7f9cc4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:15.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40aa330 0x7f9cb0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cb0003f00 0x7f9cb00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cb0004d10 0x7f9cb0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc4156100 con 0x7f9cb0003f00 2023-10-24T23:57:15.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc41527c0 con 0x7f9cb0004d10 2023-10-24T23:57:15.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb000ba70 con 0x7f9cc40aa330 2023-10-24T23:57:15.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc9001640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cb0003f00 0x7f9cb00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cca003640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cb0004d10 0x7f9cb0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc9802640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40aa330 0x7f9cb0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc9001640 1 -- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40aa330 msgr2=0x7f9cb0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:15.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc9001640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40aa330 0x7f9cb0003980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc9001640 1 -- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cb0004d10 msgr2=0x7f9cb0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:15.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc9001640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cb0004d10 0x7f9cb0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cc9001640 1 -- 172.21.15.107:0/38739374 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb0003530 con 0x7f9cb0003f00 2023-10-24T23:57:15.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.122+0000 7f9cca003640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cb0004d10 0x7f9cb0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:15.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.126+0000 7f9cc9001640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cb0003f00 0x7f9cb00047d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009ab0 tx=0x7f9cb401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424e9bffe48e20f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:15.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.126+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb40a6500 con 0x7f9cb0003f00 2023-10-24T23:57:15.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.126+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb40a6660 con 0x7f9cb0003f00 2023-10-24T23:57:15.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.126+0000 7f9cc8800640 1 -- 172.21.15.107:0/38739374 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb40a6a10 con 0x7f9cb0003f00 2023-10-24T23:57:15.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.126+0000 7f9ccba66640 1 -- 172.21.15.107:0/38739374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cb0003f00 msgr2=0x7f9cb00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:15.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.126+0000 7f9ccba66640 1 --2- 172.21.15.107:0/38739374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cb0003f00 0x7f9cb00047d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009ab0 tx=0x7f9cb401ca30 comp rx=0 tx=0).stop 2023-10-24T23:57:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.126+0000 7f9cca003640 1 -- 172.21.15.107:0/38739374 reap_dead start 2023-10-24T23:57:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.126+0000 7f9ccba66640 1 -- 172.21.15.107:0/38739374 shutdown_connections 2023-10-24T23:57:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.126+0000 7f9ccba66640 1 -- 172.21.15.107:0/38739374 >> 172.21.15.107:0/38739374 conn(0x7f9cc4070f60 msgr2=0x7f9cc4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.126+0000 7f9ccba66640 1 -- 172.21.15.107:0/38739374 shutdown_connections 2023-10-24T23:57:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.126+0000 7f9ccba66640 1 -- 172.21.15.107:0/38739374 wait complete. 2023-10-24T23:57:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9ccba66640 1 Processor -- start 2023-10-24T23:57:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9ccba66640 1 -- start start 2023-10-24T23:57:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9ccba66640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cb0003f00 0x7f9cc41387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9ccba66640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc40adaa0 0x7f9cc414f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9ccba66640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40ad550 0x7f9cc414fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9ccba66640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc4150c00 con 0x7f9cc40adaa0 2023-10-24T23:57:15.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9ccba66640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc4150080 con 0x7f9cb0003f00 2023-10-24T23:57:15.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9ccba66640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc4136580 con 0x7f9cc40ad550 2023-10-24T23:57:15.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cc9001640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc40adaa0 0x7f9cc414f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cc9001640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc40adaa0 0x7f9cc414f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46900/0 (socket says 172.21.15.107:46900) 2023-10-24T23:57:15.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cc9001640 1 -- 172.21.15.107:0/3520309734 learned_addr learned my addr 172.21.15.107:0/3520309734 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:15.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cc9802640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cb0003f00 0x7f9cc41387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cca003640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40ad550 0x7f9cc414fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cc9001640 1 -- 172.21.15.107:0/3520309734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40ad550 msgr2=0x7f9cc414fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:15.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cc9001640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40ad550 0x7f9cc414fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cc9001640 1 -- 172.21.15.107:0/3520309734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cb0003f00 msgr2=0x7f9cc41387a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:15.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cc9001640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cb0003f00 0x7f9cc41387a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cc9001640 1 -- 172.21.15.107:0/3520309734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc4136860 con 0x7f9cc40adaa0 2023-10-24T23:57:15.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cc9001640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc40adaa0 0x7f9cc414f600 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40b0040 tx=0x7f9cb0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:15.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cba7fc640 1 -- 172.21.15.107:0/3520309734 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb400f0c0 con 0x7f9cc40adaa0 2023-10-24T23:57:15.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cba7fc640 1 -- 172.21.15.107:0/3520309734 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb40a67d0 con 0x7f9cc40adaa0 2023-10-24T23:57:15.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9ccba66640 1 -- 172.21.15.107:0/3520309734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc4136b40 con 0x7f9cc40adaa0 2023-10-24T23:57:15.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9cba7fc640 1 -- 172.21.15.107:0/3520309734 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb40a6b80 con 0x7f9cc40adaa0 2023-10-24T23:57:15.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9ccba66640 1 -- 172.21.15.107:0/3520309734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc4137100 con 0x7f9cc40adaa0 2023-10-24T23:57:15.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.130+0000 7f9ccba66640 1 -- 172.21.15.107:0/3520309734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc41372f0 con 0x7f9cc40adaa0 2023-10-24T23:57:15.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.134+0000 7f9cba7fc640 1 -- 172.21.15.107:0/3520309734 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9cb40a6ce0 con 0x7f9cc40adaa0 2023-10-24T23:57:15.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.134+0000 7f9cba7fc640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ca8065e60 0x7f9ca8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.134+0000 7f9cba7fc640 1 -- 172.21.15.107:0/3520309734 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f9cb40aa070 con 0x7f9cc40adaa0 2023-10-24T23:57:15.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.134+0000 7f9cc9802640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ca8065e60 0x7f9ca8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.134+0000 7f9cc9802640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ca8065e60 0x7f9ca8068320 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0009ab0 tx=0x7f9cc0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:15.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.138+0000 7f9cba7fc640 1 -- 172.21.15.107:0/3520309734 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9cb40eba40 con 0x7f9cc40adaa0 2023-10-24T23:57:15.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.170+0000 7faee053f640 1 Processor -- start 2023-10-24T23:57:15.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.170+0000 7faee053f640 1 -- start start 2023-10-24T23:57:15.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faee053f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faed8152060 0x7faed80aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faee053f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faed80c20b0 0x7faed80c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faee053f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed80a9aa0 con 0x7faed80c2990 2023-10-24T23:57:15.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faee053f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed8155480 con 0x7faed80c20b0 2023-10-24T23:57:15.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faee053f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed8155760 con 0x7faed8152060 2023-10-24T23:57:15.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faedeadc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faed80c20b0 0x7faed80c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faedeadc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faed80c20b0 0x7faed80c1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46910/0 (socket says 172.21.15.107:46910) 2023-10-24T23:57:15.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faedeadc640 1 -- 172.21.15.107:0/786985619 learned_addr learned my addr 172.21.15.107:0/786985619 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:15.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faede2db640 1 --2- 172.21.15.107:0/786985619 >> v2:172.21.15.110:3300/0 conn(0x7faed8152060 0x7faed80aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faedeadc640 1 -- 172.21.15.107:0/786985619 >> v2:172.21.15.110:3300/0 conn(0x7faed8152060 msgr2=0x7faed80aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:15.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faedeadc640 1 --2- 172.21.15.107:0/786985619 >> v2:172.21.15.110:3300/0 conn(0x7faed8152060 0x7faed80aada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faedeadc640 1 -- 172.21.15.107:0/786985619 >> v1:172.21.15.110:6789/0 conn(0x7faed80c2990 legacy=0x7faed80a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:15.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faedeadc640 1 -- 172.21.15.107:0/786985619 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed80a9450 con 0x7faed80c20b0 2023-10-24T23:57:15.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faedeadc640 1 --2- 172.21.15.107:0/786985619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80c20b0 0x7faed80c1480 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7faed8073d70 tx=0x7faed001c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee256a7cfc52858c server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:15.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faedd2d9640 1 -- 172.21.15.107:0/786985619 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed00a56c0 con 0x7faed80c20b0 2023-10-24T23:57:15.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faedd2d9640 1 -- 172.21.15.107:0/786985619 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed00a5820 con 0x7faed80c20b0 2023-10-24T23:57:15.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faedd2d9640 1 -- 172.21.15.107:0/786985619 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed00a5bb0 con 0x7faed80c20b0 2023-10-24T23:57:15.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faee053f640 1 -- 172.21.15.107:0/786985619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80c20b0 msgr2=0x7faed80c1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:15.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.174+0000 7faee053f640 1 --2- 172.21.15.107:0/786985619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80c20b0 0x7faed80c1480 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7faed8073d70 tx=0x7faed001c840 comp rx=0 tx=0).stop 2023-10-24T23:57:15.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 -- 172.21.15.107:0/786985619 shutdown_connections 2023-10-24T23:57:15.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 --2- 172.21.15.107:0/786985619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80c20b0 0x7faed80c1480 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 --2- 172.21.15.107:0/786985619 >> v2:172.21.15.110:3300/0 conn(0x7faed8152060 0x7faed80aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 -- 172.21.15.107:0/786985619 >> 172.21.15.107:0/786985619 conn(0x7faed80c4a90 msgr2=0x7faed80c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:15.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 -- 172.21.15.107:0/786985619 shutdown_connections 2023-10-24T23:57:15.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 -- 172.21.15.107:0/786985619 wait complete. 2023-10-24T23:57:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 Processor -- start 2023-10-24T23:57:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 -- start start 2023-10-24T23:57:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80c20b0 0x7faed80c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faed80c2990 0x7faed81f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed8152060 0x7faed81f0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed80c7dc0 con 0x7faed80c20b0 2023-10-24T23:57:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed80c6810 con 0x7faed8152060 2023-10-24T23:57:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed80c6af0 con 0x7faed80c2990 2023-10-24T23:57:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faede2db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80c20b0 0x7faed80c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faede2db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80c20b0 0x7faed80c7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46924/0 (socket says 172.21.15.107:46924) 2023-10-24T23:57:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faede2db640 1 -- 172.21.15.107:0/1971637839 learned_addr learned my addr 172.21.15.107:0/1971637839 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faeddada640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faed80c2990 0x7faed81f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faedeadc640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed8152060 0x7faed81f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faede2db640 1 -- 172.21.15.107:0/1971637839 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faed80c2990 msgr2=0x7faed81f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faede2db640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faed80c2990 0x7faed81f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faede2db640 1 -- 172.21.15.107:0/1971637839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed8152060 msgr2=0x7faed81f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:15.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faede2db640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed8152060 0x7faed81f0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faede2db640 1 -- 172.21.15.107:0/1971637839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed0009b30 con 0x7faed80c20b0 2023-10-24T23:57:15.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faede2db640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80c20b0 0x7faed80c7880 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7faed400d800 tx=0x7faed400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:15.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faeceffd640 1 -- 172.21.15.107:0/1971637839 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed400b6c0 con 0x7faed80c20b0 2023-10-24T23:57:15.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faeceffd640 1 -- 172.21.15.107:0/1971637839 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed400b820 con 0x7faed80c20b0 2023-10-24T23:57:15.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faeceffd640 1 -- 172.21.15.107:0/1971637839 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed400bb90 con 0x7faed80c20b0 2023-10-24T23:57:15.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 -- 172.21.15.107:0/1971637839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed80c6dd0 con 0x7faed80c20b0 2023-10-24T23:57:15.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.178+0000 7faee053f640 1 -- 172.21.15.107:0/1971637839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed8151140 con 0x7faed80c20b0 2023-10-24T23:57:15.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.182+0000 7faee053f640 1 -- 172.21.15.107:0/1971637839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed80746e0 con 0x7faed80c20b0 2023-10-24T23:57:15.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.182+0000 7faeceffd640 1 -- 172.21.15.107:0/1971637839 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7faed400bcf0 con 0x7faed80c20b0 2023-10-24T23:57:15.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.182+0000 7faeceffd640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faebc065e90 0x7faebc068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:15.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.182+0000 7faeceffd640 1 -- 172.21.15.107:0/1971637839 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7faed4084140 con 0x7faed80c20b0 2023-10-24T23:57:15.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.182+0000 7faeddada640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faebc065e90 0x7faebc068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:15.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.182+0000 7faeddada640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faebc065e90 0x7faebc068350 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7faed8154340 tx=0x7faec8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:15.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.186+0000 7faeceffd640 1 -- 172.21.15.107:0/1971637839 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7faed4051340 con 0x7faed80c20b0 2023-10-24T23:57:15.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.334+0000 7f9ccba66640 1 -- 172.21.15.107:0/3520309734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cc40746e0 con 0x7f9cc40adaa0 2023-10-24T23:57:15.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.334+0000 7f9cba7fc640 1 -- 172.21.15.107:0/3520309734 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f9cb40ef060 con 0x7f9cc40adaa0 2023-10-24T23:57:15.340 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:15.340 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:15.341 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:15.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 -- 172.21.15.107:0/3520309734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ca8065e60 msgr2=0x7f9ca8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:15.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ca8065e60 0x7f9ca8068320 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0009ab0 tx=0x7f9cc0008040 comp rx=0 tx=0).stop 2023-10-24T23:57:15.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 -- 172.21.15.107:0/3520309734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc40adaa0 msgr2=0x7f9cc414f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:15.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc40adaa0 0x7f9cc414f600 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40b0040 tx=0x7f9cb0005800 comp rx=0 tx=0).stop 2023-10-24T23:57:15.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 -- 172.21.15.107:0/3520309734 shutdown_connections 2023-10-24T23:57:15.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ca8065e60 0x7f9ca8068320 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9cc40ad550 0x7f9cc414fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cc40adaa0 0x7f9cc414f600 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 --2- 172.21.15.107:0/3520309734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cb0003f00 0x7f9cc41387a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 -- 172.21.15.107:0/3520309734 >> 172.21.15.107:0/3520309734 conn(0x7f9cc4070f60 msgr2=0x7f9cc4158880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:15.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 -- 172.21.15.107:0/3520309734 shutdown_connections 2023-10-24T23:57:15.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.338+0000 7f9ccba66640 1 -- 172.21.15.107:0/3520309734 wait complete. 2023-10-24T23:57:15.361 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:57:15.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 -- 172.21.15.107:0/1971637839 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faed80c1e90 con 0x7faed80c20b0 2023-10-24T23:57:15.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faeceffd640 1 -- 172.21.15.107:0/1971637839 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7faed4054960 con 0x7faed80c20b0 2023-10-24T23:57:15.407 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:15.407 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:15.408 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 -- 172.21.15.107:0/1971637839 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faebc065e90 msgr2=0x7faebc068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faebc065e90 0x7faebc068350 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7faed8154340 tx=0x7faec8009040 comp rx=0 tx=0).stop 2023-10-24T23:57:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 -- 172.21.15.107:0/1971637839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80c20b0 msgr2=0x7faed80c7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80c20b0 0x7faed80c7880 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7faed400d800 tx=0x7faed400dcc0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 -- 172.21.15.107:0/1971637839 shutdown_connections 2023-10-24T23:57:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed8152060 0x7faed81f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faebc065e90 0x7faebc068350 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faed80c2990 0x7faed81f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 --2- 172.21.15.107:0/1971637839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faed80c20b0 0x7faed80c7880 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 -- 172.21.15.107:0/1971637839 >> 172.21.15.107:0/1971637839 conn(0x7faed80c4a90 msgr2=0x7faed80a9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 -- 172.21.15.107:0/1971637839 shutdown_connections 2023-10-24T23:57:15.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:15.402+0000 7faee053f640 1 -- 172.21.15.107:0/1971637839 wait complete. 2023-10-24T23:57:15.424 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:57:17.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:17.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:17.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f58374a9640 1 Processor -- start 2023-10-24T23:57:17.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f58374a9640 1 -- start start 2023-10-24T23:57:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f58374a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f583014eb70 0x7f58300c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f58374a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58300c22f0 con 0x7f583006df30 2023-10-24T23:57:17.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f58374a9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58300c25d0 con 0x7f5830070b00 2023-10-24T23:57:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f58374a9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5830071700 con 0x7f583014eb70 2023-10-24T23:57:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f5835245640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f583014eb70 0x7f58300c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:17.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f5835245640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f583014eb70 0x7f58300c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46928/0 (socket says 172.21.15.107:46928) 2023-10-24T23:57:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f5835a46640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f583006df30 0x7f583015be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52070/0 (socket says 172.21.15.107:52070) 2023-10-24T23:57:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f5835245640 1 -- 172.21.15.107:0/2860575836 learned_addr learned my addr 172.21.15.107:0/2860575836 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:17.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f5835245640 1 -- 172.21.15.107:0/2860575836 >> v1:172.21.15.110:6789/0 conn(0x7f5830070b00 legacy=0x7f58301514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f5835245640 1 -- 172.21.15.107:0/2860575836 >> v1:172.21.15.107:6789/0 conn(0x7f583006df30 legacy=0x7f583015be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.486+0000 7f5835245640 1 -- 172.21.15.107:0/2860575836 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58300c2be0 con 0x7f583014eb70 2023-10-24T23:57:17.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.490+0000 7f5835245640 1 --2- 172.21.15.107:0/2860575836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f583014eb70 0x7f58300c33e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f582400bb10 tx=0x7f582401ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b3623ecf21b2bc server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:17.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.490+0000 7f5823fff640 1 -- 172.21.15.107:0/2860575836 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58240a7ab0 con 0x7f583014eb70 2023-10-24T23:57:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.490+0000 7f5823fff640 1 -- 172.21.15.107:0/2860575836 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58240a7c10 con 0x7f583014eb70 2023-10-24T23:57:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.490+0000 7f5823fff640 1 -- 172.21.15.107:0/2860575836 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58240af440 con 0x7f583014eb70 2023-10-24T23:57:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.490+0000 7f58374a9640 1 -- 172.21.15.107:0/2860575836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f583014eb70 msgr2=0x7f58300c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.490+0000 7f58374a9640 1 --2- 172.21.15.107:0/2860575836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f583014eb70 0x7f58300c33e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f582400bb10 tx=0x7f582401ebd0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.490+0000 7f58374a9640 1 -- 172.21.15.107:0/2860575836 shutdown_connections 2023-10-24T23:57:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.490+0000 7f58374a9640 1 --2- 172.21.15.107:0/2860575836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f583014eb70 0x7f58300c33e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.490+0000 7f58374a9640 1 -- 172.21.15.107:0/2860575836 >> 172.21.15.107:0/2860575836 conn(0x7f58300c39a0 msgr2=0x7f58300c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.490+0000 7f58374a9640 1 -- 172.21.15.107:0/2860575836 shutdown_connections 2023-10-24T23:57:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.490+0000 7f58374a9640 1 -- 172.21.15.107:0/2860575836 wait complete. 2023-10-24T23:57:17.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f58374a9640 1 Processor -- start 2023-10-24T23:57:17.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f58374a9640 1 -- start start 2023-10-24T23:57:17.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f58374a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f583006df30 0x7f5830147ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:17.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f58374a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5830070b00 0x7f583013b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:17.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f58374a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f583014eb70 0x7f583013bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:17.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f58374a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5830146600 con 0x7f5830070b00 2023-10-24T23:57:17.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f58374a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583013c110 con 0x7f583006df30 2023-10-24T23:57:17.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f58374a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58301471b0 con 0x7f583014eb70 2023-10-24T23:57:17.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5834a44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5830070b00 0x7f583013b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:17.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5834a44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5830070b00 0x7f583013b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46940/0 (socket says 172.21.15.107:46940) 2023-10-24T23:57:17.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5834a44640 1 -- 172.21.15.107:0/288667704 learned_addr learned my addr 172.21.15.107:0/288667704 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:17.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5835a46640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f583014eb70 0x7f583013bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:17.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5834a44640 1 -- 172.21.15.107:0/288667704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f583014eb70 msgr2=0x7f583013bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5835245640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f583006df30 0x7f5830147ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:17.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5834a44640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f583014eb70 0x7f583013bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5834a44640 1 -- 172.21.15.107:0/288667704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f583006df30 msgr2=0x7f5830147ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5834a44640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f583006df30 0x7f5830147ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5834a44640 1 -- 172.21.15.107:0/288667704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582400b770 con 0x7f5830070b00 2023-10-24T23:57:17.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5834a44640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5830070b00 0x7f583013b690 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f582c00c440 tx=0x7f582c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:17.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5821ffb640 1 -- 172.21.15.107:0/288667704 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c012830 con 0x7f5830070b00 2023-10-24T23:57:17.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5821ffb640 1 -- 172.21.15.107:0/288667704 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c012990 con 0x7f5830070b00 2023-10-24T23:57:17.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f58374a9640 1 -- 172.21.15.107:0/288667704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5830145810 con 0x7f5830070b00 2023-10-24T23:57:17.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f5821ffb640 1 -- 172.21.15.107:0/288667704 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c012d40 con 0x7f5830070b00 2023-10-24T23:57:17.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f58374a9640 1 -- 172.21.15.107:0/288667704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5830147400 con 0x7f5830070b00 2023-10-24T23:57:17.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.494+0000 7f58374a9640 1 -- 172.21.15.107:0/288667704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58300746e0 con 0x7f5830070b00 2023-10-24T23:57:17.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.498+0000 7f5821ffb640 1 -- 172.21.15.107:0/288667704 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f582c01b400 con 0x7f5830070b00 2023-10-24T23:57:17.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.498+0000 7f5821ffb640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5814065e60 0x7f5814068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:17.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.498+0000 7f5821ffb640 1 -- 172.21.15.107:0/288667704 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f582c083c00 con 0x7f5830070b00 2023-10-24T23:57:17.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.498+0000 7f5835245640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5814065e60 0x7f5814068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:17.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.498+0000 7f5835245640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5814065e60 0x7f5814068320 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f58240b1000 tx=0x7f5824008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:17.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.502+0000 7f5821ffb640 1 -- 172.21.15.107:0/288667704 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f582c08a080 con 0x7f5830070b00 2023-10-24T23:57:17.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c7b0c640 1 Processor -- start 2023-10-24T23:57:17.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c7b0c640 1 -- start start 2023-10-24T23:57:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c7b0c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03c00c4e20 0x7f03c00c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c7b0c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f03c00c5ba0 0x7f03c00c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:17.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c7b0c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c0155f20 con 0x7f03c014fbd0 2023-10-24T23:57:17.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c7b0c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c00c6d40 con 0x7f03c00c4e20 2023-10-24T23:57:17.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c7b0c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c00c2320 con 0x7f03c00c5ba0 2023-10-24T23:57:17.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c50a7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03c00c4e20 0x7f03c00c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:17.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c58a8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f03c014fbd0 0x7f03c006df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52084/0 (socket says 172.21.15.107:52084) 2023-10-24T23:57:17.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c58a8640 1 -- 172.21.15.107:0/3149457090 learned_addr learned my addr 172.21.15.107:0/3149457090 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:17.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c60a9640 1 --2- 172.21.15.107:0/3149457090 >> v2:172.21.15.110:3300/0 conn(0x7f03c00c5ba0 0x7f03c00c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:17.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c50a7640 1 -- 172.21.15.107:0/3149457090 >> v2:172.21.15.110:3300/0 conn(0x7f03c00c5ba0 msgr2=0x7f03c00c1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:17.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c50a7640 1 --2- 172.21.15.107:0/3149457090 >> v2:172.21.15.110:3300/0 conn(0x7f03c00c5ba0 0x7f03c00c1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c50a7640 1 -- 172.21.15.107:0/3149457090 >> v1:172.21.15.107:6789/0 conn(0x7f03c014fbd0 legacy=0x7f03c006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c50a7640 1 -- 172.21.15.107:0/3149457090 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c0153560 con 0x7f03c00c4e20 2023-10-24T23:57:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c48a6640 1 -- 172.21.15.107:0/3149457090 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180153827 0 0) 0x7f03c0155f20 con 0x7f03c014fbd0 2023-10-24T23:57:17.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.550+0000 7f03c60a9640 1 --2- 172.21.15.107:0/3149457090 >> v2:172.21.15.110:3300/0 conn(0x7f03c00c5ba0 0x7f03c00c1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:17.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c50a7640 1 --2- 172.21.15.107:0/3149457090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c00c4e20 0x7f03c00c18a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f03c00c4990 tx=0x7f03b001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb2c6901d4cd987 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:17.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c48a6640 1 -- 172.21.15.107:0/3149457090 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b00ab600 con 0x7f03c00c4e20 2023-10-24T23:57:17.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c48a6640 1 -- 172.21.15.107:0/3149457090 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b00a7050 con 0x7f03c00c4e20 2023-10-24T23:57:17.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c48a6640 1 -- 172.21.15.107:0/3149457090 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b00ab980 con 0x7f03c00c4e20 2023-10-24T23:57:17.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c7b0c640 1 -- 172.21.15.107:0/3149457090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c00c4e20 msgr2=0x7f03c00c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c7b0c640 1 --2- 172.21.15.107:0/3149457090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c00c4e20 0x7f03c00c18a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f03c00c4990 tx=0x7f03b001c3d0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c7b0c640 1 -- 172.21.15.107:0/3149457090 shutdown_connections 2023-10-24T23:57:17.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c7b0c640 1 --2- 172.21.15.107:0/3149457090 >> v2:172.21.15.110:3300/0 conn(0x7f03c00c5ba0 0x7f03c00c1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c7b0c640 1 --2- 172.21.15.107:0/3149457090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c00c4e20 0x7f03c00c18a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c7b0c640 1 -- 172.21.15.107:0/3149457090 >> 172.21.15.107:0/3149457090 conn(0x7f03c00c39a0 msgr2=0x7f03c00c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:17.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c7b0c640 1 -- 172.21.15.107:0/3149457090 shutdown_connections 2023-10-24T23:57:17.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.554+0000 7f03c7b0c640 1 -- 172.21.15.107:0/3149457090 wait complete. 2023-10-24T23:57:17.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c7b0c640 1 Processor -- start 2023-10-24T23:57:17.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c7b0c640 1 -- start start 2023-10-24T23:57:17.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c7b0c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c00c4e20 0x7f03c014db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:17.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c7b0c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03c00c5ba0 0x7f03c006c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:17.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c7b0c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c014fbd0 0x7f03c014c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:17.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c7b0c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c01ed380 con 0x7f03c00c4e20 2023-10-24T23:57:17.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c7b0c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c01f0d30 con 0x7f03c014fbd0 2023-10-24T23:57:17.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c7b0c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c01ed810 con 0x7f03c00c5ba0 2023-10-24T23:57:17.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c58a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c00c4e20 0x7f03c014db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:17.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c58a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c00c4e20 0x7f03c014db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46962/0 (socket says 172.21.15.107:46962) 2023-10-24T23:57:17.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c58a8640 1 -- 172.21.15.107:0/2017871211 learned_addr learned my addr 172.21.15.107:0/2017871211 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:17.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c60a9640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c014fbd0 0x7f03c014c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:17.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c50a7640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03c00c5ba0 0x7f03c006c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:17.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c58a8640 1 -- 172.21.15.107:0/2017871211 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03c00c5ba0 msgr2=0x7f03c006c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c58a8640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03c00c5ba0 0x7f03c006c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c58a8640 1 -- 172.21.15.107:0/2017871211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c014fbd0 msgr2=0x7f03c014c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c58a8640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c014fbd0 0x7f03c014c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c58a8640 1 -- 172.21.15.107:0/2017871211 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b0009420 con 0x7f03c00c4e20 2023-10-24T23:57:17.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c60a9640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c014fbd0 0x7f03c014c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:17.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c58a8640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c00c4e20 0x7f03c014db10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c480 tx=0x7f03bc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:17.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03b67fc640 1 -- 172.21.15.107:0/2017871211 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0074e0 con 0x7f03c00c4e20 2023-10-24T23:57:17.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03b67fc640 1 -- 172.21.15.107:0/2017871211 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc007660 con 0x7f03c00c4e20 2023-10-24T23:57:17.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c7b0c640 1 -- 172.21.15.107:0/2017871211 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c01eca50 con 0x7f03c00c4e20 2023-10-24T23:57:17.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03b67fc640 1 -- 172.21.15.107:0/2017871211 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc004a30 con 0x7f03c00c4e20 2023-10-24T23:57:17.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c7b0c640 1 -- 172.21.15.107:0/2017871211 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c01ed010 con 0x7f03c00c4e20 2023-10-24T23:57:17.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.558+0000 7f03c7b0c640 1 -- 172.21.15.107:0/2017871211 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c00746e0 con 0x7f03c00c4e20 2023-10-24T23:57:17.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.562+0000 7f03b67fc640 1 -- 172.21.15.107:0/2017871211 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f03bc004b90 con 0x7f03c00c4e20 2023-10-24T23:57:17.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.562+0000 7f03b67fc640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03a4065e60 0x7f03a4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:17.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.562+0000 7f03b67fc640 1 -- 172.21.15.107:0/2017871211 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f03bc082e50 con 0x7f03c00c4e20 2023-10-24T23:57:17.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.562+0000 7f03c50a7640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03a4065e60 0x7f03a4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:17.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.562+0000 7f03c50a7640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03a4065e60 0x7f03a4068320 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f03b00ad000 tx=0x7f03b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:17.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.566+0000 7f03b67fc640 1 -- 172.21.15.107:0/2017871211 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f03bc053ae0 con 0x7f03c00c4e20 2023-10-24T23:57:17.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.694+0000 7f58374a9640 1 -- 172.21.15.107:0/288667704 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f583015d360 con 0x7f5830070b00 2023-10-24T23:57:17.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.694+0000 7f5821ffb640 1 -- 172.21.15.107:0/288667704 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f582c016090 con 0x7f5830070b00 2023-10-24T23:57:17.699 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:17.699 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:17.700 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:17.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.694+0000 7f58374a9640 1 -- 172.21.15.107:0/288667704 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5814065e60 msgr2=0x7f5814068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.694+0000 7f58374a9640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5814065e60 0x7f5814068320 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f58240b1000 tx=0x7f5824008040 comp rx=0 tx=0).stop 2023-10-24T23:57:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.694+0000 7f58374a9640 1 -- 172.21.15.107:0/288667704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5830070b00 msgr2=0x7f583013b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.694+0000 7f58374a9640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5830070b00 0x7f583013b690 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f582c00c440 tx=0x7f582c00c900 comp rx=0 tx=0).stop 2023-10-24T23:57:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.698+0000 7f58374a9640 1 -- 172.21.15.107:0/288667704 shutdown_connections 2023-10-24T23:57:17.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.698+0000 7f58374a9640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5814065e60 0x7f5814068320 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.698+0000 7f58374a9640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f583014eb70 0x7f583013bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.698+0000 7f58374a9640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5830070b00 0x7f583013b690 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.698+0000 7f58374a9640 1 --2- 172.21.15.107:0/288667704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f583006df30 0x7f5830147ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.698+0000 7f58374a9640 1 -- 172.21.15.107:0/288667704 >> 172.21.15.107:0/288667704 conn(0x7f58300c39a0 msgr2=0x7f58300c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:17.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.698+0000 7f58374a9640 1 -- 172.21.15.107:0/288667704 shutdown_connections 2023-10-24T23:57:17.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.698+0000 7f58374a9640 1 -- 172.21.15.107:0/288667704 wait complete. 2023-10-24T23:57:17.718 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:57:17.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.754+0000 7f03c7b0c640 1 -- 172.21.15.107:0/2017871211 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03c015cd60 con 0x7f03c00c4e20 2023-10-24T23:57:17.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.754+0000 7f03b67fc640 1 -- 172.21.15.107:0/2017871211 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f03bc018020 con 0x7f03c00c4e20 2023-10-24T23:57:17.761 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:17.761 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:17.763 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:17.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 -- 172.21.15.107:0/2017871211 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03a4065e60 msgr2=0x7f03a4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03a4065e60 0x7f03a4068320 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f03b00ad000 tx=0x7f03b0008040 comp rx=0 tx=0).stop 2023-10-24T23:57:17.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 -- 172.21.15.107:0/2017871211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c00c4e20 msgr2=0x7f03c014db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:17.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c00c4e20 0x7f03c014db10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00c480 tx=0x7f03bc00c940 comp rx=0 tx=0).stop 2023-10-24T23:57:17.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 -- 172.21.15.107:0/2017871211 shutdown_connections 2023-10-24T23:57:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03c014fbd0 0x7f03c014c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03a4065e60 0x7f03a4068320 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03c00c5ba0 0x7f03c006c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 --2- 172.21.15.107:0/2017871211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03c00c4e20 0x7f03c014db10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 -- 172.21.15.107:0/2017871211 >> 172.21.15.107:0/2017871211 conn(0x7f03c00c39a0 msgr2=0x7f03c00c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 -- 172.21.15.107:0/2017871211 shutdown_connections 2023-10-24T23:57:17.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:17.758+0000 7f03c7b0c640 1 -- 172.21.15.107:0/2017871211 wait complete. 2023-10-24T23:57:17.779 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:57:19.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:19.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:19.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee78753640 1 Processor -- start 2023-10-24T23:57:19.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee78753640 1 -- start start 2023-10-24T23:57:19.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee78753640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee700c70e0 0x7fee700bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee78753640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee700c2130 0x7fee700be4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee78753640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee700c1ca0 con 0x7fee70156770 2023-10-24T23:57:19.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee78753640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee700c27e0 con 0x7fee700c70e0 2023-10-24T23:57:19.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee78753640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee700c10c0 con 0x7fee700c2130 2023-10-24T23:57:19.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee75cee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee700c70e0 0x7fee700bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee75cee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee700c70e0 0x7fee700bdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46968/0 (socket says 172.21.15.107:46968) 2023-10-24T23:57:19.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee75cee640 1 -- 172.21.15.107:0/1832567191 learned_addr learned my addr 172.21.15.107:0/1832567191 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:19.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee76cf0640 1 --2- 172.21.15.107:0/1832567191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee700c2130 0x7fee700be4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee75cee640 1 -- 172.21.15.107:0/1832567191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee700c2130 msgr2=0x7fee700be4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:19.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee75cee640 1 --2- 172.21.15.107:0/1832567191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee700c2130 0x7fee700be4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee75cee640 1 -- 172.21.15.107:0/1832567191 >> v1:172.21.15.107:6789/0 conn(0x7fee70156770 legacy=0x7fee700ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:19.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee754ed640 1 -- 172.21.15.107:0/1832567191 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458204888 0 0) 0x7fee700c1ca0 con 0x7fee70156770 2023-10-24T23:57:19.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.846+0000 7fee75cee640 1 -- 172.21.15.107:0/1832567191 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee700c13a0 con 0x7fee700c70e0 2023-10-24T23:57:19.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee75cee640 1 --2- 172.21.15.107:0/1832567191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee700c70e0 0x7fee700bdfa0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fee64009870 tx=0x7fee6401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6694c971ca5378a server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:19.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee754ed640 1 -- 172.21.15.107:0/1832567191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee640ab600 con 0x7fee700c70e0 2023-10-24T23:57:19.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee754ed640 1 -- 172.21.15.107:0/1832567191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee640a7050 con 0x7fee700c70e0 2023-10-24T23:57:19.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee754ed640 1 -- 172.21.15.107:0/1832567191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee640ab980 con 0x7fee700c70e0 2023-10-24T23:57:19.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 -- 172.21.15.107:0/1832567191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee700c70e0 msgr2=0x7fee700bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:19.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 --2- 172.21.15.107:0/1832567191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee700c70e0 0x7fee700bdfa0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fee64009870 tx=0x7fee6401c420 comp rx=0 tx=0).stop 2023-10-24T23:57:19.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 -- 172.21.15.107:0/1832567191 shutdown_connections 2023-10-24T23:57:19.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 --2- 172.21.15.107:0/1832567191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee700c2130 0x7fee700be4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 --2- 172.21.15.107:0/1832567191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee700c70e0 0x7fee700bdfa0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 -- 172.21.15.107:0/1832567191 >> 172.21.15.107:0/1832567191 conn(0x7fee700c65c0 msgr2=0x7fee700c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:19.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 -- 172.21.15.107:0/1832567191 shutdown_connections 2023-10-24T23:57:19.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 -- 172.21.15.107:0/1832567191 wait complete. 2023-10-24T23:57:19.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 Processor -- start 2023-10-24T23:57:19.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 -- start start 2023-10-24T23:57:19.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee700c2130 0x7fee701537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee700c70e0 0x7fee7006fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee70156770 0x7fee70071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee700709d0 con 0x7fee70156770 2023-10-24T23:57:19.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee701ebb70 con 0x7fee700c70e0 2023-10-24T23:57:19.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee78753640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee701eaf80 con 0x7fee700c2130 2023-10-24T23:57:19.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee76cf0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee70156770 0x7fee70071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee76cf0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee70156770 0x7fee70071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46970/0 (socket says 172.21.15.107:46970) 2023-10-24T23:57:19.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.850+0000 7fee76cf0640 1 -- 172.21.15.107:0/571166373 learned_addr learned my addr 172.21.15.107:0/571166373 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:19.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee764ef640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee700c2130 0x7fee701537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee75cee640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee700c70e0 0x7fee7006fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee76cf0640 1 -- 172.21.15.107:0/571166373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee700c2130 msgr2=0x7fee701537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:19.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee76cf0640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee700c2130 0x7fee701537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee76cf0640 1 -- 172.21.15.107:0/571166373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee700c70e0 msgr2=0x7fee7006fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:19.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee76cf0640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee700c70e0 0x7fee7006fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee76cf0640 1 -- 172.21.15.107:0/571166373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee640094d0 con 0x7fee70156770 2023-10-24T23:57:19.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee75cee640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee700c70e0 0x7fee7006fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:19.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee76cf0640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee70156770 0x7fee70071ce0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fee60007e40 tx=0x7fee6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:19.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee6effd640 1 -- 172.21.15.107:0/571166373 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee60014620 con 0x7fee70156770 2023-10-24T23:57:19.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee6effd640 1 -- 172.21.15.107:0/571166373 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6000ee90 con 0x7fee70156770 2023-10-24T23:57:19.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee6effd640 1 -- 172.21.15.107:0/571166373 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee600149e0 con 0x7fee70156770 2023-10-24T23:57:19.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee78753640 1 -- 172.21.15.107:0/571166373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee701eb240 con 0x7fee70156770 2023-10-24T23:57:19.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee78753640 1 -- 172.21.15.107:0/571166373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee701eb800 con 0x7fee70156770 2023-10-24T23:57:19.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee78753640 1 -- 172.21.15.107:0/571166373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee701ebce0 con 0x7fee70156770 2023-10-24T23:57:19.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee6effd640 1 -- 172.21.15.107:0/571166373 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fee60014b40 con 0x7fee70156770 2023-10-24T23:57:19.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee6effd640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee5c065e90 0x7fee5c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee6effd640 1 -- 172.21.15.107:0/571166373 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7fee60084640 con 0x7fee70156770 2023-10-24T23:57:19.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee764ef640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee5c065e90 0x7fee5c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.854+0000 7fee764ef640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee5c065e90 0x7fee5c068350 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fee68000f70 tx=0x7fee68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:19.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.858+0000 7fee6effd640 1 -- 172.21.15.107:0/571166373 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fee60051840 con 0x7fee70156770 2023-10-24T23:57:19.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7bc02640 1 Processor -- start 2023-10-24T23:57:19.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7bc02640 1 -- start start 2023-10-24T23:57:19.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7bc02640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcd74069d40 0x7fcd74158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7bc02640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcd74065bc0 0x7fcd7406a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7bc02640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd740a94b0 0x7fcd740aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7bc02640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd740ac650 con 0x7fcd74065bc0 2023-10-24T23:57:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7bc02640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd740a9e00 con 0x7fcd74069d40 2023-10-24T23:57:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7bc02640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7406cea0 con 0x7fcd740a94b0 2023-10-24T23:57:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7919d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcd74065bc0 0x7fcd7406a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7919d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcd74065bc0 0x7fcd7406a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46984/0 (socket says 172.21.15.107:46984) 2023-10-24T23:57:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7919d640 1 -- 172.21.15.107:0/4281569003 learned_addr learned my addr 172.21.15.107:0/4281569003 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7a19f640 1 --2- 172.21.15.107:0/4281569003 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd740a94b0 0x7fcd740aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7919d640 1 -- 172.21.15.107:0/4281569003 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd740a94b0 msgr2=0x7fcd740aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7919d640 1 --2- 172.21.15.107:0/4281569003 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd740a94b0 0x7fcd740aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7919d640 1 -- 172.21.15.107:0/4281569003 >> v2:172.21.15.110:3300/0 conn(0x7fcd74069d40 msgr2=0x7fcd74158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7999e640 1 --2- 172.21.15.107:0/4281569003 >> v2:172.21.15.110:3300/0 conn(0x7fcd74069d40 0x7fcd74158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7919d640 1 --2- 172.21.15.107:0/4281569003 >> v2:172.21.15.110:3300/0 conn(0x7fcd74069d40 0x7fcd74158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7919d640 1 -- 172.21.15.107:0/4281569003 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd741584e0 con 0x7fcd74065bc0 2023-10-24T23:57:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.898+0000 7fcd7a19f640 1 --2- 172.21.15.107:0/4281569003 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd740a94b0 0x7fcd740aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:19.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7919d640 1 --2- 172.21.15.107:0/4281569003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd74065bc0 0x7fcd7406a120 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fcd7406e070 tx=0x7fcd6401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a9a103b0711339 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:19.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7899c640 1 -- 172.21.15.107:0/4281569003 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd640ab600 con 0x7fcd74065bc0 2023-10-24T23:57:19.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7899c640 1 -- 172.21.15.107:0/4281569003 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd640a7050 con 0x7fcd74065bc0 2023-10-24T23:57:19.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7899c640 1 -- 172.21.15.107:0/4281569003 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd640ab980 con 0x7fcd74065bc0 2023-10-24T23:57:19.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7bc02640 1 -- 172.21.15.107:0/4281569003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd74065bc0 msgr2=0x7fcd7406a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:19.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7bc02640 1 --2- 172.21.15.107:0/4281569003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd74065bc0 0x7fcd7406a120 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fcd7406e070 tx=0x7fcd6401c470 comp rx=0 tx=0).stop 2023-10-24T23:57:19.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7bc02640 1 -- 172.21.15.107:0/4281569003 shutdown_connections 2023-10-24T23:57:19.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7bc02640 1 --2- 172.21.15.107:0/4281569003 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd740a94b0 0x7fcd740aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7bc02640 1 --2- 172.21.15.107:0/4281569003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd74065bc0 0x7fcd7406a120 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7bc02640 1 --2- 172.21.15.107:0/4281569003 >> v2:172.21.15.110:3300/0 conn(0x7fcd74069d40 0x7fcd74158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7bc02640 1 -- 172.21.15.107:0/4281569003 >> 172.21.15.107:0/4281569003 conn(0x7fcd740c65c0 msgr2=0x7fcd740c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:19.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7bc02640 1 -- 172.21.15.107:0/4281569003 shutdown_connections 2023-10-24T23:57:19.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7bc02640 1 -- 172.21.15.107:0/4281569003 wait complete. 2023-10-24T23:57:19.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7bc02640 1 Processor -- start 2023-10-24T23:57:19.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.902+0000 7fcd7bc02640 1 -- start start 2023-10-24T23:57:19.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7bc02640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd74065bc0 0x7fcd74071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7bc02640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd74069d40 0x7fcd740bca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7bc02640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd740a94b0 0x7fcd740bd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7bc02640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd74072680 con 0x7fcd740a94b0 2023-10-24T23:57:19.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7bc02640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd74071150 con 0x7fcd74069d40 2023-10-24T23:57:19.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7bc02640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd74157c10 con 0x7fcd74065bc0 2023-10-24T23:57:19.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7a19f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd740a94b0 0x7fcd740bd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7a19f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd740a94b0 0x7fcd740bd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46996/0 (socket says 172.21.15.107:46996) 2023-10-24T23:57:19.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7a19f640 1 -- 172.21.15.107:0/2125280603 learned_addr learned my addr 172.21.15.107:0/2125280603 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:19.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7999e640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd74065bc0 0x7fcd74071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7a19f640 1 -- 172.21.15.107:0/2125280603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd74065bc0 msgr2=0x7fcd74071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:19.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7919d640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd74069d40 0x7fcd740bca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7a19f640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd74065bc0 0x7fcd74071910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7a19f640 1 -- 172.21.15.107:0/2125280603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd74069d40 msgr2=0x7fcd740bca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:19.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7a19f640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd74069d40 0x7fcd740bca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:19.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7a19f640 1 -- 172.21.15.107:0/2125280603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd640094d0 con 0x7fcd740a94b0 2023-10-24T23:57:19.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7999e640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd74065bc0 0x7fcd74071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:57:19.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7a19f640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd740a94b0 0x7fcd740bd690 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00cef0 tx=0x7fcd6c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:19.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd6a7fc640 1 -- 172.21.15.107:0/2125280603 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c00d840 con 0x7fcd740a94b0 2023-10-24T23:57:19.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd6a7fc640 1 -- 172.21.15.107:0/2125280603 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c00d9c0 con 0x7fcd740a94b0 2023-10-24T23:57:19.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7bc02640 1 -- 172.21.15.107:0/2125280603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd740c02b0 con 0x7fcd740a94b0 2023-10-24T23:57:19.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd6a7fc640 1 -- 172.21.15.107:0/2125280603 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c00dd70 con 0x7fcd740a94b0 2023-10-24T23:57:19.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7bc02640 1 -- 172.21.15.107:0/2125280603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd740723a0 con 0x7fcd740a94b0 2023-10-24T23:57:19.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7bc02640 1 -- 172.21.15.107:0/2125280603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd74070f90 con 0x7fcd740a94b0 2023-10-24T23:57:19.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd6a7fc640 1 -- 172.21.15.107:0/2125280603 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcd6c01e3b0 con 0x7fcd740a94b0 2023-10-24T23:57:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd6a7fc640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd58065e90 0x7fcd58068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd6a7fc640 1 -- 172.21.15.107:0/2125280603 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7fcd6c084650 con 0x7fcd740a94b0 2023-10-24T23:57:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7999e640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd58065e90 0x7fcd58068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.906+0000 7fcd7999e640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd58065e90 0x7fcd58068350 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcd740c5c50 tx=0x7fcd70008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:19.910+0000 7fcd6a7fc640 1 -- 172.21.15.107:0/2125280603 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fcd6c051910 con 0x7fcd740a94b0 2023-10-24T23:57:20.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.050+0000 7fee78753640 1 -- 172.21.15.107:0/571166373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee700746e0 con 0x7fee70156770 2023-10-24T23:57:20.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.050+0000 7fee6effd640 1 -- 172.21.15.107:0/571166373 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7fee60054e60 con 0x7fee70156770 2023-10-24T23:57:20.057 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:20.057 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:20.058 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:20.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 -- 172.21.15.107:0/571166373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee5c065e90 msgr2=0x7fee5c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:20.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee5c065e90 0x7fee5c068350 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fee68000f70 tx=0x7fee68009040 comp rx=0 tx=0).stop 2023-10-24T23:57:20.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 -- 172.21.15.107:0/571166373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee70156770 msgr2=0x7fee70071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:20.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee70156770 0x7fee70071ce0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fee60007e40 tx=0x7fee6000e4b0 comp rx=0 tx=0).stop 2023-10-24T23:57:20.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 -- 172.21.15.107:0/571166373 shutdown_connections 2023-10-24T23:57:20.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee70156770 0x7fee70071ce0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:20.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee700c70e0 0x7fee7006fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:20.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee5c065e90 0x7fee5c068350 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:20.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 --2- 172.21.15.107:0/571166373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee700c2130 0x7fee701537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:20.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 -- 172.21.15.107:0/571166373 >> 172.21.15.107:0/571166373 conn(0x7fee700c65c0 msgr2=0x7fee701549a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:20.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 -- 172.21.15.107:0/571166373 shutdown_connections 2023-10-24T23:57:20.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.054+0000 7fee78753640 1 -- 172.21.15.107:0/571166373 wait complete. 2023-10-24T23:57:20.076 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:57:20.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.098+0000 7fcd7bc02640 1 -- 172.21.15.107:0/2125280603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd740746e0 con 0x7fcd740a94b0 2023-10-24T23:57:20.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.098+0000 7fcd6a7fc640 1 -- 172.21.15.107:0/2125280603 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7fcd6c054f30 con 0x7fcd740a94b0 2023-10-24T23:57:20.105 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:20.105 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:20.106 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:20.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 -- 172.21.15.107:0/2125280603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd58065e90 msgr2=0x7fcd58068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:20.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd58065e90 0x7fcd58068350 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcd740c5c50 tx=0x7fcd70008040 comp rx=0 tx=0).stop 2023-10-24T23:57:20.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 -- 172.21.15.107:0/2125280603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd740a94b0 msgr2=0x7fcd740bd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:20.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd740a94b0 0x7fcd740bd690 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00cef0 tx=0x7fcd6c00a710 comp rx=0 tx=0).stop 2023-10-24T23:57:20.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 -- 172.21.15.107:0/2125280603 shutdown_connections 2023-10-24T23:57:20.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd740a94b0 0x7fcd740bd690 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:20.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd74069d40 0x7fcd740bca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:20.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd58065e90 0x7fcd58068350 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:20.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 --2- 172.21.15.107:0/2125280603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd74065bc0 0x7fcd74071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:20.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 -- 172.21.15.107:0/2125280603 >> 172.21.15.107:0/2125280603 conn(0x7fcd740c65c0 msgr2=0x7fcd74150fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:20.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 -- 172.21.15.107:0/2125280603 shutdown_connections 2023-10-24T23:57:20.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:20.102+0000 7fcd7bc02640 1 -- 172.21.15.107:0/2125280603 wait complete. 2023-10-24T23:57:20.121 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:57:22.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:22.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:22.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.198+0000 7f7a6d539640 1 Processor -- start 2023-10-24T23:57:22.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.198+0000 7f7a6d539640 1 -- start start 2023-10-24T23:57:22.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a6d539640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a6814c470 0x7f7a6814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a6d539640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7a6806f490 0x7f7a68071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a6d539640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a68155cd0 con 0x7f7a68071fc0 2023-10-24T23:57:22.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a6d539640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a681526e0 con 0x7f7a6806f490 2023-10-24T23:57:22.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a6d539640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680c4380 con 0x7f7a6814c470 2023-10-24T23:57:22.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a66ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a6814c470 0x7f7a6814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a5ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7a6806f490 0x7f7a68071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a5ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7a6806f490 0x7f7a68071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:51072/0 (socket says 172.21.15.107:51072) 2023-10-24T23:57:22.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a66ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a6814c470 0x7f7a6814c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:37270/0 (socket says 172.21.15.107:37270) 2023-10-24T23:57:22.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a5ffff640 1 -- 172.21.15.107:0/129625595 learned_addr learned my addr 172.21.15.107:0/129625595 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:22.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a66ffd640 1 -- 172.21.15.107:0/129625595 >> v2:172.21.15.110:3300/0 conn(0x7f7a6806f490 msgr2=0x7f7a68071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a66ffd640 1 --2- 172.21.15.107:0/129625595 >> v2:172.21.15.110:3300/0 conn(0x7f7a6806f490 0x7f7a68071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a66ffd640 1 -- 172.21.15.107:0/129625595 >> v1:172.21.15.110:6789/0 conn(0x7f7a68071fc0 legacy=0x7f7a68152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a66ffd640 1 -- 172.21.15.107:0/129625595 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a680c48b0 con 0x7f7a6814c470 2023-10-24T23:57:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a5ffff640 1 --2- 172.21.15.107:0/129625595 >> v2:172.21.15.110:3300/0 conn(0x7f7a6806f490 0x7f7a68071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:57:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a66ffd640 1 --2- 172.21.15.107:0/129625595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a6814c470 0x7f7a6814c850 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a580097c0 tx=0x7f7a5801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72820103201a9191 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a667fc640 1 -- 172.21.15.107:0/129625595 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a580ab600 con 0x7f7a6814c470 2023-10-24T23:57:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a667fc640 1 -- 172.21.15.107:0/129625595 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a580a7050 con 0x7f7a6814c470 2023-10-24T23:57:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.202+0000 7f7a667fc640 1 -- 172.21.15.107:0/129625595 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a580ab980 con 0x7f7a6814c470 2023-10-24T23:57:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 -- 172.21.15.107:0/129625595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a6814c470 msgr2=0x7f7a6814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 --2- 172.21.15.107:0/129625595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a6814c470 0x7f7a6814c850 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7a580097c0 tx=0x7f7a5801c360 comp rx=0 tx=0).stop 2023-10-24T23:57:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 -- 172.21.15.107:0/129625595 shutdown_connections 2023-10-24T23:57:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 --2- 172.21.15.107:0/129625595 >> v2:172.21.15.110:3300/0 conn(0x7f7a6806f490 0x7f7a68071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 --2- 172.21.15.107:0/129625595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a6814c470 0x7f7a6814c850 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 -- 172.21.15.107:0/129625595 >> 172.21.15.107:0/129625595 conn(0x7f7a68070f60 msgr2=0x7f7a68071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 -- 172.21.15.107:0/129625595 shutdown_connections 2023-10-24T23:57:22.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 -- 172.21.15.107:0/129625595 wait complete. 2023-10-24T23:57:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 Processor -- start 2023-10-24T23:57:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 -- start start 2023-10-24T23:57:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a6806f490 0x7f7a68143780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a68071fc0 0x7f7a680c3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a6814c470 0x7f7a68148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a680ab2f0 con 0x7f7a6806f490 2023-10-24T23:57:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a681eee00 con 0x7f7a6814c470 2023-10-24T23:57:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a681eadd0 con 0x7f7a68071fc0 2023-10-24T23:57:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a66ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a6806f490 0x7f7a68143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a66ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a6806f490 0x7f7a68143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47012/0 (socket says 172.21.15.107:47012) 2023-10-24T23:57:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a66ffd640 1 -- 172.21.15.107:0/4168407411 learned_addr learned my addr 172.21.15.107:0/4168407411 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a5ffff640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a68071fc0 0x7f7a680c3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a677fe640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a6814c470 0x7f7a68148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a66ffd640 1 -- 172.21.15.107:0/4168407411 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a68071fc0 msgr2=0x7f7a680c3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a66ffd640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a68071fc0 0x7f7a680c3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a66ffd640 1 -- 172.21.15.107:0/4168407411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a6814c470 msgr2=0x7f7a68148240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a66ffd640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a6814c470 0x7f7a68148240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a66ffd640 1 -- 172.21.15.107:0/4168407411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a500094d0 con 0x7f7a6806f490 2023-10-24T23:57:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a677fe640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a6814c470 0x7f7a68148240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:57:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a66ffd640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a6806f490 0x7f7a68143780 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7a58019f90 tx=0x7f7a58019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a5f7fe640 1 -- 172.21.15.107:0/4168407411 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a580abd90 con 0x7f7a6806f490 2023-10-24T23:57:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a5f7fe640 1 -- 172.21.15.107:0/4168407411 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a58006e20 con 0x7f7a6806f490 2023-10-24T23:57:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 -- 172.21.15.107:0/4168407411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a58009420 con 0x7f7a6806f490 2023-10-24T23:57:22.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a5f7fe640 1 -- 172.21.15.107:0/4168407411 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a580b66c0 con 0x7f7a6806f490 2023-10-24T23:57:22.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.206+0000 7f7a6d539640 1 -- 172.21.15.107:0/4168407411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a681eb390 con 0x7f7a6806f490 2023-10-24T23:57:22.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.210+0000 7f7a6d539640 1 -- 172.21.15.107:0/4168407411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a680746e0 con 0x7f7a6806f490 2023-10-24T23:57:22.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.210+0000 7f7a5f7fe640 1 -- 172.21.15.107:0/4168407411 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7a580b2040 con 0x7f7a6806f490 2023-10-24T23:57:22.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.210+0000 7f7a5f7fe640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a48065e60 0x7f7a48068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.210+0000 7f7a5f7fe640 1 -- 172.21.15.107:0/4168407411 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7f7a5811e510 con 0x7f7a6806f490 2023-10-24T23:57:22.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.210+0000 7f7a5ffff640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a48065e60 0x7f7a48068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.210+0000 7f7a5ffff640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a48065e60 0x7f7a48068320 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7a68143cc0 tx=0x7f7a50009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:22.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.214+0000 7f7a5f7fe640 1 -- 172.21.15.107:0/4168407411 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7a580eb7c0 con 0x7f7a6806f490 2023-10-24T23:57:22.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb681723640 1 Processor -- start 2023-10-24T23:57:22.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb681723640 1 -- start start 2023-10-24T23:57:22.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb681723640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb67c14fc30 0x7fb67c153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb681723640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb67c0c5c20 0x7fb67c06df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb681723640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c072330 con 0x7fb67c06e470 2023-10-24T23:57:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb681723640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67c155ca0 con 0x7fb67c0c5c20 2023-10-24T23:57:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb681723640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c155f80 con 0x7fb67c14fc30 2023-10-24T23:57:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb67b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb67c06e470 0x7fb67c0c18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52112/0 (socket says 172.21.15.107:52112) 2023-10-24T23:57:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb67b7fe640 1 -- 172.21.15.107:0/2617367393 learned_addr learned my addr 172.21.15.107:0/2617367393 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb67affd640 1 --2- 172.21.15.107:0/2617367393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb67c14fc30 0x7fb67c153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb67a7fc640 1 --2- 172.21.15.107:0/2617367393 >> v2:172.21.15.110:3300/0 conn(0x7fb67c0c5c20 0x7fb67c06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb679ffb640 1 -- 172.21.15.107:0/2617367393 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343102431 0 0) 0x7fb67c072330 con 0x7fb67c06e470 2023-10-24T23:57:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb679ffb640 1 -- 172.21.15.107:0/2617367393 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb668003530 con 0x7fb67c06e470 2023-10-24T23:57:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb679ffb640 1 -- 172.21.15.107:0/2617367393 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1791262659 0 0) 0x7fb668003530 con 0x7fb67c06e470 2023-10-24T23:57:22.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb67affd640 1 -- 172.21.15.107:0/2617367393 >> v2:172.21.15.110:3300/0 conn(0x7fb67c0c5c20 msgr2=0x7fb67c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:22.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb67affd640 1 --2- 172.21.15.107:0/2617367393 >> v2:172.21.15.110:3300/0 conn(0x7fb67c0c5c20 0x7fb67c06df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb67affd640 1 -- 172.21.15.107:0/2617367393 >> v1:172.21.15.107:6789/0 conn(0x7fb67c06e470 legacy=0x7fb67c0c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.242+0000 7fb67affd640 1 -- 172.21.15.107:0/2617367393 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c156260 con 0x7fb67c14fc30 2023-10-24T23:57:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb67affd640 1 --2- 172.21.15.107:0/2617367393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb67c14fc30 0x7fb67c153190 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb6740097c0 tx=0x7fb67401c2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2feffb8d2918923a server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb679ffb640 1 -- 172.21.15.107:0/2617367393 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c001a00 con 0x7fb67c14fc30 2023-10-24T23:57:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb679ffb640 1 -- 172.21.15.107:0/2617367393 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6740a7050 con 0x7fb67c14fc30 2023-10-24T23:57:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb679ffb640 1 -- 172.21.15.107:0/2617367393 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740ab600 con 0x7fb67c14fc30 2023-10-24T23:57:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb681723640 1 -- 172.21.15.107:0/2617367393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb67c14fc30 msgr2=0x7fb67c153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb681723640 1 --2- 172.21.15.107:0/2617367393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb67c14fc30 0x7fb67c153190 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb6740097c0 tx=0x7fb67401c2e0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb681723640 1 -- 172.21.15.107:0/2617367393 shutdown_connections 2023-10-24T23:57:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb681723640 1 --2- 172.21.15.107:0/2617367393 >> v2:172.21.15.110:3300/0 conn(0x7fb67c0c5c20 0x7fb67c06df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb681723640 1 --2- 172.21.15.107:0/2617367393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb67c14fc30 0x7fb67c153190 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb681723640 1 -- 172.21.15.107:0/2617367393 >> 172.21.15.107:0/2617367393 conn(0x7fb67c0c39a0 msgr2=0x7fb67c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb681723640 1 -- 172.21.15.107:0/2617367393 shutdown_connections 2023-10-24T23:57:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.246+0000 7fb681723640 1 -- 172.21.15.107:0/2617367393 wait complete. 2023-10-24T23:57:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb681723640 1 Processor -- start 2023-10-24T23:57:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb681723640 1 -- start start 2023-10-24T23:57:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb681723640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c06e470 0x7fb67c138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb681723640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb67c0c5c20 0x7fb67c154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb681723640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb67c14fc30 0x7fb67c1ea600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb681723640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c153390 con 0x7fb67c14fc30 2023-10-24T23:57:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb681723640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c13a610 con 0x7fb67c06e470 2023-10-24T23:57:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb681723640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c137960 con 0x7fb67c0c5c20 2023-10-24T23:57:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb67c14fc30 0x7fb67c1ea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb67c14fc30 0x7fb67c1ea600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47026/0 (socket says 172.21.15.107:47026) 2023-10-24T23:57:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67b7fe640 1 -- 172.21.15.107:0/420406319 learned_addr learned my addr 172.21.15.107:0/420406319 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67a7fc640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb67c0c5c20 0x7fb67c154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67affd640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c06e470 0x7fb67c138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67b7fe640 1 -- 172.21.15.107:0/420406319 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb67c0c5c20 msgr2=0x7fb67c154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67b7fe640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb67c0c5c20 0x7fb67c154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67b7fe640 1 -- 172.21.15.107:0/420406319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c06e470 msgr2=0x7fb67c138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67b7fe640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c06e470 0x7fb67c138220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67b7fe640 1 -- 172.21.15.107:0/420406319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb674009420 con 0x7fb67c14fc30 2023-10-24T23:57:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67affd640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c06e470 0x7fb67c138220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:57:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67b7fe640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb67c14fc30 0x7fb67c1ea600 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb66c001870 tx=0x7fb66c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb6737fe640 1 -- 172.21.15.107:0/420406319 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c012400 con 0x7fb67c14fc30 2023-10-24T23:57:22.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb6737fe640 1 -- 172.21.15.107:0/420406319 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c012560 con 0x7fb67c14fc30 2023-10-24T23:57:22.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb681723640 1 -- 172.21.15.107:0/420406319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c06cba0 con 0x7fb67c14fc30 2023-10-24T23:57:22.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb6737fe640 1 -- 172.21.15.107:0/420406319 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c012910 con 0x7fb67c14fc30 2023-10-24T23:57:22.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb681723640 1 -- 172.21.15.107:0/420406319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c06b5a0 con 0x7fb67c14fc30 2023-10-24T23:57:22.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb681723640 1 -- 172.21.15.107:0/420406319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67c0746e0 con 0x7fb67c14fc30 2023-10-24T23:57:22.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb6737fe640 1 -- 172.21.15.107:0/420406319 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb66c012a70 con 0x7fb67c14fc30 2023-10-24T23:57:22.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb6737fe640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb65c065e90 0x7fb65c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:22.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb6737fe640 1 -- 172.21.15.107:0/420406319 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 7305+0+0 (secure 0 0 0) 0x7fb66c0842e0 con 0x7fb67c14fc30 2023-10-24T23:57:22.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.250+0000 7fb67affd640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb65c065e90 0x7fb65c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:22.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.254+0000 7fb67affd640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb65c065e90 0x7fb65c068350 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb6740ad000 tx=0x7fb6740a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:22.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.254+0000 7fb6737fe640 1 -- 172.21.15.107:0/420406319 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb66c0514e0 con 0x7fb67c14fc30 2023-10-24T23:57:22.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.406+0000 7f7a6d539640 1 -- 172.21.15.107:0/4168407411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a6806f870 con 0x7f7a6806f490 2023-10-24T23:57:22.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.406+0000 7f7a5f7fe640 1 -- 172.21.15.107:0/4168407411 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7f7a580eede0 con 0x7f7a6806f490 2023-10-24T23:57:22.413 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:22.413 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:22.414 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:22.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 -- 172.21.15.107:0/4168407411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a48065e60 msgr2=0x7f7a48068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a48065e60 0x7f7a48068320 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7a68143cc0 tx=0x7f7a50009240 comp rx=0 tx=0).stop 2023-10-24T23:57:22.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 -- 172.21.15.107:0/4168407411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a6806f490 msgr2=0x7f7a68143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a6806f490 0x7f7a68143780 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7a58019f90 tx=0x7f7a58019100 comp rx=0 tx=0).stop 2023-10-24T23:57:22.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 -- 172.21.15.107:0/4168407411 shutdown_connections 2023-10-24T23:57:22.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a6814c470 0x7f7a68148240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a48065e60 0x7f7a48068320 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a68071fc0 0x7f7a680c3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 --2- 172.21.15.107:0/4168407411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a6806f490 0x7f7a68143780 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 -- 172.21.15.107:0/4168407411 >> 172.21.15.107:0/4168407411 conn(0x7f7a68070f60 msgr2=0x7f7a68073170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:22.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 -- 172.21.15.107:0/4168407411 shutdown_connections 2023-10-24T23:57:22.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.410+0000 7f7a6d539640 1 -- 172.21.15.107:0/4168407411 wait complete. 2023-10-24T23:57:22.431 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:57:22.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.446+0000 7fb681723640 1 -- 172.21.15.107:0/420406319 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb67c0c5a00 con 0x7fb67c14fc30 2023-10-24T23:57:22.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.446+0000 7fb6737fe640 1 -- 172.21.15.107:0/420406319 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+5772 (secure 0 0 0) 0x7fb66c054b00 con 0x7fb67c14fc30 2023-10-24T23:57:22.452 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:22.452 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":5562,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:56:21.585439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4799},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4799":{"gid":4799,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/1031240534","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1031240534},{"type":"v1","addr":"172.21.15.110:6835","nonce":1031240534}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:22.453 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 33 2023-10-24T23:57:22.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 -- 172.21.15.107:0/420406319 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb65c065e90 msgr2=0x7fb65c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb65c065e90 0x7fb65c068350 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb6740ad000 tx=0x7fb6740a6000 comp rx=0 tx=0).stop 2023-10-24T23:57:22.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 -- 172.21.15.107:0/420406319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb67c14fc30 msgr2=0x7fb67c1ea600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:22.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb67c14fc30 0x7fb67c1ea600 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb66c001870 tx=0x7fb66c00a430 comp rx=0 tx=0).stop 2023-10-24T23:57:22.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 -- 172.21.15.107:0/420406319 shutdown_connections 2023-10-24T23:57:22.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb67c14fc30 0x7fb67c1ea600 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb65c065e90 0x7fb65c068350 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb67c0c5c20 0x7fb67c154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 --2- 172.21.15.107:0/420406319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb67c06e470 0x7fb67c138220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:22.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 -- 172.21.15.107:0/420406319 >> 172.21.15.107:0/420406319 conn(0x7fb67c0c39a0 msgr2=0x7fb67c0c17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:22.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 -- 172.21.15.107:0/420406319 shutdown_connections 2023-10-24T23:57:22.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:22.450+0000 7fb681723640 1 -- 172.21.15.107:0/420406319 wait complete. 2023-10-24T23:57:22.470 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:57:24.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:24.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.554+0000 7f1ebc1c2640 1 Processor -- start 2023-10-24T23:57:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.554+0000 7f1ebc1c2640 1 -- start start 2023-10-24T23:57:24.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.554+0000 7f1ebc1c2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1eb40720c0 0x7f1eb413d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.554+0000 7f1ebc1c2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb40adf70 con 0x7f1eb4154860 2023-10-24T23:57:24.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.554+0000 7f1ebc1c2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb406e710 con 0x7f1eb4071ce0 2023-10-24T23:57:24.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.554+0000 7f1ebc1c2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb406d900 con 0x7f1eb40720c0 2023-10-24T23:57:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eba75f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1eb4154860 0x7f1eb4158230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60978/0 (socket says 172.21.15.107:60978) 2023-10-24T23:57:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eba75f640 1 -- 172.21.15.107:0/3942767722 learned_addr learned my addr 172.21.15.107:0/3942767722 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb975d640 1 --2- 172.21.15.107:0/3942767722 >> v2:172.21.15.110:3300/0 conn(0x7f1eb40720c0 0x7f1eb413d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623693871 0 0) 0x7f1eb40adf70 con 0x7f1eb4154860 2023-10-24T23:57:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea0003530 con 0x7f1eb4154860 2023-10-24T23:57:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750851124 0 0) 0x7f1ea0003530 con 0x7f1eb4154860 2023-10-24T23:57:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 >> v2:172.21.15.110:3300/0 conn(0x7f1eb40720c0 msgr2=0x7f1eb413d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 --2- 172.21.15.107:0/3942767722 >> v2:172.21.15.110:3300/0 conn(0x7f1eb40720c0 0x7f1eb413d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 >> v1:172.21.15.110:6789/0 conn(0x7f1eb4071ce0 legacy=0x7f1eb40ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb4154d20 con 0x7f1eb4154860 2023-10-24T23:57:24.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f1eac001a00 con 0x7f1eb4154860 2023-10-24T23:57:24.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 >> v1:172.21.15.107:6789/0 conn(0x7f1eb4154860 legacy=0x7f1eb4158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ea0003980 0x7f1ea0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 0x7f1ea0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ea0005190 0x7f1ea000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea000c060 con 0x7f1ea00047d0 2023-10-24T23:57:24.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea000c1d0 con 0x7f1ea0005190 2023-10-24T23:57:24.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea000c390 con 0x7f1ea0003980 2023-10-24T23:57:24.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb975d640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 0x7f1ea0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb9f5e640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ea0003980 0x7f1ea0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eba75f640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ea0005190 0x7f1ea000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb975d640 1 -- 172.21.15.107:0/3942767722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ea0003980 msgr2=0x7f1ea0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:24.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb975d640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ea0003980 0x7f1ea0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb975d640 1 -- 172.21.15.107:0/3942767722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ea0005190 msgr2=0x7f1ea000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:24.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb975d640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ea0005190 0x7f1ea000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eba75f640 1 -- 172.21.15.107:0/3942767722 reap_dead start 2023-10-24T23:57:24.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb975d640 1 -- 172.21.15.107:0/3942767722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb4154d20 con 0x7f1ea00047d0 2023-10-24T23:57:24.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb9f5e640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ea0003980 0x7f1ea0003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.558+0000 7f1eb975d640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 0x7f1ea0004c50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009ae0 tx=0x7f1ea401ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b159b022292dcb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea40a55f0 con 0x7f1ea00047d0 2023-10-24T23:57:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea40a5750 con 0x7f1ea00047d0 2023-10-24T23:57:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb8f5c640 1 -- 172.21.15.107:0/3942767722 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea40a5b00 con 0x7f1ea00047d0 2023-10-24T23:57:24.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3942767722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 msgr2=0x7f1ea0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 0x7f1ea0004c50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009ae0 tx=0x7f1ea401ca20 comp rx=0 tx=0).stop 2023-10-24T23:57:24.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3942767722 shutdown_connections 2023-10-24T23:57:24.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 --2- 172.21.15.107:0/3942767722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 0x7f1ea0004c50 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3942767722 >> 172.21.15.107:0/3942767722 conn(0x7f1eb4070f60 msgr2=0x7f1eb4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:24.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3942767722 shutdown_connections 2023-10-24T23:57:24.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3942767722 wait complete. 2023-10-24T23:57:24.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 Processor -- start 2023-10-24T23:57:24.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 -- start start 2023-10-24T23:57:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 0x7f1eb4152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1eb41379d0 0x7f1eb40c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1eb40c7350 0x7f1eb4155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40c1b70 con 0x7f1ea00047d0 2023-10-24T23:57:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40c22a0 con 0x7f1eb41379d0 2023-10-24T23:57:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb4157ca0 con 0x7f1eb40c7350 2023-10-24T23:57:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb9f5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 0x7f1eb4152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb9f5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 0x7f1eb4152120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49306/0 (socket says 172.21.15.107:49306) 2023-10-24T23:57:24.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb9f5e640 1 -- 172.21.15.107:0/3458154455 learned_addr learned my addr 172.21.15.107:0/3458154455 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:24.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eba75f640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1eb40c7350 0x7f1eb4155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb975d640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1eb41379d0 0x7f1eb40c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb9f5e640 1 -- 172.21.15.107:0/3458154455 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1eb40c7350 msgr2=0x7f1eb4155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb9f5e640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1eb40c7350 0x7f1eb4155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb9f5e640 1 -- 172.21.15.107:0/3458154455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1eb41379d0 msgr2=0x7f1eb40c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb9f5e640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1eb41379d0 0x7f1eb40c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb9f5e640 1 -- 172.21.15.107:0/3458154455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb4138110 con 0x7f1ea00047d0 2023-10-24T23:57:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eba75f640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1eb40c7350 0x7f1eb4155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eb9f5e640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 0x7f1eb4152120 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000d500 tx=0x7f1eb000d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eaa7fc640 1 -- 172.21.15.107:0/3458154455 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0005500 con 0x7f1ea00047d0 2023-10-24T23:57:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1eaa7fc640 1 -- 172.21.15.107:0/3458154455 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb0007030 con 0x7f1ea00047d0 2023-10-24T23:57:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.562+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3458154455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb4138360 con 0x7f1ea00047d0 2023-10-24T23:57:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.566+0000 7f1eaa7fc640 1 -- 172.21.15.107:0/3458154455 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb00073c0 con 0x7f1ea00047d0 2023-10-24T23:57:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.566+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3458154455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb40c1d80 con 0x7f1ea00047d0 2023-10-24T23:57:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.566+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3458154455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb41ed5d0 con 0x7f1ea00047d0 2023-10-24T23:57:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.566+0000 7f1eaa7fc640 1 -- 172.21.15.107:0/3458154455 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1eb0007520 con 0x7f1ea00047d0 2023-10-24T23:57:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.566+0000 7f1eaa7fc640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e98065e60 0x7f1e98068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.566+0000 7f1eaa7fc640 1 -- 172.21.15.107:0/3458154455 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7391+0+0 (secure 0 0 0) 0x7f1eb00835e0 con 0x7f1ea00047d0 2023-10-24T23:57:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.566+0000 7f1eb975d640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e98065e60 0x7f1e98068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.566+0000 7f1eb975d640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e98065e60 0x7f1e98068320 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40af000 tx=0x7f1ea4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.570+0000 7f1eaa7fc640 1 -- 172.21.15.107:0/3458154455 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1eb0050850 con 0x7f1ea00047d0 2023-10-24T23:57:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.590+0000 7fa226513640 1 Processor -- start 2023-10-24T23:57:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.590+0000 7fa226513640 1 -- start start 2023-10-24T23:57:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa22014fd40 0x7fa2200ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa2200a9820 0x7fa2200aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa22006f730 con 0x7fa22006ee40 2023-10-24T23:57:24.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2200aa750 con 0x7fa2200a9820 2023-10-24T23:57:24.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa220156a40 con 0x7fa22014fd40 2023-10-24T23:57:24.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa21f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa2200a9820 0x7fa2200aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa224ab0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa22006ee40 0x7fa2200c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60986/0 (socket says 172.21.15.107:60986) 2023-10-24T23:57:24.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa224ab0640 1 -- 172.21.15.107:0/2966487124 learned_addr learned my addr 172.21.15.107:0/2966487124 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:24.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa21ffff640 1 --2- 172.21.15.107:0/2966487124 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa22014fd40 0x7fa2200ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa21f7fe640 1 -- 172.21.15.107:0/2966487124 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa22014fd40 msgr2=0x7fa2200ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:24.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa21f7fe640 1 --2- 172.21.15.107:0/2966487124 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa22014fd40 0x7fa2200ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa21f7fe640 1 -- 172.21.15.107:0/2966487124 >> v1:172.21.15.107:6789/0 conn(0x7fa22006ee40 legacy=0x7fa2200c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa21f7fe640 1 -- 172.21.15.107:0/2966487124 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa220156710 con 0x7fa2200a9820 2023-10-24T23:57:24.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa21effd640 1 -- 172.21.15.107:0/2966487124 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534408406 0 0) 0x7fa22006f730 con 0x7fa22006ee40 2023-10-24T23:57:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa21f7fe640 1 --2- 172.21.15.107:0/2966487124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2200a9820 0x7fa2200aa210 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa2100097c0 tx=0x7fa21001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea34a5caa43e69f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa21effd640 1 -- 172.21.15.107:0/2966487124 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2100ab600 con 0x7fa2200a9820 2023-10-24T23:57:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa21effd640 1 -- 172.21.15.107:0/2966487124 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2100a7050 con 0x7fa2200a9820 2023-10-24T23:57:24.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa21effd640 1 -- 172.21.15.107:0/2966487124 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2100ab980 con 0x7fa2200a9820 2023-10-24T23:57:24.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 -- 172.21.15.107:0/2966487124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2200a9820 msgr2=0x7fa2200aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 --2- 172.21.15.107:0/2966487124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2200a9820 0x7fa2200aa210 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa2100097c0 tx=0x7fa21001c330 comp rx=0 tx=0).stop 2023-10-24T23:57:24.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 -- 172.21.15.107:0/2966487124 shutdown_connections 2023-10-24T23:57:24.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 --2- 172.21.15.107:0/2966487124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2200a9820 0x7fa2200aa210 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 --2- 172.21.15.107:0/2966487124 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa22014fd40 0x7fa2200ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 -- 172.21.15.107:0/2966487124 >> 172.21.15.107:0/2966487124 conn(0x7fa2200c2950 msgr2=0x7fa2200c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:24.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 -- 172.21.15.107:0/2966487124 shutdown_connections 2023-10-24T23:57:24.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.594+0000 7fa226513640 1 -- 172.21.15.107:0/2966487124 wait complete. 2023-10-24T23:57:24.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa226513640 1 Processor -- start 2023-10-24T23:57:24.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa226513640 1 -- start start 2023-10-24T23:57:24.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa226513640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa22006ee40 0x7fa2200bb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa226513640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa22014fd40 0x7fa22014d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa226513640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa22014c760 0x7fa22014cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa226513640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200bbb10 con 0x7fa22014c760 2023-10-24T23:57:24.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa226513640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22014d130 con 0x7fa22006ee40 2023-10-24T23:57:24.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa226513640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200bbd90 con 0x7fa22014fd40 2023-10-24T23:57:24.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa224ab0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa22014c760 0x7fa22014cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa224ab0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa22014c760 0x7fa22014cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49324/0 (socket says 172.21.15.107:49324) 2023-10-24T23:57:24.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa224ab0640 1 -- 172.21.15.107:0/1253712878 learned_addr learned my addr 172.21.15.107:0/1253712878 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:24.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa21ffff640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa22006ee40 0x7fa2200bb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa21f7fe640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa22014fd40 0x7fa22014d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa224ab0640 1 -- 172.21.15.107:0/1253712878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa22014fd40 msgr2=0x7fa22014d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa224ab0640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa22014fd40 0x7fa22014d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa224ab0640 1 -- 172.21.15.107:0/1253712878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa22006ee40 msgr2=0x7fa2200bb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa224ab0640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa22006ee40 0x7fa2200bb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa224ab0640 1 -- 172.21.15.107:0/1253712878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa210009420 con 0x7fa22014c760 2023-10-24T23:57:24.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa21f7fe640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa22014fd40 0x7fa22014d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:24.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa224ab0640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa22014c760 0x7fa22014cbf0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c780 tx=0x7fa21800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:24.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa21cff9640 1 -- 172.21.15.107:0/1253712878 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21800ce90 con 0x7fa22014c760 2023-10-24T23:57:24.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa21cff9640 1 -- 172.21.15.107:0/1253712878 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21800a640 con 0x7fa22014c760 2023-10-24T23:57:24.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa21cff9640 1 -- 172.21.15.107:0/1253712878 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21800a9d0 con 0x7fa22014c760 2023-10-24T23:57:24.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa226513640 1 -- 172.21.15.107:0/1253712878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22015b4e0 con 0x7fa22014c760 2023-10-24T23:57:24.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa226513640 1 -- 172.21.15.107:0/1253712878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22015baa0 con 0x7fa22014c760 2023-10-24T23:57:24.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.598+0000 7fa226513640 1 -- 172.21.15.107:0/1253712878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2200c24a0 con 0x7fa22014c760 2023-10-24T23:57:24.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.602+0000 7fa21cff9640 1 -- 172.21.15.107:0/1253712878 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa21800ab30 con 0x7fa22014c760 2023-10-24T23:57:24.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.602+0000 7fa21cff9640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa20c065e60 0x7fa20c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.602+0000 7fa21ffff640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa20c065e60 0x7fa20c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.602+0000 7fa21cff9640 1 -- 172.21.15.107:0/1253712878 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7391+0+0 (secure 0 0 0) 0x7fa2180832d0 con 0x7fa22014c760 2023-10-24T23:57:24.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.602+0000 7fa21ffff640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa20c065e60 0x7fa20c068320 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa214009a30 tx=0x7fa214008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:24.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.606+0000 7fa21cff9640 1 -- 172.21.15.107:0/1253712878 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa2180503f0 con 0x7fa22014c760 2023-10-24T23:57:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3458154455 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1eb40746e0 con 0x7f1ea00047d0 2023-10-24T23:57:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1eaa7fc640 1 -- 172.21.15.107:0/3458154455 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5016 (secure 0 0 0) 0x7f1eb0053e70 con 0x7f1ea00047d0 2023-10-24T23:57:24.775 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:24.775 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:24.551466+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:24.776 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 35 2023-10-24T23:57:24.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3458154455 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e98065e60 msgr2=0x7f1e98068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e98065e60 0x7f1e98068320 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40af000 tx=0x7f1ea4008040 comp rx=0 tx=0).stop 2023-10-24T23:57:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3458154455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 msgr2=0x7f1eb4152120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 0x7f1eb4152120 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000d500 tx=0x7f1eb000d9c0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3458154455 shutdown_connections 2023-10-24T23:57:24.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e98065e60 0x7f1e98068320 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1eb40c7350 0x7f1eb4155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1eb41379d0 0x7f1eb40c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 --2- 172.21.15.107:0/3458154455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ea00047d0 0x7f1eb4152120 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3458154455 >> 172.21.15.107:0/3458154455 conn(0x7f1eb4070f60 msgr2=0x7f1eb406c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:24.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3458154455 shutdown_connections 2023-10-24T23:57:24.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.770+0000 7f1ebc1c2640 1 -- 172.21.15.107:0/3458154455 wait complete. 2023-10-24T23:57:24.791 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:57:24.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 -- 172.21.15.107:0/1253712878 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa22006f730 con 0x7fa22014c760 2023-10-24T23:57:24.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa21cff9640 1 -- 172.21.15.107:0/1253712878 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5016 (secure 0 0 0) 0x7fa218053a10 con 0x7fa22014c760 2023-10-24T23:57:24.816 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:24.816 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:24.551466+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:24.817 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 35 2023-10-24T23:57:24.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 -- 172.21.15.107:0/1253712878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa20c065e60 msgr2=0x7fa20c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa20c065e60 0x7fa20c068320 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa214009a30 tx=0x7fa214008040 comp rx=0 tx=0).stop 2023-10-24T23:57:24.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 -- 172.21.15.107:0/1253712878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa22014c760 msgr2=0x7fa22014cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa22014c760 0x7fa22014cbf0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa21800c780 tx=0x7fa21800cc40 comp rx=0 tx=0).stop 2023-10-24T23:57:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 -- 172.21.15.107:0/1253712878 shutdown_connections 2023-10-24T23:57:24.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa22014c760 0x7fa22014cbf0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa20c065e60 0x7fa20c068320 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa22014fd40 0x7fa22014d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 --2- 172.21.15.107:0/1253712878 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa22006ee40 0x7fa2200bb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 -- 172.21.15.107:0/1253712878 >> 172.21.15.107:0/1253712878 conn(0x7fa2200c2950 msgr2=0x7fa22014dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:24.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 -- 172.21.15.107:0/1253712878 shutdown_connections 2023-10-24T23:57:24.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.810+0000 7fa226513640 1 -- 172.21.15.107:0/1253712878 wait complete. 2023-10-24T23:57:24.833 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2023-10-24T23:57:24.833 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-24T23:57:24.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:24.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feebbd24640 1 Processor -- start 2023-10-24T23:57:24.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feebbd24640 1 -- start start 2023-10-24T23:57:24.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feebbd24640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feeb406ee40 0x7feeb40ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:24.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feebbd24640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb406f730 con 0x7feeb414fd40 2023-10-24T23:57:24.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feebbd24640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb4156a40 con 0x7feeb40a9820 2023-10-24T23:57:24.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feebbd24640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeb4156710 con 0x7feeb406ee40 2023-10-24T23:57:24.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb9ac0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feeb414fd40 0x7feeb40c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32770/0 (socket says 172.21.15.107:32770) 2023-10-24T23:57:24.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb9ac0640 1 -- 172.21.15.107:0/1760169583 learned_addr learned my addr 172.21.15.107:0/1760169583 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:24.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeba2c1640 1 --2- 172.21.15.107:0/1760169583 >> v2:172.21.15.110:3300/0 conn(0x7feeb406ee40 0x7feeb40ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:24.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500582392 0 0) 0x7feeb406f730 con 0x7feeb414fd40 2023-10-24T23:57:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea0003530 con 0x7feeb414fd40 2023-10-24T23:57:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 329973750 0 0) 0x7feea0003530 con 0x7feeb414fd40 2023-10-24T23:57:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 >> v2:172.21.15.110:3300/0 conn(0x7feeb406ee40 msgr2=0x7feeb40ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 --2- 172.21.15.107:0/1760169583 >> v2:172.21.15.110:3300/0 conn(0x7feeb406ee40 0x7feeb40ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:24.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 >> v1:172.21.15.110:6789/0 conn(0x7feeb40a9820 legacy=0x7feeb415af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb406f980 con 0x7feeb414fd40 2023-10-24T23:57:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7feeb0001680 con 0x7feeb414fd40 2023-10-24T23:57:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 >> v1:172.21.15.107:6789/0 conn(0x7feeb414fd40 legacy=0x7feeb40c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 --2- 172.21.15.107:0/1760169583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea0003980 0x7feea0003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:25.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 --2- 172.21.15.107:0/1760169583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feea00047d0 0x7feea0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:25.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 --2- 172.21.15.107:0/1760169583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 0x7feea000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:25.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb4156710 con 0x7feea0005190 2023-10-24T23:57:25.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea000c0e0 con 0x7feea0003980 2023-10-24T23:57:25.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea000c2a0 con 0x7feea00047d0 2023-10-24T23:57:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeba2c1640 1 --2- 172.21.15.107:0/1760169583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 0x7feea000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb9ac0640 1 --2- 172.21.15.107:0/1760169583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea0003980 0x7feea0003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeb92bf640 1 --2- 172.21.15.107:0/1760169583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feea00047d0 0x7feea0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeba2c1640 1 -- 172.21.15.107:0/1760169583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feea00047d0 msgr2=0x7feea0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeba2c1640 1 --2- 172.21.15.107:0/1760169583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feea00047d0 0x7feea0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:25.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeba2c1640 1 -- 172.21.15.107:0/1760169583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea0003980 msgr2=0x7feea0003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:25.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeba2c1640 1 --2- 172.21.15.107:0/1760169583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feea0003980 0x7feea0003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:25.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.990+0000 7feeba2c1640 1 -- 172.21.15.107:0/1760169583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea0003530 con 0x7feea0005190 2023-10-24T23:57:25.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feeba2c1640 1 -- 172.21.15.107:0/1760169583 reap_dead start 2023-10-24T23:57:25.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feeba2c1640 1 --2- 172.21.15.107:0/1760169583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 0x7feea000b990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7feeac009f90 tx=0x7feeac01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72251719f98e847 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:25.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac003ba0 con 0x7feea0005190 2023-10-24T23:57:25.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeac00f040 con 0x7feea0005190 2023-10-24T23:57:25.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feeb8abe640 1 -- 172.21.15.107:0/1760169583 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac0a5970 con 0x7feea0005190 2023-10-24T23:57:25.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 -- 172.21.15.107:0/1760169583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 msgr2=0x7feea000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:25.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 --2- 172.21.15.107:0/1760169583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 0x7feea000b990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7feeac009f90 tx=0x7feeac01cc60 comp rx=0 tx=0).stop 2023-10-24T23:57:25.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 -- 172.21.15.107:0/1760169583 shutdown_connections 2023-10-24T23:57:25.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 --2- 172.21.15.107:0/1760169583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 0x7feea000b990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:25.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 -- 172.21.15.107:0/1760169583 >> 172.21.15.107:0/1760169583 conn(0x7feeb40c2950 msgr2=0x7feeb40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:25.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 -- 172.21.15.107:0/1760169583 shutdown_connections 2023-10-24T23:57:25.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 -- 172.21.15.107:0/1760169583 wait complete. 2023-10-24T23:57:25.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 Processor -- start 2023-10-24T23:57:25.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 -- start start 2023-10-24T23:57:25.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 0x7feeb4146c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:25.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feeb414c800 0x7feeb414b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:25.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feeb414b900 0x7feeb40bf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:25.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb414afa0 con 0x7feea0005190 2023-10-24T23:57:25.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb4147880 con 0x7feeb414c800 2023-10-24T23:57:25.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.994+0000 7feebbd24640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb414a690 con 0x7feeb414b900 2023-10-24T23:57:25.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb9ac0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 0x7feeb4146c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:25.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb9ac0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 0x7feeb4146c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49346/0 (socket says 172.21.15.107:49346) 2023-10-24T23:57:25.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb9ac0640 1 -- 172.21.15.107:0/1221818846 learned_addr learned my addr 172.21.15.107:0/1221818846 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:25.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb92bf640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feeb414c800 0x7feeb414b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:25.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeba2c1640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feeb414b900 0x7feeb40bf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:25.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb9ac0640 1 -- 172.21.15.107:0/1221818846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feeb414b900 msgr2=0x7feeb40bf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:25.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb9ac0640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feeb414b900 0x7feeb40bf7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:25.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb9ac0640 1 -- 172.21.15.107:0/1221818846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feeb414c800 msgr2=0x7feeb414b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:25.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb9ac0640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feeb414c800 0x7feeb414b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:25.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb9ac0640 1 -- 172.21.15.107:0/1221818846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb41517e0 con 0x7feea0005190 2023-10-24T23:57:25.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb9ac0640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 0x7feeb4146c00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7feeb0001830 tx=0x7feeb0008560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:25.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeaa7fc640 1 -- 172.21.15.107:0/1221818846 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb000f740 con 0x7feea0005190 2023-10-24T23:57:25.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeaa7fc640 1 -- 172.21.15.107:0/1221818846 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb000f8c0 con 0x7feea0005190 2023-10-24T23:57:25.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feebbd24640 1 -- 172.21.15.107:0/1221818846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb41ebd00 con 0x7feea0005190 2023-10-24T23:57:25.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeaa7fc640 1 -- 172.21.15.107:0/1221818846 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb000fc70 con 0x7feea0005190 2023-10-24T23:57:25.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feebbd24640 1 -- 172.21.15.107:0/1221818846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb414abc0 con 0x7feea0005190 2023-10-24T23:57:25.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feebbd24640 1 -- 172.21.15.107:0/1221818846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb40746e0 con 0x7feea0005190 2023-10-24T23:57:25.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeaa7fc640 1 -- 172.21.15.107:0/1221818846 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7feeb00153e0 con 0x7feea0005190 2023-10-24T23:57:25.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeaa7fc640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee98065e90 0x7fee98068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:25.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeaa7fc640 1 -- 172.21.15.107:0/1221818846 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7391+0+0 (secure 0 0 0) 0x7feeb0083e60 con 0x7feea0005190 2023-10-24T23:57:25.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb92bf640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee98065e90 0x7fee98068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:25.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:24.998+0000 7feeb92bf640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee98065e90 0x7fee98068350 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feeb40bb360 tx=0x7feea4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:25.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.002+0000 7feeaa7fc640 1 -- 172.21.15.107:0/1221818846 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7feeb0051010 con 0x7feea0005190 2023-10-24T23:57:25.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.174+0000 7feebbd24640 1 -- 172.21.15.107:0/1221818846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feeb40a9600 con 0x7feea0005190 2023-10-24T23:57:25.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.174+0000 7feeaa7fc640 1 -- 172.21.15.107:0/1221818846 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5016 (secure 0 0 0) 0x7feeb0054630 con 0x7feea0005190 2023-10-24T23:57:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:25.181 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:24.551466+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:25.181 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 35 2023-10-24T23:57:25.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 -- 172.21.15.107:0/1221818846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee98065e90 msgr2=0x7fee98068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:25.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee98065e90 0x7fee98068350 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feeb40bb360 tx=0x7feea4009040 comp rx=0 tx=0).stop 2023-10-24T23:57:25.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 -- 172.21.15.107:0/1221818846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 msgr2=0x7feeb4146c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:25.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 0x7feeb4146c00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7feeb0001830 tx=0x7feeb0008560 comp rx=0 tx=0).stop 2023-10-24T23:57:25.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 -- 172.21.15.107:0/1221818846 shutdown_connections 2023-10-24T23:57:25.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee98065e90 0x7fee98068350 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:25.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feeb414b900 0x7feeb40bf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:25.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feeb414c800 0x7feeb414b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:25.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 --2- 172.21.15.107:0/1221818846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feea0005190 0x7feeb4146c00 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:25.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 -- 172.21.15.107:0/1221818846 >> 172.21.15.107:0/1221818846 conn(0x7feeb40c2950 msgr2=0x7feeb40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:25.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 -- 172.21.15.107:0/1221818846 shutdown_connections 2023-10-24T23:57:25.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:25.178+0000 7feebbd24640 1 -- 172.21.15.107:0/1221818846 wait complete. 2023-10-24T23:57:25.198 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 35, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 27, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5403": { "addr": "172.21.15.107:6835/1004926771", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 1004926771, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 1004926771, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5403, "incarnation": 24, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:56:01.459282+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5403 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5562": { "addr": "172.21.15.107:6837/204494054", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 204494054, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 204494054, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5562, "incarnation": 34, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-24T23:57:24.551466+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5562 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/3721101226", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 3721101226, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 3721101226, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4232, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-24T23:57:26.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:26.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762c888640 1 Processor -- start 2023-10-24T23:57:26.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762c888640 1 -- start start 2023-10-24T23:57:26.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762c888640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628155ea0 0x7f762806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:26.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762c888640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76280c59d0 con 0x7f76281566e0 2023-10-24T23:57:26.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762c888640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76280c2990 con 0x7f7628156280 2023-10-24T23:57:26.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762c888640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76280c2ec0 con 0x7f7628155ea0 2023-10-24T23:57:26.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f7626d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f76281566e0 0x7f76280c3e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32778/0 (socket says 172.21.15.107:32778) 2023-10-24T23:57:26.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f7626d9d640 1 -- 172.21.15.107:0/3825887023 learned_addr learned my addr 172.21.15.107:0/3825887023 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762659c640 1 --2- 172.21.15.107:0/3825887023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628155ea0 0x7f762806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661527393 0 0) 0x7f76280c59d0 con 0x7f76281566e0 2023-10-24T23:57:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7614003530 con 0x7f76281566e0 2023-10-24T23:57:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 353822376 0 0) 0x7f7614003530 con 0x7f76281566e0 2023-10-24T23:57:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628155ea0 msgr2=0x7f762806ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 --2- 172.21.15.107:0/3825887023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628155ea0 0x7f762806ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 >> v1:172.21.15.110:6789/0 conn(0x7f7628156280 legacy=0x7f762814c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762806d500 con 0x7f76281566e0 2023-10-24T23:57:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f7620001680 con 0x7f76281566e0 2023-10-24T23:57:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 >> v1:172.21.15.107:6789/0 conn(0x7f76281566e0 legacy=0x7f76280c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 --2- 172.21.15.107:0/3825887023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76140039b0 0x7f7614003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:26.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 --2- 172.21.15.107:0/3825887023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628155ea0 0x7f76140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:26.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 --2- 172.21.15.107:0/3825887023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7614004d10 0x7f7614005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:26.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76280c2ec0 con 0x7f7614004d10 2023-10-24T23:57:26.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761400bb50 con 0x7f76140039b0 2023-10-24T23:57:26.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761400bd10 con 0x7f7628155ea0 2023-10-24T23:57:26.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f7626d9d640 1 --2- 172.21.15.107:0/3825887023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7614004d10 0x7f7614005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:26.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f762659c640 1 --2- 172.21.15.107:0/3825887023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76140039b0 0x7f7614003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:26.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f7626d9d640 1 -- 172.21.15.107:0/3825887023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628155ea0 msgr2=0x7f76140047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:26.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f7626d9d640 1 --2- 172.21.15.107:0/3825887023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628155ea0 0x7f76140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:26.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f7626d9d640 1 -- 172.21.15.107:0/3825887023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76140039b0 msgr2=0x7f7614003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:26.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f7626d9d640 1 --2- 172.21.15.107:0/3825887023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76140039b0 0x7f7614003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:26.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.918+0000 7f7626d9d640 1 -- 172.21.15.107:0/3825887023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7614003530 con 0x7f7614004d10 2023-10-24T23:57:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f7626d9d640 1 --2- 172.21.15.107:0/3825887023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7614004d10 0x7f7614005190 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f762000c1c0 tx=0x7f762001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7251fc08ffaf820 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7620005ae0 con 0x7f7614004d10 2023-10-24T23:57:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7620005c40 con 0x7f7614004d10 2023-10-24T23:57:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762559a640 1 -- 172.21.15.107:0/3825887023 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76200a5810 con 0x7f7614004d10 2023-10-24T23:57:26.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 -- 172.21.15.107:0/3825887023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7614004d10 msgr2=0x7f7614005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:26.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 --2- 172.21.15.107:0/3825887023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7614004d10 0x7f7614005190 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f762000c1c0 tx=0x7f762001cc20 comp rx=0 tx=0).stop 2023-10-24T23:57:26.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f7626d9d640 1 -- 172.21.15.107:0/3825887023 reap_dead start 2023-10-24T23:57:26.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 -- 172.21.15.107:0/3825887023 shutdown_connections 2023-10-24T23:57:26.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 -- 172.21.15.107:0/3825887023 >> 172.21.15.107:0/3825887023 conn(0x7f76280c4a90 msgr2=0x7f76280c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:26.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 -- 172.21.15.107:0/3825887023 shutdown_connections 2023-10-24T23:57:26.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 -- 172.21.15.107:0/3825887023 wait complete. 2023-10-24T23:57:26.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 Processor -- start 2023-10-24T23:57:26.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 -- start start 2023-10-24T23:57:26.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76280c16e0 0x7f7628146190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:26.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628149090 0x7f76281494d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:26.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628149a10 0x7f762813d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:26.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76281466d0 con 0x7f76280c16e0 2023-10-24T23:57:26.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762813e6e0 con 0x7f7628149090 2023-10-24T23:57:26.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762c888640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762806c200 con 0x7f7628149a10 2023-10-24T23:57:26.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76280c16e0 0x7f7628146190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:26.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76280c16e0 0x7f7628146190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49364/0 (socket says 172.21.15.107:49364) 2023-10-24T23:57:26.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.922+0000 7f762659c640 1 -- 172.21.15.107:0/2495546874 learned_addr learned my addr 172.21.15.107:0/2495546874 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:26.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f7626d9d640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628149a10 0x7f762813d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:26.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f7625d9b640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628149090 0x7f76281494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:26.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f762659c640 1 -- 172.21.15.107:0/2495546874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628149a10 msgr2=0x7f762813d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:26.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f762659c640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628149a10 0x7f762813d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:26.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f762659c640 1 -- 172.21.15.107:0/2495546874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628149090 msgr2=0x7f76281494d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:26.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f762659c640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628149090 0x7f76281494d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:26.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f762659c640 1 -- 172.21.15.107:0/2495546874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762806caa0 con 0x7f76280c16e0 2023-10-24T23:57:26.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f7625d9b640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628149090 0x7f76281494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:26.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f762659c640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76280c16e0 0x7f7628146190 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7610011660 tx=0x7f7610011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:26.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f761effd640 1 -- 172.21.15.107:0/2495546874 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761000f960 con 0x7f76280c16e0 2023-10-24T23:57:26.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f761effd640 1 -- 172.21.15.107:0/2495546874 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761000fac0 con 0x7f76280c16e0 2023-10-24T23:57:26.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f762c888640 1 -- 172.21.15.107:0/2495546874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7628146900 con 0x7f76280c16e0 2023-10-24T23:57:26.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f761effd640 1 -- 172.21.15.107:0/2495546874 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761000fe70 con 0x7f76280c16e0 2023-10-24T23:57:26.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f762c888640 1 -- 172.21.15.107:0/2495546874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762806c730 con 0x7f76280c16e0 2023-10-24T23:57:26.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f762c888640 1 -- 172.21.15.107:0/2495546874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76280746e0 con 0x7f76280c16e0 2023-10-24T23:57:26.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f761effd640 1 -- 172.21.15.107:0/2495546874 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7610015460 con 0x7f76280c16e0 2023-10-24T23:57:26.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f761effd640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7608065e60 0x7f7608068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:26.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f761effd640 1 -- 172.21.15.107:0/2495546874 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7391+0+0 (secure 0 0 0) 0x7f7610084220 con 0x7f76280c16e0 2023-10-24T23:57:26.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f7625d9b640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7608065e60 0x7f7608068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:26.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.926+0000 7f7625d9b640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7608065e60 0x7f7608068320 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7628148890 tx=0x7f7618009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:26.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:26.930+0000 7f761effd640 1 -- 172.21.15.107:0/2495546874 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f76100513b0 con 0x7f76280c16e0 2023-10-24T23:57:27.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.114+0000 7f762c888640 1 -- 172.21.15.107:0/2495546874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76280aa220 con 0x7f76280c16e0 2023-10-24T23:57:27.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.114+0000 7f761effd640 1 -- 172.21.15.107:0/2495546874 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f76100549d0 con 0x7f76280c16e0 2023-10-24T23:57:27.120 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:27.120 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:27.121 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 37 2023-10-24T23:57:27.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 -- 172.21.15.107:0/2495546874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7608065e60 msgr2=0x7f7608068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:27.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7608065e60 0x7f7608068320 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7628148890 tx=0x7f7618009040 comp rx=0 tx=0).stop 2023-10-24T23:57:27.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 -- 172.21.15.107:0/2495546874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76280c16e0 msgr2=0x7f7628146190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76280c16e0 0x7f7628146190 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f7610011660 tx=0x7f7610011b20 comp rx=0 tx=0).stop 2023-10-24T23:57:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 -- 172.21.15.107:0/2495546874 shutdown_connections 2023-10-24T23:57:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7608065e60 0x7f7608068320 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7628149a10 0x7f762813d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7628149090 0x7f76281494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 --2- 172.21.15.107:0/2495546874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76280c16e0 0x7f7628146190 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 -- 172.21.15.107:0/2495546874 >> 172.21.15.107:0/2495546874 conn(0x7f76280c4a90 msgr2=0x7f7628073320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 -- 172.21.15.107:0/2495546874 shutdown_connections 2023-10-24T23:57:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.118+0000 7f762c888640 1 -- 172.21.15.107:0/2495546874 wait complete. 2023-10-24T23:57:27.138 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:57:27.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:27.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.314+0000 7f6e07f0d640 1 Processor -- start 2023-10-24T23:57:27.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.314+0000 7f6e07f0d640 1 -- start start 2023-10-24T23:57:27.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e07f0d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6e0014fd40 0x7f6e000ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:27.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e07f0d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e000a9820 0x7f6e000aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:27.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e07f0d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0006f730 con 0x7f6e0006ee40 2023-10-24T23:57:27.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e07f0d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e000aa750 con 0x7f6e0014fd40 2023-10-24T23:57:27.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e07f0d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e00156a40 con 0x7f6e000a9820 2023-10-24T23:57:27.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e054a8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e000a9820 0x7f6e000aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:27.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e05ca9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6e0014fd40 0x7f6e000ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:27.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e054a8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e000a9820 0x7f6e000aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:40866/0 (socket says 172.21.15.107:40866) 2023-10-24T23:57:27.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e054a8640 1 -- 172.21.15.107:0/3258662904 learned_addr learned my addr 172.21.15.107:0/3258662904 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:27.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e054a8640 1 -- 172.21.15.107:0/3258662904 >> v2:172.21.15.110:3300/0 conn(0x7f6e0014fd40 msgr2=0x7f6e000ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:27.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e054a8640 1 --2- 172.21.15.107:0/3258662904 >> v2:172.21.15.110:3300/0 conn(0x7f6e0014fd40 0x7f6e000ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e054a8640 1 -- 172.21.15.107:0/3258662904 >> v1:172.21.15.110:6789/0 conn(0x7f6e0006ee40 legacy=0x7f6e000c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:27.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.318+0000 7f6e054a8640 1 -- 172.21.15.107:0/3258662904 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e00156710 con 0x7f6e000a9820 2023-10-24T23:57:27.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e054a8640 1 --2- 172.21.15.107:0/3258662904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e000a9820 0x7f6e000aa210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009870 tx=0x7f6dfc01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25cfedcb54ca47c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:27.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e04ca7640 1 -- 172.21.15.107:0/3258662904 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc0ad640 con 0x7f6e000a9820 2023-10-24T23:57:27.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e04ca7640 1 -- 172.21.15.107:0/3258662904 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc0ad7a0 con 0x7f6e000a9820 2023-10-24T23:57:27.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e04ca7640 1 -- 172.21.15.107:0/3258662904 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc0adb30 con 0x7f6e000a9820 2023-10-24T23:57:27.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 -- 172.21.15.107:0/3258662904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e000a9820 msgr2=0x7f6e000aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:27.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 --2- 172.21.15.107:0/3258662904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e000a9820 0x7f6e000aa210 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009870 tx=0x7f6dfc01c3f0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 -- 172.21.15.107:0/3258662904 shutdown_connections 2023-10-24T23:57:27.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 --2- 172.21.15.107:0/3258662904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e000a9820 0x7f6e000aa210 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 --2- 172.21.15.107:0/3258662904 >> v2:172.21.15.110:3300/0 conn(0x7f6e0014fd40 0x7f6e000ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 -- 172.21.15.107:0/3258662904 >> 172.21.15.107:0/3258662904 conn(0x7f6e000c2950 msgr2=0x7f6e000c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:27.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 -- 172.21.15.107:0/3258662904 shutdown_connections 2023-10-24T23:57:27.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 -- 172.21.15.107:0/3258662904 wait complete. 2023-10-24T23:57:27.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 Processor -- start 2023-10-24T23:57:27.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 -- start start 2023-10-24T23:57:27.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e0006ee40 0x7f6e0014f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:27.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e0014fd40 0x7f6e0014dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:27.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e0014e0e0 0x7f6e00154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:27.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e00154f10 con 0x7f6e0014fd40 2023-10-24T23:57:27.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0014f5d0 con 0x7f6e0014e0e0 2023-10-24T23:57:27.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e001538a0 con 0x7f6e0006ee40 2023-10-24T23:57:27.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e054a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e0014fd40 0x7f6e0014dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:27.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e054a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e0014fd40 0x7f6e0014dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49368/0 (socket says 172.21.15.107:49368) 2023-10-24T23:57:27.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e054a8640 1 -- 172.21.15.107:0/2604222023 learned_addr learned my addr 172.21.15.107:0/2604222023 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:27.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e05ca9640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e0006ee40 0x7f6e0014f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:27.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e064aa640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e0014e0e0 0x7f6e00154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:27.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e054a8640 1 -- 172.21.15.107:0/2604222023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e0006ee40 msgr2=0x7f6e0014f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:27.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e054a8640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e0006ee40 0x7f6e0014f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e054a8640 1 -- 172.21.15.107:0/2604222023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e0014e0e0 msgr2=0x7f6e00154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:27.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e054a8640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e0014e0e0 0x7f6e00154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e054a8640 1 -- 172.21.15.107:0/2604222023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc0094d0 con 0x7f6e0014fd40 2023-10-24T23:57:27.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e05ca9640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e0006ee40 0x7f6e0014f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:57:27.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e054a8640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e0014fd40 0x7f6e0014dba0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc000c10 tx=0x7f6dfc019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:27.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6df67fc640 1 -- 172.21.15.107:0/2604222023 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc0b6630 con 0x7f6e0014fd40 2023-10-24T23:57:27.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6df67fc640 1 -- 172.21.15.107:0/2604222023 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc0a6050 con 0x7f6e0014fd40 2023-10-24T23:57:27.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6e07f0d640 1 -- 172.21.15.107:0/2604222023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0014e5a0 con 0x7f6e0014fd40 2023-10-24T23:57:27.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.322+0000 7f6df67fc640 1 -- 172.21.15.107:0/2604222023 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc0b6a10 con 0x7f6e0014fd40 2023-10-24T23:57:27.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.326+0000 7f6e07f0d640 1 -- 172.21.15.107:0/2604222023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0014eb60 con 0x7f6e0014fd40 2023-10-24T23:57:27.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.326+0000 7f6e07f0d640 1 -- 172.21.15.107:0/2604222023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e000c24a0 con 0x7f6e0014fd40 2023-10-24T23:57:27.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.326+0000 7f6df67fc640 1 -- 172.21.15.107:0/2604222023 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6dfc0b4040 con 0x7f6e0014fd40 2023-10-24T23:57:27.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.326+0000 7f6df67fc640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6de4065e60 0x7f6de4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:27.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.326+0000 7f6df67fc640 1 -- 172.21.15.107:0/2604222023 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 7391+0+0 (secure 0 0 0) 0x7f6dfc11de80 con 0x7f6e0014fd40 2023-10-24T23:57:27.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.326+0000 7f6e05ca9640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6de4065e60 0x7f6de4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:27.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.326+0000 7f6e05ca9640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6de4065e60 0x7f6de4068320 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009b10 tx=0x7f6df0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:27.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.330+0000 7f6df67fc640 1 -- 172.21.15.107:0/2604222023 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6dfc0eafa0 con 0x7f6e0014fd40 2023-10-24T23:57:27.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.518+0000 7f6e07f0d640 1 -- 172.21.15.107:0/2604222023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e00070840 con 0x7f6e0014fd40 2023-10-24T23:57:27.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6df67fc640 1 -- 172.21.15.107:0/2604222023 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f6dfc0b2020 con 0x7f6e0014fd40 2023-10-24T23:57:27.525 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:27.526 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:56:01.459282+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5403},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5403":{"gid":5403,"name":"c","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1004926771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1004926771},{"type":"v1","addr":"172.21.15.107:6835","nonce":1004926771}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:27.526 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 37 2023-10-24T23:57:27.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 -- 172.21.15.107:0/2604222023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6de4065e60 msgr2=0x7f6de4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:27.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6de4065e60 0x7f6de4068320 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009b10 tx=0x7f6df0008040 comp rx=0 tx=0).stop 2023-10-24T23:57:27.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 -- 172.21.15.107:0/2604222023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e0014fd40 msgr2=0x7f6e0014dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:27.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e0014fd40 0x7f6e0014dba0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc000c10 tx=0x7f6dfc019190 comp rx=0 tx=0).stop 2023-10-24T23:57:27.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 -- 172.21.15.107:0/2604222023 shutdown_connections 2023-10-24T23:57:27.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e0014e0e0 0x7f6e00154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e0014fd40 0x7f6e0014dba0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6de4065e60 0x7f6de4068320 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 --2- 172.21.15.107:0/2604222023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e0006ee40 0x7f6e0014f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:27.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 -- 172.21.15.107:0/2604222023 >> 172.21.15.107:0/2604222023 conn(0x7f6e000c2950 msgr2=0x7f6e000c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:27.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 -- 172.21.15.107:0/2604222023 shutdown_connections 2023-10-24T23:57:27.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:27.522+0000 7f6e07f0d640 1 -- 172.21.15.107:0/2604222023 wait complete. 2023-10-24T23:57:27.543 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=4799 2023-10-24T23:57:27.543 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.d 2023-10-24T23:57:28.614 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-24T23:57:28.610+0000 7fcc36cc1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:57:29.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:29.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f8a0078c640 1 Processor -- start 2023-10-24T23:57:29.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f8a0078c640 1 -- start start 2023-10-24T23:57:29.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f8a0078c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8156770 0x7f89f80ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f8a0078c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f80c1ca0 con 0x7f89f80be6b0 2023-10-24T23:57:29.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f8a0078c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f80c10c0 con 0x7f89f80c70e0 2023-10-24T23:57:29.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f8a0078c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f80c27e0 con 0x7f89f8156770 2023-10-24T23:57:29.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fed29640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f89f80be6b0 0x7f89f815d130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32782/0 (socket says 172.21.15.107:32782) 2023-10-24T23:57:29.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fed29640 1 -- 172.21.15.107:0/3622810412 learned_addr learned my addr 172.21.15.107:0/3622810412 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:29.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fe528640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8156770 0x7f89f80ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316093943 0 0) 0x7f89f80c1ca0 con 0x7f89f80be6b0 2023-10-24T23:57:29.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e4003530 con 0x7f89f80be6b0 2023-10-24T23:57:29.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671296004 0 0) 0x7f89e4003530 con 0x7f89f80be6b0 2023-10-24T23:57:29.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8156770 msgr2=0x7f89f80ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:29.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8156770 0x7f89f80ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 >> v1:172.21.15.110:6789/0 conn(0x7f89f80c70e0 legacy=0x7f89f80bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f80c13a0 con 0x7f89f80be6b0 2023-10-24T23:57:29.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f89f0001a00 con 0x7f89f80be6b0 2023-10-24T23:57:29.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 >> v1:172.21.15.107:6789/0 conn(0x7f89f80be6b0 legacy=0x7f89f815d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89e40039b0 0x7f89e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8156770 0x7f89e40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89e4004d10 0x7f89e4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e400bb20 con 0x7f89e4004d10 2023-10-24T23:57:29.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e400bc90 con 0x7f89e40039b0 2023-10-24T23:57:29.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e400be50 con 0x7f89f8156770 2023-10-24T23:57:29.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fed29640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89e4004d10 0x7f89e4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fe528640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89e40039b0 0x7f89e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fdd27640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8156770 0x7f89e40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fed29640 1 -- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8156770 msgr2=0x7f89e40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:29.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fed29640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8156770 0x7f89e40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fed29640 1 -- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89e40039b0 msgr2=0x7f89e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:29.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fed29640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89e40039b0 0x7f89e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fed29640 1 -- 172.21.15.107:0/3622810412 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e4003530 con 0x7f89e4004d10 2023-10-24T23:57:29.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fdd27640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8156770 0x7f89e40047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:29.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.266+0000 7f89fe528640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89e40039b0 0x7f89e4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:29.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f89fed29640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89e4004d10 0x7f89e4005190 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f89f80beb00 tx=0x7f89f001d2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a925ac29abffbb30 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:29.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f00072d0 con 0x7f89e4004d10 2023-10-24T23:57:29.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f00a8070 con 0x7f89e4004d10 2023-10-24T23:57:29.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f89fd526640 1 -- 172.21.15.107:0/3622810412 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f00076b0 con 0x7f89e4004d10 2023-10-24T23:57:29.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f8a0078c640 1 -- 172.21.15.107:0/3622810412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89e4004d10 msgr2=0x7f89e4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f8a0078c640 1 --2- 172.21.15.107:0/3622810412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89e4004d10 0x7f89e4005190 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f89f80beb00 tx=0x7f89f001d2b0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f89fed29640 1 -- 172.21.15.107:0/3622810412 reap_dead start 2023-10-24T23:57:29.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f8a0078c640 1 -- 172.21.15.107:0/3622810412 shutdown_connections 2023-10-24T23:57:29.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f8a0078c640 1 -- 172.21.15.107:0/3622810412 >> 172.21.15.107:0/3622810412 conn(0x7f89f80c65c0 msgr2=0x7f89f80c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:29.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f8a0078c640 1 -- 172.21.15.107:0/3622810412 shutdown_connections 2023-10-24T23:57:29.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f8a0078c640 1 -- 172.21.15.107:0/3622810412 wait complete. 2023-10-24T23:57:29.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f8a0078c640 1 Processor -- start 2023-10-24T23:57:29.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.270+0000 7f8a0078c640 1 -- start start 2023-10-24T23:57:29.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f8a0078c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89f8071790 0x7f89f81eefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f8a0078c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f81df600 0x7f89f81515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f8a0078c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8151ae0 0x7f89f81e13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f8a0078c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f81e27a0 con 0x7f89f81df600 2023-10-24T23:57:29.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f8a0078c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f8072590 con 0x7f89f8071790 2023-10-24T23:57:29.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f8a0078c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f8151f60 con 0x7f89f8151ae0 2023-10-24T23:57:29.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fdd27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f81df600 0x7f89f81515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fdd27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f81df600 0x7f89f81515a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49394/0 (socket says 172.21.15.107:49394) 2023-10-24T23:57:29.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fdd27640 1 -- 172.21.15.107:0/1167194318 learned_addr learned my addr 172.21.15.107:0/1167194318 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:29.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fe528640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89f8071790 0x7f89f81eefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fed29640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8151ae0 0x7f89f81e13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fdd27640 1 -- 172.21.15.107:0/1167194318 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8151ae0 msgr2=0x7f89f81e13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fdd27640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8151ae0 0x7f89f81e13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fdd27640 1 -- 172.21.15.107:0/1167194318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89f8071790 msgr2=0x7f89f81eefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fdd27640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89f8071790 0x7f89f81eefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fdd27640 1 -- 172.21.15.107:0/1167194318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f81ef4e0 con 0x7f89f81df600 2023-10-24T23:57:29.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fdd27640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f81df600 0x7f89f81515a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f89e800d770 tx=0x7f89e800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:29.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89eeffd640 1 -- 172.21.15.107:0/1167194318 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e8005270 con 0x7f89f81df600 2023-10-24T23:57:29.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89eeffd640 1 -- 172.21.15.107:0/1167194318 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e80053d0 con 0x7f89f81df600 2023-10-24T23:57:29.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f8a0078c640 1 -- 172.21.15.107:0/1167194318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f81e09a0 con 0x7f89f81df600 2023-10-24T23:57:29.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89eeffd640 1 -- 172.21.15.107:0/1167194318 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e8007270 con 0x7f89f81df600 2023-10-24T23:57:29.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f8a0078c640 1 -- 172.21.15.107:0/1167194318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f81e0f60 con 0x7f89f81df600 2023-10-24T23:57:29.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f8a0078c640 1 -- 172.21.15.107:0/1167194318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f81e10f0 con 0x7f89f81df600 2023-10-24T23:57:29.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89eeffd640 1 -- 172.21.15.107:0/1167194318 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f89e80073d0 con 0x7f89f81df600 2023-10-24T23:57:29.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89eeffd640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f89dc065e90 0x7f89dc068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89eeffd640 1 -- 172.21.15.107:0/1167194318 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f89e8083490 con 0x7f89f81df600 2023-10-24T23:57:29.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fe528640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f89dc065e90 0x7f89dc068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.274+0000 7f89fe528640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f89dc065e90 0x7f89dc068350 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009ac0 tx=0x7f89f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:29.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.278+0000 7f89eeffd640 1 -- 172.21.15.107:0/1167194318 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f89e80505e0 con 0x7f89f81df600 2023-10-24T23:57:29.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.470+0000 7f8a0078c640 1 -- 172.21.15.107:0/1167194318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89f81ef4e0 con 0x7f89f81df600 2023-10-24T23:57:29.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.470+0000 7f89eeffd640 1 -- 172.21.15.107:0/1167194318 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4254 (secure 0 0 0) 0x7f89f81ef4e0 con 0x7f89f81df600 2023-10-24T23:57:29.476 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:29.476 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:28.407918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:29.477 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 38 2023-10-24T23:57:29.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 -- 172.21.15.107:0/1167194318 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f89dc065e90 msgr2=0x7f89dc068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f89dc065e90 0x7f89dc068350 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f89f4009ac0 tx=0x7f89f4008040 comp rx=0 tx=0).stop 2023-10-24T23:57:29.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 -- 172.21.15.107:0/1167194318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f81df600 msgr2=0x7f89f81515a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f81df600 0x7f89f81515a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f89e800d770 tx=0x7f89e800dc30 comp rx=0 tx=0).stop 2023-10-24T23:57:29.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 -- 172.21.15.107:0/1167194318 shutdown_connections 2023-10-24T23:57:29.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f89dc065e90 0x7f89dc068350 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89f8151ae0 0x7f89f81e13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89f81df600 0x7f89f81515a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 --2- 172.21.15.107:0/1167194318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89f8071790 0x7f89f81eefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 -- 172.21.15.107:0/1167194318 >> 172.21.15.107:0/1167194318 conn(0x7f89f80c65c0 msgr2=0x7f89f8154010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:29.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 -- 172.21.15.107:0/1167194318 shutdown_connections 2023-10-24T23:57:29.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.474+0000 7f8a0078c640 1 -- 172.21.15.107:0/1167194318 wait complete. 2023-10-24T23:57:29.495 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2023-10-24T23:57:29.495 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-24T23:57:29.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:29.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f89d9f640 1 Processor -- start 2023-10-24T23:57:29.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f89d9f640 1 -- start start 2023-10-24T23:57:29.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f89d9f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4f84071220 0x7f4f8406bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f89d9f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4f840c24c0 0x7f4f8406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f89d9f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f84150bc0 con 0x7f4f84151960 2023-10-24T23:57:29.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f89d9f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f8406c7f0 con 0x7f4f840c24c0 2023-10-24T23:57:29.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f89d9f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f8406c510 con 0x7f4f84071220 2023-10-24T23:57:29.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f83fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4f840c24c0 0x7f4f8406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f83fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4f840c24c0 0x7f4f8406a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49408/0 (socket says 172.21.15.107:49408) 2023-10-24T23:57:29.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f83fff640 1 -- 172.21.15.107:0/1658946485 learned_addr learned my addr 172.21.15.107:0/1658946485 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:29.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f837fe640 1 --2- 172.21.15.107:0/1658946485 >> v2:172.21.15.110:3300/0 conn(0x7f4f84071220 0x7f4f8406bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f83fff640 1 -- 172.21.15.107:0/1658946485 >> v2:172.21.15.110:3300/0 conn(0x7f4f84071220 msgr2=0x7f4f8406bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:29.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f83fff640 1 --2- 172.21.15.107:0/1658946485 >> v2:172.21.15.110:3300/0 conn(0x7f4f84071220 0x7f4f8406bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f83fff640 1 -- 172.21.15.107:0/1658946485 >> v1:172.21.15.110:6789/0 conn(0x7f4f84151960 legacy=0x7f4f84071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f83fff640 1 -- 172.21.15.107:0/1658946485 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f84150e80 con 0x7f4f840c24c0 2023-10-24T23:57:29.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f83fff640 1 --2- 172.21.15.107:0/1658946485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f840c24c0 0x7f4f8406a9c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4f8406d0a0 tx=0x7f4f7c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483173a92c7c3a80 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:29.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.610+0000 7f4f827fc640 1 -- 172.21.15.107:0/1658946485 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0a5580 con 0x7f4f840c24c0 2023-10-24T23:57:29.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f827fc640 1 -- 172.21.15.107:0/1658946485 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7c0a56e0 con 0x7f4f840c24c0 2023-10-24T23:57:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f827fc640 1 -- 172.21.15.107:0/1658946485 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0a5a70 con 0x7f4f840c24c0 2023-10-24T23:57:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 -- 172.21.15.107:0/1658946485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f840c24c0 msgr2=0x7f4f8406a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 --2- 172.21.15.107:0/1658946485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f840c24c0 0x7f4f8406a9c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4f8406d0a0 tx=0x7f4f7c01c940 comp rx=0 tx=0).stop 2023-10-24T23:57:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 -- 172.21.15.107:0/1658946485 shutdown_connections 2023-10-24T23:57:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 --2- 172.21.15.107:0/1658946485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f840c24c0 0x7f4f8406a9c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 --2- 172.21.15.107:0/1658946485 >> v2:172.21.15.110:3300/0 conn(0x7f4f84071220 0x7f4f8406bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 -- 172.21.15.107:0/1658946485 >> 172.21.15.107:0/1658946485 conn(0x7f4f840c2950 msgr2=0x7f4f840c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 -- 172.21.15.107:0/1658946485 shutdown_connections 2023-10-24T23:57:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 -- 172.21.15.107:0/1658946485 wait complete. 2023-10-24T23:57:29.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 Processor -- start 2023-10-24T23:57:29.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 -- start start 2023-10-24T23:57:29.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f84071220 0x7f4f84143e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f840c24c0 0x7f4f8414f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f84151960 0x7f4f8414a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f84141fc0 con 0x7f4f840c24c0 2023-10-24T23:57:29.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8414ee10 con 0x7f4f84071220 2023-10-24T23:57:29.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f84143570 con 0x7f4f84151960 2023-10-24T23:57:29.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f840c24c0 0x7f4f8414f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f840c24c0 0x7f4f8414f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49416/0 (socket says 172.21.15.107:49416) 2023-10-24T23:57:29.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f82ffd640 1 -- 172.21.15.107:0/4275821013 learned_addr learned my addr 172.21.15.107:0/4275821013 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:29.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f837fe640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f84071220 0x7f4f84143e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f83fff640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f84151960 0x7f4f8414a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f82ffd640 1 -- 172.21.15.107:0/4275821013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f84151960 msgr2=0x7f4f8414a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f82ffd640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f84151960 0x7f4f8414a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f82ffd640 1 -- 172.21.15.107:0/4275821013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f84071220 msgr2=0x7f4f84143e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f82ffd640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f84071220 0x7f4f84143e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f82ffd640 1 -- 172.21.15.107:0/4275821013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7c009b30 con 0x7f4f840c24c0 2023-10-24T23:57:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f82ffd640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f840c24c0 0x7f4f8414f530 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c00c3c0 tx=0x7f4f6c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:29.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f7bfff640 1 -- 172.21.15.107:0/4275821013 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c0052d0 con 0x7f4f840c24c0 2023-10-24T23:57:29.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f7bfff640 1 -- 172.21.15.107:0/4275821013 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6c005430 con 0x7f4f840c24c0 2023-10-24T23:57:29.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 -- 172.21.15.107:0/4275821013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8414fa70 con 0x7f4f840c24c0 2023-10-24T23:57:29.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f7bfff640 1 -- 172.21.15.107:0/4275821013 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c003840 con 0x7f4f840c24c0 2023-10-24T23:57:29.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.614+0000 7f4f89d9f640 1 -- 172.21.15.107:0/4275821013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8414a710 con 0x7f4f840c24c0 2023-10-24T23:57:29.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.618+0000 7f4f89d9f640 1 -- 172.21.15.107:0/4275821013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f84143720 con 0x7f4f840c24c0 2023-10-24T23:57:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.618+0000 7f4f7bfff640 1 -- 172.21.15.107:0/4275821013 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4f6c0039a0 con 0x7f4f840c24c0 2023-10-24T23:57:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.618+0000 7f4f7bfff640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4f64065e90 0x7f4f64068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.618+0000 7f4f7bfff640 1 -- 172.21.15.107:0/4275821013 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f4f6c083170 con 0x7f4f840c24c0 2023-10-24T23:57:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.618+0000 7f4f837fe640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4f64065e90 0x7f4f64068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.618+0000 7f4f837fe640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4f64065e90 0x7f4f64068350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009ac0 tx=0x7f4f74009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:29.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.622+0000 7f4f7bfff640 1 -- 172.21.15.107:0/4275821013 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4f84143720 con 0x7f4f840c24c0 2023-10-24T23:57:29.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.810+0000 7f4f89d9f640 1 -- 172.21.15.107:0/4275821013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f841436e0 con 0x7f4f840c24c0 2023-10-24T23:57:29.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.810+0000 7f4f7bfff640 1 -- 172.21.15.107:0/4275821013 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4254 (secure 0 0 0) 0x7f4f6c050370 con 0x7f4f840c24c0 2023-10-24T23:57:29.815 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:29.815 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:28.407918+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:29.816 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 38 2023-10-24T23:57:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.810+0000 7f4f89d9f640 1 -- 172.21.15.107:0/4275821013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4f64065e90 msgr2=0x7f4f64068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.810+0000 7f4f89d9f640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4f64065e90 0x7f4f64068350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009ac0 tx=0x7f4f74009190 comp rx=0 tx=0).stop 2023-10-24T23:57:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.810+0000 7f4f89d9f640 1 -- 172.21.15.107:0/4275821013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f840c24c0 msgr2=0x7f4f8414f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.810+0000 7f4f89d9f640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f840c24c0 0x7f4f8414f530 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c00c3c0 tx=0x7f4f6c00c880 comp rx=0 tx=0).stop 2023-10-24T23:57:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.810+0000 7f4f89d9f640 1 -- 172.21.15.107:0/4275821013 shutdown_connections 2023-10-24T23:57:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.810+0000 7f4f89d9f640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4f64065e90 0x7f4f64068350 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.810+0000 7f4f89d9f640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4f84151960 0x7f4f8414a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.810+0000 7f4f89d9f640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f840c24c0 0x7f4f8414f530 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.814+0000 7f4f89d9f640 1 --2- 172.21.15.107:0/4275821013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4f84071220 0x7f4f84143e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:29.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.814+0000 7f4f89d9f640 1 -- 172.21.15.107:0/4275821013 >> 172.21.15.107:0/4275821013 conn(0x7f4f840c2950 msgr2=0x7f4f841542c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:29.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.814+0000 7f4f89d9f640 1 -- 172.21.15.107:0/4275821013 shutdown_connections 2023-10-24T23:57:29.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:29.814+0000 7f4f89d9f640 1 -- 172.21.15.107:0/4275821013 wait complete. 2023-10-24T23:57:29.834 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 38, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4232": { "addr": "172.21.15.110:6834/3721101226", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 3721101226, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 3721101226, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4232, "incarnation": 38, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:57:28.407918+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4232 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5562": { "addr": "172.21.15.107:6837/204494054", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 204494054, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 204494054, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5562, "incarnation": 34, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-24T23:57:26.574492+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5562 } } } ], "standbys": [] } 2023-10-24T23:57:31.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:31.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.962+0000 7f6df7aa8640 1 Processor -- start 2023-10-24T23:57:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.962+0000 7f6df7aa8640 1 -- start start 2023-10-24T23:57:31.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df7aa8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6df00c8100 0x7f6df00abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:31.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df7aa8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df0155680 con 0x7f6df006a4b0 2023-10-24T23:57:31.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df7aa8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df0154a70 con 0x7f6df0152040 2023-10-24T23:57:31.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df7aa8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6df0154d50 con 0x7f6df00c8100 2023-10-24T23:57:31.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df6045640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6df006a4b0 0x7f6df00ac240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:32784/0 (socket says 172.21.15.107:32784) 2023-10-24T23:57:31.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df6045640 1 -- 172.21.15.107:0/3937696001 learned_addr learned my addr 172.21.15.107:0/3937696001 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:31.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667410084 0 0) 0x7f6df0155680 con 0x7f6df006a4b0 2023-10-24T23:57:31.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ddc003530 con 0x7f6df006a4b0 2023-10-24T23:57:31.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df5043640 1 --2- 172.21.15.107:0/3937696001 >> v2:172.21.15.110:3300/0 conn(0x7f6df00c8100 0x7f6df00abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:31.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2994791259 0 0) 0x7f6ddc003530 con 0x7f6df006a4b0 2023-10-24T23:57:31.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 >> v2:172.21.15.110:3300/0 conn(0x7f6df00c8100 msgr2=0x7f6df00abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:31.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 --2- 172.21.15.107:0/3937696001 >> v2:172.21.15.110:3300/0 conn(0x7f6df00c8100 0x7f6df00abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:31.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 >> v1:172.21.15.110:6789/0 conn(0x7f6df0152040 legacy=0x7f6df00c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:31.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df0155030 con 0x7f6df006a4b0 2023-10-24T23:57:31.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6dec001680 con 0x7f6df006a4b0 2023-10-24T23:57:31.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 >> v1:172.21.15.107:6789/0 conn(0x7f6df006a4b0 legacy=0x7f6df00ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:31.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 --2- 172.21.15.107:0/3937696001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ddc003980 0x7f6ddc003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:31.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 --2- 172.21.15.107:0/3937696001 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ddc0047d0 0x7f6ddc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:31.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 --2- 172.21.15.107:0/3937696001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 0x7f6ddc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:31.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df0154d50 con 0x7f6ddc0052b0 2023-10-24T23:57:31.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc00c160 con 0x7f6ddc003980 2023-10-24T23:57:31.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc00c320 con 0x7f6ddc0047d0 2023-10-24T23:57:31.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df6045640 1 --2- 172.21.15.107:0/3937696001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 0x7f6ddc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:31.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df6045640 1 -- 172.21.15.107:0/3937696001 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ddc0047d0 msgr2=0x7f6ddc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df6045640 1 --2- 172.21.15.107:0/3937696001 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ddc0047d0 0x7f6ddc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df6045640 1 -- 172.21.15.107:0/3937696001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ddc003980 msgr2=0x7f6ddc003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df6045640 1 --2- 172.21.15.107:0/3937696001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ddc003980 0x7f6ddc003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df6045640 1 -- 172.21.15.107:0/3937696001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc003530 con 0x7f6ddc0052b0 2023-10-24T23:57:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df6045640 1 -- 172.21.15.107:0/3937696001 reap_dead start 2023-10-24T23:57:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df6045640 1 --2- 172.21.15.107:0/3937696001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 0x7f6ddc00b990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6dec001ec0 tx=0x7f6dec01f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e9c0ad5d2bead0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec005ae0 con 0x7f6ddc0052b0 2023-10-24T23:57:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dec0ab070 con 0x7f6ddc0052b0 2023-10-24T23:57:31.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.966+0000 7f6df4842640 1 -- 172.21.15.107:0/3937696001 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec005ea0 con 0x7f6ddc0052b0 2023-10-24T23:57:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 -- 172.21.15.107:0/3937696001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 msgr2=0x7f6ddc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 --2- 172.21.15.107:0/3937696001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 0x7f6ddc00b990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6dec001ec0 tx=0x7f6dec01f390 comp rx=0 tx=0).stop 2023-10-24T23:57:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 -- 172.21.15.107:0/3937696001 shutdown_connections 2023-10-24T23:57:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 --2- 172.21.15.107:0/3937696001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 0x7f6ddc00b990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 -- 172.21.15.107:0/3937696001 >> 172.21.15.107:0/3937696001 conn(0x7f6df00ca200 msgr2=0x7f6df00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 -- 172.21.15.107:0/3937696001 shutdown_connections 2023-10-24T23:57:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 -- 172.21.15.107:0/3937696001 wait complete. 2023-10-24T23:57:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 Processor -- start 2023-10-24T23:57:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 -- start start 2023-10-24T23:57:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 0x7f6df014c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6df00ac7c0 0x7f6df00c1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6df01dd370 0x7f6df01dd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df014ce90 con 0x7f6ddc0052b0 2023-10-24T23:57:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df01de590 con 0x7f6df01dd370 2023-10-24T23:57:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df00c1ac0 con 0x7f6df00ac7c0 2023-10-24T23:57:31.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df5844640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 0x7f6df014c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:31.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df5844640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 0x7f6df014c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49434/0 (socket says 172.21.15.107:49434) 2023-10-24T23:57:31.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df5844640 1 -- 172.21.15.107:0/2737790212 learned_addr learned my addr 172.21.15.107:0/2737790212 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:31.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df5043640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6df00ac7c0 0x7f6df00c1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:31.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df5844640 1 -- 172.21.15.107:0/2737790212 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6df00ac7c0 msgr2=0x7f6df00c1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:31.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df5844640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6df00ac7c0 0x7f6df00c1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df5844640 1 -- 172.21.15.107:0/2737790212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6df01dd370 msgr2=0x7f6df01dd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df6045640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6df01dd370 0x7f6df01dd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df5844640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6df01dd370 0x7f6df01dd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df5844640 1 -- 172.21.15.107:0/2737790212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df00c1da0 con 0x7f6ddc0052b0 2023-10-24T23:57:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df5844640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 0x7f6df014c950 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6de800c780 tx=0x7f6de800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:31.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6de67fc640 1 -- 172.21.15.107:0/2737790212 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de8004b70 con 0x7f6ddc0052b0 2023-10-24T23:57:31.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6de67fc640 1 -- 172.21.15.107:0/2737790212 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de800ce90 con 0x7f6ddc0052b0 2023-10-24T23:57:31.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 -- 172.21.15.107:0/2737790212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df00c2080 con 0x7f6ddc0052b0 2023-10-24T23:57:31.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6de67fc640 1 -- 172.21.15.107:0/2737790212 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de800a7f0 con 0x7f6ddc0052b0 2023-10-24T23:57:31.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.970+0000 7f6df7aa8640 1 -- 172.21.15.107:0/2737790212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df00c2640 con 0x7f6ddc0052b0 2023-10-24T23:57:31.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.974+0000 7f6df7aa8640 1 -- 172.21.15.107:0/2737790212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df00746e0 con 0x7f6ddc0052b0 2023-10-24T23:57:31.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.974+0000 7f6de67fc640 1 -- 172.21.15.107:0/2737790212 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6de800a950 con 0x7f6ddc0052b0 2023-10-24T23:57:31.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.974+0000 7f6de67fc640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dd4065e60 0x7f6dd4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:31.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.974+0000 7f6de67fc640 1 -- 172.21.15.107:0/2737790212 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f6de8083ec0 con 0x7f6ddc0052b0 2023-10-24T23:57:31.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.974+0000 7f6df5043640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dd4065e60 0x7f6dd4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:31.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.974+0000 7f6df5043640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dd4065e60 0x7f6dd4068320 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6df01de130 tx=0x7f6de0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:31.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:31.978+0000 7f6de67fc640 1 -- 172.21.15.107:0/2737790212 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6de8051010 con 0x7f6ddc0052b0 2023-10-24T23:57:32.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.166+0000 7f6df7aa8640 1 -- 172.21.15.107:0/2737790212 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6df00ac190 con 0x7f6ddc0052b0 2023-10-24T23:57:32.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6de67fc640 1 -- 172.21.15.107:0/2737790212 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4255 (secure 0 0 0) 0x7f6de8054630 con 0x7f6ddc0052b0 2023-10-24T23:57:32.174 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:32.174 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:31.610439+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:rejoin","state_seq":70,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:32.174 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 40 2023-10-24T23:57:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 -- 172.21.15.107:0/2737790212 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dd4065e60 msgr2=0x7f6dd4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dd4065e60 0x7f6dd4068320 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6df01de130 tx=0x7f6de0008040 comp rx=0 tx=0).stop 2023-10-24T23:57:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 -- 172.21.15.107:0/2737790212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 msgr2=0x7f6df014c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 0x7f6df014c950 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6de800c780 tx=0x7f6de800cc40 comp rx=0 tx=0).stop 2023-10-24T23:57:32.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 -- 172.21.15.107:0/2737790212 shutdown_connections 2023-10-24T23:57:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6df01dd370 0x7f6df01dd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dd4065e60 0x7f6dd4068320 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6df00ac7c0 0x7f6df00c1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 --2- 172.21.15.107:0/2737790212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ddc0052b0 0x7f6df014c950 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:32.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 -- 172.21.15.107:0/2737790212 >> 172.21.15.107:0/2737790212 conn(0x7f6df00ca200 msgr2=0x7f6df0153d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 -- 172.21.15.107:0/2737790212 shutdown_connections 2023-10-24T23:57:32.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:32.170+0000 7f6df7aa8640 1 -- 172.21.15.107:0/2737790212 wait complete. 2023-10-24T23:57:32.194 INFO:tasks.mds_thrash.fs.[a]:no change 2023-10-24T23:57:34.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:34.294 INFO:tasks.ceph.mds.b.smithi110.stderr:2023-10-24T23:57:34.288+0000 7f73bc8cb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:57:34.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.326+0000 7f30a85f2640 1 Processor -- start 2023-10-24T23:57:34.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a85f2640 1 -- start start 2023-10-24T23:57:34.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a85f2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f30a0152040 0x7f30a00c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a85f2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30a00c8100 0x7f30a00abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a85f2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a0155680 con 0x7f30a006a4b0 2023-10-24T23:57:34.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a85f2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a0154a70 con 0x7f30a00c8100 2023-10-24T23:57:34.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a85f2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a0154d50 con 0x7f30a0152040 2023-10-24T23:57:34.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a5b8d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30a00c8100 0x7f30a00abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a5b8d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30a00c8100 0x7f30a00abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36598/0 (socket says 172.21.15.107:36598) 2023-10-24T23:57:34.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a5b8d640 1 -- 172.21.15.107:0/419458369 learned_addr learned my addr 172.21.15.107:0/419458369 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:34.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a638e640 1 --2- 172.21.15.107:0/419458369 >> v2:172.21.15.110:3300/0 conn(0x7f30a0152040 0x7f30a00c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a5b8d640 1 -- 172.21.15.107:0/419458369 >> v2:172.21.15.110:3300/0 conn(0x7f30a0152040 msgr2=0x7f30a00c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:34.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a5b8d640 1 --2- 172.21.15.107:0/419458369 >> v2:172.21.15.110:3300/0 conn(0x7f30a0152040 0x7f30a00c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a5b8d640 1 -- 172.21.15.107:0/419458369 >> v1:172.21.15.110:6789/0 conn(0x7f30a006a4b0 legacy=0x7f30a00ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a5b8d640 1 -- 172.21.15.107:0/419458369 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a0155030 con 0x7f30a00c8100 2023-10-24T23:57:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a5b8d640 1 --2- 172.21.15.107:0/419458369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a00c8100 0x7f30a00abd00 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f30a0067ba0 tx=0x7f309001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56abd93ffdd9ec00 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a538c640 1 -- 172.21.15.107:0/419458369 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30900ad680 con 0x7f30a00c8100 2023-10-24T23:57:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a538c640 1 -- 172.21.15.107:0/419458369 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30900ad7e0 con 0x7f30a00c8100 2023-10-24T23:57:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.330+0000 7f30a538c640 1 -- 172.21.15.107:0/419458369 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30900adb70 con 0x7f30a00c8100 2023-10-24T23:57:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 -- 172.21.15.107:0/419458369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a00c8100 msgr2=0x7f30a00abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 --2- 172.21.15.107:0/419458369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a00c8100 0x7f30a00abd00 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f30a0067ba0 tx=0x7f309001c470 comp rx=0 tx=0).stop 2023-10-24T23:57:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 -- 172.21.15.107:0/419458369 shutdown_connections 2023-10-24T23:57:34.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 --2- 172.21.15.107:0/419458369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a00c8100 0x7f30a00abd00 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 --2- 172.21.15.107:0/419458369 >> v2:172.21.15.110:3300/0 conn(0x7f30a0152040 0x7f30a00c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 -- 172.21.15.107:0/419458369 >> 172.21.15.107:0/419458369 conn(0x7f30a00ca200 msgr2=0x7f30a00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:34.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 -- 172.21.15.107:0/419458369 shutdown_connections 2023-10-24T23:57:34.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 -- 172.21.15.107:0/419458369 wait complete. 2023-10-24T23:57:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 Processor -- start 2023-10-24T23:57:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 -- start start 2023-10-24T23:57:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30a006a4b0 0x7f30a00a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a00c8100 0x7f30a00aaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0152040 0x7f30a01477a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a00ab4f0 con 0x7f30a0152040 2023-10-24T23:57:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a014bb80 con 0x7f30a00c8100 2023-10-24T23:57:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a85f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a00aa2c0 con 0x7f30a006a4b0 2023-10-24T23:57:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a6b8f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0152040 0x7f30a01477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a6b8f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0152040 0x7f30a01477a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36608/0 (socket says 172.21.15.107:36608) 2023-10-24T23:57:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a6b8f640 1 -- 172.21.15.107:0/1121021277 learned_addr learned my addr 172.21.15.107:0/1121021277 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a638e640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30a006a4b0 0x7f30a00a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a5b8d640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a00c8100 0x7f30a00aaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a6b8f640 1 -- 172.21.15.107:0/1121021277 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30a006a4b0 msgr2=0x7f30a00a9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a6b8f640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30a006a4b0 0x7f30a00a9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a6b8f640 1 -- 172.21.15.107:0/1121021277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a00c8100 msgr2=0x7f30a00aaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a6b8f640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a00c8100 0x7f30a00aaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a6b8f640 1 -- 172.21.15.107:0/1121021277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30900094d0 con 0x7f30a0152040 2023-10-24T23:57:34.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a638e640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30a006a4b0 0x7f30a00a9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:57:34.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f30a6b8f640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0152040 0x7f30a01477a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f309800c730 tx=0x7f309800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:34.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.334+0000 7f3096ffd640 1 -- 172.21.15.107:0/1121021277 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3098003b80 con 0x7f30a0152040 2023-10-24T23:57:34.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.338+0000 7f3096ffd640 1 -- 172.21.15.107:0/1121021277 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3098003ce0 con 0x7f30a0152040 2023-10-24T23:57:34.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.338+0000 7f30a85f2640 1 -- 172.21.15.107:0/1121021277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a0149580 con 0x7f30a0152040 2023-10-24T23:57:34.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.338+0000 7f3096ffd640 1 -- 172.21.15.107:0/1121021277 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309800aa30 con 0x7f30a0152040 2023-10-24T23:57:34.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.338+0000 7f30a85f2640 1 -- 172.21.15.107:0/1121021277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a0149b40 con 0x7f30a0152040 2023-10-24T23:57:34.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.338+0000 7f30a85f2640 1 -- 172.21.15.107:0/1121021277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a014bcf0 con 0x7f30a0152040 2023-10-24T23:57:34.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.338+0000 7f3096ffd640 1 -- 172.21.15.107:0/1121021277 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f309800ab90 con 0x7f30a0152040 2023-10-24T23:57:34.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.338+0000 7f3096ffd640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3084065e90 0x7f3084068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.338+0000 7f3096ffd640 1 -- 172.21.15.107:0/1121021277 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f3098083ad0 con 0x7f30a0152040 2023-10-24T23:57:34.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.338+0000 7f30a638e640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3084065e90 0x7f3084068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.338+0000 7f30a638e640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3084065e90 0x7f3084068350 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f309c00b9f0 tx=0x7f309c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:34.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.342+0000 7f3096ffd640 1 -- 172.21.15.107:0/1121021277 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f30a014bcf0 con 0x7f30a0152040 2023-10-24T23:57:34.544 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2023-10-24T23:57:34.545 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-24T23:57:34.545 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-24T23:57:34.548 INFO:tasks.ceph.mds.d:Started 2023-10-24T23:57:34.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:34.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.582+0000 7f30a85f2640 1 -- 172.21.15.107:0/1121021277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30a014bcf0 con 0x7f30a0152040 2023-10-24T23:57:34.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.582+0000 7f3096ffd640 1 -- 172.21.15.107:0/1121021277 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4255 (secure 0 0 0) 0x7f30a014bcf0 con 0x7f30a0152040 2023-10-24T23:57:34.589 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:34.589 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:34.589 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 41 2023-10-24T23:57:34.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 -- 172.21.15.107:0/1121021277 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3084065e90 msgr2=0x7f3084068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3084065e90 0x7f3084068350 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f309c00b9f0 tx=0x7f309c008040 comp rx=0 tx=0).stop 2023-10-24T23:57:34.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 -- 172.21.15.107:0/1121021277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0152040 msgr2=0x7f30a01477a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0152040 0x7f30a01477a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f309800c730 tx=0x7f309800cbf0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 -- 172.21.15.107:0/1121021277 shutdown_connections 2023-10-24T23:57:34.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0152040 0x7f30a01477a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30a00c8100 0x7f30a00aaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3084065e90 0x7f3084068350 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 --2- 172.21.15.107:0/1121021277 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30a006a4b0 0x7f30a00a9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 -- 172.21.15.107:0/1121021277 >> 172.21.15.107:0/1121021277 conn(0x7f30a00ca200 msgr2=0x7f30a00ae3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:34.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 -- 172.21.15.107:0/1121021277 shutdown_connections 2023-10-24T23:57:34.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.586+0000 7f30a85f2640 1 -- 172.21.15.107:0/1121021277 wait complete. 2023-10-24T23:57:34.613 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=5403 2023-10-24T23:57:34.613 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before reviving mds.c 2023-10-24T23:57:34.626 INFO:tasks.ceph.mds.d.smithi110.stdout:starting mds.d at 2023-10-24T23:57:34.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f3674f37640 1 Processor -- start 2023-10-24T23:57:34.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f3674f37640 1 -- start start 2023-10-24T23:57:34.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f3674f37640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3670152040 0x7f36700c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f3674f37640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36700c8100 0x7f36700abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f3674f37640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3670155680 con 0x7f367006a4b0 2023-10-24T23:57:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f3674f37640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3670154a70 con 0x7f36700c8100 2023-10-24T23:57:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f3674f37640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3670154d50 con 0x7f3670152040 2023-10-24T23:57:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f366dd9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36700c8100 0x7f36700abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f366ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f367006a4b0 0x7f36700ac240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59440/0 (socket says 172.21.15.107:59440) 2023-10-24T23:57:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f366dd9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36700c8100 0x7f36700abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36620/0 (socket says 172.21.15.107:36620) 2023-10-24T23:57:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f366ed9d640 1 -- 172.21.15.107:0/1060954608 learned_addr learned my addr 172.21.15.107:0/1060954608 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f366e59c640 1 --2- 172.21.15.107:0/1060954608 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3670152040 0x7f36700c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f366dd9b640 1 -- 172.21.15.107:0/1060954608 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3670152040 msgr2=0x7f36700c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f366dd9b640 1 --2- 172.21.15.107:0/1060954608 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3670152040 0x7f36700c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f366dd9b640 1 -- 172.21.15.107:0/1060954608 >> v1:172.21.15.107:6789/0 conn(0x7f367006a4b0 legacy=0x7f36700ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.670+0000 7f366dd9b640 1 -- 172.21.15.107:0/1060954608 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3670155030 con 0x7f36700c8100 2023-10-24T23:57:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f366dd9b640 1 --2- 172.21.15.107:0/1060954608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700c8100 0x7f36700abd00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f36640097c0 tx=0x7f366401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81f28944ac8d48b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f366d59a640 1 -- 172.21.15.107:0/1060954608 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640ab600 con 0x7f36700c8100 2023-10-24T23:57:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f366d59a640 1 -- 172.21.15.107:0/1060954608 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36640a7050 con 0x7f36700c8100 2023-10-24T23:57:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f366d59a640 1 -- 172.21.15.107:0/1060954608 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36640ab980 con 0x7f36700c8100 2023-10-24T23:57:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f3674f37640 1 -- 172.21.15.107:0/1060954608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700c8100 msgr2=0x7f36700abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f3674f37640 1 --2- 172.21.15.107:0/1060954608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700c8100 0x7f36700abd00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f36640097c0 tx=0x7f366401c3f0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f3674f37640 1 -- 172.21.15.107:0/1060954608 shutdown_connections 2023-10-24T23:57:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f3674f37640 1 --2- 172.21.15.107:0/1060954608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36700c8100 0x7f36700abd00 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f3674f37640 1 --2- 172.21.15.107:0/1060954608 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3670152040 0x7f36700c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f3674f37640 1 -- 172.21.15.107:0/1060954608 >> 172.21.15.107:0/1060954608 conn(0x7f36700ca200 msgr2=0x7f36700ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f3674f37640 1 -- 172.21.15.107:0/1060954608 shutdown_connections 2023-10-24T23:57:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f3674f37640 1 -- 172.21.15.107:0/1060954608 wait complete. 2023-10-24T23:57:34.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f3674f37640 1 Processor -- start 2023-10-24T23:57:34.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.674+0000 7f3674f37640 1 -- start start 2023-10-24T23:57:34.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f3674f37640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f367006a4b0 0x7f36701ee3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f3674f37640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36700c8100 0x7f367014cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f3674f37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670152040 0x7f36701ed980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f3674f37640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367014c0a0 con 0x7f3670152040 2023-10-24T23:57:34.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f3674f37640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3670151970 con 0x7f36700c8100 2023-10-24T23:57:34.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f3674f37640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367014ba70 con 0x7f367006a4b0 2023-10-24T23:57:34.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670152040 0x7f36701ed980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670152040 0x7f36701ed980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36624/0 (socket says 172.21.15.107:36624) 2023-10-24T23:57:34.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366ed9d640 1 -- 172.21.15.107:0/2459575760 learned_addr learned my addr 172.21.15.107:0/2459575760 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:34.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366e59c640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f367006a4b0 0x7f36701ee3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366dd9b640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36700c8100 0x7f367014cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366ed9d640 1 -- 172.21.15.107:0/2459575760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f367006a4b0 msgr2=0x7f36701ee3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366ed9d640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f367006a4b0 0x7f36701ee3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366ed9d640 1 -- 172.21.15.107:0/2459575760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36700c8100 msgr2=0x7f367014cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366ed9d640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36700c8100 0x7f367014cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366ed9d640 1 -- 172.21.15.107:0/2459575760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3664009420 con 0x7f3670152040 2023-10-24T23:57:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366e59c640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f367006a4b0 0x7f36701ee3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:57:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f366ed9d640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670152040 0x7f36701ed980 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f366800ca80 tx=0x7f366800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f365effd640 1 -- 172.21.15.107:0/2459575760 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366800ac30 con 0x7f3670152040 2023-10-24T23:57:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f365effd640 1 -- 172.21.15.107:0/2459575760 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366800ad90 con 0x7f3670152040 2023-10-24T23:57:34.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f3674f37640 1 -- 172.21.15.107:0/2459575760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36701edec0 con 0x7f3670152040 2023-10-24T23:57:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f365effd640 1 -- 172.21.15.107:0/2459575760 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3668012600 con 0x7f3670152040 2023-10-24T23:57:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f3674f37640 1 -- 172.21.15.107:0/2459575760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367014d7d0 con 0x7f3670152040 2023-10-24T23:57:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f3674f37640 1 -- 172.21.15.107:0/2459575760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367014c1e0 con 0x7f3670152040 2023-10-24T23:57:34.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.678+0000 7f365effd640 1 -- 172.21.15.107:0/2459575760 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3668007070 con 0x7f3670152040 2023-10-24T23:57:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.682+0000 7f365effd640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3650065e60 0x7f3650068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:34.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.682+0000 7f365effd640 1 -- 172.21.15.107:0/2459575760 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f3668083ba0 con 0x7f3670152040 2023-10-24T23:57:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.682+0000 7f366e59c640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3650065e60 0x7f3650068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.682+0000 7f366e59c640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3650065e60 0x7f3650068320 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3660009f50 tx=0x7f3660009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:34.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.686+0000 7f365effd640 1 -- 172.21.15.107:0/2459575760 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f367014c1e0 con 0x7f3670152040 2023-10-24T23:57:34.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.922+0000 7f3674f37640 1 -- 172.21.15.107:0/2459575760 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f367014c1e0 con 0x7f3670152040 2023-10-24T23:57:34.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.922+0000 7f365effd640 1 -- 172.21.15.107:0/2459575760 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4255 (secure 0 0 0) 0x7f367014c1e0 con 0x7f3670152040 2023-10-24T23:57:34.928 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:34.928 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:34.929 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 41 2023-10-24T23:57:34.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 -- 172.21.15.107:0/2459575760 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3650065e60 msgr2=0x7f3650068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3650065e60 0x7f3650068320 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3660009f50 tx=0x7f3660009190 comp rx=0 tx=0).stop 2023-10-24T23:57:34.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 -- 172.21.15.107:0/2459575760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670152040 msgr2=0x7f36701ed980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:34.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670152040 0x7f36701ed980 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f366800ca80 tx=0x7f366800cf40 comp rx=0 tx=0).stop 2023-10-24T23:57:34.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 -- 172.21.15.107:0/2459575760 shutdown_connections 2023-10-24T23:57:34.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3670152040 0x7f36701ed980 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36700c8100 0x7f367014cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3650065e60 0x7f3650068320 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 --2- 172.21.15.107:0/2459575760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f367006a4b0 0x7f36701ee3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 -- 172.21.15.107:0/2459575760 >> 172.21.15.107:0/2459575760 conn(0x7f36700ca200 msgr2=0x7f36700ac1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 -- 172.21.15.107:0/2459575760 shutdown_connections 2023-10-24T23:57:34.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:34.926+0000 7f3674f37640 1 -- 172.21.15.107:0/2459575760 wait complete. 2023-10-24T23:57:34.948 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-24T23:57:35.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:57:35.115 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:57:36.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:37.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.090+0000 7f02afa16640 1 Processor -- start 2023-10-24T23:57:37.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.090+0000 7f02afa16640 1 -- start start 2023-10-24T23:57:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.090+0000 7f02afa16640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02a8071ce0 0x7f02a80ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.090+0000 7f02afa16640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f02a80720c0 0x7f02a813d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:37.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.090+0000 7f02afa16640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a806e710 con 0x7f02a8154860 2023-10-24T23:57:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.090+0000 7f02afa16640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02a806d900 con 0x7f02a80720c0 2023-10-24T23:57:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.090+0000 7f02afa16640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a8154d20 con 0x7f02a8071ce0 2023-10-24T23:57:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.090+0000 7f02acfb1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f02a80720c0 0x7f02a813d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.090+0000 7f02adfb3640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f02a8154860 0x7f02a8158230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59454/0 (socket says 172.21.15.107:59454) 2023-10-24T23:57:37.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.090+0000 7f02adfb3640 1 -- 172.21.15.107:0/1083827867 learned_addr learned my addr 172.21.15.107:0/1083827867 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.090+0000 7f02ad7b2640 1 --2- 172.21.15.107:0/1083827867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02a8071ce0 0x7f02a80ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02acfb1640 1 -- 172.21.15.107:0/1083827867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02a8071ce0 msgr2=0x7f02a80ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02acfb1640 1 --2- 172.21.15.107:0/1083827867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02a8071ce0 0x7f02a80ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02acfb1640 1 -- 172.21.15.107:0/1083827867 >> v1:172.21.15.107:6789/0 conn(0x7f02a8154860 legacy=0x7f02a8158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:37.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02acfb1640 1 -- 172.21.15.107:0/1083827867 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a8155000 con 0x7f02a80720c0 2023-10-24T23:57:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02acfb1640 1 --2- 172.21.15.107:0/1083827867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02a80720c0 0x7f02a813d970 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f02980097c0 tx=0x7f029801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa2c5bcbe3ea5ac server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f029ffff640 1 -- 172.21.15.107:0/1083827867 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02980ac620 con 0x7f02a80720c0 2023-10-24T23:57:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f029ffff640 1 -- 172.21.15.107:0/1083827867 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02980ac780 con 0x7f02a80720c0 2023-10-24T23:57:37.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f029ffff640 1 -- 172.21.15.107:0/1083827867 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02980acb10 con 0x7f02a80720c0 2023-10-24T23:57:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02afa16640 1 -- 172.21.15.107:0/1083827867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02a80720c0 msgr2=0x7f02a813d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02afa16640 1 --2- 172.21.15.107:0/1083827867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02a80720c0 0x7f02a813d970 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f02980097c0 tx=0x7f029801c3c0 comp rx=0 tx=0).stop 2023-10-24T23:57:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02afa16640 1 -- 172.21.15.107:0/1083827867 shutdown_connections 2023-10-24T23:57:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02afa16640 1 --2- 172.21.15.107:0/1083827867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02a80720c0 0x7f02a813d970 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:37.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02afa16640 1 --2- 172.21.15.107:0/1083827867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02a8071ce0 0x7f02a80ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02afa16640 1 -- 172.21.15.107:0/1083827867 >> 172.21.15.107:0/1083827867 conn(0x7f02a8070f60 msgr2=0x7f02a8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02afa16640 1 -- 172.21.15.107:0/1083827867 shutdown_connections 2023-10-24T23:57:37.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.094+0000 7f02afa16640 1 -- 172.21.15.107:0/1083827867 wait complete. 2023-10-24T23:57:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02afa16640 1 Processor -- start 2023-10-24T23:57:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02afa16640 1 -- start start 2023-10-24T23:57:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02afa16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02a8071ce0 0x7f02a806c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:37.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02afa16640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02a80720c0 0x7f02a80c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02afa16640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02a8154860 0x7f02a81ecc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02afa16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a813c510 con 0x7f02a8071ce0 2023-10-24T23:57:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02afa16640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a8157530 con 0x7f02a80720c0 2023-10-24T23:57:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02afa16640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a81f0c60 con 0x7f02a8154860 2023-10-24T23:57:37.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02ad7b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02a8071ce0 0x7f02a806c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02ad7b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02a8071ce0 0x7f02a806c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36646/0 (socket says 172.21.15.107:36646) 2023-10-24T23:57:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02ad7b2640 1 -- 172.21.15.107:0/2345991451 learned_addr learned my addr 172.21.15.107:0/2345991451 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02acfb1640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02a80720c0 0x7f02a80c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02adfb3640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02a8154860 0x7f02a81ecc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:37.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02ad7b2640 1 -- 172.21.15.107:0/2345991451 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02a8154860 msgr2=0x7f02a81ecc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02ad7b2640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02a8154860 0x7f02a81ecc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02ad7b2640 1 -- 172.21.15.107:0/2345991451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02a80720c0 msgr2=0x7f02a80c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02ad7b2640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02a80720c0 0x7f02a80c7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02ad7b2640 1 -- 172.21.15.107:0/2345991451 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0298009420 con 0x7f02a8071ce0 2023-10-24T23:57:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02ad7b2640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02a8071ce0 0x7f02a806c540 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f02a400d800 tx=0x7f02a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f029dffb640 1 -- 172.21.15.107:0/2345991451 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a400b6c0 con 0x7f02a8071ce0 2023-10-24T23:57:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f029dffb640 1 -- 172.21.15.107:0/2345991451 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a400b820 con 0x7f02a8071ce0 2023-10-24T23:57:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02afa16640 1 -- 172.21.15.107:0/2345991451 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a81f0f40 con 0x7f02a8071ce0 2023-10-24T23:57:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f029dffb640 1 -- 172.21.15.107:0/2345991451 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a400bb90 con 0x7f02a8071ce0 2023-10-24T23:57:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02afa16640 1 -- 172.21.15.107:0/2345991451 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a80c75b0 con 0x7f02a8071ce0 2023-10-24T23:57:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.098+0000 7f02afa16640 1 -- 172.21.15.107:0/2345991451 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a80746e0 con 0x7f02a8071ce0 2023-10-24T23:57:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.102+0000 7f029dffb640 1 -- 172.21.15.107:0/2345991451 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f02a400bcf0 con 0x7f02a8071ce0 2023-10-24T23:57:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.102+0000 7f029dffb640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f028c065e60 0x7f028c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.102+0000 7f029dffb640 1 -- 172.21.15.107:0/2345991451 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f02a4084360 con 0x7f02a8071ce0 2023-10-24T23:57:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.102+0000 7f02acfb1640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f028c065e60 0x7f028c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.102+0000 7f02acfb1640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f028c065e60 0x7f028c068320 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f02980ae000 tx=0x7f0298008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.106+0000 7f029dffb640 1 -- 172.21.15.107:0/2345991451 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f02a4051420 con 0x7f02a8071ce0 2023-10-24T23:57:37.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 -- 172.21.15.107:0/2345991451 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02a806f940 con 0x7f02a8071ce0 2023-10-24T23:57:37.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f029dffb640 1 -- 172.21.15.107:0/2345991451 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f02a4011340 con 0x7f02a8071ce0 2023-10-24T23:57:37.318 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:37.319 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:37.319 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 42 2023-10-24T23:57:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 -- 172.21.15.107:0/2345991451 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f028c065e60 msgr2=0x7f028c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f028c065e60 0x7f028c068320 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f02980ae000 tx=0x7f0298008040 comp rx=0 tx=0).stop 2023-10-24T23:57:37.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 -- 172.21.15.107:0/2345991451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02a8071ce0 msgr2=0x7f02a806c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:37.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02a8071ce0 0x7f02a806c540 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f02a400d800 tx=0x7f02a400dcc0 comp rx=0 tx=0).stop 2023-10-24T23:57:37.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 -- 172.21.15.107:0/2345991451 shutdown_connections 2023-10-24T23:57:37.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f028c065e60 0x7f028c068320 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:37.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02a8154860 0x7f02a81ecc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:37.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02a80720c0 0x7f02a80c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:37.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 --2- 172.21.15.107:0/2345991451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02a8071ce0 0x7f02a806c540 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:37.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 -- 172.21.15.107:0/2345991451 >> 172.21.15.107:0/2345991451 conn(0x7f02a8070f60 msgr2=0x7f02a80c37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:37.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 -- 172.21.15.107:0/2345991451 shutdown_connections 2023-10-24T23:57:37.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:37.314+0000 7f02afa16640 1 -- 172.21.15.107:0/2345991451 wait complete. 2023-10-24T23:57:37.341 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2023-10-24T23:57:37.342 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before thrashing 2023-10-24T23:57:42.614 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2023-10-24T23:57:42.615 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-24T23:57:42.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-24T23:57:42.617 INFO:tasks.ceph.mds.c:Started 2023-10-24T23:57:42.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:42.697 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-24T23:57:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.750+0000 7fa64598f640 1 Processor -- start 2023-10-24T23:57:42.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.750+0000 7fa64598f640 1 -- start start 2023-10-24T23:57:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.750+0000 7fa64598f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64006a770 0x7fa640152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.750+0000 7fa64598f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa64006d2d0 0x7fa640154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:42.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.750+0000 7fa64598f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64006d760 con 0x7fa64006cef0 2023-10-24T23:57:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.750+0000 7fa64598f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6400a9e60 con 0x7fa64006d2d0 2023-10-24T23:57:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.750+0000 7fa64598f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6400a99d0 con 0x7fa64006a770 2023-10-24T23:57:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa64006d2d0 0x7fa640154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:42.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa64006d2d0 0x7fa640154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57674/0 (socket says 172.21.15.107:57674) 2023-10-24T23:57:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63f7fe640 1 -- 172.21.15.107:0/3987903428 learned_addr learned my addr 172.21.15.107:0/3987903428 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63effd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa64006cef0 0x7fa640154350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59706/0 (socket says 172.21.15.107:59706) 2023-10-24T23:57:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63e7fc640 1 --2- 172.21.15.107:0/3987903428 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64006a770 0x7fa640152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63f7fe640 1 -- 172.21.15.107:0/3987903428 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64006a770 msgr2=0x7fa640152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:42.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63f7fe640 1 --2- 172.21.15.107:0/3987903428 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64006a770 0x7fa640152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:42.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63f7fe640 1 -- 172.21.15.107:0/3987903428 >> v1:172.21.15.107:6789/0 conn(0x7fa64006cef0 legacy=0x7fa640154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:42.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63f7fe640 1 -- 172.21.15.107:0/3987903428 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6400a9450 con 0x7fa64006d2d0 2023-10-24T23:57:42.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63dffb640 1 -- 172.21.15.107:0/3987903428 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502159870 0 0) 0x7fa630001120 con 0x7fa64006cef0 2023-10-24T23:57:42.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63f7fe640 1 --2- 172.21.15.107:0/3987903428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64006d2d0 0x7fa640154a60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa638009e50 tx=0x7fa63801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a754f90a5a02d3b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:42.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63dffb640 1 -- 172.21.15.107:0/3987903428 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6380a56a0 con 0x7fa64006d2d0 2023-10-24T23:57:42.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63dffb640 1 -- 172.21.15.107:0/3987903428 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6380a5800 con 0x7fa64006d2d0 2023-10-24T23:57:42.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa63dffb640 1 -- 172.21.15.107:0/3987903428 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6380a5b90 con 0x7fa64006d2d0 2023-10-24T23:57:42.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa64598f640 1 -- 172.21.15.107:0/3987903428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64006d2d0 msgr2=0x7fa640154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:42.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa64598f640 1 --2- 172.21.15.107:0/3987903428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64006d2d0 0x7fa640154a60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa638009e50 tx=0x7fa63801c8f0 comp rx=0 tx=0).stop 2023-10-24T23:57:42.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa64598f640 1 -- 172.21.15.107:0/3987903428 shutdown_connections 2023-10-24T23:57:42.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa64598f640 1 --2- 172.21.15.107:0/3987903428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64006d2d0 0x7fa640154a60 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:42.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa64598f640 1 --2- 172.21.15.107:0/3987903428 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64006a770 0x7fa640152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:42.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa64598f640 1 -- 172.21.15.107:0/3987903428 >> 172.21.15.107:0/3987903428 conn(0x7fa6400c6ed0 msgr2=0x7fa6400c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:42.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa64598f640 1 -- 172.21.15.107:0/3987903428 shutdown_connections 2023-10-24T23:57:42.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.754+0000 7fa64598f640 1 -- 172.21.15.107:0/3987903428 wait complete. 2023-10-24T23:57:42.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa64598f640 1 Processor -- start 2023-10-24T23:57:42.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa64598f640 1 -- start start 2023-10-24T23:57:42.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa64598f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64006a770 0x7fa640071170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:42.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa64598f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64006cef0 0x7fa6401ec6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:42.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa64598f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa64006d2d0 0x7fa6401f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:42.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa64598f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6400c5c80 con 0x7fa64006cef0 2023-10-24T23:57:42.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa64598f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa640071ec0 con 0x7fa64006d2d0 2023-10-24T23:57:42.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa64598f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6400716b0 con 0x7fa64006a770 2023-10-24T23:57:42.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64006cef0 0x7fa6401ec6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:42.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64006cef0 0x7fa6401ec6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57690/0 (socket says 172.21.15.107:57690) 2023-10-24T23:57:42.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63e7fc640 1 -- 172.21.15.107:0/2711278982 learned_addr learned my addr 172.21.15.107:0/2711278982 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:42.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63effd640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64006a770 0x7fa640071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:42.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63f7fe640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa64006d2d0 0x7fa6401f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:42.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63e7fc640 1 -- 172.21.15.107:0/2711278982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64006a770 msgr2=0x7fa640071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:42.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63e7fc640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64006a770 0x7fa640071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:42.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63e7fc640 1 -- 172.21.15.107:0/2711278982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa64006d2d0 msgr2=0x7fa6401f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:42.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63e7fc640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa64006d2d0 0x7fa6401f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:42.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63e7fc640 1 -- 172.21.15.107:0/2711278982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa638009b30 con 0x7fa64006cef0 2023-10-24T23:57:42.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63f7fe640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa64006d2d0 0x7fa6401f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:57:42.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa63e7fc640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64006cef0 0x7fa6401ec6a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa63400e870 tx=0x7fa63400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:42.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa627fff640 1 -- 172.21.15.107:0/2711278982 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa634009510 con 0x7fa64006cef0 2023-10-24T23:57:42.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa627fff640 1 -- 172.21.15.107:0/2711278982 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63400eec0 con 0x7fa64006cef0 2023-10-24T23:57:42.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa627fff640 1 -- 172.21.15.107:0/2711278982 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63400cb40 con 0x7fa64006cef0 2023-10-24T23:57:42.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa64598f640 1 -- 172.21.15.107:0/2711278982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa640071900 con 0x7fa64006cef0 2023-10-24T23:57:42.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.758+0000 7fa64598f640 1 -- 172.21.15.107:0/2711278982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6401f0060 con 0x7fa64006cef0 2023-10-24T23:57:42.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.762+0000 7fa64598f640 1 -- 172.21.15.107:0/2711278982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6400746e0 con 0x7fa64006cef0 2023-10-24T23:57:42.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.762+0000 7fa627fff640 1 -- 172.21.15.107:0/2711278982 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa63400cca0 con 0x7fa64006cef0 2023-10-24T23:57:42.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.762+0000 7fa627fff640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa62c065e90 0x7fa62c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:42.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.762+0000 7fa627fff640 1 -- 172.21.15.107:0/2711278982 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7fa634083d50 con 0x7fa64006cef0 2023-10-24T23:57:42.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.762+0000 7fa63effd640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa62c065e90 0x7fa62c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:42.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.762+0000 7fa63effd640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa62c065e90 0x7fa62c068350 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa630000f70 tx=0x7fa630007df0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:42.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:42.766+0000 7fa627fff640 1 -- 172.21.15.107:0/2711278982 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa634087030 con 0x7fa64006cef0 2023-10-24T23:57:43.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.026+0000 7fa64598f640 1 -- 172.21.15.107:0/2711278982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6400c19b0 con 0x7fa64006cef0 2023-10-24T23:57:43.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.026+0000 7fa627fff640 1 -- 172.21.15.107:0/2711278982 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5014 (secure 0 0 0) 0x7fa634050f50 con 0x7fa64006cef0 2023-10-24T23:57:43.033 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:43.033 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:43.033 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 42 2023-10-24T23:57:43.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 -- 172.21.15.107:0/2711278982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa62c065e90 msgr2=0x7fa62c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:43.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa62c065e90 0x7fa62c068350 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa630000f70 tx=0x7fa630007df0 comp rx=0 tx=0).stop 2023-10-24T23:57:43.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 -- 172.21.15.107:0/2711278982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64006cef0 msgr2=0x7fa6401ec6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:43.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64006cef0 0x7fa6401ec6a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa63400e870 tx=0x7fa63400ec70 comp rx=0 tx=0).stop 2023-10-24T23:57:43.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 -- 172.21.15.107:0/2711278982 shutdown_connections 2023-10-24T23:57:43.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa64006d2d0 0x7fa6401f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:43.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa64006cef0 0x7fa6401ec6a0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:43.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa62c065e90 0x7fa62c068350 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:43.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 --2- 172.21.15.107:0/2711278982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64006a770 0x7fa640071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:43.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 -- 172.21.15.107:0/2711278982 >> 172.21.15.107:0/2711278982 conn(0x7fa6400c6ed0 msgr2=0x7fa6400aae90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:43.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 -- 172.21.15.107:0/2711278982 shutdown_connections 2023-10-24T23:57:43.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:43.030+0000 7fa64598f640 1 -- 172.21.15.107:0/2711278982 wait complete. 2023-10-24T23:57:43.054 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-24T23:57:45.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:45.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.174+0000 7f97ed5b5640 1 Processor -- start 2023-10-24T23:57:45.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.174+0000 7f97ed5b5640 1 -- start start 2023-10-24T23:57:45.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.174+0000 7f97ed5b5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f97e814c470 0x7f97e814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.174+0000 7f97ed5b5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f97e806f490 0x7f97e8071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.174+0000 7f97ed5b5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97e8155cd0 con 0x7f97e8071fc0 2023-10-24T23:57:45.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.174+0000 7f97ed5b5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e81526e0 con 0x7f97e814c470 2023-10-24T23:57:45.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.174+0000 7f97ed5b5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97e80c4380 con 0x7f97e806f490 2023-10-24T23:57:45.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.174+0000 7f97e6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f97e814c470 0x7f97e814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.174+0000 7f97e77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f97e8071fc0 0x7f97e8152b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59720/0 (socket says 172.21.15.107:59720) 2023-10-24T23:57:45.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.174+0000 7f97e77fe640 1 -- 172.21.15.107:0/565361033 learned_addr learned my addr 172.21.15.107:0/565361033 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:45.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.174+0000 7f97e6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f97e814c470 0x7f97e814c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57702/0 (socket says 172.21.15.107:57702) 2023-10-24T23:57:45.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97e67fc640 1 --2- 172.21.15.107:0/565361033 >> v2:172.21.15.110:3300/0 conn(0x7f97e806f490 0x7f97e8071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97e6ffd640 1 -- 172.21.15.107:0/565361033 >> v2:172.21.15.110:3300/0 conn(0x7f97e806f490 msgr2=0x7f97e8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:45.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97e6ffd640 1 --2- 172.21.15.107:0/565361033 >> v2:172.21.15.110:3300/0 conn(0x7f97e806f490 0x7f97e8071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97e6ffd640 1 -- 172.21.15.107:0/565361033 >> v1:172.21.15.107:6789/0 conn(0x7f97e8071fc0 legacy=0x7f97e8152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97e6ffd640 1 -- 172.21.15.107:0/565361033 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e80c48b0 con 0x7f97e814c470 2023-10-24T23:57:45.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97e5ffb640 1 -- 172.21.15.107:0/565361033 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587623803 0 0) 0x7f97e8155cd0 con 0x7f97e8071fc0 2023-10-24T23:57:45.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97e6ffd640 1 --2- 172.21.15.107:0/565361033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97e814c470 0x7f97e814c850 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f97e806a7e0 tx=0x7f97d801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e558d711c1bfdcf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:45.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97e5ffb640 1 -- 172.21.15.107:0/565361033 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d80ab600 con 0x7f97e814c470 2023-10-24T23:57:45.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97e5ffb640 1 -- 172.21.15.107:0/565361033 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d80a7050 con 0x7f97e814c470 2023-10-24T23:57:45.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97e5ffb640 1 -- 172.21.15.107:0/565361033 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d80ab980 con 0x7f97e814c470 2023-10-24T23:57:45.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97ed5b5640 1 -- 172.21.15.107:0/565361033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97e814c470 msgr2=0x7f97e814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97ed5b5640 1 --2- 172.21.15.107:0/565361033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97e814c470 0x7f97e814c850 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f97e806a7e0 tx=0x7f97d801c340 comp rx=0 tx=0).stop 2023-10-24T23:57:45.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97ed5b5640 1 -- 172.21.15.107:0/565361033 shutdown_connections 2023-10-24T23:57:45.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97ed5b5640 1 --2- 172.21.15.107:0/565361033 >> v2:172.21.15.110:3300/0 conn(0x7f97e806f490 0x7f97e8071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97ed5b5640 1 --2- 172.21.15.107:0/565361033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97e814c470 0x7f97e814c850 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97ed5b5640 1 -- 172.21.15.107:0/565361033 >> 172.21.15.107:0/565361033 conn(0x7f97e8070f60 msgr2=0x7f97e8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:45.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97ed5b5640 1 -- 172.21.15.107:0/565361033 shutdown_connections 2023-10-24T23:57:45.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.178+0000 7f97ed5b5640 1 -- 172.21.15.107:0/565361033 wait complete. 2023-10-24T23:57:45.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97ed5b5640 1 Processor -- start 2023-10-24T23:57:45.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97ed5b5640 1 -- start start 2023-10-24T23:57:45.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97ed5b5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97e806f490 0x7f97e80acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97ed5b5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97e8071fc0 0x7f97e8145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97ed5b5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97e814c470 0x7f97e81473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97ed5b5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e8146f20 con 0x7f97e806f490 2023-10-24T23:57:45.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97ed5b5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e814c1f0 con 0x7f97e8071fc0 2023-10-24T23:57:45.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97ed5b5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97e814b950 con 0x7f97e814c470 2023-10-24T23:57:45.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97e806f490 0x7f97e80acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97e806f490 0x7f97e80acfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57718/0 (socket says 172.21.15.107:57718) 2023-10-24T23:57:45.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e6ffd640 1 -- 172.21.15.107:0/4263337373 learned_addr learned my addr 172.21.15.107:0/4263337373 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:45.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e67fc640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97e8071fc0 0x7f97e8145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e77fe640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97e814c470 0x7f97e81473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e6ffd640 1 -- 172.21.15.107:0/4263337373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97e814c470 msgr2=0x7f97e81473b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e6ffd640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97e814c470 0x7f97e81473b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e6ffd640 1 -- 172.21.15.107:0/4263337373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97e8071fc0 msgr2=0x7f97e8145020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e6ffd640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97e8071fc0 0x7f97e8145020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e6ffd640 1 -- 172.21.15.107:0/4263337373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d8009420 con 0x7f97e806f490 2023-10-24T23:57:45.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e67fc640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97e8071fc0 0x7f97e8145020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:57:45.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97e6ffd640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97e806f490 0x7f97e80acfd0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f97d80ae040 tx=0x7f97d801ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:45.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97cf7fe640 1 -- 172.21.15.107:0/4263337373 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d80b7650 con 0x7f97e806f490 2023-10-24T23:57:45.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97cf7fe640 1 -- 172.21.15.107:0/4263337373 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97d80b77b0 con 0x7f97e806f490 2023-10-24T23:57:45.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97ed5b5640 1 -- 172.21.15.107:0/4263337373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e814bc30 con 0x7f97e806f490 2023-10-24T23:57:45.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97cf7fe640 1 -- 172.21.15.107:0/4263337373 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d80b7b60 con 0x7f97e806f490 2023-10-24T23:57:45.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97ed5b5640 1 -- 172.21.15.107:0/4263337373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e8145560 con 0x7f97e806f490 2023-10-24T23:57:45.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.182+0000 7f97ed5b5640 1 -- 172.21.15.107:0/4263337373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e80746e0 con 0x7f97e806f490 2023-10-24T23:57:45.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.186+0000 7f97cf7fe640 1 -- 172.21.15.107:0/4263337373 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f97d80b7cc0 con 0x7f97e806f490 2023-10-24T23:57:45.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.186+0000 7f97cf7fe640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d4065e60 0x7f97d4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.186+0000 7f97cf7fe640 1 -- 172.21.15.107:0/4263337373 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f97d811e690 con 0x7f97e806f490 2023-10-24T23:57:45.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.186+0000 7f97e67fc640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d4065e60 0x7f97d4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.186+0000 7f97e67fc640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d4065e60 0x7f97d4068320 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f97e80c2430 tx=0x7f97dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:45.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.190+0000 7f97cf7fe640 1 -- 172.21.15.107:0/4263337373 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f97d80eb900 con 0x7f97e806f490 2023-10-24T23:57:45.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:45.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7ffffff640 1 Processor -- start 2023-10-24T23:57:45.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7ffffff640 1 -- start start 2023-10-24T23:57:45.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7ffffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8000151ec0 0x7f80000ab510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7ffffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80000bf750 0x7f80000bfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7ffffff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8000158c20 con 0x7f8000158090 2023-10-24T23:57:45.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7ffffff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80000c28e0 con 0x7f80000bf750 2023-10-24T23:57:45.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7ffffff640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800006d6d0 con 0x7f8000151ec0 2023-10-24T23:57:45.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7fff7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80000bf750 0x7f80000bfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7ffe7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8000158090 0x7f8000158510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59734/0 (socket says 172.21.15.107:59734) 2023-10-24T23:57:45.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7ffe7fc640 1 -- 172.21.15.107:0/2882674180 learned_addr learned my addr 172.21.15.107:0/2882674180 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:45.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7ffeffd640 1 --2- 172.21.15.107:0/2882674180 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8000151ec0 0x7f80000ab510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7fff7fe640 1 -- 172.21.15.107:0/2882674180 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8000151ec0 msgr2=0x7f80000ab510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:45.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7fff7fe640 1 --2- 172.21.15.107:0/2882674180 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8000151ec0 0x7f80000ab510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7ffdffb640 1 -- 172.21.15.107:0/2882674180 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492834911 0 0) 0x7f8000158c20 con 0x7f8000158090 2023-10-24T23:57:45.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7fff7fe640 1 -- 172.21.15.107:0/2882674180 >> v1:172.21.15.107:6789/0 conn(0x7f8000158090 legacy=0x7f8000158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.458+0000 7f7fff7fe640 1 -- 172.21.15.107:0/2882674180 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80000a9f50 con 0x7f80000bf750 2023-10-24T23:57:45.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7fff7fe640 1 --2- 172.21.15.107:0/2882674180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80000bf750 0x7f80000bfbb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8009d90 tx=0x7f7ff801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76fc384ef5ba7621 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:45.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7ffdffb640 1 -- 172.21.15.107:0/2882674180 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff80a5950 con 0x7f80000bf750 2023-10-24T23:57:45.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7ffdffb640 1 -- 172.21.15.107:0/2882674180 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff80a5ab0 con 0x7f80000bf750 2023-10-24T23:57:45.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7ffdffb640 1 -- 172.21.15.107:0/2882674180 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff80a5e40 con 0x7f80000bf750 2023-10-24T23:57:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7ffffff640 1 -- 172.21.15.107:0/2882674180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80000bf750 msgr2=0x7f80000bfbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7ffffff640 1 --2- 172.21.15.107:0/2882674180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80000bf750 0x7f80000bfbb0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8009d90 tx=0x7f7ff801c890 comp rx=0 tx=0).stop 2023-10-24T23:57:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7ffffff640 1 -- 172.21.15.107:0/2882674180 shutdown_connections 2023-10-24T23:57:45.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7ffffff640 1 --2- 172.21.15.107:0/2882674180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80000bf750 0x7f80000bfbb0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7ffffff640 1 --2- 172.21.15.107:0/2882674180 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8000151ec0 0x7f80000ab510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7ffffff640 1 -- 172.21.15.107:0/2882674180 >> 172.21.15.107:0/2882674180 conn(0x7f80000c64c0 msgr2=0x7f80000c68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:45.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7ffffff640 1 -- 172.21.15.107:0/2882674180 shutdown_connections 2023-10-24T23:57:45.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.462+0000 7f7ffffff640 1 -- 172.21.15.107:0/2882674180 wait complete. 2023-10-24T23:57:45.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffffff640 1 Processor -- start 2023-10-24T23:57:45.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffffff640 1 -- start start 2023-10-24T23:57:45.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80000bf750 0x7f8000148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8000151ec0 0x7f8000148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8000158090 0x7f80001eff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800006fd10 con 0x7f80000bf750 2023-10-24T23:57:45.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffffff640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80001f0cd0 con 0x7f8000151ec0 2023-10-24T23:57:45.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffffff640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800014edb0 con 0x7f8000158090 2023-10-24T23:57:45.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80000bf750 0x7f8000148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80000bf750 0x7f8000148ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57742/0 (socket says 172.21.15.107:57742) 2023-10-24T23:57:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffeffd640 1 -- 172.21.15.107:0/2256989347 learned_addr learned my addr 172.21.15.107:0/2256989347 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffe7fc640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8000151ec0 0x7f8000148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7fff7fe640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8000158090 0x7f80001eff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffeffd640 1 -- 172.21.15.107:0/2256989347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8000158090 msgr2=0x7f80001eff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffeffd640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8000158090 0x7f80001eff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffeffd640 1 -- 172.21.15.107:0/2256989347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8000151ec0 msgr2=0x7f8000148ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffeffd640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8000151ec0 0x7f8000148ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffeffd640 1 -- 172.21.15.107:0/2256989347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8009a70 con 0x7f80000bf750 2023-10-24T23:57:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7fff7fe640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8000158090 0x7f80001eff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffeffd640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80000bf750 0x7f8000148ab0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000e870 tx=0x7f7ff000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:45.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7fe77fe640 1 -- 172.21.15.107:0/2256989347 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0009510 con 0x7f80000bf750 2023-10-24T23:57:45.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7fe77fe640 1 -- 172.21.15.107:0/2256989347 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff000eec0 con 0x7f80000bf750 2023-10-24T23:57:45.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffffff640 1 -- 172.21.15.107:0/2256989347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800014f090 con 0x7f80000bf750 2023-10-24T23:57:45.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7fe77fe640 1 -- 172.21.15.107:0/2256989347 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff000cb40 con 0x7f80000bf750 2023-10-24T23:57:45.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7ffffff640 1 -- 172.21.15.107:0/2256989347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80001569f0 con 0x7f80000bf750 2023-10-24T23:57:45.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.466+0000 7f7fe57fa640 1 -- 172.21.15.107:0/2256989347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800004e050 con 0x7f80000bf750 2023-10-24T23:57:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 -- 172.21.15.107:0/4263337373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97e806a950 con 0x7f97e806f490 2023-10-24T23:57:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f7fe77fe640 1 -- 172.21.15.107:0/2256989347 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7ff000cca0 con 0x7f80000bf750 2023-10-24T23:57:45.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97cf7fe640 1 -- 172.21.15.107:0/4263337373 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f97d80eef20 con 0x7f97e806f490 2023-10-24T23:57:45.479 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:45.479 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:45.480 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f7fe77fe640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7fec065e90 0x7f7fec068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f7fe77fe640 1 -- 172.21.15.107:0/2256989347 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f7ff0083da0 con 0x7f80000bf750 2023-10-24T23:57:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f7ffe7fc640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7fec065e90 0x7f7fec068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f7ffe7fc640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7fec065e90 0x7f7fec068350 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8000155960 tx=0x7f7ff4007b60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:45.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 -- 172.21.15.107:0/4263337373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d4065e60 msgr2=0x7f97d4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d4065e60 0x7f97d4068320 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f97e80c2430 tx=0x7f97dc008040 comp rx=0 tx=0).stop 2023-10-24T23:57:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 -- 172.21.15.107:0/4263337373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97e806f490 msgr2=0x7f97e80acfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97e806f490 0x7f97e80acfd0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f97d80ae040 tx=0x7f97d801ab80 comp rx=0 tx=0).stop 2023-10-24T23:57:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 -- 172.21.15.107:0/4263337373 shutdown_connections 2023-10-24T23:57:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d4065e60 0x7f97d4068320 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97e814c470 0x7f97e81473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97e8071fc0 0x7f97e8145020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 --2- 172.21.15.107:0/4263337373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97e806f490 0x7f97e80acfd0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 -- 172.21.15.107:0/4263337373 >> 172.21.15.107:0/4263337373 conn(0x7f97e8070f60 msgr2=0x7f97e8073270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 -- 172.21.15.107:0/4263337373 shutdown_connections 2023-10-24T23:57:45.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.470+0000 7f97ed5b5640 1 -- 172.21.15.107:0/4263337373 wait complete. 2023-10-24T23:57:45.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.478+0000 7f7fe77fe640 1 -- 172.21.15.107:0/2256989347 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7ff0050ef0 con 0x7f80000bf750 2023-10-24T23:57:45.495 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2023-10-24T23:57:45.498 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before thrashing 2023-10-24T23:57:45.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.678+0000 7f7fe57fa640 1 -- 172.21.15.107:0/2256989347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f800006d4b0 con 0x7f80000bf750 2023-10-24T23:57:45.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe77fe640 1 -- 172.21.15.107:0/2256989347 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f7ff0054510 con 0x7f80000bf750 2023-10-24T23:57:45.686 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:45.686 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:45.687 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:45.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 -- 172.21.15.107:0/2256989347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7fec065e90 msgr2=0x7f7fec068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7fec065e90 0x7f7fec068350 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8000155960 tx=0x7f7ff4007b60 comp rx=0 tx=0).stop 2023-10-24T23:57:45.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 -- 172.21.15.107:0/2256989347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80000bf750 msgr2=0x7f8000148ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80000bf750 0x7f8000148ab0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000e870 tx=0x7f7ff000ec70 comp rx=0 tx=0).stop 2023-10-24T23:57:45.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 -- 172.21.15.107:0/2256989347 shutdown_connections 2023-10-24T23:57:45.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7fec065e90 0x7f7fec068350 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8000158090 0x7f80001eff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8000151ec0 0x7f8000148ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 --2- 172.21.15.107:0/2256989347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80000bf750 0x7f8000148ab0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 -- 172.21.15.107:0/2256989347 >> 172.21.15.107:0/2256989347 conn(0x7f80000c64c0 msgr2=0x7f80000aac50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:45.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 -- 172.21.15.107:0/2256989347 shutdown_connections 2023-10-24T23:57:45.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.682+0000 7f7fe57fa640 1 -- 172.21.15.107:0/2256989347 wait complete. 2023-10-24T23:57:45.713 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2023-10-24T23:57:45.713 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-24T23:57:45.713 INFO:teuthology.orchestra.run:waiting for 300 2023-10-24T23:57:45.724 INFO:tasks.ceph.mds.a:Stopped 2023-10-24T23:57:45.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:45.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53acd4d640 1 Processor -- start 2023-10-24T23:57:45.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53acd4d640 1 -- start start 2023-10-24T23:57:45.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53acd4d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f53a80c5c20 0x7f53a8153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53acd4d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a80c18a0 0x7f53a80c1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53acd4d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a8072330 con 0x7f53a814fc30 2023-10-24T23:57:45.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53acd4d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a8155ca0 con 0x7f53a80c5c20 2023-10-24T23:57:45.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53acd4d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a8155f80 con 0x7f53a80c18a0 2023-10-24T23:57:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f53a814fc30 0x7f53a806df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59740/0 (socket says 172.21.15.107:59740) 2023-10-24T23:57:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a659c640 1 -- 172.21.15.107:0/1614274282 learned_addr learned my addr 172.21.15.107:0/1614274282 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a5d9b640 1 --2- 172.21.15.107:0/1614274282 >> v2:172.21.15.110:3300/0 conn(0x7f53a80c5c20 0x7f53a8153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a6d9d640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a80c18a0 0x7f53a80c1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112104020 0 0) 0x7f53a8072330 con 0x7f53a814fc30 2023-10-24T23:57:45.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5390003530 con 0x7f53a814fc30 2023-10-24T23:57:45.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 674171696 0 0) 0x7f5390003530 con 0x7f53a814fc30 2023-10-24T23:57:45.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a80c18a0 msgr2=0x7f53a80c1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:45.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a80c18a0 0x7f53a80c1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 >> v2:172.21.15.110:3300/0 conn(0x7f53a80c5c20 msgr2=0x7f53a8153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:45.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 --2- 172.21.15.107:0/1614274282 >> v2:172.21.15.110:3300/0 conn(0x7f53a80c5c20 0x7f53a8153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a8156260 con 0x7f53a814fc30 2023-10-24T23:57:45.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5398001680 con 0x7f53a814fc30 2023-10-24T23:57:45.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 >> v1:172.21.15.107:6789/0 conn(0x7f53a814fc30 legacy=0x7f53a806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 0x7f5390003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53900047d0 0x7f5390004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a80c18a0 0x7f5390005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a8155ca0 con 0x7f5390003980 2023-10-24T23:57:45.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a8155f80 con 0x7f53900047d0 2023-10-24T23:57:45.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539000b990 con 0x7f53a80c18a0 2023-10-24T23:57:45.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a659c640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 0x7f5390003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a5d9b640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53900047d0 0x7f5390004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a6d9d640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a80c18a0 0x7f5390005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a659c640 1 -- 172.21.15.107:0/1614274282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a80c18a0 msgr2=0x7f5390005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:45.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a659c640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a80c18a0 0x7f5390005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a659c640 1 -- 172.21.15.107:0/1614274282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53900047d0 msgr2=0x7f5390004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:45.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a659c640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53900047d0 0x7f5390004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.882+0000 7f53a659c640 1 -- 172.21.15.107:0/1614274282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390003530 con 0x7f5390003980 2023-10-24T23:57:45.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53a659c640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 0x7f5390003da0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f539800c240 tx=0x7f539801cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24999fd1315adca server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53980058e0 con 0x7f5390003980 2023-10-24T23:57:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5398005a40 con 0x7f5390003980 2023-10-24T23:57:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53a559a640 1 -- 172.21.15.107:0/1614274282 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398005df0 con 0x7f5390003980 2023-10-24T23:57:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53acd4d640 1 -- 172.21.15.107:0/1614274282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 msgr2=0x7f5390003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53acd4d640 1 --2- 172.21.15.107:0/1614274282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 0x7f5390003da0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f539800c240 tx=0x7f539801cbf0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53a6d9d640 1 -- 172.21.15.107:0/1614274282 reap_dead start 2023-10-24T23:57:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53acd4d640 1 -- 172.21.15.107:0/1614274282 shutdown_connections 2023-10-24T23:57:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53acd4d640 1 -- 172.21.15.107:0/1614274282 >> 172.21.15.107:0/1614274282 conn(0x7f53a80c39a0 msgr2=0x7f53a80c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53acd4d640 1 -- 172.21.15.107:0/1614274282 shutdown_connections 2023-10-24T23:57:45.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53acd4d640 1 -- 172.21.15.107:0/1614274282 wait complete. 2023-10-24T23:57:45.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53acd4d640 1 Processor -- start 2023-10-24T23:57:45.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.886+0000 7f53acd4d640 1 -- start start 2023-10-24T23:57:45.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53acd4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 0x7f53a80aa630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53acd4d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53a8153b80 0x7f53a8150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53acd4d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a8152780 0x7f53a81ea760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53acd4d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a8140530 con 0x7f5390003980 2023-10-24T23:57:45.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53acd4d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a815a830 con 0x7f53a8153b80 2023-10-24T23:57:45.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53acd4d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a80ab510 con 0x7f53a8152780 2023-10-24T23:57:45.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 0x7f53a80aa630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 0x7f53a80aa630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57752/0 (socket says 172.21.15.107:57752) 2023-10-24T23:57:45.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a659c640 1 -- 172.21.15.107:0/823112063 learned_addr learned my addr 172.21.15.107:0/823112063 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a5d9b640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53a8153b80 0x7f53a8150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a6d9d640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a8152780 0x7f53a81ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a659c640 1 -- 172.21.15.107:0/823112063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a8152780 msgr2=0x7f53a81ea760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a659c640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a8152780 0x7f53a81ea760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a659c640 1 -- 172.21.15.107:0/823112063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53a8153b80 msgr2=0x7f53a8150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a659c640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53a8153b80 0x7f53a8150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a659c640 1 -- 172.21.15.107:0/823112063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a81eb550 con 0x7f5390003980 2023-10-24T23:57:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a5d9b640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53a8153b80 0x7f53a8150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:57:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a659c640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 0x7f53a80aa630 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f539800c240 tx=0x7f5390005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:45.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f5396ffd640 1 -- 172.21.15.107:0/823112063 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53980047f0 con 0x7f5390003980 2023-10-24T23:57:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f5396ffd640 1 -- 172.21.15.107:0/823112063 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5398004950 con 0x7f5390003980 2023-10-24T23:57:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53acd4d640 1 -- 172.21.15.107:0/823112063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a815aa80 con 0x7f5390003980 2023-10-24T23:57:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f5396ffd640 1 -- 172.21.15.107:0/823112063 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398004cc0 con 0x7f5390003980 2023-10-24T23:57:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53acd4d640 1 -- 172.21.15.107:0/823112063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a8152260 con 0x7f5390003980 2023-10-24T23:57:45.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53acd4d640 1 -- 172.21.15.107:0/823112063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a80746e0 con 0x7f5390003980 2023-10-24T23:57:45.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f5396ffd640 1 -- 172.21.15.107:0/823112063 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f53980a57f0 con 0x7f5390003980 2023-10-24T23:57:45.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f5396ffd640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5388065e90 0x7f5388068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:45.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f5396ffd640 1 -- 172.21.15.107:0/823112063 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f539811d700 con 0x7f5390003980 2023-10-24T23:57:45.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.890+0000 7f53a5d9b640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5388065e90 0x7f5388068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:45.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.894+0000 7f53a5d9b640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5388065e90 0x7f5388068350 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f53a815a5f0 tx=0x7f539c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:45.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:45.894+0000 7f5396ffd640 1 -- 172.21.15.107:0/823112063 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f53980ea930 con 0x7f5390003980 2023-10-24T23:57:46.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.102+0000 7f53acd4d640 1 -- 172.21.15.107:0/823112063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53a80c6130 con 0x7f5390003980 2023-10-24T23:57:46.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.102+0000 7f5396ffd640 1 -- 172.21.15.107:0/823112063 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f53980bd090 con 0x7f5390003980 2023-10-24T23:57:46.108 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:46.109 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:46.109 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:46.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 -- 172.21.15.107:0/823112063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5388065e90 msgr2=0x7f5388068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:46.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5388065e90 0x7f5388068350 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f53a815a5f0 tx=0x7f539c008040 comp rx=0 tx=0).stop 2023-10-24T23:57:46.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 -- 172.21.15.107:0/823112063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 msgr2=0x7f53a80aa630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:46.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 0x7f53a80aa630 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f539800c240 tx=0x7f5390005800 comp rx=0 tx=0).stop 2023-10-24T23:57:46.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 -- 172.21.15.107:0/823112063 shutdown_connections 2023-10-24T23:57:46.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5388065e90 0x7f5388068350 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:46.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53a8152780 0x7f53a81ea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:46.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53a8153b80 0x7f53a8150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:46.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 --2- 172.21.15.107:0/823112063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5390003980 0x7f53a80aa630 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:46.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 -- 172.21.15.107:0/823112063 >> 172.21.15.107:0/823112063 conn(0x7f53a80c39a0 msgr2=0x7f53a815b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:46.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 -- 172.21.15.107:0/823112063 shutdown_connections 2023-10-24T23:57:46.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:46.106+0000 7f53acd4d640 1 -- 172.21.15.107:0/823112063 wait complete. 2023-10-24T23:57:46.129 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:57:48.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:48.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.266+0000 7fa8c39aa640 1 Processor -- start 2023-10-24T23:57:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.266+0000 7fa8c39aa640 1 -- start start 2023-10-24T23:57:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.266+0000 7fa8c39aa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8bc070b00 0x7fa8bc0c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.266+0000 7fa8c39aa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa8bc06df30 0x7fa8bc06e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.266+0000 7fa8c39aa640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8bc0c22f0 con 0x7fa8bc14eb70 2023-10-24T23:57:48.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.266+0000 7fa8c39aa640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8bc071700 con 0x7fa8bc06df30 2023-10-24T23:57:48.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.266+0000 7fa8c39aa640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc0c25d0 con 0x7fa8bc070b00 2023-10-24T23:57:48.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.266+0000 7fa8c1f47640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa8bc06df30 0x7fa8bc06e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:48.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.266+0000 7fa8c1f47640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa8bc06df30 0x7fa8bc06e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57764/0 (socket says 172.21.15.107:57764) 2023-10-24T23:57:48.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.266+0000 7fa8c1f47640 1 -- 172.21.15.107:0/344765547 learned_addr learned my addr 172.21.15.107:0/344765547 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:48.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.266+0000 7fa8c0f45640 1 --2- 172.21.15.107:0/344765547 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8bc070b00 0x7fa8bc0c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:48.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c1f47640 1 -- 172.21.15.107:0/344765547 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8bc070b00 msgr2=0x7fa8bc0c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:48.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c1f47640 1 --2- 172.21.15.107:0/344765547 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8bc070b00 0x7fa8bc0c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:48.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c1f47640 1 -- 172.21.15.107:0/344765547 >> v1:172.21.15.110:6789/0 conn(0x7fa8bc14eb70 legacy=0x7fa8bc1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:48.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c1f47640 1 -- 172.21.15.107:0/344765547 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc0c2be0 con 0x7fa8bc06df30 2023-10-24T23:57:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c1f47640 1 --2- 172.21.15.107:0/344765547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8bc06df30 0x7fa8bc06e360 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009e50 tx=0x7fa8b401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7c73192ae22067 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8b3fff640 1 -- 172.21.15.107:0/344765547 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b40a5900 con 0x7fa8bc06df30 2023-10-24T23:57:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8b3fff640 1 -- 172.21.15.107:0/344765547 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b40a5a60 con 0x7fa8bc06df30 2023-10-24T23:57:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8b3fff640 1 -- 172.21.15.107:0/344765547 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b40a5df0 con 0x7fa8bc06df30 2023-10-24T23:57:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c39aa640 1 -- 172.21.15.107:0/344765547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8bc06df30 msgr2=0x7fa8bc06e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c39aa640 1 --2- 172.21.15.107:0/344765547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8bc06df30 0x7fa8bc06e360 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009e50 tx=0x7fa8b401ca10 comp rx=0 tx=0).stop 2023-10-24T23:57:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c39aa640 1 -- 172.21.15.107:0/344765547 shutdown_connections 2023-10-24T23:57:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c39aa640 1 --2- 172.21.15.107:0/344765547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8bc06df30 0x7fa8bc06e360 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c39aa640 1 --2- 172.21.15.107:0/344765547 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8bc070b00 0x7fa8bc0c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c39aa640 1 -- 172.21.15.107:0/344765547 >> 172.21.15.107:0/344765547 conn(0x7fa8bc0c39a0 msgr2=0x7fa8bc0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c39aa640 1 -- 172.21.15.107:0/344765547 shutdown_connections 2023-10-24T23:57:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.270+0000 7fa8c39aa640 1 -- 172.21.15.107:0/344765547 wait complete. 2023-10-24T23:57:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c39aa640 1 Processor -- start 2023-10-24T23:57:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c39aa640 1 -- start start 2023-10-24T23:57:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c39aa640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8bc06df30 0x7fa8bc147e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c39aa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8bc070b00 0x7fa8bc15a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c39aa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8bc14eb70 0x7fa8bc1ed120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c39aa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc145670 con 0x7fa8bc070b00 2023-10-24T23:57:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c39aa640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc06c720 con 0x7fa8bc06df30 2023-10-24T23:57:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c39aa640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc1eca60 con 0x7fa8bc14eb70 2023-10-24T23:57:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c0f45640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8bc070b00 0x7fa8bc15a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c0f45640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8bc070b00 0x7fa8bc15a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57770/0 (socket says 172.21.15.107:57770) 2023-10-24T23:57:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c0f45640 1 -- 172.21.15.107:0/4152040706 learned_addr learned my addr 172.21.15.107:0/4152040706 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c1746640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8bc06df30 0x7fa8bc147e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c1f47640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8bc14eb70 0x7fa8bc1ed120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c0f45640 1 -- 172.21.15.107:0/4152040706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8bc14eb70 msgr2=0x7fa8bc1ed120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c0f45640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8bc14eb70 0x7fa8bc1ed120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c0f45640 1 -- 172.21.15.107:0/4152040706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8bc06df30 msgr2=0x7fa8bc147e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c0f45640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8bc06df30 0x7fa8bc147e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c0f45640 1 -- 172.21.15.107:0/4152040706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b4009b30 con 0x7fa8bc070b00 2023-10-24T23:57:48.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c1f47640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8bc14eb70 0x7fa8bc1ed120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:48.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c0f45640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8bc070b00 0x7fa8bc15a4a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00d7d0 tx=0x7fa8ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:48.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8b1ffb640 1 -- 172.21.15.107:0/4152040706 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac00b6c0 con 0x7fa8bc070b00 2023-10-24T23:57:48.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8b1ffb640 1 -- 172.21.15.107:0/4152040706 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ac00b820 con 0x7fa8bc070b00 2023-10-24T23:57:48.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c39aa640 1 -- 172.21.15.107:0/4152040706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8bc1458c0 con 0x7fa8bc070b00 2023-10-24T23:57:48.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8b1ffb640 1 -- 172.21.15.107:0/4152040706 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac00bbd0 con 0x7fa8bc070b00 2023-10-24T23:57:48.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.274+0000 7fa8c39aa640 1 -- 172.21.15.107:0/4152040706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8bc15a9e0 con 0x7fa8bc070b00 2023-10-24T23:57:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.278+0000 7fa8c39aa640 1 -- 172.21.15.107:0/4152040706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8bc1f0060 con 0x7fa8bc070b00 2023-10-24T23:57:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.278+0000 7fa8b1ffb640 1 -- 172.21.15.107:0/4152040706 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa8ac00bd30 con 0x7fa8bc070b00 2023-10-24T23:57:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.278+0000 7fa8b1ffb640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa8a0065e60 0x7fa8a0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.278+0000 7fa8c1746640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa8a0065e60 0x7fa8a0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:48.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.278+0000 7fa8b1ffb640 1 -- 172.21.15.107:0/4152040706 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7fa8ac084300 con 0x7fa8bc070b00 2023-10-24T23:57:48.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.278+0000 7fa8c1746640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa8a0065e60 0x7fa8a0068320 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8000f70 tx=0x7fa8b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:48.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.282+0000 7fa8b1ffb640 1 -- 172.21.15.107:0/4152040706 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa8bc1f0060 con 0x7fa8bc070b00 2023-10-24T23:57:48.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 -- 172.21.15.107:0/4152040706 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8bc1f0060 con 0x7fa8bc070b00 2023-10-24T23:57:48.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8b1ffb640 1 -- 172.21.15.107:0/4152040706 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fa8bc1f0060 con 0x7fa8bc070b00 2023-10-24T23:57:48.515 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:48.515 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:48.517 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:48.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 -- 172.21.15.107:0/4152040706 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa8a0065e60 msgr2=0x7fa8a0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:48.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa8a0065e60 0x7fa8a0068320 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8000f70 tx=0x7fa8b8009040 comp rx=0 tx=0).stop 2023-10-24T23:57:48.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 -- 172.21.15.107:0/4152040706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8bc070b00 msgr2=0x7fa8bc15a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:48.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8bc070b00 0x7fa8bc15a4a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00d7d0 tx=0x7fa8ac00dc90 comp rx=0 tx=0).stop 2023-10-24T23:57:48.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 -- 172.21.15.107:0/4152040706 shutdown_connections 2023-10-24T23:57:48.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa8a0065e60 0x7fa8a0068320 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:48.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa8bc14eb70 0x7fa8bc1ed120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:48.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8bc070b00 0x7fa8bc15a4a0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:48.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 --2- 172.21.15.107:0/4152040706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8bc06df30 0x7fa8bc147e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:48.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 -- 172.21.15.107:0/4152040706 >> 172.21.15.107:0/4152040706 conn(0x7fa8bc0c39a0 msgr2=0x7fa8bc14da30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:48.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 -- 172.21.15.107:0/4152040706 shutdown_connections 2023-10-24T23:57:48.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:48.510+0000 7fa8c39aa640 1 -- 172.21.15.107:0/4152040706 wait complete. 2023-10-24T23:57:48.533 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:57:50.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:50.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:50.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ec32f640 1 Processor -- start 2023-10-24T23:57:50.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ec32f640 1 -- start start 2023-10-24T23:57:50.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ec32f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa5e4152040 0x7fa5e40c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ec32f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e4155680 con 0x7fa5e406a4b0 2023-10-24T23:57:50.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ec32f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e4154a70 con 0x7fa5e40c8100 2023-10-24T23:57:50.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ec32f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e4154d50 con 0x7fa5e4152040 2023-10-24T23:57:50.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea8cc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa5e406a4b0 0x7fa5e40ac410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59746/0 (socket says 172.21.15.107:59746) 2023-10-24T23:57:50.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea8cc640 1 -- 172.21.15.107:0/1951120657 learned_addr learned my addr 172.21.15.107:0/1951120657 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:50.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea0cb640 1 --2- 172.21.15.107:0/1951120657 >> v2:172.21.15.110:3300/0 conn(0x7fa5e4152040 0x7fa5e40c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868725156 0 0) 0x7fa5e4155680 con 0x7fa5e406a4b0 2023-10-24T23:57:50.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d0003530 con 0x7fa5e406a4b0 2023-10-24T23:57:50.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1078994647 0 0) 0x7fa5d0003530 con 0x7fa5e406a4b0 2023-10-24T23:57:50.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 >> v2:172.21.15.110:3300/0 conn(0x7fa5e4152040 msgr2=0x7fa5e40c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:50.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 --2- 172.21.15.107:0/1951120657 >> v2:172.21.15.110:3300/0 conn(0x7fa5e4152040 0x7fa5e40c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 >> v1:172.21.15.110:6789/0 conn(0x7fa5e40c8100 legacy=0x7fa5e40abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e4155030 con 0x7fa5e406a4b0 2023-10-24T23:57:50.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fa5dc001a00 con 0x7fa5e406a4b0 2023-10-24T23:57:50.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 >> v1:172.21.15.107:6789/0 conn(0x7fa5e406a4b0 legacy=0x7fa5e40ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d0003980 0x7fa5d0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5d00047d0 0x7fa5d0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d00052b0 0x7fa5d000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e4154d50 con 0x7fa5d0003980 2023-10-24T23:57:50.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d000c180 con 0x7fa5d00052b0 2023-10-24T23:57:50.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d000c340 con 0x7fa5d00047d0 2023-10-24T23:57:50.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea0cb640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d0003980 0x7fa5d0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5e98ca640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5d00047d0 0x7fa5d0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea8cc640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d00052b0 0x7fa5d000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea0cb640 1 -- 172.21.15.107:0/1951120657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5d00047d0 msgr2=0x7fa5d0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:50.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea0cb640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5d00047d0 0x7fa5d0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea0cb640 1 -- 172.21.15.107:0/1951120657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d00052b0 msgr2=0x7fa5d000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:50.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea0cb640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d00052b0 0x7fa5d000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea0cb640 1 -- 172.21.15.107:0/1951120657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d0003530 con 0x7fa5d0003980 2023-10-24T23:57:50.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea8cc640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5d00052b0 0x7fa5d000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:50.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.626+0000 7fa5ea8cc640 1 -- 172.21.15.107:0/1951120657 reap_dead start 2023-10-24T23:57:50.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.630+0000 7fa5ea0cb640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d0003980 0x7fa5d0003da0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009ab0 tx=0x7fa5e001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f906cc3c767dd090 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:50.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.630+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e00a5510 con 0x7fa5d0003980 2023-10-24T23:57:50.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.630+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e00a5670 con 0x7fa5d0003980 2023-10-24T23:57:50.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.630+0000 7fa5e90c9640 1 -- 172.21.15.107:0/1951120657 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e00a5820 con 0x7fa5d0003980 2023-10-24T23:57:50.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.630+0000 7fa5ec32f640 1 -- 172.21.15.107:0/1951120657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d0003980 msgr2=0x7fa5d0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.630+0000 7fa5ec32f640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d0003980 0x7fa5d0003da0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009ab0 tx=0x7fa5e001ca30 comp rx=0 tx=0).stop 2023-10-24T23:57:50.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.630+0000 7fa5ec32f640 1 -- 172.21.15.107:0/1951120657 shutdown_connections 2023-10-24T23:57:50.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.630+0000 7fa5ec32f640 1 --2- 172.21.15.107:0/1951120657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5d0003980 0x7fa5d0003da0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.630+0000 7fa5ec32f640 1 -- 172.21.15.107:0/1951120657 >> 172.21.15.107:0/1951120657 conn(0x7fa5e40ca200 msgr2=0x7fa5e40ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:50.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.630+0000 7fa5ec32f640 1 -- 172.21.15.107:0/1951120657 shutdown_connections 2023-10-24T23:57:50.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.630+0000 7fa5ec32f640 1 -- 172.21.15.107:0/1951120657 wait complete. 2023-10-24T23:57:50.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ec32f640 1 Processor -- start 2023-10-24T23:57:50.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ec32f640 1 -- start start 2023-10-24T23:57:50.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ec32f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5d0003980 0x7fa5e40ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ec32f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5e40ae400 0x7fa5e40aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ec32f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5e4147fa0 0x7fa5e4148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ec32f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e40ae0d0 con 0x7fa5e4147fa0 2023-10-24T23:57:50.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ec32f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e40addf0 con 0x7fa5e40ae400 2023-10-24T23:57:50.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ec32f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e41ee230 con 0x7fa5d0003980 2023-10-24T23:57:50.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ea8cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5e4147fa0 0x7fa5e4148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ea8cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5e4147fa0 0x7fa5e4148450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57800/0 (socket says 172.21.15.107:57800) 2023-10-24T23:57:50.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ea8cc640 1 -- 172.21.15.107:0/2448642416 learned_addr learned my addr 172.21.15.107:0/2448642416 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:50.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ea0cb640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5d0003980 0x7fa5e40ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5e98ca640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5e40ae400 0x7fa5e40aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ea8cc640 1 -- 172.21.15.107:0/2448642416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5d0003980 msgr2=0x7fa5e40ae950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ea8cc640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5d0003980 0x7fa5e40ae950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ea8cc640 1 -- 172.21.15.107:0/2448642416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5e40ae400 msgr2=0x7fa5e40aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ea8cc640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5e40ae400 0x7fa5e40aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ea8cc640 1 -- 172.21.15.107:0/2448642416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e41ee510 con 0x7fa5e4147fa0 2023-10-24T23:57:50.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ea8cc640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5e4147fa0 0x7fa5e4148450 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0012f0 tx=0x7fa5dc00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5daffd640 1 -- 172.21.15.107:0/2448642416 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc004d70 con 0x7fa5e4147fa0 2023-10-24T23:57:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ec32f640 1 -- 172.21.15.107:0/2448642416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e41ee7f0 con 0x7fa5e4147fa0 2023-10-24T23:57:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5ec32f640 1 -- 172.21.15.107:0/2448642416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e40ada80 con 0x7fa5e4147fa0 2023-10-24T23:57:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5daffd640 1 -- 172.21.15.107:0/2448642416 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc00bb00 con 0x7fa5e4147fa0 2023-10-24T23:57:50.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.634+0000 7fa5daffd640 1 -- 172.21.15.107:0/2448642416 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc00be90 con 0x7fa5e4147fa0 2023-10-24T23:57:50.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.638+0000 7fa5ec32f640 1 -- 172.21.15.107:0/2448642416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc005260 con 0x7fa5e4147fa0 2023-10-24T23:57:50.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.638+0000 7fa5daffd640 1 -- 172.21.15.107:0/2448642416 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa5dc007070 con 0x7fa5e4147fa0 2023-10-24T23:57:50.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.638+0000 7fa5daffd640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa5c8065e60 0x7fa5c8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.638+0000 7fa5daffd640 1 -- 172.21.15.107:0/2448642416 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7fa5dc084190 con 0x7fa5e4147fa0 2023-10-24T23:57:50.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.638+0000 7fa5ea0cb640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa5c8065e60 0x7fa5c8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.638+0000 7fa5ea0cb640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa5c8065e60 0x7fa5c8068320 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa5e00af000 tx=0x7fa5e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:50.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.642+0000 7fa5daffd640 1 -- 172.21.15.107:0/2448642416 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa5dc0511c0 con 0x7fa5e4147fa0 2023-10-24T23:57:50.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.678+0000 7f25fa1fb640 1 Processor -- start 2023-10-24T23:57:50.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.678+0000 7f25fa1fb640 1 -- start start 2023-10-24T23:57:50.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.678+0000 7f25fa1fb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25f40aa140 0x7f25f40aa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.678+0000 7f25fa1fb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25f40ac020 0x7f25f40c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.678+0000 7f25fa1fb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f4066000 con 0x7f25f40c2520 2023-10-24T23:57:50.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.678+0000 7f25fa1fb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f40aaa60 con 0x7f25f40aa140 2023-10-24T23:57:50.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.678+0000 7f25fa1fb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f40c0f40 con 0x7f25f40ac020 2023-10-24T23:57:50.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.678+0000 7f25f37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25f40aa140 0x7f25f40aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.678+0000 7f25f3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f25f40c2520 0x7f25f4154df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59758/0 (socket says 172.21.15.107:59758) 2023-10-24T23:57:50.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.678+0000 7f25f3fff640 1 -- 172.21.15.107:0/4185934214 learned_addr learned my addr 172.21.15.107:0/4185934214 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:50.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.678+0000 7f25eaffd640 1 --2- 172.21.15.107:0/4185934214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25f40ac020 0x7f25f40c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25f37fe640 1 -- 172.21.15.107:0/4185934214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25f40ac020 msgr2=0x7f25f40c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:50.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25f37fe640 1 --2- 172.21.15.107:0/4185934214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25f40ac020 0x7f25f40c1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25f37fe640 1 -- 172.21.15.107:0/4185934214 >> v1:172.21.15.107:6789/0 conn(0x7f25f40c2520 legacy=0x7f25f4154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25f37fe640 1 -- 172.21.15.107:0/4185934214 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f40c1220 con 0x7f25f40aa140 2023-10-24T23:57:50.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25f37fe640 1 --2- 172.21.15.107:0/4185934214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f40aa140 0x7f25f40aa520 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f25e40097c0 tx=0x7f25e401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a922f7e648f9152a server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25f2ffd640 1 -- 172.21.15.107:0/4185934214 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e40ad660 con 0x7f25f40aa140 2023-10-24T23:57:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25f2ffd640 1 -- 172.21.15.107:0/4185934214 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e40ad7c0 con 0x7f25f40aa140 2023-10-24T23:57:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25f2ffd640 1 -- 172.21.15.107:0/4185934214 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e40adb50 con 0x7f25f40aa140 2023-10-24T23:57:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25fa1fb640 1 -- 172.21.15.107:0/4185934214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f40aa140 msgr2=0x7f25f40aa520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25fa1fb640 1 --2- 172.21.15.107:0/4185934214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f40aa140 0x7f25f40aa520 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f25e40097c0 tx=0x7f25e401c450 comp rx=0 tx=0).stop 2023-10-24T23:57:50.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25fa1fb640 1 -- 172.21.15.107:0/4185934214 shutdown_connections 2023-10-24T23:57:50.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25fa1fb640 1 --2- 172.21.15.107:0/4185934214 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25f40ac020 0x7f25f40c1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25fa1fb640 1 --2- 172.21.15.107:0/4185934214 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f40aa140 0x7f25f40aa520 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25fa1fb640 1 -- 172.21.15.107:0/4185934214 >> 172.21.15.107:0/4185934214 conn(0x7f25f40c65c0 msgr2=0x7f25f40c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:50.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25fa1fb640 1 -- 172.21.15.107:0/4185934214 shutdown_connections 2023-10-24T23:57:50.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.682+0000 7f25fa1fb640 1 -- 172.21.15.107:0/4185934214 wait complete. 2023-10-24T23:57:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25fa1fb640 1 Processor -- start 2023-10-24T23:57:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25fa1fb640 1 -- start start 2023-10-24T23:57:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25fa1fb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f40aa140 0x7f25f4151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25fa1fb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25f40ac020 0x7f25f414c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25fa1fb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25f40c2520 0x7f25f414c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25fa1fb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f415a210 con 0x7f25f40aa140 2023-10-24T23:57:50.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25fa1fb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f40c4dd0 con 0x7f25f40c2520 2023-10-24T23:57:50.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25fa1fb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f40c5020 con 0x7f25f40ac020 2023-10-24T23:57:50.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f40aa140 0x7f25f4151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f40aa140 0x7f25f4151b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57824/0 (socket says 172.21.15.107:57824) 2023-10-24T23:57:50.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f37fe640 1 -- 172.21.15.107:0/2410772315 learned_addr learned my addr 172.21.15.107:0/2410772315 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:50.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25eaffd640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25f40ac020 0x7f25f414c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f3fff640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25f40c2520 0x7f25f414c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f37fe640 1 -- 172.21.15.107:0/2410772315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25f40ac020 msgr2=0x7f25f414c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f37fe640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25f40ac020 0x7f25f414c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f37fe640 1 -- 172.21.15.107:0/2410772315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25f40c2520 msgr2=0x7f25f414c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f37fe640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25f40c2520 0x7f25f414c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f37fe640 1 -- 172.21.15.107:0/2410772315 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e4009420 con 0x7f25f40aa140 2023-10-24T23:57:50.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25eaffd640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25f40ac020 0x7f25f414c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:50.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f37fe640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f40aa140 0x7f25f4151b80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f25e40b0040 tx=0x7f25e401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:50.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f0ff9640 1 -- 172.21.15.107:0/2410772315 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e40be070 con 0x7f25f40aa140 2023-10-24T23:57:50.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f0ff9640 1 -- 172.21.15.107:0/2410772315 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e40b9450 con 0x7f25f40aa140 2023-10-24T23:57:50.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25f0ff9640 1 -- 172.21.15.107:0/2410772315 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e40b9820 con 0x7f25f40aa140 2023-10-24T23:57:50.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25fa1fb640 1 -- 172.21.15.107:0/2410772315 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f40c5300 con 0x7f25f40aa140 2023-10-24T23:57:50.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.686+0000 7f25fa1fb640 1 -- 172.21.15.107:0/2410772315 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f40c58c0 con 0x7f25f40aa140 2023-10-24T23:57:50.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.690+0000 7f25fa1fb640 1 -- 172.21.15.107:0/2410772315 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f40746e0 con 0x7f25f40aa140 2023-10-24T23:57:50.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.690+0000 7f25f0ff9640 1 -- 172.21.15.107:0/2410772315 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f25e40b9980 con 0x7f25f40aa140 2023-10-24T23:57:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.690+0000 7f25f0ff9640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25d4065e90 0x7f25d4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.690+0000 7f25f0ff9640 1 -- 172.21.15.107:0/2410772315 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f25e411dee0 con 0x7f25f40aa140 2023-10-24T23:57:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.690+0000 7f25eaffd640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25d4065e90 0x7f25d4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:50.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.690+0000 7f25eaffd640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25d4065e90 0x7f25d4068350 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f25f4069ff0 tx=0x7f25dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:50.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.698+0000 7f25f0ff9640 1 -- 172.21.15.107:0/2410772315 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f25e40eb030 con 0x7f25f40aa140 2023-10-24T23:57:50.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.898+0000 7fa5ec32f640 1 -- 172.21.15.107:0/2448642416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5cc0050d0 con 0x7fa5e4147fa0 2023-10-24T23:57:50.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.898+0000 7fa5daffd640 1 -- 172.21.15.107:0/2448642416 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fa5dc0547e0 con 0x7fa5e4147fa0 2023-10-24T23:57:50.905 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:50.905 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 -- 172.21.15.107:0/2448642416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa5c8065e60 msgr2=0x7fa5c8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa5c8065e60 0x7fa5c8068320 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa5e00af000 tx=0x7fa5e0008040 comp rx=0 tx=0).stop 2023-10-24T23:57:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 -- 172.21.15.107:0/2448642416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5e4147fa0 msgr2=0x7fa5e4148450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5e4147fa0 0x7fa5e4148450 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc0012f0 tx=0x7fa5dc00b4a0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 -- 172.21.15.107:0/2448642416 shutdown_connections 2023-10-24T23:57:50.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa5e4147fa0 0x7fa5e4148450 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5e40ae400 0x7fa5e40aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa5c8065e60 0x7fa5c8068320 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 --2- 172.21.15.107:0/2448642416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5d0003980 0x7fa5e40ae950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 -- 172.21.15.107:0/2448642416 >> 172.21.15.107:0/2448642416 conn(0x7fa5e40ca200 msgr2=0x7fa5e414a940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:50.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 -- 172.21.15.107:0/2448642416 shutdown_connections 2023-10-24T23:57:50.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.902+0000 7fa5ec32f640 1 -- 172.21.15.107:0/2448642416 wait complete. 2023-10-24T23:57:50.924 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2023-10-24T23:57:50.925 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-24T23:57:50.925 INFO:teuthology.orchestra.run:waiting for 300 2023-10-24T23:57:50.941 INFO:tasks.ceph.mds.b:Stopped 2023-10-24T23:57:50.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:50.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.938+0000 7f25fa1fb640 1 -- 172.21.15.107:0/2410772315 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25f406a290 con 0x7f25f40aa140 2023-10-24T23:57:50.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.938+0000 7f25f0ff9640 1 -- 172.21.15.107:0/2410772315 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f25e40ee650 con 0x7f25f40aa140 2023-10-24T23:57:50.945 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:50.945 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:50.945 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:50.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 -- 172.21.15.107:0/2410772315 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25d4065e90 msgr2=0x7f25d4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25d4065e90 0x7f25d4068350 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f25f4069ff0 tx=0x7f25dc008040 comp rx=0 tx=0).stop 2023-10-24T23:57:50.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 -- 172.21.15.107:0/2410772315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f40aa140 msgr2=0x7f25f4151b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:50.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f40aa140 0x7f25f4151b80 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f25e40b0040 tx=0x7f25e401ac90 comp rx=0 tx=0).stop 2023-10-24T23:57:50.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 -- 172.21.15.107:0/2410772315 shutdown_connections 2023-10-24T23:57:50.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25f40c2520 0x7f25f414c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f25d4065e90 0x7f25d4068350 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25f40ac020 0x7f25f414c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 --2- 172.21.15.107:0/2410772315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25f40aa140 0x7f25f4151b80 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:50.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 -- 172.21.15.107:0/2410772315 >> 172.21.15.107:0/2410772315 conn(0x7f25f40c65c0 msgr2=0x7f25f414fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:50.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 -- 172.21.15.107:0/2410772315 shutdown_connections 2023-10-24T23:57:50.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:50.942+0000 7f25fa1fb640 1 -- 172.21.15.107:0/2410772315 wait complete. 2023-10-24T23:57:50.966 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:57:51.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700fb2f640 1 Processor -- start 2023-10-24T23:57:51.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700fb2f640 1 -- start start 2023-10-24T23:57:51.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700fb2f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700814fc20 0x7f7008153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:51.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700fb2f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f70080ab190 0x7f70080ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:51.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700fb2f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70081536d0 con 0x7f70080a9450 2023-10-24T23:57:51.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700fb2f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70080ad0a0 con 0x7f70080ab190 2023-10-24T23:57:51.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700fb2f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700806a6b0 con 0x7f700814fc20 2023-10-24T23:57:51.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700e0cc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f70080ab190 0x7f70080ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:51.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700e0cc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f70080ab190 0x7f70080ab5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57828/0 (socket says 172.21.15.107:57828) 2023-10-24T23:57:51.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700e0cc640 1 -- 172.21.15.107:0/1918170699 learned_addr learned my addr 172.21.15.107:0/1918170699 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:51.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700e0cc640 1 -- 172.21.15.107:0/1918170699 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700814fc20 msgr2=0x7f7008153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:51.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700e0cc640 1 --2- 172.21.15.107:0/1918170699 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700814fc20 0x7f7008153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:51.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700e0cc640 1 -- 172.21.15.107:0/1918170699 >> v1:172.21.15.110:6789/0 conn(0x7f70080a9450 legacy=0x7f70080a98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:51.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.054+0000 7f700e0cc640 1 -- 172.21.15.107:0/1918170699 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700806a380 con 0x7f70080ab190 2023-10-24T23:57:51.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700e0cc640 1 --2- 172.21.15.107:0/1918170699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70080ab190 0x7f70080ab5e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7000009870 tx=0x7f700001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588279a3d9f776cb server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:51.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700c8c9640 1 -- 172.21.15.107:0/1918170699 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70000ab600 con 0x7f70080ab190 2023-10-24T23:57:51.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700c8c9640 1 -- 172.21.15.107:0/1918170699 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70000a7050 con 0x7f70080ab190 2023-10-24T23:57:51.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700c8c9640 1 -- 172.21.15.107:0/1918170699 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70000ab980 con 0x7f70080ab190 2023-10-24T23:57:51.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 -- 172.21.15.107:0/1918170699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70080ab190 msgr2=0x7f70080ab5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:51.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 --2- 172.21.15.107:0/1918170699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70080ab190 0x7f70080ab5e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7000009870 tx=0x7f700001c470 comp rx=0 tx=0).stop 2023-10-24T23:57:51.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 -- 172.21.15.107:0/1918170699 shutdown_connections 2023-10-24T23:57:51.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 --2- 172.21.15.107:0/1918170699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70080ab190 0x7f70080ab5e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:51.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 --2- 172.21.15.107:0/1918170699 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700814fc20 0x7f7008153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:51.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 -- 172.21.15.107:0/1918170699 >> 172.21.15.107:0/1918170699 conn(0x7f70080c4e40 msgr2=0x7f70080c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:51.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 -- 172.21.15.107:0/1918170699 shutdown_connections 2023-10-24T23:57:51.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 -- 172.21.15.107:0/1918170699 wait complete. 2023-10-24T23:57:51.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 Processor -- start 2023-10-24T23:57:51.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 -- start start 2023-10-24T23:57:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70080a9450 0x7f70081509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70080ab190 0x7f700815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:51.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700814fc20 0x7f70081f0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7008070420 con 0x7f70080a9450 2023-10-24T23:57:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7008148f40 con 0x7f700814fc20 2023-10-24T23:57:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70080ab850 con 0x7f70080ab190 2023-10-24T23:57:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700d8cb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70080a9450 0x7f70081509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:51.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700d8cb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70080a9450 0x7f70081509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57838/0 (socket says 172.21.15.107:57838) 2023-10-24T23:57:51.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700d8cb640 1 -- 172.21.15.107:0/3241272598 learned_addr learned my addr 172.21.15.107:0/3241272598 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:51.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700e0cc640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700814fc20 0x7f70081f0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:51.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700d0ca640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70080ab190 0x7f700815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:51.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700d8cb640 1 -- 172.21.15.107:0/3241272598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70080ab190 msgr2=0x7f700815b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:51.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700d8cb640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70080ab190 0x7f700815b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700d8cb640 1 -- 172.21.15.107:0/3241272598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700814fc20 msgr2=0x7f70081f0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700d8cb640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700814fc20 0x7f70081f0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700d8cb640 1 -- 172.21.15.107:0/3241272598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70000094d0 con 0x7f70080a9450 2023-10-24T23:57:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700d8cb640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70080a9450 0x7f70081509e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800b550 tx=0x7f6ff800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f6ffe7fc640 1 -- 172.21.15.107:0/3241272598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8010870 con 0x7f70080a9450 2023-10-24T23:57:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f6ffe7fc640 1 -- 172.21.15.107:0/3241272598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff80109f0 con 0x7f70080a9450 2023-10-24T23:57:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 -- 172.21.15.107:0/3241272598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7008149190 con 0x7f70080a9450 2023-10-24T23:57:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f700fb2f640 1 -- 172.21.15.107:0/3241272598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70081f0790 con 0x7f70080a9450 2023-10-24T23:57:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.058+0000 7f6ffe7fc640 1 -- 172.21.15.107:0/3241272598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8010da0 con 0x7f70080a9450 2023-10-24T23:57:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.062+0000 7f700fb2f640 1 -- 172.21.15.107:0/3241272598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff0005260 con 0x7f70080a9450 2023-10-24T23:57:51.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.062+0000 7f6ffe7fc640 1 -- 172.21.15.107:0/3241272598 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6ff800f3c0 con 0x7f70080a9450 2023-10-24T23:57:51.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.062+0000 7f6ffe7fc640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065e90 0x7f6fec068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:51.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.062+0000 7f700d0ca640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065e90 0x7f6fec068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:51.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.062+0000 7f6ffe7fc640 1 -- 172.21.15.107:0/3241272598 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f6ff8083c90 con 0x7f70080a9450 2023-10-24T23:57:51.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.062+0000 7f700d0ca640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065e90 0x7f6fec068350 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f70081525a0 tx=0x7f70040072f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:51.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.066+0000 7f6ffe7fc640 1 -- 172.21.15.107:0/3241272598 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6ff8050de0 con 0x7f70080a9450 2023-10-24T23:57:51.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.242+0000 7f700fb2f640 1 -- 172.21.15.107:0/3241272598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ff0005510 con 0x7f70080a9450 2023-10-24T23:57:51.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.242+0000 7f6ffe7fc640 1 -- 172.21.15.107:0/3241272598 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f6ff8054400 con 0x7f70080a9450 2023-10-24T23:57:51.247 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:51.247 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:51.247 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:51.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.242+0000 7f700fb2f640 1 -- 172.21.15.107:0/3241272598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065e90 msgr2=0x7f6fec068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:51.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.242+0000 7f700fb2f640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065e90 0x7f6fec068350 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f70081525a0 tx=0x7f70040072f0 comp rx=0 tx=0).stop 2023-10-24T23:57:51.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.242+0000 7f700fb2f640 1 -- 172.21.15.107:0/3241272598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70080a9450 msgr2=0x7f70081509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:51.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.242+0000 7f700fb2f640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70080a9450 0x7f70081509e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800b550 tx=0x7f6ff800ba10 comp rx=0 tx=0).stop 2023-10-24T23:57:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.246+0000 7f700fb2f640 1 -- 172.21.15.107:0/3241272598 shutdown_connections 2023-10-24T23:57:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.246+0000 7f700fb2f640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700814fc20 0x7f70081f0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.246+0000 7f700fb2f640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065e90 0x7f6fec068350 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.246+0000 7f700fb2f640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70080ab190 0x7f700815b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:51.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.246+0000 7f700fb2f640 1 --2- 172.21.15.107:0/3241272598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70080a9450 0x7f70081509e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:51.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.246+0000 7f700fb2f640 1 -- 172.21.15.107:0/3241272598 >> 172.21.15.107:0/3241272598 conn(0x7f70080c4e40 msgr2=0x7f70080a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:51.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.246+0000 7f700fb2f640 1 -- 172.21.15.107:0/3241272598 shutdown_connections 2023-10-24T23:57:51.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:51.246+0000 7f700fb2f640 1 -- 172.21.15.107:0/3241272598 wait complete. 2023-10-24T23:57:51.264 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:57:52.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d20a42640 1 Processor -- start 2023-10-24T23:57:53.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d20a42640 1 -- start start 2023-10-24T23:57:53.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d20a42640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3d1c071220 0x7f3d1c06bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d20a42640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3d1c151960 0x7f3d1c071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d20a42640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d1c0c24c0 0x7f3d1c06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d20a42640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d1c06c7f0 con 0x7f3d1c071220 2023-10-24T23:57:53.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d20a42640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d1c150bc0 con 0x7f3d1c151960 2023-10-24T23:57:53.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d20a42640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c06c510 con 0x7f3d1c0c24c0 2023-10-24T23:57:53.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d1a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3d1c071220 0x7f3d1c06bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d1a59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3d1c071220 0x7f3d1c06bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33598/0 (socket says 172.21.15.107:33598) 2023-10-24T23:57:53.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d1a59c640 1 -- 172.21.15.107:0/425447474 learned_addr learned my addr 172.21.15.107:0/425447474 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:53.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d1ad9d640 1 --2- 172.21.15.107:0/425447474 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d1c0c24c0 0x7f3d1c06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d11d9b640 1 --2- 172.21.15.107:0/425447474 >> v2:172.21.15.110:3300/0 conn(0x7f3d1c151960 0x7f3d1c071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d1a59c640 1 -- 172.21.15.107:0/425447474 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d1c0c24c0 msgr2=0x7f3d1c06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:53.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d1a59c640 1 --2- 172.21.15.107:0/425447474 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d1c0c24c0 0x7f3d1c06a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d1a59c640 1 -- 172.21.15.107:0/425447474 >> v2:172.21.15.110:3300/0 conn(0x7f3d1c151960 msgr2=0x7f3d1c071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:53.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d1a59c640 1 --2- 172.21.15.107:0/425447474 >> v2:172.21.15.110:3300/0 conn(0x7f3d1c151960 0x7f3d1c071a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.094+0000 7f3d1a59c640 1 -- 172.21.15.107:0/425447474 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c150e80 con 0x7f3d1c071220 2023-10-24T23:57:53.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d1a59c640 1 --2- 172.21.15.107:0/425447474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c071220 0x7f3d1c06bce0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c06d1d0 tx=0x7f3d0c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe67ad2097c5c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:53.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d19d9b640 1 -- 172.21.15.107:0/425447474 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c0ab600 con 0x7f3d1c071220 2023-10-24T23:57:53.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d19d9b640 1 -- 172.21.15.107:0/425447474 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0c0a7050 con 0x7f3d1c071220 2023-10-24T23:57:53.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d19d9b640 1 -- 172.21.15.107:0/425447474 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c0ab980 con 0x7f3d1c071220 2023-10-24T23:57:53.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d20a42640 1 -- 172.21.15.107:0/425447474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c071220 msgr2=0x7f3d1c06bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:53.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d20a42640 1 --2- 172.21.15.107:0/425447474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c071220 0x7f3d1c06bce0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c06d1d0 tx=0x7f3d0c01c470 comp rx=0 tx=0).stop 2023-10-24T23:57:53.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d20a42640 1 -- 172.21.15.107:0/425447474 shutdown_connections 2023-10-24T23:57:53.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d20a42640 1 --2- 172.21.15.107:0/425447474 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d1c0c24c0 0x7f3d1c06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d20a42640 1 --2- 172.21.15.107:0/425447474 >> v2:172.21.15.110:3300/0 conn(0x7f3d1c151960 0x7f3d1c071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d20a42640 1 --2- 172.21.15.107:0/425447474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c071220 0x7f3d1c06bce0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d20a42640 1 -- 172.21.15.107:0/425447474 >> 172.21.15.107:0/425447474 conn(0x7f3d1c0c2950 msgr2=0x7f3d1c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:53.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d20a42640 1 -- 172.21.15.107:0/425447474 shutdown_connections 2023-10-24T23:57:53.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.098+0000 7f3d20a42640 1 -- 172.21.15.107:0/425447474 wait complete. 2023-10-24T23:57:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d20a42640 1 Processor -- start 2023-10-24T23:57:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d20a42640 1 -- start start 2023-10-24T23:57:53.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d20a42640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d1c0c24c0 0x7f3d1c150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d20a42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c151960 0x7f3d1c1529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d20a42640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d1c152f20 0x7f3d1c0c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d20a42640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c152550 con 0x7f3d1c151960 2023-10-24T23:57:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d20a42640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c147fb0 con 0x7f3d1c0c24c0 2023-10-24T23:57:53.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d20a42640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c148290 con 0x7f3d1c152f20 2023-10-24T23:57:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d11d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c151960 0x7f3d1c1529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d11d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c151960 0x7f3d1c1529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33600/0 (socket says 172.21.15.107:33600) 2023-10-24T23:57:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d11d9b640 1 -- 172.21.15.107:0/1993346657 learned_addr learned my addr 172.21.15.107:0/1993346657 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d11d9b640 1 -- 172.21.15.107:0/1993346657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d1c152f20 msgr2=0x7f3d1c0c4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d1a59c640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d1c0c24c0 0x7f3d1c150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d11d9b640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d1c152f20 0x7f3d1c0c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d11d9b640 1 -- 172.21.15.107:0/1993346657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d1c0c24c0 msgr2=0x7f3d1c150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d11d9b640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d1c0c24c0 0x7f3d1c150400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d11d9b640 1 -- 172.21.15.107:0/1993346657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0c0094d0 con 0x7f3d1c151960 2023-10-24T23:57:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d11d9b640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c151960 0x7f3d1c1529e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400d800 tx=0x7f3d0400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:53.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d137fe640 1 -- 172.21.15.107:0/1993346657 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0400b6c0 con 0x7f3d1c151960 2023-10-24T23:57:53.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d137fe640 1 -- 172.21.15.107:0/1993346657 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0400b820 con 0x7f3d1c151960 2023-10-24T23:57:53.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d20a42640 1 -- 172.21.15.107:0/1993346657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1c148570 con 0x7f3d1c151960 2023-10-24T23:57:53.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d137fe640 1 -- 172.21.15.107:0/1993346657 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0400bb90 con 0x7f3d1c151960 2023-10-24T23:57:53.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d20a42640 1 -- 172.21.15.107:0/1993346657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c150940 con 0x7f3d1c151960 2023-10-24T23:57:53.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d20a42640 1 -- 172.21.15.107:0/1993346657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1c152700 con 0x7f3d1c151960 2023-10-24T23:57:53.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d137fe640 1 -- 172.21.15.107:0/1993346657 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3d0400bcf0 con 0x7f3d1c151960 2023-10-24T23:57:53.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d137fe640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cfc065e60 0x7f3cfc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d137fe640 1 -- 172.21.15.107:0/1993346657 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f3d04084220 con 0x7f3d1c151960 2023-10-24T23:57:53.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.102+0000 7f3d1a59c640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cfc065e60 0x7f3cfc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.106+0000 7f3d1a59c640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cfc065e60 0x7f3cfc068320 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0ad000 tx=0x7f3d0c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:53.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.106+0000 7f3d137fe640 1 -- 172.21.15.107:0/1993346657 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3d1c152700 con 0x7f3d1c151960 2023-10-24T23:57:53.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 -- 172.21.15.107:0/1993346657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d1c0746e0 con 0x7f3d1c151960 2023-10-24T23:57:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d137fe640 1 -- 172.21.15.107:0/1993346657 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f3d04054e00 con 0x7f3d1c151960 2023-10-24T23:57:53.302 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:53.302 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:53.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 -- 172.21.15.107:0/1993346657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cfc065e60 msgr2=0x7f3cfc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:53.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cfc065e60 0x7f3cfc068320 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c0ad000 tx=0x7f3d0c008040 comp rx=0 tx=0).stop 2023-10-24T23:57:53.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 -- 172.21.15.107:0/1993346657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c151960 msgr2=0x7f3d1c1529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:53.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c151960 0x7f3d1c1529e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400d800 tx=0x7f3d0400dcc0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 -- 172.21.15.107:0/1993346657 shutdown_connections 2023-10-24T23:57:53.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3cfc065e60 0x7f3cfc068320 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d1c152f20 0x7f3d1c0c4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d1c151960 0x7f3d1c1529e0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 --2- 172.21.15.107:0/1993346657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d1c0c24c0 0x7f3d1c150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 -- 172.21.15.107:0/1993346657 >> 172.21.15.107:0/1993346657 conn(0x7f3d1c0c2950 msgr2=0x7f3d1c149210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:53.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 -- 172.21.15.107:0/1993346657 shutdown_connections 2023-10-24T23:57:53.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.298+0000 7f3d20a42640 1 -- 172.21.15.107:0/1993346657 wait complete. 2023-10-24T23:57:53.321 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:57:53.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23c11ef640 1 Processor -- start 2023-10-24T23:57:53.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23c11ef640 1 -- start start 2023-10-24T23:57:53.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23c11ef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f23bc14fd40 0x7f23bc0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23c11ef640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23bc06ee40 0x7f23bc0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23c11ef640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23bc06f730 con 0x7f23bc0a9820 2023-10-24T23:57:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23c11ef640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23bc0aa750 con 0x7f23bc14fd40 2023-10-24T23:57:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23c11ef640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc156a40 con 0x7f23bc06ee40 2023-10-24T23:57:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23bad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f23bc14fd40 0x7f23bc0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23bad9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f23bc14fd40 0x7f23bc0ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33610/0 (socket says 172.21.15.107:33610) 2023-10-24T23:57:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23ba59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f23bc0a9820 0x7f23bc0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58938/0 (socket says 172.21.15.107:58938) 2023-10-24T23:57:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23bad9d640 1 -- 172.21.15.107:0/2640862592 learned_addr learned my addr 172.21.15.107:0/2640862592 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23bb59e640 1 --2- 172.21.15.107:0/2640862592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23bc06ee40 0x7f23bc0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23bad9d640 1 -- 172.21.15.107:0/2640862592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23bc06ee40 msgr2=0x7f23bc0aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23bad9d640 1 --2- 172.21.15.107:0/2640862592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23bc06ee40 0x7f23bc0aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23bad9d640 1 -- 172.21.15.107:0/2640862592 >> v1:172.21.15.107:6789/0 conn(0x7f23bc0a9820 legacy=0x7f23bc0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:53.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.381+0000 7f23bad9d640 1 -- 172.21.15.107:0/2640862592 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc156710 con 0x7f23bc14fd40 2023-10-24T23:57:53.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23bad9d640 1 --2- 172.21.15.107:0/2640862592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23bc14fd40 0x7f23bc0ab090 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009870 tx=0x7f23ac01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947d5ecd13c2b21d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:53.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23b9d9b640 1 -- 172.21.15.107:0/2640862592 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0ad640 con 0x7f23bc14fd40 2023-10-24T23:57:53.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23b9d9b640 1 -- 172.21.15.107:0/2640862592 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac0ad7a0 con 0x7f23bc14fd40 2023-10-24T23:57:53.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23b9d9b640 1 -- 172.21.15.107:0/2640862592 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0adb30 con 0x7f23bc14fd40 2023-10-24T23:57:53.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 -- 172.21.15.107:0/2640862592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23bc14fd40 msgr2=0x7f23bc0ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:53.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 --2- 172.21.15.107:0/2640862592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23bc14fd40 0x7f23bc0ab090 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009870 tx=0x7f23ac01c3f0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 -- 172.21.15.107:0/2640862592 shutdown_connections 2023-10-24T23:57:53.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 --2- 172.21.15.107:0/2640862592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23bc06ee40 0x7f23bc0aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 --2- 172.21.15.107:0/2640862592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23bc14fd40 0x7f23bc0ab090 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 -- 172.21.15.107:0/2640862592 >> 172.21.15.107:0/2640862592 conn(0x7f23bc0c2950 msgr2=0x7f23bc0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:53.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 -- 172.21.15.107:0/2640862592 shutdown_connections 2023-10-24T23:57:53.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 -- 172.21.15.107:0/2640862592 wait complete. 2023-10-24T23:57:53.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 Processor -- start 2023-10-24T23:57:53.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 -- start start 2023-10-24T23:57:53.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23bc06ee40 0x7f23bc15b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23bc14fd40 0x7f23bc14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23bc1edc40 0x7f23bc14ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc1ee1b0 con 0x7f23bc14fd40 2023-10-24T23:57:53.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc0c34d0 con 0x7f23bc1edc40 2023-10-24T23:57:53.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23c11ef640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23bc069e70 con 0x7f23bc06ee40 2023-10-24T23:57:53.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23ba59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23bc14fd40 0x7f23bc14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23ba59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23bc14fd40 0x7f23bc14a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33616/0 (socket says 172.21.15.107:33616) 2023-10-24T23:57:53.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.385+0000 7f23ba59c640 1 -- 172.21.15.107:0/263058303 learned_addr learned my addr 172.21.15.107:0/263058303 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:53.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23bad9d640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23bc06ee40 0x7f23bc15b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23bb59e640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23bc1edc40 0x7f23bc14ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23ba59c640 1 -- 172.21.15.107:0/263058303 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23bc06ee40 msgr2=0x7f23bc15b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:53.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23ba59c640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23bc06ee40 0x7f23bc15b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23ba59c640 1 -- 172.21.15.107:0/263058303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23bc1edc40 msgr2=0x7f23bc14ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:53.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23ba59c640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23bc1edc40 0x7f23bc14ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23ba59c640 1 -- 172.21.15.107:0/263058303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ac0094d0 con 0x7f23bc14fd40 2023-10-24T23:57:53.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23ba59c640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23bc14fd40 0x7f23bc14a6d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f23b000c490 tx=0x7f23b000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:53.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23a37fe640 1 -- 172.21.15.107:0/263058303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0007480 con 0x7f23bc14fd40 2023-10-24T23:57:53.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23a37fe640 1 -- 172.21.15.107:0/263058303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b00075e0 con 0x7f23bc14fd40 2023-10-24T23:57:53.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23c11ef640 1 -- 172.21.15.107:0/263058303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23bc06a150 con 0x7f23bc14fd40 2023-10-24T23:57:53.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23a37fe640 1 -- 172.21.15.107:0/263058303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0004a10 con 0x7f23bc14fd40 2023-10-24T23:57:53.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23c11ef640 1 -- 172.21.15.107:0/263058303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23bc06a710 con 0x7f23bc14fd40 2023-10-24T23:57:53.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23c11ef640 1 -- 172.21.15.107:0/263058303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23bc0c24a0 con 0x7f23bc14fd40 2023-10-24T23:57:53.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23a37fe640 1 -- 172.21.15.107:0/263058303 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f23b0004b70 con 0x7f23bc14fd40 2023-10-24T23:57:53.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23a37fe640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f23a8065e60 0x7f23a8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:53.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23a37fe640 1 -- 172.21.15.107:0/263058303 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f23b00831f0 con 0x7f23bc14fd40 2023-10-24T23:57:53.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23bad9d640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f23a8065e60 0x7f23a8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:53.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.389+0000 7f23bad9d640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f23a8065e60 0x7f23a8068320 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0af000 tx=0x7f23ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:53.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.393+0000 7f23a37fe640 1 -- 172.21.15.107:0/263058303 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f23b0050340 con 0x7f23bc14fd40 2023-10-24T23:57:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.581+0000 7f23c11ef640 1 -- 172.21.15.107:0/263058303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23bc070840 con 0x7f23bc14fd40 2023-10-24T23:57:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.581+0000 7f23a37fe640 1 -- 172.21.15.107:0/263058303 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f23b0053960 con 0x7f23bc14fd40 2023-10-24T23:57:53.589 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:53.589 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:53.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 -- 172.21.15.107:0/263058303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f23a8065e60 msgr2=0x7f23a8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f23a8065e60 0x7f23a8068320 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f23ac0af000 tx=0x7f23ac008040 comp rx=0 tx=0).stop 2023-10-24T23:57:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 -- 172.21.15.107:0/263058303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23bc14fd40 msgr2=0x7f23bc14a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23bc14fd40 0x7f23bc14a6d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f23b000c490 tx=0x7f23b000c950 comp rx=0 tx=0).stop 2023-10-24T23:57:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 -- 172.21.15.107:0/263058303 shutdown_connections 2023-10-24T23:57:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f23bc1edc40 0x7f23bc14ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23bc14fd40 0x7f23bc14a6d0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f23a8065e60 0x7f23a8068320 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 --2- 172.21.15.107:0/263058303 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23bc06ee40 0x7f23bc15b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 -- 172.21.15.107:0/263058303 >> 172.21.15.107:0/263058303 conn(0x7f23bc0c2950 msgr2=0x7f23bc15c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 -- 172.21.15.107:0/263058303 shutdown_connections 2023-10-24T23:57:53.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:53.585+0000 7f23c11ef640 1 -- 172.21.15.107:0/263058303 wait complete. 2023-10-24T23:57:53.607 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:57:55.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:55.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.445+0000 7f466a4e7640 1 Processor -- start 2023-10-24T23:57:55.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f466a4e7640 1 -- start start 2023-10-24T23:57:55.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f466a4e7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4664071220 0x7f466406bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f466a4e7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4664150bc0 con 0x7f46640c24c0 2023-10-24T23:57:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f466a4e7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f466406c510 con 0x7f4664151960 2023-10-24T23:57:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f466a4e7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466406c7f0 con 0x7f4664071220 2023-10-24T23:57:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4668a84640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f46640c24c0 0x7f466406a9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58948/0 (socket says 172.21.15.107:58948) 2023-10-24T23:57:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4668a84640 1 -- 172.21.15.107:0/3682589830 learned_addr learned my addr 172.21.15.107:0/3682589830 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4663fff640 1 --2- 172.21.15.107:0/3682589830 >> v2:172.21.15.110:3300/0 conn(0x7f4664071220 0x7f466406bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962945281 0 0) 0x7f4664150bc0 con 0x7f46640c24c0 2023-10-24T23:57:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4650003530 con 0x7f46640c24c0 2023-10-24T23:57:55.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3754838236 0 0) 0x7f4650003530 con 0x7f46640c24c0 2023-10-24T23:57:55.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 >> v2:172.21.15.110:3300/0 conn(0x7f4664071220 msgr2=0x7f466406bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:55.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 --2- 172.21.15.107:0/3682589830 >> v2:172.21.15.110:3300/0 conn(0x7f4664071220 0x7f466406bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 >> v1:172.21.15.110:6789/0 conn(0x7f4664151960 legacy=0x7f4664071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664150e80 con 0x7f46640c24c0 2023-10-24T23:57:55.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f465c001a00 con 0x7f46640c24c0 2023-10-24T23:57:55.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 >> v1:172.21.15.107:6789/0 conn(0x7f46640c24c0 legacy=0x7f466406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4650003980 0x7f4650003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f46500047d0 0x7f4650004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 0x7f465000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466406c7f0 con 0x7f4650005260 2023-10-24T23:57:55.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46500099b0 con 0x7f4650003980 2023-10-24T23:57:55.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4650009b70 con 0x7f46500047d0 2023-10-24T23:57:55.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4663fff640 1 --2- 172.21.15.107:0/3682589830 >> v2:172.21.15.110:3300/0 conn(0x7f4664071220 0x7f466406bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:57:55.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4668a84640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 0x7f465000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f465bfff640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f46500047d0 0x7f4650004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4663fff640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4650003980 0x7f4650003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4668a84640 1 -- 172.21.15.107:0/3682589830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f46500047d0 msgr2=0x7f4650004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:55.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4668a84640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f46500047d0 0x7f4650004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4668a84640 1 -- 172.21.15.107:0/3682589830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4650003980 msgr2=0x7f4650003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:55.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4668a84640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4650003980 0x7f4650003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4668a84640 1 -- 172.21.15.107:0/3682589830 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4650003530 con 0x7f4650005260 2023-10-24T23:57:55.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.449+0000 7f4663fff640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4650003980 0x7f4650003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:55.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f4668a84640 1 -- 172.21.15.107:0/3682589830 reap_dead start 2023-10-24T23:57:55.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f4668a84640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 0x7f465000dba0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f465c00a830 tx=0x7f465c01faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb83bdbd1e3335f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:55.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c0aa6d0 con 0x7f4650005260 2023-10-24T23:57:55.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465c0aa870 con 0x7f4650005260 2023-10-24T23:57:55.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f46637fe640 1 -- 172.21.15.107:0/3682589830 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c0aac20 con 0x7f4650005260 2023-10-24T23:57:55.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f466a4e7640 1 -- 172.21.15.107:0/3682589830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 msgr2=0x7f465000dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f466a4e7640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 0x7f465000dba0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f465c00a830 tx=0x7f465c01faa0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f466a4e7640 1 -- 172.21.15.107:0/3682589830 shutdown_connections 2023-10-24T23:57:55.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f466a4e7640 1 --2- 172.21.15.107:0/3682589830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 0x7f465000dba0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f466a4e7640 1 -- 172.21.15.107:0/3682589830 >> 172.21.15.107:0/3682589830 conn(0x7f46640c2950 msgr2=0x7f46640c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:55.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f466a4e7640 1 -- 172.21.15.107:0/3682589830 shutdown_connections 2023-10-24T23:57:55.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f466a4e7640 1 -- 172.21.15.107:0/3682589830 wait complete. 2023-10-24T23:57:55.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f466a4e7640 1 Processor -- start 2023-10-24T23:57:55.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.453+0000 7f466a4e7640 1 -- start start 2023-10-24T23:57:55.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f466a4e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 0x7f466414b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f466a4e7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4664146720 0x7f46640c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f466a4e7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f466406b400 0x7f46641e8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f466a4e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4664150bf0 con 0x7f4650005260 2023-10-24T23:57:55.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f466a4e7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46641506a0 con 0x7f466406b400 2023-10-24T23:57:55.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f466a4e7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46640c5750 con 0x7f4664146720 2023-10-24T23:57:55.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f4663fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 0x7f466414b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f4663fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 0x7f466414b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33630/0 (socket says 172.21.15.107:33630) 2023-10-24T23:57:55.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f4663fff640 1 -- 172.21.15.107:0/4235428357 learned_addr learned my addr 172.21.15.107:0/4235428357 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:55.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f465bfff640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4664146720 0x7f46640c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f4668a84640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f466406b400 0x7f46641e8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f4663fff640 1 -- 172.21.15.107:0/4235428357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4664146720 msgr2=0x7f46640c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f4663fff640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4664146720 0x7f46640c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f4663fff640 1 -- 172.21.15.107:0/4235428357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f466406b400 msgr2=0x7f46641e8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f4663fff640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f466406b400 0x7f46641e8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f4663fff640 1 -- 172.21.15.107:0/4235428357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664150910 con 0x7f4650005260 2023-10-24T23:57:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f4663fff640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 0x7f466414b270 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f465400e910 tx=0x7f465400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f46617fa640 1 -- 172.21.15.107:0/4235428357 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465400ab20 con 0x7f4650005260 2023-10-24T23:57:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f46617fa640 1 -- 172.21.15.107:0/4235428357 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465400ac80 con 0x7f4650005260 2023-10-24T23:57:55.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f466a4e7640 1 -- 172.21.15.107:0/4235428357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466414b7b0 con 0x7f4650005260 2023-10-24T23:57:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f46617fa640 1 -- 172.21.15.107:0/4235428357 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46540113f0 con 0x7f4650005260 2023-10-24T23:57:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f466a4e7640 1 -- 172.21.15.107:0/4235428357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466414c4b0 con 0x7f4650005260 2023-10-24T23:57:55.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.457+0000 7f466a4e7640 1 -- 172.21.15.107:0/4235428357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46640746e0 con 0x7f4650005260 2023-10-24T23:57:55.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.461+0000 7f46617fa640 1 -- 172.21.15.107:0/4235428357 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4654011550 con 0x7f4650005260 2023-10-24T23:57:55.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.461+0000 7f46617fa640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4644065e60 0x7f4644068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.461+0000 7f46617fa640 1 -- 172.21.15.107:0/4235428357 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f4654084130 con 0x7f4650005260 2023-10-24T23:57:55.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.461+0000 7f465bfff640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4644065e60 0x7f4644068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.461+0000 7f465bfff640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4644065e60 0x7f4644068320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4664146e50 tx=0x7f464c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:55.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.465+0000 7f46617fa640 1 -- 172.21.15.107:0/4235428357 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4654051260 con 0x7f4650005260 2023-10-24T23:57:55.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:55.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.649+0000 7f466a4e7640 1 -- 172.21.15.107:0/4235428357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46640c22a0 con 0x7f4650005260 2023-10-24T23:57:55.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.649+0000 7f46617fa640 1 -- 172.21.15.107:0/4235428357 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f4654022020 con 0x7f4650005260 2023-10-24T23:57:55.655 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:55.655 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:55.656 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:55.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 -- 172.21.15.107:0/4235428357 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4644065e60 msgr2=0x7f4644068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4644065e60 0x7f4644068320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4664146e50 tx=0x7f464c009040 comp rx=0 tx=0).stop 2023-10-24T23:57:55.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 -- 172.21.15.107:0/4235428357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 msgr2=0x7f466414b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 0x7f466414b270 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f465400e910 tx=0x7f465400edd0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 -- 172.21.15.107:0/4235428357 shutdown_connections 2023-10-24T23:57:55.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f466406b400 0x7f46641e8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4644065e60 0x7f4644068320 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4664146720 0x7f46640c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 --2- 172.21.15.107:0/4235428357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650005260 0x7f466414b270 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 -- 172.21.15.107:0/4235428357 >> 172.21.15.107:0/4235428357 conn(0x7f46640c2950 msgr2=0x7f46640c1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:55.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 -- 172.21.15.107:0/4235428357 shutdown_connections 2023-10-24T23:57:55.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.653+0000 7f466a4e7640 1 -- 172.21.15.107:0/4235428357 wait complete. 2023-10-24T23:57:55.673 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:57:55.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.717+0000 7fb118e51640 1 Processor -- start 2023-10-24T23:57:55.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.717+0000 7fb118e51640 1 -- start start 2023-10-24T23:57:55.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.717+0000 7fb118e51640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb114151d60 0x7fb1140a9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.717+0000 7fb118e51640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb11406a780 0x7fb114154d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.717+0000 7fb118e51640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1140c24b0 0x7fb114155280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.717+0000 7fb118e51640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1140aadc0 con 0x7fb1140c24b0 2023-10-24T23:57:55.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.717+0000 7fb118e51640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11406afe0 con 0x7fb11406a780 2023-10-24T23:57:55.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.717+0000 7fb118e51640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11406b330 con 0x7fb114151d60 2023-10-24T23:57:55.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb112d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1140c24b0 0x7fb114155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb112d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb1140c24b0 0x7fb114155280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33646/0 (socket says 172.21.15.107:33646) 2023-10-24T23:57:55.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb112d9d640 1 -- 172.21.15.107:0/3292699385 learned_addr learned my addr 172.21.15.107:0/3292699385 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:55.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb111d9b640 1 --2- 172.21.15.107:0/3292699385 >> v2:172.21.15.110:3300/0 conn(0x7fb11406a780 0x7fb114154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb11259c640 1 --2- 172.21.15.107:0/3292699385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb114151d60 0x7fb1140a9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb112d9d640 1 -- 172.21.15.107:0/3292699385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb114151d60 msgr2=0x7fb1140a9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:55.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb112d9d640 1 --2- 172.21.15.107:0/3292699385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb114151d60 0x7fb1140a9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb112d9d640 1 -- 172.21.15.107:0/3292699385 >> v2:172.21.15.110:3300/0 conn(0x7fb11406a780 msgr2=0x7fb114154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:55.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb112d9d640 1 --2- 172.21.15.107:0/3292699385 >> v2:172.21.15.110:3300/0 conn(0x7fb11406a780 0x7fb114154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb112d9d640 1 -- 172.21.15.107:0/3292699385 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1141557c0 con 0x7fb1140c24b0 2023-10-24T23:57:55.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb112d9d640 1 --2- 172.21.15.107:0/3292699385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1140c24b0 0x7fb114155280 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb11406d250 tx=0x7fb10c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e2b1a53670c86e server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:55.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb11159a640 1 -- 172.21.15.107:0/3292699385 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c0ab600 con 0x7fb1140c24b0 2023-10-24T23:57:55.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb11159a640 1 -- 172.21.15.107:0/3292699385 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c0a7050 con 0x7fb1140c24b0 2023-10-24T23:57:55.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb11159a640 1 -- 172.21.15.107:0/3292699385 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c0ab980 con 0x7fb1140c24b0 2023-10-24T23:57:55.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb118e51640 1 -- 172.21.15.107:0/3292699385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1140c24b0 msgr2=0x7fb114155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb118e51640 1 --2- 172.21.15.107:0/3292699385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1140c24b0 0x7fb114155280 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fb11406d250 tx=0x7fb10c01c500 comp rx=0 tx=0).stop 2023-10-24T23:57:55.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb118e51640 1 -- 172.21.15.107:0/3292699385 shutdown_connections 2023-10-24T23:57:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb118e51640 1 --2- 172.21.15.107:0/3292699385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1140c24b0 0x7fb114155280 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb118e51640 1 --2- 172.21.15.107:0/3292699385 >> v2:172.21.15.110:3300/0 conn(0x7fb11406a780 0x7fb114154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb118e51640 1 --2- 172.21.15.107:0/3292699385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb114151d60 0x7fb1140a9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb118e51640 1 -- 172.21.15.107:0/3292699385 >> 172.21.15.107:0/3292699385 conn(0x7fb1140c2950 msgr2=0x7fb1140c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:55.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb118e51640 1 -- 172.21.15.107:0/3292699385 shutdown_connections 2023-10-24T23:57:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.721+0000 7fb118e51640 1 -- 172.21.15.107:0/3292699385 wait complete. 2023-10-24T23:57:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb118e51640 1 Processor -- start 2023-10-24T23:57:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb118e51640 1 -- start start 2023-10-24T23:57:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb118e51640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1140c24b0 0x7fb114148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb118e51640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb114151d60 0x7fb11414fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb118e51640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1141502b0 0x7fb114155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb118e51640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1141f0270 con 0x7fb1140c24b0 2023-10-24T23:57:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb118e51640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1141f04c0 con 0x7fb1141502b0 2023-10-24T23:57:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb118e51640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1141f07a0 con 0x7fb114151d60 2023-10-24T23:57:55.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb11259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1140c24b0 0x7fb114148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb11259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1140c24b0 0x7fb114148ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33662/0 (socket says 172.21.15.107:33662) 2023-10-24T23:57:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb11259c640 1 -- 172.21.15.107:0/2396244390 learned_addr learned my addr 172.21.15.107:0/2396244390 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb111d9b640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb114151d60 0x7fb11414fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb112d9d640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1141502b0 0x7fb114155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb11259c640 1 -- 172.21.15.107:0/2396244390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb114151d60 msgr2=0x7fb11414fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb11259c640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb114151d60 0x7fb11414fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb11259c640 1 -- 172.21.15.107:0/2396244390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1141502b0 msgr2=0x7fb114155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb11259c640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1141502b0 0x7fb114155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb11259c640 1 -- 172.21.15.107:0/2396244390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c0094d0 con 0x7fb1140c24b0 2023-10-24T23:57:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb11259c640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1140c24b0 0x7fb114148ea0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fb104007e40 tx=0x7fb10400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:55.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb102ffd640 1 -- 172.21.15.107:0/2396244390 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104014620 con 0x7fb1140c24b0 2023-10-24T23:57:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb102ffd640 1 -- 172.21.15.107:0/2396244390 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10400ee90 con 0x7fb1140c24b0 2023-10-24T23:57:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb118e51640 1 -- 172.21.15.107:0/2396244390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1141f0a80 con 0x7fb1140c24b0 2023-10-24T23:57:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb102ffd640 1 -- 172.21.15.107:0/2396244390 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1040149c0 con 0x7fb1140c24b0 2023-10-24T23:57:55.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb118e51640 1 -- 172.21.15.107:0/2396244390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1141496e0 con 0x7fb1140c24b0 2023-10-24T23:57:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.725+0000 7fb118e51640 1 -- 172.21.15.107:0/2396244390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1140746e0 con 0x7fb1140c24b0 2023-10-24T23:57:55.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.729+0000 7fb102ffd640 1 -- 172.21.15.107:0/2396244390 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb104014b20 con 0x7fb1140c24b0 2023-10-24T23:57:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.729+0000 7fb102ffd640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0f4065e60 0x7fb0f4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.729+0000 7fb102ffd640 1 -- 172.21.15.107:0/2396244390 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7fb104085610 con 0x7fb1140c24b0 2023-10-24T23:57:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.729+0000 7fb111d9b640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0f4065e60 0x7fb0f4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.729+0000 7fb111d9b640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0f4065e60 0x7fb0f4068320 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb1141529d0 tx=0x7fb108008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:55.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.733+0000 7fb102ffd640 1 -- 172.21.15.107:0/2396244390 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb104052740 con 0x7fb1140c24b0 2023-10-24T23:57:55.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.913+0000 7fb118e51640 1 -- 172.21.15.107:0/2396244390 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1140a9520 con 0x7fb1140c24b0 2023-10-24T23:57:55.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.913+0000 7fb102ffd640 1 -- 172.21.15.107:0/2396244390 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fb104055d60 con 0x7fb1140c24b0 2023-10-24T23:57:55.920 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:55.920 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:55.921 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:55.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 -- 172.21.15.107:0/2396244390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0f4065e60 msgr2=0x7fb0f4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0f4065e60 0x7fb0f4068320 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb1141529d0 tx=0x7fb108008040 comp rx=0 tx=0).stop 2023-10-24T23:57:55.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 -- 172.21.15.107:0/2396244390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1140c24b0 msgr2=0x7fb114148ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1140c24b0 0x7fb114148ea0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fb104007e40 tx=0x7fb10400e4b0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 -- 172.21.15.107:0/2396244390 shutdown_connections 2023-10-24T23:57:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb1141502b0 0x7fb114155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb0f4065e60 0x7fb0f4068320 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb114151d60 0x7fb11414fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 --2- 172.21.15.107:0/2396244390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1140c24b0 0x7fb114148ea0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:55.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 -- 172.21.15.107:0/2396244390 >> 172.21.15.107:0/2396244390 conn(0x7fb1140c2950 msgr2=0x7fb114158160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:55.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 -- 172.21.15.107:0/2396244390 shutdown_connections 2023-10-24T23:57:55.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:55.917+0000 7fb118e51640 1 -- 172.21.15.107:0/2396244390 wait complete. 2023-10-24T23:57:55.937 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:57:57.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:57.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc426a80640 1 Processor -- start 2023-10-24T23:57:57.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc426a80640 1 -- start start 2023-10-24T23:57:57.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc426a80640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc4200c8100 0x7fc4200abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:57.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc426a80640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc420155680 con 0x7fc42006a4b0 2023-10-24T23:57:57.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc426a80640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc420154a70 con 0x7fc420152040 2023-10-24T23:57:57.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc426a80640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc420154d50 con 0x7fc4200c8100 2023-10-24T23:57:57.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc41ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc4200c8100 0x7fc4200abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:57.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc41ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc4200c8100 0x7fc4200abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33670/0 (socket says 172.21.15.107:33670) 2023-10-24T23:57:57.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc42501d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc42006a4b0 0x7fc4200ac240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58964/0 (socket says 172.21.15.107:58964) 2023-10-24T23:57:57.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc41ffff640 1 -- 172.21.15.107:0/4088163744 learned_addr learned my addr 172.21.15.107:0/4088163744 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:57.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc41ffff640 1 -- 172.21.15.107:0/4088163744 >> v1:172.21.15.110:6789/0 conn(0x7fc420152040 legacy=0x7fc4200c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:57.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc41ffff640 1 -- 172.21.15.107:0/4088163744 >> v1:172.21.15.107:6789/0 conn(0x7fc42006a4b0 legacy=0x7fc4200ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:57.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc41ffff640 1 -- 172.21.15.107:0/4088163744 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc420155030 con 0x7fc4200c8100 2023-10-24T23:57:57.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc41ffff640 1 --2- 172.21.15.107:0/4088163744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4200c8100 0x7fc4200abd00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009870 tx=0x7fc40c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c3267e931d3895 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:57.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.801+0000 7fc41f7fe640 1 -- 172.21.15.107:0/4088163744 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c0ad680 con 0x7fc4200c8100 2023-10-24T23:57:57.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc41f7fe640 1 -- 172.21.15.107:0/4088163744 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc40c0ad7e0 con 0x7fc4200c8100 2023-10-24T23:57:57.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc41f7fe640 1 -- 172.21.15.107:0/4088163744 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c0adb70 con 0x7fc4200c8100 2023-10-24T23:57:57.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 -- 172.21.15.107:0/4088163744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4200c8100 msgr2=0x7fc4200abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:57.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 --2- 172.21.15.107:0/4088163744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4200c8100 0x7fc4200abd00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009870 tx=0x7fc40c01c470 comp rx=0 tx=0).stop 2023-10-24T23:57:57.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 -- 172.21.15.107:0/4088163744 shutdown_connections 2023-10-24T23:57:57.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 --2- 172.21.15.107:0/4088163744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4200c8100 0x7fc4200abd00 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:57.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 -- 172.21.15.107:0/4088163744 >> 172.21.15.107:0/4088163744 conn(0x7fc4200ca200 msgr2=0x7fc4200ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:57.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 -- 172.21.15.107:0/4088163744 shutdown_connections 2023-10-24T23:57:57.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 -- 172.21.15.107:0/4088163744 wait complete. 2023-10-24T23:57:57.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 Processor -- start 2023-10-24T23:57:57.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 -- start start 2023-10-24T23:57:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc42006a4b0 0x7fc4200c1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4200c8100 0x7fc4200c0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc420152040 0x7fc420148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4201f0b70 con 0x7fc42006a4b0 2023-10-24T23:57:57.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4201f1060 con 0x7fc4200c8100 2023-10-24T23:57:57.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4201482b0 con 0x7fc420152040 2023-10-24T23:57:57.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc42481c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc42006a4b0 0x7fc4200c1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:57.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc42481c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc42006a4b0 0x7fc4200c1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33672/0 (socket says 172.21.15.107:33672) 2023-10-24T23:57:57.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc42481c640 1 -- 172.21.15.107:0/1416297233 learned_addr learned my addr 172.21.15.107:0/1416297233 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:57.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc42481c640 1 -- 172.21.15.107:0/1416297233 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc420152040 msgr2=0x7fc420148a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:57.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc42481c640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc420152040 0x7fc420148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:57.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc42481c640 1 -- 172.21.15.107:0/1416297233 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4200c8100 msgr2=0x7fc4200c0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:57.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc42481c640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4200c8100 0x7fc4200c0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:57.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc42481c640 1 -- 172.21.15.107:0/1416297233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c0094d0 con 0x7fc42006a4b0 2023-10-24T23:57:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc42481c640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc42006a4b0 0x7fc4200c1240 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc41400c450 tx=0x7fc41400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc41d7fa640 1 -- 172.21.15.107:0/1416297233 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414012830 con 0x7fc42006a4b0 2023-10-24T23:57:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc41d7fa640 1 -- 172.21.15.107:0/1416297233 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4140129b0 con 0x7fc42006a4b0 2023-10-24T23:57:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 -- 172.21.15.107:0/1416297233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4201f0d80 con 0x7fc42006a4b0 2023-10-24T23:57:57.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc41d7fa640 1 -- 172.21.15.107:0/1416297233 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414012d60 con 0x7fc42006a4b0 2023-10-24T23:57:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.805+0000 7fc426a80640 1 -- 172.21.15.107:0/1416297233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc420147480 con 0x7fc42006a4b0 2023-10-24T23:57:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.809+0000 7fc426a80640 1 -- 172.21.15.107:0/1416297233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4200746e0 con 0x7fc42006a4b0 2023-10-24T23:57:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.809+0000 7fc41d7fa640 1 -- 172.21.15.107:0/1416297233 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc41401c460 con 0x7fc42006a4b0 2023-10-24T23:57:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.809+0000 7fc41d7fa640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc404065e60 0x7fc404068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:57.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.809+0000 7fc41d7fa640 1 -- 172.21.15.107:0/1416297233 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7fc414083cc0 con 0x7fc42006a4b0 2023-10-24T23:57:57.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.809+0000 7fc41ffff640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc404065e60 0x7fc404068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:57.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.809+0000 7fc41ffff640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc404065e60 0x7fc404068320 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0af000 tx=0x7fc40c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:57.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.813+0000 7fc41d7fa640 1 -- 172.21.15.107:0/1416297233 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc41408a080 con 0x7fc42006a4b0 2023-10-24T23:57:57.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:57:58.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.997+0000 7fc426a80640 1 -- 172.21.15.107:0/1416297233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc420148720 con 0x7fc42006a4b0 2023-10-24T23:57:58.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:57.997+0000 7fc41d7fa640 1 -- 172.21.15.107:0/1416297233 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fc414050d70 con 0x7fc42006a4b0 2023-10-24T23:57:58.004 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:58.004 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:58.005 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:58.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 -- 172.21.15.107:0/1416297233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc404065e60 msgr2=0x7fc404068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:58.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc404065e60 0x7fc404068320 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0af000 tx=0x7fc40c008040 comp rx=0 tx=0).stop 2023-10-24T23:57:58.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 -- 172.21.15.107:0/1416297233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc42006a4b0 msgr2=0x7fc4200c1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:58.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc42006a4b0 0x7fc4200c1240 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc41400c450 tx=0x7fc41400c910 comp rx=0 tx=0).stop 2023-10-24T23:57:58.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 -- 172.21.15.107:0/1416297233 shutdown_connections 2023-10-24T23:57:58.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc404065e60 0x7fc404068320 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc420152040 0x7fc420148a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4200c8100 0x7fc4200c0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 --2- 172.21.15.107:0/1416297233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc42006a4b0 0x7fc4200c1240 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 -- 172.21.15.107:0/1416297233 >> 172.21.15.107:0/1416297233 conn(0x7fc4200ca200 msgr2=0x7fc4200c4850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:58.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 -- 172.21.15.107:0/1416297233 shutdown_connections 2023-10-24T23:57:58.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.001+0000 7fc426a80640 1 -- 172.21.15.107:0/1416297233 wait complete. 2023-10-24T23:57:58.023 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:57:58.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0b107e640 1 Processor -- start 2023-10-24T23:57:58.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0b107e640 1 -- start start 2023-10-24T23:57:58.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0b107e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0ac06af70 0x7fc0ac06c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:58.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0b107e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0ac06a230 0x7fc0ac1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:58.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0b107e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ac0c1860 con 0x7fc0ac14b600 2023-10-24T23:57:58.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0b107e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0ac154d50 con 0x7fc0ac06af70 2023-10-24T23:57:58.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0b107e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac14b9e0 con 0x7fc0ac06a230 2023-10-24T23:57:58.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0aad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc0ac14b600 0x7fc0ac151fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58970/0 (socket says 172.21.15.107:58970) 2023-10-24T23:57:58.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0aad9d640 1 -- 172.21.15.107:0/1658840260 learned_addr learned my addr 172.21.15.107:0/1658840260 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:57:58.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0aa59c640 1 --2- 172.21.15.107:0/1658840260 >> v2:172.21.15.107:3300/0 conn(0x7fc0ac06af70 0x7fc0ac06c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:58.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0aa59c640 1 -- 172.21.15.107:0/1658840260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0ac06a230 msgr2=0x7fc0ac1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:57:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0ab59e640 1 --2- 172.21.15.107:0/1658840260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0ac06a230 0x7fc0ac1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0aa59c640 1 --2- 172.21.15.107:0/1658840260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0ac06a230 0x7fc0ac1551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0aa59c640 1 -- 172.21.15.107:0/1658840260 >> v1:172.21.15.107:6789/0 conn(0x7fc0ac14b600 legacy=0x7fc0ac151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0aa59c640 1 -- 172.21.15.107:0/1658840260 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ac0c1ab0 con 0x7fc0ac06af70 2023-10-24T23:57:58.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0a9d9b640 1 -- 172.21.15.107:0/1658840260 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288223834 0 0) 0x7fc0ac0c1860 con 0x7fc0ac14b600 2023-10-24T23:57:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.057+0000 7fc0aa59c640 1 --2- 172.21.15.107:0/1658840260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ac06af70 0x7fc0ac06c4c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fc0a00097c0 tx=0x7fc0a001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0da65bf21358cee server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0a9d9b640 1 -- 172.21.15.107:0/1658840260 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a00ac620 con 0x7fc0ac06af70 2023-10-24T23:57:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0a9d9b640 1 -- 172.21.15.107:0/1658840260 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a00ac780 con 0x7fc0ac06af70 2023-10-24T23:57:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0a9d9b640 1 -- 172.21.15.107:0/1658840260 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a00acb10 con 0x7fc0ac06af70 2023-10-24T23:57:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 -- 172.21.15.107:0/1658840260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ac06af70 msgr2=0x7fc0ac06c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 --2- 172.21.15.107:0/1658840260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ac06af70 0x7fc0ac06c4c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fc0a00097c0 tx=0x7fc0a001c3c0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 -- 172.21.15.107:0/1658840260 shutdown_connections 2023-10-24T23:57:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 --2- 172.21.15.107:0/1658840260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0ac06a230 0x7fc0ac1551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 --2- 172.21.15.107:0/1658840260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ac06af70 0x7fc0ac06c4c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 -- 172.21.15.107:0/1658840260 >> 172.21.15.107:0/1658840260 conn(0x7fc0ac0c2950 msgr2=0x7fc0ac0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 -- 172.21.15.107:0/1658840260 shutdown_connections 2023-10-24T23:57:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 -- 172.21.15.107:0/1658840260 wait complete. 2023-10-24T23:57:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 Processor -- start 2023-10-24T23:57:58.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 -- start start 2023-10-24T23:57:58.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0ac06a230 0x7fc0ac143c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:58.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ac06af70 0x7fc0ac1eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:58.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ac14b600 0x7fc0ac06b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:58.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac1eefa0 con 0x7fc0ac14b600 2023-10-24T23:57:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac1538f0 con 0x7fc0ac06af70 2023-10-24T23:57:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac1eaec0 con 0x7fc0ac06a230 2023-10-24T23:57:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0ab59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ac14b600 0x7fc0ac06b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0ab59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ac14b600 0x7fc0ac06b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33684/0 (socket says 172.21.15.107:33684) 2023-10-24T23:57:58.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0ab59e640 1 -- 172.21.15.107:0/1275091520 learned_addr learned my addr 172.21.15.107:0/1275091520 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:57:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0aad9d640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0ac06a230 0x7fc0ac143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0aa59c640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ac06af70 0x7fc0ac1eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0ab59e640 1 -- 172.21.15.107:0/1275091520 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0ac06a230 msgr2=0x7fc0ac143c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0ab59e640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0ac06a230 0x7fc0ac143c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0ab59e640 1 -- 172.21.15.107:0/1275091520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ac06af70 msgr2=0x7fc0ac1eea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:58.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0ab59e640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ac06af70 0x7fc0ac1eea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0ab59e640 1 -- 172.21.15.107:0/1275091520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a0009420 con 0x7fc0ac14b600 2023-10-24T23:57:58.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0aa59c640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ac06af70 0x7fc0ac1eea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:58.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0aad9d640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0ac06a230 0x7fc0ac143c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:57:58.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0ab59e640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ac14b600 0x7fc0ac06b5f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400cd70 tx=0x7fc0a400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:58.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0937fe640 1 -- 172.21.15.107:0/1275091520 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4003ba0 con 0x7fc0ac14b600 2023-10-24T23:57:58.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0937fe640 1 -- 172.21.15.107:0/1275091520 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a4003d00 con 0x7fc0ac14b600 2023-10-24T23:57:58.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 -- 172.21.15.107:0/1275091520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ac153b40 con 0x7fc0ac14b600 2023-10-24T23:57:58.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0937fe640 1 -- 172.21.15.107:0/1275091520 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4004240 con 0x7fc0ac14b600 2023-10-24T23:57:58.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.061+0000 7fc0b107e640 1 -- 172.21.15.107:0/1275091520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ac154100 con 0x7fc0ac14b600 2023-10-24T23:57:58.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.065+0000 7fc0b107e640 1 -- 172.21.15.107:0/1275091520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0ac0c05c0 con 0x7fc0ac14b600 2023-10-24T23:57:58.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.065+0000 7fc0937fe640 1 -- 172.21.15.107:0/1275091520 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc0a40043a0 con 0x7fc0ac14b600 2023-10-24T23:57:58.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.065+0000 7fc0937fe640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc098065e90 0x7fc098068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:57:58.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.065+0000 7fc0937fe640 1 -- 172.21.15.107:0/1275091520 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7fc0a4012070 con 0x7fc0ac14b600 2023-10-24T23:57:58.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.065+0000 7fc0aad9d640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc098065e90 0x7fc098068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:57:58.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.065+0000 7fc0aad9d640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc098065e90 0x7fc098068350 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc09c000f70 tx=0x7fc09c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:57:58.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.069+0000 7fc0937fe640 1 -- 172.21.15.107:0/1275091520 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc0a4051820 con 0x7fc0ac14b600 2023-10-24T23:57:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.257+0000 7fc0b107e640 1 -- 172.21.15.107:0/1275091520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0ac1eb030 con 0x7fc0ac14b600 2023-10-24T23:57:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.257+0000 7fc0937fe640 1 -- 172.21.15.107:0/1275091520 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fc0a4054e40 con 0x7fc0ac14b600 2023-10-24T23:57:58.263 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:57:58.263 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:57:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:57:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 -- 172.21.15.107:0/1275091520 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc098065e90 msgr2=0x7fc098068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc098065e90 0x7fc098068350 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc09c000f70 tx=0x7fc09c009040 comp rx=0 tx=0).stop 2023-10-24T23:57:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 -- 172.21.15.107:0/1275091520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ac14b600 msgr2=0x7fc0ac06b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:57:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ac14b600 0x7fc0ac06b5f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400cd70 tx=0x7fc0a400a660 comp rx=0 tx=0).stop 2023-10-24T23:57:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 -- 172.21.15.107:0/1275091520 shutdown_connections 2023-10-24T23:57:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0ac14b600 0x7fc0ac06b5f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0ac06af70 0x7fc0ac1eea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc098065e90 0x7fc098068350 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 --2- 172.21.15.107:0/1275091520 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0ac06a230 0x7fc0ac143c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:57:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 -- 172.21.15.107:0/1275091520 >> 172.21.15.107:0/1275091520 conn(0x7fc0ac0c2950 msgr2=0x7fc0ac145b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:57:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 -- 172.21.15.107:0/1275091520 shutdown_connections 2023-10-24T23:57:58.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:57:58.261+0000 7fc0b107e640 1 -- 172.21.15.107:0/1275091520 wait complete. 2023-10-24T23:57:58.283 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:58:00.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:00.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc62d31640 1 Processor -- start 2023-10-24T23:58:00.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc62d31640 1 -- start start 2023-10-24T23:58:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc62d31640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc5c14eb70 0x7fbc5c0c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc62d31640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbc5c070b00 0x7fbc5c1514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc62d31640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbc5c151a20 0x7fbc5c06df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc62d31640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc5c071700 con 0x7fbc5c070b00 2023-10-24T23:58:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc62d31640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc5c0c22f0 con 0x7fbc5c151a20 2023-10-24T23:58:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc62d31640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0c25d0 con 0x7fbc5c14eb70 2023-10-24T23:58:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc5bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbc5c070b00 0x7fbc5c1514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc5bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbc5c070b00 0x7fbc5c1514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33688/0 (socket says 172.21.15.107:33688) 2023-10-24T23:58:00.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc5bfff640 1 -- 172.21.15.107:0/304177681 learned_addr learned my addr 172.21.15.107:0/304177681 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:00.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc612ce640 1 --2- 172.21.15.107:0/304177681 >> v2:172.21.15.110:3300/0 conn(0x7fbc5c151a20 0x7fbc5c06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:00.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc60acd640 1 --2- 172.21.15.107:0/304177681 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc5c14eb70 0x7fbc5c0c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:00.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc5bfff640 1 -- 172.21.15.107:0/304177681 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc5c14eb70 msgr2=0x7fbc5c0c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:00.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc5bfff640 1 --2- 172.21.15.107:0/304177681 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc5c14eb70 0x7fbc5c0c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc5bfff640 1 -- 172.21.15.107:0/304177681 >> v2:172.21.15.110:3300/0 conn(0x7fbc5c151a20 msgr2=0x7fbc5c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:00.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc5bfff640 1 --2- 172.21.15.107:0/304177681 >> v2:172.21.15.110:3300/0 conn(0x7fbc5c151a20 0x7fbc5c06df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc5bfff640 1 -- 172.21.15.107:0/304177681 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c0c2be0 con 0x7fbc5c070b00 2023-10-24T23:58:00.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc612ce640 1 --2- 172.21.15.107:0/304177681 >> v2:172.21.15.110:3300/0 conn(0x7fbc5c151a20 0x7fbc5c06df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:00.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.153+0000 7fbc60acd640 1 --2- 172.21.15.107:0/304177681 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc5c14eb70 0x7fbc5c0c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:00.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc5bfff640 1 --2- 172.21.15.107:0/304177681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc5c070b00 0x7fbc5c1514e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c072a20 tx=0x7fbc4801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a230e03c64e238e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:00.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc5b7fe640 1 -- 172.21.15.107:0/304177681 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc480ab660 con 0x7fbc5c070b00 2023-10-24T23:58:00.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc5b7fe640 1 -- 172.21.15.107:0/304177681 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc480ab7c0 con 0x7fbc5c070b00 2023-10-24T23:58:00.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc5b7fe640 1 -- 172.21.15.107:0/304177681 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc480abb50 con 0x7fbc5c070b00 2023-10-24T23:58:00.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 -- 172.21.15.107:0/304177681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc5c070b00 msgr2=0x7fbc5c1514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:00.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 --2- 172.21.15.107:0/304177681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc5c070b00 0x7fbc5c1514e0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c072a20 tx=0x7fbc4801c500 comp rx=0 tx=0).stop 2023-10-24T23:58:00.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 -- 172.21.15.107:0/304177681 shutdown_connections 2023-10-24T23:58:00.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 --2- 172.21.15.107:0/304177681 >> v2:172.21.15.110:3300/0 conn(0x7fbc5c151a20 0x7fbc5c06df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 --2- 172.21.15.107:0/304177681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc5c070b00 0x7fbc5c1514e0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 --2- 172.21.15.107:0/304177681 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc5c14eb70 0x7fbc5c0c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 -- 172.21.15.107:0/304177681 >> 172.21.15.107:0/304177681 conn(0x7fbc5c0c39a0 msgr2=0x7fbc5c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:00.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 -- 172.21.15.107:0/304177681 shutdown_connections 2023-10-24T23:58:00.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 -- 172.21.15.107:0/304177681 wait complete. 2023-10-24T23:58:00.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 Processor -- start 2023-10-24T23:58:00.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 -- start start 2023-10-24T23:58:00.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc5c070b00 0x7fbc5c14f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc5c14eb70 0x7fbc5c137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc5c151a20 0x7fbc5c138150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c150070 con 0x7fbc5c070b00 2023-10-24T23:58:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c14c4f0 con 0x7fbc5c14eb70 2023-10-24T23:58:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c14c7d0 con 0x7fbc5c151a20 2023-10-24T23:58:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc60acd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc5c070b00 0x7fbc5c14f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc60acd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc5c070b00 0x7fbc5c14f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33704/0 (socket says 172.21.15.107:33704) 2023-10-24T23:58:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc60acd640 1 -- 172.21.15.107:0/2134394611 learned_addr learned my addr 172.21.15.107:0/2134394611 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc60acd640 1 -- 172.21.15.107:0/2134394611 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc5c151a20 msgr2=0x7fbc5c138150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:00.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc60acd640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc5c151a20 0x7fbc5c138150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc60acd640 1 -- 172.21.15.107:0/2134394611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc5c14eb70 msgr2=0x7fbc5c137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:00.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc60acd640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc5c14eb70 0x7fbc5c137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc60acd640 1 -- 172.21.15.107:0/2134394611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc480094d0 con 0x7fbc5c070b00 2023-10-24T23:58:00.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc60acd640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc5c070b00 0x7fbc5c14f5b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000c7c0 tx=0x7fbc5000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:00.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc597fa640 1 -- 172.21.15.107:0/2134394611 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50007a30 con 0x7fbc5c070b00 2023-10-24T23:58:00.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc597fa640 1 -- 172.21.15.107:0/2134394611 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc50007b90 con 0x7fbc5c070b00 2023-10-24T23:58:00.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 -- 172.21.15.107:0/2134394611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c14cab0 con 0x7fbc5c070b00 2023-10-24T23:58:00.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc597fa640 1 -- 172.21.15.107:0/2134394611 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5000ce70 con 0x7fbc5c070b00 2023-10-24T23:58:00.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.157+0000 7fbc62d31640 1 -- 172.21.15.107:0/2134394611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c14faf0 con 0x7fbc5c070b00 2023-10-24T23:58:00.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.161+0000 7fbc62d31640 1 -- 172.21.15.107:0/2134394611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c0746e0 con 0x7fbc5c070b00 2023-10-24T23:58:00.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.161+0000 7fbc597fa640 1 -- 172.21.15.107:0/2134394611 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbc500113f0 con 0x7fbc5c070b00 2023-10-24T23:58:00.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.161+0000 7fbc597fa640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc40065e60 0x7fbc40068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.161+0000 7fbc597fa640 1 -- 172.21.15.107:0/2134394611 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7fbc50083e60 con 0x7fbc5c070b00 2023-10-24T23:58:00.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.161+0000 7fbc5bfff640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc40065e60 0x7fbc40068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.161+0000 7fbc5bfff640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc40065e60 0x7fbc40068320 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbc480af000 tx=0x7fbc48008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.165+0000 7fbc597fa640 1 -- 172.21.15.107:0/2134394611 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbc50050fb0 con 0x7fbc5c070b00 2023-10-24T23:58:00.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:00.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.349+0000 7fbc62d31640 1 -- 172.21.15.107:0/2134394611 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc5c0c1460 con 0x7fbc5c070b00 2023-10-24T23:58:00.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.349+0000 7fbc597fa640 1 -- 172.21.15.107:0/2134394611 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fbc5001f090 con 0x7fbc5c070b00 2023-10-24T23:58:00.357 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:00.357 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:00.357 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:58:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 -- 172.21.15.107:0/2134394611 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc40065e60 msgr2=0x7fbc40068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc40065e60 0x7fbc40068320 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fbc480af000 tx=0x7fbc48008040 comp rx=0 tx=0).stop 2023-10-24T23:58:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 -- 172.21.15.107:0/2134394611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc5c070b00 msgr2=0x7fbc5c14f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc5c070b00 0x7fbc5c14f5b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000c7c0 tx=0x7fbc5000cc80 comp rx=0 tx=0).stop 2023-10-24T23:58:00.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 -- 172.21.15.107:0/2134394611 shutdown_connections 2023-10-24T23:58:00.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc40065e60 0x7fbc40068320 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc5c151a20 0x7fbc5c138150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc5c14eb70 0x7fbc5c137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 --2- 172.21.15.107:0/2134394611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc5c070b00 0x7fbc5c14f5b0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 -- 172.21.15.107:0/2134394611 >> 172.21.15.107:0/2134394611 conn(0x7fbc5c0c39a0 msgr2=0x7fbc5c0c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:00.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 -- 172.21.15.107:0/2134394611 shutdown_connections 2023-10-24T23:58:00.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.353+0000 7fbc62d31640 1 -- 172.21.15.107:0/2134394611 wait complete. 2023-10-24T23:58:00.374 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:58:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.393+0000 7f694e22c640 1 Processor -- start 2023-10-24T23:58:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.393+0000 7f694e22c640 1 -- start start 2023-10-24T23:58:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.393+0000 7f694e22c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f694806f490 0x7f694814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.393+0000 7f694e22c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6948152b30 0x7f6948152f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.393+0000 7f694e22c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6948155cd0 con 0x7f694814c470 2023-10-24T23:58:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.393+0000 7f694e22c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69481526e0 con 0x7f6948152b30 2023-10-24T23:58:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.393+0000 7f694e22c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69480c4380 con 0x7f694806f490 2023-10-24T23:58:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f6947fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6948152b30 0x7f6948152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f6947fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6948152b30 0x7f6948152f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33716/0 (socket says 172.21.15.107:33716) 2023-10-24T23:58:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f6947fff640 1 -- 172.21.15.107:0/3010546779 learned_addr learned my addr 172.21.15.107:0/3010546779 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f6946ffd640 1 --2- 172.21.15.107:0/3010546779 >> v2:172.21.15.110:3300/0 conn(0x7f694806f490 0x7f694814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:00.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f6947fff640 1 -- 172.21.15.107:0/3010546779 >> v2:172.21.15.110:3300/0 conn(0x7f694806f490 msgr2=0x7f694814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:00.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f6947fff640 1 --2- 172.21.15.107:0/3010546779 >> v2:172.21.15.110:3300/0 conn(0x7f694806f490 0x7f694814c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f6947fff640 1 -- 172.21.15.107:0/3010546779 >> v1:172.21.15.110:6789/0 conn(0x7f694814c470 legacy=0x7f6948071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:00.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f6947fff640 1 -- 172.21.15.107:0/3010546779 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69480c48b0 con 0x7f6948152b30 2023-10-24T23:58:00.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f6946ffd640 1 --2- 172.21.15.107:0/3010546779 >> v2:172.21.15.110:3300/0 conn(0x7f694806f490 0x7f694814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:00.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f6947fff640 1 --2- 172.21.15.107:0/3010546779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6948152b30 0x7f6948152f40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f694806a7e0 tx=0x7f694001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b205f25e8334b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:00.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f69467fc640 1 -- 172.21.15.107:0/3010546779 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69400a5620 con 0x7f6948152b30 2023-10-24T23:58:00.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f69467fc640 1 -- 172.21.15.107:0/3010546779 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69400a5780 con 0x7f6948152b30 2023-10-24T23:58:00.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f69467fc640 1 -- 172.21.15.107:0/3010546779 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69400a5b10 con 0x7f6948152b30 2023-10-24T23:58:00.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f694e22c640 1 -- 172.21.15.107:0/3010546779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6948152b30 msgr2=0x7f6948152f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:00.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f694e22c640 1 --2- 172.21.15.107:0/3010546779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6948152b30 0x7f6948152f40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f694806a7e0 tx=0x7f694001c870 comp rx=0 tx=0).stop 2023-10-24T23:58:00.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f694e22c640 1 -- 172.21.15.107:0/3010546779 shutdown_connections 2023-10-24T23:58:00.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f694e22c640 1 --2- 172.21.15.107:0/3010546779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6948152b30 0x7f6948152f40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f694e22c640 1 --2- 172.21.15.107:0/3010546779 >> v2:172.21.15.110:3300/0 conn(0x7f694806f490 0x7f694814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f694e22c640 1 -- 172.21.15.107:0/3010546779 >> 172.21.15.107:0/3010546779 conn(0x7f6948070f60 msgr2=0x7f6948071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:00.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f694e22c640 1 -- 172.21.15.107:0/3010546779 shutdown_connections 2023-10-24T23:58:00.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.397+0000 7f694e22c640 1 -- 172.21.15.107:0/3010546779 wait complete. 2023-10-24T23:58:00.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f694e22c640 1 Processor -- start 2023-10-24T23:58:00.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f694e22c640 1 -- start start 2023-10-24T23:58:00.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f694e22c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f694806f490 0x7f694813def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f694e22c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f694814c470 0x7f694813d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f694e22c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6948152b30 0x7f694813adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f694e22c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694813c890 con 0x7f694814c470 2023-10-24T23:58:00.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f694e22c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69481eac10 con 0x7f6948152b30 2023-10-24T23:58:00.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f694e22c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69481ea040 con 0x7f694806f490 2023-10-24T23:58:00.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f6946ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f694814c470 0x7f694813d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:00.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f6946ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f694814c470 0x7f694813d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33718/0 (socket says 172.21.15.107:33718) 2023-10-24T23:58:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f6946ffd640 1 -- 172.21.15.107:0/2949485619 learned_addr learned my addr 172.21.15.107:0/2949485619 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f69477fe640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f694806f490 0x7f694813def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f6947fff640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6948152b30 0x7f694813adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f6946ffd640 1 -- 172.21.15.107:0/2949485619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f694806f490 msgr2=0x7f694813def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f6946ffd640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f694806f490 0x7f694813def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f6946ffd640 1 -- 172.21.15.107:0/2949485619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6948152b30 msgr2=0x7f694813adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f6946ffd640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6948152b30 0x7f694813adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f6946ffd640 1 -- 172.21.15.107:0/2949485619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6940009b30 con 0x7f694814c470 2023-10-24T23:58:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f6946ffd640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f694814c470 0x7f694813d3c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f693c00c7c0 tx=0x7f693c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f692ffff640 1 -- 172.21.15.107:0/2949485619 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c007a30 con 0x7f694814c470 2023-10-24T23:58:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f692ffff640 1 -- 172.21.15.107:0/2949485619 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693c007b90 con 0x7f694814c470 2023-10-24T23:58:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f694e22c640 1 -- 172.21.15.107:0/2949485619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69481ea2e0 con 0x7f694814c470 2023-10-24T23:58:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f692ffff640 1 -- 172.21.15.107:0/2949485619 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c00ce70 con 0x7f694814c470 2023-10-24T23:58:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f694e22c640 1 -- 172.21.15.107:0/2949485619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69481ea8a0 con 0x7f694814c470 2023-10-24T23:58:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f694e22c640 1 -- 172.21.15.107:0/2949485619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69481ead80 con 0x7f694814c470 2023-10-24T23:58:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f692ffff640 1 -- 172.21.15.107:0/2949485619 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f693c0193f0 con 0x7f694814c470 2023-10-24T23:58:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f692ffff640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6934065e60 0x7f6934068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f692ffff640 1 -- 172.21.15.107:0/2949485619 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7f693c084370 con 0x7f694814c470 2023-10-24T23:58:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f69477fe640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6934065e60 0x7f6934068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.401+0000 7f69477fe640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6934065e60 0x7f6934068320 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6938001600 tx=0x7f6938009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.405+0000 7f692ffff640 1 -- 172.21.15.107:0/2949485619 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f693c051300 con 0x7f694814c470 2023-10-24T23:58:00.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.585+0000 7f694e22c640 1 -- 172.21.15.107:0/2949485619 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69480746e0 con 0x7f694814c470 2023-10-24T23:58:00.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.585+0000 7f692ffff640 1 -- 172.21.15.107:0/2949485619 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f693c054920 con 0x7f694814c470 2023-10-24T23:58:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:00.593 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:00.593 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:58:00.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 -- 172.21.15.107:0/2949485619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6934065e60 msgr2=0x7f6934068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:00.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6934065e60 0x7f6934068320 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6938001600 tx=0x7f6938009040 comp rx=0 tx=0).stop 2023-10-24T23:58:00.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 -- 172.21.15.107:0/2949485619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f694814c470 msgr2=0x7f694813d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:00.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f694814c470 0x7f694813d3c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f693c00c7c0 tx=0x7f693c00cc80 comp rx=0 tx=0).stop 2023-10-24T23:58:00.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 -- 172.21.15.107:0/2949485619 shutdown_connections 2023-10-24T23:58:00.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6948152b30 0x7f694813adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f694814c470 0x7f694813d3c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6934065e60 0x7f6934068320 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 --2- 172.21.15.107:0/2949485619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f694806f490 0x7f694813def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:00.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 -- 172.21.15.107:0/2949485619 >> 172.21.15.107:0/2949485619 conn(0x7f6948070f60 msgr2=0x7f6948151680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:00.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 -- 172.21.15.107:0/2949485619 shutdown_connections 2023-10-24T23:58:00.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:00.589+0000 7f694e22c640 1 -- 172.21.15.107:0/2949485619 wait complete. 2023-10-24T23:58:00.610 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:58:02.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:02.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0f1e16640 1 Processor -- start 2023-10-24T23:58:02.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0f1e16640 1 -- start start 2023-10-24T23:58:02.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0f1e16640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec06c8e0 0x7ff0ec154810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0f1e16640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ec06ccc0 con 0x7ff0ec1584a0 2023-10-24T23:58:02.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0f1e16640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ec0c3aa0 con 0x7ff0ec0a9600 2023-10-24T23:58:02.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0f1e16640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec158080 con 0x7ff0ec06c8e0 2023-10-24T23:58:02.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0ebfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff0ec1584a0 0x7ff0ec15f130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58978/0 (socket says 172.21.15.107:58978) 2023-10-24T23:58:02.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0ebfff640 1 -- 172.21.15.107:0/314221698 learned_addr learned my addr 172.21.15.107:0/314221698 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:02.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392901232 0 0) 0x7ff0ec06ccc0 con 0x7ff0ec1584a0 2023-10-24T23:58:02.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0cc003530 con 0x7ff0ec1584a0 2023-10-24T23:58:02.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eb7fe640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec06c8e0 0x7ff0ec154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 727507941 0 0) 0x7ff0cc003530 con 0x7ff0ec1584a0 2023-10-24T23:58:02.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec06c8e0 msgr2=0x7ff0ec154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec06c8e0 0x7ff0ec154810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 >> v1:172.21.15.110:6789/0 conn(0x7ff0ec0a9600 legacy=0x7ff0ec0aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec0c3200 con 0x7ff0ec1584a0 2023-10-24T23:58:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7ff0e4001a00 con 0x7ff0ec1584a0 2023-10-24T23:58:02.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 >> v1:172.21.15.107:6789/0 conn(0x7ff0ec1584a0 legacy=0x7ff0ec15f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0039b0 0x7ff0cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec06c8e0 0x7ff0cc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc004d10 0x7ff0cc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec158080 con 0x7ff0cc0039b0 2023-10-24T23:58:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc00bb70 con 0x7ff0cc004d10 2023-10-24T23:58:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc00bd30 con 0x7ff0ec06c8e0 2023-10-24T23:58:02.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eb7fe640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0039b0 0x7ff0cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0e3fff640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec06c8e0 0x7ff0cc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0ebfff640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc004d10 0x7ff0cc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eb7fe640 1 -- 172.21.15.107:0/314221698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec06c8e0 msgr2=0x7ff0cc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:02.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eb7fe640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec06c8e0 0x7ff0cc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eb7fe640 1 -- 172.21.15.107:0/314221698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc004d10 msgr2=0x7ff0cc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:02.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eb7fe640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc004d10 0x7ff0cc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.501+0000 7ff0eb7fe640 1 -- 172.21.15.107:0/314221698 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0cc003530 con 0x7ff0cc0039b0 2023-10-24T23:58:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0eb7fe640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0039b0 0x7ff0cc003dd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc00ba30 tx=0x7ff0dc01c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bfaa0241665b8e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc0ac5f0 con 0x7ff0cc0039b0 2023-10-24T23:58:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0dc0ac750 con 0x7ff0cc0039b0 2023-10-24T23:58:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0eaffd640 1 -- 172.21.15.107:0/314221698 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc0acb00 con 0x7ff0cc0039b0 2023-10-24T23:58:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0f1e16640 1 -- 172.21.15.107:0/314221698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0039b0 msgr2=0x7ff0cc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0f1e16640 1 --2- 172.21.15.107:0/314221698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0039b0 0x7ff0cc003dd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc00ba30 tx=0x7ff0dc01c650 comp rx=0 tx=0).stop 2023-10-24T23:58:02.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0ebfff640 1 -- 172.21.15.107:0/314221698 reap_dead start 2023-10-24T23:58:02.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0f1e16640 1 -- 172.21.15.107:0/314221698 shutdown_connections 2023-10-24T23:58:02.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0f1e16640 1 -- 172.21.15.107:0/314221698 >> 172.21.15.107:0/314221698 conn(0x7ff0ec0c81d0 msgr2=0x7ff0ec0c85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:02.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0f1e16640 1 -- 172.21.15.107:0/314221698 shutdown_connections 2023-10-24T23:58:02.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0f1e16640 1 -- 172.21.15.107:0/314221698 wait complete. 2023-10-24T23:58:02.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0f1e16640 1 Processor -- start 2023-10-24T23:58:02.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.505+0000 7ff0f1e16640 1 -- start start 2023-10-24T23:58:02.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0f1e16640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc0039b0 0x7ff0ec1492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0f1e16640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec0ad530 0x7ff0ec1f0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0f1e16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0ec1475b0 0x7ff0ec1ecf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0f1e16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec157910 con 0x7ff0ec1475b0 2023-10-24T23:58:02.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0f1e16640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec153420 con 0x7ff0cc0039b0 2023-10-24T23:58:02.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0f1e16640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ec153700 con 0x7ff0ec0ad530 2023-10-24T23:58:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0ec1475b0 0x7ff0ec1ecf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0ec1475b0 0x7ff0ec1ecf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33744/0 (socket says 172.21.15.107:33744) 2023-10-24T23:58:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0ebfff640 1 -- 172.21.15.107:0/3544955554 learned_addr learned my addr 172.21.15.107:0/3544955554 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:02.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0e3fff640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec0ad530 0x7ff0ec1f0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0eb7fe640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc0039b0 0x7ff0ec1492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0ebfff640 1 -- 172.21.15.107:0/3544955554 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec0ad530 msgr2=0x7ff0ec1f0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0ebfff640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec0ad530 0x7ff0ec1f0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0ebfff640 1 -- 172.21.15.107:0/3544955554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc0039b0 msgr2=0x7ff0ec1492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0ebfff640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc0039b0 0x7ff0ec1492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0ebfff640 1 -- 172.21.15.107:0/3544955554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec1f04c0 con 0x7ff0ec1475b0 2023-10-24T23:58:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0e3fff640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec0ad530 0x7ff0ec1f0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:58:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0ebfff640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0ec1475b0 0x7ff0ec1ecf50 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec158880 tx=0x7ff0e4007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:02.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0e8ff9640 1 -- 172.21.15.107:0/3544955554 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e400eaa0 con 0x7ff0ec1475b0 2023-10-24T23:58:02.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0e8ff9640 1 -- 172.21.15.107:0/3544955554 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e400ec00 con 0x7ff0ec1475b0 2023-10-24T23:58:02.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0f1e16640 1 -- 172.21.15.107:0/3544955554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0ec1f07a0 con 0x7ff0ec1475b0 2023-10-24T23:58:02.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0e8ff9640 1 -- 172.21.15.107:0/3544955554 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4013420 con 0x7ff0ec1475b0 2023-10-24T23:58:02.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0f1e16640 1 -- 172.21.15.107:0/3544955554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ec14e570 con 0x7ff0ec1475b0 2023-10-24T23:58:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0f1e16640 1 -- 172.21.15.107:0/3544955554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ec067a50 con 0x7ff0ec1475b0 2023-10-24T23:58:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0e8ff9640 1 -- 172.21.15.107:0/3544955554 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff0e4013580 con 0x7ff0ec1475b0 2023-10-24T23:58:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0e8ff9640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0d0065e90 0x7ff0d0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0e8ff9640 1 -- 172.21.15.107:0/3544955554 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7ff0e4084130 con 0x7ff0ec1475b0 2023-10-24T23:58:02.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.509+0000 7ff0eb7fe640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0d0065e90 0x7ff0d0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.513+0000 7ff0eb7fe640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0d0065e90 0x7ff0d0068350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0b0000 tx=0x7ff0dc0b7040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:02.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.513+0000 7ff0e8ff9640 1 -- 172.21.15.107:0/3544955554 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff0e4051280 con 0x7ff0ec1475b0 2023-10-24T23:58:02.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:02.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.701+0000 7ff0f1e16640 1 -- 172.21.15.107:0/3544955554 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0ec153870 con 0x7ff0ec1475b0 2023-10-24T23:58:02.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.705+0000 7ff0e8ff9640 1 -- 172.21.15.107:0/3544955554 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7ff0e40548a0 con 0x7ff0ec1475b0 2023-10-24T23:58:02.709 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:02.709 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:02.710 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:58:02.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.705+0000 7ff0f1e16640 1 -- 172.21.15.107:0/3544955554 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0d0065e90 msgr2=0x7ff0d0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.705+0000 7ff0f1e16640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0d0065e90 0x7ff0d0068350 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0b0000 tx=0x7ff0dc0b7040 comp rx=0 tx=0).stop 2023-10-24T23:58:02.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.705+0000 7ff0f1e16640 1 -- 172.21.15.107:0/3544955554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0ec1475b0 msgr2=0x7ff0ec1ecf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.705+0000 7ff0f1e16640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0ec1475b0 0x7ff0ec1ecf50 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec158880 tx=0x7ff0e4007bc0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.709+0000 7ff0f1e16640 1 -- 172.21.15.107:0/3544955554 shutdown_connections 2023-10-24T23:58:02.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.709+0000 7ff0f1e16640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0ec1475b0 0x7ff0ec1ecf50 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.709+0000 7ff0f1e16640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0d0065e90 0x7ff0d0068350 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.709+0000 7ff0f1e16640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0ec0ad530 0x7ff0ec1f0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.709+0000 7ff0f1e16640 1 --2- 172.21.15.107:0/3544955554 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc0039b0 0x7ff0ec1492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.709+0000 7ff0f1e16640 1 -- 172.21.15.107:0/3544955554 >> 172.21.15.107:0/3544955554 conn(0x7ff0ec0c81d0 msgr2=0x7ff0ec153d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:02.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.709+0000 7ff0f1e16640 1 -- 172.21.15.107:0/3544955554 shutdown_connections 2023-10-24T23:58:02.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.709+0000 7ff0f1e16640 1 -- 172.21.15.107:0/3544955554 wait complete. 2023-10-24T23:58:02.732 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-24T23:58:02.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.729+0000 7ff0d2162640 1 Processor -- start 2023-10-24T23:58:02.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.729+0000 7ff0d2162640 1 -- start start 2023-10-24T23:58:02.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.729+0000 7ff0d2162640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff0cc152040 0x7ff0cc0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.729+0000 7ff0d2162640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 0x7ff0cc0ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.729+0000 7ff0d2162640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0cc155680 con 0x7ff0cc0c8100 2023-10-24T23:58:02.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.729+0000 7ff0d2162640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0cc154a70 con 0x7ff0cc152040 2023-10-24T23:58:02.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.729+0000 7ff0d2162640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc154d50 con 0x7ff0cc06a4b0 2023-10-24T23:58:02.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.729+0000 7ff0cbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 0x7ff0cc0ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.729+0000 7ff0cb7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff0cc152040 0x7ff0cc0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.729+0000 7ff0cbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 0x7ff0cc0ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:52212/0 (socket says 172.21.15.107:52212) 2023-10-24T23:58:02.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.729+0000 7ff0cbfff640 1 -- 172.21.15.107:0/831058437 learned_addr learned my addr 172.21.15.107:0/831058437 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:02.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0cbfff640 1 -- 172.21.15.107:0/831058437 >> v2:172.21.15.110:3300/0 conn(0x7ff0cc152040 msgr2=0x7ff0cc0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:02.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0cbfff640 1 --2- 172.21.15.107:0/831058437 >> v2:172.21.15.110:3300/0 conn(0x7ff0cc152040 0x7ff0cc0c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0cbfff640 1 -- 172.21.15.107:0/831058437 >> v1:172.21.15.110:6789/0 conn(0x7ff0cc0c8100 legacy=0x7ff0cc0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0cbfff640 1 -- 172.21.15.107:0/831058437 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0cc155030 con 0x7ff0cc06a4b0 2023-10-24T23:58:02.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0cbfff640 1 --2- 172.21.15.107:0/831058437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 0x7ff0cc0ac410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009e50 tx=0x7ff0c401c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9baeb21965a4f31b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:02.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0ca7fc640 1 -- 172.21.15.107:0/831058437 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c40a5750 con 0x7ff0cc06a4b0 2023-10-24T23:58:02.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0ca7fc640 1 -- 172.21.15.107:0/831058437 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c40a58b0 con 0x7ff0cc06a4b0 2023-10-24T23:58:02.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0ca7fc640 1 -- 172.21.15.107:0/831058437 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c40a5c40 con 0x7ff0cc06a4b0 2023-10-24T23:58:02.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0d2162640 1 -- 172.21.15.107:0/831058437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 msgr2=0x7ff0cc0ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0d2162640 1 --2- 172.21.15.107:0/831058437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 0x7ff0cc0ac410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009e50 tx=0x7ff0c401c980 comp rx=0 tx=0).stop 2023-10-24T23:58:02.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0d2162640 1 -- 172.21.15.107:0/831058437 shutdown_connections 2023-10-24T23:58:02.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0d2162640 1 --2- 172.21.15.107:0/831058437 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 0x7ff0cc0ac410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0d2162640 1 --2- 172.21.15.107:0/831058437 >> v2:172.21.15.110:3300/0 conn(0x7ff0cc152040 0x7ff0cc0c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0d2162640 1 -- 172.21.15.107:0/831058437 >> 172.21.15.107:0/831058437 conn(0x7ff0cc0ca200 msgr2=0x7ff0cc0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:02.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0d2162640 1 -- 172.21.15.107:0/831058437 shutdown_connections 2023-10-24T23:58:02.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.733+0000 7ff0d2162640 1 -- 172.21.15.107:0/831058437 wait complete. 2023-10-24T23:58:02.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0d2162640 1 Processor -- start 2023-10-24T23:58:02.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0d2162640 1 -- start start 2023-10-24T23:58:02.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0d2162640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 0x7ff0cc143ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0d2162640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0c8100 0x7ff0cc147a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0d2162640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc152040 0x7ff0cc1eca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0d2162640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc153b10 con 0x7ff0cc0c8100 2023-10-24T23:58:02.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0d2162640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc152ca0 con 0x7ff0cc152040 2023-10-24T23:58:02.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0d2162640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0cc143760 con 0x7ff0cc06a4b0 2023-10-24T23:58:02.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0c8100 0x7ff0cc147a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0caffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0c8100 0x7ff0cc147a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57486/0 (socket says 172.21.15.107:57486) 2023-10-24T23:58:02.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0caffd640 1 -- 172.21.15.107:0/1065651196 learned_addr learned my addr 172.21.15.107:0/1065651196 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:02.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0cb7fe640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 0x7ff0cc143ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0cbfff640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc152040 0x7ff0cc1eca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0caffd640 1 -- 172.21.15.107:0/1065651196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 msgr2=0x7ff0cc143ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0caffd640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 0x7ff0cc143ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0caffd640 1 -- 172.21.15.107:0/1065651196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc152040 msgr2=0x7ff0cc1eca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0caffd640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc152040 0x7ff0cc1eca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0caffd640 1 -- 172.21.15.107:0/1065651196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4009b30 con 0x7ff0cc0c8100 2023-10-24T23:58:02.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0caffd640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0c8100 0x7ff0cc147a20 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c430 tx=0x7ff0bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:02.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0b3fff640 1 -- 172.21.15.107:0/1065651196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc0052d0 con 0x7ff0cc0c8100 2023-10-24T23:58:02.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0b3fff640 1 -- 172.21.15.107:0/1065651196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0bc005430 con 0x7ff0cc0c8100 2023-10-24T23:58:02.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0d2162640 1 -- 172.21.15.107:0/1065651196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0cc152950 con 0x7ff0cc0c8100 2023-10-24T23:58:02.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0b3fff640 1 -- 172.21.15.107:0/1065651196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc007260 con 0x7ff0cc0c8100 2023-10-24T23:58:02.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0d2162640 1 -- 172.21.15.107:0/1065651196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0cc147f60 con 0x7ff0cc0c8100 2023-10-24T23:58:02.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.737+0000 7ff0d2162640 1 -- 172.21.15.107:0/1065651196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0cc152e10 con 0x7ff0cc0c8100 2023-10-24T23:58:02.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.741+0000 7ff0b3fff640 1 -- 172.21.15.107:0/1065651196 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff0bc0073c0 con 0x7ff0cc0c8100 2023-10-24T23:58:02.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.741+0000 7ff0b3fff640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0b8065e60 0x7ff0b8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:02.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.741+0000 7ff0b3fff640 1 -- 172.21.15.107:0/1065651196 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 7477+0+0 (secure 0 0 0) 0x7ff0bc0830c0 con 0x7ff0cc0c8100 2023-10-24T23:58:02.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.741+0000 7ff0cb7fe640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0b8065e60 0x7ff0b8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:02.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.741+0000 7ff0cb7fe640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0b8065e60 0x7ff0b8068320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009b60 tx=0x7ff0c0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:02.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.745+0000 7ff0b3fff640 1 -- 172.21.15.107:0/1065651196 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff0bc0502c0 con 0x7ff0cc0c8100 2023-10-24T23:58:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.929+0000 7ff0d2162640 1 -- 172.21.15.107:0/1065651196 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0cc0746e0 con 0x7ff0cc0c8100 2023-10-24T23:58:02.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.929+0000 7ff0b3fff640 1 -- 172.21.15.107:0/1065651196 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+5774 (secure 0 0 0) 0x7ff0bc0538e0 con 0x7ff0cc0c8100 2023-10-24T23:58:02.937 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:02.937 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4937,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":42},{"gid":5751,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:57:32.622497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4232":{"gid":4232,"name":"b","rank":0,"incarnation":38,"state":"up:active","state_seq":71,"addr":"172.21.15.110:6834/3721101226","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3721101226},{"type":"v1","addr":"172.21.15.110:6834","nonce":3721101226}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:57:26.574492+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5562},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5562":{"gid":5562,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":17,"addr":"172.21.15.107:6837/204494054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":204494054},{"type":"v1","addr":"172.21.15.107:6837","nonce":204494054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 43 2023-10-24T23:58:02.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 -- 172.21.15.107:0/1065651196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0b8065e60 msgr2=0x7ff0b8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0b8065e60 0x7ff0b8068320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0009b60 tx=0x7ff0c0009190 comp rx=0 tx=0).stop 2023-10-24T23:58:02.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 -- 172.21.15.107:0/1065651196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0c8100 msgr2=0x7ff0cc147a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:02.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0c8100 0x7ff0cc147a20 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc00c430 tx=0x7ff0bc00c8f0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 -- 172.21.15.107:0/1065651196 shutdown_connections 2023-10-24T23:58:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0cc152040 0x7ff0cc1eca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0cc0c8100 0x7ff0cc147a20 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff0b8065e60 0x7ff0b8068320 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 --2- 172.21.15.107:0/1065651196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0cc06a4b0 0x7ff0cc143ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:02.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 -- 172.21.15.107:0/1065651196 >> 172.21.15.107:0/1065651196 conn(0x7ff0cc0ca200 msgr2=0x7ff0cc0ad680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:02.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 -- 172.21.15.107:0/1065651196 shutdown_connections 2023-10-24T23:58:02.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:02.933+0000 7ff0d2162640 1 -- 172.21.15.107:0/1065651196 wait complete. 2023-10-24T23:58:02.955 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:58:04.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7df671f640 1 Processor -- start 2023-10-24T23:58:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7df671f640 1 -- start start 2023-10-24T23:58:04.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7df671f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7df013bb00 0x7f7df01531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:04.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7df671f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7df00a9450 0x7f7df0071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:04.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7df671f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df0152d40 con 0x7f7df00aa330 2023-10-24T23:58:04.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7df671f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df0156100 con 0x7f7df013bb00 2023-10-24T23:58:04.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7df671f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df01527c0 con 0x7f7df00a9450 2023-10-24T23:58:04.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7deffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7df013bb00 0x7f7df01531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:04.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7deffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7df013bb00 0x7f7df01531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57498/0 (socket says 172.21.15.107:57498) 2023-10-24T23:58:04.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7deffff640 1 -- 172.21.15.107:0/3550717296 learned_addr learned my addr 172.21.15.107:0/3550717296 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:04.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7df4cbc640 1 --2- 172.21.15.107:0/3550717296 >> v2:172.21.15.110:3300/0 conn(0x7f7df00a9450 0x7f7df0071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:04.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7deffff640 1 -- 172.21.15.107:0/3550717296 >> v2:172.21.15.110:3300/0 conn(0x7f7df00a9450 msgr2=0x7f7df0071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:04.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7deffff640 1 --2- 172.21.15.107:0/3550717296 >> v2:172.21.15.110:3300/0 conn(0x7f7df00a9450 0x7f7df0071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:04.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7deffff640 1 -- 172.21.15.107:0/3550717296 >> v1:172.21.15.107:6789/0 conn(0x7f7df00aa330 legacy=0x7f7df006e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:04.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.857+0000 7f7deffff640 1 -- 172.21.15.107:0/3550717296 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df006d660 con 0x7f7df013bb00 2023-10-24T23:58:04.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deffff640 1 --2- 172.21.15.107:0/3550717296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7df013bb00 0x7f7df01531d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7df0072ab0 tx=0x7f7de401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d3536afcc9ad3f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:04.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deeffd640 1 -- 172.21.15.107:0/3550717296 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de40ab600 con 0x7f7df013bb00 2023-10-24T23:58:04.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deeffd640 1 -- 172.21.15.107:0/3550717296 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de40a7050 con 0x7f7df013bb00 2023-10-24T23:58:04.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deeffd640 1 -- 172.21.15.107:0/3550717296 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de40ab980 con 0x7f7df013bb00 2023-10-24T23:58:04.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 -- 172.21.15.107:0/3550717296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7df013bb00 msgr2=0x7f7df01531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:04.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 --2- 172.21.15.107:0/3550717296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7df013bb00 0x7f7df01531d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7df0072ab0 tx=0x7f7de401c3e0 comp rx=0 tx=0).stop 2023-10-24T23:58:04.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 -- 172.21.15.107:0/3550717296 shutdown_connections 2023-10-24T23:58:04.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 --2- 172.21.15.107:0/3550717296 >> v2:172.21.15.110:3300/0 conn(0x7f7df00a9450 0x7f7df0071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 --2- 172.21.15.107:0/3550717296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7df013bb00 0x7f7df01531d0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 -- 172.21.15.107:0/3550717296 >> 172.21.15.107:0/3550717296 conn(0x7f7df0070f60 msgr2=0x7f7df0071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 -- 172.21.15.107:0/3550717296 shutdown_connections 2023-10-24T23:58:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 -- 172.21.15.107:0/3550717296 wait complete. 2023-10-24T23:58:04.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 Processor -- start 2023-10-24T23:58:04.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 -- start start 2023-10-24T23:58:04.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7df00a9450 0x7f7df01eb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:04.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7df00aa330 0x7f7df014fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:04.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7df00ad250 0x7f7df01ee130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:04.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df00aacc0 con 0x7f7df00a9450 2023-10-24T23:58:04.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df00aaf10 con 0x7f7df00aa330 2023-10-24T23:58:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df01eacd0 con 0x7f7df00ad250 2023-10-24T23:58:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7df00a9450 0x7f7df01eb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7df00a9450 0x7f7df01eb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57506/0 (socket says 172.21.15.107:57506) 2023-10-24T23:58:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deffff640 1 -- 172.21.15.107:0/3103405672 learned_addr learned my addr 172.21.15.107:0/3103405672 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:04.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deffff640 1 -- 172.21.15.107:0/3103405672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7df00ad250 msgr2=0x7f7df01ee130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deffff640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7df00ad250 0x7f7df01ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deffff640 1 -- 172.21.15.107:0/3103405672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7df00aa330 msgr2=0x7f7df014fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deffff640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7df00aa330 0x7f7df014fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deffff640 1 -- 172.21.15.107:0/3103405672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de40094d0 con 0x7f7df00a9450 2023-10-24T23:58:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7deffff640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7df00a9450 0x7f7df01eb7b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7de401a010 tx=0x7f7de401ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7decff9640 1 -- 172.21.15.107:0/3103405672 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de40b6620 con 0x7f7df00a9450 2023-10-24T23:58:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7decff9640 1 -- 172.21.15.107:0/3103405672 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de40b67a0 con 0x7f7df00a9450 2023-10-24T23:58:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 -- 172.21.15.107:0/3103405672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df01eafb0 con 0x7f7df00a9450 2023-10-24T23:58:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7decff9640 1 -- 172.21.15.107:0/3103405672 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de40b6b50 con 0x7f7df00a9450 2023-10-24T23:58:04.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.861+0000 7f7df671f640 1 -- 172.21.15.107:0/3103405672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df0150490 con 0x7f7df00a9450 2023-10-24T23:58:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.865+0000 7f7df671f640 1 -- 172.21.15.107:0/3103405672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df00746e0 con 0x7f7df00a9450 2023-10-24T23:58:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.865+0000 7f7decff9640 1 -- 172.21.15.107:0/3103405672 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7de40b2040 con 0x7f7df00a9450 2023-10-24T23:58:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.865+0000 7f7decff9640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ddc065e60 0x7f7ddc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.865+0000 7f7decff9640 1 -- 172.21.15.107:0/3103405672 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f7de411e280 con 0x7f7df00a9450 2023-10-24T23:58:04.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.865+0000 7f7def7fe640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ddc065e60 0x7f7ddc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.865+0000 7f7def7fe640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ddc065e60 0x7f7ddc068320 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7de0000f70 tx=0x7f7de00079c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:04.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:04.869+0000 7f7decff9640 1 -- 172.21.15.107:0/3103405672 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7de40eb320 con 0x7f7df00a9450 2023-10-24T23:58:04.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:05.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 -- 172.21.15.107:0/3103405672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7df01eb460 con 0x7f7df00a9450 2023-10-24T23:58:05.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7decff9640 1 -- 172.21.15.107:0/3103405672 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4262 (secure 0 0 0) 0x7f7de40ee940 con 0x7f7df00a9450 2023-10-24T23:58:05.062 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:05.062 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:04.498660+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:04.498660+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:05.063 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 45 2023-10-24T23:58:05.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 -- 172.21.15.107:0/3103405672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ddc065e60 msgr2=0x7f7ddc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ddc065e60 0x7f7ddc068320 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7de0000f70 tx=0x7f7de00079c0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 -- 172.21.15.107:0/3103405672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7df00a9450 msgr2=0x7f7df01eb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7df00a9450 0x7f7df01eb7b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f7de401a010 tx=0x7f7de401ac20 comp rx=0 tx=0).stop 2023-10-24T23:58:05.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 -- 172.21.15.107:0/3103405672 shutdown_connections 2023-10-24T23:58:05.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ddc065e60 0x7f7ddc068320 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7df00ad250 0x7f7df01ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7df00aa330 0x7f7df014fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 --2- 172.21.15.107:0/3103405672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7df00a9450 0x7f7df01eb7b0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 -- 172.21.15.107:0/3103405672 >> 172.21.15.107:0/3103405672 conn(0x7f7df0070f60 msgr2=0x7f7df015c290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:05.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 -- 172.21.15.107:0/3103405672 shutdown_connections 2023-10-24T23:58:05.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.057+0000 7f7df671f640 1 -- 172.21.15.107:0/3103405672 wait complete. 2023-10-24T23:58:05.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.065+0000 7fe77d925640 1 Processor -- start 2023-10-24T23:58:05.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.065+0000 7fe77d925640 1 -- start start 2023-10-24T23:58:05.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe77d925640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe778070b00 0x7fe7780c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe77d925640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7780c22f0 con 0x7fe77806df30 2023-10-24T23:58:05.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe77d925640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7780c25d0 con 0x7fe77814eb70 2023-10-24T23:58:05.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe77d925640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe778071700 con 0x7fe778070b00 2023-10-24T23:58:05.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe7767fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe778070b00 0x7fe7780c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe7767fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe778070b00 0x7fe7780c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57508/0 (socket says 172.21.15.107:57508) 2023-10-24T23:58:05.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe7767fc640 1 -- 172.21.15.107:0/1207610711 learned_addr learned my addr 172.21.15.107:0/1207610711 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:05.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe7767fc640 1 -- 172.21.15.107:0/1207610711 >> v1:172.21.15.110:6789/0 conn(0x7fe77814eb70 legacy=0x7fe7781514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:05.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe7767fc640 1 -- 172.21.15.107:0/1207610711 >> v1:172.21.15.107:6789/0 conn(0x7fe77806df30 legacy=0x7fe77815be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe7767fc640 1 -- 172.21.15.107:0/1207610711 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7780c2be0 con 0x7fe778070b00 2023-10-24T23:58:05.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe7767fc640 1 --2- 172.21.15.107:0/1207610711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe778070b00 0x7fe7780c33e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe768009870 tx=0x7fe76801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fb3730b3dd11d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:05.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe775ffb640 1 -- 172.21.15.107:0/1207610711 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7680ab610 con 0x7fe778070b00 2023-10-24T23:58:05.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe775ffb640 1 -- 172.21.15.107:0/1207610711 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7680ab770 con 0x7fe778070b00 2023-10-24T23:58:05.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe775ffb640 1 -- 172.21.15.107:0/1207610711 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7680abb00 con 0x7fe778070b00 2023-10-24T23:58:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe77d925640 1 -- 172.21.15.107:0/1207610711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe778070b00 msgr2=0x7fe7780c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe77d925640 1 --2- 172.21.15.107:0/1207610711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe778070b00 0x7fe7780c33e0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe768009870 tx=0x7fe76801c500 comp rx=0 tx=0).stop 2023-10-24T23:58:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe77d925640 1 -- 172.21.15.107:0/1207610711 shutdown_connections 2023-10-24T23:58:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe77d925640 1 --2- 172.21.15.107:0/1207610711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe778070b00 0x7fe7780c33e0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.069+0000 7fe77d925640 1 -- 172.21.15.107:0/1207610711 >> 172.21.15.107:0/1207610711 conn(0x7fe7780c39a0 msgr2=0x7fe7780c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 -- 172.21.15.107:0/1207610711 shutdown_connections 2023-10-24T23:58:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 -- 172.21.15.107:0/1207610711 wait complete. 2023-10-24T23:58:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 Processor -- start 2023-10-24T23:58:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 -- start start 2023-10-24T23:58:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe77806df30 0x7fe778144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe778070b00 0x7fe7780a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe77814eb70 0x7fe7780aa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe778144b30 con 0x7fe778070b00 2023-10-24T23:58:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7780aa3a0 con 0x7fe77806df30 2023-10-24T23:58:05.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7781ef110 con 0x7fe77814eb70 2023-10-24T23:58:05.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe7767fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe778070b00 0x7fe7780a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe7767fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe778070b00 0x7fe7780a9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57524/0 (socket says 172.21.15.107:57524) 2023-10-24T23:58:05.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe7767fc640 1 -- 172.21.15.107:0/1847088306 learned_addr learned my addr 172.21.15.107:0/1847088306 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:05.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe776ffd640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe77806df30 0x7fe778144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe7777fe640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe77814eb70 0x7fe7780aa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe7767fc640 1 -- 172.21.15.107:0/1847088306 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe77814eb70 msgr2=0x7fe7780aa7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe7767fc640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe77814eb70 0x7fe7780aa7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe7767fc640 1 -- 172.21.15.107:0/1847088306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe77806df30 msgr2=0x7fe778144190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe7767fc640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe77806df30 0x7fe778144190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe7767fc640 1 -- 172.21.15.107:0/1847088306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680094d0 con 0x7fe778070b00 2023-10-24T23:58:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe7767fc640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe778070b00 0x7fe7780a9e60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe7680ad040 tx=0x7fe76801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:05.085 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2023-10-24T23:58:05.085 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-24T23:58:05.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe75ffff640 1 -- 172.21.15.107:0/1847088306 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7680b4800 con 0x7fe778070b00 2023-10-24T23:58:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe75ffff640 1 -- 172.21.15.107:0/1847088306 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7680b4980 con 0x7fe778070b00 2023-10-24T23:58:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 -- 172.21.15.107:0/1847088306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe778144850 con 0x7fe778070b00 2023-10-24T23:58:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe75ffff640 1 -- 172.21.15.107:0/1847088306 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7680b4d30 con 0x7fe778070b00 2023-10-24T23:58:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 -- 172.21.15.107:0/1847088306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7781eeda0 con 0x7fe778070b00 2023-10-24T23:58:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.073+0000 7fe77d925640 1 -- 172.21.15.107:0/1847088306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7780746e0 con 0x7fe778070b00 2023-10-24T23:58:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.077+0000 7fe75ffff640 1 -- 172.21.15.107:0/1847088306 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe7680be3f0 con 0x7fe778070b00 2023-10-24T23:58:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.077+0000 7fe75ffff640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe764065e60 0x7fe764068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.077+0000 7fe75ffff640 1 -- 172.21.15.107:0/1847088306 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7fe76811dec0 con 0x7fe778070b00 2023-10-24T23:58:05.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.077+0000 7fe776ffd640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe764065e60 0x7fe764068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.077+0000 7fe776ffd640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe764065e60 0x7fe764068320 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe778145620 tx=0x7fe76c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:05.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.081+0000 7fe75ffff640 1 -- 172.21.15.107:0/1847088306 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe7680eb080 con 0x7fe778070b00 2023-10-24T23:58:05.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:58:05.160 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:58:05.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd03349640 1 Processor -- start 2023-10-24T23:58:05.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd03349640 1 -- start start 2023-10-24T23:58:05.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd03349640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fccfc069d40 0x7fccfc158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd03349640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccfc06cea0 con 0x7fccfc065bc0 2023-10-24T23:58:05.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd03349640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccfc1584e0 con 0x7fccfc06a120 2023-10-24T23:58:05.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd03349640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccfc0ac650 con 0x7fccfc069d40 2023-10-24T23:58:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd008e4640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fccfc065bc0 0x7fccfc0a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40778/0 (socket says 172.21.15.107:40778) 2023-10-24T23:58:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd008e4640 1 -- 172.21.15.107:0/989292605 learned_addr learned my addr 172.21.15.107:0/989292605 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd010e5640 1 --2- 172.21.15.107:0/989292605 >> v2:172.21.15.110:3300/0 conn(0x7fccfc069d40 0x7fccfc158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584589653 0 0) 0x7fccfc06cea0 con 0x7fccfc065bc0 2023-10-24T23:58:05.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce8003530 con 0x7fccfc065bc0 2023-10-24T23:58:05.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750799352 0 0) 0x7fcce8003530 con 0x7fccfc065bc0 2023-10-24T23:58:05.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 >> v2:172.21.15.110:3300/0 conn(0x7fccfc069d40 msgr2=0x7fccfc158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:05.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 --2- 172.21.15.107:0/989292605 >> v2:172.21.15.110:3300/0 conn(0x7fccfc069d40 0x7fccfc158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 >> v1:172.21.15.110:6789/0 conn(0x7fccfc06a120 legacy=0x7fccfc0aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc06d3d0 con 0x7fccfc065bc0 2023-10-24T23:58:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fccec001680 con 0x7fccfc065bc0 2023-10-24T23:58:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 >> v1:172.21.15.107:6789/0 conn(0x7fccfc065bc0 legacy=0x7fccfc0a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce8003980 0x7fcce8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce80047d0 0x7fcce8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcce8005190 0x7fcce800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce800c100 con 0x7fcce8003980 2023-10-24T23:58:05.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce800c270 con 0x7fcce8005190 2023-10-24T23:58:05.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce800c430 con 0x7fcce80047d0 2023-10-24T23:58:05.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd010e5640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce8003980 0x7fcce8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd018e6640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcce8005190 0x7fcce800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd008e4640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce80047d0 0x7fcce8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd010e5640 1 -- 172.21.15.107:0/989292605 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce80047d0 msgr2=0x7fcce8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:05.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd010e5640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce80047d0 0x7fcce8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd010e5640 1 -- 172.21.15.107:0/989292605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcce8005190 msgr2=0x7fcce800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:05.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd010e5640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcce8005190 0x7fcce800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd018e6640 1 -- 172.21.15.107:0/989292605 reap_dead start 2023-10-24T23:58:05.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd010e5640 1 -- 172.21.15.107:0/989292605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce8003530 con 0x7fcce8003980 2023-10-24T23:58:05.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.189+0000 7fcd008e4640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce80047d0 0x7fcce8004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:05.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.193+0000 7fcd010e5640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce8003980 0x7fcce8003da0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009ae0 tx=0x7fccf801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199f312fde3023cf server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:05.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.193+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf80a6430 con 0x7fcce8003980 2023-10-24T23:58:05.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.193+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf80a6590 con 0x7fcce8003980 2023-10-24T23:58:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.193+0000 7fccf3fff640 1 -- 172.21.15.107:0/989292605 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf80a6940 con 0x7fcce8003980 2023-10-24T23:58:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.193+0000 7fcd03349640 1 -- 172.21.15.107:0/989292605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce8003980 msgr2=0x7fcce8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.193+0000 7fcd03349640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce8003980 0x7fcce8003da0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009ae0 tx=0x7fccf801ca50 comp rx=0 tx=0).stop 2023-10-24T23:58:05.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.193+0000 7fcd03349640 1 -- 172.21.15.107:0/989292605 shutdown_connections 2023-10-24T23:58:05.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.193+0000 7fcd03349640 1 --2- 172.21.15.107:0/989292605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcce8003980 0x7fcce8003da0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.193+0000 7fcd03349640 1 -- 172.21.15.107:0/989292605 >> 172.21.15.107:0/989292605 conn(0x7fccfc0c65c0 msgr2=0x7fccfc0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:05.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.193+0000 7fcd03349640 1 -- 172.21.15.107:0/989292605 shutdown_connections 2023-10-24T23:58:05.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.193+0000 7fcd03349640 1 -- 172.21.15.107:0/989292605 wait complete. 2023-10-24T23:58:05.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd03349640 1 Processor -- start 2023-10-24T23:58:05.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd03349640 1 -- start start 2023-10-24T23:58:05.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd03349640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce8003980 0x7fccfc14c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd03349640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc070320 0x7fccfc1f0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd03349640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc0707a0 0x7fccfc1ed4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd03349640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc14cf30 con 0x7fccfc070320 2023-10-24T23:58:05.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd03349640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc14bf90 con 0x7fccfc0707a0 2023-10-24T23:58:05.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd03349640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc14c960 con 0x7fcce8003980 2023-10-24T23:58:05.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd008e4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc070320 0x7fccfc1f0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd008e4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc070320 0x7fccfc1f0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57546/0 (socket says 172.21.15.107:57546) 2023-10-24T23:58:05.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd008e4640 1 -- 172.21.15.107:0/75114900 learned_addr learned my addr 172.21.15.107:0/75114900 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:05.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd018e6640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc0707a0 0x7fccfc1ed4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd010e5640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce8003980 0x7fccfc14c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd008e4640 1 -- 172.21.15.107:0/75114900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce8003980 msgr2=0x7fccfc14c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd008e4640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce8003980 0x7fccfc14c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd008e4640 1 -- 172.21.15.107:0/75114900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc0707a0 msgr2=0x7fccfc1ed4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd008e4640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc0707a0 0x7fccfc1ed4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd008e4640 1 -- 172.21.15.107:0/75114900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc14b9d0 con 0x7fccfc070320 2023-10-24T23:58:05.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd010e5640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce8003980 0x7fccfc14c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:05.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd018e6640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc0707a0 0x7fccfc1ed4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:58:05.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd008e4640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc070320 0x7fccfc1f0ca0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fccec00d7b0 tx=0x7fccec00dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:05.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fccf1ffb640 1 -- 172.21.15.107:0/75114900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec0074e0 con 0x7fccfc070320 2023-10-24T23:58:05.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fccf1ffb640 1 -- 172.21.15.107:0/75114900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccec007640 con 0x7fccfc070320 2023-10-24T23:58:05.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd03349640 1 -- 172.21.15.107:0/75114900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccfc1f15f0 con 0x7fccfc070320 2023-10-24T23:58:05.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fccf1ffb640 1 -- 172.21.15.107:0/75114900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccec004a10 con 0x7fccfc070320 2023-10-24T23:58:05.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd03349640 1 -- 172.21.15.107:0/75114900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccfc0bfab0 con 0x7fccfc070320 2023-10-24T23:58:05.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd03349640 1 -- 172.21.15.107:0/75114900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccfc0746e0 con 0x7fccfc070320 2023-10-24T23:58:05.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fccf1ffb640 1 -- 172.21.15.107:0/75114900 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fccec004b70 con 0x7fccfc070320 2023-10-24T23:58:05.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fccf1ffb640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcce0065e60 0x7fcce0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fccf1ffb640 1 -- 172.21.15.107:0/75114900 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7fccec0839c0 con 0x7fccfc070320 2023-10-24T23:58:05.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.197+0000 7fcd010e5640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcce0065e60 0x7fcce0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.201+0000 7fcd010e5640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcce0065e60 0x7fcce0068320 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fccf80ad000 tx=0x7fccf80046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:05.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.201+0000 7fccf1ffb640 1 -- 172.21.15.107:0/75114900 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fccec050a60 con 0x7fccfc070320 2023-10-24T23:58:05.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.261+0000 7fe77d925640 1 -- 172.21.15.107:0/1847088306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7781ef250 con 0x7fe778070b00 2023-10-24T23:58:05.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.261+0000 7fe75ffff640 1 -- 172.21.15.107:0/1847088306 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4262 (secure 0 0 0) 0x7fe7680ee6a0 con 0x7fe778070b00 2023-10-24T23:58:05.268 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:05.268 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:04.498660+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:04.498660+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:05.269 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 45 2023-10-24T23:58:05.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 -- 172.21.15.107:0/1847088306 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe764065e60 msgr2=0x7fe764068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe764065e60 0x7fe764068320 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe778145620 tx=0x7fe76c009040 comp rx=0 tx=0).stop 2023-10-24T23:58:05.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 -- 172.21.15.107:0/1847088306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe778070b00 msgr2=0x7fe7780a9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe778070b00 0x7fe7780a9e60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe7680ad040 tx=0x7fe76801ad40 comp rx=0 tx=0).stop 2023-10-24T23:58:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 -- 172.21.15.107:0/1847088306 shutdown_connections 2023-10-24T23:58:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe764065e60 0x7fe764068320 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe77814eb70 0x7fe7780aa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe778070b00 0x7fe7780a9e60 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 --2- 172.21.15.107:0/1847088306 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe77806df30 0x7fe778144190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 -- 172.21.15.107:0/1847088306 >> 172.21.15.107:0/1847088306 conn(0x7fe7780c39a0 msgr2=0x7fe7780c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 -- 172.21.15.107:0/1847088306 shutdown_connections 2023-10-24T23:58:05.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.265+0000 7fe77d925640 1 -- 172.21.15.107:0/1847088306 wait complete. 2023-10-24T23:58:05.287 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2023-10-24T23:58:05.287 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-24T23:58:05.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:05.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.381+0000 7fcd03349640 1 -- 172.21.15.107:0/75114900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccfc0c0730 con 0x7fccfc070320 2023-10-24T23:58:05.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.381+0000 7fccf1ffb640 1 -- 172.21.15.107:0/75114900 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4262 (secure 0 0 0) 0x7fccec019090 con 0x7fccfc070320 2023-10-24T23:58:05.389 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:05.389 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:04.498660+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:04.498660+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:05.389 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 45 2023-10-24T23:58:05.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 -- 172.21.15.107:0/75114900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcce0065e60 msgr2=0x7fcce0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcce0065e60 0x7fcce0068320 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fccf80ad000 tx=0x7fccf80046d0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 -- 172.21.15.107:0/75114900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc070320 msgr2=0x7fccfc1f0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc070320 0x7fccfc1f0ca0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fccec00d7b0 tx=0x7fccec00dc70 comp rx=0 tx=0).stop 2023-10-24T23:58:05.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 -- 172.21.15.107:0/75114900 shutdown_connections 2023-10-24T23:58:05.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccfc0707a0 0x7fccfc1ed4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccfc070320 0x7fccfc1f0ca0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcce0065e60 0x7fcce0068320 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 --2- 172.21.15.107:0/75114900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcce8003980 0x7fccfc14c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 -- 172.21.15.107:0/75114900 >> 172.21.15.107:0/75114900 conn(0x7fccfc0c65c0 msgr2=0x7fccfc0bdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:05.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 -- 172.21.15.107:0/75114900 shutdown_connections 2023-10-24T23:58:05.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.385+0000 7fcd03349640 1 -- 172.21.15.107:0/75114900 wait complete. 2023-10-24T23:58:05.414 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 45, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 45, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5751": { "addr": "172.21.15.107:6835/1508236506", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 1508236506, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 1508236506, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5751, "incarnation": 44, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:58:04.498660+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5751 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 45, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4937": { "addr": "172.21.15.110:6835/2348043270", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2348043270, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2348043270, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4937, "incarnation": 44, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-24T23:58:04.498660+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4937 } } } ], "standbys": [] } 2023-10-24T23:58:05.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30ca6cf640 1 Processor -- start 2023-10-24T23:58:05.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30ca6cf640 1 -- start start 2023-10-24T23:58:05.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30ca6cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c413dc20 0x7f30c40c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30ca6cf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c4154860 con 0x7f30c40720c0 2023-10-24T23:58:05.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30ca6cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c40c1f10 con 0x7f30c4157e50 2023-10-24T23:58:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30ca6cf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4154b20 con 0x7f30c413dc20 2023-10-24T23:58:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c37fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f30c40720c0 0x7f30c406df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40786/0 (socket says 172.21.15.107:40786) 2023-10-24T23:58:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c37fe640 1 -- 172.21.15.107:0/3918082472 learned_addr learned my addr 172.21.15.107:0/3918082472 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:05.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c3fff640 1 --2- 172.21.15.107:0/3918082472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c413dc20 0x7f30c40c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326874066 0 0) 0x7f30c4154860 con 0x7f30c40720c0 2023-10-24T23:58:05.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30ac003530 con 0x7f30c40720c0 2023-10-24T23:58:05.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797898770 0 0) 0x7f30ac003530 con 0x7f30c40720c0 2023-10-24T23:58:05.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c413dc20 msgr2=0x7f30c40c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 --2- 172.21.15.107:0/3918082472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c413dc20 0x7f30c40c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 >> v1:172.21.15.110:6789/0 conn(0x7f30c4157e50 legacy=0x7f30c415f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c4154e00 con 0x7f30c40720c0 2023-10-24T23:58:05.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f30b8001680 con 0x7f30c40720c0 2023-10-24T23:58:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 >> v1:172.21.15.107:6789/0 conn(0x7f30c40720c0 legacy=0x7f30c406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 --2- 172.21.15.107:0/3918082472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30ac0039b0 0x7f30ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 --2- 172.21.15.107:0/3918082472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c413dc20 0x7f30ac0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 --2- 172.21.15.107:0/3918082472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30ac004d60 0x7f30ac0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4154b20 con 0x7f30ac0039b0 2023-10-24T23:58:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac00bc10 con 0x7f30ac004d60 2023-10-24T23:58:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac00bdd0 con 0x7f30c413dc20 2023-10-24T23:58:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c3fff640 1 --2- 172.21.15.107:0/3918082472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30ac0039b0 0x7f30ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c3fff640 1 -- 172.21.15.107:0/3918082472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c413dc20 msgr2=0x7f30ac0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c8c6c640 1 --2- 172.21.15.107:0/3918082472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30ac004d60 0x7f30ac0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c3fff640 1 --2- 172.21.15.107:0/3918082472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c413dc20 0x7f30ac0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c3fff640 1 -- 172.21.15.107:0/3918082472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30ac004d60 msgr2=0x7f30ac0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c3fff640 1 --2- 172.21.15.107:0/3918082472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30ac004d60 0x7f30ac0051e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.437+0000 7f30c3fff640 1 -- 172.21.15.107:0/3918082472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac003530 con 0x7f30ac0039b0 2023-10-24T23:58:05.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c3fff640 1 --2- 172.21.15.107:0/3918082472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30ac0039b0 0x7f30ac003dd0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f30b400ba00 tx=0x7f30b401c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921f83fc5da8bdeb server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:05.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b40aa050 con 0x7f30ac0039b0 2023-10-24T23:58:05.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b40ae5f0 con 0x7f30ac0039b0 2023-10-24T23:58:05.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c2ffd640 1 -- 172.21.15.107:0/3918082472 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b40ae9a0 con 0x7f30ac0039b0 2023-10-24T23:58:05.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3918082472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30ac0039b0 msgr2=0x7f30ac003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 --2- 172.21.15.107:0/3918082472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30ac0039b0 0x7f30ac003dd0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f30b400ba00 tx=0x7f30b401c660 comp rx=0 tx=0).stop 2023-10-24T23:58:05.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c8c6c640 1 -- 172.21.15.107:0/3918082472 reap_dead start 2023-10-24T23:58:05.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3918082472 shutdown_connections 2023-10-24T23:58:05.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3918082472 >> 172.21.15.107:0/3918082472 conn(0x7f30c40c39a0 msgr2=0x7f30c40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:05.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3918082472 shutdown_connections 2023-10-24T23:58:05.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3918082472 wait complete. 2023-10-24T23:58:05.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 Processor -- start 2023-10-24T23:58:05.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 -- start start 2023-10-24T23:58:05.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30ac0039b0 0x7f30c40c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c41514d0 0x7f30c4152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30c41365b0 0x7f30c4136a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4157460 con 0x7f30c41365b0 2023-10-24T23:58:05.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4156770 con 0x7f30ac0039b0 2023-10-24T23:58:05.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4156a50 con 0x7f30c41514d0 2023-10-24T23:58:05.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c8c6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30c41365b0 0x7f30c4136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c8c6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30c41365b0 0x7f30c4136a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57566/0 (socket says 172.21.15.107:57566) 2023-10-24T23:58:05.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c8c6c640 1 -- 172.21.15.107:0/3478603472 learned_addr learned my addr 172.21.15.107:0/3478603472 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:05.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c3fff640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30ac0039b0 0x7f30c40c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c37fe640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c41514d0 0x7f30c4152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c8c6c640 1 -- 172.21.15.107:0/3478603472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c41514d0 msgr2=0x7f30c4152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c8c6c640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c41514d0 0x7f30c4152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c8c6c640 1 -- 172.21.15.107:0/3478603472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30ac0039b0 msgr2=0x7f30c40c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c8c6c640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30ac0039b0 0x7f30c40c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c8c6c640 1 -- 172.21.15.107:0/3478603472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c4156d30 con 0x7f30c41365b0 2023-10-24T23:58:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c8c6c640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30c41365b0 0x7f30c4136a60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f30c4158230 tx=0x7f30bc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c0ff9640 1 -- 172.21.15.107:0/3478603472 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc003b80 con 0x7f30c41365b0 2023-10-24T23:58:05.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c0ff9640 1 -- 172.21.15.107:0/3478603472 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30bc003ce0 con 0x7f30c41365b0 2023-10-24T23:58:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3478603472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c40c58d0 con 0x7f30c41365b0 2023-10-24T23:58:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30c0ff9640 1 -- 172.21.15.107:0/3478603472 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30bc0052b0 con 0x7f30c41365b0 2023-10-24T23:58:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.441+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3478603472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c4136fa0 con 0x7f30c41365b0 2023-10-24T23:58:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.445+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3478603472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c40746e0 con 0x7f30c41365b0 2023-10-24T23:58:05.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.445+0000 7f30c0ff9640 1 -- 172.21.15.107:0/3478603472 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f30bc005410 con 0x7f30c41365b0 2023-10-24T23:58:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.445+0000 7f30c0ff9640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f30b0065e90 0x7f30b0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.445+0000 7f30c0ff9640 1 -- 172.21.15.107:0/3478603472 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f30bc0838c0 con 0x7f30c41365b0 2023-10-24T23:58:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.445+0000 7f30c3fff640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f30b0065e90 0x7f30b0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:05.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.445+0000 7f30c3fff640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f30b0065e90 0x7f30b0068350 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f30b40b0000 tx=0x7f30b40046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:05.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.449+0000 7f30c0ff9640 1 -- 172.21.15.107:0/3478603472 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f30bc0508c0 con 0x7f30c41365b0 2023-10-24T23:58:05.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3478603472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30c40c56c0 con 0x7f30c41365b0 2023-10-24T23:58:05.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30c0ff9640 1 -- 172.21.15.107:0/3478603472 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4257 (secure 0 0 0) 0x7f30bc018020 con 0x7f30c41365b0 2023-10-24T23:58:05.634 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:05.635 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:05.511797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:05.511797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:05.636 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 46 2023-10-24T23:58:05.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3478603472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f30b0065e90 msgr2=0x7f30b0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f30b0065e90 0x7f30b0068350 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f30b40b0000 tx=0x7f30b40046d0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3478603472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30c41365b0 msgr2=0x7f30c4136a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:05.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30c41365b0 0x7f30c4136a60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f30c4158230 tx=0x7f30bc00dc00 comp rx=0 tx=0).stop 2023-10-24T23:58:05.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3478603472 shutdown_connections 2023-10-24T23:58:05.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30c41365b0 0x7f30c4136a60 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f30b0065e90 0x7f30b0068350 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30c41514d0 0x7f30c4152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 --2- 172.21.15.107:0/3478603472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30ac0039b0 0x7f30c40c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3478603472 >> 172.21.15.107:0/3478603472 conn(0x7f30c40c39a0 msgr2=0x7f30c4071920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3478603472 shutdown_connections 2023-10-24T23:58:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:05.629+0000 7f30ca6cf640 1 -- 172.21.15.107:0/3478603472 wait complete. 2023-10-24T23:58:05.652 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 46, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 46, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5751": { "addr": "172.21.15.107:6835/1508236506", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 1508236506, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 1508236506, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5751, "incarnation": 44, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:58:05.511797+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5751 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 46, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4937": { "addr": "172.21.15.110:6835/2348043270", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2348043270, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2348043270, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4937, "incarnation": 44, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-24T23:58:05.511797+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4937 } } } ], "standbys": [] } 2023-10-24T23:58:07.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:07.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f13546de640 1 Processor -- start 2023-10-24T23:58:07.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f13546de640 1 -- start start 2023-10-24T23:58:07.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f13546de640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f134c0c8640 0x7f134c06a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f13546de640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c152400 con 0x7f134c1553d0 2023-10-24T23:58:07.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f13546de640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c1558d0 con 0x7f134c152020 2023-10-24T23:58:07.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f13546de640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134c0c6ce0 con 0x7f134c0c8640 2023-10-24T23:58:07.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1352c7b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f134c1553d0 0x7f134c06ac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40788/0 (socket says 172.21.15.107:40788) 2023-10-24T23:58:07.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1352c7b640 1 -- 172.21.15.107:0/330614977 learned_addr learned my addr 172.21.15.107:0/330614977 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:07.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065637133 0 0) 0x7f134c152400 con 0x7f134c1553d0 2023-10-24T23:58:07.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351c79640 1 --2- 172.21.15.107:0/330614977 >> v2:172.21.15.110:3300/0 conn(0x7f134c0c8640 0x7f134c06a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1338003530 con 0x7f134c1553d0 2023-10-24T23:58:07.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2605830217 0 0) 0x7f1338003530 con 0x7f134c1553d0 2023-10-24T23:58:07.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 >> v2:172.21.15.110:3300/0 conn(0x7f134c0c8640 msgr2=0x7f134c06a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:07.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 --2- 172.21.15.107:0/330614977 >> v2:172.21.15.110:3300/0 conn(0x7f134c0c8640 0x7f134c06a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 >> v1:172.21.15.110:6789/0 conn(0x7f134c152020 legacy=0x7f134c0c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:07.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c0c6fc0 con 0x7f134c1553d0 2023-10-24T23:58:07.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f1344001a00 con 0x7f134c1553d0 2023-10-24T23:58:07.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 >> v1:172.21.15.107:6789/0 conn(0x7f134c1553d0 legacy=0x7f134c06ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:07.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 --2- 172.21.15.107:0/330614977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1338003980 0x7f1338003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 --2- 172.21.15.107:0/330614977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13380047d0 0x7f1338004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 --2- 172.21.15.107:0/330614977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13380052b0 0x7f133800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c0c6ce0 con 0x7f13380052b0 2023-10-24T23:58:07.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133800c160 con 0x7f1338003980 2023-10-24T23:58:07.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133800c320 con 0x7f13380047d0 2023-10-24T23:58:07.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1352c7b640 1 --2- 172.21.15.107:0/330614977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13380052b0 0x7f133800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f135247a640 1 --2- 172.21.15.107:0/330614977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1338003980 0x7f1338003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1351c79640 1 --2- 172.21.15.107:0/330614977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13380047d0 0x7f1338004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1352c7b640 1 -- 172.21.15.107:0/330614977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13380047d0 msgr2=0x7f1338004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:07.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1352c7b640 1 --2- 172.21.15.107:0/330614977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13380047d0 0x7f1338004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1352c7b640 1 -- 172.21.15.107:0/330614977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1338003980 msgr2=0x7f1338003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:07.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1352c7b640 1 --2- 172.21.15.107:0/330614977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1338003980 0x7f1338003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.545+0000 7f1352c7b640 1 -- 172.21.15.107:0/330614977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1338003530 con 0x7f13380052b0 2023-10-24T23:58:07.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f1352c7b640 1 -- 172.21.15.107:0/330614977 reap_dead start 2023-10-24T23:58:07.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f1352c7b640 1 --2- 172.21.15.107:0/330614977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13380052b0 0x7f133800b990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1344002240 tx=0x7f134401fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f97c5024bd9690 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:07.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13440af070 con 0x7f13380052b0 2023-10-24T23:58:07.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13440aa470 con 0x7f13380052b0 2023-10-24T23:58:07.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f1351478640 1 -- 172.21.15.107:0/330614977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13440aa800 con 0x7f13380052b0 2023-10-24T23:58:07.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f13546de640 1 -- 172.21.15.107:0/330614977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13380052b0 msgr2=0x7f133800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:07.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f13546de640 1 --2- 172.21.15.107:0/330614977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13380052b0 0x7f133800b990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1344002240 tx=0x7f134401fa40 comp rx=0 tx=0).stop 2023-10-24T23:58:07.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f13546de640 1 -- 172.21.15.107:0/330614977 shutdown_connections 2023-10-24T23:58:07.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f13546de640 1 --2- 172.21.15.107:0/330614977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13380052b0 0x7f133800b990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f13546de640 1 -- 172.21.15.107:0/330614977 >> 172.21.15.107:0/330614977 conn(0x7f134c0abd60 msgr2=0x7f134c0ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:07.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f13546de640 1 -- 172.21.15.107:0/330614977 shutdown_connections 2023-10-24T23:58:07.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f13546de640 1 -- 172.21.15.107:0/330614977 wait complete. 2023-10-24T23:58:07.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.549+0000 7f13546de640 1 Processor -- start 2023-10-24T23:58:07.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f13546de640 1 -- start start 2023-10-24T23:58:07.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f13546de640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13380052b0 0x7f134c0af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f13546de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f134c14fea0 0x7f134c0af970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f13546de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f134c1ed010 0x7f134c1ed4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f13546de640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c0afeb0 con 0x7f134c14fea0 2023-10-24T23:58:07.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f13546de640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c14a290 con 0x7f13380052b0 2023-10-24T23:58:07.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f13546de640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c0bee50 con 0x7f134c1ed010 2023-10-24T23:58:07.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1351c79640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f134c14fea0 0x7f134c0af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1351c79640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f134c14fea0 0x7f134c0af970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57586/0 (socket says 172.21.15.107:57586) 2023-10-24T23:58:07.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1351c79640 1 -- 172.21.15.107:0/3666566036 learned_addr learned my addr 172.21.15.107:0/3666566036 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:07.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f135247a640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13380052b0 0x7f134c0af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1352c7b640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f134c1ed010 0x7f134c1ed4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1351c79640 1 -- 172.21.15.107:0/3666566036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f134c1ed010 msgr2=0x7f134c1ed4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:07.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1351c79640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f134c1ed010 0x7f134c1ed4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1351c79640 1 -- 172.21.15.107:0/3666566036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13380052b0 msgr2=0x7f134c0af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:07.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1351c79640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13380052b0 0x7f134c0af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1351c79640 1 -- 172.21.15.107:0/3666566036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c0c1ff0 con 0x7f134c14fea0 2023-10-24T23:58:07.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1352c7b640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f134c1ed010 0x7f134c1ed4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:07.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1351c79640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f134c14fea0 0x7f134c0af970 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f133c00e570 tx=0x7f133c00ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:07.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1342ffd640 1 -- 172.21.15.107:0/3666566036 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c013610 con 0x7f134c14fea0 2023-10-24T23:58:07.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1342ffd640 1 -- 172.21.15.107:0/3666566036 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c013770 con 0x7f134c14fea0 2023-10-24T23:58:07.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1342ffd640 1 -- 172.21.15.107:0/3666566036 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c013ae0 con 0x7f134c14fea0 2023-10-24T23:58:07.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f13546de640 1 -- 172.21.15.107:0/3666566036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c0c22d0 con 0x7f134c14fea0 2023-10-24T23:58:07.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f13546de640 1 -- 172.21.15.107:0/3666566036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c0c25b0 con 0x7f134c14fea0 2023-10-24T23:58:07.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f13546de640 1 -- 172.21.15.107:0/3666566036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134c0746e0 con 0x7f134c14fea0 2023-10-24T23:58:07.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.553+0000 7f1342ffd640 1 -- 172.21.15.107:0/3666566036 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f133c013c40 con 0x7f134c14fea0 2023-10-24T23:58:07.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.557+0000 7f1342ffd640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1330065e60 0x7f1330068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.557+0000 7f135247a640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1330065e60 0x7f1330068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.557+0000 7f1342ffd640 1 -- 172.21.15.107:0/3666566036 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f133c083a90 con 0x7f134c14fea0 2023-10-24T23:58:07.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.557+0000 7f135247a640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1330065e60 0x7f1330068320 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1348004e30 tx=0x7f1348009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:07.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.561+0000 7f1342ffd640 1 -- 172.21.15.107:0/3666566036 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f133c050b30 con 0x7f134c14fea0 2023-10-24T23:58:07.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:07.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.773+0000 7f13546de640 1 -- 172.21.15.107:0/3666566036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f134c0c7800 con 0x7f134c14fea0 2023-10-24T23:58:07.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.773+0000 7f1342ffd640 1 -- 172.21.15.107:0/3666566036 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4258 (secure 0 0 0) 0x7f133c054150 con 0x7f134c14fea0 2023-10-24T23:58:07.852 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:07.852 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:07.853 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 48 2023-10-24T23:58:07.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 -- 172.21.15.107:0/3666566036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1330065e60 msgr2=0x7f1330068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:07.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1330065e60 0x7f1330068320 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1348004e30 tx=0x7f1348009040 comp rx=0 tx=0).stop 2023-10-24T23:58:07.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 -- 172.21.15.107:0/3666566036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f134c14fea0 msgr2=0x7f134c0af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:07.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f134c14fea0 0x7f134c0af970 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f133c00e570 tx=0x7f133c00ea30 comp rx=0 tx=0).stop 2023-10-24T23:58:07.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 -- 172.21.15.107:0/3666566036 shutdown_connections 2023-10-24T23:58:07.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1330065e60 0x7f1330068320 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f134c1ed010 0x7f134c1ed4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f134c14fea0 0x7f134c0af970 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 --2- 172.21.15.107:0/3666566036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13380052b0 0x7f134c0af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 -- 172.21.15.107:0/3666566036 >> 172.21.15.107:0/3666566036 conn(0x7f134c0abd60 msgr2=0x7f134c0c9210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:07.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 -- 172.21.15.107:0/3666566036 shutdown_connections 2023-10-24T23:58:07.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.777+0000 7f13546de640 1 -- 172.21.15.107:0/3666566036 wait complete. 2023-10-24T23:58:07.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.789+0000 7f7a835e9640 1 Processor -- start 2023-10-24T23:58:07.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.789+0000 7f7a835e9640 1 -- start start 2023-10-24T23:58:07.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a835e9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7a7c152040 0x7f7a7c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a835e9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7c155680 con 0x7f7a7c0c8100 2023-10-24T23:58:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a835e9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7c154a70 con 0x7f7a7c06a4b0 2023-10-24T23:58:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a835e9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7c154d50 con 0x7f7a7c152040 2023-10-24T23:58:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a81385640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7a7c152040 0x7f7a7c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a81385640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7a7c152040 0x7f7a7c0c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57588/0 (socket says 172.21.15.107:57588) 2023-10-24T23:58:07.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a81385640 1 -- 172.21.15.107:0/1101755234 learned_addr learned my addr 172.21.15.107:0/1101755234 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:07.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a81385640 1 -- 172.21.15.107:0/1101755234 >> v1:172.21.15.110:6789/0 conn(0x7f7a7c06a4b0 legacy=0x7f7a7c0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:07.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a81385640 1 -- 172.21.15.107:0/1101755234 >> v1:172.21.15.107:6789/0 conn(0x7f7a7c0c8100 legacy=0x7f7a7c0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:07.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a81385640 1 -- 172.21.15.107:0/1101755234 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7c155030 con 0x7f7a7c152040 2023-10-24T23:58:07.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a81385640 1 --2- 172.21.15.107:0/1101755234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a7c152040 0x7f7a7c0c4b80 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009870 tx=0x7f7a7801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869121eaaa5122dc server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a73fff640 1 -- 172.21.15.107:0/1101755234 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a780ab600 con 0x7f7a7c152040 2023-10-24T23:58:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a73fff640 1 -- 172.21.15.107:0/1101755234 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a780a7050 con 0x7f7a7c152040 2023-10-24T23:58:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.793+0000 7f7a73fff640 1 -- 172.21.15.107:0/1101755234 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a780ab980 con 0x7f7a7c152040 2023-10-24T23:58:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 -- 172.21.15.107:0/1101755234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a7c152040 msgr2=0x7f7a7c0c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 --2- 172.21.15.107:0/1101755234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a7c152040 0x7f7a7c0c4b80 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7a78009870 tx=0x7f7a7801c3e0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 -- 172.21.15.107:0/1101755234 shutdown_connections 2023-10-24T23:58:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 --2- 172.21.15.107:0/1101755234 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a7c152040 0x7f7a7c0c4b80 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 -- 172.21.15.107:0/1101755234 >> 172.21.15.107:0/1101755234 conn(0x7f7a7c0ca200 msgr2=0x7f7a7c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 -- 172.21.15.107:0/1101755234 shutdown_connections 2023-10-24T23:58:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 -- 172.21.15.107:0/1101755234 wait complete. 2023-10-24T23:58:07.863 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=5562 2023-10-24T23:58:07.864 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.a 2023-10-24T23:58:07.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 Processor -- start 2023-10-24T23:58:07.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 -- start start 2023-10-24T23:58:07.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a7c06a4b0 0x7f7a7c06aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a7c0c8100 0x7f7a7c0c2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a7c152040 0x7f7a7c0c2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0aaf40 con 0x7f7a7c0c8100 2023-10-24T23:58:07.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0ab220 con 0x7f7a7c06a4b0 2023-10-24T23:58:07.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a835e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0c3090 con 0x7f7a7c152040 2023-10-24T23:58:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a80b84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a7c0c8100 0x7f7a7c0c2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a80b84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a7c0c8100 0x7f7a7c0c2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57598/0 (socket says 172.21.15.107:57598) 2023-10-24T23:58:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a80b84640 1 -- 172.21.15.107:0/2743189111 learned_addr learned my addr 172.21.15.107:0/2743189111 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a81385640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a7c06a4b0 0x7f7a7c06aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a81b86640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a7c152040 0x7f7a7c0c2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a80b84640 1 -- 172.21.15.107:0/2743189111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a7c152040 msgr2=0x7f7a7c0c2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:07.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a80b84640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a7c152040 0x7f7a7c0c2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a80b84640 1 -- 172.21.15.107:0/2743189111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a7c06a4b0 msgr2=0x7f7a7c06aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:07.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a80b84640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a7c06a4b0 0x7f7a7c06aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:07.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a80b84640 1 -- 172.21.15.107:0/2743189111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a780094d0 con 0x7f7a7c0c8100 2023-10-24T23:58:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a81385640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a7c06a4b0 0x7f7a7c06aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:58:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.797+0000 7f7a80b84640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a7c0c8100 0x7f7a7c0c2610 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00c500 tx=0x7f7a6c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.801+0000 7f7a71ffb640 1 -- 172.21.15.107:0/2743189111 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c0052d0 con 0x7f7a7c0c8100 2023-10-24T23:58:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.801+0000 7f7a71ffb640 1 -- 172.21.15.107:0/2743189111 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6c005430 con 0x7f7a7c0c8100 2023-10-24T23:58:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.801+0000 7f7a835e9640 1 -- 172.21.15.107:0/2743189111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7c0ab6b0 con 0x7f7a7c0c8100 2023-10-24T23:58:07.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.801+0000 7f7a71ffb640 1 -- 172.21.15.107:0/2743189111 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6c012830 con 0x7f7a7c0c8100 2023-10-24T23:58:07.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.801+0000 7f7a835e9640 1 -- 172.21.15.107:0/2743189111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7c0abb70 con 0x7f7a7c0c8100 2023-10-24T23:58:07.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.801+0000 7f7a835e9640 1 -- 172.21.15.107:0/2743189111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7c0ab3b0 con 0x7f7a7c0c8100 2023-10-24T23:58:07.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.801+0000 7f7a71ffb640 1 -- 172.21.15.107:0/2743189111 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7a6c012990 con 0x7f7a7c0c8100 2023-10-24T23:58:07.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.801+0000 7f7a71ffb640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a60065e60 0x7f7a60068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:07.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.801+0000 7f7a71ffb640 1 -- 172.21.15.107:0/2743189111 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f7a6c084420 con 0x7f7a7c0c8100 2023-10-24T23:58:07.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.801+0000 7f7a81385640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a60065e60 0x7f7a60068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:07.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.801+0000 7f7a81385640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a60065e60 0x7f7a60068320 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7a780ad000 tx=0x7f7a780a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:07.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:07.805+0000 7f7a71ffb640 1 -- 172.21.15.107:0/2743189111 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7a6c051460 con 0x7f7a7c0c8100 2023-10-24T23:58:08.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.025+0000 7f7a835e9640 1 -- 172.21.15.107:0/2743189111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a7c0746e0 con 0x7f7a7c0c8100 2023-10-24T23:58:08.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.025+0000 7f7a71ffb640 1 -- 172.21.15.107:0/2743189111 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4258 (secure 0 0 0) 0x7f7a6c019300 con 0x7f7a7c0c8100 2023-10-24T23:58:08.032 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:08.032 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:08.033 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 48 2023-10-24T23:58:08.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 -- 172.21.15.107:0/2743189111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a60065e60 msgr2=0x7f7a60068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:08.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a60065e60 0x7f7a60068320 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7a780ad000 tx=0x7f7a780a6000 comp rx=0 tx=0).stop 2023-10-24T23:58:08.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 -- 172.21.15.107:0/2743189111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a7c0c8100 msgr2=0x7f7a7c0c2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:08.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a7c0c8100 0x7f7a7c0c2610 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c00c500 tx=0x7f7a6c00c9c0 comp rx=0 tx=0).stop 2023-10-24T23:58:08.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 -- 172.21.15.107:0/2743189111 shutdown_connections 2023-10-24T23:58:08.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a60065e60 0x7f7a60068320 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:08.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a7c152040 0x7f7a7c0c2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:08.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a7c0c8100 0x7f7a7c0c2610 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:08.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 --2- 172.21.15.107:0/2743189111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a7c06a4b0 0x7f7a7c06aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:08.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 -- 172.21.15.107:0/2743189111 >> 172.21.15.107:0/2743189111 conn(0x7f7a7c0ca200 msgr2=0x7f7a7c14a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:08.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 -- 172.21.15.107:0/2743189111 shutdown_connections 2023-10-24T23:58:08.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.029+0000 7f7a835e9640 1 -- 172.21.15.107:0/2743189111 wait complete. 2023-10-24T23:58:08.050 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=4232 2023-10-24T23:58:08.050 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before reviving mds.b 2023-10-24T23:58:08.687 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-24T23:58:08.681+0000 7f304e439640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:58:08.864 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2023-10-24T23:58:08.865 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-24T23:58:08.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-24T23:58:08.868 INFO:tasks.ceph.mds.a:Started 2023-10-24T23:58:08.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:08.926 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-24T23:58:08.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde9fda640 1 Processor -- start 2023-10-24T23:58:08.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde9fda640 1 -- start start 2023-10-24T23:58:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde9fda640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbde4156770 0x7fbde40ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde9fda640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde40c1ca0 con 0x7fbde40c70e0 2023-10-24T23:58:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde9fda640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde40c10c0 con 0x7fbde40be6b0 2023-10-24T23:58:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde9fda640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde40c27e0 con 0x7fbde4156770 2023-10-24T23:58:08.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbde4156770 0x7fbde40ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:08.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbde4156770 0x7fbde40ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57618/0 (socket says 172.21.15.107:57618) 2023-10-24T23:58:08.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde37fe640 1 -- 172.21.15.107:0/675021843 learned_addr learned my addr 172.21.15.107:0/675021843 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:08.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde37fe640 1 -- 172.21.15.107:0/675021843 >> v1:172.21.15.110:6789/0 conn(0x7fbde40be6b0 legacy=0x7fbde415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:08.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde37fe640 1 -- 172.21.15.107:0/675021843 >> v1:172.21.15.107:6789/0 conn(0x7fbde40c70e0 legacy=0x7fbde40bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:08.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.977+0000 7fbde37fe640 1 -- 172.21.15.107:0/675021843 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde40c13a0 con 0x7fbde4156770 2023-10-24T23:58:08.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde37fe640 1 --2- 172.21.15.107:0/675021843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbde4156770 0x7fbde40ab7c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4009870 tx=0x7fbdd401bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68f0e4db9af791f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:08.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde2ffd640 1 -- 172.21.15.107:0/675021843 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd40a4750 con 0x7fbde4156770 2023-10-24T23:58:08.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde2ffd640 1 -- 172.21.15.107:0/675021843 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd40a48b0 con 0x7fbde4156770 2023-10-24T23:58:08.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde2ffd640 1 -- 172.21.15.107:0/675021843 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd40a4c40 con 0x7fbde4156770 2023-10-24T23:58:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde9fda640 1 -- 172.21.15.107:0/675021843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbde4156770 msgr2=0x7fbde40ab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde9fda640 1 --2- 172.21.15.107:0/675021843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbde4156770 0x7fbde40ab7c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4009870 tx=0x7fbdd401bf90 comp rx=0 tx=0).stop 2023-10-24T23:58:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde9fda640 1 -- 172.21.15.107:0/675021843 shutdown_connections 2023-10-24T23:58:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde9fda640 1 --2- 172.21.15.107:0/675021843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbde4156770 0x7fbde40ab7c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde9fda640 1 -- 172.21.15.107:0/675021843 >> 172.21.15.107:0/675021843 conn(0x7fbde40c65c0 msgr2=0x7fbde40c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:08.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde9fda640 1 -- 172.21.15.107:0/675021843 shutdown_connections 2023-10-24T23:58:08.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde9fda640 1 -- 172.21.15.107:0/675021843 wait complete. 2023-10-24T23:58:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde9fda640 1 Processor -- start 2023-10-24T23:58:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.981+0000 7fbde9fda640 1 -- start start 2023-10-24T23:58:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde9fda640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbde40be6b0 0x7fbde414c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde9fda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbde40c70e0 0x7fbde414b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde9fda640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbde4156770 0x7fbde4071a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde9fda640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde4071610 con 0x7fbde40c70e0 2023-10-24T23:58:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde9fda640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde4149770 con 0x7fbde40be6b0 2023-10-24T23:58:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde9fda640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde4148ed0 con 0x7fbde4156770 2023-10-24T23:58:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbddbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbde40c70e0 0x7fbde414b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbddbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbde40c70e0 0x7fbde414b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57622/0 (socket says 172.21.15.107:57622) 2023-10-24T23:58:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbddbfff640 1 -- 172.21.15.107:0/742666316 learned_addr learned my addr 172.21.15.107:0/742666316 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde37fe640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbde40be6b0 0x7fbde414c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde3fff640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbde4156770 0x7fbde4071a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbddbfff640 1 -- 172.21.15.107:0/742666316 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbde4156770 msgr2=0x7fbde4071a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbddbfff640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbde4156770 0x7fbde4071a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbddbfff640 1 -- 172.21.15.107:0/742666316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbde40be6b0 msgr2=0x7fbde414c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbddbfff640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbde40be6b0 0x7fbde414c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbddbfff640 1 -- 172.21.15.107:0/742666316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd40094d0 con 0x7fbde40c70e0 2023-10-24T23:58:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbddbfff640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbde40c70e0 0x7fbde414b760 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc00c750 tx=0x7fbdcc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde0ff9640 1 -- 172.21.15.107:0/742666316 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc005280 con 0x7fbde40c70e0 2023-10-24T23:58:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde0ff9640 1 -- 172.21.15.107:0/742666316 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdcc0053e0 con 0x7fbde40c70e0 2023-10-24T23:58:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde0ff9640 1 -- 172.21.15.107:0/742666316 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc00a860 con 0x7fbde40c70e0 2023-10-24T23:58:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde9fda640 1 -- 172.21.15.107:0/742666316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde41491b0 con 0x7fbde40c70e0 2023-10-24T23:58:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde9fda640 1 -- 172.21.15.107:0/742666316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde4070ec0 con 0x7fbde40c70e0 2023-10-24T23:58:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde9fda640 1 -- 172.21.15.107:0/742666316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde4071750 con 0x7fbde40c70e0 2023-10-24T23:58:08.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde0ff9640 1 -- 172.21.15.107:0/742666316 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbdcc00a9c0 con 0x7fbde40c70e0 2023-10-24T23:58:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde0ff9640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdc4065e60 0x7fbdc4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde37fe640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdc4065e60 0x7fbdc4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.985+0000 7fbde0ff9640 1 -- 172.21.15.107:0/742666316 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7fbdcc0834c0 con 0x7fbde40c70e0 2023-10-24T23:58:08.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.989+0000 7fbde37fe640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdc4065e60 0x7fbdc4068320 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbdd40ae000 tx=0x7fbdd4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:08.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:08.993+0000 7fbde0ff9640 1 -- 172.21.15.107:0/742666316 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbde4071750 con 0x7fbde40c70e0 2023-10-24T23:58:09.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.209+0000 7fbde9fda640 1 -- 172.21.15.107:0/742666316 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbde4071750 con 0x7fbde40c70e0 2023-10-24T23:58:09.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.209+0000 7fbde0ff9640 1 -- 172.21.15.107:0/742666316 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4258 (secure 0 0 0) 0x7fbde4071750 con 0x7fbde40c70e0 2023-10-24T23:58:09.217 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:09.217 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:09.219 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 48 2023-10-24T23:58:09.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 -- 172.21.15.107:0/742666316 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdc4065e60 msgr2=0x7fbdc4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:09.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdc4065e60 0x7fbdc4068320 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fbdd40ae000 tx=0x7fbdd4009040 comp rx=0 tx=0).stop 2023-10-24T23:58:09.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 -- 172.21.15.107:0/742666316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbde40c70e0 msgr2=0x7fbde414b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:09.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbde40c70e0 0x7fbde414b760 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc00c750 tx=0x7fbdcc00cc10 comp rx=0 tx=0).stop 2023-10-24T23:58:09.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 -- 172.21.15.107:0/742666316 shutdown_connections 2023-10-24T23:58:09.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdc4065e60 0x7fbdc4068320 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:09.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbde4156770 0x7fbde4071a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:09.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbde40c70e0 0x7fbde414b760 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:09.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 --2- 172.21.15.107:0/742666316 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbde40be6b0 0x7fbde414c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:09.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 -- 172.21.15.107:0/742666316 >> 172.21.15.107:0/742666316 conn(0x7fbde40c65c0 msgr2=0x7fbde40c4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:09.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 -- 172.21.15.107:0/742666316 shutdown_connections 2023-10-24T23:58:09.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:09.213+0000 7fbde9fda640 1 -- 172.21.15.107:0/742666316 wait complete. 2023-10-24T23:58:09.238 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-24T23:58:10.652 INFO:tasks.ceph.mds.d.smithi110.stderr:2023-10-24T23:58:10.648+0000 7fcc15e6c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:58:11.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6f8cd640 1 Processor -- start 2023-10-24T23:58:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6f8cd640 1 -- start start 2023-10-24T23:58:11.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6f8cd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c68152040 0x7f5c680c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:11.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6f8cd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5c6806a4b0 0x7f5c680ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:11.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6f8cd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c68155680 con 0x7f5c680c8100 2023-10-24T23:58:11.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6f8cd640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c68154a70 con 0x7f5c68152040 2023-10-24T23:58:11.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6f8cd640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c68154d50 con 0x7f5c6806a4b0 2023-10-24T23:58:11.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6d669640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c68152040 0x7f5c680c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:11.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6d669640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5c68152040 0x7f5c680c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57626/0 (socket says 172.21.15.107:57626) 2023-10-24T23:58:11.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6d669640 1 -- 172.21.15.107:0/636906063 learned_addr learned my addr 172.21.15.107:0/636906063 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:11.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6de6a640 1 --2- 172.21.15.107:0/636906063 >> v2:172.21.15.110:3300/0 conn(0x7f5c6806a4b0 0x7f5c680ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:11.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6d669640 1 -- 172.21.15.107:0/636906063 >> v2:172.21.15.110:3300/0 conn(0x7f5c6806a4b0 msgr2=0x7f5c680ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6d669640 1 --2- 172.21.15.107:0/636906063 >> v2:172.21.15.110:3300/0 conn(0x7f5c6806a4b0 0x7f5c680ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6d669640 1 -- 172.21.15.107:0/636906063 >> v1:172.21.15.110:6789/0 conn(0x7f5c680c8100 legacy=0x7f5c680abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6d669640 1 -- 172.21.15.107:0/636906063 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c68155030 con 0x7f5c68152040 2023-10-24T23:58:11.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c6d669640 1 --2- 172.21.15.107:0/636906063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c68152040 0x7f5c680c4b80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5c6806b590 tx=0x7f5c6401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5595f21a78771fc server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:11.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c5ffff640 1 -- 172.21.15.107:0/636906063 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c640ab600 con 0x7f5c68152040 2023-10-24T23:58:11.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c5ffff640 1 -- 172.21.15.107:0/636906063 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c640a7050 con 0x7f5c68152040 2023-10-24T23:58:11.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.373+0000 7f5c5ffff640 1 -- 172.21.15.107:0/636906063 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c640ab980 con 0x7f5c68152040 2023-10-24T23:58:11.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/636906063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c68152040 msgr2=0x7f5c680c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:11.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 --2- 172.21.15.107:0/636906063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c68152040 0x7f5c680c4b80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5c6806b590 tx=0x7f5c6401c430 comp rx=0 tx=0).stop 2023-10-24T23:58:11.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/636906063 shutdown_connections 2023-10-24T23:58:11.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 --2- 172.21.15.107:0/636906063 >> v2:172.21.15.110:3300/0 conn(0x7f5c6806a4b0 0x7f5c680ac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:11.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 --2- 172.21.15.107:0/636906063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c68152040 0x7f5c680c4b80 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:11.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/636906063 >> 172.21.15.107:0/636906063 conn(0x7f5c680ca200 msgr2=0x7f5c680ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/636906063 shutdown_connections 2023-10-24T23:58:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/636906063 wait complete. 2023-10-24T23:58:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 Processor -- start 2023-10-24T23:58:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 -- start start 2023-10-24T23:58:11.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c6806a4b0 0x7f5c681513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:11.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c680c8100 0x7f5c6814a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:11.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c68152040 0x7f5c681474e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:11.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6814f770 con 0x7f5c68152040 2023-10-24T23:58:11.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c68150330 con 0x7f5c6806a4b0 2023-10-24T23:58:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c68150000 con 0x7f5c680c8100 2023-10-24T23:58:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6de6a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c68152040 0x7f5c681474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6de6a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c68152040 0x7f5c681474e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57640/0 (socket says 172.21.15.107:57640) 2023-10-24T23:58:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6de6a640 1 -- 172.21.15.107:0/874999093 learned_addr learned my addr 172.21.15.107:0/874999093 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6ce68640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c680c8100 0x7f5c6814a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6d669640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c6806a4b0 0x7f5c681513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6de6a640 1 -- 172.21.15.107:0/874999093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c680c8100 msgr2=0x7f5c6814a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6de6a640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c680c8100 0x7f5c6814a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6de6a640 1 -- 172.21.15.107:0/874999093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c6806a4b0 msgr2=0x7f5c681513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6de6a640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c6806a4b0 0x7f5c681513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:11.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6de6a640 1 -- 172.21.15.107:0/874999093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c640094d0 con 0x7f5c68152040 2023-10-24T23:58:11.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6ce68640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c680c8100 0x7f5c6814a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:58:11.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6de6a640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c68152040 0x7f5c681474e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f5c60007e60 tx=0x7f5c6000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:11.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c5dffb640 1 -- 172.21.15.107:0/874999093 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60014420 con 0x7f5c68152040 2023-10-24T23:58:11.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c5dffb640 1 -- 172.21.15.107:0/874999093 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c600145a0 con 0x7f5c68152040 2023-10-24T23:58:11.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/874999093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c681f0c10 con 0x7f5c68152040 2023-10-24T23:58:11.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c5dffb640 1 -- 172.21.15.107:0/874999093 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60014950 con 0x7f5c68152040 2023-10-24T23:58:11.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.377+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/874999093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c681f0ef0 con 0x7f5c68152040 2023-10-24T23:58:11.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.381+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/874999093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6814fe50 con 0x7f5c68152040 2023-10-24T23:58:11.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.381+0000 7f5c5dffb640 1 -- 172.21.15.107:0/874999093 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5c60014ab0 con 0x7f5c68152040 2023-10-24T23:58:11.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.381+0000 7f5c5dffb640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c4c065e60 0x7f5c4c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:11.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.381+0000 7f5c6d669640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c4c065e60 0x7f5c4c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:11.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.381+0000 7f5c5dffb640 1 -- 172.21.15.107:0/874999093 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f5c60084700 con 0x7f5c68152040 2023-10-24T23:58:11.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.381+0000 7f5c6d669640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c4c065e60 0x7f5c4c068320 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5c680c4500 tx=0x7f5c64009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:11.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.385+0000 7f5c5dffb640 1 -- 172.21.15.107:0/874999093 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5c60051710 con 0x7f5c68152040 2023-10-24T23:58:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.621+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/874999093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c681504a0 con 0x7f5c68152040 2023-10-24T23:58:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c5dffb640 1 -- 172.21.15.107:0/874999093 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5017 (secure 0 0 0) 0x7f5c60054d30 con 0x7f5c68152040 2023-10-24T23:58:11.629 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:11.630 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:11.630 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 49 2023-10-24T23:58:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/874999093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c4c065e60 msgr2=0x7f5c4c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c4c065e60 0x7f5c4c068320 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5c680c4500 tx=0x7f5c64009040 comp rx=0 tx=0).stop 2023-10-24T23:58:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/874999093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c68152040 msgr2=0x7f5c681474e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c68152040 0x7f5c681474e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f5c60007e60 tx=0x7f5c6000e4b0 comp rx=0 tx=0).stop 2023-10-24T23:58:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/874999093 shutdown_connections 2023-10-24T23:58:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c68152040 0x7f5c681474e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c4c065e60 0x7f5c4c068320 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c680c8100 0x7f5c6814a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 --2- 172.21.15.107:0/874999093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c6806a4b0 0x7f5c681513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/874999093 >> 172.21.15.107:0/874999093 conn(0x7f5c680ca200 msgr2=0x7f5c68154040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/874999093 shutdown_connections 2023-10-24T23:58:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:11.625+0000 7f5c6f8cd640 1 -- 172.21.15.107:0/874999093 wait complete. 2023-10-24T23:58:11.656 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2023-10-24T23:58:11.656 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2023-10-24T23:58:13.051 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2023-10-24T23:58:13.051 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-24T23:58:13.052 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-24T23:58:13.054 INFO:tasks.ceph.mds.b:Started 2023-10-24T23:58:13.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:13.144 INFO:tasks.ceph.mds.b.smithi110.stdout:starting mds.b at 2023-10-24T23:58:13.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.205+0000 7f707c81a640 1 Processor -- start 2023-10-24T23:58:13.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.205+0000 7f707c81a640 1 -- start start 2023-10-24T23:58:13.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f707c81a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f707813bb00 0x7f70781531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:13.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f707c81a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f70780aa330 0x7f707806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:13.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f707c81a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7078152d40 con 0x7f70780a9450 2023-10-24T23:58:13.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f707c81a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7078156100 con 0x7f70780aa330 2023-10-24T23:58:13.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f707c81a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70781527c0 con 0x7f707813bb00 2023-10-24T23:58:13.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f706ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f70780aa330 0x7f707806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f707659c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f707813bb00 0x7f70781531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f7076d9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f70780a9450 0x7f7078071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:40100/0 (socket says 172.21.15.107:40100) 2023-10-24T23:58:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f7076d9d640 1 -- 172.21.15.107:0/2511734156 learned_addr learned my addr 172.21.15.107:0/2511734156 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f707659c640 1 -- 172.21.15.107:0/2511734156 >> v2:172.21.15.110:3300/0 conn(0x7f70780aa330 msgr2=0x7f707806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f707659c640 1 --2- 172.21.15.107:0/2511734156 >> v2:172.21.15.110:3300/0 conn(0x7f70780aa330 0x7f707806e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f707659c640 1 -- 172.21.15.107:0/2511734156 >> v1:172.21.15.110:6789/0 conn(0x7f70780a9450 legacy=0x7f7078071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:13.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f707659c640 1 -- 172.21.15.107:0/2511734156 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707806d660 con 0x7f707813bb00 2023-10-24T23:58:13.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f706ffff640 1 --2- 172.21.15.107:0/2511734156 >> v2:172.21.15.110:3300/0 conn(0x7f70780aa330 0x7f707806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:58:13.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.209+0000 7f707659c640 1 --2- 172.21.15.107:0/2511734156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f707813bb00 0x7f70781531d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f70680097f0 tx=0x7f706801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4831a9c3c83b6da9 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:13.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f7075d9b640 1 -- 172.21.15.107:0/2511734156 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70680ab600 con 0x7f707813bb00 2023-10-24T23:58:13.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f7075d9b640 1 -- 172.21.15.107:0/2511734156 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70680a7050 con 0x7f707813bb00 2023-10-24T23:58:13.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f7075d9b640 1 -- 172.21.15.107:0/2511734156 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70680ab980 con 0x7f707813bb00 2023-10-24T23:58:13.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 -- 172.21.15.107:0/2511734156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f707813bb00 msgr2=0x7f70781531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:13.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 --2- 172.21.15.107:0/2511734156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f707813bb00 0x7f70781531d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f70680097f0 tx=0x7f706801c360 comp rx=0 tx=0).stop 2023-10-24T23:58:13.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 -- 172.21.15.107:0/2511734156 shutdown_connections 2023-10-24T23:58:13.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 --2- 172.21.15.107:0/2511734156 >> v2:172.21.15.110:3300/0 conn(0x7f70780aa330 0x7f707806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:13.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 --2- 172.21.15.107:0/2511734156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f707813bb00 0x7f70781531d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:13.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 -- 172.21.15.107:0/2511734156 >> 172.21.15.107:0/2511734156 conn(0x7f7078070f60 msgr2=0x7f7078071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:13.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 -- 172.21.15.107:0/2511734156 shutdown_connections 2023-10-24T23:58:13.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 -- 172.21.15.107:0/2511734156 wait complete. 2023-10-24T23:58:13.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 Processor -- start 2023-10-24T23:58:13.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 -- start start 2023-10-24T23:58:13.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70780a9450 0x7f70781389a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:13.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70780aa330 0x7f70781335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:13.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707813bb00 0x7f7078133ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:13.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780ab3b0 con 0x7f707813bb00 2023-10-24T23:58:13.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70781ee3b0 con 0x7f70780a9450 2023-10-24T23:58:13.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707c81a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70781ee8e0 con 0x7f70780aa330 2023-10-24T23:58:13.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f7076d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707813bb00 0x7f7078133ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:13.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70780a9450 0x7f70781389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:13.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f706ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70780aa330 0x7f70781335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:13.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f7076d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707813bb00 0x7f7078133ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40660/0 (socket says 172.21.15.107:40660) 2023-10-24T23:58:13.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f7076d9d640 1 -- 172.21.15.107:0/3509364821 learned_addr learned my addr 172.21.15.107:0/3509364821 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707659c640 1 -- 172.21.15.107:0/3509364821 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70780aa330 msgr2=0x7f70781335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707659c640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70780aa330 0x7f70781335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707659c640 1 -- 172.21.15.107:0/3509364821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707813bb00 msgr2=0x7f7078133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707659c640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707813bb00 0x7f7078133ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f707659c640 1 -- 172.21.15.107:0/3509364821 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7060009420 con 0x7f70780a9450 2023-10-24T23:58:13.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f7076d9d640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707813bb00 0x7f7078133ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:58:13.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.213+0000 7f706ffff640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70780aa330 0x7f70781335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:58:13.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f707659c640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70780a9450 0x7f70781389a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f70680ab000 tx=0x7f7068019100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:13.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f706effd640 1 -- 172.21.15.107:0/3509364821 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70680b6890 con 0x7f70780a9450 2023-10-24T23:58:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f706effd640 1 -- 172.21.15.107:0/3509364821 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70680b6a30 con 0x7f70780a9450 2023-10-24T23:58:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f707c81a640 1 -- 172.21.15.107:0/3509364821 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7068009420 con 0x7f70780a9450 2023-10-24T23:58:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f706effd640 1 -- 172.21.15.107:0/3509364821 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70680b6de0 con 0x7f70780a9450 2023-10-24T23:58:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f707c81a640 1 -- 172.21.15.107:0/3509364821 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70780aadf0 con 0x7f70780a9450 2023-10-24T23:58:13.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f707c81a640 1 -- 172.21.15.107:0/3509364821 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7078151c50 con 0x7f70780a9450 2023-10-24T23:58:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f706effd640 1 -- 172.21.15.107:0/3509364821 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f70680b2040 con 0x7f70780a9450 2023-10-24T23:58:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f706effd640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7058065e60 0x7f7058068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f706effd640 1 -- 172.21.15.107:0/3509364821 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f706811e790 con 0x7f70780a9450 2023-10-24T23:58:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f706ffff640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7058065e60 0x7f7058068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.217+0000 7f706ffff640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7058065e60 0x7f7058068320 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f70780aa770 tx=0x7f7060009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.221+0000 7f706effd640 1 -- 172.21.15.107:0/3509364821 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f70680eba00 con 0x7f70780a9450 2023-10-24T23:58:13.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.441+0000 7f707c81a640 1 -- 172.21.15.107:0/3509364821 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70780746e0 con 0x7f70780a9450 2023-10-24T23:58:13.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.441+0000 7f706effd640 1 -- 172.21.15.107:0/3509364821 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5017 (secure 0 0 0) 0x7f70680b0020 con 0x7f70780a9450 2023-10-24T23:58:13.448 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:13.448 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:13.449 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 49 2023-10-24T23:58:13.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 -- 172.21.15.107:0/3509364821 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7058065e60 msgr2=0x7f7058068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:13.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7058065e60 0x7f7058068320 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f70780aa770 tx=0x7f7060009190 comp rx=0 tx=0).stop 2023-10-24T23:58:13.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 -- 172.21.15.107:0/3509364821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70780a9450 msgr2=0x7f70781389a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:13.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70780a9450 0x7f70781389a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f70680ab000 tx=0x7f7068019100 comp rx=0 tx=0).stop 2023-10-24T23:58:13.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 -- 172.21.15.107:0/3509364821 shutdown_connections 2023-10-24T23:58:13.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f707813bb00 0x7f7078133ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:13.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7058065e60 0x7f7058068320 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:13.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70780aa330 0x7f70781335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:13.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 --2- 172.21.15.107:0/3509364821 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70780a9450 0x7f70781389a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:13.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 -- 172.21.15.107:0/3509364821 >> 172.21.15.107:0/3509364821 conn(0x7f7078070f60 msgr2=0x7f7078150fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:13.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 -- 172.21.15.107:0/3509364821 shutdown_connections 2023-10-24T23:58:13.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:13.445+0000 7f707c81a640 1 -- 172.21.15.107:0/3509364821 wait complete. 2023-10-24T23:58:13.469 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-24T23:58:15.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:15.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f025cb1f640 1 Processor -- start 2023-10-24T23:58:15.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f025cb1f640 1 -- start start 2023-10-24T23:58:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f025cb1f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02580c27d0 0x7f02580c2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f025cb1f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f02580724b0 0x7f02580c1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f025cb1f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0258069ef0 con 0x7f0258072030 2023-10-24T23:58:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f025cb1f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02581556d0 con 0x7f02580724b0 2023-10-24T23:58:15.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f025cb1f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0258154860 con 0x7f02580c27d0 2023-10-24T23:58:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f0256ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0258072030 0x7f02580c1720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48182/0 (socket says 172.21.15.107:48182) 2023-10-24T23:58:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f0256ffd640 1 -- 172.21.15.107:0/2357960819 learned_addr learned my addr 172.21.15.107:0/2357960819 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f0257fff640 1 --2- 172.21.15.107:0/2357960819 >> v2:172.21.15.110:3300/0 conn(0x7f02580724b0 0x7f02580c1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f02577fe640 1 --2- 172.21.15.107:0/2357960819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02580c27d0 0x7f02580c2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f02567fc640 1 -- 172.21.15.107:0/2357960819 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524477084 0 0) 0x7f024c001120 con 0x7f0258072030 2023-10-24T23:58:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f02567fc640 1 -- 172.21.15.107:0/2357960819 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0240003530 con 0x7f0258072030 2023-10-24T23:58:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f02577fe640 1 -- 172.21.15.107:0/2357960819 >> v2:172.21.15.110:3300/0 conn(0x7f02580724b0 msgr2=0x7f02580c1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f02577fe640 1 --2- 172.21.15.107:0/2357960819 >> v2:172.21.15.110:3300/0 conn(0x7f02580724b0 0x7f02580c1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f02577fe640 1 -- 172.21.15.107:0/2357960819 >> v1:172.21.15.107:6789/0 conn(0x7f0258072030 legacy=0x7f02580c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.597+0000 7f02577fe640 1 -- 172.21.15.107:0/2357960819 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025806dab0 con 0x7f02580c27d0 2023-10-24T23:58:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f02577fe640 1 --2- 172.21.15.107:0/2357960819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02580c27d0 0x7f02580c2bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f02480097c0 tx=0x7f024801c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7625614c479a070c server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f02567fc640 1 -- 172.21.15.107:0/2357960819 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02480ab600 con 0x7f02580c27d0 2023-10-24T23:58:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f02567fc640 1 -- 172.21.15.107:0/2357960819 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02480a7050 con 0x7f02580c27d0 2023-10-24T23:58:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f02567fc640 1 -- 172.21.15.107:0/2357960819 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02480ab960 con 0x7f02580c27d0 2023-10-24T23:58:15.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f025cb1f640 1 -- 172.21.15.107:0/2357960819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02580c27d0 msgr2=0x7f02580c2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:15.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f025cb1f640 1 --2- 172.21.15.107:0/2357960819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02580c27d0 0x7f02580c2bb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f02480097c0 tx=0x7f024801c3f0 comp rx=0 tx=0).stop 2023-10-24T23:58:15.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f025cb1f640 1 -- 172.21.15.107:0/2357960819 shutdown_connections 2023-10-24T23:58:15.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f025cb1f640 1 --2- 172.21.15.107:0/2357960819 >> v2:172.21.15.110:3300/0 conn(0x7f02580724b0 0x7f02580c1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:15.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f025cb1f640 1 --2- 172.21.15.107:0/2357960819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02580c27d0 0x7f02580c2bb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:15.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f025cb1f640 1 -- 172.21.15.107:0/2357960819 >> 172.21.15.107:0/2357960819 conn(0x7f02580c6cc0 msgr2=0x7f02580c70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:15.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f025cb1f640 1 -- 172.21.15.107:0/2357960819 shutdown_connections 2023-10-24T23:58:15.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.601+0000 7f025cb1f640 1 -- 172.21.15.107:0/2357960819 wait complete. 2023-10-24T23:58:15.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f025cb1f640 1 Processor -- start 2023-10-24T23:58:15.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f025cb1f640 1 -- start start 2023-10-24T23:58:15.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f025cb1f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0258072030 0x7f0258151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:15.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f025cb1f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02580724b0 0x7f02580c0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:15.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f025cb1f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02580c27d0 0x7f025814c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:15.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f025cb1f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02581efe80 con 0x7f0258072030 2023-10-24T23:58:15.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f025cb1f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02581f0a00 con 0x7f02580724b0 2023-10-24T23:58:15.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f025cb1f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02580aa330 con 0x7f02580c27d0 2023-10-24T23:58:15.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f02577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0258072030 0x7f0258151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:15.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f02577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0258072030 0x7f0258151060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40674/0 (socket says 172.21.15.107:40674) 2023-10-24T23:58:15.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f02577fe640 1 -- 172.21.15.107:0/63736443 learned_addr learned my addr 172.21.15.107:0/63736443 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:15.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f0257fff640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02580c27d0 0x7f025814c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:15.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f0256ffd640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02580724b0 0x7f02580c0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:15.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f02577fe640 1 -- 172.21.15.107:0/63736443 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02580c27d0 msgr2=0x7f025814c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:15.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f02577fe640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02580c27d0 0x7f025814c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:15.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f02577fe640 1 -- 172.21.15.107:0/63736443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02580724b0 msgr2=0x7f02580c0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:15.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f02577fe640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02580724b0 0x7f02580c0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:15.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f02577fe640 1 -- 172.21.15.107:0/63736443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0248009420 con 0x7f0258072030 2023-10-24T23:58:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f02577fe640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0258072030 0x7f0258151060 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f024801a020 tx=0x7f0248019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f023ffff640 1 -- 172.21.15.107:0/63736443 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02480038b0 con 0x7f0258072030 2023-10-24T23:58:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f023ffff640 1 -- 172.21.15.107:0/63736443 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0248003a10 con 0x7f0258072030 2023-10-24T23:58:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f025cb1f640 1 -- 172.21.15.107:0/63736443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02581f00d0 con 0x7f0258072030 2023-10-24T23:58:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f023ffff640 1 -- 172.21.15.107:0/63736443 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0248003d80 con 0x7f0258072030 2023-10-24T23:58:15.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.605+0000 7f025cb1f640 1 -- 172.21.15.107:0/63736443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02581f0690 con 0x7f0258072030 2023-10-24T23:58:15.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.609+0000 7f025cb1f640 1 -- 172.21.15.107:0/63736443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f025804e050 con 0x7f0258072030 2023-10-24T23:58:15.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.609+0000 7f023ffff640 1 -- 172.21.15.107:0/63736443 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f02480b6420 con 0x7f0258072030 2023-10-24T23:58:15.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.609+0000 7f023ffff640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0244065e60 0x7f0244068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:15.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.609+0000 7f023ffff640 1 -- 172.21.15.107:0/63736443 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f024811e290 con 0x7f0258072030 2023-10-24T23:58:15.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.609+0000 7f0256ffd640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0244065e60 0x7f0244068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:15.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.609+0000 7f0256ffd640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0244065e60 0x7f0244068320 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f02580c7ba0 tx=0x7f024c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:15.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.617+0000 7f023ffff640 1 -- 172.21.15.107:0/63736443 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f02480eb330 con 0x7f0258072030 2023-10-24T23:58:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.849+0000 7f025cb1f640 1 -- 172.21.15.107:0/63736443 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0258154230 con 0x7f0258072030 2023-10-24T23:58:15.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.849+0000 7f023ffff640 1 -- 172.21.15.107:0/63736443 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f02480ee950 con 0x7f0258072030 2023-10-24T23:58:15.856 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:15.856 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:15.856 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 -- 172.21.15.107:0/63736443 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0244065e60 msgr2=0x7f0244068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0244065e60 0x7f0244068320 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f02580c7ba0 tx=0x7f024c009040 comp rx=0 tx=0).stop 2023-10-24T23:58:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 -- 172.21.15.107:0/63736443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0258072030 msgr2=0x7f0258151060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0258072030 0x7f0258151060 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f024801a020 tx=0x7f0248019190 comp rx=0 tx=0).stop 2023-10-24T23:58:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 -- 172.21.15.107:0/63736443 shutdown_connections 2023-10-24T23:58:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0244065e60 0x7f0244068320 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02580c27d0 0x7f025814c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02580724b0 0x7f02580c0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 --2- 172.21.15.107:0/63736443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0258072030 0x7f0258151060 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 -- 172.21.15.107:0/63736443 >> 172.21.15.107:0/63736443 conn(0x7f02580c6cc0 msgr2=0x7f025815e330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 -- 172.21.15.107:0/63736443 shutdown_connections 2023-10-24T23:58:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:15.853+0000 7f025cb1f640 1 -- 172.21.15.107:0/63736443 wait complete. 2023-10-24T23:58:15.876 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2023-10-24T23:58:15.877 INFO:tasks.mds_thrash.fs.[a]:waiting for 29 secs before thrashing 2023-10-24T23:58:35.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:58:35.425 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:58:38.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:38.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a9171640 1 Processor -- start 2023-10-24T23:58:38.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a9171640 1 -- start start 2023-10-24T23:58:38.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a9171640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f11a414fc30 0x7f11a4153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a9171640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11a40c18a0 0x7f11a40c1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a9171640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a4072330 con 0x7f11a40c5c20 2023-10-24T23:58:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a9171640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a4155ca0 con 0x7f11a414fc30 2023-10-24T23:58:38.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a9171640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a4155f80 con 0x7f11a40c18a0 2023-10-24T23:58:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a2d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f11a414fc30 0x7f11a4153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a2d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f11a414fc30 0x7f11a4153190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53788/0 (socket says 172.21.15.107:53788) 2023-10-24T23:58:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a2d9d640 1 -- 172.21.15.107:0/2427399724 learned_addr learned my addr 172.21.15.107:0/2427399724 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a359e640 1 --2- 172.21.15.107:0/2427399724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11a40c18a0 0x7f11a40c1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a2d9d640 1 -- 172.21.15.107:0/2427399724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11a40c18a0 msgr2=0x7f11a40c1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a2d9d640 1 --2- 172.21.15.107:0/2427399724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11a40c18a0 0x7f11a40c1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a2d9d640 1 -- 172.21.15.107:0/2427399724 >> v1:172.21.15.110:6789/0 conn(0x7f11a40c5c20 legacy=0x7f11a406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:38.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.809+0000 7f11a2d9d640 1 -- 172.21.15.107:0/2427399724 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a4156260 con 0x7f11a414fc30 2023-10-24T23:58:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a2d9d640 1 --2- 172.21.15.107:0/2427399724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11a414fc30 0x7f11a4153190 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1194009870 tx=0x7f119401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd293c5967e6f0c server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a1d9b640 1 -- 172.21.15.107:0/2427399724 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11940ab600 con 0x7f11a414fc30 2023-10-24T23:58:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a1d9b640 1 -- 172.21.15.107:0/2427399724 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11940ab760 con 0x7f11a414fc30 2023-10-24T23:58:38.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a1d9b640 1 -- 172.21.15.107:0/2427399724 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11940abaf0 con 0x7f11a414fc30 2023-10-24T23:58:38.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 -- 172.21.15.107:0/2427399724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11a414fc30 msgr2=0x7f11a4153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:38.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 --2- 172.21.15.107:0/2427399724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11a414fc30 0x7f11a4153190 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1194009870 tx=0x7f119401c440 comp rx=0 tx=0).stop 2023-10-24T23:58:38.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 -- 172.21.15.107:0/2427399724 shutdown_connections 2023-10-24T23:58:38.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 --2- 172.21.15.107:0/2427399724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11a40c18a0 0x7f11a40c1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 --2- 172.21.15.107:0/2427399724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11a414fc30 0x7f11a4153190 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 -- 172.21.15.107:0/2427399724 >> 172.21.15.107:0/2427399724 conn(0x7f11a40c39a0 msgr2=0x7f11a40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 -- 172.21.15.107:0/2427399724 shutdown_connections 2023-10-24T23:58:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 -- 172.21.15.107:0/2427399724 wait complete. 2023-10-24T23:58:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 Processor -- start 2023-10-24T23:58:38.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 -- start start 2023-10-24T23:58:38.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11a40c18a0 0x7f11a41eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:38.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11a40c5c20 0x7f11a4154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11a414fc30 0x7f11a413a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a41508b0 con 0x7f11a40c18a0 2023-10-24T23:58:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a41ebdf0 con 0x7f11a414fc30 2023-10-24T23:58:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a9171640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a4139fd0 con 0x7f11a40c5c20 2023-10-24T23:58:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11a40c18a0 0x7f11a41eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11a40c18a0 0x7f11a41eb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53804/0 (socket says 172.21.15.107:53804) 2023-10-24T23:58:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a2d9d640 1 -- 172.21.15.107:0/4232003547 learned_addr learned my addr 172.21.15.107:0/4232003547 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a359e640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11a414fc30 0x7f11a413a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a259c640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11a40c5c20 0x7f11a4154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a2d9d640 1 -- 172.21.15.107:0/4232003547 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11a40c5c20 msgr2=0x7f11a4154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a2d9d640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11a40c5c20 0x7f11a4154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a2d9d640 1 -- 172.21.15.107:0/4232003547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11a414fc30 msgr2=0x7f11a413a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a2d9d640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11a414fc30 0x7f11a413a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.813+0000 7f11a2d9d640 1 -- 172.21.15.107:0/4232003547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11940094d0 con 0x7f11a40c18a0 2023-10-24T23:58:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f11a2d9d640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11a40c18a0 0x7f11a41eb8b0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f11940ad040 tx=0x7f119401ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:38.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f118b7fe640 1 -- 172.21.15.107:0/4232003547 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11940b4800 con 0x7f11a40c18a0 2023-10-24T23:58:38.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f118b7fe640 1 -- 172.21.15.107:0/4232003547 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11940b4980 con 0x7f11a40c18a0 2023-10-24T23:58:38.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f11a9171640 1 -- 172.21.15.107:0/4232003547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a4153bd0 con 0x7f11a40c18a0 2023-10-24T23:58:38.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f118b7fe640 1 -- 172.21.15.107:0/4232003547 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11940b4d30 con 0x7f11a40c18a0 2023-10-24T23:58:38.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f11a9171640 1 -- 172.21.15.107:0/4232003547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a4139cf0 con 0x7f11a40c18a0 2023-10-24T23:58:38.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f11a9171640 1 -- 172.21.15.107:0/4232003547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a40746e0 con 0x7f11a40c18a0 2023-10-24T23:58:38.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f118b7fe640 1 -- 172.21.15.107:0/4232003547 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f11940c7440 con 0x7f11a40c18a0 2023-10-24T23:58:38.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f118b7fe640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1190065e60 0x7f1190068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f118b7fe640 1 -- 172.21.15.107:0/4232003547 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f119411e970 con 0x7f11a40c18a0 2023-10-24T23:58:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f11a259c640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1190065e60 0x7f1190068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.817+0000 7f11a259c640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1190065e60 0x7f1190068320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f11a4154560 tx=0x7f1198009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:38.825+0000 7f118b7fe640 1 -- 172.21.15.107:0/4232003547 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f11940ebac0 con 0x7f11a40c18a0 2023-10-24T23:58:39.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.113+0000 7f11a9171640 1 -- 172.21.15.107:0/4232003547 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11a414f440 con 0x7f11a40c18a0 2023-10-24T23:58:39.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f118b7fe640 1 -- 172.21.15.107:0/4232003547 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f11940ef0e0 con 0x7f11a40c18a0 2023-10-24T23:58:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:39.121 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:39.121 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:39.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 -- 172.21.15.107:0/4232003547 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1190065e60 msgr2=0x7f1190068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:39.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1190065e60 0x7f1190068320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f11a4154560 tx=0x7f1198009040 comp rx=0 tx=0).stop 2023-10-24T23:58:39.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 -- 172.21.15.107:0/4232003547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11a40c18a0 msgr2=0x7f11a41eb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:39.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11a40c18a0 0x7f11a41eb8b0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f11940ad040 tx=0x7f119401ac80 comp rx=0 tx=0).stop 2023-10-24T23:58:39.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 -- 172.21.15.107:0/4232003547 shutdown_connections 2023-10-24T23:58:39.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11a414fc30 0x7f11a413a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1190065e60 0x7f1190068320 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11a40c5c20 0x7f11a4154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 --2- 172.21.15.107:0/4232003547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11a40c18a0 0x7f11a41eb8b0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 -- 172.21.15.107:0/4232003547 >> 172.21.15.107:0/4232003547 conn(0x7f11a40c39a0 msgr2=0x7f11a40c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:39.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 -- 172.21.15.107:0/4232003547 shutdown_connections 2023-10-24T23:58:39.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.117+0000 7f11a9171640 1 -- 172.21.15.107:0/4232003547 wait complete. 2023-10-24T23:58:39.145 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2023-10-24T23:58:39.145 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-24T23:58:39.146 INFO:teuthology.orchestra.run:waiting for 300 2023-10-24T23:58:39.167 INFO:tasks.ceph.mds.d:Stopped 2023-10-24T23:58:39.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:39.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb83a58d640 1 Processor -- start 2023-10-24T23:58:39.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb83a58d640 1 -- start start 2023-10-24T23:58:39.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb83a58d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb834071ce0 0x7fb8340ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:39.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb83a58d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb8340720c0 0x7fb83413d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:39.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb83a58d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb83406e710 con 0x7fb834154860 2023-10-24T23:58:39.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb83a58d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb83406d900 con 0x7fb8340720c0 2023-10-24T23:58:39.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb83a58d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb834154d20 con 0x7fb834071ce0 2023-10-24T23:58:39.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb82bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb8340720c0 0x7fb83413d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:39.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb838b2a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb834154860 0x7fb834158230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:35854/0 (socket says 172.21.15.107:35854) 2023-10-24T23:58:39.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb838b2a640 1 -- 172.21.15.107:0/773413152 learned_addr learned my addr 172.21.15.107:0/773413152 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:39.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb833fff640 1 --2- 172.21.15.107:0/773413152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb834071ce0 0x7fb8340ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:39.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb8337fe640 1 -- 172.21.15.107:0/773413152 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152503722 0 0) 0x7fb83406e710 con 0x7fb834154860 2023-10-24T23:58:39.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb8337fe640 1 -- 172.21.15.107:0/773413152 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb820003530 con 0x7fb834154860 2023-10-24T23:58:39.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb82bfff640 1 -- 172.21.15.107:0/773413152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb834071ce0 msgr2=0x7fb8340ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:39.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb82bfff640 1 --2- 172.21.15.107:0/773413152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb834071ce0 0x7fb8340ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb82bfff640 1 -- 172.21.15.107:0/773413152 >> v1:172.21.15.110:6789/0 conn(0x7fb834154860 legacy=0x7fb834158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:39.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb82bfff640 1 -- 172.21.15.107:0/773413152 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb834155000 con 0x7fb8340720c0 2023-10-24T23:58:39.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.297+0000 7fb833fff640 1 --2- 172.21.15.107:0/773413152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb834071ce0 0x7fb8340ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:58:39.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb82bfff640 1 --2- 172.21.15.107:0/773413152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340720c0 0x7fb83413d970 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb81c0097c0 tx=0x7fb81c01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a87fc4b41699b20f server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:39.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb8337fe640 1 -- 172.21.15.107:0/773413152 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c0ac620 con 0x7fb8340720c0 2023-10-24T23:58:39.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb8337fe640 1 -- 172.21.15.107:0/773413152 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c0ac780 con 0x7fb8340720c0 2023-10-24T23:58:39.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb8337fe640 1 -- 172.21.15.107:0/773413152 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c0acaf0 con 0x7fb8340720c0 2023-10-24T23:58:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb83a58d640 1 -- 172.21.15.107:0/773413152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340720c0 msgr2=0x7fb83413d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb83a58d640 1 --2- 172.21.15.107:0/773413152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340720c0 0x7fb83413d970 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb81c0097c0 tx=0x7fb81c01c3c0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb83a58d640 1 -- 172.21.15.107:0/773413152 shutdown_connections 2023-10-24T23:58:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb83a58d640 1 --2- 172.21.15.107:0/773413152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340720c0 0x7fb83413d970 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb83a58d640 1 --2- 172.21.15.107:0/773413152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb834071ce0 0x7fb8340ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb83a58d640 1 -- 172.21.15.107:0/773413152 >> 172.21.15.107:0/773413152 conn(0x7fb834070f60 msgr2=0x7fb834071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb83a58d640 1 -- 172.21.15.107:0/773413152 shutdown_connections 2023-10-24T23:58:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb83a58d640 1 -- 172.21.15.107:0/773413152 wait complete. 2023-10-24T23:58:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb83a58d640 1 Processor -- start 2023-10-24T23:58:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.301+0000 7fb83a58d640 1 -- start start 2023-10-24T23:58:39.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb83a58d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834071ce0 0x7fb834134520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:39.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb83a58d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340720c0 0x7fb834138580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:39.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb83a58d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb834154860 0x7fb8341513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:39.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb83a58d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb834150f50 con 0x7fb834071ce0 2023-10-24T23:58:39.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb83a58d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb834155d40 con 0x7fb8340720c0 2023-10-24T23:58:39.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb83a58d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb834150340 con 0x7fb834154860 2023-10-24T23:58:39.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb833fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834071ce0 0x7fb834134520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:39.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb833fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834071ce0 0x7fb834134520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53820/0 (socket says 172.21.15.107:53820) 2023-10-24T23:58:39.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb833fff640 1 -- 172.21.15.107:0/2327801457 learned_addr learned my addr 172.21.15.107:0/2327801457 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:39.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb82bfff640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340720c0 0x7fb834138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:39.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb838b2a640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb834154860 0x7fb8341513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:39.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb833fff640 1 -- 172.21.15.107:0/2327801457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb834154860 msgr2=0x7fb8341513e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:39.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb833fff640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb834154860 0x7fb8341513e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb833fff640 1 -- 172.21.15.107:0/2327801457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340720c0 msgr2=0x7fb834138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:39.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb833fff640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340720c0 0x7fb834138580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb833fff640 1 -- 172.21.15.107:0/2327801457 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb824009420 con 0x7fb834071ce0 2023-10-24T23:58:39.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb833fff640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834071ce0 0x7fb834134520 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb82400ebe0 tx=0x7fb82400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:39.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb8317fa640 1 -- 172.21.15.107:0/2327801457 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82400eee0 con 0x7fb834071ce0 2023-10-24T23:58:39.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb8317fa640 1 -- 172.21.15.107:0/2327801457 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82400cc30 con 0x7fb834071ce0 2023-10-24T23:58:39.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb8317fa640 1 -- 172.21.15.107:0/2327801457 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824013470 con 0x7fb834071ce0 2023-10-24T23:58:39.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb83a58d640 1 -- 172.21.15.107:0/2327801457 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81c009420 con 0x7fb834071ce0 2023-10-24T23:58:39.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb83a58d640 1 -- 172.21.15.107:0/2327801457 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb834150900 con 0x7fb834071ce0 2023-10-24T23:58:39.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb83a58d640 1 -- 172.21.15.107:0/2327801457 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb834157e90 con 0x7fb834071ce0 2023-10-24T23:58:39.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb8317fa640 1 -- 172.21.15.107:0/2327801457 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb8240135d0 con 0x7fb834071ce0 2023-10-24T23:58:39.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb8317fa640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 0x7fb814068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:39.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb8317fa640 1 -- 172.21.15.107:0/2327801457 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7fb824084170 con 0x7fb834071ce0 2023-10-24T23:58:39.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.305+0000 7fb82bfff640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 0x7fb814068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:39.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.309+0000 7fb82bfff640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 0x7fb814068320 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb81c0ae000 tx=0x7fb81c0a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:39.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.313+0000 7fb8317fa640 1 -- 172.21.15.107:0/2327801457 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb824051180 con 0x7fb834071ce0 2023-10-24T23:58:39.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.553+0000 7fb83a58d640 1 -- 172.21.15.107:0/2327801457 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8340746e0 con 0x7fb834071ce0 2023-10-24T23:58:39.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.553+0000 7fb8317fa640 1 -- 172.21.15.107:0/2327801457 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7fb8240547a0 con 0x7fb834071ce0 2023-10-24T23:58:39.561 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:39.561 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:39.561 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:39.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 -- 172.21.15.107:0/2327801457 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 msgr2=0x7fb814068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:39.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 0x7fb814068320 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb81c0ae000 tx=0x7fb81c0a7000 comp rx=0 tx=0).stop 2023-10-24T23:58:39.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 -- 172.21.15.107:0/2327801457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834071ce0 msgr2=0x7fb834134520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:39.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834071ce0 0x7fb834134520 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb82400ebe0 tx=0x7fb82400c520 comp rx=0 tx=0).stop 2023-10-24T23:58:39.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 -- 172.21.15.107:0/2327801457 shutdown_connections 2023-10-24T23:58:39.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 0x7fb814068320 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb834154860 0x7fb8341513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340720c0 0x7fb834138580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 --2- 172.21.15.107:0/2327801457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834071ce0 0x7fb834134520 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:39.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 -- 172.21.15.107:0/2327801457 >> 172.21.15.107:0/2327801457 conn(0x7fb834070f60 msgr2=0x7fb8340c4260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:39.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 -- 172.21.15.107:0/2327801457 shutdown_connections 2023-10-24T23:58:39.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:39.557+0000 7fb83a58d640 1 -- 172.21.15.107:0/2327801457 wait complete. 2023-10-24T23:58:39.582 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:58:41.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:41.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443fb33640 1 Processor -- start 2023-10-24T23:58:41.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443fb33640 1 -- start start 2023-10-24T23:58:41.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443fb33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4438156770 0x7f44380ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443fb33640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f44380c70e0 0x7f44380bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443fb33640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44380c1ca0 con 0x7f44380c2130 2023-10-24T23:58:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443fb33640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44380c27e0 con 0x7f4438156770 2023-10-24T23:58:41.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443fb33640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44380c10c0 con 0x7f44380c70e0 2023-10-24T23:58:41.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443d8cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4438156770 0x7f44380ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:41.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443d8cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4438156770 0x7f44380ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53834/0 (socket says 172.21.15.107:53834) 2023-10-24T23:58:41.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443e0d0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f44380c2130 0x7f4438069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47282/0 (socket says 172.21.15.107:47282) 2023-10-24T23:58:41.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443d8cf640 1 -- 172.21.15.107:0/2648964905 learned_addr learned my addr 172.21.15.107:0/2648964905 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:41.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443d0ce640 1 --2- 172.21.15.107:0/2648964905 >> v2:172.21.15.110:3300/0 conn(0x7f44380c70e0 0x7f44380bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:41.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443d8cf640 1 -- 172.21.15.107:0/2648964905 >> v2:172.21.15.110:3300/0 conn(0x7f44380c70e0 msgr2=0x7f44380bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:41.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443d8cf640 1 --2- 172.21.15.107:0/2648964905 >> v2:172.21.15.110:3300/0 conn(0x7f44380c70e0 0x7f44380bdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:41.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443d8cf640 1 -- 172.21.15.107:0/2648964905 >> v1:172.21.15.107:6789/0 conn(0x7f44380c2130 legacy=0x7f4438069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:41.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.729+0000 7f443d8cf640 1 -- 172.21.15.107:0/2648964905 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44380c13a0 con 0x7f4438156770 2023-10-24T23:58:41.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443d8cf640 1 --2- 172.21.15.107:0/2648964905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4438156770 0x7f44380ab7c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f44380a8180 tx=0x7f443401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442f92778db4c995 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:41.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443c8cd640 1 -- 172.21.15.107:0/2648964905 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44340ab600 con 0x7f4438156770 2023-10-24T23:58:41.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443c8cd640 1 -- 172.21.15.107:0/2648964905 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44340a7050 con 0x7f4438156770 2023-10-24T23:58:41.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443c8cd640 1 -- 172.21.15.107:0/2648964905 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44340ab980 con 0x7f4438156770 2023-10-24T23:58:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443fb33640 1 -- 172.21.15.107:0/2648964905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4438156770 msgr2=0x7f44380ab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443fb33640 1 --2- 172.21.15.107:0/2648964905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4438156770 0x7f44380ab7c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f44380a8180 tx=0x7f443401c3b0 comp rx=0 tx=0).stop 2023-10-24T23:58:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443fb33640 1 -- 172.21.15.107:0/2648964905 shutdown_connections 2023-10-24T23:58:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443fb33640 1 --2- 172.21.15.107:0/2648964905 >> v2:172.21.15.110:3300/0 conn(0x7f44380c70e0 0x7f44380bdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:41.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443fb33640 1 --2- 172.21.15.107:0/2648964905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4438156770 0x7f44380ab7c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443fb33640 1 -- 172.21.15.107:0/2648964905 >> 172.21.15.107:0/2648964905 conn(0x7f44380c65c0 msgr2=0x7f44380c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443fb33640 1 -- 172.21.15.107:0/2648964905 shutdown_connections 2023-10-24T23:58:41.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443fb33640 1 -- 172.21.15.107:0/2648964905 wait complete. 2023-10-24T23:58:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443fb33640 1 Processor -- start 2023-10-24T23:58:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.733+0000 7f443fb33640 1 -- start start 2023-10-24T23:58:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443fb33640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44380c2130 0x7f44380a4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:41.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443fb33640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f44380c70e0 0x7f44381f1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:41.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443fb33640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4438156770 0x7f44381f4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:41.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443fb33640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44381f1fc0 con 0x7f44380c2130 2023-10-24T23:58:41.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443fb33640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44380a3940 con 0x7f4438156770 2023-10-24T23:58:41.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443fb33640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44380a41c0 con 0x7f44380c70e0 2023-10-24T23:58:41.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443d8cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44380c2130 0x7f44380a4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:41.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443d8cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44380c2130 0x7f44380a4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53842/0 (socket says 172.21.15.107:53842) 2023-10-24T23:58:41.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443d8cf640 1 -- 172.21.15.107:0/2082645453 learned_addr learned my addr 172.21.15.107:0/2082645453 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:41.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443e0d0640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4438156770 0x7f44381f4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:41.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443d0ce640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f44380c70e0 0x7f44381f1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:41.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443d8cf640 1 -- 172.21.15.107:0/2082645453 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f44380c70e0 msgr2=0x7f44381f1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:41.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443d8cf640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f44380c70e0 0x7f44381f1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:41.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443d8cf640 1 -- 172.21.15.107:0/2082645453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4438156770 msgr2=0x7f44381f4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:41.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443d8cf640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4438156770 0x7f44381f4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:41.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443d8cf640 1 -- 172.21.15.107:0/2082645453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44340094d0 con 0x7f44380c2130 2023-10-24T23:58:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443d8cf640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44380c2130 0x7f44380a4b40 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f44340ae040 tx=0x7f4434019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f442e7fc640 1 -- 172.21.15.107:0/2082645453 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44340b4640 con 0x7f44380c2130 2023-10-24T23:58:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f442e7fc640 1 -- 172.21.15.107:0/2082645453 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44340b47c0 con 0x7f44380c2130 2023-10-24T23:58:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443fb33640 1 -- 172.21.15.107:0/2082645453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44380a3c00 con 0x7f44380c2130 2023-10-24T23:58:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f442e7fc640 1 -- 172.21.15.107:0/2082645453 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44340b4b70 con 0x7f44380c2130 2023-10-24T23:58:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443fb33640 1 -- 172.21.15.107:0/2082645453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44381f4670 con 0x7f44380c2130 2023-10-24T23:58:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443fb33640 1 -- 172.21.15.107:0/2082645453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44380746e0 con 0x7f44380c2130 2023-10-24T23:58:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f442e7fc640 1 -- 172.21.15.107:0/2082645453 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f44340b2070 con 0x7f44380c2130 2023-10-24T23:58:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f442e7fc640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f441c065e60 0x7f441c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:41.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f442e7fc640 1 -- 172.21.15.107:0/2082645453 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f443411dd50 con 0x7f44380c2130 2023-10-24T23:58:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.737+0000 7f443d0ce640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f441c065e60 0x7f441c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.741+0000 7f443d0ce640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f441c065e60 0x7f441c068320 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4438153dd0 tx=0x7f4428008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.745+0000 7f442e7fc640 1 -- 172.21.15.107:0/2082645453 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4434121030 con 0x7f44380c2130 2023-10-24T23:58:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.977+0000 7f443fb33640 1 -- 172.21.15.107:0/2082645453 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4438065a00 con 0x7f44380c2130 2023-10-24T23:58:41.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f442e7fc640 1 -- 172.21.15.107:0/2082645453 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f44340ee880 con 0x7f44380c2130 2023-10-24T23:58:41.986 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:41.986 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:41.987 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 -- 172.21.15.107:0/2082645453 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f441c065e60 msgr2=0x7f441c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:41.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f441c065e60 0x7f441c068320 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4438153dd0 tx=0x7f4428008040 comp rx=0 tx=0).stop 2023-10-24T23:58:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 -- 172.21.15.107:0/2082645453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44380c2130 msgr2=0x7f44380a4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44380c2130 0x7f44380a4b40 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f44340ae040 tx=0x7f4434019150 comp rx=0 tx=0).stop 2023-10-24T23:58:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 -- 172.21.15.107:0/2082645453 shutdown_connections 2023-10-24T23:58:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4438156770 0x7f44381f4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f441c065e60 0x7f441c068320 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:41.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f44380c70e0 0x7f44381f1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 --2- 172.21.15.107:0/2082645453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44380c2130 0x7f44380a4b40 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 -- 172.21.15.107:0/2082645453 >> 172.21.15.107:0/2082645453 conn(0x7f44380c65c0 msgr2=0x7f4438069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 -- 172.21.15.107:0/2082645453 shutdown_connections 2023-10-24T23:58:41.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:41.981+0000 7f443fb33640 1 -- 172.21.15.107:0/2082645453 wait complete. 2023-10-24T23:58:42.004 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:58:44.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:44.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6421274640 1 Processor -- start 2023-10-24T23:58:44.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6421274640 1 -- start start 2023-10-24T23:58:44.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6421274640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f641c069d40 0x7f641c158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:44.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6421274640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f641c06a120 0x7f641c0aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:44.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6421274640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641c06cea0 con 0x7f641c065bc0 2023-10-24T23:58:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6421274640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f641c0ac650 con 0x7f641c06a120 2023-10-24T23:58:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6421274640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641c0a9e00 con 0x7f641c069d40 2023-10-24T23:58:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f641a7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f641c065bc0 0x7f641c0a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40442/0 (socket says 172.21.15.107:40442) 2023-10-24T23:58:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f641a7fc640 1 -- 172.21.15.107:0/1216640457 learned_addr learned my addr 172.21.15.107:0/1216640457 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:44.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f641b7fe640 1 --2- 172.21.15.107:0/1216640457 >> v2:172.21.15.110:3300/0 conn(0x7f641c06a120 0x7f641c0aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966160494 0 0) 0x7f641c06cea0 con 0x7f641c065bc0 2023-10-24T23:58:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6404003530 con 0x7f641c065bc0 2023-10-24T23:58:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f641affd640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f641c069d40 0x7f641c158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3567951553 0 0) 0x7f6404003530 con 0x7f641c065bc0 2023-10-24T23:58:44.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f641c069d40 msgr2=0x7f641c158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f641c069d40 0x7f641c158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 >> v2:172.21.15.110:3300/0 conn(0x7f641c06a120 msgr2=0x7f641c0aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 --2- 172.21.15.107:0/1216640457 >> v2:172.21.15.110:3300/0 conn(0x7f641c06a120 0x7f641c0aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c1584e0 con 0x7f641c065bc0 2023-10-24T23:58:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6410001680 con 0x7f641c065bc0 2023-10-24T23:58:44.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 >> v1:172.21.15.107:6789/0 conn(0x7f641c065bc0 legacy=0x7f641c0a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:44.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f641c069d40 0x7f6404003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:44.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6404003f00 0x7f64040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:44.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6404004d10 0x7f6404005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:44.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640400ba40 con 0x7f6404003f00 2023-10-24T23:58:44.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640400bb80 con 0x7f6404004d10 2023-10-24T23:58:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640400bd10 con 0x7f641c069d40 2023-10-24T23:58:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f641a7fc640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6404003f00 0x7f64040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.141+0000 7f641b7fe640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6404004d10 0x7f6404005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f641affd640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f641c069d40 0x7f6404003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f641a7fc640 1 -- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f641c069d40 msgr2=0x7f6404003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f641a7fc640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f641c069d40 0x7f6404003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f641a7fc640 1 -- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6404004d10 msgr2=0x7f6404005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:44.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f641a7fc640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6404004d10 0x7f6404005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:44.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f641a7fc640 1 -- 172.21.15.107:0/1216640457 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6404003530 con 0x7f6404003f00 2023-10-24T23:58:44.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f641affd640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f641c069d40 0x7f6404003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:44.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f641a7fc640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6404003f00 0x7f64040047d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f641000c1c0 tx=0x7f641001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb2fee32a2c2c789 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:44.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64100058e0 con 0x7f6404003f00 2023-10-24T23:58:44.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6410005a60 con 0x7f6404003f00 2023-10-24T23:58:44.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f6419ffb640 1 -- 172.21.15.107:0/1216640457 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410005e10 con 0x7f6404003f00 2023-10-24T23:58:44.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f6421274640 1 -- 172.21.15.107:0/1216640457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6404003f00 msgr2=0x7f64040047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:44.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f6421274640 1 --2- 172.21.15.107:0/1216640457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6404003f00 0x7f64040047d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f641000c1c0 tx=0x7f641001cc00 comp rx=0 tx=0).stop 2023-10-24T23:58:44.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f641b7fe640 1 -- 172.21.15.107:0/1216640457 reap_dead start 2023-10-24T23:58:44.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f6421274640 1 -- 172.21.15.107:0/1216640457 shutdown_connections 2023-10-24T23:58:44.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f6421274640 1 -- 172.21.15.107:0/1216640457 >> 172.21.15.107:0/1216640457 conn(0x7f641c0c65c0 msgr2=0x7f641c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:44.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f6421274640 1 -- 172.21.15.107:0/1216640457 shutdown_connections 2023-10-24T23:58:44.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.145+0000 7f6421274640 1 -- 172.21.15.107:0/1216640457 wait complete. 2023-10-24T23:58:44.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f6421274640 1 Processor -- start 2023-10-24T23:58:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f6421274640 1 -- start start 2023-10-24T23:58:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f6421274640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6404003f00 0x7f641c1f0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f6421274640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f641c152660 0x7f641c14e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:44.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f6421274640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f641c155460 0x7f641c14eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:44.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f6421274640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c1f1530 con 0x7f641c152660 2023-10-24T23:58:44.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f6421274640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641c154980 con 0x7f641c155460 2023-10-24T23:58:44.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f6421274640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641c154c60 con 0x7f6404003f00 2023-10-24T23:58:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f641c152660 0x7f641c14e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f641c152660 0x7f641c14e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44618/0 (socket says 172.21.15.107:44618) 2023-10-24T23:58:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641a7fc640 1 -- 172.21.15.107:0/467636510 learned_addr learned my addr 172.21.15.107:0/467636510 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641affd640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6404003f00 0x7f641c1f0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641b7fe640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f641c155460 0x7f641c14eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641a7fc640 1 -- 172.21.15.107:0/467636510 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6404003f00 msgr2=0x7f641c1f0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641a7fc640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6404003f00 0x7f641c1f0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641a7fc640 1 -- 172.21.15.107:0/467636510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f641c155460 msgr2=0x7f641c14eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641a7fc640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f641c155460 0x7f641c14eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641a7fc640 1 -- 172.21.15.107:0/467636510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c154f40 con 0x7f641c152660 2023-10-24T23:58:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641b7fe640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f641c155460 0x7f641c14eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f641a7fc640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f641c152660 0x7f641c14e620 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f64100ab040 tx=0x7f6404005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:44.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f64037fe640 1 -- 172.21.15.107:0/467636510 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64100047f0 con 0x7f641c152660 2023-10-24T23:58:44.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f64037fe640 1 -- 172.21.15.107:0/467636510 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6410004950 con 0x7f641c152660 2023-10-24T23:58:44.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f64037fe640 1 -- 172.21.15.107:0/467636510 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410004ce0 con 0x7f641c152660 2023-10-24T23:58:44.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f6421274640 1 -- 172.21.15.107:0/467636510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641c0be460 con 0x7f641c152660 2023-10-24T23:58:44.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f6421274640 1 -- 172.21.15.107:0/467636510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641c1f11b0 con 0x7f641c152660 2023-10-24T23:58:44.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.149+0000 7f6421274640 1 -- 172.21.15.107:0/467636510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641c1550d0 con 0x7f641c152660 2023-10-24T23:58:44.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.153+0000 7f64037fe640 1 -- 172.21.15.107:0/467636510 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f64100a56e0 con 0x7f641c152660 2023-10-24T23:58:44.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.153+0000 7f64037fe640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6408065e60 0x7f6408068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:44.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.153+0000 7f64037fe640 1 -- 172.21.15.107:0/467636510 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f641011d140 con 0x7f641c152660 2023-10-24T23:58:44.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.153+0000 7f641affd640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6408065e60 0x7f6408068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:44.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.153+0000 7f641affd640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6408065e60 0x7f6408068320 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f640c00d800 tx=0x7f640c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:44.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.157+0000 7f64037fe640 1 -- 172.21.15.107:0/467636510 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f64100ea150 con 0x7f641c152660 2023-10-24T23:58:44.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.401+0000 7f6421274640 1 -- 172.21.15.107:0/467636510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f641c15c6e0 con 0x7f641c152660 2023-10-24T23:58:44.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f64037fe640 1 -- 172.21.15.107:0/467636510 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f64100ed770 con 0x7f641c152660 2023-10-24T23:58:44.409 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:44.409 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:44.410 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:44.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 -- 172.21.15.107:0/467636510 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6408065e60 msgr2=0x7f6408068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:44.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6408065e60 0x7f6408068320 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f640c00d800 tx=0x7f640c008040 comp rx=0 tx=0).stop 2023-10-24T23:58:44.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 -- 172.21.15.107:0/467636510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f641c152660 msgr2=0x7f641c14e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:44.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f641c152660 0x7f641c14e620 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f64100ab040 tx=0x7f6404005800 comp rx=0 tx=0).stop 2023-10-24T23:58:44.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 -- 172.21.15.107:0/467636510 shutdown_connections 2023-10-24T23:58:44.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f641c155460 0x7f641c14eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:44.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f641c152660 0x7f641c14e620 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:44.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6408065e60 0x7f6408068320 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:44.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 --2- 172.21.15.107:0/467636510 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6404003f00 0x7f641c1f0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:44.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 -- 172.21.15.107:0/467636510 >> 172.21.15.107:0/467636510 conn(0x7f641c0c65c0 msgr2=0x7f641c0c6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:44.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 -- 172.21.15.107:0/467636510 shutdown_connections 2023-10-24T23:58:44.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:44.405+0000 7f6421274640 1 -- 172.21.15.107:0/467636510 wait complete. 2023-10-24T23:58:44.429 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:58:44.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:45.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.001+0000 7f47b614f640 1 Processor -- start 2023-10-24T23:58:45.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.001+0000 7f47b614f640 1 -- start start 2023-10-24T23:58:45.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47b614f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47b01562b0 0x7f47b0158d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47b614f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47b015c340 0x7f47b00c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47b614f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f47b00c50c0 0x7f47b00ac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47b614f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b0159290 con 0x7f47b01562b0 2023-10-24T23:58:45.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47b614f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b0159d40 con 0x7f47b00c50c0 2023-10-24T23:58:45.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47b614f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b00c83a0 con 0x7f47b015c340 2023-10-24T23:58:45.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47af7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47b01562b0 0x7f47b0158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:45.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47af7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47b01562b0 0x7f47b0158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44626/0 (socket says 172.21.15.107:44626) 2023-10-24T23:58:45.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47af7fe640 1 -- 172.21.15.107:0/1838685366 learned_addr learned my addr 172.21.15.107:0/1838685366 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:45.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47af7fe640 1 -- 172.21.15.107:0/1838685366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47b015c340 msgr2=0x7f47b00c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:45.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47af7fe640 1 --2- 172.21.15.107:0/1838685366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47b015c340 0x7f47b00c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47af7fe640 1 -- 172.21.15.107:0/1838685366 >> v2:172.21.15.110:3300/0 conn(0x7f47b00c50c0 msgr2=0x7f47b00ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:45.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47af7fe640 1 --2- 172.21.15.107:0/1838685366 >> v2:172.21.15.110:3300/0 conn(0x7f47b00c50c0 0x7f47b00ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47af7fe640 1 -- 172.21.15.107:0/1838685366 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b01594a0 con 0x7f47b01562b0 2023-10-24T23:58:45.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47af7fe640 1 --2- 172.21.15.107:0/1838685366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47b01562b0 0x7f47b0158d50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f47b00a9970 tx=0x7f47a001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792a7ac8c31e1da2 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:45.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47ae7fc640 1 -- 172.21.15.107:0/1838685366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a00ab620 con 0x7f47b01562b0 2023-10-24T23:58:45.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47ae7fc640 1 -- 172.21.15.107:0/1838685366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a00ab780 con 0x7f47b01562b0 2023-10-24T23:58:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.005+0000 7f47ae7fc640 1 -- 172.21.15.107:0/1838685366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a00abb10 con 0x7f47b01562b0 2023-10-24T23:58:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 -- 172.21.15.107:0/1838685366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47b01562b0 msgr2=0x7f47b0158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 --2- 172.21.15.107:0/1838685366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47b01562b0 0x7f47b0158d50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f47b00a9970 tx=0x7f47a001c440 comp rx=0 tx=0).stop 2023-10-24T23:58:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 -- 172.21.15.107:0/1838685366 shutdown_connections 2023-10-24T23:58:45.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 --2- 172.21.15.107:0/1838685366 >> v2:172.21.15.110:3300/0 conn(0x7f47b00c50c0 0x7f47b00ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 --2- 172.21.15.107:0/1838685366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47b015c340 0x7f47b00c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 --2- 172.21.15.107:0/1838685366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47b01562b0 0x7f47b0158d50 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 -- 172.21.15.107:0/1838685366 >> 172.21.15.107:0/1838685366 conn(0x7f47b00ca200 msgr2=0x7f47b00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 -- 172.21.15.107:0/1838685366 shutdown_connections 2023-10-24T23:58:45.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 -- 172.21.15.107:0/1838685366 wait complete. 2023-10-24T23:58:45.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 Processor -- start 2023-10-24T23:58:45.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 -- start start 2023-10-24T23:58:45.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47b00c50c0 0x7f47b0151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47b01562b0 0x7f47b014f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47b015c340 0x7f47b014fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0157700 con 0x7f47b015c340 2023-10-24T23:58:45.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b0156c60 con 0x7f47b00c50c0 2023-10-24T23:58:45.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47b614f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b01f21f0 con 0x7f47b01562b0 2023-10-24T23:58:45.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47b015c340 0x7f47b014fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:45.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47b015c340 0x7f47b014fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44628/0 (socket says 172.21.15.107:44628) 2023-10-24T23:58:45.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47affff640 1 -- 172.21.15.107:0/1890885016 learned_addr learned my addr 172.21.15.107:0/1890885016 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:45.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47aeffd640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47b01562b0 0x7f47b014f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:45.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47af7fe640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47b00c50c0 0x7f47b0151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:45.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47affff640 1 -- 172.21.15.107:0/1890885016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47b01562b0 msgr2=0x7f47b014f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:45.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47affff640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47b01562b0 0x7f47b014f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47affff640 1 -- 172.21.15.107:0/1890885016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47b00c50c0 msgr2=0x7f47b0151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47affff640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47b00c50c0 0x7f47b0151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47affff640 1 -- 172.21.15.107:0/1890885016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a00094d0 con 0x7f47b015c340 2023-10-24T23:58:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47af7fe640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47b00c50c0 0x7f47b0151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:58:45.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.009+0000 7f47affff640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47b015c340 0x7f47b014fef0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f47a800b970 tx=0x7f47a800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:45.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.013+0000 7f4797fff640 1 -- 172.21.15.107:0/1890885016 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a800cb40 con 0x7f47b015c340 2023-10-24T23:58:45.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.013+0000 7f4797fff640 1 -- 172.21.15.107:0/1890885016 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a800ccc0 con 0x7f47b015c340 2023-10-24T23:58:45.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.013+0000 7f47b614f640 1 -- 172.21.15.107:0/1890885016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b0156f00 con 0x7f47b015c340 2023-10-24T23:58:45.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.013+0000 7f4797fff640 1 -- 172.21.15.107:0/1890885016 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a8014680 con 0x7f47b015c340 2023-10-24T23:58:45.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.013+0000 7f47b614f640 1 -- 172.21.15.107:0/1890885016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b01f24d0 con 0x7f47b015c340 2023-10-24T23:58:45.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.013+0000 7f47b614f640 1 -- 172.21.15.107:0/1890885016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b0157420 con 0x7f47b015c340 2023-10-24T23:58:45.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.013+0000 7f4797fff640 1 -- 172.21.15.107:0/1890885016 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f47a801e440 con 0x7f47b015c340 2023-10-24T23:58:45.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.013+0000 7f4797fff640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f479c065e60 0x7f479c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.013+0000 7f4797fff640 1 -- 172.21.15.107:0/1890885016 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f47a8083c50 con 0x7f47b015c340 2023-10-24T23:58:45.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.013+0000 7f47af7fe640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f479c065e60 0x7f479c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:45.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.013+0000 7f47af7fe640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f479c065e60 0x7f479c068320 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f47a00af000 tx=0x7f47a00a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:45.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.021+0000 7f4797fff640 1 -- 172.21.15.107:0/1890885016 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f47b0157420 con 0x7f47b015c340 2023-10-24T23:58:45.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 -- 172.21.15.107:0/1890885016 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47b0157420 con 0x7f47b015c340 2023-10-24T23:58:45.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f4797fff640 1 -- 172.21.15.107:0/1890885016 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f47b0157420 con 0x7f47b015c340 2023-10-24T23:58:45.251 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:45.252 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:45.253 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:45.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 -- 172.21.15.107:0/1890885016 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f479c065e60 msgr2=0x7f479c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:45.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f479c065e60 0x7f479c068320 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f47a00af000 tx=0x7f47a00a7000 comp rx=0 tx=0).stop 2023-10-24T23:58:45.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 -- 172.21.15.107:0/1890885016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47b015c340 msgr2=0x7f47b014fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:45.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47b015c340 0x7f47b014fef0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f47a800b970 tx=0x7f47a800be30 comp rx=0 tx=0).stop 2023-10-24T23:58:45.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 -- 172.21.15.107:0/1890885016 shutdown_connections 2023-10-24T23:58:45.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47b015c340 0x7f47b014fef0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f479c065e60 0x7f479c068320 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47b01562b0 0x7f47b014f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 --2- 172.21.15.107:0/1890885016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47b00c50c0 0x7f47b0151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 -- 172.21.15.107:0/1890885016 >> 172.21.15.107:0/1890885016 conn(0x7f47b00ca200 msgr2=0x7f47b00ad9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:45.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 -- 172.21.15.107:0/1890885016 shutdown_connections 2023-10-24T23:58:45.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.245+0000 7f47b614f640 1 -- 172.21.15.107:0/1890885016 wait complete. 2023-10-24T23:58:45.269 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2023-10-24T23:58:45.270 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-24T23:58:45.270 INFO:teuthology.orchestra.run:waiting for 300 2023-10-24T23:58:45.279 INFO:tasks.ceph.mds.c:Stopped 2023-10-24T23:58:45.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:45.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf70621640 1 Processor -- start 2023-10-24T23:58:45.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf70621640 1 -- start start 2023-10-24T23:58:45.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf70621640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdf6814c470 0x7fdf6814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf70621640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf68152b30 0x7fdf68152f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf70621640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf68155cd0 con 0x7fdf6806f490 2023-10-24T23:58:45.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf70621640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf681526e0 con 0x7fdf6814c470 2023-10-24T23:58:45.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf70621640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680c4380 con 0x7fdf68152b30 2023-10-24T23:58:45.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf6e3bd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdf6814c470 0x7fdf6814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:45.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf6ebbe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf68152b30 0x7fdf68152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:45.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf6dbbc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fdf6806f490 0x7fdf68071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:36738/0 (socket says 172.21.15.107:36738) 2023-10-24T23:58:45.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf6dbbc640 1 -- 172.21.15.107:0/946351500 learned_addr learned my addr 172.21.15.107:0/946351500 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:45.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf6ebbe640 1 -- 172.21.15.107:0/946351500 >> v2:172.21.15.110:3300/0 conn(0x7fdf6814c470 msgr2=0x7fdf6814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:45.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf6ebbe640 1 --2- 172.21.15.107:0/946351500 >> v2:172.21.15.110:3300/0 conn(0x7fdf6814c470 0x7fdf6814c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf6ebbe640 1 -- 172.21.15.107:0/946351500 >> v1:172.21.15.110:6789/0 conn(0x7fdf6806f490 legacy=0x7fdf68071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:45.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.445+0000 7fdf6ebbe640 1 -- 172.21.15.107:0/946351500 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf680c48b0 con 0x7fdf68152b30 2023-10-24T23:58:45.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6ebbe640 1 --2- 172.21.15.107:0/946351500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf68152b30 0x7fdf68152f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009d90 tx=0x7fdf6001c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7731a582dc0da360 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:45.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6d3bb640 1 -- 172.21.15.107:0/946351500 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf600a57e0 con 0x7fdf68152b30 2023-10-24T23:58:45.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6d3bb640 1 -- 172.21.15.107:0/946351500 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf600a5940 con 0x7fdf68152b30 2023-10-24T23:58:45.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6d3bb640 1 -- 172.21.15.107:0/946351500 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf600a5cd0 con 0x7fdf68152b30 2023-10-24T23:58:45.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 -- 172.21.15.107:0/946351500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf68152b30 msgr2=0x7fdf68152f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:45.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 --2- 172.21.15.107:0/946351500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf68152b30 0x7fdf68152f80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009d90 tx=0x7fdf6001c830 comp rx=0 tx=0).stop 2023-10-24T23:58:45.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 -- 172.21.15.107:0/946351500 shutdown_connections 2023-10-24T23:58:45.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 --2- 172.21.15.107:0/946351500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf68152b30 0x7fdf68152f80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 --2- 172.21.15.107:0/946351500 >> v2:172.21.15.110:3300/0 conn(0x7fdf6814c470 0x7fdf6814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 -- 172.21.15.107:0/946351500 >> 172.21.15.107:0/946351500 conn(0x7fdf68070f60 msgr2=0x7fdf68071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:45.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 -- 172.21.15.107:0/946351500 shutdown_connections 2023-10-24T23:58:45.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 -- 172.21.15.107:0/946351500 wait complete. 2023-10-24T23:58:45.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 Processor -- start 2023-10-24T23:58:45.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 -- start start 2023-10-24T23:58:45.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf6806f490 0x7fdf681ed100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf6814c470 0x7fdf680c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf68152b30 0x7fdf68141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf681ed640 con 0x7fdf6814c470 2023-10-24T23:58:45.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf681425e0 con 0x7fdf68152b30 2023-10-24T23:58:45.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf70621640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6806c720 con 0x7fdf6806f490 2023-10-24T23:58:45.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6dbbc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf6814c470 0x7fdf680c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:45.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6dbbc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf6814c470 0x7fdf680c1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44644/0 (socket says 172.21.15.107:44644) 2023-10-24T23:58:45.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6dbbc640 1 -- 172.21.15.107:0/1145942502 learned_addr learned my addr 172.21.15.107:0/1145942502 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:45.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6ebbe640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf68152b30 0x7fdf68141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:45.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6e3bd640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf6806f490 0x7fdf681ed100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:45.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6dbbc640 1 -- 172.21.15.107:0/1145942502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf6806f490 msgr2=0x7fdf681ed100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:45.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6dbbc640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf6806f490 0x7fdf681ed100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6dbbc640 1 -- 172.21.15.107:0/1145942502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf68152b30 msgr2=0x7fdf68141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:45.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6dbbc640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf68152b30 0x7fdf68141810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6dbbc640 1 -- 172.21.15.107:0/1145942502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf60009a70 con 0x7fdf6814c470 2023-10-24T23:58:45.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.449+0000 7fdf6dbbc640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf6814c470 0x7fdf680c1570 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800c500 tx=0x7fdf5800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:45.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.453+0000 7fdf5effd640 1 -- 172.21.15.107:0/1145942502 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf58005cf0 con 0x7fdf6814c470 2023-10-24T23:58:45.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.453+0000 7fdf5effd640 1 -- 172.21.15.107:0/1145942502 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf58005e50 con 0x7fdf6814c470 2023-10-24T23:58:45.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.453+0000 7fdf70621640 1 -- 172.21.15.107:0/1145942502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6806ca00 con 0x7fdf6814c470 2023-10-24T23:58:45.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.453+0000 7fdf5effd640 1 -- 172.21.15.107:0/1145942502 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf580072f0 con 0x7fdf6814c470 2023-10-24T23:58:45.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.453+0000 7fdf70621640 1 -- 172.21.15.107:0/1145942502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf68142830 con 0x7fdf6814c470 2023-10-24T23:58:45.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.453+0000 7fdf70621640 1 -- 172.21.15.107:0/1145942502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf680746e0 con 0x7fdf6814c470 2023-10-24T23:58:45.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.453+0000 7fdf5effd640 1 -- 172.21.15.107:0/1145942502 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdf58007450 con 0x7fdf6814c470 2023-10-24T23:58:45.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.453+0000 7fdf5effd640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf4c065e60 0x7fdf4c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:45.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.453+0000 7fdf5effd640 1 -- 172.21.15.107:0/1145942502 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7fdf58083300 con 0x7fdf6814c470 2023-10-24T23:58:45.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.453+0000 7fdf6e3bd640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf4c065e60 0x7fdf4c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:45.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.453+0000 7fdf6e3bd640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf4c065e60 0x7fdf4c068320 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009af0 tx=0x7fdf64008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:45.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.461+0000 7fdf5effd640 1 -- 172.21.15.107:0/1145942502 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdf58050450 con 0x7fdf6814c470 2023-10-24T23:58:45.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 -- 172.21.15.107:0/1145942502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf680adcb0 con 0x7fdf6814c470 2023-10-24T23:58:45.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf5effd640 1 -- 172.21.15.107:0/1145942502 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7fdf58053a70 con 0x7fdf6814c470 2023-10-24T23:58:45.722 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:45.722 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:45.723 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:45.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 -- 172.21.15.107:0/1145942502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf4c065e60 msgr2=0x7fdf4c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:45.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf4c065e60 0x7fdf4c068320 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdf64009af0 tx=0x7fdf64008040 comp rx=0 tx=0).stop 2023-10-24T23:58:45.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 -- 172.21.15.107:0/1145942502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf6814c470 msgr2=0x7fdf680c1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:45.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf6814c470 0x7fdf680c1570 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fdf5800c500 tx=0x7fdf5800c9c0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 -- 172.21.15.107:0/1145942502 shutdown_connections 2023-10-24T23:58:45.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf68152b30 0x7fdf68141810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf6814c470 0x7fdf680c1570 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf4c065e60 0x7fdf4c068320 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 --2- 172.21.15.107:0/1145942502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf6806f490 0x7fdf681ed100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:45.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 -- 172.21.15.107:0/1145942502 >> 172.21.15.107:0/1145942502 conn(0x7fdf68070f60 msgr2=0x7fdf68152f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:45.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 -- 172.21.15.107:0/1145942502 shutdown_connections 2023-10-24T23:58:45.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:45.717+0000 7fdf70621640 1 -- 172.21.15.107:0/1145942502 wait complete. 2023-10-24T23:58:45.742 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:58:46.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:46.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.569+0000 7fc94b867640 1 Processor -- start 2023-10-24T23:58:46.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.569+0000 7fc94b867640 1 -- start start 2023-10-24T23:58:46.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc94b867640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc9440c70e0 0x7fc9440bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:46.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc94b867640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9440c2130 0x7fc9440be4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:46.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc94b867640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9440c1ca0 con 0x7fc944156770 2023-10-24T23:58:46.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc94b867640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9440c27e0 con 0x7fc9440c70e0 2023-10-24T23:58:46.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc94b867640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9440c10c0 con 0x7fc9440c2130 2023-10-24T23:58:46.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc948e02640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc9440c70e0 0x7fc9440bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:46.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc949e04640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9440c2130 0x7fc9440be4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:46.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc949603640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc944156770 0x7fc9440ab7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:36742/0 (socket says 172.21.15.107:36742) 2023-10-24T23:58:46.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc949603640 1 -- 172.21.15.107:0/3106731454 learned_addr learned my addr 172.21.15.107:0/3106731454 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:46.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc949e04640 1 -- 172.21.15.107:0/3106731454 >> v2:172.21.15.110:3300/0 conn(0x7fc9440c70e0 msgr2=0x7fc9440bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:46.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc949e04640 1 --2- 172.21.15.107:0/3106731454 >> v2:172.21.15.110:3300/0 conn(0x7fc9440c70e0 0x7fc9440bdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:46.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc949e04640 1 -- 172.21.15.107:0/3106731454 >> v1:172.21.15.110:6789/0 conn(0x7fc944156770 legacy=0x7fc9440ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:46.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.573+0000 7fc949e04640 1 -- 172.21.15.107:0/3106731454 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9440c13a0 con 0x7fc9440c2130 2023-10-24T23:58:46.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc949e04640 1 --2- 172.21.15.107:0/3106731454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9440c2130 0x7fc9440be4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009d90 tx=0x7fc93c01c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e73766b82f97987e server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:46.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc93bfff640 1 -- 172.21.15.107:0/3106731454 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c0a57a0 con 0x7fc9440c2130 2023-10-24T23:58:46.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc93bfff640 1 -- 172.21.15.107:0/3106731454 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c0a5900 con 0x7fc9440c2130 2023-10-24T23:58:46.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc93bfff640 1 -- 172.21.15.107:0/3106731454 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c0a5c90 con 0x7fc9440c2130 2023-10-24T23:58:46.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 -- 172.21.15.107:0/3106731454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9440c2130 msgr2=0x7fc9440be4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:46.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 --2- 172.21.15.107:0/3106731454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9440c2130 0x7fc9440be4e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009d90 tx=0x7fc93c01c830 comp rx=0 tx=0).stop 2023-10-24T23:58:46.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 -- 172.21.15.107:0/3106731454 shutdown_connections 2023-10-24T23:58:46.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 --2- 172.21.15.107:0/3106731454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9440c2130 0x7fc9440be4e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:46.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 --2- 172.21.15.107:0/3106731454 >> v2:172.21.15.110:3300/0 conn(0x7fc9440c70e0 0x7fc9440bdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:46.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 -- 172.21.15.107:0/3106731454 >> 172.21.15.107:0/3106731454 conn(0x7fc9440c65c0 msgr2=0x7fc9440c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:46.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 -- 172.21.15.107:0/3106731454 shutdown_connections 2023-10-24T23:58:46.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 -- 172.21.15.107:0/3106731454 wait complete. 2023-10-24T23:58:46.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 Processor -- start 2023-10-24T23:58:46.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 -- start start 2023-10-24T23:58:46.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9440c2130 0x7fc94414e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9440c70e0 0x7fc9441eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc944156770 0x7fc9441eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc944154260 con 0x7fc944156770 2023-10-24T23:58:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc944153dd0 con 0x7fc9440c2130 2023-10-24T23:58:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc94b867640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc944070a90 con 0x7fc9440c70e0 2023-10-24T23:58:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc949e04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc944156770 0x7fc9441eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:46.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc949e04640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc944156770 0x7fc9441eaf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44656/0 (socket says 172.21.15.107:44656) 2023-10-24T23:58:46.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc949e04640 1 -- 172.21.15.107:0/2112859332 learned_addr learned my addr 172.21.15.107:0/2112859332 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:46.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc949603640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9440c2130 0x7fc94414e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:46.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc948e02640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9440c70e0 0x7fc9441eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:46.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc949e04640 1 -- 172.21.15.107:0/2112859332 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9440c70e0 msgr2=0x7fc9441eaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:46.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc949e04640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9440c70e0 0x7fc9441eaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:46.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc949e04640 1 -- 172.21.15.107:0/2112859332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9440c2130 msgr2=0x7fc94414e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:46.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc949e04640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9440c2130 0x7fc94414e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:46.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.577+0000 7fc949e04640 1 -- 172.21.15.107:0/2112859332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c009a70 con 0x7fc944156770 2023-10-24T23:58:46.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc949e04640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc944156770 0x7fc9441eaf80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009d90 tx=0x7fc93c0195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc939ffb640 1 -- 172.21.15.107:0/2112859332 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c0ad640 con 0x7fc944156770 2023-10-24T23:58:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc939ffb640 1 -- 172.21.15.107:0/2112859332 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c0ad7a0 con 0x7fc944156770 2023-10-24T23:58:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc94b867640 1 -- 172.21.15.107:0/2112859332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9441eeaf0 con 0x7fc944156770 2023-10-24T23:58:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc94b867640 1 -- 172.21.15.107:0/2112859332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94414e620 con 0x7fc944156770 2023-10-24T23:58:46.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc939ffb640 1 -- 172.21.15.107:0/2112859332 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c0adb30 con 0x7fc944156770 2023-10-24T23:58:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc94b867640 1 -- 172.21.15.107:0/2112859332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc924005260 con 0x7fc944156770 2023-10-24T23:58:46.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc939ffb640 1 -- 172.21.15.107:0/2112859332 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc93c0b4070 con 0x7fc944156770 2023-10-24T23:58:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc939ffb640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc928065eb0 0x7fc928068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc939ffb640 1 -- 172.21.15.107:0/2112859332 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7fc93c11e570 con 0x7fc944156770 2023-10-24T23:58:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc949603640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc928065eb0 0x7fc928068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:46.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.581+0000 7fc949603640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc928065eb0 0x7fc928068370 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc9440bccf0 tx=0x7fc940007e90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:46.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.585+0000 7fc939ffb640 1 -- 172.21.15.107:0/2112859332 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc93c0eb7a0 con 0x7fc944156770 2023-10-24T23:58:46.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.785+0000 7fc94b867640 1 -- 172.21.15.107:0/2112859332 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc924005510 con 0x7fc944156770 2023-10-24T23:58:46.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc939ffb640 1 -- 172.21.15.107:0/2112859332 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7fc93c0eedc0 con 0x7fc944156770 2023-10-24T23:58:46.793 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:46.794 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:46.794 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:46.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 -- 172.21.15.107:0/2112859332 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc928065eb0 msgr2=0x7fc928068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc928065eb0 0x7fc928068370 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc9440bccf0 tx=0x7fc940007e90 comp rx=0 tx=0).stop 2023-10-24T23:58:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 -- 172.21.15.107:0/2112859332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc944156770 msgr2=0x7fc9441eaf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc944156770 0x7fc9441eaf80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc93c009d90 tx=0x7fc93c0195d0 comp rx=0 tx=0).stop 2023-10-24T23:58:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 -- 172.21.15.107:0/2112859332 shutdown_connections 2023-10-24T23:58:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc944156770 0x7fc9441eaf80 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc928065eb0 0x7fc928068370 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9440c70e0 0x7fc9441eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 --2- 172.21.15.107:0/2112859332 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc9440c2130 0x7fc94414e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 -- 172.21.15.107:0/2112859332 >> 172.21.15.107:0/2112859332 conn(0x7fc9440c65c0 msgr2=0x7fc9440c5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:46.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 -- 172.21.15.107:0/2112859332 shutdown_connections 2023-10-24T23:58:46.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:46.789+0000 7fc94b867640 1 -- 172.21.15.107:0/2112859332 wait complete. 2023-10-24T23:58:46.812 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:58:47.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.869+0000 7f97fc1aa640 1 Processor -- start 2023-10-24T23:58:47.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.869+0000 7f97fc1aa640 1 -- start start 2023-10-24T23:58:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97fc1aa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97f40a9820 0x7f97f40ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97fc1aa640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f97f406ee40 0x7f97f40aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97fc1aa640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f406f730 con 0x7f97f414fd40 2023-10-24T23:58:47.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97fc1aa640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f40aa750 con 0x7f97f406ee40 2023-10-24T23:58:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97fc1aa640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f4156a40 con 0x7f97f40a9820 2023-10-24T23:58:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97fa747640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f97f406ee40 0x7f97f40aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97f9f46640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f97f414fd40 0x7f97f40c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:36750/0 (socket says 172.21.15.107:36750) 2023-10-24T23:58:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97f9f46640 1 -- 172.21.15.107:0/1469005069 learned_addr learned my addr 172.21.15.107:0/1469005069 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:47.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97f9745640 1 --2- 172.21.15.107:0/1469005069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97f40a9820 0x7f97f40ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:47.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97f8f44640 1 -- 172.21.15.107:0/1469005069 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463296075 0 0) 0x7f97f406f730 con 0x7f97f414fd40 2023-10-24T23:58:47.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97f8f44640 1 -- 172.21.15.107:0/1469005069 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97e0003530 con 0x7f97f414fd40 2023-10-24T23:58:47.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97f9745640 1 -- 172.21.15.107:0/1469005069 >> v2:172.21.15.110:3300/0 conn(0x7f97f406ee40 msgr2=0x7f97f40aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:47.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97f9745640 1 --2- 172.21.15.107:0/1469005069 >> v2:172.21.15.110:3300/0 conn(0x7f97f406ee40 0x7f97f40aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:47.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97f9745640 1 -- 172.21.15.107:0/1469005069 >> v1:172.21.15.110:6789/0 conn(0x7f97f414fd40 legacy=0x7f97f40c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:47.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97f9745640 1 -- 172.21.15.107:0/1469005069 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f4156710 con 0x7f97f40a9820 2023-10-24T23:58:47.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.873+0000 7f97f9745640 1 --2- 172.21.15.107:0/1469005069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97f40a9820 0x7f97f40ab090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f97e40097c0 tx=0x7f97e401c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26a93ecd480fb936 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97f8f44640 1 -- 172.21.15.107:0/1469005069 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40ab600 con 0x7f97f40a9820 2023-10-24T23:58:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97f8f44640 1 -- 172.21.15.107:0/1469005069 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e40a7050 con 0x7f97f40a9820 2023-10-24T23:58:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97f8f44640 1 -- 172.21.15.107:0/1469005069 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40ab960 con 0x7f97f40a9820 2023-10-24T23:58:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 -- 172.21.15.107:0/1469005069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97f40a9820 msgr2=0x7f97f40ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:47.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 --2- 172.21.15.107:0/1469005069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97f40a9820 0x7f97f40ab090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f97e40097c0 tx=0x7f97e401c330 comp rx=0 tx=0).stop 2023-10-24T23:58:47.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 -- 172.21.15.107:0/1469005069 shutdown_connections 2023-10-24T23:58:47.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 --2- 172.21.15.107:0/1469005069 >> v2:172.21.15.110:3300/0 conn(0x7f97f406ee40 0x7f97f40aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:47.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 --2- 172.21.15.107:0/1469005069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97f40a9820 0x7f97f40ab090 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:47.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 -- 172.21.15.107:0/1469005069 >> 172.21.15.107:0/1469005069 conn(0x7f97f40c2950 msgr2=0x7f97f40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:47.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 -- 172.21.15.107:0/1469005069 shutdown_connections 2023-10-24T23:58:47.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 -- 172.21.15.107:0/1469005069 wait complete. 2023-10-24T23:58:47.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 Processor -- start 2023-10-24T23:58:47.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 -- start start 2023-10-24T23:58:47.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97f406ee40 0x7f97f41ed890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:47.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f414fd40 0x7f97f415b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:47.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97f414de70 0x7f97f41468a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:47.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f41ed400 con 0x7f97f414de70 2023-10-24T23:58:47.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f41ecdd0 con 0x7f97f414fd40 2023-10-24T23:58:47.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f414f1b0 con 0x7f97f406ee40 2023-10-24T23:58:47.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fa747640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97f414de70 0x7f97f41468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fa747640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97f414de70 0x7f97f41468a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44668/0 (socket says 172.21.15.107:44668) 2023-10-24T23:58:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fa747640 1 -- 172.21.15.107:0/4105621077 learned_addr learned my addr 172.21.15.107:0/4105621077 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97f9f46640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97f406ee40 0x7f97f41ed890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97f9745640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f414fd40 0x7f97f415b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fa747640 1 -- 172.21.15.107:0/4105621077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97f406ee40 msgr2=0x7f97f41ed890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:47.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fa747640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97f406ee40 0x7f97f41ed890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:47.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fa747640 1 -- 172.21.15.107:0/4105621077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f414fd40 msgr2=0x7f97f415b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:47.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fa747640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f414fd40 0x7f97f415b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:47.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fa747640 1 -- 172.21.15.107:0/4105621077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4009420 con 0x7f97f414de70 2023-10-24T23:58:47.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fa747640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97f414de70 0x7f97f41468a0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00efc0 tx=0x7f97ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:47.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97ea7fc640 1 -- 172.21.15.107:0/4105621077 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec014480 con 0x7f97f414de70 2023-10-24T23:58:47.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97ea7fc640 1 -- 172.21.15.107:0/4105621077 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ec00f040 con 0x7f97f414de70 2023-10-24T23:58:47.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 -- 172.21.15.107:0/4105621077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f41ed090 con 0x7f97f414de70 2023-10-24T23:58:47.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97ea7fc640 1 -- 172.21.15.107:0/4105621077 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec014820 con 0x7f97f414de70 2023-10-24T23:58:47.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.877+0000 7f97fc1aa640 1 -- 172.21.15.107:0/4105621077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f415bae0 con 0x7f97f414de70 2023-10-24T23:58:47.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.881+0000 7f97fc1aa640 1 -- 172.21.15.107:0/4105621077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f40c24a0 con 0x7f97f414de70 2023-10-24T23:58:47.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.881+0000 7f97ea7fc640 1 -- 172.21.15.107:0/4105621077 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f97ec014980 con 0x7f97f414de70 2023-10-24T23:58:47.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.881+0000 7f97ea7fc640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d8065e90 0x7f97d8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:47.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.881+0000 7f97ea7fc640 1 -- 172.21.15.107:0/4105621077 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f97ec0848a0 con 0x7f97f414de70 2023-10-24T23:58:47.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.881+0000 7f97f9f46640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d8065e90 0x7f97d8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.881+0000 7f97f9f46640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d8065e90 0x7f97d8068350 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f97f0007fd0 tx=0x7f97f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:47.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:47.885+0000 7f97ea7fc640 1 -- 172.21.15.107:0/4105621077 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f97ec051940 con 0x7f97f414de70 2023-10-24T23:58:48.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.069+0000 7f97fc1aa640 1 -- 172.21.15.107:0/4105621077 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97f4070840 con 0x7f97f414de70 2023-10-24T23:58:48.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97ea7fc640 1 -- 172.21.15.107:0/4105621077 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f97ec018090 con 0x7f97f414de70 2023-10-24T23:58:48.077 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:48.077 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:48.077 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:48.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 -- 172.21.15.107:0/4105621077 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d8065e90 msgr2=0x7f97d8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:48.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d8065e90 0x7f97d8068350 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f97f0007fd0 tx=0x7f97f0009040 comp rx=0 tx=0).stop 2023-10-24T23:58:48.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 -- 172.21.15.107:0/4105621077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97f414de70 msgr2=0x7f97f41468a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:48.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97f414de70 0x7f97f41468a0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00efc0 tx=0x7f97ec00c490 comp rx=0 tx=0).stop 2023-10-24T23:58:48.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 -- 172.21.15.107:0/4105621077 shutdown_connections 2023-10-24T23:58:48.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97f414de70 0x7f97f41468a0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:48.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97f414fd40 0x7f97f415b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:48.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f97d8065e90 0x7f97d8068350 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:48.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 --2- 172.21.15.107:0/4105621077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97f406ee40 0x7f97f41ed890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:48.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 -- 172.21.15.107:0/4105621077 >> 172.21.15.107:0/4105621077 conn(0x7f97f40c2950 msgr2=0x7f97f40c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:48.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 -- 172.21.15.107:0/4105621077 shutdown_connections 2023-10-24T23:58:48.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.073+0000 7f97fc1aa640 1 -- 172.21.15.107:0/4105621077 wait complete. 2023-10-24T23:58:48.095 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:58:48.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:48.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.937+0000 7f72745ad640 1 Processor -- start 2023-10-24T23:58:48.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.937+0000 7f72745ad640 1 -- start start 2023-10-24T23:58:48.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f72745ad640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f726c14fd40 0x7f726c0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:48.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f72745ad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f726c0a9820 0x7f726c0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:48.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f72745ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726c06f730 con 0x7f726c06ee40 2023-10-24T23:58:48.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f72745ad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f726c0aa750 con 0x7f726c0a9820 2023-10-24T23:58:48.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f72745ad640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f726c156a40 con 0x7f726c14fd40 2023-10-24T23:58:48.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7272b4a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f726c06ee40 0x7f726c0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40450/0 (socket says 172.21.15.107:40450) 2023-10-24T23:58:48.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7272b4a640 1 -- 172.21.15.107:0/321786465 learned_addr learned my addr 172.21.15.107:0/321786465 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:48.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7271b48640 1 --2- 172.21.15.107:0/321786465 >> v2:172.21.15.107:3300/0 conn(0x7f726c0a9820 0x7f726c0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:48.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7272349640 1 --2- 172.21.15.107:0/321786465 >> v2:172.21.15.110:3300/0 conn(0x7f726c14fd40 0x7f726c0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:48.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7271347640 1 -- 172.21.15.107:0/321786465 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680763478 0 0) 0x7f726c06f730 con 0x7f726c06ee40 2023-10-24T23:58:48.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7271347640 1 -- 172.21.15.107:0/321786465 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7258003530 con 0x7f726c06ee40 2023-10-24T23:58:48.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7271b48640 1 -- 172.21.15.107:0/321786465 >> v2:172.21.15.110:3300/0 conn(0x7f726c14fd40 msgr2=0x7f726c0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:48.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7271b48640 1 --2- 172.21.15.107:0/321786465 >> v2:172.21.15.110:3300/0 conn(0x7f726c14fd40 0x7f726c0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:48.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7271b48640 1 -- 172.21.15.107:0/321786465 >> v1:172.21.15.107:6789/0 conn(0x7f726c06ee40 legacy=0x7f726c0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:48.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7271b48640 1 -- 172.21.15.107:0/321786465 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c156710 con 0x7f726c0a9820 2023-10-24T23:58:48.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7272349640 1 --2- 172.21.15.107:0/321786465 >> v2:172.21.15.110:3300/0 conn(0x7f726c14fd40 0x7f726c0ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7271b48640 1 --2- 172.21.15.107:0/321786465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726c0a9820 0x7f726c0aa210 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f726c0c28b0 tx=0x7f725c01c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220f4a9a2b9904c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7271347640 1 -- 172.21.15.107:0/321786465 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c0ab600 con 0x7f726c0a9820 2023-10-24T23:58:48.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7271347640 1 -- 172.21.15.107:0/321786465 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725c0a7050 con 0x7f726c0a9820 2023-10-24T23:58:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.941+0000 7f7271347640 1 -- 172.21.15.107:0/321786465 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c0ab980 con 0x7f726c0a9820 2023-10-24T23:58:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 -- 172.21.15.107:0/321786465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726c0a9820 msgr2=0x7f726c0aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 --2- 172.21.15.107:0/321786465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726c0a9820 0x7f726c0aa210 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f726c0c28b0 tx=0x7f725c01c310 comp rx=0 tx=0).stop 2023-10-24T23:58:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 -- 172.21.15.107:0/321786465 shutdown_connections 2023-10-24T23:58:48.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 --2- 172.21.15.107:0/321786465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726c0a9820 0x7f726c0aa210 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:48.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 --2- 172.21.15.107:0/321786465 >> v2:172.21.15.110:3300/0 conn(0x7f726c14fd40 0x7f726c0ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:48.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 -- 172.21.15.107:0/321786465 >> 172.21.15.107:0/321786465 conn(0x7f726c0c2950 msgr2=0x7f726c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:48.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 -- 172.21.15.107:0/321786465 shutdown_connections 2023-10-24T23:58:48.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 -- 172.21.15.107:0/321786465 wait complete. 2023-10-24T23:58:48.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 Processor -- start 2023-10-24T23:58:48.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 -- start start 2023-10-24T23:58:48.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f726c06ee40 0x7f726c147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:48.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726c14fd40 0x7f726c1ee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:48.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f726c1eaac0 0x7f726c1eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:48.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c14f590 con 0x7f726c14fd40 2023-10-24T23:58:48.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c1ee1f0 con 0x7f726c06ee40 2023-10-24T23:58:48.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726c0c5870 con 0x7f726c1eaac0 2023-10-24T23:58:48.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7271b48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726c14fd40 0x7f726c1ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:48.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7271b48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726c14fd40 0x7f726c1ee680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44678/0 (socket says 172.21.15.107:44678) 2023-10-24T23:58:48.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7271b48640 1 -- 172.21.15.107:0/598063489 learned_addr learned my addr 172.21.15.107:0/598063489 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:48.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7271b48640 1 -- 172.21.15.107:0/598063489 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f726c1eaac0 msgr2=0x7f726c1eebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:48.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7271b48640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f726c1eaac0 0x7f726c1eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:48.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7271b48640 1 -- 172.21.15.107:0/598063489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f726c06ee40 msgr2=0x7f726c147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:48.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7272349640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f726c06ee40 0x7f726c147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:48.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7271b48640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f726c06ee40 0x7f726c147b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:48.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7271b48640 1 -- 172.21.15.107:0/598063489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725c009420 con 0x7f726c14fd40 2023-10-24T23:58:48.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7271b48640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726c14fd40 0x7f726c1ee680 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f725c01c7f0 tx=0x7f725c0190b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:48.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7262ffd640 1 -- 172.21.15.107:0/598063489 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c0b4430 con 0x7f726c14fd40 2023-10-24T23:58:48.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7262ffd640 1 -- 172.21.15.107:0/598063489 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725c0b4590 con 0x7f726c14fd40 2023-10-24T23:58:48.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 -- 172.21.15.107:0/598063489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726c0c5590 con 0x7f726c14fd40 2023-10-24T23:58:48.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f7262ffd640 1 -- 172.21.15.107:0/598063489 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c0b4940 con 0x7f726c14fd40 2023-10-24T23:58:48.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.945+0000 7f72745ad640 1 -- 172.21.15.107:0/598063489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726c148050 con 0x7f726c14fd40 2023-10-24T23:58:48.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.949+0000 7f72745ad640 1 -- 172.21.15.107:0/598063489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726c0c24a0 con 0x7f726c14fd40 2023-10-24T23:58:48.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.949+0000 7f7262ffd640 1 -- 172.21.15.107:0/598063489 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f725c0b2040 con 0x7f726c14fd40 2023-10-24T23:58:48.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.949+0000 7f7262ffd640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e60 0x7f7250068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:48.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.949+0000 7f7272349640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e60 0x7f7250068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:48.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.949+0000 7f7262ffd640 1 -- 172.21.15.107:0/598063489 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f725c11dba0 con 0x7f726c14fd40 2023-10-24T23:58:48.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.949+0000 7f7272349640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e60 0x7f7250068320 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7268006fd0 tx=0x7f7268008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:48.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:48.953+0000 7f7262ffd640 1 -- 172.21.15.107:0/598063489 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f725c0eac40 con 0x7f726c14fd40 2023-10-24T23:58:49.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.145+0000 7f72745ad640 1 -- 172.21.15.107:0/598063489 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f726c070840 con 0x7f726c14fd40 2023-10-24T23:58:49.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.145+0000 7f7262ffd640 1 -- 172.21.15.107:0/598063489 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f725c0ee260 con 0x7f726c14fd40 2023-10-24T23:58:49.152 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:49.152 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:49.152 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:49.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 -- 172.21.15.107:0/598063489 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e60 msgr2=0x7f7250068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:49.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e60 0x7f7250068320 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7268006fd0 tx=0x7f7268008040 comp rx=0 tx=0).stop 2023-10-24T23:58:49.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 -- 172.21.15.107:0/598063489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726c14fd40 msgr2=0x7f726c1ee680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:49.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726c14fd40 0x7f726c1ee680 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f725c01c7f0 tx=0x7f725c0190b0 comp rx=0 tx=0).stop 2023-10-24T23:58:49.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 -- 172.21.15.107:0/598063489 shutdown_connections 2023-10-24T23:58:49.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e60 0x7f7250068320 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:49.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f726c1eaac0 0x7f726c1eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:49.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726c14fd40 0x7f726c1ee680 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:49.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 --2- 172.21.15.107:0/598063489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f726c06ee40 0x7f726c147b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:49.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 -- 172.21.15.107:0/598063489 >> 172.21.15.107:0/598063489 conn(0x7f726c0c2950 msgr2=0x7f726c0a9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:49.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 -- 172.21.15.107:0/598063489 shutdown_connections 2023-10-24T23:58:49.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:49.149+0000 7f72745ad640 1 -- 172.21.15.107:0/598063489 wait complete. 2023-10-24T23:58:49.170 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:58:50.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:50.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf47699640 1 Processor -- start 2023-10-24T23:58:50.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf47699640 1 -- start start 2023-10-24T23:58:50.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf47699640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbf400ac9d0 0x7fbf40158890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:50.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf47699640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbf40154a10 0x7fbf40154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:50.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf47699640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf400ad0d0 con 0x7fbf40152160 2023-10-24T23:58:50.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf47699640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4006fe90 con 0x7fbf400ac9d0 2023-10-24T23:58:50.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf47699640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf40158dd0 con 0x7fbf40154a10 2023-10-24T23:58:50.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf44c34640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbf400ac9d0 0x7fbf40158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:50.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf45435640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbf40152160 0x7fbf4006cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40460/0 (socket says 172.21.15.107:40460) 2023-10-24T23:58:50.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf45435640 1 -- 172.21.15.107:0/1267978134 learned_addr learned my addr 172.21.15.107:0/1267978134 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:50.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf45c36640 1 --2- 172.21.15.107:0/1267978134 >> v2:172.21.15.110:3300/0 conn(0x7fbf40154a10 0x7fbf40154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:50.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf44c34640 1 -- 172.21.15.107:0/1267978134 >> v2:172.21.15.110:3300/0 conn(0x7fbf40154a10 msgr2=0x7fbf40154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:50.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf44c34640 1 --2- 172.21.15.107:0/1267978134 >> v2:172.21.15.110:3300/0 conn(0x7fbf40154a10 0x7fbf40154ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:50.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf44c34640 1 -- 172.21.15.107:0/1267978134 >> v1:172.21.15.107:6789/0 conn(0x7fbf40152160 legacy=0x7fbf4006cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:50.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.225+0000 7fbf44c34640 1 -- 172.21.15.107:0/1267978134 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf400c0300 con 0x7fbf400ac9d0 2023-10-24T23:58:50.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf44c34640 1 --2- 172.21.15.107:0/1267978134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf400ac9d0 0x7fbf40158890 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fbf4006a640 tx=0x7fbf3401c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec95f94322275e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:50.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf3ffff640 1 -- 172.21.15.107:0/1267978134 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf340ac640 con 0x7fbf400ac9d0 2023-10-24T23:58:50.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf3ffff640 1 -- 172.21.15.107:0/1267978134 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf340ac7a0 con 0x7fbf400ac9d0 2023-10-24T23:58:50.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf3ffff640 1 -- 172.21.15.107:0/1267978134 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf340acb30 con 0x7fbf400ac9d0 2023-10-24T23:58:50.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 -- 172.21.15.107:0/1267978134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf400ac9d0 msgr2=0x7fbf40158890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:50.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 --2- 172.21.15.107:0/1267978134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf400ac9d0 0x7fbf40158890 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fbf4006a640 tx=0x7fbf3401c310 comp rx=0 tx=0).stop 2023-10-24T23:58:50.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 -- 172.21.15.107:0/1267978134 shutdown_connections 2023-10-24T23:58:50.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 --2- 172.21.15.107:0/1267978134 >> v2:172.21.15.110:3300/0 conn(0x7fbf40154a10 0x7fbf40154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:50.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 --2- 172.21.15.107:0/1267978134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf400ac9d0 0x7fbf40158890 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:50.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 -- 172.21.15.107:0/1267978134 >> 172.21.15.107:0/1267978134 conn(0x7fbf400c6ed0 msgr2=0x7fbf400c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:50.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 -- 172.21.15.107:0/1267978134 shutdown_connections 2023-10-24T23:58:50.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 -- 172.21.15.107:0/1267978134 wait complete. 2023-10-24T23:58:50.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 Processor -- start 2023-10-24T23:58:50.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 -- start start 2023-10-24T23:58:50.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf400ac9d0 0x7fbf400c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:50.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbf40152160 0x7fbf4014d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:50.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf40154a10 0x7fbf401f0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:50.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf400c5b60 con 0x7fbf40154a10 2023-10-24T23:58:50.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf400c6820 con 0x7fbf400ac9d0 2023-10-24T23:58:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf47699640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf400c6b00 con 0x7fbf40152160 2023-10-24T23:58:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf45c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf40154a10 0x7fbf401f0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf45c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf40154a10 0x7fbf401f0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44694/0 (socket says 172.21.15.107:44694) 2023-10-24T23:58:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf45c36640 1 -- 172.21.15.107:0/1930271062 learned_addr learned my addr 172.21.15.107:0/1930271062 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:50.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf45435640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf400ac9d0 0x7fbf400c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:50.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.229+0000 7fbf44c34640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbf40152160 0x7fbf4014d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:50.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf45c36640 1 -- 172.21.15.107:0/1930271062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbf40152160 msgr2=0x7fbf4014d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:50.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf45c36640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbf40152160 0x7fbf4014d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf45c36640 1 -- 172.21.15.107:0/1930271062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf400ac9d0 msgr2=0x7fbf400c5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf45c36640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf400ac9d0 0x7fbf400c5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf45c36640 1 -- 172.21.15.107:0/1930271062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf34009420 con 0x7fbf40154a10 2023-10-24T23:58:50.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf44c34640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbf40152160 0x7fbf4014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf45c36640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf40154a10 0x7fbf401f0110 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000c7f0 tx=0x7fbf3000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf3dffb640 1 -- 172.21.15.107:0/1930271062 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30004220 con 0x7fbf40154a10 2023-10-24T23:58:50.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf3dffb640 1 -- 172.21.15.107:0/1930271062 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf30004380 con 0x7fbf40154a10 2023-10-24T23:58:50.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf47699640 1 -- 172.21.15.107:0/1930271062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf400c5510 con 0x7fbf40154a10 2023-10-24T23:58:50.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf3dffb640 1 -- 172.21.15.107:0/1930271062 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30003840 con 0x7fbf40154a10 2023-10-24T23:58:50.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf47699640 1 -- 172.21.15.107:0/1930271062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4014d750 con 0x7fbf40154a10 2023-10-24T23:58:50.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf47699640 1 -- 172.21.15.107:0/1930271062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf400746e0 con 0x7fbf40154a10 2023-10-24T23:58:50.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf3dffb640 1 -- 172.21.15.107:0/1930271062 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbf30003bb0 con 0x7fbf40154a10 2023-10-24T23:58:50.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf3dffb640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbf2c065e60 0x7fbf2c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:50.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf3dffb640 1 -- 172.21.15.107:0/1930271062 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7fbf30083e10 con 0x7fbf40154a10 2023-10-24T23:58:50.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf45435640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbf2c065e60 0x7fbf2c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:50.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.233+0000 7fbf45435640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbf2c065e60 0x7fbf2c068320 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fbf38005c10 tx=0x7fbf38009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:50.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.237+0000 7fbf3dffb640 1 -- 172.21.15.107:0/1930271062 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbf30050ee0 con 0x7fbf40154a10 2023-10-24T23:58:50.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.425+0000 7fbf47699640 1 -- 172.21.15.107:0/1930271062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf400c5ca0 con 0x7fbf40154a10 2023-10-24T23:58:50.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.425+0000 7fbf3dffb640 1 -- 172.21.15.107:0/1930271062 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7fbf3001e090 con 0x7fbf40154a10 2023-10-24T23:58:50.431 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:50.431 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:50.432 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 -- 172.21.15.107:0/1930271062 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbf2c065e60 msgr2=0x7fbf2c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbf2c065e60 0x7fbf2c068320 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fbf38005c10 tx=0x7fbf38009040 comp rx=0 tx=0).stop 2023-10-24T23:58:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 -- 172.21.15.107:0/1930271062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf40154a10 msgr2=0x7fbf401f0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf40154a10 0x7fbf401f0110 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000c7f0 tx=0x7fbf3000ccb0 comp rx=0 tx=0).stop 2023-10-24T23:58:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 -- 172.21.15.107:0/1930271062 shutdown_connections 2023-10-24T23:58:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbf40154a10 0x7fbf401f0110 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbf2c065e60 0x7fbf2c068320 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbf40152160 0x7fbf4014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 --2- 172.21.15.107:0/1930271062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbf400ac9d0 0x7fbf400c5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 -- 172.21.15.107:0/1930271062 >> 172.21.15.107:0/1930271062 conn(0x7fbf400c6ed0 msgr2=0x7fbf4014f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 -- 172.21.15.107:0/1930271062 shutdown_connections 2023-10-24T23:58:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:50.429+0000 7fbf47699640 1 -- 172.21.15.107:0/1930271062 wait complete. 2023-10-24T23:58:50.449 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:58:51.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f9016043640 1 Processor -- start 2023-10-24T23:58:51.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f9016043640 1 -- start start 2023-10-24T23:58:51.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f9016043640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f901014fd40 0x7f90100ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:51.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f9016043640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f901006ee40 0x7f90100aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f9016043640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f901006f730 con 0x7f90100a9820 2023-10-24T23:58:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f9016043640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90100aa750 con 0x7f901006ee40 2023-10-24T23:58:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f9016043640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9010156a40 con 0x7f901014fd40 2023-10-24T23:58:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f900ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f901006ee40 0x7f90100aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:51.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f900ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f901006ee40 0x7f90100aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44706/0 (socket says 172.21.15.107:44706) 2023-10-24T23:58:51.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f900ffff640 1 -- 172.21.15.107:0/510455974 learned_addr learned my addr 172.21.15.107:0/510455974 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:51.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f900ffff640 1 -- 172.21.15.107:0/510455974 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f901014fd40 msgr2=0x7f90100ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:51.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f900ffff640 1 --2- 172.21.15.107:0/510455974 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f901014fd40 0x7f90100ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:51.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f900ffff640 1 -- 172.21.15.107:0/510455974 >> v1:172.21.15.110:6789/0 conn(0x7f90100a9820 legacy=0x7f90100c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:51.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.297+0000 7f900ffff640 1 -- 172.21.15.107:0/510455974 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9010156710 con 0x7f901006ee40 2023-10-24T23:58:51.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900ffff640 1 --2- 172.21.15.107:0/510455974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006ee40 0x7f90100aa210 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f9008009e50 tx=0x7f900801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efebfe9f1db7bb server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:51.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900effd640 1 -- 172.21.15.107:0/510455974 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90080a56a0 con 0x7f901006ee40 2023-10-24T23:58:51.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900effd640 1 -- 172.21.15.107:0/510455974 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90080a5800 con 0x7f901006ee40 2023-10-24T23:58:51.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900effd640 1 -- 172.21.15.107:0/510455974 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90080a5b90 con 0x7f901006ee40 2023-10-24T23:58:51.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 -- 172.21.15.107:0/510455974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006ee40 msgr2=0x7f90100aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:51.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 --2- 172.21.15.107:0/510455974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006ee40 0x7f90100aa210 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f9008009e50 tx=0x7f900801c880 comp rx=0 tx=0).stop 2023-10-24T23:58:51.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 -- 172.21.15.107:0/510455974 shutdown_connections 2023-10-24T23:58:51.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 --2- 172.21.15.107:0/510455974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006ee40 0x7f90100aa210 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:51.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 --2- 172.21.15.107:0/510455974 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f901014fd40 0x7f90100ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:51.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 -- 172.21.15.107:0/510455974 >> 172.21.15.107:0/510455974 conn(0x7f90100c2950 msgr2=0x7f90100c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:51.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 -- 172.21.15.107:0/510455974 shutdown_connections 2023-10-24T23:58:51.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 -- 172.21.15.107:0/510455974 wait complete. 2023-10-24T23:58:51.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 Processor -- start 2023-10-24T23:58:51.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 -- start start 2023-10-24T23:58:51.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006ee40 0x7f901015b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:51.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f901014fd40 0x7f901006a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:51.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f901006b6f0 0x7f901014c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:51.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90101554f0 con 0x7f901006ee40 2023-10-24T23:58:51.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90100bae70 con 0x7f901006b6f0 2023-10-24T23:58:51.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f901006bbf0 con 0x7f901014fd40 2023-10-24T23:58:51.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006ee40 0x7f901015b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:51.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006ee40 0x7f901015b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44720/0 (socket says 172.21.15.107:44720) 2023-10-24T23:58:51.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900f7fe640 1 -- 172.21.15.107:0/262368813 learned_addr learned my addr 172.21.15.107:0/262368813 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:51.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900ffff640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f901006b6f0 0x7f901014c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:51.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9007fff640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f901014fd40 0x7f901006a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:51.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900f7fe640 1 -- 172.21.15.107:0/262368813 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f901014fd40 msgr2=0x7f901006a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:51.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900f7fe640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f901014fd40 0x7f901006a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:51.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900f7fe640 1 -- 172.21.15.107:0/262368813 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f901006b6f0 msgr2=0x7f901014c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:51.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900f7fe640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f901006b6f0 0x7f901014c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:51.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900f7fe640 1 -- 172.21.15.107:0/262368813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9008009b30 con 0x7f901006ee40 2023-10-24T23:58:51.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900ffff640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f901006b6f0 0x7f901014c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:58:51.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900f7fe640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006ee40 0x7f901015b5a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800b550 tx=0x7f8ff800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:51.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900cff9640 1 -- 172.21.15.107:0/262368813 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8010870 con 0x7f901006ee40 2023-10-24T23:58:51.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900cff9640 1 -- 172.21.15.107:0/262368813 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff80109f0 con 0x7f901006ee40 2023-10-24T23:58:51.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 -- 172.21.15.107:0/262368813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9010155740 con 0x7f901006ee40 2023-10-24T23:58:51.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f900cff9640 1 -- 172.21.15.107:0/262368813 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8010da0 con 0x7f901006ee40 2023-10-24T23:58:51.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.301+0000 7f9016043640 1 -- 172.21.15.107:0/262368813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f901015bae0 con 0x7f901006ee40 2023-10-24T23:58:51.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.305+0000 7f9016043640 1 -- 172.21.15.107:0/262368813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90100c24a0 con 0x7f901006ee40 2023-10-24T23:58:51.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.305+0000 7f900cff9640 1 -- 172.21.15.107:0/262368813 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8ff800f3c0 con 0x7f901006ee40 2023-10-24T23:58:51.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.305+0000 7f900cff9640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8ff0065e60 0x7f8ff0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:51.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.305+0000 7f900cff9640 1 -- 172.21.15.107:0/262368813 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f8ff8083e90 con 0x7f901006ee40 2023-10-24T23:58:51.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.305+0000 7f9007fff640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8ff0065e60 0x7f8ff0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:51.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.305+0000 7f9007fff640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8ff0065e60 0x7f8ff0068320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f901006ffb0 tx=0x7f9000009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:51.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.309+0000 7f900cff9640 1 -- 172.21.15.107:0/262368813 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8ff8050ea0 con 0x7f901006ee40 2023-10-24T23:58:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.493+0000 7f9016043640 1 -- 172.21.15.107:0/262368813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90100bafe0 con 0x7f901006ee40 2023-10-24T23:58:51.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.493+0000 7f900cff9640 1 -- 172.21.15.107:0/262368813 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f8ff8014090 con 0x7f901006ee40 2023-10-24T23:58:51.499 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:51.499 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:51.500 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 -- 172.21.15.107:0/262368813 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8ff0065e60 msgr2=0x7f8ff0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8ff0065e60 0x7f8ff0068320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f901006ffb0 tx=0x7f9000009040 comp rx=0 tx=0).stop 2023-10-24T23:58:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 -- 172.21.15.107:0/262368813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006ee40 msgr2=0x7f901015b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006ee40 0x7f901015b5a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800b550 tx=0x7f8ff800ba10 comp rx=0 tx=0).stop 2023-10-24T23:58:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 -- 172.21.15.107:0/262368813 shutdown_connections 2023-10-24T23:58:51.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f901006b6f0 0x7f901014c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:51.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8ff0065e60 0x7f8ff0068320 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:51.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f901014fd40 0x7f901006a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:51.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 --2- 172.21.15.107:0/262368813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f901006ee40 0x7f901015b5a0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:51.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 -- 172.21.15.107:0/262368813 >> 172.21.15.107:0/262368813 conn(0x7f90100c2950 msgr2=0x7f90100c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 -- 172.21.15.107:0/262368813 shutdown_connections 2023-10-24T23:58:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:51.497+0000 7f9016043640 1 -- 172.21.15.107:0/262368813 wait complete. 2023-10-24T23:58:51.517 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-24T23:58:52.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:52.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.573+0000 7f6d0dd0d640 1 Processor -- start 2023-10-24T23:58:52.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.573+0000 7f6d0dd0d640 1 -- start start 2023-10-24T23:58:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d0dd0d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6d0806a230 0x7f6d081551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d0dd0d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d080c1860 con 0x7f6d0814b600 2023-10-24T23:58:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d0dd0d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d08154d50 con 0x7f6d0806af70 2023-10-24T23:58:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d0dd0d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d0814b9e0 con 0x7f6d0806a230 2023-10-24T23:58:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d077fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6d0814b600 0x7f6d08151fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56102/0 (socket says 172.21.15.107:56102) 2023-10-24T23:58:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d077fe640 1 -- 172.21.15.107:0/1799144819 learned_addr learned my addr 172.21.15.107:0/1799144819 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:52.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d07fff640 1 --2- 172.21.15.107:0/1799144819 >> v2:172.21.15.110:3300/0 conn(0x7f6d0806a230 0x7f6d081551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568036843 0 0) 0x7f6d080c1860 con 0x7f6d0814b600 2023-10-24T23:58:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cf4005740 con 0x7f6d0814b600 2023-10-24T23:58:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3819121640 0 0) 0x7f6cf4005740 con 0x7f6d0814b600 2023-10-24T23:58:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 >> v2:172.21.15.110:3300/0 conn(0x7f6d0806a230 msgr2=0x7f6d081551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:52.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 --2- 172.21.15.107:0/1799144819 >> v2:172.21.15.110:3300/0 conn(0x7f6d0806a230 0x7f6d081551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 >> v1:172.21.15.110:6789/0 conn(0x7f6d0806af70 legacy=0x7f6d0806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d080c1ab0 con 0x7f6d0814b600 2023-10-24T23:58:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6d00001680 con 0x7f6d0814b600 2023-10-24T23:58:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 >> v1:172.21.15.107:6789/0 conn(0x7f6d0814b600 legacy=0x7f6d08151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:52.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 --2- 172.21.15.107:0/1799144819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf4005b90 0x7f6cf4005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 --2- 172.21.15.107:0/1799144819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf40069e0 0x7f6cf4006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 --2- 172.21.15.107:0/1799144819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf40073a0 0x7f6cf400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0814b9e0 con 0x7f6cf40073a0 2023-10-24T23:58:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf400e340 con 0x7f6cf40069e0 2023-10-24T23:58:52.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf400e500 con 0x7f6cf4005b90 2023-10-24T23:58:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d07fff640 1 --2- 172.21.15.107:0/1799144819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf40073a0 0x7f6cf400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d06ffd640 1 --2- 172.21.15.107:0/1799144819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf40069e0 0x7f6cf4006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d07fff640 1 -- 172.21.15.107:0/1799144819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf4005b90 msgr2=0x7f6cf4005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d07fff640 1 --2- 172.21.15.107:0/1799144819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf4005b90 0x7f6cf4005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:52.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d07fff640 1 -- 172.21.15.107:0/1799144819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf40069e0 msgr2=0x7f6cf4006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d07fff640 1 --2- 172.21.15.107:0/1799144819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf40069e0 0x7f6cf4006e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:52.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.577+0000 7f6d07fff640 1 -- 172.21.15.107:0/1799144819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf4005740 con 0x7f6cf40073a0 2023-10-24T23:58:52.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d07fff640 1 -- 172.21.15.107:0/1799144819 reap_dead start 2023-10-24T23:58:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d07fff640 1 --2- 172.21.15.107:0/1799144819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf40073a0 0x7f6cf400dba0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009f90 tx=0x7f6cf801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c4d6aedcc3fca7 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf80a5c70 con 0x7f6cf40073a0 2023-10-24T23:58:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf800f040 con 0x7f6cf40073a0 2023-10-24T23:58:52.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d067fc640 1 -- 172.21.15.107:0/1799144819 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf80039a0 con 0x7f6cf40073a0 2023-10-24T23:58:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/1799144819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf40073a0 msgr2=0x7f6cf400dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 --2- 172.21.15.107:0/1799144819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf40073a0 0x7f6cf400dba0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8009f90 tx=0x7f6cf801cc60 comp rx=0 tx=0).stop 2023-10-24T23:58:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/1799144819 shutdown_connections 2023-10-24T23:58:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 --2- 172.21.15.107:0/1799144819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf40073a0 0x7f6cf400dba0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/1799144819 >> 172.21.15.107:0/1799144819 conn(0x7f6d080c2950 msgr2=0x7f6d080c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/1799144819 shutdown_connections 2023-10-24T23:58:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/1799144819 wait complete. 2023-10-24T23:58:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 Processor -- start 2023-10-24T23:58:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 -- start start 2023-10-24T23:58:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf40073a0 0x7f6d08147670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d08070640 0x7f6d08152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d08147bb0 0x7f6d08152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d081e5820 con 0x7f6d08070640 2023-10-24T23:58:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080701b0 con 0x7f6cf40073a0 2023-10-24T23:58:52.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d0dd0d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d080a95f0 con 0x7f6d08147bb0 2023-10-24T23:58:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d06ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d08070640 0x7f6d08152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d06ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d08070640 0x7f6d08152920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37982/0 (socket says 172.21.15.107:37982) 2023-10-24T23:58:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d06ffd640 1 -- 172.21.15.107:0/171726796 learned_addr learned my addr 172.21.15.107:0/171726796 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.581+0000 7f6d077fe640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf40073a0 0x7f6d08147670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d07fff640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d08147bb0 0x7f6d08152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d06ffd640 1 -- 172.21.15.107:0/171726796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d08147bb0 msgr2=0x7f6d08152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d06ffd640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d08147bb0 0x7f6d08152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d06ffd640 1 -- 172.21.15.107:0/171726796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf40073a0 msgr2=0x7f6d08147670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d06ffd640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf40073a0 0x7f6d08147670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:52.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d06ffd640 1 -- 172.21.15.107:0/171726796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d081e5a60 con 0x7f6d08070640 2023-10-24T23:58:52.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d07fff640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d08147bb0 0x7f6d08152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:58:52.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d06ffd640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d08070640 0x7f6d08152920 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000d4a0 tx=0x7f6cf000d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:52.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6cfffff640 1 -- 172.21.15.107:0/171726796 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0005480 con 0x7f6d08070640 2023-10-24T23:58:52.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6cfffff640 1 -- 172.21.15.107:0/171726796 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf0003620 con 0x7f6d08070640 2023-10-24T23:58:52.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6cfffff640 1 -- 172.21.15.107:0/171726796 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf00039f0 con 0x7f6d08070640 2023-10-24T23:58:52.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/171726796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d081e5d00 con 0x7f6d08070640 2023-10-24T23:58:52.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/171726796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d080a9890 con 0x7f6d08070640 2023-10-24T23:58:52.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/171726796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d080746e0 con 0x7f6d08070640 2023-10-24T23:58:52.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6cfffff640 1 -- 172.21.15.107:0/171726796 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6cf0003b50 con 0x7f6d08070640 2023-10-24T23:58:52.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6cfffff640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ce8065e90 0x7f6ce8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:52.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d077fe640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ce8065e90 0x7f6ce8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:52.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6cfffff640 1 -- 172.21.15.107:0/171726796 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 7649+0+0 (secure 0 0 0) 0x7f6cf0083be0 con 0x7f6d08070640 2023-10-24T23:58:52.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.585+0000 7f6d077fe640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ce8065e90 0x7f6ce8068350 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6d08151570 tx=0x7f6d00001700 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:52.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.589+0000 7f6cfffff640 1 -- 172.21.15.107:0/171726796 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6cf0050bf0 con 0x7f6d08070640 2023-10-24T23:58:52.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.781+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/171726796 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d08151f80 con 0x7f6d08070640 2023-10-24T23:58:52.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.781+0000 7f6cfffff640 1 -- 172.21.15.107:0/171726796 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f6cf0054210 con 0x7f6d08070640 2023-10-24T23:58:52.790 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:52.790 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50},{"gid":5925,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:06.525380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4937":{"gid":4937,"name":"d","rank":0,"incarnation":44,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6835/2348043270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2348043270},{"type":"v1","addr":"172.21.15.110:6835","nonce":2348043270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 50 2023-10-24T23:58:52.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/171726796 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ce8065e90 msgr2=0x7f6ce8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ce8065e90 0x7f6ce8068350 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6d08151570 tx=0x7f6d00001700 comp rx=0 tx=0).stop 2023-10-24T23:58:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/171726796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d08070640 msgr2=0x7f6d08152920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:52.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d08070640 0x7f6d08152920 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000d4a0 tx=0x7f6cf000d960 comp rx=0 tx=0).stop 2023-10-24T23:58:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/171726796 shutdown_connections 2023-10-24T23:58:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ce8065e90 0x7f6ce8068350 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6d08147bb0 0x7f6d08152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d08070640 0x7f6d08152920 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 --2- 172.21.15.107:0/171726796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf40073a0 0x7f6d08147670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/171726796 >> 172.21.15.107:0/171726796 conn(0x7f6d080c2950 msgr2=0x7f6d0806a640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:52.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/171726796 shutdown_connections 2023-10-24T23:58:52.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:52.785+0000 7f6d0dd0d640 1 -- 172.21.15.107:0/171726796 wait complete. 2023-10-24T23:58:52.807 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:58:53.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.641+0000 7fb9031c7640 1 Processor -- start 2023-10-24T23:58:53.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.641+0000 7fb9031c7640 1 -- start start 2023-10-24T23:58:53.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb9031c7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb8fc150930 0x7fb8fc0a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:53.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb9031c7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc1510c0 0x7fb8fc151550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb9031c7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8fc0a9b90 con 0x7fb8fc0c1570 2023-10-24T23:58:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb9031c7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8fc0c1390 con 0x7fb8fc150930 2023-10-24T23:58:53.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb9031c7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc154dd0 con 0x7fb8fc1510c0 2023-10-24T23:58:53.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb8fc0c1570 0x7fb8fc069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56116/0 (socket says 172.21.15.107:56116) 2023-10-24T23:58:53.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fbfff640 1 -- 172.21.15.107:0/2678129731 learned_addr learned my addr 172.21.15.107:0/2678129731 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:53.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb901764640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc1510c0 0x7fb8fc151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:53.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb900f63640 1 --2- 172.21.15.107:0/2678129731 >> v2:172.21.15.110:3300/0 conn(0x7fb8fc150930 0x7fb8fc0a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:53.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120838632 0 0) 0x7fb8fc0a9b90 con 0x7fb8fc0c1570 2023-10-24T23:58:53.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8ec003530 con 0x7fb8fc0c1570 2023-10-24T23:58:53.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 448418543 0 0) 0x7fb8ec003530 con 0x7fb8fc0c1570 2023-10-24T23:58:53.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc1510c0 msgr2=0x7fb8fc151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:53.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc1510c0 0x7fb8fc151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:53.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 >> v2:172.21.15.110:3300/0 conn(0x7fb8fc150930 msgr2=0x7fb8fc0a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:53.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 --2- 172.21.15.107:0/2678129731 >> v2:172.21.15.110:3300/0 conn(0x7fb8fc150930 0x7fb8fc0a94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:53.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc1550b0 con 0x7fb8fc0c1570 2023-10-24T23:58:53.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb8e8001680 con 0x7fb8fc0c1570 2023-10-24T23:58:53.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 >> v1:172.21.15.107:6789/0 conn(0x7fb8fc0c1570 legacy=0x7fb8fc069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:53.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8ec003980 0x7fb8ec003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:53.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc1510c0 0x7fb8ec0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:53.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec004d10 0x7fb8ec005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:53.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0c1390 con 0x7fb8ec004d10 2023-10-24T23:58:53.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc154dd0 con 0x7fb8ec003980 2023-10-24T23:58:53.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec00b990 con 0x7fb8fc1510c0 2023-10-24T23:58:53.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb901764640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec004d10 0x7fb8ec005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:53.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb900f63640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8ec003980 0x7fb8ec003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:53.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fbfff640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc1510c0 0x7fb8ec0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:53.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb901764640 1 -- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc1510c0 msgr2=0x7fb8ec0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:53.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb901764640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc1510c0 0x7fb8ec0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:53.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb901764640 1 -- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8ec003980 msgr2=0x7fb8ec003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:53.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb901764640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8ec003980 0x7fb8ec003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:53.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb901764640 1 -- 172.21.15.107:0/2678129731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec003530 con 0x7fb8ec004d10 2023-10-24T23:58:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.645+0000 7fb8fbfff640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc1510c0 0x7fb8ec0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:53.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb901764640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec004d10 0x7fb8ec005190 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009fc0 tx=0x7fb8f401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926eb2cc7980b65c server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f40a5b60 con 0x7fb8ec004d10 2023-10-24T23:58:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f400f040 con 0x7fb8ec004d10 2023-10-24T23:58:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fb7fe640 1 -- 172.21.15.107:0/2678129731 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f40ad440 con 0x7fb8ec004d10 2023-10-24T23:58:53.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 -- 172.21.15.107:0/2678129731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec004d10 msgr2=0x7fb8ec005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 --2- 172.21.15.107:0/2678129731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8ec004d10 0x7fb8ec005190 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009fc0 tx=0x7fb8f401cc60 comp rx=0 tx=0).stop 2023-10-24T23:58:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb901764640 1 -- 172.21.15.107:0/2678129731 reap_dead start 2023-10-24T23:58:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 -- 172.21.15.107:0/2678129731 shutdown_connections 2023-10-24T23:58:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 -- 172.21.15.107:0/2678129731 >> 172.21.15.107:0/2678129731 conn(0x7fb8fc0c65c0 msgr2=0x7fb8fc0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:53.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 -- 172.21.15.107:0/2678129731 shutdown_connections 2023-10-24T23:58:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 -- 172.21.15.107:0/2678129731 wait complete. 2023-10-24T23:58:53.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 Processor -- start 2023-10-24T23:58:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 -- start start 2023-10-24T23:58:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8ec004d10 0x7fb8fc0a74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8fc14acc0 0x7fb8fc0bb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:53.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc0bbc90 0x7fb8fc0708a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:53.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0724d0 con 0x7fb8fc14acc0 2023-10-24T23:58:53.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0ba4b0 con 0x7fb8ec004d10 2023-10-24T23:58:53.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb9031c7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc1e7a20 con 0x7fb8fc0bbc90 2023-10-24T23:58:53.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8fc14acc0 0x7fb8fc0bb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:53.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8fc14acc0 0x7fb8fc0bb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37998/0 (socket says 172.21.15.107:37998) 2023-10-24T23:58:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fbfff640 1 -- 172.21.15.107:0/2159373633 learned_addr learned my addr 172.21.15.107:0/2159373633 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb901764640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc0bbc90 0x7fb8fc0708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb900f63640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8ec004d10 0x7fb8fc0a74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fbfff640 1 -- 172.21.15.107:0/2159373633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc0bbc90 msgr2=0x7fb8fc0708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:53.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fbfff640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc0bbc90 0x7fb8fc0708a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fbfff640 1 -- 172.21.15.107:0/2159373633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8ec004d10 msgr2=0x7fb8fc0a74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fbfff640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8ec004d10 0x7fb8fc0a74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fbfff640 1 -- 172.21.15.107:0/2159373633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc1e7d00 con 0x7fb8fc14acc0 2023-10-24T23:58:53.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb901764640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc0bbc90 0x7fb8fc0708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8fbfff640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8fc14acc0 0x7fb8fc0bb750 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8000c10 tx=0x7fb8e800e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8f97fa640 1 -- 172.21.15.107:0/2159373633 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8010e30 con 0x7fb8fc14acc0 2023-10-24T23:58:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.649+0000 7fb8f97fa640 1 -- 172.21.15.107:0/2159373633 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e80143a0 con 0x7fb8fc14acc0 2023-10-24T23:58:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.653+0000 7fb9031c7640 1 -- 172.21.15.107:0/2159373633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8fc1e7fe0 con 0x7fb8fc14acc0 2023-10-24T23:58:53.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.653+0000 7fb8f97fa640 1 -- 172.21.15.107:0/2159373633 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8014710 con 0x7fb8fc14acc0 2023-10-24T23:58:53.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.653+0000 7fb9031c7640 1 -- 172.21.15.107:0/2159373633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8fc0bc110 con 0x7fb8fc14acc0 2023-10-24T23:58:53.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.653+0000 7fb9031c7640 1 -- 172.21.15.107:0/2159373633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8fc0746e0 con 0x7fb8fc14acc0 2023-10-24T23:58:53.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.653+0000 7fb8f97fa640 1 -- 172.21.15.107:0/2159373633 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb8e8014870 con 0x7fb8fc14acc0 2023-10-24T23:58:53.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.653+0000 7fb8f97fa640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8e0065e60 0x7fb8e0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:53.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.653+0000 7fb8f97fa640 1 -- 172.21.15.107:0/2159373633 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 7735+0+0 (secure 0 0 0) 0x7fb8e8086f30 con 0x7fb8fc14acc0 2023-10-24T23:58:53.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.653+0000 7fb900f63640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8e0065e60 0x7fb8e0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:53.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.653+0000 7fb900f63640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8e0065e60 0x7fb8e0068320 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc150350 tx=0x7fb8f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:53.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.657+0000 7fb8f97fa640 1 -- 172.21.15.107:0/2159373633 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb8e8057a10 con 0x7fb8fc14acc0 2023-10-24T23:58:53.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 -- 172.21.15.107:0/2159373633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8fc0c0f40 con 0x7fb8fc14acc0 2023-10-24T23:58:53.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb8f97fa640 1 -- 172.21.15.107:0/2159373633 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5016 (secure 0 0 0) 0x7fb8e801b020 con 0x7fb8fc14acc0 2023-10-24T23:58:53.932 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:53.932 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:53.437912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 51 2023-10-24T23:58:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 -- 172.21.15.107:0/2159373633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8e0065e60 msgr2=0x7fb8e0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:53.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8e0065e60 0x7fb8e0068320 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc150350 tx=0x7fb8f0008040 comp rx=0 tx=0).stop 2023-10-24T23:58:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 -- 172.21.15.107:0/2159373633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8fc14acc0 msgr2=0x7fb8fc0bb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8fc14acc0 0x7fb8fc0bb750 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8000c10 tx=0x7fb8e800e4d0 comp rx=0 tx=0).stop 2023-10-24T23:58:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 -- 172.21.15.107:0/2159373633 shutdown_connections 2023-10-24T23:58:53.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8e0065e60 0x7fb8e0068320 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8fc0bbc90 0x7fb8fc0708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8fc14acc0 0x7fb8fc0bb750 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 --2- 172.21.15.107:0/2159373633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8ec004d10 0x7fb8fc0a74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 -- 172.21.15.107:0/2159373633 >> 172.21.15.107:0/2159373633 conn(0x7fb8fc0c65c0 msgr2=0x7fb8fc0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 -- 172.21.15.107:0/2159373633 shutdown_connections 2023-10-24T23:58:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:53.845+0000 7fb9031c7640 1 -- 172.21.15.107:0/2159373633 wait complete. 2023-10-24T23:58:53.937 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2023-10-24T23:58:53.937 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-24T23:58:53.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:54.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c96d4640 1 Processor -- start 2023-10-24T23:58:54.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c96d4640 1 -- start start 2023-10-24T23:58:54.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c96d4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb3c414b600 0x7fb3c4151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c96d4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb3c406af70 0x7fb3c406c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c96d4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3c40c1860 con 0x7fb3c406a230 2023-10-24T23:58:54.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c96d4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c4154d50 con 0x7fb3c414b600 2023-10-24T23:58:54.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c96d4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3c41524f0 con 0x7fb3c406af70 2023-10-24T23:58:54.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb3c414b600 0x7fb3c4151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb3c414b600 0x7fb3c4151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38008/0 (socket says 172.21.15.107:38008) 2023-10-24T23:58:54.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c2ffd640 1 -- 172.21.15.107:0/1994699507 learned_addr learned my addr 172.21.15.107:0/1994699507 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:54.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c2ffd640 1 -- 172.21.15.107:0/1994699507 >> v2:172.21.15.110:3300/0 conn(0x7fb3c406af70 msgr2=0x7fb3c406c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:54.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c27fc640 1 --2- 172.21.15.107:0/1994699507 >> v2:172.21.15.110:3300/0 conn(0x7fb3c406af70 0x7fb3c406c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c2ffd640 1 --2- 172.21.15.107:0/1994699507 >> v2:172.21.15.110:3300/0 conn(0x7fb3c406af70 0x7fb3c406c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c2ffd640 1 -- 172.21.15.107:0/1994699507 >> v1:172.21.15.110:6789/0 conn(0x7fb3c406a230 legacy=0x7fb3c41551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:54.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c2ffd640 1 -- 172.21.15.107:0/1994699507 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c414b9e0 con 0x7fb3c414b600 2023-10-24T23:58:54.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.049+0000 7fb3c27fc640 1 --2- 172.21.15.107:0/1994699507 >> v2:172.21.15.110:3300/0 conn(0x7fb3c406af70 0x7fb3c406c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:54.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c2ffd640 1 --2- 172.21.15.107:0/1994699507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c414b600 0x7fb3c4151fb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb3c40c2870 tx=0x7fb3b401ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86da6117bad16d29 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:54.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c1ffb640 1 -- 172.21.15.107:0/1994699507 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b40a7aa0 con 0x7fb3c414b600 2023-10-24T23:58:54.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c1ffb640 1 -- 172.21.15.107:0/1994699507 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b40a7c00 con 0x7fb3c414b600 2023-10-24T23:58:54.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c1ffb640 1 -- 172.21.15.107:0/1994699507 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b40af440 con 0x7fb3c414b600 2023-10-24T23:58:54.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 -- 172.21.15.107:0/1994699507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c414b600 msgr2=0x7fb3c4151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:54.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 --2- 172.21.15.107:0/1994699507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c414b600 0x7fb3c4151fb0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb3c40c2870 tx=0x7fb3b401ead0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 -- 172.21.15.107:0/1994699507 shutdown_connections 2023-10-24T23:58:54.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 --2- 172.21.15.107:0/1994699507 >> v2:172.21.15.110:3300/0 conn(0x7fb3c406af70 0x7fb3c406c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 --2- 172.21.15.107:0/1994699507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c414b600 0x7fb3c4151fb0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 -- 172.21.15.107:0/1994699507 >> 172.21.15.107:0/1994699507 conn(0x7fb3c40c2950 msgr2=0x7fb3c40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:54.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 -- 172.21.15.107:0/1994699507 shutdown_connections 2023-10-24T23:58:54.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 -- 172.21.15.107:0/1994699507 wait complete. 2023-10-24T23:58:54.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 Processor -- start 2023-10-24T23:58:54.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 -- start start 2023-10-24T23:58:54.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c406a230 0x7fb3c4147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c406af70 0x7fb3c41416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c414b600 0x7fb3c4141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c41ec940 con 0x7fb3c414b600 2023-10-24T23:58:54.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c41eba50 con 0x7fb3c406a230 2023-10-24T23:58:54.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c96d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c41ebd30 con 0x7fb3c406af70 2023-10-24T23:58:54.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c414b600 0x7fb3c4141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c414b600 0x7fb3c4141c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38022/0 (socket says 172.21.15.107:38022) 2023-10-24T23:58:54.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c37fe640 1 -- 172.21.15.107:0/2438203550 learned_addr learned my addr 172.21.15.107:0/2438203550 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:54.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c2ffd640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c406a230 0x7fb3c4147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c27fc640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c406af70 0x7fb3c41416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c37fe640 1 -- 172.21.15.107:0/2438203550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c406af70 msgr2=0x7fb3c41416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c37fe640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c406af70 0x7fb3c41416d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c37fe640 1 -- 172.21.15.107:0/2438203550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c406a230 msgr2=0x7fb3c4147330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c37fe640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c406a230 0x7fb3c4147330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.053+0000 7fb3c37fe640 1 -- 172.21.15.107:0/2438203550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b400b770 con 0x7fb3c414b600 2023-10-24T23:58:54.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3c2ffd640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c406a230 0x7fb3c4147330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:58:54.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3c37fe640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c414b600 0x7fb3c4141c10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00c780 tx=0x7fb3bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:54.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3ab7fe640 1 -- 172.21.15.107:0/2438203550 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc00a9f0 con 0x7fb3c414b600 2023-10-24T23:58:54.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3ab7fe640 1 -- 172.21.15.107:0/2438203550 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc00ce90 con 0x7fb3c414b600 2023-10-24T23:58:54.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3c96d4640 1 -- 172.21.15.107:0/2438203550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c41ec010 con 0x7fb3c414b600 2023-10-24T23:58:54.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3ab7fe640 1 -- 172.21.15.107:0/2438203550 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc00add0 con 0x7fb3c414b600 2023-10-24T23:58:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3c96d4640 1 -- 172.21.15.107:0/2438203550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c41ec5d0 con 0x7fb3c414b600 2023-10-24T23:58:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3c96d4640 1 -- 172.21.15.107:0/2438203550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c41e8530 con 0x7fb3c414b600 2023-10-24T23:58:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3ab7fe640 1 -- 172.21.15.107:0/2438203550 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb3bc012420 con 0x7fb3c414b600 2023-10-24T23:58:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3ab7fe640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb3b0065e60 0x7fb3b0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3ab7fe640 1 -- 172.21.15.107:0/2438203550 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 7735+0+0 (secure 0 0 0) 0x7fb3bc0843b0 con 0x7fb3c414b600 2023-10-24T23:58:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3c2ffd640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb3b0065e60 0x7fb3b0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.057+0000 7fb3c2ffd640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb3b0065e60 0x7fb3b0068320 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb3b40b1000 tx=0x7fb3b40a9070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:54.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.061+0000 7fb3ab7fe640 1 -- 172.21.15.107:0/2438203550 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb3bc051370 con 0x7fb3c414b600 2023-10-24T23:58:54.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.245+0000 7fb3c96d4640 1 -- 172.21.15.107:0/2438203550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3c40746e0 con 0x7fb3c414b600 2023-10-24T23:58:54.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3ab7fe640 1 -- 172.21.15.107:0/2438203550 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+5016 (secure 0 0 0) 0x7fb3bc054990 con 0x7fb3c414b600 2023-10-24T23:58:54.253 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:54.253 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:53.437912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 51 2023-10-24T23:58:54.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 -- 172.21.15.107:0/2438203550 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb3b0065e60 msgr2=0x7fb3b0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:54.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb3b0065e60 0x7fb3b0068320 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb3b40b1000 tx=0x7fb3b40a9070 comp rx=0 tx=0).stop 2023-10-24T23:58:54.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 -- 172.21.15.107:0/2438203550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c414b600 msgr2=0x7fb3c4141c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:54.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c414b600 0x7fb3c4141c10 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00c780 tx=0x7fb3bc00cc40 comp rx=0 tx=0).stop 2023-10-24T23:58:54.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 -- 172.21.15.107:0/2438203550 shutdown_connections 2023-10-24T23:58:54.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3c414b600 0x7fb3c4141c10 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb3b0065e60 0x7fb3b0068320 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb3c406af70 0x7fb3c41416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 --2- 172.21.15.107:0/2438203550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb3c406a230 0x7fb3c4147330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 -- 172.21.15.107:0/2438203550 >> 172.21.15.107:0/2438203550 conn(0x7fb3c40c2950 msgr2=0x7fb3c40aadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:54.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 -- 172.21.15.107:0/2438203550 shutdown_connections 2023-10-24T23:58:54.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.249+0000 7fb3c96d4640 1 -- 172.21.15.107:0/2438203550 wait complete. 2023-10-24T23:58:54.271 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 51, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5751": { "addr": "172.21.15.107:6835/1508236506", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 1508236506, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 1508236506, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5751, "incarnation": 44, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:58:07.539380+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5751 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 51, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5925": { "addr": "172.21.15.107:6837/1292188015", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 1292188015, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 1292188015, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5925, "incarnation": 51, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-24T23:58:53.437912+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5925 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/1101018338", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 1101018338, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 1101018338, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 50, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5293, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-24T23:58:54.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:54.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa66745a640 1 Processor -- start 2023-10-24T23:58:54.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa66745a640 1 -- start start 2023-10-24T23:58:54.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa66745a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa66014b600 0x7fa660151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa66745a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66006a230 0x7fa6601551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa66745a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6600c1860 con 0x7fa66006af70 2023-10-24T23:58:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa66745a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa660154d50 con 0x7fa66014b600 2023-10-24T23:58:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa66745a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6601524f0 con 0x7fa66006a230 2023-10-24T23:58:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa6651f6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa66014b600 0x7fa660151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa6649f5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa66006af70 0x7fa66006c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56132/0 (socket says 172.21.15.107:56132) 2023-10-24T23:58:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa6649f5640 1 -- 172.21.15.107:0/2002746269 learned_addr learned my addr 172.21.15.107:0/2002746269 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:54.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa6659f7640 1 --2- 172.21.15.107:0/2002746269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66006a230 0x7fa6601551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa6651f6640 1 -- 172.21.15.107:0/2002746269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66006a230 msgr2=0x7fa6601551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:54.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa6651f6640 1 --2- 172.21.15.107:0/2002746269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66006a230 0x7fa6601551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa6651f6640 1 -- 172.21.15.107:0/2002746269 >> v1:172.21.15.107:6789/0 conn(0x7fa66006af70 legacy=0x7fa66006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:54.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa6651f6640 1 -- 172.21.15.107:0/2002746269 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66014b9e0 con 0x7fa66014b600 2023-10-24T23:58:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.933+0000 7fa657fff640 1 -- 172.21.15.107:0/2002746269 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32770773 0 0) 0x7fa6600c1860 con 0x7fa66006af70 2023-10-24T23:58:54.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6651f6640 1 --2- 172.21.15.107:0/2002746269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66014b600 0x7fa660151fb0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009d90 tx=0x7fa65c01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7244a14a26613ad8 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:54.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa657fff640 1 -- 172.21.15.107:0/2002746269 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0a5860 con 0x7fa66014b600 2023-10-24T23:58:54.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa657fff640 1 -- 172.21.15.107:0/2002746269 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c0a59c0 con 0x7fa66014b600 2023-10-24T23:58:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa657fff640 1 -- 172.21.15.107:0/2002746269 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0a5d50 con 0x7fa66014b600 2023-10-24T23:58:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 -- 172.21.15.107:0/2002746269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66014b600 msgr2=0x7fa660151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 --2- 172.21.15.107:0/2002746269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66014b600 0x7fa660151fb0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009d90 tx=0x7fa65c01c890 comp rx=0 tx=0).stop 2023-10-24T23:58:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 -- 172.21.15.107:0/2002746269 shutdown_connections 2023-10-24T23:58:54.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 --2- 172.21.15.107:0/2002746269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66006a230 0x7fa6601551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 --2- 172.21.15.107:0/2002746269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66014b600 0x7fa660151fb0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 -- 172.21.15.107:0/2002746269 >> 172.21.15.107:0/2002746269 conn(0x7fa6600c2950 msgr2=0x7fa6600c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:54.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 -- 172.21.15.107:0/2002746269 shutdown_connections 2023-10-24T23:58:54.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 -- 172.21.15.107:0/2002746269 wait complete. 2023-10-24T23:58:54.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 Processor -- start 2023-10-24T23:58:54.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 -- start start 2023-10-24T23:58:54.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66006a230 0x7fa66014a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66006af70 0x7fa660154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66014b600 0x7fa6601422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa660152920 con 0x7fa66006a230 2023-10-24T23:58:54.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6601f08b0 con 0x7fa66014b600 2023-10-24T23:58:54.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6601f0e10 con 0x7fa66006af70 2023-10-24T23:58:54.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6651f6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66006a230 0x7fa66014a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6651f6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66006a230 0x7fa66014a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38038/0 (socket says 172.21.15.107:38038) 2023-10-24T23:58:54.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6651f6640 1 -- 172.21.15.107:0/706551659 learned_addr learned my addr 172.21.15.107:0/706551659 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:54.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6651f6640 1 -- 172.21.15.107:0/706551659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66006af70 msgr2=0x7fa660154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:54.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6649f5640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66006af70 0x7fa660154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6651f6640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66006af70 0x7fa660154260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6659f7640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66014b600 0x7fa6601422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6651f6640 1 -- 172.21.15.107:0/706551659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66014b600 msgr2=0x7fa6601422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:54.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6651f6640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66014b600 0x7fa6601422a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:54.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6651f6640 1 -- 172.21.15.107:0/706551659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c009a70 con 0x7fa66006a230 2023-10-24T23:58:54.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa6651f6640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66006a230 0x7fa66014a210 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009d90 tx=0x7fa65c019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:54.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa655ffb640 1 -- 172.21.15.107:0/706551659 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c003ba0 con 0x7fa66006a230 2023-10-24T23:58:54.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa655ffb640 1 -- 172.21.15.107:0/706551659 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c003d00 con 0x7fa66006a230 2023-10-24T23:58:54.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 -- 172.21.15.107:0/706551659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa660152b70 con 0x7fa66006a230 2023-10-24T23:58:54.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa655ffb640 1 -- 172.21.15.107:0/706551659 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0ad850 con 0x7fa66006a230 2023-10-24T23:58:54.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.937+0000 7fa66745a640 1 -- 172.21.15.107:0/706551659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa66014a750 con 0x7fa66006a230 2023-10-24T23:58:54.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.941+0000 7fa66745a640 1 -- 172.21.15.107:0/706551659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa660152db0 con 0x7fa66006a230 2023-10-24T23:58:54.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.941+0000 7fa655ffb640 1 -- 172.21.15.107:0/706551659 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa65c0b4070 con 0x7fa66006a230 2023-10-24T23:58:54.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.941+0000 7fa655ffb640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa644065e60 0x7fa644068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:54.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.941+0000 7fa655ffb640 1 -- 172.21.15.107:0/706551659 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 7735+0+0 (secure 0 0 0) 0x7fa65c11e940 con 0x7fa66006a230 2023-10-24T23:58:54.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.941+0000 7fa6649f5640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa644065e60 0x7fa644068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:54.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.941+0000 7fa6649f5640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa644065e60 0x7fa644068320 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa6600c1690 tx=0x7fa650009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:54.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:54.945+0000 7fa655ffb640 1 -- 172.21.15.107:0/706551659 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa65c0ebab0 con 0x7fa66006a230 2023-10-24T23:58:55.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.129+0000 7fa66745a640 1 -- 172.21.15.107:0/706551659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6600746e0 con 0x7fa66006a230 2023-10-24T23:58:55.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa655ffb640 1 -- 172.21.15.107:0/706551659 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+5020 (secure 0 0 0) 0x7fa65c0b2070 con 0x7fa66006a230 2023-10-24T23:58:55.137 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:55.137 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:54.568019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:55.137 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 52 2023-10-24T23:58:55.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 -- 172.21.15.107:0/706551659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa644065e60 msgr2=0x7fa644068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa644065e60 0x7fa644068320 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa6600c1690 tx=0x7fa650009040 comp rx=0 tx=0).stop 2023-10-24T23:58:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 -- 172.21.15.107:0/706551659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66006a230 msgr2=0x7fa66014a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66006a230 0x7fa66014a210 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009d90 tx=0x7fa65c019630 comp rx=0 tx=0).stop 2023-10-24T23:58:55.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 -- 172.21.15.107:0/706551659 shutdown_connections 2023-10-24T23:58:55.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66014b600 0x7fa6601422a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:55.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa644065e60 0x7fa644068320 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:55.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66006af70 0x7fa660154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:55.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 --2- 172.21.15.107:0/706551659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66006a230 0x7fa66014a210 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:55.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 -- 172.21.15.107:0/706551659 >> 172.21.15.107:0/706551659 conn(0x7fa6600c2950 msgr2=0x7fa6600a9450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:55.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 -- 172.21.15.107:0/706551659 shutdown_connections 2023-10-24T23:58:55.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:55.133+0000 7fa66745a640 1 -- 172.21.15.107:0/706551659 wait complete. 2023-10-24T23:58:55.155 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:58:56.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:56.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd43aea4640 1 Processor -- start 2023-10-24T23:58:56.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd43aea4640 1 -- start start 2023-10-24T23:58:56.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd43aea4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd43414fd40 0x7fd4340ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:56.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd43aea4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd43406ee40 0x7fd4340aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:56.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd43aea4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd43406f730 con 0x7fd4340a9820 2023-10-24T23:58:56.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd43aea4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4340aa750 con 0x7fd43406ee40 2023-10-24T23:58:56.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd43aea4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd434156a40 con 0x7fd43414fd40 2023-10-24T23:58:56.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd433fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd4340a9820 0x7fd4340c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56146/0 (socket says 172.21.15.107:56146) 2023-10-24T23:58:56.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd433fff640 1 -- 172.21.15.107:0/2806429430 learned_addr learned my addr 172.21.15.107:0/2806429430 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:56.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd438c40640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd43414fd40 0x7fd4340ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:56.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd439441640 1 --2- 172.21.15.107:0/2806429430 >> v2:172.21.15.110:3300/0 conn(0x7fd43406ee40 0x7fd4340aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:56.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004463437 0 0) 0x7fd43406f730 con 0x7fd4340a9820 2023-10-24T23:58:56.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd424003530 con 0x7fd4340a9820 2023-10-24T23:58:56.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2984316530 0 0) 0x7fd424003530 con 0x7fd4340a9820 2023-10-24T23:58:56.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd43414fd40 msgr2=0x7fd4340ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:56.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd43414fd40 0x7fd4340ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:56.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 >> v2:172.21.15.110:3300/0 conn(0x7fd43406ee40 msgr2=0x7fd4340aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:56.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 --2- 172.21.15.107:0/2806429430 >> v2:172.21.15.110:3300/0 conn(0x7fd43406ee40 0x7fd4340aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:56.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd434156710 con 0x7fd4340a9820 2023-10-24T23:58:56.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fd420001680 con 0x7fd4340a9820 2023-10-24T23:58:56.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 >> v1:172.21.15.107:6789/0 conn(0x7fd4340a9820 legacy=0x7fd4340c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:56.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd424003980 0x7fd424003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:56.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.401+0000 7fd4337fe640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd43414fd40 0x7fd4240047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:56.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd4337fe640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd424004d10 0x7fd424005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:56.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4340aa750 con 0x7fd424003980 2023-10-24T23:58:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd434156a40 con 0x7fd424004d10 2023-10-24T23:58:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd42400ba60 con 0x7fd43414fd40 2023-10-24T23:58:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd438c40640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd424003980 0x7fd424003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd439441640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd424004d10 0x7fd424005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd433fff640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd43414fd40 0x7fd4240047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd438c40640 1 -- 172.21.15.107:0/2806429430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd43414fd40 msgr2=0x7fd4240047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd438c40640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd43414fd40 0x7fd4240047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd438c40640 1 -- 172.21.15.107:0/2806429430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd424004d10 msgr2=0x7fd424005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd438c40640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd424004d10 0x7fd424005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd438c40640 1 -- 172.21.15.107:0/2806429430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd424003530 con 0x7fd424003980 2023-10-24T23:58:56.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd438c40640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd424003980 0x7fd424003da0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd428009ae0 tx=0x7fd42801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171d4ccb2cc4aeb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:56.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4280a5590 con 0x7fd424003980 2023-10-24T23:58:56.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4280a56f0 con 0x7fd424003980 2023-10-24T23:58:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd4337fe640 1 -- 172.21.15.107:0/2806429430 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4280a5aa0 con 0x7fd424003980 2023-10-24T23:58:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd43aea4640 1 -- 172.21.15.107:0/2806429430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd424003980 msgr2=0x7fd424003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd43aea4640 1 --2- 172.21.15.107:0/2806429430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd424003980 0x7fd424003da0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd428009ae0 tx=0x7fd42801ca20 comp rx=0 tx=0).stop 2023-10-24T23:58:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd439441640 1 -- 172.21.15.107:0/2806429430 reap_dead start 2023-10-24T23:58:56.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd43aea4640 1 -- 172.21.15.107:0/2806429430 shutdown_connections 2023-10-24T23:58:56.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd43aea4640 1 -- 172.21.15.107:0/2806429430 >> 172.21.15.107:0/2806429430 conn(0x7fd4340c2950 msgr2=0x7fd4340c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:56.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd43aea4640 1 -- 172.21.15.107:0/2806429430 shutdown_connections 2023-10-24T23:58:56.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.405+0000 7fd43aea4640 1 -- 172.21.15.107:0/2806429430 wait complete. 2023-10-24T23:58:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd43aea4640 1 Processor -- start 2023-10-24T23:58:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd43aea4640 1 -- start start 2023-10-24T23:58:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd43aea4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd424003980 0x7fd4340bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd43aea4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4340c5830 0x7fd4341ebb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd43aea4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340bb8c0 0x7fd4340bbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd43aea4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4340c0390 con 0x7fd4340c5830 2023-10-24T23:58:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd43aea4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4340bd8b0 con 0x7fd4340bb8c0 2023-10-24T23:58:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd43aea4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4340bfc40 con 0x7fd424003980 2023-10-24T23:58:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd433fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4340c5830 0x7fd4341ebb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd433fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4340c5830 0x7fd4341ebb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38056/0 (socket says 172.21.15.107:38056) 2023-10-24T23:58:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd433fff640 1 -- 172.21.15.107:0/1348471288 learned_addr learned my addr 172.21.15.107:0/1348471288 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:56.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd438c40640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd424003980 0x7fd4340bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:56.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd439441640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340bb8c0 0x7fd4340bbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:56.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd433fff640 1 -- 172.21.15.107:0/1348471288 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd424003980 msgr2=0x7fd4340bfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:56.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd433fff640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd424003980 0x7fd4340bfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:56.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd433fff640 1 -- 172.21.15.107:0/1348471288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340bb8c0 msgr2=0x7fd4340bbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:56.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd433fff640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340bb8c0 0x7fd4340bbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:56.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd433fff640 1 -- 172.21.15.107:0/1348471288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4340bb470 con 0x7fd4340c5830 2023-10-24T23:58:56.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd438c40640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd424003980 0x7fd4340bfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:58:56.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd433fff640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4340c5830 0x7fd4341ebb00 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd42000e7b0 tx=0x7fd42000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:56.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd4317fa640 1 -- 172.21.15.107:0/1348471288 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42000c6c0 con 0x7fd4340c5830 2023-10-24T23:58:56.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd4317fa640 1 -- 172.21.15.107:0/1348471288 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd42000c840 con 0x7fd4340c5830 2023-10-24T23:58:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd43aea4640 1 -- 172.21.15.107:0/1348471288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4341ec040 con 0x7fd4340c5830 2023-10-24T23:58:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd4317fa640 1 -- 172.21.15.107:0/1348471288 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42000cbf0 con 0x7fd4340c5830 2023-10-24T23:58:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd43aea4640 1 -- 172.21.15.107:0/1348471288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4340bb190 con 0x7fd4340c5830 2023-10-24T23:58:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.409+0000 7fd43aea4640 1 -- 172.21.15.107:0/1348471288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4340c24a0 con 0x7fd4340c5830 2023-10-24T23:58:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.413+0000 7fd4317fa640 1 -- 172.21.15.107:0/1348471288 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd42000cd50 con 0x7fd4340c5830 2023-10-24T23:58:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.413+0000 7fd4317fa640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd418065e60 0x7fd418068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.413+0000 7fd4317fa640 1 -- 172.21.15.107:0/1348471288 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 7735+0+0 (secure 0 0 0) 0x7fd4200847e0 con 0x7fd4340c5830 2023-10-24T23:58:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.413+0000 7fd438c40640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd418065e60 0x7fd418068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.413+0000 7fd438c40640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd418065e60 0x7fd418068320 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd4280af000 tx=0x7fd4280046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.417+0000 7fd4317fa640 1 -- 172.21.15.107:0/1348471288 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd420051880 con 0x7fd4340c5830 2023-10-24T23:58:56.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 -- 172.21.15.107:0/1348471288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd43415bba0 con 0x7fd4340c5830 2023-10-24T23:58:56.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd4317fa640 1 -- 172.21.15.107:0/1348471288 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5017 (secure 0 0 0) 0x7fd43415bba0 con 0x7fd4340c5830 2023-10-24T23:58:56.610 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:56.611 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:56.611 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 54 2023-10-24T23:58:56.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 -- 172.21.15.107:0/1348471288 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd418065e60 msgr2=0x7fd418068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:56.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd418065e60 0x7fd418068320 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd4280af000 tx=0x7fd4280046d0 comp rx=0 tx=0).stop 2023-10-24T23:58:56.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 -- 172.21.15.107:0/1348471288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4340c5830 msgr2=0x7fd4341ebb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:56.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4340c5830 0x7fd4341ebb00 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd42000e7b0 tx=0x7fd42000ec70 comp rx=0 tx=0).stop 2023-10-24T23:58:56.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 -- 172.21.15.107:0/1348471288 shutdown_connections 2023-10-24T23:58:56.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4340bb8c0 0x7fd4340bbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:56.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4340c5830 0x7fd4341ebb00 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:56.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd418065e60 0x7fd418068320 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:56.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 --2- 172.21.15.107:0/1348471288 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd424003980 0x7fd4340bfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:56.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 -- 172.21.15.107:0/1348471288 >> 172.21.15.107:0/1348471288 conn(0x7fd4340c2950 msgr2=0x7fd43415c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:56.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 -- 172.21.15.107:0/1348471288 shutdown_connections 2023-10-24T23:58:56.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:56.605+0000 7fd43aea4640 1 -- 172.21.15.107:0/1348471288 wait complete. 2023-10-24T23:58:56.628 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=4937 2023-10-24T23:58:56.629 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.d 2023-10-24T23:58:57.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:57.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.277+0000 7f36b862a640 1 Processor -- start 2023-10-24T23:58:57.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b862a640 1 -- start start 2023-10-24T23:58:57.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b862a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36b00a9820 0x7f36b00ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:57.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b862a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36b006ee40 0x7f36b00aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:57.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b862a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b006f730 con 0x7f36b014fd40 2023-10-24T23:58:57.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b862a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36b00aa750 con 0x7f36b006ee40 2023-10-24T23:58:57.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b862a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0156a40 con 0x7f36b00a9820 2023-10-24T23:58:57.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b5bc5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36b00a9820 0x7f36b00ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:57.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b6bc7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36b006ee40 0x7f36b00aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:57.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b63c6640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f36b014fd40 0x7f36b00c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:47002/0 (socket says 172.21.15.107:47002) 2023-10-24T23:58:57.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b63c6640 1 -- 172.21.15.107:0/4009853067 learned_addr learned my addr 172.21.15.107:0/4009853067 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:57.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b5bc5640 1 -- 172.21.15.107:0/4009853067 >> v2:172.21.15.110:3300/0 conn(0x7f36b006ee40 msgr2=0x7f36b00aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:57.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b5bc5640 1 --2- 172.21.15.107:0/4009853067 >> v2:172.21.15.110:3300/0 conn(0x7f36b006ee40 0x7f36b00aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:57.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b5bc5640 1 -- 172.21.15.107:0/4009853067 >> v1:172.21.15.110:6789/0 conn(0x7f36b014fd40 legacy=0x7f36b00c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:57.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b5bc5640 1 -- 172.21.15.107:0/4009853067 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b0156710 con 0x7f36b00a9820 2023-10-24T23:58:57.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.281+0000 7f36b6bc7640 1 --2- 172.21.15.107:0/4009853067 >> v2:172.21.15.110:3300/0 conn(0x7f36b006ee40 0x7f36b00aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-24T23:58:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b5bc5640 1 --2- 172.21.15.107:0/4009853067 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36b00a9820 0x7f36b00ab090 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f36ac0097c0 tx=0x7f36ac01c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d93ef7a884f23f89 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b53c4640 1 -- 172.21.15.107:0/4009853067 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac0ab600 con 0x7f36b00a9820 2023-10-24T23:58:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b53c4640 1 -- 172.21.15.107:0/4009853067 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac0a7050 con 0x7f36b00a9820 2023-10-24T23:58:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b53c4640 1 -- 172.21.15.107:0/4009853067 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac0ab980 con 0x7f36b00a9820 2023-10-24T23:58:57.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 -- 172.21.15.107:0/4009853067 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36b00a9820 msgr2=0x7f36b00ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:57.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 --2- 172.21.15.107:0/4009853067 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36b00a9820 0x7f36b00ab090 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f36ac0097c0 tx=0x7f36ac01c330 comp rx=0 tx=0).stop 2023-10-24T23:58:57.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 -- 172.21.15.107:0/4009853067 shutdown_connections 2023-10-24T23:58:57.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 --2- 172.21.15.107:0/4009853067 >> v2:172.21.15.110:3300/0 conn(0x7f36b006ee40 0x7f36b00aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:57.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 --2- 172.21.15.107:0/4009853067 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36b00a9820 0x7f36b00ab090 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:57.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 -- 172.21.15.107:0/4009853067 >> 172.21.15.107:0/4009853067 conn(0x7f36b00c2950 msgr2=0x7f36b00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:57.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 -- 172.21.15.107:0/4009853067 shutdown_connections 2023-10-24T23:58:57.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 -- 172.21.15.107:0/4009853067 wait complete. 2023-10-24T23:58:57.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 Processor -- start 2023-10-24T23:58:57.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 -- start start 2023-10-24T23:58:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b006ee40 0x7f36b0144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b014fd40 0x7f36b014b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36b0155b10 0x7f36b014db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0146360 con 0x7f36b006ee40 2023-10-24T23:58:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0145b50 con 0x7f36b014fd40 2023-10-24T23:58:57.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b862a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b014d6e0 con 0x7f36b0155b10 2023-10-24T23:58:57.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b63c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b006ee40 0x7f36b0144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:57.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b63c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b006ee40 0x7f36b0144770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38064/0 (socket says 172.21.15.107:38064) 2023-10-24T23:58:57.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b63c6640 1 -- 172.21.15.107:0/3436907106 learned_addr learned my addr 172.21.15.107:0/3436907106 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:57.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b6bc7640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36b0155b10 0x7f36b014db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b5bc5640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b014fd40 0x7f36b014b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b63c6640 1 -- 172.21.15.107:0/3436907106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36b0155b10 msgr2=0x7f36b014db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b63c6640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36b0155b10 0x7f36b014db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b63c6640 1 -- 172.21.15.107:0/3436907106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b014fd40 msgr2=0x7f36b014b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:57.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b63c6640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b014fd40 0x7f36b014b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:57.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b63c6640 1 -- 172.21.15.107:0/3436907106 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a8009a70 con 0x7f36b006ee40 2023-10-24T23:58:57.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.285+0000 7f36b5bc5640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b014fd40 0x7f36b014b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:58:57.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36b63c6640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b006ee40 0x7f36b0144770 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f36a000c450 tx=0x7f36a000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:57.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36a6ffd640 1 -- 172.21.15.107:0/3436907106 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a0005cf0 con 0x7f36b006ee40 2023-10-24T23:58:57.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36a6ffd640 1 -- 172.21.15.107:0/3436907106 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a0005e50 con 0x7f36b006ee40 2023-10-24T23:58:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36a6ffd640 1 -- 172.21.15.107:0/3436907106 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00072f0 con 0x7f36b006ee40 2023-10-24T23:58:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36b862a640 1 -- 172.21.15.107:0/3436907106 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ac009420 con 0x7f36b006ee40 2023-10-24T23:58:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36b862a640 1 -- 172.21.15.107:0/3436907106 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b0146080 con 0x7f36b006ee40 2023-10-24T23:58:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36b862a640 1 -- 172.21.15.107:0/3436907106 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b00c24a0 con 0x7f36b006ee40 2023-10-24T23:58:57.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36a6ffd640 1 -- 172.21.15.107:0/3436907106 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f36a0007450 con 0x7f36b006ee40 2023-10-24T23:58:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36a6ffd640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065e60 0x7f3694068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36a6ffd640 1 -- 172.21.15.107:0/3436907106 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 7735+0+0 (secure 0 0 0) 0x7f36a0083340 con 0x7f36b006ee40 2023-10-24T23:58:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36b5bc5640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065e60 0x7f3694068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:57.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.289+0000 7f36b5bc5640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065e60 0x7f3694068320 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f36ac0ad000 tx=0x7f36ac0a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:57.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.293+0000 7f36a6ffd640 1 -- 172.21.15.107:0/3436907106 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f36a0050390 con 0x7f36b006ee40 2023-10-24T23:58:57.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.481+0000 7f36b862a640 1 -- 172.21.15.107:0/3436907106 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36b0070840 con 0x7f36b006ee40 2023-10-24T23:58:57.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.481+0000 7f36a6ffd640 1 -- 172.21.15.107:0/3436907106 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+5017 (secure 0 0 0) 0x7f36a00539b0 con 0x7f36b006ee40 2023-10-24T23:58:57.488 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:57.488 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:07.539380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5751},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5751":{"gid":5751,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/1508236506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1508236506},{"type":"v1","addr":"172.21.15.107:6835","nonce":1508236506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:57.489 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 54 2023-10-24T23:58:57.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 -- 172.21.15.107:0/3436907106 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065e60 msgr2=0x7f3694068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:57.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065e60 0x7f3694068320 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f36ac0ad000 tx=0x7f36ac0a6000 comp rx=0 tx=0).stop 2023-10-24T23:58:57.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 -- 172.21.15.107:0/3436907106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b006ee40 msgr2=0x7f36b0144770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:57.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b006ee40 0x7f36b0144770 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f36a000c450 tx=0x7f36a000c850 comp rx=0 tx=0).stop 2023-10-24T23:58:57.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 -- 172.21.15.107:0/3436907106 shutdown_connections 2023-10-24T23:58:57.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065e60 0x7f3694068320 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:57.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36b0155b10 0x7f36b014db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:57.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b014fd40 0x7f36b014b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:57.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 --2- 172.21.15.107:0/3436907106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b006ee40 0x7f36b0144770 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:57.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 -- 172.21.15.107:0/3436907106 >> 172.21.15.107:0/3436907106 conn(0x7f36b00c2950 msgr2=0x7f36b01501b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:57.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 -- 172.21.15.107:0/3436907106 shutdown_connections 2023-10-24T23:58:57.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:57.485+0000 7f36b862a640 1 -- 172.21.15.107:0/3436907106 wait complete. 2023-10-24T23:58:57.506 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-24T23:58:59.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:59.561 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-24T23:58:59.557+0000 7effef1e5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:58:59.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.637+0000 7f86ab945640 1 Processor -- start 2023-10-24T23:58:59.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.637+0000 7f86ab945640 1 -- start start 2023-10-24T23:58:59.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86a40a9820 0x7f86a40ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:59.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a406ee40 0x7f86a40aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:59.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a406f730 con 0x7f86a414fd40 2023-10-24T23:58:59.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a40aa750 con 0x7f86a40a9820 2023-10-24T23:58:59.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a4156a40 con 0x7f86a406ee40 2023-10-24T23:58:59.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86a8ee0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86a40a9820 0x7f86a40ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:59.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86a96e1640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f86a414fd40 0x7f86a40c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56154/0 (socket says 172.21.15.107:56154) 2023-10-24T23:58:59.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86a96e1640 1 -- 172.21.15.107:0/427434108 learned_addr learned my addr 172.21.15.107:0/427434108 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:59.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86a9ee2640 1 --2- 172.21.15.107:0/427434108 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a406ee40 0x7f86a40aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:59.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86a8ee0640 1 -- 172.21.15.107:0/427434108 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a406ee40 msgr2=0x7f86a40aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:59.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86a8ee0640 1 --2- 172.21.15.107:0/427434108 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a406ee40 0x7f86a40aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86a8ee0640 1 -- 172.21.15.107:0/427434108 >> v1:172.21.15.107:6789/0 conn(0x7f86a414fd40 legacy=0x7f86a40c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:59.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f869bfff640 1 -- 172.21.15.107:0/427434108 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066202896 0 0) 0x7f86a406f730 con 0x7f86a414fd40 2023-10-24T23:58:59.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86a8ee0640 1 -- 172.21.15.107:0/427434108 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a4156710 con 0x7f86a40a9820 2023-10-24T23:58:59.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86a9ee2640 1 --2- 172.21.15.107:0/427434108 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a406ee40 0x7f86a40aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:58:59.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86a8ee0640 1 --2- 172.21.15.107:0/427434108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a40a9820 0x7f86a40ab090 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f86a00097c0 tx=0x7f86a001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37853011344da48 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:59.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f869bfff640 1 -- 172.21.15.107:0/427434108 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a00ab600 con 0x7f86a40a9820 2023-10-24T23:58:59.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f869bfff640 1 -- 172.21.15.107:0/427434108 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a00a7050 con 0x7f86a40a9820 2023-10-24T23:58:59.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f869bfff640 1 -- 172.21.15.107:0/427434108 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a00ab980 con 0x7f86a40a9820 2023-10-24T23:58:59.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 -- 172.21.15.107:0/427434108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a40a9820 msgr2=0x7f86a40ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:59.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 --2- 172.21.15.107:0/427434108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a40a9820 0x7f86a40ab090 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f86a00097c0 tx=0x7f86a001c330 comp rx=0 tx=0).stop 2023-10-24T23:58:59.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 -- 172.21.15.107:0/427434108 shutdown_connections 2023-10-24T23:58:59.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 --2- 172.21.15.107:0/427434108 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a406ee40 0x7f86a40aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 --2- 172.21.15.107:0/427434108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a40a9820 0x7f86a40ab090 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 -- 172.21.15.107:0/427434108 >> 172.21.15.107:0/427434108 conn(0x7f86a40c2950 msgr2=0x7f86a40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:59.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 -- 172.21.15.107:0/427434108 shutdown_connections 2023-10-24T23:58:59.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.641+0000 7f86ab945640 1 -- 172.21.15.107:0/427434108 wait complete. 2023-10-24T23:58:59.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86ab945640 1 Processor -- start 2023-10-24T23:58:59.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86ab945640 1 -- start start 2023-10-24T23:58:59.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86ab945640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86a406ee40 0x7f86a415bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:59.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86ab945640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a414fd40 0x7f86a414bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:59.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86ab945640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a41eb3d0 0x7f86a4147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:59.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86ab945640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a41eb940 con 0x7f86a414fd40 2023-10-24T23:58:59.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86ab945640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a414ca80 con 0x7f86a406ee40 2023-10-24T23:58:59.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86ab945640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a40c5270 con 0x7f86a41eb3d0 2023-10-24T23:58:59.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a8ee0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a414fd40 0x7f86a414bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:59.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a8ee0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a414fd40 0x7f86a414bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38072/0 (socket says 172.21.15.107:38072) 2023-10-24T23:58:59.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a8ee0640 1 -- 172.21.15.107:0/2821819540 learned_addr learned my addr 172.21.15.107:0/2821819540 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:58:59.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a96e1640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86a406ee40 0x7f86a415bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:59.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a9ee2640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a41eb3d0 0x7f86a4147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:59.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a8ee0640 1 -- 172.21.15.107:0/2821819540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a41eb3d0 msgr2=0x7f86a4147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:59.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a8ee0640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a41eb3d0 0x7f86a4147730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a8ee0640 1 -- 172.21.15.107:0/2821819540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86a406ee40 msgr2=0x7f86a415bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:59.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a8ee0640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86a406ee40 0x7f86a415bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a8ee0640 1 -- 172.21.15.107:0/2821819540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a0009420 con 0x7f86a414fd40 2023-10-24T23:58:59.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a9ee2640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a41eb3d0 0x7f86a4147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:58:59.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86a8ee0640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a414fd40 0x7f86a414bce0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f86a001a9e0 tx=0x7f86a00190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:59.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f8699ffb640 1 -- 172.21.15.107:0/2821819540 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a0004230 con 0x7f86a414fd40 2023-10-24T23:58:59.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f8699ffb640 1 -- 172.21.15.107:0/2821819540 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a0004390 con 0x7f86a414fd40 2023-10-24T23:58:59.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86ab945640 1 -- 172.21.15.107:0/2821819540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a40c5550 con 0x7f86a414fd40 2023-10-24T23:58:59.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f8699ffb640 1 -- 172.21.15.107:0/2821819540 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a0003820 con 0x7f86a414fd40 2023-10-24T23:58:59.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86ab945640 1 -- 172.21.15.107:0/2821819540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a414f0c0 con 0x7f86a414fd40 2023-10-24T23:58:59.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f86ab945640 1 -- 172.21.15.107:0/2821819540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a40c24a0 con 0x7f86a414fd40 2023-10-24T23:58:59.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.645+0000 7f8699ffb640 1 -- 172.21.15.107:0/2821819540 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f86a00b2040 con 0x7f86a414fd40 2023-10-24T23:58:59.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.649+0000 7f8699ffb640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8688065e90 0x7f8688068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:59.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.649+0000 7f8699ffb640 1 -- 172.21.15.107:0/2821819540 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7f86a011dd10 con 0x7f86a414fd40 2023-10-24T23:58:59.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.649+0000 7f86a96e1640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8688065e90 0x7f8688068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:59.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.649+0000 7f86a96e1640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8688065e90 0x7f8688068350 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8694007fd0 tx=0x7f8694007d60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:58:59.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.653+0000 7f8699ffb640 1 -- 172.21.15.107:0/2821819540 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f86a00ead30 con 0x7f86a414fd40 2023-10-24T23:58:59.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 -- 172.21.15.107:0/2821819540 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86a4070840 con 0x7f86a414fd40 2023-10-24T23:58:59.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f8699ffb640 1 -- 172.21.15.107:0/2821819540 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4261 (secure 0 0 0) 0x7f86a00c2070 con 0x7f86a414fd40 2023-10-24T23:58:59.846 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:58:59.846 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:59.617148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:58:59.847 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 56 2023-10-24T23:58:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 -- 172.21.15.107:0/2821819540 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8688065e90 msgr2=0x7f8688068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8688065e90 0x7f8688068350 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8694007fd0 tx=0x7f8694007d60 comp rx=0 tx=0).stop 2023-10-24T23:58:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 -- 172.21.15.107:0/2821819540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a414fd40 msgr2=0x7f86a414bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:59.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a414fd40 0x7f86a414bce0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f86a001a9e0 tx=0x7f86a00190d0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 -- 172.21.15.107:0/2821819540 shutdown_connections 2023-10-24T23:58:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8688065e90 0x7f8688068350 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86a41eb3d0 0x7f86a4147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86a414fd40 0x7f86a414bce0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 --2- 172.21.15.107:0/2821819540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86a406ee40 0x7f86a415bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 -- 172.21.15.107:0/2821819540 >> 172.21.15.107:0/2821819540 conn(0x7f86a40c2950 msgr2=0x7f86a414d530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:58:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 -- 172.21.15.107:0/2821819540 shutdown_connections 2023-10-24T23:58:59.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.841+0000 7f86ab945640 1 -- 172.21.15.107:0/2821819540 wait complete. 2023-10-24T23:58:59.865 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2023-10-24T23:58:59.865 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-24T23:58:59.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:58:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.981+0000 7fb76762f640 1 Processor -- start 2023-10-24T23:58:59.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.981+0000 7fb76762f640 1 -- start start 2023-10-24T23:58:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb76762f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb76006bc30 0x7fb76015c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb76762f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7601565c0 con 0x7fb76015c680 2023-10-24T23:58:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb76762f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76006a2b0 con 0x7fb7601561e0 2023-10-24T23:58:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb76762f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76006a590 con 0x7fb76006bc30 2023-10-24T23:58:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb765bcc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb76015c680 0x7fb7601634d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56156/0 (socket says 172.21.15.107:56156) 2023-10-24T23:58:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb765bcc640 1 -- 172.21.15.107:0/3418238924 learned_addr learned my addr 172.21.15.107:0/3418238924 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:58:59.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb764bca640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb76006bc30 0x7fb76015c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531099257 0 0) 0x7fb7601565c0 con 0x7fb76015c680 2023-10-24T23:58:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb74c003530 con 0x7fb76015c680 2023-10-24T23:58:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3623169345 0 0) 0x7fb74c003530 con 0x7fb76015c680 2023-10-24T23:58:59.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb76006bc30 msgr2=0x7fb76015c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb76006bc30 0x7fb76015c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 >> v1:172.21.15.110:6789/0 conn(0x7fb7601561e0 legacy=0x7fb7600c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76006a870 con 0x7fb76015c680 2023-10-24T23:58:59.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb758001a00 con 0x7fb76015c680 2023-10-24T23:58:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 >> v1:172.21.15.107:6789/0 conn(0x7fb76015c680 legacy=0x7fb7601634d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:58:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c0039b0 0x7fb74c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c0047d0 0x7fb74c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:59.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb74c005150 0x7fb74c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:58:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76006a590 con 0x7fb74c005150 2023-10-24T23:58:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c00bed0 con 0x7fb74c0039b0 2023-10-24T23:58:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c00c060 con 0x7fb74c0047d0 2023-10-24T23:58:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb765bcc640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb74c005150 0x7fb74c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:59.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb7653cb640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c0039b0 0x7fb74c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:59.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb764bca640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c0047d0 0x7fb74c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:58:59.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb765bcc640 1 -- 172.21.15.107:0/3418238924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c0047d0 msgr2=0x7fb74c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:59.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb765bcc640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb74c0047d0 0x7fb74c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb765bcc640 1 -- 172.21.15.107:0/3418238924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c0039b0 msgr2=0x7fb74c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:58:59.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb765bcc640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb74c0039b0 0x7fb74c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:58:59.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.985+0000 7fb765bcc640 1 -- 172.21.15.107:0/3418238924 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c003530 con 0x7fb74c005150 2023-10-24T23:59:00.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb765bcc640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb74c005150 0x7fb74c00b990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb75800a830 tx=0x7fb75801faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2a6d185018d92a server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:00.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7580aa6d0 con 0x7fb74c005150 2023-10-24T23:59:00.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7580aa850 con 0x7fb74c005150 2023-10-24T23:59:00.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb757fff640 1 -- 172.21.15.107:0/3418238924 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7580aac00 con 0x7fb74c005150 2023-10-24T23:59:00.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 -- 172.21.15.107:0/3418238924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb74c005150 msgr2=0x7fb74c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:00.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 --2- 172.21.15.107:0/3418238924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb74c005150 0x7fb74c00b990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fb75800a830 tx=0x7fb75801faa0 comp rx=0 tx=0).stop 2023-10-24T23:59:00.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb765bcc640 1 -- 172.21.15.107:0/3418238924 reap_dead start 2023-10-24T23:59:00.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 -- 172.21.15.107:0/3418238924 shutdown_connections 2023-10-24T23:59:00.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 -- 172.21.15.107:0/3418238924 >> 172.21.15.107:0/3418238924 conn(0x7fb7600c2950 msgr2=0x7fb7600c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:00.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 -- 172.21.15.107:0/3418238924 shutdown_connections 2023-10-24T23:59:00.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 -- 172.21.15.107:0/3418238924 wait complete. 2023-10-24T23:59:00.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 Processor -- start 2023-10-24T23:59:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 -- start start 2023-10-24T23:59:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7601f1190 0x7fb7601f1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7600bff40 0x7fb76014ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:00.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb760071b50 0x7fb76014b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7601ef650 con 0x7fb760071b50 2023-10-24T23:59:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7601f1f00 con 0x7fb7601f1190 2023-10-24T23:59:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb76762f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7601eefd0 con 0x7fb7600bff40 2023-10-24T23:59:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb765bcc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb760071b50 0x7fb76014b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:00.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb765bcc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb760071b50 0x7fb76014b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38092/0 (socket says 172.21.15.107:38092) 2023-10-24T23:59:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb765bcc640 1 -- 172.21.15.107:0/2522313755 learned_addr learned my addr 172.21.15.107:0/2522313755 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb7653cb640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7601f1190 0x7fb7601f1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb764bca640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7600bff40 0x7fb76014ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb765bcc640 1 -- 172.21.15.107:0/2522313755 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7600bff40 msgr2=0x7fb76014ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:00.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb765bcc640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7600bff40 0x7fb76014ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:00.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb765bcc640 1 -- 172.21.15.107:0/2522313755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7601f1190 msgr2=0x7fb7601f1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:00.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb765bcc640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7601f1190 0x7fb7601f1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:00.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb765bcc640 1 -- 172.21.15.107:0/2522313755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7601f1c20 con 0x7fb760071b50 2023-10-24T23:59:00.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb765bcc640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb760071b50 0x7fb76014b2a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb75800a830 tx=0x7fb74c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:00.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.989+0000 7fb755ffb640 1 -- 172.21.15.107:0/2522313755 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7580aa4d0 con 0x7fb760071b50 2023-10-24T23:59:00.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.993+0000 7fb755ffb640 1 -- 172.21.15.107:0/2522313755 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7580aa630 con 0x7fb760071b50 2023-10-24T23:59:00.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.993+0000 7fb76762f640 1 -- 172.21.15.107:0/2522313755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7601ef270 con 0x7fb760071b50 2023-10-24T23:59:00.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.993+0000 7fb755ffb640 1 -- 172.21.15.107:0/2522313755 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7580aabe0 con 0x7fb760071b50 2023-10-24T23:59:00.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.993+0000 7fb76762f640 1 -- 172.21.15.107:0/2522313755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7601efa70 con 0x7fb760071b50 2023-10-24T23:59:00.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.993+0000 7fb76762f640 1 -- 172.21.15.107:0/2522313755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7601eee30 con 0x7fb760071b50 2023-10-24T23:59:00.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.993+0000 7fb755ffb640 1 -- 172.21.15.107:0/2522313755 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb7580aad40 con 0x7fb760071b50 2023-10-24T23:59:00.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.993+0000 7fb755ffb640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb744065e60 0x7fb744068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:00.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.993+0000 7fb755ffb640 1 -- 172.21.15.107:0/2522313755 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7fb7581212a0 con 0x7fb760071b50 2023-10-24T23:59:00.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.993+0000 7fb7653cb640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb744065e60 0x7fb744068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:00.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.993+0000 7fb7653cb640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb744065e60 0x7fb744068320 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb75c005510 tx=0x7fb75c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:00.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:58:59.997+0000 7fb755ffb640 1 -- 172.21.15.107:0/2522313755 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb7580ee3b0 con 0x7fb760071b50 2023-10-24T23:59:00.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.201+0000 7fb76762f640 1 -- 172.21.15.107:0/2522313755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7601eee30 con 0x7fb760071b50 2023-10-24T23:59:00.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb755ffb640 1 -- 172.21.15.107:0/2522313755 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4261 (secure 0 0 0) 0x7fb7601eee30 con 0x7fb760071b50 2023-10-24T23:59:00.209 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:00.210 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:58:59.617148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:00.210 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 56 2023-10-24T23:59:00.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 -- 172.21.15.107:0/2522313755 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb744065e60 msgr2=0x7fb744068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:00.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb744065e60 0x7fb744068320 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb75c005510 tx=0x7fb75c009040 comp rx=0 tx=0).stop 2023-10-24T23:59:00.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 -- 172.21.15.107:0/2522313755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb760071b50 msgr2=0x7fb76014b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:00.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb760071b50 0x7fb76014b2a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb75800a830 tx=0x7fb74c005800 comp rx=0 tx=0).stop 2023-10-24T23:59:00.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 -- 172.21.15.107:0/2522313755 shutdown_connections 2023-10-24T23:59:00.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb760071b50 0x7fb76014b2a0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb744065e60 0x7fb744068320 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7600bff40 0x7fb76014ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 --2- 172.21.15.107:0/2522313755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7601f1190 0x7fb7601f1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:00.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 -- 172.21.15.107:0/2522313755 >> 172.21.15.107:0/2522313755 conn(0x7fb7600c2950 msgr2=0x7fb760153a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:00.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 -- 172.21.15.107:0/2522313755 shutdown_connections 2023-10-24T23:59:00.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:00.205+0000 7fb76762f640 1 -- 172.21.15.107:0/2522313755 wait complete. 2023-10-24T23:59:00.232 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 56, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5293": { "addr": "172.21.15.110:6834/1101018338", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 1101018338, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 1101018338, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5293, "incarnation": 55, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:58:59.617148+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5293 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 54, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5925": { "addr": "172.21.15.107:6837/1292188015", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 1292188015, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 1292188015, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5925, "incarnation": 51, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-24T23:58:56.590011+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5925 } } } ], "standbys": [] } 2023-10-24T23:59:02.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:02.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2e1ad3640 1 Processor -- start 2023-10-24T23:59:02.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2e1ad3640 1 -- start start 2023-10-24T23:59:02.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2e1ad3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2dc150930 0x7ff2dc0a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2e1ad3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2dc1510c0 0x7ff2dc151550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2e1ad3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2dc0a9b90 con 0x7ff2dc0c1570 2023-10-24T23:59:02.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2e1ad3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2dc0c1390 con 0x7ff2dc150930 2023-10-24T23:59:02.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2e1ad3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc154dd0 con 0x7ff2dc1510c0 2023-10-24T23:59:02.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2db7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2dc150930 0x7ff2dc0a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2daffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff2dc0c1570 0x7ff2dc069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56160/0 (socket says 172.21.15.107:56160) 2023-10-24T23:59:02.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2daffd640 1 -- 172.21.15.107:0/4124494574 learned_addr learned my addr 172.21.15.107:0/4124494574 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:59:02.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2dbfff640 1 --2- 172.21.15.107:0/4124494574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2dc1510c0 0x7ff2dc151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2db7fe640 1 -- 172.21.15.107:0/4124494574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2dc1510c0 msgr2=0x7ff2dc151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:02.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2db7fe640 1 --2- 172.21.15.107:0/4124494574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2dc1510c0 0x7ff2dc151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2db7fe640 1 -- 172.21.15.107:0/4124494574 >> v1:172.21.15.107:6789/0 conn(0x7ff2dc0c1570 legacy=0x7ff2dc069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:02.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2db7fe640 1 -- 172.21.15.107:0/4124494574 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2dc1550b0 con 0x7ff2dc150930 2023-10-24T23:59:02.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.381+0000 7ff2dbfff640 1 --2- 172.21.15.107:0/4124494574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2dc1510c0 0x7ff2dc151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:59:02.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2db7fe640 1 --2- 172.21.15.107:0/4124494574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2dc150930 0x7ff2dc0a94b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0097c0 tx=0x7ff2cc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe58a64a62bc0811 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:02.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2da7fc640 1 -- 172.21.15.107:0/4124494574 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc0ab600 con 0x7ff2dc150930 2023-10-24T23:59:02.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2da7fc640 1 -- 172.21.15.107:0/4124494574 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2cc0a7050 con 0x7ff2dc150930 2023-10-24T23:59:02.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2da7fc640 1 -- 172.21.15.107:0/4124494574 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc0ab980 con 0x7ff2dc150930 2023-10-24T23:59:02.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/4124494574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2dc150930 msgr2=0x7ff2dc0a94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:02.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2e1ad3640 1 --2- 172.21.15.107:0/4124494574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2dc150930 0x7ff2dc0a94b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0097c0 tx=0x7ff2cc01c3f0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/4124494574 shutdown_connections 2023-10-24T23:59:02.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2e1ad3640 1 --2- 172.21.15.107:0/4124494574 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2dc1510c0 0x7ff2dc151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2e1ad3640 1 --2- 172.21.15.107:0/4124494574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2dc150930 0x7ff2dc0a94b0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/4124494574 >> 172.21.15.107:0/4124494574 conn(0x7ff2dc0c65c0 msgr2=0x7ff2dc0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:02.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/4124494574 shutdown_connections 2023-10-24T23:59:02.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.385+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/4124494574 wait complete. 2023-10-24T23:59:02.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2e1ad3640 1 Processor -- start 2023-10-24T23:59:02.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2e1ad3640 1 -- start start 2023-10-24T23:59:02.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2e1ad3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2dc0c1570 0x7ff2dc153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2e1ad3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2dc150930 0x7ff2dc15f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2e1ad3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2dc1510c0 0x7ff2dc1ee420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2e1ad3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc1542b0 con 0x7ff2dc150930 2023-10-24T23:59:02.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2e1ad3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc153080 con 0x7ff2dc1510c0 2023-10-24T23:59:02.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2e1ad3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc1522c0 con 0x7ff2dc0c1570 2023-10-24T23:59:02.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2daffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2dc150930 0x7ff2dc15f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2daffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2dc150930 0x7ff2dc15f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38112/0 (socket says 172.21.15.107:38112) 2023-10-24T23:59:02.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2daffd640 1 -- 172.21.15.107:0/2554859279 learned_addr learned my addr 172.21.15.107:0/2554859279 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:02.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2db7fe640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2dc0c1570 0x7ff2dc153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2dbfff640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2dc1510c0 0x7ff2dc1ee420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2daffd640 1 -- 172.21.15.107:0/2554859279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2dc0c1570 msgr2=0x7ff2dc153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:02.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2daffd640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2dc0c1570 0x7ff2dc153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2daffd640 1 -- 172.21.15.107:0/2554859279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2dc1510c0 msgr2=0x7ff2dc1ee420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2daffd640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2dc1510c0 0x7ff2dc1ee420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2daffd640 1 -- 172.21.15.107:0/2554859279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc009420 con 0x7ff2dc150930 2023-10-24T23:59:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2daffd640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2dc150930 0x7ff2dc15f120 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000c500 tx=0x7ff2d000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2c3fff640 1 -- 172.21.15.107:0/2554859279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d0005cf0 con 0x7ff2dc150930 2023-10-24T23:59:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2c3fff640 1 -- 172.21.15.107:0/2554859279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d0005e50 con 0x7ff2dc150930 2023-10-24T23:59:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/2554859279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2dc1525a0 con 0x7ff2dc150930 2023-10-24T23:59:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2c3fff640 1 -- 172.21.15.107:0/2554859279 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d00072f0 con 0x7ff2dc150930 2023-10-24T23:59:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/2554859279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2dc152b60 con 0x7ff2dc150930 2023-10-24T23:59:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.389+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/2554859279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2dc152d50 con 0x7ff2dc150930 2023-10-24T23:59:02.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.393+0000 7ff2c3fff640 1 -- 172.21.15.107:0/2554859279 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff2d0007450 con 0x7ff2dc150930 2023-10-24T23:59:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.393+0000 7ff2c3fff640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2c8065e60 0x7ff2c8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.393+0000 7ff2c3fff640 1 -- 172.21.15.107:0/2554859279 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7ff2d0083370 con 0x7ff2dc150930 2023-10-24T23:59:02.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.393+0000 7ff2db7fe640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2c8065e60 0x7ff2c8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.393+0000 7ff2db7fe640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2c8065e60 0x7ff2c8068320 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0ad000 tx=0x7ff2cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:02.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.397+0000 7ff2c3fff640 1 -- 172.21.15.107:0/2554859279 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff2d0050420 con 0x7ff2dc150930 2023-10-24T23:59:02.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.617+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/2554859279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2dc0746e0 con 0x7ff2dc150930 2023-10-24T23:59:02.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.617+0000 7ff2c3fff640 1 -- 172.21.15.107:0/2554859279 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4258 (secure 0 0 0) 0x7ff2d0017090 con 0x7ff2dc150930 2023-10-24T23:59:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:02.623 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:02.624 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 58 2023-10-24T23:59:02.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/2554859279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2c8065e60 msgr2=0x7ff2c8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:02.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2c8065e60 0x7ff2c8068320 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0ad000 tx=0x7ff2cc008040 comp rx=0 tx=0).stop 2023-10-24T23:59:02.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/2554859279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2dc150930 msgr2=0x7ff2dc15f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:02.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2dc150930 0x7ff2dc15f120 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff2d000c500 tx=0x7ff2d000c9c0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/2554859279 shutdown_connections 2023-10-24T23:59:02.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2dc1510c0 0x7ff2dc1ee420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2dc150930 0x7ff2dc15f120 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2c8065e60 0x7ff2c8068320 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 --2- 172.21.15.107:0/2554859279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2dc0c1570 0x7ff2dc153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/2554859279 >> 172.21.15.107:0/2554859279 conn(0x7ff2dc0c65c0 msgr2=0x7ff2dc0c3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:02.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/2554859279 shutdown_connections 2023-10-24T23:59:02.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.621+0000 7ff2e1ad3640 1 -- 172.21.15.107:0/2554859279 wait complete. 2023-10-24T23:59:02.629 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2023-10-24T23:59:02.629 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-24T23:59:02.630 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-24T23:59:02.633 INFO:tasks.ceph.mds.d:Started 2023-10-24T23:59:02.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:02.645 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=5751 2023-10-24T23:59:02.645 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.c 2023-10-24T23:59:02.710 INFO:tasks.ceph.mds.d.smithi110.stdout:starting mds.d at 2023-10-24T23:59:02.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.757+0000 7f77d6bd0640 1 Processor -- start 2023-10-24T23:59:02.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d6bd0640 1 -- start start 2023-10-24T23:59:02.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d6bd0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f77d0151fb0 0x7f77d0155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d6bd0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f77d00c82e0 0x7f77d00c8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d6bd0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d0152390 con 0x7f77d0158bf0 2023-10-24T23:59:02.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d6bd0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d00c5630 con 0x7f77d0151fb0 2023-10-24T23:59:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d6bd0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d0158760 con 0x7f77d00c82e0 2023-10-24T23:59:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d496c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f77d0151fb0 0x7f77d0155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d496c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f77d0151fb0 0x7f77d0155560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39040/0 (socket says 172.21.15.107:39040) 2023-10-24T23:59:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d496c640 1 -- 172.21.15.107:0/3482590580 learned_addr learned my addr 172.21.15.107:0/3482590580 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:02.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d516d640 1 --2- 172.21.15.107:0/3482590580 >> v2:172.21.15.110:3300/0 conn(0x7f77d00c82e0 0x7f77d00c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d496c640 1 -- 172.21.15.107:0/3482590580 >> v2:172.21.15.110:3300/0 conn(0x7f77d00c82e0 msgr2=0x7f77d00c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:02.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d496c640 1 --2- 172.21.15.107:0/3482590580 >> v2:172.21.15.110:3300/0 conn(0x7f77d00c82e0 0x7f77d00c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d496c640 1 -- 172.21.15.107:0/3482590580 >> v1:172.21.15.110:6789/0 conn(0x7f77d0158bf0 legacy=0x7f77d00c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:02.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d496c640 1 -- 172.21.15.107:0/3482590580 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d00c5350 con 0x7f77d0151fb0 2023-10-24T23:59:02.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d516d640 1 --2- 172.21.15.107:0/3482590580 >> v2:172.21.15.110:3300/0 conn(0x7f77d00c82e0 0x7f77d00c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:59:02.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77d496c640 1 --2- 172.21.15.107:0/3482590580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d0151fb0 0x7f77d0155560 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f77d0066770 tx=0x7f77c401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc08ee18ec6e6c4b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:02.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77cf7fe640 1 -- 172.21.15.107:0/3482590580 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c40ab640 con 0x7f77d0151fb0 2023-10-24T23:59:02.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77cf7fe640 1 -- 172.21.15.107:0/3482590580 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c40a7070 con 0x7f77d0151fb0 2023-10-24T23:59:02.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.761+0000 7f77cf7fe640 1 -- 172.21.15.107:0/3482590580 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c40ab9c0 con 0x7f77d0151fb0 2023-10-24T23:59:02.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 -- 172.21.15.107:0/3482590580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d0151fb0 msgr2=0x7f77d0155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:02.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 --2- 172.21.15.107:0/3482590580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d0151fb0 0x7f77d0155560 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f77d0066770 tx=0x7f77c401c440 comp rx=0 tx=0).stop 2023-10-24T23:59:02.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 -- 172.21.15.107:0/3482590580 shutdown_connections 2023-10-24T23:59:02.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 --2- 172.21.15.107:0/3482590580 >> v2:172.21.15.110:3300/0 conn(0x7f77d00c82e0 0x7f77d00c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 --2- 172.21.15.107:0/3482590580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d0151fb0 0x7f77d0155560 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 -- 172.21.15.107:0/3482590580 >> 172.21.15.107:0/3482590580 conn(0x7f77d00abcf0 msgr2=0x7f77d00ac100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:02.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 -- 172.21.15.107:0/3482590580 shutdown_connections 2023-10-24T23:59:02.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 -- 172.21.15.107:0/3482590580 wait complete. 2023-10-24T23:59:02.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 Processor -- start 2023-10-24T23:59:02.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 -- start start 2023-10-24T23:59:02.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d00c82e0 0x7f77d0148470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77d0151fb0 0x7f77d01489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77d0158bf0 0x7f77d0156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00c15e0 con 0x7f77d00c82e0 2023-10-24T23:59:02.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00c40e0 con 0x7f77d0151fb0 2023-10-24T23:59:02.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d6bd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d00c35b0 con 0x7f77d0158bf0 2023-10-24T23:59:02.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d496c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d00c82e0 0x7f77d0148470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d496c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d00c82e0 0x7f77d0148470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39052/0 (socket says 172.21.15.107:39052) 2023-10-24T23:59:02.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d496c640 1 -- 172.21.15.107:0/2019080901 learned_addr learned my addr 172.21.15.107:0/2019080901 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:02.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77cffff640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77d0151fb0 0x7f77d01489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d516d640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77d0158bf0 0x7f77d0156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d496c640 1 -- 172.21.15.107:0/2019080901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77d0158bf0 msgr2=0x7f77d0156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:02.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d496c640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77d0158bf0 0x7f77d0156630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d496c640 1 -- 172.21.15.107:0/2019080901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77d0151fb0 msgr2=0x7f77d01489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:02.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d496c640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77d0151fb0 0x7f77d01489b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:02.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d496c640 1 -- 172.21.15.107:0/2019080901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c40094d0 con 0x7f77d00c82e0 2023-10-24T23:59:02.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.765+0000 7f77d496c640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d00c82e0 0x7f77d0148470 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f77c4019050 tx=0x7f77c401ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:02.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.769+0000 7f77cd7fa640 1 -- 172.21.15.107:0/2019080901 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c40be3f0 con 0x7f77d00c82e0 2023-10-24T23:59:02.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.769+0000 7f77d6bd0640 1 -- 172.21.15.107:0/2019080901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d0157850 con 0x7f77d00c82e0 2023-10-24T23:59:02.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.769+0000 7f77cd7fa640 1 -- 172.21.15.107:0/2019080901 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77c40be550 con 0x7f77d00c82e0 2023-10-24T23:59:02.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.769+0000 7f77cd7fa640 1 -- 172.21.15.107:0/2019080901 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c40be8e0 con 0x7f77d00c82e0 2023-10-24T23:59:02.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.769+0000 7f77d6bd0640 1 -- 172.21.15.107:0/2019080901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d0157e10 con 0x7f77d00c82e0 2023-10-24T23:59:02.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.769+0000 7f77d6bd0640 1 -- 172.21.15.107:0/2019080901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d00746e0 con 0x7f77d00c82e0 2023-10-24T23:59:02.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.769+0000 7f77cd7fa640 1 -- 172.21.15.107:0/2019080901 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f77c40c7440 con 0x7f77d00c82e0 2023-10-24T23:59:02.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.769+0000 7f77cd7fa640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f77b4065e60 0x7f77b4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:02.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.769+0000 7f77cd7fa640 1 -- 172.21.15.107:0/2019080901 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7f77c411e8c0 con 0x7f77d00c82e0 2023-10-24T23:59:02.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.769+0000 7f77cffff640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f77b4065e60 0x7f77b4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:02.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.769+0000 7f77cffff640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f77b4065e60 0x7f77b4068320 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f77d01471e0 tx=0x7f77bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:02.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.773+0000 7f77cd7fa640 1 -- 172.21.15.107:0/2019080901 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f77c40eb8f0 con 0x7f77d00c82e0 2023-10-24T23:59:03.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.997+0000 7f77d6bd0640 1 -- 172.21.15.107:0/2019080901 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77d00c0e60 con 0x7f77d00c82e0 2023-10-24T23:59:03.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:02.997+0000 7f77cd7fa640 1 -- 172.21.15.107:0/2019080901 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+4258 (secure 0 0 0) 0x7f77c40eef10 con 0x7f77d00c82e0 2023-10-24T23:59:03.003 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:03.003 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:03.003 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 58 2023-10-24T23:59:03.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 -- 172.21.15.107:0/2019080901 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f77b4065e60 msgr2=0x7f77b4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:03.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f77b4065e60 0x7f77b4068320 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f77d01471e0 tx=0x7f77bc009040 comp rx=0 tx=0).stop 2023-10-24T23:59:03.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 -- 172.21.15.107:0/2019080901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d00c82e0 msgr2=0x7f77d0148470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:03.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d00c82e0 0x7f77d0148470 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f77c4019050 tx=0x7f77c401ac80 comp rx=0 tx=0).stop 2023-10-24T23:59:03.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 -- 172.21.15.107:0/2019080901 shutdown_connections 2023-10-24T23:59:03.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f77b4065e60 0x7f77b4068320 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:03.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77d0158bf0 0x7f77d0156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:03.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77d0151fb0 0x7f77d01489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:03.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 --2- 172.21.15.107:0/2019080901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d00c82e0 0x7f77d0148470 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:03.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 -- 172.21.15.107:0/2019080901 >> 172.21.15.107:0/2019080901 conn(0x7f77d00abcf0 msgr2=0x7f77d00c4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:03.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 -- 172.21.15.107:0/2019080901 shutdown_connections 2023-10-24T23:59:03.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:03.001+0000 7f77d6bd0640 1 -- 172.21.15.107:0/2019080901 wait complete. 2023-10-24T23:59:03.024 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-24T23:59:03.575 INFO:tasks.ceph.mds.b.smithi110.stderr:2023-10-24T23:59:03.571+0000 7f0d46e11640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-24T23:59:04.647 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2023-10-24T23:59:04.647 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-24T23:59:04.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-24T23:59:04.650 INFO:tasks.ceph.mds.c:Started 2023-10-24T23:59:04.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:04.726 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-24T23:59:04.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff0e309640 1 Processor -- start 2023-10-24T23:59:04.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff0e309640 1 -- start start 2023-10-24T23:59:04.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff0e309640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7eff08069d40 0x7eff08158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:04.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff0e309640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7eff0806a120 0x7eff080aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:04.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff0e309640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0806cea0 con 0x7eff08065bc0 2023-10-24T23:59:04.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff0e309640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff080ac650 con 0x7eff08069d40 2023-10-24T23:59:04.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff0e309640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff080a9e00 con 0x7eff0806a120 2023-10-24T23:59:04.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff07fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7eff08069d40 0x7eff08158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:04.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff07fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7eff08069d40 0x7eff08158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39088/0 (socket says 172.21.15.107:39088) 2023-10-24T23:59:04.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff07fff640 1 -- 172.21.15.107:0/46675701 learned_addr learned my addr 172.21.15.107:0/46675701 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:04.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff0c8a6640 1 --2- 172.21.15.107:0/46675701 >> v2:172.21.15.110:3300/0 conn(0x7eff0806a120 0x7eff080aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:04.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff07fff640 1 -- 172.21.15.107:0/46675701 >> v2:172.21.15.110:3300/0 conn(0x7eff0806a120 msgr2=0x7eff080aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:04.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff07fff640 1 --2- 172.21.15.107:0/46675701 >> v2:172.21.15.110:3300/0 conn(0x7eff0806a120 0x7eff080aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:04.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff07fff640 1 -- 172.21.15.107:0/46675701 >> v1:172.21.15.110:6789/0 conn(0x7eff08065bc0 legacy=0x7eff080a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:04.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff07fff640 1 -- 172.21.15.107:0/46675701 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff081584e0 con 0x7eff08069d40 2023-10-24T23:59:04.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.781+0000 7eff0c8a6640 1 --2- 172.21.15.107:0/46675701 >> v2:172.21.15.110:3300/0 conn(0x7eff0806a120 0x7eff080aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:59:04.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff07fff640 1 --2- 172.21.15.107:0/46675701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff08069d40 0x7eff08158970 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7eff080c7400 tx=0x7efef801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ce5e3fffe3bb7b server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:04.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff06ffd640 1 -- 172.21.15.107:0/46675701 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef80ad660 con 0x7eff08069d40 2023-10-24T23:59:04.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff06ffd640 1 -- 172.21.15.107:0/46675701 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef80ad7c0 con 0x7eff08069d40 2023-10-24T23:59:04.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff06ffd640 1 -- 172.21.15.107:0/46675701 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef80adb50 con 0x7eff08069d40 2023-10-24T23:59:04.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff0e309640 1 -- 172.21.15.107:0/46675701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff08069d40 msgr2=0x7eff08158970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:04.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff0e309640 1 --2- 172.21.15.107:0/46675701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff08069d40 0x7eff08158970 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7eff080c7400 tx=0x7efef801c470 comp rx=0 tx=0).stop 2023-10-24T23:59:04.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff0e309640 1 -- 172.21.15.107:0/46675701 shutdown_connections 2023-10-24T23:59:04.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff0e309640 1 --2- 172.21.15.107:0/46675701 >> v2:172.21.15.110:3300/0 conn(0x7eff0806a120 0x7eff080aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:04.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff0e309640 1 --2- 172.21.15.107:0/46675701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff08069d40 0x7eff08158970 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:04.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff0e309640 1 -- 172.21.15.107:0/46675701 >> 172.21.15.107:0/46675701 conn(0x7eff080c65c0 msgr2=0x7eff080c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:04.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff0e309640 1 -- 172.21.15.107:0/46675701 shutdown_connections 2023-10-24T23:59:04.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff0e309640 1 -- 172.21.15.107:0/46675701 wait complete. 2023-10-24T23:59:04.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.785+0000 7eff0e309640 1 Processor -- start 2023-10-24T23:59:04.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0e309640 1 -- start start 2023-10-24T23:59:04.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0e309640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff08065bc0 0x7eff080be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:04.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0e309640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff08069d40 0x7eff0814dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:04.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0e309640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff0806a120 0x7eff080a72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:04.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0e309640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0814eda0 con 0x7eff08065bc0 2023-10-24T23:59:04.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0e309640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0814f080 con 0x7eff0806a120 2023-10-24T23:59:04.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0e309640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0814f360 con 0x7eff08069d40 2023-10-24T23:59:04.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff07fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff08065bc0 0x7eff080be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:04.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff07fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff08065bc0 0x7eff080be9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39094/0 (socket says 172.21.15.107:39094) 2023-10-24T23:59:04.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff07fff640 1 -- 172.21.15.107:0/3348253652 learned_addr learned my addr 172.21.15.107:0/3348253652 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:04.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0c8a6640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff0806a120 0x7eff080a72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:04.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff077fe640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff08069d40 0x7eff0814dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:04.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff07fff640 1 -- 172.21.15.107:0/3348253652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff08069d40 msgr2=0x7eff0814dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:04.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff07fff640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff08069d40 0x7eff0814dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:04.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff07fff640 1 -- 172.21.15.107:0/3348253652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff0806a120 msgr2=0x7eff080a72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff07fff640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff0806a120 0x7eff080a72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff07fff640 1 -- 172.21.15.107:0/3348253652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef80094d0 con 0x7eff08065bc0 2023-10-24T23:59:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff077fe640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff08069d40 0x7eff0814dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:59:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0c8a6640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff0806a120 0x7eff080a72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:59:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff07fff640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff08065bc0 0x7eff080be9f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7efef80b0040 tx=0x7efef8019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:04.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff04ff9640 1 -- 172.21.15.107:0/3348253652 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef80ba070 con 0x7eff08065bc0 2023-10-24T23:59:04.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff04ff9640 1 -- 172.21.15.107:0/3348253652 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef80b55d0 con 0x7eff08065bc0 2023-10-24T23:59:04.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0e309640 1 -- 172.21.15.107:0/3348253652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0814d520 con 0x7eff08065bc0 2023-10-24T23:59:04.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff04ff9640 1 -- 172.21.15.107:0/3348253652 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef80b5980 con 0x7eff08065bc0 2023-10-24T23:59:04.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0e309640 1 -- 172.21.15.107:0/3348253652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff080be5e0 con 0x7eff08065bc0 2023-10-24T23:59:04.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff0e309640 1 -- 172.21.15.107:0/3348253652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff080746e0 con 0x7eff08065bc0 2023-10-24T23:59:04.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff04ff9640 1 -- 172.21.15.107:0/3348253652 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7efef80b4070 con 0x7eff08065bc0 2023-10-24T23:59:04.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.789+0000 7eff04ff9640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efef4065e60 0x7efef4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:04.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.793+0000 7eff04ff9640 1 -- 172.21.15.107:0/3348253652 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7efef811dd50 con 0x7eff08065bc0 2023-10-24T23:59:04.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.793+0000 7eff077fe640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efef4065e60 0x7efef4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:04.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.793+0000 7eff077fe640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efef4065e60 0x7efef4068320 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7eff080a96f0 tx=0x7efefc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:04.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:04.797+0000 7eff04ff9640 1 -- 172.21.15.107:0/3348253652 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7efef8121030 con 0x7eff08065bc0 2023-10-24T23:59:05.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:05.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.053+0000 7eff0e309640 1 -- 172.21.15.107:0/3348253652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff0814d9d0 con 0x7eff08065bc0 2023-10-24T23:59:05.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.053+0000 7eff04ff9640 1 -- 172.21.15.107:0/3348253652 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5017 (secure 0 0 0) 0x7efef80eacd0 con 0x7eff08065bc0 2023-10-24T23:59:05.059 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:05.059 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:05.060 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 59 2023-10-24T23:59:05.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 -- 172.21.15.107:0/3348253652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efef4065e60 msgr2=0x7efef4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:05.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efef4065e60 0x7efef4068320 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7eff080a96f0 tx=0x7efefc009040 comp rx=0 tx=0).stop 2023-10-24T23:59:05.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 -- 172.21.15.107:0/3348253652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff08065bc0 msgr2=0x7eff080be9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:05.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff08065bc0 0x7eff080be9f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7efef80b0040 tx=0x7efef8019210 comp rx=0 tx=0).stop 2023-10-24T23:59:05.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 -- 172.21.15.107:0/3348253652 shutdown_connections 2023-10-24T23:59:05.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff0806a120 0x7eff080a72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efef4065e60 0x7efef4068320 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff08069d40 0x7eff0814dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 --2- 172.21.15.107:0/3348253652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff08065bc0 0x7eff080be9f0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 -- 172.21.15.107:0/3348253652 >> 172.21.15.107:0/3348253652 conn(0x7eff080c65c0 msgr2=0x7eff080c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:05.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 -- 172.21.15.107:0/3348253652 shutdown_connections 2023-10-24T23:59:05.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.057+0000 7eff0e309640 1 -- 172.21.15.107:0/3348253652 wait complete. 2023-10-24T23:59:05.086 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-24T23:59:05.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.141+0000 7f57b65ce640 1 Processor -- start 2023-10-24T23:59:05.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.141+0000 7f57b65ce640 1 -- start start 2023-10-24T23:59:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.141+0000 7f57b65ce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57b00c4e20 0x7f57b00c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:05.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.141+0000 7f57b65ce640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b0155f20 con 0x7f57b00c5ba0 2023-10-24T23:59:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.141+0000 7f57b65ce640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b00c6d40 con 0x7f57b014fbd0 2023-10-24T23:59:05.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57b65ce640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b0153560 con 0x7f57b00c4e20 2023-10-24T23:59:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57b4b6b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f57b00c5ba0 0x7f57b00c1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46074/0 (socket says 172.21.15.107:46074) 2023-10-24T23:59:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57b4b6b640 1 -- 172.21.15.107:0/2909303251 learned_addr learned my addr 172.21.15.107:0/2909303251 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:59:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57af7fe640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57b00c4e20 0x7f57b00c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:05.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585212584 0 0) 0x7f57b0155f20 con 0x7f57b00c5ba0 2023-10-24T23:59:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5798003530 con 0x7f57b00c5ba0 2023-10-24T23:59:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118091332 0 0) 0x7f57b00c6d40 con 0x7f57b014fbd0 2023-10-24T23:59:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b0155f20 con 0x7f57b014fbd0 2023-10-24T23:59:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2184004354 0 0) 0x7f5798003530 con 0x7f57b00c5ba0 2023-10-24T23:59:05.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57b00c4e20 msgr2=0x7f57b00c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57b00c4e20 0x7f57b00c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 >> v1:172.21.15.110:6789/0 conn(0x7f57b014fbd0 legacy=0x7f57b006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b01527a0 con 0x7f57b00c5ba0 2023-10-24T23:59:05.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f57a8001a00 con 0x7f57b00c5ba0 2023-10-24T23:59:05.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 >> v1:172.21.15.107:6789/0 conn(0x7f57b00c5ba0 legacy=0x7f57b00c1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:05.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5798003b40 0x7f5798003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:05.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57b00c4e20 0x7f57980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:05.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5798004d10 0x7f5798005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:05.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b0153560 con 0x7f5798003b40 2023-10-24T23:59:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579800bb20 con 0x7f5798004d10 2023-10-24T23:59:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f579800bcb0 con 0x7f57b00c4e20 2023-10-24T23:59:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57affff640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5798003b40 0x7f5798003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:05.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57b4b6b640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5798004d10 0x7f5798005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57af7fe640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57b00c4e20 0x7f57980047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57affff640 1 -- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57b00c4e20 msgr2=0x7f57980047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57affff640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57b00c4e20 0x7f57980047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57affff640 1 -- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5798004d10 msgr2=0x7f5798005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:05.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57affff640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5798004d10 0x7f5798005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57affff640 1 -- 172.21.15.107:0/2909303251 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5798003530 con 0x7f5798003b40 2023-10-24T23:59:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.145+0000 7f57b4b6b640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5798004d10 0x7f5798005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-24T23:59:05.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57affff640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5798003b40 0x7f5798003f40 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f57a0001900 tx=0x7f57a001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebae5efadb05a971 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:05.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a00052d0 con 0x7f5798003b40 2023-10-24T23:59:05.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a00a7050 con 0x7f5798003b40 2023-10-24T23:59:05.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57aeffd640 1 -- 172.21.15.107:0/2909303251 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a0003600 con 0x7f5798003b40 2023-10-24T23:59:05.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 -- 172.21.15.107:0/2909303251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5798003b40 msgr2=0x7f5798003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 --2- 172.21.15.107:0/2909303251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5798003b40 0x7f5798003f40 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f57a0001900 tx=0x7f57a001c560 comp rx=0 tx=0).stop 2023-10-24T23:59:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b4b6b640 1 -- 172.21.15.107:0/2909303251 reap_dead start 2023-10-24T23:59:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 -- 172.21.15.107:0/2909303251 shutdown_connections 2023-10-24T23:59:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 -- 172.21.15.107:0/2909303251 >> 172.21.15.107:0/2909303251 conn(0x7f57b00c39a0 msgr2=0x7f57b00c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:05.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 -- 172.21.15.107:0/2909303251 shutdown_connections 2023-10-24T23:59:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 -- 172.21.15.107:0/2909303251 wait complete. 2023-10-24T23:59:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 Processor -- start 2023-10-24T23:59:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 -- start start 2023-10-24T23:59:05.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5798003b40 0x7f57b0154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:05.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57b0071cd0 0x7f57b0149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:05.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57b006dd80 0x7f57b006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:05.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b0154e60 con 0x7f57b0071cd0 2023-10-24T23:59:05.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b0155100 con 0x7f57b006dd80 2023-10-24T23:59:05.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b65ce640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b01553e0 con 0x7f5798003b40 2023-10-24T23:59:05.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57af7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57b0071cd0 0x7f57b0149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:05.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57af7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57b0071cd0 0x7f57b0149820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39116/0 (socket says 172.21.15.107:39116) 2023-10-24T23:59:05.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57af7fe640 1 -- 172.21.15.107:0/2770404342 learned_addr learned my addr 172.21.15.107:0/2770404342 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:05.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57affff640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5798003b40 0x7f57b0154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:05.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57b4b6b640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57b006dd80 0x7f57b006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:05.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57af7fe640 1 -- 172.21.15.107:0/2770404342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5798003b40 msgr2=0x7f57b0154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:05.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57af7fe640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5798003b40 0x7f57b0154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57af7fe640 1 -- 172.21.15.107:0/2770404342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57b006dd80 msgr2=0x7f57b006e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:05.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57af7fe640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57b006dd80 0x7f57b006e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.149+0000 7f57af7fe640 1 -- 172.21.15.107:0/2770404342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b01556c0 con 0x7f57b0071cd0 2023-10-24T23:59:05.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57af7fe640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57b0071cd0 0x7f57b0149820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f57a40119d0 tx=0x7f57a4011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:05.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57acff9640 1 -- 172.21.15.107:0/2770404342 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a400fad0 con 0x7f57b0071cd0 2023-10-24T23:59:05.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57acff9640 1 -- 172.21.15.107:0/2770404342 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a400fc30 con 0x7f57b0071cd0 2023-10-24T23:59:05.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57b65ce640 1 -- 172.21.15.107:0/2770404342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b0149d60 con 0x7f57b0071cd0 2023-10-24T23:59:05.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57acff9640 1 -- 172.21.15.107:0/2770404342 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a4014440 con 0x7f57b0071cd0 2023-10-24T23:59:05.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57b65ce640 1 -- 172.21.15.107:0/2770404342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b014a0a0 con 0x7f57b0071cd0 2023-10-24T23:59:05.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57b65ce640 1 -- 172.21.15.107:0/2770404342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b00746e0 con 0x7f57b0071cd0 2023-10-24T23:59:05.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57acff9640 1 -- 172.21.15.107:0/2770404342 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f57a40145a0 con 0x7f57b0071cd0 2023-10-24T23:59:05.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57acff9640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f579c065e90 0x7f579c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:05.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57acff9640 1 -- 172.21.15.107:0/2770404342 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7f57a4084780 con 0x7f57b0071cd0 2023-10-24T23:59:05.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57affff640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f579c065e90 0x7f579c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:05.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.153+0000 7f57affff640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f579c065e90 0x7f579c068350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f57a00ac000 tx=0x7f57a00a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:05.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.157+0000 7f57acff9640 1 -- 172.21.15.107:0/2770404342 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f57a40517b0 con 0x7f57b0071cd0 2023-10-24T23:59:05.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.365+0000 7f57b65ce640 1 -- 172.21.15.107:0/2770404342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57b0155850 con 0x7f57b0071cd0 2023-10-24T23:59:05.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.365+0000 7f57acff9640 1 -- 172.21.15.107:0/2770404342 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5017 (secure 0 0 0) 0x7f57a4054dd0 con 0x7f57b0071cd0 2023-10-24T23:59:05.373 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:05.373 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:05.373 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 59 2023-10-24T23:59:05.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 -- 172.21.15.107:0/2770404342 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f579c065e90 msgr2=0x7f579c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:05.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f579c065e90 0x7f579c068350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f57a00ac000 tx=0x7f57a00a6000 comp rx=0 tx=0).stop 2023-10-24T23:59:05.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 -- 172.21.15.107:0/2770404342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57b0071cd0 msgr2=0x7f57b0149820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:05.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57b0071cd0 0x7f57b0149820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f57a40119d0 tx=0x7f57a4011e90 comp rx=0 tx=0).stop 2023-10-24T23:59:05.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 -- 172.21.15.107:0/2770404342 shutdown_connections 2023-10-24T23:59:05.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57b006dd80 0x7f57b006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57b0071cd0 0x7f57b0149820 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f579c065e90 0x7f579c068350 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 --2- 172.21.15.107:0/2770404342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5798003b40 0x7f57b0154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:05.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 -- 172.21.15.107:0/2770404342 >> 172.21.15.107:0/2770404342 conn(0x7f57b00c39a0 msgr2=0x7f57b0158490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:05.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 -- 172.21.15.107:0/2770404342 shutdown_connections 2023-10-24T23:59:05.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:05.369+0000 7f57b65ce640 1 -- 172.21.15.107:0/2770404342 wait complete. 2023-10-24T23:59:05.391 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2023-10-24T23:59:05.391 INFO:tasks.mds_thrash.fs.[b]:waiting for 89 secs before thrashing 2023-10-24T23:59:05.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:59:05.471 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:59:07.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.189+0000 7f5f6c662640 1 Processor -- start 2023-10-24T23:59:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.189+0000 7f5f6c662640 1 -- start start 2023-10-24T23:59:07.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6c662640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5f64070b00 0x7f5f640c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6c662640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5f6406df30 0x7f5f6406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6c662640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f640c22f0 con 0x7f5f6414eb70 2023-10-24T23:59:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6c662640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f64071700 con 0x7f5f6406df30 2023-10-24T23:59:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6c662640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f640c25d0 con 0x7f5f64070b00 2023-10-24T23:59:07.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6abff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5f6406df30 0x7f5f6406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6abff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5f6406df30 0x7f5f6406e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39122/0 (socket says 172.21.15.107:39122) 2023-10-24T23:59:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6abff640 1 -- 172.21.15.107:0/655723813 learned_addr learned my addr 172.21.15.107:0/655723813 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f69bfd640 1 --2- 172.21.15.107:0/655723813 >> v2:172.21.15.110:3300/0 conn(0x7f5f64070b00 0x7f5f640c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6abff640 1 -- 172.21.15.107:0/655723813 >> v2:172.21.15.110:3300/0 conn(0x7f5f64070b00 msgr2=0x7f5f640c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6abff640 1 --2- 172.21.15.107:0/655723813 >> v2:172.21.15.110:3300/0 conn(0x7f5f64070b00 0x7f5f640c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6abff640 1 -- 172.21.15.107:0/655723813 >> v1:172.21.15.107:6789/0 conn(0x7f5f6414eb70 legacy=0x7f5f641514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6abff640 1 -- 172.21.15.107:0/655723813 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f640c2be0 con 0x7f5f6406df30 2023-10-24T23:59:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f693fc640 1 -- 172.21.15.107:0/655723813 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969366131 0 0) 0x7f5f640c22f0 con 0x7f5f6414eb70 2023-10-24T23:59:07.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f6abff640 1 --2- 172.21.15.107:0/655723813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f6406df30 0x7f5f6406e3a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5f640c1e70 tx=0x7f5f5c01ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2927e5b0d0126408 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:07.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f693fc640 1 -- 172.21.15.107:0/655723813 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c00b2d0 con 0x7f5f6406df30 2023-10-24T23:59:07.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f693fc640 1 -- 172.21.15.107:0/655723813 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c00b430 con 0x7f5f6406df30 2023-10-24T23:59:07.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.193+0000 7f5f693fc640 1 -- 172.21.15.107:0/655723813 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0a7be0 con 0x7f5f6406df30 2023-10-24T23:59:07.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 -- 172.21.15.107:0/655723813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f6406df30 msgr2=0x7f5f6406e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:07.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 --2- 172.21.15.107:0/655723813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f6406df30 0x7f5f6406e3a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f5f640c1e70 tx=0x7f5f5c01ebd0 comp rx=0 tx=0).stop 2023-10-24T23:59:07.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 -- 172.21.15.107:0/655723813 shutdown_connections 2023-10-24T23:59:07.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 --2- 172.21.15.107:0/655723813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f6406df30 0x7f5f6406e3a0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:07.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 --2- 172.21.15.107:0/655723813 >> v2:172.21.15.110:3300/0 conn(0x7f5f64070b00 0x7f5f640c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:07.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 -- 172.21.15.107:0/655723813 >> 172.21.15.107:0/655723813 conn(0x7f5f640c39a0 msgr2=0x7f5f640c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:07.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 -- 172.21.15.107:0/655723813 shutdown_connections 2023-10-24T23:59:07.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 -- 172.21.15.107:0/655723813 wait complete. 2023-10-24T23:59:07.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 Processor -- start 2023-10-24T23:59:07.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 -- start start 2023-10-24T23:59:07.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6406df30 0x7f5f640a9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f64070b00 0x7f5f64154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f6414eb70 0x7f5f64154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f64153970 con 0x7f5f6414eb70 2023-10-24T23:59:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6414a790 con 0x7f5f6406df30 2023-10-24T23:59:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f64149c10 con 0x7f5f64070b00 2023-10-24T23:59:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6abff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f6414eb70 0x7f5f64154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6abff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f6414eb70 0x7f5f64154640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39132/0 (socket says 172.21.15.107:39132) 2023-10-24T23:59:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6abff640 1 -- 172.21.15.107:0/2560601134 learned_addr learned my addr 172.21.15.107:0/2560601134 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6a3fe640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6406df30 0x7f5f640a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f69bfd640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f64070b00 0x7f5f64154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:07.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6abff640 1 -- 172.21.15.107:0/2560601134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f64070b00 msgr2=0x7f5f64154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:07.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6abff640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f64070b00 0x7f5f64154100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:07.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6abff640 1 -- 172.21.15.107:0/2560601134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6406df30 msgr2=0x7f5f640a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:07.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6abff640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6406df30 0x7f5f640a9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:07.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6abff640 1 -- 172.21.15.107:0/2560601134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c00bd40 con 0x7f5f6414eb70 2023-10-24T23:59:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6abff640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f6414eb70 0x7f5f64154640 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c01c800 tx=0x7f5f5c01b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f5affd640 1 -- 172.21.15.107:0/2560601134 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0bd070 con 0x7f5f6414eb70 2023-10-24T23:59:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f5affd640 1 -- 172.21.15.107:0/2560601134 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c009600 con 0x7f5f6414eb70 2023-10-24T23:59:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 -- 172.21.15.107:0/2560601134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f64149ef0 con 0x7f5f6414eb70 2023-10-24T23:59:07.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f5affd640 1 -- 172.21.15.107:0/2560601134 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0099b0 con 0x7f5f6414eb70 2023-10-24T23:59:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.197+0000 7f5f6c662640 1 -- 172.21.15.107:0/2560601134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6414a4b0 con 0x7f5f6414eb70 2023-10-24T23:59:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.201+0000 7f5f6c662640 1 -- 172.21.15.107:0/2560601134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f640746e0 con 0x7f5f6414eb70 2023-10-24T23:59:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.201+0000 7f5f5affd640 1 -- 172.21.15.107:0/2560601134 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5f5c009b10 con 0x7f5f6414eb70 2023-10-24T23:59:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.201+0000 7f5f5affd640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f48065e90 0x7f5f48068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:07.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.201+0000 7f5f5affd640 1 -- 172.21.15.107:0/2560601134 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7f5f5c11ebd0 con 0x7f5f6414eb70 2023-10-24T23:59:07.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.201+0000 7f5f6a3fe640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f48065e90 0x7f5f48068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:07.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.201+0000 7f5f6a3fe640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f48065e90 0x7f5f48068350 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5f60000f70 tx=0x7f5f60009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:07.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.205+0000 7f5f5affd640 1 -- 172.21.15.107:0/2560601134 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5f5c0ebb30 con 0x7f5f6414eb70 2023-10-24T23:59:07.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 -- 172.21.15.107:0/2560601134 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f640c1a50 con 0x7f5f6414eb70 2023-10-24T23:59:07.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f5affd640 1 -- 172.21.15.107:0/2560601134 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f5f5c0ef150 con 0x7f5f6414eb70 2023-10-24T23:59:07.438 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:07.438 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6102,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:07.438 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-10-24T23:59:07.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 -- 172.21.15.107:0/2560601134 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f48065e90 msgr2=0x7f5f48068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:07.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f48065e90 0x7f5f48068350 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5f60000f70 tx=0x7f5f60009040 comp rx=0 tx=0).stop 2023-10-24T23:59:07.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 -- 172.21.15.107:0/2560601134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f6414eb70 msgr2=0x7f5f64154640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:07.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f6414eb70 0x7f5f64154640 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c01c800 tx=0x7f5f5c01b970 comp rx=0 tx=0).stop 2023-10-24T23:59:07.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 -- 172.21.15.107:0/2560601134 shutdown_connections 2023-10-24T23:59:07.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f6414eb70 0x7f5f64154640 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:07.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f48065e90 0x7f5f48068350 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:07.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f64070b00 0x7f5f64154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:07.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 --2- 172.21.15.107:0/2560601134 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f6406df30 0x7f5f640a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:07.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 -- 172.21.15.107:0/2560601134 >> 172.21.15.107:0/2560601134 conn(0x7f5f640c39a0 msgr2=0x7f5f640c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 -- 172.21.15.107:0/2560601134 shutdown_connections 2023-10-24T23:59:07.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:07.433+0000 7f5f6c662640 1 -- 172.21.15.107:0/2560601134 wait complete. 2023-10-24T23:59:07.456 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2023-10-24T23:59:07.457 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2023-10-24T23:59:33.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:33.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7fa052f640 1 Processor -- start 2023-10-24T23:59:33.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7fa052f640 1 -- start start 2023-10-24T23:59:33.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7fa052f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7f98070b00 0x7f7f980c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7fa052f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7f9806df30 0x7f7f9806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7fa052f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f980c22f0 con 0x7f7f9814eb70 2023-10-24T23:59:33.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7fa052f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f98071700 con 0x7f7f98070b00 2023-10-24T23:59:33.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7fa052f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f980c25d0 con 0x7f7f9806df30 2023-10-24T23:59:33.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7f9daca640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7f98070b00 0x7f7f980c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7f9e2cb640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7f9814eb70 0x7f7f981514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57054/0 (socket says 172.21.15.107:57054) 2023-10-24T23:59:33.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7f9e2cb640 1 -- 172.21.15.107:0/1817543878 learned_addr learned my addr 172.21.15.107:0/1817543878 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:59:33.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7f9eacc640 1 --2- 172.21.15.107:0/1817543878 >> v2:172.21.15.110:3300/0 conn(0x7f7f9806df30 0x7f7f9806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7f9daca640 1 -- 172.21.15.107:0/1817543878 >> v2:172.21.15.110:3300/0 conn(0x7f7f9806df30 msgr2=0x7f7f9806e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:33.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7f9daca640 1 --2- 172.21.15.107:0/1817543878 >> v2:172.21.15.110:3300/0 conn(0x7f7f9806df30 0x7f7f9806e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.593+0000 7f7f9daca640 1 -- 172.21.15.107:0/1817543878 >> v1:172.21.15.107:6789/0 conn(0x7f7f9814eb70 legacy=0x7f7f981514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:33.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7f9d2c9640 1 -- 172.21.15.107:0/1817543878 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321673157 0 0) 0x7f7f980c22f0 con 0x7f7f9814eb70 2023-10-24T23:59:33.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7f9daca640 1 -- 172.21.15.107:0/1817543878 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f980c2be0 con 0x7f7f98070b00 2023-10-24T23:59:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7f9daca640 1 --2- 172.21.15.107:0/1817543878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f98070b00 0x7f7f980c33e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7f981525e0 tx=0x7f7f8801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722c3c68a3b4ad10 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7f9d2c9640 1 -- 172.21.15.107:0/1817543878 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f880ab680 con 0x7f7f98070b00 2023-10-24T23:59:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7f9d2c9640 1 -- 172.21.15.107:0/1817543878 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f880ab7e0 con 0x7f7f98070b00 2023-10-24T23:59:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7f9d2c9640 1 -- 172.21.15.107:0/1817543878 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f880abb70 con 0x7f7f98070b00 2023-10-24T23:59:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7fa052f640 1 -- 172.21.15.107:0/1817543878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f98070b00 msgr2=0x7f7f980c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7fa052f640 1 --2- 172.21.15.107:0/1817543878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f98070b00 0x7f7f980c33e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7f981525e0 tx=0x7f7f8801c460 comp rx=0 tx=0).stop 2023-10-24T23:59:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7fa052f640 1 -- 172.21.15.107:0/1817543878 shutdown_connections 2023-10-24T23:59:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7fa052f640 1 --2- 172.21.15.107:0/1817543878 >> v2:172.21.15.110:3300/0 conn(0x7f7f9806df30 0x7f7f9806e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7fa052f640 1 --2- 172.21.15.107:0/1817543878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f98070b00 0x7f7f980c33e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7fa052f640 1 -- 172.21.15.107:0/1817543878 >> 172.21.15.107:0/1817543878 conn(0x7f7f980c39a0 msgr2=0x7f7f980c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:33.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7fa052f640 1 -- 172.21.15.107:0/1817543878 shutdown_connections 2023-10-24T23:59:33.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.597+0000 7f7fa052f640 1 -- 172.21.15.107:0/1817543878 wait complete. 2023-10-24T23:59:33.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7fa052f640 1 Processor -- start 2023-10-24T23:59:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7fa052f640 1 -- start start 2023-10-24T23:59:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7fa052f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f9806df30 0x7f7f98144cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7fa052f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f98070b00 0x7f7f9813c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7fa052f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9814eb70 0x7f7f9813cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7fa052f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f981e9840 con 0x7f7f9814eb70 2023-10-24T23:59:33.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7fa052f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9813d190 con 0x7f7f98070b00 2023-10-24T23:59:33.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7fa052f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f981426d0 con 0x7f7f9806df30 2023-10-24T23:59:33.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f9eacc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9814eb70 0x7f7f9813cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f9eacc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9814eb70 0x7f7f9813cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35818/0 (socket says 172.21.15.107:35818) 2023-10-24T23:59:33.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f9eacc640 1 -- 172.21.15.107:0/4260064126 learned_addr learned my addr 172.21.15.107:0/4260064126 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:33.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f9e2cb640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f9806df30 0x7f7f98144cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f9daca640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f98070b00 0x7f7f9813c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f9eacc640 1 -- 172.21.15.107:0/4260064126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f9806df30 msgr2=0x7f7f98144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:33.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f9eacc640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f9806df30 0x7f7f98144cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f9eacc640 1 -- 172.21.15.107:0/4260064126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f98070b00 msgr2=0x7f7f9813c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f9eacc640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f98070b00 0x7f7f9813c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f9eacc640 1 -- 172.21.15.107:0/4260064126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f88009420 con 0x7f7f9814eb70 2023-10-24T23:59:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f9eacc640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9814eb70 0x7f7f9813cc50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000efc0 tx=0x7f7f9000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f8effd640 1 -- 172.21.15.107:0/4260064126 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90009ba0 con 0x7f7f9814eb70 2023-10-24T23:59:33.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f8effd640 1 -- 172.21.15.107:0/4260064126 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9000f040 con 0x7f7f9814eb70 2023-10-24T23:59:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7f8effd640 1 -- 172.21.15.107:0/4260064126 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f900146c0 con 0x7f7f9814eb70 2023-10-24T23:59:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7fa052f640 1 -- 172.21.15.107:0/4260064126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f981429b0 con 0x7f7f9814eb70 2023-10-24T23:59:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7fa052f640 1 -- 172.21.15.107:0/4260064126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f98142f70 con 0x7f7f9814eb70 2023-10-24T23:59:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.601+0000 7f7fa052f640 1 -- 172.21.15.107:0/4260064126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f980746e0 con 0x7f7f9814eb70 2023-10-24T23:59:33.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.605+0000 7f7f8effd640 1 -- 172.21.15.107:0/4260064126 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7f90014820 con 0x7f7f9814eb70 2023-10-24T23:59:33.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.605+0000 7f7f8effd640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f7c065e60 0x7f7f7c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.605+0000 7f7f8effd640 1 -- 172.21.15.107:0/4260064126 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7f7f90084990 con 0x7f7f9814eb70 2023-10-24T23:59:33.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.605+0000 7f7f9e2cb640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f7c065e60 0x7f7f7c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.605+0000 7f7f9e2cb640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f7c065e60 0x7f7f7c068320 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7f94000f70 tx=0x7f7f94007b80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:33.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.609+0000 7f7f8effd640 1 -- 172.21.15.107:0/4260064126 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7f90051990 con 0x7f7f9814eb70 2023-10-24T23:59:33.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.829+0000 7f7fa052f640 1 -- 172.21.15.107:0/4260064126 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f98143160 con 0x7f7f9814eb70 2023-10-24T23:59:33.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.829+0000 7f7f8effd640 1 -- 172.21.15.107:0/4260064126 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f7f90018070 con 0x7f7f9814eb70 2023-10-24T23:59:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:33.835 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6102,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:33.835 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-10-24T23:59:33.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 -- 172.21.15.107:0/4260064126 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f7c065e60 msgr2=0x7f7f7c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:33.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f7c065e60 0x7f7f7c068320 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7f94000f70 tx=0x7f7f94007b80 comp rx=0 tx=0).stop 2023-10-24T23:59:33.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 -- 172.21.15.107:0/4260064126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9814eb70 msgr2=0x7f7f9813cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:33.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9814eb70 0x7f7f9813cc50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000efc0 tx=0x7f7f9000c490 comp rx=0 tx=0).stop 2023-10-24T23:59:33.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 -- 172.21.15.107:0/4260064126 shutdown_connections 2023-10-24T23:59:33.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f9814eb70 0x7f7f9813cc50 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f98070b00 0x7f7f9813c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7f7c065e60 0x7f7f7c068320 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 --2- 172.21.15.107:0/4260064126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7f9806df30 0x7f7f98144cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 -- 172.21.15.107:0/4260064126 >> 172.21.15.107:0/4260064126 conn(0x7f7f980c39a0 msgr2=0x7f7f980c55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:33.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 -- 172.21.15.107:0/4260064126 shutdown_connections 2023-10-24T23:59:33.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.833+0000 7f7fa052f640 1 -- 172.21.15.107:0/4260064126 wait complete. 2023-10-24T23:59:33.854 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2023-10-24T23:59:33.855 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-24T23:59:33.855 INFO:teuthology.orchestra.run:waiting for 300 2023-10-24T23:59:33.870 INFO:tasks.ceph.mds.b:Stopped 2023-10-24T23:59:33.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:33.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.977+0000 7f31f83e8640 1 Processor -- start 2023-10-24T23:59:33.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.977+0000 7f31f83e8640 1 -- start start 2023-10-24T23:59:33.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.977+0000 7f31f83e8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31f00aa140 0x7f31f00aa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.977+0000 7f31f83e8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f31f00ac020 0x7f31f00c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.977+0000 7f31f83e8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f0066000 con 0x7f31f00c2520 2023-10-24T23:59:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.977+0000 7f31f83e8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f00aaa60 con 0x7f31f00ac020 2023-10-24T23:59:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.977+0000 7f31f83e8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00c0f40 con 0x7f31f00aa140 2023-10-24T23:59:33.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.977+0000 7f31f5983640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f31f00ac020 0x7f31f00c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.977+0000 7f31f5983640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f31f00ac020 0x7f31f00c1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35834/0 (socket says 172.21.15.107:35834) 2023-10-24T23:59:33.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.977+0000 7f31f5983640 1 -- 172.21.15.107:0/2944022467 learned_addr learned my addr 172.21.15.107:0/2944022467 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:33.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.977+0000 7f31f6184640 1 --2- 172.21.15.107:0/2944022467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31f00aa140 0x7f31f00aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f5983640 1 -- 172.21.15.107:0/2944022467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31f00aa140 msgr2=0x7f31f00aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:33.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f5983640 1 --2- 172.21.15.107:0/2944022467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31f00aa140 0x7f31f00aa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f5983640 1 -- 172.21.15.107:0/2944022467 >> v1:172.21.15.110:6789/0 conn(0x7f31f00c2520 legacy=0x7f31f0154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:33.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f5983640 1 -- 172.21.15.107:0/2944022467 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f00c1220 con 0x7f31f00ac020 2023-10-24T23:59:33.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f5983640 1 --2- 172.21.15.107:0/2944022467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f00ac020 0x7f31f00c1fe0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009870 tx=0x7f31e001e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5124d9b550106a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:33.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f5182640 1 -- 172.21.15.107:0/2944022467 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e00aa660 con 0x7f31f00ac020 2023-10-24T23:59:33.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f5182640 1 -- 172.21.15.107:0/2944022467 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e00aa7c0 con 0x7f31f00ac020 2023-10-24T23:59:33.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f5182640 1 -- 172.21.15.107:0/2944022467 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e00aab50 con 0x7f31f00ac020 2023-10-24T23:59:33.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f83e8640 1 -- 172.21.15.107:0/2944022467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f00ac020 msgr2=0x7f31f00c1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:33.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f83e8640 1 --2- 172.21.15.107:0/2944022467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f00ac020 0x7f31f00c1fe0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009870 tx=0x7f31e001e720 comp rx=0 tx=0).stop 2023-10-24T23:59:33.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f83e8640 1 -- 172.21.15.107:0/2944022467 shutdown_connections 2023-10-24T23:59:33.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f83e8640 1 --2- 172.21.15.107:0/2944022467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f00ac020 0x7f31f00c1fe0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f83e8640 1 --2- 172.21.15.107:0/2944022467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31f00aa140 0x7f31f00aa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f83e8640 1 -- 172.21.15.107:0/2944022467 >> 172.21.15.107:0/2944022467 conn(0x7f31f00c65c0 msgr2=0x7f31f00c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:33.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f83e8640 1 -- 172.21.15.107:0/2944022467 shutdown_connections 2023-10-24T23:59:33.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.981+0000 7f31f83e8640 1 -- 172.21.15.107:0/2944022467 wait complete. 2023-10-24T23:59:33.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f83e8640 1 Processor -- start 2023-10-24T23:59:33.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f83e8640 1 -- start start 2023-10-24T23:59:33.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f83e8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31f00aa140 0x7f31f006f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f83e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f00ac020 0x7f31f014bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f83e8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f00c2520 0x7f31f01e8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f83e8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f00c38b0 con 0x7f31f00ac020 2023-10-24T23:59:33.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f83e8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f014f650 con 0x7f31f00c2520 2023-10-24T23:59:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f83e8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f01e8670 con 0x7f31f00aa140 2023-10-24T23:59:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f5983640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f00ac020 0x7f31f014bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f5983640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f00ac020 0x7f31f014bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35838/0 (socket says 172.21.15.107:35838) 2023-10-24T23:59:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f5983640 1 -- 172.21.15.107:0/3039318658 learned_addr learned my addr 172.21.15.107:0/3039318658 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f6184640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31f00aa140 0x7f31f006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f6985640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f00c2520 0x7f31f01e8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f5983640 1 -- 172.21.15.107:0/3039318658 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31f00aa140 msgr2=0x7f31f006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:33.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f5983640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31f00aa140 0x7f31f006f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f5983640 1 -- 172.21.15.107:0/3039318658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f00c2520 msgr2=0x7f31f01e8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:33.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f5983640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f00c2520 0x7f31f01e8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:33.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f5983640 1 -- 172.21.15.107:0/3039318658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e00094d0 con 0x7f31f00ac020 2023-10-24T23:59:33.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f5983640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f00ac020 0x7f31f014bd60 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f31e001cdd0 tx=0x7f31e001b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:33.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31e6ffd640 1 -- 172.21.15.107:0/3039318658 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e00b2810 con 0x7f31f00ac020 2023-10-24T23:59:33.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31e6ffd640 1 -- 172.21.15.107:0/3039318658 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e00b2970 con 0x7f31f00ac020 2023-10-24T23:59:33.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f83e8640 1 -- 172.21.15.107:0/3039318658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f014d140 con 0x7f31f00ac020 2023-10-24T23:59:33.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31e6ffd640 1 -- 172.21.15.107:0/3039318658 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e00b2d20 con 0x7f31f00ac020 2023-10-24T23:59:33.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f83e8640 1 -- 172.21.15.107:0/3039318658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f01e8930 con 0x7f31f00ac020 2023-10-24T23:59:33.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.985+0000 7f31f83e8640 1 -- 172.21.15.107:0/3039318658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f00746e0 con 0x7f31f00ac020 2023-10-24T23:59:33.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.989+0000 7f31e6ffd640 1 -- 172.21.15.107:0/3039318658 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f31e00c5440 con 0x7f31f00ac020 2023-10-24T23:59:33.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.989+0000 7f31e6ffd640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f31d4065e60 0x7f31d4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:33.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.989+0000 7f31e6ffd640 1 -- 172.21.15.107:0/3039318658 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7f31e0120fa0 con 0x7f31f00ac020 2023-10-24T23:59:33.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.989+0000 7f31f6184640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f31d4065e60 0x7f31d4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:33.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.989+0000 7f31f6184640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f31d4065e60 0x7f31d4068320 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f31f00c5990 tx=0x7f31ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:33.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:33.993+0000 7f31e6ffd640 1 -- 172.21.15.107:0/3039318658 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f31e00ee080 con 0x7f31f00ac020 2023-10-24T23:59:34.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 -- 172.21.15.107:0/3039318658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31f00a7f90 con 0x7f31f00ac020 2023-10-24T23:59:34.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31e6ffd640 1 -- 172.21.15.107:0/3039318658 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f31e00aa620 con 0x7f31f00ac020 2023-10-24T23:59:34.229 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:34.229 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6102,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:34.230 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-10-24T23:59:34.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 -- 172.21.15.107:0/3039318658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f31d4065e60 msgr2=0x7f31d4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:34.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f31d4065e60 0x7f31d4068320 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f31f00c5990 tx=0x7f31ec008040 comp rx=0 tx=0).stop 2023-10-24T23:59:34.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 -- 172.21.15.107:0/3039318658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f00ac020 msgr2=0x7f31f014bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:34.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f00ac020 0x7f31f014bd60 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f31e001cdd0 tx=0x7f31e001b4c0 comp rx=0 tx=0).stop 2023-10-24T23:59:34.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 -- 172.21.15.107:0/3039318658 shutdown_connections 2023-10-24T23:59:34.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31f00c2520 0x7f31f01e8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:34.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31f00ac020 0x7f31f014bd60 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:34.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f31d4065e60 0x7f31d4068320 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:34.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 --2- 172.21.15.107:0/3039318658 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f31f00aa140 0x7f31f006f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:34.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 -- 172.21.15.107:0/3039318658 >> 172.21.15.107:0/3039318658 conn(0x7f31f00c65c0 msgr2=0x7f31f00c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:34.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 -- 172.21.15.107:0/3039318658 shutdown_connections 2023-10-24T23:59:34.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:34.225+0000 7f31f83e8640 1 -- 172.21.15.107:0/3039318658 wait complete. 2023-10-24T23:59:34.254 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:59:35.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:59:35.594 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-24T23:59:36.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:36.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b88cf3640 1 Processor -- start 2023-10-24T23:59:36.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b88cf3640 1 -- start start 2023-10-24T23:59:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b88cf3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2b8414b600 0x7f2b84151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b88cf3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8406a230 0x7f2b841551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b88cf3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b840c1860 con 0x7f2b8406af70 2023-10-24T23:59:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b88cf3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b84154d50 con 0x7f2b8414b600 2023-10-24T23:59:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b88cf3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b841524f0 con 0x7f2b8406a230 2023-10-24T23:59:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b8259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2b8414b600 0x7f2b84151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b7bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2b8406af70 0x7f2b8406c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57060/0 (socket says 172.21.15.107:57060) 2023-10-24T23:59:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b7bfff640 1 -- 172.21.15.107:0/2594174533 learned_addr learned my addr 172.21.15.107:0/2594174533 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:59:36.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b82d9d640 1 --2- 172.21.15.107:0/2594174533 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8406a230 0x7f2b841551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b8259c640 1 -- 172.21.15.107:0/2594174533 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8406a230 msgr2=0x7f2b841551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b8259c640 1 --2- 172.21.15.107:0/2594174533 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8406a230 0x7f2b841551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b8259c640 1 -- 172.21.15.107:0/2594174533 >> v1:172.21.15.107:6789/0 conn(0x7f2b8406af70 legacy=0x7f2b8406c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b81d9b640 1 -- 172.21.15.107:0/2594174533 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505268022 0 0) 0x7f2b840c1860 con 0x7f2b8406af70 2023-10-24T23:59:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.381+0000 7f2b8259c640 1 -- 172.21.15.107:0/2594174533 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8414b9e0 con 0x7f2b8414b600 2023-10-24T23:59:36.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b8259c640 1 --2- 172.21.15.107:0/2594174533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b8414b600 0x7f2b84151fb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2b740097c0 tx=0x7f2b7401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b4cc3cf4c78fc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:36.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b81d9b640 1 -- 172.21.15.107:0/2594174533 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b740ab600 con 0x7f2b8414b600 2023-10-24T23:59:36.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b81d9b640 1 -- 172.21.15.107:0/2594174533 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b740a7050 con 0x7f2b8414b600 2023-10-24T23:59:36.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b81d9b640 1 -- 172.21.15.107:0/2594174533 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b740ab980 con 0x7f2b8414b600 2023-10-24T23:59:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 -- 172.21.15.107:0/2594174533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b8414b600 msgr2=0x7f2b84151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 --2- 172.21.15.107:0/2594174533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b8414b600 0x7f2b84151fb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2b740097c0 tx=0x7f2b7401c3c0 comp rx=0 tx=0).stop 2023-10-24T23:59:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 -- 172.21.15.107:0/2594174533 shutdown_connections 2023-10-24T23:59:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 --2- 172.21.15.107:0/2594174533 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8406a230 0x7f2b841551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 --2- 172.21.15.107:0/2594174533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b8414b600 0x7f2b84151fb0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 -- 172.21.15.107:0/2594174533 >> 172.21.15.107:0/2594174533 conn(0x7f2b840c2950 msgr2=0x7f2b840c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 -- 172.21.15.107:0/2594174533 shutdown_connections 2023-10-24T23:59:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 -- 172.21.15.107:0/2594174533 wait complete. 2023-10-24T23:59:36.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 Processor -- start 2023-10-24T23:59:36.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 -- start start 2023-10-24T23:59:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b8406a230 0x7f2b841ec940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8406af70 0x7f2b841f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8414b600 0x7f2b84143420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b84152f60 con 0x7f2b8406a230 2023-10-24T23:59:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b841440e0 con 0x7f2b8406af70 2023-10-24T23:59:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b88cf3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b841efeb0 con 0x7f2b8414b600 2023-10-24T23:59:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b8259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b8406a230 0x7f2b841ec940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b8259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b8406a230 0x7f2b841ec940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35862/0 (socket says 172.21.15.107:35862) 2023-10-24T23:59:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b8259c640 1 -- 172.21.15.107:0/604649017 learned_addr learned my addr 172.21.15.107:0/604649017 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:36.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b82d9d640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8414b600 0x7f2b84143420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b7bfff640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8406af70 0x7f2b841f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b8259c640 1 -- 172.21.15.107:0/604649017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8414b600 msgr2=0x7f2b84143420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b8259c640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8414b600 0x7f2b84143420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b8259c640 1 -- 172.21.15.107:0/604649017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8406af70 msgr2=0x7f2b841f05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:36.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b8259c640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8406af70 0x7f2b841f05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:36.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.385+0000 7f2b8259c640 1 -- 172.21.15.107:0/604649017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b74009420 con 0x7f2b8406a230 2023-10-24T23:59:36.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b8259c640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b8406a230 0x7f2b841ec940 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f2b74019ff0 tx=0x7f2b74019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:36.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b7affd640 1 -- 172.21.15.107:0/604649017 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b740b4610 con 0x7f2b8406a230 2023-10-24T23:59:36.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b7affd640 1 -- 172.21.15.107:0/604649017 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b740b4790 con 0x7f2b8406a230 2023-10-24T23:59:36.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b88cf3640 1 -- 172.21.15.107:0/604649017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b841f0150 con 0x7f2b8406a230 2023-10-24T23:59:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b7affd640 1 -- 172.21.15.107:0/604649017 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b740b4b40 con 0x7f2b8406a230 2023-10-24T23:59:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b88cf3640 1 -- 172.21.15.107:0/604649017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b84071a90 con 0x7f2b8406a230 2023-10-24T23:59:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b88cf3640 1 -- 172.21.15.107:0/604649017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b840718f0 con 0x7f2b8406a230 2023-10-24T23:59:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b7affd640 1 -- 172.21.15.107:0/604649017 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2b740b2040 con 0x7f2b8406a230 2023-10-24T23:59:36.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b7affd640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2b64065e60 0x7f2b64068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:36.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b7affd640 1 -- 172.21.15.107:0/604649017 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7f2b7411e380 con 0x7f2b8406a230 2023-10-24T23:59:36.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b7bfff640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2b64065e60 0x7f2b64068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:36.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.389+0000 7f2b7bfff640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2b64065e60 0x7f2b64068320 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2b8414ed60 tx=0x7f2b6c007b60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:36.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.393+0000 7f2b7affd640 1 -- 172.21.15.107:0/604649017 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2b740eb460 con 0x7f2b8406a230 2023-10-24T23:59:36.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.585+0000 7f2b88cf3640 1 -- 172.21.15.107:0/604649017 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b840718f0 con 0x7f2b8406a230 2023-10-24T23:59:36.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b7affd640 1 -- 172.21.15.107:0/604649017 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f2b840718f0 con 0x7f2b8406a230 2023-10-24T23:59:36.588 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:36.589 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6102,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:36.589 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-10-24T23:59:36.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 -- 172.21.15.107:0/604649017 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2b64065e60 msgr2=0x7f2b64068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:36.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2b64065e60 0x7f2b64068320 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f2b8414ed60 tx=0x7f2b6c007b60 comp rx=0 tx=0).stop 2023-10-24T23:59:36.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 -- 172.21.15.107:0/604649017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b8406a230 msgr2=0x7f2b841ec940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:36.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b8406a230 0x7f2b841ec940 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f2b74019ff0 tx=0x7f2b74019160 comp rx=0 tx=0).stop 2023-10-24T23:59:36.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 -- 172.21.15.107:0/604649017 shutdown_connections 2023-10-24T23:59:36.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2b64065e60 0x7f2b64068320 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:36.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2b8414b600 0x7f2b84143420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:36.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2b8406af70 0x7f2b841f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:36.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 --2- 172.21.15.107:0/604649017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b8406a230 0x7f2b841ec940 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:36.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 -- 172.21.15.107:0/604649017 >> 172.21.15.107:0/604649017 conn(0x7f2b840c2950 msgr2=0x7f2b840bf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:36.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 -- 172.21.15.107:0/604649017 shutdown_connections 2023-10-24T23:59:36.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:36.589+0000 7f2b88cf3640 1 -- 172.21.15.107:0/604649017 wait complete. 2023-10-24T23:59:36.607 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:59:38.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:38.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.741+0000 7f0a01248640 1 Processor -- start 2023-10-24T23:59:38.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.741+0000 7f0a01248640 1 -- start start 2023-10-24T23:59:38.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.741+0000 7f0a01248640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09fc0a9820 0x7f09fc0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:38.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.741+0000 7f0a01248640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f09fc06ee40 0x7f09fc0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:38.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.741+0000 7f0a01248640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09fc06f730 con 0x7f09fc14fd40 2023-10-24T23:59:38.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.741+0000 7f0a01248640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09fc0aa750 con 0x7f09fc06ee40 2023-10-24T23:59:38.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.741+0000 7f0a01248640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc156a40 con 0x7f09fc0a9820 2023-10-24T23:59:38.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f09fc14fd40 0x7f09fc0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57064/0 (socket says 172.21.15.107:57064) 2023-10-24T23:59:38.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fad9d640 1 -- 172.21.15.107:0/892325024 learned_addr learned my addr 172.21.15.107:0/892325024 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:59:38.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fa59c640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09fc0a9820 0x7f09fc0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:38.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fb59e640 1 --2- 172.21.15.107:0/892325024 >> v2:172.21.15.110:3300/0 conn(0x7f09fc06ee40 0x7f09fc0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:38.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82485232 0 0) 0x7f09fc06f730 con 0x7f09fc14fd40 2023-10-24T23:59:38.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e4003530 con 0x7f09fc14fd40 2023-10-24T23:59:38.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1378107948 0 0) 0x7f09e4003530 con 0x7f09fc14fd40 2023-10-24T23:59:38.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09fc0a9820 msgr2=0x7f09fc0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:38.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09fc0a9820 0x7f09fc0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:38.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 >> v2:172.21.15.110:3300/0 conn(0x7f09fc06ee40 msgr2=0x7f09fc0aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:38.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 --2- 172.21.15.107:0/892325024 >> v2:172.21.15.110:3300/0 conn(0x7f09fc06ee40 0x7f09fc0aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:38.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc156710 con 0x7f09fc14fd40 2023-10-24T23:59:38.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f09ec001680 con 0x7f09fc14fd40 2023-10-24T23:59:38.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 >> v1:172.21.15.107:6789/0 conn(0x7f09fc14fd40 legacy=0x7f09fc0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:38.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09e4003980 0x7f09e4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:38.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e40047d0 0x7f09e4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:38.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09e4005150 0x7f09e400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:38.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0aa750 con 0x7f09e4005150 2023-10-24T23:59:38.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc156a40 con 0x7f09e4003980 2023-10-24T23:59:38.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e400bed0 con 0x7f09e40047d0 2023-10-24T23:59:38.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fb59e640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09e4005150 0x7f09e400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:38.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fad9d640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09e4003980 0x7f09e4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:38.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fa59c640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e40047d0 0x7f09e4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:38.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fb59e640 1 -- 172.21.15.107:0/892325024 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e40047d0 msgr2=0x7f09e4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:38.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fb59e640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09e40047d0 0x7f09e4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:38.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fb59e640 1 -- 172.21.15.107:0/892325024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09e4003980 msgr2=0x7f09e4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:38.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fb59e640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09e4003980 0x7f09e4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:38.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fb59e640 1 -- 172.21.15.107:0/892325024 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e4003530 con 0x7f09e4005150 2023-10-24T23:59:38.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09fb59e640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09e4005150 0x7f09e400b990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009fc0 tx=0x7f09f401cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d244dcea081590 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:38.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f40a5c00 con 0x7f09e4005150 2023-10-24T23:59:38.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f400f040 con 0x7f09e4005150 2023-10-24T23:59:38.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.745+0000 7f09f9d9b640 1 -- 172.21.15.107:0/892325024 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f40ad440 con 0x7f09e4005150 2023-10-24T23:59:38.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- 172.21.15.107:0/892325024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09e4005150 msgr2=0x7f09e400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:38.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 --2- 172.21.15.107:0/892325024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09e4005150 0x7f09e400b990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009fc0 tx=0x7f09f401cc70 comp rx=0 tx=0).stop 2023-10-24T23:59:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fb59e640 1 -- 172.21.15.107:0/892325024 reap_dead start 2023-10-24T23:59:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- 172.21.15.107:0/892325024 shutdown_connections 2023-10-24T23:59:38.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- 172.21.15.107:0/892325024 >> 172.21.15.107:0/892325024 conn(0x7f09fc0c2950 msgr2=0x7f09fc0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:38.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- 172.21.15.107:0/892325024 shutdown_connections 2023-10-24T23:59:38.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- 172.21.15.107:0/892325024 wait complete. 2023-10-24T23:59:38.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 Processor -- start 2023-10-24T23:59:38.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- start start 2023-10-24T23:59:38.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09fc14ca30 0x7f09fc153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:38.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09fc1557e0 0x7f09fc06a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:38.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09fc06ac70 0x7f09fc148720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:38.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc153b10 con 0x7f09fc06ac70 2023-10-24T23:59:38.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc1f0a70 con 0x7f09fc14ca30 2023-10-24T23:59:38.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc155150 con 0x7f09fc1557e0 2023-10-24T23:59:38.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09fc06ac70 0x7f09fc148720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:38.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09fc06ac70 0x7f09fc148720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35878/0 (socket says 172.21.15.107:35878) 2023-10-24T23:59:38.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fb59e640 1 -- 172.21.15.107:0/3679979899 learned_addr learned my addr 172.21.15.107:0/3679979899 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:38.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fad9d640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09fc14ca30 0x7f09fc153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:38.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fa59c640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09fc1557e0 0x7f09fc06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:38.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fb59e640 1 -- 172.21.15.107:0/3679979899 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09fc1557e0 msgr2=0x7f09fc06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:38.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fb59e640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09fc1557e0 0x7f09fc06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:38.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fb59e640 1 -- 172.21.15.107:0/3679979899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09fc14ca30 msgr2=0x7f09fc153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:38.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fb59e640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09fc14ca30 0x7f09fc153f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:38.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fb59e640 1 -- 172.21.15.107:0/3679979899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc1f0f60 con 0x7f09fc06ac70 2023-10-24T23:59:38.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09fb59e640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09fc06ac70 0x7f09fc148720 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f09f40b0040 tx=0x7f09f4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:38.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09e37fe640 1 -- 172.21.15.107:0/3679979899 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f40039a0 con 0x7f09fc06ac70 2023-10-24T23:59:38.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09e37fe640 1 -- 172.21.15.107:0/3679979899 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f4003b20 con 0x7f09fc06ac70 2023-10-24T23:59:38.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- 172.21.15.107:0/3679979899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09fc069e70 con 0x7f09fc06ac70 2023-10-24T23:59:38.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f09e37fe640 1 -- 172.21.15.107:0/3679979899 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4004030 con 0x7f09fc06ac70 2023-10-24T23:59:38.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- 172.21.15.107:0/3679979899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09fc0bf280 con 0x7f09fc06ac70 2023-10-24T23:59:38.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.749+0000 7f0a01248640 1 -- 172.21.15.107:0/3679979899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc153c50 con 0x7f09fc06ac70 2023-10-24T23:59:38.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.753+0000 7f09e37fe640 1 -- 172.21.15.107:0/3679979899 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f09f40b4040 con 0x7f09fc06ac70 2023-10-24T23:59:38.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.753+0000 7f09e37fe640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09e8065e90 0x7f09e8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:38.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.753+0000 7f09e37fe640 1 -- 172.21.15.107:0/3679979899 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7f09f411d5f0 con 0x7f09fc06ac70 2023-10-24T23:59:38.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.753+0000 7f09fad9d640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09e8065e90 0x7f09e8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:38.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.753+0000 7f09fad9d640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09e8065e90 0x7f09e8068350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f09ec001210 tx=0x7f09ec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:38.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.757+0000 7f09e37fe640 1 -- 172.21.15.107:0/3679979899 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f09f40ea680 con 0x7f09fc06ac70 2023-10-24T23:59:38.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 -- 172.21.15.107:0/3679979899 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09fc153c50 con 0x7f09fc06ac70 2023-10-24T23:59:38.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f09e37fe640 1 -- 172.21.15.107:0/3679979899 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f09fc153c50 con 0x7f09fc06ac70 2023-10-24T23:59:38.949 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:38.950 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6102,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:38.950 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-10-24T23:59:38.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 -- 172.21.15.107:0/3679979899 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09e8065e90 msgr2=0x7f09e8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:38.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09e8065e90 0x7f09e8068350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f09ec001210 tx=0x7f09ec009040 comp rx=0 tx=0).stop 2023-10-24T23:59:38.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 -- 172.21.15.107:0/3679979899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09fc06ac70 msgr2=0x7f09fc148720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:38.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09fc06ac70 0x7f09fc148720 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f09f40b0040 tx=0x7f09f4009750 comp rx=0 tx=0).stop 2023-10-24T23:59:38.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 -- 172.21.15.107:0/3679979899 shutdown_connections 2023-10-24T23:59:38.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09fc06ac70 0x7f09fc148720 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:38.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09e8065e90 0x7f09e8068350 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:38.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09fc1557e0 0x7f09fc06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:38.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 --2- 172.21.15.107:0/3679979899 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09fc14ca30 0x7f09fc153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:38.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 -- 172.21.15.107:0/3679979899 >> 172.21.15.107:0/3679979899 conn(0x7f09fc0c2950 msgr2=0x7f09fc0c0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:38.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 -- 172.21.15.107:0/3679979899 shutdown_connections 2023-10-24T23:59:38.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:38.949+0000 7f0a01248640 1 -- 172.21.15.107:0/3679979899 wait complete. 2023-10-24T23:59:38.967 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:59:40.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:41.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620d060640 1 Processor -- start 2023-10-24T23:59:41.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620d060640 1 -- start start 2023-10-24T23:59:41.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620d060640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f62080c8100 0x7f62080abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:41.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620d060640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f620806a4b0 0x7f62080ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620d060640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6208155680 con 0x7f6208152040 2023-10-24T23:59:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620d060640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6208154a70 con 0x7f62080c8100 2023-10-24T23:59:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620d060640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6208154d50 con 0x7f620806a4b0 2023-10-24T23:59:41.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f62080c8100 0x7f62080abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f6206d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6208152040 0x7f62080c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57076/0 (socket says 172.21.15.107:57076) 2023-10-24T23:59:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f6206d9d640 1 -- 172.21.15.107:0/3924689006 learned_addr learned my addr 172.21.15.107:0/3924689006 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:59:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620759e640 1 --2- 172.21.15.107:0/3924689006 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f620806a4b0 0x7f62080ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620659c640 1 -- 172.21.15.107:0/3924689006 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f620806a4b0 msgr2=0x7f62080ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620659c640 1 --2- 172.21.15.107:0/3924689006 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f620806a4b0 0x7f62080ac240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:41.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f6205d9b640 1 -- 172.21.15.107:0/3924689006 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878044390 0 0) 0x7f6208155680 con 0x7f6208152040 2023-10-24T23:59:41.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620659c640 1 -- 172.21.15.107:0/3924689006 >> v1:172.21.15.107:6789/0 conn(0x7f6208152040 legacy=0x7f62080c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:41.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.097+0000 7f620659c640 1 -- 172.21.15.107:0/3924689006 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6208155030 con 0x7f62080c8100 2023-10-24T23:59:41.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620659c640 1 --2- 172.21.15.107:0/3924689006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62080c8100 0x7f62080abd00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0097c0 tx=0x7f61fc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ebf836c23db13d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:41.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6205d9b640 1 -- 172.21.15.107:0/3924689006 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0a4ea0 con 0x7f62080c8100 2023-10-24T23:59:41.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6205d9b640 1 -- 172.21.15.107:0/3924689006 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61fc0a7050 con 0x7f62080c8100 2023-10-24T23:59:41.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6205d9b640 1 -- 172.21.15.107:0/3924689006 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0ab820 con 0x7f62080c8100 2023-10-24T23:59:41.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 -- 172.21.15.107:0/3924689006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62080c8100 msgr2=0x7f62080abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:41.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 --2- 172.21.15.107:0/3924689006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62080c8100 0x7f62080abd00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0097c0 tx=0x7f61fc01c3f0 comp rx=0 tx=0).stop 2023-10-24T23:59:41.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 -- 172.21.15.107:0/3924689006 shutdown_connections 2023-10-24T23:59:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 --2- 172.21.15.107:0/3924689006 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f620806a4b0 0x7f62080ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 --2- 172.21.15.107:0/3924689006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62080c8100 0x7f62080abd00 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 -- 172.21.15.107:0/3924689006 >> 172.21.15.107:0/3924689006 conn(0x7f62080ca200 msgr2=0x7f62080ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 -- 172.21.15.107:0/3924689006 shutdown_connections 2023-10-24T23:59:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 -- 172.21.15.107:0/3924689006 wait complete. 2023-10-24T23:59:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 Processor -- start 2023-10-24T23:59:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 -- start start 2023-10-24T23:59:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f620806a4b0 0x7f62080bd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62080c8100 0x7f6208150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6208152040 0x7f62081ec9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62081f0eb0 con 0x7f620806a4b0 2023-10-24T23:59:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62081f00f0 con 0x7f6208152040 2023-10-24T23:59:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620d060640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62081f03d0 con 0x7f62080c8100 2023-10-24T23:59:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6206d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f620806a4b0 0x7f62080bd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6206d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f620806a4b0 0x7f62080bd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35908/0 (socket says 172.21.15.107:35908) 2023-10-24T23:59:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6206d9d640 1 -- 172.21.15.107:0/3494210872 learned_addr learned my addr 172.21.15.107:0/3494210872 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f620659c640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62080c8100 0x7f6208150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6206d9d640 1 -- 172.21.15.107:0/3494210872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62080c8100 msgr2=0x7f6208150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6206d9d640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62080c8100 0x7f6208150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6206d9d640 1 -- 172.21.15.107:0/3494210872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6208152040 msgr2=0x7f62081ec9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6206d9d640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6208152040 0x7f62081ec9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6206d9d640 1 -- 172.21.15.107:0/3494210872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc009420 con 0x7f620806a4b0 2023-10-24T23:59:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f6206d9d640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f620806a4b0 0x7f62080bd1a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f61f80011f0 tx=0x7f61f800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.101+0000 7f61ef7fe640 1 -- 172.21.15.107:0/3494210872 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f80074e0 con 0x7f620806a4b0 2023-10-24T23:59:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.105+0000 7f61ef7fe640 1 -- 172.21.15.107:0/3494210872 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f8007640 con 0x7f620806a4b0 2023-10-24T23:59:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.105+0000 7f61ef7fe640 1 -- 172.21.15.107:0/3494210872 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f8004a10 con 0x7f620806a4b0 2023-10-24T23:59:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.105+0000 7f620d060640 1 -- 172.21.15.107:0/3494210872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62081f06b0 con 0x7f620806a4b0 2023-10-24T23:59:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.105+0000 7f620d060640 1 -- 172.21.15.107:0/3494210872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62080a9fa0 con 0x7f620806a4b0 2023-10-24T23:59:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.105+0000 7f620d060640 1 -- 172.21.15.107:0/3494210872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62080746e0 con 0x7f620806a4b0 2023-10-24T23:59:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.105+0000 7f61ef7fe640 1 -- 172.21.15.107:0/3494210872 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f61f8004b70 con 0x7f620806a4b0 2023-10-24T23:59:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.105+0000 7f61ef7fe640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f61f4065e60 0x7f61f4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.105+0000 7f61ef7fe640 1 -- 172.21.15.107:0/3494210872 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7f61f80831a0 con 0x7f620806a4b0 2023-10-24T23:59:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.105+0000 7f620659c640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f61f4065e60 0x7f61f4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.105+0000 7f620659c640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f61f4065e60 0x7f61f4068320 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0ad000 tx=0x7f61fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.109+0000 7f61ef7fe640 1 -- 172.21.15.107:0/3494210872 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f61f80501a0 con 0x7f620806a4b0 2023-10-24T23:59:41.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.297+0000 7f620d060640 1 -- 172.21.15.107:0/3494210872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62081f0b60 con 0x7f620806a4b0 2023-10-24T23:59:41.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.297+0000 7f61ef7fe640 1 -- 172.21.15.107:0/3494210872 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f61f8018070 con 0x7f620806a4b0 2023-10-24T23:59:41.300 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:41.300 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6102,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:41.301 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-10-24T23:59:41.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 -- 172.21.15.107:0/3494210872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f61f4065e60 msgr2=0x7f61f4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:41.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f61f4065e60 0x7f61f4068320 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0ad000 tx=0x7f61fc008040 comp rx=0 tx=0).stop 2023-10-24T23:59:41.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 -- 172.21.15.107:0/3494210872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f620806a4b0 msgr2=0x7f62080bd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:41.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f620806a4b0 0x7f62080bd1a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f61f80011f0 tx=0x7f61f800c9b0 comp rx=0 tx=0).stop 2023-10-24T23:59:41.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 -- 172.21.15.107:0/3494210872 shutdown_connections 2023-10-24T23:59:41.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6208152040 0x7f62081ec9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:41.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f61f4065e60 0x7f61f4068320 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:41.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62080c8100 0x7f6208150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:41.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 --2- 172.21.15.107:0/3494210872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f620806a4b0 0x7f62080bd1a0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:41.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 -- 172.21.15.107:0/3494210872 >> 172.21.15.107:0/3494210872 conn(0x7f62080ca200 msgr2=0x7f62080ad7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:41.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 -- 172.21.15.107:0/3494210872 shutdown_connections 2023-10-24T23:59:41.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:41.301+0000 7f620d060640 1 -- 172.21.15.107:0/3494210872 wait complete. 2023-10-24T23:59:41.319 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:59:43.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:43.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb2229d8640 1 Processor -- start 2023-10-24T23:59:43.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb2229d8640 1 -- start start 2023-10-24T23:59:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb2229d8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb21c071220 0x7fb21c06bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb2229d8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb21c0c24c0 0x7fb21c06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:43.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb2229d8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21c150bc0 con 0x7fb21c151960 2023-10-24T23:59:43.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb2229d8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21c06c7f0 con 0x7fb21c071220 2023-10-24T23:59:43.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb2229d8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21c06c510 con 0x7fb21c0c24c0 2023-10-24T23:59:43.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb21bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb21c071220 0x7fb21c06bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:43.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb21bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb21c071220 0x7fb21c06bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49562/0 (socket says 172.21.15.107:49562) 2023-10-24T23:59:43.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb21bfff640 1 -- 172.21.15.107:0/3301469103 learned_addr learned my addr 172.21.15.107:0/3301469103 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:43.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb220f75640 1 --2- 172.21.15.107:0/3301469103 >> v2:172.21.15.110:3300/0 conn(0x7fb21c0c24c0 0x7fb21c06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:43.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb21bfff640 1 -- 172.21.15.107:0/3301469103 >> v2:172.21.15.110:3300/0 conn(0x7fb21c0c24c0 msgr2=0x7fb21c06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:43.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb21bfff640 1 --2- 172.21.15.107:0/3301469103 >> v2:172.21.15.110:3300/0 conn(0x7fb21c0c24c0 0x7fb21c06a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:43.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb21bfff640 1 -- 172.21.15.107:0/3301469103 >> v1:172.21.15.110:6789/0 conn(0x7fb21c151960 legacy=0x7fb21c071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:43.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.449+0000 7fb21bfff640 1 -- 172.21.15.107:0/3301469103 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c150e80 con 0x7fb21c071220 2023-10-24T23:59:43.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb21bfff640 1 --2- 172.21.15.107:0/3301469103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c071220 0x7fb21c06bce0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb21c0725a0 tx=0x7fb21401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9802335990ec0bd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:43.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb21affd640 1 -- 172.21.15.107:0/3301469103 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2140ab600 con 0x7fb21c071220 2023-10-24T23:59:43.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb21affd640 1 -- 172.21.15.107:0/3301469103 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2140a7050 con 0x7fb21c071220 2023-10-24T23:59:43.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb21affd640 1 -- 172.21.15.107:0/3301469103 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2140ab980 con 0x7fb21c071220 2023-10-24T23:59:43.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 -- 172.21.15.107:0/3301469103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c071220 msgr2=0x7fb21c06bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:43.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 --2- 172.21.15.107:0/3301469103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c071220 0x7fb21c06bce0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb21c0725a0 tx=0x7fb21401c430 comp rx=0 tx=0).stop 2023-10-24T23:59:43.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 -- 172.21.15.107:0/3301469103 shutdown_connections 2023-10-24T23:59:43.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 --2- 172.21.15.107:0/3301469103 >> v2:172.21.15.110:3300/0 conn(0x7fb21c0c24c0 0x7fb21c06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:43.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 --2- 172.21.15.107:0/3301469103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c071220 0x7fb21c06bce0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:43.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 -- 172.21.15.107:0/3301469103 >> 172.21.15.107:0/3301469103 conn(0x7fb21c0c2950 msgr2=0x7fb21c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:43.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 -- 172.21.15.107:0/3301469103 shutdown_connections 2023-10-24T23:59:43.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 -- 172.21.15.107:0/3301469103 wait complete. 2023-10-24T23:59:43.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 Processor -- start 2023-10-24T23:59:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 -- start start 2023-10-24T23:59:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb21c071220 0x7fb21c14dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb21c0c24c0 0x7fb21c0c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c151960 0x7fb21c14beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:43.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c14cba0 con 0x7fb21c151960 2023-10-24T23:59:43.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c149b70 con 0x7fb21c071220 2023-10-24T23:59:43.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb2229d8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c14e1c0 con 0x7fb21c0c24c0 2023-10-24T23:59:43.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb220f75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c151960 0x7fb21c14beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:43.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb220f75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c151960 0x7fb21c14beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49566/0 (socket says 172.21.15.107:49566) 2023-10-24T23:59:43.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb220f75640 1 -- 172.21.15.107:0/4198722475 learned_addr learned my addr 172.21.15.107:0/4198722475 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:43.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb21b7fe640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb21c0c24c0 0x7fb21c0c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:43.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb21bfff640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb21c071220 0x7fb21c14dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb220f75640 1 -- 172.21.15.107:0/4198722475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb21c0c24c0 msgr2=0x7fb21c0c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb220f75640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb21c0c24c0 0x7fb21c0c45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb220f75640 1 -- 172.21.15.107:0/4198722475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb21c071220 msgr2=0x7fb21c14dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:43.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.453+0000 7fb220f75640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb21c071220 0x7fb21c14dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.457+0000 7fb220f75640 1 -- 172.21.15.107:0/4198722475 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2140094d0 con 0x7fb21c151960 2023-10-24T23:59:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.457+0000 7fb220f75640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c151960 0x7fb21c14beb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb20c007e60 tx=0x7fb20c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.457+0000 7fb218ff9640 1 -- 172.21.15.107:0/4198722475 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c014420 con 0x7fb21c151960 2023-10-24T23:59:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.457+0000 7fb218ff9640 1 -- 172.21.15.107:0/4198722475 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c0145a0 con 0x7fb21c151960 2023-10-24T23:59:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.457+0000 7fb218ff9640 1 -- 172.21.15.107:0/4198722475 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c014950 con 0x7fb21c151960 2023-10-24T23:59:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.457+0000 7fb2229d8640 1 -- 172.21.15.107:0/4198722475 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21c149530 con 0x7fb21c151960 2023-10-24T23:59:43.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.457+0000 7fb2229d8640 1 -- 172.21.15.107:0/4198722475 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21c1eb830 con 0x7fb21c151960 2023-10-24T23:59:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.457+0000 7fb2229d8640 1 -- 172.21.15.107:0/4198722475 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c0746e0 con 0x7fb21c151960 2023-10-24T23:59:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.461+0000 7fb218ff9640 1 -- 172.21.15.107:0/4198722475 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb20c014ab0 con 0x7fb21c151960 2023-10-24T23:59:43.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.461+0000 7fb218ff9640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb208065e60 0x7fb208068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.461+0000 7fb21bfff640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb208065e60 0x7fb208068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.461+0000 7fb218ff9640 1 -- 172.21.15.107:0/4198722475 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7fb20c085620 con 0x7fb21c151960 2023-10-24T23:59:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.461+0000 7fb21bfff640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb208065e60 0x7fb208068320 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb2140ad000 tx=0x7fb214008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:43.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.465+0000 7fb218ff9640 1 -- 172.21.15.107:0/4198722475 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb20c052700 con 0x7fb21c151960 2023-10-24T23:59:43.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.653+0000 7fb2229d8640 1 -- 172.21.15.107:0/4198722475 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb21c0c22a0 con 0x7fb21c151960 2023-10-24T23:59:43.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.653+0000 7fb218ff9640 1 -- 172.21.15.107:0/4198722475 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5777 (secure 0 0 0) 0x7fb20c055d20 con 0x7fb21c151960 2023-10-24T23:59:43.658 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:43.658 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6102,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:43.659 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-10-24T23:59:43.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 -- 172.21.15.107:0/4198722475 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb208065e60 msgr2=0x7fb208068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:43.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb208065e60 0x7fb208068320 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb2140ad000 tx=0x7fb214008040 comp rx=0 tx=0).stop 2023-10-24T23:59:43.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 -- 172.21.15.107:0/4198722475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c151960 msgr2=0x7fb21c14beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:43.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c151960 0x7fb21c14beb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb20c007e60 tx=0x7fb20c00e4b0 comp rx=0 tx=0).stop 2023-10-24T23:59:43.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 -- 172.21.15.107:0/4198722475 shutdown_connections 2023-10-24T23:59:43.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c151960 0x7fb21c14beb0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:43.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb208065e60 0x7fb208068320 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:43.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb21c0c24c0 0x7fb21c0c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:43.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 --2- 172.21.15.107:0/4198722475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb21c071220 0x7fb21c14dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:43.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 -- 172.21.15.107:0/4198722475 >> 172.21.15.107:0/4198722475 conn(0x7fb21c0c2950 msgr2=0x7fb21c073170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:43.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 -- 172.21.15.107:0/4198722475 shutdown_connections 2023-10-24T23:59:43.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:43.657+0000 7fb2229d8640 1 -- 172.21.15.107:0/4198722475 wait complete. 2023-10-24T23:59:43.675 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:59:45.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:45.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f068b482640 1 Processor -- start 2023-10-24T23:59:45.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f068b482640 1 -- start start 2023-10-24T23:59:45.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f068b482640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f068406a230 0x7f06841551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:45.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f068b482640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06840c1860 con 0x7f068406af70 2023-10-24T23:59:45.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f068b482640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0684154d50 con 0x7f068414b600 2023-10-24T23:59:45.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f068b482640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f068414b9e0 con 0x7f068406a230 2023-10-24T23:59:45.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f0688a1d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f068406af70 0x7f068406c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39742/0 (socket says 172.21.15.107:39742) 2023-10-24T23:59:45.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f0688a1d640 1 -- 172.21.15.107:0/194085457 learned_addr learned my addr 172.21.15.107:0/194085457 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:59:45.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f0689a1f640 1 --2- 172.21.15.107:0/194085457 >> v2:172.21.15.110:3300/0 conn(0x7f068406a230 0x7f06841551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:45.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834776489 0 0) 0x7f06840c1860 con 0x7f068406af70 2023-10-24T23:59:45.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0670003530 con 0x7f068406af70 2023-10-24T23:59:45.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3170385897 0 0) 0x7f0670003530 con 0x7f068406af70 2023-10-24T23:59:45.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 >> v2:172.21.15.110:3300/0 conn(0x7f068406a230 msgr2=0x7f06841551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:45.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 --2- 172.21.15.107:0/194085457 >> v2:172.21.15.110:3300/0 conn(0x7f068406a230 0x7f06841551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:45.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 >> v1:172.21.15.110:6789/0 conn(0x7f068414b600 legacy=0x7f0684151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:45.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06840c1ab0 con 0x7f068406af70 2023-10-24T23:59:45.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f0674001680 con 0x7f068406af70 2023-10-24T23:59:45.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 >> v1:172.21.15.107:6789/0 conn(0x7f068406af70 legacy=0x7f068406c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:45.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 --2- 172.21.15.107:0/194085457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0670003980 0x7f0670003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:45.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 --2- 172.21.15.107:0/194085457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06700047d0 0x7f0670004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:45.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 --2- 172.21.15.107:0/194085457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0670005190 0x7f067000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:45.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068414b9e0 con 0x7f06700047d0 2023-10-24T23:59:45.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067000c130 con 0x7f0670005190 2023-10-24T23:59:45.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067000c2f0 con 0x7f0670003980 2023-10-24T23:59:45.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f0688a1d640 1 --2- 172.21.15.107:0/194085457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06700047d0 0x7f0670004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:45.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f068921e640 1 --2- 172.21.15.107:0/194085457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0670003980 0x7f0670003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:45.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f0688a1d640 1 -- 172.21.15.107:0/194085457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0670003980 msgr2=0x7f0670003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:45.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f0688a1d640 1 --2- 172.21.15.107:0/194085457 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0670003980 0x7f0670003da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:45.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f0688a1d640 1 -- 172.21.15.107:0/194085457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0670005190 msgr2=0x7f067000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:45.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f0688a1d640 1 --2- 172.21.15.107:0/194085457 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0670005190 0x7f067000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:45.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f0688a1d640 1 -- 172.21.15.107:0/194085457 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0670003530 con 0x7f06700047d0 2023-10-24T23:59:45.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.804+0000 7f0689a1f640 1 -- 172.21.15.107:0/194085457 reap_dead start 2023-10-24T23:59:45.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0688a1d640 1 --2- 172.21.15.107:0/194085457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06700047d0 0x7f0670004c50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f067400c210 tx=0x7f067401cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a83b708f0e240c server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:45.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06740058e0 con 0x7f06700047d0 2023-10-24T23:59:45.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0674005a60 con 0x7f06700047d0 2023-10-24T23:59:45.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f067bfff640 1 -- 172.21.15.107:0/194085457 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674005e10 con 0x7f06700047d0 2023-10-24T23:59:45.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 -- 172.21.15.107:0/194085457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06700047d0 msgr2=0x7f0670004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:45.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 --2- 172.21.15.107:0/194085457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06700047d0 0x7f0670004c50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f067400c210 tx=0x7f067401cbf0 comp rx=0 tx=0).stop 2023-10-24T23:59:45.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 -- 172.21.15.107:0/194085457 shutdown_connections 2023-10-24T23:59:45.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 --2- 172.21.15.107:0/194085457 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06700047d0 0x7f0670004c50 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:45.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 -- 172.21.15.107:0/194085457 >> 172.21.15.107:0/194085457 conn(0x7f06840c2950 msgr2=0x7f06840c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:45.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 -- 172.21.15.107:0/194085457 shutdown_connections 2023-10-24T23:59:45.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 -- 172.21.15.107:0/194085457 wait complete. 2023-10-24T23:59:45.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 Processor -- start 2023-10-24T23:59:45.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 -- start start 2023-10-24T23:59:45.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f06700047d0 0x7f0684147c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:45.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06841df7c0 0x7f0684144960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:45.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06840c4530 0x7f068414fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:45.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06841dfc40 con 0x7f06841df7c0 2023-10-24T23:59:45.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0684150870 con 0x7f06840c4530 2023-10-24T23:59:45.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0684144ea0 con 0x7f06700047d0 2023-10-24T23:59:45.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0688a1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06841df7c0 0x7f0684144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:45.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0688a1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06841df7c0 0x7f0684144960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49576/0 (socket says 172.21.15.107:49576) 2023-10-24T23:59:45.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0688a1d640 1 -- 172.21.15.107:0/3203677650 learned_addr learned my addr 172.21.15.107:0/3203677650 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:45.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0689a1f640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06840c4530 0x7f068414fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:45.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068921e640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f06700047d0 0x7f0684147c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:45.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0688a1d640 1 -- 172.21.15.107:0/3203677650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f06700047d0 msgr2=0x7f0684147c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:45.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0688a1d640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f06700047d0 0x7f0684147c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:45.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0688a1d640 1 -- 172.21.15.107:0/3203677650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06840c4530 msgr2=0x7f068414fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:45.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0688a1d640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06840c4530 0x7f068414fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:45.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0688a1d640 1 -- 172.21.15.107:0/3203677650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684152540 con 0x7f06841df7c0 2023-10-24T23:59:45.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068921e640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f06700047d0 0x7f0684147c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:59:45.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0688a1d640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06841df7c0 0x7f0684144960 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0674001940 tx=0x7f0670005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:45.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0679ffb640 1 -- 172.21.15.107:0/3203677650 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06740b4070 con 0x7f06841df7c0 2023-10-24T23:59:45.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0679ffb640 1 -- 172.21.15.107:0/3203677650 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0674000ee0 con 0x7f06841df7c0 2023-10-24T23:59:45.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f068b482640 1 -- 172.21.15.107:0/3203677650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06841de760 con 0x7f06841df7c0 2023-10-24T23:59:45.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.808+0000 7f0679ffb640 1 -- 172.21.15.107:0/3203677650 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674001640 con 0x7f06841df7c0 2023-10-24T23:59:45.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.812+0000 7f068b482640 1 -- 172.21.15.107:0/3203677650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06841dece0 con 0x7f06841df7c0 2023-10-24T23:59:45.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.812+0000 7f068b482640 1 -- 172.21.15.107:0/3203677650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06840746e0 con 0x7f06841df7c0 2023-10-24T23:59:45.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.812+0000 7f0679ffb640 1 -- 172.21.15.107:0/3203677650 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0674005bd0 con 0x7f06841df7c0 2023-10-24T23:59:45.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.812+0000 7f0679ffb640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0668065e90 0x7f0668068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:45.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.812+0000 7f0679ffb640 1 -- 172.21.15.107:0/3203677650 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7f067411d420 con 0x7f06841df7c0 2023-10-24T23:59:45.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.812+0000 7f068921e640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0668065e90 0x7f0668068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:45.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.812+0000 7f068921e640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0668065e90 0x7f0668068350 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0684150b50 tx=0x7f0680009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:45.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:45.816+0000 7f0679ffb640 1 -- 172.21.15.107:0/3203677650 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f06740ea390 con 0x7f06841df7c0 2023-10-24T23:59:46.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 -- 172.21.15.107:0/3203677650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0684069fa0 con 0x7f06841df7c0 2023-10-24T23:59:46.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f0679ffb640 1 -- 172.21.15.107:0/3203677650 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5777 (secure 0 0 0) 0x7f06740ed9b0 con 0x7f06841df7c0 2023-10-24T23:59:46.010 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:46.010 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6102,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:46.010 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-10-24T23:59:46.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 -- 172.21.15.107:0/3203677650 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0668065e90 msgr2=0x7f0668068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0668065e90 0x7f0668068350 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0684150b50 tx=0x7f0680009040 comp rx=0 tx=0).stop 2023-10-24T23:59:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 -- 172.21.15.107:0/3203677650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06841df7c0 msgr2=0x7f0684144960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06841df7c0 0x7f0684144960 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0674001940 tx=0x7f0670005800 comp rx=0 tx=0).stop 2023-10-24T23:59:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 -- 172.21.15.107:0/3203677650 shutdown_connections 2023-10-24T23:59:46.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f06840c4530 0x7f068414fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06841df7c0 0x7f0684144960 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0668065e90 0x7f0668068350 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 --2- 172.21.15.107:0/3203677650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f06700047d0 0x7f0684147c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 -- 172.21.15.107:0/3203677650 >> 172.21.15.107:0/3203677650 conn(0x7f06840c2950 msgr2=0x7f06840a9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:46.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 -- 172.21.15.107:0/3203677650 shutdown_connections 2023-10-24T23:59:46.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:46.008+0000 7f068b482640 1 -- 172.21.15.107:0/3203677650 wait complete. 2023-10-24T23:59:46.028 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:59:48.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:48.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece74be640 1 Processor -- start 2023-10-24T23:59:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece74be640 1 -- start start 2023-10-24T23:59:48.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece74be640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fece0152040 0x7fece00c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:48.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece74be640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fece006a4b0 0x7fece00ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:48.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece74be640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece0155680 con 0x7fece00c8100 2023-10-24T23:59:48.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece74be640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece0154a70 con 0x7fece006a4b0 2023-10-24T23:59:48.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece74be640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece0154d50 con 0x7fece0152040 2023-10-24T23:59:48.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece5a5b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fece006a4b0 0x7fece00ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:48.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece5a5b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fece006a4b0 0x7fece00ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49588/0 (socket says 172.21.15.107:49588) 2023-10-24T23:59:48.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece5a5b640 1 -- 172.21.15.107:0/3972235121 learned_addr learned my addr 172.21.15.107:0/3972235121 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:48.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece4a59640 1 --1- 172.21.15.107:0/3972235121 >> v1:172.21.15.107:6789/0 conn(0x7fece00c8100 0x7fece00abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39756/0 (socket says 172.21.15.107:39756) 2023-10-24T23:59:48.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece525a640 1 --2- 172.21.15.107:0/3972235121 >> v2:172.21.15.110:3300/0 conn(0x7fece0152040 0x7fece00c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:48.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece5a5b640 1 -- 172.21.15.107:0/3972235121 >> v2:172.21.15.110:3300/0 conn(0x7fece0152040 msgr2=0x7fece00c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:48.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece5a5b640 1 --2- 172.21.15.107:0/3972235121 >> v2:172.21.15.110:3300/0 conn(0x7fece0152040 0x7fece00c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:48.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece5a5b640 1 -- 172.21.15.107:0/3972235121 >> v1:172.21.15.107:6789/0 conn(0x7fece00c8100 legacy=0x7fece00abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:48.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.160+0000 7fece5a5b640 1 -- 172.21.15.107:0/3972235121 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece0155030 con 0x7fece006a4b0 2023-10-24T23:59:48.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece5a5b640 1 --2- 172.21.15.107:0/3972235121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fece006a4b0 0x7fece00ac410 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fece0067ba0 tx=0x7fecd801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d1335fc7c45653 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:48.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fecdffff640 1 -- 172.21.15.107:0/3972235121 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd80ad680 con 0x7fece006a4b0 2023-10-24T23:59:48.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fecdffff640 1 -- 172.21.15.107:0/3972235121 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd80ad7e0 con 0x7fece006a4b0 2023-10-24T23:59:48.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fecdffff640 1 -- 172.21.15.107:0/3972235121 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd80adb70 con 0x7fece006a4b0 2023-10-24T23:59:48.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 -- 172.21.15.107:0/3972235121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fece006a4b0 msgr2=0x7fece00ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:48.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 --2- 172.21.15.107:0/3972235121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fece006a4b0 0x7fece00ac410 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fece0067ba0 tx=0x7fecd801c470 comp rx=0 tx=0).stop 2023-10-24T23:59:48.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 -- 172.21.15.107:0/3972235121 shutdown_connections 2023-10-24T23:59:48.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 --2- 172.21.15.107:0/3972235121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fece006a4b0 0x7fece00ac410 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:48.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 --2- 172.21.15.107:0/3972235121 >> v2:172.21.15.110:3300/0 conn(0x7fece0152040 0x7fece00c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:48.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 -- 172.21.15.107:0/3972235121 >> 172.21.15.107:0/3972235121 conn(0x7fece00ca200 msgr2=0x7fece00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:48.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 -- 172.21.15.107:0/3972235121 shutdown_connections 2023-10-24T23:59:48.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 -- 172.21.15.107:0/3972235121 wait complete. 2023-10-24T23:59:48.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 Processor -- start 2023-10-24T23:59:48.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 -- start start 2023-10-24T23:59:48.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece006a4b0 0x7fece01f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:48.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fece00c8100 0x7fece0149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:48.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece0152040 0x7fece01ec970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:48.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece01f05d0 con 0x7fece00c8100 2023-10-24T23:59:48.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece0151e10 con 0x7fece006a4b0 2023-10-24T23:59:48.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.164+0000 7fece74be640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece0151b30 con 0x7fece0152040 2023-10-24T23:59:48.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece4a59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fece00c8100 0x7fece0149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:48.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece4a59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fece00c8100 0x7fece0149a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49590/0 (socket says 172.21.15.107:49590) 2023-10-24T23:59:48.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece4a59640 1 -- 172.21.15.107:0/2894693384 learned_addr learned my addr 172.21.15.107:0/2894693384 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:48.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece525a640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece006a4b0 0x7fece01f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:48.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece5a5b640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece0152040 0x7fece01ec970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:48.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece4a59640 1 -- 172.21.15.107:0/2894693384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece0152040 msgr2=0x7fece01ec970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:48.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece4a59640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece0152040 0x7fece01ec970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:48.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece4a59640 1 -- 172.21.15.107:0/2894693384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece006a4b0 msgr2=0x7fece01f0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:48.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece4a59640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece006a4b0 0x7fece01f0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:48.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece4a59640 1 -- 172.21.15.107:0/2894693384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd80094d0 con 0x7fece00c8100 2023-10-24T23:59:48.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece5a5b640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece0152040 0x7fece01ec970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-24T23:59:48.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece4a59640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fece00c8100 0x7fece0149a70 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fecd400c7a0 tx=0x7fecd400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:48.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fecddffb640 1 -- 172.21.15.107:0/2894693384 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd400a5e0 con 0x7fece00c8100 2023-10-24T23:59:48.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fecddffb640 1 -- 172.21.15.107:0/2894693384 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd400a740 con 0x7fece00c8100 2023-10-24T23:59:48.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fecddffb640 1 -- 172.21.15.107:0/2894693384 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd400aaf0 con 0x7fece00c8100 2023-10-24T23:59:48.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece74be640 1 -- 172.21.15.107:0/2894693384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece01eff80 con 0x7fece00c8100 2023-10-24T23:59:48.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece74be640 1 -- 172.21.15.107:0/2894693384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece0149fb0 con 0x7fece00c8100 2023-10-24T23:59:48.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece74be640 1 -- 172.21.15.107:0/2894693384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece01f0780 con 0x7fece00c8100 2023-10-24T23:59:48.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fecddffb640 1 -- 172.21.15.107:0/2894693384 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fecd400ac50 con 0x7fece00c8100 2023-10-24T23:59:48.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fecddffb640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feccc065e60 0x7feccc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:48.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fecddffb640 1 -- 172.21.15.107:0/2894693384 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 7821+0+0 (secure 0 0 0) 0x7fecd40835a0 con 0x7fece00c8100 2023-10-24T23:59:48.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece525a640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feccc065e60 0x7feccc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:48.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.168+0000 7fece525a640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feccc065e60 0x7feccc068320 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fece014a390 tx=0x7fecd0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:48.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.176+0000 7fecddffb640 1 -- 172.21.15.107:0/2894693384 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fece01f0780 con 0x7fece00c8100 2023-10-24T23:59:48.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 -- 172.21.15.107:0/2894693384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fece00746e0 con 0x7fece00c8100 2023-10-24T23:59:48.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fecddffb640 1 -- 172.21.15.107:0/2894693384 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+5777 (secure 0 0 0) 0x7fecd4017090 con 0x7fece00c8100 2023-10-24T23:59:48.366 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:48.366 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6102,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:01.641563+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5293":{"gid":5293,"name":"b","rank":0,"incarnation":55,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/1101018338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1101018338},{"type":"v1","addr":"172.21.15.110:6834","nonce":1101018338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:48.367 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 60 2023-10-24T23:59:48.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 -- 172.21.15.107:0/2894693384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feccc065e60 msgr2=0x7feccc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:48.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feccc065e60 0x7feccc068320 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fece014a390 tx=0x7fecd0008040 comp rx=0 tx=0).stop 2023-10-24T23:59:48.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 -- 172.21.15.107:0/2894693384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fece00c8100 msgr2=0x7fece0149a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:48.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fece00c8100 0x7fece0149a70 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fecd400c7a0 tx=0x7fecd400cc60 comp rx=0 tx=0).stop 2023-10-24T23:59:48.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 -- 172.21.15.107:0/2894693384 shutdown_connections 2023-10-24T23:59:48.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feccc065e60 0x7feccc068320 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:48.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece0152040 0x7fece01ec970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:48.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fece00c8100 0x7fece0149a70 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:48.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 --2- 172.21.15.107:0/2894693384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece006a4b0 0x7fece01f0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:48.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 -- 172.21.15.107:0/2894693384 >> 172.21.15.107:0/2894693384 conn(0x7fece00ca200 msgr2=0x7fece00c9980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:48.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 -- 172.21.15.107:0/2894693384 shutdown_connections 2023-10-24T23:59:48.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:48.364+0000 7fece74be640 1 -- 172.21.15.107:0/2894693384 wait complete. 2023-10-24T23:59:48.383 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-24T23:59:50.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:50.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.508+0000 7ff8f1a2c640 1 Processor -- start 2023-10-24T23:59:50.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.508+0000 7ff8f1a2c640 1 -- start start 2023-10-24T23:59:50.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8f1a2c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff8ec14b600 0x7ff8ec151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:50.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8f1a2c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8ec06a230 0x7ff8ec1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:50.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8f1a2c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec0c1860 con 0x7ff8ec06af70 2023-10-24T23:59:50.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8f1a2c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ec154d50 con 0x7ff8ec14b600 2023-10-24T23:59:50.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8f1a2c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec1524f0 con 0x7ff8ec06a230 2023-10-24T23:59:50.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8ea7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff8ec06af70 0x7ff8ec06c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39768/0 (socket says 172.21.15.107:39768) 2023-10-24T23:59:50.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8ea7fc640 1 -- 172.21.15.107:0/3508234232 learned_addr learned my addr 172.21.15.107:0/3508234232 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:59:50.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8eaffd640 1 --2- 172.21.15.107:0/3508234232 >> v2:172.21.15.110:3300/0 conn(0x7ff8ec14b600 0x7ff8ec151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:50.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8eb7fe640 1 --2- 172.21.15.107:0/3508234232 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8ec06a230 0x7ff8ec1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:50.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8e9ffb640 1 -- 172.21.15.107:0/3508234232 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038755983 0 0) 0x7ff8ec0c1860 con 0x7ff8ec06af70 2023-10-24T23:59:50.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8e9ffb640 1 -- 172.21.15.107:0/3508234232 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d8003530 con 0x7ff8ec06af70 2023-10-24T23:59:50.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8eaffd640 1 -- 172.21.15.107:0/3508234232 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8ec06a230 msgr2=0x7ff8ec1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8eaffd640 1 --2- 172.21.15.107:0/3508234232 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8ec06a230 0x7ff8ec1551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8eaffd640 1 -- 172.21.15.107:0/3508234232 >> v1:172.21.15.107:6789/0 conn(0x7ff8ec06af70 legacy=0x7ff8ec06c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8eaffd640 1 -- 172.21.15.107:0/3508234232 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec14b9e0 con 0x7ff8ec14b600 2023-10-24T23:59:50.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.512+0000 7ff8eaffd640 1 --2- 172.21.15.107:0/3508234232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8ec14b600 0x7ff8ec151fb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0097c0 tx=0x7ff8dc01e5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc037da562517482 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:50.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8e9ffb640 1 -- 172.21.15.107:0/3508234232 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0ad5f0 con 0x7ff8ec14b600 2023-10-24T23:59:50.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8e9ffb640 1 -- 172.21.15.107:0/3508234232 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc0ad750 con 0x7ff8ec14b600 2023-10-24T23:59:50.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8e9ffb640 1 -- 172.21.15.107:0/3508234232 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0adac0 con 0x7ff8ec14b600 2023-10-24T23:59:50.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/3508234232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8ec14b600 msgr2=0x7ff8ec151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:50.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 --2- 172.21.15.107:0/3508234232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8ec14b600 0x7ff8ec151fb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0097c0 tx=0x7ff8dc01e5d0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/3508234232 shutdown_connections 2023-10-24T23:59:50.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 --2- 172.21.15.107:0/3508234232 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8ec06a230 0x7ff8ec1551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 --2- 172.21.15.107:0/3508234232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8ec14b600 0x7ff8ec151fb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/3508234232 >> 172.21.15.107:0/3508234232 conn(0x7ff8ec0c2950 msgr2=0x7ff8ec0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:50.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/3508234232 shutdown_connections 2023-10-24T23:59:50.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/3508234232 wait complete. 2023-10-24T23:59:50.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 Processor -- start 2023-10-24T23:59:50.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 -- start start 2023-10-24T23:59:50.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8ec06a230 0x7ff8ec0bae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:50.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8ec06af70 0x7ff8ec1f08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:50.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec14b600 0x7ff8ec15a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:50.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec145850 con 0x7ff8ec14b600 2023-10-24T23:59:50.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec1f0430 con 0x7ff8ec06af70 2023-10-24T23:59:50.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec149270 con 0x7ff8ec06a230 2023-10-24T23:59:50.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8eb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec14b600 0x7ff8ec15a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:50.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8eb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec14b600 0x7ff8ec15a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49592/0 (socket says 172.21.15.107:49592) 2023-10-24T23:59:50.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8eb7fe640 1 -- 172.21.15.107:0/1443910069 learned_addr learned my addr 172.21.15.107:0/1443910069 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:50.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8ea7fc640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8ec06af70 0x7ff8ec1f08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:50.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8eaffd640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8ec06a230 0x7ff8ec0bae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:50.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8eb7fe640 1 -- 172.21.15.107:0/1443910069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8ec06a230 msgr2=0x7ff8ec0bae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:50.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8eb7fe640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8ec06a230 0x7ff8ec0bae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8eb7fe640 1 -- 172.21.15.107:0/1443910069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8ec06af70 msgr2=0x7ff8ec1f08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:50.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8eb7fe640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8ec06af70 0x7ff8ec1f08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8eb7fe640 1 -- 172.21.15.107:0/1443910069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc009420 con 0x7ff8ec14b600 2023-10-24T23:59:50.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8eb7fe640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec14b600 0x7ff8ec15a290 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400efc0 tx=0x7ff8e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:50.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8e3fff640 1 -- 172.21.15.107:0/1443910069 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4014480 con 0x7ff8ec14b600 2023-10-24T23:59:50.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8e3fff640 1 -- 172.21.15.107:0/1443910069 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e400f040 con 0x7ff8ec14b600 2023-10-24T23:59:50.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8e3fff640 1 -- 172.21.15.107:0/1443910069 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4014820 con 0x7ff8ec14b600 2023-10-24T23:59:50.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/1443910069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ec154860 con 0x7ff8ec14b600 2023-10-24T23:59:50.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.516+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/1443910069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ec154580 con 0x7ff8ec14b600 2023-10-24T23:59:50.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.520+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/1443910069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ec1f05c0 con 0x7ff8ec14b600 2023-10-24T23:59:50.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.520+0000 7ff8e3fff640 1 -- 172.21.15.107:0/1443910069 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff8e4014980 con 0x7ff8ec14b600 2023-10-24T23:59:50.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.520+0000 7ff8e3fff640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8cc065e60 0x7ff8cc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:50.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.520+0000 7ff8e3fff640 1 -- 172.21.15.107:0/1443910069 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7ff8e4084c30 con 0x7ff8ec14b600 2023-10-24T23:59:50.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.520+0000 7ff8eaffd640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8cc065e60 0x7ff8cc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:50.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.520+0000 7ff8eaffd640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8cc065e60 0x7ff8cc068320 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0b1000 tx=0x7ff8dc0a9000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:50.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.524+0000 7ff8e3fff640 1 -- 172.21.15.107:0/1443910069 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff8e4051c80 con 0x7ff8ec14b600 2023-10-24T23:59:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.716+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/1443910069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8ec0746e0 con 0x7ff8ec14b600 2023-10-24T23:59:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.716+0000 7ff8e3fff640 1 -- 172.21.15.107:0/1443910069 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5017 (secure 0 0 0) 0x7ff8e4018070 con 0x7ff8ec14b600 2023-10-24T23:59:50.718 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:50.718 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:50.603881+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 63 2023-10-24T23:59:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.716+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/1443910069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8cc065e60 msgr2=0x7ff8cc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.716+0000 7ff8f1a2c640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8cc065e60 0x7ff8cc068320 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc0b1000 tx=0x7ff8dc0a9000 comp rx=0 tx=0).stop 2023-10-24T23:59:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.720+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/1443910069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec14b600 msgr2=0x7ff8ec15a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.720+0000 7ff8f1a2c640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec14b600 0x7ff8ec15a290 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400efc0 tx=0x7ff8e400c490 comp rx=0 tx=0).stop 2023-10-24T23:59:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.720+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/1443910069 shutdown_connections 2023-10-24T23:59:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.720+0000 7ff8f1a2c640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec14b600 0x7ff8ec15a290 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.720+0000 7ff8f1a2c640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8ec06af70 0x7ff8ec1f08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.720+0000 7ff8f1a2c640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8cc065e60 0x7ff8cc068320 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.720+0000 7ff8f1a2c640 1 --2- 172.21.15.107:0/1443910069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8ec06a230 0x7ff8ec0bae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.720+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/1443910069 >> 172.21.15.107:0/1443910069 conn(0x7ff8ec0c2950 msgr2=0x7ff8ec073170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:50.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.720+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/1443910069 shutdown_connections 2023-10-24T23:59:50.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.720+0000 7ff8f1a2c640 1 -- 172.21.15.107:0/1443910069 wait complete. 2023-10-24T23:59:50.739 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2023-10-24T23:59:50.739 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-24T23:59:50.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:50.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.864+0000 7fe518083640 1 Processor -- start 2023-10-24T23:59:50.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.864+0000 7fe518083640 1 -- start start 2023-10-24T23:59:50.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe518083640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe51006ee40 0x7fe5100ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:50.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe518083640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51006f730 con 0x7fe5100a9820 2023-10-24T23:59:50.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe518083640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe510156a40 con 0x7fe51014fd40 2023-10-24T23:59:50.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe518083640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe510156710 con 0x7fe51006ee40 2023-10-24T23:59:50.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe516620640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe51006ee40 0x7fe5100ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:50.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe51561e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe5100a9820 0x7fe51015af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:39772/0 (socket says 172.21.15.107:39772) 2023-10-24T23:59:50.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe51561e640 1 -- 172.21.15.107:0/1454822508 learned_addr learned my addr 172.21.15.107:0/1454822508 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:59:50.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe516620640 1 -- 172.21.15.107:0/1454822508 >> v1:172.21.15.110:6789/0 conn(0x7fe51014fd40 legacy=0x7fe5100c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:50.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe516620640 1 -- 172.21.15.107:0/1454822508 >> v1:172.21.15.107:6789/0 conn(0x7fe5100a9820 legacy=0x7fe51015af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:50.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe514e1d640 1 -- 172.21.15.107:0/1454822508 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254089939 0 0) 0x7fe51006f730 con 0x7fe5100a9820 2023-10-24T23:59:50.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe516620640 1 -- 172.21.15.107:0/1454822508 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51006f980 con 0x7fe51006ee40 2023-10-24T23:59:50.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe516620640 1 --2- 172.21.15.107:0/1454822508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51006ee40 0x7fe5100ab090 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe508009d90 tx=0x7fe50801c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba4ec460296db7d server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:50.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe514e1d640 1 -- 172.21.15.107:0/1454822508 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5080a5810 con 0x7fe51006ee40 2023-10-24T23:59:50.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe514e1d640 1 -- 172.21.15.107:0/1454822508 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5080a5970 con 0x7fe51006ee40 2023-10-24T23:59:50.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe514e1d640 1 -- 172.21.15.107:0/1454822508 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5080a5d00 con 0x7fe51006ee40 2023-10-24T23:59:50.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe518083640 1 -- 172.21.15.107:0/1454822508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51006ee40 msgr2=0x7fe5100ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:50.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe518083640 1 --2- 172.21.15.107:0/1454822508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51006ee40 0x7fe5100ab090 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fe508009d90 tx=0x7fe50801c830 comp rx=0 tx=0).stop 2023-10-24T23:59:50.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe518083640 1 -- 172.21.15.107:0/1454822508 shutdown_connections 2023-10-24T23:59:50.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe518083640 1 --2- 172.21.15.107:0/1454822508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe51006ee40 0x7fe5100ab090 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe518083640 1 -- 172.21.15.107:0/1454822508 >> 172.21.15.107:0/1454822508 conn(0x7fe5100c2950 msgr2=0x7fe5100c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:50.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe518083640 1 -- 172.21.15.107:0/1454822508 shutdown_connections 2023-10-24T23:59:50.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.868+0000 7fe518083640 1 -- 172.21.15.107:0/1454822508 wait complete. 2023-10-24T23:59:50.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe518083640 1 Processor -- start 2023-10-24T23:59:50.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe518083640 1 -- start start 2023-10-24T23:59:50.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe518083640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe51006ee40 0x7fe510149610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:50.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe518083640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe51014fd40 0x7fe5100bf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:50.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe518083640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510153910 0x7fe510146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:50.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe518083640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe510069ee0 con 0x7fe510153910 2023-10-24T23:59:50.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe518083640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5100c02b0 con 0x7fe51014fd40 2023-10-24T23:59:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe518083640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5101edfd0 con 0x7fe51006ee40 2023-10-24T23:59:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe516620640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510153910 0x7fe510146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:50.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe516620640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510153910 0x7fe510146240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49614/0 (socket says 172.21.15.107:49614) 2023-10-24T23:59:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe516620640 1 -- 172.21.15.107:0/2969359280 learned_addr learned my addr 172.21.15.107:0/2969359280 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:50.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe51561e640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe51014fd40 0x7fe5100bf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe515e1f640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe51006ee40 0x7fe510149610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe516620640 1 -- 172.21.15.107:0/2969359280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe51006ee40 msgr2=0x7fe510149610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:50.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe516620640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe51006ee40 0x7fe510149610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe516620640 1 -- 172.21.15.107:0/2969359280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe51014fd40 msgr2=0x7fe5100bf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe516620640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe51014fd40 0x7fe5100bf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe516620640 1 -- 172.21.15.107:0/2969359280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe508009a70 con 0x7fe510153910 2023-10-24T23:59:50.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe516620640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510153910 0x7fe510146240 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe50801a460 tx=0x7fe5080195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:50.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe5067fc640 1 -- 172.21.15.107:0/2969359280 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5080b9860 con 0x7fe510153910 2023-10-24T23:59:50.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe5067fc640 1 -- 172.21.15.107:0/2969359280 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5080b99c0 con 0x7fe510153910 2023-10-24T23:59:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe518083640 1 -- 172.21.15.107:0/2969359280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5101ee500 con 0x7fe510153910 2023-10-24T23:59:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe5067fc640 1 -- 172.21.15.107:0/2969359280 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5080b9d50 con 0x7fe510153910 2023-10-24T23:59:50.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe518083640 1 -- 172.21.15.107:0/2969359280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe51006b7c0 con 0x7fe510153910 2023-10-24T23:59:50.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.872+0000 7fe518083640 1 -- 172.21.15.107:0/2969359280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5100c24a0 con 0x7fe510153910 2023-10-24T23:59:50.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.876+0000 7fe5067fc640 1 -- 172.21.15.107:0/2969359280 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe5080ad440 con 0x7fe510153910 2023-10-24T23:59:50.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.876+0000 7fe5067fc640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4f4065e90 0x7fe4f4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:50.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.876+0000 7fe515e1f640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4f4065e90 0x7fe4f4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:50.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.876+0000 7fe5067fc640 1 -- 172.21.15.107:0/2969359280 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7fe50811f0c0 con 0x7fe510153910 2023-10-24T23:59:50.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.876+0000 7fe515e1f640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4f4065e90 0x7fe4f4068350 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe5100c04b0 tx=0x7fe50c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:50.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:50.880+0000 7fe5067fc640 1 -- 172.21.15.107:0/2969359280 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe5080ec240 con 0x7fe510153910 2023-10-24T23:59:51.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.068+0000 7fe518083640 1 -- 172.21.15.107:0/2969359280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe51006a020 con 0x7fe510153910 2023-10-24T23:59:51.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.068+0000 7fe5067fc640 1 -- 172.21.15.107:0/2969359280 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5017 (secure 0 0 0) 0x7fe51006a020 con 0x7fe510153910 2023-10-24T23:59:51.071 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:51.071 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:50.603881+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:51.071 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 63 2023-10-24T23:59:51.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 -- 172.21.15.107:0/2969359280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4f4065e90 msgr2=0x7fe4f4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:51.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4f4065e90 0x7fe4f4068350 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe5100c04b0 tx=0x7fe50c009040 comp rx=0 tx=0).stop 2023-10-24T23:59:51.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 -- 172.21.15.107:0/2969359280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510153910 msgr2=0x7fe510146240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:51.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510153910 0x7fe510146240 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe50801a460 tx=0x7fe5080195d0 comp rx=0 tx=0).stop 2023-10-24T23:59:51.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 -- 172.21.15.107:0/2969359280 shutdown_connections 2023-10-24T23:59:51.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510153910 0x7fe510146240 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:51.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe51014fd40 0x7fe5100bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:51.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4f4065e90 0x7fe4f4068350 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:51.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 --2- 172.21.15.107:0/2969359280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe51006ee40 0x7fe510149610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:51.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 -- 172.21.15.107:0/2969359280 >> 172.21.15.107:0/2969359280 conn(0x7fe5100c2950 msgr2=0x7fe5100c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:51.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 -- 172.21.15.107:0/2969359280 shutdown_connections 2023-10-24T23:59:51.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:51.072+0000 7fe518083640 1 -- 172.21.15.107:0/2969359280 wait complete. 2023-10-24T23:59:51.090 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 63, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 63, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6102": { "addr": "172.21.15.107:6835/1381726420", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 1381726420, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 1381726420, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6102, "incarnation": 61, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:59:50.603881+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6102 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 54, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5925": { "addr": "172.21.15.107:6837/1292188015", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 1292188015, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 1292188015, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5925, "incarnation": 51, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-24T23:58:56.590011+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5925 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 59, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-24T23:59:53.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-24T23:59:53.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.232+0000 7f63eb09a640 1 Processor -- start 2023-10-24T23:59:53.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.232+0000 7f63eb09a640 1 -- start start 2023-10-24T23:59:53.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63eb09a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f63e413bb00 0x7f63e41531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63eb09a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f63e40aa330 0x7f63e406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63eb09a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e4152d40 con 0x7f63e40a9450 2023-10-24T23:59:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63eb09a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e4156100 con 0x7f63e413bb00 2023-10-24T23:59:53.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63eb09a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e41527c0 con 0x7f63e40aa330 2023-10-24T23:59:53.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e8e36640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f63e413bb00 0x7f63e41531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:53.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e9637640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f63e40a9450 0x7f63e4071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43296/0 (socket says 172.21.15.107:43296) 2023-10-24T23:59:53.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e8e36640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f63e413bb00 0x7f63e41531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39792/0 (socket says 172.21.15.107:39792) 2023-10-24T23:59:53.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e9637640 1 -- 172.21.15.107:0/4140917892 learned_addr learned my addr 172.21.15.107:0/4140917892 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-24T23:59:53.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e3fff640 1 --2- 172.21.15.107:0/4140917892 >> v2:172.21.15.110:3300/0 conn(0x7f63e40aa330 0x7f63e406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:53.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e8e36640 1 -- 172.21.15.107:0/4140917892 >> v2:172.21.15.110:3300/0 conn(0x7f63e40aa330 msgr2=0x7f63e406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-24T23:59:53.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e8e36640 1 --2- 172.21.15.107:0/4140917892 >> v2:172.21.15.110:3300/0 conn(0x7f63e40aa330 0x7f63e406e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:53.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e8e36640 1 -- 172.21.15.107:0/4140917892 >> v1:172.21.15.107:6789/0 conn(0x7f63e40a9450 legacy=0x7f63e4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:53.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e37fe640 1 -- 172.21.15.107:0/4140917892 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655279931 0 0) 0x7f63e4152d40 con 0x7f63e40a9450 2023-10-24T23:59:53.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e8e36640 1 -- 172.21.15.107:0/4140917892 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e406d660 con 0x7f63e413bb00 2023-10-24T23:59:53.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e8e36640 1 --2- 172.21.15.107:0/4140917892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e413bb00 0x7f63e41531d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f63e4066a60 tx=0x7f63dc01c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b547f9023bdeda server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:53.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e37fe640 1 -- 172.21.15.107:0/4140917892 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc0a57c0 con 0x7f63e413bb00 2023-10-24T23:59:53.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e37fe640 1 -- 172.21.15.107:0/4140917892 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63dc0a5920 con 0x7f63e413bb00 2023-10-24T23:59:53.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.236+0000 7f63e37fe640 1 -- 172.21.15.107:0/4140917892 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc0a5cb0 con 0x7f63e413bb00 2023-10-24T23:59:53.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 -- 172.21.15.107:0/4140917892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e413bb00 msgr2=0x7f63e41531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:53.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 --2- 172.21.15.107:0/4140917892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e413bb00 0x7f63e41531d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f63e4066a60 tx=0x7f63dc01c7f0 comp rx=0 tx=0).stop 2023-10-24T23:59:53.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 -- 172.21.15.107:0/4140917892 shutdown_connections 2023-10-24T23:59:53.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 --2- 172.21.15.107:0/4140917892 >> v2:172.21.15.110:3300/0 conn(0x7f63e40aa330 0x7f63e406e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:53.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 --2- 172.21.15.107:0/4140917892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e413bb00 0x7f63e41531d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:53.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 -- 172.21.15.107:0/4140917892 >> 172.21.15.107:0/4140917892 conn(0x7f63e4070f60 msgr2=0x7f63e4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:53.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 -- 172.21.15.107:0/4140917892 shutdown_connections 2023-10-24T23:59:53.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 -- 172.21.15.107:0/4140917892 wait complete. 2023-10-24T23:59:53.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 Processor -- start 2023-10-24T23:59:53.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 -- start start 2023-10-24T23:59:53.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e40a9450 0x7f63e40c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:53.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63e40aa330 0x7f63e40c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:53.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63e413bb00 0x7f63e40c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:53.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e4130b50 con 0x7f63e40a9450 2023-10-24T23:59:53.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e41316d0 con 0x7f63e413bb00 2023-10-24T23:59:53.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e4130da0 con 0x7f63e40aa330 2023-10-24T23:59:53.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e8e36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e40a9450 0x7f63e40c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:53.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e8e36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e40a9450 0x7f63e40c7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39808/0 (socket says 172.21.15.107:39808) 2023-10-24T23:59:53.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e8e36640 1 -- 172.21.15.107:0/1409134236 learned_addr learned my addr 172.21.15.107:0/1409134236 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-24T23:59:53.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e3fff640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63e40aa330 0x7f63e40c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:53.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e9637640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63e413bb00 0x7f63e40c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:53.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e8e36640 1 -- 172.21.15.107:0/1409134236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63e40aa330 msgr2=0x7f63e40c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:53.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e8e36640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63e40aa330 0x7f63e40c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:53.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e8e36640 1 -- 172.21.15.107:0/1409134236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63e413bb00 msgr2=0x7f63e40c6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:53.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e8e36640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63e413bb00 0x7f63e40c6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:53.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e8e36640 1 -- 172.21.15.107:0/1409134236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc009a70 con 0x7f63e40a9450 2023-10-24T23:59:53.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e8e36640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e40a9450 0x7f63e40c7000 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f63dc01c830 tx=0x7f63dc019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:53.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e17fa640 1 -- 172.21.15.107:0/1409134236 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc0ad650 con 0x7f63e40a9450 2023-10-24T23:59:53.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e17fa640 1 -- 172.21.15.107:0/1409134236 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63dc0ad7b0 con 0x7f63e40a9450 2023-10-24T23:59:53.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 -- 172.21.15.107:0/1409134236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e4131080 con 0x7f63e40a9450 2023-10-24T23:59:53.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63e17fa640 1 -- 172.21.15.107:0/1409134236 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc0adb20 con 0x7f63e40a9450 2023-10-24T23:59:53.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.240+0000 7f63eb09a640 1 -- 172.21.15.107:0/1409134236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e4135f40 con 0x7f63e40a9450 2023-10-24T23:59:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.244+0000 7f63eb09a640 1 -- 172.21.15.107:0/1409134236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e40746e0 con 0x7f63e40a9450 2023-10-24T23:59:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.244+0000 7f63e17fa640 1 -- 172.21.15.107:0/1409134236 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f63dc0adc80 con 0x7f63e40a9450 2023-10-24T23:59:53.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.244+0000 7f63e17fa640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c4065e90 0x7f63c4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-24T23:59:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.244+0000 7f63e17fa640 1 -- 172.21.15.107:0/1409134236 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7f63dc11f460 con 0x7f63e40a9450 2023-10-24T23:59:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.244+0000 7f63e3fff640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c4065e90 0x7f63c4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-24T23:59:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.244+0000 7f63e3fff640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c4065e90 0x7f63c4068350 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f63e4152320 tx=0x7f63cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-24T23:59:53.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.248+0000 7f63e17fa640 1 -- 172.21.15.107:0/1409134236 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f63dc0ec520 con 0x7f63e40a9450 2023-10-24T23:59:53.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.488+0000 7f63eb09a640 1 -- 172.21.15.107:0/1409134236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63e4151e40 con 0x7f63e40a9450 2023-10-24T23:59:53.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.488+0000 7f63e17fa640 1 -- 172.21.15.107:0/1409134236 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5017 (secure 0 0 0) 0x7f63dc0bc090 con 0x7f63e40a9450 2023-10-24T23:59:53.491 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-24T23:59:53.491 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-24T23:59:53.491 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 64 2023-10-24T23:59:53.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 -- 172.21.15.107:0/1409134236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c4065e90 msgr2=0x7f63c4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:53.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c4065e90 0x7f63c4068350 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f63e4152320 tx=0x7f63cc008040 comp rx=0 tx=0).stop 2023-10-24T23:59:53.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 -- 172.21.15.107:0/1409134236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e40a9450 msgr2=0x7f63e40c7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-24T23:59:53.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e40a9450 0x7f63e40c7000 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f63dc01c830 tx=0x7f63dc019590 comp rx=0 tx=0).stop 2023-10-24T23:59:53.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 -- 172.21.15.107:0/1409134236 shutdown_connections 2023-10-24T23:59:53.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63e413bb00 0x7f63e40c6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:53.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c4065e90 0x7f63c4068350 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:53.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63e40aa330 0x7f63e40c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:53.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 --2- 172.21.15.107:0/1409134236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e40a9450 0x7f63e40c7000 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-24T23:59:53.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 -- 172.21.15.107:0/1409134236 >> 172.21.15.107:0/1409134236 conn(0x7f63e4070f60 msgr2=0x7f63e4151730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-24T23:59:53.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 -- 172.21.15.107:0/1409134236 shutdown_connections 2023-10-24T23:59:53.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-24T23:59:53.492+0000 7f63eb09a640 1 -- 172.21.15.107:0/1409134236 wait complete. 2023-10-24T23:59:53.510 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=5293 2023-10-24T23:59:53.511 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before reviving mds.b 2023-10-24T23:59:55.677 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-24T23:59:55.676+0000 7fdcd7180640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:00:00.360 INFO:tasks.ceph.mon.c.smithi110.stderr:2023-10-25T00:00:00.359+0000 7f8362cbc640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 45992) UID: 0 2023-10-25T00:00:00.361 INFO:tasks.ceph.mon.c.smithi110.stderr:2023-10-25T00:00:00.359+0000 7f8362cbc640 -1 mon.c@2(peon) e1 *** Got Signal Hangup *** 2023-10-25T00:00:00.361 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-25T00:00:00.359+0000 7fd748e98640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 45992) UID: 0 2023-10-25T00:00:00.362 INFO:tasks.ceph.mon.b.smithi110.stderr:2023-10-25T00:00:00.359+0000 7f62ad598640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 45992) UID: 0 2023-10-25T00:00:00.362 INFO:tasks.ceph.mon.b.smithi110.stderr:2023-10-25T00:00:00.359+0000 7f62ad598640 -1 mon.b@1(peon) e1 *** Got Signal Hangup *** 2023-10-25T00:00:00.362 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-10-25T00:00:00.359+0000 7fa70c681640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 45992) UID: 0 2023-10-25T00:00:00.363 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-10-25T00:00:00.359+0000 7f3847a92640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 45992) UID: 0 2023-10-25T00:00:00.363 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-10-25T00:00:00.359+0000 7f59769ca640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 45992) UID: 0 2023-10-25T00:00:00.364 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-10-25T00:00:00.359+0000 7fab93f92640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 45992) UID: 0 2023-10-25T00:00:00.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi110.stderr:2023-10-25T00:00:00.359+0000 7fa083336640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 45992) UID: 0 2023-10-25T00:00:00.365 INFO:tasks.ceph.mds.d.smithi110.stderr:2023-10-25T00:00:00.359+0000 7f6f6a052640 -1 received signal: Hangup from (PID: 45992) UID: 0 2023-10-25T00:00:00.375 INFO:tasks.ceph.mon.b.smithi110.stderr:2023-10-25T00:00:00.375+0000 7f62ad598640 -1 received signal: Hangup from (PID: 45993) UID: 0 2023-10-25T00:00:00.375 INFO:tasks.ceph.mon.b.smithi110.stderr:2023-10-25T00:00:00.375+0000 7f62ad598640 -1 mon.b@1(peon) e1 *** Got Signal Hangup *** 2023-10-25T00:00:00.375 INFO:tasks.ceph.mon.c.smithi110.stderr:2023-10-25T00:00:00.375+0000 7f8362cbc640 -1 received signal: Hangup from (PID: 45993) UID: 0 2023-10-25T00:00:00.376 INFO:tasks.ceph.mon.c.smithi110.stderr:2023-10-25T00:00:00.375+0000 7f8362cbc640 -1 mon.c@2(peon) e1 *** Got Signal Hangup *** 2023-10-25T00:00:00.376 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-10-25T00:00:00.375+0000 7f3847a92640 -1 received signal: Hangup from (PID: 45993) UID: 0 2023-10-25T00:00:00.376 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-10-25T00:00:00.375+0000 7f59769ca640 -1 received signal: Hangup from (PID: 45993) UID: 0 2023-10-25T00:00:00.377 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-10-25T00:00:00.375+0000 7fab93f92640 -1 received signal: Hangup from (PID: 45993) UID: 0 2023-10-25T00:00:00.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi110.stderr:2023-10-25T00:00:00.375+0000 7fa083336640 -1 received signal: Hangup from (PID: 45993) UID: 0 2023-10-25T00:00:00.378 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-10-25T00:00:00.375+0000 7fa70c681640 -1 received signal: Hangup from (PID: 45993) UID: 0 2023-10-25T00:00:00.378 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-10-25T00:00:00.375+0000 7fd748e98640 -1 received signal: Hangup from (PID: 45993) UID: 0 2023-10-25T00:00:00.379 INFO:tasks.ceph.mds.d.smithi110.stderr:2023-10-25T00:00:00.375+0000 7f6f6a052640 -1 Fail to open '/proc/45993/cmdline' error = (2) No such file or directory 2023-10-25T00:00:00.379 INFO:tasks.ceph.mds.d.smithi110.stderr:2023-10-25T00:00:00.375+0000 7f6f6a052640 -1 received signal: Hangup from (PID: 45993) UID: 0 2023-10-25T00:00:01.511 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2023-10-25T00:00:01.512 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-25T00:00:01.512 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-25T00:00:01.514 INFO:tasks.ceph.mds.b:Started 2023-10-25T00:00:01.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:01.582 INFO:tasks.ceph.mds.b.smithi110.stdout:starting mds.b at 2023-10-25T00:00:01.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.652+0000 7f1e9d71f640 1 Processor -- start 2023-10-25T00:00:01.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.652+0000 7f1e9d71f640 1 -- start start 2023-10-25T00:00:01.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e9d71f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1e9814c470 0x7f1e9814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:01.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e9d71f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e98155cd0 con 0x7f1e98152b30 2023-10-25T00:00:01.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e9d71f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e980c4380 con 0x7f1e9806f490 2023-10-25T00:00:01.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e9d71f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e981526e0 con 0x7f1e9814c470 2023-10-25T00:00:01.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e977fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1e98152b30 0x7f1e98152f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43302/0 (socket says 172.21.15.107:43302) 2023-10-25T00:00:01.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e977fe640 1 -- 172.21.15.107:0/1759616271 learned_addr learned my addr 172.21.15.107:0/1759616271 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:00:01.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e96ffd640 1 --2- 172.21.15.107:0/1759616271 >> v2:172.21.15.110:3300/0 conn(0x7f1e9814c470 0x7f1e9814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:01.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e95ffb640 1 -- 172.21.15.107:0/1759616271 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791781576 0 0) 0x7f1e98155cd0 con 0x7f1e98152b30 2023-10-25T00:00:01.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e95ffb640 1 -- 172.21.15.107:0/1759616271 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e80003530 con 0x7f1e98152b30 2023-10-25T00:00:01.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e95ffb640 1 -- 172.21.15.107:0/1759616271 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065151064 0 0) 0x7f1e980c4380 con 0x7f1e9806f490 2023-10-25T00:00:01.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e95ffb640 1 -- 172.21.15.107:0/1759616271 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e98155cd0 con 0x7f1e9806f490 2023-10-25T00:00:01.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e96ffd640 1 -- 172.21.15.107:0/1759616271 >> v1:172.21.15.110:6789/0 conn(0x7f1e9806f490 legacy=0x7f1e98071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:01.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e96ffd640 1 -- 172.21.15.107:0/1759616271 >> v1:172.21.15.107:6789/0 conn(0x7f1e98152b30 legacy=0x7f1e98152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:01.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e96ffd640 1 -- 172.21.15.107:0/1759616271 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e980c48b0 con 0x7f1e9814c470 2023-10-25T00:00:01.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e96ffd640 1 --2- 172.21.15.107:0/1759616271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e9814c470 0x7f1e9814c850 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1e880097c0 tx=0x7f1e8801c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d5256d41c3d3d40 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:01.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e95ffb640 1 -- 172.21.15.107:0/1759616271 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e880ab600 con 0x7f1e9814c470 2023-10-25T00:00:01.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e95ffb640 1 -- 172.21.15.107:0/1759616271 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e880a7050 con 0x7f1e9814c470 2023-10-25T00:00:01.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.656+0000 7f1e95ffb640 1 -- 172.21.15.107:0/1759616271 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e880ab980 con 0x7f1e9814c470 2023-10-25T00:00:01.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1759616271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e9814c470 msgr2=0x7f1e9814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:01.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 --2- 172.21.15.107:0/1759616271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e9814c470 0x7f1e9814c850 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1e880097c0 tx=0x7f1e8801c360 comp rx=0 tx=0).stop 2023-10-25T00:00:01.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1759616271 shutdown_connections 2023-10-25T00:00:01.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 --2- 172.21.15.107:0/1759616271 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e9814c470 0x7f1e9814c850 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:01.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1759616271 >> 172.21.15.107:0/1759616271 conn(0x7f1e98070f60 msgr2=0x7f1e98071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:01.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1759616271 shutdown_connections 2023-10-25T00:00:01.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1759616271 wait complete. 2023-10-25T00:00:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 Processor -- start 2023-10-25T00:00:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 -- start start 2023-10-25T00:00:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e9806f490 0x7f1e98154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:01.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e9814c470 0x7f1e98144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e98152b30 0x7f1e98144e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e98143e20 con 0x7f1e98152b30 2023-10-25T00:00:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e980c1ae0 con 0x7f1e9806f490 2023-10-25T00:00:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e980c14d0 con 0x7f1e9814c470 2023-10-25T00:00:01.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e977fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e98152b30 0x7f1e98144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e977fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e98152b30 0x7f1e98144e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39816/0 (socket says 172.21.15.107:39816) 2023-10-25T00:00:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e977fe640 1 -- 172.21.15.107:0/1835982498 learned_addr learned my addr 172.21.15.107:0/1835982498 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e96ffd640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e9806f490 0x7f1e98154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e967fc640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e9814c470 0x7f1e98144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:01.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e977fe640 1 -- 172.21.15.107:0/1835982498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e9814c470 msgr2=0x7f1e98144910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:01.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e977fe640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e9814c470 0x7f1e98144910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:01.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e977fe640 1 -- 172.21.15.107:0/1835982498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e9806f490 msgr2=0x7f1e98154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:01.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e977fe640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e9806f490 0x7f1e98154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:01.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e977fe640 1 -- 172.21.15.107:0/1835982498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e88009420 con 0x7f1e98152b30 2023-10-25T00:00:01.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e96ffd640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e9806f490 0x7f1e98154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:00:01.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e977fe640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e98152b30 0x7f1e98144e50 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000caa0 tx=0x7f1e9000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e7f7fe640 1 -- 172.21.15.107:0/1835982498 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9000ac00 con 0x7f1e98152b30 2023-10-25T00:00:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e7f7fe640 1 -- 172.21.15.107:0/1835982498 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9000ad60 con 0x7f1e98152b30 2023-10-25T00:00:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1835982498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e981f0650 con 0x7f1e98152b30 2023-10-25T00:00:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e7f7fe640 1 -- 172.21.15.107:0/1835982498 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e90012600 con 0x7f1e98152b30 2023-10-25T00:00:01.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.660+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1835982498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e981f0930 con 0x7f1e98152b30 2023-10-25T00:00:01.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.664+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1835982498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e980c1c70 con 0x7f1e98152b30 2023-10-25T00:00:01.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.664+0000 7f1e7f7fe640 1 -- 172.21.15.107:0/1835982498 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1e90007070 con 0x7f1e98152b30 2023-10-25T00:00:01.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.664+0000 7f1e7f7fe640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e84065e90 0x7f1e84068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:01.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.664+0000 7f1e7f7fe640 1 -- 172.21.15.107:0/1835982498 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7f1e90083d90 con 0x7f1e98152b30 2023-10-25T00:00:01.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.664+0000 7f1e96ffd640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e84065e90 0x7f1e84068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:01.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.664+0000 7f1e96ffd640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e84065e90 0x7f1e84068350 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1e880ad000 tx=0x7f1e88008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:01.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.668+0000 7f1e7f7fe640 1 -- 172.21.15.107:0/1835982498 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1e90050d40 con 0x7f1e98152b30 2023-10-25T00:00:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.916+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1835982498 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e980746e0 con 0x7f1e98152b30 2023-10-25T00:00:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.916+0000 7f1e7f7fe640 1 -- 172.21.15.107:0/1835982498 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5017 (secure 0 0 0) 0x7f1e90017090 con 0x7f1e98152b30 2023-10-25T00:00:01.917 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:01.917 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 64 2023-10-25T00:00:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.916+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1835982498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e84065e90 msgr2=0x7f1e84068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.916+0000 7f1e9d71f640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e84065e90 0x7f1e84068350 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1e880ad000 tx=0x7f1e88008040 comp rx=0 tx=0).stop 2023-10-25T00:00:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.916+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1835982498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e98152b30 msgr2=0x7f1e98144e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.916+0000 7f1e9d71f640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e98152b30 0x7f1e98144e50 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1e9000caa0 tx=0x7f1e9000cf60 comp rx=0 tx=0).stop 2023-10-25T00:00:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.920+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1835982498 shutdown_connections 2023-10-25T00:00:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.920+0000 7f1e9d71f640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e98152b30 0x7f1e98144e50 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:01.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.920+0000 7f1e9d71f640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e84065e90 0x7f1e84068350 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.920+0000 7f1e9d71f640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e9814c470 0x7f1e98144910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.920+0000 7f1e9d71f640 1 --2- 172.21.15.107:0/1835982498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e9806f490 0x7f1e98154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.920+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1835982498 >> 172.21.15.107:0/1835982498 conn(0x7f1e98070f60 msgr2=0x7f1e98151bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.920+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1835982498 shutdown_connections 2023-10-25T00:00:01.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:01.920+0000 7f1e9d71f640 1 -- 172.21.15.107:0/1835982498 wait complete. 2023-10-25T00:00:01.941 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-25T00:00:03.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:04.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b7fbe640 1 Processor -- start 2023-10-25T00:00:04.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b7fbe640 1 -- start start 2023-10-25T00:00:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b7fbe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0b0156220 0x7fc0b006df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b7fbe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b015c200 0x7fc0b0071100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b7fbe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b00c5d40 con 0x7fc0b00c6a30 2023-10-25T00:00:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b7fbe640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b00c6130 con 0x7fc0b0156220 2023-10-25T00:00:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b7fbe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b0158f60 con 0x7fc0b015c200 2023-10-25T00:00:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b5d5a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0b0156220 0x7fc0b006df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b5559640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b015c200 0x7fc0b0071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b655b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc0b00c6a30 0x7fc0b0071640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:57592/0 (socket says 172.21.15.107:57592) 2023-10-25T00:00:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b5d5a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0b0156220 0x7fc0b006df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43146/0 (socket says 172.21.15.107:43146) 2023-10-25T00:00:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b655b640 1 -- 172.21.15.107:0/2612298484 learned_addr learned my addr 172.21.15.107:0/2612298484 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:00:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b4d58640 1 -- 172.21.15.107:0/2612298484 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282245036 0 0) 0x7fc0b00c5d40 con 0x7fc0b00c6a30 2023-10-25T00:00:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b5d5a640 1 -- 172.21.15.107:0/2612298484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b015c200 msgr2=0x7fc0b0071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b5d5a640 1 --2- 172.21.15.107:0/2612298484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b015c200 0x7fc0b0071100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b5d5a640 1 -- 172.21.15.107:0/2612298484 >> v1:172.21.15.110:6789/0 conn(0x7fc0b00c6a30 legacy=0x7fc0b0071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b5d5a640 1 -- 172.21.15.107:0/2612298484 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b0159240 con 0x7fc0b0156220 2023-10-25T00:00:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.088+0000 7fc0b5559640 1 --2- 172.21.15.107:0/2612298484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b015c200 0x7fc0b0071100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:00:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b5d5a640 1 --2- 172.21.15.107:0/2612298484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b0156220 0x7fc0b006df30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac0097c0 tx=0x7fc0ac01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e85f40b5e9bd561 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b4d58640 1 -- 172.21.15.107:0/2612298484 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac0ac620 con 0x7fc0b0156220 2023-10-25T00:00:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b4d58640 1 -- 172.21.15.107:0/2612298484 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ac0ac780 con 0x7fc0b0156220 2023-10-25T00:00:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b4d58640 1 -- 172.21.15.107:0/2612298484 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac0acb10 con 0x7fc0b0156220 2023-10-25T00:00:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/2612298484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b0156220 msgr2=0x7fc0b006df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b7fbe640 1 --2- 172.21.15.107:0/2612298484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b0156220 0x7fc0b006df30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac0097c0 tx=0x7fc0ac01c3c0 comp rx=0 tx=0).stop 2023-10-25T00:00:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/2612298484 shutdown_connections 2023-10-25T00:00:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b7fbe640 1 --2- 172.21.15.107:0/2612298484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b015c200 0x7fc0b0071100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b7fbe640 1 --2- 172.21.15.107:0/2612298484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b0156220 0x7fc0b006df30 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/2612298484 >> 172.21.15.107:0/2612298484 conn(0x7fc0b00c39a0 msgr2=0x7fc0b00c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/2612298484 shutdown_connections 2023-10-25T00:00:04.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/2612298484 wait complete. 2023-10-25T00:00:04.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b7fbe640 1 Processor -- start 2023-10-25T00:00:04.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.092+0000 7fc0b7fbe640 1 -- start start 2023-10-25T00:00:04.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b7fbe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b00c6a30 0x7fc0b00a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:04.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b7fbe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b0156220 0x7fc0b01f49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:04.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b7fbe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015c200 0x7fc0b0151390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:04.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b7fbe640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b01f4f00 con 0x7fc0b015c200 2023-10-25T00:00:04.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b7fbe640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b00aad80 con 0x7fc0b0156220 2023-10-25T00:00:04.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b7fbe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b01f5130 con 0x7fc0b00c6a30 2023-10-25T00:00:04.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b655b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015c200 0x7fc0b0151390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:04.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b655b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015c200 0x7fc0b0151390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43152/0 (socket says 172.21.15.107:43152) 2023-10-25T00:00:04.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b5d5a640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b00c6a30 0x7fc0b00a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:04.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b5559640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b0156220 0x7fc0b01f49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:04.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b655b640 1 -- 172.21.15.107:0/298038353 learned_addr learned my addr 172.21.15.107:0/298038353 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:04.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b655b640 1 -- 172.21.15.107:0/298038353 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b00c6a30 msgr2=0x7fc0b00a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:04.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b655b640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b00c6a30 0x7fc0b00a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:04.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b655b640 1 -- 172.21.15.107:0/298038353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b0156220 msgr2=0x7fc0b01f49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:04.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b655b640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b0156220 0x7fc0b01f49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:04.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b655b640 1 -- 172.21.15.107:0/298038353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a0009420 con 0x7fc0b015c200 2023-10-25T00:00:04.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b5559640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b0156220 0x7fc0b01f49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:00:04.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b655b640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015c200 0x7fc0b0151390 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8001570 tx=0x7fc0a800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:04.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0a67fc640 1 -- 172.21.15.107:0/298038353 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a800a640 con 0x7fc0b015c200 2023-10-25T00:00:04.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0a67fc640 1 -- 172.21.15.107:0/298038353 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a800a7c0 con 0x7fc0b015c200 2023-10-25T00:00:04.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0a67fc640 1 -- 172.21.15.107:0/298038353 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a800ab70 con 0x7fc0b015c200 2023-10-25T00:00:04.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/298038353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ac009420 con 0x7fc0b015c200 2023-10-25T00:00:04.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/298038353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b01f56f0 con 0x7fc0b015c200 2023-10-25T00:00:04.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/298038353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b0070480 con 0x7fc0b015c200 2023-10-25T00:00:04.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0a67fc640 1 -- 172.21.15.107:0/298038353 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc0a800acd0 con 0x7fc0b015c200 2023-10-25T00:00:04.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0a67fc640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc094065e60 0x7fc094068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:04.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0a67fc640 1 -- 172.21.15.107:0/298038353 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7fc0a8083320 con 0x7fc0b015c200 2023-10-25T00:00:04.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.096+0000 7fc0b5d5a640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc094065e60 0x7fc094068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:04.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.100+0000 7fc0b5d5a640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc094065e60 0x7fc094068320 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac0ae000 tx=0x7fc0ac0a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:04.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.104+0000 7fc0a67fc640 1 -- 172.21.15.107:0/298038353 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc0a8050370 con 0x7fc0b015c200 2023-10-25T00:00:04.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/298038353 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0b015c5e0 con 0x7fc0b015c200 2023-10-25T00:00:04.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0a67fc640 1 -- 172.21.15.107:0/298038353 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fc0a8018020 con 0x7fc0b015c200 2023-10-25T00:00:04.413 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:04.413 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:04.414 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 65 2023-10-25T00:00:04.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/298038353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc094065e60 msgr2=0x7fc094068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:04.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc094065e60 0x7fc094068320 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac0ae000 tx=0x7fc0ac0a7000 comp rx=0 tx=0).stop 2023-10-25T00:00:04.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/298038353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015c200 msgr2=0x7fc0b0151390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:04.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015c200 0x7fc0b0151390 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8001570 tx=0x7fc0a800cca0 comp rx=0 tx=0).stop 2023-10-25T00:00:04.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/298038353 shutdown_connections 2023-10-25T00:00:04.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b015c200 0x7fc0b0151390 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:04.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b0156220 0x7fc0b01f49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:04.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc094065e60 0x7fc094068320 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:04.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 --2- 172.21.15.107:0/298038353 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b00c6a30 0x7fc0b00a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:04.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/298038353 >> 172.21.15.107:0/298038353 conn(0x7fc0b00c39a0 msgr2=0x7fc0b0157790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:04.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/298038353 shutdown_connections 2023-10-25T00:00:04.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:04.412+0000 7fc0b7fbe640 1 -- 172.21.15.107:0/298038353 wait complete. 2023-10-25T00:00:04.433 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2023-10-25T00:00:04.433 INFO:tasks.mds_thrash.fs.[a]:waiting for 69 secs before thrashing 2023-10-25T00:00:05.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-25T00:00:05.657 DEBUG:teuthology.orchestra.run.smithi110:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-25T00:00:05.667 INFO:teuthology.orchestra.run.smithi110.stderr:error: state file /var/lib/logrotate/status is already locked 2023-10-25T00:00:05.667 INFO:teuthology.orchestra.run.smithi110.stderr:logrotate does not support parallel execution on the same set of logfiles. 2023-10-25T00:00:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 3 2023-10-25T00:00:06.022 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/ceph.py", line 188, in invoke_logrotate run.wait(procs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-10-25T00:00:11.086 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-10-25T00:00:11.084+0000 7fba47438640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 54342) UID: 0 2023-10-25T00:00:11.086 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-10-25T00:00:11.084+0000 7fba47438640 -1 mon.a@0(leader) e1 *** Got Signal Hangup *** 2023-10-25T00:00:11.087 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-25T00:00:11.084+0000 7fc8f6b15640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 54342) UID: 0 2023-10-25T00:00:11.087 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-10-25T00:00:11.084+0000 7ff5f852c640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 54342) UID: 0 2023-10-25T00:00:11.088 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-10-25T00:00:11.084+0000 7fced08b2640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 54342) UID: 0 2023-10-25T00:00:11.088 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-10-25T00:00:11.084+0000 7f267b2c8640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 54342) UID: 0 2023-10-25T00:00:11.089 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-10-25T00:00:11.084+0000 7f148bb3e640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 54342) UID: 0 2023-10-25T00:00:11.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:2023-10-25T00:00:11.084+0000 7f46bf5ac640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 54342) UID: 0 2023-10-25T00:00:11.089 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:00:11.088+0000 7efff31ed640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 54342) UID: 0 2023-10-25T00:00:11.090 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:00:11.088+0000 7fdcdb188640 -1 received signal: Hangup from killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror (PID: 54342) UID: 0 2023-10-25T00:00:11.111 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-10-25T00:00:11.108+0000 7fba47438640 -1 received signal: Hangup from (PID: 54343) UID: 0 2023-10-25T00:00:11.112 INFO:tasks.ceph.mon.a.smithi107.stderr:2023-10-25T00:00:11.108+0000 7fba47438640 -1 mon.a@0(leader) e1 *** Got Signal Hangup *** 2023-10-25T00:00:11.112 INFO:tasks.ceph.osd.0.smithi107.stderr:2023-10-25T00:00:11.108+0000 7f148bb3e640 -1 received signal: Hangup from (PID: 54343) UID: 0 2023-10-25T00:00:11.113 INFO:tasks.ceph.osd.3.smithi107.stderr:2023-10-25T00:00:11.108+0000 7f267b2c8640 -1 received signal: Hangup from (PID: 54343) UID: 0 2023-10-25T00:00:11.113 INFO:tasks.ceph.mgr.y.smithi107.stderr:2023-10-25T00:00:11.108+0000 7fc8f6b15640 -1 received signal: Hangup from (PID: 54343) UID: 0 2023-10-25T00:00:11.113 INFO:tasks.ceph.osd.2.smithi107.stderr:2023-10-25T00:00:11.108+0000 7fced08b2640 -1 received signal: Hangup from (PID: 54343) UID: 0 2023-10-25T00:00:11.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:2023-10-25T00:00:11.108+0000 7f46bf5ac640 -1 received signal: Hangup from (PID: 54343) UID: 0 2023-10-25T00:00:11.114 INFO:tasks.ceph.osd.1.smithi107.stderr:2023-10-25T00:00:11.108+0000 7ff5f852c640 -1 received signal: Hangup from (PID: 54343) UID: 0 2023-10-25T00:00:11.115 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:00:11.108+0000 7efff31ed640 -1 received signal: Hangup from (PID: 54343) UID: 0 2023-10-25T00:00:11.115 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:00:11.108+0000 7fdcdb188640 -1 received signal: Hangup from (PID: 54343) UID: 0 2023-10-25T00:00:34.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.516+0000 7f07795ca640 1 Processor -- start 2023-10-25T00:00:34.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.516+0000 7f07795ca640 1 -- start start 2023-10-25T00:00:34.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f07795ca640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f07740c2bb0 0x7f07740c1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:34.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f07795ca640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f07740c1c60 0x7f077415f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:34.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f07795ca640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0774069ef0 con 0x7f07740c27d0 2023-10-25T00:00:34.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f07795ca640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07741556d0 con 0x7f07740c1c60 2023-10-25T00:00:34.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f07795ca640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0774154860 con 0x7f07740c2bb0 2023-10-25T00:00:34.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f0778dc9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f07740c1c60 0x7f077415f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:34.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f0778dc9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f07740c1c60 0x7f077415f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43344/0 (socket says 172.21.15.107:43344) 2023-10-25T00:00:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f0778dc9640 1 -- 172.21.15.107:0/749370930 learned_addr learned my addr 172.21.15.107:0/749370930 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f076bfff640 1 --2- 172.21.15.107:0/749370930 >> v2:172.21.15.110:3300/0 conn(0x7f07740c2bb0 0x7f07740c1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f0778dc9640 1 -- 172.21.15.107:0/749370930 >> v2:172.21.15.110:3300/0 conn(0x7f07740c2bb0 msgr2=0x7f07740c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f0778dc9640 1 --2- 172.21.15.107:0/749370930 >> v2:172.21.15.110:3300/0 conn(0x7f07740c2bb0 0x7f07740c1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f0778dc9640 1 -- 172.21.15.107:0/749370930 >> v1:172.21.15.110:6789/0 conn(0x7f07740c27d0 legacy=0x7f0774072030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:34.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f0778dc9640 1 -- 172.21.15.107:0/749370930 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07740c3030 con 0x7f07740c1c60 2023-10-25T00:00:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f0778dc9640 1 --2- 172.21.15.107:0/749370930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07740c1c60 0x7f077415f100 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f07740669b0 tx=0x7f076c01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd5f99dca98cf89 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f07737fe640 1 -- 172.21.15.107:0/749370930 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c0a56b0 con 0x7f07740c1c60 2023-10-25T00:00:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f07737fe640 1 -- 172.21.15.107:0/749370930 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c0a5810 con 0x7f07740c1c60 2023-10-25T00:00:34.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f07737fe640 1 -- 172.21.15.107:0/749370930 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c0a5ba0 con 0x7f07740c1c60 2023-10-25T00:00:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f07795ca640 1 -- 172.21.15.107:0/749370930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07740c1c60 msgr2=0x7f077415f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.520+0000 7f07795ca640 1 --2- 172.21.15.107:0/749370930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07740c1c60 0x7f077415f100 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f07740669b0 tx=0x7f076c01c900 comp rx=0 tx=0).stop 2023-10-25T00:00:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 -- 172.21.15.107:0/749370930 shutdown_connections 2023-10-25T00:00:34.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 --2- 172.21.15.107:0/749370930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07740c1c60 0x7f077415f100 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 --2- 172.21.15.107:0/749370930 >> v2:172.21.15.110:3300/0 conn(0x7f07740c2bb0 0x7f07740c1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 -- 172.21.15.107:0/749370930 >> 172.21.15.107:0/749370930 conn(0x7f07740c6cc0 msgr2=0x7f07740c70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 -- 172.21.15.107:0/749370930 shutdown_connections 2023-10-25T00:00:34.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 -- 172.21.15.107:0/749370930 wait complete. 2023-10-25T00:00:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 Processor -- start 2023-10-25T00:00:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 -- start start 2023-10-25T00:00:34.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f07740c1c60 0x7f077414e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:34.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07740c27d0 0x7f07740a9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:34.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07740c2bb0 0x7f07740aa430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:34.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0774155bd0 con 0x7f07740c2bb0 2023-10-25T00:00:34.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07740c09f0 con 0x7f07740c27d0 2023-10-25T00:00:34.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0774157000 con 0x7f07740c1c60 2023-10-25T00:00:34.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f0778dc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07740c2bb0 0x7f07740aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:34.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f0778dc9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07740c2bb0 0x7f07740aa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43352/0 (socket says 172.21.15.107:43352) 2023-10-25T00:00:34.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f0778dc9640 1 -- 172.21.15.107:0/1063071079 learned_addr learned my addr 172.21.15.107:0/1063071079 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:34.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f076bfff640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07740c27d0 0x7f07740a9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:34.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f0773fff640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f07740c1c60 0x7f077414e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f0778dc9640 1 -- 172.21.15.107:0/1063071079 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f07740c1c60 msgr2=0x7f077414e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f0778dc9640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f07740c1c60 0x7f077414e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f0778dc9640 1 -- 172.21.15.107:0/1063071079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07740c27d0 msgr2=0x7f07740a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f0778dc9640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07740c27d0 0x7f07740a9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f0778dc9640 1 -- 172.21.15.107:0/1063071079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c009b30 con 0x7f07740c2bb0 2023-10-25T00:00:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f0778dc9640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07740c2bb0 0x7f07740aa430 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f076c01a530 tx=0x7f076c0196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07717fa640 1 -- 172.21.15.107:0/1063071079 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c0b8670 con 0x7f07740c2bb0 2023-10-25T00:00:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07717fa640 1 -- 172.21.15.107:0/1063071079 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c0b87d0 con 0x7f07740c2bb0 2023-10-25T00:00:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07717fa640 1 -- 172.21.15.107:0/1063071079 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c0b8b60 con 0x7f07740c2bb0 2023-10-25T00:00:34.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 -- 172.21.15.107:0/1063071079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0774158c00 con 0x7f07740c2bb0 2023-10-25T00:00:34.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.524+0000 7f07795ca640 1 -- 172.21.15.107:0/1063071079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07741edcc0 con 0x7f07740c2bb0 2023-10-25T00:00:34.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.528+0000 7f07795ca640 1 -- 172.21.15.107:0/1063071079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0774156240 con 0x7f07740c2bb0 2023-10-25T00:00:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.528+0000 7f07717fa640 1 -- 172.21.15.107:0/1063071079 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f076c0b8cc0 con 0x7f07740c2bb0 2023-10-25T00:00:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.528+0000 7f07717fa640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0754065e90 0x7f0754068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.528+0000 7f07717fa640 1 -- 172.21.15.107:0/1063071079 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7f076c11ef90 con 0x7f07740c2bb0 2023-10-25T00:00:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.528+0000 7f0773fff640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0754065e90 0x7f0754068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.528+0000 7f0773fff640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0754065e90 0x7f0754068350 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f076400b810 tx=0x7f0764008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:34.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.532+0000 7f07717fa640 1 -- 172.21.15.107:0/1063071079 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f076c150040 con 0x7f07740c2bb0 2023-10-25T00:00:34.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.752+0000 7f07795ca640 1 -- 172.21.15.107:0/1063071079 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0774156060 con 0x7f07740c2bb0 2023-10-25T00:00:34.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.752+0000 7f07717fa640 1 -- 172.21.15.107:0/1063071079 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f0774156060 con 0x7f07740c2bb0 2023-10-25T00:00:34.755 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:34.756 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:34.756 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 65 2023-10-25T00:00:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 -- 172.21.15.107:0/1063071079 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0754065e90 msgr2=0x7f0754068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0754065e90 0x7f0754068350 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f076400b810 tx=0x7f0764008040 comp rx=0 tx=0).stop 2023-10-25T00:00:34.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 -- 172.21.15.107:0/1063071079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07740c2bb0 msgr2=0x7f07740aa430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07740c2bb0 0x7f07740aa430 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f076c01a530 tx=0x7f076c0196a0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 -- 172.21.15.107:0/1063071079 shutdown_connections 2023-10-25T00:00:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f07740c2bb0 0x7f07740aa430 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f07740c27d0 0x7f07740a9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0754065e90 0x7f0754068350 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 --2- 172.21.15.107:0/1063071079 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f07740c1c60 0x7f077414e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 -- 172.21.15.107:0/1063071079 >> 172.21.15.107:0/1063071079 conn(0x7f07740c6cc0 msgr2=0x7f07740c1220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 -- 172.21.15.107:0/1063071079 shutdown_connections 2023-10-25T00:00:34.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.756+0000 7f07795ca640 1 -- 172.21.15.107:0/1063071079 wait complete. 2023-10-25T00:00:34.779 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2023-10-25T00:00:34.779 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:00:34.779 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:00:34.791 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:00:34.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:34.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.952+0000 7f318fa8e640 1 Processor -- start 2023-10-25T00:00:34.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.952+0000 7f318fa8e640 1 -- start start 2023-10-25T00:00:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.952+0000 7f318fa8e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f318806cef0 0x7f3188152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.952+0000 7f318fa8e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31880a99d0 con 0x7f318806d2d0 2023-10-25T00:00:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.952+0000 7f318fa8e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318806d740 con 0x7f318806a770 2023-10-25T00:00:34.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.952+0000 7f318fa8e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31880a9e60 con 0x7f318806cef0 2023-10-25T00:00:34.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.952+0000 7f318d82a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f318806cef0 0x7f3188152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:34.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.952+0000 7f318e02b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f318806d2d0 0x7f3188154a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55482/0 (socket says 172.21.15.107:55482) 2023-10-25T00:00:34.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.952+0000 7f318e02b640 1 -- 172.21.15.107:0/3151649231 learned_addr learned my addr 172.21.15.107:0/3151649231 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:00:34.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318d82a640 1 -- 172.21.15.107:0/3151649231 >> v1:172.21.15.110:6789/0 conn(0x7f318806a770 legacy=0x7f3188154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:34.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318d82a640 1 -- 172.21.15.107:0/3151649231 >> v1:172.21.15.107:6789/0 conn(0x7f318806d2d0 legacy=0x7f3188154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:34.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318d82a640 1 -- 172.21.15.107:0/3151649231 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880a9450 con 0x7f318806cef0 2023-10-25T00:00:34.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318d82a640 1 --2- 172.21.15.107:0/3151649231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318806cef0 0x7f3188152fb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f31840097c0 tx=0x7f318401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f76adbc53381d1a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:34.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318c828640 1 -- 172.21.15.107:0/3151649231 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31840a4ea0 con 0x7f318806cef0 2023-10-25T00:00:34.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318c828640 1 -- 172.21.15.107:0/3151649231 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31840a7050 con 0x7f318806cef0 2023-10-25T00:00:34.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318c828640 1 -- 172.21.15.107:0/3151649231 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31840ab820 con 0x7f318806cef0 2023-10-25T00:00:34.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318fa8e640 1 -- 172.21.15.107:0/3151649231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318806cef0 msgr2=0x7f3188152fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:34.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318fa8e640 1 --2- 172.21.15.107:0/3151649231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318806cef0 0x7f3188152fb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f31840097c0 tx=0x7f318401c320 comp rx=0 tx=0).stop 2023-10-25T00:00:34.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318fa8e640 1 -- 172.21.15.107:0/3151649231 shutdown_connections 2023-10-25T00:00:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318fa8e640 1 --2- 172.21.15.107:0/3151649231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318806cef0 0x7f3188152fb0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318fa8e640 1 -- 172.21.15.107:0/3151649231 >> 172.21.15.107:0/3151649231 conn(0x7f31880c6ed0 msgr2=0x7f31880c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318fa8e640 1 -- 172.21.15.107:0/3151649231 shutdown_connections 2023-10-25T00:00:34.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.956+0000 7f318fa8e640 1 -- 172.21.15.107:0/3151649231 wait complete. 2023-10-25T00:00:34.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318fa8e640 1 Processor -- start 2023-10-25T00:00:34.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318fa8e640 1 -- start start 2023-10-25T00:00:34.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318fa8e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318806a770 0x7f31880c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:34.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318fa8e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f318806cef0 0x7f31881503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:34.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318fa8e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f318806d2d0 0x7f31881520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318fa8e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31881512b0 con 0x7f318806a770 2023-10-25T00:00:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318fa8e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31881ea7c0 con 0x7f318806d2d0 2023-10-25T00:00:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318fa8e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318813c610 con 0x7f318806cef0 2023-10-25T00:00:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318d82a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318806a770 0x7f31880c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318d82a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318806a770 0x7f31880c3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43366/0 (socket says 172.21.15.107:43366) 2023-10-25T00:00:34.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318d82a640 1 -- 172.21.15.107:0/2366208663 learned_addr learned my addr 172.21.15.107:0/2366208663 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318d029640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f318806cef0 0x7f31881503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318e02b640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f318806d2d0 0x7f31881520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318d82a640 1 -- 172.21.15.107:0/2366208663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f318806cef0 msgr2=0x7f31881503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318d82a640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f318806cef0 0x7f31881503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318d82a640 1 -- 172.21.15.107:0/2366208663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f318806d2d0 msgr2=0x7f31881520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318d82a640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f318806d2d0 0x7f31881520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318d82a640 1 -- 172.21.15.107:0/2366208663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3184009420 con 0x7f318806a770 2023-10-25T00:00:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318d82a640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318806a770 0x7f31880c3c30 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f31840097c0 tx=0x7f318401ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f317e7fc640 1 -- 172.21.15.107:0/2366208663 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31840b7650 con 0x7f318806a770 2023-10-25T00:00:34.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f317e7fc640 1 -- 172.21.15.107:0/2366208663 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31840abe90 con 0x7f318806a770 2023-10-25T00:00:34.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318fa8e640 1 -- 172.21.15.107:0/2366208663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3188142ac0 con 0x7f318806a770 2023-10-25T00:00:34.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f317e7fc640 1 -- 172.21.15.107:0/2366208663 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31840b7a10 con 0x7f318806a770 2023-10-25T00:00:34.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.960+0000 7f318fa8e640 1 -- 172.21.15.107:0/2366208663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31881eb770 con 0x7f318806a770 2023-10-25T00:00:34.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.964+0000 7f318fa8e640 1 -- 172.21.15.107:0/2366208663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31880746e0 con 0x7f318806a770 2023-10-25T00:00:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.964+0000 7f317e7fc640 1 -- 172.21.15.107:0/2366208663 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f31840b7b70 con 0x7f318806a770 2023-10-25T00:00:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.964+0000 7f317e7fc640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f316c065e30 0x7f316c0682f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.964+0000 7f317e7fc640 1 -- 172.21.15.107:0/2366208663 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7f318411e730 con 0x7f318806a770 2023-10-25T00:00:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.964+0000 7f318d029640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f316c065e30 0x7f316c0682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:34.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.964+0000 7f318d029640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f316c065e30 0x7f316c0682f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f318814f7f0 tx=0x7f3178008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:34.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:34.968+0000 7f317e7fc640 1 -- 172.21.15.107:0/2366208663 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f31840eb640 con 0x7f318806a770 2023-10-25T00:00:35.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.204+0000 7f318fa8e640 1 -- 172.21.15.107:0/2366208663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31880c72b0 con 0x7f318806a770 2023-10-25T00:00:35.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.204+0000 7f317e7fc640 1 -- 172.21.15.107:0/2366208663 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f31840bb090 con 0x7f318806a770 2023-10-25T00:00:35.206 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:35.207 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:35.207 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 65 2023-10-25T00:00:35.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 -- 172.21.15.107:0/2366208663 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f316c065e30 msgr2=0x7f316c0682f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:35.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f316c065e30 0x7f316c0682f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f318814f7f0 tx=0x7f3178008040 comp rx=0 tx=0).stop 2023-10-25T00:00:35.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 -- 172.21.15.107:0/2366208663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318806a770 msgr2=0x7f31880c3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:35.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318806a770 0x7f31880c3c30 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f31840097c0 tx=0x7f318401ab60 comp rx=0 tx=0).stop 2023-10-25T00:00:35.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 -- 172.21.15.107:0/2366208663 shutdown_connections 2023-10-25T00:00:35.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f318806d2d0 0x7f31881520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:35.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f316c065e30 0x7f316c0682f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:35.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f318806cef0 0x7f31881503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:35.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 --2- 172.21.15.107:0/2366208663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f318806a770 0x7f31880c3c30 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:35.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 -- 172.21.15.107:0/2366208663 >> 172.21.15.107:0/2366208663 conn(0x7f31880c6ed0 msgr2=0x7f31880ac1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:35.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 -- 172.21.15.107:0/2366208663 shutdown_connections 2023-10-25T00:00:35.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:35.208+0000 7f318fa8e640 1 -- 172.21.15.107:0/2366208663 wait complete. 2023-10-25T00:00:35.228 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:00:37.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:37.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.372+0000 7f056e32e640 1 Processor -- start 2023-10-25T00:00:37.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.372+0000 7f056e32e640 1 -- start start 2023-10-25T00:00:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.372+0000 7f056e32e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f05680c8640 0x7f056806a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.372+0000 7f056e32e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05681553d0 0x7f056806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.372+0000 7f056e32e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0568152400 con 0x7f0568152020 2023-10-25T00:00:37.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.372+0000 7f056e32e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05681558d0 con 0x7f05680c8640 2023-10-25T00:00:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.372+0000 7f056e32e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05680c6ce0 con 0x7f05681553d0 2023-10-25T00:00:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f055f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f05680c8640 0x7f056806a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f056c8cb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05681553d0 0x7f056806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f055f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f05680c8640 0x7f056806a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:53820/0 (socket says 172.21.15.107:53820) 2023-10-25T00:00:37.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f056c8cb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05681553d0 0x7f056806ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:44796/0 (socket says 172.21.15.107:44796) 2023-10-25T00:00:37.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f055f7fe640 1 -- 172.21.15.107:0/2199753618 learned_addr learned my addr 172.21.15.107:0/2199753618 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:37.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f055f7fe640 1 -- 172.21.15.107:0/2199753618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05681553d0 msgr2=0x7f056806ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:37.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f055f7fe640 1 --2- 172.21.15.107:0/2199753618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05681553d0 0x7f056806ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:37.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f055f7fe640 1 -- 172.21.15.107:0/2199753618 >> v1:172.21.15.110:6789/0 conn(0x7f0568152020 legacy=0x7f05680c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:37.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f055f7fe640 1 -- 172.21.15.107:0/2199753618 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05680c6fc0 con 0x7f05680c8640 2023-10-25T00:00:37.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f055f7fe640 1 --2- 172.21.15.107:0/2199753618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05680c8640 0x7f056806a710 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0550009870 tx=0x7f055001cee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30c32fbaf9d9325f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f05677fe640 1 -- 172.21.15.107:0/2199753618 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05500a6040 con 0x7f05680c8640 2023-10-25T00:00:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.376+0000 7f05677fe640 1 -- 172.21.15.107:0/2199753618 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05500aa600 con 0x7f05680c8640 2023-10-25T00:00:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f05677fe640 1 -- 172.21.15.107:0/2199753618 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05500aa990 con 0x7f05680c8640 2023-10-25T00:00:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 -- 172.21.15.107:0/2199753618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05680c8640 msgr2=0x7f056806a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:37.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 --2- 172.21.15.107:0/2199753618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05680c8640 0x7f056806a710 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0550009870 tx=0x7f055001cee0 comp rx=0 tx=0).stop 2023-10-25T00:00:37.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 -- 172.21.15.107:0/2199753618 shutdown_connections 2023-10-25T00:00:37.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 --2- 172.21.15.107:0/2199753618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05681553d0 0x7f056806ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:37.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 --2- 172.21.15.107:0/2199753618 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05680c8640 0x7f056806a710 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:37.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 -- 172.21.15.107:0/2199753618 >> 172.21.15.107:0/2199753618 conn(0x7f05680abd60 msgr2=0x7f05680ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:37.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 -- 172.21.15.107:0/2199753618 shutdown_connections 2023-10-25T00:00:37.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 -- 172.21.15.107:0/2199753618 wait complete. 2023-10-25T00:00:37.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 Processor -- start 2023-10-25T00:00:37.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 -- start start 2023-10-25T00:00:37.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05680c8640 0x7f056814ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:37.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0568152020 0x7f05681ed550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:37.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05681553d0 0x7f05681eda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:37.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05680aa020 con 0x7f0568152020 2023-10-25T00:00:37.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05680ab2b0 con 0x7f05680c8640 2023-10-25T00:00:37.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056e32e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05680abb30 con 0x7f05681553d0 2023-10-25T00:00:37.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f055f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0568152020 0x7f05681ed550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:37.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f055f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0568152020 0x7f05681ed550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43376/0 (socket says 172.21.15.107:43376) 2023-10-25T00:00:37.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f055f7fe640 1 -- 172.21.15.107:0/1503465192 learned_addr learned my addr 172.21.15.107:0/1503465192 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:37.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f0567fff640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05680c8640 0x7f056814ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:37.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f056c8cb640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05681553d0 0x7f05681eda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:37.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f055f7fe640 1 -- 172.21.15.107:0/1503465192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05681553d0 msgr2=0x7f05681eda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:37.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f055f7fe640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05681553d0 0x7f05681eda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:37.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f055f7fe640 1 -- 172.21.15.107:0/1503465192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05680c8640 msgr2=0x7f056814ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:37.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f055f7fe640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05680c8640 0x7f056814ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:37.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f055f7fe640 1 -- 172.21.15.107:0/1503465192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05500094d0 con 0x7f0568152020 2023-10-25T00:00:37.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f0567fff640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05680c8640 0x7f056814ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:00:37.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.380+0000 7f055f7fe640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0568152020 0x7f05681ed550 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f055001a0b0 tx=0x7f055001acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:37.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.384+0000 7f05657fa640 1 -- 172.21.15.107:0/1503465192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05500b3600 con 0x7f0568152020 2023-10-25T00:00:37.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.384+0000 7f05657fa640 1 -- 172.21.15.107:0/1503465192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05500b3780 con 0x7f0568152020 2023-10-25T00:00:37.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.384+0000 7f056e32e640 1 -- 172.21.15.107:0/1503465192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05680ab570 con 0x7f0568152020 2023-10-25T00:00:37.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.384+0000 7f05657fa640 1 -- 172.21.15.107:0/1503465192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05500b3b30 con 0x7f0568152020 2023-10-25T00:00:37.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.384+0000 7f056e32e640 1 -- 172.21.15.107:0/1503465192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05681ed030 con 0x7f0568152020 2023-10-25T00:00:37.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.384+0000 7f056e32e640 1 -- 172.21.15.107:0/1503465192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05681ed220 con 0x7f0568152020 2023-10-25T00:00:37.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.384+0000 7f05657fa640 1 -- 172.21.15.107:0/1503465192 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f05500b3c90 con 0x7f0568152020 2023-10-25T00:00:37.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.384+0000 7f05657fa640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0548065e60 0x7f0548068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:37.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.384+0000 7f05657fa640 1 -- 172.21.15.107:0/1503465192 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7f055011e3f0 con 0x7f0568152020 2023-10-25T00:00:37.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.384+0000 7f0567fff640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0548065e60 0x7f0548068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:37.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.384+0000 7f0567fff640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0548065e60 0x7f0548068320 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0568143b40 tx=0x7f0558009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:37.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.388+0000 7f05657fa640 1 -- 172.21.15.107:0/1503465192 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f05500eb260 con 0x7f0568152020 2023-10-25T00:00:37.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.604+0000 7f056e32e640 1 -- 172.21.15.107:0/1503465192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05680746e0 con 0x7f0568152020 2023-10-25T00:00:37.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.604+0000 7f05657fa640 1 -- 172.21.15.107:0/1503465192 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f05500ee880 con 0x7f0568152020 2023-10-25T00:00:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:37.607 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:37.608 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 65 2023-10-25T00:00:37.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 -- 172.21.15.107:0/1503465192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0548065e60 msgr2=0x7f0548068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:37.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0548065e60 0x7f0548068320 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0568143b40 tx=0x7f0558009040 comp rx=0 tx=0).stop 2023-10-25T00:00:37.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 -- 172.21.15.107:0/1503465192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0568152020 msgr2=0x7f05681ed550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:37.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0568152020 0x7f05681ed550 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f055001a0b0 tx=0x7f055001acc0 comp rx=0 tx=0).stop 2023-10-25T00:00:37.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 -- 172.21.15.107:0/1503465192 shutdown_connections 2023-10-25T00:00:37.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0548065e60 0x7f0548068320 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:37.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05681553d0 0x7f05681eda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:37.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0568152020 0x7f05681ed550 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:37.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 --2- 172.21.15.107:0/1503465192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05680c8640 0x7f056814ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:37.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 -- 172.21.15.107:0/1503465192 >> 172.21.15.107:0/1503465192 conn(0x7f05680abd60 msgr2=0x7f05680c4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:37.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 -- 172.21.15.107:0/1503465192 shutdown_connections 2023-10-25T00:00:37.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:37.608+0000 7f056e32e640 1 -- 172.21.15.107:0/1503465192 wait complete. 2023-10-25T00:00:37.631 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:00:39.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:39.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.764+0000 7fa7bf3dc640 1 Processor -- start 2023-10-25T00:00:39.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.764+0000 7fa7bf3dc640 1 -- start start 2023-10-25T00:00:39.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.764+0000 7fa7bf3dc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa7b813dc20 0x7fa7b80c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:39.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.764+0000 7fa7bf3dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa7b80720c0 0x7fa7b806df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:39.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.764+0000 7fa7bf3dc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b8154860 con 0x7fa7b806e470 2023-10-25T00:00:39.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.764+0000 7fa7bf3dc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b80c1f10 con 0x7fa7b813dc20 2023-10-25T00:00:39.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.764+0000 7fa7bf3dc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b8154b20 con 0x7fa7b80720c0 2023-10-25T00:00:39.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.764+0000 7fa7bd178640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa7b813dc20 0x7fa7b80c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:39.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.764+0000 7fa7bd979640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa7b806e470 0x7fa7b8157e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55498/0 (socket says 172.21.15.107:55498) 2023-10-25T00:00:39.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.764+0000 7fa7bd979640 1 -- 172.21.15.107:0/920244363 learned_addr learned my addr 172.21.15.107:0/920244363 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:00:39.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bc977640 1 --2- 172.21.15.107:0/920244363 >> v2:172.21.15.110:3300/0 conn(0x7fa7b80720c0 0x7fa7b806df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:39.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bd178640 1 -- 172.21.15.107:0/920244363 >> v2:172.21.15.110:3300/0 conn(0x7fa7b80720c0 msgr2=0x7fa7b806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:39.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bd178640 1 --2- 172.21.15.107:0/920244363 >> v2:172.21.15.110:3300/0 conn(0x7fa7b80720c0 0x7fa7b806df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:39.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bd178640 1 -- 172.21.15.107:0/920244363 >> v1:172.21.15.107:6789/0 conn(0x7fa7b806e470 legacy=0x7fa7b8157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:39.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bd178640 1 -- 172.21.15.107:0/920244363 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b8154e00 con 0x7fa7b813dc20 2023-10-25T00:00:39.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bd178640 1 --2- 172.21.15.107:0/920244363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b813dc20 0x7fa7b80c18a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80c06a0 tx=0x7fa7b401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ddce30e6a20de4 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:39.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7affff640 1 -- 172.21.15.107:0/920244363 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b40ab600 con 0x7fa7b813dc20 2023-10-25T00:00:39.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7affff640 1 -- 172.21.15.107:0/920244363 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b40a7050 con 0x7fa7b813dc20 2023-10-25T00:00:39.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7affff640 1 -- 172.21.15.107:0/920244363 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b40ab980 con 0x7fa7b813dc20 2023-10-25T00:00:39.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/920244363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b813dc20 msgr2=0x7fa7b80c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:39.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bf3dc640 1 --2- 172.21.15.107:0/920244363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b813dc20 0x7fa7b80c18a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80c06a0 tx=0x7fa7b401c340 comp rx=0 tx=0).stop 2023-10-25T00:00:39.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/920244363 shutdown_connections 2023-10-25T00:00:39.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bf3dc640 1 --2- 172.21.15.107:0/920244363 >> v2:172.21.15.110:3300/0 conn(0x7fa7b80720c0 0x7fa7b806df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:39.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bf3dc640 1 --2- 172.21.15.107:0/920244363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b813dc20 0x7fa7b80c18a0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:39.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/920244363 >> 172.21.15.107:0/920244363 conn(0x7fa7b80c39a0 msgr2=0x7fa7b80c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:39.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/920244363 shutdown_connections 2023-10-25T00:00:39.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.768+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/920244363 wait complete. 2023-10-25T00:00:39.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bf3dc640 1 Processor -- start 2023-10-25T00:00:39.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bf3dc640 1 -- start start 2023-10-25T00:00:39.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bf3dc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7b806e470 0x7fa7b80c5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:39.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bf3dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7b80720c0 0x7fa7b8151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:39.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bf3dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b813dc20 0x7fa7b8151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bf3dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b813c5e0 con 0x7fa7b813dc20 2023-10-25T00:00:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bf3dc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b8132b90 con 0x7fa7b806e470 2023-10-25T00:00:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bf3dc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b8132e70 con 0x7fa7b80720c0 2023-10-25T00:00:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bd979640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b813dc20 0x7fa7b8151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:39.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bd979640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b813dc20 0x7fa7b8151ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43404/0 (socket says 172.21.15.107:43404) 2023-10-25T00:00:39.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bd979640 1 -- 172.21.15.107:0/3537276649 learned_addr learned my addr 172.21.15.107:0/3537276649 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:39.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bc977640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7b80720c0 0x7fa7b8151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:39.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bd178640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7b806e470 0x7fa7b80c5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:39.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bd979640 1 -- 172.21.15.107:0/3537276649 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7b80720c0 msgr2=0x7fa7b8151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:39.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bd979640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7b80720c0 0x7fa7b8151960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:39.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bd979640 1 -- 172.21.15.107:0/3537276649 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7b806e470 msgr2=0x7fa7b80c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:39.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bd979640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7b806e470 0x7fa7b80c5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:39.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bd979640 1 -- 172.21.15.107:0/3537276649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b4009420 con 0x7fa7b813dc20 2023-10-25T00:00:39.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bd178640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7b806e470 0x7fa7b80c5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:00:39.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bd979640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b813dc20 0x7fa7b8151ea0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa7b000cbc0 tx=0x7fa7b000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:39.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7adffb640 1 -- 172.21.15.107:0/3537276649 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b000ad40 con 0x7fa7b813dc20 2023-10-25T00:00:39.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7adffb640 1 -- 172.21.15.107:0/3537276649 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b0012400 con 0x7fa7b813dc20 2023-10-25T00:00:39.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7adffb640 1 -- 172.21.15.107:0/3537276649 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b000aea0 con 0x7fa7b813dc20 2023-10-25T00:00:39.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/3537276649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b813c830 con 0x7fa7b813dc20 2023-10-25T00:00:39.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/3537276649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b80c5ed0 con 0x7fa7b813dc20 2023-10-25T00:00:39.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.772+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/3537276649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b80746e0 con 0x7fa7b813dc20 2023-10-25T00:00:39.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.776+0000 7fa7adffb640 1 -- 172.21.15.107:0/3537276649 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa7b0007070 con 0x7fa7b813dc20 2023-10-25T00:00:39.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.776+0000 7fa7adffb640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa79c065e60 0x7fa79c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:39.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.776+0000 7fa7adffb640 1 -- 172.21.15.107:0/3537276649 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7fa7b0083d20 con 0x7fa7b813dc20 2023-10-25T00:00:39.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.776+0000 7fa7bd178640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa79c065e60 0x7fa79c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:39.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.776+0000 7fa7bd178640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa79c065e60 0x7fa79c068320 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40ad000 tx=0x7fa7b4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:39.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:39.780+0000 7fa7adffb640 1 -- 172.21.15.107:0/3537276649 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa7b0050d70 con 0x7fa7b813dc20 2023-10-25T00:00:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.032+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/3537276649 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7b806ab30 con 0x7fa7b813dc20 2023-10-25T00:00:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.032+0000 7fa7adffb640 1 -- 172.21.15.107:0/3537276649 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fa7b0017070 con 0x7fa7b813dc20 2023-10-25T00:00:40.035 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:40.035 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:40.036 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 65 2023-10-25T00:00:40.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/3537276649 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa79c065e60 msgr2=0x7fa79c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa79c065e60 0x7fa79c068320 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40ad000 tx=0x7fa7b4008040 comp rx=0 tx=0).stop 2023-10-25T00:00:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/3537276649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b813dc20 msgr2=0x7fa7b8151ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b813dc20 0x7fa7b8151ea0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa7b000cbc0 tx=0x7fa7b000a430 comp rx=0 tx=0).stop 2023-10-25T00:00:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/3537276649 shutdown_connections 2023-10-25T00:00:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b813dc20 0x7fa7b8151ea0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa79c065e60 0x7fa79c068320 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7b80720c0 0x7fa7b8151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 --2- 172.21.15.107:0/3537276649 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7b806e470 0x7fa7b80c5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/3537276649 >> 172.21.15.107:0/3537276649 conn(0x7fa7b80c39a0 msgr2=0x7fa7b806fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/3537276649 shutdown_connections 2023-10-25T00:00:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:40.036+0000 7fa7bf3dc640 1 -- 172.21.15.107:0/3537276649 wait complete. 2023-10-25T00:00:40.059 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:00:42.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:42.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2ba9f640 1 Processor -- start 2023-10-25T00:00:42.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2ba9f640 1 -- start start 2023-10-25T00:00:42.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2ba9f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee2414eb70 0x7fee240c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:42.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2ba9f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee2406df30 0x7fee2406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:42.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2ba9f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee240c22f0 con 0x7fee24070b00 2023-10-25T00:00:42.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2ba9f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee24071700 con 0x7fee2406df30 2023-10-25T00:00:42.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2ba9f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee240c25d0 con 0x7fee2414eb70 2023-10-25T00:00:42.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2a03c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee2406df30 0x7fee2406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:42.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2a03c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee2406df30 0x7fee2406e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43410/0 (socket says 172.21.15.107:43410) 2023-10-25T00:00:42.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2a03c640 1 -- 172.21.15.107:0/2947928592 learned_addr learned my addr 172.21.15.107:0/2947928592 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:42.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2983b640 1 --2- 172.21.15.107:0/2947928592 >> v2:172.21.15.110:3300/0 conn(0x7fee2414eb70 0x7fee240c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:42.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2a03c640 1 -- 172.21.15.107:0/2947928592 >> v2:172.21.15.110:3300/0 conn(0x7fee2414eb70 msgr2=0x7fee240c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:42.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2a03c640 1 --2- 172.21.15.107:0/2947928592 >> v2:172.21.15.110:3300/0 conn(0x7fee2414eb70 0x7fee240c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:42.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2a03c640 1 -- 172.21.15.107:0/2947928592 >> v1:172.21.15.110:6789/0 conn(0x7fee24070b00 legacy=0x7fee241514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:42.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2a03c640 1 -- 172.21.15.107:0/2947928592 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee240c2be0 con 0x7fee2406df30 2023-10-25T00:00:42.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.200+0000 7fee2983b640 1 --2- 172.21.15.107:0/2947928592 >> v2:172.21.15.110:3300/0 conn(0x7fee2414eb70 0x7fee240c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:00:42.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2a03c640 1 --2- 172.21.15.107:0/2947928592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2406df30 0x7fee2406e3a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fee240c6730 tx=0x7fee2001c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d858a583e655d290 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:42.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee28839640 1 -- 172.21.15.107:0/2947928592 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee200a5870 con 0x7fee2406df30 2023-10-25T00:00:42.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee28839640 1 -- 172.21.15.107:0/2947928592 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee200a59d0 con 0x7fee2406df30 2023-10-25T00:00:42.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee28839640 1 -- 172.21.15.107:0/2947928592 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee200a5d60 con 0x7fee2406df30 2023-10-25T00:00:42.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 -- 172.21.15.107:0/2947928592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2406df30 msgr2=0x7fee2406e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:42.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 --2- 172.21.15.107:0/2947928592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2406df30 0x7fee2406e3a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fee240c6730 tx=0x7fee2001c990 comp rx=0 tx=0).stop 2023-10-25T00:00:42.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 -- 172.21.15.107:0/2947928592 shutdown_connections 2023-10-25T00:00:42.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 --2- 172.21.15.107:0/2947928592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2406df30 0x7fee2406e3a0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 --2- 172.21.15.107:0/2947928592 >> v2:172.21.15.110:3300/0 conn(0x7fee2414eb70 0x7fee240c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 -- 172.21.15.107:0/2947928592 >> 172.21.15.107:0/2947928592 conn(0x7fee240c39a0 msgr2=0x7fee240c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 -- 172.21.15.107:0/2947928592 shutdown_connections 2023-10-25T00:00:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 -- 172.21.15.107:0/2947928592 wait complete. 2023-10-25T00:00:42.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 Processor -- start 2023-10-25T00:00:42.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 -- start start 2023-10-25T00:00:42.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2406df30 0x7fee240aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:42.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee24070b00 0x7fee2414b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:42.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2414eb70 0x7fee241ee160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:42.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee241edd40 con 0x7fee2414eb70 2023-10-25T00:00:42.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee241ed9a0 con 0x7fee2406df30 2023-10-25T00:00:42.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2ba9f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee2406c660 con 0x7fee24070b00 2023-10-25T00:00:42.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2a03c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2414eb70 0x7fee241ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2a03c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2414eb70 0x7fee241ee160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43424/0 (socket says 172.21.15.107:43424) 2023-10-25T00:00:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2a03c640 1 -- 172.21.15.107:0/117896875 learned_addr learned my addr 172.21.15.107:0/117896875 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2983b640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2406df30 0x7fee240aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.204+0000 7fee2903a640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee24070b00 0x7fee2414b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:42.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2a03c640 1 -- 172.21.15.107:0/117896875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee24070b00 msgr2=0x7fee2414b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:42.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2a03c640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee24070b00 0x7fee2414b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:42.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2a03c640 1 -- 172.21.15.107:0/117896875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2406df30 msgr2=0x7fee240aa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:42.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2a03c640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2406df30 0x7fee240aa720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:42.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2a03c640 1 -- 172.21.15.107:0/117896875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20009b30 con 0x7fee2414eb70 2023-10-25T00:00:42.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2983b640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2406df30 0x7fee240aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:00:42.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2a03c640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2414eb70 0x7fee241ee160 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fee2001a5c0 tx=0x7fee20019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:42.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee1a7fc640 1 -- 172.21.15.107:0/117896875 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee200b8670 con 0x7fee2414eb70 2023-10-25T00:00:42.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee1a7fc640 1 -- 172.21.15.107:0/117896875 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee200b87d0 con 0x7fee2414eb70 2023-10-25T00:00:42.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2ba9f640 1 -- 172.21.15.107:0/117896875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2406c940 con 0x7fee2414eb70 2023-10-25T00:00:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee1a7fc640 1 -- 172.21.15.107:0/117896875 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee200b8b80 con 0x7fee2414eb70 2023-10-25T00:00:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2ba9f640 1 -- 172.21.15.107:0/117896875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2406cc20 con 0x7fee2414eb70 2023-10-25T00:00:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2ba9f640 1 -- 172.21.15.107:0/117896875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee241ede80 con 0x7fee2414eb70 2023-10-25T00:00:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee1a7fc640 1 -- 172.21.15.107:0/117896875 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fee200b8ce0 con 0x7fee2414eb70 2023-10-25T00:00:42.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee1a7fc640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee08065e60 0x7fee08068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:42.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee1a7fc640 1 -- 172.21.15.107:0/117896875 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7fee2011f5e0 con 0x7fee2414eb70 2023-10-25T00:00:42.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2983b640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee08065e60 0x7fee08068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:42.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.208+0000 7fee2983b640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee08065e60 0x7fee08068320 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fee1c006fd0 tx=0x7fee1c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:42.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.212+0000 7fee1a7fc640 1 -- 172.21.15.107:0/117896875 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fee241ede80 con 0x7fee2414eb70 2023-10-25T00:00:42.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.432+0000 7fee2ba9f640 1 -- 172.21.15.107:0/117896875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee241ede80 con 0x7fee2414eb70 2023-10-25T00:00:42.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.432+0000 7fee1a7fc640 1 -- 172.21.15.107:0/117896875 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fee241ede80 con 0x7fee2414eb70 2023-10-25T00:00:42.435 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:42.435 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:42.435 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 65 2023-10-25T00:00:42.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 -- 172.21.15.107:0/117896875 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee08065e60 msgr2=0x7fee08068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:42.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee08065e60 0x7fee08068320 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fee1c006fd0 tx=0x7fee1c008040 comp rx=0 tx=0).stop 2023-10-25T00:00:42.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 -- 172.21.15.107:0/117896875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2414eb70 msgr2=0x7fee241ee160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:42.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2414eb70 0x7fee241ee160 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fee2001a5c0 tx=0x7fee20019730 comp rx=0 tx=0).stop 2023-10-25T00:00:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 -- 172.21.15.107:0/117896875 shutdown_connections 2023-10-25T00:00:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee2414eb70 0x7fee241ee160 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee08065e60 0x7fee08068320 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:42.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee24070b00 0x7fee2414b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:42.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 --2- 172.21.15.107:0/117896875 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee2406df30 0x7fee240aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:42.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 -- 172.21.15.107:0/117896875 >> 172.21.15.107:0/117896875 conn(0x7fee240c39a0 msgr2=0x7fee240c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:42.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 -- 172.21.15.107:0/117896875 shutdown_connections 2023-10-25T00:00:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:42.436+0000 7fee2ba9f640 1 -- 172.21.15.107:0/117896875 wait complete. 2023-10-25T00:00:42.455 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:00:44.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa64aab6640 1 Processor -- start 2023-10-25T00:00:44.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa64aab6640 1 -- start start 2023-10-25T00:00:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa64aab6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa6441587a0 0x7fa6440bfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa64aab6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6440a9f50 con 0x7fa644151ec0 2023-10-25T00:00:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa64aab6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6441554d0 con 0x7fa6440ab510 2023-10-25T00:00:44.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa64aab6640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6440c28e0 con 0x7fa6441587a0 2023-10-25T00:00:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa649ab4640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa644151ec0 0x7fa644158090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51354/0 (socket says 172.21.15.107:51354) 2023-10-25T00:00:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa649ab4640 1 -- 172.21.15.107:0/3291142996 learned_addr learned my addr 172.21.15.107:0/3291142996 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:00:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa64a2b5640 1 --2- 172.21.15.107:0/3291142996 >> v2:172.21.15.110:3300/0 conn(0x7fa6441587a0 0x7fa6440bfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498692244 0 0) 0x7fa6440a9f50 con 0x7fa644151ec0 2023-10-25T00:00:44.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa630003530 con 0x7fa644151ec0 2023-10-25T00:00:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547791170 0 0) 0x7fa630003530 con 0x7fa644151ec0 2023-10-25T00:00:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 >> v2:172.21.15.110:3300/0 conn(0x7fa6441587a0 msgr2=0x7fa6440bfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 --2- 172.21.15.107:0/3291142996 >> v2:172.21.15.110:3300/0 conn(0x7fa6441587a0 0x7fa6440bfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:44.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 >> v1:172.21.15.110:6789/0 conn(0x7fa6440ab510 legacy=0x7fa6440bf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64406d8e0 con 0x7fa644151ec0 2023-10-25T00:00:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fa640001680 con 0x7fa644151ec0 2023-10-25T00:00:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 >> v1:172.21.15.107:6789/0 conn(0x7fa644151ec0 legacy=0x7fa644158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:44.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 --2- 172.21.15.107:0/3291142996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa630003980 0x7fa630003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 --2- 172.21.15.107:0/3291142996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6300047d0 0x7fa630004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 --2- 172.21.15.107:0/3291142996 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa630005190 0x7fa63000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63000c100 con 0x7fa6300047d0 2023-10-25T00:00:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63000c270 con 0x7fa630003980 2023-10-25T00:00:44.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63000c430 con 0x7fa630005190 2023-10-25T00:00:44.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa6492b3640 1 --2- 172.21.15.107:0/3291142996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6300047d0 0x7fa630004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:44.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa649ab4640 1 --2- 172.21.15.107:0/3291142996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa630003980 0x7fa630003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:44.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa6492b3640 1 -- 172.21.15.107:0/3291142996 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa630005190 msgr2=0x7fa63000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:44.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa6492b3640 1 --2- 172.21.15.107:0/3291142996 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa630005190 0x7fa63000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:44.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa6492b3640 1 -- 172.21.15.107:0/3291142996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa630003980 msgr2=0x7fa630003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:44.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa6492b3640 1 --2- 172.21.15.107:0/3291142996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa630003980 0x7fa630003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:44.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa64a2b5640 1 -- 172.21.15.107:0/3291142996 reap_dead start 2023-10-25T00:00:44.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa6492b3640 1 -- 172.21.15.107:0/3291142996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa630003530 con 0x7fa6300047d0 2023-10-25T00:00:44.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.592+0000 7fa649ab4640 1 --2- 172.21.15.107:0/3291142996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa630003980 0x7fa630003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:00:44.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.596+0000 7fa6492b3640 1 --2- 172.21.15.107:0/3291142996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6300047d0 0x7fa630004c50 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa634001780 tx=0x7fa63401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f4443555869768 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:44.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.596+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6340052d0 con 0x7fa6300047d0 2023-10-25T00:00:44.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.596+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6340a7050 con 0x7fa6300047d0 2023-10-25T00:00:44.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.596+0000 7fa648ab2640 1 -- 172.21.15.107:0/3291142996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa634003600 con 0x7fa6300047d0 2023-10-25T00:00:44.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.596+0000 7fa64aab6640 1 -- 172.21.15.107:0/3291142996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6300047d0 msgr2=0x7fa630004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:44.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.596+0000 7fa64aab6640 1 --2- 172.21.15.107:0/3291142996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6300047d0 0x7fa630004c50 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fa634001780 tx=0x7fa63401c3e0 comp rx=0 tx=0).stop 2023-10-25T00:00:44.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.596+0000 7fa64aab6640 1 -- 172.21.15.107:0/3291142996 shutdown_connections 2023-10-25T00:00:44.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.596+0000 7fa64aab6640 1 --2- 172.21.15.107:0/3291142996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6300047d0 0x7fa630004c50 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:44.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.596+0000 7fa64aab6640 1 -- 172.21.15.107:0/3291142996 >> 172.21.15.107:0/3291142996 conn(0x7fa6440c64c0 msgr2=0x7fa6440c68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:44.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.596+0000 7fa64aab6640 1 -- 172.21.15.107:0/3291142996 shutdown_connections 2023-10-25T00:00:44.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.596+0000 7fa64aab6640 1 -- 172.21.15.107:0/3291142996 wait complete. 2023-10-25T00:00:44.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64aab6640 1 Processor -- start 2023-10-25T00:00:44.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64aab6640 1 -- start start 2023-10-25T00:00:44.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64aab6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6300047d0 0x7fa6441f0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:44.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64aab6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440c1570 0x7fa64414ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:44.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64aab6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64414c680 0x7fa64406fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:44.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64aab6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6441e4b30 con 0x7fa6440c1570 2023-10-25T00:00:44.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64aab6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6441e4850 con 0x7fa6300047d0 2023-10-25T00:00:44.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64aab6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6441f02c0 con 0x7fa64414c680 2023-10-25T00:00:44.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa6492b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440c1570 0x7fa64414ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:44.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa6492b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440c1570 0x7fa64414ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49400/0 (socket says 172.21.15.107:49400) 2023-10-25T00:00:44.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa6492b3640 1 -- 172.21.15.107:0/4114194997 learned_addr learned my addr 172.21.15.107:0/4114194997 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:44.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa649ab4640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6300047d0 0x7fa6441f0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:44.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64a2b5640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64414c680 0x7fa64406fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:44.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa6492b3640 1 -- 172.21.15.107:0/4114194997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64414c680 msgr2=0x7fa64406fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:44.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa6492b3640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64414c680 0x7fa64406fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:44.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa6492b3640 1 -- 172.21.15.107:0/4114194997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6300047d0 msgr2=0x7fa6441f0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:44.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa6492b3640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6300047d0 0x7fa6441f0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:44.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa6492b3640 1 -- 172.21.15.107:0/4114194997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6441f05a0 con 0x7fa6440c1570 2023-10-25T00:00:44.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa649ab4640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6300047d0 0x7fa6441f0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:00:44.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa6492b3640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440c1570 0x7fa64414ded0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa634000c10 tx=0x7fa630005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:44.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa63a7fc640 1 -- 172.21.15.107:0/4114194997 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6340072f0 con 0x7fa6440c1570 2023-10-25T00:00:44.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa63a7fc640 1 -- 172.21.15.107:0/4114194997 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa634007450 con 0x7fa6440c1570 2023-10-25T00:00:44.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64aab6640 1 -- 172.21.15.107:0/4114194997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6441f0880 con 0x7fa6440c1570 2023-10-25T00:00:44.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa63a7fc640 1 -- 172.21.15.107:0/4114194997 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6340067e0 con 0x7fa6440c1570 2023-10-25T00:00:44.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64aab6640 1 -- 172.21.15.107:0/4114194997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6441e44e0 con 0x7fa6440c1570 2023-10-25T00:00:44.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa64aab6640 1 -- 172.21.15.107:0/4114194997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6441f0a10 con 0x7fa6440c1570 2023-10-25T00:00:44.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.600+0000 7fa63a7fc640 1 -- 172.21.15.107:0/4114194997 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa634006940 con 0x7fa6440c1570 2023-10-25T00:00:44.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.604+0000 7fa63a7fc640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa628065e90 0x7fa628068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:44.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.604+0000 7fa63a7fc640 1 -- 172.21.15.107:0/4114194997 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7fa63411df80 con 0x7fa6440c1570 2023-10-25T00:00:44.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.604+0000 7fa649ab4640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa628065e90 0x7fa628068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:44.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.604+0000 7fa649ab4640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa628065e90 0x7fa628068350 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa640001ec0 tx=0x7fa640009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:44.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.608+0000 7fa63a7fc640 1 -- 172.21.15.107:0/4114194997 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa6340eaf20 con 0x7fa6440c1570 2023-10-25T00:00:44.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.844+0000 7fa64aab6640 1 -- 172.21.15.107:0/4114194997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa64404e050 con 0x7fa6440c1570 2023-10-25T00:00:44.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.844+0000 7fa63a7fc640 1 -- 172.21.15.107:0/4114194997 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fa6340ee540 con 0x7fa6440c1570 2023-10-25T00:00:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:44.846 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:44.847 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 65 2023-10-25T00:00:44.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 -- 172.21.15.107:0/4114194997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa628065e90 msgr2=0x7fa628068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:44.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa628065e90 0x7fa628068350 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa640001ec0 tx=0x7fa640009040 comp rx=0 tx=0).stop 2023-10-25T00:00:44.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 -- 172.21.15.107:0/4114194997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440c1570 msgr2=0x7fa64414ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:44.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440c1570 0x7fa64414ded0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa634000c10 tx=0x7fa630005800 comp rx=0 tx=0).stop 2023-10-25T00:00:44.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 -- 172.21.15.107:0/4114194997 shutdown_connections 2023-10-25T00:00:44.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa628065e90 0x7fa628068350 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:44.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa64414c680 0x7fa64406fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:44.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440c1570 0x7fa64414ded0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:44.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 --2- 172.21.15.107:0/4114194997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6300047d0 0x7fa6441f0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:44.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 -- 172.21.15.107:0/4114194997 >> 172.21.15.107:0/4114194997 conn(0x7fa6440c64c0 msgr2=0x7fa6440c68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:44.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 -- 172.21.15.107:0/4114194997 shutdown_connections 2023-10-25T00:00:44.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:44.848+0000 7fa64aab6640 1 -- 172.21.15.107:0/4114194997 wait complete. 2023-10-25T00:00:44.865 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:00:46.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.004+0000 7fba81434640 1 Processor -- start 2023-10-25T00:00:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.004+0000 7fba81434640 1 -- start start 2023-10-25T00:00:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba81434640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fba7c0720a0 0x7fba7c0ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:47.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba81434640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fba7c158500 0x7fba7c15f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba81434640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba7c0aa150 con 0x7fba7c0c0aa0 2023-10-25T00:00:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba81434640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba7c0a9900 con 0x7fba7c158500 2023-10-25T00:00:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba81434640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba7c0a9700 con 0x7fba7c0720a0 2023-10-25T00:00:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba7affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fba7c0c0aa0 0x7fba7c157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51356/0 (socket says 172.21.15.107:51356) 2023-10-25T00:00:47.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba7affd640 1 -- 172.21.15.107:0/2382311151 learned_addr learned my addr 172.21.15.107:0/2382311151 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:00:47.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba7b7fe640 1 --2- 172.21.15.107:0/2382311151 >> v2:172.21.15.107:3300/0 conn(0x7fba7c158500 0x7fba7c15f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:47.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba7a7fc640 1 --2- 172.21.15.107:0/2382311151 >> v2:172.21.15.110:3300/0 conn(0x7fba7c0720a0 0x7fba7c0ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba7b7fe640 1 -- 172.21.15.107:0/2382311151 >> v2:172.21.15.110:3300/0 conn(0x7fba7c0720a0 msgr2=0x7fba7c0ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba7b7fe640 1 --2- 172.21.15.107:0/2382311151 >> v2:172.21.15.110:3300/0 conn(0x7fba7c0720a0 0x7fba7c0ab5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba7b7fe640 1 -- 172.21.15.107:0/2382311151 >> v1:172.21.15.107:6789/0 conn(0x7fba7c0c0aa0 legacy=0x7fba7c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.008+0000 7fba7b7fe640 1 -- 172.21.15.107:0/2382311151 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c1549a0 con 0x7fba7c158500 2023-10-25T00:00:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7b7fe640 1 --2- 172.21.15.107:0/2382311151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba7c158500 0x7fba7c15f180 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fba7c069a90 tx=0x7fba7401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c635efec65607a05 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba79ffb640 1 -- 172.21.15.107:0/2382311151 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba740b0440 con 0x7fba7c158500 2023-10-25T00:00:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba79ffb640 1 -- 172.21.15.107:0/2382311151 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba740b05a0 con 0x7fba7c158500 2023-10-25T00:00:47.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba79ffb640 1 -- 172.21.15.107:0/2382311151 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba740b0930 con 0x7fba7c158500 2023-10-25T00:00:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 -- 172.21.15.107:0/2382311151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba7c158500 msgr2=0x7fba7c15f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 --2- 172.21.15.107:0/2382311151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba7c158500 0x7fba7c15f180 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fba7c069a90 tx=0x7fba7401f0f0 comp rx=0 tx=0).stop 2023-10-25T00:00:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 -- 172.21.15.107:0/2382311151 shutdown_connections 2023-10-25T00:00:47.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 --2- 172.21.15.107:0/2382311151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba7c158500 0x7fba7c15f180 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:47.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 --2- 172.21.15.107:0/2382311151 >> v2:172.21.15.110:3300/0 conn(0x7fba7c0720a0 0x7fba7c0ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:47.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 -- 172.21.15.107:0/2382311151 >> 172.21.15.107:0/2382311151 conn(0x7fba7c0c4e40 msgr2=0x7fba7c0c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:47.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 -- 172.21.15.107:0/2382311151 shutdown_connections 2023-10-25T00:00:47.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 -- 172.21.15.107:0/2382311151 wait complete. 2023-10-25T00:00:47.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 Processor -- start 2023-10-25T00:00:47.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 -- start start 2023-10-25T00:00:47.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba7c0720a0 0x7fba7c14fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:47.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba7c0c0aa0 0x7fba7c153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:47.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba7c158500 0x7fba7c1ea580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:47.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c1e11f0 con 0x7fba7c0720a0 2023-10-25T00:00:47.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c1e16b0 con 0x7fba7c158500 2023-10-25T00:00:47.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c1e1a90 con 0x7fba7c0c0aa0 2023-10-25T00:00:47.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba7c0720a0 0x7fba7c14fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:47.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba7c0720a0 0x7fba7c14fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49422/0 (socket says 172.21.15.107:49422) 2023-10-25T00:00:47.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7affd640 1 -- 172.21.15.107:0/1140181101 learned_addr learned my addr 172.21.15.107:0/1140181101 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:47.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7b7fe640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba7c158500 0x7fba7c1ea580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:47.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7a7fc640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba7c0c0aa0 0x7fba7c153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:47.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7affd640 1 -- 172.21.15.107:0/1140181101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba7c0c0aa0 msgr2=0x7fba7c153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:47.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7affd640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba7c0c0aa0 0x7fba7c153090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:47.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7affd640 1 -- 172.21.15.107:0/1140181101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba7c158500 msgr2=0x7fba7c1ea580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:47.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7affd640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba7c158500 0x7fba7c1ea580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:47.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7affd640 1 -- 172.21.15.107:0/1140181101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba74009a70 con 0x7fba7c0720a0 2023-10-25T00:00:47.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7a7fc640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba7c0c0aa0 0x7fba7c153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:00:47.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba7affd640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba7c0720a0 0x7fba7c14fa20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00c500 tx=0x7fba6c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:47.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba637fe640 1 -- 172.21.15.107:0/1140181101 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c005cf0 con 0x7fba7c0720a0 2023-10-25T00:00:47.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba637fe640 1 -- 172.21.15.107:0/1140181101 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6c005e50 con 0x7fba7c0720a0 2023-10-25T00:00:47.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 -- 172.21.15.107:0/1140181101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7c1e13d0 con 0x7fba7c0720a0 2023-10-25T00:00:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba637fe640 1 -- 172.21.15.107:0/1140181101 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c0072f0 con 0x7fba7c0720a0 2023-10-25T00:00:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.012+0000 7fba81434640 1 -- 172.21.15.107:0/1140181101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7c14f640 con 0x7fba7c0720a0 2023-10-25T00:00:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.016+0000 7fba81434640 1 -- 172.21.15.107:0/1140181101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7c0746e0 con 0x7fba7c0720a0 2023-10-25T00:00:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.016+0000 7fba637fe640 1 -- 172.21.15.107:0/1140181101 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fba6c007450 con 0x7fba7c0720a0 2023-10-25T00:00:47.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.016+0000 7fba637fe640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba68065e90 0x7fba68068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:47.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.016+0000 7fba637fe640 1 -- 172.21.15.107:0/1140181101 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 7907+0+0 (secure 0 0 0) 0x7fba6c083470 con 0x7fba7c0720a0 2023-10-25T00:00:47.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.016+0000 7fba7a7fc640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba68065e90 0x7fba68068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:47.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.016+0000 7fba7a7fc640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba68065e90 0x7fba68068350 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fba70006fd0 tx=0x7fba70008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:47.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.020+0000 7fba637fe640 1 -- 172.21.15.107:0/1140181101 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fba6c050440 con 0x7fba7c0720a0 2023-10-25T00:00:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.232+0000 7fba81434640 1 -- 172.21.15.107:0/1140181101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba7c071e80 con 0x7fba7c0720a0 2023-10-25T00:00:47.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba637fe640 1 -- 172.21.15.107:0/1140181101 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fba6c053a60 con 0x7fba7c0720a0 2023-10-25T00:00:47.239 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:47.239 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":65}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-24T23:58:56.590011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":51,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6837/1292188015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1292188015},{"type":"v1","addr":"172.21.15.107:6837","nonce":1292188015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 65 2023-10-25T00:00:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 -- 172.21.15.107:0/1140181101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba68065e90 msgr2=0x7fba68068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:47.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba68065e90 0x7fba68068350 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fba70006fd0 tx=0x7fba70008040 comp rx=0 tx=0).stop 2023-10-25T00:00:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 -- 172.21.15.107:0/1140181101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba7c0720a0 msgr2=0x7fba7c14fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba7c0720a0 0x7fba7c14fa20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fba6c00c500 tx=0x7fba6c00c9c0 comp rx=0 tx=0).stop 2023-10-25T00:00:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 -- 172.21.15.107:0/1140181101 shutdown_connections 2023-10-25T00:00:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba7c158500 0x7fba7c1ea580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:47.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba68065e90 0x7fba68068350 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba7c0c0aa0 0x7fba7c153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 --2- 172.21.15.107:0/1140181101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba7c0720a0 0x7fba7c14fa20 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 -- 172.21.15.107:0/1140181101 >> 172.21.15.107:0/1140181101 conn(0x7fba7c0c4e40 msgr2=0x7fba7c0c1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 -- 172.21.15.107:0/1140181101 shutdown_connections 2023-10-25T00:00:47.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:47.236+0000 7fba81434640 1 -- 172.21.15.107:0/1140181101 wait complete. 2023-10-25T00:00:47.259 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:00:49.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:49.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.392+0000 7f81747f2640 1 Processor -- start 2023-10-25T00:00:49.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.392+0000 7f81747f2640 1 -- start start 2023-10-25T00:00:49.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f81747f2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f816c156770 0x7f816c0ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:49.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f81747f2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f816c0be6b0 0x7f816c15d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:49.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f81747f2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c0c1ca0 con 0x7f816c0c70e0 2023-10-25T00:00:49.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f81747f2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f816c0c27e0 con 0x7f816c0be6b0 2023-10-25T00:00:49.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f81747f2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f816c0c10c0 con 0x7f816c156770 2023-10-25T00:00:49.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f8172d8f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f816c0be6b0 0x7f816c15d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:49.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f8171d8d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f816c0c70e0 0x7f816c0bdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51358/0 (socket says 172.21.15.107:51358) 2023-10-25T00:00:49.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f8171d8d640 1 -- 172.21.15.107:0/2730294495 learned_addr learned my addr 172.21.15.107:0/2730294495 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:00:49.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f8172d8f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f816c0be6b0 0x7f816c15d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49436/0 (socket says 172.21.15.107:49436) 2023-10-25T00:00:49.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f817258e640 1 --2- 172.21.15.107:0/2730294495 >> v2:172.21.15.110:3300/0 conn(0x7f816c156770 0x7f816c0ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:49.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f8172d8f640 1 -- 172.21.15.107:0/2730294495 >> v2:172.21.15.110:3300/0 conn(0x7f816c156770 msgr2=0x7f816c0ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:49.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f8172d8f640 1 --2- 172.21.15.107:0/2730294495 >> v2:172.21.15.110:3300/0 conn(0x7f816c156770 0x7f816c0ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f8172d8f640 1 -- 172.21.15.107:0/2730294495 >> v1:172.21.15.107:6789/0 conn(0x7f816c0c70e0 legacy=0x7f816c0bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:49.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f817158c640 1 -- 172.21.15.107:0/2730294495 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503807955 0 0) 0x7f816c0c1ca0 con 0x7f816c0c70e0 2023-10-25T00:00:49.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.396+0000 7f8172d8f640 1 -- 172.21.15.107:0/2730294495 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c0c13a0 con 0x7f816c0be6b0 2023-10-25T00:00:49.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f8172d8f640 1 --2- 172.21.15.107:0/2730294495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f816c0be6b0 0x7f816c15d130 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f816c0ac6f0 tx=0x7f816401c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c82aaf81b42f43b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f817158c640 1 -- 172.21.15.107:0/2730294495 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81640a5840 con 0x7f816c0be6b0 2023-10-25T00:00:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f817158c640 1 -- 172.21.15.107:0/2730294495 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81640a59a0 con 0x7f816c0be6b0 2023-10-25T00:00:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f817158c640 1 -- 172.21.15.107:0/2730294495 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81640a5d30 con 0x7f816c0be6b0 2023-10-25T00:00:49.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f81747f2640 1 -- 172.21.15.107:0/2730294495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f816c0be6b0 msgr2=0x7f816c15d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f81747f2640 1 --2- 172.21.15.107:0/2730294495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f816c0be6b0 0x7f816c15d130 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f816c0ac6f0 tx=0x7f816401c7c0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f81747f2640 1 -- 172.21.15.107:0/2730294495 shutdown_connections 2023-10-25T00:00:49.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f81747f2640 1 --2- 172.21.15.107:0/2730294495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f816c0be6b0 0x7f816c15d130 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f81747f2640 1 --2- 172.21.15.107:0/2730294495 >> v2:172.21.15.110:3300/0 conn(0x7f816c156770 0x7f816c0ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f81747f2640 1 -- 172.21.15.107:0/2730294495 >> 172.21.15.107:0/2730294495 conn(0x7f816c0c65c0 msgr2=0x7f816c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f81747f2640 1 -- 172.21.15.107:0/2730294495 shutdown_connections 2023-10-25T00:00:49.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f81747f2640 1 -- 172.21.15.107:0/2730294495 wait complete. 2023-10-25T00:00:49.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f81747f2640 1 Processor -- start 2023-10-25T00:00:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.400+0000 7f81747f2640 1 -- start start 2023-10-25T00:00:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f81747f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f816c0be6b0 0x7f816c071b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f81747f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f816c0c70e0 0x7f816c0bfe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f81747f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c156770 0x7f816c0aa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:49.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f81747f2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c070040 con 0x7f816c0be6b0 2023-10-25T00:00:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f81747f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c14b3a0 con 0x7f816c156770 2023-10-25T00:00:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f81747f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816c06fc90 con 0x7f816c0c70e0 2023-10-25T00:00:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f817258e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f816c0be6b0 0x7f816c071b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f817258e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f816c0be6b0 0x7f816c071b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49444/0 (socket says 172.21.15.107:49444) 2023-10-25T00:00:49.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f817258e640 1 -- 172.21.15.107:0/2202775598 learned_addr learned my addr 172.21.15.107:0/2202775598 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f8172d8f640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c156770 0x7f816c0aa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f8171d8d640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f816c0c70e0 0x7f816c0bfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f817258e640 1 -- 172.21.15.107:0/2202775598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f816c0c70e0 msgr2=0x7f816c0bfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f817258e640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f816c0c70e0 0x7f816c0bfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f817258e640 1 -- 172.21.15.107:0/2202775598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c156770 msgr2=0x7f816c0aa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f817258e640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c156770 0x7f816c0aa4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f817258e640 1 -- 172.21.15.107:0/2202775598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164009a70 con 0x7f816c0be6b0 2023-10-25T00:00:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f8172d8f640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c156770 0x7f816c0aa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:00:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f817258e640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f816c0be6b0 0x7f816c071b80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f816800d620 tx=0x7f816800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:49.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f8162ffd640 1 -- 172.21.15.107:0/2202775598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816800b620 con 0x7f816c0be6b0 2023-10-25T00:00:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f8162ffd640 1 -- 172.21.15.107:0/2202775598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816800b780 con 0x7f816c0be6b0 2023-10-25T00:00:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f8162ffd640 1 -- 172.21.15.107:0/2202775598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816800bb10 con 0x7f816c0be6b0 2023-10-25T00:00:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f81747f2640 1 -- 172.21.15.107:0/2202775598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816c06f6d0 con 0x7f816c0be6b0 2023-10-25T00:00:49.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f81747f2640 1 -- 172.21.15.107:0/2202775598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f816c14b030 con 0x7f816c0be6b0 2023-10-25T00:00:49.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.404+0000 7f81747f2640 1 -- 172.21.15.107:0/2202775598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816c14ae90 con 0x7f816c0be6b0 2023-10-25T00:00:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.408+0000 7f8162ffd640 1 -- 172.21.15.107:0/2202775598 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f816800bc70 con 0x7f816c0be6b0 2023-10-25T00:00:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.408+0000 7f8162ffd640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8150065e60 0x7f8150068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.408+0000 7f8162ffd640 1 -- 172.21.15.107:0/2202775598 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7f8168084d00 con 0x7f816c0be6b0 2023-10-25T00:00:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.408+0000 7f8171d8d640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8150065e60 0x7f8150068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:49.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.408+0000 7f8171d8d640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8150065e60 0x7f8150068320 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f816c1556c0 tx=0x7f815c007b60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.412+0000 7f8162ffd640 1 -- 172.21.15.107:0/2202775598 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8168051ca0 con 0x7f816c0be6b0 2023-10-25T00:00:49.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.652+0000 7f81747f2640 1 -- 172.21.15.107:0/2202775598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f816c14ae90 con 0x7f816c0be6b0 2023-10-25T00:00:49.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.652+0000 7f8162ffd640 1 -- 172.21.15.107:0/2202775598 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f816c14ae90 con 0x7f816c0be6b0 2023-10-25T00:00:49.654 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:49.654 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:48.480888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:49.655 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 66 2023-10-25T00:00:49.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.652+0000 7f81747f2640 1 -- 172.21.15.107:0/2202775598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8150065e60 msgr2=0x7f8150068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:49.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.652+0000 7f81747f2640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8150065e60 0x7f8150068320 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f816c1556c0 tx=0x7f815c007b60 comp rx=0 tx=0).stop 2023-10-25T00:00:49.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.652+0000 7f81747f2640 1 -- 172.21.15.107:0/2202775598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f816c0be6b0 msgr2=0x7f816c071b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:49.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.652+0000 7f81747f2640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f816c0be6b0 0x7f816c071b80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f816800d620 tx=0x7f816800dae0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.652+0000 7f81747f2640 1 -- 172.21.15.107:0/2202775598 shutdown_connections 2023-10-25T00:00:49.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.652+0000 7f81747f2640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f816c156770 0x7f816c0aa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.652+0000 7f81747f2640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8150065e60 0x7f8150068320 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.656+0000 7f81747f2640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f816c0c70e0 0x7f816c0bfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.656+0000 7f81747f2640 1 --2- 172.21.15.107:0/2202775598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f816c0be6b0 0x7f816c071b80 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.656+0000 7f81747f2640 1 -- 172.21.15.107:0/2202775598 >> 172.21.15.107:0/2202775598 conn(0x7f816c0c65c0 msgr2=0x7f816c0c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:49.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.656+0000 7f81747f2640 1 -- 172.21.15.107:0/2202775598 shutdown_connections 2023-10-25T00:00:49.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.656+0000 7f81747f2640 1 -- 172.21.15.107:0/2202775598 wait complete. 2023-10-25T00:00:49.672 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2023-10-25T00:00:49.673 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:00:49.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:49.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc63139640 1 Processor -- start 2023-10-25T00:00:49.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc63139640 1 -- start start 2023-10-25T00:00:49.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc63139640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcc5c0c8100 0x7fcc5c0abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc63139640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c155680 con 0x7fcc5c152040 2023-10-25T00:00:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc63139640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5c154a70 con 0x7fcc5c06a4b0 2023-10-25T00:00:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc63139640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc5c154d50 con 0x7fcc5c0c8100 2023-10-25T00:00:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc5bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcc5c0c8100 0x7fcc5c0abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc60ed5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcc5c152040 0x7fcc5c0c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51362/0 (socket says 172.21.15.107:51362) 2023-10-25T00:00:49.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc60ed5640 1 -- 172.21.15.107:0/101374484 learned_addr learned my addr 172.21.15.107:0/101374484 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:00:49.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc5bfff640 1 -- 172.21.15.107:0/101374484 >> v1:172.21.15.110:6789/0 conn(0x7fcc5c06a4b0 legacy=0x7fcc5c0ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:49.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc5bfff640 1 -- 172.21.15.107:0/101374484 >> v1:172.21.15.107:6789/0 conn(0x7fcc5c152040 legacy=0x7fcc5c0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:49.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc5bfff640 1 -- 172.21.15.107:0/101374484 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c155030 con 0x7fcc5c0c8100 2023-10-25T00:00:49.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.796+0000 7fcc5b7fe640 1 -- 172.21.15.107:0/101374484 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031686117 0 0) 0x7fcc5c155680 con 0x7fcc5c152040 2023-10-25T00:00:49.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc5bfff640 1 --2- 172.21.15.107:0/101374484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc5c0c8100 0x7fcc5c0abd00 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcc480097c0 tx=0x7fcc4801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e6d79a0d0ed700 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:49.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc5b7fe640 1 -- 172.21.15.107:0/101374484 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc480ab600 con 0x7fcc5c0c8100 2023-10-25T00:00:49.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc5b7fe640 1 -- 172.21.15.107:0/101374484 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc480a7050 con 0x7fcc5c0c8100 2023-10-25T00:00:49.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc5b7fe640 1 -- 172.21.15.107:0/101374484 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc480ab980 con 0x7fcc5c0c8100 2023-10-25T00:00:49.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 -- 172.21.15.107:0/101374484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc5c0c8100 msgr2=0x7fcc5c0abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:49.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 --2- 172.21.15.107:0/101374484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc5c0c8100 0x7fcc5c0abd00 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fcc480097c0 tx=0x7fcc4801c3b0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 -- 172.21.15.107:0/101374484 shutdown_connections 2023-10-25T00:00:49.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 --2- 172.21.15.107:0/101374484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc5c0c8100 0x7fcc5c0abd00 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 -- 172.21.15.107:0/101374484 >> 172.21.15.107:0/101374484 conn(0x7fcc5c0ca200 msgr2=0x7fcc5c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:49.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 -- 172.21.15.107:0/101374484 shutdown_connections 2023-10-25T00:00:49.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 -- 172.21.15.107:0/101374484 wait complete. 2023-10-25T00:00:49.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 Processor -- start 2023-10-25T00:00:49.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 -- start start 2023-10-25T00:00:49.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc5c06a4b0 0x7fcc5c1489d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:49.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc5c0c8100 0x7fcc5c147440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:49.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc5c152040 0x7fcc5c1efde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:49.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0c2230 con 0x7fcc5c06a4b0 2023-10-25T00:00:49.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0c1240 con 0x7fcc5c0c8100 2023-10-25T00:00:49.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc63139640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c0c1770 con 0x7fcc5c152040 2023-10-25T00:00:49.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc60ed5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc5c06a4b0 0x7fcc5c1489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:49.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc60ed5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc5c06a4b0 0x7fcc5c1489d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49456/0 (socket says 172.21.15.107:49456) 2023-10-25T00:00:49.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc60ed5640 1 -- 172.21.15.107:0/2688642898 learned_addr learned my addr 172.21.15.107:0/2688642898 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:49.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc5bfff640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc5c0c8100 0x7fcc5c147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:49.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc616d6640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc5c152040 0x7fcc5c1efde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:49.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc60ed5640 1 -- 172.21.15.107:0/2688642898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc5c152040 msgr2=0x7fcc5c1efde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:49.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc60ed5640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc5c152040 0x7fcc5c1efde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc60ed5640 1 -- 172.21.15.107:0/2688642898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc5c0c8100 msgr2=0x7fcc5c147440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:49.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc60ed5640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc5c0c8100 0x7fcc5c147440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:49.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc60ed5640 1 -- 172.21.15.107:0/2688642898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc48009420 con 0x7fcc5c06a4b0 2023-10-25T00:00:49.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.800+0000 7fcc5bfff640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc5c0c8100 0x7fcc5c147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:00:49.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc60ed5640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc5c06a4b0 0x7fcc5c1489d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fcc500011f0 tx=0x7fcc5000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:49.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc597fa640 1 -- 172.21.15.107:0/2688642898 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc500074e0 con 0x7fcc5c06a4b0 2023-10-25T00:00:49.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc597fa640 1 -- 172.21.15.107:0/2688642898 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc50007640 con 0x7fcc5c06a4b0 2023-10-25T00:00:49.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc597fa640 1 -- 172.21.15.107:0/2688642898 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50004a10 con 0x7fcc5c06a4b0 2023-10-25T00:00:49.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc63139640 1 -- 172.21.15.107:0/2688642898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5c147980 con 0x7fcc5c06a4b0 2023-10-25T00:00:49.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc63139640 1 -- 172.21.15.107:0/2688642898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5c146c40 con 0x7fcc5c06a4b0 2023-10-25T00:00:49.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc63139640 1 -- 172.21.15.107:0/2688642898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5c0746e0 con 0x7fcc5c06a4b0 2023-10-25T00:00:49.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc597fa640 1 -- 172.21.15.107:0/2688642898 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcc50004b70 con 0x7fcc5c06a4b0 2023-10-25T00:00:49.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc597fa640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc40065e60 0x7fcc40068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:49.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc597fa640 1 -- 172.21.15.107:0/2688642898 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7fcc500832f0 con 0x7fcc5c06a4b0 2023-10-25T00:00:49.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc5bfff640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc40065e60 0x7fcc40068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:49.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.804+0000 7fcc5bfff640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc40065e60 0x7fcc40068320 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcc480ad000 tx=0x7fcc48008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:49.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:49.812+0000 7fcc597fa640 1 -- 172.21.15.107:0/2688642898 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fcc500501a0 con 0x7fcc5c06a4b0 2023-10-25T00:00:50.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.060+0000 7fcc63139640 1 -- 172.21.15.107:0/2688642898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc5c1f0320 con 0x7fcc5c06a4b0 2023-10-25T00:00:50.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.060+0000 7fcc597fa640 1 -- 172.21.15.107:0/2688642898 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5017 (secure 0 0 0) 0x7fcc50018070 con 0x7fcc5c06a4b0 2023-10-25T00:00:50.065 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:50.065 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:49.700593+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:50.067 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 67 2023-10-25T00:00:50.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 -- 172.21.15.107:0/2688642898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc40065e60 msgr2=0x7fcc40068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:50.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc40065e60 0x7fcc40068320 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcc480ad000 tx=0x7fcc48008040 comp rx=0 tx=0).stop 2023-10-25T00:00:50.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 -- 172.21.15.107:0/2688642898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc5c06a4b0 msgr2=0x7fcc5c1489d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:50.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc5c06a4b0 0x7fcc5c1489d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fcc500011f0 tx=0x7fcc5000c9b0 comp rx=0 tx=0).stop 2023-10-25T00:00:50.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 -- 172.21.15.107:0/2688642898 shutdown_connections 2023-10-25T00:00:50.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc40065e60 0x7fcc40068320 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:50.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc5c152040 0x7fcc5c1efde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:50.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc5c0c8100 0x7fcc5c147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:50.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 --2- 172.21.15.107:0/2688642898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc5c06a4b0 0x7fcc5c1489d0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:50.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 -- 172.21.15.107:0/2688642898 >> 172.21.15.107:0/2688642898 conn(0x7fcc5c0ca200 msgr2=0x7fcc5c153b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:50.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 -- 172.21.15.107:0/2688642898 shutdown_connections 2023-10-25T00:00:50.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:50.064+0000 7fcc63139640 1 -- 172.21.15.107:0/2688642898 wait complete. 2023-10-25T00:00:50.084 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 67, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 64, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6102": { "addr": "172.21.15.107:6835/1381726420", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 1381726420, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 1381726420, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6102, "incarnation": 61, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-24T23:59:51.614770+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6102 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 67, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5506": { "addr": "172.21.15.110:6834/201284779", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 201284779, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 201284779, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5506, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:00:49.700593+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5506 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 59, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:00:52.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:52.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb73be0640 1 Processor -- start 2023-10-25T00:00:52.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb73be0640 1 -- start start 2023-10-25T00:00:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb73be0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcb6c152020 0x7fcb6c0c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:52.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb73be0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0c8640 0x7fcb6c06a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb73be0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c152400 con 0x7fcb6c1553d0 2023-10-25T00:00:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb73be0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6c1558d0 con 0x7fcb6c152020 2023-10-25T00:00:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb73be0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0c6ce0 con 0x7fcb6c0c8640 2023-10-25T00:00:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb7197c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcb6c152020 0x7fcb6c0c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:52.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb7217d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcb6c1553d0 0x7fcb6c06ac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51370/0 (socket says 172.21.15.107:51370) 2023-10-25T00:00:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb7217d640 1 -- 172.21.15.107:0/2988069622 learned_addr learned my addr 172.21.15.107:0/2988069622 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:00:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb7117b640 1 --2- 172.21.15.107:0/2988069622 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0c8640 0x7fcb6c06a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb7197c640 1 -- 172.21.15.107:0/2988069622 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0c8640 msgr2=0x7fcb6c06a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb7197c640 1 --2- 172.21.15.107:0/2988069622 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0c8640 0x7fcb6c06a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb7197c640 1 -- 172.21.15.107:0/2988069622 >> v1:172.21.15.107:6789/0 conn(0x7fcb6c1553d0 legacy=0x7fcb6c06ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb7197c640 1 -- 172.21.15.107:0/2988069622 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c0c6fc0 con 0x7fcb6c152020 2023-10-25T00:00:52.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.224+0000 7fcb7097a640 1 -- 172.21.15.107:0/2988069622 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317512970 0 0) 0x7fcb6c152400 con 0x7fcb6c1553d0 2023-10-25T00:00:52.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb7197c640 1 --2- 172.21.15.107:0/2988069622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb6c152020 0x7fcb6c0c4bb0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fcb680097c0 tx=0x7fcb6801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=103a74141a4c1c14 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:52.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb7097a640 1 -- 172.21.15.107:0/2988069622 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb680ac640 con 0x7fcb6c152020 2023-10-25T00:00:52.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb7097a640 1 -- 172.21.15.107:0/2988069622 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb680ac7a0 con 0x7fcb6c152020 2023-10-25T00:00:52.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb7097a640 1 -- 172.21.15.107:0/2988069622 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb680acb30 con 0x7fcb6c152020 2023-10-25T00:00:52.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 -- 172.21.15.107:0/2988069622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb6c152020 msgr2=0x7fcb6c0c4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:52.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 --2- 172.21.15.107:0/2988069622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb6c152020 0x7fcb6c0c4bb0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fcb680097c0 tx=0x7fcb6801c3c0 comp rx=0 tx=0).stop 2023-10-25T00:00:52.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 -- 172.21.15.107:0/2988069622 shutdown_connections 2023-10-25T00:00:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 --2- 172.21.15.107:0/2988069622 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0c8640 0x7fcb6c06a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 --2- 172.21.15.107:0/2988069622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb6c152020 0x7fcb6c0c4bb0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 -- 172.21.15.107:0/2988069622 >> 172.21.15.107:0/2988069622 conn(0x7fcb6c0abd60 msgr2=0x7fcb6c0ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 -- 172.21.15.107:0/2988069622 shutdown_connections 2023-10-25T00:00:52.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 -- 172.21.15.107:0/2988069622 wait complete. 2023-10-25T00:00:52.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 Processor -- start 2023-10-25T00:00:52.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 -- start start 2023-10-25T00:00:52.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0c8640 0x7fcb6c148730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:52.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb6c152020 0x7fcb6c0bc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:52.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb6c1553d0 0x7fcb6c0bc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:52.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0aa610 con 0x7fcb6c1553d0 2023-10-25T00:00:52.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0bcee0 con 0x7fcb6c152020 2023-10-25T00:00:52.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb73be0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c1459e0 con 0x7fcb6c0c8640 2023-10-25T00:00:52.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb7217d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb6c1553d0 0x7fcb6c0bc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:52.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb7217d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb6c1553d0 0x7fcb6c0bc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49482/0 (socket says 172.21.15.107:49482) 2023-10-25T00:00:52.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb7217d640 1 -- 172.21.15.107:0/1919907592 learned_addr learned my addr 172.21.15.107:0/1919907592 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:52.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb7197c640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0c8640 0x7fcb6c148730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:52.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.228+0000 7fcb7117b640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb6c152020 0x7fcb6c0bc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:52.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb7217d640 1 -- 172.21.15.107:0/1919907592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0c8640 msgr2=0x7fcb6c148730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:52.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb7217d640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0c8640 0x7fcb6c148730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:52.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb7217d640 1 -- 172.21.15.107:0/1919907592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb6c152020 msgr2=0x7fcb6c0bc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:52.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb7217d640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb6c152020 0x7fcb6c0bc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:52.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb7217d640 1 -- 172.21.15.107:0/1919907592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb68009420 con 0x7fcb6c1553d0 2023-10-25T00:00:52.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb7217d640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb6c1553d0 0x7fcb6c0bc9a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fcb6400cb30 tx=0x7fcb6400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:52.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb627fc640 1 -- 172.21.15.107:0/1919907592 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6400ac40 con 0x7fcb6c1553d0 2023-10-25T00:00:52.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb627fc640 1 -- 172.21.15.107:0/1919907592 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6400adc0 con 0x7fcb6c1553d0 2023-10-25T00:00:52.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb627fc640 1 -- 172.21.15.107:0/1919907592 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb64012640 con 0x7fcb6c1553d0 2023-10-25T00:00:52.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb73be0640 1 -- 172.21.15.107:0/1919907592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c149380 con 0x7fcb6c1553d0 2023-10-25T00:00:52.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb73be0640 1 -- 172.21.15.107:0/1919907592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c0a9c70 con 0x7fcb6c1553d0 2023-10-25T00:00:52.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb73be0640 1 -- 172.21.15.107:0/1919907592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c0aa750 con 0x7fcb6c1553d0 2023-10-25T00:00:52.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb627fc640 1 -- 172.21.15.107:0/1919907592 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcb64007070 con 0x7fcb6c1553d0 2023-10-25T00:00:52.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb627fc640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcb50065e60 0x7fcb50068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:52.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb627fc640 1 -- 172.21.15.107:0/1919907592 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7fcb64083f60 con 0x7fcb6c1553d0 2023-10-25T00:00:52.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb7197c640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcb50065e60 0x7fcb50068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:52.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.232+0000 7fcb7197c640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcb50065e60 0x7fcb50068320 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcb680ae000 tx=0x7fcb68008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:52.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.240+0000 7fcb627fc640 1 -- 172.21.15.107:0/1919907592 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fcb6c0aa750 con 0x7fcb6c1553d0 2023-10-25T00:00:52.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.432+0000 7fcb73be0640 1 -- 172.21.15.107:0/1919907592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb6c0aa750 con 0x7fcb6c1553d0 2023-10-25T00:00:52.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.432+0000 7fcb627fc640 1 -- 172.21.15.107:0/1919907592 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5014 (secure 0 0 0) 0x7fcb6c0aa750 con 0x7fcb6c1553d0 2023-10-25T00:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:52.435 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:52.436 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 69 2023-10-25T00:00:52.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 -- 172.21.15.107:0/1919907592 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcb50065e60 msgr2=0x7fcb50068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcb50065e60 0x7fcb50068320 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcb680ae000 tx=0x7fcb68008040 comp rx=0 tx=0).stop 2023-10-25T00:00:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 -- 172.21.15.107:0/1919907592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb6c1553d0 msgr2=0x7fcb6c0bc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb6c1553d0 0x7fcb6c0bc9a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fcb6400cb30 tx=0x7fcb6400a060 comp rx=0 tx=0).stop 2023-10-25T00:00:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 -- 172.21.15.107:0/1919907592 shutdown_connections 2023-10-25T00:00:52.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb6c1553d0 0x7fcb6c0bc9a0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:52.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb6c152020 0x7fcb6c0bc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:52.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcb50065e60 0x7fcb50068320 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 --2- 172.21.15.107:0/1919907592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb6c0c8640 0x7fcb6c148730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 -- 172.21.15.107:0/1919907592 >> 172.21.15.107:0/1919907592 conn(0x7fcb6c0abd60 msgr2=0x7fcb6c152ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:52.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 -- 172.21.15.107:0/1919907592 shutdown_connections 2023-10-25T00:00:52.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:52.436+0000 7fcb73be0640 1 -- 172.21.15.107:0/1919907592 wait complete. 2023-10-25T00:00:52.456 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=5925 2023-10-25T00:00:52.457 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.a 2023-10-25T00:00:52.501 INFO:tasks.ceph.mds.b.smithi110.stderr:2023-10-25T00:00:52.498+0000 7f3a39f4f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:00:55.458 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2023-10-25T00:00:55.458 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:00:55.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:00:55.462 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:00:55.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:55.541 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:00:55.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49f6838640 1 Processor -- start 2023-10-25T00:00:55.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49f6838640 1 -- start start 2023-10-25T00:00:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49f6838640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f49f014eb70 0x7f49f00c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49f6838640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49f0070b00 0x7f49f01514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:55.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49f6838640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f00c22f0 con 0x7f49f0151a20 2023-10-25T00:00:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49f6838640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f0071700 con 0x7f49f014eb70 2023-10-25T00:00:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49f6838640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f00c25d0 con 0x7f49f0070b00 2023-10-25T00:00:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49f4dd5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f49f0151a20 0x7f49f006df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37344/0 (socket says 172.21.15.107:37344) 2023-10-25T00:00:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49f4dd5640 1 -- 172.21.15.107:0/3911940011 learned_addr learned my addr 172.21.15.107:0/3911940011 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:00:55.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49effff640 1 --2- 172.21.15.107:0/3911940011 >> v2:172.21.15.110:3300/0 conn(0x7f49f014eb70 0x7f49f00c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49ef7fe640 1 --2- 172.21.15.107:0/3911940011 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49f0070b00 0x7f49f01514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49eeffd640 1 -- 172.21.15.107:0/3911940011 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645756910 0 0) 0x7f49f00c22f0 con 0x7f49f0151a20 2023-10-25T00:00:55.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49eeffd640 1 -- 172.21.15.107:0/3911940011 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d8003530 con 0x7f49f0151a20 2023-10-25T00:00:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49ef7fe640 1 -- 172.21.15.107:0/3911940011 >> v2:172.21.15.110:3300/0 conn(0x7f49f014eb70 msgr2=0x7f49f00c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49ef7fe640 1 --2- 172.21.15.107:0/3911940011 >> v2:172.21.15.110:3300/0 conn(0x7f49f014eb70 0x7f49f00c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49ef7fe640 1 -- 172.21.15.107:0/3911940011 >> v1:172.21.15.107:6789/0 conn(0x7f49f0151a20 legacy=0x7f49f006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49eeffd640 1 -- 172.21.15.107:0/3911940011 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1715342341 0 0) 0x7f49d8003530 con 0x7f49f0151a20 2023-10-25T00:00:55.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49ef7fe640 1 -- 172.21.15.107:0/3911940011 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f00c2be0 con 0x7f49f0070b00 2023-10-25T00:00:55.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.600+0000 7f49effff640 1 --2- 172.21.15.107:0/3911940011 >> v2:172.21.15.110:3300/0 conn(0x7f49f014eb70 0x7f49f00c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:00:55.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49ef7fe640 1 --2- 172.21.15.107:0/3911940011 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49f0070b00 0x7f49f01514e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f49e40097c0 tx=0x7f49e401c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22e854ac7a7d6678 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:55.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49eeffd640 1 -- 172.21.15.107:0/3911940011 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8001a00 con 0x7f49f0070b00 2023-10-25T00:00:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49eeffd640 1 -- 172.21.15.107:0/3911940011 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e40a7050 con 0x7f49f0070b00 2023-10-25T00:00:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49eeffd640 1 -- 172.21.15.107:0/3911940011 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e40ab620 con 0x7f49f0070b00 2023-10-25T00:00:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49f6838640 1 -- 172.21.15.107:0/3911940011 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49f0070b00 msgr2=0x7f49f01514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49f6838640 1 --2- 172.21.15.107:0/3911940011 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49f0070b00 0x7f49f01514e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f49e40097c0 tx=0x7f49e401c3a0 comp rx=0 tx=0).stop 2023-10-25T00:00:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49f6838640 1 -- 172.21.15.107:0/3911940011 shutdown_connections 2023-10-25T00:00:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49f6838640 1 --2- 172.21.15.107:0/3911940011 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49f0070b00 0x7f49f01514e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49f6838640 1 --2- 172.21.15.107:0/3911940011 >> v2:172.21.15.110:3300/0 conn(0x7f49f014eb70 0x7f49f00c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49f6838640 1 -- 172.21.15.107:0/3911940011 >> 172.21.15.107:0/3911940011 conn(0x7f49f00c39a0 msgr2=0x7f49f00c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:55.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49f6838640 1 -- 172.21.15.107:0/3911940011 shutdown_connections 2023-10-25T00:00:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49f6838640 1 -- 172.21.15.107:0/3911940011 wait complete. 2023-10-25T00:00:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49f6838640 1 Processor -- start 2023-10-25T00:00:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.604+0000 7f49f6838640 1 -- start start 2023-10-25T00:00:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49f6838640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49f0070b00 0x7f49f00ac090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:55.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49f6838640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49f014eb70 0x7f49f0149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:55.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49f6838640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49f0151a20 0x7f49f00aa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:55.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49f6838640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f014b130 con 0x7f49f0070b00 2023-10-25T00:00:55.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49f6838640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f01e8610 con 0x7f49f014eb70 2023-10-25T00:00:55.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49f6838640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f00a95d0 con 0x7f49f0151a20 2023-10-25T00:00:55.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49f0070b00 0x7f49f00ac090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:55.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49effff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49f0070b00 0x7f49f00ac090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44736/0 (socket says 172.21.15.107:44736) 2023-10-25T00:00:55.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49effff640 1 -- 172.21.15.107:0/3698313161 learned_addr learned my addr 172.21.15.107:0/3698313161 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:55.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49ef7fe640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49f014eb70 0x7f49f0149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:55.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49f4dd5640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49f0151a20 0x7f49f00aa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:55.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49effff640 1 -- 172.21.15.107:0/3698313161 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49f0151a20 msgr2=0x7f49f00aa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:55.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49effff640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49f0151a20 0x7f49f00aa0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:55.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49effff640 1 -- 172.21.15.107:0/3698313161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49f014eb70 msgr2=0x7f49f0149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:55.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49effff640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49f014eb70 0x7f49f0149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:55.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49effff640 1 -- 172.21.15.107:0/3698313161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e000b6c0 con 0x7f49f0070b00 2023-10-25T00:00:55.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49ef7fe640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49f014eb70 0x7f49f0149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:00:55.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49effff640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49f0070b00 0x7f49f00ac090 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f49e0010f10 tx=0x7f49e000e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:55.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49ecff9640 1 -- 172.21.15.107:0/3698313161 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e000ba20 con 0x7f49f0070b00 2023-10-25T00:00:55.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49ecff9640 1 -- 172.21.15.107:0/3698313161 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e0011860 con 0x7f49f0070b00 2023-10-25T00:00:55.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49f6838640 1 -- 172.21.15.107:0/3698313161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e4009420 con 0x7f49f0070b00 2023-10-25T00:00:55.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49ecff9640 1 -- 172.21.15.107:0/3698313161 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e0011bd0 con 0x7f49f0070b00 2023-10-25T00:00:55.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49f6838640 1 -- 172.21.15.107:0/3698313161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f00a9b90 con 0x7f49f0070b00 2023-10-25T00:00:55.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49f6838640 1 -- 172.21.15.107:0/3698313161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f00746e0 con 0x7f49f0070b00 2023-10-25T00:00:55.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49ecff9640 1 -- 172.21.15.107:0/3698313161 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f49e0011d30 con 0x7f49f0070b00 2023-10-25T00:00:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49ecff9640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f49dc065e60 0x7f49dc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49ecff9640 1 -- 172.21.15.107:0/3698313161 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7f49e0086f30 con 0x7f49f0070b00 2023-10-25T00:00:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49ef7fe640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f49dc065e60 0x7f49dc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.608+0000 7f49ef7fe640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f49dc065e60 0x7f49dc068320 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f49e40ad000 tx=0x7f49e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:55.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.612+0000 7f49ecff9640 1 -- 172.21.15.107:0/3698313161 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f49e0053df0 con 0x7f49f0070b00 2023-10-25T00:00:55.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.852+0000 7f49f6838640 1 -- 172.21.15.107:0/3698313161 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49f00c1ed0 con 0x7f49f0070b00 2023-10-25T00:00:55.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.852+0000 7f49ecff9640 1 -- 172.21.15.107:0/3698313161 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f49e001e070 con 0x7f49f0070b00 2023-10-25T00:00:55.853 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:55.854 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:55.854 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 69 2023-10-25T00:00:55.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.852+0000 7f49f6838640 1 -- 172.21.15.107:0/3698313161 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f49dc065e60 msgr2=0x7f49dc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:55.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.852+0000 7f49f6838640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f49dc065e60 0x7f49dc068320 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f49e40ad000 tx=0x7f49e4008040 comp rx=0 tx=0).stop 2023-10-25T00:00:55.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.852+0000 7f49f6838640 1 -- 172.21.15.107:0/3698313161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49f0070b00 msgr2=0x7f49f00ac090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:55.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.852+0000 7f49f6838640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49f0070b00 0x7f49f00ac090 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f49e0010f10 tx=0x7f49e000e730 comp rx=0 tx=0).stop 2023-10-25T00:00:55.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.852+0000 7f49f6838640 1 -- 172.21.15.107:0/3698313161 shutdown_connections 2023-10-25T00:00:55.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.852+0000 7f49f6838640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f49dc065e60 0x7f49dc068320 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:55.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.856+0000 7f49f6838640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49f0151a20 0x7f49f00aa0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:55.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.856+0000 7f49f6838640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49f014eb70 0x7f49f0149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:55.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.856+0000 7f49f6838640 1 --2- 172.21.15.107:0/3698313161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49f0070b00 0x7f49f00ac090 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:55.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.856+0000 7f49f6838640 1 -- 172.21.15.107:0/3698313161 >> 172.21.15.107:0/3698313161 conn(0x7f49f00c39a0 msgr2=0x7f49f00c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:55.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.856+0000 7f49f6838640 1 -- 172.21.15.107:0/3698313161 shutdown_connections 2023-10-25T00:00:55.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:55.856+0000 7f49f6838640 1 -- 172.21.15.107:0/3698313161 wait complete. 2023-10-25T00:00:55.874 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-25T00:00:57.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:00:58.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9711f7e640 1 Processor -- start 2023-10-25T00:00:58.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9711f7e640 1 -- start start 2023-10-25T00:00:58.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9711f7e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c33e0 0x7f970c15ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:58.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9711f7e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c0c22f0 con 0x7f970c070b00 2023-10-25T00:00:58.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9711f7e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c0c25d0 con 0x7f970c14eb70 2023-10-25T00:00:58.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9711f7e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970c071700 con 0x7f970c0c33e0 2023-10-25T00:00:58.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9702ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f970c070b00 0x7f970c06df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37354/0 (socket says 172.21.15.107:37354) 2023-10-25T00:00:58.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9702ffd640 1 -- 172.21.15.107:0/1194350463 learned_addr learned my addr 172.21.15.107:0/1194350463 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:00:58.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970bfff640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c33e0 0x7f970c15ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:58.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580225186 0 0) 0x7f970c0c22f0 con 0x7f970c070b00 2023-10-25T00:00:58.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f8003530 con 0x7f970c070b00 2023-10-25T00:00:58.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1134893710 0 0) 0x7f96f8003530 con 0x7f970c070b00 2023-10-25T00:00:58.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c33e0 msgr2=0x7f970c15ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:58.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c33e0 0x7f970c15ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:58.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 >> v1:172.21.15.110:6789/0 conn(0x7f970c14eb70 legacy=0x7f970c1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:58.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c0c2be0 con 0x7f970c070b00 2023-10-25T00:00:58.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f96f4001680 con 0x7f970c070b00 2023-10-25T00:00:58.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 >> v1:172.21.15.107:6789/0 conn(0x7f970c070b00 legacy=0x7f970c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:58.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c33e0 0x7f96f80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:58.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96f8003f30 0x7f96f80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:58.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96f8004d10 0x7f96f8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:58.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f800ba40 con 0x7f96f8003f30 2023-10-25T00:00:58.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f800ddc0 con 0x7f96f8004d10 2023-10-25T00:00:58.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f800df80 con 0x7f970c0c33e0 2023-10-25T00:00:58.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9702ffd640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96f8003f30 0x7f96f80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:58.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970b7fe640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c33e0 0x7f96f80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:58.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f970bfff640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96f8004d10 0x7f96f8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:58.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9702ffd640 1 -- 172.21.15.107:0/1194350463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c33e0 msgr2=0x7f96f80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:58.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9702ffd640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c33e0 0x7f96f80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:58.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9702ffd640 1 -- 172.21.15.107:0/1194350463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96f8004d10 msgr2=0x7f96f8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:00:58.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9702ffd640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96f8004d10 0x7f96f8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:58.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.008+0000 7f9702ffd640 1 -- 172.21.15.107:0/1194350463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f8003530 con 0x7f96f8003f30 2023-10-25T00:00:58.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f9702ffd640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96f8003f30 0x7f96f80047d0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f96f400c210 tx=0x7f96f401f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf388054b00368e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:58.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4005ae0 con 0x7f96f8003f30 2023-10-25T00:00:58.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f40ab070 con 0x7f96f8003f30 2023-10-25T00:00:58.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f970affd640 1 -- 172.21.15.107:0/1194350463 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4005ea0 con 0x7f96f8003f30 2023-10-25T00:00:58.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f9711f7e640 1 -- 172.21.15.107:0/1194350463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96f8003f30 msgr2=0x7f96f80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:58.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f9711f7e640 1 --2- 172.21.15.107:0/1194350463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96f8003f30 0x7f96f80047d0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f96f400c210 tx=0x7f96f401f370 comp rx=0 tx=0).stop 2023-10-25T00:00:58.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f970bfff640 1 -- 172.21.15.107:0/1194350463 reap_dead start 2023-10-25T00:00:58.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f9711f7e640 1 -- 172.21.15.107:0/1194350463 shutdown_connections 2023-10-25T00:00:58.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f9711f7e640 1 -- 172.21.15.107:0/1194350463 >> 172.21.15.107:0/1194350463 conn(0x7f970c0c39a0 msgr2=0x7f970c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:58.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f9711f7e640 1 -- 172.21.15.107:0/1194350463 shutdown_connections 2023-10-25T00:00:58.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f9711f7e640 1 -- 172.21.15.107:0/1194350463 wait complete. 2023-10-25T00:00:58.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f9711f7e640 1 Processor -- start 2023-10-25T00:00:58.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.012+0000 7f9711f7e640 1 -- start start 2023-10-25T00:00:58.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9711f7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f970c149060 0x7f970c14ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:58.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9711f7e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c1060 0x7f970c06c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:58.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9711f7e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f970c06c560 0x7f970c1eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:58.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9711f7e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c1eb0f0 con 0x7f970c149060 2023-10-25T00:00:58.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9711f7e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c06c9e0 con 0x7f970c06c560 2023-10-25T00:00:58.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9711f7e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970c1489c0 con 0x7f970c0c1060 2023-10-25T00:00:58.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f970b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f970c149060 0x7f970c14ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:58.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f970b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f970c149060 0x7f970c14ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44766/0 (socket says 172.21.15.107:44766) 2023-10-25T00:00:58.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f970b7fe640 1 -- 172.21.15.107:0/256978335 learned_addr learned my addr 172.21.15.107:0/256978335 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:00:58.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9702ffd640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c1060 0x7f970c06c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:58.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f970bfff640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f970c06c560 0x7f970c1eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:58.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f970b7fe640 1 -- 172.21.15.107:0/256978335 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c1060 msgr2=0x7f970c06c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:58.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f970b7fe640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c1060 0x7f970c06c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:58.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f970b7fe640 1 -- 172.21.15.107:0/256978335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f970c06c560 msgr2=0x7f970c1eb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:58.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f970b7fe640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f970c06c560 0x7f970c1eb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:58.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f970b7fe640 1 -- 172.21.15.107:0/256978335 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c1ead50 con 0x7f970c149060 2023-10-25T00:00:58.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f970b7fe640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f970c149060 0x7f970c14ac70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00e550 tx=0x7f96fc00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:58.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9708ff9640 1 -- 172.21.15.107:0/256978335 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc0052e0 con 0x7f970c149060 2023-10-25T00:00:58.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9708ff9640 1 -- 172.21.15.107:0/256978335 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc005440 con 0x7f970c149060 2023-10-25T00:00:58.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9711f7e640 1 -- 172.21.15.107:0/256978335 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970c14b300 con 0x7f970c149060 2023-10-25T00:00:58.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9708ff9640 1 -- 172.21.15.107:0/256978335 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc003820 con 0x7f970c149060 2023-10-25T00:00:58.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9711f7e640 1 -- 172.21.15.107:0/256978335 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970c14b8c0 con 0x7f970c149060 2023-10-25T00:00:58.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9711f7e640 1 -- 172.21.15.107:0/256978335 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970c1eb360 con 0x7f970c149060 2023-10-25T00:00:58.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9708ff9640 1 -- 172.21.15.107:0/256978335 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f96fc003980 con 0x7f970c149060 2023-10-25T00:00:58.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9708ff9640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f96ec065e60 0x7f96ec068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:00:58.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9702ffd640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f96ec065e60 0x7f96ec068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:00:58.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.016+0000 7f9708ff9640 1 -- 172.21.15.107:0/256978335 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7f96fc083e60 con 0x7f970c149060 2023-10-25T00:00:58.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.020+0000 7f9702ffd640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f96ec065e60 0x7f96ec068320 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f96f40af000 tx=0x7f96f40aa000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:00:58.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.024+0000 7f9708ff9640 1 -- 172.21.15.107:0/256978335 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f96fc050c90 con 0x7f970c149060 2023-10-25T00:00:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 -- 172.21.15.107:0/256978335 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f970c0746e0 con 0x7f970c149060 2023-10-25T00:00:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9708ff9640 1 -- 172.21.15.107:0/256978335 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f96fc0542b0 con 0x7f970c149060 2023-10-25T00:00:58.249 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:00:58.249 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:00:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-10-25T00:00:58.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 -- 172.21.15.107:0/256978335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f96ec065e60 msgr2=0x7f96ec068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:58.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f96ec065e60 0x7f96ec068320 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f96f40af000 tx=0x7f96f40aa000 comp rx=0 tx=0).stop 2023-10-25T00:00:58.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 -- 172.21.15.107:0/256978335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f970c149060 msgr2=0x7f970c14ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:00:58.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f970c149060 0x7f970c14ac70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00e550 tx=0x7f96fc00ea10 comp rx=0 tx=0).stop 2023-10-25T00:00:58.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 -- 172.21.15.107:0/256978335 shutdown_connections 2023-10-25T00:00:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f970c06c560 0x7f970c1eb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f96ec065e60 0x7f96ec068320 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f970c0c1060 0x7f970c06c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 --2- 172.21.15.107:0/256978335 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f970c149060 0x7f970c14ac70 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:00:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 -- 172.21.15.107:0/256978335 >> 172.21.15.107:0/256978335 conn(0x7f970c0c39a0 msgr2=0x7f970c06fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:00:58.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 -- 172.21.15.107:0/256978335 shutdown_connections 2023-10-25T00:00:58.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:00:58.248+0000 7f9711f7e640 1 -- 172.21.15.107:0/256978335 wait complete. 2023-10-25T00:00:58.271 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2023-10-25T00:00:58.271 INFO:tasks.mds_thrash.fs.[b]:waiting for 71 secs before thrashing 2023-10-25T00:01:13.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:13.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff0a4a87640 1 Processor -- start 2023-10-25T00:01:13.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff0a4a87640 1 -- start start 2023-10-25T00:01:13.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff0a4a87640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff0a006f490 0x7ff0a014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:13.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff0a4a87640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff0a0152b30 0x7ff0a0152f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:13.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff0a4a87640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a0155cd0 con 0x7ff0a014c470 2023-10-25T00:01:13.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff0a4a87640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a01526e0 con 0x7ff0a0152b30 2023-10-25T00:01:13.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff0a4a87640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a00c4380 con 0x7ff0a006f490 2023-10-25T00:01:13.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff09ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff0a0152b30 0x7ff0a0152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:13.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff09ed9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff0a0152b30 0x7ff0a0152f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43048/0 (socket says 172.21.15.107:43048) 2023-10-25T00:01:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff09ed9d640 1 -- 172.21.15.107:0/1010565151 learned_addr learned my addr 172.21.15.107:0/1010565151 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff09e59c640 1 --1- 172.21.15.107:0/1010565151 >> v1:172.21.15.107:6789/0 conn(0x7ff0a014c470 0x7ff0a0071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33100/0 (socket says 172.21.15.107:33100) 2023-10-25T00:01:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.568+0000 7ff09dd9b640 1 --2- 172.21.15.107:0/1010565151 >> v2:172.21.15.110:3300/0 conn(0x7ff0a006f490 0x7ff0a014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff09ed9d640 1 -- 172.21.15.107:0/1010565151 >> v2:172.21.15.110:3300/0 conn(0x7ff0a006f490 msgr2=0x7ff0a014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:13.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff09ed9d640 1 --2- 172.21.15.107:0/1010565151 >> v2:172.21.15.110:3300/0 conn(0x7ff0a006f490 0x7ff0a014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:13.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff09ed9d640 1 -- 172.21.15.107:0/1010565151 >> v1:172.21.15.107:6789/0 conn(0x7ff0a014c470 legacy=0x7ff0a0071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:13.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff09d59a640 1 -- 172.21.15.107:0/1010565151 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835781599 0 0) 0x7ff0a0155cd0 con 0x7ff0a014c470 2023-10-25T00:01:13.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff09ed9d640 1 -- 172.21.15.107:0/1010565151 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a00c48b0 con 0x7ff0a0152b30 2023-10-25T00:01:13.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff09ed9d640 1 --2- 172.21.15.107:0/1010565151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0a0152b30 0x7ff0a0152f40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff0a006a7e0 tx=0x7ff09801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c3e3b8ca2af792 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:13.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff09d59a640 1 -- 172.21.15.107:0/1010565151 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0980a5660 con 0x7ff0a0152b30 2023-10-25T00:01:13.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff09d59a640 1 -- 172.21.15.107:0/1010565151 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0980a57c0 con 0x7ff0a0152b30 2023-10-25T00:01:13.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff09d59a640 1 -- 172.21.15.107:0/1010565151 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0980a5b50 con 0x7ff0a0152b30 2023-10-25T00:01:13.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1010565151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0a0152b30 msgr2=0x7ff0a0152f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:13.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff0a4a87640 1 --2- 172.21.15.107:0/1010565151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0a0152b30 0x7ff0a0152f40 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7ff0a006a7e0 tx=0x7ff09801c8b0 comp rx=0 tx=0).stop 2023-10-25T00:01:13.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1010565151 shutdown_connections 2023-10-25T00:01:13.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff0a4a87640 1 --2- 172.21.15.107:0/1010565151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0a0152b30 0x7ff0a0152f40 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:13.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff0a4a87640 1 --2- 172.21.15.107:0/1010565151 >> v2:172.21.15.110:3300/0 conn(0x7ff0a006f490 0x7ff0a014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:13.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1010565151 >> 172.21.15.107:0/1010565151 conn(0x7ff0a0070f60 msgr2=0x7ff0a0071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:13.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1010565151 shutdown_connections 2023-10-25T00:01:13.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.572+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1010565151 wait complete. 2023-10-25T00:01:13.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff0a4a87640 1 Processor -- start 2023-10-25T00:01:13.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff0a4a87640 1 -- start start 2023-10-25T00:01:13.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff0a4a87640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0a006f490 0x7ff0a00c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:13.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff0a4a87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0a014c470 0x7ff0a014b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:13.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff0a4a87640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0a0152b30 0x7ff0a01ecd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:13.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff0a4a87640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a014a700 con 0x7ff0a014c470 2023-10-25T00:01:13.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff0a4a87640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a006bff0 con 0x7ff0a006f490 2023-10-25T00:01:13.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff0a4a87640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a01ec820 con 0x7ff0a0152b30 2023-10-25T00:01:13.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0a014c470 0x7ff0a014b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:13.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0a014c470 0x7ff0a014b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43054/0 (socket says 172.21.15.107:43054) 2023-10-25T00:01:13.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09dd9b640 1 -- 172.21.15.107:0/1546144223 learned_addr learned my addr 172.21.15.107:0/1546144223 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:13.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09e59c640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0a006f490 0x7ff0a00c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:13.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09ed9d640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0a0152b30 0x7ff0a01ecd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:13.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09dd9b640 1 -- 172.21.15.107:0/1546144223 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0a0152b30 msgr2=0x7ff0a01ecd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:13.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09dd9b640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0a0152b30 0x7ff0a01ecd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:13.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09dd9b640 1 -- 172.21.15.107:0/1546144223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0a006f490 msgr2=0x7ff0a00c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:13.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09dd9b640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0a006f490 0x7ff0a00c1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:13.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09dd9b640 1 -- 172.21.15.107:0/1546144223 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff098009b30 con 0x7ff0a014c470 2023-10-25T00:01:13.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09e59c640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0a006f490 0x7ff0a00c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:01:13.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff09dd9b640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0a014c470 0x7ff0a014b510 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff09400e930 tx=0x7ff09400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:13.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff08effd640 1 -- 172.21.15.107:0/1546144223 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff094009690 con 0x7ff0a014c470 2023-10-25T00:01:13.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff08effd640 1 -- 172.21.15.107:0/1546144223 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09400ca00 con 0x7ff0a014c470 2023-10-25T00:01:13.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1546144223 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a006c240 con 0x7ff0a014c470 2023-10-25T00:01:13.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff08effd640 1 -- 172.21.15.107:0/1546144223 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09400cd70 con 0x7ff0a014c470 2023-10-25T00:01:13.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1546144223 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a006c800 con 0x7ff0a014c470 2023-10-25T00:01:13.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.576+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1546144223 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a00746e0 con 0x7ff0a014c470 2023-10-25T00:01:13.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.580+0000 7ff08effd640 1 -- 172.21.15.107:0/1546144223 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff094013470 con 0x7ff0a014c470 2023-10-25T00:01:13.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.580+0000 7ff08effd640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff080065e60 0x7ff080068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:13.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.580+0000 7ff08effd640 1 -- 172.21.15.107:0/1546144223 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7ff0940840c0 con 0x7ff0a014c470 2023-10-25T00:01:13.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.580+0000 7ff09e59c640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff080065e60 0x7ff080068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:13.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.580+0000 7ff09e59c640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff080065e60 0x7ff080068320 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff090000f70 tx=0x7ff090007c40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:13.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.584+0000 7ff08effd640 1 -- 172.21.15.107:0/1546144223 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff094051180 con 0x7ff0a014c470 2023-10-25T00:01:13.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.820+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1546144223 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0a01537d0 con 0x7ff0a014c470 2023-10-25T00:01:13.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff08effd640 1 -- 172.21.15.107:0/1546144223 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5774 (secure 0 0 0) 0x7ff094017090 con 0x7ff0a014c470 2023-10-25T00:01:13.824 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:13.824 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:13.825 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-10-25T00:01:13.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1546144223 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff080065e60 msgr2=0x7ff080068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:13.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff080065e60 0x7ff080068320 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff090000f70 tx=0x7ff090007c40 comp rx=0 tx=0).stop 2023-10-25T00:01:13.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1546144223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0a014c470 msgr2=0x7ff0a014b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:13.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0a014c470 0x7ff0a014b510 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ff09400e930 tx=0x7ff09400edf0 comp rx=0 tx=0).stop 2023-10-25T00:01:13.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1546144223 shutdown_connections 2023-10-25T00:01:13.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff080065e60 0x7ff080068320 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:13.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff0a0152b30 0x7ff0a01ecd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:13.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0a014c470 0x7ff0a014b510 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:13.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 --2- 172.21.15.107:0/1546144223 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0a006f490 0x7ff0a00c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:13.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1546144223 >> 172.21.15.107:0/1546144223 conn(0x7ff0a0070f60 msgr2=0x7ff0a00c3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:13.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1546144223 shutdown_connections 2023-10-25T00:01:13.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:13.824+0000 7ff0a4a87640 1 -- 172.21.15.107:0/1546144223 wait complete. 2023-10-25T00:01:13.844 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2023-10-25T00:01:13.845 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:01:13.845 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:01:13.858 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:01:13.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:14.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e8718a640 1 Processor -- start 2023-10-25T00:01:14.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e8718a640 1 -- start start 2023-10-25T00:01:14.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e8718a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e8014c850 0x7f3e80153240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:14.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e8718a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e80155cd0 con 0x7f3e8014c470 2023-10-25T00:01:14.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e8718a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e800c4380 con 0x7f3e8006f490 2023-10-25T00:01:14.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e8718a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8014cc60 con 0x7f3e8014c850 2023-10-25T00:01:14.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e85727640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e8014c850 0x7f3e80153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:14.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e84f26640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3e8014c470 0x7f3e80071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33116/0 (socket says 172.21.15.107:33116) 2023-10-25T00:01:14.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e84f26640 1 -- 172.21.15.107:0/3985530042 learned_addr learned my addr 172.21.15.107:0/3985530042 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:14.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e85727640 1 -- 172.21.15.107:0/3985530042 >> v1:172.21.15.110:6789/0 conn(0x7f3e8006f490 legacy=0x7f3e80152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:14.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e85727640 1 -- 172.21.15.107:0/3985530042 >> v1:172.21.15.107:6789/0 conn(0x7f3e8014c470 legacy=0x7f3e80071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:14.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e85727640 1 -- 172.21.15.107:0/3985530042 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e800c48b0 con 0x7f3e8014c850 2023-10-25T00:01:14.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e85727640 1 --2- 172.21.15.107:0/3985530042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e8014c850 0x7f3e80153240 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009d90 tx=0x7f3e7801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f8782de753060f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:14.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e7f7fe640 1 -- 172.21.15.107:0/3985530042 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e780a66c0 con 0x7f3e8014c850 2023-10-25T00:01:14.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e7f7fe640 1 -- 172.21.15.107:0/3985530042 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e780a6820 con 0x7f3e8014c850 2023-10-25T00:01:14.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.024+0000 7f3e7f7fe640 1 -- 172.21.15.107:0/3985530042 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e780a6bb0 con 0x7f3e8014c850 2023-10-25T00:01:14.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 -- 172.21.15.107:0/3985530042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e8014c850 msgr2=0x7f3e80153240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:14.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 --2- 172.21.15.107:0/3985530042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e8014c850 0x7f3e80153240 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009d90 tx=0x7f3e7801c7f0 comp rx=0 tx=0).stop 2023-10-25T00:01:14.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 -- 172.21.15.107:0/3985530042 shutdown_connections 2023-10-25T00:01:14.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 --2- 172.21.15.107:0/3985530042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e8014c850 0x7f3e80153240 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:14.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 -- 172.21.15.107:0/3985530042 >> 172.21.15.107:0/3985530042 conn(0x7f3e80070f60 msgr2=0x7f3e80071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:14.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 -- 172.21.15.107:0/3985530042 shutdown_connections 2023-10-25T00:01:14.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 -- 172.21.15.107:0/3985530042 wait complete. 2023-10-25T00:01:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 Processor -- start 2023-10-25T00:01:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 -- start start 2023-10-25T00:01:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e8006f490 0x7f3e80141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e8014c470 0x7f3e800c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e8014c850 0x7f3e80134f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e80135dd0 con 0x7f3e8014c470 2023-10-25T00:01:14.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e800c72b0 con 0x7f3e8006f490 2023-10-25T00:01:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e80136570 con 0x7f3e8014c850 2023-10-25T00:01:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e8014c470 0x7f3e800c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e8014c470 0x7f3e800c7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43070/0 (socket says 172.21.15.107:43070) 2023-10-25T00:01:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7ffff640 1 -- 172.21.15.107:0/2511083725 learned_addr learned my addr 172.21.15.107:0/2511083725 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e84f26640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e8006f490 0x7f3e80141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:14.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e85727640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e8014c850 0x7f3e80134f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:14.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7ffff640 1 -- 172.21.15.107:0/2511083725 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e8014c850 msgr2=0x7f3e80134f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:14.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7ffff640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e8014c850 0x7f3e80134f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:14.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7ffff640 1 -- 172.21.15.107:0/2511083725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e8006f490 msgr2=0x7f3e80141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:14.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7ffff640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e8006f490 0x7f3e80141640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:14.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7ffff640 1 -- 172.21.15.107:0/2511083725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e78009a70 con 0x7f3e8014c470 2023-10-25T00:01:14.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7ffff640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e8014c470 0x7f3e800c7720 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00c480 tx=0x7f3e6c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:14.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7d7fa640 1 -- 172.21.15.107:0/2511083725 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c012830 con 0x7f3e8014c470 2023-10-25T00:01:14.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7d7fa640 1 -- 172.21.15.107:0/2511083725 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6c0129b0 con 0x7f3e8014c470 2023-10-25T00:01:14.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 -- 172.21.15.107:0/2511083725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e80135990 con 0x7f3e8014c470 2023-10-25T00:01:14.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e7d7fa640 1 -- 172.21.15.107:0/2511083725 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c012d60 con 0x7f3e8014c470 2023-10-25T00:01:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.028+0000 7f3e8718a640 1 -- 172.21.15.107:0/2511083725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e800c7c60 con 0x7f3e8014c470 2023-10-25T00:01:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.032+0000 7f3e8718a640 1 -- 172.21.15.107:0/2511083725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e80146710 con 0x7f3e8014c470 2023-10-25T00:01:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.032+0000 7f3e7d7fa640 1 -- 172.21.15.107:0/2511083725 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3e6c01c460 con 0x7f3e8014c470 2023-10-25T00:01:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.032+0000 7f3e7d7fa640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e64065e60 0x7f3e64068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.032+0000 7f3e7d7fa640 1 -- 172.21.15.107:0/2511083725 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7f3e6c083df0 con 0x7f3e8014c470 2023-10-25T00:01:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.032+0000 7f3e84f26640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e64065e60 0x7f3e64068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:14.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.032+0000 7f3e84f26640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e64065e60 0x7f3e64068320 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3e74005890 tx=0x7f3e74009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:14.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.036+0000 7f3e7d7fa640 1 -- 172.21.15.107:0/2511083725 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3e6c050d40 con 0x7f3e8014c470 2023-10-25T00:01:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 -- 172.21.15.107:0/2511083725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e800746e0 con 0x7f3e8014c470 2023-10-25T00:01:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e7d7fa640 1 -- 172.21.15.107:0/2511083725 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f3e6c054360 con 0x7f3e8014c470 2023-10-25T00:01:14.229 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:14.229 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:14.230 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-10-25T00:01:14.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 -- 172.21.15.107:0/2511083725 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e64065e60 msgr2=0x7f3e64068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:14.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e64065e60 0x7f3e64068320 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3e74005890 tx=0x7f3e74009040 comp rx=0 tx=0).stop 2023-10-25T00:01:14.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 -- 172.21.15.107:0/2511083725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e8014c470 msgr2=0x7f3e800c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:14.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e8014c470 0x7f3e800c7720 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00c480 tx=0x7f3e6c00c940 comp rx=0 tx=0).stop 2023-10-25T00:01:14.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 -- 172.21.15.107:0/2511083725 shutdown_connections 2023-10-25T00:01:14.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e64065e60 0x7f3e64068320 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:14.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e8014c850 0x7f3e80134f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:14.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e8014c470 0x7f3e800c7720 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:14.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 --2- 172.21.15.107:0/2511083725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e8006f490 0x7f3e80141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:14.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 -- 172.21.15.107:0/2511083725 >> 172.21.15.107:0/2511083725 conn(0x7f3e80070f60 msgr2=0x7f3e800c20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:14.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 -- 172.21.15.107:0/2511083725 shutdown_connections 2023-10-25T00:01:14.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:14.228+0000 7f3e8718a640 1 -- 172.21.15.107:0/2511083725 wait complete. 2023-10-25T00:01:14.248 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:01:16.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:16.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.380+0000 7f6b4ada3640 1 Processor -- start 2023-10-25T00:01:16.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b4ada3640 1 -- start start 2023-10-25T00:01:16.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b4ada3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b44071ce0 0x7f6b440ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:16.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b4ada3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6b440720c0 0x7f6b4413d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:16.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b4ada3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b4406e710 con 0x7f6b44154860 2023-10-25T00:01:16.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b4ada3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4406d900 con 0x7f6b440720c0 2023-10-25T00:01:16.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b4ada3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b44154d20 con 0x7f6b44071ce0 2023-10-25T00:01:16.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b43fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6b440720c0 0x7f6b4413d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:16.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b43fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6b440720c0 0x7f6b4413d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43082/0 (socket says 172.21.15.107:43082) 2023-10-25T00:01:16.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b43fff640 1 -- 172.21.15.107:0/3457384659 learned_addr learned my addr 172.21.15.107:0/3457384659 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:16.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b48b3f640 1 --2- 172.21.15.107:0/3457384659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b44071ce0 0x7f6b440ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:16.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b43fff640 1 -- 172.21.15.107:0/3457384659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b44071ce0 msgr2=0x7f6b440ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:16.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b43fff640 1 --2- 172.21.15.107:0/3457384659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b44071ce0 0x7f6b440ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:16.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b43fff640 1 -- 172.21.15.107:0/3457384659 >> v1:172.21.15.110:6789/0 conn(0x7f6b44154860 legacy=0x7f6b44158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:16.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b43fff640 1 -- 172.21.15.107:0/3457384659 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b44155000 con 0x7f6b440720c0 2023-10-25T00:01:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.384+0000 7f6b43fff640 1 --2- 172.21.15.107:0/3457384659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b440720c0 0x7f6b4413d970 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009870 tx=0x7f6b3001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82284e563ffab80 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b437fe640 1 -- 172.21.15.107:0/3457384659 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b300ab680 con 0x7f6b440720c0 2023-10-25T00:01:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b437fe640 1 -- 172.21.15.107:0/3457384659 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b300ab7e0 con 0x7f6b440720c0 2023-10-25T00:01:16.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b437fe640 1 -- 172.21.15.107:0/3457384659 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b300abb70 con 0x7f6b440720c0 2023-10-25T00:01:16.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 -- 172.21.15.107:0/3457384659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b440720c0 msgr2=0x7f6b4413d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:16.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 --2- 172.21.15.107:0/3457384659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b440720c0 0x7f6b4413d970 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f6b30009870 tx=0x7f6b3001c480 comp rx=0 tx=0).stop 2023-10-25T00:01:16.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 -- 172.21.15.107:0/3457384659 shutdown_connections 2023-10-25T00:01:16.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 --2- 172.21.15.107:0/3457384659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b440720c0 0x7f6b4413d970 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:16.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 --2- 172.21.15.107:0/3457384659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b44071ce0 0x7f6b440ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:16.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 -- 172.21.15.107:0/3457384659 >> 172.21.15.107:0/3457384659 conn(0x7f6b44070f60 msgr2=0x7f6b44071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:16.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 -- 172.21.15.107:0/3457384659 shutdown_connections 2023-10-25T00:01:16.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 -- 172.21.15.107:0/3457384659 wait complete. 2023-10-25T00:01:16.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 Processor -- start 2023-10-25T00:01:16.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 -- start start 2023-10-25T00:01:16.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b44071ce0 0x7f6b441ed790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:16.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b440720c0 0x7f6b4413bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:16.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b44154860 0x7f6b441367f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:16.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b441edcd0 con 0x7f6b44071ce0 2023-10-25T00:01:16.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440c7050 con 0x7f6b440720c0 2023-10-25T00:01:16.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b4ada3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b440c7860 con 0x7f6b44154860 2023-10-25T00:01:16.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b48b3f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b44071ce0 0x7f6b441ed790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:16.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b48b3f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b44071ce0 0x7f6b441ed790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43088/0 (socket says 172.21.15.107:43088) 2023-10-25T00:01:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b48b3f640 1 -- 172.21.15.107:0/2418442250 learned_addr learned my addr 172.21.15.107:0/2418442250 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b48b3f640 1 -- 172.21.15.107:0/2418442250 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b44154860 msgr2=0x7f6b441367f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b48b3f640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b44154860 0x7f6b441367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b48b3f640 1 -- 172.21.15.107:0/2418442250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b440720c0 msgr2=0x7f6b4413bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b48b3f640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b440720c0 0x7f6b4413bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:16.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.388+0000 7f6b48b3f640 1 -- 172.21.15.107:0/2418442250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b300094d0 con 0x7f6b44071ce0 2023-10-25T00:01:16.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b48b3f640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b44071ce0 0x7f6b441ed790 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6b3800d800 tx=0x7f6b3800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:16.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b417fa640 1 -- 172.21.15.107:0/2418442250 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3800b6c0 con 0x7f6b44071ce0 2023-10-25T00:01:16.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b417fa640 1 -- 172.21.15.107:0/2418442250 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3800b820 con 0x7f6b44071ce0 2023-10-25T00:01:16.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b417fa640 1 -- 172.21.15.107:0/2418442250 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3800bb90 con 0x7f6b44071ce0 2023-10-25T00:01:16.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b4ada3640 1 -- 172.21.15.107:0/2418442250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b44139e50 con 0x7f6b44071ce0 2023-10-25T00:01:16.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b4ada3640 1 -- 172.21.15.107:0/2418442250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b440c7580 con 0x7f6b44071ce0 2023-10-25T00:01:16.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b4ada3640 1 -- 172.21.15.107:0/2418442250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b44157a40 con 0x7f6b44071ce0 2023-10-25T00:01:16.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b417fa640 1 -- 172.21.15.107:0/2418442250 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6b3800bcf0 con 0x7f6b44071ce0 2023-10-25T00:01:16.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b417fa640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b28065e60 0x7f6b28068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b417fa640 1 -- 172.21.15.107:0/2418442250 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7f6b38084570 con 0x7f6b44071ce0 2023-10-25T00:01:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b43fff640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b28065e60 0x7f6b28068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.392+0000 7f6b43fff640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b28065e60 0x7f6b28068320 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6b300af000 tx=0x7f6b30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.396+0000 7f6b417fa640 1 -- 172.21.15.107:0/2418442250 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6b38051430 con 0x7f6b44071ce0 2023-10-25T00:01:16.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.584+0000 7f6b4ada3640 1 -- 172.21.15.107:0/2418442250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b440746e0 con 0x7f6b44071ce0 2023-10-25T00:01:16.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b417fa640 1 -- 172.21.15.107:0/2418442250 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f6b38054a50 con 0x7f6b44071ce0 2023-10-25T00:01:16.588 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:16.589 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:16.589 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-10-25T00:01:16.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 -- 172.21.15.107:0/2418442250 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b28065e60 msgr2=0x7f6b28068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:16.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b28065e60 0x7f6b28068320 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6b300af000 tx=0x7f6b30008040 comp rx=0 tx=0).stop 2023-10-25T00:01:16.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 -- 172.21.15.107:0/2418442250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b44071ce0 msgr2=0x7f6b441ed790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:16.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b44071ce0 0x7f6b441ed790 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6b3800d800 tx=0x7f6b3800dcc0 comp rx=0 tx=0).stop 2023-10-25T00:01:16.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 -- 172.21.15.107:0/2418442250 shutdown_connections 2023-10-25T00:01:16.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b28065e60 0x7f6b28068320 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:16.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b44154860 0x7f6b441367f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:16.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b440720c0 0x7f6b4413bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:16.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 --2- 172.21.15.107:0/2418442250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b44071ce0 0x7f6b441ed790 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:16.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 -- 172.21.15.107:0/2418442250 >> 172.21.15.107:0/2418442250 conn(0x7f6b44070f60 msgr2=0x7f6b44073170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:16.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 -- 172.21.15.107:0/2418442250 shutdown_connections 2023-10-25T00:01:16.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:16.588+0000 7f6b4ada3640 1 -- 172.21.15.107:0/2418442250 wait complete. 2023-10-25T00:01:16.609 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:01:18.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:18.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f6019f640 1 Processor -- start 2023-10-25T00:01:18.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f6019f640 1 -- start start 2023-10-25T00:01:18.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f6019f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f58065bc0 0x7f5f58158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:18.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f6019f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5806a5b0 con 0x7f5f58069d40 2023-10-25T00:01:18.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f6019f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5806cea0 con 0x7f5f5806a120 2023-10-25T00:01:18.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f6019f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580ac650 con 0x7f5f58065bc0 2023-10-25T00:01:18.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5df3b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5f58069d40 0x7f5f580a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33120/0 (socket says 172.21.15.107:33120) 2023-10-25T00:01:18.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5df3b640 1 -- 172.21.15.107:0/2606849871 learned_addr learned my addr 172.21.15.107:0/2606849871 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5d73a640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f58065bc0 0x7f5f58158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582249154 0 0) 0x7f5f54001120 con 0x7f5f58069d40 2023-10-25T00:01:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f44003530 con 0x7f5f58069d40 2023-10-25T00:01:18.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3004338480 0 0) 0x7f5f44003530 con 0x7f5f58069d40 2023-10-25T00:01:18.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f58065bc0 msgr2=0x7f5f58158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:18.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f58065bc0 0x7f5f58158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:18.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 >> v1:172.21.15.110:6789/0 conn(0x7f5f5806a120 legacy=0x7f5f580aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f581584e0 con 0x7f5f58069d40 2023-10-25T00:01:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5f54001810 con 0x7f5f58069d40 2023-10-25T00:01:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5d73a640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f58065bc0 0x7f5f58158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:01:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 >> v1:172.21.15.107:6789/0 conn(0x7f5f58069d40 legacy=0x7f5f580a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 0x7f5f44003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f58065bc0 0x7f5f440047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f44004d10 0x7f5f44005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4400ba40 con 0x7f5f440039b0 2023-10-25T00:01:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4400bb80 con 0x7f5f44004d10 2023-10-25T00:01:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4400bd40 con 0x7f5f58065bc0 2023-10-25T00:01:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5df3b640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 0x7f5f44003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5e73c640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f44004d10 0x7f5f44005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5d73a640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f58065bc0 0x7f5f440047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5df3b640 1 -- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f58065bc0 msgr2=0x7f5f440047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5df3b640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f58065bc0 0x7f5f440047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5df3b640 1 -- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f44004d10 msgr2=0x7f5f44005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5df3b640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f44004d10 0x7f5f44005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5df3b640 1 -- 172.21.15.107:0/2606849871 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f48009420 con 0x7f5f440039b0 2023-10-25T00:01:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.740+0000 7f5f5e73c640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f44004d10 0x7f5f44005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:01:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f5df3b640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 0x7f5f44003d90 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400c110 tx=0x7f5f5401cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a969516cdd1f0b79 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f540058e0 con 0x7f5f440039b0 2023-10-25T00:01:18.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f54005a40 con 0x7f5f440039b0 2023-10-25T00:01:18.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f5cf39640 1 -- 172.21.15.107:0/2606849871 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54005df0 con 0x7f5f440039b0 2023-10-25T00:01:18.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f6019f640 1 -- 172.21.15.107:0/2606849871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 msgr2=0x7f5f44003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:18.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f6019f640 1 --2- 172.21.15.107:0/2606849871 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 0x7f5f44003d90 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400c110 tx=0x7f5f5401cba0 comp rx=0 tx=0).stop 2023-10-25T00:01:18.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f5e73c640 1 -- 172.21.15.107:0/2606849871 reap_dead start 2023-10-25T00:01:18.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f6019f640 1 -- 172.21.15.107:0/2606849871 shutdown_connections 2023-10-25T00:01:18.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f6019f640 1 -- 172.21.15.107:0/2606849871 >> 172.21.15.107:0/2606849871 conn(0x7f5f580c65c0 msgr2=0x7f5f580c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:18.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f6019f640 1 -- 172.21.15.107:0/2606849871 shutdown_connections 2023-10-25T00:01:18.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f6019f640 1 -- 172.21.15.107:0/2606849871 wait complete. 2023-10-25T00:01:18.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f6019f640 1 Processor -- start 2023-10-25T00:01:18.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.744+0000 7f5f6019f640 1 -- start start 2023-10-25T00:01:18.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f6019f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 0x7f5f580a9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:18.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f6019f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f580bd690 0x7f5f581f1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:18.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f6019f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f581f16c0 0x7f5f580bdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:18.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f6019f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580c0580 con 0x7f5f440039b0 2023-10-25T00:01:18.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f6019f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580bffc0 con 0x7f5f580bd690 2023-10-25T00:01:18.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f6019f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580c02a0 con 0x7f5f581f16c0 2023-10-25T00:01:18.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5df3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 0x7f5f580a9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:18.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5df3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 0x7f5f580a9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43096/0 (socket says 172.21.15.107:43096) 2023-10-25T00:01:18.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5df3b640 1 -- 172.21.15.107:0/814706479 learned_addr learned my addr 172.21.15.107:0/814706479 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:18.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5d73a640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f580bd690 0x7f5f581f1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:18.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5e73c640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f581f16c0 0x7f5f580bdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:18.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5df3b640 1 -- 172.21.15.107:0/814706479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f581f16c0 msgr2=0x7f5f580bdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:18.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5df3b640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f581f16c0 0x7f5f580bdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:18.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5df3b640 1 -- 172.21.15.107:0/814706479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f580bd690 msgr2=0x7f5f581f1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5df3b640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f580bd690 0x7f5f581f1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5df3b640 1 -- 172.21.15.107:0/814706479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f581f0c60 con 0x7f5f440039b0 2023-10-25T00:01:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5df3b640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 0x7f5f580a9500 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f5f540ab040 tx=0x7f5f44005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f4e7fc640 1 -- 172.21.15.107:0/814706479 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f540017d0 con 0x7f5f440039b0 2023-10-25T00:01:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f4e7fc640 1 -- 172.21.15.107:0/814706479 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f540047f0 con 0x7f5f440039b0 2023-10-25T00:01:18.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f4e7fc640 1 -- 172.21.15.107:0/814706479 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54004b60 con 0x7f5f440039b0 2023-10-25T00:01:18.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f6019f640 1 -- 172.21.15.107:0/814706479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f580a9a40 con 0x7f5f440039b0 2023-10-25T00:01:18.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f6019f640 1 -- 172.21.15.107:0/814706479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5814e5c0 con 0x7f5f440039b0 2023-10-25T00:01:18.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f6019f640 1 -- 172.21.15.107:0/814706479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f581f0df0 con 0x7f5f440039b0 2023-10-25T00:01:18.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f4e7fc640 1 -- 172.21.15.107:0/814706479 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5f54004cc0 con 0x7f5f440039b0 2023-10-25T00:01:18.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f4e7fc640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f3c065e90 0x7f5f3c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:18.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f4e7fc640 1 -- 172.21.15.107:0/814706479 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7f5f5411d110 con 0x7f5f440039b0 2023-10-25T00:01:18.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5d73a640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f3c065e90 0x7f5f3c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:18.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.748+0000 7f5f5d73a640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f3c065e90 0x7f5f3c068350 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5f5815eef0 tx=0x7f5f48008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:18.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.752+0000 7f5f4e7fc640 1 -- 172.21.15.107:0/814706479 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5f540ea110 con 0x7f5f440039b0 2023-10-25T00:01:18.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.944+0000 7f5f6019f640 1 -- 172.21.15.107:0/814706479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f5806a5b0 con 0x7f5f440039b0 2023-10-25T00:01:18.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.944+0000 7f5f4e7fc640 1 -- 172.21.15.107:0/814706479 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f5f5806a5b0 con 0x7f5f440039b0 2023-10-25T00:01:18.945 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:18.946 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:18.946 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-10-25T00:01:18.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 -- 172.21.15.107:0/814706479 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f3c065e90 msgr2=0x7f5f3c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:18.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f3c065e90 0x7f5f3c068350 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5f5815eef0 tx=0x7f5f48008040 comp rx=0 tx=0).stop 2023-10-25T00:01:18.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 -- 172.21.15.107:0/814706479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 msgr2=0x7f5f580a9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:18.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 0x7f5f580a9500 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f5f540ab040 tx=0x7f5f44005800 comp rx=0 tx=0).stop 2023-10-25T00:01:18.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 -- 172.21.15.107:0/814706479 shutdown_connections 2023-10-25T00:01:18.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f3c065e90 0x7f5f3c068350 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:18.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f581f16c0 0x7f5f580bdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:18.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f580bd690 0x7f5f581f1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:18.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 --2- 172.21.15.107:0/814706479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f440039b0 0x7f5f580a9500 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:18.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 -- 172.21.15.107:0/814706479 >> 172.21.15.107:0/814706479 conn(0x7f5f580c65c0 msgr2=0x7f5f580c5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:18.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 -- 172.21.15.107:0/814706479 shutdown_connections 2023-10-25T00:01:18.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:18.948+0000 7f5f6019f640 1 -- 172.21.15.107:0/814706479 wait complete. 2023-10-25T00:01:18.964 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:01:20.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:21.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f616dd3c640 1 Processor -- start 2023-10-25T00:01:21.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f616dd3c640 1 -- start start 2023-10-25T00:01:21.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f616dd3c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f616814fd40 0x7f61680ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:21.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f616dd3c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61680a9820 0x7f61680aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:21.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f616dd3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f616806ee40 0x7f61680c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:21.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f616dd3c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61680aa750 con 0x7f61680a9820 2023-10-25T00:01:21.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f616dd3c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616806f730 con 0x7f616814fd40 2023-10-25T00:01:21.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f616dd3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6168156a40 con 0x7f616806ee40 2023-10-25T00:01:21.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f6166ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61680a9820 0x7f61680aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:21.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f6166ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61680a9820 0x7f61680aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43110/0 (socket says 172.21.15.107:43110) 2023-10-25T00:01:21.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f6166ffd640 1 -- 172.21.15.107:0/3426074171 learned_addr learned my addr 172.21.15.107:0/3426074171 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:21.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f61677fe640 1 --2- 172.21.15.107:0/3426074171 >> v2:172.21.15.110:3300/0 conn(0x7f616814fd40 0x7f61680ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:21.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f6167fff640 1 --2- 172.21.15.107:0/3426074171 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f616806ee40 0x7f61680c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:21.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f6166ffd640 1 -- 172.21.15.107:0/3426074171 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f616806ee40 msgr2=0x7f61680c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:21.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f6166ffd640 1 --2- 172.21.15.107:0/3426074171 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f616806ee40 0x7f61680c1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f6166ffd640 1 -- 172.21.15.107:0/3426074171 >> v2:172.21.15.110:3300/0 conn(0x7f616814fd40 msgr2=0x7f61680ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:21.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f6166ffd640 1 --2- 172.21.15.107:0/3426074171 >> v2:172.21.15.110:3300/0 conn(0x7f616814fd40 0x7f61680ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f6166ffd640 1 -- 172.21.15.107:0/3426074171 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6168156710 con 0x7f61680a9820 2023-10-25T00:01:21.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.100+0000 7f6167fff640 1 --2- 172.21.15.107:0/3426074171 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f616806ee40 0x7f61680c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:01:21.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f6166ffd640 1 --2- 172.21.15.107:0/3426074171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61680a9820 0x7f61680aa210 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f61680c28b0 tx=0x7f615c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c555b82168037771 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:21.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61667fc640 1 -- 172.21.15.107:0/3426074171 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c0ac640 con 0x7f61680a9820 2023-10-25T00:01:21.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61667fc640 1 -- 172.21.15.107:0/3426074171 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c0ac7a0 con 0x7f61680a9820 2023-10-25T00:01:21.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61667fc640 1 -- 172.21.15.107:0/3426074171 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c0acb30 con 0x7f61680a9820 2023-10-25T00:01:21.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 -- 172.21.15.107:0/3426074171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61680a9820 msgr2=0x7f61680aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:21.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 --2- 172.21.15.107:0/3426074171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61680a9820 0x7f61680aa210 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f61680c28b0 tx=0x7f615c01c3b0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 -- 172.21.15.107:0/3426074171 shutdown_connections 2023-10-25T00:01:21.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 --2- 172.21.15.107:0/3426074171 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f616806ee40 0x7f61680c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 --2- 172.21.15.107:0/3426074171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61680a9820 0x7f61680aa210 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 --2- 172.21.15.107:0/3426074171 >> v2:172.21.15.110:3300/0 conn(0x7f616814fd40 0x7f61680ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 -- 172.21.15.107:0/3426074171 >> 172.21.15.107:0/3426074171 conn(0x7f61680c2950 msgr2=0x7f61680c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:21.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 -- 172.21.15.107:0/3426074171 shutdown_connections 2023-10-25T00:01:21.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 -- 172.21.15.107:0/3426074171 wait complete. 2023-10-25T00:01:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 Processor -- start 2023-10-25T00:01:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 -- start start 2023-10-25T00:01:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f616806ee40 0x7f616815b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f616814fd40 0x7f616813b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6168069d40 0x7f61681ed450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61680c4ad0 con 0x7f616806ee40 2023-10-25T00:01:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6168155340 con 0x7f6168069d40 2023-10-25T00:01:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f616dd3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6168155620 con 0x7f616814fd40 2023-10-25T00:01:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f616806ee40 0x7f616815b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f616806ee40 0x7f616815b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43120/0 (socket says 172.21.15.107:43120) 2023-10-25T00:01:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61677fe640 1 -- 172.21.15.107:0/2859000215 learned_addr learned my addr 172.21.15.107:0/2859000215 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f6166ffd640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f616814fd40 0x7f616813b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61677fe640 1 -- 172.21.15.107:0/2859000215 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f616814fd40 msgr2=0x7f616813b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61677fe640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f616814fd40 0x7f616813b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f6167fff640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6168069d40 0x7f61681ed450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61677fe640 1 -- 172.21.15.107:0/2859000215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6168069d40 msgr2=0x7f61681ed450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61677fe640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6168069d40 0x7f61681ed450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61677fe640 1 -- 172.21.15.107:0/2859000215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615c0094d0 con 0x7f616806ee40 2023-10-25T00:01:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f6167fff640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6168069d40 0x7f61681ed450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:01:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f61677fe640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f616806ee40 0x7f616815b5a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f615800d5f0 tx=0x7f615800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:21.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.104+0000 7f614ffff640 1 -- 172.21.15.107:0/2859000215 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61580125f0 con 0x7f616806ee40 2023-10-25T00:01:21.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.108+0000 7f614ffff640 1 -- 172.21.15.107:0/2859000215 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6158012750 con 0x7f616806ee40 2023-10-25T00:01:21.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.108+0000 7f616dd3c640 1 -- 172.21.15.107:0/2859000215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61680c34d0 con 0x7f616806ee40 2023-10-25T00:01:21.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.108+0000 7f614ffff640 1 -- 172.21.15.107:0/2859000215 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6158012ae0 con 0x7f616806ee40 2023-10-25T00:01:21.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.108+0000 7f616dd3c640 1 -- 172.21.15.107:0/2859000215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61680c3a90 con 0x7f616806ee40 2023-10-25T00:01:21.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.108+0000 7f616dd3c640 1 -- 172.21.15.107:0/2859000215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61680c24a0 con 0x7f616806ee40 2023-10-25T00:01:21.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.108+0000 7f614ffff640 1 -- 172.21.15.107:0/2859000215 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6158012c40 con 0x7f616806ee40 2023-10-25T00:01:21.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.108+0000 7f614ffff640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6154065e90 0x7f6154068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:21.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.108+0000 7f6166ffd640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6154065e90 0x7f6154068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:21.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.108+0000 7f614ffff640 1 -- 172.21.15.107:0/2859000215 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7f6158084370 con 0x7f616806ee40 2023-10-25T00:01:21.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.108+0000 7f6166ffd640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6154065e90 0x7f6154068350 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f615c0ae000 tx=0x7f615c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:21.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.112+0000 7f614ffff640 1 -- 172.21.15.107:0/2859000215 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f61580512c0 con 0x7f616806ee40 2023-10-25T00:01:21.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.304+0000 7f616dd3c640 1 -- 172.21.15.107:0/2859000215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6168070840 con 0x7f616806ee40 2023-10-25T00:01:21.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.304+0000 7f614ffff640 1 -- 172.21.15.107:0/2859000215 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f6158007380 con 0x7f616806ee40 2023-10-25T00:01:21.306 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:21.306 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:21.306 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-10-25T00:01:21.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.304+0000 7f616dd3c640 1 -- 172.21.15.107:0/2859000215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6154065e90 msgr2=0x7f6154068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:21.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.304+0000 7f616dd3c640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6154065e90 0x7f6154068350 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f615c0ae000 tx=0x7f615c008040 comp rx=0 tx=0).stop 2023-10-25T00:01:21.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.304+0000 7f616dd3c640 1 -- 172.21.15.107:0/2859000215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f616806ee40 msgr2=0x7f616815b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:21.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.304+0000 7f616dd3c640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f616806ee40 0x7f616815b5a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f615800d5f0 tx=0x7f615800dab0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.308+0000 7f616dd3c640 1 -- 172.21.15.107:0/2859000215 shutdown_connections 2023-10-25T00:01:21.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.308+0000 7f616dd3c640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6168069d40 0x7f61681ed450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.308+0000 7f616dd3c640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6154065e90 0x7f6154068350 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.308+0000 7f616dd3c640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f616814fd40 0x7f616813b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.308+0000 7f616dd3c640 1 --2- 172.21.15.107:0/2859000215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f616806ee40 0x7f616815b5a0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:21.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.308+0000 7f616dd3c640 1 -- 172.21.15.107:0/2859000215 >> 172.21.15.107:0/2859000215 conn(0x7f61680c2950 msgr2=0x7f616814e760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:21.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.308+0000 7f616dd3c640 1 -- 172.21.15.107:0/2859000215 shutdown_connections 2023-10-25T00:01:21.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:21.308+0000 7f616dd3c640 1 -- 172.21.15.107:0/2859000215 wait complete. 2023-10-25T00:01:21.325 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:01:23.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:23.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.456+0000 7f75c8e0c640 1 Processor -- start 2023-10-25T00:01:23.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c8e0c640 1 -- start start 2023-10-25T00:01:23.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c8e0c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f75c4152040 0x7f75c40c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:23.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c8e0c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c40c8100 0x7f75c40abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:23.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c8e0c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c4155680 con 0x7f75c406a4b0 2023-10-25T00:01:23.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c8e0c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c4154a70 con 0x7f75c4152040 2023-10-25T00:01:23.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c8e0c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c4154d50 con 0x7f75c40c8100 2023-10-25T00:01:23.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f75c4152040 0x7f75c40c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:23.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c2d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f75c406a4b0 0x7f75c40ac240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42904/0 (socket says 172.21.15.107:42904) 2023-10-25T00:01:23.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f75c4152040 0x7f75c40c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44420/0 (socket says 172.21.15.107:44420) 2023-10-25T00:01:23.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c2d9d640 1 -- 172.21.15.107:0/1084008089 learned_addr learned my addr 172.21.15.107:0/1084008089 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:23.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c1d9b640 1 --2- 172.21.15.107:0/1084008089 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c40c8100 0x7f75c40abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:23.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c259c640 1 -- 172.21.15.107:0/1084008089 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c40c8100 msgr2=0x7f75c40abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:23.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c259c640 1 --2- 172.21.15.107:0/1084008089 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c40c8100 0x7f75c40abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:23.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c259c640 1 -- 172.21.15.107:0/1084008089 >> v1:172.21.15.107:6789/0 conn(0x7f75c406a4b0 legacy=0x7f75c40ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:23.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c259c640 1 -- 172.21.15.107:0/1084008089 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c4155030 con 0x7f75c4152040 2023-10-25T00:01:23.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c159a640 1 -- 172.21.15.107:0/1084008089 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793281927 0 0) 0x7f75c4155680 con 0x7f75c406a4b0 2023-10-25T00:01:23.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c1d9b640 1 --2- 172.21.15.107:0/1084008089 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c40c8100 0x7f75c40abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:01:23.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c259c640 1 --2- 172.21.15.107:0/1084008089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75c4152040 0x7f75c40c4b80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009d90 tx=0x7f75b401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad77bd886ad070c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:23.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c159a640 1 -- 172.21.15.107:0/1084008089 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b40a5890 con 0x7f75c4152040 2023-10-25T00:01:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c159a640 1 -- 172.21.15.107:0/1084008089 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b40a59f0 con 0x7f75c4152040 2023-10-25T00:01:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.460+0000 7f75c159a640 1 -- 172.21.15.107:0/1084008089 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b40a5d80 con 0x7f75c4152040 2023-10-25T00:01:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 -- 172.21.15.107:0/1084008089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75c4152040 msgr2=0x7f75c40c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 --2- 172.21.15.107:0/1084008089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75c4152040 0x7f75c40c4b80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009d90 tx=0x7f75b401c8c0 comp rx=0 tx=0).stop 2023-10-25T00:01:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 -- 172.21.15.107:0/1084008089 shutdown_connections 2023-10-25T00:01:23.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 --2- 172.21.15.107:0/1084008089 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c40c8100 0x7f75c40abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:23.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 --2- 172.21.15.107:0/1084008089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75c4152040 0x7f75c40c4b80 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:23.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 -- 172.21.15.107:0/1084008089 >> 172.21.15.107:0/1084008089 conn(0x7f75c40ca200 msgr2=0x7f75c40ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:23.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 -- 172.21.15.107:0/1084008089 shutdown_connections 2023-10-25T00:01:23.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 -- 172.21.15.107:0/1084008089 wait complete. 2023-10-25T00:01:23.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 Processor -- start 2023-10-25T00:01:23.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 -- start start 2023-10-25T00:01:23.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c406a4b0 0x7f75c41426f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:23.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75c40c8100 0x7f75c414cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:23.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75c4152040 0x7f75c414a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:23.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c4142c30 con 0x7f75c40c8100 2023-10-25T00:01:23.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c414c3c0 con 0x7f75c4152040 2023-10-25T00:01:23.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c4153fe0 con 0x7f75c406a4b0 2023-10-25T00:01:23.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75c40c8100 0x7f75c414cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:23.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75c40c8100 0x7f75c414cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44422/0 (socket says 172.21.15.107:44422) 2023-10-25T00:01:23.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c1d9b640 1 -- 172.21.15.107:0/3281350862 learned_addr learned my addr 172.21.15.107:0/3281350862 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c259c640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c406a4b0 0x7f75c41426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c2d9d640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75c4152040 0x7f75c414a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c1d9b640 1 -- 172.21.15.107:0/3281350862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c406a4b0 msgr2=0x7f75c41426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c1d9b640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c406a4b0 0x7f75c41426f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c1d9b640 1 -- 172.21.15.107:0/3281350862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75c4152040 msgr2=0x7f75c414a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:23.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c1d9b640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75c4152040 0x7f75c414a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:23.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c1d9b640 1 -- 172.21.15.107:0/3281350862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b4009a70 con 0x7f75c40c8100 2023-10-25T00:01:23.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c1d9b640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75c40c8100 0x7f75c414cfe0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00c820 tx=0x7f75ac00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:23.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75baffd640 1 -- 172.21.15.107:0/3281350862 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac007a30 con 0x7f75c40c8100 2023-10-25T00:01:23.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75baffd640 1 -- 172.21.15.107:0/3281350862 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac007b90 con 0x7f75c40c8100 2023-10-25T00:01:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 -- 172.21.15.107:0/3281350862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c414c620 con 0x7f75c40c8100 2023-10-25T00:01:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75baffd640 1 -- 172.21.15.107:0/3281350862 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00ce10 con 0x7f75c40c8100 2023-10-25T00:01:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.464+0000 7f75c8e0c640 1 -- 172.21.15.107:0/3281350862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c4154250 con 0x7f75c40c8100 2023-10-25T00:01:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.468+0000 7f75c8e0c640 1 -- 172.21.15.107:0/3281350862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c414cb40 con 0x7f75c40c8100 2023-10-25T00:01:23.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.468+0000 7f75baffd640 1 -- 172.21.15.107:0/3281350862 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f75ac0113f0 con 0x7f75c40c8100 2023-10-25T00:01:23.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.468+0000 7f75baffd640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75a4065e60 0x7f75a4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:23.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.468+0000 7f75baffd640 1 -- 172.21.15.107:0/3281350862 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7f75ac0840e0 con 0x7f75c40c8100 2023-10-25T00:01:23.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.468+0000 7f75c259c640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75a4065e60 0x7f75a4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.468+0000 7f75c259c640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75a4065e60 0x7f75a4068320 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f75b40af000 tx=0x7f75b40a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.472+0000 7f75baffd640 1 -- 172.21.15.107:0/3281350862 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f75ac050fa0 con 0x7f75c40c8100 2023-10-25T00:01:23.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.660+0000 7f75c8e0c640 1 -- 172.21.15.107:0/3281350862 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75c40746e0 con 0x7f75c40c8100 2023-10-25T00:01:23.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.660+0000 7f75baffd640 1 -- 172.21.15.107:0/3281350862 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f75ac0545c0 con 0x7f75c40c8100 2023-10-25T00:01:23.662 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:23.662 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:23.662 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-10-25T00:01:23.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 -- 172.21.15.107:0/3281350862 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75a4065e60 msgr2=0x7f75a4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:23.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75a4065e60 0x7f75a4068320 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f75b40af000 tx=0x7f75b40a7070 comp rx=0 tx=0).stop 2023-10-25T00:01:23.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 -- 172.21.15.107:0/3281350862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75c40c8100 msgr2=0x7f75c414cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:23.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75c40c8100 0x7f75c414cfe0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00c820 tx=0x7f75ac00cce0 comp rx=0 tx=0).stop 2023-10-25T00:01:23.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 -- 172.21.15.107:0/3281350862 shutdown_connections 2023-10-25T00:01:23.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75c4152040 0x7f75c414a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:23.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75c40c8100 0x7f75c414cfe0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:23.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75a4065e60 0x7f75a4068320 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:23.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 --2- 172.21.15.107:0/3281350862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75c406a4b0 0x7f75c41426f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:23.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 -- 172.21.15.107:0/3281350862 >> 172.21.15.107:0/3281350862 conn(0x7f75c40ca200 msgr2=0x7f75c40ada00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:23.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 -- 172.21.15.107:0/3281350862 shutdown_connections 2023-10-25T00:01:23.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:23.664+0000 7f75c8e0c640 1 -- 172.21.15.107:0/3281350862 wait complete. 2023-10-25T00:01:23.682 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:01:25.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:25.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.812+0000 7f7112ead640 1 Processor -- start 2023-10-25T00:01:25.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.812+0000 7f7112ead640 1 -- start start 2023-10-25T00:01:25.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7112ead640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f710c14fd40 0x7f710c0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:25.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7112ead640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f710c0a9820 0x7f710c0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:25.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7112ead640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c06f730 con 0x7f710c06ee40 2023-10-25T00:01:25.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7112ead640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f710c0aa750 con 0x7f710c14fd40 2023-10-25T00:01:25.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7112ead640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710c156a40 con 0x7f710c0a9820 2023-10-25T00:01:25.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7110c49640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f710c14fd40 0x7f710c0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:25.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7110c49640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f710c14fd40 0x7f710c0ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44424/0 (socket says 172.21.15.107:44424) 2023-10-25T00:01:25.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7110c49640 1 -- 172.21.15.107:0/592171894 learned_addr learned my addr 172.21.15.107:0/592171894 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:25.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f710bfff640 1 --2- 172.21.15.107:0/592171894 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f710c0a9820 0x7f710c0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:25.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7110c49640 1 -- 172.21.15.107:0/592171894 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f710c0a9820 msgr2=0x7f710c0aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:25.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7110c49640 1 --2- 172.21.15.107:0/592171894 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f710c0a9820 0x7f710c0aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:25.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7110c49640 1 -- 172.21.15.107:0/592171894 >> v1:172.21.15.110:6789/0 conn(0x7f710c06ee40 legacy=0x7f710c0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:25.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7110c49640 1 -- 172.21.15.107:0/592171894 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c156710 con 0x7f710c14fd40 2023-10-25T00:01:25.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f7110c49640 1 --2- 172.21.15.107:0/592171894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c14fd40 0x7f710c0ab090 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009870 tx=0x7f70fc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f358e0ecd7418 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:25.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f710b7fe640 1 -- 172.21.15.107:0/592171894 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc0ad640 con 0x7f710c14fd40 2023-10-25T00:01:25.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f710b7fe640 1 -- 172.21.15.107:0/592171894 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70fc0ad7a0 con 0x7f710c14fd40 2023-10-25T00:01:25.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.816+0000 7f710b7fe640 1 -- 172.21.15.107:0/592171894 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc0adb30 con 0x7f710c14fd40 2023-10-25T00:01:25.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 -- 172.21.15.107:0/592171894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c14fd40 msgr2=0x7f710c0ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:25.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 --2- 172.21.15.107:0/592171894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c14fd40 0x7f710c0ab090 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009870 tx=0x7f70fc01c3f0 comp rx=0 tx=0).stop 2023-10-25T00:01:25.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 -- 172.21.15.107:0/592171894 shutdown_connections 2023-10-25T00:01:25.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 --2- 172.21.15.107:0/592171894 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f710c0a9820 0x7f710c0aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:25.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 --2- 172.21.15.107:0/592171894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c14fd40 0x7f710c0ab090 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:25.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 -- 172.21.15.107:0/592171894 >> 172.21.15.107:0/592171894 conn(0x7f710c0c2950 msgr2=0x7f710c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:25.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 -- 172.21.15.107:0/592171894 shutdown_connections 2023-10-25T00:01:25.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 -- 172.21.15.107:0/592171894 wait complete. 2023-10-25T00:01:25.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 Processor -- start 2023-10-25T00:01:25.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 -- start start 2023-10-25T00:01:25.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f710c06ee40 0x7f710c1ebfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:25.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c14fd40 0x7f710c15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:25.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f710c151410 0x7f710c148490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:25.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c0bc570 con 0x7f710c14fd40 2023-10-25T00:01:25.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c15a630 con 0x7f710c151410 2023-10-25T00:01:25.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710c15a910 con 0x7f710c06ee40 2023-10-25T00:01:25.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f710bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c14fd40 0x7f710c15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:25.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f710bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c14fd40 0x7f710c15b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44426/0 (socket says 172.21.15.107:44426) 2023-10-25T00:01:25.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f710bfff640 1 -- 172.21.15.107:0/10968801 learned_addr learned my addr 172.21.15.107:0/10968801 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:25.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7110c49640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f710c06ee40 0x7f710c1ebfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:25.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f711144a640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f710c151410 0x7f710c148490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:25.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f710bfff640 1 -- 172.21.15.107:0/10968801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f710c06ee40 msgr2=0x7f710c1ebfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f710bfff640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f710c06ee40 0x7f710c1ebfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f710bfff640 1 -- 172.21.15.107:0/10968801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f710c151410 msgr2=0x7f710c148490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f710bfff640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f710c151410 0x7f710c148490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f710bfff640 1 -- 172.21.15.107:0/10968801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc0094d0 con 0x7f710c14fd40 2023-10-25T00:01:25.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f711144a640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f710c151410 0x7f710c148490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:01:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f710bfff640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c14fd40 0x7f710c15b6d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f710000d800 tx=0x7f710000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f71097fa640 1 -- 172.21.15.107:0/10968801 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710000b6c0 con 0x7f710c14fd40 2023-10-25T00:01:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f71097fa640 1 -- 172.21.15.107:0/10968801 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710000b820 con 0x7f710c14fd40 2023-10-25T00:01:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 -- 172.21.15.107:0/10968801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710c15abf0 con 0x7f710c14fd40 2023-10-25T00:01:25.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f71097fa640 1 -- 172.21.15.107:0/10968801 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710000bb90 con 0x7f710c14fd40 2023-10-25T00:01:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.820+0000 7f7112ead640 1 -- 172.21.15.107:0/10968801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710c15b1b0 con 0x7f710c14fd40 2023-10-25T00:01:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.824+0000 7f7112ead640 1 -- 172.21.15.107:0/10968801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710c15b370 con 0x7f710c14fd40 2023-10-25T00:01:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.824+0000 7f71097fa640 1 -- 172.21.15.107:0/10968801 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f710000bcf0 con 0x7f710c14fd40 2023-10-25T00:01:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.824+0000 7f71097fa640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70f8065e60 0x7f70f8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:25.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.824+0000 7f71097fa640 1 -- 172.21.15.107:0/10968801 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7f71000844d0 con 0x7f710c14fd40 2023-10-25T00:01:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.824+0000 7f7110c49640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70f8065e60 0x7f70f8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.824+0000 7f7110c49640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70f8065e60 0x7f70f8068320 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0af000 tx=0x7f70fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:25.828+0000 7f71097fa640 1 -- 172.21.15.107:0/10968801 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7100051420 con 0x7f710c14fd40 2023-10-25T00:01:26.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 -- 172.21.15.107:0/10968801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f710c0c24a0 con 0x7f710c14fd40 2023-10-25T00:01:26.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f71097fa640 1 -- 172.21.15.107:0/10968801 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f7100011340 con 0x7f710c14fd40 2023-10-25T00:01:26.025 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:26.025 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:26.026 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-10-25T00:01:26.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 -- 172.21.15.107:0/10968801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70f8065e60 msgr2=0x7f70f8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:26.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70f8065e60 0x7f70f8068320 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0af000 tx=0x7f70fc008040 comp rx=0 tx=0).stop 2023-10-25T00:01:26.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 -- 172.21.15.107:0/10968801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c14fd40 msgr2=0x7f710c15b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:26.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c14fd40 0x7f710c15b6d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f710000d800 tx=0x7f710000dcc0 comp rx=0 tx=0).stop 2023-10-25T00:01:26.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 -- 172.21.15.107:0/10968801 shutdown_connections 2023-10-25T00:01:26.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f710c151410 0x7f710c148490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:26.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f710c14fd40 0x7f710c15b6d0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70f8065e60 0x7f70f8068320 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 --2- 172.21.15.107:0/10968801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f710c06ee40 0x7f710c1ebfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 -- 172.21.15.107:0/10968801 >> 172.21.15.107:0/10968801 conn(0x7f710c0c2950 msgr2=0x7f710c1501b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 -- 172.21.15.107:0/10968801 shutdown_connections 2023-10-25T00:01:26.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:26.024+0000 7f7112ead640 1 -- 172.21.15.107:0/10968801 wait complete. 2023-10-25T00:01:26.043 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:01:28.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:28.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc093ac5640 1 Processor -- start 2023-10-25T00:01:28.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc093ac5640 1 -- start start 2023-10-25T00:01:28.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc093ac5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c0c8100 0x7fc08c0abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:28.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc093ac5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc08c06a4b0 0x7fc08c0ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:28.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc093ac5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c155680 con 0x7fc08c152040 2023-10-25T00:01:28.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc093ac5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc08c154a70 con 0x7fc08c06a4b0 2023-10-25T00:01:28.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc093ac5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c154d50 con 0x7fc08c0c8100 2023-10-25T00:01:28.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc092062640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc08c06a4b0 0x7fc08c0ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:28.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc091060640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c0c8100 0x7fc08c0abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:28.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc092062640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc08c06a4b0 0x7fc08c0ac240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:43248/0 (socket says 172.21.15.107:43248) 2023-10-25T00:01:28.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc092062640 1 -- 172.21.15.107:0/1797679039 learned_addr learned my addr 172.21.15.107:0/1797679039 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:28.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc091060640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c0c8100 0x7fc08c0abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:49296/0 (socket says 172.21.15.107:49296) 2023-10-25T00:01:28.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc091060640 1 -- 172.21.15.107:0/1797679039 >> v2:172.21.15.110:3300/0 conn(0x7fc08c06a4b0 msgr2=0x7fc08c0ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:28.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc091060640 1 --2- 172.21.15.107:0/1797679039 >> v2:172.21.15.110:3300/0 conn(0x7fc08c06a4b0 0x7fc08c0ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:28.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc091060640 1 -- 172.21.15.107:0/1797679039 >> v1:172.21.15.110:6789/0 conn(0x7fc08c152040 legacy=0x7fc08c0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:28.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.172+0000 7fc091060640 1 -- 172.21.15.107:0/1797679039 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c155030 con 0x7fc08c0c8100 2023-10-25T00:01:28.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc091060640 1 --2- 172.21.15.107:0/1797679039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c0c8100 0x7fc08c0abd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0097c0 tx=0x7fc07c01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78ddf82c862a4696 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc09085f640 1 -- 172.21.15.107:0/1797679039 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c0a4ea0 con 0x7fc08c0c8100 2023-10-25T00:01:28.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc09085f640 1 -- 172.21.15.107:0/1797679039 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07c0a7050 con 0x7fc08c0c8100 2023-10-25T00:01:28.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc09085f640 1 -- 172.21.15.107:0/1797679039 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c0ab820 con 0x7fc08c0c8100 2023-10-25T00:01:28.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 -- 172.21.15.107:0/1797679039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c0c8100 msgr2=0x7fc08c0abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:28.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 --2- 172.21.15.107:0/1797679039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c0c8100 0x7fc08c0abd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0097c0 tx=0x7fc07c01c3f0 comp rx=0 tx=0).stop 2023-10-25T00:01:28.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 -- 172.21.15.107:0/1797679039 shutdown_connections 2023-10-25T00:01:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 --2- 172.21.15.107:0/1797679039 >> v2:172.21.15.110:3300/0 conn(0x7fc08c06a4b0 0x7fc08c0ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 --2- 172.21.15.107:0/1797679039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c0c8100 0x7fc08c0abd00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 -- 172.21.15.107:0/1797679039 >> 172.21.15.107:0/1797679039 conn(0x7fc08c0ca200 msgr2=0x7fc08c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 -- 172.21.15.107:0/1797679039 shutdown_connections 2023-10-25T00:01:28.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 -- 172.21.15.107:0/1797679039 wait complete. 2023-10-25T00:01:28.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 Processor -- start 2023-10-25T00:01:28.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 -- start start 2023-10-25T00:01:28.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c06a4b0 0x7fc08c14cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:28.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc08c0c8100 0x7fc08c147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:28.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc08c152040 0x7fc08c1479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:28.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c14f450 con 0x7fc08c152040 2023-10-25T00:01:28.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c0c0d70 con 0x7fc08c0c8100 2023-10-25T00:01:28.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc093ac5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c150470 con 0x7fc08c06a4b0 2023-10-25T00:01:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc092062640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc08c152040 0x7fc08c1479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc092062640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc08c152040 0x7fc08c1479b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44442/0 (socket says 172.21.15.107:44442) 2023-10-25T00:01:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc092062640 1 -- 172.21.15.107:0/935914449 learned_addr learned my addr 172.21.15.107:0/935914449 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc091861640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c06a4b0 0x7fc08c14cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:28.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.176+0000 7fc091060640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc08c0c8100 0x7fc08c147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc092062640 1 -- 172.21.15.107:0/935914449 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c06a4b0 msgr2=0x7fc08c14cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc092062640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c06a4b0 0x7fc08c14cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc092062640 1 -- 172.21.15.107:0/935914449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc08c0c8100 msgr2=0x7fc08c147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc092062640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc08c0c8100 0x7fc08c147470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:28.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc092062640 1 -- 172.21.15.107:0/935914449 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c009420 con 0x7fc08c152040 2023-10-25T00:01:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc091060640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc08c0c8100 0x7fc08c147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:01:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc092062640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc08c152040 0x7fc08c1479b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc08400efc0 tx=0x7fc08400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc0827fc640 1 -- 172.21.15.107:0/935914449 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084009b60 con 0x7fc08c152040 2023-10-25T00:01:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc0827fc640 1 -- 172.21.15.107:0/935914449 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08400f040 con 0x7fc08c152040 2023-10-25T00:01:28.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc093ac5640 1 -- 172.21.15.107:0/935914449 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08c14f6a0 con 0x7fc08c152040 2023-10-25T00:01:28.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc0827fc640 1 -- 172.21.15.107:0/935914449 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0840039a0 con 0x7fc08c152040 2023-10-25T00:01:28.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc093ac5640 1 -- 172.21.15.107:0/935914449 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08c151480 con 0x7fc08c152040 2023-10-25T00:01:28.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc093ac5640 1 -- 172.21.15.107:0/935914449 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c0af0c0 con 0x7fc08c152040 2023-10-25T00:01:28.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc0827fc640 1 -- 172.21.15.107:0/935914449 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc084003b00 con 0x7fc08c152040 2023-10-25T00:01:28.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc0827fc640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc070065e90 0x7fc070068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:28.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc0827fc640 1 -- 172.21.15.107:0/935914449 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 7993+0+0 (secure 0 0 0) 0x7fc0840843f0 con 0x7fc08c152040 2023-10-25T00:01:28.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc091861640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc070065e90 0x7fc070068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:28.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.180+0000 7fc091861640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc070065e90 0x7fc070068350 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc088001810 tx=0x7fc088009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:28.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.184+0000 7fc0827fc640 1 -- 172.21.15.107:0/935914449 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc084051340 con 0x7fc08c152040 2023-10-25T00:01:28.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.376+0000 7fc093ac5640 1 -- 172.21.15.107:0/935914449 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc08c0c0ee0 con 0x7fc08c152040 2023-10-25T00:01:28.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.376+0000 7fc0827fc640 1 -- 172.21.15.107:0/935914449 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fc084054960 con 0x7fc08c152040 2023-10-25T00:01:28.379 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:28.379 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-24T23:59:51.614770+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6102},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6102":{"gid":6102,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":15,"addr":"172.21.15.107:6835/1381726420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1381726420},{"type":"v1","addr":"172.21.15.107:6835","nonce":1381726420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:28.380 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 70 2023-10-25T00:01:28.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 -- 172.21.15.107:0/935914449 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc070065e90 msgr2=0x7fc070068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:28.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc070065e90 0x7fc070068350 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc088001810 tx=0x7fc088009040 comp rx=0 tx=0).stop 2023-10-25T00:01:28.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 -- 172.21.15.107:0/935914449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc08c152040 msgr2=0x7fc08c1479b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:28.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc08c152040 0x7fc08c1479b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc08400efc0 tx=0x7fc08400c490 comp rx=0 tx=0).stop 2023-10-25T00:01:28.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 -- 172.21.15.107:0/935914449 shutdown_connections 2023-10-25T00:01:28.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc08c152040 0x7fc08c1479b0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:28.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc08c0c8100 0x7fc08c147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:28.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc070065e90 0x7fc070068350 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:28.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 --2- 172.21.15.107:0/935914449 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc08c06a4b0 0x7fc08c14cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:28.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 -- 172.21.15.107:0/935914449 >> 172.21.15.107:0/935914449 conn(0x7fc08c0ca200 msgr2=0x7fc08c0ad070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:28.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 -- 172.21.15.107:0/935914449 shutdown_connections 2023-10-25T00:01:28.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:28.380+0000 7fc093ac5640 1 -- 172.21.15.107:0/935914449 wait complete. 2023-10-25T00:01:28.398 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:01:30.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:30.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbfff4640 1 Processor -- start 2023-10-25T00:01:30.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbfff4640 1 -- start start 2023-10-25T00:01:30.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbfff4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1cb814fd40 0x7f1cb80ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbfff4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb806f730 con 0x7f1cb80a9820 2023-10-25T00:01:30.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbfff4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb8156a40 con 0x7f1cb806ee40 2023-10-25T00:01:30.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbfff4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb8156710 con 0x7f1cb814fd40 2023-10-25T00:01:30.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbdd90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1cb814fd40 0x7f1cb80ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbd58f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1cb80a9820 0x7f1cb80c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42918/0 (socket says 172.21.15.107:42918) 2023-10-25T00:01:30.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbdd90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1cb814fd40 0x7f1cb80ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44450/0 (socket says 172.21.15.107:44450) 2023-10-25T00:01:30.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbd58f640 1 -- 172.21.15.107:0/2893756203 learned_addr learned my addr 172.21.15.107:0/2893756203 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:30.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbdd90640 1 -- 172.21.15.107:0/2893756203 >> v1:172.21.15.110:6789/0 conn(0x7f1cb806ee40 legacy=0x7f1cb815cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbcd8e640 1 -- 172.21.15.107:0/2893756203 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693797691 0 0) 0x7f1cb806f730 con 0x7f1cb80a9820 2023-10-25T00:01:30.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbdd90640 1 -- 172.21.15.107:0/2893756203 >> v1:172.21.15.107:6789/0 conn(0x7f1cb80a9820 legacy=0x7f1cb80c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.532+0000 7f1cbdd90640 1 -- 172.21.15.107:0/2893756203 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb806f980 con 0x7f1cb814fd40 2023-10-25T00:01:30.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbdd90640 1 --2- 172.21.15.107:0/2893756203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb814fd40 0x7f1cb80ab090 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40097c0 tx=0x7f1cb401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c252121e2b01ee7c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:30.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbcd8e640 1 -- 172.21.15.107:0/2893756203 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb40ab600 con 0x7f1cb814fd40 2023-10-25T00:01:30.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbcd8e640 1 -- 172.21.15.107:0/2893756203 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb40a7050 con 0x7f1cb814fd40 2023-10-25T00:01:30.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbcd8e640 1 -- 172.21.15.107:0/2893756203 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb40ab980 con 0x7f1cb814fd40 2023-10-25T00:01:30.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 -- 172.21.15.107:0/2893756203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb814fd40 msgr2=0x7f1cb80ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 --2- 172.21.15.107:0/2893756203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb814fd40 0x7f1cb80ab090 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40097c0 tx=0x7f1cb401c360 comp rx=0 tx=0).stop 2023-10-25T00:01:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 -- 172.21.15.107:0/2893756203 shutdown_connections 2023-10-25T00:01:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 --2- 172.21.15.107:0/2893756203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb814fd40 0x7f1cb80ab090 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 -- 172.21.15.107:0/2893756203 >> 172.21.15.107:0/2893756203 conn(0x7f1cb80c2950 msgr2=0x7f1cb80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 -- 172.21.15.107:0/2893756203 shutdown_connections 2023-10-25T00:01:30.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 -- 172.21.15.107:0/2893756203 wait complete. 2023-10-25T00:01:30.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 Processor -- start 2023-10-25T00:01:30.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 -- start start 2023-10-25T00:01:30.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cb806ee40 0x7f1cb80bf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb814fd40 0x7f1cb81f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cb80c4a80 0x7f1cb81f0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb80c3f70 con 0x7f1cb814fd40 2023-10-25T00:01:30.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb80c3c40 con 0x7f1cb806ee40 2023-10-25T00:01:30.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb80c34d0 con 0x7f1cb80c4a80 2023-10-25T00:01:30.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbd58f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb814fd40 0x7f1cb81f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbd58f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb814fd40 0x7f1cb81f05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44462/0 (socket says 172.21.15.107:44462) 2023-10-25T00:01:30.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbd58f640 1 -- 172.21.15.107:0/1310996545 learned_addr learned my addr 172.21.15.107:0/1310996545 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:30.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbdd90640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cb806ee40 0x7f1cb80bf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbe591640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cb80c4a80 0x7f1cb81f0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbd58f640 1 -- 172.21.15.107:0/1310996545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cb80c4a80 msgr2=0x7f1cb81f0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbd58f640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cb80c4a80 0x7f1cb81f0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbd58f640 1 -- 172.21.15.107:0/1310996545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cb806ee40 msgr2=0x7f1cb80bf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbd58f640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cb806ee40 0x7f1cb80bf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbd58f640 1 -- 172.21.15.107:0/1310996545 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb4009420 con 0x7f1cb814fd40 2023-10-25T00:01:30.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbe591640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cb80c4a80 0x7f1cb81f0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:01:30.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbd58f640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb814fd40 0x7f1cb81f05f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80011f0 tx=0x7f1ca800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:30.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cae7fc640 1 -- 172.21.15.107:0/1310996545 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca80074e0 con 0x7f1cb814fd40 2023-10-25T00:01:30.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cae7fc640 1 -- 172.21.15.107:0/1310996545 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca8007640 con 0x7f1cb814fd40 2023-10-25T00:01:30.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 -- 172.21.15.107:0/1310996545 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb80c37b0 con 0x7f1cb814fd40 2023-10-25T00:01:30.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cae7fc640 1 -- 172.21.15.107:0/1310996545 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8004a10 con 0x7f1cb814fd40 2023-10-25T00:01:30.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.536+0000 7f1cbfff4640 1 -- 172.21.15.107:0/1310996545 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb8151080 con 0x7f1cb814fd40 2023-10-25T00:01:30.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.540+0000 7f1cbfff4640 1 -- 172.21.15.107:0/1310996545 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb80c24a0 con 0x7f1cb814fd40 2023-10-25T00:01:30.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.540+0000 7f1cae7fc640 1 -- 172.21.15.107:0/1310996545 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1ca8004b70 con 0x7f1cb814fd40 2023-10-25T00:01:30.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.540+0000 7f1cae7fc640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c9c065e90 0x7f1c9c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.540+0000 7f1cae7fc640 1 -- 172.21.15.107:0/1310996545 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7f1ca8083500 con 0x7f1cb814fd40 2023-10-25T00:01:30.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.540+0000 7f1cbdd90640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c9c065e90 0x7f1c9c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.540+0000 7f1cbdd90640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c9c065e90 0x7f1c9c068350 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40ad000 tx=0x7f1cb4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:30.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.544+0000 7f1cae7fc640 1 -- 172.21.15.107:0/1310996545 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1ca8050400 con 0x7f1cb814fd40 2023-10-25T00:01:30.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.736+0000 7f1cbfff4640 1 -- 172.21.15.107:0/1310996545 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cb80c3940 con 0x7f1cb814fd40 2023-10-25T00:01:30.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.736+0000 7f1cae7fc640 1 -- 172.21.15.107:0/1310996545 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f1ca8018090 con 0x7f1cb814fd40 2023-10-25T00:01:30.738 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:30.738 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:30.634248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:30.739 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 73 2023-10-25T00:01:30.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 -- 172.21.15.107:0/1310996545 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c9c065e90 msgr2=0x7f1c9c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c9c065e90 0x7f1c9c068350 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1cb40ad000 tx=0x7f1cb4008040 comp rx=0 tx=0).stop 2023-10-25T00:01:30.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 -- 172.21.15.107:0/1310996545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb814fd40 msgr2=0x7f1cb81f05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb814fd40 0x7f1cb81f05f0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80011f0 tx=0x7f1ca800c9b0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 -- 172.21.15.107:0/1310996545 shutdown_connections 2023-10-25T00:01:30.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c9c065e90 0x7f1c9c068350 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cb80c4a80 0x7f1cb81f0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cb814fd40 0x7f1cb81f05f0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 --2- 172.21.15.107:0/1310996545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cb806ee40 0x7f1cb80bf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 -- 172.21.15.107:0/1310996545 >> 172.21.15.107:0/1310996545 conn(0x7f1cb80c2950 msgr2=0x7f1cb80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:30.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 -- 172.21.15.107:0/1310996545 shutdown_connections 2023-10-25T00:01:30.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.740+0000 7f1cbfff4640 1 -- 172.21.15.107:0/1310996545 wait complete. 2023-10-25T00:01:30.758 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2023-10-25T00:01:30.758 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:01:30.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:30.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f152b0bc640 1 Processor -- start 2023-10-25T00:01:30.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f152b0bc640 1 -- start start 2023-10-25T00:01:30.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f152b0bc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1524152040 0x7f15240c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f152b0bc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1524155680 con 0x7f15240c8100 2023-10-25T00:01:30.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f152b0bc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1524154a70 con 0x7f152406a4b0 2023-10-25T00:01:30.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f152b0bc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1524154d50 con 0x7f1524152040 2023-10-25T00:01:30.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1523fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f15240c8100 0x7f15240abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42926/0 (socket says 172.21.15.107:42926) 2023-10-25T00:01:30.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1523fff640 1 -- 172.21.15.107:0/124996026 learned_addr learned my addr 172.21.15.107:0/124996026 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:30.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1528e58640 1 --2- 172.21.15.107:0/124996026 >> v2:172.21.15.110:3300/0 conn(0x7f1524152040 0x7f15240c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246025587 0 0) 0x7f1524155680 con 0x7f15240c8100 2023-10-25T00:01:30.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f150c003530 con 0x7f15240c8100 2023-10-25T00:01:30.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 50589149 0 0) 0x7f150c003530 con 0x7f15240c8100 2023-10-25T00:01:30.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 >> v2:172.21.15.110:3300/0 conn(0x7f1524152040 msgr2=0x7f15240c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:30.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 --2- 172.21.15.107:0/124996026 >> v2:172.21.15.110:3300/0 conn(0x7f1524152040 0x7f15240c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 >> v1:172.21.15.110:6789/0 conn(0x7f152406a4b0 legacy=0x7f15240ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1524155030 con 0x7f15240c8100 2023-10-25T00:01:30.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f1518001680 con 0x7f15240c8100 2023-10-25T00:01:30.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 >> v1:172.21.15.107:6789/0 conn(0x7f15240c8100 legacy=0x7f15240abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 --2- 172.21.15.107:0/124996026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f150c003980 0x7f150c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 --2- 172.21.15.107:0/124996026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 0x7f150c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 --2- 172.21.15.107:0/124996026 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f150c0052b0 0x7f150c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1524154d50 con 0x7f150c0047d0 2023-10-25T00:01:30.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150c00c180 con 0x7f150c003980 2023-10-25T00:01:30.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150c00c340 con 0x7f150c0052b0 2023-10-25T00:01:30.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1523fff640 1 --2- 172.21.15.107:0/124996026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 0x7f150c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1528e58640 1 --2- 172.21.15.107:0/124996026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f150c003980 0x7f150c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1529659640 1 --2- 172.21.15.107:0/124996026 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f150c0052b0 0x7f150c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1523fff640 1 -- 172.21.15.107:0/124996026 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f150c0052b0 msgr2=0x7f150c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:30.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1523fff640 1 --2- 172.21.15.107:0/124996026 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f150c0052b0 0x7f150c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1523fff640 1 -- 172.21.15.107:0/124996026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f150c003980 msgr2=0x7f150c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:30.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1523fff640 1 --2- 172.21.15.107:0/124996026 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f150c003980 0x7f150c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1529659640 1 -- 172.21.15.107:0/124996026 reap_dead start 2023-10-25T00:01:30.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.880+0000 7f1523fff640 1 -- 172.21.15.107:0/124996026 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c003530 con 0x7f150c0047d0 2023-10-25T00:01:30.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f1523fff640 1 --2- 172.21.15.107:0/124996026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 0x7f150c004c50 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f15240c8560 tx=0x7f151801cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2898a615dce4b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:30.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15180058e0 con 0x7f150c0047d0 2023-10-25T00:01:30.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1518005a40 con 0x7f150c0047d0 2023-10-25T00:01:30.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f15237fe640 1 -- 172.21.15.107:0/124996026 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518005e70 con 0x7f150c0047d0 2023-10-25T00:01:30.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 -- 172.21.15.107:0/124996026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 msgr2=0x7f150c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 --2- 172.21.15.107:0/124996026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 0x7f150c004c50 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f15240c8560 tx=0x7f151801cbc0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 -- 172.21.15.107:0/124996026 shutdown_connections 2023-10-25T00:01:30.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 --2- 172.21.15.107:0/124996026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 0x7f150c004c50 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 -- 172.21.15.107:0/124996026 >> 172.21.15.107:0/124996026 conn(0x7f15240ca200 msgr2=0x7f15240ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:30.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 -- 172.21.15.107:0/124996026 shutdown_connections 2023-10-25T00:01:30.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 -- 172.21.15.107:0/124996026 wait complete. 2023-10-25T00:01:30.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 Processor -- start 2023-10-25T00:01:30.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 -- start start 2023-10-25T00:01:30.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 0x7f15241e1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f152414b160 0x7f15240aa480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15241ecfc0 0x7f152414a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15241e2a20 con 0x7f150c0047d0 2023-10-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15241ecb30 con 0x7f15241ecfc0 2023-10-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f152b0bc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15241ec7e0 con 0x7f152414b160 2023-10-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f1528e58640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 0x7f15241e1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f1528e58640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 0x7f15241e1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44476/0 (socket says 172.21.15.107:44476) 2023-10-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.884+0000 7f1528e58640 1 -- 172.21.15.107:0/1451612114 learned_addr learned my addr 172.21.15.107:0/1451612114 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:30.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f1529659640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15241ecfc0 0x7f152414a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f1523fff640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f152414b160 0x7f15240aa480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f1528e58640 1 -- 172.21.15.107:0/1451612114 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f152414b160 msgr2=0x7f15240aa480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f1528e58640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f152414b160 0x7f15240aa480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f1528e58640 1 -- 172.21.15.107:0/1451612114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15241ecfc0 msgr2=0x7f152414a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f1528e58640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15241ecfc0 0x7f152414a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f1528e58640 1 -- 172.21.15.107:0/1451612114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15241dffd0 con 0x7f150c0047d0 2023-10-25T00:01:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f1523fff640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f152414b160 0x7f15240aa480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:01:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f1528e58640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 0x7f15241e1610 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1514011930 tx=0x7f1514011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:30.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f15217fa640 1 -- 172.21.15.107:0/1451612114 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15140095b0 con 0x7f150c0047d0 2023-10-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f15217fa640 1 -- 172.21.15.107:0/1451612114 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1514004240 con 0x7f150c0047d0 2023-10-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f15217fa640 1 -- 172.21.15.107:0/1451612114 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f151400f8e0 con 0x7f150c0047d0 2023-10-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f152b0bc640 1 -- 172.21.15.107:0/1451612114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15241e02b0 con 0x7f150c0047d0 2023-10-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f152b0bc640 1 -- 172.21.15.107:0/1451612114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15241e0870 con 0x7f150c0047d0 2023-10-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f152b0bc640 1 -- 172.21.15.107:0/1451612114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152414b540 con 0x7f150c0047d0 2023-10-25T00:01:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f15217fa640 1 -- 172.21.15.107:0/1451612114 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f151400fa40 con 0x7f150c0047d0 2023-10-25T00:01:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f15217fa640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e90 0x7f1510068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f15217fa640 1 -- 172.21.15.107:0/1451612114 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7f1514084580 con 0x7f150c0047d0 2023-10-25T00:01:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f1523fff640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e90 0x7f1510068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:30.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.888+0000 7f1523fff640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e90 0x7f1510068350 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f15180ab000 tx=0x7f15180a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:30.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:30.892+0000 7f15217fa640 1 -- 172.21.15.107:0/1451612114 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f152414b540 con 0x7f150c0047d0 2023-10-25T00:01:31.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.084+0000 7f152b0bc640 1 -- 172.21.15.107:0/1451612114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f152414b540 con 0x7f150c0047d0 2023-10-25T00:01:31.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.084+0000 7f15217fa640 1 -- 172.21.15.107:0/1451612114 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f152414b540 con 0x7f150c0047d0 2023-10-25T00:01:31.087 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:31.087 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:30.634248+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:31.088 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 73 2023-10-25T00:01:31.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 -- 172.21.15.107:0/1451612114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e90 msgr2=0x7f1510068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:31.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e90 0x7f1510068350 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f15180ab000 tx=0x7f15180a7070 comp rx=0 tx=0).stop 2023-10-25T00:01:31.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 -- 172.21.15.107:0/1451612114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 msgr2=0x7f15241e1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:31.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 0x7f15241e1610 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1514011930 tx=0x7f1514011df0 comp rx=0 tx=0).stop 2023-10-25T00:01:31.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 -- 172.21.15.107:0/1451612114 shutdown_connections 2023-10-25T00:01:31.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15241ecfc0 0x7f152414a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:31.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e90 0x7f1510068350 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:31.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f152414b160 0x7f15240aa480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:31.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 --2- 172.21.15.107:0/1451612114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f150c0047d0 0x7f15241e1610 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:31.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 -- 172.21.15.107:0/1451612114 >> 172.21.15.107:0/1451612114 conn(0x7f15240ca200 msgr2=0x7f152414db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:31.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 -- 172.21.15.107:0/1451612114 shutdown_connections 2023-10-25T00:01:31.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:31.088+0000 7f152b0bc640 1 -- 172.21.15.107:0/1451612114 wait complete. 2023-10-25T00:01:31.105 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 73, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 73, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6303": { "addr": "172.21.15.107:6837/3103569740", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 3103569740, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 3103569740, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6303, "incarnation": 71, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:01:30.634248+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6303 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 69, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5506": { "addr": "172.21.15.110:6834/201284779", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 201284779, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 201284779, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5506, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:00:51.726634+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5506 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 59, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:01:33.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:33.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.240+0000 7f309b920640 1 Processor -- start 2023-10-25T00:01:33.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.240+0000 7f309b920640 1 -- start start 2023-10-25T00:01:33.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.240+0000 7f309b920640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3094156770 0x7f30940ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:33.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.240+0000 7f309b920640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30940c70e0 0x7f30940bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:33.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.240+0000 7f309b920640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30940c1ca0 con 0x7f30940c2130 2023-10-25T00:01:33.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.240+0000 7f309b920640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30940c27e0 con 0x7f30940c70e0 2023-10-25T00:01:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.240+0000 7f309b920640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30940c10c0 con 0x7f3094156770 2023-10-25T00:01:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f3098ebb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30940c70e0 0x7f30940bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f3098ebb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30940c70e0 0x7f30940bdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54468/0 (socket says 172.21.15.107:54468) 2023-10-25T00:01:33.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f3098ebb640 1 -- 172.21.15.107:0/1684613936 learned_addr learned my addr 172.21.15.107:0/1684613936 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f30996bc640 1 --2- 172.21.15.107:0/1684613936 >> v2:172.21.15.110:3300/0 conn(0x7f3094156770 0x7f30940ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f3098ebb640 1 -- 172.21.15.107:0/1684613936 >> v2:172.21.15.110:3300/0 conn(0x7f3094156770 msgr2=0x7f30940ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f3098ebb640 1 --2- 172.21.15.107:0/1684613936 >> v2:172.21.15.110:3300/0 conn(0x7f3094156770 0x7f30940ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:33.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f3098ebb640 1 -- 172.21.15.107:0/1684613936 >> v1:172.21.15.110:6789/0 conn(0x7f30940c2130 legacy=0x7f3094069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:33.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f3098ebb640 1 -- 172.21.15.107:0/1684613936 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30940c13a0 con 0x7f30940c70e0 2023-10-25T00:01:33.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f3098ebb640 1 --2- 172.21.15.107:0/1684613936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30940c70e0 0x7f30940bdfa0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f30940ac6f0 tx=0x7f308401c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aea2e7a3d7a8c95 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:33.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f308bfff640 1 -- 172.21.15.107:0/1684613936 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30840ab600 con 0x7f30940c70e0 2023-10-25T00:01:33.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f308bfff640 1 -- 172.21.15.107:0/1684613936 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30840a7050 con 0x7f30940c70e0 2023-10-25T00:01:33.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f308bfff640 1 -- 172.21.15.107:0/1684613936 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30840ab980 con 0x7f30940c70e0 2023-10-25T00:01:33.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f309b920640 1 -- 172.21.15.107:0/1684613936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30940c70e0 msgr2=0x7f30940bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:33.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f309b920640 1 --2- 172.21.15.107:0/1684613936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30940c70e0 0x7f30940bdfa0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f30940ac6f0 tx=0x7f308401c370 comp rx=0 tx=0).stop 2023-10-25T00:01:33.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f309b920640 1 -- 172.21.15.107:0/1684613936 shutdown_connections 2023-10-25T00:01:33.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f309b920640 1 --2- 172.21.15.107:0/1684613936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30940c70e0 0x7f30940bdfa0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:33.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f309b920640 1 --2- 172.21.15.107:0/1684613936 >> v2:172.21.15.110:3300/0 conn(0x7f3094156770 0x7f30940ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:33.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f309b920640 1 -- 172.21.15.107:0/1684613936 >> 172.21.15.107:0/1684613936 conn(0x7f30940c65c0 msgr2=0x7f30940c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:33.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f309b920640 1 -- 172.21.15.107:0/1684613936 shutdown_connections 2023-10-25T00:01:33.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.244+0000 7f309b920640 1 -- 172.21.15.107:0/1684613936 wait complete. 2023-10-25T00:01:33.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f309b920640 1 Processor -- start 2023-10-25T00:01:33.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f309b920640 1 -- start start 2023-10-25T00:01:33.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f309b920640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30940c2130 0x7f30941edb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:33.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f309b920640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30940c70e0 0x7f3094153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:33.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f309b920640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3094156770 0x7f309414d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:33.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f309b920640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309415cd90 con 0x7f30940c70e0 2023-10-25T00:01:33.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f309b920640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3094152850 con 0x7f3094156770 2023-10-25T00:01:33.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f309b920640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30941ee060 con 0x7f30940c2130 2023-10-25T00:01:33.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3098ebb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30940c70e0 0x7f3094153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:33.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3098ebb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30940c70e0 0x7f3094153440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54472/0 (socket says 172.21.15.107:54472) 2023-10-25T00:01:33.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3098ebb640 1 -- 172.21.15.107:0/2806865448 learned_addr learned my addr 172.21.15.107:0/2806865448 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:33.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3099ebd640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3094156770 0x7f309414d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:33.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f30996bc640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30940c2130 0x7f30941edb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:33.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3098ebb640 1 -- 172.21.15.107:0/2806865448 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30940c2130 msgr2=0x7f30941edb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:33.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3098ebb640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30940c2130 0x7f30941edb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:33.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3098ebb640 1 -- 172.21.15.107:0/2806865448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3094156770 msgr2=0x7f309414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:33.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3098ebb640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3094156770 0x7f309414d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:33.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3098ebb640 1 -- 172.21.15.107:0/2806865448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30840094d0 con 0x7f30940c70e0 2023-10-25T00:01:33.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f30996bc640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30940c2130 0x7f30941edb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:01:33.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3098ebb640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30940c70e0 0x7f3094153440 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3084019fa0 tx=0x7f3084019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:33.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3089ffb640 1 -- 172.21.15.107:0/2806865448 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30840b4440 con 0x7f30940c70e0 2023-10-25T00:01:33.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f309b920640 1 -- 172.21.15.107:0/2806865448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30941525b0 con 0x7f30940c70e0 2023-10-25T00:01:33.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3089ffb640 1 -- 172.21.15.107:0/2806865448 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30840b45a0 con 0x7f30940c70e0 2023-10-25T00:01:33.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f3089ffb640 1 -- 172.21.15.107:0/2806865448 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30840b4930 con 0x7f30940c70e0 2023-10-25T00:01:33.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.248+0000 7f309b920640 1 -- 172.21.15.107:0/2806865448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30940ab2c0 con 0x7f30940c70e0 2023-10-25T00:01:33.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.252+0000 7f309b920640 1 -- 172.21.15.107:0/2806865448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309415ced0 con 0x7f30940c70e0 2023-10-25T00:01:33.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.252+0000 7f3089ffb640 1 -- 172.21.15.107:0/2806865448 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f30840b2070 con 0x7f30940c70e0 2023-10-25T00:01:33.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.252+0000 7f3089ffb640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3078065e60 0x7f3078068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:33.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.252+0000 7f3089ffb640 1 -- 172.21.15.107:0/2806865448 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7f308411df30 con 0x7f30940c70e0 2023-10-25T00:01:33.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.252+0000 7f30996bc640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3078065e60 0x7f3078068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:33.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.252+0000 7f30996bc640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3078065e60 0x7f3078068320 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f309000b9f0 tx=0x7f3090008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:33.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.260+0000 7f3089ffb640 1 -- 172.21.15.107:0/2806865448 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f309415ced0 con 0x7f30940c70e0 2023-10-25T00:01:33.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.520+0000 7f309b920640 1 -- 172.21.15.107:0/2806865448 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f309415ced0 con 0x7f30940c70e0 2023-10-25T00:01:33.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.520+0000 7f3089ffb640 1 -- 172.21.15.107:0/2806865448 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f309415ced0 con 0x7f30940c70e0 2023-10-25T00:01:33.521 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:33.521 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:31.647132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:33.522 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 74 2023-10-25T00:01:33.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 -- 172.21.15.107:0/2806865448 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3078065e60 msgr2=0x7f3078068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3078065e60 0x7f3078068320 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f309000b9f0 tx=0x7f3090008040 comp rx=0 tx=0).stop 2023-10-25T00:01:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 -- 172.21.15.107:0/2806865448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30940c70e0 msgr2=0x7f3094153440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30940c70e0 0x7f3094153440 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f3084019fa0 tx=0x7f3084019110 comp rx=0 tx=0).stop 2023-10-25T00:01:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 -- 172.21.15.107:0/2806865448 shutdown_connections 2023-10-25T00:01:33.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3094156770 0x7f309414d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:33.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30940c70e0 0x7f3094153440 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:33.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3078065e60 0x7f3078068320 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:33.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 --2- 172.21.15.107:0/2806865448 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30940c2130 0x7f30941edb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:33.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 -- 172.21.15.107:0/2806865448 >> 172.21.15.107:0/2806865448 conn(0x7f30940c65c0 msgr2=0x7f30940c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:33.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 -- 172.21.15.107:0/2806865448 shutdown_connections 2023-10-25T00:01:33.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:33.524+0000 7f309b920640 1 -- 172.21.15.107:0/2806865448 wait complete. 2023-10-25T00:01:33.546 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6102 2023-10-25T00:01:33.546 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.c 2023-10-25T00:01:34.547 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2023-10-25T00:01:34.547 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:01:34.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:01:34.550 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:01:34.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:34.619 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:01:34.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cf36c640 1 Processor -- start 2023-10-25T00:01:34.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cf36c640 1 -- start start 2023-10-25T00:01:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cf36c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe7c8152040 0x7fe7c80c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cf36c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe7c806a4b0 0x7fe7c80ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:34.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cf36c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c8155680 con 0x7fe7c80c8100 2023-10-25T00:01:34.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cf36c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c8154a70 con 0x7fe7c806a4b0 2023-10-25T00:01:34.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cf36c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7c8154d50 con 0x7fe7c8152040 2023-10-25T00:01:34.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cd909640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe7c806a4b0 0x7fe7c80ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:34.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cd909640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe7c806a4b0 0x7fe7c80ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54486/0 (socket says 172.21.15.107:54486) 2023-10-25T00:01:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cd909640 1 -- 172.21.15.107:0/1829260031 learned_addr learned my addr 172.21.15.107:0/1829260031 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cd108640 1 --2- 172.21.15.107:0/1829260031 >> v2:172.21.15.110:3300/0 conn(0x7fe7c8152040 0x7fe7c80c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cd909640 1 -- 172.21.15.107:0/1829260031 >> v2:172.21.15.110:3300/0 conn(0x7fe7c8152040 msgr2=0x7fe7c80c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cd909640 1 --2- 172.21.15.107:0/1829260031 >> v2:172.21.15.110:3300/0 conn(0x7fe7c8152040 0x7fe7c80c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:34.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cd909640 1 -- 172.21.15.107:0/1829260031 >> v1:172.21.15.110:6789/0 conn(0x7fe7c80c8100 legacy=0x7fe7c80abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:34.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cd909640 1 -- 172.21.15.107:0/1829260031 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c8155030 con 0x7fe7c806a4b0 2023-10-25T00:01:34.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7cd909640 1 --2- 172.21.15.107:0/1829260031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c806a4b0 0x7fe7c80ac410 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8067ba0 tx=0x7fe7c001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=413d8892d8adbb5d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:34.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7bffff640 1 -- 172.21.15.107:0/1829260031 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c00ab600 con 0x7fe7c806a4b0 2023-10-25T00:01:34.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7bffff640 1 -- 172.21.15.107:0/1829260031 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c00a7050 con 0x7fe7c806a4b0 2023-10-25T00:01:34.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.664+0000 7fe7bffff640 1 -- 172.21.15.107:0/1829260031 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c00ab980 con 0x7fe7c806a4b0 2023-10-25T00:01:34.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1829260031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c806a4b0 msgr2=0x7fe7c80ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:34.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 --2- 172.21.15.107:0/1829260031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c806a4b0 0x7fe7c80ac410 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8067ba0 tx=0x7fe7c001c430 comp rx=0 tx=0).stop 2023-10-25T00:01:34.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1829260031 shutdown_connections 2023-10-25T00:01:34.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 --2- 172.21.15.107:0/1829260031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c806a4b0 0x7fe7c80ac410 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:34.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 --2- 172.21.15.107:0/1829260031 >> v2:172.21.15.110:3300/0 conn(0x7fe7c8152040 0x7fe7c80c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:34.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1829260031 >> 172.21.15.107:0/1829260031 conn(0x7fe7c80ca200 msgr2=0x7fe7c80ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:34.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1829260031 shutdown_connections 2023-10-25T00:01:34.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1829260031 wait complete. 2023-10-25T00:01:34.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 Processor -- start 2023-10-25T00:01:34.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- start start 2023-10-25T00:01:34.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c806a4b0 0x7fe7c814fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:34.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7c80c8100 0x7fe7c8147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:34.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7c8152040 0x7fe7c81479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:34.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c8148d90 con 0x7fe7c806a4b0 2023-10-25T00:01:34.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c814f210 con 0x7fe7c80c8100 2023-10-25T00:01:34.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c814f4d0 con 0x7fe7c8152040 2023-10-25T00:01:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cd108640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c806a4b0 0x7fe7c814fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cd108640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c806a4b0 0x7fe7c814fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54490/0 (socket says 172.21.15.107:54490) 2023-10-25T00:01:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cd108640 1 -- 172.21.15.107:0/1063000904 learned_addr learned my addr 172.21.15.107:0/1063000904 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cc907640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7c80c8100 0x7fe7c8147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cd909640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7c8152040 0x7fe7c81479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cd108640 1 -- 172.21.15.107:0/1063000904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7c8152040 msgr2=0x7fe7c81479b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cd108640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7c8152040 0x7fe7c81479b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cd108640 1 -- 172.21.15.107:0/1063000904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7c80c8100 msgr2=0x7fe7c8147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:34.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cd108640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7c80c8100 0x7fe7c8147470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:34.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cd108640 1 -- 172.21.15.107:0/1063000904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c00094d0 con 0x7fe7c806a4b0 2023-10-25T00:01:34.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cd108640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c806a4b0 0x7fe7c814fd20 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4007ef0 tx=0x7fe7c400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:34.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7bdffb640 1 -- 172.21.15.107:0/1063000904 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4014620 con 0x7fe7c806a4b0 2023-10-25T00:01:34.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7bdffb640 1 -- 172.21.15.107:0/1063000904 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c4014780 con 0x7fe7c806a4b0 2023-10-25T00:01:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1063000904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c814f7b0 con 0x7fe7c806a4b0 2023-10-25T00:01:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7bdffb640 1 -- 172.21.15.107:0/1063000904 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4014b30 con 0x7fe7c806a4b0 2023-10-25T00:01:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1063000904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c8151cf0 con 0x7fe7c806a4b0 2023-10-25T00:01:34.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.668+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1063000904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c8151eb0 con 0x7fe7c806a4b0 2023-10-25T00:01:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.672+0000 7fe7bdffb640 1 -- 172.21.15.107:0/1063000904 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe7c4014c90 con 0x7fe7c806a4b0 2023-10-25T00:01:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.672+0000 7fe7bdffb640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe7ac065e60 0x7fe7ac068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.672+0000 7fe7bdffb640 1 -- 172.21.15.107:0/1063000904 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7fe7c40848b0 con 0x7fe7c806a4b0 2023-10-25T00:01:34.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.672+0000 7fe7cc907640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe7ac065e60 0x7fe7ac068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.672+0000 7fe7cc907640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe7ac065e60 0x7fe7ac068320 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe7c80c9d90 tx=0x7fe7b8009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.676+0000 7fe7bdffb640 1 -- 172.21.15.107:0/1063000904 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe7c40517b0 con 0x7fe7c806a4b0 2023-10-25T00:01:34.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.888+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1063000904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7c80746e0 con 0x7fe7c806a4b0 2023-10-25T00:01:34.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7bdffb640 1 -- 172.21.15.107:0/1063000904 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fe7c4054dd0 con 0x7fe7c806a4b0 2023-10-25T00:01:34.892 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:31.647132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:34.894 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 75 2023-10-25T00:01:34.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1063000904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe7ac065e60 msgr2=0x7fe7ac068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:34.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe7ac065e60 0x7fe7ac068320 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe7c80c9d90 tx=0x7fe7b8009060 comp rx=0 tx=0).stop 2023-10-25T00:01:34.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1063000904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c806a4b0 msgr2=0x7fe7c814fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:34.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c806a4b0 0x7fe7c814fd20 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4007ef0 tx=0x7fe7c400e4b0 comp rx=0 tx=0).stop 2023-10-25T00:01:34.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1063000904 shutdown_connections 2023-10-25T00:01:34.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe7ac065e60 0x7fe7ac068320 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:34.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe7c8152040 0x7fe7c81479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:34.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe7c80c8100 0x7fe7c8147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:34.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 --2- 172.21.15.107:0/1063000904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c806a4b0 0x7fe7c814fd20 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:34.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1063000904 >> 172.21.15.107:0/1063000904 conn(0x7fe7c80ca200 msgr2=0x7fe7c814b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:34.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1063000904 shutdown_connections 2023-10-25T00:01:34.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:34.892+0000 7fe7cf36c640 1 -- 172.21.15.107:0/1063000904 wait complete. 2023-10-25T00:01:34.914 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2023-10-25T00:01:34.915 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before thrashing 2023-10-25T00:01:36.501 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:01:36.499+0000 7f178f60f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:01:42.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:43.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f239db42640 1 Processor -- start 2023-10-25T00:01:43.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f239db42640 1 -- start start 2023-10-25T00:01:43.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f239db42640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2398070b00 0x7f23980c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f239db42640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980c22f0 con 0x7f239814eb70 2023-10-25T00:01:43.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f239db42640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980c25d0 con 0x7f239806df30 2023-10-25T00:01:43.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f239db42640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2398071700 con 0x7f2398070b00 2023-10-25T00:01:43.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f23977fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f239814eb70 0x7f23981514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48878/0 (socket says 172.21.15.107:48878) 2023-10-25T00:01:43.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f23977fe640 1 -- 172.21.15.107:0/1718577623 learned_addr learned my addr 172.21.15.107:0/1718577623 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:43.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f238ffff640 1 --2- 172.21.15.107:0/1718577623 >> v2:172.21.15.110:3300/0 conn(0x7f2398070b00 0x7f23980c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336180603 0 0) 0x7f23980c22f0 con 0x7f239814eb70 2023-10-25T00:01:43.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2384003530 con 0x7f239814eb70 2023-10-25T00:01:43.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196368775 0 0) 0x7f2384003530 con 0x7f239814eb70 2023-10-25T00:01:43.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 >> v2:172.21.15.110:3300/0 conn(0x7f2398070b00 msgr2=0x7f23980c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:43.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 --2- 172.21.15.107:0/1718577623 >> v2:172.21.15.110:3300/0 conn(0x7f2398070b00 0x7f23980c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 >> v1:172.21.15.110:6789/0 conn(0x7f239806df30 legacy=0x7f239815be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23980c2be0 con 0x7f239814eb70 2023-10-25T00:01:43.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f2388001680 con 0x7f239814eb70 2023-10-25T00:01:43.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 >> v1:172.21.15.107:6789/0 conn(0x7f239814eb70 legacy=0x7f23981514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 --2- 172.21.15.107:0/1718577623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2384003980 0x7f2384003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 --2- 172.21.15.107:0/1718577623 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23840047d0 0x7f2384004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 --2- 172.21.15.107:0/1718577623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2384005190 0x7f238400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238400c000 con 0x7f2384003980 2023-10-25T00:01:43.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238400c170 con 0x7f2384005190 2023-10-25T00:01:43.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f238400c330 con 0x7f23840047d0 2023-10-25T00:01:43.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.067+0000 7f23977fe640 1 --2- 172.21.15.107:0/1718577623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2384003980 0x7f2384003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f23977fe640 1 -- 172.21.15.107:0/1718577623 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23840047d0 msgr2=0x7f2384004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:43.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f23977fe640 1 --2- 172.21.15.107:0/1718577623 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23840047d0 0x7f2384004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f23977fe640 1 -- 172.21.15.107:0/1718577623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2384005190 msgr2=0x7f238400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:43.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f2397fff640 1 --2- 172.21.15.107:0/1718577623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2384005190 0x7f238400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f23977fe640 1 --2- 172.21.15.107:0/1718577623 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2384005190 0x7f238400b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f2397fff640 1 -- 172.21.15.107:0/1718577623 reap_dead start 2023-10-25T00:01:43.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f23977fe640 1 -- 172.21.15.107:0/1718577623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2384003530 con 0x7f2384003980 2023-10-25T00:01:43.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f23977fe640 1 --2- 172.21.15.107:0/1718577623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2384003980 0x7f2384003da0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f238800c240 tx=0x7f238801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7e28aaa948aafb server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:43.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2388005ae0 con 0x7f2384003980 2023-10-25T00:01:43.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2388005c40 con 0x7f2384003980 2023-10-25T00:01:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f2396ffd640 1 -- 172.21.15.107:0/1718577623 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23880a5830 con 0x7f2384003980 2023-10-25T00:01:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f239db42640 1 -- 172.21.15.107:0/1718577623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2384003980 msgr2=0x7f2384003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f239db42640 1 --2- 172.21.15.107:0/1718577623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2384003980 0x7f2384003da0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f238800c240 tx=0x7f238801cc40 comp rx=0 tx=0).stop 2023-10-25T00:01:43.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f239db42640 1 -- 172.21.15.107:0/1718577623 shutdown_connections 2023-10-25T00:01:43.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f239db42640 1 --2- 172.21.15.107:0/1718577623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2384003980 0x7f2384003da0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f239db42640 1 -- 172.21.15.107:0/1718577623 >> 172.21.15.107:0/1718577623 conn(0x7f23980c39a0 msgr2=0x7f23980c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:43.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f239db42640 1 -- 172.21.15.107:0/1718577623 shutdown_connections 2023-10-25T00:01:43.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.071+0000 7f239db42640 1 -- 172.21.15.107:0/1718577623 wait complete. 2023-10-25T00:01:43.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f239db42640 1 Processor -- start 2023-10-25T00:01:43.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f239db42640 1 -- start start 2023-10-25T00:01:43.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f239db42640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2384003980 0x7f2398153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f239db42640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f239814a050 0x7f23981ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f239db42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f239814a4b0 0x7f23981ee580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f239db42640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23981eb500 con 0x7f239814a4b0 2023-10-25T00:01:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f239db42640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23981eeac0 con 0x7f2384003980 2023-10-25T00:01:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f239db42640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239814aa60 con 0x7f239814a050 2023-10-25T00:01:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2397fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f239814a4b0 0x7f23981ee580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2397fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f239814a4b0 0x7f23981ee580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37344/0 (socket says 172.21.15.107:37344) 2023-10-25T00:01:43.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2397fff640 1 -- 172.21.15.107:0/898824935 learned_addr learned my addr 172.21.15.107:0/898824935 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:43.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f23977fe640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2384003980 0x7f2398153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f238ffff640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f239814a050 0x7f23981ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2397fff640 1 -- 172.21.15.107:0/898824935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f239814a050 msgr2=0x7f23981ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2397fff640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f239814a050 0x7f23981ee040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2397fff640 1 -- 172.21.15.107:0/898824935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2384003980 msgr2=0x7f2398153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2397fff640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2384003980 0x7f2398153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2397fff640 1 -- 172.21.15.107:0/898824935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23981eab60 con 0x7f239814a4b0 2023-10-25T00:01:43.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2397fff640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f239814a4b0 0x7f23981ee580 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f23980c69c0 tx=0x7f239000d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:43.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2394ff9640 1 -- 172.21.15.107:0/898824935 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2390003d90 con 0x7f239814a4b0 2023-10-25T00:01:43.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2394ff9640 1 -- 172.21.15.107:0/898824935 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2390005070 con 0x7f239814a4b0 2023-10-25T00:01:43.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f239db42640 1 -- 172.21.15.107:0/898824935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23981eae40 con 0x7f239814a4b0 2023-10-25T00:01:43.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f2394ff9640 1 -- 172.21.15.107:0/898824935 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2390005420 con 0x7f239814a4b0 2023-10-25T00:01:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f239db42640 1 -- 172.21.15.107:0/898824935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2398154460 con 0x7f239814a4b0 2023-10-25T00:01:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.075+0000 7f239db42640 1 -- 172.21.15.107:0/898824935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23981eb640 con 0x7f239814a4b0 2023-10-25T00:01:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.079+0000 7f2394ff9640 1 -- 172.21.15.107:0/898824935 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2390005580 con 0x7f239814a4b0 2023-10-25T00:01:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.079+0000 7f2394ff9640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2378065e90 0x7f2378068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.079+0000 7f2394ff9640 1 -- 172.21.15.107:0/898824935 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7f23900837e0 con 0x7f239814a4b0 2023-10-25T00:01:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.079+0000 7f23977fe640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2378065e90 0x7f2378068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.079+0000 7f23977fe640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2378065e90 0x7f2378068350 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f23880ab000 tx=0x7f23880b1040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.083+0000 7f2394ff9640 1 -- 172.21.15.107:0/898824935 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f23981eb640 con 0x7f239814a4b0 2023-10-25T00:01:43.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.311+0000 7f239db42640 1 -- 172.21.15.107:0/898824935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23981eb640 con 0x7f239814a4b0 2023-10-25T00:01:43.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.311+0000 7f2394ff9640 1 -- 172.21.15.107:0/898824935 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f23981eb640 con 0x7f239814a4b0 2023-10-25T00:01:43.313 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:43.313 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:31.647132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:43.314 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 75 2023-10-25T00:01:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.311+0000 7f239db42640 1 -- 172.21.15.107:0/898824935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2378065e90 msgr2=0x7f2378068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.311+0000 7f239db42640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2378065e90 0x7f2378068350 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f23880ab000 tx=0x7f23880b1040 comp rx=0 tx=0).stop 2023-10-25T00:01:43.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.311+0000 7f239db42640 1 -- 172.21.15.107:0/898824935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f239814a4b0 msgr2=0x7f23981ee580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.311+0000 7f239db42640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f239814a4b0 0x7f23981ee580 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f23980c69c0 tx=0x7f239000d950 comp rx=0 tx=0).stop 2023-10-25T00:01:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.315+0000 7f239db42640 1 -- 172.21.15.107:0/898824935 shutdown_connections 2023-10-25T00:01:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.315+0000 7f239db42640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f239814a4b0 0x7f23981ee580 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.315+0000 7f239db42640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2378065e90 0x7f2378068350 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.315+0000 7f239db42640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f239814a050 0x7f23981ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.315+0000 7f239db42640 1 --2- 172.21.15.107:0/898824935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2384003980 0x7f2398153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.315+0000 7f239db42640 1 -- 172.21.15.107:0/898824935 >> 172.21.15.107:0/898824935 conn(0x7f23980c39a0 msgr2=0x7f239806fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.315+0000 7f239db42640 1 -- 172.21.15.107:0/898824935 shutdown_connections 2023-10-25T00:01:43.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.315+0000 7f239db42640 1 -- 172.21.15.107:0/898824935 wait complete. 2023-10-25T00:01:43.334 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2023-10-25T00:01:43.335 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:01:43.335 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:01:43.345 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:01:43.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:43.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab76b1e640 1 Processor -- start 2023-10-25T00:01:43.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab76b1e640 1 -- start start 2023-10-25T00:01:43.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab76b1e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fab70071ce0 0x7fab700ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab76b1e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab70154860 0x7fab70158230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab76b1e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7006e710 con 0x7fab700720c0 2023-10-25T00:01:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab76b1e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7006d900 con 0x7fab70071ce0 2023-10-25T00:01:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab76b1e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab70154d20 con 0x7fab70154860 2023-10-25T00:01:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fab700720c0 0x7fab7013d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48894/0 (socket says 172.21.15.107:48894) 2023-10-25T00:01:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6ffff640 1 -- 172.21.15.107:0/1791545782 learned_addr learned my addr 172.21.15.107:0/1791545782 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:43.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab750bb640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab70154860 0x7fab70158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab748ba640 1 --2- 172.21.15.107:0/1791545782 >> v2:172.21.15.110:3300/0 conn(0x7fab70071ce0 0x7fab700ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792240782 0 0) 0x7fab7006e710 con 0x7fab700720c0 2023-10-25T00:01:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab60003530 con 0x7fab700720c0 2023-10-25T00:01:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2776970279 0 0) 0x7fab60003530 con 0x7fab700720c0 2023-10-25T00:01:43.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab70154860 msgr2=0x7fab70158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:43.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab70154860 0x7fab70158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 >> v2:172.21.15.110:3300/0 conn(0x7fab70071ce0 msgr2=0x7fab700ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:43.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 --2- 172.21.15.107:0/1791545782 >> v2:172.21.15.110:3300/0 conn(0x7fab70071ce0 0x7fab700ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab70155000 con 0x7fab700720c0 2023-10-25T00:01:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fab64001680 con 0x7fab700720c0 2023-10-25T00:01:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 >> v1:172.21.15.107:6789/0 conn(0x7fab700720c0 legacy=0x7fab7013d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 0x7fab60003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab600047d0 0x7fab60004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab70154860 0x7fab60005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab70154d20 con 0x7fab60003980 2023-10-25T00:01:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6000b990 con 0x7fab600047d0 2023-10-25T00:01:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6000bb00 con 0x7fab70154860 2023-10-25T00:01:43.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab748ba640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 0x7fab60003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab6ffff640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab600047d0 0x7fab60004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab750bb640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab70154860 0x7fab60005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab748ba640 1 -- 172.21.15.107:0/1791545782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab70154860 msgr2=0x7fab60005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab748ba640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab70154860 0x7fab60005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab748ba640 1 -- 172.21.15.107:0/1791545782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab600047d0 msgr2=0x7fab60004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:43.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab748ba640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab600047d0 0x7fab60004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.507+0000 7fab748ba640 1 -- 172.21.15.107:0/1791545782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60003530 con 0x7fab60003980 2023-10-25T00:01:43.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab748ba640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 0x7fab60003da0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009ae0 tx=0x7fab5c01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed2efea99bf5eac server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:43.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0a5550 con 0x7fab60003980 2023-10-25T00:01:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c0a56b0 con 0x7fab60003980 2023-10-25T00:01:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab6f7fe640 1 -- 172.21.15.107:0/1791545782 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0a5a60 con 0x7fab60003980 2023-10-25T00:01:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab76b1e640 1 -- 172.21.15.107:0/1791545782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 msgr2=0x7fab60003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab76b1e640 1 --2- 172.21.15.107:0/1791545782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 0x7fab60003da0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009ae0 tx=0x7fab5c01ca00 comp rx=0 tx=0).stop 2023-10-25T00:01:43.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab750bb640 1 -- 172.21.15.107:0/1791545782 reap_dead start 2023-10-25T00:01:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab76b1e640 1 -- 172.21.15.107:0/1791545782 shutdown_connections 2023-10-25T00:01:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab76b1e640 1 -- 172.21.15.107:0/1791545782 >> 172.21.15.107:0/1791545782 conn(0x7fab70070f60 msgr2=0x7fab70071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab76b1e640 1 -- 172.21.15.107:0/1791545782 shutdown_connections 2023-10-25T00:01:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab76b1e640 1 -- 172.21.15.107:0/1791545782 wait complete. 2023-10-25T00:01:43.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab76b1e640 1 Processor -- start 2023-10-25T00:01:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.511+0000 7fab76b1e640 1 -- start start 2023-10-25T00:01:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab76b1e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 0x7fab701395a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab76b1e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab700c6e10 0x7fab700c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab76b1e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab70155bb0 0x7fab70156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab76b1e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab70139bb0 con 0x7fab60003980 2023-10-25T00:01:43.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab76b1e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7013a190 con 0x7fab70155bb0 2023-10-25T00:01:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab76b1e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab701ec650 con 0x7fab700c6e10 2023-10-25T00:01:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab748ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 0x7fab701395a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab748ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 0x7fab701395a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37368/0 (socket says 172.21.15.107:37368) 2023-10-25T00:01:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab748ba640 1 -- 172.21.15.107:0/599597922 learned_addr learned my addr 172.21.15.107:0/599597922 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:43.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab750bb640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab70155bb0 0x7fab70156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab6ffff640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab700c6e10 0x7fab700c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab748ba640 1 -- 172.21.15.107:0/599597922 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab700c6e10 msgr2=0x7fab700c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab748ba640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab700c6e10 0x7fab700c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab748ba640 1 -- 172.21.15.107:0/599597922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab70155bb0 msgr2=0x7fab70156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab748ba640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab70155bb0 0x7fab70156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab748ba640 1 -- 172.21.15.107:0/599597922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7013e200 con 0x7fab60003980 2023-10-25T00:01:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab748ba640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 0x7fab701395a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009ae0 tx=0x7fab60005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab6d7fa640 1 -- 172.21.15.107:0/599597922 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0a5e20 con 0x7fab60003980 2023-10-25T00:01:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab6d7fa640 1 -- 172.21.15.107:0/599597922 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c004020 con 0x7fab60003980 2023-10-25T00:01:43.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab6d7fa640 1 -- 172.21.15.107:0/599597922 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c004390 con 0x7fab60003980 2023-10-25T00:01:43.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab76b1e640 1 -- 172.21.15.107:0/599597922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab701ec930 con 0x7fab60003980 2023-10-25T00:01:43.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab76b1e640 1 -- 172.21.15.107:0/599597922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab701ecef0 con 0x7fab60003980 2023-10-25T00:01:43.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab76b1e640 1 -- 172.21.15.107:0/599597922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab700746e0 con 0x7fab60003980 2023-10-25T00:01:43.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab6d7fa640 1 -- 172.21.15.107:0/599597922 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fab5c0b4040 con 0x7fab60003980 2023-10-25T00:01:43.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.515+0000 7fab6d7fa640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab54065e60 0x7fab54068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.519+0000 7fab6d7fa640 1 -- 172.21.15.107:0/599597922 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7fab5c11df70 con 0x7fab60003980 2023-10-25T00:01:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.519+0000 7fab6ffff640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab54065e60 0x7fab54068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.519+0000 7fab6ffff640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab54065e60 0x7fab54068320 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fab7006cf30 tx=0x7fab64009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:43.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.523+0000 7fab6d7fa640 1 -- 172.21.15.107:0/599597922 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fab5c0eaf20 con 0x7fab60003980 2023-10-25T00:01:43.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 -- 172.21.15.107:0/599597922 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab70139bb0 con 0x7fab60003980 2023-10-25T00:01:43.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab6d7fa640 1 -- 172.21.15.107:0/599597922 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fab5c0ee540 con 0x7fab60003980 2023-10-25T00:01:43.761 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:43.761 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:31.647132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 75 2023-10-25T00:01:43.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 -- 172.21.15.107:0/599597922 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab54065e60 msgr2=0x7fab54068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab54065e60 0x7fab54068320 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fab7006cf30 tx=0x7fab64009040 comp rx=0 tx=0).stop 2023-10-25T00:01:43.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 -- 172.21.15.107:0/599597922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 msgr2=0x7fab701395a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:43.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 0x7fab701395a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009ae0 tx=0x7fab60005800 comp rx=0 tx=0).stop 2023-10-25T00:01:43.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 -- 172.21.15.107:0/599597922 shutdown_connections 2023-10-25T00:01:43.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab70155bb0 0x7fab70156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab54065e60 0x7fab54068320 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab700c6e10 0x7fab700c7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 --2- 172.21.15.107:0/599597922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60003980 0x7fab701395a0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:43.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 -- 172.21.15.107:0/599597922 >> 172.21.15.107:0/599597922 conn(0x7fab70070f60 msgr2=0x7fab701526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:43.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 -- 172.21.15.107:0/599597922 shutdown_connections 2023-10-25T00:01:43.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:43.759+0000 7fab76b1e640 1 -- 172.21.15.107:0/599597922 wait complete. 2023-10-25T00:01:43.780 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:01:45.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:45.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cb58ae640 1 Processor -- start 2023-10-25T00:01:45.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cb58ae640 1 -- start start 2023-10-25T00:01:45.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cb58ae640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7cb00c8100 0x7f7cb00abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:45.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cb58ae640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cb006a4b0 0x7f7cb00ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:45.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cb58ae640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb0155680 con 0x7f7cb0152040 2023-10-25T00:01:45.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cb58ae640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb0154a70 con 0x7f7cb00c8100 2023-10-25T00:01:45.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cb58ae640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb0154d50 con 0x7f7cb006a4b0 2023-10-25T00:01:45.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cae7fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7cb00c8100 0x7f7cb00abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:45.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cae7fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7cb00c8100 0x7f7cb00abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:52842/0 (socket says 172.21.15.107:52842) 2023-10-25T00:01:45.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cae7fc640 1 -- 172.21.15.107:0/4292888991 learned_addr learned my addr 172.21.15.107:0/4292888991 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:45.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7caf7fe640 1 --2- 172.21.15.107:0/4292888991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cb006a4b0 0x7f7cb00ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:45.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cae7fc640 1 -- 172.21.15.107:0/4292888991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cb006a4b0 msgr2=0x7f7cb00ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:45.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cae7fc640 1 --2- 172.21.15.107:0/4292888991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cb006a4b0 0x7f7cb00ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:45.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cae7fc640 1 -- 172.21.15.107:0/4292888991 >> v1:172.21.15.110:6789/0 conn(0x7f7cb0152040 legacy=0x7f7cb00c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:45.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.911+0000 7f7cae7fc640 1 -- 172.21.15.107:0/4292888991 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb0155030 con 0x7f7cb00c8100 2023-10-25T00:01:45.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cae7fc640 1 --2- 172.21.15.107:0/4292888991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb00c8100 0x7f7cb00abd00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009870 tx=0x7f7ca401c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=622f91cf395ebfcb server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:45.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cadffb640 1 -- 172.21.15.107:0/4292888991 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40ab600 con 0x7f7cb00c8100 2023-10-25T00:01:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cadffb640 1 -- 172.21.15.107:0/4292888991 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca40a7050 con 0x7f7cb00c8100 2023-10-25T00:01:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cadffb640 1 -- 172.21.15.107:0/4292888991 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40ab980 con 0x7f7cb00c8100 2023-10-25T00:01:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4292888991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb00c8100 msgr2=0x7f7cb00abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cb58ae640 1 --2- 172.21.15.107:0/4292888991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb00c8100 0x7f7cb00abd00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009870 tx=0x7f7ca401c4b0 comp rx=0 tx=0).stop 2023-10-25T00:01:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4292888991 shutdown_connections 2023-10-25T00:01:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cb58ae640 1 --2- 172.21.15.107:0/4292888991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cb006a4b0 0x7f7cb00ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cb58ae640 1 --2- 172.21.15.107:0/4292888991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb00c8100 0x7f7cb00abd00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4292888991 >> 172.21.15.107:0/4292888991 conn(0x7f7cb00ca200 msgr2=0x7f7cb00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4292888991 shutdown_connections 2023-10-25T00:01:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4292888991 wait complete. 2023-10-25T00:01:45.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cb58ae640 1 Processor -- start 2023-10-25T00:01:45.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.915+0000 7f7cb58ae640 1 -- start start 2023-10-25T00:01:45.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cb58ae640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cb006a4b0 0x7f7cb0150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:45.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cb58ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb00c8100 0x7f7cb01f0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:45.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cb58ae640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb0152040 0x7f7cb014bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:45.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cb58ae640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb01f0f30 con 0x7f7cb00c8100 2023-10-25T00:01:45.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cb58ae640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb01519e0 con 0x7f7cb0152040 2023-10-25T00:01:45.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cb58ae640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb0151cc0 con 0x7f7cb006a4b0 2023-10-25T00:01:45.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cae7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb00c8100 0x7f7cb01f0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:45.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cae7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb00c8100 0x7f7cb01f0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37374/0 (socket says 172.21.15.107:37374) 2023-10-25T00:01:45.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cae7fc640 1 -- 172.21.15.107:0/4134443177 learned_addr learned my addr 172.21.15.107:0/4134443177 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:45.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7caf7fe640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb0152040 0x7f7cb014bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:45.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7caeffd640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cb006a4b0 0x7f7cb0150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:45.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cae7fc640 1 -- 172.21.15.107:0/4134443177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cb006a4b0 msgr2=0x7f7cb0150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:45.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cae7fc640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cb006a4b0 0x7f7cb0150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:45.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cae7fc640 1 -- 172.21.15.107:0/4134443177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb0152040 msgr2=0x7f7cb014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:45.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cae7fc640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb0152040 0x7f7cb014bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:45.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cae7fc640 1 -- 172.21.15.107:0/4134443177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca40094d0 con 0x7f7cb00c8100 2023-10-25T00:01:45.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7caeffd640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cb006a4b0 0x7f7cb0150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:01:45.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cae7fc640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb00c8100 0x7f7cb01f0250 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40ae040 tx=0x7f7ca4019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:45.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7c97fff640 1 -- 172.21.15.107:0/4134443177 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40b6470 con 0x7f7cb00c8100 2023-10-25T00:01:45.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7c97fff640 1 -- 172.21.15.107:0/4134443177 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca40b65d0 con 0x7f7cb00c8100 2023-10-25T00:01:45.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4134443177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb01f0c50 con 0x7f7cb00c8100 2023-10-25T00:01:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7c97fff640 1 -- 172.21.15.107:0/4134443177 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40b6980 con 0x7f7cb00c8100 2023-10-25T00:01:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4134443177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb00abd60 con 0x7f7cb00c8100 2023-10-25T00:01:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4134443177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb00746e0 con 0x7f7cb00c8100 2023-10-25T00:01:45.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7c97fff640 1 -- 172.21.15.107:0/4134443177 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7ca40b2070 con 0x7f7cb00c8100 2023-10-25T00:01:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7c97fff640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7c9c065e90 0x7f7c9c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7caeffd640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7c9c065e90 0x7f7c9c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7c97fff640 1 -- 172.21.15.107:0/4134443177 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7f7ca411e590 con 0x7f7cb00c8100 2023-10-25T00:01:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.919+0000 7f7caeffd640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7c9c065e90 0x7f7c9c068350 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0000f70 tx=0x7f7ca0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:45.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:45.923+0000 7f7c97fff640 1 -- 172.21.15.107:0/4134443177 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7ca40eb360 con 0x7f7cb00c8100 2023-10-25T00:01:46.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.111+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4134443177 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cb014b2d0 con 0x7f7cb00c8100 2023-10-25T00:01:46.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.111+0000 7f7c97fff640 1 -- 172.21.15.107:0/4134443177 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f7ca40b0020 con 0x7f7cb00c8100 2023-10-25T00:01:46.115 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:46.115 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:31.647132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:46.115 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 75 2023-10-25T00:01:46.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4134443177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7c9c065e90 msgr2=0x7f7c9c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:46.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7c9c065e90 0x7f7c9c068350 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0000f70 tx=0x7f7ca0009040 comp rx=0 tx=0).stop 2023-10-25T00:01:46.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4134443177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb00c8100 msgr2=0x7f7cb01f0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:46.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb00c8100 0x7f7cb01f0250 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40ae040 tx=0x7f7ca4019250 comp rx=0 tx=0).stop 2023-10-25T00:01:46.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4134443177 shutdown_connections 2023-10-25T00:01:46.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cb0152040 0x7f7cb014bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:46.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb00c8100 0x7f7cb01f0250 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:46.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7c9c065e90 0x7f7c9c068350 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:46.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 --2- 172.21.15.107:0/4134443177 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cb006a4b0 0x7f7cb0150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:46.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4134443177 >> 172.21.15.107:0/4134443177 conn(0x7f7cb00ca200 msgr2=0x7f7cb00ad070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:46.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4134443177 shutdown_connections 2023-10-25T00:01:46.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:46.115+0000 7f7cb58ae640 1 -- 172.21.15.107:0/4134443177 wait complete. 2023-10-25T00:01:46.133 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:01:48.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:48.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.263+0000 7f36d1b77640 1 Processor -- start 2023-10-25T00:01:48.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.263+0000 7f36d1b77640 1 -- start start 2023-10-25T00:01:48.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36d1b77640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc14fd40 0x7f36cc0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:48.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36d1b77640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36cc06ee40 0x7f36cc0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:48.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36d1b77640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36cc06f730 con 0x7f36cc0a9820 2023-10-25T00:01:48.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36d1b77640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36cc0aa750 con 0x7f36cc06ee40 2023-10-25T00:01:48.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36d1b77640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc156a40 con 0x7f36cc14fd40 2023-10-25T00:01:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36caffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f36cc0a9820 0x7f36cc0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48908/0 (socket says 172.21.15.107:48908) 2023-10-25T00:01:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36caffd640 1 -- 172.21.15.107:0/2050530194 learned_addr learned my addr 172.21.15.107:0/2050530194 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36cb7fe640 1 --2- 172.21.15.107:0/2050530194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc14fd40 0x7f36cc0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36cbfff640 1 --2- 172.21.15.107:0/2050530194 >> v2:172.21.15.110:3300/0 conn(0x7f36cc06ee40 0x7f36cc0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675581497 0 0) 0x7f36cc06f730 con 0x7f36cc0a9820 2023-10-25T00:01:48.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b4003530 con 0x7f36cc0a9820 2023-10-25T00:01:48.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3747223301 0 0) 0x7f36b4003530 con 0x7f36cc0a9820 2023-10-25T00:01:48.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc14fd40 msgr2=0x7f36cc0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:48.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 --2- 172.21.15.107:0/2050530194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc14fd40 0x7f36cc0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 >> v2:172.21.15.110:3300/0 conn(0x7f36cc06ee40 msgr2=0x7f36cc0aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 --2- 172.21.15.107:0/2050530194 >> v2:172.21.15.110:3300/0 conn(0x7f36cc06ee40 0x7f36cc0aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc156710 con 0x7f36cc0a9820 2023-10-25T00:01:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f36c0001680 con 0x7f36cc0a9820 2023-10-25T00:01:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 >> v1:172.21.15.107:6789/0 conn(0x7f36cc0a9820 legacy=0x7f36cc0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:48.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 --2- 172.21.15.107:0/2050530194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b4003980 0x7f36b4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:48.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 --2- 172.21.15.107:0/2050530194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b40047d0 0x7f36b4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:48.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 --2- 172.21.15.107:0/2050530194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc14fd40 0x7f36b4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:48.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc0aa750 con 0x7f36b4003980 2023-10-25T00:01:48.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc156a40 con 0x7f36b40047d0 2023-10-25T00:01:48.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36b400fe20 con 0x7f36cc14fd40 2023-10-25T00:01:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36cb7fe640 1 --2- 172.21.15.107:0/2050530194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b4003980 0x7f36b4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36cbfff640 1 --2- 172.21.15.107:0/2050530194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc14fd40 0x7f36b4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36cb7fe640 1 -- 172.21.15.107:0/2050530194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc14fd40 msgr2=0x7f36b4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36cb7fe640 1 --2- 172.21.15.107:0/2050530194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc14fd40 0x7f36b4005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36cb7fe640 1 -- 172.21.15.107:0/2050530194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b40047d0 msgr2=0x7f36b4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:48.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36cb7fe640 1 --2- 172.21.15.107:0/2050530194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b40047d0 0x7f36b4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36cb7fe640 1 -- 172.21.15.107:0/2050530194 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b4003530 con 0x7f36b4003980 2023-10-25T00:01:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36cb7fe640 1 --2- 172.21.15.107:0/2050530194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b4003980 0x7f36b4003da0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f36bc014f00 tx=0x7f36bc024ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ef5aae32988ea1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc0adc40 con 0x7f36b4003980 2023-10-25T00:01:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.267+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36bc0adda0 con 0x7f36b4003980 2023-10-25T00:01:48.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36ca7fc640 1 -- 172.21.15.107:0/2050530194 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc0b5680 con 0x7f36b4003980 2023-10-25T00:01:48.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 -- 172.21.15.107:0/2050530194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b4003980 msgr2=0x7f36b4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:48.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 --2- 172.21.15.107:0/2050530194 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b4003980 0x7f36b4003da0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f36bc014f00 tx=0x7f36bc024ec0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36cbfff640 1 -- 172.21.15.107:0/2050530194 reap_dead start 2023-10-25T00:01:48.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 -- 172.21.15.107:0/2050530194 shutdown_connections 2023-10-25T00:01:48.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 -- 172.21.15.107:0/2050530194 >> 172.21.15.107:0/2050530194 conn(0x7f36cc0c2950 msgr2=0x7f36cc0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 -- 172.21.15.107:0/2050530194 shutdown_connections 2023-10-25T00:01:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 -- 172.21.15.107:0/2050530194 wait complete. 2023-10-25T00:01:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 Processor -- start 2023-10-25T00:01:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 -- start start 2023-10-25T00:01:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b4003980 0x7f36cc0c34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc1494e0 0x7f36cc1e02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36cc1eca60 0x7f36cc148260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc149370 con 0x7f36cc1eca60 2023-10-25T00:01:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc1ddcf0 con 0x7f36b4003980 2023-10-25T00:01:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc1ed3b0 con 0x7f36cc1494e0 2023-10-25T00:01:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36cc1eca60 0x7f36cc148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36cbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36cc1eca60 0x7f36cc148260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37394/0 (socket says 172.21.15.107:37394) 2023-10-25T00:01:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36cbfff640 1 -- 172.21.15.107:0/3524445929 learned_addr learned my addr 172.21.15.107:0/3524445929 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36caffd640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc1494e0 0x7f36cc1e02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36cb7fe640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b4003980 0x7f36cc0c34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36cbfff640 1 -- 172.21.15.107:0/3524445929 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc1494e0 msgr2=0x7f36cc1e02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36cbfff640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc1494e0 0x7f36cc1e02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36cbfff640 1 -- 172.21.15.107:0/3524445929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b4003980 msgr2=0x7f36cc0c34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36cbfff640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b4003980 0x7f36cc0c34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36cbfff640 1 -- 172.21.15.107:0/3524445929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc1ddf90 con 0x7f36cc1eca60 2023-10-25T00:01:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36cbfff640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36cc1eca60 0x7f36cc148260 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f36cc06d200 tx=0x7f36c400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36b3fff640 1 -- 172.21.15.107:0/3524445929 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4017400 con 0x7f36cc1eca60 2023-10-25T00:01:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36b3fff640 1 -- 172.21.15.107:0/3524445929 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c4011040 con 0x7f36cc1eca60 2023-10-25T00:01:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 -- 172.21.15.107:0/3524445929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36cc1ed690 con 0x7f36cc1eca60 2023-10-25T00:01:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36b3fff640 1 -- 172.21.15.107:0/3524445929 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c40177c0 con 0x7f36cc1eca60 2023-10-25T00:01:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.271+0000 7f36d1b77640 1 -- 172.21.15.107:0/3524445929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36cc148fb0 con 0x7f36cc1eca60 2023-10-25T00:01:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.275+0000 7f36d1b77640 1 -- 172.21.15.107:0/3524445929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36cc0c24a0 con 0x7f36cc1eca60 2023-10-25T00:01:48.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.275+0000 7f36b3fff640 1 -- 172.21.15.107:0/3524445929 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f36c4017920 con 0x7f36cc1eca60 2023-10-25T00:01:48.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.275+0000 7f36b3fff640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36b8065e90 0x7f36b8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:48.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.275+0000 7f36b3fff640 1 -- 172.21.15.107:0/3524445929 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7f36c4084ba0 con 0x7f36cc1eca60 2023-10-25T00:01:48.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.275+0000 7f36cb7fe640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36b8065e90 0x7f36b8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:48.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.275+0000 7f36cb7fe640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36b8065e90 0x7f36b8068350 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f36bc0b7000 tx=0x7f36bc0046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:48.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.279+0000 7f36b3fff640 1 -- 172.21.15.107:0/3524445929 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f36c4051a80 con 0x7f36cc1eca60 2023-10-25T00:01:48.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.463+0000 7f36d1b77640 1 -- 172.21.15.107:0/3524445929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36cc0746e0 con 0x7f36cc1eca60 2023-10-25T00:01:48.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.463+0000 7f36b3fff640 1 -- 172.21.15.107:0/3524445929 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f36c401c070 con 0x7f36cc1eca60 2023-10-25T00:01:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:48.468 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:31.647132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:48.468 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 75 2023-10-25T00:01:48.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 -- 172.21.15.107:0/3524445929 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36b8065e90 msgr2=0x7f36b8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:48.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36b8065e90 0x7f36b8068350 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f36bc0b7000 tx=0x7f36bc0046d0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 -- 172.21.15.107:0/3524445929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36cc1eca60 msgr2=0x7f36cc148260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:48.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36cc1eca60 0x7f36cc148260 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f36cc06d200 tx=0x7f36c400f4a0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 -- 172.21.15.107:0/3524445929 shutdown_connections 2023-10-25T00:01:48.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36cc1eca60 0x7f36cc148260 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36b8065e90 0x7f36b8068350 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36cc1494e0 0x7f36cc1e02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 --2- 172.21.15.107:0/3524445929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36b4003980 0x7f36cc0c34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:48.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 -- 172.21.15.107:0/3524445929 >> 172.21.15.107:0/3524445929 conn(0x7f36cc0c2950 msgr2=0x7f36cc14d750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:48.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 -- 172.21.15.107:0/3524445929 shutdown_connections 2023-10-25T00:01:48.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:48.467+0000 7f36d1b77640 1 -- 172.21.15.107:0/3524445929 wait complete. 2023-10-25T00:01:48.484 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:01:50.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:50.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8ee6a2640 1 Processor -- start 2023-10-25T00:01:50.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8ee6a2640 1 -- start start 2023-10-25T00:01:50.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8ee6a2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe8e814b600 0x7fe8e8151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:50.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8ee6a2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe8e806af70 0x7fe8e806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:50.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8ee6a2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e80c1860 con 0x7fe8e806a230 2023-10-25T00:01:50.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8ee6a2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e8154d50 con 0x7fe8e806af70 2023-10-25T00:01:50.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8ee6a2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e81524f0 con 0x7fe8e814b600 2023-10-25T00:01:50.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8e77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe8e806af70 0x7fe8e806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:50.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8ecc3f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe8e806a230 0x7fe8e81551e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48912/0 (socket says 172.21.15.107:48912) 2023-10-25T00:01:50.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8e77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe8e806af70 0x7fe8e806c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37410/0 (socket says 172.21.15.107:37410) 2023-10-25T00:01:50.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8ecc3f640 1 -- 172.21.15.107:0/3386736363 learned_addr learned my addr 172.21.15.107:0/3386736363 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:50.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8e7fff640 1 --2- 172.21.15.107:0/3386736363 >> v2:172.21.15.110:3300/0 conn(0x7fe8e814b600 0x7fe8e8151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:50.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8e77fe640 1 -- 172.21.15.107:0/3386736363 >> v2:172.21.15.110:3300/0 conn(0x7fe8e814b600 msgr2=0x7fe8e8151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:50.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8e77fe640 1 --2- 172.21.15.107:0/3386736363 >> v2:172.21.15.110:3300/0 conn(0x7fe8e814b600 0x7fe8e8151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:50.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8e77fe640 1 -- 172.21.15.107:0/3386736363 >> v1:172.21.15.107:6789/0 conn(0x7fe8e806a230 legacy=0x7fe8e81551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:50.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8e77fe640 1 -- 172.21.15.107:0/3386736363 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e814b9e0 con 0x7fe8e806af70 2023-10-25T00:01:50.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8e77fe640 1 --2- 172.21.15.107:0/3386736363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8e806af70 0x7fe8e806c4c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe8e806f860 tx=0x7fe8dc01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c08adf7ab89a71a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:50.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8e6ffd640 1 -- 172.21.15.107:0/3386736363 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc0ac600 con 0x7fe8e806af70 2023-10-25T00:01:50.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8e6ffd640 1 -- 172.21.15.107:0/3386736363 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc0a8050 con 0x7fe8e806af70 2023-10-25T00:01:50.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.619+0000 7fe8e6ffd640 1 -- 172.21.15.107:0/3386736363 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc0ac980 con 0x7fe8e806af70 2023-10-25T00:01:50.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3386736363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8e806af70 msgr2=0x7fe8e806c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:50.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 --2- 172.21.15.107:0/3386736363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8e806af70 0x7fe8e806c4c0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fe8e806f860 tx=0x7fe8dc01c3a0 comp rx=0 tx=0).stop 2023-10-25T00:01:50.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3386736363 shutdown_connections 2023-10-25T00:01:50.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 --2- 172.21.15.107:0/3386736363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8e806af70 0x7fe8e806c4c0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:50.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 --2- 172.21.15.107:0/3386736363 >> v2:172.21.15.110:3300/0 conn(0x7fe8e814b600 0x7fe8e8151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:50.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3386736363 >> 172.21.15.107:0/3386736363 conn(0x7fe8e80c2950 msgr2=0x7fe8e80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:50.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3386736363 shutdown_connections 2023-10-25T00:01:50.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3386736363 wait complete. 2023-10-25T00:01:50.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 Processor -- start 2023-10-25T00:01:50.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 -- start start 2023-10-25T00:01:50.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8e806a230 0x7fe8e81ee8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:50.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8e806af70 0x7fe8e80aa830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:50.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8e814b600 0x7fe8e80c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:50.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e8143aa0 con 0x7fe8e806af70 2023-10-25T00:01:50.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e81eee30 con 0x7fe8e806a230 2023-10-25T00:01:50.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80c43c0 con 0x7fe8e814b600 2023-10-25T00:01:50.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8e806af70 0x7fe8e80aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:50.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8e806af70 0x7fe8e80aa830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37414/0 (socket says 172.21.15.107:37414) 2023-10-25T00:01:50.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e77fe640 1 -- 172.21.15.107:0/3719073866 learned_addr learned my addr 172.21.15.107:0/3719073866 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:50.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e7fff640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8e806a230 0x7fe8e81ee8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:50.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ecc3f640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8e814b600 0x7fe8e80c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:50.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e77fe640 1 -- 172.21.15.107:0/3719073866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8e814b600 msgr2=0x7fe8e80c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:50.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e77fe640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8e814b600 0x7fe8e80c4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:50.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e77fe640 1 -- 172.21.15.107:0/3719073866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8e806a230 msgr2=0x7fe8e81ee8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:50.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e77fe640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8e806a230 0x7fe8e81ee8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:50.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e77fe640 1 -- 172.21.15.107:0/3719073866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc009420 con 0x7fe8e806af70 2023-10-25T00:01:50.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e77fe640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8e806af70 0x7fe8e80aa830 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc018fb0 tx=0x7fe8dc01abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:50.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e4ff9640 1 -- 172.21.15.107:0/3719073866 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc0b7450 con 0x7fe8e806af70 2023-10-25T00:01:50.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e4ff9640 1 -- 172.21.15.107:0/3719073866 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc0b75b0 con 0x7fe8e806af70 2023-10-25T00:01:50.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3719073866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e81429d0 con 0x7fe8e806af70 2023-10-25T00:01:50.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8e4ff9640 1 -- 172.21.15.107:0/3719073866 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc0b7920 con 0x7fe8e806af70 2023-10-25T00:01:50.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.623+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3719073866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e8142f90 con 0x7fe8e806af70 2023-10-25T00:01:50.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.627+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3719073866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e80c4570 con 0x7fe8e806af70 2023-10-25T00:01:50.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.627+0000 7fe8e4ff9640 1 -- 172.21.15.107:0/3719073866 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe8dc0b3070 con 0x7fe8e806af70 2023-10-25T00:01:50.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.627+0000 7fe8e4ff9640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8d4065e60 0x7fe8d4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:50.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.627+0000 7fe8e7fff640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8d4065e60 0x7fe8d4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:50.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.627+0000 7fe8e4ff9640 1 -- 172.21.15.107:0/3719073866 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7fe8dc11e2b0 con 0x7fe8e806af70 2023-10-25T00:01:50.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.627+0000 7fe8e7fff640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8d4065e60 0x7fe8d4068320 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe8e814ef00 tx=0x7fe8e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:50.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.631+0000 7fe8e4ff9640 1 -- 172.21.15.107:0/3719073866 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe8e80c4570 con 0x7fe8e806af70 2023-10-25T00:01:50.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.831+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3719073866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8e80c4530 con 0x7fe8e806af70 2023-10-25T00:01:50.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.831+0000 7fe8e4ff9640 1 -- 172.21.15.107:0/3719073866 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fe8dc0eb1b0 con 0x7fe8e806af70 2023-10-25T00:01:50.834 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:50.834 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:31.647132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:50.835 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 75 2023-10-25T00:01:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3719073866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8d4065e60 msgr2=0x7fe8d4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8d4065e60 0x7fe8d4068320 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe8e814ef00 tx=0x7fe8e0008040 comp rx=0 tx=0).stop 2023-10-25T00:01:50.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3719073866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8e806af70 msgr2=0x7fe8e80aa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8e806af70 0x7fe8e80aa830 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc018fb0 tx=0x7fe8dc01abe0 comp rx=0 tx=0).stop 2023-10-25T00:01:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3719073866 shutdown_connections 2023-10-25T00:01:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe8d4065e60 0x7fe8d4068320 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8e814b600 0x7fe8e80c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:50.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8e806af70 0x7fe8e80aa830 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:50.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 --2- 172.21.15.107:0/3719073866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8e806a230 0x7fe8e81ee8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:50.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3719073866 >> 172.21.15.107:0/3719073866 conn(0x7fe8e80c2950 msgr2=0x7fe8e8150d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:50.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3719073866 shutdown_connections 2023-10-25T00:01:50.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:50.835+0000 7fe8ee6a2640 1 -- 172.21.15.107:0/3719073866 wait complete. 2023-10-25T00:01:50.853 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:01:52.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:52.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f68cd640 1 Processor -- start 2023-10-25T00:01:52.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f68cd640 1 -- start start 2023-10-25T00:01:52.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f68cd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc0f00c2990 0x7fc0f00aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:52.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f68cd640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f00a9aa0 con 0x7fc0f0152060 2023-10-25T00:01:52.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f68cd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f0155480 con 0x7fc0f00c20b0 2023-10-25T00:01:52.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f68cd640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f0155760 con 0x7fc0f00c2990 2023-10-25T00:01:52.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0effff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc0f0152060 0x7fc0f00a9f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53162/0 (socket says 172.21.15.107:53162) 2023-10-25T00:01:52.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0effff640 1 -- 172.21.15.107:0/1752195659 learned_addr learned my addr 172.21.15.107:0/1752195659 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:52.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0ef7fe640 1 --2- 172.21.15.107:0/1752195659 >> v2:172.21.15.110:3300/0 conn(0x7fc0f00c2990 0x7fc0f00aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:52.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475134695 0 0) 0x7fc0f00a9aa0 con 0x7fc0f0152060 2023-10-25T00:01:52.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d8003530 con 0x7fc0f0152060 2023-10-25T00:01:52.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 941877157 0 0) 0x7fc0d8003530 con 0x7fc0f0152060 2023-10-25T00:01:52.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 >> v2:172.21.15.110:3300/0 conn(0x7fc0f00c2990 msgr2=0x7fc0f00aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:52.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 --2- 172.21.15.107:0/1752195659 >> v2:172.21.15.110:3300/0 conn(0x7fc0f00c2990 0x7fc0f00aada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:52.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 >> v1:172.21.15.110:6789/0 conn(0x7fc0f00c20b0 legacy=0x7fc0f00c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:52.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f00a9450 con 0x7fc0f0152060 2023-10-25T00:01:52.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fc0e0001680 con 0x7fc0f0152060 2023-10-25T00:01:52.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 >> v1:172.21.15.107:6789/0 conn(0x7fc0f0152060 legacy=0x7fc0f00a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:52.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 --2- 172.21.15.107:0/1752195659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d8003980 0x7fc0d8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:52.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 --2- 172.21.15.107:0/1752195659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80047d0 0x7fc0d8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:52.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 --2- 172.21.15.107:0/1752195659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80052b0 0x7fc0d800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:52.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0155760 con 0x7fc0d80052b0 2023-10-25T00:01:52.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d800c180 con 0x7fc0d80047d0 2023-10-25T00:01:52.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d800c340 con 0x7fc0d8003980 2023-10-25T00:01:52.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f4e6a640 1 --2- 172.21.15.107:0/1752195659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80052b0 0x7fc0d800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:52.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0effff640 1 --2- 172.21.15.107:0/1752195659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d8003980 0x7fc0d8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0ef7fe640 1 --2- 172.21.15.107:0/1752195659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80047d0 0x7fc0d8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f4e6a640 1 -- 172.21.15.107:0/1752195659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d8003980 msgr2=0x7fc0d8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f4e6a640 1 --2- 172.21.15.107:0/1752195659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d8003980 0x7fc0d8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f4e6a640 1 -- 172.21.15.107:0/1752195659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80047d0 msgr2=0x7fc0d8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:52.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f4e6a640 1 --2- 172.21.15.107:0/1752195659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0d80047d0 0x7fc0d8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:52.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.983+0000 7fc0f4e6a640 1 -- 172.21.15.107:0/1752195659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d8003530 con 0x7fc0d80052b0 2023-10-25T00:01:52.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f4e6a640 1 -- 172.21.15.107:0/1752195659 reap_dead start 2023-10-25T00:01:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f4e6a640 1 --2- 172.21.15.107:0/1752195659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80052b0 0x7fc0d800b990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8001e40 tx=0x7fc0e801c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be22db6ccc5f04bb server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8003b80 con 0x7fc0d80052b0 2023-10-25T00:01:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e8003ce0 con 0x7fc0d80052b0 2023-10-25T00:01:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0eeffd640 1 -- 172.21.15.107:0/1752195659 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8005310 con 0x7fc0d80052b0 2023-10-25T00:01:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 -- 172.21.15.107:0/1752195659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80052b0 msgr2=0x7fc0d800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 --2- 172.21.15.107:0/1752195659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80052b0 0x7fc0d800b990 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8001e40 tx=0x7fc0e801c710 comp rx=0 tx=0).stop 2023-10-25T00:01:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 -- 172.21.15.107:0/1752195659 shutdown_connections 2023-10-25T00:01:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 --2- 172.21.15.107:0/1752195659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0d80052b0 0x7fc0d800b990 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 -- 172.21.15.107:0/1752195659 >> 172.21.15.107:0/1752195659 conn(0x7fc0f00c4a90 msgr2=0x7fc0f00c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:52.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 -- 172.21.15.107:0/1752195659 shutdown_connections 2023-10-25T00:01:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 -- 172.21.15.107:0/1752195659 wait complete. 2023-10-25T00:01:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 Processor -- start 2023-10-25T00:01:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 -- start start 2023-10-25T00:01:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d80052b0 0x7fc0f01efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f014f890 0x7fc0f00c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:52.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0f00c6450 0x7fc0f0144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f014f710 con 0x7fc0f014f890 2023-10-25T00:01:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0150590 con 0x7fc0f00c6450 2023-10-25T00:01:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f68cd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f0150260 con 0x7fc0d80052b0 2023-10-25T00:01:52.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f014f890 0x7fc0f00c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0ef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f014f890 0x7fc0f00c5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49884/0 (socket says 172.21.15.107:49884) 2023-10-25T00:01:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0ef7fe640 1 -- 172.21.15.107:0/2401440909 learned_addr learned my addr 172.21.15.107:0/2401440909 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0effff640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d80052b0 0x7fc0f01efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.987+0000 7fc0f4e6a640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0f00c6450 0x7fc0f0144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ef7fe640 1 -- 172.21.15.107:0/2401440909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d80052b0 msgr2=0x7fc0f01efdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:53.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ef7fe640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d80052b0 0x7fc0f01efdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ef7fe640 1 -- 172.21.15.107:0/2401440909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0f00c6450 msgr2=0x7fc0f0144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ef7fe640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0f00c6450 0x7fc0f0144770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ef7fe640 1 -- 172.21.15.107:0/2401440909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f01f0740 con 0x7fc0f014f890 2023-10-25T00:01:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0f4e6a640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0f00c6450 0x7fc0f0144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:01:53.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0effff640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d80052b0 0x7fc0f01efdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:01:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ef7fe640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f014f890 0x7fc0f00c5f10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4011540 tx=0x7fc0e4011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ecff9640 1 -- 172.21.15.107:0/2401440909 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4015930 con 0x7fc0f014f890 2023-10-25T00:01:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ecff9640 1 -- 172.21.15.107:0/2401440909 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e4015a90 con 0x7fc0f014f890 2023-10-25T00:01:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0f68cd640 1 -- 172.21.15.107:0/2401440909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f01516b0 con 0x7fc0f014f890 2023-10-25T00:01:53.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ecff9640 1 -- 172.21.15.107:0/2401440909 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e4015e40 con 0x7fc0f014f890 2023-10-25T00:01:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0f68cd640 1 -- 172.21.15.107:0/2401440909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f014acb0 con 0x7fc0f014f890 2023-10-25T00:01:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0f68cd640 1 -- 172.21.15.107:0/2401440909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f00746e0 con 0x7fc0f014f890 2023-10-25T00:01:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ecff9640 1 -- 172.21.15.107:0/2401440909 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc0e400f460 con 0x7fc0f014f890 2023-10-25T00:01:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ecff9640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0dc065e90 0x7fc0dc068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:53.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0ecff9640 1 -- 172.21.15.107:0/2401440909 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7fc0e4084800 con 0x7fc0f014f890 2023-10-25T00:01:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0effff640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0dc065e90 0x7fc0dc068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.991+0000 7fc0effff640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0dc065e90 0x7fc0dc068350 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc0f00c7f70 tx=0x7fc0e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:53.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:52.995+0000 7fc0ecff9640 1 -- 172.21.15.107:0/2401440909 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc0e4051660 con 0x7fc0f014f890 2023-10-25T00:01:53.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.183+0000 7fc0f68cd640 1 -- 172.21.15.107:0/2401440909 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0f00c1e90 con 0x7fc0f014f890 2023-10-25T00:01:53.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.183+0000 7fc0ecff9640 1 -- 172.21.15.107:0/2401440909 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5774 (secure 0 0 0) 0x7fc0e4054c80 con 0x7fc0f014f890 2023-10-25T00:01:53.186 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:53.187 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:31.647132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:53.187 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 75 2023-10-25T00:01:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 -- 172.21.15.107:0/2401440909 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0dc065e90 msgr2=0x7fc0dc068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0dc065e90 0x7fc0dc068350 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc0f00c7f70 tx=0x7fc0e0009040 comp rx=0 tx=0).stop 2023-10-25T00:01:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 -- 172.21.15.107:0/2401440909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f014f890 msgr2=0x7fc0f00c5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f014f890 0x7fc0f00c5f10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc0e4011540 tx=0x7fc0e4011a00 comp rx=0 tx=0).stop 2023-10-25T00:01:53.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 -- 172.21.15.107:0/2401440909 shutdown_connections 2023-10-25T00:01:53.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0f00c6450 0x7fc0f0144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:53.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f014f890 0x7fc0f00c5f10 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:53.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc0dc065e90 0x7fc0dc068350 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:53.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 --2- 172.21.15.107:0/2401440909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0d80052b0 0x7fc0f01efdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:53.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 -- 172.21.15.107:0/2401440909 >> 172.21.15.107:0/2401440909 conn(0x7fc0f00c4a90 msgr2=0x7fc0f00c2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:53.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 -- 172.21.15.107:0/2401440909 shutdown_connections 2023-10-25T00:01:53.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:53.187+0000 7fc0f68cd640 1 -- 172.21.15.107:0/2401440909 wait complete. 2023-10-25T00:01:53.205 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:01:55.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:55.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.335+0000 7f9d5173c640 1 Processor -- start 2023-10-25T00:01:55.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.335+0000 7f9d5173c640 1 -- start start 2023-10-25T00:01:55.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d5173c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9d4c0a9820 0x7f9d4c0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:55.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d5173c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9d4c06ee40 0x7f9d4c0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d5173c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c06f730 con 0x7f9d4c14fd40 2023-10-25T00:01:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d5173c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c0aa750 con 0x7f9d4c06ee40 2023-10-25T00:01:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d5173c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c156a40 con 0x7f9d4c0a9820 2023-10-25T00:01:55.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d4affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9d4c14fd40 0x7f9d4c0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53174/0 (socket says 172.21.15.107:53174) 2023-10-25T00:01:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d4affd640 1 -- 172.21.15.107:0/1658138547 learned_addr learned my addr 172.21.15.107:0/1658138547 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d4b7fe640 1 --2- 172.21.15.107:0/1658138547 >> v2:172.21.15.107:3300/0 conn(0x7f9d4c06ee40 0x7f9d4c0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d4a7fc640 1 --2- 172.21.15.107:0/1658138547 >> v2:172.21.15.110:3300/0 conn(0x7f9d4c0a9820 0x7f9d4c0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d49ffb640 1 -- 172.21.15.107:0/1658138547 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489585203 0 0) 0x7f9d4c06f730 con 0x7f9d4c14fd40 2023-10-25T00:01:55.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d49ffb640 1 -- 172.21.15.107:0/1658138547 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d34003530 con 0x7f9d4c14fd40 2023-10-25T00:01:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d4b7fe640 1 -- 172.21.15.107:0/1658138547 >> v2:172.21.15.110:3300/0 conn(0x7f9d4c0a9820 msgr2=0x7f9d4c0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d4b7fe640 1 --2- 172.21.15.107:0/1658138547 >> v2:172.21.15.110:3300/0 conn(0x7f9d4c0a9820 0x7f9d4c0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d4b7fe640 1 -- 172.21.15.107:0/1658138547 >> v1:172.21.15.107:6789/0 conn(0x7f9d4c14fd40 legacy=0x7f9d4c0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:55.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d4b7fe640 1 -- 172.21.15.107:0/1658138547 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c156710 con 0x7f9d4c06ee40 2023-10-25T00:01:55.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.339+0000 7f9d4a7fc640 1 --2- 172.21.15.107:0/1658138547 >> v2:172.21.15.110:3300/0 conn(0x7f9d4c0a9820 0x7f9d4c0ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:01:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d4b7fe640 1 --2- 172.21.15.107:0/1658138547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d4c06ee40 0x7f9d4c0aa210 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c06d2f0 tx=0x7f9d4401c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b0d8451bcd3a86 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d49ffb640 1 -- 172.21.15.107:0/1658138547 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440a5820 con 0x7f9d4c06ee40 2023-10-25T00:01:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d49ffb640 1 -- 172.21.15.107:0/1658138547 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d440a5980 con 0x7f9d4c06ee40 2023-10-25T00:01:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d49ffb640 1 -- 172.21.15.107:0/1658138547 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440a5cf0 con 0x7f9d4c06ee40 2023-10-25T00:01:55.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d5173c640 1 -- 172.21.15.107:0/1658138547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d4c06ee40 msgr2=0x7f9d4c0aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:55.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d5173c640 1 --2- 172.21.15.107:0/1658138547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d4c06ee40 0x7f9d4c0aa210 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c06d2f0 tx=0x7f9d4401c7c0 comp rx=0 tx=0).stop 2023-10-25T00:01:55.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d5173c640 1 -- 172.21.15.107:0/1658138547 shutdown_connections 2023-10-25T00:01:55.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d5173c640 1 --2- 172.21.15.107:0/1658138547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d4c06ee40 0x7f9d4c0aa210 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:55.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d5173c640 1 --2- 172.21.15.107:0/1658138547 >> v2:172.21.15.110:3300/0 conn(0x7f9d4c0a9820 0x7f9d4c0ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:55.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d5173c640 1 -- 172.21.15.107:0/1658138547 >> 172.21.15.107:0/1658138547 conn(0x7f9d4c0c2950 msgr2=0x7f9d4c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:55.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d5173c640 1 -- 172.21.15.107:0/1658138547 shutdown_connections 2023-10-25T00:01:55.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.343+0000 7f9d5173c640 1 -- 172.21.15.107:0/1658138547 wait complete. 2023-10-25T00:01:55.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d5173c640 1 Processor -- start 2023-10-25T00:01:55.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d5173c640 1 -- start start 2023-10-25T00:01:55.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d5173c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d4c06ee40 0x7f9d4c0bfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:55.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d5173c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d4c14fd40 0x7f9d4c146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:55.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d5173c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d4c15bc40 0x7f9d4c1472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:55.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d5173c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c15b360 con 0x7f9d4c15bc40 2023-10-25T00:01:55.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d5173c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c1596e0 con 0x7f9d4c06ee40 2023-10-25T00:01:55.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d5173c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c1ef230 con 0x7f9d4c14fd40 2023-10-25T00:01:55.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d4b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d4c15bc40 0x7f9d4c1472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:55.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d4b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d4c15bc40 0x7f9d4c1472e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49908/0 (socket says 172.21.15.107:49908) 2023-10-25T00:01:55.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d4b7fe640 1 -- 172.21.15.107:0/3732929397 learned_addr learned my addr 172.21.15.107:0/3732929397 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:55.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d4affd640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d4c06ee40 0x7f9d4c0bfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:55.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d4a7fc640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d4c14fd40 0x7f9d4c146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:55.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d4b7fe640 1 -- 172.21.15.107:0/3732929397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d4c14fd40 msgr2=0x7f9d4c146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:55.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d4b7fe640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d4c14fd40 0x7f9d4c146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:55.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d4b7fe640 1 -- 172.21.15.107:0/3732929397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d4c06ee40 msgr2=0x7f9d4c0bfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:55.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d4b7fe640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d4c06ee40 0x7f9d4c0bfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:55.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d4b7fe640 1 -- 172.21.15.107:0/3732929397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d44009a70 con 0x7f9d4c15bc40 2023-10-25T00:01:55.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d4b7fe640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d4c15bc40 0x7f9d4c1472e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9d440193d0 tx=0x7f9d44019560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:55.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d337fe640 1 -- 172.21.15.107:0/3732929397 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440ad640 con 0x7f9d4c15bc40 2023-10-25T00:01:55.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d337fe640 1 -- 172.21.15.107:0/3732929397 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d440ad7a0 con 0x7f9d4c15bc40 2023-10-25T00:01:55.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d5173c640 1 -- 172.21.15.107:0/3732929397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4c15aac0 con 0x7f9d4c15bc40 2023-10-25T00:01:55.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d337fe640 1 -- 172.21.15.107:0/3732929397 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440adb30 con 0x7f9d4c15bc40 2023-10-25T00:01:55.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d5173c640 1 -- 172.21.15.107:0/3732929397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4c15b080 con 0x7f9d4c15bc40 2023-10-25T00:01:55.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.347+0000 7f9d5173c640 1 -- 172.21.15.107:0/3732929397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4c070840 con 0x7f9d4c15bc40 2023-10-25T00:01:55.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.351+0000 7f9d337fe640 1 -- 172.21.15.107:0/3732929397 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9d440b4070 con 0x7f9d4c15bc40 2023-10-25T00:01:55.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.351+0000 7f9d337fe640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d38065e60 0x7f9d38068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:55.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.351+0000 7f9d337fe640 1 -- 172.21.15.107:0/3732929397 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7f9d4411e980 con 0x7f9d4c15bc40 2023-10-25T00:01:55.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.351+0000 7f9d4affd640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d38065e60 0x7f9d38068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:55.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.351+0000 7f9d4affd640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d38065e60 0x7f9d38068320 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c005890 tx=0x7f9d3c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:55.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.355+0000 7f9d337fe640 1 -- 172.21.15.107:0/3732929397 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9d440eb880 con 0x7f9d4c15bc40 2023-10-25T00:01:55.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d5173c640 1 -- 172.21.15.107:0/3732929397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d4c159850 con 0x7f9d4c15bc40 2023-10-25T00:01:55.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d337fe640 1 -- 172.21.15.107:0/3732929397 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f9d440b2090 con 0x7f9d4c15bc40 2023-10-25T00:01:55.545 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:55.545 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:31.647132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:55.545 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 75 2023-10-25T00:01:55.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d5173c640 1 -- 172.21.15.107:0/3732929397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d38065e60 msgr2=0x7f9d38068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:55.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d5173c640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d38065e60 0x7f9d38068320 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c005890 tx=0x7f9d3c009040 comp rx=0 tx=0).stop 2023-10-25T00:01:55.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d5173c640 1 -- 172.21.15.107:0/3732929397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d4c15bc40 msgr2=0x7f9d4c1472e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:55.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d5173c640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d4c15bc40 0x7f9d4c1472e0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9d440193d0 tx=0x7f9d44019560 comp rx=0 tx=0).stop 2023-10-25T00:01:55.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d5173c640 1 -- 172.21.15.107:0/3732929397 shutdown_connections 2023-10-25T00:01:55.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d5173c640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d4c15bc40 0x7f9d4c1472e0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:55.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d5173c640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9d38065e60 0x7f9d38068320 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:55.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d5173c640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9d4c14fd40 0x7f9d4c146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:55.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d5173c640 1 --2- 172.21.15.107:0/3732929397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d4c06ee40 0x7f9d4c0bfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:55.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.543+0000 7f9d5173c640 1 -- 172.21.15.107:0/3732929397 >> 172.21.15.107:0/3732929397 conn(0x7f9d4c0c2950 msgr2=0x7f9d4c15c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:55.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.547+0000 7f9d5173c640 1 -- 172.21.15.107:0/3732929397 shutdown_connections 2023-10-25T00:01:55.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:55.547+0000 7f9d5173c640 1 -- 172.21.15.107:0/3732929397 wait complete. 2023-10-25T00:01:55.564 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:01:57.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:01:57.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.691+0000 7f9311a69640 1 Processor -- start 2023-10-25T00:01:57.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.691+0000 7f9311a69640 1 -- start start 2023-10-25T00:01:57.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f9311a69640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f930c151960 0x7f930c071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:57.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f9311a69640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c150bc0 con 0x7f930c071220 2023-10-25T00:01:57.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f9311a69640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c06c510 con 0x7f930c0c24c0 2023-10-25T00:01:57.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f9311a69640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930c06c7f0 con 0x7f930c151960 2023-10-25T00:01:57.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f930c071220 0x7f930c06bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53184/0 (socket says 172.21.15.107:53184) 2023-10-25T00:01:57.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930b7fe640 1 -- 172.21.15.107:0/1680206274 learned_addr learned my addr 172.21.15.107:0/1680206274 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:01:57.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930affd640 1 --2- 172.21.15.107:0/1680206274 >> v2:172.21.15.110:3300/0 conn(0x7f930c151960 0x7f930c071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:57.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199858192 0 0) 0x7f930c150bc0 con 0x7f930c071220 2023-10-25T00:01:57.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f4003530 con 0x7f930c071220 2023-10-25T00:01:57.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3984233991 0 0) 0x7f92f4003530 con 0x7f930c071220 2023-10-25T00:01:57.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 >> v2:172.21.15.110:3300/0 conn(0x7f930c151960 msgr2=0x7f930c071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:57.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 --2- 172.21.15.107:0/1680206274 >> v2:172.21.15.110:3300/0 conn(0x7f930c151960 0x7f930c071a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:57.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 >> v1:172.21.15.110:6789/0 conn(0x7f930c0c24c0 legacy=0x7f930c06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:57.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c150e80 con 0x7f930c071220 2023-10-25T00:01:57.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f9304001a00 con 0x7f930c071220 2023-10-25T00:01:57.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 >> v1:172.21.15.107:6789/0 conn(0x7f930c071220 legacy=0x7f930c06bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:57.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 --2- 172.21.15.107:0/1680206274 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92f4003980 0x7f92f4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:57.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 --2- 172.21.15.107:0/1680206274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92f40047d0 0x7f92f4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:57.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 --2- 172.21.15.107:0/1680206274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92f4005190 0x7f92f4005960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:57.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c06c7f0 con 0x7f92f4005190 2023-10-25T00:01:57.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4006100 con 0x7f92f40047d0 2023-10-25T00:01:57.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f40062c0 con 0x7f92f4003980 2023-10-25T00:01:57.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930bfff640 1 --2- 172.21.15.107:0/1680206274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92f4005190 0x7f92f4005960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:57.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930affd640 1 --2- 172.21.15.107:0/1680206274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92f40047d0 0x7f92f4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:57.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930b7fe640 1 --2- 172.21.15.107:0/1680206274 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92f4003980 0x7f92f4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:57.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930bfff640 1 -- 172.21.15.107:0/1680206274 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92f4003980 msgr2=0x7f92f4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:57.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930bfff640 1 --2- 172.21.15.107:0/1680206274 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92f4003980 0x7f92f4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:57.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930bfff640 1 -- 172.21.15.107:0/1680206274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92f40047d0 msgr2=0x7f92f4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:01:57.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930bfff640 1 --2- 172.21.15.107:0/1680206274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92f40047d0 0x7f92f4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:57.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930bfff640 1 -- 172.21.15.107:0/1680206274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f4003530 con 0x7f92f4005190 2023-10-25T00:01:57.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.695+0000 7f930bfff640 1 -- 172.21.15.107:0/1680206274 reap_dead start 2023-10-25T00:01:57.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930bfff640 1 --2- 172.21.15.107:0/1680206274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92f4005190 0x7f92f4005960 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f92fc001780 tx=0x7f92fc01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0de14ffa4f3828e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:57.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc0052d0 con 0x7f92f4005190 2023-10-25T00:01:57.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc0a7050 con 0x7f92f4005190 2023-10-25T00:01:57.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930a7fc640 1 -- 172.21.15.107:0/1680206274 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc003840 con 0x7f92f4005190 2023-10-25T00:01:57.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 -- 172.21.15.107:0/1680206274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92f4005190 msgr2=0x7f92f4005960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:57.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 --2- 172.21.15.107:0/1680206274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92f4005190 0x7f92f4005960 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f92fc001780 tx=0x7f92fc01c400 comp rx=0 tx=0).stop 2023-10-25T00:01:57.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 -- 172.21.15.107:0/1680206274 shutdown_connections 2023-10-25T00:01:57.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 --2- 172.21.15.107:0/1680206274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92f4005190 0x7f92f4005960 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:57.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 -- 172.21.15.107:0/1680206274 >> 172.21.15.107:0/1680206274 conn(0x7f930c0c2950 msgr2=0x7f930c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:57.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 -- 172.21.15.107:0/1680206274 shutdown_connections 2023-10-25T00:01:57.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 -- 172.21.15.107:0/1680206274 wait complete. 2023-10-25T00:01:57.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 Processor -- start 2023-10-25T00:01:57.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 -- start start 2023-10-25T00:01:57.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92f4005190 0x7f930c1e9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:57.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c1484e0 0x7f930c13d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:57.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f930c1e95b0 0x7f930c13da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:57.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0baf20 con 0x7f930c1484e0 2023-10-25T00:01:57.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c151fc0 con 0x7f930c1e95b0 2023-10-25T00:01:57.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c1e9b00 con 0x7f92f4005190 2023-10-25T00:01:57.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c1484e0 0x7f930c13d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:57.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c1484e0 0x7f930c13d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49916/0 (socket says 172.21.15.107:49916) 2023-10-25T00:01:57.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930affd640 1 -- 172.21.15.107:0/4058885166 learned_addr learned my addr 172.21.15.107:0/4058885166 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:01:57.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930b7fe640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92f4005190 0x7f930c1e9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:57.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930bfff640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f930c1e95b0 0x7f930c13da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:57.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930affd640 1 -- 172.21.15.107:0/4058885166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92f4005190 msgr2=0x7f930c1e9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:57.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930affd640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92f4005190 0x7f930c1e9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:57.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930affd640 1 -- 172.21.15.107:0/4058885166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f930c1e95b0 msgr2=0x7f930c13da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:57.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930affd640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f930c1e95b0 0x7f930c13da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:57.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930affd640 1 -- 172.21.15.107:0/4058885166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c152280 con 0x7f930c1484e0 2023-10-25T00:01:57.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f930affd640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c1484e0 0x7f930c13d4e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9300011560 tx=0x7f9300011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:57.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f92f3fff640 1 -- 172.21.15.107:0/4058885166 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9300015930 con 0x7f930c1484e0 2023-10-25T00:01:57.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f92f3fff640 1 -- 172.21.15.107:0/4058885166 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9300015a90 con 0x7f930c1484e0 2023-10-25T00:01:57.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 -- 172.21.15.107:0/4058885166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930c13df60 con 0x7f930c1484e0 2023-10-25T00:01:57.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f92f3fff640 1 -- 172.21.15.107:0/4058885166 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9300015e60 con 0x7f930c1484e0 2023-10-25T00:01:57.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.699+0000 7f9311a69640 1 -- 172.21.15.107:0/4058885166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f930c148a30 con 0x7f930c1484e0 2023-10-25T00:01:57.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.703+0000 7f9311a69640 1 -- 172.21.15.107:0/4058885166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930c148d10 con 0x7f930c1484e0 2023-10-25T00:01:57.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.703+0000 7f92f3fff640 1 -- 172.21.15.107:0/4058885166 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f930000f460 con 0x7f930c1484e0 2023-10-25T00:01:57.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.703+0000 7f92f3fff640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f92f8065e90 0x7f92f8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:01:57.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.703+0000 7f92f3fff640 1 -- 172.21.15.107:0/4058885166 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 8079+0+0 (secure 0 0 0) 0x7f93000846c0 con 0x7f930c1484e0 2023-10-25T00:01:57.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.703+0000 7f930b7fe640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f92f8065e90 0x7f92f8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:01:57.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.703+0000 7f930b7fe640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f92f8065e90 0x7f92f8068350 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f93040020c0 tx=0x7f9304002050 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:01:57.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.707+0000 7f92f3fff640 1 -- 172.21.15.107:0/4058885166 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f93000515e0 con 0x7f930c1484e0 2023-10-25T00:01:57.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.899+0000 7f9311a69640 1 -- 172.21.15.107:0/4058885166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f930c152410 con 0x7f930c1484e0 2023-10-25T00:01:57.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f92f3fff640 1 -- 172.21.15.107:0/4058885166 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5774 (secure 0 0 0) 0x7f9300054c00 con 0x7f930c1484e0 2023-10-25T00:01:57.904 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:01:57.904 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6396,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:31.647132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6303},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6303":{"gid":6303,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6837/3103569740","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3103569740},{"type":"v1","addr":"172.21.15.107:6837","nonce":3103569740}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:01:57.904 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 75 2023-10-25T00:01:57.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 -- 172.21.15.107:0/4058885166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f92f8065e90 msgr2=0x7f92f8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:57.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f92f8065e90 0x7f92f8068350 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f93040020c0 tx=0x7f9304002050 comp rx=0 tx=0).stop 2023-10-25T00:01:57.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 -- 172.21.15.107:0/4058885166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c1484e0 msgr2=0x7f930c13d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:01:57.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c1484e0 0x7f930c13d4e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9300011560 tx=0x7f9300011a20 comp rx=0 tx=0).stop 2023-10-25T00:01:57.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 -- 172.21.15.107:0/4058885166 shutdown_connections 2023-10-25T00:01:57.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f930c1e95b0 0x7f930c13da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:57.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f930c1484e0 0x7f930c13d4e0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:57.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f92f8065e90 0x7f92f8068350 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:57.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 --2- 172.21.15.107:0/4058885166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f92f4005190 0x7f930c1e9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:01:57.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 -- 172.21.15.107:0/4058885166 >> 172.21.15.107:0/4058885166 conn(0x7f930c0c2950 msgr2=0x7f930c14c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:01:57.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 -- 172.21.15.107:0/4058885166 shutdown_connections 2023-10-25T00:01:57.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:01:57.903+0000 7f9311a69640 1 -- 172.21.15.107:0/4058885166 wait complete. 2023-10-25T00:01:57.923 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:01:59.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:00.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b235a1640 1 Processor -- start 2023-10-25T00:02:00.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b235a1640 1 -- start start 2023-10-25T00:02:00.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b235a1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b1c069d40 0x7f5b1c158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:00.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b235a1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c06cea0 con 0x7f5b1c06a120 2023-10-25T00:02:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b235a1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c1584e0 con 0x7f5b1c065bc0 2023-10-25T00:02:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b235a1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b1c0ac650 con 0x7f5b1c069d40 2023-10-25T00:02:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b2133d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b1c069d40 0x7f5b1c158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b2133d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b1c069d40 0x7f5b1c158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49928/0 (socket says 172.21.15.107:49928) 2023-10-25T00:02:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b2133d640 1 -- 172.21.15.107:0/395503182 learned_addr learned my addr 172.21.15.107:0/395503182 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:00.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b2133d640 1 -- 172.21.15.107:0/395503182 >> v1:172.21.15.110:6789/0 conn(0x7f5b1c065bc0 legacy=0x7f5b1c0a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:00.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b2133d640 1 -- 172.21.15.107:0/395503182 >> v1:172.21.15.107:6789/0 conn(0x7f5b1c06a120 legacy=0x7f5b1c0aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.055+0000 7f5b2133d640 1 -- 172.21.15.107:0/395503182 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c06d3d0 con 0x7f5b1c069d40 2023-10-25T00:02:00.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b2133d640 1 --2- 172.21.15.107:0/395503182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c069d40 0x7f5b1c158970 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f5b18009e50 tx=0x7f5b1801c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2522dbe792d6eda server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:00.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b13fff640 1 -- 172.21.15.107:0/395503182 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180a5730 con 0x7f5b1c069d40 2023-10-25T00:02:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b13fff640 1 -- 172.21.15.107:0/395503182 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b180a5890 con 0x7f5b1c069d40 2023-10-25T00:02:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b13fff640 1 -- 172.21.15.107:0/395503182 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180a5c20 con 0x7f5b1c069d40 2023-10-25T00:02:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b235a1640 1 -- 172.21.15.107:0/395503182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c069d40 msgr2=0x7f5b1c158970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b235a1640 1 --2- 172.21.15.107:0/395503182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c069d40 0x7f5b1c158970 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f5b18009e50 tx=0x7f5b1801c980 comp rx=0 tx=0).stop 2023-10-25T00:02:00.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b235a1640 1 -- 172.21.15.107:0/395503182 shutdown_connections 2023-10-25T00:02:00.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b235a1640 1 --2- 172.21.15.107:0/395503182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c069d40 0x7f5b1c158970 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b235a1640 1 -- 172.21.15.107:0/395503182 >> 172.21.15.107:0/395503182 conn(0x7f5b1c0c65c0 msgr2=0x7f5b1c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:00.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b235a1640 1 -- 172.21.15.107:0/395503182 shutdown_connections 2023-10-25T00:02:00.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b235a1640 1 -- 172.21.15.107:0/395503182 wait complete. 2023-10-25T00:02:00.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b235a1640 1 Processor -- start 2023-10-25T00:02:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.059+0000 7f5b235a1640 1 -- start start 2023-10-25T00:02:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b235a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c065bc0 0x7f5b1c070f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b235a1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b1c06a120 0x7f5b1c0bc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:00.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b235a1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b1c15f670 0x7f5b1c1f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b235a1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c1ee940 con 0x7f5b1c065bc0 2023-10-25T00:02:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b235a1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c071440 con 0x7f5b1c15f670 2023-10-25T00:02:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b235a1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c157110 con 0x7f5b1c06a120 2023-10-25T00:02:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b2133d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c065bc0 0x7f5b1c070f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b2133d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c065bc0 0x7f5b1c070f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49936/0 (socket says 172.21.15.107:49936) 2023-10-25T00:02:00.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b2133d640 1 -- 172.21.15.107:0/3668369463 learned_addr learned my addr 172.21.15.107:0/3668369463 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:00.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b20b3c640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b1c06a120 0x7f5b1c0bc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:00.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b21b3e640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b1c15f670 0x7f5b1c1f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:00.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b2133d640 1 -- 172.21.15.107:0/3668369463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b1c06a120 msgr2=0x7f5b1c0bc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b2133d640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b1c06a120 0x7f5b1c0bc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b2133d640 1 -- 172.21.15.107:0/3668369463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b1c15f670 msgr2=0x7f5b1c1f03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b2133d640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b1c15f670 0x7f5b1c1f03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b2133d640 1 -- 172.21.15.107:0/3668369463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b18009b30 con 0x7f5b1c065bc0 2023-10-25T00:02:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b2133d640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c065bc0 0x7f5b1c070f00 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f5b180af040 tx=0x7f5b18019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:00.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b11ffb640 1 -- 172.21.15.107:0/3668369463 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180b8880 con 0x7f5b1c065bc0 2023-10-25T00:02:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b11ffb640 1 -- 172.21.15.107:0/3668369463 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b180b8a20 con 0x7f5b1c065bc0 2023-10-25T00:02:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b235a1640 1 -- 172.21.15.107:0/3668369463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c1f0f30 con 0x7f5b1c065bc0 2023-10-25T00:02:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b11ffb640 1 -- 172.21.15.107:0/3668369463 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180b8dd0 con 0x7f5b1c065bc0 2023-10-25T00:02:00.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b235a1640 1 -- 172.21.15.107:0/3668369463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c1ee2f0 con 0x7f5b1c065bc0 2023-10-25T00:02:00.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b235a1640 1 -- 172.21.15.107:0/3668369463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c0a9c30 con 0x7f5b1c065bc0 2023-10-25T00:02:00.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b11ffb640 1 -- 172.21.15.107:0/3668369463 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5b180b6440 con 0x7f5b1c065bc0 2023-10-25T00:02:00.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b11ffb640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b00065e90 0x7f5b00068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:00.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b11ffb640 1 -- 172.21.15.107:0/3668369463 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7f5b1811f580 con 0x7f5b1c065bc0 2023-10-25T00:02:00.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.063+0000 7f5b20b3c640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b00065e90 0x7f5b00068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:00.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.067+0000 7f5b20b3c640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b00065e90 0x7f5b00068350 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c15fb70 tx=0x7f5b0c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:00.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.067+0000 7f5b11ffb640 1 -- 172.21.15.107:0/3668369463 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5b180ec300 con 0x7f5b1c065bc0 2023-10-25T00:02:00.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.263+0000 7f5b235a1640 1 -- 172.21.15.107:0/3668369463 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b1c157f20 con 0x7f5b1c065bc0 2023-10-25T00:02:00.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.263+0000 7f5b11ffb640 1 -- 172.21.15.107:0/3668369463 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5016 (secure 0 0 0) 0x7f5b180ef920 con 0x7f5b1c065bc0 2023-10-25T00:02:00.266 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:00.266 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:59.596353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:00.267 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 77 2023-10-25T00:02:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 -- 172.21.15.107:0/3668369463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b00065e90 msgr2=0x7f5b00068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b00065e90 0x7f5b00068350 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c15fb70 tx=0x7f5b0c009040 comp rx=0 tx=0).stop 2023-10-25T00:02:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 -- 172.21.15.107:0/3668369463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c065bc0 msgr2=0x7f5b1c070f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c065bc0 0x7f5b1c070f00 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f5b180af040 tx=0x7f5b18019720 comp rx=0 tx=0).stop 2023-10-25T00:02:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 -- 172.21.15.107:0/3668369463 shutdown_connections 2023-10-25T00:02:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b1c15f670 0x7f5b1c1f03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b00065e90 0x7f5b00068350 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b1c06a120 0x7f5b1c0bc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 --2- 172.21.15.107:0/3668369463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b1c065bc0 0x7f5b1c070f00 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 -- 172.21.15.107:0/3668369463 >> 172.21.15.107:0/3668369463 conn(0x7f5b1c0c65c0 msgr2=0x7f5b1c155fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:00.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 -- 172.21.15.107:0/3668369463 shutdown_connections 2023-10-25T00:02:00.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.267+0000 7f5b235a1640 1 -- 172.21.15.107:0/3668369463 wait complete. 2023-10-25T00:02:00.285 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2023-10-25T00:02:00.285 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:02:00.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:00.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.403+0000 7f8737a67640 1 Processor -- start 2023-10-25T00:02:00.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.403+0000 7f8737a67640 1 -- start start 2023-10-25T00:02:00.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8737a67640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f873014fd40 0x7f87300ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:00.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8737a67640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f87300a9820 0x7f87300aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:00.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8737a67640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873006f730 con 0x7f873006ee40 2023-10-25T00:02:00.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8737a67640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87300aa750 con 0x7f87300a9820 2023-10-25T00:02:00.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8737a67640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8730156a40 con 0x7f873014fd40 2023-10-25T00:02:00.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8735002640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f87300a9820 0x7f87300aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:00.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8735002640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f87300a9820 0x7f87300aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49950/0 (socket says 172.21.15.107:49950) 2023-10-25T00:02:00.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8736004640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f873006ee40 0x7f87300c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53202/0 (socket says 172.21.15.107:53202) 2023-10-25T00:02:00.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8735002640 1 -- 172.21.15.107:0/268984509 learned_addr learned my addr 172.21.15.107:0/268984509 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:00.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8735803640 1 --2- 172.21.15.107:0/268984509 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f873014fd40 0x7f87300ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:00.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8735002640 1 -- 172.21.15.107:0/268984509 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f873014fd40 msgr2=0x7f87300ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:00.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8735002640 1 --2- 172.21.15.107:0/268984509 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f873014fd40 0x7f87300ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8735002640 1 -- 172.21.15.107:0/268984509 >> v1:172.21.15.107:6789/0 conn(0x7f873006ee40 legacy=0x7f87300c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8735002640 1 -- 172.21.15.107:0/268984509 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8730156710 con 0x7f87300a9820 2023-10-25T00:02:00.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8735803640 1 --2- 172.21.15.107:0/268984509 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f873014fd40 0x7f87300ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:02:00.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8734801640 1 -- 172.21.15.107:0/268984509 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224906206 0 0) 0x7f873006f730 con 0x7f873006ee40 2023-10-25T00:02:00.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8735002640 1 --2- 172.21.15.107:0/268984509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87300a9820 0x7f87300aa210 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f8724009870 tx=0x7f872401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93aa21d7be81b373 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:00.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8734801640 1 -- 172.21.15.107:0/268984509 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87240ad640 con 0x7f87300a9820 2023-10-25T00:02:00.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8734801640 1 -- 172.21.15.107:0/268984509 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87240ad7a0 con 0x7f87300a9820 2023-10-25T00:02:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8734801640 1 -- 172.21.15.107:0/268984509 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87240adb30 con 0x7f87300a9820 2023-10-25T00:02:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8737a67640 1 -- 172.21.15.107:0/268984509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87300a9820 msgr2=0x7f87300aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.407+0000 7f8737a67640 1 --2- 172.21.15.107:0/268984509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87300a9820 0x7f87300aa210 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f8724009870 tx=0x7f872401c3f0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 -- 172.21.15.107:0/268984509 shutdown_connections 2023-10-25T00:02:00.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 --2- 172.21.15.107:0/268984509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87300a9820 0x7f87300aa210 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 --2- 172.21.15.107:0/268984509 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f873014fd40 0x7f87300ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 -- 172.21.15.107:0/268984509 >> 172.21.15.107:0/268984509 conn(0x7f87300c2950 msgr2=0x7f87300c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 -- 172.21.15.107:0/268984509 shutdown_connections 2023-10-25T00:02:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 -- 172.21.15.107:0/268984509 wait complete. 2023-10-25T00:02:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 Processor -- start 2023-10-25T00:02:00.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 -- start start 2023-10-25T00:02:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f873006ee40 0x7f87300c4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f873014fd40 0x7f873015b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8730155a20 0x7f8730146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87301eda10 con 0x7f873006ee40 2023-10-25T00:02:00.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87301ecd90 con 0x7f873014fd40 2023-10-25T00:02:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87301ed070 con 0x7f8730155a20 2023-10-25T00:02:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8735803640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f873006ee40 0x7f87300c4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8735803640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f873006ee40 0x7f87300c4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49952/0 (socket says 172.21.15.107:49952) 2023-10-25T00:02:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8735803640 1 -- 172.21.15.107:0/2313284785 learned_addr learned my addr 172.21.15.107:0/2313284785 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8735002640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f873014fd40 0x7f873015b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8736004640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8730155a20 0x7f8730146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8735803640 1 -- 172.21.15.107:0/2313284785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8730155a20 msgr2=0x7f8730146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8735803640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8730155a20 0x7f8730146470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8735803640 1 -- 172.21.15.107:0/2313284785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f873014fd40 msgr2=0x7f873015b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8735803640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f873014fd40 0x7f873015b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8735803640 1 -- 172.21.15.107:0/2313284785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87240094d0 con 0x7f873006ee40 2023-10-25T00:02:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8735002640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f873014fd40 0x7f873015b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8735803640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f873006ee40 0x7f87300c4610 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f872c00e810 tx=0x7f872c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f87227fc640 1 -- 172.21.15.107:0/2313284785 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c00c8d0 con 0x7f873006ee40 2023-10-25T00:02:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f87227fc640 1 -- 172.21.15.107:0/2313284785 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f872c00ca30 con 0x7f873006ee40 2023-10-25T00:02:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 -- 172.21.15.107:0/2313284785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87301ed350 con 0x7f873006ee40 2023-10-25T00:02:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f87227fc640 1 -- 172.21.15.107:0/2313284785 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f872c00cde0 con 0x7f873006ee40 2023-10-25T00:02:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 -- 172.21.15.107:0/2313284785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873015bae0 con 0x7f873006ee40 2023-10-25T00:02:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.411+0000 7f8737a67640 1 -- 172.21.15.107:0/2313284785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87300c24a0 con 0x7f873006ee40 2023-10-25T00:02:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.415+0000 7f87227fc640 1 -- 172.21.15.107:0/2313284785 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f872c013410 con 0x7f873006ee40 2023-10-25T00:02:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.415+0000 7f87227fc640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8714065e60 0x7f8714068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.415+0000 7f8735002640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8714065e60 0x7f8714068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.415+0000 7f87227fc640 1 -- 172.21.15.107:0/2313284785 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7f872c083e40 con 0x7f873006ee40 2023-10-25T00:02:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.415+0000 7f8735002640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8714065e60 0x7f8714068320 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f87240af000 tx=0x7f87240a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:00.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.419+0000 7f87227fc640 1 -- 172.21.15.107:0/2313284785 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f872c050ce0 con 0x7f873006ee40 2023-10-25T00:02:00.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.611+0000 7f8737a67640 1 -- 172.21.15.107:0/2313284785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8730070840 con 0x7f873006ee40 2023-10-25T00:02:00.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.611+0000 7f87227fc640 1 -- 172.21.15.107:0/2313284785 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+5016 (secure 0 0 0) 0x7f872c021090 con 0x7f873006ee40 2023-10-25T00:02:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:00.615 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:01:59.596353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:00.615 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 77 2023-10-25T00:02:00.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 -- 172.21.15.107:0/2313284785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8714065e60 msgr2=0x7f8714068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8714065e60 0x7f8714068320 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f87240af000 tx=0x7f87240a7070 comp rx=0 tx=0).stop 2023-10-25T00:02:00.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 -- 172.21.15.107:0/2313284785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f873006ee40 msgr2=0x7f87300c4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:00.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f873006ee40 0x7f87300c4610 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f872c00e810 tx=0x7f872c00ecd0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 -- 172.21.15.107:0/2313284785 shutdown_connections 2023-10-25T00:02:00.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8714065e60 0x7f8714068320 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8730155a20 0x7f8730146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f873014fd40 0x7f873015b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 --2- 172.21.15.107:0/2313284785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f873006ee40 0x7f87300c4610 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:00.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 -- 172.21.15.107:0/2313284785 >> 172.21.15.107:0/2313284785 conn(0x7f87300c2950 msgr2=0x7f87300719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:00.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 -- 172.21.15.107:0/2313284785 shutdown_connections 2023-10-25T00:02:00.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:00.615+0000 7f8737a67640 1 -- 172.21.15.107:0/2313284785 wait complete. 2023-10-25T00:02:00.637 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 77, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 77, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6396": { "addr": "172.21.15.107:6835/3232233949", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3232233949, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 3232233949, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6396, "incarnation": 76, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:01:59.596353+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6396 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 69, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5506": { "addr": "172.21.15.110:6834/201284779", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 201284779, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 201284779, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5506, "incarnation": 66, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:00:51.726634+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5506 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 59, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:02:02.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:02.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f700a1f0640 1 Processor -- start 2023-10-25T00:02:02.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f700a1f0640 1 -- start start 2023-10-25T00:02:02.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f700a1f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700415c200 0x7f7004071100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:02.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f700a1f0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70040c5d40 con 0x7f7004156220 2023-10-25T00:02:02.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f700a1f0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70040c6130 con 0x7f70040c6a30 2023-10-25T00:02:02.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f700a1f0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7004158f60 con 0x7f700415c200 2023-10-25T00:02:02.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f70037fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7004156220 0x7f700406df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41136/0 (socket says 172.21.15.107:41136) 2023-10-25T00:02:02.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f70037fe640 1 -- 172.21.15.107:0/409137446 learned_addr learned my addr 172.21.15.107:0/409137446 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:02:02.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f6ffbfff640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700415c200 0x7f7004071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:02.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872080530 0 0) 0x7f70040c5d40 con 0x7f7004156220 2023-10-25T00:02:02.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff0003530 con 0x7f7004156220 2023-10-25T00:02:02.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183356182 0 0) 0x7f6ff0003530 con 0x7f7004156220 2023-10-25T00:02:02.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700415c200 msgr2=0x7f7004071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:02.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700415c200 0x7f7004071100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:02.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 >> v1:172.21.15.110:6789/0 conn(0x7f70040c6a30 legacy=0x7f7004071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:02.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7004159240 con 0x7f7004156220 2023-10-25T00:02:02.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6ff4001680 con 0x7f7004156220 2023-10-25T00:02:02.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 >> v1:172.21.15.107:6789/0 conn(0x7f7004156220 legacy=0x7f700406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:02.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 0x7f6ff0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:02.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700415c200 0x7f6ff00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:02.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff0004d10 0x7f6ff0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:02.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7004158f60 con 0x7f6ff00039b0 2023-10-25T00:02:02.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff000ba40 con 0x7f6ff0004d10 2023-10-25T00:02:02.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff000bc00 con 0x7f700415c200 2023-10-25T00:02:02.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f70037fe640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 0x7f6ff0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:02.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f6ffbfff640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700415c200 0x7f6ff00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:02.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7003fff640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff0004d10 0x7f6ff0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:02.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f70037fe640 1 -- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700415c200 msgr2=0x7f6ff00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:02.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f70037fe640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700415c200 0x7f6ff00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:02.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f70037fe640 1 -- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff0004d10 msgr2=0x7f6ff0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:02.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f70037fe640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff0004d10 0x7f6ff0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:02.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f70037fe640 1 -- 172.21.15.107:0/409137446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff0003530 con 0x7f6ff00039b0 2023-10-25T00:02:02.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.771+0000 7f7003fff640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ff0004d10 0x7f6ff0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:02.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.775+0000 7f70037fe640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 0x7f6ff0003d90 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400c190 tx=0x7f6ff401cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4bf8fcee62d22e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:02.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.775+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4005ae0 con 0x7f6ff00039b0 2023-10-25T00:02:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.775+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff4005c60 con 0x7f6ff00039b0 2023-10-25T00:02:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.775+0000 7f7002ffd640 1 -- 172.21.15.107:0/409137446 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40a57a0 con 0x7f6ff00039b0 2023-10-25T00:02:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.775+0000 7f700a1f0640 1 -- 172.21.15.107:0/409137446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 msgr2=0x7f6ff0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.775+0000 7f700a1f0640 1 --2- 172.21.15.107:0/409137446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 0x7f6ff0003d90 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400c190 tx=0x7f6ff401cbb0 comp rx=0 tx=0).stop 2023-10-25T00:02:02.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.775+0000 7f7003fff640 1 -- 172.21.15.107:0/409137446 reap_dead start 2023-10-25T00:02:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.775+0000 7f700a1f0640 1 -- 172.21.15.107:0/409137446 shutdown_connections 2023-10-25T00:02:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.775+0000 7f700a1f0640 1 -- 172.21.15.107:0/409137446 >> 172.21.15.107:0/409137446 conn(0x7f70040c39a0 msgr2=0x7f70040c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.775+0000 7f700a1f0640 1 -- 172.21.15.107:0/409137446 shutdown_connections 2023-10-25T00:02:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.775+0000 7f700a1f0640 1 -- 172.21.15.107:0/409137446 wait complete. 2023-10-25T00:02:02.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f700a1f0640 1 Processor -- start 2023-10-25T00:02:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f700a1f0640 1 -- start start 2023-10-25T00:02:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f700a1f0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 0x7f70041f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f700a1f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7004157b80 0x7f7004158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f700a1f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7004150b90 0x7f7004151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:02.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f700a1f0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70041f22d0 con 0x7f6ff00039b0 2023-10-25T00:02:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f700a1f0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700415a480 con 0x7f7004157b80 2023-10-25T00:02:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f700a1f0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700406d760 con 0x7f7004150b90 2023-10-25T00:02:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f70037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 0x7f70041f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f70037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 0x7f70041f1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45780/0 (socket says 172.21.15.107:45780) 2023-10-25T00:02:02.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f70037fe640 1 -- 172.21.15.107:0/2519537500 learned_addr learned my addr 172.21.15.107:0/2519537500 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f6ffbfff640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7004157b80 0x7f7004158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f7003fff640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7004150b90 0x7f7004151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f70037fe640 1 -- 172.21.15.107:0/2519537500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7004150b90 msgr2=0x7f7004151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f70037fe640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7004150b90 0x7f7004151040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f70037fe640 1 -- 172.21.15.107:0/2519537500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7004157b80 msgr2=0x7f7004158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f70037fe640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7004157b80 0x7f7004158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f70037fe640 1 -- 172.21.15.107:0/2519537500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700406da40 con 0x7f6ff00039b0 2023-10-25T00:02:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f70037fe640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 0x7f70041f1d90 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6ff401c9e0 tx=0x7f6ff0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f7000ff9640 1 -- 172.21.15.107:0/2519537500 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40b4070 con 0x7f6ff00039b0 2023-10-25T00:02:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f7000ff9640 1 -- 172.21.15.107:0/2519537500 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff4005e50 con 0x7f6ff00039b0 2023-10-25T00:02:02.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f7000ff9640 1 -- 172.21.15.107:0/2519537500 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4001640 con 0x7f6ff00039b0 2023-10-25T00:02:02.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f700a1f0640 1 -- 172.21.15.107:0/2519537500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700415a110 con 0x7f6ff00039b0 2023-10-25T00:02:02.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f700a1f0640 1 -- 172.21.15.107:0/2519537500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70041515c0 con 0x7f6ff00039b0 2023-10-25T00:02:02.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f700a1f0640 1 -- 172.21.15.107:0/2519537500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7004070480 con 0x7f6ff00039b0 2023-10-25T00:02:02.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.779+0000 7f7000ff9640 1 -- 172.21.15.107:0/2519537500 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6ff40a57a0 con 0x7f6ff00039b0 2023-10-25T00:02:02.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.783+0000 7f7000ff9640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fe4065e90 0x7f6fe4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:02.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.783+0000 7f7000ff9640 1 -- 172.21.15.107:0/2519537500 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7f6ff411da80 con 0x7f6ff00039b0 2023-10-25T00:02:02.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.783+0000 7f6ffbfff640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fe4065e90 0x7f6fe4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:02.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.783+0000 7f6ffbfff640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fe4065e90 0x7f6fe4068350 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f70040c34f0 tx=0x7f6fec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:02.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:02.787+0000 7f7000ff9640 1 -- 172.21.15.107:0/2519537500 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6ff40ea880 con 0x7f6ff00039b0 2023-10-25T00:02:03.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.015+0000 7f700a1f0640 1 -- 172.21.15.107:0/2519537500 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70040746e0 con 0x7f6ff00039b0 2023-10-25T00:02:03.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.015+0000 7f7000ff9640 1 -- 172.21.15.107:0/2519537500 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f6ff40edea0 con 0x7f6ff00039b0 2023-10-25T00:02:03.019 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:03.019 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:03.020 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 79 2023-10-25T00:02:03.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 -- 172.21.15.107:0/2519537500 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fe4065e90 msgr2=0x7f6fe4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:03.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fe4065e90 0x7f6fe4068350 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f70040c34f0 tx=0x7f6fec008040 comp rx=0 tx=0).stop 2023-10-25T00:02:03.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 -- 172.21.15.107:0/2519537500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 msgr2=0x7f70041f1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:03.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 0x7f70041f1d90 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6ff401c9e0 tx=0x7f6ff0005800 comp rx=0 tx=0).stop 2023-10-25T00:02:03.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 -- 172.21.15.107:0/2519537500 shutdown_connections 2023-10-25T00:02:03.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fe4065e90 0x7f6fe4068350 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:03.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7004150b90 0x7f7004151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:03.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7004157b80 0x7f7004158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:03.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 --2- 172.21.15.107:0/2519537500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ff00039b0 0x7f70041f1d90 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:03.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 -- 172.21.15.107:0/2519537500 >> 172.21.15.107:0/2519537500 conn(0x7f70040c39a0 msgr2=0x7f700415e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:03.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 -- 172.21.15.107:0/2519537500 shutdown_connections 2023-10-25T00:02:03.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:03.019+0000 7f700a1f0640 1 -- 172.21.15.107:0/2519537500 wait complete. 2023-10-25T00:02:03.038 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=6303 2023-10-25T00:02:03.039 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.a 2023-10-25T00:02:04.687 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:02:04.683+0000 7f3f3a103640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:02:05.040 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2023-10-25T00:02:05.040 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:02:05.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:02:05.043 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:02:05.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:05.116 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:02:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc130eb640 1 Processor -- start 2023-10-25T00:02:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc130eb640 1 -- start start 2023-10-25T00:02:05.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc130eb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc0c0ac020 0x7fdc0c0c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:05.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc130eb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdc0c0c2520 0x7fdc0c06c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:05.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc130eb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c066000 con 0x7fdc0c0aa140 2023-10-25T00:02:05.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc130eb640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c0c0f40 con 0x7fdc0c0c2520 2023-10-25T00:02:05.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc130eb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0c1220 con 0x7fdc0c0ac020 2023-10-25T00:02:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc11688640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdc0c0c2520 0x7fdc0c06c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc11688640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdc0c0c2520 0x7fdc0c06c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35508/0 (socket says 172.21.15.107:35508) 2023-10-25T00:02:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc11688640 1 -- 172.21.15.107:0/3641664891 learned_addr learned my addr 172.21.15.107:0/3641664891 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:05.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc0bfff640 1 --2- 172.21.15.107:0/3641664891 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc0c0ac020 0x7fdc0c0c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc11688640 1 -- 172.21.15.107:0/3641664891 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc0c0ac020 msgr2=0x7fdc0c0c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc11688640 1 --2- 172.21.15.107:0/3641664891 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc0c0ac020 0x7fdc0c0c1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc11688640 1 -- 172.21.15.107:0/3641664891 >> v1:172.21.15.110:6789/0 conn(0x7fdc0c0aa140 legacy=0x7fdc0c0aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.183+0000 7fdc11688640 1 -- 172.21.15.107:0/3641664891 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0c1500 con 0x7fdc0c0c2520 2023-10-25T00:02:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc11688640 1 --2- 172.21.15.107:0/3641664891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc0c0c2520 0x7fdc0c06c490 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400d150 tx=0x7fdc0401ebe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bab838753e273f2f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:05.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc0b7fe640 1 -- 172.21.15.107:0/3641664891 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400b2d0 con 0x7fdc0c0c2520 2023-10-25T00:02:05.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc0b7fe640 1 -- 172.21.15.107:0/3641664891 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0400b430 con 0x7fdc0c0c2520 2023-10-25T00:02:05.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc0b7fe640 1 -- 172.21.15.107:0/3641664891 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc040a7cd0 con 0x7fdc0c0c2520 2023-10-25T00:02:05.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc130eb640 1 -- 172.21.15.107:0/3641664891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc0c0c2520 msgr2=0x7fdc0c06c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc130eb640 1 --2- 172.21.15.107:0/3641664891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc0c0c2520 0x7fdc0c06c490 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400d150 tx=0x7fdc0401ebe0 comp rx=0 tx=0).stop 2023-10-25T00:02:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc130eb640 1 -- 172.21.15.107:0/3641664891 shutdown_connections 2023-10-25T00:02:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc130eb640 1 --2- 172.21.15.107:0/3641664891 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc0c0c2520 0x7fdc0c06c490 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc130eb640 1 --2- 172.21.15.107:0/3641664891 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc0c0ac020 0x7fdc0c0c1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc130eb640 1 -- 172.21.15.107:0/3641664891 >> 172.21.15.107:0/3641664891 conn(0x7fdc0c0c65c0 msgr2=0x7fdc0c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:05.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc130eb640 1 -- 172.21.15.107:0/3641664891 shutdown_connections 2023-10-25T00:02:05.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc130eb640 1 -- 172.21.15.107:0/3641664891 wait complete. 2023-10-25T00:02:05.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc130eb640 1 Processor -- start 2023-10-25T00:02:05.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.187+0000 7fdc130eb640 1 -- start start 2023-10-25T00:02:05.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc130eb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc0c0aa140 0x7fdc0c152c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:05.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc130eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc0c0ac020 0x7fdc0c153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:05.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc130eb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc0c0c2520 0x7fdc0c1ea150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:05.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc130eb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c157a80 con 0x7fdc0c0ac020 2023-10-25T00:02:05.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc130eb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c071d70 con 0x7fdc0c0aa140 2023-10-25T00:02:05.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc130eb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c072050 con 0x7fdc0c0c2520 2023-10-25T00:02:05.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc0bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc0c0ac020 0x7fdc0c153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:05.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc0bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc0c0ac020 0x7fdc0c153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45800/0 (socket says 172.21.15.107:45800) 2023-10-25T00:02:05.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc0bfff640 1 -- 172.21.15.107:0/355948632 learned_addr learned my addr 172.21.15.107:0/355948632 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:05.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc10e87640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc0c0aa140 0x7fdc0c152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:05.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc11688640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc0c0c2520 0x7fdc0c1ea150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc0bfff640 1 -- 172.21.15.107:0/355948632 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc0c0c2520 msgr2=0x7fdc0c1ea150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc0bfff640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc0c0c2520 0x7fdc0c1ea150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc0bfff640 1 -- 172.21.15.107:0/355948632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc0c0aa140 msgr2=0x7fdc0c152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc0bfff640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc0c0aa140 0x7fdc0c152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc0bfff640 1 -- 172.21.15.107:0/355948632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0400bd40 con 0x7fdc0c0ac020 2023-10-25T00:02:05.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc11688640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc0c0c2520 0x7fdc0c1ea150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:02:05.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc0bfff640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc0c0ac020 0x7fdc0c153140 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800d5f0 tx=0x7fdbf800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:05.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc097fa640 1 -- 172.21.15.107:0/355948632 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf80125f0 con 0x7fdc0c0ac020 2023-10-25T00:02:05.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc097fa640 1 -- 172.21.15.107:0/355948632 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf8012750 con 0x7fdc0c0ac020 2023-10-25T00:02:05.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc130eb640 1 -- 172.21.15.107:0/355948632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c06f6f0 con 0x7fdc0c0ac020 2023-10-25T00:02:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc097fa640 1 -- 172.21.15.107:0/355948632 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8012ae0 con 0x7fdc0c0ac020 2023-10-25T00:02:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc130eb640 1 -- 172.21.15.107:0/355948632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c06fc10 con 0x7fdc0c0ac020 2023-10-25T00:02:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc130eb640 1 -- 172.21.15.107:0/355948632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c06f550 con 0x7fdc0c0ac020 2023-10-25T00:02:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc097fa640 1 -- 172.21.15.107:0/355948632 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdbf8012c40 con 0x7fdc0c0ac020 2023-10-25T00:02:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.191+0000 7fdc097fa640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbf0065e60 0x7fdbf0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:05.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.195+0000 7fdc097fa640 1 -- 172.21.15.107:0/355948632 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7fdbf80843f0 con 0x7fdc0c0ac020 2023-10-25T00:02:05.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.195+0000 7fdc10e87640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbf0065e60 0x7fdbf0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:05.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.195+0000 7fdc10e87640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbf0065e60 0x7fdbf0068320 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdc00001770 tx=0x7fdc00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:05.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.199+0000 7fdc097fa640 1 -- 172.21.15.107:0/355948632 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdc0c06f550 con 0x7fdc0c0ac020 2023-10-25T00:02:05.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 -- 172.21.15.107:0/355948632 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc0c06f550 con 0x7fdc0c0ac020 2023-10-25T00:02:05.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc097fa640 1 -- 172.21.15.107:0/355948632 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5014 (secure 0 0 0) 0x7fdc0c06f550 con 0x7fdc0c0ac020 2023-10-25T00:02:05.448 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:05.448 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:05.449 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 79 2023-10-25T00:02:05.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 -- 172.21.15.107:0/355948632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbf0065e60 msgr2=0x7fdbf0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:05.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbf0065e60 0x7fdbf0068320 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fdc00001770 tx=0x7fdc00009040 comp rx=0 tx=0).stop 2023-10-25T00:02:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 -- 172.21.15.107:0/355948632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc0c0ac020 msgr2=0x7fdc0c153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc0c0ac020 0x7fdc0c153140 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800d5f0 tx=0x7fdbf800dab0 comp rx=0 tx=0).stop 2023-10-25T00:02:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 -- 172.21.15.107:0/355948632 shutdown_connections 2023-10-25T00:02:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbf0065e60 0x7fdbf0068320 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:05.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc0c0c2520 0x7fdc0c1ea150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:05.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc0c0ac020 0x7fdc0c153140 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:05.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 --2- 172.21.15.107:0/355948632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc0c0aa140 0x7fdc0c152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:05.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 -- 172.21.15.107:0/355948632 >> 172.21.15.107:0/355948632 conn(0x7fdc0c0c65c0 msgr2=0x7fdc0c0a94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:05.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 -- 172.21.15.107:0/355948632 shutdown_connections 2023-10-25T00:02:05.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:05.447+0000 7fdc130eb640 1 -- 172.21.15.107:0/355948632 wait complete. 2023-10-25T00:02:05.468 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-25T00:02:07.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.607+0000 7fe2f2407640 1 Processor -- start 2023-10-25T00:02:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.607+0000 7fe2f2407640 1 -- start start 2023-10-25T00:02:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe2ec151f00 0x7fe2ec070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe2ec155370 0x7fe2ec071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2ec155870 con 0x7fe2ec0c1080 2023-10-25T00:02:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2ec1522e0 con 0x7fe2ec155370 2023-10-25T00:02:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec0c02c0 con 0x7fe2ec151f00 2023-10-25T00:02:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe2ec155370 0x7fe2ec071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe2ec155370 0x7fe2ec071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45810/0 (socket says 172.21.15.107:45810) 2023-10-25T00:02:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eb7fe640 1 -- 172.21.15.107:0/1755274210 learned_addr learned my addr 172.21.15.107:0/1755274210 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2ebfff640 1 --2- 172.21.15.107:0/1755274210 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe2ec151f00 0x7fe2ec070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eaffd640 1 -- 172.21.15.107:0/1755274210 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083195387 0 0) 0x7fe2ec155870 con 0x7fe2ec0c1080 2023-10-25T00:02:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eaffd640 1 -- 172.21.15.107:0/1755274210 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d4003530 con 0x7fe2ec0c1080 2023-10-25T00:02:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eb7fe640 1 -- 172.21.15.107:0/1755274210 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe2ec151f00 msgr2=0x7fe2ec070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eb7fe640 1 --2- 172.21.15.107:0/1755274210 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe2ec151f00 0x7fe2ec070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eb7fe640 1 -- 172.21.15.107:0/1755274210 >> v1:172.21.15.107:6789/0 conn(0x7fe2ec0c1080 legacy=0x7fe2ec0ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:07.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eb7fe640 1 -- 172.21.15.107:0/1755274210 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec0c05a0 con 0x7fe2ec155370 2023-10-25T00:02:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eb7fe640 1 --2- 172.21.15.107:0/1755274210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2ec155370 0x7fe2ec071350 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009870 tx=0x7fe2e001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d957d71cc67b74d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eaffd640 1 -- 172.21.15.107:0/1755274210 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e00a9400 con 0x7fe2ec155370 2023-10-25T00:02:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eaffd640 1 -- 172.21.15.107:0/1755274210 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e00a9560 con 0x7fe2ec155370 2023-10-25T00:02:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2eaffd640 1 -- 172.21.15.107:0/1755274210 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e00a98f0 con 0x7fe2ec155370 2023-10-25T00:02:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 -- 172.21.15.107:0/1755274210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2ec155370 msgr2=0x7fe2ec071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 --2- 172.21.15.107:0/1755274210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2ec155370 0x7fe2ec071350 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009870 tx=0x7fe2e001c540 comp rx=0 tx=0).stop 2023-10-25T00:02:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 -- 172.21.15.107:0/1755274210 shutdown_connections 2023-10-25T00:02:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 --2- 172.21.15.107:0/1755274210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2ec155370 0x7fe2ec071350 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 --2- 172.21.15.107:0/1755274210 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe2ec151f00 0x7fe2ec070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 -- 172.21.15.107:0/1755274210 >> 172.21.15.107:0/1755274210 conn(0x7fe2ec0c5cd0 msgr2=0x7fe2ec0c60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 -- 172.21.15.107:0/1755274210 shutdown_connections 2023-10-25T00:02:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.611+0000 7fe2f2407640 1 -- 172.21.15.107:0/1755274210 wait complete. 2023-10-25T00:02:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2f2407640 1 Processor -- start 2023-10-25T00:02:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2f2407640 1 -- start start 2023-10-25T00:02:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2f2407640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2ec0c1080 0x7fe2ec148e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2f2407640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe2ec151f00 0x7fe2ec145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2f2407640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2ec155370 0x7fe2ec145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:07.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2f2407640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec14b2e0 con 0x7fe2ec0c1080 2023-10-25T00:02:07.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2f2407640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec14ef80 con 0x7fe2ec155370 2023-10-25T00:02:07.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2f2407640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2ec1f05b0 con 0x7fe2ec151f00 2023-10-25T00:02:07.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2ec0c1080 0x7fe2ec148e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:07.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2ebfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2ec0c1080 0x7fe2ec148e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45818/0 (socket says 172.21.15.107:45818) 2023-10-25T00:02:07.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2ebfff640 1 -- 172.21.15.107:0/3362796934 learned_addr learned my addr 172.21.15.107:0/3362796934 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:07.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2eb7fe640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe2ec151f00 0x7fe2ec145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:07.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2f09a4640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2ec155370 0x7fe2ec145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:07.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2ebfff640 1 -- 172.21.15.107:0/3362796934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe2ec151f00 msgr2=0x7fe2ec145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:07.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2ebfff640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe2ec151f00 0x7fe2ec145790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:07.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2ebfff640 1 -- 172.21.15.107:0/3362796934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2ec155370 msgr2=0x7fe2ec145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:07.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2ebfff640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2ec155370 0x7fe2ec145cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:07.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2ebfff640 1 -- 172.21.15.107:0/3362796934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e00094d0 con 0x7fe2ec0c1080 2023-10-25T00:02:07.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2ebfff640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2ec0c1080 0x7fe2ec148e20 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00d800 tx=0x7fe2dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2e8ff9640 1 -- 172.21.15.107:0/3362796934 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc00b6c0 con 0x7fe2ec0c1080 2023-10-25T00:02:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2e8ff9640 1 -- 172.21.15.107:0/3362796934 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2dc00b820 con 0x7fe2ec0c1080 2023-10-25T00:02:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2f2407640 1 -- 172.21.15.107:0/3362796934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2ec1f0890 con 0x7fe2ec0c1080 2023-10-25T00:02:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2e8ff9640 1 -- 172.21.15.107:0/3362796934 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc00bb90 con 0x7fe2ec0c1080 2023-10-25T00:02:07.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.615+0000 7fe2f2407640 1 -- 172.21.15.107:0/3362796934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2ec1f0e50 con 0x7fe2ec0c1080 2023-10-25T00:02:07.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.619+0000 7fe2f2407640 1 -- 172.21.15.107:0/3362796934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2ec0746e0 con 0x7fe2ec0c1080 2023-10-25T00:02:07.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.619+0000 7fe2e8ff9640 1 -- 172.21.15.107:0/3362796934 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe2dc00bcf0 con 0x7fe2ec0c1080 2023-10-25T00:02:07.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.619+0000 7fe2e8ff9640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe2d8065e60 0x7fe2d8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:07.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.619+0000 7fe2e8ff9640 1 -- 172.21.15.107:0/3362796934 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7fe2dc0845c0 con 0x7fe2ec0c1080 2023-10-25T00:02:07.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.619+0000 7fe2eb7fe640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe2d8065e60 0x7fe2d8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:07.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.619+0000 7fe2eb7fe640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe2d8065e60 0x7fe2d8068320 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00ae000 tx=0x7fe2e00b3040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:07.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.623+0000 7fe2e8ff9640 1 -- 172.21.15.107:0/3362796934 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe2dc0513e0 con 0x7fe2ec0c1080 2023-10-25T00:02:07.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.855+0000 7fe2f2407640 1 -- 172.21.15.107:0/3362796934 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2ec15f050 con 0x7fe2ec0c1080 2023-10-25T00:02:07.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.855+0000 7fe2e8ff9640 1 -- 172.21.15.107:0/3362796934 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5771 (secure 0 0 0) 0x7fe2dc011340 con 0x7fe2ec0c1080 2023-10-25T00:02:07.859 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:07.859 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 80 2023-10-25T00:02:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 -- 172.21.15.107:0/3362796934 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe2d8065e60 msgr2=0x7fe2d8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe2d8065e60 0x7fe2d8068320 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00ae000 tx=0x7fe2e00b3040 comp rx=0 tx=0).stop 2023-10-25T00:02:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 -- 172.21.15.107:0/3362796934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2ec0c1080 msgr2=0x7fe2ec148e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2ec0c1080 0x7fe2ec148e20 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00d800 tx=0x7fe2dc00dcc0 comp rx=0 tx=0).stop 2023-10-25T00:02:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 -- 172.21.15.107:0/3362796934 shutdown_connections 2023-10-25T00:02:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2ec155370 0x7fe2ec145cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe2d8065e60 0x7fe2d8068320 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe2ec151f00 0x7fe2ec145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 --2- 172.21.15.107:0/3362796934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2ec0c1080 0x7fe2ec148e20 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 -- 172.21.15.107:0/3362796934 >> 172.21.15.107:0/3362796934 conn(0x7fe2ec0c5cd0 msgr2=0x7fe2ec152c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 -- 172.21.15.107:0/3362796934 shutdown_connections 2023-10-25T00:02:07.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:07.859+0000 7fe2f2407640 1 -- 172.21.15.107:0/3362796934 wait complete. 2023-10-25T00:02:07.879 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2023-10-25T00:02:07.880 INFO:tasks.mds_thrash.fs.[a]:waiting for 66 secs before thrashing 2023-10-25T00:02:09.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:09.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f434dfe8640 1 Processor -- start 2023-10-25T00:02:09.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f434dfe8640 1 -- start start 2023-10-25T00:02:09.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f434dfe8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f43480720c0 0x7f43480c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f434dfe8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4348154860 con 0x7f4348157e50 2023-10-25T00:02:09.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f434dfe8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43480c1f10 con 0x7f434813dc20 2023-10-25T00:02:09.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f434dfe8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4348154b20 con 0x7f43480720c0 2023-10-25T00:02:09.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f4347fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4348157e50 0x7f434815f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41158/0 (socket says 172.21.15.107:41158) 2023-10-25T00:02:09.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f4347fff640 1 -- 172.21.15.107:0/3415378470 learned_addr learned my addr 172.21.15.107:0/3415378470 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:02:09.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f4346ffd640 1 --2- 172.21.15.107:0/3415378470 >> v2:172.21.15.110:3300/0 conn(0x7f43480720c0 0x7f43480c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941679299 0 0) 0x7f4348154860 con 0x7f4348157e50 2023-10-25T00:02:09.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4330003530 con 0x7f4348157e50 2023-10-25T00:02:09.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1417002166 0 0) 0x7f4330003530 con 0x7f4348157e50 2023-10-25T00:02:09.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 >> v2:172.21.15.110:3300/0 conn(0x7f43480720c0 msgr2=0x7f43480c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:09.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f43467fc640 1 --2- 172.21.15.107:0/3415378470 >> v2:172.21.15.110:3300/0 conn(0x7f43480720c0 0x7f43480c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 >> v1:172.21.15.110:6789/0 conn(0x7f434813dc20 legacy=0x7f434806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:09.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4348154e00 con 0x7f4348157e50 2023-10-25T00:02:09.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.407+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f4340001a00 con 0x7f4348157e50 2023-10-25T00:02:09.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 >> v1:172.21.15.107:6789/0 conn(0x7f4348157e50 legacy=0x7f434815f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:09.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f43467fc640 1 --2- 172.21.15.107:0/3415378470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4330003980 0x7f4330003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f43467fc640 1 --2- 172.21.15.107:0/3415378470 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f43300047d0 0x7f4330004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f43467fc640 1 --2- 172.21.15.107:0/3415378470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4330005200 0x7f433000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4348154b20 con 0x7f4330005200 2023-10-25T00:02:09.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433000c160 con 0x7f4330003980 2023-10-25T00:02:09.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f433000c320 con 0x7f43300047d0 2023-10-25T00:02:09.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f4347fff640 1 --2- 172.21.15.107:0/3415378470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4330005200 0x7f433000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f4346ffd640 1 --2- 172.21.15.107:0/3415378470 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f43300047d0 0x7f4330004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f4347fff640 1 -- 172.21.15.107:0/3415378470 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f43300047d0 msgr2=0x7f4330004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f4347fff640 1 --2- 172.21.15.107:0/3415378470 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f43300047d0 0x7f4330004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f4347fff640 1 -- 172.21.15.107:0/3415378470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4330003980 msgr2=0x7f4330003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:09.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f4347fff640 1 --2- 172.21.15.107:0/3415378470 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4330003980 0x7f4330003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f4347fff640 1 -- 172.21.15.107:0/3415378470 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4330003530 con 0x7f4330005200 2023-10-25T00:02:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f4347fff640 1 -- 172.21.15.107:0/3415378470 reap_dead start 2023-10-25T00:02:09.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f4347fff640 1 --2- 172.21.15.107:0/3415378470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4330005200 0x7f433000b990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f434000a830 tx=0x7f434001d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed57ae3a7989330 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400ac620 con 0x7f4330005200 2023-10-25T00:02:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43400a8070 con 0x7f4330005200 2023-10-25T00:02:09.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.411+0000 7f43467fc640 1 -- 172.21.15.107:0/3415378470 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400ac9e0 con 0x7f4330005200 2023-10-25T00:02:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 -- 172.21.15.107:0/3415378470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4330005200 msgr2=0x7f433000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 --2- 172.21.15.107:0/3415378470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4330005200 0x7f433000b990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f434000a830 tx=0x7f434001d360 comp rx=0 tx=0).stop 2023-10-25T00:02:09.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 -- 172.21.15.107:0/3415378470 shutdown_connections 2023-10-25T00:02:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 --2- 172.21.15.107:0/3415378470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4330005200 0x7f433000b990 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 -- 172.21.15.107:0/3415378470 >> 172.21.15.107:0/3415378470 conn(0x7f43480c39a0 msgr2=0x7f43480c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 -- 172.21.15.107:0/3415378470 shutdown_connections 2023-10-25T00:02:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 -- 172.21.15.107:0/3415378470 wait complete. 2023-10-25T00:02:09.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 Processor -- start 2023-10-25T00:02:09.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 -- start start 2023-10-25T00:02:09.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4330005200 0x7f43481315c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4348135550 0x7f43480c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43480c6b20 0x7f43481f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4348155e70 con 0x7f43480c6b20 2023-10-25T00:02:09.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4348156ca0 con 0x7f4330005200 2023-10-25T00:02:09.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43480c5b30 con 0x7f4348135550 2023-10-25T00:02:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f4347fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43480c6b20 0x7f43481f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f4347fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43480c6b20 0x7f43481f0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45844/0 (socket says 172.21.15.107:45844) 2023-10-25T00:02:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f4347fff640 1 -- 172.21.15.107:0/1950451587 learned_addr learned my addr 172.21.15.107:0/1950451587 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f43477fe640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4330005200 0x7f43481315c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f4346ffd640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4348135550 0x7f43480c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f4347fff640 1 -- 172.21.15.107:0/1950451587 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4348135550 msgr2=0x7f43480c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f4347fff640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4348135550 0x7f43480c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f4347fff640 1 -- 172.21.15.107:0/1950451587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4330005200 msgr2=0x7f43481315c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:09.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f4347fff640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4330005200 0x7f43481315c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f4347fff640 1 -- 172.21.15.107:0/1950451587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43480c60b0 con 0x7f43480c6b20 2023-10-25T00:02:09.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f4347fff640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43480c6b20 0x7f43481f0640 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f434000a830 tx=0x7f4330005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:09.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f432ffff640 1 -- 172.21.15.107:0/1950451587 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400b0070 con 0x7f43480c6b20 2023-10-25T00:02:09.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f432ffff640 1 -- 172.21.15.107:0/1950451587 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4340001a40 con 0x7f43480c6b20 2023-10-25T00:02:09.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f432ffff640 1 -- 172.21.15.107:0/1950451587 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43400b3040 con 0x7f43480c6b20 2023-10-25T00:02:09.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 -- 172.21.15.107:0/1950451587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4348155b90 con 0x7f43480c6b20 2023-10-25T00:02:09.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.415+0000 7f434dfe8640 1 -- 172.21.15.107:0/1950451587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4348131b00 con 0x7f43480c6b20 2023-10-25T00:02:09.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.419+0000 7f434dfe8640 1 -- 172.21.15.107:0/1950451587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4348155fd0 con 0x7f43480c6b20 2023-10-25T00:02:09.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.419+0000 7f432ffff640 1 -- 172.21.15.107:0/1950451587 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f43400aa070 con 0x7f43480c6b20 2023-10-25T00:02:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.419+0000 7f432ffff640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4334065e90 0x7f4334068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.419+0000 7f432ffff640 1 -- 172.21.15.107:0/1950451587 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7f434011d510 con 0x7f43480c6b20 2023-10-25T00:02:09.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.419+0000 7f43477fe640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4334065e90 0x7f4334068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.419+0000 7f43477fe640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4334065e90 0x7f4334068350 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4338004db0 tx=0x7f4338009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:09.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.423+0000 7f432ffff640 1 -- 172.21.15.107:0/1950451587 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4348155fd0 con 0x7f43480c6b20 2023-10-25T00:02:09.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.631+0000 7f434dfe8640 1 -- 172.21.15.107:0/1950451587 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4348155fd0 con 0x7f43480c6b20 2023-10-25T00:02:09.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.631+0000 7f432ffff640 1 -- 172.21.15.107:0/1950451587 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5771 (secure 0 0 0) 0x7f4348155fd0 con 0x7f43480c6b20 2023-10-25T00:02:09.633 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:09.634 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 80 2023-10-25T00:02:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 -- 172.21.15.107:0/1950451587 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4334065e90 msgr2=0x7f4334068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:09.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4334065e90 0x7f4334068350 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4338004db0 tx=0x7f4338009040 comp rx=0 tx=0).stop 2023-10-25T00:02:09.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 -- 172.21.15.107:0/1950451587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43480c6b20 msgr2=0x7f43481f0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:09.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43480c6b20 0x7f43481f0640 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f434000a830 tx=0x7f4330005800 comp rx=0 tx=0).stop 2023-10-25T00:02:09.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 -- 172.21.15.107:0/1950451587 shutdown_connections 2023-10-25T00:02:09.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43480c6b20 0x7f43481f0640 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4334065e90 0x7f4334068350 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4348135550 0x7f43480c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 --2- 172.21.15.107:0/1950451587 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4330005200 0x7f43481315c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 -- 172.21.15.107:0/1950451587 >> 172.21.15.107:0/1950451587 conn(0x7f43480c39a0 msgr2=0x7f43480c03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:09.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 -- 172.21.15.107:0/1950451587 shutdown_connections 2023-10-25T00:02:09.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.635+0000 7f434dfe8640 1 -- 172.21.15.107:0/1950451587 wait complete. 2023-10-25T00:02:09.655 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2023-10-25T00:02:09.656 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-25T00:02:09.656 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:02:09.671 INFO:tasks.ceph.mds.b:Stopped 2023-10-25T00:02:09.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:09.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be33a0640 1 Processor -- start 2023-10-25T00:02:09.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be33a0640 1 -- start start 2023-10-25T00:02:09.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be33a0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6bdc0aa140 0x7f6bdc0aa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be33a0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc066000 con 0x7f6bdc0ac020 2023-10-25T00:02:09.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be33a0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc0aaa60 con 0x7f6bdc0c26f0 2023-10-25T00:02:09.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be33a0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bdc0c0f40 con 0x7f6bdc0aa140 2023-10-25T00:02:09.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be093b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6bdc0ac020 0x7f6bdc0c1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41160/0 (socket says 172.21.15.107:41160) 2023-10-25T00:02:09.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be093b640 1 -- 172.21.15.107:0/2149455833 learned_addr learned my addr 172.21.15.107:0/2149455833 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:02:09.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be113c640 1 --2- 172.21.15.107:0/2149455833 >> v2:172.21.15.110:3300/0 conn(0x7f6bdc0aa140 0x7f6bdc0aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109624423 0 0) 0x7f6bdc066000 con 0x7f6bdc0ac020 2023-10-25T00:02:09.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc8003530 con 0x7f6bdc0ac020 2023-10-25T00:02:09.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2814053090 0 0) 0x7f6bc8003530 con 0x7f6bdc0ac020 2023-10-25T00:02:09.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 >> v2:172.21.15.110:3300/0 conn(0x7f6bdc0aa140 msgr2=0x7f6bdc0aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:09.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 --2- 172.21.15.107:0/2149455833 >> v2:172.21.15.110:3300/0 conn(0x7f6bdc0aa140 0x7f6bdc0aa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 >> v1:172.21.15.110:6789/0 conn(0x7f6bdc0c26f0 legacy=0x7f6bdc06c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:09.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc0c1220 con 0x7f6bdc0ac020 2023-10-25T00:02:09.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6bd00016a0 con 0x7f6bdc0ac020 2023-10-25T00:02:09.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 >> v1:172.21.15.107:6789/0 conn(0x7f6bdc0ac020 legacy=0x7f6bdc0c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:09.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bc8003980 0x7f6bc8003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bc8004380 0x7f6bc8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bc8005050 0x7f6bc800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc0c0f40 con 0x7f6bc8005050 2023-10-25T00:02:09.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc800e1c0 con 0x7f6bc8003980 2023-10-25T00:02:09.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc800e380 con 0x7f6bc8004380 2023-10-25T00:02:09.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be193d640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bc8005050 0x7f6bc800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be113c640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bc8003980 0x7f6bc8003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be093b640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bc8004380 0x7f6bc8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be193d640 1 -- 172.21.15.107:0/2149455833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bc8004380 msgr2=0x7f6bc8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:09.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be193d640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bc8004380 0x7f6bc8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be193d640 1 -- 172.21.15.107:0/2149455833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bc8003980 msgr2=0x7f6bc8003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:09.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be193d640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bc8003980 0x7f6bc8003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be193d640 1 -- 172.21.15.107:0/2149455833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc8003530 con 0x7f6bc8005050 2023-10-25T00:02:09.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.783+0000 7f6be093b640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bc8004380 0x7f6bc8004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:09.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6be193d640 1 -- 172.21.15.107:0/2149455833 reap_dead start 2023-10-25T00:02:09.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6be193d640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bc8005050 0x7f6bc800b990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4001f60 tx=0x7f6bd401c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37054cf19de88e0b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:09.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40a5750 con 0x7f6bc8005050 2023-10-25T00:02:09.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd40a58b0 con 0x7f6bc8005050 2023-10-25T00:02:09.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6bcffff640 1 -- 172.21.15.107:0/2149455833 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40a5c40 con 0x7f6bc8005050 2023-10-25T00:02:09.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6be33a0640 1 -- 172.21.15.107:0/2149455833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bc8005050 msgr2=0x7f6bc800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:09.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6be33a0640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bc8005050 0x7f6bc800b990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4001f60 tx=0x7f6bd401c860 comp rx=0 tx=0).stop 2023-10-25T00:02:09.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6be33a0640 1 -- 172.21.15.107:0/2149455833 shutdown_connections 2023-10-25T00:02:09.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6be33a0640 1 --2- 172.21.15.107:0/2149455833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bc8005050 0x7f6bc800b990 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6be33a0640 1 -- 172.21.15.107:0/2149455833 >> 172.21.15.107:0/2149455833 conn(0x7f6bdc0c65c0 msgr2=0x7f6bdc0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:09.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6be33a0640 1 -- 172.21.15.107:0/2149455833 shutdown_connections 2023-10-25T00:02:09.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6be33a0640 1 -- 172.21.15.107:0/2149455833 wait complete. 2023-10-25T00:02:09.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6be33a0640 1 Processor -- start 2023-10-25T00:02:09.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.787+0000 7f6be33a0640 1 -- start start 2023-10-25T00:02:09.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be33a0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bc8005050 0x7f6bdc1e5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be33a0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bdc0bd690 0x7f6bdc0bdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be33a0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bdc071230 0x7f6bdc0716b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be33a0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc1e5f60 con 0x7f6bdc0bd690 2023-10-25T00:02:09.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be33a0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc0bcbf0 con 0x7f6bc8005050 2023-10-25T00:02:09.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be33a0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc1d6ca0 con 0x7f6bdc071230 2023-10-25T00:02:09.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be093b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bdc0bd690 0x7f6bdc0bdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be093b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bdc0bd690 0x7f6bdc0bdb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45868/0 (socket says 172.21.15.107:45868) 2023-10-25T00:02:09.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be093b640 1 -- 172.21.15.107:0/2085512785 learned_addr learned my addr 172.21.15.107:0/2085512785 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:09.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be113c640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bc8005050 0x7f6bdc1e5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be193d640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bdc071230 0x7f6bdc0716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be093b640 1 -- 172.21.15.107:0/2085512785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bdc071230 msgr2=0x7f6bdc0716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:09.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be093b640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bdc071230 0x7f6bdc0716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be093b640 1 -- 172.21.15.107:0/2085512785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bc8005050 msgr2=0x7f6bdc1e5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:09.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be093b640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bc8005050 0x7f6bdc1e5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:09.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be093b640 1 -- 172.21.15.107:0/2085512785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc0bce40 con 0x7f6bdc0bd690 2023-10-25T00:02:09.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be093b640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bdc0bd690 0x7f6bdc0bdb10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0000f70 tx=0x7f6bd0010f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:09.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6bcdffb640 1 -- 172.21.15.107:0/2085512785 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd000ec70 con 0x7f6bdc0bd690 2023-10-25T00:02:09.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6bcdffb640 1 -- 172.21.15.107:0/2085512785 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd000edd0 con 0x7f6bdc0bd690 2023-10-25T00:02:09.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6bcdffb640 1 -- 172.21.15.107:0/2085512785 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd00145e0 con 0x7f6bdc0bd690 2023-10-25T00:02:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be33a0640 1 -- 172.21.15.107:0/2085512785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc0bd120 con 0x7f6bdc0bd690 2023-10-25T00:02:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be33a0640 1 -- 172.21.15.107:0/2085512785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc1d6900 con 0x7f6bdc0bd690 2023-10-25T00:02:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be33a0640 1 -- 172.21.15.107:0/2085512785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc1d52b0 con 0x7f6bdc0bd690 2023-10-25T00:02:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6bcdffb640 1 -- 172.21.15.107:0/2085512785 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6bd0014740 con 0x7f6bdc0bd690 2023-10-25T00:02:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6bcdffb640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6bc0065e90 0x7f6bc0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:09.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6bcdffb640 1 -- 172.21.15.107:0/2085512785 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7f6bd00869f0 con 0x7f6bdc0bd690 2023-10-25T00:02:09.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be113c640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6bc0065e90 0x7f6bc0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:09.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.791+0000 7f6be113c640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6bc0065e90 0x7f6bc0068350 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80099c0 tx=0x7f6bd8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:09.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:09.799+0000 7f6bcdffb640 1 -- 172.21.15.107:0/2085512785 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6bdc1d52b0 con 0x7f6bdc0bd690 2023-10-25T00:02:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.031+0000 7f6be33a0640 1 -- 172.21.15.107:0/2085512785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bdc1d52b0 con 0x7f6bdc0bd690 2023-10-25T00:02:10.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.031+0000 7f6bcdffb640 1 -- 172.21.15.107:0/2085512785 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5771 (secure 0 0 0) 0x7f6bdc1d52b0 con 0x7f6bdc0bd690 2023-10-25T00:02:10.035 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:10.035 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 80 2023-10-25T00:02:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 -- 172.21.15.107:0/2085512785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6bc0065e90 msgr2=0x7f6bc0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6bc0065e90 0x7f6bc0068350 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80099c0 tx=0x7f6bd8008040 comp rx=0 tx=0).stop 2023-10-25T00:02:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 -- 172.21.15.107:0/2085512785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bdc0bd690 msgr2=0x7f6bdc0bdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bdc0bd690 0x7f6bdc0bdb10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0000f70 tx=0x7f6bd0010f30 comp rx=0 tx=0).stop 2023-10-25T00:02:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 -- 172.21.15.107:0/2085512785 shutdown_connections 2023-10-25T00:02:10.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6bc0065e90 0x7f6bc0068350 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:10.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6bdc071230 0x7f6bdc0716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:10.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6bdc0bd690 0x7f6bdc0bdb10 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:10.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 --2- 172.21.15.107:0/2085512785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bc8005050 0x7f6bdc1e5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:10.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 -- 172.21.15.107:0/2085512785 >> 172.21.15.107:0/2085512785 conn(0x7f6bdc0c65c0 msgr2=0x7f6bdc0c5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:10.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 -- 172.21.15.107:0/2085512785 shutdown_connections 2023-10-25T00:02:10.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:10.035+0000 7f6be33a0640 1 -- 172.21.15.107:0/2085512785 wait complete. 2023-10-25T00:02:10.055 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:02:12.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:12.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcc5f45640 1 Processor -- start 2023-10-25T00:02:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcc5f45640 1 -- start start 2023-10-25T00:02:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcc5f45640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00ac020 0x7fdcc00c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcc5f45640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc0066000 con 0x7fdcc00c2520 2023-10-25T00:02:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcc5f45640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc00c0f40 con 0x7fdcc00aa140 2023-10-25T00:02:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcc5f45640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00c1220 con 0x7fdcc00ac020 2023-10-25T00:02:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdcc00c2520 0x7fdcc006c490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41170/0 (socket says 172.21.15.107:41170) 2023-10-25T00:02:12.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbffff640 1 -- 172.21.15.107:0/3219406168 learned_addr learned my addr 172.21.15.107:0/3219406168 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:02:12.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbeffd640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00ac020 0x7fdcc00c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:12.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932285819 0 0) 0x7fdcc0066000 con 0x7fdcc00c2520 2023-10-25T00:02:12.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca8003530 con 0x7fdcc00c2520 2023-10-25T00:02:12.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3020095028 0 0) 0x7fdca8003530 con 0x7fdcc00c2520 2023-10-25T00:02:12.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00ac020 msgr2=0x7fdcc00c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:12.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00ac020 0x7fdcc00c1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:12.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 >> v1:172.21.15.110:6789/0 conn(0x7fdcc00aa140 legacy=0x7fdcc00aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:12.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00c1500 con 0x7fdcc00c2520 2023-10-25T00:02:12.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fdcb8001a00 con 0x7fdcc00c2520 2023-10-25T00:02:12.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 >> v1:172.21.15.107:6789/0 conn(0x7fdcc00c2520 legacy=0x7fdcc006c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:12.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 0x7fdca8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:12.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00ac020 0x7fdca80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:12.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdca8004d10 0x7fdca8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:12.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00c1220 con 0x7fdca80039b0 2023-10-25T00:02:12.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca800ba40 con 0x7fdca8004d10 2023-10-25T00:02:12.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdca800bbb0 con 0x7fdcc00ac020 2023-10-25T00:02:12.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbf7fe640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 0x7fdca8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:12.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00ac020 msgr2=0x7fdca80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:12.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbeffd640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00ac020 0x7fdca80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:12.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbffff640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdca8004d10 0x7fdca8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:12.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbf7fe640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00ac020 0x7fdca80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:12.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdca8004d10 msgr2=0x7fdca8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:12.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbf7fe640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdca8004d10 0x7fdca8005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:12.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/3219406168 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00c1500 con 0x7fdca80039b0 2023-10-25T00:02:12.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.195+0000 7fdcbeffd640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00ac020 0x7fdca80047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:12.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcbf7fe640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 0x7fdca8003dd0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0001780 tx=0x7fdcb001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74e07e72a9d2cc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:12.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb00052d0 con 0x7fdca80039b0 2023-10-25T00:02:12.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb00a7050 con 0x7fdca80039b0 2023-10-25T00:02:12.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcbe7fc640 1 -- 172.21.15.107:0/3219406168 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0003600 con 0x7fdca80039b0 2023-10-25T00:02:12.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcc5f45640 1 -- 172.21.15.107:0/3219406168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 msgr2=0x7fdca8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:12.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcc5f45640 1 --2- 172.21.15.107:0/3219406168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 0x7fdca8003dd0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0001780 tx=0x7fdcb001c3d0 comp rx=0 tx=0).stop 2023-10-25T00:02:12.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcbffff640 1 -- 172.21.15.107:0/3219406168 reap_dead start 2023-10-25T00:02:12.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcc5f45640 1 -- 172.21.15.107:0/3219406168 shutdown_connections 2023-10-25T00:02:12.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcc5f45640 1 -- 172.21.15.107:0/3219406168 >> 172.21.15.107:0/3219406168 conn(0x7fdcc00c65c0 msgr2=0x7fdcc00c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:12.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcc5f45640 1 -- 172.21.15.107:0/3219406168 shutdown_connections 2023-10-25T00:02:12.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcc5f45640 1 -- 172.21.15.107:0/3219406168 wait complete. 2023-10-25T00:02:12.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcc5f45640 1 Processor -- start 2023-10-25T00:02:12.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.199+0000 7fdcc5f45640 1 -- start start 2023-10-25T00:02:12.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcc5f45640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 0x7fdcc00bcbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:12.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcc5f45640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00bc460 0x7fdcc01f0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:12.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcc5f45640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcc00b9b80 0x7fdcc0151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:12.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcc5f45640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc014c6c0 con 0x7fdca80039b0 2023-10-25T00:02:12.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcc5f45640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc00ba8d0 con 0x7fdcc00b9b80 2023-10-25T00:02:12.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcc5f45640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc006c820 con 0x7fdcc00bc460 2023-10-25T00:02:12.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 0x7fdcc00bcbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:12.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 0x7fdcc00bcbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45888/0 (socket says 172.21.15.107:45888) 2023-10-25T00:02:12.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/1104316501 learned_addr learned my addr 172.21.15.107:0/1104316501 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:12.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbeffd640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00bc460 0x7fdcc01f0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:12.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbffff640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcc00b9b80 0x7fdcc0151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:12.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/1104316501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00bc460 msgr2=0x7fdcc01f0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:12.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbf7fe640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00bc460 0x7fdcc01f0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:12.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/1104316501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcc00b9b80 msgr2=0x7fdcc0151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:12.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbf7fe640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcc00b9b80 0x7fdcc0151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:12.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbf7fe640 1 -- 172.21.15.107:0/1104316501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00bd0f0 con 0x7fdca80039b0 2023-10-25T00:02:12.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbf7fe640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 0x7fdcc00bcbb0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0005040 tx=0x7fdca8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:12.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdca7fff640 1 -- 172.21.15.107:0/1104316501 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0007040 con 0x7fdca80039b0 2023-10-25T00:02:12.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdca7fff640 1 -- 172.21.15.107:0/1104316501 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb00071a0 con 0x7fdca80039b0 2023-10-25T00:02:12.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcc5f45640 1 -- 172.21.15.107:0/1104316501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc0158030 con 0x7fdca80039b0 2023-10-25T00:02:12.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdca7fff640 1 -- 172.21.15.107:0/1104316501 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0007510 con 0x7fdca80039b0 2023-10-25T00:02:12.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcc5f45640 1 -- 172.21.15.107:0/1104316501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc00ba170 con 0x7fdca80039b0 2023-10-25T00:02:12.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcc5f45640 1 -- 172.21.15.107:0/1104316501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc00c7580 con 0x7fdca80039b0 2023-10-25T00:02:12.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdca7fff640 1 -- 172.21.15.107:0/1104316501 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdcb00067e0 con 0x7fdca80039b0 2023-10-25T00:02:12.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdca7fff640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdcac065e20 0x7fdcac0682e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:12.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbeffd640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdcac065e20 0x7fdcac0682e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:12.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdca7fff640 1 -- 172.21.15.107:0/1104316501 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7fdcb011e230 con 0x7fdca80039b0 2023-10-25T00:02:12.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.203+0000 7fdcbeffd640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdcac065e20 0x7fdcac0682e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00c6d20 tx=0x7fdcb4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:12.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.207+0000 7fdca7fff640 1 -- 172.21.15.107:0/1104316501 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdcb00eb100 con 0x7fdca80039b0 2023-10-25T00:02:12.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.427+0000 7fdcc5f45640 1 -- 172.21.15.107:0/1104316501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcc00746e0 con 0x7fdca80039b0 2023-10-25T00:02:12.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.427+0000 7fdca7fff640 1 -- 172.21.15.107:0/1104316501 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5771 (secure 0 0 0) 0x7fdcb00ee720 con 0x7fdca80039b0 2023-10-25T00:02:12.430 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:12.430 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:12.431 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 80 2023-10-25T00:02:12.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 -- 172.21.15.107:0/1104316501 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdcac065e20 msgr2=0x7fdcac0682e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:12.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdcac065e20 0x7fdcac0682e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00c6d20 tx=0x7fdcb4008040 comp rx=0 tx=0).stop 2023-10-25T00:02:12.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 -- 172.21.15.107:0/1104316501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 msgr2=0x7fdcc00bcbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:12.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 0x7fdcc00bcbb0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0005040 tx=0x7fdca8005800 comp rx=0 tx=0).stop 2023-10-25T00:02:12.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 -- 172.21.15.107:0/1104316501 shutdown_connections 2023-10-25T00:02:12.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcc00b9b80 0x7fdcc0151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:12.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdcac065e20 0x7fdcac0682e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:12.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdcc00bc460 0x7fdcc01f0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:12.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 --2- 172.21.15.107:0/1104316501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdca80039b0 0x7fdcc00bcbb0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:12.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 -- 172.21.15.107:0/1104316501 >> 172.21.15.107:0/1104316501 conn(0x7fdcc00c65c0 msgr2=0x7fdcc00a94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:12.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 -- 172.21.15.107:0/1104316501 shutdown_connections 2023-10-25T00:02:12.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:12.431+0000 7fdcc5f45640 1 -- 172.21.15.107:0/1104316501 wait complete. 2023-10-25T00:02:12.453 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:02:14.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:14.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.583+0000 7f16cbf14640 1 Processor -- start 2023-10-25T00:02:14.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.583+0000 7f16cbf14640 1 -- start start 2023-10-25T00:02:14.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16cbf14640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 0x7f16c4153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:14.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16cbf14640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f16c40c18a0 0x7f16c40c1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:14.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16cbf14640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c4072330 con 0x7f16c414fc30 2023-10-25T00:02:14.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16cbf14640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c4155ca0 con 0x7f16c40c18a0 2023-10-25T00:02:14.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16cbf14640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c4155f80 con 0x7f16c40c5c20 2023-10-25T00:02:14.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16ca4b1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f16c40c18a0 0x7f16c40c1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:14.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16ca4b1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f16c40c18a0 0x7f16c40c1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40112/0 (socket says 172.21.15.107:40112) 2023-10-25T00:02:14.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16ca4b1640 1 -- 172.21.15.107:0/2774889692 learned_addr learned my addr 172.21.15.107:0/2774889692 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:14.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16c94af640 1 --2- 172.21.15.107:0/2774889692 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 0x7f16c4153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:14.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16ca4b1640 1 -- 172.21.15.107:0/2774889692 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 msgr2=0x7f16c4153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:14.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16ca4b1640 1 --2- 172.21.15.107:0/2774889692 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 0x7f16c4153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:14.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16ca4b1640 1 -- 172.21.15.107:0/2774889692 >> v1:172.21.15.107:6789/0 conn(0x7f16c414fc30 legacy=0x7f16c406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:14.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16ca4b1640 1 -- 172.21.15.107:0/2774889692 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c4156260 con 0x7f16c40c18a0 2023-10-25T00:02:14.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16c8cae640 1 -- 172.21.15.107:0/2774889692 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683138427 0 0) 0x7f16c4072330 con 0x7f16c414fc30 2023-10-25T00:02:14.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16c94af640 1 --2- 172.21.15.107:0/2774889692 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 0x7f16c4153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:14.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16ca4b1640 1 --2- 172.21.15.107:0/2774889692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c40c18a0 0x7f16c40c1d10 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009e50 tx=0x7f16bc01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2dd7395262c0071 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:14.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.587+0000 7f16c8cae640 1 -- 172.21.15.107:0/2774889692 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc0a5810 con 0x7f16c40c18a0 2023-10-25T00:02:14.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c8cae640 1 -- 172.21.15.107:0/2774889692 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16bc0a5970 con 0x7f16c40c18a0 2023-10-25T00:02:14.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c8cae640 1 -- 172.21.15.107:0/2774889692 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16bc0a5d00 con 0x7f16c40c18a0 2023-10-25T00:02:14.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 -- 172.21.15.107:0/2774889692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c40c18a0 msgr2=0x7f16c40c1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:14.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 --2- 172.21.15.107:0/2774889692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c40c18a0 0x7f16c40c1d10 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009e50 tx=0x7f16bc01c950 comp rx=0 tx=0).stop 2023-10-25T00:02:14.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 -- 172.21.15.107:0/2774889692 shutdown_connections 2023-10-25T00:02:14.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 --2- 172.21.15.107:0/2774889692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c40c18a0 0x7f16c40c1d10 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:14.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 --2- 172.21.15.107:0/2774889692 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 0x7f16c4153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:14.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 -- 172.21.15.107:0/2774889692 >> 172.21.15.107:0/2774889692 conn(0x7f16c40c39a0 msgr2=0x7f16c40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:14.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 -- 172.21.15.107:0/2774889692 shutdown_connections 2023-10-25T00:02:14.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 -- 172.21.15.107:0/2774889692 wait complete. 2023-10-25T00:02:14.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 Processor -- start 2023-10-25T00:02:14.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 -- start start 2023-10-25T00:02:14.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c40c18a0 0x7f16c4137270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:14.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 0x7f16c415c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:14.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c414fc30 0x7f16c40bf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:14.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c40bfd00 con 0x7f16c40c18a0 2023-10-25T00:02:14.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c41eb040 con 0x7f16c414fc30 2023-10-25T00:02:14.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c41eb320 con 0x7f16c40c5c20 2023-10-25T00:02:14.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c9cb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c40c18a0 0x7f16c4137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:14.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c9cb0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c40c18a0 0x7f16c4137270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40114/0 (socket says 172.21.15.107:40114) 2023-10-25T00:02:14.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c9cb0640 1 -- 172.21.15.107:0/3103665108 learned_addr learned my addr 172.21.15.107:0/3103665108 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:14.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c94af640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 0x7f16c415c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:14.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16ca4b1640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c414fc30 0x7f16c40bf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:14.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c9cb0640 1 -- 172.21.15.107:0/3103665108 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 msgr2=0x7f16c415c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:14.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c9cb0640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 0x7f16c415c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:14.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c9cb0640 1 -- 172.21.15.107:0/3103665108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c414fc30 msgr2=0x7f16c40bf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:14.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c9cb0640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c414fc30 0x7f16c40bf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:14.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c9cb0640 1 -- 172.21.15.107:0/3103665108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc009b30 con 0x7f16c40c18a0 2023-10-25T00:02:14.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c94af640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 0x7f16c415c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:14.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16ca4b1640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c414fc30 0x7f16c40bf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:14.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16c9cb0640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c40c18a0 0x7f16c4137270 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f16c000c4d0 tx=0x7f16c000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:14.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16ba7fc640 1 -- 172.21.15.107:0/3103665108 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c0005480 con 0x7f16c40c18a0 2023-10-25T00:02:14.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16ba7fc640 1 -- 172.21.15.107:0/3103665108 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16c0003600 con 0x7f16c40c18a0 2023-10-25T00:02:14.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 -- 172.21.15.107:0/3103665108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c40be590 con 0x7f16c40c18a0 2023-10-25T00:02:14.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16ba7fc640 1 -- 172.21.15.107:0/3103665108 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c00039b0 con 0x7f16c40c18a0 2023-10-25T00:02:14.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.591+0000 7f16cbf14640 1 -- 172.21.15.107:0/3103665108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c40beb50 con 0x7f16c40c18a0 2023-10-25T00:02:14.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.595+0000 7f16cbf14640 1 -- 172.21.15.107:0/3103665108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c40746e0 con 0x7f16c40c18a0 2023-10-25T00:02:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.595+0000 7f16ba7fc640 1 -- 172.21.15.107:0/3103665108 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f16c0003b10 con 0x7f16c40c18a0 2023-10-25T00:02:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.595+0000 7f16ba7fc640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f16a8065ee0 0x7f16a80683a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.595+0000 7f16ba7fc640 1 -- 172.21.15.107:0/3103665108 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7f16c0083550 con 0x7f16c40c18a0 2023-10-25T00:02:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.595+0000 7f16c94af640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f16a8065ee0 0x7f16a80683a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.595+0000 7f16c94af640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f16a8065ee0 0x7f16a80683a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f16c414f050 tx=0x7f16b4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:14.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.603+0000 7f16ba7fc640 1 -- 172.21.15.107:0/3103665108 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f16c0050590 con 0x7f16c40c18a0 2023-10-25T00:02:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.811+0000 7f16cbf14640 1 -- 172.21.15.107:0/3103665108 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16c40c5a00 con 0x7f16c40c18a0 2023-10-25T00:02:14.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.811+0000 7f16ba7fc640 1 -- 172.21.15.107:0/3103665108 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5771 (secure 0 0 0) 0x7f16c0017020 con 0x7f16c40c18a0 2023-10-25T00:02:14.814 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:14.814 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:14.816 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 80 2023-10-25T00:02:14.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 -- 172.21.15.107:0/3103665108 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f16a8065ee0 msgr2=0x7f16a80683a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:14.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f16a8065ee0 0x7f16a80683a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f16c414f050 tx=0x7f16b4008040 comp rx=0 tx=0).stop 2023-10-25T00:02:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 -- 172.21.15.107:0/3103665108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c40c18a0 msgr2=0x7f16c4137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c40c18a0 0x7f16c4137270 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f16c000c4d0 tx=0x7f16c000c990 comp rx=0 tx=0).stop 2023-10-25T00:02:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 -- 172.21.15.107:0/3103665108 shutdown_connections 2023-10-25T00:02:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16c414fc30 0x7f16c40bf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f16a8065ee0 0x7f16a80683a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16c40c5c20 0x7f16c415c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 --2- 172.21.15.107:0/3103665108 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16c40c18a0 0x7f16c4137270 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 -- 172.21.15.107:0/3103665108 >> 172.21.15.107:0/3103665108 conn(0x7f16c40c39a0 msgr2=0x7f16c40ab990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 -- 172.21.15.107:0/3103665108 shutdown_connections 2023-10-25T00:02:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:14.815+0000 7f16cbf14640 1 -- 172.21.15.107:0/3103665108 wait complete. 2023-10-25T00:02:14.833 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:02:16.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f3354fc0640 1 Processor -- start 2023-10-25T00:02:16.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f3354fc0640 1 -- start start 2023-10-25T00:02:16.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f3354fc0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3350154860 0x7f3350158230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f3354fc0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33500adf70 con 0x7f3350071ce0 2023-10-25T00:02:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f3354fc0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f335006e710 con 0x7f33500720c0 2023-10-25T00:02:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f3354fc0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335006d900 con 0x7f3350154860 2023-10-25T00:02:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334e59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3350071ce0 0x7f33500ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45426/0 (socket says 172.21.15.107:45426) 2023-10-25T00:02:16.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334e59c640 1 -- 172.21.15.107:0/2693220041 learned_addr learned my addr 172.21.15.107:0/2693220041 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:02:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334ed9d640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3350154860 0x7f3350158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786569876 0 0) 0x7f33500adf70 con 0x7f3350071ce0 2023-10-25T00:02:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f333c003530 con 0x7f3350071ce0 2023-10-25T00:02:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3893741442 0 0) 0x7f333c003530 con 0x7f3350071ce0 2023-10-25T00:02:16.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3350154860 msgr2=0x7f3350158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3350154860 0x7f3350158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 >> v1:172.21.15.110:6789/0 conn(0x7f33500720c0 legacy=0x7f335013d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3350154d20 con 0x7f3350071ce0 2023-10-25T00:02:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f3348001680 con 0x7f3350071ce0 2023-10-25T00:02:16.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 >> v1:172.21.15.107:6789/0 conn(0x7f3350071ce0 legacy=0x7f33500ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3350154860 0x7f333c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f333c003f30 0x7f333c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c004d10 0x7f333c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c00bb20 con 0x7f333c004d10 2023-10-25T00:02:16.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c00bc60 con 0x7f333c003f30 2023-10-25T00:02:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333c00be20 con 0x7f3350154860 2023-10-25T00:02:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334ed9d640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c004d10 0x7f333c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334e59c640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3350154860 0x7f333c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334dd9b640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f333c003f30 0x7f333c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334ed9d640 1 -- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3350154860 msgr2=0x7f333c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334ed9d640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3350154860 0x7f333c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334ed9d640 1 -- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f333c003f30 msgr2=0x7f333c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334ed9d640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f333c003f30 0x7f333c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334ed9d640 1 -- 172.21.15.107:0/2693220041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333c003530 con 0x7f333c004d10 2023-10-25T00:02:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.979+0000 7f334dd9b640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f333c003f30 0x7f333c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.983+0000 7f334ed9d640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c004d10 0x7f333c005190 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3340009f90 tx=0x7f334001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af982d7fa18d23 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.983+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33400a5b30 con 0x7f333c004d10 2023-10-25T00:02:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.983+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334000f040 con 0x7f333c004d10 2023-10-25T00:02:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.983+0000 7f334d59a640 1 -- 172.21.15.107:0/2693220041 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33400a5e90 con 0x7f333c004d10 2023-10-25T00:02:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.983+0000 7f3354fc0640 1 -- 172.21.15.107:0/2693220041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c004d10 msgr2=0x7f333c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.983+0000 7f3354fc0640 1 --2- 172.21.15.107:0/2693220041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f333c004d10 0x7f333c005190 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f3340009f90 tx=0x7f334001cc20 comp rx=0 tx=0).stop 2023-10-25T00:02:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.983+0000 7f334ed9d640 1 -- 172.21.15.107:0/2693220041 reap_dead start 2023-10-25T00:02:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.983+0000 7f3354fc0640 1 -- 172.21.15.107:0/2693220041 shutdown_connections 2023-10-25T00:02:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.983+0000 7f3354fc0640 1 -- 172.21.15.107:0/2693220041 >> 172.21.15.107:0/2693220041 conn(0x7f3350070f60 msgr2=0x7f3350071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:16.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.983+0000 7f3354fc0640 1 -- 172.21.15.107:0/2693220041 shutdown_connections 2023-10-25T00:02:16.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.983+0000 7f3354fc0640 1 -- 172.21.15.107:0/2693220041 wait complete. 2023-10-25T00:02:16.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3354fc0640 1 Processor -- start 2023-10-25T00:02:16.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3354fc0640 1 -- start start 2023-10-25T00:02:16.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3354fc0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3350157690 0x7f335014e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3354fc0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33501e9650 0x7f33501ec060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3354fc0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33501ec5a0 0x7f33501561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3354fc0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3350155d10 con 0x7f3350157690 2023-10-25T00:02:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3354fc0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33501e9ca0 con 0x7f33501ec5a0 2023-10-25T00:02:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3354fc0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33501e9f80 con 0x7f33501e9650 2023-10-25T00:02:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3350157690 0x7f335014e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3350157690 0x7f335014e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40134/0 (socket says 172.21.15.107:40134) 2023-10-25T00:02:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334e59c640 1 -- 172.21.15.107:0/2718440241 learned_addr learned my addr 172.21.15.107:0/2718440241 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334dd9b640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33501e9650 0x7f33501ec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334ed9d640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33501ec5a0 0x7f33501561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334e59c640 1 -- 172.21.15.107:0/2718440241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33501e9650 msgr2=0x7f33501ec060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:16.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334e59c640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33501e9650 0x7f33501ec060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:16.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334e59c640 1 -- 172.21.15.107:0/2718440241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33501ec5a0 msgr2=0x7f33501561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:16.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334e59c640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33501ec5a0 0x7f33501561a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:16.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334e59c640 1 -- 172.21.15.107:0/2718440241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33501ea260 con 0x7f3350157690 2023-10-25T00:02:16.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334dd9b640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33501e9650 0x7f33501ec060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:16.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334e59c640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3350157690 0x7f335014e570 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3348012830 tx=0x7f3348012cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:16.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3346ffd640 1 -- 172.21.15.107:0/2718440241 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33480107c0 con 0x7f3350157690 2023-10-25T00:02:16.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3346ffd640 1 -- 172.21.15.107:0/2718440241 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3348010920 con 0x7f3350157690 2023-10-25T00:02:16.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3354fc0640 1 -- 172.21.15.107:0/2718440241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33501ea540 con 0x7f3350157690 2023-10-25T00:02:16.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3346ffd640 1 -- 172.21.15.107:0/2718440241 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348010cb0 con 0x7f3350157690 2023-10-25T00:02:17.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3354fc0640 1 -- 172.21.15.107:0/2718440241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33501eaac0 con 0x7f3350157690 2023-10-25T00:02:17.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3354fc0640 1 -- 172.21.15.107:0/2718440241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3350155b40 con 0x7f3350157690 2023-10-25T00:02:17.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3346ffd640 1 -- 172.21.15.107:0/2718440241 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3348016480 con 0x7f3350157690 2023-10-25T00:02:17.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3346ffd640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3330065e60 0x7f3330068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:17.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f3346ffd640 1 -- 172.21.15.107:0/2718440241 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7f3348087590 con 0x7f3350157690 2023-10-25T00:02:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.987+0000 7f334dd9b640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3330065e60 0x7f3330068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.991+0000 7f334dd9b640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3330065e60 0x7f3330068320 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f33500c25a0 tx=0x7f3338009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:16.995+0000 7f3346ffd640 1 -- 172.21.15.107:0/2718440241 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3348054430 con 0x7f3350157690 2023-10-25T00:02:17.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.187+0000 7f3354fc0640 1 -- 172.21.15.107:0/2718440241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33501e9a30 con 0x7f3350157690 2023-10-25T00:02:17.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3346ffd640 1 -- 172.21.15.107:0/2718440241 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5771 (secure 0 0 0) 0x7f3348057a50 con 0x7f3350157690 2023-10-25T00:02:17.191 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:17.192 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:17.192 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 80 2023-10-25T00:02:17.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 -- 172.21.15.107:0/2718440241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3330065e60 msgr2=0x7f3330068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:17.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3330065e60 0x7f3330068320 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f33500c25a0 tx=0x7f3338009040 comp rx=0 tx=0).stop 2023-10-25T00:02:17.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 -- 172.21.15.107:0/2718440241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3350157690 msgr2=0x7f335014e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:17.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3350157690 0x7f335014e570 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f3348012830 tx=0x7f3348012cf0 comp rx=0 tx=0).stop 2023-10-25T00:02:17.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 -- 172.21.15.107:0/2718440241 shutdown_connections 2023-10-25T00:02:17.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33501ec5a0 0x7f33501561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:17.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3330065e60 0x7f3330068320 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:17.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f33501e9650 0x7f33501ec060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:17.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 --2- 172.21.15.107:0/2718440241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3350157690 0x7f335014e570 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:17.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 -- 172.21.15.107:0/2718440241 >> 172.21.15.107:0/2718440241 conn(0x7f3350070f60 msgr2=0x7f335013cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 -- 172.21.15.107:0/2718440241 shutdown_connections 2023-10-25T00:02:17.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:17.191+0000 7f3354fc0640 1 -- 172.21.15.107:0/2718440241 wait complete. 2023-10-25T00:02:17.212 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:02:19.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:19.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cfe63640 1 Processor -- start 2023-10-25T00:02:19.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cfe63640 1 -- start start 2023-10-25T00:02:19.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cfe63640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4c814c470 0x7fd4c814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:19.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cfe63640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd4c806f490 0x7fd4c8071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:19.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cfe63640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd4c8071fc0 0x7fd4c8152b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:19.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cfe63640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c81526e0 con 0x7fd4c806f490 2023-10-25T00:02:19.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cfe63640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c8155cd0 con 0x7fd4c8071fc0 2023-10-25T00:02:19.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cfe63640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c80c4380 con 0x7fd4c814c470 2023-10-25T00:02:19.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cd3fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd4c806f490 0x7fd4c8071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:19.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cd3fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd4c806f490 0x7fd4c8071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40148/0 (socket says 172.21.15.107:40148) 2023-10-25T00:02:19.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cd3fe640 1 -- 172.21.15.107:0/3915408226 learned_addr learned my addr 172.21.15.107:0/3915408226 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:19.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4ce400640 1 --2- 172.21.15.107:0/3915408226 >> v2:172.21.15.110:3300/0 conn(0x7fd4c8071fc0 0x7fd4c8152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:19.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cd3fe640 1 -- 172.21.15.107:0/3915408226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4c814c470 msgr2=0x7fd4c814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:19.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cdbff640 1 --2- 172.21.15.107:0/3915408226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4c814c470 0x7fd4c814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:19.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cd3fe640 1 --2- 172.21.15.107:0/3915408226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4c814c470 0x7fd4c814c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cd3fe640 1 -- 172.21.15.107:0/3915408226 >> v2:172.21.15.110:3300/0 conn(0x7fd4c8071fc0 msgr2=0x7fd4c8152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:19.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cd3fe640 1 --2- 172.21.15.107:0/3915408226 >> v2:172.21.15.110:3300/0 conn(0x7fd4c8071fc0 0x7fd4c8152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4cd3fe640 1 -- 172.21.15.107:0/3915408226 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c80c48b0 con 0x7fd4c806f490 2023-10-25T00:02:19.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.331+0000 7fd4ce400640 1 --2- 172.21.15.107:0/3915408226 >> v2:172.21.15.110:3300/0 conn(0x7fd4c8071fc0 0x7fd4c8152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:19.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cd3fe640 1 --2- 172.21.15.107:0/3915408226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c806f490 0x7fd4c8071a80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8073c60 tx=0x7fd4b801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40dc59c081a0bc46 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:19.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4ccbfd640 1 -- 172.21.15.107:0/3915408226 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80ab600 con 0x7fd4c806f490 2023-10-25T00:02:19.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4ccbfd640 1 -- 172.21.15.107:0/3915408226 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b80a7050 con 0x7fd4c806f490 2023-10-25T00:02:19.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4ccbfd640 1 -- 172.21.15.107:0/3915408226 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80ab980 con 0x7fd4c806f490 2023-10-25T00:02:19.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3915408226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c806f490 msgr2=0x7fd4c8071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:19.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cfe63640 1 --2- 172.21.15.107:0/3915408226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c806f490 0x7fd4c8071a80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd4c8073c60 tx=0x7fd4b801c3e0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3915408226 shutdown_connections 2023-10-25T00:02:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cfe63640 1 --2- 172.21.15.107:0/3915408226 >> v2:172.21.15.110:3300/0 conn(0x7fd4c8071fc0 0x7fd4c8152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cfe63640 1 --2- 172.21.15.107:0/3915408226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c806f490 0x7fd4c8071a80 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cfe63640 1 --2- 172.21.15.107:0/3915408226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4c814c470 0x7fd4c814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3915408226 >> 172.21.15.107:0/3915408226 conn(0x7fd4c8070f60 msgr2=0x7fd4c8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:19.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3915408226 shutdown_connections 2023-10-25T00:02:19.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3915408226 wait complete. 2023-10-25T00:02:19.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cfe63640 1 Processor -- start 2023-10-25T00:02:19.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.335+0000 7fd4cfe63640 1 -- start start 2023-10-25T00:02:19.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cfe63640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4c806f490 0x7fd4c806bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:19.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cfe63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c8071fc0 0x7fd4c80aced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:19.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cfe63640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c814c470 0x7fd4c80ad410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:19.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cfe63640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c81ecab0 con 0x7fd4c8071fc0 2023-10-25T00:02:19.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cfe63640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c81eda80 con 0x7fd4c814c470 2023-10-25T00:02:19.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cfe63640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c80ad950 con 0x7fd4c806f490 2023-10-25T00:02:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cd3fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c8071fc0 0x7fd4c80aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cd3fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c8071fc0 0x7fd4c80aced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40156/0 (socket says 172.21.15.107:40156) 2023-10-25T00:02:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cd3fe640 1 -- 172.21.15.107:0/3432068731 learned_addr learned my addr 172.21.15.107:0/3432068731 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4ce400640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c814c470 0x7fd4c80ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:19.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cdbff640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4c806f490 0x7fd4c806bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cd3fe640 1 -- 172.21.15.107:0/3432068731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4c806f490 msgr2=0x7fd4c806bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cd3fe640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4c806f490 0x7fd4c806bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cd3fe640 1 -- 172.21.15.107:0/3432068731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c814c470 msgr2=0x7fd4c80ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cd3fe640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c814c470 0x7fd4c80ad410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cd3fe640 1 -- 172.21.15.107:0/3432068731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b80094d0 con 0x7fd4c8071fc0 2023-10-25T00:02:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4ce400640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c814c470 0x7fd4c80ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cd3fe640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c8071fc0 0x7fd4c80aced0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80ae040 tx=0x7fd4b801ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4be7fc640 1 -- 172.21.15.107:0/3432068731 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80b6620 con 0x7fd4c8071fc0 2023-10-25T00:02:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4be7fc640 1 -- 172.21.15.107:0/3432068731 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b80abe90 con 0x7fd4c8071fc0 2023-10-25T00:02:19.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3432068731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c81eccc0 con 0x7fd4c8071fc0 2023-10-25T00:02:19.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4be7fc640 1 -- 172.21.15.107:0/3432068731 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80b69e0 con 0x7fd4c8071fc0 2023-10-25T00:02:19.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3432068731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c81ed280 con 0x7fd4c8071fc0 2023-10-25T00:02:19.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3432068731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c80746e0 con 0x7fd4c8071fc0 2023-10-25T00:02:19.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4be7fc640 1 -- 172.21.15.107:0/3432068731 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd4b80b2040 con 0x7fd4c8071fc0 2023-10-25T00:02:19.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4be7fc640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd4ac065e60 0x7fd4ac068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:19.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4be7fc640 1 -- 172.21.15.107:0/3432068731 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7fd4b811e540 con 0x7fd4c8071fc0 2023-10-25T00:02:19.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cdbff640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd4ac065e60 0x7fd4ac068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:19.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.339+0000 7fd4cdbff640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd4ac065e60 0x7fd4ac068320 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd4c400b7c0 tx=0x7fd4c4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:19.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.347+0000 7fd4be7fc640 1 -- 172.21.15.107:0/3432068731 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd4b80eb350 con 0x7fd4c8071fc0 2023-10-25T00:02:19.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.563+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3432068731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4c8152e30 con 0x7fd4c8071fc0 2023-10-25T00:02:19.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4be7fc640 1 -- 172.21.15.107:0/3432068731 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5771 (secure 0 0 0) 0x7fd4b80ee970 con 0x7fd4c8071fc0 2023-10-25T00:02:19.568 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:19.568 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:19.568 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 80 2023-10-25T00:02:19.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3432068731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd4ac065e60 msgr2=0x7fd4ac068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:19.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd4ac065e60 0x7fd4ac068320 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd4c400b7c0 tx=0x7fd4c4008040 comp rx=0 tx=0).stop 2023-10-25T00:02:19.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3432068731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c8071fc0 msgr2=0x7fd4c80aced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:19.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c8071fc0 0x7fd4c80aced0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80ae040 tx=0x7fd4b801ac20 comp rx=0 tx=0).stop 2023-10-25T00:02:19.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3432068731 shutdown_connections 2023-10-25T00:02:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4c814c470 0x7fd4c80ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4c8071fc0 0x7fd4c80aced0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd4ac065e60 0x7fd4ac068320 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 --2- 172.21.15.107:0/3432068731 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4c806f490 0x7fd4c806bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3432068731 >> 172.21.15.107:0/3432068731 conn(0x7fd4c8070f60 msgr2=0x7fd4c8153bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:19.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3432068731 shutdown_connections 2023-10-25T00:02:19.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:19.567+0000 7fd4cfe63640 1 -- 172.21.15.107:0/3432068731 wait complete. 2023-10-25T00:02:19.593 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:02:21.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:21.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5b54f2640 1 Processor -- start 2023-10-25T00:02:21.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5b54f2640 1 -- start start 2023-10-25T00:02:21.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5b54f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5b00c8100 0x7fe5b00abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:21.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5b54f2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5b006a4b0 0x7fe5b00ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:21.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5b54f2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b0155680 con 0x7fe5b0152040 2023-10-25T00:02:21.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5b54f2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b0154a70 con 0x7fe5b006a4b0 2023-10-25T00:02:21.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5b54f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b0154d50 con 0x7fe5b00c8100 2023-10-25T00:02:21.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5af7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5b006a4b0 0x7fe5b00ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:21.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5af7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5b006a4b0 0x7fe5b00ac240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40164/0 (socket says 172.21.15.107:40164) 2023-10-25T00:02:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5af7fe640 1 -- 172.21.15.107:0/51388261 learned_addr learned my addr 172.21.15.107:0/51388261 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5a67fc640 1 --2- 172.21.15.107:0/51388261 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5b00c8100 0x7fe5b00abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5af7fe640 1 -- 172.21.15.107:0/51388261 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5b00c8100 msgr2=0x7fe5b00abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:21.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5af7fe640 1 --2- 172.21.15.107:0/51388261 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5b00c8100 0x7fe5b00abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:21.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5af7fe640 1 -- 172.21.15.107:0/51388261 >> v1:172.21.15.110:6789/0 conn(0x7fe5b0152040 legacy=0x7fe5b00c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:21.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5af7fe640 1 -- 172.21.15.107:0/51388261 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b0155030 con 0x7fe5b006a4b0 2023-10-25T00:02:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.735+0000 7fe5a67fc640 1 --2- 172.21.15.107:0/51388261 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5b00c8100 0x7fe5b00abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5af7fe640 1 --2- 172.21.15.107:0/51388261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b006a4b0 0x7fe5b00ac240 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8009e50 tx=0x7fe5a801c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd1814453b5cb8a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5ae7fc640 1 -- 172.21.15.107:0/51388261 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a80a5660 con 0x7fe5b006a4b0 2023-10-25T00:02:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5ae7fc640 1 -- 172.21.15.107:0/51388261 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a80a57c0 con 0x7fe5b006a4b0 2023-10-25T00:02:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5ae7fc640 1 -- 172.21.15.107:0/51388261 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a80a5b50 con 0x7fe5b006a4b0 2023-10-25T00:02:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 -- 172.21.15.107:0/51388261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b006a4b0 msgr2=0x7fe5b00ac240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 --2- 172.21.15.107:0/51388261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b006a4b0 0x7fe5b00ac240 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8009e50 tx=0x7fe5a801c980 comp rx=0 tx=0).stop 2023-10-25T00:02:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 -- 172.21.15.107:0/51388261 shutdown_connections 2023-10-25T00:02:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 --2- 172.21.15.107:0/51388261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b006a4b0 0x7fe5b00ac240 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 --2- 172.21.15.107:0/51388261 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5b00c8100 0x7fe5b00abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 -- 172.21.15.107:0/51388261 >> 172.21.15.107:0/51388261 conn(0x7fe5b00ca200 msgr2=0x7fe5b00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 -- 172.21.15.107:0/51388261 shutdown_connections 2023-10-25T00:02:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 -- 172.21.15.107:0/51388261 wait complete. 2023-10-25T00:02:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 Processor -- start 2023-10-25T00:02:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 -- start start 2023-10-25T00:02:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b006a4b0 0x7fe5b00abc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:21.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5b00c8100 0x7fe5b0149830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:21.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5b0152040 0x7fe5b014ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:21.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b00aaf50 con 0x7fe5b006a4b0 2023-10-25T00:02:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b00ab7f0 con 0x7fe5b0152040 2023-10-25T00:02:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5b54f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b00aa740 con 0x7fe5b00c8100 2023-10-25T00:02:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5aeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b006a4b0 0x7fe5b00abc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:21.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5aeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b006a4b0 0x7fe5b00abc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40176/0 (socket says 172.21.15.107:40176) 2023-10-25T00:02:21.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.739+0000 7fe5aeffd640 1 -- 172.21.15.107:0/1475489364 learned_addr learned my addr 172.21.15.107:0/1475489364 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:21.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5a67fc640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5b00c8100 0x7fe5b0149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:21.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5af7fe640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5b0152040 0x7fe5b014ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5aeffd640 1 -- 172.21.15.107:0/1475489364 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5b00c8100 msgr2=0x7fe5b0149830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5aeffd640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5b00c8100 0x7fe5b0149830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5aeffd640 1 -- 172.21.15.107:0/1475489364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5b0152040 msgr2=0x7fe5b014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5aeffd640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5b0152040 0x7fe5b014ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5aeffd640 1 -- 172.21.15.107:0/1475489364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a8009b30 con 0x7fe5b006a4b0 2023-10-25T00:02:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5aeffd640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b006a4b0 0x7fe5b00abc80 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000c500 tx=0x7fe5a000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5a7fff640 1 -- 172.21.15.107:0/1475489364 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0005220 con 0x7fe5b006a4b0 2023-10-25T00:02:21.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5a7fff640 1 -- 172.21.15.107:0/1475489364 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a0005380 con 0x7fe5b006a4b0 2023-10-25T00:02:21.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5b54f2640 1 -- 172.21.15.107:0/1475489364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b00ab1a0 con 0x7fe5b006a4b0 2023-10-25T00:02:21.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5a7fff640 1 -- 172.21.15.107:0/1475489364 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0007270 con 0x7fe5b006a4b0 2023-10-25T00:02:21.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5b54f2640 1 -- 172.21.15.107:0/1475489364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b0149d70 con 0x7fe5b006a4b0 2023-10-25T00:02:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5b54f2640 1 -- 172.21.15.107:0/1475489364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b00746e0 con 0x7fe5b006a4b0 2023-10-25T00:02:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5a7fff640 1 -- 172.21.15.107:0/1475489364 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe5a00073d0 con 0x7fe5b006a4b0 2023-10-25T00:02:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5a7fff640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe590065e60 0x7fe590068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5a7fff640 1 -- 172.21.15.107:0/1475489364 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 8165+0+0 (secure 0 0 0) 0x7fe5a00839e0 con 0x7fe5b006a4b0 2023-10-25T00:02:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5a67fc640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe590065e60 0x7fe590068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.743+0000 7fe5a67fc640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe590065e60 0x7fe590068320 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe5b01edbb0 tx=0x7fe598008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:21.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.747+0000 7fe5a7fff640 1 -- 172.21.15.107:0/1475489364 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe5a0050800 con 0x7fe5b006a4b0 2023-10-25T00:02:21.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.967+0000 7fe5b54f2640 1 -- 172.21.15.107:0/1475489364 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5b00ad2c0 con 0x7fe5b006a4b0 2023-10-25T00:02:21.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.967+0000 7fe5a7fff640 1 -- 172.21.15.107:0/1475489364 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5771 (secure 0 0 0) 0x7fe5a0053e20 con 0x7fe5b006a4b0 2023-10-25T00:02:21.970 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:21.971 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":80}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:00:51.726634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5506":{"gid":5506,"name":"b","rank":0,"incarnation":66,"state":"up:active","state_seq":15,"addr":"172.21.15.110:6834/201284779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":201284779},{"type":"v1","addr":"172.21.15.110:6834","nonce":201284779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:21.971 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 80 2023-10-25T00:02:21.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 -- 172.21.15.107:0/1475489364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe590065e60 msgr2=0x7fe590068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe590065e60 0x7fe590068320 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe5b01edbb0 tx=0x7fe598008040 comp rx=0 tx=0).stop 2023-10-25T00:02:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 -- 172.21.15.107:0/1475489364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b006a4b0 msgr2=0x7fe5b00abc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:21.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b006a4b0 0x7fe5b00abc80 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000c500 tx=0x7fe5a000c9c0 comp rx=0 tx=0).stop 2023-10-25T00:02:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 -- 172.21.15.107:0/1475489364 shutdown_connections 2023-10-25T00:02:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5b0152040 0x7fe5b014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:21.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe590065e60 0x7fe590068320 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5b00c8100 0x7fe5b0149830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 --2- 172.21.15.107:0/1475489364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5b006a4b0 0x7fe5b00abc80 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 -- 172.21.15.107:0/1475489364 >> 172.21.15.107:0/1475489364 conn(0x7fe5b00ca200 msgr2=0x7fe5b0154090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:21.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 -- 172.21.15.107:0/1475489364 shutdown_connections 2023-10-25T00:02:21.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:21.971+0000 7fe5b54f2640 1 -- 172.21.15.107:0/1475489364 wait complete. 2023-10-25T00:02:21.991 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:02:23.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.115+0000 7f7afdea4640 1 Processor -- start 2023-10-25T00:02:24.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.115+0000 7f7afdea4640 1 -- start start 2023-10-25T00:02:24.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7afdea4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7af814c470 0x7f7af814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:24.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7afdea4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af8155cd0 con 0x7f7af8152b30 2023-10-25T00:02:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7afdea4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af80c4380 con 0x7f7af806f490 2023-10-25T00:02:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7afdea4640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af81526e0 con 0x7f7af814c470 2023-10-25T00:02:24.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7af814c470 0x7f7af814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7af8152b30 0x7f7af8152f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41386/0 (socket says 172.21.15.107:41386) 2023-10-25T00:02:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7af814c470 0x7f7af814c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52966/0 (socket says 172.21.15.107:52966) 2023-10-25T00:02:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af7fff640 1 -- 172.21.15.107:0/3540788495 learned_addr learned my addr 172.21.15.107:0/3540788495 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:02:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af77fe640 1 -- 172.21.15.107:0/3540788495 >> v1:172.21.15.110:6789/0 conn(0x7f7af806f490 legacy=0x7f7af8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:24.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af67fc640 1 -- 172.21.15.107:0/3540788495 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149270677 0 0) 0x7f7af8155cd0 con 0x7f7af8152b30 2023-10-25T00:02:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af77fe640 1 -- 172.21.15.107:0/3540788495 >> v1:172.21.15.107:6789/0 conn(0x7f7af8152b30 legacy=0x7f7af8152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af77fe640 1 -- 172.21.15.107:0/3540788495 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af80c48b0 con 0x7f7af814c470 2023-10-25T00:02:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af77fe640 1 --2- 172.21.15.107:0/3540788495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af814c470 0x7f7af814c850 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80097c0 tx=0x7f7ae801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dea9644d0b57158 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af67fc640 1 -- 172.21.15.107:0/3540788495 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80ab600 con 0x7f7af814c470 2023-10-25T00:02:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af67fc640 1 -- 172.21.15.107:0/3540788495 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae80a7050 con 0x7f7af814c470 2023-10-25T00:02:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7af67fc640 1 -- 172.21.15.107:0/3540788495 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80ab980 con 0x7f7af814c470 2023-10-25T00:02:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7afdea4640 1 -- 172.21.15.107:0/3540788495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af814c470 msgr2=0x7f7af814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7afdea4640 1 --2- 172.21.15.107:0/3540788495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af814c470 0x7f7af814c850 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80097c0 tx=0x7f7ae801c360 comp rx=0 tx=0).stop 2023-10-25T00:02:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7afdea4640 1 -- 172.21.15.107:0/3540788495 shutdown_connections 2023-10-25T00:02:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7afdea4640 1 --2- 172.21.15.107:0/3540788495 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af814c470 0x7f7af814c850 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7afdea4640 1 -- 172.21.15.107:0/3540788495 >> 172.21.15.107:0/3540788495 conn(0x7f7af8070f60 msgr2=0x7f7af8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:24.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7afdea4640 1 -- 172.21.15.107:0/3540788495 shutdown_connections 2023-10-25T00:02:24.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.119+0000 7f7afdea4640 1 -- 172.21.15.107:0/3540788495 wait complete. 2023-10-25T00:02:24.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7afdea4640 1 Processor -- start 2023-10-25T00:02:24.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7afdea4640 1 -- start start 2023-10-25T00:02:24.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7afdea4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af806f490 0x7f7af80c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:24.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7afdea4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af814c470 0x7f7af806c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:24.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7afdea4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7af8152b30 0x7f7af81418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:24.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7afdea4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af8154060 con 0x7f7af806f490 2023-10-25T00:02:24.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7afdea4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af81efbd0 con 0x7f7af8152b30 2023-10-25T00:02:24.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7afdea4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af81efeb0 con 0x7f7af814c470 2023-10-25T00:02:24.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7af77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af806f490 0x7f7af80c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:24.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7af77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af806f490 0x7f7af80c1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52972/0 (socket says 172.21.15.107:52972) 2023-10-25T00:02:24.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7af77fe640 1 -- 172.21.15.107:0/1216593534 learned_addr learned my addr 172.21.15.107:0/1216593534 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:24.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7af7fff640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7af8152b30 0x7f7af81418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:24.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7af6ffd640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af814c470 0x7f7af806c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:24.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7af77fe640 1 -- 172.21.15.107:0/1216593534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af814c470 msgr2=0x7f7af806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7af77fe640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af814c470 0x7f7af806c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7af77fe640 1 -- 172.21.15.107:0/1216593534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7af8152b30 msgr2=0x7f7af81418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7af77fe640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7af8152b30 0x7f7af81418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7af77fe640 1 -- 172.21.15.107:0/1216593534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae8009420 con 0x7f7af806f490 2023-10-25T00:02:24.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7af77fe640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af806f490 0x7f7af80c1570 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8019f90 tx=0x7f7ae801aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:24.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7aeffff640 1 -- 172.21.15.107:0/1216593534 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80abde0 con 0x7f7af806f490 2023-10-25T00:02:24.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7aeffff640 1 -- 172.21.15.107:0/1216593534 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae8004030 con 0x7f7af806f490 2023-10-25T00:02:24.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7afdea4640 1 -- 172.21.15.107:0/1216593534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af81f0190 con 0x7f7af806f490 2023-10-25T00:02:24.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7aeffff640 1 -- 172.21.15.107:0/1216593534 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80043a0 con 0x7f7af806f490 2023-10-25T00:02:24.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.123+0000 7f7afdea4640 1 -- 172.21.15.107:0/1216593534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af81f0750 con 0x7f7af806f490 2023-10-25T00:02:24.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.127+0000 7f7afdea4640 1 -- 172.21.15.107:0/1216593534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af80746e0 con 0x7f7af806f490 2023-10-25T00:02:24.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.127+0000 7f7aeffff640 1 -- 172.21.15.107:0/1216593534 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7ae8003620 con 0x7f7af806f490 2023-10-25T00:02:24.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.127+0000 7f7aeffff640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad8065e60 0x7f7ad8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:24.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.127+0000 7f7aeffff640 1 -- 172.21.15.107:0/1216593534 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7f7ae811e820 con 0x7f7af806f490 2023-10-25T00:02:24.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.127+0000 7f7af6ffd640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad8065e60 0x7f7ad8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:24.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.127+0000 7f7af6ffd640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad8065e60 0x7f7ad8068320 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7af814c440 tx=0x7f7ae0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:24.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.131+0000 7f7aeffff640 1 -- 172.21.15.107:0/1216593534 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7ae80eb740 con 0x7f7af806f490 2023-10-25T00:02:24.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.351+0000 7f7afdea4640 1 -- 172.21.15.107:0/1216593534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7af80c2410 con 0x7f7af806f490 2023-10-25T00:02:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.351+0000 7f7aeffff640 1 -- 172.21.15.107:0/1216593534 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f7ae80c4020 con 0x7f7af806f490 2023-10-25T00:02:24.353 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:24.353 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:23.543029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 81 2023-10-25T00:02:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 -- 172.21.15.107:0/1216593534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad8065e60 msgr2=0x7f7ad8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad8065e60 0x7f7ad8068320 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7af814c440 tx=0x7f7ae0009040 comp rx=0 tx=0).stop 2023-10-25T00:02:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 -- 172.21.15.107:0/1216593534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af806f490 msgr2=0x7f7af80c1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af806f490 0x7f7af80c1570 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8019f90 tx=0x7f7ae801aba0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 -- 172.21.15.107:0/1216593534 shutdown_connections 2023-10-25T00:02:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7af8152b30 0x7f7af81418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad8065e60 0x7f7ad8068320 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af814c470 0x7f7af806c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 --2- 172.21.15.107:0/1216593534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af806f490 0x7f7af80c1570 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 -- 172.21.15.107:0/1216593534 >> 172.21.15.107:0/1216593534 conn(0x7f7af8070f60 msgr2=0x7f7af8150ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:24.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 -- 172.21.15.107:0/1216593534 shutdown_connections 2023-10-25T00:02:24.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.355+0000 7f7afdea4640 1 -- 172.21.15.107:0/1216593534 wait complete. 2023-10-25T00:02:24.372 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2023-10-25T00:02:24.372 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:02:24.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:24.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29dfe50640 1 Processor -- start 2023-10-25T00:02:24.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29dfe50640 1 -- start start 2023-10-25T00:02:24.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29dfe50640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f29d8071770 0x7f29d8151480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:24.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29dfe50640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d80c44c0 con 0x7f29d80c2790 2023-10-25T00:02:24.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29dfe50640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d80c38b0 con 0x7f29d806fde0 2023-10-25T00:02:24.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29dfe50640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d8071b50 con 0x7f29d8071770 2023-10-25T00:02:24.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29ddbec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f29d8071770 0x7f29d8151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:24.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29ddbec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f29d8071770 0x7f29d8151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52980/0 (socket says 172.21.15.107:52980) 2023-10-25T00:02:24.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29dd3eb640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f29d80c2790 0x7f29d806cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41388/0 (socket says 172.21.15.107:41388) 2023-10-25T00:02:24.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29ddbec640 1 -- 172.21.15.107:0/2338913353 learned_addr learned my addr 172.21.15.107:0/2338913353 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:24.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29ddbec640 1 -- 172.21.15.107:0/2338913353 >> v1:172.21.15.110:6789/0 conn(0x7f29d806fde0 legacy=0x7f29d8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:24.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29ddbec640 1 -- 172.21.15.107:0/2338913353 >> v1:172.21.15.107:6789/0 conn(0x7f29d80c2790 legacy=0x7f29d806cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29ddbec640 1 -- 172.21.15.107:0/2338913353 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d80c3b90 con 0x7f29d8071770 2023-10-25T00:02:24.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.495+0000 7f29dcbea640 1 -- 172.21.15.107:0/2338913353 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977160254 0 0) 0x7f29d80c44c0 con 0x7f29d80c2790 2023-10-25T00:02:24.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29ddbec640 1 --2- 172.21.15.107:0/2338913353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29d8071770 0x7f29d8151480 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009870 tx=0x7f29d401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f035e38083024d2d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:24.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dcbea640 1 -- 172.21.15.107:0/2338913353 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d40ab600 con 0x7f29d8071770 2023-10-25T00:02:24.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dcbea640 1 -- 172.21.15.107:0/2338913353 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d40a7050 con 0x7f29d8071770 2023-10-25T00:02:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dcbea640 1 -- 172.21.15.107:0/2338913353 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d40ab980 con 0x7f29d8071770 2023-10-25T00:02:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 -- 172.21.15.107:0/2338913353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29d8071770 msgr2=0x7f29d8151480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 --2- 172.21.15.107:0/2338913353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29d8071770 0x7f29d8151480 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009870 tx=0x7f29d401c3e0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 -- 172.21.15.107:0/2338913353 shutdown_connections 2023-10-25T00:02:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 --2- 172.21.15.107:0/2338913353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29d8071770 0x7f29d8151480 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 -- 172.21.15.107:0/2338913353 >> 172.21.15.107:0/2338913353 conn(0x7f29d80c6ed0 msgr2=0x7f29d80c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:24.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 -- 172.21.15.107:0/2338913353 shutdown_connections 2023-10-25T00:02:24.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 -- 172.21.15.107:0/2338913353 wait complete. 2023-10-25T00:02:24.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 Processor -- start 2023-10-25T00:02:24.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 -- start start 2023-10-25T00:02:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29d806fde0 0x7f29d81402f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29d8071770 0x7f29d806a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d80c2790 0x7f29d80c0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d8140830 con 0x7f29d806fde0 2023-10-25T00:02:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d8141f20 con 0x7f29d80c2790 2023-10-25T00:02:24.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29dfe50640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d80aa7a0 con 0x7f29d8071770 2023-10-25T00:02:24.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.499+0000 7f29ddbec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29d806fde0 0x7f29d81402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:24.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ddbec640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29d806fde0 0x7f29d81402f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52986/0 (socket says 172.21.15.107:52986) 2023-10-25T00:02:24.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ddbec640 1 -- 172.21.15.107:0/2461442653 learned_addr learned my addr 172.21.15.107:0/2461442653 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:24.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29de3ed640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d80c2790 0x7f29d80c0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29dd3eb640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29d8071770 0x7f29d806a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ddbec640 1 -- 172.21.15.107:0/2461442653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29d8071770 msgr2=0x7f29d806a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ddbec640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29d8071770 0x7f29d806a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ddbec640 1 -- 172.21.15.107:0/2461442653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d80c2790 msgr2=0x7f29d80c0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ddbec640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d80c2790 0x7f29d80c0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ddbec640 1 -- 172.21.15.107:0/2461442653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d40094d0 con 0x7f29d806fde0 2023-10-25T00:02:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29dd3eb640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29d8071770 0x7f29d806a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ddbec640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29d806fde0 0x7f29d81402f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f29d40ae040 tx=0x7f29d401ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ce7fc640 1 -- 172.21.15.107:0/2461442653 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d40b6620 con 0x7f29d806fde0 2023-10-25T00:02:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ce7fc640 1 -- 172.21.15.107:0/2461442653 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29d40b67a0 con 0x7f29d806fde0 2023-10-25T00:02:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29dfe50640 1 -- 172.21.15.107:0/2461442653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d8069ee0 con 0x7f29d806fde0 2023-10-25T00:02:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ce7fc640 1 -- 172.21.15.107:0/2461442653 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d40b6b50 con 0x7f29d806fde0 2023-10-25T00:02:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29dfe50640 1 -- 172.21.15.107:0/2461442653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d806a4a0 con 0x7f29d806fde0 2023-10-25T00:02:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29dfe50640 1 -- 172.21.15.107:0/2461442653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d8069d40 con 0x7f29d806fde0 2023-10-25T00:02:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ce7fc640 1 -- 172.21.15.107:0/2461442653 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f29d40b2040 con 0x7f29d806fde0 2023-10-25T00:02:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ce7fc640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29bc065e60 0x7f29bc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29ce7fc640 1 -- 172.21.15.107:0/2461442653 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7f29d411e620 con 0x7f29d806fde0 2023-10-25T00:02:24.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29dd3eb640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29bc065e60 0x7f29bc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:24.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.503+0000 7f29dd3eb640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29bc065e60 0x7f29bc068320 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f29c8007fd0 tx=0x7f29c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:24.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.511+0000 7f29ce7fc640 1 -- 172.21.15.107:0/2461442653 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f29d8069d40 con 0x7f29d806fde0 2023-10-25T00:02:24.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.727+0000 7f29dfe50640 1 -- 172.21.15.107:0/2461442653 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29d8069d40 con 0x7f29d806fde0 2023-10-25T00:02:24.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.727+0000 7f29ce7fc640 1 -- 172.21.15.107:0/2461442653 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f29d8069d40 con 0x7f29d806fde0 2023-10-25T00:02:24.729 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:24.729 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:23.543029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:24.729 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 81 2023-10-25T00:02:24.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 -- 172.21.15.107:0/2461442653 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29bc065e60 msgr2=0x7f29bc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29bc065e60 0x7f29bc068320 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f29c8007fd0 tx=0x7f29c8009040 comp rx=0 tx=0).stop 2023-10-25T00:02:24.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 -- 172.21.15.107:0/2461442653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29d806fde0 msgr2=0x7f29d81402f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:24.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29d806fde0 0x7f29d81402f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f29d40ae040 tx=0x7f29d401ac20 comp rx=0 tx=0).stop 2023-10-25T00:02:24.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 -- 172.21.15.107:0/2461442653 shutdown_connections 2023-10-25T00:02:24.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29d80c2790 0x7f29d80c0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29bc065e60 0x7f29bc068320 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29d8071770 0x7f29d806a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 --2- 172.21.15.107:0/2461442653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29d806fde0 0x7f29d81402f0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:24.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 -- 172.21.15.107:0/2461442653 >> 172.21.15.107:0/2461442653 conn(0x7f29d80c6ed0 msgr2=0x7f29d8150f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:24.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 -- 172.21.15.107:0/2461442653 shutdown_connections 2023-10-25T00:02:24.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:24.731+0000 7f29dfe50640 1 -- 172.21.15.107:0/2461442653 wait complete. 2023-10-25T00:02:24.749 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 81, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 79, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6396": { "addr": "172.21.15.107:6835/3232233949", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3232233949, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 3232233949, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6396, "incarnation": 76, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:02:01.619602+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6396 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 81, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6483": { "addr": "172.21.15.107:6837/319918260", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 319918260, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 319918260, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6483, "incarnation": 81, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:02:23.543029+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6483 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 59, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:02:26.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.871+0000 7efd71fb9640 1 Processor -- start 2023-10-25T00:02:26.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.871+0000 7efd71fb9640 1 -- start start 2023-10-25T00:02:26.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd71fb9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efd6c065bc0 0x7efd6c158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:26.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd71fb9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6c06cea0 con 0x7efd6c069d40 2023-10-25T00:02:26.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd71fb9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6c1584e0 con 0x7efd6c06a120 2023-10-25T00:02:26.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd71fb9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd6c0ac650 con 0x7efd6c065bc0 2023-10-25T00:02:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd63fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efd6c065bc0 0x7efd6c158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd6b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7efd6c069d40 0x7efd6c0a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41398/0 (socket says 172.21.15.107:41398) 2023-10-25T00:02:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd63fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efd6c065bc0 0x7efd6c158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53000/0 (socket says 172.21.15.107:53000) 2023-10-25T00:02:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd63fff640 1 -- 172.21.15.107:0/2308249745 learned_addr learned my addr 172.21.15.107:0/2308249745 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:26.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd63fff640 1 -- 172.21.15.107:0/2308249745 >> v1:172.21.15.110:6789/0 conn(0x7efd6c06a120 legacy=0x7efd6c0aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd63fff640 1 -- 172.21.15.107:0/2308249745 >> v1:172.21.15.107:6789/0 conn(0x7efd6c069d40 legacy=0x7efd6c0a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd63fff640 1 -- 172.21.15.107:0/2308249745 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c06d3d0 con 0x7efd6c065bc0 2023-10-25T00:02:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.875+0000 7efd6affd640 1 -- 172.21.15.107:0/2308249745 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886383230 0 0) 0x7efd6c06cea0 con 0x7efd6c069d40 2023-10-25T00:02:26.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd63fff640 1 --2- 172.21.15.107:0/2308249745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd6c065bc0 0x7efd6c158970 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7efd54009870 tx=0x7efd5401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a7d145209b8722 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:26.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd6affd640 1 -- 172.21.15.107:0/2308249745 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd540ad660 con 0x7efd6c065bc0 2023-10-25T00:02:26.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd6affd640 1 -- 172.21.15.107:0/2308249745 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd540ad7c0 con 0x7efd6c065bc0 2023-10-25T00:02:26.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd6affd640 1 -- 172.21.15.107:0/2308249745 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd540adb50 con 0x7efd6c065bc0 2023-10-25T00:02:26.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 -- 172.21.15.107:0/2308249745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd6c065bc0 msgr2=0x7efd6c158970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:26.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 --2- 172.21.15.107:0/2308249745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd6c065bc0 0x7efd6c158970 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7efd54009870 tx=0x7efd5401c470 comp rx=0 tx=0).stop 2023-10-25T00:02:26.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 -- 172.21.15.107:0/2308249745 shutdown_connections 2023-10-25T00:02:26.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 --2- 172.21.15.107:0/2308249745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd6c065bc0 0x7efd6c158970 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:26.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 -- 172.21.15.107:0/2308249745 >> 172.21.15.107:0/2308249745 conn(0x7efd6c0c65c0 msgr2=0x7efd6c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:26.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 -- 172.21.15.107:0/2308249745 shutdown_connections 2023-10-25T00:02:26.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 -- 172.21.15.107:0/2308249745 wait complete. 2023-10-25T00:02:26.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 Processor -- start 2023-10-25T00:02:26.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 -- start start 2023-10-25T00:02:26.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd6c065bc0 0x7efd6c150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:26.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd6c069d40 0x7efd6c070540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:26.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd6c06a120 0x7efd6c070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:26.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c1f04c0 con 0x7efd6c069d40 2023-10-25T00:02:26.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c1ecbe0 con 0x7efd6c06a120 2023-10-25T00:02:26.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c153840 con 0x7efd6c065bc0 2023-10-25T00:02:26.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd63fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd6c069d40 0x7efd6c070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:26.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd63fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd6c069d40 0x7efd6c070540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53014/0 (socket says 172.21.15.107:53014) 2023-10-25T00:02:26.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd63fff640 1 -- 172.21.15.107:0/547408616 learned_addr learned my addr 172.21.15.107:0/547408616 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:26.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd6bfff640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd6c06a120 0x7efd6c070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:26.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd6b7fe640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd6c065bc0 0x7efd6c150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:26.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd63fff640 1 -- 172.21.15.107:0/547408616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd6c065bc0 msgr2=0x7efd6c150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd63fff640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd6c065bc0 0x7efd6c150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd63fff640 1 -- 172.21.15.107:0/547408616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd6c06a120 msgr2=0x7efd6c070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd63fff640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd6c06a120 0x7efd6c070a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd63fff640 1 -- 172.21.15.107:0/547408616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd540094d0 con 0x7efd6c069d40 2023-10-25T00:02:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd6b7fe640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd6c065bc0 0x7efd6c150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:02:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd63fff640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd6c069d40 0x7efd6c070540 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7efd540b0040 tx=0x7efd5401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd68ff9640 1 -- 172.21.15.107:0/547408616 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd540ab870 con 0x7efd6c069d40 2023-10-25T00:02:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd68ff9640 1 -- 172.21.15.107:0/547408616 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd540ab9f0 con 0x7efd6c069d40 2023-10-25T00:02:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 -- 172.21.15.107:0/547408616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6c1edbb0 con 0x7efd6c069d40 2023-10-25T00:02:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd68ff9640 1 -- 172.21.15.107:0/547408616 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd540abda0 con 0x7efd6c069d40 2023-10-25T00:02:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.879+0000 7efd71fb9640 1 -- 172.21.15.107:0/547408616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6c1ed720 con 0x7efd6c069d40 2023-10-25T00:02:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.883+0000 7efd71fb9640 1 -- 172.21.15.107:0/547408616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6c1ed8b0 con 0x7efd6c069d40 2023-10-25T00:02:26.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.883+0000 7efd68ff9640 1 -- 172.21.15.107:0/547408616 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7efd540b4040 con 0x7efd6c069d40 2023-10-25T00:02:26.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.883+0000 7efd68ff9640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd4c065e90 0x7efd4c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:26.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.883+0000 7efd68ff9640 1 -- 172.21.15.107:0/547408616 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7efd5411ef70 con 0x7efd6c069d40 2023-10-25T00:02:26.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.883+0000 7efd6b7fe640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd4c065e90 0x7efd4c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:26.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.883+0000 7efd6b7fe640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd4c065e90 0x7efd4c068350 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7efd5c000f70 tx=0x7efd5c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:26.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:26.891+0000 7efd68ff9640 1 -- 172.21.15.107:0/547408616 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7efd540ebd60 con 0x7efd6c069d40 2023-10-25T00:02:27.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.123+0000 7efd71fb9640 1 -- 172.21.15.107:0/547408616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd6c157f20 con 0x7efd6c069d40 2023-10-25T00:02:27.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.123+0000 7efd68ff9640 1 -- 172.21.15.107:0/547408616 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5013 (secure 0 0 0) 0x7efd540ef380 con 0x7efd6c069d40 2023-10-25T00:02:27.125 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:27.126 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:27.126 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 84 2023-10-25T00:02:27.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 -- 172.21.15.107:0/547408616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd4c065e90 msgr2=0x7efd4c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:27.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd4c065e90 0x7efd4c068350 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7efd5c000f70 tx=0x7efd5c009040 comp rx=0 tx=0).stop 2023-10-25T00:02:27.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 -- 172.21.15.107:0/547408616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd6c069d40 msgr2=0x7efd6c070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd6c069d40 0x7efd6c070540 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7efd540b0040 tx=0x7efd5401acb0 comp rx=0 tx=0).stop 2023-10-25T00:02:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 -- 172.21.15.107:0/547408616 shutdown_connections 2023-10-25T00:02:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd6c06a120 0x7efd6c070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd6c069d40 0x7efd6c070540 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:27.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd4c065e90 0x7efd4c068350 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 --2- 172.21.15.107:0/547408616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd6c065bc0 0x7efd6c150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 -- 172.21.15.107:0/547408616 >> 172.21.15.107:0/547408616 conn(0x7efd6c0c65c0 msgr2=0x7efd6c0a9580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 -- 172.21.15.107:0/547408616 shutdown_connections 2023-10-25T00:02:27.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:27.127+0000 7efd71fb9640 1 -- 172.21.15.107:0/547408616 wait complete. 2023-10-25T00:02:27.147 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5506 2023-10-25T00:02:27.148 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.b 2023-10-25T00:02:29.148 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2023-10-25T00:02:29.149 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-25T00:02:29.150 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-25T00:02:29.153 INFO:tasks.ceph.mds.b:Started 2023-10-25T00:02:29.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:29.237 INFO:tasks.ceph.mds.b.smithi110.stdout:starting mds.b at 2023-10-25T00:02:29.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681eb4a640 1 Processor -- start 2023-10-25T00:02:29.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681eb4a640 1 -- start start 2023-10-25T00:02:29.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681eb4a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f681813dc20 0x7f68180c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:29.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681eb4a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6818157e50 0x7f68181582a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:29.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681eb4a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6818154860 con 0x7f68180720c0 2023-10-25T00:02:29.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681eb4a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68181587e0 con 0x7f681813dc20 2023-10-25T00:02:29.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681eb4a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68180c1f10 con 0x7f6818157e50 2023-10-25T00:02:29.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681c8e6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f681813dc20 0x7f68180c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:29.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f6817fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f68180720c0 0x7f681806df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:42008/0 (socket says 172.21.15.107:42008) 2023-10-25T00:02:29.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f6817fff640 1 -- 172.21.15.107:0/882285300 learned_addr learned my addr 172.21.15.107:0/882285300 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:02:29.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681d0e7640 1 --2- 172.21.15.107:0/882285300 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6818157e50 0x7f68181582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:29.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681c8e6640 1 -- 172.21.15.107:0/882285300 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6818157e50 msgr2=0x7f68181582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:29.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681c8e6640 1 --2- 172.21.15.107:0/882285300 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6818157e50 0x7f68181582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681c8e6640 1 -- 172.21.15.107:0/882285300 >> v1:172.21.15.110:6789/0 conn(0x7f68180720c0 legacy=0x7f681806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681c8e6640 1 -- 172.21.15.107:0/882285300 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6818154b20 con 0x7f681813dc20 2023-10-25T00:02:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.291+0000 7f681d0e7640 1 --2- 172.21.15.107:0/882285300 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6818157e50 0x7f68181582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:02:29.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681c8e6640 1 --2- 172.21.15.107:0/882285300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f681813dc20 0x7f68180c18a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f680c0097c0 tx=0x7f680c01c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dfb262a4f2280d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:29.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f68177fe640 1 -- 172.21.15.107:0/882285300 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c0ab600 con 0x7f681813dc20 2023-10-25T00:02:29.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f68177fe640 1 -- 172.21.15.107:0/882285300 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c0a7050 con 0x7f681813dc20 2023-10-25T00:02:29.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f68177fe640 1 -- 172.21.15.107:0/882285300 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c0ab980 con 0x7f681813dc20 2023-10-25T00:02:29.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 -- 172.21.15.107:0/882285300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f681813dc20 msgr2=0x7f68180c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:29.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 --2- 172.21.15.107:0/882285300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f681813dc20 0x7f68180c18a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f680c0097c0 tx=0x7f680c01c360 comp rx=0 tx=0).stop 2023-10-25T00:02:29.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 -- 172.21.15.107:0/882285300 shutdown_connections 2023-10-25T00:02:29.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 --2- 172.21.15.107:0/882285300 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6818157e50 0x7f68181582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:29.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 --2- 172.21.15.107:0/882285300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f681813dc20 0x7f68180c18a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:29.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 -- 172.21.15.107:0/882285300 >> 172.21.15.107:0/882285300 conn(0x7f68180c39a0 msgr2=0x7f68180c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:29.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 -- 172.21.15.107:0/882285300 shutdown_connections 2023-10-25T00:02:29.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 -- 172.21.15.107:0/882285300 wait complete. 2023-10-25T00:02:29.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 Processor -- start 2023-10-25T00:02:29.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 -- start start 2023-10-25T00:02:29.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68180720c0 0x7f6818138b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:29.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f681813dc20 0x7f68181efb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:29.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6818157e50 0x7f68181f00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:29.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681813a830 con 0x7f6818157e50 2023-10-25T00:02:29.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68180abe40 con 0x7f681813dc20 2023-10-25T00:02:29.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681eb4a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68180ac120 con 0x7f68180720c0 2023-10-25T00:02:29.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681d0e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6818157e50 0x7f68181f00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:29.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681d0e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6818157e50 0x7f68181f00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53032/0 (socket says 172.21.15.107:53032) 2023-10-25T00:02:29.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681d0e7640 1 -- 172.21.15.107:0/3953108556 learned_addr learned my addr 172.21.15.107:0/3953108556 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:29.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f6817fff640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f681813dc20 0x7f68181efb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:29.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681c8e6640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68180720c0 0x7f6818138b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:29.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681d0e7640 1 -- 172.21.15.107:0/3953108556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68180720c0 msgr2=0x7f6818138b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:29.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681d0e7640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68180720c0 0x7f6818138b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:29.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681d0e7640 1 -- 172.21.15.107:0/3953108556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f681813dc20 msgr2=0x7f68181efb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:29.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681d0e7640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f681813dc20 0x7f68181efb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:29.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681d0e7640 1 -- 172.21.15.107:0/3953108556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6810009a70 con 0x7f6818157e50 2023-10-25T00:02:29.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f681c8e6640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68180720c0 0x7f6818138b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:02:29.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.295+0000 7f6817fff640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f681813dc20 0x7f68181efb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:02:29.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.299+0000 7f681d0e7640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6818157e50 0x7f68181f00d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f681000efc0 tx=0x7f6810009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:29.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.299+0000 7f68157fa640 1 -- 172.21.15.107:0/3953108556 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810009d70 con 0x7f6818157e50 2023-10-25T00:02:29.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.299+0000 7f68157fa640 1 -- 172.21.15.107:0/3953108556 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681000f040 con 0x7f6818157e50 2023-10-25T00:02:29.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.299+0000 7f681eb4a640 1 -- 172.21.15.107:0/3953108556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c009420 con 0x7f6818157e50 2023-10-25T00:02:29.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.299+0000 7f68157fa640 1 -- 172.21.15.107:0/3953108556 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68100039a0 con 0x7f6818157e50 2023-10-25T00:02:29.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.299+0000 7f681eb4a640 1 -- 172.21.15.107:0/3953108556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681813ad60 con 0x7f6818157e50 2023-10-25T00:02:29.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.299+0000 7f681eb4a640 1 -- 172.21.15.107:0/3953108556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681813af80 con 0x7f6818157e50 2023-10-25T00:02:29.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.299+0000 7f68157fa640 1 -- 172.21.15.107:0/3953108556 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6810014480 con 0x7f6818157e50 2023-10-25T00:02:29.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.299+0000 7f68157fa640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f67fc065e90 0x7f67fc068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:29.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.299+0000 7f68157fa640 1 -- 172.21.15.107:0/3953108556 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7f6810084e20 con 0x7f6818157e50 2023-10-25T00:02:29.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.299+0000 7f681c8e6640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f67fc065e90 0x7f67fc068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:29.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.303+0000 7f681c8e6640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f67fc065e90 0x7f67fc068350 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f680c0ad000 tx=0x7f680c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:29.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.307+0000 7f68157fa640 1 -- 172.21.15.107:0/3953108556 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6810051b10 con 0x7f6818157e50 2023-10-25T00:02:29.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.507+0000 7f681eb4a640 1 -- 172.21.15.107:0/3953108556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68180746e0 con 0x7f6818157e50 2023-10-25T00:02:29.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f68157fa640 1 -- 172.21.15.107:0/3953108556 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f6810018090 con 0x7f6818157e50 2023-10-25T00:02:29.511 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:29.512 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:29.512 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 84 2023-10-25T00:02:29.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 -- 172.21.15.107:0/3953108556 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f67fc065e90 msgr2=0x7f67fc068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:29.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f67fc065e90 0x7f67fc068350 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f680c0ad000 tx=0x7f680c008040 comp rx=0 tx=0).stop 2023-10-25T00:02:29.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 -- 172.21.15.107:0/3953108556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6818157e50 msgr2=0x7f68181f00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:29.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6818157e50 0x7f68181f00d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f681000efc0 tx=0x7f6810009f90 comp rx=0 tx=0).stop 2023-10-25T00:02:29.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 -- 172.21.15.107:0/3953108556 shutdown_connections 2023-10-25T00:02:29.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6818157e50 0x7f68181f00d0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:29.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f681813dc20 0x7f68181efb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f67fc065e90 0x7f67fc068350 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 --2- 172.21.15.107:0/3953108556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f68180720c0 0x7f6818138b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 -- 172.21.15.107:0/3953108556 >> 172.21.15.107:0/3953108556 conn(0x7f68180c39a0 msgr2=0x7f68180c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 -- 172.21.15.107:0/3953108556 shutdown_connections 2023-10-25T00:02:29.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:29.511+0000 7f681eb4a640 1 -- 172.21.15.107:0/3953108556 wait complete. 2023-10-25T00:02:29.530 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-25T00:02:30.644 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:02:30.643+0000 7fccda747640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:02:31.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:02:31.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb69f10640 1 Processor -- start 2023-10-25T00:02:31.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb69f10640 1 -- start start 2023-10-25T00:02:31.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb69f10640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdb6413dc20 0x7fdb640c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:31.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb69f10640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb640720c0 0x7fdb6406df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:31.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb69f10640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb64154860 con 0x7fdb6406e470 2023-10-25T00:02:31.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb69f10640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb640c1f10 con 0x7fdb6413dc20 2023-10-25T00:02:31.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb69f10640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb64154b20 con 0x7fdb640720c0 2023-10-25T00:02:31.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb637fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdb6413dc20 0x7fdb640c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:31.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb637fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdb6413dc20 0x7fdb640c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53044/0 (socket says 172.21.15.107:53044) 2023-10-25T00:02:31.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb63fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdb6406e470 0x7fdb64157e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41404/0 (socket says 172.21.15.107:41404) 2023-10-25T00:02:31.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb637fe640 1 -- 172.21.15.107:0/1995817086 learned_addr learned my addr 172.21.15.107:0/1995817086 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:31.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb62ffd640 1 --2- 172.21.15.107:0/1995817086 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb640720c0 0x7fdb6406df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:31.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb637fe640 1 -- 172.21.15.107:0/1995817086 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb640720c0 msgr2=0x7fdb6406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:02:31.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb637fe640 1 --2- 172.21.15.107:0/1995817086 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb640720c0 0x7fdb6406df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:31.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb637fe640 1 -- 172.21.15.107:0/1995817086 >> v1:172.21.15.107:6789/0 conn(0x7fdb6406e470 legacy=0x7fdb64157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:31.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.655+0000 7fdb637fe640 1 -- 172.21.15.107:0/1995817086 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb64154e00 con 0x7fdb6413dc20 2023-10-25T00:02:31.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb637fe640 1 --2- 172.21.15.107:0/1995817086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb6413dc20 0x7fdb640c18a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009870 tx=0x7fdb5401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0aa1c07fd89333 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:31.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb627fc640 1 -- 172.21.15.107:0/1995817086 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb540ab600 con 0x7fdb6413dc20 2023-10-25T00:02:31.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb627fc640 1 -- 172.21.15.107:0/1995817086 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb540a7050 con 0x7fdb6413dc20 2023-10-25T00:02:31.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb627fc640 1 -- 172.21.15.107:0/1995817086 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb540ab980 con 0x7fdb6413dc20 2023-10-25T00:02:31.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 -- 172.21.15.107:0/1995817086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb6413dc20 msgr2=0x7fdb640c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:31.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 --2- 172.21.15.107:0/1995817086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb6413dc20 0x7fdb640c18a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009870 tx=0x7fdb5401c420 comp rx=0 tx=0).stop 2023-10-25T00:02:31.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 -- 172.21.15.107:0/1995817086 shutdown_connections 2023-10-25T00:02:31.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 --2- 172.21.15.107:0/1995817086 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb640720c0 0x7fdb6406df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:31.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 --2- 172.21.15.107:0/1995817086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb6413dc20 0x7fdb640c18a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:31.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 -- 172.21.15.107:0/1995817086 >> 172.21.15.107:0/1995817086 conn(0x7fdb640c39a0 msgr2=0x7fdb640c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:31.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 -- 172.21.15.107:0/1995817086 shutdown_connections 2023-10-25T00:02:31.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 -- 172.21.15.107:0/1995817086 wait complete. 2023-10-25T00:02:31.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 Processor -- start 2023-10-25T00:02:31.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 -- start start 2023-10-25T00:02:31.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb6406e470 0x7fdb641f0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:31.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640720c0 0x7fdb6413c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:31.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb6413dc20 0x7fdb64137860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:31.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb640abd30 con 0x7fdb640720c0 2023-10-25T00:02:31.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb641f0f40 con 0x7fdb6406e470 2023-10-25T00:02:31.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb69f10640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb640ac460 con 0x7fdb6413dc20 2023-10-25T00:02:31.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640720c0 0x7fdb6413c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:31.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb62ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640720c0 0x7fdb6413c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53060/0 (socket says 172.21.15.107:53060) 2023-10-25T00:02:31.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb62ffd640 1 -- 172.21.15.107:0/1481600601 learned_addr learned my addr 172.21.15.107:0/1481600601 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:02:31.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb637fe640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb6406e470 0x7fdb641f0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:31.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb63fff640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb6413dc20 0x7fdb64137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:31.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb62ffd640 1 -- 172.21.15.107:0/1481600601 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb6413dc20 msgr2=0x7fdb64137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:31.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb62ffd640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb6413dc20 0x7fdb64137860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:31.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb62ffd640 1 -- 172.21.15.107:0/1481600601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb6406e470 msgr2=0x7fdb641f0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:31.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb62ffd640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb6406e470 0x7fdb641f0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:31.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.659+0000 7fdb62ffd640 1 -- 172.21.15.107:0/1481600601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb540094d0 con 0x7fdb640720c0 2023-10-25T00:02:31.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb62ffd640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640720c0 0x7fdb6413c1e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fdb5800d5f0 tx=0x7fdb5800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:31.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb4bfff640 1 -- 172.21.15.107:0/1481600601 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb580125f0 con 0x7fdb640720c0 2023-10-25T00:02:31.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb4bfff640 1 -- 172.21.15.107:0/1481600601 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb58012750 con 0x7fdb640720c0 2023-10-25T00:02:31.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb69f10640 1 -- 172.21.15.107:0/1481600601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb640c6560 con 0x7fdb640720c0 2023-10-25T00:02:31.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb4bfff640 1 -- 172.21.15.107:0/1481600601 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb58012ae0 con 0x7fdb640720c0 2023-10-25T00:02:31.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb69f10640 1 -- 172.21.15.107:0/1481600601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb640a94f0 con 0x7fdb640720c0 2023-10-25T00:02:31.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb69f10640 1 -- 172.21.15.107:0/1481600601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb640746e0 con 0x7fdb640720c0 2023-10-25T00:02:31.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb4bfff640 1 -- 172.21.15.107:0/1481600601 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdb58012c40 con 0x7fdb640720c0 2023-10-25T00:02:31.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb4bfff640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb50065e60 0x7fdb50068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:02:31.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb4bfff640 1 -- 172.21.15.107:0/1481600601 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7fdb58084590 con 0x7fdb640720c0 2023-10-25T00:02:31.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb637fe640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb50065e60 0x7fdb50068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:02:31.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.663+0000 7fdb637fe640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb50065e60 0x7fdb50068320 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fdb540ad000 tx=0x7fdb54008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:02:31.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.667+0000 7fdb4bfff640 1 -- 172.21.15.107:0/1481600601 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdb58051390 con 0x7fdb640720c0 2023-10-25T00:02:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb69f10640 1 -- 172.21.15.107:0/1481600601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb6406ae50 con 0x7fdb640720c0 2023-10-25T00:02:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb4bfff640 1 -- 172.21.15.107:0/1481600601 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fdb580549b0 con 0x7fdb640720c0 2023-10-25T00:02:31.893 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:02:31.893 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5764,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:02:31.893 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 85 2023-10-25T00:02:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb69f10640 1 -- 172.21.15.107:0/1481600601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb50065e60 msgr2=0x7fdb50068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb69f10640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb50065e60 0x7fdb50068320 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fdb540ad000 tx=0x7fdb54008040 comp rx=0 tx=0).stop 2023-10-25T00:02:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb69f10640 1 -- 172.21.15.107:0/1481600601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640720c0 msgr2=0x7fdb6413c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:02:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb69f10640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640720c0 0x7fdb6413c1e0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fdb5800d5f0 tx=0x7fdb5800dab0 comp rx=0 tx=0).stop 2023-10-25T00:02:31.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb69f10640 1 -- 172.21.15.107:0/1481600601 shutdown_connections 2023-10-25T00:02:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb69f10640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb50065e60 0x7fdb50068320 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb69f10640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb6413dc20 0x7fdb64137860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb69f10640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb640720c0 0x7fdb6413c1e0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb69f10640 1 --2- 172.21.15.107:0/1481600601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb6406e470 0x7fdb641f0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:02:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.891+0000 7fdb69f10640 1 -- 172.21.15.107:0/1481600601 >> 172.21.15.107:0/1481600601 conn(0x7fdb640c39a0 msgr2=0x7fdb6406a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:02:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.895+0000 7fdb69f10640 1 -- 172.21.15.107:0/1481600601 shutdown_connections 2023-10-25T00:02:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:02:31.895+0000 7fdb69f10640 1 -- 172.21.15.107:0/1481600601 wait complete. 2023-10-25T00:02:31.911 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2023-10-25T00:02:31.911 INFO:tasks.mds_thrash.fs.[b]:waiting for 69 secs before thrashing 2023-10-25T00:03:13.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:14.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68f6b9640 1 Processor -- start 2023-10-25T00:03:14.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68f6b9640 1 -- start start 2023-10-25T00:03:14.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68f6b9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb688071770 0x7fb688151480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68f6b9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb68806fde0 0x7fb688070250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68f6b9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6880c44c0 con 0x7fb6880c2790 2023-10-25T00:03:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68f6b9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb688071b50 con 0x7fb68806fde0 2023-10-25T00:03:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68f6b9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6880c38b0 con 0x7fb688071770 2023-10-25T00:03:14.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68dc56640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb68806fde0 0x7fb688070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68dc56640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb68806fde0 0x7fb688070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48292/0 (socket says 172.21.15.107:48292) 2023-10-25T00:03:14.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68dc56640 1 -- 172.21.15.107:0/1083538960 learned_addr learned my addr 172.21.15.107:0/1083538960 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:14.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68d455640 1 --2- 172.21.15.107:0/1083538960 >> v2:172.21.15.110:3300/0 conn(0x7fb688071770 0x7fb688151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68dc56640 1 -- 172.21.15.107:0/1083538960 >> v2:172.21.15.110:3300/0 conn(0x7fb688071770 msgr2=0x7fb688151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:14.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68dc56640 1 --2- 172.21.15.107:0/1083538960 >> v2:172.21.15.110:3300/0 conn(0x7fb688071770 0x7fb688151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68dc56640 1 -- 172.21.15.107:0/1083538960 >> v1:172.21.15.110:6789/0 conn(0x7fb6880c2790 legacy=0x7fb68806cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:14.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.007+0000 7fb68dc56640 1 -- 172.21.15.107:0/1083538960 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6880c3b90 con 0x7fb68806fde0 2023-10-25T00:03:14.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb68dc56640 1 --2- 172.21.15.107:0/1083538960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb68806fde0 0x7fb688070250 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb6880c1e60 tx=0x7fb68001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b186c940c9f53314 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:14.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb67ffff640 1 -- 172.21.15.107:0/1083538960 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6800a5620 con 0x7fb68806fde0 2023-10-25T00:03:14.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb67ffff640 1 -- 172.21.15.107:0/1083538960 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6800a5780 con 0x7fb68806fde0 2023-10-25T00:03:14.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb67ffff640 1 -- 172.21.15.107:0/1083538960 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6800a5b10 con 0x7fb68806fde0 2023-10-25T00:03:14.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb68f6b9640 1 -- 172.21.15.107:0/1083538960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb68806fde0 msgr2=0x7fb688070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:14.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb68f6b9640 1 --2- 172.21.15.107:0/1083538960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb68806fde0 0x7fb688070250 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fb6880c1e60 tx=0x7fb68001c870 comp rx=0 tx=0).stop 2023-10-25T00:03:14.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb68f6b9640 1 -- 172.21.15.107:0/1083538960 shutdown_connections 2023-10-25T00:03:14.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb68f6b9640 1 --2- 172.21.15.107:0/1083538960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb68806fde0 0x7fb688070250 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb68f6b9640 1 --2- 172.21.15.107:0/1083538960 >> v2:172.21.15.110:3300/0 conn(0x7fb688071770 0x7fb688151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb68f6b9640 1 -- 172.21.15.107:0/1083538960 >> 172.21.15.107:0/1083538960 conn(0x7fb6880c6ed0 msgr2=0x7fb6880c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:14.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb68f6b9640 1 -- 172.21.15.107:0/1083538960 shutdown_connections 2023-10-25T00:03:14.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb68f6b9640 1 -- 172.21.15.107:0/1083538960 wait complete. 2023-10-25T00:03:14.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb68f6b9640 1 Processor -- start 2023-10-25T00:03:14.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.011+0000 7fb68f6b9640 1 -- start start 2023-10-25T00:03:14.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68f6b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb68806fde0 0x7fb6881429e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68f6b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb688071770 0x7fb68814c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68f6b9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6880c2790 0x7fb68814d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68f6b9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68813f430 con 0x7fb688071770 2023-10-25T00:03:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68f6b9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6881ee620 con 0x7fb6880c2790 2023-10-25T00:03:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68f6b9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68814bbe0 con 0x7fb68806fde0 2023-10-25T00:03:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68cc54640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb688071770 0x7fb68814c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68cc54640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb688071770 0x7fb68814c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48302/0 (socket says 172.21.15.107:48302) 2023-10-25T00:03:14.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68cc54640 1 -- 172.21.15.107:0/3157790042 learned_addr learned my addr 172.21.15.107:0/3157790042 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:14.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68d455640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb68806fde0 0x7fb6881429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68dc56640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6880c2790 0x7fb68814d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68cc54640 1 -- 172.21.15.107:0/3157790042 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb68806fde0 msgr2=0x7fb6881429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:14.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68cc54640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb68806fde0 0x7fb6881429e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68cc54640 1 -- 172.21.15.107:0/3157790042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6880c2790 msgr2=0x7fb68814d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:14.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68cc54640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6880c2790 0x7fb68814d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68cc54640 1 -- 172.21.15.107:0/3157790042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb680009b30 con 0x7fb688071770 2023-10-25T00:03:14.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68cc54640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb688071770 0x7fb68814c6c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb67800b3e0 tx=0x7fb67800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:14.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb67dffb640 1 -- 172.21.15.107:0/3157790042 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678011830 con 0x7fb688071770 2023-10-25T00:03:14.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb67dffb640 1 -- 172.21.15.107:0/3157790042 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb678011990 con 0x7fb688071770 2023-10-25T00:03:14.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb67dffb640 1 -- 172.21.15.107:0/3157790042 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678011d40 con 0x7fb688071770 2023-10-25T00:03:14.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68f6b9640 1 -- 172.21.15.107:0/3157790042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68814bec0 con 0x7fb688071770 2023-10-25T00:03:14.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68f6b9640 1 -- 172.21.15.107:0/3157790042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb688141050 con 0x7fb688071770 2023-10-25T00:03:14.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb68f6b9640 1 -- 172.21.15.107:0/3157790042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6880746e0 con 0x7fb688071770 2023-10-25T00:03:14.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.015+0000 7fb67dffb640 1 -- 172.21.15.107:0/3157790042 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb67801a400 con 0x7fb688071770 2023-10-25T00:03:14.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.019+0000 7fb67dffb640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb66c065e60 0x7fb66c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.019+0000 7fb67dffb640 1 -- 172.21.15.107:0/3157790042 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7fb678083e50 con 0x7fb688071770 2023-10-25T00:03:14.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.019+0000 7fb68d455640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb66c065e60 0x7fb66c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.019+0000 7fb68d455640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb66c065e60 0x7fb66c068320 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb688152130 tx=0x7fb684008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:14.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.023+0000 7fb67dffb640 1 -- 172.21.15.107:0/3157790042 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb678050c50 con 0x7fb688071770 2023-10-25T00:03:14.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.219+0000 7fb68f6b9640 1 -- 172.21.15.107:0/3157790042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb68814e8d0 con 0x7fb688071770 2023-10-25T00:03:14.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.219+0000 7fb67dffb640 1 -- 172.21.15.107:0/3157790042 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fb678015020 con 0x7fb688071770 2023-10-25T00:03:14.222 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:14.223 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5764,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:14.223 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 85 2023-10-25T00:03:14.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 -- 172.21.15.107:0/3157790042 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb66c065e60 msgr2=0x7fb66c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:14.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb66c065e60 0x7fb66c068320 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb688152130 tx=0x7fb684008040 comp rx=0 tx=0).stop 2023-10-25T00:03:14.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 -- 172.21.15.107:0/3157790042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb688071770 msgr2=0x7fb68814c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:14.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb688071770 0x7fb68814c6c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb67800b3e0 tx=0x7fb67800b8a0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 -- 172.21.15.107:0/3157790042 shutdown_connections 2023-10-25T00:03:14.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6880c2790 0x7fb68814d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb688071770 0x7fb68814c6c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb66c065e60 0x7fb66c068320 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 --2- 172.21.15.107:0/3157790042 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb68806fde0 0x7fb6881429e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 -- 172.21.15.107:0/3157790042 >> 172.21.15.107:0/3157790042 conn(0x7fb6880c6ed0 msgr2=0x7fb68806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:14.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 -- 172.21.15.107:0/3157790042 shutdown_connections 2023-10-25T00:03:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.223+0000 7fb68f6b9640 1 -- 172.21.15.107:0/3157790042 wait complete. 2023-10-25T00:03:14.243 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2023-10-25T00:03:14.243 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:03:14.244 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:03:14.254 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:03:14.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:14.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facc03f1640 1 Processor -- start 2023-10-25T00:03:14.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facc03f1640 1 -- start start 2023-10-25T00:03:14.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facc03f1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7facb8156770 0x7facb80ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facc03f1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7facb80c70e0 0x7facb80bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facc03f1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb80c1ca0 con 0x7facb80c2130 2023-10-25T00:03:14.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facc03f1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb80c27e0 con 0x7facb80c70e0 2023-10-25T00:03:14.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facc03f1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb80c10c0 con 0x7facb8156770 2023-10-25T00:03:14.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facbd98c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7facb80c70e0 0x7facb80bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facbd98c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7facb80c70e0 0x7facb80bdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48318/0 (socket says 172.21.15.107:48318) 2023-10-25T00:03:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facbd98c640 1 -- 172.21.15.107:0/1317249526 learned_addr learned my addr 172.21.15.107:0/1317249526 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facbe18d640 1 --2- 172.21.15.107:0/1317249526 >> v2:172.21.15.110:3300/0 conn(0x7facb8156770 0x7facb80ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facbd98c640 1 -- 172.21.15.107:0/1317249526 >> v2:172.21.15.110:3300/0 conn(0x7facb8156770 msgr2=0x7facb80ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facbd98c640 1 --2- 172.21.15.107:0/1317249526 >> v2:172.21.15.110:3300/0 conn(0x7facb8156770 0x7facb80ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facbd98c640 1 -- 172.21.15.107:0/1317249526 >> v1:172.21.15.110:6789/0 conn(0x7facb80c2130 legacy=0x7facb8069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:14.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facbd98c640 1 -- 172.21.15.107:0/1317249526 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb80c13a0 con 0x7facb80c70e0 2023-10-25T00:03:14.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facbd98c640 1 --2- 172.21.15.107:0/1317249526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facb80c70e0 0x7facb80bdfa0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7facb80ac6f0 tx=0x7faca801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce31acbb85c344b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facbd18b640 1 -- 172.21.15.107:0/1317249526 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca80ab600 con 0x7facb80c70e0 2023-10-25T00:03:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.435+0000 7facbd18b640 1 -- 172.21.15.107:0/1317249526 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca80a7050 con 0x7facb80c70e0 2023-10-25T00:03:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbd18b640 1 -- 172.21.15.107:0/1317249526 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca80ab980 con 0x7facb80c70e0 2023-10-25T00:03:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 -- 172.21.15.107:0/1317249526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facb80c70e0 msgr2=0x7facb80bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 --2- 172.21.15.107:0/1317249526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facb80c70e0 0x7facb80bdfa0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7facb80ac6f0 tx=0x7faca801c3b0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 -- 172.21.15.107:0/1317249526 shutdown_connections 2023-10-25T00:03:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 --2- 172.21.15.107:0/1317249526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facb80c70e0 0x7facb80bdfa0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 --2- 172.21.15.107:0/1317249526 >> v2:172.21.15.110:3300/0 conn(0x7facb8156770 0x7facb80ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 -- 172.21.15.107:0/1317249526 >> 172.21.15.107:0/1317249526 conn(0x7facb80c65c0 msgr2=0x7facb80c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 -- 172.21.15.107:0/1317249526 shutdown_connections 2023-10-25T00:03:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 -- 172.21.15.107:0/1317249526 wait complete. 2023-10-25T00:03:14.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 Processor -- start 2023-10-25T00:03:14.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 -- start start 2023-10-25T00:03:14.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb80c2130 0x7facb814e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb80c70e0 0x7facb80bfb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facb8156770 0x7facb80bbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb8158380 con 0x7facb8156770 2023-10-25T00:03:14.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb8158050 con 0x7facb80c2130 2023-10-25T00:03:14.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facc03f1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb81ef1f0 con 0x7facb80c70e0 2023-10-25T00:03:14.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbe98e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facb8156770 0x7facb80bbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbe98e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facb8156770 0x7facb80bbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48334/0 (socket says 172.21.15.107:48334) 2023-10-25T00:03:14.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbe98e640 1 -- 172.21.15.107:0/4280523652 learned_addr learned my addr 172.21.15.107:0/4280523652 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:14.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbe18d640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb80c2130 0x7facb814e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbd98c640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb80c70e0 0x7facb80bfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbe98e640 1 -- 172.21.15.107:0/4280523652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb80c70e0 msgr2=0x7facb80bfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:14.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbe98e640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb80c70e0 0x7facb80bfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbe98e640 1 -- 172.21.15.107:0/4280523652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb80c2130 msgr2=0x7facb814e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:14.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbe98e640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb80c2130 0x7facb814e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbe98e640 1 -- 172.21.15.107:0/4280523652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca80094d0 con 0x7facb8156770 2023-10-25T00:03:14.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.439+0000 7facbe98e640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facb8156770 0x7facb80bbd30 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7facb000c770 tx=0x7facb000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:14.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.443+0000 7facaeffd640 1 -- 172.21.15.107:0/4280523652 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb0003b80 con 0x7facb8156770 2023-10-25T00:03:14.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.443+0000 7facaeffd640 1 -- 172.21.15.107:0/4280523652 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb0003ce0 con 0x7facb8156770 2023-10-25T00:03:14.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.443+0000 7facc03f1640 1 -- 172.21.15.107:0/4280523652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb8157a90 con 0x7facb8156770 2023-10-25T00:03:14.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.443+0000 7facaeffd640 1 -- 172.21.15.107:0/4280523652 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb000aa90 con 0x7facb8156770 2023-10-25T00:03:14.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.443+0000 7facc03f1640 1 -- 172.21.15.107:0/4280523652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb81ec3f0 con 0x7facb8156770 2023-10-25T00:03:14.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.443+0000 7facc03f1640 1 -- 172.21.15.107:0/4280523652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb81ec250 con 0x7facb8156770 2023-10-25T00:03:14.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.443+0000 7facaeffd640 1 -- 172.21.15.107:0/4280523652 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7facb000abf0 con 0x7facb8156770 2023-10-25T00:03:14.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.443+0000 7facaeffd640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac9c065e60 0x7fac9c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:14.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.443+0000 7facaeffd640 1 -- 172.21.15.107:0/4280523652 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7facb0083d90 con 0x7facb8156770 2023-10-25T00:03:14.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.443+0000 7facbe18d640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac9c065e60 0x7fac9c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:14.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.443+0000 7facbe18d640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac9c065e60 0x7fac9c068320 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7facb40079c0 tx=0x7facb4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:14.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.451+0000 7facaeffd640 1 -- 172.21.15.107:0/4280523652 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7facb81ec250 con 0x7facb8156770 2023-10-25T00:03:14.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.667+0000 7facc03f1640 1 -- 172.21.15.107:0/4280523652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facb81ec250 con 0x7facb8156770 2023-10-25T00:03:14.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facaeffd640 1 -- 172.21.15.107:0/4280523652 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5773 (secure 0 0 0) 0x7facb81ec250 con 0x7facb8156770 2023-10-25T00:03:14.671 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:14.672 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5764,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:14.672 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 85 2023-10-25T00:03:14.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 -- 172.21.15.107:0/4280523652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac9c065e60 msgr2=0x7fac9c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:14.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac9c065e60 0x7fac9c068320 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7facb40079c0 tx=0x7facb4008040 comp rx=0 tx=0).stop 2023-10-25T00:03:14.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 -- 172.21.15.107:0/4280523652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facb8156770 msgr2=0x7facb80bbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:14.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facb8156770 0x7facb80bbd30 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7facb000c770 tx=0x7facb000cc30 comp rx=0 tx=0).stop 2023-10-25T00:03:14.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 -- 172.21.15.107:0/4280523652 shutdown_connections 2023-10-25T00:03:14.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facb8156770 0x7facb80bbd30 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac9c065e60 0x7fac9c068320 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facb80c70e0 0x7facb80bfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 --2- 172.21.15.107:0/4280523652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facb80c2130 0x7facb814e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:14.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 -- 172.21.15.107:0/4280523652 >> 172.21.15.107:0/4280523652 conn(0x7facb80c65c0 msgr2=0x7facb8069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:14.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 -- 172.21.15.107:0/4280523652 shutdown_connections 2023-10-25T00:03:14.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:14.671+0000 7facc03f1640 1 -- 172.21.15.107:0/4280523652 wait complete. 2023-10-25T00:03:14.692 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:03:16.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:16.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.839+0000 7fd9f19e9640 1 Processor -- start 2023-10-25T00:03:16.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9f19e9640 1 -- start start 2023-10-25T00:03:16.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9f19e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9ec06a4b0 0x7fd9ec0ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:16.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9f19e9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ec155680 con 0x7fd9ec0c8100 2023-10-25T00:03:16.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9f19e9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ec154a70 con 0x7fd9ec152040 2023-10-25T00:03:16.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9f19e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec154d50 con 0x7fd9ec06a4b0 2023-10-25T00:03:16.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9e3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd9ec0c8100 0x7fd9ec0abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60022/0 (socket says 172.21.15.107:60022) 2023-10-25T00:03:16.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9e3fff640 1 -- 172.21.15.107:0/35646818 learned_addr learned my addr 172.21.15.107:0/35646818 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:16.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9ec06a4b0 0x7fd9ec0ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:16.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720366631 0 0) 0x7fd9ec155680 con 0x7fd9ec0c8100 2023-10-25T00:03:16.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d8003530 con 0x7fd9ec0c8100 2023-10-25T00:03:16.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2809131872 0 0) 0x7fd9d8003530 con 0x7fd9ec0c8100 2023-10-25T00:03:16.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9ec06a4b0 msgr2=0x7fd9ec0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:16.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9ec06a4b0 0x7fd9ec0ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:16.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 >> v1:172.21.15.110:6789/0 conn(0x7fd9ec152040 legacy=0x7fd9ec0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:16.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec155030 con 0x7fd9ec0c8100 2023-10-25T00:03:16.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fd9d4001680 con 0x7fd9ec0c8100 2023-10-25T00:03:16.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 >> v1:172.21.15.107:6789/0 conn(0x7fd9ec0c8100 legacy=0x7fd9ec0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:16.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d80039b0 0x7fd9d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:16.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9ec06a4b0 0x7fd9d80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:16.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9d8004d10 0x7fd9d8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:16.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec154d50 con 0x7fd9d80039b0 2023-10-25T00:03:16.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d800baf0 con 0x7fd9d8004d10 2023-10-25T00:03:16.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d800bcb0 con 0x7fd9ec06a4b0 2023-10-25T00:03:16.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d80039b0 0x7fd9d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:16.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9d8004d10 0x7fd9d8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:16.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9e3fff640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9ec06a4b0 0x7fd9d80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:16.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9eaffd640 1 -- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9ec06a4b0 msgr2=0x7fd9d80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:16.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9ec06a4b0 0x7fd9d80047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:16.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9eaffd640 1 -- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9d8004d10 msgr2=0x7fd9d8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:16.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9d8004d10 0x7fd9d8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:16.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9eaffd640 1 -- 172.21.15.107:0/35646818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d8003530 con 0x7fd9d80039b0 2023-10-25T00:03:16.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.843+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9d8004d10 0x7fd9d8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:16.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d80039b0 0x7fd9d8003dd0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc001780 tx=0x7fd9dc01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d7945eb73f929f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:16.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc0052d0 con 0x7fd9d80039b0 2023-10-25T00:03:16.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc0a7050 con 0x7fd9d80039b0 2023-10-25T00:03:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9ea7fc640 1 -- 172.21.15.107:0/35646818 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc003600 con 0x7fd9d80039b0 2023-10-25T00:03:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9f19e9640 1 -- 172.21.15.107:0/35646818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d80039b0 msgr2=0x7fd9d8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9f19e9640 1 --2- 172.21.15.107:0/35646818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9d80039b0 0x7fd9d8003dd0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc001780 tx=0x7fd9dc01c340 comp rx=0 tx=0).stop 2023-10-25T00:03:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/35646818 reap_dead start 2023-10-25T00:03:16.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9f19e9640 1 -- 172.21.15.107:0/35646818 shutdown_connections 2023-10-25T00:03:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9f19e9640 1 -- 172.21.15.107:0/35646818 >> 172.21.15.107:0/35646818 conn(0x7fd9ec0ca200 msgr2=0x7fd9ec0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9f19e9640 1 -- 172.21.15.107:0/35646818 shutdown_connections 2023-10-25T00:03:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9f19e9640 1 -- 172.21.15.107:0/35646818 wait complete. 2023-10-25T00:03:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9f19e9640 1 Processor -- start 2023-10-25T00:03:16.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.847+0000 7fd9f19e9640 1 -- start start 2023-10-25T00:03:16.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9f19e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9d80039b0 0x7fd9ec153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:16.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9f19e9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9ec153db0 0x7fd9ec1f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:16.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9f19e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1f0810 0x7fd9ec1ecb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:16.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9f19e9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec149420 con 0x7fd9ec1f0810 2023-10-25T00:03:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9f19e9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec151880 con 0x7fd9ec153db0 2023-10-25T00:03:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9f19e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec151b60 con 0x7fd9d80039b0 2023-10-25T00:03:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9eb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1f0810 0x7fd9ec1ecb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9eb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1f0810 0x7fd9ec1ecb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48362/0 (socket says 172.21.15.107:48362) 2023-10-25T00:03:16.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/1384079815 learned_addr learned my addr 172.21.15.107:0/1384079815 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9d80039b0 0x7fd9ec153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9e3fff640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9ec153db0 0x7fd9ec1f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/1384079815 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9d80039b0 msgr2=0x7fd9ec153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9d80039b0 0x7fd9ec153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:16.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/1384079815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9ec153db0 msgr2=0x7fd9ec1f02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:16.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9ec153db0 0x7fd9ec1f02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:16.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9eb7fe640 1 -- 172.21.15.107:0/1384079815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec1434d0 con 0x7fd9ec1f0810 2023-10-25T00:03:16.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9eb7fe640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1f0810 0x7fd9ec1ecb00 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec08c090 tx=0x7fd9e400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:16.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9e8ff9640 1 -- 172.21.15.107:0/1384079815 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4017400 con 0x7fd9ec1f0810 2023-10-25T00:03:16.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9e8ff9640 1 -- 172.21.15.107:0/1384079815 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e4011040 con 0x7fd9ec1f0810 2023-10-25T00:03:16.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9f19e9640 1 -- 172.21.15.107:0/1384079815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ec1437b0 con 0x7fd9ec1f0810 2023-10-25T00:03:16.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9e8ff9640 1 -- 172.21.15.107:0/1384079815 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e40177c0 con 0x7fd9ec1f0810 2023-10-25T00:03:16.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9f19e9640 1 -- 172.21.15.107:0/1384079815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ec154230 con 0x7fd9ec1f0810 2023-10-25T00:03:16.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9f19e9640 1 -- 172.21.15.107:0/1384079815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ec0746e0 con 0x7fd9ec1f0810 2023-10-25T00:03:16.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9e8ff9640 1 -- 172.21.15.107:0/1384079815 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd9e40203b0 con 0x7fd9ec1f0810 2023-10-25T00:03:16.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9e8ff9640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd9cc065e60 0x7fd9cc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:16.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9e8ff9640 1 -- 172.21.15.107:0/1384079815 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7fd9e4084320 con 0x7fd9ec1f0810 2023-10-25T00:03:16.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.851+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd9cc065e60 0x7fd9cc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:16.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.855+0000 7fd9eaffd640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd9cc065e60 0x7fd9cc068320 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc001750 tx=0x7fd9dc0011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:16.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:16.859+0000 7fd9e8ff9640 1 -- 172.21.15.107:0/1384079815 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd9e4051120 con 0x7fd9ec1f0810 2023-10-25T00:03:17.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.051+0000 7fd9f19e9640 1 -- 172.21.15.107:0/1384079815 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9ec14c160 con 0x7fd9ec1f0810 2023-10-25T00:03:17.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.051+0000 7fd9e8ff9640 1 -- 172.21.15.107:0/1384079815 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fd9e4054740 con 0x7fd9ec1f0810 2023-10-25T00:03:17.053 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:17.053 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5764,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:17.054 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 85 2023-10-25T00:03:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 -- 172.21.15.107:0/1384079815 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd9cc065e60 msgr2=0x7fd9cc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd9cc065e60 0x7fd9cc068320 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc001750 tx=0x7fd9dc0011a0 comp rx=0 tx=0).stop 2023-10-25T00:03:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 -- 172.21.15.107:0/1384079815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1f0810 msgr2=0x7fd9ec1ecb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1f0810 0x7fd9ec1ecb00 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec08c090 tx=0x7fd9e400f4a0 comp rx=0 tx=0).stop 2023-10-25T00:03:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 -- 172.21.15.107:0/1384079815 shutdown_connections 2023-10-25T00:03:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1f0810 0x7fd9ec1ecb00 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd9ec153db0 0x7fd9ec1f02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd9cc065e60 0x7fd9cc068320 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:17.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 --2- 172.21.15.107:0/1384079815 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd9d80039b0 0x7fd9ec153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:17.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 -- 172.21.15.107:0/1384079815 >> 172.21.15.107:0/1384079815 conn(0x7fd9ec0ca200 msgr2=0x7fd9ec14b970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:17.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 -- 172.21.15.107:0/1384079815 shutdown_connections 2023-10-25T00:03:17.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:17.055+0000 7fd9f19e9640 1 -- 172.21.15.107:0/1384079815 wait complete. 2023-10-25T00:03:17.072 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:03:19.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:19.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45cbf05640 1 Processor -- start 2023-10-25T00:03:19.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45cbf05640 1 -- start start 2023-10-25T00:03:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45cbf05640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4152040 0x7f45c40c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45cbf05640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f45c406a4b0 0x7f45c40ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:19.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45cbf05640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c4155680 con 0x7f45c40c8100 2023-10-25T00:03:19.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45cbf05640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45c4154a70 con 0x7f45c406a4b0 2023-10-25T00:03:19.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45cbf05640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c4154d50 con 0x7f45c4152040 2023-10-25T00:03:19.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c94a0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f45c40c8100 0x7f45c40abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60032/0 (socket says 172.21.15.107:60032) 2023-10-25T00:03:19.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c94a0640 1 -- 172.21.15.107:0/2942792641 learned_addr learned my addr 172.21.15.107:0/2942792641 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:19.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c9ca1640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4152040 0x7f45c40c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:19.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45ca4a2640 1 --2- 172.21.15.107:0/2942792641 >> v2:172.21.15.110:3300/0 conn(0x7f45c406a4b0 0x7f45c40ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:19.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776015241 0 0) 0x7f45c4155680 con 0x7f45c40c8100 2023-10-25T00:03:19.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b0003530 con 0x7f45c40c8100 2023-10-25T00:03:19.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2207113226 0 0) 0x7f45b0003530 con 0x7f45c40c8100 2023-10-25T00:03:19.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4152040 msgr2=0x7f45c40c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:19.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4152040 0x7f45c40c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:19.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 >> v2:172.21.15.110:3300/0 conn(0x7f45c406a4b0 msgr2=0x7f45c40ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:19.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 --2- 172.21.15.107:0/2942792641 >> v2:172.21.15.110:3300/0 conn(0x7f45c406a4b0 0x7f45c40ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:19.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c4155030 con 0x7f45c40c8100 2023-10-25T00:03:19.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f45b4001680 con 0x7f45c40c8100 2023-10-25T00:03:19.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 >> v1:172.21.15.107:6789/0 conn(0x7f45c40c8100 legacy=0x7f45c40abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:19.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45b0003980 0x7f45b0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:19.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4152040 0x7f45b00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:19.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 0x7f45b0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:19.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c4154a70 con 0x7f45b0004d10 2023-10-25T00:03:19.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c4154d50 con 0x7f45b0003980 2023-10-25T00:03:19.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b000b990 con 0x7f45c4152040 2023-10-25T00:03:19.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45ca4a2640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 0x7f45b0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:19.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c94a0640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4152040 0x7f45b00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:19.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c9ca1640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45b0003980 0x7f45b0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:19.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45ca4a2640 1 -- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4152040 msgr2=0x7f45b00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:19.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45ca4a2640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4152040 0x7f45b00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:19.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45ca4a2640 1 -- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45b0003980 msgr2=0x7f45b0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:19.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45ca4a2640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45b0003980 0x7f45b0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:19.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45ca4a2640 1 -- 172.21.15.107:0/2942792641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b0003530 con 0x7f45b0004d10 2023-10-25T00:03:19.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c94a0640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4152040 0x7f45b00047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:03:19.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.207+0000 7f45c9ca1640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45b0003980 0x7f45b0003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:03:19.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45ca4a2640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 0x7f45b0005190 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f45bc009fc0 tx=0x7f45bc01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c65589771dcf0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:19.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0a5cb0 con 0x7f45b0004d10 2023-10-25T00:03:19.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45bc00f040 con 0x7f45b0004d10 2023-10-25T00:03:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45c8c9f640 1 -- 172.21.15.107:0/2942792641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0ad680 con 0x7f45b0004d10 2023-10-25T00:03:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45cbf05640 1 -- 172.21.15.107:0/2942792641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 msgr2=0x7f45b0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45cbf05640 1 --2- 172.21.15.107:0/2942792641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 0x7f45b0005190 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f45bc009fc0 tx=0x7f45bc01cca0 comp rx=0 tx=0).stop 2023-10-25T00:03:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45ca4a2640 1 -- 172.21.15.107:0/2942792641 reap_dead start 2023-10-25T00:03:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45cbf05640 1 -- 172.21.15.107:0/2942792641 shutdown_connections 2023-10-25T00:03:19.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45cbf05640 1 -- 172.21.15.107:0/2942792641 >> 172.21.15.107:0/2942792641 conn(0x7f45c40ca200 msgr2=0x7f45c40ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45cbf05640 1 -- 172.21.15.107:0/2942792641 shutdown_connections 2023-10-25T00:03:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45cbf05640 1 -- 172.21.15.107:0/2942792641 wait complete. 2023-10-25T00:03:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45cbf05640 1 Processor -- start 2023-10-25T00:03:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.211+0000 7f45cbf05640 1 -- start start 2023-10-25T00:03:19.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45cbf05640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 0x7f45c40b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45cbf05640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45c40b15d0 0x7f45c41e9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45cbf05640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4150fa0 0x7f45c41502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45cbf05640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c40bf820 con 0x7f45b0004d10 2023-10-25T00:03:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45cbf05640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c41e9500 con 0x7f45c40b15d0 2023-10-25T00:03:19.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45cbf05640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c41e97a0 con 0x7f45c4150fa0 2023-10-25T00:03:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c9ca1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 0x7f45c40b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c9ca1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 0x7f45c40b0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48388/0 (socket says 172.21.15.107:48388) 2023-10-25T00:03:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c9ca1640 1 -- 172.21.15.107:0/3411737091 learned_addr learned my addr 172.21.15.107:0/3411737091 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45ca4a2640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4150fa0 0x7f45c41502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c94a0640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45c40b15d0 0x7f45c41e9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c9ca1640 1 -- 172.21.15.107:0/3411737091 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4150fa0 msgr2=0x7f45c41502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c9ca1640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4150fa0 0x7f45c41502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c9ca1640 1 -- 172.21.15.107:0/3411737091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45c40b15d0 msgr2=0x7f45c41e9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c9ca1640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45c40b15d0 0x7f45c41e9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:19.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c9ca1640 1 -- 172.21.15.107:0/3411737091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c40bd5e0 con 0x7f45b0004d10 2023-10-25T00:03:19.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c9ca1640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 0x7f45c40b0090 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009160 tx=0x7f45c0011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:19.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45ba7fc640 1 -- 172.21.15.107:0/3411737091 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0004220 con 0x7f45b0004d10 2023-10-25T00:03:19.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45ba7fc640 1 -- 172.21.15.107:0/3411737091 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c0004380 con 0x7f45b0004d10 2023-10-25T00:03:19.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45cbf05640 1 -- 172.21.15.107:0/3411737091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c40bf2a0 con 0x7f45b0004d10 2023-10-25T00:03:19.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45ba7fc640 1 -- 172.21.15.107:0/3411737091 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0015890 con 0x7f45b0004d10 2023-10-25T00:03:19.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45cbf05640 1 -- 172.21.15.107:0/3411737091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c40b05d0 con 0x7f45b0004d10 2023-10-25T00:03:19.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45cbf05640 1 -- 172.21.15.107:0/3411737091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c40746e0 con 0x7f45b0004d10 2023-10-25T00:03:19.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45ba7fc640 1 -- 172.21.15.107:0/3411737091 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f45c00159f0 con 0x7f45b0004d10 2023-10-25T00:03:19.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45ba7fc640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f45a8065e60 0x7f45a8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45ba7fc640 1 -- 172.21.15.107:0/3411737091 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7f45c0084520 con 0x7f45b0004d10 2023-10-25T00:03:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c94a0640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f45a8065e60 0x7f45a8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.215+0000 7f45c94a0640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f45a8065e60 0x7f45a8068320 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f45c40adc60 tx=0x7f45b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.219+0000 7f45ba7fc640 1 -- 172.21.15.107:0/3411737091 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f45c00513d0 con 0x7f45b0004d10 2023-10-25T00:03:19.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.411+0000 7f45cbf05640 1 -- 172.21.15.107:0/3411737091 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45c40c99a0 con 0x7f45b0004d10 2023-10-25T00:03:19.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.411+0000 7f45ba7fc640 1 -- 172.21.15.107:0/3411737091 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5773 (secure 0 0 0) 0x7f45c0022090 con 0x7f45b0004d10 2023-10-25T00:03:19.413 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:19.413 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5764,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 85 2023-10-25T00:03:19.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 -- 172.21.15.107:0/3411737091 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f45a8065e60 msgr2=0x7f45a8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f45a8065e60 0x7f45a8068320 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f45c40adc60 tx=0x7f45b4009040 comp rx=0 tx=0).stop 2023-10-25T00:03:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 -- 172.21.15.107:0/3411737091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 msgr2=0x7f45c40b0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 0x7f45c40b0090 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009160 tx=0x7f45c0011a60 comp rx=0 tx=0).stop 2023-10-25T00:03:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 -- 172.21.15.107:0/3411737091 shutdown_connections 2023-10-25T00:03:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f45a8065e60 0x7f45a8068320 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f45c4150fa0 0x7f45c41502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45c40b15d0 0x7f45c41e9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 --2- 172.21.15.107:0/3411737091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45b0004d10 0x7f45c40b0090 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 -- 172.21.15.107:0/3411737091 >> 172.21.15.107:0/3411737091 conn(0x7f45c40ca200 msgr2=0x7f45c40c4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 -- 172.21.15.107:0/3411737091 shutdown_connections 2023-10-25T00:03:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:19.415+0000 7f45cbf05640 1 -- 172.21.15.107:0/3411737091 wait complete. 2023-10-25T00:03:19.431 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:03:21.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:21.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.563+0000 7ff428f86640 1 Processor -- start 2023-10-25T00:03:21.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.563+0000 7ff428f86640 1 -- start start 2023-10-25T00:03:21.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff428f86640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff42406ee40 0x7ff4240ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:21.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff428f86640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42406f730 con 0x7ff4240a9820 2023-10-25T00:03:21.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff428f86640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff424156a40 con 0x7ff42414fd40 2023-10-25T00:03:21.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff428f86640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff424156710 con 0x7ff42406ee40 2023-10-25T00:03:21.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff421d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff4240a9820 0x7ff42415af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60046/0 (socket says 172.21.15.107:60046) 2023-10-25T00:03:21.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff421d9b640 1 -- 172.21.15.107:0/1953754598 learned_addr learned my addr 172.21.15.107:0/1953754598 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:21.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff422d9d640 1 --2- 172.21.15.107:0/1953754598 >> v2:172.21.15.107:3300/0 conn(0x7ff42406ee40 0x7ff4240ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:21.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff422d9d640 1 -- 172.21.15.107:0/1953754598 >> v1:172.21.15.110:6789/0 conn(0x7ff42414fd40 legacy=0x7ff4240c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:21.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff422d9d640 1 -- 172.21.15.107:0/1953754598 >> v1:172.21.15.107:6789/0 conn(0x7ff4240a9820 legacy=0x7ff42415af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:21.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff42159a640 1 -- 172.21.15.107:0/1953754598 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216218351 0 0) 0x7ff42406f730 con 0x7ff4240a9820 2023-10-25T00:03:21.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff422d9d640 1 -- 172.21.15.107:0/1953754598 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42406f980 con 0x7ff42406ee40 2023-10-25T00:03:21.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff422d9d640 1 --2- 172.21.15.107:0/1953754598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff42406ee40 0x7ff4240ab090 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009d90 tx=0x7ff41c01c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76bf15671b31a6d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:21.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff42159a640 1 -- 172.21.15.107:0/1953754598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0a57d0 con 0x7ff42406ee40 2023-10-25T00:03:21.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff42159a640 1 -- 172.21.15.107:0/1953754598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41c0a5930 con 0x7ff42406ee40 2023-10-25T00:03:21.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.567+0000 7ff42159a640 1 -- 172.21.15.107:0/1953754598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0a5cc0 con 0x7ff42406ee40 2023-10-25T00:03:21.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- 172.21.15.107:0/1953754598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff42406ee40 msgr2=0x7ff4240ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:21.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 --2- 172.21.15.107:0/1953754598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff42406ee40 0x7ff4240ab090 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009d90 tx=0x7ff41c01c7f0 comp rx=0 tx=0).stop 2023-10-25T00:03:21.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- 172.21.15.107:0/1953754598 shutdown_connections 2023-10-25T00:03:21.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 --2- 172.21.15.107:0/1953754598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff42406ee40 0x7ff4240ab090 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:21.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- 172.21.15.107:0/1953754598 >> 172.21.15.107:0/1953754598 conn(0x7ff4240c2950 msgr2=0x7ff4240c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:21.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- 172.21.15.107:0/1953754598 shutdown_connections 2023-10-25T00:03:21.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- 172.21.15.107:0/1953754598 wait complete. 2023-10-25T00:03:21.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 Processor -- start 2023-10-25T00:03:21.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- start start 2023-10-25T00:03:21.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff42406ee40 0x7ff42414b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:21.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff42414fd40 0x7ff4240bb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:21.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4240c58d0 0x7ff4240bbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:21.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff424151700 con 0x7ff4240c58d0 2023-10-25T00:03:21.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240bae70 con 0x7ff42414fd40 2023-10-25T00:03:21.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42414bc60 con 0x7ff42406ee40 2023-10-25T00:03:21.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff422d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4240c58d0 0x7ff4240bbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:21.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff422d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4240c58d0 0x7ff4240bbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48410/0 (socket says 172.21.15.107:48410) 2023-10-25T00:03:21.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff422d9d640 1 -- 172.21.15.107:0/1645562347 learned_addr learned my addr 172.21.15.107:0/1645562347 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:21.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff421d9b640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff42414fd40 0x7ff4240bb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:21.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff42259c640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff42406ee40 0x7ff42414b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:21.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff422d9d640 1 -- 172.21.15.107:0/1645562347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff42406ee40 msgr2=0x7ff42414b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:21.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff422d9d640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff42406ee40 0x7ff42414b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:21.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff422d9d640 1 -- 172.21.15.107:0/1645562347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff42414fd40 msgr2=0x7ff4240bb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:21.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff422d9d640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff42414fd40 0x7ff4240bb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:21.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff422d9d640 1 -- 172.21.15.107:0/1645562347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c009a70 con 0x7ff4240c58d0 2023-10-25T00:03:21.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff422d9d640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4240c58d0 0x7ff4240bbb70 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff41c0b0040 tx=0x7ff41c019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:21.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff412ffd640 1 -- 172.21.15.107:0/1645562347 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0ad850 con 0x7ff4240c58d0 2023-10-25T00:03:21.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff412ffd640 1 -- 172.21.15.107:0/1645562347 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41c0ad9b0 con 0x7ff4240c58d0 2023-10-25T00:03:21.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- 172.21.15.107:0/1645562347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42414bf40 con 0x7ff4240c58d0 2023-10-25T00:03:21.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff412ffd640 1 -- 172.21.15.107:0/1645562347 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c0adda0 con 0x7ff4240c58d0 2023-10-25T00:03:21.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- 172.21.15.107:0/1645562347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4240bb110 con 0x7ff4240c58d0 2023-10-25T00:03:21.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.571+0000 7ff428f86640 1 -- 172.21.15.107:0/1645562347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4240c24a0 con 0x7ff4240c58d0 2023-10-25T00:03:21.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.575+0000 7ff412ffd640 1 -- 172.21.15.107:0/1645562347 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff41c0b4050 con 0x7ff4240c58d0 2023-10-25T00:03:21.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.575+0000 7ff412ffd640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff404065e60 0x7ff404068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:21.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.575+0000 7ff412ffd640 1 -- 172.21.15.107:0/1645562347 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7ff41c11eff0 con 0x7ff4240c58d0 2023-10-25T00:03:21.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.575+0000 7ff42259c640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff404065e60 0x7ff404068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:21.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.575+0000 7ff42259c640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff404065e60 0x7ff404068320 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff41400b810 tx=0x7ff414008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:21.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.579+0000 7ff412ffd640 1 -- 172.21.15.107:0/1645562347 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff41c0ebe90 con 0x7ff4240c58d0 2023-10-25T00:03:21.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.767+0000 7ff428f86640 1 -- 172.21.15.107:0/1645562347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4240bb300 con 0x7ff4240c58d0 2023-10-25T00:03:21.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff412ffd640 1 -- 172.21.15.107:0/1645562347 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5773 (secure 0 0 0) 0x7ff41c0c7090 con 0x7ff4240c58d0 2023-10-25T00:03:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:21.771 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5764,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:21.772 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 85 2023-10-25T00:03:21.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 -- 172.21.15.107:0/1645562347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff404065e60 msgr2=0x7ff404068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:21.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff404065e60 0x7ff404068320 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff41400b810 tx=0x7ff414008040 comp rx=0 tx=0).stop 2023-10-25T00:03:21.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 -- 172.21.15.107:0/1645562347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4240c58d0 msgr2=0x7ff4240bbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:21.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4240c58d0 0x7ff4240bbb70 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7ff41c0b0040 tx=0x7ff41c019590 comp rx=0 tx=0).stop 2023-10-25T00:03:21.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 -- 172.21.15.107:0/1645562347 shutdown_connections 2023-10-25T00:03:21.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4240c58d0 0x7ff4240bbb70 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:21.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff42414fd40 0x7ff4240bb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:21.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff404065e60 0x7ff404068320 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:21.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 --2- 172.21.15.107:0/1645562347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff42406ee40 0x7ff42414b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:21.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 -- 172.21.15.107:0/1645562347 >> 172.21.15.107:0/1645562347 conn(0x7ff4240c2950 msgr2=0x7ff424150350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:21.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 -- 172.21.15.107:0/1645562347 shutdown_connections 2023-10-25T00:03:21.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:21.771+0000 7ff428f86640 1 -- 172.21.15.107:0/1645562347 wait complete. 2023-10-25T00:03:21.789 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:03:23.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:23.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bfbee640 1 Processor -- start 2023-10-25T00:03:23.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bfbee640 1 -- start start 2023-10-25T00:03:23.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bfbee640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb8b8152040 0x7fb8b80c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:23.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bfbee640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b8155680 con 0x7fb8b806a4b0 2023-10-25T00:03:23.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bfbee640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b8154a70 con 0x7fb8b80c8100 2023-10-25T00:03:23.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bfbee640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b8154d50 con 0x7fb8b8152040 2023-10-25T00:03:23.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8be18b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb8b806a4b0 0x7fb8b80ac410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49898/0 (socket says 172.21.15.107:49898) 2023-10-25T00:03:23.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8be18b640 1 -- 172.21.15.107:0/1667661854 learned_addr learned my addr 172.21.15.107:0/1667661854 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:23.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bd98a640 1 --2- 172.21.15.107:0/1667661854 >> v2:172.21.15.110:3300/0 conn(0x7fb8b8152040 0x7fb8b80c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:23.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900995155 0 0) 0x7fb8b8155680 con 0x7fb8b806a4b0 2023-10-25T00:03:23.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a0003530 con 0x7fb8b806a4b0 2023-10-25T00:03:23.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3628295740 0 0) 0x7fb8a0003530 con 0x7fb8b806a4b0 2023-10-25T00:03:23.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 >> v2:172.21.15.110:3300/0 conn(0x7fb8b8152040 msgr2=0x7fb8b80c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:23.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 --2- 172.21.15.107:0/1667661854 >> v2:172.21.15.110:3300/0 conn(0x7fb8b8152040 0x7fb8b80c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:23.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 >> v1:172.21.15.110:6789/0 conn(0x7fb8b80c8100 legacy=0x7fb8b80abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:23.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b8155030 con 0x7fb8b806a4b0 2023-10-25T00:03:23.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb8a8001a00 con 0x7fb8b806a4b0 2023-10-25T00:03:23.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 >> v1:172.21.15.107:6789/0 conn(0x7fb8b806a4b0 legacy=0x7fb8b80ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:23.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8a0003980 0x7fb8a0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:23.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 0x7fb8a0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:23.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8a00052b0 0x7fb8a000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b8154d50 con 0x7fb8a00047d0 2023-10-25T00:03:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a000c180 con 0x7fb8a0003980 2023-10-25T00:03:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a000c340 con 0x7fb8a00052b0 2023-10-25T00:03:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bd189640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 0x7fb8a0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bd98a640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8a0003980 0x7fb8a0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8be18b640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8a00052b0 0x7fb8a000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bd189640 1 -- 172.21.15.107:0/1667661854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8a00052b0 msgr2=0x7fb8a000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bd189640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8a00052b0 0x7fb8a000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:23.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bd189640 1 -- 172.21.15.107:0/1667661854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8a0003980 msgr2=0x7fb8a0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:23.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bd189640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8a0003980 0x7fb8a0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:23.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8bd189640 1 -- 172.21.15.107:0/1667661854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a0003530 con 0x7fb8a00047d0 2023-10-25T00:03:23.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8be18b640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8a00052b0 0x7fb8a000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:23.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.923+0000 7fb8be18b640 1 -- 172.21.15.107:0/1667661854 reap_dead start 2023-10-25T00:03:23.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bd189640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 0x7fb8a0004c50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80c8560 tx=0x7fb8ac01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ccc2f932f9d9f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:23.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac0050d0 con 0x7fb8a00047d0 2023-10-25T00:03:23.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac0a7050 con 0x7fb8a00047d0 2023-10-25T00:03:23.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bc988640 1 -- 172.21.15.107:0/1667661854 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac005490 con 0x7fb8a00047d0 2023-10-25T00:03:23.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1667661854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 msgr2=0x7fb8a0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:23.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 0x7fb8a0004c50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fb8b80c8560 tx=0x7fb8ac01c360 comp rx=0 tx=0).stop 2023-10-25T00:03:23.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1667661854 shutdown_connections 2023-10-25T00:03:23.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 --2- 172.21.15.107:0/1667661854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 0x7fb8a0004c50 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:23.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1667661854 >> 172.21.15.107:0/1667661854 conn(0x7fb8b80ca200 msgr2=0x7fb8b80ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:23.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1667661854 shutdown_connections 2023-10-25T00:03:23.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1667661854 wait complete. 2023-10-25T00:03:23.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 Processor -- start 2023-10-25T00:03:23.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 -- start start 2023-10-25T00:03:23.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 0x7fb8b81e2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b80bd080 0x7fb8b81f0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8b8150560 0x7fb8b81f1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b81f1590 con 0x7fb8a00047d0 2023-10-25T00:03:23.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b81ee670 con 0x7fb8b80bd080 2023-10-25T00:03:23.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.927+0000 7fb8bfbee640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b814fe80 con 0x7fb8b8150560 2023-10-25T00:03:23.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd98a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 0x7fb8b81e2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:23.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd98a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 0x7fb8b81e2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44326/0 (socket says 172.21.15.107:44326) 2023-10-25T00:03:23.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd98a640 1 -- 172.21.15.107:0/1987629156 learned_addr learned my addr 172.21.15.107:0/1987629156 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:23.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8be18b640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8b8150560 0x7fb8b81f1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:23.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd189640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b80bd080 0x7fb8b81f0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:23.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd98a640 1 -- 172.21.15.107:0/1987629156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8b8150560 msgr2=0x7fb8b81f1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:23.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd98a640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8b8150560 0x7fb8b81f1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:23.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd98a640 1 -- 172.21.15.107:0/1987629156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b80bd080 msgr2=0x7fb8b81f0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:23.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd98a640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b80bd080 0x7fb8b81f0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:23.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd98a640 1 -- 172.21.15.107:0/1987629156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b8150160 con 0x7fb8a00047d0 2023-10-25T00:03:23.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd98a640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 0x7fb8b81e2330 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0011660 tx=0x7fb8b0011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:23.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8b67fc640 1 -- 172.21.15.107:0/1987629156 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b000f960 con 0x7fb8a00047d0 2023-10-25T00:03:23.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8b67fc640 1 -- 172.21.15.107:0/1987629156 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b000fac0 con 0x7fb8a00047d0 2023-10-25T00:03:23.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8b67fc640 1 -- 172.21.15.107:0/1987629156 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b000fe30 con 0x7fb8a00047d0 2023-10-25T00:03:23.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1987629156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b814f650 con 0x7fb8a00047d0 2023-10-25T00:03:23.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1987629156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b81e2870 con 0x7fb8a00047d0 2023-10-25T00:03:23.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1987629156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b814f4b0 con 0x7fb8a00047d0 2023-10-25T00:03:23.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8b67fc640 1 -- 172.21.15.107:0/1987629156 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb8b0015460 con 0x7fb8a00047d0 2023-10-25T00:03:23.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8b67fc640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8a4065e60 0x7fb8a4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:23.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8b67fc640 1 -- 172.21.15.107:0/1987629156 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7fb8b00845b0 con 0x7fb8a00047d0 2023-10-25T00:03:23.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd189640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8a4065e60 0x7fb8a4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:23.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.931+0000 7fb8bd189640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8a4065e60 0x7fb8a4068320 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00057b0 tx=0x7fb8ac0011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:23.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:23.935+0000 7fb8b67fc640 1 -- 172.21.15.107:0/1987629156 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb8b814f4b0 con 0x7fb8a00047d0 2023-10-25T00:03:24.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.123+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1987629156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8b814f4b0 con 0x7fb8a00047d0 2023-10-25T00:03:24.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.123+0000 7fb8b67fc640 1 -- 172.21.15.107:0/1987629156 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fb8b814f4b0 con 0x7fb8a00047d0 2023-10-25T00:03:24.125 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:24.125 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5764,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:24.125 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 85 2023-10-25T00:03:24.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1987629156 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8a4065e60 msgr2=0x7fb8a4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:24.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8a4065e60 0x7fb8a4068320 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00057b0 tx=0x7fb8ac0011a0 comp rx=0 tx=0).stop 2023-10-25T00:03:24.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1987629156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 msgr2=0x7fb8b81e2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:24.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 0x7fb8b81e2330 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0011660 tx=0x7fb8b0011b20 comp rx=0 tx=0).stop 2023-10-25T00:03:24.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1987629156 shutdown_connections 2023-10-25T00:03:24.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8a4065e60 0x7fb8a4068320 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:24.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8b8150560 0x7fb8b81f1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:24.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8b80bd080 0x7fb8b81f0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:24.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 --2- 172.21.15.107:0/1987629156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8a00047d0 0x7fb8b81e2330 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:24.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1987629156 >> 172.21.15.107:0/1987629156 conn(0x7fb8b80ca200 msgr2=0x7fb8b80ca5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:24.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1987629156 shutdown_connections 2023-10-25T00:03:24.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:24.127+0000 7fb8bfbee640 1 -- 172.21.15.107:0/1987629156 wait complete. 2023-10-25T00:03:24.142 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:03:26.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.275+0000 7f7aa0394640 1 Processor -- start 2023-10-25T00:03:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.275+0000 7f7aa0394640 1 -- start start 2023-10-25T00:03:26.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7aa0394640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7a98151d60 0x7f7a980a9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7aa0394640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a98154840 con 0x7f7a980c24b0 2023-10-25T00:03:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7aa0394640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a980aa640 con 0x7f7a9806a780 2023-10-25T00:03:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7aa0394640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a980aadc0 con 0x7f7a98151d60 2023-10-25T00:03:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9e931640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7a980c24b0 0x7f7a9815f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49902/0 (socket says 172.21.15.107:49902) 2023-10-25T00:03:26.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9e931640 1 -- 172.21.15.107:0/695687478 learned_addr learned my addr 172.21.15.107:0/695687478 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9e130640 1 --2- 172.21.15.107:0/695687478 >> v2:172.21.15.110:3300/0 conn(0x7f7a98151d60 0x7f7a980a9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231612347 0 0) 0x7f7a98154840 con 0x7f7a980c24b0 2023-10-25T00:03:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a84003530 con 0x7f7a980c24b0 2023-10-25T00:03:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 160260021 0 0) 0x7f7a84003530 con 0x7f7a980c24b0 2023-10-25T00:03:26.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 >> v2:172.21.15.110:3300/0 conn(0x7f7a98151d60 msgr2=0x7f7a980a9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:26.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 --2- 172.21.15.107:0/695687478 >> v2:172.21.15.110:3300/0 conn(0x7f7a98151d60 0x7f7a980a9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 >> v1:172.21.15.110:6789/0 conn(0x7f7a9806a780 legacy=0x7f7a98154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:26.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a980aafa0 con 0x7f7a980c24b0 2023-10-25T00:03:26.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f7a90001a00 con 0x7f7a980c24b0 2023-10-25T00:03:26.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 >> v1:172.21.15.107:6789/0 conn(0x7f7a980c24b0 legacy=0x7f7a9815f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 --2- 172.21.15.107:0/695687478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a84003980 0x7f7a84003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 --2- 172.21.15.107:0/695687478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 0x7f7a84004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 --2- 172.21.15.107:0/695687478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a84005190 0x7f7a8400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a980aadc0 con 0x7f7a840047d0 2023-10-25T00:03:26.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8400c030 con 0x7f7a84003980 2023-10-25T00:03:26.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8400c1f0 con 0x7f7a84005190 2023-10-25T00:03:26.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d92f640 1 --2- 172.21.15.107:0/695687478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 0x7f7a84004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:26.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9e130640 1 --2- 172.21.15.107:0/695687478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a84003980 0x7f7a84003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:26.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d92f640 1 -- 172.21.15.107:0/695687478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a84005190 msgr2=0x7f7a8400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:26.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d92f640 1 --2- 172.21.15.107:0/695687478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a84005190 0x7f7a8400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d92f640 1 -- 172.21.15.107:0/695687478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a84003980 msgr2=0x7f7a84003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:26.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d92f640 1 --2- 172.21.15.107:0/695687478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a84003980 0x7f7a84003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d92f640 1 -- 172.21.15.107:0/695687478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a84003530 con 0x7f7a840047d0 2023-10-25T00:03:26.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9e130640 1 --2- 172.21.15.107:0/695687478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a84003980 0x7f7a84003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:26.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9e931640 1 -- 172.21.15.107:0/695687478 reap_dead start 2023-10-25T00:03:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d92f640 1 --2- 172.21.15.107:0/695687478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 0x7f7a84004c50 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f7a94001780 tx=0x7f7a9401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6a3bc4b14eee13 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a940052d0 con 0x7f7a840047d0 2023-10-25T00:03:26.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a940a7050 con 0x7f7a840047d0 2023-10-25T00:03:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.279+0000 7f7a9d12e640 1 -- 172.21.15.107:0/695687478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a94003600 con 0x7f7a840047d0 2023-10-25T00:03:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- 172.21.15.107:0/695687478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 msgr2=0x7f7a84004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 --2- 172.21.15.107:0/695687478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 0x7f7a84004c50 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f7a94001780 tx=0x7f7a9401c3e0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- 172.21.15.107:0/695687478 shutdown_connections 2023-10-25T00:03:26.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 --2- 172.21.15.107:0/695687478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 0x7f7a84004c50 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- 172.21.15.107:0/695687478 >> 172.21.15.107:0/695687478 conn(0x7f7a980c2950 msgr2=0x7f7a980c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:26.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- 172.21.15.107:0/695687478 shutdown_connections 2023-10-25T00:03:26.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- 172.21.15.107:0/695687478 wait complete. 2023-10-25T00:03:26.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 Processor -- start 2023-10-25T00:03:26.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- start start 2023-10-25T00:03:26.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 0x7f7a98149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:26.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a98153970 0x7f7a98153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:26.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a98157210 0x7f7a98157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:26.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a981ed4d0 con 0x7f7a840047d0 2023-10-25T00:03:26.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a981ed770 con 0x7f7a98157210 2023-10-25T00:03:26.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a981eda50 con 0x7f7a98153970 2023-10-25T00:03:26.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9e130640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 0x7f7a98149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:26.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9e130640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 0x7f7a98149c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44340/0 (socket says 172.21.15.107:44340) 2023-10-25T00:03:26.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9e130640 1 -- 172.21.15.107:0/2720002600 learned_addr learned my addr 172.21.15.107:0/2720002600 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:26.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9d92f640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a98153970 0x7f7a98153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:26.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9e931640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a98157210 0x7f7a98157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:26.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9e130640 1 -- 172.21.15.107:0/2720002600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a98153970 msgr2=0x7f7a98153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:26.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9e130640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a98153970 0x7f7a98153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9e130640 1 -- 172.21.15.107:0/2720002600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a98157210 msgr2=0x7f7a98157690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:26.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9e130640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a98157210 0x7f7a98157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9e130640 1 -- 172.21.15.107:0/2720002600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a981edd30 con 0x7f7a840047d0 2023-10-25T00:03:26.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9d92f640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a98153970 0x7f7a98153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:26.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a9e130640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 0x7f7a98149c50 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f7a88010790 tx=0x7f7a88010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:26.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a8effd640 1 -- 172.21.15.107:0/2720002600 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a88004220 con 0x7f7a840047d0 2023-10-25T00:03:26.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a8effd640 1 -- 172.21.15.107:0/2720002600 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a88004380 con 0x7f7a840047d0 2023-10-25T00:03:26.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- 172.21.15.107:0/2720002600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a98154430 con 0x7f7a840047d0 2023-10-25T00:03:26.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7a8effd640 1 -- 172.21.15.107:0/2720002600 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8800eb20 con 0x7f7a840047d0 2023-10-25T00:03:26.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- 172.21.15.107:0/2720002600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9814a190 con 0x7f7a840047d0 2023-10-25T00:03:26.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.283+0000 7f7aa0394640 1 -- 172.21.15.107:0/2720002600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a98157e00 con 0x7f7a840047d0 2023-10-25T00:03:26.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.287+0000 7f7a8effd640 1 -- 172.21.15.107:0/2720002600 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7a8800ec80 con 0x7f7a840047d0 2023-10-25T00:03:26.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.287+0000 7f7a8effd640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a7c065e90 0x7f7a7c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:26.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.287+0000 7f7a8effd640 1 -- 172.21.15.107:0/2720002600 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 8251+0+0 (secure 0 0 0) 0x7f7a88084610 con 0x7f7a840047d0 2023-10-25T00:03:26.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.287+0000 7f7a9d92f640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a7c065e90 0x7f7a7c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:26.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.287+0000 7f7a9d92f640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a7c065e90 0x7f7a7c068350 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7a94001750 tx=0x7f7a940011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:26.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.291+0000 7f7a8effd640 1 -- 172.21.15.107:0/2720002600 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7a88051490 con 0x7f7a840047d0 2023-10-25T00:03:26.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7aa0394640 1 -- 172.21.15.107:0/2720002600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a981edec0 con 0x7f7a840047d0 2023-10-25T00:03:26.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7a8effd640 1 -- 172.21.15.107:0/2720002600 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+5773 (secure 0 0 0) 0x7f7a88010ea0 con 0x7f7a840047d0 2023-10-25T00:03:26.476 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:26.477 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":5764,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:02:01.619602+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6396},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6396":{"gid":6396,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6835/3232233949","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3232233949},{"type":"v1","addr":"172.21.15.107:6835","nonce":3232233949}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:26.477 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 85 2023-10-25T00:03:26.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7aa0394640 1 -- 172.21.15.107:0/2720002600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a7c065e90 msgr2=0x7f7a7c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:26.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7aa0394640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a7c065e90 0x7f7a7c068350 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7a94001750 tx=0x7f7a940011a0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7aa0394640 1 -- 172.21.15.107:0/2720002600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 msgr2=0x7f7a98149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:26.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7aa0394640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 0x7f7a98149c50 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f7a88010790 tx=0x7f7a88010c50 comp rx=0 tx=0).stop 2023-10-25T00:03:26.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7aa0394640 1 -- 172.21.15.107:0/2720002600 shutdown_connections 2023-10-25T00:03:26.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7aa0394640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a98157210 0x7f7a98157690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7aa0394640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a7c065e90 0x7f7a7c068350 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7aa0394640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a98153970 0x7f7a98153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7aa0394640 1 --2- 172.21.15.107:0/2720002600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a840047d0 0x7f7a98149c50 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:26.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.475+0000 7f7aa0394640 1 -- 172.21.15.107:0/2720002600 >> 172.21.15.107:0/2720002600 conn(0x7f7a980c2950 msgr2=0x7f7a9814eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:26.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.479+0000 7f7aa0394640 1 -- 172.21.15.107:0/2720002600 shutdown_connections 2023-10-25T00:03:26.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:26.479+0000 7f7aa0394640 1 -- 172.21.15.107:0/2720002600 wait complete. 2023-10-25T00:03:26.495 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:03:28.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:28.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.631+0000 7fcc32a97640 1 Processor -- start 2023-10-25T00:03:28.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.631+0000 7fcc32a97640 1 -- start start 2023-10-25T00:03:28.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc32a97640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc2c154860 0x7fcc2c158230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:28.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc32a97640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c0adf70 con 0x7fcc2c0720c0 2023-10-25T00:03:28.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc32a97640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c06e710 con 0x7fcc2c071ce0 2023-10-25T00:03:28.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc32a97640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c06d900 con 0x7fcc2c154860 2023-10-25T00:03:28.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc2bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcc2c0720c0 0x7fcc2c13d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49912/0 (socket says 172.21.15.107:49912) 2023-10-25T00:03:28.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc2bfff640 1 -- 172.21.15.107:0/3918171726 learned_addr learned my addr 172.21.15.107:0/3918171726 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:28.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc31034640 1 --2- 172.21.15.107:0/3918171726 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc2c154860 0x7fcc2c158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:28.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc2b7fe640 1 -- 172.21.15.107:0/3918171726 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314921116 0 0) 0x7fcc2c0adf70 con 0x7fcc2c0720c0 2023-10-25T00:03:28.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc2b7fe640 1 -- 172.21.15.107:0/3918171726 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc14003530 con 0x7fcc2c0720c0 2023-10-25T00:03:28.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc2b7fe640 1 -- 172.21.15.107:0/3918171726 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530766592 0 0) 0x7fcc2c06e710 con 0x7fcc2c071ce0 2023-10-25T00:03:28.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc2b7fe640 1 -- 172.21.15.107:0/3918171726 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2c0adf70 con 0x7fcc2c071ce0 2023-10-25T00:03:28.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc31034640 1 -- 172.21.15.107:0/3918171726 >> v1:172.21.15.110:6789/0 conn(0x7fcc2c071ce0 legacy=0x7fcc2c0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:28.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc31034640 1 -- 172.21.15.107:0/3918171726 >> v1:172.21.15.107:6789/0 conn(0x7fcc2c0720c0 legacy=0x7fcc2c13d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:28.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.635+0000 7fcc31034640 1 -- 172.21.15.107:0/3918171726 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c154d20 con 0x7fcc2c154860 2023-10-25T00:03:28.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc31034640 1 --2- 172.21.15.107:0/3918171726 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc2c154860 0x7fcc2c158230 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009dc0 tx=0x7fcc1c01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cd7038c9cf8ef77 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:28.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc2b7fe640 1 -- 172.21.15.107:0/3918171726 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c0a5970 con 0x7fcc2c154860 2023-10-25T00:03:28.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc2b7fe640 1 -- 172.21.15.107:0/3918171726 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c0a5ad0 con 0x7fcc2c154860 2023-10-25T00:03:28.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc2b7fe640 1 -- 172.21.15.107:0/3918171726 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c0a5e60 con 0x7fcc2c154860 2023-10-25T00:03:28.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 -- 172.21.15.107:0/3918171726 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc2c154860 msgr2=0x7fcc2c158230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:28.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 --2- 172.21.15.107:0/3918171726 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc2c154860 0x7fcc2c158230 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009dc0 tx=0x7fcc1c01c8b0 comp rx=0 tx=0).stop 2023-10-25T00:03:28.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 -- 172.21.15.107:0/3918171726 shutdown_connections 2023-10-25T00:03:28.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 --2- 172.21.15.107:0/3918171726 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc2c154860 0x7fcc2c158230 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:28.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 -- 172.21.15.107:0/3918171726 >> 172.21.15.107:0/3918171726 conn(0x7fcc2c070f60 msgr2=0x7fcc2c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:28.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 -- 172.21.15.107:0/3918171726 shutdown_connections 2023-10-25T00:03:28.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 -- 172.21.15.107:0/3918171726 wait complete. 2023-10-25T00:03:28.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 Processor -- start 2023-10-25T00:03:28.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 -- start start 2023-10-25T00:03:28.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c071ce0 0x7fcc2c0c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:28.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc2c0720c0 0x7fcc2c1ecdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:28.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c154860 0x7fcc2c1ed310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:28.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c1f0a00 con 0x7fcc2c071ce0 2023-10-25T00:03:28.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c1f0ce0 con 0x7fcc2c154860 2023-10-25T00:03:28.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc32a97640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c156e90 con 0x7fcc2c0720c0 2023-10-25T00:03:28.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc30833640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c071ce0 0x7fcc2c0c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:28.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc30833640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c071ce0 0x7fcc2c0c74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44342/0 (socket says 172.21.15.107:44342) 2023-10-25T00:03:28.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc30833640 1 -- 172.21.15.107:0/2422449410 learned_addr learned my addr 172.21.15.107:0/2422449410 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:28.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc2bfff640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc2c0720c0 0x7fcc2c1ecdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:28.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.639+0000 7fcc31034640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c154860 0x7fcc2c1ed310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:28.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc30833640 1 -- 172.21.15.107:0/2422449410 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc2c0720c0 msgr2=0x7fcc2c1ecdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:28.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc30833640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc2c0720c0 0x7fcc2c1ecdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:28.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc30833640 1 -- 172.21.15.107:0/2422449410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c154860 msgr2=0x7fcc2c1ed310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:28.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc30833640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c154860 0x7fcc2c1ed310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:28.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc30833640 1 -- 172.21.15.107:0/2422449410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c009a70 con 0x7fcc2c071ce0 2023-10-25T00:03:28.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc31034640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c154860 0x7fcc2c1ed310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:28.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc30833640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c071ce0 0x7fcc2c0c74e0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400c430 tx=0x7fcc2400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:28.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc297fa640 1 -- 172.21.15.107:0/2422449410 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc240052d0 con 0x7fcc2c071ce0 2023-10-25T00:03:28.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc32a97640 1 -- 172.21.15.107:0/2422449410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2c13e200 con 0x7fcc2c071ce0 2023-10-25T00:03:28.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc297fa640 1 -- 172.21.15.107:0/2422449410 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc24005450 con 0x7fcc2c071ce0 2023-10-25T00:03:28.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc297fa640 1 -- 172.21.15.107:0/2422449410 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24003840 con 0x7fcc2c071ce0 2023-10-25T00:03:28.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc32a97640 1 -- 172.21.15.107:0/2422449410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2c157450 con 0x7fcc2c071ce0 2023-10-25T00:03:28.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc32a97640 1 -- 172.21.15.107:0/2422449410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2c0746e0 con 0x7fcc2c071ce0 2023-10-25T00:03:28.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc297fa640 1 -- 172.21.15.107:0/2422449410 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcc240039a0 con 0x7fcc2c071ce0 2023-10-25T00:03:28.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc297fa640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc18065e60 0x7fcc18068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:28.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc297fa640 1 -- 172.21.15.107:0/2422449410 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7fcc24083760 con 0x7fcc2c071ce0 2023-10-25T00:03:28.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc2bfff640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc18065e60 0x7fcc18068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:28.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.643+0000 7fcc2bfff640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc18065e60 0x7fcc18068320 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c1ea930 tx=0x7fcc20007d40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:28.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.647+0000 7fcc297fa640 1 -- 172.21.15.107:0/2422449410 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fcc24050530 con 0x7fcc2c071ce0 2023-10-25T00:03:28.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.847+0000 7fcc32a97640 1 -- 172.21.15.107:0/2422449410 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc2c06f940 con 0x7fcc2c071ce0 2023-10-25T00:03:28.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc297fa640 1 -- 172.21.15.107:0/2422449410 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5012 (secure 0 0 0) 0x7fcc24053b50 con 0x7fcc2c071ce0 2023-10-25T00:03:28.851 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:28.851 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:28.569856+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:28.852 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 86 2023-10-25T00:03:28.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 -- 172.21.15.107:0/2422449410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc18065e60 msgr2=0x7fcc18068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:28.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc18065e60 0x7fcc18068320 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c1ea930 tx=0x7fcc20007d40 comp rx=0 tx=0).stop 2023-10-25T00:03:28.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 -- 172.21.15.107:0/2422449410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c071ce0 msgr2=0x7fcc2c0c74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:28.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c071ce0 0x7fcc2c0c74e0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400c430 tx=0x7fcc2400c8f0 comp rx=0 tx=0).stop 2023-10-25T00:03:28.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 -- 172.21.15.107:0/2422449410 shutdown_connections 2023-10-25T00:03:28.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc2c154860 0x7fcc2c1ed310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:28.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc18065e60 0x7fcc18068320 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:28.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcc2c0720c0 0x7fcc2c1ecdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:28.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 --2- 172.21.15.107:0/2422449410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c071ce0 0x7fcc2c0c74e0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:28.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 -- 172.21.15.107:0/2422449410 >> 172.21.15.107:0/2422449410 conn(0x7fcc2c070f60 msgr2=0x7fcc2c1525a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:28.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 -- 172.21.15.107:0/2422449410 shutdown_connections 2023-10-25T00:03:28.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.851+0000 7fcc32a97640 1 -- 172.21.15.107:0/2422449410 wait complete. 2023-10-25T00:03:28.870 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2023-10-25T00:03:28.871 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:03:28.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:28.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d3b139640 1 Processor -- start 2023-10-25T00:03:28.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d3b139640 1 -- start start 2023-10-25T00:03:28.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d3b139640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3d340c24c0 0x7f3d3406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:28.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d3b139640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d34150bc0 con 0x7f3d34071220 2023-10-25T00:03:28.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d3b139640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3406c510 con 0x7f3d34151960 2023-10-25T00:03:28.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d3b139640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3406c7f0 con 0x7f3d340c24c0 2023-10-25T00:03:28.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d38ed5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3d34071220 0x7f3d3406bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49928/0 (socket says 172.21.15.107:49928) 2023-10-25T00:03:28.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d38ed5640 1 -- 172.21.15.107:0/1975884634 learned_addr learned my addr 172.21.15.107:0/1975884634 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:28.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d396d6640 1 --2- 172.21.15.107:0/1975884634 >> v2:172.21.15.110:3300/0 conn(0x7f3d340c24c0 0x7f3d3406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:28.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d337fe640 1 -- 172.21.15.107:0/1975884634 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487510848 0 0) 0x7f3d34150bc0 con 0x7f3d34071220 2023-10-25T00:03:28.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d337fe640 1 -- 172.21.15.107:0/1975884634 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d20003530 con 0x7f3d34071220 2023-10-25T00:03:28.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d337fe640 1 -- 172.21.15.107:0/1975884634 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531657070 0 0) 0x7f3d3406c510 con 0x7f3d34151960 2023-10-25T00:03:28.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d337fe640 1 -- 172.21.15.107:0/1975884634 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d34150bc0 con 0x7f3d34151960 2023-10-25T00:03:28.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d396d6640 1 -- 172.21.15.107:0/1975884634 >> v1:172.21.15.110:6789/0 conn(0x7f3d34151960 legacy=0x7f3d34071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:28.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d396d6640 1 -- 172.21.15.107:0/1975884634 >> v1:172.21.15.107:6789/0 conn(0x7f3d34071220 legacy=0x7f3d3406bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:28.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.991+0000 7f3d396d6640 1 -- 172.21.15.107:0/1975884634 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34150e80 con 0x7f3d340c24c0 2023-10-25T00:03:28.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d396d6640 1 --2- 172.21.15.107:0/1975884634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d340c24c0 0x7f3d3406a9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00aea0 tx=0x7f3d2c01d9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe4d0b4fef130c1c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:28.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d337fe640 1 -- 172.21.15.107:0/1975884634 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c00a2d0 con 0x7f3d340c24c0 2023-10-25T00:03:28.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d337fe640 1 -- 172.21.15.107:0/1975884634 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c0a6620 con 0x7f3d340c24c0 2023-10-25T00:03:28.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d337fe640 1 -- 172.21.15.107:0/1975884634 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c0a6990 con 0x7f3d340c24c0 2023-10-25T00:03:28.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 -- 172.21.15.107:0/1975884634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d340c24c0 msgr2=0x7f3d3406a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:28.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 --2- 172.21.15.107:0/1975884634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d340c24c0 0x7f3d3406a9c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c00aea0 tx=0x7f3d2c01d9c0 comp rx=0 tx=0).stop 2023-10-25T00:03:28.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 -- 172.21.15.107:0/1975884634 shutdown_connections 2023-10-25T00:03:28.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 --2- 172.21.15.107:0/1975884634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d340c24c0 0x7f3d3406a9c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:28.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 -- 172.21.15.107:0/1975884634 >> 172.21.15.107:0/1975884634 conn(0x7f3d340c2950 msgr2=0x7f3d340c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:28.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 -- 172.21.15.107:0/1975884634 shutdown_connections 2023-10-25T00:03:28.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 -- 172.21.15.107:0/1975884634 wait complete. 2023-10-25T00:03:28.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 Processor -- start 2023-10-25T00:03:28.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 -- start start 2023-10-25T00:03:29.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d34071220 0x7f3d34146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:29.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d340c24c0 0x7f3d3414dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:29.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d34151960 0x7f3d34149300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:29.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34154140 con 0x7f3d340c24c0 2023-10-25T00:03:29.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d34147160 con 0x7f3d34151960 2023-10-25T00:03:29.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d3b139640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3414d590 con 0x7f3d34071220 2023-10-25T00:03:29.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d33fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d340c24c0 0x7f3d3414dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:29.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d33fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d340c24c0 0x7f3d3414dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44356/0 (socket says 172.21.15.107:44356) 2023-10-25T00:03:29.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d33fff640 1 -- 172.21.15.107:0/4257546670 learned_addr learned my addr 172.21.15.107:0/4257546670 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:29.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d38ed5640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d34071220 0x7f3d34146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:29.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d396d6640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d34151960 0x7f3d34149300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:29.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d33fff640 1 -- 172.21.15.107:0/4257546670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d34071220 msgr2=0x7f3d34146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:29.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d33fff640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d34071220 0x7f3d34146c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:29.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d33fff640 1 -- 172.21.15.107:0/4257546670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d34151960 msgr2=0x7f3d34149300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:29.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d33fff640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d34151960 0x7f3d34149300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:29.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.995+0000 7f3d33fff640 1 -- 172.21.15.107:0/4257546670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c00ab80 con 0x7f3d340c24c0 2023-10-25T00:03:29.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d33fff640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d340c24c0 0x7f3d3414dd50 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00c530 tx=0x7f3d1c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d317fa640 1 -- 172.21.15.107:0/4257546670 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0052d0 con 0x7f3d340c24c0 2023-10-25T00:03:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d317fa640 1 -- 172.21.15.107:0/4257546670 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c005450 con 0x7f3d340c24c0 2023-10-25T00:03:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d3b139640 1 -- 172.21.15.107:0/4257546670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d34153e60 con 0x7f3d340c24c0 2023-10-25T00:03:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d317fa640 1 -- 172.21.15.107:0/4257546670 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c012870 con 0x7f3d340c24c0 2023-10-25T00:03:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d3b139640 1 -- 172.21.15.107:0/4257546670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d34153b60 con 0x7f3d340c24c0 2023-10-25T00:03:29.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d3b139640 1 -- 172.21.15.107:0/4257546670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3414d3f0 con 0x7f3d340c24c0 2023-10-25T00:03:29.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d317fa640 1 -- 172.21.15.107:0/4257546670 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3d1c0129d0 con 0x7f3d340c24c0 2023-10-25T00:03:29.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d317fa640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d14065e90 0x7f3d14068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:29.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d317fa640 1 -- 172.21.15.107:0/4257546670 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7f3d1c083b50 con 0x7f3d340c24c0 2023-10-25T00:03:29.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d38ed5640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d14065e90 0x7f3d14068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:29.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:28.999+0000 7f3d38ed5640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d14065e90 0x7f3d14068350 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3d240011f0 tx=0x7f3d24007d40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:29.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.003+0000 7f3d317fa640 1 -- 172.21.15.107:0/4257546670 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3d3414d3f0 con 0x7f3d340c24c0 2023-10-25T00:03:29.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.179+0000 7f3d3b139640 1 -- 172.21.15.107:0/4257546670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d3414d3f0 con 0x7f3d340c24c0 2023-10-25T00:03:29.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.179+0000 7f3d317fa640 1 -- 172.21.15.107:0/4257546670 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5012 (secure 0 0 0) 0x7f3d3414d3f0 con 0x7f3d340c24c0 2023-10-25T00:03:29.182 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:29.182 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:28.569856+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:29.182 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 86 2023-10-25T00:03:29.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 -- 172.21.15.107:0/4257546670 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d14065e90 msgr2=0x7f3d14068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:29.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d14065e90 0x7f3d14068350 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3d240011f0 tx=0x7f3d24007d40 comp rx=0 tx=0).stop 2023-10-25T00:03:29.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 -- 172.21.15.107:0/4257546670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d340c24c0 msgr2=0x7f3d3414dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:29.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d340c24c0 0x7f3d3414dd50 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00c530 tx=0x7f3d1c00c9f0 comp rx=0 tx=0).stop 2023-10-25T00:03:29.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 -- 172.21.15.107:0/4257546670 shutdown_connections 2023-10-25T00:03:29.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d34151960 0x7f3d34149300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:29.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d340c24c0 0x7f3d3414dd50 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:29.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d14065e90 0x7f3d14068350 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:29.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 --2- 172.21.15.107:0/4257546670 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d34071220 0x7f3d34146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:29.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 -- 172.21.15.107:0/4257546670 >> 172.21.15.107:0/4257546670 conn(0x7f3d340c2950 msgr2=0x7f3d3406bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:29.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 -- 172.21.15.107:0/4257546670 shutdown_connections 2023-10-25T00:03:29.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:29.183+0000 7f3d3b139640 1 -- 172.21.15.107:0/4257546670 wait complete. 2023-10-25T00:03:29.199 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 86, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 86, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5764": { "addr": "172.21.15.110:6834/2577776351", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 2577776351, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 2577776351, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5764, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:03:28.569856+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5764 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 84, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6483": { "addr": "172.21.15.107:6837/319918260", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 319918260, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 319918260, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6483, "incarnation": 81, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:02:26.743229+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6483 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 59, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:03:31.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:31.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.331+0000 7f2233ebc640 1 Processor -- start 2023-10-25T00:03:31.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.331+0000 7f2233ebc640 1 -- start start 2023-10-25T00:03:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2233ebc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f222c151960 0x7f222c071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2233ebc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f222c0c24c0 0x7f222c06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:31.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2233ebc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f222c150bc0 con 0x7f222c071220 2023-10-25T00:03:31.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2233ebc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f222c06c7f0 con 0x7f222c151960 2023-10-25T00:03:31.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2233ebc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222c06c510 con 0x7f222c0c24c0 2023-10-25T00:03:31.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2231457640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f222c151960 0x7f222c071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:31.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2231c58640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f222c071220 0x7f222c06bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49938/0 (socket says 172.21.15.107:49938) 2023-10-25T00:03:31.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2231c58640 1 -- 172.21.15.107:0/2665487804 learned_addr learned my addr 172.21.15.107:0/2665487804 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2232459640 1 --2- 172.21.15.107:0/2665487804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f222c0c24c0 0x7f222c06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2231457640 1 -- 172.21.15.107:0/2665487804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f222c0c24c0 msgr2=0x7f222c06a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2231457640 1 --2- 172.21.15.107:0/2665487804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f222c0c24c0 0x7f222c06a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2231457640 1 -- 172.21.15.107:0/2665487804 >> v1:172.21.15.107:6789/0 conn(0x7f222c071220 legacy=0x7f222c06bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2230c56640 1 -- 172.21.15.107:0/2665487804 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026704847 0 0) 0x7f222c150bc0 con 0x7f222c071220 2023-10-25T00:03:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2231457640 1 -- 172.21.15.107:0/2665487804 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c150e80 con 0x7f222c151960 2023-10-25T00:03:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2231457640 1 --2- 172.21.15.107:0/2665487804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f222c151960 0x7f222c071a90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f221c0097c0 tx=0x7f221c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b8ee34335c1ee server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2230c56640 1 -- 172.21.15.107:0/2665487804 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c0ab600 con 0x7f222c151960 2023-10-25T00:03:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2230c56640 1 -- 172.21.15.107:0/2665487804 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221c0a7050 con 0x7f222c151960 2023-10-25T00:03:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.335+0000 7f2230c56640 1 -- 172.21.15.107:0/2665487804 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c0ab980 con 0x7f222c151960 2023-10-25T00:03:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 -- 172.21.15.107:0/2665487804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f222c151960 msgr2=0x7f222c071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 --2- 172.21.15.107:0/2665487804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f222c151960 0x7f222c071a90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f221c0097c0 tx=0x7f221c01c3f0 comp rx=0 tx=0).stop 2023-10-25T00:03:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 -- 172.21.15.107:0/2665487804 shutdown_connections 2023-10-25T00:03:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 --2- 172.21.15.107:0/2665487804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f222c0c24c0 0x7f222c06a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 --2- 172.21.15.107:0/2665487804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f222c151960 0x7f222c071a90 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 -- 172.21.15.107:0/2665487804 >> 172.21.15.107:0/2665487804 conn(0x7f222c0c2950 msgr2=0x7f222c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 -- 172.21.15.107:0/2665487804 shutdown_connections 2023-10-25T00:03:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 -- 172.21.15.107:0/2665487804 wait complete. 2023-10-25T00:03:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 Processor -- start 2023-10-25T00:03:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 -- start start 2023-10-25T00:03:31.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f222c071220 0x7f222c148ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:31.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f222c0c24c0 0x7f222c1ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:31.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f222c151960 0x7f222c0c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:31.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c145f90 con 0x7f222c0c24c0 2023-10-25T00:03:31.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c148850 con 0x7f222c071220 2023-10-25T00:03:31.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222c149220 con 0x7f222c151960 2023-10-25T00:03:31.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2231457640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f222c0c24c0 0x7f222c1ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:31.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2231457640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f222c0c24c0 0x7f222c1ee590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44388/0 (socket says 172.21.15.107:44388) 2023-10-25T00:03:31.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2231457640 1 -- 172.21.15.107:0/3611070228 learned_addr learned my addr 172.21.15.107:0/3611070228 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:31.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2232459640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f222c151960 0x7f222c0c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:31.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2231c58640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f222c071220 0x7f222c148ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:31.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2231457640 1 -- 172.21.15.107:0/3611070228 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f222c151960 msgr2=0x7f222c0c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:31.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2231457640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f222c151960 0x7f222c0c3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:31.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2231457640 1 -- 172.21.15.107:0/3611070228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f222c071220 msgr2=0x7f222c148ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:31.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2231457640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f222c071220 0x7f222c148ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:31.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2231457640 1 -- 172.21.15.107:0/3611070228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221c009420 con 0x7f222c0c24c0 2023-10-25T00:03:31.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2231457640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f222c0c24c0 0x7f222c1ee590 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f221c0097c0 tx=0x7f221c019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:31.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f22227fc640 1 -- 172.21.15.107:0/3611070228 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c0b4400 con 0x7f222c0c24c0 2023-10-25T00:03:31.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f22227fc640 1 -- 172.21.15.107:0/3611070228 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221c0b4560 con 0x7f222c0c24c0 2023-10-25T00:03:31.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 -- 172.21.15.107:0/3611070228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222c141770 con 0x7f222c0c24c0 2023-10-25T00:03:31.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f22227fc640 1 -- 172.21.15.107:0/3611070228 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c0b4910 con 0x7f222c0c24c0 2023-10-25T00:03:31.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.339+0000 7f2233ebc640 1 -- 172.21.15.107:0/3611070228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c141d30 con 0x7f222c0c24c0 2023-10-25T00:03:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.343+0000 7f2233ebc640 1 -- 172.21.15.107:0/3611070228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f222c1489e0 con 0x7f222c0c24c0 2023-10-25T00:03:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.343+0000 7f22227fc640 1 -- 172.21.15.107:0/3611070228 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f221c0b2040 con 0x7f222c0c24c0 2023-10-25T00:03:31.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.343+0000 7f22227fc640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2210065e90 0x7f2210068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:31.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.343+0000 7f22227fc640 1 -- 172.21.15.107:0/3611070228 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7f221c11e080 con 0x7f222c0c24c0 2023-10-25T00:03:31.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.343+0000 7f2231c58640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2210065e90 0x7f2210068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:31.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.343+0000 7f2231c58640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2210065e90 0x7f2210068350 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2228000f70 tx=0x7f2228007b80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:31.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.347+0000 7f22227fc640 1 -- 172.21.15.107:0/3611070228 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f221c0eae20 con 0x7f222c0c24c0 2023-10-25T00:03:31.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.539+0000 7f2233ebc640 1 -- 172.21.15.107:0/3611070228 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f222c0746e0 con 0x7f222c0c24c0 2023-10-25T00:03:31.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.539+0000 7f22227fc640 1 -- 172.21.15.107:0/3611070228 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f221c0b0090 con 0x7f222c0c24c0 2023-10-25T00:03:31.542 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:31.542 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:30.691662+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:31.542 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 88 2023-10-25T00:03:31.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 -- 172.21.15.107:0/3611070228 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2210065e90 msgr2=0x7f2210068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:31.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2210065e90 0x7f2210068350 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2228000f70 tx=0x7f2228007b80 comp rx=0 tx=0).stop 2023-10-25T00:03:31.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 -- 172.21.15.107:0/3611070228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f222c0c24c0 msgr2=0x7f222c1ee590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:31.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f222c0c24c0 0x7f222c1ee590 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f221c0097c0 tx=0x7f221c019190 comp rx=0 tx=0).stop 2023-10-25T00:03:31.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 -- 172.21.15.107:0/3611070228 shutdown_connections 2023-10-25T00:03:31.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2210065e90 0x7f2210068350 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:31.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f222c151960 0x7f222c0c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:31.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f222c0c24c0 0x7f222c1ee590 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:31.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 --2- 172.21.15.107:0/3611070228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f222c071220 0x7f222c148ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:31.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 -- 172.21.15.107:0/3611070228 >> 172.21.15.107:0/3611070228 conn(0x7f222c0c2950 msgr2=0x7f222c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:31.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 -- 172.21.15.107:0/3611070228 shutdown_connections 2023-10-25T00:03:31.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:31.543+0000 7f2233ebc640 1 -- 172.21.15.107:0/3611070228 wait complete. 2023-10-25T00:03:31.562 INFO:tasks.mds_thrash.fs.[a]:no change 2023-10-25T00:03:33.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:33.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3453604640 1 Processor -- start 2023-10-25T00:03:33.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3453604640 1 -- start start 2023-10-25T00:03:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3453604640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f344c158560 0x7f344c0c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3453604640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344c154860 con 0x7f344c13dc20 2023-10-25T00:03:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3453604640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344c0c1f10 con 0x7f344c0720c0 2023-10-25T00:03:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3453604640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344c154b20 con 0x7f344c158560 2023-10-25T00:03:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f34513a0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f344c13dc20 0x7f344c06df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59028/0 (socket says 172.21.15.107:59028) 2023-10-25T00:03:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f34513a0640 1 -- 172.21.15.107:0/2032449064 learned_addr learned my addr 172.21.15.107:0/2032449064 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3451ba1640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f344c158560 0x7f344c0c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:33.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127165468 0 0) 0x7f344c154860 con 0x7f344c13dc20 2023-10-25T00:03:33.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3438003530 con 0x7f344c13dc20 2023-10-25T00:03:33.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368179592 0 0) 0x7f3438003530 con 0x7f344c13dc20 2023-10-25T00:03:33.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f344c158560 msgr2=0x7f344c0c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:33.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f344c158560 0x7f344c0c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 >> v1:172.21.15.110:6789/0 conn(0x7f344c0720c0 legacy=0x7f344c157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c154e00 con 0x7f344c13dc20 2023-10-25T00:03:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f3448001680 con 0x7f344c13dc20 2023-10-25T00:03:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 >> v1:172.21.15.107:6789/0 conn(0x7f344c13dc20 legacy=0x7f344c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34380039b0 0x7f3438003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:33.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380047d0 0x7f3438004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:33.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34380052b0 0x7f343800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:33.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c154b20 con 0x7f34380039b0 2023-10-25T00:03:33.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343800bf80 con 0x7f34380047d0 2023-10-25T00:03:33.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343800c140 con 0x7f34380052b0 2023-10-25T00:03:33.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f34513a0640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34380039b0 0x7f3438003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:33.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3451ba1640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34380052b0 0x7f343800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:33.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f34513a0640 1 -- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34380052b0 msgr2=0x7f343800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:33.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3450b9f640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380047d0 0x7f3438004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:33.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f34513a0640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34380052b0 0x7f343800b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:33.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f34513a0640 1 -- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380047d0 msgr2=0x7f3438004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:33.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f34513a0640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380047d0 0x7f3438004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:33.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f34513a0640 1 -- 172.21.15.107:0/2032449064 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3438003530 con 0x7f34380039b0 2023-10-25T00:03:33.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.695+0000 7f3450b9f640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380047d0 0x7f3438004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f34513a0640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34380039b0 0x7f3438003df0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f344800c240 tx=0x7f344801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ce468219b73f73 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448005ae0 con 0x7f34380039b0 2023-10-25T00:03:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3448005c60 con 0x7f34380039b0 2023-10-25T00:03:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3443fff640 1 -- 172.21.15.107:0/2032449064 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448007070 con 0x7f34380039b0 2023-10-25T00:03:33.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3453604640 1 -- 172.21.15.107:0/2032449064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34380039b0 msgr2=0x7f3438003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:33.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3453604640 1 --2- 172.21.15.107:0/2032449064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34380039b0 0x7f3438003df0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f344800c240 tx=0x7f344801cc40 comp rx=0 tx=0).stop 2023-10-25T00:03:33.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3451ba1640 1 -- 172.21.15.107:0/2032449064 reap_dead start 2023-10-25T00:03:33.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3453604640 1 -- 172.21.15.107:0/2032449064 shutdown_connections 2023-10-25T00:03:33.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3453604640 1 -- 172.21.15.107:0/2032449064 >> 172.21.15.107:0/2032449064 conn(0x7f344c0c39a0 msgr2=0x7f344c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:33.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3453604640 1 -- 172.21.15.107:0/2032449064 shutdown_connections 2023-10-25T00:03:33.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3453604640 1 -- 172.21.15.107:0/2032449064 wait complete. 2023-10-25T00:03:33.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3453604640 1 Processor -- start 2023-10-25T00:03:33.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.699+0000 7f3453604640 1 -- start start 2023-10-25T00:03:33.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3453604640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380039b0 0x7f344c0aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:33.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3453604640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f344c06bda0 0x7f344c134100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3453604640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f344c1536d0 0x7f344c153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3453604640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0c50a0 con 0x7f344c06bda0 2023-10-25T00:03:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3453604640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0a9580 con 0x7f34380039b0 2023-10-25T00:03:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3453604640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0c4dc0 con 0x7f344c1536d0 2023-10-25T00:03:33.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3450b9f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f344c06bda0 0x7f344c134100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3450b9f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f344c06bda0 0x7f344c134100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44584/0 (socket says 172.21.15.107:44584) 2023-10-25T00:03:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3450b9f640 1 -- 172.21.15.107:0/55500696 learned_addr learned my addr 172.21.15.107:0/55500696 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f34513a0640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380039b0 0x7f344c0aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3451ba1640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f344c1536d0 0x7f344c153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:33.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3450b9f640 1 -- 172.21.15.107:0/55500696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f344c1536d0 msgr2=0x7f344c153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:33.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3450b9f640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f344c1536d0 0x7f344c153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:33.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3450b9f640 1 -- 172.21.15.107:0/55500696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380039b0 msgr2=0x7f344c0aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:33.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3450b9f640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380039b0 0x7f344c0aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:33.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3450b9f640 1 -- 172.21.15.107:0/55500696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c1f0540 con 0x7f344c06bda0 2023-10-25T00:03:33.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3450b9f640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f344c06bda0 0x7f344c134100 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f343c00d770 tx=0x7f343c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3441ffb640 1 -- 172.21.15.107:0/55500696 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c005270 con 0x7f344c06bda0 2023-10-25T00:03:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3441ffb640 1 -- 172.21.15.107:0/55500696 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343c0053f0 con 0x7f344c06bda0 2023-10-25T00:03:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3453604640 1 -- 172.21.15.107:0/55500696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344c0aaa30 con 0x7f344c06bda0 2023-10-25T00:03:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3441ffb640 1 -- 172.21.15.107:0/55500696 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c00b870 con 0x7f344c06bda0 2023-10-25T00:03:33.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3453604640 1 -- 172.21.15.107:0/55500696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344c134640 con 0x7f344c06bda0 2023-10-25T00:03:33.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3453604640 1 -- 172.21.15.107:0/55500696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344c1f03a0 con 0x7f344c06bda0 2023-10-25T00:03:33.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3441ffb640 1 -- 172.21.15.107:0/55500696 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f343c00b9d0 con 0x7f344c06bda0 2023-10-25T00:03:33.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3441ffb640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3430065e60 0x7f3430068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:33.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f3441ffb640 1 -- 172.21.15.107:0/55500696 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7f343c083f00 con 0x7f344c06bda0 2023-10-25T00:03:33.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.703+0000 7f34513a0640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3430065e60 0x7f3430068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.707+0000 7f34513a0640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3430065e60 0x7f3430068320 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f34480ab000 tx=0x7f34480a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:33.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.711+0000 7f3441ffb640 1 -- 172.21.15.107:0/55500696 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f344c1f03a0 con 0x7f344c06bda0 2023-10-25T00:03:33.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 -- 172.21.15.107:0/55500696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f344c1f03a0 con 0x7f344c06bda0 2023-10-25T00:03:33.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3441ffb640 1 -- 172.21.15.107:0/55500696 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5013 (secure 0 0 0) 0x7f344c1f03a0 con 0x7f344c06bda0 2023-10-25T00:03:33.943 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:33.943 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 89 2023-10-25T00:03:33.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 -- 172.21.15.107:0/55500696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3430065e60 msgr2=0x7f3430068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:33.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3430065e60 0x7f3430068320 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f34480ab000 tx=0x7f34480a7070 comp rx=0 tx=0).stop 2023-10-25T00:03:33.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 -- 172.21.15.107:0/55500696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f344c06bda0 msgr2=0x7f344c134100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:33.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f344c06bda0 0x7f344c134100 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f343c00d770 tx=0x7f343c00dc30 comp rx=0 tx=0).stop 2023-10-25T00:03:33.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 -- 172.21.15.107:0/55500696 shutdown_connections 2023-10-25T00:03:33.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3430065e60 0x7f3430068320 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:33.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f344c1536d0 0x7f344c153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:33.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f344c06bda0 0x7f344c134100 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:33.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 --2- 172.21.15.107:0/55500696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380039b0 0x7f344c0aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:33.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 -- 172.21.15.107:0/55500696 >> 172.21.15.107:0/55500696 conn(0x7f344c0c39a0 msgr2=0x7f344c06cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:33.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 -- 172.21.15.107:0/55500696 shutdown_connections 2023-10-25T00:03:33.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:33.943+0000 7f3453604640 1 -- 172.21.15.107:0/55500696 wait complete. 2023-10-25T00:03:33.965 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=6396 2023-10-25T00:03:33.965 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.c 2023-10-25T00:03:33.965 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2023-10-25T00:03:33.966 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:03:33.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:03:33.968 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:03:33.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:34.038 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:03:34.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdad1376640 1 Processor -- start 2023-10-25T00:03:34.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdad1376640 1 -- start start 2023-10-25T00:03:34.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdad1376640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdacc070b00 0x7fdacc0c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:34.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdad1376640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdacc06df30 0x7fdacc06e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:34.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdad1376640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdacc0c22f0 con 0x7fdacc14eb70 2023-10-25T00:03:34.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdad1376640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdacc071700 con 0x7fdacc06df30 2023-10-25T00:03:34.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdad1376640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc0c25d0 con 0x7fdacc070b00 2023-10-25T00:03:34.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdacb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdacc06df30 0x7fdacc06e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdacb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdacc06df30 0x7fdacc06e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44624/0 (socket says 172.21.15.107:44624) 2023-10-25T00:03:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdacb7fe640 1 -- 172.21.15.107:0/2313440535 learned_addr learned my addr 172.21.15.107:0/2313440535 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdac3fff640 1 --2- 172.21.15.107:0/2313440535 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdacc070b00 0x7fdacc0c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdacb7fe640 1 -- 172.21.15.107:0/2313440535 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdacc070b00 msgr2=0x7fdacc0c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:34.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdacb7fe640 1 --2- 172.21.15.107:0/2313440535 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdacc070b00 0x7fdacc0c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:34.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdacb7fe640 1 -- 172.21.15.107:0/2313440535 >> v1:172.21.15.110:6789/0 conn(0x7fdacc14eb70 legacy=0x7fdacc1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:34.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.099+0000 7fdacb7fe640 1 -- 172.21.15.107:0/2313440535 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc0c2be0 con 0x7fdacc06df30 2023-10-25T00:03:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdacb7fe640 1 --2- 172.21.15.107:0/2313440535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc06df30 0x7fdacc06e360 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009e50 tx=0x7fdac401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7315e63d79b760 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdaca7fc640 1 -- 172.21.15.107:0/2313440535 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac40a5870 con 0x7fdacc06df30 2023-10-25T00:03:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdaca7fc640 1 -- 172.21.15.107:0/2313440535 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac40a59d0 con 0x7fdacc06df30 2023-10-25T00:03:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdaca7fc640 1 -- 172.21.15.107:0/2313440535 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac40a5d60 con 0x7fdacc06df30 2023-10-25T00:03:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdad1376640 1 -- 172.21.15.107:0/2313440535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc06df30 msgr2=0x7fdacc06e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdad1376640 1 --2- 172.21.15.107:0/2313440535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc06df30 0x7fdacc06e360 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009e50 tx=0x7fdac401c9d0 comp rx=0 tx=0).stop 2023-10-25T00:03:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdad1376640 1 -- 172.21.15.107:0/2313440535 shutdown_connections 2023-10-25T00:03:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdad1376640 1 --2- 172.21.15.107:0/2313440535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc06df30 0x7fdacc06e360 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdad1376640 1 --2- 172.21.15.107:0/2313440535 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdacc070b00 0x7fdacc0c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdad1376640 1 -- 172.21.15.107:0/2313440535 >> 172.21.15.107:0/2313440535 conn(0x7fdacc0c39a0 msgr2=0x7fdacc0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdad1376640 1 -- 172.21.15.107:0/2313440535 shutdown_connections 2023-10-25T00:03:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.103+0000 7fdad1376640 1 -- 172.21.15.107:0/2313440535 wait complete. 2023-10-25T00:03:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdad1376640 1 Processor -- start 2023-10-25T00:03:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdad1376640 1 -- start start 2023-10-25T00:03:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdad1376640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdacc06df30 0x7fdacc1ecb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdad1376640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc070b00 0x7fdacc15b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdad1376640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdacc14eb70 0x7fdacc14fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdad1376640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc069d40 con 0x7fdacc070b00 2023-10-25T00:03:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdad1376640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc14f6a0 con 0x7fdacc06df30 2023-10-25T00:03:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdad1376640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc06a830 con 0x7fdacc14eb70 2023-10-25T00:03:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc070b00 0x7fdacc15b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc070b00 0x7fdacc15b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44638/0 (socket says 172.21.15.107:44638) 2023-10-25T00:03:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac3fff640 1 -- 172.21.15.107:0/2827193155 learned_addr learned my addr 172.21.15.107:0/2827193155 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdacb7fe640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdacc14eb70 0x7fdacc14fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdacaffd640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdacc06df30 0x7fdacc1ecb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac3fff640 1 -- 172.21.15.107:0/2827193155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdacc14eb70 msgr2=0x7fdacc14fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac3fff640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdacc14eb70 0x7fdacc14fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac3fff640 1 -- 172.21.15.107:0/2827193155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdacc06df30 msgr2=0x7fdacc1ecb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:34.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac3fff640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdacc06df30 0x7fdacc1ecb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:34.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac3fff640 1 -- 172.21.15.107:0/2827193155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac4009b30 con 0x7fdacc070b00 2023-10-25T00:03:34.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdacaffd640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdacc06df30 0x7fdacc1ecb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:03:34.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac3fff640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc070b00 0x7fdacc15b610 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fdab400d800 tx=0x7fdab400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:34.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac37fe640 1 -- 172.21.15.107:0/2827193155 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab400b6c0 con 0x7fdacc070b00 2023-10-25T00:03:34.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdad1376640 1 -- 172.21.15.107:0/2827193155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdacc069f90 con 0x7fdacc070b00 2023-10-25T00:03:34.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac37fe640 1 -- 172.21.15.107:0/2827193155 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab400b820 con 0x7fdacc070b00 2023-10-25T00:03:34.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdac37fe640 1 -- 172.21.15.107:0/2827193155 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab400bbb0 con 0x7fdacc070b00 2023-10-25T00:03:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdad1376640 1 -- 172.21.15.107:0/2827193155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdacc06a550 con 0x7fdacc070b00 2023-10-25T00:03:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.107+0000 7fdad1376640 1 -- 172.21.15.107:0/2827193155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdacc0c6280 con 0x7fdacc070b00 2023-10-25T00:03:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.111+0000 7fdac37fe640 1 -- 172.21.15.107:0/2827193155 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdab400bd10 con 0x7fdacc070b00 2023-10-25T00:03:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.111+0000 7fdac37fe640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdaac065e60 0x7fdaac068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:34.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.111+0000 7fdac37fe640 1 -- 172.21.15.107:0/2827193155 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7fdab4084620 con 0x7fdacc070b00 2023-10-25T00:03:34.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.111+0000 7fdacaffd640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdaac065e60 0x7fdaac068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:34.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.111+0000 7fdacaffd640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdaac065e60 0x7fdaac068320 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0098b0 tx=0x7fdabc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:34.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.115+0000 7fdac37fe640 1 -- 172.21.15.107:0/2827193155 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdab4051330 con 0x7fdacc070b00 2023-10-25T00:03:34.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 -- 172.21.15.107:0/2827193155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdacc0c1410 con 0x7fdacc070b00 2023-10-25T00:03:34.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdac37fe640 1 -- 172.21.15.107:0/2827193155 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5013 (secure 0 0 0) 0x7fdab4054950 con 0x7fdacc070b00 2023-10-25T00:03:34.336 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:34.336 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:34.336 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 89 2023-10-25T00:03:34.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 -- 172.21.15.107:0/2827193155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdaac065e60 msgr2=0x7fdaac068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdaac065e60 0x7fdaac068320 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdabc0098b0 tx=0x7fdabc008040 comp rx=0 tx=0).stop 2023-10-25T00:03:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 -- 172.21.15.107:0/2827193155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc070b00 msgr2=0x7fdacc15b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc070b00 0x7fdacc15b610 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fdab400d800 tx=0x7fdab400dcc0 comp rx=0 tx=0).stop 2023-10-25T00:03:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 -- 172.21.15.107:0/2827193155 shutdown_connections 2023-10-25T00:03:34.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdaac065e60 0x7fdaac068320 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdacc14eb70 0x7fdacc14fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc070b00 0x7fdacc15b610 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 --2- 172.21.15.107:0/2827193155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdacc06df30 0x7fdacc1ecb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 -- 172.21.15.107:0/2827193155 >> 172.21.15.107:0/2827193155 conn(0x7fdacc0c39a0 msgr2=0x7fdacc0c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 -- 172.21.15.107:0/2827193155 shutdown_connections 2023-10-25T00:03:34.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:34.335+0000 7fdad1376640 1 -- 172.21.15.107:0/2827193155 wait complete. 2023-10-25T00:03:34.359 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-25T00:03:34.556 INFO:tasks.ceph.mds.b.smithi110.stderr:2023-10-25T00:03:34.557+0000 7f71e9cd1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:03:36.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:36.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.495+0000 7fdb4d2ba640 1 Processor -- start 2023-10-25T00:03:36.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.495+0000 7fdb4d2ba640 1 -- start start 2023-10-25T00:03:36.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb4d2ba640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdb480c8100 0x7fdb480abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:36.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb4d2ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb4806a4b0 0x7fdb480ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:36.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb4d2ba640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb48155680 con 0x7fdb48152040 2023-10-25T00:03:36.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb4d2ba640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb48154a70 con 0x7fdb480c8100 2023-10-25T00:03:36.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb4d2ba640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48154d50 con 0x7fdb4806a4b0 2023-10-25T00:03:36.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb46ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdb48152040 0x7fdb480c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59040/0 (socket says 172.21.15.107:59040) 2023-10-25T00:03:36.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb46ffd640 1 -- 172.21.15.107:0/3910969634 learned_addr learned my addr 172.21.15.107:0/3910969634 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:36.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb3ffff640 1 --2- 172.21.15.107:0/3910969634 >> v2:172.21.15.110:3300/0 conn(0x7fdb480c8100 0x7fdb480abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:36.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb477fe640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb4806a4b0 0x7fdb480ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:36.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883455794 0 0) 0x7fdb48155680 con 0x7fdb48152040 2023-10-25T00:03:36.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb34003530 con 0x7fdb48152040 2023-10-25T00:03:36.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2046227001 0 0) 0x7fdb34003530 con 0x7fdb48152040 2023-10-25T00:03:36.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb4806a4b0 msgr2=0x7fdb480ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:36.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb4806a4b0 0x7fdb480ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:36.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 >> v2:172.21.15.110:3300/0 conn(0x7fdb480c8100 msgr2=0x7fdb480abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:36.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 --2- 172.21.15.107:0/3910969634 >> v2:172.21.15.110:3300/0 conn(0x7fdb480c8100 0x7fdb480abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb48155030 con 0x7fdb48152040 2023-10-25T00:03:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fdb38001680 con 0x7fdb48152040 2023-10-25T00:03:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 >> v1:172.21.15.107:6789/0 conn(0x7fdb48152040 legacy=0x7fdb480c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb4806a4b0 0x7fdb34003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb34003f00 0x7fdb340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:36.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 0x7fdb34005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48154a70 con 0x7fdb34004d10 2023-10-25T00:03:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48154d50 con 0x7fdb34003f00 2023-10-25T00:03:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb3400b9e0 con 0x7fdb4806a4b0 2023-10-25T00:03:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb477fe640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 0x7fdb34005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:36.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb46ffd640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb4806a4b0 0x7fdb34003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb3ffff640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb34003f00 0x7fdb340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb477fe640 1 -- 172.21.15.107:0/3910969634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb4806a4b0 msgr2=0x7fdb34003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb477fe640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb4806a4b0 0x7fdb34003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb477fe640 1 -- 172.21.15.107:0/3910969634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb34003f00 msgr2=0x7fdb340047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:36.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb477fe640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb34003f00 0x7fdb340047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:36.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.499+0000 7fdb477fe640 1 -- 172.21.15.107:0/3910969634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb34003530 con 0x7fdb34004d10 2023-10-25T00:03:36.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb477fe640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 0x7fdb34005190 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000e170 tx=0x7fdb4001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdfd0d5908fdca82 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:36.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb400a5b40 con 0x7fdb34004d10 2023-10-25T00:03:36.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4000f040 con 0x7fdb34004d10 2023-10-25T00:03:36.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb467fc640 1 -- 172.21.15.107:0/3910969634 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb400ad440 con 0x7fdb34004d10 2023-10-25T00:03:36.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/3910969634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 msgr2=0x7fdb34005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:36.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 --2- 172.21.15.107:0/3910969634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 0x7fdb34005190 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000e170 tx=0x7fdb4001cc40 comp rx=0 tx=0).stop 2023-10-25T00:03:36.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb477fe640 1 -- 172.21.15.107:0/3910969634 reap_dead start 2023-10-25T00:03:36.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/3910969634 shutdown_connections 2023-10-25T00:03:36.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/3910969634 >> 172.21.15.107:0/3910969634 conn(0x7fdb480ca200 msgr2=0x7fdb480ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:36.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/3910969634 shutdown_connections 2023-10-25T00:03:36.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/3910969634 wait complete. 2023-10-25T00:03:36.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 Processor -- start 2023-10-25T00:03:36.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 -- start start 2023-10-25T00:03:36.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 0x7fdb481ed590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:36.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb480c37a0 0x7fdb481e9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:36.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb480bd200 0x7fdb48150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:36.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb481edad0 con 0x7fdb34004d10 2023-10-25T00:03:36.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480c12e0 con 0x7fdb480c37a0 2023-10-25T00:03:36.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb4d2ba640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480c15c0 con 0x7fdb480bd200 2023-10-25T00:03:36.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb46ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 0x7fdb481ed590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:36.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb46ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 0x7fdb481ed590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44650/0 (socket says 172.21.15.107:44650) 2023-10-25T00:03:36.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb46ffd640 1 -- 172.21.15.107:0/579521671 learned_addr learned my addr 172.21.15.107:0/579521671 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:36.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.503+0000 7fdb3ffff640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb480c37a0 0x7fdb481e9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:36.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb477fe640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb480bd200 0x7fdb48150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:36.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb46ffd640 1 -- 172.21.15.107:0/579521671 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb480bd200 msgr2=0x7fdb48150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:36.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb46ffd640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb480bd200 0x7fdb48150cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:36.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb46ffd640 1 -- 172.21.15.107:0/579521671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb480c37a0 msgr2=0x7fdb481e9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:36.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb46ffd640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb480c37a0 0x7fdb481e9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:36.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb46ffd640 1 -- 172.21.15.107:0/579521671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb481ecd90 con 0x7fdb34004d10 2023-10-25T00:03:36.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb477fe640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb480bd200 0x7fdb48150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:03:36.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb46ffd640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 0x7fdb481ed590 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fdb38011bb0 tx=0x7fdb3800f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:36.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb3f7fe640 1 -- 172.21.15.107:0/579521671 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3800fdf0 con 0x7fdb34004d10 2023-10-25T00:03:36.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb3f7fe640 1 -- 172.21.15.107:0/579521671 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb38015410 con 0x7fdb34004d10 2023-10-25T00:03:36.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/579521671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb481ed070 con 0x7fdb34004d10 2023-10-25T00:03:36.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb3f7fe640 1 -- 172.21.15.107:0/579521671 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb380157c0 con 0x7fdb34004d10 2023-10-25T00:03:36.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/579521671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4814cd10 con 0x7fdb34004d10 2023-10-25T00:03:36.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/579521671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb481ecbf0 con 0x7fdb34004d10 2023-10-25T00:03:36.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb3f7fe640 1 -- 172.21.15.107:0/579521671 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdb38015920 con 0x7fdb34004d10 2023-10-25T00:03:36.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb3f7fe640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb28065e90 0x7fdb28068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:36.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb3f7fe640 1 -- 172.21.15.107:0/579521671 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7fdb380875e0 con 0x7fdb34004d10 2023-10-25T00:03:36.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb3ffff640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb28065e90 0x7fdb28068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:36.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.507+0000 7fdb3ffff640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb28065e90 0x7fdb28068350 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdb481ea7b0 tx=0x7fdb30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:36.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.511+0000 7fdb3f7fe640 1 -- 172.21.15.107:0/579521671 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdb481ecbf0 con 0x7fdb34004d10 2023-10-25T00:03:36.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/579521671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb481ecbf0 con 0x7fdb34004d10 2023-10-25T00:03:36.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb3f7fe640 1 -- 172.21.15.107:0/579521671 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fdb481ecbf0 con 0x7fdb34004d10 2023-10-25T00:03:36.733 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:36.733 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6672,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:36.734 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 90 2023-10-25T00:03:36.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/579521671 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb28065e90 msgr2=0x7fdb28068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:36.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb4d2ba640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb28065e90 0x7fdb28068350 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdb481ea7b0 tx=0x7fdb30008040 comp rx=0 tx=0).stop 2023-10-25T00:03:36.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/579521671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 msgr2=0x7fdb481ed590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:36.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb4d2ba640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 0x7fdb481ed590 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fdb38011bb0 tx=0x7fdb3800f540 comp rx=0 tx=0).stop 2023-10-25T00:03:36.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/579521671 shutdown_connections 2023-10-25T00:03:36.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb4d2ba640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb28065e90 0x7fdb28068350 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:36.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb4d2ba640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdb480bd200 0x7fdb48150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:36.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb4d2ba640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb480c37a0 0x7fdb481e9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:36.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb4d2ba640 1 --2- 172.21.15.107:0/579521671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdb34004d10 0x7fdb481ed590 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.731+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/579521671 >> 172.21.15.107:0/579521671 conn(0x7fdb480ca200 msgr2=0x7fdb480c4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.735+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/579521671 shutdown_connections 2023-10-25T00:03:36.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:36.735+0000 7fdb4d2ba640 1 -- 172.21.15.107:0/579521671 wait complete. 2023-10-25T00:03:36.751 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2023-10-25T00:03:36.752 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2023-10-25T00:03:40.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:41.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.043+0000 7fb58af6e640 1 Processor -- start 2023-10-25T00:03:41.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.043+0000 7fb58af6e640 1 -- start start 2023-10-25T00:03:41.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.043+0000 7fb58af6e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb584071ce0 0x7fb5840ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.043+0000 7fb58af6e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5840720c0 0x7fb58413d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.043+0000 7fb58af6e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb584154860 0x7fb584158230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.043+0000 7fb58af6e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb58406d900 con 0x7fb584154860 2023-10-25T00:03:41.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.043+0000 7fb58af6e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb58406e710 con 0x7fb584071ce0 2023-10-25T00:03:41.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.043+0000 7fb58af6e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb584154d20 con 0x7fb5840720c0 2023-10-25T00:03:41.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.043+0000 7fb58950b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb584154860 0x7fb584158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.043+0000 7fb58950b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb584154860 0x7fb584158230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44664/0 (socket says 172.21.15.107:44664) 2023-10-25T00:03:41.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.043+0000 7fb58950b640 1 -- 172.21.15.107:0/3001498950 learned_addr learned my addr 172.21.15.107:0/3001498950 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:41.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb588d0a640 1 --2- 172.21.15.107:0/3001498950 >> v2:172.21.15.110:3300/0 conn(0x7fb584071ce0 0x7fb5840ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb583fff640 1 --2- 172.21.15.107:0/3001498950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5840720c0 0x7fb58413d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58950b640 1 -- 172.21.15.107:0/3001498950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5840720c0 msgr2=0x7fb58413d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:41.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58950b640 1 --2- 172.21.15.107:0/3001498950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5840720c0 0x7fb58413d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58950b640 1 -- 172.21.15.107:0/3001498950 >> v2:172.21.15.110:3300/0 conn(0x7fb584071ce0 msgr2=0x7fb5840ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:41.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58950b640 1 --2- 172.21.15.107:0/3001498950 >> v2:172.21.15.110:3300/0 conn(0x7fb584071ce0 0x7fb5840ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58950b640 1 -- 172.21.15.107:0/3001498950 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb584155000 con 0x7fb584154860 2023-10-25T00:03:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58950b640 1 --2- 172.21.15.107:0/3001498950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb584154860 0x7fb584158230 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb584067460 tx=0x7fb57c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e2837c5ac4bd7 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:41.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb5837fe640 1 -- 172.21.15.107:0/3001498950 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c0a5790 con 0x7fb584154860 2023-10-25T00:03:41.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb5837fe640 1 -- 172.21.15.107:0/3001498950 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c0a58f0 con 0x7fb584154860 2023-10-25T00:03:41.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb5837fe640 1 -- 172.21.15.107:0/3001498950 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c0a5c80 con 0x7fb584154860 2023-10-25T00:03:41.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58af6e640 1 -- 172.21.15.107:0/3001498950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb584154860 msgr2=0x7fb584158230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:41.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58af6e640 1 --2- 172.21.15.107:0/3001498950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb584154860 0x7fb584158230 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb584067460 tx=0x7fb57c01c910 comp rx=0 tx=0).stop 2023-10-25T00:03:41.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58af6e640 1 -- 172.21.15.107:0/3001498950 shutdown_connections 2023-10-25T00:03:41.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58af6e640 1 --2- 172.21.15.107:0/3001498950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb584154860 0x7fb584158230 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58af6e640 1 --2- 172.21.15.107:0/3001498950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5840720c0 0x7fb58413d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58af6e640 1 --2- 172.21.15.107:0/3001498950 >> v2:172.21.15.110:3300/0 conn(0x7fb584071ce0 0x7fb5840ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58af6e640 1 -- 172.21.15.107:0/3001498950 >> 172.21.15.107:0/3001498950 conn(0x7fb584070f60 msgr2=0x7fb584071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:41.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58af6e640 1 -- 172.21.15.107:0/3001498950 shutdown_connections 2023-10-25T00:03:41.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.047+0000 7fb58af6e640 1 -- 172.21.15.107:0/3001498950 wait complete. 2023-10-25T00:03:41.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58af6e640 1 Processor -- start 2023-10-25T00:03:41.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58af6e640 1 -- start start 2023-10-25T00:03:41.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58af6e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb584071ce0 0x7fb5840c1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58af6e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5840720c0 0x7fb5841514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58af6e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb584154860 0x7fb5840bdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58af6e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5841f0fa0 con 0x7fb584154860 2023-10-25T00:03:41.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58af6e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5840c1b30 con 0x7fb584071ce0 2023-10-25T00:03:41.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58af6e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb58413c9f0 con 0x7fb5840720c0 2023-10-25T00:03:41.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58950b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb584154860 0x7fb5840bdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58950b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb584154860 0x7fb5840bdb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44670/0 (socket says 172.21.15.107:44670) 2023-10-25T00:03:41.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58950b640 1 -- 172.21.15.107:0/1918919824 learned_addr learned my addr 172.21.15.107:0/1918919824 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:41.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb588d0a640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb584071ce0 0x7fb5840c1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb583fff640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5840720c0 0x7fb5841514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58950b640 1 -- 172.21.15.107:0/1918919824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5840720c0 msgr2=0x7fb5841514c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:41.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58950b640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5840720c0 0x7fb5841514c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58950b640 1 -- 172.21.15.107:0/1918919824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb584071ce0 msgr2=0x7fb5840c1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:41.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58950b640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb584071ce0 0x7fb5840c1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58950b640 1 -- 172.21.15.107:0/1918919824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c009b30 con 0x7fb584154860 2023-10-25T00:03:41.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb588d0a640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb584071ce0 0x7fb5840c1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:03:41.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58950b640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb584154860 0x7fb5840bdb60 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0af040 tx=0x7fb57c0196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:41.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb5817fa640 1 -- 172.21.15.107:0/1918919824 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c0b8670 con 0x7fb584154860 2023-10-25T00:03:41.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb5817fa640 1 -- 172.21.15.107:0/1918919824 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c0b87d0 con 0x7fb584154860 2023-10-25T00:03:41.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58af6e640 1 -- 172.21.15.107:0/1918919824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb584151a00 con 0x7fb584154860 2023-10-25T00:03:41.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb5817fa640 1 -- 172.21.15.107:0/1918919824 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c0b8b60 con 0x7fb584154860 2023-10-25T00:03:41.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.051+0000 7fb58af6e640 1 -- 172.21.15.107:0/1918919824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5841f0bc0 con 0x7fb584154860 2023-10-25T00:03:41.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.055+0000 7fb58af6e640 1 -- 172.21.15.107:0/1918919824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb584153af0 con 0x7fb584154860 2023-10-25T00:03:41.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.055+0000 7fb5817fa640 1 -- 172.21.15.107:0/1918919824 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb57c0b8cc0 con 0x7fb584154860 2023-10-25T00:03:41.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.055+0000 7fb5817fa640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb568065e60 0x7fb568068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.055+0000 7fb5817fa640 1 -- 172.21.15.107:0/1918919824 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7fb57c11f7d0 con 0x7fb584154860 2023-10-25T00:03:41.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.055+0000 7fb588d0a640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb568065e60 0x7fb568068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.055+0000 7fb588d0a640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb568065e60 0x7fb568068320 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb57800ba50 tx=0x7fb578008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:41.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.059+0000 7fb5817fa640 1 -- 172.21.15.107:0/1918919824 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb57c0ec6b0 con 0x7fb584154860 2023-10-25T00:03:41.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.311+0000 7fb58af6e640 1 -- 172.21.15.107:0/1918919824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5840746e0 con 0x7fb584154860 2023-10-25T00:03:41.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.311+0000 7fb5817fa640 1 -- 172.21.15.107:0/1918919824 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fb57c0bc090 con 0x7fb584154860 2023-10-25T00:03:41.313 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:41.314 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6672,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:41.314 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 90 2023-10-25T00:03:41.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 -- 172.21.15.107:0/1918919824 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb568065e60 msgr2=0x7fb568068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:41.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb568065e60 0x7fb568068320 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb57800ba50 tx=0x7fb578008040 comp rx=0 tx=0).stop 2023-10-25T00:03:41.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 -- 172.21.15.107:0/1918919824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb584154860 msgr2=0x7fb5840bdb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:41.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb584154860 0x7fb5840bdb60 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0af040 tx=0x7fb57c0196b0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 -- 172.21.15.107:0/1918919824 shutdown_connections 2023-10-25T00:03:41.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb584154860 0x7fb5840bdb60 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb568065e60 0x7fb568068320 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5840720c0 0x7fb5841514c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 --2- 172.21.15.107:0/1918919824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb584071ce0 0x7fb5840c1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 -- 172.21.15.107:0/1918919824 >> 172.21.15.107:0/1918919824 conn(0x7fb584070f60 msgr2=0x7fb584153050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:41.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 -- 172.21.15.107:0/1918919824 shutdown_connections 2023-10-25T00:03:41.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.315+0000 7fb58af6e640 1 -- 172.21.15.107:0/1918919824 wait complete. 2023-10-25T00:03:41.334 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2023-10-25T00:03:41.334 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:03:41.335 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:03:41.346 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:03:41.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:41.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0fad15640 1 Processor -- start 2023-10-25T00:03:41.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0fad15640 1 -- start start 2023-10-25T00:03:41.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0fad15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0f414fbd0 0x7fa0f406df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0fad15640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa0f40c4e20 0x7fa0f40c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0fad15640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f4155f20 con 0x7fa0f40c5ba0 2023-10-25T00:03:41.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0fad15640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f40c6d40 con 0x7fa0f40c4e20 2023-10-25T00:03:41.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0fad15640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4153560 con 0x7fa0f414fbd0 2023-10-25T00:03:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0f3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa0f40c4e20 0x7fa0f40c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0f92b2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa0f40c5ba0 0x7fa0f40c1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59052/0 (socket says 172.21.15.107:59052) 2023-10-25T00:03:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0f92b2640 1 -- 172.21.15.107:0/1921801881 learned_addr learned my addr 172.21.15.107:0/1921801881 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0f8ab1640 1 --2- 172.21.15.107:0/1921801881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0f414fbd0 0x7fa0f406df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0f3fff640 1 -- 172.21.15.107:0/1921801881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0f414fbd0 msgr2=0x7fa0f406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:41.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0f3fff640 1 --2- 172.21.15.107:0/1921801881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0f414fbd0 0x7fa0f406df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0f3fff640 1 -- 172.21.15.107:0/1921801881 >> v1:172.21.15.107:6789/0 conn(0x7fa0f40c5ba0 legacy=0x7fa0f40c1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:41.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0f3fff640 1 -- 172.21.15.107:0/1921801881 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f41527a0 con 0x7fa0f40c4e20 2023-10-25T00:03:41.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.503+0000 7fa0f37fe640 1 -- 172.21.15.107:0/1921801881 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389315082 0 0) 0x7fa0f4155f20 con 0x7fa0f40c5ba0 2023-10-25T00:03:41.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0f3fff640 1 --2- 172.21.15.107:0/1921801881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f40c4e20 0x7fa0f40c18a0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00097c0 tx=0x7fa0e001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b458639fdc8a726e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:41.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0f37fe640 1 -- 172.21.15.107:0/1921801881 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e00ab600 con 0x7fa0f40c4e20 2023-10-25T00:03:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0f37fe640 1 -- 172.21.15.107:0/1921801881 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e00a7050 con 0x7fa0f40c4e20 2023-10-25T00:03:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0f37fe640 1 -- 172.21.15.107:0/1921801881 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e00ab980 con 0x7fa0f40c4e20 2023-10-25T00:03:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 -- 172.21.15.107:0/1921801881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f40c4e20 msgr2=0x7fa0f40c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 --2- 172.21.15.107:0/1921801881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f40c4e20 0x7fa0f40c18a0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00097c0 tx=0x7fa0e001c3f0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 -- 172.21.15.107:0/1921801881 shutdown_connections 2023-10-25T00:03:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 --2- 172.21.15.107:0/1921801881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f40c4e20 0x7fa0f40c18a0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 --2- 172.21.15.107:0/1921801881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0f414fbd0 0x7fa0f406df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 -- 172.21.15.107:0/1921801881 >> 172.21.15.107:0/1921801881 conn(0x7fa0f40c39a0 msgr2=0x7fa0f40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 -- 172.21.15.107:0/1921801881 shutdown_connections 2023-10-25T00:03:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 -- 172.21.15.107:0/1921801881 wait complete. 2023-10-25T00:03:41.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 Processor -- start 2023-10-25T00:03:41.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 -- start start 2023-10-25T00:03:41.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f40c4e20 0x7fa0f41eabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0f40c5ba0 0x7fa0f40aaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0f414fbd0 0x7fa0f413f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4154f50 con 0x7fa0f40c4e20 2023-10-25T00:03:41.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4155230 con 0x7fa0f414fbd0 2023-10-25T00:03:41.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.507+0000 7fa0fad15640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4155510 con 0x7fa0f40c5ba0 2023-10-25T00:03:41.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f8ab1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f40c4e20 0x7fa0f41eabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f8ab1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f40c4e20 0x7fa0f41eabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44684/0 (socket says 172.21.15.107:44684) 2023-10-25T00:03:41.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f8ab1640 1 -- 172.21.15.107:0/2526229673 learned_addr learned my addr 172.21.15.107:0/2526229673 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:41.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f92b2640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0f414fbd0 0x7fa0f413f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f3fff640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0f40c5ba0 0x7fa0f40aaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f8ab1640 1 -- 172.21.15.107:0/2526229673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0f40c5ba0 msgr2=0x7fa0f40aaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:41.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f8ab1640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0f40c5ba0 0x7fa0f40aaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f8ab1640 1 -- 172.21.15.107:0/2526229673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0f414fbd0 msgr2=0x7fa0f413f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:41.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f8ab1640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0f414fbd0 0x7fa0f413f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f8ab1640 1 -- 172.21.15.107:0/2526229673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e0009420 con 0x7fa0f40c4e20 2023-10-25T00:03:41.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f92b2640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0f414fbd0 0x7fa0f413f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:41.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f3fff640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0f40c5ba0 0x7fa0f40aaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:41.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f8ab1640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f40c4e20 0x7fa0f41eabf0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fa0e800d5f0 tx=0x7fa0e800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:41.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f17fa640 1 -- 172.21.15.107:0/2526229673 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e80125f0 con 0x7fa0f40c4e20 2023-10-25T00:03:41.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f17fa640 1 -- 172.21.15.107:0/2526229673 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e8012750 con 0x7fa0f40c4e20 2023-10-25T00:03:41.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0fad15640 1 -- 172.21.15.107:0/2526229673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f41557f0 con 0x7fa0f40c4e20 2023-10-25T00:03:41.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f17fa640 1 -- 172.21.15.107:0/2526229673 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8012ae0 con 0x7fa0f40c4e20 2023-10-25T00:03:41.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0fad15640 1 -- 172.21.15.107:0/2526229673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f4147660 con 0x7fa0f40c4e20 2023-10-25T00:03:41.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0fad15640 1 -- 172.21.15.107:0/2526229673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f40746e0 con 0x7fa0f40c4e20 2023-10-25T00:03:41.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f17fa640 1 -- 172.21.15.107:0/2526229673 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa0e8012c40 con 0x7fa0f40c4e20 2023-10-25T00:03:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f17fa640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa0d8065e60 0x7fa0d8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f3fff640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa0d8065e60 0x7fa0d8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f17fa640 1 -- 172.21.15.107:0/2526229673 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7fa0e80845b0 con 0x7fa0f40c4e20 2023-10-25T00:03:41.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.511+0000 7fa0f3fff640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa0d8065e60 0x7fa0d8068320 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00ad000 tx=0x7fa0e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:41.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.519+0000 7fa0f17fa640 1 -- 172.21.15.107:0/2526229673 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa0e8051350 con 0x7fa0f40c4e20 2023-10-25T00:03:41.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.771+0000 7fa0fad15640 1 -- 172.21.15.107:0/2526229673 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0f40c5980 con 0x7fa0f40c4e20 2023-10-25T00:03:41.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.771+0000 7fa0f17fa640 1 -- 172.21.15.107:0/2526229673 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5773 (secure 0 0 0) 0x7fa0e8007380 con 0x7fa0f40c4e20 2023-10-25T00:03:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:41.775 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6672,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:41.775 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 90 2023-10-25T00:03:41.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 -- 172.21.15.107:0/2526229673 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa0d8065e60 msgr2=0x7fa0d8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:41.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa0d8065e60 0x7fa0d8068320 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa0e00ad000 tx=0x7fa0e0008040 comp rx=0 tx=0).stop 2023-10-25T00:03:41.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 -- 172.21.15.107:0/2526229673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f40c4e20 msgr2=0x7fa0f41eabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:41.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f40c4e20 0x7fa0f41eabf0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fa0e800d5f0 tx=0x7fa0e800dab0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 -- 172.21.15.107:0/2526229673 shutdown_connections 2023-10-25T00:03:41.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa0f414fbd0 0x7fa0f413f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa0d8065e60 0x7fa0d8068320 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa0f40c5ba0 0x7fa0f40aaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 --2- 172.21.15.107:0/2526229673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f40c4e20 0x7fa0f41eabf0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:41.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 -- 172.21.15.107:0/2526229673 >> 172.21.15.107:0/2526229673 conn(0x7fa0f40c39a0 msgr2=0x7fa0f406b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:41.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 -- 172.21.15.107:0/2526229673 shutdown_connections 2023-10-25T00:03:41.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:41.775+0000 7fa0fad15640 1 -- 172.21.15.107:0/2526229673 wait complete. 2023-10-25T00:03:41.794 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:03:43.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:43.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.938+0000 7f0588b7b640 1 Processor -- start 2023-10-25T00:03:43.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.938+0000 7f0588b7b640 1 -- start start 2023-10-25T00:03:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0588b7b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f05840c70e0 0x7f05840bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0588b7b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05840c1ca0 con 0x7f0584156770 2023-10-25T00:03:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0588b7b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05840c10c0 con 0x7f05840c2130 2023-10-25T00:03:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0588b7b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05840c27e0 con 0x7f05840c70e0 2023-10-25T00:03:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0584156770 0x7f05840ab7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52394/0 (socket says 172.21.15.107:52394) 2023-10-25T00:03:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058259c640 1 -- 172.21.15.107:0/2145992545 learned_addr learned my addr 172.21.15.107:0/2145992545 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0581d9b640 1 --2- 172.21.15.107:0/2145992545 >> v2:172.21.15.110:3300/0 conn(0x7f05840c70e0 0x7f05840bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544897508 0 0) 0x7f05840c1ca0 con 0x7f0584156770 2023-10-25T00:03:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056c003530 con 0x7f0584156770 2023-10-25T00:03:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2613233108 0 0) 0x7f056c003530 con 0x7f0584156770 2023-10-25T00:03:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 >> v2:172.21.15.110:3300/0 conn(0x7f05840c70e0 msgr2=0x7f05840bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 --2- 172.21.15.107:0/2145992545 >> v2:172.21.15.110:3300/0 conn(0x7f05840c70e0 0x7f05840bdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 >> v1:172.21.15.110:6789/0 conn(0x7f05840c2130 legacy=0x7f058415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05840c13a0 con 0x7f0584156770 2023-10-25T00:03:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f0574001680 con 0x7f0584156770 2023-10-25T00:03:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 >> v1:172.21.15.107:6789/0 conn(0x7f0584156770 legacy=0x7f05840ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 --2- 172.21.15.107:0/2145992545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056c003980 0x7f056c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 --2- 172.21.15.107:0/2145992545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f056c0047d0 0x7f056c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 --2- 172.21.15.107:0/2145992545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 0x7f056c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00c150 con 0x7f056c0052b0 2023-10-25T00:03:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00c2c0 con 0x7f056c003980 2023-10-25T00:03:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f056c00c480 con 0x7f056c0047d0 2023-10-25T00:03:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0582d9d640 1 --2- 172.21.15.107:0/2145992545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 0x7f056c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f058259c640 1 --2- 172.21.15.107:0/2145992545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056c003980 0x7f056c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0581d9b640 1 --2- 172.21.15.107:0/2145992545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f056c0047d0 0x7f056c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0582d9d640 1 -- 172.21.15.107:0/2145992545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f056c0047d0 msgr2=0x7f056c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0582d9d640 1 --2- 172.21.15.107:0/2145992545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f056c0047d0 0x7f056c004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:43.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0582d9d640 1 -- 172.21.15.107:0/2145992545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056c003980 msgr2=0x7f056c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:43.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0582d9d640 1 --2- 172.21.15.107:0/2145992545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f056c003980 0x7f056c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:43.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.942+0000 7f0582d9d640 1 -- 172.21.15.107:0/2145992545 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c003530 con 0x7f056c0052b0 2023-10-25T00:03:43.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0582d9d640 1 -- 172.21.15.107:0/2145992545 reap_dead start 2023-10-25T00:03:43.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0582d9d640 1 --2- 172.21.15.107:0/2145992545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 0x7f056c00b990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f057c001e40 tx=0x7f057c01c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb0899bff72fd40 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:43.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c003b80 con 0x7f056c0052b0 2023-10-25T00:03:43.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057c003ce0 con 0x7f056c0052b0 2023-10-25T00:03:43.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f058159a640 1 -- 172.21.15.107:0/2145992545 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c005310 con 0x7f056c0052b0 2023-10-25T00:03:43.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 -- 172.21.15.107:0/2145992545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 msgr2=0x7f056c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:43.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 --2- 172.21.15.107:0/2145992545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 0x7f056c00b990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f057c001e40 tx=0x7f057c01c710 comp rx=0 tx=0).stop 2023-10-25T00:03:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 -- 172.21.15.107:0/2145992545 shutdown_connections 2023-10-25T00:03:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 --2- 172.21.15.107:0/2145992545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 0x7f056c00b990 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 -- 172.21.15.107:0/2145992545 >> 172.21.15.107:0/2145992545 conn(0x7f05840c65c0 msgr2=0x7f05840c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 -- 172.21.15.107:0/2145992545 shutdown_connections 2023-10-25T00:03:43.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 -- 172.21.15.107:0/2145992545 wait complete. 2023-10-25T00:03:43.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 Processor -- start 2023-10-25T00:03:43.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 -- start start 2023-10-25T00:03:43.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 0x7f05840c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:43.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05840c5000 0x7f05841ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:43.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05840c5480 0x7f05841eea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:43.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05840a7510 con 0x7f056c0052b0 2023-10-25T00:03:43.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05840a7000 con 0x7f05840c5000 2023-10-25T00:03:43.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0588b7b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05840b8730 con 0x7f05840c5480 2023-10-25T00:03:43.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f058259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 0x7f05840c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f058259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 0x7f05840c5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40114/0 (socket says 172.21.15.107:40114) 2023-10-25T00:03:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f058259c640 1 -- 172.21.15.107:0/1845187560 learned_addr learned my addr 172.21.15.107:0/1845187560 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0582d9d640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05840c5480 0x7f05841eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f0581d9b640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05840c5000 0x7f05841ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:43.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f058259c640 1 -- 172.21.15.107:0/1845187560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05840c5480 msgr2=0x7f05841eea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:43.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.946+0000 7f058259c640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05840c5480 0x7f05841eea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:43.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f058259c640 1 -- 172.21.15.107:0/1845187560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05840c5000 msgr2=0x7f05841ee4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:43.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f058259c640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05840c5000 0x7f05841ee4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:43.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f058259c640 1 -- 172.21.15.107:0/1845187560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05840a7230 con 0x7f056c0052b0 2023-10-25T00:03:43.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0581d9b640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05840c5000 0x7f05841ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:03:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f058259c640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 0x7f05840c5b80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f057400e780 tx=0x7f057400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0572ffd640 1 -- 172.21.15.107:0/1845187560 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057400c6c0 con 0x7f056c0052b0 2023-10-25T00:03:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0572ffd640 1 -- 172.21.15.107:0/1845187560 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057400c840 con 0x7f056c0052b0 2023-10-25T00:03:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0588b7b640 1 -- 172.21.15.107:0/1845187560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05840c43c0 con 0x7f056c0052b0 2023-10-25T00:03:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0572ffd640 1 -- 172.21.15.107:0/1845187560 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057400cbf0 con 0x7f056c0052b0 2023-10-25T00:03:43.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0588b7b640 1 -- 172.21.15.107:0/1845187560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05840c60c0 con 0x7f056c0052b0 2023-10-25T00:03:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0588b7b640 1 -- 172.21.15.107:0/1845187560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05840746e0 con 0x7f056c0052b0 2023-10-25T00:03:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0572ffd640 1 -- 172.21.15.107:0/1845187560 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f057400cd50 con 0x7f056c0052b0 2023-10-25T00:03:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0572ffd640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e90 0x7f0564068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0581d9b640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e90 0x7f0564068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:43.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0572ffd640 1 -- 172.21.15.107:0/1845187560 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7f0574083cc0 con 0x7f056c0052b0 2023-10-25T00:03:43.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.950+0000 7f0581d9b640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e90 0x7f0564068350 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f058414bcb0 tx=0x7f0578008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:43.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:43.954+0000 7f0572ffd640 1 -- 172.21.15.107:0/1845187560 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0574050a60 con 0x7f056c0052b0 2023-10-25T00:03:44.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 -- 172.21.15.107:0/1845187560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0584065a00 con 0x7f056c0052b0 2023-10-25T00:03:44.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0572ffd640 1 -- 172.21.15.107:0/1845187560 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5773 (secure 0 0 0) 0x7f0574054080 con 0x7f056c0052b0 2023-10-25T00:03:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:44.200 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6672,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:44.200 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 90 2023-10-25T00:03:44.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 -- 172.21.15.107:0/1845187560 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e90 msgr2=0x7f0564068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:44.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e90 0x7f0564068350 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f058414bcb0 tx=0x7f0578008040 comp rx=0 tx=0).stop 2023-10-25T00:03:44.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 -- 172.21.15.107:0/1845187560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 msgr2=0x7f05840c5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:44.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 0x7f05840c5b80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f057400e780 tx=0x7f057400ec40 comp rx=0 tx=0).stop 2023-10-25T00:03:44.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 -- 172.21.15.107:0/1845187560 shutdown_connections 2023-10-25T00:03:44.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0564065e90 0x7f0564068350 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:44.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05840c5480 0x7f05841eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:44.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05840c5000 0x7f05841ee4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:44.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 --2- 172.21.15.107:0/1845187560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f056c0052b0 0x7f05840c5b80 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:44.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 -- 172.21.15.107:0/1845187560 >> 172.21.15.107:0/1845187560 conn(0x7f05840c65c0 msgr2=0x7f05840ab7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:44.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 -- 172.21.15.107:0/1845187560 shutdown_connections 2023-10-25T00:03:44.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:44.198+0000 7f0588b7b640 1 -- 172.21.15.107:0/1845187560 wait complete. 2023-10-25T00:03:44.222 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:03:46.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:46.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.362+0000 7f4cf3e48640 1 Processor -- start 2023-10-25T00:03:46.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.362+0000 7f4cf3e48640 1 -- start start 2023-10-25T00:03:46.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf3e48640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec13bb00 0x7f4cec1531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:46.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf3e48640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec152d40 con 0x7f4cec0a9450 2023-10-25T00:03:46.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf3e48640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec06d660 con 0x7f4cec0aa330 2023-10-25T00:03:46.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf3e48640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec156100 con 0x7f4cec13bb00 2023-10-25T00:03:46.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf23e5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4cec0a9450 0x7f4cec071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52400/0 (socket says 172.21.15.107:52400) 2023-10-25T00:03:46.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf23e5640 1 -- 172.21.15.107:0/1812264915 learned_addr learned my addr 172.21.15.107:0/1812264915 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:46.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf1be4640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec13bb00 0x7f4cec1531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:46.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020961468 0 0) 0x7f4cec152d40 con 0x7f4cec0a9450 2023-10-25T00:03:46.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd8003530 con 0x7f4cec0a9450 2023-10-25T00:03:46.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1862822185 0 0) 0x7f4cd8003530 con 0x7f4cec0a9450 2023-10-25T00:03:46.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec13bb00 msgr2=0x7f4cec1531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:46.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec13bb00 0x7f4cec1531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:46.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 >> v1:172.21.15.110:6789/0 conn(0x7f4cec0aa330 legacy=0x7f4cec06e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:46.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec155e60 con 0x7f4cec0a9450 2023-10-25T00:03:46.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f4ce4001a00 con 0x7f4cec0a9450 2023-10-25T00:03:46.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 >> v1:172.21.15.107:6789/0 conn(0x7f4cec0a9450 legacy=0x7f4cec071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:46.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cd80039b0 0x7f4cd8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:46.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec13bb00 0x7f4cd80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:46.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cd8004d10 0x7f4cd8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:46.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec156100 con 0x7f4cd80039b0 2023-10-25T00:03:46.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd800bb20 con 0x7f4cd8004d10 2023-10-25T00:03:46.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd800bcb0 con 0x7f4cec13bb00 2023-10-25T00:03:46.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf1be4640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cd80039b0 0x7f4cd8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:46.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf13e3640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec13bb00 0x7f4cd80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:46.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf23e5640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cd8004d10 0x7f4cd8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:46.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf1be4640 1 -- 172.21.15.107:0/1812264915 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec13bb00 msgr2=0x7f4cd80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf1be4640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec13bb00 0x7f4cd80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf1be4640 1 -- 172.21.15.107:0/1812264915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cd8004d10 msgr2=0x7f4cd8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf1be4640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cd8004d10 0x7f4cd8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.366+0000 7f4cf1be4640 1 -- 172.21.15.107:0/1812264915 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd8003530 con 0x7f4cd80039b0 2023-10-25T00:03:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.370+0000 7f4cf1be4640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cd80039b0 0x7f4cd8003d90 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009b10 tx=0x7f4ce801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29704bd3f1441516 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.370+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80a5510 con 0x7f4cd80039b0 2023-10-25T00:03:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.370+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce80a5670 con 0x7f4cd80039b0 2023-10-25T00:03:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.370+0000 7f4cf0be2640 1 -- 172.21.15.107:0/1812264915 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80a5a20 con 0x7f4cd80039b0 2023-10-25T00:03:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.370+0000 7f4cf3e48640 1 -- 172.21.15.107:0/1812264915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cd80039b0 msgr2=0x7f4cd8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:46.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.370+0000 7f4cf3e48640 1 --2- 172.21.15.107:0/1812264915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cd80039b0 0x7f4cd8003d90 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009b10 tx=0x7f4ce801ca50 comp rx=0 tx=0).stop 2023-10-25T00:03:46.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.370+0000 7f4cf23e5640 1 -- 172.21.15.107:0/1812264915 reap_dead start 2023-10-25T00:03:46.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.370+0000 7f4cf3e48640 1 -- 172.21.15.107:0/1812264915 shutdown_connections 2023-10-25T00:03:46.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.370+0000 7f4cf3e48640 1 -- 172.21.15.107:0/1812264915 >> 172.21.15.107:0/1812264915 conn(0x7f4cec070f60 msgr2=0x7f4cec071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:46.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.370+0000 7f4cf3e48640 1 -- 172.21.15.107:0/1812264915 shutdown_connections 2023-10-25T00:03:46.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.370+0000 7f4cf3e48640 1 -- 172.21.15.107:0/1812264915 wait complete. 2023-10-25T00:03:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf3e48640 1 Processor -- start 2023-10-25T00:03:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf3e48640 1 -- start start 2023-10-25T00:03:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf3e48640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cd80039b0 0x7f4cec0c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:46.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf3e48640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec06c110 0x7f4cec14f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf3e48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec06aec0 0x7f4cec14f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf3e48640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0c7250 con 0x7f4cec06aec0 2023-10-25T00:03:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf3e48640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec06bc80 con 0x7f4cd80039b0 2023-10-25T00:03:46.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf3e48640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0c5e50 con 0x7f4cec06c110 2023-10-25T00:03:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf23e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec06aec0 0x7f4cec14f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf23e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec06aec0 0x7f4cec14f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40134/0 (socket says 172.21.15.107:40134) 2023-10-25T00:03:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf23e5640 1 -- 172.21.15.107:0/3208725001 learned_addr learned my addr 172.21.15.107:0/3208725001 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf13e3640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec06c110 0x7f4cec14f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf1be4640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cd80039b0 0x7f4cec0c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:46.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf23e5640 1 -- 172.21.15.107:0/3208725001 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec06c110 msgr2=0x7f4cec14f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:46.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf23e5640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec06c110 0x7f4cec14f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:46.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf23e5640 1 -- 172.21.15.107:0/3208725001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cd80039b0 msgr2=0x7f4cec0c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:46.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf23e5640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cd80039b0 0x7f4cec0c1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:46.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf23e5640 1 -- 172.21.15.107:0/3208725001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec0c60b0 con 0x7f4cec06aec0 2023-10-25T00:03:46.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf23e5640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec06aec0 0x7f4cec14f870 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800be40 tx=0x7f4ce400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:46.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4ce27fc640 1 -- 172.21.15.107:0/3208725001 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce400ec10 con 0x7f4cec06aec0 2023-10-25T00:03:46.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4ce27fc640 1 -- 172.21.15.107:0/3208725001 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce400ed70 con 0x7f4cec06aec0 2023-10-25T00:03:46.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf3e48640 1 -- 172.21.15.107:0/3208725001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cec06b910 con 0x7f4cec06aec0 2023-10-25T00:03:46.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4ce27fc640 1 -- 172.21.15.107:0/3208725001 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce400eee0 con 0x7f4cec06aec0 2023-10-25T00:03:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf3e48640 1 -- 172.21.15.107:0/3208725001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cec0ade50 con 0x7f4cec06aec0 2023-10-25T00:03:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.374+0000 7f4cf3e48640 1 -- 172.21.15.107:0/3208725001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cec06bdf0 con 0x7f4cec06aec0 2023-10-25T00:03:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.378+0000 7f4ce27fc640 1 -- 172.21.15.107:0/3208725001 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4ce4013660 con 0x7f4cec06aec0 2023-10-25T00:03:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.378+0000 7f4ce27fc640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e90 0x7f4cd0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.378+0000 7f4ce27fc640 1 -- 172.21.15.107:0/3208725001 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7f4ce4084410 con 0x7f4cec06aec0 2023-10-25T00:03:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.378+0000 7f4cf1be4640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e90 0x7f4cd0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.378+0000 7f4cf1be4640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e90 0x7f4cd0068350 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80af000 tx=0x7f4ce80046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:46.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.382+0000 7f4ce27fc640 1 -- 172.21.15.107:0/3208725001 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4ce40511b0 con 0x7f4cec06aec0 2023-10-25T00:03:46.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.614+0000 7f4cf3e48640 1 -- 172.21.15.107:0/3208725001 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cec0746e0 con 0x7f4cec06aec0 2023-10-25T00:03:46.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.614+0000 7f4ce27fc640 1 -- 172.21.15.107:0/3208725001 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5773 (secure 0 0 0) 0x7f4ce40547d0 con 0x7f4cec06aec0 2023-10-25T00:03:46.619 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:46.619 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6672,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:46.619 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 90 2023-10-25T00:03:46.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 -- 172.21.15.107:0/3208725001 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e90 msgr2=0x7f4cd0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:46.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e90 0x7f4cd0068350 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80af000 tx=0x7f4ce80046d0 comp rx=0 tx=0).stop 2023-10-25T00:03:46.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 -- 172.21.15.107:0/3208725001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec06aec0 msgr2=0x7f4cec14f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:46.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec06aec0 0x7f4cec14f870 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800be40 tx=0x7f4ce400e530 comp rx=0 tx=0).stop 2023-10-25T00:03:46.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 -- 172.21.15.107:0/3208725001 shutdown_connections 2023-10-25T00:03:46.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec06aec0 0x7f4cec14f870 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:46.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e90 0x7f4cd0068350 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:46.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec06c110 0x7f4cec14f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:46.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 --2- 172.21.15.107:0/3208725001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cd80039b0 0x7f4cec0c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:46.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 -- 172.21.15.107:0/3208725001 >> 172.21.15.107:0/3208725001 conn(0x7f4cec070f60 msgr2=0x7f4cec151bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:46.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 -- 172.21.15.107:0/3208725001 shutdown_connections 2023-10-25T00:03:46.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:46.618+0000 7f4cf3e48640 1 -- 172.21.15.107:0/3208725001 wait complete. 2023-10-25T00:03:46.642 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:03:48.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:48.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e9d00640 1 Processor -- start 2023-10-25T00:03:48.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e9d00640 1 -- start start 2023-10-25T00:03:48.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e9d00640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e4152040 0x7f70e40c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e9d00640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f70e406a4b0 0x7f70e40ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e9d00640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e4155680 con 0x7f70e40c8100 2023-10-25T00:03:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e9d00640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e4154a70 con 0x7f70e406a4b0 2023-10-25T00:03:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e9d00640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e4154d50 con 0x7f70e4152040 2023-10-25T00:03:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f70e40c8100 0x7f70e40abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52408/0 (socket says 172.21.15.107:52408) 2023-10-25T00:03:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e2ffd640 1 -- 172.21.15.107:0/602054005 learned_addr learned my addr 172.21.15.107:0/602054005 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e3fff640 1 --2- 172.21.15.107:0/602054005 >> v2:172.21.15.107:3300/0 conn(0x7f70e406a4b0 0x7f70e40ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e27fc640 1 -- 172.21.15.107:0/602054005 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411680581 0 0) 0x7f70e4155680 con 0x7f70e40c8100 2023-10-25T00:03:48.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e37fe640 1 --2- 172.21.15.107:0/602054005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e4152040 0x7f70e40c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e27fc640 1 -- 172.21.15.107:0/602054005 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70cc003530 con 0x7f70e40c8100 2023-10-25T00:03:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e3fff640 1 -- 172.21.15.107:0/602054005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e4152040 msgr2=0x7f70e40c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e3fff640 1 --2- 172.21.15.107:0/602054005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e4152040 0x7f70e40c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e3fff640 1 -- 172.21.15.107:0/602054005 >> v1:172.21.15.107:6789/0 conn(0x7f70e40c8100 legacy=0x7f70e40abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:48.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e3fff640 1 -- 172.21.15.107:0/602054005 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e4155030 con 0x7f70e406a4b0 2023-10-25T00:03:48.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.778+0000 7f70e37fe640 1 --2- 172.21.15.107:0/602054005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e4152040 0x7f70e40c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e3fff640 1 --2- 172.21.15.107:0/602054005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e406a4b0 0x7f70e40ac410 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009d90 tx=0x7f70dc01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884a032f45084fed server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e27fc640 1 -- 172.21.15.107:0/602054005 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0a5890 con 0x7f70e406a4b0 2023-10-25T00:03:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e27fc640 1 -- 172.21.15.107:0/602054005 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70dc0a59f0 con 0x7f70e406a4b0 2023-10-25T00:03:48.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e27fc640 1 -- 172.21.15.107:0/602054005 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70dc0a5d80 con 0x7f70e406a4b0 2023-10-25T00:03:48.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 -- 172.21.15.107:0/602054005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e406a4b0 msgr2=0x7f70e40ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:48.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 --2- 172.21.15.107:0/602054005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e406a4b0 0x7f70e40ac410 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f70dc009d90 tx=0x7f70dc01c8c0 comp rx=0 tx=0).stop 2023-10-25T00:03:48.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 -- 172.21.15.107:0/602054005 shutdown_connections 2023-10-25T00:03:48.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 --2- 172.21.15.107:0/602054005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e406a4b0 0x7f70e40ac410 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 --2- 172.21.15.107:0/602054005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e4152040 0x7f70e40c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 -- 172.21.15.107:0/602054005 >> 172.21.15.107:0/602054005 conn(0x7f70e40ca200 msgr2=0x7f70e40ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 -- 172.21.15.107:0/602054005 shutdown_connections 2023-10-25T00:03:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 -- 172.21.15.107:0/602054005 wait complete. 2023-10-25T00:03:48.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 Processor -- start 2023-10-25T00:03:48.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 -- start start 2023-10-25T00:03:48.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e406a4b0 0x7f70e4153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:48.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e40c8100 0x7f70e4144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:48.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e4152040 0x7f70e4144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:48.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e41544b0 con 0x7f70e406a4b0 2023-10-25T00:03:48.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e414d890 con 0x7f70e4152040 2023-10-25T00:03:48.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.782+0000 7f70e9d00640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e4143be0 con 0x7f70e40c8100 2023-10-25T00:03:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e406a4b0 0x7f70e4153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e406a4b0 0x7f70e4153940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40152/0 (socket says 172.21.15.107:40152) 2023-10-25T00:03:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e37fe640 1 -- 172.21.15.107:0/247339873 learned_addr learned my addr 172.21.15.107:0/247339873 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e3fff640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e4152040 0x7f70e4144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:48.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e2ffd640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e40c8100 0x7f70e4144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:48.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e37fe640 1 -- 172.21.15.107:0/247339873 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e40c8100 msgr2=0x7f70e4144350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:48.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e37fe640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e40c8100 0x7f70e4144350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:48.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e37fe640 1 -- 172.21.15.107:0/247339873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e4152040 msgr2=0x7f70e4144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:48.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e37fe640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e4152040 0x7f70e4144890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:48.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e37fe640 1 -- 172.21.15.107:0/247339873 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc009a70 con 0x7f70e406a4b0 2023-10-25T00:03:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e3fff640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e4152040 0x7f70e4144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e2ffd640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e40c8100 0x7f70e4144350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e37fe640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e406a4b0 0x7f70e4153940 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f70d400c7c0 tx=0x7f70d400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70cbfff640 1 -- 172.21.15.107:0/247339873 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d4007a30 con 0x7f70e406a4b0 2023-10-25T00:03:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70cbfff640 1 -- 172.21.15.107:0/247339873 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d4007b90 con 0x7f70e406a4b0 2023-10-25T00:03:48.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e9d00640 1 -- 172.21.15.107:0/247339873 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e4153470 con 0x7f70e406a4b0 2023-10-25T00:03:48.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70cbfff640 1 -- 172.21.15.107:0/247339873 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d400ce70 con 0x7f70e406a4b0 2023-10-25T00:03:48.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e9d00640 1 -- 172.21.15.107:0/247339873 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e41ed980 con 0x7f70e406a4b0 2023-10-25T00:03:48.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e9d00640 1 -- 172.21.15.107:0/247339873 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e40746e0 con 0x7f70e406a4b0 2023-10-25T00:03:48.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70cbfff640 1 -- 172.21.15.107:0/247339873 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f70d40113f0 con 0x7f70e406a4b0 2023-10-25T00:03:48.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70cbfff640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d0065e60 0x7f70d0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70cbfff640 1 -- 172.21.15.107:0/247339873 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7f70d40845c0 con 0x7f70e406a4b0 2023-10-25T00:03:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e2ffd640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d0065e60 0x7f70d0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.786+0000 7f70e2ffd640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d0065e60 0x7f70d0068320 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f70e406b4b0 tx=0x7f70d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:48.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:48.794+0000 7f70cbfff640 1 -- 172.21.15.107:0/247339873 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f70d4051360 con 0x7f70e406a4b0 2023-10-25T00:03:49.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.014+0000 7f70e9d00640 1 -- 172.21.15.107:0/247339873 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70e40c8cc0 con 0x7f70e406a4b0 2023-10-25T00:03:49.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.014+0000 7f70cbfff640 1 -- 172.21.15.107:0/247339873 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5773 (secure 0 0 0) 0x7f70d4054980 con 0x7f70e406a4b0 2023-10-25T00:03:49.018 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:49.018 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6672,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 90 2023-10-25T00:03:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 -- 172.21.15.107:0/247339873 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d0065e60 msgr2=0x7f70d0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d0065e60 0x7f70d0068320 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f70e406b4b0 tx=0x7f70d8009040 comp rx=0 tx=0).stop 2023-10-25T00:03:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 -- 172.21.15.107:0/247339873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e406a4b0 msgr2=0x7f70e4153940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e406a4b0 0x7f70e4153940 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f70d400c7c0 tx=0x7f70d400cc80 comp rx=0 tx=0).stop 2023-10-25T00:03:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 -- 172.21.15.107:0/247339873 shutdown_connections 2023-10-25T00:03:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70e4152040 0x7f70e4144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d0065e60 0x7f70d0068320 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70e40c8100 0x7f70e4144350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 --2- 172.21.15.107:0/247339873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e406a4b0 0x7f70e4153940 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 -- 172.21.15.107:0/247339873 >> 172.21.15.107:0/247339873 conn(0x7f70e40ca200 msgr2=0x7f70e40c91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 -- 172.21.15.107:0/247339873 shutdown_connections 2023-10-25T00:03:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:49.018+0000 7f70e9d00640 1 -- 172.21.15.107:0/247339873 wait complete. 2023-10-25T00:03:49.043 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:03:51.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:51.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29ecb17640 1 Processor -- start 2023-10-25T00:03:51.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29ecb17640 1 -- start start 2023-10-25T00:03:51.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29ecb17640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29e814eb70 0x7f29e80c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:51.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29ecb17640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f29e806df30 0x7f29e806e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:51.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29ecb17640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e80c22f0 con 0x7f29e8070b00 2023-10-25T00:03:51.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29ecb17640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29e8071700 con 0x7f29e806df30 2023-10-25T00:03:51.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29ecb17640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e80c25d0 con 0x7f29e814eb70 2023-10-25T00:03:51.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29e5d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f29e8070b00 0x7f29e81514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52422/0 (socket says 172.21.15.107:52422) 2023-10-25T00:03:51.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29e5d9b640 1 -- 172.21.15.107:0/2098600424 learned_addr learned my addr 172.21.15.107:0/2098600424 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:51.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29e6d9d640 1 --2- 172.21.15.107:0/2098600424 >> v2:172.21.15.107:3300/0 conn(0x7f29e806df30 0x7f29e806e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:51.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29e659c640 1 --2- 172.21.15.107:0/2098600424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29e814eb70 0x7f29e80c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:51.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29e6d9d640 1 -- 172.21.15.107:0/2098600424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29e814eb70 msgr2=0x7f29e80c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:51.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29e6d9d640 1 --2- 172.21.15.107:0/2098600424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29e814eb70 0x7f29e80c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:51.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29e6d9d640 1 -- 172.21.15.107:0/2098600424 >> v1:172.21.15.107:6789/0 conn(0x7f29e8070b00 legacy=0x7f29e81514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:51.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29e559a640 1 -- 172.21.15.107:0/2098600424 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338318961 0 0) 0x7f29e80c22f0 con 0x7f29e8070b00 2023-10-25T00:03:51.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.178+0000 7f29e6d9d640 1 -- 172.21.15.107:0/2098600424 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e80c2be0 con 0x7f29e806df30 2023-10-25T00:03:51.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29e6d9d640 1 --2- 172.21.15.107:0/2098600424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29e806df30 0x7f29e806e360 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009d90 tx=0x7f29e0020d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615d7648f88cf692 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:51.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29e559a640 1 -- 172.21.15.107:0/2098600424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e00a9d50 con 0x7f29e806df30 2023-10-25T00:03:51.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29e559a640 1 -- 172.21.15.107:0/2098600424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e00b1440 con 0x7f29e806df30 2023-10-25T00:03:51.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29e559a640 1 -- 172.21.15.107:0/2098600424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e00b17d0 con 0x7f29e806df30 2023-10-25T00:03:51.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29ecb17640 1 -- 172.21.15.107:0/2098600424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29e806df30 msgr2=0x7f29e806e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:51.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29ecb17640 1 --2- 172.21.15.107:0/2098600424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29e806df30 0x7f29e806e360 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f29e0009d90 tx=0x7f29e0020d60 comp rx=0 tx=0).stop 2023-10-25T00:03:51.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29ecb17640 1 -- 172.21.15.107:0/2098600424 shutdown_connections 2023-10-25T00:03:51.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29ecb17640 1 --2- 172.21.15.107:0/2098600424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29e806df30 0x7f29e806e360 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:51.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29ecb17640 1 --2- 172.21.15.107:0/2098600424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29e814eb70 0x7f29e80c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:51.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29ecb17640 1 -- 172.21.15.107:0/2098600424 >> 172.21.15.107:0/2098600424 conn(0x7f29e80c39a0 msgr2=0x7f29e80c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:51.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29ecb17640 1 -- 172.21.15.107:0/2098600424 shutdown_connections 2023-10-25T00:03:51.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29ecb17640 1 -- 172.21.15.107:0/2098600424 wait complete. 2023-10-25T00:03:51.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29ecb17640 1 Processor -- start 2023-10-25T00:03:51.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.182+0000 7f29ecb17640 1 -- start start 2023-10-25T00:03:51.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29ecb17640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29e806df30 0x7f29e80abec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:51.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29ecb17640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29e8070b00 0x7f29e80aaa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:51.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29ecb17640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29e814eb70 0x7f29e8153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:51.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29ecb17640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e81532e0 con 0x7f29e814eb70 2023-10-25T00:03:51.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29ecb17640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e81ec690 con 0x7f29e806df30 2023-10-25T00:03:51.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29ecb17640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e81ecbc0 con 0x7f29e8070b00 2023-10-25T00:03:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29e814eb70 0x7f29e8153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29e814eb70 0x7f29e8153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40158/0 (socket says 172.21.15.107:40158) 2023-10-25T00:03:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e6d9d640 1 -- 172.21.15.107:0/278922833 learned_addr learned my addr 172.21.15.107:0/278922833 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e5d9b640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29e8070b00 0x7f29e80aaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e659c640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29e806df30 0x7f29e80abec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:51.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e6d9d640 1 -- 172.21.15.107:0/278922833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29e8070b00 msgr2=0x7f29e80aaa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:51.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e6d9d640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29e8070b00 0x7f29e80aaa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:51.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e6d9d640 1 -- 172.21.15.107:0/278922833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29e806df30 msgr2=0x7f29e80abec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:51.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e6d9d640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29e806df30 0x7f29e80abec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:51.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e6d9d640 1 -- 172.21.15.107:0/278922833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e0009a70 con 0x7f29e814eb70 2023-10-25T00:03:51.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e659c640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29e806df30 0x7f29e80abec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:51.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e6d9d640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29e814eb70 0x7f29e8153de0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f29e00b4040 tx=0x7f29e001db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:51.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29d6ffd640 1 -- 172.21.15.107:0/278922833 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e00b1c30 con 0x7f29e814eb70 2023-10-25T00:03:51.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29d6ffd640 1 -- 172.21.15.107:0/278922833 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e00b1d90 con 0x7f29e814eb70 2023-10-25T00:03:51.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29ecb17640 1 -- 172.21.15.107:0/278922833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e80ac400 con 0x7f29e814eb70 2023-10-25T00:03:51.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29d6ffd640 1 -- 172.21.15.107:0/278922833 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e00bd690 con 0x7f29e814eb70 2023-10-25T00:03:51.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29ecb17640 1 -- 172.21.15.107:0/278922833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e81ec8e0 con 0x7f29e814eb70 2023-10-25T00:03:51.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29d4ff9640 1 -- 172.21.15.107:0/278922833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c0005260 con 0x7f29e814eb70 2023-10-25T00:03:51.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29d6ffd640 1 -- 172.21.15.107:0/278922833 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f29e00bd7f0 con 0x7f29e814eb70 2023-10-25T00:03:51.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29d6ffd640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29c8065e60 0x7f29c8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:51.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29d6ffd640 1 -- 172.21.15.107:0/278922833 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 8337+0+0 (secure 0 0 0) 0x7f29e0123880 con 0x7f29e814eb70 2023-10-25T00:03:51.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.186+0000 7f29e659c640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29c8065e60 0x7f29c8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:51.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.190+0000 7f29e659c640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29c8065e60 0x7f29c8068320 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f29d800c9e0 tx=0x7f29d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:51.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.194+0000 7f29d6ffd640 1 -- 172.21.15.107:0/278922833 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f29e00f0620 con 0x7f29e814eb70 2023-10-25T00:03:51.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.422+0000 7f29d4ff9640 1 -- 172.21.15.107:0/278922833 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29c0005a90 con 0x7f29e814eb70 2023-10-25T00:03:51.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.422+0000 7f29d6ffd640 1 -- 172.21.15.107:0/278922833 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5773 (secure 0 0 0) 0x7f29e00f3c40 con 0x7f29e814eb70 2023-10-25T00:03:51.426 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:51.426 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6672,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":90}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:02:26.743229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6483":{"gid":6483,"name":"a","rank":0,"incarnation":81,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/319918260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":319918260},{"type":"v1","addr":"172.21.15.107:6837","nonce":319918260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:51.427 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 90 2023-10-25T00:03:51.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 -- 172.21.15.107:0/278922833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29c8065e60 msgr2=0x7f29c8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:51.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29c8065e60 0x7f29c8068320 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f29d800c9e0 tx=0x7f29d8008040 comp rx=0 tx=0).stop 2023-10-25T00:03:51.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 -- 172.21.15.107:0/278922833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29e814eb70 msgr2=0x7f29e8153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:51.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29e814eb70 0x7f29e8153de0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f29e00b4040 tx=0x7f29e001db00 comp rx=0 tx=0).stop 2023-10-25T00:03:51.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 -- 172.21.15.107:0/278922833 shutdown_connections 2023-10-25T00:03:51.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29e814eb70 0x7f29e8153de0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:51.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f29c8065e60 0x7f29c8068320 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:51.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f29e8070b00 0x7f29e80aaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:51.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 --2- 172.21.15.107:0/278922833 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f29e806df30 0x7f29e80abec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 -- 172.21.15.107:0/278922833 >> 172.21.15.107:0/278922833 conn(0x7f29e80c39a0 msgr2=0x7f29e80c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 -- 172.21.15.107:0/278922833 shutdown_connections 2023-10-25T00:03:51.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:51.426+0000 7f29d4ff9640 1 -- 172.21.15.107:0/278922833 wait complete. 2023-10-25T00:03:51.446 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:03:53.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1cb177d640 1 Processor -- start 2023-10-25T00:03:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1cb177d640 1 -- start start 2023-10-25T00:03:53.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1cb177d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cac13dc20 0x7f1cac0c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:53.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1cb177d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1cac0720c0 0x7f1cac06df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:53.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1cb177d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cac154860 con 0x7f1cac06e470 2023-10-25T00:03:53.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1cb177d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cac0c1f10 con 0x7f1cac0720c0 2023-10-25T00:03:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1cb177d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac154b20 con 0x7f1cac13dc20 2023-10-25T00:03:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1caa7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1cac0720c0 0x7f1cac06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1caa7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1cac0720c0 0x7f1cac06df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50392/0 (socket says 172.21.15.107:50392) 2023-10-25T00:03:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1caa7fc640 1 -- 172.21.15.107:0/3178370690 learned_addr learned my addr 172.21.15.107:0/3178370690 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1caa7fc640 1 -- 172.21.15.107:0/3178370690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cac13dc20 msgr2=0x7f1cac0c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1caa7fc640 1 --2- 172.21.15.107:0/3178370690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cac13dc20 0x7f1cac0c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1caa7fc640 1 -- 172.21.15.107:0/3178370690 >> v1:172.21.15.110:6789/0 conn(0x7f1cac06e470 legacy=0x7f1cac157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.586+0000 7f1caa7fc640 1 -- 172.21.15.107:0/3178370690 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac154e00 con 0x7f1cac0720c0 2023-10-25T00:03:53.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.590+0000 7f1caa7fc640 1 --2- 172.21.15.107:0/3178370690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cac0720c0 0x7f1cac06df30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1ca0009870 tx=0x7f1ca001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8a4366ec78a16e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1ca9ffb640 1 -- 172.21.15.107:0/3178370690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca00ab600 con 0x7f1cac0720c0 2023-10-25T00:03:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1ca9ffb640 1 -- 172.21.15.107:0/3178370690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca00a7050 con 0x7f1cac0720c0 2023-10-25T00:03:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1ca9ffb640 1 -- 172.21.15.107:0/3178370690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca00ab980 con 0x7f1cac0720c0 2023-10-25T00:03:53.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1cb177d640 1 -- 172.21.15.107:0/3178370690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cac0720c0 msgr2=0x7f1cac06df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:53.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1cb177d640 1 --2- 172.21.15.107:0/3178370690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cac0720c0 0x7f1cac06df30 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1ca0009870 tx=0x7f1ca001c3e0 comp rx=0 tx=0).stop 2023-10-25T00:03:53.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1cb177d640 1 -- 172.21.15.107:0/3178370690 shutdown_connections 2023-10-25T00:03:53.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1cb177d640 1 --2- 172.21.15.107:0/3178370690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cac0720c0 0x7f1cac06df30 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:53.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1cb177d640 1 --2- 172.21.15.107:0/3178370690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cac13dc20 0x7f1cac0c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:53.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1cb177d640 1 -- 172.21.15.107:0/3178370690 >> 172.21.15.107:0/3178370690 conn(0x7f1cac0c39a0 msgr2=0x7f1cac0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:53.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1cb177d640 1 -- 172.21.15.107:0/3178370690 shutdown_connections 2023-10-25T00:03:53.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1cb177d640 1 -- 172.21.15.107:0/3178370690 wait complete. 2023-10-25T00:03:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1cb177d640 1 Processor -- start 2023-10-25T00:03:53.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.594+0000 7f1cb177d640 1 -- start start 2023-10-25T00:03:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1cb177d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cac06e470 0x7f1cac0aa220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1cb177d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cac0720c0 0x7f1cac137b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1cb177d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cac13dc20 0x7f1cac153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1cb177d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac0a99d0 con 0x7f1cac06e470 2023-10-25T00:03:53.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1cb177d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac1f11f0 con 0x7f1cac13dc20 2023-10-25T00:03:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1cb177d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac158db0 con 0x7f1cac0720c0 2023-10-25T00:03:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1caaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cac06e470 0x7f1cac0aa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1caaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cac06e470 0x7f1cac0aa220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50406/0 (socket says 172.21.15.107:50406) 2023-10-25T00:03:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1caaffd640 1 -- 172.21.15.107:0/2540505192 learned_addr learned my addr 172.21.15.107:0/2540505192 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:53.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1caa7fc640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cac0720c0 0x7f1cac137b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:53.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1cab7fe640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cac13dc20 0x7f1cac153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:53.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1caaffd640 1 -- 172.21.15.107:0/2540505192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cac0720c0 msgr2=0x7f1cac137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:53.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1caaffd640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cac0720c0 0x7f1cac137b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:53.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1caaffd640 1 -- 172.21.15.107:0/2540505192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cac13dc20 msgr2=0x7f1cac153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:53.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1caaffd640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cac13dc20 0x7f1cac153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1caaffd640 1 -- 172.21.15.107:0/2540505192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca00094d0 con 0x7f1cac06e470 2023-10-25T00:03:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1caaffd640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cac06e470 0x7f1cac0aa220 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00b550 tx=0x7f1c9c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1c937fe640 1 -- 172.21.15.107:0/2540505192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c010870 con 0x7f1cac06e470 2023-10-25T00:03:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1c937fe640 1 -- 172.21.15.107:0/2540505192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c0109f0 con 0x7f1cac06e470 2023-10-25T00:03:53.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1cb177d640 1 -- 172.21.15.107:0/2540505192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cac1f0c70 con 0x7f1cac06e470 2023-10-25T00:03:53.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1c937fe640 1 -- 172.21.15.107:0/2540505192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c010da0 con 0x7f1cac06e470 2023-10-25T00:03:53.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1cb177d640 1 -- 172.21.15.107:0/2540505192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cac1ecee0 con 0x7f1cac06e470 2023-10-25T00:03:53.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1cb177d640 1 -- 172.21.15.107:0/2540505192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cac0746e0 con 0x7f1cac06e470 2023-10-25T00:03:53.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1c937fe640 1 -- 172.21.15.107:0/2540505192 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1c9c00f3c0 con 0x7f1cac06e470 2023-10-25T00:03:53.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1c937fe640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c98065e60 0x7f1c98068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:53.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1caa7fc640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c98065e60 0x7f1c98068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:53.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.598+0000 7f1c937fe640 1 -- 172.21.15.107:0/2540505192 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f1c9c088080 con 0x7f1cac06e470 2023-10-25T00:03:53.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.602+0000 7f1caa7fc640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c98065e60 0x7f1c98068320 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00ad000 tx=0x7f1ca00a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:53.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.606+0000 7f1c937fe640 1 -- 172.21.15.107:0/2540505192 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1c9c050e80 con 0x7f1cac06e470 2023-10-25T00:03:53.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.846+0000 7f1cb177d640 1 -- 172.21.15.107:0/2540505192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cac0c07d0 con 0x7f1cac06e470 2023-10-25T00:03:53.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.846+0000 7f1c937fe640 1 -- 172.21.15.107:0/2540505192 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5016 (secure 0 0 0) 0x7f1c9c01e020 con 0x7f1cac06e470 2023-10-25T00:03:53.850 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:53.850 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:53.580110+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:53.851 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 91 2023-10-25T00:03:53.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 -- 172.21.15.107:0/2540505192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c98065e60 msgr2=0x7f1c98068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:53.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c98065e60 0x7f1c98068320 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00ad000 tx=0x7f1ca00a6000 comp rx=0 tx=0).stop 2023-10-25T00:03:53.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 -- 172.21.15.107:0/2540505192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cac06e470 msgr2=0x7f1cac0aa220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:53.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cac06e470 0x7f1cac0aa220 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00b550 tx=0x7f1c9c00ba10 comp rx=0 tx=0).stop 2023-10-25T00:03:53.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 -- 172.21.15.107:0/2540505192 shutdown_connections 2023-10-25T00:03:53.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cac13dc20 0x7f1cac153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:53.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c98065e60 0x7f1c98068320 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:53.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1cac0720c0 0x7f1cac137b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:53.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 --2- 172.21.15.107:0/2540505192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1cac06e470 0x7f1cac0aa220 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:53.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 -- 172.21.15.107:0/2540505192 >> 172.21.15.107:0/2540505192 conn(0x7f1cac0c39a0 msgr2=0x7f1cac0c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:53.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 -- 172.21.15.107:0/2540505192 shutdown_connections 2023-10-25T00:03:53.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.850+0000 7f1cb177d640 1 -- 172.21.15.107:0/2540505192 wait complete. 2023-10-25T00:03:53.868 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2023-10-25T00:03:53.869 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:03:53.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:53.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee3c241640 1 Processor -- start 2023-10-25T00:03:54.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee3c241640 1 -- start start 2023-10-25T00:03:54.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee3c241640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee34156770 0x7fee340ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:54.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee3c241640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee340c70e0 0x7fee340bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee3c241640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee340c1ca0 con 0x7fee340c2130 2023-10-25T00:03:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee3c241640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee340c27e0 con 0x7fee34156770 2023-10-25T00:03:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee3c241640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee340c10c0 con 0x7fee340c70e0 2023-10-25T00:03:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee39fdd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee34156770 0x7fee340ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:54.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee39fdd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fee34156770 0x7fee340ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50420/0 (socket says 172.21.15.107:50420) 2023-10-25T00:03:54.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee39fdd640 1 -- 172.21.15.107:0/4129748322 learned_addr learned my addr 172.21.15.107:0/4129748322 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:54.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee397dc640 1 --2- 172.21.15.107:0/4129748322 >> v2:172.21.15.110:3300/0 conn(0x7fee340c70e0 0x7fee340bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:54.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee39fdd640 1 -- 172.21.15.107:0/4129748322 >> v2:172.21.15.110:3300/0 conn(0x7fee340c70e0 msgr2=0x7fee340bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:54.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee39fdd640 1 --2- 172.21.15.107:0/4129748322 >> v2:172.21.15.110:3300/0 conn(0x7fee340c70e0 0x7fee340bdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:54.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee39fdd640 1 -- 172.21.15.107:0/4129748322 >> v1:172.21.15.110:6789/0 conn(0x7fee340c2130 legacy=0x7fee34069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:54.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee39fdd640 1 -- 172.21.15.107:0/4129748322 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee340c13a0 con 0x7fee34156770 2023-10-25T00:03:54.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:53.998+0000 7fee397dc640 1 --2- 172.21.15.107:0/4129748322 >> v2:172.21.15.110:3300/0 conn(0x7fee340c70e0 0x7fee340bdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:54.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee39fdd640 1 --2- 172.21.15.107:0/4129748322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee34156770 0x7fee340ab7c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fee340a8180 tx=0x7fee3001c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e943272e4a799eb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:54.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee38fdb640 1 -- 172.21.15.107:0/4129748322 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300ab600 con 0x7fee34156770 2023-10-25T00:03:54.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee38fdb640 1 -- 172.21.15.107:0/4129748322 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee300a7050 con 0x7fee34156770 2023-10-25T00:03:54.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee38fdb640 1 -- 172.21.15.107:0/4129748322 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300ab980 con 0x7fee34156770 2023-10-25T00:03:54.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee3c241640 1 -- 172.21.15.107:0/4129748322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee34156770 msgr2=0x7fee340ab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:54.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee3c241640 1 --2- 172.21.15.107:0/4129748322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee34156770 0x7fee340ab7c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fee340a8180 tx=0x7fee3001c370 comp rx=0 tx=0).stop 2023-10-25T00:03:54.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee3c241640 1 -- 172.21.15.107:0/4129748322 shutdown_connections 2023-10-25T00:03:54.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee3c241640 1 --2- 172.21.15.107:0/4129748322 >> v2:172.21.15.110:3300/0 conn(0x7fee340c70e0 0x7fee340bdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:54.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee3c241640 1 --2- 172.21.15.107:0/4129748322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee34156770 0x7fee340ab7c0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:54.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee3c241640 1 -- 172.21.15.107:0/4129748322 >> 172.21.15.107:0/4129748322 conn(0x7fee340c65c0 msgr2=0x7fee340c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:54.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee3c241640 1 -- 172.21.15.107:0/4129748322 shutdown_connections 2023-10-25T00:03:54.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.002+0000 7fee3c241640 1 -- 172.21.15.107:0/4129748322 wait complete. 2023-10-25T00:03:54.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3c241640 1 Processor -- start 2023-10-25T00:03:54.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3c241640 1 -- start start 2023-10-25T00:03:54.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3c241640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee340c2130 0x7fee34150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:54.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3c241640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee340c70e0 0x7fee341512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:54.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3c241640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee34156770 0x7fee3415b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:54.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3c241640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee340a8070 con 0x7fee34156770 2023-10-25T00:03:54.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3c241640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee340aaac0 con 0x7fee340c70e0 2023-10-25T00:03:54.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3c241640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee340a7d90 con 0x7fee340c2130 2023-10-25T00:03:54.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3a7de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee34156770 0x7fee3415b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:54.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3a7de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee34156770 0x7fee3415b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50424/0 (socket says 172.21.15.107:50424) 2023-10-25T00:03:54.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3a7de640 1 -- 172.21.15.107:0/348733870 learned_addr learned my addr 172.21.15.107:0/348733870 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:54.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee397dc640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee340c70e0 0x7fee341512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:54.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee39fdd640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee340c2130 0x7fee34150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:54.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3a7de640 1 -- 172.21.15.107:0/348733870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee340c2130 msgr2=0x7fee34150db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:54.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3a7de640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee340c2130 0x7fee34150db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:54.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3a7de640 1 -- 172.21.15.107:0/348733870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee340c70e0 msgr2=0x7fee341512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:54.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3a7de640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee340c70e0 0x7fee341512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:54.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3a7de640 1 -- 172.21.15.107:0/348733870 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee300094d0 con 0x7fee34156770 2023-10-25T00:03:54.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3a7de640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee34156770 0x7fee3415b510 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00c780 tx=0x7fee2c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:54.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee2a7fc640 1 -- 172.21.15.107:0/348733870 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c004b70 con 0x7fee34156770 2023-10-25T00:03:54.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee2a7fc640 1 -- 172.21.15.107:0/348733870 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2c00ce90 con 0x7fee34156770 2023-10-25T00:03:54.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3c241640 1 -- 172.21.15.107:0/348733870 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee340aa470 con 0x7fee34156770 2023-10-25T00:03:54.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee2a7fc640 1 -- 172.21.15.107:0/348733870 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00a850 con 0x7fee34156770 2023-10-25T00:03:54.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3c241640 1 -- 172.21.15.107:0/348733870 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee340a95f0 con 0x7fee34156770 2023-10-25T00:03:54.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee3c241640 1 -- 172.21.15.107:0/348733870 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee340aa2d0 con 0x7fee34156770 2023-10-25T00:03:54.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.006+0000 7fee2a7fc640 1 -- 172.21.15.107:0/348733870 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fee2c00a9b0 con 0x7fee34156770 2023-10-25T00:03:54.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.010+0000 7fee2a7fc640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee18065e60 0x7fee18068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:54.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.010+0000 7fee2a7fc640 1 -- 172.21.15.107:0/348733870 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7fee2c084300 con 0x7fee34156770 2023-10-25T00:03:54.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.010+0000 7fee39fdd640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee18065e60 0x7fee18068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:54.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.010+0000 7fee39fdd640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee18065e60 0x7fee18068320 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fee300ad000 tx=0x7fee300a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:54.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.014+0000 7fee2a7fc640 1 -- 172.21.15.107:0/348733870 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fee340aa2d0 con 0x7fee34156770 2023-10-25T00:03:54.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.250+0000 7fee3c241640 1 -- 172.21.15.107:0/348733870 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee340aac70 con 0x7fee34156770 2023-10-25T00:03:54.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.250+0000 7fee2a7fc640 1 -- 172.21.15.107:0/348733870 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+5016 (secure 0 0 0) 0x7fee340aac70 con 0x7fee34156770 2023-10-25T00:03:54.254 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:54.255 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:53.580110+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:54.255 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 91 2023-10-25T00:03:54.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 -- 172.21.15.107:0/348733870 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee18065e60 msgr2=0x7fee18068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:54.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee18065e60 0x7fee18068320 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fee300ad000 tx=0x7fee300a6000 comp rx=0 tx=0).stop 2023-10-25T00:03:54.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 -- 172.21.15.107:0/348733870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee34156770 msgr2=0x7fee3415b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:54.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee34156770 0x7fee3415b510 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00c780 tx=0x7fee2c00cc40 comp rx=0 tx=0).stop 2023-10-25T00:03:54.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 -- 172.21.15.107:0/348733870 shutdown_connections 2023-10-25T00:03:54.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee34156770 0x7fee3415b510 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:54.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee340c70e0 0x7fee341512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:54.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee18065e60 0x7fee18068320 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:54.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 --2- 172.21.15.107:0/348733870 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee340c2130 0x7fee34150db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:54.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 -- 172.21.15.107:0/348733870 >> 172.21.15.107:0/348733870 conn(0x7fee340c65c0 msgr2=0x7fee34069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:54.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 -- 172.21.15.107:0/348733870 shutdown_connections 2023-10-25T00:03:54.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:54.254+0000 7fee3c241640 1 -- 172.21.15.107:0/348733870 wait complete. 2023-10-25T00:03:54.272 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 91, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5764": { "addr": "172.21.15.110:6834/2577776351", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 2577776351, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 2577776351, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5764, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:03:31.705318+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5764 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6672": { "addr": "172.21.15.107:6835/4180698155", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 4180698155, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 4180698155, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6672, "incarnation": 91, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:03:53.580110+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6672 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 59, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:03:56.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:56.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.410+0000 7fab680e3640 1 Processor -- start 2023-10-25T00:03:56.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.410+0000 7fab680e3640 1 -- start start 2023-10-25T00:03:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.410+0000 7fab680e3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600c33e0 0x7fab6015ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.410+0000 7fab680e3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab600c22f0 con 0x7fab60070b00 2023-10-25T00:03:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.410+0000 7fab680e3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab600c25d0 con 0x7fab6014eb70 2023-10-25T00:03:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.410+0000 7fab680e3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab60071700 con 0x7fab600c33e0 2023-10-25T00:03:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.410+0000 7fab6567e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fab60070b00 0x7fab6006df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45682/0 (socket says 172.21.15.107:45682) 2023-10-25T00:03:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.410+0000 7fab6567e640 1 -- 172.21.15.107:0/1469823226 learned_addr learned my addr 172.21.15.107:0/1469823226 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab66680640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600c33e0 0x7fab6015ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:56.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745683087 0 0) 0x7fab600c22f0 con 0x7fab60070b00 2023-10-25T00:03:56.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c003530 con 0x7fab60070b00 2023-10-25T00:03:56.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3792432071 0 0) 0x7fab4c003530 con 0x7fab60070b00 2023-10-25T00:03:56.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600c33e0 msgr2=0x7fab6015ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:56.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600c33e0 0x7fab6015ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:56.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 >> v1:172.21.15.110:6789/0 conn(0x7fab6014eb70 legacy=0x7fab601514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:56.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab600c2be0 con 0x7fab60070b00 2023-10-25T00:03:56.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fab50001680 con 0x7fab60070b00 2023-10-25T00:03:56.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 >> v1:172.21.15.107:6789/0 conn(0x7fab60070b00 legacy=0x7fab6006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:56.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab4c0039b0 0x7fab4c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:56.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab4c0047d0 0x7fab4c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:56.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600c33e0 0x7fab4c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:56.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c00dc50 con 0x7fab4c0039b0 2023-10-25T00:03:56.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c00b990 con 0x7fab4c0047d0 2023-10-25T00:03:56.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4c00bb50 con 0x7fab600c33e0 2023-10-25T00:03:56.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab65e7f640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab4c0039b0 0x7fab4c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:56.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab6567e640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab4c0047d0 0x7fab4c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:56.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab66680640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600c33e0 0x7fab4c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:56.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab65e7f640 1 -- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600c33e0 msgr2=0x7fab4c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:56.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab65e7f640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600c33e0 0x7fab4c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:56.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab65e7f640 1 -- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab4c0047d0 msgr2=0x7fab4c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:56.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab65e7f640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab4c0047d0 0x7fab4c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:56.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab65e7f640 1 -- 172.21.15.107:0/1469823226 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c003530 con 0x7fab4c0039b0 2023-10-25T00:03:56.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab66680640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600c33e0 0x7fab4c005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:03:56.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab65e7f640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab4c0039b0 0x7fab4c003dd0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fab5c001750 tx=0x7fab5c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e98a33a73a88ca2 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:56.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0052d0 con 0x7fab4c0039b0 2023-10-25T00:03:56.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c0a7050 con 0x7fab4c0039b0 2023-10-25T00:03:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.414+0000 7fab64e7d640 1 -- 172.21.15.107:0/1469823226 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c003600 con 0x7fab4c0039b0 2023-10-25T00:03:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 -- 172.21.15.107:0/1469823226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab4c0039b0 msgr2=0x7fab4c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 --2- 172.21.15.107:0/1469823226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab4c0039b0 0x7fab4c003dd0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fab5c001750 tx=0x7fab5c01c3a0 comp rx=0 tx=0).stop 2023-10-25T00:03:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab66680640 1 -- 172.21.15.107:0/1469823226 reap_dead start 2023-10-25T00:03:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 -- 172.21.15.107:0/1469823226 shutdown_connections 2023-10-25T00:03:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 -- 172.21.15.107:0/1469823226 >> 172.21.15.107:0/1469823226 conn(0x7fab600c39a0 msgr2=0x7fab600c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 -- 172.21.15.107:0/1469823226 shutdown_connections 2023-10-25T00:03:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 -- 172.21.15.107:0/1469823226 wait complete. 2023-10-25T00:03:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 Processor -- start 2023-10-25T00:03:56.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 -- start start 2023-10-25T00:03:56.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60147ef0 0x7fab601482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:56.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab60154070 0x7fab600a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:56.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600a9990 0x7fab60144b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:56.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab60145ba0 con 0x7fab60147ef0 2023-10-25T00:03:56.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab601538b0 con 0x7fab60154070 2023-10-25T00:03:56.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6006e370 con 0x7fab600a9990 2023-10-25T00:03:56.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab65e7f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60147ef0 0x7fab601482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab65e7f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60147ef0 0x7fab601482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50442/0 (socket says 172.21.15.107:50442) 2023-10-25T00:03:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab65e7f640 1 -- 172.21.15.107:0/4105308583 learned_addr learned my addr 172.21.15.107:0/4105308583 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab66680640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600a9990 0x7fab60144b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab6567e640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab60154070 0x7fab600a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:56.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab65e7f640 1 -- 172.21.15.107:0/4105308583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600a9990 msgr2=0x7fab60144b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab65e7f640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600a9990 0x7fab60144b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab65e7f640 1 -- 172.21.15.107:0/4105308583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab60154070 msgr2=0x7fab600a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab65e7f640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab60154070 0x7fab600a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab65e7f640 1 -- 172.21.15.107:0/4105308583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab601548f0 con 0x7fab60147ef0 2023-10-25T00:03:56.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab65e7f640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60147ef0 0x7fab601482d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fab5c01c1d0 tx=0x7fab4c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:56.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab567fc640 1 -- 172.21.15.107:0/4105308583 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0072f0 con 0x7fab60147ef0 2023-10-25T00:03:56.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab567fc640 1 -- 172.21.15.107:0/4105308583 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c007450 con 0x7fab60147ef0 2023-10-25T00:03:56.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 -- 172.21.15.107:0/4105308583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab60154bd0 con 0x7fab60147ef0 2023-10-25T00:03:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab567fc640 1 -- 172.21.15.107:0/4105308583 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c0067e0 con 0x7fab60147ef0 2023-10-25T00:03:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.418+0000 7fab680e3640 1 -- 172.21.15.107:0/4105308583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6015a1a0 con 0x7fab60147ef0 2023-10-25T00:03:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.422+0000 7fab680e3640 1 -- 172.21.15.107:0/4105308583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab600746e0 con 0x7fab60147ef0 2023-10-25T00:03:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.422+0000 7fab567fc640 1 -- 172.21.15.107:0/4105308583 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fab5c006940 con 0x7fab60147ef0 2023-10-25T00:03:56.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.422+0000 7fab567fc640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab44065e60 0x7fab44068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:56.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.422+0000 7fab567fc640 1 -- 172.21.15.107:0/4105308583 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7fab5c11e060 con 0x7fab60147ef0 2023-10-25T00:03:56.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.422+0000 7fab6567e640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab44065e60 0x7fab44068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:56.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.422+0000 7fab6567e640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab44065e60 0x7fab44068320 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fab60148ef0 tx=0x7fab50009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:56.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.426+0000 7fab567fc640 1 -- 172.21.15.107:0/4105308583 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fab5c0eae50 con 0x7fab60147ef0 2023-10-25T00:03:56.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.662+0000 7fab680e3640 1 -- 172.21.15.107:0/4105308583 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab600c1a50 con 0x7fab60147ef0 2023-10-25T00:03:56.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.662+0000 7fab567fc640 1 -- 172.21.15.107:0/4105308583 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5016 (secure 0 0 0) 0x7fab5c0ee470 con 0x7fab60147ef0 2023-10-25T00:03:56.665 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:56.665 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:55.666415+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:56.665 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 93 2023-10-25T00:03:56.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 -- 172.21.15.107:0/4105308583 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab44065e60 msgr2=0x7fab44068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:56.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab44065e60 0x7fab44068320 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fab60148ef0 tx=0x7fab50009040 comp rx=0 tx=0).stop 2023-10-25T00:03:56.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 -- 172.21.15.107:0/4105308583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60147ef0 msgr2=0x7fab601482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:56.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60147ef0 0x7fab601482d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fab5c01c1d0 tx=0x7fab4c005800 comp rx=0 tx=0).stop 2023-10-25T00:03:56.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 -- 172.21.15.107:0/4105308583 shutdown_connections 2023-10-25T00:03:56.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fab44065e60 0x7fab44068320 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:56.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fab600a9990 0x7fab60144b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:56.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fab60154070 0x7fab600a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:56.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 --2- 172.21.15.107:0/4105308583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab60147ef0 0x7fab601482d0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:56.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 -- 172.21.15.107:0/4105308583 >> 172.21.15.107:0/4105308583 conn(0x7fab600c39a0 msgr2=0x7fab6006c270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:56.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 -- 172.21.15.107:0/4105308583 shutdown_connections 2023-10-25T00:03:56.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:56.666+0000 7fab680e3640 1 -- 172.21.15.107:0/4105308583 wait complete. 2023-10-25T00:03:56.687 INFO:tasks.mds_thrash.fs.[b]:no change 2023-10-25T00:03:58.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:03:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d3139640 1 Processor -- start 2023-10-25T00:03:58.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d3139640 1 -- start start 2023-10-25T00:03:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d3139640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84cc0ac020 0x7f84cc0c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d3139640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f84cc0c2520 0x7f84cc06c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d3139640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84cc066000 con 0x7f84cc0aa140 2023-10-25T00:03:58.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d3139640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84cc0c0f40 con 0x7f84cc0c2520 2023-10-25T00:03:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d3139640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc0c1220 con 0x7f84cc0ac020 2023-10-25T00:03:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d16d6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f84cc0c2520 0x7f84cc06c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d0ed5640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f84cc0aa140 0x7f84cc0aa520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:56014/0 (socket says 172.21.15.107:56014) 2023-10-25T00:03:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d0ed5640 1 -- 172.21.15.107:0/1961098331 learned_addr learned my addr 172.21.15.107:0/1961098331 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:03:58.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84cbfff640 1 --2- 172.21.15.107:0/1961098331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84cc0ac020 0x7f84cc0c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d16d6640 1 -- 172.21.15.107:0/1961098331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84cc0ac020 msgr2=0x7f84cc0c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:03:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d16d6640 1 --2- 172.21.15.107:0/1961098331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84cc0ac020 0x7f84cc0c1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d16d6640 1 -- 172.21.15.107:0/1961098331 >> v1:172.21.15.110:6789/0 conn(0x7f84cc0aa140 legacy=0x7f84cc0aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:58.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.822+0000 7f84d16d6640 1 -- 172.21.15.107:0/1961098331 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc0c1500 con 0x7f84cc0c2520 2023-10-25T00:03:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84d16d6640 1 --2- 172.21.15.107:0/1961098331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84cc0c2520 0x7f84cc06c490 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009d90 tx=0x7f84c401c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3402b1df69cd81c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:58.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84cb7fe640 1 -- 172.21.15.107:0/1961098331 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c40a59e0 con 0x7f84cc0c2520 2023-10-25T00:03:58.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84cb7fe640 1 -- 172.21.15.107:0/1961098331 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c40a5b40 con 0x7f84cc0c2520 2023-10-25T00:03:58.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84cb7fe640 1 -- 172.21.15.107:0/1961098331 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c40ad440 con 0x7f84cc0c2520 2023-10-25T00:03:58.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84d3139640 1 -- 172.21.15.107:0/1961098331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84cc0c2520 msgr2=0x7f84cc06c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:58.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84d3139640 1 --2- 172.21.15.107:0/1961098331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84cc0c2520 0x7f84cc06c490 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009d90 tx=0x7f84c401c920 comp rx=0 tx=0).stop 2023-10-25T00:03:58.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84d3139640 1 -- 172.21.15.107:0/1961098331 shutdown_connections 2023-10-25T00:03:58.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84d3139640 1 --2- 172.21.15.107:0/1961098331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84cc0c2520 0x7f84cc06c490 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:58.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84d3139640 1 --2- 172.21.15.107:0/1961098331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84cc0ac020 0x7f84cc0c1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:58.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84d3139640 1 -- 172.21.15.107:0/1961098331 >> 172.21.15.107:0/1961098331 conn(0x7f84cc0c65c0 msgr2=0x7f84cc0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:58.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84d3139640 1 -- 172.21.15.107:0/1961098331 shutdown_connections 2023-10-25T00:03:58.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.826+0000 7f84d3139640 1 -- 172.21.15.107:0/1961098331 wait complete. 2023-10-25T00:03:58.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d3139640 1 Processor -- start 2023-10-25T00:03:58.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d3139640 1 -- start start 2023-10-25T00:03:58.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d3139640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84cc0aa140 0x7f84cc1505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:58.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d3139640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84cc0ac020 0x7f84cc152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:58.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d3139640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84cc0c2520 0x7f84cc14ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:58.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d3139640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc1514c0 con 0x7f84cc0aa140 2023-10-25T00:03:58.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d3139640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc0c7310 con 0x7f84cc0ac020 2023-10-25T00:03:58.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d3139640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84cc0720f0 con 0x7f84cc0c2520 2023-10-25T00:03:58.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d0ed5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84cc0aa140 0x7f84cc1505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:58.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d0ed5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84cc0aa140 0x7f84cc1505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50458/0 (socket says 172.21.15.107:50458) 2023-10-25T00:03:58.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d0ed5640 1 -- 172.21.15.107:0/1941072664 learned_addr learned my addr 172.21.15.107:0/1941072664 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:03:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d16d6640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84cc0c2520 0x7f84cc14ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84cbfff640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84cc0ac020 0x7f84cc152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d0ed5640 1 -- 172.21.15.107:0/1941072664 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84cc0c2520 msgr2=0x7f84cc14ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d0ed5640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84cc0c2520 0x7f84cc14ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:58.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d0ed5640 1 -- 172.21.15.107:0/1941072664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84cc0ac020 msgr2=0x7f84cc152d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d0ed5640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84cc0ac020 0x7f84cc152d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d0ed5640 1 -- 172.21.15.107:0/1941072664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c4009a70 con 0x7f84cc0aa140 2023-10-25T00:03:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84cbfff640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84cc0ac020 0x7f84cc152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:03:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d0ed5640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84cc0aa140 0x7f84cc1505f0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f84c000c400 tx=0x7f84c000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84c97fa640 1 -- 172.21.15.107:0/1941072664 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0005cf0 con 0x7f84cc0aa140 2023-10-25T00:03:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84c97fa640 1 -- 172.21.15.107:0/1941072664 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c0005e50 con 0x7f84cc0aa140 2023-10-25T00:03:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d3139640 1 -- 172.21.15.107:0/1941072664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84cc150e00 con 0x7f84cc0aa140 2023-10-25T00:03:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84c97fa640 1 -- 172.21.15.107:0/1941072664 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c00072d0 con 0x7f84cc0aa140 2023-10-25T00:03:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d3139640 1 -- 172.21.15.107:0/1941072664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84cc153270 con 0x7f84cc0aa140 2023-10-25T00:03:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.830+0000 7f84d3139640 1 -- 172.21.15.107:0/1941072664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84cc14e990 con 0x7f84cc0aa140 2023-10-25T00:03:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.834+0000 7f84c97fa640 1 -- 172.21.15.107:0/1941072664 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f84c0007430 con 0x7f84cc0aa140 2023-10-25T00:03:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.834+0000 7f84c97fa640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f84b0065e60 0x7f84b0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:03:58.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.834+0000 7f84c97fa640 1 -- 172.21.15.107:0/1941072664 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f84c0083e00 con 0x7f84cc0aa140 2023-10-25T00:03:58.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.834+0000 7f84cbfff640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f84b0065e60 0x7f84b0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:03:58.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.834+0000 7f84cbfff640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f84b0065e60 0x7f84b0068320 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f84cc14e710 tx=0x7f84b8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:03:58.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:58.838+0000 7f84c97fa640 1 -- 172.21.15.107:0/1941072664 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f84c0050c80 con 0x7f84cc0aa140 2023-10-25T00:03:59.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.050+0000 7f84d3139640 1 -- 172.21.15.107:0/1941072664 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84cc0746e0 con 0x7f84cc0aa140 2023-10-25T00:03:59.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.054+0000 7f84c97fa640 1 -- 172.21.15.107:0/1941072664 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5016 (secure 0 0 0) 0x7f84c0021090 con 0x7f84cc0aa140 2023-10-25T00:03:59.056 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:03:59.056 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:03:59.056 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 94 2023-10-25T00:03:59.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 -- 172.21.15.107:0/1941072664 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f84b0065e60 msgr2=0x7f84b0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:59.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f84b0065e60 0x7f84b0068320 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f84cc14e710 tx=0x7f84b8008040 comp rx=0 tx=0).stop 2023-10-25T00:03:59.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 -- 172.21.15.107:0/1941072664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84cc0aa140 msgr2=0x7f84cc1505f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:03:59.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84cc0aa140 0x7f84cc1505f0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f84c000c400 tx=0x7f84c000c8c0 comp rx=0 tx=0).stop 2023-10-25T00:03:59.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 -- 172.21.15.107:0/1941072664 shutdown_connections 2023-10-25T00:03:59.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f84b0065e60 0x7f84b0068320 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:59.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f84cc0c2520 0x7f84cc14ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:59.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84cc0ac020 0x7f84cc152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:59.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 --2- 172.21.15.107:0/1941072664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84cc0aa140 0x7f84cc1505f0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:03:59.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 -- 172.21.15.107:0/1941072664 >> 172.21.15.107:0/1941072664 conn(0x7f84cc0c65c0 msgr2=0x7f84cc0c55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:03:59.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 -- 172.21.15.107:0/1941072664 shutdown_connections 2023-10-25T00:03:59.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:03:59.058+0000 7f84d3139640 1 -- 172.21.15.107:0/1941072664 wait complete. 2023-10-25T00:03:59.084 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=6483 2023-10-25T00:03:59.084 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.a 2023-10-25T00:03:59.741 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:03:59.738+0000 7f6240cdc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:04:06.085 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2023-10-25T00:04:06.086 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:04:06.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:04:06.089 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:04:06.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:06.161 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:04:06.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.218+0000 7f75e3076640 1 Processor -- start 2023-10-25T00:04:06.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.218+0000 7f75e3076640 1 -- start start 2023-10-25T00:04:06.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e3076640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f75dc14fbd0 0x7f75dc06df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:06.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e3076640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 0x7f75dc0c1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:06.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e3076640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc155f20 con 0x7f75dc0c4e20 2023-10-25T00:04:06.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e3076640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75dc0c6d40 con 0x7f75dc14fbd0 2023-10-25T00:04:06.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e3076640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc153560 con 0x7f75dc0c5ba0 2023-10-25T00:04:06.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e1613640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 0x7f75dc0c1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:06.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e1613640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 0x7f75dc0c1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:34320/0 (socket says 172.21.15.107:34320) 2023-10-25T00:04:06.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e0e12640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f75dc14fbd0 0x7f75dc06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:06.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e1613640 1 -- 172.21.15.107:0/4147035262 learned_addr learned my addr 172.21.15.107:0/4147035262 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:06.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e0e12640 1 -- 172.21.15.107:0/4147035262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 msgr2=0x7f75dc0c1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:06.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e0e12640 1 --2- 172.21.15.107:0/4147035262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 0x7f75dc0c1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:06.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e0e12640 1 -- 172.21.15.107:0/4147035262 >> v1:172.21.15.110:6789/0 conn(0x7f75dc0c4e20 legacy=0x7f75dc0c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e0e12640 1 -- 172.21.15.107:0/4147035262 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc1527a0 con 0x7f75dc14fbd0 2023-10-25T00:04:06.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.222+0000 7f75e1613640 1 --2- 172.21.15.107:0/4147035262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 0x7f75dc0c1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:04:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e0e12640 1 --2- 172.21.15.107:0/4147035262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc14fbd0 0x7f75dc06df30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f75d00097c0 tx=0x7f75d001c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f658c52f2293db69 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75db7fe640 1 -- 172.21.15.107:0/4147035262 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d00ab600 con 0x7f75dc14fbd0 2023-10-25T00:04:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75db7fe640 1 -- 172.21.15.107:0/4147035262 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d00a7050 con 0x7f75dc14fbd0 2023-10-25T00:04:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75db7fe640 1 -- 172.21.15.107:0/4147035262 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d00ab980 con 0x7f75dc14fbd0 2023-10-25T00:04:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 -- 172.21.15.107:0/4147035262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc14fbd0 msgr2=0x7f75dc06df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 --2- 172.21.15.107:0/4147035262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc14fbd0 0x7f75dc06df30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f75d00097c0 tx=0x7f75d001c3f0 comp rx=0 tx=0).stop 2023-10-25T00:04:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 -- 172.21.15.107:0/4147035262 shutdown_connections 2023-10-25T00:04:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 --2- 172.21.15.107:0/4147035262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 0x7f75dc0c1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 --2- 172.21.15.107:0/4147035262 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc14fbd0 0x7f75dc06df30 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 -- 172.21.15.107:0/4147035262 >> 172.21.15.107:0/4147035262 conn(0x7f75dc0c39a0 msgr2=0x7f75dc0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:06.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 -- 172.21.15.107:0/4147035262 shutdown_connections 2023-10-25T00:04:06.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 -- 172.21.15.107:0/4147035262 wait complete. 2023-10-25T00:04:06.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 Processor -- start 2023-10-25T00:04:06.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 -- start start 2023-10-25T00:04:06.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc0c4e20 0x7f75dc150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:06.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 0x7f75dc149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:06.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75dc14fbd0 0x7f75dc149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:06.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc149ac0 con 0x7f75dc14fbd0 2023-10-25T00:04:06.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc152120 con 0x7f75dc0c4e20 2023-10-25T00:04:06.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e3076640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc1eebf0 con 0x7f75dc0c5ba0 2023-10-25T00:04:06.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e1613640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75dc14fbd0 0x7f75dc149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:06.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e1613640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75dc14fbd0 0x7f75dc149580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54424/0 (socket says 172.21.15.107:54424) 2023-10-25T00:04:06.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e1613640 1 -- 172.21.15.107:0/2069302801 learned_addr learned my addr 172.21.15.107:0/2069302801 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:06.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e0e12640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc0c4e20 0x7f75dc150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:06.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75dbfff640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 0x7f75dc149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:06.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e1613640 1 -- 172.21.15.107:0/2069302801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 msgr2=0x7f75dc149040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:06.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e1613640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 0x7f75dc149040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:06.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e1613640 1 -- 172.21.15.107:0/2069302801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc0c4e20 msgr2=0x7f75dc150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:06.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e1613640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc0c4e20 0x7f75dc150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:06.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e1613640 1 -- 172.21.15.107:0/2069302801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d4009b30 con 0x7f75dc14fbd0 2023-10-25T00:04:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75e1613640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75dc14fbd0 0x7f75dc149580 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f75d400efc0 tx=0x7f75d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.226+0000 7f75d97fa640 1 -- 172.21.15.107:0/2069302801 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d4018070 con 0x7f75dc14fbd0 2023-10-25T00:04:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.230+0000 7f75d97fa640 1 -- 172.21.15.107:0/2069302801 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d400f040 con 0x7f75dc14fbd0 2023-10-25T00:04:06.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.230+0000 7f75d97fa640 1 -- 172.21.15.107:0/2069302801 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d400ce00 con 0x7f75dc14fbd0 2023-10-25T00:04:06.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.230+0000 7f75e3076640 1 -- 172.21.15.107:0/2069302801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d0009420 con 0x7f75dc14fbd0 2023-10-25T00:04:06.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.230+0000 7f75e3076640 1 -- 172.21.15.107:0/2069302801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75dc147990 con 0x7f75dc14fbd0 2023-10-25T00:04:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.230+0000 7f75e3076640 1 -- 172.21.15.107:0/2069302801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75dc0746e0 con 0x7f75dc14fbd0 2023-10-25T00:04:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.230+0000 7f75d97fa640 1 -- 172.21.15.107:0/2069302801 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f75d4014480 con 0x7f75dc14fbd0 2023-10-25T00:04:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.230+0000 7f75d97fa640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75c0065e90 0x7f75c0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:06.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.230+0000 7f75d97fa640 1 -- 172.21.15.107:0/2069302801 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f75d40843b0 con 0x7f75dc14fbd0 2023-10-25T00:04:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.230+0000 7f75e0e12640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75c0065e90 0x7f75c0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.230+0000 7f75e0e12640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75c0065e90 0x7f75c0068350 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f75d00ad000 tx=0x7f75d00a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:06.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.238+0000 7f75d97fa640 1 -- 172.21.15.107:0/2069302801 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f75d4051100 con 0x7f75dc14fbd0 2023-10-25T00:04:06.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.454+0000 7f75e3076640 1 -- 172.21.15.107:0/2069302801 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75dc153010 con 0x7f75dc14fbd0 2023-10-25T00:04:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.454+0000 7f75d97fa640 1 -- 172.21.15.107:0/2069302801 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5016 (secure 0 0 0) 0x7f75d4054720 con 0x7f75dc14fbd0 2023-10-25T00:04:06.458 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:06.458 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:06.459 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 94 2023-10-25T00:04:06.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 -- 172.21.15.107:0/2069302801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75c0065e90 msgr2=0x7f75c0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:06.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75c0065e90 0x7f75c0068350 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f75d00ad000 tx=0x7f75d00a6000 comp rx=0 tx=0).stop 2023-10-25T00:04:06.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 -- 172.21.15.107:0/2069302801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75dc14fbd0 msgr2=0x7f75dc149580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:06.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75dc14fbd0 0x7f75dc149580 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f75d400efc0 tx=0x7f75d400c490 comp rx=0 tx=0).stop 2023-10-25T00:04:06.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 -- 172.21.15.107:0/2069302801 shutdown_connections 2023-10-25T00:04:06.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75dc14fbd0 0x7f75dc149580 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:06.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f75c0065e90 0x7f75c0068350 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:06.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75dc0c5ba0 0x7f75dc149040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:06.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 --2- 172.21.15.107:0/2069302801 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc0c4e20 0x7f75dc150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:06.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 -- 172.21.15.107:0/2069302801 >> 172.21.15.107:0/2069302801 conn(0x7f75dc0c39a0 msgr2=0x7f75dc15c670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:06.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 -- 172.21.15.107:0/2069302801 shutdown_connections 2023-10-25T00:04:06.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:06.458+0000 7f75e3076640 1 -- 172.21.15.107:0/2069302801 wait complete. 2023-10-25T00:04:06.478 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-25T00:04:08.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.610+0000 7fd58ec9f640 1 Processor -- start 2023-10-25T00:04:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.610+0000 7fd58ec9f640 1 -- start start 2023-10-25T00:04:08.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd588152020 0x7fd5880c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:08.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5881553d0 0x7fd58806ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:08.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd588152400 con 0x7fd5880c8640 2023-10-25T00:04:08.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5881558d0 con 0x7fd5881553d0 2023-10-25T00:04:08.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5880c6ce0 con 0x7fd588152020 2023-10-25T00:04:08.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd587fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd5880c8640 0x7fd58806a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41886/0 (socket says 172.21.15.107:41886) 2023-10-25T00:04:08.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd587fff640 1 -- 172.21.15.107:0/368136683 learned_addr learned my addr 172.21.15.107:0/368136683 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:08.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58d23c640 1 --2- 172.21.15.107:0/368136683 >> v2:172.21.15.107:3300/0 conn(0x7fd5881553d0 0x7fd58806ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:08.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ca3b640 1 --2- 172.21.15.107:0/368136683 >> v2:172.21.15.110:3300/0 conn(0x7fd588152020 0x7fd5880c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:08.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd5877fe640 1 -- 172.21.15.107:0/368136683 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273448705 0 0) 0x7fd588152400 con 0x7fd5880c8640 2023-10-25T00:04:08.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd5877fe640 1 -- 172.21.15.107:0/368136683 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd578003530 con 0x7fd5880c8640 2023-10-25T00:04:08.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58d23c640 1 -- 172.21.15.107:0/368136683 >> v2:172.21.15.110:3300/0 conn(0x7fd588152020 msgr2=0x7fd5880c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:08.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58d23c640 1 --2- 172.21.15.107:0/368136683 >> v2:172.21.15.110:3300/0 conn(0x7fd588152020 0x7fd5880c4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:08.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58d23c640 1 -- 172.21.15.107:0/368136683 >> v1:172.21.15.107:6789/0 conn(0x7fd5880c8640 legacy=0x7fd58806a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:08.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58d23c640 1 -- 172.21.15.107:0/368136683 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5880c6fc0 con 0x7fd5881553d0 2023-10-25T00:04:08.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ca3b640 1 --2- 172.21.15.107:0/368136683 >> v2:172.21.15.110:3300/0 conn(0x7fd588152020 0x7fd5880c4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:04:08.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58d23c640 1 --2- 172.21.15.107:0/368136683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5881553d0 0x7fd58806ae20 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd5880cb2e0 tx=0x7fd58001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41ad65f4b78a2a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:08.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd5877fe640 1 -- 172.21.15.107:0/368136683 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5800a5930 con 0x7fd5881553d0 2023-10-25T00:04:08.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd5877fe640 1 -- 172.21.15.107:0/368136683 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5800a5a90 con 0x7fd5881553d0 2023-10-25T00:04:08.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd5877fe640 1 -- 172.21.15.107:0/368136683 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5800a5e20 con 0x7fd5881553d0 2023-10-25T00:04:08.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 -- 172.21.15.107:0/368136683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5881553d0 msgr2=0x7fd58806ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:08.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 --2- 172.21.15.107:0/368136683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5881553d0 0x7fd58806ae20 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd5880cb2e0 tx=0x7fd58001c850 comp rx=0 tx=0).stop 2023-10-25T00:04:08.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 -- 172.21.15.107:0/368136683 shutdown_connections 2023-10-25T00:04:08.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 --2- 172.21.15.107:0/368136683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5881553d0 0x7fd58806ae20 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:08.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 --2- 172.21.15.107:0/368136683 >> v2:172.21.15.110:3300/0 conn(0x7fd588152020 0x7fd5880c4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:08.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 -- 172.21.15.107:0/368136683 >> 172.21.15.107:0/368136683 conn(0x7fd5880abd60 msgr2=0x7fd5880ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:08.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 -- 172.21.15.107:0/368136683 shutdown_connections 2023-10-25T00:04:08.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.614+0000 7fd58ec9f640 1 -- 172.21.15.107:0/368136683 wait complete. 2023-10-25T00:04:08.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ec9f640 1 Processor -- start 2023-10-25T00:04:08.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ec9f640 1 -- start start 2023-10-25T00:04:08.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ec9f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5880c8640 0x7fd5881f05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:08.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ec9f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd588152020 0x7fd5881f0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:08.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ec9f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5881553d0 0x7fd5881461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:08.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ec9f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd588150770 con 0x7fd5880c8640 2023-10-25T00:04:08.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ec9f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5880c18c0 con 0x7fd588152020 2023-10-25T00:04:08.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ec9f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5881f1030 con 0x7fd5881553d0 2023-10-25T00:04:08.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ca3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5880c8640 0x7fd5881f05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:08.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ca3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5880c8640 0x7fd5881f05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54452/0 (socket says 172.21.15.107:54452) 2023-10-25T00:04:08.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ca3b640 1 -- 172.21.15.107:0/2607948152 learned_addr learned my addr 172.21.15.107:0/2607948152 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd587fff640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd588152020 0x7fd5881f0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58d23c640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5881553d0 0x7fd5881461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ca3b640 1 -- 172.21.15.107:0/2607948152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5881553d0 msgr2=0x7fd5881461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ca3b640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5881553d0 0x7fd5881461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:08.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ca3b640 1 -- 172.21.15.107:0/2607948152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd588152020 msgr2=0x7fd5881f0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:08.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ca3b640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd588152020 0x7fd5881f0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:08.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ca3b640 1 -- 172.21.15.107:0/2607948152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd580009a70 con 0x7fd5880c8640 2023-10-25T00:04:08.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ca3b640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5880c8640 0x7fd5881f05b0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00e810 tx=0x7fd57c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:08.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd5857fa640 1 -- 172.21.15.107:0/2607948152 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c00c8d0 con 0x7fd5880c8640 2023-10-25T00:04:08.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd5857fa640 1 -- 172.21.15.107:0/2607948152 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c00ca30 con 0x7fd5880c8640 2023-10-25T00:04:08.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ec9f640 1 -- 172.21.15.107:0/2607948152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5880c3640 con 0x7fd5880c8640 2023-10-25T00:04:08.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd5857fa640 1 -- 172.21.15.107:0/2607948152 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c00cdc0 con 0x7fd5880c8640 2023-10-25T00:04:08.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.618+0000 7fd58ec9f640 1 -- 172.21.15.107:0/2607948152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58814bde0 con 0x7fd5880c8640 2023-10-25T00:04:08.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.622+0000 7fd58ec9f640 1 -- 172.21.15.107:0/2607948152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5880746e0 con 0x7fd5880c8640 2023-10-25T00:04:08.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.622+0000 7fd5857fa640 1 -- 172.21.15.107:0/2607948152 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd57c013410 con 0x7fd5880c8640 2023-10-25T00:04:08.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.622+0000 7fd5857fa640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd56c065e60 0x7fd56c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:08.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.622+0000 7fd5857fa640 1 -- 172.21.15.107:0/2607948152 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7fd57c083f30 con 0x7fd5880c8640 2023-10-25T00:04:08.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.622+0000 7fd587fff640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd56c065e60 0x7fd56c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:08.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.622+0000 7fd587fff640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd56c065e60 0x7fd56c068320 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd5740011f0 tx=0x7fd574009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:08.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.626+0000 7fd5857fa640 1 -- 172.21.15.107:0/2607948152 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd57c050c80 con 0x7fd5880c8640 2023-10-25T00:04:08.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.846+0000 7fd58ec9f640 1 -- 172.21.15.107:0/2607948152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5880c16a0 con 0x7fd5880c8640 2023-10-25T00:04:08.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd5857fa640 1 -- 172.21.15.107:0/2607948152 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7fd57c021090 con 0x7fd5880c8640 2023-10-25T00:04:08.851 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:08.851 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:08.851 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 -- 172.21.15.107:0/2607948152 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd56c065e60 msgr2=0x7fd56c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd56c065e60 0x7fd56c068320 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd5740011f0 tx=0x7fd574009040 comp rx=0 tx=0).stop 2023-10-25T00:04:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 -- 172.21.15.107:0/2607948152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5880c8640 msgr2=0x7fd5881f05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:08.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5880c8640 0x7fd5881f05b0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00e810 tx=0x7fd57c00ecd0 comp rx=0 tx=0).stop 2023-10-25T00:04:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 -- 172.21.15.107:0/2607948152 shutdown_connections 2023-10-25T00:04:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd56c065e60 0x7fd56c068320 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5881553d0 0x7fd5881461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd588152020 0x7fd5881f0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 --2- 172.21.15.107:0/2607948152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5880c8640 0x7fd5881f05b0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 -- 172.21.15.107:0/2607948152 >> 172.21.15.107:0/2607948152 conn(0x7fd5880abd60 msgr2=0x7fd58815e500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:08.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 -- 172.21.15.107:0/2607948152 shutdown_connections 2023-10-25T00:04:08.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:08.850+0000 7fd58ec9f640 1 -- 172.21.15.107:0/2607948152 wait complete. 2023-10-25T00:04:08.873 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2023-10-25T00:04:08.873 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before thrashing 2023-10-25T00:04:11.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:12.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb02846640 1 Processor -- start 2023-10-25T00:04:12.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb02846640 1 -- start start 2023-10-25T00:04:12.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb02846640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbafc0c8100 0x7fbafc0abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb02846640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc06a4b0 0x7fbafc0ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb02846640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc155680 con 0x7fbafc152040 2023-10-25T00:04:12.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb02846640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbafc154a70 con 0x7fbafc0c8100 2023-10-25T00:04:12.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb02846640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc154d50 con 0x7fbafc06a4b0 2023-10-25T00:04:12.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafbfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbafc152040 0x7fbafc0c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41894/0 (socket says 172.21.15.107:41894) 2023-10-25T00:04:12.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafbfff640 1 -- 172.21.15.107:0/1605175487 learned_addr learned my addr 172.21.15.107:0/1605175487 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:12.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb00de3640 1 --2- 172.21.15.107:0/1605175487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc06a4b0 0x7fbafc0ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506630336 0 0) 0x7fbafc155680 con 0x7fbafc152040 2023-10-25T00:04:12.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbaf3fff640 1 --2- 172.21.15.107:0/1605175487 >> v2:172.21.15.110:3300/0 conn(0x7fbafc0c8100 0x7fbafc0abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae8003530 con 0x7fbafc152040 2023-10-25T00:04:12.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3562099958 0 0) 0x7fbae8003530 con 0x7fbafc152040 2023-10-25T00:04:12.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc06a4b0 msgr2=0x7fbafc0ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:12.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 --2- 172.21.15.107:0/1605175487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc06a4b0 0x7fbafc0ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 >> v2:172.21.15.110:3300/0 conn(0x7fbafc0c8100 msgr2=0x7fbafc0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:12.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 --2- 172.21.15.107:0/1605175487 >> v2:172.21.15.110:3300/0 conn(0x7fbafc0c8100 0x7fbafc0abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc155030 con 0x7fbafc152040 2023-10-25T00:04:12.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fbaec001680 con 0x7fbafc152040 2023-10-25T00:04:12.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 >> v1:172.21.15.107:6789/0 conn(0x7fbafc152040 legacy=0x7fbafc0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 --2- 172.21.15.107:0/1605175487 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8003980 0x7fbae8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 --2- 172.21.15.107:0/1605175487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc06a4b0 0x7fbae80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 --2- 172.21.15.107:0/1605175487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbae8004d10 0x7fbae8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc154a70 con 0x7fbae8004d10 2023-10-25T00:04:12.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc154d50 con 0x7fbae8003980 2023-10-25T00:04:12.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae800b9c0 con 0x7fbafc06a4b0 2023-10-25T00:04:12.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb00de3640 1 --2- 172.21.15.107:0/1605175487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbae8004d10 0x7fbae8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbafbfff640 1 --2- 172.21.15.107:0/1605175487 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8003980 0x7fbae8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb00de3640 1 -- 172.21.15.107:0/1605175487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc06a4b0 msgr2=0x7fbae80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:12.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb00de3640 1 --2- 172.21.15.107:0/1605175487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc06a4b0 0x7fbae80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb00de3640 1 -- 172.21.15.107:0/1605175487 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8003980 msgr2=0x7fbae8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:12.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb00de3640 1 --2- 172.21.15.107:0/1605175487 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8003980 0x7fbae8003da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.030+0000 7fbb00de3640 1 -- 172.21.15.107:0/1605175487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae8003530 con 0x7fbae8004d10 2023-10-25T00:04:12.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbb00de3640 1 --2- 172.21.15.107:0/1605175487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbae8004d10 0x7fbae8005190 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4009fc0 tx=0x7fbaf401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780eef1ad8adb233 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:12.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40a5ab0 con 0x7fbae8004d10 2023-10-25T00:04:12.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf400f040 con 0x7fbae8004d10 2023-10-25T00:04:12.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbafb7fe640 1 -- 172.21.15.107:0/1605175487 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40a5e50 con 0x7fbae8004d10 2023-10-25T00:04:12.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbb02846640 1 -- 172.21.15.107:0/1605175487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbae8004d10 msgr2=0x7fbae8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbb02846640 1 --2- 172.21.15.107:0/1605175487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbae8004d10 0x7fbae8005190 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4009fc0 tx=0x7fbaf401cca0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbb00de3640 1 -- 172.21.15.107:0/1605175487 reap_dead start 2023-10-25T00:04:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbb02846640 1 -- 172.21.15.107:0/1605175487 shutdown_connections 2023-10-25T00:04:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbb02846640 1 -- 172.21.15.107:0/1605175487 >> 172.21.15.107:0/1605175487 conn(0x7fbafc0ca200 msgr2=0x7fbafc0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbb02846640 1 -- 172.21.15.107:0/1605175487 shutdown_connections 2023-10-25T00:04:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbb02846640 1 -- 172.21.15.107:0/1605175487 wait complete. 2023-10-25T00:04:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbb02846640 1 Processor -- start 2023-10-25T00:04:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbb02846640 1 -- start start 2023-10-25T00:04:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.034+0000 7fbb02846640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8004d10 0x7fbafc0ac5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb02846640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc1ecc30 0x7fbafc1df600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb02846640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbafc1e9db0 0x7fbafc1ea230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb02846640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc0ab020 con 0x7fbafc1e9db0 2023-10-25T00:04:12.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb02846640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1e9680 con 0x7fbae8004d10 2023-10-25T00:04:12.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb02846640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbafc1e9920 con 0x7fbafc1ecc30 2023-10-25T00:04:12.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb00de3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbafc1e9db0 0x7fbafc1ea230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb00de3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbafc1e9db0 0x7fbafc1ea230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54484/0 (socket says 172.21.15.107:54484) 2023-10-25T00:04:12.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb00de3640 1 -- 172.21.15.107:0/1551655354 learned_addr learned my addr 172.21.15.107:0/1551655354 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:12.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbaf3fff640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc1ecc30 0x7fbafc1df600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbafbfff640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8004d10 0x7fbafc0ac5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb00de3640 1 -- 172.21.15.107:0/1551655354 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc1ecc30 msgr2=0x7fbafc1df600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb00de3640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc1ecc30 0x7fbafc1df600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb00de3640 1 -- 172.21.15.107:0/1551655354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8004d10 msgr2=0x7fbafc0ac5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb00de3640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8004d10 0x7fbafc0ac5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb00de3640 1 -- 172.21.15.107:0/1551655354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0aa6b0 con 0x7fbafc1e9db0 2023-10-25T00:04:12.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb00de3640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbafc1e9db0 0x7fbafc1ea230 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4002240 tx=0x7fbaf4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:12.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbaf97fa640 1 -- 172.21.15.107:0/1551655354 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40039a0 con 0x7fbafc1e9db0 2023-10-25T00:04:12.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbaf97fa640 1 -- 172.21.15.107:0/1551655354 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf4003b00 con 0x7fbafc1e9db0 2023-10-25T00:04:12.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb02846640 1 -- 172.21.15.107:0/1551655354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc067110 con 0x7fbafc1e9db0 2023-10-25T00:04:12.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbaf97fa640 1 -- 172.21.15.107:0/1551655354 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40ad710 con 0x7fbafc1e9db0 2023-10-25T00:04:12.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb02846640 1 -- 172.21.15.107:0/1551655354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc0aa0f0 con 0x7fbafc1e9db0 2023-10-25T00:04:12.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbb02846640 1 -- 172.21.15.107:0/1551655354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbafc0746e0 con 0x7fbafc1e9db0 2023-10-25T00:04:12.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbaf97fa640 1 -- 172.21.15.107:0/1551655354 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbaf40b4040 con 0x7fbafc1e9db0 2023-10-25T00:04:12.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbaf97fa640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbadc065e90 0x7fbadc068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbaf97fa640 1 -- 172.21.15.107:0/1551655354 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7fbaf411dc80 con 0x7fbafc1e9db0 2023-10-25T00:04:12.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbafbfff640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbadc065e90 0x7fbadc068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.038+0000 7fbafbfff640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbadc065e90 0x7fbadc068350 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbaec004880 tx=0x7fbaec001730 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:12.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.042+0000 7fbaf97fa640 1 -- 172.21.15.107:0/1551655354 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbaf40eaa80 con 0x7fbafc1e9db0 2023-10-25T00:04:12.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 -- 172.21.15.107:0/1551655354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbafc1e9a60 con 0x7fbafc1e9db0 2023-10-25T00:04:12.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbaf97fa640 1 -- 172.21.15.107:0/1551655354 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7fbaf40bc030 con 0x7fbafc1e9db0 2023-10-25T00:04:12.252 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:12.252 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:12.253 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 -- 172.21.15.107:0/1551655354 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbadc065e90 msgr2=0x7fbadc068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbadc065e90 0x7fbadc068350 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fbaec004880 tx=0x7fbaec001730 comp rx=0 tx=0).stop 2023-10-25T00:04:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 -- 172.21.15.107:0/1551655354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbafc1e9db0 msgr2=0x7fbafc1ea230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbafc1e9db0 0x7fbafc1ea230 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4002240 tx=0x7fbaf4009750 comp rx=0 tx=0).stop 2023-10-25T00:04:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 -- 172.21.15.107:0/1551655354 shutdown_connections 2023-10-25T00:04:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbafc1e9db0 0x7fbafc1ea230 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbadc065e90 0x7fbadc068350 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbafc1ecc30 0x7fbafc1df600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 --2- 172.21.15.107:0/1551655354 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbae8004d10 0x7fbafc0ac5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 -- 172.21.15.107:0/1551655354 >> 172.21.15.107:0/1551655354 conn(0x7fbafc0ca200 msgr2=0x7fbafc06a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 -- 172.21.15.107:0/1551655354 shutdown_connections 2023-10-25T00:04:12.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.250+0000 7fbb02846640 1 -- 172.21.15.107:0/1551655354 wait complete. 2023-10-25T00:04:12.271 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2023-10-25T00:04:12.271 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:04:12.272 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:04:12.286 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:04:12.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:12.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.458+0000 7fd89dc8d640 1 Processor -- start 2023-10-25T00:04:12.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.458+0000 7fd89dc8d640 1 -- start start 2023-10-25T00:04:12.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd89dc8d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd898152040 0x7fd898158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd89dc8d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8980adb80 0x7fd898071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd89dc8d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd898152420 con 0x7fd898071fc0 2023-10-25T00:04:12.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd89dc8d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8981556e0 con 0x7fd898152040 2023-10-25T00:04:12.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd89dc8d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd898154d40 con 0x7fd8980adb80 2023-10-25T00:04:12.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd8977fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd898152040 0x7fd898158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd8977fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd898152040 0x7fd898158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54490/0 (socket says 172.21.15.107:54490) 2023-10-25T00:04:12.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd8977fe640 1 -- 172.21.15.107:0/346274906 learned_addr learned my addr 172.21.15.107:0/346274906 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd896ffd640 1 --2- 172.21.15.107:0/346274906 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8980adb80 0x7fd898071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd8977fe640 1 -- 172.21.15.107:0/346274906 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8980adb80 msgr2=0x7fd898071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd8977fe640 1 --2- 172.21.15.107:0/346274906 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8980adb80 0x7fd898071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd8977fe640 1 -- 172.21.15.107:0/346274906 >> v1:172.21.15.107:6789/0 conn(0x7fd898071fc0 legacy=0x7fd89806e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd8977fe640 1 -- 172.21.15.107:0/346274906 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd898155020 con 0x7fd898152040 2023-10-25T00:04:12.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd8977fe640 1 --2- 172.21.15.107:0/346274906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd898152040 0x7fd898158940 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fd888009870 tx=0x7fd88801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b118dbe769b9ca44 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:12.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd8967fc640 1 -- 172.21.15.107:0/346274906 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8880ab600 con 0x7fd898152040 2023-10-25T00:04:12.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd8967fc640 1 -- 172.21.15.107:0/346274906 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8880a7050 con 0x7fd898152040 2023-10-25T00:04:12.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.462+0000 7fd8967fc640 1 -- 172.21.15.107:0/346274906 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8880ab980 con 0x7fd898152040 2023-10-25T00:04:12.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 -- 172.21.15.107:0/346274906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd898152040 msgr2=0x7fd898158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 --2- 172.21.15.107:0/346274906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd898152040 0x7fd898158940 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fd888009870 tx=0x7fd88801c490 comp rx=0 tx=0).stop 2023-10-25T00:04:12.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 -- 172.21.15.107:0/346274906 shutdown_connections 2023-10-25T00:04:12.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 --2- 172.21.15.107:0/346274906 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8980adb80 0x7fd898071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 --2- 172.21.15.107:0/346274906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd898152040 0x7fd898158940 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 -- 172.21.15.107:0/346274906 >> 172.21.15.107:0/346274906 conn(0x7fd898070f60 msgr2=0x7fd898071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 -- 172.21.15.107:0/346274906 shutdown_connections 2023-10-25T00:04:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 -- 172.21.15.107:0/346274906 wait complete. 2023-10-25T00:04:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 Processor -- start 2023-10-25T00:04:12.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 -- start start 2023-10-25T00:04:12.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd898071fc0 0x7fd8981f06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8980adb80 0x7fd8981ec770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd898152040 0x7fd898150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8981f02d0 con 0x7fd8980adb80 2023-10-25T00:04:12.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8981efce0 con 0x7fd898152040 2023-10-25T00:04:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8981f0c30 con 0x7fd898071fc0 2023-10-25T00:04:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd896ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8980adb80 0x7fd8981ec770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd896ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8980adb80 0x7fd8981ec770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54492/0 (socket says 172.21.15.107:54492) 2023-10-25T00:04:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd896ffd640 1 -- 172.21.15.107:0/2444176197 learned_addr learned my addr 172.21.15.107:0/2444176197 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd8977fe640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd898071fc0 0x7fd8981f06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd897fff640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd898152040 0x7fd898150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd896ffd640 1 -- 172.21.15.107:0/2444176197 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd898071fc0 msgr2=0x7fd8981f06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd896ffd640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd898071fc0 0x7fd8981f06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd896ffd640 1 -- 172.21.15.107:0/2444176197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd898152040 msgr2=0x7fd898150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd896ffd640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd898152040 0x7fd898150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd896ffd640 1 -- 172.21.15.107:0/2444176197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8880094d0 con 0x7fd8980adb80 2023-10-25T00:04:12.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd897fff640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd898152040 0x7fd898150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:12.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd896ffd640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8980adb80 0x7fd8981ec770 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00d800 tx=0x7fd88c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:12.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd87ffff640 1 -- 172.21.15.107:0/2444176197 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c00b660 con 0x7fd8980adb80 2023-10-25T00:04:12.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 -- 172.21.15.107:0/2444176197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd898154070 con 0x7fd8980adb80 2023-10-25T00:04:12.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd87ffff640 1 -- 172.21.15.107:0/2444176197 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88c00b7c0 con 0x7fd8980adb80 2023-10-25T00:04:12.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd87ffff640 1 -- 172.21.15.107:0/2444176197 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c00bb30 con 0x7fd8980adb80 2023-10-25T00:04:12.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.466+0000 7fd89dc8d640 1 -- 172.21.15.107:0/2444176197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd898154630 con 0x7fd8980adb80 2023-10-25T00:04:12.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.470+0000 7fd89dc8d640 1 -- 172.21.15.107:0/2444176197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd898154970 con 0x7fd8980adb80 2023-10-25T00:04:12.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.470+0000 7fd87ffff640 1 -- 172.21.15.107:0/2444176197 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd88c00bc90 con 0x7fd8980adb80 2023-10-25T00:04:12.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.470+0000 7fd87ffff640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd884065e60 0x7fd884068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:12.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.470+0000 7fd87ffff640 1 -- 172.21.15.107:0/2444176197 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7fd88c084750 con 0x7fd8980adb80 2023-10-25T00:04:12.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.470+0000 7fd8977fe640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd884065e60 0x7fd884068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:12.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.470+0000 7fd8977fe640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd884065e60 0x7fd884068320 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd8880ad000 tx=0x7fd888008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:12.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.474+0000 7fd87ffff640 1 -- 172.21.15.107:0/2444176197 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd88c051390 con 0x7fd8980adb80 2023-10-25T00:04:12.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.722+0000 7fd89dc8d640 1 -- 172.21.15.107:0/2444176197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8980746e0 con 0x7fd8980adb80 2023-10-25T00:04:12.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.722+0000 7fd87ffff640 1 -- 172.21.15.107:0/2444176197 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7fd88c0549b0 con 0x7fd8980adb80 2023-10-25T00:04:12.724 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:12.725 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:12.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 -- 172.21.15.107:0/2444176197 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd884065e60 msgr2=0x7fd884068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd884065e60 0x7fd884068320 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd8880ad000 tx=0x7fd888008040 comp rx=0 tx=0).stop 2023-10-25T00:04:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 -- 172.21.15.107:0/2444176197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8980adb80 msgr2=0x7fd8981ec770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8980adb80 0x7fd8981ec770 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00d800 tx=0x7fd88c00dcc0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 -- 172.21.15.107:0/2444176197 shutdown_connections 2023-10-25T00:04:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd898152040 0x7fd898150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8980adb80 0x7fd8981ec770 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd884065e60 0x7fd884068320 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 --2- 172.21.15.107:0/2444176197 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd898071fc0 0x7fd8981f06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 -- 172.21.15.107:0/2444176197 >> 172.21.15.107:0/2444176197 conn(0x7fd898070f60 msgr2=0x7fd89806c080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 -- 172.21.15.107:0/2444176197 shutdown_connections 2023-10-25T00:04:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:12.726+0000 7fd89dc8d640 1 -- 172.21.15.107:0/2444176197 wait complete. 2023-10-25T00:04:12.744 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:04:14.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:14.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4c302640 1 Processor -- start 2023-10-25T00:04:14.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4c302640 1 -- start start 2023-10-25T00:04:14.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4c302640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5a4406d370 0x7f5a440adc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:14.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4c302640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4406d9f0 con 0x7f5a440c4350 2023-10-25T00:04:14.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4c302640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a44154a40 con 0x7f5a440ad860 2023-10-25T00:04:14.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4c302640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a44154850 con 0x7f5a4406d370 2023-10-25T00:04:14.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4a89f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5a440c4350 0x7f5a4415f1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33978/0 (socket says 172.21.15.107:33978) 2023-10-25T00:04:14.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4a89f640 1 -- 172.21.15.107:0/680070997 learned_addr learned my addr 172.21.15.107:0/680070997 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:14.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4989d640 1 --2- 172.21.15.107:0/680070997 >> v2:172.21.15.110:3300/0 conn(0x7f5a4406d370 0x7f5a440adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:14.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920946160 0 0) 0x7f5a4406d9f0 con 0x7f5a440c4350 2023-10-25T00:04:14.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c003530 con 0x7f5a440c4350 2023-10-25T00:04:14.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1789158678 0 0) 0x7f5a2c003530 con 0x7f5a440c4350 2023-10-25T00:04:14.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 >> v2:172.21.15.110:3300/0 conn(0x7f5a4406d370 msgr2=0x7f5a440adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:14.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 --2- 172.21.15.107:0/680070997 >> v2:172.21.15.110:3300/0 conn(0x7f5a4406d370 0x7f5a440adc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:14.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 >> v1:172.21.15.110:6789/0 conn(0x7f5a440ad860 legacy=0x7f5a4406bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:14.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a44154d20 con 0x7f5a440c4350 2023-10-25T00:04:14.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5a34001680 con 0x7f5a440c4350 2023-10-25T00:04:14.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 >> v1:172.21.15.107:6789/0 conn(0x7f5a440c4350 legacy=0x7f5a4415f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:14.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 --2- 172.21.15.107:0/680070997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a2c003980 0x7f5a2c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:14.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 --2- 172.21.15.107:0/680070997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a2c0047d0 0x7f5a2c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:14.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 --2- 172.21.15.107:0/680070997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a2c005190 0x7f5a2c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:14.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c00c100 con 0x7f5a2c003980 2023-10-25T00:04:14.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c00c270 con 0x7f5a2c0047d0 2023-10-25T00:04:14.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c00c430 con 0x7f5a2c005190 2023-10-25T00:04:14.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4a09e640 1 --2- 172.21.15.107:0/680070997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a2c003980 0x7f5a2c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:14.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4989d640 1 --2- 172.21.15.107:0/680070997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a2c0047d0 0x7f5a2c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:14.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4a89f640 1 --2- 172.21.15.107:0/680070997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a2c005190 0x7f5a2c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:14.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4a09e640 1 -- 172.21.15.107:0/680070997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a2c005190 msgr2=0x7f5a2c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:14.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4a09e640 1 --2- 172.21.15.107:0/680070997 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a2c005190 0x7f5a2c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:14.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4a09e640 1 -- 172.21.15.107:0/680070997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a2c0047d0 msgr2=0x7f5a2c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:14.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4a09e640 1 --2- 172.21.15.107:0/680070997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a2c0047d0 0x7f5a2c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:14.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4a09e640 1 -- 172.21.15.107:0/680070997 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c003530 con 0x7f5a2c003980 2023-10-25T00:04:14.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.878+0000 7f5a4a89f640 1 -- 172.21.15.107:0/680070997 reap_dead start 2023-10-25T00:04:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.882+0000 7f5a4a09e640 1 --2- 172.21.15.107:0/680070997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a2c003980 0x7f5a2c003da0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c001e70 tx=0x7f5a3c01c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c5937e31976889 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.882+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c003b80 con 0x7f5a2c003980 2023-10-25T00:04:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.882+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c003ce0 con 0x7f5a2c003980 2023-10-25T00:04:14.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.882+0000 7f5a4909c640 1 -- 172.21.15.107:0/680070997 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0052c0 con 0x7f5a2c003980 2023-10-25T00:04:14.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.882+0000 7f5a4c302640 1 -- 172.21.15.107:0/680070997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a2c003980 msgr2=0x7f5a2c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:14.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.882+0000 7f5a4c302640 1 --2- 172.21.15.107:0/680070997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a2c003980 0x7f5a2c003da0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c001e70 tx=0x7f5a3c01c740 comp rx=0 tx=0).stop 2023-10-25T00:04:14.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.882+0000 7f5a4c302640 1 -- 172.21.15.107:0/680070997 shutdown_connections 2023-10-25T00:04:14.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.882+0000 7f5a4c302640 1 --2- 172.21.15.107:0/680070997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a2c003980 0x7f5a2c003da0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:14.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.882+0000 7f5a4c302640 1 -- 172.21.15.107:0/680070997 >> 172.21.15.107:0/680070997 conn(0x7f5a44065930 msgr2=0x7f5a44065d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:14.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.882+0000 7f5a4c302640 1 -- 172.21.15.107:0/680070997 shutdown_connections 2023-10-25T00:04:14.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.882+0000 7f5a4c302640 1 -- 172.21.15.107:0/680070997 wait complete. 2023-10-25T00:04:14.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4c302640 1 Processor -- start 2023-10-25T00:04:14.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4c302640 1 -- start start 2023-10-25T00:04:14.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4c302640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a2c003980 0x7f5a441e3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:14.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4c302640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a44141db0 0x7f5a441ed4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:14.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4c302640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a44154180 0x7f5a441eda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:14.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4c302640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4414d4a0 con 0x7f5a44141db0 2023-10-25T00:04:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4c302640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a441f0a60 con 0x7f5a2c003980 2023-10-25T00:04:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4c302640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a441422b0 con 0x7f5a44154180 2023-10-25T00:04:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4989d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a44141db0 0x7f5a441ed4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4989d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a44141db0 0x7f5a441ed4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47558/0 (socket says 172.21.15.107:47558) 2023-10-25T00:04:14.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4989d640 1 -- 172.21.15.107:0/2727412781 learned_addr learned my addr 172.21.15.107:0/2727412781 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:14.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4a09e640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a2c003980 0x7f5a441e3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:14.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4a89f640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a44154180 0x7f5a441eda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:14.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4989d640 1 -- 172.21.15.107:0/2727412781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a44154180 msgr2=0x7f5a441eda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:14.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4989d640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a44154180 0x7f5a441eda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:14.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4989d640 1 -- 172.21.15.107:0/2727412781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a2c003980 msgr2=0x7f5a441e3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:14.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4989d640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a2c003980 0x7f5a441e3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:14.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4989d640 1 -- 172.21.15.107:0/2727412781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a441f0cb0 con 0x7f5a44141db0 2023-10-25T00:04:14.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4a89f640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a44154180 0x7f5a441eda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:04:14.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4989d640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a44141db0 0x7f5a441ed4f0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f5a38011aa0 tx=0x7f5a38011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a42ffd640 1 -- 172.21.15.107:0/2727412781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3800fb80 con 0x7f5a44141db0 2023-10-25T00:04:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a42ffd640 1 -- 172.21.15.107:0/2727412781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3800fce0 con 0x7f5a44141db0 2023-10-25T00:04:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a42ffd640 1 -- 172.21.15.107:0/2727412781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a38014680 con 0x7f5a44141db0 2023-10-25T00:04:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4c302640 1 -- 172.21.15.107:0/2727412781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a441f0f90 con 0x7f5a44141db0 2023-10-25T00:04:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4c302640 1 -- 172.21.15.107:0/2727412781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a441e37d0 con 0x7f5a44141db0 2023-10-25T00:04:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.886+0000 7f5a4c302640 1 -- 172.21.15.107:0/2727412781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a440746e0 con 0x7f5a44141db0 2023-10-25T00:04:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.890+0000 7f5a42ffd640 1 -- 172.21.15.107:0/2727412781 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5a380147e0 con 0x7f5a44141db0 2023-10-25T00:04:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.890+0000 7f5a42ffd640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a30065e60 0x7f5a30068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.890+0000 7f5a42ffd640 1 -- 172.21.15.107:0/2727412781 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f5a38084900 con 0x7f5a44141db0 2023-10-25T00:04:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.890+0000 7f5a4a09e640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a30065e60 0x7f5a30068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.890+0000 7f5a4a09e640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a30065e60 0x7f5a30068320 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c005000 tx=0x7f5a3c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:14.894+0000 7f5a42ffd640 1 -- 172.21.15.107:0/2727412781 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5a38051650 con 0x7f5a44141db0 2023-10-25T00:04:15.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.134+0000 7f5a4c302640 1 -- 172.21.15.107:0/2727412781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a440c4830 con 0x7f5a44141db0 2023-10-25T00:04:15.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a42ffd640 1 -- 172.21.15.107:0/2727412781 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f5a38054c70 con 0x7f5a44141db0 2023-10-25T00:04:15.139 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:15.139 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:15.140 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 -- 172.21.15.107:0/2727412781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a30065e60 msgr2=0x7f5a30068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:15.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a30065e60 0x7f5a30068320 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c005000 tx=0x7f5a3c009040 comp rx=0 tx=0).stop 2023-10-25T00:04:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 -- 172.21.15.107:0/2727412781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a44141db0 msgr2=0x7f5a441ed4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a44141db0 0x7f5a441ed4f0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f5a38011aa0 tx=0x7f5a38011f60 comp rx=0 tx=0).stop 2023-10-25T00:04:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 -- 172.21.15.107:0/2727412781 shutdown_connections 2023-10-25T00:04:15.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a30065e60 0x7f5a30068320 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:15.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a44154180 0x7f5a441eda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:15.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a44141db0 0x7f5a441ed4f0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:15.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 --2- 172.21.15.107:0/2727412781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a2c003980 0x7f5a441e3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:15.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 -- 172.21.15.107:0/2727412781 >> 172.21.15.107:0/2727412781 conn(0x7f5a44065930 msgr2=0x7f5a441e4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:15.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 -- 172.21.15.107:0/2727412781 shutdown_connections 2023-10-25T00:04:15.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:15.138+0000 7f5a4c302640 1 -- 172.21.15.107:0/2727412781 wait complete. 2023-10-25T00:04:15.161 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:04:17.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:17.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.302+0000 7f09c1d42640 1 Processor -- start 2023-10-25T00:04:17.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.302+0000 7f09c1d42640 1 -- start start 2023-10-25T00:04:17.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09c1d42640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f09bc14fc30 0x7f09bc153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:17.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09c1d42640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc0c18a0 0x7f09bc0c1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:17.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09c1d42640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09bc072330 con 0x7f09bc0c5c20 2023-10-25T00:04:17.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09c1d42640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09bc155ca0 con 0x7f09bc14fc30 2023-10-25T00:04:17.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09c1d42640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc155f80 con 0x7f09bc0c18a0 2023-10-25T00:04:17.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09baffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f09bc0c5c20 0x7f09bc06df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33984/0 (socket says 172.21.15.107:33984) 2023-10-25T00:04:17.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09baffd640 1 -- 172.21.15.107:0/1662264519 learned_addr learned my addr 172.21.15.107:0/1662264519 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:17.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09bb7fe640 1 --2- 172.21.15.107:0/1662264519 >> v2:172.21.15.110:3300/0 conn(0x7f09bc14fc30 0x7f09bc153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:17.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09bbfff640 1 --2- 172.21.15.107:0/1662264519 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc0c18a0 0x7f09bc0c1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:17.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820914597 0 0) 0x7f09bc072330 con 0x7f09bc0c5c20 2023-10-25T00:04:17.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09a4003530 con 0x7f09bc0c5c20 2023-10-25T00:04:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2399053165 0 0) 0x7f09a4003530 con 0x7f09bc0c5c20 2023-10-25T00:04:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc0c18a0 msgr2=0x7f09bc0c1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 --2- 172.21.15.107:0/1662264519 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc0c18a0 0x7f09bc0c1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:17.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 >> v2:172.21.15.110:3300/0 conn(0x7f09bc14fc30 msgr2=0x7f09bc153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:17.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 --2- 172.21.15.107:0/1662264519 >> v2:172.21.15.110:3300/0 conn(0x7f09bc14fc30 0x7f09bc153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:17.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc156260 con 0x7f09bc0c5c20 2023-10-25T00:04:17.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f09b0001680 con 0x7f09bc0c5c20 2023-10-25T00:04:17.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 >> v1:172.21.15.107:6789/0 conn(0x7f09bc0c5c20 legacy=0x7f09bc06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:17.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 --2- 172.21.15.107:0/1662264519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 0x7f09a4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:17.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 --2- 172.21.15.107:0/1662264519 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc0c18a0 0x7f09a40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:17.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 --2- 172.21.15.107:0/1662264519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09a4004d10 0x7f09a4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:17.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc155ca0 con 0x7f09a4003980 2023-10-25T00:04:17.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc155f80 con 0x7f09a4004d10 2023-10-25T00:04:17.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a400b990 con 0x7f09bc0c18a0 2023-10-25T00:04:17.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09bb7fe640 1 --2- 172.21.15.107:0/1662264519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 0x7f09a4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:17.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09bb7fe640 1 -- 172.21.15.107:0/1662264519 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc0c18a0 msgr2=0x7f09a40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:17.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09bb7fe640 1 --2- 172.21.15.107:0/1662264519 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc0c18a0 0x7f09a40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:17.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09bb7fe640 1 -- 172.21.15.107:0/1662264519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09a4004d10 msgr2=0x7f09a4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:17.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09bb7fe640 1 --2- 172.21.15.107:0/1662264519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09a4004d10 0x7f09a4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:17.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09bb7fe640 1 -- 172.21.15.107:0/1662264519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4003530 con 0x7f09a4003980 2023-10-25T00:04:17.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.306+0000 7f09bb7fe640 1 --2- 172.21.15.107:0/1662264519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 0x7f09a4003da0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f09ac009ae0 tx=0x7f09ac01ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20941cd4359175ff server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:17.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac0a5630 con 0x7f09a4003980 2023-10-25T00:04:17.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ac0a5790 con 0x7f09a4003980 2023-10-25T00:04:17.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09ba7fc640 1 -- 172.21.15.107:0/1662264519 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac0a5b40 con 0x7f09a4003980 2023-10-25T00:04:17.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 -- 172.21.15.107:0/1662264519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 msgr2=0x7f09a4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:17.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 --2- 172.21.15.107:0/1662264519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 0x7f09a4003da0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f09ac009ae0 tx=0x7f09ac01ca60 comp rx=0 tx=0).stop 2023-10-25T00:04:17.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09bbfff640 1 -- 172.21.15.107:0/1662264519 reap_dead start 2023-10-25T00:04:17.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 -- 172.21.15.107:0/1662264519 shutdown_connections 2023-10-25T00:04:17.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 -- 172.21.15.107:0/1662264519 >> 172.21.15.107:0/1662264519 conn(0x7f09bc0c39a0 msgr2=0x7f09bc0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:17.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 -- 172.21.15.107:0/1662264519 shutdown_connections 2023-10-25T00:04:17.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 -- 172.21.15.107:0/1662264519 wait complete. 2023-10-25T00:04:17.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 Processor -- start 2023-10-25T00:04:17.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 -- start start 2023-10-25T00:04:17.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 0x7f09bc149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:17.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc15aa50 0x7f09bc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:17.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09bc1477c0 0x7f09bc147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:17.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc14b0c0 con 0x7f09a4003980 2023-10-25T00:04:17.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc1487d0 con 0x7f09bc1477c0 2023-10-25T00:04:17.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09c1d42640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc14b640 con 0x7f09bc15aa50 2023-10-25T00:04:17.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 0x7f09bc149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:17.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 0x7f09bc149850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47590/0 (socket says 172.21.15.107:47590) 2023-10-25T00:04:17.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09bb7fe640 1 -- 172.21.15.107:0/2228221659 learned_addr learned my addr 172.21.15.107:0/2228221659 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:17.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09bbfff640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09bc1477c0 0x7f09bc147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:17.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09baffd640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc15aa50 0x7f09bc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:17.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09bb7fe640 1 -- 172.21.15.107:0/2228221659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc15aa50 msgr2=0x7f09bc06d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:17.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09bb7fe640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc15aa50 0x7f09bc06d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:17.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09bb7fe640 1 -- 172.21.15.107:0/2228221659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09bc1477c0 msgr2=0x7f09bc147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:17.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09bb7fe640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09bc1477c0 0x7f09bc147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:17.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09bb7fe640 1 -- 172.21.15.107:0/2228221659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc150630 con 0x7f09a4003980 2023-10-25T00:04:17.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09bb7fe640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 0x7f09bc149850 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f09ac01c890 tx=0x7f09a4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:17.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.310+0000 7f09a3fff640 1 -- 172.21.15.107:0/2228221659 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac0a5630 con 0x7f09a4003980 2023-10-25T00:04:17.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.314+0000 7f09a3fff640 1 -- 172.21.15.107:0/2228221659 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ac004020 con 0x7f09a4003980 2023-10-25T00:04:17.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.314+0000 7f09c1d42640 1 -- 172.21.15.107:0/2228221659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc148a20 con 0x7f09a4003980 2023-10-25T00:04:17.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.314+0000 7f09a3fff640 1 -- 172.21.15.107:0/2228221659 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac004390 con 0x7f09a4003980 2023-10-25T00:04:17.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.314+0000 7f09c1d42640 1 -- 172.21.15.107:0/2228221659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09bc1525e0 con 0x7f09a4003980 2023-10-25T00:04:17.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.314+0000 7f09c1d42640 1 -- 172.21.15.107:0/2228221659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09bc148c10 con 0x7f09a4003980 2023-10-25T00:04:17.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.314+0000 7f09a3fff640 1 -- 172.21.15.107:0/2228221659 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f09ac0b4040 con 0x7f09a4003980 2023-10-25T00:04:17.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.314+0000 7f09a3fff640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09a8065e90 0x7f09a8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:17.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.314+0000 7f09baffd640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09a8065e90 0x7f09a8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:17.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.314+0000 7f09a3fff640 1 -- 172.21.15.107:0/2228221659 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f09ac11e250 con 0x7f09a4003980 2023-10-25T00:04:17.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.314+0000 7f09baffd640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09a8065e90 0x7f09a8068350 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f09bc1486a0 tx=0x7f09b0001710 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:17.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.322+0000 7f09a3fff640 1 -- 172.21.15.107:0/2228221659 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f09ac0eafc0 con 0x7f09a4003980 2023-10-25T00:04:17.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.574+0000 7f09c1d42640 1 -- 172.21.15.107:0/2228221659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09bc0746e0 con 0x7f09a4003980 2023-10-25T00:04:17.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.574+0000 7f09a3fff640 1 -- 172.21.15.107:0/2228221659 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f09ac0ee5e0 con 0x7f09a4003980 2023-10-25T00:04:17.577 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:17.577 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:17.577 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:17.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 -- 172.21.15.107:0/2228221659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09a8065e90 msgr2=0x7f09a8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:17.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09a8065e90 0x7f09a8068350 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f09bc1486a0 tx=0x7f09b0001710 comp rx=0 tx=0).stop 2023-10-25T00:04:17.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 -- 172.21.15.107:0/2228221659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 msgr2=0x7f09bc149850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:17.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 0x7f09bc149850 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f09ac01c890 tx=0x7f09a4005800 comp rx=0 tx=0).stop 2023-10-25T00:04:17.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 -- 172.21.15.107:0/2228221659 shutdown_connections 2023-10-25T00:04:17.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09bc1477c0 0x7f09bc147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:17.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f09a8065e90 0x7f09a8068350 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:17.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f09bc15aa50 0x7f09bc06d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:17.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 --2- 172.21.15.107:0/2228221659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f09a4003980 0x7f09bc149850 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:17.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 -- 172.21.15.107:0/2228221659 >> 172.21.15.107:0/2228221659 conn(0x7f09bc0c39a0 msgr2=0x7f09bc15b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:17.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 -- 172.21.15.107:0/2228221659 shutdown_connections 2023-10-25T00:04:17.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:17.578+0000 7f09c1d42640 1 -- 172.21.15.107:0/2228221659 wait complete. 2023-10-25T00:04:17.595 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:04:18.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:18.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f337b115640 1 Processor -- start 2023-10-25T00:04:18.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f337b115640 1 -- start start 2023-10-25T00:04:18.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f337b115640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f337413dc20 0x7f33740c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:18.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f337b115640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3374157e50 0x7f33741582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:18.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f337b115640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3374154860 con 0x7f33740720c0 2023-10-25T00:04:18.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f337b115640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3374158800 con 0x7f3374157e50 2023-10-25T00:04:18.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f337b115640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33740c1f10 con 0x7f337413dc20 2023-10-25T00:04:18.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f33796b2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3374157e50 0x7f33741582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f33796b2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3374157e50 0x7f33741582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47594/0 (socket says 172.21.15.107:47594) 2023-10-25T00:04:18.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f33796b2640 1 -- 172.21.15.107:0/4184542734 learned_addr learned my addr 172.21.15.107:0/4184542734 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f3378eb1640 1 --2- 172.21.15.107:0/4184542734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f337413dc20 0x7f33740c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f33796b2640 1 -- 172.21.15.107:0/4184542734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f337413dc20 msgr2=0x7f33740c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f33796b2640 1 --2- 172.21.15.107:0/4184542734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f337413dc20 0x7f33740c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:18.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f33796b2640 1 -- 172.21.15.107:0/4184542734 >> v1:172.21.15.110:6789/0 conn(0x7f33740720c0 legacy=0x7f337406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f33796b2640 1 -- 172.21.15.107:0/4184542734 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3374154b20 con 0x7f3374157e50 2023-10-25T00:04:18.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.890+0000 7f3378eb1640 1 --2- 172.21.15.107:0/4184542734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f337413dc20 0x7f33740c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:18.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f33796b2640 1 --2- 172.21.15.107:0/4184542734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3374157e50 0x7f33741582c0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f336c009e50 tx=0x7f336c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d79baec2ff51c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f33737fe640 1 -- 172.21.15.107:0/4184542734 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c0a5620 con 0x7f3374157e50 2023-10-25T00:04:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f33737fe640 1 -- 172.21.15.107:0/4184542734 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336c0a5780 con 0x7f3374157e50 2023-10-25T00:04:18.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f33737fe640 1 -- 172.21.15.107:0/4184542734 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336c0a5b10 con 0x7f3374157e50 2023-10-25T00:04:18.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f337b115640 1 -- 172.21.15.107:0/4184542734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3374157e50 msgr2=0x7f33741582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:18.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f337b115640 1 --2- 172.21.15.107:0/4184542734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3374157e50 0x7f33741582c0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f336c009e50 tx=0x7f336c01c8b0 comp rx=0 tx=0).stop 2023-10-25T00:04:18.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f337b115640 1 -- 172.21.15.107:0/4184542734 shutdown_connections 2023-10-25T00:04:18.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f337b115640 1 --2- 172.21.15.107:0/4184542734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3374157e50 0x7f33741582c0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:18.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f337b115640 1 --2- 172.21.15.107:0/4184542734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f337413dc20 0x7f33740c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:18.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f337b115640 1 -- 172.21.15.107:0/4184542734 >> 172.21.15.107:0/4184542734 conn(0x7f33740c39a0 msgr2=0x7f33740c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:18.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f337b115640 1 -- 172.21.15.107:0/4184542734 shutdown_connections 2023-10-25T00:04:18.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.894+0000 7f337b115640 1 -- 172.21.15.107:0/4184542734 wait complete. 2023-10-25T00:04:18.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f337b115640 1 Processor -- start 2023-10-25T00:04:18.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f337b115640 1 -- start start 2023-10-25T00:04:18.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f337b115640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33740720c0 0x7f3374153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:18.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f337b115640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f337413dc20 0x7f33741ecd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:18.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f337b115640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3374157e50 0x7f33741ed2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:18.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f337b115640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33741ed7f0 con 0x7f337413dc20 2023-10-25T00:04:18.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f337b115640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33741ec870 con 0x7f33740720c0 2023-10-25T00:04:18.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f337b115640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33741f02e0 con 0x7f3374157e50 2023-10-25T00:04:18.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f3373fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f337413dc20 0x7f33741ecd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:18.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f3373fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f337413dc20 0x7f33741ecd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47606/0 (socket says 172.21.15.107:47606) 2023-10-25T00:04:18.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f3373fff640 1 -- 172.21.15.107:0/969152236 learned_addr learned my addr 172.21.15.107:0/969152236 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:18.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f3378eb1640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33740720c0 0x7f3374153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f33796b2640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3374157e50 0x7f33741ed2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f3373fff640 1 -- 172.21.15.107:0/969152236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3374157e50 msgr2=0x7f33741ed2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:18.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f3373fff640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3374157e50 0x7f33741ed2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f3373fff640 1 -- 172.21.15.107:0/969152236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33740720c0 msgr2=0x7f3374153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f3373fff640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33740720c0 0x7f3374153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:18.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f3373fff640 1 -- 172.21.15.107:0/969152236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c009b30 con 0x7f337413dc20 2023-10-25T00:04:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f3373fff640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f337413dc20 0x7f33741ecd70 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f336000b410 tx=0x7f336000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:18.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f33717fa640 1 -- 172.21.15.107:0/969152236 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360004020 con 0x7f337413dc20 2023-10-25T00:04:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f33717fa640 1 -- 172.21.15.107:0/969152236 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3360004180 con 0x7f337413dc20 2023-10-25T00:04:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f33717fa640 1 -- 172.21.15.107:0/969152236 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360011630 con 0x7f337413dc20 2023-10-25T00:04:18.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f337b115640 1 -- 172.21.15.107:0/969152236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33741f05c0 con 0x7f337413dc20 2023-10-25T00:04:18.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.898+0000 7f337b115640 1 -- 172.21.15.107:0/969152236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33741f0b80 con 0x7f337413dc20 2023-10-25T00:04:18.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.902+0000 7f337b115640 1 -- 172.21.15.107:0/969152236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33740746e0 con 0x7f337413dc20 2023-10-25T00:04:18.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.902+0000 7f33717fa640 1 -- 172.21.15.107:0/969152236 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3360011790 con 0x7f337413dc20 2023-10-25T00:04:18.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.902+0000 7f33717fa640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3358065e60 0x7f3358068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:18.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.902+0000 7f3378eb1640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3358065e60 0x7f3358068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:18.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.902+0000 7f33717fa640 1 -- 172.21.15.107:0/969152236 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f3360083fa0 con 0x7f337413dc20 2023-10-25T00:04:18.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.902+0000 7f3378eb1640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3358065e60 0x7f3358068320 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3368006fd0 tx=0x7f3368008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:18.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:18.906+0000 7f33717fa640 1 -- 172.21.15.107:0/969152236 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f33600b5040 con 0x7f337413dc20 2023-10-25T00:04:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.114+0000 7f337b115640 1 -- 172.21.15.107:0/969152236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33740c1190 con 0x7f337413dc20 2023-10-25T00:04:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.114+0000 7f33717fa640 1 -- 172.21.15.107:0/969152236 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f3360015020 con 0x7f337413dc20 2023-10-25T00:04:19.119 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:19.119 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:19.121 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:19.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 -- 172.21.15.107:0/969152236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3358065e60 msgr2=0x7f3358068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3358065e60 0x7f3358068320 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f3368006fd0 tx=0x7f3368008040 comp rx=0 tx=0).stop 2023-10-25T00:04:19.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 -- 172.21.15.107:0/969152236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f337413dc20 msgr2=0x7f33741ecd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f337413dc20 0x7f33741ecd70 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f336000b410 tx=0x7f336000b8d0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 -- 172.21.15.107:0/969152236 shutdown_connections 2023-10-25T00:04:19.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3358065e60 0x7f3358068320 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3374157e50 0x7f33741ed2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f337413dc20 0x7f33741ecd70 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 --2- 172.21.15.107:0/969152236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33740720c0 0x7f3374153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 -- 172.21.15.107:0/969152236 >> 172.21.15.107:0/969152236 conn(0x7f33740c39a0 msgr2=0x7f33740c05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:19.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 -- 172.21.15.107:0/969152236 shutdown_connections 2023-10-25T00:04:19.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.118+0000 7f337b115640 1 -- 172.21.15.107:0/969152236 wait complete. 2023-10-25T00:04:19.138 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2023-10-25T00:04:19.138 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-25T00:04:19.139 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:04:19.156 INFO:tasks.ceph.mds.b:Stopped 2023-10-25T00:04:19.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:19.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f883cf640 1 Processor -- start 2023-10-25T00:04:19.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f883cf640 1 -- start start 2023-10-25T00:04:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f883cf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0f800c33e0 0x7f0f8015ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f883cf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f800c22f0 con 0x7f0f8014eb70 2023-10-25T00:04:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f883cf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f800c25d0 con 0x7f0f80070b00 2023-10-25T00:04:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f883cf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f80071700 con 0x7f0f800c33e0 2023-10-25T00:04:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f8696c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0f800c33e0 0x7f0f8015ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f8616b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0f8014eb70 0x7f0f801514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33992/0 (socket says 172.21.15.107:33992) 2023-10-25T00:04:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f8696c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0f800c33e0 0x7f0f8015ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47614/0 (socket says 172.21.15.107:47614) 2023-10-25T00:04:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f8616b640 1 -- 172.21.15.107:0/3108920443 learned_addr learned my addr 172.21.15.107:0/3108920443 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f8696c640 1 -- 172.21.15.107:0/3108920443 >> v1:172.21.15.110:6789/0 conn(0x7f0f80070b00 legacy=0x7f0f8006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f8696c640 1 -- 172.21.15.107:0/3108920443 >> v1:172.21.15.107:6789/0 conn(0x7f0f8014eb70 legacy=0x7f0f801514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f85169640 1 -- 172.21.15.107:0/3108920443 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717586447 0 0) 0x7f0f800c22f0 con 0x7f0f8014eb70 2023-10-25T00:04:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.290+0000 7f0f8696c640 1 -- 172.21.15.107:0/3108920443 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f800c2be0 con 0x7f0f800c33e0 2023-10-25T00:04:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f8696c640 1 --2- 172.21.15.107:0/3108920443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f800c33e0 0x7f0f8015ce70 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009d90 tx=0x7f0f7801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3b14373a6fc052 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f85169640 1 -- 172.21.15.107:0/3108920443 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f780a58e0 con 0x7f0f800c33e0 2023-10-25T00:04:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f85169640 1 -- 172.21.15.107:0/3108920443 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f780a5a40 con 0x7f0f800c33e0 2023-10-25T00:04:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f85169640 1 -- 172.21.15.107:0/3108920443 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f780a5dd0 con 0x7f0f800c33e0 2023-10-25T00:04:19.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 -- 172.21.15.107:0/3108920443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f800c33e0 msgr2=0x7f0f8015ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 --2- 172.21.15.107:0/3108920443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f800c33e0 0x7f0f8015ce70 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009d90 tx=0x7f0f7801c910 comp rx=0 tx=0).stop 2023-10-25T00:04:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 -- 172.21.15.107:0/3108920443 shutdown_connections 2023-10-25T00:04:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 --2- 172.21.15.107:0/3108920443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f800c33e0 0x7f0f8015ce70 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 -- 172.21.15.107:0/3108920443 >> 172.21.15.107:0/3108920443 conn(0x7f0f800c39a0 msgr2=0x7f0f800c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 -- 172.21.15.107:0/3108920443 shutdown_connections 2023-10-25T00:04:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 -- 172.21.15.107:0/3108920443 wait complete. 2023-10-25T00:04:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 Processor -- start 2023-10-25T00:04:19.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 -- start start 2023-10-25T00:04:19.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f80070b00 0x7f0f8014b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:19.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f800c33e0 0x7f0f80149080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:19.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8014eb70 0x7f0f8013cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8014be30 con 0x7f0f800c33e0 2023-10-25T00:04:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8013f550 con 0x7f0f80070b00 2023-10-25T00:04:19.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.294+0000 7f0f883cf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8014f470 con 0x7f0f8014eb70 2023-10-25T00:04:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8596a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f800c33e0 0x7f0f80149080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8596a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f800c33e0 0x7f0f80149080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47620/0 (socket says 172.21.15.107:47620) 2023-10-25T00:04:19.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8596a640 1 -- 172.21.15.107:0/1503414418 learned_addr learned my addr 172.21.15.107:0/1503414418 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8616b640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f80070b00 0x7f0f8014b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:19.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8696c640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8014eb70 0x7f0f8013cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:19.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8596a640 1 -- 172.21.15.107:0/1503414418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8014eb70 msgr2=0x7f0f8013cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8596a640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8014eb70 0x7f0f8013cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8596a640 1 -- 172.21.15.107:0/1503414418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f80070b00 msgr2=0x7f0f8014b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8596a640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f80070b00 0x7f0f8014b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8596a640 1 -- 172.21.15.107:0/1503414418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f78009a70 con 0x7f0f800c33e0 2023-10-25T00:04:19.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8596a640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f800c33e0 0x7f0f80149080 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000b410 tx=0x7f0f7000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:19.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f76ffd640 1 -- 172.21.15.107:0/1503414418 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70004020 con 0x7f0f800c33e0 2023-10-25T00:04:19.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f76ffd640 1 -- 172.21.15.107:0/1503414418 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f70004180 con 0x7f0f800c33e0 2023-10-25T00:04:19.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f883cf640 1 -- 172.21.15.107:0/1503414418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8006c880 con 0x7f0f800c33e0 2023-10-25T00:04:19.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f76ffd640 1 -- 172.21.15.107:0/1503414418 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70011630 con 0x7f0f800c33e0 2023-10-25T00:04:19.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f883cf640 1 -- 172.21.15.107:0/1503414418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8013ef00 con 0x7f0f800c33e0 2023-10-25T00:04:19.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f883cf640 1 -- 172.21.15.107:0/1503414418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f800746e0 con 0x7f0f800c33e0 2023-10-25T00:04:19.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f76ffd640 1 -- 172.21.15.107:0/1503414418 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0f70011790 con 0x7f0f800c33e0 2023-10-25T00:04:19.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f76ffd640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f64065e60 0x7f0f64068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:19.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f76ffd640 1 -- 172.21.15.107:0/1503414418 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f0f70083f60 con 0x7f0f800c33e0 2023-10-25T00:04:19.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8616b640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f64065e60 0x7f0f64068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:19.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.298+0000 7f0f8616b640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f64065e60 0x7f0f64068320 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c000f70 tx=0x7f0f7c007b60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:19.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.306+0000 7f0f76ffd640 1 -- 172.21.15.107:0/1503414418 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0f70050c20 con 0x7f0f800c33e0 2023-10-25T00:04:19.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 -- 172.21.15.107:0/1503414418 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f800c1ed0 con 0x7f0f800c33e0 2023-10-25T00:04:19.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f76ffd640 1 -- 172.21.15.107:0/1503414418 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f0f70015020 con 0x7f0f800c33e0 2023-10-25T00:04:19.580 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:19.580 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:19.580 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:19.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 -- 172.21.15.107:0/1503414418 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f64065e60 msgr2=0x7f0f64068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f64065e60 0x7f0f64068320 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c000f70 tx=0x7f0f7c007b60 comp rx=0 tx=0).stop 2023-10-25T00:04:19.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 -- 172.21.15.107:0/1503414418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f800c33e0 msgr2=0x7f0f80149080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f800c33e0 0x7f0f80149080 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000b410 tx=0x7f0f7000b8d0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 -- 172.21.15.107:0/1503414418 shutdown_connections 2023-10-25T00:04:19.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f64065e60 0x7f0f64068320 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8014eb70 0x7f0f8013cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f800c33e0 0x7f0f80149080 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 --2- 172.21.15.107:0/1503414418 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f80070b00 0x7f0f8014b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 -- 172.21.15.107:0/1503414418 >> 172.21.15.107:0/1503414418 conn(0x7f0f800c39a0 msgr2=0x7f0f8014ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:19.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 -- 172.21.15.107:0/1503414418 shutdown_connections 2023-10-25T00:04:19.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.578+0000 7f0f883cf640 1 -- 172.21.15.107:0/1503414418 wait complete. 2023-10-25T00:04:19.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:19.599 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:04:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb293872640 1 Processor -- start 2023-10-25T00:04:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb293872640 1 -- start start 2023-10-25T00:04:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb293872640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb28c156770 0x7fb28c0ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb293872640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb28c0c70e0 0x7fb28c0bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb293872640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28c0c1ca0 con 0x7fb28c0c2130 2023-10-25T00:04:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb293872640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb28c0c27e0 con 0x7fb28c156770 2023-10-25T00:04:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb293872640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb28c0c10c0 con 0x7fb28c0c70e0 2023-10-25T00:04:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb29160e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb28c156770 0x7fb28c0ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb29160e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb28c156770 0x7fb28c0ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47630/0 (socket says 172.21.15.107:47630) 2023-10-25T00:04:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb29160e640 1 -- 172.21.15.107:0/664510476 learned_addr learned my addr 172.21.15.107:0/664510476 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb29160e640 1 -- 172.21.15.107:0/664510476 >> v2:172.21.15.110:3300/0 conn(0x7fb28c0c70e0 msgr2=0x7fb28c0bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb29160e640 1 --2- 172.21.15.107:0/664510476 >> v2:172.21.15.110:3300/0 conn(0x7fb28c0c70e0 0x7fb28c0bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb29160e640 1 -- 172.21.15.107:0/664510476 >> v1:172.21.15.110:6789/0 conn(0x7fb28c0c2130 legacy=0x7fb28c069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb29160e640 1 -- 172.21.15.107:0/664510476 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28c0c13a0 con 0x7fb28c156770 2023-10-25T00:04:19.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.750+0000 7fb29160e640 1 --2- 172.21.15.107:0/664510476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb28c156770 0x7fb28c0ab7c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0a8180 tx=0x7fb28801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bce6180858f4b98 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:19.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb283fff640 1 -- 172.21.15.107:0/664510476 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2880ab600 con 0x7fb28c156770 2023-10-25T00:04:19.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb283fff640 1 -- 172.21.15.107:0/664510476 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2880a7050 con 0x7fb28c156770 2023-10-25T00:04:19.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb283fff640 1 -- 172.21.15.107:0/664510476 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2880ab980 con 0x7fb28c156770 2023-10-25T00:04:19.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- 172.21.15.107:0/664510476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb28c156770 msgr2=0x7fb28c0ab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 --2- 172.21.15.107:0/664510476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb28c156770 0x7fb28c0ab7c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fb28c0a8180 tx=0x7fb28801c370 comp rx=0 tx=0).stop 2023-10-25T00:04:19.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- 172.21.15.107:0/664510476 shutdown_connections 2023-10-25T00:04:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 --2- 172.21.15.107:0/664510476 >> v2:172.21.15.110:3300/0 conn(0x7fb28c0c70e0 0x7fb28c0bdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 --2- 172.21.15.107:0/664510476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb28c156770 0x7fb28c0ab7c0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- 172.21.15.107:0/664510476 >> 172.21.15.107:0/664510476 conn(0x7fb28c0c65c0 msgr2=0x7fb28c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- 172.21.15.107:0/664510476 shutdown_connections 2023-10-25T00:04:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- 172.21.15.107:0/664510476 wait complete. 2023-10-25T00:04:19.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 Processor -- start 2023-10-25T00:04:19.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- start start 2023-10-25T00:04:19.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb28c0c2130 0x7fb28c15ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb28c0c70e0 0x7fb28c14e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb28c156770 0x7fb28c1f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c15c460 con 0x7fb28c156770 2023-10-25T00:04:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c1f0e90 con 0x7fb28c0c2130 2023-10-25T00:04:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28c1f1170 con 0x7fb28c0c70e0 2023-10-25T00:04:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb291e0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb28c156770 0x7fb28c1f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb291e0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb28c156770 0x7fb28c1f0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47638/0 (socket says 172.21.15.107:47638) 2023-10-25T00:04:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb291e0f640 1 -- 172.21.15.107:0/3359415135 learned_addr learned my addr 172.21.15.107:0/3359415135 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb291e0f640 1 -- 172.21.15.107:0/3359415135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb28c0c70e0 msgr2=0x7fb28c14e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb291e0f640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb28c0c70e0 0x7fb28c14e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb291e0f640 1 -- 172.21.15.107:0/3359415135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb28c0c2130 msgr2=0x7fb28c15ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb291e0f640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb28c0c2130 0x7fb28c15ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb291e0f640 1 -- 172.21.15.107:0/3359415135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2880094d0 con 0x7fb28c156770 2023-10-25T00:04:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb291e0f640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb28c156770 0x7fb28c1f0640 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fb28400c780 tx=0x7fb28400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:19.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb281ffb640 1 -- 172.21.15.107:0/3359415135 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb284003d90 con 0x7fb28c156770 2023-10-25T00:04:19.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb281ffb640 1 -- 172.21.15.107:0/3359415135 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28400ce90 con 0x7fb28c156770 2023-10-25T00:04:19.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- 172.21.15.107:0/3359415135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb28c15c0f0 con 0x7fb28c156770 2023-10-25T00:04:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb281ffb640 1 -- 172.21.15.107:0/3359415135 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2840052b0 con 0x7fb28c156770 2023-10-25T00:04:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- 172.21.15.107:0/3359415135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb28c06fca0 con 0x7fb28c156770 2023-10-25T00:04:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.754+0000 7fb293872640 1 -- 172.21.15.107:0/3359415135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28c0746e0 con 0x7fb28c156770 2023-10-25T00:04:19.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.758+0000 7fb281ffb640 1 -- 172.21.15.107:0/3359415135 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb284005410 con 0x7fb28c156770 2023-10-25T00:04:19.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.758+0000 7fb281ffb640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb270065e60 0x7fb270068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:19.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.758+0000 7fb281ffb640 1 -- 172.21.15.107:0/3359415135 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7fb284083a70 con 0x7fb28c156770 2023-10-25T00:04:19.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.758+0000 7fb29160e640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb270065e60 0x7fb270068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:19.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.758+0000 7fb29160e640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb270065e60 0x7fb270068320 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb2880ad000 tx=0x7fb288008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:19.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.762+0000 7fb281ffb640 1 -- 172.21.15.107:0/3359415135 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb284050730 con 0x7fb28c156770 2023-10-25T00:04:19.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 -- 172.21.15.107:0/3359415135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb28c15c280 con 0x7fb28c156770 2023-10-25T00:04:19.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb281ffb640 1 -- 172.21.15.107:0/3359415135 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7fb284017090 con 0x7fb28c156770 2023-10-25T00:04:19.952 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:19.952 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:19.953 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:19.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 -- 172.21.15.107:0/3359415135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb270065e60 msgr2=0x7fb270068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb270065e60 0x7fb270068320 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb2880ad000 tx=0x7fb288008040 comp rx=0 tx=0).stop 2023-10-25T00:04:19.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 -- 172.21.15.107:0/3359415135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb28c156770 msgr2=0x7fb28c1f0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:19.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb28c156770 0x7fb28c1f0640 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fb28400c780 tx=0x7fb28400cc40 comp rx=0 tx=0).stop 2023-10-25T00:04:19.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 -- 172.21.15.107:0/3359415135 shutdown_connections 2023-10-25T00:04:19.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb28c156770 0x7fb28c1f0640 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb270065e60 0x7fb270068320 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb28c0c70e0 0x7fb28c14e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 --2- 172.21.15.107:0/3359415135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb28c0c2130 0x7fb28c15ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:19.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 -- 172.21.15.107:0/3359415135 >> 172.21.15.107:0/3359415135 conn(0x7fb28c0c65c0 msgr2=0x7fb28c0c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:19.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 -- 172.21.15.107:0/3359415135 shutdown_connections 2023-10-25T00:04:19.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:19.950+0000 7fb293872640 1 -- 172.21.15.107:0/3359415135 wait complete. 2023-10-25T00:04:19.970 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:04:21.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:21.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783e676640 1 Processor -- start 2023-10-25T00:04:21.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783e676640 1 -- start start 2023-10-25T00:04:21.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783e676640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7838071220 0x7f783806bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:21.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783e676640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f78380c24c0 0x7f783806a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:21.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783e676640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7838150bc0 con 0x7f7838151960 2023-10-25T00:04:21.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783e676640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f783806c7f0 con 0x7f78380c24c0 2023-10-25T00:04:21.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783e676640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783806c510 con 0x7f7838071220 2023-10-25T00:04:21.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783cc13640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f78380c24c0 0x7f783806a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:21.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f78377fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7838151960 0x7f7838071a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33994/0 (socket says 172.21.15.107:33994) 2023-10-25T00:04:21.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f78377fe640 1 -- 172.21.15.107:0/3003071715 learned_addr learned my addr 172.21.15.107:0/3003071715 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:21.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f7837fff640 1 --2- 172.21.15.107:0/3003071715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7838071220 0x7f783806bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:21.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f7836ffd640 1 -- 172.21.15.107:0/3003071715 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517745840 0 0) 0x7f7838150bc0 con 0x7f7838151960 2023-10-25T00:04:21.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f7836ffd640 1 -- 172.21.15.107:0/3003071715 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7820003530 con 0x7f7838151960 2023-10-25T00:04:21.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783cc13640 1 -- 172.21.15.107:0/3003071715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7838071220 msgr2=0x7f783806bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:21.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783cc13640 1 --2- 172.21.15.107:0/3003071715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7838071220 0x7f783806bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:21.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783cc13640 1 -- 172.21.15.107:0/3003071715 >> v1:172.21.15.107:6789/0 conn(0x7f7838151960 legacy=0x7f7838071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:21.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.730+0000 7f783cc13640 1 -- 172.21.15.107:0/3003071715 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7838150e80 con 0x7f78380c24c0 2023-10-25T00:04:21.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783cc13640 1 --2- 172.21.15.107:0/3003071715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78380c24c0 0x7f783806a9c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f7830009dc0 tx=0x7f783001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4519bf9426b3d4f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7836ffd640 1 -- 172.21.15.107:0/3003071715 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78300a66f0 con 0x7f78380c24c0 2023-10-25T00:04:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7836ffd640 1 -- 172.21.15.107:0/3003071715 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78300a6850 con 0x7f78380c24c0 2023-10-25T00:04:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7836ffd640 1 -- 172.21.15.107:0/3003071715 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78300a6be0 con 0x7f78380c24c0 2023-10-25T00:04:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 -- 172.21.15.107:0/3003071715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78380c24c0 msgr2=0x7f783806a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:21.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 --2- 172.21.15.107:0/3003071715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78380c24c0 0x7f783806a9c0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f7830009dc0 tx=0x7f783001c8a0 comp rx=0 tx=0).stop 2023-10-25T00:04:21.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 -- 172.21.15.107:0/3003071715 shutdown_connections 2023-10-25T00:04:21.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 --2- 172.21.15.107:0/3003071715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78380c24c0 0x7f783806a9c0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:21.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 --2- 172.21.15.107:0/3003071715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7838071220 0x7f783806bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:21.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 -- 172.21.15.107:0/3003071715 >> 172.21.15.107:0/3003071715 conn(0x7f78380c2950 msgr2=0x7f78380c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 -- 172.21.15.107:0/3003071715 shutdown_connections 2023-10-25T00:04:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 -- 172.21.15.107:0/3003071715 wait complete. 2023-10-25T00:04:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 Processor -- start 2023-10-25T00:04:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 -- start start 2023-10-25T00:04:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838071220 0x7f7838147d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:21.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78380c24c0 0x7f7838148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:21.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838151960 0x7f78381e9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:21.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7838139790 con 0x7f7838071220 2023-10-25T00:04:21.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78381ed8f0 con 0x7f7838151960 2023-10-25T00:04:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783813c680 con 0x7f78380c24c0 2023-10-25T00:04:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7837fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838071220 0x7f7838147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7837fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838071220 0x7f7838147d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47666/0 (socket says 172.21.15.107:47666) 2023-10-25T00:04:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7837fff640 1 -- 172.21.15.107:0/1775752907 learned_addr learned my addr 172.21.15.107:0/1775752907 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783cc13640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838151960 0x7f78381e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:21.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f78377fe640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78380c24c0 0x7f7838148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:21.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7837fff640 1 -- 172.21.15.107:0/1775752907 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78380c24c0 msgr2=0x7f7838148ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:21.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7837fff640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78380c24c0 0x7f7838148ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:21.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7837fff640 1 -- 172.21.15.107:0/1775752907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838151960 msgr2=0x7f78381e9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7837fff640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838151960 0x7f78381e9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7837fff640 1 -- 172.21.15.107:0/1775752907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7830009a70 con 0x7f7838071220 2023-10-25T00:04:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f78377fe640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78380c24c0 0x7f7838148ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7837fff640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838071220 0x7f7838147d80 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f782c00d620 tx=0x7f782c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7834ff9640 1 -- 172.21.15.107:0/1775752907 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c00b620 con 0x7f7838071220 2023-10-25T00:04:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f783e676640 1 -- 172.21.15.107:0/1775752907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78381ed370 con 0x7f7838071220 2023-10-25T00:04:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.734+0000 7f7834ff9640 1 -- 172.21.15.107:0/1775752907 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782c00b780 con 0x7f7838071220 2023-10-25T00:04:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.738+0000 7f7834ff9640 1 -- 172.21.15.107:0/1775752907 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c00bb10 con 0x7f7838071220 2023-10-25T00:04:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.738+0000 7f783e676640 1 -- 172.21.15.107:0/1775752907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f783813bc90 con 0x7f7838071220 2023-10-25T00:04:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.738+0000 7f783e676640 1 -- 172.21.15.107:0/1775752907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78380746e0 con 0x7f7838071220 2023-10-25T00:04:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.738+0000 7f7834ff9640 1 -- 172.21.15.107:0/1775752907 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f782c00bc70 con 0x7f7838071220 2023-10-25T00:04:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.738+0000 7f7834ff9640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7824065e60 0x7f7824068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.738+0000 7f7834ff9640 1 -- 172.21.15.107:0/1775752907 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f782c084610 con 0x7f7838071220 2023-10-25T00:04:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.738+0000 7f78377fe640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7824065e60 0x7f7824068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:21.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.738+0000 7f78377fe640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7824065e60 0x7f7824068320 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f78380c5bc0 tx=0x7f7828009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:21.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.742+0000 7f7834ff9640 1 -- 172.21.15.107:0/1775752907 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f782c051360 con 0x7f7838071220 2023-10-25T00:04:21.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.926+0000 7f783e676640 1 -- 172.21.15.107:0/1775752907 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78380c22a0 con 0x7f7838071220 2023-10-25T00:04:21.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.926+0000 7f7834ff9640 1 -- 172.21.15.107:0/1775752907 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f782c007380 con 0x7f7838071220 2023-10-25T00:04:21.929 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:21.929 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:21.930 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:21.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 -- 172.21.15.107:0/1775752907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7824065e60 msgr2=0x7f7824068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:21.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7824065e60 0x7f7824068320 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f78380c5bc0 tx=0x7f7828009040 comp rx=0 tx=0).stop 2023-10-25T00:04:21.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 -- 172.21.15.107:0/1775752907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838071220 msgr2=0x7f7838147d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:21.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838071220 0x7f7838147d80 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f782c00d620 tx=0x7f782c00dae0 comp rx=0 tx=0).stop 2023-10-25T00:04:21.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 -- 172.21.15.107:0/1775752907 shutdown_connections 2023-10-25T00:04:21.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7838151960 0x7f78381e9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:21.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7824065e60 0x7f7824068320 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:21.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78380c24c0 0x7f7838148ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:21.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 --2- 172.21.15.107:0/1775752907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7838071220 0x7f7838147d80 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:21.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 -- 172.21.15.107:0/1775752907 >> 172.21.15.107:0/1775752907 conn(0x7f78380c2950 msgr2=0x7f78380c5070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:21.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 -- 172.21.15.107:0/1775752907 shutdown_connections 2023-10-25T00:04:21.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:21.930+0000 7f783e676640 1 -- 172.21.15.107:0/1775752907 wait complete. 2023-10-25T00:04:21.946 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:04:21.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fc66e8640 1 Processor -- start 2023-10-25T00:04:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fc66e8640 1 -- start start 2023-10-25T00:04:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fc66e8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc014fd40 0x7f1fc00ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fc66e8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fc00a9820 0x7f1fc00aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:22.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fc66e8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1fc006ee40 0x7f1fc00c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fc66e8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc00aa750 con 0x7f1fc00a9820 2023-10-25T00:04:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fc66e8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc006f730 con 0x7f1fc006ee40 2023-10-25T00:04:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fc66e8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc0156a40 con 0x7f1fc014fd40 2023-10-25T00:04:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fb77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fc00a9820 0x7f1fc00aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:22.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fb77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fc00a9820 0x7f1fc00aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47676/0 (socket says 172.21.15.107:47676) 2023-10-25T00:04:22.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fb77fe640 1 -- 172.21.15.107:0/1311421416 learned_addr learned my addr 172.21.15.107:0/1311421416 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:22.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fbffff640 1 --2- 172.21.15.107:0/1311421416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc014fd40 0x7f1fc00ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:22.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fc4c85640 1 --2- 172.21.15.107:0/1311421416 >> v2:172.21.15.110:3300/0 conn(0x7f1fc006ee40 0x7f1fc00c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:22.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fb77fe640 1 -- 172.21.15.107:0/1311421416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc014fd40 msgr2=0x7f1fc00ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:22.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fb77fe640 1 --2- 172.21.15.107:0/1311421416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc014fd40 0x7f1fc00ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fb77fe640 1 -- 172.21.15.107:0/1311421416 >> v2:172.21.15.110:3300/0 conn(0x7f1fc006ee40 msgr2=0x7f1fc00c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:22.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fb77fe640 1 --2- 172.21.15.107:0/1311421416 >> v2:172.21.15.110:3300/0 conn(0x7f1fc006ee40 0x7f1fc00c1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fb77fe640 1 -- 172.21.15.107:0/1311421416 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc0156710 con 0x7f1fc00a9820 2023-10-25T00:04:22.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.094+0000 7f1fc4c85640 1 --2- 172.21.15.107:0/1311421416 >> v2:172.21.15.110:3300/0 conn(0x7f1fc006ee40 0x7f1fc00c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:22.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fb77fe640 1 --2- 172.21.15.107:0/1311421416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00a9820 0x7f1fc00aa210 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1fc006f260 tx=0x7f1fa801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bc46294255e68e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:22.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fbf7fe640 1 -- 172.21.15.107:0/1311421416 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80ac640 con 0x7f1fc00a9820 2023-10-25T00:04:22.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fbf7fe640 1 -- 172.21.15.107:0/1311421416 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa80ac7a0 con 0x7f1fc00a9820 2023-10-25T00:04:22.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fbf7fe640 1 -- 172.21.15.107:0/1311421416 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa80acb30 con 0x7f1fc00a9820 2023-10-25T00:04:22.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fc66e8640 1 -- 172.21.15.107:0/1311421416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00a9820 msgr2=0x7f1fc00aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:22.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fc66e8640 1 --2- 172.21.15.107:0/1311421416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00a9820 0x7f1fc00aa210 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f1fc006f260 tx=0x7f1fa801c3b0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fc66e8640 1 -- 172.21.15.107:0/1311421416 shutdown_connections 2023-10-25T00:04:22.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fc66e8640 1 --2- 172.21.15.107:0/1311421416 >> v2:172.21.15.110:3300/0 conn(0x7f1fc006ee40 0x7f1fc00c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fc66e8640 1 --2- 172.21.15.107:0/1311421416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc00a9820 0x7f1fc00aa210 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fc66e8640 1 --2- 172.21.15.107:0/1311421416 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc014fd40 0x7f1fc00ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fc66e8640 1 -- 172.21.15.107:0/1311421416 >> 172.21.15.107:0/1311421416 conn(0x7f1fc00c2950 msgr2=0x7f1fc00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:22.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fc66e8640 1 -- 172.21.15.107:0/1311421416 shutdown_connections 2023-10-25T00:04:22.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fc66e8640 1 -- 172.21.15.107:0/1311421416 wait complete. 2023-10-25T00:04:22.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fc66e8640 1 Processor -- start 2023-10-25T00:04:22.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.098+0000 7f1fc66e8640 1 -- start start 2023-10-25T00:04:22.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fc66e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc006ee40 0x7f1fc0149310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:22.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fc66e8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc014fd40 0x7f1fc0149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:22.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fc66e8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc00c3c60 0x7f1fc00c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:22.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fc66e8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc014ad70 con 0x7f1fc006ee40 2023-10-25T00:04:22.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fc66e8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc00bf820 con 0x7f1fc00c3c60 2023-10-25T00:04:22.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fc66e8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc014b2b0 con 0x7f1fc014fd40 2023-10-25T00:04:22.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc006ee40 0x7f1fc0149310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:22.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc006ee40 0x7f1fc0149310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47690/0 (socket says 172.21.15.107:47690) 2023-10-25T00:04:22.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbffff640 1 -- 172.21.15.107:0/603248460 learned_addr learned my addr 172.21.15.107:0/603248460 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:22.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fc4c85640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc00c3c60 0x7f1fc00c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:22.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fb77fe640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc014fd40 0x7f1fc0149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:22.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbffff640 1 -- 172.21.15.107:0/603248460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc014fd40 msgr2=0x7f1fc0149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:22.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbffff640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc014fd40 0x7f1fc0149850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbffff640 1 -- 172.21.15.107:0/603248460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc00c3c60 msgr2=0x7f1fc00c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:22.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbffff640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc00c3c60 0x7f1fc00c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbffff640 1 -- 172.21.15.107:0/603248460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa80094d0 con 0x7f1fc006ee40 2023-10-25T00:04:22.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbffff640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc006ee40 0x7f1fc0149310 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000d620 tx=0x7f1fb000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbd7fa640 1 -- 172.21.15.107:0/603248460 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb000b620 con 0x7f1fc006ee40 2023-10-25T00:04:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbd7fa640 1 -- 172.21.15.107:0/603248460 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fb000b780 con 0x7f1fc006ee40 2023-10-25T00:04:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fc66e8640 1 -- 172.21.15.107:0/603248460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc0149d90 con 0x7f1fc006ee40 2023-10-25T00:04:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbd7fa640 1 -- 172.21.15.107:0/603248460 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb000bb10 con 0x7f1fc006ee40 2023-10-25T00:04:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fc66e8640 1 -- 172.21.15.107:0/603248460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc0155380 con 0x7f1fc006ee40 2023-10-25T00:04:22.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fc66e8640 1 -- 172.21.15.107:0/603248460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc00c24a0 con 0x7f1fc006ee40 2023-10-25T00:04:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbd7fa640 1 -- 172.21.15.107:0/603248460 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1fb000bc70 con 0x7f1fc006ee40 2023-10-25T00:04:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbd7fa640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fa0065e90 0x7f1fa0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fbd7fa640 1 -- 172.21.15.107:0/603248460 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f1fb00845a0 con 0x7f1fc006ee40 2023-10-25T00:04:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fb77fe640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fa0065e90 0x7f1fa0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.102+0000 7f1fb77fe640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fa0065e90 0x7f1fa0068350 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80ae000 tx=0x7f1fa8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.106+0000 7f1fbd7fa640 1 -- 172.21.15.107:0/603248460 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1fb00512f0 con 0x7f1fc006ee40 2023-10-25T00:04:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 -- 172.21.15.107:0/603248460 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fc0070840 con 0x7f1fc006ee40 2023-10-25T00:04:22.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fbd7fa640 1 -- 172.21.15.107:0/603248460 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f1fb0007380 con 0x7f1fc006ee40 2023-10-25T00:04:22.300 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:22.300 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 -- 172.21.15.107:0/603248460 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fa0065e90 msgr2=0x7f1fa0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:22.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fa0065e90 0x7f1fa0068350 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80ae000 tx=0x7f1fa8008040 comp rx=0 tx=0).stop 2023-10-25T00:04:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 -- 172.21.15.107:0/603248460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc006ee40 msgr2=0x7f1fc0149310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc006ee40 0x7f1fc0149310 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f1fb000d620 tx=0x7f1fb000dae0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 -- 172.21.15.107:0/603248460 shutdown_connections 2023-10-25T00:04:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc00c3c60 0x7f1fc00c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fa0065e90 0x7f1fa0068350 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc014fd40 0x7f1fc0149850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 --2- 172.21.15.107:0/603248460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc006ee40 0x7f1fc0149310 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:22.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 -- 172.21.15.107:0/603248460 >> 172.21.15.107:0/603248460 conn(0x7f1fc00c2950 msgr2=0x7f1fc0151d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:22.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 -- 172.21.15.107:0/603248460 shutdown_connections 2023-10-25T00:04:22.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:22.298+0000 7f1fc66e8640 1 -- 172.21.15.107:0/603248460 wait complete. 2023-10-25T00:04:22.318 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:04:23.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:24.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.074+0000 7f347af90640 1 Processor -- start 2023-10-25T00:04:24.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.074+0000 7f347af90640 1 -- start start 2023-10-25T00:04:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347af90640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f347414fd40 0x7f34740ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347af90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f347406ee40 0x7f34740aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347af90640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347406f730 con 0x7f34740a9820 2023-10-25T00:04:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347af90640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34740aa750 con 0x7f347406ee40 2023-10-25T00:04:24.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347af90640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3474156a40 con 0x7f347414fd40 2023-10-25T00:04:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347952d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f347406ee40 0x7f34740aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347952d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f347406ee40 0x7f34740aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52548/0 (socket says 172.21.15.107:52548) 2023-10-25T00:04:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347952d640 1 -- 172.21.15.107:0/4195871290 learned_addr learned my addr 172.21.15.107:0/4195871290 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f3478d2c640 1 --2- 172.21.15.107:0/4195871290 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f347414fd40 0x7f34740ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347952d640 1 -- 172.21.15.107:0/4195871290 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f347414fd40 msgr2=0x7f34740ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347952d640 1 --2- 172.21.15.107:0/4195871290 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f347414fd40 0x7f34740ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347952d640 1 -- 172.21.15.107:0/4195871290 >> v1:172.21.15.110:6789/0 conn(0x7f34740a9820 legacy=0x7f34740c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347952d640 1 -- 172.21.15.107:0/4195871290 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3474156710 con 0x7f347406ee40 2023-10-25T00:04:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f347952d640 1 --2- 172.21.15.107:0/4195871290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347406ee40 0x7f34740aa210 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f346c009870 tx=0x7f346c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e8520c23274552 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:24.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f34737fe640 1 -- 172.21.15.107:0/4195871290 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c0ad640 con 0x7f347406ee40 2023-10-25T00:04:24.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f34737fe640 1 -- 172.21.15.107:0/4195871290 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346c0ad7a0 con 0x7f347406ee40 2023-10-25T00:04:24.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.078+0000 7f34737fe640 1 -- 172.21.15.107:0/4195871290 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346c0adb30 con 0x7f347406ee40 2023-10-25T00:04:24.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 -- 172.21.15.107:0/4195871290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347406ee40 msgr2=0x7f34740aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 --2- 172.21.15.107:0/4195871290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347406ee40 0x7f34740aa210 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f346c009870 tx=0x7f346c01c3f0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 -- 172.21.15.107:0/4195871290 shutdown_connections 2023-10-25T00:04:24.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 --2- 172.21.15.107:0/4195871290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347406ee40 0x7f34740aa210 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 --2- 172.21.15.107:0/4195871290 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f347414fd40 0x7f34740ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 -- 172.21.15.107:0/4195871290 >> 172.21.15.107:0/4195871290 conn(0x7f34740c2950 msgr2=0x7f34740c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 -- 172.21.15.107:0/4195871290 shutdown_connections 2023-10-25T00:04:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 -- 172.21.15.107:0/4195871290 wait complete. 2023-10-25T00:04:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 Processor -- start 2023-10-25T00:04:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 -- start start 2023-10-25T00:04:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347406ee40 0x7f3474153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f347414fd40 0x7f34741489d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3474155990 0x7f3474148f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347414f560 con 0x7f347406ee40 2023-10-25T00:04:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f347414d570 con 0x7f3474155990 2023-10-25T00:04:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f347414eea0 con 0x7f347414fd40 2023-10-25T00:04:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f3478d2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347406ee40 0x7f3474153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f3478d2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347406ee40 0x7f3474153c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52556/0 (socket says 172.21.15.107:52556) 2023-10-25T00:04:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f3478d2c640 1 -- 172.21.15.107:0/4132826111 learned_addr learned my addr 172.21.15.107:0/4132826111 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347952d640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3474155990 0x7f3474148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f3473fff640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f347414fd40 0x7f34741489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f3478d2c640 1 -- 172.21.15.107:0/4132826111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f347414fd40 msgr2=0x7f34741489d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f3478d2c640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f347414fd40 0x7f34741489d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f3478d2c640 1 -- 172.21.15.107:0/4132826111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3474155990 msgr2=0x7f3474148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f3478d2c640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3474155990 0x7f3474148f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f3478d2c640 1 -- 172.21.15.107:0/4132826111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c0094d0 con 0x7f347406ee40 2023-10-25T00:04:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347952d640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3474155990 0x7f3474148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f3473fff640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f347414fd40 0x7f34741489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f3478d2c640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347406ee40 0x7f3474153c30 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f3464007e70 tx=0x7f346400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f34717fa640 1 -- 172.21.15.107:0/4132826111 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464014620 con 0x7f347406ee40 2023-10-25T00:04:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f34717fa640 1 -- 172.21.15.107:0/4132826111 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3464014780 con 0x7f347406ee40 2023-10-25T00:04:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 -- 172.21.15.107:0/4132826111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347414f180 con 0x7f347406ee40 2023-10-25T00:04:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f34717fa640 1 -- 172.21.15.107:0/4132826111 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3464014b30 con 0x7f347406ee40 2023-10-25T00:04:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.082+0000 7f347af90640 1 -- 172.21.15.107:0/4132826111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347414d7c0 con 0x7f347406ee40 2023-10-25T00:04:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.086+0000 7f347af90640 1 -- 172.21.15.107:0/4132826111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34740c24a0 con 0x7f347406ee40 2023-10-25T00:04:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.086+0000 7f34717fa640 1 -- 172.21.15.107:0/4132826111 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3464014c90 con 0x7f347406ee40 2023-10-25T00:04:24.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.086+0000 7f34717fa640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3454065e60 0x7f3454068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.086+0000 7f34717fa640 1 -- 172.21.15.107:0/4132826111 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f3464084b70 con 0x7f347406ee40 2023-10-25T00:04:24.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.086+0000 7f3473fff640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3454065e60 0x7f3454068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.086+0000 7f3473fff640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3454065e60 0x7f3454068320 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f34741eef80 tx=0x7f345c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:24.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.090+0000 7f34717fa640 1 -- 172.21.15.107:0/4132826111 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3464051830 con 0x7f347406ee40 2023-10-25T00:04:24.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.278+0000 7f347af90640 1 -- 172.21.15.107:0/4132826111 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3474070840 con 0x7f347406ee40 2023-10-25T00:04:24.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.278+0000 7f34717fa640 1 -- 172.21.15.107:0/4132826111 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f3464054e50 con 0x7f347406ee40 2023-10-25T00:04:24.282 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:24.283 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:24.284 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:24.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 -- 172.21.15.107:0/4132826111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3454065e60 msgr2=0x7f3454068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3454065e60 0x7f3454068320 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f34741eef80 tx=0x7f345c009040 comp rx=0 tx=0).stop 2023-10-25T00:04:24.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 -- 172.21.15.107:0/4132826111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347406ee40 msgr2=0x7f3474153c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347406ee40 0x7f3474153c30 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f3464007e70 tx=0x7f346400e4b0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 -- 172.21.15.107:0/4132826111 shutdown_connections 2023-10-25T00:04:24.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3474155990 0x7f3474148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3454065e60 0x7f3454068320 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f347414fd40 0x7f34741489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 --2- 172.21.15.107:0/4132826111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347406ee40 0x7f3474153c30 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 -- 172.21.15.107:0/4132826111 >> 172.21.15.107:0/4132826111 conn(0x7f34740c2950 msgr2=0x7f34740c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:24.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 -- 172.21.15.107:0/4132826111 shutdown_connections 2023-10-25T00:04:24.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.282+0000 7f347af90640 1 -- 172.21.15.107:0/4132826111 wait complete. 2023-10-25T00:04:24.300 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:04:24.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:24.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.434+0000 7f726afbd640 1 Processor -- start 2023-10-25T00:04:24.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.434+0000 7f726afbd640 1 -- start start 2023-10-25T00:04:24.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726afbd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f726414fd40 0x7f72640ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726afbd640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726406f730 con 0x7f726406ee40 2023-10-25T00:04:24.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726afbd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264156a40 con 0x7f72640a9820 2023-10-25T00:04:24.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726afbd640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7264156710 con 0x7f726414fd40 2023-10-25T00:04:24.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726955a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f726406ee40 0x7f726415cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37320/0 (socket says 172.21.15.107:37320) 2023-10-25T00:04:24.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726955a640 1 -- 172.21.15.107:0/1645162545 learned_addr learned my addr 172.21.15.107:0/1645162545 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:24.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645569043 0 0) 0x7f726406f730 con 0x7f726406ee40 2023-10-25T00:04:24.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f7268d59640 1 --2- 172.21.15.107:0/1645162545 >> v2:172.21.15.110:3300/0 conn(0x7f726414fd40 0x7f72640ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f724c003530 con 0x7f726406ee40 2023-10-25T00:04:24.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3183267796 0 0) 0x7f724c003530 con 0x7f726406ee40 2023-10-25T00:04:24.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 >> v2:172.21.15.110:3300/0 conn(0x7f726414fd40 msgr2=0x7f72640ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:24.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 --2- 172.21.15.107:0/1645162545 >> v2:172.21.15.110:3300/0 conn(0x7f726414fd40 0x7f72640ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 >> v1:172.21.15.110:6789/0 conn(0x7f72640a9820 legacy=0x7f72640c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726406f980 con 0x7f726406ee40 2023-10-25T00:04:24.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f725c001a00 con 0x7f726406ee40 2023-10-25T00:04:24.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 >> v1:172.21.15.107:6789/0 conn(0x7f726406ee40 legacy=0x7f726415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f724c003980 0x7f724c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f724c0047d0 0x7f724c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 0x7f724c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264156710 con 0x7f724c005390 2023-10-25T00:04:24.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f724c00c220 con 0x7f724c0047d0 2023-10-25T00:04:24.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f724c00c3e0 con 0x7f724c003980 2023-10-25T00:04:24.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726955a640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 0x7f724c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f7268d59640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f724c003980 0x7f724c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725bfff640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f724c0047d0 0x7f724c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726955a640 1 -- 172.21.15.107:0/1645162545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f724c003980 msgr2=0x7f724c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:24.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726955a640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f724c003980 0x7f724c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726955a640 1 -- 172.21.15.107:0/1645162545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f724c0047d0 msgr2=0x7f724c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:24.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726955a640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f724c0047d0 0x7f724c004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726955a640 1 -- 172.21.15.107:0/1645162545 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c003530 con 0x7f724c005390 2023-10-25T00:04:24.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f7268d59640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f724c003980 0x7f724c003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:24.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726955a640 1 -- 172.21.15.107:0/1645162545 reap_dead start 2023-10-25T00:04:24.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f726955a640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 0x7f724c00b990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f725c00a800 tx=0x7f725c01d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845f346afb5a3b41 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:24.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c0072d0 con 0x7f724c005390 2023-10-25T00:04:24.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725c0a8070 con 0x7f724c005390 2023-10-25T00:04:24.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.438+0000 7f725b7fe640 1 -- 172.21.15.107:0/1645162545 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c007670 con 0x7f724c005390 2023-10-25T00:04:24.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 -- 172.21.15.107:0/1645162545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 msgr2=0x7f724c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 0x7f724c00b990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f725c00a800 tx=0x7f725c01d300 comp rx=0 tx=0).stop 2023-10-25T00:04:24.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 -- 172.21.15.107:0/1645162545 shutdown_connections 2023-10-25T00:04:24.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 --2- 172.21.15.107:0/1645162545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 0x7f724c00b990 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 -- 172.21.15.107:0/1645162545 >> 172.21.15.107:0/1645162545 conn(0x7f72640c2950 msgr2=0x7f72640c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:24.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 -- 172.21.15.107:0/1645162545 shutdown_connections 2023-10-25T00:04:24.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 -- 172.21.15.107:0/1645162545 wait complete. 2023-10-25T00:04:24.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 Processor -- start 2023-10-25T00:04:24.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 -- start start 2023-10-25T00:04:24.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 0x7f72640c4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72640bbe20 0x7f72641ead00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72640c4a60 0x7f72641eb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72640bb080 con 0x7f724c005390 2023-10-25T00:04:24.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264149170 con 0x7f72640c4a60 2023-10-25T00:04:24.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72640c02f0 con 0x7f72640bbe20 2023-10-25T00:04:24.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f7268d59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 0x7f72640c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f7268d59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 0x7f72640c4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52574/0 (socket says 172.21.15.107:52574) 2023-10-25T00:04:24.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f7268d59640 1 -- 172.21.15.107:0/2838699865 learned_addr learned my addr 172.21.15.107:0/2838699865 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:24.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726955a640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72640c4a60 0x7f72641eb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f725bfff640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72640bbe20 0x7f72641ead00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f7268d59640 1 -- 172.21.15.107:0/2838699865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72640bbe20 msgr2=0x7f72641ead00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f7268d59640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72640bbe20 0x7f72641ead00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f7268d59640 1 -- 172.21.15.107:0/2838699865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72640c4a60 msgr2=0x7f72641eb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f7268d59640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72640c4a60 0x7f72641eb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f7268d59640 1 -- 172.21.15.107:0/2838699865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72641eeff0 con 0x7f724c005390 2023-10-25T00:04:24.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726955a640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72640c4a60 0x7f72641eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:24.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f7268d59640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 0x7f72640c4520 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f725400e810 tx=0x7f725400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:24.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f72597fa640 1 -- 172.21.15.107:0/2838699865 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725400c6d0 con 0x7f724c005390 2023-10-25T00:04:24.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f72597fa640 1 -- 172.21.15.107:0/2838699865 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725400c830 con 0x7f724c005390 2023-10-25T00:04:24.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f72597fa640 1 -- 172.21.15.107:0/2838699865 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725400cba0 con 0x7f724c005390 2023-10-25T00:04:24.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 -- 172.21.15.107:0/2838699865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72640bb2d0 con 0x7f724c005390 2023-10-25T00:04:24.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.442+0000 7f726afbd640 1 -- 172.21.15.107:0/2838699865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72640bb900 con 0x7f724c005390 2023-10-25T00:04:24.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.446+0000 7f726afbd640 1 -- 172.21.15.107:0/2838699865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72640c24a0 con 0x7f724c005390 2023-10-25T00:04:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.446+0000 7f72597fa640 1 -- 172.21.15.107:0/2838699865 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f725400cd00 con 0x7f724c005390 2023-10-25T00:04:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.446+0000 7f72597fa640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e90 0x7f7250068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.446+0000 7f72597fa640 1 -- 172.21.15.107:0/2838699865 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f7254083ea0 con 0x7f724c005390 2023-10-25T00:04:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.446+0000 7f725bfff640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e90 0x7f7250068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:24.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.446+0000 7f725bfff640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e90 0x7f7250068350 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7260007fd0 tx=0x7f7260009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:24.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.450+0000 7f72597fa640 1 -- 172.21.15.107:0/2838699865 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7254050bf0 con 0x7f724c005390 2023-10-25T00:04:24.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.642+0000 7f726afbd640 1 -- 172.21.15.107:0/2838699865 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72640746e0 con 0x7f724c005390 2023-10-25T00:04:24.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f72597fa640 1 -- 172.21.15.107:0/2838699865 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f7254021020 con 0x7f724c005390 2023-10-25T00:04:24.647 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:24.648 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:24.649 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:24.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 -- 172.21.15.107:0/2838699865 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e90 msgr2=0x7f7250068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e90 0x7f7250068350 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7260007fd0 tx=0x7f7260009040 comp rx=0 tx=0).stop 2023-10-25T00:04:24.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 -- 172.21.15.107:0/2838699865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 msgr2=0x7f72640c4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:24.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 0x7f72640c4520 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f725400e810 tx=0x7f725400ecd0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 -- 172.21.15.107:0/2838699865 shutdown_connections 2023-10-25T00:04:24.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72640c4a60 0x7f72641eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7250065e90 0x7f7250068350 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72640bbe20 0x7f72641ead00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 --2- 172.21.15.107:0/2838699865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f724c005390 0x7f72640c4520 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:24.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 -- 172.21.15.107:0/2838699865 >> 172.21.15.107:0/2838699865 conn(0x7f72640c2950 msgr2=0x7f726406bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:24.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 -- 172.21.15.107:0/2838699865 shutdown_connections 2023-10-25T00:04:24.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:24.646+0000 7f726afbd640 1 -- 172.21.15.107:0/2838699865 wait complete. 2023-10-25T00:04:24.666 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:04:26.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:26.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.426+0000 7f02844f5640 1 Processor -- start 2023-10-25T00:04:26.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.426+0000 7f02844f5640 1 -- start start 2023-10-25T00:04:26.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f02844f5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f027c14fd40 0x7f027c0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:26.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f02844f5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027c06f730 con 0x7f027c06ee40 2023-10-25T00:04:26.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f02844f5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027c156a40 con 0x7f027c0a9820 2023-10-25T00:04:26.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f02844f5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027c156710 con 0x7f027c14fd40 2023-10-25T00:04:26.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f0282291640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f027c14fd40 0x7f027c0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:26.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f0282a92640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f027c06ee40 0x7f027c15cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37322/0 (socket says 172.21.15.107:37322) 2023-10-25T00:04:26.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f0282a92640 1 -- 172.21.15.107:0/3368105817 learned_addr learned my addr 172.21.15.107:0/3368105817 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:26.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f0282291640 1 -- 172.21.15.107:0/3368105817 >> v1:172.21.15.110:6789/0 conn(0x7f027c0a9820 legacy=0x7f027c0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:26.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f0282291640 1 -- 172.21.15.107:0/3368105817 >> v1:172.21.15.107:6789/0 conn(0x7f027c06ee40 legacy=0x7f027c15cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f0282291640 1 -- 172.21.15.107:0/3368105817 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c06f980 con 0x7f027c14fd40 2023-10-25T00:04:26.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f0282291640 1 --2- 172.21.15.107:0/3368105817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f027c14fd40 0x7f027c0ab090 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f02780097c0 tx=0x7f027801c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2326c04e7d9c698c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:26.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f028128f640 1 -- 172.21.15.107:0/3368105817 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02780ab600 con 0x7f027c14fd40 2023-10-25T00:04:26.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f028128f640 1 -- 172.21.15.107:0/3368105817 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02780a7050 con 0x7f027c14fd40 2023-10-25T00:04:26.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.430+0000 7f028128f640 1 -- 172.21.15.107:0/3368105817 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02780ab980 con 0x7f027c14fd40 2023-10-25T00:04:26.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 -- 172.21.15.107:0/3368105817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f027c14fd40 msgr2=0x7f027c0ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 --2- 172.21.15.107:0/3368105817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f027c14fd40 0x7f027c0ab090 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f02780097c0 tx=0x7f027801c320 comp rx=0 tx=0).stop 2023-10-25T00:04:26.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 -- 172.21.15.107:0/3368105817 shutdown_connections 2023-10-25T00:04:26.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 --2- 172.21.15.107:0/3368105817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f027c14fd40 0x7f027c0ab090 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 -- 172.21.15.107:0/3368105817 >> 172.21.15.107:0/3368105817 conn(0x7f027c0c2950 msgr2=0x7f027c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:26.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 -- 172.21.15.107:0/3368105817 shutdown_connections 2023-10-25T00:04:26.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 -- 172.21.15.107:0/3368105817 wait complete. 2023-10-25T00:04:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 Processor -- start 2023-10-25T00:04:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 -- start start 2023-10-25T00:04:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f027c06ee40 0x7f027c1ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:26.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f027c14fd40 0x7f027c0c09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:26.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f027c0be520 0x7f027c156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:26.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c1ebaf0 con 0x7f027c14fd40 2023-10-25T00:04:26.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c1eb4c0 con 0x7f027c06ee40 2023-10-25T00:04:26.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027c15dfe0 con 0x7f027c0be520 2023-10-25T00:04:26.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0281a90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f027c14fd40 0x7f027c0c09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:26.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0281a90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f027c14fd40 0x7f027c0c09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52586/0 (socket says 172.21.15.107:52586) 2023-10-25T00:04:26.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0281a90640 1 -- 172.21.15.107:0/1566787063 learned_addr learned my addr 172.21.15.107:0/1566787063 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:26.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0282291640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f027c06ee40 0x7f027c1ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:26.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0281a90640 1 -- 172.21.15.107:0/1566787063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f027c0be520 msgr2=0x7f027c156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0282a92640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f027c0be520 0x7f027c156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:26.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0281a90640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f027c0be520 0x7f027c156f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0281a90640 1 -- 172.21.15.107:0/1566787063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f027c06ee40 msgr2=0x7f027c1ebf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0281a90640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f027c06ee40 0x7f027c1ebf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0281a90640 1 -- 172.21.15.107:0/1566787063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0278009420 con 0x7f027c14fd40 2023-10-25T00:04:26.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0282291640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f027c06ee40 0x7f027c1ebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:26.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0281a90640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f027c14fd40 0x7f027c0c09b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f026c00c450 tx=0x7f026c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:26.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0272ffd640 1 -- 172.21.15.107:0/1566787063 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c012830 con 0x7f027c14fd40 2023-10-25T00:04:26.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0272ffd640 1 -- 172.21.15.107:0/1566787063 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026c0129b0 con 0x7f027c14fd40 2023-10-25T00:04:26.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 -- 172.21.15.107:0/1566787063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027c15e2c0 con 0x7f027c14fd40 2023-10-25T00:04:26.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f0272ffd640 1 -- 172.21.15.107:0/1566787063 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c012d60 con 0x7f027c14fd40 2023-10-25T00:04:26.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.434+0000 7f02844f5640 1 -- 172.21.15.107:0/1566787063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027c0c0ef0 con 0x7f027c14fd40 2023-10-25T00:04:26.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.438+0000 7f02844f5640 1 -- 172.21.15.107:0/1566787063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027c0c24a0 con 0x7f027c14fd40 2023-10-25T00:04:26.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.438+0000 7f0272ffd640 1 -- 172.21.15.107:0/1566787063 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f026c01c460 con 0x7f027c14fd40 2023-10-25T00:04:26.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.438+0000 7f0272ffd640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0260065e60 0x7f0260068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:26.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.438+0000 7f0272ffd640 1 -- 172.21.15.107:0/1566787063 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f026c083f70 con 0x7f027c14fd40 2023-10-25T00:04:26.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.438+0000 7f0282291640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0260065e60 0x7f0260068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:26.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.438+0000 7f0282291640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0260065e60 0x7f0260068320 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f02780ad000 tx=0x7f02780a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:26.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.442+0000 7f0272ffd640 1 -- 172.21.15.107:0/1566787063 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f026c0b5040 con 0x7f027c14fd40 2023-10-25T00:04:26.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.630+0000 7f02844f5640 1 -- 172.21.15.107:0/1566787063 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f027c0746e0 con 0x7f027c14fd40 2023-10-25T00:04:26.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.630+0000 7f0272ffd640 1 -- 172.21.15.107:0/1566787063 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f026c050cc0 con 0x7f027c14fd40 2023-10-25T00:04:26.633 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:26.634 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:26.634 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:26.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 -- 172.21.15.107:0/1566787063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0260065e60 msgr2=0x7f0260068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0260065e60 0x7f0260068320 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f02780ad000 tx=0x7f02780a6000 comp rx=0 tx=0).stop 2023-10-25T00:04:26.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 -- 172.21.15.107:0/1566787063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f027c14fd40 msgr2=0x7f027c0c09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f027c14fd40 0x7f027c0c09b0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f026c00c450 tx=0x7f026c00c910 comp rx=0 tx=0).stop 2023-10-25T00:04:26.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 -- 172.21.15.107:0/1566787063 shutdown_connections 2023-10-25T00:04:26.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0260065e60 0x7f0260068320 secure :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f02780ad000 tx=0x7f02780a6000 comp rx=0 tx=0).stop 2023-10-25T00:04:26.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f027c0be520 0x7f027c156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f027c14fd40 0x7f027c0c09b0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 --2- 172.21.15.107:0/1566787063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f027c06ee40 0x7f027c1ebf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 -- 172.21.15.107:0/1566787063 >> 172.21.15.107:0/1566787063 conn(0x7f027c0c2950 msgr2=0x7f027c0719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:26.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 -- 172.21.15.107:0/1566787063 shutdown_connections 2023-10-25T00:04:26.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.634+0000 7f02844f5640 1 -- 172.21.15.107:0/1566787063 wait complete. 2023-10-25T00:04:26.652 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:04:26.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:26.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.786+0000 7f841a468640 1 Processor -- start 2023-10-25T00:04:26.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.786+0000 7f841a468640 1 -- start start 2023-10-25T00:04:26.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8414151d60 0x7f84140a9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:26.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f841406a780 0x7f8414154d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:26.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8414154840 con 0x7f84140c24b0 2023-10-25T00:04:26.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84140aadc0 con 0x7f841406a780 2023-10-25T00:04:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f841406afe0 con 0x7f8414151d60 2023-10-25T00:04:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f84137fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f841406a780 0x7f8414154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f84137fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f841406a780 0x7f8414154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52600/0 (socket says 172.21.15.107:52600) 2023-10-25T00:04:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f84137fe640 1 -- 172.21.15.107:0/4137575061 learned_addr learned my addr 172.21.15.107:0/4137575061 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f8418a05640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f84140c24b0 0x7f84140bf280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37328/0 (socket says 172.21.15.107:37328) 2023-10-25T00:04:26.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f8413fff640 1 --2- 172.21.15.107:0/4137575061 >> v2:172.21.15.110:3300/0 conn(0x7f8414151d60 0x7f84140a9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:26.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f84137fe640 1 -- 172.21.15.107:0/4137575061 >> v2:172.21.15.110:3300/0 conn(0x7f8414151d60 msgr2=0x7f84140a9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:26.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f84137fe640 1 --2- 172.21.15.107:0/4137575061 >> v2:172.21.15.110:3300/0 conn(0x7f8414151d60 0x7f84140a9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f84137fe640 1 -- 172.21.15.107:0/4137575061 >> v1:172.21.15.107:6789/0 conn(0x7f84140c24b0 legacy=0x7f84140bf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f84137fe640 1 -- 172.21.15.107:0/4137575061 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84140aa640 con 0x7f841406a780 2023-10-25T00:04:26.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f8413fff640 1 --2- 172.21.15.107:0/4137575061 >> v2:172.21.15.110:3300/0 conn(0x7f8414151d60 0x7f84140a9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:26.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f84137fe640 1 --2- 172.21.15.107:0/4137575061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f841406a780 0x7f8414154d40 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f841406d3e0 tx=0x7f840401c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a872201719bf3a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:26.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f8412ffd640 1 -- 172.21.15.107:0/4137575061 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040a56c0 con 0x7f841406a780 2023-10-25T00:04:26.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f8412ffd640 1 -- 172.21.15.107:0/4137575061 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84040a5820 con 0x7f841406a780 2023-10-25T00:04:26.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f8412ffd640 1 -- 172.21.15.107:0/4137575061 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040a5bb0 con 0x7f841406a780 2023-10-25T00:04:26.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 -- 172.21.15.107:0/4137575061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f841406a780 msgr2=0x7f8414154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 --2- 172.21.15.107:0/4137575061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f841406a780 0x7f8414154d40 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f841406d3e0 tx=0x7f840401c860 comp rx=0 tx=0).stop 2023-10-25T00:04:26.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 -- 172.21.15.107:0/4137575061 shutdown_connections 2023-10-25T00:04:26.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 --2- 172.21.15.107:0/4137575061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f841406a780 0x7f8414154d40 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 --2- 172.21.15.107:0/4137575061 >> v2:172.21.15.110:3300/0 conn(0x7f8414151d60 0x7f84140a9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 -- 172.21.15.107:0/4137575061 >> 172.21.15.107:0/4137575061 conn(0x7f84140c2950 msgr2=0x7f84140c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:26.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 -- 172.21.15.107:0/4137575061 shutdown_connections 2023-10-25T00:04:26.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.790+0000 7f841a468640 1 -- 172.21.15.107:0/4137575061 wait complete. 2023-10-25T00:04:26.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f841a468640 1 Processor -- start 2023-10-25T00:04:26.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f841a468640 1 -- start start 2023-10-25T00:04:26.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f841a468640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140c24b0 0x7f8414157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f841a468640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414151d60 0x7f841414a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f841a468640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414070fa0 0x7f841414ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f841a468640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841414f3b0 con 0x7f8414151d60 2023-10-25T00:04:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f841a468640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841406f9d0 con 0x7f84140c24b0 2023-10-25T00:04:26.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f841a468640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8414071430 con 0x7f8414070fa0 2023-10-25T00:04:26.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f84137fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414151d60 0x7f841414a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:26.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f84137fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414151d60 0x7f841414a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52612/0 (socket says 172.21.15.107:52612) 2023-10-25T00:04:26.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f84137fe640 1 -- 172.21.15.107:0/2159886218 learned_addr learned my addr 172.21.15.107:0/2159886218 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:26.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f8413fff640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140c24b0 0x7f8414157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:26.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f8418a05640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414070fa0 0x7f841414ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:26.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f84137fe640 1 -- 172.21.15.107:0/2159886218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414070fa0 msgr2=0x7f841414ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f84137fe640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414070fa0 0x7f841414ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f84137fe640 1 -- 172.21.15.107:0/2159886218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140c24b0 msgr2=0x7f8414157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f84137fe640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140c24b0 0x7f8414157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f84137fe640 1 -- 172.21.15.107:0/2159886218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8404009b30 con 0x7f8414151d60 2023-10-25T00:04:26.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f8413fff640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140c24b0 0x7f8414157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:26.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f8418a05640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414070fa0 0x7f841414ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:26.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f84137fe640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414151d60 0x7f841414a640 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f84040af040 tx=0x7f8404019600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:26.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f8410ff9640 1 -- 172.21.15.107:0/2159886218 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040a56c0 con 0x7f8414151d60 2023-10-25T00:04:26.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f8410ff9640 1 -- 172.21.15.107:0/2159886218 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8404007c50 con 0x7f8414151d60 2023-10-25T00:04:26.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f841a468640 1 -- 172.21.15.107:0/2159886218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841414f010 con 0x7f8414151d60 2023-10-25T00:04:26.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f8410ff9640 1 -- 172.21.15.107:0/2159886218 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404009510 con 0x7f8414151d60 2023-10-25T00:04:26.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f841a468640 1 -- 172.21.15.107:0/2159886218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841414b0c0 con 0x7f8414151d60 2023-10-25T00:04:26.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f841a468640 1 -- 172.21.15.107:0/2159886218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84140746e0 con 0x7f8414151d60 2023-10-25T00:04:26.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.794+0000 7f8410ff9640 1 -- 172.21.15.107:0/2159886218 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f84040b8470 con 0x7f8414151d60 2023-10-25T00:04:26.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.798+0000 7f8410ff9640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f4065e60 0x7f83f4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:26.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.798+0000 7f8410ff9640 1 -- 172.21.15.107:0/2159886218 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 8423+0+0 (secure 0 0 0) 0x7f840411fd50 con 0x7f8414151d60 2023-10-25T00:04:26.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.798+0000 7f8413fff640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f4065e60 0x7f83f4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:26.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.798+0000 7f8413fff640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f4065e60 0x7f83f4068320 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8414151780 tx=0x7f83fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:26.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.802+0000 7f8410ff9640 1 -- 172.21.15.107:0/2159886218 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8404123030 con 0x7f8414151d60 2023-10-25T00:04:26.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.986+0000 7f841a468640 1 -- 172.21.15.107:0/2159886218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84140a9520 con 0x7f8414151d60 2023-10-25T00:04:26.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f8410ff9640 1 -- 172.21.15.107:0/2159886218 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5776 (secure 0 0 0) 0x7f84040bc020 con 0x7f8414151d60 2023-10-25T00:04:26.991 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:26.991 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:03:56.676082+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6672},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6672":{"gid":6672,"name":"c","rank":0,"incarnation":91,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6835/4180698155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":4180698155},{"type":"v1","addr":"172.21.15.107:6835","nonce":4180698155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:26.991 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 95 2023-10-25T00:04:26.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 -- 172.21.15.107:0/2159886218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f4065e60 msgr2=0x7f83f4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f4065e60 0x7f83f4068320 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f8414151780 tx=0x7f83fc008040 comp rx=0 tx=0).stop 2023-10-25T00:04:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 -- 172.21.15.107:0/2159886218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414151d60 msgr2=0x7f841414a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414151d60 0x7f841414a640 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f84040af040 tx=0x7f8404019600 comp rx=0 tx=0).stop 2023-10-25T00:04:26.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 -- 172.21.15.107:0/2159886218 shutdown_connections 2023-10-25T00:04:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f83f4065e60 0x7f83f4068320 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8414070fa0 0x7f841414ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414151d60 0x7f841414a640 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 --2- 172.21.15.107:0/2159886218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f84140c24b0 0x7f8414157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 -- 172.21.15.107:0/2159886218 >> 172.21.15.107:0/2159886218 conn(0x7f84140c2950 msgr2=0x7f84141530e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 -- 172.21.15.107:0/2159886218 shutdown_connections 2023-10-25T00:04:26.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:26.990+0000 7f841a468640 1 -- 172.21.15.107:0/2159886218 wait complete. 2023-10-25T00:04:27.009 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:04:28.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:28.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.778+0000 7fc2ef8cc640 1 Processor -- start 2023-10-25T00:04:28.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.778+0000 7fc2ef8cc640 1 -- start start 2023-10-25T00:04:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.778+0000 7fc2ef8cc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc2e80c8100 0x7fc2e80abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.778+0000 7fc2ef8cc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2e806a4b0 0x7fc2e80ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:28.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ef8cc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e8155680 con 0x7fc2e8152040 2023-10-25T00:04:28.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ef8cc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e8154a70 con 0x7fc2e80c8100 2023-10-25T00:04:28.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ef8cc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8154d50 con 0x7fc2e806a4b0 2023-10-25T00:04:28.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ed668640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc2e8152040 0x7fc2e80c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37344/0 (socket says 172.21.15.107:37344) 2023-10-25T00:04:28.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ed668640 1 -- 172.21.15.107:0/1385375225 learned_addr learned my addr 172.21.15.107:0/1385375225 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ece67640 1 --2- 172.21.15.107:0/1385375225 >> v2:172.21.15.107:3300/0 conn(0x7fc2e80c8100 0x7fc2e80abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ede69640 1 --2- 172.21.15.107:0/1385375225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2e806a4b0 0x7fc2e80ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2dffff640 1 -- 172.21.15.107:0/1385375225 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846724104 0 0) 0x7fc2e8155680 con 0x7fc2e8152040 2023-10-25T00:04:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2dffff640 1 -- 172.21.15.107:0/1385375225 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d4003530 con 0x7fc2e8152040 2023-10-25T00:04:28.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ece67640 1 -- 172.21.15.107:0/1385375225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2e806a4b0 msgr2=0x7fc2e80ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ece67640 1 --2- 172.21.15.107:0/1385375225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2e806a4b0 0x7fc2e80ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ece67640 1 -- 172.21.15.107:0/1385375225 >> v1:172.21.15.107:6789/0 conn(0x7fc2e8152040 legacy=0x7fc2e80c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:28.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ece67640 1 -- 172.21.15.107:0/1385375225 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e8155030 con 0x7fc2e80c8100 2023-10-25T00:04:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ece67640 1 --2- 172.21.15.107:0/1385375225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2e80c8100 0x7fc2e80abd00 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80097c0 tx=0x7fc2d801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5aab9c1a06b2217 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2dffff640 1 -- 172.21.15.107:0/1385375225 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80a4ea0 con 0x7fc2e80c8100 2023-10-25T00:04:28.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2dffff640 1 -- 172.21.15.107:0/1385375225 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d80a7050 con 0x7fc2e80c8100 2023-10-25T00:04:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2dffff640 1 -- 172.21.15.107:0/1385375225 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80ab800 con 0x7fc2e80c8100 2023-10-25T00:04:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/1385375225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2e80c8100 msgr2=0x7fc2e80abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ef8cc640 1 --2- 172.21.15.107:0/1385375225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2e80c8100 0x7fc2e80abd00 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80097c0 tx=0x7fc2d801c3f0 comp rx=0 tx=0).stop 2023-10-25T00:04:28.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/1385375225 shutdown_connections 2023-10-25T00:04:28.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ef8cc640 1 --2- 172.21.15.107:0/1385375225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2e806a4b0 0x7fc2e80ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:28.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ef8cc640 1 --2- 172.21.15.107:0/1385375225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2e80c8100 0x7fc2e80abd00 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:28.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/1385375225 >> 172.21.15.107:0/1385375225 conn(0x7fc2e80ca200 msgr2=0x7fc2e80ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:28.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/1385375225 shutdown_connections 2023-10-25T00:04:28.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.782+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/1385375225 wait complete. 2023-10-25T00:04:28.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ef8cc640 1 Processor -- start 2023-10-25T00:04:28.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ef8cc640 1 -- start start 2023-10-25T00:04:28.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ef8cc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2e806a4b0 0x7fc2e814b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:28.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ef8cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2e80c8100 0x7fc2e8145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:28.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ef8cc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2e8152040 0x7fc2e81527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:28.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ef8cc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e814d780 con 0x7fc2e80c8100 2023-10-25T00:04:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ef8cc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e814d430 con 0x7fc2e8152040 2023-10-25T00:04:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ef8cc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8148d10 con 0x7fc2e806a4b0 2023-10-25T00:04:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ece67640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2e80c8100 0x7fc2e8145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ece67640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2e80c8100 0x7fc2e8145b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52630/0 (socket says 172.21.15.107:52630) 2023-10-25T00:04:28.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ece67640 1 -- 172.21.15.107:0/3583435411 learned_addr learned my addr 172.21.15.107:0/3583435411 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ed668640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2e806a4b0 0x7fc2e814b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ede69640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2e8152040 0x7fc2e81527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ece67640 1 -- 172.21.15.107:0/3583435411 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2e806a4b0 msgr2=0x7fc2e814b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ece67640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2e806a4b0 0x7fc2e814b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:28.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ece67640 1 -- 172.21.15.107:0/3583435411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2e8152040 msgr2=0x7fc2e81527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ece67640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2e8152040 0x7fc2e81527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ece67640 1 -- 172.21.15.107:0/3583435411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d8009420 con 0x7fc2e80c8100 2023-10-25T00:04:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ed668640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2e806a4b0 0x7fc2e814b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:04:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ede69640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2e8152040 0x7fc2e81527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:04:28.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ece67640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2e80c8100 0x7fc2e8145b10 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8000c10 tx=0x7fc2d8019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ddffb640 1 -- 172.21.15.107:0/3583435411 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8003a30 con 0x7fc2e80c8100 2023-10-25T00:04:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/3583435411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e8148fd0 con 0x7fc2e80c8100 2023-10-25T00:04:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ddffb640 1 -- 172.21.15.107:0/3583435411 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d8003b90 con 0x7fc2e80c8100 2023-10-25T00:04:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ddffb640 1 -- 172.21.15.107:0/3583435411 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80ab400 con 0x7fc2e80c8100 2023-10-25T00:04:28.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/3583435411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e81f10d0 con 0x7fc2e80c8100 2023-10-25T00:04:28.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.786+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/3583435411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e814d8c0 con 0x7fc2e80c8100 2023-10-25T00:04:28.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.790+0000 7fc2ddffb640 1 -- 172.21.15.107:0/3583435411 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc2d80b2040 con 0x7fc2e80c8100 2023-10-25T00:04:28.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.790+0000 7fc2ddffb640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc2cc065e60 0x7fc2cc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:28.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.790+0000 7fc2ddffb640 1 -- 172.21.15.107:0/3583435411 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 8509+0+0 (secure 0 0 0) 0x7fc2d811e080 con 0x7fc2e80c8100 2023-10-25T00:04:28.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.790+0000 7fc2ed668640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc2cc065e60 0x7fc2cc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:28.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.790+0000 7fc2ed668640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc2cc065e60 0x7fc2cc068320 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4005890 tx=0x7fc2e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:28.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.794+0000 7fc2ddffb640 1 -- 172.21.15.107:0/3583435411 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc2e814d8c0 con 0x7fc2e80c8100 2023-10-25T00:04:28.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.978+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/3583435411 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2e814d8c0 con 0x7fc2e80c8100 2023-10-25T00:04:28.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ddffb640 1 -- 172.21.15.107:0/3583435411 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5016 (secure 0 0 0) 0x7fc2e814d8c0 con 0x7fc2e80c8100 2023-10-25T00:04:28.983 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:28.983 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:28.593178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:28.983 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 96 2023-10-25T00:04:28.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/3583435411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc2cc065e60 msgr2=0x7fc2cc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:28.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc2cc065e60 0x7fc2cc068320 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4005890 tx=0x7fc2e4009040 comp rx=0 tx=0).stop 2023-10-25T00:04:28.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/3583435411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2e80c8100 msgr2=0x7fc2e8145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:28.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2e80c8100 0x7fc2e8145b10 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8000c10 tx=0x7fc2d8019190 comp rx=0 tx=0).stop 2023-10-25T00:04:28.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/3583435411 shutdown_connections 2023-10-25T00:04:28.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc2e8152040 0x7fc2e81527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:28.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2e80c8100 0x7fc2e8145b10 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:28.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc2cc065e60 0x7fc2cc068320 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:28.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 --2- 172.21.15.107:0/3583435411 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2e806a4b0 0x7fc2e814b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:28.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/3583435411 >> 172.21.15.107:0/3583435411 conn(0x7fc2e80ca200 msgr2=0x7fc2e80ad4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:28.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/3583435411 shutdown_connections 2023-10-25T00:04:28.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:28.982+0000 7fc2ef8cc640 1 -- 172.21.15.107:0/3583435411 wait complete. 2023-10-25T00:04:29.003 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:04:29.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:29.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff2aafa640 1 Processor -- start 2023-10-25T00:04:29.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff2aafa640 1 -- start start 2023-10-25T00:04:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff2aafa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7eff24071220 0x7eff2406bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff2aafa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff240c24c0 0x7eff2406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff2aafa640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff24150bc0 con 0x7eff24151960 2023-10-25T00:04:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff2aafa640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2406c7f0 con 0x7eff24071220 2023-10-25T00:04:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff2aafa640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2406c510 con 0x7eff240c24c0 2023-10-25T00:04:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff28896640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7eff24071220 0x7eff2406bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff28896640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7eff24071220 0x7eff2406bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52636/0 (socket says 172.21.15.107:52636) 2023-10-25T00:04:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff28896640 1 -- 172.21.15.107:0/1083994533 learned_addr learned my addr 172.21.15.107:0/1083994533 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff29097640 1 --2- 172.21.15.107:0/1083994533 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff240c24c0 0x7eff2406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff28896640 1 -- 172.21.15.107:0/1083994533 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff240c24c0 msgr2=0x7eff2406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff28896640 1 --2- 172.21.15.107:0/1083994533 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff240c24c0 0x7eff2406a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff28896640 1 -- 172.21.15.107:0/1083994533 >> v1:172.21.15.110:6789/0 conn(0x7eff24151960 legacy=0x7eff24071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.110+0000 7eff28896640 1 -- 172.21.15.107:0/1083994533 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff24150e80 con 0x7eff24071220 2023-10-25T00:04:29.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff28896640 1 --2- 172.21.15.107:0/1083994533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff24071220 0x7eff2406bce0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7eff2000aa10 tx=0x7eff2001d880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78fa911b30b0c8ca server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1b7fe640 1 -- 172.21.15.107:0/1083994533 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff200a76e0 con 0x7eff24071220 2023-10-25T00:04:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1b7fe640 1 -- 172.21.15.107:0/1083994533 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff200a7840 con 0x7eff24071220 2023-10-25T00:04:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1b7fe640 1 -- 172.21.15.107:0/1083994533 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff200a7bd0 con 0x7eff24071220 2023-10-25T00:04:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 -- 172.21.15.107:0/1083994533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff24071220 msgr2=0x7eff2406bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 --2- 172.21.15.107:0/1083994533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff24071220 0x7eff2406bce0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7eff2000aa10 tx=0x7eff2001d880 comp rx=0 tx=0).stop 2023-10-25T00:04:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 -- 172.21.15.107:0/1083994533 shutdown_connections 2023-10-25T00:04:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 --2- 172.21.15.107:0/1083994533 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff240c24c0 0x7eff2406a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 --2- 172.21.15.107:0/1083994533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff24071220 0x7eff2406bce0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 -- 172.21.15.107:0/1083994533 >> 172.21.15.107:0/1083994533 conn(0x7eff240c2950 msgr2=0x7eff240c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 -- 172.21.15.107:0/1083994533 shutdown_connections 2023-10-25T00:04:29.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 -- 172.21.15.107:0/1083994533 wait complete. 2023-10-25T00:04:29.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 Processor -- start 2023-10-25T00:04:29.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 -- start start 2023-10-25T00:04:29.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff24071220 0x7eff24148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff240c24c0 0x7eff24146b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff24151960 0x7eff241470b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff241530a0 con 0x7eff240c24c0 2023-10-25T00:04:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff241475f0 con 0x7eff24071220 2023-10-25T00:04:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff241ebbf0 con 0x7eff24151960 2023-10-25T00:04:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff240c24c0 0x7eff24146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff240c24c0 0x7eff24146b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52646/0 (socket says 172.21.15.107:52646) 2023-10-25T00:04:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1bfff640 1 -- 172.21.15.107:0/2310439709 learned_addr learned my addr 172.21.15.107:0/2310439709 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff28896640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff24071220 0x7eff24148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff29097640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff24151960 0x7eff241470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1bfff640 1 -- 172.21.15.107:0/2310439709 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff24151960 msgr2=0x7eff241470b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1bfff640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff24151960 0x7eff241470b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1bfff640 1 -- 172.21.15.107:0/2310439709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff24071220 msgr2=0x7eff24148eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1bfff640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff24071220 0x7eff24148eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1bfff640 1 -- 172.21.15.107:0/2310439709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2000a670 con 0x7eff240c24c0 2023-10-25T00:04:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff28896640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff24071220 0x7eff24148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:29.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff1bfff640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff240c24c0 0x7eff24146b70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7eff1400c500 tx=0x7eff1400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:29.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff197fa640 1 -- 172.21.15.107:0/2310439709 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14005220 con 0x7eff240c24c0 2023-10-25T00:04:29.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff197fa640 1 -- 172.21.15.107:0/2310439709 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff14005380 con 0x7eff240c24c0 2023-10-25T00:04:29.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff2aafa640 1 -- 172.21.15.107:0/2310439709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff241520e0 con 0x7eff240c24c0 2023-10-25T00:04:29.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.114+0000 7eff197fa640 1 -- 172.21.15.107:0/2310439709 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14007270 con 0x7eff240c24c0 2023-10-25T00:04:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.118+0000 7eff2aafa640 1 -- 172.21.15.107:0/2310439709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff241526a0 con 0x7eff240c24c0 2023-10-25T00:04:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.118+0000 7eff2aafa640 1 -- 172.21.15.107:0/2310439709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2406c200 con 0x7eff240c24c0 2023-10-25T00:04:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.118+0000 7eff197fa640 1 -- 172.21.15.107:0/2310439709 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7eff140073d0 con 0x7eff240c24c0 2023-10-25T00:04:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.118+0000 7eff197fa640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff10065e90 0x7eff10068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.118+0000 7eff28896640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff10065e90 0x7eff10068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.118+0000 7eff197fa640 1 -- 172.21.15.107:0/2310439709 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 8509+0+0 (secure 0 0 0) 0x7eff14083b30 con 0x7eff240c24c0 2023-10-25T00:04:29.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.118+0000 7eff28896640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff10065e90 0x7eff10068350 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7eff200b0000 tx=0x7eff20008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:29.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.122+0000 7eff197fa640 1 -- 172.21.15.107:0/2310439709 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7eff2406c200 con 0x7eff240c24c0 2023-10-25T00:04:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 -- 172.21.15.107:0/2310439709 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff2406c200 con 0x7eff240c24c0 2023-10-25T00:04:29.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff197fa640 1 -- 172.21.15.107:0/2310439709 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5016 (secure 0 0 0) 0x7eff2406c200 con 0x7eff240c24c0 2023-10-25T00:04:29.296 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:29.296 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:28.593178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:29.297 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 96 2023-10-25T00:04:29.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 -- 172.21.15.107:0/2310439709 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff10065e90 msgr2=0x7eff10068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff10065e90 0x7eff10068350 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7eff200b0000 tx=0x7eff20008040 comp rx=0 tx=0).stop 2023-10-25T00:04:29.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 -- 172.21.15.107:0/2310439709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff240c24c0 msgr2=0x7eff24146b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff240c24c0 0x7eff24146b70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7eff1400c500 tx=0x7eff1400c9c0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 -- 172.21.15.107:0/2310439709 shutdown_connections 2023-10-25T00:04:29.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff10065e90 0x7eff10068350 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff24151960 0x7eff241470b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff240c24c0 0x7eff24146b70 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 --2- 172.21.15.107:0/2310439709 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff24071220 0x7eff24148eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 -- 172.21.15.107:0/2310439709 >> 172.21.15.107:0/2310439709 conn(0x7eff240c2950 msgr2=0x7eff2414e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:29.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 -- 172.21.15.107:0/2310439709 shutdown_connections 2023-10-25T00:04:29.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.294+0000 7eff2aafa640 1 -- 172.21.15.107:0/2310439709 wait complete. 2023-10-25T00:04:29.312 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2023-10-25T00:04:29.312 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:04:29.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:29.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc47a17640 1 Processor -- start 2023-10-25T00:04:29.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc47a17640 1 -- start start 2023-10-25T00:04:29.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc47a17640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdc40151960 0x7fdc40071220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc47a17640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc40150bc0 con 0x7fdc400c1860 2023-10-25T00:04:29.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc47a17640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4006c510 con 0x7fdc400c24c0 2023-10-25T00:04:29.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc47a17640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc4006c7f0 con 0x7fdc40151960 2023-10-25T00:04:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc45fb4640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdc400c1860 0x7fdc40071a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37352/0 (socket says 172.21.15.107:37352) 2023-10-25T00:04:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc45fb4640 1 -- 172.21.15.107:0/277636662 learned_addr learned my addr 172.21.15.107:0/277636662 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc457b3640 1 --2- 172.21.15.107:0/277636662 >> v2:172.21.15.110:3300/0 conn(0x7fdc40151960 0x7fdc40071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005452433 0 0) 0x7fdc40150bc0 con 0x7fdc400c1860 2023-10-25T00:04:29.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc2c003530 con 0x7fdc400c1860 2023-10-25T00:04:29.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1943139890 0 0) 0x7fdc2c003530 con 0x7fdc400c1860 2023-10-25T00:04:29.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 >> v2:172.21.15.110:3300/0 conn(0x7fdc40151960 msgr2=0x7fdc40071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:29.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 --2- 172.21.15.107:0/277636662 >> v2:172.21.15.110:3300/0 conn(0x7fdc40151960 0x7fdc40071220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 >> v1:172.21.15.110:6789/0 conn(0x7fdc400c24c0 legacy=0x7fdc4006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc40150e80 con 0x7fdc400c1860 2023-10-25T00:04:29.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fdc38001680 con 0x7fdc400c1860 2023-10-25T00:04:29.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 >> v1:172.21.15.107:6789/0 conn(0x7fdc400c1860 legacy=0x7fdc40071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc2c003980 0x7fdc2c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc2c0047d0 0x7fdc2c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 0x7fdc2c00dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4006c7f0 con 0x7fdc2c005190 2023-10-25T00:04:29.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c00e340 con 0x7fdc2c003980 2023-10-25T00:04:29.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c00e500 con 0x7fdc2c0047d0 2023-10-25T00:04:29.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc45fb4640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 0x7fdc2c00dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc457b3640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc2c003980 0x7fdc2c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc44fb2640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc2c0047d0 0x7fdc2c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc45fb4640 1 -- 172.21.15.107:0/277636662 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc2c0047d0 msgr2=0x7fdc2c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:29.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc45fb4640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc2c0047d0 0x7fdc2c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc45fb4640 1 -- 172.21.15.107:0/277636662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc2c003980 msgr2=0x7fdc2c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc45fb4640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc2c003980 0x7fdc2c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc45fb4640 1 -- 172.21.15.107:0/277636662 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c003530 con 0x7fdc2c005190 2023-10-25T00:04:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc44fb2640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc2c0047d0 0x7fdc2c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.426+0000 7fdc457b3640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc2c003980 0x7fdc2c003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc45fb4640 1 -- 172.21.15.107:0/277636662 reap_dead start 2023-10-25T00:04:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc45fb4640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 0x7fdc2c00dba0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800c240 tx=0x7fdc3801f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddca84c7be802dce server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38005ae0 con 0x7fdc2c005190 2023-10-25T00:04:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc380ab070 con 0x7fdc2c005190 2023-10-25T00:04:29.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc33fff640 1 -- 172.21.15.107:0/277636662 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38007070 con 0x7fdc2c005190 2023-10-25T00:04:29.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 -- 172.21.15.107:0/277636662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 msgr2=0x7fdc2c00dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 0x7fdc2c00dba0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800c240 tx=0x7fdc3801f3b0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 -- 172.21.15.107:0/277636662 shutdown_connections 2023-10-25T00:04:29.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 --2- 172.21.15.107:0/277636662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 0x7fdc2c00dba0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 -- 172.21.15.107:0/277636662 >> 172.21.15.107:0/277636662 conn(0x7fdc400c2950 msgr2=0x7fdc400c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:29.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 -- 172.21.15.107:0/277636662 shutdown_connections 2023-10-25T00:04:29.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 -- 172.21.15.107:0/277636662 wait complete. 2023-10-25T00:04:29.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 Processor -- start 2023-10-25T00:04:29.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 -- start start 2023-10-25T00:04:29.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 0x7fdc4014eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc401e57a0 0x7fdc400c3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc40152400 0x7fdc4014ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc40149af0 con 0x7fdc2c005190 2023-10-25T00:04:29.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc40149d40 con 0x7fdc401e57a0 2023-10-25T00:04:29.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.430+0000 7fdc47a17640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4014a020 con 0x7fdc40152400 2023-10-25T00:04:29.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc457b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 0x7fdc4014eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc457b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 0x7fdc4014eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52664/0 (socket says 172.21.15.107:52664) 2023-10-25T00:04:29.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc457b3640 1 -- 172.21.15.107:0/3528778723 learned_addr learned my addr 172.21.15.107:0/3528778723 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:29.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc44fb2640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc401e57a0 0x7fdc400c3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc45fb4640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc40152400 0x7fdc4014ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc457b3640 1 -- 172.21.15.107:0/3528778723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc40152400 msgr2=0x7fdc4014ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc457b3640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc40152400 0x7fdc4014ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc457b3640 1 -- 172.21.15.107:0/3528778723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc401e57a0 msgr2=0x7fdc400c3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc457b3640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc401e57a0 0x7fdc400c3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc457b3640 1 -- 172.21.15.107:0/3528778723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4014a300 con 0x7fdc2c005190 2023-10-25T00:04:29.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc45fb4640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc40152400 0x7fdc4014ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:29.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc457b3640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 0x7fdc4014eb30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00f350 tx=0x7fdc3c00f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:29.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc31ffb640 1 -- 172.21.15.107:0/3528778723 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c003ba0 con 0x7fdc2c005190 2023-10-25T00:04:29.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc31ffb640 1 -- 172.21.15.107:0/3528778723 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc3c011040 con 0x7fdc2c005190 2023-10-25T00:04:29.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc47a17640 1 -- 172.21.15.107:0/3528778723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4014a5e0 con 0x7fdc2c005190 2023-10-25T00:04:29.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc31ffb640 1 -- 172.21.15.107:0/3528778723 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c004020 con 0x7fdc2c005190 2023-10-25T00:04:29.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc47a17640 1 -- 172.21.15.107:0/3528778723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4006adc0 con 0x7fdc2c005190 2023-10-25T00:04:29.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc47a17640 1 -- 172.21.15.107:0/3528778723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc40072410 con 0x7fdc2c005190 2023-10-25T00:04:29.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc31ffb640 1 -- 172.21.15.107:0/3528778723 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdc3c004180 con 0x7fdc2c005190 2023-10-25T00:04:29.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc31ffb640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdc24065e60 0x7fdc24068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:29.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc31ffb640 1 -- 172.21.15.107:0/3528778723 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 8509+0+0 (secure 0 0 0) 0x7fdc3c084960 con 0x7fdc2c005190 2023-10-25T00:04:29.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc44fb2640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdc24065e60 0x7fdc24068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:29.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.434+0000 7fdc44fb2640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdc24065e60 0x7fdc24068320 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdc34007fd0 tx=0x7fdc34009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:29.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.438+0000 7fdc31ffb640 1 -- 172.21.15.107:0/3528778723 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdc3c051660 con 0x7fdc2c005190 2023-10-25T00:04:29.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.614+0000 7fdc47a17640 1 -- 172.21.15.107:0/3528778723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc400746e0 con 0x7fdc2c005190 2023-10-25T00:04:29.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.614+0000 7fdc31ffb640 1 -- 172.21.15.107:0/3528778723 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5016 (secure 0 0 0) 0x7fdc3c054c80 con 0x7fdc2c005190 2023-10-25T00:04:29.616 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:29.617 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:28.593178+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:29.617 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 96 2023-10-25T00:04:29.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.614+0000 7fdc47a17640 1 -- 172.21.15.107:0/3528778723 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdc24065e60 msgr2=0x7fdc24068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.614+0000 7fdc47a17640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdc24065e60 0x7fdc24068320 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fdc34007fd0 tx=0x7fdc34009040 comp rx=0 tx=0).stop 2023-10-25T00:04:29.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.614+0000 7fdc47a17640 1 -- 172.21.15.107:0/3528778723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 msgr2=0x7fdc4014eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.614+0000 7fdc47a17640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 0x7fdc4014eb30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00f350 tx=0x7fdc3c00f810 comp rx=0 tx=0).stop 2023-10-25T00:04:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.618+0000 7fdc47a17640 1 -- 172.21.15.107:0/3528778723 shutdown_connections 2023-10-25T00:04:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.618+0000 7fdc47a17640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdc24065e60 0x7fdc24068320 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.618+0000 7fdc47a17640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdc40152400 0x7fdc4014ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.618+0000 7fdc47a17640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdc401e57a0 0x7fdc400c3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.618+0000 7fdc47a17640 1 --2- 172.21.15.107:0/3528778723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdc2c005190 0x7fdc4014eb30 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.618+0000 7fdc47a17640 1 -- 172.21.15.107:0/3528778723 >> 172.21.15.107:0/3528778723 conn(0x7fdc400c2950 msgr2=0x7fdc4006bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.618+0000 7fdc47a17640 1 -- 172.21.15.107:0/3528778723 shutdown_connections 2023-10-25T00:04:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:29.618+0000 7fdc47a17640 1 -- 172.21.15.107:0/3528778723 wait complete. 2023-10-25T00:04:29.633 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 96, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5764": { "addr": "172.21.15.110:6834/2577776351", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 2577776351, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 2577776351, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5764, "incarnation": 86, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:03:31.705318+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5764 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 96, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6753": { "addr": "172.21.15.107:6837/1555840329", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 1555840329, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 1555840329, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6753, "incarnation": 96, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:04:28.593178+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6753 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 59, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:04:31.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:31.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.130+0000 7feffd83a640 1 Processor -- start 2023-10-25T00:04:31.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feffd83a640 1 -- start start 2023-10-25T00:04:31.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feffd83a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feff80a9820 0x7feff80ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:31.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feffd83a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feff806ee40 0x7feff80aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:31.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feffd83a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff806f730 con 0x7feff814fd40 2023-10-25T00:04:31.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feffd83a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff80aa750 con 0x7feff806ee40 2023-10-25T00:04:31.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feffd83a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff8156a40 con 0x7feff80a9820 2023-10-25T00:04:31.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feff77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feff806ee40 0x7feff80aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:31.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feff77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feff806ee40 0x7feff80aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52676/0 (socket says 172.21.15.107:52676) 2023-10-25T00:04:31.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feff77fe640 1 -- 172.21.15.107:0/531586039 learned_addr learned my addr 172.21.15.107:0/531586039 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:31.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feff77fe640 1 -- 172.21.15.107:0/531586039 >> v2:172.21.15.110:3300/0 conn(0x7feff80a9820 msgr2=0x7feff80ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:31.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feff77fe640 1 --2- 172.21.15.107:0/531586039 >> v2:172.21.15.110:3300/0 conn(0x7feff80a9820 0x7feff80ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feff77fe640 1 -- 172.21.15.107:0/531586039 >> v1:172.21.15.110:6789/0 conn(0x7feff814fd40 legacy=0x7feff80c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:31.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feff77fe640 1 -- 172.21.15.107:0/531586039 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff8156710 con 0x7feff806ee40 2023-10-25T00:04:31.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feff77fe640 1 --2- 172.21.15.107:0/531586039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff806ee40 0x7feff80aa210 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7feff806d2f0 tx=0x7feff001c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1070197b9d71e2b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:31.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feff67fc640 1 -- 172.21.15.107:0/531586039 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff00a56a0 con 0x7feff806ee40 2023-10-25T00:04:31.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feff67fc640 1 -- 172.21.15.107:0/531586039 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff00a5800 con 0x7feff806ee40 2023-10-25T00:04:31.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.134+0000 7feff67fc640 1 -- 172.21.15.107:0/531586039 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff00a5b90 con 0x7feff806ee40 2023-10-25T00:04:31.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 -- 172.21.15.107:0/531586039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff806ee40 msgr2=0x7feff80aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 --2- 172.21.15.107:0/531586039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff806ee40 0x7feff80aa210 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7feff806d2f0 tx=0x7feff001c840 comp rx=0 tx=0).stop 2023-10-25T00:04:31.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 -- 172.21.15.107:0/531586039 shutdown_connections 2023-10-25T00:04:31.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 --2- 172.21.15.107:0/531586039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff806ee40 0x7feff80aa210 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 --2- 172.21.15.107:0/531586039 >> v2:172.21.15.110:3300/0 conn(0x7feff80a9820 0x7feff80ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 -- 172.21.15.107:0/531586039 >> 172.21.15.107:0/531586039 conn(0x7feff80c2950 msgr2=0x7feff80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:31.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 -- 172.21.15.107:0/531586039 shutdown_connections 2023-10-25T00:04:31.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 -- 172.21.15.107:0/531586039 wait complete. 2023-10-25T00:04:31.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 Processor -- start 2023-10-25T00:04:31.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 -- start start 2023-10-25T00:04:31.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff806ee40 0x7feff814e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:31.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff814fd40 0x7feff8144910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:31.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feff8149c70 0x7feff81ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:31.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff81ebb10 con 0x7feff814fd40 2023-10-25T00:04:31.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff81eac40 con 0x7feff806ee40 2023-10-25T00:04:31.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff81eaf00 con 0x7feff8149c70 2023-10-25T00:04:31.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7fefee7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff814fd40 0x7feff8144910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:31.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7fefee7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff814fd40 0x7feff8144910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52680/0 (socket says 172.21.15.107:52680) 2023-10-25T00:04:31.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7fefee7fc640 1 -- 172.21.15.107:0/3735214013 learned_addr learned my addr 172.21.15.107:0/3735214013 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:31.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feff6ffd640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff806ee40 0x7feff814e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:31.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feff77fe640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feff8149c70 0x7feff81ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:31.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7fefee7fc640 1 -- 172.21.15.107:0/3735214013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feff8149c70 msgr2=0x7feff81ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7fefee7fc640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feff8149c70 0x7feff81ee3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7fefee7fc640 1 -- 172.21.15.107:0/3735214013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff806ee40 msgr2=0x7feff814e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7fefee7fc640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff806ee40 0x7feff814e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7fefee7fc640 1 -- 172.21.15.107:0/3735214013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff0009b30 con 0x7feff814fd40 2023-10-25T00:04:31.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feff77fe640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feff8149c70 0x7feff81ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7fefee7fc640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff814fd40 0x7feff8144910 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fefe000b550 tx=0x7fefe000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feff4ff9640 1 -- 172.21.15.107:0/3735214013 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0010870 con 0x7feff814fd40 2023-10-25T00:04:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feff4ff9640 1 -- 172.21.15.107:0/3735214013 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe00109f0 con 0x7feff814fd40 2023-10-25T00:04:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 -- 172.21.15.107:0/3735214013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff81eb1e0 con 0x7feff814fd40 2023-10-25T00:04:31.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feff4ff9640 1 -- 172.21.15.107:0/3735214013 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0010da0 con 0x7feff814fd40 2023-10-25T00:04:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.138+0000 7feffd83a640 1 -- 172.21.15.107:0/3735214013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff81eb7a0 con 0x7feff814fd40 2023-10-25T00:04:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.142+0000 7feffd83a640 1 -- 172.21.15.107:0/3735214013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff80c24a0 con 0x7feff814fd40 2023-10-25T00:04:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.142+0000 7feff4ff9640 1 -- 172.21.15.107:0/3735214013 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fefe000f3c0 con 0x7feff814fd40 2023-10-25T00:04:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.142+0000 7feff4ff9640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd8065e60 0x7fefd8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:31.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.142+0000 7feff6ffd640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd8065e60 0x7fefd8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:31.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.142+0000 7feff4ff9640 1 -- 172.21.15.107:0/3735214013 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 8509+0+0 (secure 0 0 0) 0x7fefe0084200 con 0x7feff814fd40 2023-10-25T00:04:31.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.142+0000 7feff6ffd640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd8065e60 0x7fefd8068320 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fefe800b810 tx=0x7fefe8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:31.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.146+0000 7feff4ff9640 1 -- 172.21.15.107:0/3735214013 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fefe0050e70 con 0x7feff814fd40 2023-10-25T00:04:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.334+0000 7feffd83a640 1 -- 172.21.15.107:0/3735214013 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feff80746e0 con 0x7feff814fd40 2023-10-25T00:04:31.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feff4ff9640 1 -- 172.21.15.107:0/3735214013 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+5016 (secure 0 0 0) 0x7fefe0014090 con 0x7feff814fd40 2023-10-25T00:04:31.339 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:31.339 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:30.692411+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:31.339 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 98 2023-10-25T00:04:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 -- 172.21.15.107:0/3735214013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd8065e60 msgr2=0x7fefd8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd8065e60 0x7fefd8068320 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fefe800b810 tx=0x7fefe8008040 comp rx=0 tx=0).stop 2023-10-25T00:04:31.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 -- 172.21.15.107:0/3735214013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff814fd40 msgr2=0x7feff8144910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff814fd40 0x7feff8144910 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fefe000b550 tx=0x7fefe000ba10 comp rx=0 tx=0).stop 2023-10-25T00:04:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 -- 172.21.15.107:0/3735214013 shutdown_connections 2023-10-25T00:04:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fefd8065e60 0x7fefd8068320 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feff8149c70 0x7feff81ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff814fd40 0x7feff8144910 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 --2- 172.21.15.107:0/3735214013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feff806ee40 0x7feff814e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 -- 172.21.15.107:0/3735214013 >> 172.21.15.107:0/3735214013 conn(0x7feff80c2950 msgr2=0x7feff80c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 -- 172.21.15.107:0/3735214013 shutdown_connections 2023-10-25T00:04:31.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.338+0000 7feffd83a640 1 -- 172.21.15.107:0/3735214013 wait complete. 2023-10-25T00:04:31.357 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:04:31.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:31.737 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:04:31.734+0000 7f716d249640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:04:31.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9c65eb640 1 Processor -- start 2023-10-25T00:04:31.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9c65eb640 1 -- start start 2023-10-25T00:04:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9c65eb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9c0153190 0x7ff9c00c1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9c65eb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c0072330 con 0x7ff9c00c5c20 2023-10-25T00:04:31.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9c65eb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9c01535c0 con 0x7ff9c014fc30 2023-10-25T00:04:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9c65eb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9c00c24f0 con 0x7ff9c0153190 2023-10-25T00:04:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9c4b88640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9c0153190 0x7ff9c00c1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9bf7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff9c00c5c20 0x7ff9c00c18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37358/0 (socket says 172.21.15.107:37358) 2023-10-25T00:04:31.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9bf7fe640 1 -- 172.21.15.107:0/1705550912 learned_addr learned my addr 172.21.15.107:0/1705550912 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9c4b88640 1 -- 172.21.15.107:0/1705550912 >> v1:172.21.15.110:6789/0 conn(0x7ff9c014fc30 legacy=0x7ff9c006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9c4b88640 1 -- 172.21.15.107:0/1705550912 >> v1:172.21.15.107:6789/0 conn(0x7ff9c00c5c20 legacy=0x7ff9c00c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9c4b88640 1 -- 172.21.15.107:0/1705550912 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c0155ca0 con 0x7ff9c0153190 2023-10-25T00:04:31.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.770+0000 7ff9beffd640 1 -- 172.21.15.107:0/1705550912 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890537605 0 0) 0x7ff9c0072330 con 0x7ff9c00c5c20 2023-10-25T00:04:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c4b88640 1 --2- 172.21.15.107:0/1705550912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9c0153190 0x7ff9c00c1fb0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009d90 tx=0x7ff9b801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360dc75a2cffaa68 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:31.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9beffd640 1 -- 172.21.15.107:0/1705550912 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b80a5950 con 0x7ff9c0153190 2023-10-25T00:04:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9beffd640 1 -- 172.21.15.107:0/1705550912 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b80a5ab0 con 0x7ff9c0153190 2023-10-25T00:04:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9beffd640 1 -- 172.21.15.107:0/1705550912 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b80a5e40 con 0x7ff9c0153190 2023-10-25T00:04:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 -- 172.21.15.107:0/1705550912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9c0153190 msgr2=0x7ff9c00c1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 --2- 172.21.15.107:0/1705550912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9c0153190 0x7ff9c00c1fb0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009d90 tx=0x7ff9b801c890 comp rx=0 tx=0).stop 2023-10-25T00:04:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 -- 172.21.15.107:0/1705550912 shutdown_connections 2023-10-25T00:04:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 --2- 172.21.15.107:0/1705550912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9c0153190 0x7ff9c00c1fb0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 -- 172.21.15.107:0/1705550912 >> 172.21.15.107:0/1705550912 conn(0x7ff9c00c39a0 msgr2=0x7ff9c00c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 -- 172.21.15.107:0/1705550912 shutdown_connections 2023-10-25T00:04:31.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 -- 172.21.15.107:0/1705550912 wait complete. 2023-10-25T00:04:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 Processor -- start 2023-10-25T00:04:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 -- start start 2023-10-25T00:04:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9c00c5c20 0x7ff9c00bcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:31.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9c014fc30 0x7ff9c01eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9c0153190 0x7ff9c0151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c00c0440 con 0x7ff9c00c5c20 2023-10-25T00:04:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c015dee0 con 0x7ff9c014fc30 2023-10-25T00:04:31.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c01eaf80 con 0x7ff9c0153190 2023-10-25T00:04:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9c00c5c20 0x7ff9c00bcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9c00c5c20 0x7ff9c00bcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52712/0 (socket says 172.21.15.107:52712) 2023-10-25T00:04:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bffff640 1 -- 172.21.15.107:0/3968895170 learned_addr learned my addr 172.21.15.107:0/3968895170 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bf7fe640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9c014fc30 0x7ff9c01eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c4b88640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9c0153190 0x7ff9c0151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bffff640 1 -- 172.21.15.107:0/3968895170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9c0153190 msgr2=0x7ff9c0151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bffff640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9c0153190 0x7ff9c0151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bffff640 1 -- 172.21.15.107:0/3968895170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9c014fc30 msgr2=0x7ff9c01eaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bffff640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9c014fc30 0x7ff9c01eaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bffff640 1 -- 172.21.15.107:0/3968895170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b8009a70 con 0x7ff9c00c5c20 2023-10-25T00:04:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bf7fe640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9c014fc30 0x7ff9c01eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bffff640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9c00c5c20 0x7ff9c00bcfb0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000c500 tx=0x7ff9b000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:31.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bcff9640 1 -- 172.21.15.107:0/3968895170 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0005220 con 0x7ff9c00c5c20 2023-10-25T00:04:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bcff9640 1 -- 172.21.15.107:0/3968895170 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b0005380 con 0x7ff9c00c5c20 2023-10-25T00:04:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 -- 172.21.15.107:0/3968895170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c01ed530 con 0x7ff9c00c5c20 2023-10-25T00:04:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9bcff9640 1 -- 172.21.15.107:0/3968895170 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0007270 con 0x7ff9c00c5c20 2023-10-25T00:04:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.774+0000 7ff9c65eb640 1 -- 172.21.15.107:0/3968895170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c01eddd0 con 0x7ff9c00c5c20 2023-10-25T00:04:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.778+0000 7ff9c65eb640 1 -- 172.21.15.107:0/3968895170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c00746e0 con 0x7ff9c00c5c20 2023-10-25T00:04:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.778+0000 7ff9bcff9640 1 -- 172.21.15.107:0/3968895170 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff9b00073d0 con 0x7ff9c00c5c20 2023-10-25T00:04:31.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.778+0000 7ff9bcff9640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9ac065e60 0x7ff9ac068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:31.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.778+0000 7ff9bcff9640 1 -- 172.21.15.107:0/3968895170 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 8509+0+0 (secure 0 0 0) 0x7ff9b00848a0 con 0x7ff9c00c5c20 2023-10-25T00:04:31.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.778+0000 7ff9bf7fe640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9ac065e60 0x7ff9ac068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:31.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.778+0000 7ff9bf7fe640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9ac065e60 0x7ff9ac068320 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff9c01ee630 tx=0x7ff9b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:31.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.782+0000 7ff9bcff9640 1 -- 172.21.15.107:0/3968895170 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff9b00515a0 con 0x7ff9c00c5c20 2023-10-25T00:04:31.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.970+0000 7ff9c65eb640 1 -- 172.21.15.107:0/3968895170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9c00c5a00 con 0x7ff9c00c5c20 2023-10-25T00:04:31.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9bcff9640 1 -- 172.21.15.107:0/3968895170 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5016 (secure 0 0 0) 0x7ff9b0054bc0 con 0x7ff9c00c5c20 2023-10-25T00:04:31.975 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:31.975 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 99 2023-10-25T00:04:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 -- 172.21.15.107:0/3968895170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9ac065e60 msgr2=0x7ff9ac068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9ac065e60 0x7ff9ac068320 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff9c01ee630 tx=0x7ff9b4009040 comp rx=0 tx=0).stop 2023-10-25T00:04:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 -- 172.21.15.107:0/3968895170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9c00c5c20 msgr2=0x7ff9c00bcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9c00c5c20 0x7ff9c00bcfb0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000c500 tx=0x7ff9b000c9c0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 -- 172.21.15.107:0/3968895170 shutdown_connections 2023-10-25T00:04:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff9ac065e60 0x7ff9ac068320 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9c0153190 0x7ff9c0151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9c014fc30 0x7ff9c01eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 --2- 172.21.15.107:0/3968895170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9c00c5c20 0x7ff9c00bcfb0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 -- 172.21.15.107:0/3968895170 >> 172.21.15.107:0/3968895170 conn(0x7ff9c00c39a0 msgr2=0x7ff9c00ab950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 -- 172.21.15.107:0/3968895170 shutdown_connections 2023-10-25T00:04:31.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:31.974+0000 7ff9c65eb640 1 -- 172.21.15.107:0/3968895170 wait complete. 2023-10-25T00:04:31.993 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=6672 2023-10-25T00:04:31.994 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.c 2023-10-25T00:04:33.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:33.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.486+0000 7f8621290640 1 Processor -- start 2023-10-25T00:04:33.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.486+0000 7f8621290640 1 -- start start 2023-10-25T00:04:33.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f8621290640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f861c14b600 0x7f861c151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:33.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f8621290640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f861c06af70 0x7f861c06c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:33.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f8621290640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c0c1860 con 0x7f861c06a230 2023-10-25T00:04:33.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f8621290640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f861c154d50 con 0x7f861c06af70 2023-10-25T00:04:33.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f8621290640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861c1524f0 con 0x7f861c14b600 2023-10-25T00:04:33.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f861a7fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f861c06af70 0x7f861c06c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:33.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f861b7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f861c06a230 0x7f861c1551e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:60412/0 (socket says 172.21.15.107:60412) 2023-10-25T00:04:33.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f861b7fe640 1 -- 172.21.15.107:0/2127867475 learned_addr learned my addr 172.21.15.107:0/2127867475 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:33.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f861affd640 1 --2- 172.21.15.107:0/2127867475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f861c14b600 0x7f861c151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:33.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f861a7fc640 1 -- 172.21.15.107:0/2127867475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f861c14b600 msgr2=0x7f861c151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:33.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f861a7fc640 1 --2- 172.21.15.107:0/2127867475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f861c14b600 0x7f861c151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:33.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f861a7fc640 1 -- 172.21.15.107:0/2127867475 >> v1:172.21.15.110:6789/0 conn(0x7f861c06a230 legacy=0x7f861c1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:33.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.490+0000 7f861a7fc640 1 -- 172.21.15.107:0/2127867475 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c14b9e0 con 0x7f861c06af70 2023-10-25T00:04:33.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861a7fc640 1 --2- 172.21.15.107:0/2127867475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f861c06af70 0x7f861c06c4c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f86100097c0 tx=0x7f861001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4266ff37e6960fd server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:33.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8619ffb640 1 -- 172.21.15.107:0/2127867475 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86100ac620 con 0x7f861c06af70 2023-10-25T00:04:33.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8619ffb640 1 -- 172.21.15.107:0/2127867475 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86100ac780 con 0x7f861c06af70 2023-10-25T00:04:33.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8619ffb640 1 -- 172.21.15.107:0/2127867475 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86100acb10 con 0x7f861c06af70 2023-10-25T00:04:33.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 -- 172.21.15.107:0/2127867475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f861c06af70 msgr2=0x7f861c06c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:33.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 --2- 172.21.15.107:0/2127867475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f861c06af70 0x7f861c06c4c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f86100097c0 tx=0x7f861001c3c0 comp rx=0 tx=0).stop 2023-10-25T00:04:33.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 -- 172.21.15.107:0/2127867475 shutdown_connections 2023-10-25T00:04:33.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 --2- 172.21.15.107:0/2127867475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f861c06af70 0x7f861c06c4c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:33.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 --2- 172.21.15.107:0/2127867475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f861c14b600 0x7f861c151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:33.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 -- 172.21.15.107:0/2127867475 >> 172.21.15.107:0/2127867475 conn(0x7f861c0c2950 msgr2=0x7f861c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:33.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 -- 172.21.15.107:0/2127867475 shutdown_connections 2023-10-25T00:04:33.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 -- 172.21.15.107:0/2127867475 wait complete. 2023-10-25T00:04:33.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 Processor -- start 2023-10-25T00:04:33.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 -- start start 2023-10-25T00:04:33.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f861c06a230 0x7f861c13c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:33.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f861c06af70 0x7f861c149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:33.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861c14b600 0x7f861c140440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:33.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c149110 con 0x7f861c14b600 2023-10-25T00:04:33.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c147210 con 0x7f861c06af70 2023-10-25T00:04:33.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f8621290640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861c1ea660 con 0x7f861c06a230 2023-10-25T00:04:33.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861c14b600 0x7f861c140440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:33.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861c14b600 0x7f861c140440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58136/0 (socket says 172.21.15.107:58136) 2023-10-25T00:04:33.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861b7fe640 1 -- 172.21.15.107:0/82365949 learned_addr learned my addr 172.21.15.107:0/82365949 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:33.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861a7fc640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f861c06af70 0x7f861c149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:33.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861affd640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f861c06a230 0x7f861c13c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:33.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861b7fe640 1 -- 172.21.15.107:0/82365949 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f861c06a230 msgr2=0x7f861c13c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:33.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861b7fe640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f861c06a230 0x7f861c13c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861b7fe640 1 -- 172.21.15.107:0/82365949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f861c06af70 msgr2=0x7f861c149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861b7fe640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f861c06af70 0x7f861c149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861b7fe640 1 -- 172.21.15.107:0/82365949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8610009420 con 0x7f861c14b600 2023-10-25T00:04:33.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.494+0000 7f861affd640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f861c06a230 0x7f861c13c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:04:33.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f861b7fe640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861c14b600 0x7f861c140440 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f861400caf0 tx=0x7f861400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:33.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f86037fe640 1 -- 172.21.15.107:0/82365949 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861400aca0 con 0x7f861c14b600 2023-10-25T00:04:33.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f86037fe640 1 -- 172.21.15.107:0/82365949 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861400ae00 con 0x7f861c14b600 2023-10-25T00:04:33.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f86037fe640 1 -- 172.21.15.107:0/82365949 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8614012600 con 0x7f861c14b600 2023-10-25T00:04:33.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f8621290640 1 -- 172.21.15.107:0/82365949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861c1ea920 con 0x7f861c14b600 2023-10-25T00:04:33.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f8621290640 1 -- 172.21.15.107:0/82365949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f861c1eaee0 con 0x7f861c14b600 2023-10-25T00:04:33.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f8621290640 1 -- 172.21.15.107:0/82365949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f861c0746e0 con 0x7f861c14b600 2023-10-25T00:04:33.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f86037fe640 1 -- 172.21.15.107:0/82365949 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8614007070 con 0x7f861c14b600 2023-10-25T00:04:33.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f86037fe640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8608065e60 0x7f8608068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:33.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f86037fe640 1 -- 172.21.15.107:0/82365949 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 8509+0+0 (secure 0 0 0) 0x7f8614083f50 con 0x7f861c14b600 2023-10-25T00:04:33.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f861affd640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8608065e60 0x7f8608068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:33.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.498+0000 7f861affd640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8608065e60 0x7f8608068320 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f860c009b10 tx=0x7f860c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:33.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.502+0000 7f86037fe640 1 -- 172.21.15.107:0/82365949 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8614050d00 con 0x7f861c14b600 2023-10-25T00:04:33.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.690+0000 7f8621290640 1 -- 172.21.15.107:0/82365949 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f861c069fa0 con 0x7f861c14b600 2023-10-25T00:04:33.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.690+0000 7f86037fe640 1 -- 172.21.15.107:0/82365949 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5016 (secure 0 0 0) 0x7f8614017070 con 0x7f861c14b600 2023-10-25T00:04:33.694 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:33.695 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:33.696 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 99 2023-10-25T00:04:33.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 -- 172.21.15.107:0/82365949 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8608065e60 msgr2=0x7f8608068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:33.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8608065e60 0x7f8608068320 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f860c009b10 tx=0x7f860c008040 comp rx=0 tx=0).stop 2023-10-25T00:04:33.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 -- 172.21.15.107:0/82365949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861c14b600 msgr2=0x7f861c140440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861c14b600 0x7f861c140440 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f861400caf0 tx=0x7f861400a060 comp rx=0 tx=0).stop 2023-10-25T00:04:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 -- 172.21.15.107:0/82365949 shutdown_connections 2023-10-25T00:04:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861c14b600 0x7f861c140440 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f861c06af70 0x7f861c149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:33.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8608065e60 0x7f8608068320 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 --2- 172.21.15.107:0/82365949 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f861c06a230 0x7f861c13c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 -- 172.21.15.107:0/82365949 >> 172.21.15.107:0/82365949 conn(0x7f861c0c2950 msgr2=0x7f861c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 -- 172.21.15.107:0/82365949 shutdown_connections 2023-10-25T00:04:33.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:33.694+0000 7f8621290640 1 -- 172.21.15.107:0/82365949 wait complete. 2023-10-25T00:04:33.713 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:04:33.995 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2023-10-25T00:04:33.995 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:04:33.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:04:33.998 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:04:33.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:34.077 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:04:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.114+0000 7fd54d858640 1 Processor -- start 2023-10-25T00:04:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.114+0000 7fd54d858640 1 -- start start 2023-10-25T00:04:34.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd54d858640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd548152250 0x7fd5480c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:34.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd54d858640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5480bb5c0 con 0x7fd5480bc6b0 2023-10-25T00:04:34.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd54d858640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd548065be0 con 0x7fd5480ac0d0 2023-10-25T00:04:34.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd54d858640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5480bbaf0 con 0x7fd548152250 2023-10-25T00:04:34.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd546ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd548152250 0x7fd5480c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:34.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd5467fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd5480bc6b0 0x7fd5481580c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36406/0 (socket says 172.21.15.107:36406) 2023-10-25T00:04:34.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd5467fc640 1 -- 172.21.15.107:0/3939626383 learned_addr learned my addr 172.21.15.107:0/3939626383 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:34.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd545ffb640 1 -- 172.21.15.107:0/3939626383 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979579667 0 0) 0x7fd5480bb5c0 con 0x7fd5480bc6b0 2023-10-25T00:04:34.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd545ffb640 1 -- 172.21.15.107:0/3939626383 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd534003530 con 0x7fd5480bc6b0 2023-10-25T00:04:34.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd546ffd640 1 -- 172.21.15.107:0/3939626383 >> v1:172.21.15.110:6789/0 conn(0x7fd5480ac0d0 legacy=0x7fd54806d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:34.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd546ffd640 1 -- 172.21.15.107:0/3939626383 >> v1:172.21.15.107:6789/0 conn(0x7fd5480bc6b0 legacy=0x7fd5481580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:34.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd546ffd640 1 -- 172.21.15.107:0/3939626383 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd548065ec0 con 0x7fd548152250 2023-10-25T00:04:34.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd546ffd640 1 --2- 172.21.15.107:0/3939626383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd548152250 0x7fd5480c2bc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fd540009d90 tx=0x7fd54001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6547bf42fd9e4280 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:34.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd545ffb640 1 -- 172.21.15.107:0/3939626383 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5400a5a00 con 0x7fd548152250 2023-10-25T00:04:34.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd545ffb640 1 -- 172.21.15.107:0/3939626383 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5400a5b60 con 0x7fd548152250 2023-10-25T00:04:34.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.118+0000 7fd545ffb640 1 -- 172.21.15.107:0/3939626383 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5400ad440 con 0x7fd548152250 2023-10-25T00:04:34.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- 172.21.15.107:0/3939626383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd548152250 msgr2=0x7fd5480c2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:34.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 --2- 172.21.15.107:0/3939626383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd548152250 0x7fd5480c2bc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fd540009d90 tx=0x7fd54001c920 comp rx=0 tx=0).stop 2023-10-25T00:04:34.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- 172.21.15.107:0/3939626383 shutdown_connections 2023-10-25T00:04:34.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 --2- 172.21.15.107:0/3939626383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd548152250 0x7fd5480c2bc0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:34.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- 172.21.15.107:0/3939626383 >> 172.21.15.107:0/3939626383 conn(0x7fd548069d40 msgr2=0x7fd54806a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:34.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- 172.21.15.107:0/3939626383 shutdown_connections 2023-10-25T00:04:34.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- 172.21.15.107:0/3939626383 wait complete. 2023-10-25T00:04:34.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 Processor -- start 2023-10-25T00:04:34.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- start start 2023-10-25T00:04:34.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5480ac0d0 0x7fd5481f0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:34.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5480bc6b0 0x7fd5481537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:34.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd548152250 0x7fd548153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:34.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd548155a30 con 0x7fd548152250 2023-10-25T00:04:34.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd548154230 con 0x7fd5480ac0d0 2023-10-25T00:04:34.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5481f0c40 con 0x7fd5480bc6b0 2023-10-25T00:04:34.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd5477fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd548152250 0x7fd548153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:34.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd5477fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd548152250 0x7fd548153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58182/0 (socket says 172.21.15.107:58182) 2023-10-25T00:04:34.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd5477fe640 1 -- 172.21.15.107:0/832159598 learned_addr learned my addr 172.21.15.107:0/832159598 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:34.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd546ffd640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5480ac0d0 0x7fd5481f0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:34.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd5467fc640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5480bc6b0 0x7fd5481537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:34.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd5477fe640 1 -- 172.21.15.107:0/832159598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5480bc6b0 msgr2=0x7fd5481537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:34.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd5477fe640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5480bc6b0 0x7fd5481537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:34.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd5477fe640 1 -- 172.21.15.107:0/832159598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5480ac0d0 msgr2=0x7fd5481f0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:34.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd5477fe640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5480ac0d0 0x7fd5481f0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:34.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd5477fe640 1 -- 172.21.15.107:0/832159598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540009a70 con 0x7fd548152250 2023-10-25T00:04:34.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd546ffd640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5480ac0d0 0x7fd5481f0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:34.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd5477fe640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd548152250 0x7fd548153cf0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd53800c460 tx=0x7fd53800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:34.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd53ffff640 1 -- 172.21.15.107:0/832159598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd538005220 con 0x7fd548152250 2023-10-25T00:04:34.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd53ffff640 1 -- 172.21.15.107:0/832159598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd538005380 con 0x7fd548152250 2023-10-25T00:04:34.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- 172.21.15.107:0/832159598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5481f0f20 con 0x7fd548152250 2023-10-25T00:04:34.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd53ffff640 1 -- 172.21.15.107:0/832159598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd538007270 con 0x7fd548152250 2023-10-25T00:04:34.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- 172.21.15.107:0/832159598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd548155750 con 0x7fd548152250 2023-10-25T00:04:34.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.122+0000 7fd54d858640 1 -- 172.21.15.107:0/832159598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5480c7680 con 0x7fd548152250 2023-10-25T00:04:34.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.126+0000 7fd53ffff640 1 -- 172.21.15.107:0/832159598 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd5380073d0 con 0x7fd548152250 2023-10-25T00:04:34.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.126+0000 7fd53ffff640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd528065e60 0x7fd528068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:34.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.126+0000 7fd53ffff640 1 -- 172.21.15.107:0/832159598 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 8509+0+0 (secure 0 0 0) 0x7fd538083a50 con 0x7fd548152250 2023-10-25T00:04:34.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.126+0000 7fd546ffd640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd528065e60 0x7fd528068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:34.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.126+0000 7fd546ffd640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd528065e60 0x7fd528068320 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd5400af000 tx=0x7fd5400a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:34.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.130+0000 7fd53ffff640 1 -- 172.21.15.107:0/832159598 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd538050750 con 0x7fd548152250 2023-10-25T00:04:34.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.314+0000 7fd54d858640 1 -- 172.21.15.107:0/832159598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5480746e0 con 0x7fd548152250 2023-10-25T00:04:34.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd53ffff640 1 -- 172.21.15.107:0/832159598 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+5016 (secure 0 0 0) 0x7fd538053d70 con 0x7fd548152250 2023-10-25T00:04:34.318 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:34.319 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:03:31.705318+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5764":{"gid":5764,"name":"b","rank":0,"incarnation":86,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/2577776351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":2577776351},{"type":"v1","addr":"172.21.15.110:6834","nonce":2577776351}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:34.319 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 99 2023-10-25T00:04:34.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 -- 172.21.15.107:0/832159598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd528065e60 msgr2=0x7fd528068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:34.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd528065e60 0x7fd528068320 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd5400af000 tx=0x7fd5400a7070 comp rx=0 tx=0).stop 2023-10-25T00:04:34.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 -- 172.21.15.107:0/832159598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd548152250 msgr2=0x7fd548153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:34.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd548152250 0x7fd548153cf0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd53800c460 tx=0x7fd53800c920 comp rx=0 tx=0).stop 2023-10-25T00:04:34.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 -- 172.21.15.107:0/832159598 shutdown_connections 2023-10-25T00:04:34.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd548152250 0x7fd548153cf0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:34.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd528065e60 0x7fd528068320 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:34.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5480bc6b0 0x7fd5481537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:34.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 --2- 172.21.15.107:0/832159598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5480ac0d0 0x7fd5481f0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:34.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 -- 172.21.15.107:0/832159598 >> 172.21.15.107:0/832159598 conn(0x7fd548069d40 msgr2=0x7fd54806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:34.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 -- 172.21.15.107:0/832159598 shutdown_connections 2023-10-25T00:04:34.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:34.318+0000 7fd54d858640 1 -- 172.21.15.107:0/832159598 wait complete. 2023-10-25T00:04:34.337 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-25T00:04:35.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:35.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.842+0000 7f7ee8b8b640 1 Processor -- start 2023-10-25T00:04:35.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.842+0000 7f7ee8b8b640 1 -- start start 2023-10-25T00:04:35.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.842+0000 7f7ee8b8b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7ee4151fb0 0x7f7ee4155560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:35.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.842+0000 7f7ee8b8b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee4152390 con 0x7f7ee40c82e0 2023-10-25T00:04:35.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.842+0000 7f7ee8b8b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee40c5630 con 0x7f7ee4158bf0 2023-10-25T00:04:35.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.842+0000 7f7ee8b8b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee4158760 con 0x7f7ee4151fb0 2023-10-25T00:04:35.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.842+0000 7f7ee259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7ee4151fb0 0x7f7ee4155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:35.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.842+0000 7f7ee2d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7ee40c82e0 0x7f7ee40c6bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36414/0 (socket says 172.21.15.107:36414) 2023-10-25T00:04:35.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.846+0000 7f7ee2d9d640 1 -- 172.21.15.107:0/1872349442 learned_addr learned my addr 172.21.15.107:0/1872349442 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:35.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.846+0000 7f7ee259c640 1 -- 172.21.15.107:0/1872349442 >> v1:172.21.15.110:6789/0 conn(0x7f7ee4158bf0 legacy=0x7f7ee40c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:35.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.846+0000 7f7ee259c640 1 -- 172.21.15.107:0/1872349442 >> v1:172.21.15.107:6789/0 conn(0x7f7ee40c82e0 legacy=0x7f7ee40c6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:35.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.846+0000 7f7ee159a640 1 -- 172.21.15.107:0/1872349442 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579452783 0 0) 0x7f7edc0014a0 con 0x7f7ee40c82e0 2023-10-25T00:04:35.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.846+0000 7f7ee259c640 1 -- 172.21.15.107:0/1872349442 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee40c5350 con 0x7f7ee4151fb0 2023-10-25T00:04:35.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.846+0000 7f7ee259c640 1 --2- 172.21.15.107:0/1872349442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ee4151fb0 0x7f7ee4155560 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40097c0 tx=0x7f7ed401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730d5507945f28ed server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:35.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.846+0000 7f7ee159a640 1 -- 172.21.15.107:0/1872349442 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed40ab600 con 0x7f7ee4151fb0 2023-10-25T00:04:35.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.846+0000 7f7ee159a640 1 -- 172.21.15.107:0/1872349442 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed40a7050 con 0x7f7ee4151fb0 2023-10-25T00:04:35.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.846+0000 7f7ee159a640 1 -- 172.21.15.107:0/1872349442 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed40ab980 con 0x7f7ee4151fb0 2023-10-25T00:04:35.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/1872349442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ee4151fb0 msgr2=0x7f7ee4155560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:35.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 --2- 172.21.15.107:0/1872349442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ee4151fb0 0x7f7ee4155560 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40097c0 tx=0x7f7ed401c3c0 comp rx=0 tx=0).stop 2023-10-25T00:04:35.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/1872349442 shutdown_connections 2023-10-25T00:04:35.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 --2- 172.21.15.107:0/1872349442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ee4151fb0 0x7f7ee4155560 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:35.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/1872349442 >> 172.21.15.107:0/1872349442 conn(0x7f7ee40abcf0 msgr2=0x7f7ee40ac100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:35.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/1872349442 shutdown_connections 2023-10-25T00:04:35.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/1872349442 wait complete. 2023-10-25T00:04:35.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 Processor -- start 2023-10-25T00:04:35.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 -- start start 2023-10-25T00:04:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ee40c82e0 0x7f7ee4152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:35.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ee4151fb0 0x7f7ee40aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ee4158bf0 0x7f7ee41565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee41561d0 con 0x7f7ee4158bf0 2023-10-25T00:04:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee40aba70 con 0x7f7ee40c82e0 2023-10-25T00:04:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee4155e30 con 0x7f7ee4151fb0 2023-10-25T00:04:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ee4158bf0 0x7f7ee41565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee2d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ee4158bf0 0x7f7ee41565f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58196/0 (socket says 172.21.15.107:58196) 2023-10-25T00:04:35.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee2d9d640 1 -- 172.21.15.107:0/2254902600 learned_addr learned my addr 172.21.15.107:0/2254902600 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee259c640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ee40c82e0 0x7f7ee4152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee1d9b640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ee4151fb0 0x7f7ee40aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee2d9d640 1 -- 172.21.15.107:0/2254902600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ee4151fb0 msgr2=0x7f7ee40aaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee2d9d640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ee4151fb0 0x7f7ee40aaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee2d9d640 1 -- 172.21.15.107:0/2254902600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ee40c82e0 msgr2=0x7f7ee4152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee2d9d640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ee40c82e0 0x7f7ee4152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee2d9d640 1 -- 172.21.15.107:0/2254902600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed4009420 con 0x7f7ee4158bf0 2023-10-25T00:04:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee2d9d640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ee4158bf0 0x7f7ee41565f0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00c730 tx=0x7f7edc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:35.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ed2ffd640 1 -- 172.21.15.107:0/2254902600 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00ce40 con 0x7f7ee4158bf0 2023-10-25T00:04:35.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ed2ffd640 1 -- 172.21.15.107:0/2254902600 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7edc00a640 con 0x7f7ee4158bf0 2023-10-25T00:04:35.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/2254902600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee41ed860 con 0x7f7ee4158bf0 2023-10-25T00:04:35.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ed2ffd640 1 -- 172.21.15.107:0/2254902600 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00aa30 con 0x7f7ee4158bf0 2023-10-25T00:04:35.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.850+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/2254902600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee40ab530 con 0x7f7ee4158bf0 2023-10-25T00:04:35.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.854+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/2254902600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee4156310 con 0x7f7ee4158bf0 2023-10-25T00:04:35.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.854+0000 7f7ed2ffd640 1 -- 172.21.15.107:0/2254902600 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7edc00ab90 con 0x7f7ee4158bf0 2023-10-25T00:04:35.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.854+0000 7f7ed2ffd640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ec4065e60 0x7f7ec4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:35.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.854+0000 7f7ee259c640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ec4065e60 0x7f7ec4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:35.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.854+0000 7f7ed2ffd640 1 -- 172.21.15.107:0/2254902600 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f7edc083970 con 0x7f7ee4158bf0 2023-10-25T00:04:35.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.854+0000 7f7ee259c640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ec4065e60 0x7f7ec4068320 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40ad000 tx=0x7f7ed40a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:35.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:35.858+0000 7f7ed2ffd640 1 -- 172.21.15.107:0/2254902600 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7edc050630 con 0x7f7ee4158bf0 2023-10-25T00:04:36.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/2254902600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ee4152390 con 0x7f7ee4158bf0 2023-10-25T00:04:36.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ed2ffd640 1 -- 172.21.15.107:0/2254902600 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5021 (secure 0 0 0) 0x7f7ee4152390 con 0x7f7ee4158bf0 2023-10-25T00:04:36.048 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:36.048 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:35.748026+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:36.048 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 102 2023-10-25T00:04:36.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/2254902600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ec4065e60 msgr2=0x7f7ec4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ec4065e60 0x7f7ec4068320 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7ed40ad000 tx=0x7f7ed40a6000 comp rx=0 tx=0).stop 2023-10-25T00:04:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/2254902600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ee4158bf0 msgr2=0x7f7ee41565f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ee4158bf0 0x7f7ee41565f0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00c730 tx=0x7f7edc00cbf0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/2254902600 shutdown_connections 2023-10-25T00:04:36.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ee4158bf0 0x7f7ee41565f0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ec4065e60 0x7f7ec4068320 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ee4151fb0 0x7f7ee40aaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 --2- 172.21.15.107:0/2254902600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ee40c82e0 0x7f7ee4152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/2254902600 >> 172.21.15.107:0/2254902600 conn(0x7f7ee40abcf0 msgr2=0x7f7ee40c4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:36.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/2254902600 shutdown_connections 2023-10-25T00:04:36.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.046+0000 7f7ee8b8b640 1 -- 172.21.15.107:0/2254902600 wait complete. 2023-10-25T00:04:36.068 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2023-10-25T00:04:36.068 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:04:36.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:36.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffc0db640 1 Processor -- start 2023-10-25T00:04:36.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffc0db640 1 -- start start 2023-10-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffc0db640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faff406d8d0 0x7faff40a8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffc0db640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faff4153510 0x7faff415d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffc0db640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff40bb940 con 0x7faff406aaa0 2023-10-25T00:04:36.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffc0db640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff40ac7c0 con 0x7faff4153510 2023-10-25T00:04:36.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffc0db640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff40a9220 con 0x7faff406d8d0 2023-10-25T00:04:36.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffa678640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faff4153510 0x7faff415d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faff9676640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7faff406aaa0 0x7faff40c25d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36418/0 (socket says 172.21.15.107:36418) 2023-10-25T00:04:36.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faff9676640 1 -- 172.21.15.107:0/3276248229 learned_addr learned my addr 172.21.15.107:0/3276248229 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:36.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffa678640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faff4153510 0x7faff415d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58198/0 (socket says 172.21.15.107:58198) 2023-10-25T00:04:36.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faff9e77640 1 --2- 172.21.15.107:0/3276248229 >> v2:172.21.15.110:3300/0 conn(0x7faff406d8d0 0x7faff40a8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffa678640 1 -- 172.21.15.107:0/3276248229 >> v2:172.21.15.110:3300/0 conn(0x7faff406d8d0 msgr2=0x7faff40a8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffa678640 1 --2- 172.21.15.107:0/3276248229 >> v2:172.21.15.110:3300/0 conn(0x7faff406d8d0 0x7faff40a8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffa678640 1 -- 172.21.15.107:0/3276248229 >> v1:172.21.15.107:6789/0 conn(0x7faff406aaa0 legacy=0x7faff40c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.198+0000 7faffa678640 1 -- 172.21.15.107:0/3276248229 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff40abcb0 con 0x7faff4153510 2023-10-25T00:04:36.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffa678640 1 --2- 172.21.15.107:0/3276248229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff4153510 0x7faff415d110 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7faff40695d0 tx=0x7fafec01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bad6ffb006ffc5b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:36.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faff8e75640 1 -- 172.21.15.107:0/3276248229 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec0a59c0 con 0x7faff4153510 2023-10-25T00:04:36.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faff8e75640 1 -- 172.21.15.107:0/3276248229 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafec0a5b20 con 0x7faff4153510 2023-10-25T00:04:36.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faff8e75640 1 -- 172.21.15.107:0/3276248229 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec0ad440 con 0x7faff4153510 2023-10-25T00:04:36.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 -- 172.21.15.107:0/3276248229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff4153510 msgr2=0x7faff415d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 --2- 172.21.15.107:0/3276248229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff4153510 0x7faff415d110 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7faff40695d0 tx=0x7fafec01c900 comp rx=0 tx=0).stop 2023-10-25T00:04:36.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 -- 172.21.15.107:0/3276248229 shutdown_connections 2023-10-25T00:04:36.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 --2- 172.21.15.107:0/3276248229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff4153510 0x7faff415d110 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 --2- 172.21.15.107:0/3276248229 >> v2:172.21.15.110:3300/0 conn(0x7faff406d8d0 0x7faff40a8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 -- 172.21.15.107:0/3276248229 >> 172.21.15.107:0/3276248229 conn(0x7faff4069d40 msgr2=0x7faff406a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 -- 172.21.15.107:0/3276248229 shutdown_connections 2023-10-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 -- 172.21.15.107:0/3276248229 wait complete. 2023-10-25T00:04:36.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 Processor -- start 2023-10-25T00:04:36.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 -- start start 2023-10-25T00:04:36.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff406aaa0 0x7faff41f0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faff406d8d0 0x7faff4147970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faff4153510 0x7faff4147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff41483f0 con 0x7faff406aaa0 2023-10-25T00:04:36.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff41519c0 con 0x7faff4153510 2023-10-25T00:04:36.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faffc0db640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff4150d70 con 0x7faff406d8d0 2023-10-25T00:04:36.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faff9e77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff406aaa0 0x7faff41f0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faff9e77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff406aaa0 0x7faff41f0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58202/0 (socket says 172.21.15.107:58202) 2023-10-25T00:04:36.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.202+0000 7faff9e77640 1 -- 172.21.15.107:0/805767537 learned_addr learned my addr 172.21.15.107:0/805767537 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:36.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faffa678640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faff4153510 0x7faff4147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faff9676640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faff406d8d0 0x7faff4147970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faff9e77640 1 -- 172.21.15.107:0/805767537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faff406d8d0 msgr2=0x7faff4147970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faff9e77640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faff406d8d0 0x7faff4147970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faff9e77640 1 -- 172.21.15.107:0/805767537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faff4153510 msgr2=0x7faff4147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faff9e77640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faff4153510 0x7faff4147eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faff9e77640 1 -- 172.21.15.107:0/805767537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec009a70 con 0x7faff406aaa0 2023-10-25T00:04:36.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faff9e77640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff406aaa0 0x7faff41f0180 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7faff000d5f0 tx=0x7faff000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:36.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7fafea7fc640 1 -- 172.21.15.107:0/805767537 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff00125f0 con 0x7faff406aaa0 2023-10-25T00:04:36.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7fafea7fc640 1 -- 172.21.15.107:0/805767537 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff0012750 con 0x7faff406aaa0 2023-10-25T00:04:36.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faffc0db640 1 -- 172.21.15.107:0/805767537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff40a9750 con 0x7faff406aaa0 2023-10-25T00:04:36.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7fafea7fc640 1 -- 172.21.15.107:0/805767537 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0012ae0 con 0x7faff406aaa0 2023-10-25T00:04:36.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faffc0db640 1 -- 172.21.15.107:0/805767537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff41f0d40 con 0x7faff406aaa0 2023-10-25T00:04:36.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faffc0db640 1 -- 172.21.15.107:0/805767537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff40746e0 con 0x7faff406aaa0 2023-10-25T00:04:36.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7fafea7fc640 1 -- 172.21.15.107:0/805767537 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7faff0012c40 con 0x7faff406aaa0 2023-10-25T00:04:36.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7fafea7fc640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fafd8065e60 0x7fafd8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7fafea7fc640 1 -- 172.21.15.107:0/805767537 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7faff0084780 con 0x7faff406aaa0 2023-10-25T00:04:36.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faff9676640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fafd8065e60 0x7fafd8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.206+0000 7faff9676640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fafd8065e60 0x7fafd8068320 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7faff41502e0 tx=0x7fafe4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:36.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.210+0000 7fafea7fc640 1 -- 172.21.15.107:0/805767537 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7faff0051390 con 0x7faff406aaa0 2023-10-25T00:04:36.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:36.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 -- 172.21.15.107:0/805767537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faff40bd250 con 0x7faff406aaa0 2023-10-25T00:04:36.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7fafea7fc640 1 -- 172.21.15.107:0/805767537 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5021 (secure 0 0 0) 0x7faff00549b0 con 0x7faff406aaa0 2023-10-25T00:04:36.403 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:36.404 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:35.748026+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:36.404 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 102 2023-10-25T00:04:36.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 -- 172.21.15.107:0/805767537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fafd8065e60 msgr2=0x7fafd8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fafd8065e60 0x7fafd8068320 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7faff41502e0 tx=0x7fafe4009040 comp rx=0 tx=0).stop 2023-10-25T00:04:36.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 -- 172.21.15.107:0/805767537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff406aaa0 msgr2=0x7faff41f0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff406aaa0 0x7faff41f0180 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7faff000d5f0 tx=0x7faff000dab0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 -- 172.21.15.107:0/805767537 shutdown_connections 2023-10-25T00:04:36.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faff4153510 0x7faff4147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fafd8065e60 0x7fafd8068320 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faff406d8d0 0x7faff4147970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 --2- 172.21.15.107:0/805767537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faff406aaa0 0x7faff41f0180 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 -- 172.21.15.107:0/805767537 >> 172.21.15.107:0/805767537 conn(0x7faff4069d40 msgr2=0x7faff40bd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:36.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 -- 172.21.15.107:0/805767537 shutdown_connections 2023-10-25T00:04:36.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.402+0000 7faffc0db640 1 -- 172.21.15.107:0/805767537 wait complete. 2023-10-25T00:04:36.424 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 102, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 102, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6906": { "addr": "172.21.15.107:6835/1246928633", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 1246928633, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 1246928633, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6906, "incarnation": 101, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:04:35.748026+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6906 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 99, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6753": { "addr": "172.21.15.107:6837/1555840329", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 1555840329, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 1555840329, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6753, "incarnation": 96, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:04:31.705767+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6753 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 59, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:04:36.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb31b22640 1 Processor -- start 2023-10-25T00:04:36.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb31b22640 1 -- start start 2023-10-25T00:04:36.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb31b22640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb2c06af70 0x7fbb2c06c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb31b22640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2c0c1860 con 0x7fbb2c14b600 2023-10-25T00:04:36.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb31b22640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb2c154d50 con 0x7fbb2c06a230 2023-10-25T00:04:36.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb31b22640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2c14b9e0 con 0x7fbb2c06af70 2023-10-25T00:04:36.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbb2c14b600 0x7fbb2c151fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36430/0 (socket says 172.21.15.107:36430) 2023-10-25T00:04:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2b7fe640 1 -- 172.21.15.107:0/1403056342 learned_addr learned my addr 172.21.15.107:0/1403056342 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2affd640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb2c06af70 0x7fbb2c06c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071853428 0 0) 0x7fbb2c0c1860 con 0x7fbb2c14b600 2023-10-25T00:04:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb14003530 con 0x7fbb2c14b600 2023-10-25T00:04:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1173449536 0 0) 0x7fbb14003530 con 0x7fbb2c14b600 2023-10-25T00:04:36.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb2c06af70 msgr2=0x7fbb2c06c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:36.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb2c06af70 0x7fbb2c06c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 >> v1:172.21.15.110:6789/0 conn(0x7fbb2c06a230 legacy=0x7fbb2c1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2c0c1ab0 con 0x7fbb2c14b600 2023-10-25T00:04:36.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fbb1c001680 con 0x7fbb2c14b600 2023-10-25T00:04:36.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 >> v1:172.21.15.107:6789/0 conn(0x7fbb2c14b600 legacy=0x7fbb2c151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb140039b0 0x7fbb14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb2c06af70 0x7fbb140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb14004d10 0x7fbb14005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2c14b9e0 con 0x7fbb14004d10 2023-10-25T00:04:36.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1400bb70 con 0x7fbb140039b0 2023-10-25T00:04:36.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1400bd30 con 0x7fbb2c06af70 2023-10-25T00:04:36.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2bfff640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb14004d10 0x7fbb14005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2b7fe640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb140039b0 0x7fbb14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2affd640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb2c06af70 0x7fbb140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2bfff640 1 -- 172.21.15.107:0/1403056342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb2c06af70 msgr2=0x7fbb140047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:36.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2bfff640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb2c06af70 0x7fbb140047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2bfff640 1 -- 172.21.15.107:0/1403056342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb140039b0 msgr2=0x7fbb14003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:36.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2bfff640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb140039b0 0x7fbb14003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.458+0000 7fbb2bfff640 1 -- 172.21.15.107:0/1403056342 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb14003530 con 0x7fbb14004d10 2023-10-25T00:04:36.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb2bfff640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb14004d10 0x7fbb14005190 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fbb24001e40 tx=0x7fbb2401c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff64a9c4d5fcba0d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:36.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24003b80 con 0x7fbb14004d10 2023-10-25T00:04:36.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb24003ce0 con 0x7fbb14004d10 2023-10-25T00:04:36.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb2a7fc640 1 -- 172.21.15.107:0/1403056342 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb240052c0 con 0x7fbb14004d10 2023-10-25T00:04:36.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb31b22640 1 -- 172.21.15.107:0/1403056342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb14004d10 msgr2=0x7fbb14005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb31b22640 1 --2- 172.21.15.107:0/1403056342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb14004d10 0x7fbb14005190 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fbb24001e40 tx=0x7fbb2401c740 comp rx=0 tx=0).stop 2023-10-25T00:04:36.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb2bfff640 1 -- 172.21.15.107:0/1403056342 reap_dead start 2023-10-25T00:04:36.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb31b22640 1 -- 172.21.15.107:0/1403056342 shutdown_connections 2023-10-25T00:04:36.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb31b22640 1 -- 172.21.15.107:0/1403056342 >> 172.21.15.107:0/1403056342 conn(0x7fbb2c0c2950 msgr2=0x7fbb2c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:36.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb31b22640 1 -- 172.21.15.107:0/1403056342 shutdown_connections 2023-10-25T00:04:36.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb31b22640 1 -- 172.21.15.107:0/1403056342 wait complete. 2023-10-25T00:04:36.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb31b22640 1 Processor -- start 2023-10-25T00:04:36.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.462+0000 7fbb31b22640 1 -- start start 2023-10-25T00:04:36.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb31b22640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb14004d10 0x7fbb2c1409c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb31b22640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb2c1401e0 0x7fbb2c0a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb31b22640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb2c0a9990 0x7fbb2c147850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb31b22640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2c1e91d0 con 0x7fbb2c1401e0 2023-10-25T00:04:36.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb31b22640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2c1e9660 con 0x7fbb2c0a9990 2023-10-25T00:04:36.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb31b22640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2c0c5b50 con 0x7fbb14004d10 2023-10-25T00:04:36.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb2c1401e0 0x7fbb2c0a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb2c1401e0 0x7fbb2c0a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58218/0 (socket says 172.21.15.107:58218) 2023-10-25T00:04:36.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2affd640 1 -- 172.21.15.107:0/3862553233 learned_addr learned my addr 172.21.15.107:0/3862553233 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:36.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2b7fe640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb14004d10 0x7fbb2c1409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2bfff640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb2c0a9990 0x7fbb2c147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2affd640 1 -- 172.21.15.107:0/3862553233 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb14004d10 msgr2=0x7fbb2c1409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2affd640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb14004d10 0x7fbb2c1409c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2affd640 1 -- 172.21.15.107:0/3862553233 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb2c0a9990 msgr2=0x7fbb2c147850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2affd640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb2c0a9990 0x7fbb2c147850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2affd640 1 -- 172.21.15.107:0/3862553233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2c0a9f60 con 0x7fbb2c1401e0 2023-10-25T00:04:36.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2bfff640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb2c0a9990 0x7fbb2c147850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:36.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2affd640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb2c1401e0 0x7fbb2c0a9450 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fbb20011dd0 tx=0x7fbb2000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:36.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb13fff640 1 -- 172.21.15.107:0/3862553233 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2000b750 con 0x7fbb2c1401e0 2023-10-25T00:04:36.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb13fff640 1 -- 172.21.15.107:0/3862553233 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2000b8b0 con 0x7fbb2c1401e0 2023-10-25T00:04:36.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb31b22640 1 -- 172.21.15.107:0/3862553233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2c1e88a0 con 0x7fbb2c1401e0 2023-10-25T00:04:36.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb13fff640 1 -- 172.21.15.107:0/3862553233 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2000bc40 con 0x7fbb2c1401e0 2023-10-25T00:04:36.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb31b22640 1 -- 172.21.15.107:0/3862553233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2c1e8e60 con 0x7fbb2c1401e0 2023-10-25T00:04:36.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb31b22640 1 -- 172.21.15.107:0/3862553233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb2c0746e0 con 0x7fbb2c1401e0 2023-10-25T00:04:36.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb13fff640 1 -- 172.21.15.107:0/3862553233 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbb2001b3f0 con 0x7fbb2c1401e0 2023-10-25T00:04:36.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb13fff640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 0x7fbb18068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:36.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb13fff640 1 -- 172.21.15.107:0/3862553233 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7fbb20086ea0 con 0x7fbb2c1401e0 2023-10-25T00:04:36.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2b7fe640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 0x7fbb18068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:36.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.466+0000 7fbb2b7fe640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 0x7fbb18068320 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c1e3040 tx=0x7fbb1c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:36.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.470+0000 7fbb13fff640 1 -- 172.21.15.107:0/3862553233 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbb20053b40 con 0x7fbb2c1401e0 2023-10-25T00:04:36.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.662+0000 7fbb31b22640 1 -- 172.21.15.107:0/3862553233 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb2c069fa0 con 0x7fbb2c1401e0 2023-10-25T00:04:36.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb13fff640 1 -- 172.21.15.107:0/3862553233 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5021 (secure 0 0 0) 0x7fbb20057160 con 0x7fbb2c1401e0 2023-10-25T00:04:36.666 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:36.667 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:35.748026+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:36.667 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 102 2023-10-25T00:04:36.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 -- 172.21.15.107:0/3862553233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 msgr2=0x7fbb18068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 0x7fbb18068320 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c1e3040 tx=0x7fbb1c009040 comp rx=0 tx=0).stop 2023-10-25T00:04:36.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 -- 172.21.15.107:0/3862553233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb2c1401e0 msgr2=0x7fbb2c0a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:36.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb2c1401e0 0x7fbb2c0a9450 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fbb20011dd0 tx=0x7fbb2000f740 comp rx=0 tx=0).stop 2023-10-25T00:04:36.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 -- 172.21.15.107:0/3862553233 shutdown_connections 2023-10-25T00:04:36.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb2c0a9990 0x7fbb2c147850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb2c1401e0 0x7fbb2c0a9450 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 0x7fbb18068320 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 --2- 172.21.15.107:0/3862553233 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb14004d10 0x7fbb2c1409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:36.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 -- 172.21.15.107:0/3862553233 >> 172.21.15.107:0/3862553233 conn(0x7fbb2c0c2950 msgr2=0x7fbb2c1489f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:36.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 -- 172.21.15.107:0/3862553233 shutdown_connections 2023-10-25T00:04:36.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:36.666+0000 7fbb31b22640 1 -- 172.21.15.107:0/3862553233 wait complete. 2023-10-25T00:04:36.686 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-25T00:04:38.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:38.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.554+0000 7fd863197640 1 Processor -- start 2023-10-25T00:04:38.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.554+0000 7fd863197640 1 -- start start 2023-10-25T00:04:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd863197640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd85c14fbd0 0x7fd85c06df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd863197640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd85c0c5ba0 0x7fd85c0c1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd863197640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd85c155f20 con 0x7fd85c0c4e20 2023-10-25T00:04:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd863197640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd85c0c6d40 con 0x7fd85c14fbd0 2023-10-25T00:04:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd863197640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd85c153560 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd860f33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd85c14fbd0 0x7fd85c06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd860f33640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd85c14fbd0 0x7fd85c06df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58228/0 (socket says 172.21.15.107:58228) 2023-10-25T00:04:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd860f33640 1 -- 172.21.15.107:0/3439770270 learned_addr learned my addr 172.21.15.107:0/3439770270 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:38.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd861734640 1 --2- 172.21.15.107:0/3439770270 >> v2:172.21.15.110:3300/0 conn(0x7fd85c0c5ba0 0x7fd85c0c1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd860f33640 1 -- 172.21.15.107:0/3439770270 >> v2:172.21.15.110:3300/0 conn(0x7fd85c0c5ba0 msgr2=0x7fd85c0c1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd860f33640 1 --2- 172.21.15.107:0/3439770270 >> v2:172.21.15.110:3300/0 conn(0x7fd85c0c5ba0 0x7fd85c0c1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd860f33640 1 -- 172.21.15.107:0/3439770270 >> v1:172.21.15.110:6789/0 conn(0x7fd85c0c4e20 legacy=0x7fd85c0c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd860f33640 1 -- 172.21.15.107:0/3439770270 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c1527a0 con 0x7fd85c14fbd0 2023-10-25T00:04:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.558+0000 7fd860f33640 1 --2- 172.21.15.107:0/3439770270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd85c14fbd0 0x7fd85c06df30 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0c4990 tx=0x7fd85001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a21d40bdd5c845 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85b7fe640 1 -- 172.21.15.107:0/3439770270 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8500ab600 con 0x7fd85c14fbd0 2023-10-25T00:04:38.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85b7fe640 1 -- 172.21.15.107:0/3439770270 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8500a7050 con 0x7fd85c14fbd0 2023-10-25T00:04:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85b7fe640 1 -- 172.21.15.107:0/3439770270 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8500ab980 con 0x7fd85c14fbd0 2023-10-25T00:04:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 -- 172.21.15.107:0/3439770270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd85c14fbd0 msgr2=0x7fd85c06df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 --2- 172.21.15.107:0/3439770270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd85c14fbd0 0x7fd85c06df30 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0c4990 tx=0x7fd85001c430 comp rx=0 tx=0).stop 2023-10-25T00:04:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 -- 172.21.15.107:0/3439770270 shutdown_connections 2023-10-25T00:04:38.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 --2- 172.21.15.107:0/3439770270 >> v2:172.21.15.110:3300/0 conn(0x7fd85c0c5ba0 0x7fd85c0c1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 --2- 172.21.15.107:0/3439770270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd85c14fbd0 0x7fd85c06df30 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 -- 172.21.15.107:0/3439770270 >> 172.21.15.107:0/3439770270 conn(0x7fd85c0c39a0 msgr2=0x7fd85c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 -- 172.21.15.107:0/3439770270 shutdown_connections 2023-10-25T00:04:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 -- 172.21.15.107:0/3439770270 wait complete. 2023-10-25T00:04:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 Processor -- start 2023-10-25T00:04:38.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 -- start start 2023-10-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd85c0c4e20 0x7fd85c153fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd85c0c5ba0 0x7fd85c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd85c14fbd0 0x7fd85c06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c153d70 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c147a00 con 0x7fd85c14fbd0 2023-10-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd863197640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c1405b0 con 0x7fd85c0c4e20 2023-10-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd85c0c5ba0 0x7fd85c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd85c0c5ba0 0x7fd85c15b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58230/0 (socket says 172.21.15.107:58230) 2023-10-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85bfff640 1 -- 172.21.15.107:0/1731229600 learned_addr learned my addr 172.21.15.107:0/1731229600 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd860f33640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd85c0c4e20 0x7fd85c153fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd861734640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd85c14fbd0 0x7fd85c06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85bfff640 1 -- 172.21.15.107:0/1731229600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd85c0c4e20 msgr2=0x7fd85c153fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85bfff640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd85c0c4e20 0x7fd85c153fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85bfff640 1 -- 172.21.15.107:0/1731229600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd85c14fbd0 msgr2=0x7fd85c06fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85bfff640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd85c14fbd0 0x7fd85c06fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85bfff640 1 -- 172.21.15.107:0/1731229600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8500094d0 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd861734640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd85c14fbd0 0x7fd85c06fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.562+0000 7fd85bfff640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd85c0c5ba0 0x7fd85c15b5c0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd84800c480 tx=0x7fd84800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.566+0000 7fd8597fa640 1 -- 172.21.15.107:0/1731229600 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd848012830 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.566+0000 7fd8597fa640 1 -- 172.21.15.107:0/1731229600 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8480129b0 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.566+0000 7fd8597fa640 1 -- 172.21.15.107:0/1731229600 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd848012d60 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.566+0000 7fd863197640 1 -- 172.21.15.107:0/1731229600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85c1544f0 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.566+0000 7fd863197640 1 -- 172.21.15.107:0/1731229600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85c147720 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.566+0000 7fd863197640 1 -- 172.21.15.107:0/1731229600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85c0746e0 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.566+0000 7fd8597fa640 1 -- 172.21.15.107:0/1731229600 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd84801c460 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.566+0000 7fd8597fa640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd840065e60 0x7fd840068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.566+0000 7fd860f33640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd840065e60 0x7fd840068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.566+0000 7fd8597fa640 1 -- 172.21.15.107:0/1731229600 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7fd8480849c0 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.566+0000 7fd860f33640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd840065e60 0x7fd840068320 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd8500ad000 tx=0x7fd8500a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:38.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.570+0000 7fd8597fa640 1 -- 172.21.15.107:0/1731229600 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd8480516e0 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:38.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd863197640 1 -- 172.21.15.107:0/1731229600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd85c1544f0 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd8597fa640 1 -- 172.21.15.107:0/1731229600 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5018 (secure 0 0 0) 0x7fd85c1544f0 con 0x7fd85c0c5ba0 2023-10-25T00:04:38.800 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:38.800 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:38.801 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 104 2023-10-25T00:04:38.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd863197640 1 -- 172.21.15.107:0/1731229600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd840065e60 msgr2=0x7fd840068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:38.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd863197640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd840065e60 0x7fd840068320 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd8500ad000 tx=0x7fd8500a6000 comp rx=0 tx=0).stop 2023-10-25T00:04:38.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd863197640 1 -- 172.21.15.107:0/1731229600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd85c0c5ba0 msgr2=0x7fd85c15b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:38.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd863197640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd85c0c5ba0 0x7fd85c15b5c0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd84800c480 tx=0x7fd84800c940 comp rx=0 tx=0).stop 2023-10-25T00:04:38.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd863197640 1 -- 172.21.15.107:0/1731229600 shutdown_connections 2023-10-25T00:04:38.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd863197640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd85c14fbd0 0x7fd85c06fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd863197640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd85c0c5ba0 0x7fd85c15b5c0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd863197640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd840065e60 0x7fd840068320 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd863197640 1 --2- 172.21.15.107:0/1731229600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd85c0c4e20 0x7fd85c153fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.798+0000 7fd863197640 1 -- 172.21.15.107:0/1731229600 >> 172.21.15.107:0/1731229600 conn(0x7fd85c0c39a0 msgr2=0x7fd85c069f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:38.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.802+0000 7fd863197640 1 -- 172.21.15.107:0/1731229600 shutdown_connections 2023-10-25T00:04:38.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.802+0000 7fd863197640 1 -- 172.21.15.107:0/1731229600 wait complete. 2023-10-25T00:04:38.821 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=5764 2023-10-25T00:04:38.821 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.b 2023-10-25T00:04:38.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f370687d640 1 Processor -- start 2023-10-25T00:04:38.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f370687d640 1 -- start start 2023-10-25T00:04:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f370687d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3700154bf0 0x7f3700154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f370687d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37000c0bf0 0x7f370006bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f370687d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f370014ea40 0x7f370014ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f370687d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3700152af0 con 0x7f37000c0bf0 2023-10-25T00:04:38.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f370687d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f370006c2a0 con 0x7f3700154bf0 2023-10-25T00:04:38.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f370687d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3700154960 con 0x7f370014ea40 2023-10-25T00:04:38.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f36ff7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37000c0bf0 0x7f370006bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f36ff7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f37000c0bf0 0x7f370006bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58238/0 (socket says 172.21.15.107:58238) 2023-10-25T00:04:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f36ff7fe640 1 -- 172.21.15.107:0/3963059972 learned_addr learned my addr 172.21.15.107:0/3963059972 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f36fffff640 1 --2- 172.21.15.107:0/3963059972 >> v2:172.21.15.110:3300/0 conn(0x7f3700154bf0 0x7f3700154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f3704e1a640 1 --2- 172.21.15.107:0/3963059972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f370014ea40 0x7f370014ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f36ff7fe640 1 -- 172.21.15.107:0/3963059972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f370014ea40 msgr2=0x7f370014ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f36ff7fe640 1 --2- 172.21.15.107:0/3963059972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f370014ea40 0x7f370014ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f36ff7fe640 1 -- 172.21.15.107:0/3963059972 >> v2:172.21.15.110:3300/0 conn(0x7f3700154bf0 msgr2=0x7f3700154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:38.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f36ff7fe640 1 --2- 172.21.15.107:0/3963059972 >> v2:172.21.15.110:3300/0 conn(0x7f3700154bf0 0x7f3700154fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.822+0000 7f36ff7fe640 1 -- 172.21.15.107:0/3963059972 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370006c540 con 0x7f37000c0bf0 2023-10-25T00:04:38.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f36ff7fe640 1 --2- 172.21.15.107:0/3963059972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37000c0bf0 0x7f370006bd60 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f37000c2ac0 tx=0x7f36f401e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b1146b64a3f2ef server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:38.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f36feffd640 1 -- 172.21.15.107:0/3963059972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f40ab620 con 0x7f37000c0bf0 2023-10-25T00:04:38.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f36feffd640 1 -- 172.21.15.107:0/3963059972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f40ab780 con 0x7f37000c0bf0 2023-10-25T00:04:38.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f36feffd640 1 -- 172.21.15.107:0/3963059972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f40abb10 con 0x7f37000c0bf0 2023-10-25T00:04:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f370687d640 1 -- 172.21.15.107:0/3963059972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37000c0bf0 msgr2=0x7f370006bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f370687d640 1 --2- 172.21.15.107:0/3963059972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37000c0bf0 0x7f370006bd60 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f37000c2ac0 tx=0x7f36f401e680 comp rx=0 tx=0).stop 2023-10-25T00:04:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f370687d640 1 -- 172.21.15.107:0/3963059972 shutdown_connections 2023-10-25T00:04:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f370687d640 1 --2- 172.21.15.107:0/3963059972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f370014ea40 0x7f370014ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f370687d640 1 --2- 172.21.15.107:0/3963059972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37000c0bf0 0x7f370006bd60 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f370687d640 1 --2- 172.21.15.107:0/3963059972 >> v2:172.21.15.110:3300/0 conn(0x7f3700154bf0 0x7f3700154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f370687d640 1 -- 172.21.15.107:0/3963059972 >> 172.21.15.107:0/3963059972 conn(0x7f37000c6270 msgr2=0x7f37000c6680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:38.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f370687d640 1 -- 172.21.15.107:0/3963059972 shutdown_connections 2023-10-25T00:04:38.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.826+0000 7f370687d640 1 -- 172.21.15.107:0/3963059972 wait complete. 2023-10-25T00:04:38.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f370687d640 1 Processor -- start 2023-10-25T00:04:38.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f370687d640 1 -- start start 2023-10-25T00:04:38.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f370687d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37000c0bf0 0x7f37000bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f370687d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f370014ea40 0x7f37001ec310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f370687d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3700154bf0 0x7f370014da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f370687d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370014e610 con 0x7f37000c0bf0 2023-10-25T00:04:38.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f370687d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37000ba060 con 0x7f370014ea40 2023-10-25T00:04:38.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f370687d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37000bb800 con 0x7f3700154bf0 2023-10-25T00:04:38.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37000c0bf0 0x7f37000bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37000c0bf0 0x7f37000bbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58248/0 (socket says 172.21.15.107:58248) 2023-10-25T00:04:38.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fffff640 1 -- 172.21.15.107:0/1523942049 learned_addr learned my addr 172.21.15.107:0/1523942049 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:38.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36ff7fe640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f370014ea40 0x7f37001ec310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f3704e1a640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3700154bf0 0x7f370014da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fffff640 1 -- 172.21.15.107:0/1523942049 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3700154bf0 msgr2=0x7f370014da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:38.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fffff640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3700154bf0 0x7f370014da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fffff640 1 -- 172.21.15.107:0/1523942049 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f370014ea40 msgr2=0x7f37001ec310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:38.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fffff640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f370014ea40 0x7f37001ec310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:38.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fffff640 1 -- 172.21.15.107:0/1523942049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f40094d0 con 0x7f37000c0bf0 2023-10-25T00:04:38.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fffff640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37000c0bf0 0x7f37000bbc90 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f36f000d620 tx=0x7f36f000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:38.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fcff9640 1 -- 172.21.15.107:0/1523942049 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f000b620 con 0x7f37000c0bf0 2023-10-25T00:04:38.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fcff9640 1 -- 172.21.15.107:0/1523942049 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f000b780 con 0x7f37000c0bf0 2023-10-25T00:04:38.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f370687d640 1 -- 172.21.15.107:0/1523942049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37000bc1d0 con 0x7f37000c0bf0 2023-10-25T00:04:38.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f36fcff9640 1 -- 172.21.15.107:0/1523942049 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f000bb10 con 0x7f37000c0bf0 2023-10-25T00:04:38.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f370687d640 1 -- 172.21.15.107:0/1523942049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37000b9d80 con 0x7f37000c0bf0 2023-10-25T00:04:38.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.830+0000 7f370687d640 1 -- 172.21.15.107:0/1523942049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37000746e0 con 0x7f37000c0bf0 2023-10-25T00:04:38.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.834+0000 7f36fcff9640 1 -- 172.21.15.107:0/1523942049 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f36f000bc70 con 0x7f37000c0bf0 2023-10-25T00:04:38.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.834+0000 7f36fcff9640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36ec065e90 0x7f36ec068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:38.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.834+0000 7f36fcff9640 1 -- 172.21.15.107:0/1523942049 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f36f00850c0 con 0x7f37000c0bf0 2023-10-25T00:04:38.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.834+0000 7f36ff7fe640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36ec065e90 0x7f36ec068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.834+0000 7f36ff7fe640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36ec065e90 0x7f36ec068350 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f36f40b0000 tx=0x7f36f40b5040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:38.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:38.838+0000 7f36fcff9640 1 -- 172.21.15.107:0/1523942049 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f36f0051cf0 con 0x7f37000c0bf0 2023-10-25T00:04:39.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 -- 172.21.15.107:0/1523942049 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37000bf910 con 0x7f37000c0bf0 2023-10-25T00:04:39.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f36fcff9640 1 -- 172.21.15.107:0/1523942049 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5018 (secure 0 0 0) 0x7f36f0007380 con 0x7f37000c0bf0 2023-10-25T00:04:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:39.108 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:39.110 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 104 2023-10-25T00:04:39.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 -- 172.21.15.107:0/1523942049 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36ec065e90 msgr2=0x7f36ec068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:39.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36ec065e90 0x7f36ec068350 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f36f40b0000 tx=0x7f36f40b5040 comp rx=0 tx=0).stop 2023-10-25T00:04:39.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 -- 172.21.15.107:0/1523942049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37000c0bf0 msgr2=0x7f37000bbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:39.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37000c0bf0 0x7f37000bbc90 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f36f000d620 tx=0x7f36f000dae0 comp rx=0 tx=0).stop 2023-10-25T00:04:39.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 -- 172.21.15.107:0/1523942049 shutdown_connections 2023-10-25T00:04:39.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36ec065e90 0x7f36ec068350 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:39.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3700154bf0 0x7f370014da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:39.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f370014ea40 0x7f37001ec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:39.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 --2- 172.21.15.107:0/1523942049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37000c0bf0 0x7f37000bbc90 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:39.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 -- 172.21.15.107:0/1523942049 >> 172.21.15.107:0/1523942049 conn(0x7f37000c6270 msgr2=0x7f3700071080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:39.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 -- 172.21.15.107:0/1523942049 shutdown_connections 2023-10-25T00:04:39.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:39.106+0000 7f370687d640 1 -- 172.21.15.107:0/1523942049 wait complete. 2023-10-25T00:04:39.125 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:active state 2023-10-25T00:04:39.125 INFO:tasks.mds_thrash.fs.[b]:waiting for 27 secs before thrashing 2023-10-25T00:04:39.734 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:04:39.734+0000 7f9ab30a8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:04:42.822 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2023-10-25T00:04:42.822 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-25T00:04:42.823 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-25T00:04:42.825 INFO:tasks.ceph.mds.b:Started 2023-10-25T00:04:42.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:42.908 INFO:tasks.ceph.mds.b.smithi110.stdout:starting mds.b at 2023-10-25T00:04:42.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.978+0000 7faa9377c640 1 Processor -- start 2023-10-25T00:04:42.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.978+0000 7faa9377c640 1 -- start start 2023-10-25T00:04:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa9377c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c0adb80 0x7faa8c158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa9377c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faa8c06e800 0x7faa8c06ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:42.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa9377c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8c152420 con 0x7faa8c152040 2023-10-25T00:04:42.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa9377c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8c1556e0 con 0x7faa8c06e800 2023-10-25T00:04:42.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa9377c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c154d40 con 0x7faa8c0adb80 2023-10-25T00:04:42.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa91d19640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faa8c06e800 0x7faa8c06ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:42.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa91d19640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faa8c06e800 0x7faa8c06ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60018/0 (socket says 172.21.15.107:60018) 2023-10-25T00:04:42.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa91518640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7faa8c152040 0x7faa8c071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:58806/0 (socket says 172.21.15.107:58806) 2023-10-25T00:04:42.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa91d19640 1 -- 172.21.15.107:0/1735022283 learned_addr learned my addr 172.21.15.107:0/1735022283 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:42.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa90d17640 1 --2- 172.21.15.107:0/1735022283 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c0adb80 0x7faa8c158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:42.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa91d19640 1 -- 172.21.15.107:0/1735022283 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c0adb80 msgr2=0x7faa8c158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:42.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa91d19640 1 --2- 172.21.15.107:0/1735022283 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c0adb80 0x7faa8c158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:42.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa91d19640 1 -- 172.21.15.107:0/1735022283 >> v1:172.21.15.110:6789/0 conn(0x7faa8c152040 legacy=0x7faa8c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:42.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa83fff640 1 -- 172.21.15.107:0/1735022283 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371084203 0 0) 0x7faa8c152420 con 0x7faa8c152040 2023-10-25T00:04:42.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa91d19640 1 -- 172.21.15.107:0/1735022283 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c155020 con 0x7faa8c06e800 2023-10-25T00:04:42.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa91d19640 1 --2- 172.21.15.107:0/1735022283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c06e800 0x7faa8c06ec90 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7faa84009e50 tx=0x7faa8401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4100d6ec39fd83 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:42.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa83fff640 1 -- 172.21.15.107:0/1735022283 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa840a5660 con 0x7faa8c06e800 2023-10-25T00:04:42.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa83fff640 1 -- 172.21.15.107:0/1735022283 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa840a57c0 con 0x7faa8c06e800 2023-10-25T00:04:42.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.982+0000 7faa83fff640 1 -- 172.21.15.107:0/1735022283 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa840a5b50 con 0x7faa8c06e800 2023-10-25T00:04:42.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 -- 172.21.15.107:0/1735022283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c06e800 msgr2=0x7faa8c06ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:42.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 --2- 172.21.15.107:0/1735022283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c06e800 0x7faa8c06ec90 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7faa84009e50 tx=0x7faa8401c9a0 comp rx=0 tx=0).stop 2023-10-25T00:04:42.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 -- 172.21.15.107:0/1735022283 shutdown_connections 2023-10-25T00:04:42.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 --2- 172.21.15.107:0/1735022283 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c06e800 0x7faa8c06ec90 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:42.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 --2- 172.21.15.107:0/1735022283 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c0adb80 0x7faa8c158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:42.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 -- 172.21.15.107:0/1735022283 >> 172.21.15.107:0/1735022283 conn(0x7faa8c070f60 msgr2=0x7faa8c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:42.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 -- 172.21.15.107:0/1735022283 shutdown_connections 2023-10-25T00:04:42.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 -- 172.21.15.107:0/1735022283 wait complete. 2023-10-25T00:04:42.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 Processor -- start 2023-10-25T00:04:42.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 -- start start 2023-10-25T00:04:42.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa8c06e800 0x7faa8c1f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:42.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c0adb80 0x7faa8c1ef980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:42.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c152040 0x7faa8c1538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:42.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c0c60d0 con 0x7faa8c0adb80 2023-10-25T00:04:42.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1ee1d0 con 0x7faa8c06e800 2023-10-25T00:04:42.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1426f0 con 0x7faa8c152040 2023-10-25T00:04:42.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa90d17640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c0adb80 0x7faa8c1ef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:42.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa90d17640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c0adb80 0x7faa8c1ef980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60034/0 (socket says 172.21.15.107:60034) 2023-10-25T00:04:42.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa90d17640 1 -- 172.21.15.107:0/1300782182 learned_addr learned my addr 172.21.15.107:0/1300782182 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:42.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa91d19640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c152040 0x7faa8c1538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:42.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa91518640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa8c06e800 0x7faa8c1f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:42.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa90d17640 1 -- 172.21.15.107:0/1300782182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c152040 msgr2=0x7faa8c1538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:42.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa90d17640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c152040 0x7faa8c1538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:42.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa90d17640 1 -- 172.21.15.107:0/1300782182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa8c06e800 msgr2=0x7faa8c1f0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:42.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa90d17640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa8c06e800 0x7faa8c1f0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:42.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa90d17640 1 -- 172.21.15.107:0/1300782182 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa84009b30 con 0x7faa8c0adb80 2023-10-25T00:04:42.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa90d17640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c0adb80 0x7faa8c1ef980 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7faa8800e960 tx=0x7faa8800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:42.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa81ffb640 1 -- 172.21.15.107:0/1300782182 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa88009690 con 0x7faa8c0adb80 2023-10-25T00:04:42.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa81ffb640 1 -- 172.21.15.107:0/1300782182 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8800ca00 con 0x7faa8c0adb80 2023-10-25T00:04:42.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa81ffb640 1 -- 172.21.15.107:0/1300782182 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8800cd70 con 0x7faa8c0adb80 2023-10-25T00:04:42.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 -- 172.21.15.107:0/1300782182 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c1f1220 con 0x7faa8c0adb80 2023-10-25T00:04:42.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.986+0000 7faa9377c640 1 -- 172.21.15.107:0/1300782182 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c0c7790 con 0x7faa8c0adb80 2023-10-25T00:04:42.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.990+0000 7faa9377c640 1 -- 172.21.15.107:0/1300782182 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8c0c6210 con 0x7faa8c0adb80 2023-10-25T00:04:42.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.990+0000 7faa81ffb640 1 -- 172.21.15.107:0/1300782182 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7faa88013470 con 0x7faa8c0adb80 2023-10-25T00:04:42.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.990+0000 7faa81ffb640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa70065e90 0x7faa70068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:42.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.990+0000 7faa81ffb640 1 -- 172.21.15.107:0/1300782182 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7faa88084340 con 0x7faa8c0adb80 2023-10-25T00:04:42.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.990+0000 7faa91518640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa70065e90 0x7faa70068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:42.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.990+0000 7faa91518640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa70065e90 0x7faa70068350 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7faa7c000f70 tx=0x7faa7c007df0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:43.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:42.994+0000 7faa81ffb640 1 -- 172.21.15.107:0/1300782182 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7faa8c0c6210 con 0x7faa8c0adb80 2023-10-25T00:04:43.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 -- 172.21.15.107:0/1300782182 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa8c0c6210 con 0x7faa8c0adb80 2023-10-25T00:04:43.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa81ffb640 1 -- 172.21.15.107:0/1300782182 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5018 (secure 0 0 0) 0x7faa8c0c6210 con 0x7faa8c0adb80 2023-10-25T00:04:43.268 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:43.268 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:43.268 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 104 2023-10-25T00:04:43.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 -- 172.21.15.107:0/1300782182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa70065e90 msgr2=0x7faa70068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:43.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa70065e90 0x7faa70068350 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7faa7c000f70 tx=0x7faa7c007df0 comp rx=0 tx=0).stop 2023-10-25T00:04:43.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 -- 172.21.15.107:0/1300782182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c0adb80 msgr2=0x7faa8c1ef980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:43.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c0adb80 0x7faa8c1ef980 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7faa8800e960 tx=0x7faa8800ee20 comp rx=0 tx=0).stop 2023-10-25T00:04:43.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 -- 172.21.15.107:0/1300782182 shutdown_connections 2023-10-25T00:04:43.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa70065e90 0x7faa70068350 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:43.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faa8c152040 0x7faa8c1538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:43.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa8c0adb80 0x7faa8c1ef980 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:43.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 --2- 172.21.15.107:0/1300782182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faa8c06e800 0x7faa8c1f0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:43.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 -- 172.21.15.107:0/1300782182 >> 172.21.15.107:0/1300782182 conn(0x7faa8c070f60 msgr2=0x7faa8c158240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:43.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 -- 172.21.15.107:0/1300782182 shutdown_connections 2023-10-25T00:04:43.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:43.266+0000 7faa9377c640 1 -- 172.21.15.107:0/1300782182 wait complete. 2023-10-25T00:04:43.289 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-25T00:04:45.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:45.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751faae640 1 Processor -- start 2023-10-25T00:04:45.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751faae640 1 -- start start 2023-10-25T00:04:45.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751faae640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f751806f490 0x7f751814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:45.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751faae640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7518152b30 0x7f7518152f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:45.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751faae640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7518155cd0 con 0x7f751814c470 2023-10-25T00:04:45.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751faae640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75181526e0 con 0x7f7518152b30 2023-10-25T00:04:45.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751faae640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75180c4380 con 0x7f751806f490 2023-10-25T00:04:45.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751e04b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7518152b30 0x7f7518152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:45.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751e04b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7518152b30 0x7f7518152f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60048/0 (socket says 172.21.15.107:60048) 2023-10-25T00:04:45.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751e04b640 1 -- 172.21.15.107:0/1735376623 learned_addr learned my addr 172.21.15.107:0/1735376623 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:45.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751d049640 1 --2- 172.21.15.107:0/1735376623 >> v2:172.21.15.110:3300/0 conn(0x7f751806f490 0x7f751814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:45.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751e04b640 1 -- 172.21.15.107:0/1735376623 >> v2:172.21.15.110:3300/0 conn(0x7f751806f490 msgr2=0x7f751814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:45.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751e04b640 1 --2- 172.21.15.107:0/1735376623 >> v2:172.21.15.110:3300/0 conn(0x7f751806f490 0x7f751814c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:45.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751e04b640 1 -- 172.21.15.107:0/1735376623 >> v1:172.21.15.110:6789/0 conn(0x7f751814c470 legacy=0x7f7518071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:45.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.430+0000 7f751e04b640 1 -- 172.21.15.107:0/1735376623 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180c48b0 con 0x7f7518152b30 2023-10-25T00:04:45.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751e04b640 1 --2- 172.21.15.107:0/1735376623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7518152b30 0x7f7518152f40 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f751806a7e0 tx=0x7f751001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8545ef1d50850f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:45.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751c848640 1 -- 172.21.15.107:0/1735376623 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75100a5660 con 0x7f7518152b30 2023-10-25T00:04:45.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751c848640 1 -- 172.21.15.107:0/1735376623 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75100a57c0 con 0x7f7518152b30 2023-10-25T00:04:45.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751c848640 1 -- 172.21.15.107:0/1735376623 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75100a5b50 con 0x7f7518152b30 2023-10-25T00:04:45.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751faae640 1 -- 172.21.15.107:0/1735376623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7518152b30 msgr2=0x7f7518152f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:45.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751faae640 1 --2- 172.21.15.107:0/1735376623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7518152b30 0x7f7518152f40 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f751806a7e0 tx=0x7f751001c8b0 comp rx=0 tx=0).stop 2023-10-25T00:04:45.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751faae640 1 -- 172.21.15.107:0/1735376623 shutdown_connections 2023-10-25T00:04:45.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751faae640 1 --2- 172.21.15.107:0/1735376623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7518152b30 0x7f7518152f40 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:45.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751faae640 1 --2- 172.21.15.107:0/1735376623 >> v2:172.21.15.110:3300/0 conn(0x7f751806f490 0x7f751814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:45.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751faae640 1 -- 172.21.15.107:0/1735376623 >> 172.21.15.107:0/1735376623 conn(0x7f7518070f60 msgr2=0x7f7518071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:45.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751faae640 1 -- 172.21.15.107:0/1735376623 shutdown_connections 2023-10-25T00:04:45.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751faae640 1 -- 172.21.15.107:0/1735376623 wait complete. 2023-10-25T00:04:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751faae640 1 Processor -- start 2023-10-25T00:04:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.434+0000 7f751faae640 1 -- start start 2023-10-25T00:04:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751faae640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f751806f490 0x7f7518154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751faae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f751814c470 0x7f75181ebca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:45.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751faae640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7518152b30 0x7f75181ec1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:45.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751faae640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75181eb810 con 0x7f751814c470 2023-10-25T00:04:45.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751faae640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75181ee350 con 0x7f751806f490 2023-10-25T00:04:45.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751faae640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751814c9d0 con 0x7f7518152b30 2023-10-25T00:04:45.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d049640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f751814c470 0x7f75181ebca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:45.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d049640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f751814c470 0x7f75181ebca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60060/0 (socket says 172.21.15.107:60060) 2023-10-25T00:04:45.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d049640 1 -- 172.21.15.107:0/1708517566 learned_addr learned my addr 172.21.15.107:0/1708517566 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:45.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d84a640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f751806f490 0x7f7518154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:45.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751e04b640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7518152b30 0x7f75181ec1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:45.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d049640 1 -- 172.21.15.107:0/1708517566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7518152b30 msgr2=0x7f75181ec1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:45.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d049640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7518152b30 0x7f75181ec1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:45.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d049640 1 -- 172.21.15.107:0/1708517566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f751806f490 msgr2=0x7f7518154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:45.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d049640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f751806f490 0x7f7518154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:45.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d049640 1 -- 172.21.15.107:0/1708517566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7510009b30 con 0x7f751814c470 2023-10-25T00:04:45.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d049640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f751814c470 0x7f75181ebca0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f750800d800 tx=0x7f750800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:45.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f750e7fc640 1 -- 172.21.15.107:0/1708517566 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750800b6c0 con 0x7f751814c470 2023-10-25T00:04:45.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f750e7fc640 1 -- 172.21.15.107:0/1708517566 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750800b820 con 0x7f751814c470 2023-10-25T00:04:45.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751faae640 1 -- 172.21.15.107:0/1708517566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75181ee5f0 con 0x7f751814c470 2023-10-25T00:04:45.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f750e7fc640 1 -- 172.21.15.107:0/1708517566 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750800bb90 con 0x7f751814c470 2023-10-25T00:04:45.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751faae640 1 -- 172.21.15.107:0/1708517566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7518151bd0 con 0x7f751814c470 2023-10-25T00:04:45.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751faae640 1 -- 172.21.15.107:0/1708517566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75181eb9c0 con 0x7f751814c470 2023-10-25T00:04:45.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f750e7fc640 1 -- 172.21.15.107:0/1708517566 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f750800bcf0 con 0x7f751814c470 2023-10-25T00:04:45.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f750e7fc640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f74fc065e60 0x7f74fc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:45.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f750e7fc640 1 -- 172.21.15.107:0/1708517566 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f7508084740 con 0x7f751814c470 2023-10-25T00:04:45.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d84a640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f74fc065e60 0x7f74fc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:45.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.438+0000 7f751d84a640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f74fc065e60 0x7f74fc068320 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7518153670 tx=0x7f7514009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:45.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.442+0000 7f750e7fc640 1 -- 172.21.15.107:0/1708517566 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f75181eb9c0 con 0x7f751814c470 2023-10-25T00:04:45.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 -- 172.21.15.107:0/1708517566 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75180746e0 con 0x7f751814c470 2023-10-25T00:04:45.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f750e7fc640 1 -- 172.21.15.107:0/1708517566 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f75080513e0 con 0x7f751814c470 2023-10-25T00:04:45.688 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:45.688 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:45.688 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:04:45.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 -- 172.21.15.107:0/1708517566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f74fc065e60 msgr2=0x7f74fc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:45.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f74fc065e60 0x7f74fc068320 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7518153670 tx=0x7f7514009040 comp rx=0 tx=0).stop 2023-10-25T00:04:45.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 -- 172.21.15.107:0/1708517566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f751814c470 msgr2=0x7f75181ebca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:45.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f751814c470 0x7f75181ebca0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f750800d800 tx=0x7f750800dcc0 comp rx=0 tx=0).stop 2023-10-25T00:04:45.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 -- 172.21.15.107:0/1708517566 shutdown_connections 2023-10-25T00:04:45.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f74fc065e60 0x7f74fc068320 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:45.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7518152b30 0x7f75181ec1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:45.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f751814c470 0x7f75181ebca0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:45.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 --2- 172.21.15.107:0/1708517566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f751806f490 0x7f7518154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:45.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 -- 172.21.15.107:0/1708517566 >> 172.21.15.107:0/1708517566 conn(0x7f7518070f60 msgr2=0x7f7518152f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:45.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 -- 172.21.15.107:0/1708517566 shutdown_connections 2023-10-25T00:04:45.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:45.686+0000 7f751faae640 1 -- 172.21.15.107:0/1708517566 wait complete. 2023-10-25T00:04:45.707 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2023-10-25T00:04:45.708 INFO:tasks.mds_thrash.fs.[a]:waiting for 13 secs before thrashing 2023-10-25T00:04:58.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:58.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb8c6f640 1 Processor -- start 2023-10-25T00:04:58.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb8c6f640 1 -- start start 2023-10-25T00:04:58.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb8c6f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40a94b0 0x7fefb41517d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:58.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb8c6f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb40a9b90 con 0x7fefb4150930 2023-10-25T00:04:58.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb8c6f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb40c1390 con 0x7fefb40c1570 2023-10-25T00:04:58.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb8c6f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb40a9940 con 0x7fefb40a94b0 2023-10-25T00:04:58.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fefb4150930 0x7fefb4069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43084/0 (socket says 172.21.15.107:43084) 2023-10-25T00:04:58.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb259c640 1 -- 172.21.15.107:0/2196281552 learned_addr learned my addr 172.21.15.107:0/2196281552 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:58.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb2d9d640 1 --2- 172.21.15.107:0/2196281552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40a94b0 0x7fefb41517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:58.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb159a640 1 -- 172.21.15.107:0/2196281552 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617671237 0 0) 0x7fefb40a9b90 con 0x7fefb4150930 2023-10-25T00:04:58.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb159a640 1 -- 172.21.15.107:0/2196281552 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c003530 con 0x7fefb4150930 2023-10-25T00:04:58.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb2d9d640 1 -- 172.21.15.107:0/2196281552 >> v1:172.21.15.110:6789/0 conn(0x7fefb40c1570 legacy=0x7fefb41510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:58.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb2d9d640 1 -- 172.21.15.107:0/2196281552 >> v1:172.21.15.107:6789/0 conn(0x7fefb4150930 legacy=0x7fefb4069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:58.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb159a640 1 -- 172.21.15.107:0/2196281552 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1273868992 0 0) 0x7fef9c003530 con 0x7fefb4150930 2023-10-25T00:04:58.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.837+0000 7fefb2d9d640 1 -- 172.21.15.107:0/2196281552 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb4151d10 con 0x7fefb40a94b0 2023-10-25T00:04:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb2d9d640 1 --2- 172.21.15.107:0/2196281552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40a94b0 0x7fefb41517d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fefac009dc0 tx=0x7fefac01c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=297b888eb497128a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb159a640 1 -- 172.21.15.107:0/2196281552 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4001680 con 0x7fefb40a94b0 2023-10-25T00:04:58.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb159a640 1 -- 172.21.15.107:0/2196281552 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefac0a6430 con 0x7fefb40a94b0 2023-10-25T00:04:58.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb159a640 1 -- 172.21.15.107:0/2196281552 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefac0a67a0 con 0x7fefb40a94b0 2023-10-25T00:04:58.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb8c6f640 1 -- 172.21.15.107:0/2196281552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40a94b0 msgr2=0x7fefb41517d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:58.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb8c6f640 1 --2- 172.21.15.107:0/2196281552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40a94b0 0x7fefb41517d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fefac009dc0 tx=0x7fefac01c800 comp rx=0 tx=0).stop 2023-10-25T00:04:58.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb8c6f640 1 -- 172.21.15.107:0/2196281552 shutdown_connections 2023-10-25T00:04:58.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb8c6f640 1 --2- 172.21.15.107:0/2196281552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40a94b0 0x7fefb41517d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:58.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb8c6f640 1 -- 172.21.15.107:0/2196281552 >> 172.21.15.107:0/2196281552 conn(0x7fefb40c65c0 msgr2=0x7fefb40c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:58.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb8c6f640 1 -- 172.21.15.107:0/2196281552 shutdown_connections 2023-10-25T00:04:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb8c6f640 1 -- 172.21.15.107:0/2196281552 wait complete. 2023-10-25T00:04:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb8c6f640 1 Processor -- start 2023-10-25T00:04:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.841+0000 7fefb8c6f640 1 -- start start 2023-10-25T00:04:58.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb8c6f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefb40a94b0 0x7fefb4147d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb8c6f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40c1570 0x7fefb4149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb8c6f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefb4150930 0x7fefb414d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:58.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb8c6f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb414f150 con 0x7fefb40a94b0 2023-10-25T00:04:58.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb8c6f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb4148250 con 0x7fefb4150930 2023-10-25T00:04:58.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb8c6f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb414e720 con 0x7fefb40c1570 2023-10-25T00:04:58.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefb40a94b0 0x7fefb4147d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:58.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefb40a94b0 0x7fefb4147d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49672/0 (socket says 172.21.15.107:49672) 2023-10-25T00:04:58.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb259c640 1 -- 172.21.15.107:0/382062832 learned_addr learned my addr 172.21.15.107:0/382062832 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:58.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb1d9b640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40c1570 0x7fefb4149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:58.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb2d9d640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefb4150930 0x7fefb414d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:58.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb259c640 1 -- 172.21.15.107:0/382062832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40c1570 msgr2=0x7fefb4149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:58.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb259c640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40c1570 0x7fefb4149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:58.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb259c640 1 -- 172.21.15.107:0/382062832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefb4150930 msgr2=0x7fefb414d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:58.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb259c640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefb4150930 0x7fefb414d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:58.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb259c640 1 -- 172.21.15.107:0/382062832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac009a70 con 0x7fefb40a94b0 2023-10-25T00:04:58.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb1d9b640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40c1570 0x7fefb4149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:04:58.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb259c640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefb40a94b0 0x7fefb4147d10 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fefa400c780 tx=0x7fefa400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:58.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefa2ffd640 1 -- 172.21.15.107:0/382062832 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4005ae0 con 0x7fefb40a94b0 2023-10-25T00:04:58.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefa2ffd640 1 -- 172.21.15.107:0/382062832 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa4005c40 con 0x7fefb40a94b0 2023-10-25T00:04:58.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefa2ffd640 1 -- 172.21.15.107:0/382062832 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4007070 con 0x7fefb40a94b0 2023-10-25T00:04:58.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb8c6f640 1 -- 172.21.15.107:0/382062832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb414ea00 con 0x7fefb40a94b0 2023-10-25T00:04:58.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb8c6f640 1 -- 172.21.15.107:0/382062832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb41477f0 con 0x7fefb40a94b0 2023-10-25T00:04:58.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefb8c6f640 1 -- 172.21.15.107:0/382062832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb4147a10 con 0x7fefb40a94b0 2023-10-25T00:04:58.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefa2ffd640 1 -- 172.21.15.107:0/382062832 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fefa40071d0 con 0x7fefb40a94b0 2023-10-25T00:04:58.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefa2ffd640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef94065e60 0x7fef94068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:58.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.845+0000 7fefa2ffd640 1 -- 172.21.15.107:0/382062832 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7fefa4083f40 con 0x7fefb40a94b0 2023-10-25T00:04:58.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.849+0000 7fefb1d9b640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef94065e60 0x7fef94068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:58.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.849+0000 7fefb1d9b640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef94065e60 0x7fef94068320 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fefb406a2a0 tx=0x7fefa8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:58.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:58.853+0000 7fefa2ffd640 1 -- 172.21.15.107:0/382062832 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fefa4050ca0 con 0x7fefb40a94b0 2023-10-25T00:04:59.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.073+0000 7fefb8c6f640 1 -- 172.21.15.107:0/382062832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefb40746e0 con 0x7fefb40a94b0 2023-10-25T00:04:59.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.073+0000 7fefa2ffd640 1 -- 172.21.15.107:0/382062832 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7fefa40542c0 con 0x7fefb40a94b0 2023-10-25T00:04:59.079 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:59.079 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:59.080 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:04:59.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 -- 172.21.15.107:0/382062832 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef94065e60 msgr2=0x7fef94068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:59.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef94065e60 0x7fef94068320 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fefb406a2a0 tx=0x7fefa8009040 comp rx=0 tx=0).stop 2023-10-25T00:04:59.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 -- 172.21.15.107:0/382062832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefb40a94b0 msgr2=0x7fefb4147d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:59.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefb40a94b0 0x7fefb4147d10 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fefa400c780 tx=0x7fefa400cc40 comp rx=0 tx=0).stop 2023-10-25T00:04:59.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 -- 172.21.15.107:0/382062832 shutdown_connections 2023-10-25T00:04:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fefb4150930 0x7fefb414d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef94065e60 0x7fef94068320 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fefb40c1570 0x7fefb4149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 --2- 172.21.15.107:0/382062832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fefb40a94b0 0x7fefb4147d10 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 -- 172.21.15.107:0/382062832 >> 172.21.15.107:0/382062832 conn(0x7fefb40c65c0 msgr2=0x7fefb41510c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:59.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 -- 172.21.15.107:0/382062832 shutdown_connections 2023-10-25T00:04:59.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.077+0000 7fefb8c6f640 1 -- 172.21.15.107:0/382062832 wait complete. 2023-10-25T00:04:59.097 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2023-10-25T00:04:59.098 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:04:59.098 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:04:59.108 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:04:59.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:04:59.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c7e27b640 1 Processor -- start 2023-10-25T00:04:59.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c7e27b640 1 -- start start 2023-10-25T00:04:59.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c7e27b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5c780720c0 0x7f5c780c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:59.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c7e27b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78157e50 0x7f5c781582a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:59.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c7e27b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c78154860 con 0x7f5c7813dc20 2023-10-25T00:04:59.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c7e27b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c781587e0 con 0x7f5c780720c0 2023-10-25T00:04:59.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c7e27b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c780c1f10 con 0x7f5c78157e50 2023-10-25T00:04:59.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c77fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5c7813dc20 0x7f5c7806df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43096/0 (socket says 172.21.15.107:43096) 2023-10-25T00:04:59.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c77fff640 1 -- 172.21.15.107:0/3467999440 learned_addr learned my addr 172.21.15.107:0/3467999440 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:04:59.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c777fe640 1 --2- 172.21.15.107:0/3467999440 >> v2:172.21.15.110:3300/0 conn(0x7f5c780720c0 0x7f5c780c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:59.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c7c818640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78157e50 0x7f5c781582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:59.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734579536 0 0) 0x7f5c78154860 con 0x7f5c7813dc20 2023-10-25T00:04:59.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c60003530 con 0x7f5c7813dc20 2023-10-25T00:04:59.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 472610371 0 0) 0x7f5c60003530 con 0x7f5c7813dc20 2023-10-25T00:04:59.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78157e50 msgr2=0x7f5c781582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:59.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78157e50 0x7f5c781582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 >> v2:172.21.15.110:3300/0 conn(0x7f5c780720c0 msgr2=0x7f5c780c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:59.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 --2- 172.21.15.107:0/3467999440 >> v2:172.21.15.110:3300/0 conn(0x7f5c780720c0 0x7f5c780c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c78154b20 con 0x7f5c7813dc20 2023-10-25T00:04:59.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5c68001a00 con 0x7f5c7813dc20 2023-10-25T00:04:59.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 >> v1:172.21.15.107:6789/0 conn(0x7f5c7813dc20 legacy=0x7f5c7806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:59.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c60003980 0x7f5c60003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:59.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78157e50 0x7f5c600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:59.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c60004d10 0x7f5c60005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:59.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c781587e0 con 0x7f5c60003980 2023-10-25T00:04:59.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c780c1f10 con 0x7f5c60004d10 2023-10-25T00:04:59.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c6000ba10 con 0x7f5c78157e50 2023-10-25T00:04:59.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c77fff640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c60003980 0x7f5c60003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:59.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.265+0000 7f5c777fe640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78157e50 0x7f5c600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:59.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c7c818640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c60004d10 0x7f5c60005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c77fff640 1 -- 172.21.15.107:0/3467999440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78157e50 msgr2=0x7f5c600047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c77fff640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78157e50 0x7f5c600047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c77fff640 1 -- 172.21.15.107:0/3467999440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c60004d10 msgr2=0x7f5c60005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:04:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c77fff640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c60004d10 0x7f5c60005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c77fff640 1 -- 172.21.15.107:0/3467999440 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c60003530 con 0x7f5c60003980 2023-10-25T00:04:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c77fff640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c60003980 0x7f5c60003da0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800a830 tx=0x7f5c6801d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb1e35586c23101 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c680ad630 con 0x7f5c60003980 2023-10-25T00:04:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c680a8070 con 0x7f5c60003980 2023-10-25T00:04:59.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c76ffd640 1 -- 172.21.15.107:0/3467999440 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c680ad9d0 con 0x7f5c60003980 2023-10-25T00:04:59.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3467999440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c60003980 msgr2=0x7f5c60003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:59.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c7e27b640 1 --2- 172.21.15.107:0/3467999440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c60003980 0x7f5c60003da0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800a830 tx=0x7f5c6801d340 comp rx=0 tx=0).stop 2023-10-25T00:04:59.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c7c818640 1 -- 172.21.15.107:0/3467999440 reap_dead start 2023-10-25T00:04:59.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3467999440 shutdown_connections 2023-10-25T00:04:59.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3467999440 >> 172.21.15.107:0/3467999440 conn(0x7f5c780c39a0 msgr2=0x7f5c780c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:59.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3467999440 shutdown_connections 2023-10-25T00:04:59.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.269+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3467999440 wait complete. 2023-10-25T00:04:59.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7e27b640 1 Processor -- start 2023-10-25T00:04:59.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7e27b640 1 -- start start 2023-10-25T00:04:59.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7e27b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c60003980 0x7f5c781368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7e27b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c781579f0 0x7f5c780706a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7e27b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78070be0 0x7f5c781ecf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7e27b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c781ec880 con 0x7f5c781579f0 2023-10-25T00:04:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7e27b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c780c56b0 con 0x7f5c60003980 2023-10-25T00:04:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7e27b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c780c53d0 con 0x7f5c78070be0 2023-10-25T00:04:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c781579f0 0x7f5c780706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c781579f0 0x7f5c780706a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49692/0 (socket says 172.21.15.107:49692) 2023-10-25T00:04:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c777fe640 1 -- 172.21.15.107:0/3602063626 learned_addr learned my addr 172.21.15.107:0/3602063626 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:04:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c77fff640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c60003980 0x7f5c781368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7c818640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78070be0 0x7f5c781ecf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c777fe640 1 -- 172.21.15.107:0/3602063626 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78070be0 msgr2=0x7f5c781ecf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c777fe640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78070be0 0x7f5c781ecf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c777fe640 1 -- 172.21.15.107:0/3602063626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c60003980 msgr2=0x7f5c781368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c777fe640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c60003980 0x7f5c781368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c777fe640 1 -- 172.21.15.107:0/3602063626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c780abe70 con 0x7f5c781579f0 2023-10-25T00:04:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c777fe640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c781579f0 0x7f5c780706a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c011660 tx=0x7f5c6c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c74ff9640 1 -- 172.21.15.107:0/3602063626 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c00f9c0 con 0x7f5c781579f0 2023-10-25T00:04:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c74ff9640 1 -- 172.21.15.107:0/3602063626 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c00fb20 con 0x7f5c781579f0 2023-10-25T00:04:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3602063626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c780ac150 con 0x7f5c781579f0 2023-10-25T00:04:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c74ff9640 1 -- 172.21.15.107:0/3602063626 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c00fe90 con 0x7f5c781579f0 2023-10-25T00:04:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3602063626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c781ed4a0 con 0x7f5c781579f0 2023-10-25T00:04:59.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.273+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3602063626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c780ac390 con 0x7f5c781579f0 2023-10-25T00:04:59.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.277+0000 7f5c74ff9640 1 -- 172.21.15.107:0/3602063626 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5c6c015460 con 0x7f5c781579f0 2023-10-25T00:04:59.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.277+0000 7f5c74ff9640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c64065e60 0x7f5c64068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:04:59.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.277+0000 7f5c77fff640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c64065e60 0x7f5c64068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:04:59.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.277+0000 7f5c74ff9640 1 -- 172.21.15.107:0/3602063626 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f5c6c084a20 con 0x7f5c781579f0 2023-10-25T00:04:59.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.277+0000 7f5c77fff640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c64065e60 0x7f5c64068320 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5c680ad000 tx=0x7f5c68001370 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:04:59.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.281+0000 7f5c74ff9640 1 -- 172.21.15.107:0/3602063626 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5c780ac390 con 0x7f5c781579f0 2023-10-25T00:04:59.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.505+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3602063626 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c780ac390 con 0x7f5c781579f0 2023-10-25T00:04:59.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.505+0000 7f5c74ff9640 1 -- 172.21.15.107:0/3602063626 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f5c780ac390 con 0x7f5c781579f0 2023-10-25T00:04:59.510 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:04:59.510 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:04:59.511 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:04:59.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3602063626 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c64065e60 msgr2=0x7f5c64068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:59.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c64065e60 0x7f5c64068320 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5c680ad000 tx=0x7f5c68001370 comp rx=0 tx=0).stop 2023-10-25T00:04:59.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3602063626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c781579f0 msgr2=0x7f5c780706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:04:59.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c781579f0 0x7f5c780706a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c011660 tx=0x7f5c6c011b20 comp rx=0 tx=0).stop 2023-10-25T00:04:59.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3602063626 shutdown_connections 2023-10-25T00:04:59.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5c64065e60 0x7f5c64068320 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5c78070be0 0x7f5c781ecf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c781579f0 0x7f5c780706a0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 --2- 172.21.15.107:0/3602063626 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5c60003980 0x7f5c781368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:04:59.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3602063626 >> 172.21.15.107:0/3602063626 conn(0x7f5c780c39a0 msgr2=0x7f5c78139980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:04:59.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3602063626 shutdown_connections 2023-10-25T00:04:59.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:04:59.509+0000 7f5c7e27b640 1 -- 172.21.15.107:0/3602063626 wait complete. 2023-10-25T00:04:59.533 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:05:01.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:01.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df4017640 1 Processor -- start 2023-10-25T00:05:01.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df4017640 1 -- start start 2023-10-25T00:05:01.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df4017640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4dec14fd40 0x7f4dec0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:01.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df4017640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4dec06ee40 0x7f4dec0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:01.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df4017640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dec06f730 con 0x7f4dec0a9820 2023-10-25T00:05:01.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df4017640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dec0aa750 con 0x7f4dec14fd40 2023-10-25T00:05:01.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df4017640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dec156a40 con 0x7f4dec06ee40 2023-10-25T00:05:01.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df1db3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4dec14fd40 0x7f4dec0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:01.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df15b2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4dec0a9820 0x7f4dec0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43100/0 (socket says 172.21.15.107:43100) 2023-10-25T00:05:01.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df15b2640 1 -- 172.21.15.107:0/2024251232 learned_addr learned my addr 172.21.15.107:0/2024251232 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:01.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df25b4640 1 --2- 172.21.15.107:0/2024251232 >> v2:172.21.15.110:3300/0 conn(0x7f4dec06ee40 0x7f4dec0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:01.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df1db3640 1 -- 172.21.15.107:0/2024251232 >> v2:172.21.15.110:3300/0 conn(0x7f4dec06ee40 msgr2=0x7f4dec0aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:01.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df1db3640 1 --2- 172.21.15.107:0/2024251232 >> v2:172.21.15.110:3300/0 conn(0x7f4dec06ee40 0x7f4dec0aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:01.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.669+0000 7f4df1db3640 1 -- 172.21.15.107:0/2024251232 >> v1:172.21.15.107:6789/0 conn(0x7f4dec0a9820 legacy=0x7f4dec0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:01.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.673+0000 7f4df1db3640 1 -- 172.21.15.107:0/2024251232 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec156710 con 0x7f4dec14fd40 2023-10-25T00:05:01.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.673+0000 7f4df1db3640 1 --2- 172.21.15.107:0/2024251232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec14fd40 0x7f4dec0ab090 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f4dec1569b0 tx=0x7f4de801bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2231ccb15e58b4e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:01.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.673+0000 7f4df0db1640 1 -- 172.21.15.107:0/2024251232 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de80a85f0 con 0x7f4dec14fd40 2023-10-25T00:05:01.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df0db1640 1 -- 172.21.15.107:0/2024251232 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de80a8750 con 0x7f4dec14fd40 2023-10-25T00:05:01.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df0db1640 1 -- 172.21.15.107:0/2024251232 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de80a8ae0 con 0x7f4dec14fd40 2023-10-25T00:05:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 -- 172.21.15.107:0/2024251232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec14fd40 msgr2=0x7f4dec0ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:01.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 --2- 172.21.15.107:0/2024251232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec14fd40 0x7f4dec0ab090 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f4dec1569b0 tx=0x7f4de801bf10 comp rx=0 tx=0).stop 2023-10-25T00:05:01.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 -- 172.21.15.107:0/2024251232 shutdown_connections 2023-10-25T00:05:01.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 --2- 172.21.15.107:0/2024251232 >> v2:172.21.15.110:3300/0 conn(0x7f4dec06ee40 0x7f4dec0aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:01.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 --2- 172.21.15.107:0/2024251232 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec14fd40 0x7f4dec0ab090 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:01.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 -- 172.21.15.107:0/2024251232 >> 172.21.15.107:0/2024251232 conn(0x7f4dec0c2950 msgr2=0x7f4dec0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:01.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 -- 172.21.15.107:0/2024251232 shutdown_connections 2023-10-25T00:05:01.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 -- 172.21.15.107:0/2024251232 wait complete. 2023-10-25T00:05:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 Processor -- start 2023-10-25T00:05:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 -- start start 2023-10-25T00:05:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec06ee40 0x7f4dec148260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec14fd40 0x7f4dec1592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dec0bae70 0x7f4dec1597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec071d90 con 0x7f4dec06ee40 2023-10-25T00:05:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec0bfa50 con 0x7f4dec14fd40 2023-10-25T00:05:01.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df4017640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec1487a0 con 0x7f4dec0bae70 2023-10-25T00:05:01.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df1db3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec06ee40 0x7f4dec148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:01.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df1db3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec06ee40 0x7f4dec148260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49700/0 (socket says 172.21.15.107:49700) 2023-10-25T00:05:01.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df1db3640 1 -- 172.21.15.107:0/1724428809 learned_addr learned my addr 172.21.15.107:0/1724428809 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:01.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df1db3640 1 -- 172.21.15.107:0/1724428809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dec0bae70 msgr2=0x7f4dec1597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:01.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df1db3640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dec0bae70 0x7f4dec1597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:01.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df15b2640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec14fd40 0x7f4dec1592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:01.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df1db3640 1 -- 172.21.15.107:0/1724428809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec14fd40 msgr2=0x7f4dec1592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:01.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df1db3640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec14fd40 0x7f4dec1592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:01.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df1db3640 1 -- 172.21.15.107:0/1724428809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de8009420 con 0x7f4dec06ee40 2023-10-25T00:05:01.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.677+0000 7f4df1db3640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec06ee40 0x7f4dec148260 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f4de80aa040 tx=0x7f4de801a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:01.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.681+0000 7f4de27fc640 1 -- 172.21.15.107:0/1724428809 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de80a8570 con 0x7f4dec06ee40 2023-10-25T00:05:01.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.681+0000 7f4de27fc640 1 -- 172.21.15.107:0/1724428809 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de80b9410 con 0x7f4dec06ee40 2023-10-25T00:05:01.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.681+0000 7f4df4017640 1 -- 172.21.15.107:0/1724428809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dec1efb60 con 0x7f4dec06ee40 2023-10-25T00:05:01.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.681+0000 7f4de27fc640 1 -- 172.21.15.107:0/1724428809 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de80b97a0 con 0x7f4dec06ee40 2023-10-25T00:05:01.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.681+0000 7f4df4017640 1 -- 172.21.15.107:0/1724428809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dec0bff80 con 0x7f4dec06ee40 2023-10-25T00:05:01.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.681+0000 7f4df4017640 1 -- 172.21.15.107:0/1724428809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dec0c24a0 con 0x7f4dec06ee40 2023-10-25T00:05:01.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.681+0000 7f4de27fc640 1 -- 172.21.15.107:0/1724428809 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4de80b03f0 con 0x7f4dec06ee40 2023-10-25T00:05:01.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.681+0000 7f4de27fc640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4dd0065e60 0x7f4dd0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:01.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.681+0000 7f4de27fc640 1 -- 172.21.15.107:0/1724428809 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f4de811d770 con 0x7f4dec06ee40 2023-10-25T00:05:01.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.681+0000 7f4df15b2640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4dd0065e60 0x7f4dd0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:01.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.681+0000 7f4df15b2640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4dd0065e60 0x7f4dd0068320 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4dec153030 tx=0x7f4ddc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:01.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.685+0000 7f4de27fc640 1 -- 172.21.15.107:0/1724428809 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4de80ea530 con 0x7f4dec06ee40 2023-10-25T00:05:01.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.873+0000 7f4df4017640 1 -- 172.21.15.107:0/1724428809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dec070840 con 0x7f4dec06ee40 2023-10-25T00:05:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.873+0000 7f4de27fc640 1 -- 172.21.15.107:0/1724428809 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f4de80edb50 con 0x7f4dec06ee40 2023-10-25T00:05:01.877 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:01.877 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:05:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 -- 172.21.15.107:0/1724428809 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4dd0065e60 msgr2=0x7f4dd0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4dd0065e60 0x7f4dd0068320 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f4dec153030 tx=0x7f4ddc009040 comp rx=0 tx=0).stop 2023-10-25T00:05:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 -- 172.21.15.107:0/1724428809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec06ee40 msgr2=0x7f4dec148260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec06ee40 0x7f4dec148260 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f4de80aa040 tx=0x7f4de801a750 comp rx=0 tx=0).stop 2023-10-25T00:05:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 -- 172.21.15.107:0/1724428809 shutdown_connections 2023-10-25T00:05:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4dd0065e60 0x7f4dd0068320 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dec0bae70 0x7f4dec1597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec14fd40 0x7f4dec1592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 --2- 172.21.15.107:0/1724428809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dec06ee40 0x7f4dec148260 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 -- 172.21.15.107:0/1724428809 >> 172.21.15.107:0/1724428809 conn(0x7f4dec0c2950 msgr2=0x7f4dec0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 -- 172.21.15.107:0/1724428809 shutdown_connections 2023-10-25T00:05:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:01.877+0000 7f4df4017640 1 -- 172.21.15.107:0/1724428809 wait complete. 2023-10-25T00:05:01.894 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:05:03.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:04.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.021+0000 7f37f1258640 1 Processor -- start 2023-10-25T00:05:04.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.021+0000 7f37f1258640 1 -- start start 2023-10-25T00:05:04.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37f1258640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec06ee40 0x7f37ec0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:04.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37f1258640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec06f730 con 0x7f37ec14fd40 2023-10-25T00:05:04.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37f1258640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec156a40 con 0x7f37ec0a9820 2023-10-25T00:05:04.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37f1258640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec156710 con 0x7f37ec06ee40 2023-10-25T00:05:04.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37ead9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f37ec14fd40 0x7f37ec0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54020/0 (socket says 172.21.15.107:54020) 2023-10-25T00:05:04.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37ead9d640 1 -- 172.21.15.107:0/977325339 learned_addr learned my addr 172.21.15.107:0/977325339 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:04.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37eb59e640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec06ee40 0x7f37ec0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:04.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526919346 0 0) 0x7f37ec06f730 con 0x7f37ec14fd40 2023-10-25T00:05:04.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d4003530 con 0x7f37ec14fd40 2023-10-25T00:05:04.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 799129544 0 0) 0x7f37d4003530 con 0x7f37ec14fd40 2023-10-25T00:05:04.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec06ee40 msgr2=0x7f37ec0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:04.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec06ee40 0x7f37ec0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:04.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 >> v1:172.21.15.110:6789/0 conn(0x7f37ec0a9820 legacy=0x7f37ec15af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:04.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec06f980 con 0x7f37ec14fd40 2023-10-25T00:05:04.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f37dc001680 con 0x7f37ec14fd40 2023-10-25T00:05:04.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 >> v1:172.21.15.107:6789/0 conn(0x7f37ec14fd40 legacy=0x7f37ec0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:04.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec06ee40 0x7f37d40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:04.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 0x7f37d40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:04.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37d4004d10 0x7f37d4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:04.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec156710 con 0x7f37d4003f30 2023-10-25T00:05:04.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d400bbd0 con 0x7f37d4004d10 2023-10-25T00:05:04.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d400bd90 con 0x7f37ec06ee40 2023-10-25T00:05:04.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37ea59c640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 0x7f37d40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:04.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37ead9d640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec06ee40 0x7f37d40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:04.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37eb59e640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37d4004d10 0x7f37d4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:04.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37ea59c640 1 -- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec06ee40 msgr2=0x7f37d40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37ea59c640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec06ee40 0x7f37d40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37ea59c640 1 -- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37d4004d10 msgr2=0x7f37d4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37ea59c640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37d4004d10 0x7f37d4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37ea59c640 1 -- 172.21.15.107:0/977325339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d4003530 con 0x7f37d4003f30 2023-10-25T00:05:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.025+0000 7f37ead9d640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec06ee40 0x7f37d40039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:05:04.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37ea59c640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 0x7f37d40047d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f37e0001780 tx=0x7f37e001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e9690786375da8c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:04.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e00052d0 con 0x7f37d4003f30 2023-10-25T00:05:04.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e00a7050 con 0x7f37d4003f30 2023-10-25T00:05:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37e9d9b640 1 -- 172.21.15.107:0/977325339 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e0003600 con 0x7f37d4003f30 2023-10-25T00:05:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 -- 172.21.15.107:0/977325339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 msgr2=0x7f37d40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 --2- 172.21.15.107:0/977325339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 0x7f37d40047d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f37e0001780 tx=0x7f37e001c400 comp rx=0 tx=0).stop 2023-10-25T00:05:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37eb59e640 1 -- 172.21.15.107:0/977325339 reap_dead start 2023-10-25T00:05:04.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 -- 172.21.15.107:0/977325339 shutdown_connections 2023-10-25T00:05:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 -- 172.21.15.107:0/977325339 >> 172.21.15.107:0/977325339 conn(0x7f37ec0c2950 msgr2=0x7f37ec0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 -- 172.21.15.107:0/977325339 shutdown_connections 2023-10-25T00:05:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 -- 172.21.15.107:0/977325339 wait complete. 2023-10-25T00:05:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 Processor -- start 2023-10-25T00:05:04.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 -- start start 2023-10-25T00:05:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 0x7f37ec1eb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37ec0bae70 0x7f37ec0bfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec13b3c0 0x7f37ec13b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec0bb2f0 con 0x7f37d4003f30 2023-10-25T00:05:04.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec14c5a0 con 0x7f37ec0bae70 2023-10-25T00:05:04.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37f1258640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ec0bb500 con 0x7f37ec13b3c0 2023-10-25T00:05:04.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37ead9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 0x7f37ec1eb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:04.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37ead9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 0x7f37ec1eb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46680/0 (socket says 172.21.15.107:46680) 2023-10-25T00:05:04.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.029+0000 7f37ead9d640 1 -- 172.21.15.107:0/2856980553 learned_addr learned my addr 172.21.15.107:0/2856980553 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:04.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37ea59c640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37ec0bae70 0x7f37ec0bfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:04.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37eb59e640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec13b3c0 0x7f37ec13b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:04.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37ead9d640 1 -- 172.21.15.107:0/2856980553 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec13b3c0 msgr2=0x7f37ec13b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:04.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37ead9d640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec13b3c0 0x7f37ec13b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:04.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37ead9d640 1 -- 172.21.15.107:0/2856980553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37ec0bae70 msgr2=0x7f37ec0bfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:04.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37ead9d640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37ec0bae70 0x7f37ec0bfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:04.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37ead9d640 1 -- 172.21.15.107:0/2856980553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec0bb7e0 con 0x7f37d4003f30 2023-10-25T00:05:04.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37ea59c640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37ec0bae70 0x7f37ec0bfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:05:04.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37ead9d640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 0x7f37ec1eb2f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00e7b0 tx=0x7f37dc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:04.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37d37fe640 1 -- 172.21.15.107:0/2856980553 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc00ee60 con 0x7f37d4003f30 2023-10-25T00:05:04.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37d37fe640 1 -- 172.21.15.107:0/2856980553 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc00c8d0 con 0x7f37d4003f30 2023-10-25T00:05:04.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37f1258640 1 -- 172.21.15.107:0/2856980553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ec0bbac0 con 0x7f37d4003f30 2023-10-25T00:05:04.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37d37fe640 1 -- 172.21.15.107:0/2856980553 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37dc00cc40 con 0x7f37d4003f30 2023-10-25T00:05:04.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37f1258640 1 -- 172.21.15.107:0/2856980553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ec149b70 con 0x7f37d4003f30 2023-10-25T00:05:04.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37f1258640 1 -- 172.21.15.107:0/2856980553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ec0c24a0 con 0x7f37d4003f30 2023-10-25T00:05:04.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37d37fe640 1 -- 172.21.15.107:0/2856980553 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f37dc01d460 con 0x7f37d4003f30 2023-10-25T00:05:04.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37d37fe640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37d8065e60 0x7f37d8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:04.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37d37fe640 1 -- 172.21.15.107:0/2856980553 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f37dc0840b0 con 0x7f37d4003f30 2023-10-25T00:05:04.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37ea59c640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37d8065e60 0x7f37d8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:04.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.033+0000 7f37ea59c640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37d8065e60 0x7f37d8068320 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f37e0001750 tx=0x7f37e00011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:04.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.037+0000 7f37d37fe640 1 -- 172.21.15.107:0/2856980553 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f37dc054890 con 0x7f37d4003f30 2023-10-25T00:05:04.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 -- 172.21.15.107:0/2856980553 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37ec0746e0 con 0x7f37d4003f30 2023-10-25T00:05:04.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37d37fe640 1 -- 172.21.15.107:0/2856980553 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f37dc054420 con 0x7f37d4003f30 2023-10-25T00:05:04.235 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:04.236 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:05:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 -- 172.21.15.107:0/2856980553 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37d8065e60 msgr2=0x7f37d8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37d8065e60 0x7f37d8068320 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f37e0001750 tx=0x7f37e00011a0 comp rx=0 tx=0).stop 2023-10-25T00:05:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 -- 172.21.15.107:0/2856980553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 msgr2=0x7f37ec1eb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 0x7f37ec1eb2f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00e7b0 tx=0x7f37dc00ec70 comp rx=0 tx=0).stop 2023-10-25T00:05:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 -- 172.21.15.107:0/2856980553 shutdown_connections 2023-10-25T00:05:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37d8065e60 0x7f37d8068320 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37ec13b3c0 0x7f37ec13b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37ec0bae70 0x7f37ec0bfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 --2- 172.21.15.107:0/2856980553 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37d4003f30 0x7f37ec1eb2f0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 -- 172.21.15.107:0/2856980553 >> 172.21.15.107:0/2856980553 conn(0x7f37ec0c2950 msgr2=0x7f37ec14b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 -- 172.21.15.107:0/2856980553 shutdown_connections 2023-10-25T00:05:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:04.233+0000 7f37f1258640 1 -- 172.21.15.107:0/2856980553 wait complete. 2023-10-25T00:05:04.252 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:05:06.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:06.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:06.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03ebfa2640 1 Processor -- start 2023-10-25T00:05:06.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03ebfa2640 1 -- start start 2023-10-25T00:05:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03ebfa2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03e413dc20 0x7f03e40c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03ebfa2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f03e4157e50 0x7f03e4158260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03ebfa2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e41587a0 con 0x7f03e40720c0 2023-10-25T00:05:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03ebfa2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e4154860 con 0x7f03e413dc20 2023-10-25T00:05:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03ebfa2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e40c1f10 con 0x7f03e4157e50 2023-10-25T00:05:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03e9d3e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03e413dc20 0x7f03e40c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03e9d3e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f03e413dc20 0x7f03e40c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46690/0 (socket says 172.21.15.107:46690) 2023-10-25T00:05:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03e953d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f03e40720c0 0x7f03e406df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54022/0 (socket says 172.21.15.107:54022) 2023-10-25T00:05:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03e9d3e640 1 -- 172.21.15.107:0/590723883 learned_addr learned my addr 172.21.15.107:0/590723883 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03ea53f640 1 --2- 172.21.15.107:0/590723883 >> v2:172.21.15.110:3300/0 conn(0x7f03e4157e50 0x7f03e4158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03e9d3e640 1 -- 172.21.15.107:0/590723883 >> v2:172.21.15.110:3300/0 conn(0x7f03e4157e50 msgr2=0x7f03e4158260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03e9d3e640 1 --2- 172.21.15.107:0/590723883 >> v2:172.21.15.110:3300/0 conn(0x7f03e4157e50 0x7f03e4158260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03e9d3e640 1 -- 172.21.15.107:0/590723883 >> v1:172.21.15.107:6789/0 conn(0x7f03e40720c0 legacy=0x7f03e406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.261+0000 7f03e9d3e640 1 -- 172.21.15.107:0/590723883 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e4154b20 con 0x7f03e413dc20 2023-10-25T00:05:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e9d3e640 1 --2- 172.21.15.107:0/590723883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03e413dc20 0x7f03e40c18a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f03e40c05a0 tx=0x7f03e001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c826da903d163b6b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e8d3c640 1 -- 172.21.15.107:0/590723883 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e00ab600 con 0x7f03e413dc20 2023-10-25T00:05:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e8d3c640 1 -- 172.21.15.107:0/590723883 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e00a7050 con 0x7f03e413dc20 2023-10-25T00:05:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e8d3c640 1 -- 172.21.15.107:0/590723883 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e00ab980 con 0x7f03e413dc20 2023-10-25T00:05:06.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 -- 172.21.15.107:0/590723883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03e413dc20 msgr2=0x7f03e40c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 --2- 172.21.15.107:0/590723883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03e413dc20 0x7f03e40c18a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f03e40c05a0 tx=0x7f03e001c3e0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 -- 172.21.15.107:0/590723883 shutdown_connections 2023-10-25T00:05:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 --2- 172.21.15.107:0/590723883 >> v2:172.21.15.110:3300/0 conn(0x7f03e4157e50 0x7f03e4158260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 --2- 172.21.15.107:0/590723883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03e413dc20 0x7f03e40c18a0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 -- 172.21.15.107:0/590723883 >> 172.21.15.107:0/590723883 conn(0x7f03e40c39a0 msgr2=0x7f03e40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:06.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 -- 172.21.15.107:0/590723883 shutdown_connections 2023-10-25T00:05:06.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 -- 172.21.15.107:0/590723883 wait complete. 2023-10-25T00:05:06.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 Processor -- start 2023-10-25T00:05:06.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 -- start start 2023-10-25T00:05:06.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03e40720c0 0x7f03e40c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03e413dc20 0x7f03e41526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e4157e50 0x7f03e4152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e4135d20 con 0x7f03e413dc20 2023-10-25T00:05:06.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e41ed390 con 0x7f03e4157e50 2023-10-25T00:05:06.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ebfa2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e41ed670 con 0x7f03e40720c0 2023-10-25T00:05:06.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e953d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03e413dc20 0x7f03e41526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e953d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03e413dc20 0x7f03e41526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46692/0 (socket says 172.21.15.107:46692) 2023-10-25T00:05:06.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e953d640 1 -- 172.21.15.107:0/926221450 learned_addr learned my addr 172.21.15.107:0/926221450 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:06.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03ea53f640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e4157e50 0x7f03e4152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e9d3e640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03e40720c0 0x7f03e40c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e953d640 1 -- 172.21.15.107:0/926221450 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03e40720c0 msgr2=0x7f03e40c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e953d640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03e40720c0 0x7f03e40c4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e953d640 1 -- 172.21.15.107:0/926221450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e4157e50 msgr2=0x7f03e4152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e953d640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e4157e50 0x7f03e4152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.265+0000 7f03e953d640 1 -- 172.21.15.107:0/926221450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e00094d0 con 0x7f03e413dc20 2023-10-25T00:05:06.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03e953d640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03e413dc20 0x7f03e41526b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f03d400c430 tx=0x7f03d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:06.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03da7fc640 1 -- 172.21.15.107:0/926221450 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d40074e0 con 0x7f03e413dc20 2023-10-25T00:05:06.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03da7fc640 1 -- 172.21.15.107:0/926221450 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d4007640 con 0x7f03e413dc20 2023-10-25T00:05:06.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03ebfa2640 1 -- 172.21.15.107:0/926221450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e41540b0 con 0x7f03e413dc20 2023-10-25T00:05:06.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03da7fc640 1 -- 172.21.15.107:0/926221450 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d4004a10 con 0x7f03e413dc20 2023-10-25T00:05:06.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03ebfa2640 1 -- 172.21.15.107:0/926221450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e40c54c0 con 0x7f03e413dc20 2023-10-25T00:05:06.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03ebfa2640 1 -- 172.21.15.107:0/926221450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e40746e0 con 0x7f03e413dc20 2023-10-25T00:05:06.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03da7fc640 1 -- 172.21.15.107:0/926221450 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f03d4004b70 con 0x7f03e413dc20 2023-10-25T00:05:06.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03da7fc640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03c8065e90 0x7f03c8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03da7fc640 1 -- 172.21.15.107:0/926221450 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f03d4083bd0 con 0x7f03e413dc20 2023-10-25T00:05:06.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03e9d3e640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03c8065e90 0x7f03c8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.269+0000 7f03e9d3e640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03c8065e90 0x7f03c8068350 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f03e00ad000 tx=0x7f03e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:06.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.273+0000 7f03da7fc640 1 -- 172.21.15.107:0/926221450 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f03d4050870 con 0x7f03e413dc20 2023-10-25T00:05:06.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4875adc640 1 Processor -- start 2023-10-25T00:05:06.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4875adc640 1 -- start start 2023-10-25T00:05:06.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4875adc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f487014eb70 0x7f48700c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4875adc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48701514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4875adc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48700c22f0 con 0x7f4870151a20 2023-10-25T00:05:06.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4875adc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4870071700 con 0x7f487014eb70 2023-10-25T00:05:06.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4875adc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48700c25d0 con 0x7f4870070b00 2023-10-25T00:05:06.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4867fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48701514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f486f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f487014eb70 0x7f48700c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4867fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48701514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:44654/0 (socket says 172.21.15.107:44654) 2023-10-25T00:05:06.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4867fff640 1 -- 172.21.15.107:0/4129835908 learned_addr learned my addr 172.21.15.107:0/4129835908 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:06.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4867fff640 1 -- 172.21.15.107:0/4129835908 >> v2:172.21.15.110:3300/0 conn(0x7f487014eb70 msgr2=0x7f48700c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:06.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4867fff640 1 --2- 172.21.15.107:0/4129835908 >> v2:172.21.15.110:3300/0 conn(0x7f487014eb70 0x7f48700c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4867fff640 1 -- 172.21.15.107:0/4129835908 >> v1:172.21.15.110:6789/0 conn(0x7f4870151a20 legacy=0x7f487006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f4867fff640 1 -- 172.21.15.107:0/4129835908 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700c2be0 con 0x7f4870070b00 2023-10-25T00:05:06.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.369+0000 7f486f7fe640 1 --2- 172.21.15.107:0/4129835908 >> v2:172.21.15.110:3300/0 conn(0x7f487014eb70 0x7f48700c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:05:06.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4867fff640 1 --2- 172.21.15.107:0/4129835908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48701514e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4858009870 tx=0x7f485801c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4b8cabbcd2c9b81 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:06.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f486effd640 1 -- 172.21.15.107:0/4129835908 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48580aa640 con 0x7f4870070b00 2023-10-25T00:05:06.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f486effd640 1 -- 172.21.15.107:0/4129835908 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48580aa7a0 con 0x7f4870070b00 2023-10-25T00:05:06.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f486effd640 1 -- 172.21.15.107:0/4129835908 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48580aab30 con 0x7f4870070b00 2023-10-25T00:05:06.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 -- 172.21.15.107:0/4129835908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 msgr2=0x7f48701514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 --2- 172.21.15.107:0/4129835908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48701514e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4858009870 tx=0x7f485801c540 comp rx=0 tx=0).stop 2023-10-25T00:05:06.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 -- 172.21.15.107:0/4129835908 shutdown_connections 2023-10-25T00:05:06.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 --2- 172.21.15.107:0/4129835908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870070b00 0x7f48701514e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 --2- 172.21.15.107:0/4129835908 >> v2:172.21.15.110:3300/0 conn(0x7f487014eb70 0x7f48700c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 -- 172.21.15.107:0/4129835908 >> 172.21.15.107:0/4129835908 conn(0x7f48700c39a0 msgr2=0x7f48700c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:06.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 -- 172.21.15.107:0/4129835908 shutdown_connections 2023-10-25T00:05:06.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 -- 172.21.15.107:0/4129835908 wait complete. 2023-10-25T00:05:06.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 Processor -- start 2023-10-25T00:05:06.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 -- start start 2023-10-25T00:05:06.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4870070b00 0x7f4870155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487014eb70 0x7f487014dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870151a20 0x7f4870140750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48701556b0 con 0x7f487014eb70 2023-10-25T00:05:06.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48701eb4d0 con 0x7f4870070b00 2023-10-25T00:05:06.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4875adc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4870155910 con 0x7f4870151a20 2023-10-25T00:05:06.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4867fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487014eb70 0x7f487014dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4867fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487014eb70 0x7f487014dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46706/0 (socket says 172.21.15.107:46706) 2023-10-25T00:05:06.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4867fff640 1 -- 172.21.15.107:0/3361454135 learned_addr learned my addr 172.21.15.107:0/3361454135 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:06.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f486f7fe640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4870070b00 0x7f4870155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f486ffff640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870151a20 0x7f4870140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4867fff640 1 -- 172.21.15.107:0/3361454135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870151a20 msgr2=0x7f4870140750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4867fff640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870151a20 0x7f4870140750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4867fff640 1 -- 172.21.15.107:0/3361454135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4870070b00 msgr2=0x7f4870155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4867fff640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4870070b00 0x7f4870155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f4867fff640 1 -- 172.21.15.107:0/3361454135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4860009420 con 0x7f487014eb70 2023-10-25T00:05:06.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.373+0000 7f486f7fe640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4870070b00 0x7f4870155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:05:06.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f4867fff640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487014eb70 0x7f487014dc90 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f48580ac040 tx=0x7f485801ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:06.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f486cff9640 1 -- 172.21.15.107:0/3361454135 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48580bd3f0 con 0x7f487014eb70 2023-10-25T00:05:06.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f486cff9640 1 -- 172.21.15.107:0/3361454135 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48580bd550 con 0x7f487014eb70 2023-10-25T00:05:06.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f4875adc640 1 -- 172.21.15.107:0/3361454135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48580094d0 con 0x7f487014eb70 2023-10-25T00:05:06.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f486cff9640 1 -- 172.21.15.107:0/3361454135 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48580bd900 con 0x7f487014eb70 2023-10-25T00:05:06.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f4875adc640 1 -- 172.21.15.107:0/3361454135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48701ea9f0 con 0x7f487014eb70 2023-10-25T00:05:06.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f4875adc640 1 -- 172.21.15.107:0/3361454135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48700746e0 con 0x7f487014eb70 2023-10-25T00:05:06.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f486cff9640 1 -- 172.21.15.107:0/3361454135 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f48580a8050 con 0x7f487014eb70 2023-10-25T00:05:06.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f486cff9640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4850065e60 0x7f4850068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f486cff9640 1 -- 172.21.15.107:0/3361454135 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f485811e600 con 0x7f487014eb70 2023-10-25T00:05:06.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f486f7fe640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4850065e60 0x7f4850068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.377+0000 7f486f7fe640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4850065e60 0x7f4850068320 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f487006c580 tx=0x7f4860008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:06.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.381+0000 7f486cff9640 1 -- 172.21.15.107:0/3361454135 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f48580eb440 con 0x7f487014eb70 2023-10-25T00:05:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.465+0000 7f03ebfa2640 1 -- 172.21.15.107:0/926221450 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03e4154560 con 0x7f03e413dc20 2023-10-25T00:05:06.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.465+0000 7f03da7fc640 1 -- 172.21.15.107:0/926221450 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f03d4053e90 con 0x7f03e413dc20 2023-10-25T00:05:06.469 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:06.469 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:06.470 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:05:06.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 -- 172.21.15.107:0/926221450 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03c8065e90 msgr2=0x7f03c8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03c8065e90 0x7f03c8068350 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f03e00ad000 tx=0x7f03e0008040 comp rx=0 tx=0).stop 2023-10-25T00:05:06.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 -- 172.21.15.107:0/926221450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03e413dc20 msgr2=0x7f03e41526b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03e413dc20 0x7f03e41526b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f03d400c430 tx=0x7f03d400c8f0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 -- 172.21.15.107:0/926221450 shutdown_connections 2023-10-25T00:05:06.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03e4157e50 0x7f03e4152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03e413dc20 0x7f03e41526b0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03c8065e90 0x7f03c8068350 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 --2- 172.21.15.107:0/926221450 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03e40720c0 0x7f03e40c4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 -- 172.21.15.107:0/926221450 >> 172.21.15.107:0/926221450 conn(0x7f03e40c39a0 msgr2=0x7f03e406c750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:06.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 -- 172.21.15.107:0/926221450 shutdown_connections 2023-10-25T00:05:06.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.469+0000 7f03ebfa2640 1 -- 172.21.15.107:0/926221450 wait complete. 2023-10-25T00:05:06.488 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2023-10-25T00:05:06.488 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:05:06.489 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:05:06.544 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:05:06.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:06.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.569+0000 7f4875adc640 1 -- 172.21.15.107:0/3361454135 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48700c1a50 con 0x7f487014eb70 2023-10-25T00:05:06.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.569+0000 7f486cff9640 1 -- 172.21.15.107:0/3361454135 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f48580eea60 con 0x7f487014eb70 2023-10-25T00:05:06.573 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:06.574 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:06.574 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:05:06.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 -- 172.21.15.107:0/3361454135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4850065e60 msgr2=0x7f4850068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4850065e60 0x7f4850068320 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f487006c580 tx=0x7f4860008040 comp rx=0 tx=0).stop 2023-10-25T00:05:06.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 -- 172.21.15.107:0/3361454135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487014eb70 msgr2=0x7f487014dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487014eb70 0x7f487014dc90 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f48580ac040 tx=0x7f485801ad80 comp rx=0 tx=0).stop 2023-10-25T00:05:06.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 -- 172.21.15.107:0/3361454135 shutdown_connections 2023-10-25T00:05:06.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4850065e60 0x7f4850068320 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4870151a20 0x7f4870140750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f487014eb70 0x7f487014dc90 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 --2- 172.21.15.107:0/3361454135 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4870070b00 0x7f4870155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 -- 172.21.15.107:0/3361454135 >> 172.21.15.107:0/3361454135 conn(0x7f48700c39a0 msgr2=0x7f487014cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:06.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 -- 172.21.15.107:0/3361454135 shutdown_connections 2023-10-25T00:05:06.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.573+0000 7f4875adc640 1 -- 172.21.15.107:0/3361454135 wait complete. 2023-10-25T00:05:06.593 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:05:06.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6eea30c640 1 Processor -- start 2023-10-25T00:05:06.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6eea30c640 1 -- start start 2023-10-25T00:05:06.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6eea30c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6ee4151960 0x7f6ee4071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6eea30c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ee40c24c0 0x7f6ee406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6eea30c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee4150bc0 con 0x7f6ee4071220 2023-10-25T00:05:06.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6eea30c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee406c7f0 con 0x7f6ee4151960 2023-10-25T00:05:06.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6eea30c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee406c510 con 0x7f6ee40c24c0 2023-10-25T00:05:06.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6ee37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6ee4151960 0x7f6ee4071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6ee37fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6ee4151960 0x7f6ee4071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46712/0 (socket says 172.21.15.107:46712) 2023-10-25T00:05:06.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6ee37fe640 1 -- 172.21.15.107:0/1389616203 learned_addr learned my addr 172.21.15.107:0/1389616203 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:06.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6ee88a9640 1 --2- 172.21.15.107:0/1389616203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ee40c24c0 0x7f6ee406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6ee37fe640 1 -- 172.21.15.107:0/1389616203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ee40c24c0 msgr2=0x7f6ee406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:06.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6ee37fe640 1 --2- 172.21.15.107:0/1389616203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ee40c24c0 0x7f6ee406a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6ee37fe640 1 -- 172.21.15.107:0/1389616203 >> v1:172.21.15.110:6789/0 conn(0x7f6ee4071220 legacy=0x7f6ee406bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:06.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.685+0000 7f6ee37fe640 1 -- 172.21.15.107:0/1389616203 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee4150e80 con 0x7f6ee4151960 2023-10-25T00:05:06.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6ee37fe640 1 --2- 172.21.15.107:0/1389616203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ee4151960 0x7f6ee4071a90 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009870 tx=0x7f6ed801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee74c017ac0f5406 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:06.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6ee2ffd640 1 -- 172.21.15.107:0/1389616203 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed80ab600 con 0x7f6ee4151960 2023-10-25T00:05:06.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6ee2ffd640 1 -- 172.21.15.107:0/1389616203 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed80a7050 con 0x7f6ee4151960 2023-10-25T00:05:06.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6ee2ffd640 1 -- 172.21.15.107:0/1389616203 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed80ab980 con 0x7f6ee4151960 2023-10-25T00:05:06.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6eea30c640 1 -- 172.21.15.107:0/1389616203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ee4151960 msgr2=0x7f6ee4071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6eea30c640 1 --2- 172.21.15.107:0/1389616203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ee4151960 0x7f6ee4071a90 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009870 tx=0x7f6ed801c470 comp rx=0 tx=0).stop 2023-10-25T00:05:06.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6eea30c640 1 -- 172.21.15.107:0/1389616203 shutdown_connections 2023-10-25T00:05:06.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6eea30c640 1 --2- 172.21.15.107:0/1389616203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ee40c24c0 0x7f6ee406a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6eea30c640 1 --2- 172.21.15.107:0/1389616203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ee4151960 0x7f6ee4071a90 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6eea30c640 1 -- 172.21.15.107:0/1389616203 >> 172.21.15.107:0/1389616203 conn(0x7f6ee40c2950 msgr2=0x7f6ee40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:06.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6eea30c640 1 -- 172.21.15.107:0/1389616203 shutdown_connections 2023-10-25T00:05:06.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6eea30c640 1 -- 172.21.15.107:0/1389616203 wait complete. 2023-10-25T00:05:06.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6eea30c640 1 Processor -- start 2023-10-25T00:05:06.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.689+0000 7f6eea30c640 1 -- start start 2023-10-25T00:05:06.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6eea30c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ee4071220 0x7f6ee413a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6eea30c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ee40c24c0 0x7f6ee413a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6eea30c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ee4151960 0x7f6ee4148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6eea30c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee414d900 con 0x7f6ee40c24c0 2023-10-25T00:05:06.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6eea30c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40bfb50 con 0x7f6ee4071220 2023-10-25T00:05:06.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6eea30c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee40baf10 con 0x7f6ee4151960 2023-10-25T00:05:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ee40c24c0 0x7f6ee413a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ee40c24c0 0x7f6ee413a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46720/0 (socket says 172.21.15.107:46720) 2023-10-25T00:05:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee37fe640 1 -- 172.21.15.107:0/1038512695 learned_addr learned my addr 172.21.15.107:0/1038512695 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:06.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee88a9640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ee4151960 0x7f6ee4148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee3fff640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ee4071220 0x7f6ee413a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee37fe640 1 -- 172.21.15.107:0/1038512695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ee4151960 msgr2=0x7f6ee4148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee37fe640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ee4151960 0x7f6ee4148d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee37fe640 1 -- 172.21.15.107:0/1038512695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ee4071220 msgr2=0x7f6ee413a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee37fe640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ee4071220 0x7f6ee413a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee37fe640 1 -- 172.21.15.107:0/1038512695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed80094d0 con 0x7f6ee40c24c0 2023-10-25T00:05:06.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee3fff640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ee4071220 0x7f6ee413a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:06.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee37fe640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ee40c24c0 0x7f6ee413a6c0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80ae040 tx=0x7f6ed8019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:06.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee0ff9640 1 -- 172.21.15.107:0/1038512695 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed80b6620 con 0x7f6ee40c24c0 2023-10-25T00:05:06.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee0ff9640 1 -- 172.21.15.107:0/1038512695 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed80b6780 con 0x7f6ee40c24c0 2023-10-25T00:05:06.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee0ff9640 1 -- 172.21.15.107:0/1038512695 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed80b6b30 con 0x7f6ee40c24c0 2023-10-25T00:05:06.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6eea30c640 1 -- 172.21.15.107:0/1038512695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee40bb1f0 con 0x7f6ee40c24c0 2023-10-25T00:05:06.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6eea30c640 1 -- 172.21.15.107:0/1038512695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee40bfe10 con 0x7f6ee40c24c0 2023-10-25T00:05:06.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6eea30c640 1 -- 172.21.15.107:0/1038512695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee40746e0 con 0x7f6ee40c24c0 2023-10-25T00:05:06.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.693+0000 7f6ee0ff9640 1 -- 172.21.15.107:0/1038512695 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6ed80b2040 con 0x7f6ee40c24c0 2023-10-25T00:05:06.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.697+0000 7f6ee0ff9640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed0065e60 0x7f6ed0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.697+0000 7f6ee3fff640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed0065e60 0x7f6ed0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.697+0000 7f6ee0ff9640 1 -- 172.21.15.107:0/1038512695 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f6ed811e650 con 0x7f6ee40c24c0 2023-10-25T00:05:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.697+0000 7f6ee3fff640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed0065e60 0x7f6ed0068320 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4003e40 tx=0x7f6ed4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:06.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.701+0000 7f6ee0ff9640 1 -- 172.21.15.107:0/1038512695 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6ed80eb2f0 con 0x7f6ee40c24c0 2023-10-25T00:05:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 -- 172.21.15.107:0/1038512695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ee40c22a0 con 0x7f6ee40c24c0 2023-10-25T00:05:06.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6ee0ff9640 1 -- 172.21.15.107:0/1038512695 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f6ed80ee910 con 0x7f6ee40c24c0 2023-10-25T00:05:06.884 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:06.884 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:06.885 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:05:06.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 -- 172.21.15.107:0/1038512695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed0065e60 msgr2=0x7f6ed0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed0065e60 0x7f6ed0068320 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4003e40 tx=0x7f6ed4009040 comp rx=0 tx=0).stop 2023-10-25T00:05:06.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 -- 172.21.15.107:0/1038512695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ee40c24c0 msgr2=0x7f6ee413a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:06.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ee40c24c0 0x7f6ee413a6c0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80ae040 tx=0x7f6ed8019210 comp rx=0 tx=0).stop 2023-10-25T00:05:06.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 -- 172.21.15.107:0/1038512695 shutdown_connections 2023-10-25T00:05:06.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6ed0065e60 0x7f6ed0068320 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ee4151960 0x7f6ee4148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ee40c24c0 0x7f6ee413a6c0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 --2- 172.21.15.107:0/1038512695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ee4071220 0x7f6ee413a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:06.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 -- 172.21.15.107:0/1038512695 >> 172.21.15.107:0/1038512695 conn(0x7f6ee40c2950 msgr2=0x7f6ee406bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:06.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 -- 172.21.15.107:0/1038512695 shutdown_connections 2023-10-25T00:05:06.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:06.881+0000 7f6eea30c640 1 -- 172.21.15.107:0/1038512695 wait complete. 2023-10-25T00:05:06.901 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:05:08.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54cbb7e640 1 Processor -- start 2023-10-25T00:05:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54cbb7e640 1 -- start start 2023-10-25T00:05:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54cbb7e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f54c40c1c60 0x7f54c406c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:08.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54cbb7e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c4155850 0x7f54c40bcb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54cbb7e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c4154cb0 con 0x7f54c40c2040 2023-10-25T00:05:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54cbb7e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c40bd290 con 0x7f54c40c1c60 2023-10-25T00:05:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54cbb7e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c40bd0a0 con 0x7f54c4155850 2023-10-25T00:05:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54c991a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f54c40c1c60 0x7f54c406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54c991a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f54c40c1c60 0x7f54c406c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46728/0 (socket says 172.21.15.107:46728) 2023-10-25T00:05:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54c991a640 1 -- 172.21.15.107:0/2382622462 learned_addr learned my addr 172.21.15.107:0/2382622462 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54ca11b640 1 --2- 172.21.15.107:0/2382622462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c4155850 0x7f54c40bcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54c991a640 1 -- 172.21.15.107:0/2382622462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c4155850 msgr2=0x7f54c40bcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54c991a640 1 --2- 172.21.15.107:0/2382622462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c4155850 0x7f54c40bcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54c991a640 1 -- 172.21.15.107:0/2382622462 >> v1:172.21.15.110:6789/0 conn(0x7f54c40c2040 legacy=0x7f54c4155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:08.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54c991a640 1 -- 172.21.15.107:0/2382622462 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c4152550 con 0x7f54c40c1c60 2023-10-25T00:05:08.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.721+0000 7f54ca11b640 1 --2- 172.21.15.107:0/2382622462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c4155850 0x7f54c40bcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:08.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54c991a640 1 --2- 172.21.15.107:0/2382622462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40c1c60 0x7f54c406c370 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f54c0009870 tx=0x7f54c001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=222788361a21a7f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:08.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54c8918640 1 -- 172.21.15.107:0/2382622462 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00ab650 con 0x7f54c40c1c60 2023-10-25T00:05:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54c8918640 1 -- 172.21.15.107:0/2382622462 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c00ab7b0 con 0x7f54c40c1c60 2023-10-25T00:05:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54c8918640 1 -- 172.21.15.107:0/2382622462 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00abb40 con 0x7f54c40c1c60 2023-10-25T00:05:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 -- 172.21.15.107:0/2382622462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40c1c60 msgr2=0x7f54c406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 --2- 172.21.15.107:0/2382622462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40c1c60 0x7f54c406c370 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f54c0009870 tx=0x7f54c001c540 comp rx=0 tx=0).stop 2023-10-25T00:05:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 -- 172.21.15.107:0/2382622462 shutdown_connections 2023-10-25T00:05:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 --2- 172.21.15.107:0/2382622462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c4155850 0x7f54c40bcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 --2- 172.21.15.107:0/2382622462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40c1c60 0x7f54c406c370 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 -- 172.21.15.107:0/2382622462 >> 172.21.15.107:0/2382622462 conn(0x7f54c40c2d50 msgr2=0x7f54c40c3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 -- 172.21.15.107:0/2382622462 shutdown_connections 2023-10-25T00:05:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 -- 172.21.15.107:0/2382622462 wait complete. 2023-10-25T00:05:08.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 Processor -- start 2023-10-25T00:05:08.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 -- start start 2023-10-25T00:05:08.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40c1c60 0x7f54c41480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:08.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c40c2040 0x7f54c40bfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:08.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c4155850 0x7f54c40c0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:08.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c40bd8e0 con 0x7f54c40c1c60 2023-10-25T00:05:08.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c40be260 con 0x7f54c40c2040 2023-10-25T00:05:08.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54cbb7e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c40bf550 con 0x7f54c4155850 2023-10-25T00:05:08.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54c991a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40c1c60 0x7f54c41480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:08.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54c991a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40c1c60 0x7f54c41480b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46736/0 (socket says 172.21.15.107:46736) 2023-10-25T00:05:08.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.725+0000 7f54c991a640 1 -- 172.21.15.107:0/4249758812 learned_addr learned my addr 172.21.15.107:0/4249758812 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:08.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54c9119640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c40c2040 0x7f54c40bfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:08.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54ca11b640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c4155850 0x7f54c40c0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:08.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54c991a640 1 -- 172.21.15.107:0/4249758812 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c4155850 msgr2=0x7f54c40c0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:08.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54c991a640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c4155850 0x7f54c40c0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:08.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54c991a640 1 -- 172.21.15.107:0/4249758812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c40c2040 msgr2=0x7f54c40bfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:08.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54c991a640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c40c2040 0x7f54c40bfbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:08.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54c991a640 1 -- 172.21.15.107:0/4249758812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c00094d0 con 0x7f54c40c1c60 2023-10-25T00:05:08.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54c991a640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40c1c60 0x7f54c41480b0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f54c00ad040 tx=0x7f54c001ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:08.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54ba7fc640 1 -- 172.21.15.107:0/4249758812 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00b4800 con 0x7f54c40c1c60 2023-10-25T00:05:08.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54ba7fc640 1 -- 172.21.15.107:0/4249758812 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c00b4980 con 0x7f54c40c1c60 2023-10-25T00:05:08.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54cbb7e640 1 -- 172.21.15.107:0/4249758812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c41edb70 con 0x7f54c40c1c60 2023-10-25T00:05:08.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54ba7fc640 1 -- 172.21.15.107:0/4249758812 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00b4d30 con 0x7f54c40c1c60 2023-10-25T00:05:08.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54cbb7e640 1 -- 172.21.15.107:0/4249758812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c414ae90 con 0x7f54c40c1c60 2023-10-25T00:05:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54cbb7e640 1 -- 172.21.15.107:0/4249758812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c40c2450 con 0x7f54c40c1c60 2023-10-25T00:05:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54ba7fc640 1 -- 172.21.15.107:0/4249758812 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f54c00b3380 con 0x7f54c40c1c60 2023-10-25T00:05:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54ba7fc640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54a8065e60 0x7f54a8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54ba7fc640 1 -- 172.21.15.107:0/4249758812 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f54c011e960 con 0x7f54c40c1c60 2023-10-25T00:05:08.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54c9119640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54a8065e60 0x7f54a8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.729+0000 7f54c9119640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54a8065e60 0x7f54a8068320 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f54c40ab300 tx=0x7f54b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.733+0000 7f54ba7fc640 1 -- 172.21.15.107:0/4249758812 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f54c40c2450 con 0x7f54c40c1c60 2023-10-25T00:05:08.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:08.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.913+0000 7f54cbb7e640 1 -- 172.21.15.107:0/4249758812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54c40c2450 con 0x7f54c40c1c60 2023-10-25T00:05:08.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54ba7fc640 1 -- 172.21.15.107:0/4249758812 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f54c40c2450 con 0x7f54c40c1c60 2023-10-25T00:05:08.918 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:08.919 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:05:08.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 -- 172.21.15.107:0/4249758812 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54a8065e60 msgr2=0x7f54a8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:08.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54a8065e60 0x7f54a8068320 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f54c40ab300 tx=0x7f54b4009040 comp rx=0 tx=0).stop 2023-10-25T00:05:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 -- 172.21.15.107:0/4249758812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40c1c60 msgr2=0x7f54c41480b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40c1c60 0x7f54c41480b0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f54c00ad040 tx=0x7f54c001ad80 comp rx=0 tx=0).stop 2023-10-25T00:05:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 -- 172.21.15.107:0/4249758812 shutdown_connections 2023-10-25T00:05:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f54a8065e60 0x7f54a8068320 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54c4155850 0x7f54c40c0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54c40c2040 0x7f54c40bfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 --2- 172.21.15.107:0/4249758812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54c40c1c60 0x7f54c41480b0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 -- 172.21.15.107:0/4249758812 >> 172.21.15.107:0/4249758812 conn(0x7f54c40c2d50 msgr2=0x7f54c40c3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 -- 172.21.15.107:0/4249758812 shutdown_connections 2023-10-25T00:05:08.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:08.917+0000 7f54cbb7e640 1 -- 172.21.15.107:0/4249758812 wait complete. 2023-10-25T00:05:08.936 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:05:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.013+0000 7f17ac456640 1 Processor -- start 2023-10-25T00:05:09.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.013+0000 7f17ac456640 1 -- start start 2023-10-25T00:05:09.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17ac456640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a414fd40 0x7f17a40ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17ac456640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f17a40a9820 0x7f17a40aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17ac456640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a406f730 con 0x7f17a406ee40 2023-10-25T00:05:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17ac456640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a40aa750 con 0x7f17a40a9820 2023-10-25T00:05:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17ac456640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a4156a40 con 0x7f17a414fd40 2023-10-25T00:05:09.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17aa9f3640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f17a406ee40 0x7f17a40c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54026/0 (socket says 172.21.15.107:54026) 2023-10-25T00:05:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17aa9f3640 1 -- 172.21.15.107:0/765725004 learned_addr learned my addr 172.21.15.107:0/765725004 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a99f1640 1 --2- 172.21.15.107:0/765725004 >> v2:172.21.15.110:3300/0 conn(0x7f17a40a9820 0x7f17a40aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17aa1f2640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a414fd40 0x7f17a40ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783883975 0 0) 0x7f17a406f730 con 0x7f17a406ee40 2023-10-25T00:05:09.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f178c003530 con 0x7f17a406ee40 2023-10-25T00:05:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1963734795 0 0) 0x7f178c003530 con 0x7f17a406ee40 2023-10-25T00:05:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a414fd40 msgr2=0x7f17a40ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a414fd40 0x7f17a40ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:09.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 >> v2:172.21.15.110:3300/0 conn(0x7f17a40a9820 msgr2=0x7f17a40aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 --2- 172.21.15.107:0/765725004 >> v2:172.21.15.110:3300/0 conn(0x7f17a40a9820 0x7f17a40aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a4156710 con 0x7f17a406ee40 2023-10-25T00:05:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f179c001a00 con 0x7f17a406ee40 2023-10-25T00:05:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 >> v1:172.21.15.107:6789/0 conn(0x7f17a406ee40 legacy=0x7f17a40c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:09.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f178c003980 0x7f178c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 0x7f178c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a414fd40 0x7f178c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a40aa750 con 0x7f178c0047d0 2023-10-25T00:05:09.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a4156a40 con 0x7f178c003980 2023-10-25T00:05:09.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178c00ba10 con 0x7f17a414fd40 2023-10-25T00:05:09.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a99f1640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 0x7f178c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:09.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17aa1f2640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f178c003980 0x7f178c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:09.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17aa9f3640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a414fd40 0x7f178c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:09.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a99f1640 1 -- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a414fd40 msgr2=0x7f178c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a99f1640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a414fd40 0x7f178c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a99f1640 1 -- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f178c003980 msgr2=0x7f178c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:09.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a99f1640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f178c003980 0x7f178c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17a99f1640 1 -- 172.21.15.107:0/765725004 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c003530 con 0x7f178c0047d0 2023-10-25T00:05:09.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.017+0000 7f17aa1f2640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f178c003980 0x7f178c003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:05:09.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17a99f1640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 0x7f178c004c50 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f1798009ae0 tx=0x7f179801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a175ccb7f3e5855 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:09.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17980a55d0 con 0x7f178c0047d0 2023-10-25T00:05:09.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17980a5730 con 0x7f178c0047d0 2023-10-25T00:05:09.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17a91f0640 1 -- 172.21.15.107:0/765725004 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17980a5ae0 con 0x7f178c0047d0 2023-10-25T00:05:09.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 -- 172.21.15.107:0/765725004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 msgr2=0x7f178c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:09.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 --2- 172.21.15.107:0/765725004 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 0x7f178c004c50 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f1798009ae0 tx=0x7f179801ca60 comp rx=0 tx=0).stop 2023-10-25T00:05:09.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17aa9f3640 1 -- 172.21.15.107:0/765725004 reap_dead start 2023-10-25T00:05:09.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 -- 172.21.15.107:0/765725004 shutdown_connections 2023-10-25T00:05:09.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 -- 172.21.15.107:0/765725004 >> 172.21.15.107:0/765725004 conn(0x7f17a40c2950 msgr2=0x7f17a40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:09.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 -- 172.21.15.107:0/765725004 shutdown_connections 2023-10-25T00:05:09.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 -- 172.21.15.107:0/765725004 wait complete. 2023-10-25T00:05:09.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 Processor -- start 2023-10-25T00:05:09.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 -- start start 2023-10-25T00:05:09.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 0x7f17a40c4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:09.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a41eeac0 0x7f17a41eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:09.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17a41eb7b0 0x7f17a414c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:09.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a41ef030 con 0x7f178c0047d0 2023-10-25T00:05:09.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a41eab40 con 0x7f17a41eb7b0 2023-10-25T00:05:09.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17ac456640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a40bbe50 con 0x7f17a41eeac0 2023-10-25T00:05:09.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.021+0000 7f17aa1f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 0x7f17a40c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:09.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17aa1f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 0x7f17a40c4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46756/0 (socket says 172.21.15.107:46756) 2023-10-25T00:05:09.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17aa1f2640 1 -- 172.21.15.107:0/1726920204 learned_addr learned my addr 172.21.15.107:0/1726920204 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:09.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17aa9f3640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17a41eb7b0 0x7f17a414c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:09.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17a99f1640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a41eeac0 0x7f17a41eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:09.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17aa1f2640 1 -- 172.21.15.107:0/1726920204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a41eeac0 msgr2=0x7f17a41eb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:09.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17aa1f2640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a41eeac0 0x7f17a41eb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:09.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17aa1f2640 1 -- 172.21.15.107:0/1726920204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17a41eb7b0 msgr2=0x7f17a414c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:09.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17aa1f2640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17a41eb7b0 0x7f17a414c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:09.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17aa1f2640 1 -- 172.21.15.107:0/1726920204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a41eade0 con 0x7f178c0047d0 2023-10-25T00:05:09.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17a99f1640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a41eeac0 0x7f17a41eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17aa1f2640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 0x7f17a40c4780 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1794009fd0 tx=0x7f1794011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17a2ffd640 1 -- 172.21.15.107:0/1726920204 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1794004220 con 0x7f178c0047d0 2023-10-25T00:05:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17a2ffd640 1 -- 172.21.15.107:0/1726920204 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1794004380 con 0x7f178c0047d0 2023-10-25T00:05:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17a2ffd640 1 -- 172.21.15.107:0/1726920204 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179400f9e0 con 0x7f178c0047d0 2023-10-25T00:05:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17ac456640 1 -- 172.21.15.107:0/1726920204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a40bb890 con 0x7f178c0047d0 2023-10-25T00:05:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17ac456640 1 -- 172.21.15.107:0/1726920204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a40c4cc0 con 0x7f178c0047d0 2023-10-25T00:05:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17ac456640 1 -- 172.21.15.107:0/1726920204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a40c24a0 con 0x7f178c0047d0 2023-10-25T00:05:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17a2ffd640 1 -- 172.21.15.107:0/1726920204 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f179400fb40 con 0x7f178c0047d0 2023-10-25T00:05:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17a2ffd640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1790065e90 0x7f1790068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:09.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17a2ffd640 1 -- 172.21.15.107:0/1726920204 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f1794084690 con 0x7f178c0047d0 2023-10-25T00:05:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17a99f1640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1790065e90 0x7f1790068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.025+0000 7f17a99f1640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1790065e90 0x7f1790068350 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f17980af000 tx=0x7f17980046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.033+0000 7f17a2ffd640 1 -- 172.21.15.107:0/1726920204 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1794051330 con 0x7f178c0047d0 2023-10-25T00:05:09.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17ac456640 1 -- 172.21.15.107:0/1726920204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17a40746e0 con 0x7f178c0047d0 2023-10-25T00:05:09.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17a2ffd640 1 -- 172.21.15.107:0/1726920204 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f1794022090 con 0x7f178c0047d0 2023-10-25T00:05:09.208 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:09.208 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:05:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17ac456640 1 -- 172.21.15.107:0/1726920204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1790065e90 msgr2=0x7f1790068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17ac456640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1790065e90 0x7f1790068350 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f17980af000 tx=0x7f17980046d0 comp rx=0 tx=0).stop 2023-10-25T00:05:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17ac456640 1 -- 172.21.15.107:0/1726920204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 msgr2=0x7f17a40c4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17ac456640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 0x7f17a40c4780 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1794009fd0 tx=0x7f1794011b30 comp rx=0 tx=0).stop 2023-10-25T00:05:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17ac456640 1 -- 172.21.15.107:0/1726920204 shutdown_connections 2023-10-25T00:05:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17ac456640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17a41eb7b0 0x7f17a414c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17ac456640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1790065e90 0x7f1790068350 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17ac456640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f17a41eeac0 0x7f17a41eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17ac456640 1 --2- 172.21.15.107:0/1726920204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f178c0047d0 0x7f17a40c4780 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.205+0000 7f17ac456640 1 -- 172.21.15.107:0/1726920204 >> 172.21.15.107:0/1726920204 conn(0x7f17a40c2950 msgr2=0x7f17a4151790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.209+0000 7f17ac456640 1 -- 172.21.15.107:0/1726920204 shutdown_connections 2023-10-25T00:05:09.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:09.209+0000 7f17ac456640 1 -- 172.21.15.107:0/1726920204 wait complete. 2023-10-25T00:05:09.225 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:05:10.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:11.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbaf8a8640 1 Processor -- start 2023-10-25T00:05:11.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbaf8a8640 1 -- start start 2023-10-25T00:05:11.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbaf8a8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba80a9820 0x7fdba80ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbaf8a8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdba806ee40 0x7fdba80aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbaf8a8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba806f730 con 0x7fdba814fd40 2023-10-25T00:05:11.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbaf8a8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba80aa750 con 0x7fdba806ee40 2023-10-25T00:05:11.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbaf8a8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba8156a40 con 0x7fdba80a9820 2023-10-25T00:05:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbade45640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdba806ee40 0x7fdba80aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbade45640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdba806ee40 0x7fdba80aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46768/0 (socket says 172.21.15.107:46768) 2023-10-25T00:05:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbade45640 1 -- 172.21.15.107:0/3435918814 learned_addr learned my addr 172.21.15.107:0/3435918814 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbace43640 1 --2- 172.21.15.107:0/3435918814 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba80a9820 0x7fdba80ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbade45640 1 -- 172.21.15.107:0/3435918814 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba80a9820 msgr2=0x7fdba80ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:11.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbade45640 1 --2- 172.21.15.107:0/3435918814 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba80a9820 0x7fdba80ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbade45640 1 -- 172.21.15.107:0/3435918814 >> v1:172.21.15.110:6789/0 conn(0x7fdba814fd40 legacy=0x7fdba80c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:11.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbade45640 1 -- 172.21.15.107:0/3435918814 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba8156710 con 0x7fdba806ee40 2023-10-25T00:05:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.069+0000 7fdbade45640 1 --2- 172.21.15.107:0/3435918814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba806ee40 0x7fdba80aa210 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fdba0009e50 tx=0x7fdba001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247c17bf0c0642ef server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdb9ffff640 1 -- 172.21.15.107:0/3435918814 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba00a56a0 con 0x7fdba806ee40 2023-10-25T00:05:11.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdb9ffff640 1 -- 172.21.15.107:0/3435918814 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba00a5800 con 0x7fdba806ee40 2023-10-25T00:05:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdb9ffff640 1 -- 172.21.15.107:0/3435918814 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba00a5b90 con 0x7fdba806ee40 2023-10-25T00:05:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/3435918814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba806ee40 msgr2=0x7fdba80aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 --2- 172.21.15.107:0/3435918814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba806ee40 0x7fdba80aa210 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fdba0009e50 tx=0x7fdba001c880 comp rx=0 tx=0).stop 2023-10-25T00:05:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/3435918814 shutdown_connections 2023-10-25T00:05:11.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 --2- 172.21.15.107:0/3435918814 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba806ee40 0x7fdba80aa210 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 --2- 172.21.15.107:0/3435918814 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba80a9820 0x7fdba80ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/3435918814 >> 172.21.15.107:0/3435918814 conn(0x7fdba80c2950 msgr2=0x7fdba80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/3435918814 shutdown_connections 2023-10-25T00:05:11.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/3435918814 wait complete. 2023-10-25T00:05:11.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 Processor -- start 2023-10-25T00:05:11.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 -- start start 2023-10-25T00:05:11.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba806ee40 0x7fdba8145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdba814fd40 0x7fdba80c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba8147f90 0x7fdba814e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba814e010 con 0x7fdba806ee40 2023-10-25T00:05:11.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba814f590 con 0x7fdba814fd40 2023-10-25T00:05:11.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbaf8a8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba80c4e20 con 0x7fdba8147f90 2023-10-25T00:05:11.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbad644640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba806ee40 0x7fdba8145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbad644640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba806ee40 0x7fdba8145330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46778/0 (socket says 172.21.15.107:46778) 2023-10-25T00:05:11.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbad644640 1 -- 172.21.15.107:0/2392447217 learned_addr learned my addr 172.21.15.107:0/2392447217 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:11.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbace43640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdba814fd40 0x7fdba80c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbade45640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba8147f90 0x7fdba814e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbad644640 1 -- 172.21.15.107:0/2392447217 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba8147f90 msgr2=0x7fdba814e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbad644640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba8147f90 0x7fdba814e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbad644640 1 -- 172.21.15.107:0/2392447217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdba814fd40 msgr2=0x7fdba80c3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbad644640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdba814fd40 0x7fdba80c3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbad644640 1 -- 172.21.15.107:0/2392447217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba0009b30 con 0x7fdba806ee40 2023-10-25T00:05:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbade45640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba8147f90 0x7fdba814e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdbad644640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba806ee40 0x7fdba8145330 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fdba400b440 tx=0x7fdba400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.073+0000 7fdb9dffb640 1 -- 172.21.15.107:0/2392447217 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4011890 con 0x7fdba806ee40 2023-10-25T00:05:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.077+0000 7fdb9dffb640 1 -- 172.21.15.107:0/2392447217 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba40119f0 con 0x7fdba806ee40 2023-10-25T00:05:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.077+0000 7fdb9dffb640 1 -- 172.21.15.107:0/2392447217 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4011da0 con 0x7fdba806ee40 2023-10-25T00:05:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.077+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/2392447217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba80c5500 con 0x7fdba806ee40 2023-10-25T00:05:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.077+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/2392447217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba814efd0 con 0x7fdba806ee40 2023-10-25T00:05:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.077+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/2392447217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba80c24a0 con 0x7fdba806ee40 2023-10-25T00:05:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.077+0000 7fdb9dffb640 1 -- 172.21.15.107:0/2392447217 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdba401a400 con 0x7fdba806ee40 2023-10-25T00:05:11.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.077+0000 7fdb9dffb640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb8c065e60 0x7fdb8c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.077+0000 7fdb9dffb640 1 -- 172.21.15.107:0/2392447217 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7fdba4084280 con 0x7fdba806ee40 2023-10-25T00:05:11.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.077+0000 7fdbace43640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb8c065e60 0x7fdb8c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.077+0000 7fdbace43640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb8c065e60 0x7fdb8c068320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdba814f760 tx=0x7fdb98008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:11.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.081+0000 7fdb9dffb640 1 -- 172.21.15.107:0/2392447217 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdba4050f20 con 0x7fdba806ee40 2023-10-25T00:05:11.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:11.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/2392447217 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdba8070840 con 0x7fdba806ee40 2023-10-25T00:05:11.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdb9dffb640 1 -- 172.21.15.107:0/2392447217 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7fdba4015090 con 0x7fdba806ee40 2023-10-25T00:05:11.280 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:11.280 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:11.280 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:05:11.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/2392447217 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb8c065e60 msgr2=0x7fdb8c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb8c065e60 0x7fdb8c068320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdba814f760 tx=0x7fdb98008040 comp rx=0 tx=0).stop 2023-10-25T00:05:11.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/2392447217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba806ee40 msgr2=0x7fdba8145330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba806ee40 0x7fdba8145330 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fdba400b440 tx=0x7fdba400b900 comp rx=0 tx=0).stop 2023-10-25T00:05:11.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/2392447217 shutdown_connections 2023-10-25T00:05:11.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdb8c065e60 0x7fdb8c068320 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdba8147f90 0x7fdba814e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdba814fd40 0x7fdba80c3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 --2- 172.21.15.107:0/2392447217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba806ee40 0x7fdba8145330 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/2392447217 >> 172.21.15.107:0/2392447217 conn(0x7fdba80c2950 msgr2=0x7fdba80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:11.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/2392447217 shutdown_connections 2023-10-25T00:05:11.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.277+0000 7fdbaf8a8640 1 -- 172.21.15.107:0/2392447217 wait complete. 2023-10-25T00:05:11.300 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:05:11.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.349+0000 7f17a45fc640 1 Processor -- start 2023-10-25T00:05:11.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.349+0000 7f17a45fc640 1 -- start start 2023-10-25T00:05:11.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a45fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c152040 0x7f179c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a45fc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c155680 con 0x7f179c0c8100 2023-10-25T00:05:11.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a45fc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c154a70 con 0x7f179c06a4b0 2023-10-25T00:05:11.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a45fc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179c154d50 con 0x7f179c152040 2023-10-25T00:05:11.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1b97640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f179c0c8100 0x7f179c0abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54042/0 (socket says 172.21.15.107:54042) 2023-10-25T00:05:11.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1b97640 1 -- 172.21.15.107:0/4246196576 learned_addr learned my addr 172.21.15.107:0/4246196576 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:11.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a2398640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c152040 0x7f179c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681191886 0 0) 0x7f179c155680 con 0x7f179c0c8100 2023-10-25T00:05:11.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1788003530 con 0x7f179c0c8100 2023-10-25T00:05:11.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2347969090 0 0) 0x7f1788003530 con 0x7f179c0c8100 2023-10-25T00:05:11.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c152040 msgr2=0x7f179c0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:11.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c152040 0x7f179c0c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 >> v1:172.21.15.110:6789/0 conn(0x7f179c06a4b0 legacy=0x7f179c0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c155030 con 0x7f179c0c8100 2023-10-25T00:05:11.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f1798001680 con 0x7f179c0c8100 2023-10-25T00:05:11.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 >> v1:172.21.15.107:6789/0 conn(0x7f179c0c8100 legacy=0x7f179c0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17880039b0 0x7f1788003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c152040 0x7f17880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1788004d10 0x7f1788005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c154d50 con 0x7f17880039b0 2023-10-25T00:05:11.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178800bb50 con 0x7f1788004d10 2023-10-25T00:05:11.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178800bd10 con 0x7f179c152040 2023-10-25T00:05:11.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a2398640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17880039b0 0x7f1788003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a2b99640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1788004d10 0x7f1788005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1b97640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c152040 0x7f17880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a2398640 1 -- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c152040 msgr2=0x7f17880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:11.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a2398640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c152040 0x7f17880047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a2398640 1 -- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1788004d10 msgr2=0x7f1788005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a2398640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1788004d10 0x7f1788005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a2398640 1 -- 172.21.15.107:0/4246196576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1788003530 con 0x7f17880039b0 2023-10-25T00:05:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a2b99640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1788004d10 0x7f1788005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a2398640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17880039b0 0x7f1788003dd0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f178c009ae0 tx=0x7f178c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63096c94734acc9c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:11.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c0a5530 con 0x7f17880039b0 2023-10-25T00:05:11.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c0a5690 con 0x7f17880039b0 2023-10-25T00:05:11.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.353+0000 7f17a1396640 1 -- 172.21.15.107:0/4246196576 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c0a5840 con 0x7f17880039b0 2023-10-25T00:05:11.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 -- 172.21.15.107:0/4246196576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17880039b0 msgr2=0x7f1788003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 --2- 172.21.15.107:0/4246196576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17880039b0 0x7f1788003dd0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f178c009ae0 tx=0x7f178c01ca50 comp rx=0 tx=0).stop 2023-10-25T00:05:11.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2b99640 1 -- 172.21.15.107:0/4246196576 reap_dead start 2023-10-25T00:05:11.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 -- 172.21.15.107:0/4246196576 shutdown_connections 2023-10-25T00:05:11.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 -- 172.21.15.107:0/4246196576 >> 172.21.15.107:0/4246196576 conn(0x7f179c0ca200 msgr2=0x7f179c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:11.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 -- 172.21.15.107:0/4246196576 shutdown_connections 2023-10-25T00:05:11.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 -- 172.21.15.107:0/4246196576 wait complete. 2023-10-25T00:05:11.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 Processor -- start 2023-10-25T00:05:11.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 -- start start 2023-10-25T00:05:11.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17880039b0 0x7f179c0aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c0bd3c0 0x7f179c1585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f179c158b10 0x7f179c0c14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c1f09b0 con 0x7f179c158b10 2023-10-25T00:05:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c156210 con 0x7f17880039b0 2023-10-25T00:05:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179c156ee0 con 0x7f179c0bd3c0 2023-10-25T00:05:11.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2b99640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f179c158b10 0x7f179c0c14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2b99640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f179c158b10 0x7f179c0c14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46788/0 (socket says 172.21.15.107:46788) 2023-10-25T00:05:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2b99640 1 -- 172.21.15.107:0/4009533043 learned_addr learned my addr 172.21.15.107:0/4009533043 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2398640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17880039b0 0x7f179c0aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a1b97640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c0bd3c0 0x7f179c1585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2b99640 1 -- 172.21.15.107:0/4009533043 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c0bd3c0 msgr2=0x7f179c1585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2b99640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c0bd3c0 0x7f179c1585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2b99640 1 -- 172.21.15.107:0/4009533043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17880039b0 msgr2=0x7f179c0aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2b99640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17880039b0 0x7f179c0aa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2b99640 1 -- 172.21.15.107:0/4009533043 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c1f0dd0 con 0x7f179c158b10 2023-10-25T00:05:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a1b97640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c0bd3c0 0x7f179c1585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2398640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17880039b0 0x7f179c0aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a2b99640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f179c158b10 0x7f179c0c14e0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f179c069810 tx=0x7f179400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f1792ffd640 1 -- 172.21.15.107:0/4009533043 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1794003b80 con 0x7f179c158b10 2023-10-25T00:05:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f1792ffd640 1 -- 172.21.15.107:0/4009533043 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1794003ce0 con 0x7f179c158b10 2023-10-25T00:05:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 -- 172.21.15.107:0/4009533043 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f179c1f0010 con 0x7f179c158b10 2023-10-25T00:05:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f1792ffd640 1 -- 172.21.15.107:0/4009533043 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17940052b0 con 0x7f179c158b10 2023-10-25T00:05:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.357+0000 7f17a45fc640 1 -- 172.21.15.107:0/4009533043 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f179c1f05d0 con 0x7f179c158b10 2023-10-25T00:05:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.361+0000 7f17a45fc640 1 -- 172.21.15.107:0/4009533043 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f179c1f0af0 con 0x7f179c158b10 2023-10-25T00:05:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.361+0000 7f1792ffd640 1 -- 172.21.15.107:0/4009533043 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1794005410 con 0x7f179c158b10 2023-10-25T00:05:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.361+0000 7f1792ffd640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1780065e90 0x7f1780068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.361+0000 7f1792ffd640 1 -- 172.21.15.107:0/4009533043 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f1794083a50 con 0x7f179c158b10 2023-10-25T00:05:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.361+0000 7f17a2398640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1780065e90 0x7f1780068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.361+0000 7f17a2398640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1780065e90 0x7f1780068350 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f178c0af000 tx=0x7f178c0046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.365+0000 7f1792ffd640 1 -- 172.21.15.107:0/4009533043 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1794050770 con 0x7f179c158b10 2023-10-25T00:05:11.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.557+0000 7f17a45fc640 1 -- 172.21.15.107:0/4009533043 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f179c1f0af0 con 0x7f179c158b10 2023-10-25T00:05:11.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.557+0000 7f1792ffd640 1 -- 172.21.15.107:0/4009533043 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f179c1f0af0 con 0x7f179c158b10 2023-10-25T00:05:11.561 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:11.561 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59},{"gid":6022,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:04:37.754802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6906},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6906":{"gid":6906,"name":"c","rank":0,"incarnation":101,"state":"up:active","state_seq":4,"addr":"172.21.15.107:6835/1246928633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1246928633},{"type":"v1","addr":"172.21.15.107:6835","nonce":1246928633}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:11.562 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 105 2023-10-25T00:05:11.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.557+0000 7f17a45fc640 1 -- 172.21.15.107:0/4009533043 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1780065e90 msgr2=0x7f1780068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.557+0000 7f17a45fc640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1780065e90 0x7f1780068350 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f178c0af000 tx=0x7f178c0046d0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.557+0000 7f17a45fc640 1 -- 172.21.15.107:0/4009533043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f179c158b10 msgr2=0x7f179c0c14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:11.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.557+0000 7f17a45fc640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f179c158b10 0x7f179c0c14e0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f179c069810 tx=0x7f179400dc00 comp rx=0 tx=0).stop 2023-10-25T00:05:11.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.561+0000 7f17a45fc640 1 -- 172.21.15.107:0/4009533043 shutdown_connections 2023-10-25T00:05:11.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.561+0000 7f17a45fc640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f179c158b10 0x7f179c0c14e0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.561+0000 7f17a45fc640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1780065e90 0x7f1780068350 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.561+0000 7f17a45fc640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f179c0bd3c0 0x7f179c1585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.561+0000 7f17a45fc640 1 --2- 172.21.15.107:0/4009533043 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17880039b0 0x7f179c0aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.561+0000 7f17a45fc640 1 -- 172.21.15.107:0/4009533043 >> 172.21.15.107:0/4009533043 conn(0x7f179c0ca200 msgr2=0x7f179c153f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.561+0000 7f17a45fc640 1 -- 172.21.15.107:0/4009533043 shutdown_connections 2023-10-25T00:05:11.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:11.561+0000 7f17a45fc640 1 -- 172.21.15.107:0/4009533043 wait complete. 2023-10-25T00:05:11.579 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:05:13.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:13.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f08aa0f2640 1 Processor -- start 2023-10-25T00:05:13.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f08aa0f2640 1 -- start start 2023-10-25T00:05:13.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f08aa0f2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f08a40c8100 0x7f08a40abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f08aa0f2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08a406a4b0 0x7f08a40ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f08aa0f2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a4155680 con 0x7f08a4152040 2023-10-25T00:05:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f08aa0f2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a4154a70 con 0x7f08a40c8100 2023-10-25T00:05:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f08aa0f2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a4154d50 con 0x7f08a406a4b0 2023-10-25T00:05:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f089affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f08a40c8100 0x7f08a40abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f089affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f08a40c8100 0x7f08a40abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49712/0 (socket says 172.21.15.107:49712) 2023-10-25T00:05:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f089affd640 1 -- 172.21.15.107:0/2548771930 learned_addr learned my addr 172.21.15.107:0/2548771930 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f08a3fff640 1 --2- 172.21.15.107:0/2548771930 >> v2:172.21.15.110:3300/0 conn(0x7f08a406a4b0 0x7f08a40ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f089affd640 1 -- 172.21.15.107:0/2548771930 >> v2:172.21.15.110:3300/0 conn(0x7f08a406a4b0 msgr2=0x7f08a40ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f089affd640 1 --2- 172.21.15.107:0/2548771930 >> v2:172.21.15.110:3300/0 conn(0x7f08a406a4b0 0x7f08a40ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f089affd640 1 -- 172.21.15.107:0/2548771930 >> v1:172.21.15.110:6789/0 conn(0x7f08a4152040 legacy=0x7f08a40c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f089affd640 1 -- 172.21.15.107:0/2548771930 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a4155030 con 0x7f08a40c8100 2023-10-25T00:05:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.429+0000 7f08a3fff640 1 --2- 172.21.15.107:0/2548771930 >> v2:172.21.15.110:3300/0 conn(0x7f08a406a4b0 0x7f08a40ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f089affd640 1 --2- 172.21.15.107:0/2548771930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a40c8100 0x7f08a40abd00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f08a406b590 tx=0x7f088c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a94af6fecc125be8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a2ffd640 1 -- 172.21.15.107:0/2548771930 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c0ab600 con 0x7f08a40c8100 2023-10-25T00:05:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a2ffd640 1 -- 172.21.15.107:0/2548771930 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c0a7050 con 0x7f08a40c8100 2023-10-25T00:05:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a2ffd640 1 -- 172.21.15.107:0/2548771930 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c0ab980 con 0x7f08a40c8100 2023-10-25T00:05:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2548771930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a40c8100 msgr2=0x7f08a40abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 --2- 172.21.15.107:0/2548771930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a40c8100 0x7f08a40abd00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f08a406b590 tx=0x7f088c01c430 comp rx=0 tx=0).stop 2023-10-25T00:05:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2548771930 shutdown_connections 2023-10-25T00:05:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 --2- 172.21.15.107:0/2548771930 >> v2:172.21.15.110:3300/0 conn(0x7f08a406a4b0 0x7f08a40ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 --2- 172.21.15.107:0/2548771930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a40c8100 0x7f08a40abd00 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2548771930 >> 172.21.15.107:0/2548771930 conn(0x7f08a40ca200 msgr2=0x7f08a40ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2548771930 shutdown_connections 2023-10-25T00:05:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2548771930 wait complete. 2023-10-25T00:05:13.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 Processor -- start 2023-10-25T00:05:13.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 -- start start 2023-10-25T00:05:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a406a4b0 0x7f08a414ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a40c8100 0x7f08a41edb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a4152040 0x7f08a40a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a4151750 con 0x7f08a406a4b0 2023-10-25T00:05:13.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a41ed050 con 0x7f08a4152040 2023-10-25T00:05:13.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08aa0f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a414f540 con 0x7f08a40c8100 2023-10-25T00:05:13.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a406a4b0 0x7f08a414ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a406a4b0 0x7f08a414ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49724/0 (socket says 172.21.15.107:49724) 2023-10-25T00:05:13.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a37fe640 1 -- 172.21.15.107:0/2747966303 learned_addr learned my addr 172.21.15.107:0/2747966303 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:13.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a3fff640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a4152040 0x7f08a40a98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a37fe640 1 -- 172.21.15.107:0/2747966303 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a40c8100 msgr2=0x7f08a41edb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f089affd640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a40c8100 0x7f08a41edb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a37fe640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a40c8100 0x7f08a41edb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a37fe640 1 -- 172.21.15.107:0/2747966303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a4152040 msgr2=0x7f08a40a98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a37fe640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a4152040 0x7f08a40a98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a37fe640 1 -- 172.21.15.107:0/2747966303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c0094d0 con 0x7f08a406a4b0 2023-10-25T00:05:13.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a3fff640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a4152040 0x7f08a40a98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:13.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.433+0000 7f08a37fe640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a406a4b0 0x7f08a414ef60 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f089400c490 tx=0x7f0894001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:13.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.437+0000 7f08a0ff9640 1 -- 172.21.15.107:0/2747966303 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894003600 con 0x7f08a406a4b0 2023-10-25T00:05:13.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.437+0000 7f08a0ff9640 1 -- 172.21.15.107:0/2747966303 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0894003780 con 0x7f08a406a4b0 2023-10-25T00:05:13.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.437+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2747966303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a414f820 con 0x7f08a406a4b0 2023-10-25T00:05:13.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.437+0000 7f08a0ff9640 1 -- 172.21.15.107:0/2747966303 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0894003b70 con 0x7f08a406a4b0 2023-10-25T00:05:13.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.437+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2747966303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a41ed5f0 con 0x7f08a406a4b0 2023-10-25T00:05:13.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.437+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2747966303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a40746e0 con 0x7f08a406a4b0 2023-10-25T00:05:13.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.437+0000 7f08a0ff9640 1 -- 172.21.15.107:0/2747966303 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0894003cd0 con 0x7f08a406a4b0 2023-10-25T00:05:13.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.437+0000 7f08a0ff9640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0888065e60 0x7f0888068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.437+0000 7f089affd640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0888065e60 0x7f0888068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.437+0000 7f08a0ff9640 1 -- 172.21.15.107:0/2747966303 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 8595+0+0 (secure 0 0 0) 0x7f0894083850 con 0x7f08a406a4b0 2023-10-25T00:05:13.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.437+0000 7f089affd640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0888065e60 0x7f0888068320 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f088c0ad000 tx=0x7f088c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:13.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.441+0000 7f08a0ff9640 1 -- 172.21.15.107:0/2747966303 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f08940504f0 con 0x7f08a406a4b0 2023-10-25T00:05:13.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:13.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.637+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2747966303 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08a40c4530 con 0x7f08a406a4b0 2023-10-25T00:05:13.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.637+0000 7f08a0ff9640 1 -- 172.21.15.107:0/2747966303 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5015 (secure 0 0 0) 0x7f0894017070 con 0x7f08a406a4b0 2023-10-25T00:05:13.642 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:13.642 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:13.610083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:13.643 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 106 2023-10-25T00:05:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2747966303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0888065e60 msgr2=0x7f0888068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0888065e60 0x7f0888068320 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f088c0ad000 tx=0x7f088c008040 comp rx=0 tx=0).stop 2023-10-25T00:05:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2747966303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a406a4b0 msgr2=0x7f08a414ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a406a4b0 0x7f08a414ef60 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f089400c490 tx=0x7f0894001e80 comp rx=0 tx=0).stop 2023-10-25T00:05:13.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2747966303 shutdown_connections 2023-10-25T00:05:13.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08a4152040 0x7f08a40a98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0888065e60 0x7f0888068320 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f08a40c8100 0x7f08a41edb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 --2- 172.21.15.107:0/2747966303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08a406a4b0 0x7f08a414ef60 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2747966303 >> 172.21.15.107:0/2747966303 conn(0x7f08a40ca200 msgr2=0x7f08a406a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:13.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2747966303 shutdown_connections 2023-10-25T00:05:13.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.641+0000 7f08aa0f2640 1 -- 172.21.15.107:0/2747966303 wait complete. 2023-10-25T00:05:13.667 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2023-10-25T00:05:13.667 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:05:13.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:13.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9ab8fd640 1 Processor -- start 2023-10-25T00:05:13.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9ab8fd640 1 -- start start 2023-10-25T00:05:13.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9ab8fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9a40adb80 0x7ff9a4158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9ab8fd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9a406e800 0x7ff9a406ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9ab8fd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a4152420 con 0x7ff9a4152040 2023-10-25T00:05:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9ab8fd640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a41556e0 con 0x7ff9a406e800 2023-10-25T00:05:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9ab8fd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a4154d40 con 0x7ff9a40adb80 2023-10-25T00:05:13.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9a9e9a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9a406e800 0x7ff9a406ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9a9e9a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff9a406e800 0x7ff9a406ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49734/0 (socket says 172.21.15.107:49734) 2023-10-25T00:05:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9a9e9a640 1 -- 172.21.15.107:0/2339978402 learned_addr learned my addr 172.21.15.107:0/2339978402 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:13.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9a8e98640 1 --2- 172.21.15.107:0/2339978402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9a40adb80 0x7ff9a4158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9a9e9a640 1 -- 172.21.15.107:0/2339978402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9a40adb80 msgr2=0x7ff9a4158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9a9e9a640 1 --2- 172.21.15.107:0/2339978402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9a40adb80 0x7ff9a4158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9a9e9a640 1 -- 172.21.15.107:0/2339978402 >> v1:172.21.15.110:6789/0 conn(0x7ff9a4152040 legacy=0x7ff9a4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:13.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9a9e9a640 1 -- 172.21.15.107:0/2339978402 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a4155020 con 0x7ff9a406e800 2023-10-25T00:05:13.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.701+0000 7ff9a8e98640 1 --2- 172.21.15.107:0/2339978402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9a40adb80 0x7ff9a4158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:13.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9a9e9a640 1 --2- 172.21.15.107:0/2339978402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9a406e800 0x7ff9a406ec90 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7ff99c009870 tx=0x7ff99c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd55b9ef18f01d22 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:13.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9a3fff640 1 -- 172.21.15.107:0/2339978402 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c0ab600 con 0x7ff9a406e800 2023-10-25T00:05:13.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9a3fff640 1 -- 172.21.15.107:0/2339978402 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c0a7050 con 0x7ff9a406e800 2023-10-25T00:05:13.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9a3fff640 1 -- 172.21.15.107:0/2339978402 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c0ab980 con 0x7ff9a406e800 2023-10-25T00:05:13.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/2339978402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9a406e800 msgr2=0x7ff9a406ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9ab8fd640 1 --2- 172.21.15.107:0/2339978402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9a406e800 0x7ff9a406ec90 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7ff99c009870 tx=0x7ff99c01c470 comp rx=0 tx=0).stop 2023-10-25T00:05:13.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/2339978402 shutdown_connections 2023-10-25T00:05:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9ab8fd640 1 --2- 172.21.15.107:0/2339978402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9a406e800 0x7ff9a406ec90 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9ab8fd640 1 --2- 172.21.15.107:0/2339978402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9a40adb80 0x7ff9a4158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/2339978402 >> 172.21.15.107:0/2339978402 conn(0x7ff9a4070f60 msgr2=0x7ff9a4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/2339978402 shutdown_connections 2023-10-25T00:05:13.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/2339978402 wait complete. 2023-10-25T00:05:13.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9ab8fd640 1 Processor -- start 2023-10-25T00:05:13.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.705+0000 7ff9ab8fd640 1 -- start start 2023-10-25T00:05:13.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9ab8fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9a406e800 0x7ff9a40c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9ab8fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9a40adb80 0x7ff9a4150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9ab8fd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9a4152040 0x7ff9a41ecff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9ab8fd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a4157cc0 con 0x7ff9a406e800 2023-10-25T00:05:13.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9ab8fd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a41467f0 con 0x7ff9a4152040 2023-10-25T00:05:13.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9ab8fd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a4156580 con 0x7ff9a40adb80 2023-10-25T00:05:13.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a9699640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9a406e800 0x7ff9a40c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a9699640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9a406e800 0x7ff9a40c7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49748/0 (socket says 172.21.15.107:49748) 2023-10-25T00:05:13.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a9699640 1 -- 172.21.15.107:0/3949216099 learned_addr learned my addr 172.21.15.107:0/3949216099 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:13.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a8e98640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9a40adb80 0x7ff9a4150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a9e9a640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9a4152040 0x7ff9a41ecff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a9699640 1 -- 172.21.15.107:0/3949216099 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9a40adb80 msgr2=0x7ff9a4150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a9699640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9a40adb80 0x7ff9a4150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a9699640 1 -- 172.21.15.107:0/3949216099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9a4152040 msgr2=0x7ff9a41ecff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a9699640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9a4152040 0x7ff9a41ecff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a9699640 1 -- 172.21.15.107:0/3949216099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99c0094d0 con 0x7ff9a406e800 2023-10-25T00:05:13.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a9e9a640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9a4152040 0x7ff9a41ecff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a9699640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9a406e800 0x7ff9a40c7720 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ff99400ba80 tx=0x7ff99400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a1ffb640 1 -- 172.21.15.107:0/3949216099 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99400cc00 con 0x7ff9a406e800 2023-10-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a1ffb640 1 -- 172.21.15.107:0/3949216099 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99400cd60 con 0x7ff9a406e800 2023-10-25T00:05:13.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/3949216099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a41ecb70 con 0x7ff9a406e800 2023-10-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a1ffb640 1 -- 172.21.15.107:0/3949216099 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994003bc0 con 0x7ff9a406e800 2023-10-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/3949216099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a41579e0 con 0x7ff9a406e800 2023-10-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/3949216099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a41ecd10 con 0x7ff9a406e800 2023-10-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a1ffb640 1 -- 172.21.15.107:0/3949216099 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff994014440 con 0x7ff9a406e800 2023-10-25T00:05:13.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a1ffb640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff990065e60 0x7ff990068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a1ffb640 1 -- 172.21.15.107:0/3949216099 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 8681+0+0 (secure 0 0 0) 0x7ff994083ed0 con 0x7ff9a406e800 2023-10-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.709+0000 7ff9a8e98640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff990065e60 0x7ff990068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.713+0000 7ff9a8e98640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff990065e60 0x7ff990068320 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4151960 tx=0x7ff998008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:13.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.713+0000 7ff9a1ffb640 1 -- 172.21.15.107:0/3949216099 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff9a41ecd10 con 0x7ff9a406e800 2023-10-25T00:05:13.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.785+0000 7ff4cc60e640 1 Processor -- start 2023-10-25T00:05:13.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.785+0000 7ff4cc60e640 1 -- start start 2023-10-25T00:05:13.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.785+0000 7ff4cc60e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff4c4156770 0x7ff4c40ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.785+0000 7ff4cc60e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4c40be6b0 0x7ff4c415d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.785+0000 7ff4cc60e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c40c1ca0 con 0x7ff4c40c70e0 2023-10-25T00:05:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.785+0000 7ff4cc60e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c40c27e0 con 0x7ff4c40be6b0 2023-10-25T00:05:13.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.785+0000 7ff4cc60e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c40c10c0 con 0x7ff4c4156770 2023-10-25T00:05:13.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.785+0000 7ff4cabab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4c40be6b0 0x7ff4c415d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.785+0000 7ff4cabab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff4c40be6b0 0x7ff4c415d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49758/0 (socket says 172.21.15.107:49758) 2023-10-25T00:05:13.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.785+0000 7ff4cabab640 1 -- 172.21.15.107:0/2353006972 learned_addr learned my addr 172.21.15.107:0/2353006972 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:13.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.785+0000 7ff4ca3aa640 1 --2- 172.21.15.107:0/2353006972 >> v2:172.21.15.110:3300/0 conn(0x7ff4c4156770 0x7ff4c40ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cabab640 1 -- 172.21.15.107:0/2353006972 >> v2:172.21.15.110:3300/0 conn(0x7ff4c4156770 msgr2=0x7ff4c40ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:13.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cabab640 1 --2- 172.21.15.107:0/2353006972 >> v2:172.21.15.110:3300/0 conn(0x7ff4c4156770 0x7ff4c40ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cabab640 1 -- 172.21.15.107:0/2353006972 >> v1:172.21.15.107:6789/0 conn(0x7ff4c40c70e0 legacy=0x7ff4c40bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4c93a8640 1 -- 172.21.15.107:0/2353006972 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912160276 0 0) 0x7ff4c40c1ca0 con 0x7ff4c40c70e0 2023-10-25T00:05:13.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cabab640 1 -- 172.21.15.107:0/2353006972 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c40c13a0 con 0x7ff4c40be6b0 2023-10-25T00:05:13.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cabab640 1 --2- 172.21.15.107:0/2353006972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40be6b0 0x7ff4c415d130 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40ac6f0 tx=0x7ff4bc01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f64798cbd76723 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:13.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4c93a8640 1 -- 172.21.15.107:0/2353006972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0a5700 con 0x7ff4c40be6b0 2023-10-25T00:05:13.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4c93a8640 1 -- 172.21.15.107:0/2353006972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc0a5860 con 0x7ff4c40be6b0 2023-10-25T00:05:13.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4c93a8640 1 -- 172.21.15.107:0/2353006972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0a5bf0 con 0x7ff4c40be6b0 2023-10-25T00:05:13.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cc60e640 1 -- 172.21.15.107:0/2353006972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40be6b0 msgr2=0x7ff4c415d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cc60e640 1 --2- 172.21.15.107:0/2353006972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40be6b0 0x7ff4c415d130 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff4c40ac6f0 tx=0x7ff4bc01c880 comp rx=0 tx=0).stop 2023-10-25T00:05:13.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cc60e640 1 -- 172.21.15.107:0/2353006972 shutdown_connections 2023-10-25T00:05:13.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cc60e640 1 --2- 172.21.15.107:0/2353006972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40be6b0 0x7ff4c415d130 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cc60e640 1 --2- 172.21.15.107:0/2353006972 >> v2:172.21.15.110:3300/0 conn(0x7ff4c4156770 0x7ff4c40ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cc60e640 1 -- 172.21.15.107:0/2353006972 >> 172.21.15.107:0/2353006972 conn(0x7ff4c40c65c0 msgr2=0x7ff4c40c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:13.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cc60e640 1 -- 172.21.15.107:0/2353006972 shutdown_connections 2023-10-25T00:05:13.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.789+0000 7ff4cc60e640 1 -- 172.21.15.107:0/2353006972 wait complete. 2023-10-25T00:05:13.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cc60e640 1 Processor -- start 2023-10-25T00:05:13.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cc60e640 1 -- start start 2023-10-25T00:05:13.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cc60e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c40be6b0 0x7ff4c40bbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cc60e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c70e0 0x7ff4c40bc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cc60e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4156770 0x7ff4c41ebf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cc60e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c406f530 con 0x7ff4c40c70e0 2023-10-25T00:05:13.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cc60e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c414e290 con 0x7ff4c40be6b0 2023-10-25T00:05:13.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cc60e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c406f710 con 0x7ff4c4156770 2023-10-25T00:05:13.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4c9ba9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c70e0 0x7ff4c40bc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4c9ba9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c70e0 0x7ff4c40bc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49764/0 (socket says 172.21.15.107:49764) 2023-10-25T00:05:13.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4c9ba9640 1 -- 172.21.15.107:0/198652659 learned_addr learned my addr 172.21.15.107:0/198652659 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:13.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4ca3aa640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c40be6b0 0x7ff4c40bbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cabab640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4156770 0x7ff4c41ebf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4c9ba9640 1 -- 172.21.15.107:0/198652659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4156770 msgr2=0x7ff4c41ebf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4c9ba9640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4156770 0x7ff4c41ebf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4c9ba9640 1 -- 172.21.15.107:0/198652659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c40be6b0 msgr2=0x7ff4c40bbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4c9ba9640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c40be6b0 0x7ff4c40bbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4c9ba9640 1 -- 172.21.15.107:0/198652659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc009b30 con 0x7ff4c40c70e0 2023-10-25T00:05:13.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4c9ba9640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c70e0 0x7ff4c40bc000 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400c500 tx=0x7ff4b400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:13.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4baffd640 1 -- 172.21.15.107:0/198652659 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4003600 con 0x7ff4c40c70e0 2023-10-25T00:05:13.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4baffd640 1 -- 172.21.15.107:0/198652659 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b4003760 con 0x7ff4c40c70e0 2023-10-25T00:05:13.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cc60e640 1 -- 172.21.15.107:0/198652659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c41505b0 con 0x7ff4c40c70e0 2023-10-25T00:05:13.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4baffd640 1 -- 172.21.15.107:0/198652659 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4003b10 con 0x7ff4c40c70e0 2023-10-25T00:05:13.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cc60e640 1 -- 172.21.15.107:0/198652659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c414f890 con 0x7ff4c40c70e0 2023-10-25T00:05:13.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4cc60e640 1 -- 172.21.15.107:0/198652659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c414fa80 con 0x7ff4c40c70e0 2023-10-25T00:05:13.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4baffd640 1 -- 172.21.15.107:0/198652659 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff4b4003c70 con 0x7ff4c40c70e0 2023-10-25T00:05:13.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4baffd640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4a8065e60 0x7ff4a8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:13.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4baffd640 1 -- 172.21.15.107:0/198652659 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 8681+0+0 (secure 0 0 0) 0x7ff4b4083920 con 0x7ff4c40c70e0 2023-10-25T00:05:13.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.793+0000 7ff4ca3aa640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4a8065e60 0x7ff4a8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:13.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.797+0000 7ff4ca3aa640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4a8065e60 0x7ff4a8068320 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000b930 tx=0x7ff4c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:13.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.797+0000 7ff4baffd640 1 -- 172.21.15.107:0/198652659 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff4b4050630 con 0x7ff4c40c70e0 2023-10-25T00:05:13.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.909+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/3949216099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9a40746e0 con 0x7ff9a406e800 2023-10-25T00:05:13.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.909+0000 7ff9a1ffb640 1 -- 172.21.15.107:0/3949216099 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5015 (secure 0 0 0) 0x7ff994050b20 con 0x7ff9a406e800 2023-10-25T00:05:13.914 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:13.914 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:13.610083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:13.914 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 106 2023-10-25T00:05:13.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/3949216099 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff990065e60 msgr2=0x7ff990068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff990065e60 0x7ff990068320 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4151960 tx=0x7ff998008040 comp rx=0 tx=0).stop 2023-10-25T00:05:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/3949216099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9a406e800 msgr2=0x7ff9a40c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9a406e800 0x7ff9a40c7720 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ff99400ba80 tx=0x7ff99400bf40 comp rx=0 tx=0).stop 2023-10-25T00:05:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/3949216099 shutdown_connections 2023-10-25T00:05:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff9a4152040 0x7ff9a41ecff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff990065e60 0x7ff990068320 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff9a40adb80 0x7ff9a4150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 --2- 172.21.15.107:0/3949216099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff9a406e800 0x7ff9a40c7720 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/3949216099 >> 172.21.15.107:0/3949216099 conn(0x7ff9a4070f60 msgr2=0x7ff9a41541a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:13.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/3949216099 shutdown_connections 2023-10-25T00:05:13.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.913+0000 7ff9ab8fd640 1 -- 172.21.15.107:0/3949216099 wait complete. 2023-10-25T00:05:13.940 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:05:14.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.997+0000 7ff4cc60e640 1 -- 172.21.15.107:0/198652659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4c40746e0 con 0x7ff4c40c70e0 2023-10-25T00:05:14.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:13.997+0000 7ff4baffd640 1 -- 172.21.15.107:0/198652659 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+5015 (secure 0 0 0) 0x7ff4b4017070 con 0x7ff4c40c70e0 2023-10-25T00:05:14.002 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:14.002 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:13.610083+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:14.002 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 106 2023-10-25T00:05:14.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 -- 172.21.15.107:0/198652659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4a8065e60 msgr2=0x7ff4a8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:14.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4a8065e60 0x7ff4a8068320 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000b930 tx=0x7ff4c0008040 comp rx=0 tx=0).stop 2023-10-25T00:05:14.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 -- 172.21.15.107:0/198652659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c70e0 msgr2=0x7ff4c40bc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:14.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c70e0 0x7ff4c40bc000 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400c500 tx=0x7ff4b400c9c0 comp rx=0 tx=0).stop 2023-10-25T00:05:14.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 -- 172.21.15.107:0/198652659 shutdown_connections 2023-10-25T00:05:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff4a8065e60 0x7ff4a8068320 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4156770 0x7ff4c41ebf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4c40c70e0 0x7ff4c40bc000 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 --2- 172.21.15.107:0/198652659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c40be6b0 0x7ff4c40bbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:14.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 -- 172.21.15.107:0/198652659 >> 172.21.15.107:0/198652659 conn(0x7ff4c40c65c0 msgr2=0x7ff4c40c2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 -- 172.21.15.107:0/198652659 shutdown_connections 2023-10-25T00:05:14.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:14.001+0000 7ff4cc60e640 1 -- 172.21.15.107:0/198652659 wait complete. 2023-10-25T00:05:14.021 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 106, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 106, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6022": { "addr": "172.21.15.110:6834/174863522", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 174863522, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 174863522, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6022, "incarnation": 106, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:05:13.610083+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6022 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 99, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6753": { "addr": "172.21.15.107:6837/1555840329", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 1555840329, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 1555840329, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6753, "incarnation": 96, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:04:31.705767+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6753 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 59, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:05:15.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:16.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:16.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763facc640 1 Processor -- start 2023-10-25T00:05:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763facc640 1 -- start start 2023-10-25T00:05:16.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763facc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f76380720c0 0x7f763813d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:16.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763facc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7638154860 0x7f7638158230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:16.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763facc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76380adf70 con 0x7f7638071ce0 2023-10-25T00:05:16.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763facc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763806d900 con 0x7f7638154860 2023-10-25T00:05:16.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763facc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763806e710 con 0x7f76380720c0 2023-10-25T00:05:16.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763d868640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7638071ce0 0x7f76380ad860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45666/0 (socket says 172.21.15.107:45666) 2023-10-25T00:05:16.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763d868640 1 -- 172.21.15.107:0/1071344160 learned_addr learned my addr 172.21.15.107:0/1071344160 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:16.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763e069640 1 --2- 172.21.15.107:0/1071344160 >> v2:172.21.15.107:3300/0 conn(0x7f7638154860 0x7f7638158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:16.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763d067640 1 --2- 172.21.15.107:0/1071344160 >> v2:172.21.15.110:3300/0 conn(0x7f76380720c0 0x7f763813d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:16.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.065+0000 7f763c866640 1 -- 172.21.15.107:0/1071344160 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669837697 0 0) 0x7f76380adf70 con 0x7f7638071ce0 2023-10-25T00:05:16.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763c866640 1 -- 172.21.15.107:0/1071344160 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7624003530 con 0x7f7638071ce0 2023-10-25T00:05:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763c866640 1 -- 172.21.15.107:0/1071344160 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1880468201 0 0) 0x7f7624003530 con 0x7f7638071ce0 2023-10-25T00:05:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763e069640 1 -- 172.21.15.107:0/1071344160 >> v2:172.21.15.110:3300/0 conn(0x7f76380720c0 msgr2=0x7f763813d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763e069640 1 --2- 172.21.15.107:0/1071344160 >> v2:172.21.15.110:3300/0 conn(0x7f76380720c0 0x7f763813d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763e069640 1 -- 172.21.15.107:0/1071344160 >> v1:172.21.15.107:6789/0 conn(0x7f7638071ce0 legacy=0x7f76380ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763e069640 1 -- 172.21.15.107:0/1071344160 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7638154d20 con 0x7f7638154860 2023-10-25T00:05:16.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763e069640 1 --2- 172.21.15.107:0/1071344160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638154860 0x7f7638158230 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7638073d60 tx=0x7f763401c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a362c09a431d2f4c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:16.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763c866640 1 -- 172.21.15.107:0/1071344160 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630001680 con 0x7f7638154860 2023-10-25T00:05:16.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763c866640 1 -- 172.21.15.107:0/1071344160 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76340a55b0 con 0x7f7638154860 2023-10-25T00:05:16.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763c866640 1 -- 172.21.15.107:0/1071344160 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76340a5940 con 0x7f7638154860 2023-10-25T00:05:16.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763facc640 1 -- 172.21.15.107:0/1071344160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638154860 msgr2=0x7f7638158230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763facc640 1 --2- 172.21.15.107:0/1071344160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638154860 0x7f7638158230 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7638073d60 tx=0x7f763401c770 comp rx=0 tx=0).stop 2023-10-25T00:05:16.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763facc640 1 -- 172.21.15.107:0/1071344160 shutdown_connections 2023-10-25T00:05:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763facc640 1 --2- 172.21.15.107:0/1071344160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638154860 0x7f7638158230 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763facc640 1 --2- 172.21.15.107:0/1071344160 >> v2:172.21.15.110:3300/0 conn(0x7f76380720c0 0x7f763813d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763facc640 1 -- 172.21.15.107:0/1071344160 >> 172.21.15.107:0/1071344160 conn(0x7f7638070f60 msgr2=0x7f7638071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763facc640 1 -- 172.21.15.107:0/1071344160 shutdown_connections 2023-10-25T00:05:16.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.069+0000 7f763facc640 1 -- 172.21.15.107:0/1071344160 wait complete. 2023-10-25T00:05:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763facc640 1 Processor -- start 2023-10-25T00:05:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763facc640 1 -- start start 2023-10-25T00:05:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763facc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638071ce0 0x7f76381ed720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763facc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76380720c0 0x7f76381f0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:16.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763facc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7638154860 0x7f76380c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:16.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763facc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76381f1090 con 0x7f7638071ce0 2023-10-25T00:05:16.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763facc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7638151ad0 con 0x7f76380720c0 2023-10-25T00:05:16.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763facc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7638151db0 con 0x7f7638154860 2023-10-25T00:05:16.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763d868640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638071ce0 0x7f76381ed720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:16.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763d868640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638071ce0 0x7f76381ed720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49784/0 (socket says 172.21.15.107:49784) 2023-10-25T00:05:16.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763d868640 1 -- 172.21.15.107:0/3043906616 learned_addr learned my addr 172.21.15.107:0/3043906616 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:16.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763d868640 1 -- 172.21.15.107:0/3043906616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7638154860 msgr2=0x7f76380c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763d868640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7638154860 0x7f76380c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763d067640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76380720c0 0x7f76381f0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:16.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763d868640 1 -- 172.21.15.107:0/3043906616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76380720c0 msgr2=0x7f76381f0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763d868640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76380720c0 0x7f76381f0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763d868640 1 -- 172.21.15.107:0/3043906616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7634009a70 con 0x7f7638071ce0 2023-10-25T00:05:16.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763d868640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638071ce0 0x7f76381ed720 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f763000c780 tx=0x7f763000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:16.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f762e7fc640 1 -- 172.21.15.107:0/3043906616 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7630005cf0 con 0x7f7638071ce0 2023-10-25T00:05:16.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f762e7fc640 1 -- 172.21.15.107:0/3043906616 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7630005e50 con 0x7f7638071ce0 2023-10-25T00:05:16.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763facc640 1 -- 172.21.15.107:0/3043906616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76381ecf20 con 0x7f7638071ce0 2023-10-25T00:05:16.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f762e7fc640 1 -- 172.21.15.107:0/3043906616 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76300072b0 con 0x7f7638071ce0 2023-10-25T00:05:16.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763facc640 1 -- 172.21.15.107:0/3043906616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763813d380 con 0x7f7638071ce0 2023-10-25T00:05:16.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f762e7fc640 1 -- 172.21.15.107:0/3043906616 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7630007410 con 0x7f7638071ce0 2023-10-25T00:05:16.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f763facc640 1 -- 172.21.15.107:0/3043906616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76380746e0 con 0x7f7638071ce0 2023-10-25T00:05:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.073+0000 7f762e7fc640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f761c065e60 0x7f761c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.077+0000 7f762e7fc640 1 -- 172.21.15.107:0/3043906616 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 8681+0+0 (secure 0 0 0) 0x7f7630083ff0 con 0x7f7638071ce0 2023-10-25T00:05:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.077+0000 7f763d067640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f761c065e60 0x7f761c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:16.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.077+0000 7f763d067640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f761c065e60 0x7f761c068320 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7638152aa0 tx=0x7f7628008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:16.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.081+0000 7f762e7fc640 1 -- 172.21.15.107:0/3043906616 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7630050c40 con 0x7f7638071ce0 2023-10-25T00:05:16.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.137+0000 7fbd0b5d6640 1 Processor -- start 2023-10-25T00:05:16.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.137+0000 7fbd0b5d6640 1 -- start start 2023-10-25T00:05:16.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd0b5d6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd04154860 0x7fbd04158230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:16.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd0b5d6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd040adf70 con 0x7fbd040720c0 2023-10-25T00:05:16.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd0b5d6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0406e710 con 0x7fbd04071ce0 2023-10-25T00:05:16.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd0b5d6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd0406d900 con 0x7fbd04154860 2023-10-25T00:05:16.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd08b71640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbd040720c0 0x7fbd0413d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45680/0 (socket says 172.21.15.107:45680) 2023-10-25T00:05:16.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd08b71640 1 -- 172.21.15.107:0/91523748 learned_addr learned my addr 172.21.15.107:0/91523748 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:16.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd09b73640 1 --2- 172.21.15.107:0/91523748 >> v2:172.21.15.107:3300/0 conn(0x7fbd04154860 0x7fbd04158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:16.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbcfbfff640 1 -- 172.21.15.107:0/91523748 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581167692 0 0) 0x7fbd040adf70 con 0x7fbd040720c0 2023-10-25T00:05:16.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbcfbfff640 1 -- 172.21.15.107:0/91523748 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf0003530 con 0x7fbd040720c0 2023-10-25T00:05:16.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd09b73640 1 -- 172.21.15.107:0/91523748 >> v1:172.21.15.110:6789/0 conn(0x7fbd04071ce0 legacy=0x7fbd040ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd09b73640 1 -- 172.21.15.107:0/91523748 >> v1:172.21.15.107:6789/0 conn(0x7fbd040720c0 legacy=0x7fbd0413d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd09b73640 1 -- 172.21.15.107:0/91523748 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd04154d20 con 0x7fbd04154860 2023-10-25T00:05:16.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd09b73640 1 --2- 172.21.15.107:0/91523748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd04154860 0x7fbd04158230 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc009d90 tx=0x7fbcfc01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a975aeee37eee1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:16.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbcfbfff640 1 -- 172.21.15.107:0/91523748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc0a5950 con 0x7fbd04154860 2023-10-25T00:05:16.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbcfbfff640 1 -- 172.21.15.107:0/91523748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcfc0a5ab0 con 0x7fbd04154860 2023-10-25T00:05:16.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbcfbfff640 1 -- 172.21.15.107:0/91523748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc0a5e40 con 0x7fbd04154860 2023-10-25T00:05:16.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/91523748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd04154860 msgr2=0x7fbd04158230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd0b5d6640 1 --2- 172.21.15.107:0/91523748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd04154860 0x7fbd04158230 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc009d90 tx=0x7fbcfc01c890 comp rx=0 tx=0).stop 2023-10-25T00:05:16.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/91523748 shutdown_connections 2023-10-25T00:05:16.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd0b5d6640 1 --2- 172.21.15.107:0/91523748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd04154860 0x7fbd04158230 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/91523748 >> 172.21.15.107:0/91523748 conn(0x7fbd04070f60 msgr2=0x7fbd04071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:16.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.141+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/91523748 shutdown_connections 2023-10-25T00:05:16.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/91523748 wait complete. 2023-10-25T00:05:16.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 Processor -- start 2023-10-25T00:05:16.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 -- start start 2023-10-25T00:05:16.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd04071ce0 0x7fbd04152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:16.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd040720c0 0x7fbd041ecf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:16.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd04154860 0x7fbd041ed450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:16.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd04156530 con 0x7fbd040720c0 2023-10-25T00:05:16.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd04157c70 con 0x7fbd04071ce0 2023-10-25T00:05:16.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041ec8e0 con 0x7fbd04154860 2023-10-25T00:05:16.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd08b71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd040720c0 0x7fbd041ecf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:16.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd08b71640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd040720c0 0x7fbd041ecf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49810/0 (socket says 172.21.15.107:49810) 2023-10-25T00:05:16.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd08b71640 1 -- 172.21.15.107:0/731071536 learned_addr learned my addr 172.21.15.107:0/731071536 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:16.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd09372640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd04071ce0 0x7fbd04152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:16.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd09b73640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd04154860 0x7fbd041ed450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:16.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd08b71640 1 -- 172.21.15.107:0/731071536 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd04154860 msgr2=0x7fbd041ed450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd08b71640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd04154860 0x7fbd041ed450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd08b71640 1 -- 172.21.15.107:0/731071536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd04071ce0 msgr2=0x7fbd04152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd08b71640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd04071ce0 0x7fbd04152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd08b71640 1 -- 172.21.15.107:0/731071536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc009a70 con 0x7fbd040720c0 2023-10-25T00:05:16.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd08b71640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd040720c0 0x7fbd041ecf10 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000c4d0 tx=0x7fbd0000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:16.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbcf9ffb640 1 -- 172.21.15.107:0/731071536 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00005cf0 con 0x7fbd040720c0 2023-10-25T00:05:16.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbcf9ffb640 1 -- 172.21.15.107:0/731071536 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd00005e50 con 0x7fbd040720c0 2023-10-25T00:05:16.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/731071536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0413e200 con 0x7fbd040720c0 2023-10-25T00:05:16.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbcf9ffb640 1 -- 172.21.15.107:0/731071536 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00007270 con 0x7fbd040720c0 2023-10-25T00:05:16.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/731071536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd041576b0 con 0x7fbd040720c0 2023-10-25T00:05:16.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.145+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/731071536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd041efee0 con 0x7fbd040720c0 2023-10-25T00:05:16.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.149+0000 7fbcf9ffb640 1 -- 172.21.15.107:0/731071536 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbd000073d0 con 0x7fbd040720c0 2023-10-25T00:05:16.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.149+0000 7fbcf9ffb640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbce8065e60 0x7fbce8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:16.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.149+0000 7fbcf9ffb640 1 -- 172.21.15.107:0/731071536 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 8681+0+0 (secure 0 0 0) 0x7fbd00083b70 con 0x7fbd040720c0 2023-10-25T00:05:16.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.149+0000 7fbd09372640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbce8065e60 0x7fbce8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:16.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.149+0000 7fbd09372640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbce8065e60 0x7fbce8068320 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbd040670d0 tx=0x7fbcf4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:16.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.157+0000 7fbcf9ffb640 1 -- 172.21.15.107:0/731071536 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbd000508f0 con 0x7fbd040720c0 2023-10-25T00:05:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.269+0000 7f763facc640 1 -- 172.21.15.107:0/3043906616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f763815efc0 con 0x7f7638071ce0 2023-10-25T00:05:16.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.269+0000 7f762e7fc640 1 -- 172.21.15.107:0/3043906616 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f7630054260 con 0x7f7638071ce0 2023-10-25T00:05:16.273 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:16.273 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:15.760207+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:16.273 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 108 2023-10-25T00:05:16.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 -- 172.21.15.107:0/3043906616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f761c065e60 msgr2=0x7f761c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f761c065e60 0x7f761c068320 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7638152aa0 tx=0x7f7628008040 comp rx=0 tx=0).stop 2023-10-25T00:05:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 -- 172.21.15.107:0/3043906616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638071ce0 msgr2=0x7f76381ed720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638071ce0 0x7f76381ed720 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f763000c780 tx=0x7f763000cc40 comp rx=0 tx=0).stop 2023-10-25T00:05:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 -- 172.21.15.107:0/3043906616 shutdown_connections 2023-10-25T00:05:16.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f761c065e60 0x7f761c068320 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7638154860 0x7f76380c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76380720c0 0x7f76381f0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 --2- 172.21.15.107:0/3043906616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7638071ce0 0x7f76381ed720 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 -- 172.21.15.107:0/3043906616 >> 172.21.15.107:0/3043906616 conn(0x7f7638070f60 msgr2=0x7f763806afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:16.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 -- 172.21.15.107:0/3043906616 shutdown_connections 2023-10-25T00:05:16.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.273+0000 7f763facc640 1 -- 172.21.15.107:0/3043906616 wait complete. 2023-10-25T00:05:16.290 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:05:16.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.341+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/731071536 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd040746e0 con 0x7fbd040720c0 2023-10-25T00:05:16.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.341+0000 7fbcf9ffb640 1 -- 172.21.15.107:0/731071536 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+5016 (secure 0 0 0) 0x7fbd00017090 con 0x7fbd040720c0 2023-10-25T00:05:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:16.346 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5410,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:15.760207+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:04:31.705767+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6753},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6753":{"gid":6753,"name":"a","rank":0,"incarnation":96,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/1555840329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1555840329},{"type":"v1","addr":"172.21.15.107:6837","nonce":1555840329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:16.346 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 108 2023-10-25T00:05:16.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/731071536 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbce8065e60 msgr2=0x7fbce8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbce8065e60 0x7fbce8068320 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbd040670d0 tx=0x7fbcf4009040 comp rx=0 tx=0).stop 2023-10-25T00:05:16.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/731071536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd040720c0 msgr2=0x7fbd041ecf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:16.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd040720c0 0x7fbd041ecf10 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fbd0000c4d0 tx=0x7fbd0000c990 comp rx=0 tx=0).stop 2023-10-25T00:05:16.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/731071536 shutdown_connections 2023-10-25T00:05:16.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbce8065e60 0x7fbce8068320 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd04154860 0x7fbd041ed450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd040720c0 0x7fbd041ecf10 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 --2- 172.21.15.107:0/731071536 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd04071ce0 0x7fbd04152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:16.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/731071536 >> 172.21.15.107:0/731071536 conn(0x7fbd04070f60 msgr2=0x7fbd040c3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:16.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/731071536 shutdown_connections 2023-10-25T00:05:16.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:16.345+0000 7fbd0b5d6640 1 -- 172.21.15.107:0/731071536 wait complete. 2023-10-25T00:05:16.364 INFO:tasks.mds_thrash.fs.[a]:no change 2023-10-25T00:05:18.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:18.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:18.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f4857c2a640 1 Processor -- start 2023-10-25T00:05:18.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f4857c2a640 1 -- start start 2023-10-25T00:05:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f4857c2a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f485014c470 0x7f485014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f4857c2a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4850152b30 0x7f4850152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f4857c2a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4850155cd0 con 0x7f485006f490 2023-10-25T00:05:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f4857c2a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48501526e0 con 0x7f4850152b30 2023-10-25T00:05:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f4857c2a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48500c4380 con 0x7f485014c470 2023-10-25T00:05:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f48561c7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4850152b30 0x7f4850152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f48559c6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f485014c470 0x7f485014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f48561c7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4850152b30 0x7f4850152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35688/0 (socket says 172.21.15.107:35688) 2023-10-25T00:05:18.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f48561c7640 1 -- 172.21.15.107:0/3272403433 learned_addr learned my addr 172.21.15.107:0/3272403433 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:18.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f48561c7640 1 -- 172.21.15.107:0/3272403433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f485014c470 msgr2=0x7f485014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:18.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f48561c7640 1 --2- 172.21.15.107:0/3272403433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f485014c470 0x7f485014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f48561c7640 1 -- 172.21.15.107:0/3272403433 >> v1:172.21.15.110:6789/0 conn(0x7f485006f490 legacy=0x7f4850071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.437+0000 7f48561c7640 1 -- 172.21.15.107:0/3272403433 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48500c48b0 con 0x7f4850152b30 2023-10-25T00:05:18.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f48561c7640 1 --2- 172.21.15.107:0/3272403433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4850152b30 0x7f4850152fa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4848009e50 tx=0x7f484801c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=197ca19d24baabd3 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:18.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f48549c4640 1 -- 172.21.15.107:0/3272403433 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48480a56a0 con 0x7f4850152b30 2023-10-25T00:05:18.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f48549c4640 1 -- 172.21.15.107:0/3272403433 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48480a5800 con 0x7f4850152b30 2023-10-25T00:05:18.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f48549c4640 1 -- 172.21.15.107:0/3272403433 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48480a5b90 con 0x7f4850152b30 2023-10-25T00:05:18.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 -- 172.21.15.107:0/3272403433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4850152b30 msgr2=0x7f4850152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 --2- 172.21.15.107:0/3272403433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4850152b30 0x7f4850152fa0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4848009e50 tx=0x7f484801c8f0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 -- 172.21.15.107:0/3272403433 shutdown_connections 2023-10-25T00:05:18.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 --2- 172.21.15.107:0/3272403433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4850152b30 0x7f4850152fa0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 --2- 172.21.15.107:0/3272403433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f485014c470 0x7f485014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 -- 172.21.15.107:0/3272403433 >> 172.21.15.107:0/3272403433 conn(0x7f4850070f60 msgr2=0x7f4850071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:18.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 -- 172.21.15.107:0/3272403433 shutdown_connections 2023-10-25T00:05:18.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 -- 172.21.15.107:0/3272403433 wait complete. 2023-10-25T00:05:18.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 Processor -- start 2023-10-25T00:05:18.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 -- start start 2023-10-25T00:05:18.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f485006f490 0x7f48500c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f485014c470 0x7f48501405e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4850152b30 0x7f4850140b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4850141310 con 0x7f485006f490 2023-10-25T00:05:18.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4850141560 con 0x7f485014c470 2023-10-25T00:05:18.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.441+0000 7f4857c2a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f485006c630 con 0x7f4850152b30 2023-10-25T00:05:18.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48559c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f485006f490 0x7f48500c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48559c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f485006f490 0x7f48500c3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49820/0 (socket says 172.21.15.107:49820) 2023-10-25T00:05:18.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48559c6640 1 -- 172.21.15.107:0/594496407 learned_addr learned my addr 172.21.15.107:0/594496407 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:18.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48551c5640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f485014c470 0x7f48501405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48561c7640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4850152b30 0x7f4850140b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48559c6640 1 -- 172.21.15.107:0/594496407 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4850152b30 msgr2=0x7f4850140b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48559c6640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4850152b30 0x7f4850140b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48559c6640 1 -- 172.21.15.107:0/594496407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f485014c470 msgr2=0x7f48501405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48559c6640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f485014c470 0x7f48501405e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48559c6640 1 -- 172.21.15.107:0/594496407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4848009b30 con 0x7f485006f490 2023-10-25T00:05:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48561c7640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4850152b30 0x7f4850140b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:05:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48559c6640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f485006f490 0x7f48500c3a80 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f484c00e930 tx=0x7f484c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:18.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48467fc640 1 -- 172.21.15.107:0/594496407 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c009690 con 0x7f485006f490 2023-10-25T00:05:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48467fc640 1 -- 172.21.15.107:0/594496407 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c00ca00 con 0x7f485006f490 2023-10-25T00:05:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f4857c2a640 1 -- 172.21.15.107:0/594496407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485006c910 con 0x7f485006f490 2023-10-25T00:05:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48467fc640 1 -- 172.21.15.107:0/594496407 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c00cd70 con 0x7f485006f490 2023-10-25T00:05:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f4857c2a640 1 -- 172.21.15.107:0/594496407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4850141060 con 0x7f485006f490 2023-10-25T00:05:18.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f4857c2a640 1 -- 172.21.15.107:0/594496407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485014ff60 con 0x7f485006f490 2023-10-25T00:05:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48467fc640 1 -- 172.21.15.107:0/594496407 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f484c013470 con 0x7f485006f490 2023-10-25T00:05:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.445+0000 7f48467fc640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4834065e60 0x7f4834068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.449+0000 7f48467fc640 1 -- 172.21.15.107:0/594496407 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 8681+0+0 (secure 0 0 0) 0x7f484c084cd0 con 0x7f485006f490 2023-10-25T00:05:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.449+0000 7f48551c5640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4834065e60 0x7f4834068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.449+0000 7f48551c5640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4834065e60 0x7f4834068320 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f48500c2580 tx=0x7f4840009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.449+0000 7f48467fc640 1 -- 172.21.15.107:0/594496407 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f484c0519d0 con 0x7f485006f490 2023-10-25T00:05:18.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f6658251640 1 Processor -- start 2023-10-25T00:05:18.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f6658251640 1 -- start start 2023-10-25T00:05:18.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f6658251640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6650065bc0 0x7f6650158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f6658251640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f665006a120 0x7f66500aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f6658251640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665006a5b0 con 0x7f6650069d40 2023-10-25T00:05:18.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f6658251640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66500ac650 con 0x7f665006a120 2023-10-25T00:05:18.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f6658251640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66500a9e00 con 0x7f6650065bc0 2023-10-25T00:05:18.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f66567ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f665006a120 0x7f66500aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f66567ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f665006a120 0x7f66500aaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49828/0 (socket says 172.21.15.107:49828) 2023-10-25T00:05:18.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f66567ee640 1 -- 172.21.15.107:0/109305500 learned_addr learned my addr 172.21.15.107:0/109305500 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:18.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f66557ec640 1 --2- 172.21.15.107:0/109305500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6650065bc0 0x7f6650158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f66567ee640 1 -- 172.21.15.107:0/109305500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6650065bc0 msgr2=0x7f6650158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:18.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f66567ee640 1 --2- 172.21.15.107:0/109305500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6650065bc0 0x7f6650158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f66567ee640 1 -- 172.21.15.107:0/109305500 >> v1:172.21.15.110:6789/0 conn(0x7f6650069d40 legacy=0x7f66500a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.501+0000 7f66567ee640 1 -- 172.21.15.107:0/109305500 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665006cea0 con 0x7f665006a120 2023-10-25T00:05:18.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f66567ee640 1 --2- 172.21.15.107:0/109305500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665006a120 0x7f66500aaef0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6648009e50 tx=0x7f664801c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c49cd8a7184aecc server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:18.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f6654feb640 1 -- 172.21.15.107:0/109305500 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66480a5640 con 0x7f665006a120 2023-10-25T00:05:18.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f6654feb640 1 -- 172.21.15.107:0/109305500 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66480a57a0 con 0x7f665006a120 2023-10-25T00:05:18.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f6654feb640 1 -- 172.21.15.107:0/109305500 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66480a5b30 con 0x7f665006a120 2023-10-25T00:05:18.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f6658251640 1 -- 172.21.15.107:0/109305500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665006a120 msgr2=0x7f66500aaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f6658251640 1 --2- 172.21.15.107:0/109305500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665006a120 0x7f66500aaef0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6648009e50 tx=0x7f664801c980 comp rx=0 tx=0).stop 2023-10-25T00:05:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f6658251640 1 -- 172.21.15.107:0/109305500 shutdown_connections 2023-10-25T00:05:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f6658251640 1 --2- 172.21.15.107:0/109305500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665006a120 0x7f66500aaef0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f6658251640 1 --2- 172.21.15.107:0/109305500 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6650065bc0 0x7f6650158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f6658251640 1 -- 172.21.15.107:0/109305500 >> 172.21.15.107:0/109305500 conn(0x7f66500c65c0 msgr2=0x7f66500c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:18.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f6658251640 1 -- 172.21.15.107:0/109305500 shutdown_connections 2023-10-25T00:05:18.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.505+0000 7f6658251640 1 -- 172.21.15.107:0/109305500 wait complete. 2023-10-25T00:05:18.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6658251640 1 Processor -- start 2023-10-25T00:05:18.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6658251640 1 -- start start 2023-10-25T00:05:18.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6658251640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6650065bc0 0x7f66501f0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6658251640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6650069d40 0x7f66501f0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6658251640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665006a120 0x7f66500bc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6658251640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500ba0f0 con 0x7f665006a120 2023-10-25T00:05:18.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6658251640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66501f08d0 con 0x7f6650069d40 2023-10-25T00:05:18.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6658251640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665006f6d0 con 0x7f6650065bc0 2023-10-25T00:05:18.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66567ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665006a120 0x7f66500bc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66567ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665006a120 0x7f66500bc5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49840/0 (socket says 172.21.15.107:49840) 2023-10-25T00:05:18.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66567ee640 1 -- 172.21.15.107:0/736547762 learned_addr learned my addr 172.21.15.107:0/736547762 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:18.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6655fed640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6650065bc0 0x7f66501f0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66557ec640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6650069d40 0x7f66501f0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66567ee640 1 -- 172.21.15.107:0/736547762 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6650065bc0 msgr2=0x7f66501f0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66567ee640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6650065bc0 0x7f66501f0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66567ee640 1 -- 172.21.15.107:0/736547762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6650069d40 msgr2=0x7f66501f0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66567ee640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6650069d40 0x7f66501f0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66567ee640 1 -- 172.21.15.107:0/736547762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6648009b30 con 0x7f665006a120 2023-10-25T00:05:18.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66557ec640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6650069d40 0x7f66501f0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:05:18.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6655fed640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6650065bc0 0x7f66501f0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:05:18.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66567ee640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665006a120 0x7f66500bc5d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f66480af040 tx=0x7f6648019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:18.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66467fc640 1 -- 172.21.15.107:0/736547762 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66480b8670 con 0x7f665006a120 2023-10-25T00:05:18.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66467fc640 1 -- 172.21.15.107:0/736547762 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66480b87d0 con 0x7f665006a120 2023-10-25T00:05:18.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6658251640 1 -- 172.21.15.107:0/736547762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66501f15a0 con 0x7f665006a120 2023-10-25T00:05:18.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f66467fc640 1 -- 172.21.15.107:0/736547762 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66480b8b40 con 0x7f665006a120 2023-10-25T00:05:18.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6658251640 1 -- 172.21.15.107:0/736547762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66500c4110 con 0x7f665006a120 2023-10-25T00:05:18.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.509+0000 7f6658251640 1 -- 172.21.15.107:0/736547762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66500c0a80 con 0x7f665006a120 2023-10-25T00:05:18.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.513+0000 7f66467fc640 1 -- 172.21.15.107:0/736547762 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f66480b8ca0 con 0x7f665006a120 2023-10-25T00:05:18.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.513+0000 7f66467fc640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6634065e90 0x7f6634068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.513+0000 7f66467fc640 1 -- 172.21.15.107:0/736547762 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 8681+0+0 (secure 0 0 0) 0x7f664811f690 con 0x7f665006a120 2023-10-25T00:05:18.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.513+0000 7f6655fed640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6634065e90 0x7f6634068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.513+0000 7f6655fed640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6634065e90 0x7f6634068350 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6650158e10 tx=0x7f664c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:18.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.517+0000 7f66467fc640 1 -- 172.21.15.107:0/736547762 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f66480ec4b0 con 0x7f665006a120 2023-10-25T00:05:18.638 INFO:tasks.ceph.mds.b.smithi110.stderr:2023-10-25T00:05:18.636+0000 7ff0289b3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:05:18.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.693+0000 7f4857c2a640 1 -- 172.21.15.107:0/594496407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48500746e0 con 0x7f485006f490 2023-10-25T00:05:18.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.693+0000 7f48467fc640 1 -- 172.21.15.107:0/594496407 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4259 (secure 0 0 0) 0x7f484c021090 con 0x7f485006f490 2023-10-25T00:05:18.698 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:18.698 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:18.611450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:18.698 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 110 2023-10-25T00:05:18.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 -- 172.21.15.107:0/594496407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4834065e60 msgr2=0x7f4834068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4834065e60 0x7f4834068320 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f48500c2580 tx=0x7f4840009040 comp rx=0 tx=0).stop 2023-10-25T00:05:18.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 -- 172.21.15.107:0/594496407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f485006f490 msgr2=0x7f48500c3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f485006f490 0x7f48500c3a80 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f484c00e930 tx=0x7f484c00edf0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 -- 172.21.15.107:0/594496407 shutdown_connections 2023-10-25T00:05:18.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4834065e60 0x7f4834068320 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4850152b30 0x7f4850140b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f485014c470 0x7f48501405e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 --2- 172.21.15.107:0/594496407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f485006f490 0x7f48500c3a80 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 -- 172.21.15.107:0/594496407 >> 172.21.15.107:0/594496407 conn(0x7f4850070f60 msgr2=0x7f4850155560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:18.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 -- 172.21.15.107:0/594496407 shutdown_connections 2023-10-25T00:05:18.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.697+0000 7f4857c2a640 1 -- 172.21.15.107:0/594496407 wait complete. 2023-10-25T00:05:18.718 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2023-10-25T00:05:18.719 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:05:18.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:18.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.773+0000 7f6658251640 1 -- 172.21.15.107:0/736547762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66500746e0 con 0x7f665006a120 2023-10-25T00:05:18.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.773+0000 7f66467fc640 1 -- 172.21.15.107:0/736547762 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4259 (secure 0 0 0) 0x7f6648009510 con 0x7f665006a120 2023-10-25T00:05:18.777 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:18.777 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:18.611450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:18.777 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 110 2023-10-25T00:05:18.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 -- 172.21.15.107:0/736547762 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6634065e90 msgr2=0x7f6634068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6634065e90 0x7f6634068350 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f6650158e10 tx=0x7f664c009040 comp rx=0 tx=0).stop 2023-10-25T00:05:18.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 -- 172.21.15.107:0/736547762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665006a120 msgr2=0x7f66500bc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665006a120 0x7f66500bc5d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f66480af040 tx=0x7f6648019720 comp rx=0 tx=0).stop 2023-10-25T00:05:18.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 -- 172.21.15.107:0/736547762 shutdown_connections 2023-10-25T00:05:18.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665006a120 0x7f66500bc5d0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6650069d40 0x7f66501f0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6634065e90 0x7f6634068350 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 --2- 172.21.15.107:0/736547762 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6650065bc0 0x7f66501f0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 -- 172.21.15.107:0/736547762 >> 172.21.15.107:0/736547762 conn(0x7f66500c65c0 msgr2=0x7f665006c490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:18.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 -- 172.21.15.107:0/736547762 shutdown_connections 2023-10-25T00:05:18.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.777+0000 7f6658251640 1 -- 172.21.15.107:0/736547762 wait complete. 2023-10-25T00:05:18.800 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=6906 2023-10-25T00:05:18.800 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.c 2023-10-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f59835c0640 1 Processor -- start 2023-10-25T00:05:18.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f59835c0640 1 -- start start 2023-10-25T00:05:18.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f59835c0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f597c152040 0x7f597c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f59835c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f597c0c8100 0x7f597c0abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f59835c0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f597c155680 con 0x7f597c06a4b0 2023-10-25T00:05:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f59835c0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597c154a70 con 0x7f597c152040 2023-10-25T00:05:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f59835c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597c154d50 con 0x7f597c0c8100 2023-10-25T00:05:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f598135c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f597c152040 0x7f597c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f598135c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f597c152040 0x7f597c0c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49850/0 (socket says 172.21.15.107:49850) 2023-10-25T00:05:18.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f598135c640 1 -- 172.21.15.107:0/3532631830 learned_addr learned my addr 172.21.15.107:0/3532631830 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:18.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f5980b5b640 1 --2- 172.21.15.107:0/3532631830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f597c0c8100 0x7f597c0abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f598135c640 1 -- 172.21.15.107:0/3532631830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f597c0c8100 msgr2=0x7f597c0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:18.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f598135c640 1 --2- 172.21.15.107:0/3532631830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f597c0c8100 0x7f597c0abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f598135c640 1 -- 172.21.15.107:0/3532631830 >> v1:172.21.15.110:6789/0 conn(0x7f597c06a4b0 legacy=0x7f597c0ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.849+0000 7f598135c640 1 -- 172.21.15.107:0/3532631830 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597c155030 con 0x7f597c152040 2023-10-25T00:05:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f598135c640 1 --2- 172.21.15.107:0/3532631830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f597c152040 0x7f597c0c4b80 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f5978009870 tx=0x7f597801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe0d87085c1287d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f5973fff640 1 -- 172.21.15.107:0/3532631830 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59780ac620 con 0x7f597c152040 2023-10-25T00:05:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f5973fff640 1 -- 172.21.15.107:0/3532631830 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59780ac780 con 0x7f597c152040 2023-10-25T00:05:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f5973fff640 1 -- 172.21.15.107:0/3532631830 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59780acb10 con 0x7f597c152040 2023-10-25T00:05:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f59835c0640 1 -- 172.21.15.107:0/3532631830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f597c152040 msgr2=0x7f597c0c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f59835c0640 1 --2- 172.21.15.107:0/3532631830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f597c152040 0x7f597c0c4b80 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f5978009870 tx=0x7f597801c4b0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f59835c0640 1 -- 172.21.15.107:0/3532631830 shutdown_connections 2023-10-25T00:05:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f59835c0640 1 --2- 172.21.15.107:0/3532631830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f597c0c8100 0x7f597c0abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f59835c0640 1 --2- 172.21.15.107:0/3532631830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f597c152040 0x7f597c0c4b80 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f59835c0640 1 -- 172.21.15.107:0/3532631830 >> 172.21.15.107:0/3532631830 conn(0x7f597c0ca200 msgr2=0x7f597c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f59835c0640 1 -- 172.21.15.107:0/3532631830 shutdown_connections 2023-10-25T00:05:18.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.853+0000 7f59835c0640 1 -- 172.21.15.107:0/3532631830 wait complete. 2023-10-25T00:05:18.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f59835c0640 1 Processor -- start 2023-10-25T00:05:18.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f59835c0640 1 -- start start 2023-10-25T00:05:18.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f59835c0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f597c06a4b0 0x7f597c1ed570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f59835c0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f597c0c8100 0x7f597c144bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f59835c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f597c152040 0x7f597c143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f59835c0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c14da10 con 0x7f597c0c8100 2023-10-25T00:05:18.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f59835c0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c1edab0 con 0x7f597c06a4b0 2023-10-25T00:05:18.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f59835c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597c141b80 con 0x7f597c152040 2023-10-25T00:05:18.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5980b5b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f597c0c8100 0x7f597c144bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5980b5b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f597c0c8100 0x7f597c144bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49862/0 (socket says 172.21.15.107:49862) 2023-10-25T00:05:18.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5980b5b640 1 -- 172.21.15.107:0/77987986 learned_addr learned my addr 172.21.15.107:0/77987986 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:18.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f598135c640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f597c06a4b0 0x7f597c1ed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5981b5d640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f597c152040 0x7f597c143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5980b5b640 1 -- 172.21.15.107:0/77987986 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f597c152040 msgr2=0x7f597c143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5980b5b640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f597c152040 0x7f597c143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5980b5b640 1 -- 172.21.15.107:0/77987986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f597c06a4b0 msgr2=0x7f597c1ed570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:18.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5980b5b640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f597c06a4b0 0x7f597c1ed570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:18.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5980b5b640 1 -- 172.21.15.107:0/77987986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59780094d0 con 0x7f597c0c8100 2023-10-25T00:05:18.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5981b5d640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f597c152040 0x7f597c143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5980b5b640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f597c0c8100 0x7f597c144bc0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f596c00d5c0 tx=0x7f596c00da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5971ffb640 1 -- 172.21.15.107:0/77987986 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c0125f0 con 0x7f597c0c8100 2023-10-25T00:05:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5971ffb640 1 -- 172.21.15.107:0/77987986 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c012750 con 0x7f597c0c8100 2023-10-25T00:05:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f59835c0640 1 -- 172.21.15.107:0/77987986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597c14ce90 con 0x7f597c0c8100 2023-10-25T00:05:18.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5971ffb640 1 -- 172.21.15.107:0/77987986 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c012b00 con 0x7f597c0c8100 2023-10-25T00:05:18.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f59835c0640 1 -- 172.21.15.107:0/77987986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f597c14d450 con 0x7f597c0c8100 2023-10-25T00:05:18.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f59835c0640 1 -- 172.21.15.107:0/77987986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597c0746e0 con 0x7f597c0c8100 2023-10-25T00:05:18.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.857+0000 7f5971ffb640 1 -- 172.21.15.107:0/77987986 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f596c012c60 con 0x7f597c0c8100 2023-10-25T00:05:18.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.861+0000 7f5971ffb640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5960065e60 0x7f5960068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:18.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.861+0000 7f5971ffb640 1 -- 172.21.15.107:0/77987986 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7f596c0845d0 con 0x7f597c0c8100 2023-10-25T00:05:18.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.861+0000 7f598135c640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5960065e60 0x7f5960068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:18.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.861+0000 7f598135c640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5960065e60 0x7f5960068320 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f59780ae000 tx=0x7f59780a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:18.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:18.865+0000 7f5971ffb640 1 -- 172.21.15.107:0/77987986 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f596c051140 con 0x7f597c0c8100 2023-10-25T00:05:19.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.093+0000 7f59835c0640 1 -- 172.21.15.107:0/77987986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f597c141950 con 0x7f597c0c8100 2023-10-25T00:05:19.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.093+0000 7f5971ffb640 1 -- 172.21.15.107:0/77987986 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+4259 (secure 0 0 0) 0x7f596c054760 con 0x7f597c0c8100 2023-10-25T00:05:19.096 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:19.096 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:18.611450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:19.097 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 110 2023-10-25T00:05:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.093+0000 7f59835c0640 1 -- 172.21.15.107:0/77987986 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5960065e60 msgr2=0x7f5960068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.093+0000 7f59835c0640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5960065e60 0x7f5960068320 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f59780ae000 tx=0x7f59780a7000 comp rx=0 tx=0).stop 2023-10-25T00:05:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.093+0000 7f59835c0640 1 -- 172.21.15.107:0/77987986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f597c0c8100 msgr2=0x7f597c144bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.093+0000 7f59835c0640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f597c0c8100 0x7f597c144bc0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f596c00d5c0 tx=0x7f596c00da80 comp rx=0 tx=0).stop 2023-10-25T00:05:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.097+0000 7f59835c0640 1 -- 172.21.15.107:0/77987986 shutdown_connections 2023-10-25T00:05:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.097+0000 7f59835c0640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5960065e60 0x7f5960068320 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.097+0000 7f59835c0640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f597c152040 0x7f597c143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.097+0000 7f59835c0640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f597c0c8100 0x7f597c144bc0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.097+0000 7f59835c0640 1 --2- 172.21.15.107:0/77987986 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f597c06a4b0 0x7f597c1ed570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.097+0000 7f59835c0640 1 -- 172.21.15.107:0/77987986 >> 172.21.15.107:0/77987986 conn(0x7f597c0ca200 msgr2=0x7f597c0ada90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.097+0000 7f59835c0640 1 -- 172.21.15.107:0/77987986 shutdown_connections 2023-10-25T00:05:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:19.097+0000 7f59835c0640 1 -- 172.21.15.107:0/77987986 wait complete. 2023-10-25T00:05:19.120 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 110, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 109, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6022": { "addr": "172.21.15.110:6834/174863522", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 174863522, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 174863522, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6022, "incarnation": 106, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:05:16.770996+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6022 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 110, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5410": { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 110, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:05:18.611450+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5410 } } } ], "standbys": [] } 2023-10-25T00:05:21.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:21.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.257+0000 7fbd3f0ce640 1 Processor -- start 2023-10-25T00:05:21.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.257+0000 7fbd3f0ce640 1 -- start start 2023-10-25T00:05:21.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3f0ce640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd3814eb70 0x7fbd380c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:21.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3f0ce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd38070b00 0x7fbd381514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:21.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3f0ce640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbd38151a20 0x7fbd3806df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:21.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3f0ce640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd38071700 con 0x7fbd3814eb70 2023-10-25T00:05:21.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3f0ce640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd380c22f0 con 0x7fbd38151a20 2023-10-25T00:05:21.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3f0ce640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd380c25d0 con 0x7fbd38070b00 2023-10-25T00:05:21.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3ce6a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd3814eb70 0x7fbd380c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:21.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3ce6a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd3814eb70 0x7fbd380c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49878/0 (socket says 172.21.15.107:49878) 2023-10-25T00:05:21.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3ce6a640 1 -- 172.21.15.107:0/227548511 learned_addr learned my addr 172.21.15.107:0/227548511 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:21.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3d66b640 1 --2- 172.21.15.107:0/227548511 >> v2:172.21.15.110:3300/0 conn(0x7fbd38151a20 0x7fbd3806df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:21.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd37fff640 1 --2- 172.21.15.107:0/227548511 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd38070b00 0x7fbd381514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:21.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3ce6a640 1 -- 172.21.15.107:0/227548511 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd38070b00 msgr2=0x7fbd381514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:21.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3ce6a640 1 --2- 172.21.15.107:0/227548511 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd38070b00 0x7fbd381514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:21.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3ce6a640 1 -- 172.21.15.107:0/227548511 >> v2:172.21.15.110:3300/0 conn(0x7fbd38151a20 msgr2=0x7fbd3806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:21.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3ce6a640 1 --2- 172.21.15.107:0/227548511 >> v2:172.21.15.110:3300/0 conn(0x7fbd38151a20 0x7fbd3806df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:21.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.261+0000 7fbd3ce6a640 1 -- 172.21.15.107:0/227548511 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd380c2be0 con 0x7fbd3814eb70 2023-10-25T00:05:21.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3ce6a640 1 --2- 172.21.15.107:0/227548511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3814eb70 0x7fbd380c33e0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fbd38072a20 tx=0x7fbd2c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b253757a415ce00 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:21.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd377fe640 1 -- 172.21.15.107:0/227548511 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c0ab660 con 0x7fbd3814eb70 2023-10-25T00:05:21.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd377fe640 1 -- 172.21.15.107:0/227548511 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2c0ab7c0 con 0x7fbd3814eb70 2023-10-25T00:05:21.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd377fe640 1 -- 172.21.15.107:0/227548511 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c0abb50 con 0x7fbd3814eb70 2023-10-25T00:05:21.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/227548511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3814eb70 msgr2=0x7fbd380c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:21.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 --2- 172.21.15.107:0/227548511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3814eb70 0x7fbd380c33e0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fbd38072a20 tx=0x7fbd2c01c500 comp rx=0 tx=0).stop 2023-10-25T00:05:21.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/227548511 shutdown_connections 2023-10-25T00:05:21.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 --2- 172.21.15.107:0/227548511 >> v2:172.21.15.110:3300/0 conn(0x7fbd38151a20 0x7fbd3806df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:21.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 --2- 172.21.15.107:0/227548511 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd38070b00 0x7fbd381514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:21.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 --2- 172.21.15.107:0/227548511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd3814eb70 0x7fbd380c33e0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:21.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/227548511 >> 172.21.15.107:0/227548511 conn(0x7fbd380c39a0 msgr2=0x7fbd380c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:21.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/227548511 shutdown_connections 2023-10-25T00:05:21.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/227548511 wait complete. 2023-10-25T00:05:21.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 Processor -- start 2023-10-25T00:05:21.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 -- start start 2023-10-25T00:05:21.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd38070b00 0x7fbd3814d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:21.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd3814eb70 0x7fbd380c04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:21.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38151a20 0x7fbd3813dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:21.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd38142170 con 0x7fbd38151a20 2023-10-25T00:05:21.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3813f3e0 con 0x7fbd38070b00 2023-10-25T00:05:21.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3f0ce640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3814da00 con 0x7fbd3814eb70 2023-10-25T00:05:21.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3d66b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38151a20 0x7fbd3813dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:21.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3d66b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38151a20 0x7fbd3813dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49880/0 (socket says 172.21.15.107:49880) 2023-10-25T00:05:21.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3d66b640 1 -- 172.21.15.107:0/2986463750 learned_addr learned my addr 172.21.15.107:0/2986463750 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:21.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3ce6a640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd38070b00 0x7fbd3814d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:21.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd37fff640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd3814eb70 0x7fbd380c04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:21.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3d66b640 1 -- 172.21.15.107:0/2986463750 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd3814eb70 msgr2=0x7fbd380c04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:21.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3d66b640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd3814eb70 0x7fbd380c04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:21.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3d66b640 1 -- 172.21.15.107:0/2986463750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd38070b00 msgr2=0x7fbd3814d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:21.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3d66b640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd38070b00 0x7fbd3814d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:21.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3d66b640 1 -- 172.21.15.107:0/2986463750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c0094d0 con 0x7fbd38151a20 2023-10-25T00:05:21.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3ce6a640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd38070b00 0x7fbd3814d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:21.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.265+0000 7fbd3d66b640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38151a20 0x7fbd3813dc70 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000efc0 tx=0x7fbd3000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:21.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.269+0000 7fbd357fa640 1 -- 172.21.15.107:0/2986463750 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd30009ba0 con 0x7fbd38151a20 2023-10-25T00:05:21.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.269+0000 7fbd357fa640 1 -- 172.21.15.107:0/2986463750 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3000f040 con 0x7fbd38151a20 2023-10-25T00:05:21.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.269+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/2986463750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd3813f680 con 0x7fbd38151a20 2023-10-25T00:05:21.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.269+0000 7fbd357fa640 1 -- 172.21.15.107:0/2986463750 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd300146c0 con 0x7fbd38151a20 2023-10-25T00:05:21.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.269+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/2986463750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd381423c0 con 0x7fbd38151a20 2023-10-25T00:05:21.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.269+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/2986463750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd3813fba0 con 0x7fbd38151a20 2023-10-25T00:05:21.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.269+0000 7fbd357fa640 1 -- 172.21.15.107:0/2986463750 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbd30014820 con 0x7fbd38151a20 2023-10-25T00:05:21.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.269+0000 7fbd357fa640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd1c065e60 0x7fbd1c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:21.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.269+0000 7fbd357fa640 1 -- 172.21.15.107:0/2986463750 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7fbd30085050 con 0x7fbd38151a20 2023-10-25T00:05:21.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.269+0000 7fbd3ce6a640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd1c065e60 0x7fbd1c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:21.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.269+0000 7fbd3ce6a640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd1c065e60 0x7fbd1c068320 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c0af000 tx=0x7fbd2c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:21.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.273+0000 7fbd357fa640 1 -- 172.21.15.107:0/2986463750 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbd30051c50 con 0x7fbd38151a20 2023-10-25T00:05:21.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.477+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/2986463750 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd380746e0 con 0x7fbd38151a20 2023-10-25T00:05:21.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.477+0000 7fbd357fa640 1 -- 172.21.15.107:0/2986463750 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4260 (secure 0 0 0) 0x7fbd30018020 con 0x7fbd38151a20 2023-10-25T00:05:21.482 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:21.483 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:20.812179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:rejoin","state_seq":97,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:21.483 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 112 2023-10-25T00:05:21.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/2986463750 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd1c065e60 msgr2=0x7fbd1c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:21.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd1c065e60 0x7fbd1c068320 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c0af000 tx=0x7fbd2c008040 comp rx=0 tx=0).stop 2023-10-25T00:05:21.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/2986463750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38151a20 msgr2=0x7fbd3813dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:21.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38151a20 0x7fbd3813dc70 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000efc0 tx=0x7fbd3000c490 comp rx=0 tx=0).stop 2023-10-25T00:05:21.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/2986463750 shutdown_connections 2023-10-25T00:05:21.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd38151a20 0x7fbd3813dc70 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:21.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbd1c065e60 0x7fbd1c068320 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:21.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd3814eb70 0x7fbd380c04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:21.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 --2- 172.21.15.107:0/2986463750 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd38070b00 0x7fbd3814d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:21.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/2986463750 >> 172.21.15.107:0/2986463750 conn(0x7fbd380c39a0 msgr2=0x7fbd38150840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:21.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/2986463750 shutdown_connections 2023-10-25T00:05:21.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:21.481+0000 7fbd3f0ce640 1 -- 172.21.15.107:0/2986463750 wait complete. 2023-10-25T00:05:21.503 INFO:tasks.mds_thrash.fs.[b]:no change 2023-10-25T00:05:23.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:23.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f01457c8640 1 Processor -- start 2023-10-25T00:05:23.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f01457c8640 1 -- start start 2023-10-25T00:05:23.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f01457c8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f014014fd40 0x7f01400ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:23.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f01457c8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f014006ee40 0x7f01400aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:23.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f01457c8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014006f730 con 0x7f01400a9820 2023-10-25T00:05:23.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f01457c8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01400aa750 con 0x7f014014fd40 2023-10-25T00:05:23.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f01457c8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0140156a40 con 0x7f014006ee40 2023-10-25T00:05:23.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f013effd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f014014fd40 0x7f01400ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:23.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f013effd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f014014fd40 0x7f01400ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:48072/0 (socket says 172.21.15.107:48072) 2023-10-25T00:05:23.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f013effd640 1 -- 172.21.15.107:0/3722743467 learned_addr learned my addr 172.21.15.107:0/3722743467 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:23.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f013f7fe640 1 --2- 172.21.15.107:0/3722743467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f014006ee40 0x7f01400aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:23.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f013effd640 1 -- 172.21.15.107:0/3722743467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f014006ee40 msgr2=0x7f01400aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:23.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f013effd640 1 --2- 172.21.15.107:0/3722743467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f014006ee40 0x7f01400aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:23.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f013effd640 1 -- 172.21.15.107:0/3722743467 >> v1:172.21.15.110:6789/0 conn(0x7f01400a9820 legacy=0x7f01400c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:23.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.633+0000 7f013effd640 1 -- 172.21.15.107:0/3722743467 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0140156710 con 0x7f014014fd40 2023-10-25T00:05:23.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f013effd640 1 --2- 172.21.15.107:0/3722743467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f014014fd40 0x7f01400ab090 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0128009870 tx=0x7f012801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6203ee935839dc46 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f013dffb640 1 -- 172.21.15.107:0/3722743467 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01280ad640 con 0x7f014014fd40 2023-10-25T00:05:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f013dffb640 1 -- 172.21.15.107:0/3722743467 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01280ad7a0 con 0x7f014014fd40 2023-10-25T00:05:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f013dffb640 1 -- 172.21.15.107:0/3722743467 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01280adb30 con 0x7f014014fd40 2023-10-25T00:05:23.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 -- 172.21.15.107:0/3722743467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f014014fd40 msgr2=0x7f01400ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:23.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 --2- 172.21.15.107:0/3722743467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f014014fd40 0x7f01400ab090 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0128009870 tx=0x7f012801c3f0 comp rx=0 tx=0).stop 2023-10-25T00:05:23.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 -- 172.21.15.107:0/3722743467 shutdown_connections 2023-10-25T00:05:23.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 --2- 172.21.15.107:0/3722743467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f014006ee40 0x7f01400aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:23.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 --2- 172.21.15.107:0/3722743467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f014014fd40 0x7f01400ab090 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:23.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 -- 172.21.15.107:0/3722743467 >> 172.21.15.107:0/3722743467 conn(0x7f01400c2950 msgr2=0x7f01400c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:23.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 -- 172.21.15.107:0/3722743467 shutdown_connections 2023-10-25T00:05:23.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 -- 172.21.15.107:0/3722743467 wait complete. 2023-10-25T00:05:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 Processor -- start 2023-10-25T00:05:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 -- start start 2023-10-25T00:05:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f014006ee40 0x7f014014edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f014014fd40 0x7f0140145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0140146c20 0x7f0140146010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0140146550 con 0x7f014014fd40 2023-10-25T00:05:23.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01401466d0 con 0x7f014006ee40 2023-10-25T00:05:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f01457c8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0140146920 con 0x7f0140146c20 2023-10-25T00:05:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f013e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f014014fd40 0x7f0140145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f013e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f014014fd40 0x7f0140145ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44326/0 (socket says 172.21.15.107:44326) 2023-10-25T00:05:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f013e7fc640 1 -- 172.21.15.107:0/3741571270 learned_addr learned my addr 172.21.15.107:0/3741571270 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:23.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f013effd640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f014006ee40 0x7f014014edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.637+0000 7f013f7fe640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0140146c20 0x7f0140146010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f013e7fc640 1 -- 172.21.15.107:0/3741571270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0140146c20 msgr2=0x7f0140146010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f013e7fc640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0140146c20 0x7f0140146010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f013e7fc640 1 -- 172.21.15.107:0/3741571270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f014006ee40 msgr2=0x7f014014edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:23.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f013e7fc640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f014006ee40 0x7f014014edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:23.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f013e7fc640 1 -- 172.21.15.107:0/3741571270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01280094d0 con 0x7f014014fd40 2023-10-25T00:05:23.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f013e7fc640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f014014fd40 0x7f0140145ad0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f013000c450 tx=0x7f013000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:23.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f01377fe640 1 -- 172.21.15.107:0/3741571270 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01300052d0 con 0x7f014014fd40 2023-10-25T00:05:23.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f01377fe640 1 -- 172.21.15.107:0/3741571270 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0130005430 con 0x7f014014fd40 2023-10-25T00:05:23.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f01457c8640 1 -- 172.21.15.107:0/3741571270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0140148780 con 0x7f014014fd40 2023-10-25T00:05:23.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f01377fe640 1 -- 172.21.15.107:0/3741571270 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130007260 con 0x7f014014fd40 2023-10-25T00:05:23.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f01457c8640 1 -- 172.21.15.107:0/3741571270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0140148d40 con 0x7f014014fd40 2023-10-25T00:05:23.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f01457c8640 1 -- 172.21.15.107:0/3741571270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01400c24a0 con 0x7f014014fd40 2023-10-25T00:05:23.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f01377fe640 1 -- 172.21.15.107:0/3741571270 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f01300073c0 con 0x7f014014fd40 2023-10-25T00:05:23.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f01377fe640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0120065e90 0x7f0120068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:23.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f01377fe640 1 -- 172.21.15.107:0/3741571270 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7f01300839e0 con 0x7f014014fd40 2023-10-25T00:05:23.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f013effd640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0120065e90 0x7f0120068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:23.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.641+0000 7f013effd640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0120065e90 0x7f0120068350 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f01280af000 tx=0x7f01280a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:23.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.645+0000 7f01377fe640 1 -- 172.21.15.107:0/3741571270 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0130050690 con 0x7f014014fd40 2023-10-25T00:05:23.679 INFO:tasks.ceph.mds.d.smithi110.stderr:2023-10-25T00:05:23.676+0000 7f6f6604a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:05:23.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.837+0000 7f01457c8640 1 -- 172.21.15.107:0/3741571270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0140070840 con 0x7f014014fd40 2023-10-25T00:05:23.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01377fe640 1 -- 172.21.15.107:0/3741571270 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4260 (secure 0 0 0) 0x7f0130053cb0 con 0x7f014014fd40 2023-10-25T00:05:23.842 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:23.842 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 113 2023-10-25T00:05:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 -- 172.21.15.107:0/3741571270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0120065e90 msgr2=0x7f0120068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:23.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0120065e90 0x7f0120068350 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f01280af000 tx=0x7f01280a7070 comp rx=0 tx=0).stop 2023-10-25T00:05:23.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 -- 172.21.15.107:0/3741571270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f014014fd40 msgr2=0x7f0140145ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:23.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f014014fd40 0x7f0140145ad0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f013000c450 tx=0x7f013000c910 comp rx=0 tx=0).stop 2023-10-25T00:05:23.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 -- 172.21.15.107:0/3741571270 shutdown_connections 2023-10-25T00:05:23.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0120065e90 0x7f0120068350 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0140146c20 0x7f0140146010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f014014fd40 0x7f0140145ad0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 --2- 172.21.15.107:0/3741571270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f014006ee40 0x7f014014edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 -- 172.21.15.107:0/3741571270 >> 172.21.15.107:0/3741571270 conn(0x7f01400c2950 msgr2=0x7f01400bf280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 -- 172.21.15.107:0/3741571270 shutdown_connections 2023-10-25T00:05:23.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:23.841+0000 7f01457c8640 1 -- 172.21.15.107:0/3741571270 wait complete. 2023-10-25T00:05:23.861 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=6753 2023-10-25T00:05:23.862 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.a 2023-10-25T00:05:24.802 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2023-10-25T00:05:24.802 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:05:24.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:05:24.806 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:05:24.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:24.893 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:05:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.941+0000 7fbaa70f4640 1 Processor -- start 2023-10-25T00:05:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.941+0000 7fbaa70f4640 1 -- start start 2023-10-25T00:05:24.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa70f4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaa006f490 0x7fbaa014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:24.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa70f4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbaa0152b30 0x7fbaa0152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:24.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa70f4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa0155cd0 con 0x7fbaa014c470 2023-10-25T00:05:24.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa70f4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa01526e0 con 0x7fbaa0152b30 2023-10-25T00:05:24.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa70f4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa00c4380 con 0x7fbaa006f490 2023-10-25T00:05:24.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa5691640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbaa0152b30 0x7fbaa0152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:24.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa5691640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbaa0152b30 0x7fbaa0152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:48112/0 (socket says 172.21.15.107:48112) 2023-10-25T00:05:24.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa5691640 1 -- 172.21.15.107:0/736335630 learned_addr learned my addr 172.21.15.107:0/736335630 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:24.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fba9ffff640 1 --2- 172.21.15.107:0/736335630 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaa006f490 0x7fbaa014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:24.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa5691640 1 -- 172.21.15.107:0/736335630 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaa006f490 msgr2=0x7fbaa014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:24.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa5691640 1 --2- 172.21.15.107:0/736335630 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaa006f490 0x7fbaa014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:24.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa5691640 1 -- 172.21.15.107:0/736335630 >> v1:172.21.15.110:6789/0 conn(0x7fbaa014c470 legacy=0x7fbaa0071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:24.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa5691640 1 -- 172.21.15.107:0/736335630 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa00c48b0 con 0x7fbaa0152b30 2023-10-25T00:05:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fbaa5691640 1 --2- 172.21.15.107:0/736335630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaa0152b30 0x7fbaa0152fa0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fba98009e50 tx=0x7fba9801c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cd3b3b54a294931 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:24.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fba9f7fe640 1 -- 172.21.15.107:0/736335630 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba980a56a0 con 0x7fbaa0152b30 2023-10-25T00:05:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fba9f7fe640 1 -- 172.21.15.107:0/736335630 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba980a5800 con 0x7fbaa0152b30 2023-10-25T00:05:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.945+0000 7fba9f7fe640 1 -- 172.21.15.107:0/736335630 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba980a5b90 con 0x7fbaa0152b30 2023-10-25T00:05:24.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 -- 172.21.15.107:0/736335630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaa0152b30 msgr2=0x7fbaa0152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:24.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 --2- 172.21.15.107:0/736335630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaa0152b30 0x7fbaa0152fa0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fba98009e50 tx=0x7fba9801c8f0 comp rx=0 tx=0).stop 2023-10-25T00:05:24.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 -- 172.21.15.107:0/736335630 shutdown_connections 2023-10-25T00:05:24.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 --2- 172.21.15.107:0/736335630 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaa0152b30 0x7fbaa0152fa0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:24.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 --2- 172.21.15.107:0/736335630 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaa006f490 0x7fbaa014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:24.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 -- 172.21.15.107:0/736335630 >> 172.21.15.107:0/736335630 conn(0x7fbaa0070f60 msgr2=0x7fbaa0071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:24.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 -- 172.21.15.107:0/736335630 shutdown_connections 2023-10-25T00:05:24.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 -- 172.21.15.107:0/736335630 wait complete. 2023-10-25T00:05:24.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 Processor -- start 2023-10-25T00:05:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 -- start start 2023-10-25T00:05:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaa006f490 0x7fbaa01456a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:24.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaa014c470 0x7fbaa006b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:24.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaa0152b30 0x7fbaa00c14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:24.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa014c200 con 0x7fbaa014c470 2023-10-25T00:05:24.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa0147650 con 0x7fbaa0152b30 2023-10-25T00:05:24.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa006bd00 con 0x7fbaa006f490 2023-10-25T00:05:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaa014c470 0x7fbaa006b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaa014c470 0x7fbaa006b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44340/0 (socket says 172.21.15.107:44340) 2023-10-25T00:05:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9ffff640 1 -- 172.21.15.107:0/725236848 learned_addr learned my addr 172.21.15.107:0/725236848 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa4e90640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaa006f490 0x7fbaa01456a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:24.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa5691640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaa0152b30 0x7fbaa00c14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9ffff640 1 -- 172.21.15.107:0/725236848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaa006f490 msgr2=0x7fbaa01456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9ffff640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaa006f490 0x7fbaa01456a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9ffff640 1 -- 172.21.15.107:0/725236848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaa0152b30 msgr2=0x7fbaa00c14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9ffff640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaa0152b30 0x7fbaa00c14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:24.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9ffff640 1 -- 172.21.15.107:0/725236848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba98009b30 con 0x7fbaa014c470 2023-10-25T00:05:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa4e90640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaa006f490 0x7fbaa01456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:05:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9ffff640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaa014c470 0x7fbaa006b000 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00e960 tx=0x7fba8c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9d7fa640 1 -- 172.21.15.107:0/725236848 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c009690 con 0x7fbaa014c470 2023-10-25T00:05:24.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9d7fa640 1 -- 172.21.15.107:0/725236848 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c00ca00 con 0x7fbaa014c470 2023-10-25T00:05:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 -- 172.21.15.107:0/725236848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa0147370 con 0x7fbaa014c470 2023-10-25T00:05:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fba9d7fa640 1 -- 172.21.15.107:0/725236848 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c00cd70 con 0x7fbaa014c470 2023-10-25T00:05:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.949+0000 7fbaa70f4640 1 -- 172.21.15.107:0/725236848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa006b540 con 0x7fbaa014c470 2023-10-25T00:05:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.953+0000 7fbaa70f4640 1 -- 172.21.15.107:0/725236848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa006be40 con 0x7fbaa014c470 2023-10-25T00:05:24.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.953+0000 7fba9d7fa640 1 -- 172.21.15.107:0/725236848 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fba8c013470 con 0x7fbaa014c470 2023-10-25T00:05:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.953+0000 7fba9d7fa640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba84065e60 0x7fba84068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.953+0000 7fbaa4e90640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba84065e60 0x7fba84068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.953+0000 7fba9d7fa640 1 -- 172.21.15.107:0/725236848 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7fba8c084500 con 0x7fbaa014c470 2023-10-25T00:05:24.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.953+0000 7fbaa4e90640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba84065e60 0x7fba84068320 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fba94000f70 tx=0x7fba94009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:24.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:24.957+0000 7fba9d7fa640 1 -- 172.21.15.107:0/725236848 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fba8c051100 con 0x7fbaa014c470 2023-10-25T00:05:25.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.185+0000 7fbaa70f4640 1 -- 172.21.15.107:0/725236848 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbaa00746e0 con 0x7fbaa014c470 2023-10-25T00:05:25.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.185+0000 7fba9d7fa640 1 -- 172.21.15.107:0/725236848 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+4260 (secure 0 0 0) 0x7fba8c017090 con 0x7fbaa014c470 2023-10-25T00:05:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:25.191 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:25.191 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 113 2023-10-25T00:05:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 -- 172.21.15.107:0/725236848 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba84065e60 msgr2=0x7fba84068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba84065e60 0x7fba84068320 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fba94000f70 tx=0x7fba94009040 comp rx=0 tx=0).stop 2023-10-25T00:05:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 -- 172.21.15.107:0/725236848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaa014c470 msgr2=0x7fbaa006b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:25.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaa014c470 0x7fbaa006b000 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00e960 tx=0x7fba8c00ee20 comp rx=0 tx=0).stop 2023-10-25T00:05:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 -- 172.21.15.107:0/725236848 shutdown_connections 2023-10-25T00:05:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbaa0152b30 0x7fbaa00c14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbaa014c470 0x7fbaa006b000 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba84065e60 0x7fba84068320 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:25.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 --2- 172.21.15.107:0/725236848 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbaa006f490 0x7fbaa01456a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 -- 172.21.15.107:0/725236848 >> 172.21.15.107:0/725236848 conn(0x7fbaa0070f60 msgr2=0x7fbaa0150730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 -- 172.21.15.107:0/725236848 shutdown_connections 2023-10-25T00:05:25.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:25.189+0000 7fbaa70f4640 1 -- 172.21.15.107:0/725236848 wait complete. 2023-10-25T00:05:25.214 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-25T00:05:27.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:27.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbfc00640 1 Processor -- start 2023-10-25T00:05:27.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbfc00640 1 -- start start 2023-10-25T00:05:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbfc00640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5bb8070b00 0x7f5bb80c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbfc00640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bb806df30 0x7f5bb806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbfc00640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb80c22f0 con 0x7f5bb814eb70 2023-10-25T00:05:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbfc00640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb8071700 con 0x7f5bb8070b00 2023-10-25T00:05:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbfc00640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb80c25d0 con 0x7f5bb806df30 2023-10-25T00:05:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbd19b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5bb8070b00 0x7f5bb80c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbd19b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5bb8070b00 0x7f5bb80c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44342/0 (socket says 172.21.15.107:44342) 2023-10-25T00:05:27.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbd19b640 1 -- 172.21.15.107:0/4233740673 learned_addr learned my addr 172.21.15.107:0/4233740673 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:27.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbe19d640 1 --2- 172.21.15.107:0/4233740673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bb806df30 0x7f5bb806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:27.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbd19b640 1 -- 172.21.15.107:0/4233740673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bb806df30 msgr2=0x7f5bb806e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:27.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbd19b640 1 --2- 172.21.15.107:0/4233740673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bb806df30 0x7f5bb806e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:27.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbd19b640 1 -- 172.21.15.107:0/4233740673 >> v1:172.21.15.110:6789/0 conn(0x7f5bb814eb70 legacy=0x7f5bb81514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:27.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.357+0000 7f5bbd19b640 1 -- 172.21.15.107:0/4233740673 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb80c2be0 con 0x7f5bb8070b00 2023-10-25T00:05:27.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.361+0000 7f5bbd19b640 1 --2- 172.21.15.107:0/4233740673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb8070b00 0x7f5bb80c33e0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800bb10 tx=0x7f5ba801ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deed5ea6e810466e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:27.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.361+0000 7f5bbc99a640 1 -- 172.21.15.107:0/4233740673 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80a7c00 con 0x7f5bb8070b00 2023-10-25T00:05:27.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.361+0000 7f5bbc99a640 1 -- 172.21.15.107:0/4233740673 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba80a7d60 con 0x7f5bb8070b00 2023-10-25T00:05:27.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.361+0000 7f5bbc99a640 1 -- 172.21.15.107:0/4233740673 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80af660 con 0x7f5bb8070b00 2023-10-25T00:05:27.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4233740673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb8070b00 msgr2=0x7f5bb80c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:27.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 --2- 172.21.15.107:0/4233740673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb8070b00 0x7f5bb80c33e0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800bb10 tx=0x7f5ba801ec10 comp rx=0 tx=0).stop 2023-10-25T00:05:27.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4233740673 shutdown_connections 2023-10-25T00:05:27.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 --2- 172.21.15.107:0/4233740673 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bb806df30 0x7f5bb806e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:27.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 --2- 172.21.15.107:0/4233740673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb8070b00 0x7f5bb80c33e0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:27.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4233740673 >> 172.21.15.107:0/4233740673 conn(0x7f5bb80c39a0 msgr2=0x7f5bb80c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:27.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4233740673 shutdown_connections 2023-10-25T00:05:27.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4233740673 wait complete. 2023-10-25T00:05:27.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 Processor -- start 2023-10-25T00:05:27.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 -- start start 2023-10-25T00:05:27.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bb806df30 0x7f5bb81eca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:27.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb8070b00 0x7f5bb80c04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:27.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bb814eb70 0x7f5bb8145670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:27.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb81454e0 con 0x7f5bb8070b00 2023-10-25T00:05:27.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb81f0990 con 0x7f5bb806df30 2023-10-25T00:05:27.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb81f0ec0 con 0x7f5bb814eb70 2023-10-25T00:05:27.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbd19b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb8070b00 0x7f5bb80c04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:27.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbd19b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb8070b00 0x7f5bb80c04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44356/0 (socket says 172.21.15.107:44356) 2023-10-25T00:05:27.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbd19b640 1 -- 172.21.15.107:0/4248371162 learned_addr learned my addr 172.21.15.107:0/4248371162 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:27.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbe19d640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bb814eb70 0x7f5bb8145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:27.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbd99c640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bb806df30 0x7f5bb81eca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:27.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbd19b640 1 -- 172.21.15.107:0/4248371162 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bb814eb70 msgr2=0x7f5bb8145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:27.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbd19b640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bb814eb70 0x7f5bb8145670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:27.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbd19b640 1 -- 172.21.15.107:0/4248371162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bb806df30 msgr2=0x7f5bb81eca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:27.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbd19b640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bb806df30 0x7f5bb81eca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:27.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbd19b640 1 -- 172.21.15.107:0/4248371162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba800b770 con 0x7f5bb8070b00 2023-10-25T00:05:27.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbd19b640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb8070b00 0x7f5bb80c04f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f5ba801d2c0 tx=0x7f5ba801b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:27.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bae7fc640 1 -- 172.21.15.107:0/4248371162 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80ba660 con 0x7f5bb8070b00 2023-10-25T00:05:27.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bae7fc640 1 -- 172.21.15.107:0/4248371162 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba80ba7c0 con 0x7f5bb8070b00 2023-10-25T00:05:27.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4248371162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb81f0be0 con 0x7f5bb8070b00 2023-10-25T00:05:27.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bae7fc640 1 -- 172.21.15.107:0/4248371162 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80bab70 con 0x7f5bb8070b00 2023-10-25T00:05:27.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.365+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4248371162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb80aa1c0 con 0x7f5bb8070b00 2023-10-25T00:05:27.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.369+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4248371162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb80746e0 con 0x7f5bb8070b00 2023-10-25T00:05:27.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.369+0000 7f5bae7fc640 1 -- 172.21.15.107:0/4248371162 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5ba80c33b0 con 0x7f5bb8070b00 2023-10-25T00:05:27.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.369+0000 7f5bae7fc640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b9c065e60 0x7f5b9c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:27.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.369+0000 7f5bae7fc640 1 -- 172.21.15.107:0/4248371162 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7f5ba811f2e0 con 0x7f5bb8070b00 2023-10-25T00:05:27.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.369+0000 7f5bbd99c640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b9c065e60 0x7f5b9c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:27.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.369+0000 7f5bbd99c640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b9c065e60 0x7f5b9c068320 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4000f70 tx=0x7f5bb4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:27.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.373+0000 7f5bae7fc640 1 -- 172.21.15.107:0/4248371162 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5ba80ec000 con 0x7f5bb8070b00 2023-10-25T00:05:27.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.609+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4248371162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bb80aa3b0 con 0x7f5bb8070b00 2023-10-25T00:05:27.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.609+0000 7f5bae7fc640 1 -- 172.21.15.107:0/4248371162 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5017 (secure 0 0 0) 0x7f5ba80ef620 con 0x7f5bb8070b00 2023-10-25T00:05:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:27.614 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:27.615 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 114 2023-10-25T00:05:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4248371162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b9c065e60 msgr2=0x7f5b9c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b9c065e60 0x7f5b9c068320 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4000f70 tx=0x7f5bb4009040 comp rx=0 tx=0).stop 2023-10-25T00:05:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4248371162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb8070b00 msgr2=0x7f5bb80c04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb8070b00 0x7f5bb80c04f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f5ba801d2c0 tx=0x7f5ba801b9b0 comp rx=0 tx=0).stop 2023-10-25T00:05:27.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4248371162 shutdown_connections 2023-10-25T00:05:27.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b9c065e60 0x7f5b9c068320 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:27.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5bb814eb70 0x7f5bb8145670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:27.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb8070b00 0x7f5bb80c04f0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:27.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 --2- 172.21.15.107:0/4248371162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bb806df30 0x7f5bb81eca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:27.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4248371162 >> 172.21.15.107:0/4248371162 conn(0x7f5bb80c39a0 msgr2=0x7f5bb80c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:27.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4248371162 shutdown_connections 2023-10-25T00:05:27.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:27.613+0000 7f5bbfc00640 1 -- 172.21.15.107:0/4248371162 wait complete. 2023-10-25T00:05:27.633 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2023-10-25T00:05:27.634 INFO:tasks.mds_thrash.fs.[a]:waiting for 11 secs before thrashing 2023-10-25T00:05:31.863 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2023-10-25T00:05:31.864 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:05:31.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:05:31.867 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:05:31.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:31.951 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:05:32.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb13cc3640 1 Processor -- start 2023-10-25T00:05:32.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb13cc3640 1 -- start start 2023-10-25T00:05:32.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb13cc3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcb0c0c70e0 0x7fcb0c0bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:32.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb13cc3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c0c2130 0x7fcb0c0be4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:32.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb13cc3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c0c1ca0 con 0x7fcb0c156770 2023-10-25T00:05:32.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb13cc3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb0c0c27e0 con 0x7fcb0c0c70e0 2023-10-25T00:05:32.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb13cc3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c0c10c0 con 0x7fcb0c0c2130 2023-10-25T00:05:32.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb11a5f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcb0c156770 0x7fcb0c0ab7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49164/0 (socket says 172.21.15.107:49164) 2023-10-25T00:05:32.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb11a5f640 1 -- 172.21.15.107:0/2117346478 learned_addr learned my addr 172.21.15.107:0/2117346478 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:32.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb1125e640 1 --2- 172.21.15.107:0/2117346478 >> v2:172.21.15.110:3300/0 conn(0x7fcb0c0c70e0 0x7fcb0c0bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:32.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb12260640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c0c2130 0x7fcb0c0be4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:32.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035930643 0 0) 0x7fcb0c0c1ca0 con 0x7fcb0c156770 2023-10-25T00:05:32.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf8003530 con 0x7fcb0c156770 2023-10-25T00:05:32.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1266066703 0 0) 0x7fcaf8003530 con 0x7fcb0c156770 2023-10-25T00:05:32.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c0c2130 msgr2=0x7fcb0c0be4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:32.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c0c2130 0x7fcb0c0be4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:32.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 >> v2:172.21.15.110:3300/0 conn(0x7fcb0c0c70e0 msgr2=0x7fcb0c0bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:32.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 --2- 172.21.15.107:0/2117346478 >> v2:172.21.15.110:3300/0 conn(0x7fcb0c0c70e0 0x7fcb0c0bdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:32.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c0c13a0 con 0x7fcb0c156770 2023-10-25T00:05:32.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fcb08001680 con 0x7fcb0c156770 2023-10-25T00:05:32.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 >> v1:172.21.15.107:6789/0 conn(0x7fcb0c156770 legacy=0x7fcb0c0ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:32.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcaf8003980 0x7fcaf8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:32.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c0c2130 0x7fcaf80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:32.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcaf8004d10 0x7fcaf8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:32.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c0c10c0 con 0x7fcaf8004d10 2023-10-25T00:05:32.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf800b990 con 0x7fcaf8003980 2023-10-25T00:05:32.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:31.997+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf800bb50 con 0x7fcb0c0c2130 2023-10-25T00:05:32.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb12260640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcaf8004d10 0x7fcaf8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:32.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb11a5f640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcaf8003980 0x7fcaf8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:32.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb1125e640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c0c2130 0x7fcaf80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:32.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb12260640 1 -- 172.21.15.107:0/2117346478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c0c2130 msgr2=0x7fcaf80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:32.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb12260640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c0c2130 0x7fcaf80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:32.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb12260640 1 -- 172.21.15.107:0/2117346478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcaf8003980 msgr2=0x7fcaf8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:32.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb12260640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcaf8003980 0x7fcaf8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:32.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb12260640 1 -- 172.21.15.107:0/2117346478 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf8003530 con 0x7fcaf8004d10 2023-10-25T00:05:32.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb12260640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcaf8004d10 0x7fcaf8005190 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009fc0 tx=0x7fcb0401cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498bdfd6d5fac98a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:32.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb040a59e0 con 0x7fcaf8004d10 2023-10-25T00:05:32.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0400f040 con 0x7fcaf8004d10 2023-10-25T00:05:32.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb10a5d640 1 -- 172.21.15.107:0/2117346478 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb040a5d80 con 0x7fcaf8004d10 2023-10-25T00:05:32.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb13cc3640 1 -- 172.21.15.107:0/2117346478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcaf8004d10 msgr2=0x7fcaf8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:32.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb13cc3640 1 --2- 172.21.15.107:0/2117346478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcaf8004d10 0x7fcaf8005190 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009fc0 tx=0x7fcb0401cc30 comp rx=0 tx=0).stop 2023-10-25T00:05:32.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb12260640 1 -- 172.21.15.107:0/2117346478 reap_dead start 2023-10-25T00:05:32.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb13cc3640 1 -- 172.21.15.107:0/2117346478 shutdown_connections 2023-10-25T00:05:32.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb13cc3640 1 -- 172.21.15.107:0/2117346478 >> 172.21.15.107:0/2117346478 conn(0x7fcb0c0c65c0 msgr2=0x7fcb0c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:32.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb13cc3640 1 -- 172.21.15.107:0/2117346478 shutdown_connections 2023-10-25T00:05:32.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.001+0000 7fcb13cc3640 1 -- 172.21.15.107:0/2117346478 wait complete. 2023-10-25T00:05:32.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb13cc3640 1 Processor -- start 2023-10-25T00:05:32.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb13cc3640 1 -- start start 2023-10-25T00:05:32.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb13cc3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb0c1ec2f0 0x7fcb0c1ec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:32.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb13cc3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c159340 0x7fcb0c070450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:32.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb13cc3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb0c070990 0x7fcb0c157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:32.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb13cc3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c06f9b0 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb13cc3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c1597d0 con 0x7fcb0c070990 2023-10-25T00:05:32.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb13cc3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c06fc50 con 0x7fcb0c159340 2023-10-25T00:05:32.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb11a5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb0c1ec2f0 0x7fcb0c1ec6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:32.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb11a5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb0c1ec2f0 0x7fcb0c1ec6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44406/0 (socket says 172.21.15.107:44406) 2023-10-25T00:05:32.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb11a5f640 1 -- 172.21.15.107:0/421976652 learned_addr learned my addr 172.21.15.107:0/421976652 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:32.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb12260640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb0c070990 0x7fcb0c157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb1125e640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c159340 0x7fcb0c070450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb11a5f640 1 -- 172.21.15.107:0/421976652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c159340 msgr2=0x7fcb0c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb11a5f640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c159340 0x7fcb0c070450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb11a5f640 1 -- 172.21.15.107:0/421976652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb0c070990 msgr2=0x7fcb0c157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:32.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb11a5f640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb0c070990 0x7fcb0c157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:32.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb11a5f640 1 -- 172.21.15.107:0/421976652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c06ff30 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb12260640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb0c070990 0x7fcb0c157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:32.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb11a5f640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb0c1ec2f0 0x7fcb0c1ec6d0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fcb08001130 tx=0x7fcb08007570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:32.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb027fc640 1 -- 172.21.15.107:0/421976652 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0800f990 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb027fc640 1 -- 172.21.15.107:0/421976652 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0800faf0 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb13cc3640 1 -- 172.21.15.107:0/421976652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0c159ab0 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb027fc640 1 -- 172.21.15.107:0/421976652 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0800fea0 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb13cc3640 1 -- 172.21.15.107:0/421976652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0c1ecc10 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.005+0000 7fcb13cc3640 1 -- 172.21.15.107:0/421976652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0c070170 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.009+0000 7fcb027fc640 1 -- 172.21.15.107:0/421976652 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcb0801d460 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.009+0000 7fcb027fc640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcaf0065e60 0x7fcaf0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:32.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.009+0000 7fcb027fc640 1 -- 172.21.15.107:0/421976652 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7fcb08084490 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.009+0000 7fcb1125e640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcaf0065e60 0x7fcaf0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:32.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.009+0000 7fcb1125e640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcaf0065e60 0x7fcaf0068320 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c1e6960 tx=0x7fcafc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:32.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.013+0000 7fcb027fc640 1 -- 172.21.15.107:0/421976652 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fcb0c070170 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.233+0000 7fcb13cc3640 1 -- 172.21.15.107:0/421976652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb0c070170 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.233+0000 7fcb027fc640 1 -- 172.21.15.107:0/421976652 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5017 (secure 0 0 0) 0x7fcb0c070170 con 0x7fcb0c1ec2f0 2023-10-25T00:05:32.237 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:32.237 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 114 2023-10-25T00:05:32.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 -- 172.21.15.107:0/421976652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcaf0065e60 msgr2=0x7fcaf0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:32.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcaf0065e60 0x7fcaf0068320 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c1e6960 tx=0x7fcafc008040 comp rx=0 tx=0).stop 2023-10-25T00:05:32.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 -- 172.21.15.107:0/421976652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb0c1ec2f0 msgr2=0x7fcb0c1ec6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:32.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb0c1ec2f0 0x7fcb0c1ec6d0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fcb08001130 tx=0x7fcb08007570 comp rx=0 tx=0).stop 2023-10-25T00:05:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 -- 172.21.15.107:0/421976652 shutdown_connections 2023-10-25T00:05:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcb0c070990 0x7fcb0c157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcaf0065e60 0x7fcaf0068320 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcb0c159340 0x7fcb0c070450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 --2- 172.21.15.107:0/421976652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb0c1ec2f0 0x7fcb0c1ec6d0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 -- 172.21.15.107:0/421976652 >> 172.21.15.107:0/421976652 conn(0x7fcb0c0c65c0 msgr2=0x7fcb0c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 -- 172.21.15.107:0/421976652 shutdown_connections 2023-10-25T00:05:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:32.237+0000 7fcb13cc3640 1 -- 172.21.15.107:0/421976652 wait complete. 2023-10-25T00:05:32.257 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-25T00:05:34.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:34.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3e2b7640 1 Processor -- start 2023-10-25T00:05:34.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3e2b7640 1 -- start start 2023-10-25T00:05:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3e2b7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7a38065bc0 0x7f7a38158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3e2b7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7a3806a120 0x7f7a380aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3e2b7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3806cea0 con 0x7f7a38069d40 2023-10-25T00:05:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3e2b7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a380ac650 con 0x7f7a3806a120 2023-10-25T00:05:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3e2b7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a380a9e00 con 0x7f7a38065bc0 2023-10-25T00:05:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a37fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7a38069d40 0x7f7a380a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52386/0 (socket says 172.21.15.107:52386) 2023-10-25T00:05:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a37fff640 1 -- 172.21.15.107:0/823315834 learned_addr learned my addr 172.21.15.107:0/823315834 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:34.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3c854640 1 --2- 172.21.15.107:0/823315834 >> v2:172.21.15.107:3300/0 conn(0x7f7a3806a120 0x7f7a380aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a377fe640 1 --2- 172.21.15.107:0/823315834 >> v2:172.21.15.110:3300/0 conn(0x7f7a38065bc0 0x7f7a38158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a36ffd640 1 -- 172.21.15.107:0/823315834 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130891194 0 0) 0x7f7a3806cea0 con 0x7f7a38069d40 2023-10-25T00:05:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a36ffd640 1 -- 172.21.15.107:0/823315834 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a20003530 con 0x7f7a38069d40 2023-10-25T00:05:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3c854640 1 -- 172.21.15.107:0/823315834 >> v2:172.21.15.110:3300/0 conn(0x7f7a38065bc0 msgr2=0x7f7a38158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:34.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3c854640 1 --2- 172.21.15.107:0/823315834 >> v2:172.21.15.110:3300/0 conn(0x7f7a38065bc0 0x7f7a38158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:34.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3c854640 1 -- 172.21.15.107:0/823315834 >> v1:172.21.15.107:6789/0 conn(0x7f7a38069d40 legacy=0x7f7a380a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:34.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.393+0000 7f7a3c854640 1 -- 172.21.15.107:0/823315834 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a381584e0 con 0x7f7a3806a120 2023-10-25T00:05:34.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a3c854640 1 --2- 172.21.15.107:0/823315834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3806a120 0x7f7a380aaef0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f7a38066170 tx=0x7f7a3001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af2338d5a64fec1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:34.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a36ffd640 1 -- 172.21.15.107:0/823315834 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a300a5830 con 0x7f7a3806a120 2023-10-25T00:05:34.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a36ffd640 1 -- 172.21.15.107:0/823315834 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a300a5990 con 0x7f7a3806a120 2023-10-25T00:05:34.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a36ffd640 1 -- 172.21.15.107:0/823315834 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a300a5d00 con 0x7f7a3806a120 2023-10-25T00:05:34.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/823315834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3806a120 msgr2=0x7f7a380aaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:34.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a3e2b7640 1 --2- 172.21.15.107:0/823315834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3806a120 0x7f7a380aaef0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f7a38066170 tx=0x7f7a3001c880 comp rx=0 tx=0).stop 2023-10-25T00:05:34.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/823315834 shutdown_connections 2023-10-25T00:05:34.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a3e2b7640 1 --2- 172.21.15.107:0/823315834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a3806a120 0x7f7a380aaef0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:34.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a3e2b7640 1 --2- 172.21.15.107:0/823315834 >> v2:172.21.15.110:3300/0 conn(0x7f7a38065bc0 0x7f7a38158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:34.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/823315834 >> 172.21.15.107:0/823315834 conn(0x7f7a380c65c0 msgr2=0x7f7a380c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:34.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/823315834 shutdown_connections 2023-10-25T00:05:34.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.397+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/823315834 wait complete. 2023-10-25T00:05:34.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3e2b7640 1 Processor -- start 2023-10-25T00:05:34.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3e2b7640 1 -- start start 2023-10-25T00:05:34.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3e2b7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a38065bc0 0x7f7a381546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:34.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3e2b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a38069d40 0x7f7a381ed910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:34.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3e2b7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3806a120 0x7f7a380bc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:34.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3e2b7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a38152660 con 0x7f7a38069d40 2023-10-25T00:05:34.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3e2b7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a381528b0 con 0x7f7a38065bc0 2023-10-25T00:05:34.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3e2b7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a38152b90 con 0x7f7a3806a120 2023-10-25T00:05:34.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a38069d40 0x7f7a381ed910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:34.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a38069d40 0x7f7a381ed910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44540/0 (socket says 172.21.15.107:44540) 2023-10-25T00:05:34.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a377fe640 1 -- 172.21.15.107:0/2566185635 learned_addr learned my addr 172.21.15.107:0/2566185635 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:34.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a37fff640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a38065bc0 0x7f7a381546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:34.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3c854640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3806a120 0x7f7a380bc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:34.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a377fe640 1 -- 172.21.15.107:0/2566185635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3806a120 msgr2=0x7f7a380bc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:34.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a377fe640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3806a120 0x7f7a380bc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:34.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a377fe640 1 -- 172.21.15.107:0/2566185635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a38065bc0 msgr2=0x7f7a381546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:34.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a377fe640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a38065bc0 0x7f7a381546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:34.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a377fe640 1 -- 172.21.15.107:0/2566185635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a30009a70 con 0x7f7a38069d40 2023-10-25T00:05:34.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a377fe640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a38069d40 0x7f7a381ed910 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00e870 tx=0x7f7a2c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:34.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a34ff9640 1 -- 172.21.15.107:0/2566185635 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c009510 con 0x7f7a38069d40 2023-10-25T00:05:34.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/2566185635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a380baa10 con 0x7f7a38069d40 2023-10-25T00:05:34.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a34ff9640 1 -- 172.21.15.107:0/2566185635 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c00eec0 con 0x7f7a38069d40 2023-10-25T00:05:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a34ff9640 1 -- 172.21.15.107:0/2566185635 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c00cb40 con 0x7f7a38069d40 2023-10-25T00:05:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/2566185635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a38152e70 con 0x7f7a38069d40 2023-10-25T00:05:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.401+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/2566185635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a380746e0 con 0x7f7a38069d40 2023-10-25T00:05:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.405+0000 7f7a34ff9640 1 -- 172.21.15.107:0/2566185635 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7a2c00ccc0 con 0x7f7a38069d40 2023-10-25T00:05:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.405+0000 7f7a34ff9640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a24065e60 0x7f7a24068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.405+0000 7f7a34ff9640 1 -- 172.21.15.107:0/2566185635 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7f7a2c084430 con 0x7f7a38069d40 2023-10-25T00:05:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.405+0000 7f7a37fff640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a24065e60 0x7f7a24068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:34.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.405+0000 7f7a37fff640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a24065e60 0x7f7a24068320 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7a28005890 tx=0x7f7a28009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:34.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.409+0000 7f7a34ff9640 1 -- 172.21.15.107:0/2566185635 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7a2c050fb0 con 0x7f7a38069d40 2023-10-25T00:05:34.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.661+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/2566185635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a3806acc0 con 0x7f7a38069d40 2023-10-25T00:05:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a34ff9640 1 -- 172.21.15.107:0/2566185635 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5775 (secure 0 0 0) 0x7f7a2c0545d0 con 0x7f7a38069d40 2023-10-25T00:05:34.667 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:34.668 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:34.668 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 115 2023-10-25T00:05:34.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/2566185635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a24065e60 msgr2=0x7f7a24068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:34.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a24065e60 0x7f7a24068320 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f7a28005890 tx=0x7f7a28009040 comp rx=0 tx=0).stop 2023-10-25T00:05:34.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/2566185635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a38069d40 msgr2=0x7f7a381ed910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a38069d40 0x7f7a381ed910 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00e870 tx=0x7f7a2c00ec70 comp rx=0 tx=0).stop 2023-10-25T00:05:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/2566185635 shutdown_connections 2023-10-25T00:05:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7a24065e60 0x7f7a24068320 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a3806a120 0x7f7a380bc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:34.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a38069d40 0x7f7a381ed910 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:34.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 --2- 172.21.15.107:0/2566185635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a38065bc0 0x7f7a381546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:34.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/2566185635 >> 172.21.15.107:0/2566185635 conn(0x7f7a380c65c0 msgr2=0x7f7a381567e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:34.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/2566185635 shutdown_connections 2023-10-25T00:05:34.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:34.665+0000 7f7a3e2b7640 1 -- 172.21.15.107:0/2566185635 wait complete. 2023-10-25T00:05:34.688 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2023-10-25T00:05:34.688 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2023-10-25T00:05:38.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:38.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0c947640 1 Processor -- start 2023-10-25T00:05:38.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0c947640 1 -- start start 2023-10-25T00:05:38.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0c947640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9c08071ce0 0x7f9c080ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:38.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0c947640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c080720c0 0x7f9c0813d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:38.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0c947640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9c08154860 0x7f9c08158230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:38.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0c947640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0806d900 con 0x7f9c08071ce0 2023-10-25T00:05:38.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0c947640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0806e710 con 0x7f9c08154860 2023-10-25T00:05:38.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0c947640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c08154d20 con 0x7f9c080720c0 2023-10-25T00:05:38.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9c08071ce0 0x7f9c080ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:38.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9c08071ce0 0x7f9c080ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44548/0 (socket says 172.21.15.107:44548) 2023-10-25T00:05:38.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0659c640 1 -- 172.21.15.107:0/4067648854 learned_addr learned my addr 172.21.15.107:0/4067648854 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:38.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c06d9d640 1 --2- 172.21.15.107:0/4067648854 >> v2:172.21.15.110:3300/0 conn(0x7f9c08154860 0x7f9c08158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:38.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c05d9b640 1 --2- 172.21.15.107:0/4067648854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c080720c0 0x7f9c0813d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:38.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0659c640 1 -- 172.21.15.107:0/4067648854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c080720c0 msgr2=0x7f9c0813d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:38.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0659c640 1 --2- 172.21.15.107:0/4067648854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c080720c0 0x7f9c0813d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:38.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0659c640 1 -- 172.21.15.107:0/4067648854 >> v2:172.21.15.110:3300/0 conn(0x7f9c08154860 msgr2=0x7f9c08158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:38.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0659c640 1 --2- 172.21.15.107:0/4067648854 >> v2:172.21.15.110:3300/0 conn(0x7f9c08154860 0x7f9c08158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:38.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c0659c640 1 -- 172.21.15.107:0/4067648854 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c08155000 con 0x7f9c08071ce0 2023-10-25T00:05:38.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.785+0000 7f9c05d9b640 1 --2- 172.21.15.107:0/4067648854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c080720c0 0x7f9c0813d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0659c640 1 --2- 172.21.15.107:0/4067648854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c08071ce0 0x7f9c080ad860 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f9c08072a90 tx=0x7f9bf801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2583c27cc49c9744 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0559a640 1 -- 172.21.15.107:0/4067648854 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf80ab600 con 0x7f9c08071ce0 2023-10-25T00:05:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0559a640 1 -- 172.21.15.107:0/4067648854 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf80ab760 con 0x7f9c08071ce0 2023-10-25T00:05:38.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0559a640 1 -- 172.21.15.107:0/4067648854 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf80abaf0 con 0x7f9c08071ce0 2023-10-25T00:05:38.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0c947640 1 -- 172.21.15.107:0/4067648854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c08071ce0 msgr2=0x7f9c080ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:38.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0c947640 1 --2- 172.21.15.107:0/4067648854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c08071ce0 0x7f9c080ad860 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f9c08072a90 tx=0x7f9bf801c440 comp rx=0 tx=0).stop 2023-10-25T00:05:38.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0c947640 1 -- 172.21.15.107:0/4067648854 shutdown_connections 2023-10-25T00:05:38.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0c947640 1 --2- 172.21.15.107:0/4067648854 >> v2:172.21.15.110:3300/0 conn(0x7f9c08154860 0x7f9c08158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:38.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0c947640 1 --2- 172.21.15.107:0/4067648854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c080720c0 0x7f9c0813d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:38.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0c947640 1 --2- 172.21.15.107:0/4067648854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c08071ce0 0x7f9c080ad860 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:38.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0c947640 1 -- 172.21.15.107:0/4067648854 >> 172.21.15.107:0/4067648854 conn(0x7f9c08070f60 msgr2=0x7f9c08071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:38.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0c947640 1 -- 172.21.15.107:0/4067648854 shutdown_connections 2023-10-25T00:05:38.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0c947640 1 -- 172.21.15.107:0/4067648854 wait complete. 2023-10-25T00:05:38.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0c947640 1 Processor -- start 2023-10-25T00:05:38.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.789+0000 7f9c0c947640 1 -- start start 2023-10-25T00:05:38.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0c947640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c08071ce0 0x7f9c080c1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:38.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0c947640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c080720c0 0x7f9c081510f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:38.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0c947640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c08154860 0x7f9c080be980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:38.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0c947640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c081ed810 con 0x7f9c080720c0 2023-10-25T00:05:38.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0c947640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c080bff10 con 0x7f9c08154860 2023-10-25T00:05:38.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0c947640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c08151f40 con 0x7f9c08071ce0 2023-10-25T00:05:38.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c05d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c080720c0 0x7f9c081510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:38.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c05d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c080720c0 0x7f9c081510f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44556/0 (socket says 172.21.15.107:44556) 2023-10-25T00:05:38.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c05d9b640 1 -- 172.21.15.107:0/3513523142 learned_addr learned my addr 172.21.15.107:0/3513523142 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:38.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c06d9d640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c08154860 0x7f9c080be980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:38.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0659c640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c08071ce0 0x7f9c080c1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:38.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c05d9b640 1 -- 172.21.15.107:0/3513523142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c08071ce0 msgr2=0x7f9c080c1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:38.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c05d9b640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c08071ce0 0x7f9c080c1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:38.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c05d9b640 1 -- 172.21.15.107:0/3513523142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c08154860 msgr2=0x7f9c080be980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:38.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c05d9b640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c08154860 0x7f9c080be980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:38.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c05d9b640 1 -- 172.21.15.107:0/3513523142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf80094d0 con 0x7f9c080720c0 2023-10-25T00:05:38.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0659c640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c08071ce0 0x7f9c080c1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:05:38.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c05d9b640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c080720c0 0x7f9c081510f0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00c7c0 tx=0x7f9bfc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:38.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9bf6ffd640 1 -- 172.21.15.107:0/3513523142 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc0079d0 con 0x7f9c080720c0 2023-10-25T00:05:38.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9bf6ffd640 1 -- 172.21.15.107:0/3513523142 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc007b30 con 0x7f9c080720c0 2023-10-25T00:05:38.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0c947640 1 -- 172.21.15.107:0/3513523142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c08152220 con 0x7f9c080720c0 2023-10-25T00:05:38.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9bf6ffd640 1 -- 172.21.15.107:0/3513523142 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc00ce70 con 0x7f9c080720c0 2023-10-25T00:05:38.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0c947640 1 -- 172.21.15.107:0/3513523142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c080c2e00 con 0x7f9c080720c0 2023-10-25T00:05:38.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0c947640 1 -- 172.21.15.107:0/3513523142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c08152410 con 0x7f9c080720c0 2023-10-25T00:05:38.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9bf6ffd640 1 -- 172.21.15.107:0/3513523142 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9bfc0113f0 con 0x7f9c080720c0 2023-10-25T00:05:38.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9bf6ffd640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9be8065e60 0x7f9be8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:38.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9bf6ffd640 1 -- 172.21.15.107:0/3513523142 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7f9bfc0842d0 con 0x7f9c080720c0 2023-10-25T00:05:38.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.793+0000 7f9c0659c640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9be8065e60 0x7f9be8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:38.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.797+0000 7f9c0659c640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9be8065e60 0x7f9be8068320 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9bf80af000 tx=0x7f9bf80a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:38.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:38.801+0000 7f9bf6ffd640 1 -- 172.21.15.107:0/3513523142 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9bfc050f80 con 0x7f9c080720c0 2023-10-25T00:05:39.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.029+0000 7f9c0c947640 1 -- 172.21.15.107:0/3513523142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c080746e0 con 0x7f9c080720c0 2023-10-25T00:05:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.029+0000 7f9bf6ffd640 1 -- 172.21.15.107:0/3513523142 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5775 (secure 0 0 0) 0x7f9bfc01f090 con 0x7f9c080720c0 2023-10-25T00:05:39.034 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:39.035 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:39.036 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 115 2023-10-25T00:05:39.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 -- 172.21.15.107:0/3513523142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9be8065e60 msgr2=0x7f9be8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:39.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9be8065e60 0x7f9be8068320 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9bf80af000 tx=0x7f9bf80a7000 comp rx=0 tx=0).stop 2023-10-25T00:05:39.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 -- 172.21.15.107:0/3513523142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c080720c0 msgr2=0x7f9c081510f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:39.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c080720c0 0x7f9c081510f0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00c7c0 tx=0x7f9bfc00cc80 comp rx=0 tx=0).stop 2023-10-25T00:05:39.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 -- 172.21.15.107:0/3513523142 shutdown_connections 2023-10-25T00:05:39.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9c08154860 0x7f9c080be980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c080720c0 0x7f9c081510f0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9be8065e60 0x7f9be8068320 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 --2- 172.21.15.107:0/3513523142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9c08071ce0 0x7f9c080c1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 -- 172.21.15.107:0/3513523142 >> 172.21.15.107:0/3513523142 conn(0x7f9c08070f60 msgr2=0x7f9c0806b870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 -- 172.21.15.107:0/3513523142 shutdown_connections 2023-10-25T00:05:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.033+0000 7f9c0c947640 1 -- 172.21.15.107:0/3513523142 wait complete. 2023-10-25T00:05:39.053 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2023-10-25T00:05:39.053 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-25T00:05:39.053 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:05:39.074 INFO:tasks.ceph.mds.b:Stopped 2023-10-25T00:05:39.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:39.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47ea03b640 1 Processor -- start 2023-10-25T00:05:39.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47ea03b640 1 -- start start 2023-10-25T00:05:39.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47ea03b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47e413bb00 0x7f47e41531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47ea03b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47e40aa330 0x7f47e406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47ea03b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e4152d40 con 0x7f47e40a9450 2023-10-25T00:05:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47ea03b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e4156100 con 0x7f47e40aa330 2023-10-25T00:05:39.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47ea03b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e41527c0 con 0x7f47e413bb00 2023-10-25T00:05:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47dbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f47e40aa330 0x7f47e406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47e3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f47e40a9450 0x7f47e4071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52396/0 (socket says 172.21.15.107:52396) 2023-10-25T00:05:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47e3fff640 1 -- 172.21.15.107:0/1525951889 learned_addr learned my addr 172.21.15.107:0/1525951889 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47e37fe640 1 --2- 172.21.15.107:0/1525951889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47e413bb00 0x7f47e41531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47dbfff640 1 -- 172.21.15.107:0/1525951889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47e413bb00 msgr2=0x7f47e41531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47dbfff640 1 --2- 172.21.15.107:0/1525951889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47e413bb00 0x7f47e41531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47dbfff640 1 -- 172.21.15.107:0/1525951889 >> v1:172.21.15.107:6789/0 conn(0x7f47e40a9450 legacy=0x7f47e4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47e2ffd640 1 -- 172.21.15.107:0/1525951889 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96265171 0 0) 0x7f47e4152d40 con 0x7f47e40a9450 2023-10-25T00:05:39.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.201+0000 7f47dbfff640 1 -- 172.21.15.107:0/1525951889 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e406d660 con 0x7f47e40aa330 2023-10-25T00:05:39.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47dbfff640 1 --2- 172.21.15.107:0/1525951889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e40aa330 0x7f47e406e250 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0097f0 tx=0x7f47cc01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e14bb0c4db5d79f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:39.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47e2ffd640 1 -- 172.21.15.107:0/1525951889 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc0ab600 con 0x7f47e40aa330 2023-10-25T00:05:39.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47e2ffd640 1 -- 172.21.15.107:0/1525951889 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47cc0a7050 con 0x7f47e40aa330 2023-10-25T00:05:39.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47e2ffd640 1 -- 172.21.15.107:0/1525951889 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc0ab980 con 0x7f47e40aa330 2023-10-25T00:05:39.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47ea03b640 1 -- 172.21.15.107:0/1525951889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e40aa330 msgr2=0x7f47e406e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:39.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47ea03b640 1 --2- 172.21.15.107:0/1525951889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e40aa330 0x7f47e406e250 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0097f0 tx=0x7f47cc01c360 comp rx=0 tx=0).stop 2023-10-25T00:05:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47ea03b640 1 -- 172.21.15.107:0/1525951889 shutdown_connections 2023-10-25T00:05:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47ea03b640 1 --2- 172.21.15.107:0/1525951889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e40aa330 0x7f47e406e250 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47ea03b640 1 --2- 172.21.15.107:0/1525951889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47e413bb00 0x7f47e41531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47ea03b640 1 -- 172.21.15.107:0/1525951889 >> 172.21.15.107:0/1525951889 conn(0x7f47e4070f60 msgr2=0x7f47e4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:39.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47ea03b640 1 -- 172.21.15.107:0/1525951889 shutdown_connections 2023-10-25T00:05:39.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.205+0000 7f47ea03b640 1 -- 172.21.15.107:0/1525951889 wait complete. 2023-10-25T00:05:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47ea03b640 1 Processor -- start 2023-10-25T00:05:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47ea03b640 1 -- start start 2023-10-25T00:05:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47ea03b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47e40a9450 0x7f47e41f0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47ea03b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47e40aa330 0x7f47e415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47ea03b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e413bb00 0x7f47e4130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47ea03b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e4132150 con 0x7f47e413bb00 2023-10-25T00:05:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47ea03b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e41efc30 con 0x7f47e40aa330 2023-10-25T00:05:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47ea03b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e41eff10 con 0x7f47e40a9450 2023-10-25T00:05:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e413bb00 0x7f47e4130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e413bb00 0x7f47e4130b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44566/0 (socket says 172.21.15.107:44566) 2023-10-25T00:05:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e3fff640 1 -- 172.21.15.107:0/2118190384 learned_addr learned my addr 172.21.15.107:0/2118190384 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e37fe640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47e40a9450 0x7f47e41f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47dbfff640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47e40aa330 0x7f47e415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e3fff640 1 -- 172.21.15.107:0/2118190384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47e40a9450 msgr2=0x7f47e41f0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:39.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e3fff640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47e40a9450 0x7f47e41f0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e3fff640 1 -- 172.21.15.107:0/2118190384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47e40aa330 msgr2=0x7f47e415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:39.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e3fff640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47e40aa330 0x7f47e415b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e3fff640 1 -- 172.21.15.107:0/2118190384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47cc009420 con 0x7f47e413bb00 2023-10-25T00:05:39.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e3fff640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e413bb00 0x7f47e4130b50 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c7a0 tx=0x7f47dc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:39.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e0ff9640 1 -- 172.21.15.107:0/2118190384 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc00a640 con 0x7f47e413bb00 2023-10-25T00:05:39.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e0ff9640 1 -- 172.21.15.107:0/2118190384 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc00a7a0 con 0x7f47e413bb00 2023-10-25T00:05:39.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47ea03b640 1 -- 172.21.15.107:0/2118190384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e41f01f0 con 0x7f47e413bb00 2023-10-25T00:05:39.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47ea03b640 1 -- 172.21.15.107:0/2118190384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e414fe10 con 0x7f47e413bb00 2023-10-25T00:05:39.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47e0ff9640 1 -- 172.21.15.107:0/2118190384 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc00abd0 con 0x7f47e413bb00 2023-10-25T00:05:39.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.209+0000 7f47ea03b640 1 -- 172.21.15.107:0/2118190384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c8005260 con 0x7f47e413bb00 2023-10-25T00:05:39.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.213+0000 7f47e0ff9640 1 -- 172.21.15.107:0/2118190384 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f47dc00ad30 con 0x7f47e413bb00 2023-10-25T00:05:39.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.213+0000 7f47e0ff9640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c4065e90 0x7f47c4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:39.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.213+0000 7f47e0ff9640 1 -- 172.21.15.107:0/2118190384 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7f47dc083c60 con 0x7f47e413bb00 2023-10-25T00:05:39.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.213+0000 7f47e37fe640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c4065e90 0x7f47c4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:39.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.213+0000 7f47e37fe640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c4065e90 0x7f47c4068350 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009a50 tx=0x7f47d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:39.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.217+0000 7f47e0ff9640 1 -- 172.21.15.107:0/2118190384 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f47dc050740 con 0x7f47e413bb00 2023-10-25T00:05:39.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.421+0000 7f47ea03b640 1 -- 172.21.15.107:0/2118190384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47c8005a90 con 0x7f47e413bb00 2023-10-25T00:05:39.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47e0ff9640 1 -- 172.21.15.107:0/2118190384 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5775 (secure 0 0 0) 0x7f47dc018020 con 0x7f47e413bb00 2023-10-25T00:05:39.427 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:39.427 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:39.427 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 115 2023-10-25T00:05:39.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 -- 172.21.15.107:0/2118190384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c4065e90 msgr2=0x7f47c4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:39.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c4065e90 0x7f47c4068350 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009a50 tx=0x7f47d4008040 comp rx=0 tx=0).stop 2023-10-25T00:05:39.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 -- 172.21.15.107:0/2118190384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e413bb00 msgr2=0x7f47e4130b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:39.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e413bb00 0x7f47e4130b50 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c7a0 tx=0x7f47dc00cc60 comp rx=0 tx=0).stop 2023-10-25T00:05:39.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 -- 172.21.15.107:0/2118190384 shutdown_connections 2023-10-25T00:05:39.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47e413bb00 0x7f47e4130b50 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47e40aa330 0x7f47e415b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f47c4065e90 0x7f47c4068350 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 --2- 172.21.15.107:0/2118190384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47e40a9450 0x7f47e41f0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 -- 172.21.15.107:0/2118190384 >> 172.21.15.107:0/2118190384 conn(0x7f47e4070f60 msgr2=0x7f47e41510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 -- 172.21.15.107:0/2118190384 shutdown_connections 2023-10-25T00:05:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:39.425+0000 7f47ea03b640 1 -- 172.21.15.107:0/2118190384 wait complete. 2023-10-25T00:05:39.449 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:05:41.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:41.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.577+0000 7fc8e7916640 1 Processor -- start 2023-10-25T00:05:41.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.577+0000 7fc8e7916640 1 -- start start 2023-10-25T00:05:41.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e7916640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e00a9820 0x7fc8e00ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:41.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e7916640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc8e006ee40 0x7fc8e00aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:41.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e7916640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e006f730 con 0x7fc8e014fd40 2023-10-25T00:05:41.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e7916640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8e00aa750 con 0x7fc8e006ee40 2023-10-25T00:05:41.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e7916640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e0156a40 con 0x7fc8e00a9820 2023-10-25T00:05:41.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e5eb3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc8e006ee40 0x7fc8e00aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:41.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e56b2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc8e014fd40 0x7fc8e00c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52398/0 (socket says 172.21.15.107:52398) 2023-10-25T00:05:41.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e56b2640 1 -- 172.21.15.107:0/2705385545 learned_addr learned my addr 172.21.15.107:0/2705385545 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:41.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e4eb1640 1 --2- 172.21.15.107:0/2705385545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e00a9820 0x7fc8e00ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:41.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e5eb3640 1 -- 172.21.15.107:0/2705385545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e00a9820 msgr2=0x7fc8e00ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:41.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e5eb3640 1 --2- 172.21.15.107:0/2705385545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e00a9820 0x7fc8e00ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:41.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e5eb3640 1 -- 172.21.15.107:0/2705385545 >> v1:172.21.15.107:6789/0 conn(0x7fc8e014fd40 legacy=0x7fc8e00c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:41.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8d7fff640 1 -- 172.21.15.107:0/2705385545 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665862019 0 0) 0x7fc8e006f730 con 0x7fc8e014fd40 2023-10-25T00:05:41.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e5eb3640 1 -- 172.21.15.107:0/2705385545 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e0156710 con 0x7fc8e006ee40 2023-10-25T00:05:41.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.581+0000 7fc8e4eb1640 1 --2- 172.21.15.107:0/2705385545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e00a9820 0x7fc8e00ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:05:41.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e5eb3640 1 --2- 172.21.15.107:0/2705385545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e006ee40 0x7fc8e00aa210 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009d90 tx=0x7fc8d801c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=890b56716a85696d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:41.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8d7fff640 1 -- 172.21.15.107:0/2705385545 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d80a5860 con 0x7fc8e006ee40 2023-10-25T00:05:41.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8d7fff640 1 -- 172.21.15.107:0/2705385545 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d80a59c0 con 0x7fc8e006ee40 2023-10-25T00:05:41.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8d7fff640 1 -- 172.21.15.107:0/2705385545 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d80a5d50 con 0x7fc8e006ee40 2023-10-25T00:05:41.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 -- 172.21.15.107:0/2705385545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e006ee40 msgr2=0x7fc8e00aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:41.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 --2- 172.21.15.107:0/2705385545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e006ee40 0x7fc8e00aa210 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009d90 tx=0x7fc8d801c800 comp rx=0 tx=0).stop 2023-10-25T00:05:41.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 -- 172.21.15.107:0/2705385545 shutdown_connections 2023-10-25T00:05:41.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 --2- 172.21.15.107:0/2705385545 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e006ee40 0x7fc8e00aa210 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:41.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 --2- 172.21.15.107:0/2705385545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e00a9820 0x7fc8e00ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:41.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 -- 172.21.15.107:0/2705385545 >> 172.21.15.107:0/2705385545 conn(0x7fc8e00c2950 msgr2=0x7fc8e00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:41.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 -- 172.21.15.107:0/2705385545 shutdown_connections 2023-10-25T00:05:41.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 -- 172.21.15.107:0/2705385545 wait complete. 2023-10-25T00:05:41.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 Processor -- start 2023-10-25T00:05:41.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 -- start start 2023-10-25T00:05:41.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e006ee40 0x7fc8e01498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:41.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e014fd40 0x7fc8e0155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:41.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e0154ac0 0x7fc8e0144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:41.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e0153e50 con 0x7fc8e014fd40 2023-10-25T00:05:41.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e0153960 con 0x7fc8e006ee40 2023-10-25T00:05:41.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e0149e30 con 0x7fc8e0154ac0 2023-10-25T00:05:41.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e4eb1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e014fd40 0x7fc8e0155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:41.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e4eb1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e014fd40 0x7fc8e0155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44582/0 (socket says 172.21.15.107:44582) 2023-10-25T00:05:41.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e4eb1640 1 -- 172.21.15.107:0/219630270 learned_addr learned my addr 172.21.15.107:0/219630270 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:41.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e56b2640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e006ee40 0x7fc8e01498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:41.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e5eb3640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e0154ac0 0x7fc8e0144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:41.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e4eb1640 1 -- 172.21.15.107:0/219630270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e0154ac0 msgr2=0x7fc8e0144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:41.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e4eb1640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e0154ac0 0x7fc8e0144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:41.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e4eb1640 1 -- 172.21.15.107:0/219630270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e006ee40 msgr2=0x7fc8e01498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:41.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e4eb1640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e006ee40 0x7fc8e01498f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:41.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e4eb1640 1 -- 172.21.15.107:0/219630270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d8009a70 con 0x7fc8e014fd40 2023-10-25T00:05:41.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e5eb3640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e0154ac0 0x7fc8e0144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:41.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e4eb1640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e014fd40 0x7fc8e0155380 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e840 tx=0x7fc8d000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:41.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8d5ffb640 1 -- 172.21.15.107:0/219630270 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000c870 con 0x7fc8e014fd40 2023-10-25T00:05:41.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8d5ffb640 1 -- 172.21.15.107:0/219630270 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d000c9d0 con 0x7fc8e014fd40 2023-10-25T00:05:41.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 -- 172.21.15.107:0/219630270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e01eeb00 con 0x7fc8e014fd40 2023-10-25T00:05:41.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8d5ffb640 1 -- 172.21.15.107:0/219630270 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000cd80 con 0x7fc8e014fd40 2023-10-25T00:05:41.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.585+0000 7fc8e7916640 1 -- 172.21.15.107:0/219630270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e0153b70 con 0x7fc8e014fd40 2023-10-25T00:05:41.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.589+0000 7fc8e7916640 1 -- 172.21.15.107:0/219630270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e00c24a0 con 0x7fc8e014fd40 2023-10-25T00:05:41.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.589+0000 7fc8d5ffb640 1 -- 172.21.15.107:0/219630270 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc8d0013410 con 0x7fc8e014fd40 2023-10-25T00:05:41.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.589+0000 7fc8d5ffb640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc8c4065e60 0x7fc8c4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:41.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.589+0000 7fc8d5ffb640 1 -- 172.21.15.107:0/219630270 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7fc8d0084010 con 0x7fc8e014fd40 2023-10-25T00:05:41.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.589+0000 7fc8e56b2640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc8c4065e60 0x7fc8c4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:41.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.589+0000 7fc8e56b2640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc8c4065e60 0x7fc8c4068320 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc000f70 tx=0x7fc8dc007c10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:41.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.593+0000 7fc8d5ffb640 1 -- 172.21.15.107:0/219630270 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc8d0050c10 con 0x7fc8e014fd40 2023-10-25T00:05:41.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.777+0000 7fc8e7916640 1 -- 172.21.15.107:0/219630270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8e0070840 con 0x7fc8e014fd40 2023-10-25T00:05:41.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8d5ffb640 1 -- 172.21.15.107:0/219630270 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5775 (secure 0 0 0) 0x7fc8d0054230 con 0x7fc8e014fd40 2023-10-25T00:05:41.782 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:41.782 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:41.783 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 115 2023-10-25T00:05:41.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 -- 172.21.15.107:0/219630270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc8c4065e60 msgr2=0x7fc8c4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:41.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc8c4065e60 0x7fc8c4068320 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc000f70 tx=0x7fc8dc007c10 comp rx=0 tx=0).stop 2023-10-25T00:05:41.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 -- 172.21.15.107:0/219630270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e014fd40 msgr2=0x7fc8e0155380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:41.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e014fd40 0x7fc8e0155380 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e840 tx=0x7fc8d000ed00 comp rx=0 tx=0).stop 2023-10-25T00:05:41.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 -- 172.21.15.107:0/219630270 shutdown_connections 2023-10-25T00:05:41.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc8c4065e60 0x7fc8c4068320 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8e0154ac0 0x7fc8e0144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8e014fd40 0x7fc8e0155380 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 --2- 172.21.15.107:0/219630270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8e006ee40 0x7fc8e01498f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 -- 172.21.15.107:0/219630270 >> 172.21.15.107:0/219630270 conn(0x7fc8e00c2950 msgr2=0x7fc8e0151e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 -- 172.21.15.107:0/219630270 shutdown_connections 2023-10-25T00:05:41.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:41.781+0000 7fc8e7916640 1 -- 172.21.15.107:0/219630270 wait complete. 2023-10-25T00:05:41.800 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:05:43.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:43.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.929+0000 7f14e752f640 1 Processor -- start 2023-10-25T00:05:43.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.929+0000 7f14e752f640 1 -- start start 2023-10-25T00:05:43.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e752f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14e00a9820 0x7f14e00ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:43.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e752f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f14e006ee40 0x7f14e00aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:43.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e752f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e006f730 con 0x7f14e014fd40 2023-10-25T00:05:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e752f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e00aa750 con 0x7f14e00a9820 2023-10-25T00:05:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e752f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e0156a40 con 0x7f14e006ee40 2023-10-25T00:05:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e4aca640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14e00a9820 0x7f14e00ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e4aca640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f14e00a9820 0x7f14e00ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35260/0 (socket says 172.21.15.107:35260) 2023-10-25T00:05:43.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e4aca640 1 -- 172.21.15.107:0/1382060872 learned_addr learned my addr 172.21.15.107:0/1382060872 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:43.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e5acc640 1 --2- 172.21.15.107:0/1382060872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f14e006ee40 0x7f14e00aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:43.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e4aca640 1 -- 172.21.15.107:0/1382060872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f14e006ee40 msgr2=0x7f14e00aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:43.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e4aca640 1 --2- 172.21.15.107:0/1382060872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f14e006ee40 0x7f14e00aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:43.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e4aca640 1 -- 172.21.15.107:0/1382060872 >> v1:172.21.15.110:6789/0 conn(0x7f14e014fd40 legacy=0x7f14e00c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:43.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e4aca640 1 -- 172.21.15.107:0/1382060872 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e0156710 con 0x7f14e00a9820 2023-10-25T00:05:43.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14e4aca640 1 --2- 172.21.15.107:0/1382060872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14e00a9820 0x7f14e00ab090 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009e50 tx=0x7f14dc01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21ea776cc91635c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:43.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14d7fff640 1 -- 172.21.15.107:0/1382060872 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc0a56a0 con 0x7f14e00a9820 2023-10-25T00:05:43.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14d7fff640 1 -- 172.21.15.107:0/1382060872 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14dc0a5800 con 0x7f14e00a9820 2023-10-25T00:05:43.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.933+0000 7f14d7fff640 1 -- 172.21.15.107:0/1382060872 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14dc0a5b90 con 0x7f14e00a9820 2023-10-25T00:05:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 -- 172.21.15.107:0/1382060872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14e00a9820 msgr2=0x7f14e00ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 --2- 172.21.15.107:0/1382060872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14e00a9820 0x7f14e00ab090 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f14dc009e50 tx=0x7f14dc01c880 comp rx=0 tx=0).stop 2023-10-25T00:05:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 -- 172.21.15.107:0/1382060872 shutdown_connections 2023-10-25T00:05:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 --2- 172.21.15.107:0/1382060872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f14e006ee40 0x7f14e00aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:43.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 --2- 172.21.15.107:0/1382060872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14e00a9820 0x7f14e00ab090 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:43.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 -- 172.21.15.107:0/1382060872 >> 172.21.15.107:0/1382060872 conn(0x7f14e00c2950 msgr2=0x7f14e00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:43.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 -- 172.21.15.107:0/1382060872 shutdown_connections 2023-10-25T00:05:43.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 -- 172.21.15.107:0/1382060872 wait complete. 2023-10-25T00:05:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 Processor -- start 2023-10-25T00:05:43.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 -- start start 2023-10-25T00:05:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14e006ee40 0x7f14e0149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f14e014fd40 0x7f14e014e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14e0144ac0 0x7f14e00c34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00c4920 con 0x7f14e0144ac0 2023-10-25T00:05:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e0149030 con 0x7f14e006ee40 2023-10-25T00:05:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e014f420 con 0x7f14e014fd40 2023-10-25T00:05:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e5acc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14e0144ac0 0x7f14e00c34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e5acc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14e0144ac0 0x7f14e00c34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35272/0 (socket says 172.21.15.107:35272) 2023-10-25T00:05:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e5acc640 1 -- 172.21.15.107:0/4229464616 learned_addr learned my addr 172.21.15.107:0/4229464616 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:43.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e4aca640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f14e014fd40 0x7f14e014e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e52cb640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14e006ee40 0x7f14e0149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e5acc640 1 -- 172.21.15.107:0/4229464616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f14e014fd40 msgr2=0x7f14e014e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e5acc640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f14e014fd40 0x7f14e014e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e5acc640 1 -- 172.21.15.107:0/4229464616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14e006ee40 msgr2=0x7f14e0149450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:43.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e5acc640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14e006ee40 0x7f14e0149450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e5acc640 1 -- 172.21.15.107:0/4229464616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc009b30 con 0x7f14e0144ac0 2023-10-25T00:05:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e52cb640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14e006ee40 0x7f14e0149450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:05:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e5acc640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14e0144ac0 0x7f14e00c34d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f14d8006e90 tx=0x7f14d800d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:43.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14d5ffb640 1 -- 172.21.15.107:0/4229464616 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d80125f0 con 0x7f14e0144ac0 2023-10-25T00:05:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14d5ffb640 1 -- 172.21.15.107:0/4229464616 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d8012750 con 0x7f14e0144ac0 2023-10-25T00:05:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 -- 172.21.15.107:0/4229464616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e014f140 con 0x7f14e0144ac0 2023-10-25T00:05:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14d5ffb640 1 -- 172.21.15.107:0/4229464616 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8012b00 con 0x7f14e0144ac0 2023-10-25T00:05:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.937+0000 7f14e752f640 1 -- 172.21.15.107:0/4229464616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e0148c50 con 0x7f14e0144ac0 2023-10-25T00:05:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.941+0000 7f14e752f640 1 -- 172.21.15.107:0/4229464616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e00c24a0 con 0x7f14e0144ac0 2023-10-25T00:05:43.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.941+0000 7f14d5ffb640 1 -- 172.21.15.107:0/4229464616 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f14d8012c60 con 0x7f14e0144ac0 2023-10-25T00:05:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.941+0000 7f14d5ffb640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f14c4065e90 0x7f14c4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.941+0000 7f14d5ffb640 1 -- 172.21.15.107:0/4229464616 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7f14d8084640 con 0x7f14e0144ac0 2023-10-25T00:05:43.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.941+0000 7f14e52cb640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f14c4065e90 0x7f14c4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.941+0000 7f14e52cb640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f14c4065e90 0x7f14c4068350 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f14d0003ec0 tx=0x7f14d0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:43.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:43.945+0000 7f14d5ffb640 1 -- 172.21.15.107:0/4229464616 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f14d8051240 con 0x7f14e0144ac0 2023-10-25T00:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 -- 172.21.15.107:0/4229464616 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14e0070840 con 0x7f14e0144ac0 2023-10-25T00:05:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14d5ffb640 1 -- 172.21.15.107:0/4229464616 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5775 (secure 0 0 0) 0x7f14d8054860 con 0x7f14e0144ac0 2023-10-25T00:05:44.144 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:44.144 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 115 2023-10-25T00:05:44.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 -- 172.21.15.107:0/4229464616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f14c4065e90 msgr2=0x7f14c4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:44.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f14c4065e90 0x7f14c4068350 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f14d0003ec0 tx=0x7f14d0009040 comp rx=0 tx=0).stop 2023-10-25T00:05:44.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 -- 172.21.15.107:0/4229464616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14e0144ac0 msgr2=0x7f14e00c34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:44.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14e0144ac0 0x7f14e00c34d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f14d8006e90 tx=0x7f14d800d9e0 comp rx=0 tx=0).stop 2023-10-25T00:05:44.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 -- 172.21.15.107:0/4229464616 shutdown_connections 2023-10-25T00:05:44.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14e0144ac0 0x7f14e00c34d0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:44.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f14c4065e90 0x7f14c4068350 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:44.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f14e014fd40 0x7f14e014e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:44.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 --2- 172.21.15.107:0/4229464616 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f14e006ee40 0x7f14e0149450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:44.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 -- 172.21.15.107:0/4229464616 >> 172.21.15.107:0/4229464616 conn(0x7f14e00c2950 msgr2=0x7f14e014a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:44.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 -- 172.21.15.107:0/4229464616 shutdown_connections 2023-10-25T00:05:44.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:44.141+0000 7f14e752f640 1 -- 172.21.15.107:0/4229464616 wait complete. 2023-10-25T00:05:44.162 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:05:46.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:46.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.285+0000 7f2f5e401640 1 Processor -- start 2023-10-25T00:05:46.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.285+0000 7f2f5e401640 1 -- start start 2023-10-25T00:05:46.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f5e401640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2f5806a780 0x7f2f58154d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:46.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f5e401640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f580c24b0 0x7f2f58155280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:46.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f5e401640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f581557c0 con 0x7f2f58151d60 2023-10-25T00:05:46.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f5e401640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f580aadc0 con 0x7f2f5806a780 2023-10-25T00:05:46.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f5e401640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5806afe0 con 0x7f2f580c24b0 2023-10-25T00:05:46.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f577fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2f5806a780 0x7f2f58154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:46.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f5c99e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f580c24b0 0x7f2f58155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:46.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f57fff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2f58151d60 0x7f2f580a9a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:36614/0 (socket says 172.21.15.107:36614) 2023-10-25T00:05:46.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f57fff640 1 -- 172.21.15.107:0/2796763930 learned_addr learned my addr 172.21.15.107:0/2796763930 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:46.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f5c99e640 1 -- 172.21.15.107:0/2796763930 >> v2:172.21.15.110:3300/0 conn(0x7f2f5806a780 msgr2=0x7f2f58154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:46.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f5c99e640 1 --2- 172.21.15.107:0/2796763930 >> v2:172.21.15.110:3300/0 conn(0x7f2f5806a780 0x7f2f58154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:46.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f5c99e640 1 -- 172.21.15.107:0/2796763930 >> v1:172.21.15.110:6789/0 conn(0x7f2f58151d60 legacy=0x7f2f580a9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:46.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.289+0000 7f2f5c99e640 1 -- 172.21.15.107:0/2796763930 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f58154840 con 0x7f2f580c24b0 2023-10-25T00:05:46.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5c99e640 1 --2- 172.21.15.107:0/2796763930 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f580c24b0 0x7f2f58155280 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009d90 tx=0x7f2f5001c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f66a17a784db4f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:46.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f56ffd640 1 -- 172.21.15.107:0/2796763930 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500a5810 con 0x7f2f580c24b0 2023-10-25T00:05:46.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f56ffd640 1 -- 172.21.15.107:0/2796763930 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f500a5970 con 0x7f2f580c24b0 2023-10-25T00:05:46.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f56ffd640 1 -- 172.21.15.107:0/2796763930 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500a5d00 con 0x7f2f580c24b0 2023-10-25T00:05:46.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 -- 172.21.15.107:0/2796763930 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f580c24b0 msgr2=0x7f2f58155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:46.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 --2- 172.21.15.107:0/2796763930 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f580c24b0 0x7f2f58155280 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009d90 tx=0x7f2f5001c950 comp rx=0 tx=0).stop 2023-10-25T00:05:46.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 -- 172.21.15.107:0/2796763930 shutdown_connections 2023-10-25T00:05:46.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 --2- 172.21.15.107:0/2796763930 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f580c24b0 0x7f2f58155280 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:46.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 --2- 172.21.15.107:0/2796763930 >> v2:172.21.15.110:3300/0 conn(0x7f2f5806a780 0x7f2f58154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:46.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 -- 172.21.15.107:0/2796763930 >> 172.21.15.107:0/2796763930 conn(0x7f2f580c2950 msgr2=0x7f2f580c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:46.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 -- 172.21.15.107:0/2796763930 shutdown_connections 2023-10-25T00:05:46.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 -- 172.21.15.107:0/2796763930 wait complete. 2023-10-25T00:05:46.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 Processor -- start 2023-10-25T00:05:46.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 -- start start 2023-10-25T00:05:46.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f580c24b0 0x7f2f581f0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:46.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f58151d60 0x7f2f58153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:46.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f58143560 0x7f2f581439f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:46.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f581f10e0 con 0x7f2f58151d60 2023-10-25T00:05:46.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f581515b0 con 0x7f2f580c24b0 2023-10-25T00:05:46.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5e401640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f58150490 con 0x7f2f58143560 2023-10-25T00:05:46.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f58151d60 0x7f2f58153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:46.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f58151d60 0x7f2f58153900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35282/0 (socket says 172.21.15.107:35282) 2023-10-25T00:05:46.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f577fe640 1 -- 172.21.15.107:0/3978477268 learned_addr learned my addr 172.21.15.107:0/3978477268 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:46.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f5c99e640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f58143560 0x7f2f581439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:46.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f57fff640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f580c24b0 0x7f2f581f0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:46.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f577fe640 1 -- 172.21.15.107:0/3978477268 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f58143560 msgr2=0x7f2f581439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:46.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f577fe640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f58143560 0x7f2f581439f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:46.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f577fe640 1 -- 172.21.15.107:0/3978477268 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f580c24b0 msgr2=0x7f2f581f0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:46.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f577fe640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f580c24b0 0x7f2f581f0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:46.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f577fe640 1 -- 172.21.15.107:0/3978477268 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f50009a70 con 0x7f2f58151d60 2023-10-25T00:05:46.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f577fe640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f58151d60 0x7f2f58153900 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800e930 tx=0x7f2f4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:46.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.293+0000 7f2f54ff9640 1 -- 172.21.15.107:0/3978477268 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f48009690 con 0x7f2f58151d60 2023-10-25T00:05:46.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.297+0000 7f2f54ff9640 1 -- 172.21.15.107:0/3978477268 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4800ca00 con 0x7f2f58151d60 2023-10-25T00:05:46.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.297+0000 7f2f5e401640 1 -- 172.21.15.107:0/3978477268 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f58150770 con 0x7f2f58151d60 2023-10-25T00:05:46.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.297+0000 7f2f54ff9640 1 -- 172.21.15.107:0/3978477268 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4800cd70 con 0x7f2f58151d60 2023-10-25T00:05:46.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.297+0000 7f2f5e401640 1 -- 172.21.15.107:0/3978477268 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f58150d30 con 0x7f2f58151d60 2023-10-25T00:05:46.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.297+0000 7f2f5e401640 1 -- 172.21.15.107:0/3978477268 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f581450a0 con 0x7f2f58151d60 2023-10-25T00:05:46.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.297+0000 7f2f54ff9640 1 -- 172.21.15.107:0/3978477268 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2f48013470 con 0x7f2f58151d60 2023-10-25T00:05:46.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.297+0000 7f2f54ff9640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f44065e60 0x7f2f44068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:46.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.297+0000 7f2f54ff9640 1 -- 172.21.15.107:0/3978477268 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7f2f480844b0 con 0x7f2f58151d60 2023-10-25T00:05:46.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.297+0000 7f2f57fff640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f44065e60 0x7f2f44068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:46.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.297+0000 7f2f57fff640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f44065e60 0x7f2f44068320 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c005890 tx=0x7f2f4c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:46.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.301+0000 7f2f54ff9640 1 -- 172.21.15.107:0/3978477268 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2f581450a0 con 0x7f2f58151d60 2023-10-25T00:05:46.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.485+0000 7f2f5e401640 1 -- 172.21.15.107:0/3978477268 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f581450a0 con 0x7f2f58151d60 2023-10-25T00:05:46.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.485+0000 7f2f54ff9640 1 -- 172.21.15.107:0/3978477268 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5775 (secure 0 0 0) 0x7f2f581450a0 con 0x7f2f58151d60 2023-10-25T00:05:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:46.489 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:46.489 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 115 2023-10-25T00:05:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 -- 172.21.15.107:0/3978477268 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f44065e60 msgr2=0x7f2f44068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:46.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f44065e60 0x7f2f44068320 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c005890 tx=0x7f2f4c009040 comp rx=0 tx=0).stop 2023-10-25T00:05:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 -- 172.21.15.107:0/3978477268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f58151d60 msgr2=0x7f2f58153900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f58151d60 0x7f2f58153900 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f2f4800e930 tx=0x7f2f4800edf0 comp rx=0 tx=0).stop 2023-10-25T00:05:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 -- 172.21.15.107:0/3978477268 shutdown_connections 2023-10-25T00:05:46.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f44065e60 0x7f2f44068320 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2f58143560 0x7f2f581439f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f58151d60 0x7f2f58153900 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 --2- 172.21.15.107:0/3978477268 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2f580c24b0 0x7f2f581f0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 -- 172.21.15.107:0/3978477268 >> 172.21.15.107:0/3978477268 conn(0x7f2f580c2950 msgr2=0x7f2f580c1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:46.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 -- 172.21.15.107:0/3978477268 shutdown_connections 2023-10-25T00:05:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:46.489+0000 7f2f5e401640 1 -- 172.21.15.107:0/3978477268 wait complete. 2023-10-25T00:05:46.505 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:05:48.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:48.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb792950640 1 Processor -- start 2023-10-25T00:05:48.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb792950640 1 -- start start 2023-10-25T00:05:48.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb792950640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c14b600 0x7fb78c151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:48.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb792950640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb78c06af70 0x7fb78c06c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:48.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb792950640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c0c1860 con 0x7fb78c06a230 2023-10-25T00:05:48.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb792950640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c154d50 con 0x7fb78c06af70 2023-10-25T00:05:48.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb792950640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1524f0 con 0x7fb78c14b600 2023-10-25T00:05:48.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb78b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb78c06af70 0x7fb78c06c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:48.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb78b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb78c06af70 0x7fb78c06c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35294/0 (socket says 172.21.15.107:35294) 2023-10-25T00:05:48.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb78b7fe640 1 -- 172.21.15.107:0/1122428077 learned_addr learned my addr 172.21.15.107:0/1122428077 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:48.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb78bfff640 1 --2- 172.21.15.107:0/1122428077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c14b600 0x7fb78c151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:48.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb78b7fe640 1 -- 172.21.15.107:0/1122428077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c14b600 msgr2=0x7fb78c151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb78b7fe640 1 --2- 172.21.15.107:0/1122428077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c14b600 0x7fb78c151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb78b7fe640 1 -- 172.21.15.107:0/1122428077 >> v1:172.21.15.110:6789/0 conn(0x7fb78c06a230 legacy=0x7fb78c1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb78b7fe640 1 -- 172.21.15.107:0/1122428077 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c14b9e0 con 0x7fb78c06af70 2023-10-25T00:05:48.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.633+0000 7fb78b7fe640 1 --2- 172.21.15.107:0/1122428077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c06af70 0x7fb78c06c4c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb780009870 tx=0x7fb78001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a921e8c7bdee1ad8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:48.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78affd640 1 -- 172.21.15.107:0/1122428077 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7800ab600 con 0x7fb78c06af70 2023-10-25T00:05:48.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78affd640 1 -- 172.21.15.107:0/1122428077 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7800ab760 con 0x7fb78c06af70 2023-10-25T00:05:48.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78affd640 1 -- 172.21.15.107:0/1122428077 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7800abaf0 con 0x7fb78c06af70 2023-10-25T00:05:48.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 -- 172.21.15.107:0/1122428077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c06af70 msgr2=0x7fb78c06c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:48.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 --2- 172.21.15.107:0/1122428077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c06af70 0x7fb78c06c4c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb780009870 tx=0x7fb78001c440 comp rx=0 tx=0).stop 2023-10-25T00:05:48.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 -- 172.21.15.107:0/1122428077 shutdown_connections 2023-10-25T00:05:48.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 --2- 172.21.15.107:0/1122428077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c06af70 0x7fb78c06c4c0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:48.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 --2- 172.21.15.107:0/1122428077 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c14b600 0x7fb78c151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:48.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 -- 172.21.15.107:0/1122428077 >> 172.21.15.107:0/1122428077 conn(0x7fb78c0c2950 msgr2=0x7fb78c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:48.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 -- 172.21.15.107:0/1122428077 shutdown_connections 2023-10-25T00:05:48.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 -- 172.21.15.107:0/1122428077 wait complete. 2023-10-25T00:05:48.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 Processor -- start 2023-10-25T00:05:48.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 -- start start 2023-10-25T00:05:48.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c06a230 0x7fb78c1f0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:48.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb78c06af70 0x7fb78c1f0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:48.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c14b600 0x7fb78c0c4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:48.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c14f9f0 con 0x7fb78c06a230 2023-10-25T00:05:48.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1f1050 con 0x7fb78c06af70 2023-10-25T00:05:48.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0c5bd0 con 0x7fb78c14b600 2023-10-25T00:05:48.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c06a230 0x7fb78c1f0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:48.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c06a230 0x7fb78c1f0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35304/0 (socket says 172.21.15.107:35304) 2023-10-25T00:05:48.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78bfff640 1 -- 172.21.15.107:0/2831247241 learned_addr learned my addr 172.21.15.107:0/2831247241 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:48.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78b7fe640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb78c06af70 0x7fb78c1f0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:48.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb790eed640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c14b600 0x7fb78c0c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:48.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78bfff640 1 -- 172.21.15.107:0/2831247241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c14b600 msgr2=0x7fb78c0c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:48.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78bfff640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c14b600 0x7fb78c0c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:48.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78bfff640 1 -- 172.21.15.107:0/2831247241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb78c06af70 msgr2=0x7fb78c1f0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:48.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78bfff640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb78c06af70 0x7fb78c1f0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:48.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78bfff640 1 -- 172.21.15.107:0/2831247241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7800094d0 con 0x7fb78c06a230 2023-10-25T00:05:48.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78b7fe640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb78c06af70 0x7fb78c1f0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:48.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb790eed640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c14b600 0x7fb78c0c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:48.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb78bfff640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c06a230 0x7fb78c1f0500 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fb78400c7f0 tx=0x7fb78400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:48.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb788ff9640 1 -- 172.21.15.107:0/2831247241 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784007820 con 0x7fb78c06a230 2023-10-25T00:05:48.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb788ff9640 1 -- 172.21.15.107:0/2831247241 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb784007980 con 0x7fb78c06a230 2023-10-25T00:05:48.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb788ff9640 1 -- 172.21.15.107:0/2831247241 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784007d30 con 0x7fb78c06a230 2023-10-25T00:05:48.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 -- 172.21.15.107:0/2831247241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c152040 con 0x7fb78c06a230 2023-10-25T00:05:48.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.637+0000 7fb792950640 1 -- 172.21.15.107:0/2831247241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c1519f0 con 0x7fb78c06a230 2023-10-25T00:05:48.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.641+0000 7fb792950640 1 -- 172.21.15.107:0/2831247241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c0746e0 con 0x7fb78c06a230 2023-10-25T00:05:48.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.641+0000 7fb788ff9640 1 -- 172.21.15.107:0/2831247241 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb7840113f0 con 0x7fb78c06a230 2023-10-25T00:05:48.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.641+0000 7fb788ff9640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e90 0x7fb778068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:48.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.641+0000 7fb78b7fe640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e90 0x7fb778068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:48.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.641+0000 7fb788ff9640 1 -- 172.21.15.107:0/2831247241 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7fb7840845d0 con 0x7fb78c06a230 2023-10-25T00:05:48.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.641+0000 7fb78b7fe640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e90 0x7fb778068350 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb7800af000 tx=0x7fb780008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:48.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.645+0000 7fb788ff9640 1 -- 172.21.15.107:0/2831247241 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb7840510c0 con 0x7fb78c06a230 2023-10-25T00:05:48.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.833+0000 7fb792950640 1 -- 172.21.15.107:0/2831247241 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb78c069fa0 con 0x7fb78c06a230 2023-10-25T00:05:48.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb788ff9640 1 -- 172.21.15.107:0/2831247241 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5775 (secure 0 0 0) 0x7fb7840546e0 con 0x7fb78c06a230 2023-10-25T00:05:48.838 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:48.839 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:48.839 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 115 2023-10-25T00:05:48.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 -- 172.21.15.107:0/2831247241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e90 msgr2=0x7fb778068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:48.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e90 0x7fb778068350 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb7800af000 tx=0x7fb780008040 comp rx=0 tx=0).stop 2023-10-25T00:05:48.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 -- 172.21.15.107:0/2831247241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c06a230 msgr2=0x7fb78c1f0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:48.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c06a230 0x7fb78c1f0500 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fb78400c7f0 tx=0x7fb78400ccb0 comp rx=0 tx=0).stop 2023-10-25T00:05:48.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 -- 172.21.15.107:0/2831247241 shutdown_connections 2023-10-25T00:05:48.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb778065e90 0x7fb778068350 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:48.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb78c14b600 0x7fb78c0c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:48.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb78c06af70 0x7fb78c1f0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:48.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 --2- 172.21.15.107:0/2831247241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb78c06a230 0x7fb78c1f0500 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:48.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 -- 172.21.15.107:0/2831247241 >> 172.21.15.107:0/2831247241 conn(0x7fb78c0c2950 msgr2=0x7fb78c06b390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:48.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 -- 172.21.15.107:0/2831247241 shutdown_connections 2023-10-25T00:05:48.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:48.837+0000 7fb792950640 1 -- 172.21.15.107:0/2831247241 wait complete. 2023-10-25T00:05:48.857 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:05:50.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:50.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.993+0000 7ff2cccdb640 1 Processor -- start 2023-10-25T00:05:50.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.993+0000 7ff2cccdb640 1 -- start start 2023-10-25T00:05:50.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2cccdb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2c813bb00 0x7ff2c81531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:50.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2cccdb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2c80aa330 0x7ff2c806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:50.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2cccdb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c8152d40 con 0x7ff2c80a9450 2023-10-25T00:05:50.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2cccdb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c8156100 con 0x7ff2c813bb00 2023-10-25T00:05:50.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2cccdb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81527c0 con 0x7ff2c80aa330 2023-10-25T00:05:50.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2c813bb00 0x7ff2c81531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:50.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2c813bb00 0x7ff2c81531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35310/0 (socket says 172.21.15.107:35310) 2023-10-25T00:05:50.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c659c640 1 -- 172.21.15.107:0/2934093900 learned_addr learned my addr 172.21.15.107:0/2934093900 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:50.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c5d9b640 1 --2- 172.21.15.107:0/2934093900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2c80aa330 0x7ff2c806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:50.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c659c640 1 -- 172.21.15.107:0/2934093900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2c80aa330 msgr2=0x7ff2c806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:50.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c659c640 1 --2- 172.21.15.107:0/2934093900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2c80aa330 0x7ff2c806e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:50.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c659c640 1 -- 172.21.15.107:0/2934093900 >> v1:172.21.15.110:6789/0 conn(0x7ff2c80a9450 legacy=0x7ff2c8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:50.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c659c640 1 -- 172.21.15.107:0/2934093900 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c806d660 con 0x7ff2c813bb00 2023-10-25T00:05:50.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c659c640 1 --2- 172.21.15.107:0/2934093900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2c813bb00 0x7ff2c81531d0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80098a0 tx=0x7ff2b801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b54ee39dc74e874 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:50.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c559a640 1 -- 172.21.15.107:0/2934093900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80ab600 con 0x7ff2c813bb00 2023-10-25T00:05:51.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c559a640 1 -- 172.21.15.107:0/2934093900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b80a7050 con 0x7ff2c813bb00 2023-10-25T00:05:51.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:50.997+0000 7ff2c559a640 1 -- 172.21.15.107:0/2934093900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80ab980 con 0x7ff2c813bb00 2023-10-25T00:05:51.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 -- 172.21.15.107:0/2934093900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2c813bb00 msgr2=0x7ff2c81531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:51.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 --2- 172.21.15.107:0/2934093900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2c813bb00 0x7ff2c81531d0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80098a0 tx=0x7ff2b801c3c0 comp rx=0 tx=0).stop 2023-10-25T00:05:51.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 -- 172.21.15.107:0/2934093900 shutdown_connections 2023-10-25T00:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 --2- 172.21.15.107:0/2934093900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2c80aa330 0x7ff2c806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 --2- 172.21.15.107:0/2934093900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2c813bb00 0x7ff2c81531d0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 -- 172.21.15.107:0/2934093900 >> 172.21.15.107:0/2934093900 conn(0x7ff2c8070f60 msgr2=0x7ff2c8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 -- 172.21.15.107:0/2934093900 shutdown_connections 2023-10-25T00:05:51.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 -- 172.21.15.107:0/2934093900 wait complete. 2023-10-25T00:05:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 Processor -- start 2023-10-25T00:05:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 -- start start 2023-10-25T00:05:51.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2c80a9450 0x7ff2c8155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:51.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2c80aa330 0x7ff2c80c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:51.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2c813bb00 0x7ff2c80c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:51.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8154d50 con 0x7ff2c80a9450 2023-10-25T00:05:51.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81548c0 con 0x7ff2c813bb00 2023-10-25T00:05:51.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8132c10 con 0x7ff2c80aa330 2023-10-25T00:05:51.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2c659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2c80a9450 0x7ff2c8155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:51.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2c659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2c80a9450 0x7ff2c8155490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35312/0 (socket says 172.21.15.107:35312) 2023-10-25T00:05:51.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2c659c640 1 -- 172.21.15.107:0/844316782 learned_addr learned my addr 172.21.15.107:0/844316782 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:51.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2c6d9d640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2c813bb00 0x7ff2c80c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:51.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2c5d9b640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2c80aa330 0x7ff2c80c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:51.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2c659c640 1 -- 172.21.15.107:0/844316782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2c80aa330 msgr2=0x7ff2c80c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:51.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2c659c640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2c80aa330 0x7ff2c80c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:51.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2c659c640 1 -- 172.21.15.107:0/844316782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2c813bb00 msgr2=0x7ff2c80c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:51.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2c659c640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2c813bb00 0x7ff2c80c2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:51.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2c659c640 1 -- 172.21.15.107:0/844316782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b80094d0 con 0x7ff2c80a9450 2023-10-25T00:05:51.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2c659c640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2c80a9450 0x7ff2c8155490 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff2b801aa70 tx=0x7ff2b801ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:51.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2b6ffd640 1 -- 172.21.15.107:0/844316782 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80b6830 con 0x7ff2c80a9450 2023-10-25T00:05:51.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2b6ffd640 1 -- 172.21.15.107:0/844316782 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b80abe90 con 0x7ff2c80a9450 2023-10-25T00:05:51.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 -- 172.21.15.107:0/844316782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c8153f90 con 0x7ff2c80a9450 2023-10-25T00:05:51.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2b6ffd640 1 -- 172.21.15.107:0/844316782 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80b6bf0 con 0x7ff2c80a9450 2023-10-25T00:05:51.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.001+0000 7ff2cccdb640 1 -- 172.21.15.107:0/844316782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c8154550 con 0x7ff2c80a9450 2023-10-25T00:05:51.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.005+0000 7ff2cccdb640 1 -- 172.21.15.107:0/844316782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c80746e0 con 0x7ff2c80a9450 2023-10-25T00:05:51.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.005+0000 7ff2b6ffd640 1 -- 172.21.15.107:0/844316782 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff2b80b2040 con 0x7ff2c80a9450 2023-10-25T00:05:51.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.005+0000 7ff2b6ffd640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2a8065e60 0x7ff2a8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:51.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.005+0000 7ff2b6ffd640 1 -- 172.21.15.107:0/844316782 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7ff2b811e900 con 0x7ff2c80a9450 2023-10-25T00:05:51.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.005+0000 7ff2c5d9b640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2a8065e60 0x7ff2a8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:51.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.005+0000 7ff2c5d9b640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2a8065e60 0x7ff2a8068320 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8130740 tx=0x7ff2bc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:51.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.009+0000 7ff2b6ffd640 1 -- 172.21.15.107:0/844316782 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff2b80eb500 con 0x7ff2c80a9450 2023-10-25T00:05:51.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.197+0000 7ff2cccdb640 1 -- 172.21.15.107:0/844316782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2c806f350 con 0x7ff2c80a9450 2023-10-25T00:05:51.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2b6ffd640 1 -- 172.21.15.107:0/844316782 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5775 (secure 0 0 0) 0x7ff2b80ba090 con 0x7ff2c80a9450 2023-10-25T00:05:51.199 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:51.199 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:51.200 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 115 2023-10-25T00:05:51.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 -- 172.21.15.107:0/844316782 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2a8065e60 msgr2=0x7ff2a8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:51.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2a8065e60 0x7ff2a8068320 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8130740 tx=0x7ff2bc008040 comp rx=0 tx=0).stop 2023-10-25T00:05:51.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 -- 172.21.15.107:0/844316782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2c80a9450 msgr2=0x7ff2c8155490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:51.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2c80a9450 0x7ff2c8155490 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff2b801aa70 tx=0x7ff2b801ac00 comp rx=0 tx=0).stop 2023-10-25T00:05:51.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 -- 172.21.15.107:0/844316782 shutdown_connections 2023-10-25T00:05:51.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2c813bb00 0x7ff2c80c2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:51.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2a8065e60 0x7ff2a8068320 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:51.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2c80aa330 0x7ff2c80c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:51.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 --2- 172.21.15.107:0/844316782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2c80a9450 0x7ff2c8155490 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:51.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 -- 172.21.15.107:0/844316782 >> 172.21.15.107:0/844316782 conn(0x7ff2c8070f60 msgr2=0x7ff2c8071340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:51.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 -- 172.21.15.107:0/844316782 shutdown_connections 2023-10-25T00:05:51.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:51.201+0000 7ff2cccdb640 1 -- 172.21.15.107:0/844316782 wait complete. 2023-10-25T00:05:51.217 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:05:53.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:53.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd299885640 1 Processor -- start 2023-10-25T00:05:53.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd299885640 1 -- start start 2023-10-25T00:05:53.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd299885640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd29414fd40 0x7fd2940ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:53.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd299885640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29406f730 con 0x7fd29406ee40 2023-10-25T00:05:53.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd299885640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd294156a40 con 0x7fd2940a9820 2023-10-25T00:05:53.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd299885640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd294156710 con 0x7fd29414fd40 2023-10-25T00:05:53.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2937fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd29406ee40 0x7fd29415cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40326/0 (socket says 172.21.15.107:40326) 2023-10-25T00:05:53.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2937fe640 1 -- 172.21.15.107:0/1937299991 learned_addr learned my addr 172.21.15.107:0/1937299991 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:53.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd292ffd640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd29414fd40 0x7fd2940ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:53.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199646033 0 0) 0x7fd29406f730 con 0x7fd29406ee40 2023-10-25T00:05:53.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd280003530 con 0x7fd29406ee40 2023-10-25T00:05:53.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 388891190 0 0) 0x7fd280003530 con 0x7fd29406ee40 2023-10-25T00:05:53.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd29414fd40 msgr2=0x7fd2940ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:53.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd29414fd40 0x7fd2940ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:53.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 >> v1:172.21.15.110:6789/0 conn(0x7fd2940a9820 legacy=0x7fd2940c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:53.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29406f980 con 0x7fd29406ee40 2023-10-25T00:05:53.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fd28c001a00 con 0x7fd29406ee40 2023-10-25T00:05:53.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 >> v1:172.21.15.107:6789/0 conn(0x7fd29406ee40 legacy=0x7fd29415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:53.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd29414fd40 0x7fd2800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:53.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd280003f30 0x7fd2800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:53.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd280004d10 0x7fd280005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:53.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd294156710 con 0x7fd280003f30 2023-10-25T00:05:53.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28000bbd0 con 0x7fd280004d10 2023-10-25T00:05:53.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28000bd90 con 0x7fd29414fd40 2023-10-25T00:05:53.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd28bfff640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd280003f30 0x7fd2800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:53.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd292ffd640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd29414fd40 0x7fd2800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:53.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2937fe640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd280004d10 0x7fd280005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:53.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd28bfff640 1 -- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd29414fd40 msgr2=0x7fd2800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:53.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd28bfff640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd29414fd40 0x7fd2800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:53.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd28bfff640 1 -- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd280004d10 msgr2=0x7fd280005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:53.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd28bfff640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd280004d10 0x7fd280005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:53.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd28bfff640 1 -- 172.21.15.107:0/1937299991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd280003530 con 0x7fd280003f30 2023-10-25T00:05:53.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.349+0000 7fd2937fe640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd280004d10 0x7fd280005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:53.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd28bfff640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd280003f30 0x7fd2800047d0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fd27c001780 tx=0x7fd27c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df37d21a96aa0243 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:53.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c0052d0 con 0x7fd280003f30 2023-10-25T00:05:53.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27c0a7050 con 0x7fd280003f30 2023-10-25T00:05:53.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2927fc640 1 -- 172.21.15.107:0/1937299991 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c003600 con 0x7fd280003f30 2023-10-25T00:05:53.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 -- 172.21.15.107:0/1937299991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd280003f30 msgr2=0x7fd2800047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:53.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 --2- 172.21.15.107:0/1937299991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd280003f30 0x7fd2800047d0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fd27c001780 tx=0x7fd27c01c400 comp rx=0 tx=0).stop 2023-10-25T00:05:53.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2937fe640 1 -- 172.21.15.107:0/1937299991 reap_dead start 2023-10-25T00:05:53.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 -- 172.21.15.107:0/1937299991 shutdown_connections 2023-10-25T00:05:53.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 -- 172.21.15.107:0/1937299991 >> 172.21.15.107:0/1937299991 conn(0x7fd2940c2950 msgr2=0x7fd2940c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:53.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 -- 172.21.15.107:0/1937299991 shutdown_connections 2023-10-25T00:05:53.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 -- 172.21.15.107:0/1937299991 wait complete. 2023-10-25T00:05:53.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 Processor -- start 2023-10-25T00:05:53.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 -- start start 2023-10-25T00:05:53.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd280003f30 0x7fd2941ec560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:53.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2941ecaa0 0x7fd294149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:53.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd29414a330 0x7fd294144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:53.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29414a7b0 con 0x7fd29414a330 2023-10-25T00:05:53.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2940c4b80 con 0x7fd280003f30 2023-10-25T00:05:53.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd299885640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd294145820 con 0x7fd2941ecaa0 2023-10-25T00:05:53.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd29414a330 0x7fd294144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:53.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd29414a330 0x7fd294144b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33602/0 (socket says 172.21.15.107:33602) 2023-10-25T00:05:53.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2937fe640 1 -- 172.21.15.107:0/712828754 learned_addr learned my addr 172.21.15.107:0/712828754 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:53.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd28bfff640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2941ecaa0 0x7fd294149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:53.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd292ffd640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd280003f30 0x7fd2941ec560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:53.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2937fe640 1 -- 172.21.15.107:0/712828754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2941ecaa0 msgr2=0x7fd294149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:53.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2937fe640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2941ecaa0 0x7fd294149df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:53.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2937fe640 1 -- 172.21.15.107:0/712828754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd280003f30 msgr2=0x7fd2941ec560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:53.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2937fe640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd280003f30 0x7fd2941ec560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:53.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.353+0000 7fd2937fe640 1 -- 172.21.15.107:0/712828754 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd294145b00 con 0x7fd29414a330 2023-10-25T00:05:53.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd2937fe640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd29414a330 0x7fd294144b30 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd29406d200 tx=0x7fd28c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:53.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd290ff9640 1 -- 172.21.15.107:0/712828754 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c001a80 con 0x7fd29414a330 2023-10-25T00:05:53.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd290ff9640 1 -- 172.21.15.107:0/712828754 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c00de70 con 0x7fd29414a330 2023-10-25T00:05:53.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd299885640 1 -- 172.21.15.107:0/712828754 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd294155a30 con 0x7fd29414a330 2023-10-25T00:05:53.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd290ff9640 1 -- 172.21.15.107:0/712828754 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c00bdc0 con 0x7fd29414a330 2023-10-25T00:05:53.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd299885640 1 -- 172.21.15.107:0/712828754 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2940c2d30 con 0x7fd29414a330 2023-10-25T00:05:53.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd299885640 1 -- 172.21.15.107:0/712828754 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2940c24a0 con 0x7fd29414a330 2023-10-25T00:05:53.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd290ff9640 1 -- 172.21.15.107:0/712828754 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd28c007110 con 0x7fd29414a330 2023-10-25T00:05:53.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd290ff9640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd274065e90 0x7fd274068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:53.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd290ff9640 1 -- 172.21.15.107:0/712828754 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 8767+0+0 (secure 0 0 0) 0x7fd28c084680 con 0x7fd29414a330 2023-10-25T00:05:53.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd292ffd640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd274065e90 0x7fd274068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:53.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.357+0000 7fd292ffd640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd274065e90 0x7fd274068350 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd284009a30 tx=0x7fd284008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:53.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.361+0000 7fd290ff9640 1 -- 172.21.15.107:0/712828754 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd28c051280 con 0x7fd29414a330 2023-10-25T00:05:53.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.545+0000 7fd299885640 1 -- 172.21.15.107:0/712828754 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2940746e0 con 0x7fd29414a330 2023-10-25T00:05:53.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.545+0000 7fd290ff9640 1 -- 172.21.15.107:0/712828754 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5775 (secure 0 0 0) 0x7fd28c0548a0 con 0x7fd29414a330 2023-10-25T00:05:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:53.545 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114},{"gid":7122,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:16.770996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6022":{"gid":6022,"name":"b","rank":0,"incarnation":106,"state":"up:active","state_seq":11,"addr":"172.21.15.110:6834/174863522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":174863522},{"type":"v1","addr":"172.21.15.110:6834","nonce":174863522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:53.546 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 115 2023-10-25T00:05:53.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 -- 172.21.15.107:0/712828754 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd274065e90 msgr2=0x7fd274068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:53.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd274065e90 0x7fd274068350 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd284009a30 tx=0x7fd284008040 comp rx=0 tx=0).stop 2023-10-25T00:05:53.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 -- 172.21.15.107:0/712828754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd29414a330 msgr2=0x7fd294144b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:53.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd29414a330 0x7fd294144b30 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd29406d200 tx=0x7fd28c00b4a0 comp rx=0 tx=0).stop 2023-10-25T00:05:53.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 -- 172.21.15.107:0/712828754 shutdown_connections 2023-10-25T00:05:53.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd29414a330 0x7fd294144b30 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:53.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd274065e90 0x7fd274068350 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:53.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2941ecaa0 0x7fd294149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:53.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 --2- 172.21.15.107:0/712828754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd280003f30 0x7fd2941ec560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:53.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 -- 172.21.15.107:0/712828754 >> 172.21.15.107:0/712828754 conn(0x7fd2940c2950 msgr2=0x7fd2940c16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:53.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 -- 172.21.15.107:0/712828754 shutdown_connections 2023-10-25T00:05:53.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:53.549+0000 7fd299885640 1 -- 172.21.15.107:0/712828754 wait complete. 2023-10-25T00:05:53.562 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:05:55.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:55.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c478e9640 1 Processor -- start 2023-10-25T00:05:55.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c478e9640 1 -- start start 2023-10-25T00:05:55.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c478e9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4c400aa330 0x7f4c4006e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:55.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c478e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c400a9450 0x7f4c40071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:55.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c478e9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c40152d40 con 0x7f4c4013bb00 2023-10-25T00:05:55.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c478e9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c40156100 con 0x7f4c400aa330 2023-10-25T00:05:55.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c478e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c401527c0 con 0x7f4c400a9450 2023-10-25T00:05:55.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c44e84640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4c400aa330 0x7f4c4006e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:55.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c45685640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4c4013bb00 0x7f4c401531d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40328/0 (socket says 172.21.15.107:40328) 2023-10-25T00:05:55.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c45685640 1 -- 172.21.15.107:0/3361168701 learned_addr learned my addr 172.21.15.107:0/3361168701 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:55.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c45e86640 1 --2- 172.21.15.107:0/3361168701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c400a9450 0x7f4c40071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:55.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c44e84640 1 -- 172.21.15.107:0/3361168701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c400a9450 msgr2=0x7f4c40071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:55.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c44e84640 1 --2- 172.21.15.107:0/3361168701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c400a9450 0x7f4c40071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:55.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c44e84640 1 -- 172.21.15.107:0/3361168701 >> v1:172.21.15.107:6789/0 conn(0x7f4c4013bb00 legacy=0x7f4c401531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:55.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.700+0000 7f4c44e84640 1 -- 172.21.15.107:0/3361168701 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4006d660 con 0x7f4c400aa330 2023-10-25T00:05:55.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c44e84640 1 --2- 172.21.15.107:0/3361168701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c400aa330 0x7f4c4006e250 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f4c300097c0 tx=0x7f4c3001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594b94ebfd0f3269 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:55.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c37fff640 1 -- 172.21.15.107:0/3361168701 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c300ab600 con 0x7f4c400aa330 2023-10-25T00:05:55.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c37fff640 1 -- 172.21.15.107:0/3361168701 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c300a7050 con 0x7f4c400aa330 2023-10-25T00:05:55.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c37fff640 1 -- 172.21.15.107:0/3361168701 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c300ab980 con 0x7f4c400aa330 2023-10-25T00:05:55.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 -- 172.21.15.107:0/3361168701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c400aa330 msgr2=0x7f4c4006e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:55.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 --2- 172.21.15.107:0/3361168701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c400aa330 0x7f4c4006e250 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f4c300097c0 tx=0x7f4c3001c360 comp rx=0 tx=0).stop 2023-10-25T00:05:55.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 -- 172.21.15.107:0/3361168701 shutdown_connections 2023-10-25T00:05:55.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 --2- 172.21.15.107:0/3361168701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c400a9450 0x7f4c40071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:55.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 --2- 172.21.15.107:0/3361168701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c400aa330 0x7f4c4006e250 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:55.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 -- 172.21.15.107:0/3361168701 >> 172.21.15.107:0/3361168701 conn(0x7f4c40070f60 msgr2=0x7f4c40071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:55.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 -- 172.21.15.107:0/3361168701 shutdown_connections 2023-10-25T00:05:55.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 -- 172.21.15.107:0/3361168701 wait complete. 2023-10-25T00:05:55.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 Processor -- start 2023-10-25T00:05:55.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 -- start start 2023-10-25T00:05:55.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c400a9450 0x7f4c400c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:55.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c400aa330 0x7f4c401eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:55.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c4013bb00 0x7f4c400c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:55.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c400c61d0 con 0x7f4c400a9450 2023-10-25T00:05:55.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c401ebd00 con 0x7f4c4013bb00 2023-10-25T00:05:55.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c478e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c400c3270 con 0x7f4c400aa330 2023-10-25T00:05:55.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c45685640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c400a9450 0x7f4c400c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:55.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c45685640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c400a9450 0x7f4c400c2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33624/0 (socket says 172.21.15.107:33624) 2023-10-25T00:05:55.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c45685640 1 -- 172.21.15.107:0/2681230010 learned_addr learned my addr 172.21.15.107:0/2681230010 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:55.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c44e84640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c400aa330 0x7f4c401eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:55.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.704+0000 7f4c45e86640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c4013bb00 0x7f4c400c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:55.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c45685640 1 -- 172.21.15.107:0/2681230010 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c400aa330 msgr2=0x7f4c401eb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:55.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c45685640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c400aa330 0x7f4c401eb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:55.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c45685640 1 -- 172.21.15.107:0/2681230010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c4013bb00 msgr2=0x7f4c400c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:55.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c45685640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c4013bb00 0x7f4c400c75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:55.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c45685640 1 -- 172.21.15.107:0/2681230010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c30009420 con 0x7f4c400a9450 2023-10-25T00:05:55.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c45685640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c400a9450 0x7f4c400c2c40 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00c400 tx=0x7f4c3c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:55.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c35ffb640 1 -- 172.21.15.107:0/2681230010 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c005cf0 con 0x7f4c400a9450 2023-10-25T00:05:55.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c35ffb640 1 -- 172.21.15.107:0/2681230010 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c005e50 con 0x7f4c400a9450 2023-10-25T00:05:55.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c478e9640 1 -- 172.21.15.107:0/2681230010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c400c3550 con 0x7f4c400a9450 2023-10-25T00:05:55.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c35ffb640 1 -- 172.21.15.107:0/2681230010 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c0072d0 con 0x7f4c400a9450 2023-10-25T00:05:55.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c478e9640 1 -- 172.21.15.107:0/2681230010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c400c3830 con 0x7f4c400a9450 2023-10-25T00:05:55.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c478e9640 1 -- 172.21.15.107:0/2681230010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c400746e0 con 0x7f4c400a9450 2023-10-25T00:05:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c35ffb640 1 -- 172.21.15.107:0/2681230010 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4c3c007430 con 0x7f4c400a9450 2023-10-25T00:05:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c35ffb640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4c24065e90 0x7f4c24068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c35ffb640 1 -- 172.21.15.107:0/2681230010 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7f4c3c083be0 con 0x7f4c400a9450 2023-10-25T00:05:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c44e84640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4c24065e90 0x7f4c24068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:55.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.708+0000 7f4c44e84640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4c24065e90 0x7f4c24068350 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4c300ad000 tx=0x7f4c30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:55.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.712+0000 7f4c35ffb640 1 -- 172.21.15.107:0/2681230010 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4c3c083e60 con 0x7f4c400a9450 2023-10-25T00:05:55.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 -- 172.21.15.107:0/2681230010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c400c2380 con 0x7f4c400a9450 2023-10-25T00:05:55.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c35ffb640 1 -- 172.21.15.107:0/2681230010 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f4c3c017090 con 0x7f4c400a9450 2023-10-25T00:05:55.907 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:55.907 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:55.773036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:55.908 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 118 2023-10-25T00:05:55.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 -- 172.21.15.107:0/2681230010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4c24065e90 msgr2=0x7f4c24068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:55.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4c24065e90 0x7f4c24068350 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4c300ad000 tx=0x7f4c30008040 comp rx=0 tx=0).stop 2023-10-25T00:05:55.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 -- 172.21.15.107:0/2681230010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c400a9450 msgr2=0x7f4c400c2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:55.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c400a9450 0x7f4c400c2c40 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00c400 tx=0x7f4c3c00c8c0 comp rx=0 tx=0).stop 2023-10-25T00:05:55.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 -- 172.21.15.107:0/2681230010 shutdown_connections 2023-10-25T00:05:55.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c4013bb00 0x7f4c400c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:55.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4c24065e90 0x7f4c24068350 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:55.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c400aa330 0x7f4c401eb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:55.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 --2- 172.21.15.107:0/2681230010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c400a9450 0x7f4c400c2c40 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:55.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 -- 172.21.15.107:0/2681230010 >> 172.21.15.107:0/2681230010 conn(0x7f4c40070f60 msgr2=0x7f4c401505f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:55.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 -- 172.21.15.107:0/2681230010 shutdown_connections 2023-10-25T00:05:55.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:55.908+0000 7f4c478e9640 1 -- 172.21.15.107:0/2681230010 wait complete. 2023-10-25T00:05:55.926 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2023-10-25T00:05:55.926 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:05:55.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:56.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.048+0000 7f1c74348640 1 Processor -- start 2023-10-25T00:05:56.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.048+0000 7f1c74348640 1 -- start start 2023-10-25T00:05:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c74348640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c6c0a9820 0x7f1c6c0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c74348640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6c06f730 con 0x7f1c6c14fd40 2023-10-25T00:05:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c74348640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6c156a40 con 0x7f1c6c06ee40 2023-10-25T00:05:56.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c74348640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6c156710 con 0x7f1c6c0a9820 2023-10-25T00:05:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c720e4640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1c6c14fd40 0x7f1c6c0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40330/0 (socket says 172.21.15.107:40330) 2023-10-25T00:05:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c720e4640 1 -- 172.21.15.107:0/1274621989 learned_addr learned my addr 172.21.15.107:0/1274621989 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c718e3640 1 --2- 172.21.15.107:0/1274621989 >> v2:172.21.15.107:3300/0 conn(0x7f1c6c0a9820 0x7f1c6c0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c710e2640 1 -- 172.21.15.107:0/1274621989 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438487904 0 0) 0x7f1c6c06f730 con 0x7f1c6c14fd40 2023-10-25T00:05:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c710e2640 1 -- 172.21.15.107:0/1274621989 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c58003530 con 0x7f1c6c14fd40 2023-10-25T00:05:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c710e2640 1 -- 172.21.15.107:0/1274621989 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3258361231 0 0) 0x7f1c58003530 con 0x7f1c6c14fd40 2023-10-25T00:05:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c718e3640 1 -- 172.21.15.107:0/1274621989 >> v1:172.21.15.110:6789/0 conn(0x7f1c6c06ee40 legacy=0x7f1c6c15cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c718e3640 1 -- 172.21.15.107:0/1274621989 >> v1:172.21.15.107:6789/0 conn(0x7f1c6c14fd40 legacy=0x7f1c6c0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:56.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c718e3640 1 -- 172.21.15.107:0/1274621989 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c06f980 con 0x7f1c6c0a9820 2023-10-25T00:05:56.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c718e3640 1 --2- 172.21.15.107:0/1274621989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c0a9820 0x7f1c6c0ab090 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0097c0 tx=0x7f1c5c01c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c6832de082cd49 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:56.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c710e2640 1 -- 172.21.15.107:0/1274621989 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68001680 con 0x7f1c6c0a9820 2023-10-25T00:05:56.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c710e2640 1 -- 172.21.15.107:0/1274621989 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5c0a7050 con 0x7f1c6c0a9820 2023-10-25T00:05:56.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c710e2640 1 -- 172.21.15.107:0/1274621989 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0ab620 con 0x7f1c6c0a9820 2023-10-25T00:05:56.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c74348640 1 -- 172.21.15.107:0/1274621989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c0a9820 msgr2=0x7f1c6c0ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:56.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c74348640 1 --2- 172.21.15.107:0/1274621989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c0a9820 0x7f1c6c0ab090 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0097c0 tx=0x7f1c5c01c280 comp rx=0 tx=0).stop 2023-10-25T00:05:56.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c74348640 1 -- 172.21.15.107:0/1274621989 shutdown_connections 2023-10-25T00:05:56.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c74348640 1 --2- 172.21.15.107:0/1274621989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c0a9820 0x7f1c6c0ab090 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:56.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c74348640 1 -- 172.21.15.107:0/1274621989 >> 172.21.15.107:0/1274621989 conn(0x7f1c6c0c2950 msgr2=0x7f1c6c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c74348640 1 -- 172.21.15.107:0/1274621989 shutdown_connections 2023-10-25T00:05:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.052+0000 7f1c74348640 1 -- 172.21.15.107:0/1274621989 wait complete. 2023-10-25T00:05:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c74348640 1 Processor -- start 2023-10-25T00:05:56.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c74348640 1 -- start start 2023-10-25T00:05:56.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c74348640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c06ee40 0x7f1c6c0c4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:56.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c74348640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c14fd40 0x7f1c6c0c4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:56.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c74348640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14b0f0 0x7f1c6c14b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:56.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c74348640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c14a810 con 0x7f1c6c14fd40 2023-10-25T00:05:56.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c74348640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c1516d0 con 0x7f1c6c06ee40 2023-10-25T00:05:56.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c74348640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c0bb1e0 con 0x7f1c6c14b0f0 2023-10-25T00:05:56.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c718e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c14fd40 0x7f1c6c0c4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:56.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c718e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c14fd40 0x7f1c6c0c4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33644/0 (socket says 172.21.15.107:33644) 2023-10-25T00:05:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c718e3640 1 -- 172.21.15.107:0/203932958 learned_addr learned my addr 172.21.15.107:0/203932958 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c728e5640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14b0f0 0x7f1c6c14b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c720e4640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c06ee40 0x7f1c6c0c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c718e3640 1 -- 172.21.15.107:0/203932958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14b0f0 msgr2=0x7f1c6c14b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c718e3640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14b0f0 0x7f1c6c14b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c718e3640 1 -- 172.21.15.107:0/203932958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c06ee40 msgr2=0x7f1c6c0c4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c718e3640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c06ee40 0x7f1c6c0c4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c718e3640 1 -- 172.21.15.107:0/203932958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c009420 con 0x7f1c6c14fd40 2023-10-25T00:05:56.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c728e5640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14b0f0 0x7f1c6c14b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:56.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c720e4640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c06ee40 0x7f1c6c0c4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:56.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c718e3640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c14fd40 0x7f1c6c0c4cc0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0097c0 tx=0x7f1c5c019020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:56.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c62ffd640 1 -- 172.21.15.107:0/203932958 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c004020 con 0x7f1c6c14fd40 2023-10-25T00:05:56.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c62ffd640 1 -- 172.21.15.107:0/203932958 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5c004180 con 0x7f1c6c14fd40 2023-10-25T00:05:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c74348640 1 -- 172.21.15.107:0/203932958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6c14a150 con 0x7f1c6c14fd40 2023-10-25T00:05:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c62ffd640 1 -- 172.21.15.107:0/203932958 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0044f0 con 0x7f1c6c14fd40 2023-10-25T00:05:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c74348640 1 -- 172.21.15.107:0/203932958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6c0c5200 con 0x7f1c6c14fd40 2023-10-25T00:05:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.056+0000 7f1c74348640 1 -- 172.21.15.107:0/203932958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6c0c24a0 con 0x7f1c6c14fd40 2023-10-25T00:05:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.060+0000 7f1c62ffd640 1 -- 172.21.15.107:0/203932958 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1c5c0b2040 con 0x7f1c6c14fd40 2023-10-25T00:05:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.060+0000 7f1c62ffd640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c50065e60 0x7f1c50068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.060+0000 7f1c720e4640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c50065e60 0x7f1c50068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.060+0000 7f1c62ffd640 1 -- 172.21.15.107:0/203932958 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7f1c5c11e260 con 0x7f1c6c14fd40 2023-10-25T00:05:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.060+0000 7f1c720e4640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c50065e60 0x7f1c50068320 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1c68000f70 tx=0x7f1c68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.064+0000 7f1c62ffd640 1 -- 172.21.15.107:0/203932958 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1c5c0eae00 con 0x7f1c6c14fd40 2023-10-25T00:05:56.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.248+0000 7f1c74348640 1 -- 172.21.15.107:0/203932958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c6c14a950 con 0x7f1c6c14fd40 2023-10-25T00:05:56.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c62ffd640 1 -- 172.21.15.107:0/203932958 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f1c6c14a950 con 0x7f1c6c14fd40 2023-10-25T00:05:56.251 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:56.251 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:55.773036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:56.251 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 118 2023-10-25T00:05:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 -- 172.21.15.107:0/203932958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c50065e60 msgr2=0x7f1c50068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c50065e60 0x7f1c50068320 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f1c68000f70 tx=0x7f1c68009040 comp rx=0 tx=0).stop 2023-10-25T00:05:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 -- 172.21.15.107:0/203932958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c14fd40 msgr2=0x7f1c6c0c4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c14fd40 0x7f1c6c0c4cc0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0097c0 tx=0x7f1c5c019020 comp rx=0 tx=0).stop 2023-10-25T00:05:56.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 -- 172.21.15.107:0/203932958 shutdown_connections 2023-10-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c50065e60 0x7f1c50068320 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14b0f0 0x7f1c6c14b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c14fd40 0x7f1c6c0c4cc0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 --2- 172.21.15.107:0/203932958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c06ee40 0x7f1c6c0c4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:56.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 -- 172.21.15.107:0/203932958 >> 172.21.15.107:0/203932958 conn(0x7f1c6c0c2950 msgr2=0x7f1c6c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 -- 172.21.15.107:0/203932958 shutdown_connections 2023-10-25T00:05:56.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:56.252+0000 7f1c74348640 1 -- 172.21.15.107:0/203932958 wait complete. 2023-10-25T00:05:56.270 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 118, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 118, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7122": { "addr": "172.21.15.107:6837/780830443", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 780830443, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 780830443, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7122, "incarnation": 116, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:05:55.773036+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7122 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 113, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5410": { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 110, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 98 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:05:21.824320+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5410 } } } ], "standbys": [ { "addr": "172.21.15.107:6835/913067382", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 913067382, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 913067382, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 114, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7107, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:05:57.663 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:05:57.664+0000 7f0808071640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:05:58.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:05:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.412+0000 7f5e59f38640 1 Processor -- start 2023-10-25T00:05:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.412+0000 7f5e59f38640 1 -- start start 2023-10-25T00:05:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.412+0000 7f5e59f38640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e5406a4b0 0x7f5e540ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.412+0000 7f5e59f38640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e54155680 con 0x7f5e54152040 2023-10-25T00:05:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.412+0000 7f5e59f38640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e54154a70 con 0x7f5e540c8100 2023-10-25T00:05:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.412+0000 7f5e59f38640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e54154d50 con 0x7f5e5406a4b0 2023-10-25T00:05:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e537fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5e54152040 0x7f5e540c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:40334/0 (socket says 172.21.15.107:40334) 2023-10-25T00:05:58.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e537fe640 1 -- 172.21.15.107:0/4012762099 learned_addr learned my addr 172.21.15.107:0/4012762099 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:05:58.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e53fff640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e5406a4b0 0x7f5e540ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:58.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593981379 0 0) 0x7f5e54155680 con 0x7f5e54152040 2023-10-25T00:05:58.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e3c003530 con 0x7f5e54152040 2023-10-25T00:05:58.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2589935763 0 0) 0x7f5e3c003530 con 0x7f5e54152040 2023-10-25T00:05:58.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e5406a4b0 msgr2=0x7f5e540ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:58.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e5406a4b0 0x7f5e540ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:58.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 >> v1:172.21.15.110:6789/0 conn(0x7f5e540c8100 legacy=0x7f5e540abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:58.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e54155030 con 0x7f5e54152040 2023-10-25T00:05:58.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5e44001680 con 0x7f5e54152040 2023-10-25T00:05:58.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 >> v1:172.21.15.107:6789/0 conn(0x7f5e54152040 legacy=0x7f5e540c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 0x7f5e3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e3c0047d0 0x7f5e3c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e5406a4b0 0x7f5e3c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e54154d50 con 0x7f5e3c0039b0 2023-10-25T00:05:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c00baf0 con 0x7f5e3c0047d0 2023-10-25T00:05:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c00bcb0 con 0x7f5e5406a4b0 2023-10-25T00:05:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e537fe640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 0x7f5e3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e52ffd640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e3c0047d0 0x7f5e3c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e537fe640 1 -- 172.21.15.107:0/4012762099 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e5406a4b0 msgr2=0x7f5e3c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:58.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e537fe640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e5406a4b0 0x7f5e3c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:58.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e537fe640 1 -- 172.21.15.107:0/4012762099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e3c0047d0 msgr2=0x7f5e3c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:05:58.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e537fe640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e3c0047d0 0x7f5e3c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:58.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e537fe640 1 -- 172.21.15.107:0/4012762099 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c003530 con 0x7f5e3c0039b0 2023-10-25T00:05:58.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e52ffd640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e3c0047d0 0x7f5e3c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:05:58.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e537fe640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 0x7f5e3c003dd0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400c240 tx=0x7f5e4401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18174cb962ff747c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:58.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44005ae0 con 0x7f5e3c0039b0 2023-10-25T00:05:58.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e44005c60 con 0x7f5e3c0039b0 2023-10-25T00:05:58.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.416+0000 7f5e527fc640 1 -- 172.21.15.107:0/4012762099 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44007070 con 0x7f5e3c0039b0 2023-10-25T00:05:58.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- 172.21.15.107:0/4012762099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 msgr2=0x7f5e3c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:58.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 --2- 172.21.15.107:0/4012762099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 0x7f5e3c003dd0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400c240 tx=0x7f5e4401cc40 comp rx=0 tx=0).stop 2023-10-25T00:05:58.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e53fff640 1 -- 172.21.15.107:0/4012762099 reap_dead start 2023-10-25T00:05:58.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- 172.21.15.107:0/4012762099 shutdown_connections 2023-10-25T00:05:58.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- 172.21.15.107:0/4012762099 >> 172.21.15.107:0/4012762099 conn(0x7f5e540ca200 msgr2=0x7f5e540ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:58.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- 172.21.15.107:0/4012762099 shutdown_connections 2023-10-25T00:05:58.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- 172.21.15.107:0/4012762099 wait complete. 2023-10-25T00:05:58.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 Processor -- start 2023-10-25T00:05:58.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- start start 2023-10-25T00:05:58.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 0x7f5e5414a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:58.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e54148d60 0x7f5e54150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:58.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e540c4090 0x7f5e54150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:58.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e541ecce0 con 0x7f5e3c0039b0 2023-10-25T00:05:58.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e541edcb0 con 0x7f5e54148d60 2023-10-25T00:05:58.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e541ecef0 con 0x7f5e540c4090 2023-10-25T00:05:58.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e537fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 0x7f5e5414a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:58.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e537fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 0x7f5e5414a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33668/0 (socket says 172.21.15.107:33668) 2023-10-25T00:05:58.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e537fe640 1 -- 172.21.15.107:0/3521119326 learned_addr learned my addr 172.21.15.107:0/3521119326 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:05:58.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e52ffd640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e54148d60 0x7f5e54150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:58.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e53fff640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e540c4090 0x7f5e54150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:58.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e537fe640 1 -- 172.21.15.107:0/3521119326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e540c4090 msgr2=0x7f5e54150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:58.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e537fe640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e540c4090 0x7f5e54150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:58.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e537fe640 1 -- 172.21.15.107:0/3521119326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e54148d60 msgr2=0x7f5e54150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:58.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e537fe640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e54148d60 0x7f5e54150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:58.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e537fe640 1 -- 172.21.15.107:0/3521119326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e541ed1d0 con 0x7f5e3c0039b0 2023-10-25T00:05:58.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e537fe640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 0x7f5e5414a680 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f5e44001940 tx=0x7f5e3c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:58.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e3bfff640 1 -- 172.21.15.107:0/3521119326 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e440b4070 con 0x7f5e3c0039b0 2023-10-25T00:05:58.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e3bfff640 1 -- 172.21.15.107:0/3521119326 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e44005dd0 con 0x7f5e3c0039b0 2023-10-25T00:05:58.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- 172.21.15.107:0/3521119326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e541ed4b0 con 0x7f5e3c0039b0 2023-10-25T00:05:58.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e3bfff640 1 -- 172.21.15.107:0/3521119326 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44007090 con 0x7f5e3c0039b0 2023-10-25T00:05:58.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- 172.21.15.107:0/3521119326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e540c17a0 con 0x7f5e3c0039b0 2023-10-25T00:05:58.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.420+0000 7f5e59f38640 1 -- 172.21.15.107:0/3521119326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e541ed9d0 con 0x7f5e3c0039b0 2023-10-25T00:05:58.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.424+0000 7f5e3bfff640 1 -- 172.21.15.107:0/3521119326 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5e440071f0 con 0x7f5e3c0039b0 2023-10-25T00:05:58.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.424+0000 7f5e3bfff640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e40065e60 0x7f5e40068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:05:58.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.424+0000 7f5e52ffd640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e40065e60 0x7f5e40068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:05:58.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.424+0000 7f5e3bfff640 1 -- 172.21.15.107:0/3521119326 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7f5e4411dd20 con 0x7f5e3c0039b0 2023-10-25T00:05:58.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.424+0000 7f5e52ffd640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e40065e60 0x7f5e40068320 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5e540ae2c0 tx=0x7f5e48009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:05:58.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.428+0000 7f5e3bfff640 1 -- 172.21.15.107:0/3521119326 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5e541ed9d0 con 0x7f5e3c0039b0 2023-10-25T00:05:58.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.628+0000 7f5e59f38640 1 -- 172.21.15.107:0/3521119326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e541ed9d0 con 0x7f5e3c0039b0 2023-10-25T00:05:58.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e3bfff640 1 -- 172.21.15.107:0/3521119326 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f5e440ea8c0 con 0x7f5e3c0039b0 2023-10-25T00:05:58.632 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:05:58.632 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:56.785175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:05:58.632 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 119 2023-10-25T00:05:58.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e59f38640 1 -- 172.21.15.107:0/3521119326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e40065e60 msgr2=0x7f5e40068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:58.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e59f38640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e40065e60 0x7f5e40068320 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5e540ae2c0 tx=0x7f5e48009040 comp rx=0 tx=0).stop 2023-10-25T00:05:58.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e59f38640 1 -- 172.21.15.107:0/3521119326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 msgr2=0x7f5e5414a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:05:58.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e59f38640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 0x7f5e5414a680 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f5e44001940 tx=0x7f5e3c005800 comp rx=0 tx=0).stop 2023-10-25T00:05:58.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e59f38640 1 -- 172.21.15.107:0/3521119326 shutdown_connections 2023-10-25T00:05:58.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e59f38640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5e40065e60 0x7f5e40068320 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:58.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e59f38640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e540c4090 0x7f5e54150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:58.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e59f38640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e54148d60 0x7f5e54150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:58.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e59f38640 1 --2- 172.21.15.107:0/3521119326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5e3c0039b0 0x7f5e5414a680 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:05:58.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e59f38640 1 -- 172.21.15.107:0/3521119326 >> 172.21.15.107:0/3521119326 conn(0x7f5e540ca200 msgr2=0x7f5e54153af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:05:58.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.632+0000 7f5e59f38640 1 -- 172.21.15.107:0/3521119326 shutdown_connections 2023-10-25T00:05:58.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:05:58.636+0000 7f5e59f38640 1 -- 172.21.15.107:0/3521119326 wait complete. 2023-10-25T00:05:58.652 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6022 2023-10-25T00:05:58.652 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.b 2023-10-25T00:06:01.653 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2023-10-25T00:06:01.653 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-25T00:06:01.653 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-25T00:06:01.656 INFO:tasks.ceph.mds.b:Started 2023-10-25T00:06:01.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:01.733 INFO:tasks.ceph.mds.b.smithi110.stdout:starting mds.b at 2023-10-25T00:06:01.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5ec199640 1 Processor -- start 2023-10-25T00:06:01.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5ec199640 1 -- start start 2023-10-25T00:06:01.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5ec199640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5e40c8100 0x7fc5e40abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:01.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5ec199640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc5e406a4b0 0x7fc5e40ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:01.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5ec199640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e4155680 con 0x7fc5e4152040 2023-10-25T00:06:01.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5ec199640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e4154a70 con 0x7fc5e40c8100 2023-10-25T00:06:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5ec199640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e4154d50 con 0x7fc5e406a4b0 2023-10-25T00:06:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5e9734640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5e40c8100 0x7fc5e40abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5e9734640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5e40c8100 0x7fc5e40abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33670/0 (socket says 172.21.15.107:33670) 2023-10-25T00:06:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5e9734640 1 -- 172.21.15.107:0/4046134241 learned_addr learned my addr 172.21.15.107:0/4046134241 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5e9f35640 1 --1- 172.21.15.107:0/4046134241 >> v1:172.21.15.110:6789/0 conn(0x7fc5e4152040 0x7fc5e40c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:43652/0 (socket says 172.21.15.107:43652) 2023-10-25T00:06:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5ea736640 1 --2- 172.21.15.107:0/4046134241 >> v2:172.21.15.110:3300/0 conn(0x7fc5e406a4b0 0x7fc5e40ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5e9734640 1 -- 172.21.15.107:0/4046134241 >> v2:172.21.15.110:3300/0 conn(0x7fc5e406a4b0 msgr2=0x7fc5e40ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5e9734640 1 --2- 172.21.15.107:0/4046134241 >> v2:172.21.15.110:3300/0 conn(0x7fc5e406a4b0 0x7fc5e40ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5e9734640 1 -- 172.21.15.107:0/4046134241 >> v1:172.21.15.110:6789/0 conn(0x7fc5e4152040 legacy=0x7fc5e40c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.792+0000 7fc5e9734640 1 -- 172.21.15.107:0/4046134241 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e4155030 con 0x7fc5e40c8100 2023-10-25T00:06:01.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5e9734640 1 --2- 172.21.15.107:0/4046134241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e40c8100 0x7fc5e40abd00 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc5e406b590 tx=0x7fc5d401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7936254dc25857 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:01.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5e8f33640 1 -- 172.21.15.107:0/4046134241 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40ab600 con 0x7fc5e40c8100 2023-10-25T00:06:01.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5e8f33640 1 -- 172.21.15.107:0/4046134241 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d40a7050 con 0x7fc5e40c8100 2023-10-25T00:06:01.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5e8f33640 1 -- 172.21.15.107:0/4046134241 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40ab980 con 0x7fc5e40c8100 2023-10-25T00:06:01.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5ec199640 1 -- 172.21.15.107:0/4046134241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e40c8100 msgr2=0x7fc5e40abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:01.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5ec199640 1 --2- 172.21.15.107:0/4046134241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e40c8100 0x7fc5e40abd00 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc5e406b590 tx=0x7fc5d401c470 comp rx=0 tx=0).stop 2023-10-25T00:06:01.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5ec199640 1 -- 172.21.15.107:0/4046134241 shutdown_connections 2023-10-25T00:06:01.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5ec199640 1 --2- 172.21.15.107:0/4046134241 >> v2:172.21.15.110:3300/0 conn(0x7fc5e406a4b0 0x7fc5e40ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:01.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5ec199640 1 --2- 172.21.15.107:0/4046134241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e40c8100 0x7fc5e40abd00 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:01.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5ec199640 1 -- 172.21.15.107:0/4046134241 >> 172.21.15.107:0/4046134241 conn(0x7fc5e40ca200 msgr2=0x7fc5e40ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:01.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5ec199640 1 -- 172.21.15.107:0/4046134241 shutdown_connections 2023-10-25T00:06:01.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.796+0000 7fc5ec199640 1 -- 172.21.15.107:0/4046134241 wait complete. 2023-10-25T00:06:01.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ec199640 1 Processor -- start 2023-10-25T00:06:01.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ec199640 1 -- start start 2023-10-25T00:06:01.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ec199640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5e406a4b0 0x7fc5e4148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:01.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ec199640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e4152040 0x7fc5e40c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:01.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ec199640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5e41eda30 0x7fc5e41495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:01.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ec199640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e41f1240 con 0x7fc5e4152040 2023-10-25T00:06:01.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ec199640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e41ecd40 con 0x7fc5e41eda30 2023-10-25T00:06:01.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ec199640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e41ed5a0 con 0x7fc5e406a4b0 2023-10-25T00:06:01.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5e9734640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e4152040 0x7fc5e40c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:01.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5e9734640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e4152040 0x7fc5e40c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33682/0 (socket says 172.21.15.107:33682) 2023-10-25T00:06:01.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5e9734640 1 -- 172.21.15.107:0/4114395066 learned_addr learned my addr 172.21.15.107:0/4114395066 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:01.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5e9f35640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5e406a4b0 0x7fc5e4148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:01.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ea736640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5e41eda30 0x7fc5e41495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:01.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5e9734640 1 -- 172.21.15.107:0/4114395066 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5e406a4b0 msgr2=0x7fc5e4148740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:01.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5e9734640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5e406a4b0 0x7fc5e4148740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:01.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5e9734640 1 -- 172.21.15.107:0/4114395066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5e41eda30 msgr2=0x7fc5e41495d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:01.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5e9734640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5e41eda30 0x7fc5e41495d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:01.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5e9734640 1 -- 172.21.15.107:0/4114395066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d40094d0 con 0x7fc5e4152040 2023-10-25T00:06:01.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5e9734640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e4152040 0x7fc5e40c8f80 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40ae040 tx=0x7fc5d4019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:01.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5da7fc640 1 -- 172.21.15.107:0/4114395066 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4004030 con 0x7fc5e4152040 2023-10-25T00:06:01.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5da7fc640 1 -- 172.21.15.107:0/4114395066 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d4004190 con 0x7fc5e4152040 2023-10-25T00:06:01.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ec199640 1 -- 172.21.15.107:0/4114395066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e41f0cc0 con 0x7fc5e4152040 2023-10-25T00:06:01.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5da7fc640 1 -- 172.21.15.107:0/4114395066 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4004500 con 0x7fc5e4152040 2023-10-25T00:06:01.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ec199640 1 -- 172.21.15.107:0/4114395066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e41ecf50 con 0x7fc5e4152040 2023-10-25T00:06:01.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5ec199640 1 -- 172.21.15.107:0/4114395066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e41ed730 con 0x7fc5e4152040 2023-10-25T00:06:01.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.800+0000 7fc5da7fc640 1 -- 172.21.15.107:0/4114395066 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc5d40b2070 con 0x7fc5e4152040 2023-10-25T00:06:01.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.804+0000 7fc5da7fc640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5c8065e60 0x7fc5c8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:01.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.804+0000 7fc5e9f35640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5c8065e60 0x7fc5c8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:01.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.804+0000 7fc5da7fc640 1 -- 172.21.15.107:0/4114395066 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7fc5d411dd60 con 0x7fc5e4152040 2023-10-25T00:06:01.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.804+0000 7fc5e9f35640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5c8065e60 0x7fc5c8068320 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0000f70 tx=0x7fc5e0007e50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:01.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:01.808+0000 7fc5da7fc640 1 -- 172.21.15.107:0/4114395066 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc5d40ea9b0 con 0x7fc5e4152040 2023-10-25T00:06:02.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.036+0000 7fc5ec199640 1 -- 172.21.15.107:0/4114395066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5e40746e0 con 0x7fc5e4152040 2023-10-25T00:06:02.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5da7fc640 1 -- 172.21.15.107:0/4114395066 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5016 (secure 0 0 0) 0x7fc5d40bb020 con 0x7fc5e4152040 2023-10-25T00:06:02.038 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:02.039 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:56.785175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:02.039 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 119 2023-10-25T00:06:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 -- 172.21.15.107:0/4114395066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5c8065e60 msgr2=0x7fc5c8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:02.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5c8065e60 0x7fc5c8068320 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0000f70 tx=0x7fc5e0007e50 comp rx=0 tx=0).stop 2023-10-25T00:06:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 -- 172.21.15.107:0/4114395066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e4152040 msgr2=0x7fc5e40c8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e4152040 0x7fc5e40c8f80 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40ae040 tx=0x7fc5d4019210 comp rx=0 tx=0).stop 2023-10-25T00:06:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 -- 172.21.15.107:0/4114395066 shutdown_connections 2023-10-25T00:06:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5e41eda30 0x7fc5e41495d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:02.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5e4152040 0x7fc5e40c8f80 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:02.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5c8065e60 0x7fc5c8068320 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:02.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 --2- 172.21.15.107:0/4114395066 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5e406a4b0 0x7fc5e4148740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:02.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 -- 172.21.15.107:0/4114395066 >> 172.21.15.107:0/4114395066 conn(0x7fc5e40ca200 msgr2=0x7fc5e40ae2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:02.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 -- 172.21.15.107:0/4114395066 shutdown_connections 2023-10-25T00:06:02.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:02.040+0000 7fc5ec199640 1 -- 172.21.15.107:0/4114395066 wait complete. 2023-10-25T00:06:02.057 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-25T00:06:04.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:04.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1c5f52640 1 Processor -- start 2023-10-25T00:06:04.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1c5f52640 1 -- start start 2023-10-25T00:06:04.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1c5f52640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc1c006f490 0x7fc1c014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:04.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1c5f52640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1c0152b30 0x7fc1c0152f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:04.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1c5f52640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c0155cd0 con 0x7fc1c014c470 2023-10-25T00:06:04.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1c5f52640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c01526e0 con 0x7fc1c006f490 2023-10-25T00:06:04.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1c5f52640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00c4380 con 0x7fc1c0152b30 2023-10-25T00:06:04.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1beffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc1c006f490 0x7fc1c014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:04.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1bf7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc1c014c470 0x7fc1c0071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56728/0 (socket says 172.21.15.107:56728) 2023-10-25T00:06:04.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1beffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc1c006f490 0x7fc1c014c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42838/0 (socket says 172.21.15.107:42838) 2023-10-25T00:06:04.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1bf7fe640 1 -- 172.21.15.107:0/2938130195 learned_addr learned my addr 172.21.15.107:0/2938130195 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:06:04.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.192+0000 7fc1bffff640 1 --2- 172.21.15.107:0/2938130195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1c0152b30 0x7fc1c0152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:04.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1beffd640 1 -- 172.21.15.107:0/2938130195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1c0152b30 msgr2=0x7fc1c0152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:04.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1beffd640 1 --2- 172.21.15.107:0/2938130195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1c0152b30 0x7fc1c0152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:04.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1beffd640 1 -- 172.21.15.107:0/2938130195 >> v1:172.21.15.107:6789/0 conn(0x7fc1c014c470 legacy=0x7fc1c0071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:04.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1beffd640 1 -- 172.21.15.107:0/2938130195 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c00c48b0 con 0x7fc1c006f490 2023-10-25T00:06:04.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1be7fc640 1 -- 172.21.15.107:0/2938130195 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508489870 0 0) 0x7fc1c0155cd0 con 0x7fc1c014c470 2023-10-25T00:06:04.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1beffd640 1 --2- 172.21.15.107:0/2938130195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c006f490 0x7fc1c014c850 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc1b40097c0 tx=0x7fc1b401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63558ace7cc37954 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:04.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1be7fc640 1 -- 172.21.15.107:0/2938130195 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b40ab600 con 0x7fc1c006f490 2023-10-25T00:06:04.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1be7fc640 1 -- 172.21.15.107:0/2938130195 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b40a7050 con 0x7fc1c006f490 2023-10-25T00:06:04.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1be7fc640 1 -- 172.21.15.107:0/2938130195 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b40ab980 con 0x7fc1c006f490 2023-10-25T00:06:04.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1c5f52640 1 -- 172.21.15.107:0/2938130195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c006f490 msgr2=0x7fc1c014c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:04.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1c5f52640 1 --2- 172.21.15.107:0/2938130195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c006f490 0x7fc1c014c850 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc1b40097c0 tx=0x7fc1b401c360 comp rx=0 tx=0).stop 2023-10-25T00:06:04.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1c5f52640 1 -- 172.21.15.107:0/2938130195 shutdown_connections 2023-10-25T00:06:04.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1c5f52640 1 --2- 172.21.15.107:0/2938130195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1c0152b30 0x7fc1c0152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:04.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1c5f52640 1 --2- 172.21.15.107:0/2938130195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c006f490 0x7fc1c014c850 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:04.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1c5f52640 1 -- 172.21.15.107:0/2938130195 >> 172.21.15.107:0/2938130195 conn(0x7fc1c0070f60 msgr2=0x7fc1c0071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:04.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1c5f52640 1 -- 172.21.15.107:0/2938130195 shutdown_connections 2023-10-25T00:06:04.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.196+0000 7fc1c5f52640 1 -- 172.21.15.107:0/2938130195 wait complete. 2023-10-25T00:06:04.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1c5f52640 1 Processor -- start 2023-10-25T00:06:04.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1c5f52640 1 -- start start 2023-10-25T00:06:04.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1c5f52640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1c006f490 0x7fc1c00c7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:04.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1c5f52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c014c470 0x7fc1c00c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:04.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1c5f52640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1c0152b30 0x7fc1c0155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:04.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1c5f52640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c01f0a10 con 0x7fc1c014c470 2023-10-25T00:06:04.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1c5f52640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00c7d90 con 0x7fc1c006f490 2023-10-25T00:06:04.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1c5f52640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c0153f50 con 0x7fc1c0152b30 2023-10-25T00:06:04.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1beffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c014c470 0x7fc1c00c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:04.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1beffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c014c470 0x7fc1c00c1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42844/0 (socket says 172.21.15.107:42844) 2023-10-25T00:06:04.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1beffd640 1 -- 172.21.15.107:0/1334681530 learned_addr learned my addr 172.21.15.107:0/1334681530 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:04.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1bffff640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1c0152b30 0x7fc1c0155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:04.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1bf7fe640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1c006f490 0x7fc1c00c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1beffd640 1 -- 172.21.15.107:0/1334681530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1c0152b30 msgr2=0x7fc1c0155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1beffd640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1c0152b30 0x7fc1c0155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1beffd640 1 -- 172.21.15.107:0/1334681530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1c006f490 msgr2=0x7fc1c00c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1beffd640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1c006f490 0x7fc1c00c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:04.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1beffd640 1 -- 172.21.15.107:0/1334681530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4009420 con 0x7fc1c014c470 2023-10-25T00:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1beffd640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c014c470 0x7fc1c00c1570 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4019f90 tx=0x7fc1b4019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1a7fff640 1 -- 172.21.15.107:0/1334681530 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b40b4610 con 0x7fc1c014c470 2023-10-25T00:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1a7fff640 1 -- 172.21.15.107:0/1334681530 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b40abe90 con 0x7fc1c014c470 2023-10-25T00:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1c5f52640 1 -- 172.21.15.107:0/1334681530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c0153c70 con 0x7fc1c014c470 2023-10-25T00:06:04.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1a7fff640 1 -- 172.21.15.107:0/1334681530 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b40b49d0 con 0x7fc1c014c470 2023-10-25T00:06:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.200+0000 7fc1c5f52640 1 -- 172.21.15.107:0/1334681530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c00c1ab0 con 0x7fc1c014c470 2023-10-25T00:06:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.204+0000 7fc1c5f52640 1 -- 172.21.15.107:0/1334681530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c00c76a0 con 0x7fc1c014c470 2023-10-25T00:06:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.204+0000 7fc1a7fff640 1 -- 172.21.15.107:0/1334681530 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc1b40b2040 con 0x7fc1c014c470 2023-10-25T00:06:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.204+0000 7fc1a7fff640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1ac065e60 0x7fc1ac068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:04.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.204+0000 7fc1a7fff640 1 -- 172.21.15.107:0/1334681530 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7fc1b411e3e0 con 0x7fc1c014c470 2023-10-25T00:06:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.204+0000 7fc1bf7fe640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1ac065e60 0x7fc1ac068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.204+0000 7fc1bf7fe640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1ac065e60 0x7fc1ac068320 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0000f70 tx=0x7fc1b0007b80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:04.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.208+0000 7fc1a7fff640 1 -- 172.21.15.107:0/1334681530 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc1b40eaef0 con 0x7fc1c014c470 2023-10-25T00:06:04.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1c5f52640 1 -- 172.21.15.107:0/1334681530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1c00746e0 con 0x7fc1c014c470 2023-10-25T00:06:04.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1a7fff640 1 -- 172.21.15.107:0/1334681530 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5774 (secure 0 0 0) 0x7fc1b40ee510 con 0x7fc1c014c470 2023-10-25T00:06:04.448 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:04.448 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:56.785175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 120 2023-10-25T00:06:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1c5f52640 1 -- 172.21.15.107:0/1334681530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1ac065e60 msgr2=0x7fc1ac068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1c5f52640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1ac065e60 0x7fc1ac068320 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0000f70 tx=0x7fc1b0007b80 comp rx=0 tx=0).stop 2023-10-25T00:06:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1c5f52640 1 -- 172.21.15.107:0/1334681530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c014c470 msgr2=0x7fc1c00c1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1c5f52640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c014c470 0x7fc1c00c1570 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4019f90 tx=0x7fc1b4019100 comp rx=0 tx=0).stop 2023-10-25T00:06:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1c5f52640 1 -- 172.21.15.107:0/1334681530 shutdown_connections 2023-10-25T00:06:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1c5f52640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1ac065e60 0x7fc1ac068320 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1c5f52640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1c0152b30 0x7fc1c0155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1c5f52640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1c014c470 0x7fc1c00c1570 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1c5f52640 1 --2- 172.21.15.107:0/1334681530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1c006f490 0x7fc1c00c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.448+0000 7fc1c5f52640 1 -- 172.21.15.107:0/1334681530 >> 172.21.15.107:0/1334681530 conn(0x7fc1c0070f60 msgr2=0x7fc1c0151bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.452+0000 7fc1c5f52640 1 -- 172.21.15.107:0/1334681530 shutdown_connections 2023-10-25T00:06:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:04.452+0000 7fc1c5f52640 1 -- 172.21.15.107:0/1334681530 wait complete. 2023-10-25T00:06:04.468 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2023-10-25T00:06:04.469 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before thrashing 2023-10-25T00:06:07.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.604+0000 7f2876ac8640 1 Processor -- start 2023-10-25T00:06:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.604+0000 7f2876ac8640 1 -- start start 2023-10-25T00:06:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.604+0000 7f2876ac8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f287006f490 0x7f287014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:07.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.604+0000 7f2876ac8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 0x7f2870152f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:07.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.604+0000 7f2876ac8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2870155cd0 con 0x7f287014c470 2023-10-25T00:06:07.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.604+0000 7f2876ac8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28701526e0 con 0x7f287006f490 2023-10-25T00:06:07.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.604+0000 7f2876ac8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28700c4380 con 0x7f2870152b30 2023-10-25T00:06:07.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.604+0000 7f2875065640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 0x7f2870152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:07.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.604+0000 7f286ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f287006f490 0x7f287014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:07.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.604+0000 7f2875065640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 0x7f2870152f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:46044/0 (socket says 172.21.15.107:46044) 2023-10-25T00:06:07.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.604+0000 7f2875065640 1 -- 172.21.15.107:0/610229834 learned_addr learned my addr 172.21.15.107:0/610229834 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f286ffff640 1 -- 172.21.15.107:0/610229834 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 msgr2=0x7f2870152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f286ffff640 1 --2- 172.21.15.107:0/610229834 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 0x7f2870152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f286ffff640 1 -- 172.21.15.107:0/610229834 >> v1:172.21.15.110:6789/0 conn(0x7f287014c470 legacy=0x7f2870071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f286ffff640 1 -- 172.21.15.107:0/610229834 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28700c48b0 con 0x7f287006f490 2023-10-25T00:06:07.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f2875065640 1 --2- 172.21.15.107:0/610229834 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 0x7f2870152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:06:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f286ffff640 1 --2- 172.21.15.107:0/610229834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287006f490 0x7f287014c850 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f285c0097c0 tx=0x7f285c01c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8e770e001ac9f51 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f286f7fe640 1 -- 172.21.15.107:0/610229834 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c0ab600 con 0x7f287006f490 2023-10-25T00:06:07.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f286f7fe640 1 -- 172.21.15.107:0/610229834 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c0a7050 con 0x7f287006f490 2023-10-25T00:06:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f286f7fe640 1 -- 172.21.15.107:0/610229834 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c0ab980 con 0x7f287006f490 2023-10-25T00:06:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f2876ac8640 1 -- 172.21.15.107:0/610229834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287006f490 msgr2=0x7f287014c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f2876ac8640 1 --2- 172.21.15.107:0/610229834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287006f490 0x7f287014c850 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f285c0097c0 tx=0x7f285c01c360 comp rx=0 tx=0).stop 2023-10-25T00:06:07.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f2876ac8640 1 -- 172.21.15.107:0/610229834 shutdown_connections 2023-10-25T00:06:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f2876ac8640 1 --2- 172.21.15.107:0/610229834 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 0x7f2870152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f2876ac8640 1 --2- 172.21.15.107:0/610229834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287006f490 0x7f287014c850 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:07.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f2876ac8640 1 -- 172.21.15.107:0/610229834 >> 172.21.15.107:0/610229834 conn(0x7f2870070f60 msgr2=0x7f2870071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f2876ac8640 1 -- 172.21.15.107:0/610229834 shutdown_connections 2023-10-25T00:06:07.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.608+0000 7f2876ac8640 1 -- 172.21.15.107:0/610229834 wait complete. 2023-10-25T00:06:07.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2876ac8640 1 Processor -- start 2023-10-25T00:06:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2876ac8640 1 -- start start 2023-10-25T00:06:07.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2876ac8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287006f490 0x7f2870143870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2876ac8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f287014c470 0x7f28700c1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2876ac8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 0x7f28701ec840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:07.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2876ac8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287014d010 con 0x7f287014c470 2023-10-25T00:06:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2876ac8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287006caa0 con 0x7f287006f490 2023-10-25T00:06:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2876ac8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28700c73c0 con 0x7f2870152b30 2023-10-25T00:06:07.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f287014c470 0x7f28700c1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:07.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f287014c470 0x7f28700c1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42846/0 (socket says 172.21.15.107:42846) 2023-10-25T00:06:07.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286ffff640 1 -- 172.21.15.107:0/1190919349 learned_addr learned my addr 172.21.15.107:0/1190919349 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:07.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2874864640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287006f490 0x7f2870143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:07.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2875065640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 0x7f28701ec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:07.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286ffff640 1 -- 172.21.15.107:0/1190919349 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 msgr2=0x7f28701ec840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:07.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286ffff640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 0x7f28701ec840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:07.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286ffff640 1 -- 172.21.15.107:0/1190919349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287006f490 msgr2=0x7f2870143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:07.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286ffff640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287006f490 0x7f2870143870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:07.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286ffff640 1 -- 172.21.15.107:0/1190919349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2868009b30 con 0x7f287014c470 2023-10-25T00:06:07.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2875065640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 0x7f28701ec840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:06:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2874864640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287006f490 0x7f2870143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:06:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286ffff640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f287014c470 0x7f28700c1310 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f285c019f90 tx=0x7f285c019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286d7fa640 1 -- 172.21.15.107:0/1190919349 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c0b4400 con 0x7f287014c470 2023-10-25T00:06:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286d7fa640 1 -- 172.21.15.107:0/1190919349 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285c0b4580 con 0x7f287014c470 2023-10-25T00:06:07.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2876ac8640 1 -- 172.21.15.107:0/1190919349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285c009420 con 0x7f287014c470 2023-10-25T00:06:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f286d7fa640 1 -- 172.21.15.107:0/1190919349 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285c0b4930 con 0x7f287014c470 2023-10-25T00:06:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.612+0000 7f2876ac8640 1 -- 172.21.15.107:0/1190919349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28701f0710 con 0x7f287014c470 2023-10-25T00:06:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.616+0000 7f2876ac8640 1 -- 172.21.15.107:0/1190919349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2870150a10 con 0x7f287014c470 2023-10-25T00:06:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.616+0000 7f286d7fa640 1 -- 172.21.15.107:0/1190919349 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f285c0b2040 con 0x7f287014c470 2023-10-25T00:06:07.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.616+0000 7f286d7fa640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2854065e60 0x7f2854068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:07.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.616+0000 7f286d7fa640 1 -- 172.21.15.107:0/1190919349 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7f285c11deb0 con 0x7f287014c470 2023-10-25T00:06:07.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.616+0000 7f2874864640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2854065e60 0x7f2854068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:07.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.616+0000 7f2874864640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2854065e60 0x7f2854068320 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2864001710 tx=0x7f2864009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:07.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.620+0000 7f286d7fa640 1 -- 172.21.15.107:0/1190919349 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f285c0ea9d0 con 0x7f287014c470 2023-10-25T00:06:07.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.860+0000 7f2876ac8640 1 -- 172.21.15.107:0/1190919349 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28701f08d0 con 0x7f287014c470 2023-10-25T00:06:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.860+0000 7f286d7fa640 1 -- 172.21.15.107:0/1190919349 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f285c0edff0 con 0x7f287014c470 2023-10-25T00:06:07.862 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:07.862 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:56.785175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 120 2023-10-25T00:06:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 -- 172.21.15.107:0/1190919349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2854065e60 msgr2=0x7f2854068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2854065e60 0x7f2854068320 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2864001710 tx=0x7f2864009040 comp rx=0 tx=0).stop 2023-10-25T00:06:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 -- 172.21.15.107:0/1190919349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f287014c470 msgr2=0x7f28700c1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f287014c470 0x7f28700c1310 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f285c019f90 tx=0x7f285c019100 comp rx=0 tx=0).stop 2023-10-25T00:06:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 -- 172.21.15.107:0/1190919349 shutdown_connections 2023-10-25T00:06:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2854065e60 0x7f2854068320 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2870152b30 0x7f28701ec840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f287014c470 0x7f28700c1310 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 --2- 172.21.15.107:0/1190919349 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f287006f490 0x7f2870143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 -- 172.21.15.107:0/1190919349 >> 172.21.15.107:0/1190919349 conn(0x7f2870070f60 msgr2=0x7f2870144690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:07.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 -- 172.21.15.107:0/1190919349 shutdown_connections 2023-10-25T00:06:07.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:07.864+0000 7f2876ac8640 1 -- 172.21.15.107:0/1190919349 wait complete. 2023-10-25T00:06:07.889 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2023-10-25T00:06:07.890 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:06:07.890 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:06:07.900 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:06:07.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:08.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd85b5640 1 Processor -- start 2023-10-25T00:06:08.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd85b5640 1 -- start start 2023-10-25T00:06:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd85b5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efdd014c470 0x7efdd014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd85b5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd0155cd0 con 0x7efdd006f490 2023-10-25T00:06:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd85b5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd00c4380 con 0x7efdd0152b30 2023-10-25T00:06:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd85b5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdd01526e0 con 0x7efdd014c470 2023-10-25T00:06:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd6351640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efdd014c470 0x7efdd014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd6351640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efdd014c470 0x7efdd014c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42850/0 (socket says 172.21.15.107:42850) 2023-10-25T00:06:08.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd6351640 1 -- 172.21.15.107:0/2440260171 learned_addr learned my addr 172.21.15.107:0/2440260171 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd6351640 1 -- 172.21.15.107:0/2440260171 >> v1:172.21.15.110:6789/0 conn(0x7efdd0152b30 legacy=0x7efdd0152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd6351640 1 -- 172.21.15.107:0/2440260171 >> v1:172.21.15.107:6789/0 conn(0x7efdd006f490 legacy=0x7efdd0071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:08.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.064+0000 7efdd6351640 1 -- 172.21.15.107:0/2440260171 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd00c48b0 con 0x7efdd014c470 2023-10-25T00:06:08.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.068+0000 7efdd6351640 1 --2- 172.21.15.107:0/2440260171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd014c470 0x7efdd014c850 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7efdcc009870 tx=0x7efdcc01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c77872684c2416 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:08.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.068+0000 7efdd534f640 1 -- 172.21.15.107:0/2440260171 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdcc0ab600 con 0x7efdd014c470 2023-10-25T00:06:08.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.068+0000 7efdd534f640 1 -- 172.21.15.107:0/2440260171 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdcc0a7050 con 0x7efdd014c470 2023-10-25T00:06:08.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.068+0000 7efdd534f640 1 -- 172.21.15.107:0/2440260171 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdcc0ab980 con 0x7efdd014c470 2023-10-25T00:06:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.068+0000 7efdd85b5640 1 -- 172.21.15.107:0/2440260171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd014c470 msgr2=0x7efdd014c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.068+0000 7efdd85b5640 1 --2- 172.21.15.107:0/2440260171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd014c470 0x7efdd014c850 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7efdcc009870 tx=0x7efdcc01c420 comp rx=0 tx=0).stop 2023-10-25T00:06:08.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.068+0000 7efdd85b5640 1 -- 172.21.15.107:0/2440260171 shutdown_connections 2023-10-25T00:06:08.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.068+0000 7efdd85b5640 1 --2- 172.21.15.107:0/2440260171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd014c470 0x7efdd014c850 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:08.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.068+0000 7efdd85b5640 1 -- 172.21.15.107:0/2440260171 >> 172.21.15.107:0/2440260171 conn(0x7efdd0070f60 msgr2=0x7efdd0071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:08.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.068+0000 7efdd85b5640 1 -- 172.21.15.107:0/2440260171 shutdown_connections 2023-10-25T00:06:08.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.068+0000 7efdd85b5640 1 -- 172.21.15.107:0/2440260171 wait complete. 2023-10-25T00:06:08.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd85b5640 1 Processor -- start 2023-10-25T00:06:08.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd85b5640 1 -- start start 2023-10-25T00:06:08.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd85b5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdd006f490 0x7efdd006c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:08.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd85b5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd014c470 0x7efdd00c1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:08.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd85b5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0152b30 0x7efdd00c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:08.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd85b5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd0144c10 con 0x7efdd0152b30 2023-10-25T00:06:08.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd85b5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd006b3d0 con 0x7efdd006f490 2023-10-25T00:06:08.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd85b5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd01446e0 con 0x7efdd014c470 2023-10-25T00:06:08.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd6b52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0152b30 0x7efdd00c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:08.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd6b52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0152b30 0x7efdd00c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42860/0 (socket says 172.21.15.107:42860) 2023-10-25T00:06:08.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd6b52640 1 -- 172.21.15.107:0/937383785 learned_addr learned my addr 172.21.15.107:0/937383785 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:08.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd6351640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdd006f490 0x7efdd006c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:08.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd5b50640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd014c470 0x7efdd00c1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd6b52640 1 -- 172.21.15.107:0/937383785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd014c470 msgr2=0x7efdd00c1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd6b52640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd014c470 0x7efdd00c1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd6b52640 1 -- 172.21.15.107:0/937383785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdd006f490 msgr2=0x7efdd006c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd6b52640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdd006f490 0x7efdd006c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:08.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd6b52640 1 -- 172.21.15.107:0/937383785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc0094d0 con 0x7efdd0152b30 2023-10-25T00:06:08.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd6351640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdd006f490 0x7efdd006c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:06:08.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd6b52640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0152b30 0x7efdd00c7430 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7efdc800c760 tx=0x7efdc800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:08.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdc6ffd640 1 -- 172.21.15.107:0/937383785 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8003b80 con 0x7efdd0152b30 2023-10-25T00:06:08.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdc6ffd640 1 -- 172.21.15.107:0/937383785 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc8003ce0 con 0x7efdd0152b30 2023-10-25T00:06:08.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd85b5640 1 -- 172.21.15.107:0/937383785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd0142960 con 0x7efdd0152b30 2023-10-25T00:06:08.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdc6ffd640 1 -- 172.21.15.107:0/937383785 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc800aa30 con 0x7efdd0152b30 2023-10-25T00:06:08.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd85b5640 1 -- 172.21.15.107:0/937383785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd01f0b20 con 0x7efdd0152b30 2023-10-25T00:06:08.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.072+0000 7efdd85b5640 1 -- 172.21.15.107:0/937383785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd01f0980 con 0x7efdd0152b30 2023-10-25T00:06:08.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.076+0000 7efdc6ffd640 1 -- 172.21.15.107:0/937383785 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7efdc800ab90 con 0x7efdd0152b30 2023-10-25T00:06:08.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.076+0000 7efdc6ffd640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efdb4065e60 0x7efdb4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:08.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.076+0000 7efdc6ffd640 1 -- 172.21.15.107:0/937383785 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7efdc8083fc0 con 0x7efdd0152b30 2023-10-25T00:06:08.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.076+0000 7efdd6351640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efdb4065e60 0x7efdb4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:08.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.076+0000 7efdd6351640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efdb4065e60 0x7efdb4068320 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0ad000 tx=0x7efdcc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:08.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.080+0000 7efdc6ffd640 1 -- 172.21.15.107:0/937383785 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7efdc80b5040 con 0x7efdd0152b30 2023-10-25T00:06:08.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 -- 172.21.15.107:0/937383785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdd00746e0 con 0x7efdd0152b30 2023-10-25T00:06:08.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdc6ffd640 1 -- 172.21.15.107:0/937383785 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5774 (secure 0 0 0) 0x7efdd01f0980 con 0x7efdd0152b30 2023-10-25T00:06:08.299 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:08.300 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:56.785175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:08.300 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 120 2023-10-25T00:06:08.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 -- 172.21.15.107:0/937383785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efdb4065e60 msgr2=0x7efdb4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:08.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efdb4065e60 0x7efdb4068320 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0ad000 tx=0x7efdcc008040 comp rx=0 tx=0).stop 2023-10-25T00:06:08.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 -- 172.21.15.107:0/937383785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0152b30 msgr2=0x7efdd00c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:08.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0152b30 0x7efdd00c7430 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7efdc800c760 tx=0x7efdc800cc20 comp rx=0 tx=0).stop 2023-10-25T00:06:08.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 -- 172.21.15.107:0/937383785 shutdown_connections 2023-10-25T00:06:08.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdd0152b30 0x7efdd00c7430 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:08.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efdb4065e60 0x7efdb4068320 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:08.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efdd014c470 0x7efdd00c1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:08.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 --2- 172.21.15.107:0/937383785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efdd006f490 0x7efdd006c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:08.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 -- 172.21.15.107:0/937383785 >> 172.21.15.107:0/937383785 conn(0x7efdd0070f60 msgr2=0x7efdd0150ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:08.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 -- 172.21.15.107:0/937383785 shutdown_connections 2023-10-25T00:06:08.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:08.300+0000 7efdd85b5640 1 -- 172.21.15.107:0/937383785 wait complete. 2023-10-25T00:06:08.322 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:06:10.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:10.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.452+0000 7fc37eda9640 1 Processor -- start 2023-10-25T00:06:10.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37eda9640 1 -- start start 2023-10-25T00:06:10.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37eda9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc37806af70 0x7fc37806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:10.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37eda9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc37806a230 0x7fc3781551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:10.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37eda9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3780c1860 con 0x7fc37814b600 2023-10-25T00:06:10.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37eda9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc378154d50 con 0x7fc37806a230 2023-10-25T00:06:10.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37eda9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37814b9e0 con 0x7fc37806af70 2023-10-25T00:06:10.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37cb45640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc37814b600 0x7fc378151fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56740/0 (socket says 172.21.15.107:56740) 2023-10-25T00:06:10.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37cb45640 1 -- 172.21.15.107:0/258584337 learned_addr learned my addr 172.21.15.107:0/258584337 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:06:10.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc377fff640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc37806af70 0x7fc37806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:10.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37d346640 1 --2- 172.21.15.107:0/258584337 >> v2:172.21.15.110:3300/0 conn(0x7fc37806a230 0x7fc3781551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:10.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953079084 0 0) 0x7fc3780c1860 con 0x7fc37814b600 2023-10-25T00:06:10.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc368003530 con 0x7fc37814b600 2023-10-25T00:06:10.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2766303739 0 0) 0x7fc368003530 con 0x7fc37814b600 2023-10-25T00:06:10.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc37806af70 msgr2=0x7fc37806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:10.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc37806af70 0x7fc37806c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:10.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 >> v2:172.21.15.110:3300/0 conn(0x7fc37806a230 msgr2=0x7fc3781551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:10.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 --2- 172.21.15.107:0/258584337 >> v2:172.21.15.110:3300/0 conn(0x7fc37806a230 0x7fc3781551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:10.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780c1ab0 con 0x7fc37814b600 2023-10-25T00:06:10.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fc36c001680 con 0x7fc37814b600 2023-10-25T00:06:10.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 >> v1:172.21.15.107:6789/0 conn(0x7fc37814b600 legacy=0x7fc378151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:10.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc368003980 0x7fc368003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:10.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3680047d0 0x7fc368004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:10.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc37806af70 0x7fc368005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:10.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc378154d50 con 0x7fc3680047d0 2023-10-25T00:06:10.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37814b9e0 con 0x7fc368003980 2023-10-25T00:06:10.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36800ba40 con 0x7fc37806af70 2023-10-25T00:06:10.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc377fff640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3680047d0 0x7fc368004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:10.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37cb45640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc368003980 0x7fc368003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:10.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc377fff640 1 -- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc37806af70 msgr2=0x7fc368005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:10.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37d346640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc37806af70 0x7fc368005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:10.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc377fff640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc37806af70 0x7fc368005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:10.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc377fff640 1 -- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc368003980 msgr2=0x7fc368003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:10.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc377fff640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc368003980 0x7fc368003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:10.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc377fff640 1 -- 172.21.15.107:0/258584337 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc368003530 con 0x7fc3680047d0 2023-10-25T00:06:10.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.456+0000 7fc37cb45640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc368003980 0x7fc368003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:06:10.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc377fff640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3680047d0 0x7fc368004c50 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fc37806b390 tx=0x7fc36401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c696eec2a64f29 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:10.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3640a8430 con 0x7fc3680047d0 2023-10-25T00:06:10.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3640a8590 con 0x7fc3680047d0 2023-10-25T00:06:10.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc3777fe640 1 -- 172.21.15.107:0/258584337 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3640a8740 con 0x7fc3680047d0 2023-10-25T00:06:10.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 -- 172.21.15.107:0/258584337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3680047d0 msgr2=0x7fc368004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:10.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 --2- 172.21.15.107:0/258584337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3680047d0 0x7fc368004c50 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fc37806b390 tx=0x7fc36401c8e0 comp rx=0 tx=0).stop 2023-10-25T00:06:10.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37d346640 1 -- 172.21.15.107:0/258584337 reap_dead start 2023-10-25T00:06:10.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 -- 172.21.15.107:0/258584337 shutdown_connections 2023-10-25T00:06:10.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 -- 172.21.15.107:0/258584337 >> 172.21.15.107:0/258584337 conn(0x7fc3780c2950 msgr2=0x7fc3780c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:10.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 -- 172.21.15.107:0/258584337 shutdown_connections 2023-10-25T00:06:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 -- 172.21.15.107:0/258584337 wait complete. 2023-10-25T00:06:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 Processor -- start 2023-10-25T00:06:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 -- start start 2023-10-25T00:06:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc3680047d0 0x7fc378152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc378070d00 0x7fc378071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc378143ca0 0x7fc3781ec840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc378072020 con 0x7fc378070d00 2023-10-25T00:06:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc378152fa0 con 0x7fc378143ca0 2023-10-25T00:06:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc378153280 con 0x7fc3680047d0 2023-10-25T00:06:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc377fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc378070d00 0x7fc378071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc377fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc378070d00 0x7fc378071ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42874/0 (socket says 172.21.15.107:42874) 2023-10-25T00:06:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc377fff640 1 -- 172.21.15.107:0/3784431856 learned_addr learned my addr 172.21.15.107:0/3784431856 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37cb45640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc3680047d0 0x7fc378152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37d346640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc378143ca0 0x7fc3781ec840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc377fff640 1 -- 172.21.15.107:0/3784431856 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc3680047d0 msgr2=0x7fc378152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc377fff640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc3680047d0 0x7fc378152910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc377fff640 1 -- 172.21.15.107:0/3784431856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc378143ca0 msgr2=0x7fc3781ec840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc377fff640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc378143ca0 0x7fc3781ec840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc377fff640 1 -- 172.21.15.107:0/3784431856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3780712e0 con 0x7fc378070d00 2023-10-25T00:06:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc377fff640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc378070d00 0x7fc378071ae0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fc36401c710 tx=0x7fc364019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc3757fa640 1 -- 172.21.15.107:0/3784431856 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36400f000 con 0x7fc378070d00 2023-10-25T00:06:10.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc3757fa640 1 -- 172.21.15.107:0/3784431856 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3640a8db0 con 0x7fc378070d00 2023-10-25T00:06:10.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 -- 172.21.15.107:0/3784431856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc378072020 con 0x7fc378070d00 2023-10-25T00:06:10.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc3757fa640 1 -- 172.21.15.107:0/3784431856 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc364004260 con 0x7fc378070d00 2023-10-25T00:06:10.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.460+0000 7fc37eda9640 1 -- 172.21.15.107:0/3784431856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3780c0e40 con 0x7fc378070d00 2023-10-25T00:06:10.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.464+0000 7fc37eda9640 1 -- 172.21.15.107:0/3784431856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3780746e0 con 0x7fc378070d00 2023-10-25T00:06:10.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.464+0000 7fc3757fa640 1 -- 172.21.15.107:0/3784431856 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc3640043c0 con 0x7fc378070d00 2023-10-25T00:06:10.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.464+0000 7fc3757fa640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc35c065e60 0x7fc35c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:10.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.464+0000 7fc3757fa640 1 -- 172.21.15.107:0/3784431856 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7fc36411e5f0 con 0x7fc378070d00 2023-10-25T00:06:10.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.464+0000 7fc37cb45640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc35c065e60 0x7fc35c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:10.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.464+0000 7fc37cb45640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc35c065e60 0x7fc35c068320 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc36c000f70 tx=0x7fc36c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:10.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.472+0000 7fc3757fa640 1 -- 172.21.15.107:0/3784431856 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc3640eb100 con 0x7fc378070d00 2023-10-25T00:06:10.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.664+0000 7fc37eda9640 1 -- 172.21.15.107:0/3784431856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc378069fa0 con 0x7fc378070d00 2023-10-25T00:06:10.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.664+0000 7fc3757fa640 1 -- 172.21.15.107:0/3784431856 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5774 (secure 0 0 0) 0x7fc3640ee720 con 0x7fc378070d00 2023-10-25T00:06:10.664 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:10.665 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:56.785175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:10.665 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 120 2023-10-25T00:06:10.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 -- 172.21.15.107:0/3784431856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc35c065e60 msgr2=0x7fc35c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:10.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc35c065e60 0x7fc35c068320 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc36c000f70 tx=0x7fc36c009040 comp rx=0 tx=0).stop 2023-10-25T00:06:10.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 -- 172.21.15.107:0/3784431856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc378070d00 msgr2=0x7fc378071ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:10.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc378070d00 0x7fc378071ae0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fc36401c710 tx=0x7fc364019680 comp rx=0 tx=0).stop 2023-10-25T00:06:10.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 -- 172.21.15.107:0/3784431856 shutdown_connections 2023-10-25T00:06:10.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc378143ca0 0x7fc3781ec840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:10.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc378070d00 0x7fc378071ae0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:10.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc35c065e60 0x7fc35c068320 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:10.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 --2- 172.21.15.107:0/3784431856 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc3680047d0 0x7fc378152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:10.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 -- 172.21.15.107:0/3784431856 >> 172.21.15.107:0/3784431856 conn(0x7fc3780c2950 msgr2=0x7fc378144e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:10.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 -- 172.21.15.107:0/3784431856 shutdown_connections 2023-10-25T00:06:10.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:10.668+0000 7fc37eda9640 1 -- 172.21.15.107:0/3784431856 wait complete. 2023-10-25T00:06:10.683 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:06:12.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:12.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.816+0000 7f39648b1640 1 Processor -- start 2023-10-25T00:06:12.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.816+0000 7f39648b1640 1 -- start start 2023-10-25T00:06:12.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f39648b1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f396014b600 0x7f3960151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f39648b1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f396006a230 0x7f39601551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f39648b1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39600c1860 con 0x7f396006af70 2023-10-25T00:06:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f39648b1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3960154d50 con 0x7f396014b600 2023-10-25T00:06:12.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f39648b1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39601524f0 con 0x7f396006a230 2023-10-25T00:06:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f395e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f396014b600 0x7f3960151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f395e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f396014b600 0x7f3960151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53806/0 (socket says 172.21.15.107:53806) 2023-10-25T00:06:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f395e59c640 1 -- 172.21.15.107:0/2260005534 learned_addr learned my addr 172.21.15.107:0/2260005534 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f395ed9d640 1 --2- 172.21.15.107:0/2260005534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f396006a230 0x7f39601551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f395e59c640 1 -- 172.21.15.107:0/2260005534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f396006a230 msgr2=0x7f39601551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:12.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f395e59c640 1 --2- 172.21.15.107:0/2260005534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f396006a230 0x7f39601551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:12.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f395e59c640 1 -- 172.21.15.107:0/2260005534 >> v1:172.21.15.107:6789/0 conn(0x7f396006af70 legacy=0x7f396006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:12.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f395e59c640 1 -- 172.21.15.107:0/2260005534 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396014b9e0 con 0x7f396014b600 2023-10-25T00:06:12.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f395d59a640 1 -- 172.21.15.107:0/2260005534 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189298722 0 0) 0x7f39600c1860 con 0x7f396006af70 2023-10-25T00:06:12.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.820+0000 7f395ed9d640 1 --2- 172.21.15.107:0/2260005534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f396006a230 0x7f39601551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:06:12.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f395e59c640 1 --2- 172.21.15.107:0/2260005534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396014b600 0x7f3960151fb0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f3948009870 tx=0x7f394801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b615f771e0ad1a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:12.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f395d59a640 1 -- 172.21.15.107:0/2260005534 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39480ad640 con 0x7f396014b600 2023-10-25T00:06:12.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f395d59a640 1 -- 172.21.15.107:0/2260005534 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39480ad7a0 con 0x7f396014b600 2023-10-25T00:06:12.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f395d59a640 1 -- 172.21.15.107:0/2260005534 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39480adb30 con 0x7f396014b600 2023-10-25T00:06:12.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 -- 172.21.15.107:0/2260005534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396014b600 msgr2=0x7f3960151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:12.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 --2- 172.21.15.107:0/2260005534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396014b600 0x7f3960151fb0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f3948009870 tx=0x7f394801c480 comp rx=0 tx=0).stop 2023-10-25T00:06:12.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 -- 172.21.15.107:0/2260005534 shutdown_connections 2023-10-25T00:06:12.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 --2- 172.21.15.107:0/2260005534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f396006a230 0x7f39601551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:12.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 --2- 172.21.15.107:0/2260005534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396014b600 0x7f3960151fb0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:12.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 -- 172.21.15.107:0/2260005534 >> 172.21.15.107:0/2260005534 conn(0x7f39600c2950 msgr2=0x7f39600c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:12.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 -- 172.21.15.107:0/2260005534 shutdown_connections 2023-10-25T00:06:12.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 -- 172.21.15.107:0/2260005534 wait complete. 2023-10-25T00:06:12.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 Processor -- start 2023-10-25T00:06:12.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 -- start start 2023-10-25T00:06:12.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f396006a230 0x7f39601eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:12.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396006af70 0x7f3960159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:12.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f396014b600 0x7f3960146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:12.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39601ebe50 con 0x7f396006af70 2023-10-25T00:06:12.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39601ec130 con 0x7f396014b600 2023-10-25T00:06:12.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f39648b1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39601ec410 con 0x7f396006a230 2023-10-25T00:06:12.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f395dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396006af70 0x7f3960159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:12.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f395dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396006af70 0x7f3960159590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53816/0 (socket says 172.21.15.107:53816) 2023-10-25T00:06:12.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f395dd9b640 1 -- 172.21.15.107:0/1013942993 learned_addr learned my addr 172.21.15.107:0/1013942993 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:12.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.824+0000 7f395e59c640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f396006a230 0x7f39601eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:12.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f395ed9d640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f396014b600 0x7f3960146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:12.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f395dd9b640 1 -- 172.21.15.107:0/1013942993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f396006a230 msgr2=0x7f39601eb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:12.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f395dd9b640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f396006a230 0x7f39601eb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:12.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f395dd9b640 1 -- 172.21.15.107:0/1013942993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f396014b600 msgr2=0x7f3960146a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:12.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f395dd9b640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f396014b600 0x7f3960146a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:12.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f395dd9b640 1 -- 172.21.15.107:0/1013942993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39480094d0 con 0x7f396006af70 2023-10-25T00:06:12.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f395dd9b640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396006af70 0x7f3960159590 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f395000c820 tx=0x7f395000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:12.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f3956ffd640 1 -- 172.21.15.107:0/1013942993 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950003db0 con 0x7f396006af70 2023-10-25T00:06:12.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f3956ffd640 1 -- 172.21.15.107:0/1013942993 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39500050f0 con 0x7f396006af70 2023-10-25T00:06:12.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f39648b1640 1 -- 172.21.15.107:0/1013942993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39601ec6f0 con 0x7f396006af70 2023-10-25T00:06:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f3956ffd640 1 -- 172.21.15.107:0/1013942993 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39500054a0 con 0x7f396006af70 2023-10-25T00:06:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f39648b1640 1 -- 172.21.15.107:0/1013942993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3960159ad0 con 0x7f396006af70 2023-10-25T00:06:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f39648b1640 1 -- 172.21.15.107:0/1013942993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39601ec930 con 0x7f396006af70 2023-10-25T00:06:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f3956ffd640 1 -- 172.21.15.107:0/1013942993 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3950005600 con 0x7f396006af70 2023-10-25T00:06:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f3956ffd640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3940065e90 0x7f3940068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f3956ffd640 1 -- 172.21.15.107:0/1013942993 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7f3950083b70 con 0x7f396006af70 2023-10-25T00:06:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f395e59c640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3940065e90 0x7f3940068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.828+0000 7f395e59c640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3940065e90 0x7f3940068350 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f39480af000 tx=0x7f3948009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:12.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:12.832+0000 7f3956ffd640 1 -- 172.21.15.107:0/1013942993 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f39601ec930 con 0x7f396006af70 2023-10-25T00:06:13.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.020+0000 7f39648b1640 1 -- 172.21.15.107:0/1013942993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39601ebf90 con 0x7f396006af70 2023-10-25T00:06:13.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f3956ffd640 1 -- 172.21.15.107:0/1013942993 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f39601ebf90 con 0x7f396006af70 2023-10-25T00:06:13.022 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:13.023 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:56.785175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:13.023 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 120 2023-10-25T00:06:13.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 -- 172.21.15.107:0/1013942993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3940065e90 msgr2=0x7f3940068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3940065e90 0x7f3940068350 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f39480af000 tx=0x7f3948009040 comp rx=0 tx=0).stop 2023-10-25T00:06:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 -- 172.21.15.107:0/1013942993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396006af70 msgr2=0x7f3960159590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396006af70 0x7f3960159590 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f395000c820 tx=0x7f395000cce0 comp rx=0 tx=0).stop 2023-10-25T00:06:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 -- 172.21.15.107:0/1013942993 shutdown_connections 2023-10-25T00:06:13.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f396014b600 0x7f3960146a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f396006af70 0x7f3960159590 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3940065e90 0x7f3940068350 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 --2- 172.21.15.107:0/1013942993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f396006a230 0x7f39601eb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 -- 172.21.15.107:0/1013942993 >> 172.21.15.107:0/1013942993 conn(0x7f39600c2950 msgr2=0x7f396006b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 -- 172.21.15.107:0/1013942993 shutdown_connections 2023-10-25T00:06:13.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:13.024+0000 7f39648b1640 1 -- 172.21.15.107:0/1013942993 wait complete. 2023-10-25T00:06:13.041 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:06:15.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:15.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f425ed2d640 1 Processor -- start 2023-10-25T00:06:15.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f425ed2d640 1 -- start start 2023-10-25T00:06:15.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f425ed2d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f425806af70 0x7f425806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:15.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f425ed2d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42580c1860 con 0x7f425806a230 2023-10-25T00:06:15.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f425ed2d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4258154d50 con 0x7f425814b600 2023-10-25T00:06:15.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f425ed2d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425814b9e0 con 0x7f425806af70 2023-10-25T00:06:15.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f425d2ca640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f425806a230 0x7f42581551e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33872/0 (socket says 172.21.15.107:33872) 2023-10-25T00:06:15.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f425d2ca640 1 -- 172.21.15.107:0/2318583935 learned_addr learned my addr 172.21.15.107:0/2318583935 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:06:15.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f4257fff640 1 --2- 172.21.15.107:0/2318583935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f425806af70 0x7f425806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:15.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f42577fe640 1 -- 172.21.15.107:0/2318583935 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161459400 0 0) 0x7f42580c1860 con 0x7f425806a230 2023-10-25T00:06:15.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f42577fe640 1 -- 172.21.15.107:0/2318583935 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4248003530 con 0x7f425806a230 2023-10-25T00:06:15.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f4257fff640 1 -- 172.21.15.107:0/2318583935 >> v1:172.21.15.110:6789/0 conn(0x7f425814b600 legacy=0x7f4258151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:15.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f4257fff640 1 -- 172.21.15.107:0/2318583935 >> v1:172.21.15.107:6789/0 conn(0x7f425806a230 legacy=0x7f42581551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:15.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f42577fe640 1 -- 172.21.15.107:0/2318583935 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 812230413 0 0) 0x7f4248003530 con 0x7f425806a230 2023-10-25T00:06:15.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.172+0000 7f4257fff640 1 -- 172.21.15.107:0/2318583935 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42580c1ab0 con 0x7f425806af70 2023-10-25T00:06:15.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f4257fff640 1 --2- 172.21.15.107:0/2318583935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f425806af70 0x7f425806c4c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f42440097f0 tx=0x7f424401c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86c46bd577f7c1ff server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:15.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f42577fe640 1 -- 172.21.15.107:0/2318583935 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250001a00 con 0x7f425806af70 2023-10-25T00:06:15.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f42577fe640 1 -- 172.21.15.107:0/2318583935 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42440a7050 con 0x7f425806af70 2023-10-25T00:06:15.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f42577fe640 1 -- 172.21.15.107:0/2318583935 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42440ab600 con 0x7f425806af70 2023-10-25T00:06:15.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 -- 172.21.15.107:0/2318583935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f425806af70 msgr2=0x7f425806c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:15.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 --2- 172.21.15.107:0/2318583935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f425806af70 0x7f425806c4c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f42440097f0 tx=0x7f424401c320 comp rx=0 tx=0).stop 2023-10-25T00:06:15.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 -- 172.21.15.107:0/2318583935 shutdown_connections 2023-10-25T00:06:15.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 --2- 172.21.15.107:0/2318583935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f425806af70 0x7f425806c4c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:15.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 -- 172.21.15.107:0/2318583935 >> 172.21.15.107:0/2318583935 conn(0x7f42580c2950 msgr2=0x7f42580c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:15.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 -- 172.21.15.107:0/2318583935 shutdown_connections 2023-10-25T00:06:15.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 -- 172.21.15.107:0/2318583935 wait complete. 2023-10-25T00:06:15.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 Processor -- start 2023-10-25T00:06:15.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 -- start start 2023-10-25T00:06:15.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f425806a230 0x7f42581eff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:15.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f425806af70 0x7f4258143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:15.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f425814b600 0x7f4258143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:15.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4258156180 con 0x7f425806af70 2023-10-25T00:06:15.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4258144950 con 0x7f425814b600 2023-10-25T00:06:15.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425ed2d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4258152200 con 0x7f425806a230 2023-10-25T00:06:15.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f4257fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f425806af70 0x7f4258143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:15.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f4257fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f425806af70 0x7f4258143ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53832/0 (socket says 172.21.15.107:53832) 2023-10-25T00:06:15.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f4257fff640 1 -- 172.21.15.107:0/3353474186 learned_addr learned my addr 172.21.15.107:0/3353474186 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:15.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425cac9640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f425806a230 0x7f42581eff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:15.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.176+0000 7f425d2ca640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f425814b600 0x7f4258143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f4257fff640 1 -- 172.21.15.107:0/3353474186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f425806a230 msgr2=0x7f42581eff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f4257fff640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f425806a230 0x7f42581eff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f4257fff640 1 -- 172.21.15.107:0/3353474186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f425814b600 msgr2=0x7f4258143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f4257fff640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f425814b600 0x7f4258143ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f4257fff640 1 -- 172.21.15.107:0/3353474186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4244009420 con 0x7f425806af70 2023-10-25T00:06:15.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f4257fff640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f425806af70 0x7f4258143ab0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4244000c10 tx=0x7f42440190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f42557fa640 1 -- 172.21.15.107:0/3353474186 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244003b30 con 0x7f425806af70 2023-10-25T00:06:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f42557fa640 1 -- 172.21.15.107:0/3353474186 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4244003c90 con 0x7f425806af70 2023-10-25T00:06:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f42557fa640 1 -- 172.21.15.107:0/3353474186 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42440ab400 con 0x7f425806af70 2023-10-25T00:06:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f425ed2d640 1 -- 172.21.15.107:0/3353474186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42581524e0 con 0x7f425806af70 2023-10-25T00:06:15.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f425ed2d640 1 -- 172.21.15.107:0/3353474186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42581f0470 con 0x7f425806af70 2023-10-25T00:06:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f425ed2d640 1 -- 172.21.15.107:0/3353474186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42580746e0 con 0x7f425806af70 2023-10-25T00:06:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f42557fa640 1 -- 172.21.15.107:0/3353474186 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f42440b2040 con 0x7f425806af70 2023-10-25T00:06:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f42557fa640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f423c065e60 0x7f423c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f42557fa640 1 -- 172.21.15.107:0/3353474186 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7f424411e240 con 0x7f425806af70 2023-10-25T00:06:15.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f425cac9640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f423c065e60 0x7f423c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.180+0000 7f425cac9640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f423c065e60 0x7f423c068320 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f424c007fd0 tx=0x7f424c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.184+0000 7f42557fa640 1 -- 172.21.15.107:0/3353474186 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f42440eae00 con 0x7f425806af70 2023-10-25T00:06:15.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.372+0000 7f425ed2d640 1 -- 172.21.15.107:0/3353474186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42581f0950 con 0x7f425806af70 2023-10-25T00:06:15.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f42557fa640 1 -- 172.21.15.107:0/3353474186 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f42440ee420 con 0x7f425806af70 2023-10-25T00:06:15.374 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:15.374 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:56.785175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:15.375 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 120 2023-10-25T00:06:15.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 -- 172.21.15.107:0/3353474186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f423c065e60 msgr2=0x7f423c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:15.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f423c065e60 0x7f423c068320 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f424c007fd0 tx=0x7f424c009040 comp rx=0 tx=0).stop 2023-10-25T00:06:15.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 -- 172.21.15.107:0/3353474186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f425806af70 msgr2=0x7f4258143ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f425806af70 0x7f4258143ab0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4244000c10 tx=0x7f42440190c0 comp rx=0 tx=0).stop 2023-10-25T00:06:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 -- 172.21.15.107:0/3353474186 shutdown_connections 2023-10-25T00:06:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f425814b600 0x7f4258143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f425806af70 0x7f4258143ab0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:15.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f423c065e60 0x7f423c068320 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:15.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 --2- 172.21.15.107:0/3353474186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f425806a230 0x7f42581eff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:15.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 -- 172.21.15.107:0/3353474186 >> 172.21.15.107:0/3353474186 conn(0x7f42580c2950 msgr2=0x7f42581482b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:15.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 -- 172.21.15.107:0/3353474186 shutdown_connections 2023-10-25T00:06:15.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:15.376+0000 7f425ed2d640 1 -- 172.21.15.107:0/3353474186 wait complete. 2023-10-25T00:06:15.391 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:06:17.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:17.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.524+0000 7f77a3ef0640 1 Processor -- start 2023-10-25T00:06:17.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.524+0000 7f77a3ef0640 1 -- start start 2023-10-25T00:06:17.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a3ef0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c14fd40 0x7f779c0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:17.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a3ef0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c06f730 con 0x7f779c0a9820 2023-10-25T00:06:17.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a3ef0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c156a40 con 0x7f779c06ee40 2023-10-25T00:06:17.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a3ef0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779c156710 con 0x7f779c14fd40 2023-10-25T00:06:17.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a148b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f779c0a9820 0x7f779c0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33876/0 (socket says 172.21.15.107:33876) 2023-10-25T00:06:17.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a148b640 1 -- 172.21.15.107:0/2629880250 learned_addr learned my addr 172.21.15.107:0/2629880250 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:06:17.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a1c8c640 1 --2- 172.21.15.107:0/2629880250 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c14fd40 0x7f779c0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:17.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784252124 0 0) 0x7f779c06f730 con 0x7f779c0a9820 2023-10-25T00:06:17.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7788003530 con 0x7f779c0a9820 2023-10-25T00:06:17.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 311705815 0 0) 0x7f7788003530 con 0x7f779c0a9820 2023-10-25T00:06:17.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c14fd40 msgr2=0x7f779c0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:17.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 --2- 172.21.15.107:0/2629880250 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c14fd40 0x7f779c0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:17.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 >> v1:172.21.15.110:6789/0 conn(0x7f779c06ee40 legacy=0x7f779c15cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:17.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c06f980 con 0x7f779c0a9820 2023-10-25T00:06:17.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f778c001680 con 0x7f779c0a9820 2023-10-25T00:06:17.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 >> v1:172.21.15.107:6789/0 conn(0x7f779c0a9820 legacy=0x7f779c0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:17.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 --2- 172.21.15.107:0/2629880250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77880039b0 0x7f7788003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:17.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 --2- 172.21.15.107:0/2629880250 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c14fd40 0x7f77880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:17.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 --2- 172.21.15.107:0/2629880250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7788004d10 0x7f7788005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:17.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c156710 con 0x7f77880039b0 2023-10-25T00:06:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778800bb50 con 0x7f7788004d10 2023-10-25T00:06:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778800bd10 con 0x7f779c14fd40 2023-10-25T00:06:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a1c8c640 1 --2- 172.21.15.107:0/2629880250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77880039b0 0x7f7788003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a1c8c640 1 -- 172.21.15.107:0/2629880250 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c14fd40 msgr2=0x7f77880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:17.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a148b640 1 --2- 172.21.15.107:0/2629880250 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c14fd40 0x7f77880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a1c8c640 1 --2- 172.21.15.107:0/2629880250 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c14fd40 0x7f77880047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a1c8c640 1 -- 172.21.15.107:0/2629880250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7788004d10 msgr2=0x7f7788005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a1c8c640 1 --2- 172.21.15.107:0/2629880250 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7788004d10 0x7f7788005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:17.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.528+0000 7f77a1c8c640 1 -- 172.21.15.107:0/2629880250 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7788003530 con 0x7f77880039b0 2023-10-25T00:06:17.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a1c8c640 1 --2- 172.21.15.107:0/2629880250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77880039b0 0x7f7788003dd0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f7798009ae0 tx=0x7f779801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e297867f0cf7bf server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77980a5540 con 0x7f77880039b0 2023-10-25T00:06:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77980a56a0 con 0x7f77880039b0 2023-10-25T00:06:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a0c8a640 1 -- 172.21.15.107:0/2629880250 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77980a5a50 con 0x7f77880039b0 2023-10-25T00:06:17.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 -- 172.21.15.107:0/2629880250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77880039b0 msgr2=0x7f7788003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 --2- 172.21.15.107:0/2629880250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77880039b0 0x7f7788003dd0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f7798009ae0 tx=0x7f779801ca50 comp rx=0 tx=0).stop 2023-10-25T00:06:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a248d640 1 -- 172.21.15.107:0/2629880250 reap_dead start 2023-10-25T00:06:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 -- 172.21.15.107:0/2629880250 shutdown_connections 2023-10-25T00:06:17.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 -- 172.21.15.107:0/2629880250 >> 172.21.15.107:0/2629880250 conn(0x7f779c0c2950 msgr2=0x7f779c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 -- 172.21.15.107:0/2629880250 shutdown_connections 2023-10-25T00:06:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 -- 172.21.15.107:0/2629880250 wait complete. 2023-10-25T00:06:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 Processor -- start 2023-10-25T00:06:17.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 -- start start 2023-10-25T00:06:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77880039b0 0x7f779c0c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f779c0c34d0 0x7f779c154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c1ee920 0x7f779c1eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:17.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c1547b0 con 0x7f779c0c34d0 2023-10-25T00:06:17.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c155ab0 con 0x7f77880039b0 2023-10-25T00:06:17.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779c154440 con 0x7f779c1ee920 2023-10-25T00:06:17.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a148b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f779c0c34d0 0x7f779c154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:17.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a148b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f779c0c34d0 0x7f779c154c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53852/0 (socket says 172.21.15.107:53852) 2023-10-25T00:06:17.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a148b640 1 -- 172.21.15.107:0/218984491 learned_addr learned my addr 172.21.15.107:0/218984491 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:17.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a1c8c640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77880039b0 0x7f779c0c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:17.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a248d640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c1ee920 0x7f779c1eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:17.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a148b640 1 -- 172.21.15.107:0/218984491 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c1ee920 msgr2=0x7f779c1eeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a148b640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c1ee920 0x7f779c1eeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a148b640 1 -- 172.21.15.107:0/218984491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77880039b0 msgr2=0x7f779c0c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a148b640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77880039b0 0x7f779c0c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a148b640 1 -- 172.21.15.107:0/218984491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c0bb000 con 0x7f779c0c34d0 2023-10-25T00:06:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a248d640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c1ee920 0x7f779c1eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:06:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a1c8c640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77880039b0 0x7f779c0c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:06:17.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a148b640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f779c0c34d0 0x7f779c154c40 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f778c007fd0 tx=0x7f778c00dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77927fc640 1 -- 172.21.15.107:0/218984491 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c0074e0 con 0x7f779c0c34d0 2023-10-25T00:06:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77927fc640 1 -- 172.21.15.107:0/218984491 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778c007640 con 0x7f779c0c34d0 2023-10-25T00:06:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 -- 172.21.15.107:0/218984491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779c155d90 con 0x7f779c0c34d0 2023-10-25T00:06:17.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77927fc640 1 -- 172.21.15.107:0/218984491 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c0049f0 con 0x7f779c0c34d0 2023-10-25T00:06:17.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.532+0000 7f77a3ef0640 1 -- 172.21.15.107:0/218984491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f779c14ca20 con 0x7f779c0c34d0 2023-10-25T00:06:17.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.536+0000 7f77a3ef0640 1 -- 172.21.15.107:0/218984491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779c0c24a0 con 0x7f779c0c34d0 2023-10-25T00:06:17.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.536+0000 7f77927fc640 1 -- 172.21.15.107:0/218984491 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f778c004b50 con 0x7f779c0c34d0 2023-10-25T00:06:17.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.536+0000 7f77927fc640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7780065e60 0x7f7780068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:17.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.536+0000 7f77927fc640 1 -- 172.21.15.107:0/218984491 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7f778c083d10 con 0x7f779c0c34d0 2023-10-25T00:06:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.536+0000 7f77a1c8c640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7780065e60 0x7f7780068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.536+0000 7f77a1c8c640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7780065e60 0x7f7780068320 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f77980af000 tx=0x7f77980046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:17.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.540+0000 7f77927fc640 1 -- 172.21.15.107:0/218984491 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f778c089020 con 0x7f779c0c34d0 2023-10-25T00:06:17.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.732+0000 7f77a3ef0640 1 -- 172.21.15.107:0/218984491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f779c155bf0 con 0x7f779c0c34d0 2023-10-25T00:06:17.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77927fc640 1 -- 172.21.15.107:0/218984491 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f779c155bf0 con 0x7f779c0c34d0 2023-10-25T00:06:17.735 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:17.735 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:56.785175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:17.736 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 120 2023-10-25T00:06:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 -- 172.21.15.107:0/218984491 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7780065e60 msgr2=0x7f7780068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7780065e60 0x7f7780068320 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f77980af000 tx=0x7f77980046d0 comp rx=0 tx=0).stop 2023-10-25T00:06:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 -- 172.21.15.107:0/218984491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f779c0c34d0 msgr2=0x7f779c154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:17.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f779c0c34d0 0x7f779c154c40 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f778c007fd0 tx=0x7f778c00dc70 comp rx=0 tx=0).stop 2023-10-25T00:06:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 -- 172.21.15.107:0/218984491 shutdown_connections 2023-10-25T00:06:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7780065e60 0x7f7780068320 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f779c1ee920 0x7f779c1eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f779c0c34d0 0x7f779c154c40 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 --2- 172.21.15.107:0/218984491 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77880039b0 0x7f779c0c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 -- 172.21.15.107:0/218984491 >> 172.21.15.107:0/218984491 conn(0x7f779c0c2950 msgr2=0x7f779c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 -- 172.21.15.107:0/218984491 shutdown_connections 2023-10-25T00:06:17.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:17.736+0000 7f77a3ef0640 1 -- 172.21.15.107:0/218984491 wait complete. 2023-10-25T00:06:17.752 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:06:19.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:19.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.884+0000 7f9bcbb69640 1 Processor -- start 2023-10-25T00:06:19.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.884+0000 7f9bcbb69640 1 -- start start 2023-10-25T00:06:19.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bcbb69640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9bc40c8100 0x7f9bc40abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:19.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bcbb69640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 0x7f9bc40ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:19.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bcbb69640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc4155680 con 0x7f9bc4152040 2023-10-25T00:06:19.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bcbb69640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc4154a70 con 0x7f9bc40c8100 2023-10-25T00:06:19.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bcbb69640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc4154d50 con 0x7f9bc406a4b0 2023-10-25T00:06:19.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bc9905640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9bc4152040 0x7f9bc40c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:39694/0 (socket says 172.21.15.107:39694) 2023-10-25T00:06:19.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bc9905640 1 -- 172.21.15.107:0/3680753967 learned_addr learned my addr 172.21.15.107:0/3680753967 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:06:19.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bca106640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 0x7f9bc40ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:19.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bc9104640 1 --2- 172.21.15.107:0/3680753967 >> v2:172.21.15.110:3300/0 conn(0x7f9bc40c8100 0x7f9bc40abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:19.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bca106640 1 -- 172.21.15.107:0/3680753967 >> v2:172.21.15.110:3300/0 conn(0x7f9bc40c8100 msgr2=0x7f9bc40abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:19.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bca106640 1 --2- 172.21.15.107:0/3680753967 >> v2:172.21.15.110:3300/0 conn(0x7f9bc40c8100 0x7f9bc40abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:19.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bca106640 1 -- 172.21.15.107:0/3680753967 >> v1:172.21.15.110:6789/0 conn(0x7f9bc4152040 legacy=0x7f9bc40c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:19.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bca106640 1 -- 172.21.15.107:0/3680753967 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc4155030 con 0x7f9bc406a4b0 2023-10-25T00:06:19.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.888+0000 7f9bc9104640 1 --2- 172.21.15.107:0/3680753967 >> v2:172.21.15.110:3300/0 conn(0x7f9bc40c8100 0x7f9bc40abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:06:19.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bca106640 1 --2- 172.21.15.107:0/3680753967 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 0x7f9bc40ac240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00097c0 tx=0x7f9bc001c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e35b5b9df84e1105 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:19.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc8903640 1 -- 172.21.15.107:0/3680753967 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc00a4ea0 con 0x7f9bc406a4b0 2023-10-25T00:06:19.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc8903640 1 -- 172.21.15.107:0/3680753967 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc00a7050 con 0x7f9bc406a4b0 2023-10-25T00:06:19.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc8903640 1 -- 172.21.15.107:0/3680753967 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc00ab820 con 0x7f9bc406a4b0 2023-10-25T00:06:19.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3680753967 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 msgr2=0x7f9bc40ac240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:19.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 --2- 172.21.15.107:0/3680753967 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 0x7f9bc40ac240 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00097c0 tx=0x7f9bc001c3f0 comp rx=0 tx=0).stop 2023-10-25T00:06:19.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3680753967 shutdown_connections 2023-10-25T00:06:19.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 --2- 172.21.15.107:0/3680753967 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 0x7f9bc40ac240 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:19.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 --2- 172.21.15.107:0/3680753967 >> v2:172.21.15.110:3300/0 conn(0x7f9bc40c8100 0x7f9bc40abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:19.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3680753967 >> 172.21.15.107:0/3680753967 conn(0x7f9bc40ca200 msgr2=0x7f9bc40ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:19.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3680753967 shutdown_connections 2023-10-25T00:06:19.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3680753967 wait complete. 2023-10-25T00:06:19.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 Processor -- start 2023-10-25T00:06:19.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 -- start start 2023-10-25T00:06:19.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 0x7f9bc4151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:19.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc40c8100 0x7f9bc40aea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:19.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bc4152040 0x7f9bc41473e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:19.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc4150b50 con 0x7f9bc40c8100 2023-10-25T00:06:19.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc414f650 con 0x7f9bc4152040 2023-10-25T00:06:19.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bcbb69640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc414f090 con 0x7f9bc406a4b0 2023-10-25T00:06:19.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc9104640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc40c8100 0x7f9bc40aea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc9104640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc40c8100 0x7f9bc40aea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53862/0 (socket says 172.21.15.107:53862) 2023-10-25T00:06:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc9104640 1 -- 172.21.15.107:0/3911860230 learned_addr learned my addr 172.21.15.107:0/3911860230 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bca106640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bc4152040 0x7f9bc41473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc9905640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 0x7f9bc4151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc9104640 1 -- 172.21.15.107:0/3911860230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 msgr2=0x7f9bc4151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc9104640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 0x7f9bc4151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc9104640 1 -- 172.21.15.107:0/3911860230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bc4152040 msgr2=0x7f9bc41473e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc9104640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bc4152040 0x7f9bc41473e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc9104640 1 -- 172.21.15.107:0/3911860230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc009420 con 0x7f9bc40c8100 2023-10-25T00:06:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.892+0000 7f9bc9905640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 0x7f9bc4151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:06:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bc9104640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc40c8100 0x7f9bc40aea90 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00ebb0 tx=0x7f9bbc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bb67fc640 1 -- 172.21.15.107:0/3911860230 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc00eeb0 con 0x7f9bc40c8100 2023-10-25T00:06:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bb67fc640 1 -- 172.21.15.107:0/3911860230 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bbc00cc30 con 0x7f9bc40c8100 2023-10-25T00:06:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3911860230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc0009420 con 0x7f9bc40c8100 2023-10-25T00:06:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bb67fc640 1 -- 172.21.15.107:0/3911860230 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc013470 con 0x7f9bc40c8100 2023-10-25T00:06:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3911860230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc40c3da0 con 0x7f9bc40c8100 2023-10-25T00:06:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3911860230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc40746e0 con 0x7f9bc40c8100 2023-10-25T00:06:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bb67fc640 1 -- 172.21.15.107:0/3911860230 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9bbc0135d0 con 0x7f9bc40c8100 2023-10-25T00:06:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bb67fc640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ba8065e60 0x7f9ba8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bb67fc640 1 -- 172.21.15.107:0/3911860230 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7f9bbc0849b0 con 0x7f9bc40c8100 2023-10-25T00:06:19.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bc9905640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ba8065e60 0x7f9ba8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:19.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.896+0000 7f9bc9905640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ba8065e60 0x7f9ba8068320 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8005c10 tx=0x7f9bb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:19.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:19.900+0000 7f9bb67fc640 1 -- 172.21.15.107:0/3911860230 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9bbc051550 con 0x7f9bc40c8100 2023-10-25T00:06:20.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.092+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3911860230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bc4150430 con 0x7f9bc40c8100 2023-10-25T00:06:20.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.092+0000 7f9bb67fc640 1 -- 172.21.15.107:0/3911860230 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f9bbc017090 con 0x7f9bc40c8100 2023-10-25T00:06:20.094 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:20.094 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:56.785175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:20.094 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 120 2023-10-25T00:06:20.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3911860230 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ba8065e60 msgr2=0x7f9ba8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:20.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ba8065e60 0x7f9ba8068320 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8005c10 tx=0x7f9bb8009040 comp rx=0 tx=0).stop 2023-10-25T00:06:20.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3911860230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc40c8100 msgr2=0x7f9bc40aea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:20.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc40c8100 0x7f9bc40aea90 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00ebb0 tx=0x7f9bbc00c520 comp rx=0 tx=0).stop 2023-10-25T00:06:20.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3911860230 shutdown_connections 2023-10-25T00:06:20.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9bc4152040 0x7f9bc41473e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:20.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bc40c8100 0x7f9bc40aea90 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:20.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ba8065e60 0x7f9ba8068320 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:20.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 --2- 172.21.15.107:0/3911860230 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9bc406a4b0 0x7f9bc4151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:20.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3911860230 >> 172.21.15.107:0/3911860230 conn(0x7f9bc40ca200 msgr2=0x7f9bc414a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:20.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3911860230 shutdown_connections 2023-10-25T00:06:20.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:20.096+0000 7f9bcbb69640 1 -- 172.21.15.107:0/3911860230 wait complete. 2023-10-25T00:06:20.112 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:06:22.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:22.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e354b640 1 Processor -- start 2023-10-25T00:06:22.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e354b640 1 -- start start 2023-10-25T00:06:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e354b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc14fd40 0x7f62dc0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e354b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f62dc06ee40 0x7f62dc0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e354b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62dc06f730 con 0x7f62dc0a9820 2023-10-25T00:06:22.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e354b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62dc0aa750 con 0x7f62dc06ee40 2023-10-25T00:06:22.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e354b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc156a40 con 0x7f62dc14fd40 2023-10-25T00:06:22.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e1ae8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f62dc06ee40 0x7f62dc0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:22.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e1ae8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f62dc06ee40 0x7f62dc0aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53870/0 (socket says 172.21.15.107:53870) 2023-10-25T00:06:22.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e1ae8640 1 -- 172.21.15.107:0/3464467178 learned_addr learned my addr 172.21.15.107:0/3464467178 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:22.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e12e7640 1 --2- 172.21.15.107:0/3464467178 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc14fd40 0x7f62dc0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:22.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e1ae8640 1 -- 172.21.15.107:0/3464467178 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc14fd40 msgr2=0x7f62dc0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:22.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e1ae8640 1 --2- 172.21.15.107:0/3464467178 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc14fd40 0x7f62dc0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:22.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e1ae8640 1 -- 172.21.15.107:0/3464467178 >> v1:172.21.15.107:6789/0 conn(0x7f62dc0a9820 legacy=0x7f62dc0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:22.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62d3fff640 1 -- 172.21.15.107:0/3464467178 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180906507 0 0) 0x7f62dc06f730 con 0x7f62dc0a9820 2023-10-25T00:06:22.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.244+0000 7f62e1ae8640 1 -- 172.21.15.107:0/3464467178 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc156710 con 0x7f62dc06ee40 2023-10-25T00:06:22.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e1ae8640 1 --2- 172.21.15.107:0/3464467178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc06ee40 0x7f62dc0aa210 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009e50 tx=0x7f62d401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34cfbf5b12f99ac6 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:22.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62d3fff640 1 -- 172.21.15.107:0/3464467178 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d40a5720 con 0x7f62dc06ee40 2023-10-25T00:06:22.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62d3fff640 1 -- 172.21.15.107:0/3464467178 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d40a5880 con 0x7f62dc06ee40 2023-10-25T00:06:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62d3fff640 1 -- 172.21.15.107:0/3464467178 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d40a5c10 con 0x7f62dc06ee40 2023-10-25T00:06:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 -- 172.21.15.107:0/3464467178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc06ee40 msgr2=0x7f62dc0aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 --2- 172.21.15.107:0/3464467178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc06ee40 0x7f62dc0aa210 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009e50 tx=0x7f62d401c8c0 comp rx=0 tx=0).stop 2023-10-25T00:06:22.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 -- 172.21.15.107:0/3464467178 shutdown_connections 2023-10-25T00:06:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 --2- 172.21.15.107:0/3464467178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc06ee40 0x7f62dc0aa210 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 --2- 172.21.15.107:0/3464467178 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc14fd40 0x7f62dc0ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 -- 172.21.15.107:0/3464467178 >> 172.21.15.107:0/3464467178 conn(0x7f62dc0c2950 msgr2=0x7f62dc0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 -- 172.21.15.107:0/3464467178 shutdown_connections 2023-10-25T00:06:22.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 -- 172.21.15.107:0/3464467178 wait complete. 2023-10-25T00:06:22.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 Processor -- start 2023-10-25T00:06:22.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 -- start start 2023-10-25T00:06:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62dc06ee40 0x7f62dc0c4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc14fd40 0x7f62dc156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:22.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc157470 0x7f62dc153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc1ed760 con 0x7f62dc157470 2023-10-25T00:06:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc1ece30 con 0x7f62dc06ee40 2023-10-25T00:06:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc1ed110 con 0x7f62dc14fd40 2023-10-25T00:06:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e1ae8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc157470 0x7f62dc153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:22.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e1ae8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc157470 0x7f62dc153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53876/0 (socket says 172.21.15.107:53876) 2023-10-25T00:06:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e1ae8640 1 -- 172.21.15.107:0/3625667565 learned_addr learned my addr 172.21.15.107:0/3625667565 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e0ae6640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc14fd40 0x7f62dc156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e1ae8640 1 -- 172.21.15.107:0/3625667565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc14fd40 msgr2=0x7f62dc156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e1ae8640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc14fd40 0x7f62dc156f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e1ae8640 1 -- 172.21.15.107:0/3625667565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62dc06ee40 msgr2=0x7f62dc0c4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:22.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e1ae8640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62dc06ee40 0x7f62dc0c4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:22.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e1ae8640 1 -- 172.21.15.107:0/3625667565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d4009b30 con 0x7f62dc157470 2023-10-25T00:06:22.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e1ae8640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc157470 0x7f62dc153910 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f62d40af040 tx=0x7f62d4019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:22.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62d1ffb640 1 -- 172.21.15.107:0/3625667565 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d40b8470 con 0x7f62dc157470 2023-10-25T00:06:22.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62d1ffb640 1 -- 172.21.15.107:0/3625667565 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d40b85d0 con 0x7f62dc157470 2023-10-25T00:06:22.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 -- 172.21.15.107:0/3625667565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62dc1ed3f0 con 0x7f62dc157470 2023-10-25T00:06:22.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62d1ffb640 1 -- 172.21.15.107:0/3625667565 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d40b8940 con 0x7f62dc157470 2023-10-25T00:06:22.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.248+0000 7f62e354b640 1 -- 172.21.15.107:0/3625667565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62dc1ecae0 con 0x7f62dc157470 2023-10-25T00:06:22.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.252+0000 7f62e354b640 1 -- 172.21.15.107:0/3625667565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62dc0c24a0 con 0x7f62dc157470 2023-10-25T00:06:22.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.252+0000 7f62d1ffb640 1 -- 172.21.15.107:0/3625667565 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f62d40b8aa0 con 0x7f62dc157470 2023-10-25T00:06:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.252+0000 7f62d1ffb640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62c0065e60 0x7f62c0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.252+0000 7f62d1ffb640 1 -- 172.21.15.107:0/3625667565 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 8853+0+0 (secure 0 0 0) 0x7f62d411fed0 con 0x7f62dc157470 2023-10-25T00:06:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.252+0000 7f62e12e7640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62c0065e60 0x7f62c0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:22.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.252+0000 7f62e12e7640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62c0065e60 0x7f62c0068320 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f62d80098f0 tx=0x7f62d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:22.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.256+0000 7f62d1ffb640 1 -- 172.21.15.107:0/3625667565 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f62d40ecaf0 con 0x7f62dc157470 2023-10-25T00:06:22.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.440+0000 7f62e354b640 1 -- 172.21.15.107:0/3625667565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62dc070840 con 0x7f62dc157470 2023-10-25T00:06:22.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62d1ffb640 1 -- 172.21.15.107:0/3625667565 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f62d40f0110 con 0x7f62dc157470 2023-10-25T00:06:22.442 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:22.442 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7107,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":114}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:05:56.785175+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7122},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7122":{"gid":7122,"name":"a","rank":0,"incarnation":116,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6837/780830443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":780830443},{"type":"v1","addr":"172.21.15.107:6837","nonce":780830443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:22.443 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 120 2023-10-25T00:06:22.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 -- 172.21.15.107:0/3625667565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62c0065e60 msgr2=0x7f62c0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:22.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62c0065e60 0x7f62c0068320 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f62d80098f0 tx=0x7f62d8008040 comp rx=0 tx=0).stop 2023-10-25T00:06:22.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 -- 172.21.15.107:0/3625667565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc157470 msgr2=0x7f62dc153910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:22.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc157470 0x7f62dc153910 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f62d40af040 tx=0x7f62d4019660 comp rx=0 tx=0).stop 2023-10-25T00:06:22.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 -- 172.21.15.107:0/3625667565 shutdown_connections 2023-10-25T00:06:22.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc157470 0x7f62dc153910 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:22.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62c0065e60 0x7f62c0068320 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:22.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc14fd40 0x7f62dc156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:22.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 --2- 172.21.15.107:0/3625667565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62dc06ee40 0x7f62dc0c4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:22.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 -- 172.21.15.107:0/3625667565 >> 172.21.15.107:0/3625667565 conn(0x7f62dc0c2950 msgr2=0x7f62dc15c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:22.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 -- 172.21.15.107:0/3625667565 shutdown_connections 2023-10-25T00:06:22.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:22.444+0000 7f62e354b640 1 -- 172.21.15.107:0/3625667565 wait complete. 2023-10-25T00:06:22.459 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:06:24.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:24.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2e6f3640 1 Processor -- start 2023-10-25T00:06:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2e6f3640 1 -- start start 2023-10-25T00:06:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2e6f3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c28152040 0x7f6c280c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2e6f3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6c2806a4b0 0x7f6c280ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2e6f3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c28155680 con 0x7f6c280c8100 2023-10-25T00:06:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2e6f3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c28154a70 con 0x7f6c2806a4b0 2023-10-25T00:06:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2e6f3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c28154d50 con 0x7f6c28152040 2023-10-25T00:06:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2cc90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6c2806a4b0 0x7f6c280ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2cc90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6c2806a4b0 0x7f6c280ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59078/0 (socket says 172.21.15.107:59078) 2023-10-25T00:06:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2cc90640 1 -- 172.21.15.107:0/1194506634 learned_addr learned my addr 172.21.15.107:0/1194506634 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c27fff640 1 --2- 172.21.15.107:0/1194506634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c28152040 0x7f6c280c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2cc90640 1 -- 172.21.15.107:0/1194506634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c28152040 msgr2=0x7f6c280c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2cc90640 1 --2- 172.21.15.107:0/1194506634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c28152040 0x7f6c280c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2cc90640 1 -- 172.21.15.107:0/1194506634 >> v1:172.21.15.110:6789/0 conn(0x7f6c280c8100 legacy=0x7f6c280abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:24.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.592+0000 7f6c2cc90640 1 -- 172.21.15.107:0/1194506634 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c28155030 con 0x7f6c2806a4b0 2023-10-25T00:06:24.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2cc90640 1 --2- 172.21.15.107:0/1194506634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c2806a4b0 0x7f6c280ac410 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009e50 tx=0x7f6c2001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25dea76262043278 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:24.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c26ffd640 1 -- 172.21.15.107:0/1194506634 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c200a5750 con 0x7f6c2806a4b0 2023-10-25T00:06:24.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c26ffd640 1 -- 172.21.15.107:0/1194506634 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c200a58b0 con 0x7f6c2806a4b0 2023-10-25T00:06:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c26ffd640 1 -- 172.21.15.107:0/1194506634 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c200a5c40 con 0x7f6c2806a4b0 2023-10-25T00:06:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1194506634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c2806a4b0 msgr2=0x7f6c280ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 --2- 172.21.15.107:0/1194506634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c2806a4b0 0x7f6c280ac410 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009e50 tx=0x7f6c2001c980 comp rx=0 tx=0).stop 2023-10-25T00:06:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1194506634 shutdown_connections 2023-10-25T00:06:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 --2- 172.21.15.107:0/1194506634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c2806a4b0 0x7f6c280ac410 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 --2- 172.21.15.107:0/1194506634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c28152040 0x7f6c280c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1194506634 >> 172.21.15.107:0/1194506634 conn(0x7f6c280ca200 msgr2=0x7f6c280ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1194506634 shutdown_connections 2023-10-25T00:06:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1194506634 wait complete. 2023-10-25T00:06:24.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 Processor -- start 2023-10-25T00:06:24.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 -- start start 2023-10-25T00:06:24.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c2806a4b0 0x7f6c2814f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c280c8100 0x7f6c281f01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c28152040 0x7f6c28146990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c281ed090 con 0x7f6c28152040 2023-10-25T00:06:24.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c28151660 con 0x7f6c2806a4b0 2023-10-25T00:06:24.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2e6f3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c281f0a00 con 0x7f6c280c8100 2023-10-25T00:06:24.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2cc90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c28152040 0x7f6c28146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2cc90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c28152040 0x7f6c28146990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59090/0 (socket says 172.21.15.107:59090) 2023-10-25T00:06:24.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c2cc90640 1 -- 172.21.15.107:0/1055748945 learned_addr learned my addr 172.21.15.107:0/1055748945 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:24.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c27fff640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c2806a4b0 0x7f6c2814f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.596+0000 7f6c277fe640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c280c8100 0x7f6c281f01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c2cc90640 1 -- 172.21.15.107:0/1055748945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c280c8100 msgr2=0x7f6c281f01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:24.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c2cc90640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c280c8100 0x7f6c281f01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c2cc90640 1 -- 172.21.15.107:0/1055748945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c2806a4b0 msgr2=0x7f6c2814f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:24.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c2cc90640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c2806a4b0 0x7f6c2814f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c2cc90640 1 -- 172.21.15.107:0/1055748945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c20009b30 con 0x7f6c28152040 2023-10-25T00:06:24.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c2cc90640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c28152040 0x7f6c28146990 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f6c2001a5b0 tx=0x7f6c20019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:24.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c24ff9640 1 -- 172.21.15.107:0/1055748945 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c200b8670 con 0x7f6c28152040 2023-10-25T00:06:24.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c24ff9640 1 -- 172.21.15.107:0/1055748945 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c200b87d0 con 0x7f6c28152040 2023-10-25T00:06:24.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1055748945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c281f0ce0 con 0x7f6c28152040 2023-10-25T00:06:24.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c24ff9640 1 -- 172.21.15.107:0/1055748945 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c200b8b60 con 0x7f6c28152040 2023-10-25T00:06:24.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1055748945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c281f0710 con 0x7f6c28152040 2023-10-25T00:06:24.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1055748945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c281ed1d0 con 0x7f6c28152040 2023-10-25T00:06:24.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c24ff9640 1 -- 172.21.15.107:0/1055748945 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6c200b8cc0 con 0x7f6c28152040 2023-10-25T00:06:24.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c24ff9640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c14065e60 0x7f6c14068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c24ff9640 1 -- 172.21.15.107:0/1055748945 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7f6c2011f890 con 0x7f6c28152040 2023-10-25T00:06:24.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c27fff640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c14065e60 0x7f6c14068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.600+0000 7f6c27fff640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c14065e60 0x7f6c14068320 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6c18007a60 tx=0x7f6c18008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:24.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.604+0000 7f6c24ff9640 1 -- 172.21.15.107:0/1055748945 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6c281ed1d0 con 0x7f6c28152040 2023-10-25T00:06:24.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.796+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1055748945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c281ed1d0 con 0x7f6c28152040 2023-10-25T00:06:24.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.796+0000 7f6c24ff9640 1 -- 172.21.15.107:0/1055748945 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5020 (secure 0 0 0) 0x7f6c281ed1d0 con 0x7f6c28152040 2023-10-25T00:06:24.797 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:24.797 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:24.733781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:24.798 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 122 2023-10-25T00:06:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1055748945 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c14065e60 msgr2=0x7f6c14068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:24.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c14065e60 0x7f6c14068320 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6c18007a60 tx=0x7f6c18008040 comp rx=0 tx=0).stop 2023-10-25T00:06:24.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1055748945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c28152040 msgr2=0x7f6c28146990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:24.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c28152040 0x7f6c28146990 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f6c2001a5b0 tx=0x7f6c20019720 comp rx=0 tx=0).stop 2023-10-25T00:06:24.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1055748945 shutdown_connections 2023-10-25T00:06:24.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c28152040 0x7f6c28146990 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c14065e60 0x7f6c14068320 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c280c8100 0x7f6c281f01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 --2- 172.21.15.107:0/1055748945 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c2806a4b0 0x7f6c2814f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1055748945 >> 172.21.15.107:0/1055748945 conn(0x7f6c280ca200 msgr2=0x7f6c280c9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:24.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1055748945 shutdown_connections 2023-10-25T00:06:24.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.800+0000 7f6c2e6f3640 1 -- 172.21.15.107:0/1055748945 wait complete. 2023-10-25T00:06:24.816 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2023-10-25T00:06:24.816 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:06:24.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:24.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.924+0000 7f8147057640 1 Processor -- start 2023-10-25T00:06:24.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.924+0000 7f8147057640 1 -- start start 2023-10-25T00:06:24.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f8147057640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 0x7f81400abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f8147057640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f814006a4b0 0x7f81400ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f8147057640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8140155680 con 0x7f8140152040 2023-10-25T00:06:24.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f8147057640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8140154a70 con 0x7f814006a4b0 2023-10-25T00:06:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f8147057640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8140154d50 con 0x7f81400c8100 2023-10-25T00:06:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f813ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 0x7f81400abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f813ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 0x7f81400abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:45670/0 (socket says 172.21.15.107:45670) 2023-10-25T00:06:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f813ffff640 1 -- 172.21.15.107:0/171810762 learned_addr learned my addr 172.21.15.107:0/171810762 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f81455f4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f814006a4b0 0x7f81400ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f81455f4640 1 -- 172.21.15.107:0/171810762 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 msgr2=0x7f81400abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f81455f4640 1 --2- 172.21.15.107:0/171810762 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 0x7f81400abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f81455f4640 1 -- 172.21.15.107:0/171810762 >> v1:172.21.15.110:6789/0 conn(0x7f8140152040 legacy=0x7f81400c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f81455f4640 1 -- 172.21.15.107:0/171810762 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8140155030 con 0x7f814006a4b0 2023-10-25T00:06:24.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f813ffff640 1 --2- 172.21.15.107:0/171810762 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 0x7f81400abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:06:24.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f81455f4640 1 --2- 172.21.15.107:0/171810762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f814006a4b0 0x7f81400ac240 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8138009d90 tx=0x7f813801c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7230cc70fabaa90a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:24.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.928+0000 7f813f7fe640 1 -- 172.21.15.107:0/171810762 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81380a57a0 con 0x7f814006a4b0 2023-10-25T00:06:24.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f813f7fe640 1 -- 172.21.15.107:0/171810762 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81380a5900 con 0x7f814006a4b0 2023-10-25T00:06:24.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f813f7fe640 1 -- 172.21.15.107:0/171810762 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81380a5c90 con 0x7f814006a4b0 2023-10-25T00:06:24.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 -- 172.21.15.107:0/171810762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f814006a4b0 msgr2=0x7f81400ac240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:24.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 --2- 172.21.15.107:0/171810762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f814006a4b0 0x7f81400ac240 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8138009d90 tx=0x7f813801c8c0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 -- 172.21.15.107:0/171810762 shutdown_connections 2023-10-25T00:06:24.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 --2- 172.21.15.107:0/171810762 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f814006a4b0 0x7f81400ac240 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 --2- 172.21.15.107:0/171810762 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 0x7f81400abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 -- 172.21.15.107:0/171810762 >> 172.21.15.107:0/171810762 conn(0x7f81400ca200 msgr2=0x7f81400ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:24.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 -- 172.21.15.107:0/171810762 shutdown_connections 2023-10-25T00:06:24.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 -- 172.21.15.107:0/171810762 wait complete. 2023-10-25T00:06:24.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 Processor -- start 2023-10-25T00:06:24.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 -- start start 2023-10-25T00:06:24.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f814006a4b0 0x7f8140150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 0x7f81400c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140152040 0x7f81401474e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81400c4560 con 0x7f8140152040 2023-10-25T00:06:24.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8140150d50 con 0x7f814006a4b0 2023-10-25T00:06:24.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8147057640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814014f580 con 0x7f81400c8100 2023-10-25T00:06:24.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f81455f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140152040 0x7f81401474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f81455f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140152040 0x7f81401474e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59092/0 (socket says 172.21.15.107:59092) 2023-10-25T00:06:24.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f81455f4640 1 -- 172.21.15.107:0/1334727565 learned_addr learned my addr 172.21.15.107:0/1334727565 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:24.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f8144df3640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f814006a4b0 0x7f8140150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f813ffff640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 0x7f81400c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f81455f4640 1 -- 172.21.15.107:0/1334727565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 msgr2=0x7f81400c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:24.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f81455f4640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 0x7f81400c8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f81455f4640 1 -- 172.21.15.107:0/1334727565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f814006a4b0 msgr2=0x7f8140150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f81455f4640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f814006a4b0 0x7f8140150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f81455f4640 1 -- 172.21.15.107:0/1334727565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81340094d0 con 0x7f8140152040 2023-10-25T00:06:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f81455f4640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140152040 0x7f81401474e0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f81380b0040 tx=0x7f8138019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.932+0000 7f813d7fa640 1 -- 172.21.15.107:0/1334727565 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81380b9650 con 0x7f8140152040 2023-10-25T00:06:24.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.936+0000 7f813d7fa640 1 -- 172.21.15.107:0/1334727565 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81380b97b0 con 0x7f8140152040 2023-10-25T00:06:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.936+0000 7f8147057640 1 -- 172.21.15.107:0/1334727565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8138009a70 con 0x7f8140152040 2023-10-25T00:06:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.936+0000 7f813d7fa640 1 -- 172.21.15.107:0/1334727565 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81380b9b20 con 0x7f8140152040 2023-10-25T00:06:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.936+0000 7f8147057640 1 -- 172.21.15.107:0/1334727565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81400c4280 con 0x7f8140152040 2023-10-25T00:06:24.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.936+0000 7f8147057640 1 -- 172.21.15.107:0/1334727565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81400746e0 con 0x7f8140152040 2023-10-25T00:06:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.936+0000 7f813d7fa640 1 -- 172.21.15.107:0/1334727565 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f81380b9c80 con 0x7f8140152040 2023-10-25T00:06:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.936+0000 7f813d7fa640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f812c065e60 0x7f812c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:24.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.936+0000 7f813d7fa640 1 -- 172.21.15.107:0/1334727565 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7f813811f4b0 con 0x7f8140152040 2023-10-25T00:06:24.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.936+0000 7f8144df3640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f812c065e60 0x7f812c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:24.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.936+0000 7f8144df3640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f812c065e60 0x7f812c068320 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8130007fd0 tx=0x7f8130009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:24.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:24.940+0000 7f813d7fa640 1 -- 172.21.15.107:0/1334727565 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f81380ec000 con 0x7f8140152040 2023-10-25T00:06:25.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f8147057640 1 -- 172.21.15.107:0/1334727565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f814014f6c0 con 0x7f8140152040 2023-10-25T00:06:25.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f813d7fa640 1 -- 172.21.15.107:0/1334727565 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5020 (secure 0 0 0) 0x7f81380ef620 con 0x7f8140152040 2023-10-25T00:06:25.116 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:25.116 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:24.733781+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:25.117 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 122 2023-10-25T00:06:25.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f8147057640 1 -- 172.21.15.107:0/1334727565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f812c065e60 msgr2=0x7f812c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:25.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f8147057640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f812c065e60 0x7f812c068320 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8130007fd0 tx=0x7f8130009040 comp rx=0 tx=0).stop 2023-10-25T00:06:25.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f8147057640 1 -- 172.21.15.107:0/1334727565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140152040 msgr2=0x7f81401474e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:25.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f8147057640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140152040 0x7f81401474e0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f81380b0040 tx=0x7f8138019660 comp rx=0 tx=0).stop 2023-10-25T00:06:25.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f8147057640 1 -- 172.21.15.107:0/1334727565 shutdown_connections 2023-10-25T00:06:25.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f8147057640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8140152040 0x7f81401474e0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:25.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f8147057640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f812c065e60 0x7f812c068320 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:25.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f8147057640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f81400c8100 0x7f81400c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:25.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f8147057640 1 --2- 172.21.15.107:0/1334727565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f814006a4b0 0x7f8140150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:25.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.116+0000 7f8147057640 1 -- 172.21.15.107:0/1334727565 >> 172.21.15.107:0/1334727565 conn(0x7f81400ca200 msgr2=0x7f814006a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:25.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.120+0000 7f8147057640 1 -- 172.21.15.107:0/1334727565 shutdown_connections 2023-10-25T00:06:25.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:25.120+0000 7f8147057640 1 -- 172.21.15.107:0/1334727565 wait complete. 2023-10-25T00:06:25.134 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 122, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 122, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7107": { "addr": "172.21.15.107:6835/913067382", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 913067382, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 913067382, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7107, "incarnation": 121, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:06:24.733781+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7107 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 113, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5410": { "addr": "172.21.15.110:6835/2150827048", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2150827048, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2150827048, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5410, "incarnation": 110, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 98 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:05:21.824320+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5410 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:06:27.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.276+0000 7fb8d52ce640 1 Processor -- start 2023-10-25T00:06:27.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.276+0000 7fb8d52ce640 1 -- start start 2023-10-25T00:06:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8d52ce640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb8d014c470 0x7fb8d014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8d52ce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d0152b30 0x7fb8d0152f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8d52ce640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d0155cd0 con 0x7fb8d006f490 2023-10-25T00:06:27.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8d52ce640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d01526e0 con 0x7fb8d014c470 2023-10-25T00:06:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8d52ce640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d00c4380 con 0x7fb8d0152b30 2023-10-25T00:06:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8ce7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb8d006f490 0x7fb8d0071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44340/0 (socket says 172.21.15.107:44340) 2023-10-25T00:06:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8ce7fc640 1 -- 172.21.15.107:0/1160849501 learned_addr learned my addr 172.21.15.107:0/1160849501 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:06:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cf7fe640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d0152b30 0x7fb8d0152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8ceffd640 1 --2- 172.21.15.107:0/1160849501 >> v2:172.21.15.110:3300/0 conn(0x7fb8d014c470 0x7fb8d014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:27.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281419813 0 0) 0x7fb8d0155cd0 con 0x7fb8d006f490 2023-10-25T00:06:27.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8bc003530 con 0x7fb8d006f490 2023-10-25T00:06:27.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415297547 0 0) 0x7fb8bc003530 con 0x7fb8d006f490 2023-10-25T00:06:27.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d0152b30 msgr2=0x7fb8d0152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:27.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d0152b30 0x7fb8d0152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:27.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 >> v2:172.21.15.110:3300/0 conn(0x7fb8d014c470 msgr2=0x7fb8d014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:27.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 --2- 172.21.15.107:0/1160849501 >> v2:172.21.15.110:3300/0 conn(0x7fb8d014c470 0x7fb8d014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:27.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d00c48b0 con 0x7fb8d006f490 2023-10-25T00:06:27.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb8b8001680 con 0x7fb8d006f490 2023-10-25T00:06:27.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 >> v1:172.21.15.107:6789/0 conn(0x7fb8d006f490 legacy=0x7fb8d0071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:27.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d0152b30 0x7fb8bc003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:27.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8bc003f00 0x7fb8bc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:27.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8bc004d10 0x7fb8bc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:27.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d01526e0 con 0x7fb8bc004d10 2023-10-25T00:06:27.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d00c4380 con 0x7fb8bc003f00 2023-10-25T00:06:27.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc00ba70 con 0x7fb8d0152b30 2023-10-25T00:06:27.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cf7fe640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8bc004d10 0x7fb8bc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:27.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8ceffd640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d0152b30 0x7fb8bc003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:27.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8ce7fc640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8bc003f00 0x7fb8bc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:27.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/1160849501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d0152b30 msgr2=0x7fb8bc003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:27.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cf7fe640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d0152b30 0x7fb8bc003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:27.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/1160849501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8bc003f00 msgr2=0x7fb8bc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:27.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cf7fe640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8bc003f00 0x7fb8bc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:27.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.280+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/1160849501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc003530 con 0x7fb8bc004d10 2023-10-25T00:06:27.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cf7fe640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8bc004d10 0x7fb8bc005190 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8009fc0 tx=0x7fb8c801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2124d67cee685ac8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c80a5b40 con 0x7fb8bc004d10 2023-10-25T00:06:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c800f040 con 0x7fb8bc004d10 2023-10-25T00:06:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cdffb640 1 -- 172.21.15.107:0/1160849501 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c80ad440 con 0x7fb8bc004d10 2023-10-25T00:06:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 -- 172.21.15.107:0/1160849501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8bc004d10 msgr2=0x7fb8bc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:27.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 --2- 172.21.15.107:0/1160849501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8bc004d10 0x7fb8bc005190 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8009fc0 tx=0x7fb8c801cc60 comp rx=0 tx=0).stop 2023-10-25T00:06:27.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/1160849501 reap_dead start 2023-10-25T00:06:27.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 -- 172.21.15.107:0/1160849501 shutdown_connections 2023-10-25T00:06:27.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 -- 172.21.15.107:0/1160849501 >> 172.21.15.107:0/1160849501 conn(0x7fb8d0070f60 msgr2=0x7fb8d0071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:27.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 -- 172.21.15.107:0/1160849501 shutdown_connections 2023-10-25T00:06:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 -- 172.21.15.107:0/1160849501 wait complete. 2023-10-25T00:06:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 Processor -- start 2023-10-25T00:06:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 -- start start 2023-10-25T00:06:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d014ca20 0x7fb8d01449b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8d014e0e0 0x7fb8d0143930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d014ce00 0x7fb8d0143e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d01ea020 con 0x7fb8d014ce00 2023-10-25T00:06:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d01447d0 con 0x7fb8d014e0e0 2023-10-25T00:06:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8d52ce640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d01eafd0 con 0x7fb8d014ca20 2023-10-25T00:06:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d014ce00 0x7fb8d0143e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cf7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d014ce00 0x7fb8d0143e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59112/0 (socket says 172.21.15.107:59112) 2023-10-25T00:06:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/3531695855 learned_addr learned my addr 172.21.15.107:0/3531695855 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8ceffd640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d014ca20 0x7fb8d01449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8ce7fc640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8d014e0e0 0x7fb8d0143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:27.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/3531695855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d014ca20 msgr2=0x7fb8d01449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cf7fe640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d014ca20 0x7fb8d01449b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/3531695855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8d014e0e0 msgr2=0x7fb8d0143930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cf7fe640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8d014e0e0 0x7fb8d0143930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.284+0000 7fb8cf7fe640 1 -- 172.21.15.107:0/3531695855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d01ea210 con 0x7fb8d014ce00 2023-10-25T00:06:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8cf7fe640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d014ce00 0x7fb8d0143e70 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8002240 tx=0x7fb8c8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:27.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8c77fe640 1 -- 172.21.15.107:0/3531695855 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c80039a0 con 0x7fb8d014ce00 2023-10-25T00:06:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8c77fe640 1 -- 172.21.15.107:0/3531695855 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c8003b00 con 0x7fb8d014ce00 2023-10-25T00:06:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8d52ce640 1 -- 172.21.15.107:0/3531695855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d01ea4f0 con 0x7fb8d014ce00 2023-10-25T00:06:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8c77fe640 1 -- 172.21.15.107:0/3531695855 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8003e70 con 0x7fb8d014ce00 2023-10-25T00:06:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8d52ce640 1 -- 172.21.15.107:0/3531695855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d01eaab0 con 0x7fb8d014ce00 2023-10-25T00:06:27.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8d52ce640 1 -- 172.21.15.107:0/3531695855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d00746e0 con 0x7fb8d014ce00 2023-10-25T00:06:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8c77fe640 1 -- 172.21.15.107:0/3531695855 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb8c80b4040 con 0x7fb8d014ce00 2023-10-25T00:06:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8c77fe640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8b0065e60 0x7fb8b0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8c77fe640 1 -- 172.21.15.107:0/3531695855 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7fb8c80a5e40 con 0x7fb8d014ce00 2023-10-25T00:06:27.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8ceffd640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8b0065e60 0x7fb8b0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:27.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.288+0000 7fb8ceffd640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8b0065e60 0x7fb8b0068320 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0072160 tx=0x7fb8c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:27.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.296+0000 7fb8c77fe640 1 -- 172.21.15.107:0/3531695855 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb8c80ea970 con 0x7fb8d014ce00 2023-10-25T00:06:27.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.500+0000 7fb8d52ce640 1 -- 172.21.15.107:0/3531695855 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8d0156530 con 0x7fb8d014ce00 2023-10-25T00:06:27.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.500+0000 7fb8c77fe640 1 -- 172.21.15.107:0/3531695855 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5017 (secure 0 0 0) 0x7fb8c80b2020 con 0x7fb8d014ce00 2023-10-25T00:06:27.500 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:27.500 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:27.501 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 124 2023-10-25T00:06:27.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 -- 172.21.15.107:0/3531695855 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8b0065e60 msgr2=0x7fb8b0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:27.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8b0065e60 0x7fb8b0068320 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0072160 tx=0x7fb8c0008040 comp rx=0 tx=0).stop 2023-10-25T00:06:27.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 -- 172.21.15.107:0/3531695855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d014ce00 msgr2=0x7fb8d0143e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:27.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d014ce00 0x7fb8d0143e70 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8002240 tx=0x7fb8c8009750 comp rx=0 tx=0).stop 2023-10-25T00:06:27.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 -- 172.21.15.107:0/3531695855 shutdown_connections 2023-10-25T00:06:27.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8d014ce00 0x7fb8d0143e70 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:27.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8d014e0e0 0x7fb8d0143930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:27.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb8b0065e60 0x7fb8b0068320 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:27.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 --2- 172.21.15.107:0/3531695855 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8d014ca20 0x7fb8d01449b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:27.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 -- 172.21.15.107:0/3531695855 >> 172.21.15.107:0/3531695855 conn(0x7fb8d0070f60 msgr2=0x7fb8d014ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:27.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 -- 172.21.15.107:0/3531695855 shutdown_connections 2023-10-25T00:06:27.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:27.504+0000 7fb8d52ce640 1 -- 172.21.15.107:0/3531695855 wait complete. 2023-10-25T00:06:27.519 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=7122 2023-10-25T00:06:27.520 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before reviving mds.a 2023-10-25T00:06:30.633 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:06:30.632+0000 7f8ec3dba640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:06:36.520 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2023-10-25T00:06:36.521 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:06:36.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:06:36.524 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:06:36.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:06:36.603 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:06:36.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.676+0000 7f726c258640 1 Processor -- start 2023-10-25T00:06:36.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.676+0000 7f726c258640 1 -- start start 2023-10-25T00:06:36.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.676+0000 7f726c258640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f726413bb00 0x7f72641531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:36.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.676+0000 7f726c258640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f72640a9450 0x7f7264071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:36.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.676+0000 7f726c258640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7264152d40 con 0x7f72640aa330 2023-10-25T00:06:36.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.676+0000 7f726c258640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7264156100 con 0x7f72640a9450 2023-10-25T00:06:36.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.676+0000 7f726c258640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72641527c0 con 0x7f726413bb00 2023-10-25T00:06:36.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.676+0000 7f7269ff4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f726413bb00 0x7f72641531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:36.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.676+0000 7f726a7f5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f72640a9450 0x7f7264071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:36.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.676+0000 7f7269ff4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f726413bb00 0x7f72641531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:59064/0 (socket says 172.21.15.107:59064) 2023-10-25T00:06:36.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.676+0000 7f7269ff4640 1 -- 172.21.15.107:0/2108423934 learned_addr learned my addr 172.21.15.107:0/2108423934 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:36.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f7269ff4640 1 -- 172.21.15.107:0/2108423934 >> v2:172.21.15.110:3300/0 conn(0x7f72640a9450 msgr2=0x7f7264071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:06:36.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f7269ff4640 1 --2- 172.21.15.107:0/2108423934 >> v2:172.21.15.110:3300/0 conn(0x7f72640a9450 0x7f7264071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:36.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f7269ff4640 1 -- 172.21.15.107:0/2108423934 >> v1:172.21.15.110:6789/0 conn(0x7f72640aa330 legacy=0x7f726406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:36.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f7269ff4640 1 -- 172.21.15.107:0/2108423934 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726406d660 con 0x7f726413bb00 2023-10-25T00:06:36.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f7269ff4640 1 --2- 172.21.15.107:0/2108423934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f726413bb00 0x7f72641531d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72600098a0 tx=0x7f726001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8815cf0c7f79f3c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:36.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f7268ff2640 1 -- 172.21.15.107:0/2108423934 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72600ab600 con 0x7f726413bb00 2023-10-25T00:06:36.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f7268ff2640 1 -- 172.21.15.107:0/2108423934 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72600a7050 con 0x7f726413bb00 2023-10-25T00:06:36.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f7268ff2640 1 -- 172.21.15.107:0/2108423934 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72600ab980 con 0x7f726413bb00 2023-10-25T00:06:36.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f726c258640 1 -- 172.21.15.107:0/2108423934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f726413bb00 msgr2=0x7f72641531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:36.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f726c258640 1 --2- 172.21.15.107:0/2108423934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f726413bb00 0x7f72641531d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72600098a0 tx=0x7f726001c400 comp rx=0 tx=0).stop 2023-10-25T00:06:36.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f726c258640 1 -- 172.21.15.107:0/2108423934 shutdown_connections 2023-10-25T00:06:36.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f726c258640 1 --2- 172.21.15.107:0/2108423934 >> v2:172.21.15.110:3300/0 conn(0x7f72640a9450 0x7f7264071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:36.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f726c258640 1 --2- 172.21.15.107:0/2108423934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f726413bb00 0x7f72641531d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:36.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f726c258640 1 -- 172.21.15.107:0/2108423934 >> 172.21.15.107:0/2108423934 conn(0x7f7264070f60 msgr2=0x7f7264071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:36.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f726c258640 1 -- 172.21.15.107:0/2108423934 shutdown_connections 2023-10-25T00:06:36.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.680+0000 7f726c258640 1 -- 172.21.15.107:0/2108423934 wait complete. 2023-10-25T00:06:36.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726c258640 1 Processor -- start 2023-10-25T00:06:36.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726c258640 1 -- start start 2023-10-25T00:06:36.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726c258640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72640a9450 0x7f72640c3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:36.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726c258640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72640aa330 0x7f726414fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:36.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726c258640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726413bb00 0x7f7264150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:36.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726c258640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72640aba00 con 0x7f726413bb00 2023-10-25T00:06:36.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726c258640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72640ab4a0 con 0x7f72640a9450 2023-10-25T00:06:36.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726c258640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72641508e0 con 0x7f72640aa330 2023-10-25T00:06:36.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726a7f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726413bb00 0x7f7264150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:36.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726a7f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726413bb00 0x7f7264150210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33446/0 (socket says 172.21.15.107:33446) 2023-10-25T00:06:36.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726a7f5640 1 -- 172.21.15.107:0/3665697695 learned_addr learned my addr 172.21.15.107:0/3665697695 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:06:36.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f72697f3640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72640aa330 0x7f726414fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:36.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f7269ff4640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72640a9450 0x7f72640c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:36.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726a7f5640 1 -- 172.21.15.107:0/3665697695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72640aa330 msgr2=0x7f726414fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:36.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726a7f5640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72640aa330 0x7f726414fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:36.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726a7f5640 1 -- 172.21.15.107:0/3665697695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72640a9450 msgr2=0x7f72640c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:36.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726a7f5640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72640a9450 0x7f72640c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:36.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726a7f5640 1 -- 172.21.15.107:0/3665697695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72600094d0 con 0x7f726413bb00 2023-10-25T00:06:36.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726a7f5640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726413bb00 0x7f7264150210 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f725c00efc0 tx=0x7f725c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:36.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f72567fc640 1 -- 172.21.15.107:0/3665697695 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c014480 con 0x7f726413bb00 2023-10-25T00:06:36.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f72567fc640 1 -- 172.21.15.107:0/3665697695 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725c00f040 con 0x7f726413bb00 2023-10-25T00:06:36.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726c258640 1 -- 172.21.15.107:0/3665697695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72640aacc0 con 0x7f726413bb00 2023-10-25T00:06:36.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f72567fc640 1 -- 172.21.15.107:0/3665697695 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c014820 con 0x7f726413bb00 2023-10-25T00:06:36.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.684+0000 7f726c258640 1 -- 172.21.15.107:0/3665697695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7264155a30 con 0x7f726413bb00 2023-10-25T00:06:36.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.688+0000 7f726c258640 1 -- 172.21.15.107:0/3665697695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72640746e0 con 0x7f726413bb00 2023-10-25T00:06:36.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.688+0000 7f72567fc640 1 -- 172.21.15.107:0/3665697695 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f725c014980 con 0x7f726413bb00 2023-10-25T00:06:36.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.688+0000 7f72567fc640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7248065e60 0x7f7248068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:06:36.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.688+0000 7f72567fc640 1 -- 172.21.15.107:0/3665697695 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7f725c084eb0 con 0x7f726413bb00 2023-10-25T00:06:36.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.688+0000 7f7269ff4640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7248065e60 0x7f7248068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:06:36.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.688+0000 7f7269ff4640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7248065e60 0x7f7248068320 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f72600ad000 tx=0x7f72600a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:06:36.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.692+0000 7f72567fc640 1 -- 172.21.15.107:0/3665697695 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f725c051a00 con 0x7f726413bb00 2023-10-25T00:06:36.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.928+0000 7f726c258640 1 -- 172.21.15.107:0/3665697695 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f726406d160 con 0x7f726413bb00 2023-10-25T00:06:36.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.928+0000 7f72567fc640 1 -- 172.21.15.107:0/3665697695 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f725c018070 con 0x7f726413bb00 2023-10-25T00:06:36.930 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:06:36.930 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:06:36.930 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 125 2023-10-25T00:06:36.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 -- 172.21.15.107:0/3665697695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7248065e60 msgr2=0x7f7248068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:36.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7248065e60 0x7f7248068320 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f72600ad000 tx=0x7f72600a6000 comp rx=0 tx=0).stop 2023-10-25T00:06:36.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 -- 172.21.15.107:0/3665697695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726413bb00 msgr2=0x7f7264150210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:06:36.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726413bb00 0x7f7264150210 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f725c00efc0 tx=0x7f725c00c490 comp rx=0 tx=0).stop 2023-10-25T00:06:36.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 -- 172.21.15.107:0/3665697695 shutdown_connections 2023-10-25T00:06:36.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f726413bb00 0x7f7264150210 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:36.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7248065e60 0x7f7248068320 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:36.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f72640aa330 0x7f726414fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 --2- 172.21.15.107:0/3665697695 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f72640a9450 0x7f72640c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:06:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 -- 172.21.15.107:0/3665697695 >> 172.21.15.107:0/3665697695 conn(0x7f7264070f60 msgr2=0x7f7264071340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:06:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 -- 172.21.15.107:0/3665697695 shutdown_connections 2023-10-25T00:06:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:06:36.932+0000 7f726c258640 1 -- 172.21.15.107:0/3665697695 wait complete. 2023-10-25T00:06:36.954 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2023-10-25T00:06:36.954 INFO:tasks.mds_thrash.fs.[a]:waiting for 73 secs before thrashing 2023-10-25T00:07:30.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:30.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.835+0000 7fac94296640 1 Processor -- start 2023-10-25T00:07:30.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.835+0000 7fac94296640 1 -- start start 2023-10-25T00:07:30.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.835+0000 7fac94296640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fac8c065bc0 0x7fac8c158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:30.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.835+0000 7fac94296640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c06a120 0x7fac8c0aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:30.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.835+0000 7fac94296640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8c06cea0 con 0x7fac8c069d40 2023-10-25T00:07:30.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.835+0000 7fac94296640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac8c0ac650 con 0x7fac8c065bc0 2023-10-25T00:07:30.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.835+0000 7fac94296640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c0a9e00 con 0x7fac8c06a120 2023-10-25T00:07:30.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.835+0000 7fac92032640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fac8c069d40 0x7fac8c0a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:55584/0 (socket says 172.21.15.107:55584) 2023-10-25T00:07:30.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.835+0000 7fac92032640 1 -- 172.21.15.107:0/2581673039 learned_addr learned my addr 172.21.15.107:0/2581673039 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:30.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91831640 1 --2- 172.21.15.107:0/2581673039 >> v2:172.21.15.110:3300/0 conn(0x7fac8c065bc0 0x7fac8c158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:30.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac92833640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c06a120 0x7fac8c0aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:30.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889077442 0 0) 0x7fac8c06cea0 con 0x7fac8c069d40 2023-10-25T00:07:30.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac78003530 con 0x7fac8c069d40 2023-10-25T00:07:30.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2643296640 0 0) 0x7fac78003530 con 0x7fac8c069d40 2023-10-25T00:07:30.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c06a120 msgr2=0x7fac8c0aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:30.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c06a120 0x7fac8c0aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:30.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 >> v2:172.21.15.110:3300/0 conn(0x7fac8c065bc0 msgr2=0x7fac8c158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:30.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 --2- 172.21.15.107:0/2581673039 >> v2:172.21.15.110:3300/0 conn(0x7fac8c065bc0 0x7fac8c158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:30.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8c1584e0 con 0x7fac8c069d40 2023-10-25T00:07:30.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fac88001680 con 0x7fac8c069d40 2023-10-25T00:07:30.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 >> v1:172.21.15.107:6789/0 conn(0x7fac8c069d40 legacy=0x7fac8c0a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:30.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac78003980 0x7fac78003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:30.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac780047d0 0x7fac78004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:30.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c06a120 0x7fac78005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:30.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7800b990 con 0x7fac78003980 2023-10-25T00:07:30.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7800bad0 con 0x7fac780047d0 2023-10-25T00:07:30.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7800bc40 con 0x7fac8c06a120 2023-10-25T00:07:30.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac92032640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac78003980 0x7fac78003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:30.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac92833640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c06a120 0x7fac78005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:30.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac92032640 1 -- 172.21.15.107:0/2581673039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c06a120 msgr2=0x7fac78005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:30.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac92032640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c06a120 0x7fac78005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:30.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac92032640 1 -- 172.21.15.107:0/2581673039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac780047d0 msgr2=0x7fac78004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:30.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91831640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac780047d0 0x7fac78004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:30.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac92032640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac780047d0 0x7fac78004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:30.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac92032640 1 -- 172.21.15.107:0/2581673039 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac78003530 con 0x7fac78003980 2023-10-25T00:07:30.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac92032640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac78003980 0x7fac78003da0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fac8800c1c0 tx=0x7fac8801cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48cd2cd0093eb566 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:30.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac880058e0 con 0x7fac78003980 2023-10-25T00:07:30.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac88005a60 con 0x7fac78003980 2023-10-25T00:07:30.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac91030640 1 -- 172.21.15.107:0/2581673039 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88005e10 con 0x7fac78003980 2023-10-25T00:07:30.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac94296640 1 -- 172.21.15.107:0/2581673039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac78003980 msgr2=0x7fac78003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:30.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.839+0000 7fac94296640 1 --2- 172.21.15.107:0/2581673039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac78003980 0x7fac78003da0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fac8800c1c0 tx=0x7fac8801cbe0 comp rx=0 tx=0).stop 2023-10-25T00:07:30.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac92833640 1 -- 172.21.15.107:0/2581673039 reap_dead start 2023-10-25T00:07:30.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 -- 172.21.15.107:0/2581673039 shutdown_connections 2023-10-25T00:07:30.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 -- 172.21.15.107:0/2581673039 >> 172.21.15.107:0/2581673039 conn(0x7fac8c0c65c0 msgr2=0x7fac8c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:30.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 -- 172.21.15.107:0/2581673039 shutdown_connections 2023-10-25T00:07:30.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 -- 172.21.15.107:0/2581673039 wait complete. 2023-10-25T00:07:30.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 Processor -- start 2023-10-25T00:07:30.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 -- start start 2023-10-25T00:07:30.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac78003980 0x7fac8c070300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:30.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8c0bdfc0 0x7fac8c14ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:30.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c1edf10 0x7fac8c14d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:30.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c1f1580 con 0x7fac8c0bdfc0 2023-10-25T00:07:30.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c06fd80 con 0x7fac78003980 2023-10-25T00:07:30.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c1eda80 con 0x7fac8c1edf10 2023-10-25T00:07:30.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac91831640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8c0bdfc0 0x7fac8c14ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:30.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac91831640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8c0bdfc0 0x7fac8c14ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33788/0 (socket says 172.21.15.107:33788) 2023-10-25T00:07:30.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac91831640 1 -- 172.21.15.107:0/362945387 learned_addr learned my addr 172.21.15.107:0/362945387 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:30.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac92032640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac78003980 0x7fac8c070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:30.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac92833640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c1edf10 0x7fac8c14d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:30.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac91831640 1 -- 172.21.15.107:0/362945387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c1edf10 msgr2=0x7fac8c14d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:30.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac91831640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c1edf10 0x7fac8c14d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:30.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac91831640 1 -- 172.21.15.107:0/362945387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac78003980 msgr2=0x7fac8c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:30.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac91831640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac78003980 0x7fac8c070300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:30.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac91831640 1 -- 172.21.15.107:0/362945387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8c070840 con 0x7fac8c0bdfc0 2023-10-25T00:07:30.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac91831640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8c0bdfc0 0x7fac8c14ed00 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fac7c010890 tx=0x7fac7c010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:30.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac82ffd640 1 -- 172.21.15.107:0/362945387 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c009630 con 0x7fac8c0bdfc0 2023-10-25T00:07:30.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac82ffd640 1 -- 172.21.15.107:0/362945387 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c00eae0 con 0x7fac8c0bdfc0 2023-10-25T00:07:30.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 -- 172.21.15.107:0/362945387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac8c1f1a40 con 0x7fac8c0bdfc0 2023-10-25T00:07:30.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac82ffd640 1 -- 172.21.15.107:0/362945387 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c00ee90 con 0x7fac8c0bdfc0 2023-10-25T00:07:30.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.843+0000 7fac94296640 1 -- 172.21.15.107:0/362945387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8c1f1760 con 0x7fac8c0bdfc0 2023-10-25T00:07:30.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.847+0000 7fac94296640 1 -- 172.21.15.107:0/362945387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac8c1edc10 con 0x7fac8c0bdfc0 2023-10-25T00:07:30.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.847+0000 7fac82ffd640 1 -- 172.21.15.107:0/362945387 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fac7c01c400 con 0x7fac8c0bdfc0 2023-10-25T00:07:30.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.847+0000 7fac82ffd640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac70065e60 0x7fac70068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:30.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.847+0000 7fac82ffd640 1 -- 172.21.15.107:0/362945387 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7fac7c0847a0 con 0x7fac8c0bdfc0 2023-10-25T00:07:30.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.847+0000 7fac92032640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac70065e60 0x7fac70068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:30.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.847+0000 7fac92032640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac70065e60 0x7fac70068320 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fac880aa000 tx=0x7fac8c06e350 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:30.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:30.851+0000 7fac82ffd640 1 -- 172.21.15.107:0/362945387 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fac7c0512f0 con 0x7fac8c0bdfc0 2023-10-25T00:07:31.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 -- 172.21.15.107:0/362945387 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac8c0746e0 con 0x7fac8c0bdfc0 2023-10-25T00:07:31.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac82ffd640 1 -- 172.21.15.107:0/362945387 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5778 (secure 0 0 0) 0x7fac7c054910 con 0x7fac8c0bdfc0 2023-10-25T00:07:31.075 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:31.075 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:31.075 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 125 2023-10-25T00:07:31.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 -- 172.21.15.107:0/362945387 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac70065e60 msgr2=0x7fac70068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:31.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac70065e60 0x7fac70068320 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fac880aa000 tx=0x7fac8c06e350 comp rx=0 tx=0).stop 2023-10-25T00:07:31.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 -- 172.21.15.107:0/362945387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8c0bdfc0 msgr2=0x7fac8c14ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:31.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8c0bdfc0 0x7fac8c14ed00 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fac7c010890 tx=0x7fac7c010d50 comp rx=0 tx=0).stop 2023-10-25T00:07:31.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 -- 172.21.15.107:0/362945387 shutdown_connections 2023-10-25T00:07:31.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac70065e60 0x7fac70068320 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8c1edf10 0x7fac8c14d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8c0bdfc0 0x7fac8c14ed00 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 --2- 172.21.15.107:0/362945387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac78003980 0x7fac8c070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 -- 172.21.15.107:0/362945387 >> 172.21.15.107:0/362945387 conn(0x7fac8c0c65c0 msgr2=0x7fac8c06c490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:31.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 -- 172.21.15.107:0/362945387 shutdown_connections 2023-10-25T00:07:31.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.075+0000 7fac94296640 1 -- 172.21.15.107:0/362945387 wait complete. 2023-10-25T00:07:31.096 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2023-10-25T00:07:31.096 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-25T00:07:31.096 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:07:31.113 INFO:tasks.ceph.mds.d:Stopped 2023-10-25T00:07:31.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:31.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.243+0000 7f74b7720640 1 Processor -- start 2023-10-25T00:07:31.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.243+0000 7f74b7720640 1 -- start start 2023-10-25T00:07:31.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b7720640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f74b0065bc0 0x7f74b0158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:31.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b7720640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74b006a120 0x7f74b00aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:31.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b7720640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b006cea0 con 0x7f74b0069d40 2023-10-25T00:07:31.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b7720640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b00ac650 con 0x7f74b006a120 2023-10-25T00:07:31.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b7720640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b00a9e00 con 0x7f74b0065bc0 2023-10-25T00:07:31.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b5cbd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74b006a120 0x7f74b00aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:31.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b5cbd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f74b006a120 0x7f74b00aaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33796/0 (socket says 172.21.15.107:33796) 2023-10-25T00:07:31.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b5cbd640 1 -- 172.21.15.107:0/293022858 learned_addr learned my addr 172.21.15.107:0/293022858 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:31.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b4cbb640 1 --2- 172.21.15.107:0/293022858 >> v2:172.21.15.110:3300/0 conn(0x7f74b0065bc0 0x7f74b0158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:31.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b5cbd640 1 -- 172.21.15.107:0/293022858 >> v2:172.21.15.110:3300/0 conn(0x7f74b0065bc0 msgr2=0x7f74b0158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:31.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b5cbd640 1 --2- 172.21.15.107:0/293022858 >> v2:172.21.15.110:3300/0 conn(0x7f74b0065bc0 0x7f74b0158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b5cbd640 1 -- 172.21.15.107:0/293022858 >> v1:172.21.15.110:6789/0 conn(0x7f74b0069d40 legacy=0x7f74b00a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:31.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.247+0000 7f74b5cbd640 1 -- 172.21.15.107:0/293022858 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b01584e0 con 0x7f74b006a120 2023-10-25T00:07:31.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74b5cbd640 1 --2- 172.21.15.107:0/293022858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74b006a120 0x7f74b00aaef0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f74b0066170 tx=0x7f74a801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da23472a9ba103ad server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:31.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74a7fff640 1 -- 172.21.15.107:0/293022858 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a80a56b0 con 0x7f74b006a120 2023-10-25T00:07:31.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74a7fff640 1 -- 172.21.15.107:0/293022858 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a80a5810 con 0x7f74b006a120 2023-10-25T00:07:31.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74a7fff640 1 -- 172.21.15.107:0/293022858 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a80a5ba0 con 0x7f74b006a120 2023-10-25T00:07:31.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74b7720640 1 -- 172.21.15.107:0/293022858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74b006a120 msgr2=0x7f74b00aaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:31.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74b7720640 1 --2- 172.21.15.107:0/293022858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74b006a120 0x7f74b00aaef0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f74b0066170 tx=0x7f74a801c900 comp rx=0 tx=0).stop 2023-10-25T00:07:31.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74b7720640 1 -- 172.21.15.107:0/293022858 shutdown_connections 2023-10-25T00:07:31.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74b7720640 1 --2- 172.21.15.107:0/293022858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74b006a120 0x7f74b00aaef0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74b7720640 1 --2- 172.21.15.107:0/293022858 >> v2:172.21.15.110:3300/0 conn(0x7f74b0065bc0 0x7f74b0158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74b7720640 1 -- 172.21.15.107:0/293022858 >> 172.21.15.107:0/293022858 conn(0x7f74b00c65c0 msgr2=0x7f74b00c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:31.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74b7720640 1 -- 172.21.15.107:0/293022858 shutdown_connections 2023-10-25T00:07:31.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74b7720640 1 -- 172.21.15.107:0/293022858 wait complete. 2023-10-25T00:07:31.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.251+0000 7f74b7720640 1 Processor -- start 2023-10-25T00:07:31.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b7720640 1 -- start start 2023-10-25T00:07:31.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b7720640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74b0065bc0 0x7f74b0153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:31.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b7720640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74b0069d40 0x7f74b01f0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:31.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b7720640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74b006a120 0x7f74b00a9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:31.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b7720640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b0154470 con 0x7f74b006a120 2023-10-25T00:07:31.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b7720640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b01ecf60 con 0x7f74b0069d40 2023-10-25T00:07:31.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b7720640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b01ed7c0 con 0x7f74b0065bc0 2023-10-25T00:07:31.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b5cbd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74b006a120 0x7f74b00a9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:31.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b5cbd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74b006a120 0x7f74b00a9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33798/0 (socket says 172.21.15.107:33798) 2023-10-25T00:07:31.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b5cbd640 1 -- 172.21.15.107:0/540248601 learned_addr learned my addr 172.21.15.107:0/540248601 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:31.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b54bc640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74b0065bc0 0x7f74b0153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:31.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b4cbb640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74b0069d40 0x7f74b01f0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:31.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b5cbd640 1 -- 172.21.15.107:0/540248601 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74b0065bc0 msgr2=0x7f74b0153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:31.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b5cbd640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74b0065bc0 0x7f74b0153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b5cbd640 1 -- 172.21.15.107:0/540248601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74b0069d40 msgr2=0x7f74b01f0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:31.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b5cbd640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74b0069d40 0x7f74b01f0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b5cbd640 1 -- 172.21.15.107:0/540248601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a8009b30 con 0x7f74b006a120 2023-10-25T00:07:31.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b5cbd640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74b006a120 0x7f74b00a9500 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f74a801afb0 tx=0x7f74a80196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:31.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74a5ffb640 1 -- 172.21.15.107:0/540248601 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a80b8880 con 0x7f74b006a120 2023-10-25T00:07:31.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74a5ffb640 1 -- 172.21.15.107:0/540248601 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a80b8a00 con 0x7f74b006a120 2023-10-25T00:07:31.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b7720640 1 -- 172.21.15.107:0/540248601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b0154080 con 0x7f74b006a120 2023-10-25T00:07:31.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74a5ffb640 1 -- 172.21.15.107:0/540248601 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a80b8db0 con 0x7f74b006a120 2023-10-25T00:07:31.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.255+0000 7f74b7720640 1 -- 172.21.15.107:0/540248601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b0153da0 con 0x7f74b006a120 2023-10-25T00:07:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.259+0000 7f74b7720640 1 -- 172.21.15.107:0/540248601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b01ed930 con 0x7f74b006a120 2023-10-25T00:07:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.259+0000 7f74a5ffb640 1 -- 172.21.15.107:0/540248601 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f74a80b6440 con 0x7f74b006a120 2023-10-25T00:07:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.259+0000 7f74a5ffb640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7494065e60 0x7f7494068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.259+0000 7f74a5ffb640 1 -- 172.21.15.107:0/540248601 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7f74a811f9f0 con 0x7f74b006a120 2023-10-25T00:07:31.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.259+0000 7f74b54bc640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7494065e60 0x7f7494068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.259+0000 7f74b54bc640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7494065e60 0x7f7494068320 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f74ac007fd0 tx=0x7f74ac007e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.263+0000 7f74a5ffb640 1 -- 172.21.15.107:0/540248601 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f74a80ec570 con 0x7f74b006a120 2023-10-25T00:07:31.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.475+0000 7f74b7720640 1 -- 172.21.15.107:0/540248601 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74b00746e0 con 0x7f74b006a120 2023-10-25T00:07:31.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74a5ffb640 1 -- 172.21.15.107:0/540248601 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f74a8009510 con 0x7f74b006a120 2023-10-25T00:07:31.478 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:31.479 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:31.479 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 125 2023-10-25T00:07:31.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 -- 172.21.15.107:0/540248601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7494065e60 msgr2=0x7f7494068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:31.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7494065e60 0x7f7494068320 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f74ac007fd0 tx=0x7f74ac007e40 comp rx=0 tx=0).stop 2023-10-25T00:07:31.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 -- 172.21.15.107:0/540248601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74b006a120 msgr2=0x7f74b00a9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:31.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74b006a120 0x7f74b00a9500 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f74a801afb0 tx=0x7f74a80196a0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 -- 172.21.15.107:0/540248601 shutdown_connections 2023-10-25T00:07:31.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74b006a120 0x7f74b00a9500 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74b0069d40 0x7f74b01f0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7494065e60 0x7f7494068320 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 --2- 172.21.15.107:0/540248601 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74b0065bc0 0x7f74b0153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 -- 172.21.15.107:0/540248601 >> 172.21.15.107:0/540248601 conn(0x7f74b00c65c0 msgr2=0x7f74b01572d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 -- 172.21.15.107:0/540248601 shutdown_connections 2023-10-25T00:07:31.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:31.479+0000 7f74b7720640 1 -- 172.21.15.107:0/540248601 wait complete. 2023-10-25T00:07:31.497 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:07:33.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:33.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f854a491640 1 Processor -- start 2023-10-25T00:07:33.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f854a491640 1 -- start start 2023-10-25T00:07:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f854a491640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f854413dc20 0x7f85440c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:33.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f854a491640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8544157e50 0x7f85441582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:33.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f854a491640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8544154860 con 0x7f85440720c0 2023-10-25T00:07:33.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f854a491640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8544158800 con 0x7f8544157e50 2023-10-25T00:07:33.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f854a491640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85440c1f10 con 0x7f854413dc20 2023-10-25T00:07:33.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f8548a2e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8544157e50 0x7f85441582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:33.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f85437fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f85440720c0 0x7f854406df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49104/0 (socket says 172.21.15.107:49104) 2023-10-25T00:07:33.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f85437fe640 1 -- 172.21.15.107:0/2983300492 learned_addr learned my addr 172.21.15.107:0/2983300492 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:33.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f8543fff640 1 --2- 172.21.15.107:0/2983300492 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f854413dc20 0x7f85440c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:33.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f8548a2e640 1 -- 172.21.15.107:0/2983300492 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f854413dc20 msgr2=0x7f85440c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:33.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f8548a2e640 1 --2- 172.21.15.107:0/2983300492 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f854413dc20 0x7f85440c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:33.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f8548a2e640 1 -- 172.21.15.107:0/2983300492 >> v1:172.21.15.107:6789/0 conn(0x7f85440720c0 legacy=0x7f854406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:33.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f8548a2e640 1 -- 172.21.15.107:0/2983300492 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8544154b20 con 0x7f8544157e50 2023-10-25T00:07:33.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.635+0000 7f8543fff640 1 --2- 172.21.15.107:0/2983300492 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f854413dc20 0x7f85440c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:07:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f8548a2e640 1 --2- 172.21.15.107:0/2983300492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8544157e50 0x7f85441582c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f853c009d90 tx=0x7f853c01c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffdd9f3c5e8fd594 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f8542ffd640 1 -- 172.21.15.107:0/2983300492 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c0a57e0 con 0x7f8544157e50 2023-10-25T00:07:33.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f8542ffd640 1 -- 172.21.15.107:0/2983300492 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c0a5940 con 0x7f8544157e50 2023-10-25T00:07:33.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f8542ffd640 1 -- 172.21.15.107:0/2983300492 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c0a5cd0 con 0x7f8544157e50 2023-10-25T00:07:33.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f854a491640 1 -- 172.21.15.107:0/2983300492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8544157e50 msgr2=0x7f85441582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:33.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f854a491640 1 --2- 172.21.15.107:0/2983300492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8544157e50 0x7f85441582c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f853c009d90 tx=0x7f853c01c830 comp rx=0 tx=0).stop 2023-10-25T00:07:33.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f854a491640 1 -- 172.21.15.107:0/2983300492 shutdown_connections 2023-10-25T00:07:33.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f854a491640 1 --2- 172.21.15.107:0/2983300492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8544157e50 0x7f85441582c0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:33.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f854a491640 1 --2- 172.21.15.107:0/2983300492 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f854413dc20 0x7f85440c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:33.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f854a491640 1 -- 172.21.15.107:0/2983300492 >> 172.21.15.107:0/2983300492 conn(0x7f85440c39a0 msgr2=0x7f85440c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:33.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f854a491640 1 -- 172.21.15.107:0/2983300492 shutdown_connections 2023-10-25T00:07:33.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.639+0000 7f854a491640 1 -- 172.21.15.107:0/2983300492 wait complete. 2023-10-25T00:07:33.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f854a491640 1 Processor -- start 2023-10-25T00:07:33.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f854a491640 1 -- start start 2023-10-25T00:07:33.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f854a491640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85440720c0 0x7f85440c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:33.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f854a491640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f854413dc20 0x7f8544139c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:33.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f854a491640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8544157e50 0x7f8544135460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:33.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f854a491640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85440ac330 con 0x7f85440720c0 2023-10-25T00:07:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f854a491640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854413b9d0 con 0x7f8544157e50 2023-10-25T00:07:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f854a491640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85440ac050 con 0x7f854413dc20 2023-10-25T00:07:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8543fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85440720c0 0x7f85440c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8543fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85440720c0 0x7f85440c64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54884/0 (socket says 172.21.15.107:54884) 2023-10-25T00:07:33.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8543fff640 1 -- 172.21.15.107:0/2362228576 learned_addr learned my addr 172.21.15.107:0/2362228576 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:33.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f85437fe640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f854413dc20 0x7f8544139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:33.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8543fff640 1 -- 172.21.15.107:0/2362228576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f854413dc20 msgr2=0x7f8544139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:33.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8543fff640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f854413dc20 0x7f8544139c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:33.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8543fff640 1 -- 172.21.15.107:0/2362228576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8544157e50 msgr2=0x7f8544135460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:33.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8543fff640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8544157e50 0x7f8544135460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:33.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8543fff640 1 -- 172.21.15.107:0/2362228576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c009a70 con 0x7f85440720c0 2023-10-25T00:07:33.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8543fff640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85440720c0 0x7f85440c64e0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f853400c820 tx=0x7f853400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:33.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8540ff9640 1 -- 172.21.15.107:0/2362228576 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8534007a30 con 0x7f85440720c0 2023-10-25T00:07:33.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8540ff9640 1 -- 172.21.15.107:0/2362228576 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8534007b90 con 0x7f85440720c0 2023-10-25T00:07:33.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f854a491640 1 -- 172.21.15.107:0/2362228576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85441557a0 con 0x7f85440720c0 2023-10-25T00:07:33.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8540ff9640 1 -- 172.21.15.107:0/2362228576 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853400ce10 con 0x7f85440720c0 2023-10-25T00:07:33.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f854a491640 1 -- 172.21.15.107:0/2362228576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8544155d60 con 0x7f85440720c0 2023-10-25T00:07:33.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f854a491640 1 -- 172.21.15.107:0/2362228576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8544155600 con 0x7f85440720c0 2023-10-25T00:07:33.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.643+0000 7f8540ff9640 1 -- 172.21.15.107:0/2362228576 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f85340113f0 con 0x7f85440720c0 2023-10-25T00:07:33.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.647+0000 7f8540ff9640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8530065e60 0x7f8530068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:33.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.647+0000 7f8540ff9640 1 -- 172.21.15.107:0/2362228576 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7f8534084490 con 0x7f85440720c0 2023-10-25T00:07:33.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.647+0000 7f85437fe640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8530065e60 0x7f8530068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:33.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.647+0000 7f85437fe640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8530065e60 0x7f8530068320 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f854414e7f0 tx=0x7f8538009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:33.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.651+0000 7f8540ff9640 1 -- 172.21.15.107:0/2362228576 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8544155600 con 0x7f85440720c0 2023-10-25T00:07:33.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.907+0000 7f854a491640 1 -- 172.21.15.107:0/2362228576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8544155600 con 0x7f85440720c0 2023-10-25T00:07:33.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.907+0000 7f8540ff9640 1 -- 172.21.15.107:0/2362228576 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f8544155600 con 0x7f85440720c0 2023-10-25T00:07:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:33.910 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:33.910 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 125 2023-10-25T00:07:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 -- 172.21.15.107:0/2362228576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8530065e60 msgr2=0x7f8530068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8530065e60 0x7f8530068320 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f854414e7f0 tx=0x7f8538009040 comp rx=0 tx=0).stop 2023-10-25T00:07:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 -- 172.21.15.107:0/2362228576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85440720c0 msgr2=0x7f85440c64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85440720c0 0x7f85440c64e0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f853400c820 tx=0x7f853400cce0 comp rx=0 tx=0).stop 2023-10-25T00:07:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 -- 172.21.15.107:0/2362228576 shutdown_connections 2023-10-25T00:07:33.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8544157e50 0x7f8544135460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:33.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8530065e60 0x7f8530068320 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:33.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f854413dc20 0x7f8544139c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 --2- 172.21.15.107:0/2362228576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85440720c0 0x7f85440c64e0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 -- 172.21.15.107:0/2362228576 >> 172.21.15.107:0/2362228576 conn(0x7f85440c39a0 msgr2=0x7f85440c05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 -- 172.21.15.107:0/2362228576 shutdown_connections 2023-10-25T00:07:33.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:33.911+0000 7f854a491640 1 -- 172.21.15.107:0/2362228576 wait complete. 2023-10-25T00:07:33.931 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:07:35.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:36.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f9991ec3640 1 Processor -- start 2023-10-25T00:07:36.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f9991ec3640 1 -- start start 2023-10-25T00:07:36.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f9991ec3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f998c0c4e20 0x7f998c0c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:36.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f9991ec3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998c155f20 con 0x7f998c0c5ba0 2023-10-25T00:07:36.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f9991ec3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998c0c6d40 con 0x7f998c14fbd0 2023-10-25T00:07:36.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f9991ec3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998c153560 con 0x7f998c0c4e20 2023-10-25T00:07:36.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f9982ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f998c0c4e20 0x7f998c0c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:36.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f998bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f998c0c5ba0 0x7f998c0c1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49106/0 (socket says 172.21.15.107:49106) 2023-10-25T00:07:36.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f998bfff640 1 -- 172.21.15.107:0/20940991 learned_addr learned my addr 172.21.15.107:0/20940991 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:36.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f9982ffd640 1 -- 172.21.15.107:0/20940991 >> v1:172.21.15.110:6789/0 conn(0x7f998c14fbd0 legacy=0x7f998c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:36.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f9982ffd640 1 -- 172.21.15.107:0/20940991 >> v1:172.21.15.107:6789/0 conn(0x7f998c0c5ba0 legacy=0x7f998c0c1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:36.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.079+0000 7f9982ffd640 1 -- 172.21.15.107:0/20940991 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c1527a0 con 0x7f998c0c4e20 2023-10-25T00:07:36.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9982ffd640 1 --2- 172.21.15.107:0/20940991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c0c4e20 0x7f998c0c18a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f99740097c0 tx=0x7f997401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6609bfda7bfff592 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f998affd640 1 -- 172.21.15.107:0/20940991 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740ab600 con 0x7f998c0c4e20 2023-10-25T00:07:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f998affd640 1 -- 172.21.15.107:0/20940991 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99740a7050 con 0x7f998c0c4e20 2023-10-25T00:07:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f998affd640 1 -- 172.21.15.107:0/20940991 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740ab980 con 0x7f998c0c4e20 2023-10-25T00:07:36.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 -- 172.21.15.107:0/20940991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c0c4e20 msgr2=0x7f998c0c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:36.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 --2- 172.21.15.107:0/20940991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c0c4e20 0x7f998c0c18a0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f99740097c0 tx=0x7f997401c3b0 comp rx=0 tx=0).stop 2023-10-25T00:07:36.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 -- 172.21.15.107:0/20940991 shutdown_connections 2023-10-25T00:07:36.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 --2- 172.21.15.107:0/20940991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c0c4e20 0x7f998c0c18a0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:36.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 -- 172.21.15.107:0/20940991 >> 172.21.15.107:0/20940991 conn(0x7f998c0c39a0 msgr2=0x7f998c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:36.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 -- 172.21.15.107:0/20940991 shutdown_connections 2023-10-25T00:07:36.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 -- 172.21.15.107:0/20940991 wait complete. 2023-10-25T00:07:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 Processor -- start 2023-10-25T00:07:36.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 -- start start 2023-10-25T00:07:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c0c4e20 0x7f998c154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f998c0c5ba0 0x7f998c146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f998c14fbd0 0x7f998c146dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:36.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c14ce00 con 0x7f998c0c4e20 2023-10-25T00:07:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c14c190 con 0x7f998c0c5ba0 2023-10-25T00:07:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9991ec3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998c14f510 con 0x7f998c14fbd0 2023-10-25T00:07:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f998b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c0c4e20 0x7f998c154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:36.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f998b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c0c4e20 0x7f998c154a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54898/0 (socket says 172.21.15.107:54898) 2023-10-25T00:07:36.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f998b7fe640 1 -- 172.21.15.107:0/2424161066 learned_addr learned my addr 172.21.15.107:0/2424161066 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:36.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f9982ffd640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f998c0c5ba0 0x7f998c146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:36.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f998b7fe640 1 -- 172.21.15.107:0/2424161066 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f998c14fbd0 msgr2=0x7f998c146dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:36.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.083+0000 7f998bfff640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f998c14fbd0 0x7f998c146dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:36.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f998b7fe640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f998c14fbd0 0x7f998c146dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f998b7fe640 1 -- 172.21.15.107:0/2424161066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f998c0c5ba0 msgr2=0x7f998c146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f998b7fe640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f998c0c5ba0 0x7f998c146890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f998b7fe640 1 -- 172.21.15.107:0/2424161066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9974009420 con 0x7f998c0c4e20 2023-10-25T00:07:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f998bfff640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f998c14fbd0 0x7f998c146dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:07:36.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9982ffd640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f998c0c5ba0 0x7f998c146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:07:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f998b7fe640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c0c4e20 0x7f998c154a50 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f997c00b410 tx=0x7f997c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9988ff9640 1 -- 172.21.15.107:0/2424161066 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c004020 con 0x7f998c0c4e20 2023-10-25T00:07:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9988ff9640 1 -- 172.21.15.107:0/2424161066 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997c004180 con 0x7f998c0c4e20 2023-10-25T00:07:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9988ff9640 1 -- 172.21.15.107:0/2424161066 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c011630 con 0x7f998c0c4e20 2023-10-25T00:07:36.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9991ec3640 1 -- 172.21.15.107:0/2424161066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998c0c14c0 con 0x7f998c0c4e20 2023-10-25T00:07:36.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9991ec3640 1 -- 172.21.15.107:0/2424161066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f998c14c680 con 0x7f998c0c4e20 2023-10-25T00:07:36.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9991ec3640 1 -- 172.21.15.107:0/2424161066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998c1502e0 con 0x7f998c0c4e20 2023-10-25T00:07:36.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9988ff9640 1 -- 172.21.15.107:0/2424161066 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f997c011790 con 0x7f998c0c4e20 2023-10-25T00:07:36.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9988ff9640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f996c065e60 0x7f996c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:36.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9988ff9640 1 -- 172.21.15.107:0/2424161066 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7f997c084060 con 0x7f998c0c4e20 2023-10-25T00:07:36.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9982ffd640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f996c065e60 0x7f996c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:36.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.087+0000 7f9982ffd640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f996c065e60 0x7f996c068320 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f99740ad000 tx=0x7f9974008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:36.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.091+0000 7f9988ff9640 1 -- 172.21.15.107:0/2424161066 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f997c050bb0 con 0x7f998c0c4e20 2023-10-25T00:07:36.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 -- 172.21.15.107:0/2424161066 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f998c0746e0 con 0x7f998c0c4e20 2023-10-25T00:07:36.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9988ff9640 1 -- 172.21.15.107:0/2424161066 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f997c015090 con 0x7f998c0c4e20 2023-10-25T00:07:36.312 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:36.312 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 125 2023-10-25T00:07:36.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 -- 172.21.15.107:0/2424161066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f996c065e60 msgr2=0x7f996c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:36.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f996c065e60 0x7f996c068320 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f99740ad000 tx=0x7f9974008040 comp rx=0 tx=0).stop 2023-10-25T00:07:36.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 -- 172.21.15.107:0/2424161066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c0c4e20 msgr2=0x7f998c154a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:36.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c0c4e20 0x7f998c154a50 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f997c00b410 tx=0x7f997c00b8d0 comp rx=0 tx=0).stop 2023-10-25T00:07:36.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 -- 172.21.15.107:0/2424161066 shutdown_connections 2023-10-25T00:07:36.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f996c065e60 0x7f996c068320 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:36.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f998c14fbd0 0x7f998c146dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:36.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f998c0c5ba0 0x7f998c146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:36.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 --2- 172.21.15.107:0/2424161066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f998c0c4e20 0x7f998c154a50 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:36.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 -- 172.21.15.107:0/2424161066 >> 172.21.15.107:0/2424161066 conn(0x7f998c0c39a0 msgr2=0x7f998c06cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:36.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 -- 172.21.15.107:0/2424161066 shutdown_connections 2023-10-25T00:07:36.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:36.311+0000 7f9991ec3640 1 -- 172.21.15.107:0/2424161066 wait complete. 2023-10-25T00:07:36.334 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:07:38.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:38.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a8359640 1 Processor -- start 2023-10-25T00:07:38.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a8359640 1 -- start start 2023-10-25T00:07:38.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a8359640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc9a00c33e0 0x7fc9a015ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:38.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a8359640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a00c22f0 con 0x7fc9a014eb70 2023-10-25T00:07:38.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a8359640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a00c25d0 con 0x7fc9a0070b00 2023-10-25T00:07:38.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a8359640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a0071700 con 0x7fc9a00c33e0 2023-10-25T00:07:38.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a60f5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc9a014eb70 0x7fc9a01514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49114/0 (socket says 172.21.15.107:49114) 2023-10-25T00:07:38.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a60f5640 1 -- 172.21.15.107:0/2487107487 learned_addr learned my addr 172.21.15.107:0/2487107487 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:38.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453764728 0 0) 0x7fc9a00c22f0 con 0x7fc9a014eb70 2023-10-25T00:07:38.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a68f6640 1 --2- 172.21.15.107:0/2487107487 >> v2:172.21.15.110:3300/0 conn(0x7fc9a00c33e0 0x7fc9a015ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:38.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc98c003530 con 0x7fc9a014eb70 2023-10-25T00:07:38.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 912874777 0 0) 0x7fc98c003530 con 0x7fc9a014eb70 2023-10-25T00:07:38.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 >> v2:172.21.15.110:3300/0 conn(0x7fc9a00c33e0 msgr2=0x7fc9a015ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:38.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 --2- 172.21.15.107:0/2487107487 >> v2:172.21.15.110:3300/0 conn(0x7fc9a00c33e0 0x7fc9a015ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:38.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 >> v1:172.21.15.110:6789/0 conn(0x7fc9a0070b00 legacy=0x7fc9a006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:38.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a00c2be0 con 0x7fc9a014eb70 2023-10-25T00:07:38.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fc99c001680 con 0x7fc9a014eb70 2023-10-25T00:07:38.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 >> v1:172.21.15.107:6789/0 conn(0x7fc9a014eb70 legacy=0x7fc9a01514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:38.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc98c003980 0x7fc98c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:38.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98c0047d0 0x7fc98c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:38.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc98c005190 0x7fc98c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:38.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c00c000 con 0x7fc98c005190 2023-10-25T00:07:38.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c00c170 con 0x7fc98c0047d0 2023-10-25T00:07:38.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc98c00c330 con 0x7fc98c003980 2023-10-25T00:07:38.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a68f6640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc98c005190 0x7fc98c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:38.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a60f5640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc98c003980 0x7fc98c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:38.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a58f4640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98c0047d0 0x7fc98c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:38.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a68f6640 1 -- 172.21.15.107:0/2487107487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc98c003980 msgr2=0x7fc98c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:38.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a68f6640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc98c003980 0x7fc98c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:38.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a68f6640 1 -- 172.21.15.107:0/2487107487 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98c0047d0 msgr2=0x7fc98c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:38.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a68f6640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98c0047d0 0x7fc98c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:38.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a68f6640 1 -- 172.21.15.107:0/2487107487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98c003530 con 0x7fc98c005190 2023-10-25T00:07:38.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.487+0000 7fc9a60f5640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc98c003980 0x7fc98c003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:07:38.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a68f6640 1 -- 172.21.15.107:0/2487107487 reap_dead start 2023-10-25T00:07:38.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a68f6640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc98c005190 0x7fc98c00b990 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fc998009fc0 tx=0x7fc99801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1f468e38286fe3 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:38.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9980a5cb0 con 0x7fc98c005190 2023-10-25T00:07:38.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99800f040 con 0x7fc98c005190 2023-10-25T00:07:38.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a50f3640 1 -- 172.21.15.107:0/2487107487 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc998003be0 con 0x7fc98c005190 2023-10-25T00:07:38.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a8359640 1 -- 172.21.15.107:0/2487107487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc98c005190 msgr2=0x7fc98c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:38.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a8359640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc98c005190 0x7fc98c00b990 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fc998009fc0 tx=0x7fc99801cca0 comp rx=0 tx=0).stop 2023-10-25T00:07:38.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a8359640 1 -- 172.21.15.107:0/2487107487 shutdown_connections 2023-10-25T00:07:38.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a8359640 1 --2- 172.21.15.107:0/2487107487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc98c005190 0x7fc98c00b990 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:38.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a8359640 1 -- 172.21.15.107:0/2487107487 >> 172.21.15.107:0/2487107487 conn(0x7fc9a00c39a0 msgr2=0x7fc9a00c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:38.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a8359640 1 -- 172.21.15.107:0/2487107487 shutdown_connections 2023-10-25T00:07:38.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.491+0000 7fc9a8359640 1 -- 172.21.15.107:0/2487107487 wait complete. 2023-10-25T00:07:38.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a8359640 1 Processor -- start 2023-10-25T00:07:38.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a8359640 1 -- start start 2023-10-25T00:07:38.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a8359640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98c005190 0x7fc9a0142950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:38.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a8359640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a0150100 0x7fc9a0150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:38.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a8359640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9a01dd160 0x7fc9a01dd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:38.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a8359640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a013c610 con 0x7fc9a0150100 2023-10-25T00:07:38.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a8359640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a01db570 con 0x7fc98c005190 2023-10-25T00:07:38.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a8359640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a006afd0 con 0x7fc9a01dd160 2023-10-25T00:07:38.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a58f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a0150100 0x7fc9a0150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:38.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a58f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a0150100 0x7fc9a0150560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54908/0 (socket says 172.21.15.107:54908) 2023-10-25T00:07:38.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a58f4640 1 -- 172.21.15.107:0/3825738785 learned_addr learned my addr 172.21.15.107:0/3825738785 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:38.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a60f5640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98c005190 0x7fc9a0142950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:38.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a68f6640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9a01dd160 0x7fc9a01dd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:38.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a58f4640 1 -- 172.21.15.107:0/3825738785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9a01dd160 msgr2=0x7fc9a01dd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:38.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a58f4640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9a01dd160 0x7fc9a01dd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:38.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a58f4640 1 -- 172.21.15.107:0/3825738785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98c005190 msgr2=0x7fc9a0142950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:38.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a58f4640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98c005190 0x7fc9a0142950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:38.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a58f4640 1 -- 172.21.15.107:0/3825738785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a0150cc0 con 0x7fc9a0150100 2023-10-25T00:07:38.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a58f4640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a0150100 0x7fc9a0150560 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fc99000f5c0 tx=0x7fc99000fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:38.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc996ffd640 1 -- 172.21.15.107:0/3825738785 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9900052e0 con 0x7fc9a0150100 2023-10-25T00:07:38.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc996ffd640 1 -- 172.21.15.107:0/3825738785 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc990005440 con 0x7fc9a0150100 2023-10-25T00:07:38.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a8359640 1 -- 172.21.15.107:0/3825738785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a0150fa0 con 0x7fc9a0150100 2023-10-25T00:07:38.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc996ffd640 1 -- 172.21.15.107:0/3825738785 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990003820 con 0x7fc9a0150100 2023-10-25T00:07:38.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a8359640 1 -- 172.21.15.107:0/3825738785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a0142e90 con 0x7fc9a0150100 2023-10-25T00:07:38.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc9a8359640 1 -- 172.21.15.107:0/3825738785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a006b1f0 con 0x7fc9a0150100 2023-10-25T00:07:38.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc996ffd640 1 -- 172.21.15.107:0/3825738785 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc990003980 con 0x7fc9a0150100 2023-10-25T00:07:38.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.495+0000 7fc996ffd640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc984065e90 0x7fc984068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:38.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.499+0000 7fc996ffd640 1 -- 172.21.15.107:0/3825738785 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7fc990084170 con 0x7fc9a0150100 2023-10-25T00:07:38.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.499+0000 7fc9a60f5640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc984065e90 0x7fc984068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:38.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.499+0000 7fc9a60f5640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc984065e90 0x7fc984068350 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0059a0 tx=0x7fc99c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:38.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.503+0000 7fc996ffd640 1 -- 172.21.15.107:0/3825738785 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc9a006b1f0 con 0x7fc9a0150100 2023-10-25T00:07:38.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.711+0000 7fc9a8359640 1 -- 172.21.15.107:0/3825738785 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9a006b1f0 con 0x7fc9a0150100 2023-10-25T00:07:38.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.711+0000 7fc996ffd640 1 -- 172.21.15.107:0/3825738785 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5778 (secure 0 0 0) 0x7fc9a006b1f0 con 0x7fc9a0150100 2023-10-25T00:07:38.712 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:38.712 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:38.712 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 125 2023-10-25T00:07:38.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.711+0000 7fc9a8359640 1 -- 172.21.15.107:0/3825738785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc984065e90 msgr2=0x7fc984068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:38.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.711+0000 7fc9a8359640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc984065e90 0x7fc984068350 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0059a0 tx=0x7fc99c009040 comp rx=0 tx=0).stop 2023-10-25T00:07:38.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.711+0000 7fc9a8359640 1 -- 172.21.15.107:0/3825738785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a0150100 msgr2=0x7fc9a0150560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:38.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.711+0000 7fc9a8359640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a0150100 0x7fc9a0150560 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fc99000f5c0 tx=0x7fc99000fa80 comp rx=0 tx=0).stop 2023-10-25T00:07:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.715+0000 7fc9a8359640 1 -- 172.21.15.107:0/3825738785 shutdown_connections 2023-10-25T00:07:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.715+0000 7fc9a8359640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc984065e90 0x7fc984068350 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.715+0000 7fc9a8359640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc9a01dd160 0x7fc9a01dd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.715+0000 7fc9a8359640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9a0150100 0x7fc9a0150560 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:38.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.715+0000 7fc9a8359640 1 --2- 172.21.15.107:0/3825738785 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc98c005190 0x7fc9a0142950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.715+0000 7fc9a8359640 1 -- 172.21.15.107:0/3825738785 >> 172.21.15.107:0/3825738785 conn(0x7fc9a00c39a0 msgr2=0x7fc9a00c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.715+0000 7fc9a8359640 1 -- 172.21.15.107:0/3825738785 shutdown_connections 2023-10-25T00:07:38.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:38.715+0000 7fc9a8359640 1 -- 172.21.15.107:0/3825738785 wait complete. 2023-10-25T00:07:38.731 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:07:40.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:40.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f38473d4640 1 Processor -- start 2023-10-25T00:07:40.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f38473d4640 1 -- start start 2023-10-25T00:07:40.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f38473d4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f38400a9450 0x7f3840071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:40.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f38473d4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3840152d40 con 0x7f384013bb00 2023-10-25T00:07:40.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f38473d4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384006d660 con 0x7f38400aa330 2023-10-25T00:07:40.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f38473d4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3840156100 con 0x7f38400a9450 2023-10-25T00:07:40.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3845170640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f384013bb00 0x7f38401531d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49128/0 (socket says 172.21.15.107:49128) 2023-10-25T00:07:40.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3845170640 1 -- 172.21.15.107:0/3857718644 learned_addr learned my addr 172.21.15.107:0/3857718644 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:40.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3845971640 1 --2- 172.21.15.107:0/3857718644 >> v2:172.21.15.110:3300/0 conn(0x7f38400a9450 0x7f3840071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:40.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969653494 0 0) 0x7f3840152d40 con 0x7f384013bb00 2023-10-25T00:07:40.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f382c003530 con 0x7f384013bb00 2023-10-25T00:07:40.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3112017179 0 0) 0x7f382c003530 con 0x7f384013bb00 2023-10-25T00:07:40.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 >> v2:172.21.15.110:3300/0 conn(0x7f38400a9450 msgr2=0x7f3840071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:40.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 --2- 172.21.15.107:0/3857718644 >> v2:172.21.15.110:3300/0 conn(0x7f38400a9450 0x7f3840071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:40.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 >> v1:172.21.15.110:6789/0 conn(0x7f38400aa330 legacy=0x7f384006e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:40.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3840155e60 con 0x7f384013bb00 2023-10-25T00:07:40.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f383c001680 con 0x7f384013bb00 2023-10-25T00:07:40.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 >> v1:172.21.15.107:6789/0 conn(0x7f384013bb00 legacy=0x7f38401531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:40.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 --2- 172.21.15.107:0/3857718644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f382c0039b0 0x7f382c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:40.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 --2- 172.21.15.107:0/3857718644 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f382c0047d0 0x7f382c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:40.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 --2- 172.21.15.107:0/3857718644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f382c0052b0 0x7f382c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:40.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840156100 con 0x7f382c0052b0 2023-10-25T00:07:40.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382c00c1b0 con 0x7f382c0039b0 2023-10-25T00:07:40.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.867+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382c00c370 con 0x7f382c0047d0 2023-10-25T00:07:40.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3845971640 1 --2- 172.21.15.107:0/3857718644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f382c0052b0 0x7f382c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:40.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3845170640 1 --2- 172.21.15.107:0/3857718644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f382c0039b0 0x7f382c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:40.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f384496f640 1 --2- 172.21.15.107:0/3857718644 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f382c0047d0 0x7f382c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:40.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3845971640 1 -- 172.21.15.107:0/3857718644 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f382c0047d0 msgr2=0x7f382c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:40.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3845971640 1 --2- 172.21.15.107:0/3857718644 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f382c0047d0 0x7f382c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:40.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3845971640 1 -- 172.21.15.107:0/3857718644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f382c0039b0 msgr2=0x7f382c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:40.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3845971640 1 --2- 172.21.15.107:0/3857718644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f382c0039b0 0x7f382c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:40.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3845971640 1 -- 172.21.15.107:0/3857718644 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382c003530 con 0x7f382c0052b0 2023-10-25T00:07:40.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3845971640 1 -- 172.21.15.107:0/3857718644 reap_dead start 2023-10-25T00:07:40.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3845971640 1 --2- 172.21.15.107:0/3857718644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f382c0052b0 0x7f382c00b990 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f3838009f90 tx=0x7f383801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3c6fcb8ab0d75c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:40.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838003ba0 con 0x7f382c0052b0 2023-10-25T00:07:40.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383800f040 con 0x7f382c0052b0 2023-10-25T00:07:40.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f3837fff640 1 -- 172.21.15.107:0/3857718644 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38380a5940 con 0x7f382c0052b0 2023-10-25T00:07:40.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f38473d4640 1 -- 172.21.15.107:0/3857718644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f382c0052b0 msgr2=0x7f382c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:40.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f38473d4640 1 --2- 172.21.15.107:0/3857718644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f382c0052b0 0x7f382c00b990 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f3838009f90 tx=0x7f383801cc60 comp rx=0 tx=0).stop 2023-10-25T00:07:40.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f38473d4640 1 -- 172.21.15.107:0/3857718644 shutdown_connections 2023-10-25T00:07:40.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f38473d4640 1 --2- 172.21.15.107:0/3857718644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f382c0052b0 0x7f382c00b990 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:40.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f38473d4640 1 -- 172.21.15.107:0/3857718644 >> 172.21.15.107:0/3857718644 conn(0x7f3840070f60 msgr2=0x7f3840071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:40.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f38473d4640 1 -- 172.21.15.107:0/3857718644 shutdown_connections 2023-10-25T00:07:40.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.871+0000 7f38473d4640 1 -- 172.21.15.107:0/3857718644 wait complete. 2023-10-25T00:07:40.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f38473d4640 1 Processor -- start 2023-10-25T00:07:40.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f38473d4640 1 -- start start 2023-10-25T00:07:40.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f38473d4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f382c0052b0 0x7f38401ee420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:40.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f38473d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f384014f960 0x7f384014f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:40.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f38473d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38400c6f40 0x7f38400c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f38473d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38401eb910 con 0x7f38400c6f40 2023-10-25T00:07:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f38473d4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38400aae40 con 0x7f382c0052b0 2023-10-25T00:07:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f38473d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38400ab120 con 0x7f384014f960 2023-10-25T00:07:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3845971640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38400c6f40 0x7f38400c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:40.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3845971640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38400c6f40 0x7f38400c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54926/0 (socket says 172.21.15.107:54926) 2023-10-25T00:07:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3845971640 1 -- 172.21.15.107:0/1440082603 learned_addr learned my addr 172.21.15.107:0/1440082603 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3845170640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f382c0052b0 0x7f38401ee420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f384496f640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f384014f960 0x7f384014f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3845971640 1 -- 172.21.15.107:0/1440082603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f384014f960 msgr2=0x7f384014f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3845971640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f384014f960 0x7f384014f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:40.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3845971640 1 -- 172.21.15.107:0/1440082603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f382c0052b0 msgr2=0x7f38401ee420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:40.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3845971640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f382c0052b0 0x7f38401ee420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:40.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3845971640 1 -- 172.21.15.107:0/1440082603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38401ebb60 con 0x7f38400c6f40 2023-10-25T00:07:40.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3845971640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38400c6f40 0x7f38400c73c0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f383801d500 tx=0x7f3838009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:40.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3835ffb640 1 -- 172.21.15.107:0/1440082603 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38380a5c00 con 0x7f38400c6f40 2023-10-25T00:07:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3835ffb640 1 -- 172.21.15.107:0/1440082603 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38380a5d60 con 0x7f38400c6f40 2023-10-25T00:07:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f38473d4640 1 -- 172.21.15.107:0/1440082603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384014f6e0 con 0x7f38400c6f40 2023-10-25T00:07:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3835ffb640 1 -- 172.21.15.107:0/1440082603 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838004260 con 0x7f38400c6f40 2023-10-25T00:07:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f38473d4640 1 -- 172.21.15.107:0/1440082603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38401ec080 con 0x7f38400c6f40 2023-10-25T00:07:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f38473d4640 1 -- 172.21.15.107:0/1440082603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38400746e0 con 0x7f38400c6f40 2023-10-25T00:07:40.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.875+0000 7f3835ffb640 1 -- 172.21.15.107:0/1440082603 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f38380043c0 con 0x7f38400c6f40 2023-10-25T00:07:40.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.879+0000 7f3835ffb640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3824065e90 0x7f3824068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:40.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.879+0000 7f3845170640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3824065e90 0x7f3824068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:40.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.879+0000 7f3835ffb640 1 -- 172.21.15.107:0/1440082603 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7f383811f220 con 0x7f38400c6f40 2023-10-25T00:07:40.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.879+0000 7f3845170640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3824065e90 0x7f3824068350 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f383c0017a0 tx=0x7f383c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:40.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:40.883+0000 7f3835ffb640 1 -- 172.21.15.107:0/1440082603 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f38380ebd70 con 0x7f38400c6f40 2023-10-25T00:07:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.107+0000 7f38473d4640 1 -- 172.21.15.107:0/1440082603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f384006d160 con 0x7f38400c6f40 2023-10-25T00:07:41.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.107+0000 7f3835ffb640 1 -- 172.21.15.107:0/1440082603 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f38380ef390 con 0x7f38400c6f40 2023-10-25T00:07:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:41.109 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 125 2023-10-25T00:07:41.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 -- 172.21.15.107:0/1440082603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3824065e90 msgr2=0x7f3824068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3824065e90 0x7f3824068350 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f383c0017a0 tx=0x7f383c009040 comp rx=0 tx=0).stop 2023-10-25T00:07:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 -- 172.21.15.107:0/1440082603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38400c6f40 msgr2=0x7f38400c73c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38400c6f40 0x7f38400c73c0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f383801d500 tx=0x7f3838009750 comp rx=0 tx=0).stop 2023-10-25T00:07:41.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 -- 172.21.15.107:0/1440082603 shutdown_connections 2023-10-25T00:07:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38400c6f40 0x7f38400c73c0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3824065e90 0x7f3824068350 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f384014f960 0x7f384014f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 --2- 172.21.15.107:0/1440082603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f382c0052b0 0x7f38401ee420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:41.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 -- 172.21.15.107:0/1440082603 >> 172.21.15.107:0/1440082603 conn(0x7f3840070f60 msgr2=0x7f3840151b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 -- 172.21.15.107:0/1440082603 shutdown_connections 2023-10-25T00:07:41.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:41.111+0000 7f38473d4640 1 -- 172.21.15.107:0/1440082603 wait complete. 2023-10-25T00:07:41.125 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:07:43.127 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:43.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f8dfe640 1 Processor -- start 2023-10-25T00:07:43.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f8dfe640 1 -- start start 2023-10-25T00:07:43.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f8dfe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f62f4069d40 0x7f62f4158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f8dfe640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f406cea0 con 0x7f62f406a120 2023-10-25T00:07:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f8dfe640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f41584e0 con 0x7f62f4065bc0 2023-10-25T00:07:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f8dfe640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f40ac650 con 0x7f62f4069d40 2023-10-25T00:07:43.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f2d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f62f406a120 0x7f62f40aaef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33666/0 (socket says 172.21.15.107:33666) 2023-10-25T00:07:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f2d9d640 1 -- 172.21.15.107:0/1351834744 learned_addr learned my addr 172.21.15.107:0/1351834744 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f259c640 1 --2- 172.21.15.107:0/1351834744 >> v2:172.21.15.110:3300/0 conn(0x7f62f4069d40 0x7f62f4158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710978107 0 0) 0x7f62f406cea0 con 0x7f62f406a120 2023-10-25T00:07:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62dc003530 con 0x7f62f406a120 2023-10-25T00:07:43.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3410850249 0 0) 0x7f62dc003530 con 0x7f62f406a120 2023-10-25T00:07:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 >> v2:172.21.15.110:3300/0 conn(0x7f62f4069d40 msgr2=0x7f62f4158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 --2- 172.21.15.107:0/1351834744 >> v2:172.21.15.110:3300/0 conn(0x7f62f4069d40 0x7f62f4158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 >> v1:172.21.15.110:6789/0 conn(0x7f62f4065bc0 legacy=0x7f62f40a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f406d3d0 con 0x7f62f406a120 2023-10-25T00:07:43.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f62ec001a00 con 0x7f62f406a120 2023-10-25T00:07:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 >> v1:172.21.15.107:6789/0 conn(0x7f62f406a120 legacy=0x7f62f40aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 --2- 172.21.15.107:0/1351834744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc003980 0x7f62dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 --2- 172.21.15.107:0/1351834744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 0x7f62dc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 --2- 172.21.15.107:0/1351834744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62dc005190 0x7f62dc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:43.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc00c100 con 0x7f62dc0047d0 2023-10-25T00:07:43.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc00c270 con 0x7f62dc005190 2023-10-25T00:07:43.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc00c430 con 0x7f62dc003980 2023-10-25T00:07:43.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f1d9b640 1 --2- 172.21.15.107:0/1351834744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 0x7f62dc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:43.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f259c640 1 --2- 172.21.15.107:0/1351834744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc003980 0x7f62dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:43.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f2d9d640 1 --2- 172.21.15.107:0/1351834744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62dc005190 0x7f62dc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f1d9b640 1 -- 172.21.15.107:0/1351834744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc003980 msgr2=0x7f62dc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f1d9b640 1 --2- 172.21.15.107:0/1351834744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62dc003980 0x7f62dc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f1d9b640 1 -- 172.21.15.107:0/1351834744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62dc005190 msgr2=0x7f62dc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f1d9b640 1 --2- 172.21.15.107:0/1351834744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62dc005190 0x7f62dc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f1d9b640 1 -- 172.21.15.107:0/1351834744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc003530 con 0x7f62dc0047d0 2023-10-25T00:07:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.271+0000 7f62f2d9d640 1 -- 172.21.15.107:0/1351834744 reap_dead start 2023-10-25T00:07:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.275+0000 7f62f1d9b640 1 --2- 172.21.15.107:0/1351834744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 0x7f62dc004c50 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f62e8001780 tx=0x7f62e801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa26c2fc2513e92 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:43.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.275+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e80052d0 con 0x7f62dc0047d0 2023-10-25T00:07:43.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.275+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e80a7050 con 0x7f62dc0047d0 2023-10-25T00:07:43.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.275+0000 7f62f159a640 1 -- 172.21.15.107:0/1351834744 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e8003600 con 0x7f62dc0047d0 2023-10-25T00:07:43.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.275+0000 7f62f8dfe640 1 -- 172.21.15.107:0/1351834744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 msgr2=0x7f62dc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:43.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.275+0000 7f62f8dfe640 1 --2- 172.21.15.107:0/1351834744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 0x7f62dc004c50 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f62e8001780 tx=0x7f62e801c3e0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.275+0000 7f62f8dfe640 1 -- 172.21.15.107:0/1351834744 shutdown_connections 2023-10-25T00:07:43.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.275+0000 7f62f8dfe640 1 --2- 172.21.15.107:0/1351834744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 0x7f62dc004c50 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.275+0000 7f62f8dfe640 1 -- 172.21.15.107:0/1351834744 >> 172.21.15.107:0/1351834744 conn(0x7f62f40c65c0 msgr2=0x7f62f40c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:43.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.275+0000 7f62f8dfe640 1 -- 172.21.15.107:0/1351834744 shutdown_connections 2023-10-25T00:07:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.275+0000 7f62f8dfe640 1 -- 172.21.15.107:0/1351834744 wait complete. 2023-10-25T00:07:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f8dfe640 1 Processor -- start 2023-10-25T00:07:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f8dfe640 1 -- start start 2023-10-25T00:07:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f8dfe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 0x7f62f41ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f8dfe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62f40a7cd0 0x7f62f4071af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f8dfe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62f40bc8d0 0x7f62f414b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f8dfe640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41e03c0 con 0x7f62dc0047d0 2023-10-25T00:07:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f8dfe640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f406f7b0 con 0x7f62f40a7cd0 2023-10-25T00:07:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f8dfe640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41e05d0 con 0x7f62f40bc8d0 2023-10-25T00:07:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 0x7f62f41ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 0x7f62f41ee380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59204/0 (socket says 172.21.15.107:59204) 2023-10-25T00:07:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f259c640 1 -- 172.21.15.107:0/11431076 learned_addr learned my addr 172.21.15.107:0/11431076 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f2d9d640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62f40bc8d0 0x7f62f414b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f1d9b640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62f40a7cd0 0x7f62f4071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f259c640 1 -- 172.21.15.107:0/11431076 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62f40bc8d0 msgr2=0x7f62f414b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:43.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f259c640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62f40bc8d0 0x7f62f414b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f259c640 1 -- 172.21.15.107:0/11431076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62f40a7cd0 msgr2=0x7f62f4071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f259c640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62f40a7cd0 0x7f62f4071af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f259c640 1 -- 172.21.15.107:0/11431076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f4151450 con 0x7f62dc0047d0 2023-10-25T00:07:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f2d9d640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62f40bc8d0 0x7f62f414b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:07:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f259c640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 0x7f62f41ee380 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f62e4011690 tx=0x7f62e4011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62e2ffd640 1 -- 172.21.15.107:0/11431076 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e400f9c0 con 0x7f62dc0047d0 2023-10-25T00:07:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62e2ffd640 1 -- 172.21.15.107:0/11431076 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e400fb20 con 0x7f62dc0047d0 2023-10-25T00:07:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f8dfe640 1 -- 172.21.15.107:0/11431076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f4151730 con 0x7f62dc0047d0 2023-10-25T00:07:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62e2ffd640 1 -- 172.21.15.107:0/11431076 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e400fe90 con 0x7f62dc0047d0 2023-10-25T00:07:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f8dfe640 1 -- 172.21.15.107:0/11431076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f4072030 con 0x7f62dc0047d0 2023-10-25T00:07:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62f8dfe640 1 -- 172.21.15.107:0/11431076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f4157e70 con 0x7f62dc0047d0 2023-10-25T00:07:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62e2ffd640 1 -- 172.21.15.107:0/11431076 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f62e4015460 con 0x7f62dc0047d0 2023-10-25T00:07:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.279+0000 7f62e2ffd640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62d4065e90 0x7f62d4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.283+0000 7f62e2ffd640 1 -- 172.21.15.107:0/11431076 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7f62e40848a0 con 0x7f62dc0047d0 2023-10-25T00:07:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.283+0000 7f62f1d9b640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62d4065e90 0x7f62d4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.283+0000 7f62f1d9b640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62d4065e90 0x7f62d4068350 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f62e8001750 tx=0x7f62e80011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.287+0000 7f62e2ffd640 1 -- 172.21.15.107:0/11431076 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f62f4157e70 con 0x7f62dc0047d0 2023-10-25T00:07:43.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 -- 172.21.15.107:0/11431076 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62f4157e70 con 0x7f62dc0047d0 2023-10-25T00:07:43.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62e2ffd640 1 -- 172.21.15.107:0/11431076 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f62f4157e70 con 0x7f62dc0047d0 2023-10-25T00:07:43.531 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:43.532 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 125 2023-10-25T00:07:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 -- 172.21.15.107:0/11431076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62d4065e90 msgr2=0x7f62d4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62d4065e90 0x7f62d4068350 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f62e8001750 tx=0x7f62e80011a0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 -- 172.21.15.107:0/11431076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 msgr2=0x7f62f41ee380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:43.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 0x7f62f41ee380 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f62e4011690 tx=0x7f62e4011b50 comp rx=0 tx=0).stop 2023-10-25T00:07:43.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 -- 172.21.15.107:0/11431076 shutdown_connections 2023-10-25T00:07:43.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f62d4065e90 0x7f62d4068350 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62f40bc8d0 0x7f62f414b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62f40a7cd0 0x7f62f4071af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 --2- 172.21.15.107:0/11431076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62dc0047d0 0x7f62f41ee380 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:43.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 -- 172.21.15.107:0/11431076 >> 172.21.15.107:0/11431076 conn(0x7f62f40c65c0 msgr2=0x7f62f406c490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:43.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 -- 172.21.15.107:0/11431076 shutdown_connections 2023-10-25T00:07:43.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:43.531+0000 7f62f8dfe640 1 -- 172.21.15.107:0/11431076 wait complete. 2023-10-25T00:07:43.552 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:07:45.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:45.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095fe90640 1 Processor -- start 2023-10-25T00:07:45.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095fe90640 1 -- start start 2023-10-25T00:07:45.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095fe90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f095806d8d0 0x7f09580a8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:45.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095fe90640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f095806aaa0 0x7f09580c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:45.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095fe90640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 0x7f09580c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:45.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095fe90640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09580ac7c0 con 0x7f095806d8d0 2023-10-25T00:07:45.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095fe90640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09580bb940 con 0x7f095806aaa0 2023-10-25T00:07:45.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095fe90640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09580a9220 con 0x7f0958153510 2023-10-25T00:07:45.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095dc2c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f095806d8d0 0x7f09580a8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:45.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095dc2c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f095806d8d0 0x7f09580a8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59206/0 (socket says 172.21.15.107:59206) 2023-10-25T00:07:45.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095dc2c640 1 -- 172.21.15.107:0/170791392 learned_addr learned my addr 172.21.15.107:0/170791392 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:45.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.679+0000 7f095e42d640 1 --2- 172.21.15.107:0/170791392 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 0x7f09580c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:45.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.683+0000 7f095d42b640 1 --2- 172.21.15.107:0/170791392 >> v2:172.21.15.110:3300/0 conn(0x7f095806aaa0 0x7f09580c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:45.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.683+0000 7f095dc2c640 1 -- 172.21.15.107:0/170791392 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 msgr2=0x7f09580c2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:45.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.683+0000 7f095dc2c640 1 --2- 172.21.15.107:0/170791392 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 0x7f09580c2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.683+0000 7f095dc2c640 1 -- 172.21.15.107:0/170791392 >> v2:172.21.15.110:3300/0 conn(0x7f095806aaa0 msgr2=0x7f09580c25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:45.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.683+0000 7f095dc2c640 1 --2- 172.21.15.107:0/170791392 >> v2:172.21.15.110:3300/0 conn(0x7f095806aaa0 0x7f09580c25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.683+0000 7f095dc2c640 1 -- 172.21.15.107:0/170791392 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09580abcb0 con 0x7f095806d8d0 2023-10-25T00:07:45.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.683+0000 7f095e42d640 1 --2- 172.21.15.107:0/170791392 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 0x7f09580c2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:07:45.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.683+0000 7f095dc2c640 1 --2- 172.21.15.107:0/170791392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f095806d8d0 0x7f09580a8ce0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f095806c400 tx=0x7f095401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e616a1e817698b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:45.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.683+0000 7f095cc2a640 1 -- 172.21.15.107:0/170791392 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09540a5840 con 0x7f095806d8d0 2023-10-25T00:07:45.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.683+0000 7f095cc2a640 1 -- 172.21.15.107:0/170791392 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09540a59a0 con 0x7f095806d8d0 2023-10-25T00:07:45.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.683+0000 7f095cc2a640 1 -- 172.21.15.107:0/170791392 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09540a5d30 con 0x7f095806d8d0 2023-10-25T00:07:45.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 -- 172.21.15.107:0/170791392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f095806d8d0 msgr2=0x7f09580a8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:45.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 --2- 172.21.15.107:0/170791392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f095806d8d0 0x7f09580a8ce0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f095806c400 tx=0x7f095401c9a0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 -- 172.21.15.107:0/170791392 shutdown_connections 2023-10-25T00:07:45.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 --2- 172.21.15.107:0/170791392 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 0x7f09580c2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 --2- 172.21.15.107:0/170791392 >> v2:172.21.15.110:3300/0 conn(0x7f095806aaa0 0x7f09580c25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 --2- 172.21.15.107:0/170791392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f095806d8d0 0x7f09580a8ce0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 -- 172.21.15.107:0/170791392 >> 172.21.15.107:0/170791392 conn(0x7f0958069d40 msgr2=0x7f095806a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:45.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 -- 172.21.15.107:0/170791392 shutdown_connections 2023-10-25T00:07:45.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 -- 172.21.15.107:0/170791392 wait complete. 2023-10-25T00:07:45.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 Processor -- start 2023-10-25T00:07:45.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 -- start start 2023-10-25T00:07:45.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f095806aaa0 0x7f09581517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:45.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f095806d8d0 0x7f095814db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:45.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 0x7f095814e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:45.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095814e580 con 0x7f095806d8d0 2023-10-25T00:07:45.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095808bd00 con 0x7f095806aaa0 2023-10-25T00:07:45.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095fe90640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0958070cf0 con 0x7f0958153510 2023-10-25T00:07:45.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095d42b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f095806d8d0 0x7f095814db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:45.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095d42b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f095806d8d0 0x7f095814db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59216/0 (socket says 172.21.15.107:59216) 2023-10-25T00:07:45.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095d42b640 1 -- 172.21.15.107:0/481562472 learned_addr learned my addr 172.21.15.107:0/481562472 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:45.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095e42d640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 0x7f095814e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:45.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095dc2c640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f095806aaa0 0x7f09581517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:45.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095d42b640 1 -- 172.21.15.107:0/481562472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 msgr2=0x7f095814e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:45.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095d42b640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 0x7f095814e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095d42b640 1 -- 172.21.15.107:0/481562472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f095806aaa0 msgr2=0x7f09581517a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:45.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095d42b640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f095806aaa0 0x7f09581517a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095d42b640 1 -- 172.21.15.107:0/481562472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0954009b30 con 0x7f095806d8d0 2023-10-25T00:07:45.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095e42d640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 0x7f095814e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:07:45.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f095d42b640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f095806d8d0 0x7f095814db00 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f094800d800 tx=0x7f094800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:45.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f094e7fc640 1 -- 172.21.15.107:0/481562472 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09480096a0 con 0x7f095806d8d0 2023-10-25T00:07:45.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f094e7fc640 1 -- 172.21.15.107:0/481562472 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0948009800 con 0x7f095806d8d0 2023-10-25T00:07:45.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.687+0000 7f094e7fc640 1 -- 172.21.15.107:0/481562472 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948009990 con 0x7f095806d8d0 2023-10-25T00:07:45.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.691+0000 7f095fe90640 1 -- 172.21.15.107:0/481562472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0958070fd0 con 0x7f095806d8d0 2023-10-25T00:07:45.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.691+0000 7f095fe90640 1 -- 172.21.15.107:0/481562472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0958071590 con 0x7f095806d8d0 2023-10-25T00:07:45.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.691+0000 7f095fe90640 1 -- 172.21.15.107:0/481562472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09580746e0 con 0x7f095806d8d0 2023-10-25T00:07:45.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.691+0000 7f094e7fc640 1 -- 172.21.15.107:0/481562472 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0948009af0 con 0x7f095806d8d0 2023-10-25T00:07:45.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.691+0000 7f094e7fc640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f093c065e60 0x7f093c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:45.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.691+0000 7f094e7fc640 1 -- 172.21.15.107:0/481562472 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7f0948084810 con 0x7f095806d8d0 2023-10-25T00:07:45.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.691+0000 7f095dc2c640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f093c065e60 0x7f093c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:45.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.691+0000 7f095dc2c640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f093c065e60 0x7f093c068320 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f09540ae000 tx=0x7f09540a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:45.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.695+0000 7f094e7fc640 1 -- 172.21.15.107:0/481562472 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0948051360 con 0x7f095806d8d0 2023-10-25T00:07:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 -- 172.21.15.107:0/481562472 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09580c17e0 con 0x7f095806d8d0 2023-10-25T00:07:45.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f094e7fc640 1 -- 172.21.15.107:0/481562472 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f0948054980 con 0x7f095806d8d0 2023-10-25T00:07:45.951 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:45.951 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:45.951 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 125 2023-10-25T00:07:45.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 -- 172.21.15.107:0/481562472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f093c065e60 msgr2=0x7f093c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:45.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f093c065e60 0x7f093c068320 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f09540ae000 tx=0x7f09540a7070 comp rx=0 tx=0).stop 2023-10-25T00:07:45.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 -- 172.21.15.107:0/481562472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f095806d8d0 msgr2=0x7f095814db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:45.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f095806d8d0 0x7f095814db00 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f094800d800 tx=0x7f094800dcc0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 -- 172.21.15.107:0/481562472 shutdown_connections 2023-10-25T00:07:45.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f093c065e60 0x7f093c068320 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0958153510 0x7f095814e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f095806d8d0 0x7f095814db00 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 --2- 172.21.15.107:0/481562472 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f095806aaa0 0x7f09581517a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:45.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 -- 172.21.15.107:0/481562472 >> 172.21.15.107:0/481562472 conn(0x7f0958069d40 msgr2=0x7f0958073170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:45.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 -- 172.21.15.107:0/481562472 shutdown_connections 2023-10-25T00:07:45.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:45.951+0000 7f095fe90640 1 -- 172.21.15.107:0/481562472 wait complete. 2023-10-25T00:07:45.977 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:07:47.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:48.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.115+0000 7f4340672640 1 Processor -- start 2023-10-25T00:07:48.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.115+0000 7f4340672640 1 -- start start 2023-10-25T00:07:48.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.115+0000 7f4340672640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f433813dc20 0x7f43380c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:48.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.115+0000 7f4340672640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f43380720c0 0x7f433806df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:48.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.115+0000 7f4340672640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4338154860 con 0x7f433806e470 2023-10-25T00:07:48.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.115+0000 7f4340672640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43380c1f10 con 0x7f43380720c0 2023-10-25T00:07:48.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.115+0000 7f4340672640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4338154b20 con 0x7f433813dc20 2023-10-25T00:07:48.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.115+0000 7f433dc0d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f43380720c0 0x7f433806df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.119+0000 7f433dc0d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f43380720c0 0x7f433806df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59230/0 (socket says 172.21.15.107:59230) 2023-10-25T00:07:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.119+0000 7f433dc0d640 1 -- 172.21.15.107:0/2738107231 learned_addr learned my addr 172.21.15.107:0/2738107231 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.119+0000 7f433e40e640 1 --2- 172.21.15.107:0/2738107231 >> v2:172.21.15.110:3300/0 conn(0x7f433813dc20 0x7f43380c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.119+0000 7f433dc0d640 1 -- 172.21.15.107:0/2738107231 >> v2:172.21.15.110:3300/0 conn(0x7f433813dc20 msgr2=0x7f43380c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:48.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.119+0000 7f433dc0d640 1 --2- 172.21.15.107:0/2738107231 >> v2:172.21.15.110:3300/0 conn(0x7f433813dc20 0x7f43380c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.119+0000 7f433dc0d640 1 -- 172.21.15.107:0/2738107231 >> v1:172.21.15.110:6789/0 conn(0x7f433806e470 legacy=0x7f4338157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.119+0000 7f433dc0d640 1 -- 172.21.15.107:0/2738107231 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4338154e00 con 0x7f43380720c0 2023-10-25T00:07:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.119+0000 7f433e40e640 1 --2- 172.21.15.107:0/2738107231 >> v2:172.21.15.110:3300/0 conn(0x7f433813dc20 0x7f43380c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:07:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.119+0000 7f433dc0d640 1 --2- 172.21.15.107:0/2738107231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43380720c0 0x7f433806df30 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f43380c8020 tx=0x7f432801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82728adbd22cc74c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.119+0000 7f433d40c640 1 -- 172.21.15.107:0/2738107231 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43280ab600 con 0x7f43380720c0 2023-10-25T00:07:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f433d40c640 1 -- 172.21.15.107:0/2738107231 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43280a7050 con 0x7f43380720c0 2023-10-25T00:07:48.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f433d40c640 1 -- 172.21.15.107:0/2738107231 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43280ab980 con 0x7f43380720c0 2023-10-25T00:07:48.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 -- 172.21.15.107:0/2738107231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43380720c0 msgr2=0x7f433806df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 --2- 172.21.15.107:0/2738107231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43380720c0 0x7f433806df30 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f43380c8020 tx=0x7f432801c3a0 comp rx=0 tx=0).stop 2023-10-25T00:07:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 -- 172.21.15.107:0/2738107231 shutdown_connections 2023-10-25T00:07:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 --2- 172.21.15.107:0/2738107231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43380720c0 0x7f433806df30 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 --2- 172.21.15.107:0/2738107231 >> v2:172.21.15.110:3300/0 conn(0x7f433813dc20 0x7f43380c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:48.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 -- 172.21.15.107:0/2738107231 >> 172.21.15.107:0/2738107231 conn(0x7f43380c39a0 msgr2=0x7f43380c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 -- 172.21.15.107:0/2738107231 shutdown_connections 2023-10-25T00:07:48.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 -- 172.21.15.107:0/2738107231 wait complete. 2023-10-25T00:07:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 Processor -- start 2023-10-25T00:07:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 -- start start 2023-10-25T00:07:48.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f433806e470 0x7f4338135420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:48.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380720c0 0x7f433813a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:48.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f433813dc20 0x7f43381eca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:48.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380aadf0 con 0x7f433813dc20 2023-10-25T00:07:48.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43380a9450 con 0x7f43380720c0 2023-10-25T00:07:48.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f4340672640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4338135960 con 0x7f433806e470 2023-10-25T00:07:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.123+0000 7f433ec0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f433813dc20 0x7f43381eca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433ec0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f433813dc20 0x7f43381eca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59240/0 (socket says 172.21.15.107:59240) 2023-10-25T00:07:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433ec0f640 1 -- 172.21.15.107:0/2089704805 learned_addr learned my addr 172.21.15.107:0/2089704805 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433dc0d640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380720c0 0x7f433813a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:48.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433e40e640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f433806e470 0x7f4338135420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:48.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433ec0f640 1 -- 172.21.15.107:0/2089704805 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f433806e470 msgr2=0x7f4338135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:48.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433ec0f640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f433806e470 0x7f4338135420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:48.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433ec0f640 1 -- 172.21.15.107:0/2089704805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380720c0 msgr2=0x7f433813a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:48.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433ec0f640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380720c0 0x7f433813a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:48.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433ec0f640 1 -- 172.21.15.107:0/2089704805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43280094d0 con 0x7f433813dc20 2023-10-25T00:07:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433ec0f640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f433813dc20 0x7f43381eca80 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f433000c780 tx=0x7f433000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f432effd640 1 -- 172.21.15.107:0/2089704805 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330004b70 con 0x7f433813dc20 2023-10-25T00:07:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f432effd640 1 -- 172.21.15.107:0/2089704805 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433000ce90 con 0x7f433813dc20 2023-10-25T00:07:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f4340672640 1 -- 172.21.15.107:0/2089704805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43380c6830 con 0x7f433813dc20 2023-10-25T00:07:48.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f432effd640 1 -- 172.21.15.107:0/2089704805 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433000a850 con 0x7f433813dc20 2023-10-25T00:07:48.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f4340672640 1 -- 172.21.15.107:0/2089704805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43380a96a0 con 0x7f433813dc20 2023-10-25T00:07:48.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f4340672640 1 -- 172.21.15.107:0/2089704805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43380746e0 con 0x7f433813dc20 2023-10-25T00:07:48.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f432effd640 1 -- 172.21.15.107:0/2089704805 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f433000a9b0 con 0x7f433813dc20 2023-10-25T00:07:48.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f432effd640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f431c065e60 0x7f431c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:48.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f432effd640 1 -- 172.21.15.107:0/2089704805 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 8939+0+0 (secure 0 0 0) 0x7f4330084500 con 0x7f433813dc20 2023-10-25T00:07:48.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433e40e640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f431c065e60 0x7f431c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:48.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.127+0000 7f433e40e640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f431c065e60 0x7f431c068320 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4334006fd0 tx=0x7f4334008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:48.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.135+0000 7f432effd640 1 -- 172.21.15.107:0/2089704805 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4330051050 con 0x7f433813dc20 2023-10-25T00:07:48.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 -- 172.21.15.107:0/2089704805 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4338158d10 con 0x7f433813dc20 2023-10-25T00:07:48.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f432effd640 1 -- 172.21.15.107:0/2089704805 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f4330054670 con 0x7f433813dc20 2023-10-25T00:07:48.359 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:48.359 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:05:21.824320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5410},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5410":{"gid":5410,"name":"d","rank":0,"incarnation":110,"state":"up:active","state_seq":98,"addr":"172.21.15.110:6835/2150827048","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2150827048},{"type":"v1","addr":"172.21.15.110:6835","nonce":2150827048}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:48.359 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 125 2023-10-25T00:07:48.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 -- 172.21.15.107:0/2089704805 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f431c065e60 msgr2=0x7f431c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:48.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f431c065e60 0x7f431c068320 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4334006fd0 tx=0x7f4334008040 comp rx=0 tx=0).stop 2023-10-25T00:07:48.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 -- 172.21.15.107:0/2089704805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f433813dc20 msgr2=0x7f43381eca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:48.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f433813dc20 0x7f43381eca80 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f433000c780 tx=0x7f433000cc40 comp rx=0 tx=0).stop 2023-10-25T00:07:48.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 -- 172.21.15.107:0/2089704805 shutdown_connections 2023-10-25T00:07:48.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f433813dc20 0x7f43381eca80 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:48.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43380720c0 0x7f433813a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:48.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f431c065e60 0x7f431c068320 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:48.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 --2- 172.21.15.107:0/2089704805 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f433806e470 0x7f4338135420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:48.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 -- 172.21.15.107:0/2089704805 >> 172.21.15.107:0/2089704805 conn(0x7f43380c39a0 msgr2=0x7f433806a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:48.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 -- 172.21.15.107:0/2089704805 shutdown_connections 2023-10-25T00:07:48.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:48.359+0000 7f4340672640 1 -- 172.21.15.107:0/2089704805 wait complete. 2023-10-25T00:07:48.379 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:07:49.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:50.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.067+0000 7f2718c3b640 1 Processor -- start 2023-10-25T00:07:50.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.067+0000 7f2718c3b640 1 -- start start 2023-10-25T00:07:50.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f2718c3b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27140c44e0 0x7f271415d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f2718c3b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27140c1930 con 0x7f271414ed90 2023-10-25T00:07:50.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f2718c3b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2714152380 con 0x7f2714072780 2023-10-25T00:07:50.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f2718c3b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27140c16b0 con 0x7f27140c44e0 2023-10-25T00:07:50.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f271414ed90 0x7f271406cca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33674/0 (socket says 172.21.15.107:33674) 2023-10-25T00:07:50.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271259c640 1 -- 172.21.15.107:0/2506380321 learned_addr learned my addr 172.21.15.107:0/2506380321 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:50.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f2712d9d640 1 --2- 172.21.15.107:0/2506380321 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27140c44e0 0x7f271415d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814500963 0 0) 0x7f27140c1930 con 0x7f271414ed90 2023-10-25T00:07:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26fc003530 con 0x7f271414ed90 2023-10-25T00:07:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3609424611 0 0) 0x7f26fc003530 con 0x7f271414ed90 2023-10-25T00:07:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27140c44e0 msgr2=0x7f271415d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 --2- 172.21.15.107:0/2506380321 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27140c44e0 0x7f271415d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 >> v1:172.21.15.110:6789/0 conn(0x7f2714072780 legacy=0x7f271406bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27140c1bd0 con 0x7f271414ed90 2023-10-25T00:07:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f2704001680 con 0x7f271414ed90 2023-10-25T00:07:50.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 >> v1:172.21.15.107:6789/0 conn(0x7f271414ed90 legacy=0x7f271406cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 --2- 172.21.15.107:0/2506380321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26fc0039b0 0x7f26fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 --2- 172.21.15.107:0/2506380321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26fc0047d0 0x7f26fc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 --2- 172.21.15.107:0/2506380321 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27140c44e0 0x7f26fc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27140c16b0 con 0x7f26fc0047d0 2023-10-25T00:07:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc00ba70 con 0x7f26fc0039b0 2023-10-25T00:07:50.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc00de40 con 0x7f27140c44e0 2023-10-25T00:07:50.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f2711d9b640 1 --2- 172.21.15.107:0/2506380321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26fc0047d0 0x7f26fc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f271259c640 1 --2- 172.21.15.107:0/2506380321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26fc0039b0 0x7f26fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f2711d9b640 1 -- 172.21.15.107:0/2506380321 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27140c44e0 msgr2=0x7f26fc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:50.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f2711d9b640 1 --2- 172.21.15.107:0/2506380321 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27140c44e0 0x7f26fc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f2711d9b640 1 -- 172.21.15.107:0/2506380321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26fc0039b0 msgr2=0x7f26fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:50.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f2711d9b640 1 --2- 172.21.15.107:0/2506380321 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f26fc0039b0 0x7f26fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.071+0000 7f2711d9b640 1 -- 172.21.15.107:0/2506380321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc003530 con 0x7f26fc0047d0 2023-10-25T00:07:50.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f2711d9b640 1 --2- 172.21.15.107:0/2506380321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26fc0047d0 0x7f26fc004c50 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f2714072bc0 tx=0x7f270801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a8a784667beff server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27080052d0 con 0x7f26fc0047d0 2023-10-25T00:07:50.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27080a7050 con 0x7f26fc0047d0 2023-10-25T00:07:50.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f271159a640 1 -- 172.21.15.107:0/2506380321 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2708003600 con 0x7f26fc0047d0 2023-10-25T00:07:50.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f2718c3b640 1 -- 172.21.15.107:0/2506380321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26fc0047d0 msgr2=0x7f26fc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f2718c3b640 1 --2- 172.21.15.107:0/2506380321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26fc0047d0 0x7f26fc004c50 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f2714072bc0 tx=0x7f270801c380 comp rx=0 tx=0).stop 2023-10-25T00:07:50.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f2712d9d640 1 -- 172.21.15.107:0/2506380321 reap_dead start 2023-10-25T00:07:50.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f2718c3b640 1 -- 172.21.15.107:0/2506380321 shutdown_connections 2023-10-25T00:07:50.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f2718c3b640 1 -- 172.21.15.107:0/2506380321 >> 172.21.15.107:0/2506380321 conn(0x7f27140a8cb0 msgr2=0x7f27140c65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:50.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f2718c3b640 1 -- 172.21.15.107:0/2506380321 shutdown_connections 2023-10-25T00:07:50.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f2718c3b640 1 -- 172.21.15.107:0/2506380321 wait complete. 2023-10-25T00:07:50.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f2718c3b640 1 Processor -- start 2023-10-25T00:07:50.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.075+0000 7f2718c3b640 1 -- start start 2023-10-25T00:07:50.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2718c3b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26fc0047d0 0x7f271414bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2718c3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2714149950 0x7f2714149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2718c3b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27141e7660 0x7f27141e7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2718c3b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27141e8b70 con 0x7f2714149950 2023-10-25T00:07:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2718c3b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2714072000 con 0x7f27141e7660 2023-10-25T00:07:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2718c3b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27141e71d0 con 0x7f26fc0047d0 2023-10-25T00:07:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2711d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2714149950 0x7f2714149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2711d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2714149950 0x7f2714149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59264/0 (socket says 172.21.15.107:59264) 2023-10-25T00:07:50.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2711d9b640 1 -- 172.21.15.107:0/3705869235 learned_addr learned my addr 172.21.15.107:0/3705869235 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:50.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f271259c640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26fc0047d0 0x7f271414bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2712d9d640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27141e7660 0x7f27141e7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2711d9b640 1 -- 172.21.15.107:0/3705869235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26fc0047d0 msgr2=0x7f271414bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2711d9b640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26fc0047d0 0x7f271414bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2711d9b640 1 -- 172.21.15.107:0/3705869235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27141e7660 msgr2=0x7f27141e7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2711d9b640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27141e7660 0x7f27141e7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2711d9b640 1 -- 172.21.15.107:0/3705869235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27141e68a0 con 0x7f2714149950 2023-10-25T00:07:50.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2712d9d640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27141e7660 0x7f27141e7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:07:50.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2711d9b640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2714149950 0x7f2714149db0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f2708005040 tx=0x7f26fc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2702ffd640 1 -- 172.21.15.107:0/3705869235 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2708007a40 con 0x7f2714149950 2023-10-25T00:07:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2702ffd640 1 -- 172.21.15.107:0/3705869235 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2708007680 con 0x7f2714149950 2023-10-25T00:07:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2718c3b640 1 -- 172.21.15.107:0/3705869235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27141e6b80 con 0x7f2714149950 2023-10-25T00:07:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2702ffd640 1 -- 172.21.15.107:0/3705869235 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2708006a00 con 0x7f2714149950 2023-10-25T00:07:50.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2718c3b640 1 -- 172.21.15.107:0/3705869235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271414c2f0 con 0x7f2714149950 2023-10-25T00:07:50.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2718c3b640 1 -- 172.21.15.107:0/3705869235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27141e7360 con 0x7f2714149950 2023-10-25T00:07:50.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2702ffd640 1 -- 172.21.15.107:0/3705869235 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2708006b60 con 0x7f2714149950 2023-10-25T00:07:50.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2702ffd640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f26f4065e90 0x7f26f4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f2702ffd640 1 -- 172.21.15.107:0/3705869235 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7f27080a4e40 con 0x7f2714149950 2023-10-25T00:07:50.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.079+0000 7f271259c640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f26f4065e90 0x7f26f4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.083+0000 7f271259c640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f26f4065e90 0x7f26f4068350 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2704001060 tx=0x7f2704009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.087+0000 7f2702ffd640 1 -- 172.21.15.107:0/3705869235 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f27080eb780 con 0x7f2714149950 2023-10-25T00:07:50.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 -- 172.21.15.107:0/3705869235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27140c49c0 con 0x7f2714149950 2023-10-25T00:07:50.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2702ffd640 1 -- 172.21.15.107:0/3705869235 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+5020 (secure 0 0 0) 0x7f27080eeda0 con 0x7f2714149950 2023-10-25T00:07:50.283 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:50.283 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:49.711549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:50.284 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 127 2023-10-25T00:07:50.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 -- 172.21.15.107:0/3705869235 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f26f4065e90 msgr2=0x7f26f4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f26f4065e90 0x7f26f4068350 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2704001060 tx=0x7f2704009040 comp rx=0 tx=0).stop 2023-10-25T00:07:50.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 -- 172.21.15.107:0/3705869235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2714149950 msgr2=0x7f2714149db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2714149950 0x7f2714149db0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f2708005040 tx=0x7f26fc005800 comp rx=0 tx=0).stop 2023-10-25T00:07:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 -- 172.21.15.107:0/3705869235 shutdown_connections 2023-10-25T00:07:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27141e7660 0x7f27141e7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2714149950 0x7f2714149db0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f26f4065e90 0x7f26f4068350 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 --2- 172.21.15.107:0/3705869235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f26fc0047d0 0x7f271414bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 -- 172.21.15.107:0/3705869235 >> 172.21.15.107:0/3705869235 conn(0x7f27140a8cb0 msgr2=0x7f271414a710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 -- 172.21.15.107:0/3705869235 shutdown_connections 2023-10-25T00:07:50.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.283+0000 7f2718c3b640 1 -- 172.21.15.107:0/3705869235 wait complete. 2023-10-25T00:07:50.305 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2023-10-25T00:07:50.305 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:07:50.305 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:07:50.316 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:07:50.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:50.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:50.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1faf997640 1 Processor -- start 2023-10-25T00:07:50.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1faf997640 1 -- start start 2023-10-25T00:07:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1faf997640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa806f490 0x7f1fa814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1faf997640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fa8152b30 0x7f1fa8152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1faf997640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa8155cd0 con 0x7f1fa814c470 2023-10-25T00:07:50.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1faf997640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa81526e0 con 0x7f1fa8152b30 2023-10-25T00:07:50.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1faf997640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa80c4380 con 0x7f1fa806f490 2023-10-25T00:07:50.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1fadf34640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fa8152b30 0x7f1fa8152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1fadf34640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fa8152b30 0x7f1fa8152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59280/0 (socket says 172.21.15.107:59280) 2023-10-25T00:07:50.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1fadf34640 1 -- 172.21.15.107:0/4174869008 learned_addr learned my addr 172.21.15.107:0/4174869008 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:50.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1facf32640 1 --2- 172.21.15.107:0/4174869008 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa806f490 0x7f1fa814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1fadf34640 1 -- 172.21.15.107:0/4174869008 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa806f490 msgr2=0x7f1fa814c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:50.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1fadf34640 1 --2- 172.21.15.107:0/4174869008 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa806f490 0x7f1fa814c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1fadf34640 1 -- 172.21.15.107:0/4174869008 >> v1:172.21.15.110:6789/0 conn(0x7f1fa814c470 legacy=0x7f1fa8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.483+0000 7f1fadf34640 1 -- 172.21.15.107:0/4174869008 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa80c48b0 con 0x7f1fa8152b30 2023-10-25T00:07:50.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1fadf34640 1 --2- 172.21.15.107:0/4174869008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fa8152b30 0x7f1fa8152fa0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009e50 tx=0x7f1fa001c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f8cffe365b41656 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1f9ffff640 1 -- 172.21.15.107:0/4174869008 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa00a56a0 con 0x7f1fa8152b30 2023-10-25T00:07:50.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1f9ffff640 1 -- 172.21.15.107:0/4174869008 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa00a5800 con 0x7f1fa8152b30 2023-10-25T00:07:50.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1f9ffff640 1 -- 172.21.15.107:0/4174869008 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa00a5b90 con 0x7f1fa8152b30 2023-10-25T00:07:50.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1faf997640 1 -- 172.21.15.107:0/4174869008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fa8152b30 msgr2=0x7f1fa8152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1faf997640 1 --2- 172.21.15.107:0/4174869008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fa8152b30 0x7f1fa8152fa0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0009e50 tx=0x7f1fa001c8f0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1faf997640 1 -- 172.21.15.107:0/4174869008 shutdown_connections 2023-10-25T00:07:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1faf997640 1 --2- 172.21.15.107:0/4174869008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fa8152b30 0x7f1fa8152fa0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1faf997640 1 --2- 172.21.15.107:0/4174869008 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa806f490 0x7f1fa814c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1faf997640 1 -- 172.21.15.107:0/4174869008 >> 172.21.15.107:0/4174869008 conn(0x7f1fa8070f60 msgr2=0x7f1fa8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1faf997640 1 -- 172.21.15.107:0/4174869008 shutdown_connections 2023-10-25T00:07:50.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.487+0000 7f1faf997640 1 -- 172.21.15.107:0/4174869008 wait complete. 2023-10-25T00:07:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1faf997640 1 Processor -- start 2023-10-25T00:07:50.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1faf997640 1 -- start start 2023-10-25T00:07:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1faf997640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fa806f490 0x7f1fa81448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1faf997640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa814c470 0x7f1fa81427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1faf997640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fa8152b30 0x7f1fa8142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1faf997640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa8143230 con 0x7f1fa806f490 2023-10-25T00:07:50.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1faf997640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa80aabf0 con 0x7f1fa8152b30 2023-10-25T00:07:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1faf997640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa80aaed0 con 0x7f1fa814c470 2023-10-25T00:07:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1fad733640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fa806f490 0x7f1fa81448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1fad733640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fa806f490 0x7f1fa81448a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59292/0 (socket says 172.21.15.107:59292) 2023-10-25T00:07:50.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1fad733640 1 -- 172.21.15.107:0/3484639031 learned_addr learned my addr 172.21.15.107:0/3484639031 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:50.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1facf32640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa814c470 0x7f1fa81427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1fadf34640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fa8152b30 0x7f1fa8142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1fad733640 1 -- 172.21.15.107:0/3484639031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa814c470 msgr2=0x7f1fa81427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1fad733640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa814c470 0x7f1fa81427b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1fad733640 1 -- 172.21.15.107:0/3484639031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fa8152b30 msgr2=0x7f1fa8142cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1fad733640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fa8152b30 0x7f1fa8142cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1fad733640 1 -- 172.21.15.107:0/3484639031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa0009b30 con 0x7f1fa806f490 2023-10-25T00:07:50.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1fad733640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fa806f490 0x7f1fa81448a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c500 tx=0x7f1fa400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1f9dffb640 1 -- 172.21.15.107:0/3484639031 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4005220 con 0x7f1fa806f490 2023-10-25T00:07:50.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1f9dffb640 1 -- 172.21.15.107:0/3484639031 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa4005380 con 0x7f1fa806f490 2023-10-25T00:07:50.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1faf997640 1 -- 172.21.15.107:0/3484639031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa80ab1b0 con 0x7f1fa806f490 2023-10-25T00:07:50.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1f9dffb640 1 -- 172.21.15.107:0/3484639031 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4007270 con 0x7f1fa806f490 2023-10-25T00:07:50.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1faf997640 1 -- 172.21.15.107:0/3484639031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa80ab770 con 0x7f1fa806f490 2023-10-25T00:07:50.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.491+0000 7f1faf997640 1 -- 172.21.15.107:0/3484639031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa80ab990 con 0x7f1fa806f490 2023-10-25T00:07:50.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.495+0000 7f1f9dffb640 1 -- 172.21.15.107:0/3484639031 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1fa40073d0 con 0x7f1fa806f490 2023-10-25T00:07:50.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.495+0000 7f1f9dffb640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f8c065e90 0x7f1f8c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.495+0000 7f1f9dffb640 1 -- 172.21.15.107:0/3484639031 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7f1fa4083ec0 con 0x7f1fa806f490 2023-10-25T00:07:50.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.495+0000 7f1facf32640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f8c065e90 0x7f1f8c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.495+0000 7f1facf32640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f8c065e90 0x7f1f8c068350 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8154ec0 tx=0x7f1f98009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.499+0000 7f1f9dffb640 1 -- 172.21.15.107:0/3484639031 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1fa4050a60 con 0x7f1fa806f490 2023-10-25T00:07:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501fd9c640 1 Processor -- start 2023-10-25T00:07:50.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501fd9c640 1 -- start start 2023-10-25T00:07:50.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501fd9c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f501814c850 0x7f5018153240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501fd9c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5018155cd0 con 0x7f501814c470 2023-10-25T00:07:50.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501fd9c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50180c4380 con 0x7f501806f490 2023-10-25T00:07:50.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501fd9c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50181526e0 con 0x7f501814c850 2023-10-25T00:07:50.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501db38640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f501814c470 0x7f5018071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33682/0 (socket says 172.21.15.107:33682) 2023-10-25T00:07:50.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501db38640 1 -- 172.21.15.107:0/1058307869 learned_addr learned my addr 172.21.15.107:0/1058307869 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:50.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501e339640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f501814c850 0x7f5018153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696920100 0 0) 0x7f5018155cd0 con 0x7f501814c470 2023-10-25T00:07:50.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5004003530 con 0x7f501814c470 2023-10-25T00:07:50.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 57702560 0 0) 0x7f5004003530 con 0x7f501814c470 2023-10-25T00:07:50.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f501814c850 msgr2=0x7f5018153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:50.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f501814c850 0x7f5018153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 >> v1:172.21.15.110:6789/0 conn(0x7f501806f490 legacy=0x7f5018152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50180c48b0 con 0x7f501814c470 2023-10-25T00:07:50.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5014001680 con 0x7f501814c470 2023-10-25T00:07:50.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 >> v1:172.21.15.107:6789/0 conn(0x7f501814c470 legacy=0x7f5018071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50040039b0 0x7f5004003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50040047d0 0x7f5004004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f501814c850 0x7f5004005260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50181526e0 con 0x7f50040047d0 2023-10-25T00:07:50.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500400bb50 con 0x7f50040039b0 2023-10-25T00:07:50.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500400bd10 con 0x7f501814c850 2023-10-25T00:07:50.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501d337640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50040047d0 0x7f5004004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501e339640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f501814c850 0x7f5004005260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501db38640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50040039b0 0x7f5004003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501d337640 1 -- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f501814c850 msgr2=0x7f5004005260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:50.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501d337640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f501814c850 0x7f5004005260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501d337640 1 -- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50040039b0 msgr2=0x7f5004003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:50.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501d337640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50040039b0 0x7f5004003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501d337640 1 -- 172.21.15.107:0/1058307869 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5004003530 con 0x7f50040047d0 2023-10-25T00:07:50.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.519+0000 7f501db38640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50040039b0 0x7f5004003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:07:50.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.523+0000 7f501d337640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50040047d0 0x7f5004004c50 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f5008001780 tx=0x7f5008020810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=154759dfa99db095 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.523+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5008005080 con 0x7f50040047d0 2023-10-25T00:07:50.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.523+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5008005200 con 0x7f50040047d0 2023-10-25T00:07:50.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.523+0000 7f501cb36640 1 -- 172.21.15.107:0/1058307869 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50080053b0 con 0x7f50040047d0 2023-10-25T00:07:50.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.523+0000 7f501fd9c640 1 -- 172.21.15.107:0/1058307869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50040047d0 msgr2=0x7f5004004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.523+0000 7f501fd9c640 1 --2- 172.21.15.107:0/1058307869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50040047d0 0x7f5004004c50 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f5008001780 tx=0x7f5008020810 comp rx=0 tx=0).stop 2023-10-25T00:07:50.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.523+0000 7f501e339640 1 -- 172.21.15.107:0/1058307869 reap_dead start 2023-10-25T00:07:50.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.523+0000 7f501fd9c640 1 -- 172.21.15.107:0/1058307869 shutdown_connections 2023-10-25T00:07:50.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.523+0000 7f501fd9c640 1 -- 172.21.15.107:0/1058307869 >> 172.21.15.107:0/1058307869 conn(0x7f5018070f60 msgr2=0x7f5018071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:50.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.523+0000 7f501fd9c640 1 -- 172.21.15.107:0/1058307869 shutdown_connections 2023-10-25T00:07:50.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.523+0000 7f501fd9c640 1 -- 172.21.15.107:0/1058307869 wait complete. 2023-10-25T00:07:50.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501fd9c640 1 Processor -- start 2023-10-25T00:07:50.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501fd9c640 1 -- start start 2023-10-25T00:07:50.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501fd9c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50040047d0 0x7f50181457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501fd9c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50180c3b10 0x7f501814ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501fd9c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f501814efd0 0x7f50181eba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501fd9c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5018146cb0 con 0x7f501814efd0 2023-10-25T00:07:50.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501fd9c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501814e600 con 0x7f50040047d0 2023-10-25T00:07:50.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501fd9c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5018145ce0 con 0x7f50180c3b10 2023-10-25T00:07:50.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501e339640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f501814efd0 0x7f50181eba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501e339640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f501814efd0 0x7f50181eba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59318/0 (socket says 172.21.15.107:59318) 2023-10-25T00:07:50.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501e339640 1 -- 172.21.15.107:0/2662769259 learned_addr learned my addr 172.21.15.107:0/2662769259 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:50.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501db38640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50040047d0 0x7f50181457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501d337640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50180c3b10 0x7f501814ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501e339640 1 -- 172.21.15.107:0/2662769259 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50180c3b10 msgr2=0x7f501814ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501e339640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50180c3b10 0x7f501814ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501e339640 1 -- 172.21.15.107:0/2662769259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50040047d0 msgr2=0x7f50181457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501e339640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50040047d0 0x7f50181457a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501e339640 1 -- 172.21.15.107:0/2662769259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5018146410 con 0x7f501814efd0 2023-10-25T00:07:50.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501e339640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f501814efd0 0x7f50181eba10 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f5018073c60 tx=0x7f50100115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f500e7fc640 1 -- 172.21.15.107:0/2662769259 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501000e070 con 0x7f501814efd0 2023-10-25T00:07:50.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f500e7fc640 1 -- 172.21.15.107:0/2662769259 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5010011e40 con 0x7f501814efd0 2023-10-25T00:07:50.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501fd9c640 1 -- 172.21.15.107:0/2662769259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50181466f0 con 0x7f501814efd0 2023-10-25T00:07:50.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f500e7fc640 1 -- 172.21.15.107:0/2662769259 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50100039a0 con 0x7f501814efd0 2023-10-25T00:07:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501fd9c640 1 -- 172.21.15.107:0/2662769259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501814ce10 con 0x7f501814efd0 2023-10-25T00:07:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f501fd9c640 1 -- 172.21.15.107:0/2662769259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50180746e0 con 0x7f501814efd0 2023-10-25T00:07:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.527+0000 7f500e7fc640 1 -- 172.21.15.107:0/2662769259 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5010003b00 con 0x7f501814efd0 2023-10-25T00:07:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.531+0000 7f500e7fc640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ffc065e60 0x7f4ffc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.531+0000 7f500e7fc640 1 -- 172.21.15.107:0/2662769259 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7f5010084a00 con 0x7f501814efd0 2023-10-25T00:07:50.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.531+0000 7f501db38640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ffc065e60 0x7f4ffc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.531+0000 7f501db38640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ffc065e60 0x7f4ffc068320 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f50140018f0 tx=0x7f5014009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.535+0000 7f500e7fc640 1 -- 172.21.15.107:0/2662769259 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f50100514f0 con 0x7f501814efd0 2023-10-25T00:07:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.715+0000 7f1faf997640 1 -- 172.21.15.107:0/3484639031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fa80746e0 con 0x7f1fa806f490 2023-10-25T00:07:50.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.715+0000 7f1f9dffb640 1 -- 172.21.15.107:0/3484639031 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+5020 (secure 0 0 0) 0x7f1fa4021020 con 0x7f1fa806f490 2023-10-25T00:07:50.717 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:50.717 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:49.711549+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 127 2023-10-25T00:07:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 -- 172.21.15.107:0/3484639031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f8c065e90 msgr2=0x7f1f8c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f8c065e90 0x7f1f8c068350 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8154ec0 tx=0x7f1f98009190 comp rx=0 tx=0).stop 2023-10-25T00:07:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 -- 172.21.15.107:0/3484639031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fa806f490 msgr2=0x7f1fa81448a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fa806f490 0x7f1fa81448a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c500 tx=0x7f1fa400c9c0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 -- 172.21.15.107:0/3484639031 shutdown_connections 2023-10-25T00:07:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fa8152b30 0x7f1fa8142cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f8c065e90 0x7f1f8c068350 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fa814c470 0x7f1fa81427b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 --2- 172.21.15.107:0/3484639031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fa806f490 0x7f1fa81448a0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 -- 172.21.15.107:0/3484639031 >> 172.21.15.107:0/3484639031 conn(0x7f1fa8070f60 msgr2=0x7f1fa8150730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 -- 172.21.15.107:0/3484639031 shutdown_connections 2023-10-25T00:07:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.719+0000 7f1faf997640 1 -- 172.21.15.107:0/3484639031 wait complete. 2023-10-25T00:07:50.740 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:07:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.767+0000 7f501fd9c640 1 -- 172.21.15.107:0/2662769259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5018073170 con 0x7f501814efd0 2023-10-25T00:07:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.767+0000 7f500e7fc640 1 -- 172.21.15.107:0/2662769259 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5017 (secure 0 0 0) 0x7f5010054b10 con 0x7f501814efd0 2023-10-25T00:07:50.769 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:50.769 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:50.722370+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:50.769 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 128 2023-10-25T00:07:50.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 -- 172.21.15.107:0/2662769259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ffc065e60 msgr2=0x7f4ffc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ffc065e60 0x7f4ffc068320 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f50140018f0 tx=0x7f5014009040 comp rx=0 tx=0).stop 2023-10-25T00:07:50.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 -- 172.21.15.107:0/2662769259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f501814efd0 msgr2=0x7f50181eba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f501814efd0 0x7f50181eba10 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f5018073c60 tx=0x7f50100115a0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 -- 172.21.15.107:0/2662769259 shutdown_connections 2023-10-25T00:07:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f501814efd0 0x7f50181eba10 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ffc065e60 0x7f4ffc068320 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50180c3b10 0x7f501814ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 --2- 172.21.15.107:0/2662769259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50040047d0 0x7f50181457a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 -- 172.21.15.107:0/2662769259 >> 172.21.15.107:0/2662769259 conn(0x7f5018070f60 msgr2=0x7f50180c62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:50.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 -- 172.21.15.107:0/2662769259 shutdown_connections 2023-10-25T00:07:50.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.771+0000 7f501fd9c640 1 -- 172.21.15.107:0/2662769259 wait complete. 2023-10-25T00:07:50.787 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2023-10-25T00:07:50.787 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:07:50.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:50.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.951+0000 7fc7e41ab640 1 Processor -- start 2023-10-25T00:07:50.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.951+0000 7fc7e41ab640 1 -- start start 2023-10-25T00:07:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.951+0000 7fc7e41ab640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc7dc0720c0 0x7fc7dc0c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.951+0000 7fc7e41ab640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc154860 con 0x7fc7dc157e50 2023-10-25T00:07:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.951+0000 7fc7e41ab640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc0c1f10 con 0x7fc7dc13dc20 2023-10-25T00:07:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.951+0000 7fc7e41ab640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7dc154b20 con 0x7fc7dc0720c0 2023-10-25T00:07:50.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.951+0000 7fc7e1746640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc7dc0720c0 0x7fc7dc0c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.951+0000 7fc7e2748640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc7dc157e50 0x7fc7dc15f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33696/0 (socket says 172.21.15.107:33696) 2023-10-25T00:07:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.951+0000 7fc7e2748640 1 -- 172.21.15.107:0/2952879239 learned_addr learned my addr 172.21.15.107:0/2952879239 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e1746640 1 -- 172.21.15.107:0/2952879239 >> v1:172.21.15.110:6789/0 conn(0x7fc7dc13dc20 legacy=0x7fc7dc06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e1746640 1 -- 172.21.15.107:0/2952879239 >> v1:172.21.15.107:6789/0 conn(0x7fc7dc157e50 legacy=0x7fc7dc15f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e0f45640 1 -- 172.21.15.107:0/2952879239 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569514609 0 0) 0x7fc7dc154860 con 0x7fc7dc157e50 2023-10-25T00:07:50.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e1746640 1 -- 172.21.15.107:0/2952879239 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc154e00 con 0x7fc7dc0720c0 2023-10-25T00:07:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e1746640 1 --2- 172.21.15.107:0/2952879239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7dc0720c0 0x7fc7dc0c18a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097c0 tx=0x7fc7cc01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28353c9edd694ddb server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e0f45640 1 -- 172.21.15.107:0/2952879239 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0ab600 con 0x7fc7dc0720c0 2023-10-25T00:07:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e0f45640 1 -- 172.21.15.107:0/2952879239 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc0a7050 con 0x7fc7dc0720c0 2023-10-25T00:07:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e0f45640 1 -- 172.21.15.107:0/2952879239 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0ab980 con 0x7fc7dc0720c0 2023-10-25T00:07:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2952879239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7dc0720c0 msgr2=0x7fc7dc0c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e41ab640 1 --2- 172.21.15.107:0/2952879239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7dc0720c0 0x7fc7dc0c18a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097c0 tx=0x7fc7cc01c320 comp rx=0 tx=0).stop 2023-10-25T00:07:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2952879239 shutdown_connections 2023-10-25T00:07:50.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e41ab640 1 --2- 172.21.15.107:0/2952879239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7dc0720c0 0x7fc7dc0c18a0 secure :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097c0 tx=0x7fc7cc01c320 comp rx=0 tx=0).stop 2023-10-25T00:07:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2952879239 >> 172.21.15.107:0/2952879239 conn(0x7fc7dc0c39a0 msgr2=0x7fc7dc0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2952879239 shutdown_connections 2023-10-25T00:07:50.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.955+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2952879239 wait complete. 2023-10-25T00:07:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e41ab640 1 Processor -- start 2023-10-25T00:07:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e41ab640 1 -- start start 2023-10-25T00:07:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e41ab640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc7dc13dc20 0x7fc7dc137750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e41ab640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7dc157e50 0x7fc7dc137c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e41ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7dc1ee250 0x7fc7dc0ab490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e41ab640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc1ee750 con 0x7fc7dc1ee250 2023-10-25T00:07:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e41ab640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc0a9580 con 0x7fc7dc157e50 2023-10-25T00:07:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e41ab640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc0a9860 con 0x7fc7dc13dc20 2023-10-25T00:07:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e2748640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7dc1ee250 0x7fc7dc0ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e2748640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7dc1ee250 0x7fc7dc0ab490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59340/0 (socket says 172.21.15.107:59340) 2023-10-25T00:07:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e2748640 1 -- 172.21.15.107:0/2596619153 learned_addr learned my addr 172.21.15.107:0/2596619153 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e1f47640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc7dc13dc20 0x7fc7dc137750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e2748640 1 -- 172.21.15.107:0/2596619153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc7dc13dc20 msgr2=0x7fc7dc137750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e2748640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc7dc13dc20 0x7fc7dc137750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e1746640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7dc157e50 0x7fc7dc137c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e2748640 1 -- 172.21.15.107:0/2596619153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7dc157e50 msgr2=0x7fc7dc137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e2748640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7dc157e50 0x7fc7dc137c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e2748640 1 -- 172.21.15.107:0/2596619153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc009420 con 0x7fc7dc1ee250 2023-10-25T00:07:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e1f47640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc7dc13dc20 0x7fc7dc137750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:07:50.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e2748640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7dc1ee250 0x7fc7dc0ab490 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4001570 tx=0x7fc7d400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7d27fc640 1 -- 172.21.15.107:0/2596619153 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d400a640 con 0x7fc7dc1ee250 2023-10-25T00:07:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7d27fc640 1 -- 172.21.15.107:0/2596619153 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d400a7c0 con 0x7fc7dc1ee250 2023-10-25T00:07:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7d27fc640 1 -- 172.21.15.107:0/2596619153 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d400ab70 con 0x7fc7dc1ee250 2023-10-25T00:07:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2596619153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7dc0a9b40 con 0x7fc7dc1ee250 2023-10-25T00:07:50.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2596619153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7dc1521f0 con 0x7fc7dc1ee250 2023-10-25T00:07:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.959+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2596619153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7dc0746e0 con 0x7fc7dc1ee250 2023-10-25T00:07:50.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.963+0000 7fc7d27fc640 1 -- 172.21.15.107:0/2596619153 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc7d400acd0 con 0x7fc7dc1ee250 2023-10-25T00:07:50.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.963+0000 7fc7d27fc640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc7c0065e60 0x7fc7c0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:50.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.963+0000 7fc7d27fc640 1 -- 172.21.15.107:0/2596619153 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7fc7d4083980 con 0x7fc7dc1ee250 2023-10-25T00:07:50.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.963+0000 7fc7e1f47640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc7c0065e60 0x7fc7c0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:50.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.963+0000 7fc7e1f47640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc7c0065e60 0x7fc7c0068320 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc157df0 tx=0x7fc7d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:50.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:50.967+0000 7fc7d27fc640 1 -- 172.21.15.107:0/2596619153 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc7d4050470 con 0x7fc7dc1ee250 2023-10-25T00:07:51.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2596619153 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7dc155b60 con 0x7fc7dc1ee250 2023-10-25T00:07:51.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7d27fc640 1 -- 172.21.15.107:0/2596619153 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+5017 (secure 0 0 0) 0x7fc7d4018070 con 0x7fc7dc1ee250 2023-10-25T00:07:51.235 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:51.236 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:50.722370+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:51.236 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 128 2023-10-25T00:07:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2596619153 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc7c0065e60 msgr2=0x7fc7c0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc7c0065e60 0x7fc7c0068320 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc157df0 tx=0x7fc7d8009040 comp rx=0 tx=0).stop 2023-10-25T00:07:51.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2596619153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7dc1ee250 msgr2=0x7fc7dc0ab490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7dc1ee250 0x7fc7dc0ab490 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4001570 tx=0x7fc7d400cca0 comp rx=0 tx=0).stop 2023-10-25T00:07:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2596619153 shutdown_connections 2023-10-25T00:07:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc7dc1ee250 0x7fc7dc0ab490 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc7dc157e50 0x7fc7dc137c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:51.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc7c0065e60 0x7fc7c0068320 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:51.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 --2- 172.21.15.107:0/2596619153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc7dc13dc20 0x7fc7dc137750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:51.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2596619153 >> 172.21.15.107:0/2596619153 conn(0x7fc7dc0c39a0 msgr2=0x7fc7dc06fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:51.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2596619153 shutdown_connections 2023-10-25T00:07:51.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:51.235+0000 7fc7e41ab640 1 -- 172.21.15.107:0/2596619153 wait complete. 2023-10-25T00:07:51.253 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 128, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 124, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7107": { "addr": "172.21.15.107:6835/913067382", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 913067382, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 913067382, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7107, "incarnation": 121, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:06:26.749209+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7107 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 128, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7305": { "addr": "172.21.15.107:6837/1317796403", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 1317796403, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 1317796403, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7305, "incarnation": 126, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:07:50.722370+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7305 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:07:52.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:52.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.871+0000 7fa2e4a20640 1 Processor -- start 2023-10-25T00:07:52.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2e4a20640 1 -- start start 2023-10-25T00:07:52.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2e4a20640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e0152040 0x7fa2e00c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:52.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2e4a20640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa2e006a4b0 0x7fa2e00ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:52.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2e4a20640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e0155680 con 0x7fa2e00c8100 2023-10-25T00:07:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2e4a20640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e0154a70 con 0x7fa2e006a4b0 2023-10-25T00:07:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2e4a20640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e0154d50 con 0x7fa2e0152040 2023-10-25T00:07:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2ddd9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa2e00c8100 0x7fa2e00abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50374/0 (socket says 172.21.15.107:50374) 2023-10-25T00:07:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2ddd9b640 1 -- 172.21.15.107:0/1134358703 learned_addr learned my addr 172.21.15.107:0/1134358703 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2de59c640 1 --2- 172.21.15.107:0/1134358703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e0152040 0x7fa2e00c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2ded9d640 1 --2- 172.21.15.107:0/1134358703 >> v2:172.21.15.110:3300/0 conn(0x7fa2e006a4b0 0x7fa2e00ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913269485 0 0) 0x7fa2e0155680 con 0x7fa2e00c8100 2023-10-25T00:07:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c8003530 con 0x7fa2e00c8100 2023-10-25T00:07:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 513310952 0 0) 0x7fa2c8003530 con 0x7fa2e00c8100 2023-10-25T00:07:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e0152040 msgr2=0x7fa2e00c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:52.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 --2- 172.21.15.107:0/1134358703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e0152040 0x7fa2e00c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:52.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 >> v2:172.21.15.110:3300/0 conn(0x7fa2e006a4b0 msgr2=0x7fa2e00ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:52.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 --2- 172.21.15.107:0/1134358703 >> v2:172.21.15.110:3300/0 conn(0x7fa2e006a4b0 0x7fa2e00ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:52.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e0155030 con 0x7fa2e00c8100 2023-10-25T00:07:52.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fa2d0001680 con 0x7fa2e00c8100 2023-10-25T00:07:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 >> v1:172.21.15.107:6789/0 conn(0x7fa2e00c8100 legacy=0x7fa2e00abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 --2- 172.21.15.107:0/1134358703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e0152040 0x7fa2c8003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 --2- 172.21.15.107:0/1134358703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2c8003f00 0x7fa2c80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 --2- 172.21.15.107:0/1134358703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2c8004d10 0x7fa2c8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e0154a70 con 0x7fa2c8003f00 2023-10-25T00:07:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e0154d50 con 0x7fa2c8004d10 2023-10-25T00:07:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c800b9e0 con 0x7fa2e0152040 2023-10-25T00:07:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2ddd9b640 1 --2- 172.21.15.107:0/1134358703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2c8003f00 0x7fa2c80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2ddd9b640 1 -- 172.21.15.107:0/1134358703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e0152040 msgr2=0x7fa2c8003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2ddd9b640 1 --2- 172.21.15.107:0/1134358703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e0152040 0x7fa2c8003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2ddd9b640 1 -- 172.21.15.107:0/1134358703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2c8004d10 msgr2=0x7fa2c8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2ddd9b640 1 --2- 172.21.15.107:0/1134358703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2c8004d10 0x7fa2c8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.875+0000 7fa2ddd9b640 1 -- 172.21.15.107:0/1134358703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8003530 con 0x7fa2c8003f00 2023-10-25T00:07:52.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ddd9b640 1 --2- 172.21.15.107:0/1134358703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2c8003f00 0x7fa2c80047d0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000c290 tx=0x7fa2d001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1732a52d9f2b67c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00058e0 con 0x7fa2c8003f00 2023-10-25T00:07:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d0005a40 con 0x7fa2c8003f00 2023-10-25T00:07:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2dd59a640 1 -- 172.21.15.107:0/1134358703 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0005bf0 con 0x7fa2c8003f00 2023-10-25T00:07:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1134358703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2c8003f00 msgr2=0x7fa2c80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 --2- 172.21.15.107:0/1134358703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2c8003f00 0x7fa2c80047d0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000c290 tx=0x7fa2d001cca0 comp rx=0 tx=0).stop 2023-10-25T00:07:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ded9d640 1 -- 172.21.15.107:0/1134358703 reap_dead start 2023-10-25T00:07:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1134358703 shutdown_connections 2023-10-25T00:07:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1134358703 >> 172.21.15.107:0/1134358703 conn(0x7fa2e00ca200 msgr2=0x7fa2e00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1134358703 shutdown_connections 2023-10-25T00:07:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1134358703 wait complete. 2023-10-25T00:07:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 Processor -- start 2023-10-25T00:07:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 -- start start 2023-10-25T00:07:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2c8003f00 0x7fa2e00aac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e00c40d0 0x7fa2e014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e014b510 0x7fa2e00c25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e00ada30 con 0x7fa2e014b510 2023-10-25T00:07:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e00aecc0 con 0x7fa2c8003f00 2023-10-25T00:07:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e00adcd0 con 0x7fa2e00c40d0 2023-10-25T00:07:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ded9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e014b510 0x7fa2e00c25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ded9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e014b510 0x7fa2e00c25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33152/0 (socket says 172.21.15.107:33152) 2023-10-25T00:07:52.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ded9d640 1 -- 172.21.15.107:0/1522569736 learned_addr learned my addr 172.21.15.107:0/1522569736 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:52.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ddd9b640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e00c40d0 0x7fa2e014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:52.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2de59c640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2c8003f00 0x7fa2e00aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:52.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ded9d640 1 -- 172.21.15.107:0/1522569736 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e00c40d0 msgr2=0x7fa2e014afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:52.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ded9d640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e00c40d0 0x7fa2e014afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:52.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ded9d640 1 -- 172.21.15.107:0/1522569736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2c8003f00 msgr2=0x7fa2e00aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:52.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ded9d640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2c8003f00 0x7fa2e00aac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:52.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ded9d640 1 -- 172.21.15.107:0/1522569736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e00adfb0 con 0x7fa2e014b510 2023-10-25T00:07:52.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ddd9b640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e00c40d0 0x7fa2e014afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:07:52.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ded9d640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e014b510 0x7fa2e00c25c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0069810 tx=0x7fa2d800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ceffd640 1 -- 172.21.15.107:0/1522569736 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8010040 con 0x7fa2e014b510 2023-10-25T00:07:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ceffd640 1 -- 172.21.15.107:0/1522569736 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d800bae0 con 0x7fa2e014b510 2023-10-25T00:07:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1522569736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e00c20a0 con 0x7fa2e014b510 2023-10-25T00:07:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2ceffd640 1 -- 172.21.15.107:0/1522569736 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d800be70 con 0x7fa2e014b510 2023-10-25T00:07:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.879+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1522569736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e00ab1a0 con 0x7fa2e014b510 2023-10-25T00:07:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.883+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1522569736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e00c2290 con 0x7fa2e014b510 2023-10-25T00:07:52.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.883+0000 7fa2ceffd640 1 -- 172.21.15.107:0/1522569736 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa2d801d400 con 0x7fa2e014b510 2023-10-25T00:07:52.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.883+0000 7fa2ceffd640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa2c0065e60 0x7fa2c0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:52.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.883+0000 7fa2ceffd640 1 -- 172.21.15.107:0/1522569736 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7fa2d8084a60 con 0x7fa2e014b510 2023-10-25T00:07:52.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.883+0000 7fa2de59c640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa2c0065e60 0x7fa2c0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:52.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.883+0000 7fa2de59c640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa2c0065e60 0x7fa2c0068320 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa2d40047c0 tx=0x7fa2d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:52.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:52.887+0000 7fa2ceffd640 1 -- 172.21.15.107:0/1522569736 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa2d8054fe0 con 0x7fa2e014b510 2023-10-25T00:07:53.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1522569736 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2e00746e0 con 0x7fa2e014b510 2023-10-25T00:07:53.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2ceffd640 1 -- 172.21.15.107:0/1522569736 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5017 (secure 0 0 0) 0x7fa2d8018090 con 0x7fa2e014b510 2023-10-25T00:07:53.076 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:53.076 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:53.076 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 129 2023-10-25T00:07:53.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1522569736 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa2c0065e60 msgr2=0x7fa2c0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:53.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa2c0065e60 0x7fa2c0068320 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa2d40047c0 tx=0x7fa2d4008040 comp rx=0 tx=0).stop 2023-10-25T00:07:53.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1522569736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e014b510 msgr2=0x7fa2e00c25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:53.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e014b510 0x7fa2e00c25c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0069810 tx=0x7fa2d800b480 comp rx=0 tx=0).stop 2023-10-25T00:07:53.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1522569736 shutdown_connections 2023-10-25T00:07:53.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2e014b510 0x7fa2e00c25c0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa2c0065e60 0x7fa2c0068320 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2e00c40d0 0x7fa2e014afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 --2- 172.21.15.107:0/1522569736 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2c8003f00 0x7fa2e00aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1522569736 >> 172.21.15.107:0/1522569736 conn(0x7fa2e00ca200 msgr2=0x7fa2e014dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:53.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.075+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1522569736 shutdown_connections 2023-10-25T00:07:53.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.079+0000 7fa2e4a20640 1 -- 172.21.15.107:0/1522569736 wait complete. 2023-10-25T00:07:53.093 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:07:53.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5fa5fb6640 1 Processor -- start 2023-10-25T00:07:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5fa5fb6640 1 -- start start 2023-10-25T00:07:53.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5fa5fb6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014fd40 0x7f5fa00ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:53.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5fa5fb6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5fa006ee40 0x7f5fa00aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:53.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5fa5fb6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa006f730 con 0x7f5fa00a9820 2023-10-25T00:07:53.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5fa5fb6640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa00aa750 con 0x7f5fa006ee40 2023-10-25T00:07:53.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5fa5fb6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0156a40 con 0x7f5fa014fd40 2023-10-25T00:07:53.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f97fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5fa00a9820 0x7f5fa00c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50386/0 (socket says 172.21.15.107:50386) 2023-10-25T00:07:53.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f97fff640 1 -- 172.21.15.107:0/1654670475 learned_addr learned my addr 172.21.15.107:0/1654670475 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:53.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9f7fe640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014fd40 0x7f5fa00ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:53.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9ffff640 1 --2- 172.21.15.107:0/1654670475 >> v2:172.21.15.110:3300/0 conn(0x7f5fa006ee40 0x7f5fa00aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:53.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445175506 0 0) 0x7f5fa006f730 con 0x7f5fa00a9820 2023-10-25T00:07:53.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f80003530 con 0x7f5fa00a9820 2023-10-25T00:07:53.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 176606683 0 0) 0x7f5f80003530 con 0x7f5fa00a9820 2023-10-25T00:07:53.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014fd40 msgr2=0x7f5fa00ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014fd40 0x7f5fa00ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 >> v2:172.21.15.110:3300/0 conn(0x7f5fa006ee40 msgr2=0x7f5fa00aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 --2- 172.21.15.107:0/1654670475 >> v2:172.21.15.110:3300/0 conn(0x7f5fa006ee40 0x7f5fa00aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa0156710 con 0x7f5fa00a9820 2023-10-25T00:07:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5f88001680 con 0x7f5fa00a9820 2023-10-25T00:07:53.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 >> v1:172.21.15.107:6789/0 conn(0x7f5fa00a9820 legacy=0x7f5fa00c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f80003980 0x7f5f80003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 0x7f5f80004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014fd40 0x7f5f80005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa00aa750 con 0x7f5f800047d0 2023-10-25T00:07:53.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa0156a40 con 0x7f5f80003980 2023-10-25T00:07:53.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8000ba10 con 0x7f5fa014fd40 2023-10-25T00:07:53.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f97fff640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 0x7f5f80004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:53.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9f7fe640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f80003980 0x7f5f80003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:53.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9ffff640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014fd40 0x7f5f80005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:53.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f97fff640 1 -- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014fd40 msgr2=0x7f5f80005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:53.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f97fff640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014fd40 0x7f5f80005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f97fff640 1 -- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f80003980 msgr2=0x7f5f80003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:53.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f97fff640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f80003980 0x7f5f80003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f97fff640 1 -- 172.21.15.107:0/1654670475 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f80003530 con 0x7f5f800047d0 2023-10-25T00:07:53.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.383+0000 7f5f9ffff640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014fd40 0x7f5f80005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:07:53.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f97fff640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 0x7f5f80004c50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f5f8800c1c0 tx=0x7f5f8801cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69046dc19f9d80c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:53.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f880058e0 con 0x7f5f800047d0 2023-10-25T00:07:53.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f88005a40 con 0x7f5f800047d0 2023-10-25T00:07:53.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9effd640 1 -- 172.21.15.107:0/1654670475 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f88005e30 con 0x7f5f800047d0 2023-10-25T00:07:53.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/1654670475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 msgr2=0x7f5f80004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:53.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 --2- 172.21.15.107:0/1654670475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 0x7f5f80004c50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f5f8800c1c0 tx=0x7f5f8801cc10 comp rx=0 tx=0).stop 2023-10-25T00:07:53.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9ffff640 1 -- 172.21.15.107:0/1654670475 reap_dead start 2023-10-25T00:07:53.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/1654670475 shutdown_connections 2023-10-25T00:07:53.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/1654670475 >> 172.21.15.107:0/1654670475 conn(0x7f5fa00c2950 msgr2=0x7f5fa00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:53.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/1654670475 shutdown_connections 2023-10-25T00:07:53.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/1654670475 wait complete. 2023-10-25T00:07:53.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 Processor -- start 2023-10-25T00:07:53.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 -- start start 2023-10-25T00:07:53.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 0x7f5fa014cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:53.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fa01ee2d0 0x7f5fa0154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:53.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014c2a0 0x7f5fa014c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:53.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa00c0380 con 0x7f5f800047d0 2023-10-25T00:07:53.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa015a200 con 0x7f5fa01ee2d0 2023-10-25T00:07:53.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5fa5fb6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa01ee840 con 0x7f5fa014c2a0 2023-10-25T00:07:53.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 0x7f5fa014cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:53.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 0x7f5fa014cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33170/0 (socket says 172.21.15.107:33170) 2023-10-25T00:07:53.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9f7fe640 1 -- 172.21.15.107:0/117905107 learned_addr learned my addr 172.21.15.107:0/117905107 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:53.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9ffff640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014c2a0 0x7f5fa014c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:53.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f97fff640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fa01ee2d0 0x7f5fa0154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:53.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9f7fe640 1 -- 172.21.15.107:0/117905107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014c2a0 msgr2=0x7f5fa014c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:53.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9f7fe640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014c2a0 0x7f5fa014c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9f7fe640 1 -- 172.21.15.107:0/117905107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fa01ee2d0 msgr2=0x7f5fa0154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:53.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9f7fe640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fa01ee2d0 0x7f5fa0154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.387+0000 7f5f9f7fe640 1 -- 172.21.15.107:0/117905107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa01eeb20 con 0x7f5f800047d0 2023-10-25T00:07:53.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5f9f7fe640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 0x7f5fa014cd40 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f5f90011590 tx=0x7f5f90011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:53.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5f9cff9640 1 -- 172.21.15.107:0/117905107 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90015930 con 0x7f5f800047d0 2023-10-25T00:07:53.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5f9cff9640 1 -- 172.21.15.107:0/117905107 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f90015a90 con 0x7f5f800047d0 2023-10-25T00:07:53.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/117905107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa01eee00 con 0x7f5f800047d0 2023-10-25T00:07:53.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5f9cff9640 1 -- 172.21.15.107:0/117905107 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90015e20 con 0x7f5f800047d0 2023-10-25T00:07:53.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/117905107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa01ef0e0 con 0x7f5f800047d0 2023-10-25T00:07:53.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/117905107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa00c24a0 con 0x7f5f800047d0 2023-10-25T00:07:53.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5f9cff9640 1 -- 172.21.15.107:0/117905107 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5f9001d400 con 0x7f5f800047d0 2023-10-25T00:07:53.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5f9cff9640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f84065e90 0x7f5f84068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:53.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5f9cff9640 1 -- 172.21.15.107:0/117905107 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7f5f90084d00 con 0x7f5f800047d0 2023-10-25T00:07:53.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5f97fff640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f84065e90 0x7f5f84068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:53.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.391+0000 7f5f97fff640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f84065e90 0x7f5f84068350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5f880aa000 tx=0x7f5fa006ffa0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:53.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.395+0000 7f5f9cff9640 1 -- 172.21.15.107:0/117905107 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5f90051750 con 0x7f5f800047d0 2023-10-25T00:07:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.583+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/117905107 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fa015a340 con 0x7f5f800047d0 2023-10-25T00:07:53.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.583+0000 7f5f9cff9640 1 -- 172.21.15.107:0/117905107 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5017 (secure 0 0 0) 0x7f5fa015a340 con 0x7f5f800047d0 2023-10-25T00:07:53.584 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:53.584 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:53.584 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 129 2023-10-25T00:07:53.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/117905107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f84065e90 msgr2=0x7f5f84068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:53.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f84065e90 0x7f5f84068350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5f880aa000 tx=0x7f5fa006ffa0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/117905107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 msgr2=0x7f5fa014cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:53.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 0x7f5fa014cd40 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f5f90011590 tx=0x7f5f90011a50 comp rx=0 tx=0).stop 2023-10-25T00:07:53.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/117905107 shutdown_connections 2023-10-25T00:07:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5f84065e90 0x7f5f84068350 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fa014c2a0 0x7f5fa014c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fa01ee2d0 0x7f5fa0154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 --2- 172.21.15.107:0/117905107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f800047d0 0x7f5fa014cd40 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/117905107 >> 172.21.15.107:0/117905107 conn(0x7f5fa00c2950 msgr2=0x7f5fa00c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/117905107 shutdown_connections 2023-10-25T00:07:53.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:53.587+0000 7f5fa5fb6640 1 -- 172.21.15.107:0/117905107 wait complete. 2023-10-25T00:07:53.602 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5410 2023-10-25T00:07:53.603 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.d 2023-10-25T00:07:55.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:55.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.223+0000 7f6ec05d7640 1 Processor -- start 2023-10-25T00:07:55.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.223+0000 7f6ec05d7640 1 -- start start 2023-10-25T00:07:55.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ec05d7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6eb8155110 0x7f6eb81554f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:55.229 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ec05d7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb80c2060 con 0x7f6eb8155a30 2023-10-25T00:07:55.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ec05d7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb80c1450 con 0x7f6eb8072790 2023-10-25T00:07:55.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ec05d7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb80c1730 con 0x7f6eb8155110 2023-10-25T00:07:55.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebdb72640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6eb8155a30 0x7f6eb8072080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50396/0 (socket says 172.21.15.107:50396) 2023-10-25T00:07:55.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebdb72640 1 -- 172.21.15.107:0/416624993 learned_addr learned my addr 172.21.15.107:0/416624993 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:55.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebe373640 1 --2- 172.21.15.107:0/416624993 >> v2:172.21.15.110:3300/0 conn(0x7f6eb8155110 0x7f6eb81554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:55.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066367955 0 0) 0x7f6eb80c2060 con 0x7f6eb8155a30 2023-10-25T00:07:55.231 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea4003530 con 0x7f6eb8155a30 2023-10-25T00:07:55.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3983330911 0 0) 0x7f6ea4003530 con 0x7f6eb8155a30 2023-10-25T00:07:55.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 >> v2:172.21.15.110:3300/0 conn(0x7f6eb8155110 msgr2=0x7f6eb81554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:55.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 --2- 172.21.15.107:0/416624993 >> v2:172.21.15.110:3300/0 conn(0x7f6eb8155110 0x7f6eb81554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:55.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 >> v1:172.21.15.110:6789/0 conn(0x7f6eb8072790 legacy=0x7f6eb80c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:55.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb80c1a10 con 0x7f6eb8155a30 2023-10-25T00:07:55.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6ea8001680 con 0x7f6eb8155a30 2023-10-25T00:07:55.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 >> v1:172.21.15.107:6789/0 conn(0x7f6eb8155a30 legacy=0x7f6eb8072080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:55.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 --2- 172.21.15.107:0/416624993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ea4003980 0x7f6ea4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:55.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 --2- 172.21.15.107:0/416624993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ea40047d0 0x7f6ea4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:55.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 --2- 172.21.15.107:0/416624993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 0x7f6ea400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:55.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb80c1730 con 0x7f6ea4005190 2023-10-25T00:07:55.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea400c030 con 0x7f6ea40047d0 2023-10-25T00:07:55.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea400c1f0 con 0x7f6ea4003980 2023-10-25T00:07:55.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebeb74640 1 --2- 172.21.15.107:0/416624993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 0x7f6ea400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:55.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebdb72640 1 --2- 172.21.15.107:0/416624993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ea40047d0 0x7f6ea4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:55.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebeb74640 1 -- 172.21.15.107:0/416624993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ea4003980 msgr2=0x7f6ea4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:55.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebe373640 1 --2- 172.21.15.107:0/416624993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ea4003980 0x7f6ea4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:55.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebeb74640 1 --2- 172.21.15.107:0/416624993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6ea4003980 0x7f6ea4003da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:55.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebeb74640 1 -- 172.21.15.107:0/416624993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ea40047d0 msgr2=0x7f6ea4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:55.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebeb74640 1 --2- 172.21.15.107:0/416624993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6ea40047d0 0x7f6ea4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:55.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.227+0000 7f6ebeb74640 1 -- 172.21.15.107:0/416624993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea4003530 con 0x7f6ea4005190 2023-10-25T00:07:55.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebeb74640 1 -- 172.21.15.107:0/416624993 reap_dead start 2023-10-25T00:07:55.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebeb74640 1 --2- 172.21.15.107:0/416624993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 0x7f6ea400b990 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0001780 tx=0x7f6eb001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd56bc591d770dc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:55.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb00052d0 con 0x7f6ea4005190 2023-10-25T00:07:55.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb00a7050 con 0x7f6ea4005190 2023-10-25T00:07:55.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebd371640 1 -- 172.21.15.107:0/416624993 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0003840 con 0x7f6ea4005190 2023-10-25T00:07:55.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 -- 172.21.15.107:0/416624993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 msgr2=0x7f6ea400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:55.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 --2- 172.21.15.107:0/416624993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 0x7f6ea400b990 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0001780 tx=0x7f6eb001c400 comp rx=0 tx=0).stop 2023-10-25T00:07:55.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 -- 172.21.15.107:0/416624993 shutdown_connections 2023-10-25T00:07:55.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 --2- 172.21.15.107:0/416624993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 0x7f6ea400b990 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:55.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 -- 172.21.15.107:0/416624993 >> 172.21.15.107:0/416624993 conn(0x7f6eb80c5d50 msgr2=0x7f6eb80c6160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:55.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 -- 172.21.15.107:0/416624993 shutdown_connections 2023-10-25T00:07:55.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 -- 172.21.15.107:0/416624993 wait complete. 2023-10-25T00:07:55.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 Processor -- start 2023-10-25T00:07:55.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 -- start start 2023-10-25T00:07:55.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 0x7f6eb80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:55.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6eb80b91e0 0x7f6eb8152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:55.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6eb81531f0 0x7f6eb80a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:55.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb8070eb0 con 0x7f6ea4005190 2023-10-25T00:07:55.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb8071190 con 0x7f6eb80b91e0 2023-10-25T00:07:55.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb8071470 con 0x7f6eb81531f0 2023-10-25T00:07:55.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebe373640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 0x7f6eb80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:55.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebe373640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 0x7f6eb80a5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33182/0 (socket says 172.21.15.107:33182) 2023-10-25T00:07:55.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebe373640 1 -- 172.21.15.107:0/579435248 learned_addr learned my addr 172.21.15.107:0/579435248 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:55.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebdb72640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6eb80b91e0 0x7f6eb8152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:55.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebeb74640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6eb81531f0 0x7f6eb80a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:55.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebe373640 1 -- 172.21.15.107:0/579435248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6eb81531f0 msgr2=0x7f6eb80a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:55.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebe373640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6eb81531f0 0x7f6eb80a67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:55.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebe373640 1 -- 172.21.15.107:0/579435248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6eb80b91e0 msgr2=0x7f6eb8152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:55.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebe373640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6eb80b91e0 0x7f6eb8152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:55.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebe373640 1 -- 172.21.15.107:0/579435248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb8071750 con 0x7f6ea4005190 2023-10-25T00:07:55.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ebe373640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 0x7f6eb80a5790 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400f350 tx=0x7f6eb400f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:55.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6eaeffd640 1 -- 172.21.15.107:0/579435248 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4017600 con 0x7f6ea4005190 2023-10-25T00:07:55.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6eaeffd640 1 -- 172.21.15.107:0/579435248 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb4011040 con 0x7f6ea4005190 2023-10-25T00:07:55.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 -- 172.21.15.107:0/579435248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb8070bb0 con 0x7f6ea4005190 2023-10-25T00:07:55.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6eaeffd640 1 -- 172.21.15.107:0/579435248 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb40179c0 con 0x7f6ea4005190 2023-10-25T00:07:55.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.231+0000 7f6ec05d7640 1 -- 172.21.15.107:0/579435248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb80a5cd0 con 0x7f6ea4005190 2023-10-25T00:07:55.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.235+0000 7f6ec05d7640 1 -- 172.21.15.107:0/579435248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb80c58c0 con 0x7f6ea4005190 2023-10-25T00:07:55.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.235+0000 7f6eaeffd640 1 -- 172.21.15.107:0/579435248 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6eb4017b20 con 0x7f6ea4005190 2023-10-25T00:07:55.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.235+0000 7f6eaeffd640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e9c065e60 0x7f6e9c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:55.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.235+0000 7f6eaeffd640 1 -- 172.21.15.107:0/579435248 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7f6eb4084bd0 con 0x7f6ea4005190 2023-10-25T00:07:55.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.235+0000 7f6ebdb72640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e9c065e60 0x7f6e9c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:55.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.235+0000 7f6ebdb72640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e9c065e60 0x7f6e9c068320 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8001060 tx=0x7f6ea8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:55.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.239+0000 7f6eaeffd640 1 -- 172.21.15.107:0/579435248 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6eb4051660 con 0x7f6ea4005190 2023-10-25T00:07:55.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 -- 172.21.15.107:0/579435248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6eb8070ff0 con 0x7f6ea4005190 2023-10-25T00:07:55.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6eaeffd640 1 -- 172.21.15.107:0/579435248 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5017 (secure 0 0 0) 0x7f6eb8070ff0 con 0x7f6ea4005190 2023-10-25T00:07:55.427 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:55.428 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:55.428 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 129 2023-10-25T00:07:55.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 -- 172.21.15.107:0/579435248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e9c065e60 msgr2=0x7f6e9c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:55.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e9c065e60 0x7f6e9c068320 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8001060 tx=0x7f6ea8009040 comp rx=0 tx=0).stop 2023-10-25T00:07:55.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 -- 172.21.15.107:0/579435248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 msgr2=0x7f6eb80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:55.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 0x7f6eb80a5790 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400f350 tx=0x7f6eb400f810 comp rx=0 tx=0).stop 2023-10-25T00:07:55.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 -- 172.21.15.107:0/579435248 shutdown_connections 2023-10-25T00:07:55.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e9c065e60 0x7f6e9c068320 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6eb81531f0 0x7f6eb80a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6eb80b91e0 0x7f6eb8152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 --2- 172.21.15.107:0/579435248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ea4005190 0x7f6eb80a5790 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 -- 172.21.15.107:0/579435248 >> 172.21.15.107:0/579435248 conn(0x7f6eb80c5d50 msgr2=0x7f6eb80aa250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 -- 172.21.15.107:0/579435248 shutdown_connections 2023-10-25T00:07:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:55.427+0000 7f6ec05d7640 1 -- 172.21.15.107:0/579435248 wait complete. 2023-10-25T00:07:55.445 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:07:56.669 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:07:56.667+0000 7f96d8672640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:07:57.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:57.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946ce1b640 1 Processor -- start 2023-10-25T00:07:57.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946ce1b640 1 -- start start 2023-10-25T00:07:57.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946ce1b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f946806a4b0 0x7f94680ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:57.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946ce1b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9468155680 con 0x7f94680c8100 2023-10-25T00:07:57.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946ce1b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9468154a70 con 0x7f9468152040 2023-10-25T00:07:57.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946ce1b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9468154d50 con 0x7f946806a4b0 2023-10-25T00:07:57.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f9465d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f94680c8100 0x7f94680abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50400/0 (socket says 172.21.15.107:50400) 2023-10-25T00:07:57.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f9465d9b640 1 -- 172.21.15.107:0/2607186078 learned_addr learned my addr 172.21.15.107:0/2607186078 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:07:57.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f9466d9d640 1 --2- 172.21.15.107:0/2607186078 >> v2:172.21.15.110:3300/0 conn(0x7f946806a4b0 0x7f94680ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:57.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78107263 0 0) 0x7f9468155680 con 0x7f94680c8100 2023-10-25T00:07:57.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9450003530 con 0x7f94680c8100 2023-10-25T00:07:57.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3278313091 0 0) 0x7f9450003530 con 0x7f94680c8100 2023-10-25T00:07:57.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 >> v2:172.21.15.110:3300/0 conn(0x7f946806a4b0 msgr2=0x7f94680ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:57.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 --2- 172.21.15.107:0/2607186078 >> v2:172.21.15.110:3300/0 conn(0x7f946806a4b0 0x7f94680ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:57.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 >> v1:172.21.15.110:6789/0 conn(0x7f9468152040 legacy=0x7f94680c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:57.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9468155030 con 0x7f94680c8100 2023-10-25T00:07:57.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f945c001680 con 0x7f94680c8100 2023-10-25T00:07:57.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f9466d9d640 1 --2- 172.21.15.107:0/2607186078 >> v2:172.21.15.110:3300/0 conn(0x7f946806a4b0 0x7f94680ac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:07:57.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 >> v1:172.21.15.107:6789/0 conn(0x7f94680c8100 legacy=0x7f94680abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:57.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9450003980 0x7f9450003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:57.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94500047d0 0x7f9450004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:57.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500052b0 0x7f945000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:57.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9468154d50 con 0x7f94500052b0 2023-10-25T00:07:57.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945000c180 con 0x7f94500047d0 2023-10-25T00:07:57.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945000c340 con 0x7f9450003980 2023-10-25T00:07:57.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f9466d9d640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500052b0 0x7f945000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:57.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f9465d9b640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94500047d0 0x7f9450004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:57.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946659c640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9450003980 0x7f9450003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:57.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f9466d9d640 1 -- 172.21.15.107:0/2607186078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9450003980 msgr2=0x7f9450003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:57.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f9466d9d640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9450003980 0x7f9450003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:57.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f9466d9d640 1 -- 172.21.15.107:0/2607186078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94500047d0 msgr2=0x7f9450004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:07:57.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f9466d9d640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94500047d0 0x7f9450004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:57.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f9466d9d640 1 -- 172.21.15.107:0/2607186078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9450003530 con 0x7f94500052b0 2023-10-25T00:07:57.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.579+0000 7f946659c640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9450003980 0x7f9450003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:07:57.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f9466d9d640 1 -- 172.21.15.107:0/2607186078 reap_dead start 2023-10-25T00:07:57.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f9466d9d640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500052b0 0x7f945000b990 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f94600073b0 tx=0x7f946001cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2c09f397d6404d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:57.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94600a5cf0 con 0x7f94500052b0 2023-10-25T00:07:57.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94600a5e50 con 0x7f94500052b0 2023-10-25T00:07:57.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946559a640 1 -- 172.21.15.107:0/2607186078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94600ad680 con 0x7f94500052b0 2023-10-25T00:07:57.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 -- 172.21.15.107:0/2607186078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500052b0 msgr2=0x7f945000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:57.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500052b0 0x7f945000b990 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f94600073b0 tx=0x7f946001cf00 comp rx=0 tx=0).stop 2023-10-25T00:07:57.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 -- 172.21.15.107:0/2607186078 shutdown_connections 2023-10-25T00:07:57.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 --2- 172.21.15.107:0/2607186078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94500052b0 0x7f945000b990 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:57.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 -- 172.21.15.107:0/2607186078 >> 172.21.15.107:0/2607186078 conn(0x7f94680ca200 msgr2=0x7f94680ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:57.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 -- 172.21.15.107:0/2607186078 shutdown_connections 2023-10-25T00:07:57.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 -- 172.21.15.107:0/2607186078 wait complete. 2023-10-25T00:07:57.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 Processor -- start 2023-10-25T00:07:57.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 -- start start 2023-10-25T00:07:57.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94500052b0 0x7f94681e9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:57.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f946814c600 0x7f94681ed4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:57.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94681de720 0x7f94680ac580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:57.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94681de330 con 0x7f946814c600 2023-10-25T00:07:57.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94681ed020 con 0x7f94500052b0 2023-10-25T00:07:57.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.583+0000 7f946ce1b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94681dddb0 con 0x7f94681de720 2023-10-25T00:07:57.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9465d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f946814c600 0x7f94681ed4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:57.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9465d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f946814c600 0x7f94681ed4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33202/0 (socket says 172.21.15.107:33202) 2023-10-25T00:07:57.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9465d9b640 1 -- 172.21.15.107:0/2481449998 learned_addr learned my addr 172.21.15.107:0/2481449998 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:57.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f946659c640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94500052b0 0x7f94681e9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:57.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9466d9d640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94681de720 0x7f94680ac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:57.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9465d9b640 1 -- 172.21.15.107:0/2481449998 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94681de720 msgr2=0x7f94680ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:57.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9465d9b640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94681de720 0x7f94680ac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:57.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9465d9b640 1 -- 172.21.15.107:0/2481449998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94500052b0 msgr2=0x7f94681e9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:57.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9465d9b640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94500052b0 0x7f94681e9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:57.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9465d9b640 1 -- 172.21.15.107:0/2481449998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94681e9b50 con 0x7f946814c600 2023-10-25T00:07:57.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9465d9b640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f946814c600 0x7f94681ed4b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f945c001060 tx=0x7f945c011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:57.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9456ffd640 1 -- 172.21.15.107:0/2481449998 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c00fb00 con 0x7f946814c600 2023-10-25T00:07:57.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9456ffd640 1 -- 172.21.15.107:0/2481449998 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c00fca0 con 0x7f946814c600 2023-10-25T00:07:57.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f946ce1b640 1 -- 172.21.15.107:0/2481449998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94681de050 con 0x7f946814c600 2023-10-25T00:07:57.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9456ffd640 1 -- 172.21.15.107:0/2481449998 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c015650 con 0x7f946814c600 2023-10-25T00:07:57.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f946ce1b640 1 -- 172.21.15.107:0/2481449998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94681ed9f0 con 0x7f946814c600 2023-10-25T00:07:57.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f946ce1b640 1 -- 172.21.15.107:0/2481449998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94680746e0 con 0x7f946814c600 2023-10-25T00:07:57.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9456ffd640 1 -- 172.21.15.107:0/2481449998 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f945c0157b0 con 0x7f946814c600 2023-10-25T00:07:57.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9456ffd640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9448065e90 0x7f9448068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:57.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f9456ffd640 1 -- 172.21.15.107:0/2481449998 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7f945c087540 con 0x7f946814c600 2023-10-25T00:07:57.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f946659c640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9448065e90 0x7f9448068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:57.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.587+0000 7f946659c640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9448065e90 0x7f9448068350 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9468152b60 tx=0x7f9458009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:57.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.591+0000 7f9456ffd640 1 -- 172.21.15.107:0/2481449998 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f945c054030 con 0x7f946814c600 2023-10-25T00:07:57.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.783+0000 7f946ce1b640 1 -- 172.21.15.107:0/2481449998 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94681ed160 con 0x7f946814c600 2023-10-25T00:07:57.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.783+0000 7f9456ffd640 1 -- 172.21.15.107:0/2481449998 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5017 (secure 0 0 0) 0x7f945c057650 con 0x7f946814c600 2023-10-25T00:07:57.785 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:07:57.785 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:07:57.786 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 129 2023-10-25T00:07:57.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 -- 172.21.15.107:0/2481449998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9448065e90 msgr2=0x7f9448068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:57.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9448065e90 0x7f9448068350 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9468152b60 tx=0x7f9458009040 comp rx=0 tx=0).stop 2023-10-25T00:07:57.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 -- 172.21.15.107:0/2481449998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f946814c600 msgr2=0x7f94681ed4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:57.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f946814c600 0x7f94681ed4b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f945c001060 tx=0x7f945c011cc0 comp rx=0 tx=0).stop 2023-10-25T00:07:57.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 -- 172.21.15.107:0/2481449998 shutdown_connections 2023-10-25T00:07:57.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9448065e90 0x7f9448068350 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:57.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94681de720 0x7f94680ac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:57.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f946814c600 0x7f94681ed4b0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:57.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 --2- 172.21.15.107:0/2481449998 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94500052b0 0x7f94681e9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:57.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 -- 172.21.15.107:0/2481449998 >> 172.21.15.107:0/2481449998 conn(0x7f94680ca200 msgr2=0x7f94680ae940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:57.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 -- 172.21.15.107:0/2481449998 shutdown_connections 2023-10-25T00:07:57.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:57.787+0000 7f946ce1b640 1 -- 172.21.15.107:0/2481449998 wait complete. 2023-10-25T00:07:57.804 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:07:59.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:07:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e32e0640 1 Processor -- start 2023-10-25T00:07:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e32e0640 1 -- start start 2023-10-25T00:07:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e32e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86dc06ee40 0x7f86dc0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e32e0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc06f730 con 0x7f86dc0a9820 2023-10-25T00:07:59.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e32e0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc156a40 con 0x7f86dc14fd40 2023-10-25T00:07:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e32e0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86dc156710 con 0x7f86dc06ee40 2023-10-25T00:07:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e187d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86dc06ee40 0x7f86dc0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e187d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f86dc06ee40 0x7f86dc0ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33216/0 (socket says 172.21.15.107:33216) 2023-10-25T00:07:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e187d640 1 -- 172.21.15.107:0/1529300749 learned_addr learned my addr 172.21.15.107:0/1529300749 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e187d640 1 -- 172.21.15.107:0/1529300749 >> v1:172.21.15.110:6789/0 conn(0x7f86dc14fd40 legacy=0x7f86dc0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e187d640 1 -- 172.21.15.107:0/1529300749 >> v1:172.21.15.107:6789/0 conn(0x7f86dc0a9820 legacy=0x7f86dc15af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86dbfff640 1 -- 172.21.15.107:0/1529300749 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589069176 0 0) 0x7f86dc06f730 con 0x7f86dc0a9820 2023-10-25T00:07:59.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.939+0000 7f86e187d640 1 -- 172.21.15.107:0/1529300749 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc06f980 con 0x7f86dc06ee40 2023-10-25T00:07:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e187d640 1 --2- 172.21.15.107:0/1529300749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86dc06ee40 0x7f86dc0ab090 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009870 tx=0x7f86d401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42a380e6ee709cf server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86dbfff640 1 -- 172.21.15.107:0/1529300749 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d40ab600 con 0x7f86dc06ee40 2023-10-25T00:07:59.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86dbfff640 1 -- 172.21.15.107:0/1529300749 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d40a7050 con 0x7f86dc06ee40 2023-10-25T00:07:59.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86dbfff640 1 -- 172.21.15.107:0/1529300749 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d40ab980 con 0x7f86dc06ee40 2023-10-25T00:07:59.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 -- 172.21.15.107:0/1529300749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86dc06ee40 msgr2=0x7f86dc0ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:59.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 --2- 172.21.15.107:0/1529300749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86dc06ee40 0x7f86dc0ab090 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f86d4009870 tx=0x7f86d401c420 comp rx=0 tx=0).stop 2023-10-25T00:07:59.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 -- 172.21.15.107:0/1529300749 shutdown_connections 2023-10-25T00:07:59.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 --2- 172.21.15.107:0/1529300749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86dc06ee40 0x7f86dc0ab090 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:59.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 -- 172.21.15.107:0/1529300749 >> 172.21.15.107:0/1529300749 conn(0x7f86dc0c2950 msgr2=0x7f86dc0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:07:59.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 -- 172.21.15.107:0/1529300749 shutdown_connections 2023-10-25T00:07:59.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 -- 172.21.15.107:0/1529300749 wait complete. 2023-10-25T00:07:59.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 Processor -- start 2023-10-25T00:07:59.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 -- start start 2023-10-25T00:07:59.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86dc06ee40 0x7f86dc154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:59.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86dc14fd40 0x7f86dc148720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:59.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86dc14aaa0 0x7f86dc148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:59.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0bd000 con 0x7f86dc14fd40 2023-10-25T00:07:59.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc1491a0 con 0x7f86dc06ee40 2023-10-25T00:07:59.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e32e0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc1ee620 con 0x7f86dc14aaa0 2023-10-25T00:07:59.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e087b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86dc14fd40 0x7f86dc148720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:59.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e087b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86dc14fd40 0x7f86dc148720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33218/0 (socket says 172.21.15.107:33218) 2023-10-25T00:07:59.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e087b640 1 -- 172.21.15.107:0/1544345576 learned_addr learned my addr 172.21.15.107:0/1544345576 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:07:59.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e107c640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86dc06ee40 0x7f86dc154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:59.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e187d640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86dc14aaa0 0x7f86dc148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:59.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e087b640 1 -- 172.21.15.107:0/1544345576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86dc14aaa0 msgr2=0x7f86dc148c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:59.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e087b640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86dc14aaa0 0x7f86dc148c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:59.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e087b640 1 -- 172.21.15.107:0/1544345576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86dc06ee40 msgr2=0x7f86dc154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:07:59.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e087b640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86dc06ee40 0x7f86dc154b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:07:59.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e087b640 1 -- 172.21.15.107:0/1544345576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d40094d0 con 0x7f86dc14fd40 2023-10-25T00:07:59.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e107c640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86dc06ee40 0x7f86dc154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:07:59.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.943+0000 7f86e187d640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86dc14aaa0 0x7f86dc148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:07:59.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86e087b640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86dc14fd40 0x7f86dc148720 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f86d000c480 tx=0x7f86d000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:59.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86d9ffb640 1 -- 172.21.15.107:0/1544345576 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d0005480 con 0x7f86dc14fd40 2023-10-25T00:07:59.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86d9ffb640 1 -- 172.21.15.107:0/1544345576 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d0003600 con 0x7f86dc14fd40 2023-10-25T00:07:59.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86e32e0640 1 -- 172.21.15.107:0/1544345576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc1ee900 con 0x7f86dc14fd40 2023-10-25T00:07:59.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86d9ffb640 1 -- 172.21.15.107:0/1544345576 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d00039b0 con 0x7f86dc14fd40 2023-10-25T00:07:59.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86e32e0640 1 -- 172.21.15.107:0/1544345576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc1eeec0 con 0x7f86dc14fd40 2023-10-25T00:07:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86e32e0640 1 -- 172.21.15.107:0/1544345576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc0c24a0 con 0x7f86dc14fd40 2023-10-25T00:07:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86d9ffb640 1 -- 172.21.15.107:0/1544345576 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f86d0003b10 con 0x7f86dc14fd40 2023-10-25T00:07:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86d9ffb640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86c8065e90 0x7f86c8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:07:59.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86d9ffb640 1 -- 172.21.15.107:0/1544345576 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7f86d0083f30 con 0x7f86dc14fd40 2023-10-25T00:07:59.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86e107c640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86c8065e90 0x7f86c8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:07:59.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.947+0000 7f86e107c640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86c8065e90 0x7f86c8068350 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0012f0 tx=0x7f86cc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:07:59.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:07:59.951+0000 7f86d9ffb640 1 -- 172.21.15.107:0/1544345576 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f86d0050a20 con 0x7f86dc14fd40 2023-10-25T00:08:00.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.143+0000 7f86e32e0640 1 -- 172.21.15.107:0/1544345576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86dc0746e0 con 0x7f86dc14fd40 2023-10-25T00:08:00.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.143+0000 7f86d9ffb640 1 -- 172.21.15.107:0/1544345576 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5017 (secure 0 0 0) 0x7f86d0054040 con 0x7f86dc14fd40 2023-10-25T00:08:00.147 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:00.147 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:00.148 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 129 2023-10-25T00:08:00.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 -- 172.21.15.107:0/1544345576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86c8065e90 msgr2=0x7f86c8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:00.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86c8065e90 0x7f86c8068350 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0012f0 tx=0x7f86cc009040 comp rx=0 tx=0).stop 2023-10-25T00:08:00.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 -- 172.21.15.107:0/1544345576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86dc14fd40 msgr2=0x7f86dc148720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:00.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86dc14fd40 0x7f86dc148720 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f86d000c480 tx=0x7f86d000c940 comp rx=0 tx=0).stop 2023-10-25T00:08:00.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 -- 172.21.15.107:0/1544345576 shutdown_connections 2023-10-25T00:08:00.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86c8065e90 0x7f86c8068350 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86dc14aaa0 0x7f86dc148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86dc14fd40 0x7f86dc148720 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 --2- 172.21.15.107:0/1544345576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86dc06ee40 0x7f86dc154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 -- 172.21.15.107:0/1544345576 >> 172.21.15.107:0/1544345576 conn(0x7f86dc0c2950 msgr2=0x7f86dc0719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:00.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 -- 172.21.15.107:0/1544345576 shutdown_connections 2023-10-25T00:08:00.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.147+0000 7f86e32e0640 1 -- 172.21.15.107:0/1544345576 wait complete. 2023-10-25T00:08:00.163 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:08:00.604 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2023-10-25T00:08:00.605 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-25T00:08:00.605 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-25T00:08:00.607 INFO:tasks.ceph.mds.d:Started 2023-10-25T00:08:00.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:00.685 INFO:tasks.ceph.mds.d.smithi110.stdout:starting mds.d at 2023-10-25T00:08:00.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.739+0000 7fd08baa9640 1 Processor -- start 2023-10-25T00:08:00.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.739+0000 7fd08baa9640 1 -- start start 2023-10-25T00:08:00.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.739+0000 7fd08baa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0840720a0 0x7fd0840ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:00.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.739+0000 7fd08baa9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd084158500 0x7fd08415f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:00.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.739+0000 7fd08baa9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0840aa150 con 0x7fd0840c0aa0 2023-10-25T00:08:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.739+0000 7fd08baa9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0840a9900 con 0x7fd084158500 2023-10-25T00:08:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.739+0000 7fd08baa9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0840a9700 con 0x7fd0840720a0 2023-10-25T00:08:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd089845640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd0840c0aa0 0x7fd084157df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50414/0 (socket says 172.21.15.107:50414) 2023-10-25T00:08:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd089845640 1 -- 172.21.15.107:0/2610839874 learned_addr learned my addr 172.21.15.107:0/2610839874 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:00.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd089044640 1 --2- 172.21.15.107:0/2610839874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0840720a0 0x7fd0840ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08a046640 1 --2- 172.21.15.107:0/2610839874 >> v2:172.21.15.110:3300/0 conn(0x7fd084158500 0x7fd08415f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd088843640 1 -- 172.21.15.107:0/2610839874 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507110814 0 0) 0x7fd0840aa150 con 0x7fd0840c0aa0 2023-10-25T00:08:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd088843640 1 -- 172.21.15.107:0/2610839874 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd070003530 con 0x7fd0840c0aa0 2023-10-25T00:08:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08a046640 1 -- 172.21.15.107:0/2610839874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0840720a0 msgr2=0x7fd0840ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08a046640 1 --2- 172.21.15.107:0/2610839874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0840720a0 0x7fd0840ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08a046640 1 -- 172.21.15.107:0/2610839874 >> v1:172.21.15.107:6789/0 conn(0x7fd0840c0aa0 legacy=0x7fd084157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:00.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08a046640 1 -- 172.21.15.107:0/2610839874 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0841549a0 con 0x7fd084158500 2023-10-25T00:08:00.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd089044640 1 --2- 172.21.15.107:0/2610839874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0840720a0 0x7fd0840ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:08:00.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08a046640 1 --2- 172.21.15.107:0/2610839874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd084158500 0x7fd08415f180 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009d90 tx=0x7fd07c01f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6354a1a505677c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:00.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd088843640 1 -- 172.21.15.107:0/2610839874 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c0af600 con 0x7fd084158500 2023-10-25T00:08:00.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd088843640 1 -- 172.21.15.107:0/2610839874 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07c0ab050 con 0x7fd084158500 2023-10-25T00:08:00.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd088843640 1 -- 172.21.15.107:0/2610839874 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c0af980 con 0x7fd084158500 2023-10-25T00:08:00.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08baa9640 1 -- 172.21.15.107:0/2610839874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd084158500 msgr2=0x7fd08415f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:00.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08baa9640 1 --2- 172.21.15.107:0/2610839874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd084158500 0x7fd08415f180 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009d90 tx=0x7fd07c01f130 comp rx=0 tx=0).stop 2023-10-25T00:08:00.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08baa9640 1 -- 172.21.15.107:0/2610839874 shutdown_connections 2023-10-25T00:08:00.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08baa9640 1 --2- 172.21.15.107:0/2610839874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd084158500 0x7fd08415f180 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08baa9640 1 --2- 172.21.15.107:0/2610839874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0840720a0 0x7fd0840ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08baa9640 1 -- 172.21.15.107:0/2610839874 >> 172.21.15.107:0/2610839874 conn(0x7fd0840c4e40 msgr2=0x7fd0840c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:00.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08baa9640 1 -- 172.21.15.107:0/2610839874 shutdown_connections 2023-10-25T00:08:00.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.743+0000 7fd08baa9640 1 -- 172.21.15.107:0/2610839874 wait complete. 2023-10-25T00:08:00.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08baa9640 1 Processor -- start 2023-10-25T00:08:00.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08baa9640 1 -- start start 2023-10-25T00:08:00.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08baa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0840720a0 0x7fd0841ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:00.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08baa9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0840c0aa0 0x7fd0840bdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:00.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08baa9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd084158500 0x7fd0841ed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:00.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08baa9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0841ea9e0 con 0x7fd084158500 2023-10-25T00:08:00.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08baa9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0840be0b0 con 0x7fd0840c0aa0 2023-10-25T00:08:00.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08baa9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0840bce60 con 0x7fd0840720a0 2023-10-25T00:08:00.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08a046640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd084158500 0x7fd0841ed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:00.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08a046640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd084158500 0x7fd0841ed9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33222/0 (socket says 172.21.15.107:33222) 2023-10-25T00:08:00.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08a046640 1 -- 172.21.15.107:0/3468047464 learned_addr learned my addr 172.21.15.107:0/3468047464 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:00.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd089845640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0840720a0 0x7fd0841ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:00.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd089044640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0840c0aa0 0x7fd0840bdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:00.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08a046640 1 -- 172.21.15.107:0/3468047464 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0840720a0 msgr2=0x7fd0841ea4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:00.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08a046640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0840720a0 0x7fd0841ea4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08a046640 1 -- 172.21.15.107:0/3468047464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0840c0aa0 msgr2=0x7fd0840bdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:00.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08a046640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0840c0aa0 0x7fd0840bdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08a046640 1 -- 172.21.15.107:0/3468047464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd074009420 con 0x7fd084158500 2023-10-25T00:08:00.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08a046640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd084158500 0x7fd0841ed9b0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fd07c01d7e0 tx=0x7fd07c01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:00.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd07a7fc640 1 -- 172.21.15.107:0/3468047464 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c0afe30 con 0x7fd084158500 2023-10-25T00:08:00.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd07a7fc640 1 -- 172.21.15.107:0/3468047464 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07c009450 con 0x7fd084158500 2023-10-25T00:08:00.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd07a7fc640 1 -- 172.21.15.107:0/3468047464 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c0b9610 con 0x7fd084158500 2023-10-25T00:08:00.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08baa9640 1 -- 172.21.15.107:0/3468047464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c009a70 con 0x7fd084158500 2023-10-25T00:08:00.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08baa9640 1 -- 172.21.15.107:0/3468047464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0841ed1b0 con 0x7fd084158500 2023-10-25T00:08:00.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.747+0000 7fd08baa9640 1 -- 172.21.15.107:0/3468047464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0841ed010 con 0x7fd084158500 2023-10-25T00:08:00.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.751+0000 7fd07a7fc640 1 -- 172.21.15.107:0/3468047464 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd07c0b6050 con 0x7fd084158500 2023-10-25T00:08:00.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.751+0000 7fd07a7fc640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd068065e90 0x7fd068068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:00.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.751+0000 7fd089845640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd068065e90 0x7fd068068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:00.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.751+0000 7fd07a7fc640 1 -- 172.21.15.107:0/3468047464 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7fd07c1221e0 con 0x7fd084158500 2023-10-25T00:08:00.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.751+0000 7fd089845640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd068065e90 0x7fd068068350 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd0800011f0 tx=0x7fd080007d40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:00.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.755+0000 7fd07a7fc640 1 -- 172.21.15.107:0/3468047464 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd0841ed010 con 0x7fd084158500 2023-10-25T00:08:00.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.943+0000 7fd08baa9640 1 -- 172.21.15.107:0/3468047464 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0841ed010 con 0x7fd084158500 2023-10-25T00:08:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd07a7fc640 1 -- 172.21.15.107:0/3468047464 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5778 (secure 0 0 0) 0x7fd0841ed010 con 0x7fd084158500 2023-10-25T00:08:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:00.947 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:00.947 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 130 2023-10-25T00:08:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 -- 172.21.15.107:0/3468047464 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd068065e90 msgr2=0x7fd068068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd068065e90 0x7fd068068350 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd0800011f0 tx=0x7fd080007d40 comp rx=0 tx=0).stop 2023-10-25T00:08:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 -- 172.21.15.107:0/3468047464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd084158500 msgr2=0x7fd0841ed9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:00.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd084158500 0x7fd0841ed9b0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fd07c01d7e0 tx=0x7fd07c01bed0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 -- 172.21.15.107:0/3468047464 shutdown_connections 2023-10-25T00:08:00.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd084158500 0x7fd0841ed9b0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0840c0aa0 0x7fd0840bdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd068065e90 0x7fd068068350 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 --2- 172.21.15.107:0/3468047464 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd0840720a0 0x7fd0841ea4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:00.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 -- 172.21.15.107:0/3468047464 >> 172.21.15.107:0/3468047464 conn(0x7fd0840c4e40 msgr2=0x7fd084151a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:00.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 -- 172.21.15.107:0/3468047464 shutdown_connections 2023-10-25T00:08:00.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:00.947+0000 7fd08baa9640 1 -- 172.21.15.107:0/3468047464 wait complete. 2023-10-25T00:08:00.965 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2023-10-25T00:08:00.966 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before thrashing 2023-10-25T00:08:02.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:02.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.291+0000 7f86f2de1640 1 Processor -- start 2023-10-25T00:08:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.291+0000 7f86f2de1640 1 -- start start 2023-10-25T00:08:02.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f2de1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86ec14fd40 0x7f86ec0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:02.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f2de1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f86ec06ee40 0x7f86ec0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:02.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f2de1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ec06f730 con 0x7f86ec0a9820 2023-10-25T00:08:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f2de1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ec0aa750 con 0x7f86ec06ee40 2023-10-25T00:08:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f2de1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec156a40 con 0x7f86ec14fd40 2023-10-25T00:08:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86ebfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f86ec0a9820 0x7f86ec0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50422/0 (socket says 172.21.15.107:50422) 2023-10-25T00:08:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86ebfff640 1 -- 172.21.15.107:0/4211442557 learned_addr learned my addr 172.21.15.107:0/4211442557 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f0b7d640 1 --2- 172.21.15.107:0/4211442557 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86ec14fd40 0x7f86ec0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:02.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f137e640 1 --2- 172.21.15.107:0/4211442557 >> v2:172.21.15.110:3300/0 conn(0x7f86ec06ee40 0x7f86ec0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:02.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86eb7fe640 1 -- 172.21.15.107:0/4211442557 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487557459 0 0) 0x7f86ec06f730 con 0x7f86ec0a9820 2023-10-25T00:08:02.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86eb7fe640 1 -- 172.21.15.107:0/4211442557 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d4003530 con 0x7f86ec0a9820 2023-10-25T00:08:02.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f0b7d640 1 -- 172.21.15.107:0/4211442557 >> v2:172.21.15.110:3300/0 conn(0x7f86ec06ee40 msgr2=0x7f86ec0aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:02.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f0b7d640 1 --2- 172.21.15.107:0/4211442557 >> v2:172.21.15.110:3300/0 conn(0x7f86ec06ee40 0x7f86ec0aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:02.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f0b7d640 1 -- 172.21.15.107:0/4211442557 >> v1:172.21.15.107:6789/0 conn(0x7f86ec0a9820 legacy=0x7f86ec0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:02.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f0b7d640 1 -- 172.21.15.107:0/4211442557 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ec156710 con 0x7f86ec14fd40 2023-10-25T00:08:02.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.295+0000 7f86f0b7d640 1 --2- 172.21.15.107:0/4211442557 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86ec14fd40 0x7f86ec0ab090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0097c0 tx=0x7f86dc01c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18c7f38dfb6e6c1e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:02.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86eb7fe640 1 -- 172.21.15.107:0/4211442557 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc0ab600 con 0x7f86ec14fd40 2023-10-25T00:08:02.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86eb7fe640 1 -- 172.21.15.107:0/4211442557 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86dc0a7050 con 0x7f86ec14fd40 2023-10-25T00:08:02.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86eb7fe640 1 -- 172.21.15.107:0/4211442557 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc0ab980 con 0x7f86ec14fd40 2023-10-25T00:08:02.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 -- 172.21.15.107:0/4211442557 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86ec14fd40 msgr2=0x7f86ec0ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:02.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 --2- 172.21.15.107:0/4211442557 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86ec14fd40 0x7f86ec0ab090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f86dc0097c0 tx=0x7f86dc01c330 comp rx=0 tx=0).stop 2023-10-25T00:08:02.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 -- 172.21.15.107:0/4211442557 shutdown_connections 2023-10-25T00:08:02.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 --2- 172.21.15.107:0/4211442557 >> v2:172.21.15.110:3300/0 conn(0x7f86ec06ee40 0x7f86ec0aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:02.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 --2- 172.21.15.107:0/4211442557 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86ec14fd40 0x7f86ec0ab090 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:02.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 -- 172.21.15.107:0/4211442557 >> 172.21.15.107:0/4211442557 conn(0x7f86ec0c2950 msgr2=0x7f86ec0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:02.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 -- 172.21.15.107:0/4211442557 shutdown_connections 2023-10-25T00:08:02.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 -- 172.21.15.107:0/4211442557 wait complete. 2023-10-25T00:08:02.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 Processor -- start 2023-10-25T00:08:02.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 -- start start 2023-10-25T00:08:02.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86ec06ee40 0x7f86ec1eb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:02.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86ec14fd40 0x7f86ec1eeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:02.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86ec1ec100 0x7f86ec14da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:02.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec158f60 con 0x7f86ec06ee40 2023-10-25T00:08:02.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1ebbd0 con 0x7f86ec1ec100 2023-10-25T00:08:02.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec1ef0e0 con 0x7f86ec14fd40 2023-10-25T00:08:02.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f0b7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86ec06ee40 0x7f86ec1eb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:02.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f0b7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86ec06ee40 0x7f86ec1eb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33230/0 (socket says 172.21.15.107:33230) 2023-10-25T00:08:02.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f0b7d640 1 -- 172.21.15.107:0/4029111074 learned_addr learned my addr 172.21.15.107:0/4029111074 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:02.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86ebfff640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86ec14fd40 0x7f86ec1eeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:02.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f137e640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86ec1ec100 0x7f86ec14da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:02.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f0b7d640 1 -- 172.21.15.107:0/4029111074 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86ec14fd40 msgr2=0x7f86ec1eeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:02.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f0b7d640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86ec14fd40 0x7f86ec1eeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:02.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f0b7d640 1 -- 172.21.15.107:0/4029111074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86ec1ec100 msgr2=0x7f86ec14da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:02.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f0b7d640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86ec1ec100 0x7f86ec14da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:02.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f0b7d640 1 -- 172.21.15.107:0/4029111074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc009420 con 0x7f86ec06ee40 2023-10-25T00:08:02.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f137e640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86ec1ec100 0x7f86ec14da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:08:02.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f0b7d640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86ec06ee40 0x7f86ec1eb690 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f86dc019f60 tx=0x7f86dc0190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:02.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86e97fa640 1 -- 172.21.15.107:0/4029111074 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc006e20 con 0x7f86ec06ee40 2023-10-25T00:08:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86e97fa640 1 -- 172.21.15.107:0/4029111074 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86dc0b4410 con 0x7f86ec06ee40 2023-10-25T00:08:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 -- 172.21.15.107:0/4029111074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ec1586c0 con 0x7f86ec06ee40 2023-10-25T00:08:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86e97fa640 1 -- 172.21.15.107:0/4029111074 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc0b47a0 con 0x7f86ec06ee40 2023-10-25T00:08:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.299+0000 7f86f2de1640 1 -- 172.21.15.107:0/4029111074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ec158c80 con 0x7f86ec06ee40 2023-10-25T00:08:02.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.303+0000 7f86f2de1640 1 -- 172.21.15.107:0/4029111074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ec0c24a0 con 0x7f86ec06ee40 2023-10-25T00:08:02.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.303+0000 7f86e97fa640 1 -- 172.21.15.107:0/4029111074 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f86dc0b2040 con 0x7f86ec06ee40 2023-10-25T00:08:02.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.303+0000 7f86e97fa640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86d8065e60 0x7f86d8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:02.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.303+0000 7f86e97fa640 1 -- 172.21.15.107:0/4029111074 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7f86dc11e880 con 0x7f86ec06ee40 2023-10-25T00:08:02.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.303+0000 7f86ebfff640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86d8065e60 0x7f86d8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:02.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.303+0000 7f86ebfff640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86d8065e60 0x7f86d8068320 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f86ec1590c0 tx=0x7f86e0007d40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:02.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.307+0000 7f86e97fa640 1 -- 172.21.15.107:0/4029111074 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f86dc0eb3f0 con 0x7f86ec06ee40 2023-10-25T00:08:02.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.495+0000 7f86f2de1640 1 -- 172.21.15.107:0/4029111074 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86ec070840 con 0x7f86ec06ee40 2023-10-25T00:08:02.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.495+0000 7f86e97fa640 1 -- 172.21.15.107:0/4029111074 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f86dc0b0020 con 0x7f86ec06ee40 2023-10-25T00:08:02.496 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:02.496 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:02.496 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 130 2023-10-25T00:08:02.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 -- 172.21.15.107:0/4029111074 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86d8065e60 msgr2=0x7f86d8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:02.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86d8065e60 0x7f86d8068320 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f86ec1590c0 tx=0x7f86e0007d40 comp rx=0 tx=0).stop 2023-10-25T00:08:02.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 -- 172.21.15.107:0/4029111074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86ec06ee40 msgr2=0x7f86ec1eb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:02.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86ec06ee40 0x7f86ec1eb690 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f86dc019f60 tx=0x7f86dc0190d0 comp rx=0 tx=0).stop 2023-10-25T00:08:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 -- 172.21.15.107:0/4029111074 shutdown_connections 2023-10-25T00:08:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86ec1ec100 0x7f86ec14da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86d8065e60 0x7f86d8068320 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86ec14fd40 0x7f86ec1eeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:02.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 --2- 172.21.15.107:0/4029111074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86ec06ee40 0x7f86ec1eb690 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:02.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 -- 172.21.15.107:0/4029111074 >> 172.21.15.107:0/4029111074 conn(0x7f86ec0c2950 msgr2=0x7f86ec1559e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:02.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 -- 172.21.15.107:0/4029111074 shutdown_connections 2023-10-25T00:08:02.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:02.499+0000 7f86f2de1640 1 -- 172.21.15.107:0/4029111074 wait complete. 2023-10-25T00:08:02.514 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:08:04.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:04.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.643+0000 7fbc0563e640 1 Processor -- start 2023-10-25T00:08:04.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.643+0000 7fbc0563e640 1 -- start start 2023-10-25T00:08:04.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbc0563e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbc0014fd40 0x7fbc000ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:04.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbc0563e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc0006ee40 0x7fbc000aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:04.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbc0563e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc0006f730 con 0x7fbc000a9820 2023-10-25T00:08:04.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbc0563e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc000aa750 con 0x7fbc0014fd40 2023-10-25T00:08:04.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbc0563e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00156a40 con 0x7fbc0006ee40 2023-10-25T00:08:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfeffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbc0014fd40 0x7fbc000ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfe7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbc000a9820 0x7fbc000c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54464/0 (socket says 172.21.15.107:54464) 2023-10-25T00:08:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfe7fc640 1 -- 172.21.15.107:0/3126663418 learned_addr learned my addr 172.21.15.107:0/3126663418 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbff7fe640 1 --2- 172.21.15.107:0/3126663418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc0006ee40 0x7fbc000aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:04.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfeffd640 1 -- 172.21.15.107:0/3126663418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc0006ee40 msgr2=0x7fbc000aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:04.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfeffd640 1 --2- 172.21.15.107:0/3126663418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc0006ee40 0x7fbc000aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:04.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfeffd640 1 -- 172.21.15.107:0/3126663418 >> v1:172.21.15.107:6789/0 conn(0x7fbc000a9820 legacy=0x7fbc000c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:04.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfeffd640 1 -- 172.21.15.107:0/3126663418 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc00156710 con 0x7fbc0014fd40 2023-10-25T00:08:04.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfdffb640 1 -- 172.21.15.107:0/3126663418 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574679422 0 0) 0x7fbc0006f730 con 0x7fbc000a9820 2023-10-25T00:08:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfeffd640 1 --2- 172.21.15.107:0/3126663418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc0014fd40 0x7fbc000ab090 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40097c0 tx=0x7fbbf401c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f407268d32cf59da server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfdffb640 1 -- 172.21.15.107:0/3126663418 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf40ab600 con 0x7fbc0014fd40 2023-10-25T00:08:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfdffb640 1 -- 172.21.15.107:0/3126663418 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf40a7050 con 0x7fbc0014fd40 2023-10-25T00:08:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.647+0000 7fbbfdffb640 1 -- 172.21.15.107:0/3126663418 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf40ab980 con 0x7fbc0014fd40 2023-10-25T00:08:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- 172.21.15.107:0/3126663418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc0014fd40 msgr2=0x7fbc000ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 --2- 172.21.15.107:0/3126663418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc0014fd40 0x7fbc000ab090 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40097c0 tx=0x7fbbf401c330 comp rx=0 tx=0).stop 2023-10-25T00:08:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- 172.21.15.107:0/3126663418 shutdown_connections 2023-10-25T00:08:04.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 --2- 172.21.15.107:0/3126663418 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc0006ee40 0x7fbc000aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 --2- 172.21.15.107:0/3126663418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc0014fd40 0x7fbc000ab090 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- 172.21.15.107:0/3126663418 >> 172.21.15.107:0/3126663418 conn(0x7fbc000c2950 msgr2=0x7fbc000c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- 172.21.15.107:0/3126663418 shutdown_connections 2023-10-25T00:08:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- 172.21.15.107:0/3126663418 wait complete. 2023-10-25T00:08:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 Processor -- start 2023-10-25T00:08:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- start start 2023-10-25T00:08:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc0006ee40 0x7fbc001eb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc0014fd40 0x7fbc000bae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:04.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc000bfc20 0x7fbc0014d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00159f30 con 0x7fbc0006ee40 2023-10-25T00:08:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00159600 con 0x7fbc0014fd40 2023-10-25T00:08:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc001598e0 con 0x7fbc000bfc20 2023-10-25T00:08:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbfeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc0006ee40 0x7fbc001eb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:04.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbfeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc0006ee40 0x7fbc001eb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46786/0 (socket says 172.21.15.107:46786) 2023-10-25T00:08:04.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbfeffd640 1 -- 172.21.15.107:0/2188788397 learned_addr learned my addr 172.21.15.107:0/2188788397 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:04.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbff7fe640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc000bfc20 0x7fbc0014d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:04.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbfe7fc640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc0014fd40 0x7fbc000bae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:04.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbfeffd640 1 -- 172.21.15.107:0/2188788397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc000bfc20 msgr2=0x7fbc0014d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:04.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbfeffd640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc000bfc20 0x7fbc0014d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbfeffd640 1 -- 172.21.15.107:0/2188788397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc0014fd40 msgr2=0x7fbc000bae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbfeffd640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc0014fd40 0x7fbc000bae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbfeffd640 1 -- 172.21.15.107:0/2188788397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf4009420 con 0x7fbc0006ee40 2023-10-25T00:08:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbfe7fc640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc0014fd40 0x7fbc000bae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbfeffd640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc0006ee40 0x7fbc001eb510 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4019f60 tx=0x7fbbf40190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbe77fe640 1 -- 172.21.15.107:0/2188788397 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf40b4610 con 0x7fbc0006ee40 2023-10-25T00:08:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbe77fe640 1 -- 172.21.15.107:0/2188788397 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf40b4790 con 0x7fbc0006ee40 2023-10-25T00:08:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbbe77fe640 1 -- 172.21.15.107:0/2188788397 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf40b4b40 con 0x7fbc0006ee40 2023-10-25T00:08:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- 172.21.15.107:0/2188788397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc00159bc0 con 0x7fbc0006ee40 2023-10-25T00:08:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- 172.21.15.107:0/2188788397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc000c00d0 con 0x7fbc0006ee40 2023-10-25T00:08:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.651+0000 7fbc0563e640 1 -- 172.21.15.107:0/2188788397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc000c24a0 con 0x7fbc0006ee40 2023-10-25T00:08:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.655+0000 7fbbe77fe640 1 -- 172.21.15.107:0/2188788397 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbbf40b2040 con 0x7fbc0006ee40 2023-10-25T00:08:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.655+0000 7fbbe77fe640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbbec065e90 0x7fbbec068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.655+0000 7fbbe77fe640 1 -- 172.21.15.107:0/2188788397 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7fbbf411e560 con 0x7fbc0006ee40 2023-10-25T00:08:04.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.655+0000 7fbbfe7fc640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbbec065e90 0x7fbbec068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.655+0000 7fbbfe7fc640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbbec065e90 0x7fbbec068350 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbc0014f760 tx=0x7fbbf0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:04.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.659+0000 7fbbe77fe640 1 -- 172.21.15.107:0/2188788397 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbbf40eaff0 con 0x7fbc0006ee40 2023-10-25T00:08:04.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.843+0000 7fbc0563e640 1 -- 172.21.15.107:0/2188788397 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc00070840 con 0x7fbc0006ee40 2023-10-25T00:08:04.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.843+0000 7fbbe77fe640 1 -- 172.21.15.107:0/2188788397 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5778 (secure 0 0 0) 0x7fbbf40b0090 con 0x7fbc0006ee40 2023-10-25T00:08:04.845 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:04.845 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 130 2023-10-25T00:08:04.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 -- 172.21.15.107:0/2188788397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbbec065e90 msgr2=0x7fbbec068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:04.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbbec065e90 0x7fbbec068350 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbc0014f760 tx=0x7fbbf0009040 comp rx=0 tx=0).stop 2023-10-25T00:08:04.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 -- 172.21.15.107:0/2188788397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc0006ee40 msgr2=0x7fbc001eb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:04.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc0006ee40 0x7fbc001eb510 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4019f60 tx=0x7fbbf40190d0 comp rx=0 tx=0).stop 2023-10-25T00:08:04.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 -- 172.21.15.107:0/2188788397 shutdown_connections 2023-10-25T00:08:04.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbbec065e90 0x7fbbec068350 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:04.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc000bfc20 0x7fbc0014d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:04.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc0014fd40 0x7fbc000bae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:04.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 --2- 172.21.15.107:0/2188788397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc0006ee40 0x7fbc001eb510 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:04.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 -- 172.21.15.107:0/2188788397 >> 172.21.15.107:0/2188788397 conn(0x7fbc000c2950 msgr2=0x7fbc0015c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:04.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 -- 172.21.15.107:0/2188788397 shutdown_connections 2023-10-25T00:08:04.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:04.847+0000 7fbc0563e640 1 -- 172.21.15.107:0/2188788397 wait complete. 2023-10-25T00:08:04.863 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:08:06.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:06.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:06.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.991+0000 7fb35d0be640 1 Processor -- start 2023-10-25T00:08:06.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.991+0000 7fb35d0be640 1 -- start start 2023-10-25T00:08:06.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.991+0000 7fb35d0be640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb35814fd40 0x7fb3580ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:06.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.991+0000 7fb35d0be640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb35806ee40 0x7fb3580aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:06.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.991+0000 7fb35d0be640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35806f730 con 0x7fb3580a9820 2023-10-25T00:08:06.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.991+0000 7fb35d0be640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3580aa750 con 0x7fb35814fd40 2023-10-25T00:08:06.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.991+0000 7fb35d0be640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb358156a40 con 0x7fb35806ee40 2023-10-25T00:08:06.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.991+0000 7fb35759e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb35806ee40 0x7fb3580aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:06.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.991+0000 7fb356d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb35814fd40 0x7fb3580ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:06.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.995+0000 7fb35659c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb3580a9820 0x7fb3580c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:58630/0 (socket says 172.21.15.107:58630) 2023-10-25T00:08:06.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.995+0000 7fb35659c640 1 -- 172.21.15.107:0/3216878641 learned_addr learned my addr 172.21.15.107:0/3216878641 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:06.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.995+0000 7fb356d9d640 1 -- 172.21.15.107:0/3216878641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb35806ee40 msgr2=0x7fb3580aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:06.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.995+0000 7fb356d9d640 1 --2- 172.21.15.107:0/3216878641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb35806ee40 0x7fb3580aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:06.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.995+0000 7fb356d9d640 1 -- 172.21.15.107:0/3216878641 >> v1:172.21.15.110:6789/0 conn(0x7fb3580a9820 legacy=0x7fb3580c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.995+0000 7fb356d9d640 1 -- 172.21.15.107:0/3216878641 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358156710 con 0x7fb35814fd40 2023-10-25T00:08:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.995+0000 7fb35759e640 1 --2- 172.21.15.107:0/3216878641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb35806ee40 0x7fb3580aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:08:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.995+0000 7fb356d9d640 1 --2- 172.21.15.107:0/3216878641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb35814fd40 0x7fb3580ab090 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb3500097c0 tx=0x7fb35001c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1d101b11a4b77df server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.995+0000 7fb355d9b640 1 -- 172.21.15.107:0/3216878641 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3500ab600 con 0x7fb35814fd40 2023-10-25T00:08:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.995+0000 7fb355d9b640 1 -- 172.21.15.107:0/3216878641 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3500a7050 con 0x7fb35814fd40 2023-10-25T00:08:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.995+0000 7fb355d9b640 1 -- 172.21.15.107:0/3216878641 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3500ab980 con 0x7fb35814fd40 2023-10-25T00:08:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 -- 172.21.15.107:0/3216878641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb35814fd40 msgr2=0x7fb3580ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 --2- 172.21.15.107:0/3216878641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb35814fd40 0x7fb3580ab090 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb3500097c0 tx=0x7fb35001c330 comp rx=0 tx=0).stop 2023-10-25T00:08:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 -- 172.21.15.107:0/3216878641 shutdown_connections 2023-10-25T00:08:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 --2- 172.21.15.107:0/3216878641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb35806ee40 0x7fb3580aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 --2- 172.21.15.107:0/3216878641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb35814fd40 0x7fb3580ab090 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 -- 172.21.15.107:0/3216878641 >> 172.21.15.107:0/3216878641 conn(0x7fb3580c2950 msgr2=0x7fb3580c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:07.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 -- 172.21.15.107:0/3216878641 shutdown_connections 2023-10-25T00:08:07.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 -- 172.21.15.107:0/3216878641 wait complete. 2023-10-25T00:08:07.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 Processor -- start 2023-10-25T00:08:07.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 -- start start 2023-10-25T00:08:07.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb35806ee40 0x7fb3580bfb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb35814fd40 0x7fb35814dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35815ca80 0x7fb3581ee260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580bf790 con 0x7fb35815ca80 2023-10-25T00:08:07.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3580bb5f0 con 0x7fb35806ee40 2023-10-25T00:08:07.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3581eb750 con 0x7fb35814fd40 2023-10-25T00:08:07.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35815ca80 0x7fb3581ee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35815ca80 0x7fb3581ee260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46792/0 (socket says 172.21.15.107:46792) 2023-10-25T00:08:07.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35759e640 1 -- 172.21.15.107:0/1419585035 learned_addr learned my addr 172.21.15.107:0/1419585035 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:07.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35659c640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb35814fd40 0x7fb35814dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb356d9d640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb35806ee40 0x7fb3580bfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35759e640 1 -- 172.21.15.107:0/1419585035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb35814fd40 msgr2=0x7fb35814dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35759e640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb35814fd40 0x7fb35814dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35759e640 1 -- 172.21.15.107:0/1419585035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb35806ee40 msgr2=0x7fb3580bfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35759e640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb35806ee40 0x7fb3580bfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35759e640 1 -- 172.21.15.107:0/1419585035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb348009a70 con 0x7fb35815ca80 2023-10-25T00:08:07.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35759e640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35815ca80 0x7fb3581ee260 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fb34800ef90 tx=0x7fb348009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:07.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb34f7fe640 1 -- 172.21.15.107:0/1419585035 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348009d70 con 0x7fb35815ca80 2023-10-25T00:08:07.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb34f7fe640 1 -- 172.21.15.107:0/1419585035 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34800f040 con 0x7fb35815ca80 2023-10-25T00:08:07.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 -- 172.21.15.107:0/1419585035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb350009420 con 0x7fb35815ca80 2023-10-25T00:08:07.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb34f7fe640 1 -- 172.21.15.107:0/1419585035 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3480146c0 con 0x7fb35815ca80 2023-10-25T00:08:07.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:06.999+0000 7fb35d0be640 1 -- 172.21.15.107:0/1419585035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3581ebc80 con 0x7fb35815ca80 2023-10-25T00:08:07.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.003+0000 7fb35d0be640 1 -- 172.21.15.107:0/1419585035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3580c24a0 con 0x7fb35815ca80 2023-10-25T00:08:07.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.003+0000 7fb34f7fe640 1 -- 172.21.15.107:0/1419585035 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb348014820 con 0x7fb35815ca80 2023-10-25T00:08:07.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.003+0000 7fb34f7fe640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb338065e60 0x7fb338068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.003+0000 7fb34f7fe640 1 -- 172.21.15.107:0/1419585035 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7fb348085730 con 0x7fb35815ca80 2023-10-25T00:08:07.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.003+0000 7fb356d9d640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb338065e60 0x7fb338068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.003+0000 7fb356d9d640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb338065e60 0x7fb338068320 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb3500ad000 tx=0x7fb350008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:07.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.007+0000 7fb34f7fe640 1 -- 172.21.15.107:0/1419585035 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb348052350 con 0x7fb35815ca80 2023-10-25T00:08:07.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.095+0000 7f249de5e640 1 Processor -- start 2023-10-25T00:08:07.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.095+0000 7f249de5e640 1 -- start start 2023-10-25T00:08:07.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f249814c470 0x7f249814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2498152b30 0x7f2498152f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2498155cd0 con 0x7f249806f490 2023-10-25T00:08:07.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24981526e0 con 0x7f249814c470 2023-10-25T00:08:07.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24980c4380 con 0x7f2498152b30 2023-10-25T00:08:07.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f24977fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f249814c470 0x7f249814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f24977fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f249814c470 0x7f249814c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46808/0 (socket says 172.21.15.107:46808) 2023-10-25T00:08:07.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f24977fe640 1 -- 172.21.15.107:0/2079982570 learned_addr learned my addr 172.21.15.107:0/2079982570 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:07.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f2497fff640 1 --2- 172.21.15.107:0/2079982570 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2498152b30 0x7f2498152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f24977fe640 1 -- 172.21.15.107:0/2079982570 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2498152b30 msgr2=0x7f2498152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f24977fe640 1 --2- 172.21.15.107:0/2079982570 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2498152b30 0x7f2498152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f24977fe640 1 -- 172.21.15.107:0/2079982570 >> v1:172.21.15.110:6789/0 conn(0x7f249806f490 legacy=0x7f2498071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:07.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f24977fe640 1 -- 172.21.15.107:0/2079982570 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24980c48b0 con 0x7f249814c470 2023-10-25T00:08:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f24977fe640 1 --2- 172.21.15.107:0/2079982570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249814c470 0x7f249814c850 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2490009870 tx=0x7f249001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8e4861ee2d8189 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f24967fc640 1 -- 172.21.15.107:0/2079982570 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24900ab600 con 0x7f249814c470 2023-10-25T00:08:07.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f24967fc640 1 -- 172.21.15.107:0/2079982570 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24900a7050 con 0x7f249814c470 2023-10-25T00:08:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f24967fc640 1 -- 172.21.15.107:0/2079982570 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24900ab980 con 0x7f249814c470 2023-10-25T00:08:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 -- 172.21.15.107:0/2079982570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249814c470 msgr2=0x7f249814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 --2- 172.21.15.107:0/2079982570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249814c470 0x7f249814c850 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f2490009870 tx=0x7f249001c3e0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 -- 172.21.15.107:0/2079982570 shutdown_connections 2023-10-25T00:08:07.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 --2- 172.21.15.107:0/2079982570 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2498152b30 0x7f2498152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 --2- 172.21.15.107:0/2079982570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249814c470 0x7f249814c850 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 -- 172.21.15.107:0/2079982570 >> 172.21.15.107:0/2079982570 conn(0x7f2498070f60 msgr2=0x7f2498071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 -- 172.21.15.107:0/2079982570 shutdown_connections 2023-10-25T00:08:07.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.099+0000 7f249de5e640 1 -- 172.21.15.107:0/2079982570 wait complete. 2023-10-25T00:08:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f249de5e640 1 Processor -- start 2023-10-25T00:08:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f249de5e640 1 -- start start 2023-10-25T00:08:07.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f249de5e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f249806f490 0x7f24980c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f249de5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249814c470 0x7f24980c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f249de5e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2498152b30 0x7f24981eda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f249de5e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24980c6860 con 0x7f249814c470 2023-10-25T00:08:07.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f249de5e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249814ffb0 con 0x7f2498152b30 2023-10-25T00:08:07.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f249de5e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24981457b0 con 0x7f249806f490 2023-10-25T00:08:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f2496ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249814c470 0x7f24980c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f2496ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249814c470 0x7f24980c7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46820/0 (socket says 172.21.15.107:46820) 2023-10-25T00:08:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f2496ffd640 1 -- 172.21.15.107:0/1450981696 learned_addr learned my addr 172.21.15.107:0/1450981696 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:07.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f24977fe640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f249806f490 0x7f24980c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f2497fff640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2498152b30 0x7f24981eda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f2496ffd640 1 -- 172.21.15.107:0/1450981696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f249806f490 msgr2=0x7f24980c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f2496ffd640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f249806f490 0x7f24980c6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f2496ffd640 1 -- 172.21.15.107:0/1450981696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2498152b30 msgr2=0x7f24981eda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f2496ffd640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2498152b30 0x7f24981eda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f2496ffd640 1 -- 172.21.15.107:0/1450981696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24900094d0 con 0x7f249814c470 2023-10-25T00:08:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f2496ffd640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249814c470 0x7f24980c7230 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f248c00c450 tx=0x7f248c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f247ffff640 1 -- 172.21.15.107:0/1450981696 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c012830 con 0x7f249814c470 2023-10-25T00:08:07.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f247ffff640 1 -- 172.21.15.107:0/1450981696 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c0129b0 con 0x7f249814c470 2023-10-25T00:08:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f249de5e640 1 -- 172.21.15.107:0/1450981696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2498145a90 con 0x7f249814c470 2023-10-25T00:08:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f247ffff640 1 -- 172.21.15.107:0/1450981696 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c012d60 con 0x7f249814c470 2023-10-25T00:08:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.103+0000 7f249de5e640 1 -- 172.21.15.107:0/1450981696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f249814a110 con 0x7f249814c470 2023-10-25T00:08:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.107+0000 7f249de5e640 1 -- 172.21.15.107:0/1450981696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24980746e0 con 0x7f249814c470 2023-10-25T00:08:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.107+0000 7f247ffff640 1 -- 172.21.15.107:0/1450981696 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f248c01c460 con 0x7f249814c470 2023-10-25T00:08:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.107+0000 7f247ffff640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2484065e60 0x7f2484068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.107+0000 7f247ffff640 1 -- 172.21.15.107:0/1450981696 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7f248c084280 con 0x7f249814c470 2023-10-25T00:08:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.107+0000 7f24977fe640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2484065e60 0x7f2484068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.107+0000 7f24977fe640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2484065e60 0x7f2484068320 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f24900ad000 tx=0x7f2490008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:07.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.111+0000 7f247ffff640 1 -- 172.21.15.107:0/1450981696 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f248c050ce0 con 0x7f249814c470 2023-10-25T00:08:07.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.203+0000 7fb35d0be640 1 -- 172.21.15.107:0/1419585035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb358070840 con 0x7fb35815ca80 2023-10-25T00:08:07.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.203+0000 7fb34f7fe640 1 -- 172.21.15.107:0/1419585035 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5778 (secure 0 0 0) 0x7fb348018020 con 0x7fb35815ca80 2023-10-25T00:08:07.204 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:07.204 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:07.205 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 130 2023-10-25T00:08:07.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 -- 172.21.15.107:0/1419585035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb338065e60 msgr2=0x7fb338068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb338065e60 0x7fb338068320 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb3500ad000 tx=0x7fb350008040 comp rx=0 tx=0).stop 2023-10-25T00:08:07.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 -- 172.21.15.107:0/1419585035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35815ca80 msgr2=0x7fb3581ee260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35815ca80 0x7fb3581ee260 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fb34800ef90 tx=0x7fb348009f90 comp rx=0 tx=0).stop 2023-10-25T00:08:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 -- 172.21.15.107:0/1419585035 shutdown_connections 2023-10-25T00:08:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35815ca80 0x7fb3581ee260 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb338065e60 0x7fb338068320 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb35814fd40 0x7fb35814dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 --2- 172.21.15.107:0/1419585035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb35806ee40 0x7fb3580bfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 -- 172.21.15.107:0/1419585035 >> 172.21.15.107:0/1419585035 conn(0x7fb3580c2950 msgr2=0x7fb3580c0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 -- 172.21.15.107:0/1419585035 shutdown_connections 2023-10-25T00:08:07.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.207+0000 7fb35d0be640 1 -- 172.21.15.107:0/1419585035 wait complete. 2023-10-25T00:08:07.222 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:08:07.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.299+0000 7f249de5e640 1 -- 172.21.15.107:0/1450981696 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24980677b0 con 0x7f249814c470 2023-10-25T00:08:07.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f247ffff640 1 -- 172.21.15.107:0/1450981696 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f248c054300 con 0x7f249814c470 2023-10-25T00:08:07.302 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:07.303 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:07.303 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 130 2023-10-25T00:08:07.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 -- 172.21.15.107:0/1450981696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2484065e60 msgr2=0x7f2484068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2484065e60 0x7f2484068320 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f24900ad000 tx=0x7f2490008040 comp rx=0 tx=0).stop 2023-10-25T00:08:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 -- 172.21.15.107:0/1450981696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249814c470 msgr2=0x7f24980c7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249814c470 0x7f24980c7230 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f248c00c450 tx=0x7f248c00c910 comp rx=0 tx=0).stop 2023-10-25T00:08:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 -- 172.21.15.107:0/1450981696 shutdown_connections 2023-10-25T00:08:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2498152b30 0x7f24981eda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f249814c470 0x7f24980c7230 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2484065e60 0x7f2484068320 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 --2- 172.21.15.107:0/1450981696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f249806f490 0x7f24980c6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 -- 172.21.15.107:0/1450981696 >> 172.21.15.107:0/1450981696 conn(0x7f2498070f60 msgr2=0x7f2498071910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:07.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 -- 172.21.15.107:0/1450981696 shutdown_connections 2023-10-25T00:08:07.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.303+0000 7f249de5e640 1 -- 172.21.15.107:0/1450981696 wait complete. 2023-10-25T00:08:07.321 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2023-10-25T00:08:07.321 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:08:07.322 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:08:07.332 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:08:07.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:07.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb9b96640 1 Processor -- start 2023-10-25T00:08:07.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb9b96640 1 -- start start 2023-10-25T00:08:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb9b96640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb413b0f0 0x7f2bb406cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb9b96640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2bb406d430 0x7f2bb40c3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb9b96640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb40a9680 con 0x7f2bb406a7e0 2023-10-25T00:08:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb9b96640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb40a9b10 con 0x7f2bb406d430 2023-10-25T00:08:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb9b96640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb4156c70 con 0x7f2bb413b0f0 2023-10-25T00:08:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb413b0f0 0x7f2bb406cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb37fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb413b0f0 0x7f2bb406cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:48276/0 (socket says 172.21.15.107:48276) 2023-10-25T00:08:07.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb37fe640 1 -- 172.21.15.107:0/1448788431 learned_addr learned my addr 172.21.15.107:0/1448788431 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb2ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2bb406d430 0x7f2bb40c3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb37fe640 1 -- 172.21.15.107:0/1448788431 >> v2:172.21.15.110:3300/0 conn(0x7f2bb406d430 msgr2=0x7f2bb40c3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb37fe640 1 --2- 172.21.15.107:0/1448788431 >> v2:172.21.15.110:3300/0 conn(0x7f2bb406d430 0x7f2bb40c3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb37fe640 1 -- 172.21.15.107:0/1448788431 >> v1:172.21.15.110:6789/0 conn(0x7f2bb406a7e0 legacy=0x7f2bb40c3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.495+0000 7f2bb37fe640 1 -- 172.21.15.107:0/1448788431 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb40c4370 con 0x7f2bb413b0f0 2023-10-25T00:08:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb37fe640 1 --2- 172.21.15.107:0/1448788431 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb413b0f0 0x7f2bb406cef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4009870 tx=0x7f2ba401c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36abe0fc41d0f9de server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb27fc640 1 -- 172.21.15.107:0/1448788431 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba40ac600 con 0x7f2bb413b0f0 2023-10-25T00:08:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb27fc640 1 -- 172.21.15.107:0/1448788431 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba40a8050 con 0x7f2bb413b0f0 2023-10-25T00:08:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb27fc640 1 -- 172.21.15.107:0/1448788431 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba40ac980 con 0x7f2bb413b0f0 2023-10-25T00:08:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1448788431 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb413b0f0 msgr2=0x7f2bb406cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb9b96640 1 --2- 172.21.15.107:0/1448788431 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb413b0f0 0x7f2bb406cef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4009870 tx=0x7f2ba401c4b0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1448788431 shutdown_connections 2023-10-25T00:08:07.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb9b96640 1 --2- 172.21.15.107:0/1448788431 >> v2:172.21.15.110:3300/0 conn(0x7f2bb406d430 0x7f2bb40c3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb9b96640 1 --2- 172.21.15.107:0/1448788431 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb413b0f0 0x7f2bb406cef0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1448788431 >> 172.21.15.107:0/1448788431 conn(0x7f2bb40c6ed0 msgr2=0x7f2bb40c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1448788431 shutdown_connections 2023-10-25T00:08:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1448788431 wait complete. 2023-10-25T00:08:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb9b96640 1 Processor -- start 2023-10-25T00:08:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.499+0000 7f2bb9b96640 1 -- start start 2023-10-25T00:08:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb9b96640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb406a7e0 0x7f2bb41eeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb9b96640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bb406d430 0x7f2bb415b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb9b96640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bb413b0f0 0x7f2bb40c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb9b96640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb41557a0 con 0x7f2bb413b0f0 2023-10-25T00:08:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb9b96640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb4155a80 con 0x7f2bb406d430 2023-10-25T00:08:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb9b96640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb41ee050 con 0x7f2bb406a7e0 2023-10-25T00:08:07.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bb413b0f0 0x7f2bb40c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bb413b0f0 0x7f2bb40c29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46830/0 (socket says 172.21.15.107:46830) 2023-10-25T00:08:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb3fff640 1 -- 172.21.15.107:0/1951964732 learned_addr learned my addr 172.21.15.107:0/1951964732 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb37fe640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb406a7e0 0x7f2bb41eeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb2ffd640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bb406d430 0x7f2bb415b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb3fff640 1 -- 172.21.15.107:0/1951964732 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb406a7e0 msgr2=0x7f2bb41eeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb3fff640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb406a7e0 0x7f2bb41eeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb3fff640 1 -- 172.21.15.107:0/1951964732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bb406d430 msgr2=0x7f2bb415b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb3fff640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bb406d430 0x7f2bb415b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb3fff640 1 -- 172.21.15.107:0/1951964732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba40094d0 con 0x7f2bb413b0f0 2023-10-25T00:08:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb3fff640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bb413b0f0 0x7f2bb40c29f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00c770 tx=0x7f2bac00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2b9bfff640 1 -- 172.21.15.107:0/1951964732 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac003b80 con 0x7f2bb413b0f0 2023-10-25T00:08:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2b9bfff640 1 -- 172.21.15.107:0/1951964732 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bac003ce0 con 0x7f2bb413b0f0 2023-10-25T00:08:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2b9bfff640 1 -- 172.21.15.107:0/1951964732 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac00aa90 con 0x7f2bb413b0f0 2023-10-25T00:08:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1951964732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb41ee330 con 0x7f2bb413b0f0 2023-10-25T00:08:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1951964732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb40c2f30 con 0x7f2bb413b0f0 2023-10-25T00:08:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1951964732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb40746e0 con 0x7f2bb413b0f0 2023-10-25T00:08:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2b9bfff640 1 -- 172.21.15.107:0/1951964732 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2bac00abf0 con 0x7f2bb413b0f0 2023-10-25T00:08:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2b9bfff640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2ba0065e60 0x7f2ba0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2b9bfff640 1 -- 172.21.15.107:0/1951964732 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 9025+0+0 (secure 0 0 0) 0x7f2bac0841c0 con 0x7f2bb413b0f0 2023-10-25T00:08:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb37fe640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2ba0065e60 0x7f2ba0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.503+0000 7f2bb37fe640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2ba0065e60 0x7f2ba0068320 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40ae000 tx=0x7f2ba40a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:07.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.507+0000 7f2b9bfff640 1 -- 172.21.15.107:0/1951964732 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2bac050c20 con 0x7f2bb413b0f0 2023-10-25T00:08:07.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.703+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1951964732 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bb41ee7e0 con 0x7f2bb413b0f0 2023-10-25T00:08:07.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.707+0000 7f2b9bfff640 1 -- 172.21.15.107:0/1951964732 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f2bac054240 con 0x7f2bb413b0f0 2023-10-25T00:08:07.707 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:07.708 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6394,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":130}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:06:26.749209+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7107},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7107":{"gid":7107,"name":"c","rank":0,"incarnation":121,"state":"up:active","state_seq":18,"addr":"172.21.15.107:6835/913067382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":913067382},{"type":"v1","addr":"172.21.15.107:6835","nonce":913067382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:07.708 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 130 2023-10-25T00:08:07.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.707+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1951964732 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2ba0065e60 msgr2=0x7f2ba0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.707+0000 7f2bb9b96640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2ba0065e60 0x7f2ba0068320 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40ae000 tx=0x7f2ba40a7000 comp rx=0 tx=0).stop 2023-10-25T00:08:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.707+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1951964732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bb413b0f0 msgr2=0x7f2bb40c29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.707+0000 7f2bb9b96640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bb413b0f0 0x7f2bb40c29f0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00c770 tx=0x7f2bac00cc30 comp rx=0 tx=0).stop 2023-10-25T00:08:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.711+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1951964732 shutdown_connections 2023-10-25T00:08:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.711+0000 7f2bb9b96640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2bb413b0f0 0x7f2bb40c29f0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.711+0000 7f2bb9b96640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bb406d430 0x7f2bb415b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.711+0000 7f2bb9b96640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2ba0065e60 0x7f2ba0068320 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.711+0000 7f2bb9b96640 1 --2- 172.21.15.107:0/1951964732 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2bb406a7e0 0x7f2bb41eeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.711+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1951964732 >> 172.21.15.107:0/1951964732 conn(0x7f2bb40c6ed0 msgr2=0x7f2bb4135f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:07.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.711+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1951964732 shutdown_connections 2023-10-25T00:08:07.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:07.711+0000 7f2bb9b96640 1 -- 172.21.15.107:0/1951964732 wait complete. 2023-10-25T00:08:07.727 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:08:09.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:09.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.351+0000 7fdf433a1640 1 Processor -- start 2023-10-25T00:08:09.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.351+0000 7fdf433a1640 1 -- start start 2023-10-25T00:08:09.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf433a1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdf3c0a9820 0x7fdf3c0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf433a1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf3c06ee40 0x7fdf3c0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf433a1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3c06f730 con 0x7fdf3c14fd40 2023-10-25T00:08:09.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf433a1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3c0aa750 con 0x7fdf3c06ee40 2023-10-25T00:08:09.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf433a1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3c156a40 con 0x7fdf3c0a9820 2023-10-25T00:08:09.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf4193e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf3c06ee40 0x7fdf3c0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf4193e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdf3c06ee40 0x7fdf3c0aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46836/0 (socket says 172.21.15.107:46836) 2023-10-25T00:08:09.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf4193e640 1 -- 172.21.15.107:0/1479164488 learned_addr learned my addr 172.21.15.107:0/1479164488 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:09.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf4093c640 1 --2- 172.21.15.107:0/1479164488 >> v2:172.21.15.110:3300/0 conn(0x7fdf3c0a9820 0x7fdf3c0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf4193e640 1 -- 172.21.15.107:0/1479164488 >> v2:172.21.15.110:3300/0 conn(0x7fdf3c0a9820 msgr2=0x7fdf3c0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:09.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf4193e640 1 --2- 172.21.15.107:0/1479164488 >> v2:172.21.15.110:3300/0 conn(0x7fdf3c0a9820 0x7fdf3c0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf4193e640 1 -- 172.21.15.107:0/1479164488 >> v1:172.21.15.110:6789/0 conn(0x7fdf3c14fd40 legacy=0x7fdf3c0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:09.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf4193e640 1 -- 172.21.15.107:0/1479164488 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c156710 con 0x7fdf3c06ee40 2023-10-25T00:08:09.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf4193e640 1 --2- 172.21.15.107:0/1479164488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c06ee40 0x7fdf3c0aa210 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c06d2f0 tx=0x7fdf3401c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c3ed5beb1c1f6e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:09.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf33fff640 1 -- 172.21.15.107:0/1479164488 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf340ac640 con 0x7fdf3c06ee40 2023-10-25T00:08:09.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf33fff640 1 -- 172.21.15.107:0/1479164488 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf340ac7a0 con 0x7fdf3c06ee40 2023-10-25T00:08:09.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.355+0000 7fdf33fff640 1 -- 172.21.15.107:0/1479164488 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf340acb30 con 0x7fdf3c06ee40 2023-10-25T00:08:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 -- 172.21.15.107:0/1479164488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c06ee40 msgr2=0x7fdf3c0aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 --2- 172.21.15.107:0/1479164488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c06ee40 0x7fdf3c0aa210 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c06d2f0 tx=0x7fdf3401c370 comp rx=0 tx=0).stop 2023-10-25T00:08:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 -- 172.21.15.107:0/1479164488 shutdown_connections 2023-10-25T00:08:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 --2- 172.21.15.107:0/1479164488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c06ee40 0x7fdf3c0aa210 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 --2- 172.21.15.107:0/1479164488 >> v2:172.21.15.110:3300/0 conn(0x7fdf3c0a9820 0x7fdf3c0ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 -- 172.21.15.107:0/1479164488 >> 172.21.15.107:0/1479164488 conn(0x7fdf3c0c2950 msgr2=0x7fdf3c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:09.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 -- 172.21.15.107:0/1479164488 shutdown_connections 2023-10-25T00:08:09.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 -- 172.21.15.107:0/1479164488 wait complete. 2023-10-25T00:08:09.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 Processor -- start 2023-10-25T00:08:09.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 -- start start 2023-10-25T00:08:09.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c06ee40 0x7fdf3c1ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c14fd40 0x7fdf3c1580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf3c14cf10 0x7fdf3c13c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c157940 con 0x7fdf3c14fd40 2023-10-25T00:08:09.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c14e320 con 0x7fdf3c06ee40 2023-10-25T00:08:09.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c15bc20 con 0x7fdf3c14cf10 2023-10-25T00:08:09.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4093c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c14fd40 0x7fdf3c1580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4093c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c14fd40 0x7fdf3c1580b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46848/0 (socket says 172.21.15.107:46848) 2023-10-25T00:08:09.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4093c640 1 -- 172.21.15.107:0/2626442982 learned_addr learned my addr 172.21.15.107:0/2626442982 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:09.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4113d640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c06ee40 0x7fdf3c1ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4193e640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf3c14cf10 0x7fdf3c13c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4093c640 1 -- 172.21.15.107:0/2626442982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf3c14cf10 msgr2=0x7fdf3c13c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4093c640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf3c14cf10 0x7fdf3c13c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4093c640 1 -- 172.21.15.107:0/2626442982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c06ee40 msgr2=0x7fdf3c1ebe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4093c640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c06ee40 0x7fdf3c1ebe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4093c640 1 -- 172.21.15.107:0/2626442982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf340094d0 con 0x7fdf3c14fd40 2023-10-25T00:08:09.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4113d640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c06ee40 0x7fdf3c1ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:09.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4193e640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf3c14cf10 0x7fdf3c13c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:09.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf4093c640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c14fd40 0x7fdf3c1580b0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00d800 tx=0x7fdf2c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:09.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf31ffb640 1 -- 172.21.15.107:0/2626442982 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c00b6c0 con 0x7fdf3c14fd40 2023-10-25T00:08:09.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf31ffb640 1 -- 172.21.15.107:0/2626442982 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2c00b820 con 0x7fdf3c14fd40 2023-10-25T00:08:09.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 -- 172.21.15.107:0/2626442982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3c14e5c0 con 0x7fdf3c14fd40 2023-10-25T00:08:09.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf31ffb640 1 -- 172.21.15.107:0/2626442982 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c00bb90 con 0x7fdf3c14fd40 2023-10-25T00:08:09.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.359+0000 7fdf433a1640 1 -- 172.21.15.107:0/2626442982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3c157b90 con 0x7fdf3c14fd40 2023-10-25T00:08:09.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.363+0000 7fdf433a1640 1 -- 172.21.15.107:0/2626442982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3c0c24a0 con 0x7fdf3c14fd40 2023-10-25T00:08:09.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.363+0000 7fdf31ffb640 1 -- 172.21.15.107:0/2626442982 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdf2c00bcf0 con 0x7fdf3c14fd40 2023-10-25T00:08:09.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.363+0000 7fdf31ffb640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e60 0x7fdf20068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.363+0000 7fdf31ffb640 1 -- 172.21.15.107:0/2626442982 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 9111+0+0 (secure 0 0 0) 0x7fdf2c084980 con 0x7fdf3c14fd40 2023-10-25T00:08:09.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.363+0000 7fdf4113d640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e60 0x7fdf20068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.363+0000 7fdf4113d640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e60 0x7fdf20068320 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdf38007fd0 tx=0x7fdf38009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:09.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.367+0000 7fdf31ffb640 1 -- 172.21.15.107:0/2626442982 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdf2c051420 con 0x7fdf3c14fd40 2023-10-25T00:08:09.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.555+0000 7fdf433a1640 1 -- 172.21.15.107:0/2626442982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf3c070840 con 0x7fdf3c14fd40 2023-10-25T00:08:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.555+0000 7fdf31ffb640 1 -- 172.21.15.107:0/2626442982 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+5019 (secure 0 0 0) 0x7fdf2c011340 con 0x7fdf3c14fd40 2023-10-25T00:08:09.556 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:09.556 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:08.676800+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:09.556 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 131 2023-10-25T00:08:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 -- 172.21.15.107:0/2626442982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e60 msgr2=0x7fdf20068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e60 0x7fdf20068320 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdf38007fd0 tx=0x7fdf38009040 comp rx=0 tx=0).stop 2023-10-25T00:08:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 -- 172.21.15.107:0/2626442982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c14fd40 msgr2=0x7fdf3c1580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c14fd40 0x7fdf3c1580b0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00d800 tx=0x7fdf2c00dcc0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 -- 172.21.15.107:0/2626442982 shutdown_connections 2023-10-25T00:08:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e60 0x7fdf20068320 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf3c14cf10 0x7fdf3c13c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c14fd40 0x7fdf3c1580b0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 --2- 172.21.15.107:0/2626442982 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf3c06ee40 0x7fdf3c1ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 -- 172.21.15.107:0/2626442982 >> 172.21.15.107:0/2626442982 conn(0x7fdf3c0c2950 msgr2=0x7fdf3c14d530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:09.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 -- 172.21.15.107:0/2626442982 shutdown_connections 2023-10-25T00:08:09.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.559+0000 7fdf433a1640 1 -- 172.21.15.107:0/2626442982 wait complete. 2023-10-25T00:08:09.574 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2023-10-25T00:08:09.575 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:08:09.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:09.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6de4c5f640 1 Processor -- start 2023-10-25T00:08:09.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6de4c5f640 1 -- start start 2023-10-25T00:08:09.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6de4c5f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6de0152040 0x7f6de00c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6de4c5f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6de00c8100 0x7f6de00abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6de4c5f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de0155680 con 0x7f6de006a4b0 2023-10-25T00:08:09.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6de4c5f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de0154a70 con 0x7f6de0152040 2023-10-25T00:08:09.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6de4c5f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de0154d50 con 0x7f6de00c8100 2023-10-25T00:08:09.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6dde59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6de0152040 0x7f6de00c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6dded9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6de006a4b0 0x7f6de00ac240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54470/0 (socket says 172.21.15.107:54470) 2023-10-25T00:08:09.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6dded9d640 1 -- 172.21.15.107:0/1077666302 learned_addr learned my addr 172.21.15.107:0/1077666302 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:09.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6dddd9b640 1 --2- 172.21.15.107:0/1077666302 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6de00c8100 0x7f6de00abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6dde59c640 1 -- 172.21.15.107:0/1077666302 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6de00c8100 msgr2=0x7f6de00abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:09.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6ddd59a640 1 -- 172.21.15.107:0/1077666302 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573732376 0 0) 0x7f6de0155680 con 0x7f6de006a4b0 2023-10-25T00:08:09.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6dde59c640 1 --2- 172.21.15.107:0/1077666302 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6de00c8100 0x7f6de00abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6dde59c640 1 -- 172.21.15.107:0/1077666302 >> v1:172.21.15.107:6789/0 conn(0x7f6de006a4b0 legacy=0x7f6de00ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6dde59c640 1 -- 172.21.15.107:0/1077666302 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de0155030 con 0x7f6de0152040 2023-10-25T00:08:09.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.687+0000 7f6dddd9b640 1 --2- 172.21.15.107:0/1077666302 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6de00c8100 0x7f6de00abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:09.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dde59c640 1 --2- 172.21.15.107:0/1077666302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0152040 0x7f6de00c4b80 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00097c0 tx=0x7f6dd001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9fe843d01414333 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:09.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6ddd59a640 1 -- 172.21.15.107:0/1077666302 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00ab600 con 0x7f6de0152040 2023-10-25T00:08:09.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6ddd59a640 1 -- 172.21.15.107:0/1077666302 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd00a7050 con 0x7f6de0152040 2023-10-25T00:08:09.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6ddd59a640 1 -- 172.21.15.107:0/1077666302 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00ab980 con 0x7f6de0152040 2023-10-25T00:08:09.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 -- 172.21.15.107:0/1077666302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0152040 msgr2=0x7f6de00c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 --2- 172.21.15.107:0/1077666302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0152040 0x7f6de00c4b80 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00097c0 tx=0x7f6dd001c3f0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 -- 172.21.15.107:0/1077666302 shutdown_connections 2023-10-25T00:08:09.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 --2- 172.21.15.107:0/1077666302 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6de00c8100 0x7f6de00abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 --2- 172.21.15.107:0/1077666302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de0152040 0x7f6de00c4b80 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 -- 172.21.15.107:0/1077666302 >> 172.21.15.107:0/1077666302 conn(0x7f6de00ca200 msgr2=0x7f6de00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:09.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 -- 172.21.15.107:0/1077666302 shutdown_connections 2023-10-25T00:08:09.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 -- 172.21.15.107:0/1077666302 wait complete. 2023-10-25T00:08:09.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 Processor -- start 2023-10-25T00:08:09.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 -- start start 2023-10-25T00:08:09.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de006a4b0 0x7f6de01426f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de00c8100 0x7f6de014cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6de0152040 0x7f6de014a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de0142c30 con 0x7f6de006a4b0 2023-10-25T00:08:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de014c3c0 con 0x7f6de00c8100 2023-10-25T00:08:09.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de0153fe0 con 0x7f6de0152040 2023-10-25T00:08:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dde59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de006a4b0 0x7f6de01426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dde59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de006a4b0 0x7f6de01426f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46866/0 (socket says 172.21.15.107:46866) 2023-10-25T00:08:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dde59c640 1 -- 172.21.15.107:0/2211438798 learned_addr learned my addr 172.21.15.107:0/2211438798 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dde59c640 1 -- 172.21.15.107:0/2211438798 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6de0152040 msgr2=0x7f6de014a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dde59c640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6de0152040 0x7f6de014a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dde59c640 1 -- 172.21.15.107:0/2211438798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de00c8100 msgr2=0x7f6de014cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dde59c640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de00c8100 0x7f6de014cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dde59c640 1 -- 172.21.15.107:0/2211438798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd0009420 con 0x7f6de006a4b0 2023-10-25T00:08:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dde59c640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de006a4b0 0x7f6de01426f0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00ae040 tx=0x7f6dd001ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:09.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dceffd640 1 -- 172.21.15.107:0/2211438798 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00b7650 con 0x7f6de006a4b0 2023-10-25T00:08:09.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dceffd640 1 -- 172.21.15.107:0/2211438798 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd00b77b0 con 0x7f6de006a4b0 2023-10-25T00:08:09.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 -- 172.21.15.107:0/2211438798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de014c620 con 0x7f6de006a4b0 2023-10-25T00:08:09.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6dceffd640 1 -- 172.21.15.107:0/2211438798 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00b7b60 con 0x7f6de006a4b0 2023-10-25T00:08:09.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.691+0000 7f6de4c5f640 1 -- 172.21.15.107:0/2211438798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de0154250 con 0x7f6de006a4b0 2023-10-25T00:08:09.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.695+0000 7f6de4c5f640 1 -- 172.21.15.107:0/2211438798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de00746e0 con 0x7f6de006a4b0 2023-10-25T00:08:09.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.695+0000 7f6dceffd640 1 -- 172.21.15.107:0/2211438798 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6dd00b7cc0 con 0x7f6de006a4b0 2023-10-25T00:08:09.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.695+0000 7f6dceffd640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dc0065e60 0x7f6dc0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.695+0000 7f6dceffd640 1 -- 172.21.15.107:0/2211438798 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 9111+0+0 (secure 0 0 0) 0x7f6dd011ee30 con 0x7f6de006a4b0 2023-10-25T00:08:09.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.695+0000 7f6dddd9b640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dc0065e60 0x7f6dc0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.695+0000 7f6dddd9b640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dc0065e60 0x7f6dc0068320 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6de00c4b00 tx=0x7f6dd4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:09.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.699+0000 7f6dceffd640 1 -- 172.21.15.107:0/2211438798 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6dd00eb960 con 0x7f6de006a4b0 2023-10-25T00:08:09.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:09.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.839+0000 7ff019e9a640 1 Processor -- start 2023-10-25T00:08:09.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.839+0000 7ff019e9a640 1 -- start start 2023-10-25T00:08:09.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff019e9a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff014156770 0x7ff0140ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff019e9a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff0140be6b0 0x7ff01415d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff019e9a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0140c1ca0 con 0x7ff0140c70e0 2023-10-25T00:08:09.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff019e9a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0140c27e0 con 0x7ff0140be6b0 2023-10-25T00:08:09.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff019e9a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0140c10c0 con 0x7ff014156770 2023-10-25T00:08:09.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff013fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff0140be6b0 0x7ff01415d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff013fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff0140be6b0 0x7ff01415d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46872/0 (socket says 172.21.15.107:46872) 2023-10-25T00:08:09.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff013fff640 1 -- 172.21.15.107:0/2577807112 learned_addr learned my addr 172.21.15.107:0/2577807112 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:09.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff0137fe640 1 --2- 172.21.15.107:0/2577807112 >> v2:172.21.15.110:3300/0 conn(0x7ff014156770 0x7ff0140ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff013fff640 1 -- 172.21.15.107:0/2577807112 >> v2:172.21.15.110:3300/0 conn(0x7ff014156770 msgr2=0x7ff0140ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:09.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff013fff640 1 --2- 172.21.15.107:0/2577807112 >> v2:172.21.15.110:3300/0 conn(0x7ff014156770 0x7ff0140ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff013fff640 1 -- 172.21.15.107:0/2577807112 >> v1:172.21.15.110:6789/0 conn(0x7ff0140c70e0 legacy=0x7ff0140bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:09.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff013fff640 1 -- 172.21.15.107:0/2577807112 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0140c13a0 con 0x7ff0140be6b0 2023-10-25T00:08:09.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff013fff640 1 --2- 172.21.15.107:0/2577807112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0140be6b0 0x7ff01415d130 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7ff0140ac6f0 tx=0x7ff00c01c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4393b94e20c44b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff0127fc640 1 -- 172.21.15.107:0/2577807112 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c0a56c0 con 0x7ff0140be6b0 2023-10-25T00:08:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff0127fc640 1 -- 172.21.15.107:0/2577807112 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00c0a5820 con 0x7ff0140be6b0 2023-10-25T00:08:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.843+0000 7ff0127fc640 1 -- 172.21.15.107:0/2577807112 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c0a5bb0 con 0x7ff0140be6b0 2023-10-25T00:08:09.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 -- 172.21.15.107:0/2577807112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0140be6b0 msgr2=0x7ff01415d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 --2- 172.21.15.107:0/2577807112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0140be6b0 0x7ff01415d130 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7ff0140ac6f0 tx=0x7ff00c01c840 comp rx=0 tx=0).stop 2023-10-25T00:08:09.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 -- 172.21.15.107:0/2577807112 shutdown_connections 2023-10-25T00:08:09.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 --2- 172.21.15.107:0/2577807112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0140be6b0 0x7ff01415d130 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 --2- 172.21.15.107:0/2577807112 >> v2:172.21.15.110:3300/0 conn(0x7ff014156770 0x7ff0140ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 -- 172.21.15.107:0/2577807112 >> 172.21.15.107:0/2577807112 conn(0x7ff0140c65c0 msgr2=0x7ff0140c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:09.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 -- 172.21.15.107:0/2577807112 shutdown_connections 2023-10-25T00:08:09.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 -- 172.21.15.107:0/2577807112 wait complete. 2023-10-25T00:08:09.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 Processor -- start 2023-10-25T00:08:09.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 -- start start 2023-10-25T00:08:09.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0140be6b0 0x7ff0140c4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0140c70e0 0x7ff01414dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff014156770 0x7ff01414e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0141f1650 con 0x7ff0140c70e0 2023-10-25T00:08:09.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01406ff00 con 0x7ff0140be6b0 2023-10-25T00:08:09.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff019e9a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0140ab650 con 0x7ff014156770 2023-10-25T00:08:09.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff012ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0140c70e0 0x7ff01414dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff012ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0140c70e0 0x7ff01414dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46888/0 (socket says 172.21.15.107:46888) 2023-10-25T00:08:09.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff012ffd640 1 -- 172.21.15.107:0/756223994 learned_addr learned my addr 172.21.15.107:0/756223994 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:09.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff0137fe640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0140be6b0 0x7ff0140c4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff013fff640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff014156770 0x7ff01414e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff012ffd640 1 -- 172.21.15.107:0/756223994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff014156770 msgr2=0x7ff01414e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff012ffd640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff014156770 0x7ff01414e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff012ffd640 1 -- 172.21.15.107:0/756223994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0140be6b0 msgr2=0x7ff0140c4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff012ffd640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0140be6b0 0x7ff0140c4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff012ffd640 1 -- 172.21.15.107:0/756223994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c009b30 con 0x7ff0140c70e0 2023-10-25T00:08:09.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.847+0000 7ff012ffd640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0140c70e0 0x7ff01414dfb0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff00400c450 tx=0x7ff00400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:09.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.851+0000 7feffbfff640 1 -- 172.21.15.107:0/756223994 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff004011790 con 0x7ff0140c70e0 2023-10-25T00:08:09.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.851+0000 7feffbfff640 1 -- 172.21.15.107:0/756223994 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff004011910 con 0x7ff0140c70e0 2023-10-25T00:08:09.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.851+0000 7feffbfff640 1 -- 172.21.15.107:0/756223994 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff004011cc0 con 0x7ff0140c70e0 2023-10-25T00:08:09.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.851+0000 7ff019e9a640 1 -- 172.21.15.107:0/756223994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0141ed160 con 0x7ff0140c70e0 2023-10-25T00:08:09.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.851+0000 7ff019e9a640 1 -- 172.21.15.107:0/756223994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01415cfd0 con 0x7ff0140c70e0 2023-10-25T00:08:09.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.851+0000 7ff019e9a640 1 -- 172.21.15.107:0/756223994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0141ed380 con 0x7ff0140c70e0 2023-10-25T00:08:09.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.851+0000 7feffbfff640 1 -- 172.21.15.107:0/756223994 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff00401c460 con 0x7ff0140c70e0 2023-10-25T00:08:09.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.851+0000 7feffbfff640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff000065e60 0x7ff000068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:09.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.851+0000 7feffbfff640 1 -- 172.21.15.107:0/756223994 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 9111+0+0 (secure 0 0 0) 0x7ff004084cf0 con 0x7ff0140c70e0 2023-10-25T00:08:09.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.851+0000 7ff0137fe640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff000065e60 0x7ff000068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:09.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.851+0000 7ff0137fe640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff000065e60 0x7ff000068320 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff0140c0090 tx=0x7ff008008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:09.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.855+0000 7feffbfff640 1 -- 172.21.15.107:0/756223994 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff004051780 con 0x7ff0140c70e0 2023-10-25T00:08:09.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 -- 172.21.15.107:0/2211438798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6de00c1000 con 0x7f6de006a4b0 2023-10-25T00:08:09.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6dceffd640 1 -- 172.21.15.107:0/2211438798 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5022 (secure 0 0 0) 0x7f6dd00b0070 con 0x7f6de006a4b0 2023-10-25T00:08:09.883 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:09.883 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:09.712335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 132 2023-10-25T00:08:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 -- 172.21.15.107:0/2211438798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dc0065e60 msgr2=0x7f6dc0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dc0065e60 0x7f6dc0068320 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6de00c4b00 tx=0x7f6dd4008040 comp rx=0 tx=0).stop 2023-10-25T00:08:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 -- 172.21.15.107:0/2211438798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de006a4b0 msgr2=0x7f6de01426f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:09.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de006a4b0 0x7f6de01426f0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00ae040 tx=0x7f6dd001ac30 comp rx=0 tx=0).stop 2023-10-25T00:08:09.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 -- 172.21.15.107:0/2211438798 shutdown_connections 2023-10-25T00:08:09.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dc0065e60 0x7f6dc0068320 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6de0152040 0x7f6de014a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de00c8100 0x7f6de014cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 --2- 172.21.15.107:0/2211438798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6de006a4b0 0x7f6de01426f0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:09.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 -- 172.21.15.107:0/2211438798 >> 172.21.15.107:0/2211438798 conn(0x7f6de00ca200 msgr2=0x7f6de00ada00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:09.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 -- 172.21.15.107:0/2211438798 shutdown_connections 2023-10-25T00:08:09.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:09.883+0000 7f6de4c5f640 1 -- 172.21.15.107:0/2211438798 wait complete. 2023-10-25T00:08:09.901 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 132, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 132, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6394": { "addr": "172.21.15.110:6835/3836115666", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 3836115666, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 3836115666, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6394, "incarnation": 131, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:08:09.712335+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6394 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 129, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7305": { "addr": "172.21.15.107:6837/1317796403", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 1317796403, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 1317796403, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7305, "incarnation": 126, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:07:51.732943+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7305 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:08:10.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.039+0000 7ff019e9a640 1 -- 172.21.15.107:0/756223994 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0141ed350 con 0x7ff0140c70e0 2023-10-25T00:08:10.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.039+0000 7feffbfff640 1 -- 172.21.15.107:0/756223994 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5022 (secure 0 0 0) 0x7ff004011e20 con 0x7ff0140c70e0 2023-10-25T00:08:10.041 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:10.041 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:09.712335+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:10.042 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 132 2023-10-25T00:08:10.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 -- 172.21.15.107:0/756223994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff000065e60 msgr2=0x7ff000068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:10.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff000065e60 0x7ff000068320 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff0140c0090 tx=0x7ff008008040 comp rx=0 tx=0).stop 2023-10-25T00:08:10.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 -- 172.21.15.107:0/756223994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0140c70e0 msgr2=0x7ff01414dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:10.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0140c70e0 0x7ff01414dfb0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7ff00400c450 tx=0x7ff00400c910 comp rx=0 tx=0).stop 2023-10-25T00:08:10.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 -- 172.21.15.107:0/756223994 shutdown_connections 2023-10-25T00:08:10.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff000065e60 0x7ff000068320 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:10.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff014156770 0x7ff01414e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:10.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff0140c70e0 0x7ff01414dfb0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:10.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 --2- 172.21.15.107:0/756223994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff0140be6b0 0x7ff0140c4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:10.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 -- 172.21.15.107:0/756223994 >> 172.21.15.107:0/756223994 conn(0x7ff0140c65c0 msgr2=0x7ff0141546f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:10.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 -- 172.21.15.107:0/756223994 shutdown_connections 2023-10-25T00:08:10.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:10.043+0000 7ff019e9a640 1 -- 172.21.15.107:0/756223994 wait complete. 2023-10-25T00:08:10.059 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:08:11.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:12.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.039+0000 7fe0db9fd640 1 Processor -- start 2023-10-25T00:08:12.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.039+0000 7fe0db9fd640 1 -- start start 2023-10-25T00:08:12.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0db9fd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0d406af70 0x7fe0d406c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0db9fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d406a230 0x7fe0d41551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0db9fd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d40c1860 con 0x7fe0d414b600 2023-10-25T00:08:12.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0db9fd640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d4154d50 con 0x7fe0d406af70 2023-10-25T00:08:12.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0db9fd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d414b9e0 con 0x7fe0d406a230 2023-10-25T00:08:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0d8f98640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0d406af70 0x7fe0d406c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0d8f98640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0d406af70 0x7fe0d406c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46902/0 (socket says 172.21.15.107:46902) 2023-10-25T00:08:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0d8f98640 1 -- 172.21.15.107:0/837098019 learned_addr learned my addr 172.21.15.107:0/837098019 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0d9f9a640 1 --2- 172.21.15.107:0/837098019 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d406a230 0x7fe0d41551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0d8f98640 1 -- 172.21.15.107:0/837098019 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d406a230 msgr2=0x7fe0d41551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0d8f98640 1 --2- 172.21.15.107:0/837098019 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d406a230 0x7fe0d41551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0d8f98640 1 -- 172.21.15.107:0/837098019 >> v1:172.21.15.110:6789/0 conn(0x7fe0d414b600 legacy=0x7fe0d4151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:12.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0d8f98640 1 -- 172.21.15.107:0/837098019 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d40c1ab0 con 0x7fe0d406af70 2023-10-25T00:08:12.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0d8f98640 1 --2- 172.21.15.107:0/837098019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d406af70 0x7fe0d406c4c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4009870 tx=0x7fe0c401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5758b4d5242ee669 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:12.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0cbfff640 1 -- 172.21.15.107:0/837098019 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c40ab600 con 0x7fe0d406af70 2023-10-25T00:08:12.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0cbfff640 1 -- 172.21.15.107:0/837098019 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c40ab760 con 0x7fe0d406af70 2023-10-25T00:08:12.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0cbfff640 1 -- 172.21.15.107:0/837098019 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c40abaf0 con 0x7fe0d406af70 2023-10-25T00:08:12.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0db9fd640 1 -- 172.21.15.107:0/837098019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d406af70 msgr2=0x7fe0d406c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:12.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.043+0000 7fe0db9fd640 1 --2- 172.21.15.107:0/837098019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d406af70 0x7fe0d406c4c0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe0c4009870 tx=0x7fe0c401c440 comp rx=0 tx=0).stop 2023-10-25T00:08:12.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 -- 172.21.15.107:0/837098019 shutdown_connections 2023-10-25T00:08:12.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 --2- 172.21.15.107:0/837098019 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d406a230 0x7fe0d41551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 --2- 172.21.15.107:0/837098019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d406af70 0x7fe0d406c4c0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 -- 172.21.15.107:0/837098019 >> 172.21.15.107:0/837098019 conn(0x7fe0d40c2950 msgr2=0x7fe0d40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:12.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 -- 172.21.15.107:0/837098019 shutdown_connections 2023-10-25T00:08:12.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 -- 172.21.15.107:0/837098019 wait complete. 2023-10-25T00:08:12.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 Processor -- start 2023-10-25T00:08:12.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 -- start start 2023-10-25T00:08:12.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d406a230 0x7fe0d41eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d406af70 0x7fe0d41ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d414b600 0x7fe0d41568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d415abb0 con 0x7fe0d406a230 2023-10-25T00:08:12.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d4155e80 con 0x7fe0d406af70 2023-10-25T00:08:12.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d4158050 con 0x7fe0d414b600 2023-10-25T00:08:12.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0d9799640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d406a230 0x7fe0d41eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0d9799640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d406a230 0x7fe0d41eeb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46916/0 (socket says 172.21.15.107:46916) 2023-10-25T00:08:12.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0d9799640 1 -- 172.21.15.107:0/3912043293 learned_addr learned my addr 172.21.15.107:0/3912043293 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:12.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0d8f98640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d406af70 0x7fe0d41ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0d9f9a640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d414b600 0x7fe0d41568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0d9799640 1 -- 172.21.15.107:0/3912043293 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d414b600 msgr2=0x7fe0d41568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:12.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0d9799640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d414b600 0x7fe0d41568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0d9799640 1 -- 172.21.15.107:0/3912043293 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d406af70 msgr2=0x7fe0d41ee2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:12.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0d9799640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d406af70 0x7fe0d41ee2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0d9799640 1 -- 172.21.15.107:0/3912043293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c40094d0 con 0x7fe0d406a230 2023-10-25T00:08:12.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0d9799640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d406a230 0x7fe0d41eeb40 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00ba50 tx=0x7fe0cc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:12.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0c9ffb640 1 -- 172.21.15.107:0/3912043293 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc00cb80 con 0x7fe0d406a230 2023-10-25T00:08:12.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0c9ffb640 1 -- 172.21.15.107:0/3912043293 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc00cd20 con 0x7fe0d406a230 2023-10-25T00:08:12.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 -- 172.21.15.107:0/3912043293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d4154620 con 0x7fe0d406a230 2023-10-25T00:08:12.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0c9ffb640 1 -- 172.21.15.107:0/3912043293 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc014680 con 0x7fe0d406a230 2023-10-25T00:08:12.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.047+0000 7fe0db9fd640 1 -- 172.21.15.107:0/3912043293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d41560d0 con 0x7fe0d406a230 2023-10-25T00:08:12.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.051+0000 7fe0db9fd640 1 -- 172.21.15.107:0/3912043293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d40746e0 con 0x7fe0d406a230 2023-10-25T00:08:12.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.051+0000 7fe0c9ffb640 1 -- 172.21.15.107:0/3912043293 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe0cc0147e0 con 0x7fe0d406a230 2023-10-25T00:08:12.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.051+0000 7fe0c9ffb640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065eb0 0x7fe0b8068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.051+0000 7fe0c9ffb640 1 -- 172.21.15.107:0/3912043293 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 9111+0+0 (secure 0 0 0) 0x7fe0cc084220 con 0x7fe0d406a230 2023-10-25T00:08:12.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.051+0000 7fe0d8f98640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065eb0 0x7fe0b8068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.051+0000 7fe0d8f98640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065eb0 0x7fe0b8068370 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe0c40af000 tx=0x7fe0c40b6040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:12.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.055+0000 7fe0c9ffb640 1 -- 172.21.15.107:0/3912043293 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe0cc050df0 con 0x7fe0d406a230 2023-10-25T00:08:12.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:12.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a86f2640 1 Processor -- start 2023-10-25T00:08:12.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a86f2640 1 -- start start 2023-10-25T00:08:12.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a86f2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20a00c70e0 0x7f20a00bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a86f2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f20a00c2130 0x7f20a00be4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a86f2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a00c1ca0 con 0x7f20a0156770 2023-10-25T00:08:12.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a86f2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a00c27e0 con 0x7f20a00c70e0 2023-10-25T00:08:12.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a86f2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a00c10c0 con 0x7f20a00c2130 2023-10-25T00:08:12.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a648e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f20a0156770 0x7f20a00ab7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54480/0 (socket says 172.21.15.107:54480) 2023-10-25T00:08:12.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a648e640 1 -- 172.21.15.107:0/1169037559 learned_addr learned my addr 172.21.15.107:0/1169037559 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:12.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a5c8d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f20a00c70e0 0x7f20a00bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a6c8f640 1 --2- 172.21.15.107:0/1169037559 >> v2:172.21.15.110:3300/0 conn(0x7f20a00c2130 0x7f20a00be4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a5c8d640 1 -- 172.21.15.107:0/1169037559 >> v2:172.21.15.110:3300/0 conn(0x7f20a00c2130 msgr2=0x7f20a00be4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a5c8d640 1 --2- 172.21.15.107:0/1169037559 >> v2:172.21.15.110:3300/0 conn(0x7f20a00c2130 0x7f20a00be4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a5c8d640 1 -- 172.21.15.107:0/1169037559 >> v1:172.21.15.107:6789/0 conn(0x7f20a0156770 legacy=0x7f20a00ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a548c640 1 -- 172.21.15.107:0/1169037559 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109479434 0 0) 0x7f20a00c1ca0 con 0x7f20a0156770 2023-10-25T00:08:12.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.183+0000 7f20a5c8d640 1 -- 172.21.15.107:0/1169037559 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a00c13a0 con 0x7f20a00c70e0 2023-10-25T00:08:12.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a5c8d640 1 --2- 172.21.15.107:0/1169037559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a00c70e0 0x7f20a00bdfa0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f20a00c2fc0 tx=0x7f209001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa2f86a783daea1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:12.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a548c640 1 -- 172.21.15.107:0/1169037559 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900ab600 con 0x7f20a00c70e0 2023-10-25T00:08:12.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a548c640 1 -- 172.21.15.107:0/1169037559 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20900a7050 con 0x7f20a00c70e0 2023-10-25T00:08:12.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a548c640 1 -- 172.21.15.107:0/1169037559 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900ab980 con 0x7f20a00c70e0 2023-10-25T00:08:12.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a86f2640 1 -- 172.21.15.107:0/1169037559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a00c70e0 msgr2=0x7f20a00bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:12.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a86f2640 1 --2- 172.21.15.107:0/1169037559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a00c70e0 0x7f20a00bdfa0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f20a00c2fc0 tx=0x7f209001c340 comp rx=0 tx=0).stop 2023-10-25T00:08:12.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a86f2640 1 -- 172.21.15.107:0/1169037559 shutdown_connections 2023-10-25T00:08:12.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a86f2640 1 --2- 172.21.15.107:0/1169037559 >> v2:172.21.15.110:3300/0 conn(0x7f20a00c2130 0x7f20a00be4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a86f2640 1 --2- 172.21.15.107:0/1169037559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a00c70e0 0x7f20a00bdfa0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a86f2640 1 -- 172.21.15.107:0/1169037559 >> 172.21.15.107:0/1169037559 conn(0x7f20a00c65c0 msgr2=0x7f20a00c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:12.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a86f2640 1 -- 172.21.15.107:0/1169037559 shutdown_connections 2023-10-25T00:08:12.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a86f2640 1 -- 172.21.15.107:0/1169037559 wait complete. 2023-10-25T00:08:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a86f2640 1 Processor -- start 2023-10-25T00:08:12.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.187+0000 7f20a86f2640 1 -- start start 2023-10-25T00:08:12.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a86f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20a00c2130 0x7f20a01588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a86f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a00c70e0 0x7f20a0157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a86f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20a0156770 0x7f20a01f0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a86f2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a01f0dd0 con 0x7f20a00c70e0 2023-10-25T00:08:12.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a86f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a0157100 con 0x7f20a00c2130 2023-10-25T00:08:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a86f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a01579a0 con 0x7f20a0156770 2023-10-25T00:08:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a5c8d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a00c70e0 0x7f20a0157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a5c8d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a00c70e0 0x7f20a0157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46936/0 (socket says 172.21.15.107:46936) 2023-10-25T00:08:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a5c8d640 1 -- 172.21.15.107:0/4016303260 learned_addr learned my addr 172.21.15.107:0/4016303260 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:12.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a648e640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20a00c2130 0x7f20a01588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a6c8f640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20a0156770 0x7f20a01f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a5c8d640 1 -- 172.21.15.107:0/4016303260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20a0156770 msgr2=0x7f20a01f0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a5c8d640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20a0156770 0x7f20a01f0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a5c8d640 1 -- 172.21.15.107:0/4016303260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20a00c2130 msgr2=0x7f20a01588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:12.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a5c8d640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20a00c2130 0x7f20a01588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a5c8d640 1 -- 172.21.15.107:0/4016303260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2090009420 con 0x7f20a00c70e0 2023-10-25T00:08:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a5c8d640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a00c70e0 0x7f20a0157e30 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f20900ae040 tx=0x7f20900190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f2096ffd640 1 -- 172.21.15.107:0/4016303260 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900b4840 con 0x7f20a00c70e0 2023-10-25T00:08:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f2096ffd640 1 -- 172.21.15.107:0/4016303260 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20900b49a0 con 0x7f20a00c70e0 2023-10-25T00:08:12.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a86f2640 1 -- 172.21.15.107:0/4016303260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a0158ea0 con 0x7f20a00c70e0 2023-10-25T00:08:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f2096ffd640 1 -- 172.21.15.107:0/4016303260 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900b4d50 con 0x7f20a00c70e0 2023-10-25T00:08:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a86f2640 1 -- 172.21.15.107:0/4016303260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a0157630 con 0x7f20a00c70e0 2023-10-25T00:08:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f20a86f2640 1 -- 172.21.15.107:0/4016303260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a00746e0 con 0x7f20a00c70e0 2023-10-25T00:08:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.191+0000 7f2096ffd640 1 -- 172.21.15.107:0/4016303260 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f20900b2040 con 0x7f20a00c70e0 2023-10-25T00:08:12.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.195+0000 7f2096ffd640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2084065e60 0x7f2084068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:12.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.195+0000 7f20a648e640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2084065e60 0x7f2084068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:12.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.195+0000 7f2096ffd640 1 -- 172.21.15.107:0/4016303260 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 9111+0+0 (secure 0 0 0) 0x7f209011ebd0 con 0x7f20a00c70e0 2023-10-25T00:08:12.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.195+0000 7f20a648e640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2084065e60 0x7f2084068320 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f209c000f70 tx=0x7f209c007c10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:12.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.199+0000 7f2096ffd640 1 -- 172.21.15.107:0/4016303260 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f20900eb810 con 0x7f20a00c70e0 2023-10-25T00:08:12.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.295+0000 7fe0db9fd640 1 -- 172.21.15.107:0/3912043293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0d4069fa0 con 0x7fe0d406a230 2023-10-25T00:08:12.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0c9ffb640 1 -- 172.21.15.107:0/3912043293 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5019 (secure 0 0 0) 0x7fe0cc019020 con 0x7fe0d406a230 2023-10-25T00:08:12.299 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:12.299 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:12.299 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 134 2023-10-25T00:08:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 -- 172.21.15.107:0/3912043293 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065eb0 msgr2=0x7fe0b8068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065eb0 0x7fe0b8068370 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe0c40af000 tx=0x7fe0c40b6040 comp rx=0 tx=0).stop 2023-10-25T00:08:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 -- 172.21.15.107:0/3912043293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d406a230 msgr2=0x7fe0d41eeb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:12.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d406a230 0x7fe0d41eeb40 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00ba50 tx=0x7fe0cc00bf10 comp rx=0 tx=0).stop 2023-10-25T00:08:12.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 -- 172.21.15.107:0/3912043293 shutdown_connections 2023-10-25T00:08:12.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065eb0 0x7fe0b8068370 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d414b600 0x7fe0d41568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d406af70 0x7fe0d41ee2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 --2- 172.21.15.107:0/3912043293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d406a230 0x7fe0d41eeb40 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 -- 172.21.15.107:0/3912043293 >> 172.21.15.107:0/3912043293 conn(0x7fe0d40c2950 msgr2=0x7fe0d4149940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:12.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 -- 172.21.15.107:0/3912043293 shutdown_connections 2023-10-25T00:08:12.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.299+0000 7fe0db9fd640 1 -- 172.21.15.107:0/3912043293 wait complete. 2023-10-25T00:08:12.319 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=7107 2023-10-25T00:08:12.320 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.c 2023-10-25T00:08:12.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.407+0000 7f20a86f2640 1 -- 172.21.15.107:0/4016303260 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20a0065a00 con 0x7f20a00c70e0 2023-10-25T00:08:12.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.407+0000 7f2096ffd640 1 -- 172.21.15.107:0/4016303260 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5019 (secure 0 0 0) 0x7f20900b0020 con 0x7f20a00c70e0 2023-10-25T00:08:12.410 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:12.410 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:12.410 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 134 2023-10-25T00:08:12.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 -- 172.21.15.107:0/4016303260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2084065e60 msgr2=0x7f2084068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:12.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2084065e60 0x7f2084068320 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f209c000f70 tx=0x7f209c007c10 comp rx=0 tx=0).stop 2023-10-25T00:08:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 -- 172.21.15.107:0/4016303260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a00c70e0 msgr2=0x7f20a0157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a00c70e0 0x7f20a0157e30 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f20900ae040 tx=0x7f20900190e0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 -- 172.21.15.107:0/4016303260 shutdown_connections 2023-10-25T00:08:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2084065e60 0x7f2084068320 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20a0156770 0x7f20a01f0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20a00c70e0 0x7f20a0157e30 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 --2- 172.21.15.107:0/4016303260 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f20a00c2130 0x7f20a01588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 -- 172.21.15.107:0/4016303260 >> 172.21.15.107:0/4016303260 conn(0x7f20a00c65c0 msgr2=0x7f20a00ab6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 -- 172.21.15.107:0/4016303260 shutdown_connections 2023-10-25T00:08:12.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:12.411+0000 7f20a86f2640 1 -- 172.21.15.107:0/4016303260 wait complete. 2023-10-25T00:08:12.427 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:08:14.320 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2023-10-25T00:08:14.321 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:08:14.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:08:14.325 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:08:14.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:14.409 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:08:14.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:14.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62af90640 1 Processor -- start 2023-10-25T00:08:14.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62af90640 1 -- start start 2023-10-25T00:08:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62af90640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa62414fc60 0x7fa6241566d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62af90640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6241533f0 con 0x7fa6240c18a0 2023-10-25T00:08:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62af90640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa624156c10 con 0x7fa6240c5c00 2023-10-25T00:08:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62af90640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6241527e0 con 0x7fa62414fc60 2023-10-25T00:08:14.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62952d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa6240c18a0 0x7fa6240c1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56644/0 (socket says 172.21.15.107:56644) 2023-10-25T00:08:14.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62952d640 1 -- 172.21.15.107:0/1779117761 learned_addr learned my addr 172.21.15.107:0/1779117761 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:14.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa628d2c640 1 --2- 172.21.15.107:0/1779117761 >> v2:172.21.15.110:3300/0 conn(0x7fa62414fc60 0x7fa6241566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107778912 0 0) 0x7fa6241533f0 con 0x7fa6240c18a0 2023-10-25T00:08:14.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa614003530 con 0x7fa6240c18a0 2023-10-25T00:08:14.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3155894370 0 0) 0x7fa614003530 con 0x7fa6240c18a0 2023-10-25T00:08:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 >> v2:172.21.15.110:3300/0 conn(0x7fa62414fc60 msgr2=0x7fa6241566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 --2- 172.21.15.107:0/1779117761 >> v2:172.21.15.110:3300/0 conn(0x7fa62414fc60 0x7fa6241566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 >> v1:172.21.15.110:6789/0 conn(0x7fa6240c5c00 legacy=0x7fa62406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa624152ac0 con 0x7fa6240c18a0 2023-10-25T00:08:14.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fa61c001a00 con 0x7fa6240c18a0 2023-10-25T00:08:14.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 >> v1:172.21.15.107:6789/0 conn(0x7fa6240c18a0 legacy=0x7fa6240c1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 --2- 172.21.15.107:0/1779117761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa614003980 0x7fa614003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 --2- 172.21.15.107:0/1779117761 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6140047d0 0x7fa614004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 --2- 172.21.15.107:0/1779117761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 0x7fa61400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6241527e0 con 0x7fa614005190 2023-10-25T00:08:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61400c130 con 0x7fa614003980 2023-10-25T00:08:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61400c2f0 con 0x7fa6140047d0 2023-10-25T00:08:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62952d640 1 --2- 172.21.15.107:0/1779117761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 0x7fa61400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa623fff640 1 --2- 172.21.15.107:0/1779117761 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6140047d0 0x7fa614004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa628d2c640 1 --2- 172.21.15.107:0/1779117761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa614003980 0x7fa614003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62952d640 1 -- 172.21.15.107:0/1779117761 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6140047d0 msgr2=0x7fa614004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:14.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62952d640 1 --2- 172.21.15.107:0/1779117761 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6140047d0 0x7fa614004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62952d640 1 -- 172.21.15.107:0/1779117761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa614003980 msgr2=0x7fa614003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:14.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62952d640 1 --2- 172.21.15.107:0/1779117761 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa614003980 0x7fa614003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.467+0000 7fa62952d640 1 -- 172.21.15.107:0/1779117761 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa614003530 con 0x7fa614005190 2023-10-25T00:08:14.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa62952d640 1 -- 172.21.15.107:0/1779117761 reap_dead start 2023-10-25T00:08:14.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa62952d640 1 --2- 172.21.15.107:0/1779117761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 0x7fa61400b990 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00a820 tx=0x7fa61c01fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690afa0ffb242c9e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:14.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c0aa6b0 con 0x7fa614005190 2023-10-25T00:08:14.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61c0aa850 con 0x7fa614005190 2023-10-25T00:08:14.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa6237fe640 1 -- 172.21.15.107:0/1779117761 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c0aac00 con 0x7fa614005190 2023-10-25T00:08:14.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa62af90640 1 -- 172.21.15.107:0/1779117761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 msgr2=0x7fa61400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa62af90640 1 --2- 172.21.15.107:0/1779117761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 0x7fa61400b990 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00a820 tx=0x7fa61c01fa80 comp rx=0 tx=0).stop 2023-10-25T00:08:14.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa62af90640 1 -- 172.21.15.107:0/1779117761 shutdown_connections 2023-10-25T00:08:14.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa62af90640 1 --2- 172.21.15.107:0/1779117761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 0x7fa61400b990 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa62af90640 1 -- 172.21.15.107:0/1779117761 >> 172.21.15.107:0/1779117761 conn(0x7fa6240c39a0 msgr2=0x7fa6240c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:14.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa62af90640 1 -- 172.21.15.107:0/1779117761 shutdown_connections 2023-10-25T00:08:14.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.471+0000 7fa62af90640 1 -- 172.21.15.107:0/1779117761 wait complete. 2023-10-25T00:08:14.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62af90640 1 Processor -- start 2023-10-25T00:08:14.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62af90640 1 -- start start 2023-10-25T00:08:14.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62af90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 0x7fa6240c1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62af90640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62406b7a0 0x7fa62414b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62af90640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6241e7630 0x7fa62414ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62af90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6240bef60 con 0x7fa614005190 2023-10-25T00:08:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62af90640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6240bbf20 con 0x7fa6241e7630 2023-10-25T00:08:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62af90640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6240c1690 con 0x7fa62406b7a0 2023-10-25T00:08:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa628d2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 0x7fa6240c1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa628d2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 0x7fa6240c1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56074/0 (socket says 172.21.15.107:56074) 2023-10-25T00:08:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa628d2c640 1 -- 172.21.15.107:0/1051578057 learned_addr learned my addr 172.21.15.107:0/1051578057 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62952d640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6241e7630 0x7fa62414ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa623fff640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62406b7a0 0x7fa62414b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa628d2c640 1 -- 172.21.15.107:0/1051578057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62406b7a0 msgr2=0x7fa62414b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa628d2c640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62406b7a0 0x7fa62414b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa628d2c640 1 -- 172.21.15.107:0/1051578057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6241e7630 msgr2=0x7fa62414ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa628d2c640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6241e7630 0x7fa62414ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa628d2c640 1 -- 172.21.15.107:0/1051578057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6241e7be0 con 0x7fa614005190 2023-10-25T00:08:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa628d2c640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 0x7fa6240c1150 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fa618011560 tx=0x7fa618011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa6217fa640 1 -- 172.21.15.107:0/1051578057 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618015930 con 0x7fa614005190 2023-10-25T00:08:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa6217fa640 1 -- 172.21.15.107:0/1051578057 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa618015a90 con 0x7fa614005190 2023-10-25T00:08:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62af90640 1 -- 172.21.15.107:0/1051578057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62406c930 con 0x7fa614005190 2023-10-25T00:08:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa6217fa640 1 -- 172.21.15.107:0/1051578057 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618015e60 con 0x7fa614005190 2023-10-25T00:08:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62af90640 1 -- 172.21.15.107:0/1051578057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62406cc10 con 0x7fa614005190 2023-10-25T00:08:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.475+0000 7fa62af90640 1 -- 172.21.15.107:0/1051578057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6240746e0 con 0x7fa614005190 2023-10-25T00:08:14.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.479+0000 7fa6217fa640 1 -- 172.21.15.107:0/1051578057 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa61800f460 con 0x7fa614005190 2023-10-25T00:08:14.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.479+0000 7fa6217fa640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa608065e60 0x7fa608068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.479+0000 7fa6217fa640 1 -- 172.21.15.107:0/1051578057 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 9111+0+0 (secure 0 0 0) 0x7fa618084b30 con 0x7fa614005190 2023-10-25T00:08:14.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.483+0000 7fa623fff640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa608065e60 0x7fa608068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.483+0000 7fa623fff640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa608065e60 0x7fa608068320 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa6240bfa90 tx=0x7fa610009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:14.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.483+0000 7fa6217fa640 1 -- 172.21.15.107:0/1051578057 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa6180515d0 con 0x7fa614005190 2023-10-25T00:08:14.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.559+0000 7f80ed968640 1 Processor -- start 2023-10-25T00:08:14.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.559+0000 7f80ed968640 1 -- start start 2023-10-25T00:08:14.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80ed968640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80e8155ea0 0x7f80e806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80ed968640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80e81566e0 0x7f80e80c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80ed968640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e80c59d0 con 0x7f80e8156280 2023-10-25T00:08:14.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80ed968640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e80c2990 con 0x7f80e8155ea0 2023-10-25T00:08:14.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80ed968640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e80c2ec0 con 0x7f80e81566e0 2023-10-25T00:08:14.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80e6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80e8155ea0 0x7f80e806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80e6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80e8155ea0 0x7f80e806ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56076/0 (socket says 172.21.15.107:56076) 2023-10-25T00:08:14.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80e6ffd640 1 -- 172.21.15.107:0/2549440793 learned_addr learned my addr 172.21.15.107:0/2549440793 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:14.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80e77fe640 1 --2- 172.21.15.107:0/2549440793 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80e81566e0 0x7f80e80c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80e6ffd640 1 -- 172.21.15.107:0/2549440793 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80e81566e0 msgr2=0x7f80e80c3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:14.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80e6ffd640 1 --2- 172.21.15.107:0/2549440793 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80e81566e0 0x7f80e80c3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80e6ffd640 1 -- 172.21.15.107:0/2549440793 >> v1:172.21.15.110:6789/0 conn(0x7f80e8156280 legacy=0x7f80e814c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80e6ffd640 1 -- 172.21.15.107:0/2549440793 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e806d500 con 0x7f80e8155ea0 2023-10-25T00:08:14.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.563+0000 7f80e6ffd640 1 --2- 172.21.15.107:0/2549440793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80e8155ea0 0x7f80e806ec90 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009870 tx=0x7f80d801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36db4f4c14f98b89 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:14.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e67fc640 1 -- 172.21.15.107:0/2549440793 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d80ab600 con 0x7f80e8155ea0 2023-10-25T00:08:14.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e67fc640 1 -- 172.21.15.107:0/2549440793 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d80a7050 con 0x7f80e8155ea0 2023-10-25T00:08:14.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e67fc640 1 -- 172.21.15.107:0/2549440793 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d80ab980 con 0x7f80e8155ea0 2023-10-25T00:08:14.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 -- 172.21.15.107:0/2549440793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80e8155ea0 msgr2=0x7f80e806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 --2- 172.21.15.107:0/2549440793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80e8155ea0 0x7f80e806ec90 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009870 tx=0x7f80d801c420 comp rx=0 tx=0).stop 2023-10-25T00:08:14.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 -- 172.21.15.107:0/2549440793 shutdown_connections 2023-10-25T00:08:14.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 --2- 172.21.15.107:0/2549440793 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80e81566e0 0x7f80e80c3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 --2- 172.21.15.107:0/2549440793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80e8155ea0 0x7f80e806ec90 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 -- 172.21.15.107:0/2549440793 >> 172.21.15.107:0/2549440793 conn(0x7f80e80c4a90 msgr2=0x7f80e80c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:14.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 -- 172.21.15.107:0/2549440793 shutdown_connections 2023-10-25T00:08:14.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 -- 172.21.15.107:0/2549440793 wait complete. 2023-10-25T00:08:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 Processor -- start 2023-10-25T00:08:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 -- start start 2023-10-25T00:08:14.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80e8155ea0 0x7f80e81eeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80e8156280 0x7f80e813d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80e81566e0 0x7f80e8145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e814a040 con 0x7f80e81566e0 2023-10-25T00:08:14.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e8142d30 con 0x7f80e8155ea0 2023-10-25T00:08:14.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80ed968640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e8147f00 con 0x7f80e8156280 2023-10-25T00:08:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80e81566e0 0x7f80e8145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80e81566e0 0x7f80e8145910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56092/0 (socket says 172.21.15.107:56092) 2023-10-25T00:08:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e77fe640 1 -- 172.21.15.107:0/1551000434 learned_addr learned my addr 172.21.15.107:0/1551000434 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e6ffd640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80e8155ea0 0x7f80e81eeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80dffff640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80e8156280 0x7f80e813d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e77fe640 1 -- 172.21.15.107:0/1551000434 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80e8156280 msgr2=0x7f80e813d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e77fe640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80e8156280 0x7f80e813d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e77fe640 1 -- 172.21.15.107:0/1551000434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80e8155ea0 msgr2=0x7f80e81eeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e77fe640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80e8155ea0 0x7f80e81eeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e77fe640 1 -- 172.21.15.107:0/1551000434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d80094d0 con 0x7f80e81566e0 2023-10-25T00:08:14.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.567+0000 7f80e77fe640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80e81566e0 0x7f80e8145910 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f80e0007d90 tx=0x7f80e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:14.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.571+0000 7f80e4ff9640 1 -- 172.21.15.107:0/1551000434 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e0014420 con 0x7f80e81566e0 2023-10-25T00:08:14.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.571+0000 7f80e4ff9640 1 -- 172.21.15.107:0/1551000434 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e00145a0 con 0x7f80e81566e0 2023-10-25T00:08:14.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.571+0000 7f80ed968640 1 -- 172.21.15.107:0/1551000434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e81481a0 con 0x7f80e81566e0 2023-10-25T00:08:14.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.571+0000 7f80e4ff9640 1 -- 172.21.15.107:0/1551000434 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e0014950 con 0x7f80e81566e0 2023-10-25T00:08:14.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.571+0000 7f80ed968640 1 -- 172.21.15.107:0/1551000434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e81429c0 con 0x7f80e81566e0 2023-10-25T00:08:14.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.571+0000 7f80ed968640 1 -- 172.21.15.107:0/1551000434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e80746e0 con 0x7f80e81566e0 2023-10-25T00:08:14.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.571+0000 7f80e4ff9640 1 -- 172.21.15.107:0/1551000434 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f80e0014ab0 con 0x7f80e81566e0 2023-10-25T00:08:14.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.571+0000 7f80e4ff9640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80c8065e60 0x7f80c8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:14.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.571+0000 7f80e4ff9640 1 -- 172.21.15.107:0/1551000434 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 9111+0+0 (secure 0 0 0) 0x7f80e0084cd0 con 0x7f80e81566e0 2023-10-25T00:08:14.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.571+0000 7f80e6ffd640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80c8065e60 0x7f80c8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:14.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.571+0000 7f80e6ffd640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80c8065e60 0x7f80c8068320 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f80d80ad000 tx=0x7f80d80a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:14.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.575+0000 7f80e4ff9640 1 -- 172.21.15.107:0/1551000434 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f80e0051770 con 0x7f80e81566e0 2023-10-25T00:08:14.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.715+0000 7fa62af90640 1 -- 172.21.15.107:0/1051578057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6240c5090 con 0x7fa614005190 2023-10-25T00:08:14.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.715+0000 7fa6217fa640 1 -- 172.21.15.107:0/1051578057 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5019 (secure 0 0 0) 0x7fa618054bf0 con 0x7fa614005190 2023-10-25T00:08:14.718 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:14.718 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:14.718 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 134 2023-10-25T00:08:14.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 -- 172.21.15.107:0/1051578057 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa608065e60 msgr2=0x7fa608068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa608065e60 0x7fa608068320 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa6240bfa90 tx=0x7fa610009040 comp rx=0 tx=0).stop 2023-10-25T00:08:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 -- 172.21.15.107:0/1051578057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 msgr2=0x7fa6240c1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 0x7fa6240c1150 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fa618011560 tx=0x7fa618011a20 comp rx=0 tx=0).stop 2023-10-25T00:08:14.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 -- 172.21.15.107:0/1051578057 shutdown_connections 2023-10-25T00:08:14.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6241e7630 0x7fa62414ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa608065e60 0x7fa608068320 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62406b7a0 0x7fa62414b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 --2- 172.21.15.107:0/1051578057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa614005190 0x7fa6240c1150 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 -- 172.21.15.107:0/1051578057 >> 172.21.15.107:0/1051578057 conn(0x7fa6240c39a0 msgr2=0x7fa6240c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:14.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 -- 172.21.15.107:0/1051578057 shutdown_connections 2023-10-25T00:08:14.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.719+0000 7fa62af90640 1 -- 172.21.15.107:0/1051578057 wait complete. 2023-10-25T00:08:14.738 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-25T00:08:14.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.827+0000 7f80ed968640 1 -- 172.21.15.107:0/1551000434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80e81533d0 con 0x7f80e81566e0 2023-10-25T00:08:14.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.827+0000 7f80e4ff9640 1 -- 172.21.15.107:0/1551000434 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5780 (secure 0 0 0) 0x7f80e0054d90 con 0x7f80e81566e0 2023-10-25T00:08:14.829 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:14.829 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:14.830 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 135 2023-10-25T00:08:14.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 -- 172.21.15.107:0/1551000434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80c8065e60 msgr2=0x7f80c8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80c8065e60 0x7f80c8068320 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f80d80ad000 tx=0x7f80d80a6000 comp rx=0 tx=0).stop 2023-10-25T00:08:14.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 -- 172.21.15.107:0/1551000434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80e81566e0 msgr2=0x7f80e8145910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:14.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80e81566e0 0x7f80e8145910 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f80e0007d90 tx=0x7f80e000e4b0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 -- 172.21.15.107:0/1551000434 shutdown_connections 2023-10-25T00:08:14.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80e81566e0 0x7f80e8145910 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80c8065e60 0x7f80c8068320 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80e8156280 0x7f80e813d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 --2- 172.21.15.107:0/1551000434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80e8155ea0 0x7f80e81eeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:14.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 -- 172.21.15.107:0/1551000434 >> 172.21.15.107:0/1551000434 conn(0x7f80e80c4a90 msgr2=0x7f80e80abf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:14.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 -- 172.21.15.107:0/1551000434 shutdown_connections 2023-10-25T00:08:14.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:14.831+0000 7f80ed968640 1 -- 172.21.15.107:0/1551000434 wait complete. 2023-10-25T00:08:14.848 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:08:15.846 INFO:tasks.ceph.mds.d.smithi110.stderr:2023-10-25T00:08:15.845+0000 7f0125d36640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:08:16.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:16.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:16.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.871+0000 7f9df918d640 1 Processor -- start 2023-10-25T00:08:16.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.871+0000 7f9df918d640 1 -- start start 2023-10-25T00:08:16.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df918d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9df414fc30 0x7f9df4153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:16.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df918d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9df40c5c20 0x7f9df406df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:16.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df918d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df4072330 con 0x7f9df406e470 2023-10-25T00:08:16.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df918d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df4155ca0 con 0x7f9df40c5c20 2023-10-25T00:08:16.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df918d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4155f80 con 0x7f9df414fc30 2023-10-25T00:08:16.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9df40c5c20 0x7f9df406df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:16.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df359e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9df406e470 0x7f9df40c18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56660/0 (socket says 172.21.15.107:56660) 2023-10-25T00:08:16.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df359e640 1 -- 172.21.15.107:0/158694202 learned_addr learned my addr 172.21.15.107:0/158694202 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:16.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df2d9d640 1 --2- 172.21.15.107:0/158694202 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9df414fc30 0x7f9df4153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:16.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df259c640 1 -- 172.21.15.107:0/158694202 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9df414fc30 msgr2=0x7f9df4153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:16.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df259c640 1 --2- 172.21.15.107:0/158694202 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9df414fc30 0x7f9df4153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df259c640 1 -- 172.21.15.107:0/158694202 >> v1:172.21.15.107:6789/0 conn(0x7f9df406e470 legacy=0x7f9df40c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df259c640 1 -- 172.21.15.107:0/158694202 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df4156260 con 0x7f9df40c5c20 2023-10-25T00:08:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.875+0000 7f9df1d9b640 1 -- 172.21.15.107:0/158694202 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10206561 0 0) 0x7f9df4072330 con 0x7f9df406e470 2023-10-25T00:08:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df259c640 1 --2- 172.21.15.107:0/158694202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df40c5c20 0x7f9df406df30 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f9de80097c0 tx=0x7f9de801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef38d00b38eb71a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df1d9b640 1 -- 172.21.15.107:0/158694202 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de80ac620 con 0x7f9df40c5c20 2023-10-25T00:08:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df1d9b640 1 -- 172.21.15.107:0/158694202 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de80ac780 con 0x7f9df40c5c20 2023-10-25T00:08:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df1d9b640 1 -- 172.21.15.107:0/158694202 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de80acb10 con 0x7f9df40c5c20 2023-10-25T00:08:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 -- 172.21.15.107:0/158694202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df40c5c20 msgr2=0x7f9df406df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 --2- 172.21.15.107:0/158694202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df40c5c20 0x7f9df406df30 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f9de80097c0 tx=0x7f9de801c3c0 comp rx=0 tx=0).stop 2023-10-25T00:08:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 -- 172.21.15.107:0/158694202 shutdown_connections 2023-10-25T00:08:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 --2- 172.21.15.107:0/158694202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df40c5c20 0x7f9df406df30 secure :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f9de80097c0 tx=0x7f9de801c3c0 comp rx=0 tx=0).stop 2023-10-25T00:08:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 --2- 172.21.15.107:0/158694202 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9df414fc30 0x7f9df4153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:16.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 -- 172.21.15.107:0/158694202 >> 172.21.15.107:0/158694202 conn(0x7f9df40c39a0 msgr2=0x7f9df40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:16.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 -- 172.21.15.107:0/158694202 shutdown_connections 2023-10-25T00:08:16.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 -- 172.21.15.107:0/158694202 wait complete. 2023-10-25T00:08:16.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 Processor -- start 2023-10-25T00:08:16.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 -- start start 2023-10-25T00:08:16.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9df406e470 0x7f9df40c04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:16.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9df414fc30 0x7f9df413a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df4157000 0x7f9df413aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4071f00 con 0x7f9df4157000 2023-10-25T00:08:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df413b0e0 con 0x7f9df414fc30 2023-10-25T00:08:16.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df918d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4158870 con 0x7f9df406e470 2023-10-25T00:08:16.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df4157000 0x7f9df413aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:16.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df4157000 0x7f9df413aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56106/0 (socket says 172.21.15.107:56106) 2023-10-25T00:08:16.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df359e640 1 -- 172.21.15.107:0/2312067798 learned_addr learned my addr 172.21.15.107:0/2312067798 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:16.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df2d9d640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9df406e470 0x7f9df40c04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:16.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df259c640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9df414fc30 0x7f9df413a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df359e640 1 -- 172.21.15.107:0/2312067798 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9df406e470 msgr2=0x7f9df40c04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df359e640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9df406e470 0x7f9df40c04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df359e640 1 -- 172.21.15.107:0/2312067798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9df414fc30 msgr2=0x7f9df413a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df359e640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9df414fc30 0x7f9df413a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.879+0000 7f9df359e640 1 -- 172.21.15.107:0/2312067798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8009420 con 0x7f9df4157000 2023-10-25T00:08:16.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.883+0000 7f9df359e640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df4157000 0x7f9df413aba0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00c7a0 tx=0x7f9dec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:16.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.883+0000 7f9ddb7fe640 1 -- 172.21.15.107:0/2312067798 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec00a640 con 0x7f9df4157000 2023-10-25T00:08:16.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.883+0000 7f9ddb7fe640 1 -- 172.21.15.107:0/2312067798 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dec00a7a0 con 0x7f9df4157000 2023-10-25T00:08:16.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.883+0000 7f9ddb7fe640 1 -- 172.21.15.107:0/2312067798 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec00abd0 con 0x7f9df4157000 2023-10-25T00:08:16.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.883+0000 7f9df918d640 1 -- 172.21.15.107:0/2312067798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df4158590 con 0x7f9df4157000 2023-10-25T00:08:16.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.883+0000 7f9df918d640 1 -- 172.21.15.107:0/2312067798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df406a770 con 0x7f9df4157000 2023-10-25T00:08:16.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.883+0000 7f9df918d640 1 -- 172.21.15.107:0/2312067798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df406a960 con 0x7f9df4157000 2023-10-25T00:08:16.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.883+0000 7f9ddb7fe640 1 -- 172.21.15.107:0/2312067798 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9dec00ad30 con 0x7f9df4157000 2023-10-25T00:08:16.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.883+0000 7f9ddb7fe640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9de0065e60 0x7f9de0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:16.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.883+0000 7f9ddb7fe640 1 -- 172.21.15.107:0/2312067798 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 9111+0+0 (secure 0 0 0) 0x7f9dec083960 con 0x7f9df4157000 2023-10-25T00:08:16.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.883+0000 7f9df2d9d640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9de0065e60 0x7f9de0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:16.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.887+0000 7f9df2d9d640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9de0065e60 0x7f9de0068320 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9de400b930 tx=0x7f9de4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:16.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.891+0000 7f9ddb7fe640 1 -- 172.21.15.107:0/2312067798 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9dec050400 con 0x7f9df4157000 2023-10-25T00:08:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.987+0000 7f53b947f640 1 Processor -- start 2023-10-25T00:08:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.987+0000 7f53b947f640 1 -- start start 2023-10-25T00:08:16.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.987+0000 7f53b947f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b40c8640 0x7f53b406a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:16.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.987+0000 7f53b947f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b4152400 con 0x7f53b4152020 2023-10-25T00:08:16.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.987+0000 7f53b947f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b41558d0 con 0x7f53b41553d0 2023-10-25T00:08:16.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.987+0000 7f53b947f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b40c6ce0 con 0x7f53b40c8640 2023-10-25T00:08:16.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.987+0000 7f53b2ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f53b4152020 0x7f53b40c4bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56670/0 (socket says 172.21.15.107:56670) 2023-10-25T00:08:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b2ffd640 1 -- 172.21.15.107:0/4090488117 learned_addr learned my addr 172.21.15.107:0/4090488117 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b27fc640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b40c8640 0x7f53b406a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733944623 0 0) 0x7f53b4152400 con 0x7f53b4152020 2023-10-25T00:08:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f539c003530 con 0x7f53b4152020 2023-10-25T00:08:16.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431068492 0 0) 0x7f539c003530 con 0x7f53b4152020 2023-10-25T00:08:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b40c8640 msgr2=0x7f53b406a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b40c8640 0x7f53b406a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 >> v1:172.21.15.110:6789/0 conn(0x7f53b41553d0 legacy=0x7f53b406ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:16.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40c6fc0 con 0x7f53b4152020 2023-10-25T00:08:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f53a40016a0 con 0x7f53b4152020 2023-10-25T00:08:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 >> v1:172.21.15.107:6789/0 conn(0x7f53b4152020 legacy=0x7f53b40c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f539c0039b0 0x7f539c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f539c0047d0 0x7f539c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:16.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b40c8640 0x7f539c005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:16.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b40c6ce0 con 0x7f539c0039b0 2023-10-25T00:08:16.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539c00bb50 con 0x7f539c0047d0 2023-10-25T00:08:16.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539c00bd10 con 0x7f53b40c8640 2023-10-25T00:08:16.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b2ffd640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f539c0039b0 0x7f539c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:16.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b27fc640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f539c0047d0 0x7f539c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b37fe640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b40c8640 0x7f539c005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b2ffd640 1 -- 172.21.15.107:0/4090488117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b40c8640 msgr2=0x7f539c005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b2ffd640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b40c8640 0x7f539c005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b2ffd640 1 -- 172.21.15.107:0/4090488117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f539c0047d0 msgr2=0x7f539c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:16.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b2ffd640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f539c0047d0 0x7f539c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:16.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.991+0000 7f53b2ffd640 1 -- 172.21.15.107:0/4090488117 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c003530 con 0x7f539c0039b0 2023-10-25T00:08:16.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b2ffd640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f539c0039b0 0x7f539c003dd0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f53b40ca110 tx=0x7f53a401caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9104b9af5664db87 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:17.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4005ae0 con 0x7f539c0039b0 2023-10-25T00:08:17.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a4005c40 con 0x7f539c0039b0 2023-10-25T00:08:17.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b1ffb640 1 -- 172.21.15.107:0/4090488117 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a40a57f0 con 0x7f539c0039b0 2023-10-25T00:08:17.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b947f640 1 -- 172.21.15.107:0/4090488117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f539c0039b0 msgr2=0x7f539c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:17.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b947f640 1 --2- 172.21.15.107:0/4090488117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f539c0039b0 0x7f539c003dd0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f53b40ca110 tx=0x7f53a401caf0 comp rx=0 tx=0).stop 2023-10-25T00:08:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b37fe640 1 -- 172.21.15.107:0/4090488117 reap_dead start 2023-10-25T00:08:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b947f640 1 -- 172.21.15.107:0/4090488117 shutdown_connections 2023-10-25T00:08:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b947f640 1 -- 172.21.15.107:0/4090488117 >> 172.21.15.107:0/4090488117 conn(0x7f53b40abd60 msgr2=0x7f53b40ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b947f640 1 -- 172.21.15.107:0/4090488117 shutdown_connections 2023-10-25T00:08:17.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b947f640 1 -- 172.21.15.107:0/4090488117 wait complete. 2023-10-25T00:08:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b947f640 1 Processor -- start 2023-10-25T00:08:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.995+0000 7f53b947f640 1 -- start start 2023-10-25T00:08:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b947f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b4158840 0x7f53b414b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:17.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b947f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53b4155d40 0x7f53b414ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:17.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b947f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53b414ee00 0x7f53b41ed270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:17.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b947f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b4156ed0 con 0x7f53b4155d40 2023-10-25T00:08:17.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b947f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b41562c0 con 0x7f53b414ee00 2023-10-25T00:08:17.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b947f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b41565a0 con 0x7f53b4158840 2023-10-25T00:08:17.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53b4155d40 0x7f53b414ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:17.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53b4155d40 0x7f53b414ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56130/0 (socket says 172.21.15.107:56130) 2023-10-25T00:08:17.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b27fc640 1 -- 172.21.15.107:0/1198979819 learned_addr learned my addr 172.21.15.107:0/1198979819 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:17.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b2ffd640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b4158840 0x7f53b414b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:17.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b37fe640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53b414ee00 0x7f53b41ed270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:17.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b27fc640 1 -- 172.21.15.107:0/1198979819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b4158840 msgr2=0x7f53b414b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b27fc640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b4158840 0x7f53b414b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b27fc640 1 -- 172.21.15.107:0/1198979819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53b414ee00 msgr2=0x7f53b41ed270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b27fc640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53b414ee00 0x7f53b41ed270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b27fc640 1 -- 172.21.15.107:0/1198979819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b4156880 con 0x7f53b4155d40 2023-10-25T00:08:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b27fc640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53b4155d40 0x7f53b414ba90 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f53a80115d0 tx=0x7f53a8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:17.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f539b7fe640 1 -- 172.21.15.107:0/1198979819 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a800f8f0 con 0x7f53b4155d40 2023-10-25T00:08:17.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f539b7fe640 1 -- 172.21.15.107:0/1198979819 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a800fa50 con 0x7f53b4155d40 2023-10-25T00:08:17.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b947f640 1 -- 172.21.15.107:0/1198979819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b4156b60 con 0x7f53b4155d40 2023-10-25T00:08:17.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f539b7fe640 1 -- 172.21.15.107:0/1198979819 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a800fe00 con 0x7f53b4155d40 2023-10-25T00:08:17.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b947f640 1 -- 172.21.15.107:0/1198979819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b4158c20 con 0x7f53b4155d40 2023-10-25T00:08:17.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b947f640 1 -- 172.21.15.107:0/1198979819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b41ed050 con 0x7f53b4155d40 2023-10-25T00:08:17.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f539b7fe640 1 -- 172.21.15.107:0/1198979819 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f53a8015460 con 0x7f53b4155d40 2023-10-25T00:08:17.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f539b7fe640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53a0065e60 0x7f53a0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:17.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f539b7fe640 1 -- 172.21.15.107:0/1198979819 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 9111+0+0 (secure 0 0 0) 0x7f53a8084ae0 con 0x7f53b4155d40 2023-10-25T00:08:17.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:16.999+0000 7f53b2ffd640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53a0065e60 0x7f53a0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:17.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.003+0000 7f53b2ffd640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53a0065e60 0x7f53a0068320 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f53a40ab000 tx=0x7f53a40b1040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:17.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.007+0000 7f539b7fe640 1 -- 172.21.15.107:0/1198979819 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f53a8051580 con 0x7f53b4155d40 2023-10-25T00:08:17.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.131+0000 7f9df918d640 1 -- 172.21.15.107:0/2312067798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9df40746e0 con 0x7f9df4157000 2023-10-25T00:08:17.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.131+0000 7f9ddb7fe640 1 -- 172.21.15.107:0/2312067798 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5780 (secure 0 0 0) 0x7f9dec018070 con 0x7f9df4157000 2023-10-25T00:08:17.132 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:17.132 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:17.132 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 135 2023-10-25T00:08:17.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 -- 172.21.15.107:0/2312067798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9de0065e60 msgr2=0x7f9de0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:17.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9de0065e60 0x7f9de0068320 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9de400b930 tx=0x7f9de4008040 comp rx=0 tx=0).stop 2023-10-25T00:08:17.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 -- 172.21.15.107:0/2312067798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df4157000 msgr2=0x7f9df413aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:17.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df4157000 0x7f9df413aba0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00c7a0 tx=0x7f9dec00cc60 comp rx=0 tx=0).stop 2023-10-25T00:08:17.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 -- 172.21.15.107:0/2312067798 shutdown_connections 2023-10-25T00:08:17.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df4157000 0x7f9df413aba0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:17.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9df414fc30 0x7f9df413a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:17.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9de0065e60 0x7f9de0068320 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:17.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 --2- 172.21.15.107:0/2312067798 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9df406e470 0x7f9df40c04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:17.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 -- 172.21.15.107:0/2312067798 >> 172.21.15.107:0/2312067798 conn(0x7f9df40c39a0 msgr2=0x7f9df406be00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:17.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 -- 172.21.15.107:0/2312067798 shutdown_connections 2023-10-25T00:08:17.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.135+0000 7f9df918d640 1 -- 172.21.15.107:0/2312067798 wait complete. 2023-10-25T00:08:17.159 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2023-10-25T00:08:17.159 INFO:tasks.mds_thrash.fs.[a]:waiting for 29 secs before thrashing 2023-10-25T00:08:17.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.251+0000 7f53b947f640 1 -- 172.21.15.107:0/1198979819 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53b40746e0 con 0x7f53b4155d40 2023-10-25T00:08:17.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.251+0000 7f539b7fe640 1 -- 172.21.15.107:0/1198979819 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5780 (secure 0 0 0) 0x7f53a8054ba0 con 0x7f53b4155d40 2023-10-25T00:08:17.252 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:17.252 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:17.253 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 135 2023-10-25T00:08:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 -- 172.21.15.107:0/1198979819 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53a0065e60 msgr2=0x7f53a0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53a0065e60 0x7f53a0068320 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f53a40ab000 tx=0x7f53a40b1040 comp rx=0 tx=0).stop 2023-10-25T00:08:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 -- 172.21.15.107:0/1198979819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53b4155d40 msgr2=0x7f53b414ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:17.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53b4155d40 0x7f53b414ba90 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f53a80115d0 tx=0x7f53a8011a90 comp rx=0 tx=0).stop 2023-10-25T00:08:17.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 -- 172.21.15.107:0/1198979819 shutdown_connections 2023-10-25T00:08:17.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53b414ee00 0x7f53b41ed270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:17.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53b4155d40 0x7f53b414ba90 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:17.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53a0065e60 0x7f53a0068320 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:17.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 --2- 172.21.15.107:0/1198979819 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53b4158840 0x7f53b414b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:17.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 -- 172.21.15.107:0/1198979819 >> 172.21.15.107:0/1198979819 conn(0x7f53b40abd60 msgr2=0x7f53b40c8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:17.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 -- 172.21.15.107:0/1198979819 shutdown_connections 2023-10-25T00:08:17.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:17.255+0000 7f53b947f640 1 -- 172.21.15.107:0/1198979819 wait complete. 2023-10-25T00:08:17.272 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:08:19.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:19.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f94915f5640 1 Processor -- start 2023-10-25T00:08:19.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f94915f5640 1 -- start start 2023-10-25T00:08:19.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f94915f5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f948c152040 0x7f948c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:19.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f94915f5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f948c06a4b0 0x7f948c0ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:19.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f94915f5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948c155680 con 0x7f948c0c8100 2023-10-25T00:08:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f94915f5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f948c154a70 con 0x7f948c06a4b0 2023-10-25T00:08:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f94915f5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f948c154d50 con 0x7f948c152040 2023-10-25T00:08:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f948b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f948c06a4b0 0x7f948c0ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f948b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f948c06a4b0 0x7f948c0ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56144/0 (socket says 172.21.15.107:56144) 2023-10-25T00:08:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f948b7fe640 1 -- 172.21.15.107:0/2800993064 learned_addr learned my addr 172.21.15.107:0/2800993064 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:19.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f948b7fe640 1 -- 172.21.15.107:0/2800993064 >> v2:172.21.15.110:3300/0 conn(0x7f948c152040 msgr2=0x7f948c0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:19.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f948b7fe640 1 --2- 172.21.15.107:0/2800993064 >> v2:172.21.15.110:3300/0 conn(0x7f948c152040 0x7f948c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:19.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f948b7fe640 1 -- 172.21.15.107:0/2800993064 >> v1:172.21.15.110:6789/0 conn(0x7f948c0c8100 legacy=0x7f948c0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:19.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.411+0000 7f948b7fe640 1 -- 172.21.15.107:0/2800993064 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c155030 con 0x7f948c06a4b0 2023-10-25T00:08:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f948b7fe640 1 --2- 172.21.15.107:0/2800993064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f948c06a4b0 0x7f948c0ac410 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f948c067ba0 tx=0x7f948001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ac51797780cab96 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f9489ffb640 1 -- 172.21.15.107:0/2800993064 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94800a56d0 con 0x7f948c06a4b0 2023-10-25T00:08:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f9489ffb640 1 -- 172.21.15.107:0/2800993064 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94800a5830 con 0x7f948c06a4b0 2023-10-25T00:08:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f9489ffb640 1 -- 172.21.15.107:0/2800993064 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94800a5bc0 con 0x7f948c06a4b0 2023-10-25T00:08:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f94915f5640 1 -- 172.21.15.107:0/2800993064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f948c06a4b0 msgr2=0x7f948c0ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f94915f5640 1 --2- 172.21.15.107:0/2800993064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f948c06a4b0 0x7f948c0ac410 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f948c067ba0 tx=0x7f948001c900 comp rx=0 tx=0).stop 2023-10-25T00:08:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f94915f5640 1 -- 172.21.15.107:0/2800993064 shutdown_connections 2023-10-25T00:08:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f94915f5640 1 --2- 172.21.15.107:0/2800993064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f948c06a4b0 0x7f948c0ac410 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f94915f5640 1 --2- 172.21.15.107:0/2800993064 >> v2:172.21.15.110:3300/0 conn(0x7f948c152040 0x7f948c0c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f94915f5640 1 -- 172.21.15.107:0/2800993064 >> 172.21.15.107:0/2800993064 conn(0x7f948c0ca200 msgr2=0x7f948c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f94915f5640 1 -- 172.21.15.107:0/2800993064 shutdown_connections 2023-10-25T00:08:19.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f94915f5640 1 -- 172.21.15.107:0/2800993064 wait complete. 2023-10-25T00:08:19.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f94915f5640 1 Processor -- start 2023-10-25T00:08:19.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.415+0000 7f94915f5640 1 -- start start 2023-10-25T00:08:19.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94915f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f948c06a4b0 0x7f948c1514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:19.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94915f5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f948c0c8100 0x7f948c0c3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:19.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94915f5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f948c152040 0x7f948c149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:19.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94915f5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c1ee290 con 0x7f948c06a4b0 2023-10-25T00:08:19.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94915f5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c14ff90 con 0x7f948c152040 2023-10-25T00:08:19.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94915f5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948c150270 con 0x7f948c0c8100 2023-10-25T00:08:19.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f948c06a4b0 0x7f948c1514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:19.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f948c06a4b0 0x7f948c1514a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56150/0 (socket says 172.21.15.107:56150) 2023-10-25T00:08:19.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948affd640 1 -- 172.21.15.107:0/3988924430 learned_addr learned my addr 172.21.15.107:0/3988924430 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:19.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948b7fe640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f948c152040 0x7f948c149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:19.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948a7fc640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f948c0c8100 0x7f948c0c3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:19.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948affd640 1 -- 172.21.15.107:0/3988924430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f948c0c8100 msgr2=0x7f948c0c3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:19.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948affd640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f948c0c8100 0x7f948c0c3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:19.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948affd640 1 -- 172.21.15.107:0/3988924430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f948c152040 msgr2=0x7f948c149640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:19.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948affd640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f948c152040 0x7f948c149640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:19.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948affd640 1 -- 172.21.15.107:0/3988924430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9480009b30 con 0x7f948c06a4b0 2023-10-25T00:08:19.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948a7fc640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f948c0c8100 0x7f948c0c3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:19.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948affd640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f948c06a4b0 0x7f948c1514a0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f948400b550 tx=0x7f948400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:19.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94737fe640 1 -- 172.21.15.107:0/3988924430 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9484010870 con 0x7f948c06a4b0 2023-10-25T00:08:19.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94737fe640 1 -- 172.21.15.107:0/3988924430 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94840109f0 con 0x7f948c06a4b0 2023-10-25T00:08:19.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94915f5640 1 -- 172.21.15.107:0/3988924430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948c150550 con 0x7f948c06a4b0 2023-10-25T00:08:19.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94737fe640 1 -- 172.21.15.107:0/3988924430 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9484010da0 con 0x7f948c06a4b0 2023-10-25T00:08:19.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94915f5640 1 -- 172.21.15.107:0/3988924430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f948c0aff00 con 0x7f948c06a4b0 2023-10-25T00:08:19.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94915f5640 1 -- 172.21.15.107:0/3988924430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f948c0c1090 con 0x7f948c06a4b0 2023-10-25T00:08:19.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94737fe640 1 -- 172.21.15.107:0/3988924430 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f948400f3c0 con 0x7f948c06a4b0 2023-10-25T00:08:19.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94737fe640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9478065e60 0x7f9478068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:19.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f94737fe640 1 -- 172.21.15.107:0/3988924430 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 9111+0+0 (secure 0 0 0) 0x7f9484084330 con 0x7f948c06a4b0 2023-10-25T00:08:19.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.419+0000 7f948a7fc640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9478065e60 0x7f9478068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:19.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.423+0000 7f948a7fc640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9478065e60 0x7f9478068320 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f948c152010 tx=0x7f947c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:19.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.427+0000 7f94737fe640 1 -- 172.21.15.107:0/3988924430 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9484050e70 con 0x7f948c06a4b0 2023-10-25T00:08:19.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.631+0000 7f94915f5640 1 -- 172.21.15.107:0/3988924430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f948c0746e0 con 0x7f948c06a4b0 2023-10-25T00:08:19.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94737fe640 1 -- 172.21.15.107:0/3988924430 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5780 (secure 0 0 0) 0x7f9484014090 con 0x7f948c06a4b0 2023-10-25T00:08:19.634 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:19.634 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:19.634 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 135 2023-10-25T00:08:19.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 -- 172.21.15.107:0/3988924430 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9478065e60 msgr2=0x7f9478068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:19.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9478065e60 0x7f9478068320 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f948c152010 tx=0x7f947c008040 comp rx=0 tx=0).stop 2023-10-25T00:08:19.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 -- 172.21.15.107:0/3988924430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f948c06a4b0 msgr2=0x7f948c1514a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:19.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f948c06a4b0 0x7f948c1514a0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f948400b550 tx=0x7f948400ba10 comp rx=0 tx=0).stop 2023-10-25T00:08:19.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 -- 172.21.15.107:0/3988924430 shutdown_connections 2023-10-25T00:08:19.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f948c152040 0x7f948c149640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:19.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9478065e60 0x7f9478068320 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:19.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f948c0c8100 0x7f948c0c3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:19.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 --2- 172.21.15.107:0/3988924430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f948c06a4b0 0x7f948c1514a0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:19.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 -- 172.21.15.107:0/3988924430 >> 172.21.15.107:0/3988924430 conn(0x7f948c0ca200 msgr2=0x7f948c0c9720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:19.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 -- 172.21.15.107:0/3988924430 shutdown_connections 2023-10-25T00:08:19.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:19.635+0000 7f94915f5640 1 -- 172.21.15.107:0/3988924430 wait complete. 2023-10-25T00:08:19.654 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:08:21.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:21.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.783+0000 7fb5cb59e640 1 Processor -- start 2023-10-25T00:08:21.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.783+0000 7fb5cb59e640 1 -- start start 2023-10-25T00:08:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.783+0000 7fb5cb59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb5cc0c2bb0 0x7fb5cc0c1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.783+0000 7fb5cb59e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb5cc0c1c60 0x7fb5cc15f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:21.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.783+0000 7fb5cb59e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc069ef0 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.783+0000 7fb5cb59e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5cc1556d0 con 0x7fb5cc0c2bb0 2023-10-25T00:08:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.783+0000 7fb5cb59e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5cc154860 con 0x7fb5cc0c1c60 2023-10-25T00:08:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5ca59c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb5cc0c27d0 0x7fb5cc072030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56682/0 (socket says 172.21.15.107:56682) 2023-10-25T00:08:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5ca59c640 1 -- 172.21.15.107:0/1585653500 learned_addr learned my addr 172.21.15.107:0/1585653500 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:21.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5c9d9b640 1 --2- 172.21.15.107:0/1585653500 >> v2:172.21.15.107:3300/0 conn(0x7fb5cc0c2bb0 0x7fb5cc0c1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5cad9d640 1 --2- 172.21.15.107:0/1585653500 >> v2:172.21.15.110:3300/0 conn(0x7fb5cc0c1c60 0x7fb5cc15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5c959a640 1 -- 172.21.15.107:0/1585653500 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115067875 0 0) 0x7fb5bc001120 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5c959a640 1 -- 172.21.15.107:0/1585653500 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b4003530 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5c9d9b640 1 -- 172.21.15.107:0/1585653500 >> v2:172.21.15.110:3300/0 conn(0x7fb5cc0c1c60 msgr2=0x7fb5cc15f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:21.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5c9d9b640 1 --2- 172.21.15.107:0/1585653500 >> v2:172.21.15.110:3300/0 conn(0x7fb5cc0c1c60 0x7fb5cc15f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:21.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5c9d9b640 1 -- 172.21.15.107:0/1585653500 >> v1:172.21.15.107:6789/0 conn(0x7fb5cc0c27d0 legacy=0x7fb5cc072030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:21.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5c9d9b640 1 -- 172.21.15.107:0/1585653500 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc0c3030 con 0x7fb5cc0c2bb0 2023-10-25T00:08:21.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5c9d9b640 1 --2- 172.21.15.107:0/1585653500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5cc0c2bb0 0x7fb5cc0c1720 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc06bfa0 tx=0x7fb5c001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8c42e4e4c9fafd server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:21.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5c959a640 1 -- 172.21.15.107:0/1585653500 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c00ab600 con 0x7fb5cc0c2bb0 2023-10-25T00:08:21.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5c959a640 1 -- 172.21.15.107:0/1585653500 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c00a7050 con 0x7fb5cc0c2bb0 2023-10-25T00:08:21.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5c959a640 1 -- 172.21.15.107:0/1585653500 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c00ab980 con 0x7fb5cc0c2bb0 2023-10-25T00:08:21.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5cb59e640 1 -- 172.21.15.107:0/1585653500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5cc0c2bb0 msgr2=0x7fb5cc0c1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:21.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5cb59e640 1 --2- 172.21.15.107:0/1585653500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5cc0c2bb0 0x7fb5cc0c1720 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc06bfa0 tx=0x7fb5c001c3d0 comp rx=0 tx=0).stop 2023-10-25T00:08:21.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5cb59e640 1 -- 172.21.15.107:0/1585653500 shutdown_connections 2023-10-25T00:08:21.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5cb59e640 1 --2- 172.21.15.107:0/1585653500 >> v2:172.21.15.110:3300/0 conn(0x7fb5cc0c1c60 0x7fb5cc15f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:21.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5cb59e640 1 --2- 172.21.15.107:0/1585653500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5cc0c2bb0 0x7fb5cc0c1720 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:21.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5cb59e640 1 -- 172.21.15.107:0/1585653500 >> 172.21.15.107:0/1585653500 conn(0x7fb5cc0c6cc0 msgr2=0x7fb5cc0c70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:21.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5cb59e640 1 -- 172.21.15.107:0/1585653500 shutdown_connections 2023-10-25T00:08:21.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.787+0000 7fb5cb59e640 1 -- 172.21.15.107:0/1585653500 wait complete. 2023-10-25T00:08:21.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5cb59e640 1 Processor -- start 2023-10-25T00:08:21.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5cb59e640 1 -- start start 2023-10-25T00:08:21.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5cb59e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc0c1c60 0x7fb5cc14c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:21.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5cb59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5cc0c27d0 0x7fb5cc14c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:21.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5cb59e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5cc0c2bb0 0x7fb5cc0bdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:21.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5cb59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc0bf6b0 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5cb59e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc0c0920 con 0x7fb5cc0c1c60 2023-10-25T00:08:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5cb59e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc0c0310 con 0x7fb5cc0c2bb0 2023-10-25T00:08:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5c9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5cc0c27d0 0x7fb5cc14c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5c9d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5cc0c27d0 0x7fb5cc14c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56154/0 (socket says 172.21.15.107:56154) 2023-10-25T00:08:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5c9d9b640 1 -- 172.21.15.107:0/2579404672 learned_addr learned my addr 172.21.15.107:0/2579404672 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:21.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5cad9d640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5cc0c2bb0 0x7fb5cc0bdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5ca59c640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc0c1c60 0x7fb5cc14c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5c9d9b640 1 -- 172.21.15.107:0/2579404672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5cc0c2bb0 msgr2=0x7fb5cc0bdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5c9d9b640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5cc0c2bb0 0x7fb5cc0bdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5c9d9b640 1 -- 172.21.15.107:0/2579404672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc0c1c60 msgr2=0x7fb5cc14c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:21.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5c9d9b640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc0c1c60 0x7fb5cc14c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:21.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5c9d9b640 1 -- 172.21.15.107:0/2579404672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c0009420 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.791+0000 7fb5c9d9b640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5cc0c27d0 0x7fb5cc14c9a0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0000c10 tx=0x7fb5c0019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:21.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.795+0000 7fb5baffd640 1 -- 172.21.15.107:0/2579404672 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c00abe70 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.795+0000 7fb5baffd640 1 -- 172.21.15.107:0/2579404672 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c00b4400 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.795+0000 7fb5cb59e640 1 -- 172.21.15.107:0/2579404672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5cc0a8240 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.795+0000 7fb5baffd640 1 -- 172.21.15.107:0/2579404672 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c00b47b0 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.795+0000 7fb5cb59e640 1 -- 172.21.15.107:0/2579404672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5cc0a8520 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.795+0000 7fb5cb59e640 1 -- 172.21.15.107:0/2579404672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5cc04e050 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.795+0000 7fb5baffd640 1 -- 172.21.15.107:0/2579404672 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb5c00b2040 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.795+0000 7fb5baffd640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5ac065e90 0x7fb5ac068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:21.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.795+0000 7fb5baffd640 1 -- 172.21.15.107:0/2579404672 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 9111+0+0 (secure 0 0 0) 0x7fb5c011e280 con 0x7fb5cc0c27d0 2023-10-25T00:08:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.795+0000 7fb5ca59c640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5ac065e90 0x7fb5ac068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.795+0000 7fb5ca59c640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5ac065e90 0x7fb5ac068350 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc001280 tx=0x7fb5bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:21.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:21.803+0000 7fb5baffd640 1 -- 172.21.15.107:0/2579404672 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb5c00ead20 con 0x7fb5cc0c27d0 2023-10-25T00:08:22.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.007+0000 7fb5cb59e640 1 -- 172.21.15.107:0/2579404672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5cc0acc40 con 0x7fb5cc0c27d0 2023-10-25T00:08:22.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.007+0000 7fb5baffd640 1 -- 172.21.15.107:0/2579404672 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5780 (secure 0 0 0) 0x7fb5c00c3090 con 0x7fb5cc0c27d0 2023-10-25T00:08:22.009 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:22.009 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7512,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":135}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:07:51.732943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7305":{"gid":7305,"name":"a","rank":0,"incarnation":126,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6837/1317796403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1317796403},{"type":"v1","addr":"172.21.15.107:6837","nonce":1317796403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:22.009 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 135 2023-10-25T00:08:22.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 -- 172.21.15.107:0/2579404672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5ac065e90 msgr2=0x7fb5ac068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:22.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5ac065e90 0x7fb5ac068350 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc001280 tx=0x7fb5bc009040 comp rx=0 tx=0).stop 2023-10-25T00:08:22.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 -- 172.21.15.107:0/2579404672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5cc0c27d0 msgr2=0x7fb5cc14c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:22.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5cc0c27d0 0x7fb5cc14c9a0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0000c10 tx=0x7fb5c0019170 comp rx=0 tx=0).stop 2023-10-25T00:08:22.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 -- 172.21.15.107:0/2579404672 shutdown_connections 2023-10-25T00:08:22.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb5ac065e90 0x7fb5ac068350 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:22.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb5cc0c2bb0 0x7fb5cc0bdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:22.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb5cc0c27d0 0x7fb5cc14c9a0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:22.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 --2- 172.21.15.107:0/2579404672 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb5cc0c1c60 0x7fb5cc14c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:22.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 -- 172.21.15.107:0/2579404672 >> 172.21.15.107:0/2579404672 conn(0x7fb5cc0c6cc0 msgr2=0x7fb5cc0c36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:22.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 -- 172.21.15.107:0/2579404672 shutdown_connections 2023-10-25T00:08:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:22.011+0000 7fb5cb59e640 1 -- 172.21.15.107:0/2579404672 wait complete. 2023-10-25T00:08:22.031 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:08:24.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:24.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f85c2640 1 Processor -- start 2023-10-25T00:08:24.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f85c2640 1 -- start start 2023-10-25T00:08:24.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f85c2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f70f014fc30 0x7f70f0153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f85c2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70f00c5c20 0x7f70f006df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f85c2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f0072330 con 0x7f70f006e470 2023-10-25T00:08:24.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f85c2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f0155ca0 con 0x7f70f014fc30 2023-10-25T00:08:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f85c2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f0155f80 con 0x7f70f00c5c20 2023-10-25T00:08:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f5b5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70f00c5c20 0x7f70f006df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f635e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f70f014fc30 0x7f70f0153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f635e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f70f014fc30 0x7f70f0153190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35440/0 (socket says 172.21.15.107:35440) 2023-10-25T00:08:24.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f635e640 1 -- 172.21.15.107:0/2110563145 learned_addr learned my addr 172.21.15.107:0/2110563145 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f6b5f640 1 --1- 172.21.15.107:0/2110563145 >> v1:172.21.15.110:6789/0 conn(0x7f70f006e470 0x7f70f00c18a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:44224/0 (socket says 172.21.15.107:44224) 2023-10-25T00:08:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f635e640 1 -- 172.21.15.107:0/2110563145 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70f00c5c20 msgr2=0x7f70f006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f635e640 1 --2- 172.21.15.107:0/2110563145 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70f00c5c20 0x7f70f006df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f635e640 1 -- 172.21.15.107:0/2110563145 >> v1:172.21.15.110:6789/0 conn(0x7f70f006e470 legacy=0x7f70f00c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:24.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.163+0000 7f70f635e640 1 -- 172.21.15.107:0/2110563145 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f0156260 con 0x7f70f014fc30 2023-10-25T00:08:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f635e640 1 --2- 172.21.15.107:0/2110563145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70f014fc30 0x7f70f0153190 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f70ec009870 tx=0x7f70ec01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=610cae6c8aed9ca8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f535c640 1 -- 172.21.15.107:0/2110563145 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec0ab680 con 0x7f70f014fc30 2023-10-25T00:08:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f535c640 1 -- 172.21.15.107:0/2110563145 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ec0ab7e0 con 0x7f70f014fc30 2023-10-25T00:08:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f535c640 1 -- 172.21.15.107:0/2110563145 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec0abb70 con 0x7f70f014fc30 2023-10-25T00:08:24.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f85c2640 1 -- 172.21.15.107:0/2110563145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70f014fc30 msgr2=0x7f70f0153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f85c2640 1 --2- 172.21.15.107:0/2110563145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70f014fc30 0x7f70f0153190 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f70ec009870 tx=0x7f70ec01c480 comp rx=0 tx=0).stop 2023-10-25T00:08:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f85c2640 1 -- 172.21.15.107:0/2110563145 shutdown_connections 2023-10-25T00:08:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f85c2640 1 --2- 172.21.15.107:0/2110563145 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70f00c5c20 0x7f70f006df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f85c2640 1 --2- 172.21.15.107:0/2110563145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70f014fc30 0x7f70f0153190 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f85c2640 1 -- 172.21.15.107:0/2110563145 >> 172.21.15.107:0/2110563145 conn(0x7f70f00c39a0 msgr2=0x7f70f00c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f85c2640 1 -- 172.21.15.107:0/2110563145 shutdown_connections 2023-10-25T00:08:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f85c2640 1 -- 172.21.15.107:0/2110563145 wait complete. 2023-10-25T00:08:24.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f85c2640 1 Processor -- start 2023-10-25T00:08:24.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.167+0000 7f70f85c2640 1 -- start start 2023-10-25T00:08:24.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f85c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70f006e470 0x7f70f0154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f85c2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70f00c5c20 0x7f70f0154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f85c2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70f014fc30 0x7f70f01332a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f85c2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f00c10e0 con 0x7f70f006e470 2023-10-25T00:08:24.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f85c2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f01551c0 con 0x7f70f00c5c20 2023-10-25T00:08:24.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f85c2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f0153aa0 con 0x7f70f014fc30 2023-10-25T00:08:24.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f635e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70f006e470 0x7f70f0154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f635e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70f006e470 0x7f70f0154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59180/0 (socket says 172.21.15.107:59180) 2023-10-25T00:08:24.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f635e640 1 -- 172.21.15.107:0/3606649379 learned_addr learned my addr 172.21.15.107:0/3606649379 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:24.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f5b5d640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70f00c5c20 0x7f70f0154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f6b5f640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70f014fc30 0x7f70f01332a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f635e640 1 -- 172.21.15.107:0/3606649379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70f014fc30 msgr2=0x7f70f01332a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:24.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f635e640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70f014fc30 0x7f70f01332a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f635e640 1 -- 172.21.15.107:0/3606649379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70f00c5c20 msgr2=0x7f70f0154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:24.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f635e640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70f00c5c20 0x7f70f0154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f635e640 1 -- 172.21.15.107:0/3606649379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ec0094d0 con 0x7f70f006e470 2023-10-25T00:08:24.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f5b5d640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70f00c5c20 0x7f70f0154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f635e640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70f006e470 0x7f70f0154630 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f70ec01cd20 tx=0x7f70ec01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70e6ffd640 1 -- 172.21.15.107:0/3606649379 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec0b4800 con 0x7f70f006e470 2023-10-25T00:08:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70e6ffd640 1 -- 172.21.15.107:0/3606649379 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ec0b4960 con 0x7f70f006e470 2023-10-25T00:08:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f85c2640 1 -- 172.21.15.107:0/3606649379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f01345c0 con 0x7f70f006e470 2023-10-25T00:08:24.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70e6ffd640 1 -- 172.21.15.107:0/3606649379 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ec0b4cf0 con 0x7f70f006e470 2023-10-25T00:08:24.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f85c2640 1 -- 172.21.15.107:0/3606649379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f0153d60 con 0x7f70f006e470 2023-10-25T00:08:24.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f85c2640 1 -- 172.21.15.107:0/3606649379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f00746e0 con 0x7f70f006e470 2023-10-25T00:08:24.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70e6ffd640 1 -- 172.21.15.107:0/3606649379 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f70ec0c7440 con 0x7f70f006e470 2023-10-25T00:08:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70e6ffd640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d4065e60 0x7f70d4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70e6ffd640 1 -- 172.21.15.107:0/3606649379 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7f70ec11f3b0 con 0x7f70f006e470 2023-10-25T00:08:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.171+0000 7f70f5b5d640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d4065e60 0x7f70d4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.175+0000 7f70f5b5d640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d4065e60 0x7f70d4068320 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f70f006c560 tx=0x7f70e0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:24.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.179+0000 7f70e6ffd640 1 -- 172.21.15.107:0/3606649379 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f70ec0ebd80 con 0x7f70f006e470 2023-10-25T00:08:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 -- 172.21.15.107:0/3606649379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70f00c5a00 con 0x7f70f006e470 2023-10-25T00:08:24.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70e6ffd640 1 -- 172.21.15.107:0/3606649379 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+5018 (secure 0 0 0) 0x7f70ec0ef3a0 con 0x7f70f006e470 2023-10-25T00:08:24.395 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:24.396 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:23.684070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:24.396 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 136 2023-10-25T00:08:24.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 -- 172.21.15.107:0/3606649379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d4065e60 msgr2=0x7f70d4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:24.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d4065e60 0x7f70d4068320 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f70f006c560 tx=0x7f70e0008040 comp rx=0 tx=0).stop 2023-10-25T00:08:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 -- 172.21.15.107:0/3606649379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70f006e470 msgr2=0x7f70f0154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70f006e470 0x7f70f0154630 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f70ec01cd20 tx=0x7f70ec01acc0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 -- 172.21.15.107:0/3606649379 shutdown_connections 2023-10-25T00:08:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f70d4065e60 0x7f70d4068320 secure :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f70f006c560 tx=0x7f70e0008040 comp rx=0 tx=0).stop 2023-10-25T00:08:24.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70f014fc30 0x7f70f01332a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70f00c5c20 0x7f70f0154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 --2- 172.21.15.107:0/3606649379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70f006e470 0x7f70f0154630 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 -- 172.21.15.107:0/3606649379 >> 172.21.15.107:0/3606649379 conn(0x7f70f00c39a0 msgr2=0x7f70f00c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 -- 172.21.15.107:0/3606649379 shutdown_connections 2023-10-25T00:08:24.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.395+0000 7f70f85c2640 1 -- 172.21.15.107:0/3606649379 wait complete. 2023-10-25T00:08:24.414 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2023-10-25T00:08:24.415 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:08:24.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56eef5640 1 Processor -- start 2023-10-25T00:08:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56eef5640 1 -- start start 2023-10-25T00:08:24.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56eef5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe56806ec90 0x7fe56814c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56eef5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe56814c9e0 0x7fe5680c3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56eef5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5680c59d0 con 0x7fe568155ea0 2023-10-25T00:08:24.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56eef5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5680c2990 con 0x7fe56814c9e0 2023-10-25T00:08:24.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56eef5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5680c2ec0 con 0x7fe56806ec90 2023-10-25T00:08:24.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56d492640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe56814c9e0 0x7fe5680c3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56d492640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe56814c9e0 0x7fe5680c3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59184/0 (socket says 172.21.15.107:59184) 2023-10-25T00:08:24.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56d492640 1 -- 172.21.15.107:0/3945581745 learned_addr learned my addr 172.21.15.107:0/3945581745 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:24.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe567fff640 1 --2- 172.21.15.107:0/3945581745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe56806ec90 0x7fe56814c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56d492640 1 -- 172.21.15.107:0/3945581745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe56806ec90 msgr2=0x7fe56814c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:24.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56d492640 1 --2- 172.21.15.107:0/3945581745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe56806ec90 0x7fe56814c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56d492640 1 -- 172.21.15.107:0/3945581745 >> v1:172.21.15.110:6789/0 conn(0x7fe568155ea0 legacy=0x7fe568156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:24.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.535+0000 7fe56d492640 1 -- 172.21.15.107:0/3945581745 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56806d500 con 0x7fe56814c9e0 2023-10-25T00:08:24.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe56d492640 1 --2- 172.21.15.107:0/3945581745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe56814c9e0 0x7fe5680c3e00 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fe560009e50 tx=0x7fe56001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3030ae9ad7aa0b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:24.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe5677fe640 1 -- 172.21.15.107:0/3945581745 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600a5620 con 0x7fe56814c9e0 2023-10-25T00:08:24.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe5677fe640 1 -- 172.21.15.107:0/3945581745 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5600a5780 con 0x7fe56814c9e0 2023-10-25T00:08:24.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe5677fe640 1 -- 172.21.15.107:0/3945581745 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600a5b10 con 0x7fe56814c9e0 2023-10-25T00:08:24.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe56eef5640 1 -- 172.21.15.107:0/3945581745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe56814c9e0 msgr2=0x7fe5680c3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:24.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe56eef5640 1 --2- 172.21.15.107:0/3945581745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe56814c9e0 0x7fe5680c3e00 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fe560009e50 tx=0x7fe56001c8b0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe56eef5640 1 -- 172.21.15.107:0/3945581745 shutdown_connections 2023-10-25T00:08:24.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe56eef5640 1 --2- 172.21.15.107:0/3945581745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe56814c9e0 0x7fe5680c3e00 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe56eef5640 1 --2- 172.21.15.107:0/3945581745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe56806ec90 0x7fe56814c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe56eef5640 1 -- 172.21.15.107:0/3945581745 >> 172.21.15.107:0/3945581745 conn(0x7fe5680c4a90 msgr2=0x7fe5680c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe56eef5640 1 -- 172.21.15.107:0/3945581745 shutdown_connections 2023-10-25T00:08:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe56eef5640 1 -- 172.21.15.107:0/3945581745 wait complete. 2023-10-25T00:08:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe56eef5640 1 Processor -- start 2023-10-25T00:08:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.539+0000 7fe56eef5640 1 -- start start 2023-10-25T00:08:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56eef5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe56806ec90 0x7fe5681eed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56eef5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe56814c9e0 0x7fe568147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56eef5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe568155ea0 0x7fe5681eac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56eef5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56814db10 con 0x7fe56806ec90 2023-10-25T00:08:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56eef5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe568146740 con 0x7fe568155ea0 2023-10-25T00:08:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56eef5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe568144bc0 con 0x7fe56814c9e0 2023-10-25T00:08:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56cc91640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe56806ec90 0x7fe5681eed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56cc91640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe56806ec90 0x7fe5681eed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59196/0 (socket says 172.21.15.107:59196) 2023-10-25T00:08:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56cc91640 1 -- 172.21.15.107:0/1358764138 learned_addr learned my addr 172.21.15.107:0/1358764138 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56d492640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe568155ea0 0x7fe5681eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe567fff640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe56814c9e0 0x7fe568147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56cc91640 1 -- 172.21.15.107:0/1358764138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe56814c9e0 msgr2=0x7fe568147050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56cc91640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe56814c9e0 0x7fe568147050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56cc91640 1 -- 172.21.15.107:0/1358764138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe568155ea0 msgr2=0x7fe5681eac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56cc91640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe568155ea0 0x7fe5681eac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56cc91640 1 -- 172.21.15.107:0/1358764138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe560009b30 con 0x7fe56806ec90 2023-10-25T00:08:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56d492640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe568155ea0 0x7fe5681eac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:08:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56cc91640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe56806ec90 0x7fe5681eed30 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00c450 tx=0x7fe55c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe5657fa640 1 -- 172.21.15.107:0/1358764138 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c012830 con 0x7fe56806ec90 2023-10-25T00:08:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe5657fa640 1 -- 172.21.15.107:0/1358764138 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55c0129b0 con 0x7fe56806ec90 2023-10-25T00:08:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe5657fa640 1 -- 172.21.15.107:0/1358764138 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c012d60 con 0x7fe56806ec90 2023-10-25T00:08:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56eef5640 1 -- 172.21.15.107:0/1358764138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5681ee250 con 0x7fe56806ec90 2023-10-25T00:08:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56eef5640 1 -- 172.21.15.107:0/1358764138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5681ee810 con 0x7fe56806ec90 2023-10-25T00:08:24.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe56eef5640 1 -- 172.21.15.107:0/1358764138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5680746e0 con 0x7fe56806ec90 2023-10-25T00:08:24.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe5657fa640 1 -- 172.21.15.107:0/1358764138 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe55c01c460 con 0x7fe56806ec90 2023-10-25T00:08:24.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe5657fa640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe54c065e60 0x7fe54c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:24.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe5657fa640 1 -- 172.21.15.107:0/1358764138 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7fe55c084270 con 0x7fe56806ec90 2023-10-25T00:08:24.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.543+0000 7fe567fff640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe54c065e60 0x7fe54c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:24.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.547+0000 7fe567fff640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe54c065e60 0x7fe54c068320 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe568147920 tx=0x7fe554008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:24.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.551+0000 7fe5657fa640 1 -- 172.21.15.107:0/1358764138 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe55c050cc0 con 0x7fe56806ec90 2023-10-25T00:08:24.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 -- 172.21.15.107:0/1358764138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe568155a20 con 0x7fe56806ec90 2023-10-25T00:08:24.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe5657fa640 1 -- 172.21.15.107:0/1358764138 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+5018 (secure 0 0 0) 0x7fe55c0542e0 con 0x7fe56806ec90 2023-10-25T00:08:24.755 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:24.755 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:23.684070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 136 2023-10-25T00:08:24.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 -- 172.21.15.107:0/1358764138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe54c065e60 msgr2=0x7fe54c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe54c065e60 0x7fe54c068320 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe568147920 tx=0x7fe554008040 comp rx=0 tx=0).stop 2023-10-25T00:08:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 -- 172.21.15.107:0/1358764138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe56806ec90 msgr2=0x7fe5681eed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe56806ec90 0x7fe5681eed30 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00c450 tx=0x7fe55c00c910 comp rx=0 tx=0).stop 2023-10-25T00:08:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 -- 172.21.15.107:0/1358764138 shutdown_connections 2023-10-25T00:08:24.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe568155ea0 0x7fe5681eac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe54c065e60 0x7fe54c068320 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe56814c9e0 0x7fe568147050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 --2- 172.21.15.107:0/1358764138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe56806ec90 0x7fe5681eed30 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 -- 172.21.15.107:0/1358764138 >> 172.21.15.107:0/1358764138 conn(0x7fe5680c4a90 msgr2=0x7fe5680ab030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:24.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 -- 172.21.15.107:0/1358764138 shutdown_connections 2023-10-25T00:08:24.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:24.755+0000 7fe56eef5640 1 -- 172.21.15.107:0/1358764138 wait complete. 2023-10-25T00:08:24.776 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 136, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 134, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6394": { "addr": "172.21.15.110:6835/3836115666", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 3836115666, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 3836115666, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6394, "incarnation": 131, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:08:11.734690+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6394 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 136, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7512": { "addr": "172.21.15.107:6835/2276136116", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2276136116, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 2276136116, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7512, "incarnation": 136, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:08:23.684070+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7512 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:08:26.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:26.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fc453640 1 Processor -- start 2023-10-25T00:08:26.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fc453640 1 -- start start 2023-10-25T00:08:26.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fc453640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98f4070b00 0x7f98f40c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:26.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fc453640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f98f406df30 0x7f98f406e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:26.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fc453640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f40c22f0 con 0x7f98f414eb70 2023-10-25T00:08:26.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fc453640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f4071700 con 0x7f98f406df30 2023-10-25T00:08:26.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fc453640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f40c25d0 con 0x7f98f4070b00 2023-10-25T00:08:26.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fa9f0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f98f406df30 0x7f98f406e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:26.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fa9f0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f98f406df30 0x7f98f406e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59198/0 (socket says 172.21.15.107:59198) 2023-10-25T00:08:26.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fa9f0640 1 -- 172.21.15.107:0/917987781 learned_addr learned my addr 172.21.15.107:0/917987781 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:26.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98f99ee640 1 --2- 172.21.15.107:0/917987781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98f4070b00 0x7f98f40c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:26.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fa9f0640 1 -- 172.21.15.107:0/917987781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98f4070b00 msgr2=0x7f98f40c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:26.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fa9f0640 1 --2- 172.21.15.107:0/917987781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98f4070b00 0x7f98f40c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:26.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fa9f0640 1 -- 172.21.15.107:0/917987781 >> v1:172.21.15.110:6789/0 conn(0x7f98f414eb70 legacy=0x7f98f41514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:26.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.919+0000 7f98fa9f0640 1 -- 172.21.15.107:0/917987781 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f40c2be0 con 0x7f98f406df30 2023-10-25T00:08:26.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98fa9f0640 1 --2- 172.21.15.107:0/917987781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f406df30 0x7f98f406e360 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00d150 tx=0x7f98ec01ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9f58f6469b38f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:26.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98f91ed640 1 -- 172.21.15.107:0/917987781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec00b2d0 con 0x7f98f406df30 2023-10-25T00:08:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98f91ed640 1 -- 172.21.15.107:0/917987781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec00b430 con 0x7f98f406df30 2023-10-25T00:08:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98f91ed640 1 -- 172.21.15.107:0/917987781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec0a7ba0 con 0x7f98f406df30 2023-10-25T00:08:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98fc453640 1 -- 172.21.15.107:0/917987781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f406df30 msgr2=0x7f98f406e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98fc453640 1 --2- 172.21.15.107:0/917987781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f406df30 0x7f98f406e360 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00d150 tx=0x7f98ec01ebd0 comp rx=0 tx=0).stop 2023-10-25T00:08:26.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98fc453640 1 -- 172.21.15.107:0/917987781 shutdown_connections 2023-10-25T00:08:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98fc453640 1 --2- 172.21.15.107:0/917987781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f406df30 0x7f98f406e360 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98fc453640 1 --2- 172.21.15.107:0/917987781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98f4070b00 0x7f98f40c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98fc453640 1 -- 172.21.15.107:0/917987781 >> 172.21.15.107:0/917987781 conn(0x7f98f40c39a0 msgr2=0x7f98f40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98fc453640 1 -- 172.21.15.107:0/917987781 shutdown_connections 2023-10-25T00:08:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.923+0000 7f98fc453640 1 -- 172.21.15.107:0/917987781 wait complete. 2023-10-25T00:08:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fc453640 1 Processor -- start 2023-10-25T00:08:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fc453640 1 -- start start 2023-10-25T00:08:26.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fc453640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f406df30 0x7f98f41e9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:26.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fc453640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98f4070b00 0x7f98f4144db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:26.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fc453640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98f414eb70 0x7f98f41ec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:26.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fc453640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f41452f0 con 0x7f98f406df30 2023-10-25T00:08:26.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fc453640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f4153170 con 0x7f98f4070b00 2023-10-25T00:08:26.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fc453640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f41528d0 con 0x7f98f414eb70 2023-10-25T00:08:26.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fa1ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f406df30 0x7f98f41e9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:26.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fa1ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f406df30 0x7f98f41e9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59208/0 (socket says 172.21.15.107:59208) 2023-10-25T00:08:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fa1ef640 1 -- 172.21.15.107:0/403959096 learned_addr learned my addr 172.21.15.107:0/403959096 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98f99ee640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98f4070b00 0x7f98f4144db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fa9f0640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98f414eb70 0x7f98f41ec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fa1ef640 1 -- 172.21.15.107:0/403959096 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98f414eb70 msgr2=0x7f98f41ec0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fa1ef640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98f414eb70 0x7f98f41ec0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fa1ef640 1 -- 172.21.15.107:0/403959096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98f4070b00 msgr2=0x7f98f4144db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fa1ef640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98f4070b00 0x7f98f4144db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fa1ef640 1 -- 172.21.15.107:0/403959096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec00bd40 con 0x7f98f406df30 2023-10-25T00:08:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fa1ef640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f406df30 0x7f98f41e9740 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f98f000c450 tx=0x7f98f000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98eaffd640 1 -- 172.21.15.107:0/403959096 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0012830 con 0x7f98f406df30 2023-10-25T00:08:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98eaffd640 1 -- 172.21.15.107:0/403959096 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f00129b0 con 0x7f98f406df30 2023-10-25T00:08:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fc453640 1 -- 172.21.15.107:0/403959096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f4152bb0 con 0x7f98f406df30 2023-10-25T00:08:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98eaffd640 1 -- 172.21.15.107:0/403959096 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0012d60 con 0x7f98f406df30 2023-10-25T00:08:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fc453640 1 -- 172.21.15.107:0/403959096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f41eb8e0 con 0x7f98f406df30 2023-10-25T00:08:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98fc453640 1 -- 172.21.15.107:0/403959096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f40746e0 con 0x7f98f406df30 2023-10-25T00:08:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.927+0000 7f98eaffd640 1 -- 172.21.15.107:0/403959096 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f98f001c460 con 0x7f98f406df30 2023-10-25T00:08:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.931+0000 7f98eaffd640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f98d8065e90 0x7f98d8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.931+0000 7f98eaffd640 1 -- 172.21.15.107:0/403959096 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7f98f00842e0 con 0x7f98f406df30 2023-10-25T00:08:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.931+0000 7f98f99ee640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f98d8065e90 0x7f98d8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.931+0000 7f98f99ee640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f98d8065e90 0x7f98d8068350 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f98f40c4cd0 tx=0x7f98e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:26.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:26.935+0000 7f98eaffd640 1 -- 172.21.15.107:0/403959096 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f98f0050cd0 con 0x7f98f406df30 2023-10-25T00:08:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.143+0000 7f98fc453640 1 -- 172.21.15.107:0/403959096 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98f4070540 con 0x7f98f406df30 2023-10-25T00:08:27.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.143+0000 7f98eaffd640 1 -- 172.21.15.107:0/403959096 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5018 (secure 0 0 0) 0x7f98f00542f0 con 0x7f98f406df30 2023-10-25T00:08:27.145 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:27.145 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:27.146 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 139 2023-10-25T00:08:27.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 -- 172.21.15.107:0/403959096 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f98d8065e90 msgr2=0x7f98d8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:27.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f98d8065e90 0x7f98d8068350 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f98f40c4cd0 tx=0x7f98e4008040 comp rx=0 tx=0).stop 2023-10-25T00:08:27.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 -- 172.21.15.107:0/403959096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f406df30 msgr2=0x7f98f41e9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:27.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f406df30 0x7f98f41e9740 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f98f000c450 tx=0x7f98f000c910 comp rx=0 tx=0).stop 2023-10-25T00:08:27.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 -- 172.21.15.107:0/403959096 shutdown_connections 2023-10-25T00:08:27.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f98d8065e90 0x7f98d8068350 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:27.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98f414eb70 0x7f98f41ec0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:27.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98f4070b00 0x7f98f4144db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:27.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 --2- 172.21.15.107:0/403959096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98f406df30 0x7f98f41e9740 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:27.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 -- 172.21.15.107:0/403959096 >> 172.21.15.107:0/403959096 conn(0x7f98f40c39a0 msgr2=0x7f98f40c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:27.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 -- 172.21.15.107:0/403959096 shutdown_connections 2023-10-25T00:08:27.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:27.147+0000 7f98fc453640 1 -- 172.21.15.107:0/403959096 wait complete. 2023-10-25T00:08:27.165 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=7305 2023-10-25T00:08:27.165 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.a 2023-10-25T00:08:28.166 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2023-10-25T00:08:28.167 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:08:28.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:08:28.170 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:08:28.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:28.245 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:08:28.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.303+0000 7f0e433aa640 1 Processor -- start 2023-10-25T00:08:28.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.303+0000 7f0e433aa640 1 -- start start 2023-10-25T00:08:28.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.303+0000 7f0e433aa640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0e3c14fbd0 0x7f0e3c06df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:28.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.303+0000 7f0e433aa640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3c155f20 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.303+0000 7f0e433aa640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3c0c6d40 con 0x7f0e3c0c4e20 2023-10-25T00:08:28.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.303+0000 7f0e433aa640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e3c153560 con 0x7f0e3c14fbd0 2023-10-25T00:08:28.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.303+0000 7f0e41947640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0e3c0c5ba0 0x7f0e3c15cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46956/0 (socket says 172.21.15.107:46956) 2023-10-25T00:08:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.303+0000 7f0e41947640 1 -- 172.21.15.107:0/751799282 learned_addr learned my addr 172.21.15.107:0/751799282 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.303+0000 7f0e41146640 1 --2- 172.21.15.107:0/751799282 >> v2:172.21.15.110:3300/0 conn(0x7f0e3c14fbd0 0x7f0e3c06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.303+0000 7f0e33fff640 1 -- 172.21.15.107:0/751799282 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55302127 0 0) 0x7f0e3c155f20 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e33fff640 1 -- 172.21.15.107:0/751799282 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e28003530 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e33fff640 1 -- 172.21.15.107:0/751799282 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2403501259 0 0) 0x7f0e28003530 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e41146640 1 -- 172.21.15.107:0/751799282 >> v1:172.21.15.110:6789/0 conn(0x7f0e3c0c4e20 legacy=0x7f0e3c0c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:28.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e41146640 1 -- 172.21.15.107:0/751799282 >> v1:172.21.15.107:6789/0 conn(0x7f0e3c0c5ba0 legacy=0x7f0e3c15cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:28.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e41146640 1 -- 172.21.15.107:0/751799282 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c1527a0 con 0x7f0e3c14fbd0 2023-10-25T00:08:28.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e41146640 1 --2- 172.21.15.107:0/751799282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e3c14fbd0 0x7f0e3c06df30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0e380097c0 tx=0x7f0e3801c310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4687076ffbd7c551 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:28.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e33fff640 1 -- 172.21.15.107:0/751799282 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e34001a00 con 0x7f0e3c14fbd0 2023-10-25T00:08:28.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e33fff640 1 -- 172.21.15.107:0/751799282 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e380a7050 con 0x7f0e3c14fbd0 2023-10-25T00:08:28.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e33fff640 1 -- 172.21.15.107:0/751799282 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e380ab620 con 0x7f0e3c14fbd0 2023-10-25T00:08:28.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e433aa640 1 -- 172.21.15.107:0/751799282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e3c14fbd0 msgr2=0x7f0e3c06df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:28.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e433aa640 1 --2- 172.21.15.107:0/751799282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e3c14fbd0 0x7f0e3c06df30 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f0e380097c0 tx=0x7f0e3801c310 comp rx=0 tx=0).stop 2023-10-25T00:08:28.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e433aa640 1 -- 172.21.15.107:0/751799282 shutdown_connections 2023-10-25T00:08:28.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e433aa640 1 --2- 172.21.15.107:0/751799282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e3c14fbd0 0x7f0e3c06df30 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:28.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e433aa640 1 -- 172.21.15.107:0/751799282 >> 172.21.15.107:0/751799282 conn(0x7f0e3c0c39a0 msgr2=0x7f0e3c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:28.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e433aa640 1 -- 172.21.15.107:0/751799282 shutdown_connections 2023-10-25T00:08:28.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.307+0000 7f0e433aa640 1 -- 172.21.15.107:0/751799282 wait complete. 2023-10-25T00:08:28.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e433aa640 1 Processor -- start 2023-10-25T00:08:28.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e433aa640 1 -- start start 2023-10-25T00:08:28.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e433aa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e3c0c4e20 0x7f0e3c14b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:28.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e433aa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e3c0c5ba0 0x7f0e3c154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:28.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e433aa640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e3c14fbd0 0x7f0e3c155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:28.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e433aa640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c1479d0 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e433aa640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c148c00 con 0x7f0e3c14fbd0 2023-10-25T00:08:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e433aa640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c147030 con 0x7f0e3c0c4e20 2023-10-25T00:08:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e40945640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e3c0c5ba0 0x7f0e3c154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e40945640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e3c0c5ba0 0x7f0e3c154ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59238/0 (socket says 172.21.15.107:59238) 2023-10-25T00:08:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e40945640 1 -- 172.21.15.107:0/1183010905 learned_addr learned my addr 172.21.15.107:0/1183010905 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e41146640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e3c0c4e20 0x7f0e3c14b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e41947640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e3c14fbd0 0x7f0e3c155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e40945640 1 -- 172.21.15.107:0/1183010905 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e3c0c4e20 msgr2=0x7f0e3c14b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e40945640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e3c0c4e20 0x7f0e3c14b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e40945640 1 -- 172.21.15.107:0/1183010905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e3c14fbd0 msgr2=0x7f0e3c155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e40945640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e3c14fbd0 0x7f0e3c155430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e40945640 1 -- 172.21.15.107:0/1183010905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e38009420 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e41146640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e3c0c4e20 0x7f0e3c14b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:08:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e41947640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e3c14fbd0 0x7f0e3c155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:08:28.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e40945640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e3c0c5ba0 0x7f0e3c154ef0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00c500 tx=0x7f0e2c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e31ffb640 1 -- 172.21.15.107:0/1183010905 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c005220 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e31ffb640 1 -- 172.21.15.107:0/1183010905 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c005380 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e433aa640 1 -- 172.21.15.107:0/1183010905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3c147310 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e31ffb640 1 -- 172.21.15.107:0/1183010905 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c007270 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.311+0000 7f0e433aa640 1 -- 172.21.15.107:0/1183010905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3c14bd60 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.315+0000 7f0e433aa640 1 -- 172.21.15.107:0/1183010905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3c0746e0 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.315+0000 7f0e31ffb640 1 -- 172.21.15.107:0/1183010905 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0e2c0073d0 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.315+0000 7f0e31ffb640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e20065e60 0x7f0e20068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.315+0000 7f0e41146640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e20065e60 0x7f0e20068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.315+0000 7f0e31ffb640 1 -- 172.21.15.107:0/1183010905 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7f0e2c083870 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.315+0000 7f0e41146640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e20065e60 0x7f0e20068320 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0e380ad000 tx=0x7f0e380a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:28.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.319+0000 7f0e31ffb640 1 -- 172.21.15.107:0/1183010905 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0e2c050240 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.543+0000 7f0e433aa640 1 -- 172.21.15.107:0/1183010905 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e3c15e480 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e31ffb640 1 -- 172.21.15.107:0/1183010905 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+5018 (secure 0 0 0) 0x7f0e2c053860 con 0x7f0e3c0c5ba0 2023-10-25T00:08:28.546 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:28.546 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:28.547 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 139 2023-10-25T00:08:28.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 -- 172.21.15.107:0/1183010905 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e20065e60 msgr2=0x7f0e20068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:28.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e20065e60 0x7f0e20068320 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0e380ad000 tx=0x7f0e380a6000 comp rx=0 tx=0).stop 2023-10-25T00:08:28.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 -- 172.21.15.107:0/1183010905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e3c0c5ba0 msgr2=0x7f0e3c154ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e3c0c5ba0 0x7f0e3c154ef0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00c500 tx=0x7f0e2c00c9c0 comp rx=0 tx=0).stop 2023-10-25T00:08:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 -- 172.21.15.107:0/1183010905 shutdown_connections 2023-10-25T00:08:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e3c14fbd0 0x7f0e3c155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e3c0c5ba0 0x7f0e3c154ef0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:28.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e20065e60 0x7f0e20068320 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:28.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 --2- 172.21.15.107:0/1183010905 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e3c0c4e20 0x7f0e3c14b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:28.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 -- 172.21.15.107:0/1183010905 >> 172.21.15.107:0/1183010905 conn(0x7f0e3c0c39a0 msgr2=0x7f0e3c0ab4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:28.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 -- 172.21.15.107:0/1183010905 shutdown_connections 2023-10-25T00:08:28.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:28.547+0000 7f0e433aa640 1 -- 172.21.15.107:0/1183010905 wait complete. 2023-10-25T00:08:28.569 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-25T00:08:29.757 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:08:29.755+0000 7f77c62f5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:08:30.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:30.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f4156d81640 1 Processor -- start 2023-10-25T00:08:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f4156d81640 1 -- start start 2023-10-25T00:08:30.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f4156d81640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f415013bb00 0x7f41501531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:30.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f4156d81640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41500aa330 0x7f415006e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:30.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f4156d81640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f41500a9450 0x7f4150071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:30.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f4156d81640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4150156100 con 0x7f415013bb00 2023-10-25T00:08:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f4156d81640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41501527c0 con 0x7f41500a9450 2023-10-25T00:08:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f4156d81640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4150152d40 con 0x7f41500aa330 2023-10-25T00:08:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f4154b1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f415013bb00 0x7f41501531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f4154b1d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f415013bb00 0x7f41501531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59246/0 (socket says 172.21.15.107:59246) 2023-10-25T00:08:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f4154b1d640 1 -- 172.21.15.107:0/3782270327 learned_addr learned my addr 172.21.15.107:0/3782270327 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f414ffff640 1 --2- 172.21.15.107:0/3782270327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41500aa330 0x7f415006e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.695+0000 7f415531e640 1 --2- 172.21.15.107:0/3782270327 >> v2:172.21.15.110:3300/0 conn(0x7f41500a9450 0x7f4150071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4154b1d640 1 -- 172.21.15.107:0/3782270327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41500aa330 msgr2=0x7f415006e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4154b1d640 1 --2- 172.21.15.107:0/3782270327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41500aa330 0x7f415006e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4154b1d640 1 -- 172.21.15.107:0/3782270327 >> v2:172.21.15.110:3300/0 conn(0x7f41500a9450 msgr2=0x7f4150071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:30.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4154b1d640 1 --2- 172.21.15.107:0/3782270327 >> v2:172.21.15.110:3300/0 conn(0x7f41500a9450 0x7f4150071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4154b1d640 1 -- 172.21.15.107:0/3782270327 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415006d660 con 0x7f415013bb00 2023-10-25T00:08:30.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f415531e640 1 --2- 172.21.15.107:0/3782270327 >> v2:172.21.15.110:3300/0 conn(0x7f41500a9450 0x7f4150071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:08:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4154b1d640 1 --2- 172.21.15.107:0/3782270327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f415013bb00 0x7f41501531d0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f4144009870 tx=0x7f414401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772e473a21d50742 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f414f7fe640 1 -- 172.21.15.107:0/3782270327 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41440ab600 con 0x7f415013bb00 2023-10-25T00:08:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f414f7fe640 1 -- 172.21.15.107:0/3782270327 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41440a7050 con 0x7f415013bb00 2023-10-25T00:08:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f414f7fe640 1 -- 172.21.15.107:0/3782270327 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41440ab980 con 0x7f415013bb00 2023-10-25T00:08:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4156d81640 1 -- 172.21.15.107:0/3782270327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f415013bb00 msgr2=0x7f41501531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4156d81640 1 --2- 172.21.15.107:0/3782270327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f415013bb00 0x7f41501531d0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f4144009870 tx=0x7f414401c3e0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4156d81640 1 -- 172.21.15.107:0/3782270327 shutdown_connections 2023-10-25T00:08:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4156d81640 1 --2- 172.21.15.107:0/3782270327 >> v2:172.21.15.110:3300/0 conn(0x7f41500a9450 0x7f4150071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4156d81640 1 --2- 172.21.15.107:0/3782270327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41500aa330 0x7f415006e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4156d81640 1 --2- 172.21.15.107:0/3782270327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f415013bb00 0x7f41501531d0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4156d81640 1 -- 172.21.15.107:0/3782270327 >> 172.21.15.107:0/3782270327 conn(0x7f4150070f60 msgr2=0x7f4150071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4156d81640 1 -- 172.21.15.107:0/3782270327 shutdown_connections 2023-10-25T00:08:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.699+0000 7f4156d81640 1 -- 172.21.15.107:0/3782270327 wait complete. 2023-10-25T00:08:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4156d81640 1 Processor -- start 2023-10-25T00:08:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4156d81640 1 -- start start 2023-10-25T00:08:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4156d81640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41500a9450 0x7f41501361e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4156d81640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41500aa330 0x7f4150131a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4156d81640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f415013bb00 0x7f4150131f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4156d81640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41501ebcd0 con 0x7f41500aa330 2023-10-25T00:08:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4156d81640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41501eb330 con 0x7f415013bb00 2023-10-25T00:08:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4156d81640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41501eb610 con 0x7f41500a9450 2023-10-25T00:08:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41500aa330 0x7f4150131a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41500aa330 0x7f4150131a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59252/0 (socket says 172.21.15.107:59252) 2023-10-25T00:08:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414ffff640 1 -- 172.21.15.107:0/280768378 learned_addr learned my addr 172.21.15.107:0/280768378 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f415531e640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f415013bb00 0x7f4150131f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4154b1d640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41500a9450 0x7f41501361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414ffff640 1 -- 172.21.15.107:0/280768378 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41500a9450 msgr2=0x7f41501361e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414ffff640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41500a9450 0x7f41501361e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414ffff640 1 -- 172.21.15.107:0/280768378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f415013bb00 msgr2=0x7f4150131f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414ffff640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f415013bb00 0x7f4150131f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414ffff640 1 -- 172.21.15.107:0/280768378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41440094d0 con 0x7f41500aa330 2023-10-25T00:08:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414ffff640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41500aa330 0x7f4150131a40 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f413c00d7d0 tx=0x7f413c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414d7fa640 1 -- 172.21.15.107:0/280768378 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c00b6c0 con 0x7f41500aa330 2023-10-25T00:08:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4156d81640 1 -- 172.21.15.107:0/280768378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41501eb8f0 con 0x7f41500aa330 2023-10-25T00:08:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414d7fa640 1 -- 172.21.15.107:0/280768378 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f413c00b820 con 0x7f41500aa330 2023-10-25T00:08:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f414d7fa640 1 -- 172.21.15.107:0/280768378 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f413c00bbd0 con 0x7f41500aa330 2023-10-25T00:08:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4156d81640 1 -- 172.21.15.107:0/280768378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4150136720 con 0x7f41500aa330 2023-10-25T00:08:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.703+0000 7f4156d81640 1 -- 172.21.15.107:0/280768378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41500746e0 con 0x7f41500aa330 2023-10-25T00:08:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.707+0000 7f414d7fa640 1 -- 172.21.15.107:0/280768378 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f413c00bd30 con 0x7f41500aa330 2023-10-25T00:08:30.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.707+0000 7f414d7fa640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4134065e60 0x7f4134068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:30.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.707+0000 7f414d7fa640 1 -- 172.21.15.107:0/280768378 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7f413c084960 con 0x7f41500aa330 2023-10-25T00:08:30.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.707+0000 7f4154b1d640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4134065e60 0x7f4134068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:30.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.707+0000 7f4154b1d640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4134065e60 0x7f4134068320 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f41440ad000 tx=0x7f4144008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:30.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.711+0000 7f414d7fa640 1 -- 172.21.15.107:0/280768378 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f413c051350 con 0x7f41500aa330 2023-10-25T00:08:30.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 -- 172.21.15.107:0/280768378 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4150071930 con 0x7f41500aa330 2023-10-25T00:08:30.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f414d7fa640 1 -- 172.21.15.107:0/280768378 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5779 (secure 0 0 0) 0x7f413c054970 con 0x7f41500aa330 2023-10-25T00:08:30.963 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:30.963 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:30.964 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 140 2023-10-25T00:08:30.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 -- 172.21.15.107:0/280768378 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4134065e60 msgr2=0x7f4134068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:30.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4134065e60 0x7f4134068320 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f41440ad000 tx=0x7f4144008040 comp rx=0 tx=0).stop 2023-10-25T00:08:30.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 -- 172.21.15.107:0/280768378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41500aa330 msgr2=0x7f4150131a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:30.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41500aa330 0x7f4150131a40 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f413c00d7d0 tx=0x7f413c00dc90 comp rx=0 tx=0).stop 2023-10-25T00:08:30.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 -- 172.21.15.107:0/280768378 shutdown_connections 2023-10-25T00:08:30.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f415013bb00 0x7f4150131f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41500aa330 0x7f4150131a40 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4134065e60 0x7f4134068320 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 --2- 172.21.15.107:0/280768378 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41500a9450 0x7f41501361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:30.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 -- 172.21.15.107:0/280768378 >> 172.21.15.107:0/280768378 conn(0x7f4150070f60 msgr2=0x7f4150073200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:30.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 -- 172.21.15.107:0/280768378 shutdown_connections 2023-10-25T00:08:30.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:30.963+0000 7f4156d81640 1 -- 172.21.15.107:0/280768378 wait complete. 2023-10-25T00:08:30.982 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2023-10-25T00:08:30.982 INFO:tasks.mds_thrash.fs.[b]:waiting for 32 secs before thrashing 2023-10-25T00:08:46.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:46.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.278+0000 7faabd1e1640 1 Processor -- start 2023-10-25T00:08:46.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.278+0000 7faabd1e1640 1 -- start start 2023-10-25T00:08:46.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faabd1e1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faab813bb00 0x7faab81531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faabd1e1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab80aa330 0x7faab806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faabd1e1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab8152d40 con 0x7faab80a9450 2023-10-25T00:08:46.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faabd1e1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab8156100 con 0x7faab813bb00 2023-10-25T00:08:46.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faabd1e1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab81527c0 con 0x7faab80aa330 2023-10-25T00:08:46.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab6d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faab813bb00 0x7faab81531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab6d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faab813bb00 0x7faab81531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36016/0 (socket says 172.21.15.107:36016) 2023-10-25T00:08:46.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab6d9d640 1 -- 172.21.15.107:0/3950867289 learned_addr learned my addr 172.21.15.107:0/3950867289 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:46.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab759e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7faab80a9450 0x7faab8071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45276/0 (socket says 172.21.15.107:45276) 2023-10-25T00:08:46.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab659c640 1 --2- 172.21.15.107:0/3950867289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab80aa330 0x7faab806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab6d9d640 1 -- 172.21.15.107:0/3950867289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab80aa330 msgr2=0x7faab806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:46.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab6d9d640 1 --2- 172.21.15.107:0/3950867289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab80aa330 0x7faab806e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab6d9d640 1 -- 172.21.15.107:0/3950867289 >> v1:172.21.15.107:6789/0 conn(0x7faab80a9450 legacy=0x7faab8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab5d9b640 1 -- 172.21.15.107:0/3950867289 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517003080 0 0) 0x7faab8152d40 con 0x7faab80a9450 2023-10-25T00:08:46.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab6d9d640 1 -- 172.21.15.107:0/3950867289 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab806d660 con 0x7faab813bb00 2023-10-25T00:08:46.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab6d9d640 1 --2- 172.21.15.107:0/3950867289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab813bb00 0x7faab81531d0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7faaa80098a0 tx=0x7faaa801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20721443c18c12b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:46.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab5d9b640 1 -- 172.21.15.107:0/3950867289 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa80ac640 con 0x7faab813bb00 2023-10-25T00:08:46.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab5d9b640 1 -- 172.21.15.107:0/3950867289 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa80ac7a0 con 0x7faab813bb00 2023-10-25T00:08:46.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.282+0000 7faab5d9b640 1 -- 172.21.15.107:0/3950867289 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa80acb30 con 0x7faab813bb00 2023-10-25T00:08:46.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 -- 172.21.15.107:0/3950867289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab813bb00 msgr2=0x7faab81531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 --2- 172.21.15.107:0/3950867289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab813bb00 0x7faab81531d0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7faaa80098a0 tx=0x7faaa801c400 comp rx=0 tx=0).stop 2023-10-25T00:08:46.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 -- 172.21.15.107:0/3950867289 shutdown_connections 2023-10-25T00:08:46.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 --2- 172.21.15.107:0/3950867289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab80aa330 0x7faab806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 --2- 172.21.15.107:0/3950867289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab813bb00 0x7faab81531d0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 -- 172.21.15.107:0/3950867289 >> 172.21.15.107:0/3950867289 conn(0x7faab8070f60 msgr2=0x7faab8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:46.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 -- 172.21.15.107:0/3950867289 shutdown_connections 2023-10-25T00:08:46.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 -- 172.21.15.107:0/3950867289 wait complete. 2023-10-25T00:08:46.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 Processor -- start 2023-10-25T00:08:46.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 -- start start 2023-10-25T00:08:46.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faab80a9450 0x7faab81346d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab80aa330 0x7faab8132cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab813bb00 0x7faab8130b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab8138370 con 0x7faab813bb00 2023-10-25T00:08:46.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab8132810 con 0x7faab80a9450 2023-10-25T00:08:46.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faabd1e1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab8137db0 con 0x7faab80aa330 2023-10-25T00:08:46.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab813bb00 0x7faab8130b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab759e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab813bb00 0x7faab8130b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36018/0 (socket says 172.21.15.107:36018) 2023-10-25T00:08:46.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab759e640 1 -- 172.21.15.107:0/3886558603 learned_addr learned my addr 172.21.15.107:0/3886558603 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:46.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab659c640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab80aa330 0x7faab8132cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab6d9d640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faab80a9450 0x7faab81346d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab759e640 1 -- 172.21.15.107:0/3886558603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab80aa330 msgr2=0x7faab8132cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab759e640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab80aa330 0x7faab8132cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab759e640 1 -- 172.21.15.107:0/3886558603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faab80a9450 msgr2=0x7faab81346d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab759e640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faab80a9450 0x7faab81346d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab759e640 1 -- 172.21.15.107:0/3886558603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa80094d0 con 0x7faab813bb00 2023-10-25T00:08:46.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab659c640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab80aa330 0x7faab8132cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:08:46.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.286+0000 7faab759e640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab813bb00 0x7faab8130b70 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7faab000c700 tx=0x7faab000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:46.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.290+0000 7faa9f7fe640 1 -- 172.21.15.107:0/3886558603 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab000ce10 con 0x7faab813bb00 2023-10-25T00:08:46.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.290+0000 7faabd1e1640 1 -- 172.21.15.107:0/3886558603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab8138090 con 0x7faab813bb00 2023-10-25T00:08:46.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.290+0000 7faa9f7fe640 1 -- 172.21.15.107:0/3886558603 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab000a640 con 0x7faab813bb00 2023-10-25T00:08:46.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.290+0000 7faa9f7fe640 1 -- 172.21.15.107:0/3886558603 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab000aa70 con 0x7faab813bb00 2023-10-25T00:08:46.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.290+0000 7faabd1e1640 1 -- 172.21.15.107:0/3886558603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab8139070 con 0x7faab813bb00 2023-10-25T00:08:46.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.290+0000 7faabd1e1640 1 -- 172.21.15.107:0/3886558603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab80746e0 con 0x7faab813bb00 2023-10-25T00:08:46.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.290+0000 7faa9f7fe640 1 -- 172.21.15.107:0/3886558603 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7faab000abd0 con 0x7faab813bb00 2023-10-25T00:08:46.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.290+0000 7faa9f7fe640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faaa4065e90 0x7faaa4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.290+0000 7faa9f7fe640 1 -- 172.21.15.107:0/3886558603 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7faab0083860 con 0x7faab813bb00 2023-10-25T00:08:46.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.290+0000 7faab6d9d640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faaa4065e90 0x7faaa4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.290+0000 7faab6d9d640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faaa4065e90 0x7faaa4068350 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7faaa80ae000 tx=0x7faaa80a6070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:46.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.298+0000 7faa9f7fe640 1 -- 172.21.15.107:0/3886558603 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7faab0050220 con 0x7faab813bb00 2023-10-25T00:08:46.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 -- 172.21.15.107:0/3886558603 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faab8139260 con 0x7faab813bb00 2023-10-25T00:08:46.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faa9f7fe640 1 -- 172.21.15.107:0/3886558603 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5779 (secure 0 0 0) 0x7faab0018070 con 0x7faab813bb00 2023-10-25T00:08:46.567 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:46.567 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:46.567 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 140 2023-10-25T00:08:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 -- 172.21.15.107:0/3886558603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faaa4065e90 msgr2=0x7faaa4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faaa4065e90 0x7faaa4068350 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7faaa80ae000 tx=0x7faaa80a6070 comp rx=0 tx=0).stop 2023-10-25T00:08:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 -- 172.21.15.107:0/3886558603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab813bb00 msgr2=0x7faab8130b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab813bb00 0x7faab8130b70 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7faab000c700 tx=0x7faab000cbc0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 -- 172.21.15.107:0/3886558603 shutdown_connections 2023-10-25T00:08:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab813bb00 0x7faab8130b70 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faaa4065e90 0x7faaa4068350 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab80aa330 0x7faab8132cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 --2- 172.21.15.107:0/3886558603 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faab80a9450 0x7faab81346d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 -- 172.21.15.107:0/3886558603 >> 172.21.15.107:0/3886558603 conn(0x7faab8070f60 msgr2=0x7faab8073170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 -- 172.21.15.107:0/3886558603 shutdown_connections 2023-10-25T00:08:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.566+0000 7faabd1e1640 1 -- 172.21.15.107:0/3886558603 wait complete. 2023-10-25T00:08:46.586 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2023-10-25T00:08:46.586 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-25T00:08:46.586 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:08:46.605 INFO:tasks.ceph.mds.d:Stopped 2023-10-25T00:08:46.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:46.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.722+0000 7f465817c640 1 Processor -- start 2023-10-25T00:08:46.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.722+0000 7f465817c640 1 -- start start 2023-10-25T00:08:46.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f465817c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465014c470 0x7f465014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f465817c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4650152b30 0x7f4650152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f465817c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4650155cd0 con 0x7f465006f490 2023-10-25T00:08:46.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f465817c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46501526e0 con 0x7f4650152b30 2023-10-25T00:08:46.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f465817c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46500c4380 con 0x7f465014c470 2023-10-25T00:08:46.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4655f18640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465014c470 0x7f465014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4655f18640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465014c470 0x7f465014c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:46168/0 (socket says 172.21.15.107:46168) 2023-10-25T00:08:46.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4655f18640 1 -- 172.21.15.107:0/3167430978 learned_addr learned my addr 172.21.15.107:0/3167430978 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:46.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4656719640 1 --2- 172.21.15.107:0/3167430978 >> v2:172.21.15.110:3300/0 conn(0x7f4650152b30 0x7f4650152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4655f18640 1 -- 172.21.15.107:0/3167430978 >> v2:172.21.15.110:3300/0 conn(0x7f4650152b30 msgr2=0x7f4650152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:46.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4655f18640 1 --2- 172.21.15.107:0/3167430978 >> v2:172.21.15.110:3300/0 conn(0x7f4650152b30 0x7f4650152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4655f18640 1 -- 172.21.15.107:0/3167430978 >> v1:172.21.15.110:6789/0 conn(0x7f465006f490 legacy=0x7f4650071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4655f18640 1 -- 172.21.15.107:0/3167430978 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46500c48b0 con 0x7f465014c470 2023-10-25T00:08:46.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4655f18640 1 --2- 172.21.15.107:0/3167430978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465014c470 0x7f465014c850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f464c009870 tx=0x7f464c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54471fdd41deeb93 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:46.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4654f16640 1 -- 172.21.15.107:0/3167430978 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c0ac640 con 0x7f465014c470 2023-10-25T00:08:46.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4654f16640 1 -- 172.21.15.107:0/3167430978 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c0ac7a0 con 0x7f465014c470 2023-10-25T00:08:46.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.726+0000 7f4654f16640 1 -- 172.21.15.107:0/3167430978 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c0acb30 con 0x7f465014c470 2023-10-25T00:08:46.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 -- 172.21.15.107:0/3167430978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465014c470 msgr2=0x7f465014c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 --2- 172.21.15.107:0/3167430978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465014c470 0x7f465014c850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f464c009870 tx=0x7f464c01c420 comp rx=0 tx=0).stop 2023-10-25T00:08:46.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 -- 172.21.15.107:0/3167430978 shutdown_connections 2023-10-25T00:08:46.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 --2- 172.21.15.107:0/3167430978 >> v2:172.21.15.110:3300/0 conn(0x7f4650152b30 0x7f4650152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 --2- 172.21.15.107:0/3167430978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465014c470 0x7f465014c850 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 -- 172.21.15.107:0/3167430978 >> 172.21.15.107:0/3167430978 conn(0x7f4650070f60 msgr2=0x7f4650071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:46.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 -- 172.21.15.107:0/3167430978 shutdown_connections 2023-10-25T00:08:46.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 -- 172.21.15.107:0/3167430978 wait complete. 2023-10-25T00:08:46.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 Processor -- start 2023-10-25T00:08:46.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 -- start start 2023-10-25T00:08:46.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465006f490 0x7f465014a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f465014c470 0x7f465014aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650152b30 0x7f4650146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46501433c0 con 0x7f4650152b30 2023-10-25T00:08:46.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46500c6ce0 con 0x7f465014c470 2023-10-25T00:08:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f465817c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46500c7860 con 0x7f465006f490 2023-10-25T00:08:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f4656719640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650152b30 0x7f4650146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f4656719640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650152b30 0x7f4650146f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36028/0 (socket says 172.21.15.107:36028) 2023-10-25T00:08:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f4656719640 1 -- 172.21.15.107:0/3394414546 learned_addr learned my addr 172.21.15.107:0/3394414546 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:46.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f4655f18640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465006f490 0x7f465014a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f4655717640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f465014c470 0x7f465014aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f4656719640 1 -- 172.21.15.107:0/3394414546 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465006f490 msgr2=0x7f465014a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f4656719640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465006f490 0x7f465014a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f4656719640 1 -- 172.21.15.107:0/3394414546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f465014c470 msgr2=0x7f465014aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f4656719640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f465014c470 0x7f465014aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f4656719640 1 -- 172.21.15.107:0/3394414546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c0094d0 con 0x7f4650152b30 2023-10-25T00:08:46.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f4656719640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650152b30 0x7f4650146f70 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f464800efc0 tx=0x7f464800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:46.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f46467fc640 1 -- 172.21.15.107:0/3394414546 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4648009ba0 con 0x7f4650152b30 2023-10-25T00:08:46.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f46467fc640 1 -- 172.21.15.107:0/3394414546 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464800f040 con 0x7f4650152b30 2023-10-25T00:08:46.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.730+0000 7f46467fc640 1 -- 172.21.15.107:0/3394414546 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46480146c0 con 0x7f4650152b30 2023-10-25T00:08:46.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.734+0000 7f465817c640 1 -- 172.21.15.107:0/3394414546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f465014a7a0 con 0x7f4650152b30 2023-10-25T00:08:46.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.734+0000 7f465817c640 1 -- 172.21.15.107:0/3394414546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46500c7210 con 0x7f4650152b30 2023-10-25T00:08:46.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.734+0000 7f46467fc640 1 -- 172.21.15.107:0/3394414546 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4648014820 con 0x7f4650152b30 2023-10-25T00:08:46.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.734+0000 7f465817c640 1 -- 172.21.15.107:0/3394414546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46500746e0 con 0x7f4650152b30 2023-10-25T00:08:46.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.734+0000 7f46467fc640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4634065e90 0x7f4634068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.734+0000 7f46467fc640 1 -- 172.21.15.107:0/3394414546 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7f4648084e40 con 0x7f4650152b30 2023-10-25T00:08:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.734+0000 7f4655f18640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4634065e90 0x7f4634068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.734+0000 7f4655f18640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4634065e90 0x7f4634068350 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f464c0ae000 tx=0x7f464c0a6070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.738+0000 7f46467fc640 1 -- 172.21.15.107:0/3394414546 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4648051940 con 0x7f4650152b30 2023-10-25T00:08:46.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.966+0000 7f465817c640 1 -- 172.21.15.107:0/3394414546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f465014fd70 con 0x7f4650152b30 2023-10-25T00:08:46.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.966+0000 7f46467fc640 1 -- 172.21.15.107:0/3394414546 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5779 (secure 0 0 0) 0x7f4648018070 con 0x7f4650152b30 2023-10-25T00:08:46.968 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:46.968 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:46.969 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 140 2023-10-25T00:08:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 -- 172.21.15.107:0/3394414546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4634065e90 msgr2=0x7f4634068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4634065e90 0x7f4634068350 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f464c0ae000 tx=0x7f464c0a6070 comp rx=0 tx=0).stop 2023-10-25T00:08:46.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 -- 172.21.15.107:0/3394414546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650152b30 msgr2=0x7f4650146f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:46.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650152b30 0x7f4650146f70 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f464800efc0 tx=0x7f464800c490 comp rx=0 tx=0).stop 2023-10-25T00:08:46.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 -- 172.21.15.107:0/3394414546 shutdown_connections 2023-10-25T00:08:46.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4650152b30 0x7f4650146f70 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f465014c470 0x7f465014aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4634065e90 0x7f4634068350 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 --2- 172.21.15.107:0/3394414546 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f465006f490 0x7f465014a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:46.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 -- 172.21.15.107:0/3394414546 >> 172.21.15.107:0/3394414546 conn(0x7f4650070f60 msgr2=0x7f46501510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:46.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 -- 172.21.15.107:0/3394414546 shutdown_connections 2023-10-25T00:08:46.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:46.970+0000 7f465817c640 1 -- 172.21.15.107:0/3394414546 wait complete. 2023-10-25T00:08:46.990 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:08:48.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:49.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.122+0000 7f69bdc3a640 1 Processor -- start 2023-10-25T00:08:49.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.122+0000 7f69bdc3a640 1 -- start start 2023-10-25T00:08:49.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69b80a9820 0x7f69b80ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:49.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f69b806ee40 0x7f69b80aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b806f730 con 0x7f69b814fd40 2023-10-25T00:08:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b80aa750 con 0x7f69b806ee40 2023-10-25T00:08:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8156a40 con 0x7f69b80a9820 2023-10-25T00:08:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f69b806ee40 0x7f69b80aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f69b814fd40 0x7f69b80c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45290/0 (socket says 172.21.15.107:45290) 2023-10-25T00:08:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b77fe640 1 -- 172.21.15.107:0/1445598875 learned_addr learned my addr 172.21.15.107:0/1445598875 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b6ffd640 1 --2- 172.21.15.107:0/1445598875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69b80a9820 0x7f69b80ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b7fff640 1 -- 172.21.15.107:0/1445598875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69b80a9820 msgr2=0x7f69b80ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b7fff640 1 --2- 172.21.15.107:0/1445598875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69b80a9820 0x7f69b80ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b7fff640 1 -- 172.21.15.107:0/1445598875 >> v1:172.21.15.107:6789/0 conn(0x7f69b814fd40 legacy=0x7f69b80c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:49.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b7fff640 1 -- 172.21.15.107:0/1445598875 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b8156710 con 0x7f69b806ee40 2023-10-25T00:08:49.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b67fc640 1 -- 172.21.15.107:0/1445598875 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492249945 0 0) 0x7f69b806f730 con 0x7f69b814fd40 2023-10-25T00:08:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b7fff640 1 --2- 172.21.15.107:0/1445598875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69b806ee40 0x7f69b80aa210 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f69a80097c0 tx=0x7f69a801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f25fda1d1e9b08 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b67fc640 1 -- 172.21.15.107:0/1445598875 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a80ab600 con 0x7f69b806ee40 2023-10-25T00:08:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b67fc640 1 -- 172.21.15.107:0/1445598875 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a80a7050 con 0x7f69b806ee40 2023-10-25T00:08:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69b67fc640 1 -- 172.21.15.107:0/1445598875 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a80ab980 con 0x7f69b806ee40 2023-10-25T00:08:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 -- 172.21.15.107:0/1445598875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69b806ee40 msgr2=0x7f69b80aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 --2- 172.21.15.107:0/1445598875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69b806ee40 0x7f69b80aa210 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f69a80097c0 tx=0x7f69a801c330 comp rx=0 tx=0).stop 2023-10-25T00:08:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 -- 172.21.15.107:0/1445598875 shutdown_connections 2023-10-25T00:08:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 --2- 172.21.15.107:0/1445598875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69b806ee40 0x7f69b80aa210 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 --2- 172.21.15.107:0/1445598875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69b80a9820 0x7f69b80ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 -- 172.21.15.107:0/1445598875 >> 172.21.15.107:0/1445598875 conn(0x7f69b80c2950 msgr2=0x7f69b80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 -- 172.21.15.107:0/1445598875 shutdown_connections 2023-10-25T00:08:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.126+0000 7f69bdc3a640 1 -- 172.21.15.107:0/1445598875 wait complete. 2023-10-25T00:08:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69bdc3a640 1 Processor -- start 2023-10-25T00:08:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69bdc3a640 1 -- start start 2023-10-25T00:08:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69bdc3a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69b806ee40 0x7f69b815b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69bdc3a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69b814fd40 0x7f69b8153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:49.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69bdc3a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69b8153ea0 0x7f69b8143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:49.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69bdc3a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b814d530 con 0x7f69b806ee40 2023-10-25T00:08:49.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69bdc3a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8144850 con 0x7f69b814fd40 2023-10-25T00:08:49.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69bdc3a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8155ac0 con 0x7f69b8153ea0 2023-10-25T00:08:49.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69b806ee40 0x7f69b815b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:49.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69b806ee40 0x7f69b815b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36044/0 (socket says 172.21.15.107:36044) 2023-10-25T00:08:49.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b77fe640 1 -- 172.21.15.107:0/425077036 learned_addr learned my addr 172.21.15.107:0/425077036 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:49.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b6ffd640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69b814fd40 0x7f69b8153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:49.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b7fff640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69b8153ea0 0x7f69b8143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:49.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b77fe640 1 -- 172.21.15.107:0/425077036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69b8153ea0 msgr2=0x7f69b8143970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b77fe640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69b8153ea0 0x7f69b8143970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b77fe640 1 -- 172.21.15.107:0/425077036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69b814fd40 msgr2=0x7f69b8153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b77fe640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69b814fd40 0x7f69b8153960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b77fe640 1 -- 172.21.15.107:0/425077036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8009420 con 0x7f69b806ee40 2023-10-25T00:08:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b7fff640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69b8153ea0 0x7f69b8143970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b6ffd640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69b814fd40 0x7f69b8153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69b77fe640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69b806ee40 0x7f69b815b5a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f69b000c7f0 tx=0x7f69b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69affff640 1 -- 172.21.15.107:0/425077036 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b000a640 con 0x7f69b806ee40 2023-10-25T00:08:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69affff640 1 -- 172.21.15.107:0/425077036 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b000a7c0 con 0x7f69b806ee40 2023-10-25T00:08:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69bdc3a640 1 -- 172.21.15.107:0/425077036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b8144570 con 0x7f69b806ee40 2023-10-25T00:08:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69affff640 1 -- 172.21.15.107:0/425077036 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b000ab70 con 0x7f69b806ee40 2023-10-25T00:08:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69bdc3a640 1 -- 172.21.15.107:0/425077036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b8154fb0 con 0x7f69b806ee40 2023-10-25T00:08:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.130+0000 7f69bdc3a640 1 -- 172.21.15.107:0/425077036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b80c24a0 con 0x7f69b806ee40 2023-10-25T00:08:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.134+0000 7f69affff640 1 -- 172.21.15.107:0/425077036 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f69b000acd0 con 0x7f69b806ee40 2023-10-25T00:08:49.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.134+0000 7f69affff640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6998065e90 0x7f6998068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:49.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.134+0000 7f69b6ffd640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6998065e90 0x7f6998068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.134+0000 7f69affff640 1 -- 172.21.15.107:0/425077036 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7f69b0083c50 con 0x7f69b806ee40 2023-10-25T00:08:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.134+0000 7f69b6ffd640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6998065e90 0x7f6998068350 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f69b815be10 tx=0x7f69a0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.138+0000 7f69affff640 1 -- 172.21.15.107:0/425077036 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f69b0089020 con 0x7f69b806ee40 2023-10-25T00:08:49.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.326+0000 7f69bdc3a640 1 -- 172.21.15.107:0/425077036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69b8070840 con 0x7f69b806ee40 2023-10-25T00:08:49.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.326+0000 7f69affff640 1 -- 172.21.15.107:0/425077036 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5779 (secure 0 0 0) 0x7f69b0018020 con 0x7f69b806ee40 2023-10-25T00:08:49.328 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:49.328 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:49.329 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 140 2023-10-25T00:08:49.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 -- 172.21.15.107:0/425077036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6998065e90 msgr2=0x7f6998068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:49.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6998065e90 0x7f6998068350 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f69b815be10 tx=0x7f69a0008040 comp rx=0 tx=0).stop 2023-10-25T00:08:49.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 -- 172.21.15.107:0/425077036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69b806ee40 msgr2=0x7f69b815b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:49.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69b806ee40 0x7f69b815b5a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f69b000c7f0 tx=0x7f69b000ccb0 comp rx=0 tx=0).stop 2023-10-25T00:08:49.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 -- 172.21.15.107:0/425077036 shutdown_connections 2023-10-25T00:08:49.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6998065e90 0x7f6998068350 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:49.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69b8153ea0 0x7f69b8143970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:49.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69b814fd40 0x7f69b8153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:49.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 --2- 172.21.15.107:0/425077036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69b806ee40 0x7f69b815b5a0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:49.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 -- 172.21.15.107:0/425077036 >> 172.21.15.107:0/425077036 conn(0x7f69b80c2950 msgr2=0x7f69b8071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:49.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 -- 172.21.15.107:0/425077036 shutdown_connections 2023-10-25T00:08:49.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:49.330+0000 7f69bdc3a640 1 -- 172.21.15.107:0/425077036 wait complete. 2023-10-25T00:08:49.345 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:08:51.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:51.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f95517fb640 1 Processor -- start 2023-10-25T00:08:51.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f95517fb640 1 -- start start 2023-10-25T00:08:51.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f95517fb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f954c14fd40 0x7f954c0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:51.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f95517fb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0a9820 0x7f954c0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:51.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f95517fb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954c06f730 con 0x7f954c06ee40 2023-10-25T00:08:51.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f95517fb640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954c0aa750 con 0x7f954c14fd40 2023-10-25T00:08:51.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f95517fb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954c156a40 con 0x7f954c0a9820 2023-10-25T00:08:51.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f954a7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0a9820 0x7f954c0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:51.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f954affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f954c14fd40 0x7f954c0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:51.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f954a7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0a9820 0x7f954c0aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:46212/0 (socket says 172.21.15.107:46212) 2023-10-25T00:08:51.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f954a7fc640 1 -- 172.21.15.107:0/2328109148 learned_addr learned my addr 172.21.15.107:0/2328109148 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:51.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f954affd640 1 -- 172.21.15.107:0/2328109148 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0a9820 msgr2=0x7f954c0aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:51.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f954affd640 1 --2- 172.21.15.107:0/2328109148 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0a9820 0x7f954c0aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:51.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f954affd640 1 -- 172.21.15.107:0/2328109148 >> v1:172.21.15.110:6789/0 conn(0x7f954c06ee40 legacy=0x7f954c0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:51.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f954affd640 1 -- 172.21.15.107:0/2328109148 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c156710 con 0x7f954c14fd40 2023-10-25T00:08:51.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.478+0000 7f954a7fc640 1 --2- 172.21.15.107:0/2328109148 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0a9820 0x7f954c0aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:08:51.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f954affd640 1 --2- 172.21.15.107:0/2328109148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c14fd40 0x7f954c0ab090 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f95400097c0 tx=0x7f954001c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23231ba41f4b90bf server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:51.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f9549ffb640 1 -- 172.21.15.107:0/2328109148 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95400ab600 con 0x7f954c14fd40 2023-10-25T00:08:51.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f9549ffb640 1 -- 172.21.15.107:0/2328109148 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95400a7050 con 0x7f954c14fd40 2023-10-25T00:08:51.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f9549ffb640 1 -- 172.21.15.107:0/2328109148 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95400ab980 con 0x7f954c14fd40 2023-10-25T00:08:51.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 -- 172.21.15.107:0/2328109148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c14fd40 msgr2=0x7f954c0ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:51.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 --2- 172.21.15.107:0/2328109148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c14fd40 0x7f954c0ab090 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f95400097c0 tx=0x7f954001c330 comp rx=0 tx=0).stop 2023-10-25T00:08:51.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 -- 172.21.15.107:0/2328109148 shutdown_connections 2023-10-25T00:08:51.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 --2- 172.21.15.107:0/2328109148 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0a9820 0x7f954c0aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:51.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 --2- 172.21.15.107:0/2328109148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c14fd40 0x7f954c0ab090 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:51.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 -- 172.21.15.107:0/2328109148 >> 172.21.15.107:0/2328109148 conn(0x7f954c0c2950 msgr2=0x7f954c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:51.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 -- 172.21.15.107:0/2328109148 shutdown_connections 2023-10-25T00:08:51.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 -- 172.21.15.107:0/2328109148 wait complete. 2023-10-25T00:08:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 Processor -- start 2023-10-25T00:08:51.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 -- start start 2023-10-25T00:08:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c06ee40 0x7f954c0bcea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.482+0000 7f95517fb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954c14fd40 0x7f954c15b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95517fb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0bbec0 0x7f954c1f0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95517fb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0bca10 con 0x7f954c14fd40 2023-10-25T00:08:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95517fb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0c5100 con 0x7f954c06ee40 2023-10-25T00:08:51.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95517fb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0c53e0 con 0x7f954c0bbec0 2023-10-25T00:08:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954c14fd40 0x7f954c15b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954c14fd40 0x7f954c15b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36060/0 (socket says 172.21.15.107:36060) 2023-10-25T00:08:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954a7fc640 1 -- 172.21.15.107:0/3463737219 learned_addr learned my addr 172.21.15.107:0/3463737219 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954affd640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c06ee40 0x7f954c0bcea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:51.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954b7fe640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0bbec0 0x7f954c1f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954a7fc640 1 -- 172.21.15.107:0/3463737219 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0bbec0 msgr2=0x7f954c1f0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954a7fc640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0bbec0 0x7f954c1f0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954a7fc640 1 -- 172.21.15.107:0/3463737219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c06ee40 msgr2=0x7f954c0bcea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954a7fc640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c06ee40 0x7f954c0bcea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:51.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954a7fc640 1 -- 172.21.15.107:0/3463737219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953c0094d0 con 0x7f954c14fd40 2023-10-25T00:08:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954affd640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c06ee40 0x7f954c0bcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:08:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954a7fc640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954c14fd40 0x7f954c15b5a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f953c00e9d0 tx=0x7f953c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95337fe640 1 -- 172.21.15.107:0/3463737219 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c00ca60 con 0x7f954c14fd40 2023-10-25T00:08:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95337fe640 1 -- 172.21.15.107:0/3463737219 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953c00cbc0 con 0x7f954c14fd40 2023-10-25T00:08:51.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95337fe640 1 -- 172.21.15.107:0/3463737219 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c013470 con 0x7f954c14fd40 2023-10-25T00:08:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95517fb640 1 -- 172.21.15.107:0/3463737219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9540009420 con 0x7f954c14fd40 2023-10-25T00:08:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95517fb640 1 -- 172.21.15.107:0/3463737219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954c0c59a0 con 0x7f954c14fd40 2023-10-25T00:08:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95517fb640 1 -- 172.21.15.107:0/3463737219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954c0c24a0 con 0x7f954c14fd40 2023-10-25T00:08:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95337fe640 1 -- 172.21.15.107:0/3463737219 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f953c0135d0 con 0x7f954c14fd40 2023-10-25T00:08:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95337fe640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9538065e60 0x7f9538068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f95337fe640 1 -- 172.21.15.107:0/3463737219 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7f953c084740 con 0x7f954c14fd40 2023-10-25T00:08:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954affd640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9538065e60 0x7f9538068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.486+0000 7f954affd640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9538065e60 0x7f9538068320 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f95400ad000 tx=0x7f9540008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:51.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.490+0000 7f95337fe640 1 -- 172.21.15.107:0/3463737219 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f953c051190 con 0x7f954c14fd40 2023-10-25T00:08:51.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.674+0000 7f95517fb640 1 -- 172.21.15.107:0/3463737219 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f954c070840 con 0x7f954c14fd40 2023-10-25T00:08:51.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95337fe640 1 -- 172.21.15.107:0/3463737219 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5779 (secure 0 0 0) 0x7f953c017090 con 0x7f954c14fd40 2023-10-25T00:08:51.677 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:51.678 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:51.678 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 140 2023-10-25T00:08:51.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 -- 172.21.15.107:0/3463737219 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9538065e60 msgr2=0x7f9538068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:51.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9538065e60 0x7f9538068320 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f95400ad000 tx=0x7f9540008040 comp rx=0 tx=0).stop 2023-10-25T00:08:51.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 -- 172.21.15.107:0/3463737219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954c14fd40 msgr2=0x7f954c15b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:51.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954c14fd40 0x7f954c15b5a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f953c00e9d0 tx=0x7f953c00ee90 comp rx=0 tx=0).stop 2023-10-25T00:08:51.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 -- 172.21.15.107:0/3463737219 shutdown_connections 2023-10-25T00:08:51.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9538065e60 0x7f9538068320 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:51.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f954c0bbec0 0x7f954c1f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:51.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954c14fd40 0x7f954c15b5a0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:51.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 --2- 172.21.15.107:0/3463737219 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f954c06ee40 0x7f954c0bcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:51.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 -- 172.21.15.107:0/3463737219 >> 172.21.15.107:0/3463737219 conn(0x7f954c0c2950 msgr2=0x7f954c151810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:51.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 -- 172.21.15.107:0/3463737219 shutdown_connections 2023-10-25T00:08:51.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:51.678+0000 7f95517fb640 1 -- 172.21.15.107:0/3463737219 wait complete. 2023-10-25T00:08:51.694 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:08:53.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:53.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.822+0000 7f1fcfed2640 1 Processor -- start 2023-10-25T00:08:53.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.822+0000 7f1fcfed2640 1 -- start start 2023-10-25T00:08:53.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fcfed2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc80a9820 0x7f1fc80ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:53.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fcfed2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fc806ee40 0x7f1fc80aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:53.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fcfed2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc806f730 con 0x7f1fc814fd40 2023-10-25T00:08:53.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fcfed2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc80aa750 con 0x7f1fc806ee40 2023-10-25T00:08:53.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fcfed2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc8156a40 con 0x7f1fc80a9820 2023-10-25T00:08:53.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fce46f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fc806ee40 0x7f1fc80aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:53.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fce46f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fc806ee40 0x7f1fc80aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47110/0 (socket says 172.21.15.107:47110) 2023-10-25T00:08:53.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fce46f640 1 -- 172.21.15.107:0/1829701910 learned_addr learned my addr 172.21.15.107:0/1829701910 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:53.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fcd46d640 1 --2- 172.21.15.107:0/1829701910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc80a9820 0x7f1fc80ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:53.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fce46f640 1 -- 172.21.15.107:0/1829701910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc80a9820 msgr2=0x7f1fc80ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:53.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fce46f640 1 --2- 172.21.15.107:0/1829701910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc80a9820 0x7f1fc80ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:53.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fce46f640 1 -- 172.21.15.107:0/1829701910 >> v1:172.21.15.107:6789/0 conn(0x7f1fc814fd40 legacy=0x7f1fc80c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:53.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fccc6c640 1 -- 172.21.15.107:0/1829701910 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824490406 0 0) 0x7f1fc806f730 con 0x7f1fc814fd40 2023-10-25T00:08:53.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fce46f640 1 -- 172.21.15.107:0/1829701910 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc8156710 con 0x7f1fc806ee40 2023-10-25T00:08:53.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fce46f640 1 --2- 172.21.15.107:0/1829701910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc806ee40 0x7f1fc80aa210 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0009e50 tx=0x7f1fc001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a20d39a89c56979f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:53.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fccc6c640 1 -- 172.21.15.107:0/1829701910 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00a5720 con 0x7f1fc806ee40 2023-10-25T00:08:53.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fccc6c640 1 -- 172.21.15.107:0/1829701910 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc00a5880 con 0x7f1fc806ee40 2023-10-25T00:08:53.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fccc6c640 1 -- 172.21.15.107:0/1829701910 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00a5c10 con 0x7f1fc806ee40 2023-10-25T00:08:53.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fcfed2640 1 -- 172.21.15.107:0/1829701910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc806ee40 msgr2=0x7f1fc80aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:53.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.826+0000 7f1fcfed2640 1 --2- 172.21.15.107:0/1829701910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc806ee40 0x7f1fc80aa210 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0009e50 tx=0x7f1fc001c8c0 comp rx=0 tx=0).stop 2023-10-25T00:08:53.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 -- 172.21.15.107:0/1829701910 shutdown_connections 2023-10-25T00:08:53.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 --2- 172.21.15.107:0/1829701910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc806ee40 0x7f1fc80aa210 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:53.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 --2- 172.21.15.107:0/1829701910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc80a9820 0x7f1fc80ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:53.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 -- 172.21.15.107:0/1829701910 >> 172.21.15.107:0/1829701910 conn(0x7f1fc80c2950 msgr2=0x7f1fc80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:53.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 -- 172.21.15.107:0/1829701910 shutdown_connections 2023-10-25T00:08:53.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 -- 172.21.15.107:0/1829701910 wait complete. 2023-10-25T00:08:53.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 Processor -- start 2023-10-25T00:08:53.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 -- start start 2023-10-25T00:08:53.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc806ee40 0x7f1fc815b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:53.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc814fd40 0x7f1fc8147bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc81eb2f0 0x7f1fc81541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc80c54f0 con 0x7f1fc81eb2f0 2023-10-25T00:08:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc8148100 con 0x7f1fc806ee40 2023-10-25T00:08:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc80c4f50 con 0x7f1fc814fd40 2023-10-25T00:08:53.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fce46f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc81eb2f0 0x7f1fc81541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:53.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fce46f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc81eb2f0 0x7f1fc81541c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47114/0 (socket says 172.21.15.107:47114) 2023-10-25T00:08:53.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fce46f640 1 -- 172.21.15.107:0/3538931036 learned_addr learned my addr 172.21.15.107:0/3538931036 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:53.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcd46d640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc814fd40 0x7f1fc8147bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:53.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcdc6e640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc806ee40 0x7f1fc815b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:53.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fce46f640 1 -- 172.21.15.107:0/3538931036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc814fd40 msgr2=0x7f1fc8147bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:53.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fce46f640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc814fd40 0x7f1fc8147bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:53.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fce46f640 1 -- 172.21.15.107:0/3538931036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc806ee40 msgr2=0x7f1fc815b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:53.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fce46f640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc806ee40 0x7f1fc815b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:53.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fce46f640 1 -- 172.21.15.107:0/3538931036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc0009b30 con 0x7f1fc81eb2f0 2023-10-25T00:08:53.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcdc6e640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc806ee40 0x7f1fc815b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:53.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fce46f640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc81eb2f0 0x7f1fc81541c0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00af040 tx=0x7f1fc0019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:53.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fbe7fc640 1 -- 172.21.15.107:0/3538931036 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00b8670 con 0x7f1fc81eb2f0 2023-10-25T00:08:53.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fbe7fc640 1 -- 172.21.15.107:0/3538931036 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc00b87d0 con 0x7f1fc81eb2f0 2023-10-25T00:08:53.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 -- 172.21.15.107:0/3538931036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc80c5210 con 0x7f1fc81eb2f0 2023-10-25T00:08:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fbe7fc640 1 -- 172.21.15.107:0/3538931036 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00b8b60 con 0x7f1fc81eb2f0 2023-10-25T00:08:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 -- 172.21.15.107:0/3538931036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc814d670 con 0x7f1fc81eb2f0 2023-10-25T00:08:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.830+0000 7f1fcfed2640 1 -- 172.21.15.107:0/3538931036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc80c24a0 con 0x7f1fc81eb2f0 2023-10-25T00:08:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.834+0000 7f1fbe7fc640 1 -- 172.21.15.107:0/3538931036 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1fc00b8cc0 con 0x7f1fc81eb2f0 2023-10-25T00:08:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.834+0000 7f1fbe7fc640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 0x7f1fac068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:53.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.834+0000 7f1fbe7fc640 1 -- 172.21.15.107:0/3538931036 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7f1fc011fa00 con 0x7f1fc81eb2f0 2023-10-25T00:08:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.834+0000 7f1fcdc6e640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 0x7f1fac068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.834+0000 7f1fcdc6e640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 0x7f1fac068320 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8000f70 tx=0x7f1fb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:53.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:53.838+0000 7f1fbe7fc640 1 -- 172.21.15.107:0/3538931036 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1fc00ec500 con 0x7f1fc81eb2f0 2023-10-25T00:08:54.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.026+0000 7f1fcfed2640 1 -- 172.21.15.107:0/3538931036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fc8070840 con 0x7f1fc81eb2f0 2023-10-25T00:08:54.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.026+0000 7f1fbe7fc640 1 -- 172.21.15.107:0/3538931036 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5779 (secure 0 0 0) 0x7f1fc00efb20 con 0x7f1fc81eb2f0 2023-10-25T00:08:54.029 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:54.029 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:54.029 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 140 2023-10-25T00:08:54.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 -- 172.21.15.107:0/3538931036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 msgr2=0x7f1fac068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:54.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 0x7f1fac068320 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8000f70 tx=0x7f1fb8009040 comp rx=0 tx=0).stop 2023-10-25T00:08:54.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 -- 172.21.15.107:0/3538931036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc81eb2f0 msgr2=0x7f1fc81541c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:54.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc81eb2f0 0x7f1fc81541c0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00af040 tx=0x7f1fc0019660 comp rx=0 tx=0).stop 2023-10-25T00:08:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 -- 172.21.15.107:0/3538931036 shutdown_connections 2023-10-25T00:08:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fc81eb2f0 0x7f1fc81541c0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fac065e60 0x7f1fac068320 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fc814fd40 0x7f1fc8147bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 --2- 172.21.15.107:0/3538931036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fc806ee40 0x7f1fc815b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:54.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 -- 172.21.15.107:0/3538931036 >> 172.21.15.107:0/3538931036 conn(0x7f1fc80c2950 msgr2=0x7f1fc80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:54.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 -- 172.21.15.107:0/3538931036 shutdown_connections 2023-10-25T00:08:54.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:54.030+0000 7f1fcfed2640 1 -- 172.21.15.107:0/3538931036 wait complete. 2023-10-25T00:08:54.047 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:08:56.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:56.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.178+0000 7fa7cbb12640 1 Processor -- start 2023-10-25T00:08:56.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.178+0000 7fa7cbb12640 1 -- start start 2023-10-25T00:08:56.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7cbb12640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa7c414b600 0x7fa7c4151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:56.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7cbb12640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7c406a230 0x7fa7c41551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:56.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7cbb12640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c40c1860 con 0x7fa7c406af70 2023-10-25T00:08:56.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7cbb12640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7c4154d50 con 0x7fa7c414b600 2023-10-25T00:08:56.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7cbb12640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c41524f0 con 0x7fa7c406a230 2023-10-25T00:08:56.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c90ad640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa7c406af70 0x7fa7c406c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33380/0 (socket says 172.21.15.107:33380) 2023-10-25T00:08:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c90ad640 1 -- 172.21.15.107:0/4244080005 learned_addr learned my addr 172.21.15.107:0/4244080005 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c98ae640 1 --2- 172.21.15.107:0/4244080005 >> v2:172.21.15.110:3300/0 conn(0x7fa7c414b600 0x7fa7c4151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7ca0af640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7c406a230 0x7fa7c41551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764318006 0 0) 0x7fa7c40c1860 con 0x7fa7c406af70 2023-10-25T00:08:56.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b0003530 con 0x7fa7c406af70 2023-10-25T00:08:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4006240841 0 0) 0x7fa7b0003530 con 0x7fa7c406af70 2023-10-25T00:08:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7c406a230 msgr2=0x7fa7c41551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7c406a230 0x7fa7c41551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 >> v2:172.21.15.110:3300/0 conn(0x7fa7c414b600 msgr2=0x7fa7c4151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:56.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 --2- 172.21.15.107:0/4244080005 >> v2:172.21.15.110:3300/0 conn(0x7fa7c414b600 0x7fa7c4151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c414b9e0 con 0x7fa7c406af70 2023-10-25T00:08:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fa7b4001680 con 0x7fa7c406af70 2023-10-25T00:08:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 >> v1:172.21.15.107:6789/0 conn(0x7fa7c406af70 legacy=0x7fa7c406c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7b0003980 0x7fa7b0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b00047d0 0x7fa7b0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:56.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7c406a230 0x7fa7b0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c4154d50 con 0x7fa7b00047d0 2023-10-25T00:08:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c41524f0 con 0x7fa7b0003980 2023-10-25T00:08:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b000b990 con 0x7fa7c406a230 2023-10-25T00:08:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c90ad640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b00047d0 0x7fa7b0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:56.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c98ae640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7b0003980 0x7fa7b0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7ca0af640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7c406a230 0x7fa7b0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c90ad640 1 -- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7c406a230 msgr2=0x7fa7b0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c90ad640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7c406a230 0x7fa7b0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c90ad640 1 -- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7b0003980 msgr2=0x7fa7b0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:56.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c90ad640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7b0003980 0x7fa7b0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c90ad640 1 -- 172.21.15.107:0/4244080005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b0003530 con 0x7fa7b00047d0 2023-10-25T00:08:56.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c98ae640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7b0003980 0x7fa7b0003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:56.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7ca0af640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7c406a230 0x7fa7b0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:56.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.182+0000 7fa7c90ad640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b00047d0 0x7fa7b0004c50 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400c1c0 tx=0x7fa7b401ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4400cb0213b00 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:56.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b40a7a00 con 0x7fa7b00047d0 2023-10-25T00:08:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b40a7b80 con 0x7fa7b00047d0 2023-10-25T00:08:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7c88ac640 1 -- 172.21.15.107:0/4244080005 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b40058e0 con 0x7fa7b00047d0 2023-10-25T00:08:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 -- 172.21.15.107:0/4244080005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b00047d0 msgr2=0x7fa7b0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 --2- 172.21.15.107:0/4244080005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7b00047d0 0x7fa7b0004c50 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400c1c0 tx=0x7fa7b401ede0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ca0af640 1 -- 172.21.15.107:0/4244080005 reap_dead start 2023-10-25T00:08:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 -- 172.21.15.107:0/4244080005 shutdown_connections 2023-10-25T00:08:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 -- 172.21.15.107:0/4244080005 >> 172.21.15.107:0/4244080005 conn(0x7fa7c40c2950 msgr2=0x7fa7c40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 -- 172.21.15.107:0/4244080005 shutdown_connections 2023-10-25T00:08:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 -- 172.21.15.107:0/4244080005 wait complete. 2023-10-25T00:08:56.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 Processor -- start 2023-10-25T00:08:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 -- start start 2023-10-25T00:08:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7b00047d0 0x7fa7c40c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7c414be20 0x7fa7c41528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7c414eb60 0x7fa7c40aa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:56.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c4144890 con 0x7fa7c414eb60 2023-10-25T00:08:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c40c4d00 con 0x7fa7c414be20 2023-10-25T00:08:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c4144dc0 con 0x7fa7b00047d0 2023-10-25T00:08:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ca0af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7c414eb60 0x7fa7c40aa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:56.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ca0af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7c414eb60 0x7fa7c40aa4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47136/0 (socket says 172.21.15.107:47136) 2023-10-25T00:08:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ca0af640 1 -- 172.21.15.107:0/3386800635 learned_addr learned my addr 172.21.15.107:0/3386800635 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7c98ae640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7b00047d0 0x7fa7c40c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7c90ad640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7c414be20 0x7fa7c41528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ca0af640 1 -- 172.21.15.107:0/3386800635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7b00047d0 msgr2=0x7fa7c40c47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:56.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ca0af640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7b00047d0 0x7fa7c40c47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ca0af640 1 -- 172.21.15.107:0/3386800635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7c414be20 msgr2=0x7fa7c41528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:56.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ca0af640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7c414be20 0x7fa7c41528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ca0af640 1 -- 172.21.15.107:0/3386800635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c4144ae0 con 0x7fa7c414eb60 2023-10-25T00:08:56.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ca0af640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7c414eb60 0x7fa7c40aa4a0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fa7c406ebb0 tx=0x7fa7bc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:56.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ba7fc640 1 -- 172.21.15.107:0/3386800635 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc017400 con 0x7fa7c414eb60 2023-10-25T00:08:56.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ba7fc640 1 -- 172.21.15.107:0/3386800635 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7bc011040 con 0x7fa7c414eb60 2023-10-25T00:08:56.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 -- 172.21.15.107:0/3386800635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c40c4fc0 con 0x7fa7c414eb60 2023-10-25T00:08:56.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7ba7fc640 1 -- 172.21.15.107:0/3386800635 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc017780 con 0x7fa7c414eb60 2023-10-25T00:08:56.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.186+0000 7fa7cbb12640 1 -- 172.21.15.107:0/3386800635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c4152e00 con 0x7fa7c414eb60 2023-10-25T00:08:56.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.190+0000 7fa7cbb12640 1 -- 172.21.15.107:0/3386800635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c40746e0 con 0x7fa7c414eb60 2023-10-25T00:08:56.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.190+0000 7fa7ba7fc640 1 -- 172.21.15.107:0/3386800635 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa7bc0203b0 con 0x7fa7c414eb60 2023-10-25T00:08:56.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.190+0000 7fa7ba7fc640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7a8065e60 0x7fa7a8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:56.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.190+0000 7fa7ba7fc640 1 -- 172.21.15.107:0/3386800635 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7fa7bc084710 con 0x7fa7c414eb60 2023-10-25T00:08:56.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.190+0000 7fa7c98ae640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7a8065e60 0x7fa7a8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:56.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.190+0000 7fa7c98ae640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7a8065e60 0x7fa7a8068320 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009b40 tx=0x7fa7c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:56.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.194+0000 7fa7ba7fc640 1 -- 172.21.15.107:0/3386800635 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa7bc051160 con 0x7fa7c414eb60 2023-10-25T00:08:56.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.386+0000 7fa7cbb12640 1 -- 172.21.15.107:0/3386800635 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7c414aea0 con 0x7fa7c414eb60 2023-10-25T00:08:56.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.386+0000 7fa7ba7fc640 1 -- 172.21.15.107:0/3386800635 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5779 (secure 0 0 0) 0x7fa7bc054780 con 0x7fa7c414eb60 2023-10-25T00:08:56.389 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:56.389 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7578,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":140}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:11.734690+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6394":{"gid":6394,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":6,"addr":"172.21.15.110:6835/3836115666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":3836115666},{"type":"v1","addr":"172.21.15.110:6835","nonce":3836115666}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:56.389 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 140 2023-10-25T00:08:56.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 -- 172.21.15.107:0/3386800635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7a8065e60 msgr2=0x7fa7a8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:56.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7a8065e60 0x7fa7a8068320 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009b40 tx=0x7fa7c0008040 comp rx=0 tx=0).stop 2023-10-25T00:08:56.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 -- 172.21.15.107:0/3386800635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7c414eb60 msgr2=0x7fa7c40aa4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:56.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7c414eb60 0x7fa7c40aa4a0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fa7c406ebb0 tx=0x7fa7bc00f4a0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 -- 172.21.15.107:0/3386800635 shutdown_connections 2023-10-25T00:08:56.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa7c414eb60 0x7fa7c40aa4a0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa7c414be20 0x7fa7c41528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa7a8065e60 0x7fa7a8068320 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 --2- 172.21.15.107:0/3386800635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa7b00047d0 0x7fa7c40c47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:56.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 -- 172.21.15.107:0/3386800635 >> 172.21.15.107:0/3386800635 conn(0x7fa7c40c2950 msgr2=0x7fa7c414feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:56.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 -- 172.21.15.107:0/3386800635 shutdown_connections 2023-10-25T00:08:56.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:56.390+0000 7fa7cbb12640 1 -- 172.21.15.107:0/3386800635 wait complete. 2023-10-25T00:08:56.407 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:08:58.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:58.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f042e5e6640 1 Processor -- start 2023-10-25T00:08:58.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f042e5e6640 1 -- start start 2023-10-25T00:08:58.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f042e5e6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042806a230 0x7f04281551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f042e5e6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04280c1860 con 0x7f042806af70 2023-10-25T00:08:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f042e5e6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0428154d50 con 0x7f042814b600 2023-10-25T00:08:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f042e5e6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042814b9e0 con 0x7f042806a230 2023-10-25T00:08:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f041ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f042806af70 0x7f042806c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33390/0 (socket says 172.21.15.107:33390) 2023-10-25T00:08:58.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f041ffff640 1 -- 172.21.15.107:0/2121032266 learned_addr learned my addr 172.21.15.107:0/2121032266 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f042cb83640 1 --2- 172.21.15.107:0/2121032266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042806a230 0x7f04281551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587055890 0 0) 0x7f04280c1860 con 0x7f042806af70 2023-10-25T00:08:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0414003530 con 0x7f042806af70 2023-10-25T00:08:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1403628783 0 0) 0x7f0414003530 con 0x7f042806af70 2023-10-25T00:08:58.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042806a230 msgr2=0x7f04281551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 --2- 172.21.15.107:0/2121032266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042806a230 0x7f04281551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 >> v1:172.21.15.110:6789/0 conn(0x7f042814b600 legacy=0x7f0428151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04280c1ab0 con 0x7f042806af70 2023-10-25T00:08:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f0410001680 con 0x7f042806af70 2023-10-25T00:08:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 >> v1:172.21.15.107:6789/0 conn(0x7f042806af70 legacy=0x7f042806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:58.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 --2- 172.21.15.107:0/2121032266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04140039b0 0x7f0414003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 --2- 172.21.15.107:0/2121032266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04140047d0 0x7f0414004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 --2- 172.21.15.107:0/2121032266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042806a230 0x7f04140051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042814b9e0 con 0x7f04140039b0 2023-10-25T00:08:58.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041400bac0 con 0x7f04140047d0 2023-10-25T00:08:58.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041400bc80 con 0x7f042806a230 2023-10-25T00:08:58.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f0427fff640 1 --2- 172.21.15.107:0/2121032266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04140039b0 0x7f0414003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f0427fff640 1 -- 172.21.15.107:0/2121032266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042806a230 msgr2=0x7f04140051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:58.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f0427fff640 1 --2- 172.21.15.107:0/2121032266 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042806a230 0x7f04140051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f041ffff640 1 --2- 172.21.15.107:0/2121032266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04140047d0 0x7f0414004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f0427fff640 1 -- 172.21.15.107:0/2121032266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04140047d0 msgr2=0x7f0414004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f0427fff640 1 --2- 172.21.15.107:0/2121032266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04140047d0 0x7f0414004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.538+0000 7f0427fff640 1 -- 172.21.15.107:0/2121032266 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0414003530 con 0x7f04140039b0 2023-10-25T00:08:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f0427fff640 1 --2- 172.21.15.107:0/2121032266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04140039b0 0x7f0414003dd0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f0418001360 tx=0x7f041801e5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8dac34c413018f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:58.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04180052d0 con 0x7f04140039b0 2023-10-25T00:08:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0418005450 con 0x7f04140039b0 2023-10-25T00:08:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f04277fe640 1 -- 172.21.15.107:0/2121032266 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0418003840 con 0x7f04140039b0 2023-10-25T00:08:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 -- 172.21.15.107:0/2121032266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04140039b0 msgr2=0x7f0414003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 --2- 172.21.15.107:0/2121032266 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04140039b0 0x7f0414003dd0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f0418001360 tx=0x7f041801e5e0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042cb83640 1 -- 172.21.15.107:0/2121032266 reap_dead start 2023-10-25T00:08:58.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 -- 172.21.15.107:0/2121032266 shutdown_connections 2023-10-25T00:08:58.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 -- 172.21.15.107:0/2121032266 >> 172.21.15.107:0/2121032266 conn(0x7f04280c2950 msgr2=0x7f04280c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:58.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 -- 172.21.15.107:0/2121032266 shutdown_connections 2023-10-25T00:08:58.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 -- 172.21.15.107:0/2121032266 wait complete. 2023-10-25T00:08:58.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 Processor -- start 2023-10-25T00:08:58.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 -- start start 2023-10-25T00:08:58.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04140039b0 0x7f04281ed480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042814e8c0 0x7f04280c4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04281e1bd0 0x7f04281e3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042814c520 con 0x7f04281e1bd0 2023-10-25T00:08:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042814c800 con 0x7f04140039b0 2023-10-25T00:08:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042e5e6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04281ec9a0 con 0x7f042814e8c0 2023-10-25T00:08:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042cb83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04281e1bd0 0x7f04281e3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042cb83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04281e1bd0 0x7f04281e3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47146/0 (socket says 172.21.15.107:47146) 2023-10-25T00:08:58.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042cb83640 1 -- 172.21.15.107:0/3109028404 learned_addr learned my addr 172.21.15.107:0/3109028404 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:58.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f041ffff640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042814e8c0 0x7f04280c4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f0427fff640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04140039b0 0x7f04281ed480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.542+0000 7f042cb83640 1 -- 172.21.15.107:0/3109028404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042814e8c0 msgr2=0x7f04280c4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:58.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f042cb83640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042814e8c0 0x7f04280c4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f042cb83640 1 -- 172.21.15.107:0/3109028404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04140039b0 msgr2=0x7f04281ed480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:58.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f042cb83640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04140039b0 0x7f04281ed480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f042cb83640 1 -- 172.21.15.107:0/3109028404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04281ecc80 con 0x7f04281e1bd0 2023-10-25T00:08:58.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f041ffff640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042814e8c0 0x7f04280c4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:58.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f0427fff640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04140039b0 0x7f04281ed480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:08:58.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f042cb83640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04281e1bd0 0x7f04281e3fb0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f042806d370 tx=0x7f042000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:58.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f04257fa640 1 -- 172.21.15.107:0/3109028404 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420010040 con 0x7f04281e1bd0 2023-10-25T00:08:58.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f04257fa640 1 -- 172.21.15.107:0/3109028404 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042000bce0 con 0x7f04281e1bd0 2023-10-25T00:08:58.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f042e5e6640 1 -- 172.21.15.107:0/3109028404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04281ecf60 con 0x7f04281e1bd0 2023-10-25T00:08:58.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f04257fa640 1 -- 172.21.15.107:0/3109028404 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420014640 con 0x7f04281e1bd0 2023-10-25T00:08:58.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f042e5e6640 1 -- 172.21.15.107:0/3109028404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042814c160 con 0x7f04281e1bd0 2023-10-25T00:08:58.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f042e5e6640 1 -- 172.21.15.107:0/3109028404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042814eca0 con 0x7f04281e1bd0 2023-10-25T00:08:58.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f04257fa640 1 -- 172.21.15.107:0/3109028404 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f04200147a0 con 0x7f04281e1bd0 2023-10-25T00:08:58.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f04257fa640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0408065e90 0x7f0408068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f04257fa640 1 -- 172.21.15.107:0/3109028404 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 9197+0+0 (secure 0 0 0) 0x7f0420084b70 con 0x7f04281e1bd0 2023-10-25T00:08:58.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f0427fff640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0408065e90 0x7f0408068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.546+0000 7f0427fff640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0408065e90 0x7f0408068350 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0418001750 tx=0x7f04180011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:58.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.550+0000 7f04257fa640 1 -- 172.21.15.107:0/3109028404 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f04200515c0 con 0x7f04281e1bd0 2023-10-25T00:08:58.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.746+0000 7f042e5e6640 1 -- 172.21.15.107:0/3109028404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04280746e0 con 0x7f04281e1bd0 2023-10-25T00:08:58.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.746+0000 7f04257fa640 1 -- 172.21.15.107:0/3109028404 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+5018 (secure 0 0 0) 0x7f0420018090 con 0x7f04281e1bd0 2023-10-25T00:08:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:58.748 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:58.701553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:58.748 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 141 2023-10-25T00:08:58.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 -- 172.21.15.107:0/3109028404 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0408065e90 msgr2=0x7f0408068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:58.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0408065e90 0x7f0408068350 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0418001750 tx=0x7f04180011a0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 -- 172.21.15.107:0/3109028404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04281e1bd0 msgr2=0x7f04281e3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:58.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04281e1bd0 0x7f04281e3fb0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f042806d370 tx=0x7f042000b480 comp rx=0 tx=0).stop 2023-10-25T00:08:58.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 -- 172.21.15.107:0/3109028404 shutdown_connections 2023-10-25T00:08:58.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04281e1bd0 0x7f04281e3fb0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0408065e90 0x7f0408068350 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f042814e8c0 0x7f04280c4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 --2- 172.21.15.107:0/3109028404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04140039b0 0x7f04281ed480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 -- 172.21.15.107:0/3109028404 >> 172.21.15.107:0/3109028404 conn(0x7f04280c2950 msgr2=0x7f042815a290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:58.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 -- 172.21.15.107:0/3109028404 shutdown_connections 2023-10-25T00:08:58.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.750+0000 7f042e5e6640 1 -- 172.21.15.107:0/3109028404 wait complete. 2023-10-25T00:08:58.772 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2023-10-25T00:08:58.772 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:08:58.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:08:58.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.878+0000 7fbc857fd640 1 Processor -- start 2023-10-25T00:08:58.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.878+0000 7fbc857fd640 1 -- start start 2023-10-25T00:08:58.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbc80152020 0x7fbc800c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbc801553d0 0x7fbc8006ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc80152400 con 0x7fbc800c8640 2023-10-25T00:08:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc801558d0 con 0x7fbc80152020 2023-10-25T00:08:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc800c6ce0 con 0x7fbc801553d0 2023-10-25T00:08:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbc80152020 0x7fbc800c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7e7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbc800c8640 0x7fbc8006a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:33400/0 (socket says 172.21.15.107:33400) 2023-10-25T00:08:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbc80152020 0x7fbc800c4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47156/0 (socket says 172.21.15.107:47156) 2023-10-25T00:08:58.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7e7fc640 1 -- 172.21.15.107:0/3927975499 learned_addr learned my addr 172.21.15.107:0/3927975499 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:08:58.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7f7fe640 1 --2- 172.21.15.107:0/3927975499 >> v2:172.21.15.110:3300/0 conn(0x7fbc801553d0 0x7fbc8006ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7effd640 1 -- 172.21.15.107:0/3927975499 >> v2:172.21.15.110:3300/0 conn(0x7fbc801553d0 msgr2=0x7fbc8006ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:08:58.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7effd640 1 --2- 172.21.15.107:0/3927975499 >> v2:172.21.15.110:3300/0 conn(0x7fbc801553d0 0x7fbc8006ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7effd640 1 -- 172.21.15.107:0/3927975499 >> v1:172.21.15.107:6789/0 conn(0x7fbc800c8640 legacy=0x7fbc8006a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:58.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7effd640 1 -- 172.21.15.107:0/3927975499 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc800c6fc0 con 0x7fbc80152020 2023-10-25T00:08:58.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7effd640 1 --2- 172.21.15.107:0/3927975499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc80152020 0x7fbc800c4bb0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fbc800ca110 tx=0x7fbc7001c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec203be9bffd9256 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:58.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7dffb640 1 -- 172.21.15.107:0/3927975499 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc700ab600 con 0x7fbc80152020 2023-10-25T00:08:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7dffb640 1 -- 172.21.15.107:0/3927975499 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc700a7050 con 0x7fbc80152020 2023-10-25T00:08:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc7dffb640 1 -- 172.21.15.107:0/3927975499 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc700ab980 con 0x7fbc80152020 2023-10-25T00:08:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 -- 172.21.15.107:0/3927975499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc80152020 msgr2=0x7fbc800c4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 --2- 172.21.15.107:0/3927975499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc80152020 0x7fbc800c4bb0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fbc800ca110 tx=0x7fbc7001c230 comp rx=0 tx=0).stop 2023-10-25T00:08:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 -- 172.21.15.107:0/3927975499 shutdown_connections 2023-10-25T00:08:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 --2- 172.21.15.107:0/3927975499 >> v2:172.21.15.110:3300/0 conn(0x7fbc801553d0 0x7fbc8006ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 --2- 172.21.15.107:0/3927975499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc80152020 0x7fbc800c4bb0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 -- 172.21.15.107:0/3927975499 >> 172.21.15.107:0/3927975499 conn(0x7fbc800abd60 msgr2=0x7fbc800ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 -- 172.21.15.107:0/3927975499 shutdown_connections 2023-10-25T00:08:58.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.882+0000 7fbc857fd640 1 -- 172.21.15.107:0/3927975499 wait complete. 2023-10-25T00:08:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc857fd640 1 Processor -- start 2023-10-25T00:08:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc857fd640 1 -- start start 2023-10-25T00:08:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc857fd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc800c8640 0x7fbc800a9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc857fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc80152020 0x7fbc800c05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc857fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc801553d0 0x7fbc8014fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc857fd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800be740 con 0x7fbc80152020 2023-10-25T00:08:58.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc857fd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc80151420 con 0x7fbc800c8640 2023-10-25T00:08:58.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc857fd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc801516c0 con 0x7fbc801553d0 2023-10-25T00:08:58.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc7e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc80152020 0x7fbc800c05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc7e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc80152020 0x7fbc800c05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47168/0 (socket says 172.21.15.107:47168) 2023-10-25T00:08:58.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc7e7fc640 1 -- 172.21.15.107:0/444215933 learned_addr learned my addr 172.21.15.107:0/444215933 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:08:58.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc7effd640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc800c8640 0x7fbc800a9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc7f7fe640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc801553d0 0x7fbc8014fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc7e7fc640 1 -- 172.21.15.107:0/444215933 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc801553d0 msgr2=0x7fbc8014fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:58.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc7e7fc640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc801553d0 0x7fbc8014fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc7e7fc640 1 -- 172.21.15.107:0/444215933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc800c8640 msgr2=0x7fbc800a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:58.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc7e7fc640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc800c8640 0x7fbc800a9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:58.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc7e7fc640 1 -- 172.21.15.107:0/444215933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc70009420 con 0x7fbc80152020 2023-10-25T00:08:58.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc7e7fc640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc80152020 0x7fbc800c05b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400c750 tx=0x7fbc7400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:58.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc677fe640 1 -- 172.21.15.107:0/444215933 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc740072d0 con 0x7fbc80152020 2023-10-25T00:08:58.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc677fe640 1 -- 172.21.15.107:0/444215933 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc74007430 con 0x7fbc80152020 2023-10-25T00:08:58.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc677fe640 1 -- 172.21.15.107:0/444215933 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc740047f0 con 0x7fbc80152020 2023-10-25T00:08:58.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc857fd640 1 -- 172.21.15.107:0/444215933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc800be950 con 0x7fbc80152020 2023-10-25T00:08:58.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc857fd640 1 -- 172.21.15.107:0/444215933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8014f110 con 0x7fbc80152020 2023-10-25T00:08:58.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc857fd640 1 -- 172.21.15.107:0/444215933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc80151880 con 0x7fbc80152020 2023-10-25T00:08:58.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.886+0000 7fbc677fe640 1 -- 172.21.15.107:0/444215933 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbc74004950 con 0x7fbc80152020 2023-10-25T00:08:58.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.890+0000 7fbc677fe640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc6c065e60 0x7fbc6c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:08:58.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.890+0000 7fbc7effd640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc6c065e60 0x7fbc6c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:08:58.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.890+0000 7fbc677fe640 1 -- 172.21.15.107:0/444215933 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7fbc740838c0 con 0x7fbc80152020 2023-10-25T00:08:58.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.890+0000 7fbc7effd640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc6c065e60 0x7fbc6c068320 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fbc700ad000 tx=0x7fbc70008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:08:58.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:58.894+0000 7fbc677fe640 1 -- 172.21.15.107:0/444215933 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbc74050220 con 0x7fbc80152020 2023-10-25T00:08:59.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.098+0000 7fbc857fd640 1 -- 172.21.15.107:0/444215933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc800beae0 con 0x7fbc80152020 2023-10-25T00:08:59.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.098+0000 7fbc677fe640 1 -- 172.21.15.107:0/444215933 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+5018 (secure 0 0 0) 0x7fbc74018030 con 0x7fbc80152020 2023-10-25T00:08:59.100 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:08:59.100 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:08:58.701553+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:08:59.100 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 141 2023-10-25T00:08:59.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 -- 172.21.15.107:0/444215933 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc6c065e60 msgr2=0x7fbc6c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:59.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc6c065e60 0x7fbc6c068320 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fbc700ad000 tx=0x7fbc70008040 comp rx=0 tx=0).stop 2023-10-25T00:08:59.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 -- 172.21.15.107:0/444215933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc80152020 msgr2=0x7fbc800c05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:08:59.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc80152020 0x7fbc800c05b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400c750 tx=0x7fbc7400cc10 comp rx=0 tx=0).stop 2023-10-25T00:08:59.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 -- 172.21.15.107:0/444215933 shutdown_connections 2023-10-25T00:08:59.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbc6c065e60 0x7fbc6c068320 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:59.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc801553d0 0x7fbc8014fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:59.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc80152020 0x7fbc800c05b0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:59.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 --2- 172.21.15.107:0/444215933 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc800c8640 0x7fbc800a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:08:59.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 -- 172.21.15.107:0/444215933 >> 172.21.15.107:0/444215933 conn(0x7fbc800abd60 msgr2=0x7fbc8006a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:08:59.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 -- 172.21.15.107:0/444215933 shutdown_connections 2023-10-25T00:08:59.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:08:59.102+0000 7fbc857fd640 1 -- 172.21.15.107:0/444215933 wait complete. 2023-10-25T00:08:59.118 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 141, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 141, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7578": { "addr": "172.21.15.107:6837/1788268787", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 1788268787, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 1788268787, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7578, "incarnation": 141, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:08:58.701553+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7578 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 139, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7512": { "addr": "172.21.15.107:6835/2276136116", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2276136116, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 2276136116, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7512, "incarnation": 136, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:08:26.829276+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7512 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:09:01.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:01.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1befc640 1 Processor -- start 2023-10-25T00:09:01.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1befc640 1 -- start start 2023-10-25T00:09:01.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1befc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcf14152160 0x7fcf14158890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:01.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1befc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf140ac9d0 0x7fcf1406cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:01.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1befc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcf1406d430 0x7fcf14154a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:01.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1befc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1406fe90 con 0x7fcf1406d430 2023-10-25T00:09:01.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1befc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf14158dd0 con 0x7fcf14152160 2023-10-25T00:09:01.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1befc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf140ad0d0 con 0x7fcf140ac9d0 2023-10-25T00:09:01.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1a499640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcf1406d430 0x7fcf14154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:01.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1a499640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcf1406d430 0x7fcf14154a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47170/0 (socket says 172.21.15.107:47170) 2023-10-25T00:09:01.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1a499640 1 -- 172.21.15.107:0/778824938 learned_addr learned my addr 172.21.15.107:0/778824938 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:01.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf19c98640 1 --2- 172.21.15.107:0/778824938 >> v2:172.21.15.110:3300/0 conn(0x7fcf14152160 0x7fcf14158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:01.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf19497640 1 --2- 172.21.15.107:0/778824938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf140ac9d0 0x7fcf1406cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:01.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1a499640 1 -- 172.21.15.107:0/778824938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf140ac9d0 msgr2=0x7fcf1406cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:01.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1a499640 1 --2- 172.21.15.107:0/778824938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf140ac9d0 0x7fcf1406cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:01.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1a499640 1 -- 172.21.15.107:0/778824938 >> v2:172.21.15.110:3300/0 conn(0x7fcf14152160 msgr2=0x7fcf14158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:01.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1a499640 1 --2- 172.21.15.107:0/778824938 >> v2:172.21.15.110:3300/0 conn(0x7fcf14152160 0x7fcf14158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:01.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf1a499640 1 -- 172.21.15.107:0/778824938 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf140c0300 con 0x7fcf1406d430 2023-10-25T00:09:01.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.254+0000 7fcf19497640 1 --2- 172.21.15.107:0/778824938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf140ac9d0 0x7fcf1406cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:09:01.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1a499640 1 --2- 172.21.15.107:0/778824938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf1406d430 0x7fcf14154a10 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fcf1406df70 tx=0x7fcf0c01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d911814cb6192a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:01.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf18c96640 1 -- 172.21.15.107:0/778824938 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0a5700 con 0x7fcf1406d430 2023-10-25T00:09:01.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf18c96640 1 -- 172.21.15.107:0/778824938 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c0a5860 con 0x7fcf1406d430 2023-10-25T00:09:01.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf18c96640 1 -- 172.21.15.107:0/778824938 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0a5bf0 con 0x7fcf1406d430 2023-10-25T00:09:01.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 -- 172.21.15.107:0/778824938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf1406d430 msgr2=0x7fcf14154a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:01.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 --2- 172.21.15.107:0/778824938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf1406d430 0x7fcf14154a10 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fcf1406df70 tx=0x7fcf0c01c880 comp rx=0 tx=0).stop 2023-10-25T00:09:01.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 -- 172.21.15.107:0/778824938 shutdown_connections 2023-10-25T00:09:01.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 --2- 172.21.15.107:0/778824938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf1406d430 0x7fcf14154a10 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:01.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 --2- 172.21.15.107:0/778824938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf140ac9d0 0x7fcf1406cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:01.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 --2- 172.21.15.107:0/778824938 >> v2:172.21.15.110:3300/0 conn(0x7fcf14152160 0x7fcf14158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:01.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 -- 172.21.15.107:0/778824938 >> 172.21.15.107:0/778824938 conn(0x7fcf140c6ed0 msgr2=0x7fcf140c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:01.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 -- 172.21.15.107:0/778824938 shutdown_connections 2023-10-25T00:09:01.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 -- 172.21.15.107:0/778824938 wait complete. 2023-10-25T00:09:01.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 Processor -- start 2023-10-25T00:09:01.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 -- start start 2023-10-25T00:09:01.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1406d430 0x7fcf140a9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:01.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf140ac9d0 0x7fcf1414d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:01.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf14152160 0x7fcf140c1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:01.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1414dd60 con 0x7fcf14152160 2023-10-25T00:09:01.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1415e000 con 0x7fcf140ac9d0 2023-10-25T00:09:01.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1415da40 con 0x7fcf1406d430 2023-10-25T00:09:01.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1a499640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf14152160 0x7fcf140c1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:01.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1a499640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf14152160 0x7fcf140c1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47180/0 (socket says 172.21.15.107:47180) 2023-10-25T00:09:01.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1a499640 1 -- 172.21.15.107:0/1246462867 learned_addr learned my addr 172.21.15.107:0/1246462867 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:01.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf19c98640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1406d430 0x7fcf140a9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:01.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf19497640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf140ac9d0 0x7fcf1414d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:01.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1a499640 1 -- 172.21.15.107:0/1246462867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1406d430 msgr2=0x7fcf140a9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:01.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1a499640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1406d430 0x7fcf140a9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:01.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1a499640 1 -- 172.21.15.107:0/1246462867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf140ac9d0 msgr2=0x7fcf1414d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:01.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1a499640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf140ac9d0 0x7fcf1414d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:01.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1a499640 1 -- 172.21.15.107:0/1246462867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c009b30 con 0x7fcf14152160 2023-10-25T00:09:01.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1a499640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf14152160 0x7fcf140c1f80 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0af040 tx=0x7fcf0c019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:01.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf0a7fc640 1 -- 172.21.15.107:0/1246462867 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0b8670 con 0x7fcf14152160 2023-10-25T00:09:01.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf0a7fc640 1 -- 172.21.15.107:0/1246462867 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c0b87d0 con 0x7fcf14152160 2023-10-25T00:09:01.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 -- 172.21.15.107:0/1246462867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1415dd20 con 0x7fcf14152160 2023-10-25T00:09:01.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf0a7fc640 1 -- 172.21.15.107:0/1246462867 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0b8b60 con 0x7fcf14152160 2023-10-25T00:09:01.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.258+0000 7fcf1befc640 1 -- 172.21.15.107:0/1246462867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf14152b70 con 0x7fcf14152160 2023-10-25T00:09:01.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.262+0000 7fcf1befc640 1 -- 172.21.15.107:0/1246462867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf140c38e0 con 0x7fcf14152160 2023-10-25T00:09:01.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.262+0000 7fcf0a7fc640 1 -- 172.21.15.107:0/1246462867 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcf0c0b8cc0 con 0x7fcf14152160 2023-10-25T00:09:01.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.262+0000 7fcf0a7fc640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcef8065e60 0x7fcef8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:01.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.262+0000 7fcf0a7fc640 1 -- 172.21.15.107:0/1246462867 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7fcf0c11f8f0 con 0x7fcf14152160 2023-10-25T00:09:01.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.262+0000 7fcf19c98640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcef8065e60 0x7fcef8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:01.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.262+0000 7fcf19c98640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcef8065e60 0x7fcef8068320 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000b930 tx=0x7fcf10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:01.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.266+0000 7fcf0a7fc640 1 -- 172.21.15.107:0/1246462867 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fcf0c0ec2e0 con 0x7fcf14152160 2023-10-25T00:09:01.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.458+0000 7fcf1befc640 1 -- 172.21.15.107:0/1246462867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf140746e0 con 0x7fcf14152160 2023-10-25T00:09:01.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.458+0000 7fcf0a7fc640 1 -- 172.21.15.107:0/1246462867 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+5019 (secure 0 0 0) 0x7fcf0c0bc070 con 0x7fcf14152160 2023-10-25T00:09:01.460 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:01.460 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:00.751428+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:01.461 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 143 2023-10-25T00:09:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 -- 172.21.15.107:0/1246462867 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcef8065e60 msgr2=0x7fcef8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcef8065e60 0x7fcef8068320 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000b930 tx=0x7fcf10008040 comp rx=0 tx=0).stop 2023-10-25T00:09:01.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 -- 172.21.15.107:0/1246462867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf14152160 msgr2=0x7fcf140c1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf14152160 0x7fcf140c1f80 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0af040 tx=0x7fcf0c019620 comp rx=0 tx=0).stop 2023-10-25T00:09:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 -- 172.21.15.107:0/1246462867 shutdown_connections 2023-10-25T00:09:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf14152160 0x7fcf140c1f80 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:01.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf140ac9d0 0x7fcf1414d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcef8065e60 0x7fcef8068320 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 --2- 172.21.15.107:0/1246462867 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf1406d430 0x7fcf140a9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 -- 172.21.15.107:0/1246462867 >> 172.21.15.107:0/1246462867 conn(0x7fcf140c6ed0 msgr2=0x7fcf14150810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 -- 172.21.15.107:0/1246462867 shutdown_connections 2023-10-25T00:09:01.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:01.462+0000 7fcf1befc640 1 -- 172.21.15.107:0/1246462867 wait complete. 2023-10-25T00:09:01.479 INFO:tasks.mds_thrash.fs.[a]:no change 2023-10-25T00:09:02.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:03.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf682f3640 1 Processor -- start 2023-10-25T00:09:03.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf682f3640 1 -- start start 2023-10-25T00:09:03.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf682f3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf60156770 0x7fdf600ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf682f3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdf600be6b0 0x7fdf6015d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf682f3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf600c1ca0 con 0x7fdf600c70e0 2023-10-25T00:09:03.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf682f3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf600c27e0 con 0x7fdf600be6b0 2023-10-25T00:09:03.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf682f3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf600c10c0 con 0x7fdf60156770 2023-10-25T00:09:03.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf6608f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf60156770 0x7fdf600ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf66890640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdf600be6b0 0x7fdf6015d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf66890640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdf600be6b0 0x7fdf6015d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:55666/0 (socket says 172.21.15.107:55666) 2023-10-25T00:09:03.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf66890640 1 -- 172.21.15.107:0/961153074 learned_addr learned my addr 172.21.15.107:0/961153074 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:03.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf6588e640 1 --1- 172.21.15.107:0/961153074 >> v1:172.21.15.110:6789/0 conn(0x7fdf600c70e0 0x7fdf600bdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:51924/0 (socket says 172.21.15.107:51924) 2023-10-25T00:09:03.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf66890640 1 -- 172.21.15.107:0/961153074 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf60156770 msgr2=0x7fdf600ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:03.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf66890640 1 --2- 172.21.15.107:0/961153074 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf60156770 0x7fdf600ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf66890640 1 -- 172.21.15.107:0/961153074 >> v1:172.21.15.110:6789/0 conn(0x7fdf600c70e0 legacy=0x7fdf600bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf66890640 1 -- 172.21.15.107:0/961153074 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf600c13a0 con 0x7fdf600be6b0 2023-10-25T00:09:03.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.126+0000 7fdf6608f640 1 --2- 172.21.15.107:0/961153074 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf60156770 0x7fdf600ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:09:03.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf66890640 1 --2- 172.21.15.107:0/961153074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf600be6b0 0x7fdf6015d130 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fdf58009e50 tx=0x7fdf5801c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8cf1fb8c91d4f2d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:03.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf6508d640 1 -- 172.21.15.107:0/961153074 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf580a5740 con 0x7fdf600be6b0 2023-10-25T00:09:03.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf6508d640 1 -- 172.21.15.107:0/961153074 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf580a58a0 con 0x7fdf600be6b0 2023-10-25T00:09:03.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf6508d640 1 -- 172.21.15.107:0/961153074 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf580a5c30 con 0x7fdf600be6b0 2023-10-25T00:09:03.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf682f3640 1 -- 172.21.15.107:0/961153074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf600be6b0 msgr2=0x7fdf6015d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf682f3640 1 --2- 172.21.15.107:0/961153074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf600be6b0 0x7fdf6015d130 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fdf58009e50 tx=0x7fdf5801c8c0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf682f3640 1 -- 172.21.15.107:0/961153074 shutdown_connections 2023-10-25T00:09:03.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf682f3640 1 --2- 172.21.15.107:0/961153074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf600be6b0 0x7fdf6015d130 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf682f3640 1 --2- 172.21.15.107:0/961153074 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf60156770 0x7fdf600ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf682f3640 1 -- 172.21.15.107:0/961153074 >> 172.21.15.107:0/961153074 conn(0x7fdf600c65c0 msgr2=0x7fdf600c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:03.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf682f3640 1 -- 172.21.15.107:0/961153074 shutdown_connections 2023-10-25T00:09:03.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.130+0000 7fdf682f3640 1 -- 172.21.15.107:0/961153074 wait complete. 2023-10-25T00:09:03.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf682f3640 1 Processor -- start 2023-10-25T00:09:03.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf682f3640 1 -- start start 2023-10-25T00:09:03.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf682f3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf600be6b0 0x7fdf600a9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf682f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf600c70e0 0x7fdf60150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf682f3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf60156770 0x7fdf60071310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf682f3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf60072470 con 0x7fdf600c70e0 2023-10-25T00:09:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf682f3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf601f09e0 con 0x7fdf60156770 2023-10-25T00:09:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf682f3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf60070830 con 0x7fdf600be6b0 2023-10-25T00:09:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf6588e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf600c70e0 0x7fdf60150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf6588e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf600c70e0 0x7fdf60150360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35366/0 (socket says 172.21.15.107:35366) 2023-10-25T00:09:03.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf6588e640 1 -- 172.21.15.107:0/817096069 learned_addr learned my addr 172.21.15.107:0/817096069 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:03.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf6608f640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf600be6b0 0x7fdf600a9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf66890640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf60156770 0x7fdf60071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf6588e640 1 -- 172.21.15.107:0/817096069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf600be6b0 msgr2=0x7fdf600a9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf6588e640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf600be6b0 0x7fdf600a9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf6588e640 1 -- 172.21.15.107:0/817096069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf60156770 msgr2=0x7fdf60071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf6588e640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf60156770 0x7fdf60071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf6588e640 1 -- 172.21.15.107:0/817096069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c009420 con 0x7fdf600c70e0 2023-10-25T00:09:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf6588e640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf600c70e0 0x7fdf60150360 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000c420 tx=0x7fdf5000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:03.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf56ffd640 1 -- 172.21.15.107:0/817096069 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf500074e0 con 0x7fdf600c70e0 2023-10-25T00:09:03.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf56ffd640 1 -- 172.21.15.107:0/817096069 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf50007640 con 0x7fdf600c70e0 2023-10-25T00:09:03.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf56ffd640 1 -- 172.21.15.107:0/817096069 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf50004a30 con 0x7fdf600c70e0 2023-10-25T00:09:03.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf682f3640 1 -- 172.21.15.107:0/817096069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf58009b30 con 0x7fdf600c70e0 2023-10-25T00:09:03.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf682f3640 1 -- 172.21.15.107:0/817096069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf60070df0 con 0x7fdf600c70e0 2023-10-25T00:09:03.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf682f3640 1 -- 172.21.15.107:0/817096069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf60070690 con 0x7fdf600c70e0 2023-10-25T00:09:03.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.134+0000 7fdf56ffd640 1 -- 172.21.15.107:0/817096069 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdf50004b90 con 0x7fdf600c70e0 2023-10-25T00:09:03.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.138+0000 7fdf56ffd640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf44065e60 0x7fdf44068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.138+0000 7fdf56ffd640 1 -- 172.21.15.107:0/817096069 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7fdf50083f30 con 0x7fdf600c70e0 2023-10-25T00:09:03.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.138+0000 7fdf6608f640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf44065e60 0x7fdf44068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.138+0000 7fdf6608f640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf44065e60 0x7fdf44068320 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009be0 tx=0x7fdf5c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:03.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.142+0000 7fdf56ffd640 1 -- 172.21.15.107:0/817096069 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdf60070690 con 0x7fdf600c70e0 2023-10-25T00:09:03.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.374+0000 7fdf682f3640 1 -- 172.21.15.107:0/817096069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf60070690 con 0x7fdf600c70e0 2023-10-25T00:09:03.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.374+0000 7fdf56ffd640 1 -- 172.21.15.107:0/817096069 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5019 (secure 0 0 0) 0x7fdf60070690 con 0x7fdf600c70e0 2023-10-25T00:09:03.375 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:03.375 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:03.376 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 144 2023-10-25T00:09:03.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 -- 172.21.15.107:0/817096069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf44065e60 msgr2=0x7fdf44068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf44065e60 0x7fdf44068320 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009be0 tx=0x7fdf5c008040 comp rx=0 tx=0).stop 2023-10-25T00:09:03.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 -- 172.21.15.107:0/817096069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf600c70e0 msgr2=0x7fdf60150360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf600c70e0 0x7fdf60150360 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fdf5000c420 tx=0x7fdf5000c8e0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 -- 172.21.15.107:0/817096069 shutdown_connections 2023-10-25T00:09:03.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf60156770 0x7fdf60071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf600c70e0 0x7fdf60150360 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf44065e60 0x7fdf44068320 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 --2- 172.21.15.107:0/817096069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf600be6b0 0x7fdf600a9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 -- 172.21.15.107:0/817096069 >> 172.21.15.107:0/817096069 conn(0x7fdf600c65c0 msgr2=0x7fdf600bea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:03.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 -- 172.21.15.107:0/817096069 shutdown_connections 2023-10-25T00:09:03.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.378+0000 7fdf682f3640 1 -- 172.21.15.107:0/817096069 wait complete. 2023-10-25T00:09:03.395 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2023-10-25T00:09:03.395 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:09:03.396 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:09:03.405 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:09:03.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:03.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:03.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.582+0000 7f22bdf62640 1 Processor -- start 2023-10-25T00:09:03.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.582+0000 7f22bdf62640 1 -- start start 2023-10-25T00:09:03.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.582+0000 7f22bdf62640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22b8156770 0x7f22b80ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.582+0000 7f22bdf62640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f22b80c70e0 0x7f22b80bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.582+0000 7f22bdf62640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f22b80c2130 0x7f22b80be4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.582+0000 7f22bdf62640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b80c27e0 con 0x7f22b80c70e0 2023-10-25T00:09:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.582+0000 7f22bdf62640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b80c1ca0 con 0x7f22b80c2130 2023-10-25T00:09:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.582+0000 7f22bdf62640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b80c10c0 con 0x7f22b8156770 2023-10-25T00:09:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.582+0000 7f22affff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f22b80c70e0 0x7f22b80bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22affff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f22b80c70e0 0x7f22b80bdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35378/0 (socket says 172.21.15.107:35378) 2023-10-25T00:09:03.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22affff640 1 -- 172.21.15.107:0/2307498396 learned_addr learned my addr 172.21.15.107:0/2307498396 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:03.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22b77fe640 1 --2- 172.21.15.107:0/2307498396 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22b8156770 0x7f22b80ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22b7fff640 1 --2- 172.21.15.107:0/2307498396 >> v2:172.21.15.110:3300/0 conn(0x7f22b80c2130 0x7f22b80be4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22affff640 1 -- 172.21.15.107:0/2307498396 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22b8156770 msgr2=0x7f22b80ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:03.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22affff640 1 --2- 172.21.15.107:0/2307498396 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22b8156770 0x7f22b80ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22affff640 1 -- 172.21.15.107:0/2307498396 >> v2:172.21.15.110:3300/0 conn(0x7f22b80c2130 msgr2=0x7f22b80be4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:03.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22affff640 1 --2- 172.21.15.107:0/2307498396 >> v2:172.21.15.110:3300/0 conn(0x7f22b80c2130 0x7f22b80be4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22affff640 1 -- 172.21.15.107:0/2307498396 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b80c13a0 con 0x7f22b80c70e0 2023-10-25T00:09:03.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22affff640 1 --2- 172.21.15.107:0/2307498396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22b80c70e0 0x7f22b80bdfa0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f22b80c0190 tx=0x7f22a001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20ce2788fa0eac0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:03.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22b6ffd640 1 -- 172.21.15.107:0/2307498396 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a00ab600 con 0x7f22b80c70e0 2023-10-25T00:09:03.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22b6ffd640 1 -- 172.21.15.107:0/2307498396 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a00a7050 con 0x7f22b80c70e0 2023-10-25T00:09:03.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22b6ffd640 1 -- 172.21.15.107:0/2307498396 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a00ab980 con 0x7f22b80c70e0 2023-10-25T00:09:03.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22bdf62640 1 -- 172.21.15.107:0/2307498396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22b80c70e0 msgr2=0x7f22b80bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22bdf62640 1 --2- 172.21.15.107:0/2307498396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22b80c70e0 0x7f22b80bdfa0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f22b80c0190 tx=0x7f22a001c3e0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22bdf62640 1 -- 172.21.15.107:0/2307498396 shutdown_connections 2023-10-25T00:09:03.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22bdf62640 1 --2- 172.21.15.107:0/2307498396 >> v2:172.21.15.110:3300/0 conn(0x7f22b80c2130 0x7f22b80be4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22bdf62640 1 --2- 172.21.15.107:0/2307498396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22b80c70e0 0x7f22b80bdfa0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22bdf62640 1 --2- 172.21.15.107:0/2307498396 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22b8156770 0x7f22b80ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22bdf62640 1 -- 172.21.15.107:0/2307498396 >> 172.21.15.107:0/2307498396 conn(0x7f22b80c65c0 msgr2=0x7f22b80c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:03.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.586+0000 7f22bdf62640 1 -- 172.21.15.107:0/2307498396 shutdown_connections 2023-10-25T00:09:03.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22bdf62640 1 -- 172.21.15.107:0/2307498396 wait complete. 2023-10-25T00:09:03.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22bdf62640 1 Processor -- start 2023-10-25T00:09:03.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22bdf62640 1 -- start start 2023-10-25T00:09:03.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22bdf62640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22b80c2130 0x7f22b815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22bdf62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22b80c70e0 0x7f22b80bcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22bdf62640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22b8156770 0x7f22b80a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22bdf62640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b81edb70 con 0x7f22b80c70e0 2023-10-25T00:09:03.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22bdf62640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b81ee0a0 con 0x7f22b80c2130 2023-10-25T00:09:03.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22bdf62640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b80ab540 con 0x7f22b8156770 2023-10-25T00:09:03.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22b80c70e0 0x7f22b80bcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22affff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22b80c70e0 0x7f22b80bcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35386/0 (socket says 172.21.15.107:35386) 2023-10-25T00:09:03.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22affff640 1 -- 172.21.15.107:0/1231213040 learned_addr learned my addr 172.21.15.107:0/1231213040 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:03.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22b77fe640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22b80c2130 0x7f22b815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22b7fff640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22b8156770 0x7f22b80a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22affff640 1 -- 172.21.15.107:0/1231213040 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22b8156770 msgr2=0x7f22b80a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22affff640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22b8156770 0x7f22b80a95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22affff640 1 -- 172.21.15.107:0/1231213040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22b80c2130 msgr2=0x7f22b815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22affff640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22b80c2130 0x7f22b815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22affff640 1 -- 172.21.15.107:0/1231213040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a00094d0 con 0x7f22b80c70e0 2023-10-25T00:09:03.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22b77fe640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22b80c2130 0x7f22b815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:09:03.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22affff640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22b80c70e0 0x7f22b80bcfb0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f22a001aa90 tx=0x7f22a0019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:03.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22b4ff9640 1 -- 172.21.15.107:0/1231213040 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a00b4420 con 0x7f22b80c70e0 2023-10-25T00:09:03.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22b4ff9640 1 -- 172.21.15.107:0/1231213040 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a00b4580 con 0x7f22b80c70e0 2023-10-25T00:09:03.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22bdf62640 1 -- 172.21.15.107:0/1231213040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b81eddc0 con 0x7f22b80c70e0 2023-10-25T00:09:03.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22b4ff9640 1 -- 172.21.15.107:0/1231213040 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a00b48f0 con 0x7f22b80c70e0 2023-10-25T00:09:03.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.590+0000 7f22bdf62640 1 -- 172.21.15.107:0/1231213040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b80bc490 con 0x7f22b80c70e0 2023-10-25T00:09:03.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.594+0000 7f22bdf62640 1 -- 172.21.15.107:0/1231213040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b80bcce0 con 0x7f22b80c70e0 2023-10-25T00:09:03.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.594+0000 7f22b4ff9640 1 -- 172.21.15.107:0/1231213040 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f22a00b2070 con 0x7f22b80c70e0 2023-10-25T00:09:03.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.594+0000 7f22b4ff9640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2298065e60 0x7f2298068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.594+0000 7f22b4ff9640 1 -- 172.21.15.107:0/1231213040 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7f22a011dfb0 con 0x7f22b80c70e0 2023-10-25T00:09:03.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.594+0000 7f22b77fe640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2298065e60 0x7f2298068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.594+0000 7f22b77fe640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2298065e60 0x7f2298068320 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f22a8009a90 tx=0x7f22a8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:03.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.598+0000 7f22b4ff9640 1 -- 172.21.15.107:0/1231213040 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f22a014f040 con 0x7f22b80c70e0 2023-10-25T00:09:03.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b9176640 1 Processor -- start 2023-10-25T00:09:03.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b9176640 1 -- start start 2023-10-25T00:09:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b9176640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6b4152040 0x7ff6b40c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b9176640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b4155680 con 0x7ff6b40c8100 2023-10-25T00:09:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b9176640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b4154a70 con 0x7ff6b406a4b0 2023-10-25T00:09:03.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b9176640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b4154d50 con 0x7ff6b4152040 2023-10-25T00:09:03.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6abfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff6b40c8100 0x7ff6b40abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56564/0 (socket says 172.21.15.107:56564) 2023-10-25T00:09:03.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6abfff640 1 -- 172.21.15.107:0/329987430 learned_addr learned my addr 172.21.15.107:0/329987430 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:09:03.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b2d9d640 1 --2- 172.21.15.107:0/329987430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6b4152040 0x7ff6b40c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b259c640 1 -- 172.21.15.107:0/329987430 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333432975 0 0) 0x7ff6b4155680 con 0x7ff6b40c8100 2023-10-25T00:09:03.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b259c640 1 -- 172.21.15.107:0/329987430 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6a0003530 con 0x7ff6b40c8100 2023-10-25T00:09:03.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b259c640 1 -- 172.21.15.107:0/329987430 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052027490 0 0) 0x7ff6b4154a70 con 0x7ff6b406a4b0 2023-10-25T00:09:03.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b2d9d640 1 -- 172.21.15.107:0/329987430 >> v1:172.21.15.110:6789/0 conn(0x7ff6b406a4b0 legacy=0x7ff6b40ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b2d9d640 1 -- 172.21.15.107:0/329987430 >> v1:172.21.15.107:6789/0 conn(0x7ff6b40c8100 legacy=0x7ff6b40abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.626+0000 7ff6b2d9d640 1 -- 172.21.15.107:0/329987430 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b4155030 con 0x7ff6b4152040 2023-10-25T00:09:03.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.630+0000 7ff6b2d9d640 1 --2- 172.21.15.107:0/329987430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6b4152040 0x7ff6b40c4b80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40097f0 tx=0x7ff6a401c1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=272ea738330ea9ad server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:03.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.630+0000 7ff6b259c640 1 -- 172.21.15.107:0/329987430 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c001680 con 0x7ff6b4152040 2023-10-25T00:09:03.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.630+0000 7ff6b259c640 1 -- 172.21.15.107:0/329987430 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a40a7050 con 0x7ff6b4152040 2023-10-25T00:09:03.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.630+0000 7ff6b259c640 1 -- 172.21.15.107:0/329987430 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a40ab600 con 0x7ff6b4152040 2023-10-25T00:09:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.630+0000 7ff6b9176640 1 -- 172.21.15.107:0/329987430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6b4152040 msgr2=0x7ff6b40c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.630+0000 7ff6b9176640 1 --2- 172.21.15.107:0/329987430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6b4152040 0x7ff6b40c4b80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40097f0 tx=0x7ff6a401c1e0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.630+0000 7ff6b9176640 1 -- 172.21.15.107:0/329987430 shutdown_connections 2023-10-25T00:09:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.630+0000 7ff6b9176640 1 --2- 172.21.15.107:0/329987430 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6b4152040 0x7ff6b40c4b80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.630+0000 7ff6b9176640 1 -- 172.21.15.107:0/329987430 >> 172.21.15.107:0/329987430 conn(0x7ff6b40ca200 msgr2=0x7ff6b40ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.630+0000 7ff6b9176640 1 -- 172.21.15.107:0/329987430 shutdown_connections 2023-10-25T00:09:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.630+0000 7ff6b9176640 1 -- 172.21.15.107:0/329987430 wait complete. 2023-10-25T00:09:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b9176640 1 Processor -- start 2023-10-25T00:09:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b9176640 1 -- start start 2023-10-25T00:09:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b9176640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6b406a4b0 0x7ff6b40c2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b9176640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6b40c8100 0x7ff6b4146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b9176640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6b4152040 0x7ff6b4146c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b9176640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b414a6b0 con 0x7ff6b40c8100 2023-10-25T00:09:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b9176640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b40aaf00 con 0x7ff6b4152040 2023-10-25T00:09:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b9176640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b40c3c50 con 0x7ff6b406a4b0 2023-10-25T00:09:03.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b2d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6b406a4b0 0x7ff6b40c2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b359e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6b4152040 0x7ff6b4146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b359e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6b4152040 0x7ff6b4146c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:55708/0 (socket says 172.21.15.107:55708) 2023-10-25T00:09:03.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b359e640 1 -- 172.21.15.107:0/3885673573 learned_addr learned my addr 172.21.15.107:0/3885673573 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:03.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b359e640 1 -- 172.21.15.107:0/3885673573 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6b406a4b0 msgr2=0x7ff6b40c2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b359e640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6b406a4b0 0x7ff6b40c2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b359e640 1 -- 172.21.15.107:0/3885673573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6b40c8100 msgr2=0x7ff6b4146700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:09:03.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b359e640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6b40c8100 0x7ff6b4146700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b359e640 1 -- 172.21.15.107:0/3885673573 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a4009420 con 0x7ff6b4152040 2023-10-25T00:09:03.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b359e640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6b4152040 0x7ff6b4146c40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00c7b0 tx=0x7ff6ac00cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:03.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6ab7fe640 1 -- 172.21.15.107:0/3885673573 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac003db0 con 0x7ff6b4152040 2023-10-25T00:09:03.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6ab7fe640 1 -- 172.21.15.107:0/3885673573 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ac00ce60 con 0x7ff6b4152040 2023-10-25T00:09:03.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b9176640 1 -- 172.21.15.107:0/3885673573 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b40a9c50 con 0x7ff6b4152040 2023-10-25T00:09:03.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6ab7fe640 1 -- 172.21.15.107:0/3885673573 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac0052b0 con 0x7ff6b4152040 2023-10-25T00:09:03.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.634+0000 7ff6b9176640 1 -- 172.21.15.107:0/3885673573 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b40ab150 con 0x7ff6b4152040 2023-10-25T00:09:03.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.638+0000 7ff6b9176640 1 -- 172.21.15.107:0/3885673573 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b414a860 con 0x7ff6b4152040 2023-10-25T00:09:03.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.638+0000 7ff6ab7fe640 1 -- 172.21.15.107:0/3885673573 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff6ac005410 con 0x7ff6b4152040 2023-10-25T00:09:03.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.638+0000 7ff6ab7fe640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff694065e60 0x7ff694068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:03.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.638+0000 7ff6ab7fe640 1 -- 172.21.15.107:0/3885673573 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7ff6ac084b70 con 0x7ff6b4152040 2023-10-25T00:09:03.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.638+0000 7ff6b2d9d640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff694065e60 0x7ff694068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:03.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.638+0000 7ff6b2d9d640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff694065e60 0x7ff694068320 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40ad000 tx=0x7ff6a40a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:03.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.658+0000 7ff6ab7fe640 1 -- 172.21.15.107:0/3885673573 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff6ac051560 con 0x7ff6b4152040 2023-10-25T00:09:03.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.838+0000 7f22bdf62640 1 -- 172.21.15.107:0/1231213040 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22b80746e0 con 0x7f22b80c70e0 2023-10-25T00:09:03.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22b4ff9640 1 -- 172.21.15.107:0/1231213040 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5019 (secure 0 0 0) 0x7f22a00b0090 con 0x7f22b80c70e0 2023-10-25T00:09:03.842 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:03.842 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:03.843 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 144 2023-10-25T00:09:03.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 -- 172.21.15.107:0/1231213040 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2298065e60 msgr2=0x7f2298068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2298065e60 0x7f2298068320 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f22a8009a90 tx=0x7f22a8008040 comp rx=0 tx=0).stop 2023-10-25T00:09:03.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 -- 172.21.15.107:0/1231213040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22b80c70e0 msgr2=0x7f22b80bcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22b80c70e0 0x7f22b80bcfb0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f22a001aa90 tx=0x7f22a0019180 comp rx=0 tx=0).stop 2023-10-25T00:09:03.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 -- 172.21.15.107:0/1231213040 shutdown_connections 2023-10-25T00:09:03.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2298065e60 0x7f2298068320 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f22b8156770 0x7f22b80a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22b80c70e0 0x7f22b80bcfb0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 --2- 172.21.15.107:0/1231213040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f22b80c2130 0x7f22b815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 -- 172.21.15.107:0/1231213040 >> 172.21.15.107:0/1231213040 conn(0x7f22b80c65c0 msgr2=0x7f22b80c4e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:03.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 -- 172.21.15.107:0/1231213040 shutdown_connections 2023-10-25T00:09:03.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.842+0000 7f22bdf62640 1 -- 172.21.15.107:0/1231213040 wait complete. 2023-10-25T00:09:03.864 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:09:03.873 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:09:03.870+0000 7ff712fba640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:09:03.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.906+0000 7ff6b9176640 1 -- 172.21.15.107:0/3885673573 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6b414a860 con 0x7ff6b4152040 2023-10-25T00:09:03.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.906+0000 7ff6ab7fe640 1 -- 172.21.15.107:0/3885673573 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5019 (secure 0 0 0) 0x7ff6b414a860 con 0x7ff6b4152040 2023-10-25T00:09:03.908 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:03.908 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:03.909 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 144 2023-10-25T00:09:03.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 -- 172.21.15.107:0/3885673573 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff694065e60 msgr2=0x7ff694068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff694065e60 0x7ff694068320 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff6a40ad000 tx=0x7ff6a40a6000 comp rx=0 tx=0).stop 2023-10-25T00:09:03.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 -- 172.21.15.107:0/3885673573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6b4152040 msgr2=0x7ff6b4146c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:03.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6b4152040 0x7ff6b4146c40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00c7b0 tx=0x7ff6ac00cc70 comp rx=0 tx=0).stop 2023-10-25T00:09:03.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 -- 172.21.15.107:0/3885673573 shutdown_connections 2023-10-25T00:09:03.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6b4152040 0x7ff6b4146c40 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6b40c8100 0x7ff6b4146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff694065e60 0x7ff694068320 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 --2- 172.21.15.107:0/3885673573 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6b406a4b0 0x7ff6b40c2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:03.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 -- 172.21.15.107:0/3885673573 >> 172.21.15.107:0/3885673573 conn(0x7ff6b40ca200 msgr2=0x7ff6b4149350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:03.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 -- 172.21.15.107:0/3885673573 shutdown_connections 2023-10-25T00:09:03.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:03.910+0000 7ff6b9176640 1 -- 172.21.15.107:0/3885673573 wait complete. 2023-10-25T00:09:03.927 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6394 2023-10-25T00:09:03.928 INFO:tasks.mds_thrash.fs.[a]:waiting for 2 secs before reviving mds.d 2023-10-25T00:09:05.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:05.929 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2023-10-25T00:09:05.929 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-25T00:09:05.929 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-25T00:09:05.932 INFO:tasks.ceph.mds.d:Started 2023-10-25T00:09:05.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:06.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d9ad3640 1 Processor -- start 2023-10-25T00:09:06.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d9ad3640 1 -- start start 2023-10-25T00:09:06.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d9ad3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50d4070b00 0x7f50d40c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:06.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d9ad3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50d406df30 0x7f50d406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:06.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d9ad3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d40c22f0 con 0x7f50d414eb70 2023-10-25T00:09:06.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d9ad3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d4071700 con 0x7f50d4070b00 2023-10-25T00:09:06.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d9ad3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d40c25d0 con 0x7f50d406df30 2023-10-25T00:09:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50d4070b00 0x7f50d40c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50d4070b00 0x7f50d40c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35400/0 (socket says 172.21.15.107:35400) 2023-10-25T00:09:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d2ffd640 1 -- 172.21.15.107:0/3772022896 learned_addr learned my addr 172.21.15.107:0/3772022896 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d3fff640 1 --2- 172.21.15.107:0/3772022896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50d406df30 0x7f50d406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:06.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d2ffd640 1 -- 172.21.15.107:0/3772022896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50d406df30 msgr2=0x7f50d406e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:06.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d2ffd640 1 --2- 172.21.15.107:0/3772022896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50d406df30 0x7f50d406e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d2ffd640 1 -- 172.21.15.107:0/3772022896 >> v1:172.21.15.107:6789/0 conn(0x7f50d414eb70 legacy=0x7f50d41514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d2ffd640 1 -- 172.21.15.107:0/3772022896 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d40c2be0 con 0x7f50d4070b00 2023-10-25T00:09:06.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.006+0000 7f50d27fc640 1 -- 172.21.15.107:0/3772022896 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79290414 0 0) 0x7f50d40c22f0 con 0x7f50d414eb70 2023-10-25T00:09:06.011 INFO:tasks.ceph.mds.d.smithi110.stdout:starting mds.d at 2023-10-25T00:09:06.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d2ffd640 1 --2- 172.21.15.107:0/3772022896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50d4070b00 0x7f50d40c33e0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f50c800bb10 tx=0x7f50c801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6d462d768054fc server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:06.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d27fc640 1 -- 172.21.15.107:0/3772022896 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80a5a00 con 0x7f50d4070b00 2023-10-25T00:09:06.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d27fc640 1 -- 172.21.15.107:0/3772022896 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c80a5b60 con 0x7f50d4070b00 2023-10-25T00:09:06.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d27fc640 1 -- 172.21.15.107:0/3772022896 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80ad440 con 0x7f50d4070b00 2023-10-25T00:09:06.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d9ad3640 1 -- 172.21.15.107:0/3772022896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50d4070b00 msgr2=0x7f50d40c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d9ad3640 1 --2- 172.21.15.107:0/3772022896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50d4070b00 0x7f50d40c33e0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f50c800bb10 tx=0x7f50c801ca10 comp rx=0 tx=0).stop 2023-10-25T00:09:06.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d9ad3640 1 -- 172.21.15.107:0/3772022896 shutdown_connections 2023-10-25T00:09:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d9ad3640 1 --2- 172.21.15.107:0/3772022896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50d406df30 0x7f50d406e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d9ad3640 1 --2- 172.21.15.107:0/3772022896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50d4070b00 0x7f50d40c33e0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d9ad3640 1 -- 172.21.15.107:0/3772022896 >> 172.21.15.107:0/3772022896 conn(0x7f50d40c39a0 msgr2=0x7f50d40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d9ad3640 1 -- 172.21.15.107:0/3772022896 shutdown_connections 2023-10-25T00:09:06.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d9ad3640 1 -- 172.21.15.107:0/3772022896 wait complete. 2023-10-25T00:09:06.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d9ad3640 1 Processor -- start 2023-10-25T00:09:06.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.010+0000 7f50d9ad3640 1 -- start start 2023-10-25T00:09:06.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d9ad3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50d406df30 0x7f50d41474f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:06.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d9ad3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50d4070b00 0x7f50d41458d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:06.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d9ad3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50d414eb70 0x7f50d4144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:06.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d9ad3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d40a9450 con 0x7f50d414eb70 2023-10-25T00:09:06.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d9ad3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d4147ad0 con 0x7f50d406df30 2023-10-25T00:09:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d9ad3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d4147db0 con 0x7f50d4070b00 2023-10-25T00:09:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50d414eb70 0x7f50d4144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d37fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50d406df30 0x7f50d41474f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50d414eb70 0x7f50d4144930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35414/0 (socket says 172.21.15.107:35414) 2023-10-25T00:09:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50d4070b00 0x7f50d41458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d3fff640 1 -- 172.21.15.107:0/746172765 learned_addr learned my addr 172.21.15.107:0/746172765 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d3fff640 1 -- 172.21.15.107:0/746172765 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50d4070b00 msgr2=0x7f50d41458d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d3fff640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50d4070b00 0x7f50d41458d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d3fff640 1 -- 172.21.15.107:0/746172765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50d406df30 msgr2=0x7f50d41474f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d3fff640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50d406df30 0x7f50d41474f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d3fff640 1 -- 172.21.15.107:0/746172765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c800b770 con 0x7f50d414eb70 2023-10-25T00:09:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d2ffd640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50d4070b00 0x7f50d41458d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:09:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d3fff640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50d414eb70 0x7f50d4144930 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f50cc012320 tx=0x7f50cc0127e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50bbfff640 1 -- 172.21.15.107:0/746172765 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc014040 con 0x7f50d414eb70 2023-10-25T00:09:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50bbfff640 1 -- 172.21.15.107:0/746172765 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50cc018810 con 0x7f50d414eb70 2023-10-25T00:09:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50bbfff640 1 -- 172.21.15.107:0/746172765 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc009c40 con 0x7f50d414eb70 2023-10-25T00:09:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d9ad3640 1 -- 172.21.15.107:0/746172765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d4148090 con 0x7f50d414eb70 2023-10-25T00:09:06.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d9ad3640 1 -- 172.21.15.107:0/746172765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d4145e10 con 0x7f50d414eb70 2023-10-25T00:09:06.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.014+0000 7f50d9ad3640 1 -- 172.21.15.107:0/746172765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d40a95c0 con 0x7f50d414eb70 2023-10-25T00:09:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.018+0000 7f50bbfff640 1 -- 172.21.15.107:0/746172765 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f50cc018bc0 con 0x7f50d414eb70 2023-10-25T00:09:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.018+0000 7f50bbfff640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50c0065e90 0x7f50c0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.018+0000 7f50bbfff640 1 -- 172.21.15.107:0/746172765 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7f50cc089020 con 0x7f50d414eb70 2023-10-25T00:09:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.018+0000 7f50d37fe640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50c0065e90 0x7f50c0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:06.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.018+0000 7f50d37fe640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50c0065e90 0x7f50c0068350 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f50c4007fd0 tx=0x7f50c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:06.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.022+0000 7f50bbfff640 1 -- 172.21.15.107:0/746172765 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f50cc055a30 con 0x7f50d414eb70 2023-10-25T00:09:06.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.062+0000 7f1534fa3640 1 Processor -- start 2023-10-25T00:09:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.062+0000 7f1534fa3640 1 -- start start 2023-10-25T00:09:06.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f1534fa3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15300aa140 0x7f15300aa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:06.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f1534fa3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1530066000 con 0x7f15300c26f0 2023-10-25T00:09:06.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f1534fa3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15300aaa60 con 0x7f15300ac020 2023-10-25T00:09:06.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f1534fa3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15300c0f40 con 0x7f15300aa140 2023-10-25T00:09:06.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f152ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f15300c26f0 0x7f153006c490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56592/0 (socket says 172.21.15.107:56592) 2023-10-25T00:09:06.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f152ed9d640 1 -- 172.21.15.107:0/2558463807 learned_addr learned my addr 172.21.15.107:0/2558463807 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:09:06.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f152e59c640 1 --2- 172.21.15.107:0/2558463807 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15300aa140 0x7f15300aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:06.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f152dd9b640 1 -- 172.21.15.107:0/2558463807 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549368866 0 0) 0x7f1530066000 con 0x7f15300c26f0 2023-10-25T00:09:06.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f152dd9b640 1 -- 172.21.15.107:0/2558463807 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c003530 con 0x7f15300c26f0 2023-10-25T00:09:06.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f152dd9b640 1 -- 172.21.15.107:0/2558463807 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743172049 0 0) 0x7f15300aaa60 con 0x7f15300ac020 2023-10-25T00:09:06.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f152dd9b640 1 -- 172.21.15.107:0/2558463807 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1530066000 con 0x7f15300ac020 2023-10-25T00:09:06.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f152e59c640 1 -- 172.21.15.107:0/2558463807 >> v1:172.21.15.110:6789/0 conn(0x7f15300ac020 legacy=0x7f15300c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f152e59c640 1 -- 172.21.15.107:0/2558463807 >> v1:172.21.15.107:6789/0 conn(0x7f15300c26f0 legacy=0x7f153006c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.066+0000 7f152e59c640 1 -- 172.21.15.107:0/2558463807 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15300c1220 con 0x7f15300aa140 2023-10-25T00:09:06.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f152e59c640 1 --2- 172.21.15.107:0/2558463807 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15300aa140 0x7f15300aa520 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f15200097f0 tx=0x7f152001c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f8a5894c9533242 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:06.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f152dd9b640 1 -- 172.21.15.107:0/2558463807 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15200aa5f0 con 0x7f15300aa140 2023-10-25T00:09:06.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f152dd9b640 1 -- 172.21.15.107:0/2558463807 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15200aa750 con 0x7f15300aa140 2023-10-25T00:09:06.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f152dd9b640 1 -- 172.21.15.107:0/2558463807 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15200aaac0 con 0x7f15300aa140 2023-10-25T00:09:06.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f1534fa3640 1 -- 172.21.15.107:0/2558463807 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15300aa140 msgr2=0x7f15300aa520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f1534fa3640 1 --2- 172.21.15.107:0/2558463807 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15300aa140 0x7f15300aa520 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f15200097f0 tx=0x7f152001c490 comp rx=0 tx=0).stop 2023-10-25T00:09:06.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f1534fa3640 1 -- 172.21.15.107:0/2558463807 shutdown_connections 2023-10-25T00:09:06.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f1534fa3640 1 --2- 172.21.15.107:0/2558463807 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15300aa140 0x7f15300aa520 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f1534fa3640 1 -- 172.21.15.107:0/2558463807 >> 172.21.15.107:0/2558463807 conn(0x7f15300c65c0 msgr2=0x7f15300c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:06.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f1534fa3640 1 -- 172.21.15.107:0/2558463807 shutdown_connections 2023-10-25T00:09:06.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f1534fa3640 1 -- 172.21.15.107:0/2558463807 wait complete. 2023-10-25T00:09:06.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f1534fa3640 1 Processor -- start 2023-10-25T00:09:06.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.070+0000 7f1534fa3640 1 -- start start 2023-10-25T00:09:06.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f1534fa3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15300aa140 0x7f153015c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:06.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f1534fa3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15300ac020 0x7f15301ec6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:06.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f1534fa3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15300c26f0 0x7f15301ecbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:06.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f1534fa3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15301521c0 con 0x7f15300aa140 2023-10-25T00:09:06.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f1534fa3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15301ed130 con 0x7f15300c26f0 2023-10-25T00:09:06.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f1534fa3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15301f0630 con 0x7f15300ac020 2023-10-25T00:09:06.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f152e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15300aa140 0x7f153015c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:06.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f152e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15300aa140 0x7f153015c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35430/0 (socket says 172.21.15.107:35430) 2023-10-25T00:09:06.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f152e59c640 1 -- 172.21.15.107:0/1886535339 learned_addr learned my addr 172.21.15.107:0/1886535339 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:06.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f1525d9b640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15300ac020 0x7f15301ec6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:06.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f152ed9d640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15300c26f0 0x7f15301ecbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:06.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f152e59c640 1 -- 172.21.15.107:0/1886535339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15300ac020 msgr2=0x7f15301ec6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f152e59c640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15300ac020 0x7f15301ec6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f152e59c640 1 -- 172.21.15.107:0/1886535339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15300c26f0 msgr2=0x7f15301ecbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f152e59c640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15300c26f0 0x7f15301ecbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f152e59c640 1 -- 172.21.15.107:0/1886535339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1520009420 con 0x7f15300aa140 2023-10-25T00:09:06.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f152ed9d640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15300c26f0 0x7f15301ecbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:09:06.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f152e59c640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15300aa140 0x7f153015c680 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f15200ac040 tx=0x7f1520019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:06.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f15277fe640 1 -- 172.21.15.107:0/1886535339 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520004450 con 0x7f15300aa140 2023-10-25T00:09:06.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f15277fe640 1 -- 172.21.15.107:0/1886535339 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15200aa8c0 con 0x7f15300aa140 2023-10-25T00:09:06.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f1534fa3640 1 -- 172.21.15.107:0/1886535339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15301f0b60 con 0x7f15300aa140 2023-10-25T00:09:06.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f15277fe640 1 -- 172.21.15.107:0/1886535339 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15200aac70 con 0x7f15300aa140 2023-10-25T00:09:06.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f1534fa3640 1 -- 172.21.15.107:0/1886535339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1530152f40 con 0x7f15300aa140 2023-10-25T00:09:06.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f1534fa3640 1 -- 172.21.15.107:0/1886535339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1530152cf0 con 0x7f15300aa140 2023-10-25T00:09:06.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.074+0000 7f15277fe640 1 -- 172.21.15.107:0/1886535339 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1520003620 con 0x7f15300aa140 2023-10-25T00:09:06.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.078+0000 7f15277fe640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e60 0x7f1510068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:06.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.078+0000 7f15277fe640 1 -- 172.21.15.107:0/1886535339 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7f15200a7070 con 0x7f15300aa140 2023-10-25T00:09:06.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.082+0000 7f1525d9b640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e60 0x7f1510068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:06.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.082+0000 7f1525d9b640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e60 0x7f1510068320 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f15300c3980 tx=0x7f1518009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:06.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.086+0000 7f15277fe640 1 -- 172.21.15.107:0/1886535339 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f15200eae30 con 0x7f15300aa140 2023-10-25T00:09:06.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.258+0000 7f50d9ad3640 1 -- 172.21.15.107:0/746172765 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50d40746e0 con 0x7f50d414eb70 2023-10-25T00:09:06.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.258+0000 7f50bbfff640 1 -- 172.21.15.107:0/746172765 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5019 (secure 0 0 0) 0x7f50cc01c020 con 0x7f50d414eb70 2023-10-25T00:09:06.262 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:06.262 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:06.263 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 144 2023-10-25T00:09:06.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.262+0000 7f50d9ad3640 1 -- 172.21.15.107:0/746172765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50c0065e90 msgr2=0x7f50c0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.262+0000 7f50d9ad3640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50c0065e90 0x7f50c0068350 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f50c4007fd0 tx=0x7f50c4009040 comp rx=0 tx=0).stop 2023-10-25T00:09:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.262+0000 7f50d9ad3640 1 -- 172.21.15.107:0/746172765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50d414eb70 msgr2=0x7f50d4144930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.262+0000 7f50d9ad3640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50d414eb70 0x7f50d4144930 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f50cc012320 tx=0x7f50cc0127e0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.266+0000 7f50d9ad3640 1 -- 172.21.15.107:0/746172765 shutdown_connections 2023-10-25T00:09:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.266+0000 7f50d9ad3640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50d414eb70 0x7f50d4144930 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.266+0000 7f50d9ad3640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50c0065e90 0x7f50c0068350 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.266+0000 7f50d9ad3640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50d4070b00 0x7f50d41458d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.266+0000 7f50d9ad3640 1 --2- 172.21.15.107:0/746172765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50d406df30 0x7f50d41474f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.266+0000 7f50d9ad3640 1 -- 172.21.15.107:0/746172765 >> 172.21.15.107:0/746172765 conn(0x7f50d40c39a0 msgr2=0x7f50d40c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.266+0000 7f50d9ad3640 1 -- 172.21.15.107:0/746172765 shutdown_connections 2023-10-25T00:09:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.266+0000 7f50d9ad3640 1 -- 172.21.15.107:0/746172765 wait complete. 2023-10-25T00:09:06.284 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:09:06.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.326+0000 7f1534fa3640 1 -- 172.21.15.107:0/1886535339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15300746e0 con 0x7f15300aa140 2023-10-25T00:09:06.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.326+0000 7f15277fe640 1 -- 172.21.15.107:0/1886535339 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5019 (secure 0 0 0) 0x7f15200ee450 con 0x7f15300aa140 2023-10-25T00:09:06.328 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:06.329 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:06.330 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 144 2023-10-25T00:09:06.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.326+0000 7f1534fa3640 1 -- 172.21.15.107:0/1886535339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e60 msgr2=0x7f1510068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.326+0000 7f1534fa3640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e60 0x7f1510068320 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f15300c3980 tx=0x7f1518009040 comp rx=0 tx=0).stop 2023-10-25T00:09:06.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.326+0000 7f1534fa3640 1 -- 172.21.15.107:0/1886535339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15300aa140 msgr2=0x7f153015c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:06.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.326+0000 7f1534fa3640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15300aa140 0x7f153015c680 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f15200ac040 tx=0x7f1520019230 comp rx=0 tx=0).stop 2023-10-25T00:09:06.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.330+0000 7f1534fa3640 1 -- 172.21.15.107:0/1886535339 shutdown_connections 2023-10-25T00:09:06.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.330+0000 7f1534fa3640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f15300c26f0 0x7f15301ecbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.330+0000 7f1534fa3640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1510065e60 0x7f1510068320 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.330+0000 7f1534fa3640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f15300ac020 0x7f15301ec6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.330+0000 7f1534fa3640 1 --2- 172.21.15.107:0/1886535339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15300aa140 0x7f153015c680 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:06.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.330+0000 7f1534fa3640 1 -- 172.21.15.107:0/1886535339 >> 172.21.15.107:0/1886535339 conn(0x7f15300c65c0 msgr2=0x7f15300c2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:06.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.330+0000 7f1534fa3640 1 -- 172.21.15.107:0/1886535339 shutdown_connections 2023-10-25T00:09:06.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:06.330+0000 7f1534fa3640 1 -- 172.21.15.107:0/1886535339 wait complete. 2023-10-25T00:09:06.348 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-25T00:09:08.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:08.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.406+0000 7fa2a0bf0640 1 Processor -- start 2023-10-25T00:09:08.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.406+0000 7fa2a0bf0640 1 -- start start 2023-10-25T00:09:08.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa2a0bf0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa29c0c70e0 0x7fa29c0bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:08.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa2a0bf0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa29c0c2130 0x7fa29c0be4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa2a0bf0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29c0c1ca0 con 0x7fa29c156770 2023-10-25T00:09:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa2a0bf0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29c0c27e0 con 0x7fa29c0c70e0 2023-10-25T00:09:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa2a0bf0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0c10c0 con 0x7fa29c0c2130 2023-10-25T00:09:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa299d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa29c0c70e0 0x7fa29c0bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa299d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa29c0c70e0 0x7fa29c0bdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35440/0 (socket says 172.21.15.107:35440) 2023-10-25T00:09:08.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa299d9b640 1 -- 172.21.15.107:0/4155532153 learned_addr learned my addr 172.21.15.107:0/4155532153 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:08.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa29ad9d640 1 --2- 172.21.15.107:0/4155532153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa29c0c2130 0x7fa29c0be4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:08.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa299d9b640 1 -- 172.21.15.107:0/4155532153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa29c0c2130 msgr2=0x7fa29c0be4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:08.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa299d9b640 1 --2- 172.21.15.107:0/4155532153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa29c0c2130 0x7fa29c0be4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa299d9b640 1 -- 172.21.15.107:0/4155532153 >> v1:172.21.15.110:6789/0 conn(0x7fa29c156770 legacy=0x7fa29c0ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa299d9b640 1 -- 172.21.15.107:0/4155532153 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c0c13a0 con 0x7fa29c0c70e0 2023-10-25T00:09:08.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa299d9b640 1 --2- 172.21.15.107:0/4155532153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa29c0c70e0 0x7fa29c0bdfa0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fa290009870 tx=0x7fa29001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9715b95fcd946b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:08.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa29959a640 1 -- 172.21.15.107:0/4155532153 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900ab600 con 0x7fa29c0c70e0 2023-10-25T00:09:08.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa29959a640 1 -- 172.21.15.107:0/4155532153 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2900a7050 con 0x7fa29c0c70e0 2023-10-25T00:09:08.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.410+0000 7fa29959a640 1 -- 172.21.15.107:0/4155532153 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900ab980 con 0x7fa29c0c70e0 2023-10-25T00:09:08.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/4155532153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa29c0c70e0 msgr2=0x7fa29c0bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 --2- 172.21.15.107:0/4155532153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa29c0c70e0 0x7fa29c0bdfa0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fa290009870 tx=0x7fa29001c420 comp rx=0 tx=0).stop 2023-10-25T00:09:08.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/4155532153 shutdown_connections 2023-10-25T00:09:08.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 --2- 172.21.15.107:0/4155532153 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa29c0c2130 0x7fa29c0be4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 --2- 172.21.15.107:0/4155532153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa29c0c70e0 0x7fa29c0bdfa0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/4155532153 >> 172.21.15.107:0/4155532153 conn(0x7fa29c0c65c0 msgr2=0x7fa29c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:08.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/4155532153 shutdown_connections 2023-10-25T00:09:08.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/4155532153 wait complete. 2023-10-25T00:09:08.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 Processor -- start 2023-10-25T00:09:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 -- start start 2023-10-25T00:09:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa29c0c2130 0x7fa29c15b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa29c0c70e0 0x7fa29c1ecc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa29c156770 0x7fa29c1ed160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:08.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0aa3d0 con 0x7fa29c0c70e0 2023-10-25T00:09:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0c4d40 con 0x7fa29c0c2130 2023-10-25T00:09:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.414+0000 7fa2a0bf0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0aa0f0 con 0x7fa29c156770 2023-10-25T00:09:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa299d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa29c0c70e0 0x7fa29c1ecc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:08.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa299d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa29c0c70e0 0x7fa29c1ecc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35442/0 (socket says 172.21.15.107:35442) 2023-10-25T00:09:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa299d9b640 1 -- 172.21.15.107:0/2447584078 learned_addr learned my addr 172.21.15.107:0/2447584078 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa29a59c640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa29c0c2130 0x7fa29c15b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa299d9b640 1 -- 172.21.15.107:0/2447584078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa29c156770 msgr2=0x7fa29c1ed160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa29ad9d640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa29c156770 0x7fa29c1ed160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:08.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa299d9b640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa29c156770 0x7fa29c1ed160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa299d9b640 1 -- 172.21.15.107:0/2447584078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa29c0c2130 msgr2=0x7fa29c15b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa299d9b640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa29c0c2130 0x7fa29c15b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa299d9b640 1 -- 172.21.15.107:0/2447584078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2900094d0 con 0x7fa29c0c70e0 2023-10-25T00:09:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa29a59c640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa29c0c2130 0x7fa29c15b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:09:08.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa29ad9d640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa29c156770 0x7fa29c1ed160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:09:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa299d9b640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa29c0c70e0 0x7fa29c1ecc20 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fa2900ae040 tx=0x7fa2900191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa28affd640 1 -- 172.21.15.107:0/2447584078 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900b4420 con 0x7fa29c0c70e0 2023-10-25T00:09:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa28affd640 1 -- 172.21.15.107:0/2447584078 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2900b4580 con 0x7fa29c0c70e0 2023-10-25T00:09:08.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/2447584078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29c153d30 con 0x7fa29c0c70e0 2023-10-25T00:09:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa28affd640 1 -- 172.21.15.107:0/2447584078 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900b4930 con 0x7fa29c0c70e0 2023-10-25T00:09:08.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/2447584078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29c15bcb0 con 0x7fa29c0c70e0 2023-10-25T00:09:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/2447584078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29c0c4f60 con 0x7fa29c0c70e0 2023-10-25T00:09:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa28affd640 1 -- 172.21.15.107:0/2447584078 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa2900b2070 con 0x7fa29c0c70e0 2023-10-25T00:09:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa28affd640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa27c065e60 0x7fa27c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa28affd640 1 -- 172.21.15.107:0/2447584078 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7fa29011e170 con 0x7fa29c0c70e0 2023-10-25T00:09:08.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa29a59c640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa27c065e60 0x7fa27c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:08.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.418+0000 7fa29a59c640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa27c065e60 0x7fa27c068320 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa28c000f70 tx=0x7fa28c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.426+0000 7fa28affd640 1 -- 172.21.15.107:0/2447584078 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa29c0c4f60 con 0x7fa29c0c70e0 2023-10-25T00:09:08.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b25f31640 1 Processor -- start 2023-10-25T00:09:08.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b25f31640 1 -- start start 2023-10-25T00:09:08.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b25f31640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6b200c70e0 0x7f6b200bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:08.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b25f31640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b200c1ca0 con 0x7f6b20156770 2023-10-25T00:09:08.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b25f31640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b200c10c0 con 0x7f6b200c2130 2023-10-25T00:09:08.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b25f31640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b200c27e0 con 0x7f6b200c70e0 2023-10-25T00:09:08.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b1effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6b200c70e0 0x7f6b200bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:08.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b1f7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6b20156770 0x7f6b200ab7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56596/0 (socket says 172.21.15.107:56596) 2023-10-25T00:09:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b1f7fe640 1 -- 172.21.15.107:0/1510099421 learned_addr learned my addr 172.21.15.107:0/1510099421 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:09:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b1effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6b200c70e0 0x7f6b200bdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35452/0 (socket says 172.21.15.107:35452) 2023-10-25T00:09:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b1effd640 1 -- 172.21.15.107:0/1510099421 >> v1:172.21.15.110:6789/0 conn(0x7f6b200c2130 legacy=0x7f6b2015d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b1effd640 1 -- 172.21.15.107:0/1510099421 >> v1:172.21.15.107:6789/0 conn(0x7f6b20156770 legacy=0x7f6b200ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b1e7fc640 1 -- 172.21.15.107:0/1510099421 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513902304 0 0) 0x7f6b200c1ca0 con 0x7f6b20156770 2023-10-25T00:09:08.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.466+0000 7f6b1effd640 1 -- 172.21.15.107:0/1510099421 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b200c13a0 con 0x7f6b200c70e0 2023-10-25T00:09:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.470+0000 7f6b1effd640 1 --2- 172.21.15.107:0/1510099421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b200c70e0 0x7f6b200bdfa0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f6b140097c0 tx=0x7f6b1401c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530412c814cb2190 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.470+0000 7f6b1e7fc640 1 -- 172.21.15.107:0/1510099421 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b140ab600 con 0x7f6b200c70e0 2023-10-25T00:09:08.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.470+0000 7f6b1e7fc640 1 -- 172.21.15.107:0/1510099421 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b140a7050 con 0x7f6b200c70e0 2023-10-25T00:09:08.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.470+0000 7f6b1e7fc640 1 -- 172.21.15.107:0/1510099421 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b140ab980 con 0x7f6b200c70e0 2023-10-25T00:09:08.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.470+0000 7f6b25f31640 1 -- 172.21.15.107:0/1510099421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b200c70e0 msgr2=0x7f6b200bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.470+0000 7f6b25f31640 1 --2- 172.21.15.107:0/1510099421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b200c70e0 0x7f6b200bdfa0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f6b140097c0 tx=0x7f6b1401c2f0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.470+0000 7f6b25f31640 1 -- 172.21.15.107:0/1510099421 shutdown_connections 2023-10-25T00:09:08.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.470+0000 7f6b25f31640 1 --2- 172.21.15.107:0/1510099421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b200c70e0 0x7f6b200bdfa0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.470+0000 7f6b25f31640 1 -- 172.21.15.107:0/1510099421 >> 172.21.15.107:0/1510099421 conn(0x7f6b200c65c0 msgr2=0x7f6b200c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:08.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.470+0000 7f6b25f31640 1 -- 172.21.15.107:0/1510099421 shutdown_connections 2023-10-25T00:09:08.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.470+0000 7f6b25f31640 1 -- 172.21.15.107:0/1510099421 wait complete. 2023-10-25T00:09:08.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b25f31640 1 Processor -- start 2023-10-25T00:09:08.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b25f31640 1 -- start start 2023-10-25T00:09:08.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b25f31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b200c2130 0x7f6b2014d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:08.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b25f31640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b20156770 0x7f6b2014d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:08.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b25f31640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b200ba080 0x7f6b201ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:08.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b25f31640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2006ff70 con 0x7f6b200c2130 2023-10-25T00:09:08.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b25f31640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b20071fc0 con 0x7f6b200ba080 2023-10-25T00:09:08.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b25f31640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b200722a0 con 0x7f6b20156770 2023-10-25T00:09:08.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b200c2130 0x7f6b2014d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:08.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b200c2130 0x7f6b2014d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35460/0 (socket says 172.21.15.107:35460) 2023-10-25T00:09:08.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b1f7fe640 1 -- 172.21.15.107:0/37809023 learned_addr learned my addr 172.21.15.107:0/37809023 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:08.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b1effd640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b20156770 0x7f6b2014d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:08.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b1ffff640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b200ba080 0x7f6b201ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:08.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b1f7fe640 1 -- 172.21.15.107:0/37809023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b20156770 msgr2=0x7f6b2014d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b1f7fe640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b20156770 0x7f6b2014d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b1f7fe640 1 -- 172.21.15.107:0/37809023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b200ba080 msgr2=0x7f6b201ee7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b1f7fe640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b200ba080 0x7f6b201ee7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b1f7fe640 1 -- 172.21.15.107:0/37809023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b14009420 con 0x7f6b200c2130 2023-10-25T00:09:08.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b1f7fe640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b200c2130 0x7f6b2014d030 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000c480 tx=0x7f6b1000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:08.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b07fff640 1 -- 172.21.15.107:0/37809023 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b100074e0 con 0x7f6b200c2130 2023-10-25T00:09:08.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b07fff640 1 -- 172.21.15.107:0/37809023 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b10007660 con 0x7f6b200c2130 2023-10-25T00:09:08.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b07fff640 1 -- 172.21.15.107:0/37809023 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b10004a30 con 0x7f6b200c2130 2023-10-25T00:09:08.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b25f31640 1 -- 172.21.15.107:0/37809023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2006fc00 con 0x7f6b200c2130 2023-10-25T00:09:08.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.474+0000 7f6b25f31640 1 -- 172.21.15.107:0/37809023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b201eeec0 con 0x7f6b200c2130 2023-10-25T00:09:08.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.478+0000 7f6b05ffb640 1 -- 172.21.15.107:0/37809023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af8005260 con 0x7f6b200c2130 2023-10-25T00:09:08.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.478+0000 7f6b07fff640 1 -- 172.21.15.107:0/37809023 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6b10004b90 con 0x7f6b200c2130 2023-10-25T00:09:08.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.478+0000 7f6b07fff640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b0c065e90 0x7f6b0c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:08.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.478+0000 7f6b07fff640 1 -- 172.21.15.107:0/37809023 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7f6b10083570 con 0x7f6b200c2130 2023-10-25T00:09:08.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.478+0000 7f6b1effd640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b0c065e90 0x7f6b0c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:08.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.478+0000 7f6b1effd640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b0c065e90 0x7f6b0c068350 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6b140ad000 tx=0x7f6b14008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:08.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.482+0000 7f6b07fff640 1 -- 172.21.15.107:0/37809023 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6b10053aa0 con 0x7f6b200c2130 2023-10-25T00:09:08.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.698+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/2447584078 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa29c0c4f60 con 0x7fa29c0c70e0 2023-10-25T00:09:08.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa28affd640 1 -- 172.21.15.107:0/2447584078 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5777 (secure 0 0 0) 0x7fa29c0c4f60 con 0x7fa29c0c70e0 2023-10-25T00:09:08.702 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:08.702 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:08.703 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 145 2023-10-25T00:09:08.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/2447584078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa27c065e60 msgr2=0x7fa27c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa27c065e60 0x7fa27c068320 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa28c000f70 tx=0x7fa28c009040 comp rx=0 tx=0).stop 2023-10-25T00:09:08.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/2447584078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa29c0c70e0 msgr2=0x7fa29c1ecc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa29c0c70e0 0x7fa29c1ecc20 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fa2900ae040 tx=0x7fa2900191c0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/2447584078 shutdown_connections 2023-10-25T00:09:08.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa27c065e60 0x7fa27c068320 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa29c156770 0x7fa29c1ed160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa29c0c70e0 0x7fa29c1ecc20 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 --2- 172.21.15.107:0/2447584078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa29c0c2130 0x7fa29c15b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/2447584078 >> 172.21.15.107:0/2447584078 conn(0x7fa29c0c65c0 msgr2=0x7fa29c15c360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/2447584078 shutdown_connections 2023-10-25T00:09:08.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.702+0000 7fa2a0bf0640 1 -- 172.21.15.107:0/2447584078 wait complete. 2023-10-25T00:09:08.723 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:09:08.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 -- 172.21.15.107:0/37809023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6af8005a90 con 0x7f6b200c2130 2023-10-25T00:09:08.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b07fff640 1 -- 172.21.15.107:0/37809023 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f6b10018020 con 0x7f6b200c2130 2023-10-25T00:09:08.726 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:08.726 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:08.727 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 145 2023-10-25T00:09:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 -- 172.21.15.107:0/37809023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b0c065e90 msgr2=0x7f6b0c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b0c065e90 0x7f6b0c068350 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6b140ad000 tx=0x7f6b14008040 comp rx=0 tx=0).stop 2023-10-25T00:09:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 -- 172.21.15.107:0/37809023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b200c2130 msgr2=0x7f6b2014d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b200c2130 0x7f6b2014d030 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000c480 tx=0x7f6b1000c940 comp rx=0 tx=0).stop 2023-10-25T00:09:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 -- 172.21.15.107:0/37809023 shutdown_connections 2023-10-25T00:09:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b200ba080 0x7f6b201ee7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b0c065e90 0x7f6b0c068350 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b20156770 0x7f6b2014d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 --2- 172.21.15.107:0/37809023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b200c2130 0x7f6b2014d030 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:08.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 -- 172.21.15.107:0/37809023 >> 172.21.15.107:0/37809023 conn(0x7f6b200c65c0 msgr2=0x7f6b200c3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:08.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 -- 172.21.15.107:0/37809023 shutdown_connections 2023-10-25T00:09:08.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:08.726+0000 7f6b05ffb640 1 -- 172.21.15.107:0/37809023 wait complete. 2023-10-25T00:09:08.747 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2023-10-25T00:09:08.747 INFO:tasks.mds_thrash.fs.[a]:waiting for 72 secs before thrashing 2023-10-25T00:09:10.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:10.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd90a989640 1 Processor -- start 2023-10-25T00:09:10.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd90a989640 1 -- start start 2023-10-25T00:09:10.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd90a989640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd90406cef0 0x7fd904152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:10.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd90a989640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd90406a770 0x7fd90406d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:10.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd90a989640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd904154350 0x7fd9041547e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:10.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd90a989640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9040a9e60 con 0x7fd90406a770 2023-10-25T00:09:10.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd90a989640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9040a99d0 con 0x7fd90406cef0 2023-10-25T00:09:10.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd90a989640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040a9450 con 0x7fd904154350 2023-10-25T00:09:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd8fbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd90406a770 0x7fd90406d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd8fbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd90406a770 0x7fd90406d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35476/0 (socket says 172.21.15.107:35476) 2023-10-25T00:09:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd8fbfff640 1 -- 172.21.15.107:0/1434650468 learned_addr learned my addr 172.21.15.107:0/1434650468 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd903fff640 1 --2- 172.21.15.107:0/1434650468 >> v2:172.21.15.110:3300/0 conn(0x7fd90406cef0 0x7fd904152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.866+0000 7fd908f26640 1 --2- 172.21.15.107:0/1434650468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd904154350 0x7fd9041547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:10.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd8fbfff640 1 -- 172.21.15.107:0/1434650468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd904154350 msgr2=0x7fd9041547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:10.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd8fbfff640 1 --2- 172.21.15.107:0/1434650468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd904154350 0x7fd9041547e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:10.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd8fbfff640 1 -- 172.21.15.107:0/1434650468 >> v2:172.21.15.110:3300/0 conn(0x7fd90406cef0 msgr2=0x7fd904152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:10.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd8fbfff640 1 --2- 172.21.15.107:0/1434650468 >> v2:172.21.15.110:3300/0 conn(0x7fd90406cef0 0x7fd904152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:10.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd8fbfff640 1 -- 172.21.15.107:0/1434650468 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90406a2f0 con 0x7fd90406a770 2023-10-25T00:09:10.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd8fbfff640 1 --2- 172.21.15.107:0/1434650468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd90406a770 0x7fd90406d2d0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fd90406be60 tx=0x7fd8ec01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6c0298728f9705 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:10.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd9037fe640 1 -- 172.21.15.107:0/1434650468 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0ab600 con 0x7fd90406a770 2023-10-25T00:09:10.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd9037fe640 1 -- 172.21.15.107:0/1434650468 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec0a7050 con 0x7fd90406a770 2023-10-25T00:09:10.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd9037fe640 1 -- 172.21.15.107:0/1434650468 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0ab980 con 0x7fd90406a770 2023-10-25T00:09:10.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd90a989640 1 -- 172.21.15.107:0/1434650468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd90406a770 msgr2=0x7fd90406d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:10.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd90a989640 1 --2- 172.21.15.107:0/1434650468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd90406a770 0x7fd90406d2d0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fd90406be60 tx=0x7fd8ec01c3e0 comp rx=0 tx=0).stop 2023-10-25T00:09:10.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd90a989640 1 -- 172.21.15.107:0/1434650468 shutdown_connections 2023-10-25T00:09:10.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd90a989640 1 --2- 172.21.15.107:0/1434650468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd904154350 0x7fd9041547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:10.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd90a989640 1 --2- 172.21.15.107:0/1434650468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd90406a770 0x7fd90406d2d0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:10.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd90a989640 1 --2- 172.21.15.107:0/1434650468 >> v2:172.21.15.110:3300/0 conn(0x7fd90406cef0 0x7fd904152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:10.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd90a989640 1 -- 172.21.15.107:0/1434650468 >> 172.21.15.107:0/1434650468 conn(0x7fd9040c6ed0 msgr2=0x7fd9040c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:10.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd90a989640 1 -- 172.21.15.107:0/1434650468 shutdown_connections 2023-10-25T00:09:10.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.870+0000 7fd90a989640 1 -- 172.21.15.107:0/1434650468 wait complete. 2023-10-25T00:09:10.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd90a989640 1 Processor -- start 2023-10-25T00:09:10.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd90a989640 1 -- start start 2023-10-25T00:09:10.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd90a989640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd90406a770 0x7fd904071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:10.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd90a989640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd90406cef0 0x7fd9040c04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:10.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd90a989640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd904154350 0x7fd9041ec970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:10.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd90a989640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd904070a30 con 0x7fd90406a770 2023-10-25T00:09:10.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd90a989640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd904157cc0 con 0x7fd904154350 2023-10-25T00:09:10.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd90a989640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90415c690 con 0x7fd90406cef0 2023-10-25T00:09:10.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd903fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd90406a770 0x7fd904071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:10.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd903fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd90406a770 0x7fd904071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35486/0 (socket says 172.21.15.107:35486) 2023-10-25T00:09:10.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd903fff640 1 -- 172.21.15.107:0/3959056282 learned_addr learned my addr 172.21.15.107:0/3959056282 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:10.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd908f26640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd904154350 0x7fd9041ec970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:10.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd903fff640 1 -- 172.21.15.107:0/3959056282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd90406cef0 msgr2=0x7fd9040c04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:10.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd903fff640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd90406cef0 0x7fd9040c04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:10.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd903fff640 1 -- 172.21.15.107:0/3959056282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd904154350 msgr2=0x7fd9041ec970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:10.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd903fff640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd904154350 0x7fd9041ec970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:10.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd903fff640 1 -- 172.21.15.107:0/3959056282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec0094d0 con 0x7fd90406a770 2023-10-25T00:09:10.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd903fff640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd90406a770 0x7fd904071130 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400d830 tx=0x7fd8f400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:10.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd9017fa640 1 -- 172.21.15.107:0/3959056282 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f400be10 con 0x7fd90406a770 2023-10-25T00:09:10.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd9017fa640 1 -- 172.21.15.107:0/3959056282 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f40098b0 con 0x7fd90406a770 2023-10-25T00:09:10.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd90a989640 1 -- 172.21.15.107:0/3959056282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9041578c0 con 0x7fd90406a770 2023-10-25T00:09:10.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd9017fa640 1 -- 172.21.15.107:0/3959056282 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4009a40 con 0x7fd90406a770 2023-10-25T00:09:10.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd90a989640 1 -- 172.21.15.107:0/3959056282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd904071ce0 con 0x7fd90406a770 2023-10-25T00:09:10.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.874+0000 7fd90a989640 1 -- 172.21.15.107:0/3959056282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9040746e0 con 0x7fd90406a770 2023-10-25T00:09:10.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.878+0000 7fd9017fa640 1 -- 172.21.15.107:0/3959056282 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd8f4009ba0 con 0x7fd90406a770 2023-10-25T00:09:10.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.878+0000 7fd9017fa640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd8e4065e60 0x7fd8e4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:10.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.878+0000 7fd9017fa640 1 -- 172.21.15.107:0/3959056282 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7fd8f4084860 con 0x7fd90406a770 2023-10-25T00:09:10.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.878+0000 7fd8fbfff640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd8e4065e60 0x7fd8e4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:10.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.878+0000 7fd8fbfff640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd8e4065e60 0x7fd8e4068320 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0ad000 tx=0x7fd8ec004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:10.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:10.882+0000 7fd9017fa640 1 -- 172.21.15.107:0/3959056282 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd8f4051250 con 0x7fd90406a770 2023-10-25T00:09:11.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.086+0000 7fd90a989640 1 -- 172.21.15.107:0/3959056282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd90406a550 con 0x7fd90406a770 2023-10-25T00:09:11.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.086+0000 7fd9017fa640 1 -- 172.21.15.107:0/3959056282 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5777 (secure 0 0 0) 0x7fd8f4018090 con 0x7fd90406a770 2023-10-25T00:09:11.089 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:11.089 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:11.090 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 145 2023-10-25T00:09:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 -- 172.21.15.107:0/3959056282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd8e4065e60 msgr2=0x7fd8e4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd8e4065e60 0x7fd8e4068320 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0ad000 tx=0x7fd8ec004740 comp rx=0 tx=0).stop 2023-10-25T00:09:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 -- 172.21.15.107:0/3959056282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd90406a770 msgr2=0x7fd904071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd90406a770 0x7fd904071130 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400d830 tx=0x7fd8f400dcf0 comp rx=0 tx=0).stop 2023-10-25T00:09:11.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 -- 172.21.15.107:0/3959056282 shutdown_connections 2023-10-25T00:09:11.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd904154350 0x7fd9041ec970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:11.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd8e4065e60 0x7fd8e4068320 secure :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0ad000 tx=0x7fd8ec004740 comp rx=0 tx=0).stop 2023-10-25T00:09:11.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd90406cef0 0x7fd9040c04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:11.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 --2- 172.21.15.107:0/3959056282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd90406a770 0x7fd904071130 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:11.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 -- 172.21.15.107:0/3959056282 >> 172.21.15.107:0/3959056282 conn(0x7fd9040c6ed0 msgr2=0x7fd9040c72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:11.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 -- 172.21.15.107:0/3959056282 shutdown_connections 2023-10-25T00:09:11.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:11.090+0000 7fd90a989640 1 -- 172.21.15.107:0/3959056282 wait complete. 2023-10-25T00:09:11.109 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:09:13.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:13.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f9c2e640 1 Processor -- start 2023-10-25T00:09:13.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f9c2e640 1 -- start start 2023-10-25T00:09:13.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f9c2e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f40aa140 0x7fc1f40aa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:13.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f9c2e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc1f40ac020 0x7fc1f40c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:13.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f9c2e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f4066000 con 0x7fc1f40c2520 2023-10-25T00:09:13.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f9c2e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f40aaa60 con 0x7fc1f40ac020 2023-10-25T00:09:13.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f9c2e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f40c0f40 con 0x7fc1f40aa140 2023-10-25T00:09:13.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f3fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc1f40c2520 0x7fc1f4154df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49950/0 (socket says 172.21.15.107:49950) 2023-10-25T00:09:13.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f3fff640 1 -- 172.21.15.107:0/364189749 learned_addr learned my addr 172.21.15.107:0/364189749 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:09:13.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f37fe640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f40aa140 0x7fc1f40aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:13.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1eaffd640 1 --2- 172.21.15.107:0/364189749 >> v2:172.21.15.110:3300/0 conn(0x7fc1f40ac020 0x7fc1f40c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:13.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371837924 0 0) 0x7fc1f4066000 con 0x7fc1f40c2520 2023-10-25T00:09:13.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e0003530 con 0x7fc1f40c2520 2023-10-25T00:09:13.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 718333521 0 0) 0x7fc1e0003530 con 0x7fc1f40c2520 2023-10-25T00:09:13.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f40aa140 msgr2=0x7fc1f40aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:13.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f40aa140 0x7fc1f40aa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:13.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 >> v2:172.21.15.110:3300/0 conn(0x7fc1f40ac020 msgr2=0x7fc1f40c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:13.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 --2- 172.21.15.107:0/364189749 >> v2:172.21.15.110:3300/0 conn(0x7fc1f40ac020 0x7fc1f40c1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:13.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f40c1220 con 0x7fc1f40c2520 2023-10-25T00:09:13.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fc1ec001a00 con 0x7fc1f40c2520 2023-10-25T00:09:13.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 >> v1:172.21.15.107:6789/0 conn(0x7fc1f40c2520 legacy=0x7fc1f4154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:13.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1e0003980 0x7fc1e0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:13.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1e00047d0 0x7fc1e0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:13.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f40aa140 0x7fc1e0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:13.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f40aaa60 con 0x7fc1e0003980 2023-10-25T00:09:13.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f40c0f40 con 0x7fc1e00047d0 2023-10-25T00:09:13.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e000b990 con 0x7fc1f40aa140 2023-10-25T00:09:13.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1f37fe640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1e0003980 0x7fc1e0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:13.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.246+0000 7fc1eaffd640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1e00047d0 0x7fc1e0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:13.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f3fff640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f40aa140 0x7fc1e0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:13.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f37fe640 1 -- 172.21.15.107:0/364189749 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f40aa140 msgr2=0x7fc1e0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:13.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f37fe640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f40aa140 0x7fc1e0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:13.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f37fe640 1 -- 172.21.15.107:0/364189749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1e00047d0 msgr2=0x7fc1e0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:13.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f37fe640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1e00047d0 0x7fc1e0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:13.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f37fe640 1 -- 172.21.15.107:0/364189749 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e0003530 con 0x7fc1e0003980 2023-10-25T00:09:13.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f37fe640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1e0003980 0x7fc1e0003da0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009ae0 tx=0x7fc1e401ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ef540e17353824 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:13.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e40a6430 con 0x7fc1e0003980 2023-10-25T00:09:13.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e40a6590 con 0x7fc1e0003980 2023-10-25T00:09:13.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f2ffd640 1 -- 172.21.15.107:0/364189749 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e40a6940 con 0x7fc1e0003980 2023-10-25T00:09:13.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/364189749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1e0003980 msgr2=0x7fc1e0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:13.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f9c2e640 1 --2- 172.21.15.107:0/364189749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1e0003980 0x7fc1e0003da0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4009ae0 tx=0x7fc1e401ca20 comp rx=0 tx=0).stop 2023-10-25T00:09:13.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f3fff640 1 -- 172.21.15.107:0/364189749 reap_dead start 2023-10-25T00:09:13.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/364189749 shutdown_connections 2023-10-25T00:09:13.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/364189749 >> 172.21.15.107:0/364189749 conn(0x7fc1f40c65c0 msgr2=0x7fc1f40c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:13.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/364189749 shutdown_connections 2023-10-25T00:09:13.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.250+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/364189749 wait complete. 2023-10-25T00:09:13.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f9c2e640 1 Processor -- start 2023-10-25T00:09:13.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f9c2e640 1 -- start start 2023-10-25T00:09:13.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f9c2e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1e0003980 0x7fc1f41e67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f9c2e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1f40bed80 0x7fc1f41eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f9c2e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f41ec010 0x7fc1f40a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f9c2e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f41eb090 con 0x7fc1f40bed80 2023-10-25T00:09:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f9c2e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f41eca60 con 0x7fc1e0003980 2023-10-25T00:09:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f9c2e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f41ea760 con 0x7fc1f41ec010 2023-10-25T00:09:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1f40bed80 0x7fc1f41eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1f40bed80 0x7fc1f41eb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51694/0 (socket says 172.21.15.107:51694) 2023-10-25T00:09:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1eaffd640 1 -- 172.21.15.107:0/1884777589 learned_addr learned my addr 172.21.15.107:0/1884777589 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f3fff640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f41ec010 0x7fc1f40a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:13.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f37fe640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1e0003980 0x7fc1f41e67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1eaffd640 1 -- 172.21.15.107:0/1884777589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f41ec010 msgr2=0x7fc1f40a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1eaffd640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f41ec010 0x7fc1f40a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1eaffd640 1 -- 172.21.15.107:0/1884777589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1e0003980 msgr2=0x7fc1f41e67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1eaffd640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1e0003980 0x7fc1f41e67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:13.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1eaffd640 1 -- 172.21.15.107:0/1884777589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f41eaa40 con 0x7fc1f40bed80 2023-10-25T00:09:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f37fe640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1e0003980 0x7fc1f41e67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:09:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1eaffd640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1f40bed80 0x7fc1f41eb520 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc011660 tx=0x7fc1dc011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f0ff9640 1 -- 172.21.15.107:0/1884777589 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc00f9c0 con 0x7fc1f40bed80 2023-10-25T00:09:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f0ff9640 1 -- 172.21.15.107:0/1884777589 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1dc00fb20 con 0x7fc1f40bed80 2023-10-25T00:09:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/1884777589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f41ead20 con 0x7fc1f40bed80 2023-10-25T00:09:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f0ff9640 1 -- 172.21.15.107:0/1884777589 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc00fe90 con 0x7fc1f40bed80 2023-10-25T00:09:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.254+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/1884777589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f41eba60 con 0x7fc1f40bed80 2023-10-25T00:09:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.258+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/1884777589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f41eb240 con 0x7fc1f40bed80 2023-10-25T00:09:13.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.258+0000 7fc1f0ff9640 1 -- 172.21.15.107:0/1884777589 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc1dc015460 con 0x7fc1f40bed80 2023-10-25T00:09:13.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.258+0000 7fc1f0ff9640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1d4065e50 0x7fc1d4068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:13.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.258+0000 7fc1f0ff9640 1 -- 172.21.15.107:0/1884777589 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7fc1dc084bf0 con 0x7fc1f40bed80 2023-10-25T00:09:13.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.258+0000 7fc1f37fe640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1d4065e50 0x7fc1d4068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:13.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.258+0000 7fc1f37fe640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1d4065e50 0x7fc1d4068310 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40ad000 tx=0x7fc1e40046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:13.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.262+0000 7fc1f0ff9640 1 -- 172.21.15.107:0/1884777589 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc1f41eb240 con 0x7fc1f40bed80 2023-10-25T00:09:13.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.482+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/1884777589 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1f40746e0 con 0x7fc1f40bed80 2023-10-25T00:09:13.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.482+0000 7fc1f0ff9640 1 -- 172.21.15.107:0/1884777589 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5777 (secure 0 0 0) 0x7fc1dc0515e0 con 0x7fc1f40bed80 2023-10-25T00:09:13.484 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:13.484 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:13.485 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 145 2023-10-25T00:09:13.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/1884777589 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1d4065e50 msgr2=0x7fc1d4068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:13.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1d4065e50 0x7fc1d4068310 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40ad000 tx=0x7fc1e40046d0 comp rx=0 tx=0).stop 2023-10-25T00:09:13.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/1884777589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1f40bed80 msgr2=0x7fc1f41eb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:13.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1f40bed80 0x7fc1f41eb520 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc011660 tx=0x7fc1dc011b20 comp rx=0 tx=0).stop 2023-10-25T00:09:13.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/1884777589 shutdown_connections 2023-10-25T00:09:13.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc1d4065e50 0x7fc1d4068310 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:13.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc1f41ec010 0x7fc1f40a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:13.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1f40bed80 0x7fc1f41eb520 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:13.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 --2- 172.21.15.107:0/1884777589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc1e0003980 0x7fc1f41e67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:13.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/1884777589 >> 172.21.15.107:0/1884777589 conn(0x7fc1f40c65c0 msgr2=0x7fc1f4154660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:13.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/1884777589 shutdown_connections 2023-10-25T00:09:13.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:13.486+0000 7fc1f9c2e640 1 -- 172.21.15.107:0/1884777589 wait complete. 2023-10-25T00:09:13.504 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:09:15.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:15.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f643de4d640 1 Processor -- start 2023-10-25T00:09:15.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f643de4d640 1 -- start start 2023-10-25T00:09:15.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f643de4d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f64381557e0 0x7f643815ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:15.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f643de4d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6438151fc0 con 0x7f643806d3d0 2023-10-25T00:09:15.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f643de4d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64381522a0 con 0x7f643814ec50 2023-10-25T00:09:15.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f643de4d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6438152580 con 0x7f64381557e0 2023-10-25T00:09:15.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f6436ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f643806d3d0 0x7f64381550d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49952/0 (socket says 172.21.15.107:49952) 2023-10-25T00:09:15.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f6436ffd640 1 -- 172.21.15.107:0/421763526 learned_addr learned my addr 172.21.15.107:0/421763526 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:09:15.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917074595 0 0) 0x7f6438151fc0 con 0x7f643806d3d0 2023-10-25T00:09:15.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f6437fff640 1 --2- 172.21.15.107:0/421763526 >> v2:172.21.15.110:3300/0 conn(0x7f64381557e0 0x7f643815ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:15.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6420003530 con 0x7f643806d3d0 2023-10-25T00:09:15.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1951503006 0 0) 0x7f6420003530 con 0x7f643806d3d0 2023-10-25T00:09:15.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 >> v2:172.21.15.110:3300/0 conn(0x7f64381557e0 msgr2=0x7f643815ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:15.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 --2- 172.21.15.107:0/421763526 >> v2:172.21.15.110:3300/0 conn(0x7f64381557e0 0x7f643815ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 >> v1:172.21.15.110:6789/0 conn(0x7f643814ec50 legacy=0x7f643806c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:15.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64380726a0 con 0x7f643806d3d0 2023-10-25T00:09:15.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f642c001680 con 0x7f643806d3d0 2023-10-25T00:09:15.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 >> v1:172.21.15.107:6789/0 conn(0x7f643806d3d0 legacy=0x7f64381550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:15.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 --2- 172.21.15.107:0/421763526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6420003980 0x7f6420003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:15.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 --2- 172.21.15.107:0/421763526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64200047d0 0x7f6420004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:15.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 --2- 172.21.15.107:0/421763526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6420005190 0x7f642000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:15.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6438152580 con 0x7f6420003980 2023-10-25T00:09:15.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642000c130 con 0x7f64200047d0 2023-10-25T00:09:15.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642000c2f0 con 0x7f6420005190 2023-10-25T00:09:15.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64377fe640 1 --2- 172.21.15.107:0/421763526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6420003980 0x7f6420003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:15.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f6436ffd640 1 --2- 172.21.15.107:0/421763526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64200047d0 0x7f6420004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:15.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f6437fff640 1 --2- 172.21.15.107:0/421763526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6420005190 0x7f642000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:15.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64377fe640 1 -- 172.21.15.107:0/421763526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6420005190 msgr2=0x7f642000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:15.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64377fe640 1 --2- 172.21.15.107:0/421763526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6420005190 0x7f642000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64377fe640 1 -- 172.21.15.107:0/421763526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64200047d0 msgr2=0x7f6420004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:15.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64377fe640 1 --2- 172.21.15.107:0/421763526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64200047d0 0x7f6420004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f64377fe640 1 -- 172.21.15.107:0/421763526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6420003530 con 0x7f6420003980 2023-10-25T00:09:15.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.666+0000 7f6437fff640 1 -- 172.21.15.107:0/421763526 reap_dead start 2023-10-25T00:09:15.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.670+0000 7f64377fe640 1 --2- 172.21.15.107:0/421763526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6420003980 0x7f6420003da0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f6428001780 tx=0x7f642801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fc34ed97fd7a49 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:15.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.670+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64280052d0 con 0x7f6420003980 2023-10-25T00:09:15.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.670+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64280a7050 con 0x7f6420003980 2023-10-25T00:09:15.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.670+0000 7f64367fc640 1 -- 172.21.15.107:0/421763526 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428003600 con 0x7f6420003980 2023-10-25T00:09:15.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.670+0000 7f643de4d640 1 -- 172.21.15.107:0/421763526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6420003980 msgr2=0x7f6420003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:15.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.670+0000 7f643de4d640 1 --2- 172.21.15.107:0/421763526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6420003980 0x7f6420003da0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f6428001780 tx=0x7f642801c3e0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.670+0000 7f643de4d640 1 -- 172.21.15.107:0/421763526 shutdown_connections 2023-10-25T00:09:15.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.670+0000 7f643de4d640 1 --2- 172.21.15.107:0/421763526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6420003980 0x7f6420003da0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.670+0000 7f643de4d640 1 -- 172.21.15.107:0/421763526 >> 172.21.15.107:0/421763526 conn(0x7f64380c6660 msgr2=0x7f64380c6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:15.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.670+0000 7f643de4d640 1 -- 172.21.15.107:0/421763526 shutdown_connections 2023-10-25T00:09:15.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.670+0000 7f643de4d640 1 -- 172.21.15.107:0/421763526 wait complete. 2023-10-25T00:09:15.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f643de4d640 1 Processor -- start 2023-10-25T00:09:15.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f643de4d640 1 -- start start 2023-10-25T00:09:15.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f643de4d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6420003980 0x7f643814ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:15.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f643de4d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f643806a8d0 0x7f64381dcb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:15.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f643de4d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64381ee850 0x7f64381dd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:15.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f643de4d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64381de4d0 con 0x7f643806a8d0 2023-10-25T00:09:15.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f643de4d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64381eee10 con 0x7f64381ee850 2023-10-25T00:09:15.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f643de4d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6438071350 con 0x7f6420003980 2023-10-25T00:09:15.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f6436ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f643806a8d0 0x7f64381dcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:15.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f6436ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f643806a8d0 0x7f64381dcb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51710/0 (socket says 172.21.15.107:51710) 2023-10-25T00:09:15.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f6436ffd640 1 -- 172.21.15.107:0/2475823972 learned_addr learned my addr 172.21.15.107:0/2475823972 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:15.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f6437fff640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64381ee850 0x7f64381dd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:15.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f64377fe640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6420003980 0x7f643814ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:15.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f6436ffd640 1 -- 172.21.15.107:0/2475823972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6420003980 msgr2=0x7f643814ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:15.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f6436ffd640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6420003980 0x7f643814ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f6436ffd640 1 -- 172.21.15.107:0/2475823972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64381ee850 msgr2=0x7f64381dd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:15.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f6436ffd640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64381ee850 0x7f64381dd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f6436ffd640 1 -- 172.21.15.107:0/2475823972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64380715f0 con 0x7f643806a8d0 2023-10-25T00:09:15.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f64377fe640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6420003980 0x7f643814ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:09:15.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f6436ffd640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f643806a8d0 0x7f64381dcb90 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f642c00e7b0 tx=0x7f642c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:15.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f641ffff640 1 -- 172.21.15.107:0/2475823972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c00c660 con 0x7f643806a8d0 2023-10-25T00:09:15.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f641ffff640 1 -- 172.21.15.107:0/2475823972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642c00c7e0 con 0x7f643806a8d0 2023-10-25T00:09:15.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f643de4d640 1 -- 172.21.15.107:0/2475823972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64380718d0 con 0x7f643806a8d0 2023-10-25T00:09:15.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f641ffff640 1 -- 172.21.15.107:0/2475823972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c00cb90 con 0x7f643806a8d0 2023-10-25T00:09:15.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.674+0000 7f643de4d640 1 -- 172.21.15.107:0/2475823972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643814da60 con 0x7f643806a8d0 2023-10-25T00:09:15.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.678+0000 7f641ffff640 1 -- 172.21.15.107:0/2475823972 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f642c00ccf0 con 0x7f643806a8d0 2023-10-25T00:09:15.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.678+0000 7f643de4d640 1 -- 172.21.15.107:0/2475823972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6438071df0 con 0x7f643806a8d0 2023-10-25T00:09:15.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.678+0000 7f641ffff640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6424065e60 0x7f6424068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:15.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.678+0000 7f641ffff640 1 -- 172.21.15.107:0/2475823972 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7f642c0842e0 con 0x7f643806a8d0 2023-10-25T00:09:15.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.678+0000 7f64377fe640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6424065e60 0x7f6424068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.678+0000 7f64377fe640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6424065e60 0x7f6424068320 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6428001750 tx=0x7f64280011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:15.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.682+0000 7f641ffff640 1 -- 172.21.15.107:0/2475823972 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6438071df0 con 0x7f643806a8d0 2023-10-25T00:09:15.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.914+0000 7f643de4d640 1 -- 172.21.15.107:0/2475823972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6438071df0 con 0x7f643806a8d0 2023-10-25T00:09:15.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.914+0000 7f641ffff640 1 -- 172.21.15.107:0/2475823972 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f6438071df0 con 0x7f643806a8d0 2023-10-25T00:09:15.915 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:15.915 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:15.916 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 145 2023-10-25T00:09:15.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.914+0000 7f643de4d640 1 -- 172.21.15.107:0/2475823972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6424065e60 msgr2=0x7f6424068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:15.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.914+0000 7f643de4d640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6424065e60 0x7f6424068320 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6428001750 tx=0x7f64280011a0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.914+0000 7f643de4d640 1 -- 172.21.15.107:0/2475823972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f643806a8d0 msgr2=0x7f64381dcb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:15.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.914+0000 7f643de4d640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f643806a8d0 0x7f64381dcb90 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f642c00e7b0 tx=0x7f642c00ec70 comp rx=0 tx=0).stop 2023-10-25T00:09:15.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.918+0000 7f643de4d640 1 -- 172.21.15.107:0/2475823972 shutdown_connections 2023-10-25T00:09:15.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.918+0000 7f643de4d640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64381ee850 0x7f64381dd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.918+0000 7f643de4d640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f643806a8d0 0x7f64381dcb90 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.918+0000 7f643de4d640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6424065e60 0x7f6424068320 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.918+0000 7f643de4d640 1 --2- 172.21.15.107:0/2475823972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6420003980 0x7f643814ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:15.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.918+0000 7f643de4d640 1 -- 172.21.15.107:0/2475823972 >> 172.21.15.107:0/2475823972 conn(0x7f64380c6660 msgr2=0x7f6438151a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:15.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.918+0000 7f643de4d640 1 -- 172.21.15.107:0/2475823972 shutdown_connections 2023-10-25T00:09:15.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:15.918+0000 7f643de4d640 1 -- 172.21.15.107:0/2475823972 wait complete. 2023-10-25T00:09:15.935 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:09:17.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:18.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.070+0000 7f2d2f154640 1 Processor -- start 2023-10-25T00:09:18.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.070+0000 7f2d2f154640 1 -- start start 2023-10-25T00:09:18.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2f154640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d28070b00 0x7f2d280c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:18.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2f154640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280c22f0 con 0x7f2d2814eb70 2023-10-25T00:09:18.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2f154640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d280c25d0 con 0x7f2d2806df30 2023-10-25T00:09:18.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2f154640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d28071700 con 0x7f2d28070b00 2023-10-25T00:09:18.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2cef0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2d2814eb70 0x7f2d281514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49958/0 (socket says 172.21.15.107:49958) 2023-10-25T00:09:18.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2cef0640 1 -- 172.21.15.107:0/3561483784 learned_addr learned my addr 172.21.15.107:0/3561483784 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:09:18.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d27fff640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d28070b00 0x7f2d280c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:18.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707284847 0 0) 0x7f2d280c22f0 con 0x7f2d2814eb70 2023-10-25T00:09:18.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d18003530 con 0x7f2d2814eb70 2023-10-25T00:09:18.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393395268 0 0) 0x7f2d18003530 con 0x7f2d2814eb70 2023-10-25T00:09:18.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d28070b00 msgr2=0x7f2d280c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:18.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d28070b00 0x7f2d280c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:18.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 >> v1:172.21.15.110:6789/0 conn(0x7f2d2806df30 legacy=0x7f2d2815be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:18.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d280c2be0 con 0x7f2d2814eb70 2023-10-25T00:09:18.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f2d1c001680 con 0x7f2d2814eb70 2023-10-25T00:09:18.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 >> v1:172.21.15.107:6789/0 conn(0x7f2d2814eb70 legacy=0x7f2d281514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:18.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 0x7f2d18003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:18.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d18004360 0x7f2d18004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:18.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d28070b00 0x7f2d18005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:18.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1800b990 con 0x7f2d180039b0 2023-10-25T00:09:18.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1800bb00 con 0x7f2d18004360 2023-10-25T00:09:18.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1800bcc0 con 0x7f2d28070b00 2023-10-25T00:09:18.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2cef0640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 0x7f2d18003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:18.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d27fff640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d18004360 0x7f2d18004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:18.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2d6f1640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d28070b00 0x7f2d18005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:18.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2cef0640 1 -- 172.21.15.107:0/3561483784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d28070b00 msgr2=0x7f2d18005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:18.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2cef0640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d28070b00 0x7f2d18005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:18.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2cef0640 1 -- 172.21.15.107:0/3561483784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d18004360 msgr2=0x7f2d18004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:18.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2cef0640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d18004360 0x7f2d18004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:18.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.074+0000 7f2d2cef0640 1 -- 172.21.15.107:0/3561483784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d18003530 con 0x7f2d180039b0 2023-10-25T00:09:18.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d2cef0640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 0x7f2d18003d90 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00c210 tx=0x7f2d1c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc34d162ee6e0474 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:18.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c005ae0 con 0x7f2d180039b0 2023-10-25T00:09:18.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c005c60 con 0x7f2d180039b0 2023-10-25T00:09:18.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d277fe640 1 -- 172.21.15.107:0/3561483784 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c007070 con 0x7f2d180039b0 2023-10-25T00:09:18.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d2f154640 1 -- 172.21.15.107:0/3561483784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 msgr2=0x7f2d18003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:18.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d2f154640 1 --2- 172.21.15.107:0/3561483784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 0x7f2d18003d90 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00c210 tx=0x7f2d1c01cc60 comp rx=0 tx=0).stop 2023-10-25T00:09:18.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d2d6f1640 1 -- 172.21.15.107:0/3561483784 reap_dead start 2023-10-25T00:09:18.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d2f154640 1 -- 172.21.15.107:0/3561483784 shutdown_connections 2023-10-25T00:09:18.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d2f154640 1 -- 172.21.15.107:0/3561483784 >> 172.21.15.107:0/3561483784 conn(0x7f2d280c39a0 msgr2=0x7f2d280c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:18.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d2f154640 1 -- 172.21.15.107:0/3561483784 shutdown_connections 2023-10-25T00:09:18.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d2f154640 1 -- 172.21.15.107:0/3561483784 wait complete. 2023-10-25T00:09:18.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d2f154640 1 Processor -- start 2023-10-25T00:09:18.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.078+0000 7f2d2f154640 1 -- start start 2023-10-25T00:09:18.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2f154640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 0x7f2d2814de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:18.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2f154640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d2814f4a0 0x7f2d2813ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:18.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2f154640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d2814d1a0 0x7f2d281ed5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:18.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2f154640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2814da40 con 0x7f2d180039b0 2023-10-25T00:09:18.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2f154640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d28150200 con 0x7f2d2814f4a0 2023-10-25T00:09:18.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2f154640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2814d6f0 con 0x7f2d2814d1a0 2023-10-25T00:09:18.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2cef0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 0x7f2d2814de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:18.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2cef0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 0x7f2d2814de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51728/0 (socket says 172.21.15.107:51728) 2023-10-25T00:09:18.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2cef0640 1 -- 172.21.15.107:0/1744763622 learned_addr learned my addr 172.21.15.107:0/1744763622 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:18.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d27fff640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d2814f4a0 0x7f2d2813ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:18.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2d6f1640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d2814d1a0 0x7f2d281ed5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:18.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2cef0640 1 -- 172.21.15.107:0/1744763622 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d2814d1a0 msgr2=0x7f2d281ed5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:18.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2cef0640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d2814d1a0 0x7f2d281ed5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:18.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2cef0640 1 -- 172.21.15.107:0/1744763622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d2814f4a0 msgr2=0x7f2d2813ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:18.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2cef0640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d2814f4a0 0x7f2d2813ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:18.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2cef0640 1 -- 172.21.15.107:0/1744763622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2814e3a0 con 0x7f2d180039b0 2023-10-25T00:09:18.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d27fff640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d2814f4a0 0x7f2d2813ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:09:18.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2cef0640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 0x7f2d2814de60 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c01ca90 tx=0x7f2d18005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:18.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d257fa640 1 -- 172.21.15.107:0/1744763622 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c0b4070 con 0x7f2d180039b0 2023-10-25T00:09:18.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d257fa640 1 -- 172.21.15.107:0/1744763622 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c005e90 con 0x7f2d180039b0 2023-10-25T00:09:18.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2f154640 1 -- 172.21.15.107:0/1744763622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2814f960 con 0x7f2d180039b0 2023-10-25T00:09:18.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d257fa640 1 -- 172.21.15.107:0/1744763622 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c001640 con 0x7f2d180039b0 2023-10-25T00:09:18.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2f154640 1 -- 172.21.15.107:0/1744763622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2814ff20 con 0x7f2d180039b0 2023-10-25T00:09:18.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d2f154640 1 -- 172.21.15.107:0/1744763622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2814db80 con 0x7f2d180039b0 2023-10-25T00:09:18.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d257fa640 1 -- 172.21.15.107:0/1744763622 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2d1c007090 con 0x7f2d180039b0 2023-10-25T00:09:18.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.082+0000 7f2d257fa640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2d0c065e90 0x7f2d0c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:18.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.086+0000 7f2d257fa640 1 -- 172.21.15.107:0/1744763622 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 9283+0+0 (secure 0 0 0) 0x7f2d1c11e060 con 0x7f2d180039b0 2023-10-25T00:09:18.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.086+0000 7f2d27fff640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2d0c065e90 0x7f2d0c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:18.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.086+0000 7f2d27fff640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2d0c065e90 0x7f2d0c068350 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2d280c12b0 tx=0x7f2d14008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:18.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.090+0000 7f2d257fa640 1 -- 172.21.15.107:0/1744763622 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2d1c0eaa50 con 0x7f2d180039b0 2023-10-25T00:09:18.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 -- 172.21.15.107:0/1744763622 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d2814db80 con 0x7f2d180039b0 2023-10-25T00:09:18.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d257fa640 1 -- 172.21.15.107:0/1744763622 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f2d2814db80 con 0x7f2d180039b0 2023-10-25T00:09:18.343 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:18.344 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:08:26.829276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7512},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7512":{"gid":7512,"name":"c","rank":0,"incarnation":136,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2276136116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2276136116},{"type":"v1","addr":"172.21.15.107:6835","nonce":2276136116}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 145 2023-10-25T00:09:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 -- 172.21.15.107:0/1744763622 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2d0c065e90 msgr2=0x7f2d0c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2d0c065e90 0x7f2d0c068350 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2d280c12b0 tx=0x7f2d14008040 comp rx=0 tx=0).stop 2023-10-25T00:09:18.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 -- 172.21.15.107:0/1744763622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 msgr2=0x7f2d2814de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:18.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 0x7f2d2814de60 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c01ca90 tx=0x7f2d18005800 comp rx=0 tx=0).stop 2023-10-25T00:09:18.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 -- 172.21.15.107:0/1744763622 shutdown_connections 2023-10-25T00:09:18.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2d0c065e90 0x7f2d0c068350 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2d2814d1a0 0x7f2d281ed5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2d2814f4a0 0x7f2d2813ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 --2- 172.21.15.107:0/1744763622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d180039b0 0x7f2d2814de60 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 -- 172.21.15.107:0/1744763622 >> 172.21.15.107:0/1744763622 conn(0x7f2d280c39a0 msgr2=0x7f2d28155e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 -- 172.21.15.107:0/1744763622 shutdown_connections 2023-10-25T00:09:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:18.342+0000 7f2d2f154640 1 -- 172.21.15.107:0/1744763622 wait complete. 2023-10-25T00:09:18.364 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:09:20.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:20.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.506+0000 7f9abf2e0640 1 Processor -- start 2023-10-25T00:09:20.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.506+0000 7f9abf2e0640 1 -- start start 2023-10-25T00:09:20.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.506+0000 7f9abf2e0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9ab8156770 0x7f9ab80ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.506+0000 7f9abf2e0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 0x7f9ab80bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.506+0000 7f9abf2e0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab80c1ca0 con 0x7f9ab80c2130 2023-10-25T00:09:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.506+0000 7f9abf2e0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab80c27e0 con 0x7f9ab8156770 2023-10-25T00:09:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.506+0000 7f9abf2e0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80c10c0 con 0x7f9ab80c70e0 2023-10-25T00:09:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.506+0000 7f9abc87b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 0x7f9ab80bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.506+0000 7f9abc87b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 0x7f9ab80bdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:53896/0 (socket says 172.21.15.107:53896) 2023-10-25T00:09:20.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.506+0000 7f9abc87b640 1 -- 172.21.15.107:0/1114950723 learned_addr learned my addr 172.21.15.107:0/1114950723 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:20.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abd07c640 1 --2- 172.21.15.107:0/1114950723 >> v2:172.21.15.110:3300/0 conn(0x7f9ab8156770 0x7f9ab80ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abc87b640 1 -- 172.21.15.107:0/1114950723 >> v2:172.21.15.110:3300/0 conn(0x7f9ab8156770 msgr2=0x7f9ab80ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:20.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abc87b640 1 --2- 172.21.15.107:0/1114950723 >> v2:172.21.15.110:3300/0 conn(0x7f9ab8156770 0x7f9ab80ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abc87b640 1 -- 172.21.15.107:0/1114950723 >> v1:172.21.15.110:6789/0 conn(0x7f9ab80c2130 legacy=0x7f9ab8069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:20.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abc87b640 1 -- 172.21.15.107:0/1114950723 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab80c13a0 con 0x7f9ab80c70e0 2023-10-25T00:09:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abc87b640 1 --2- 172.21.15.107:0/1114950723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 0x7f9ab80bdfa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009870 tx=0x7f9aa801c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcbef4b1466d6533 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:20.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9aaffff640 1 -- 172.21.15.107:0/1114950723 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa80ac410 con 0x7f9ab80c70e0 2023-10-25T00:09:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9aaffff640 1 -- 172.21.15.107:0/1114950723 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa80ac570 con 0x7f9ab80c70e0 2023-10-25T00:09:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9aaffff640 1 -- 172.21.15.107:0/1114950723 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa80ac900 con 0x7f9ab80c70e0 2023-10-25T00:09:20.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1114950723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 msgr2=0x7f9ab80bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abf2e0640 1 --2- 172.21.15.107:0/1114950723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 0x7f9ab80bdfa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009870 tx=0x7f9aa801c3f0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1114950723 shutdown_connections 2023-10-25T00:09:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abf2e0640 1 --2- 172.21.15.107:0/1114950723 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 0x7f9ab80bdfa0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abf2e0640 1 --2- 172.21.15.107:0/1114950723 >> v2:172.21.15.110:3300/0 conn(0x7f9ab8156770 0x7f9ab80ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1114950723 >> 172.21.15.107:0/1114950723 conn(0x7f9ab80c65c0 msgr2=0x7f9ab80c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:20.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.510+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1114950723 shutdown_connections 2023-10-25T00:09:20.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1114950723 wait complete. 2023-10-25T00:09:20.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abf2e0640 1 Processor -- start 2023-10-25T00:09:20.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abf2e0640 1 -- start start 2023-10-25T00:09:20.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abf2e0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ab80c2130 0x7f9ab8152790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abf2e0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 0x7f9ab814c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abf2e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ab8156770 0x7f9ab814c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abf2e0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80ab2e0 con 0x7f9ab8156770 2023-10-25T00:09:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abf2e0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80ab000 con 0x7f9ab80c2130 2023-10-25T00:09:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abf2e0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80aa970 con 0x7f9ab80c70e0 2023-10-25T00:09:20.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abd87d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ab8156770 0x7f9ab814c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abd87d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ab8156770 0x7f9ab814c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51730/0 (socket says 172.21.15.107:51730) 2023-10-25T00:09:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abd87d640 1 -- 172.21.15.107:0/1611403691 learned_addr learned my addr 172.21.15.107:0/1611403691 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:20.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abd07c640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ab80c2130 0x7f9ab8152790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abc87b640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 0x7f9ab814c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abd87d640 1 -- 172.21.15.107:0/1611403691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 msgr2=0x7f9ab814c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abd87d640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 0x7f9ab814c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abd87d640 1 -- 172.21.15.107:0/1611403691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ab80c2130 msgr2=0x7f9ab8152790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:20.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abd87d640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ab80c2130 0x7f9ab8152790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abd87d640 1 -- 172.21.15.107:0/1611403691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa80094d0 con 0x7f9ab8156770 2023-10-25T00:09:20.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abd87d640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ab8156770 0x7f9ab814c740 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c770 tx=0x7f9ab000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:20.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9aadffb640 1 -- 172.21.15.107:0/1611403691 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0003b80 con 0x7f9ab8156770 2023-10-25T00:09:20.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9aadffb640 1 -- 172.21.15.107:0/1611403691 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab0003ce0 con 0x7f9ab8156770 2023-10-25T00:09:20.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1611403691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab814d050 con 0x7f9ab8156770 2023-10-25T00:09:20.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9aadffb640 1 -- 172.21.15.107:0/1611403691 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab000aa90 con 0x7f9ab8156770 2023-10-25T00:09:20.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.514+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1611403691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab8153a90 con 0x7f9ab8156770 2023-10-25T00:09:20.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.518+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1611403691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab80aab30 con 0x7f9ab8156770 2023-10-25T00:09:20.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.518+0000 7f9aadffb640 1 -- 172.21.15.107:0/1611403691 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9ab000abf0 con 0x7f9ab8156770 2023-10-25T00:09:20.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.518+0000 7f9aadffb640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9a9c065e60 0x7f9a9c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.518+0000 7f9abd07c640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9a9c065e60 0x7f9a9c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.518+0000 7f9aadffb640 1 -- 172.21.15.107:0/1611403691 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7f9ab0084280 con 0x7f9ab8156770 2023-10-25T00:09:20.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.518+0000 7f9abd07c640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9a9c065e60 0x7f9a9c068320 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80b9a70 tx=0x7f9ab4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:20.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.522+0000 7f9aadffb640 1 -- 172.21.15.107:0/1611403691 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9ab0050cd0 con 0x7f9ab8156770 2023-10-25T00:09:20.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.762+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1611403691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ab80be460 con 0x7f9ab8156770 2023-10-25T00:09:20.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.762+0000 7f9aadffb640 1 -- 172.21.15.107:0/1611403691 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f9ab00542f0 con 0x7f9ab8156770 2023-10-25T00:09:20.763 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:20.764 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:20.741995+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:20.764 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 148 2023-10-25T00:09:20.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.762+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1611403691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9a9c065e60 msgr2=0x7f9a9c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:20.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.762+0000 7f9abf2e0640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9a9c065e60 0x7f9a9c068320 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80b9a70 tx=0x7f9ab4009190 comp rx=0 tx=0).stop 2023-10-25T00:09:20.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.762+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1611403691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ab8156770 msgr2=0x7f9ab814c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.762+0000 7f9abf2e0640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ab8156770 0x7f9ab814c740 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c770 tx=0x7f9ab000cc30 comp rx=0 tx=0).stop 2023-10-25T00:09:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.766+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1611403691 shutdown_connections 2023-10-25T00:09:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.766+0000 7f9abf2e0640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9ab8156770 0x7f9ab814c740 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.766+0000 7f9abf2e0640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9a9c065e60 0x7f9a9c068320 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.766+0000 7f9abf2e0640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ab80c70e0 0x7f9ab814c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.766+0000 7f9abf2e0640 1 --2- 172.21.15.107:0/1611403691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ab80c2130 0x7f9ab8152790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.766+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1611403691 >> 172.21.15.107:0/1611403691 conn(0x7f9ab80c65c0 msgr2=0x7f9ab8153380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:20.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.766+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1611403691 shutdown_connections 2023-10-25T00:09:20.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.766+0000 7f9abf2e0640 1 -- 172.21.15.107:0/1611403691 wait complete. 2023-10-25T00:09:20.781 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2023-10-25T00:09:20.781 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:09:20.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:20.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd862b8640 1 Processor -- start 2023-10-25T00:09:20.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd862b8640 1 -- start start 2023-10-25T00:09:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd862b8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80152040 0x7fcd800c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd862b8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd80155680 con 0x7fcd8006a4b0 2023-10-25T00:09:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd862b8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd80154a70 con 0x7fcd800c8100 2023-10-25T00:09:20.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd862b8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd80154d50 con 0x7fcd80152040 2023-10-25T00:09:20.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd84855640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcd8006a4b0 0x7fcd800ac410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49962/0 (socket says 172.21.15.107:49962) 2023-10-25T00:09:20.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd84855640 1 -- 172.21.15.107:0/1848978744 learned_addr learned my addr 172.21.15.107:0/1848978744 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:09:20.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7ffff640 1 --2- 172.21.15.107:0/1848978744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80152040 0x7fcd800c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077915317 0 0) 0x7fcd80155680 con 0x7fcd8006a4b0 2023-10-25T00:09:20.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd68003530 con 0x7fcd8006a4b0 2023-10-25T00:09:20.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3376388235 0 0) 0x7fcd68003530 con 0x7fcd8006a4b0 2023-10-25T00:09:20.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80152040 msgr2=0x7fcd800c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:20.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 --2- 172.21.15.107:0/1848978744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80152040 0x7fcd800c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 >> v1:172.21.15.110:6789/0 conn(0x7fcd800c8100 legacy=0x7fcd800abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:20.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd80155030 con 0x7fcd8006a4b0 2023-10-25T00:09:20.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fcd78001a00 con 0x7fcd8006a4b0 2023-10-25T00:09:20.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 >> v1:172.21.15.107:6789/0 conn(0x7fcd8006a4b0 legacy=0x7fcd800ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:20.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 --2- 172.21.15.107:0/1848978744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd680039b0 0x7fcd68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 --2- 172.21.15.107:0/1848978744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd680047d0 0x7fcd68004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 --2- 172.21.15.107:0/1848978744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80152040 0x7fcd68005260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd80154d50 con 0x7fcd680039b0 2023-10-25T00:09:20.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6800bb50 con 0x7fcd680047d0 2023-10-25T00:09:20.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6800bd10 con 0x7fcd80152040 2023-10-25T00:09:20.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7ffff640 1 --2- 172.21.15.107:0/1848978744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd680039b0 0x7fcd68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7f7fe640 1 --2- 172.21.15.107:0/1848978744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd680047d0 0x7fcd68004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7ffff640 1 -- 172.21.15.107:0/1848978744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80152040 msgr2=0x7fcd68005260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7ffff640 1 --2- 172.21.15.107:0/1848978744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80152040 0x7fcd68005260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7ffff640 1 -- 172.21.15.107:0/1848978744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd680047d0 msgr2=0x7fcd68004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7ffff640 1 --2- 172.21.15.107:0/1848978744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd680047d0 0x7fcd68004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.902+0000 7fcd7ffff640 1 -- 172.21.15.107:0/1848978744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd68003530 con 0x7fcd680039b0 2023-10-25T00:09:20.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd7ffff640 1 --2- 172.21.15.107:0/1848978744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd680039b0 0x7fcd68003dd0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009ae0 tx=0x7fcd7001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70969ca99def328 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:20.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd700a5570 con 0x7fcd680039b0 2023-10-25T00:09:20.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd700a56d0 con 0x7fcd680039b0 2023-10-25T00:09:20.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd7effd640 1 -- 172.21.15.107:0/1848978744 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd700a5880 con 0x7fcd680039b0 2023-10-25T00:09:20.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd862b8640 1 -- 172.21.15.107:0/1848978744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd680039b0 msgr2=0x7fcd68003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd862b8640 1 --2- 172.21.15.107:0/1848978744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd680039b0 0x7fcd68003dd0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009ae0 tx=0x7fcd7001ca90 comp rx=0 tx=0).stop 2023-10-25T00:09:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd84855640 1 -- 172.21.15.107:0/1848978744 reap_dead start 2023-10-25T00:09:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd862b8640 1 -- 172.21.15.107:0/1848978744 shutdown_connections 2023-10-25T00:09:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd862b8640 1 -- 172.21.15.107:0/1848978744 >> 172.21.15.107:0/1848978744 conn(0x7fcd800ca200 msgr2=0x7fcd800ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:20.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd862b8640 1 -- 172.21.15.107:0/1848978744 shutdown_connections 2023-10-25T00:09:20.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd862b8640 1 -- 172.21.15.107:0/1848978744 wait complete. 2023-10-25T00:09:20.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd862b8640 1 Processor -- start 2023-10-25T00:09:20.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.906+0000 7fcd862b8640 1 -- start start 2023-10-25T00:09:20.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd862b8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd680039b0 0x7fcd800ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd862b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd800c1240 0x7fcd8014e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd862b8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80158690 0x7fcd8014e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd862b8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800c2250 con 0x7fcd800c1240 2023-10-25T00:09:20.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd862b8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800c16d0 con 0x7fcd680039b0 2023-10-25T00:09:20.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd862b8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800c19b0 con 0x7fcd80158690 2023-10-25T00:09:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd800c1240 0x7fcd8014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd800c1240 0x7fcd8014e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51738/0 (socket says 172.21.15.107:51738) 2023-10-25T00:09:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7f7fe640 1 -- 172.21.15.107:0/34276279 learned_addr learned my addr 172.21.15.107:0/34276279 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7ffff640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd680039b0 0x7fcd800ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd84855640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80158690 0x7fcd8014e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7f7fe640 1 -- 172.21.15.107:0/34276279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80158690 msgr2=0x7fcd8014e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:20.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7f7fe640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80158690 0x7fcd8014e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7f7fe640 1 -- 172.21.15.107:0/34276279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd680039b0 msgr2=0x7fcd800ac0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:20.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7f7fe640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd680039b0 0x7fcd800ac0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:20.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7f7fe640 1 -- 172.21.15.107:0/34276279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd800c1c90 con 0x7fcd800c1240 2023-10-25T00:09:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7f7fe640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd800c1240 0x7fcd8014e230 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400e580 tx=0x7fcd7400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7cff9640 1 -- 172.21.15.107:0/34276279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd740052e0 con 0x7fcd800c1240 2023-10-25T00:09:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7cff9640 1 -- 172.21.15.107:0/34276279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd74005440 con 0x7fcd800c1240 2023-10-25T00:09:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7cff9640 1 -- 172.21.15.107:0/34276279 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd74003840 con 0x7fcd800c1240 2023-10-25T00:09:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd862b8640 1 -- 172.21.15.107:0/34276279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd800c1f70 con 0x7fcd800c1240 2023-10-25T00:09:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd862b8640 1 -- 172.21.15.107:0/34276279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8014ecb0 con 0x7fcd800c1240 2023-10-25T00:09:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd862b8640 1 -- 172.21.15.107:0/34276279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd800746e0 con 0x7fcd800c1240 2023-10-25T00:09:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.910+0000 7fcd7cff9640 1 -- 172.21.15.107:0/34276279 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcd740039a0 con 0x7fcd800c1240 2023-10-25T00:09:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.914+0000 7fcd7cff9640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd6c065e60 0x7fcd6c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.914+0000 7fcd7ffff640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd6c065e60 0x7fcd6c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.914+0000 7fcd7cff9640 1 -- 172.21.15.107:0/34276279 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7fcd74084430 con 0x7fcd800c1240 2023-10-25T00:09:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.914+0000 7fcd7ffff640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd6c065e60 0x7fcd6c068320 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcd700af000 tx=0x7fcd700046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:20.918+0000 7fcd7cff9640 1 -- 172.21.15.107:0/34276279 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fcd74050ea0 con 0x7fcd800c1240 2023-10-25T00:09:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.138+0000 7fcd862b8640 1 -- 172.21.15.107:0/34276279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd80155a10 con 0x7fcd800c1240 2023-10-25T00:09:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.138+0000 7fcd7cff9640 1 -- 172.21.15.107:0/34276279 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+5016 (secure 0 0 0) 0x7fcd740544c0 con 0x7fcd800c1240 2023-10-25T00:09:21.140 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:21.140 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:20.741995+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:21.141 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 148 2023-10-25T00:09:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 -- 172.21.15.107:0/34276279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd6c065e60 msgr2=0x7fcd6c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd6c065e60 0x7fcd6c068320 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcd700af000 tx=0x7fcd700046d0 comp rx=0 tx=0).stop 2023-10-25T00:09:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 -- 172.21.15.107:0/34276279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd800c1240 msgr2=0x7fcd8014e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd800c1240 0x7fcd8014e230 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400e580 tx=0x7fcd7400ea40 comp rx=0 tx=0).stop 2023-10-25T00:09:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 -- 172.21.15.107:0/34276279 shutdown_connections 2023-10-25T00:09:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd6c065e60 0x7fcd6c068320 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd80158690 0x7fcd8014e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd800c1240 0x7fcd8014e230 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 --2- 172.21.15.107:0/34276279 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd680039b0 0x7fcd800ac0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:21.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 -- 172.21.15.107:0/34276279 >> 172.21.15.107:0/34276279 conn(0x7fcd800ca200 msgr2=0x7fcd8014f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:21.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 -- 172.21.15.107:0/34276279 shutdown_connections 2023-10-25T00:09:21.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:21.142+0000 7fcd862b8640 1 -- 172.21.15.107:0/34276279 wait complete. 2023-10-25T00:09:21.158 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 148, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7578": { "addr": "172.21.15.107:6837/1788268787", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 1788268787, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 1788268787, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7578, "incarnation": 141, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:09:01.764886+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7578 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 148, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6568": { "addr": "172.21.15.110:6835/326715537", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 326715537, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 326715537, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6568, "incarnation": 146, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:09:20.741995+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6568 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:09:21.822 INFO:tasks.ceph.mds.d.smithi110.stderr:2023-10-25T00:09:21.820+0000 7f5358105640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:09:23.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:23.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.298+0000 7f3d69f7d640 1 Processor -- start 2023-10-25T00:09:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.298+0000 7f3d69f7d640 1 -- start start 2023-10-25T00:09:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.298+0000 7f3d69f7d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3d64070b00 0x7f3d640c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:23.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.298+0000 7f3d69f7d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3d6406df30 0x7f3d6406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.298+0000 7f3d69f7d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d640c22f0 con 0x7f3d6414eb70 2023-10-25T00:09:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.298+0000 7f3d69f7d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d64071700 con 0x7f3d64070b00 2023-10-25T00:09:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.298+0000 7f3d69f7d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d640c25d0 con 0x7f3d6406df30 2023-10-25T00:09:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.298+0000 7f3d62ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3d64070b00 0x7f3d640c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.298+0000 7f3d62ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3d64070b00 0x7f3d640c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49338/0 (socket says 172.21.15.107:49338) 2023-10-25T00:09:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.298+0000 7f3d62ffd640 1 -- 172.21.15.107:0/2108847556 learned_addr learned my addr 172.21.15.107:0/2108847556 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d63fff640 1 --2- 172.21.15.107:0/2108847556 >> v2:172.21.15.110:3300/0 conn(0x7f3d6406df30 0x7f3d6406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d62ffd640 1 -- 172.21.15.107:0/2108847556 >> v2:172.21.15.110:3300/0 conn(0x7f3d6406df30 msgr2=0x7f3d6406e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d62ffd640 1 --2- 172.21.15.107:0/2108847556 >> v2:172.21.15.110:3300/0 conn(0x7f3d6406df30 0x7f3d6406e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:23.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d62ffd640 1 -- 172.21.15.107:0/2108847556 >> v1:172.21.15.110:6789/0 conn(0x7f3d6414eb70 legacy=0x7f3d641514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:23.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d62ffd640 1 -- 172.21.15.107:0/2108847556 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d640c2be0 con 0x7f3d64070b00 2023-10-25T00:09:23.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d62ffd640 1 --2- 172.21.15.107:0/2108847556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d64070b00 0x7f3d640c33e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3d641525e0 tx=0x7f3d5801ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6e6303529741a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:23.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d627fc640 1 -- 172.21.15.107:0/2108847556 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d580a7bc0 con 0x7f3d64070b00 2023-10-25T00:09:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d627fc640 1 -- 172.21.15.107:0/2108847556 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d580a7d20 con 0x7f3d64070b00 2023-10-25T00:09:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d627fc640 1 -- 172.21.15.107:0/2108847556 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d580af660 con 0x7f3d64070b00 2023-10-25T00:09:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d69f7d640 1 -- 172.21.15.107:0/2108847556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d64070b00 msgr2=0x7f3d640c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d69f7d640 1 --2- 172.21.15.107:0/2108847556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d64070b00 0x7f3d640c33e0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3d641525e0 tx=0x7f3d5801ebd0 comp rx=0 tx=0).stop 2023-10-25T00:09:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d69f7d640 1 -- 172.21.15.107:0/2108847556 shutdown_connections 2023-10-25T00:09:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d69f7d640 1 --2- 172.21.15.107:0/2108847556 >> v2:172.21.15.110:3300/0 conn(0x7f3d6406df30 0x7f3d6406e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d69f7d640 1 --2- 172.21.15.107:0/2108847556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d64070b00 0x7f3d640c33e0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d69f7d640 1 -- 172.21.15.107:0/2108847556 >> 172.21.15.107:0/2108847556 conn(0x7f3d640c39a0 msgr2=0x7f3d640c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:23.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d69f7d640 1 -- 172.21.15.107:0/2108847556 shutdown_connections 2023-10-25T00:09:23.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.302+0000 7f3d69f7d640 1 -- 172.21.15.107:0/2108847556 wait complete. 2023-10-25T00:09:23.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d69f7d640 1 Processor -- start 2023-10-25T00:09:23.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d69f7d640 1 -- start start 2023-10-25T00:09:23.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d69f7d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d6406df30 0x7f3d641e8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:23.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d69f7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d64070b00 0x7f3d64150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:23.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d69f7d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d6414eb70 0x7f3d641ec190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:23.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d69f7d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d641ebd70 con 0x7f3d64070b00 2023-10-25T00:09:23.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d69f7d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d641454e0 con 0x7f3d6414eb70 2023-10-25T00:09:23.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d69f7d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6406b510 con 0x7f3d6406df30 2023-10-25T00:09:23.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d62ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d64070b00 0x7f3d64150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:23.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d62ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d64070b00 0x7f3d64150600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49352/0 (socket says 172.21.15.107:49352) 2023-10-25T00:09:23.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d62ffd640 1 -- 172.21.15.107:0/930104958 learned_addr learned my addr 172.21.15.107:0/930104958 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:23.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d637fe640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d6406df30 0x7f3d641e8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:23.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d63fff640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d6414eb70 0x7f3d641ec190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:23.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d62ffd640 1 -- 172.21.15.107:0/930104958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d6406df30 msgr2=0x7f3d641e8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d62ffd640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d6406df30 0x7f3d641e8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d62ffd640 1 -- 172.21.15.107:0/930104958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d6414eb70 msgr2=0x7f3d641ec190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d62ffd640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d6414eb70 0x7f3d641ec190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:23.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d62ffd640 1 -- 172.21.15.107:0/930104958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5800b770 con 0x7f3d64070b00 2023-10-25T00:09:23.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d63fff640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d6414eb70 0x7f3d641ec190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:09:23.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d62ffd640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d64070b00 0x7f3d64150600 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800b2c0 tx=0x7f3d5801b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:23.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d4bfff640 1 -- 172.21.15.107:0/930104958 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d58006ee0 con 0x7f3d64070b00 2023-10-25T00:09:23.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d4bfff640 1 -- 172.21.15.107:0/930104958 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d580a7e90 con 0x7f3d64070b00 2023-10-25T00:09:23.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d4bfff640 1 -- 172.21.15.107:0/930104958 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d580afea0 con 0x7f3d64070b00 2023-10-25T00:09:23.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d69f7d640 1 -- 172.21.15.107:0/930104958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6406b230 con 0x7f3d64070b00 2023-10-25T00:09:23.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d69f7d640 1 -- 172.21.15.107:0/930104958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d640ac320 con 0x7f3d64070b00 2023-10-25T00:09:23.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.306+0000 7f3d69f7d640 1 -- 172.21.15.107:0/930104958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d640746e0 con 0x7f3d64070b00 2023-10-25T00:09:23.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.310+0000 7f3d4bfff640 1 -- 172.21.15.107:0/930104958 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3d580b6040 con 0x7f3d64070b00 2023-10-25T00:09:23.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.310+0000 7f3d4bfff640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d50065e90 0x7f3d50068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:23.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.310+0000 7f3d4bfff640 1 -- 172.21.15.107:0/930104958 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7f3d58121980 con 0x7f3d64070b00 2023-10-25T00:09:23.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.310+0000 7f3d637fe640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d50065e90 0x7f3d50068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:23.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.310+0000 7f3d637fe640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d50065e90 0x7f3d50068350 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3d54000f70 tx=0x7f3d54009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:23.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.314+0000 7f3d4bfff640 1 -- 172.21.15.107:0/930104958 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3d580ee3a0 con 0x7f3d64070b00 2023-10-25T00:09:23.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.538+0000 7f3d69f7d640 1 -- 172.21.15.107:0/930104958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d640c1a50 con 0x7f3d64070b00 2023-10-25T00:09:23.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.538+0000 7f3d4bfff640 1 -- 172.21.15.107:0/930104958 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f3d580f19c0 con 0x7f3d64070b00 2023-10-25T00:09:23.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:23.539 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":149,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:23.540 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 149 2023-10-25T00:09:23.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 -- 172.21.15.107:0/930104958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d50065e90 msgr2=0x7f3d50068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:23.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d50065e90 0x7f3d50068350 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3d54000f70 tx=0x7f3d54009040 comp rx=0 tx=0).stop 2023-10-25T00:09:23.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 -- 172.21.15.107:0/930104958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d64070b00 msgr2=0x7f3d64150600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:23.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d64070b00 0x7f3d64150600 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800b2c0 tx=0x7f3d5801b970 comp rx=0 tx=0).stop 2023-10-25T00:09:23.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 -- 172.21.15.107:0/930104958 shutdown_connections 2023-10-25T00:09:23.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d6414eb70 0x7f3d641ec190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:23.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d64070b00 0x7f3d64150600 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3d50065e90 0x7f3d50068350 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 --2- 172.21.15.107:0/930104958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d6406df30 0x7f3d641e8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 -- 172.21.15.107:0/930104958 >> 172.21.15.107:0/930104958 conn(0x7f3d640c39a0 msgr2=0x7f3d640c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 -- 172.21.15.107:0/930104958 shutdown_connections 2023-10-25T00:09:23.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:23.542+0000 7f3d69f7d640 1 -- 172.21.15.107:0/930104958 wait complete. 2023-10-25T00:09:23.562 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=7512 2023-10-25T00:09:23.562 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.c 2023-10-25T00:09:24.563 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2023-10-25T00:09:24.563 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:09:24.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:09:24.566 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:09:24.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:09:24.634 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:09:24.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf66a6640 1 Processor -- start 2023-10-25T00:09:24.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf66a6640 1 -- start start 2023-10-25T00:09:24.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf66a6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf0065bc0 0x7f7cf0158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:24.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf66a6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7cf006a120 0x7f7cf00aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:24.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf66a6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf006a5b0 con 0x7f7cf0069d40 2023-10-25T00:09:24.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf66a6640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf00ac650 con 0x7f7cf006a120 2023-10-25T00:09:24.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf66a6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf00a9e00 con 0x7f7cf0065bc0 2023-10-25T00:09:24.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7cf0069d40 0x7f7cf00a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53598/0 (socket says 172.21.15.107:53598) 2023-10-25T00:09:24.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceffff640 1 -- 172.21.15.107:0/230982289 learned_addr learned my addr 172.21.15.107:0/230982289 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:09:24.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf4c43640 1 --2- 172.21.15.107:0/230982289 >> v2:172.21.15.110:3300/0 conn(0x7f7cf006a120 0x7f7cf00aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:24.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cef7fe640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf0065bc0 0x7f7cf0158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:24.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701756065 0 0) 0x7f7ce0001120 con 0x7f7cf0069d40 2023-10-25T00:09:24.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd8003530 con 0x7f7cf0069d40 2023-10-25T00:09:24.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2377453170 0 0) 0x7f7cd8003530 con 0x7f7cf0069d40 2023-10-25T00:09:24.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf0065bc0 msgr2=0x7f7cf0158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:24.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf0065bc0 0x7f7cf0158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:24.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 >> v2:172.21.15.110:3300/0 conn(0x7f7cf006a120 msgr2=0x7f7cf00aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:24.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 --2- 172.21.15.107:0/230982289 >> v2:172.21.15.110:3300/0 conn(0x7f7cf006a120 0x7f7cf00aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:24.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf006cea0 con 0x7f7cf0069d40 2023-10-25T00:09:24.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f7ce0001810 con 0x7f7cf0069d40 2023-10-25T00:09:24.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 >> v1:172.21.15.107:6789/0 conn(0x7f7cf0069d40 legacy=0x7f7cf00a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:24.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cd8003980 0x7f7cd8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:24.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf0065bc0 0x7f7cd80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cd8004d10 0x7f7cd8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd800b990 con 0x7f7cd8004d10 2023-10-25T00:09:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd800bb00 con 0x7f7cd8003980 2023-10-25T00:09:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd800bcc0 con 0x7f7cf0065bc0 2023-10-25T00:09:24.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf4c43640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cd8004d10 0x7f7cd8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cef7fe640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf0065bc0 0x7f7cd80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7ceffff640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cd8003980 0x7f7cd8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf4c43640 1 -- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf0065bc0 msgr2=0x7f7cd80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf4c43640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf0065bc0 0x7f7cd80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf4c43640 1 -- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cd8003980 msgr2=0x7f7cd8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:09:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf4c43640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cd8003980 0x7f7cd8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cf4c43640 1 -- 172.21.15.107:0/230982289 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd8003530 con 0x7f7cd8004d10 2023-10-25T00:09:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.694+0000 7f7cef7fe640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf0065bc0 0x7f7cd80047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:09:24.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.698+0000 7f7cf4c43640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cd8004d10 0x7f7cd8005190 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009fc0 tx=0x7f7ce801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de289623fc9346c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:24.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.698+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80a5cb0 con 0x7f7cd8004d10 2023-10-25T00:09:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.698+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce800f040 con 0x7f7cd8004d10 2023-10-25T00:09:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.698+0000 7f7ceeffd640 1 -- 172.21.15.107:0/230982289 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80ad680 con 0x7f7cd8004d10 2023-10-25T00:09:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.698+0000 7f7cf66a6640 1 -- 172.21.15.107:0/230982289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cd8004d10 msgr2=0x7f7cd8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.698+0000 7f7cf66a6640 1 --2- 172.21.15.107:0/230982289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cd8004d10 0x7f7cd8005190 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009fc0 tx=0x7f7ce801cca0 comp rx=0 tx=0).stop 2023-10-25T00:09:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.698+0000 7f7cf4c43640 1 -- 172.21.15.107:0/230982289 reap_dead start 2023-10-25T00:09:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.698+0000 7f7cf66a6640 1 -- 172.21.15.107:0/230982289 shutdown_connections 2023-10-25T00:09:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.698+0000 7f7cf66a6640 1 -- 172.21.15.107:0/230982289 >> 172.21.15.107:0/230982289 conn(0x7f7cf00c65c0 msgr2=0x7f7cf00c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.698+0000 7f7cf66a6640 1 -- 172.21.15.107:0/230982289 shutdown_connections 2023-10-25T00:09:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.698+0000 7f7cf66a6640 1 -- 172.21.15.107:0/230982289 wait complete. 2023-10-25T00:09:24.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cf66a6640 1 Processor -- start 2023-10-25T00:09:24.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cf66a6640 1 -- start start 2023-10-25T00:09:24.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cf66a6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cd8004d10 0x7f7cf01545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:24.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cf66a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf00c3d60 0x7f7cf00bf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:24.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cf66a6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf01ea890 0x7f7cf00c1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:24.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cf66a6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01ead90 con 0x7f7cf00c3d60 2023-10-25T00:09:24.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cf66a6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf01537e0 con 0x7f7cd8004d10 2023-10-25T00:09:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cf66a6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf0153ac0 con 0x7f7cf01ea890 2023-10-25T00:09:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf00c3d60 0x7f7cf00bf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf00c3d60 0x7f7cf00bf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49384/0 (socket says 172.21.15.107:49384) 2023-10-25T00:09:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cef7fe640 1 -- 172.21.15.107:0/2840512641 learned_addr learned my addr 172.21.15.107:0/2840512641 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:09:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cf4c43640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf01ea890 0x7f7cf00c1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:24.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7ceffff640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cd8004d10 0x7f7cf01545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:24.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cef7fe640 1 -- 172.21.15.107:0/2840512641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf01ea890 msgr2=0x7f7cf00c1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:24.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cef7fe640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf01ea890 0x7f7cf00c1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:24.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cef7fe640 1 -- 172.21.15.107:0/2840512641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cd8004d10 msgr2=0x7f7cf01545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:24.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cef7fe640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cd8004d10 0x7f7cf01545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cef7fe640 1 -- 172.21.15.107:0/2840512641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf0153da0 con 0x7f7cf00c3d60 2023-10-25T00:09:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cef7fe640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf00c3d60 0x7f7cf00bf5b0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009fd0 tx=0x7f7ce4011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cecff9640 1 -- 172.21.15.107:0/2840512641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4004220 con 0x7f7cf00c3d60 2023-10-25T00:09:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cecff9640 1 -- 172.21.15.107:0/2840512641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce4004380 con 0x7f7cf00c3d60 2023-10-25T00:09:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cf66a6640 1 -- 172.21.15.107:0/2840512641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf0154080 con 0x7f7cf00c3d60 2023-10-25T00:09:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cf66a6640 1 -- 172.21.15.107:0/2840512641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf00c2020 con 0x7f7cf00c3d60 2023-10-25T00:09:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.702+0000 7f7cecff9640 1 -- 172.21.15.107:0/2840512641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce400f9e0 con 0x7f7cf00c3d60 2023-10-25T00:09:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.706+0000 7f7cecff9640 1 -- 172.21.15.107:0/2840512641 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7ce400fb40 con 0x7f7cf00c3d60 2023-10-25T00:09:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.706+0000 7f7cecff9640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7cdc065e90 0x7f7cdc068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:09:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.706+0000 7f7cf66a6640 1 -- 172.21.15.107:0/2840512641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf0154ae0 con 0x7f7cf00c3d60 2023-10-25T00:09:24.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.706+0000 7f7cecff9640 1 -- 172.21.15.107:0/2840512641 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7f7ce4084aa0 con 0x7f7cf00c3d60 2023-10-25T00:09:24.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.706+0000 7f7ceffff640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7cdc065e90 0x7f7cdc068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:09:24.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.706+0000 7f7ceffff640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7cdc065e90 0x7f7cdc068350 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0001ec0 tx=0x7f7ce0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:09:24.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.710+0000 7f7cecff9640 1 -- 172.21.15.107:0/2840512641 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7ce4022080 con 0x7f7cf00c3d60 2023-10-25T00:09:24.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 -- 172.21.15.107:0/2840512641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cf00746e0 con 0x7f7cf00c3d60 2023-10-25T00:09:24.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cecff9640 1 -- 172.21.15.107:0/2840512641 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f7ce4051440 con 0x7f7cf00c3d60 2023-10-25T00:09:24.926 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:09:24.926 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:09:24.927 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 150 2023-10-25T00:09:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 -- 172.21.15.107:0/2840512641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7cdc065e90 msgr2=0x7f7cdc068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7cdc065e90 0x7f7cdc068350 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0001ec0 tx=0x7f7ce0009040 comp rx=0 tx=0).stop 2023-10-25T00:09:24.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 -- 172.21.15.107:0/2840512641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf00c3d60 msgr2=0x7f7cf00bf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:09:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf00c3d60 0x7f7cf00bf5b0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009fd0 tx=0x7f7ce4011b30 comp rx=0 tx=0).stop 2023-10-25T00:09:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 -- 172.21.15.107:0/2840512641 shutdown_connections 2023-10-25T00:09:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7cdc065e90 0x7f7cdc068350 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:24.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7cf01ea890 0x7f7cf00c1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:24.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cf00c3d60 0x7f7cf00bf5b0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:24.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 --2- 172.21.15.107:0/2840512641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7cd8004d10 0x7f7cf01545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:09:24.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 -- 172.21.15.107:0/2840512641 >> 172.21.15.107:0/2840512641 conn(0x7f7cf00c65c0 msgr2=0x7f7cf00c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:09:24.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 -- 172.21.15.107:0/2840512641 shutdown_connections 2023-10-25T00:09:24.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:09:24.926+0000 7f7cf66a6640 1 -- 172.21.15.107:0/2840512641 wait complete. 2023-10-25T00:09:24.948 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2023-10-25T00:09:24.949 INFO:tasks.mds_thrash.fs.[b]:waiting for 82 secs before thrashing 2023-10-25T00:10:20.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:20.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a91ea640 1 Processor -- start 2023-10-25T00:10:20.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a91ea640 1 -- start start 2023-10-25T00:10:20.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a91ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a413dc20 0x7f28a40c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:20.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a91ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f28a40720c0 0x7f28a406df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:20.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a91ea640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a4154860 con 0x7f28a406e470 2023-10-25T00:10:20.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a91ea640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a40c1f10 con 0x7f28a40720c0 2023-10-25T00:10:20.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a91ea640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a4154b20 con 0x7f28a413dc20 2023-10-25T00:10:20.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a359e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f28a406e470 0x7f28a4157e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53036/0 (socket says 172.21.15.107:53036) 2023-10-25T00:10:20.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a359e640 1 -- 172.21.15.107:0/94418733 learned_addr learned my addr 172.21.15.107:0/94418733 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:20.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a2d9d640 1 --2- 172.21.15.107:0/94418733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a413dc20 0x7f28a40c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:20.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a259c640 1 --2- 172.21.15.107:0/94418733 >> v2:172.21.15.110:3300/0 conn(0x7f28a40720c0 0x7f28a406df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:20.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155115391 0 0) 0x7f28a4154860 con 0x7f28a406e470 2023-10-25T00:10:20.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f288c003530 con 0x7f28a406e470 2023-10-25T00:10:20.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1416462014 0 0) 0x7f288c003530 con 0x7f28a406e470 2023-10-25T00:10:20.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a413dc20 msgr2=0x7f28a40c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:20.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 --2- 172.21.15.107:0/94418733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a413dc20 0x7f28a40c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:20.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 >> v2:172.21.15.110:3300/0 conn(0x7f28a40720c0 msgr2=0x7f28a406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:20.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 --2- 172.21.15.107:0/94418733 >> v2:172.21.15.110:3300/0 conn(0x7f28a40720c0 0x7f28a406df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:20.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a4154e00 con 0x7f28a406e470 2023-10-25T00:10:20.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f289c001a00 con 0x7f28a406e470 2023-10-25T00:10:20.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 >> v1:172.21.15.107:6789/0 conn(0x7f28a406e470 legacy=0x7f28a4157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:20.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 --2- 172.21.15.107:0/94418733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f288c003980 0x7f288c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:20.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 --2- 172.21.15.107:0/94418733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f288c0047d0 0x7f288c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:20.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 --2- 172.21.15.107:0/94418733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a413dc20 0x7f288c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:20.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a40c1f10 con 0x7f288c0047d0 2023-10-25T00:10:20.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a4154b20 con 0x7f288c003980 2023-10-25T00:10:20.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288c00ba10 con 0x7f28a413dc20 2023-10-25T00:10:20.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a259c640 1 --2- 172.21.15.107:0/94418733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f288c0047d0 0x7f288c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:20.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a2d9d640 1 --2- 172.21.15.107:0/94418733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f288c003980 0x7f288c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:20.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a259c640 1 -- 172.21.15.107:0/94418733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a413dc20 msgr2=0x7f288c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:20.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a259c640 1 --2- 172.21.15.107:0/94418733 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a413dc20 0x7f288c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:20.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a259c640 1 -- 172.21.15.107:0/94418733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f288c003980 msgr2=0x7f288c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:20.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a259c640 1 --2- 172.21.15.107:0/94418733 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f288c003980 0x7f288c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:20.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.881+0000 7f28a259c640 1 -- 172.21.15.107:0/94418733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c003530 con 0x7f288c0047d0 2023-10-25T00:10:20.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a259c640 1 --2- 172.21.15.107:0/94418733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f288c0047d0 0x7f288c004c50 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f2898009ae0 tx=0x7f289801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ad7b712a5338cc server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:20.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28980a6470 con 0x7f288c0047d0 2023-10-25T00:10:20.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28980a65d0 con 0x7f288c0047d0 2023-10-25T00:10:20.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a1d9b640 1 -- 172.21.15.107:0/94418733 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28980a6980 con 0x7f288c0047d0 2023-10-25T00:10:20.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a91ea640 1 -- 172.21.15.107:0/94418733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f288c0047d0 msgr2=0x7f288c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:20.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a91ea640 1 --2- 172.21.15.107:0/94418733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f288c0047d0 0x7f288c004c50 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f2898009ae0 tx=0x7f289801ca90 comp rx=0 tx=0).stop 2023-10-25T00:10:20.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a359e640 1 -- 172.21.15.107:0/94418733 reap_dead start 2023-10-25T00:10:20.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a91ea640 1 -- 172.21.15.107:0/94418733 shutdown_connections 2023-10-25T00:10:20.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a91ea640 1 -- 172.21.15.107:0/94418733 >> 172.21.15.107:0/94418733 conn(0x7f28a40c39a0 msgr2=0x7f28a40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:20.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a91ea640 1 -- 172.21.15.107:0/94418733 shutdown_connections 2023-10-25T00:10:20.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a91ea640 1 -- 172.21.15.107:0/94418733 wait complete. 2023-10-25T00:10:20.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.885+0000 7f28a91ea640 1 Processor -- start 2023-10-25T00:10:20.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a91ea640 1 -- start start 2023-10-25T00:10:20.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a91ea640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28a4070140 0x7f28a41ecb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:20.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a91ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a40abb80 0x7f28a41ed040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:20.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a91ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a41375a0 0x7f28a4137a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:20.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a91ea640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a41f0b40 con 0x7f28a40abb80 2023-10-25T00:10:20.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a91ea640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a413cdf0 con 0x7f28a4070140 2023-10-25T00:10:20.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a91ea640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a415da50 con 0x7f28a41375a0 2023-10-25T00:10:20.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a40abb80 0x7f28a41ed040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:20.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a40abb80 0x7f28a41ed040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50472/0 (socket says 172.21.15.107:50472) 2023-10-25T00:10:20.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a259c640 1 -- 172.21.15.107:0/4029819707 learned_addr learned my addr 172.21.15.107:0/4029819707 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:20.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a359e640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a41375a0 0x7f28a4137a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:20.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a2d9d640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28a4070140 0x7f28a41ecb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:20.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a259c640 1 -- 172.21.15.107:0/4029819707 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a41375a0 msgr2=0x7f28a4137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:20.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a259c640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a41375a0 0x7f28a4137a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:20.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a259c640 1 -- 172.21.15.107:0/4029819707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28a4070140 msgr2=0x7f28a41ecb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a259c640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28a4070140 0x7f28a41ecb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a259c640 1 -- 172.21.15.107:0/4029819707 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a41f0f60 con 0x7f28a40abb80 2023-10-25T00:10:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a259c640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a40abb80 0x7f28a41ed040 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f289801c8c0 tx=0x7f2898019830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f288b7fe640 1 -- 172.21.15.107:0/4029819707 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289800f000 con 0x7f28a40abb80 2023-10-25T00:10:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f288b7fe640 1 -- 172.21.15.107:0/4029819707 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28980a6dc0 con 0x7f28a40abb80 2023-10-25T00:10:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a91ea640 1 -- 172.21.15.107:0/4029819707 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a40ac420 con 0x7f28a40abb80 2023-10-25T00:10:20.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f288b7fe640 1 -- 172.21.15.107:0/4029819707 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28980a7670 con 0x7f28a40abb80 2023-10-25T00:10:20.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a91ea640 1 -- 172.21.15.107:0/4029819707 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a41f0480 con 0x7f28a40abb80 2023-10-25T00:10:20.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f28a91ea640 1 -- 172.21.15.107:0/4029819707 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a40746e0 con 0x7f28a40abb80 2023-10-25T00:10:20.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f288b7fe640 1 -- 172.21.15.107:0/4029819707 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f28980a77d0 con 0x7f28a40abb80 2023-10-25T00:10:20.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.889+0000 7f288b7fe640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2890065e90 0x7f2890068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:20.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.893+0000 7f28a2d9d640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2890065e90 0x7f2890068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:20.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.893+0000 7f288b7fe640 1 -- 172.21.15.107:0/4029819707 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7f289811e2d0 con 0x7f28a40abb80 2023-10-25T00:10:20.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.893+0000 7f28a2d9d640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2890065e90 0x7f2890068350 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2894009950 tx=0x7f2894008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:20.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:20.897+0000 7f288b7fe640 1 -- 172.21.15.107:0/4029819707 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f28980eac70 con 0x7f28a40abb80 2023-10-25T00:10:21.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 -- 172.21.15.107:0/4029819707 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28a40c1190 con 0x7f28a40abb80 2023-10-25T00:10:21.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f288b7fe640 1 -- 172.21.15.107:0/4029819707 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f28980be070 con 0x7f28a40abb80 2023-10-25T00:10:21.135 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:21.135 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:21.135 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 150 2023-10-25T00:10:21.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 -- 172.21.15.107:0/4029819707 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2890065e90 msgr2=0x7f2890068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:21.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2890065e90 0x7f2890068350 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f2894009950 tx=0x7f2894008040 comp rx=0 tx=0).stop 2023-10-25T00:10:21.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 -- 172.21.15.107:0/4029819707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a40abb80 msgr2=0x7f28a41ed040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:21.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a40abb80 0x7f28a41ed040 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f289801c8c0 tx=0x7f2898019830 comp rx=0 tx=0).stop 2023-10-25T00:10:21.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 -- 172.21.15.107:0/4029819707 shutdown_connections 2023-10-25T00:10:21.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2890065e90 0x7f2890068350 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a41375a0 0x7f28a4137a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a40abb80 0x7f28a41ed040 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 --2- 172.21.15.107:0/4029819707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28a4070140 0x7f28a41ecb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 -- 172.21.15.107:0/4029819707 >> 172.21.15.107:0/4029819707 conn(0x7f28a40c39a0 msgr2=0x7f28a413bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 -- 172.21.15.107:0/4029819707 shutdown_connections 2023-10-25T00:10:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.133+0000 7f28a91ea640 1 -- 172.21.15.107:0/4029819707 wait complete. 2023-10-25T00:10:21.155 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2023-10-25T00:10:21.155 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:10:21.155 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:10:21.167 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:10:21.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:21.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c36c1640 1 Processor -- start 2023-10-25T00:10:21.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c36c1640 1 -- start start 2023-10-25T00:10:21.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c36c1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9bc0c70e0 0x7fe9bc0bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:21.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c36c1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc0c1ca0 con 0x7fe9bc0c2130 2023-10-25T00:10:21.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c36c1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc0c10c0 con 0x7fe9bc156770 2023-10-25T00:10:21.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c36c1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9bc0c27e0 con 0x7fe9bc0c70e0 2023-10-25T00:10:21.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c1c5e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe9bc0c2130 0x7fe9bc15d130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53044/0 (socket says 172.21.15.107:53044) 2023-10-25T00:10:21.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c1c5e640 1 -- 172.21.15.107:0/2094311294 learned_addr learned my addr 172.21.15.107:0/2094311294 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:21.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c0c5c640 1 --2- 172.21.15.107:0/2094311294 >> v2:172.21.15.110:3300/0 conn(0x7fe9bc0c70e0 0x7fe9bc0bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:21.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620049667 0 0) 0x7fe9bc0c1ca0 con 0x7fe9bc0c2130 2023-10-25T00:10:21.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a8003530 con 0x7fe9bc0c2130 2023-10-25T00:10:21.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3169231270 0 0) 0x7fe9a8003530 con 0x7fe9bc0c2130 2023-10-25T00:10:21.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 >> v2:172.21.15.110:3300/0 conn(0x7fe9bc0c70e0 msgr2=0x7fe9bc0bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:21.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 --2- 172.21.15.107:0/2094311294 >> v2:172.21.15.110:3300/0 conn(0x7fe9bc0c70e0 0x7fe9bc0bdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 >> v1:172.21.15.110:6789/0 conn(0x7fe9bc156770 legacy=0x7fe9bc0ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:21.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc0c13a0 con 0x7fe9bc0c2130 2023-10-25T00:10:21.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fe9b4001a00 con 0x7fe9bc0c2130 2023-10-25T00:10:21.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 >> v1:172.21.15.107:6789/0 conn(0x7fe9bc0c2130 legacy=0x7fe9bc15d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:21.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a8003980 0x7fe9a8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:21.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a80047d0 0x7fe9a8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:21.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a8005390 0x7fe9a800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:21.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a800c1f0 con 0x7fe9a8003980 2023-10-25T00:10:21.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a800c360 con 0x7fe9a80047d0 2023-10-25T00:10:21.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a800c520 con 0x7fe9a8005390 2023-10-25T00:10:21.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c145d640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a8003980 0x7fe9a8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:21.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c0c5c640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a80047d0 0x7fe9a8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:21.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c1c5e640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a8005390 0x7fe9a800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:21.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c145d640 1 -- 172.21.15.107:0/2094311294 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a8005390 msgr2=0x7fe9a800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:21.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c145d640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a8005390 0x7fe9a800b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c145d640 1 -- 172.21.15.107:0/2094311294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a80047d0 msgr2=0x7fe9a8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:21.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c145d640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a80047d0 0x7fe9a8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c145d640 1 -- 172.21.15.107:0/2094311294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a8003530 con 0x7fe9a8003980 2023-10-25T00:10:21.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c1c5e640 1 -- 172.21.15.107:0/2094311294 reap_dead start 2023-10-25T00:10:21.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.345+0000 7fe9c0c5c640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9a80047d0 0x7fe9a8004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:10:21.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9c145d640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a8003980 0x7fe9a8003da0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8001780 tx=0x7fe9b801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3de0f1b7fef2c52 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:21.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b80050d0 con 0x7fe9a8003980 2023-10-25T00:10:21.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b80a7050 con 0x7fe9a8003980 2023-10-25T00:10:21.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9b3fff640 1 -- 172.21.15.107:0/2094311294 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8005490 con 0x7fe9a8003980 2023-10-25T00:10:21.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9c36c1640 1 -- 172.21.15.107:0/2094311294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a8003980 msgr2=0x7fe9a8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:21.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9c36c1640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a8003980 0x7fe9a8003da0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8001780 tx=0x7fe9b801c3d0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9c36c1640 1 -- 172.21.15.107:0/2094311294 shutdown_connections 2023-10-25T00:10:21.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9c36c1640 1 --2- 172.21.15.107:0/2094311294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a8003980 0x7fe9a8003da0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9c36c1640 1 -- 172.21.15.107:0/2094311294 >> 172.21.15.107:0/2094311294 conn(0x7fe9bc0c65c0 msgr2=0x7fe9bc0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:21.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9c36c1640 1 -- 172.21.15.107:0/2094311294 shutdown_connections 2023-10-25T00:10:21.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9c36c1640 1 -- 172.21.15.107:0/2094311294 wait complete. 2023-10-25T00:10:21.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9c36c1640 1 Processor -- start 2023-10-25T00:10:21.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.349+0000 7fe9c36c1640 1 -- start start 2023-10-25T00:10:21.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c36c1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a8003980 0x7fe9bc1edfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:21.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c36c1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9bc151420 0x7fe9bc0a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:21.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c36c1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9bc151970 0x7fe9bc0a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:21.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c36c1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc09f3f0 con 0x7fe9bc151970 2023-10-25T00:10:21.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c36c1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc1edbd0 con 0x7fe9bc151420 2023-10-25T00:10:21.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c36c1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc1ed650 con 0x7fe9a8003980 2023-10-25T00:10:21.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c1c5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9bc151970 0x7fe9bc0a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:21.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c1c5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9bc151970 0x7fe9bc0a6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50488/0 (socket says 172.21.15.107:50488) 2023-10-25T00:10:21.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c1c5e640 1 -- 172.21.15.107:0/1600921413 learned_addr learned my addr 172.21.15.107:0/1600921413 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:21.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c145d640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a8003980 0x7fe9bc1edfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:21.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c0c5c640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9bc151420 0x7fe9bc0a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:21.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c1c5e640 1 -- 172.21.15.107:0/1600921413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a8003980 msgr2=0x7fe9bc1edfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:21.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c1c5e640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a8003980 0x7fe9bc1edfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c1c5e640 1 -- 172.21.15.107:0/1600921413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9bc151420 msgr2=0x7fe9bc0a6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:21.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c1c5e640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9bc151420 0x7fe9bc0a6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c1c5e640 1 -- 172.21.15.107:0/1600921413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc1ed8f0 con 0x7fe9bc151970 2023-10-25T00:10:21.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c1c5e640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9bc151970 0x7fe9bc0a6ec0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4001590 tx=0x7fe9b400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:21.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9b1ffb640 1 -- 172.21.15.107:0/1600921413 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b400de90 con 0x7fe9bc151970 2023-10-25T00:10:21.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9b1ffb640 1 -- 172.21.15.107:0/1600921413 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b400bb20 con 0x7fe9bc151970 2023-10-25T00:10:21.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9b1ffb640 1 -- 172.21.15.107:0/1600921413 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b400bed0 con 0x7fe9bc151970 2023-10-25T00:10:21.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c36c1640 1 -- 172.21.15.107:0/1600921413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9bc1ee930 con 0x7fe9bc151970 2023-10-25T00:10:21.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c36c1640 1 -- 172.21.15.107:0/1600921413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9bc1ee520 con 0x7fe9bc151970 2023-10-25T00:10:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c36c1640 1 -- 172.21.15.107:0/1600921413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9bc1ee770 con 0x7fe9bc151970 2023-10-25T00:10:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9b1ffb640 1 -- 172.21.15.107:0/1600921413 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe9b4007070 con 0x7fe9bc151970 2023-10-25T00:10:21.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9b1ffb640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe9a0065e60 0x7fe9a0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:21.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9b1ffb640 1 -- 172.21.15.107:0/1600921413 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7fe9b40847a0 con 0x7fe9bc151970 2023-10-25T00:10:21.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c145d640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe9a0065e60 0x7fe9a0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:21.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.353+0000 7fe9c145d640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe9a0065e60 0x7fe9a0068320 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8001750 tx=0x7fe9b80011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:21.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.357+0000 7fe9b1ffb640 1 -- 172.21.15.107:0/1600921413 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe9b40510b0 con 0x7fe9bc151970 2023-10-25T00:10:21.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.549+0000 7fe9c36c1640 1 -- 172.21.15.107:0/1600921413 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9bc0c6a40 con 0x7fe9bc151970 2023-10-25T00:10:21.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.549+0000 7fe9b1ffb640 1 -- 172.21.15.107:0/1600921413 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5777 (secure 0 0 0) 0x7fe9b40546d0 con 0x7fe9bc151970 2023-10-25T00:10:21.553 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:21.553 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:21.554 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 150 2023-10-25T00:10:21.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 -- 172.21.15.107:0/1600921413 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe9a0065e60 msgr2=0x7fe9a0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:21.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe9a0065e60 0x7fe9a0068320 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8001750 tx=0x7fe9b80011a0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 -- 172.21.15.107:0/1600921413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9bc151970 msgr2=0x7fe9bc0a6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:21.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9bc151970 0x7fe9bc0a6ec0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4001590 tx=0x7fe9b400b4a0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 -- 172.21.15.107:0/1600921413 shutdown_connections 2023-10-25T00:10:21.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9bc151970 0x7fe9bc0a6ec0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe9bc151420 0x7fe9bc0a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe9a0065e60 0x7fe9a0068320 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 --2- 172.21.15.107:0/1600921413 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe9a8003980 0x7fe9bc1edfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:21.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 -- 172.21.15.107:0/1600921413 >> 172.21.15.107:0/1600921413 conn(0x7fe9bc0c65c0 msgr2=0x7fe9bc0c3040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:21.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 -- 172.21.15.107:0/1600921413 shutdown_connections 2023-10-25T00:10:21.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:21.553+0000 7fe9c36c1640 1 -- 172.21.15.107:0/1600921413 wait complete. 2023-10-25T00:10:21.572 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:10:23.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f88a1ae7640 1 Processor -- start 2023-10-25T00:10:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f88a1ae7640 1 -- start start 2023-10-25T00:10:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f88a1ae7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c14b600 0x7f889c151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f88a1ae7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c0c1860 con 0x7f889c06a230 2023-10-25T00:10:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f88a1ae7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c154d50 con 0x7f889c06af70 2023-10-25T00:10:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f88a1ae7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c1524f0 con 0x7f889c14b600 2023-10-25T00:10:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f889c06a230 0x7f889c1551e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51962/0 (socket says 172.21.15.107:51962) 2023-10-25T00:10:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889bfff640 1 -- 172.21.15.107:0/760354581 learned_addr learned my addr 172.21.15.107:0/760354581 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889b7fe640 1 --2- 172.21.15.107:0/760354581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c14b600 0x7f889c151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305759452 0 0) 0x7f889c0c1860 con 0x7f889c06a230 2023-10-25T00:10:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8888003530 con 0x7f889c06a230 2023-10-25T00:10:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2265847450 0 0) 0x7f8888003530 con 0x7f889c06a230 2023-10-25T00:10:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c14b600 msgr2=0x7f889c151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 --2- 172.21.15.107:0/760354581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c14b600 0x7f889c151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 >> v1:172.21.15.110:6789/0 conn(0x7f889c06af70 legacy=0x7f889c06c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c14b9e0 con 0x7f889c06a230 2023-10-25T00:10:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f8894001a00 con 0x7f889c06a230 2023-10-25T00:10:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 >> v1:172.21.15.107:6789/0 conn(0x7f889c06a230 legacy=0x7f889c1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 --2- 172.21.15.107:0/760354581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 0x7f8888003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 --2- 172.21.15.107:0/760354581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88880047d0 0x7f8888004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 --2- 172.21.15.107:0/760354581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c14b600 0x7f88880051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c1524f0 con 0x7f88880039b0 2023-10-25T00:10:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888800dcd0 con 0x7f88880047d0 2023-10-25T00:10:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888800de90 con 0x7f889c14b600 2023-10-25T00:10:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889b7fe640 1 --2- 172.21.15.107:0/760354581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 0x7f8888003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889b7fe640 1 -- 172.21.15.107:0/760354581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c14b600 msgr2=0x7f88880051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889b7fe640 1 --2- 172.21.15.107:0/760354581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c14b600 0x7f88880051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889b7fe640 1 -- 172.21.15.107:0/760354581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88880047d0 msgr2=0x7f8888004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889affd640 1 --2- 172.21.15.107:0/760354581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88880047d0 0x7f8888004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889b7fe640 1 --2- 172.21.15.107:0/760354581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f88880047d0 0x7f8888004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.709+0000 7f889b7fe640 1 -- 172.21.15.107:0/760354581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8888003530 con 0x7f88880039b0 2023-10-25T00:10:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f889b7fe640 1 --2- 172.21.15.107:0/760354581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 0x7f8888003dd0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f888c009ae0 tx=0x7f888c01ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a849b58c5de740 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c0a6450 con 0x7f88880039b0 2023-10-25T00:10:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888c0a65b0 con 0x7f88880039b0 2023-10-25T00:10:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f889a7fc640 1 -- 172.21.15.107:0/760354581 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c0a6960 con 0x7f88880039b0 2023-10-25T00:10:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 -- 172.21.15.107:0/760354581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 msgr2=0x7f8888003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 --2- 172.21.15.107:0/760354581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 0x7f8888003dd0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f888c009ae0 tx=0x7f888c01ca60 comp rx=0 tx=0).stop 2023-10-25T00:10:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f889bfff640 1 -- 172.21.15.107:0/760354581 reap_dead start 2023-10-25T00:10:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 -- 172.21.15.107:0/760354581 shutdown_connections 2023-10-25T00:10:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 -- 172.21.15.107:0/760354581 >> 172.21.15.107:0/760354581 conn(0x7f889c0c2950 msgr2=0x7f889c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 -- 172.21.15.107:0/760354581 shutdown_connections 2023-10-25T00:10:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 -- 172.21.15.107:0/760354581 wait complete. 2023-10-25T00:10:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 Processor -- start 2023-10-25T00:10:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 -- start start 2023-10-25T00:10:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 0x7f889c1e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c140ea0 0x7f889c0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f889c1e63a0 0x7f889c1e8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c149420 con 0x7f88880039b0 2023-10-25T00:10:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c148ec0 con 0x7f889c1e63a0 2023-10-25T00:10:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f88a1ae7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889c0c57e0 con 0x7f889c140ea0 2023-10-25T00:10:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f889b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 0x7f889c1e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f889b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 0x7f889c1e5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39068/0 (socket says 172.21.15.107:39068) 2023-10-25T00:10:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f889b7fe640 1 -- 172.21.15.107:0/1248866739 learned_addr learned my addr 172.21.15.107:0/1248866739 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f889affd640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c140ea0 0x7f889c0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.713+0000 7f889bfff640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f889c1e63a0 0x7f889c1e8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f889b7fe640 1 -- 172.21.15.107:0/1248866739 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c140ea0 msgr2=0x7f889c0717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f889b7fe640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c140ea0 0x7f889c0717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f889b7fe640 1 -- 172.21.15.107:0/1248866739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f889c1e63a0 msgr2=0x7f889c1e8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f889b7fe640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f889c1e63a0 0x7f889c1e8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f889b7fe640 1 -- 172.21.15.107:0/1248866739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c1e89e0 con 0x7f88880039b0 2023-10-25T00:10:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f889bfff640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f889c1e63a0 0x7f889c1e8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:10:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f889b7fe640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 0x7f889c1e5e60 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f888c01cf40 tx=0x7f8888005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f8893fff640 1 -- 172.21.15.107:0/1248866739 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c0b5070 con 0x7f88880039b0 2023-10-25T00:10:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f8893fff640 1 -- 172.21.15.107:0/1248866739 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888c00f000 con 0x7f88880039b0 2023-10-25T00:10:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f88a1ae7640 1 -- 172.21.15.107:0/1248866739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889c148b50 con 0x7f88880039b0 2023-10-25T00:10:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f8893fff640 1 -- 172.21.15.107:0/1248866739 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c0a6e60 con 0x7f88880039b0 2023-10-25T00:10:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f88a1ae7640 1 -- 172.21.15.107:0/1248866739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c0c5a80 con 0x7f88880039b0 2023-10-25T00:10:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f88a1ae7640 1 -- 172.21.15.107:0/1248866739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889c148ce0 con 0x7f88880039b0 2023-10-25T00:10:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f8893fff640 1 -- 172.21.15.107:0/1248866739 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f888c0b9070 con 0x7f88880039b0 2023-10-25T00:10:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f8893fff640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f887c065e60 0x7f887c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f8893fff640 1 -- 172.21.15.107:0/1248866739 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7f888c11e090 con 0x7f88880039b0 2023-10-25T00:10:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f889affd640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f887c065e60 0x7f887c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.717+0000 7f889affd640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f887c065e60 0x7f887c068320 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f889c073470 tx=0x7f8884009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.721+0000 7f8893fff640 1 -- 172.21.15.107:0/1248866739 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f888c0eaae0 con 0x7f88880039b0 2023-10-25T00:10:23.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.913+0000 7f88a1ae7640 1 -- 172.21.15.107:0/1248866739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f889c0746e0 con 0x7f88880039b0 2023-10-25T00:10:23.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.913+0000 7f8893fff640 1 -- 172.21.15.107:0/1248866739 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f888c0c0020 con 0x7f88880039b0 2023-10-25T00:10:23.915 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:23.916 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:23.916 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 150 2023-10-25T00:10:23.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 -- 172.21.15.107:0/1248866739 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f887c065e60 msgr2=0x7f887c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:23.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f887c065e60 0x7f887c068320 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f889c073470 tx=0x7f8884009040 comp rx=0 tx=0).stop 2023-10-25T00:10:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 -- 172.21.15.107:0/1248866739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 msgr2=0x7f889c1e5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 0x7f889c1e5e60 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f888c01cf40 tx=0x7f8888005800 comp rx=0 tx=0).stop 2023-10-25T00:10:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 -- 172.21.15.107:0/1248866739 shutdown_connections 2023-10-25T00:10:23.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f889c1e63a0 0x7f889c1e8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f887c065e60 0x7f887c068320 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f889c140ea0 0x7f889c0717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 --2- 172.21.15.107:0/1248866739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88880039b0 0x7f889c1e5e60 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:23.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 -- 172.21.15.107:0/1248866739 >> 172.21.15.107:0/1248866739 conn(0x7f889c0c2950 msgr2=0x7f889c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:23.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 -- 172.21.15.107:0/1248866739 shutdown_connections 2023-10-25T00:10:23.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:23.917+0000 7f88a1ae7640 1 -- 172.21.15.107:0/1248866739 wait complete. 2023-10-25T00:10:23.936 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:10:25.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:26.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa9336a640 1 Processor -- start 2023-10-25T00:10:26.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa9336a640 1 -- start start 2023-10-25T00:10:26.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa9336a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffa8c14b600 0x7ffa8c151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:26.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa9336a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c06a230 0x7ffa8c1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:26.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa9336a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa8c0c1860 con 0x7ffa8c06af70 2023-10-25T00:10:26.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa9336a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa8c154d50 con 0x7ffa8c14b600 2023-10-25T00:10:26.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa9336a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c1524f0 con 0x7ffa8c06a230 2023-10-25T00:10:26.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa90905640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ffa8c06af70 0x7ffa8c06c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51978/0 (socket says 172.21.15.107:51978) 2023-10-25T00:10:26.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa90905640 1 -- 172.21.15.107:0/2446435294 learned_addr learned my addr 172.21.15.107:0/2446435294 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:26.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa91907640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c06a230 0x7ffa8c1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:26.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa91106640 1 --2- 172.21.15.107:0/2446435294 >> v2:172.21.15.110:3300/0 conn(0x7ffa8c14b600 0x7ffa8c151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:26.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419876076 0 0) 0x7ffa8c0c1860 con 0x7ffa8c06af70 2023-10-25T00:10:26.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa78005740 con 0x7ffa8c06af70 2023-10-25T00:10:26.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4097118688 0 0) 0x7ffa78005740 con 0x7ffa8c06af70 2023-10-25T00:10:26.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c06a230 msgr2=0x7ffa8c1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:26.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c06a230 0x7ffa8c1551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 >> v2:172.21.15.110:3300/0 conn(0x7ffa8c14b600 msgr2=0x7ffa8c151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:26.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 --2- 172.21.15.107:0/2446435294 >> v2:172.21.15.110:3300/0 conn(0x7ffa8c14b600 0x7ffa8c151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8c14b9e0 con 0x7ffa8c06af70 2023-10-25T00:10:26.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7ffa7c001680 con 0x7ffa8c06af70 2023-10-25T00:10:26.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 >> v1:172.21.15.107:6789/0 conn(0x7ffa8c06af70 legacy=0x7ffa8c06c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:26.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa78005b90 0x7ffa78005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:26.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c06a230 0x7ffa780069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:26.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa78006f20 0x7ffa780073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:26.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c154d50 con 0x7ffa78005b90 2023-10-25T00:10:26.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c1524f0 con 0x7ffa78006f20 2023-10-25T00:10:26.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7800dba0 con 0x7ffa8c06a230 2023-10-25T00:10:26.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa91106640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa78005b90 0x7ffa78005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:26.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa91907640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa78006f20 0x7ffa780073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:26.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa91106640 1 -- 172.21.15.107:0/2446435294 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c06a230 msgr2=0x7ffa780069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:26.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa91106640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c06a230 0x7ffa780069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa91106640 1 -- 172.21.15.107:0/2446435294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa78006f20 msgr2=0x7ffa780073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:26.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa91106640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa78006f20 0x7ffa780073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa91106640 1 -- 172.21.15.107:0/2446435294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa78005740 con 0x7ffa78005b90 2023-10-25T00:10:26.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.069+0000 7ffa91907640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa78006f20 0x7ffa780073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:10:26.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa91106640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa78005b90 0x7ffa78005fb0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009ae0 tx=0x7ffa8801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9433948f3dad6e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:26.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa880a5630 con 0x7ffa78005b90 2023-10-25T00:10:26.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa880a5790 con 0x7ffa78005b90 2023-10-25T00:10:26.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa83fff640 1 -- 172.21.15.107:0/2446435294 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa880a5b40 con 0x7ffa78005b90 2023-10-25T00:10:26.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 -- 172.21.15.107:0/2446435294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa78005b90 msgr2=0x7ffa78005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:26.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 --2- 172.21.15.107:0/2446435294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa78005b90 0x7ffa78005fb0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009ae0 tx=0x7ffa8801ca60 comp rx=0 tx=0).stop 2023-10-25T00:10:26.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa91907640 1 -- 172.21.15.107:0/2446435294 reap_dead start 2023-10-25T00:10:26.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 -- 172.21.15.107:0/2446435294 shutdown_connections 2023-10-25T00:10:26.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 -- 172.21.15.107:0/2446435294 >> 172.21.15.107:0/2446435294 conn(0x7ffa8c0c2950 msgr2=0x7ffa8c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:26.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 -- 172.21.15.107:0/2446435294 shutdown_connections 2023-10-25T00:10:26.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 -- 172.21.15.107:0/2446435294 wait complete. 2023-10-25T00:10:26.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 Processor -- start 2023-10-25T00:10:26.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 -- start start 2023-10-25T00:10:26.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa78005b90 0x7ffa8c154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:26.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c14f050 0x7ffa8c1590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:26.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa8c1458b0 0x7ffa8c1f07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:26.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c147aa0 con 0x7ffa8c1458b0 2023-10-25T00:10:26.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c147200 con 0x7ffa78005b90 2023-10-25T00:10:26.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa9336a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c1474e0 con 0x7ffa8c14f050 2023-10-25T00:10:26.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa91907640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa8c1458b0 0x7ffa8c1f07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:26.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa91907640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa8c1458b0 0x7ffa8c1f07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39076/0 (socket says 172.21.15.107:39076) 2023-10-25T00:10:26.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa91907640 1 -- 172.21.15.107:0/3018748486 learned_addr learned my addr 172.21.15.107:0/3018748486 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:26.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.073+0000 7ffa91106640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa78005b90 0x7ffa8c154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:26.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa90905640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c14f050 0x7ffa8c1590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:26.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa91907640 1 -- 172.21.15.107:0/3018748486 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c14f050 msgr2=0x7ffa8c1590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:26.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa91907640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c14f050 0x7ffa8c1590c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa91907640 1 -- 172.21.15.107:0/3018748486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa78005b90 msgr2=0x7ffa8c154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:26.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa91907640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa78005b90 0x7ffa8c154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa91907640 1 -- 172.21.15.107:0/3018748486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8c1477c0 con 0x7ffa8c1458b0 2023-10-25T00:10:26.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa91907640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa8c1458b0 0x7ffa8c1f07a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c06ebb0 tx=0x7ffa8400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:26.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa81ffb640 1 -- 172.21.15.107:0/3018748486 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8400fe90 con 0x7ffa8c1458b0 2023-10-25T00:10:26.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa81ffb640 1 -- 172.21.15.107:0/3018748486 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa84011040 con 0x7ffa8c1458b0 2023-10-25T00:10:26.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa9336a640 1 -- 172.21.15.107:0/3018748486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8c1f0280 con 0x7ffa8c1458b0 2023-10-25T00:10:26.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa81ffb640 1 -- 172.21.15.107:0/3018748486 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa84017620 con 0x7ffa8c1458b0 2023-10-25T00:10:26.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa9336a640 1 -- 172.21.15.107:0/3018748486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8c146fd0 con 0x7ffa8c1458b0 2023-10-25T00:10:26.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa9336a640 1 -- 172.21.15.107:0/3018748486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa8c1f00e0 con 0x7ffa8c1458b0 2023-10-25T00:10:26.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa81ffb640 1 -- 172.21.15.107:0/3018748486 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ffa840203b0 con 0x7ffa8c1458b0 2023-10-25T00:10:26.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa81ffb640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa70065e60 0x7ffa70068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:26.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa81ffb640 1 -- 172.21.15.107:0/3018748486 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7ffa84084790 con 0x7ffa8c1458b0 2023-10-25T00:10:26.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa91106640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa70065e60 0x7ffa70068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:26.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.077+0000 7ffa91106640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa70065e60 0x7ffa70068320 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ffa880af000 tx=0x7ffa880046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:26.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.081+0000 7ffa81ffb640 1 -- 172.21.15.107:0/3018748486 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ffa8c1f00e0 con 0x7ffa8c1458b0 2023-10-25T00:10:26.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 -- 172.21.15.107:0/3018748486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa8c1f00e0 con 0x7ffa8c1458b0 2023-10-25T00:10:26.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa81ffb640 1 -- 172.21.15.107:0/3018748486 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5777 (secure 0 0 0) 0x7ffa8c1f00e0 con 0x7ffa8c1458b0 2023-10-25T00:10:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:26.271 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 150 2023-10-25T00:10:26.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 -- 172.21.15.107:0/3018748486 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa70065e60 msgr2=0x7ffa70068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:26.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa70065e60 0x7ffa70068320 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ffa880af000 tx=0x7ffa880046d0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 -- 172.21.15.107:0/3018748486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa8c1458b0 msgr2=0x7ffa8c1f07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:26.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa8c1458b0 0x7ffa8c1f07a0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c06ebb0 tx=0x7ffa8400f4a0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 -- 172.21.15.107:0/3018748486 shutdown_connections 2023-10-25T00:10:26.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa8c1458b0 0x7ffa8c1f07a0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa70065e60 0x7ffa70068320 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa8c14f050 0x7ffa8c1590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 --2- 172.21.15.107:0/3018748486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa78005b90 0x7ffa8c154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:26.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 -- 172.21.15.107:0/3018748486 >> 172.21.15.107:0/3018748486 conn(0x7ffa8c0c2950 msgr2=0x7ffa8c0bfd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:26.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 -- 172.21.15.107:0/3018748486 shutdown_connections 2023-10-25T00:10:26.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:26.269+0000 7ffa9336a640 1 -- 172.21.15.107:0/3018748486 wait complete. 2023-10-25T00:10:26.287 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:10:28.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.417+0000 7f2817d53640 1 Processor -- start 2023-10-25T00:10:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.417+0000 7f2817d53640 1 -- start start 2023-10-25T00:10:28.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2817d53640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f281014fd40 0x7f28100ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:28.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2817d53640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281006f730 con 0x7f281006ee40 2023-10-25T00:10:28.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2817d53640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2810156a40 con 0x7f28100a9820 2023-10-25T00:10:28.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2817d53640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2810156710 con 0x7f281014fd40 2023-10-25T00:10:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f28162f0640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f281006ee40 0x7f281015cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51982/0 (socket says 172.21.15.107:51982) 2023-10-25T00:10:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f28162f0640 1 -- 172.21.15.107:0/1499869680 learned_addr learned my addr 172.21.15.107:0/1499869680 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2815aef640 1 --2- 172.21.15.107:0/1499869680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f281014fd40 0x7f28100ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434563415 0 0) 0x7f281006f730 con 0x7f281006ee40 2023-10-25T00:10:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27fc003530 con 0x7f281006ee40 2023-10-25T00:10:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888522041 0 0) 0x7f27fc003530 con 0x7f281006ee40 2023-10-25T00:10:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f281014fd40 msgr2=0x7f28100ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 --2- 172.21.15.107:0/1499869680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f281014fd40 0x7f28100ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 >> v1:172.21.15.110:6789/0 conn(0x7f28100a9820 legacy=0x7f28100c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281006f980 con 0x7f281006ee40 2023-10-25T00:10:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f2808001a00 con 0x7f281006ee40 2023-10-25T00:10:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 >> v1:172.21.15.107:6789/0 conn(0x7f281006ee40 legacy=0x7f281015cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 --2- 172.21.15.107:0/1499869680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27fc0039b0 0x7f27fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:28.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 --2- 172.21.15.107:0/1499869680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f281014fd40 0x7f27fc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:28.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 --2- 172.21.15.107:0/1499869680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27fc004d10 0x7f27fc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:28.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2810156710 con 0x7f27fc0039b0 2023-10-25T00:10:28.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc00bb50 con 0x7f27fc004d10 2023-10-25T00:10:28.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc00bd10 con 0x7f281014fd40 2023-10-25T00:10:28.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2815aef640 1 --2- 172.21.15.107:0/1499869680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27fc0039b0 0x7f27fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:28.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2815aef640 1 -- 172.21.15.107:0/1499869680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f281014fd40 msgr2=0x7f27fc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:28.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2815aef640 1 --2- 172.21.15.107:0/1499869680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f281014fd40 0x7f27fc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:28.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2815aef640 1 -- 172.21.15.107:0/1499869680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27fc004d10 msgr2=0x7f27fc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:28.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f28162f0640 1 --2- 172.21.15.107:0/1499869680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27fc004d10 0x7f27fc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:28.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2815aef640 1 --2- 172.21.15.107:0/1499869680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27fc004d10 0x7f27fc005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:28.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.421+0000 7f2815aef640 1 -- 172.21.15.107:0/1499869680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc003530 con 0x7f27fc0039b0 2023-10-25T00:10:28.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2815aef640 1 --2- 172.21.15.107:0/1499869680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27fc0039b0 0x7f27fc003dd0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f280c009ae0 tx=0x7f280c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81fe58c2bb25c3c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:28.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c0a5580 con 0x7f27fc0039b0 2023-10-25T00:10:28.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c0a56e0 con 0x7f27fc0039b0 2023-10-25T00:10:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2814aed640 1 -- 172.21.15.107:0/1499869680 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c0a5a90 con 0x7f27fc0039b0 2023-10-25T00:10:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 -- 172.21.15.107:0/1499869680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27fc0039b0 msgr2=0x7f27fc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 --2- 172.21.15.107:0/1499869680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27fc0039b0 0x7f27fc003dd0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f280c009ae0 tx=0x7f280c01ca90 comp rx=0 tx=0).stop 2023-10-25T00:10:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28162f0640 1 -- 172.21.15.107:0/1499869680 reap_dead start 2023-10-25T00:10:28.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 -- 172.21.15.107:0/1499869680 shutdown_connections 2023-10-25T00:10:28.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 -- 172.21.15.107:0/1499869680 >> 172.21.15.107:0/1499869680 conn(0x7f28100c2950 msgr2=0x7f28100c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:28.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 -- 172.21.15.107:0/1499869680 shutdown_connections 2023-10-25T00:10:28.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 -- 172.21.15.107:0/1499869680 wait complete. 2023-10-25T00:10:28.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 Processor -- start 2023-10-25T00:10:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 -- start start 2023-10-25T00:10:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27fc0039b0 0x7f28101eba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281014ea70 0x7f28100c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28100c4180 0x7f2810149270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:28.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28100bd310 con 0x7f281014ea70 2023-10-25T00:10:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28101ebfc0 con 0x7f28100c4180 2023-10-25T00:10:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28100be3a0 con 0x7f27fc0039b0 2023-10-25T00:10:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28152ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281014ea70 0x7f28100c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28152ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281014ea70 0x7f28100c3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39090/0 (socket says 172.21.15.107:39090) 2023-10-25T00:10:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28152ee640 1 -- 172.21.15.107:0/1774551215 learned_addr learned my addr 172.21.15.107:0/1774551215 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2815aef640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27fc0039b0 0x7f28101eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28152ee640 1 -- 172.21.15.107:0/1774551215 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27fc0039b0 msgr2=0x7f28101eba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28152ee640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27fc0039b0 0x7f28101eba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28152ee640 1 -- 172.21.15.107:0/1774551215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28100c4180 msgr2=0x7f2810149270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28162f0640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28100c4180 0x7f2810149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:28.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28152ee640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28100c4180 0x7f2810149270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:28.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28152ee640 1 -- 172.21.15.107:0/1774551215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28100c34d0 con 0x7f281014ea70 2023-10-25T00:10:28.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28152ee640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281014ea70 0x7f28100c3c40 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f280000c4d0 tx=0x7f280000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:28.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28067fc640 1 -- 172.21.15.107:0/1774551215 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28000052d0 con 0x7f281014ea70 2023-10-25T00:10:28.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28067fc640 1 -- 172.21.15.107:0/1774551215 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2800005450 con 0x7f281014ea70 2023-10-25T00:10:28.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 -- 172.21.15.107:0/1774551215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28100c5410 con 0x7f281014ea70 2023-10-25T00:10:28.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f28067fc640 1 -- 172.21.15.107:0/1774551215 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2800012870 con 0x7f281014ea70 2023-10-25T00:10:28.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.425+0000 7f2817d53640 1 -- 172.21.15.107:0/1774551215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28100c59d0 con 0x7f281014ea70 2023-10-25T00:10:28.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.429+0000 7f2817d53640 1 -- 172.21.15.107:0/1774551215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28100c24a0 con 0x7f281014ea70 2023-10-25T00:10:28.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.429+0000 7f28067fc640 1 -- 172.21.15.107:0/1774551215 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f28000129d0 con 0x7f281014ea70 2023-10-25T00:10:28.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.429+0000 7f28067fc640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f27f4065e90 0x7f27f4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:28.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.429+0000 7f28067fc640 1 -- 172.21.15.107:0/1774551215 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7f2800083ff0 con 0x7f281014ea70 2023-10-25T00:10:28.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.429+0000 7f2815aef640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f27f4065e90 0x7f27f4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:28.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.429+0000 7f2815aef640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f27f4065e90 0x7f27f4068350 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f280c0af000 tx=0x7f280c0046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.433+0000 7f28067fc640 1 -- 172.21.15.107:0/1774551215 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2800050900 con 0x7f281014ea70 2023-10-25T00:10:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.625+0000 7f2817d53640 1 -- 172.21.15.107:0/1774551215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28100746e0 con 0x7f281014ea70 2023-10-25T00:10:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.625+0000 7f28067fc640 1 -- 172.21.15.107:0/1774551215 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f2800053f20 con 0x7f281014ea70 2023-10-25T00:10:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:28.629 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 150 2023-10-25T00:10:28.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 -- 172.21.15.107:0/1774551215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f27f4065e90 msgr2=0x7f27f4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:28.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f27f4065e90 0x7f27f4068350 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f280c0af000 tx=0x7f280c0046d0 comp rx=0 tx=0).stop 2023-10-25T00:10:28.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 -- 172.21.15.107:0/1774551215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281014ea70 msgr2=0x7f28100c3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281014ea70 0x7f28100c3c40 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f280000c4d0 tx=0x7f280000c990 comp rx=0 tx=0).stop 2023-10-25T00:10:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 -- 172.21.15.107:0/1774551215 shutdown_connections 2023-10-25T00:10:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28100c4180 0x7f2810149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f281014ea70 0x7f28100c3c40 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:28.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f27f4065e90 0x7f27f4068350 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:28.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 --2- 172.21.15.107:0/1774551215 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27fc0039b0 0x7f28101eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:28.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 -- 172.21.15.107:0/1774551215 >> 172.21.15.107:0/1774551215 conn(0x7f28100c2950 msgr2=0x7f2810150120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:28.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 -- 172.21.15.107:0/1774551215 shutdown_connections 2023-10-25T00:10:28.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:28.629+0000 7f2817d53640 1 -- 172.21.15.107:0/1774551215 wait complete. 2023-10-25T00:10:28.648 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:10:30.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:30.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a92a3640 1 Processor -- start 2023-10-25T00:10:30.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a92a3640 1 -- start start 2023-10-25T00:10:30.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a92a3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49a414fd40 0x7f49a40ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:30.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a92a3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a406f730 con 0x7f49a40a9820 2023-10-25T00:10:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a92a3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a4156a40 con 0x7f49a406ee40 2023-10-25T00:10:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a92a3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4156710 con 0x7f49a414fd40 2023-10-25T00:10:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a27fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f49a40a9820 0x7f49a40c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51996/0 (socket says 172.21.15.107:51996) 2023-10-25T00:10:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a27fc640 1 -- 172.21.15.107:0/957933862 learned_addr learned my addr 172.21.15.107:0/957933862 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:30.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a2ffd640 1 --2- 172.21.15.107:0/957933862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49a414fd40 0x7f49a40ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a1ffb640 1 -- 172.21.15.107:0/957933862 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251253540 0 0) 0x7f49a406f730 con 0x7f49a40a9820 2023-10-25T00:10:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a1ffb640 1 -- 172.21.15.107:0/957933862 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f498c003530 con 0x7f49a40a9820 2023-10-25T00:10:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a2ffd640 1 -- 172.21.15.107:0/957933862 >> v1:172.21.15.110:6789/0 conn(0x7f49a406ee40 legacy=0x7f49a415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a1ffb640 1 -- 172.21.15.107:0/957933862 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 164372342 0 0) 0x7f498c003530 con 0x7f49a40a9820 2023-10-25T00:10:30.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a2ffd640 1 -- 172.21.15.107:0/957933862 >> v1:172.21.15.107:6789/0 conn(0x7f49a40a9820 legacy=0x7f49a40c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:30.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.781+0000 7f49a2ffd640 1 -- 172.21.15.107:0/957933862 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a406f980 con 0x7f49a414fd40 2023-10-25T00:10:30.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a2ffd640 1 --2- 172.21.15.107:0/957933862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49a414fd40 0x7f49a40ab090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f49940097f0 tx=0x7f499401c2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5727b98e0570ad0b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:30.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a1ffb640 1 -- 172.21.15.107:0/957933862 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998001680 con 0x7f49a414fd40 2023-10-25T00:10:30.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a1ffb640 1 -- 172.21.15.107:0/957933862 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49940a7050 con 0x7f49a414fd40 2023-10-25T00:10:30.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a1ffb640 1 -- 172.21.15.107:0/957933862 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49940ab600 con 0x7f49a414fd40 2023-10-25T00:10:30.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 -- 172.21.15.107:0/957933862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49a414fd40 msgr2=0x7f49a40ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:30.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 --2- 172.21.15.107:0/957933862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49a414fd40 0x7f49a40ab090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f49940097f0 tx=0x7f499401c2c0 comp rx=0 tx=0).stop 2023-10-25T00:10:30.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 -- 172.21.15.107:0/957933862 shutdown_connections 2023-10-25T00:10:30.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 --2- 172.21.15.107:0/957933862 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49a414fd40 0x7f49a40ab090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:30.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 -- 172.21.15.107:0/957933862 >> 172.21.15.107:0/957933862 conn(0x7f49a40c2950 msgr2=0x7f49a40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:30.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 -- 172.21.15.107:0/957933862 shutdown_connections 2023-10-25T00:10:30.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 -- 172.21.15.107:0/957933862 wait complete. 2023-10-25T00:10:30.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 Processor -- start 2023-10-25T00:10:30.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 -- start start 2023-10-25T00:10:30.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49a406ee40 0x7f49a4155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:30.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a414fd40 0x7f49a4145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:30.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49a4153990 0x7f49a41eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:30.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a414d0d0 con 0x7f49a414fd40 2023-10-25T00:10:30.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4154b60 con 0x7f49a406ee40 2023-10-25T00:10:30.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.785+0000 7f49a92a3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4146150 con 0x7f49a4153990 2023-10-25T00:10:30.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a414fd40 0x7f49a4145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:30.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a414fd40 0x7f49a4145c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39102/0 (socket says 172.21.15.107:39102) 2023-10-25T00:10:30.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a27fc640 1 -- 172.21.15.107:0/2610858385 learned_addr learned my addr 172.21.15.107:0/2610858385 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:30.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a37fe640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49a4153990 0x7f49a41eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:30.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a2ffd640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49a406ee40 0x7f49a4155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:30.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a27fc640 1 -- 172.21.15.107:0/2610858385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49a4153990 msgr2=0x7f49a41eaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:30.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a27fc640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49a4153990 0x7f49a41eaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:30.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a27fc640 1 -- 172.21.15.107:0/2610858385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49a406ee40 msgr2=0x7f49a4155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:30.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a27fc640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49a406ee40 0x7f49a4155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:30.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a27fc640 1 -- 172.21.15.107:0/2610858385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4994009420 con 0x7f49a414fd40 2023-10-25T00:10:30.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a27fc640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a414fd40 0x7f49a4145c10 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f499800c780 tx=0x7f499800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:30.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f498b7fe640 1 -- 172.21.15.107:0/2610858385 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998005cf0 con 0x7f49a414fd40 2023-10-25T00:10:30.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f498b7fe640 1 -- 172.21.15.107:0/2610858385 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4998005e70 con 0x7f49a414fd40 2023-10-25T00:10:30.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f498b7fe640 1 -- 172.21.15.107:0/2610858385 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998007290 con 0x7f49a414fd40 2023-10-25T00:10:30.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a92a3640 1 -- 172.21.15.107:0/2610858385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a40c54e0 con 0x7f49a414fd40 2023-10-25T00:10:30.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.789+0000 7f49a92a3640 1 -- 172.21.15.107:0/2610858385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a41ef270 con 0x7f49a414fd40 2023-10-25T00:10:30.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.793+0000 7f49a92a3640 1 -- 172.21.15.107:0/2610858385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a40c24a0 con 0x7f49a414fd40 2023-10-25T00:10:30.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.793+0000 7f498b7fe640 1 -- 172.21.15.107:0/2610858385 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f49980073f0 con 0x7f49a414fd40 2023-10-25T00:10:30.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.793+0000 7f498b7fe640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4990065e90 0x7f4990068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:30.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.793+0000 7f498b7fe640 1 -- 172.21.15.107:0/2610858385 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7f4998083f40 con 0x7f49a414fd40 2023-10-25T00:10:30.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.793+0000 7f49a2ffd640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4990065e90 0x7f4990068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:30.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.793+0000 7f49a2ffd640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4990065e90 0x7f4990068350 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f49940ad000 tx=0x7f49940a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:30.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.797+0000 7f498b7fe640 1 -- 172.21.15.107:0/2610858385 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4998050a10 con 0x7f49a414fd40 2023-10-25T00:10:30.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 -- 172.21.15.107:0/2610858385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49a4155b00 con 0x7f49a414fd40 2023-10-25T00:10:30.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f498b7fe640 1 -- 172.21.15.107:0/2610858385 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f49a4155b00 con 0x7f49a414fd40 2023-10-25T00:10:30.990 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:30.990 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:30.991 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 150 2023-10-25T00:10:30.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 -- 172.21.15.107:0/2610858385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4990065e90 msgr2=0x7f4990068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:30.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4990065e90 0x7f4990068350 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f49940ad000 tx=0x7f49940a6000 comp rx=0 tx=0).stop 2023-10-25T00:10:30.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 -- 172.21.15.107:0/2610858385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a414fd40 msgr2=0x7f49a4145c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:30.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a414fd40 0x7f49a4145c10 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f499800c780 tx=0x7f499800cc40 comp rx=0 tx=0).stop 2023-10-25T00:10:30.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 -- 172.21.15.107:0/2610858385 shutdown_connections 2023-10-25T00:10:30.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4990065e90 0x7f4990068350 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:30.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f49a4153990 0x7f49a41eaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:30.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49a414fd40 0x7f49a4145c10 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:30.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 --2- 172.21.15.107:0/2610858385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f49a406ee40 0x7f49a4155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:30.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 -- 172.21.15.107:0/2610858385 >> 172.21.15.107:0/2610858385 conn(0x7f49a40c2950 msgr2=0x7f49a4150d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:30.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 -- 172.21.15.107:0/2610858385 shutdown_connections 2023-10-25T00:10:30.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:30.989+0000 7f49a92a3640 1 -- 172.21.15.107:0/2610858385 wait complete. 2023-10-25T00:10:31.009 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:10:33.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:33.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.141+0000 7fba474e9640 1 Processor -- start 2023-10-25T00:10:33.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba474e9640 1 -- start start 2023-10-25T00:10:33.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba474e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba40152040 0x7fba400c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba474e9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fba400c8100 0x7fba400abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba474e9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba40155680 con 0x7fba4006a4b0 2023-10-25T00:10:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba474e9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba40154a70 con 0x7fba400c8100 2023-10-25T00:10:33.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba474e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba40154d50 con 0x7fba40152040 2023-10-25T00:10:33.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba45a86640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fba4006a4b0 0x7fba400ac240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54590/0 (socket says 172.21.15.107:54590) 2023-10-25T00:10:33.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba45a86640 1 -- 172.21.15.107:0/3721794226 learned_addr learned my addr 172.21.15.107:0/3721794226 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:33.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba45285640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba40152040 0x7fba400c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:33.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba44a84640 1 --2- 172.21.15.107:0/3721794226 >> v2:172.21.15.110:3300/0 conn(0x7fba400c8100 0x7fba400abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:33.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780798351 0 0) 0x7fba40155680 con 0x7fba4006a4b0 2023-10-25T00:10:33.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba28003530 con 0x7fba4006a4b0 2023-10-25T00:10:33.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3253493525 0 0) 0x7fba28003530 con 0x7fba4006a4b0 2023-10-25T00:10:33.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba40152040 msgr2=0x7fba400c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:33.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba40152040 0x7fba400c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 >> v2:172.21.15.110:3300/0 conn(0x7fba400c8100 msgr2=0x7fba400abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:33.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 --2- 172.21.15.107:0/3721794226 >> v2:172.21.15.110:3300/0 conn(0x7fba400c8100 0x7fba400abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba40155030 con 0x7fba4006a4b0 2023-10-25T00:10:33.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fba38001680 con 0x7fba4006a4b0 2023-10-25T00:10:33.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 >> v1:172.21.15.107:6789/0 conn(0x7fba4006a4b0 legacy=0x7fba400ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:33.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba40152040 0x7fba28003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:33.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba28003f00 0x7fba280047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:33.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba28004d10 0x7fba28005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:33.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba40154a70 con 0x7fba28004d10 2023-10-25T00:10:33.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba40154d50 con 0x7fba28003f00 2023-10-25T00:10:33.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2800b9e0 con 0x7fba40152040 2023-10-25T00:10:33.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba45a86640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba28004d10 0x7fba28005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:33.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba44a84640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba28003f00 0x7fba280047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:33.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba45285640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba40152040 0x7fba28003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:33.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba45a86640 1 -- 172.21.15.107:0/3721794226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba40152040 msgr2=0x7fba28003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:33.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba45a86640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba40152040 0x7fba28003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba45a86640 1 -- 172.21.15.107:0/3721794226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba28003f00 msgr2=0x7fba280047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:33.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba45a86640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba28003f00 0x7fba280047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.145+0000 7fba45a86640 1 -- 172.21.15.107:0/3721794226 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba28003530 con 0x7fba28004d10 2023-10-25T00:10:33.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba45a86640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba28004d10 0x7fba28005190 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fba3800c270 tx=0x7fba3801f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a65e4f8059fd6e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:33.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba38005ae0 con 0x7fba28004d10 2023-10-25T00:10:33.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba380ab070 con 0x7fba28004d10 2023-10-25T00:10:33.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba3ffff640 1 -- 172.21.15.107:0/3721794226 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba38007070 con 0x7fba28004d10 2023-10-25T00:10:33.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 -- 172.21.15.107:0/3721794226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba28004d10 msgr2=0x7fba28005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:33.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 --2- 172.21.15.107:0/3721794226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba28004d10 0x7fba28005190 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fba3800c270 tx=0x7fba3801f3b0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba45a86640 1 -- 172.21.15.107:0/3721794226 reap_dead start 2023-10-25T00:10:33.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 -- 172.21.15.107:0/3721794226 shutdown_connections 2023-10-25T00:10:33.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 -- 172.21.15.107:0/3721794226 >> 172.21.15.107:0/3721794226 conn(0x7fba400ca200 msgr2=0x7fba400ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:33.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 -- 172.21.15.107:0/3721794226 shutdown_connections 2023-10-25T00:10:33.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 -- 172.21.15.107:0/3721794226 wait complete. 2023-10-25T00:10:33.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 Processor -- start 2023-10-25T00:10:33.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 -- start start 2023-10-25T00:10:33.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba28004d10 0x7fba401e96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:33.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba401ed500 0x7fba401e9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:33.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba401ed960 0x7fba401527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:33.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba400c1890 con 0x7fba401ed500 2023-10-25T00:10:33.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba401e9c30 con 0x7fba401ed960 2023-10-25T00:10:33.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba474e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba401ed070 con 0x7fba28004d10 2023-10-25T00:10:33.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba44a84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba401ed500 0x7fba401e9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:33.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba44a84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba401ed500 0x7fba401e9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47752/0 (socket says 172.21.15.107:47752) 2023-10-25T00:10:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba44a84640 1 -- 172.21.15.107:0/4179571142 learned_addr learned my addr 172.21.15.107:0/4179571142 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba45285640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba28004d10 0x7fba401e96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba45a86640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba401ed960 0x7fba401527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba44a84640 1 -- 172.21.15.107:0/4179571142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba28004d10 msgr2=0x7fba401e96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:33.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba44a84640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba28004d10 0x7fba401e96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba44a84640 1 -- 172.21.15.107:0/4179571142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba401ed960 msgr2=0x7fba401527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:33.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba44a84640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba401ed960 0x7fba401527d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba44a84640 1 -- 172.21.15.107:0/4179571142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4014f9f0 con 0x7fba401ed500 2023-10-25T00:10:33.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.149+0000 7fba45285640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba28004d10 0x7fba401e96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:10:33.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba44a84640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba401ed500 0x7fba401e9e50 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fba34011700 tx=0x7fba34011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:33.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba3dffb640 1 -- 172.21.15.107:0/4179571142 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3400f960 con 0x7fba401ed500 2023-10-25T00:10:33.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba3dffb640 1 -- 172.21.15.107:0/4179571142 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3400fac0 con 0x7fba401ed500 2023-10-25T00:10:33.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba474e9640 1 -- 172.21.15.107:0/4179571142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba400aa020 con 0x7fba401ed500 2023-10-25T00:10:33.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba3dffb640 1 -- 172.21.15.107:0/4179571142 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3400fe50 con 0x7fba401ed500 2023-10-25T00:10:33.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba474e9640 1 -- 172.21.15.107:0/4179571142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba401ecd00 con 0x7fba401ed500 2023-10-25T00:10:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba474e9640 1 -- 172.21.15.107:0/4179571142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba401dfdb0 con 0x7fba401ed500 2023-10-25T00:10:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba3dffb640 1 -- 172.21.15.107:0/4179571142 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fba34015460 con 0x7fba401ed500 2023-10-25T00:10:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba3dffb640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba2c065e60 0x7fba2c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba3dffb640 1 -- 172.21.15.107:0/4179571142 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7fba34084d20 con 0x7fba401ed500 2023-10-25T00:10:33.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba45285640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba2c065e60 0x7fba2c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:33.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.153+0000 7fba45285640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba2c065e60 0x7fba2c068320 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fba30007440 tx=0x7fba3000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:33.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.157+0000 7fba3dffb640 1 -- 172.21.15.107:0/4179571142 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fba340516c0 con 0x7fba401ed500 2023-10-25T00:10:33.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.341+0000 7fba474e9640 1 -- 172.21.15.107:0/4179571142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba400746e0 con 0x7fba401ed500 2023-10-25T00:10:33.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.341+0000 7fba3dffb640 1 -- 172.21.15.107:0/4179571142 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5777 (secure 0 0 0) 0x7fba34054ce0 con 0x7fba401ed500 2023-10-25T00:10:33.344 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:33.344 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:33.345 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 150 2023-10-25T00:10:33.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 -- 172.21.15.107:0/4179571142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba2c065e60 msgr2=0x7fba2c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:33.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba2c065e60 0x7fba2c068320 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fba30007440 tx=0x7fba3000b040 comp rx=0 tx=0).stop 2023-10-25T00:10:33.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 -- 172.21.15.107:0/4179571142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba401ed500 msgr2=0x7fba401e9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:33.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba401ed500 0x7fba401e9e50 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fba34011700 tx=0x7fba34011bc0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 -- 172.21.15.107:0/4179571142 shutdown_connections 2023-10-25T00:10:33.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fba401ed960 0x7fba401527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba401ed500 0x7fba401e9e50 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fba2c065e60 0x7fba2c068320 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 --2- 172.21.15.107:0/4179571142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fba28004d10 0x7fba401e96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:33.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 -- 172.21.15.107:0/4179571142 >> 172.21.15.107:0/4179571142 conn(0x7fba400ca200 msgr2=0x7fba400ad0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:33.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 -- 172.21.15.107:0/4179571142 shutdown_connections 2023-10-25T00:10:33.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:33.345+0000 7fba474e9640 1 -- 172.21.15.107:0/4179571142 wait complete. 2023-10-25T00:10:33.361 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:10:35.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.489+0000 7fd7a5c68640 1 Processor -- start 2023-10-25T00:10:35.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.489+0000 7fd7a5c68640 1 -- start start 2023-10-25T00:10:35.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd7a5c68640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd7a0152040 0x7fd7a00c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:35.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd7a5c68640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd7a006a4b0 0x7fd7a00ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd7a5c68640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a0155680 con 0x7fd7a00c8100 2023-10-25T00:10:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd7a5c68640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a0154a70 con 0x7fd7a0152040 2023-10-25T00:10:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd7a5c68640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a0154d50 con 0x7fd7a006a4b0 2023-10-25T00:10:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd7a0152040 0x7fd7a00c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:35.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79effd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd7a00c8100 0x7fd7a00abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54606/0 (socket says 172.21.15.107:54606) 2023-10-25T00:10:35.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79effd640 1 -- 172.21.15.107:0/2193196230 learned_addr learned my addr 172.21.15.107:0/2193196230 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:35.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79ffff640 1 --2- 172.21.15.107:0/2193196230 >> v2:172.21.15.110:3300/0 conn(0x7fd7a006a4b0 0x7fd7a00ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:35.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79f7fe640 1 -- 172.21.15.107:0/2193196230 >> v2:172.21.15.110:3300/0 conn(0x7fd7a006a4b0 msgr2=0x7fd7a00ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:35.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/2193196230 >> v2:172.21.15.110:3300/0 conn(0x7fd7a006a4b0 0x7fd7a00ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:35.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79f7fe640 1 -- 172.21.15.107:0/2193196230 >> v1:172.21.15.107:6789/0 conn(0x7fd7a00c8100 legacy=0x7fd7a00abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:35.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79f7fe640 1 -- 172.21.15.107:0/2193196230 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a0155030 con 0x7fd7a0152040 2023-10-25T00:10:35.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/2193196230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7a0152040 0x7fd7a00c4b80 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd7a006b590 tx=0x7fd79001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b2768716401241 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:35.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79e7fc640 1 -- 172.21.15.107:0/2193196230 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900ab600 con 0x7fd7a0152040 2023-10-25T00:10:35.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79e7fc640 1 -- 172.21.15.107:0/2193196230 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7900a7050 con 0x7fd7a0152040 2023-10-25T00:10:35.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.493+0000 7fd79e7fc640 1 -- 172.21.15.107:0/2193196230 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900ab980 con 0x7fd7a0152040 2023-10-25T00:10:35.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 -- 172.21.15.107:0/2193196230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7a0152040 msgr2=0x7fd7a00c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:35.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 --2- 172.21.15.107:0/2193196230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7a0152040 0x7fd7a00c4b80 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd7a006b590 tx=0x7fd79001c260 comp rx=0 tx=0).stop 2023-10-25T00:10:35.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 -- 172.21.15.107:0/2193196230 shutdown_connections 2023-10-25T00:10:35.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 --2- 172.21.15.107:0/2193196230 >> v2:172.21.15.110:3300/0 conn(0x7fd7a006a4b0 0x7fd7a00ac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:35.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 --2- 172.21.15.107:0/2193196230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7a0152040 0x7fd7a00c4b80 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:35.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 -- 172.21.15.107:0/2193196230 >> 172.21.15.107:0/2193196230 conn(0x7fd7a00ca200 msgr2=0x7fd7a00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:35.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 -- 172.21.15.107:0/2193196230 shutdown_connections 2023-10-25T00:10:35.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 -- 172.21.15.107:0/2193196230 wait complete. 2023-10-25T00:10:35.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 Processor -- start 2023-10-25T00:10:35.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 -- start start 2023-10-25T00:10:35.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7a006a4b0 0x7fd7a00ae7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:35.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a00c8100 0x7fd7a0148820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:35.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7a0152040 0x7fd7a0148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:35.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a014b020 con 0x7fd7a0152040 2023-10-25T00:10:35.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01f0370 con 0x7fd7a00c8100 2023-10-25T00:10:35.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01f0650 con 0x7fd7a006a4b0 2023-10-25T00:10:35.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd79ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7a0152040 0x7fd7a0148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:35.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd79ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7a0152040 0x7fd7a0148d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47764/0 (socket says 172.21.15.107:47764) 2023-10-25T00:10:35.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd79ffff640 1 -- 172.21.15.107:0/3735045803 learned_addr learned my addr 172.21.15.107:0/3735045803 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:35.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd79ffff640 1 -- 172.21.15.107:0/3735045803 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7a006a4b0 msgr2=0x7fd7a00ae7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:35.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd79ffff640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7a006a4b0 0x7fd7a00ae7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:35.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd79ffff640 1 -- 172.21.15.107:0/3735045803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a00c8100 msgr2=0x7fd7a0148820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:35.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd79effd640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a00c8100 0x7fd7a0148820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:35.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd79ffff640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a00c8100 0x7fd7a0148820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:35.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd79ffff640 1 -- 172.21.15.107:0/3735045803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd790009420 con 0x7fd7a0152040 2023-10-25T00:10:35.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd79ffff640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7a0152040 0x7fd7a0148d60 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd79800efc0 tx=0x7fd79800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:35.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd787fff640 1 -- 172.21.15.107:0/3735045803 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798009ba0 con 0x7fd7a0152040 2023-10-25T00:10:35.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd787fff640 1 -- 172.21.15.107:0/3735045803 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79800f040 con 0x7fd7a0152040 2023-10-25T00:10:35.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd787fff640 1 -- 172.21.15.107:0/3735045803 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7980146c0 con 0x7fd7a0152040 2023-10-25T00:10:35.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 -- 172.21.15.107:0/3735045803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a01f0930 con 0x7fd7a0152040 2023-10-25T00:10:35.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.497+0000 7fd7a5c68640 1 -- 172.21.15.107:0/3735045803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a01f0ef0 con 0x7fd7a0152040 2023-10-25T00:10:35.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.501+0000 7fd7a5c68640 1 -- 172.21.15.107:0/3735045803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a00746e0 con 0x7fd7a0152040 2023-10-25T00:10:35.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.501+0000 7fd787fff640 1 -- 172.21.15.107:0/3735045803 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd798014820 con 0x7fd7a0152040 2023-10-25T00:10:35.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.501+0000 7fd787fff640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd78c065e60 0x7fd78c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:35.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.501+0000 7fd787fff640 1 -- 172.21.15.107:0/3735045803 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7fd798084ff0 con 0x7fd7a0152040 2023-10-25T00:10:35.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.501+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd78c065e60 0x7fd78c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:35.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.501+0000 7fd79f7fe640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd78c065e60 0x7fd78c068320 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd7900ad000 tx=0x7fd790008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:35.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.505+0000 7fd787fff640 1 -- 172.21.15.107:0/3735045803 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd798051990 con 0x7fd7a0152040 2023-10-25T00:10:35.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.689+0000 7fd7a5c68640 1 -- 172.21.15.107:0/3735045803 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7a00bd640 con 0x7fd7a0152040 2023-10-25T00:10:35.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd787fff640 1 -- 172.21.15.107:0/3735045803 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5777 (secure 0 0 0) 0x7fd798018020 con 0x7fd7a0152040 2023-10-25T00:10:35.694 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:35.694 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:35.694 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 150 2023-10-25T00:10:35.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 -- 172.21.15.107:0/3735045803 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd78c065e60 msgr2=0x7fd78c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:35.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd78c065e60 0x7fd78c068320 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd7900ad000 tx=0x7fd790008040 comp rx=0 tx=0).stop 2023-10-25T00:10:35.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 -- 172.21.15.107:0/3735045803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7a0152040 msgr2=0x7fd7a0148d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:35.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7a0152040 0x7fd7a0148d60 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd79800efc0 tx=0x7fd79800c490 comp rx=0 tx=0).stop 2023-10-25T00:10:35.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 -- 172.21.15.107:0/3735045803 shutdown_connections 2023-10-25T00:10:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd7a0152040 0x7fd7a0148d60 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7a00c8100 0x7fd7a0148820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd78c065e60 0x7fd78c068320 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 --2- 172.21.15.107:0/3735045803 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd7a006a4b0 0x7fd7a00ae7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:35.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 -- 172.21.15.107:0/3735045803 >> 172.21.15.107:0/3735045803 conn(0x7fd7a00ca200 msgr2=0x7fd7a0151880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 -- 172.21.15.107:0/3735045803 shutdown_connections 2023-10-25T00:10:35.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:35.693+0000 7fd7a5c68640 1 -- 172.21.15.107:0/3735045803 wait complete. 2023-10-25T00:10:35.711 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:10:37.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:37.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaf4f44640 1 Processor -- start 2023-10-25T00:10:37.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaf4f44640 1 -- start start 2023-10-25T00:10:37.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaf4f44640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faaf013bb00 0x7faaf01531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:37.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaf4f44640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faaf00aa330 0x7faaf006e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:37.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaf4f44640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaf0152d40 con 0x7faaf00a9450 2023-10-25T00:10:37.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaf4f44640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaf0156100 con 0x7faaf013bb00 2023-10-25T00:10:37.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaf4f44640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf01527c0 con 0x7faaf00aa330 2023-10-25T00:10:37.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaee59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faaf013bb00 0x7faaf01531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:37.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaee59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faaf013bb00 0x7faaf01531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47774/0 (socket says 172.21.15.107:47774) 2023-10-25T00:10:37.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaee59c640 1 -- 172.21.15.107:0/4040840745 learned_addr learned my addr 172.21.15.107:0/4040840745 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:37.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaedd9b640 1 --2- 172.21.15.107:0/4040840745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faaf00aa330 0x7faaf006e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:37.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaee59c640 1 -- 172.21.15.107:0/4040840745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faaf00aa330 msgr2=0x7faaf006e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:37.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaee59c640 1 --2- 172.21.15.107:0/4040840745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faaf00aa330 0x7faaf006e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:37.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaee59c640 1 -- 172.21.15.107:0/4040840745 >> v1:172.21.15.110:6789/0 conn(0x7faaf00a9450 legacy=0x7faaf0071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:37.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.841+0000 7faaee59c640 1 -- 172.21.15.107:0/4040840745 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf006d660 con 0x7faaf013bb00 2023-10-25T00:10:37.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaee59c640 1 --2- 172.21.15.107:0/4040840745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf013bb00 0x7faaf01531d0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7faae00098a0 tx=0x7faae001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13160ebf4dec9716 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:37.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaed59a640 1 -- 172.21.15.107:0/4040840745 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae00ac640 con 0x7faaf013bb00 2023-10-25T00:10:37.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaed59a640 1 -- 172.21.15.107:0/4040840745 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae00ac7a0 con 0x7faaf013bb00 2023-10-25T00:10:37.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaed59a640 1 -- 172.21.15.107:0/4040840745 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae00acb30 con 0x7faaf013bb00 2023-10-25T00:10:37.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 -- 172.21.15.107:0/4040840745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf013bb00 msgr2=0x7faaf01531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:37.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 --2- 172.21.15.107:0/4040840745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf013bb00 0x7faaf01531d0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7faae00098a0 tx=0x7faae001c400 comp rx=0 tx=0).stop 2023-10-25T00:10:37.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 -- 172.21.15.107:0/4040840745 shutdown_connections 2023-10-25T00:10:37.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 --2- 172.21.15.107:0/4040840745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faaf00aa330 0x7faaf006e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:37.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 --2- 172.21.15.107:0/4040840745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf013bb00 0x7faaf01531d0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:37.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 -- 172.21.15.107:0/4040840745 >> 172.21.15.107:0/4040840745 conn(0x7faaf0070f60 msgr2=0x7faaf0071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:37.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 -- 172.21.15.107:0/4040840745 shutdown_connections 2023-10-25T00:10:37.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 -- 172.21.15.107:0/4040840745 wait complete. 2023-10-25T00:10:37.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 Processor -- start 2023-10-25T00:10:37.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 -- start start 2023-10-25T00:10:37.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faaf00a9450 0x7faaf0131b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:37.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faaf00aa330 0x7faaf0135320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:37.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf013bb00 0x7faaf0135860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:37.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf0135da0 con 0x7faaf013bb00 2023-10-25T00:10:37.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf0130f70 con 0x7faaf00aa330 2023-10-25T00:10:37.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf0134e90 con 0x7faaf00a9450 2023-10-25T00:10:37.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaeed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf013bb00 0x7faaf0135860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:37.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaeed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf013bb00 0x7faaf0135860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47788/0 (socket says 172.21.15.107:47788) 2023-10-25T00:10:37.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaeed9d640 1 -- 172.21.15.107:0/90455908 learned_addr learned my addr 172.21.15.107:0/90455908 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:37.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaee59c640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faaf00a9450 0x7faaf0131b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:37.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaedd9b640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faaf00aa330 0x7faaf0135320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:37.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaeed9d640 1 -- 172.21.15.107:0/90455908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faaf00a9450 msgr2=0x7faaf0131b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:37.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaeed9d640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faaf00a9450 0x7faaf0131b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:37.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaeed9d640 1 -- 172.21.15.107:0/90455908 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faaf00aa330 msgr2=0x7faaf0135320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:37.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaeed9d640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faaf00aa330 0x7faaf0135320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:37.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaeed9d640 1 -- 172.21.15.107:0/90455908 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae00094d0 con 0x7faaf013bb00 2023-10-25T00:10:37.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaeed9d640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf013bb00 0x7faaf0135860 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7faae800c770 tx=0x7faae800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:37.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faadeffd640 1 -- 172.21.15.107:0/90455908 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae8003b80 con 0x7faaf013bb00 2023-10-25T00:10:37.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faadeffd640 1 -- 172.21.15.107:0/90455908 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae8003ce0 con 0x7faaf013bb00 2023-10-25T00:10:37.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 -- 172.21.15.107:0/90455908 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf01311c0 con 0x7faaf013bb00 2023-10-25T00:10:37.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faadeffd640 1 -- 172.21.15.107:0/90455908 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae800aa90 con 0x7faaf013bb00 2023-10-25T00:10:37.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.845+0000 7faaf4f44640 1 -- 172.21.15.107:0/90455908 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf0134b20 con 0x7faaf013bb00 2023-10-25T00:10:37.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.849+0000 7faaf4f44640 1 -- 172.21.15.107:0/90455908 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf0134cb0 con 0x7faaf013bb00 2023-10-25T00:10:37.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.849+0000 7faadeffd640 1 -- 172.21.15.107:0/90455908 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7faae800abf0 con 0x7faaf013bb00 2023-10-25T00:10:37.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.849+0000 7faadeffd640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faad0065e90 0x7faad0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:37.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.849+0000 7faaee59c640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faad0065e90 0x7faad0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:37.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.849+0000 7faadeffd640 1 -- 172.21.15.107:0/90455908 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 9369+0+0 (secure 0 0 0) 0x7faae8084330 con 0x7faaf013bb00 2023-10-25T00:10:37.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.849+0000 7faaee59c640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faad0065e90 0x7faad0068350 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7faae00ae000 tx=0x7faae00a6070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:37.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:37.853+0000 7faadeffd640 1 -- 172.21.15.107:0/90455908 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7faae8050cd0 con 0x7faaf013bb00 2023-10-25T00:10:38.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.041+0000 7faaf4f44640 1 -- 172.21.15.107:0/90455908 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faaf0135000 con 0x7faaf013bb00 2023-10-25T00:10:38.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faadeffd640 1 -- 172.21.15.107:0/90455908 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+5777 (secure 0 0 0) 0x7faae80542f0 con 0x7faaf013bb00 2023-10-25T00:10:38.045 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:38.046 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7746,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:09:01.764886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7578":{"gid":7578,"name":"a","rank":0,"incarnation":141,"state":"up:active","state_seq":11,"addr":"172.21.15.107:6837/1788268787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1788268787},{"type":"v1","addr":"172.21.15.107:6837","nonce":1788268787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:38.046 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 150 2023-10-25T00:10:38.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 -- 172.21.15.107:0/90455908 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faad0065e90 msgr2=0x7faad0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:38.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faad0065e90 0x7faad0068350 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7faae00ae000 tx=0x7faae00a6070 comp rx=0 tx=0).stop 2023-10-25T00:10:38.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 -- 172.21.15.107:0/90455908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf013bb00 msgr2=0x7faaf0135860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf013bb00 0x7faaf0135860 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7faae800c770 tx=0x7faae800cc30 comp rx=0 tx=0).stop 2023-10-25T00:10:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 -- 172.21.15.107:0/90455908 shutdown_connections 2023-10-25T00:10:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf013bb00 0x7faaf0135860 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faaf00aa330 0x7faaf0135320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:38.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faad0065e90 0x7faad0068350 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 --2- 172.21.15.107:0/90455908 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faaf00a9450 0x7faaf0131b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 -- 172.21.15.107:0/90455908 >> 172.21.15.107:0/90455908 conn(0x7faaf0070f60 msgr2=0x7faaf0073170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 -- 172.21.15.107:0/90455908 shutdown_connections 2023-10-25T00:10:38.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:38.045+0000 7faaf4f44640 1 -- 172.21.15.107:0/90455908 wait complete. 2023-10-25T00:10:38.064 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:10:40.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:40.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f603d377640 1 Processor -- start 2023-10-25T00:10:40.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f603d377640 1 -- start start 2023-10-25T00:10:40.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f603d377640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f603806c9e0 0x7f60380a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f603d377640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f60380c2de0 0x7f6038154830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f603d377640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6038158570 0x7f60380aa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f603d377640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60380c4180 con 0x7f603806c9e0 2023-10-25T00:10:40.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f603d377640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6038157e00 con 0x7f60380c2de0 2023-10-25T00:10:40.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f603d377640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60380c32e0 con 0x7f6038158570 2023-10-25T00:10:40.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f6036ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f603806c9e0 0x7f60380a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f6036ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f603806c9e0 0x7f60380a96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47798/0 (socket says 172.21.15.107:47798) 2023-10-25T00:10:40.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f6036ffd640 1 -- 172.21.15.107:0/3789590914 learned_addr learned my addr 172.21.15.107:0/3789590914 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:40.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f60367fc640 1 --2- 172.21.15.107:0/3789590914 >> v2:172.21.15.110:3300/0 conn(0x7f60380c2de0 0x7f6038154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f60377fe640 1 --2- 172.21.15.107:0/3789590914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6038158570 0x7f60380aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f6036ffd640 1 -- 172.21.15.107:0/3789590914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6038158570 msgr2=0x7f60380aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:40.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f6036ffd640 1 --2- 172.21.15.107:0/3789590914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6038158570 0x7f60380aa050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f6036ffd640 1 -- 172.21.15.107:0/3789590914 >> v2:172.21.15.110:3300/0 conn(0x7f60380c2de0 msgr2=0x7f6038154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f6036ffd640 1 --2- 172.21.15.107:0/3789590914 >> v2:172.21.15.110:3300/0 conn(0x7f60380c2de0 0x7f6038154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.193+0000 7f6036ffd640 1 -- 172.21.15.107:0/3789590914 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60380c35c0 con 0x7f603806c9e0 2023-10-25T00:10:40.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f6036ffd640 1 --2- 172.21.15.107:0/3789590914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f603806c9e0 0x7f60380a96c0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f60380729f0 tx=0x7f602801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408b2fc5e1af2864 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:40.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f6035ffb640 1 -- 172.21.15.107:0/3789590914 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60280ab600 con 0x7f603806c9e0 2023-10-25T00:10:40.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f6035ffb640 1 -- 172.21.15.107:0/3789590914 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60280a7050 con 0x7f603806c9e0 2023-10-25T00:10:40.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f6035ffb640 1 -- 172.21.15.107:0/3789590914 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60280ab980 con 0x7f603806c9e0 2023-10-25T00:10:40.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 -- 172.21.15.107:0/3789590914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f603806c9e0 msgr2=0x7f60380a96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 --2- 172.21.15.107:0/3789590914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f603806c9e0 0x7f60380a96c0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f60380729f0 tx=0x7f602801c3e0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 -- 172.21.15.107:0/3789590914 shutdown_connections 2023-10-25T00:10:40.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 --2- 172.21.15.107:0/3789590914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6038158570 0x7f60380aa050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 --2- 172.21.15.107:0/3789590914 >> v2:172.21.15.110:3300/0 conn(0x7f60380c2de0 0x7f6038154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 --2- 172.21.15.107:0/3789590914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f603806c9e0 0x7f60380a96c0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 -- 172.21.15.107:0/3789590914 >> 172.21.15.107:0/3789590914 conn(0x7f60380c81d0 msgr2=0x7f60380c85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:40.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 -- 172.21.15.107:0/3789590914 shutdown_connections 2023-10-25T00:10:40.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 -- 172.21.15.107:0/3789590914 wait complete. 2023-10-25T00:10:40.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 Processor -- start 2023-10-25T00:10:40.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 -- start start 2023-10-25T00:10:40.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f603806c9e0 0x7f60381f0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60380c2de0 0x7f603806e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6038158570 0x7f60380accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60380c6ea0 con 0x7f60380c2de0 2023-10-25T00:10:40.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603814d820 con 0x7f603806c9e0 2023-10-25T00:10:40.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603814ade0 con 0x7f6038158570 2023-10-25T00:10:40.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f60367fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60380c2de0 0x7f603806e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f60367fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60380c2de0 0x7f603806e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47812/0 (socket says 172.21.15.107:47812) 2023-10-25T00:10:40.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f60367fc640 1 -- 172.21.15.107:0/2777822693 learned_addr learned my addr 172.21.15.107:0/2777822693 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:40.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f60377fe640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6038158570 0x7f60380accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f6036ffd640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f603806c9e0 0x7f60381f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f60367fc640 1 -- 172.21.15.107:0/2777822693 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6038158570 msgr2=0x7f60380accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f60367fc640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6038158570 0x7f60380accb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f60367fc640 1 -- 172.21.15.107:0/2777822693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f603806c9e0 msgr2=0x7f60381f0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f60367fc640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f603806c9e0 0x7f60381f0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f60367fc640 1 -- 172.21.15.107:0/2777822693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60280094d0 con 0x7f60380c2de0 2023-10-25T00:10:40.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f60377fe640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6038158570 0x7f60380accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:10:40.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f60367fc640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60380c2de0 0x7f603806e720 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f602c00d830 tx=0x7f602c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:40.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f601f7fe640 1 -- 172.21.15.107:0/2777822693 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c0096a0 con 0x7f60380c2de0 2023-10-25T00:10:40.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f601f7fe640 1 -- 172.21.15.107:0/2777822693 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c009800 con 0x7f60380c2de0 2023-10-25T00:10:40.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f601f7fe640 1 -- 172.21.15.107:0/2777822693 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c009b70 con 0x7f60380c2de0 2023-10-25T00:10:40.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 -- 172.21.15.107:0/2777822693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60380c6290 con 0x7f60380c2de0 2023-10-25T00:10:40.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.197+0000 7f603d377640 1 -- 172.21.15.107:0/2777822693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60380abf90 con 0x7f60380c2de0 2023-10-25T00:10:40.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.201+0000 7f603d377640 1 -- 172.21.15.107:0/2777822693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6038067a50 con 0x7f60380c2de0 2023-10-25T00:10:40.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.201+0000 7f601f7fe640 1 -- 172.21.15.107:0/2777822693 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f602c014410 con 0x7f60380c2de0 2023-10-25T00:10:40.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.201+0000 7f601f7fe640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6024065e60 0x7f6024068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.201+0000 7f601f7fe640 1 -- 172.21.15.107:0/2777822693 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7f602c084eb0 con 0x7f60380c2de0 2023-10-25T00:10:40.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.201+0000 7f6036ffd640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6024065e60 0x7f6024068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.201+0000 7f6036ffd640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6024065e60 0x7f6024068320 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f60280ad000 tx=0x7f60280a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:40.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.205+0000 7f601f7fe640 1 -- 172.21.15.107:0/2777822693 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f602c051800 con 0x7f60380c2de0 2023-10-25T00:10:40.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.377+0000 7f603d377640 1 -- 172.21.15.107:0/2777822693 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f603804e0c0 con 0x7f60380c2de0 2023-10-25T00:10:40.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f601f7fe640 1 -- 172.21.15.107:0/2777822693 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5019 (secure 0 0 0) 0x7f602c054e20 con 0x7f60380c2de0 2023-10-25T00:10:40.382 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:40.383 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:39.839906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 152 2023-10-25T00:10:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 -- 172.21.15.107:0/2777822693 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6024065e60 msgr2=0x7f6024068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6024065e60 0x7f6024068320 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f60280ad000 tx=0x7f60280a6000 comp rx=0 tx=0).stop 2023-10-25T00:10:40.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 -- 172.21.15.107:0/2777822693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60380c2de0 msgr2=0x7f603806e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60380c2de0 0x7f603806e720 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f602c00d830 tx=0x7f602c00dcf0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 -- 172.21.15.107:0/2777822693 shutdown_connections 2023-10-25T00:10:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6024065e60 0x7f6024068320 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6038158570 0x7f60380accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60380c2de0 0x7f603806e720 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 --2- 172.21.15.107:0/2777822693 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f603806c9e0 0x7f60381f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 -- 172.21.15.107:0/2777822693 >> 172.21.15.107:0/2777822693 conn(0x7f60380c81d0 msgr2=0x7f6038157030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:40.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 -- 172.21.15.107:0/2777822693 shutdown_connections 2023-10-25T00:10:40.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.381+0000 7f603d377640 1 -- 172.21.15.107:0/2777822693 wait complete. 2023-10-25T00:10:40.399 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2023-10-25T00:10:40.399 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:10:40.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:40.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa640bc7640 1 Processor -- start 2023-10-25T00:10:40.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa640bc7640 1 -- start start 2023-10-25T00:10:40.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa640bc7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c152040 0x7fa63c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa640bc7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63c155680 con 0x7fa63c06a4b0 2023-10-25T00:10:40.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa640bc7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa63c154a70 con 0x7fa63c0c8100 2023-10-25T00:10:40.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa640bc7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c154d50 con 0x7fa63c152040 2023-10-25T00:10:40.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa63c06a4b0 0x7fa63c0ac410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54618/0 (socket says 172.21.15.107:54618) 2023-10-25T00:10:40.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63ad9d640 1 -- 172.21.15.107:0/3917046959 learned_addr learned my addr 172.21.15.107:0/3917046959 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:40.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63a59c640 1 --2- 172.21.15.107:0/3917046959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c152040 0x7fa63c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128355536 0 0) 0x7fa63c155680 con 0x7fa63c06a4b0 2023-10-25T00:10:40.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa624003530 con 0x7fa63c06a4b0 2023-10-25T00:10:40.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3650964524 0 0) 0x7fa624003530 con 0x7fa63c06a4b0 2023-10-25T00:10:40.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c152040 msgr2=0x7fa63c0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:40.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 --2- 172.21.15.107:0/3917046959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c152040 0x7fa63c0c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 >> v1:172.21.15.110:6789/0 conn(0x7fa63c0c8100 legacy=0x7fa63c0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63c155030 con 0x7fa63c06a4b0 2023-10-25T00:10:40.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fa634001a00 con 0x7fa63c06a4b0 2023-10-25T00:10:40.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 >> v1:172.21.15.107:6789/0 conn(0x7fa63c06a4b0 legacy=0x7fa63c0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 --2- 172.21.15.107:0/3917046959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 0x7fa624003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 --2- 172.21.15.107:0/3917046959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6240047d0 0x7fa624004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 --2- 172.21.15.107:0/3917046959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c152040 0x7fa624005260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c154d50 con 0x7fa6240039b0 2023-10-25T00:10:40.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62400bb50 con 0x7fa6240047d0 2023-10-25T00:10:40.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62400bd10 con 0x7fa63c152040 2023-10-25T00:10:40.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63a59c640 1 --2- 172.21.15.107:0/3917046959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 0x7fa624003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa639d9b640 1 --2- 172.21.15.107:0/3917046959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6240047d0 0x7fa624004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63a59c640 1 -- 172.21.15.107:0/3917046959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c152040 msgr2=0x7fa624005260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:40.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63a59c640 1 --2- 172.21.15.107:0/3917046959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c152040 0x7fa624005260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63a59c640 1 -- 172.21.15.107:0/3917046959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6240047d0 msgr2=0x7fa624004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:40.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63a59c640 1 --2- 172.21.15.107:0/3917046959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6240047d0 0x7fa624004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.505+0000 7fa63a59c640 1 -- 172.21.15.107:0/3917046959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa624003530 con 0x7fa6240039b0 2023-10-25T00:10:40.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63a59c640 1 --2- 172.21.15.107:0/3917046959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 0x7fa624003dd0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fa630009ae0 tx=0x7fa63001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec07a6222479f6c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:40.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6300a5570 con 0x7fa6240039b0 2023-10-25T00:10:40.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6300a56d0 con 0x7fa6240039b0 2023-10-25T00:10:40.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63959a640 1 -- 172.21.15.107:0/3917046959 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6300a5880 con 0x7fa6240039b0 2023-10-25T00:10:40.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 -- 172.21.15.107:0/3917046959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 msgr2=0x7fa624003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 --2- 172.21.15.107:0/3917046959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 0x7fa624003dd0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fa630009ae0 tx=0x7fa63001ca90 comp rx=0 tx=0).stop 2023-10-25T00:10:40.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63ad9d640 1 -- 172.21.15.107:0/3917046959 reap_dead start 2023-10-25T00:10:40.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 -- 172.21.15.107:0/3917046959 shutdown_connections 2023-10-25T00:10:40.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 -- 172.21.15.107:0/3917046959 >> 172.21.15.107:0/3917046959 conn(0x7fa63c0ca200 msgr2=0x7fa63c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:40.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 -- 172.21.15.107:0/3917046959 shutdown_connections 2023-10-25T00:10:40.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 -- 172.21.15.107:0/3917046959 wait complete. 2023-10-25T00:10:40.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 Processor -- start 2023-10-25T00:10:40.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 -- start start 2023-10-25T00:10:40.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 0x7fa63c0ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c0c1240 0x7fa63c14e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa63c158690 0x7fa63c14e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c0c2250 con 0x7fa6240039b0 2023-10-25T00:10:40.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c0c16d0 con 0x7fa63c158690 2023-10-25T00:10:40.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63c0c19b0 con 0x7fa63c0c1240 2023-10-25T00:10:40.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 0x7fa63c0ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 0x7fa63c0ac0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47838/0 (socket says 172.21.15.107:47838) 2023-10-25T00:10:40.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63a59c640 1 -- 172.21.15.107:0/643662847 learned_addr learned my addr 172.21.15.107:0/643662847 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:40.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa639d9b640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c0c1240 0x7fa63c14e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63ad9d640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa63c158690 0x7fa63c14e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63a59c640 1 -- 172.21.15.107:0/643662847 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c0c1240 msgr2=0x7fa63c14e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63a59c640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c0c1240 0x7fa63c14e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63a59c640 1 -- 172.21.15.107:0/643662847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa63c158690 msgr2=0x7fa63c14e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63a59c640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa63c158690 0x7fa63c14e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63a59c640 1 -- 172.21.15.107:0/643662847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63c0c1c90 con 0x7fa6240039b0 2023-10-25T00:10:40.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63ad9d640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa63c158690 0x7fa63c14e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:10:40.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa63a59c640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 0x7fa63c0ac0a0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fa6300b0040 tx=0x7fa624005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:40.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa62affd640 1 -- 172.21.15.107:0/643662847 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63000f000 con 0x7fa6240039b0 2023-10-25T00:10:40.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa62affd640 1 -- 172.21.15.107:0/643662847 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa630004020 con 0x7fa6240039b0 2023-10-25T00:10:40.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 -- 172.21.15.107:0/643662847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63c0c1f70 con 0x7fa6240039b0 2023-10-25T00:10:40.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa62affd640 1 -- 172.21.15.107:0/643662847 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6300043b0 con 0x7fa6240039b0 2023-10-25T00:10:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.509+0000 7fa640bc7640 1 -- 172.21.15.107:0/643662847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63c14ecb0 con 0x7fa6240039b0 2023-10-25T00:10:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.513+0000 7fa640bc7640 1 -- 172.21.15.107:0/643662847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa63c0c1af0 con 0x7fa6240039b0 2023-10-25T00:10:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.513+0000 7fa62affd640 1 -- 172.21.15.107:0/643662847 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa6300b4040 con 0x7fa6240039b0 2023-10-25T00:10:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.513+0000 7fa62affd640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa61c065e60 0x7fa61c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:40.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.513+0000 7fa62affd640 1 -- 172.21.15.107:0/643662847 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7fa63011dc10 con 0x7fa6240039b0 2023-10-25T00:10:40.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.513+0000 7fa639d9b640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa61c065e60 0x7fa61c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:40.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.513+0000 7fa639d9b640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa61c065e60 0x7fa61c068320 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0aca50 tx=0x7fa62c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:40.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.517+0000 7fa62affd640 1 -- 172.21.15.107:0/643662847 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa63c0c1af0 con 0x7fa6240039b0 2023-10-25T00:10:40.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 -- 172.21.15.107:0/643662847 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa63c0c1af0 con 0x7fa6240039b0 2023-10-25T00:10:40.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa62affd640 1 -- 172.21.15.107:0/643662847 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5019 (secure 0 0 0) 0x7fa63c0c1af0 con 0x7fa6240039b0 2023-10-25T00:10:40.698 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:40.698 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:39.839906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:reconnect","state_seq":20,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:40.699 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 152 2023-10-25T00:10:40.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 -- 172.21.15.107:0/643662847 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa61c065e60 msgr2=0x7fa61c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa61c065e60 0x7fa61c068320 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0aca50 tx=0x7fa62c009040 comp rx=0 tx=0).stop 2023-10-25T00:10:40.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 -- 172.21.15.107:0/643662847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 msgr2=0x7fa63c0ac0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:40.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 0x7fa63c0ac0a0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fa6300b0040 tx=0x7fa624005800 comp rx=0 tx=0).stop 2023-10-25T00:10:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 -- 172.21.15.107:0/643662847 shutdown_connections 2023-10-25T00:10:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa63c158690 0x7fa63c14e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa61c065e60 0x7fa61c068320 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa63c0c1240 0x7fa63c14e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 --2- 172.21.15.107:0/643662847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6240039b0 0x7fa63c0ac0a0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 -- 172.21.15.107:0/643662847 >> 172.21.15.107:0/643662847 conn(0x7fa63c0ca200 msgr2=0x7fa63c14f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 -- 172.21.15.107:0/643662847 shutdown_connections 2023-10-25T00:10:40.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:40.697+0000 7fa640bc7640 1 -- 172.21.15.107:0/643662847 wait complete. 2023-10-25T00:10:40.716 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 152, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 152, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7746": { "addr": "172.21.15.107:6835/3474118649", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3474118649, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 3474118649, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7746, "incarnation": 151, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:10:39.839906+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7746 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6568": { "addr": "172.21.15.110:6835/326715537", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 326715537, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 326715537, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6568, "incarnation": 146, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:09:21.753141+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6568 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:10:42.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:42.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bf7be640 1 Processor -- start 2023-10-25T00:10:42.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bf7be640 1 -- start start 2023-10-25T00:10:42.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bf7be640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0b8156770 0x7fc0b80ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:42.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bf7be640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc0b80c70e0 0x7fc0b80bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:42.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bf7be640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80c1ca0 con 0x7fc0b80c2130 2023-10-25T00:10:42.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bf7be640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b80c27e0 con 0x7fc0b8156770 2023-10-25T00:10:42.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bf7be640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b80c10c0 con 0x7fc0b80c70e0 2023-10-25T00:10:42.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bd55a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc0b8156770 0x7fc0b80ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:42.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bdd5b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc0b80c2130 0x7fc0b8069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35156/0 (socket says 172.21.15.107:35156) 2023-10-25T00:10:42.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bdd5b640 1 -- 172.21.15.107:0/510893267 learned_addr learned my addr 172.21.15.107:0/510893267 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:42.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bcd59640 1 --2- 172.21.15.107:0/510893267 >> v2:172.21.15.110:3300/0 conn(0x7fc0b80c70e0 0x7fc0b80bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:42.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bd55a640 1 -- 172.21.15.107:0/510893267 >> v2:172.21.15.110:3300/0 conn(0x7fc0b80c70e0 msgr2=0x7fc0b80bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:42.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bd55a640 1 --2- 172.21.15.107:0/510893267 >> v2:172.21.15.110:3300/0 conn(0x7fc0b80c70e0 0x7fc0b80bdfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:42.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bd55a640 1 -- 172.21.15.107:0/510893267 >> v1:172.21.15.107:6789/0 conn(0x7fc0b80c2130 legacy=0x7fc0b8069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:42.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0affff640 1 -- 172.21.15.107:0/510893267 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698689133 0 0) 0x7fc0b80c1ca0 con 0x7fc0b80c2130 2023-10-25T00:10:42.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.869+0000 7fc0bd55a640 1 -- 172.21.15.107:0/510893267 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b80c13a0 con 0x7fc0b8156770 2023-10-25T00:10:42.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0bd55a640 1 --2- 172.21.15.107:0/510893267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b8156770 0x7fc0b80ab7c0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80a8180 tx=0x7fc0b401c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59b8b387737e57d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:42.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0affff640 1 -- 172.21.15.107:0/510893267 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b40ab600 con 0x7fc0b8156770 2023-10-25T00:10:42.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0affff640 1 -- 172.21.15.107:0/510893267 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b40a7050 con 0x7fc0b8156770 2023-10-25T00:10:42.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0affff640 1 -- 172.21.15.107:0/510893267 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b40ab980 con 0x7fc0b8156770 2023-10-25T00:10:42.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0bf7be640 1 -- 172.21.15.107:0/510893267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b8156770 msgr2=0x7fc0b80ab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:42.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0bf7be640 1 --2- 172.21.15.107:0/510893267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b8156770 0x7fc0b80ab7c0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80a8180 tx=0x7fc0b401c310 comp rx=0 tx=0).stop 2023-10-25T00:10:42.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0bf7be640 1 -- 172.21.15.107:0/510893267 shutdown_connections 2023-10-25T00:10:42.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0bf7be640 1 --2- 172.21.15.107:0/510893267 >> v2:172.21.15.110:3300/0 conn(0x7fc0b80c70e0 0x7fc0b80bdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:42.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0bf7be640 1 --2- 172.21.15.107:0/510893267 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b8156770 0x7fc0b80ab7c0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:42.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0bf7be640 1 -- 172.21.15.107:0/510893267 >> 172.21.15.107:0/510893267 conn(0x7fc0b80c65c0 msgr2=0x7fc0b80c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:42.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0bf7be640 1 -- 172.21.15.107:0/510893267 shutdown_connections 2023-10-25T00:10:42.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0bf7be640 1 -- 172.21.15.107:0/510893267 wait complete. 2023-10-25T00:10:42.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0bf7be640 1 Processor -- start 2023-10-25T00:10:42.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.873+0000 7fc0bf7be640 1 -- start start 2023-10-25T00:10:42.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bf7be640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b80c2130 0x7fc0b81ee060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:42.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bf7be640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b80c70e0 0x7fc0b80bf770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:42.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bf7be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b8156770 0x7fc0b815c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:42.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bf7be640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b815c170 con 0x7fc0b8156770 2023-10-25T00:10:42.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bf7be640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b81ee5a0 con 0x7fc0b80c70e0 2023-10-25T00:10:42.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bf7be640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b815d090 con 0x7fc0b80c2130 2023-10-25T00:10:42.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bdd5b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b8156770 0x7fc0b815c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bdd5b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b8156770 0x7fc0b815c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38634/0 (socket says 172.21.15.107:38634) 2023-10-25T00:10:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bdd5b640 1 -- 172.21.15.107:0/2330398162 learned_addr learned my addr 172.21.15.107:0/2330398162 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bcd59640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b80c70e0 0x7fc0b80bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bd55a640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b80c2130 0x7fc0b81ee060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:42.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bdd5b640 1 -- 172.21.15.107:0/2330398162 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b80c2130 msgr2=0x7fc0b81ee060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:42.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bdd5b640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b80c2130 0x7fc0b81ee060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:42.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bdd5b640 1 -- 172.21.15.107:0/2330398162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b80c70e0 msgr2=0x7fc0b80bf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:42.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bdd5b640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b80c70e0 0x7fc0b80bf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:42.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bdd5b640 1 -- 172.21.15.107:0/2330398162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b4009420 con 0x7fc0b8156770 2023-10-25T00:10:42.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bdd5b640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b8156770 0x7fc0b815c590 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000c810 tx=0x7fc0b000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:42.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0adffb640 1 -- 172.21.15.107:0/2330398162 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b000a640 con 0x7fc0b8156770 2023-10-25T00:10:42.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0adffb640 1 -- 172.21.15.107:0/2330398162 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b000a7c0 con 0x7fc0b8156770 2023-10-25T00:10:42.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bf7be640 1 -- 172.21.15.107:0/2330398162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b815d370 con 0x7fc0b8156770 2023-10-25T00:10:42.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0adffb640 1 -- 172.21.15.107:0/2330398162 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b000ab70 con 0x7fc0b8156770 2023-10-25T00:10:42.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bf7be640 1 -- 172.21.15.107:0/2330398162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b815bd90 con 0x7fc0b8156770 2023-10-25T00:10:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bf7be640 1 -- 172.21.15.107:0/2330398162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b815c2b0 con 0x7fc0b8156770 2023-10-25T00:10:42.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0adffb640 1 -- 172.21.15.107:0/2330398162 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc0b000acd0 con 0x7fc0b8156770 2023-10-25T00:10:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0adffb640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc09c065e60 0x7fc09c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0adffb640 1 -- 172.21.15.107:0/2330398162 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7fc0b0083dd0 con 0x7fc0b8156770 2023-10-25T00:10:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.877+0000 7fc0bd55a640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc09c065e60 0x7fc09c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.881+0000 7fc0bd55a640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc09c065e60 0x7fc09c068320 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40ad000 tx=0x7fc0b40a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:42.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:42.885+0000 7fc0adffb640 1 -- 172.21.15.107:0/2330398162 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc0b0050720 con 0x7fc0b8156770 2023-10-25T00:10:43.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.081+0000 7fc0bf7be640 1 -- 172.21.15.107:0/2330398162 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0b80746e0 con 0x7fc0b8156770 2023-10-25T00:10:43.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0adffb640 1 -- 172.21.15.107:0/2330398162 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5016 (secure 0 0 0) 0x7fc0b0018020 con 0x7fc0b8156770 2023-10-25T00:10:43.085 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:43.085 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:43.086 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 154 2023-10-25T00:10:43.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 -- 172.21.15.107:0/2330398162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc09c065e60 msgr2=0x7fc09c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:43.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc09c065e60 0x7fc09c068320 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40ad000 tx=0x7fc0b40a6000 comp rx=0 tx=0).stop 2023-10-25T00:10:43.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 -- 172.21.15.107:0/2330398162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b8156770 msgr2=0x7fc0b815c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:43.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b8156770 0x7fc0b815c590 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000c810 tx=0x7fc0b000ccd0 comp rx=0 tx=0).stop 2023-10-25T00:10:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 -- 172.21.15.107:0/2330398162 shutdown_connections 2023-10-25T00:10:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0b8156770 0x7fc0b815c590 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b80c70e0 0x7fc0b80bf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc09c065e60 0x7fc09c068320 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:43.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 --2- 172.21.15.107:0/2330398162 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0b80c2130 0x7fc0b81ee060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 -- 172.21.15.107:0/2330398162 >> 172.21.15.107:0/2330398162 conn(0x7fc0b80c65c0 msgr2=0x7fc0b81549a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 -- 172.21.15.107:0/2330398162 shutdown_connections 2023-10-25T00:10:43.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:43.085+0000 7fc0bf7be640 1 -- 172.21.15.107:0/2330398162 wait complete. 2023-10-25T00:10:43.104 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=7578 2023-10-25T00:10:43.104 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.a 2023-10-25T00:10:44.785 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:10:44.785+0000 7f7ff0492640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:10:46.106 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2023-10-25T00:10:46.106 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:10:46.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:10:46.109 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:10:46.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:46.196 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:10:46.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd61843f640 1 Processor -- start 2023-10-25T00:10:46.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd61843f640 1 -- start start 2023-10-25T00:10:46.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd61843f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd610151480 0x7fd6100704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:46.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd61843f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6100c44c0 con 0x7fd610071770 2023-10-25T00:10:46.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd61843f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6100c38b0 con 0x7fd6100c2790 2023-10-25T00:10:46.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd61843f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd610071b50 con 0x7fd610151480 2023-10-25T00:10:46.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6161db640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd610071770 0x7fd61006cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35172/0 (socket says 172.21.15.107:35172) 2023-10-25T00:10:46.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6161db640 1 -- 172.21.15.107:0/512127832 learned_addr learned my addr 172.21.15.107:0/512127832 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:46.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943809866 0 0) 0x7fd6100c44c0 con 0x7fd610071770 2023-10-25T00:10:46.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5fc003530 con 0x7fd610071770 2023-10-25T00:10:46.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6169dc640 1 --2- 172.21.15.107:0/512127832 >> v2:172.21.15.110:3300/0 conn(0x7fd610151480 0x7fd6100704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:46.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1423059319 0 0) 0x7fd5fc003530 con 0x7fd610071770 2023-10-25T00:10:46.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 >> v2:172.21.15.110:3300/0 conn(0x7fd610151480 msgr2=0x7fd6100704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:46.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 --2- 172.21.15.107:0/512127832 >> v2:172.21.15.110:3300/0 conn(0x7fd610151480 0x7fd6100704f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:46.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 >> v1:172.21.15.110:6789/0 conn(0x7fd6100c2790 legacy=0x7fd61006fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:46.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6100c3b90 con 0x7fd610071770 2023-10-25T00:10:46.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fd60c001680 con 0x7fd610071770 2023-10-25T00:10:46.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 >> v1:172.21.15.107:6789/0 conn(0x7fd610071770 legacy=0x7fd61006cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:46.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5fc003980 0x7fd5fc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:46.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5fc0047d0 0x7fd5fc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:46.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 0x7fd5fc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:46.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd610071b50 con 0x7fd5fc005390 2023-10-25T00:10:46.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc00c220 con 0x7fd5fc003980 2023-10-25T00:10:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc00c3e0 con 0x7fd5fc0047d0 2023-10-25T00:10:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6169dc640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 0x7fd5fc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6161db640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5fc003980 0x7fd5fc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6159da640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5fc0047d0 0x7fd5fc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:46.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6169dc640 1 -- 172.21.15.107:0/512127832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5fc0047d0 msgr2=0x7fd5fc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6169dc640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5fc0047d0 0x7fd5fc004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6169dc640 1 -- 172.21.15.107:0/512127832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5fc003980 msgr2=0x7fd5fc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6169dc640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5fc003980 0x7fd5fc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6169dc640 1 -- 172.21.15.107:0/512127832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5fc003530 con 0x7fd5fc005390 2023-10-25T00:10:46.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.253+0000 7fd6161db640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5fc003980 0x7fd5fc003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:10:46.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd6169dc640 1 -- 172.21.15.107:0/512127832 reap_dead start 2023-10-25T00:10:46.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd6169dc640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 0x7fd5fc00b990 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fd608007360 tx=0x7fd60801ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14948df5f79641df server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:46.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd608003ba0 con 0x7fd5fc005390 2023-10-25T00:10:46.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd608003d00 con 0x7fd5fc005390 2023-10-25T00:10:46.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd6151d9640 1 -- 172.21.15.107:0/512127832 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6080a5b80 con 0x7fd5fc005390 2023-10-25T00:10:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd61843f640 1 -- 172.21.15.107:0/512127832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 msgr2=0x7fd5fc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd61843f640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 0x7fd5fc00b990 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fd608007360 tx=0x7fd60801ce60 comp rx=0 tx=0).stop 2023-10-25T00:10:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd61843f640 1 -- 172.21.15.107:0/512127832 shutdown_connections 2023-10-25T00:10:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd61843f640 1 --2- 172.21.15.107:0/512127832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 0x7fd5fc00b990 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd61843f640 1 -- 172.21.15.107:0/512127832 >> 172.21.15.107:0/512127832 conn(0x7fd6100c6ed0 msgr2=0x7fd6100c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:46.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd61843f640 1 -- 172.21.15.107:0/512127832 shutdown_connections 2023-10-25T00:10:46.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd61843f640 1 -- 172.21.15.107:0/512127832 wait complete. 2023-10-25T00:10:46.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd61843f640 1 Processor -- start 2023-10-25T00:10:46.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.257+0000 7fd61843f640 1 -- start start 2023-10-25T00:10:46.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd61843f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 0x7fd610154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd61843f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6101ee530 0x7fd61014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd61843f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6101493e0 0x7fd610149890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd61843f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61014ac10 con 0x7fd5fc005390 2023-10-25T00:10:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd61843f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6101489d0 con 0x7fd6101ee530 2023-10-25T00:10:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd61843f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd610155480 con 0x7fd6101493e0 2023-10-25T00:10:46.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6161db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 0x7fd610154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6161db640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 0x7fd610154e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38660/0 (socket says 172.21.15.107:38660) 2023-10-25T00:10:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6161db640 1 -- 172.21.15.107:0/4133754280 learned_addr learned my addr 172.21.15.107:0/4133754280 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6169dc640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6101493e0 0x7fd610149890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6159da640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6101ee530 0x7fd61014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:46.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6161db640 1 -- 172.21.15.107:0/4133754280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6101493e0 msgr2=0x7fd610149890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6161db640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6101493e0 0x7fd610149890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6161db640 1 -- 172.21.15.107:0/4133754280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6101ee530 msgr2=0x7fd61014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6161db640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6101ee530 0x7fd61014fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6161db640 1 -- 172.21.15.107:0/4133754280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd610155760 con 0x7fd5fc005390 2023-10-25T00:10:46.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6159da640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6101ee530 0x7fd61014fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:10:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd6161db640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 0x7fd610154e30 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fd60c007fd0 tx=0x7fd60c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd606ffd640 1 -- 172.21.15.107:0/4133754280 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c0072d0 con 0x7fd5fc005390 2023-10-25T00:10:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd606ffd640 1 -- 172.21.15.107:0/4133754280 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd60c007430 con 0x7fd5fc005390 2023-10-25T00:10:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd61843f640 1 -- 172.21.15.107:0/4133754280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd610148be0 con 0x7fd5fc005390 2023-10-25T00:10:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd606ffd640 1 -- 172.21.15.107:0/4133754280 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c0047f0 con 0x7fd5fc005390 2023-10-25T00:10:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd61843f640 1 -- 172.21.15.107:0/4133754280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6101ee970 con 0x7fd5fc005390 2023-10-25T00:10:46.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd61843f640 1 -- 172.21.15.107:0/4133754280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6100746e0 con 0x7fd5fc005390 2023-10-25T00:10:46.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.261+0000 7fd606ffd640 1 -- 172.21.15.107:0/4133754280 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd60c00b6b0 con 0x7fd5fc005390 2023-10-25T00:10:46.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.265+0000 7fd606ffd640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd5f4065e90 0x7fd5f4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:46.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.265+0000 7fd606ffd640 1 -- 172.21.15.107:0/4133754280 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7fd60c084230 con 0x7fd5fc005390 2023-10-25T00:10:46.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.265+0000 7fd6159da640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd5f4065e90 0x7fd5f4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:46.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.265+0000 7fd6159da640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd5f4065e90 0x7fd5f4068350 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd600000c10 tx=0x7fd600009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:46.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.265+0000 7fd606ffd640 1 -- 172.21.15.107:0/4133754280 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd60c050b30 con 0x7fd5fc005390 2023-10-25T00:10:46.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.509+0000 7fd61843f640 1 -- 172.21.15.107:0/4133754280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd610149090 con 0x7fd5fc005390 2023-10-25T00:10:46.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.509+0000 7fd606ffd640 1 -- 172.21.15.107:0/4133754280 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5016 (secure 0 0 0) 0x7fd60c019070 con 0x7fd5fc005390 2023-10-25T00:10:46.513 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:46.513 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:46.513 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 154 2023-10-25T00:10:46.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 -- 172.21.15.107:0/4133754280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd5f4065e90 msgr2=0x7fd5f4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:46.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd5f4065e90 0x7fd5f4068350 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd600000c10 tx=0x7fd600009040 comp rx=0 tx=0).stop 2023-10-25T00:10:46.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 -- 172.21.15.107:0/4133754280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 msgr2=0x7fd610154e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:46.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 0x7fd610154e30 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fd60c007fd0 tx=0x7fd60c00dc00 comp rx=0 tx=0).stop 2023-10-25T00:10:46.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 -- 172.21.15.107:0/4133754280 shutdown_connections 2023-10-25T00:10:46.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd5f4065e90 0x7fd5f4068350 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:46.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6101493e0 0x7fd610149890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:46.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6101ee530 0x7fd61014fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:46.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 --2- 172.21.15.107:0/4133754280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5fc005390 0x7fd610154e30 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:46.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 -- 172.21.15.107:0/4133754280 >> 172.21.15.107:0/4133754280 conn(0x7fd6100c6ed0 msgr2=0x7fd6101519a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:46.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 -- 172.21.15.107:0/4133754280 shutdown_connections 2023-10-25T00:10:46.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:46.513+0000 7fd61843f640 1 -- 172.21.15.107:0/4133754280 wait complete. 2023-10-25T00:10:46.536 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-25T00:10:46.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:47.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc7624fd640 1 Processor -- start 2023-10-25T00:10:47.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc7624fd640 1 -- start start 2023-10-25T00:10:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc7624fd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc75c152040 0x7fc75c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc7624fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc75c0c8100 0x7fc75c0abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc7624fd640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75c155680 con 0x7fc75c06a4b0 2023-10-25T00:10:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc7624fd640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75c154a70 con 0x7fc75c152040 2023-10-25T00:10:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc7624fd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c154d50 con 0x7fc75c0c8100 2023-10-25T00:10:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc75bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc75c152040 0x7fc75c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc760a9a640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc75c06a4b0 0x7fc75c0ac240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35174/0 (socket says 172.21.15.107:35174) 2023-10-25T00:10:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc760a9a640 1 -- 172.21.15.107:0/2003043743 learned_addr learned my addr 172.21.15.107:0/2003043743 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc753fff640 1 --2- 172.21.15.107:0/2003043743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc75c0c8100 0x7fc75c0abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc75b7fe640 1 -- 172.21.15.107:0/2003043743 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631600978 0 0) 0x7fc75c155680 con 0x7fc75c06a4b0 2023-10-25T00:10:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc75bfff640 1 -- 172.21.15.107:0/2003043743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc75c0c8100 msgr2=0x7fc75c0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:47.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc75bfff640 1 --2- 172.21.15.107:0/2003043743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc75c0c8100 0x7fc75c0abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc75bfff640 1 -- 172.21.15.107:0/2003043743 >> v1:172.21.15.107:6789/0 conn(0x7fc75c06a4b0 legacy=0x7fc75c0ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.085+0000 7fc75bfff640 1 -- 172.21.15.107:0/2003043743 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c155030 con 0x7fc75c152040 2023-10-25T00:10:47.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc75bfff640 1 --2- 172.21.15.107:0/2003043743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc75c152040 0x7fc75c0c4b80 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097c0 tx=0x7fc74c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664c95f042ded88d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:47.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc75b7fe640 1 -- 172.21.15.107:0/2003043743 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c0ab600 con 0x7fc75c152040 2023-10-25T00:10:47.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc75b7fe640 1 -- 172.21.15.107:0/2003043743 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c0a7050 con 0x7fc75c152040 2023-10-25T00:10:47.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc75b7fe640 1 -- 172.21.15.107:0/2003043743 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c0ab980 con 0x7fc75c152040 2023-10-25T00:10:47.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc7624fd640 1 -- 172.21.15.107:0/2003043743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc75c152040 msgr2=0x7fc75c0c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc7624fd640 1 --2- 172.21.15.107:0/2003043743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc75c152040 0x7fc75c0c4b80 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097c0 tx=0x7fc74c01c3f0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc7624fd640 1 -- 172.21.15.107:0/2003043743 shutdown_connections 2023-10-25T00:10:47.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc7624fd640 1 --2- 172.21.15.107:0/2003043743 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc75c0c8100 0x7fc75c0abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc7624fd640 1 --2- 172.21.15.107:0/2003043743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc75c152040 0x7fc75c0c4b80 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc7624fd640 1 -- 172.21.15.107:0/2003043743 >> 172.21.15.107:0/2003043743 conn(0x7fc75c0ca200 msgr2=0x7fc75c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:47.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc7624fd640 1 -- 172.21.15.107:0/2003043743 shutdown_connections 2023-10-25T00:10:47.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.089+0000 7fc7624fd640 1 -- 172.21.15.107:0/2003043743 wait complete. 2023-10-25T00:10:47.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7624fd640 1 Processor -- start 2023-10-25T00:10:47.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7624fd640 1 -- start start 2023-10-25T00:10:47.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7624fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc75c06a4b0 0x7fc75c143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7624fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc75c0c8100 0x7fc75c0c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7624fd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc75c152040 0x7fc75c1ed1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7624fd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c152a80 con 0x7fc75c06a4b0 2023-10-25T00:10:47.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7624fd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c144be0 con 0x7fc75c152040 2023-10-25T00:10:47.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7624fd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75c144ec0 con 0x7fc75c0c8100 2023-10-25T00:10:47.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc75bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc75c06a4b0 0x7fc75c143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc75bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc75c06a4b0 0x7fc75c143520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38670/0 (socket says 172.21.15.107:38670) 2023-10-25T00:10:47.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc75bfff640 1 -- 172.21.15.107:0/3244897350 learned_addr learned my addr 172.21.15.107:0/3244897350 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:47.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc753fff640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc75c0c8100 0x7fc75c0c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc760a9a640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc75c152040 0x7fc75c1ed1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc75bfff640 1 -- 172.21.15.107:0/3244897350 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc75c0c8100 msgr2=0x7fc75c0c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc75bfff640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc75c0c8100 0x7fc75c0c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc75bfff640 1 -- 172.21.15.107:0/3244897350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc75c152040 msgr2=0x7fc75c1ed1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc75bfff640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc75c152040 0x7fc75c1ed1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc75bfff640 1 -- 172.21.15.107:0/3244897350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c009420 con 0x7fc75c06a4b0 2023-10-25T00:10:47.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc75bfff640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc75c06a4b0 0x7fc75c143520 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0ae040 tx=0x7fc74c01ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:47.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7597fa640 1 -- 172.21.15.107:0/3244897350 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c0b7450 con 0x7fc75c06a4b0 2023-10-25T00:10:47.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7597fa640 1 -- 172.21.15.107:0/3244897350 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c0b75b0 con 0x7fc75c06a4b0 2023-10-25T00:10:47.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7624fd640 1 -- 172.21.15.107:0/3244897350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc75c1451a0 con 0x7fc75c06a4b0 2023-10-25T00:10:47.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7597fa640 1 -- 172.21.15.107:0/3244897350 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c0b7960 con 0x7fc75c06a4b0 2023-10-25T00:10:47.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7624fd640 1 -- 172.21.15.107:0/3244897350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc75c1eccd0 con 0x7fc75c06a4b0 2023-10-25T00:10:47.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.093+0000 7fc7624fd640 1 -- 172.21.15.107:0/3244897350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75c0746e0 con 0x7fc75c06a4b0 2023-10-25T00:10:47.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.097+0000 7fc7597fa640 1 -- 172.21.15.107:0/3244897350 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc74c0b7ac0 con 0x7fc75c06a4b0 2023-10-25T00:10:47.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.097+0000 7fc7597fa640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc73c065e60 0x7fc73c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.097+0000 7fc7597fa640 1 -- 172.21.15.107:0/3244897350 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7fc74c11eeb0 con 0x7fc75c06a4b0 2023-10-25T00:10:47.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.097+0000 7fc753fff640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc73c065e60 0x7fc73c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.097+0000 7fc753fff640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc73c065e60 0x7fc73c068320 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc75c15efb0 tx=0x7fc744009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:47.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.101+0000 7fc7597fa640 1 -- 172.21.15.107:0/3244897350 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc74c0eb920 con 0x7fc75c06a4b0 2023-10-25T00:10:47.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.353+0000 7fc7624fd640 1 -- 172.21.15.107:0/3244897350 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc75c0bd210 con 0x7fc75c06a4b0 2023-10-25T00:10:47.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.353+0000 7fc7597fa640 1 -- 172.21.15.107:0/3244897350 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5777 (secure 0 0 0) 0x7fc74c0b0020 con 0x7fc75c06a4b0 2023-10-25T00:10:47.358 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:47.358 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:47.359 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 155 2023-10-25T00:10:47.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 -- 172.21.15.107:0/3244897350 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc73c065e60 msgr2=0x7fc73c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc73c065e60 0x7fc73c068320 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fc75c15efb0 tx=0x7fc744009190 comp rx=0 tx=0).stop 2023-10-25T00:10:47.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 -- 172.21.15.107:0/3244897350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc75c06a4b0 msgr2=0x7fc75c143520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc75c06a4b0 0x7fc75c143520 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0ae040 tx=0x7fc74c01ac30 comp rx=0 tx=0).stop 2023-10-25T00:10:47.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 -- 172.21.15.107:0/3244897350 shutdown_connections 2023-10-25T00:10:47.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc75c152040 0x7fc75c1ed1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc73c065e60 0x7fc73c068320 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc75c0c8100 0x7fc75c0c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 --2- 172.21.15.107:0/3244897350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc75c06a4b0 0x7fc75c143520 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 -- 172.21.15.107:0/3244897350 >> 172.21.15.107:0/3244897350 conn(0x7fc75c0ca200 msgr2=0x7fc75c0adbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:47.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 -- 172.21.15.107:0/3244897350 shutdown_connections 2023-10-25T00:10:47.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.357+0000 7fc7624fd640 1 -- 172.21.15.107:0/3244897350 wait complete. 2023-10-25T00:10:47.376 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2023-10-25T00:10:47.376 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-25T00:10:47.377 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:10:47.399 INFO:tasks.ceph.mds.d:Stopped 2023-10-25T00:10:47.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:47.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.513+0000 7f6e166a3640 1 Processor -- start 2023-10-25T00:10:47.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.513+0000 7f6e166a3640 1 -- start start 2023-10-25T00:10:47.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.513+0000 7f6e166a3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e10156770 0x7f6e100ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.513+0000 7f6e166a3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6e100be6b0 0x7f6e1015d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.513+0000 7f6e166a3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e100c1ca0 con 0x7f6e100c70e0 2023-10-25T00:10:47.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.513+0000 7f6e166a3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e100c27e0 con 0x7f6e100be6b0 2023-10-25T00:10:47.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.513+0000 7f6e166a3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100c10c0 con 0x7f6e10156770 2023-10-25T00:10:47.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.513+0000 7f6e0ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e10156770 0x7f6e100ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.513+0000 7f6e0f7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6e100c70e0 0x7f6e100bdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:48712/0 (socket says 172.21.15.107:48712) 2023-10-25T00:10:47.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e0f7fe640 1 -- 172.21.15.107:0/2165914334 learned_addr learned my addr 172.21.15.107:0/2165914334 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:47.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e14c40640 1 --2- 172.21.15.107:0/2165914334 >> v2:172.21.15.110:3300/0 conn(0x7f6e100be6b0 0x7f6e1015d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e0ffff640 1 -- 172.21.15.107:0/2165914334 >> v2:172.21.15.110:3300/0 conn(0x7f6e100be6b0 msgr2=0x7f6e1015d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:47.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e0ffff640 1 --2- 172.21.15.107:0/2165914334 >> v2:172.21.15.110:3300/0 conn(0x7f6e100be6b0 0x7f6e1015d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e0ffff640 1 -- 172.21.15.107:0/2165914334 >> v1:172.21.15.110:6789/0 conn(0x7f6e100c70e0 legacy=0x7f6e100bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e0ffff640 1 -- 172.21.15.107:0/2165914334 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e100c13a0 con 0x7f6e10156770 2023-10-25T00:10:47.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e14c40640 1 --2- 172.21.15.107:0/2165914334 >> v2:172.21.15.110:3300/0 conn(0x7f6e100be6b0 0x7f6e1015d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:10:47.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e0ffff640 1 --2- 172.21.15.107:0/2165914334 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e10156770 0x7f6e100ab7c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6e000097c0 tx=0x7f6e0001c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81658457da135810 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:47.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e0effd640 1 -- 172.21.15.107:0/2165914334 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e000ac640 con 0x7f6e10156770 2023-10-25T00:10:47.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e0effd640 1 -- 172.21.15.107:0/2165914334 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e000ac7a0 con 0x7f6e10156770 2023-10-25T00:10:47.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e0effd640 1 -- 172.21.15.107:0/2165914334 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e000acb70 con 0x7f6e10156770 2023-10-25T00:10:47.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e166a3640 1 -- 172.21.15.107:0/2165914334 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e10156770 msgr2=0x7f6e100ab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e166a3640 1 --2- 172.21.15.107:0/2165914334 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e10156770 0x7f6e100ab7c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6e000097c0 tx=0x7f6e0001c350 comp rx=0 tx=0).stop 2023-10-25T00:10:47.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e166a3640 1 -- 172.21.15.107:0/2165914334 shutdown_connections 2023-10-25T00:10:47.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e166a3640 1 --2- 172.21.15.107:0/2165914334 >> v2:172.21.15.110:3300/0 conn(0x7f6e100be6b0 0x7f6e1015d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e166a3640 1 --2- 172.21.15.107:0/2165914334 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e10156770 0x7f6e100ab7c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e166a3640 1 -- 172.21.15.107:0/2165914334 >> 172.21.15.107:0/2165914334 conn(0x7f6e100c65c0 msgr2=0x7f6e100c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:47.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e166a3640 1 -- 172.21.15.107:0/2165914334 shutdown_connections 2023-10-25T00:10:47.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.517+0000 7f6e166a3640 1 -- 172.21.15.107:0/2165914334 wait complete. 2023-10-25T00:10:47.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e166a3640 1 Processor -- start 2023-10-25T00:10:47.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e166a3640 1 -- start start 2023-10-25T00:10:47.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e166a3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e100be6b0 0x7f6e10070470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e166a3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e100c70e0 0x7f6e10153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e166a3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e10156770 0x7f6e100b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e166a3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1015cc50 con 0x7f6e100be6b0 2023-10-25T00:10:47.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e166a3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100709b0 con 0x7f6e100c70e0 2023-10-25T00:10:47.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e166a3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1015c970 con 0x7f6e10156770 2023-10-25T00:10:47.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e100be6b0 0x7f6e10070470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e100be6b0 0x7f6e10070470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38684/0 (socket says 172.21.15.107:38684) 2023-10-25T00:10:47.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0ffff640 1 -- 172.21.15.107:0/3506534975 learned_addr learned my addr 172.21.15.107:0/3506534975 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:47.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0f7fe640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e100c70e0 0x7f6e10153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e14c40640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e10156770 0x7f6e100b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0ffff640 1 -- 172.21.15.107:0/3506534975 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e10156770 msgr2=0x7f6e100b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0ffff640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e10156770 0x7f6e100b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0ffff640 1 -- 172.21.15.107:0/3506534975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e100c70e0 msgr2=0x7f6e10153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0ffff640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e100c70e0 0x7f6e10153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0ffff640 1 -- 172.21.15.107:0/3506534975 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e08009a70 con 0x7f6e100be6b0 2023-10-25T00:10:47.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0ffff640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e100be6b0 0x7f6e10070470 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f6e000af040 tx=0x7f6e0001ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:47.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0cff9640 1 -- 172.21.15.107:0/3506534975 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e000b8650 con 0x7f6e100be6b0 2023-10-25T00:10:47.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0cff9640 1 -- 172.21.15.107:0/3506534975 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e000b87d0 con 0x7f6e100be6b0 2023-10-25T00:10:47.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e166a3640 1 -- 172.21.15.107:0/3506534975 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e00009420 con 0x7f6e100be6b0 2023-10-25T00:10:47.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e0cff9640 1 -- 172.21.15.107:0/3506534975 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e000b8b80 con 0x7f6e100be6b0 2023-10-25T00:10:47.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e166a3640 1 -- 172.21.15.107:0/3506534975 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1006f9a0 con 0x7f6e100be6b0 2023-10-25T00:10:47.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.521+0000 7f6e166a3640 1 -- 172.21.15.107:0/3506534975 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1006f500 con 0x7f6e100be6b0 2023-10-25T00:10:47.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.525+0000 7f6e0cff9640 1 -- 172.21.15.107:0/3506534975 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6e000b8ce0 con 0x7f6e100be6b0 2023-10-25T00:10:47.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.525+0000 7f6e0cff9640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dfc065e60 0x7f6dfc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:47.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.525+0000 7f6e0f7fe640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dfc065e60 0x7f6dfc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:47.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.525+0000 7f6e0cff9640 1 -- 172.21.15.107:0/3506534975 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7f6e0011ef50 con 0x7f6e100be6b0 2023-10-25T00:10:47.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.525+0000 7f6e0f7fe640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dfc065e60 0x7f6dfc068320 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6e10072760 tx=0x7f6e04009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:47.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.529+0000 7f6e0cff9640 1 -- 172.21.15.107:0/3506534975 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6e000eb8a0 con 0x7f6e100be6b0 2023-10-25T00:10:47.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.745+0000 7f6e166a3640 1 -- 172.21.15.107:0/3506534975 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e100746e0 con 0x7f6e100be6b0 2023-10-25T00:10:47.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.745+0000 7f6e0cff9640 1 -- 172.21.15.107:0/3506534975 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f6e000b1020 con 0x7f6e100be6b0 2023-10-25T00:10:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:47.748 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:47.749 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 155 2023-10-25T00:10:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 -- 172.21.15.107:0/3506534975 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dfc065e60 msgr2=0x7f6dfc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dfc065e60 0x7f6dfc068320 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6e10072760 tx=0x7f6e04009040 comp rx=0 tx=0).stop 2023-10-25T00:10:47.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 -- 172.21.15.107:0/3506534975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e100be6b0 msgr2=0x7f6e10070470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e100be6b0 0x7f6e10070470 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f6e000af040 tx=0x7f6e0001ab90 comp rx=0 tx=0).stop 2023-10-25T00:10:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 -- 172.21.15.107:0/3506534975 shutdown_connections 2023-10-25T00:10:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6dfc065e60 0x7f6dfc068320 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e10156770 0x7f6e100b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e100c70e0 0x7f6e10153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 --2- 172.21.15.107:0/3506534975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e100be6b0 0x7f6e10070470 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:47.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 -- 172.21.15.107:0/3506534975 >> 172.21.15.107:0/3506534975 conn(0x7f6e100c65c0 msgr2=0x7f6e100c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:47.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 -- 172.21.15.107:0/3506534975 shutdown_connections 2023-10-25T00:10:47.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:47.749+0000 7f6e166a3640 1 -- 172.21.15.107:0/3506534975 wait complete. 2023-10-25T00:10:47.766 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:10:48.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:48.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc70736c640 1 Processor -- start 2023-10-25T00:10:48.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc70736c640 1 -- start start 2023-10-25T00:10:48.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc70736c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc700152020 0x7fc7000c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:48.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc70736c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc700152400 con 0x7fc7001553d0 2023-10-25T00:10:48.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc70736c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7001558d0 con 0x7fc7000c8640 2023-10-25T00:10:48.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc70736c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7000c6ce0 con 0x7fc700152020 2023-10-25T00:10:48.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc705909640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc7001553d0 0x7fc70006ae20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35184/0 (socket says 172.21.15.107:35184) 2023-10-25T00:10:48.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc705909640 1 -- 172.21.15.107:0/4186053195 learned_addr learned my addr 172.21.15.107:0/4186053195 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:48.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc705108640 1 --2- 172.21.15.107:0/4186053195 >> v2:172.21.15.110:3300/0 conn(0x7fc700152020 0x7fc7000c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:48.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759102920 0 0) 0x7fc700152400 con 0x7fc7001553d0 2023-10-25T00:10:48.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6ec003530 con 0x7fc7001553d0 2023-10-25T00:10:48.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645270110 0 0) 0x7fc7001558d0 con 0x7fc7000c8640 2023-10-25T00:10:48.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc700152400 con 0x7fc7000c8640 2023-10-25T00:10:48.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2511095162 0 0) 0x7fc6ec003530 con 0x7fc7001553d0 2023-10-25T00:10:48.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 >> v2:172.21.15.110:3300/0 conn(0x7fc700152020 msgr2=0x7fc7000c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:48.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 --2- 172.21.15.107:0/4186053195 >> v2:172.21.15.110:3300/0 conn(0x7fc700152020 0x7fc7000c4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:48.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 >> v1:172.21.15.110:6789/0 conn(0x7fc7000c8640 legacy=0x7fc70006a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:48.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7000c6fc0 con 0x7fc7001553d0 2023-10-25T00:10:48.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fc6f8001a20 con 0x7fc7001553d0 2023-10-25T00:10:48.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc705108640 1 --2- 172.21.15.107:0/4186053195 >> v2:172.21.15.110:3300/0 conn(0x7fc700152020 0x7fc7000c4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:10:48.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 >> v1:172.21.15.107:6789/0 conn(0x7fc7001553d0 legacy=0x7fc70006ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:48.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 --2- 172.21.15.107:0/4186053195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc6ec003b10 0x7fc6ec003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:48.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 --2- 172.21.15.107:0/4186053195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc6ec0047d0 0x7fc6ec004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:48.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 --2- 172.21.15.107:0/4186053195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6ec005190 0x7fc6ec00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:48.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000c6ce0 con 0x7fc6ec005190 2023-10-25T00:10:48.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec00c0c0 con 0x7fc6ec0047d0 2023-10-25T00:10:48.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec00c280 con 0x7fc6ec003b10 2023-10-25T00:10:48.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc705909640 1 --2- 172.21.15.107:0/4186053195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6ec005190 0x7fc6ec00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:48.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc704907640 1 --2- 172.21.15.107:0/4186053195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc6ec0047d0 0x7fc6ec004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:48.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc705108640 1 --2- 172.21.15.107:0/4186053195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc6ec003b10 0x7fc6ec003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:48.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc705909640 1 -- 172.21.15.107:0/4186053195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc6ec003b10 msgr2=0x7fc6ec003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:48.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc705909640 1 --2- 172.21.15.107:0/4186053195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc6ec003b10 0x7fc6ec003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:48.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc705909640 1 -- 172.21.15.107:0/4186053195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc6ec0047d0 msgr2=0x7fc6ec004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:48.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc705909640 1 --2- 172.21.15.107:0/4186053195 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc6ec0047d0 0x7fc6ec004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:48.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.657+0000 7fc705909640 1 -- 172.21.15.107:0/4186053195 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec003530 con 0x7fc6ec005190 2023-10-25T00:10:48.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc705909640 1 -- 172.21.15.107:0/4186053195 reap_dead start 2023-10-25T00:10:48.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc705909640 1 --2- 172.21.15.107:0/4186053195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6ec005190 0x7fc6ec00b990 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fc700155800 tx=0x7fc6f801d240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76e26ff63cc142f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:48.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80ac620 con 0x7fc6ec005190 2023-10-25T00:10:48.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f80a8070 con 0x7fc6ec005190 2023-10-25T00:10:48.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc6f7fff640 1 -- 172.21.15.107:0/4186053195 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80ac9e0 con 0x7fc6ec005190 2023-10-25T00:10:48.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc70736c640 1 -- 172.21.15.107:0/4186053195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6ec005190 msgr2=0x7fc6ec00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:48.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc70736c640 1 --2- 172.21.15.107:0/4186053195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6ec005190 0x7fc6ec00b990 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fc700155800 tx=0x7fc6f801d240 comp rx=0 tx=0).stop 2023-10-25T00:10:48.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc70736c640 1 -- 172.21.15.107:0/4186053195 shutdown_connections 2023-10-25T00:10:48.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc70736c640 1 --2- 172.21.15.107:0/4186053195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6ec005190 0x7fc6ec00b990 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:48.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc70736c640 1 -- 172.21.15.107:0/4186053195 >> 172.21.15.107:0/4186053195 conn(0x7fc7000abd60 msgr2=0x7fc7000ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:48.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc70736c640 1 -- 172.21.15.107:0/4186053195 shutdown_connections 2023-10-25T00:10:48.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc70736c640 1 -- 172.21.15.107:0/4186053195 wait complete. 2023-10-25T00:10:48.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc70736c640 1 Processor -- start 2023-10-25T00:10:48.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.661+0000 7fc70736c640 1 -- start start 2023-10-25T00:10:48.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc70736c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc6ec005190 0x7fc7000ad490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:48.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc70736c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc7000c09b0 0x7fc70014a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:48.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc70736c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70014acd0 0x7fc7001ecc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:48.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc70736c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000ae270 con 0x7fc70014acd0 2023-10-25T00:10:48.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc70736c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70014b150 con 0x7fc6ec005190 2023-10-25T00:10:48.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc70736c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7000afae0 con 0x7fc7000c09b0 2023-10-25T00:10:48.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705909640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70014acd0 0x7fc7001ecc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:48.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705909640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70014acd0 0x7fc7001ecc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38708/0 (socket says 172.21.15.107:38708) 2023-10-25T00:10:48.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705909640 1 -- 172.21.15.107:0/658266467 learned_addr learned my addr 172.21.15.107:0/658266467 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:48.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705108640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc6ec005190 0x7fc7000ad490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:48.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc704907640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc7000c09b0 0x7fc70014a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:48.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705909640 1 -- 172.21.15.107:0/658266467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc7000c09b0 msgr2=0x7fc70014a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:48.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705909640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc7000c09b0 0x7fc70014a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:48.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705909640 1 -- 172.21.15.107:0/658266467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc6ec005190 msgr2=0x7fc7000ad490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:48.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705909640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc6ec005190 0x7fc7000ad490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:48.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705909640 1 -- 172.21.15.107:0/658266467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7000c1520 con 0x7fc70014acd0 2023-10-25T00:10:48.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705909640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70014acd0 0x7fc7001ecc40 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fc6f801dae0 tx=0x7fc6ec005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:48.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc6f5ffb640 1 -- 172.21.15.107:0/658266467 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8004d70 con 0x7fc70014acd0 2023-10-25T00:10:48.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc6f5ffb640 1 -- 172.21.15.107:0/658266467 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f8005030 con 0x7fc70014acd0 2023-10-25T00:10:48.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc70736c640 1 -- 172.21.15.107:0/658266467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7000af1b0 con 0x7fc70014acd0 2023-10-25T00:10:48.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc6f5ffb640 1 -- 172.21.15.107:0/658266467 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f80053c0 con 0x7fc70014acd0 2023-10-25T00:10:48.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc70736c640 1 -- 172.21.15.107:0/658266467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7000af770 con 0x7fc70014acd0 2023-10-25T00:10:48.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc70736c640 1 -- 172.21.15.107:0/658266467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7000c7280 con 0x7fc70014acd0 2023-10-25T00:10:48.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc6f5ffb640 1 -- 172.21.15.107:0/658266467 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc6f80b5030 con 0x7fc70014acd0 2023-10-25T00:10:48.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc6f5ffb640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc6e4065e90 0x7fc6e4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:48.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc6f5ffb640 1 -- 172.21.15.107:0/658266467 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7fc6f80b3070 con 0x7fc70014acd0 2023-10-25T00:10:48.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705108640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc6e4065e90 0x7fc6e4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:48.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.665+0000 7fc705108640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc6e4065e90 0x7fc6e4068350 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0093f0 tx=0x7fc6fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:48.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.673+0000 7fc6f5ffb640 1 -- 172.21.15.107:0/658266467 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc7000c7280 con 0x7fc70014acd0 2023-10-25T00:10:48.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.913+0000 7fc70736c640 1 -- 172.21.15.107:0/658266467 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7000afc90 con 0x7fc70014acd0 2023-10-25T00:10:48.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.913+0000 7fc6f5ffb640 1 -- 172.21.15.107:0/658266467 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5777 (secure 0 0 0) 0x7fc7000afc90 con 0x7fc70014acd0 2023-10-25T00:10:48.915 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:48.916 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:48.916 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 155 2023-10-25T00:10:48.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 -- 172.21.15.107:0/658266467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc6e4065e90 msgr2=0x7fc6e4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:48.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc6e4065e90 0x7fc6e4068350 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0093f0 tx=0x7fc6fc008040 comp rx=0 tx=0).stop 2023-10-25T00:10:48.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 -- 172.21.15.107:0/658266467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70014acd0 msgr2=0x7fc7001ecc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:48.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70014acd0 0x7fc7001ecc40 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fc6f801dae0 tx=0x7fc6ec005800 comp rx=0 tx=0).stop 2023-10-25T00:10:48.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 -- 172.21.15.107:0/658266467 shutdown_connections 2023-10-25T00:10:48.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70014acd0 0x7fc7001ecc40 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:48.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc6e4065e90 0x7fc6e4068350 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:48.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc7000c09b0 0x7fc70014a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:48.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 --2- 172.21.15.107:0/658266467 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc6ec005190 0x7fc7000ad490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:48.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 -- 172.21.15.107:0/658266467 >> 172.21.15.107:0/658266467 conn(0x7fc7000abd60 msgr2=0x7fc70014ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:48.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 -- 172.21.15.107:0/658266467 shutdown_connections 2023-10-25T00:10:48.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:48.917+0000 7fc70736c640 1 -- 172.21.15.107:0/658266467 wait complete. 2023-10-25T00:10:48.936 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2023-10-25T00:10:48.936 INFO:tasks.mds_thrash.fs.[a]:waiting for 15 secs before thrashing 2023-10-25T00:10:49.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:49.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4207aa9640 1 Processor -- start 2023-10-25T00:10:49.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4207aa9640 1 -- start start 2023-10-25T00:10:49.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4207aa9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4200152040 0x7f4200158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:49.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4207aa9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f420006e800 0x7f420006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:49.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4207aa9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4200152420 con 0x7f42000adb80 2023-10-25T00:10:49.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4207aa9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42001556e0 con 0x7f4200152040 2023-10-25T00:10:49.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4207aa9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4200154d40 con 0x7f420006e800 2023-10-25T00:10:49.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4205845640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4200152040 0x7f4200158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:49.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4205845640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4200152040 0x7f4200158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38710/0 (socket says 172.21.15.107:38710) 2023-10-25T00:10:49.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4205845640 1 -- 172.21.15.107:0/4267294102 learned_addr learned my addr 172.21.15.107:0/4267294102 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:49.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4206046640 1 --2- 172.21.15.107:0/4267294102 >> v2:172.21.15.110:3300/0 conn(0x7f420006e800 0x7f420006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:49.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4205845640 1 -- 172.21.15.107:0/4267294102 >> v2:172.21.15.110:3300/0 conn(0x7f420006e800 msgr2=0x7f420006ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:49.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4205845640 1 --2- 172.21.15.107:0/4267294102 >> v2:172.21.15.110:3300/0 conn(0x7f420006e800 0x7f420006ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:49.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4205845640 1 -- 172.21.15.107:0/4267294102 >> v1:172.21.15.110:6789/0 conn(0x7f42000adb80 legacy=0x7f4200071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:49.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4205845640 1 -- 172.21.15.107:0/4267294102 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4200155020 con 0x7f4200152040 2023-10-25T00:10:49.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4205845640 1 --2- 172.21.15.107:0/4267294102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4200152040 0x7f4200158940 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f420006af70 tx=0x7f41fc01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d65aec38806389 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:49.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.913+0000 7f4204843640 1 -- 172.21.15.107:0/4267294102 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0a55c0 con 0x7f4200152040 2023-10-25T00:10:49.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4204843640 1 -- 172.21.15.107:0/4267294102 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41fc0a5720 con 0x7f4200152040 2023-10-25T00:10:49.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4204843640 1 -- 172.21.15.107:0/4267294102 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0a5ab0 con 0x7f4200152040 2023-10-25T00:10:49.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 -- 172.21.15.107:0/4267294102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4200152040 msgr2=0x7f4200158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:49.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 --2- 172.21.15.107:0/4267294102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4200152040 0x7f4200158940 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f420006af70 tx=0x7f41fc01c900 comp rx=0 tx=0).stop 2023-10-25T00:10:49.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 -- 172.21.15.107:0/4267294102 shutdown_connections 2023-10-25T00:10:49.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 --2- 172.21.15.107:0/4267294102 >> v2:172.21.15.110:3300/0 conn(0x7f420006e800 0x7f420006ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:49.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 --2- 172.21.15.107:0/4267294102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4200152040 0x7f4200158940 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:49.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 -- 172.21.15.107:0/4267294102 >> 172.21.15.107:0/4267294102 conn(0x7f4200070f60 msgr2=0x7f4200071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:49.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 -- 172.21.15.107:0/4267294102 shutdown_connections 2023-10-25T00:10:49.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 -- 172.21.15.107:0/4267294102 wait complete. 2023-10-25T00:10:49.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 Processor -- start 2023-10-25T00:10:49.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 -- start start 2023-10-25T00:10:49.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420006e800 0x7f4200152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:49.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42000adb80 0x7f4200148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:49.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4200152040 0x7f4200148e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:49.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42000c7d50 con 0x7f420006e800 2023-10-25T00:10:49.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420014b220 con 0x7f42000adb80 2023-10-25T00:10:49.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420014b670 con 0x7f4200152040 2023-10-25T00:10:49.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4205845640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420006e800 0x7f4200152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:49.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4205845640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420006e800 0x7f4200152f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38718/0 (socket says 172.21.15.107:38718) 2023-10-25T00:10:49.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4205845640 1 -- 172.21.15.107:0/4201068362 learned_addr learned my addr 172.21.15.107:0/4201068362 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:49.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4205044640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42000adb80 0x7f4200148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:49.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4206046640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4200152040 0x7f4200148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:49.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4205845640 1 -- 172.21.15.107:0/4201068362 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4200152040 msgr2=0x7f4200148e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:49.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4205845640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4200152040 0x7f4200148e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:49.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4205845640 1 -- 172.21.15.107:0/4201068362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42000adb80 msgr2=0x7f4200148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:49.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4205845640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42000adb80 0x7f4200148910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:49.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4205845640 1 -- 172.21.15.107:0/4201068362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc009b30 con 0x7f420006e800 2023-10-25T00:10:49.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4206046640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4200152040 0x7f4200148e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:10:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4205845640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420006e800 0x7f4200152f60 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f41fc01a530 tx=0x7f41fc0196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f41f67fc640 1 -- 172.21.15.107:0/4201068362 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0b8670 con 0x7f420006e800 2023-10-25T00:10:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f41f67fc640 1 -- 172.21.15.107:0/4201068362 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41fc0b87f0 con 0x7f420006e800 2023-10-25T00:10:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 -- 172.21.15.107:0/4201068362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42000c7790 con 0x7f420006e800 2023-10-25T00:10:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f41f67fc640 1 -- 172.21.15.107:0/4201068362 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0b8ba0 con 0x7f420006e800 2023-10-25T00:10:49.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.917+0000 7f4207aa9640 1 -- 172.21.15.107:0/4201068362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42001edb90 con 0x7f420006e800 2023-10-25T00:10:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.921+0000 7f4207aa9640 1 -- 172.21.15.107:0/4201068362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42001ed9f0 con 0x7f420006e800 2023-10-25T00:10:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.921+0000 7f41f67fc640 1 -- 172.21.15.107:0/4201068362 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f41fc0b8d00 con 0x7f420006e800 2023-10-25T00:10:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.921+0000 7f41f67fc640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f41e4065e60 0x7f41e4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:49.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.921+0000 7f41f67fc640 1 -- 172.21.15.107:0/4201068362 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7f41fc11fa60 con 0x7f420006e800 2023-10-25T00:10:49.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.921+0000 7f4205044640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f41e4065e60 0x7f41e4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:49.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.921+0000 7f4205044640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f41e4065e60 0x7f41e4068320 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4200152cd0 tx=0x7f41f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:49.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:49.925+0000 7f41f67fc640 1 -- 172.21.15.107:0/4201068362 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f42001ed9f0 con 0x7f420006e800 2023-10-25T00:10:50.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.129+0000 7f4207aa9640 1 -- 172.21.15.107:0/4201068362 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42001ed9f0 con 0x7f420006e800 2023-10-25T00:10:50.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f41f67fc640 1 -- 172.21.15.107:0/4201068362 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f42001ed9f0 con 0x7f420006e800 2023-10-25T00:10:50.133 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:50.133 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:50.134 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 155 2023-10-25T00:10:50.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 -- 172.21.15.107:0/4201068362 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f41e4065e60 msgr2=0x7f41e4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:50.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f41e4065e60 0x7f41e4068320 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4200152cd0 tx=0x7f41f0009040 comp rx=0 tx=0).stop 2023-10-25T00:10:50.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 -- 172.21.15.107:0/4201068362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420006e800 msgr2=0x7f4200152f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:50.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420006e800 0x7f4200152f60 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f41fc01a530 tx=0x7f41fc0196a0 comp rx=0 tx=0).stop 2023-10-25T00:10:50.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 -- 172.21.15.107:0/4201068362 shutdown_connections 2023-10-25T00:10:50.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f41e4065e60 0x7f41e4068320 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:50.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4200152040 0x7f4200148e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:50.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f42000adb80 0x7f4200148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:50.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 --2- 172.21.15.107:0/4201068362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f420006e800 0x7f4200152f60 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:50.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 -- 172.21.15.107:0/4201068362 >> 172.21.15.107:0/4201068362 conn(0x7f4200070f60 msgr2=0x7f42000c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:50.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 -- 172.21.15.107:0/4201068362 shutdown_connections 2023-10-25T00:10:50.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:50.133+0000 7f4207aa9640 1 -- 172.21.15.107:0/4201068362 wait complete. 2023-10-25T00:10:50.150 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:10:52.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:52.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.289+0000 7f53d52bf640 1 Processor -- start 2023-10-25T00:10:52.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.289+0000 7f53d52bf640 1 -- start start 2023-10-25T00:10:52.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.289+0000 7f53d52bf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53d014fd40 0x7f53d00ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:52.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.289+0000 7f53d52bf640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f53d00a9820 0x7f53d00aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:52.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.289+0000 7f53d52bf640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d006f730 con 0x7f53d006ee40 2023-10-25T00:10:52.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.289+0000 7f53d52bf640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d00aa750 con 0x7f53d00a9820 2023-10-25T00:10:52.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.289+0000 7f53d52bf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d0156a40 con 0x7f53d014fd40 2023-10-25T00:10:52.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.289+0000 7f53cf7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f53d006ee40 0x7f53d00c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35200/0 (socket says 172.21.15.107:35200) 2023-10-25T00:10:52.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.289+0000 7f53cf7fe640 1 -- 172.21.15.107:0/3629451365 learned_addr learned my addr 172.21.15.107:0/3629451365 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:52.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ceffd640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53d014fd40 0x7f53d00ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:52.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53c7fff640 1 --2- 172.21.15.107:0/3629451365 >> v2:172.21.15.110:3300/0 conn(0x7f53d00a9820 0x7f53d00aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:52.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045674997 0 0) 0x7f53d006f730 con 0x7f53d006ee40 2023-10-25T00:10:52.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53bc003530 con 0x7f53d006ee40 2023-10-25T00:10:52.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2698317751 0 0) 0x7f53bc003530 con 0x7f53d006ee40 2023-10-25T00:10:52.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53d014fd40 msgr2=0x7f53d00ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:52.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53d014fd40 0x7f53d00ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:52.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 >> v2:172.21.15.110:3300/0 conn(0x7f53d00a9820 msgr2=0x7f53d00aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:52.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 --2- 172.21.15.107:0/3629451365 >> v2:172.21.15.110:3300/0 conn(0x7f53d00a9820 0x7f53d00aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:52.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d0156710 con 0x7f53d006ee40 2023-10-25T00:10:52.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f53c8001a00 con 0x7f53d006ee40 2023-10-25T00:10:52.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 >> v1:172.21.15.107:6789/0 conn(0x7f53d006ee40 legacy=0x7f53d00c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:52.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc003980 0x7f53bc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:52.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc0047d0 0x7f53bc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:52.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53d014fd40 0x7f53bc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:52.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d00aa750 con 0x7f53bc003980 2023-10-25T00:10:52.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d0156a40 con 0x7f53bc0047d0 2023-10-25T00:10:52.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc00ba10 con 0x7f53d014fd40 2023-10-25T00:10:52.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ceffd640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc003980 0x7f53bc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:52.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53cf7fe640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53d014fd40 0x7f53bc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:52.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53c7fff640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc0047d0 0x7f53bc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:52.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ceffd640 1 -- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53d014fd40 msgr2=0x7f53bc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:52.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ceffd640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53d014fd40 0x7f53bc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:52.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ceffd640 1 -- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc0047d0 msgr2=0x7f53bc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:52.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ceffd640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc0047d0 0x7f53bc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:52.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ceffd640 1 -- 172.21.15.107:0/3629451365 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc003530 con 0x7f53bc003980 2023-10-25T00:10:52.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53c7fff640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53bc0047d0 0x7f53bc004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:10:52.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ceffd640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc003980 0x7f53bc003da0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f53c0009ae0 tx=0x7f53c001ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d43eca3f7bc1c24 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:52.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c00a5590 con 0x7f53bc003980 2023-10-25T00:10:52.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c00a56f0 con 0x7f53bc003980 2023-10-25T00:10:52.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.293+0000 7f53ce7fc640 1 -- 172.21.15.107:0/3629451365 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c00a5aa0 con 0x7f53bc003980 2023-10-25T00:10:52.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 -- 172.21.15.107:0/3629451365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc003980 msgr2=0x7f53bc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:52.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 --2- 172.21.15.107:0/3629451365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53bc003980 0x7f53bc003da0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f53c0009ae0 tx=0x7f53c001ca20 comp rx=0 tx=0).stop 2023-10-25T00:10:52.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53cf7fe640 1 -- 172.21.15.107:0/3629451365 reap_dead start 2023-10-25T00:10:52.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 -- 172.21.15.107:0/3629451365 shutdown_connections 2023-10-25T00:10:52.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 -- 172.21.15.107:0/3629451365 >> 172.21.15.107:0/3629451365 conn(0x7f53d00c2950 msgr2=0x7f53d00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:52.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 -- 172.21.15.107:0/3629451365 shutdown_connections 2023-10-25T00:10:52.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 -- 172.21.15.107:0/3629451365 wait complete. 2023-10-25T00:10:52.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 Processor -- start 2023-10-25T00:10:52.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 -- start start 2023-10-25T00:10:52.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53bc003980 0x7f53d014d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:52.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53d00c5840 0x7f53d006aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:52.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53d01edae0 0x7f53d0148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:52.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d00c4240 con 0x7f53d00c5840 2023-10-25T00:10:52.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d00c54b0 con 0x7f53d01edae0 2023-10-25T00:10:52.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53d52bf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d01ed650 con 0x7f53bc003980 2023-10-25T00:10:52.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53c7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53d00c5840 0x7f53d006aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:52.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53c7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53d00c5840 0x7f53d006aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38734/0 (socket says 172.21.15.107:38734) 2023-10-25T00:10:52.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53c7fff640 1 -- 172.21.15.107:0/442930296 learned_addr learned my addr 172.21.15.107:0/442930296 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:52.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53ceffd640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53bc003980 0x7f53d014d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:52.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53cf7fe640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53d01edae0 0x7f53d0148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:52.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53c7fff640 1 -- 172.21.15.107:0/442930296 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53bc003980 msgr2=0x7f53d014d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:52.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53c7fff640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53bc003980 0x7f53d014d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:52.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53c7fff640 1 -- 172.21.15.107:0/442930296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53d01edae0 msgr2=0x7f53d0148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:52.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53c7fff640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53d01edae0 0x7f53d0148950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:52.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.297+0000 7f53c7fff640 1 -- 172.21.15.107:0/442930296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d014eb10 con 0x7f53d00c5840 2023-10-25T00:10:52.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.301+0000 7f53c7fff640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53d00c5840 0x7f53d006aff0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f53b8010980 tx=0x7f53b8010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:52.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.301+0000 7f53c77fe640 1 -- 172.21.15.107:0/442930296 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8004220 con 0x7f53d00c5840 2023-10-25T00:10:52.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.301+0000 7f53c77fe640 1 -- 172.21.15.107:0/442930296 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b8004380 con 0x7f53d00c5840 2023-10-25T00:10:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.301+0000 7f53c77fe640 1 -- 172.21.15.107:0/442930296 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b800eb00 con 0x7f53d00c5840 2023-10-25T00:10:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.301+0000 7f53d52bf640 1 -- 172.21.15.107:0/442930296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d014edb0 con 0x7f53d00c5840 2023-10-25T00:10:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.301+0000 7f53d52bf640 1 -- 172.21.15.107:0/442930296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d01ed290 con 0x7f53d00c5840 2023-10-25T00:10:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.301+0000 7f53c77fe640 1 -- 172.21.15.107:0/442930296 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f53b800ec60 con 0x7f53d00c5840 2023-10-25T00:10:52.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.301+0000 7f53c77fe640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53b0065e90 0x7f53b0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:52.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.301+0000 7f53c77fe640 1 -- 172.21.15.107:0/442930296 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7f53b8084260 con 0x7f53d00c5840 2023-10-25T00:10:52.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.301+0000 7f53ceffd640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53b0065e90 0x7f53b0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:52.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.305+0000 7f53ceffd640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53b0065e90 0x7f53b0068350 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f53c00af000 tx=0x7f53c00046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:52.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.305+0000 7f53d52bf640 1 -- 172.21.15.107:0/442930296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d00c24a0 con 0x7f53d00c5840 2023-10-25T00:10:52.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.309+0000 7f53c77fe640 1 -- 172.21.15.107:0/442930296 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f53b8050c60 con 0x7f53d00c5840 2023-10-25T00:10:52.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 -- 172.21.15.107:0/442930296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53d00746e0 con 0x7f53d00c5840 2023-10-25T00:10:52.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53c77fe640 1 -- 172.21.15.107:0/442930296 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f53b8021070 con 0x7f53d00c5840 2023-10-25T00:10:52.502 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:52.502 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:52.503 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 155 2023-10-25T00:10:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 -- 172.21.15.107:0/442930296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53b0065e90 msgr2=0x7f53b0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53b0065e90 0x7f53b0068350 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f53c00af000 tx=0x7f53c00046d0 comp rx=0 tx=0).stop 2023-10-25T00:10:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 -- 172.21.15.107:0/442930296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53d00c5840 msgr2=0x7f53d006aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53d00c5840 0x7f53d006aff0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f53b8010980 tx=0x7f53b8010d80 comp rx=0 tx=0).stop 2023-10-25T00:10:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 -- 172.21.15.107:0/442930296 shutdown_connections 2023-10-25T00:10:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f53d01edae0 0x7f53d0148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53d00c5840 0x7f53d006aff0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f53b0065e90 0x7f53b0068350 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 --2- 172.21.15.107:0/442930296 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f53bc003980 0x7f53d014d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 -- 172.21.15.107:0/442930296 >> 172.21.15.107:0/442930296 conn(0x7f53d00c2950 msgr2=0x7f53d0150a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 -- 172.21.15.107:0/442930296 shutdown_connections 2023-10-25T00:10:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:52.501+0000 7f53d52bf640 1 -- 172.21.15.107:0/442930296 wait complete. 2023-10-25T00:10:52.521 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:10:54.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:54.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.653+0000 7ff301537640 1 Processor -- start 2023-10-25T00:10:54.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.653+0000 7ff301537640 1 -- start start 2023-10-25T00:10:54.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff301537640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2fc158560 0x7ff2fc0c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:54.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff301537640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc154860 con 0x7ff2fc13dc20 2023-10-25T00:10:54.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff301537640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc0c1f10 con 0x7ff2fc0720c0 2023-10-25T00:10:54.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff301537640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc154b20 con 0x7ff2fc158560 2023-10-25T00:10:54.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff2faffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff2fc13dc20 0x7ff2fc06df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60126/0 (socket says 172.21.15.107:60126) 2023-10-25T00:10:54.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff2faffd640 1 -- 172.21.15.107:0/1878029709 learned_addr learned my addr 172.21.15.107:0/1878029709 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:54.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff2fb7fe640 1 --2- 172.21.15.107:0/1878029709 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2fc158560 0x7ff2fc0c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:54.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff2f9ffb640 1 -- 172.21.15.107:0/1878029709 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487336802 0 0) 0x7ff2fc154860 con 0x7ff2fc13dc20 2023-10-25T00:10:54.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff2f9ffb640 1 -- 172.21.15.107:0/1878029709 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e8003530 con 0x7ff2fc13dc20 2023-10-25T00:10:54.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff2fb7fe640 1 -- 172.21.15.107:0/1878029709 >> v1:172.21.15.110:6789/0 conn(0x7ff2fc0720c0 legacy=0x7ff2fc157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:54.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff2fb7fe640 1 -- 172.21.15.107:0/1878029709 >> v1:172.21.15.107:6789/0 conn(0x7ff2fc13dc20 legacy=0x7ff2fc06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:54.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.657+0000 7ff2fb7fe640 1 -- 172.21.15.107:0/1878029709 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc154e00 con 0x7ff2fc158560 2023-10-25T00:10:54.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff2fb7fe640 1 --2- 172.21.15.107:0/1878029709 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2fc158560 0x7ff2fc0c18a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc154a50 tx=0x7ff2e401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c3f49e26d1b97c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:54.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff2f9ffb640 1 -- 172.21.15.107:0/1878029709 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40ab600 con 0x7ff2fc158560 2023-10-25T00:10:54.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff2f9ffb640 1 -- 172.21.15.107:0/1878029709 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e40a7050 con 0x7ff2fc158560 2023-10-25T00:10:54.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff2f9ffb640 1 -- 172.21.15.107:0/1878029709 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40ab960 con 0x7ff2fc158560 2023-10-25T00:10:54.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff301537640 1 -- 172.21.15.107:0/1878029709 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2fc158560 msgr2=0x7ff2fc0c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:54.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff301537640 1 --2- 172.21.15.107:0/1878029709 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2fc158560 0x7ff2fc0c18a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc154a50 tx=0x7ff2e401c380 comp rx=0 tx=0).stop 2023-10-25T00:10:54.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff301537640 1 -- 172.21.15.107:0/1878029709 shutdown_connections 2023-10-25T00:10:54.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff301537640 1 --2- 172.21.15.107:0/1878029709 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2fc158560 0x7ff2fc0c18a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:54.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff301537640 1 -- 172.21.15.107:0/1878029709 >> 172.21.15.107:0/1878029709 conn(0x7ff2fc0c39a0 msgr2=0x7ff2fc0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:54.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff301537640 1 -- 172.21.15.107:0/1878029709 shutdown_connections 2023-10-25T00:10:54.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff301537640 1 -- 172.21.15.107:0/1878029709 wait complete. 2023-10-25T00:10:54.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff301537640 1 Processor -- start 2023-10-25T00:10:54.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.661+0000 7ff301537640 1 -- start start 2023-10-25T00:10:54.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff301537640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2fc0720c0 0x7ff2fc1568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:54.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff301537640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc13dc20 0x7ff2fc136830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:54.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff301537640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2fc157700 0x7ff2fc0aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:54.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff301537640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc1ed890 con 0x7ff2fc157700 2023-10-25T00:10:54.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff301537640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc156440 con 0x7ff2fc13dc20 2023-10-25T00:10:54.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff301537640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc137200 con 0x7ff2fc0720c0 2023-10-25T00:10:54.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2fb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2fc157700 0x7ff2fc0aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:54.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2fb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2fc157700 0x7ff2fc0aa6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44544/0 (socket says 172.21.15.107:44544) 2023-10-25T00:10:54.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2fb7fe640 1 -- 172.21.15.107:0/3923790988 learned_addr learned my addr 172.21.15.107:0/3923790988 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:54.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2faffd640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2fc0720c0 0x7ff2fc1568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:54.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2fa7fc640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc13dc20 0x7ff2fc136830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:54.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2fb7fe640 1 -- 172.21.15.107:0/3923790988 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2fc0720c0 msgr2=0x7ff2fc1568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:54.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2fb7fe640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2fc0720c0 0x7ff2fc1568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:54.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2fb7fe640 1 -- 172.21.15.107:0/3923790988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc13dc20 msgr2=0x7ff2fc136830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:54.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2fb7fe640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc13dc20 0x7ff2fc136830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2fb7fe640 1 -- 172.21.15.107:0/3923790988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e4009420 con 0x7ff2fc157700 2023-10-25T00:10:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2fb7fe640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2fc157700 0x7ff2fc0aa6b0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4018f90 tx=0x7ff2e4019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2f37fe640 1 -- 172.21.15.107:0/3923790988 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4003a30 con 0x7ff2fc157700 2023-10-25T00:10:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2f37fe640 1 -- 172.21.15.107:0/3923790988 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e4003b90 con 0x7ff2fc157700 2023-10-25T00:10:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff301537640 1 -- 172.21.15.107:0/3923790988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2fc1ed2f0 con 0x7ff2fc157700 2023-10-25T00:10:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2f37fe640 1 -- 172.21.15.107:0/3923790988 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40ab400 con 0x7ff2fc157700 2023-10-25T00:10:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff301537640 1 -- 172.21.15.107:0/3923790988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2fc155df0 con 0x7ff2fc157700 2023-10-25T00:10:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2f37fe640 1 -- 172.21.15.107:0/3923790988 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff2e40b2040 con 0x7ff2fc157700 2023-10-25T00:10:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff301537640 1 -- 172.21.15.107:0/3923790988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2fc1565f0 con 0x7ff2fc157700 2023-10-25T00:10:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2f37fe640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2dc065e60 0x7ff2dc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2f37fe640 1 -- 172.21.15.107:0/3923790988 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7ff2e411e580 con 0x7ff2fc157700 2023-10-25T00:10:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.665+0000 7ff2faffd640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2dc065e60 0x7ff2dc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.669+0000 7ff2faffd640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2dc065e60 0x7ff2dc068320 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4001570 tx=0x7ff2f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.673+0000 7ff2f37fe640 1 -- 172.21.15.107:0/3923790988 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff2fc1565f0 con 0x7ff2fc157700 2023-10-25T00:10:54.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.933+0000 7ff301537640 1 -- 172.21.15.107:0/3923790988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2fc1565b0 con 0x7ff2fc157700 2023-10-25T00:10:54.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.933+0000 7ff2f37fe640 1 -- 172.21.15.107:0/3923790988 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5777 (secure 0 0 0) 0x7ff2e40b0090 con 0x7ff2fc157700 2023-10-25T00:10:54.937 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:54.937 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:54.938 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 155 2023-10-25T00:10:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 -- 172.21.15.107:0/3923790988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2dc065e60 msgr2=0x7ff2dc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2dc065e60 0x7ff2dc068320 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4001570 tx=0x7ff2f4009040 comp rx=0 tx=0).stop 2023-10-25T00:10:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 -- 172.21.15.107:0/3923790988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2fc157700 msgr2=0x7ff2fc0aa6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2fc157700 0x7ff2fc0aa6b0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4018f90 tx=0x7ff2e4019120 comp rx=0 tx=0).stop 2023-10-25T00:10:54.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 -- 172.21.15.107:0/3923790988 shutdown_connections 2023-10-25T00:10:54.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2fc157700 0x7ff2fc0aa6b0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:54.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2fc13dc20 0x7ff2fc136830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:54.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff2dc065e60 0x7ff2dc068320 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:54.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 --2- 172.21.15.107:0/3923790988 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2fc0720c0 0x7ff2fc1568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:54.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 -- 172.21.15.107:0/3923790988 >> 172.21.15.107:0/3923790988 conn(0x7ff2fc0c39a0 msgr2=0x7ff2fc0c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 -- 172.21.15.107:0/3923790988 shutdown_connections 2023-10-25T00:10:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:54.937+0000 7ff301537640 1 -- 172.21.15.107:0/3923790988 wait complete. 2023-10-25T00:10:54.958 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:10:56.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:57.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57d6c6c640 1 Processor -- start 2023-10-25T00:10:57.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57d6c6c640 1 -- start start 2023-10-25T00:10:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57d6c6c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57d0070b00 0x7f57d00c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57d6c6c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f57d006df30 0x7f57d006e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57d6c6c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d00c22f0 con 0x7f57d014eb70 2023-10-25T00:10:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57d6c6c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d0071700 con 0x7f57d006df30 2023-10-25T00:10:57.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57d6c6c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d00c25d0 con 0x7f57d0070b00 2023-10-25T00:10:57.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57cffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57d0070b00 0x7f57d00c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:57.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57d5209640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f57d006df30 0x7f57d006e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:57.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57d4a08640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f57d014eb70 0x7f57d01514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:40976/0 (socket says 172.21.15.107:40976) 2023-10-25T00:10:57.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57d4a08640 1 -- 172.21.15.107:0/4168521650 learned_addr learned my addr 172.21.15.107:0/4168521650 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:57.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57cffff640 1 -- 172.21.15.107:0/4168521650 >> v2:172.21.15.110:3300/0 conn(0x7f57d006df30 msgr2=0x7f57d006e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:57.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57cffff640 1 --2- 172.21.15.107:0/4168521650 >> v2:172.21.15.110:3300/0 conn(0x7f57d006df30 0x7f57d006e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:57.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57cffff640 1 -- 172.21.15.107:0/4168521650 >> v1:172.21.15.110:6789/0 conn(0x7f57d014eb70 legacy=0x7f57d01514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:57.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.081+0000 7f57cffff640 1 -- 172.21.15.107:0/4168521650 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d00c2be0 con 0x7f57d0070b00 2023-10-25T00:10:57.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.085+0000 7f57cffff640 1 --2- 172.21.15.107:0/4168521650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57d0070b00 0x7f57d00c33e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0097c0 tx=0x7f57bc01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cfa70c6e81bc415 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:57.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.085+0000 7f57cf7fe640 1 -- 172.21.15.107:0/4168521650 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0a6040 con 0x7f57d0070b00 2023-10-25T00:10:57.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.085+0000 7f57cf7fe640 1 -- 172.21.15.107:0/4168521650 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc0aa600 con 0x7f57d0070b00 2023-10-25T00:10:57.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.085+0000 7f57cf7fe640 1 -- 172.21.15.107:0/4168521650 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0aa990 con 0x7f57d0070b00 2023-10-25T00:10:57.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.085+0000 7f57d6c6c640 1 -- 172.21.15.107:0/4168521650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57d0070b00 msgr2=0x7f57d00c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:57.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.085+0000 7f57d6c6c640 1 --2- 172.21.15.107:0/4168521650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57d0070b00 0x7f57d00c33e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0097c0 tx=0x7f57bc01c480 comp rx=0 tx=0).stop 2023-10-25T00:10:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.085+0000 7f57d6c6c640 1 -- 172.21.15.107:0/4168521650 shutdown_connections 2023-10-25T00:10:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.085+0000 7f57d6c6c640 1 --2- 172.21.15.107:0/4168521650 >> v2:172.21.15.110:3300/0 conn(0x7f57d006df30 0x7f57d006e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.085+0000 7f57d6c6c640 1 --2- 172.21.15.107:0/4168521650 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57d0070b00 0x7f57d00c33e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.085+0000 7f57d6c6c640 1 -- 172.21.15.107:0/4168521650 >> 172.21.15.107:0/4168521650 conn(0x7f57d00c39a0 msgr2=0x7f57d00c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 -- 172.21.15.107:0/4168521650 shutdown_connections 2023-10-25T00:10:57.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 -- 172.21.15.107:0/4168521650 wait complete. 2023-10-25T00:10:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 Processor -- start 2023-10-25T00:10:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 -- start start 2023-10-25T00:10:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57d006df30 0x7f57d00c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57d0070b00 0x7f57d014b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57d014eb70 0x7f57d014ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0142ca0 con 0x7f57d0070b00 2023-10-25T00:10:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d00aba60 con 0x7f57d014eb70 2023-10-25T00:10:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d006a0e0 con 0x7f57d006df30 2023-10-25T00:10:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57d0070b00 0x7f57d014b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57d0070b00 0x7f57d014b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44556/0 (socket says 172.21.15.107:44556) 2023-10-25T00:10:57.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cffff640 1 -- 172.21.15.107:0/2868042528 learned_addr learned my addr 172.21.15.107:0/2868042528 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:57.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d4a08640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57d006df30 0x7f57d00c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:57.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d5209640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57d014eb70 0x7f57d014ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:57.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cffff640 1 -- 172.21.15.107:0/2868042528 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57d006df30 msgr2=0x7f57d00c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cffff640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57d006df30 0x7f57d00c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cffff640 1 -- 172.21.15.107:0/2868042528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57d014eb70 msgr2=0x7f57d014ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cffff640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57d014eb70 0x7f57d014ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cffff640 1 -- 172.21.15.107:0/2868042528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57bc009420 con 0x7f57d0070b00 2023-10-25T00:10:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d5209640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57d014eb70 0x7f57d014ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:10:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cffff640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57d0070b00 0x7f57d014b530 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f57bc01a0b0 tx=0x7f57bc01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:57.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cd7fa640 1 -- 172.21.15.107:0/2868042528 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0b3600 con 0x7f57d0070b00 2023-10-25T00:10:57.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cd7fa640 1 -- 172.21.15.107:0/2868042528 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc0b3760 con 0x7f57d0070b00 2023-10-25T00:10:57.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 -- 172.21.15.107:0/2868042528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d00ab6c0 con 0x7f57d0070b00 2023-10-25T00:10:57.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57cd7fa640 1 -- 172.21.15.107:0/2868042528 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0b3af0 con 0x7f57d0070b00 2023-10-25T00:10:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.089+0000 7f57d6c6c640 1 -- 172.21.15.107:0/2868042528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d01ed710 con 0x7f57d0070b00 2023-10-25T00:10:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.093+0000 7f57d6c6c640 1 -- 172.21.15.107:0/2868042528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d00abba0 con 0x7f57d0070b00 2023-10-25T00:10:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.093+0000 7f57cd7fa640 1 -- 172.21.15.107:0/2868042528 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f57bc0b3c50 con 0x7f57d0070b00 2023-10-25T00:10:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.093+0000 7f57cd7fa640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f57b4065e60 0x7f57b4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.093+0000 7f57cd7fa640 1 -- 172.21.15.107:0/2868042528 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7f57bc11efb0 con 0x7f57d0070b00 2023-10-25T00:10:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.093+0000 7f57d4a08640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f57b4065e60 0x7f57b4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.093+0000 7f57d4a08640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f57b4065e60 0x7f57b4068320 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f57c4005890 tx=0x7f57c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:57.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.097+0000 7f57cd7fa640 1 -- 172.21.15.107:0/2868042528 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f57d00abba0 con 0x7f57d0070b00 2023-10-25T00:10:57.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.313+0000 7f57d6c6c640 1 -- 172.21.15.107:0/2868042528 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57d00abba0 con 0x7f57d0070b00 2023-10-25T00:10:57.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.313+0000 7f57cd7fa640 1 -- 172.21.15.107:0/2868042528 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f57d00abba0 con 0x7f57d0070b00 2023-10-25T00:10:57.317 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:57.317 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:57.317 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 155 2023-10-25T00:10:57.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 -- 172.21.15.107:0/2868042528 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f57b4065e60 msgr2=0x7f57b4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:57.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f57b4065e60 0x7f57b4068320 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f57c4005890 tx=0x7f57c4009040 comp rx=0 tx=0).stop 2023-10-25T00:10:57.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 -- 172.21.15.107:0/2868042528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57d0070b00 msgr2=0x7f57d014b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:57.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57d0070b00 0x7f57d014b530 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f57bc01a0b0 tx=0x7f57bc01acc0 comp rx=0 tx=0).stop 2023-10-25T00:10:57.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 -- 172.21.15.107:0/2868042528 shutdown_connections 2023-10-25T00:10:57.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57d014eb70 0x7f57d014ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:57.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57d0070b00 0x7f57d014b530 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:57.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f57b4065e60 0x7f57b4068320 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:57.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 --2- 172.21.15.107:0/2868042528 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f57d006df30 0x7f57d00c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:57.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 -- 172.21.15.107:0/2868042528 >> 172.21.15.107:0/2868042528 conn(0x7f57d00c39a0 msgr2=0x7f57d00c0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:57.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 -- 172.21.15.107:0/2868042528 shutdown_connections 2023-10-25T00:10:57.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:57.317+0000 7f57d6c6c640 1 -- 172.21.15.107:0/2868042528 wait complete. 2023-10-25T00:10:57.334 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:10:59.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:10:59.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbf288c640 1 Processor -- start 2023-10-25T00:10:59.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbf288c640 1 -- start start 2023-10-25T00:10:59.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbf288c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffbec0c5c20 0x7ffbec153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:59.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbf288c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec072330 con 0x7ffbec0c18a0 2023-10-25T00:10:59.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbf288c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec155ca0 con 0x7ffbec14fc30 2023-10-25T00:10:59.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbf288c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbec155f80 con 0x7ffbec0c5c20 2023-10-25T00:10:59.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbf0e29640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ffbec0c18a0 0x7ffbec0c1cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60138/0 (socket says 172.21.15.107:60138) 2023-10-25T00:10:59.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbeb7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffbec0c5c20 0x7ffbec153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:59.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbf0e29640 1 -- 172.21.15.107:0/1665731173 learned_addr learned my addr 172.21.15.107:0/1665731173 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:10:59.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbeaffd640 1 -- 172.21.15.107:0/1665731173 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259383330 0 0) 0x7ffbec072330 con 0x7ffbec0c18a0 2023-10-25T00:10:59.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbeaffd640 1 -- 172.21.15.107:0/1665731173 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd4003530 con 0x7ffbec0c18a0 2023-10-25T00:10:59.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbeb7fe640 1 -- 172.21.15.107:0/1665731173 >> v1:172.21.15.110:6789/0 conn(0x7ffbec14fc30 legacy=0x7ffbec06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:10:59.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbeb7fe640 1 -- 172.21.15.107:0/1665731173 >> v1:172.21.15.107:6789/0 conn(0x7ffbec0c18a0 legacy=0x7ffbec0c1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:59.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.469+0000 7ffbeb7fe640 1 -- 172.21.15.107:0/1665731173 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec156260 con 0x7ffbec0c5c20 2023-10-25T00:10:59.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.473+0000 7ffbeb7fe640 1 --2- 172.21.15.107:0/1665731173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbec0c5c20 0x7ffbec153190 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00097c0 tx=0x7ffbe001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8272a22c5e4e134 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:59.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.473+0000 7ffbeaffd640 1 -- 172.21.15.107:0/1665731173 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe00ad680 con 0x7ffbec0c5c20 2023-10-25T00:10:59.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.473+0000 7ffbeaffd640 1 -- 172.21.15.107:0/1665731173 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe00ad7e0 con 0x7ffbec0c5c20 2023-10-25T00:10:59.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.473+0000 7ffbeaffd640 1 -- 172.21.15.107:0/1665731173 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe00adb70 con 0x7ffbec0c5c20 2023-10-25T00:10:59.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.473+0000 7ffbf288c640 1 -- 172.21.15.107:0/1665731173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbec0c5c20 msgr2=0x7ffbec153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:59.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.473+0000 7ffbf288c640 1 --2- 172.21.15.107:0/1665731173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbec0c5c20 0x7ffbec153190 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7ffbe00097c0 tx=0x7ffbe001c3a0 comp rx=0 tx=0).stop 2023-10-25T00:10:59.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.473+0000 7ffbf288c640 1 -- 172.21.15.107:0/1665731173 shutdown_connections 2023-10-25T00:10:59.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.473+0000 7ffbf288c640 1 --2- 172.21.15.107:0/1665731173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbec0c5c20 0x7ffbec153190 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:59.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.473+0000 7ffbf288c640 1 -- 172.21.15.107:0/1665731173 >> 172.21.15.107:0/1665731173 conn(0x7ffbec0c39a0 msgr2=0x7ffbec0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:59.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.473+0000 7ffbf288c640 1 -- 172.21.15.107:0/1665731173 shutdown_connections 2023-10-25T00:10:59.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.473+0000 7ffbf288c640 1 -- 172.21.15.107:0/1665731173 wait complete. 2023-10-25T00:10:59.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf288c640 1 Processor -- start 2023-10-25T00:10:59.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf288c640 1 -- start start 2023-10-25T00:10:59.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf288c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbec0c18a0 0x7ffbec0c04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:59.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf288c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbec0c5c20 0x7ffbec14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:59.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf288c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffbec14fc30 0x7ffbec14e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:59.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf288c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec071870 con 0x7ffbec0c5c20 2023-10-25T00:10:59.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf288c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec070b20 con 0x7ffbec0c18a0 2023-10-25T00:10:59.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf288c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec06b750 con 0x7ffbec14fc30 2023-10-25T00:10:59.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbeb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbec0c5c20 0x7ffbec14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:59.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbeb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbec0c5c20 0x7ffbec14e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44562/0 (socket says 172.21.15.107:44562) 2023-10-25T00:10:59.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbeb7fe640 1 -- 172.21.15.107:0/2954375677 learned_addr learned my addr 172.21.15.107:0/2954375677 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:10:59.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf0e29640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffbec14fc30 0x7ffbec14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:59.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbebfff640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbec0c18a0 0x7ffbec0c04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:59.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbeb7fe640 1 -- 172.21.15.107:0/2954375677 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffbec14fc30 msgr2=0x7ffbec14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:59.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbeb7fe640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffbec14fc30 0x7ffbec14e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:59.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbeb7fe640 1 -- 172.21.15.107:0/2954375677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbec0c18a0 msgr2=0x7ffbec0c04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:59.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbeb7fe640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbec0c18a0 0x7ffbec0c04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:59.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbeb7fe640 1 -- 172.21.15.107:0/2954375677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe0009420 con 0x7ffbec0c5c20 2023-10-25T00:10:59.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf0e29640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffbec14fc30 0x7ffbec14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:10:59.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbeb7fe640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbec0c5c20 0x7ffbec14e1f0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0018fb0 tx=0x7ffbe001abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:59.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbe8ff9640 1 -- 172.21.15.107:0/2954375677 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe00add60 con 0x7ffbec0c5c20 2023-10-25T00:10:59.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbe8ff9640 1 -- 172.21.15.107:0/2954375677 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe00b8450 con 0x7ffbec0c5c20 2023-10-25T00:10:59.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf288c640 1 -- 172.21.15.107:0/2954375677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbec06d930 con 0x7ffbec0c5c20 2023-10-25T00:10:59.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbe8ff9640 1 -- 172.21.15.107:0/2954375677 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe00b8800 con 0x7ffbec0c5c20 2023-10-25T00:10:59.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf288c640 1 -- 172.21.15.107:0/2954375677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbec070d90 con 0x7ffbec0c5c20 2023-10-25T00:10:59.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbf288c640 1 -- 172.21.15.107:0/2954375677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbec073980 con 0x7ffbec0c5c20 2023-10-25T00:10:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.477+0000 7ffbe8ff9640 1 -- 172.21.15.107:0/2954375677 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ffbe00b4070 con 0x7ffbec0c5c20 2023-10-25T00:10:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.481+0000 7ffbe8ff9640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffbd8065e90 0x7ffbd8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:10:59.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.481+0000 7ffbe8ff9640 1 -- 172.21.15.107:0/2954375677 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7ffbe011ed40 con 0x7ffbec0c5c20 2023-10-25T00:10:59.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.481+0000 7ffbebfff640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffbd8065e90 0x7ffbd8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:10:59.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.481+0000 7ffbebfff640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffbd8065e90 0x7ffbd8068350 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00b810 tx=0x7ffbdc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:10:59.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.485+0000 7ffbe8ff9640 1 -- 172.21.15.107:0/2954375677 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ffbe00eb690 con 0x7ffbec0c5c20 2023-10-25T00:10:59.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.721+0000 7ffbf288c640 1 -- 172.21.15.107:0/2954375677 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbec073980 con 0x7ffbec0c5c20 2023-10-25T00:10:59.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbe8ff9640 1 -- 172.21.15.107:0/2954375677 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5777 (secure 0 0 0) 0x7ffbe00b2020 con 0x7ffbec0c5c20 2023-10-25T00:10:59.725 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:10:59.725 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:10:59.726 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 155 2023-10-25T00:10:59.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 -- 172.21.15.107:0/2954375677 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffbd8065e90 msgr2=0x7ffbd8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:59.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffbd8065e90 0x7ffbd8068350 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00b810 tx=0x7ffbdc008040 comp rx=0 tx=0).stop 2023-10-25T00:10:59.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 -- 172.21.15.107:0/2954375677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbec0c5c20 msgr2=0x7ffbec14e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:10:59.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbec0c5c20 0x7ffbec14e1f0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0018fb0 tx=0x7ffbe001abe0 comp rx=0 tx=0).stop 2023-10-25T00:10:59.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 -- 172.21.15.107:0/2954375677 shutdown_connections 2023-10-25T00:10:59.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffbd8065e90 0x7ffbd8068350 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:59.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffbec14fc30 0x7ffbec14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:59.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbec0c5c20 0x7ffbec14e1f0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:59.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 --2- 172.21.15.107:0/2954375677 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffbec0c18a0 0x7ffbec0c04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:10:59.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 -- 172.21.15.107:0/2954375677 >> 172.21.15.107:0/2954375677 conn(0x7ffbec0c39a0 msgr2=0x7ffbec0c1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:10:59.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 -- 172.21.15.107:0/2954375677 shutdown_connections 2023-10-25T00:10:59.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:10:59.725+0000 7ffbf288c640 1 -- 172.21.15.107:0/2954375677 wait complete. 2023-10-25T00:10:59.745 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:11:01.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.881+0000 7f98bf33a640 1 Processor -- start 2023-10-25T00:11:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.881+0000 7f98bf33a640 1 -- start start 2023-10-25T00:11:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bf33a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98b8152040 0x7f98b80c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bf33a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f98b806a4b0 0x7f98b80ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bf33a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b8155680 con 0x7f98b80c8100 2023-10-25T00:11:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bf33a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b8154a70 con 0x7f98b806a4b0 2023-10-25T00:11:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bf33a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b8154d50 con 0x7f98b8152040 2023-10-25T00:11:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bd8d7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f98b806a4b0 0x7f98b80ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bd8d7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f98b806a4b0 0x7f98b80ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44574/0 (socket says 172.21.15.107:44574) 2023-10-25T00:11:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bd8d7640 1 -- 172.21.15.107:0/766165235 learned_addr learned my addr 172.21.15.107:0/766165235 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bd0d6640 1 --2- 172.21.15.107:0/766165235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98b8152040 0x7f98b80c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bd8d7640 1 -- 172.21.15.107:0/766165235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98b8152040 msgr2=0x7f98b80c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bd8d7640 1 --2- 172.21.15.107:0/766165235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98b8152040 0x7f98b80c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bd8d7640 1 -- 172.21.15.107:0/766165235 >> v1:172.21.15.110:6789/0 conn(0x7f98b80c8100 legacy=0x7f98b80abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bd8d7640 1 -- 172.21.15.107:0/766165235 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b8155030 con 0x7f98b806a4b0 2023-10-25T00:11:01.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.885+0000 7f98bd8d7640 1 --2- 172.21.15.107:0/766165235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b806a4b0 0x7f98b80ac410 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f98b0009e50 tx=0x7f98b001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=674b1703487ce210 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:01.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98affff640 1 -- 172.21.15.107:0/766165235 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b00a5750 con 0x7f98b806a4b0 2023-10-25T00:11:01.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98affff640 1 -- 172.21.15.107:0/766165235 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b00a58b0 con 0x7f98b806a4b0 2023-10-25T00:11:01.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98affff640 1 -- 172.21.15.107:0/766165235 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b00a5c40 con 0x7f98b806a4b0 2023-10-25T00:11:01.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 -- 172.21.15.107:0/766165235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b806a4b0 msgr2=0x7f98b80ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:01.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 --2- 172.21.15.107:0/766165235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b806a4b0 0x7f98b80ac410 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f98b0009e50 tx=0x7f98b001c980 comp rx=0 tx=0).stop 2023-10-25T00:11:01.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 -- 172.21.15.107:0/766165235 shutdown_connections 2023-10-25T00:11:01.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 --2- 172.21.15.107:0/766165235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b806a4b0 0x7f98b80ac410 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:01.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 --2- 172.21.15.107:0/766165235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98b8152040 0x7f98b80c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:01.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 -- 172.21.15.107:0/766165235 >> 172.21.15.107:0/766165235 conn(0x7f98b80ca200 msgr2=0x7f98b80ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:01.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 -- 172.21.15.107:0/766165235 shutdown_connections 2023-10-25T00:11:01.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 -- 172.21.15.107:0/766165235 wait complete. 2023-10-25T00:11:01.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 Processor -- start 2023-10-25T00:11:01.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 -- start start 2023-10-25T00:11:01.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b806a4b0 0x7f98b814d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:01.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98b80c8100 0x7f98b814b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:01.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98b8152040 0x7f98b814bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:01.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b81ecc60 con 0x7f98b806a4b0 2023-10-25T00:11:01.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b81545a0 con 0x7f98b80c8100 2023-10-25T00:11:01.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b81ecf20 con 0x7f98b8152040 2023-10-25T00:11:01.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bd0d6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b806a4b0 0x7f98b814d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:01.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bd0d6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b806a4b0 0x7f98b814d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44576/0 (socket says 172.21.15.107:44576) 2023-10-25T00:11:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bd0d6640 1 -- 172.21.15.107:0/1006701128 learned_addr learned my addr 172.21.15.107:0/1006701128 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bc8d5640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98b80c8100 0x7f98b814b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bd8d7640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98b8152040 0x7f98b814bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bd0d6640 1 -- 172.21.15.107:0/1006701128 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98b8152040 msgr2=0x7f98b814bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bd0d6640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98b8152040 0x7f98b814bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bd0d6640 1 -- 172.21.15.107:0/1006701128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98b80c8100 msgr2=0x7f98b814b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bd0d6640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98b80c8100 0x7f98b814b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bd0d6640 1 -- 172.21.15.107:0/1006701128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b0009b30 con 0x7f98b806a4b0 2023-10-25T00:11:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bd0d6640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b806a4b0 0x7f98b814d140 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f98b400d800 tx=0x7f98b400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98adffb640 1 -- 172.21.15.107:0/1006701128 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b400b6c0 con 0x7f98b806a4b0 2023-10-25T00:11:01.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98adffb640 1 -- 172.21.15.107:0/1006701128 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b400b820 con 0x7f98b806a4b0 2023-10-25T00:11:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.889+0000 7f98bf33a640 1 -- 172.21.15.107:0/1006701128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b81ed200 con 0x7f98b806a4b0 2023-10-25T00:11:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.893+0000 7f98adffb640 1 -- 172.21.15.107:0/1006701128 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b400bb90 con 0x7f98b806a4b0 2023-10-25T00:11:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.893+0000 7f98bf33a640 1 -- 172.21.15.107:0/1006701128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b814d680 con 0x7f98b806a4b0 2023-10-25T00:11:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.893+0000 7f98bf33a640 1 -- 172.21.15.107:0/1006701128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b80746e0 con 0x7f98b806a4b0 2023-10-25T00:11:01.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.893+0000 7f98adffb640 1 -- 172.21.15.107:0/1006701128 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f98b400bcf0 con 0x7f98b806a4b0 2023-10-25T00:11:01.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.893+0000 7f98adffb640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f989c065e60 0x7f989c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.893+0000 7f98adffb640 1 -- 172.21.15.107:0/1006701128 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 9455+0+0 (secure 0 0 0) 0x7f98b4084a90 con 0x7f98b806a4b0 2023-10-25T00:11:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.893+0000 7f98bc8d5640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f989c065e60 0x7f989c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.893+0000 7f98bc8d5640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f989c065e60 0x7f989c068320 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f98b80abef0 tx=0x7f98a8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:01.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:01.897+0000 7f98adffb640 1 -- 172.21.15.107:0/1006701128 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f98b40513e0 con 0x7f98b806a4b0 2023-10-25T00:11:02.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.181+0000 7f98bf33a640 1 -- 172.21.15.107:0/1006701128 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98b80ade90 con 0x7f98b806a4b0 2023-10-25T00:11:02.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.181+0000 7f98adffb640 1 -- 172.21.15.107:0/1006701128 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f98b4054a00 con 0x7f98b806a4b0 2023-10-25T00:11:02.185 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:02.185 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:09:21.753141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6568},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6568":{"gid":6568,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":7,"addr":"172.21.15.110:6835/326715537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":326715537},{"type":"v1","addr":"172.21.15.110:6835","nonce":326715537}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:02.185 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 155 2023-10-25T00:11:02.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 -- 172.21.15.107:0/1006701128 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f989c065e60 msgr2=0x7f989c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:02.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f989c065e60 0x7f989c068320 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f98b80abef0 tx=0x7f98a8009040 comp rx=0 tx=0).stop 2023-10-25T00:11:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 -- 172.21.15.107:0/1006701128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b806a4b0 msgr2=0x7f98b814d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b806a4b0 0x7f98b814d140 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f98b400d800 tx=0x7f98b400dcc0 comp rx=0 tx=0).stop 2023-10-25T00:11:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 -- 172.21.15.107:0/1006701128 shutdown_connections 2023-10-25T00:11:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f989c065e60 0x7f989c068320 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:02.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f98b8152040 0x7f98b814bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:02.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f98b80c8100 0x7f98b814b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:02.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 --2- 172.21.15.107:0/1006701128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b806a4b0 0x7f98b814d140 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:02.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 -- 172.21.15.107:0/1006701128 >> 172.21.15.107:0/1006701128 conn(0x7f98b80ca200 msgr2=0x7f98b80ad780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:02.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 -- 172.21.15.107:0/1006701128 shutdown_connections 2023-10-25T00:11:02.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:02.185+0000 7f98bf33a640 1 -- 172.21.15.107:0/1006701128 wait complete. 2023-10-25T00:11:02.205 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:11:03.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:04.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.077+0000 7f806ba17640 1 Processor -- start 2023-10-25T00:11:04.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.077+0000 7f806ba17640 1 -- start start 2023-10-25T00:11:04.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f806ba17640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f806413bb00 0x7f80641531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f806ba17640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f80640a9450 0x7f8064071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f806ba17640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8064152d40 con 0x7f80640aa330 2023-10-25T00:11:04.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f806ba17640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8064156100 con 0x7f806413bb00 2023-10-25T00:11:04.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f806ba17640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80641527c0 con 0x7f80640a9450 2023-10-25T00:11:04.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f80697b3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f806413bb00 0x7f80641531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f80697b3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f806413bb00 0x7f80641531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42858/0 (socket says 172.21.15.107:42858) 2023-10-25T00:11:04.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f80697b3640 1 -- 172.21.15.107:0/1979366519 learned_addr learned my addr 172.21.15.107:0/1979366519 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:04.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f8069fb4640 1 --2- 172.21.15.107:0/1979366519 >> v2:172.21.15.110:3300/0 conn(0x7f80640a9450 0x7f8064071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f80697b3640 1 -- 172.21.15.107:0/1979366519 >> v2:172.21.15.110:3300/0 conn(0x7f80640a9450 msgr2=0x7f8064071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:04.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f80697b3640 1 --2- 172.21.15.107:0/1979366519 >> v2:172.21.15.110:3300/0 conn(0x7f80640a9450 0x7f8064071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f80697b3640 1 -- 172.21.15.107:0/1979366519 >> v1:172.21.15.110:6789/0 conn(0x7f80640aa330 legacy=0x7f806406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.081+0000 7f80697b3640 1 -- 172.21.15.107:0/1979366519 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806406d660 con 0x7f806413bb00 2023-10-25T00:11:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f80697b3640 1 --2- 172.21.15.107:0/1979366519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806413bb00 0x7f80641531d0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f8064072ab0 tx=0x7f806001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d59116a6fac6ec server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f805bfff640 1 -- 172.21.15.107:0/1979366519 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80600ab600 con 0x7f806413bb00 2023-10-25T00:11:04.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f805bfff640 1 -- 172.21.15.107:0/1979366519 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80600a7050 con 0x7f806413bb00 2023-10-25T00:11:04.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f805bfff640 1 -- 172.21.15.107:0/1979366519 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80600ab980 con 0x7f806413bb00 2023-10-25T00:11:04.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 -- 172.21.15.107:0/1979366519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806413bb00 msgr2=0x7f80641531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 --2- 172.21.15.107:0/1979366519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806413bb00 0x7f80641531d0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f8064072ab0 tx=0x7f806001c3e0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 -- 172.21.15.107:0/1979366519 shutdown_connections 2023-10-25T00:11:04.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 --2- 172.21.15.107:0/1979366519 >> v2:172.21.15.110:3300/0 conn(0x7f80640a9450 0x7f8064071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 --2- 172.21.15.107:0/1979366519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806413bb00 0x7f80641531d0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 -- 172.21.15.107:0/1979366519 >> 172.21.15.107:0/1979366519 conn(0x7f8064070f60 msgr2=0x7f8064071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:04.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 -- 172.21.15.107:0/1979366519 shutdown_connections 2023-10-25T00:11:04.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 -- 172.21.15.107:0/1979366519 wait complete. 2023-10-25T00:11:04.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 Processor -- start 2023-10-25T00:11:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 -- start start 2023-10-25T00:11:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80640a9450 0x7f806406ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80640aa330 0x7f8064133000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806413bb00 0x7f806406b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806406b010 con 0x7f806413bb00 2023-10-25T00:11:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8064153b30 con 0x7f80640aa330 2023-10-25T00:11:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f806ba17640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8064153e10 con 0x7f80640a9450 2023-10-25T00:11:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f8069fb4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806413bb00 0x7f806406b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f8069fb4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806413bb00 0x7f806406b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42874/0 (socket says 172.21.15.107:42874) 2023-10-25T00:11:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f8069fb4640 1 -- 172.21.15.107:0/137955203 learned_addr learned my addr 172.21.15.107:0/137955203 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f80697b3640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80640a9450 0x7f806406ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.085+0000 7f8068fb2640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80640aa330 0x7f8064133000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8069fb4640 1 -- 172.21.15.107:0/137955203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80640a9450 msgr2=0x7f806406ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8069fb4640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80640a9450 0x7f806406ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8069fb4640 1 -- 172.21.15.107:0/137955203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80640aa330 msgr2=0x7f8064133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8069fb4640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80640aa330 0x7f8064133000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8069fb4640 1 -- 172.21.15.107:0/137955203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80600094d0 con 0x7f806413bb00 2023-10-25T00:11:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f80697b3640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80640a9450 0x7f806406ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:11:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8069fb4640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806413bb00 0x7f806406b4a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f805c007e40 tx=0x7f805c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8059ffb640 1 -- 172.21.15.107:0/137955203 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c014620 con 0x7f806413bb00 2023-10-25T00:11:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8059ffb640 1 -- 172.21.15.107:0/137955203 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c00ee90 con 0x7f806413bb00 2023-10-25T00:11:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f806ba17640 1 -- 172.21.15.107:0/137955203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80641540f0 con 0x7f806413bb00 2023-10-25T00:11:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8059ffb640 1 -- 172.21.15.107:0/137955203 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c0149e0 con 0x7f806413bb00 2023-10-25T00:11:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f806ba17640 1 -- 172.21.15.107:0/137955203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8064155590 con 0x7f806413bb00 2023-10-25T00:11:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f804f7fe640 1 -- 172.21.15.107:0/137955203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803c005260 con 0x7f806413bb00 2023-10-25T00:11:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8059ffb640 1 -- 172.21.15.107:0/137955203 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f805c014b40 con 0x7f806413bb00 2023-10-25T00:11:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8059ffb640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8048065e60 0x7f8048068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f8059ffb640 1 -- 172.21.15.107:0/137955203 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7f805c084f50 con 0x7f806413bb00 2023-10-25T00:11:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f80697b3640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8048065e60 0x7f8048068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.089+0000 7f80697b3640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8048065e60 0x7f8048068320 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f80600ad000 tx=0x7f80600a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.093+0000 7f8059ffb640 1 -- 172.21.15.107:0/137955203 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f805c051840 con 0x7f806413bb00 2023-10-25T00:11:04.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:04.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5ba1d00640 1 Processor -- start 2023-10-25T00:11:04.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5ba1d00640 1 -- start start 2023-10-25T00:11:04.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5ba1d00640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b9c070b00 0x7f5b9c0c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5ba1d00640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5b9c06df30 0x7f5b9c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5ba1d00640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9c0c22f0 con 0x7f5b9c14eb70 2023-10-25T00:11:04.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5ba1d00640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b9c071700 con 0x7f5b9c070b00 2023-10-25T00:11:04.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5ba1d00640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b9c0c25d0 con 0x7f5b9c06df30 2023-10-25T00:11:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5b9b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5b9c14eb70 0x7f5b9c1514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49554/0 (socket says 172.21.15.107:49554) 2023-10-25T00:11:04.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5b92ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b9c070b00 0x7f5b9c0c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5b9b7fe640 1 -- 172.21.15.107:0/1093938162 learned_addr learned my addr 172.21.15.107:0/1093938162 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:11:04.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5b9bfff640 1 --2- 172.21.15.107:0/1093938162 >> v2:172.21.15.110:3300/0 conn(0x7f5b9c06df30 0x7f5b9c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5b92ffd640 1 -- 172.21.15.107:0/1093938162 >> v2:172.21.15.110:3300/0 conn(0x7f5b9c06df30 msgr2=0x7f5b9c06e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:04.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5b92ffd640 1 --2- 172.21.15.107:0/1093938162 >> v2:172.21.15.110:3300/0 conn(0x7f5b9c06df30 0x7f5b9c06e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5b92ffd640 1 -- 172.21.15.107:0/1093938162 >> v1:172.21.15.107:6789/0 conn(0x7f5b9c14eb70 legacy=0x7f5b9c1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.321+0000 7f5b92ffd640 1 -- 172.21.15.107:0/1093938162 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c0c2be0 con 0x7f5b9c070b00 2023-10-25T00:11:04.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.325+0000 7f5b92ffd640 1 --2- 172.21.15.107:0/1093938162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b9c070b00 0x7f5b9c0c33e0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c1525e0 tx=0x7f5b8401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3e45d011baa1e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.325+0000 7f5b9affd640 1 -- 172.21.15.107:0/1093938162 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b840ab680 con 0x7f5b9c070b00 2023-10-25T00:11:04.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.325+0000 7f5b9affd640 1 -- 172.21.15.107:0/1093938162 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b840ab7e0 con 0x7f5b9c070b00 2023-10-25T00:11:04.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.325+0000 7f5b9affd640 1 -- 172.21.15.107:0/1093938162 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b840abb70 con 0x7f5b9c070b00 2023-10-25T00:11:04.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 -- 172.21.15.107:0/1093938162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b9c070b00 msgr2=0x7f5b9c0c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 --2- 172.21.15.107:0/1093938162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b9c070b00 0x7f5b9c0c33e0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c1525e0 tx=0x7f5b8401c460 comp rx=0 tx=0).stop 2023-10-25T00:11:04.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 -- 172.21.15.107:0/1093938162 shutdown_connections 2023-10-25T00:11:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 --2- 172.21.15.107:0/1093938162 >> v2:172.21.15.110:3300/0 conn(0x7f5b9c06df30 0x7f5b9c06e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 --2- 172.21.15.107:0/1093938162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b9c070b00 0x7f5b9c0c33e0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 -- 172.21.15.107:0/1093938162 >> 172.21.15.107:0/1093938162 conn(0x7f5b9c0c39a0 msgr2=0x7f5b9c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 -- 172.21.15.107:0/1093938162 shutdown_connections 2023-10-25T00:11:04.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 -- 172.21.15.107:0/1093938162 wait complete. 2023-10-25T00:11:04.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 Processor -- start 2023-10-25T00:11:04.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 -- start start 2023-10-25T00:11:04.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b9c06df30 0x7f5b9c144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b9c14eb70 0x7f5b9c069d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b9c15aaa0 0x7f5b9c06a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c1ed3e0 con 0x7f5b9c14eb70 2023-10-25T00:11:04.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c1ed910 con 0x7f5b9c06df30 2023-10-25T00:11:04.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c155450 con 0x7f5b9c15aaa0 2023-10-25T00:11:04.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b92ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b9c14eb70 0x7f5b9c069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b92ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b9c14eb70 0x7f5b9c069d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42896/0 (socket says 172.21.15.107:42896) 2023-10-25T00:11:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b92ffd640 1 -- 172.21.15.107:0/1821585421 learned_addr learned my addr 172.21.15.107:0/1821585421 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b9b7fe640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b9c06df30 0x7f5b9c144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b9bfff640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b9c15aaa0 0x7f5b9c06a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b92ffd640 1 -- 172.21.15.107:0/1821585421 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b9c15aaa0 msgr2=0x7f5b9c06a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b92ffd640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b9c15aaa0 0x7f5b9c06a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b92ffd640 1 -- 172.21.15.107:0/1821585421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b9c06df30 msgr2=0x7f5b9c144d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b92ffd640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b9c06df30 0x7f5b9c144d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b92ffd640 1 -- 172.21.15.107:0/1821585421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b84009420 con 0x7f5b9c14eb70 2023-10-25T00:11:04.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b92ffd640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b9c14eb70 0x7f5b9c069d40 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f5b840ad040 tx=0x7f5b8401aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b98ff9640 1 -- 172.21.15.107:0/1821585421 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b840b5810 con 0x7f5b9c14eb70 2023-10-25T00:11:04.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b98ff9640 1 -- 172.21.15.107:0/1821585421 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b840b5970 con 0x7f5b9c14eb70 2023-10-25T00:11:04.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 -- 172.21.15.107:0/1821585421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b9c1ed630 con 0x7f5b9c14eb70 2023-10-25T00:11:04.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5b98ff9640 1 -- 172.21.15.107:0/1821585421 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b840b5d20 con 0x7f5b9c14eb70 2023-10-25T00:11:04.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.329+0000 7f5ba1d00640 1 -- 172.21.15.107:0/1821585421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b9c145240 con 0x7f5b9c14eb70 2023-10-25T00:11:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.333+0000 7f5b91ffb640 1 -- 172.21.15.107:0/1821585421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b74005260 con 0x7f5b9c14eb70 2023-10-25T00:11:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.333+0000 7f5b98ff9640 1 -- 172.21.15.107:0/1821585421 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5b840bf410 con 0x7f5b9c14eb70 2023-10-25T00:11:04.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.333+0000 7f5b98ff9640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b7c065e60 0x7f5b7c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.333+0000 7f5b98ff9640 1 -- 172.21.15.107:0/1821585421 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7f5b8411f220 con 0x7f5b9c14eb70 2023-10-25T00:11:04.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.333+0000 7f5b9b7fe640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b7c065e60 0x7f5b7c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.333+0000 7f5b9b7fe640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b7c065e60 0x7f5b7c068320 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c005890 tx=0x7f5b8c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.337+0000 7f5b98ff9640 1 -- 172.21.15.107:0/1821585421 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5b840ebb10 con 0x7f5b9c14eb70 2023-10-25T00:11:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.357+0000 7f804f7fe640 1 -- 172.21.15.107:0/137955203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f803c005a90 con 0x7f806413bb00 2023-10-25T00:11:04.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.357+0000 7f8059ffb640 1 -- 172.21.15.107:0/137955203 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5019 (secure 0 0 0) 0x7f805c054e60 con 0x7f806413bb00 2023-10-25T00:11:04.359 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:04.360 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:03.826414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:04.360 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 156 2023-10-25T00:11:04.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 -- 172.21.15.107:0/137955203 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8048065e60 msgr2=0x7f8048068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8048065e60 0x7f8048068320 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f80600ad000 tx=0x7f80600a6000 comp rx=0 tx=0).stop 2023-10-25T00:11:04.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 -- 172.21.15.107:0/137955203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806413bb00 msgr2=0x7f806406b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806413bb00 0x7f806406b4a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f805c007e40 tx=0x7f805c00e4b0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 -- 172.21.15.107:0/137955203 shutdown_connections 2023-10-25T00:11:04.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806413bb00 0x7f806406b4a0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80640aa330 0x7f8064133000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8048065e60 0x7f8048068320 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 --2- 172.21.15.107:0/137955203 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80640a9450 0x7f806406ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 -- 172.21.15.107:0/137955203 >> 172.21.15.107:0/137955203 conn(0x7f8064070f60 msgr2=0x7f806406e980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:04.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 -- 172.21.15.107:0/137955203 shutdown_connections 2023-10-25T00:11:04.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.361+0000 7f804f7fe640 1 -- 172.21.15.107:0/137955203 wait complete. 2023-10-25T00:11:04.383 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2023-10-25T00:11:04.384 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:11:04.384 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:11:04.397 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:11:04.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.553+0000 7f5b91ffb640 1 -- 172.21.15.107:0/1821585421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b74005a90 con 0x7f5b9c14eb70 2023-10-25T00:11:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.553+0000 7f5b98ff9640 1 -- 172.21.15.107:0/1821585421 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5019 (secure 0 0 0) 0x7f5b840ef130 con 0x7f5b9c14eb70 2023-10-25T00:11:04.558 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:04.558 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:03.826414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 156 2023-10-25T00:11:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 -- 172.21.15.107:0/1821585421 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b7c065e60 msgr2=0x7f5b7c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b7c065e60 0x7f5b7c068320 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c005890 tx=0x7f5b8c009040 comp rx=0 tx=0).stop 2023-10-25T00:11:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 -- 172.21.15.107:0/1821585421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b9c14eb70 msgr2=0x7f5b9c069d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b9c14eb70 0x7f5b9c069d40 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f5b840ad040 tx=0x7f5b8401aca0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 -- 172.21.15.107:0/1821585421 shutdown_connections 2023-10-25T00:11:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b7c065e60 0x7f5b7c068320 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b9c15aaa0 0x7f5b9c06a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b9c14eb70 0x7f5b9c069d40 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 --2- 172.21.15.107:0/1821585421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b9c06df30 0x7f5b9c144d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 -- 172.21.15.107:0/1821585421 >> 172.21.15.107:0/1821585421 conn(0x7f5b9c0c39a0 msgr2=0x7f5b9c0c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 -- 172.21.15.107:0/1821585421 shutdown_connections 2023-10-25T00:11:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.557+0000 7f5b91ffb640 1 -- 172.21.15.107:0/1821585421 wait complete. 2023-10-25T00:11:04.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b95794640 1 Processor -- start 2023-10-25T00:11:04.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b95794640 1 -- start start 2023-10-25T00:11:04.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b95794640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6b90152040 0x7f6b900c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b95794640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b9006a4b0 0x7f6b900ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b95794640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b90155680 con 0x7f6b900c8100 2023-10-25T00:11:04.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b95794640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b90154a70 con 0x7f6b90152040 2023-10-25T00:11:04.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b95794640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90154d50 con 0x7f6b9006a4b0 2023-10-25T00:11:04.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8e7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6b900c8100 0x7f6b900abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49562/0 (socket says 172.21.15.107:49562) 2023-10-25T00:11:04.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8e7fc640 1 -- 172.21.15.107:0/601101506 learned_addr learned my addr 172.21.15.107:0/601101506 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:11:04.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8f7fe640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b9006a4b0 0x7f6b900ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8effd640 1 --2- 172.21.15.107:0/601101506 >> v2:172.21.15.110:3300/0 conn(0x7f6b90152040 0x7f6b900c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673099248 0 0) 0x7f6b90155680 con 0x7f6b900c8100 2023-10-25T00:11:04.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b78003530 con 0x7f6b900c8100 2023-10-25T00:11:04.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3079974130 0 0) 0x7f6b78003530 con 0x7f6b900c8100 2023-10-25T00:11:04.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b9006a4b0 msgr2=0x7f6b900ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:04.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b9006a4b0 0x7f6b900ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 >> v2:172.21.15.110:3300/0 conn(0x7f6b90152040 msgr2=0x7f6b900c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:04.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 --2- 172.21.15.107:0/601101506 >> v2:172.21.15.110:3300/0 conn(0x7f6b90152040 0x7f6b900c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b90155030 con 0x7f6b900c8100 2023-10-25T00:11:04.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6b84001680 con 0x7f6b900c8100 2023-10-25T00:11:04.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 >> v1:172.21.15.107:6789/0 conn(0x7f6b900c8100 legacy=0x7f6b900abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b78003980 0x7f6b78003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b780047d0 0x7f6b78004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b9006a4b0 0x7f6b78005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90154a70 con 0x7f6b78003980 2023-10-25T00:11:04.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90154d50 con 0x7f6b780047d0 2023-10-25T00:11:04.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7800b9c0 con 0x7f6b9006a4b0 2023-10-25T00:11:04.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8e7fc640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b780047d0 0x7f6b78004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.573+0000 7f6b8f7fe640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b9006a4b0 0x7f6b78005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.583 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2023-10-25T00:11:04.583 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:11:04.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:04.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.577+0000 7f6b8effd640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b78003980 0x7f6b78003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.577+0000 7f6b8f7fe640 1 -- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b780047d0 msgr2=0x7f6b78004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:04.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.577+0000 7f6b8f7fe640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b780047d0 0x7f6b78004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.577+0000 7f6b8f7fe640 1 -- 172.21.15.107:0/601101506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b78003980 msgr2=0x7f6b78003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:04.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.577+0000 7f6b8f7fe640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b78003980 0x7f6b78003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.577+0000 7f6b8f7fe640 1 -- 172.21.15.107:0/601101506 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b78003530 con 0x7f6b9006a4b0 2023-10-25T00:11:04.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.577+0000 7f6b8f7fe640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b9006a4b0 0x7f6b78005190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009fc0 tx=0x7f6b8801cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3993d7ad4c47bf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.577+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b880a5ba0 con 0x7f6b9006a4b0 2023-10-25T00:11:04.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.577+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8800f040 con 0x7f6b9006a4b0 2023-10-25T00:11:04.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.577+0000 7f6b8dffb640 1 -- 172.21.15.107:0/601101506 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b880ad440 con 0x7f6b9006a4b0 2023-10-25T00:11:04.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 -- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b9006a4b0 msgr2=0x7f6b78005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 --2- 172.21.15.107:0/601101506 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b9006a4b0 0x7f6b78005190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009fc0 tx=0x7f6b8801cc40 comp rx=0 tx=0).stop 2023-10-25T00:11:04.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8f7fe640 1 -- 172.21.15.107:0/601101506 reap_dead start 2023-10-25T00:11:04.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 -- 172.21.15.107:0/601101506 shutdown_connections 2023-10-25T00:11:04.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 -- 172.21.15.107:0/601101506 >> 172.21.15.107:0/601101506 conn(0x7f6b900ca200 msgr2=0x7f6b900ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:04.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 -- 172.21.15.107:0/601101506 shutdown_connections 2023-10-25T00:11:04.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 -- 172.21.15.107:0/601101506 wait complete. 2023-10-25T00:11:04.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 Processor -- start 2023-10-25T00:11:04.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 -- start start 2023-10-25T00:11:04.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b9014f540 0x7f6b901f0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b9014f920 0x7f6b901f0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b900c3bc0 0x7f6b90148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900c2320 con 0x7f6b9014f920 2023-10-25T00:11:04.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901f03d0 con 0x7f6b9014f540 2023-10-25T00:11:04.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900c1240 con 0x7f6b900c3bc0 2023-10-25T00:11:04.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b9014f920 0x7f6b901f0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b9014f920 0x7f6b901f0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42924/0 (socket says 172.21.15.107:42924) 2023-10-25T00:11:04.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8e7fc640 1 -- 172.21.15.107:0/1147991485 learned_addr learned my addr 172.21.15.107:0/1147991485 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:04.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8effd640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b9014f540 0x7f6b901f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8f7fe640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b900c3bc0 0x7f6b90148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8e7fc640 1 -- 172.21.15.107:0/1147991485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b900c3bc0 msgr2=0x7f6b90148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8e7fc640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b900c3bc0 0x7f6b90148fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8e7fc640 1 -- 172.21.15.107:0/1147991485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b9014f540 msgr2=0x7f6b901f0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8e7fc640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b9014f540 0x7f6b901f0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8e7fc640 1 -- 172.21.15.107:0/1147991485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b900c1520 con 0x7f6b9014f920 2023-10-25T00:11:04.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8effd640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b9014f540 0x7f6b901f0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:11:04.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b8e7fc640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b9014f920 0x7f6b901f0dc0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f6b84007340 tx=0x7f6b84004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b777fe640 1 -- 172.21.15.107:0/1147991485 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8400f9c0 con 0x7f6b9014f920 2023-10-25T00:11:04.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b777fe640 1 -- 172.21.15.107:0/1147991485 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8400fb60 con 0x7f6b9014f920 2023-10-25T00:11:04.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b777fe640 1 -- 172.21.15.107:0/1147991485 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b840133f0 con 0x7f6b9014f920 2023-10-25T00:11:04.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 -- 172.21.15.107:0/1147991485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b900c1800 con 0x7f6b9014f920 2023-10-25T00:11:04.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.581+0000 7f6b95794640 1 -- 172.21.15.107:0/1147991485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b900c1ae0 con 0x7f6b9014f920 2023-10-25T00:11:04.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.585+0000 7f6b95794640 1 -- 172.21.15.107:0/1147991485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b900746e0 con 0x7f6b9014f920 2023-10-25T00:11:04.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.585+0000 7f6b777fe640 1 -- 172.21.15.107:0/1147991485 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6b84013550 con 0x7f6b9014f920 2023-10-25T00:11:04.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.585+0000 7f6b777fe640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b7c065e60 0x7f6b7c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.585+0000 7f6b777fe640 1 -- 172.21.15.107:0/1147991485 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7f6b84084830 con 0x7f6b9014f920 2023-10-25T00:11:04.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.585+0000 7f6b8effd640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b7c065e60 0x7f6b7c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.585+0000 7f6b8effd640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b7c065e60 0x7f6b7c068320 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6b900673e0 tx=0x7f6b80008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.589+0000 7f6b777fe640 1 -- 172.21.15.107:0/1147991485 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6b84051120 con 0x7f6b9014f920 2023-10-25T00:11:04.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f8713240640 1 Processor -- start 2023-10-25T00:11:04.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f8713240640 1 -- start start 2023-10-25T00:11:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f8713240640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f870c0c5c20 0x7f870c153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f8713240640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870c072330 con 0x7f870c0c18a0 2023-10-25T00:11:04.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f8713240640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870c155ca0 con 0x7f870c14fc30 2023-10-25T00:11:04.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f8713240640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870c155f80 con 0x7f870c0c5c20 2023-10-25T00:11:04.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f87117dd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f870c0c18a0 0x7f870c0c1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49576/0 (socket says 172.21.15.107:49576) 2023-10-25T00:11:04.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f87117dd640 1 -- 172.21.15.107:0/842037602 learned_addr learned my addr 172.21.15.107:0/842037602 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:11:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870bfff640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f870c0c5c20 0x7f870c153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432923223 0 0) 0x7f870c072330 con 0x7f870c0c18a0 2023-10-25T00:11:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86fc003530 con 0x7f870c0c18a0 2023-10-25T00:11:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 466826869 0 0) 0x7f86fc003530 con 0x7f870c0c18a0 2023-10-25T00:11:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f870c0c5c20 msgr2=0x7f870c153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f870c0c5c20 0x7f870c153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 >> v1:172.21.15.110:6789/0 conn(0x7f870c14fc30 legacy=0x7f870c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c156260 con 0x7f870c0c18a0 2023-10-25T00:11:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f8704001a00 con 0x7f870c0c18a0 2023-10-25T00:11:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 >> v1:172.21.15.107:6789/0 conn(0x7f870c0c18a0 legacy=0x7f870c0c1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f870c0c5c20 0x7f86fc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86fc003f30 0x7f86fc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86fc004d10 0x7f86fc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870c155f80 con 0x7f86fc003f30 2023-10-25T00:11:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc00bad0 con 0x7f86fc004d10 2023-10-25T00:11:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc00bc90 con 0x7f870c0c5c20 2023-10-25T00:11:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870bfff640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86fc003f30 0x7f86fc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f87117dd640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86fc004d10 0x7f86fc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f8710fdc640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f870c0c5c20 0x7f86fc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870bfff640 1 -- 172.21.15.107:0/842037602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f870c0c5c20 msgr2=0x7f86fc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:04.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870bfff640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f870c0c5c20 0x7f86fc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870bfff640 1 -- 172.21.15.107:0/842037602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86fc004d10 msgr2=0x7f86fc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870bfff640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f86fc004d10 0x7f86fc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.705+0000 7f870bfff640 1 -- 172.21.15.107:0/842037602 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c156260 con 0x7f86fc003f30 2023-10-25T00:11:04.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f870bfff640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86fc003f30 0x7f86fc0047d0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8700009ae0 tx=0x7f870001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e61577ef820ba9 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000a6450 con 0x7f86fc003f30 2023-10-25T00:11:04.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87000a65b0 con 0x7f86fc003f30 2023-10-25T00:11:04.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f870b7fe640 1 -- 172.21.15.107:0/842037602 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000a6960 con 0x7f86fc003f30 2023-10-25T00:11:04.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f8713240640 1 -- 172.21.15.107:0/842037602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86fc003f30 msgr2=0x7f86fc0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f8713240640 1 --2- 172.21.15.107:0/842037602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86fc003f30 0x7f86fc0047d0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f8700009ae0 tx=0x7f870001c960 comp rx=0 tx=0).stop 2023-10-25T00:11:04.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f87117dd640 1 -- 172.21.15.107:0/842037602 reap_dead start 2023-10-25T00:11:04.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f8713240640 1 -- 172.21.15.107:0/842037602 shutdown_connections 2023-10-25T00:11:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f8713240640 1 -- 172.21.15.107:0/842037602 >> 172.21.15.107:0/842037602 conn(0x7f870c0c39a0 msgr2=0x7f870c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f8713240640 1 -- 172.21.15.107:0/842037602 shutdown_connections 2023-10-25T00:11:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f8713240640 1 -- 172.21.15.107:0/842037602 wait complete. 2023-10-25T00:11:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f8713240640 1 Processor -- start 2023-10-25T00:11:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.709+0000 7f8713240640 1 -- start start 2023-10-25T00:11:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8713240640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86fc003f30 0x7f870c1513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8713240640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f870c150390 0x7f870c06c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8713240640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f870c13a7f0 0x7f870c13ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8713240640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870c1ed770 con 0x7f870c150390 2023-10-25T00:11:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8713240640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870c1ece40 con 0x7f870c13a7f0 2023-10-25T00:11:04.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8713240640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870c1ed120 con 0x7f86fc003f30 2023-10-25T00:11:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f870bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f870c150390 0x7f870c06c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f870bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f870c150390 0x7f870c06c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42934/0 (socket says 172.21.15.107:42934) 2023-10-25T00:11:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f870bfff640 1 -- 172.21.15.107:0/3341412265 learned_addr learned my addr 172.21.15.107:0/3341412265 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8710fdc640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86fc003f30 0x7f870c1513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f87117dd640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f870c13a7f0 0x7f870c13ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f870bfff640 1 -- 172.21.15.107:0/3341412265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86fc003f30 msgr2=0x7f870c1513e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f870bfff640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86fc003f30 0x7f870c1513e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f870bfff640 1 -- 172.21.15.107:0/3341412265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f870c13a7f0 msgr2=0x7f870c13ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f870bfff640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f870c13a7f0 0x7f870c13ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f870bfff640 1 -- 172.21.15.107:0/3341412265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c1ed400 con 0x7f870c150390 2023-10-25T00:11:04.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8710fdc640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86fc003f30 0x7f870c1513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:11:04.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f87117dd640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f870c13a7f0 0x7f870c13ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:11:04.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f870bfff640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f870c150390 0x7f870c06c130 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8700009ae0 tx=0x7f86fc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f87097fa640 1 -- 172.21.15.107:0/3341412265 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000a6dd0 con 0x7f870c150390 2023-10-25T00:11:04.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8713240640 1 -- 172.21.15.107:0/3341412265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870c06c670 con 0x7f870c150390 2023-10-25T00:11:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8713240640 1 -- 172.21.15.107:0/3341412265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870c06ffc0 con 0x7f870c150390 2023-10-25T00:11:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f87097fa640 1 -- 172.21.15.107:0/3341412265 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870000f000 con 0x7f870c150390 2023-10-25T00:11:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f87097fa640 1 -- 172.21.15.107:0/3341412265 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000a7630 con 0x7f870c150390 2023-10-25T00:11:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8713240640 1 -- 172.21.15.107:0/3341412265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ec005260 con 0x7f870c150390 2023-10-25T00:11:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f87097fa640 1 -- 172.21.15.107:0/3341412265 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f87000a7790 con 0x7f870c150390 2023-10-25T00:11:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f87097fa640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86f0065e90 0x7f86f0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:04.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f87097fa640 1 -- 172.21.15.107:0/3341412265 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7f870011e410 con 0x7f870c150390 2023-10-25T00:11:04.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8710fdc640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86f0065e90 0x7f86f0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:04.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.713+0000 7f8710fdc640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86f0065e90 0x7f86f0068350 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f86f80016e0 tx=0x7f86f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:04.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.717+0000 7f87097fa640 1 -- 172.21.15.107:0/3341412265 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f87000ead30 con 0x7f870c150390 2023-10-25T00:11:04.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 -- 172.21.15.107:0/1147991485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b900ad310 con 0x7f6b9014f920 2023-10-25T00:11:04.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b777fe640 1 -- 172.21.15.107:0/1147991485 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5019 (secure 0 0 0) 0x7f6b84054740 con 0x7f6b9014f920 2023-10-25T00:11:04.823 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:04.823 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:03.826414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:04.824 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 156 2023-10-25T00:11:04.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 -- 172.21.15.107:0/1147991485 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b7c065e60 msgr2=0x7f6b7c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b7c065e60 0x7f6b7c068320 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6b900673e0 tx=0x7f6b80008040 comp rx=0 tx=0).stop 2023-10-25T00:11:04.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 -- 172.21.15.107:0/1147991485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b9014f920 msgr2=0x7f6b901f0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b9014f920 0x7f6b901f0dc0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f6b84007340 tx=0x7f6b84004f90 comp rx=0 tx=0).stop 2023-10-25T00:11:04.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 -- 172.21.15.107:0/1147991485 shutdown_connections 2023-10-25T00:11:04.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6b7c065e60 0x7f6b7c068320 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6b900c3bc0 0x7f6b90148fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b9014f920 0x7f6b901f0dc0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 --2- 172.21.15.107:0/1147991485 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b9014f540 0x7f6b901f0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 -- 172.21.15.107:0/1147991485 >> 172.21.15.107:0/1147991485 conn(0x7f6b900ca200 msgr2=0x7f6b901539e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 -- 172.21.15.107:0/1147991485 shutdown_connections 2023-10-25T00:11:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.821+0000 7f6b95794640 1 -- 172.21.15.107:0/1147991485 wait complete. 2023-10-25T00:11:04.839 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:11:04.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.933+0000 7f8713240640 1 -- 172.21.15.107:0/3341412265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86ec005a90 con 0x7f870c150390 2023-10-25T00:11:04.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.933+0000 7f87097fa640 1 -- 172.21.15.107:0/3341412265 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+5022 (secure 0 0 0) 0x7f87000ee350 con 0x7f870c150390 2023-10-25T00:11:04.936 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:04.936 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:04.846815+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:04.936 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 157 2023-10-25T00:11:04.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 -- 172.21.15.107:0/3341412265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86f0065e90 msgr2=0x7f86f0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86f0065e90 0x7f86f0068350 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f86f80016e0 tx=0x7f86f8009040 comp rx=0 tx=0).stop 2023-10-25T00:11:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 -- 172.21.15.107:0/3341412265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f870c150390 msgr2=0x7f870c06c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f870c150390 0x7f870c06c130 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f8700009ae0 tx=0x7f86fc005800 comp rx=0 tx=0).stop 2023-10-25T00:11:04.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 -- 172.21.15.107:0/3341412265 shutdown_connections 2023-10-25T00:11:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f870c13a7f0 0x7f870c13ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f870c150390 0x7f870c06c130 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f86f0065e90 0x7f86f0068350 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 --2- 172.21.15.107:0/3341412265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f86fc003f30 0x7f870c1513e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:04.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 -- 172.21.15.107:0/3341412265 >> 172.21.15.107:0/3341412265 conn(0x7f870c0c39a0 msgr2=0x7f870c0c5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 -- 172.21.15.107:0/3341412265 shutdown_connections 2023-10-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:04.937+0000 7f8713240640 1 -- 172.21.15.107:0/3341412265 wait complete. 2023-10-25T00:11:04.953 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 157, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 154, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7746": { "addr": "172.21.15.107:6835/3474118649", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3474118649, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 3474118649, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7746, "incarnation": 151, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:10:41.848063+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7746 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 157, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7839": { "addr": "172.21.15.107:6837/3370462806", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 3370462806, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 3370462806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7839, "incarnation": 156, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:11:04.846815+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7839 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:11:06.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:06.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:06.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6c2f1640 1 Processor -- start 2023-10-25T00:11:06.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6c2f1640 1 -- start start 2023-10-25T00:11:06.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6c2f1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6a6413bb00 0x7f6a641531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:06.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6c2f1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640aa330 0x7f6a6406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:06.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6c2f1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6a640a9450 0x7f6a64071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:06.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6c2f1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a64156100 con 0x7f6a6413bb00 2023-10-25T00:11:06.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6c2f1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a641527c0 con 0x7f6a640a9450 2023-10-25T00:11:06.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6c2f1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a64152d40 con 0x7f6a640aa330 2023-10-25T00:11:06.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6a08d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6a6413bb00 0x7f6a641531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:06.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6a08d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6a6413bb00 0x7f6a641531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42948/0 (socket says 172.21.15.107:42948) 2023-10-25T00:11:06.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6a08d640 1 -- 172.21.15.107:0/3456634265 learned_addr learned my addr 172.21.15.107:0/3456634265 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6988c640 1 --2- 172.21.15.107:0/3456634265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640aa330 0x7f6a6406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6a08d640 1 -- 172.21.15.107:0/3456634265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640aa330 msgr2=0x7f6a6406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6a88e640 1 --2- 172.21.15.107:0/3456634265 >> v2:172.21.15.110:3300/0 conn(0x7f6a640a9450 0x7f6a64071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6a08d640 1 --2- 172.21.15.107:0/3456634265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640aa330 0x7f6a6406e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6a08d640 1 -- 172.21.15.107:0/3456634265 >> v2:172.21.15.110:3300/0 conn(0x7f6a640a9450 msgr2=0x7f6a64071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:06.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6a08d640 1 --2- 172.21.15.107:0/3456634265 >> v2:172.21.15.110:3300/0 conn(0x7f6a640a9450 0x7f6a64071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:06.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6a08d640 1 -- 172.21.15.107:0/3456634265 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6406d660 con 0x7f6a6413bb00 2023-10-25T00:11:06.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.969+0000 7f6a6988c640 1 --2- 172.21.15.107:0/3456634265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640aa330 0x7f6a6406e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:11:06.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6a08d640 1 --2- 172.21.15.107:0/3456634265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a6413bb00 0x7f6a641531d0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009870 tx=0x7f6a6001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9e8ece14ec84c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:06.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6908b640 1 -- 172.21.15.107:0/3456634265 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a600ab600 con 0x7f6a6413bb00 2023-10-25T00:11:06.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6908b640 1 -- 172.21.15.107:0/3456634265 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a600a7050 con 0x7f6a6413bb00 2023-10-25T00:11:06.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6908b640 1 -- 172.21.15.107:0/3456634265 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a600ab980 con 0x7f6a6413bb00 2023-10-25T00:11:06.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/3456634265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a6413bb00 msgr2=0x7f6a641531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:06.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6c2f1640 1 --2- 172.21.15.107:0/3456634265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a6413bb00 0x7f6a641531d0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009870 tx=0x7f6a6001c3e0 comp rx=0 tx=0).stop 2023-10-25T00:11:06.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/3456634265 shutdown_connections 2023-10-25T00:11:06.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6c2f1640 1 --2- 172.21.15.107:0/3456634265 >> v2:172.21.15.110:3300/0 conn(0x7f6a640a9450 0x7f6a64071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:06.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6c2f1640 1 --2- 172.21.15.107:0/3456634265 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640aa330 0x7f6a6406e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:06.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6c2f1640 1 --2- 172.21.15.107:0/3456634265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a6413bb00 0x7f6a641531d0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:06.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/3456634265 >> 172.21.15.107:0/3456634265 conn(0x7f6a64070f60 msgr2=0x7f6a64071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:06.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/3456634265 shutdown_connections 2023-10-25T00:11:06.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.973+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/3456634265 wait complete. 2023-10-25T00:11:06.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6c2f1640 1 Processor -- start 2023-10-25T00:11:06.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6c2f1640 1 -- start start 2023-10-25T00:11:06.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6c2f1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640a9450 0x7f6a641361e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:06.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6c2f1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a640aa330 0x7f6a64131a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:06.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6c2f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a6413bb00 0x7f6a64131f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:06.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6c2f1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a641ebcd0 con 0x7f6a6413bb00 2023-10-25T00:11:06.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6c2f1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a641eb330 con 0x7f6a640aa330 2023-10-25T00:11:06.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6c2f1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a641eb610 con 0x7f6a640a9450 2023-10-25T00:11:06.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6a88e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a6413bb00 0x7f6a64131f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:06.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6a88e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a6413bb00 0x7f6a64131f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42950/0 (socket says 172.21.15.107:42950) 2023-10-25T00:11:06.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6a88e640 1 -- 172.21.15.107:0/963045484 learned_addr learned my addr 172.21.15.107:0/963045484 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:06.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6a08d640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640a9450 0x7f6a641361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:06.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6988c640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a640aa330 0x7f6a64131a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:06.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6a88e640 1 -- 172.21.15.107:0/963045484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640a9450 msgr2=0x7f6a641361e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:06.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6a88e640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640a9450 0x7f6a641361e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:06.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6a88e640 1 -- 172.21.15.107:0/963045484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a640aa330 msgr2=0x7f6a64131a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:06.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6a88e640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a640aa330 0x7f6a64131a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:06.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6a88e640 1 -- 172.21.15.107:0/963045484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a600094d0 con 0x7f6a6413bb00 2023-10-25T00:11:06.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6a08d640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640a9450 0x7f6a641361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:11:06.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6a88e640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a6413bb00 0x7f6a64131f80 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00ccf0 tx=0x7f6a5c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:06.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a5affd640 1 -- 172.21.15.107:0/963045484 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c003ba0 con 0x7f6a6413bb00 2023-10-25T00:11:06.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a5affd640 1 -- 172.21.15.107:0/963045484 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c003d00 con 0x7f6a6413bb00 2023-10-25T00:11:06.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a5affd640 1 -- 172.21.15.107:0/963045484 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c004240 con 0x7f6a6413bb00 2023-10-25T00:11:06.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/963045484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a641eb8f0 con 0x7f6a6413bb00 2023-10-25T00:11:06.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/963045484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a64136720 con 0x7f6a6413bb00 2023-10-25T00:11:06.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.977+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/963045484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a641ebe10 con 0x7f6a6413bb00 2023-10-25T00:11:06.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.981+0000 7f6a5affd640 1 -- 172.21.15.107:0/963045484 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6a5c0043a0 con 0x7f6a6413bb00 2023-10-25T00:11:06.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.981+0000 7f6a5affd640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6a48065e60 0x7f6a48068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:06.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.981+0000 7f6a5affd640 1 -- 172.21.15.107:0/963045484 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7f6a5c012070 con 0x7f6a6413bb00 2023-10-25T00:11:06.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.981+0000 7f6a6a08d640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6a48065e60 0x7f6a48068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:06.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.981+0000 7f6a6a08d640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6a48065e60 0x7f6a48068320 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f6a600ad000 tx=0x7f6a60008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:06.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:06.985+0000 7f6a5affd640 1 -- 172.21.15.107:0/963045484 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6a641ebe10 con 0x7f6a6413bb00 2023-10-25T00:11:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.077+0000 7fcd46252640 1 Processor -- start 2023-10-25T00:11:07.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.077+0000 7fcd46252640 1 -- start start 2023-10-25T00:11:07.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcd4014c470 0x7fcd4014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcd4006f490 0x7fcd40071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd40155cd0 con 0x7fcd40071fc0 2023-10-25T00:11:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd401526e0 con 0x7fcd4006f490 2023-10-25T00:11:07.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd400c4380 con 0x7fcd4014c470 2023-10-25T00:11:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd37fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fcd4006f490 0x7fcd40071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd3ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fcd40071fc0 0x7fcd40152b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49586/0 (socket says 172.21.15.107:49586) 2023-10-25T00:11:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd3ffff640 1 -- 172.21.15.107:0/4143644007 learned_addr learned my addr 172.21.15.107:0/4143644007 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:11:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd3f7fe640 1 --2- 172.21.15.107:0/4143644007 >> v2:172.21.15.110:3300/0 conn(0x7fcd4014c470 0x7fcd4014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:07.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd37fff640 1 -- 172.21.15.107:0/4143644007 >> v2:172.21.15.110:3300/0 conn(0x7fcd4014c470 msgr2=0x7fcd4014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd37fff640 1 --2- 172.21.15.107:0/4143644007 >> v2:172.21.15.110:3300/0 conn(0x7fcd4014c470 0x7fcd4014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd37fff640 1 -- 172.21.15.107:0/4143644007 >> v1:172.21.15.107:6789/0 conn(0x7fcd40071fc0 legacy=0x7fcd40152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd3effd640 1 -- 172.21.15.107:0/4143644007 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669029540 0 0) 0x7fcd40155cd0 con 0x7fcd40071fc0 2023-10-25T00:11:07.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd37fff640 1 -- 172.21.15.107:0/4143644007 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd400c48b0 con 0x7fcd4006f490 2023-10-25T00:11:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd37fff640 1 --2- 172.21.15.107:0/4143644007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4006f490 0x7fcd40071a80 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fcd4006a8e0 tx=0x7fcd2801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e734315f8a5de366 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd3effd640 1 -- 172.21.15.107:0/4143644007 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd280ab600 con 0x7fcd4006f490 2023-10-25T00:11:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd3effd640 1 -- 172.21.15.107:0/4143644007 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd280a7050 con 0x7fcd4006f490 2023-10-25T00:11:07.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd3effd640 1 -- 172.21.15.107:0/4143644007 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd280ab980 con 0x7fcd4006f490 2023-10-25T00:11:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 -- 172.21.15.107:0/4143644007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4006f490 msgr2=0x7fcd40071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 --2- 172.21.15.107:0/4143644007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4006f490 0x7fcd40071a80 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fcd4006a8e0 tx=0x7fcd2801c340 comp rx=0 tx=0).stop 2023-10-25T00:11:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 -- 172.21.15.107:0/4143644007 shutdown_connections 2023-10-25T00:11:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 --2- 172.21.15.107:0/4143644007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4006f490 0x7fcd40071a80 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 --2- 172.21.15.107:0/4143644007 >> v2:172.21.15.110:3300/0 conn(0x7fcd4014c470 0x7fcd4014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 -- 172.21.15.107:0/4143644007 >> 172.21.15.107:0/4143644007 conn(0x7fcd40070f60 msgr2=0x7fcd40071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:07.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 -- 172.21.15.107:0/4143644007 shutdown_connections 2023-10-25T00:11:07.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.081+0000 7fcd46252640 1 -- 172.21.15.107:0/4143644007 wait complete. 2023-10-25T00:11:07.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd46252640 1 Processor -- start 2023-10-25T00:11:07.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd46252640 1 -- start start 2023-10-25T00:11:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd46252640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4006f490 0x7fcd40143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd46252640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd40071fc0 0x7fcd40154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:07.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd46252640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd4014c470 0x7fcd4014afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:07.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd46252640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd40155750 con 0x7fcd4006f490 2023-10-25T00:11:07.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd46252640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd401409f0 con 0x7fcd40071fc0 2023-10-25T00:11:07.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd46252640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd40144d30 con 0x7fcd4014c470 2023-10-25T00:11:07.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4006f490 0x7fcd40143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:07.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4006f490 0x7fcd40143090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42964/0 (socket says 172.21.15.107:42964) 2023-10-25T00:11:07.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3f7fe640 1 -- 172.21.15.107:0/1942625270 learned_addr learned my addr 172.21.15.107:0/1942625270 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:07.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3ffff640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd4014c470 0x7fcd4014afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:07.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd37fff640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd40071fc0 0x7fcd40154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3f7fe640 1 -- 172.21.15.107:0/1942625270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd4014c470 msgr2=0x7fcd4014afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3f7fe640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd4014c470 0x7fcd4014afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3f7fe640 1 -- 172.21.15.107:0/1942625270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd40071fc0 msgr2=0x7fcd40154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3f7fe640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd40071fc0 0x7fcd40154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3f7fe640 1 -- 172.21.15.107:0/1942625270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd28009420 con 0x7fcd4006f490 2023-10-25T00:11:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3f7fe640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4006f490 0x7fcd40143090 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000d650 tx=0x7fcd3000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3cff9640 1 -- 172.21.15.107:0/1942625270 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3000b6c0 con 0x7fcd4006f490 2023-10-25T00:11:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3cff9640 1 -- 172.21.15.107:0/1942625270 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3000b820 con 0x7fcd4006f490 2023-10-25T00:11:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd3cff9640 1 -- 172.21.15.107:0/1942625270 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3000bbb0 con 0x7fcd4006f490 2023-10-25T00:11:07.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd46252640 1 -- 172.21.15.107:0/1942625270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd40145700 con 0x7fcd4006f490 2023-10-25T00:11:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd46252640 1 -- 172.21.15.107:0/1942625270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd40144530 con 0x7fcd4006f490 2023-10-25T00:11:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.085+0000 7fcd46252640 1 -- 172.21.15.107:0/1942625270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd400746e0 con 0x7fcd4006f490 2023-10-25T00:11:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.089+0000 7fcd3cff9640 1 -- 172.21.15.107:0/1942625270 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcd3000bd10 con 0x7fcd4006f490 2023-10-25T00:11:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.089+0000 7fcd3cff9640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd20065e60 0x7fcd20068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.089+0000 7fcd3cff9640 1 -- 172.21.15.107:0/1942625270 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7fcd30084920 con 0x7fcd4006f490 2023-10-25T00:11:07.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.089+0000 7fcd37fff640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd20065e60 0x7fcd20068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:07.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.089+0000 7fcd37fff640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd20065e60 0x7fcd20068320 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcd280ad000 tx=0x7fcd28008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:07.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.097+0000 7fcd3cff9640 1 -- 172.21.15.107:0/1942625270 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fcd300512c0 con 0x7fcd4006f490 2023-10-25T00:11:07.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.181+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/963045484 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a641ebe10 con 0x7f6a6413bb00 2023-10-25T00:11:07.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.181+0000 7f6a5affd640 1 -- 172.21.15.107:0/963045484 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5019 (secure 0 0 0) 0x7f6a641ebe10 con 0x7f6a6413bb00 2023-10-25T00:11:07.183 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:07.183 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:07.184 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 159 2023-10-25T00:11:07.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/963045484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6a48065e60 msgr2=0x7f6a48068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:07.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6a48065e60 0x7f6a48068320 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f6a600ad000 tx=0x7f6a60008040 comp rx=0 tx=0).stop 2023-10-25T00:11:07.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/963045484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a6413bb00 msgr2=0x7f6a64131f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:07.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a6413bb00 0x7f6a64131f80 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00ccf0 tx=0x7f6a5c00a660 comp rx=0 tx=0).stop 2023-10-25T00:11:07.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/963045484 shutdown_connections 2023-10-25T00:11:07.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a6413bb00 0x7f6a64131f80 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a640aa330 0x7f6a64131a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6a48065e60 0x7f6a48068320 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 --2- 172.21.15.107:0/963045484 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a640a9450 0x7f6a641361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/963045484 >> 172.21.15.107:0/963045484 conn(0x7f6a64070f60 msgr2=0x7f6a64073200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:07.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/963045484 shutdown_connections 2023-10-25T00:11:07.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.185+0000 7f6a6c2f1640 1 -- 172.21.15.107:0/963045484 wait complete. 2023-10-25T00:11:07.201 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:11:07.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.289+0000 7fcd46252640 1 -- 172.21.15.107:0/1942625270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd400c2440 con 0x7fcd4006f490 2023-10-25T00:11:07.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd3cff9640 1 -- 172.21.15.107:0/1942625270 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5019 (secure 0 0 0) 0x7fcd30007380 con 0x7fcd4006f490 2023-10-25T00:11:07.294 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:07.294 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:07.294 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 159 2023-10-25T00:11:07.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 -- 172.21.15.107:0/1942625270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd20065e60 msgr2=0x7fcd20068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:07.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd20065e60 0x7fcd20068320 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcd280ad000 tx=0x7fcd28008040 comp rx=0 tx=0).stop 2023-10-25T00:11:07.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 -- 172.21.15.107:0/1942625270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4006f490 msgr2=0x7fcd40143090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:07.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4006f490 0x7fcd40143090 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000d650 tx=0x7fcd3000db10 comp rx=0 tx=0).stop 2023-10-25T00:11:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 -- 172.21.15.107:0/1942625270 shutdown_connections 2023-10-25T00:11:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcd20065e60 0x7fcd20068320 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd4014c470 0x7fcd4014afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd40071fc0 0x7fcd40154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 --2- 172.21.15.107:0/1942625270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd4006f490 0x7fcd40143090 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:07.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 -- 172.21.15.107:0/1942625270 >> 172.21.15.107:0/1942625270 conn(0x7fcd40070f60 msgr2=0x7fcd40151cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:07.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 -- 172.21.15.107:0/1942625270 shutdown_connections 2023-10-25T00:11:07.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:07.293+0000 7fcd46252640 1 -- 172.21.15.107:0/1942625270 wait complete. 2023-10-25T00:11:07.311 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=6568 2023-10-25T00:11:07.312 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before reviving mds.d 2023-10-25T00:11:09.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:09.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.329+0000 7fdf3afb5640 1 Processor -- start 2023-10-25T00:11:09.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.329+0000 7fdf3afb5640 1 -- start start 2023-10-25T00:11:09.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf3afb5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34152020 0x7fdf340c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:09.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf3afb5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf34152400 con 0x7fdf340c8640 2023-10-25T00:11:09.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf3afb5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf341558d0 con 0x7fdf341553d0 2023-10-25T00:11:09.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf3afb5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340c6ce0 con 0x7fdf34152020 2023-10-25T00:11:09.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf33fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fdf340c8640 0x7fdf3406a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49592/0 (socket says 172.21.15.107:49592) 2023-10-25T00:11:09.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf33fff640 1 -- 172.21.15.107:0/4137681031 learned_addr learned my addr 172.21.15.107:0/4137681031 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:11:09.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf38d51640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34152020 0x7fdf340c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:09.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774463276 0 0) 0x7fdf34152400 con 0x7fdf340c8640 2023-10-25T00:11:09.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf1c003530 con 0x7fdf340c8640 2023-10-25T00:11:09.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2687135016 0 0) 0x7fdf1c003530 con 0x7fdf340c8640 2023-10-25T00:11:09.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34152020 msgr2=0x7fdf340c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:09.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34152020 0x7fdf340c4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:09.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 >> v1:172.21.15.110:6789/0 conn(0x7fdf341553d0 legacy=0x7fdf3406ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:09.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf340c6fc0 con 0x7fdf340c8640 2023-10-25T00:11:09.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fdf28001680 con 0x7fdf340c8640 2023-10-25T00:11:09.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 >> v1:172.21.15.107:6789/0 conn(0x7fdf340c8640 legacy=0x7fdf3406a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:09.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf1c0039b0 0x7fdf1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:09.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34152020 0x7fdf1c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:09.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 0x7fdf1c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:09.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340c6ce0 con 0x7fdf1c004d10 2023-10-25T00:11:09.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c00bb50 con 0x7fdf1c0039b0 2023-10-25T00:11:09.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c00bd10 con 0x7fdf34152020 2023-10-25T00:11:09.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf39552640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 0x7fdf1c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:09.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf38d51640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf1c0039b0 0x7fdf1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:09.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf33fff640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34152020 0x7fdf1c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf39552640 1 -- 172.21.15.107:0/4137681031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34152020 msgr2=0x7fdf1c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf39552640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34152020 0x7fdf1c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf39552640 1 -- 172.21.15.107:0/4137681031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf1c0039b0 msgr2=0x7fdf1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf39552640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf1c0039b0 0x7fdf1c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.333+0000 7fdf39552640 1 -- 172.21.15.107:0/4137681031 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c003530 con 0x7fdf1c004d10 2023-10-25T00:11:09.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf39552640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 0x7fdf1c005190 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c001e70 tx=0x7fdf2c01d190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207cce9e41cf384b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:09.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c003980 con 0x7fdf1c004d10 2023-10-25T00:11:09.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2c003ae0 con 0x7fdf1c004d10 2023-10-25T00:11:09.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf337fe640 1 -- 172.21.15.107:0/4137681031 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c003e90 con 0x7fdf1c004d10 2023-10-25T00:11:09.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 -- 172.21.15.107:0/4137681031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 msgr2=0x7fdf1c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:09.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 --2- 172.21.15.107:0/4137681031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 0x7fdf1c005190 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c001e70 tx=0x7fdf2c01d190 comp rx=0 tx=0).stop 2023-10-25T00:11:09.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf39552640 1 -- 172.21.15.107:0/4137681031 reap_dead start 2023-10-25T00:11:09.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 -- 172.21.15.107:0/4137681031 shutdown_connections 2023-10-25T00:11:09.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 -- 172.21.15.107:0/4137681031 >> 172.21.15.107:0/4137681031 conn(0x7fdf340abd60 msgr2=0x7fdf340ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:09.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 -- 172.21.15.107:0/4137681031 shutdown_connections 2023-10-25T00:11:09.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 -- 172.21.15.107:0/4137681031 wait complete. 2023-10-25T00:11:09.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 Processor -- start 2023-10-25T00:11:09.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 -- start start 2023-10-25T00:11:09.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 0x7fdf340ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:09.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf340ae540 0x7fdf3414d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34158090 0x7fdf34158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340afcd0 con 0x7fdf1c004d10 2023-10-25T00:11:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf340ad410 con 0x7fdf340ae540 2023-10-25T00:11:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf341ecc30 con 0x7fdf34158090 2023-10-25T00:11:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf38d51640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 0x7fdf340ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf38d51640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 0x7fdf340ad830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42984/0 (socket says 172.21.15.107:42984) 2023-10-25T00:11:09.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf38d51640 1 -- 172.21.15.107:0/1173838150 learned_addr learned my addr 172.21.15.107:0/1173838150 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:09.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf39552640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34158090 0x7fdf34158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:09.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf33fff640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf340ae540 0x7fdf3414d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:09.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf38d51640 1 -- 172.21.15.107:0/1173838150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34158090 msgr2=0x7fdf34158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:09.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf38d51640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34158090 0x7fdf34158510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:09.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf38d51640 1 -- 172.21.15.107:0/1173838150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf340ae540 msgr2=0x7fdf3414d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:09.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf38d51640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf340ae540 0x7fdf3414d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:09.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf38d51640 1 -- 172.21.15.107:0/1173838150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf340add70 con 0x7fdf1c004d10 2023-10-25T00:11:09.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf38d51640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 0x7fdf340ad830 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fdf240115e0 tx=0x7fdf24011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:09.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf317fa640 1 -- 172.21.15.107:0/1173838150 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2400f8f0 con 0x7fdf1c004d10 2023-10-25T00:11:09.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf317fa640 1 -- 172.21.15.107:0/1173838150 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2400fa50 con 0x7fdf1c004d10 2023-10-25T00:11:09.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 -- 172.21.15.107:0/1173838150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf341ece20 con 0x7fdf1c004d10 2023-10-25T00:11:09.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf317fa640 1 -- 172.21.15.107:0/1173838150 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2400fdc0 con 0x7fdf1c004d10 2023-10-25T00:11:09.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.337+0000 7fdf3afb5640 1 -- 172.21.15.107:0/1173838150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3414d630 con 0x7fdf1c004d10 2023-10-25T00:11:09.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.341+0000 7fdf3afb5640 1 -- 172.21.15.107:0/1173838150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf340ace20 con 0x7fdf1c004d10 2023-10-25T00:11:09.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.341+0000 7fdf317fa640 1 -- 172.21.15.107:0/1173838150 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdf24015460 con 0x7fdf1c004d10 2023-10-25T00:11:09.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.341+0000 7fdf317fa640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e90 0x7fdf20068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:09.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.341+0000 7fdf317fa640 1 -- 172.21.15.107:0/1173838150 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7fdf24084a40 con 0x7fdf1c004d10 2023-10-25T00:11:09.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.341+0000 7fdf33fff640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e90 0x7fdf20068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:09.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.341+0000 7fdf33fff640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e90 0x7fdf20068350 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fdf28000f70 tx=0x7fdf28000ee0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:09.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.345+0000 7fdf317fa640 1 -- 172.21.15.107:0/1173838150 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdf24086e60 con 0x7fdf1c004d10 2023-10-25T00:11:09.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.537+0000 7fdf3afb5640 1 -- 172.21.15.107:0/1173838150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf340746e0 con 0x7fdf1c004d10 2023-10-25T00:11:09.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.537+0000 7fdf317fa640 1 -- 172.21.15.107:0/1173838150 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5019 (secure 0 0 0) 0x7fdf24022090 con 0x7fdf1c004d10 2023-10-25T00:11:09.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:09.539 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:09.539 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 159 2023-10-25T00:11:09.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 -- 172.21.15.107:0/1173838150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e90 msgr2=0x7fdf20068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:09.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e90 0x7fdf20068350 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fdf28000f70 tx=0x7fdf28000ee0 comp rx=0 tx=0).stop 2023-10-25T00:11:09.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 -- 172.21.15.107:0/1173838150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 msgr2=0x7fdf340ad830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:09.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 0x7fdf340ad830 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fdf240115e0 tx=0x7fdf24011aa0 comp rx=0 tx=0).stop 2023-10-25T00:11:09.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 -- 172.21.15.107:0/1173838150 shutdown_connections 2023-10-25T00:11:09.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdf20065e90 0x7fdf20068350 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:09.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdf34158090 0x7fdf34158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:09.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdf340ae540 0x7fdf3414d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:09.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 --2- 172.21.15.107:0/1173838150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf1c004d10 0x7fdf340ad830 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:09.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 -- 172.21.15.107:0/1173838150 >> 172.21.15.107:0/1173838150 conn(0x7fdf340abd60 msgr2=0x7fdf34151260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:09.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 -- 172.21.15.107:0/1173838150 shutdown_connections 2023-10-25T00:11:09.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:09.541+0000 7fdf3afb5640 1 -- 172.21.15.107:0/1173838150 wait complete. 2023-10-25T00:11:09.557 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:11:11.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:11.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.685+0000 7fe05df82640 1 Processor -- start 2023-10-25T00:11:11.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe05df82640 1 -- start start 2023-10-25T00:11:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe05df82640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0580a9820 0x7fe0580ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe05df82640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe05806ee40 0x7fe0580aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe05df82640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05806f730 con 0x7fe05814fd40 2023-10-25T00:11:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe05df82640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0580aa750 con 0x7fe0580a9820 2023-10-25T00:11:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe05df82640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe058156a40 con 0x7fe05806ee40 2023-10-25T00:11:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe04effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0580a9820 0x7fe0580ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe04effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0580a9820 0x7fe0580ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42992/0 (socket says 172.21.15.107:42992) 2023-10-25T00:11:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe04effd640 1 -- 172.21.15.107:0/3626024422 learned_addr learned my addr 172.21.15.107:0/3626024422 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:11.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe057fff640 1 --2- 172.21.15.107:0/3626024422 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe05806ee40 0x7fe0580aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:11.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe04effd640 1 -- 172.21.15.107:0/3626024422 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe05806ee40 msgr2=0x7fe0580aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:11.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe04effd640 1 --2- 172.21.15.107:0/3626024422 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe05806ee40 0x7fe0580aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:11.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe04effd640 1 -- 172.21.15.107:0/3626024422 >> v1:172.21.15.110:6789/0 conn(0x7fe05814fd40 legacy=0x7fe0580c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:11.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe04effd640 1 -- 172.21.15.107:0/3626024422 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe058156710 con 0x7fe0580a9820 2023-10-25T00:11:11.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.689+0000 7fe04effd640 1 --2- 172.21.15.107:0/3626024422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0580a9820 0x7fe0580ab090 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fe040009870 tx=0x7fe04001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7b25b67fe60b97 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe056ffd640 1 -- 172.21.15.107:0/3626024422 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0400ad640 con 0x7fe0580a9820 2023-10-25T00:11:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe056ffd640 1 -- 172.21.15.107:0/3626024422 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0400ad7a0 con 0x7fe0580a9820 2023-10-25T00:11:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe056ffd640 1 -- 172.21.15.107:0/3626024422 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0400adb30 con 0x7fe0580a9820 2023-10-25T00:11:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 -- 172.21.15.107:0/3626024422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0580a9820 msgr2=0x7fe0580ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 --2- 172.21.15.107:0/3626024422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0580a9820 0x7fe0580ab090 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fe040009870 tx=0x7fe04001c3f0 comp rx=0 tx=0).stop 2023-10-25T00:11:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 -- 172.21.15.107:0/3626024422 shutdown_connections 2023-10-25T00:11:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 --2- 172.21.15.107:0/3626024422 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe05806ee40 0x7fe0580aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 --2- 172.21.15.107:0/3626024422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0580a9820 0x7fe0580ab090 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 -- 172.21.15.107:0/3626024422 >> 172.21.15.107:0/3626024422 conn(0x7fe0580c2950 msgr2=0x7fe0580c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 -- 172.21.15.107:0/3626024422 shutdown_connections 2023-10-25T00:11:11.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 -- 172.21.15.107:0/3626024422 wait complete. 2023-10-25T00:11:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 Processor -- start 2023-10-25T00:11:11.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 -- start start 2023-10-25T00:11:11.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05806ee40 0x7fe058144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:11.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe05814fd40 0x7fe05815b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:11.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe05814ef50 0x7fe0581ecb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:11.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe058069dd0 con 0x7fe05806ee40 2023-10-25T00:11:11.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe058145070 con 0x7fe05814fd40 2023-10-25T00:11:11.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0581f0a90 con 0x7fe05814ef50 2023-10-25T00:11:11.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05806ee40 0x7fe058144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:11.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe0577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05806ee40 0x7fe058144220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43002/0 (socket says 172.21.15.107:43002) 2023-10-25T00:11:11.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe0577fe640 1 -- 172.21.15.107:0/822769442 learned_addr learned my addr 172.21.15.107:0/822769442 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe04effd640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe05814fd40 0x7fe05815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe057fff640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe05814ef50 0x7fe0581ecb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe0577fe640 1 -- 172.21.15.107:0/822769442 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe05814ef50 msgr2=0x7fe0581ecb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe0577fe640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe05814ef50 0x7fe0581ecb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe0577fe640 1 -- 172.21.15.107:0/822769442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe05814fd40 msgr2=0x7fe05815b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:11.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe0577fe640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe05814fd40 0x7fe05815b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:11.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe0577fe640 1 -- 172.21.15.107:0/822769442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0400094d0 con 0x7fe05806ee40 2023-10-25T00:11:11.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe0577fe640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05806ee40 0x7fe058144220 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fe04800c3c0 tx=0x7fe04800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:11.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe054ff9640 1 -- 172.21.15.107:0/822769442 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0480052d0 con 0x7fe05806ee40 2023-10-25T00:11:11.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe054ff9640 1 -- 172.21.15.107:0/822769442 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe048005430 con 0x7fe05806ee40 2023-10-25T00:11:11.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 -- 172.21.15.107:0/822769442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0581ed620 con 0x7fe05806ee40 2023-10-25T00:11:11.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe054ff9640 1 -- 172.21.15.107:0/822769442 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe048003840 con 0x7fe05806ee40 2023-10-25T00:11:11.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.693+0000 7fe05df82640 1 -- 172.21.15.107:0/822769442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05814f3e0 con 0x7fe05806ee40 2023-10-25T00:11:11.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.697+0000 7fe05df82640 1 -- 172.21.15.107:0/822769442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0580c24a0 con 0x7fe05806ee40 2023-10-25T00:11:11.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.697+0000 7fe054ff9640 1 -- 172.21.15.107:0/822769442 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe0480039a0 con 0x7fe05806ee40 2023-10-25T00:11:11.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.697+0000 7fe054ff9640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe038065e60 0x7fe038068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:11.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.697+0000 7fe054ff9640 1 -- 172.21.15.107:0/822769442 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7fe048083b60 con 0x7fe05806ee40 2023-10-25T00:11:11.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.697+0000 7fe04effd640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe038065e60 0x7fe038068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:11.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.697+0000 7fe04effd640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe038065e60 0x7fe038068320 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe0400af000 tx=0x7fe040009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:11.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.701+0000 7fe054ff9640 1 -- 172.21.15.107:0/822769442 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe048050420 con 0x7fe05806ee40 2023-10-25T00:11:11.782 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:11:11.781+0000 7f5cf59b0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:11:11.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.893+0000 7fe05df82640 1 -- 172.21.15.107:0/822769442 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe058070840 con 0x7fe05806ee40 2023-10-25T00:11:11.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.893+0000 7fe054ff9640 1 -- 172.21.15.107:0/822769442 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5019 (secure 0 0 0) 0x7fe048053a40 con 0x7fe05806ee40 2023-10-25T00:11:11.896 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:11.897 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 159 2023-10-25T00:11:11.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 -- 172.21.15.107:0/822769442 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe038065e60 msgr2=0x7fe038068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:11.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe038065e60 0x7fe038068320 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe0400af000 tx=0x7fe040009040 comp rx=0 tx=0).stop 2023-10-25T00:11:11.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 -- 172.21.15.107:0/822769442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05806ee40 msgr2=0x7fe058144220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:11.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05806ee40 0x7fe058144220 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fe04800c3c0 tx=0x7fe04800c880 comp rx=0 tx=0).stop 2023-10-25T00:11:11.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 -- 172.21.15.107:0/822769442 shutdown_connections 2023-10-25T00:11:11.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe038065e60 0x7fe038068320 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:11.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe05814ef50 0x7fe0581ecb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:11.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe05814fd40 0x7fe05815b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:11.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 --2- 172.21.15.107:0/822769442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe05806ee40 0x7fe058144220 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:11.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 -- 172.21.15.107:0/822769442 >> 172.21.15.107:0/822769442 conn(0x7fe0580c2950 msgr2=0x7fe0580c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:11.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 -- 172.21.15.107:0/822769442 shutdown_connections 2023-10-25T00:11:11.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:11.897+0000 7fe05df82640 1 -- 172.21.15.107:0/822769442 wait complete. 2023-10-25T00:11:11.915 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:11:12.313 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2023-10-25T00:11:12.313 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-25T00:11:12.314 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-25T00:11:12.316 INFO:tasks.ceph.mds.d:Started 2023-10-25T00:11:12.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:12.398 INFO:tasks.ceph.mds.d.smithi110.stdout:starting mds.d at 2023-10-25T00:11:12.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84c4a6640 1 Processor -- start 2023-10-25T00:11:12.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84c4a6640 1 -- start start 2023-10-25T00:11:12.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84c4a6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff8440c24c0 0x7ff84406a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:12.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84c4a6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff844150bc0 con 0x7ff844071220 2023-10-25T00:11:12.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84c4a6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84406c510 con 0x7ff844151960 2023-10-25T00:11:12.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84c4a6640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84406c7f0 con 0x7ff8440c24c0 2023-10-25T00:11:12.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84a242640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff844071220 0x7ff84406bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49606/0 (socket says 172.21.15.107:49606) 2023-10-25T00:11:12.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84a242640 1 -- 172.21.15.107:0/3922470038 learned_addr learned my addr 172.21.15.107:0/3922470038 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:11:12.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84aa43640 1 --2- 172.21.15.107:0/3922470038 >> v2:172.21.15.110:3300/0 conn(0x7ff8440c24c0 0x7ff84406a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:12.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff849240640 1 -- 172.21.15.107:0/3922470038 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369058511 0 0) 0x7ff844150bc0 con 0x7ff844071220 2023-10-25T00:11:12.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff849240640 1 -- 172.21.15.107:0/3922470038 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff830003530 con 0x7ff844071220 2023-10-25T00:11:12.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff849240640 1 -- 172.21.15.107:0/3922470038 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186770457 0 0) 0x7ff84406c510 con 0x7ff844151960 2023-10-25T00:11:12.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff849240640 1 -- 172.21.15.107:0/3922470038 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff844150bc0 con 0x7ff844151960 2023-10-25T00:11:12.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84aa43640 1 -- 172.21.15.107:0/3922470038 >> v1:172.21.15.110:6789/0 conn(0x7ff844151960 legacy=0x7ff844071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:12.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84aa43640 1 -- 172.21.15.107:0/3922470038 >> v1:172.21.15.107:6789/0 conn(0x7ff844071220 legacy=0x7ff84406bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:12.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.445+0000 7ff84aa43640 1 -- 172.21.15.107:0/3922470038 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff844150e80 con 0x7ff8440c24c0 2023-10-25T00:11:12.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84aa43640 1 --2- 172.21.15.107:0/3922470038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8440c24c0 0x7ff84406a9c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00a960 tx=0x7ff83c01d7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f5b176d2d522ff8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:12.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff849240640 1 -- 172.21.15.107:0/3922470038 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c0a7620 con 0x7ff8440c24c0 2023-10-25T00:11:12.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff849240640 1 -- 172.21.15.107:0/3922470038 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83c0a7780 con 0x7ff8440c24c0 2023-10-25T00:11:12.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff849240640 1 -- 172.21.15.107:0/3922470038 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c0a7b10 con 0x7ff8440c24c0 2023-10-25T00:11:12.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 -- 172.21.15.107:0/3922470038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8440c24c0 msgr2=0x7ff84406a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:12.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 --2- 172.21.15.107:0/3922470038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8440c24c0 0x7ff84406a9c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00a960 tx=0x7ff83c01d7c0 comp rx=0 tx=0).stop 2023-10-25T00:11:12.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 -- 172.21.15.107:0/3922470038 shutdown_connections 2023-10-25T00:11:12.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 --2- 172.21.15.107:0/3922470038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8440c24c0 0x7ff84406a9c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:12.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 -- 172.21.15.107:0/3922470038 >> 172.21.15.107:0/3922470038 conn(0x7ff8440c2950 msgr2=0x7ff8440c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:12.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 -- 172.21.15.107:0/3922470038 shutdown_connections 2023-10-25T00:11:12.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 -- 172.21.15.107:0/3922470038 wait complete. 2023-10-25T00:11:12.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 Processor -- start 2023-10-25T00:11:12.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 -- start start 2023-10-25T00:11:12.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff844071220 0x7ff84414d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:12.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8440c24c0 0x7ff844145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:12.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff844151960 0x7ff8441462b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:12.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84414e2c0 con 0x7ff844071220 2023-10-25T00:11:12.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8441eddb0 con 0x7ff8440c24c0 2023-10-25T00:11:12.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8441476e0 con 0x7ff844151960 2023-10-25T00:11:12.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84a242640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff844071220 0x7ff84414d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:12.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84a242640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff844071220 0x7ff84414d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43010/0 (socket says 172.21.15.107:43010) 2023-10-25T00:11:12.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84a242640 1 -- 172.21.15.107:0/1709591036 learned_addr learned my addr 172.21.15.107:0/1709591036 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:12.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff849a41640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8440c24c0 0x7ff844145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:12.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84aa43640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff844151960 0x7ff8441462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:12.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84a242640 1 -- 172.21.15.107:0/1709591036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff844151960 msgr2=0x7ff8441462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:12.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84a242640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff844151960 0x7ff8441462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:12.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84a242640 1 -- 172.21.15.107:0/1709591036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8440c24c0 msgr2=0x7ff844145d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:12.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84a242640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8440c24c0 0x7ff844145d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:12.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84a242640 1 -- 172.21.15.107:0/1709591036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c00a5c0 con 0x7ff844071220 2023-10-25T00:11:12.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84a242640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff844071220 0x7ff84414d5e0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7ff84000cbf0 tx=0x7ff84000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:12.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff83affd640 1 -- 172.21.15.107:0/1709591036 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84000ace0 con 0x7ff844071220 2023-10-25T00:11:12.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff83affd640 1 -- 172.21.15.107:0/1709591036 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84000ae40 con 0x7ff844071220 2023-10-25T00:11:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 -- 172.21.15.107:0/1709591036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84414e510 con 0x7ff844071220 2023-10-25T00:11:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff83affd640 1 -- 172.21.15.107:0/1709591036 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840012600 con 0x7ff844071220 2023-10-25T00:11:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.449+0000 7ff84c4a6640 1 -- 172.21.15.107:0/1709591036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84414db20 con 0x7ff844071220 2023-10-25T00:11:12.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.453+0000 7ff84c4a6640 1 -- 172.21.15.107:0/1709591036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8440746e0 con 0x7ff844071220 2023-10-25T00:11:12.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.453+0000 7ff83affd640 1 -- 172.21.15.107:0/1709591036 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff840007070 con 0x7ff844071220 2023-10-25T00:11:12.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.453+0000 7ff83affd640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff828065e90 0x7ff828068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:12.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.453+0000 7ff83affd640 1 -- 172.21.15.107:0/1709591036 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7ff8400848d0 con 0x7ff844071220 2023-10-25T00:11:12.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.453+0000 7ff849a41640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff828065e90 0x7ff828068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:12.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.453+0000 7ff849a41640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff828065e90 0x7ff828068350 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff8441e85f0 tx=0x7ff834009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:12.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.457+0000 7ff83affd640 1 -- 172.21.15.107:0/1709591036 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff8400512f0 con 0x7ff844071220 2023-10-25T00:11:12.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.645+0000 7ff84c4a6640 1 -- 172.21.15.107:0/1709591036 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8440c22a0 con 0x7ff844071220 2023-10-25T00:11:12.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff83affd640 1 -- 172.21.15.107:0/1709591036 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5019 (secure 0 0 0) 0x7ff840021020 con 0x7ff844071220 2023-10-25T00:11:12.649 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:12.649 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 159 2023-10-25T00:11:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 -- 172.21.15.107:0/1709591036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff828065e90 msgr2=0x7ff828068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff828065e90 0x7ff828068350 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff8441e85f0 tx=0x7ff834009040 comp rx=0 tx=0).stop 2023-10-25T00:11:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 -- 172.21.15.107:0/1709591036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff844071220 msgr2=0x7ff84414d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff844071220 0x7ff84414d5e0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7ff84000cbf0 tx=0x7ff84000a430 comp rx=0 tx=0).stop 2023-10-25T00:11:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 -- 172.21.15.107:0/1709591036 shutdown_connections 2023-10-25T00:11:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff828065e90 0x7ff828068350 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff844151960 0x7ff8441462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8440c24c0 0x7ff844145d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 --2- 172.21.15.107:0/1709591036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff844071220 0x7ff84414d5e0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:12.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 -- 172.21.15.107:0/1709591036 >> 172.21.15.107:0/1709591036 conn(0x7ff8440c2950 msgr2=0x7ff84406bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 -- 172.21.15.107:0/1709591036 shutdown_connections 2023-10-25T00:11:12.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:12.649+0000 7ff84c4a6640 1 -- 172.21.15.107:0/1709591036 wait complete. 2023-10-25T00:11:12.668 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-25T00:11:13.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:14.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7f977e640 1 Processor -- start 2023-10-25T00:11:14.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7f977e640 1 -- start start 2023-10-25T00:11:14.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7f977e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7f40a9820 0x7fb7f40ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7f977e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7f406ee40 0x7fb7f40aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7f977e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f406f730 con 0x7fb7f414fd40 2023-10-25T00:11:14.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7f977e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f40aa750 con 0x7fb7f40a9820 2023-10-25T00:11:14.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7f977e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f4156a40 con 0x7fb7f406ee40 2023-10-25T00:11:14.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7ebfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7f40a9820 0x7fb7f40ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7ebfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb7f40a9820 0x7fb7f40ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45754/0 (socket says 172.21.15.107:45754) 2023-10-25T00:11:14.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7ebfff640 1 -- 172.21.15.107:0/3666542531 learned_addr learned my addr 172.21.15.107:0/3666542531 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:14.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7f37fe640 1 --2- 172.21.15.107:0/3666542531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7f406ee40 0x7fb7f40aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7ebfff640 1 -- 172.21.15.107:0/3666542531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7f406ee40 msgr2=0x7fb7f40aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:14.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7ebfff640 1 --2- 172.21.15.107:0/3666542531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7f406ee40 0x7fb7f40aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7ebfff640 1 -- 172.21.15.107:0/3666542531 >> v1:172.21.15.110:6789/0 conn(0x7fb7f414fd40 legacy=0x7fb7f40c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:14.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7ebfff640 1 -- 172.21.15.107:0/3666542531 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f4156710 con 0x7fb7f40a9820 2023-10-25T00:11:14.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.045+0000 7fb7f37fe640 1 --2- 172.21.15.107:0/3666542531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7f406ee40 0x7fb7f40aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:11:14.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7ebfff640 1 --2- 172.21.15.107:0/3666542531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f40a9820 0x7fb7f40ab090 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009870 tx=0x7fb7dc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d349e9eb661489cb server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:14.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f27fc640 1 -- 172.21.15.107:0/3666542531 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0ad640 con 0x7fb7f40a9820 2023-10-25T00:11:14.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f27fc640 1 -- 172.21.15.107:0/3666542531 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc0ad7a0 con 0x7fb7f40a9820 2023-10-25T00:11:14.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f27fc640 1 -- 172.21.15.107:0/3666542531 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0adb30 con 0x7fb7f40a9820 2023-10-25T00:11:14.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 -- 172.21.15.107:0/3666542531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f40a9820 msgr2=0x7fb7f40ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:14.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 --2- 172.21.15.107:0/3666542531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f40a9820 0x7fb7f40ab090 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009870 tx=0x7fb7dc01c3f0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 -- 172.21.15.107:0/3666542531 shutdown_connections 2023-10-25T00:11:14.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 --2- 172.21.15.107:0/3666542531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7f406ee40 0x7fb7f40aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 --2- 172.21.15.107:0/3666542531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f40a9820 0x7fb7f40ab090 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 -- 172.21.15.107:0/3666542531 >> 172.21.15.107:0/3666542531 conn(0x7fb7f40c2950 msgr2=0x7fb7f40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:14.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 -- 172.21.15.107:0/3666542531 shutdown_connections 2023-10-25T00:11:14.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 -- 172.21.15.107:0/3666542531 wait complete. 2023-10-25T00:11:14.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 Processor -- start 2023-10-25T00:11:14.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 -- start start 2023-10-25T00:11:14.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f406ee40 0x7fb7f4159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7f414fd40 0x7fb7f414e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7f40bb270 0x7fb7f41eeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f41eea20 con 0x7fb7f406ee40 2023-10-25T00:11:14.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f40bfce0 con 0x7fb7f414fd40 2023-10-25T00:11:14.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f40bffc0 con 0x7fb7f40bb270 2023-10-25T00:11:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f406ee40 0x7fb7f4159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f406ee40 0x7fb7f4159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45760/0 (socket says 172.21.15.107:45760) 2023-10-25T00:11:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f2ffd640 1 -- 172.21.15.107:0/2456652093 learned_addr learned my addr 172.21.15.107:0/2456652093 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7ebfff640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7f414fd40 0x7fb7f414e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f37fe640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7f40bb270 0x7fb7f41eeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f2ffd640 1 -- 172.21.15.107:0/2456652093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7f40bb270 msgr2=0x7fb7f41eeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:14.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f2ffd640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7f40bb270 0x7fb7f41eeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f2ffd640 1 -- 172.21.15.107:0/2456652093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7f414fd40 msgr2=0x7fb7f414e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:14.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f2ffd640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7f414fd40 0x7fb7f414e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f2ffd640 1 -- 172.21.15.107:0/2456652093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc0094d0 con 0x7fb7f406ee40 2023-10-25T00:11:14.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7ebfff640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7f414fd40 0x7fb7f414e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:11:14.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f2ffd640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f406ee40 0x7fb7f4159790 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400c3c0 tx=0x7fb7e400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:14.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7eb7fe640 1 -- 172.21.15.107:0/2456652093 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e40052d0 con 0x7fb7f406ee40 2023-10-25T00:11:14.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7eb7fe640 1 -- 172.21.15.107:0/2456652093 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e4005430 con 0x7fb7f406ee40 2023-10-25T00:11:14.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7f977e640 1 -- 172.21.15.107:0/2456652093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f40c02a0 con 0x7fb7f406ee40 2023-10-25T00:11:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.049+0000 7fb7eb7fe640 1 -- 172.21.15.107:0/2456652093 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4003840 con 0x7fb7f406ee40 2023-10-25T00:11:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.053+0000 7fb7f977e640 1 -- 172.21.15.107:0/2456652093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f414e6f0 con 0x7fb7f406ee40 2023-10-25T00:11:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.053+0000 7fb7f977e640 1 -- 172.21.15.107:0/2456652093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f40c24a0 con 0x7fb7f406ee40 2023-10-25T00:11:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.053+0000 7fb7eb7fe640 1 -- 172.21.15.107:0/2456652093 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb7e40039a0 con 0x7fb7f406ee40 2023-10-25T00:11:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.053+0000 7fb7eb7fe640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb7d4065e90 0x7fb7d4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.053+0000 7fb7eb7fe640 1 -- 172.21.15.107:0/2456652093 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7fb7e4083bb0 con 0x7fb7f406ee40 2023-10-25T00:11:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.053+0000 7fb7ebfff640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb7d4065e90 0x7fb7d4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.053+0000 7fb7ebfff640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb7d4065e90 0x7fb7d4068350 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0af000 tx=0x7fb7dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:14.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.057+0000 7fb7eb7fe640 1 -- 172.21.15.107:0/2456652093 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb7e40505d0 con 0x7fb7f406ee40 2023-10-25T00:11:14.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.245+0000 7fb7f977e640 1 -- 172.21.15.107:0/2456652093 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7f4070840 con 0x7fb7f406ee40 2023-10-25T00:11:14.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.245+0000 7fb7eb7fe640 1 -- 172.21.15.107:0/2456652093 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+5777 (secure 0 0 0) 0x7fb7e4017090 con 0x7fb7f406ee40 2023-10-25T00:11:14.248 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:14.248 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:14.248 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 160 2023-10-25T00:11:14.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 -- 172.21.15.107:0/2456652093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb7d4065e90 msgr2=0x7fb7d4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:14.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb7d4065e90 0x7fb7d4068350 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0af000 tx=0x7fb7dc008040 comp rx=0 tx=0).stop 2023-10-25T00:11:14.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 -- 172.21.15.107:0/2456652093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f406ee40 msgr2=0x7fb7f4159790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:14.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f406ee40 0x7fb7f4159790 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400c3c0 tx=0x7fb7e400c880 comp rx=0 tx=0).stop 2023-10-25T00:11:14.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 -- 172.21.15.107:0/2456652093 shutdown_connections 2023-10-25T00:11:14.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb7d4065e90 0x7fb7d4068350 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb7f40bb270 0x7fb7f41eeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7f414fd40 0x7fb7f414e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 --2- 172.21.15.107:0/2456652093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7f406ee40 0x7fb7f4159790 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 -- 172.21.15.107:0/2456652093 >> 172.21.15.107:0/2456652093 conn(0x7fb7f40c2950 msgr2=0x7fb7f40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:14.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 -- 172.21.15.107:0/2456652093 shutdown_connections 2023-10-25T00:11:14.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.249+0000 7fb7f977e640 1 -- 172.21.15.107:0/2456652093 wait complete. 2023-10-25T00:11:14.265 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:11:14.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:14.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.797+0000 7f847fa48640 1 Processor -- start 2023-10-25T00:11:14.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.797+0000 7f847fa48640 1 -- start start 2023-10-25T00:11:14.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f847814b600 0x7f8478151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f847806af70 0x7f847806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f847806a230 0x7f84781551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84780c1860 con 0x7f847806a230 2023-10-25T00:11:14.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8478154d50 con 0x7f847806af70 2023-10-25T00:11:14.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84781524f0 con 0x7f847814b600 2023-10-25T00:11:14.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847dfe5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f847806a230 0x7f84781551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847dfe5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f847806a230 0x7f84781551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45770/0 (socket says 172.21.15.107:45770) 2023-10-25T00:11:14.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847dfe5640 1 -- 172.21.15.107:0/3316808525 learned_addr learned my addr 172.21.15.107:0/3316808525 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:14.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847d7e4640 1 --2- 172.21.15.107:0/3316808525 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f847814b600 0x7f8478151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847cfe3640 1 --2- 172.21.15.107:0/3316808525 >> v2:172.21.15.110:3300/0 conn(0x7f847806af70 0x7f847806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847dfe5640 1 -- 172.21.15.107:0/3316808525 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f847814b600 msgr2=0x7f8478151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:14.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847dfe5640 1 --2- 172.21.15.107:0/3316808525 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f847814b600 0x7f8478151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847dfe5640 1 -- 172.21.15.107:0/3316808525 >> v2:172.21.15.110:3300/0 conn(0x7f847806af70 msgr2=0x7f847806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:14.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847dfe5640 1 --2- 172.21.15.107:0/3316808525 >> v2:172.21.15.110:3300/0 conn(0x7f847806af70 0x7f847806c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847dfe5640 1 -- 172.21.15.107:0/3316808525 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847814b9e0 con 0x7f847806a230 2023-10-25T00:11:14.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847cfe3640 1 --2- 172.21.15.107:0/3316808525 >> v2:172.21.15.110:3300/0 conn(0x7f847806af70 0x7f847806c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:11:14.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847dfe5640 1 --2- 172.21.15.107:0/3316808525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f847806a230 0x7f84781551e0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f847806d1c0 tx=0x7f847001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1cd8e817593171 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:14.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f846ffff640 1 -- 172.21.15.107:0/3316808525 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84700ab600 con 0x7f847806a230 2023-10-25T00:11:14.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f846ffff640 1 -- 172.21.15.107:0/3316808525 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84700a7050 con 0x7f847806a230 2023-10-25T00:11:14.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f846ffff640 1 -- 172.21.15.107:0/3316808525 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84700ab980 con 0x7f847806a230 2023-10-25T00:11:14.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 -- 172.21.15.107:0/3316808525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f847806a230 msgr2=0x7f84781551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:14.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 --2- 172.21.15.107:0/3316808525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f847806a230 0x7f84781551e0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f847806d1c0 tx=0x7f847001c440 comp rx=0 tx=0).stop 2023-10-25T00:11:14.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 -- 172.21.15.107:0/3316808525 shutdown_connections 2023-10-25T00:11:14.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 --2- 172.21.15.107:0/3316808525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f847806a230 0x7f84781551e0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 --2- 172.21.15.107:0/3316808525 >> v2:172.21.15.110:3300/0 conn(0x7f847806af70 0x7f847806c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 --2- 172.21.15.107:0/3316808525 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f847814b600 0x7f8478151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 -- 172.21.15.107:0/3316808525 >> 172.21.15.107:0/3316808525 conn(0x7f84780c2950 msgr2=0x7f84780c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:14.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 -- 172.21.15.107:0/3316808525 shutdown_connections 2023-10-25T00:11:14.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.801+0000 7f847fa48640 1 -- 172.21.15.107:0/3316808525 wait complete. 2023-10-25T00:11:14.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847fa48640 1 Processor -- start 2023-10-25T00:11:14.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847fa48640 1 -- start start 2023-10-25T00:11:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847fa48640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f847806a230 0x7f847814a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847fa48640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f847806af70 0x7f84781ef070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847fa48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f847814b600 0x7f84781ef5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847fa48640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781efaf0 con 0x7f847814b600 2023-10-25T00:11:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847fa48640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781548a0 con 0x7f847806a230 2023-10-25T00:11:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847fa48640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8478071e90 con 0x7f847806af70 2023-10-25T00:11:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847dfe5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f847814b600 0x7f84781ef5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847dfe5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f847814b600 0x7f84781ef5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45784/0 (socket says 172.21.15.107:45784) 2023-10-25T00:11:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847dfe5640 1 -- 172.21.15.107:0/2491104981 learned_addr learned my addr 172.21.15.107:0/2491104981 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847cfe3640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f847806af70 0x7f84781ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847d7e4640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f847806a230 0x7f847814a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847dfe5640 1 -- 172.21.15.107:0/2491104981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f847806af70 msgr2=0x7f84781ef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847dfe5640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f847806af70 0x7f84781ef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847dfe5640 1 -- 172.21.15.107:0/2491104981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f847806a230 msgr2=0x7f847814a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:14.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847dfe5640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f847806a230 0x7f847814a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:14.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847dfe5640 1 -- 172.21.15.107:0/2491104981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84700094d0 con 0x7f847814b600 2023-10-25T00:11:14.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.805+0000 7f847dfe5640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f847814b600 0x7f84781ef5b0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f84700ae040 tx=0x7f84700191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:14.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.809+0000 7f846dffb640 1 -- 172.21.15.107:0/2491104981 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84700b4620 con 0x7f847814b600 2023-10-25T00:11:14.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.809+0000 7f846dffb640 1 -- 172.21.15.107:0/2491104981 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84700b4780 con 0x7f847814b600 2023-10-25T00:11:14.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.809+0000 7f846dffb640 1 -- 172.21.15.107:0/2491104981 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84700b4af0 con 0x7f847814b600 2023-10-25T00:11:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.809+0000 7f847fa48640 1 -- 172.21.15.107:0/2491104981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84781eeb70 con 0x7f847814b600 2023-10-25T00:11:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.809+0000 7f847fa48640 1 -- 172.21.15.107:0/2491104981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8478070bb0 con 0x7f847814b600 2023-10-25T00:11:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.809+0000 7f847fa48640 1 -- 172.21.15.107:0/2491104981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84781eed60 con 0x7f847814b600 2023-10-25T00:11:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.809+0000 7f846dffb640 1 -- 172.21.15.107:0/2491104981 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f84700b4c50 con 0x7f847814b600 2023-10-25T00:11:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.809+0000 7f846dffb640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f845c065e60 0x7f845c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:14.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.809+0000 7f847d7e4640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f845c065e60 0x7f845c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:14.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.809+0000 7f846dffb640 1 -- 172.21.15.107:0/2491104981 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7f84701222b0 con 0x7f847814b600 2023-10-25T00:11:14.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.809+0000 7f847d7e4640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f845c065e60 0x7f845c068320 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8474003fd0 tx=0x7f84740039a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:14.813+0000 7f846dffb640 1 -- 172.21.15.107:0/2491104981 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f84700eeb40 con 0x7f847814b600 2023-10-25T00:11:15.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.001+0000 7f847fa48640 1 -- 172.21.15.107:0/2491104981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84780746e0 con 0x7f847814b600 2023-10-25T00:11:15.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.001+0000 7f846dffb640 1 -- 172.21.15.107:0/2491104981 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f84700b0020 con 0x7f847814b600 2023-10-25T00:11:15.004 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:15.004 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:15.005 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 160 2023-10-25T00:11:15.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 -- 172.21.15.107:0/2491104981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f845c065e60 msgr2=0x7f845c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:15.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f845c065e60 0x7f845c068320 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8474003fd0 tx=0x7f84740039a0 comp rx=0 tx=0).stop 2023-10-25T00:11:15.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 -- 172.21.15.107:0/2491104981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f847814b600 msgr2=0x7f84781ef5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f847814b600 0x7f84781ef5b0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f84700ae040 tx=0x7f84700191e0 comp rx=0 tx=0).stop 2023-10-25T00:11:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 -- 172.21.15.107:0/2491104981 shutdown_connections 2023-10-25T00:11:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f847814b600 0x7f84781ef5b0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f845c065e60 0x7f845c068320 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:15.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f847806af70 0x7f84781ef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:15.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 --2- 172.21.15.107:0/2491104981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f847806a230 0x7f847814a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:15.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 -- 172.21.15.107:0/2491104981 >> 172.21.15.107:0/2491104981 conn(0x7f84780c2950 msgr2=0x7f8478144e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:15.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 -- 172.21.15.107:0/2491104981 shutdown_connections 2023-10-25T00:11:15.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:15.005+0000 7f847fa48640 1 -- 172.21.15.107:0/2491104981 wait complete. 2023-10-25T00:11:15.023 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2023-10-25T00:11:15.023 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2023-10-25T00:11:16.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.393+0000 7f415ba5a640 1 Processor -- start 2023-10-25T00:11:16.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.393+0000 7f415ba5a640 1 -- start start 2023-10-25T00:11:16.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f415ba5a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f415414b600 0x7f4154151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:16.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f415ba5a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41540c1860 con 0x7f415406af70 2023-10-25T00:11:16.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f415ba5a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4154154d50 con 0x7f415406a230 2023-10-25T00:11:16.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f415ba5a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41541524f0 con 0x7f415414b600 2023-10-25T00:11:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f4158ff5640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f415406af70 0x7f415406c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38460/0 (socket says 172.21.15.107:38460) 2023-10-25T00:11:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f4158ff5640 1 -- 172.21.15.107:0/2544371330 learned_addr learned my addr 172.21.15.107:0/2544371330 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:11:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f41597f6640 1 --2- 172.21.15.107:0/2544371330 >> v2:172.21.15.110:3300/0 conn(0x7f415414b600 0x7f4154151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054659373 0 0) 0x7f41540c1860 con 0x7f415406af70 2023-10-25T00:11:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4140005740 con 0x7f415406af70 2023-10-25T00:11:16.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1411947039 0 0) 0x7f4140005740 con 0x7f415406af70 2023-10-25T00:11:16.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 >> v2:172.21.15.110:3300/0 conn(0x7f415414b600 msgr2=0x7f4154151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:16.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 --2- 172.21.15.107:0/2544371330 >> v2:172.21.15.110:3300/0 conn(0x7f415414b600 0x7f4154151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:16.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 >> v1:172.21.15.110:6789/0 conn(0x7f415406a230 legacy=0x7f41541551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:16.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415414b9e0 con 0x7f415406af70 2023-10-25T00:11:16.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f4144001680 con 0x7f415406af70 2023-10-25T00:11:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 >> v1:172.21.15.107:6789/0 conn(0x7f415406af70 legacy=0x7f415406c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 --2- 172.21.15.107:0/2544371330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4140005b90 0x7f4140005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 --2- 172.21.15.107:0/2544371330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41400069e0 0x7f4140006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 --2- 172.21.15.107:0/2544371330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41400073a0 0x7f414000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:16.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41541524f0 con 0x7f41400073a0 2023-10-25T00:11:16.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414000e340 con 0x7f4140005b90 2023-10-25T00:11:16.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414000e500 con 0x7f41400069e0 2023-10-25T00:11:16.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f4159ff7640 1 --2- 172.21.15.107:0/2544371330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41400073a0 0x7f414000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:16.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f4158ff5640 1 --2- 172.21.15.107:0/2544371330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41400069e0 0x7f4140006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:16.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f4159ff7640 1 -- 172.21.15.107:0/2544371330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41400069e0 msgr2=0x7f4140006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:16.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f4159ff7640 1 --2- 172.21.15.107:0/2544371330 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41400069e0 0x7f4140006e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:16.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f4159ff7640 1 -- 172.21.15.107:0/2544371330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4140005b90 msgr2=0x7f4140005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:16.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f4159ff7640 1 --2- 172.21.15.107:0/2544371330 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4140005b90 0x7f4140005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:16.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.397+0000 7f4159ff7640 1 -- 172.21.15.107:0/2544371330 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4140005740 con 0x7f41400073a0 2023-10-25T00:11:16.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4159ff7640 1 -- 172.21.15.107:0/2544371330 reap_dead start 2023-10-25T00:11:16.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4159ff7640 1 --2- 172.21.15.107:0/2544371330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41400073a0 0x7f414000dba0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f414c001e40 tx=0x7f414c01c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5853608232b844a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:16.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c003b80 con 0x7f41400073a0 2023-10-25T00:11:16.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414c003ce0 con 0x7f41400073a0 2023-10-25T00:11:16.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f414bfff640 1 -- 172.21.15.107:0/2544371330 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c005260 con 0x7f41400073a0 2023-10-25T00:11:16.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 -- 172.21.15.107:0/2544371330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41400073a0 msgr2=0x7f414000dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:16.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 --2- 172.21.15.107:0/2544371330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41400073a0 0x7f414000dba0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f414c001e40 tx=0x7f414c01c740 comp rx=0 tx=0).stop 2023-10-25T00:11:16.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 -- 172.21.15.107:0/2544371330 shutdown_connections 2023-10-25T00:11:16.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 --2- 172.21.15.107:0/2544371330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41400073a0 0x7f414000dba0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:16.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 -- 172.21.15.107:0/2544371330 >> 172.21.15.107:0/2544371330 conn(0x7f41540c2950 msgr2=0x7f41540c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:16.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 -- 172.21.15.107:0/2544371330 shutdown_connections 2023-10-25T00:11:16.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 -- 172.21.15.107:0/2544371330 wait complete. 2023-10-25T00:11:16.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 Processor -- start 2023-10-25T00:11:16.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 -- start start 2023-10-25T00:11:16.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41400073a0 0x7f4154154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f415414f0c0 0x7f4154143c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41541446b0 0x7f41541e4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41541500d0 con 0x7f415414f0c0 2023-10-25T00:11:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41540c5750 con 0x7f41541446b0 2023-10-25T00:11:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f415ba5a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41540c5a30 con 0x7f41400073a0 2023-10-25T00:11:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4158ff5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f415414f0c0 0x7f4154143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4158ff5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f415414f0c0 0x7f4154143c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45812/0 (socket says 172.21.15.107:45812) 2023-10-25T00:11:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4158ff5640 1 -- 172.21.15.107:0/2217066393 learned_addr learned my addr 172.21.15.107:0/2217066393 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f41597f6640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41400073a0 0x7f4154154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4159ff7640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41541446b0 0x7f41541e4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4158ff5640 1 -- 172.21.15.107:0/2217066393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41400073a0 msgr2=0x7f4154154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4158ff5640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41400073a0 0x7f4154154260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4158ff5640 1 -- 172.21.15.107:0/2217066393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41541446b0 msgr2=0x7f41541e4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:16.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4158ff5640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41541446b0 0x7f41541e4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:16.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4158ff5640 1 -- 172.21.15.107:0/2217066393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4154150760 con 0x7f415414f0c0 2023-10-25T00:11:16.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f41597f6640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41400073a0 0x7f4154154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:11:16.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.401+0000 7f4159ff7640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41541446b0 0x7f41541e4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:11:16.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f4158ff5640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f415414f0c0 0x7f4154143c40 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f41440018f0 tx=0x7f414400e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:16.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f4149ffb640 1 -- 172.21.15.107:0/2217066393 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414400ed50 con 0x7f415414f0c0 2023-10-25T00:11:16.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f4149ffb640 1 -- 172.21.15.107:0/2217066393 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41440143a0 con 0x7f415414f0c0 2023-10-25T00:11:16.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f415ba5a640 1 -- 172.21.15.107:0/2217066393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4154150a40 con 0x7f415414f0c0 2023-10-25T00:11:16.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f4149ffb640 1 -- 172.21.15.107:0/2217066393 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4144014710 con 0x7f415414f0c0 2023-10-25T00:11:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f415ba5a640 1 -- 172.21.15.107:0/2217066393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41541547a0 con 0x7f415414f0c0 2023-10-25T00:11:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f415ba5a640 1 -- 172.21.15.107:0/2217066393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41540746e0 con 0x7f415414f0c0 2023-10-25T00:11:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f4149ffb640 1 -- 172.21.15.107:0/2217066393 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4144014870 con 0x7f415414f0c0 2023-10-25T00:11:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f4149ffb640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4138065e60 0x7f4138068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f4149ffb640 1 -- 172.21.15.107:0/2217066393 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7f4144087a00 con 0x7f415414f0c0 2023-10-25T00:11:16.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f41597f6640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4138065e60 0x7f4138068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:16.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.405+0000 7f41597f6640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4138065e60 0x7f4138068320 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f41500047c0 tx=0x7f4150008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:16.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.409+0000 7f4149ffb640 1 -- 172.21.15.107:0/2217066393 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4144054310 con 0x7f415414f0c0 2023-10-25T00:11:16.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.593+0000 7f415ba5a640 1 -- 172.21.15.107:0/2217066393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4154069fa0 con 0x7f415414f0c0 2023-10-25T00:11:16.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.593+0000 7f4149ffb640 1 -- 172.21.15.107:0/2217066393 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+5777 (secure 0 0 0) 0x7f414401b020 con 0x7f415414f0c0 2023-10-25T00:11:16.595 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:16.595 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":160}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:10:41.848063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7746},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7746":{"gid":7746,"name":"c","rank":0,"incarnation":151,"state":"up:active","state_seq":22,"addr":"172.21.15.107:6835/3474118649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3474118649},{"type":"v1","addr":"172.21.15.107:6835","nonce":3474118649}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:16.595 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 160 2023-10-25T00:11:16.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.593+0000 7f415ba5a640 1 -- 172.21.15.107:0/2217066393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4138065e60 msgr2=0x7f4138068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:16.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.593+0000 7f415ba5a640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4138065e60 0x7f4138068320 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f41500047c0 tx=0x7f4150008040 comp rx=0 tx=0).stop 2023-10-25T00:11:16.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.593+0000 7f415ba5a640 1 -- 172.21.15.107:0/2217066393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f415414f0c0 msgr2=0x7f4154143c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:16.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.597+0000 7f415ba5a640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f415414f0c0 0x7f4154143c40 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f41440018f0 tx=0x7f414400e4d0 comp rx=0 tx=0).stop 2023-10-25T00:11:16.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.597+0000 7f415ba5a640 1 -- 172.21.15.107:0/2217066393 shutdown_connections 2023-10-25T00:11:16.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.597+0000 7f415ba5a640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41541446b0 0x7f41541e4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:16.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.597+0000 7f415ba5a640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f415414f0c0 0x7f4154143c40 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:16.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.597+0000 7f415ba5a640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4138065e60 0x7f4138068320 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:16.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.597+0000 7f415ba5a640 1 --2- 172.21.15.107:0/2217066393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f41400073a0 0x7f4154154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:16.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.597+0000 7f415ba5a640 1 -- 172.21.15.107:0/2217066393 >> 172.21.15.107:0/2217066393 conn(0x7f41540c2950 msgr2=0x7f4154147f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:16.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.597+0000 7f415ba5a640 1 -- 172.21.15.107:0/2217066393 shutdown_connections 2023-10-25T00:11:16.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:16.597+0000 7f415ba5a640 1 -- 172.21.15.107:0/2217066393 wait complete. 2023-10-25T00:11:16.612 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:11:18.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:18.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f017b1f9640 1 Processor -- start 2023-10-25T00:11:18.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f017b1f9640 1 -- start start 2023-10-25T00:11:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f017b1f9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740c33e0 0x7f017415ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f017b1f9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01740c22f0 con 0x7f017414eb70 2023-10-25T00:11:18.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f017b1f9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01740c25d0 con 0x7f0174070b00 2023-10-25T00:11:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f017b1f9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0174071700 con 0x7f01740c33e0 2023-10-25T00:11:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f0178f95640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f017414eb70 0x7f01741514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38462/0 (socket says 172.21.15.107:38462) 2023-10-25T00:11:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f0178f95640 1 -- 172.21.15.107:0/2802301910 learned_addr learned my addr 172.21.15.107:0/2802301910 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:11:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f0179796640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740c33e0 0x7f017415ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227791375 0 0) 0x7f01740c22f0 con 0x7f017414eb70 2023-10-25T00:11:18.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0164003530 con 0x7f017414eb70 2023-10-25T00:11:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2701738284 0 0) 0x7f0164003530 con 0x7f017414eb70 2023-10-25T00:11:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740c33e0 msgr2=0x7f017415ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740c33e0 0x7f017415ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 >> v1:172.21.15.110:6789/0 conn(0x7f0174070b00 legacy=0x7f017406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:18.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01740c2be0 con 0x7f017414eb70 2023-10-25T00:11:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f0168001680 con 0x7f017414eb70 2023-10-25T00:11:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 >> v1:172.21.15.107:6789/0 conn(0x7f017414eb70 legacy=0x7f01741514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740c33e0 0x7f01640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0164003fe0 0x7f01640047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:18.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0164004d10 0x7f0164005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016400bb20 con 0x7f0164004d10 2023-10-25T00:11:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016400bc90 con 0x7f0164003fe0 2023-10-25T00:11:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016400be50 con 0x7f01740c33e0 2023-10-25T00:11:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f0179796640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0164004d10 0x7f0164005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:18.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f0178f95640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740c33e0 0x7f01640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f0173fff640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0164003fe0 0x7f01640047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f0179796640 1 -- 172.21.15.107:0/2802301910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740c33e0 msgr2=0x7f01640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f0179796640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740c33e0 0x7f01640039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f0179796640 1 -- 172.21.15.107:0/2802301910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0164003fe0 msgr2=0x7f01640047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:18.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f0179796640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0164003fe0 0x7f01640047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:18.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.745+0000 7f0179796640 1 -- 172.21.15.107:0/2802301910 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0164003530 con 0x7f0164004d10 2023-10-25T00:11:18.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f0179796640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0164004d10 0x7f0164005190 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f016c007360 tx=0x7f016c01ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473f952aa1b72d77 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:18.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c0a5d20 con 0x7f0164004d10 2023-10-25T00:11:18.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c0a5e80 con 0x7f0164004d10 2023-10-25T00:11:18.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f01737fe640 1 -- 172.21.15.107:0/2802301910 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c0ad660 con 0x7f0164004d10 2023-10-25T00:11:18.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f017b1f9640 1 -- 172.21.15.107:0/2802301910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0164004d10 msgr2=0x7f0164005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:18.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f017b1f9640 1 --2- 172.21.15.107:0/2802301910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0164004d10 0x7f0164005190 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f016c007360 tx=0x7f016c01ce20 comp rx=0 tx=0).stop 2023-10-25T00:11:18.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f0179796640 1 -- 172.21.15.107:0/2802301910 reap_dead start 2023-10-25T00:11:18.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f017b1f9640 1 -- 172.21.15.107:0/2802301910 shutdown_connections 2023-10-25T00:11:18.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f017b1f9640 1 -- 172.21.15.107:0/2802301910 >> 172.21.15.107:0/2802301910 conn(0x7f01740c39a0 msgr2=0x7f01740c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:18.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f017b1f9640 1 -- 172.21.15.107:0/2802301910 shutdown_connections 2023-10-25T00:11:18.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f017b1f9640 1 -- 172.21.15.107:0/2802301910 wait complete. 2023-10-25T00:11:18.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f017b1f9640 1 Processor -- start 2023-10-25T00:11:18.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.749+0000 7f017b1f9640 1 -- start start 2023-10-25T00:11:18.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f017b1f9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01741ead70 0x7f01741eb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:18.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f017b1f9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f017406c420 0x7f01741e8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:18.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f017b1f9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740a9450 0x7f01741e8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:18.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f017b1f9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01741ebae0 con 0x7f017406c420 2023-10-25T00:11:18.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f017b1f9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017406bf90 con 0x7f01741ead70 2023-10-25T00:11:18.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f017b1f9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01741eb800 con 0x7f01740a9450 2023-10-25T00:11:18.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f0173fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f017406c420 0x7f01741e8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:18.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f0173fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f017406c420 0x7f01741e8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45840/0 (socket says 172.21.15.107:45840) 2023-10-25T00:11:18.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f0173fff640 1 -- 172.21.15.107:0/176300593 learned_addr learned my addr 172.21.15.107:0/176300593 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:18.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f0178f95640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01741ead70 0x7f01741eb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:18.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f0173fff640 1 -- 172.21.15.107:0/176300593 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740a9450 msgr2=0x7f01741e8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:18.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f0173fff640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740a9450 0x7f01741e8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:18.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f0173fff640 1 -- 172.21.15.107:0/176300593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01741ead70 msgr2=0x7f01741eb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:18.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f0173fff640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01741ead70 0x7f01741eb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f0173fff640 1 -- 172.21.15.107:0/176300593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017406b940 con 0x7f017406c420 2023-10-25T00:11:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f0173fff640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f017406c420 0x7f01741e8a10 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f016000e580 tx=0x7f016000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f01717fa640 1 -- 172.21.15.107:0/176300593 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01600052e0 con 0x7f017406c420 2023-10-25T00:11:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f01717fa640 1 -- 172.21.15.107:0/176300593 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0160005440 con 0x7f017406c420 2023-10-25T00:11:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f01717fa640 1 -- 172.21.15.107:0/176300593 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0160003820 con 0x7f017406c420 2023-10-25T00:11:18.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f017b1f9640 1 -- 172.21.15.107:0/176300593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017406bc20 con 0x7f017406c420 2023-10-25T00:11:18.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f017b1f9640 1 -- 172.21.15.107:0/176300593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0174067800 con 0x7f017406c420 2023-10-25T00:11:18.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f017b1f9640 1 -- 172.21.15.107:0/176300593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017406b7a0 con 0x7f017406c420 2023-10-25T00:11:18.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f01717fa640 1 -- 172.21.15.107:0/176300593 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0160003980 con 0x7f017406c420 2023-10-25T00:11:18.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f01717fa640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0158065e90 0x7f0158068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:18.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f0178f95640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0158065e90 0x7f0158068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:18.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.753+0000 7f01717fa640 1 -- 172.21.15.107:0/176300593 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 9541+0+0 (secure 0 0 0) 0x7f0160084010 con 0x7f017406c420 2023-10-25T00:11:18.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.757+0000 7f0178f95640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0158065e90 0x7f0158068350 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0168001ca0 tx=0x7f0168009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:18.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.761+0000 7f01717fa640 1 -- 172.21.15.107:0/176300593 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f017406b7a0 con 0x7f017406c420 2023-10-25T00:11:18.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.957+0000 7f017b1f9640 1 -- 172.21.15.107:0/176300593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f017406b7a0 con 0x7f017406c420 2023-10-25T00:11:18.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f01717fa640 1 -- 172.21.15.107:0/176300593 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+5015 (secure 0 0 0) 0x7f017406b7a0 con 0x7f017406c420 2023-10-25T00:11:18.961 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:18.961 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:18.832580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:18.962 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 161 2023-10-25T00:11:18.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 -- 172.21.15.107:0/176300593 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0158065e90 msgr2=0x7f0158068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:18.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0158065e90 0x7f0158068350 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0168001ca0 tx=0x7f0168009040 comp rx=0 tx=0).stop 2023-10-25T00:11:18.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 -- 172.21.15.107:0/176300593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f017406c420 msgr2=0x7f01741e8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:18.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f017406c420 0x7f01741e8a10 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f016000e580 tx=0x7f016000ea40 comp rx=0 tx=0).stop 2023-10-25T00:11:18.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 -- 172.21.15.107:0/176300593 shutdown_connections 2023-10-25T00:11:18.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0158065e90 0x7f0158068350 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:18.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01740a9450 0x7f01741e8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:18.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f017406c420 0x7f01741e8a10 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:18.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 --2- 172.21.15.107:0/176300593 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01741ead70 0x7f01741eb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:18.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 -- 172.21.15.107:0/176300593 >> 172.21.15.107:0/176300593 conn(0x7f01740c39a0 msgr2=0x7f017414c530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:18.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 -- 172.21.15.107:0/176300593 shutdown_connections 2023-10-25T00:11:18.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:18.961+0000 7f017b1f9640 1 -- 172.21.15.107:0/176300593 wait complete. 2023-10-25T00:11:18.979 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2023-10-25T00:11:18.979 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:11:18.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:19.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.085+0000 7f65a6cd0640 1 Processor -- start 2023-10-25T00:11:19.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.085+0000 7f65a6cd0640 1 -- start start 2023-10-25T00:11:19.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a6cd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65a00abb50 0x7f65a00abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:19.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a6cd0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a014eaa0 con 0x7f65a014e6c0 2023-10-25T00:11:19.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a6cd0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a0072130 con 0x7f65a0157e20 2023-10-25T00:11:19.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a6cd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00c5b80 con 0x7f65a00abb50 2023-10-25T00:11:19.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a5cce640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f65a014e6c0 0x7f65a0155040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38472/0 (socket says 172.21.15.107:38472) 2023-10-25T00:11:19.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a5cce640 1 -- 172.21.15.107:0/3157713081 learned_addr learned my addr 172.21.15.107:0/3157713081 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:11:19.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a54cd640 1 --2- 172.21.15.107:0/3157713081 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65a00abb50 0x7f65a00abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:19.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a4ccc640 1 -- 172.21.15.107:0/3157713081 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017310615 0 0) 0x7f65a014eaa0 con 0x7f65a014e6c0 2023-10-25T00:11:19.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a4ccc640 1 -- 172.21.15.107:0/3157713081 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f658c003530 con 0x7f65a014e6c0 2023-10-25T00:11:19.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a4ccc640 1 -- 172.21.15.107:0/3157713081 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636216390 0 0) 0x7f65a0072130 con 0x7f65a0157e20 2023-10-25T00:11:19.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a4ccc640 1 -- 172.21.15.107:0/3157713081 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a014eaa0 con 0x7f65a0157e20 2023-10-25T00:11:19.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a54cd640 1 -- 172.21.15.107:0/3157713081 >> v1:172.21.15.110:6789/0 conn(0x7f65a0157e20 legacy=0x7f65a01582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:19.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a54cd640 1 -- 172.21.15.107:0/3157713081 >> v1:172.21.15.107:6789/0 conn(0x7f65a014e6c0 legacy=0x7f65a0155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:19.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a4ccc640 1 -- 172.21.15.107:0/3157713081 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2650538152 0 0) 0x7f658c003530 con 0x7f65a014e6c0 2023-10-25T00:11:19.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a54cd640 1 -- 172.21.15.107:0/3157713081 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a0072410 con 0x7f65a00abb50 2023-10-25T00:11:19.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a54cd640 1 --2- 172.21.15.107:0/3157713081 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65a00abb50 0x7f65a00abfd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6594009dc0 tx=0x7f659401c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c6766bd5f787321 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:19.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.089+0000 7f65a4ccc640 1 -- 172.21.15.107:0/3157713081 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c001680 con 0x7f65a00abb50 2023-10-25T00:11:19.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a4ccc640 1 -- 172.21.15.107:0/3157713081 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65940a5610 con 0x7f65a00abb50 2023-10-25T00:11:19.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a4ccc640 1 -- 172.21.15.107:0/3157713081 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65940a5980 con 0x7f65a00abb50 2023-10-25T00:11:19.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 -- 172.21.15.107:0/3157713081 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65a00abb50 msgr2=0x7f65a00abfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:19.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 --2- 172.21.15.107:0/3157713081 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65a00abb50 0x7f65a00abfd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6594009dc0 tx=0x7f659401c7d0 comp rx=0 tx=0).stop 2023-10-25T00:11:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 -- 172.21.15.107:0/3157713081 shutdown_connections 2023-10-25T00:11:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 --2- 172.21.15.107:0/3157713081 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65a00abb50 0x7f65a00abfd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 -- 172.21.15.107:0/3157713081 >> 172.21.15.107:0/3157713081 conn(0x7f65a00ca050 msgr2=0x7f65a00ca460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 -- 172.21.15.107:0/3157713081 shutdown_connections 2023-10-25T00:11:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 -- 172.21.15.107:0/3157713081 wait complete. 2023-10-25T00:11:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 Processor -- start 2023-10-25T00:11:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 -- start start 2023-10-25T00:11:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00abb50 0x7f65a014a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65a014e6c0 0x7f65a014a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65a0157e20 0x7f65a014f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00c3270 con 0x7f65a00abb50 2023-10-25T00:11:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a00c3830 con 0x7f65a014e6c0 2023-10-25T00:11:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a01518f0 con 0x7f65a0157e20 2023-10-25T00:11:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a5cce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00abb50 0x7f65a014a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:19.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a5cce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00abb50 0x7f65a014a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45850/0 (socket says 172.21.15.107:45850) 2023-10-25T00:11:19.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a5cce640 1 -- 172.21.15.107:0/2031637150 learned_addr learned my addr 172.21.15.107:0/2031637150 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:19.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a64cf640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65a0157e20 0x7f65a014f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:19.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a54cd640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65a014e6c0 0x7f65a014a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:19.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a5cce640 1 -- 172.21.15.107:0/2031637150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65a0157e20 msgr2=0x7f65a014f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:19.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a5cce640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65a0157e20 0x7f65a014f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:19.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a5cce640 1 -- 172.21.15.107:0/2031637150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65a014e6c0 msgr2=0x7f65a014a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:19.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a5cce640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65a014e6c0 0x7f65a014a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:19.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a5cce640 1 -- 172.21.15.107:0/2031637150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6594009a70 con 0x7f65a00abb50 2023-10-25T00:11:19.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a5cce640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00abb50 0x7f65a014a420 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f659c00c780 tx=0x7f659c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:19.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65927fc640 1 -- 172.21.15.107:0/2031637150 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c005cf0 con 0x7f65a00abb50 2023-10-25T00:11:19.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65927fc640 1 -- 172.21.15.107:0/2031637150 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659c005e50 con 0x7f65a00abb50 2023-10-25T00:11:19.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 -- 172.21.15.107:0/2031637150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a0151bd0 con 0x7f65a00abb50 2023-10-25T00:11:19.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65927fc640 1 -- 172.21.15.107:0/2031637150 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c007270 con 0x7f65a00abb50 2023-10-25T00:11:19.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.093+0000 7f65a6cd0640 1 -- 172.21.15.107:0/2031637150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a00c1090 con 0x7f65a00abb50 2023-10-25T00:11:19.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.097+0000 7f65a6cd0640 1 -- 172.21.15.107:0/2031637150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a00c12e0 con 0x7f65a00abb50 2023-10-25T00:11:19.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.097+0000 7f65927fc640 1 -- 172.21.15.107:0/2031637150 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f659c0073d0 con 0x7f65a00abb50 2023-10-25T00:11:19.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.097+0000 7f65927fc640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6584065e60 0x7f6584068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:19.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.097+0000 7f65a54cd640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6584065e60 0x7f6584068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:19.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.097+0000 7f65927fc640 1 -- 172.21.15.107:0/2031637150 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7f659c083d10 con 0x7f65a00abb50 2023-10-25T00:11:19.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.097+0000 7f65a54cd640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6584065e60 0x7f6584068320 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f65940af000 tx=0x7f65940a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:19.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.101+0000 7f65927fc640 1 -- 172.21.15.107:0/2031637150 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f659c050660 con 0x7f65a00abb50 2023-10-25T00:11:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 -- 172.21.15.107:0/2031637150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65a004e050 con 0x7f65a00abb50 2023-10-25T00:11:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65927fc640 1 -- 172.21.15.107:0/2031637150 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+5015 (secure 0 0 0) 0x7f659c017020 con 0x7f65a00abb50 2023-10-25T00:11:19.294 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:19.294 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:18.832580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 161 2023-10-25T00:11:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 -- 172.21.15.107:0/2031637150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6584065e60 msgr2=0x7f6584068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6584065e60 0x7f6584068320 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f65940af000 tx=0x7f65940a7070 comp rx=0 tx=0).stop 2023-10-25T00:11:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 -- 172.21.15.107:0/2031637150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00abb50 msgr2=0x7f65a014a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00abb50 0x7f65a014a420 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f659c00c780 tx=0x7f659c00cc40 comp rx=0 tx=0).stop 2023-10-25T00:11:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 -- 172.21.15.107:0/2031637150 shutdown_connections 2023-10-25T00:11:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6584065e60 0x7f6584068320 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f65a0157e20 0x7f65a014f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65a014e6c0 0x7f65a014a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 --2- 172.21.15.107:0/2031637150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f65a00abb50 0x7f65a014a420 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 -- 172.21.15.107:0/2031637150 >> 172.21.15.107:0/2031637150 conn(0x7f65a00ca050 msgr2=0x7f65a00c8460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 -- 172.21.15.107:0/2031637150 shutdown_connections 2023-10-25T00:11:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:19.293+0000 7f65a6cd0640 1 -- 172.21.15.107:0/2031637150 wait complete. 2023-10-25T00:11:19.312 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 161, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 161, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6805": { "addr": "172.21.15.110:6835/898258574", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 898258574, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 898258574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6805, "incarnation": 161, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:11:18.832580+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6805 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 159, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7839": { "addr": "172.21.15.107:6837/3370462806", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 3370462806, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 3370462806, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7839, "incarnation": 156, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:11:06.860947+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7839 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:11:21.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:21.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee7913640 1 Processor -- start 2023-10-25T00:11:21.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee7913640 1 -- start start 2023-10-25T00:11:21.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee7913640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4ee01561e0 0x7f4ee00c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:21.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee7913640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4ee006bc30 0x7f4ee015c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:21.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee7913640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee00c1da0 0x7f4ee015c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:21.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee7913640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee01565c0 con 0x7f4ee01561e0 2023-10-25T00:11:21.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee7913640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee006a2b0 con 0x7f4ee006bc30 2023-10-25T00:11:21.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee7913640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee006a590 con 0x7f4ee00c1da0 2023-10-25T00:11:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee56af640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4ee01561e0 0x7f4ee00c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee56af640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4ee01561e0 0x7f4ee00c1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45860/0 (socket says 172.21.15.107:45860) 2023-10-25T00:11:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee56af640 1 -- 172.21.15.107:0/1333922269 learned_addr learned my addr 172.21.15.107:0/1333922269 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee5eb0640 1 --2- 172.21.15.107:0/1333922269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee00c1da0 0x7f4ee015c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:21.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee4eae640 1 --2- 172.21.15.107:0/1333922269 >> v2:172.21.15.110:3300/0 conn(0x7f4ee006bc30 0x7f4ee015c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee56af640 1 -- 172.21.15.107:0/1333922269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee00c1da0 msgr2=0x7f4ee015c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee56af640 1 --2- 172.21.15.107:0/1333922269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee00c1da0 0x7f4ee015c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee56af640 1 -- 172.21.15.107:0/1333922269 >> v2:172.21.15.110:3300/0 conn(0x7f4ee006bc30 msgr2=0x7f4ee015c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee56af640 1 --2- 172.21.15.107:0/1333922269 >> v2:172.21.15.110:3300/0 conn(0x7f4ee006bc30 0x7f4ee015c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:21.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee56af640 1 -- 172.21.15.107:0/1333922269 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee006a870 con 0x7f4ee01561e0 2023-10-25T00:11:21.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.441+0000 7f4ee5eb0640 1 --2- 172.21.15.107:0/1333922269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee00c1da0 0x7f4ee015c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:11:21.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee56af640 1 --2- 172.21.15.107:0/1333922269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ee01561e0 0x7f4ee00c1860 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f4ee006d220 tx=0x7f4ed801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0519ba33a76513 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:21.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ed3fff640 1 -- 172.21.15.107:0/1333922269 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed80ab600 con 0x7f4ee01561e0 2023-10-25T00:11:21.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ed3fff640 1 -- 172.21.15.107:0/1333922269 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed80a7050 con 0x7f4ee01561e0 2023-10-25T00:11:21.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ed3fff640 1 -- 172.21.15.107:0/1333922269 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed80ab980 con 0x7f4ee01561e0 2023-10-25T00:11:21.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 -- 172.21.15.107:0/1333922269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ee01561e0 msgr2=0x7f4ee00c1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:21.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 --2- 172.21.15.107:0/1333922269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ee01561e0 0x7f4ee00c1860 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f4ee006d220 tx=0x7f4ed801c470 comp rx=0 tx=0).stop 2023-10-25T00:11:21.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 -- 172.21.15.107:0/1333922269 shutdown_connections 2023-10-25T00:11:21.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 --2- 172.21.15.107:0/1333922269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee00c1da0 0x7f4ee015c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:21.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 --2- 172.21.15.107:0/1333922269 >> v2:172.21.15.110:3300/0 conn(0x7f4ee006bc30 0x7f4ee015c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:21.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 --2- 172.21.15.107:0/1333922269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ee01561e0 0x7f4ee00c1860 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:21.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 -- 172.21.15.107:0/1333922269 >> 172.21.15.107:0/1333922269 conn(0x7f4ee00c2950 msgr2=0x7f4ee00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:21.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 -- 172.21.15.107:0/1333922269 shutdown_connections 2023-10-25T00:11:21.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 -- 172.21.15.107:0/1333922269 wait complete. 2023-10-25T00:11:21.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 Processor -- start 2023-10-25T00:11:21.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 -- start start 2023-10-25T00:11:21.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ee00c1da0 0x7f4ee014dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:21.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ee01561e0 0x7f4ee0071690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:21.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee0071bd0 0x7f4ee014c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:21.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee0071240 con 0x7f4ee00c1da0 2023-10-25T00:11:21.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee0072060 con 0x7f4ee01561e0 2023-10-25T00:11:21.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee7913640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee014e360 con 0x7f4ee0071bd0 2023-10-25T00:11:21.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee56af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ee00c1da0 0x7f4ee014dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:21.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee56af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ee00c1da0 0x7f4ee014dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45866/0 (socket says 172.21.15.107:45866) 2023-10-25T00:11:21.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.445+0000 7f4ee56af640 1 -- 172.21.15.107:0/681316778 learned_addr learned my addr 172.21.15.107:0/681316778 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:21.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee4eae640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ee01561e0 0x7f4ee0071690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:21.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee5eb0640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee0071bd0 0x7f4ee014c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:21.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee56af640 1 -- 172.21.15.107:0/681316778 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee0071bd0 msgr2=0x7f4ee014c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:21.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee56af640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee0071bd0 0x7f4ee014c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:21.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee56af640 1 -- 172.21.15.107:0/681316778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ee01561e0 msgr2=0x7f4ee0071690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:21.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee56af640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ee01561e0 0x7f4ee0071690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:21.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee56af640 1 -- 172.21.15.107:0/681316778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed80094d0 con 0x7f4ee00c1da0 2023-10-25T00:11:21.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee5eb0640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee0071bd0 0x7f4ee014c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:11:21.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee56af640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ee00c1da0 0x7f4ee014dd50 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f4ed801cd10 tx=0x7f4ed8019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:21.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ed1ffb640 1 -- 172.21.15.107:0/681316778 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed80b6440 con 0x7f4ee00c1da0 2023-10-25T00:11:21.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ed1ffb640 1 -- 172.21.15.107:0/681316778 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed80b65c0 con 0x7f4ee00c1da0 2023-10-25T00:11:21.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee7913640 1 -- 172.21.15.107:0/681316778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee015d540 con 0x7f4ee00c1da0 2023-10-25T00:11:21.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ed1ffb640 1 -- 172.21.15.107:0/681316778 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed80b6970 con 0x7f4ee00c1da0 2023-10-25T00:11:21.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee7913640 1 -- 172.21.15.107:0/681316778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee014cf20 con 0x7f4ee00c1da0 2023-10-25T00:11:21.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee7913640 1 -- 172.21.15.107:0/681316778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee00746e0 con 0x7f4ee00c1da0 2023-10-25T00:11:21.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ed1ffb640 1 -- 172.21.15.107:0/681316778 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4ed80b2040 con 0x7f4ee00c1da0 2023-10-25T00:11:21.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ed1ffb640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ec4065e90 0x7f4ec4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:21.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ed1ffb640 1 -- 172.21.15.107:0/681316778 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7f4ed811e540 con 0x7f4ee00c1da0 2023-10-25T00:11:21.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee4eae640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ec4065e90 0x7f4ec4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:21.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.449+0000 7f4ee4eae640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ec4065e90 0x7f4ec4068350 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4ee014e290 tx=0x7f4edc00d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:21.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.453+0000 7f4ed1ffb640 1 -- 172.21.15.107:0/681316778 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4ed80eade0 con 0x7f4ee00c1da0 2023-10-25T00:11:21.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.649+0000 7f4ee7913640 1 -- 172.21.15.107:0/681316778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ee00c03a0 con 0x7f4ee00c1da0 2023-10-25T00:11:21.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.649+0000 7f4ed1ffb640 1 -- 172.21.15.107:0/681316778 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+5015 (secure 0 0 0) 0x7f4ed80b6ad0 con 0x7f4ee00c1da0 2023-10-25T00:11:21.651 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:21.651 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:20.872119+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:21.651 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 163 2023-10-25T00:11:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 -- 172.21.15.107:0/681316778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ec4065e90 msgr2=0x7f4ec4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ec4065e90 0x7f4ec4068350 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4ee014e290 tx=0x7f4edc00d040 comp rx=0 tx=0).stop 2023-10-25T00:11:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 -- 172.21.15.107:0/681316778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ee00c1da0 msgr2=0x7f4ee014dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ee00c1da0 0x7f4ee014dd50 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f4ed801cd10 tx=0x7f4ed8019210 comp rx=0 tx=0).stop 2023-10-25T00:11:21.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 -- 172.21.15.107:0/681316778 shutdown_connections 2023-10-25T00:11:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ec4065e90 0x7f4ec4068350 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ee0071bd0 0x7f4ee014c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ee01561e0 0x7f4ee0071690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:21.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 --2- 172.21.15.107:0/681316778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ee00c1da0 0x7f4ee014dd50 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 -- 172.21.15.107:0/681316778 >> 172.21.15.107:0/681316778 conn(0x7f4ee00c2950 msgr2=0x7f4ee0155140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 -- 172.21.15.107:0/681316778 shutdown_connections 2023-10-25T00:11:21.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:21.653+0000 7f4ee7913640 1 -- 172.21.15.107:0/681316778 wait complete. 2023-10-25T00:11:21.669 INFO:tasks.mds_thrash.fs.[a]:no change 2023-10-25T00:11:22.809 INFO:tasks.ceph.mds.d.smithi110.stderr:2023-10-25T00:11:22.809+0000 7fb5f85d8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:11:23.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:23.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bfb4b640 1 Processor -- start 2023-10-25T00:11:23.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bfb4b640 1 -- start start 2023-10-25T00:11:23.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bfb4b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f78b806f490 0x7f78b814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:23.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bfb4b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f78b8152b30 0x7f78b8152f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:23.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bfb4b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b8155cd0 con 0x7f78b814c470 2023-10-25T00:11:23.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bfb4b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b81526e0 con 0x7f78b806f490 2023-10-25T00:11:23.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bfb4b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b80c4380 con 0x7f78b8152b30 2023-10-25T00:11:23.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bd0e6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f78b806f490 0x7f78b814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:23.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bd8e7640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f78b814c470 0x7f78b8071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42338/0 (socket says 172.21.15.107:42338) 2023-10-25T00:11:23.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bd8e7640 1 -- 172.21.15.107:0/2079301353 learned_addr learned my addr 172.21.15.107:0/2079301353 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:11:23.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78be0e8640 1 --2- 172.21.15.107:0/2079301353 >> v2:172.21.15.110:3300/0 conn(0x7f78b8152b30 0x7f78b8152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:23.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bd0e6640 1 -- 172.21.15.107:0/2079301353 >> v2:172.21.15.110:3300/0 conn(0x7f78b8152b30 msgr2=0x7f78b8152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:11:23.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bd0e6640 1 --2- 172.21.15.107:0/2079301353 >> v2:172.21.15.110:3300/0 conn(0x7f78b8152b30 0x7f78b8152f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:23.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bd0e6640 1 -- 172.21.15.107:0/2079301353 >> v1:172.21.15.107:6789/0 conn(0x7f78b814c470 legacy=0x7f78b8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:23.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bc8e5640 1 -- 172.21.15.107:0/2079301353 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693854538 0 0) 0x7f78b8155cd0 con 0x7f78b814c470 2023-10-25T00:11:23.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.817+0000 7f78bd0e6640 1 -- 172.21.15.107:0/2079301353 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b80c48b0 con 0x7f78b806f490 2023-10-25T00:11:23.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bd0e6640 1 --2- 172.21.15.107:0/2079301353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78b806f490 0x7f78b814c850 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f78b806fa90 tx=0x7f78a801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de947a18516b984 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:23.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bc8e5640 1 -- 172.21.15.107:0/2079301353 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a80ab600 con 0x7f78b806f490 2023-10-25T00:11:23.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bc8e5640 1 -- 172.21.15.107:0/2079301353 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a80a7050 con 0x7f78b806f490 2023-10-25T00:11:23.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bc8e5640 1 -- 172.21.15.107:0/2079301353 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a80ab980 con 0x7f78b806f490 2023-10-25T00:11:23.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bfb4b640 1 -- 172.21.15.107:0/2079301353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78b806f490 msgr2=0x7f78b814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:23.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bfb4b640 1 --2- 172.21.15.107:0/2079301353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78b806f490 0x7f78b814c850 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f78b806fa90 tx=0x7f78a801c340 comp rx=0 tx=0).stop 2023-10-25T00:11:23.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bfb4b640 1 -- 172.21.15.107:0/2079301353 shutdown_connections 2023-10-25T00:11:23.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bfb4b640 1 --2- 172.21.15.107:0/2079301353 >> v2:172.21.15.110:3300/0 conn(0x7f78b8152b30 0x7f78b8152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:23.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bfb4b640 1 --2- 172.21.15.107:0/2079301353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78b806f490 0x7f78b814c850 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:23.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bfb4b640 1 -- 172.21.15.107:0/2079301353 >> 172.21.15.107:0/2079301353 conn(0x7f78b8070f60 msgr2=0x7f78b8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:23.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bfb4b640 1 -- 172.21.15.107:0/2079301353 shutdown_connections 2023-10-25T00:11:23.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.821+0000 7f78bfb4b640 1 -- 172.21.15.107:0/2079301353 wait complete. 2023-10-25T00:11:23.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bfb4b640 1 Processor -- start 2023-10-25T00:11:23.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bfb4b640 1 -- start start 2023-10-25T00:11:23.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bfb4b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78b806f490 0x7f78b81eb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:23.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bfb4b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78b814c470 0x7f78b81429f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:23.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bfb4b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78b8152b30 0x7f78b814e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:23.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bfb4b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b8142f30 con 0x7f78b806f490 2023-10-25T00:11:23.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bfb4b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b80ab0e0 con 0x7f78b814c470 2023-10-25T00:11:23.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bfb4b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b814d140 con 0x7f78b8152b30 2023-10-25T00:11:23.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bd8e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78b806f490 0x7f78b81eb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:23.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bd8e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78b806f490 0x7f78b81eb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58710/0 (socket says 172.21.15.107:58710) 2023-10-25T00:11:23.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bd8e7640 1 -- 172.21.15.107:0/993157456 learned_addr learned my addr 172.21.15.107:0/993157456 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:23.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bd0e6640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78b814c470 0x7f78b81429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:23.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78be0e8640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78b8152b30 0x7f78b814e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bd8e7640 1 -- 172.21.15.107:0/993157456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78b8152b30 msgr2=0x7f78b814e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bd8e7640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78b8152b30 0x7f78b814e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bd8e7640 1 -- 172.21.15.107:0/993157456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78b814c470 msgr2=0x7f78b81429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bd8e7640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78b814c470 0x7f78b81429f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bd8e7640 1 -- 172.21.15.107:0/993157456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a8009420 con 0x7f78b806f490 2023-10-25T00:11:23.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bd8e7640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78b806f490 0x7f78b81eb070 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f78b40011f0 tx=0x7f78b400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:23.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78ae7fc640 1 -- 172.21.15.107:0/993157456 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40074e0 con 0x7f78b806f490 2023-10-25T00:11:23.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78ae7fc640 1 -- 172.21.15.107:0/993157456 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b4007640 con 0x7f78b806f490 2023-10-25T00:11:23.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78ae7fc640 1 -- 172.21.15.107:0/993157456 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4004a10 con 0x7f78b806f490 2023-10-25T00:11:23.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bfb4b640 1 -- 172.21.15.107:0/993157456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b814cb80 con 0x7f78b806f490 2023-10-25T00:11:23.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.825+0000 7f78bfb4b640 1 -- 172.21.15.107:0/993157456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b8155730 con 0x7f78b806f490 2023-10-25T00:11:23.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.829+0000 7f78bfb4b640 1 -- 172.21.15.107:0/993157456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b8155f30 con 0x7f78b806f490 2023-10-25T00:11:23.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.829+0000 7f78ae7fc640 1 -- 172.21.15.107:0/993157456 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f78b4004b70 con 0x7f78b806f490 2023-10-25T00:11:23.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.829+0000 7f78ae7fc640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f789c065e60 0x7f789c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:23.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.829+0000 7f78ae7fc640 1 -- 172.21.15.107:0/993157456 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7f78b4083850 con 0x7f78b806f490 2023-10-25T00:11:23.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.829+0000 7f78bd0e6640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f789c065e60 0x7f789c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:23.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.829+0000 7f78bd0e6640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f789c065e60 0x7f789c068320 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f78a80ad000 tx=0x7f78a80a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:23.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:23.833+0000 7f78ae7fc640 1 -- 172.21.15.107:0/993157456 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f78b40501a0 con 0x7f78b806f490 2023-10-25T00:11:24.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.089+0000 7f78bfb4b640 1 -- 172.21.15.107:0/993157456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78b80746e0 con 0x7f78b806f490 2023-10-25T00:11:24.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.089+0000 7f78ae7fc640 1 -- 172.21.15.107:0/993157456 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5015 (secure 0 0 0) 0x7f78b4018030 con 0x7f78b806f490 2023-10-25T00:11:24.092 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:24.093 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:24.093 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 164 2023-10-25T00:11:24.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 -- 172.21.15.107:0/993157456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f789c065e60 msgr2=0x7f789c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f789c065e60 0x7f789c068320 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f78a80ad000 tx=0x7f78a80a6000 comp rx=0 tx=0).stop 2023-10-25T00:11:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 -- 172.21.15.107:0/993157456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78b806f490 msgr2=0x7f78b81eb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78b806f490 0x7f78b81eb070 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f78b40011f0 tx=0x7f78b400c9b0 comp rx=0 tx=0).stop 2023-10-25T00:11:24.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 -- 172.21.15.107:0/993157456 shutdown_connections 2023-10-25T00:11:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f789c065e60 0x7f789c068320 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78b8152b30 0x7f78b814e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78b814c470 0x7f78b81429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:24.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 --2- 172.21.15.107:0/993157456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78b806f490 0x7f78b81eb070 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 -- 172.21.15.107:0/993157456 >> 172.21.15.107:0/993157456 conn(0x7f78b8070f60 msgr2=0x7f78b8071340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 -- 172.21.15.107:0/993157456 shutdown_connections 2023-10-25T00:11:24.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:24.093+0000 7f78bfb4b640 1 -- 172.21.15.107:0/993157456 wait complete. 2023-10-25T00:11:24.113 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=7746 2023-10-25T00:11:24.114 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.c 2023-10-25T00:11:31.114 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2023-10-25T00:11:31.115 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:11:31.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:11:31.118 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:11:31.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:11:31.195 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:11:31.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f3028fc2640 1 Processor -- start 2023-10-25T00:11:31.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f3028fc2640 1 -- start start 2023-10-25T00:11:31.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f3028fc2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30240aa140 0x7f30240aa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:31.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f3028fc2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3024066000 con 0x7f30240ac020 2023-10-25T00:11:31.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f3028fc2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30240aaa60 con 0x7f30240c26f0 2023-10-25T00:11:31.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f3028fc2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30240c0f40 con 0x7f30240aa140 2023-10-25T00:11:31.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f302259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30240aa140 0x7f30240aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:31.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f302259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f30240aa140 0x7f30240aa520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58726/0 (socket says 172.21.15.107:58726) 2023-10-25T00:11:31.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f302259c640 1 -- 172.21.15.107:0/4101555053 learned_addr learned my addr 172.21.15.107:0/4101555053 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:31.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f3021d9b640 1 --1- 172.21.15.107:0/4101555053 >> v1:172.21.15.107:6789/0 conn(0x7f30240ac020 0x7f30240c1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42354/0 (socket says 172.21.15.107:42354) 2023-10-25T00:11:31.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f302259c640 1 -- 172.21.15.107:0/4101555053 >> v1:172.21.15.110:6789/0 conn(0x7f30240c26f0 legacy=0x7f302406c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:31.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.261+0000 7f302259c640 1 -- 172.21.15.107:0/4101555053 >> v1:172.21.15.107:6789/0 conn(0x7f30240ac020 legacy=0x7f30240c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:31.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f302259c640 1 -- 172.21.15.107:0/4101555053 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30240c1220 con 0x7f30240aa140 2023-10-25T00:11:31.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f302259c640 1 --2- 172.21.15.107:0/4101555053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30240aa140 0x7f30240aa520 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f3014009870 tx=0x7f301401e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b85ed04dd026d1b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:31.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f302159a640 1 -- 172.21.15.107:0/4101555053 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30140a86f0 con 0x7f30240aa140 2023-10-25T00:11:31.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f302159a640 1 -- 172.21.15.107:0/4101555053 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30140a8850 con 0x7f30240aa140 2023-10-25T00:11:31.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f302159a640 1 -- 172.21.15.107:0/4101555053 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30140a8be0 con 0x7f30240aa140 2023-10-25T00:11:31.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f3028fc2640 1 -- 172.21.15.107:0/4101555053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30240aa140 msgr2=0x7f30240aa520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:31.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f3028fc2640 1 --2- 172.21.15.107:0/4101555053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30240aa140 0x7f30240aa520 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f3014009870 tx=0x7f301401e720 comp rx=0 tx=0).stop 2023-10-25T00:11:31.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f3028fc2640 1 -- 172.21.15.107:0/4101555053 shutdown_connections 2023-10-25T00:11:31.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f3028fc2640 1 --2- 172.21.15.107:0/4101555053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30240aa140 0x7f30240aa520 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:31.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f3028fc2640 1 -- 172.21.15.107:0/4101555053 >> 172.21.15.107:0/4101555053 conn(0x7f30240c65c0 msgr2=0x7f30240c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:31.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f3028fc2640 1 -- 172.21.15.107:0/4101555053 shutdown_connections 2023-10-25T00:11:31.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.265+0000 7f3028fc2640 1 -- 172.21.15.107:0/4101555053 wait complete. 2023-10-25T00:11:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3028fc2640 1 Processor -- start 2023-10-25T00:11:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3028fc2640 1 -- start start 2023-10-25T00:11:31.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3028fc2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240aa140 0x7f30240bbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3028fc2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30240ac020 0x7f30240707a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3028fc2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30240c26f0 0x7f3024152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3028fc2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30240ba7c0 con 0x7f30240ac020 2023-10-25T00:11:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3028fc2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3024070ce0 con 0x7f30240c26f0 2023-10-25T00:11:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3028fc2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3024150640 con 0x7f30240aa140 2023-10-25T00:11:31.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3021d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30240ac020 0x7f30240707a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:31.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3021d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30240ac020 0x7f30240707a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58734/0 (socket says 172.21.15.107:58734) 2023-10-25T00:11:31.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3021d9b640 1 -- 172.21.15.107:0/1598279896 learned_addr learned my addr 172.21.15.107:0/1598279896 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:11:31.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3022d9d640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30240c26f0 0x7f3024152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:31.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f302259c640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240aa140 0x7f30240bbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3021d9b640 1 -- 172.21.15.107:0/1598279896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240aa140 msgr2=0x7f30240bbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3021d9b640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240aa140 0x7f30240bbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3021d9b640 1 -- 172.21.15.107:0/1598279896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30240c26f0 msgr2=0x7f3024152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3021d9b640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30240c26f0 0x7f3024152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3021d9b640 1 -- 172.21.15.107:0/1598279896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30140094d0 con 0x7f30240ac020 2023-10-25T00:11:31.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f302259c640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240aa140 0x7f30240bbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:11:31.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3021d9b640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30240ac020 0x7f30240707a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f301800c400 tx=0x7f301800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:31.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3012ffd640 1 -- 172.21.15.107:0/1598279896 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30180074e0 con 0x7f30240ac020 2023-10-25T00:11:31.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3012ffd640 1 -- 172.21.15.107:0/1598279896 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3018007660 con 0x7f30240ac020 2023-10-25T00:11:31.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3012ffd640 1 -- 172.21.15.107:0/1598279896 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018004a30 con 0x7f30240ac020 2023-10-25T00:11:31.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3028fc2640 1 -- 172.21.15.107:0/1598279896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3024150360 con 0x7f30240ac020 2023-10-25T00:11:31.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.269+0000 7f3028fc2640 1 -- 172.21.15.107:0/1598279896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30240ba450 con 0x7f30240ac020 2023-10-25T00:11:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.273+0000 7f3028fc2640 1 -- 172.21.15.107:0/1598279896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30240ba5e0 con 0x7f30240ac020 2023-10-25T00:11:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.273+0000 7f3012ffd640 1 -- 172.21.15.107:0/1598279896 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3018004b90 con 0x7f30240ac020 2023-10-25T00:11:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.273+0000 7f3012ffd640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3004065e90 0x7f3004068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:11:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.273+0000 7f302259c640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3004065e90 0x7f3004068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:11:31.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.273+0000 7f3012ffd640 1 -- 172.21.15.107:0/1598279896 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7f3018083fe0 con 0x7f30240ac020 2023-10-25T00:11:31.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.273+0000 7f302259c640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3004065e90 0x7f3004068350 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f30140b1000 tx=0x7f3014008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:11:31.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.281+0000 7f3012ffd640 1 -- 172.21.15.107:0/1598279896 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3018050880 con 0x7f30240ac020 2023-10-25T00:11:31.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.501+0000 7f3028fc2640 1 -- 172.21.15.107:0/1598279896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30240746e0 con 0x7f30240ac020 2023-10-25T00:11:31.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.501+0000 7f3012ffd640 1 -- 172.21.15.107:0/1598279896 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f3018053ea0 con 0x7f30240ac020 2023-10-25T00:11:31.499 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:11:31.500 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:11:31.500 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 165 2023-10-25T00:11:31.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 -- 172.21.15.107:0/1598279896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3004065e90 msgr2=0x7f3004068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:31.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3004065e90 0x7f3004068350 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f30140b1000 tx=0x7f3014008040 comp rx=0 tx=0).stop 2023-10-25T00:11:31.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 -- 172.21.15.107:0/1598279896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30240ac020 msgr2=0x7f30240707a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:11:31.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30240ac020 0x7f30240707a0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f301800c400 tx=0x7f301800c8c0 comp rx=0 tx=0).stop 2023-10-25T00:11:31.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 -- 172.21.15.107:0/1598279896 shutdown_connections 2023-10-25T00:11:31.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f30240c26f0 0x7f3024152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:31.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30240ac020 0x7f30240707a0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:31.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3004065e90 0x7f3004068350 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:31.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 --2- 172.21.15.107:0/1598279896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f30240aa140 0x7f30240bbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:11:31.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 -- 172.21.15.107:0/1598279896 >> 172.21.15.107:0/1598279896 conn(0x7f30240c65c0 msgr2=0x7f30240c5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:11:31.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 -- 172.21.15.107:0/1598279896 shutdown_connections 2023-10-25T00:11:31.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:11:31.505+0000 7f3028fc2640 1 -- 172.21.15.107:0/1598279896 wait complete. 2023-10-25T00:11:31.519 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2023-10-25T00:11:31.519 INFO:tasks.mds_thrash.fs.[a]:waiting for 92 secs before thrashing 2023-10-25T00:12:15.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:15.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd415a64640 1 Processor -- start 2023-10-25T00:12:15.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd415a64640 1 -- start start 2023-10-25T00:12:15.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd415a64640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4100aa330 0x7fd41006e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd415a64640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd4100a9450 0x7fd410071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd415a64640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd410152d40 con 0x7fd41013bb00 2023-10-25T00:12:15.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd415a64640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd410156100 con 0x7fd4100a9450 2023-10-25T00:12:15.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd415a64640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4101527c0 con 0x7fd4100aa330 2023-10-25T00:12:15.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd40ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd4100a9450 0x7fd410071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd40ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd4100a9450 0x7fd410071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51706/0 (socket says 172.21.15.107:51706) 2023-10-25T00:12:15.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd40ffff640 1 -- 172.21.15.107:0/640133950 learned_addr learned my addr 172.21.15.107:0/640133950 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:15.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd407fff640 1 --2- 172.21.15.107:0/640133950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4100aa330 0x7fd41006e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd40ffff640 1 -- 172.21.15.107:0/640133950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4100aa330 msgr2=0x7fd41006e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:15.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd40ffff640 1 --2- 172.21.15.107:0/640133950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4100aa330 0x7fd41006e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd40ffff640 1 -- 172.21.15.107:0/640133950 >> v1:172.21.15.110:6789/0 conn(0x7fd41013bb00 legacy=0x7fd4101531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:15.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd40ffff640 1 -- 172.21.15.107:0/640133950 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41006d660 con 0x7fd4100a9450 2023-10-25T00:12:15.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.160+0000 7fd407fff640 1 --2- 172.21.15.107:0/640133950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4100aa330 0x7fd41006e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:12:15.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd40ffff640 1 --2- 172.21.15.107:0/640133950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4100a9450 0x7fd410071a80 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fd408009e80 tx=0x7fd40801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb957cb598343bd server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:15.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd40effd640 1 -- 172.21.15.107:0/640133950 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4080a5600 con 0x7fd4100a9450 2023-10-25T00:12:15.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd40effd640 1 -- 172.21.15.107:0/640133950 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4080a5760 con 0x7fd4100a9450 2023-10-25T00:12:15.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd40effd640 1 -- 172.21.15.107:0/640133950 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4080a5af0 con 0x7fd4100a9450 2023-10-25T00:12:15.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd415a64640 1 -- 172.21.15.107:0/640133950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4100a9450 msgr2=0x7fd410071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:15.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd415a64640 1 --2- 172.21.15.107:0/640133950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4100a9450 0x7fd410071a80 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fd408009e80 tx=0x7fd40801c890 comp rx=0 tx=0).stop 2023-10-25T00:12:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd415a64640 1 -- 172.21.15.107:0/640133950 shutdown_connections 2023-10-25T00:12:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd415a64640 1 --2- 172.21.15.107:0/640133950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4100a9450 0x7fd410071a80 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd415a64640 1 --2- 172.21.15.107:0/640133950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4100aa330 0x7fd41006e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd415a64640 1 -- 172.21.15.107:0/640133950 >> 172.21.15.107:0/640133950 conn(0x7fd410070f60 msgr2=0x7fd410071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:15.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd415a64640 1 -- 172.21.15.107:0/640133950 shutdown_connections 2023-10-25T00:12:15.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.164+0000 7fd415a64640 1 -- 172.21.15.107:0/640133950 wait complete. 2023-10-25T00:12:15.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd415a64640 1 Processor -- start 2023-10-25T00:12:15.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd415a64640 1 -- start start 2023-10-25T00:12:15.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd415a64640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4100a9450 0x7fd4101eb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd415a64640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4100aa330 0x7fd4100c32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd415a64640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd41013bb00 0x7fd410154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd415a64640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4101ebb80 con 0x7fd41013bb00 2023-10-25T00:12:15.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd415a64640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd410154410 con 0x7fd4100a9450 2023-10-25T00:12:15.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd415a64640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4100c70f0 con 0x7fd4100aa330 2023-10-25T00:12:15.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd41013bb00 0x7fd410154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd41013bb00 0x7fd410154b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51714/0 (socket says 172.21.15.107:51714) 2023-10-25T00:12:15.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40ffff640 1 -- 172.21.15.107:0/4207467610 learned_addr learned my addr 172.21.15.107:0/4207467610 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:15.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd407fff640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4100aa330 0x7fd4100c32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40f7fe640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4100a9450 0x7fd4101eb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40ffff640 1 -- 172.21.15.107:0/4207467610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4100aa330 msgr2=0x7fd4100c32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:15.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40ffff640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4100aa330 0x7fd4100c32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40ffff640 1 -- 172.21.15.107:0/4207467610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4100a9450 msgr2=0x7fd4101eb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:15.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40ffff640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4100a9450 0x7fd4101eb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40ffff640 1 -- 172.21.15.107:0/4207467610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd408009b30 con 0x7fd41013bb00 2023-10-25T00:12:15.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40ffff640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd41013bb00 0x7fd410154b80 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fd40801af40 tx=0x7fd408019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:15.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40cff9640 1 -- 172.21.15.107:0/4207467610 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4080b8880 con 0x7fd41013bb00 2023-10-25T00:12:15.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40cff9640 1 -- 172.21.15.107:0/4207467610 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4080b8a00 con 0x7fd41013bb00 2023-10-25T00:12:15.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd415a64640 1 -- 172.21.15.107:0/4207467610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4100c73d0 con 0x7fd41013bb00 2023-10-25T00:12:15.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd40cff9640 1 -- 172.21.15.107:0/4207467610 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4080b8db0 con 0x7fd41013bb00 2023-10-25T00:12:15.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.168+0000 7fd415a64640 1 -- 172.21.15.107:0/4207467610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4100c7990 con 0x7fd41013bb00 2023-10-25T00:12:15.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.172+0000 7fd415a64640 1 -- 172.21.15.107:0/4207467610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd41006f3f0 con 0x7fd41013bb00 2023-10-25T00:12:15.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.172+0000 7fd40cff9640 1 -- 172.21.15.107:0/4207467610 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd4080b6440 con 0x7fd41013bb00 2023-10-25T00:12:15.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.172+0000 7fd40cff9640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3f0065e60 0x7fd3f0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.172+0000 7fd40cff9640 1 -- 172.21.15.107:0/4207467610 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7fd40811fe50 con 0x7fd41013bb00 2023-10-25T00:12:15.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.172+0000 7fd40f7fe640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3f0065e60 0x7fd3f0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.172+0000 7fd40f7fe640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3f0065e60 0x7fd3f0068320 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd4101e8a00 tx=0x7fd400008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:15.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.176+0000 7fd40cff9640 1 -- 172.21.15.107:0/4207467610 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd4080ec670 con 0x7fd41013bb00 2023-10-25T00:12:15.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd415a64640 1 -- 172.21.15.107:0/4207467610 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4100746e0 con 0x7fd41013bb00 2023-10-25T00:12:15.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd40cff9640 1 -- 172.21.15.107:0/4207467610 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5776 (secure 0 0 0) 0x7fd408009510 con 0x7fd41013bb00 2023-10-25T00:12:15.394 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:15.394 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:15.394 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 165 2023-10-25T00:12:15.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd415a64640 1 -- 172.21.15.107:0/4207467610 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3f0065e60 msgr2=0x7fd3f0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:15.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd415a64640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3f0065e60 0x7fd3f0068320 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd4101e8a00 tx=0x7fd400008040 comp rx=0 tx=0).stop 2023-10-25T00:12:15.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd415a64640 1 -- 172.21.15.107:0/4207467610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd41013bb00 msgr2=0x7fd410154b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:15.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd415a64640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd41013bb00 0x7fd410154b80 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fd40801af40 tx=0x7fd408019630 comp rx=0 tx=0).stop 2023-10-25T00:12:15.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd415a64640 1 -- 172.21.15.107:0/4207467610 shutdown_connections 2023-10-25T00:12:15.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd415a64640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd41013bb00 0x7fd410154b80 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd415a64640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd3f0065e60 0x7fd3f0068320 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd415a64640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd4100aa330 0x7fd4100c32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd415a64640 1 --2- 172.21.15.107:0/4207467610 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd4100a9450 0x7fd4101eb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.396+0000 7fd415a64640 1 -- 172.21.15.107:0/4207467610 >> 172.21.15.107:0/4207467610 conn(0x7fd410070f60 msgr2=0x7fd410138c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:15.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.400+0000 7fd415a64640 1 -- 172.21.15.107:0/4207467610 shutdown_connections 2023-10-25T00:12:15.398 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.400+0000 7fd415a64640 1 -- 172.21.15.107:0/4207467610 wait complete. 2023-10-25T00:12:15.413 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2023-10-25T00:12:15.414 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:12:15.414 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:12:15.424 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:12:15.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:15.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.592+0000 7f4786523640 1 Processor -- start 2023-10-25T00:12:15.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.592+0000 7f4786523640 1 -- start start 2023-10-25T00:12:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.592+0000 7f4786523640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f47800720c0 0x7f47800c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.592+0000 7f4786523640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4780157e50 0x7f47801582a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.592+0000 7f4786523640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4780154860 con 0x7f478013dc20 2023-10-25T00:12:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.592+0000 7f4786523640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47801587e0 con 0x7f47800720c0 2023-10-25T00:12:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.592+0000 7f4786523640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47800c1f10 con 0x7f4780157e50 2023-10-25T00:12:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.592+0000 7f477f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f47800720c0 0x7f47800c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.592+0000 7f4784ac0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4780157e50 0x7f47801582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.592+0000 7f477ffff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f478013dc20 0x7f478006df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:58036/0 (socket says 172.21.15.107:58036) 2023-10-25T00:12:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f477ffff640 1 -- 172.21.15.107:0/3306729749 learned_addr learned my addr 172.21.15.107:0/3306729749 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:12:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f477f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f47800720c0 0x7f47800c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:43218/0 (socket says 172.21.15.107:43218) 2023-10-25T00:12:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f477f7fe640 1 -- 172.21.15.107:0/3306729749 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4780157e50 msgr2=0x7f47801582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f477f7fe640 1 --2- 172.21.15.107:0/3306729749 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4780157e50 0x7f47801582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f477f7fe640 1 -- 172.21.15.107:0/3306729749 >> v1:172.21.15.110:6789/0 conn(0x7f478013dc20 legacy=0x7f478006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f477f7fe640 1 -- 172.21.15.107:0/3306729749 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4780154b20 con 0x7f47800720c0 2023-10-25T00:12:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f477f7fe640 1 --2- 172.21.15.107:0/3306729749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47800720c0 0x7f47800c18a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f47740097c0 tx=0x7f477401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89e822e526f79e32 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f477effd640 1 -- 172.21.15.107:0/3306729749 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740ab600 con 0x7f47800720c0 2023-10-25T00:12:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f477effd640 1 -- 172.21.15.107:0/3306729749 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47740a7050 con 0x7f47800720c0 2023-10-25T00:12:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f477effd640 1 -- 172.21.15.107:0/3306729749 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740ab980 con 0x7f47800720c0 2023-10-25T00:12:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f4786523640 1 -- 172.21.15.107:0/3306729749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47800720c0 msgr2=0x7f47800c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f4786523640 1 --2- 172.21.15.107:0/3306729749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47800720c0 0x7f47800c18a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f47740097c0 tx=0x7f477401c360 comp rx=0 tx=0).stop 2023-10-25T00:12:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f4786523640 1 -- 172.21.15.107:0/3306729749 shutdown_connections 2023-10-25T00:12:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f4786523640 1 --2- 172.21.15.107:0/3306729749 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4780157e50 0x7f47801582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f4786523640 1 --2- 172.21.15.107:0/3306729749 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f47800720c0 0x7f47800c18a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f4786523640 1 -- 172.21.15.107:0/3306729749 >> 172.21.15.107:0/3306729749 conn(0x7f47800c39a0 msgr2=0x7f47800c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.596+0000 7f4786523640 1 -- 172.21.15.107:0/3306729749 shutdown_connections 2023-10-25T00:12:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4786523640 1 -- 172.21.15.107:0/3306729749 wait complete. 2023-10-25T00:12:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4786523640 1 Processor -- start 2023-10-25T00:12:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4786523640 1 -- start start 2023-10-25T00:12:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4786523640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47800720c0 0x7f4780157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4786523640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f478013dc20 0x7f47801efd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4786523640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4780157e50 0x7f47801edee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4786523640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4780156a90 con 0x7f478013dc20 2023-10-25T00:12:15.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4786523640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47801eda90 con 0x7f4780157e50 2023-10-25T00:12:15.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4786523640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4780157a50 con 0x7f47800720c0 2023-10-25T00:12:15.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f478013dc20 0x7f47801efd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f478013dc20 0x7f47801efd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51726/0 (socket says 172.21.15.107:51726) 2023-10-25T00:12:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477f7fe640 1 -- 172.21.15.107:0/3532748101 learned_addr learned my addr 172.21.15.107:0/3532748101 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477ffff640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47800720c0 0x7f4780157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4784ac0640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4780157e50 0x7f47801edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477f7fe640 1 -- 172.21.15.107:0/3532748101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47800720c0 msgr2=0x7f4780157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477f7fe640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47800720c0 0x7f4780157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477f7fe640 1 -- 172.21.15.107:0/3532748101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4780157e50 msgr2=0x7f47801edee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:15.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477f7fe640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4780157e50 0x7f47801edee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477f7fe640 1 -- 172.21.15.107:0/3532748101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4774009420 con 0x7f478013dc20 2023-10-25T00:12:15.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477f7fe640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f478013dc20 0x7f47801efd80 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f4774019f90 tx=0x7f4774019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:15.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477cff9640 1 -- 172.21.15.107:0/3532748101 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774006e20 con 0x7f478013dc20 2023-10-25T00:12:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477cff9640 1 -- 172.21.15.107:0/3532748101 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47740b4400 con 0x7f478013dc20 2023-10-25T00:12:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f477cff9640 1 -- 172.21.15.107:0/3532748101 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47740b47b0 con 0x7f478013dc20 2023-10-25T00:12:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4786523640 1 -- 172.21.15.107:0/3532748101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47801f0810 con 0x7f478013dc20 2023-10-25T00:12:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.600+0000 7f4786523640 1 -- 172.21.15.107:0/3532748101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47801567b0 con 0x7f478013dc20 2023-10-25T00:12:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.604+0000 7f4786523640 1 -- 172.21.15.107:0/3532748101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47801edc00 con 0x7f478013dc20 2023-10-25T00:12:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.604+0000 7f477cff9640 1 -- 172.21.15.107:0/3532748101 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f47740b2040 con 0x7f478013dc20 2023-10-25T00:12:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.604+0000 7f477cff9640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f476c065e60 0x7f476c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.604+0000 7f477cff9640 1 -- 172.21.15.107:0/3532748101 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7f477411e2b0 con 0x7f478013dc20 2023-10-25T00:12:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.604+0000 7f477ffff640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f476c065e60 0x7f476c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.604+0000 7f477ffff640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f476c065e60 0x7f476c068320 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4770005890 tx=0x7f4770009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.608+0000 7f477cff9640 1 -- 172.21.15.107:0/3532748101 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f47801edc00 con 0x7f478013dc20 2023-10-25T00:12:15.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.832+0000 7f4786523640 1 -- 172.21.15.107:0/3532748101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47801edc00 con 0x7f478013dc20 2023-10-25T00:12:15.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f477cff9640 1 -- 172.21.15.107:0/3532748101 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f47801edc00 con 0x7f478013dc20 2023-10-25T00:12:15.832 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:15.833 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:15.833 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 165 2023-10-25T00:12:15.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 -- 172.21.15.107:0/3532748101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f476c065e60 msgr2=0x7f476c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:15.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f476c065e60 0x7f476c068320 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4770005890 tx=0x7f4770009040 comp rx=0 tx=0).stop 2023-10-25T00:12:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 -- 172.21.15.107:0/3532748101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f478013dc20 msgr2=0x7f47801efd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f478013dc20 0x7f47801efd80 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f4774019f90 tx=0x7f4774019100 comp rx=0 tx=0).stop 2023-10-25T00:12:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 -- 172.21.15.107:0/3532748101 shutdown_connections 2023-10-25T00:12:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4780157e50 0x7f47801edee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f478013dc20 0x7f47801efd80 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f476c065e60 0x7f476c068320 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 --2- 172.21.15.107:0/3532748101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f47800720c0 0x7f4780157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:15.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 -- 172.21.15.107:0/3532748101 >> 172.21.15.107:0/3532748101 conn(0x7f47800c39a0 msgr2=0x7f478013cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:15.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 -- 172.21.15.107:0/3532748101 shutdown_connections 2023-10-25T00:12:15.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:15.836+0000 7f4786523640 1 -- 172.21.15.107:0/3532748101 wait complete. 2023-10-25T00:12:15.854 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:12:17.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:18.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.008+0000 7fca1277d640 1 Processor -- start 2023-10-25T00:12:18.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.008+0000 7fca1277d640 1 -- start start 2023-10-25T00:12:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca1277d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 0x7fca0c158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:18.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca1277d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fca0c065bc0 0x7fca0c06a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca1277d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c06cea0 con 0x7fca0c0a94b0 2023-10-25T00:12:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca1277d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0c0ac650 con 0x7fca0c065bc0 2023-10-25T00:12:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca1277d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c0a9e00 con 0x7fca0c069d40 2023-10-25T00:12:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fca0c065bc0 0x7fca0c06a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:18.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fca0c065bc0 0x7fca0c06a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51728/0 (socket says 172.21.15.107:51728) 2023-10-25T00:12:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0b7fe640 1 -- 172.21.15.107:0/789458057 learned_addr learned my addr 172.21.15.107:0/789458057 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0bfff640 1 --2- 172.21.15.107:0/789458057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 0x7fca0c158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0b7fe640 1 -- 172.21.15.107:0/789458057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 msgr2=0x7fca0c158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:18.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/789458057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 0x7fca0c158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:18.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0b7fe640 1 -- 172.21.15.107:0/789458057 >> v1:172.21.15.110:6789/0 conn(0x7fca0c0a94b0 legacy=0x7fca0c0aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:18.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0b7fe640 1 -- 172.21.15.107:0/789458057 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c1584e0 con 0x7fca0c065bc0 2023-10-25T00:12:18.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0bfff640 1 --2- 172.21.15.107:0/789458057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 0x7fca0c158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:12:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/789458057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c065bc0 0x7fca0c06a120 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fca00009870 tx=0x7fca0001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba42bca1fdd43913 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0affd640 1 -- 172.21.15.107:0/789458057 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca000ac600 con 0x7fca0c065bc0 2023-10-25T00:12:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0affd640 1 -- 172.21.15.107:0/789458057 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca000a8050 con 0x7fca0c065bc0 2023-10-25T00:12:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca0affd640 1 -- 172.21.15.107:0/789458057 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca000ac980 con 0x7fca0c065bc0 2023-10-25T00:12:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.012+0000 7fca1277d640 1 -- 172.21.15.107:0/789458057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c065bc0 msgr2=0x7fca0c06a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 --2- 172.21.15.107:0/789458057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c065bc0 0x7fca0c06a120 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fca00009870 tx=0x7fca0001c4b0 comp rx=0 tx=0).stop 2023-10-25T00:12:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 -- 172.21.15.107:0/789458057 shutdown_connections 2023-10-25T00:12:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 --2- 172.21.15.107:0/789458057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c065bc0 0x7fca0c06a120 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 --2- 172.21.15.107:0/789458057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 0x7fca0c158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 -- 172.21.15.107:0/789458057 >> 172.21.15.107:0/789458057 conn(0x7fca0c0c65c0 msgr2=0x7fca0c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 -- 172.21.15.107:0/789458057 shutdown_connections 2023-10-25T00:12:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 -- 172.21.15.107:0/789458057 wait complete. 2023-10-25T00:12:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 Processor -- start 2023-10-25T00:12:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 -- start start 2023-10-25T00:12:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 0x7fca0c071110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c0a94b0 0x7fca0c1f0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c149310 0x7fca0c1f0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c14c2c0 con 0x7fca0c149310 2023-10-25T00:12:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c14cb10 con 0x7fca0c0a94b0 2023-10-25T00:12:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c0bb3c0 con 0x7fca0c069d40 2023-10-25T00:12:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca10d1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c149310 0x7fca0c1f0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca10d1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c149310 0x7fca0c1f0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51734/0 (socket says 172.21.15.107:51734) 2023-10-25T00:12:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca10d1a640 1 -- 172.21.15.107:0/4167139182 learned_addr learned my addr 172.21.15.107:0/4167139182 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca0bfff640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 0x7fca0c071110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c0a94b0 0x7fca0c1f0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca10d1a640 1 -- 172.21.15.107:0/4167139182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 msgr2=0x7fca0c071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca10d1a640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 0x7fca0c071110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca10d1a640 1 -- 172.21.15.107:0/4167139182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c0a94b0 msgr2=0x7fca0c1f0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca10d1a640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c0a94b0 0x7fca0c1f0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca10d1a640 1 -- 172.21.15.107:0/4167139182 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca000094d0 con 0x7fca0c149310 2023-10-25T00:12:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca0bfff640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 0x7fca0c071110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:12:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca0b7fe640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c0a94b0 0x7fca0c1f0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:12:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca10d1a640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c149310 0x7fca0c1f0ea0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fca0400c730 tx=0x7fca0400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca08ff9640 1 -- 172.21.15.107:0/4167139182 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca04003b80 con 0x7fca0c149310 2023-10-25T00:12:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca08ff9640 1 -- 172.21.15.107:0/4167139182 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca04003ce0 con 0x7fca0c149310 2023-10-25T00:12:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 -- 172.21.15.107:0/4167139182 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c0bb6a0 con 0x7fca0c149310 2023-10-25T00:12:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca08ff9640 1 -- 172.21.15.107:0/4167139182 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0400aa30 con 0x7fca0c149310 2023-10-25T00:12:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.016+0000 7fca1277d640 1 -- 172.21.15.107:0/4167139182 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c14c7a0 con 0x7fca0c149310 2023-10-25T00:12:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.020+0000 7fca1277d640 1 -- 172.21.15.107:0/4167139182 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c14ccc0 con 0x7fca0c149310 2023-10-25T00:12:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.020+0000 7fca08ff9640 1 -- 172.21.15.107:0/4167139182 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fca0400ab90 con 0x7fca0c149310 2023-10-25T00:12:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.020+0000 7fca08ff9640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9f8065e60 0x7fc9f8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.020+0000 7fca0bfff640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9f8065e60 0x7fc9f8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.020+0000 7fca08ff9640 1 -- 172.21.15.107:0/4167139182 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7fca04084390 con 0x7fca0c149310 2023-10-25T00:12:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.020+0000 7fca0bfff640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9f8065e60 0x7fc9f8068320 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc006fd0 tx=0x7fc9fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.024+0000 7fca08ff9640 1 -- 172.21.15.107:0/4167139182 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fca0c14ccc0 con 0x7fca0c149310 2023-10-25T00:12:18.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.272+0000 7fca1277d640 1 -- 172.21.15.107:0/4167139182 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca0c14ccc0 con 0x7fca0c149310 2023-10-25T00:12:18.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.272+0000 7fca08ff9640 1 -- 172.21.15.107:0/4167139182 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5776 (secure 0 0 0) 0x7fca0c14ccc0 con 0x7fca0c149310 2023-10-25T00:12:18.272 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:18.273 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:18.273 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 165 2023-10-25T00:12:18.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 -- 172.21.15.107:0/4167139182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9f8065e60 msgr2=0x7fc9f8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:18.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9f8065e60 0x7fc9f8068320 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc006fd0 tx=0x7fc9fc008040 comp rx=0 tx=0).stop 2023-10-25T00:12:18.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 -- 172.21.15.107:0/4167139182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c149310 msgr2=0x7fca0c1f0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:18.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c149310 0x7fca0c1f0ea0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fca0400c730 tx=0x7fca0400cbf0 comp rx=0 tx=0).stop 2023-10-25T00:12:18.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 -- 172.21.15.107:0/4167139182 shutdown_connections 2023-10-25T00:12:18.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca0c149310 0x7fca0c1f0ea0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:18.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca0c0a94b0 0x7fca0c1f0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:18.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc9f8065e60 0x7fc9f8068320 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:18.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 --2- 172.21.15.107:0/4167139182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fca0c069d40 0x7fca0c071110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:18.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 -- 172.21.15.107:0/4167139182 >> 172.21.15.107:0/4167139182 conn(0x7fca0c0c65c0 msgr2=0x7fca0c0c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:18.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 -- 172.21.15.107:0/4167139182 shutdown_connections 2023-10-25T00:12:18.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:18.276+0000 7fca1277d640 1 -- 172.21.15.107:0/4167139182 wait complete. 2023-10-25T00:12:18.296 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:12:20.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:20.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.424+0000 7f690acdd640 1 Processor -- start 2023-10-25T00:12:20.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.424+0000 7f690acdd640 1 -- start start 2023-10-25T00:12:20.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f690acdd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6904156770 0x7f69040ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:20.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f690acdd640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69040c1ca0 con 0x7f69040c70e0 2023-10-25T00:12:20.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f690acdd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69040c10c0 con 0x7f69040be6b0 2023-10-25T00:12:20.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f690acdd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69040c27e0 con 0x7f6904156770 2023-10-25T00:12:20.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f6903fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f69040c70e0 0x7f69040bdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43642/0 (socket says 172.21.15.107:43642) 2023-10-25T00:12:20.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f6903fff640 1 -- 172.21.15.107:0/2746385014 learned_addr learned my addr 172.21.15.107:0/2746385014 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:12:20.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f6908a79640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6904156770 0x7f69040ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:20.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129482094 0 0) 0x7f69040c1ca0 con 0x7f69040c70e0 2023-10-25T00:12:20.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f4003530 con 0x7f69040c70e0 2023-10-25T00:12:20.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1657937997 0 0) 0x7f68f4003530 con 0x7f69040c70e0 2023-10-25T00:12:20.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6904156770 msgr2=0x7f69040ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:20.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6904156770 0x7f69040ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:20.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 >> v1:172.21.15.110:6789/0 conn(0x7f69040be6b0 legacy=0x7f690415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:20.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69040c13a0 con 0x7f69040c70e0 2023-10-25T00:12:20.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f68f0001680 con 0x7f69040c70e0 2023-10-25T00:12:20.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 >> v1:172.21.15.107:6789/0 conn(0x7f69040c70e0 legacy=0x7f69040bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:20.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40039b0 0x7f68f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:20.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6904156770 0x7f68f40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:20.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 0x7f68f4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:20.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f400bb20 con 0x7f68f4004d10 2023-10-25T00:12:20.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f400bc90 con 0x7f68f40039b0 2023-10-25T00:12:20.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f400be50 con 0x7f6904156770 2023-10-25T00:12:20.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f690927a640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 0x7f68f4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:20.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f6908a79640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40039b0 0x7f68f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:20.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f6903fff640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6904156770 0x7f68f40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:20.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f690927a640 1 -- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6904156770 msgr2=0x7f68f40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:20.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f690927a640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6904156770 0x7f68f40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:20.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f690927a640 1 -- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40039b0 msgr2=0x7f68f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:20.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f690927a640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40039b0 0x7f68f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:20.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f690927a640 1 -- 172.21.15.107:0/2746385014 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f4003530 con 0x7f68f4004d10 2023-10-25T00:12:20.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.428+0000 7f6908a79640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f68f40039b0 0x7f68f4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:12:20.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f690927a640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 0x7f68f4005190 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f69040beb00 tx=0x7f68fc01c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50cf2bb15f421e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:20.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc003b80 con 0x7f68f4004d10 2023-10-25T00:12:20.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68fc003ce0 con 0x7f68f4004d10 2023-10-25T00:12:20.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f69037fe640 1 -- 172.21.15.107:0/2746385014 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc0052c0 con 0x7f68f4004d10 2023-10-25T00:12:20.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f690acdd640 1 -- 172.21.15.107:0/2746385014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 msgr2=0x7f68f4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:20.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f690acdd640 1 --2- 172.21.15.107:0/2746385014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 0x7f68f4005190 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f69040beb00 tx=0x7f68fc01c6d0 comp rx=0 tx=0).stop 2023-10-25T00:12:20.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f690927a640 1 -- 172.21.15.107:0/2746385014 reap_dead start 2023-10-25T00:12:20.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f690acdd640 1 -- 172.21.15.107:0/2746385014 shutdown_connections 2023-10-25T00:12:20.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f690acdd640 1 -- 172.21.15.107:0/2746385014 >> 172.21.15.107:0/2746385014 conn(0x7f69040c65c0 msgr2=0x7f69040c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:20.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f690acdd640 1 -- 172.21.15.107:0/2746385014 shutdown_connections 2023-10-25T00:12:20.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f690acdd640 1 -- 172.21.15.107:0/2746385014 wait complete. 2023-10-25T00:12:20.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.432+0000 7f690acdd640 1 Processor -- start 2023-10-25T00:12:20.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690acdd640 1 -- start start 2023-10-25T00:12:20.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690acdd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 0x7f6904152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:20.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690acdd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f690406f520 0x7f69041f0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:20.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690acdd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69041f0790 0x7f69040aa210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:20.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690acdd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904070c80 con 0x7f68f4004d10 2023-10-25T00:12:20.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690acdd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69040c60d0 con 0x7f690406f520 2023-10-25T00:12:20.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690acdd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69040705a0 con 0x7f69041f0790 2023-10-25T00:12:20.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f6908a79640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 0x7f6904152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:20.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f6908a79640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 0x7f6904152300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51740/0 (socket says 172.21.15.107:51740) 2023-10-25T00:12:20.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f6908a79640 1 -- 172.21.15.107:0/1806997775 learned_addr learned my addr 172.21.15.107:0/1806997775 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:20.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f6903fff640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f690406f520 0x7f69041f0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:20.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690927a640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69041f0790 0x7f69040aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:20.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f6908a79640 1 -- 172.21.15.107:0/1806997775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69041f0790 msgr2=0x7f69040aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:20.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f6908a79640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69041f0790 0x7f69040aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:20.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f6908a79640 1 -- 172.21.15.107:0/1806997775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f690406f520 msgr2=0x7f69041f0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:20.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f6908a79640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f690406f520 0x7f69041f0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:20.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f6908a79640 1 -- 172.21.15.107:0/1806997775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69040702c0 con 0x7f68f4004d10 2023-10-25T00:12:20.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f6903fff640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f690406f520 0x7f69041f0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:12:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690927a640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69041f0790 0x7f69040aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:12:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f6908a79640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 0x7f6904152300 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f68f8009fd0 tx=0x7f68f8011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f69017fa640 1 -- 172.21.15.107:0/1806997775 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f8004220 con 0x7f68f4004d10 2023-10-25T00:12:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f69017fa640 1 -- 172.21.15.107:0/1806997775 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f8004380 con 0x7f68f4004d10 2023-10-25T00:12:20.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690acdd640 1 -- 172.21.15.107:0/1806997775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69040c5a80 con 0x7f68f4004d10 2023-10-25T00:12:20.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f69017fa640 1 -- 172.21.15.107:0/1806997775 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f800f9e0 con 0x7f68f4004d10 2023-10-25T00:12:20.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690acdd640 1 -- 172.21.15.107:0/1806997775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6904152840 con 0x7f68f4004d10 2023-10-25T00:12:20.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f690acdd640 1 -- 172.21.15.107:0/1806997775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69040c58e0 con 0x7f68f4004d10 2023-10-25T00:12:20.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.436+0000 7f69017fa640 1 -- 172.21.15.107:0/1806997775 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f68f800fb40 con 0x7f68f4004d10 2023-10-25T00:12:20.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.440+0000 7f69017fa640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f68e8065e60 0x7f68e8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:20.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.440+0000 7f69017fa640 1 -- 172.21.15.107:0/1806997775 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7f68f8084bd0 con 0x7f68f4004d10 2023-10-25T00:12:20.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.440+0000 7f6903fff640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f68e8065e60 0x7f68e8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:20.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.440+0000 7f6903fff640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f68e8065e60 0x7f68e8068320 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f68f00010f0 tx=0x7f68f0001160 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:20.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.444+0000 7f69017fa640 1 -- 172.21.15.107:0/1806997775 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f69040c58e0 con 0x7f68f4004d10 2023-10-25T00:12:20.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.708+0000 7f690acdd640 1 -- 172.21.15.107:0/1806997775 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69040c58e0 con 0x7f68f4004d10 2023-10-25T00:12:20.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.708+0000 7f69017fa640 1 -- 172.21.15.107:0/1806997775 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f69040c58e0 con 0x7f68f4004d10 2023-10-25T00:12:20.706 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:20.707 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:20.707 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 165 2023-10-25T00:12:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 -- 172.21.15.107:0/1806997775 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f68e8065e60 msgr2=0x7f68e8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f68e8065e60 0x7f68e8068320 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f68f00010f0 tx=0x7f68f0001160 comp rx=0 tx=0).stop 2023-10-25T00:12:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 -- 172.21.15.107:0/1806997775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 msgr2=0x7f6904152300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:20.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 0x7f6904152300 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f68f8009fd0 tx=0x7f68f8011b30 comp rx=0 tx=0).stop 2023-10-25T00:12:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 -- 172.21.15.107:0/1806997775 shutdown_connections 2023-10-25T00:12:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f68e8065e60 0x7f68e8068320 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69041f0790 0x7f69040aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f690406f520 0x7f69041f0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:20.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 --2- 172.21.15.107:0/1806997775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f68f4004d10 0x7f6904152300 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:20.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 -- 172.21.15.107:0/1806997775 >> 172.21.15.107:0/1806997775 conn(0x7f69040c65c0 msgr2=0x7f69040bfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:20.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 -- 172.21.15.107:0/1806997775 shutdown_connections 2023-10-25T00:12:20.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:20.712+0000 7f690acdd640 1 -- 172.21.15.107:0/1806997775 wait complete. 2023-10-25T00:12:20.727 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:12:22.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ec1bd8640 1 Processor -- start 2023-10-25T00:12:22.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ec1bd8640 1 -- start start 2023-10-25T00:12:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ec1bd8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1ebc152040 0x7f1ebc0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ec1bd8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1ebc06a4b0 0x7f1ebc0ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ec1bd8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc155680 con 0x7f1ebc0c8100 2023-10-25T00:12:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ec1bd8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc154a70 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ec1bd8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ebc154d50 con 0x7f1ebc152040 2023-10-25T00:12:22.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ebbfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1ebc06a4b0 0x7f1ebc0ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:22.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ebaffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1ebc0c8100 0x7f1ebc0abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52122/0 (socket says 172.21.15.107:52122) 2023-10-25T00:12:22.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ebaffd640 1 -- 172.21.15.107:0/2492647284 learned_addr learned my addr 172.21.15.107:0/2492647284 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:12:22.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ebb7fe640 1 --2- 172.21.15.107:0/2492647284 >> v2:172.21.15.110:3300/0 conn(0x7f1ebc152040 0x7f1ebc0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:22.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ebbfff640 1 -- 172.21.15.107:0/2492647284 >> v2:172.21.15.110:3300/0 conn(0x7f1ebc152040 msgr2=0x7f1ebc0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ebbfff640 1 --2- 172.21.15.107:0/2492647284 >> v2:172.21.15.110:3300/0 conn(0x7f1ebc152040 0x7f1ebc0c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ebbfff640 1 -- 172.21.15.107:0/2492647284 >> v1:172.21.15.107:6789/0 conn(0x7f1ebc0c8100 legacy=0x7f1ebc0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.876+0000 7f1ebbfff640 1 -- 172.21.15.107:0/2492647284 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc155030 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1ebbfff640 1 --2- 172.21.15.107:0/2492647284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ebc06a4b0 0x7f1ebc0ac410 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc067ba0 tx=0x7f1eb401c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232f71a972e0fd3d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:22.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1eba7fc640 1 -- 172.21.15.107:0/2492647284 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb40a6670 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1eba7fc640 1 -- 172.21.15.107:0/2492647284 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb40a67d0 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1eba7fc640 1 -- 172.21.15.107:0/2492647284 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb40a6b60 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/2492647284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ebc06a4b0 msgr2=0x7f1ebc0ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1ec1bd8640 1 --2- 172.21.15.107:0/2492647284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ebc06a4b0 0x7f1ebc0ac410 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc067ba0 tx=0x7f1eb401c710 comp rx=0 tx=0).stop 2023-10-25T00:12:22.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/2492647284 shutdown_connections 2023-10-25T00:12:22.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1ec1bd8640 1 --2- 172.21.15.107:0/2492647284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ebc06a4b0 0x7f1ebc0ac410 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:22.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1ec1bd8640 1 --2- 172.21.15.107:0/2492647284 >> v2:172.21.15.110:3300/0 conn(0x7f1ebc152040 0x7f1ebc0c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:22.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/2492647284 >> 172.21.15.107:0/2492647284 conn(0x7f1ebc0ca200 msgr2=0x7f1ebc0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:22.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/2492647284 shutdown_connections 2023-10-25T00:12:22.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/2492647284 wait complete. 2023-10-25T00:12:22.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1ec1bd8640 1 Processor -- start 2023-10-25T00:12:22.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.880+0000 7f1ec1bd8640 1 -- start start 2023-10-25T00:12:22.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ec1bd8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ebc06a4b0 0x7f1ebc147c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:22.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ec1bd8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ebc0c8100 0x7f1ebc145cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:22.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ec1bd8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ebc152040 0x7f1ebc0aa820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:22.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ec1bd8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0c9fc0 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ec1bd8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc0a9d80 con 0x7f1ebc0c8100 2023-10-25T00:12:22.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ec1bd8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc1ed810 con 0x7f1ebc152040 2023-10-25T00:12:22.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ebc06a4b0 0x7f1ebc147c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:22.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ebc06a4b0 0x7f1ebc147c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35178/0 (socket says 172.21.15.107:35178) 2023-10-25T00:12:22.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebb7fe640 1 -- 172.21.15.107:0/1742284970 learned_addr learned my addr 172.21.15.107:0/1742284970 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:22.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebaffd640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ebc0c8100 0x7f1ebc145cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:22.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebbfff640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ebc152040 0x7f1ebc0aa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:22.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebb7fe640 1 -- 172.21.15.107:0/1742284970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ebc152040 msgr2=0x7f1ebc0aa820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:22.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebb7fe640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ebc152040 0x7f1ebc0aa820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:22.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebb7fe640 1 -- 172.21.15.107:0/1742284970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ebc0c8100 msgr2=0x7f1ebc145cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:22.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebb7fe640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ebc0c8100 0x7f1ebc145cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:22.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebb7fe640 1 -- 172.21.15.107:0/1742284970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb4009a70 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebbfff640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ebc152040 0x7f1ebc0aa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:12:22.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebb7fe640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ebc06a4b0 0x7f1ebc147c00 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00d800 tx=0x7f1eac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:22.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ea3fff640 1 -- 172.21.15.107:0/1742284970 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00b6c0 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ea3fff640 1 -- 172.21.15.107:0/1742284970 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac00b860 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ea3fff640 1 -- 172.21.15.107:0/1742284970 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00bc10 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/1742284970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc0aa020 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/1742284970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc1edaf0 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/1742284970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc0746e0 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ea3fff640 1 -- 172.21.15.107:0/1742284970 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1eac01e460 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ea3fff640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ea8065e60 0x7f1ea8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ea3fff640 1 -- 172.21.15.107:0/1742284970 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7f1eac084be0 con 0x7f1ebc06a4b0 2023-10-25T00:12:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.884+0000 7f1ebaffd640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ea8065e60 0x7f1ea8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:22.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.888+0000 7f1ebaffd640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ea8065e60 0x7f1ea8068320 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc1530b0 tx=0x7f1eb0007c30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:22.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:22.896+0000 7f1ea3fff640 1 -- 172.21.15.107:0/1742284970 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1eac051480 con 0x7f1ebc06a4b0 2023-10-25T00:12:23.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.120+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/1742284970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ebc152b20 con 0x7f1ebc06a4b0 2023-10-25T00:12:23.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.120+0000 7f1ea3fff640 1 -- 172.21.15.107:0/1742284970 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f1eac054aa0 con 0x7f1ebc06a4b0 2023-10-25T00:12:23.120 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:23.120 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:23.120 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 165 2023-10-25T00:12:23.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/1742284970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ea8065e60 msgr2=0x7f1ea8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:23.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ea8065e60 0x7f1ea8068320 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc1530b0 tx=0x7f1eb0007c30 comp rx=0 tx=0).stop 2023-10-25T00:12:23.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/1742284970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ebc06a4b0 msgr2=0x7f1ebc147c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:23.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ebc06a4b0 0x7f1ebc147c00 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f1eac00d800 tx=0x7f1eac00dcc0 comp rx=0 tx=0).stop 2023-10-25T00:12:23.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/1742284970 shutdown_connections 2023-10-25T00:12:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ea8065e60 0x7f1ea8068320 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1ebc152040 0x7f1ebc0aa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ebc0c8100 0x7f1ebc145cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 --2- 172.21.15.107:0/1742284970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ebc06a4b0 0x7f1ebc147c00 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/1742284970 >> 172.21.15.107:0/1742284970 conn(0x7f1ebc0ca200 msgr2=0x7f1ebc0c4850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/1742284970 shutdown_connections 2023-10-25T00:12:23.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:23.124+0000 7f1ec1bd8640 1 -- 172.21.15.107:0/1742284970 wait complete. 2023-10-25T00:12:23.139 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:12:25.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:25.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f1596c31640 1 Processor -- start 2023-10-25T00:12:25.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f1596c31640 1 -- start start 2023-10-25T00:12:25.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f1596c31640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f159013dc20 0x7f15900c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f1596c31640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1590157e50 0x7f1590158260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f1596c31640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15901587a0 con 0x7f15900720c0 2023-10-25T00:12:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f1596c31640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1590154860 con 0x7f159013dc20 2023-10-25T00:12:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f1596c31640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15900c1f10 con 0x7f1590157e50 2023-10-25T00:12:25.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f15949cd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f159013dc20 0x7f15900c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:25.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f15949cd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f159013dc20 0x7f15900c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35184/0 (socket says 172.21.15.107:35184) 2023-10-25T00:12:25.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f15949cd640 1 -- 172.21.15.107:0/3020234449 learned_addr learned my addr 172.21.15.107:0/3020234449 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:25.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f15951ce640 1 --2- 172.21.15.107:0/3020234449 >> v2:172.21.15.110:3300/0 conn(0x7f1590157e50 0x7f1590158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:25.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f15949cd640 1 -- 172.21.15.107:0/3020234449 >> v2:172.21.15.110:3300/0 conn(0x7f1590157e50 msgr2=0x7f1590158260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:25.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f15949cd640 1 --2- 172.21.15.107:0/3020234449 >> v2:172.21.15.110:3300/0 conn(0x7f1590157e50 0x7f1590158260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f15949cd640 1 -- 172.21.15.107:0/3020234449 >> v1:172.21.15.110:6789/0 conn(0x7f15900720c0 legacy=0x7f159006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:25.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.276+0000 7f15949cd640 1 -- 172.21.15.107:0/3020234449 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1590154b20 con 0x7f159013dc20 2023-10-25T00:12:25.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f15949cd640 1 --2- 172.21.15.107:0/3020234449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f159013dc20 0x7f15900c18a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f15900c05a0 tx=0x7f158401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3710551ac6462696 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:25.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f158f7fe640 1 -- 172.21.15.107:0/3020234449 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15840ab600 con 0x7f159013dc20 2023-10-25T00:12:25.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f158f7fe640 1 -- 172.21.15.107:0/3020234449 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15840a7050 con 0x7f159013dc20 2023-10-25T00:12:25.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f158f7fe640 1 -- 172.21.15.107:0/3020234449 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15840ab980 con 0x7f159013dc20 2023-10-25T00:12:25.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f1596c31640 1 -- 172.21.15.107:0/3020234449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f159013dc20 msgr2=0x7f15900c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:25.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f1596c31640 1 --2- 172.21.15.107:0/3020234449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f159013dc20 0x7f15900c18a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f15900c05a0 tx=0x7f158401c3a0 comp rx=0 tx=0).stop 2023-10-25T00:12:25.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f1596c31640 1 -- 172.21.15.107:0/3020234449 shutdown_connections 2023-10-25T00:12:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f1596c31640 1 --2- 172.21.15.107:0/3020234449 >> v2:172.21.15.110:3300/0 conn(0x7f1590157e50 0x7f1590158260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f1596c31640 1 --2- 172.21.15.107:0/3020234449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f159013dc20 0x7f15900c18a0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f1596c31640 1 -- 172.21.15.107:0/3020234449 >> 172.21.15.107:0/3020234449 conn(0x7f15900c39a0 msgr2=0x7f15900c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f1596c31640 1 -- 172.21.15.107:0/3020234449 shutdown_connections 2023-10-25T00:12:25.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.280+0000 7f1596c31640 1 -- 172.21.15.107:0/3020234449 wait complete. 2023-10-25T00:12:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f1596c31640 1 Processor -- start 2023-10-25T00:12:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f1596c31640 1 -- start start 2023-10-25T00:12:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f1596c31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15900720c0 0x7f15901356a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f1596c31640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f159013dc20 0x7f159013a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:25.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f1596c31640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1590157e50 0x7f1590133b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:25.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f1596c31640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15900c6460 con 0x7f15900720c0 2023-10-25T00:12:25.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f1596c31640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15900c58c0 con 0x7f159013dc20 2023-10-25T00:12:25.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f1596c31640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15901361b0 con 0x7f1590157e50 2023-10-25T00:12:25.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f15949cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15900720c0 0x7f15901356a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:25.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f15949cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15900720c0 0x7f15901356a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35194/0 (socket says 172.21.15.107:35194) 2023-10-25T00:12:25.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f15949cd640 1 -- 172.21.15.107:0/827061300 learned_addr learned my addr 172.21.15.107:0/827061300 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:25.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f158ffff640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f159013dc20 0x7f159013a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:25.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f15951ce640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1590157e50 0x7f1590133b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:25.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f15949cd640 1 -- 172.21.15.107:0/827061300 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1590157e50 msgr2=0x7f1590133b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:25.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f15949cd640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1590157e50 0x7f1590133b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:25.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f15949cd640 1 -- 172.21.15.107:0/827061300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f159013dc20 msgr2=0x7f159013a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f15949cd640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f159013dc20 0x7f159013a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f15949cd640 1 -- 172.21.15.107:0/827061300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15840094d0 con 0x7f15900720c0 2023-10-25T00:12:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f15949cd640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15900720c0 0x7f15901356a0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f1584000c10 tx=0x7f1584019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f158d7fa640 1 -- 172.21.15.107:0/827061300 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15840b6650 con 0x7f15900720c0 2023-10-25T00:12:25.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f158d7fa640 1 -- 172.21.15.107:0/827061300 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15840b67d0 con 0x7f15900720c0 2023-10-25T00:12:25.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f1596c31640 1 -- 172.21.15.107:0/827061300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15900c5b30 con 0x7f15900720c0 2023-10-25T00:12:25.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f158d7fa640 1 -- 172.21.15.107:0/827061300 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15840b6b80 con 0x7f15900720c0 2023-10-25T00:12:25.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f1596c31640 1 -- 172.21.15.107:0/827061300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15900c60f0 con 0x7f15900720c0 2023-10-25T00:12:25.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.284+0000 7f1596c31640 1 -- 172.21.15.107:0/827061300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15900746e0 con 0x7f15900720c0 2023-10-25T00:12:25.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.288+0000 7f158d7fa640 1 -- 172.21.15.107:0/827061300 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f15840b2040 con 0x7f15900720c0 2023-10-25T00:12:25.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.288+0000 7f158d7fa640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1574065e90 0x7f1574068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:25.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.288+0000 7f158d7fa640 1 -- 172.21.15.107:0/827061300 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7f158411ea00 con 0x7f15900720c0 2023-10-25T00:12:25.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.288+0000 7f158ffff640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1574065e90 0x7f1574068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.288+0000 7f158ffff640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1574065e90 0x7f1574068350 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1590158f50 tx=0x7f157c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.292+0000 7f158d7fa640 1 -- 172.21.15.107:0/827061300 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f15840eb2a0 con 0x7f15900720c0 2023-10-25T00:12:25.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.524+0000 7f1596c31640 1 -- 172.21.15.107:0/827061300 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15900c1190 con 0x7f15900720c0 2023-10-25T00:12:25.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f158d7fa640 1 -- 172.21.15.107:0/827061300 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f15840ee8c0 con 0x7f15900720c0 2023-10-25T00:12:25.525 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:25.525 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:25.525 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 165 2023-10-25T00:12:25.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 -- 172.21.15.107:0/827061300 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1574065e90 msgr2=0x7f1574068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:25.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1574065e90 0x7f1574068350 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1590158f50 tx=0x7f157c009040 comp rx=0 tx=0).stop 2023-10-25T00:12:25.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 -- 172.21.15.107:0/827061300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15900720c0 msgr2=0x7f15901356a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:25.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15900720c0 0x7f15901356a0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f1584000c10 tx=0x7f1584019140 comp rx=0 tx=0).stop 2023-10-25T00:12:25.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 -- 172.21.15.107:0/827061300 shutdown_connections 2023-10-25T00:12:25.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1574065e90 0x7f1574068350 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:25.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1590157e50 0x7f1590133b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:25.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f159013dc20 0x7f159013a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:25.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 --2- 172.21.15.107:0/827061300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15900720c0 0x7f15901356a0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:25.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 -- 172.21.15.107:0/827061300 >> 172.21.15.107:0/827061300 conn(0x7f15900c39a0 msgr2=0x7f159015e410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:25.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 -- 172.21.15.107:0/827061300 shutdown_connections 2023-10-25T00:12:25.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:25.528+0000 7f1596c31640 1 -- 172.21.15.107:0/827061300 wait complete. 2023-10-25T00:12:25.544 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:12:27.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:27.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.676+0000 7f3a18561640 1 Processor -- start 2023-10-25T00:12:27.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.676+0000 7f3a18561640 1 -- start start 2023-10-25T00:12:27.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a18561640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a10156770 0x7f3a100ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:27.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a18561640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3a100be6b0 0x7f3a1015d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:27.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a18561640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a100c1ca0 con 0x7f3a100c70e0 2023-10-25T00:12:27.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a18561640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a100c27e0 con 0x7f3a100be6b0 2023-10-25T00:12:27.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a18561640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100c10c0 con 0x7f3a10156770 2023-10-25T00:12:27.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a16afe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3a100be6b0 0x7f3a1015d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:27.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a16afe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3a100be6b0 0x7f3a1015d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35208/0 (socket says 172.21.15.107:35208) 2023-10-25T00:12:27.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a16afe640 1 -- 172.21.15.107:0/2372995170 learned_addr learned my addr 172.21.15.107:0/2372995170 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:27.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a162fd640 1 --2- 172.21.15.107:0/2372995170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a10156770 0x7f3a100ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:27.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a16afe640 1 -- 172.21.15.107:0/2372995170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a10156770 msgr2=0x7f3a100ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:27.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a16afe640 1 --2- 172.21.15.107:0/2372995170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a10156770 0x7f3a100ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:27.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a16afe640 1 -- 172.21.15.107:0/2372995170 >> v1:172.21.15.110:6789/0 conn(0x7f3a100c70e0 legacy=0x7f3a100bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:27.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a16afe640 1 -- 172.21.15.107:0/2372995170 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a100c13a0 con 0x7f3a100be6b0 2023-10-25T00:12:27.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a16afe640 1 --2- 172.21.15.107:0/2372995170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a100be6b0 0x7f3a1015d130 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009e50 tx=0x7f3a0801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b540fdc626a87458 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:27.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a152fb640 1 -- 172.21.15.107:0/2372995170 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a080a5740 con 0x7f3a100be6b0 2023-10-25T00:12:27.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a152fb640 1 -- 172.21.15.107:0/2372995170 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a080a58a0 con 0x7f3a100be6b0 2023-10-25T00:12:27.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.680+0000 7f3a152fb640 1 -- 172.21.15.107:0/2372995170 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a080a5c30 con 0x7f3a100be6b0 2023-10-25T00:12:27.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 -- 172.21.15.107:0/2372995170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a100be6b0 msgr2=0x7f3a1015d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:27.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 --2- 172.21.15.107:0/2372995170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a100be6b0 0x7f3a1015d130 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009e50 tx=0x7f3a0801c8c0 comp rx=0 tx=0).stop 2023-10-25T00:12:27.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 -- 172.21.15.107:0/2372995170 shutdown_connections 2023-10-25T00:12:27.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 --2- 172.21.15.107:0/2372995170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a100be6b0 0x7f3a1015d130 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:27.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 --2- 172.21.15.107:0/2372995170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a10156770 0x7f3a100ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:27.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 -- 172.21.15.107:0/2372995170 >> 172.21.15.107:0/2372995170 conn(0x7f3a100c65c0 msgr2=0x7f3a100c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:27.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 -- 172.21.15.107:0/2372995170 shutdown_connections 2023-10-25T00:12:27.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 -- 172.21.15.107:0/2372995170 wait complete. 2023-10-25T00:12:27.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 Processor -- start 2023-10-25T00:12:27.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 -- start start 2023-10-25T00:12:27.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a100be6b0 0x7f3a100c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:27.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a100c70e0 0x7f3a100a6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:27.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a10156770 0x7f3a100a7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:27.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100ba860 con 0x7f3a100c70e0 2023-10-25T00:12:27.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100c4020 con 0x7f3a10156770 2023-10-25T00:12:27.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a101eebd0 con 0x7f3a100be6b0 2023-10-25T00:12:27.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a15afc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a100c70e0 0x7f3a100a6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:27.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a15afc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a100c70e0 0x7f3a100a6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35216/0 (socket says 172.21.15.107:35216) 2023-10-25T00:12:27.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a15afc640 1 -- 172.21.15.107:0/3663328776 learned_addr learned my addr 172.21.15.107:0/3663328776 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:27.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a162fd640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a100be6b0 0x7f3a100c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a16afe640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a10156770 0x7f3a100a7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a15afc640 1 -- 172.21.15.107:0/3663328776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a100be6b0 msgr2=0x7f3a100c4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a15afc640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a100be6b0 0x7f3a100c4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a15afc640 1 -- 172.21.15.107:0/3663328776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a10156770 msgr2=0x7f3a100a7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a15afc640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a10156770 0x7f3a100a7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a15afc640 1 -- 172.21.15.107:0/3663328776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a08009b30 con 0x7f3a100c70e0 2023-10-25T00:12:27.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a16afe640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a10156770 0x7f3a100a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:12:27.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a15afc640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a100c70e0 0x7f3a100a6f00 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000c500 tx=0x7f3a0000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:27.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a06ffd640 1 -- 172.21.15.107:0/3663328776 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00005220 con 0x7f3a100c70e0 2023-10-25T00:12:27.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a06ffd640 1 -- 172.21.15.107:0/3663328776 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a00005380 con 0x7f3a100c70e0 2023-10-25T00:12:27.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 -- 172.21.15.107:0/3663328776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a101eeeb0 con 0x7f3a100c70e0 2023-10-25T00:12:27.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a06ffd640 1 -- 172.21.15.107:0/3663328776 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00007270 con 0x7f3a100c70e0 2023-10-25T00:12:27.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.684+0000 7f3a18561640 1 -- 172.21.15.107:0/3663328776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a10150760 con 0x7f3a100c70e0 2023-10-25T00:12:27.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.688+0000 7f3a18561640 1 -- 172.21.15.107:0/3663328776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a100746e0 con 0x7f3a100c70e0 2023-10-25T00:12:27.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.688+0000 7f3a06ffd640 1 -- 172.21.15.107:0/3663328776 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3a000073d0 con 0x7f3a100c70e0 2023-10-25T00:12:27.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.688+0000 7f3a06ffd640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39f4065e60 0x7f39f4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:27.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.688+0000 7f3a06ffd640 1 -- 172.21.15.107:0/3663328776 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7f3a000839e0 con 0x7f3a100c70e0 2023-10-25T00:12:27.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.688+0000 7f3a162fd640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39f4065e60 0x7f39f4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:27.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.688+0000 7f3a162fd640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39f4065e60 0x7f39f4068320 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00b930 tx=0x7f3a0c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:27.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.692+0000 7f3a06ffd640 1 -- 172.21.15.107:0/3663328776 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3a00050200 con 0x7f3a100c70e0 2023-10-25T00:12:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.944+0000 7f3a18561640 1 -- 172.21.15.107:0/3663328776 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a100b9980 con 0x7f3a100c70e0 2023-10-25T00:12:27.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.944+0000 7f3a06ffd640 1 -- 172.21.15.107:0/3663328776 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f3a00053820 con 0x7f3a100c70e0 2023-10-25T00:12:27.943 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:27.943 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:27.944 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 165 2023-10-25T00:12:27.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 -- 172.21.15.107:0/3663328776 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39f4065e60 msgr2=0x7f39f4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:27.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39f4065e60 0x7f39f4068320 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00b930 tx=0x7f3a0c008040 comp rx=0 tx=0).stop 2023-10-25T00:12:27.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 -- 172.21.15.107:0/3663328776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a100c70e0 msgr2=0x7f3a100a6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:27.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a100c70e0 0x7f3a100a6f00 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f3a0000c500 tx=0x7f3a0000c9c0 comp rx=0 tx=0).stop 2023-10-25T00:12:27.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 -- 172.21.15.107:0/3663328776 shutdown_connections 2023-10-25T00:12:27.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3a10156770 0x7f3a100a7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:27.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a100c70e0 0x7f3a100a6f00 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:27.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39f4065e60 0x7f39f4068320 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:27.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 --2- 172.21.15.107:0/3663328776 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3a100be6b0 0x7f3a100c4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:27.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 -- 172.21.15.107:0/3663328776 >> 172.21.15.107:0/3663328776 conn(0x7f3a100c65c0 msgr2=0x7f3a101556d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:27.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 -- 172.21.15.107:0/3663328776 shutdown_connections 2023-10-25T00:12:27.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:27.948+0000 7f3a18561640 1 -- 172.21.15.107:0/3663328776 wait complete. 2023-10-25T00:12:27.963 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:12:29.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.108+0000 7f0490491640 1 Processor -- start 2023-10-25T00:12:30.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.108+0000 7f0490491640 1 -- start start 2023-10-25T00:12:30.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f0490491640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f048814fc30 0x7f0488153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:30.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f0490491640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04880c18a0 0x7f04880c1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:30.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f0490491640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0488072330 con 0x7f04880c5c20 2023-10-25T00:12:30.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f0490491640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0488155ca0 con 0x7f048814fc30 2023-10-25T00:12:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f0490491640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0488155f80 con 0x7f04880c18a0 2023-10-25T00:12:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048e22d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f048814fc30 0x7f0488153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048e22d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f048814fc30 0x7f0488153190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35220/0 (socket says 172.21.15.107:35220) 2023-10-25T00:12:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048da2c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f04880c5c20 0x7f048806df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52126/0 (socket says 172.21.15.107:52126) 2023-10-25T00:12:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048e22d640 1 -- 172.21.15.107:0/4253081586 learned_addr learned my addr 172.21.15.107:0/4253081586 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048ea2e640 1 --2- 172.21.15.107:0/4253081586 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04880c18a0 0x7f04880c1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048e22d640 1 -- 172.21.15.107:0/4253081586 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04880c18a0 msgr2=0x7f04880c1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048e22d640 1 --2- 172.21.15.107:0/4253081586 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04880c18a0 0x7f04880c1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048e22d640 1 -- 172.21.15.107:0/4253081586 >> v1:172.21.15.107:6789/0 conn(0x7f04880c5c20 legacy=0x7f048806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048d22b640 1 -- 172.21.15.107:0/4253081586 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346524191 0 0) 0x7f0488072330 con 0x7f04880c5c20 2023-10-25T00:12:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048e22d640 1 -- 172.21.15.107:0/4253081586 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0488156260 con 0x7f048814fc30 2023-10-25T00:12:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048e22d640 1 --2- 172.21.15.107:0/4253081586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f048814fc30 0x7f0488153190 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f0484009870 tx=0x7f048401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be061e3c0f61148 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:30.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048d22b640 1 -- 172.21.15.107:0/4253081586 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04840ab680 con 0x7f048814fc30 2023-10-25T00:12:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048d22b640 1 -- 172.21.15.107:0/4253081586 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04840ab7e0 con 0x7f048814fc30 2023-10-25T00:12:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f048d22b640 1 -- 172.21.15.107:0/4253081586 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04840abb70 con 0x7f048814fc30 2023-10-25T00:12:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f0490491640 1 -- 172.21.15.107:0/4253081586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f048814fc30 msgr2=0x7f0488153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f0490491640 1 --2- 172.21.15.107:0/4253081586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f048814fc30 0x7f0488153190 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f0484009870 tx=0x7f048401c480 comp rx=0 tx=0).stop 2023-10-25T00:12:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f0490491640 1 -- 172.21.15.107:0/4253081586 shutdown_connections 2023-10-25T00:12:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f0490491640 1 --2- 172.21.15.107:0/4253081586 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f04880c18a0 0x7f04880c1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f0490491640 1 --2- 172.21.15.107:0/4253081586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f048814fc30 0x7f0488153190 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.112+0000 7f0490491640 1 -- 172.21.15.107:0/4253081586 >> 172.21.15.107:0/4253081586 conn(0x7f04880c39a0 msgr2=0x7f04880c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 -- 172.21.15.107:0/4253081586 shutdown_connections 2023-10-25T00:12:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 -- 172.21.15.107:0/4253081586 wait complete. 2023-10-25T00:12:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 Processor -- start 2023-10-25T00:12:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 -- start start 2023-10-25T00:12:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04880c18a0 0x7f0488138e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04880c5c20 0x7f0488134ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0488139d10 0x7f04881f0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0488139ae0 con 0x7f04880c5c20 2023-10-25T00:12:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048813a680 con 0x7f04880c18a0 2023-10-25T00:12:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048814f0c0 con 0x7f0488139d10 2023-10-25T00:12:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048da2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04880c5c20 0x7f0488134ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048da2c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04880c5c20 0x7f0488134ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35234/0 (socket says 172.21.15.107:35234) 2023-10-25T00:12:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048da2c640 1 -- 172.21.15.107:0/46402326 learned_addr learned my addr 172.21.15.107:0/46402326 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048e22d640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04880c18a0 0x7f0488138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048ea2e640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0488139d10 0x7f04881f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048da2c640 1 -- 172.21.15.107:0/46402326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0488139d10 msgr2=0x7f04881f0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048da2c640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0488139d10 0x7f04881f0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048da2c640 1 -- 172.21.15.107:0/46402326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04880c18a0 msgr2=0x7f0488138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048da2c640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04880c18a0 0x7f0488138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048da2c640 1 -- 172.21.15.107:0/46402326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04840094d0 con 0x7f04880c5c20 2023-10-25T00:12:30.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048e22d640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04880c18a0 0x7f0488138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:12:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f048da2c640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04880c5c20 0x7f0488134ce0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f047800c480 tx=0x7f047800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f047effd640 1 -- 172.21.15.107:0/46402326 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04780074e0 con 0x7f04880c5c20 2023-10-25T00:12:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f047effd640 1 -- 172.21.15.107:0/46402326 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0478007660 con 0x7f04880c5c20 2023-10-25T00:12:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 -- 172.21.15.107:0/46402326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048813a8b0 con 0x7f04880c5c20 2023-10-25T00:12:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f047effd640 1 -- 172.21.15.107:0/46402326 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0478004a30 con 0x7f04880c5c20 2023-10-25T00:12:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.116+0000 7f0490491640 1 -- 172.21.15.107:0/46402326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048813ae70 con 0x7f04880c5c20 2023-10-25T00:12:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.120+0000 7f0490491640 1 -- 172.21.15.107:0/46402326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048813b090 con 0x7f04880c5c20 2023-10-25T00:12:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.120+0000 7f047effd640 1 -- 172.21.15.107:0/46402326 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0478004b90 con 0x7f04880c5c20 2023-10-25T00:12:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.120+0000 7f047effd640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f046c065e60 0x7f046c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.120+0000 7f047effd640 1 -- 172.21.15.107:0/46402326 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7f0478083770 con 0x7f04880c5c20 2023-10-25T00:12:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.120+0000 7f048e22d640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f046c065e60 0x7f046c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.120+0000 7f048e22d640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f046c065e60 0x7f046c068320 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f04840af000 tx=0x7f0484008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.124+0000 7f047effd640 1 -- 172.21.15.107:0/46402326 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0478053aa0 con 0x7f04880c5c20 2023-10-25T00:12:30.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 -- 172.21.15.107:0/46402326 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04880746e0 con 0x7f04880c5c20 2023-10-25T00:12:30.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f047effd640 1 -- 172.21.15.107:0/46402326 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5776 (secure 0 0 0) 0x7f0478018020 con 0x7f04880c5c20 2023-10-25T00:12:30.358 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:30.358 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:30.359 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 165 2023-10-25T00:12:30.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 -- 172.21.15.107:0/46402326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f046c065e60 msgr2=0x7f046c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:30.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f046c065e60 0x7f046c068320 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f04840af000 tx=0x7f0484008040 comp rx=0 tx=0).stop 2023-10-25T00:12:30.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 -- 172.21.15.107:0/46402326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04880c5c20 msgr2=0x7f0488134ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:30.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04880c5c20 0x7f0488134ce0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f047800c480 tx=0x7f047800c940 comp rx=0 tx=0).stop 2023-10-25T00:12:30.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 -- 172.21.15.107:0/46402326 shutdown_connections 2023-10-25T00:12:30.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f046c065e60 0x7f046c068320 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:30.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0488139d10 0x7f04881f0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:30.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f04880c5c20 0x7f0488134ce0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:30.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 --2- 172.21.15.107:0/46402326 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f04880c18a0 0x7f0488138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:30.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 -- 172.21.15.107:0/46402326 >> 172.21.15.107:0/46402326 conn(0x7f04880c39a0 msgr2=0x7f048815d850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 -- 172.21.15.107:0/46402326 shutdown_connections 2023-10-25T00:12:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:30.360+0000 7f0490491640 1 -- 172.21.15.107:0/46402326 wait complete. 2023-10-25T00:12:30.380 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:12:32.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:32.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.528+0000 7feb450b2640 1 Processor -- start 2023-10-25T00:12:32.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.528+0000 7feb450b2640 1 -- start start 2023-10-25T00:12:32.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.528+0000 7feb450b2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb400adb80 0x7feb40158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:32.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.528+0000 7feb450b2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feb4006e800 0x7feb4006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:32.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.528+0000 7feb450b2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb40152420 con 0x7feb40152040 2023-10-25T00:12:32.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.528+0000 7feb450b2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb401556e0 con 0x7feb4006e800 2023-10-25T00:12:32.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.528+0000 7feb450b2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb40154d40 con 0x7feb400adb80 2023-10-25T00:12:32.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.528+0000 7feb37fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb400adb80 0x7feb40158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:32.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.528+0000 7feb3ed9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7feb40152040 0x7feb40071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:41736/0 (socket says 172.21.15.107:41736) 2023-10-25T00:12:32.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.528+0000 7feb3f59e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feb4006e800 0x7feb4006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:32.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.528+0000 7feb3ed9d640 1 -- 172.21.15.107:0/1355347074 learned_addr learned my addr 172.21.15.107:0/1355347074 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:12:32.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb3f59e640 1 -- 172.21.15.107:0/1355347074 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb400adb80 msgr2=0x7feb40158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:32.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb3f59e640 1 --2- 172.21.15.107:0/1355347074 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb400adb80 0x7feb40158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:32.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb3f59e640 1 -- 172.21.15.107:0/1355347074 >> v1:172.21.15.110:6789/0 conn(0x7feb40152040 legacy=0x7feb40071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:32.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb3f59e640 1 -- 172.21.15.107:0/1355347074 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40155020 con 0x7feb4006e800 2023-10-25T00:12:32.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb3f59e640 1 --2- 172.21.15.107:0/1355347074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb4006e800 0x7feb4006ec90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7feb38009d90 tx=0x7feb3801c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=541c859ae89fd6a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:32.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb3e59c640 1 -- 172.21.15.107:0/1355347074 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb380a5780 con 0x7feb4006e800 2023-10-25T00:12:32.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb3e59c640 1 -- 172.21.15.107:0/1355347074 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb380a58e0 con 0x7feb4006e800 2023-10-25T00:12:32.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb3e59c640 1 -- 172.21.15.107:0/1355347074 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb380a5c70 con 0x7feb4006e800 2023-10-25T00:12:32.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb450b2640 1 -- 172.21.15.107:0/1355347074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb4006e800 msgr2=0x7feb4006ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:32.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb450b2640 1 --2- 172.21.15.107:0/1355347074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb4006e800 0x7feb4006ec90 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7feb38009d90 tx=0x7feb3801c8c0 comp rx=0 tx=0).stop 2023-10-25T00:12:32.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb450b2640 1 -- 172.21.15.107:0/1355347074 shutdown_connections 2023-10-25T00:12:32.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb450b2640 1 --2- 172.21.15.107:0/1355347074 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb4006e800 0x7feb4006ec90 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:32.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb450b2640 1 --2- 172.21.15.107:0/1355347074 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb400adb80 0x7feb40158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:32.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb450b2640 1 -- 172.21.15.107:0/1355347074 >> 172.21.15.107:0/1355347074 conn(0x7feb40070f60 msgr2=0x7feb40071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:32.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb450b2640 1 -- 172.21.15.107:0/1355347074 shutdown_connections 2023-10-25T00:12:32.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.532+0000 7feb450b2640 1 -- 172.21.15.107:0/1355347074 wait complete. 2023-10-25T00:12:32.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb450b2640 1 Processor -- start 2023-10-25T00:12:32.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb450b2640 1 -- start start 2023-10-25T00:12:32.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb450b2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb4006e800 0x7feb40156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:32.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb450b2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb400adb80 0x7feb40146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:32.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb450b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb40152040 0x7feb40155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:32.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb450b2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb401514b0 con 0x7feb40152040 2023-10-25T00:12:32.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb450b2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb40151790 con 0x7feb4006e800 2023-10-25T00:12:32.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb450b2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb40157960 con 0x7feb400adb80 2023-10-25T00:12:32.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb3f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb40152040 0x7feb40155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:32.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb3f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb40152040 0x7feb40155a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39648/0 (socket says 172.21.15.107:39648) 2023-10-25T00:12:32.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb3f59e640 1 -- 172.21.15.107:0/4060688783 learned_addr learned my addr 172.21.15.107:0/4060688783 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:32.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb3ed9d640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb4006e800 0x7feb40156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:32.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb37fff640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb400adb80 0x7feb40146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:32.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb3f59e640 1 -- 172.21.15.107:0/4060688783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb400adb80 msgr2=0x7feb40146d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:32.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb3f59e640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb400adb80 0x7feb40146d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:32.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb3f59e640 1 -- 172.21.15.107:0/4060688783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb4006e800 msgr2=0x7feb40156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:32.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb3f59e640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb4006e800 0x7feb40156e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:32.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb3f59e640 1 -- 172.21.15.107:0/4060688783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb38009a70 con 0x7feb40152040 2023-10-25T00:12:32.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb3f59e640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb40152040 0x7feb40155a00 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7feb3801a4f0 tx=0x7feb38019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:32.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb377fe640 1 -- 172.21.15.107:0/4060688783 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb380ad640 con 0x7feb40152040 2023-10-25T00:12:32.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb377fe640 1 -- 172.21.15.107:0/4060688783 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb380ad7a0 con 0x7feb40152040 2023-10-25T00:12:32.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb450b2640 1 -- 172.21.15.107:0/4060688783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb40157c40 con 0x7feb40152040 2023-10-25T00:12:32.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb377fe640 1 -- 172.21.15.107:0/4060688783 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb380adb10 con 0x7feb40152040 2023-10-25T00:12:32.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb450b2640 1 -- 172.21.15.107:0/4060688783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb401472a0 con 0x7feb40152040 2023-10-25T00:12:32.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb450b2640 1 -- 172.21.15.107:0/4060688783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb400746e0 con 0x7feb40152040 2023-10-25T00:12:32.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.536+0000 7feb377fe640 1 -- 172.21.15.107:0/4060688783 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7feb380b4070 con 0x7feb40152040 2023-10-25T00:12:32.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.540+0000 7feb377fe640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb20065e90 0x7feb20068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:32.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.540+0000 7feb377fe640 1 -- 172.21.15.107:0/4060688783 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 9627+0+0 (secure 0 0 0) 0x7feb3811f450 con 0x7feb40152040 2023-10-25T00:12:32.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.540+0000 7feb3ed9d640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb20065e90 0x7feb20068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:32.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.540+0000 7feb3ed9d640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb20065e90 0x7feb20068350 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7feb30000f70 tx=0x7feb30009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:32.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.544+0000 7feb377fe640 1 -- 172.21.15.107:0/4060688783 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7feb380ebc00 con 0x7feb40152040 2023-10-25T00:12:32.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.784+0000 7feb450b2640 1 -- 172.21.15.107:0/4060688783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb400c23a0 con 0x7feb40152040 2023-10-25T00:12:32.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb377fe640 1 -- 172.21.15.107:0/4060688783 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5776 (secure 0 0 0) 0x7feb380b2090 con 0x7feb40152040 2023-10-25T00:12:32.785 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:32.785 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8016,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":165}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:11:06.860947+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7839":{"gid":7839,"name":"a","rank":0,"incarnation":156,"state":"up:active","state_seq":8,"addr":"172.21.15.107:6837/3370462806","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3370462806},{"type":"v1","addr":"172.21.15.107:6837","nonce":3370462806}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 165 2023-10-25T00:12:32.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 -- 172.21.15.107:0/4060688783 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb20065e90 msgr2=0x7feb20068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:32.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb20065e90 0x7feb20068350 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7feb30000f70 tx=0x7feb30009040 comp rx=0 tx=0).stop 2023-10-25T00:12:32.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 -- 172.21.15.107:0/4060688783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb40152040 msgr2=0x7feb40155a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:32.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb40152040 0x7feb40155a00 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7feb3801a4f0 tx=0x7feb38019660 comp rx=0 tx=0).stop 2023-10-25T00:12:32.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 -- 172.21.15.107:0/4060688783 shutdown_connections 2023-10-25T00:12:32.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb40152040 0x7feb40155a00 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:32.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feb20065e90 0x7feb20068350 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:32.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feb400adb80 0x7feb40146d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:32.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 --2- 172.21.15.107:0/4060688783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feb4006e800 0x7feb40156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:32.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 -- 172.21.15.107:0/4060688783 >> 172.21.15.107:0/4060688783 conn(0x7feb40070f60 msgr2=0x7feb400c30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:32.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 -- 172.21.15.107:0/4060688783 shutdown_connections 2023-10-25T00:12:32.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:32.788+0000 7feb450b2640 1 -- 172.21.15.107:0/4060688783 wait complete. 2023-10-25T00:12:32.803 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:12:34.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:34.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.920+0000 7fa27c2e4640 1 Processor -- start 2023-10-25T00:12:34.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.920+0000 7fa27c2e4640 1 -- start start 2023-10-25T00:12:34.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.920+0000 7fa27c2e4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740be6b0 0x7fa27415e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:34.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.920+0000 7fa27c2e4640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2740c1ca0 con 0x7fa274156770 2023-10-25T00:12:34.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.920+0000 7fa27c2e4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2740c10c0 con 0x7fa2740c70e0 2023-10-25T00:12:34.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.920+0000 7fa27c2e4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2740c27e0 con 0x7fa2740be6b0 2023-10-25T00:12:34.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.920+0000 7fa27a080640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa274156770 0x7fa2740ab7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59798/0 (socket says 172.21.15.107:59798) 2023-10-25T00:12:34.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.920+0000 7fa27a080640 1 -- 172.21.15.107:0/1097053781 learned_addr learned my addr 172.21.15.107:0/1097053781 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:12:34.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.920+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052071547 0 0) 0x7fa2740c1ca0 con 0x7fa274156770 2023-10-25T00:12:34.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.920+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa260003530 con 0x7fa274156770 2023-10-25T00:12:34.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.920+0000 7fa27a881640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740be6b0 0x7fa27415e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:34.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3248953442 0 0) 0x7fa260003530 con 0x7fa274156770 2023-10-25T00:12:34.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740be6b0 msgr2=0x7fa27415e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:34.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740be6b0 0x7fa27415e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:34.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 >> v1:172.21.15.110:6789/0 conn(0x7fa2740c70e0 legacy=0x7fa2740bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:34.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2740c13a0 con 0x7fa274156770 2023-10-25T00:12:34.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fa270001680 con 0x7fa274156770 2023-10-25T00:12:34.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 >> v1:172.21.15.107:6789/0 conn(0x7fa274156770 legacy=0x7fa2740ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:34.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740be6b0 0x7fa2600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:34.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa260003f30 0x7fa2600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:34.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa260004d10 0x7fa260005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:34.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26000bba0 con 0x7fa260004d10 2023-10-25T00:12:34.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26000bd10 con 0x7fa260003f30 2023-10-25T00:12:34.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26000bed0 con 0x7fa2740be6b0 2023-10-25T00:12:34.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27a881640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa260004d10 0x7fa260005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:34.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27a080640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740be6b0 0x7fa2600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:34.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27987f640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa260003f30 0x7fa2600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:34.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27a881640 1 -- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740be6b0 msgr2=0x7fa2600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:34.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27a881640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740be6b0 0x7fa2600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:34.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27a881640 1 -- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa260003f30 msgr2=0x7fa2600047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:34.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27a881640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa260003f30 0x7fa2600047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:34.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27a881640 1 -- 172.21.15.107:0/1097053781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa260003530 con 0x7fa260004d10 2023-10-25T00:12:34.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27987f640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa260003f30 0x7fa2600047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27a881640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa260004d10 0x7fa260005190 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fa27406f520 tx=0x7fa26c01cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39515dbdb47e1104 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c0a5b80 con 0x7fa260004d10 2023-10-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa26c00f040 con 0x7fa260004d10 2023-10-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27907e640 1 -- 172.21.15.107:0/1097053781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26c0ad440 con 0x7fa260004d10 2023-10-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27c2e4640 1 -- 172.21.15.107:0/1097053781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa260004d10 msgr2=0x7fa260005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27c2e4640 1 --2- 172.21.15.107:0/1097053781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa260004d10 0x7fa260005190 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fa27406f520 tx=0x7fa26c01cbd0 comp rx=0 tx=0).stop 2023-10-25T00:12:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27a881640 1 -- 172.21.15.107:0/1097053781 reap_dead start 2023-10-25T00:12:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27c2e4640 1 -- 172.21.15.107:0/1097053781 shutdown_connections 2023-10-25T00:12:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27c2e4640 1 -- 172.21.15.107:0/1097053781 >> 172.21.15.107:0/1097053781 conn(0x7fa2740c65c0 msgr2=0x7fa2740c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:34.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27c2e4640 1 -- 172.21.15.107:0/1097053781 shutdown_connections 2023-10-25T00:12:34.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.924+0000 7fa27c2e4640 1 -- 172.21.15.107:0/1097053781 wait complete. 2023-10-25T00:12:34.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27c2e4640 1 Processor -- start 2023-10-25T00:12:34.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27c2e4640 1 -- start start 2023-10-25T00:12:34.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27c2e4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa260004d10 0x7fa2740b9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:34.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27c2e4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa274072150 0x7fa2740c5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:34.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27c2e4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740ba790 0x7fa2740bac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:34.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27c2e4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa274071700 con 0x7fa274072150 2023-10-25T00:12:34.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27c2e4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2740ba530 con 0x7fa260004d10 2023-10-25T00:12:34.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27c2e4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa274070ec0 con 0x7fa2740ba790 2023-10-25T00:12:34.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27987f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa274072150 0x7fa2740c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:34.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27987f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa274072150 0x7fa2740c5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39678/0 (socket says 172.21.15.107:39678) 2023-10-25T00:12:34.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27987f640 1 -- 172.21.15.107:0/3149444206 learned_addr learned my addr 172.21.15.107:0/3149444206 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:34.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27a080640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa260004d10 0x7fa2740b9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:34.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27a881640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740ba790 0x7fa2740bac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:34.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27987f640 1 -- 172.21.15.107:0/3149444206 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740ba790 msgr2=0x7fa2740bac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27987f640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740ba790 0x7fa2740bac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27987f640 1 -- 172.21.15.107:0/3149444206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa260004d10 msgr2=0x7fa2740b9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27987f640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa260004d10 0x7fa2740b9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27987f640 1 -- 172.21.15.107:0/3149444206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2740711a0 con 0x7fa274072150 2023-10-25T00:12:34.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27a881640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740ba790 0x7fa2740bac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:12:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27987f640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa274072150 0x7fa2740c5160 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fa26400e550 tx=0x7fa26400e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa26affd640 1 -- 172.21.15.107:0/3149444206 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2640052e0 con 0x7fa274072150 2023-10-25T00:12:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa26affd640 1 -- 172.21.15.107:0/3149444206 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa264005440 con 0x7fa274072150 2023-10-25T00:12:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27c2e4640 1 -- 172.21.15.107:0/3149444206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa274071950 con 0x7fa274072150 2023-10-25T00:12:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa26affd640 1 -- 172.21.15.107:0/3149444206 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa264003840 con 0x7fa274072150 2023-10-25T00:12:34.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27c2e4640 1 -- 172.21.15.107:0/3149444206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2740c56a0 con 0x7fa274072150 2023-10-25T00:12:34.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.928+0000 7fa27c2e4640 1 -- 172.21.15.107:0/3149444206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2740746e0 con 0x7fa274072150 2023-10-25T00:12:34.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.932+0000 7fa26affd640 1 -- 172.21.15.107:0/3149444206 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa2640039a0 con 0x7fa274072150 2023-10-25T00:12:34.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.932+0000 7fa26affd640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa258065e60 0x7fa258068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:34.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.932+0000 7fa26affd640 1 -- 172.21.15.107:0/3149444206 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7fa264084350 con 0x7fa274072150 2023-10-25T00:12:34.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.932+0000 7fa27a080640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa258065e60 0x7fa258068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:34.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.932+0000 7fa27a080640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa258065e60 0x7fa258068320 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa2741e88d0 tx=0x7fa270007900 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:34.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:34.936+0000 7fa26affd640 1 -- 172.21.15.107:0/3149444206 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa264050b90 con 0x7fa274072150 2023-10-25T00:12:35.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.148+0000 7fa27c2e4640 1 -- 172.21.15.107:0/3149444206 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa274154f30 con 0x7fa274072150 2023-10-25T00:12:35.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.148+0000 7fa26affd640 1 -- 172.21.15.107:0/3149444206 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5019 (secure 0 0 0) 0x7fa2640541b0 con 0x7fa274072150 2023-10-25T00:12:35.146 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:35.147 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:34.907299+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:35.147 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 167 2023-10-25T00:12:35.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 -- 172.21.15.107:0/3149444206 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa258065e60 msgr2=0x7fa258068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:35.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa258065e60 0x7fa258068320 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa2741e88d0 tx=0x7fa270007900 comp rx=0 tx=0).stop 2023-10-25T00:12:35.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 -- 172.21.15.107:0/3149444206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa274072150 msgr2=0x7fa2740c5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:35.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa274072150 0x7fa2740c5160 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fa26400e550 tx=0x7fa26400e950 comp rx=0 tx=0).stop 2023-10-25T00:12:35.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 -- 172.21.15.107:0/3149444206 shutdown_connections 2023-10-25T00:12:35.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa258065e60 0x7fa258068320 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa2740ba790 0x7fa2740bac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa274072150 0x7fa2740c5160 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 --2- 172.21.15.107:0/3149444206 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa260004d10 0x7fa2740b9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 -- 172.21.15.107:0/3149444206 >> 172.21.15.107:0/3149444206 conn(0x7fa2740c65c0 msgr2=0x7fa2740ab7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:35.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 -- 172.21.15.107:0/3149444206 shutdown_connections 2023-10-25T00:12:35.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.152+0000 7fa27c2e4640 1 -- 172.21.15.107:0/3149444206 wait complete. 2023-10-25T00:12:35.165 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2023-10-25T00:12:35.166 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:12:35.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:35.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.284+0000 7faabba01640 1 Processor -- start 2023-10-25T00:12:35.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.284+0000 7faabba01640 1 -- start start 2023-10-25T00:12:35.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faabba01640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab414c470 0x7faab414c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:35.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faabba01640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faab406f490 0x7faab4071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:35.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faabba01640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faab4071fc0 0x7faab4152b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:35.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faabba01640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab41526e0 con 0x7faab4071fc0 2023-10-25T00:12:35.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faabba01640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab4155cd0 con 0x7faab406f490 2023-10-25T00:12:35.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faabba01640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab40c4380 con 0x7faab414c470 2023-10-25T00:12:35.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab9f9e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faab4071fc0 0x7faab4152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab9f9e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faab4071fc0 0x7faab4152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39686/0 (socket says 172.21.15.107:39686) 2023-10-25T00:12:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab9f9e640 1 -- 172.21.15.107:0/282152222 learned_addr learned my addr 172.21.15.107:0/282152222 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab8f9c640 1 --2- 172.21.15.107:0/282152222 >> v2:172.21.15.110:3300/0 conn(0x7faab406f490 0x7faab4071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab979d640 1 --2- 172.21.15.107:0/282152222 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab414c470 0x7faab414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:35.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab9f9e640 1 -- 172.21.15.107:0/282152222 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab414c470 msgr2=0x7faab414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:35.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab9f9e640 1 --2- 172.21.15.107:0/282152222 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab414c470 0x7faab414c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab9f9e640 1 -- 172.21.15.107:0/282152222 >> v2:172.21.15.110:3300/0 conn(0x7faab406f490 msgr2=0x7faab4071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:35.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab9f9e640 1 --2- 172.21.15.107:0/282152222 >> v2:172.21.15.110:3300/0 conn(0x7faab406f490 0x7faab4071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab9f9e640 1 -- 172.21.15.107:0/282152222 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab40c48b0 con 0x7faab4071fc0 2023-10-25T00:12:35.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab8f9c640 1 --2- 172.21.15.107:0/282152222 >> v2:172.21.15.110:3300/0 conn(0x7faab406f490 0x7faab4071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:12:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faab9f9e640 1 --2- 172.21.15.107:0/282152222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab4071fc0 0x7faab4152b30 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7faab40727a0 tx=0x7faaac01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da3f4a0f7937998 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faaabfff640 1 -- 172.21.15.107:0/282152222 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0a5620 con 0x7faab4071fc0 2023-10-25T00:12:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faaabfff640 1 -- 172.21.15.107:0/282152222 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac0a5780 con 0x7faab4071fc0 2023-10-25T00:12:35.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.288+0000 7faaabfff640 1 -- 172.21.15.107:0/282152222 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0a5b10 con 0x7faab4071fc0 2023-10-25T00:12:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 -- 172.21.15.107:0/282152222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab4071fc0 msgr2=0x7faab4152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 --2- 172.21.15.107:0/282152222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab4071fc0 0x7faab4152b30 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7faab40727a0 tx=0x7faaac01c8b0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 -- 172.21.15.107:0/282152222 shutdown_connections 2023-10-25T00:12:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 --2- 172.21.15.107:0/282152222 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab4071fc0 0x7faab4152b30 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 --2- 172.21.15.107:0/282152222 >> v2:172.21.15.110:3300/0 conn(0x7faab406f490 0x7faab4071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 --2- 172.21.15.107:0/282152222 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab414c470 0x7faab414c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 -- 172.21.15.107:0/282152222 >> 172.21.15.107:0/282152222 conn(0x7faab4070f60 msgr2=0x7faab4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:35.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 -- 172.21.15.107:0/282152222 shutdown_connections 2023-10-25T00:12:35.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 -- 172.21.15.107:0/282152222 wait complete. 2023-10-25T00:12:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 Processor -- start 2023-10-25T00:12:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 -- start start 2023-10-25T00:12:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faab406f490 0x7faab414b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab4071fc0 0x7faab4155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:35.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab414c470 0x7faab4145950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:35.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab414b690 con 0x7faab414c470 2023-10-25T00:12:35.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab40c20e0 con 0x7faab406f490 2023-10-25T00:12:35.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab40c23c0 con 0x7faab4071fc0 2023-10-25T00:12:35.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faab9f9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab414c470 0x7faab4145950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:35.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faab9f9e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab414c470 0x7faab4145950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39694/0 (socket says 172.21.15.107:39694) 2023-10-25T00:12:35.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faab9f9e640 1 -- 172.21.15.107:0/828722010 learned_addr learned my addr 172.21.15.107:0/828722010 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:35.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faab8f9c640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab4071fc0 0x7faab4155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:35.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faab979d640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faab406f490 0x7faab414b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:35.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faab9f9e640 1 -- 172.21.15.107:0/828722010 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab4071fc0 msgr2=0x7faab4155120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:35.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faab9f9e640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab4071fc0 0x7faab4155120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faab9f9e640 1 -- 172.21.15.107:0/828722010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faab406f490 msgr2=0x7faab414b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faab9f9e640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faab406f490 0x7faab414b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faab9f9e640 1 -- 172.21.15.107:0/828722010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac009b30 con 0x7faab414c470 2023-10-25T00:12:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faab9f9e640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab414c470 0x7faab4145950 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7faaac01af60 tx=0x7faaac019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:35.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faaa9ffb640 1 -- 172.21.15.107:0/828722010 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0b8670 con 0x7faab414c470 2023-10-25T00:12:35.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faaa9ffb640 1 -- 172.21.15.107:0/828722010 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac0b87d0 con 0x7faab414c470 2023-10-25T00:12:35.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 -- 172.21.15.107:0/828722010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab406ba20 con 0x7faab414c470 2023-10-25T00:12:35.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faaa9ffb640 1 -- 172.21.15.107:0/828722010 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0b8b60 con 0x7faab414c470 2023-10-25T00:12:35.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.292+0000 7faabba01640 1 -- 172.21.15.107:0/828722010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab406bfe0 con 0x7faab414c470 2023-10-25T00:12:35.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.296+0000 7faabba01640 1 -- 172.21.15.107:0/828722010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab40746e0 con 0x7faab414c470 2023-10-25T00:12:35.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.296+0000 7faaa9ffb640 1 -- 172.21.15.107:0/828722010 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7faaac0b8cc0 con 0x7faab414c470 2023-10-25T00:12:35.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.296+0000 7faaa9ffb640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa98065e90 0x7faa98068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:35.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.296+0000 7faaa9ffb640 1 -- 172.21.15.107:0/828722010 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7faaac11f840 con 0x7faab414c470 2023-10-25T00:12:35.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.296+0000 7faab979d640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa98065e90 0x7faa98068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:35.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.296+0000 7faab979d640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa98065e90 0x7faa98068350 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7faab000a970 tx=0x7faab0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:35.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.300+0000 7faaa9ffb640 1 -- 172.21.15.107:0/828722010 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7faaac0ec080 con 0x7faab414c470 2023-10-25T00:12:35.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.532+0000 7faabba01640 1 -- 172.21.15.107:0/828722010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faab4155660 con 0x7faab414c470 2023-10-25T00:12:35.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faaa9ffb640 1 -- 172.21.15.107:0/828722010 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+5019 (secure 0 0 0) 0x7faaac0bc090 con 0x7faab414c470 2023-10-25T00:12:35.533 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:35.533 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:34.907299+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:35.533 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 167 2023-10-25T00:12:35.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 -- 172.21.15.107:0/828722010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa98065e90 msgr2=0x7faa98068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:35.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa98065e90 0x7faa98068350 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7faab000a970 tx=0x7faab0008040 comp rx=0 tx=0).stop 2023-10-25T00:12:35.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 -- 172.21.15.107:0/828722010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab414c470 msgr2=0x7faab4145950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:35.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab414c470 0x7faab4145950 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7faaac01af60 tx=0x7faaac019650 comp rx=0 tx=0).stop 2023-10-25T00:12:35.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 -- 172.21.15.107:0/828722010 shutdown_connections 2023-10-25T00:12:35.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab414c470 0x7faab4145950 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faa98065e90 0x7faa98068350 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faab4071fc0 0x7faab4155120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 --2- 172.21.15.107:0/828722010 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faab406f490 0x7faab414b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:35.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 -- 172.21.15.107:0/828722010 >> 172.21.15.107:0/828722010 conn(0x7faab4070f60 msgr2=0x7faab4153c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:35.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 -- 172.21.15.107:0/828722010 shutdown_connections 2023-10-25T00:12:35.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:35.536+0000 7faabba01640 1 -- 172.21.15.107:0/828722010 wait complete. 2023-10-25T00:12:35.551 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 167, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 164, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6805": { "addr": "172.21.15.110:6835/898258574", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 898258574, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 898258574, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6805, "incarnation": 161, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:11:21.884636+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6805 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 167, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8016": { "addr": "172.21.15.107:6835/2856791739", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2856791739, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 2856791739, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8016, "incarnation": 166, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:12:34.907299+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8016 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:12:37.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:37.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f3042245640 1 Processor -- start 2023-10-25T00:12:37.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f3042245640 1 -- start start 2023-10-25T00:12:37.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f3042245640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f303c156770 0x7f303c0ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:37.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f3042245640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f303c0c70e0 0x7f303c0bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:37.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f3042245640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f303c0c1ca0 con 0x7f303c0c2130 2023-10-25T00:12:37.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f3042245640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f303c0c27e0 con 0x7f303c0c70e0 2023-10-25T00:12:37.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f3042245640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303c0c10c0 con 0x7f303c156770 2023-10-25T00:12:37.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f303affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f303c0c70e0 0x7f303c0bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:37.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f303bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f303c0c2130 0x7f303c069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:59806/0 (socket says 172.21.15.107:59806) 2023-10-25T00:12:37.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f303bfff640 1 -- 172.21.15.107:0/3192425973 learned_addr learned my addr 172.21.15.107:0/3192425973 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:12:37.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f303b7fe640 1 --2- 172.21.15.107:0/3192425973 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f303c156770 0x7f303c0ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:37.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f303affd640 1 -- 172.21.15.107:0/3192425973 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f303c156770 msgr2=0x7f303c0ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:37.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f303affd640 1 --2- 172.21.15.107:0/3192425973 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f303c156770 0x7f303c0ab7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:37.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f303affd640 1 -- 172.21.15.107:0/3192425973 >> v1:172.21.15.107:6789/0 conn(0x7f303c0c2130 legacy=0x7f303c069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:37.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f303affd640 1 -- 172.21.15.107:0/3192425973 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f303c0c13a0 con 0x7f303c0c70e0 2023-10-25T00:12:37.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.716+0000 7f303a7fc640 1 -- 172.21.15.107:0/3192425973 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482927811 0 0) 0x7f303c0c1ca0 con 0x7f303c0c2130 2023-10-25T00:12:37.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f303affd640 1 --2- 172.21.15.107:0/3192425973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f303c0c70e0 0x7f303c0bdfa0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f30300097c0 tx=0x7f303001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2bb0bc5e11a7dc2 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:37.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f303a7fc640 1 -- 172.21.15.107:0/3192425973 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30300ab600 con 0x7f303c0c70e0 2023-10-25T00:12:37.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f303a7fc640 1 -- 172.21.15.107:0/3192425973 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30300a7050 con 0x7f303c0c70e0 2023-10-25T00:12:37.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f303a7fc640 1 -- 172.21.15.107:0/3192425973 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30300ab980 con 0x7f303c0c70e0 2023-10-25T00:12:37.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 -- 172.21.15.107:0/3192425973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f303c0c70e0 msgr2=0x7f303c0bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:37.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 --2- 172.21.15.107:0/3192425973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f303c0c70e0 0x7f303c0bdfa0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f30300097c0 tx=0x7f303001c330 comp rx=0 tx=0).stop 2023-10-25T00:12:37.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 -- 172.21.15.107:0/3192425973 shutdown_connections 2023-10-25T00:12:37.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 --2- 172.21.15.107:0/3192425973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f303c0c70e0 0x7f303c0bdfa0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:37.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 --2- 172.21.15.107:0/3192425973 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f303c156770 0x7f303c0ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:37.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 -- 172.21.15.107:0/3192425973 >> 172.21.15.107:0/3192425973 conn(0x7f303c0c65c0 msgr2=0x7f303c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:37.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 -- 172.21.15.107:0/3192425973 shutdown_connections 2023-10-25T00:12:37.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 -- 172.21.15.107:0/3192425973 wait complete. 2023-10-25T00:12:37.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 Processor -- start 2023-10-25T00:12:37.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 -- start start 2023-10-25T00:12:37.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f303c0c2130 0x7f303c1ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:37.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f303c0c70e0 0x7f303c14f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:37.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f303c156770 0x7f303c14fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:37.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c1ef300 con 0x7f303c156770 2023-10-25T00:12:37.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c1ef020 con 0x7f303c0c70e0 2023-10-25T00:12:37.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f3042245640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303c155340 con 0x7f303c0c2130 2023-10-25T00:12:37.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f303bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f303c156770 0x7f303c14fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:37.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f303bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f303c156770 0x7f303c14fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39710/0 (socket says 172.21.15.107:39710) 2023-10-25T00:12:37.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f303bfff640 1 -- 172.21.15.107:0/2912855900 learned_addr learned my addr 172.21.15.107:0/2912855900 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:37.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f303b7fe640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f303c0c2130 0x7f303c1ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:37.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.720+0000 7f303affd640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f303c0c70e0 0x7f303c14f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:37.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f303bfff640 1 -- 172.21.15.107:0/2912855900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f303c0c2130 msgr2=0x7f303c1ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:37.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f303bfff640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f303c0c2130 0x7f303c1ee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:37.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f303bfff640 1 -- 172.21.15.107:0/2912855900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f303c0c70e0 msgr2=0x7f303c14f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:37.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f303bfff640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f303c0c70e0 0x7f303c14f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:37.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f303bfff640 1 -- 172.21.15.107:0/2912855900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3030009420 con 0x7f303c156770 2023-10-25T00:12:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f303b7fe640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f303c0c2130 0x7f303c1ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:12:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f303bfff640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f303c156770 0x7f303c14fc70 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f3034001570 tx=0x7f303400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f3023fff640 1 -- 172.21.15.107:0/2912855900 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303400a640 con 0x7f303c156770 2023-10-25T00:12:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f3023fff640 1 -- 172.21.15.107:0/2912855900 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303400a7c0 con 0x7f303c156770 2023-10-25T00:12:37.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f3042245640 1 -- 172.21.15.107:0/2912855900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c155620 con 0x7f303c156770 2023-10-25T00:12:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f3023fff640 1 -- 172.21.15.107:0/2912855900 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303400ab70 con 0x7f303c156770 2023-10-25T00:12:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f3042245640 1 -- 172.21.15.107:0/2912855900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c0c5600 con 0x7f303c156770 2023-10-25T00:12:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f3042245640 1 -- 172.21.15.107:0/2912855900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303c0746e0 con 0x7f303c156770 2023-10-25T00:12:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f3023fff640 1 -- 172.21.15.107:0/2912855900 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f303400acd0 con 0x7f303c156770 2023-10-25T00:12:37.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f3023fff640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3028065e60 0x7f3028068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:37.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f3023fff640 1 -- 172.21.15.107:0/2912855900 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7f3034083c40 con 0x7f303c156770 2023-10-25T00:12:37.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f303b7fe640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3028065e60 0x7f3028068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:37.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.724+0000 7f303b7fe640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3028065e60 0x7f3028068320 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f302c00b9f0 tx=0x7f302c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:37.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.728+0000 7f3023fff640 1 -- 172.21.15.107:0/2912855900 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3034050400 con 0x7f303c156770 2023-10-25T00:12:37.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.972+0000 7f3042245640 1 -- 172.21.15.107:0/2912855900 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f303c0bf990 con 0x7f303c156770 2023-10-25T00:12:37.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.972+0000 7f3023fff640 1 -- 172.21.15.107:0/2912855900 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f3034053a20 con 0x7f303c156770 2023-10-25T00:12:37.971 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:37.972 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:37.972 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 169 2023-10-25T00:12:37.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 -- 172.21.15.107:0/2912855900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3028065e60 msgr2=0x7f3028068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:37.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3028065e60 0x7f3028068320 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f302c00b9f0 tx=0x7f302c008040 comp rx=0 tx=0).stop 2023-10-25T00:12:37.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 -- 172.21.15.107:0/2912855900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f303c156770 msgr2=0x7f303c14fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:37.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f303c156770 0x7f303c14fc70 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f3034001570 tx=0x7f303400cca0 comp rx=0 tx=0).stop 2023-10-25T00:12:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 -- 172.21.15.107:0/2912855900 shutdown_connections 2023-10-25T00:12:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f303c156770 0x7f303c14fc70 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f303c0c70e0 0x7f303c14f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:37.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3028065e60 0x7f3028068320 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:37.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 --2- 172.21.15.107:0/2912855900 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f303c0c2130 0x7f303c1ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:37.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 -- 172.21.15.107:0/2912855900 >> 172.21.15.107:0/2912855900 conn(0x7f303c0c65c0 msgr2=0x7f303c1559a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:37.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 -- 172.21.15.107:0/2912855900 shutdown_connections 2023-10-25T00:12:37.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:37.976+0000 7f3042245640 1 -- 172.21.15.107:0/2912855900 wait complete. 2023-10-25T00:12:37.990 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=7839 2023-10-25T00:12:37.991 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.a 2023-10-25T00:12:41.284 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:12:41.284+0000 7f50c0546640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:12:41.992 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2023-10-25T00:12:41.992 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:12:41.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:12:41.996 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:12:41.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:42.072 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:12:42.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0c18fc640 1 Processor -- start 2023-10-25T00:12:42.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0c18fc640 1 -- start start 2023-10-25T00:12:42.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0c18fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0bc152020 0x7fe0bc0c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:42.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0c18fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0bc0c8640 0x7fe0bc06a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:42.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0c18fc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0bc152400 con 0x7fe0bc1553d0 2023-10-25T00:12:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0c18fc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0bc1558d0 con 0x7fe0bc152020 2023-10-25T00:12:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0c18fc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc0c6ce0 con 0x7fe0bc0c8640 2023-10-25T00:12:42.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0baffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0bc152020 0x7fe0bc0c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0baffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0bc152020 0x7fe0bc0c4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39726/0 (socket says 172.21.15.107:39726) 2023-10-25T00:12:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0baffd640 1 -- 172.21.15.107:0/315562279 learned_addr learned my addr 172.21.15.107:0/315562279 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0ba7fc640 1 --2- 172.21.15.107:0/315562279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0bc0c8640 0x7fe0bc06a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:42.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0baffd640 1 -- 172.21.15.107:0/315562279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0bc0c8640 msgr2=0x7fe0bc06a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:42.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0baffd640 1 --2- 172.21.15.107:0/315562279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0bc0c8640 0x7fe0bc06a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:42.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0baffd640 1 -- 172.21.15.107:0/315562279 >> v1:172.21.15.110:6789/0 conn(0x7fe0bc1553d0 legacy=0x7fe0bc06ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:42.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.144+0000 7fe0baffd640 1 -- 172.21.15.107:0/315562279 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc0c6fc0 con 0x7fe0bc152020 2023-10-25T00:12:42.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0baffd640 1 --2- 172.21.15.107:0/315562279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0bc152020 0x7fe0bc0c4bb0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac009870 tx=0x7fe0ac01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff435043bb8bd7a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:42.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0b9ffb640 1 -- 172.21.15.107:0/315562279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0a6040 con 0x7fe0bc152020 2023-10-25T00:12:42.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0b9ffb640 1 -- 172.21.15.107:0/315562279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ac0aa600 con 0x7fe0bc152020 2023-10-25T00:12:42.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0b9ffb640 1 -- 172.21.15.107:0/315562279 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0aa990 con 0x7fe0bc152020 2023-10-25T00:12:42.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 -- 172.21.15.107:0/315562279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0bc152020 msgr2=0x7fe0bc0c4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:42.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 --2- 172.21.15.107:0/315562279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0bc152020 0x7fe0bc0c4bb0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac009870 tx=0x7fe0ac01c480 comp rx=0 tx=0).stop 2023-10-25T00:12:42.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 -- 172.21.15.107:0/315562279 shutdown_connections 2023-10-25T00:12:42.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 --2- 172.21.15.107:0/315562279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0bc0c8640 0x7fe0bc06a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:42.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 --2- 172.21.15.107:0/315562279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0bc152020 0x7fe0bc0c4bb0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:42.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 -- 172.21.15.107:0/315562279 >> 172.21.15.107:0/315562279 conn(0x7fe0bc0abd60 msgr2=0x7fe0bc0ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:42.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 -- 172.21.15.107:0/315562279 shutdown_connections 2023-10-25T00:12:42.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 -- 172.21.15.107:0/315562279 wait complete. 2023-10-25T00:12:42.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 Processor -- start 2023-10-25T00:12:42.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 -- start start 2023-10-25T00:12:42.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0bc0c8640 0x7fe0bc0c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:42.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0bc152020 0x7fe0bc156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:42.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0bc1553d0 0x7fe0bc156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:42.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc14f400 con 0x7fe0bc0c8640 2023-10-25T00:12:42.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc1574a0 con 0x7fe0bc1553d0 2023-10-25T00:12:42.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0c18fc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc0c03f0 con 0x7fe0bc152020 2023-10-25T00:12:42.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0baffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0bc0c8640 0x7fe0bc0c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:42.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0baffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0bc0c8640 0x7fe0bc0c3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39742/0 (socket says 172.21.15.107:39742) 2023-10-25T00:12:42.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0baffd640 1 -- 172.21.15.107:0/3738315157 learned_addr learned my addr 172.21.15.107:0/3738315157 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:42.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0ba7fc640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0bc152020 0x7fe0bc156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:42.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0bb7fe640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0bc1553d0 0x7fe0bc156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:42.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0baffd640 1 -- 172.21.15.107:0/3738315157 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0bc152020 msgr2=0x7fe0bc156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:42.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0baffd640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0bc152020 0x7fe0bc156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:42.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0baffd640 1 -- 172.21.15.107:0/3738315157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0bc1553d0 msgr2=0x7fe0bc156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:42.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0baffd640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0bc1553d0 0x7fe0bc156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:42.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0baffd640 1 -- 172.21.15.107:0/3738315157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac0094d0 con 0x7fe0bc0c8640 2023-10-25T00:12:42.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0baffd640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0bc0c8640 0x7fe0bc0c3730 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac01ab30 tx=0x7fe0ac01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:42.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.148+0000 7fe0a3fff640 1 -- 172.21.15.107:0/3738315157 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0b3800 con 0x7fe0bc0c8640 2023-10-25T00:12:42.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.152+0000 7fe0a3fff640 1 -- 172.21.15.107:0/3738315157 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ac0b3960 con 0x7fe0bc0c8640 2023-10-25T00:12:42.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.152+0000 7fe0a3fff640 1 -- 172.21.15.107:0/3738315157 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0b3d10 con 0x7fe0bc0c8640 2023-10-25T00:12:42.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.152+0000 7fe0c18fc640 1 -- 172.21.15.107:0/3738315157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0bc158ce0 con 0x7fe0bc0c8640 2023-10-25T00:12:42.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.152+0000 7fe0c18fc640 1 -- 172.21.15.107:0/3738315157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0bc0c3c70 con 0x7fe0bc0c8640 2023-10-25T00:12:42.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.152+0000 7fe0c18fc640 1 -- 172.21.15.107:0/3738315157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0bc0746e0 con 0x7fe0bc0c8640 2023-10-25T00:12:42.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.152+0000 7fe0a3fff640 1 -- 172.21.15.107:0/3738315157 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe0ac0b2380 con 0x7fe0bc0c8640 2023-10-25T00:12:42.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.152+0000 7fe0a3fff640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0a8065e90 0x7fe0a8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:42.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.152+0000 7fe0a3fff640 1 -- 172.21.15.107:0/3738315157 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7fe0ac11e860 con 0x7fe0bc0c8640 2023-10-25T00:12:42.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.152+0000 7fe0ba7fc640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0a8065e90 0x7fe0a8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:42.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.152+0000 7fe0ba7fc640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0a8065e90 0x7fe0a8068350 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc15e5f0 tx=0x7fe0b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:42.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.160+0000 7fe0a3fff640 1 -- 172.21.15.107:0/3738315157 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe0ac0eb010 con 0x7fe0bc0c8640 2023-10-25T00:12:42.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.380+0000 7fe0c18fc640 1 -- 172.21.15.107:0/3738315157 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0bc156640 con 0x7fe0bc0c8640 2023-10-25T00:12:42.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.380+0000 7fe0a3fff640 1 -- 172.21.15.107:0/3738315157 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) v1 ==== 95+0+5016 (secure 0 0 0) 0x7fe0ac0ee630 con 0x7fe0bc0c8640 2023-10-25T00:12:42.379 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:42.379 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:42.380 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 169 2023-10-25T00:12:42.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 -- 172.21.15.107:0/3738315157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0a8065e90 msgr2=0x7fe0a8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:42.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0a8065e90 0x7fe0a8068350 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc15e5f0 tx=0x7fe0b0008040 comp rx=0 tx=0).stop 2023-10-25T00:12:42.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 -- 172.21.15.107:0/3738315157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0bc0c8640 msgr2=0x7fe0bc0c3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:42.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0bc0c8640 0x7fe0bc0c3730 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac01ab30 tx=0x7fe0ac01acc0 comp rx=0 tx=0).stop 2023-10-25T00:12:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 -- 172.21.15.107:0/3738315157 shutdown_connections 2023-10-25T00:12:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0bc1553d0 0x7fe0bc156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0a8065e90 0x7fe0a8068350 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0bc152020 0x7fe0bc156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:42.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 --2- 172.21.15.107:0/3738315157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0bc0c8640 0x7fe0bc0c3730 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:42.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 -- 172.21.15.107:0/3738315157 >> 172.21.15.107:0/3738315157 conn(0x7fe0bc0abd60 msgr2=0x7fe0bc06a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:42.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 -- 172.21.15.107:0/3738315157 shutdown_connections 2023-10-25T00:12:42.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:42.384+0000 7fe0c18fc640 1 -- 172.21.15.107:0/3738315157 wait complete. 2023-10-25T00:12:42.399 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-25T00:12:44.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:12:44.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff92ef8d640 1 Processor -- start 2023-10-25T00:12:44.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff92ef8d640 1 -- start start 2023-10-25T00:12:44.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff92ef8d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff928148110 0x7ff928151f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff92ef8d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff928071a40 0x7ff92806c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff92ef8d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92806ae50 con 0x7ff92806b8a0 2023-10-25T00:12:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff92ef8d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9280a9800 con 0x7ff928071a40 2023-10-25T00:12:44.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff92ef8d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92806b480 con 0x7ff928148110 2023-10-25T00:12:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff927fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff928071a40 0x7ff92806c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff927fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff928071a40 0x7ff92806c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37036/0 (socket says 172.21.15.107:37036) 2023-10-25T00:12:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff927fff640 1 -- 172.21.15.107:0/3884852914 learned_addr learned my addr 172.21.15.107:0/3884852914 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:44.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff92cd29640 1 --2- 172.21.15.107:0/3884852914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff928148110 0x7ff928151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:44.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff927fff640 1 -- 172.21.15.107:0/3884852914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff928148110 msgr2=0x7ff928151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:12:44.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff927fff640 1 --2- 172.21.15.107:0/3884852914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff928148110 0x7ff928151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:44.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff927fff640 1 -- 172.21.15.107:0/3884852914 >> v1:172.21.15.107:6789/0 conn(0x7ff92806b8a0 legacy=0x7ff928070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:44.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff927fff640 1 -- 172.21.15.107:0/3884852914 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9280c1550 con 0x7ff928071a40 2023-10-25T00:12:44.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.540+0000 7ff9277fe640 1 -- 172.21.15.107:0/3884852914 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650199230 0 0) 0x7ff92806ae50 con 0x7ff92806b8a0 2023-10-25T00:12:44.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff927fff640 1 --2- 172.21.15.107:0/3884852914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff928071a40 0x7ff92806c1a0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7ff91400bb40 tx=0x7ff91401ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205c26a30269389c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:44.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff9277fe640 1 -- 172.21.15.107:0/3884852914 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9140a7be0 con 0x7ff928071a40 2023-10-25T00:12:44.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff9277fe640 1 -- 172.21.15.107:0/3884852914 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9140a7d40 con 0x7ff928071a40 2023-10-25T00:12:44.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff9277fe640 1 -- 172.21.15.107:0/3884852914 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9140af660 con 0x7ff928071a40 2023-10-25T00:12:44.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 -- 172.21.15.107:0/3884852914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff928071a40 msgr2=0x7ff92806c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:44.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 --2- 172.21.15.107:0/3884852914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff928071a40 0x7ff92806c1a0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7ff91400bb40 tx=0x7ff91401ebf0 comp rx=0 tx=0).stop 2023-10-25T00:12:44.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 -- 172.21.15.107:0/3884852914 shutdown_connections 2023-10-25T00:12:44.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 --2- 172.21.15.107:0/3884852914 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff928071a40 0x7ff92806c1a0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:44.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 --2- 172.21.15.107:0/3884852914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff928148110 0x7ff928151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:44.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 -- 172.21.15.107:0/3884852914 >> 172.21.15.107:0/3884852914 conn(0x7ff9280c5cd0 msgr2=0x7ff9280c60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:44.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 -- 172.21.15.107:0/3884852914 shutdown_connections 2023-10-25T00:12:44.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 -- 172.21.15.107:0/3884852914 wait complete. 2023-10-25T00:12:44.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 Processor -- start 2023-10-25T00:12:44.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 -- start start 2023-10-25T00:12:44.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff92806b8a0 0x7ff9280c03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:44.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff928071a40 0x7ff92814c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:44.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff928148110 0x7ff92814c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:44.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff928145f60 con 0x7ff928148110 2023-10-25T00:12:44.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9281552f0 con 0x7ff928071a40 2023-10-25T00:12:44.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff928146440 con 0x7ff92806b8a0 2023-10-25T00:12:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92d52a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff928148110 0x7ff92814c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92d52a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff928148110 0x7ff92814c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37042/0 (socket says 172.21.15.107:37042) 2023-10-25T00:12:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92d52a640 1 -- 172.21.15.107:0/493173434 learned_addr learned my addr 172.21.15.107:0/493173434 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:12:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff927fff640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff928071a40 0x7ff92814c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92d52a640 1 -- 172.21.15.107:0/493173434 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff92806b8a0 msgr2=0x7ff9280c03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92cd29640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff92806b8a0 0x7ff9280c03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92d52a640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff92806b8a0 0x7ff9280c03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92d52a640 1 -- 172.21.15.107:0/493173434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff928071a40 msgr2=0x7ff92814c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92d52a640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff928071a40 0x7ff92814c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92d52a640 1 -- 172.21.15.107:0/493173434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91400b770 con 0x7ff928148110 2023-10-25T00:12:44.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff927fff640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff928071a40 0x7ff92814c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:12:44.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92d52a640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff928148110 0x7ff92814c780 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7ff920001570 tx=0x7ff92000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:44.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff9257fa640 1 -- 172.21.15.107:0/493173434 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff920003db0 con 0x7ff928148110 2023-10-25T00:12:44.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff9257fa640 1 -- 172.21.15.107:0/493173434 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff920005070 con 0x7ff928148110 2023-10-25T00:12:44.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.544+0000 7ff92ef8d640 1 -- 172.21.15.107:0/493173434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff928146720 con 0x7ff928148110 2023-10-25T00:12:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.548+0000 7ff9257fa640 1 -- 172.21.15.107:0/493173434 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff920005420 con 0x7ff928148110 2023-10-25T00:12:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.548+0000 7ff92ef8d640 1 -- 172.21.15.107:0/493173434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92814ead0 con 0x7ff928148110 2023-10-25T00:12:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.548+0000 7ff92ef8d640 1 -- 172.21.15.107:0/493173434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92814bb80 con 0x7ff928148110 2023-10-25T00:12:44.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.548+0000 7ff9257fa640 1 -- 172.21.15.107:0/493173434 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff920005580 con 0x7ff928148110 2023-10-25T00:12:44.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.548+0000 7ff9257fa640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff90c065e60 0x7ff90c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:12:44.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.548+0000 7ff9257fa640 1 -- 172.21.15.107:0/493173434 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7ff920083fb0 con 0x7ff928148110 2023-10-25T00:12:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.548+0000 7ff92cd29640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff90c065e60 0x7ff90c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:12:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.548+0000 7ff92cd29640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff90c065e60 0x7ff90c068320 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff9280ad0a0 tx=0x7ff91c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:12:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.556+0000 7ff9257fa640 1 -- 172.21.15.107:0/493173434 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff92814bb80 con 0x7ff928148110 2023-10-25T00:12:44.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.772+0000 7ff92ef8d640 1 -- 172.21.15.107:0/493173434 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff92814bf60 con 0x7ff928148110 2023-10-25T00:12:44.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.772+0000 7ff9257fa640 1 -- 172.21.15.107:0/493173434 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5774 (secure 0 0 0) 0x7ff92814bf60 con 0x7ff928148110 2023-10-25T00:12:44.772 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:12:44.773 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:12:44.773 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 170 2023-10-25T00:12:44.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 -- 172.21.15.107:0/493173434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff90c065e60 msgr2=0x7ff90c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff90c065e60 0x7ff90c068320 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff9280ad0a0 tx=0x7ff91c008040 comp rx=0 tx=0).stop 2023-10-25T00:12:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 -- 172.21.15.107:0/493173434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff928148110 msgr2=0x7ff92814c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:12:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff928148110 0x7ff92814c780 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7ff920001570 tx=0x7ff92000cd20 comp rx=0 tx=0).stop 2023-10-25T00:12:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 -- 172.21.15.107:0/493173434 shutdown_connections 2023-10-25T00:12:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff928148110 0x7ff92814c780 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff928071a40 0x7ff92814c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff90c065e60 0x7ff90c068320 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 --2- 172.21.15.107:0/493173434 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff92806b8a0 0x7ff9280c03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:12:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 -- 172.21.15.107:0/493173434 >> 172.21.15.107:0/493173434 conn(0x7ff9280c5cd0 msgr2=0x7ff9280c60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:12:44.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 -- 172.21.15.107:0/493173434 shutdown_connections 2023-10-25T00:12:44.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:12:44.776+0000 7ff92ef8d640 1 -- 172.21.15.107:0/493173434 wait complete. 2023-10-25T00:12:44.792 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2023-10-25T00:12:44.792 INFO:tasks.mds_thrash.fs.[b]:waiting for 48 secs before thrashing 2023-10-25T00:13:03.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc28a211640 1 Processor -- start 2023-10-25T00:13:03.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc28a211640 1 -- start start 2023-10-25T00:13:03.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc28a211640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2840720c0 0x7fc2840c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:03.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc28a211640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc284157e50 0x7fc2841582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:03.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc28a211640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc284154860 con 0x7fc28413dc20 2023-10-25T00:13:03.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc28a211640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc284158800 con 0x7fc284157e50 2023-10-25T00:13:03.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc28a211640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2840c1f10 con 0x7fc2840720c0 2023-10-25T00:13:03.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc27affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2840720c0 0x7fc2840c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:03.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc283fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc284157e50 0x7fc2841582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:03.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc27affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2840720c0 0x7fc2840c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:50200/0 (socket says 172.21.15.107:50200) 2023-10-25T00:13:03.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc27affd640 1 -- 172.21.15.107:0/368626545 learned_addr learned my addr 172.21.15.107:0/368626545 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:03.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc2837fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc28413dc20 0x7fc28406df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:56080/0 (socket says 172.21.15.107:56080) 2023-10-25T00:13:03.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc283fff640 1 -- 172.21.15.107:0/368626545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2840720c0 msgr2=0x7fc2840c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:03.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc283fff640 1 --2- 172.21.15.107:0/368626545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2840720c0 0x7fc2840c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:03.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc283fff640 1 -- 172.21.15.107:0/368626545 >> v1:172.21.15.110:6789/0 conn(0x7fc28413dc20 legacy=0x7fc28406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:03.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc283fff640 1 -- 172.21.15.107:0/368626545 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc284154b20 con 0x7fc284157e50 2023-10-25T00:13:03.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.663+0000 7fc27affd640 1 --2- 172.21.15.107:0/368626545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2840720c0 0x7fc2840c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:13:03.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc283fff640 1 --2- 172.21.15.107:0/368626545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc284157e50 0x7fc2841582c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc27c009d90 tx=0x7fc27c01c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c4d47f77a2b3532 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:03.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc282ffd640 1 -- 172.21.15.107:0/368626545 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27c0a57e0 con 0x7fc284157e50 2023-10-25T00:13:03.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc282ffd640 1 -- 172.21.15.107:0/368626545 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27c0a5940 con 0x7fc284157e50 2023-10-25T00:13:03.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc282ffd640 1 -- 172.21.15.107:0/368626545 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27c0a5cd0 con 0x7fc284157e50 2023-10-25T00:13:03.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc28a211640 1 -- 172.21.15.107:0/368626545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc284157e50 msgr2=0x7fc2841582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:03.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc28a211640 1 --2- 172.21.15.107:0/368626545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc284157e50 0x7fc2841582c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc27c009d90 tx=0x7fc27c01c830 comp rx=0 tx=0).stop 2023-10-25T00:13:03.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc28a211640 1 -- 172.21.15.107:0/368626545 shutdown_connections 2023-10-25T00:13:03.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc28a211640 1 --2- 172.21.15.107:0/368626545 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc284157e50 0x7fc2841582c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:03.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc28a211640 1 --2- 172.21.15.107:0/368626545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2840720c0 0x7fc2840c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc28a211640 1 -- 172.21.15.107:0/368626545 >> 172.21.15.107:0/368626545 conn(0x7fc2840c39a0 msgr2=0x7fc2840c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc28a211640 1 -- 172.21.15.107:0/368626545 shutdown_connections 2023-10-25T00:13:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.667+0000 7fc28a211640 1 -- 172.21.15.107:0/368626545 wait complete. 2023-10-25T00:13:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc28a211640 1 Processor -- start 2023-10-25T00:13:03.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc28a211640 1 -- start start 2023-10-25T00:13:03.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc28a211640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840720c0 0x7fc2840c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:03.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc28a211640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc28413dc20 0x7fc2840c6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:03.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc28a211640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc284157e50 0x7fc284156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc28a211640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28414f4a0 con 0x7fc2840720c0 2023-10-25T00:13:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc28a211640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28406c200 con 0x7fc284157e50 2023-10-25T00:13:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc28a211640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2840c6a50 con 0x7fc28413dc20 2023-10-25T00:13:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc2837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840720c0 0x7fc2840c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc2837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840720c0 0x7fc2840c5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52630/0 (socket says 172.21.15.107:52630) 2023-10-25T00:13:03.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc2837fe640 1 -- 172.21.15.107:0/3345138201 learned_addr learned my addr 172.21.15.107:0/3345138201 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:03.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc283fff640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc284157e50 0x7fc284156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:03.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc27affd640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc28413dc20 0x7fc2840c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:03.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc2837fe640 1 -- 172.21.15.107:0/3345138201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc28413dc20 msgr2=0x7fc2840c6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:03.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc2837fe640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc28413dc20 0x7fc2840c6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:03.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc2837fe640 1 -- 172.21.15.107:0/3345138201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc284157e50 msgr2=0x7fc284156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:03.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc2837fe640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc284157e50 0x7fc284156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc2837fe640 1 -- 172.21.15.107:0/3345138201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c0094d0 con 0x7fc2840720c0 2023-10-25T00:13:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc2837fe640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840720c0 0x7fc2840c5fd0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fc27400c430 tx=0x7fc27400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc280ff9640 1 -- 172.21.15.107:0/3345138201 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2740074e0 con 0x7fc2840720c0 2023-10-25T00:13:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc280ff9640 1 -- 172.21.15.107:0/3345138201 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc274007640 con 0x7fc2840720c0 2023-10-25T00:13:03.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc28a211640 1 -- 172.21.15.107:0/3345138201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc27c009a70 con 0x7fc2840720c0 2023-10-25T00:13:03.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc280ff9640 1 -- 172.21.15.107:0/3345138201 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc274004a10 con 0x7fc2840720c0 2023-10-25T00:13:03.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc28a211640 1 -- 172.21.15.107:0/3345138201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc284155b10 con 0x7fc2840720c0 2023-10-25T00:13:03.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc28a211640 1 -- 172.21.15.107:0/3345138201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2840746e0 con 0x7fc2840720c0 2023-10-25T00:13:03.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.671+0000 7fc280ff9640 1 -- 172.21.15.107:0/3345138201 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc274004b70 con 0x7fc2840720c0 2023-10-25T00:13:03.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.675+0000 7fc280ff9640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc264065e60 0x7fc264068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:03.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.675+0000 7fc27affd640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc264065e60 0x7fc264068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:03.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.675+0000 7fc280ff9640 1 -- 172.21.15.107:0/3345138201 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7fc274083fe0 con 0x7fc2840720c0 2023-10-25T00:13:03.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.675+0000 7fc27affd640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc264065e60 0x7fc264068320 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc284154830 tx=0x7fc26c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:03.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.679+0000 7fc280ff9640 1 -- 172.21.15.107:0/3345138201 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc274050820 con 0x7fc2840720c0 2023-10-25T00:13:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.923+0000 7fc28a211640 1 -- 172.21.15.107:0/3345138201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc284157c30 con 0x7fc2840720c0 2023-10-25T00:13:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc280ff9640 1 -- 172.21.15.107:0/3345138201 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5774 (secure 0 0 0) 0x7fc274053e40 con 0x7fc2840720c0 2023-10-25T00:13:03.925 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:03.925 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:03.926 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 170 2023-10-25T00:13:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 -- 172.21.15.107:0/3345138201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc264065e60 msgr2=0x7fc264068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc264065e60 0x7fc264068320 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc284154830 tx=0x7fc26c008040 comp rx=0 tx=0).stop 2023-10-25T00:13:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 -- 172.21.15.107:0/3345138201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840720c0 msgr2=0x7fc2840c5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840720c0 0x7fc2840c5fd0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fc27400c430 tx=0x7fc27400c8f0 comp rx=0 tx=0).stop 2023-10-25T00:13:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 -- 172.21.15.107:0/3345138201 shutdown_connections 2023-10-25T00:13:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc284157e50 0x7fc284156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc264065e60 0x7fc264068320 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc28413dc20 0x7fc2840c6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 --2- 172.21.15.107:0/3345138201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2840720c0 0x7fc2840c5fd0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 -- 172.21.15.107:0/3345138201 >> 172.21.15.107:0/3345138201 conn(0x7fc2840c39a0 msgr2=0x7fc28406cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 -- 172.21.15.107:0/3345138201 shutdown_connections 2023-10-25T00:13:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:03.927+0000 7fc28a211640 1 -- 172.21.15.107:0/3345138201 wait complete. 2023-10-25T00:13:03.945 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2023-10-25T00:13:03.945 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-25T00:13:03.945 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:13:03.962 INFO:tasks.ceph.mds.d:Stopped 2023-10-25T00:13:03.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:04.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f004b6a0640 1 Processor -- start 2023-10-25T00:13:04.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f004b6a0640 1 -- start start 2023-10-25T00:13:04.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f004b6a0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0044152040 0x7f00440c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f004b6a0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f00440c8100 0x7f00440abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f004b6a0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0044155680 con 0x7f004406a4b0 2023-10-25T00:13:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f004b6a0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0044154a70 con 0x7f00440c8100 2023-10-25T00:13:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f004b6a0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0044154d50 con 0x7f0044152040 2023-10-25T00:13:04.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f0048c3b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f00440c8100 0x7f00440abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:04.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f0048c3b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f00440c8100 0x7f00440abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:44658/0 (socket says 172.21.15.107:44658) 2023-10-25T00:13:04.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f0048c3b640 1 -- 172.21.15.107:0/2593077140 learned_addr learned my addr 172.21.15.107:0/2593077140 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:04.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f004943c640 1 --2- 172.21.15.107:0/2593077140 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0044152040 0x7f00440c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:04.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f0048c3b640 1 -- 172.21.15.107:0/2593077140 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0044152040 msgr2=0x7f00440c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:04.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f0048c3b640 1 --2- 172.21.15.107:0/2593077140 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0044152040 0x7f00440c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:04.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f0048c3b640 1 -- 172.21.15.107:0/2593077140 >> v1:172.21.15.110:6789/0 conn(0x7f004406a4b0 legacy=0x7f00440ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:04.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f0048c3b640 1 -- 172.21.15.107:0/2593077140 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0044155030 con 0x7f00440c8100 2023-10-25T00:13:04.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.083+0000 7f004943c640 1 --2- 172.21.15.107:0/2593077140 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0044152040 0x7f00440c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:13:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f0048c3b640 1 --2- 172.21.15.107:0/2593077140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00440c8100 0x7f00440abd00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0034009870 tx=0x7f003401c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf9e18caea667122 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f003bfff640 1 -- 172.21.15.107:0/2593077140 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00340ac620 con 0x7f00440c8100 2023-10-25T00:13:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f003bfff640 1 -- 172.21.15.107:0/2593077140 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00340ac780 con 0x7f00440c8100 2023-10-25T00:13:04.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f003bfff640 1 -- 172.21.15.107:0/2593077140 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00340acb10 con 0x7f00440c8100 2023-10-25T00:13:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f004b6a0640 1 -- 172.21.15.107:0/2593077140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00440c8100 msgr2=0x7f00440abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f004b6a0640 1 --2- 172.21.15.107:0/2593077140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00440c8100 0x7f00440abd00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0034009870 tx=0x7f003401c4b0 comp rx=0 tx=0).stop 2023-10-25T00:13:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f004b6a0640 1 -- 172.21.15.107:0/2593077140 shutdown_connections 2023-10-25T00:13:04.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f004b6a0640 1 --2- 172.21.15.107:0/2593077140 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00440c8100 0x7f00440abd00 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f004b6a0640 1 --2- 172.21.15.107:0/2593077140 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0044152040 0x7f00440c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f004b6a0640 1 -- 172.21.15.107:0/2593077140 >> 172.21.15.107:0/2593077140 conn(0x7f00440ca200 msgr2=0x7f00440ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f004b6a0640 1 -- 172.21.15.107:0/2593077140 shutdown_connections 2023-10-25T00:13:04.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f004b6a0640 1 -- 172.21.15.107:0/2593077140 wait complete. 2023-10-25T00:13:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f004b6a0640 1 Processor -- start 2023-10-25T00:13:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.087+0000 7f004b6a0640 1 -- start start 2023-10-25T00:13:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f004b6a0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f004406a4b0 0x7f00441518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:04.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f004b6a0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00440c8100 0x7f004414cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f004b6a0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0044152040 0x7f004414d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f004b6a0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00441f0990 con 0x7f0044152040 2023-10-25T00:13:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f004b6a0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0044151e00 con 0x7f00440c8100 2023-10-25T00:13:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f004b6a0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00441f0be0 con 0x7f004406a4b0 2023-10-25T00:13:04.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0049c3d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0044152040 0x7f004414d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0049c3d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0044152040 0x7f004414d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52636/0 (socket says 172.21.15.107:52636) 2023-10-25T00:13:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0049c3d640 1 -- 172.21.15.107:0/671396483 learned_addr learned my addr 172.21.15.107:0/671396483 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0048c3b640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00440c8100 0x7f004414cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f004943c640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f004406a4b0 0x7f00441518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:04.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0049c3d640 1 -- 172.21.15.107:0/671396483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f004406a4b0 msgr2=0x7f00441518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0049c3d640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f004406a4b0 0x7f00441518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0049c3d640 1 -- 172.21.15.107:0/671396483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00440c8100 msgr2=0x7f004414cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0049c3d640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00440c8100 0x7f004414cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:04.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0049c3d640 1 -- 172.21.15.107:0/671396483 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0040009420 con 0x7f0044152040 2023-10-25T00:13:04.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0049c3d640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0044152040 0x7f004414d140 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f003c00c730 tx=0x7f003c00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:04.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0039ffb640 1 -- 172.21.15.107:0/671396483 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c006d60 con 0x7f0044152040 2023-10-25T00:13:04.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0039ffb640 1 -- 172.21.15.107:0/671396483 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003c003980 con 0x7f0044152040 2023-10-25T00:13:04.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f004b6a0640 1 -- 172.21.15.107:0/671396483 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00340094d0 con 0x7f0044152040 2023-10-25T00:13:04.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0039ffb640 1 -- 172.21.15.107:0/671396483 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003c003d30 con 0x7f0044152040 2023-10-25T00:13:04.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f004b6a0640 1 -- 172.21.15.107:0/671396483 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00441f11a0 con 0x7f0044152040 2023-10-25T00:13:04.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f004b6a0640 1 -- 172.21.15.107:0/671396483 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00440746e0 con 0x7f0044152040 2023-10-25T00:13:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0039ffb640 1 -- 172.21.15.107:0/671396483 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f003c0050d0 con 0x7f0044152040 2023-10-25T00:13:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0039ffb640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0028065e90 0x7f0028068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:04.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f0039ffb640 1 -- 172.21.15.107:0/671396483 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7f003c084020 con 0x7f0044152040 2023-10-25T00:13:04.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.091+0000 7f004943c640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0028065e90 0x7f0028068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:04.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.095+0000 7f004943c640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0028065e90 0x7f0028068350 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0040009cf0 tx=0x7f0040009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:04.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.099+0000 7f0039ffb640 1 -- 172.21.15.107:0/671396483 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f003c0543a0 con 0x7f0044152040 2023-10-25T00:13:04.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 -- 172.21.15.107:0/671396483 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f004406b170 con 0x7f0044152040 2023-10-25T00:13:04.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f0039ffb640 1 -- 172.21.15.107:0/671396483 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f003c053f30 con 0x7f0044152040 2023-10-25T00:13:04.286 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:04.287 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:04.287 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 170 2023-10-25T00:13:04.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 -- 172.21.15.107:0/671396483 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0028065e90 msgr2=0x7f0028068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:04.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0028065e90 0x7f0028068350 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f0040009cf0 tx=0x7f0040009190 comp rx=0 tx=0).stop 2023-10-25T00:13:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 -- 172.21.15.107:0/671396483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0044152040 msgr2=0x7f004414d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0044152040 0x7f004414d140 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f003c00c730 tx=0x7f003c00cb30 comp rx=0 tx=0).stop 2023-10-25T00:13:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 -- 172.21.15.107:0/671396483 shutdown_connections 2023-10-25T00:13:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0044152040 0x7f004414d140 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f00440c8100 0x7f004414cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0028065e90 0x7f0028068350 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 --2- 172.21.15.107:0/671396483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f004406a4b0 0x7f00441518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 -- 172.21.15.107:0/671396483 >> 172.21.15.107:0/671396483 conn(0x7f00440ca200 msgr2=0x7f00440abf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.287+0000 7f004b6a0640 1 -- 172.21.15.107:0/671396483 shutdown_connections 2023-10-25T00:13:04.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:04.291+0000 7f004b6a0640 1 -- 172.21.15.107:0/671396483 wait complete. 2023-10-25T00:13:04.310 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:13:06.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:06.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.439+0000 7f2ab2c08640 1 Processor -- start 2023-10-25T00:13:06.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.439+0000 7f2ab2c08640 1 -- start start 2023-10-25T00:13:06.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab2c08640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2aac14b600 0x7f2aac151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:06.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab2c08640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac06af70 0x7f2aac06c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:06.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab2c08640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2aac06a230 0x7f2aac1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:06.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab2c08640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac0c1860 con 0x7f2aac14b600 2023-10-25T00:13:06.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab2c08640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aac154d50 con 0x7f2aac06a230 2023-10-25T00:13:06.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab2c08640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac1524f0 con 0x7f2aac06af70 2023-10-25T00:13:06.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab09a4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2aac14b600 0x7f2aac151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:06.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab09a4640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2aac14b600 0x7f2aac151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52642/0 (socket says 172.21.15.107:52642) 2023-10-25T00:13:06.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab09a4640 1 -- 172.21.15.107:0/1121566421 learned_addr learned my addr 172.21.15.107:0/1121566421 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:06.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab09a4640 1 -- 172.21.15.107:0/1121566421 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac06af70 msgr2=0x7f2aac06c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:06.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab09a4640 1 --2- 172.21.15.107:0/1121566421 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac06af70 0x7f2aac06c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:06.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab09a4640 1 -- 172.21.15.107:0/1121566421 >> v2:172.21.15.110:3300/0 conn(0x7f2aac06a230 msgr2=0x7f2aac1551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:06.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab09a4640 1 --2- 172.21.15.107:0/1121566421 >> v2:172.21.15.110:3300/0 conn(0x7f2aac06a230 0x7f2aac1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:06.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab09a4640 1 -- 172.21.15.107:0/1121566421 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac14b9e0 con 0x7f2aac14b600 2023-10-25T00:13:06.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2ab09a4640 1 --2- 172.21.15.107:0/1121566421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aac14b600 0x7f2aac151fb0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f2aac06d0a0 tx=0x7f2aa401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad3c9fbe195167c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:06.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2aab7fe640 1 -- 172.21.15.107:0/1121566421 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa40a5790 con 0x7f2aac14b600 2023-10-25T00:13:06.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2aab7fe640 1 -- 172.21.15.107:0/1121566421 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa40a58f0 con 0x7f2aac14b600 2023-10-25T00:13:06.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.443+0000 7f2aab7fe640 1 -- 172.21.15.107:0/1121566421 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa40a5c80 con 0x7f2aac14b600 2023-10-25T00:13:06.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- 172.21.15.107:0/1121566421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aac14b600 msgr2=0x7f2aac151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:06.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 --2- 172.21.15.107:0/1121566421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aac14b600 0x7f2aac151fb0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f2aac06d0a0 tx=0x7f2aa401c910 comp rx=0 tx=0).stop 2023-10-25T00:13:06.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- 172.21.15.107:0/1121566421 shutdown_connections 2023-10-25T00:13:06.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 --2- 172.21.15.107:0/1121566421 >> v2:172.21.15.110:3300/0 conn(0x7f2aac06a230 0x7f2aac1551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:06.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 --2- 172.21.15.107:0/1121566421 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac06af70 0x7f2aac06c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:06.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 --2- 172.21.15.107:0/1121566421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aac14b600 0x7f2aac151fb0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:06.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- 172.21.15.107:0/1121566421 >> 172.21.15.107:0/1121566421 conn(0x7f2aac0c2950 msgr2=0x7f2aac0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:06.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- 172.21.15.107:0/1121566421 shutdown_connections 2023-10-25T00:13:06.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- 172.21.15.107:0/1121566421 wait complete. 2023-10-25T00:13:06.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 Processor -- start 2023-10-25T00:13:06.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- start start 2023-10-25T00:13:06.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac06a230 0x7f2aac147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:06.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aac06af70 0x7f2aac144900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:06.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac14b600 0x7f2aac140400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:06.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac1458b0 con 0x7f2aac06af70 2023-10-25T00:13:06.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac151ef0 con 0x7f2aac06a230 2023-10-25T00:13:06.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac145da0 con 0x7f2aac14b600 2023-10-25T00:13:06.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aabfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aac06af70 0x7f2aac144900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:06.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aabfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aac06af70 0x7f2aac144900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52650/0 (socket says 172.21.15.107:52650) 2023-10-25T00:13:06.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aabfff640 1 -- 172.21.15.107:0/696559456 learned_addr learned my addr 172.21.15.107:0/696559456 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:06.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab09a4640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac06a230 0x7f2aac147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:06.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab11a5640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac14b600 0x7f2aac140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:06.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aabfff640 1 -- 172.21.15.107:0/696559456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac14b600 msgr2=0x7f2aac140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:06.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aabfff640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac14b600 0x7f2aac140400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:06.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aabfff640 1 -- 172.21.15.107:0/696559456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac06a230 msgr2=0x7f2aac147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:06.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aabfff640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac06a230 0x7f2aac147310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:06.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aabfff640 1 -- 172.21.15.107:0/696559456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa4009b30 con 0x7f2aac06af70 2023-10-25T00:13:06.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab09a4640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac06a230 0x7f2aac147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:13:06.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab11a5640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac14b600 0x7f2aac140400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:13:06.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aabfff640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aac06af70 0x7f2aac144900 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400b4b0 tx=0x7f2a9400b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:06.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aa97fa640 1 -- 172.21.15.107:0/696559456 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94010610 con 0x7f2aac06af70 2023-10-25T00:13:06.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aa97fa640 1 -- 172.21.15.107:0/696559456 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a94010770 con 0x7f2aac06af70 2023-10-25T00:13:06.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2aa97fa640 1 -- 172.21.15.107:0/696559456 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a94010b20 con 0x7f2aac06af70 2023-10-25T00:13:06.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- 172.21.15.107:0/696559456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aac1ee9c0 con 0x7f2aac06af70 2023-10-25T00:13:06.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- 172.21.15.107:0/696559456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aac145ac0 con 0x7f2aac06af70 2023-10-25T00:13:06.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.447+0000 7f2ab2c08640 1 -- 172.21.15.107:0/696559456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aac0746e0 con 0x7f2aac06af70 2023-10-25T00:13:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.451+0000 7f2aa97fa640 1 -- 172.21.15.107:0/696559456 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2a94010c80 con 0x7f2aac06af70 2023-10-25T00:13:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.451+0000 7f2aa97fa640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a90065e60 0x7f2a90068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.451+0000 7f2aa97fa640 1 -- 172.21.15.107:0/696559456 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7f2a940842d0 con 0x7f2aac06af70 2023-10-25T00:13:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.451+0000 7f2ab09a4640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a90065e60 0x7f2a90068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:06.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.451+0000 7f2ab09a4640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a90065e60 0x7f2a90068320 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2aa40ae000 tx=0x7f2aa40a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:06.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.455+0000 7f2aa97fa640 1 -- 172.21.15.107:0/696559456 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2a94050b10 con 0x7f2aac06af70 2023-10-25T00:13:06.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.639+0000 7f2ab2c08640 1 -- 172.21.15.107:0/696559456 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2aac069fa0 con 0x7f2aac06af70 2023-10-25T00:13:06.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2aa97fa640 1 -- 172.21.15.107:0/696559456 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f2a94014090 con 0x7f2aac06af70 2023-10-25T00:13:06.641 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:06.641 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:06.642 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 170 2023-10-25T00:13:06.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 -- 172.21.15.107:0/696559456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a90065e60 msgr2=0x7f2a90068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:06.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a90065e60 0x7f2a90068320 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2aa40ae000 tx=0x7f2aa40a7070 comp rx=0 tx=0).stop 2023-10-25T00:13:06.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 -- 172.21.15.107:0/696559456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aac06af70 msgr2=0x7f2aac144900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:06.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aac06af70 0x7f2aac144900 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400b4b0 tx=0x7f2a9400b970 comp rx=0 tx=0).stop 2023-10-25T00:13:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 -- 172.21.15.107:0/696559456 shutdown_connections 2023-10-25T00:13:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a90065e60 0x7f2a90068320 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2aac14b600 0x7f2aac140400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2aac06af70 0x7f2aac144900 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 --2- 172.21.15.107:0/696559456 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2aac06a230 0x7f2aac147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:06.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 -- 172.21.15.107:0/696559456 >> 172.21.15.107:0/696559456 conn(0x7f2aac0c2950 msgr2=0x7f2aac0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:06.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 -- 172.21.15.107:0/696559456 shutdown_connections 2023-10-25T00:13:06.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:06.643+0000 7f2ab2c08640 1 -- 172.21.15.107:0/696559456 wait complete. 2023-10-25T00:13:06.659 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:13:08.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:08.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa651c9640 1 Processor -- start 2023-10-25T00:13:08.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa651c9640 1 -- start start 2023-10-25T00:13:08.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa651c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa60152040 0x7ffa600c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:08.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa651c9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffa6006a4b0 0x7ffa600ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:08.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa651c9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa60155680 con 0x7ffa600c8100 2023-10-25T00:13:08.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa651c9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa60154a70 con 0x7ffa6006a4b0 2023-10-25T00:13:08.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa651c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa60154d50 con 0x7ffa60152040 2023-10-25T00:13:08.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa5f59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffa6006a4b0 0x7ffa600ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:08.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa5659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ffa600c8100 0x7ffa600abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47652/0 (socket says 172.21.15.107:47652) 2023-10-25T00:13:08.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa5f59e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ffa6006a4b0 0x7ffa600ac410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52666/0 (socket says 172.21.15.107:52666) 2023-10-25T00:13:08.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa5659c640 1 -- 172.21.15.107:0/1681293875 learned_addr learned my addr 172.21.15.107:0/1681293875 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:13:08.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa5ed9d640 1 --2- 172.21.15.107:0/1681293875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa60152040 0x7ffa600c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:08.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa5f59e640 1 -- 172.21.15.107:0/1681293875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa60152040 msgr2=0x7ffa600c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:08.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa5f59e640 1 --2- 172.21.15.107:0/1681293875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa60152040 0x7ffa600c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:08.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa5f59e640 1 -- 172.21.15.107:0/1681293875 >> v1:172.21.15.107:6789/0 conn(0x7ffa600c8100 legacy=0x7ffa600abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:08.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.791+0000 7ffa5f59e640 1 -- 172.21.15.107:0/1681293875 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa60155030 con 0x7ffa6006a4b0 2023-10-25T00:13:08.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5f59e640 1 --2- 172.21.15.107:0/1681293875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6006a4b0 0x7ffa600ac410 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009d90 tx=0x7ffa5801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99557e49125844ed server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:08.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5e59c640 1 -- 172.21.15.107:0/1681293875 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa580a5890 con 0x7ffa6006a4b0 2023-10-25T00:13:08.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5e59c640 1 -- 172.21.15.107:0/1681293875 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa580a59f0 con 0x7ffa6006a4b0 2023-10-25T00:13:08.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5e59c640 1 -- 172.21.15.107:0/1681293875 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa580a5d80 con 0x7ffa6006a4b0 2023-10-25T00:13:08.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 -- 172.21.15.107:0/1681293875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6006a4b0 msgr2=0x7ffa600ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:08.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 --2- 172.21.15.107:0/1681293875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6006a4b0 0x7ffa600ac410 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7ffa58009d90 tx=0x7ffa5801c8c0 comp rx=0 tx=0).stop 2023-10-25T00:13:08.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 -- 172.21.15.107:0/1681293875 shutdown_connections 2023-10-25T00:13:08.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 --2- 172.21.15.107:0/1681293875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6006a4b0 0x7ffa600ac410 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:08.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 --2- 172.21.15.107:0/1681293875 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa60152040 0x7ffa600c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:08.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 -- 172.21.15.107:0/1681293875 >> 172.21.15.107:0/1681293875 conn(0x7ffa600ca200 msgr2=0x7ffa600ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:08.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 -- 172.21.15.107:0/1681293875 shutdown_connections 2023-10-25T00:13:08.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 -- 172.21.15.107:0/1681293875 wait complete. 2023-10-25T00:13:08.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 Processor -- start 2023-10-25T00:13:08.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 -- start start 2023-10-25T00:13:08.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6006a4b0 0x7ffa6014fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:08.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa600c8100 0x7ffa600aaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:08.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa60152040 0x7ffa601f0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:08.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600aaaa0 con 0x7ffa6006a4b0 2023-10-25T00:13:08.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa60151c10 con 0x7ffa60152040 2023-10-25T00:13:08.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa600c0780 con 0x7ffa600c8100 2023-10-25T00:13:08.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6006a4b0 0x7ffa6014fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:08.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6006a4b0 0x7ffa6014fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52668/0 (socket says 172.21.15.107:52668) 2023-10-25T00:13:08.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5ed9d640 1 -- 172.21.15.107:0/3988081054 learned_addr learned my addr 172.21.15.107:0/3988081054 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:08.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5f59e640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa60152040 0x7ffa601f0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:08.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5659c640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa600c8100 0x7ffa600aaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:08.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5ed9d640 1 -- 172.21.15.107:0/3988081054 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa600c8100 msgr2=0x7ffa600aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:08.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5ed9d640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa600c8100 0x7ffa600aaf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:08.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5ed9d640 1 -- 172.21.15.107:0/3988081054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa60152040 msgr2=0x7ffa601f0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:08.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5ed9d640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa60152040 0x7ffa601f0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:08.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5ed9d640 1 -- 172.21.15.107:0/3988081054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa58009a70 con 0x7ffa6006a4b0 2023-10-25T00:13:08.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa5ed9d640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6006a4b0 0x7ffa6014fc60 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7ffa5000d800 tx=0x7ffa5000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:08.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa57fff640 1 -- 172.21.15.107:0/3988081054 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5000b6c0 con 0x7ffa6006a4b0 2023-10-25T00:13:08.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa57fff640 1 -- 172.21.15.107:0/3988081054 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5000b820 con 0x7ffa6006a4b0 2023-10-25T00:13:08.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 -- 172.21.15.107:0/3988081054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa600c0a60 con 0x7ffa6006a4b0 2023-10-25T00:13:08.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa57fff640 1 -- 172.21.15.107:0/3988081054 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5000bb90 con 0x7ffa6006a4b0 2023-10-25T00:13:08.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.795+0000 7ffa651c9640 1 -- 172.21.15.107:0/3988081054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa601501a0 con 0x7ffa6006a4b0 2023-10-25T00:13:08.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.799+0000 7ffa651c9640 1 -- 172.21.15.107:0/3988081054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa600c05e0 con 0x7ffa6006a4b0 2023-10-25T00:13:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.799+0000 7ffa57fff640 1 -- 172.21.15.107:0/3988081054 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ffa5000bcf0 con 0x7ffa6006a4b0 2023-10-25T00:13:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.799+0000 7ffa57fff640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa40065e60 0x7ffa40068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.799+0000 7ffa5659c640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa40065e60 0x7ffa40068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.799+0000 7ffa57fff640 1 -- 172.21.15.107:0/3988081054 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7ffa50084c70 con 0x7ffa6006a4b0 2023-10-25T00:13:08.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.799+0000 7ffa5659c640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa40065e60 0x7ffa40068320 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ffa600ac0e0 tx=0x7ffa48009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:08.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.803+0000 7ffa57fff640 1 -- 172.21.15.107:0/3988081054 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ffa600c05e0 con 0x7ffa6006a4b0 2023-10-25T00:13:08.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.991+0000 7ffa651c9640 1 -- 172.21.15.107:0/3988081054 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa600c05e0 con 0x7ffa6006a4b0 2023-10-25T00:13:08.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.991+0000 7ffa57fff640 1 -- 172.21.15.107:0/3988081054 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5774 (secure 0 0 0) 0x7ffa600c05e0 con 0x7ffa6006a4b0 2023-10-25T00:13:08.990 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:08.990 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:08.991 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 170 2023-10-25T00:13:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.991+0000 7ffa651c9640 1 -- 172.21.15.107:0/3988081054 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa40065e60 msgr2=0x7ffa40068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.991+0000 7ffa651c9640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa40065e60 0x7ffa40068320 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ffa600ac0e0 tx=0x7ffa48009040 comp rx=0 tx=0).stop 2023-10-25T00:13:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.991+0000 7ffa651c9640 1 -- 172.21.15.107:0/3988081054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6006a4b0 msgr2=0x7ffa6014fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.991+0000 7ffa651c9640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6006a4b0 0x7ffa6014fc60 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7ffa5000d800 tx=0x7ffa5000dcc0 comp rx=0 tx=0).stop 2023-10-25T00:13:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.991+0000 7ffa651c9640 1 -- 172.21.15.107:0/3988081054 shutdown_connections 2023-10-25T00:13:08.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.991+0000 7ffa651c9640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa60152040 0x7ffa601f0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.991+0000 7ffa651c9640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa40065e60 0x7ffa40068320 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.991+0000 7ffa651c9640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa600c8100 0x7ffa600aaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.995+0000 7ffa651c9640 1 --2- 172.21.15.107:0/3988081054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa6006a4b0 0x7ffa6014fc60 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.995+0000 7ffa651c9640 1 -- 172.21.15.107:0/3988081054 >> 172.21.15.107:0/3988081054 conn(0x7ffa600ca200 msgr2=0x7ffa6006a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:08.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.995+0000 7ffa651c9640 1 -- 172.21.15.107:0/3988081054 shutdown_connections 2023-10-25T00:13:08.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:08.995+0000 7ffa651c9640 1 -- 172.21.15.107:0/3988081054 wait complete. 2023-10-25T00:13:09.009 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:13:11.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:11.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.139+0000 7f78f3ea8640 1 Processor -- start 2023-10-25T00:13:11.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.139+0000 7f78f3ea8640 1 -- start start 2023-10-25T00:13:11.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f3ea8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f78ec152040 0x7f78ec0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:11.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f3ea8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f78ec0c8100 0x7f78ec0abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:11.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f3ea8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ec155680 con 0x7f78ec06a4b0 2023-10-25T00:13:11.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f3ea8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78ec154a70 con 0x7f78ec0c8100 2023-10-25T00:13:11.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f3ea8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78ec154d50 con 0x7f78ec152040 2023-10-25T00:13:11.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f1443640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f78ec0c8100 0x7f78ec0abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:11.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f1443640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f78ec0c8100 0x7f78ec0abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52684/0 (socket says 172.21.15.107:52684) 2023-10-25T00:13:11.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f1443640 1 -- 172.21.15.107:0/2788309711 learned_addr learned my addr 172.21.15.107:0/2788309711 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f1c44640 1 --2- 172.21.15.107:0/2788309711 >> v2:172.21.15.110:3300/0 conn(0x7f78ec152040 0x7f78ec0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f1443640 1 -- 172.21.15.107:0/2788309711 >> v2:172.21.15.110:3300/0 conn(0x7f78ec152040 msgr2=0x7f78ec0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f1443640 1 --2- 172.21.15.107:0/2788309711 >> v2:172.21.15.110:3300/0 conn(0x7f78ec152040 0x7f78ec0c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f1443640 1 -- 172.21.15.107:0/2788309711 >> v1:172.21.15.110:6789/0 conn(0x7f78ec06a4b0 legacy=0x7f78ec0ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:11.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f1443640 1 -- 172.21.15.107:0/2788309711 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ec155030 con 0x7f78ec0c8100 2023-10-25T00:13:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f1443640 1 --2- 172.21.15.107:0/2788309711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78ec0c8100 0x7f78ec0abd00 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f78ec067ba0 tx=0x7f78dc01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af905ef320548c84 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f0c42640 1 -- 172.21.15.107:0/2788309711 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0ab600 con 0x7f78ec0c8100 2023-10-25T00:13:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f0c42640 1 -- 172.21.15.107:0/2788309711 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78dc0a7050 con 0x7f78ec0c8100 2023-10-25T00:13:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.143+0000 7f78f0c42640 1 -- 172.21.15.107:0/2788309711 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0ab980 con 0x7f78ec0c8100 2023-10-25T00:13:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 -- 172.21.15.107:0/2788309711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78ec0c8100 msgr2=0x7f78ec0abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 --2- 172.21.15.107:0/2788309711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78ec0c8100 0x7f78ec0abd00 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f78ec067ba0 tx=0x7f78dc01c430 comp rx=0 tx=0).stop 2023-10-25T00:13:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 -- 172.21.15.107:0/2788309711 shutdown_connections 2023-10-25T00:13:11.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 --2- 172.21.15.107:0/2788309711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78ec0c8100 0x7f78ec0abd00 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 --2- 172.21.15.107:0/2788309711 >> v2:172.21.15.110:3300/0 conn(0x7f78ec152040 0x7f78ec0c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 -- 172.21.15.107:0/2788309711 >> 172.21.15.107:0/2788309711 conn(0x7f78ec0ca200 msgr2=0x7f78ec0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 -- 172.21.15.107:0/2788309711 shutdown_connections 2023-10-25T00:13:11.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 -- 172.21.15.107:0/2788309711 wait complete. 2023-10-25T00:13:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 Processor -- start 2023-10-25T00:13:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 -- start start 2023-10-25T00:13:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78ec06a4b0 0x7f78ec0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78ec0c8100 0x7f78ec1ecd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78ec152040 0x7f78ec1ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:11.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec1493a0 con 0x7f78ec0c8100 2023-10-25T00:13:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec148690 con 0x7f78ec06a4b0 2023-10-25T00:13:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f3ea8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ec1f0d80 con 0x7f78ec152040 2023-10-25T00:13:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f1443640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78ec0c8100 0x7f78ec1ecd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f1443640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78ec0c8100 0x7f78ec1ecd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52686/0 (socket says 172.21.15.107:52686) 2023-10-25T00:13:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f1443640 1 -- 172.21.15.107:0/1958091518 learned_addr learned my addr 172.21.15.107:0/1958091518 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f1c44640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78ec06a4b0 0x7f78ec0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f2445640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78ec152040 0x7f78ec1ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f1443640 1 -- 172.21.15.107:0/1958091518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78ec152040 msgr2=0x7f78ec1ed2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f1443640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78ec152040 0x7f78ec1ed2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f1443640 1 -- 172.21.15.107:0/1958091518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78ec06a4b0 msgr2=0x7f78ec0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:11.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f1443640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78ec06a4b0 0x7f78ec0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:11.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f1443640 1 -- 172.21.15.107:0/1958091518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78dc0094d0 con 0x7f78ec0c8100 2023-10-25T00:13:11.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f2445640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78ec152040 0x7f78ec1ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:13:11.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.147+0000 7f78f1443640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78ec0c8100 0x7f78ec1ecd60 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f78dc01a060 tx=0x7f78dc0191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:11.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.151+0000 7f78e27fc640 1 -- 172.21.15.107:0/1958091518 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0b4660 con 0x7f78ec0c8100 2023-10-25T00:13:11.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.151+0000 7f78e27fc640 1 -- 172.21.15.107:0/1958091518 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78dc0b47c0 con 0x7f78ec0c8100 2023-10-25T00:13:11.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.151+0000 7f78f3ea8640 1 -- 172.21.15.107:0/1958091518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ec148930 con 0x7f78ec0c8100 2023-10-25T00:13:11.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.151+0000 7f78e27fc640 1 -- 172.21.15.107:0/1958091518 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0b4b30 con 0x7f78ec0c8100 2023-10-25T00:13:11.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.151+0000 7f78f3ea8640 1 -- 172.21.15.107:0/1958091518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ec1f1060 con 0x7f78ec0c8100 2023-10-25T00:13:11.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.151+0000 7f78f3ea8640 1 -- 172.21.15.107:0/1958091518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ec0746e0 con 0x7f78ec0c8100 2023-10-25T00:13:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.151+0000 7f78e27fc640 1 -- 172.21.15.107:0/1958091518 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f78dc0b2070 con 0x7f78ec0c8100 2023-10-25T00:13:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.151+0000 7f78e27fc640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f78d0065e60 0x7f78d0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.151+0000 7f78e27fc640 1 -- 172.21.15.107:0/1958091518 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7f78dc11e3e0 con 0x7f78ec0c8100 2023-10-25T00:13:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.151+0000 7f78f1c44640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f78d0065e60 0x7f78d0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:11.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.151+0000 7f78f1c44640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f78d0065e60 0x7f78d0068320 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0bd830 tx=0x7f78e8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:11.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.155+0000 7f78e27fc640 1 -- 172.21.15.107:0/1958091518 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f78dc0eac20 con 0x7f78ec0c8100 2023-10-25T00:13:11.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.347+0000 7f78f3ea8640 1 -- 172.21.15.107:0/1958091518 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78ec0ac710 con 0x7f78ec0c8100 2023-10-25T00:13:11.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78e27fc640 1 -- 172.21.15.107:0/1958091518 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f78dc0ee240 con 0x7f78ec0c8100 2023-10-25T00:13:11.349 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:11.349 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:11.350 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 170 2023-10-25T00:13:11.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 -- 172.21.15.107:0/1958091518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f78d0065e60 msgr2=0x7f78d0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:11.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f78d0065e60 0x7f78d0068320 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0bd830 tx=0x7f78e8008040 comp rx=0 tx=0).stop 2023-10-25T00:13:11.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 -- 172.21.15.107:0/1958091518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78ec0c8100 msgr2=0x7f78ec1ecd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:11.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78ec0c8100 0x7f78ec1ecd60 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f78dc01a060 tx=0x7f78dc0191d0 comp rx=0 tx=0).stop 2023-10-25T00:13:11.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 -- 172.21.15.107:0/1958091518 shutdown_connections 2023-10-25T00:13:11.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f78d0065e60 0x7f78d0068320 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:11.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78ec152040 0x7f78ec1ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:11.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78ec0c8100 0x7f78ec1ecd60 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:11.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 --2- 172.21.15.107:0/1958091518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78ec06a4b0 0x7f78ec0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:11.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 -- 172.21.15.107:0/1958091518 >> 172.21.15.107:0/1958091518 conn(0x7f78ec0ca200 msgr2=0x7f78ec14a710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:11.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 -- 172.21.15.107:0/1958091518 shutdown_connections 2023-10-25T00:13:11.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:11.351+0000 7f78f3ea8640 1 -- 172.21.15.107:0/1958091518 wait complete. 2023-10-25T00:13:11.367 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:13:13.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.495+0000 7fc84f1f1640 1 Processor -- start 2023-10-25T00:13:13.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84f1f1640 1 -- start start 2023-10-25T00:13:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84f1f1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814ff40 0x7fc848153200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84f1f1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc8480ab8e0 0x7fc8480a9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:13.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84f1f1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc848153740 con 0x7fc8480acb10 2023-10-25T00:13:13.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84f1f1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8480abf10 con 0x7fc8480ab8e0 2023-10-25T00:13:13.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84f1f1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84806b930 con 0x7fc84814ff40 2023-10-25T00:13:13.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc847fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc8480acb10 0x7fc8480ab1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41218/0 (socket says 172.21.15.107:41218) 2023-10-25T00:13:13.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc847fff640 1 -- 172.21.15.107:0/124192315 learned_addr learned my addr 172.21.15.107:0/124192315 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:13:13.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84d78e640 1 --2- 172.21.15.107:0/124192315 >> v2:172.21.15.110:3300/0 conn(0x7fc8480ab8e0 0x7fc8480a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:13.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84cf8d640 1 --2- 172.21.15.107:0/124192315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814ff40 0x7fc848153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:13.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038982327 0 0) 0x7fc834001120 con 0x7fc8480acb10 2023-10-25T00:13:13.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc838003530 con 0x7fc8480acb10 2023-10-25T00:13:13.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1731194360 0 0) 0x7fc838003530 con 0x7fc8480acb10 2023-10-25T00:13:13.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814ff40 msgr2=0x7fc848153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:13.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 --2- 172.21.15.107:0/124192315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814ff40 0x7fc848153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:13.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 >> v2:172.21.15.110:3300/0 conn(0x7fc8480ab8e0 msgr2=0x7fc8480a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:13.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 --2- 172.21.15.107:0/124192315 >> v2:172.21.15.110:3300/0 conn(0x7fc8480ab8e0 0x7fc8480a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:13.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8480ac3a0 con 0x7fc8480acb10 2023-10-25T00:13:13.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fc834001810 con 0x7fc8480acb10 2023-10-25T00:13:13.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 >> v1:172.21.15.107:6789/0 conn(0x7fc8480acb10 legacy=0x7fc8480ab1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:13.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 --2- 172.21.15.107:0/124192315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 0x7fc838003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:13.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 --2- 172.21.15.107:0/124192315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814ff40 0x7fc8380047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:13.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 --2- 172.21.15.107:0/124192315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc838004d10 0x7fc838005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:13.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8480abf10 con 0x7fc838003980 2023-10-25T00:13:13.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84806b930 con 0x7fc838004d10 2023-10-25T00:13:13.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83800ba60 con 0x7fc84814ff40 2023-10-25T00:13:13.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84cf8d640 1 --2- 172.21.15.107:0/124192315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 0x7fc838003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:13.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84cf8d640 1 -- 172.21.15.107:0/124192315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814ff40 msgr2=0x7fc8380047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:13.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84cf8d640 1 --2- 172.21.15.107:0/124192315 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814ff40 0x7fc8380047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:13.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84cf8d640 1 -- 172.21.15.107:0/124192315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc838004d10 msgr2=0x7fc838005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:13.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84d78e640 1 --2- 172.21.15.107:0/124192315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc838004d10 0x7fc838005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:13.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84cf8d640 1 --2- 172.21.15.107:0/124192315 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc838004d10 0x7fc838005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:13.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.499+0000 7fc84cf8d640 1 -- 172.21.15.107:0/124192315 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc838003530 con 0x7fc838003980 2023-10-25T00:13:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84cf8d640 1 --2- 172.21.15.107:0/124192315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 0x7fc838003da0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009ae0 tx=0x7fc83c01ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d4a440c985cb65 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0a5540 con 0x7fc838003980 2023-10-25T00:13:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83c0a56a0 con 0x7fc838003980 2023-10-25T00:13:13.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc8477fe640 1 -- 172.21.15.107:0/124192315 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0a5a50 con 0x7fc838003980 2023-10-25T00:13:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 -- 172.21.15.107:0/124192315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 msgr2=0x7fc838003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 --2- 172.21.15.107:0/124192315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 0x7fc838003da0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fc83c009ae0 tx=0x7fc83c01ca60 comp rx=0 tx=0).stop 2023-10-25T00:13:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84d78e640 1 -- 172.21.15.107:0/124192315 reap_dead start 2023-10-25T00:13:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 -- 172.21.15.107:0/124192315 shutdown_connections 2023-10-25T00:13:13.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 -- 172.21.15.107:0/124192315 >> 172.21.15.107:0/124192315 conn(0x7fc8480c4e40 msgr2=0x7fc8480c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 -- 172.21.15.107:0/124192315 shutdown_connections 2023-10-25T00:13:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 -- 172.21.15.107:0/124192315 wait complete. 2023-10-25T00:13:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 Processor -- start 2023-10-25T00:13:13.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 -- start start 2023-10-25T00:13:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 0x7fc848144300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814b3a0 0x7fc8480c0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8481505f0 0x7fc8480718c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:13.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84806fec0 con 0x7fc838003980 2023-10-25T00:13:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc848144840 con 0x7fc8481505f0 2023-10-25T00:13:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84f1f1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8481ed950 con 0x7fc84814b3a0 2023-10-25T00:13:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84cf8d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 0x7fc848144300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84cf8d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 0x7fc848144300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58492/0 (socket says 172.21.15.107:58492) 2023-10-25T00:13:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84cf8d640 1 -- 172.21.15.107:0/932258600 learned_addr learned my addr 172.21.15.107:0/932258600 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:13.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84d78e640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8481505f0 0x7fc8480718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc847fff640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814b3a0 0x7fc8480c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84cf8d640 1 -- 172.21.15.107:0/932258600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814b3a0 msgr2=0x7fc8480c0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84cf8d640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814b3a0 0x7fc8480c0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84cf8d640 1 -- 172.21.15.107:0/932258600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8481505f0 msgr2=0x7fc8480718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:13.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84cf8d640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8481505f0 0x7fc8480718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:13.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84cf8d640 1 -- 172.21.15.107:0/932258600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc848146100 con 0x7fc838003980 2023-10-25T00:13:13.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.503+0000 7fc84cf8d640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 0x7fc848144300 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fc83c01c890 tx=0x7fc838005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:13.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.507+0000 7fc8457fa640 1 -- 172.21.15.107:0/932258600 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0a5e10 con 0x7fc838003980 2023-10-25T00:13:13.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.507+0000 7fc8457fa640 1 -- 172.21.15.107:0/932258600 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83c004020 con 0x7fc838003980 2023-10-25T00:13:13.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.507+0000 7fc8457fa640 1 -- 172.21.15.107:0/932258600 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83c0043d0 con 0x7fc838003980 2023-10-25T00:13:13.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.507+0000 7fc84f1f1640 1 -- 172.21.15.107:0/932258600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8480700d0 con 0x7fc838003980 2023-10-25T00:13:13.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.507+0000 7fc84f1f1640 1 -- 172.21.15.107:0/932258600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc848071e00 con 0x7fc838003980 2023-10-25T00:13:13.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.507+0000 7fc84f1f1640 1 -- 172.21.15.107:0/932258600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8481ed7b0 con 0x7fc838003980 2023-10-25T00:13:13.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.507+0000 7fc8457fa640 1 -- 172.21.15.107:0/932258600 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc83c0b4040 con 0x7fc838003980 2023-10-25T00:13:13.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.507+0000 7fc8457fa640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc82c065e60 0x7fc82c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:13.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.507+0000 7fc8457fa640 1 -- 172.21.15.107:0/932258600 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7fc83c11e700 con 0x7fc838003980 2023-10-25T00:13:13.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.507+0000 7fc847fff640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc82c065e60 0x7fc82c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:13.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.507+0000 7fc847fff640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc82c065e60 0x7fc82c068320 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc8480720a0 tx=0x7fc834001bb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:13.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.511+0000 7fc8457fa640 1 -- 172.21.15.107:0/932258600 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc8481ed7b0 con 0x7fc838003980 2023-10-25T00:13:13.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.699+0000 7fc84f1f1640 1 -- 172.21.15.107:0/932258600 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8481ed7b0 con 0x7fc838003980 2023-10-25T00:13:13.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.699+0000 7fc8457fa640 1 -- 172.21.15.107:0/932258600 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5774 (secure 0 0 0) 0x7fc8481ed7b0 con 0x7fc838003980 2023-10-25T00:13:13.699 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:13.700 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:13.700 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 170 2023-10-25T00:13:13.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 -- 172.21.15.107:0/932258600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc82c065e60 msgr2=0x7fc82c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:13.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc82c065e60 0x7fc82c068320 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc8480720a0 tx=0x7fc834001bb0 comp rx=0 tx=0).stop 2023-10-25T00:13:13.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 -- 172.21.15.107:0/932258600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 msgr2=0x7fc848144300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:13.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 0x7fc848144300 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fc83c01c890 tx=0x7fc838005800 comp rx=0 tx=0).stop 2023-10-25T00:13:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 -- 172.21.15.107:0/932258600 shutdown_connections 2023-10-25T00:13:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8481505f0 0x7fc8480718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc82c065e60 0x7fc82c068320 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc84814b3a0 0x7fc8480c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 --2- 172.21.15.107:0/932258600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc838003980 0x7fc848144300 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:13.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 -- 172.21.15.107:0/932258600 >> 172.21.15.107:0/932258600 conn(0x7fc8480c4e40 msgr2=0x7fc8480ab1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:13.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 -- 172.21.15.107:0/932258600 shutdown_connections 2023-10-25T00:13:13.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:13.703+0000 7fc84f1f1640 1 -- 172.21.15.107:0/932258600 wait complete. 2023-10-25T00:13:13.718 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:13:15.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd85145b640 1 Processor -- start 2023-10-25T00:13:15.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd85145b640 1 -- start start 2023-10-25T00:13:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd85145b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd84c14fd40 0x7fd84c0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd85145b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd84c06ee40 0x7fd84c0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd85145b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84c06f730 con 0x7fd84c0a9820 2023-10-25T00:13:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd85145b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84c0aa750 con 0x7fd84c14fd40 2023-10-25T00:13:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd85145b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c156a40 con 0x7fd84c06ee40 2023-10-25T00:13:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd84affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd84c14fd40 0x7fd84c0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd843fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd84c0a9820 0x7fd84c0c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41220/0 (socket says 172.21.15.107:41220) 2023-10-25T00:13:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd843fff640 1 -- 172.21.15.107:0/871270530 learned_addr learned my addr 172.21.15.107:0/871270530 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:13:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd84b7fe640 1 --2- 172.21.15.107:0/871270530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd84c06ee40 0x7fd84c0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd84a7fc640 1 -- 172.21.15.107:0/871270530 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84180985 0 0) 0x7fd84c06f730 con 0x7fd84c0a9820 2023-10-25T00:13:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd84a7fc640 1 -- 172.21.15.107:0/871270530 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd838003530 con 0x7fd84c0a9820 2023-10-25T00:13:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd84affd640 1 -- 172.21.15.107:0/871270530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd84c06ee40 msgr2=0x7fd84c0aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd84affd640 1 --2- 172.21.15.107:0/871270530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd84c06ee40 0x7fd84c0aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:15.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd84affd640 1 -- 172.21.15.107:0/871270530 >> v1:172.21.15.107:6789/0 conn(0x7fd84c0a9820 legacy=0x7fd84c0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd84affd640 1 -- 172.21.15.107:0/871270530 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c156710 con 0x7fd84c14fd40 2023-10-25T00:13:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.851+0000 7fd84affd640 1 --2- 172.21.15.107:0/871270530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c14fd40 0x7fd84c0ab090 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0097c0 tx=0x7fd83c01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332f394475f5531a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:15.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd84a7fc640 1 -- 172.21.15.107:0/871270530 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0ab600 con 0x7fd84c14fd40 2023-10-25T00:13:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd84a7fc640 1 -- 172.21.15.107:0/871270530 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c0a7050 con 0x7fd84c14fd40 2023-10-25T00:13:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd84a7fc640 1 -- 172.21.15.107:0/871270530 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0ab980 con 0x7fd84c14fd40 2023-10-25T00:13:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 -- 172.21.15.107:0/871270530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c14fd40 msgr2=0x7fd84c0ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 --2- 172.21.15.107:0/871270530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c14fd40 0x7fd84c0ab090 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0097c0 tx=0x7fd83c01c330 comp rx=0 tx=0).stop 2023-10-25T00:13:15.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 -- 172.21.15.107:0/871270530 shutdown_connections 2023-10-25T00:13:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 --2- 172.21.15.107:0/871270530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd84c06ee40 0x7fd84c0aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 --2- 172.21.15.107:0/871270530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c14fd40 0x7fd84c0ab090 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 -- 172.21.15.107:0/871270530 >> 172.21.15.107:0/871270530 conn(0x7fd84c0c2950 msgr2=0x7fd84c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:15.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 -- 172.21.15.107:0/871270530 shutdown_connections 2023-10-25T00:13:15.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 -- 172.21.15.107:0/871270530 wait complete. 2023-10-25T00:13:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 Processor -- start 2023-10-25T00:13:15.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 -- start start 2023-10-25T00:13:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd84c06ee40 0x7fd84c15b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c14fd40 0x7fd84c0bbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd84c0bb830 0x7fd84c14ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c15be00 con 0x7fd84c14fd40 2023-10-25T00:13:15.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c0bfce0 con 0x7fd84c06ee40 2023-10-25T00:13:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c0bffc0 con 0x7fd84c0bb830 2023-10-25T00:13:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd843fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c14fd40 0x7fd84c0bbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd843fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c14fd40 0x7fd84c0bbf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58510/0 (socket says 172.21.15.107:58510) 2023-10-25T00:13:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd843fff640 1 -- 172.21.15.107:0/3686838970 learned_addr learned my addr 172.21.15.107:0/3686838970 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:15.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd84affd640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd84c06ee40 0x7fd84c15b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd84b7fe640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd84c0bb830 0x7fd84c14ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd843fff640 1 -- 172.21.15.107:0/3686838970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd84c0bb830 msgr2=0x7fd84c14ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd843fff640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd84c0bb830 0x7fd84c14ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd843fff640 1 -- 172.21.15.107:0/3686838970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd84c06ee40 msgr2=0x7fd84c15b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:15.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd843fff640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd84c06ee40 0x7fd84c15b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd843fff640 1 -- 172.21.15.107:0/3686838970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c009420 con 0x7fd84c14fd40 2023-10-25T00:13:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd843fff640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c14fd40 0x7fd84c0bbf50 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fd83400c4d0 tx=0x7fd83400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd8437fe640 1 -- 172.21.15.107:0/3686838970 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd834005cf0 con 0x7fd84c14fd40 2023-10-25T00:13:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd8437fe640 1 -- 172.21.15.107:0/3686838970 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd834005e50 con 0x7fd84c14fd40 2023-10-25T00:13:15.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd8437fe640 1 -- 172.21.15.107:0/3686838970 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd834007270 con 0x7fd84c14fd40 2023-10-25T00:13:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.855+0000 7fd85145b640 1 -- 172.21.15.107:0/3686838970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84c0c02a0 con 0x7fd84c14fd40 2023-10-25T00:13:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.859+0000 7fd85145b640 1 -- 172.21.15.107:0/3686838970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84c0bb1f0 con 0x7fd84c14fd40 2023-10-25T00:13:15.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.859+0000 7fd85145b640 1 -- 172.21.15.107:0/3686838970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84c0c24a0 con 0x7fd84c14fd40 2023-10-25T00:13:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.859+0000 7fd8437fe640 1 -- 172.21.15.107:0/3686838970 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd8340073d0 con 0x7fd84c14fd40 2023-10-25T00:13:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.859+0000 7fd8437fe640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd82c065e90 0x7fd82c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:15.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.859+0000 7fd8437fe640 1 -- 172.21.15.107:0/3686838970 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7fd834083b60 con 0x7fd84c14fd40 2023-10-25T00:13:15.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.859+0000 7fd84affd640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd82c065e90 0x7fd82c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:15.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.859+0000 7fd84affd640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd82c065e90 0x7fd82c068350 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0ad000 tx=0x7fd83c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:15.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:15.863+0000 7fd8437fe640 1 -- 172.21.15.107:0/3686838970 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd834050450 con 0x7fd84c14fd40 2023-10-25T00:13:16.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 -- 172.21.15.107:0/3686838970 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd84c070840 con 0x7fd84c14fd40 2023-10-25T00:13:16.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd8437fe640 1 -- 172.21.15.107:0/3686838970 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5774 (secure 0 0 0) 0x7fd834017020 con 0x7fd84c14fd40 2023-10-25T00:13:16.054 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:16.054 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:16.055 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 170 2023-10-25T00:13:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 -- 172.21.15.107:0/3686838970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd82c065e90 msgr2=0x7fd82c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd82c065e90 0x7fd82c068350 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0ad000 tx=0x7fd83c008040 comp rx=0 tx=0).stop 2023-10-25T00:13:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 -- 172.21.15.107:0/3686838970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c14fd40 msgr2=0x7fd84c0bbf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c14fd40 0x7fd84c0bbf50 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fd83400c4d0 tx=0x7fd83400c990 comp rx=0 tx=0).stop 2023-10-25T00:13:16.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 -- 172.21.15.107:0/3686838970 shutdown_connections 2023-10-25T00:13:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd82c065e90 0x7fd82c068350 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd84c0bb830 0x7fd84c14ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c14fd40 0x7fd84c0bbf50 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 --2- 172.21.15.107:0/3686838970 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd84c06ee40 0x7fd84c15b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 -- 172.21.15.107:0/3686838970 >> 172.21.15.107:0/3686838970 conn(0x7fd84c0c2950 msgr2=0x7fd84c15c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:16.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 -- 172.21.15.107:0/3686838970 shutdown_connections 2023-10-25T00:13:16.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:16.055+0000 7fd85145b640 1 -- 172.21.15.107:0/3686838970 wait complete. 2023-10-25T00:13:16.072 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:13:18.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:18.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.207+0000 7f90a5693640 1 Processor -- start 2023-10-25T00:13:18.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.207+0000 7f90a5693640 1 -- start start 2023-10-25T00:13:18.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f90a5693640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f90a006a230 0x7f90a01551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:18.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f90a5693640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a00c1860 con 0x7f90a006af70 2023-10-25T00:13:18.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f90a5693640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a0154d50 con 0x7f90a014b600 2023-10-25T00:13:18.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f90a5693640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a014b9e0 con 0x7f90a006a230 2023-10-25T00:13:18.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f90967fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f90a006af70 0x7f90a006c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41224/0 (socket says 172.21.15.107:41224) 2023-10-25T00:13:18.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f90967fc640 1 -- 172.21.15.107:0/2205629434 learned_addr learned my addr 172.21.15.107:0/2205629434 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:13:18.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f909f7fe640 1 --2- 172.21.15.107:0/2205629434 >> v2:172.21.15.107:3300/0 conn(0x7f90a006a230 0x7f90a01551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:18.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f909e7fc640 1 -- 172.21.15.107:0/2205629434 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709543594 0 0) 0x7f90a00c1860 con 0x7f90a006af70 2023-10-25T00:13:18.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f909e7fc640 1 -- 172.21.15.107:0/2205629434 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f908c003530 con 0x7f90a006af70 2023-10-25T00:13:18.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f909f7fe640 1 -- 172.21.15.107:0/2205629434 >> v1:172.21.15.110:6789/0 conn(0x7f90a014b600 legacy=0x7f90a0151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:18.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f909f7fe640 1 -- 172.21.15.107:0/2205629434 >> v1:172.21.15.107:6789/0 conn(0x7f90a006af70 legacy=0x7f90a006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:18.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f909f7fe640 1 -- 172.21.15.107:0/2205629434 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a00c1ab0 con 0x7f90a006a230 2023-10-25T00:13:18.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f909f7fe640 1 --2- 172.21.15.107:0/2205629434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90a006a230 0x7f90a01551e0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f9098009d90 tx=0x7f909801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69569b00c45bc3f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:18.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f909e7fc640 1 -- 172.21.15.107:0/2205629434 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980a5860 con 0x7f90a006a230 2023-10-25T00:13:18.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f909e7fc640 1 -- 172.21.15.107:0/2205629434 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90980a59c0 con 0x7f90a006a230 2023-10-25T00:13:18.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.211+0000 7f909e7fc640 1 -- 172.21.15.107:0/2205629434 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980a5d50 con 0x7f90a006a230 2023-10-25T00:13:18.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 -- 172.21.15.107:0/2205629434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90a006a230 msgr2=0x7f90a01551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:18.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 --2- 172.21.15.107:0/2205629434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90a006a230 0x7f90a01551e0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f9098009d90 tx=0x7f909801c890 comp rx=0 tx=0).stop 2023-10-25T00:13:18.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 -- 172.21.15.107:0/2205629434 shutdown_connections 2023-10-25T00:13:18.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 --2- 172.21.15.107:0/2205629434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90a006a230 0x7f90a01551e0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:18.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 -- 172.21.15.107:0/2205629434 >> 172.21.15.107:0/2205629434 conn(0x7f90a00c2950 msgr2=0x7f90a00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:18.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 -- 172.21.15.107:0/2205629434 shutdown_connections 2023-10-25T00:13:18.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 -- 172.21.15.107:0/2205629434 wait complete. 2023-10-25T00:13:18.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 Processor -- start 2023-10-25T00:13:18.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 -- start start 2023-10-25T00:13:18.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90a006a230 0x7f90a0149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:18.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90a006af70 0x7f90a01f0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90a014b600 0x7f90a01f0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0152730 con 0x7f90a014b600 2023-10-25T00:13:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a01f0bb0 con 0x7f90a006a230 2023-10-25T00:13:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0153180 con 0x7f90a006af70 2023-10-25T00:13:18.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f909f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90a014b600 0x7f90a01f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:18.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f909f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90a014b600 0x7f90a01f0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58528/0 (socket says 172.21.15.107:58528) 2023-10-25T00:13:18.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f909f7fe640 1 -- 172.21.15.107:0/1833098073 learned_addr learned my addr 172.21.15.107:0/1833098073 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:18.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f909effd640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90a006a230 0x7f90a0149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:18.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90967fc640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90a006af70 0x7f90a01f0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:18.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f909f7fe640 1 -- 172.21.15.107:0/1833098073 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90a006af70 msgr2=0x7f90a01f0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:18.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f909f7fe640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90a006af70 0x7f90a01f0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:18.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f909f7fe640 1 -- 172.21.15.107:0/1833098073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90a006a230 msgr2=0x7f90a0149630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:18.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f909f7fe640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90a006a230 0x7f90a0149630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:18.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f909f7fe640 1 -- 172.21.15.107:0/1833098073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9098009a70 con 0x7f90a014b600 2023-10-25T00:13:18.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f909effd640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90a006a230 0x7f90a0149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:13:18.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f909f7fe640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90a014b600 0x7f90a01f0670 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f909801a4c0 tx=0x7f9098019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:18.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f9097fff640 1 -- 172.21.15.107:0/1833098073 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980b9860 con 0x7f90a014b600 2023-10-25T00:13:18.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f9097fff640 1 -- 172.21.15.107:0/1833098073 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90980b99c0 con 0x7f90a014b600 2023-10-25T00:13:18.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 -- 172.21.15.107:0/1833098073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a0146460 con 0x7f90a014b600 2023-10-25T00:13:18.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f9097fff640 1 -- 172.21.15.107:0/1833098073 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980b9d30 con 0x7f90a014b600 2023-10-25T00:13:18.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.215+0000 7f90a5693640 1 -- 172.21.15.107:0/1833098073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a0146a20 con 0x7f90a014b600 2023-10-25T00:13:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.219+0000 7f90a5693640 1 -- 172.21.15.107:0/1833098073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a0146c10 con 0x7f90a014b600 2023-10-25T00:13:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.219+0000 7f9097fff640 1 -- 172.21.15.107:0/1833098073 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f90980ad440 con 0x7f90a014b600 2023-10-25T00:13:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.219+0000 7f9097fff640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9080065e60 0x7f9080068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.219+0000 7f909effd640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9080065e60 0x7f9080068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.219+0000 7f9097fff640 1 -- 172.21.15.107:0/1833098073 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 9713+0+0 (secure 0 0 0) 0x7f909811f9b0 con 0x7f90a014b600 2023-10-25T00:13:18.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.219+0000 7f909effd640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9080065e60 0x7f9080068320 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f90a0149e70 tx=0x7f9090009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:18.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.223+0000 7f9097fff640 1 -- 172.21.15.107:0/1833098073 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f90980ec1f0 con 0x7f90a014b600 2023-10-25T00:13:18.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.415+0000 7f90a5693640 1 -- 172.21.15.107:0/1833098073 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90a00746e0 con 0x7f90a014b600 2023-10-25T00:13:18.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.415+0000 7f9097fff640 1 -- 172.21.15.107:0/1833098073 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f90980bd090 con 0x7f90a014b600 2023-10-25T00:13:18.415 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:18.416 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":170}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:11:21.884636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6805},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6805":{"gid":6805,"name":"d","rank":0,"incarnation":161,"state":"up:active","state_seq":5,"addr":"172.21.15.110:6835/898258574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":898258574},{"type":"v1","addr":"172.21.15.110:6835","nonce":898258574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:18.416 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 170 2023-10-25T00:13:18.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 -- 172.21.15.107:0/1833098073 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9080065e60 msgr2=0x7f9080068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:18.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9080065e60 0x7f9080068320 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f90a0149e70 tx=0x7f9090009040 comp rx=0 tx=0).stop 2023-10-25T00:13:18.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 -- 172.21.15.107:0/1833098073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90a014b600 msgr2=0x7f90a01f0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:18.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90a014b600 0x7f90a01f0670 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f909801a4c0 tx=0x7f9098019630 comp rx=0 tx=0).stop 2023-10-25T00:13:18.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 -- 172.21.15.107:0/1833098073 shutdown_connections 2023-10-25T00:13:18.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f90a014b600 0x7f90a01f0670 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:18.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9080065e60 0x7f9080068320 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:18.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90a006af70 0x7f90a01f0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:18.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 --2- 172.21.15.107:0/1833098073 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f90a006a230 0x7f90a0149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:18.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 -- 172.21.15.107:0/1833098073 >> 172.21.15.107:0/1833098073 conn(0x7f90a00c2950 msgr2=0x7f90a014a420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:18.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 -- 172.21.15.107:0/1833098073 shutdown_connections 2023-10-25T00:13:18.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:18.419+0000 7f90a5693640 1 -- 172.21.15.107:0/1833098073 wait complete. 2023-10-25T00:13:18.433 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:13:20.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f1885bed640 1 Processor -- start 2023-10-25T00:13:20.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f1885bed640 1 -- start start 2023-10-25T00:13:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f1885bed640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f188014fd40 0x7f18800ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f1885bed640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f188006ee40 0x7f18800aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f1885bed640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f188006f730 con 0x7f18800a9820 2023-10-25T00:13:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f1885bed640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18800aa750 con 0x7f188006ee40 2023-10-25T00:13:20.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f1885bed640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1880156a40 con 0x7f188014fd40 2023-10-25T00:13:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f187ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f188006ee40 0x7f18800aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f187ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f188006ee40 0x7f18800aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58538/0 (socket says 172.21.15.107:58538) 2023-10-25T00:13:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f187ffff640 1 -- 172.21.15.107:0/3496297683 learned_addr learned my addr 172.21.15.107:0/3496297683 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f187f7fe640 1 --2- 172.21.15.107:0/3496297683 >> v2:172.21.15.110:3300/0 conn(0x7f188014fd40 0x7f18800ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f187ffff640 1 -- 172.21.15.107:0/3496297683 >> v2:172.21.15.110:3300/0 conn(0x7f188014fd40 msgr2=0x7f18800ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f187ffff640 1 --2- 172.21.15.107:0/3496297683 >> v2:172.21.15.110:3300/0 conn(0x7f188014fd40 0x7f18800ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f187ffff640 1 -- 172.21.15.107:0/3496297683 >> v1:172.21.15.110:6789/0 conn(0x7f18800a9820 legacy=0x7f18800c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.567+0000 7f187ffff640 1 -- 172.21.15.107:0/3496297683 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1880156710 con 0x7f188006ee40 2023-10-25T00:13:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f187ffff640 1 --2- 172.21.15.107:0/3496297683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f188006ee40 0x7f18800aa210 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f18800c28b0 tx=0x7f187801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8051d706fa64b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f187e7fc640 1 -- 172.21.15.107:0/3496297683 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18780a56e0 con 0x7f188006ee40 2023-10-25T00:13:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f187e7fc640 1 -- 172.21.15.107:0/3496297683 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18780a5840 con 0x7f188006ee40 2023-10-25T00:13:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f187e7fc640 1 -- 172.21.15.107:0/3496297683 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18780a5bd0 con 0x7f188006ee40 2023-10-25T00:13:20.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 -- 172.21.15.107:0/3496297683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f188006ee40 msgr2=0x7f18800aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:20.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 --2- 172.21.15.107:0/3496297683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f188006ee40 0x7f18800aa210 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f18800c28b0 tx=0x7f187801c880 comp rx=0 tx=0).stop 2023-10-25T00:13:20.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 -- 172.21.15.107:0/3496297683 shutdown_connections 2023-10-25T00:13:20.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 --2- 172.21.15.107:0/3496297683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f188006ee40 0x7f18800aa210 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 --2- 172.21.15.107:0/3496297683 >> v2:172.21.15.110:3300/0 conn(0x7f188014fd40 0x7f18800ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 -- 172.21.15.107:0/3496297683 >> 172.21.15.107:0/3496297683 conn(0x7f18800c2950 msgr2=0x7f18800c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:20.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 -- 172.21.15.107:0/3496297683 shutdown_connections 2023-10-25T00:13:20.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 -- 172.21.15.107:0/3496297683 wait complete. 2023-10-25T00:13:20.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 Processor -- start 2023-10-25T00:13:20.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 -- start start 2023-10-25T00:13:20.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f188006ee40 0x7f188015b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f188014fd40 0x7f18800bfa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.571+0000 7f1885bed640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18800bff80 0x7f18800bae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1885bed640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f188014d040 con 0x7f188014fd40 2023-10-25T00:13:20.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1885bed640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18801ecc60 con 0x7f18800bff80 2023-10-25T00:13:20.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1885bed640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18801edc30 con 0x7f188006ee40 2023-10-25T00:13:20.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f188014fd40 0x7f18800bfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:20.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f188014fd40 0x7f18800bfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58542/0 (socket says 172.21.15.107:58542) 2023-10-25T00:13:20.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187effd640 1 -- 172.21.15.107:0/446935531 learned_addr learned my addr 172.21.15.107:0/446935531 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187ffff640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18800bff80 0x7f18800bae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187effd640 1 -- 172.21.15.107:0/446935531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f188006ee40 msgr2=0x7f188015b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187effd640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f188006ee40 0x7f188015b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187effd640 1 -- 172.21.15.107:0/446935531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18800bff80 msgr2=0x7f18800bae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187effd640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18800bff80 0x7f18800bae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187effd640 1 -- 172.21.15.107:0/446935531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1878009b30 con 0x7f188014fd40 2023-10-25T00:13:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187effd640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f188014fd40 0x7f18800bfa40 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f187400c500 tx=0x7f187400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1867fff640 1 -- 172.21.15.107:0/446935531 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1874005220 con 0x7f188014fd40 2023-10-25T00:13:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1867fff640 1 -- 172.21.15.107:0/446935531 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1874005380 con 0x7f188014fd40 2023-10-25T00:13:20.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1885bed640 1 -- 172.21.15.107:0/446935531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18801ece70 con 0x7f188014fd40 2023-10-25T00:13:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1867fff640 1 -- 172.21.15.107:0/446935531 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1874007270 con 0x7f188014fd40 2023-10-25T00:13:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1885bed640 1 -- 172.21.15.107:0/446935531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18801ed430 con 0x7f188014fd40 2023-10-25T00:13:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1885bed640 1 -- 172.21.15.107:0/446935531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1880070840 con 0x7f188014fd40 2023-10-25T00:13:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1867fff640 1 -- 172.21.15.107:0/446935531 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f18740073d0 con 0x7f188014fd40 2023-10-25T00:13:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1867fff640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f186c065e90 0x7f186c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187f7fe640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f186c065e90 0x7f186c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f1867fff640 1 -- 172.21.15.107:0/446935531 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 9799+0+0 (secure 0 0 0) 0x7f1874083d50 con 0x7f188014fd40 2023-10-25T00:13:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.575+0000 7f187f7fe640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f186c065e90 0x7f186c068350 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1870006aa0 tx=0x7f1870006a30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:20.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.579+0000 7f1867fff640 1 -- 172.21.15.107:0/446935531 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1874087030 con 0x7f188014fd40 2023-10-25T00:13:20.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.767+0000 7f1885bed640 1 -- 172.21.15.107:0/446935531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18800c24a0 con 0x7f188014fd40 2023-10-25T00:13:20.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1867fff640 1 -- 172.21.15.107:0/446935531 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+5020 (secure 0 0 0) 0x7f18740504c0 con 0x7f188014fd40 2023-10-25T00:13:20.769 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:20.769 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:19.912366+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:20.770 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 172 2023-10-25T00:13:20.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 -- 172.21.15.107:0/446935531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f186c065e90 msgr2=0x7f186c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:20.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f186c065e90 0x7f186c068350 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1870006aa0 tx=0x7f1870006a30 comp rx=0 tx=0).stop 2023-10-25T00:13:20.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 -- 172.21.15.107:0/446935531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f188014fd40 msgr2=0x7f18800bfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:20.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f188014fd40 0x7f18800bfa40 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f187400c500 tx=0x7f187400c9c0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 -- 172.21.15.107:0/446935531 shutdown_connections 2023-10-25T00:13:20.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18800bff80 0x7f18800bae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f188014fd40 0x7f18800bfa40 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f186c065e90 0x7f186c068350 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 --2- 172.21.15.107:0/446935531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f188006ee40 0x7f188015b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 -- 172.21.15.107:0/446935531 >> 172.21.15.107:0/446935531 conn(0x7f18800c2950 msgr2=0x7f188014d520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:20.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 -- 172.21.15.107:0/446935531 shutdown_connections 2023-10-25T00:13:20.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.771+0000 7f1885bed640 1 -- 172.21.15.107:0/446935531 wait complete. 2023-10-25T00:13:20.789 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2023-10-25T00:13:20.790 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:13:20.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:20.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.907+0000 7f69edd7e640 1 Processor -- start 2023-10-25T00:13:20.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.907+0000 7f69edd7e640 1 -- start start 2023-10-25T00:13:20.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.907+0000 7f69edd7e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f69e813bb00 0x7f69e81531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.907+0000 7f69edd7e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69e80aa330 0x7f69e806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.907+0000 7f69edd7e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e8152d40 con 0x7f69e80a9450 2023-10-25T00:13:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.907+0000 7f69edd7e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e8156100 con 0x7f69e813bb00 2023-10-25T00:13:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.907+0000 7f69edd7e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e81527c0 con 0x7f69e80aa330 2023-10-25T00:13:20.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e77fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f69e813bb00 0x7f69e81531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f69e80a9450 0x7f69e8071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41230/0 (socket says 172.21.15.107:41230) 2023-10-25T00:13:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e7fff640 1 -- 172.21.15.107:0/3397787372 learned_addr learned my addr 172.21.15.107:0/3397787372 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:13:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69dffff640 1 --2- 172.21.15.107:0/3397787372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69e80aa330 0x7f69e806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e77fe640 1 -- 172.21.15.107:0/3397787372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69e80aa330 msgr2=0x7f69e806e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:20.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e77fe640 1 --2- 172.21.15.107:0/3397787372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69e80aa330 0x7f69e806e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e77fe640 1 -- 172.21.15.107:0/3397787372 >> v1:172.21.15.107:6789/0 conn(0x7f69e80a9450 legacy=0x7f69e8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e6ffd640 1 -- 172.21.15.107:0/3397787372 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156177318 0 0) 0x7f69e8152d40 con 0x7f69e80a9450 2023-10-25T00:13:20.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e77fe640 1 -- 172.21.15.107:0/3397787372 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e806d660 con 0x7f69e813bb00 2023-10-25T00:13:20.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e77fe640 1 --2- 172.21.15.107:0/3397787372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69e813bb00 0x7f69e81531d0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f69d00097f0 tx=0x7f69d001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd4853df39ea6d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:20.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e6ffd640 1 -- 172.21.15.107:0/3397787372 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d00ab600 con 0x7f69e813bb00 2023-10-25T00:13:20.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e6ffd640 1 -- 172.21.15.107:0/3397787372 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d00a7050 con 0x7f69e813bb00 2023-10-25T00:13:20.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69e6ffd640 1 -- 172.21.15.107:0/3397787372 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d00ab980 con 0x7f69e813bb00 2023-10-25T00:13:20.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69edd7e640 1 -- 172.21.15.107:0/3397787372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69e813bb00 msgr2=0x7f69e81531d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:20.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.911+0000 7f69edd7e640 1 --2- 172.21.15.107:0/3397787372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69e813bb00 0x7f69e81531d0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f69d00097f0 tx=0x7f69d001c360 comp rx=0 tx=0).stop 2023-10-25T00:13:20.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 -- 172.21.15.107:0/3397787372 shutdown_connections 2023-10-25T00:13:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 --2- 172.21.15.107:0/3397787372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69e80aa330 0x7f69e806e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 --2- 172.21.15.107:0/3397787372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69e813bb00 0x7f69e81531d0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 -- 172.21.15.107:0/3397787372 >> 172.21.15.107:0/3397787372 conn(0x7f69e8070f60 msgr2=0x7f69e8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 -- 172.21.15.107:0/3397787372 shutdown_connections 2023-10-25T00:13:20.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 -- 172.21.15.107:0/3397787372 wait complete. 2023-10-25T00:13:20.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 Processor -- start 2023-10-25T00:13:20.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 -- start start 2023-10-25T00:13:20.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69e80a9450 0x7f69e8150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69e80aa330 0x7f69e815b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69e813bb00 0x7f69e81edd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e8150860 con 0x7f69e80aa330 2023-10-25T00:13:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e81392e0 con 0x7f69e813bb00 2023-10-25T00:13:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e80c6c70 con 0x7f69e80a9450 2023-10-25T00:13:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69dffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69e80aa330 0x7f69e815b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:20.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69dffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69e80aa330 0x7f69e815b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58562/0 (socket says 172.21.15.107:58562) 2023-10-25T00:13:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69dffff640 1 -- 172.21.15.107:0/1617127248 learned_addr learned my addr 172.21.15.107:0/1617127248 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69e77fe640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69e80a9450 0x7f69e8150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69e7fff640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69e813bb00 0x7f69e81edd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:20.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69dffff640 1 -- 172.21.15.107:0/1617127248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69e80a9450 msgr2=0x7f69e8150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69dffff640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69e80a9450 0x7f69e8150320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69dffff640 1 -- 172.21.15.107:0/1617127248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69e813bb00 msgr2=0x7f69e81edd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69dffff640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69e813bb00 0x7f69e81edd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69dffff640 1 -- 172.21.15.107:0/1617127248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d0009420 con 0x7f69e80aa330 2023-10-25T00:13:20.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69e7fff640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69e813bb00 0x7f69e81edd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:13:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69dffff640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69e80aa330 0x7f69e815b440 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f69d800d7d0 tx=0x7f69d800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69e4ff9640 1 -- 172.21.15.107:0/1617127248 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d800b6c0 con 0x7f69e80aa330 2023-10-25T00:13:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69e4ff9640 1 -- 172.21.15.107:0/1617127248 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d800b820 con 0x7f69e80aa330 2023-10-25T00:13:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 -- 172.21.15.107:0/1617127248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e80c6f50 con 0x7f69e80aa330 2023-10-25T00:13:20.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69e4ff9640 1 -- 172.21.15.107:0/1617127248 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d800bbd0 con 0x7f69e80aa330 2023-10-25T00:13:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 -- 172.21.15.107:0/1617127248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e80c7510 con 0x7f69e80aa330 2023-10-25T00:13:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.915+0000 7f69edd7e640 1 -- 172.21.15.107:0/1617127248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e80746e0 con 0x7f69e80aa330 2023-10-25T00:13:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.919+0000 7f69e4ff9640 1 -- 172.21.15.107:0/1617127248 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f69d800bd30 con 0x7f69e80aa330 2023-10-25T00:13:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.919+0000 7f69e4ff9640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69c8065e60 0x7f69c8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:20.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.919+0000 7f69e4ff9640 1 -- 172.21.15.107:0/1617127248 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 9799+0+0 (secure 0 0 0) 0x7f69d8085790 con 0x7f69e80aa330 2023-10-25T00:13:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.919+0000 7f69e77fe640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69c8065e60 0x7f69c8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.919+0000 7f69e77fe640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69c8065e60 0x7f69c8068320 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f69d00ad000 tx=0x7f69d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:20.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:20.923+0000 7f69e4ff9640 1 -- 172.21.15.107:0/1617127248 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f69d8051ee0 con 0x7f69e80aa330 2023-10-25T00:13:21.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.107+0000 7f69edd7e640 1 -- 172.21.15.107:0/1617127248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69e81529a0 con 0x7f69e80aa330 2023-10-25T00:13:21.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.107+0000 7f69e4ff9640 1 -- 172.21.15.107:0/1617127248 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+5017 (secure 0 0 0) 0x7f69d8055500 con 0x7f69e80aa330 2023-10-25T00:13:21.109 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:21.109 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:20.926467+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:21.109 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 173 2023-10-25T00:13:21.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 -- 172.21.15.107:0/1617127248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69c8065e60 msgr2=0x7f69c8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69c8065e60 0x7f69c8068320 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f69d00ad000 tx=0x7f69d0008040 comp rx=0 tx=0).stop 2023-10-25T00:13:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 -- 172.21.15.107:0/1617127248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69e80aa330 msgr2=0x7f69e815b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69e80aa330 0x7f69e815b440 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f69d800d7d0 tx=0x7f69d800dc90 comp rx=0 tx=0).stop 2023-10-25T00:13:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 -- 172.21.15.107:0/1617127248 shutdown_connections 2023-10-25T00:13:21.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69e813bb00 0x7f69e81edd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f69e80aa330 0x7f69e815b440 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f69c8065e60 0x7f69c8068320 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 --2- 172.21.15.107:0/1617127248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f69e80a9450 0x7f69e8150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 -- 172.21.15.107:0/1617127248 >> 172.21.15.107:0/1617127248 conn(0x7f69e8070f60 msgr2=0x7f69e8156240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:21.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 -- 172.21.15.107:0/1617127248 shutdown_connections 2023-10-25T00:13:21.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:21.111+0000 7f69edd7e640 1 -- 172.21.15.107:0/1617127248 wait complete. 2023-10-25T00:13:21.129 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 173, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 173, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8106": { "addr": "172.21.15.107:6837/175581297", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 175581297, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 175581297, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8106, "incarnation": 171, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:13:20.926467+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8106 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 169, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8016": { "addr": "172.21.15.107:6835/2856791739", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2856791739, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 2856791739, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8016, "incarnation": 166, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:12:36.920081+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8016 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:13:22.882 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:13:22.883+0000 7fbe8aaa9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:13:23.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:23.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.258+0000 7f54a1c69640 1 Processor -- start 2023-10-25T00:13:23.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.258+0000 7f54a1c69640 1 -- start start 2023-10-25T00:13:23.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f54a1c69640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f549c155370 0x7f549c0ac450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:23.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f54a1c69640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549c155870 con 0x7f549c0c1080 2023-10-25T00:13:23.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f54a1c69640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549c1522e0 con 0x7f549c151f00 2023-10-25T00:13:23.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f54a1c69640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f549c0c02c0 con 0x7f549c155370 2023-10-25T00:13:23.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f549affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f549c155370 0x7f549c0ac450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:23.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f549affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f549c155370 0x7f549c0ac450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46854/0 (socket says 172.21.15.107:46854) 2023-10-25T00:13:23.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f549affd640 1 -- 172.21.15.107:0/2800639865 learned_addr learned my addr 172.21.15.107:0/2800639865 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:23.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f549bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f549c0c1080 0x7f549c0ac990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36558/0 (socket says 172.21.15.107:36558) 2023-10-25T00:13:23.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f549a7fc640 1 -- 172.21.15.107:0/2800639865 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416875106 0 0) 0x7f549c155870 con 0x7f549c0c1080 2023-10-25T00:13:23.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f549a7fc640 1 -- 172.21.15.107:0/2800639865 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5484003530 con 0x7f549c0c1080 2023-10-25T00:13:23.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f549affd640 1 -- 172.21.15.107:0/2800639865 >> v1:172.21.15.110:6789/0 conn(0x7f549c151f00 legacy=0x7f549c070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:23.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f549affd640 1 -- 172.21.15.107:0/2800639865 >> v1:172.21.15.107:6789/0 conn(0x7f549c0c1080 legacy=0x7f549c0ac990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:23.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f549affd640 1 -- 172.21.15.107:0/2800639865 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549c0c05a0 con 0x7f549c155370 2023-10-25T00:13:23.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.262+0000 7f549affd640 1 --2- 172.21.15.107:0/2800639865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c155370 0x7f549c0ac450 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f5490009870 tx=0x7f549001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c232b4d495345a9d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:23.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549a7fc640 1 -- 172.21.15.107:0/2800639865 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54900a9400 con 0x7f549c155370 2023-10-25T00:13:23.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549a7fc640 1 -- 172.21.15.107:0/2800639865 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54900a9560 con 0x7f549c155370 2023-10-25T00:13:23.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549a7fc640 1 -- 172.21.15.107:0/2800639865 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54900a98f0 con 0x7f549c155370 2023-10-25T00:13:23.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 -- 172.21.15.107:0/2800639865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c155370 msgr2=0x7f549c0ac450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:23.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 --2- 172.21.15.107:0/2800639865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c155370 0x7f549c0ac450 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f5490009870 tx=0x7f549001c540 comp rx=0 tx=0).stop 2023-10-25T00:13:23.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 -- 172.21.15.107:0/2800639865 shutdown_connections 2023-10-25T00:13:23.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 --2- 172.21.15.107:0/2800639865 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c155370 0x7f549c0ac450 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:23.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 -- 172.21.15.107:0/2800639865 >> 172.21.15.107:0/2800639865 conn(0x7f549c0c5cd0 msgr2=0x7f549c0c60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:23.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 -- 172.21.15.107:0/2800639865 shutdown_connections 2023-10-25T00:13:23.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 -- 172.21.15.107:0/2800639865 wait complete. 2023-10-25T00:13:23.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 Processor -- start 2023-10-25T00:13:23.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 -- start start 2023-10-25T00:13:23.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f549c0c1080 0x7f549c14a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:23.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f549c151f00 0x7f549c155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:23.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c155370 0x7f549c155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:23.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c06dba0 con 0x7f549c155370 2023-10-25T00:13:23.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c158a80 con 0x7f549c151f00 2023-10-25T00:13:23.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f54a1c69640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549c1efca0 con 0x7f549c0c1080 2023-10-25T00:13:23.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c155370 0x7f549c155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:23.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c155370 0x7f549c155fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46866/0 (socket says 172.21.15.107:46866) 2023-10-25T00:13:23.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549bfff640 1 -- 172.21.15.107:0/2076841866 learned_addr learned my addr 172.21.15.107:0/2076841866 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:23.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549affd640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f549c151f00 0x7f549c155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:23.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549b7fe640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f549c0c1080 0x7f549c14a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:23.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549bfff640 1 -- 172.21.15.107:0/2076841866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f549c0c1080 msgr2=0x7f549c14a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:23.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549bfff640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f549c0c1080 0x7f549c14a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:23.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549bfff640 1 -- 172.21.15.107:0/2076841866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f549c151f00 msgr2=0x7f549c155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:23.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549bfff640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f549c151f00 0x7f549c155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:23.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549bfff640 1 -- 172.21.15.107:0/2076841866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54900094d0 con 0x7f549c155370 2023-10-25T00:13:23.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.266+0000 7f549bfff640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c155370 0x7f549c155fd0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f549400c7c0 tx=0x7f549400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:23.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.270+0000 7f5483fff640 1 -- 172.21.15.107:0/2076841866 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549400a640 con 0x7f549c155370 2023-10-25T00:13:23.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.270+0000 7f5483fff640 1 -- 172.21.15.107:0/2076841866 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549400a7a0 con 0x7f549c155370 2023-10-25T00:13:23.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.270+0000 7f54a1c69640 1 -- 172.21.15.107:0/2076841866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549c06e370 con 0x7f549c155370 2023-10-25T00:13:23.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.270+0000 7f5483fff640 1 -- 172.21.15.107:0/2076841866 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549400ab50 con 0x7f549c155370 2023-10-25T00:13:23.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.270+0000 7f54a1c69640 1 -- 172.21.15.107:0/2076841866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549c1f0190 con 0x7f549c155370 2023-10-25T00:13:23.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.270+0000 7f54a1c69640 1 -- 172.21.15.107:0/2076841866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549c0746e0 con 0x7f549c155370 2023-10-25T00:13:23.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.270+0000 7f5483fff640 1 -- 172.21.15.107:0/2076841866 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f549400acb0 con 0x7f549c155370 2023-10-25T00:13:23.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.270+0000 7f5483fff640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5488065e60 0x7f5488068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:23.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.270+0000 7f5483fff640 1 -- 172.21.15.107:0/2076841866 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 9799+0+0 (secure 0 0 0) 0x7f5494083ee0 con 0x7f549c155370 2023-10-25T00:13:23.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.270+0000 7f549b7fe640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5488065e60 0x7f5488068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:23.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.270+0000 7f549b7fe640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5488065e60 0x7f5488068320 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f549c150640 tx=0x7f548c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:23.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.274+0000 7f5483fff640 1 -- 172.21.15.107:0/2076841866 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5494050720 con 0x7f549c155370 2023-10-25T00:13:23.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 -- 172.21.15.107:0/2076841866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f549c0721d0 con 0x7f549c155370 2023-10-25T00:13:23.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f5483fff640 1 -- 172.21.15.107:0/2076841866 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+5017 (secure 0 0 0) 0x7f5494017020 con 0x7f549c155370 2023-10-25T00:13:23.510 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:23.511 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:23.511 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 174 2023-10-25T00:13:23.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 -- 172.21.15.107:0/2076841866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5488065e60 msgr2=0x7f5488068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:23.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5488065e60 0x7f5488068320 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f549c150640 tx=0x7f548c009040 comp rx=0 tx=0).stop 2023-10-25T00:13:23.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 -- 172.21.15.107:0/2076841866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c155370 msgr2=0x7f549c155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:23.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c155370 0x7f549c155fd0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f549400c7c0 tx=0x7f549400cc80 comp rx=0 tx=0).stop 2023-10-25T00:13:23.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 -- 172.21.15.107:0/2076841866 shutdown_connections 2023-10-25T00:13:23.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c155370 0x7f549c155fd0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:23.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f549c151f00 0x7f549c155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:23.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5488065e60 0x7f5488068320 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:23.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 --2- 172.21.15.107:0/2076841866 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f549c0c1080 0x7f549c14a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:23.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 -- 172.21.15.107:0/2076841866 >> 172.21.15.107:0/2076841866 conn(0x7f549c0c5cd0 msgr2=0x7f549c0ac680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:23.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 -- 172.21.15.107:0/2076841866 shutdown_connections 2023-10-25T00:13:23.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:23.510+0000 7f54a1c69640 1 -- 172.21.15.107:0/2076841866 wait complete. 2023-10-25T00:13:23.532 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6805 2023-10-25T00:13:23.532 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.d 2023-10-25T00:13:27.532 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2023-10-25T00:13:27.533 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-25T00:13:27.533 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-25T00:13:27.536 INFO:tasks.ceph.mds.d:Started 2023-10-25T00:13:27.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:27.617 INFO:tasks.ceph.mds.d.smithi110.stdout:starting mds.d at 2023-10-25T00:13:27.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.670+0000 7f5a4a3be640 1 Processor -- start 2023-10-25T00:13:27.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.670+0000 7f5a4a3be640 1 -- start start 2023-10-25T00:13:27.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a4a3be640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a440c8100 0x7f5a440abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:27.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a4a3be640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a44155680 con 0x7f5a44152040 2023-10-25T00:13:27.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a4a3be640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a44154a70 con 0x7f5a4406a4b0 2023-10-25T00:13:27.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a4a3be640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a44154d50 con 0x7f5a440c8100 2023-10-25T00:13:27.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a43fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5a44152040 0x7f5a440c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36560/0 (socket says 172.21.15.107:36560) 2023-10-25T00:13:27.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a43fff640 1 -- 172.21.15.107:0/2905831919 learned_addr learned my addr 172.21.15.107:0/2905831919 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:13:27.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a3bfff640 1 --2- 172.21.15.107:0/2905831919 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a440c8100 0x7f5a440abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:27.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a437fe640 1 -- 172.21.15.107:0/2905831919 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323483652 0 0) 0x7f5a44155680 con 0x7f5a44152040 2023-10-25T00:13:27.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a437fe640 1 -- 172.21.15.107:0/2905831919 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a30003530 con 0x7f5a44152040 2023-10-25T00:13:27.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a437fe640 1 -- 172.21.15.107:0/2905831919 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351216703 0 0) 0x7f5a44154a70 con 0x7f5a4406a4b0 2023-10-25T00:13:27.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a437fe640 1 -- 172.21.15.107:0/2905831919 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a44155680 con 0x7f5a4406a4b0 2023-10-25T00:13:27.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a3bfff640 1 -- 172.21.15.107:0/2905831919 >> v1:172.21.15.110:6789/0 conn(0x7f5a4406a4b0 legacy=0x7f5a440ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:27.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a3bfff640 1 -- 172.21.15.107:0/2905831919 >> v1:172.21.15.107:6789/0 conn(0x7f5a44152040 legacy=0x7f5a440c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:27.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a3bfff640 1 -- 172.21.15.107:0/2905831919 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a44155030 con 0x7f5a440c8100 2023-10-25T00:13:27.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a3bfff640 1 --2- 172.21.15.107:0/2905831919 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a440c8100 0x7f5a440abd00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0097f0 tx=0x7f5a2c01c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19ef77a2897118ba server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:27.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a437fe640 1 -- 172.21.15.107:0/2905831919 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0ab600 con 0x7f5a440c8100 2023-10-25T00:13:27.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a437fe640 1 -- 172.21.15.107:0/2905831919 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c0a7050 con 0x7f5a440c8100 2023-10-25T00:13:27.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.674+0000 7f5a437fe640 1 -- 172.21.15.107:0/2905831919 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0ab980 con 0x7f5a440c8100 2023-10-25T00:13:27.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 -- 172.21.15.107:0/2905831919 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a440c8100 msgr2=0x7f5a440abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:27.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 --2- 172.21.15.107:0/2905831919 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a440c8100 0x7f5a440abd00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0097f0 tx=0x7f5a2c01c430 comp rx=0 tx=0).stop 2023-10-25T00:13:27.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 -- 172.21.15.107:0/2905831919 shutdown_connections 2023-10-25T00:13:27.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 --2- 172.21.15.107:0/2905831919 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a440c8100 0x7f5a440abd00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:27.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 -- 172.21.15.107:0/2905831919 >> 172.21.15.107:0/2905831919 conn(0x7f5a440ca200 msgr2=0x7f5a440ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:27.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 -- 172.21.15.107:0/2905831919 shutdown_connections 2023-10-25T00:13:27.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 -- 172.21.15.107:0/2905831919 wait complete. 2023-10-25T00:13:27.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 Processor -- start 2023-10-25T00:13:27.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 -- start start 2023-10-25T00:13:27.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a4406a4b0 0x7f5a44148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:27.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440c8100 0x7f5a440c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:27.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a44152040 0x7f5a44153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:27.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a440be930 con 0x7f5a440c8100 2023-10-25T00:13:27.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a440bdca0 con 0x7f5a44152040 2023-10-25T00:13:27.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a440be4b0 con 0x7f5a4406a4b0 2023-10-25T00:13:27.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a3bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440c8100 0x7f5a440c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:27.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a3bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440c8100 0x7f5a440c3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46880/0 (socket says 172.21.15.107:46880) 2023-10-25T00:13:27.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a3bfff640 1 -- 172.21.15.107:0/685196541 learned_addr learned my addr 172.21.15.107:0/685196541 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:27.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a43fff640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a4406a4b0 0x7f5a44148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:27.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4895b640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a44152040 0x7f5a44153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:27.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a3bfff640 1 -- 172.21.15.107:0/685196541 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a4406a4b0 msgr2=0x7f5a44148890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:27.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a3bfff640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a4406a4b0 0x7f5a44148890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:27.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a3bfff640 1 -- 172.21.15.107:0/685196541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a44152040 msgr2=0x7f5a44153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:27.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a3bfff640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a44152040 0x7f5a44153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:27.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a3bfff640 1 -- 172.21.15.107:0/685196541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c009420 con 0x7f5a440c8100 2023-10-25T00:13:27.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a3bfff640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440c8100 0x7f5a440c3fd0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c019040 tx=0x7f5a2c0191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:27.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a417fa640 1 -- 172.21.15.107:0/685196541 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0b4400 con 0x7f5a440c8100 2023-10-25T00:13:27.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a417fa640 1 -- 172.21.15.107:0/685196541 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2c0b4580 con 0x7f5a440c8100 2023-10-25T00:13:27.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 -- 172.21.15.107:0/685196541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a440c4510 con 0x7f5a440c8100 2023-10-25T00:13:27.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a417fa640 1 -- 172.21.15.107:0/685196541 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0b4930 con 0x7f5a440c8100 2023-10-25T00:13:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.678+0000 7f5a4a3be640 1 -- 172.21.15.107:0/685196541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a440be1d0 con 0x7f5a440c8100 2023-10-25T00:13:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.682+0000 7f5a4a3be640 1 -- 172.21.15.107:0/685196541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a44069440 con 0x7f5a440c8100 2023-10-25T00:13:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.682+0000 7f5a417fa640 1 -- 172.21.15.107:0/685196541 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5a2c0b2040 con 0x7f5a440c8100 2023-10-25T00:13:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.682+0000 7f5a417fa640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a24065e90 0x7f5a24068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.682+0000 7f5a417fa640 1 -- 172.21.15.107:0/685196541 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 9799+0+0 (secure 0 0 0) 0x7f5a2c11dd90 con 0x7f5a440c8100 2023-10-25T00:13:27.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.682+0000 7f5a43fff640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a24065e90 0x7f5a24068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:27.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.682+0000 7f5a43fff640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a24065e90 0x7f5a24068350 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5a34007fd0 tx=0x7f5a34007d80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:27.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.686+0000 7f5a417fa640 1 -- 172.21.15.107:0/685196541 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5a2c0ea5b0 con 0x7f5a440c8100 2023-10-25T00:13:27.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.934+0000 7f5a4a3be640 1 -- 172.21.15.107:0/685196541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a440746e0 con 0x7f5a440c8100 2023-10-25T00:13:27.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a417fa640 1 -- 172.21.15.107:0/685196541 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f5a2c0edbd0 con 0x7f5a440c8100 2023-10-25T00:13:27.938 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:27.938 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:27.939 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 175 2023-10-25T00:13:27.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 -- 172.21.15.107:0/685196541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a24065e90 msgr2=0x7f5a24068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:27.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a24065e90 0x7f5a24068350 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f5a34007fd0 tx=0x7f5a34007d80 comp rx=0 tx=0).stop 2023-10-25T00:13:27.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 -- 172.21.15.107:0/685196541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440c8100 msgr2=0x7f5a440c3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:27.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440c8100 0x7f5a440c3fd0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c019040 tx=0x7f5a2c0191d0 comp rx=0 tx=0).stop 2023-10-25T00:13:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 -- 172.21.15.107:0/685196541 shutdown_connections 2023-10-25T00:13:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a44152040 0x7f5a44153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a440c8100 0x7f5a440c3fd0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a24065e90 0x7f5a24068350 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 --2- 172.21.15.107:0/685196541 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a4406a4b0 0x7f5a44148890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 -- 172.21.15.107:0/685196541 >> 172.21.15.107:0/685196541 conn(0x7f5a440ca200 msgr2=0x7f5a4406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 -- 172.21.15.107:0/685196541 shutdown_connections 2023-10-25T00:13:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:27.938+0000 7f5a4a3be640 1 -- 172.21.15.107:0/685196541 wait complete. 2023-10-25T00:13:27.956 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2023-10-25T00:13:27.957 INFO:tasks.mds_thrash.fs.[a]:waiting for 95 secs before thrashing 2023-10-25T00:13:28.702 INFO:tasks.workunit.client.0.smithi107.stdout:fs setup took 1098 secs 2023-10-25T00:13:29.992 INFO:tasks.workunit.client.0.smithi107.stdout:Syncing()...1 sec 2023-10-25T00:13:29.993 INFO:tasks.workunit.client.0.smithi107.stdout:Starting Actual Benchmark At: Wed Oct 25 00:13:29 2023 2023-10-25T00:13:29.993 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:13:32.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:32.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc44bc640 1 Processor -- start 2023-10-25T00:13:32.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc44bc640 1 -- start start 2023-10-25T00:13:32.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc44bc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4cbc0c8640 0x7f4cbc06a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:32.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc44bc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4cbc1553d0 0x7f4cbc06ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:32.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc44bc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cbc152400 con 0x7f4cbc152020 2023-10-25T00:13:32.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc2258640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4cbc152020 0x7f4cbc0c4bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:42976/0 (socket says 172.21.15.107:42976) 2023-10-25T00:13:32.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc2258640 1 -- 172.21.15.107:0/2662301486 learned_addr learned my addr 172.21.15.107:0/2662301486 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:13:32.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1a57640 1 --2- 172.21.15.107:0/2662301486 >> v2:172.21.15.107:3300/0 conn(0x7f4cbc0c8640 0x7f4cbc06a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:32.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc2a59640 1 --2- 172.21.15.107:0/2662301486 >> v2:172.21.15.110:3300/0 conn(0x7f4cbc1553d0 0x7f4cbc06ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:32.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc44bc640 1 -- 172.21.15.107:0/2662301486 --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cbc1558d0 con 0x7f4cbc0c8640 2023-10-25T00:13:32.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc44bc640 1 -- 172.21.15.107:0/2662301486 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cbc0c6ce0 con 0x7f4cbc1553d0 2023-10-25T00:13:32.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994952803 0 0) 0x7f4cbc152400 con 0x7f4cbc152020 2023-10-25T00:13:32.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ca8003530 con 0x7f4cbc152020 2023-10-25T00:13:32.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3540946276 0 0) 0x7f4ca8003530 con 0x7f4cbc152020 2023-10-25T00:13:32.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 >> v2:172.21.15.110:3300/0 conn(0x7f4cbc1553d0 msgr2=0x7f4cbc06ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:32.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 --2- 172.21.15.107:0/2662301486 >> v2:172.21.15.110:3300/0 conn(0x7f4cbc1553d0 0x7f4cbc06ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:32.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 >> v2:172.21.15.107:3300/0 conn(0x7f4cbc0c8640 msgr2=0x7f4cbc06a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:32.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 --2- 172.21.15.107:0/2662301486 >> v2:172.21.15.107:3300/0 conn(0x7f4cbc0c8640 0x7f4cbc06a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:32.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc0c6fc0 con 0x7f4cbc152020 2023-10-25T00:13:32.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f4cb8001680 con 0x7f4cbc152020 2023-10-25T00:13:32.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1a57640 1 --2- 172.21.15.107:0/2662301486 >> v2:172.21.15.107:3300/0 conn(0x7f4cbc0c8640 0x7f4cbc06a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:13:32.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 >> v1:172.21.15.110:6789/0 conn(0x7f4cbc152020 legacy=0x7f4cbc0c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:32.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 --2- 172.21.15.107:0/2662301486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca8003980 0x7f4ca8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:32.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 --2- 172.21.15.107:0/2662301486 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca80047d0 0x7f4ca8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:32.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 --2- 172.21.15.107:0/2662301486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca8005190 0x7f4ca800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:32.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc1558d0 con 0x7f4ca8005190 2023-10-25T00:13:32.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0c6ce0 con 0x7f4ca8003980 2023-10-25T00:13:32.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca800bfd0 con 0x7f4ca80047d0 2023-10-25T00:13:32.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc2a59640 1 --2- 172.21.15.107:0/2662301486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca8005190 0x7f4ca800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:32.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.954+0000 7f4cc2258640 1 --2- 172.21.15.107:0/2662301486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca8003980 0x7f4ca8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:32.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.958+0000 7f4cc2258640 1 -- 172.21.15.107:0/2662301486 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca80047d0 msgr2=0x7f4ca8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.958+0000 7f4cc2258640 1 --2- 172.21.15.107:0/2662301486 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca80047d0 0x7f4ca8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.958+0000 7f4cc2258640 1 -- 172.21.15.107:0/2662301486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca8005190 msgr2=0x7f4ca800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.958+0000 7f4cc2258640 1 --2- 172.21.15.107:0/2662301486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca8005190 0x7f4ca800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.958+0000 7f4cc2258640 1 -- 172.21.15.107:0/2662301486 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca8003530 con 0x7f4ca8003980 2023-10-25T00:13:32.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.958+0000 7f4cc2a59640 1 --2- 172.21.15.107:0/2662301486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ca8005190 0x7f4ca800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:13:32.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.958+0000 7f4cc2a59640 1 -- 172.21.15.107:0/2662301486 reap_dead start 2023-10-25T00:13:32.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.958+0000 7f4cc2258640 1 --2- 172.21.15.107:0/2662301486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca8003980 0x7f4ca8003da0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800c290 tx=0x7f4cb801f2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6415f9182d0d2e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:32.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.958+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb80a8c90 con 0x7f4ca8003980 2023-10-25T00:13:32.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.958+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb80a8df0 con 0x7f4ca8003980 2023-10-25T00:13:32.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.958+0000 7f4cc1256640 1 -- 172.21.15.107:0/2662301486 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8005b20 con 0x7f4ca8003980 2023-10-25T00:13:32.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 -- 172.21.15.107:0/2662301486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca8003980 msgr2=0x7f4ca8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:32.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 --2- 172.21.15.107:0/2662301486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca8003980 0x7f4ca8003da0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800c290 tx=0x7f4cb801f2f0 comp rx=0 tx=0).stop 2023-10-25T00:13:32.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 -- 172.21.15.107:0/2662301486 shutdown_connections 2023-10-25T00:13:32.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 --2- 172.21.15.107:0/2662301486 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ca8003980 0x7f4ca8003da0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:32.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 -- 172.21.15.107:0/2662301486 >> 172.21.15.107:0/2662301486 conn(0x7f4cbc0abd60 msgr2=0x7f4cbc0ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:32.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 -- 172.21.15.107:0/2662301486 shutdown_connections 2023-10-25T00:13:32.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 -- 172.21.15.107:0/2662301486 wait complete. 2023-10-25T00:13:32.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 Processor -- start 2023-10-25T00:13:32.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 -- start start 2023-10-25T00:13:32.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca8003980 0x7f4cbc0c3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:32.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cbc155d40 0x7f4cbc0c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:32.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc1ed2f0 0x7f4cbc1ed7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:32.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc1ee230 con 0x7f4cbc1ed2f0 2023-10-25T00:13:32.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc157010 con 0x7f4cbc155d40 2023-10-25T00:13:32.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc44bc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc1edce0 con 0x7f4ca8003980 2023-10-25T00:13:32.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc2a59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc1ed2f0 0x7f4cbc1ed7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:32.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc1a57640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cbc155d40 0x7f4cbc0c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:32.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc2a59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc1ed2f0 0x7f4cbc1ed7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45348/0 (socket says 172.21.15.107:45348) 2023-10-25T00:13:32.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc1a57640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cbc155d40 0x7f4cbc0c3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54274/0 (socket says 172.21.15.107:54274) 2023-10-25T00:13:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc2a59640 1 -- 172.21.15.107:0/1761169576 learned_addr learned my addr 172.21.15.107:0/1761169576 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc2258640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca8003980 0x7f4cbc0c3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc2a59640 1 -- 172.21.15.107:0/1761169576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca8003980 msgr2=0x7f4cbc0c3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc2a59640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca8003980 0x7f4cbc0c3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:32.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc2a59640 1 -- 172.21.15.107:0/1761169576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cbc155d40 msgr2=0x7f4cbc0c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc2a59640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cbc155d40 0x7f4cbc0c3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.962+0000 7f4cc2a59640 1 -- 172.21.15.107:0/1761169576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc157210 con 0x7f4cbc1ed2f0 2023-10-25T00:13:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.966+0000 7f4cc2a59640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc1ed2f0 0x7f4cbc1ed7a0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0698a0 tx=0x7f4cb400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:32.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.966+0000 7f4cb2ffd640 1 -- 172.21.15.107:0/1761169576 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb401c070 con 0x7f4cbc1ed2f0 2023-10-25T00:13:32.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.966+0000 7f4cb2ffd640 1 -- 172.21.15.107:0/1761169576 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb400f040 con 0x7f4cbc1ed2f0 2023-10-25T00:13:32.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.966+0000 7f4cc44bc640 1 -- 172.21.15.107:0/1761169576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cbc1edf50 con 0x7f4cbc1ed2f0 2023-10-25T00:13:32.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.966+0000 7f4cb2ffd640 1 -- 172.21.15.107:0/1761169576 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb400e040 con 0x7f4cbc1ed2f0 2023-10-25T00:13:32.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.966+0000 7f4cc44bc640 1 -- 172.21.15.107:0/1761169576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc14d8f0 con 0x7f4cbc1ed2f0 2023-10-25T00:13:32.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.966+0000 7f4cb0ff9640 1 -- 172.21.15.107:0/1761169576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cbc14d470 con 0x7f4cbc1ed2f0 2023-10-25T00:13:32.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.966+0000 7f4cb2ffd640 1 -- 172.21.15.107:0/1761169576 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4cb4015430 con 0x7f4cbc1ed2f0 2023-10-25T00:13:32.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.970+0000 7f4cb2ffd640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca0065eb0 0x7f4ca0068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:32.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.970+0000 7f4cc2258640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca0065eb0 0x7f4ca0068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:32.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.970+0000 7f4cb2ffd640 1 -- 172.21.15.107:0/1761169576 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 9799+0+0 (secure 0 0 0) 0x7f4cb4009be0 con 0x7f4cbc1ed2f0 2023-10-25T00:13:32.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.970+0000 7f4cc2258640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca0065eb0 0x7f4ca0068370 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80ad000 tx=0x7f4cb8001a10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:32.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:32.974+0000 7f4cb2ffd640 1 -- 172.21.15.107:0/1761169576 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4cbc14d470 con 0x7f4cbc1ed2f0 2023-10-25T00:13:33.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.350+0000 7f4cb0ff9640 1 -- 172.21.15.107:0/1761169576 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cbc14d470 con 0x7f4cbc1ed2f0 2023-10-25T00:13:33.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.354+0000 7f4cb2ffd640 1 -- 172.21.15.107:0/1761169576 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f4cbc14d470 con 0x7f4cbc1ed2f0 2023-10-25T00:13:33.353 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:33.354 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:33.354 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 175 2023-10-25T00:13:33.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 -- 172.21.15.107:0/1761169576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca0065eb0 msgr2=0x7f4ca0068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:33.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca0065eb0 0x7f4ca0068370 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80ad000 tx=0x7f4cb8001a10 comp rx=0 tx=0).stop 2023-10-25T00:13:33.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 -- 172.21.15.107:0/1761169576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc1ed2f0 msgr2=0x7f4cbc1ed7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:33.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc1ed2f0 0x7f4cbc1ed7a0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0698a0 tx=0x7f4cb400f580 comp rx=0 tx=0).stop 2023-10-25T00:13:33.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 -- 172.21.15.107:0/1761169576 shutdown_connections 2023-10-25T00:13:33.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cbc1ed2f0 0x7f4cbc1ed7a0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:33.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cbc155d40 0x7f4cbc0c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4ca0065eb0 0x7f4ca0068370 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 --2- 172.21.15.107:0/1761169576 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ca8003980 0x7f4cbc0c3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 -- 172.21.15.107:0/1761169576 >> 172.21.15.107:0/1761169576 conn(0x7f4cbc0abd60 msgr2=0x7f4cbc14a320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 -- 172.21.15.107:0/1761169576 shutdown_connections 2023-10-25T00:13:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.358+0000 7f4cb0ff9640 1 -- 172.21.15.107:0/1761169576 wait complete. 2023-10-25T00:13:33.380 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2023-10-25T00:13:33.381 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:13:33.381 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:13:33.395 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:13:33.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:33.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.650+0000 7fe065331640 1 Processor -- start 2023-10-25T00:13:33.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.650+0000 7fe065331640 1 -- start start 2023-10-25T00:13:33.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.654+0000 7fe065331640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 0x7fe06006af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:33.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.654+0000 7fe065331640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0600723f0 con 0x7fe0600c24c0 2023-10-25T00:13:33.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.654+0000 7fe065331640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe06006a380 con 0x7fe0601476e0 2023-10-25T00:13:33.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.654+0000 7fe065331640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe060069fd0 con 0x7fe0601516d0 2023-10-25T00:13:33.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.654+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 0x7fe06006af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:33.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.654+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 0x7fe06006af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:56126/0 (socket says 172.21.15.107:56126) 2023-10-25T00:13:33.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.654+0000 7fe05f7fe640 1 -- 172.21.15.107:0/759823826 learned_addr learned my addr 172.21.15.107:0/759823826 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:33.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.654+0000 7fe05f7fe640 1 -- 172.21.15.107:0/759823826 >> v1:172.21.15.110:6789/0 conn(0x7fe0601476e0 legacy=0x7fe06006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:33.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.654+0000 7fe05f7fe640 1 -- 172.21.15.107:0/759823826 >> v1:172.21.15.107:6789/0 conn(0x7fe0600c24c0 legacy=0x7fe06006a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:33.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.654+0000 7fe05f7fe640 1 -- 172.21.15.107:0/759823826 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe060147ac0 con 0x7fe0601516d0 2023-10-25T00:13:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.658+0000 7fe05f7fe640 1 --2- 172.21.15.107:0/759823826 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 0x7fe06006af20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe058009e80 tx=0x7fe05801c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5ecda138a12700f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.658+0000 7fe05dffb640 1 -- 172.21.15.107:0/759823826 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0580a5580 con 0x7fe0601516d0 2023-10-25T00:13:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.658+0000 7fe05dffb640 1 -- 172.21.15.107:0/759823826 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0580a56e0 con 0x7fe0601516d0 2023-10-25T00:13:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.658+0000 7fe05dffb640 1 -- 172.21.15.107:0/759823826 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0580a5a70 con 0x7fe0601516d0 2023-10-25T00:13:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.658+0000 7fe065331640 1 -- 172.21.15.107:0/759823826 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 msgr2=0x7fe06006af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.658+0000 7fe065331640 1 --2- 172.21.15.107:0/759823826 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 0x7fe06006af20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe058009e80 tx=0x7fe05801c890 comp rx=0 tx=0).stop 2023-10-25T00:13:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 -- 172.21.15.107:0/759823826 shutdown_connections 2023-10-25T00:13:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 --2- 172.21.15.107:0/759823826 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 0x7fe06006af20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:33.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 -- 172.21.15.107:0/759823826 >> 172.21.15.107:0/759823826 conn(0x7fe0600c2950 msgr2=0x7fe0600c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 -- 172.21.15.107:0/759823826 shutdown_connections 2023-10-25T00:13:33.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 -- 172.21.15.107:0/759823826 wait complete. 2023-10-25T00:13:33.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 Processor -- start 2023-10-25T00:13:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 -- start start 2023-10-25T00:13:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0600c24c0 0x7fe060150640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0601476e0 0x7fe060154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 0x7fe060151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0601394c0 con 0x7fe0601476e0 2023-10-25T00:13:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0601501b0 con 0x7fe0600c24c0 2023-10-25T00:13:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.662+0000 7fe065331640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0601546a0 con 0x7fe0601516d0 2023-10-25T00:13:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0601476e0 0x7fe060154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0600c24c0 0x7fe060150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0601476e0 0x7fe060154160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45362/0 (socket says 172.21.15.107:45362) 2023-10-25T00:13:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 0x7fe060151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05e7fc640 1 -- 172.21.15.107:0/692694331 learned_addr learned my addr 172.21.15.107:0/692694331 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05f7fe640 1 -- 172.21.15.107:0/692694331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0600c24c0 msgr2=0x7fe060150640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05f7fe640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0600c24c0 0x7fe060150640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05f7fe640 1 -- 172.21.15.107:0/692694331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0601476e0 msgr2=0x7fe060154160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05f7fe640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0601476e0 0x7fe060154160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05f7fe640 1 -- 172.21.15.107:0/692694331 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe058009b30 con 0x7fe0601516d0 2023-10-25T00:13:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05effd640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0600c24c0 0x7fe060150640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:13:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05f7fe640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 0x7fe060151e50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe0580ab000 tx=0x7fe058019630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe0477fe640 1 -- 172.21.15.107:0/692694331 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0580b8670 con 0x7fe0601516d0 2023-10-25T00:13:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe065331640 1 -- 172.21.15.107:0/692694331 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06013ee60 con 0x7fe0601516d0 2023-10-25T00:13:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe065331640 1 -- 172.21.15.107:0/692694331 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0601391e0 con 0x7fe0601516d0 2023-10-25T00:13:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe0477fe640 1 -- 172.21.15.107:0/692694331 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0580b87d0 con 0x7fe0601516d0 2023-10-25T00:13:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe0477fe640 1 -- 172.21.15.107:0/692694331 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0580b8b60 con 0x7fe0601516d0 2023-10-25T00:13:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe0477fe640 1 -- 172.21.15.107:0/692694331 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe0580b8d00 con 0x7fe0601516d0 2023-10-25T00:13:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe0477fe640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe04c065eb0 0x7fe04c068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe0477fe640 1 -- 172.21.15.107:0/692694331 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 9799+0+0 (secure 0 0 0) 0x7fe05811f9b0 con 0x7fe0601516d0 2023-10-25T00:13:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05effd640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe04c065eb0 0x7fe04c068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe065331640 1 -- 172.21.15.107:0/692694331 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0600746e0 con 0x7fe0601516d0 2023-10-25T00:13:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.666+0000 7fe05effd640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe04c065eb0 0x7fe04c068370 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe0601401a0 tx=0x7fe050009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:33.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:33.674+0000 7fe0477fe640 1 -- 172.21.15.107:0/692694331 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe0580bc080 con 0x7fe0601516d0 2023-10-25T00:13:34.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.030+0000 7fe065331640 1 -- 172.21.15.107:0/692694331 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0601502f0 con 0x7fe0601516d0 2023-10-25T00:13:34.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.030+0000 7fe0477fe640 1 -- 172.21.15.107:0/692694331 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5778 (secure 0 0 0) 0x7fe0580ec390 con 0x7fe0601516d0 2023-10-25T00:13:34.031 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:34.031 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:34.031 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 175 2023-10-25T00:13:34.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 -- 172.21.15.107:0/692694331 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe04c065eb0 msgr2=0x7fe04c068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:34.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe04c065eb0 0x7fe04c068370 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe0601401a0 tx=0x7fe050009060 comp rx=0 tx=0).stop 2023-10-25T00:13:34.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 -- 172.21.15.107:0/692694331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 msgr2=0x7fe060151e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:34.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 0x7fe060151e50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe0580ab000 tx=0x7fe058019630 comp rx=0 tx=0).stop 2023-10-25T00:13:34.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 -- 172.21.15.107:0/692694331 shutdown_connections 2023-10-25T00:13:34.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe04c065eb0 0x7fe04c068370 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:34.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0601516d0 0x7fe060151e50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:34.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0601476e0 0x7fe060154160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:34.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 --2- 172.21.15.107:0/692694331 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0600c24c0 0x7fe060150640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:34.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 -- 172.21.15.107:0/692694331 >> 172.21.15.107:0/692694331 conn(0x7fe0600c2950 msgr2=0x7fe060150df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:34.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 -- 172.21.15.107:0/692694331 shutdown_connections 2023-10-25T00:13:34.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:34.038+0000 7fe065331640 1 -- 172.21.15.107:0/692694331 wait complete. 2023-10-25T00:13:34.068 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:13:36.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:36.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.266+0000 7fc412d3c640 1 Processor -- start 2023-10-25T00:13:36.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc412d3c640 1 -- start start 2023-10-25T00:13:36.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc412d3c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc40c0ce070 0x7fc40c0caa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:36.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc412d3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc40c0c9a80 0x7fc40c165120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:36.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc412d3c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc40c0ca570 con 0x7fc40c072270 2023-10-25T00:13:36.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc412d3c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc40c06b670 con 0x7fc40c0ce070 2023-10-25T00:13:36.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc412d3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c0c9f30 con 0x7fc40c0c9a80 2023-10-25T00:13:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc40bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc40c0ce070 0x7fc40c0caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc40bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc40c0ce070 0x7fc40c0caa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45374/0 (socket says 172.21.15.107:45374) 2023-10-25T00:13:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc40bfff640 1 -- 172.21.15.107:0/2872431043 learned_addr learned my addr 172.21.15.107:0/2872431043 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc4112d9640 1 --2- 172.21.15.107:0/2872431043 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc40c0c9a80 0x7fc40c165120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:36.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc40bfff640 1 -- 172.21.15.107:0/2872431043 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc40c0c9a80 msgr2=0x7fc40c165120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:36.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc40bfff640 1 --2- 172.21.15.107:0/2872431043 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc40c0c9a80 0x7fc40c165120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:36.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc40bfff640 1 -- 172.21.15.107:0/2872431043 >> v1:172.21.15.107:6789/0 conn(0x7fc40c072270 legacy=0x7fc40c06af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:36.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.270+0000 7fc40bfff640 1 -- 172.21.15.107:0/2872431043 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c0c8f00 con 0x7fc40c0ce070 2023-10-25T00:13:36.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.274+0000 7fc40bfff640 1 --2- 172.21.15.107:0/2872431043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc40c0ce070 0x7fc40c0caa00 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0098a0 tx=0x7fc3fc01c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c733b128889c4c07 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:36.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.274+0000 7fc40b7fe640 1 -- 172.21.15.107:0/2872431043 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc0ab600 con 0x7fc40c0ce070 2023-10-25T00:13:36.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.274+0000 7fc40b7fe640 1 -- 172.21.15.107:0/2872431043 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc0a7050 con 0x7fc40c0ce070 2023-10-25T00:13:36.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.274+0000 7fc40b7fe640 1 -- 172.21.15.107:0/2872431043 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc0ab980 con 0x7fc40c0ce070 2023-10-25T00:13:36.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.274+0000 7fc412d3c640 1 -- 172.21.15.107:0/2872431043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc40c0ce070 msgr2=0x7fc40c0caa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:36.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.274+0000 7fc412d3c640 1 --2- 172.21.15.107:0/2872431043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc40c0ce070 0x7fc40c0caa00 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0098a0 tx=0x7fc3fc01c270 comp rx=0 tx=0).stop 2023-10-25T00:13:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.274+0000 7fc412d3c640 1 -- 172.21.15.107:0/2872431043 shutdown_connections 2023-10-25T00:13:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.274+0000 7fc412d3c640 1 --2- 172.21.15.107:0/2872431043 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc40c0c9a80 0x7fc40c165120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.274+0000 7fc412d3c640 1 --2- 172.21.15.107:0/2872431043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc40c0ce070 0x7fc40c0caa00 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.274+0000 7fc412d3c640 1 -- 172.21.15.107:0/2872431043 >> 172.21.15.107:0/2872431043 conn(0x7fc40c0ce580 msgr2=0x7fc40c0ce990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:36.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.274+0000 7fc412d3c640 1 -- 172.21.15.107:0/2872431043 shutdown_connections 2023-10-25T00:13:36.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc412d3c640 1 -- 172.21.15.107:0/2872431043 wait complete. 2023-10-25T00:13:36.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc412d3c640 1 Processor -- start 2023-10-25T00:13:36.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc412d3c640 1 -- start start 2023-10-25T00:13:36.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc412d3c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc40c072270 0x7fc40c0c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:36.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc412d3c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc40c0c9a80 0x7fc40c0b3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:36.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc412d3c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc40c0ce070 0x7fc40c151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:36.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc412d3c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c0b31e0 con 0x7fc40c0ce070 2023-10-25T00:13:36.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc412d3c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c164c80 con 0x7fc40c0c9a80 2023-10-25T00:13:36.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc412d3c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c0b2eb0 con 0x7fc40c072270 2023-10-25T00:13:36.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc4112d9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc40c0ce070 0x7fc40c151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:36.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc4112d9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc40c0ce070 0x7fc40c151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45384/0 (socket says 172.21.15.107:45384) 2023-10-25T00:13:36.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc4112d9640 1 -- 172.21.15.107:0/2995633419 learned_addr learned my addr 172.21.15.107:0/2995633419 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:36.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc410ad8640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc40c072270 0x7fc40c0c3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:36.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.278+0000 7fc40bfff640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc40c0c9a80 0x7fc40c0b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:36.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4112d9640 1 -- 172.21.15.107:0/2995633419 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc40c072270 msgr2=0x7fc40c0c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:36.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4112d9640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc40c072270 0x7fc40c0c3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:36.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4112d9640 1 -- 172.21.15.107:0/2995633419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc40c0c9a80 msgr2=0x7fc40c0b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4112d9640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc40c0c9a80 0x7fc40c0b3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4112d9640 1 -- 172.21.15.107:0/2995633419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3fc0094d0 con 0x7fc40c0ce070 2023-10-25T00:13:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4112d9640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc40c0ce070 0x7fc40c151cf0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fc40002f600 tx=0x7fc40002fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4097fa640 1 -- 172.21.15.107:0/2995633419 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc400036640 con 0x7fc40c0ce070 2023-10-25T00:13:36.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4097fa640 1 -- 172.21.15.107:0/2995633419 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4000367c0 con 0x7fc40c0ce070 2023-10-25T00:13:36.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc412d3c640 1 -- 172.21.15.107:0/2995633419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40c0c42f0 con 0x7fc40c0ce070 2023-10-25T00:13:36.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4097fa640 1 -- 172.21.15.107:0/2995633419 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc400036b70 con 0x7fc40c0ce070 2023-10-25T00:13:36.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc412d3c640 1 -- 172.21.15.107:0/2995633419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40c0c45d0 con 0x7fc40c0ce070 2023-10-25T00:13:36.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4097fa640 1 -- 172.21.15.107:0/2995633419 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc400036cd0 con 0x7fc40c0ce070 2023-10-25T00:13:36.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4097fa640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc3f8065e60 0x7fc3f8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:36.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc4097fa640 1 -- 172.21.15.107:0/2995633419 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 9799+0+0 (secure 0 0 0) 0x7fc4000a6860 con 0x7fc40c0ce070 2023-10-25T00:13:36.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.282+0000 7fc410ad8640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc3f8065e60 0x7fc3f8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:36.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.286+0000 7fc410ad8640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc3f8065e60 0x7fc3f8068320 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc4040012f0 tx=0x7fc404009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:36.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.286+0000 7fc412d3c640 1 -- 172.21.15.107:0/2995633419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40c0b3350 con 0x7fc40c0ce070 2023-10-25T00:13:36.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.290+0000 7fc4097fa640 1 -- 172.21.15.107:0/2995633419 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc400073100 con 0x7fc40c0ce070 2023-10-25T00:13:36.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.714+0000 7fc412d3c640 1 -- 172.21.15.107:0/2995633419 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc40c06d710 con 0x7fc40c0ce070 2023-10-25T00:13:36.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.714+0000 7fc4097fa640 1 -- 172.21.15.107:0/2995633419 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5778 (secure 0 0 0) 0x7fc400076720 con 0x7fc40c0ce070 2023-10-25T00:13:36.717 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:36.717 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:36.717 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 175 2023-10-25T00:13:36.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.718+0000 7fc412d3c640 1 -- 172.21.15.107:0/2995633419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc3f8065e60 msgr2=0x7fc3f8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:36.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.718+0000 7fc412d3c640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc3f8065e60 0x7fc3f8068320 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc4040012f0 tx=0x7fc404009040 comp rx=0 tx=0).stop 2023-10-25T00:13:36.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.718+0000 7fc412d3c640 1 -- 172.21.15.107:0/2995633419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc40c0ce070 msgr2=0x7fc40c151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:36.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.718+0000 7fc412d3c640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc40c0ce070 0x7fc40c151cf0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fc40002f600 tx=0x7fc40002fac0 comp rx=0 tx=0).stop 2023-10-25T00:13:36.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.718+0000 7fc412d3c640 1 -- 172.21.15.107:0/2995633419 shutdown_connections 2023-10-25T00:13:36.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.718+0000 7fc412d3c640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc40c0ce070 0x7fc40c151cf0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:36.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.718+0000 7fc412d3c640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc40c0c9a80 0x7fc40c0b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:36.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.718+0000 7fc412d3c640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc3f8065e60 0x7fc3f8068320 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:36.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.718+0000 7fc412d3c640 1 --2- 172.21.15.107:0/2995633419 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc40c072270 0x7fc40c0c3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:36.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.718+0000 7fc412d3c640 1 -- 172.21.15.107:0/2995633419 >> 172.21.15.107:0/2995633419 conn(0x7fc40c0ce580 msgr2=0x7fc40c0cc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:36.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.722+0000 7fc412d3c640 1 -- 172.21.15.107:0/2995633419 shutdown_connections 2023-10-25T00:13:36.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:36.722+0000 7fc412d3c640 1 -- 172.21.15.107:0/2995633419 wait complete. 2023-10-25T00:13:36.775 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:13:38.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:39.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a3d8fc640 1 Processor -- start 2023-10-25T00:13:39.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a3d8fc640 1 -- start start 2023-10-25T00:13:39.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a3d8fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5a3000b5c0 0x7f5a300475d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:39.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a3d8fc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300075b0 con 0x7f5a3003ed50 2023-10-25T00:13:39.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a3d8fc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a30007200 con 0x7f5a300dc0d0 2023-10-25T00:13:39.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a3d8fc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a300df930 con 0x7f5a3000b5c0 2023-10-25T00:13:39.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a3c8fa640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f5a300dc0d0 0x7f5a3000bbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:42982/0 (socket says 172.21.15.107:42982) 2023-10-25T00:13:39.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a3c8fa640 1 -- 172.21.15.107:0/1419171186 learned_addr learned my addr 172.21.15.107:0/1419171186 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:13:39.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a37fff640 1 --2- 172.21.15.107:0/1419171186 >> v2:172.21.15.107:3300/0 conn(0x7f5a3000b5c0 0x7f5a300475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:39.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a377fe640 1 -- 172.21.15.107:0/1419171186 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602620204 0 0) 0x7f5a30007200 con 0x7f5a300dc0d0 2023-10-25T00:13:39.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a377fe640 1 -- 172.21.15.107:0/1419171186 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a28003530 con 0x7f5a300dc0d0 2023-10-25T00:13:39.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a377fe640 1 -- 172.21.15.107:0/1419171186 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656347354 0 0) 0x7f5a300075b0 con 0x7f5a3003ed50 2023-10-25T00:13:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a37fff640 1 -- 172.21.15.107:0/1419171186 >> v1:172.21.15.110:6789/0 conn(0x7f5a300dc0d0 legacy=0x7f5a3000bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a37fff640 1 -- 172.21.15.107:0/1419171186 >> v1:172.21.15.107:6789/0 conn(0x7f5a3003ed50 legacy=0x7f5a300e9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.030+0000 7f5a37fff640 1 -- 172.21.15.107:0/1419171186 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a30007800 con 0x7f5a3000b5c0 2023-10-25T00:13:39.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.034+0000 7f5a37fff640 1 --2- 172.21.15.107:0/1419171186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3000b5c0 0x7f5a300475d0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f5a38063dd0 tx=0x7f5a38096d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48f1e8f4e078a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:39.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.034+0000 7f5a377fe640 1 -- 172.21.15.107:0/1419171186 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3811f990 con 0x7f5a3000b5c0 2023-10-25T00:13:39.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.034+0000 7f5a377fe640 1 -- 172.21.15.107:0/1419171186 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3811faf0 con 0x7f5a3000b5c0 2023-10-25T00:13:39.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.038+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/1419171186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3000b5c0 msgr2=0x7f5a300475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:39.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.038+0000 7f5a3d8fc640 1 --2- 172.21.15.107:0/1419171186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3000b5c0 0x7f5a300475d0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f5a38063dd0 tx=0x7f5a38096d90 comp rx=0 tx=0).stop 2023-10-25T00:13:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.038+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/1419171186 shutdown_connections 2023-10-25T00:13:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.038+0000 7f5a3d8fc640 1 --2- 172.21.15.107:0/1419171186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3000b5c0 0x7f5a300475d0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.038+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/1419171186 >> 172.21.15.107:0/1419171186 conn(0x7f5a30006440 msgr2=0x7f5a30006850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:39.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.038+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/1419171186 shutdown_connections 2023-10-25T00:13:39.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.038+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/1419171186 wait complete. 2023-10-25T00:13:39.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.038+0000 7f5a3d8fc640 1 Processor -- start 2023-10-25T00:13:39.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.038+0000 7f5a3d8fc640 1 -- start start 2023-10-25T00:13:39.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d8fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3000b5c0 0x7f5a300d6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:39.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d8fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a3003ed50 0x7f5a30040ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:39.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d8fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a300dc0d0 0x7f5a300411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:39.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d8fc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a30041c70 con 0x7f5a3000b5c0 2023-10-25T00:13:39.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d8fc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a30040810 con 0x7f5a300dc0d0 2023-10-25T00:13:39.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d8fc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3003fc00 con 0x7f5a3003ed50 2023-10-25T00:13:39.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d0fb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a300dc0d0 0x7f5a300411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:39.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d0fb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a300dc0d0 0x7f5a300411e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:54314/0 (socket says 172.21.15.107:54314) 2023-10-25T00:13:39.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d0fb640 1 -- 172.21.15.107:0/2105868563 learned_addr learned my addr 172.21.15.107:0/2105868563 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:39.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a37fff640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a3003ed50 0x7f5a30040ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:39.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d0fb640 1 -- 172.21.15.107:0/2105868563 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a3003ed50 msgr2=0x7f5a30040ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:39.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d0fb640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a3003ed50 0x7f5a30040ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:39.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d0fb640 1 -- 172.21.15.107:0/2105868563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3000b5c0 msgr2=0x7f5a300d6650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:13:39.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3c8fa640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3000b5c0 0x7f5a300d6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:39.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d0fb640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3000b5c0 0x7f5a300d6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:39.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d0fb640 1 -- 172.21.15.107:0/2105868563 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3804e5c0 con 0x7f5a300dc0d0 2023-10-25T00:13:39.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3c8fa640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3000b5c0 0x7f5a300d6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:13:39.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d0fb640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a300dc0d0 0x7f5a300411e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5a2400c450 tx=0x7f5a2400c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:39.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a357fa640 1 -- 172.21.15.107:0/2105868563 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a240054e0 con 0x7f5a300dc0d0 2023-10-25T00:13:39.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a357fa640 1 -- 172.21.15.107:0/2105868563 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a24003600 con 0x7f5a300dc0d0 2023-10-25T00:13:39.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a357fa640 1 -- 172.21.15.107:0/2105868563 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a24003990 con 0x7f5a300dc0d0 2023-10-25T00:13:39.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/2105868563 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3003fee0 con 0x7f5a300dc0d0 2023-10-25T00:13:39.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.042+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/2105868563 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a300404a0 con 0x7f5a300dc0d0 2023-10-25T00:13:39.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.046+0000 7f5a357fa640 1 -- 172.21.15.107:0/2105868563 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5a24003af0 con 0x7f5a300dc0d0 2023-10-25T00:13:39.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.046+0000 7f5a357fa640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a1c065e90 0x7f5a1c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:39.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.046+0000 7f5a3c8fa640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a1c065e90 0x7f5a1c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:39.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.046+0000 7f5a357fa640 1 -- 172.21.15.107:0/2105868563 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 9799+0+0 (secure 0 0 0) 0x7f5a24083c70 con 0x7f5a300dc0d0 2023-10-25T00:13:39.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.050+0000 7f5a3c8fa640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a1c065e90 0x7f5a1c068350 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0011f0 tx=0x7f5a2c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:39.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.050+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/2105868563 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a300409c0 con 0x7f5a300dc0d0 2023-10-25T00:13:39.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.058+0000 7f5a357fa640 1 -- 172.21.15.107:0/2105868563 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5a300409c0 con 0x7f5a300dc0d0 2023-10-25T00:13:39.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.510+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/2105868563 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a300409c0 con 0x7f5a300dc0d0 2023-10-25T00:13:39.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.514+0000 7f5a357fa640 1 -- 172.21.15.107:0/2105868563 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f5a300409c0 con 0x7f5a300dc0d0 2023-10-25T00:13:39.519 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:39.519 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:39.519 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 175 2023-10-25T00:13:39.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.522+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/2105868563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a1c065e90 msgr2=0x7f5a1c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:39.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.522+0000 7f5a3d8fc640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a1c065e90 0x7f5a1c068350 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c0011f0 tx=0x7f5a2c009040 comp rx=0 tx=0).stop 2023-10-25T00:13:39.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.522+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/2105868563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a300dc0d0 msgr2=0x7f5a300411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:39.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.522+0000 7f5a3d8fc640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a300dc0d0 0x7f5a300411e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5a2400c450 tx=0x7f5a2400c910 comp rx=0 tx=0).stop 2023-10-25T00:13:39.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.522+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/2105868563 shutdown_connections 2023-10-25T00:13:39.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.522+0000 7f5a3d8fc640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5a300dc0d0 0x7f5a300411e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:39.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.522+0000 7f5a3d8fc640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5a1c065e90 0x7f5a1c068350 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:39.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.522+0000 7f5a3d8fc640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5a3003ed50 0x7f5a30040ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:39.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.522+0000 7f5a3d8fc640 1 --2- 172.21.15.107:0/2105868563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a3000b5c0 0x7f5a300d6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:39.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.522+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/2105868563 >> 172.21.15.107:0/2105868563 conn(0x7f5a30006440 msgr2=0x7f5a300d2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:39.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.522+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/2105868563 shutdown_connections 2023-10-25T00:13:39.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:39.526+0000 7f5a3d8fc640 1 -- 172.21.15.107:0/2105868563 wait complete. 2023-10-25T00:13:39.551 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:13:41.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:41.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.770+0000 7f8f994ad640 1 Processor -- start 2023-10-25T00:13:41.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.770+0000 7f8f994ad640 1 -- start start 2023-10-25T00:13:41.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.770+0000 7f8f994ad640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8f940c3e00 0x7f8f940c4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:41.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.770+0000 7f8f994ad640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f940c59d0 con 0x7f8f94155ea0 2023-10-25T00:13:41.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.770+0000 7f8f994ad640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f940c2990 con 0x7f8f9406ec90 2023-10-25T00:13:41.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.770+0000 7f8f994ad640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f940c2ec0 con 0x7f8f940c3e00 2023-10-25T00:13:41.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.770+0000 7f8f937fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8f940c3e00 0x7f8f940c4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:41.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.770+0000 7f8f937fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8f940c3e00 0x7f8f940c4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45398/0 (socket says 172.21.15.107:45398) 2023-10-25T00:13:41.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.770+0000 7f8f937fe640 1 -- 172.21.15.107:0/1865015505 learned_addr learned my addr 172.21.15.107:0/1865015505 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:41.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.774+0000 7f8f937fe640 1 -- 172.21.15.107:0/1865015505 >> v1:172.21.15.110:6789/0 conn(0x7f8f9406ec90 legacy=0x7f8f9414c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:41.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.774+0000 7f8f937fe640 1 -- 172.21.15.107:0/1865015505 >> v1:172.21.15.107:6789/0 conn(0x7f8f94155ea0 legacy=0x7f8f94156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:41.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.774+0000 7f8f937fe640 1 -- 172.21.15.107:0/1865015505 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9406d500 con 0x7f8f940c3e00 2023-10-25T00:13:41.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.774+0000 7f8f937fe640 1 --2- 172.21.15.107:0/1865015505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f940c3e00 0x7f8f940c4220 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009e50 tx=0x7f8f8c01c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94caf31ddd1504d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:41.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.778+0000 7f8f91ffb640 1 -- 172.21.15.107:0/1865015505 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c0a57b0 con 0x7f8f940c3e00 2023-10-25T00:13:41.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.778+0000 7f8f91ffb640 1 -- 172.21.15.107:0/1865015505 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c0a5910 con 0x7f8f940c3e00 2023-10-25T00:13:41.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.778+0000 7f8f994ad640 1 -- 172.21.15.107:0/1865015505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f940c3e00 msgr2=0x7f8f940c4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:41.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.778+0000 7f8f91ffb640 1 -- 172.21.15.107:0/1865015505 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c0a57b0 con 0x7f8f940c3e00 2023-10-25T00:13:41.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.778+0000 7f8f994ad640 1 --2- 172.21.15.107:0/1865015505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f940c3e00 0x7f8f940c4220 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009e50 tx=0x7f8f8c01c8f0 comp rx=0 tx=0).stop 2023-10-25T00:13:41.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.778+0000 7f8f994ad640 1 -- 172.21.15.107:0/1865015505 shutdown_connections 2023-10-25T00:13:41.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.778+0000 7f8f994ad640 1 --2- 172.21.15.107:0/1865015505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f940c3e00 0x7f8f940c4220 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:41.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.778+0000 7f8f994ad640 1 -- 172.21.15.107:0/1865015505 >> 172.21.15.107:0/1865015505 conn(0x7f8f940c4a90 msgr2=0x7f8f940c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:41.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.782+0000 7f8f994ad640 1 -- 172.21.15.107:0/1865015505 shutdown_connections 2023-10-25T00:13:41.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.782+0000 7f8f994ad640 1 -- 172.21.15.107:0/1865015505 wait complete. 2023-10-25T00:13:41.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.782+0000 7f8f994ad640 1 Processor -- start 2023-10-25T00:13:41.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.782+0000 7f8f994ad640 1 -- start start 2023-10-25T00:13:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f994ad640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8f9406ec90 0x7f8f940c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f994ad640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f940c3e00 0x7f8f941f9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:41.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f994ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f94155ea0 0x7f8f94145520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:41.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f994ad640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f940c66a0 con 0x7f8f94155ea0 2023-10-25T00:13:41.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f994ad640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9414a230 con 0x7f8f940c3e00 2023-10-25T00:13:41.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f994ad640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f941fa140 con 0x7f8f9406ec90 2023-10-25T00:13:41.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f94155ea0 0x7f8f94145520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:41.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f94155ea0 0x7f8f94145520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45408/0 (socket says 172.21.15.107:45408) 2023-10-25T00:13:41.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f937fe640 1 -- 172.21.15.107:0/2191776950 learned_addr learned my addr 172.21.15.107:0/2191776950 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:41.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f927fc640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f940c3e00 0x7f8f941f9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:41.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f937fe640 1 -- 172.21.15.107:0/2191776950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8f9406ec90 msgr2=0x7f8f940c5df0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-25T00:13:41.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f92ffd640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8f9406ec90 0x7f8f940c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:41.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f937fe640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8f9406ec90 0x7f8f940c5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f937fe640 1 -- 172.21.15.107:0/2191776950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f940c3e00 msgr2=0x7f8f941f9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f937fe640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f940c3e00 0x7f8f941f9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f937fe640 1 -- 172.21.15.107:0/2191776950 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c009b30 con 0x7f8f94155ea0 2023-10-25T00:13:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f937fe640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f94155ea0 0x7f8f94145520 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c019500 tx=0x7f8f8c019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:41.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f7b7fe640 1 -- 172.21.15.107:0/2191776950 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c003db0 con 0x7f8f94155ea0 2023-10-25T00:13:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f7b7fe640 1 -- 172.21.15.107:0/2191776950 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c004040 con 0x7f8f94155ea0 2023-10-25T00:13:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f7b7fe640 1 -- 172.21.15.107:0/2191776950 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c0043b0 con 0x7f8f94155ea0 2023-10-25T00:13:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f994ad640 1 -- 172.21.15.107:0/2191776950 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f941460b0 con 0x7f8f94155ea0 2023-10-25T00:13:41.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f994ad640 1 -- 172.21.15.107:0/2191776950 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f940c7c20 con 0x7f8f94155ea0 2023-10-25T00:13:41.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.786+0000 7f8f994ad640 1 -- 172.21.15.107:0/2191776950 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f940c7a60 con 0x7f8f94155ea0 2023-10-25T00:13:41.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.790+0000 7f8f7b7fe640 1 -- 172.21.15.107:0/2191776950 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8f8c004510 con 0x7f8f94155ea0 2023-10-25T00:13:41.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.790+0000 7f8f7b7fe640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8f80065e60 0x7f8f80068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:41.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.790+0000 7f8f7b7fe640 1 -- 172.21.15.107:0/2191776950 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 9799+0+0 (secure 0 0 0) 0x7f8f8c1200f0 con 0x7f8f94155ea0 2023-10-25T00:13:41.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.798+0000 7f8f7b7fe640 1 -- 172.21.15.107:0/2191776950 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8f8c0ec800 con 0x7f8f94155ea0 2023-10-25T00:13:41.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.798+0000 7f8f92ffd640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8f80065e60 0x7f8f80068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:41.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:41.798+0000 7f8f92ffd640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8f80065e60 0x7f8f80068320 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8f84007fd0 tx=0x7f8f84009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:42.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.210+0000 7f8f994ad640 1 -- 172.21.15.107:0/2191776950 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f940c7db0 con 0x7f8f94155ea0 2023-10-25T00:13:42.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.210+0000 7f8f7b7fe640 1 -- 172.21.15.107:0/2191776950 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f8f8c0efe20 con 0x7f8f94155ea0 2023-10-25T00:13:42.212 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:42.212 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:42.212 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 175 2023-10-25T00:13:42.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 -- 172.21.15.107:0/2191776950 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8f80065e60 msgr2=0x7f8f80068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:42.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8f80065e60 0x7f8f80068320 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8f84007fd0 tx=0x7f8f84009040 comp rx=0 tx=0).stop 2023-10-25T00:13:42.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 -- 172.21.15.107:0/2191776950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f94155ea0 msgr2=0x7f8f94145520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:42.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f94155ea0 0x7f8f94145520 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c019500 tx=0x7f8f8c019690 comp rx=0 tx=0).stop 2023-10-25T00:13:42.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 -- 172.21.15.107:0/2191776950 shutdown_connections 2023-10-25T00:13:42.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f94155ea0 0x7f8f94145520 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:42.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8f940c3e00 0x7f8f941f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:42.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8f80065e60 0x7f8f80068320 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:42.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 --2- 172.21.15.107:0/2191776950 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8f9406ec90 0x7f8f940c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:42.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 -- 172.21.15.107:0/2191776950 >> 172.21.15.107:0/2191776950 conn(0x7f8f940c4a90 msgr2=0x7f8f9413cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:42.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 -- 172.21.15.107:0/2191776950 shutdown_connections 2023-10-25T00:13:42.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:42.222+0000 7f8f994ad640 1 -- 172.21.15.107:0/2191776950 wait complete. 2023-10-25T00:13:42.250 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:13:44.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:44.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac8858d640 1 Processor -- start 2023-10-25T00:13:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac8858d640 1 -- start start 2023-10-25T00:13:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac8858d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fac80070b00 0x7fac800c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac8858d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8006df30 0x7fac8006e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:44.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac8858d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac800c22f0 con 0x7fac8014eb70 2023-10-25T00:13:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac8858d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac80071700 con 0x7fac80070b00 2023-10-25T00:13:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac8858d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac800c25d0 con 0x7fac8006df30 2023-10-25T00:13:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac86329640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fac8014eb70 0x7fac801514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49944/0 (socket says 172.21.15.107:49944) 2023-10-25T00:13:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac85b28640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fac80070b00 0x7fac800c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:44.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac86329640 1 -- 172.21.15.107:0/1227562690 learned_addr learned my addr 172.21.15.107:0/1227562690 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:13:44.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac86b2a640 1 --2- 172.21.15.107:0/1227562690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8006df30 0x7fac8006e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:44.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac85b28640 1 -- 172.21.15.107:0/1227562690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8006df30 msgr2=0x7fac8006e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:44.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac85b28640 1 --2- 172.21.15.107:0/1227562690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8006df30 0x7fac8006e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:44.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac85b28640 1 -- 172.21.15.107:0/1227562690 >> v1:172.21.15.107:6789/0 conn(0x7fac8014eb70 legacy=0x7fac801514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:44.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.442+0000 7fac85b28640 1 -- 172.21.15.107:0/1227562690 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac800c2be0 con 0x7fac80070b00 2023-10-25T00:13:44.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.446+0000 7fac85b28640 1 --2- 172.21.15.107:0/1227562690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac80070b00 0x7fac800c33e0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fac700097c0 tx=0x7fac7001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55be08e1077b1296 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:44.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.446+0000 7fac85327640 1 -- 172.21.15.107:0/1227562690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac700a6040 con 0x7fac80070b00 2023-10-25T00:13:44.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.446+0000 7fac85327640 1 -- 172.21.15.107:0/1227562690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac700aa600 con 0x7fac80070b00 2023-10-25T00:13:44.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.446+0000 7fac85327640 1 -- 172.21.15.107:0/1227562690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac700a6040 con 0x7fac80070b00 2023-10-25T00:13:44.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.446+0000 7fac8858d640 1 -- 172.21.15.107:0/1227562690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac80070b00 msgr2=0x7fac800c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:44.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.446+0000 7fac8858d640 1 --2- 172.21.15.107:0/1227562690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac80070b00 0x7fac800c33e0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fac700097c0 tx=0x7fac7001c480 comp rx=0 tx=0).stop 2023-10-25T00:13:44.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.446+0000 7fac8858d640 1 -- 172.21.15.107:0/1227562690 shutdown_connections 2023-10-25T00:13:44.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.446+0000 7fac8858d640 1 --2- 172.21.15.107:0/1227562690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8006df30 0x7fac8006e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:44.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.446+0000 7fac8858d640 1 --2- 172.21.15.107:0/1227562690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac80070b00 0x7fac800c33e0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:44.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.446+0000 7fac8858d640 1 -- 172.21.15.107:0/1227562690 >> 172.21.15.107:0/1227562690 conn(0x7fac800c39a0 msgr2=0x7fac800c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.450+0000 7fac8858d640 1 -- 172.21.15.107:0/1227562690 shutdown_connections 2023-10-25T00:13:44.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.450+0000 7fac8858d640 1 -- 172.21.15.107:0/1227562690 wait complete. 2023-10-25T00:13:44.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.450+0000 7fac8858d640 1 Processor -- start 2023-10-25T00:13:44.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.450+0000 7fac8858d640 1 -- start start 2023-10-25T00:13:44.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac8858d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8006df30 0x7fac8006ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:44.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac8858d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80070b00 0x7fac801513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:44.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac8858d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8014eb70 0x7fac801f0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:44.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac8858d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8006b910 con 0x7fac8014eb70 2023-10-25T00:13:44.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac8858d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac801efcf0 con 0x7fac80070b00 2023-10-25T00:13:44.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac8858d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8014b8c0 con 0x7fac8006df30 2023-10-25T00:13:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac86b2a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8014eb70 0x7fac801f0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac86b2a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8014eb70 0x7fac801f0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53914/0 (socket says 172.21.15.107:53914) 2023-10-25T00:13:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac86b2a640 1 -- 172.21.15.107:0/2965333582 learned_addr learned my addr 172.21.15.107:0/2965333582 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac86329640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8006df30 0x7fac8006ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:44.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac85b28640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80070b00 0x7fac801513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac86329640 1 -- 172.21.15.107:0/2965333582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80070b00 msgr2=0x7fac801513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac86329640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80070b00 0x7fac801513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac86329640 1 -- 172.21.15.107:0/2965333582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8014eb70 msgr2=0x7fac801f0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac86329640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8014eb70 0x7fac801f0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:44.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac86329640 1 -- 172.21.15.107:0/2965333582 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac70009420 con 0x7fac8006df30 2023-10-25T00:13:44.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac86329640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8006df30 0x7fac8006ab80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fac8006e910 tx=0x7fac7c00c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:44.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac76ffd640 1 -- 172.21.15.107:0/2965333582 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c005cf0 con 0x7fac8006df30 2023-10-25T00:13:44.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac76ffd640 1 -- 172.21.15.107:0/2965333582 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c005e70 con 0x7fac8006df30 2023-10-25T00:13:44.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac8858d640 1 -- 172.21.15.107:0/2965333582 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac8006c7a0 con 0x7fac8006df30 2023-10-25T00:13:44.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac76ffd640 1 -- 172.21.15.107:0/2965333582 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0072f0 con 0x7fac8006df30 2023-10-25T00:13:44.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.454+0000 7fac8858d640 1 -- 172.21.15.107:0/2965333582 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac80151900 con 0x7fac8006df30 2023-10-25T00:13:44.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.458+0000 7fac76ffd640 1 -- 172.21.15.107:0/2965333582 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fac7c007450 con 0x7fac8006df30 2023-10-25T00:13:44.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.458+0000 7fac8858d640 1 -- 172.21.15.107:0/2965333582 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac800746e0 con 0x7fac8006df30 2023-10-25T00:13:44.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.458+0000 7fac76ffd640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac64065e60 0x7fac64068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:44.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.458+0000 7fac85b28640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac64065e60 0x7fac64068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:44.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.458+0000 7fac76ffd640 1 -- 172.21.15.107:0/2965333582 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 9799+0+0 (secure 0 0 0) 0x7fac7c083c00 con 0x7fac8006df30 2023-10-25T00:13:44.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.458+0000 7fac85b28640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac64065e60 0x7fac64068320 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fac700ae000 tx=0x7fac70008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:44.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.466+0000 7fac76ffd640 1 -- 172.21.15.107:0/2965333582 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fac7c050440 con 0x7fac8006df30 2023-10-25T00:13:44.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.902+0000 7fac8858d640 1 -- 172.21.15.107:0/2965333582 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac800c1a50 con 0x7fac8006df30 2023-10-25T00:13:44.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.902+0000 7fac76ffd640 1 -- 172.21.15.107:0/2965333582 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5778 (secure 0 0 0) 0x7fac7c017090 con 0x7fac8006df30 2023-10-25T00:13:44.903 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:44.903 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:44.903 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 175 2023-10-25T00:13:44.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.906+0000 7fac8858d640 1 -- 172.21.15.107:0/2965333582 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac64065e60 msgr2=0x7fac64068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:44.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.906+0000 7fac8858d640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac64065e60 0x7fac64068320 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fac700ae000 tx=0x7fac70008040 comp rx=0 tx=0).stop 2023-10-25T00:13:44.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.906+0000 7fac8858d640 1 -- 172.21.15.107:0/2965333582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8006df30 msgr2=0x7fac8006ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:44.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.906+0000 7fac8858d640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8006df30 0x7fac8006ab80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fac8006e910 tx=0x7fac7c00c820 comp rx=0 tx=0).stop 2023-10-25T00:13:44.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.910+0000 7fac8858d640 1 -- 172.21.15.107:0/2965333582 shutdown_connections 2023-10-25T00:13:44.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.910+0000 7fac8858d640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fac8014eb70 0x7fac801f0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:44.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.910+0000 7fac8858d640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fac80070b00 0x7fac801513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:44.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.910+0000 7fac8858d640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fac64065e60 0x7fac64068320 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:44.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.910+0000 7fac8858d640 1 --2- 172.21.15.107:0/2965333582 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fac8006df30 0x7fac8006ab80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:44.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.910+0000 7fac8858d640 1 -- 172.21.15.107:0/2965333582 >> 172.21.15.107:0/2965333582 conn(0x7fac800c39a0 msgr2=0x7fac800c4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:44.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.910+0000 7fac8858d640 1 -- 172.21.15.107:0/2965333582 shutdown_connections 2023-10-25T00:13:44.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:44.910+0000 7fac8858d640 1 -- 172.21.15.107:0/2965333582 wait complete. 2023-10-25T00:13:44.946 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:13:46.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:47.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.162+0000 7f859dfb3640 1 Processor -- start 2023-10-25T00:13:47.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.162+0000 7f859dfb3640 1 -- start start 2023-10-25T00:13:47.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f859dfb3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85981562b0 0x7f8598158d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:47.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f859dfb3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f859815c340 0x7f85980c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:47.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f859dfb3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f85980c50c0 0x7f85980ac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:47.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f859dfb3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8598159290 con 0x7f85980c50c0 2023-10-25T00:13:47.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f859dfb3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8598159d40 con 0x7f859815c340 2023-10-25T00:13:47.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f859dfb3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980c83a0 con 0x7f85981562b0 2023-10-25T00:13:47.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f85977fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85981562b0 0x7f8598158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:47.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f85977fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85981562b0 0x7f8598158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:52492/0 (socket says 172.21.15.107:52492) 2023-10-25T00:13:47.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f85977fe640 1 -- 172.21.15.107:0/1238078354 learned_addr learned my addr 172.21.15.107:0/1238078354 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:47.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f8597fff640 1 --2- 172.21.15.107:0/1238078354 >> v2:172.21.15.107:3300/0 conn(0x7f85980c50c0 0x7f85980ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:47.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f8596ffd640 1 --2- 172.21.15.107:0/1238078354 >> v2:172.21.15.110:3300/0 conn(0x7f859815c340 0x7f85980c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:47.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f85977fe640 1 -- 172.21.15.107:0/1238078354 >> v2:172.21.15.110:3300/0 conn(0x7f859815c340 msgr2=0x7f85980c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:47.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f85977fe640 1 --2- 172.21.15.107:0/1238078354 >> v2:172.21.15.110:3300/0 conn(0x7f859815c340 0x7f85980c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:47.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f85977fe640 1 -- 172.21.15.107:0/1238078354 >> v2:172.21.15.107:3300/0 conn(0x7f85980c50c0 msgr2=0x7f85980ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:47.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f85977fe640 1 --2- 172.21.15.107:0/1238078354 >> v2:172.21.15.107:3300/0 conn(0x7f85980c50c0 0x7f85980ac330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:47.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f85977fe640 1 -- 172.21.15.107:0/1238078354 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85981594a0 con 0x7f85981562b0 2023-10-25T00:13:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f85977fe640 1 --2- 172.21.15.107:0/1238078354 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85981562b0 0x7f8598158d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8598069840 tx=0x7f858801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c74e78fe449c5059 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.166+0000 7f85967fc640 1 -- 172.21.15.107:0/1238078354 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85880ab620 con 0x7f85981562b0 2023-10-25T00:13:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f85967fc640 1 -- 172.21.15.107:0/1238078354 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85880ab780 con 0x7f85981562b0 2023-10-25T00:13:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f85967fc640 1 -- 172.21.15.107:0/1238078354 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85880abb10 con 0x7f85981562b0 2023-10-25T00:13:47.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f859dfb3640 1 -- 172.21.15.107:0/1238078354 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85981562b0 msgr2=0x7f8598158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:47.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f859dfb3640 1 --2- 172.21.15.107:0/1238078354 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85981562b0 0x7f8598158d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8598069840 tx=0x7f858801c440 comp rx=0 tx=0).stop 2023-10-25T00:13:47.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f859dfb3640 1 -- 172.21.15.107:0/1238078354 shutdown_connections 2023-10-25T00:13:47.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f859dfb3640 1 --2- 172.21.15.107:0/1238078354 >> v2:172.21.15.107:3300/0 conn(0x7f85980c50c0 0x7f85980ac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:47.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f859dfb3640 1 --2- 172.21.15.107:0/1238078354 >> v2:172.21.15.110:3300/0 conn(0x7f859815c340 0x7f85980c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:47.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f859dfb3640 1 --2- 172.21.15.107:0/1238078354 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85981562b0 0x7f8598158d50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:47.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f859dfb3640 1 -- 172.21.15.107:0/1238078354 >> 172.21.15.107:0/1238078354 conn(0x7f85980ca200 msgr2=0x7f85980ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:47.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f859dfb3640 1 -- 172.21.15.107:0/1238078354 shutdown_connections 2023-10-25T00:13:47.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f859dfb3640 1 -- 172.21.15.107:0/1238078354 wait complete. 2023-10-25T00:13:47.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f859dfb3640 1 Processor -- start 2023-10-25T00:13:47.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.170+0000 7f859dfb3640 1 -- start start 2023-10-25T00:13:47.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f859dfb3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980c50c0 0x7f8598152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:47.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f859dfb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85981562b0 0x7f8598156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:47.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f859dfb3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f859815c340 0x7f8598157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:47.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f859dfb3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859815bb00 con 0x7f85981562b0 2023-10-25T00:13:47.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f859dfb3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859815b070 con 0x7f859815c340 2023-10-25T00:13:47.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f859dfb3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859815b4b0 con 0x7f85980c50c0 2023-10-25T00:13:47.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f8597fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f859815c340 0x7f8598157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:47.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f8597fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f859815c340 0x7f8598157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45546/0 (socket says 172.21.15.107:45546) 2023-10-25T00:13:47.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f8597fff640 1 -- 172.21.15.107:0/2192179627 learned_addr learned my addr 172.21.15.107:0/2192179627 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:47.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f85977fe640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980c50c0 0x7f8598152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:47.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f8596ffd640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85981562b0 0x7f8598156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:47.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f85977fe640 1 -- 172.21.15.107:0/2192179627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f859815c340 msgr2=0x7f8598157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:47.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f85977fe640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f859815c340 0x7f8598157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:47.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f85977fe640 1 -- 172.21.15.107:0/2192179627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85981562b0 msgr2=0x7f8598156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:47.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f85977fe640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85981562b0 0x7f8598156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:47.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f85977fe640 1 -- 172.21.15.107:0/2192179627 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85880094d0 con 0x7f85980c50c0 2023-10-25T00:13:47.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f85977fe640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980c50c0 0x7f8598152460 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f85880ac000 tx=0x7f858801ac80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:47.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f857ffff640 1 -- 172.21.15.107:0/2192179627 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85880abe40 con 0x7f85980c50c0 2023-10-25T00:13:47.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f857ffff640 1 -- 172.21.15.107:0/2192179627 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85880be3f0 con 0x7f85980c50c0 2023-10-25T00:13:47.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f859dfb3640 1 -- 172.21.15.107:0/2192179627 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859815b790 con 0x7f85980c50c0 2023-10-25T00:13:47.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f857ffff640 1 -- 172.21.15.107:0/2192179627 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85880be7c0 con 0x7f85980c50c0 2023-10-25T00:13:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f859dfb3640 1 -- 172.21.15.107:0/2192179627 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8598155e50 con 0x7f85980c50c0 2023-10-25T00:13:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.174+0000 7f859dfb3640 1 -- 172.21.15.107:0/2192179627 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8578005260 con 0x7f85980c50c0 2023-10-25T00:13:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.178+0000 7f857ffff640 1 -- 172.21.15.107:0/2192179627 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f85880be920 con 0x7f85980c50c0 2023-10-25T00:13:47.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.178+0000 7f857ffff640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8584065e60 0x7f8584068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.178+0000 7f8596ffd640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8584065e60 0x7f8584068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.178+0000 7f8596ffd640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8584065e60 0x7f8584068320 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f85981646b0 tx=0x7f858c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.178+0000 7f857ffff640 1 -- 172.21.15.107:0/2192179627 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 9799+0+0 (secure 0 0 0) 0x7f858811ef10 con 0x7f85980c50c0 2023-10-25T00:13:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.182+0000 7f857ffff640 1 -- 172.21.15.107:0/2192179627 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f85880eb870 con 0x7f85980c50c0 2023-10-25T00:13:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.670+0000 7f859dfb3640 1 -- 172.21.15.107:0/2192179627 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8578005a90 con 0x7f85980c50c0 2023-10-25T00:13:47.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.670+0000 7f857ffff640 1 -- 172.21.15.107:0/2192179627 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5778 (secure 0 0 0) 0x7f85880eee90 con 0x7f85980c50c0 2023-10-25T00:13:47.672 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:47.672 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":6976,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:12:36.920081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8016},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8016":{"gid":8016,"name":"c","rank":0,"incarnation":166,"state":"up:active","state_seq":19,"addr":"172.21.15.107:6835/2856791739","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2856791739},{"type":"v1","addr":"172.21.15.107:6835","nonce":2856791739}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:47.672 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 175 2023-10-25T00:13:47.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 -- 172.21.15.107:0/2192179627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8584065e60 msgr2=0x7f8584068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:47.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8584065e60 0x7f8584068320 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f85981646b0 tx=0x7f858c008040 comp rx=0 tx=0).stop 2023-10-25T00:13:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 -- 172.21.15.107:0/2192179627 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980c50c0 msgr2=0x7f8598152460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980c50c0 0x7f8598152460 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f85880ac000 tx=0x7f858801ac80 comp rx=0 tx=0).stop 2023-10-25T00:13:47.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 -- 172.21.15.107:0/2192179627 shutdown_connections 2023-10-25T00:13:47.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f859815c340 0x7f8598157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:47.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85981562b0 0x7f8598156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:47.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8584065e60 0x7f8584068320 secure :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f85981646b0 tx=0x7f858c008040 comp rx=0 tx=0).stop 2023-10-25T00:13:47.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 --2- 172.21.15.107:0/2192179627 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85980c50c0 0x7f8598152460 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:47.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 -- 172.21.15.107:0/2192179627 >> 172.21.15.107:0/2192179627 conn(0x7f85980ca200 msgr2=0x7f85980af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 -- 172.21.15.107:0/2192179627 shutdown_connections 2023-10-25T00:13:47.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:47.674+0000 7f857dffb640 1 -- 172.21.15.107:0/2192179627 wait complete. 2023-10-25T00:13:47.701 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:13:49.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:49.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85bb06a640 1 Processor -- start 2023-10-25T00:13:49.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85bb06a640 1 -- start start 2023-10-25T00:13:49.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85bb06a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f85b40720c0 0x7f85b40c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:49.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85bb06a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85b4157e50 0x7f85b41582a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:49.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85bb06a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b4154860 con 0x7f85b413dc20 2023-10-25T00:13:49.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85bb06a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b41587e0 con 0x7f85b40720c0 2023-10-25T00:13:49.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85bb06a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40c1f10 con 0x7f85b4157e50 2023-10-25T00:13:49.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85b8e06640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f85b413dc20 0x7f85b406df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:49960/0 (socket says 172.21.15.107:49960) 2023-10-25T00:13:49.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85b8e06640 1 -- 172.21.15.107:0/708718690 learned_addr learned my addr 172.21.15.107:0/708718690 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:13:49.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85b3fff640 1 --2- 172.21.15.107:0/708718690 >> v2:172.21.15.107:3300/0 conn(0x7f85b40720c0 0x7f85b40c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:49.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85b9607640 1 --2- 172.21.15.107:0/708718690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85b4157e50 0x7f85b41582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:49.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85b9607640 1 -- 172.21.15.107:0/708718690 >> v2:172.21.15.107:3300/0 conn(0x7f85b40720c0 msgr2=0x7f85b40c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:49.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85b9607640 1 --2- 172.21.15.107:0/708718690 >> v2:172.21.15.107:3300/0 conn(0x7f85b40720c0 0x7f85b40c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:49.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85b9607640 1 -- 172.21.15.107:0/708718690 >> v1:172.21.15.107:6789/0 conn(0x7f85b413dc20 legacy=0x7f85b406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:49.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.898+0000 7f85b9607640 1 -- 172.21.15.107:0/708718690 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b4154b20 con 0x7f85b4157e50 2023-10-25T00:13:49.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.902+0000 7f85b9607640 1 --2- 172.21.15.107:0/708718690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85b4157e50 0x7f85b41582a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f85ac009d90 tx=0x7f85ac01c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=692de253e2161c6a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:49.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.902+0000 7f85b37fe640 1 -- 172.21.15.107:0/708718690 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0a57e0 con 0x7f85b4157e50 2023-10-25T00:13:49.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.902+0000 7f85b37fe640 1 -- 172.21.15.107:0/708718690 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac0a5940 con 0x7f85b4157e50 2023-10-25T00:13:49.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.902+0000 7f85b37fe640 1 -- 172.21.15.107:0/708718690 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0a5cd0 con 0x7f85b4157e50 2023-10-25T00:13:49.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 -- 172.21.15.107:0/708718690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85b4157e50 msgr2=0x7f85b41582a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:49.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 --2- 172.21.15.107:0/708718690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85b4157e50 0x7f85b41582a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f85ac009d90 tx=0x7f85ac01c850 comp rx=0 tx=0).stop 2023-10-25T00:13:49.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 -- 172.21.15.107:0/708718690 shutdown_connections 2023-10-25T00:13:49.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 --2- 172.21.15.107:0/708718690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85b4157e50 0x7f85b41582a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:49.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 --2- 172.21.15.107:0/708718690 >> v2:172.21.15.107:3300/0 conn(0x7f85b40720c0 0x7f85b40c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:49.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 -- 172.21.15.107:0/708718690 >> 172.21.15.107:0/708718690 conn(0x7f85b40c39a0 msgr2=0x7f85b40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:49.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 -- 172.21.15.107:0/708718690 shutdown_connections 2023-10-25T00:13:49.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 -- 172.21.15.107:0/708718690 wait complete. 2023-10-25T00:13:49.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 Processor -- start 2023-10-25T00:13:49.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 -- start start 2023-10-25T00:13:49.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85b40720c0 0x7f85b40c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:49.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.906+0000 7f85bb06a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85b413dc20 0x7f85b40a9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:49.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85bb06a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85b4157e50 0x7f85b40aa100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:49.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b9607640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85b4157e50 0x7f85b40aa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:49.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b9607640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85b4157e50 0x7f85b40aa100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53948/0 (socket says 172.21.15.107:53948) 2023-10-25T00:13:49.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b9607640 1 -- 172.21.15.107:0/2749257295 learned_addr learned my addr 172.21.15.107:0/2749257295 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:49.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40a97a0 con 0x7f85b4157e50 2023-10-25T00:13:49.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40a9450 con 0x7f85b413dc20 2023-10-25T00:13:49.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b40c6b10 con 0x7f85b40720c0 2023-10-25T00:13:49.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b8e06640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85b40720c0 0x7f85b40c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:49.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b9607640 1 -- 172.21.15.107:0/2749257295 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85b40720c0 msgr2=0x7f85b40c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:49.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b9607640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85b40720c0 0x7f85b40c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:49.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b9607640 1 -- 172.21.15.107:0/2749257295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85b413dc20 msgr2=0x7f85b40a9bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-25T00:13:49.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b9607640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85b413dc20 0x7f85b40a9bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:49.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b9607640 1 -- 172.21.15.107:0/2749257295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac009a70 con 0x7f85b4157e50 2023-10-25T00:13:49.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b9607640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85b4157e50 0x7f85b40aa100 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f85ac01af00 tx=0x7f85ac0195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:49.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b17fa640 1 -- 172.21.15.107:0/2749257295 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0b9650 con 0x7f85b4157e50 2023-10-25T00:13:49.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b17fa640 1 -- 172.21.15.107:0/2749257295 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ac0b97b0 con 0x7f85b4157e50 2023-10-25T00:13:49.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85b17fa640 1 -- 172.21.15.107:0/2749257295 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85ac0b9b20 con 0x7f85b4157e50 2023-10-25T00:13:49.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b40c6830 con 0x7f85b4157e50 2023-10-25T00:13:49.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.910+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b41f9d80 con 0x7f85b4157e50 2023-10-25T00:13:49.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.914+0000 7f85b17fa640 1 -- 172.21.15.107:0/2749257295 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f85ac0b9c80 con 0x7f85b4157e50 2023-10-25T00:13:49.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.914+0000 7f85b17fa640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8598065e60 0x7f8598068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:49.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.914+0000 7f85b17fa640 1 -- 172.21.15.107:0/2749257295 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7f85ac11fb10 con 0x7f85b4157e50 2023-10-25T00:13:49.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.914+0000 7f85b8e06640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8598065e60 0x7f8598068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:49.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.914+0000 7f85b8e06640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8598065e60 0x7f8598068320 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f85a0005890 tx=0x7f85a0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:49.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.918+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8594005260 con 0x7f85b4157e50 2023-10-25T00:13:49.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:49.922+0000 7f85b17fa640 1 -- 172.21.15.107:0/2749257295 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f85ac0ec1d0 con 0x7f85b4157e50 2023-10-25T00:13:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.370+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8594005a90 con 0x7f85b4157e50 2023-10-25T00:13:50.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.370+0000 7f85b17fa640 1 -- 172.21.15.107:0/2749257295 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5019 (secure 0 0 0) 0x7f85ac0b2020 con 0x7f85b4157e50 2023-10-25T00:13:50.373 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:50.373 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:49.905332+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:50.373 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 177 2023-10-25T00:13:50.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.374+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8598065e60 msgr2=0x7f8598068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.374+0000 7f85bb06a640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8598065e60 0x7f8598068320 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f85a0005890 tx=0x7f85a0009040 comp rx=0 tx=0).stop 2023-10-25T00:13:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.374+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85b4157e50 msgr2=0x7f85b40aa100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:50.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.374+0000 7f85bb06a640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85b4157e50 0x7f85b40aa100 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f85ac01af00 tx=0x7f85ac0195f0 comp rx=0 tx=0).stop 2023-10-25T00:13:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.378+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 shutdown_connections 2023-10-25T00:13:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.378+0000 7f85bb06a640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85b4157e50 0x7f85b40aa100 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.378+0000 7f85bb06a640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85b413dc20 0x7f85b40a9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.378+0000 7f85bb06a640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8598065e60 0x7f8598068320 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:50.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.378+0000 7f85bb06a640 1 --2- 172.21.15.107:0/2749257295 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85b40720c0 0x7f85b40c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:50.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.378+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 >> 172.21.15.107:0/2749257295 conn(0x7f85b40c39a0 msgr2=0x7f85b406c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:50.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.378+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 shutdown_connections 2023-10-25T00:13:50.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.378+0000 7f85bb06a640 1 -- 172.21.15.107:0/2749257295 wait complete. 2023-10-25T00:13:50.404 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2023-10-25T00:13:50.405 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:13:50.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:50.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e17869640 1 Processor -- start 2023-10-25T00:13:50.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e17869640 1 -- start start 2023-10-25T00:13:50.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e17869640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e100aa330 0x7f3e1006e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:50.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e17869640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e10152d40 con 0x7f3e100a9450 2023-10-25T00:13:50.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e14e04640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e100aa330 0x7f3e1006e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:50.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e17869640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e1006d660 con 0x7f3e1013bb00 2023-10-25T00:13:50.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e17869640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e10156100 con 0x7f3e100aa330 2023-10-25T00:13:50.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e14e04640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e100aa330 0x7f3e1006e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53952/0 (socket says 172.21.15.107:53952) 2023-10-25T00:13:50.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e14e04640 1 -- 172.21.15.107:0/851580966 learned_addr learned my addr 172.21.15.107:0/851580966 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:50.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e14e04640 1 -- 172.21.15.107:0/851580966 >> v1:172.21.15.110:6789/0 conn(0x7f3e1013bb00 legacy=0x7f3e101531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:50.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e14e04640 1 -- 172.21.15.107:0/851580966 >> v1:172.21.15.107:6789/0 conn(0x7f3e100a9450 legacy=0x7f3e10071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:13:50.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.598+0000 7f3e14e04640 1 -- 172.21.15.107:0/851580966 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e10155e60 con 0x7f3e100aa330 2023-10-25T00:13:50.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.602+0000 7f3e14e04640 1 --2- 172.21.15.107:0/851580966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e100aa330 0x7f3e1006e250 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0098a0 tx=0x7f3e0c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a49cf0d6649dd5e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:50.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.602+0000 7f3e07fff640 1 -- 172.21.15.107:0/851580966 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c0ab600 con 0x7f3e100aa330 2023-10-25T00:13:50.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.602+0000 7f3e07fff640 1 -- 172.21.15.107:0/851580966 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0c0a7050 con 0x7f3e100aa330 2023-10-25T00:13:50.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.602+0000 7f3e07fff640 1 -- 172.21.15.107:0/851580966 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c0ab980 con 0x7f3e100aa330 2023-10-25T00:13:50.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 -- 172.21.15.107:0/851580966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e100aa330 msgr2=0x7f3e1006e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:50.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 --2- 172.21.15.107:0/851580966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e100aa330 0x7f3e1006e250 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0098a0 tx=0x7f3e0c01c380 comp rx=0 tx=0).stop 2023-10-25T00:13:50.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 -- 172.21.15.107:0/851580966 shutdown_connections 2023-10-25T00:13:50.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 --2- 172.21.15.107:0/851580966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e100aa330 0x7f3e1006e250 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:50.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 -- 172.21.15.107:0/851580966 >> 172.21.15.107:0/851580966 conn(0x7f3e10070f60 msgr2=0x7f3e10071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:50.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 -- 172.21.15.107:0/851580966 shutdown_connections 2023-10-25T00:13:50.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 -- 172.21.15.107:0/851580966 wait complete. 2023-10-25T00:13:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 Processor -- start 2023-10-25T00:13:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 -- start start 2023-10-25T00:13:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e100a9450 0x7f3e1006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e100aa330 0x7f3e10138810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:50.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1013bb00 0x7f3e10138d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e100adb50 con 0x7f3e1013bb00 2023-10-25T00:13:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e10139290 con 0x7f3e100aa330 2023-10-25T00:13:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e17869640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e101f1280 con 0x7f3e100a9450 2023-10-25T00:13:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e14e04640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e100aa330 0x7f3e10138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:50.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e14e04640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e100aa330 0x7f3e10138810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45568/0 (socket says 172.21.15.107:45568) 2023-10-25T00:13:50.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e14e04640 1 -- 172.21.15.107:0/2584583721 learned_addr learned my addr 172.21.15.107:0/2584583721 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:50.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e15605640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e100a9450 0x7f3e1006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:50.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.606+0000 7f3e15e06640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1013bb00 0x7f3e10138d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e14e04640 1 -- 172.21.15.107:0/2584583721 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e100a9450 msgr2=0x7f3e1006dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e14e04640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e100a9450 0x7f3e1006dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e14e04640 1 -- 172.21.15.107:0/2584583721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1013bb00 msgr2=0x7f3e10138d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e14e04640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1013bb00 0x7f3e10138d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e14e04640 1 -- 172.21.15.107:0/2584583721 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0c0094d0 con 0x7f3e100aa330 2023-10-25T00:13:50.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e14e04640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e100aa330 0x7f3e10138810 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c019fb0 tx=0x7f3e0c019120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:50.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e05ffb640 1 -- 172.21.15.107:0/2584583721 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c0b5810 con 0x7f3e100aa330 2023-10-25T00:13:50.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e17869640 1 -- 172.21.15.107:0/2584583721 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e101f1560 con 0x7f3e100aa330 2023-10-25T00:13:50.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e17869640 1 -- 172.21.15.107:0/2584583721 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e101f1b20 con 0x7f3e100aa330 2023-10-25T00:13:50.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e05ffb640 1 -- 172.21.15.107:0/2584583721 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0c0b5970 con 0x7f3e100aa330 2023-10-25T00:13:50.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e05ffb640 1 -- 172.21.15.107:0/2584583721 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c0b5d20 con 0x7f3e100aa330 2023-10-25T00:13:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e17869640 1 -- 172.21.15.107:0/2584583721 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e100746e0 con 0x7f3e100aa330 2023-10-25T00:13:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.610+0000 7f3e05ffb640 1 -- 172.21.15.107:0/2584583721 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3e0c0b43a0 con 0x7f3e100aa330 2023-10-25T00:13:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.614+0000 7f3e05ffb640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3df4065e60 0x7f3df4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.614+0000 7f3e05ffb640 1 -- 172.21.15.107:0/2584583721 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7f3e0c11f060 con 0x7f3e100aa330 2023-10-25T00:13:50.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.614+0000 7f3e15605640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3df4065e60 0x7f3df4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:50.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.614+0000 7f3e15605640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3df4065e60 0x7f3df4068320 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009fd0 tx=0x7f3e0800d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:50.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:50.622+0000 7f3e05ffb640 1 -- 172.21.15.107:0/2584583721 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3e0c0eb850 con 0x7f3e100aa330 2023-10-25T00:13:51.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.022+0000 7f3e17869640 1 -- 172.21.15.107:0/2584583721 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e1006ae10 con 0x7f3e100aa330 2023-10-25T00:13:51.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.022+0000 7f3e05ffb640 1 -- 172.21.15.107:0/2584583721 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f3e0c0b9090 con 0x7f3e100aa330 2023-10-25T00:13:51.024 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:51.025 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:50.907735+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:51.025 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 178 2023-10-25T00:13:51.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 -- 172.21.15.107:0/2584583721 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3df4065e60 msgr2=0x7f3df4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:51.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3df4065e60 0x7f3df4068320 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009fd0 tx=0x7f3e0800d040 comp rx=0 tx=0).stop 2023-10-25T00:13:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 -- 172.21.15.107:0/2584583721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e100aa330 msgr2=0x7f3e10138810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e100aa330 0x7f3e10138810 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c019fb0 tx=0x7f3e0c019120 comp rx=0 tx=0).stop 2023-10-25T00:13:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 -- 172.21.15.107:0/2584583721 shutdown_connections 2023-10-25T00:13:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e1013bb00 0x7f3e10138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:51.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e100aa330 0x7f3e10138810 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:51.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3df4065e60 0x7f3df4068320 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:51.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 --2- 172.21.15.107:0/2584583721 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e100a9450 0x7f3e1006dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:51.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 -- 172.21.15.107:0/2584583721 >> 172.21.15.107:0/2584583721 conn(0x7f3e10070f60 msgr2=0x7f3e100c35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:51.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 -- 172.21.15.107:0/2584583721 shutdown_connections 2023-10-25T00:13:51.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:51.034+0000 7f3e17869640 1 -- 172.21.15.107:0/2584583721 wait complete. 2023-10-25T00:13:51.064 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 178, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 174, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8106": { "addr": "172.21.15.107:6837/175581297", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 175581297, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 175581297, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8106, "incarnation": 171, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:13:21.937931+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8106 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 178, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6976": { "addr": "172.21.15.110:6835/1619327197", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 1619327197, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 1619327197, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6976, "incarnation": 176, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:13:50.907735+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6976 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:13:52.706 INFO:tasks.ceph.mds.d.smithi110.stderr:2023-10-25T00:13:52.706+0000 7fb26920d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:13:53.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:53.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.350+0000 7f1c736cb640 1 Processor -- start 2023-10-25T00:13:53.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.350+0000 7f1c736cb640 1 -- start start 2023-10-25T00:13:53.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.350+0000 7f1c736cb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c6c14c470 0x7f1c6c14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:53.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.350+0000 7f1c736cb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6c155cd0 con 0x7f1c6c152b30 2023-10-25T00:13:53.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.350+0000 7f1c736cb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6c0c4380 con 0x7f1c6c06f490 2023-10-25T00:13:53.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.350+0000 7f1c736cb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6c1526e0 con 0x7f1c6c14c470 2023-10-25T00:13:53.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.354+0000 7f1c71467640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c6c14c470 0x7f1c6c14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:53.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.354+0000 7f1c71c68640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1c6c152b30 0x7f1c6c152f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57462/0 (socket says 172.21.15.107:57462) 2023-10-25T00:13:53.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.354+0000 7f1c71c68640 1 -- 172.21.15.107:0/2290114796 learned_addr learned my addr 172.21.15.107:0/2290114796 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:13:53.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.354+0000 7f1c63fff640 1 -- 172.21.15.107:0/2290114796 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950553576 0 0) 0x7f1c6c155cd0 con 0x7f1c6c152b30 2023-10-25T00:13:53.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.354+0000 7f1c63fff640 1 -- 172.21.15.107:0/2290114796 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c58003530 con 0x7f1c6c152b30 2023-10-25T00:13:53.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.354+0000 7f1c63fff640 1 -- 172.21.15.107:0/2290114796 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500123525 0 0) 0x7f1c6c0c4380 con 0x7f1c6c06f490 2023-10-25T00:13:53.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.354+0000 7f1c71467640 1 -- 172.21.15.107:0/2290114796 >> v1:172.21.15.110:6789/0 conn(0x7f1c6c06f490 legacy=0x7f1c6c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:53.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.354+0000 7f1c71467640 1 -- 172.21.15.107:0/2290114796 >> v1:172.21.15.107:6789/0 conn(0x7f1c6c152b30 legacy=0x7f1c6c152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:53.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.354+0000 7f1c71467640 1 -- 172.21.15.107:0/2290114796 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c0c48b0 con 0x7f1c6c14c470 2023-10-25T00:13:53.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.362+0000 7f1c71467640 1 --2- 172.21.15.107:0/2290114796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c14c470 0x7f1c6c14c850 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f1c680097c0 tx=0x7f1c6801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b10c8e049a60458 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:53.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.362+0000 7f1c63fff640 1 -- 172.21.15.107:0/2290114796 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c680ab600 con 0x7f1c6c14c470 2023-10-25T00:13:53.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.362+0000 7f1c63fff640 1 -- 172.21.15.107:0/2290114796 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c680a7050 con 0x7f1c6c14c470 2023-10-25T00:13:53.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.366+0000 7f1c63fff640 1 -- 172.21.15.107:0/2290114796 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c680ab980 con 0x7f1c6c14c470 2023-10-25T00:13:53.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.366+0000 7f1c736cb640 1 -- 172.21.15.107:0/2290114796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c14c470 msgr2=0x7f1c6c14c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:53.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.366+0000 7f1c736cb640 1 --2- 172.21.15.107:0/2290114796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c14c470 0x7f1c6c14c850 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f1c680097c0 tx=0x7f1c6801c360 comp rx=0 tx=0).stop 2023-10-25T00:13:53.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.366+0000 7f1c736cb640 1 -- 172.21.15.107:0/2290114796 shutdown_connections 2023-10-25T00:13:53.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.366+0000 7f1c736cb640 1 --2- 172.21.15.107:0/2290114796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c14c470 0x7f1c6c14c850 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:53.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.366+0000 7f1c736cb640 1 -- 172.21.15.107:0/2290114796 >> 172.21.15.107:0/2290114796 conn(0x7f1c6c070f60 msgr2=0x7f1c6c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:53.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.366+0000 7f1c736cb640 1 -- 172.21.15.107:0/2290114796 shutdown_connections 2023-10-25T00:13:53.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.366+0000 7f1c736cb640 1 -- 172.21.15.107:0/2290114796 wait complete. 2023-10-25T00:13:53.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.366+0000 7f1c736cb640 1 Processor -- start 2023-10-25T00:13:53.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.366+0000 7f1c736cb640 1 -- start start 2023-10-25T00:13:53.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c736cb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c06f490 0x7f1c6c143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:53.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c736cb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14c470 0x7f1c6c14ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:53.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c70c66640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14c470 0x7f1c6c14ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:53.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c71467640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c06f490 0x7f1c6c143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:53.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c71467640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c06f490 0x7f1c6c143db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53912/0 (socket says 172.21.15.107:53912) 2023-10-25T00:13:53.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c70c66640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14c470 0x7f1c6c14ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:35176/0 (socket says 172.21.15.107:35176) 2023-10-25T00:13:53.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c736cb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c152b30 0x7f1c6c14cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:53.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c71467640 1 -- 172.21.15.107:0/4259607370 learned_addr learned my addr 172.21.15.107:0/4259607370 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:53.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c736cb640 1 -- 172.21.15.107:0/4259607370 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c14f170 con 0x7f1c6c06f490 2023-10-25T00:13:53.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c736cb640 1 -- 172.21.15.107:0/4259607370 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c14f410 con 0x7f1c6c152b30 2023-10-25T00:13:53.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c71c68640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c152b30 0x7f1c6c14cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:53.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.370+0000 7f1c736cb640 1 -- 172.21.15.107:0/4259607370 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c1432d0 con 0x7f1c6c14c470 2023-10-25T00:13:53.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c70c66640 1 -- 172.21.15.107:0/4259607370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c152b30 msgr2=0x7f1c6c14cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:53.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c70c66640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c152b30 0x7f1c6c14cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:53.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c70c66640 1 -- 172.21.15.107:0/4259607370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c06f490 msgr2=0x7f1c6c143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:53.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c70c66640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c06f490 0x7f1c6c143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:53.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c70c66640 1 -- 172.21.15.107:0/4259607370 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c68009420 con 0x7f1c6c14c470 2023-10-25T00:13:53.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c70c66640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14c470 0x7f1c6c14ec30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0011f0 tx=0x7f1c5c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:53.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c61ffb640 1 -- 172.21.15.107:0/4259607370 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0054e0 con 0x7f1c6c14c470 2023-10-25T00:13:53.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c61ffb640 1 -- 172.21.15.107:0/4259607370 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5c003600 con 0x7f1c6c14c470 2023-10-25T00:13:53.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c61ffb640 1 -- 172.21.15.107:0/4259607370 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c003a30 con 0x7f1c6c14c470 2023-10-25T00:13:53.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c736cb640 1 -- 172.21.15.107:0/4259607370 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6c1435b0 con 0x7f1c6c14c470 2023-10-25T00:13:53.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c736cb640 1 -- 172.21.15.107:0/4259607370 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6c14d520 con 0x7f1c6c14c470 2023-10-25T00:13:53.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.374+0000 7f1c577fe640 1 -- 172.21.15.107:0/4259607370 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c44005260 con 0x7f1c6c14c470 2023-10-25T00:13:53.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.378+0000 7f1c61ffb640 1 -- 172.21.15.107:0/4259607370 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1c5c003b90 con 0x7f1c6c14c470 2023-10-25T00:13:53.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.378+0000 7f1c61ffb640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c5006a2c0 0x7f1c5006c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:53.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.378+0000 7f1c61ffb640 1 -- 172.21.15.107:0/4259607370 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7f1c5c083ce0 con 0x7f1c6c14c470 2023-10-25T00:13:53.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.378+0000 7f1c71467640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c5006a2c0 0x7f1c5006c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:53.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.378+0000 7f1c71467640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c5006a2c0 0x7f1c5006c780 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1c6801c3a0 tx=0x7f1c680a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:53.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.382+0000 7f1c61ffb640 1 -- 172.21.15.107:0/4259607370 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1c5c050520 con 0x7f1c6c14c470 2023-10-25T00:13:53.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.778+0000 7f1c577fe640 1 -- 172.21.15.107:0/4259607370 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c44005510 con 0x7f1c6c14c470 2023-10-25T00:13:53.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.782+0000 7f1c61ffb640 1 -- 172.21.15.107:0/4259607370 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f1c5c017020 con 0x7f1c6c14c470 2023-10-25T00:13:53.782 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:53.782 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:53.783 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 179 2023-10-25T00:13:53.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.786+0000 7f1c577fe640 1 -- 172.21.15.107:0/4259607370 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c5006a2c0 msgr2=0x7f1c5006c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:53.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.786+0000 7f1c577fe640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c5006a2c0 0x7f1c5006c780 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1c6801c3a0 tx=0x7f1c680a6000 comp rx=0 tx=0).stop 2023-10-25T00:13:53.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.786+0000 7f1c577fe640 1 -- 172.21.15.107:0/4259607370 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14c470 msgr2=0x7f1c6c14ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:53.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.786+0000 7f1c577fe640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14c470 0x7f1c6c14ec30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c0011f0 tx=0x7f1c5c00c8c0 comp rx=0 tx=0).stop 2023-10-25T00:13:53.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.790+0000 7f1c577fe640 1 -- 172.21.15.107:0/4259607370 shutdown_connections 2023-10-25T00:13:53.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.790+0000 7f1c577fe640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c6c152b30 0x7f1c6c14cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:53.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.790+0000 7f1c577fe640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c5006a2c0 0x7f1c5006c780 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:53.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.790+0000 7f1c577fe640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c6c14c470 0x7f1c6c14ec30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:53.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.790+0000 7f1c577fe640 1 --2- 172.21.15.107:0/4259607370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c06f490 0x7f1c6c143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:53.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.790+0000 7f1c577fe640 1 -- 172.21.15.107:0/4259607370 >> 172.21.15.107:0/4259607370 conn(0x7f1c6c070f60 msgr2=0x7f1c6c0c6720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:53.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.790+0000 7f1c577fe640 1 -- 172.21.15.107:0/4259607370 shutdown_connections 2023-10-25T00:13:53.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:53.790+0000 7f1c577fe640 1 -- 172.21.15.107:0/4259607370 wait complete. 2023-10-25T00:13:53.822 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=8016 2023-10-25T00:13:53.822 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.c 2023-10-25T00:13:56.823 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2023-10-25T00:13:56.823 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:13:56.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:13:56.826 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:13:56.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:13:56.906 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:13:57.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1369ecb640 1 Processor -- start 2023-10-25T00:13:57.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1369ecb640 1 -- start start 2023-10-25T00:13:57.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1369ecb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f13640c33e0 0x7f1364069d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:57.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1369ecb640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13640c22f0 con 0x7f136414eb70 2023-10-25T00:13:57.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1369ecb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13640c25d0 con 0x7f1364070b00 2023-10-25T00:13:57.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1369ecb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1364071700 con 0x7f13640c33e0 2023-10-25T00:13:57.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1363fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f13640c33e0 0x7f1364069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1363fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f13640c33e0 0x7f1364069d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53946/0 (socket says 172.21.15.107:53946) 2023-10-25T00:13:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1363fff640 1 -- 172.21.15.107:0/3702955084 learned_addr learned my addr 172.21.15.107:0/3702955084 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1363fff640 1 -- 172.21.15.107:0/3702955084 >> v1:172.21.15.110:6789/0 conn(0x7f1364070b00 legacy=0x7f136406df30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-25T00:13:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1363fff640 1 -- 172.21.15.107:0/3702955084 >> v1:172.21.15.107:6789/0 conn(0x7f136414eb70 legacy=0x7f13641514e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:13:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.066+0000 7f1363fff640 1 -- 172.21.15.107:0/3702955084 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13640c2be0 con 0x7f13640c33e0 2023-10-25T00:13:57.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.074+0000 7f1363fff640 1 --2- 172.21.15.107:0/3702955084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13640c33e0 0x7f1364069d40 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f1358009870 tx=0x7f135801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b7bb0739d3fb31 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:57.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.074+0000 7f13627fc640 1 -- 172.21.15.107:0/3702955084 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13580ab610 con 0x7f13640c33e0 2023-10-25T00:13:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.074+0000 7f13627fc640 1 -- 172.21.15.107:0/3702955084 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13580ab770 con 0x7f13640c33e0 2023-10-25T00:13:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.074+0000 7f13627fc640 1 -- 172.21.15.107:0/3702955084 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13580ab610 con 0x7f13640c33e0 2023-10-25T00:13:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.074+0000 7f1369ecb640 1 -- 172.21.15.107:0/3702955084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13640c33e0 msgr2=0x7f1364069d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:57.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.074+0000 7f1369ecb640 1 --2- 172.21.15.107:0/3702955084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13640c33e0 0x7f1364069d40 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f1358009870 tx=0x7f135801c500 comp rx=0 tx=0).stop 2023-10-25T00:13:57.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.078+0000 7f1369ecb640 1 -- 172.21.15.107:0/3702955084 shutdown_connections 2023-10-25T00:13:57.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.078+0000 7f1369ecb640 1 --2- 172.21.15.107:0/3702955084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f13640c33e0 0x7f1364069d40 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:57.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.078+0000 7f1369ecb640 1 -- 172.21.15.107:0/3702955084 >> 172.21.15.107:0/3702955084 conn(0x7f13640c39a0 msgr2=0x7f13640c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:57.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.086+0000 7f1369ecb640 1 -- 172.21.15.107:0/3702955084 shutdown_connections 2023-10-25T00:13:57.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.086+0000 7f1369ecb640 1 -- 172.21.15.107:0/3702955084 wait complete. 2023-10-25T00:13:57.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1369ecb640 1 Processor -- start 2023-10-25T00:13:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1369ecb640 1 -- start start 2023-10-25T00:13:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1369ecb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1364070b00 0x7f13640ab620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1369ecb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13640c33e0 0x7f13640abb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:57.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1369ecb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f136414eb70 0x7f13641376b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:57.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1369ecb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136414f500 con 0x7f136414eb70 2023-10-25T00:13:57.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1369ecb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13641f24c0 con 0x7f1364070b00 2023-10-25T00:13:57.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1369ecb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13640a9640 con 0x7f13640c33e0 2023-10-25T00:13:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1363fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f136414eb70 0x7f13641376b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1363fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f136414eb70 0x7f13641376b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53962/0 (socket says 172.21.15.107:53962) 2023-10-25T00:13:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1362ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13640c33e0 0x7f13640abb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1363fff640 1 -- 172.21.15.107:0/2164444023 learned_addr learned my addr 172.21.15.107:0/2164444023 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:13:57.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f13637fe640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1364070b00 0x7f13640ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1362ffd640 1 -- 172.21.15.107:0/2164444023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1364070b00 msgr2=0x7f13640ab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1362ffd640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1364070b00 0x7f13640ab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1362ffd640 1 -- 172.21.15.107:0/2164444023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f136414eb70 msgr2=0x7f13641376b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1362ffd640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f136414eb70 0x7f13641376b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:57.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1362ffd640 1 -- 172.21.15.107:0/2164444023 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13580094d0 con 0x7f13640c33e0 2023-10-25T00:13:57.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1363fff640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f136414eb70 0x7f13641376b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:13:57.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f13637fe640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1364070b00 0x7f13640ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:13:57.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.090+0000 7f1362ffd640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13640c33e0 0x7f13640abb60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f135c00bdf0 tx=0x7f135c009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.094+0000 7f134bfff640 1 -- 172.21.15.107:0/2164444023 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c014440 con 0x7f13640c33e0 2023-10-25T00:13:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.094+0000 7f134bfff640 1 -- 172.21.15.107:0/2164444023 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c0145a0 con 0x7f13640c33e0 2023-10-25T00:13:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.094+0000 7f134bfff640 1 -- 172.21.15.107:0/2164444023 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c014440 con 0x7f13640c33e0 2023-10-25T00:13:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.094+0000 7f1369ecb640 1 -- 172.21.15.107:0/2164444023 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13640a9920 con 0x7f13640c33e0 2023-10-25T00:13:57.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.094+0000 7f1369ecb640 1 -- 172.21.15.107:0/2164444023 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13640ac0a0 con 0x7f13640c33e0 2023-10-25T00:13:57.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.094+0000 7f134bfff640 1 -- 172.21.15.107:0/2164444023 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f135c01d3b0 con 0x7f13640c33e0 2023-10-25T00:13:57.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.094+0000 7f1369ecb640 1 -- 172.21.15.107:0/2164444023 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1344005260 con 0x7f13640c33e0 2023-10-25T00:13:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.094+0000 7f134bfff640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1350065f20 0x7f13500683e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:13:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.094+0000 7f134bfff640 1 -- 172.21.15.107:0/2164444023 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7f135c084500 con 0x7f13640c33e0 2023-10-25T00:13:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.094+0000 7f13637fe640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1350065f20 0x7f13500683e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:13:57.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.098+0000 7f13637fe640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1350065f20 0x7f13500683e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f13540016f0 tx=0x7f1354009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:13:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.102+0000 7f134bfff640 1 -- 172.21.15.107:0/2164444023 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f135c050c70 con 0x7f13640c33e0 2023-10-25T00:13:57.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.470+0000 7f1369ecb640 1 -- 172.21.15.107:0/2164444023 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1344005a90 con 0x7f13640c33e0 2023-10-25T00:13:57.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.470+0000 7f134bfff640 1 -- 172.21.15.107:0/2164444023 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f135c019020 con 0x7f13640c33e0 2023-10-25T00:13:57.472 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:13:57.473 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:13:57.473 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:13:57.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.474+0000 7f1369ecb640 1 -- 172.21.15.107:0/2164444023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1350065f20 msgr2=0x7f13500683e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:57.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.474+0000 7f1369ecb640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1350065f20 0x7f13500683e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f13540016f0 tx=0x7f1354009040 comp rx=0 tx=0).stop 2023-10-25T00:13:57.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.474+0000 7f1369ecb640 1 -- 172.21.15.107:0/2164444023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13640c33e0 msgr2=0x7f13640abb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:13:57.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.474+0000 7f1369ecb640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13640c33e0 0x7f13640abb60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f135c00bdf0 tx=0x7f135c009f70 comp rx=0 tx=0).stop 2023-10-25T00:13:57.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.474+0000 7f1369ecb640 1 -- 172.21.15.107:0/2164444023 shutdown_connections 2023-10-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.474+0000 7f1369ecb640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f136414eb70 0x7f13641376b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.474+0000 7f1369ecb640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1350065f20 0x7f13500683e0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.474+0000 7f1369ecb640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f13640c33e0 0x7f13640abb60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:57.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.474+0000 7f1369ecb640 1 --2- 172.21.15.107:0/2164444023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1364070b00 0x7f13640ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:13:57.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.474+0000 7f1369ecb640 1 -- 172.21.15.107:0/2164444023 >> 172.21.15.107:0/2164444023 conn(0x7f13640c39a0 msgr2=0x7f136406c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:13:57.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.478+0000 7f1369ecb640 1 -- 172.21.15.107:0/2164444023 shutdown_connections 2023-10-25T00:13:57.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:13:57.478+0000 7f1369ecb640 1 -- 172.21.15.107:0/2164444023 wait complete. 2023-10-25T00:13:57.515 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2023-10-25T00:13:57.515 INFO:tasks.mds_thrash.fs.[b]:waiting for 70 secs before thrashing 2023-10-25T00:15:02.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:03.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbfa8a640 1 Processor -- start 2023-10-25T00:15:03.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbfa8a640 1 -- start start 2023-10-25T00:15:03.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbfa8a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fccb80c24a0 0x7fccb8154840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:03.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbfa8a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb8154d80 con 0x7fccb80a9450 2023-10-25T00:15:03.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbfa8a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb80aac40 con 0x7fccb8151ff0 2023-10-25T00:15:03.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbfa8a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb806b630 con 0x7fccb80c24a0 2023-10-25T00:15:03.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbd826640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fccb8151ff0 0x7fccb8158760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:54728/0 (socket says 172.21.15.107:54728) 2023-10-25T00:15:03.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbd826640 1 -- 172.21.15.107:0/3162457355 learned_addr learned my addr 172.21.15.107:0/3162457355 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:03.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbd025640 1 --2- 172.21.15.107:0/3162457355 >> v2:172.21.15.107:3300/0 conn(0x7fccb80c24a0 0x7fccb8154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:03.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbc824640 1 -- 172.21.15.107:0/3162457355 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170691026 0 0) 0x7fccb80aac40 con 0x7fccb8151ff0 2023-10-25T00:15:03.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbc824640 1 -- 172.21.15.107:0/3162457355 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca4003530 con 0x7fccb8151ff0 2023-10-25T00:15:03.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbd025640 1 -- 172.21.15.107:0/3162457355 >> v1:172.21.15.110:6789/0 conn(0x7fccb8151ff0 legacy=0x7fccb8158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:03.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbd025640 1 -- 172.21.15.107:0/3162457355 >> v1:172.21.15.107:6789/0 conn(0x7fccb80a9450 legacy=0x7fccb80c1860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:15:03.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.264+0000 7fccbd025640 1 -- 172.21.15.107:0/3162457355 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb80ab450 con 0x7fccb80c24a0 2023-10-25T00:15:03.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.268+0000 7fccbd025640 1 --2- 172.21.15.107:0/3162457355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb80c24a0 0x7fccb8154840 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009d90 tx=0x7fccb001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd1743a2109610 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:03.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.268+0000 7fccbc824640 1 -- 172.21.15.107:0/3162457355 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb00a66c0 con 0x7fccb80c24a0 2023-10-25T00:15:03.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.268+0000 7fccbc824640 1 -- 172.21.15.107:0/3162457355 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb00a6820 con 0x7fccb80c24a0 2023-10-25T00:15:03.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.268+0000 7fccbc824640 1 -- 172.21.15.107:0/3162457355 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb00a6bb0 con 0x7fccb80c24a0 2023-10-25T00:15:03.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.272+0000 7fccbfa8a640 1 -- 172.21.15.107:0/3162457355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb80c24a0 msgr2=0x7fccb8154840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:03.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.272+0000 7fccbfa8a640 1 --2- 172.21.15.107:0/3162457355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb80c24a0 0x7fccb8154840 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009d90 tx=0x7fccb001c7f0 comp rx=0 tx=0).stop 2023-10-25T00:15:03.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.272+0000 7fccbfa8a640 1 -- 172.21.15.107:0/3162457355 shutdown_connections 2023-10-25T00:15:03.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.272+0000 7fccbfa8a640 1 --2- 172.21.15.107:0/3162457355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb80c24a0 0x7fccb8154840 secure :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009d90 tx=0x7fccb001c7f0 comp rx=0 tx=0).stop 2023-10-25T00:15:03.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.272+0000 7fccbfa8a640 1 -- 172.21.15.107:0/3162457355 >> 172.21.15.107:0/3162457355 conn(0x7fccb80c2950 msgr2=0x7fccb80c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:03.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.272+0000 7fccbfa8a640 1 -- 172.21.15.107:0/3162457355 shutdown_connections 2023-10-25T00:15:03.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.272+0000 7fccbfa8a640 1 -- 172.21.15.107:0/3162457355 wait complete. 2023-10-25T00:15:03.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.276+0000 7fccbfa8a640 1 Processor -- start 2023-10-25T00:15:03.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbfa8a640 1 -- start start 2023-10-25T00:15:03.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbfa8a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccb80a9450 0x7fccb8152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:03.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbfa8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb8151ff0 0x7fccb81f67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:03.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbd826640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccb80a9450 0x7fccb8152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:03.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbd025640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb8151ff0 0x7fccb81f67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:03.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbd025640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb8151ff0 0x7fccb81f67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35620/0 (socket says 172.21.15.107:35620) 2023-10-25T00:15:03.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbd025640 1 -- 172.21.15.107:0/684505571 learned_addr learned my addr 172.21.15.107:0/684505571 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:03.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbfa8a640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccb81f6d30 0x7fccb8157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:03.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb813ab60 con 0x7fccb8151ff0 2023-10-25T00:15:03.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb81fd820 con 0x7fccb81f6d30 2023-10-25T00:15:03.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb81fcca0 con 0x7fccb80a9450 2023-10-25T00:15:03.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.284+0000 7fccbe027640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccb81f6d30 0x7fccb8157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:03.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.288+0000 7fccbd025640 1 -- 172.21.15.107:0/684505571 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccb80a9450 msgr2=0x7fccb8152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:03.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.288+0000 7fccbd025640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccb80a9450 0x7fccb8152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:03.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.288+0000 7fccbd025640 1 -- 172.21.15.107:0/684505571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccb81f6d30 msgr2=0x7fccb8157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:03.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.288+0000 7fccbd025640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccb81f6d30 0x7fccb8157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:03.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.288+0000 7fccbd025640 1 -- 172.21.15.107:0/684505571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0009a70 con 0x7fccb8151ff0 2023-10-25T00:15:03.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.288+0000 7fccbd025640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb8151ff0 0x7fccb81f67f0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fccb00b0040 tx=0x7fccb0019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:03.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.288+0000 7fccae7fc640 1 -- 172.21.15.107:0/684505571 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb00b8630 con 0x7fccb8151ff0 2023-10-25T00:15:03.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.288+0000 7fccae7fc640 1 -- 172.21.15.107:0/684505571 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb00b8790 con 0x7fccb8151ff0 2023-10-25T00:15:03.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.288+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb81fcf80 con 0x7fccb8151ff0 2023-10-25T00:15:03.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.288+0000 7fccae7fc640 1 -- 172.21.15.107:0/684505571 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb00b8b00 con 0x7fccb8151ff0 2023-10-25T00:15:03.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.288+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb81fd540 con 0x7fccb8151ff0 2023-10-25T00:15:03.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.292+0000 7fccae7fc640 1 -- 172.21.15.107:0/684505571 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fccb00b8c60 con 0x7fccb8151ff0 2023-10-25T00:15:03.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.292+0000 7fccae7fc640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc9c065f30 0x7fcc9c0683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:03.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.292+0000 7fccae7fc640 1 -- 172.21.15.107:0/684505571 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7fccb00aa070 con 0x7fccb8151ff0 2023-10-25T00:15:03.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.292+0000 7fccbd826640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc9c065f30 0x7fcc9c0683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:03.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.296+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca0005260 con 0x7fccb8151ff0 2023-10-25T00:15:03.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.300+0000 7fccbd826640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc9c065f30 0x7fcc9c0683f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fccb40011f0 tx=0x7fccb4007d40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:03.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.308+0000 7fccae7fc640 1 -- 172.21.15.107:0/684505571 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fccb00be080 con 0x7fccb8151ff0 2023-10-25T00:15:03.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.812+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcca0005a90 con 0x7fccb8151ff0 2023-10-25T00:15:03.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.812+0000 7fccae7fc640 1 -- 172.21.15.107:0/684505571 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7fccb00c8020 con 0x7fccb8151ff0 2023-10-25T00:15:03.817 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:03.817 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:03.818 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc9c065f30 msgr2=0x7fcc9c0683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc9c065f30 0x7fcc9c0683f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fccb40011f0 tx=0x7fccb4007d40 comp rx=0 tx=0).stop 2023-10-25T00:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb8151ff0 msgr2=0x7fccb81f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb8151ff0 0x7fccb81f67f0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fccb00b0040 tx=0x7fccb0019590 comp rx=0 tx=0).stop 2023-10-25T00:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 shutdown_connections 2023-10-25T00:15:03.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fccb81f6d30 0x7fccb8157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:03.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb8151ff0 0x7fccb81f67f0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:03.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcc9c065f30 0x7fcc9c0683f0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:03.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 --2- 172.21.15.107:0/684505571 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fccb80a9450 0x7fccb8152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:03.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 >> 172.21.15.107:0/684505571 conn(0x7fccb80c2950 msgr2=0x7fccb80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:03.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 shutdown_connections 2023-10-25T00:15:03.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:03.820+0000 7fccbfa8a640 1 -- 172.21.15.107:0/684505571 wait complete. 2023-10-25T00:15:03.853 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2023-10-25T00:15:03.853 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:15:03.854 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:15:03.871 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:15:03.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:04.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f768084a640 1 Processor -- start 2023-10-25T00:15:04.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f768084a640 1 -- start start 2023-10-25T00:15:04.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f768084a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f767c0720c0 0x7f767c0c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:04.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f768084a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f767c157e50 0x7f767c1582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:04.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f768084a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767c154860 con 0x7f767c13dc20 2023-10-25T00:15:04.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f768084a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f767c158800 con 0x7f767c157e50 2023-10-25T00:15:04.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f768084a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c0c1f10 con 0x7f767c0720c0 2023-10-25T00:15:04.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f7671d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f767c0720c0 0x7f767c0c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:04.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f767ad9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f767c157e50 0x7f767c1582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:04.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f767a59c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f767c13dc20 0x7f767c06df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:54740/0 (socket says 172.21.15.107:54740) 2023-10-25T00:15:04.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f7671d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f767c0720c0 0x7f767c0c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:57844/0 (socket says 172.21.15.107:57844) 2023-10-25T00:15:04.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f767ad9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f767c157e50 0x7f767c1582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35370/0 (socket says 172.21.15.107:35370) 2023-10-25T00:15:04.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.160+0000 7f767a59c640 1 -- 172.21.15.107:0/1224214764 learned_addr learned my addr 172.21.15.107:0/1224214764 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:04.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.164+0000 7f7671d9b640 1 -- 172.21.15.107:0/1224214764 >> v2:172.21.15.110:3300/0 conn(0x7f767c157e50 msgr2=0x7f767c1582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:04.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.164+0000 7f7671d9b640 1 --2- 172.21.15.107:0/1224214764 >> v2:172.21.15.110:3300/0 conn(0x7f767c157e50 0x7f767c1582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:04.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.164+0000 7f7671d9b640 1 -- 172.21.15.107:0/1224214764 >> v1:172.21.15.110:6789/0 conn(0x7f767c13dc20 legacy=0x7f767c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:04.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.164+0000 7f7671d9b640 1 -- 172.21.15.107:0/1224214764 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c154b20 con 0x7f767c0720c0 2023-10-25T00:15:04.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.164+0000 7f767ad9d640 1 --2- 172.21.15.107:0/1224214764 >> v2:172.21.15.110:3300/0 conn(0x7f767c157e50 0x7f767c1582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:15:04.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.164+0000 7f7671d9b640 1 --2- 172.21.15.107:0/1224214764 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f767c0720c0 0x7f767c0c18a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f76640097c0 tx=0x7f766401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0cb6e0a74209532 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:04.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.164+0000 7f7679d9b640 1 -- 172.21.15.107:0/1224214764 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76640ab600 con 0x7f767c0720c0 2023-10-25T00:15:04.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.164+0000 7f7679d9b640 1 -- 172.21.15.107:0/1224214764 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76640a7050 con 0x7f767c0720c0 2023-10-25T00:15:04.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.164+0000 7f7679d9b640 1 -- 172.21.15.107:0/1224214764 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76640ab980 con 0x7f767c0720c0 2023-10-25T00:15:04.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.168+0000 7f768084a640 1 -- 172.21.15.107:0/1224214764 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f767c0720c0 msgr2=0x7f767c0c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:04.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.168+0000 7f768084a640 1 --2- 172.21.15.107:0/1224214764 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f767c0720c0 0x7f767c0c18a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f76640097c0 tx=0x7f766401c380 comp rx=0 tx=0).stop 2023-10-25T00:15:04.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.168+0000 7f768084a640 1 -- 172.21.15.107:0/1224214764 shutdown_connections 2023-10-25T00:15:04.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.168+0000 7f768084a640 1 --2- 172.21.15.107:0/1224214764 >> v2:172.21.15.110:3300/0 conn(0x7f767c157e50 0x7f767c1582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:04.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.168+0000 7f768084a640 1 --2- 172.21.15.107:0/1224214764 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f767c0720c0 0x7f767c0c18a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:04.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.168+0000 7f768084a640 1 -- 172.21.15.107:0/1224214764 >> 172.21.15.107:0/1224214764 conn(0x7f767c0c39a0 msgr2=0x7f767c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:04.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.168+0000 7f768084a640 1 -- 172.21.15.107:0/1224214764 shutdown_connections 2023-10-25T00:15:04.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.168+0000 7f768084a640 1 -- 172.21.15.107:0/1224214764 wait complete. 2023-10-25T00:15:04.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.168+0000 7f768084a640 1 Processor -- start 2023-10-25T00:15:04.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f768084a640 1 -- start start 2023-10-25T00:15:04.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f768084a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f767c0720c0 0x7f767c0a96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:04.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f768084a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c13dc20 0x7f767c14dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:04.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f768084a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f767c157e50 0x7f767c14e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:04.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f768084a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c156650 con 0x7f767c13dc20 2023-10-25T00:15:04.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f768084a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f767c1503c0 con 0x7f767c157e50 2023-10-25T00:15:04.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f768084a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767c1f18a0 con 0x7f767c0720c0 2023-10-25T00:15:04.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f767ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f767c157e50 0x7f767c14e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:04.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f767ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f767c157e50 0x7f767c14e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35376/0 (socket says 172.21.15.107:35376) 2023-10-25T00:15:04.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f767ad9d640 1 -- 172.21.15.107:0/3601171002 learned_addr learned my addr 172.21.15.107:0/3601171002 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:04.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f7671d9b640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c13dc20 0x7f767c14dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:04.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f767ad9d640 1 -- 172.21.15.107:0/3601171002 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f767c0720c0 msgr2=0x7f767c0a96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:04.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f767ad9d640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f767c0720c0 0x7f767c0a96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:04.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f767ad9d640 1 -- 172.21.15.107:0/3601171002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c13dc20 msgr2=0x7f767c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:04.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f767ad9d640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c13dc20 0x7f767c14dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:04.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f767ad9d640 1 -- 172.21.15.107:0/3601171002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7674009a70 con 0x7f767c157e50 2023-10-25T00:15:04.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f767ad9d640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f767c157e50 0x7f767c14e170 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f767400efc0 tx=0x7f7674009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:04.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f76737fe640 1 -- 172.21.15.107:0/3601171002 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7674009d70 con 0x7f767c157e50 2023-10-25T00:15:04.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f768084a640 1 -- 172.21.15.107:0/3601171002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7664009420 con 0x7f767c157e50 2023-10-25T00:15:04.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.180+0000 7f768084a640 1 -- 172.21.15.107:0/3601171002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767c1f1b40 con 0x7f767c157e50 2023-10-25T00:15:04.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.184+0000 7f76737fe640 1 -- 172.21.15.107:0/3601171002 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767400f040 con 0x7f767c157e50 2023-10-25T00:15:04.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.184+0000 7f76737fe640 1 -- 172.21.15.107:0/3601171002 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76740039a0 con 0x7f767c157e50 2023-10-25T00:15:04.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.184+0000 7f768084a640 1 -- 172.21.15.107:0/3601171002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7660005260 con 0x7f767c157e50 2023-10-25T00:15:04.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.184+0000 7f76737fe640 1 -- 172.21.15.107:0/3601171002 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f76740144a0 con 0x7f767c157e50 2023-10-25T00:15:04.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.184+0000 7f76737fe640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f765c065f30 0x7f765c0683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:04.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.184+0000 7f76737fe640 1 -- 172.21.15.107:0/3601171002 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7f76740852f0 con 0x7f767c157e50 2023-10-25T00:15:04.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.184+0000 7f767a59c640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f765c065f30 0x7f765c0683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:04.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.184+0000 7f767a59c640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f765c065f30 0x7f765c0683f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f767c0aa580 tx=0x7f766c007a50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:04.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.192+0000 7f76737fe640 1 -- 172.21.15.107:0/3601171002 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7674051af0 con 0x7f767c157e50 2023-10-25T00:15:04.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.728+0000 7f768084a640 1 -- 172.21.15.107:0/3601171002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7660005a90 con 0x7f767c157e50 2023-10-25T00:15:04.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.728+0000 7f76737fe640 1 -- 172.21.15.107:0/3601171002 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f7674018090 con 0x7f767c157e50 2023-10-25T00:15:04.728 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:04.729 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:04.729 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:04.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 -- 172.21.15.107:0/3601171002 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f765c065f30 msgr2=0x7f765c0683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f765c065f30 0x7f765c0683f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f767c0aa580 tx=0x7f766c007a50 comp rx=0 tx=0).stop 2023-10-25T00:15:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 -- 172.21.15.107:0/3601171002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f767c157e50 msgr2=0x7f767c14e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f767c157e50 0x7f767c14e170 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f767400efc0 tx=0x7f7674009f90 comp rx=0 tx=0).stop 2023-10-25T00:15:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 -- 172.21.15.107:0/3601171002 shutdown_connections 2023-10-25T00:15:04.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f767c157e50 0x7f767c14e170 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f767c13dc20 0x7f767c14dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f765c065f30 0x7f765c0683f0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 --2- 172.21.15.107:0/3601171002 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f767c0720c0 0x7f767c0a96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 -- 172.21.15.107:0/3601171002 >> 172.21.15.107:0/3601171002 conn(0x7f767c0c39a0 msgr2=0x7f767c15e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:04.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 -- 172.21.15.107:0/3601171002 shutdown_connections 2023-10-25T00:15:04.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:04.732+0000 7f768084a640 1 -- 172.21.15.107:0/3601171002 wait complete. 2023-10-25T00:15:04.770 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:15:06.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:06.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.968+0000 7f24bee07640 1 Processor -- start 2023-10-25T00:15:06.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.968+0000 7f24bee07640 1 -- start start 2023-10-25T00:15:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.968+0000 7f24bee07640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f24b806cc00 0x7f24b8151f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.968+0000 7f24bee07640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 0x7f24b806b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.968+0000 7f24bee07640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b80c3890 con 0x7f24b806a270 2023-10-25T00:15:06.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.968+0000 7f24bee07640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b8155810 con 0x7f24b806cc00 2023-10-25T00:15:06.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.968+0000 7f24bee07640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b81524c0 con 0x7f24b806b2e0 2023-10-25T00:15:06.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.968+0000 7f24bcba3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f24b806cc00 0x7f24b8151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:06.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.968+0000 7f24bd3a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 0x7f24b806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:06.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.968+0000 7f24bcba3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f24b806cc00 0x7f24b8151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35392/0 (socket says 172.21.15.107:35392) 2023-10-25T00:15:06.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.972+0000 7f24bcba3640 1 -- 172.21.15.107:0/803755362 learned_addr learned my addr 172.21.15.107:0/803755362 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:06.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.972+0000 7f24bd3a4640 1 -- 172.21.15.107:0/803755362 >> v2:172.21.15.110:3300/0 conn(0x7f24b806cc00 msgr2=0x7f24b8151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:06.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.972+0000 7f24bd3a4640 1 --2- 172.21.15.107:0/803755362 >> v2:172.21.15.110:3300/0 conn(0x7f24b806cc00 0x7f24b8151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:06.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.972+0000 7f24bd3a4640 1 -- 172.21.15.107:0/803755362 >> v1:172.21.15.110:6789/0 conn(0x7f24b806a270 legacy=0x7f24b80c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:06.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.972+0000 7f24bd3a4640 1 -- 172.21.15.107:0/803755362 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b8154b90 con 0x7f24b806b2e0 2023-10-25T00:15:06.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.972+0000 7f24bcba3640 1 --2- 172.21.15.107:0/803755362 >> v2:172.21.15.110:3300/0 conn(0x7f24b806cc00 0x7f24b8151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:15:06.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24bd3a4640 1 --2- 172.21.15.107:0/803755362 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 0x7f24b806b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009d90 tx=0x7f24b001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b01d1a78ccf853d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:06.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24b77fe640 1 -- 172.21.15.107:0/803755362 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b00a5720 con 0x7f24b806b2e0 2023-10-25T00:15:06.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24b77fe640 1 -- 172.21.15.107:0/803755362 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b00a5880 con 0x7f24b806b2e0 2023-10-25T00:15:06.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24b77fe640 1 -- 172.21.15.107:0/803755362 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b00a5c10 con 0x7f24b806b2e0 2023-10-25T00:15:06.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24bee07640 1 -- 172.21.15.107:0/803755362 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 msgr2=0x7f24b806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:06.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24bee07640 1 --2- 172.21.15.107:0/803755362 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 0x7f24b806b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009d90 tx=0x7f24b001c970 comp rx=0 tx=0).stop 2023-10-25T00:15:06.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24bee07640 1 -- 172.21.15.107:0/803755362 shutdown_connections 2023-10-25T00:15:06.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24bee07640 1 --2- 172.21.15.107:0/803755362 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 0x7f24b806b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:06.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24bee07640 1 --2- 172.21.15.107:0/803755362 >> v2:172.21.15.110:3300/0 conn(0x7f24b806cc00 0x7f24b8151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:06.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24bee07640 1 -- 172.21.15.107:0/803755362 >> 172.21.15.107:0/803755362 conn(0x7f24b80c2950 msgr2=0x7f24b80c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:06.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24bee07640 1 -- 172.21.15.107:0/803755362 shutdown_connections 2023-10-25T00:15:06.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24bee07640 1 -- 172.21.15.107:0/803755362 wait complete. 2023-10-25T00:15:06.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.976+0000 7f24bee07640 1 Processor -- start 2023-10-25T00:15:06.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bee07640 1 -- start start 2023-10-25T00:15:06.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bee07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24b806a270 0x7f24b80c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:06.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bee07640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 0x7f24b814d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:06.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bee07640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24b806cc00 0x7f24b814d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:06.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bee07640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b82033f0 con 0x7f24b806a270 2023-10-25T00:15:06.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bee07640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b814dd40 con 0x7f24b806cc00 2023-10-25T00:15:06.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bee07640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b82027e0 con 0x7f24b806b2e0 2023-10-25T00:15:06.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bcba3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24b806a270 0x7f24b80c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:06.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bcba3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24b806a270 0x7f24b80c50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35636/0 (socket says 172.21.15.107:35636) 2023-10-25T00:15:06.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bcba3640 1 -- 172.21.15.107:0/3728812519 learned_addr learned my addr 172.21.15.107:0/3728812519 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:06.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bd3a4640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24b806cc00 0x7f24b814d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:06.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24b7fff640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 0x7f24b814d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:06.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bcba3640 1 -- 172.21.15.107:0/3728812519 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 msgr2=0x7f24b814d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:06.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bcba3640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 0x7f24b814d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:06.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bcba3640 1 -- 172.21.15.107:0/3728812519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24b806cc00 msgr2=0x7f24b814d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:06.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bcba3640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24b806cc00 0x7f24b814d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:06.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bcba3640 1 -- 172.21.15.107:0/3728812519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a40094d0 con 0x7f24b806a270 2023-10-25T00:15:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24b7fff640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 0x7f24b814d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:15:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bcba3640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24b806a270 0x7f24b80c50a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f24a400e930 tx=0x7f24a400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24b57fa640 1 -- 172.21.15.107:0/3728812519 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a400ca00 con 0x7f24b806a270 2023-10-25T00:15:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24b57fa640 1 -- 172.21.15.107:0/3728812519 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a400cb60 con 0x7f24b806a270 2023-10-25T00:15:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.988+0000 7f24bee07640 1 -- 172.21.15.107:0/3728812519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b0009a70 con 0x7f24b806a270 2023-10-25T00:15:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.992+0000 7f24b57fa640 1 -- 172.21.15.107:0/3728812519 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4013470 con 0x7f24b806a270 2023-10-25T00:15:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.992+0000 7f24b57fa640 1 -- 172.21.15.107:0/3728812519 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f24a4013650 con 0x7f24b806a270 2023-10-25T00:15:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.992+0000 7f24bee07640 1 -- 172.21.15.107:0/3728812519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b8202da0 con 0x7f24b806a270 2023-10-25T00:15:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.992+0000 7f24b57fa640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f249c065f30 0x7f249c0683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:07.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.992+0000 7f24b57fa640 1 -- 172.21.15.107:0/3728812519 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7f24a4087170 con 0x7f24b806a270 2023-10-25T00:15:07.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.992+0000 7f24b7fff640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f249c065f30 0x7f249c0683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:07.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.992+0000 7f24bee07640 1 -- 172.21.15.107:0/3728812519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b80c03f0 con 0x7f24b806a270 2023-10-25T00:15:07.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:06.996+0000 7f24b7fff640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f249c065f30 0x7f249c0683f0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f24b81484a0 tx=0x7f24ac009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:07.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.004+0000 7f24b57fa640 1 -- 172.21.15.107:0/3728812519 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f24a4017080 con 0x7f24b806a270 2023-10-25T00:15:07.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.504+0000 7f24bee07640 1 -- 172.21.15.107:0/3728812519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24b80746e0 con 0x7f24b806a270 2023-10-25T00:15:07.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.504+0000 7f24b57fa640 1 -- 172.21.15.107:0/3728812519 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f24a4088060 con 0x7f24b806a270 2023-10-25T00:15:07.503 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:07.504 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:07.504 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:07.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 -- 172.21.15.107:0/3728812519 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f249c065f30 msgr2=0x7f249c0683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f249c065f30 0x7f249c0683f0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f24b81484a0 tx=0x7f24ac009040 comp rx=0 tx=0).stop 2023-10-25T00:15:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 -- 172.21.15.107:0/3728812519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24b806a270 msgr2=0x7f24b80c50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:07.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24b806a270 0x7f24b80c50a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f24a400e930 tx=0x7f24a400edf0 comp rx=0 tx=0).stop 2023-10-25T00:15:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 -- 172.21.15.107:0/3728812519 shutdown_connections 2023-10-25T00:15:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24b806cc00 0x7f24b814d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f249c065f30 0x7f249c0683f0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:07.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24b806b2e0 0x7f24b814d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 --2- 172.21.15.107:0/3728812519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24b806a270 0x7f24b80c50a0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 -- 172.21.15.107:0/3728812519 >> 172.21.15.107:0/3728812519 conn(0x7f24b80c2950 msgr2=0x7f24b8156b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 -- 172.21.15.107:0/3728812519 shutdown_connections 2023-10-25T00:15:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.508+0000 7f24bee07640 1 -- 172.21.15.107:0/3728812519 wait complete. 2023-10-25T00:15:07.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:07.538 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:15:07.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.704+0000 7ff59430d640 1 Processor -- start 2023-10-25T00:15:07.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.704+0000 7ff59430d640 1 -- start start 2023-10-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.704+0000 7ff59430d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff58c070b00 0x7ff58c0c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.704+0000 7ff59430d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58c06df30 0x7ff58c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.704+0000 7ff59430d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c0c22f0 con 0x7ff58c14eb70 2023-10-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.704+0000 7ff59430d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c071700 con 0x7ff58c070b00 2023-10-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.704+0000 7ff59430d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c0c25d0 con 0x7ff58c06df30 2023-10-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.704+0000 7ff5920a9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff58c14eb70 0x7ff58c1514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51826/0 (socket says 172.21.15.107:51826) 2023-10-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.704+0000 7ff5920a9640 1 -- 172.21.15.107:0/3561617572 learned_addr learned my addr 172.21.15.107:0/3561617572 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.704+0000 7ff5918a8640 1 --2- 172.21.15.107:0/3561617572 >> v2:172.21.15.110:3300/0 conn(0x7ff58c070b00 0x7ff58c0c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.704+0000 7ff5928aa640 1 --2- 172.21.15.107:0/3561617572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58c06df30 0x7ff58c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:07.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.708+0000 7ff5910a7640 1 -- 172.21.15.107:0/3561617572 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569961294 0 0) 0x7ff58c0c22f0 con 0x7ff58c14eb70 2023-10-25T00:15:07.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.708+0000 7ff5910a7640 1 -- 172.21.15.107:0/3561617572 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff578003530 con 0x7ff58c14eb70 2023-10-25T00:15:07.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.708+0000 7ff5918a8640 1 -- 172.21.15.107:0/3561617572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58c06df30 msgr2=0x7ff58c06e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:07.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.708+0000 7ff5918a8640 1 --2- 172.21.15.107:0/3561617572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58c06df30 0x7ff58c06e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.708+0000 7ff5918a8640 1 -- 172.21.15.107:0/3561617572 >> v1:172.21.15.107:6789/0 conn(0x7ff58c14eb70 legacy=0x7ff58c1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.708+0000 7ff5918a8640 1 -- 172.21.15.107:0/3561617572 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c0c2be0 con 0x7ff58c070b00 2023-10-25T00:15:07.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.708+0000 7ff5928aa640 1 --2- 172.21.15.107:0/3561617572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58c06df30 0x7ff58c06e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:15:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.708+0000 7ff5918a8640 1 --2- 172.21.15.107:0/3561617572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff58c070b00 0x7ff58c0c33e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0097c0 tx=0x7ff57c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc00f8aa178a92f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.708+0000 7ff5910a7640 1 -- 172.21.15.107:0/3561617572 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0ac620 con 0x7ff58c070b00 2023-10-25T00:15:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.708+0000 7ff5910a7640 1 -- 172.21.15.107:0/3561617572 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c0ac780 con 0x7ff58c070b00 2023-10-25T00:15:07.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.708+0000 7ff5910a7640 1 -- 172.21.15.107:0/3561617572 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0acb50 con 0x7ff58c070b00 2023-10-25T00:15:07.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.712+0000 7ff59430d640 1 -- 172.21.15.107:0/3561617572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff58c070b00 msgr2=0x7ff58c0c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:07.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.712+0000 7ff59430d640 1 --2- 172.21.15.107:0/3561617572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff58c070b00 0x7ff58c0c33e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0097c0 tx=0x7ff57c01c4a0 comp rx=0 tx=0).stop 2023-10-25T00:15:07.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.712+0000 7ff59430d640 1 -- 172.21.15.107:0/3561617572 shutdown_connections 2023-10-25T00:15:07.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.712+0000 7ff59430d640 1 --2- 172.21.15.107:0/3561617572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58c06df30 0x7ff58c06e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:07.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.712+0000 7ff59430d640 1 --2- 172.21.15.107:0/3561617572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff58c070b00 0x7ff58c0c33e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:07.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.712+0000 7ff59430d640 1 -- 172.21.15.107:0/3561617572 >> 172.21.15.107:0/3561617572 conn(0x7ff58c0c39a0 msgr2=0x7ff58c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:07.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.712+0000 7ff59430d640 1 -- 172.21.15.107:0/3561617572 shutdown_connections 2023-10-25T00:15:07.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.712+0000 7ff59430d640 1 -- 172.21.15.107:0/3561617572 wait complete. 2023-10-25T00:15:07.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.712+0000 7ff59430d640 1 Processor -- start 2023-10-25T00:15:07.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.712+0000 7ff59430d640 1 -- start start 2023-10-25T00:15:07.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff59430d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58c06df30 0x7ff58c150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:07.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff59430d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff58c070b00 0x7ff58c14a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff59430d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff58c14eb70 0x7ff58c14ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff59430d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c149aa0 con 0x7ff58c14eb70 2023-10-25T00:15:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff59430d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c149ec0 con 0x7ff58c070b00 2023-10-25T00:15:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff59430d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c0ab280 con 0x7ff58c06df30 2023-10-25T00:15:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff5918a8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff58c070b00 0x7ff58c14a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff5918a8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff58c070b00 0x7ff58c14a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35412/0 (socket says 172.21.15.107:35412) 2023-10-25T00:15:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff5918a8640 1 -- 172.21.15.107:0/3634912632 learned_addr learned my addr 172.21.15.107:0/3634912632 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff5920a9640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58c06df30 0x7ff58c150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff5928aa640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff58c14eb70 0x7ff58c14ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff5918a8640 1 -- 172.21.15.107:0/3634912632 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58c06df30 msgr2=0x7ff58c150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:07.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff5918a8640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58c06df30 0x7ff58c150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:07.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff5918a8640 1 -- 172.21.15.107:0/3634912632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff58c14eb70 msgr2=0x7ff58c14ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:07.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff5918a8640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff58c14eb70 0x7ff58c14ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:07.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff5918a8640 1 -- 172.21.15.107:0/3634912632 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58400bc80 con 0x7ff58c070b00 2023-10-25T00:15:07.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff5918a8640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff58c070b00 0x7ff58c14a050 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0097c0 tx=0x7ff57c01ace0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:07.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff582ffd640 1 -- 172.21.15.107:0/3634912632 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0ac620 con 0x7ff58c070b00 2023-10-25T00:15:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff582ffd640 1 -- 172.21.15.107:0/3634912632 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c0b8450 con 0x7ff58c070b00 2023-10-25T00:15:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff582ffd640 1 -- 172.21.15.107:0/3634912632 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0b87e0 con 0x7ff58c070b00 2023-10-25T00:15:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff59430d640 1 -- 172.21.15.107:0/3634912632 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57c009420 con 0x7ff58c070b00 2023-10-25T00:15:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff59430d640 1 -- 172.21.15.107:0/3634912632 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58c149100 con 0x7ff58c070b00 2023-10-25T00:15:07.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.716+0000 7ff582ffd640 1 -- 172.21.15.107:0/3634912632 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff57c0aa070 con 0x7ff58c070b00 2023-10-25T00:15:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.720+0000 7ff582ffd640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff570065e90 0x7ff570068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.720+0000 7ff582ffd640 1 -- 172.21.15.107:0/3634912632 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7ff57c11e980 con 0x7ff58c070b00 2023-10-25T00:15:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.720+0000 7ff5920a9640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff570065e90 0x7ff570068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:07.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.720+0000 7ff5920a9640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff570065e90 0x7ff570068350 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff588005890 tx=0x7ff588009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:07.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.724+0000 7ff59430d640 1 -- 172.21.15.107:0/3634912632 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58c149be0 con 0x7ff58c070b00 2023-10-25T00:15:07.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:07.732+0000 7ff582ffd640 1 -- 172.21.15.107:0/3634912632 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff58c149be0 con 0x7ff58c070b00 2023-10-25T00:15:08.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.124+0000 7ff59430d640 1 -- 172.21.15.107:0/3634912632 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff58c149be0 con 0x7ff58c070b00 2023-10-25T00:15:08.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.124+0000 7ff582ffd640 1 -- 172.21.15.107:0/3634912632 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7ff58c149be0 con 0x7ff58c070b00 2023-10-25T00:15:08.125 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:08.126 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:08.126 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:08.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 -- 172.21.15.107:0/3634912632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff570065e90 msgr2=0x7ff570068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:08.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff570065e90 0x7ff570068350 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff588005890 tx=0x7ff588009040 comp rx=0 tx=0).stop 2023-10-25T00:15:08.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 -- 172.21.15.107:0/3634912632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff58c070b00 msgr2=0x7ff58c14a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:08.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff58c070b00 0x7ff58c14a050 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0097c0 tx=0x7ff57c01ace0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 -- 172.21.15.107:0/3634912632 shutdown_connections 2023-10-25T00:15:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff58c14eb70 0x7ff58c14ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff58c070b00 0x7ff58c14a050 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff570065e90 0x7ff570068350 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 --2- 172.21.15.107:0/3634912632 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58c06df30 0x7ff58c150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 -- 172.21.15.107:0/3634912632 >> 172.21.15.107:0/3634912632 conn(0x7ff58c0c39a0 msgr2=0x7ff58c06fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 -- 172.21.15.107:0/3634912632 shutdown_connections 2023-10-25T00:15:08.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.128+0000 7ff59430d640 1 -- 172.21.15.107:0/3634912632 wait complete. 2023-10-25T00:15:08.162 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2023-10-25T00:15:08.163 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-25T00:15:08.163 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:15:08.176 INFO:tasks.ceph.mds.d:Stopped 2023-10-25T00:15:08.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:08.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.360+0000 7f70124f1640 1 Processor -- start 2023-10-25T00:15:08.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.360+0000 7f70124f1640 1 -- start start 2023-10-25T00:15:08.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.360+0000 7f70124f1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f700c139be0 0x7f700c0aade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:08.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.360+0000 7f70124f1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f700c153fb0 0x7f700c154400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:08.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.360+0000 7f70124f1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700c139fc0 con 0x7f700c069dd0 2023-10-25T00:15:08.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.360+0000 7f70124f1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700c150870 con 0x7f700c153fb0 2023-10-25T00:15:08.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.360+0000 7f70124f1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700c150a80 con 0x7f700c139be0 2023-10-25T00:15:08.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.360+0000 7f700bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f700c139be0 0x7f700c0aade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:08.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.360+0000 7f7010a8e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f700c153fb0 0x7f700c154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:08.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.360+0000 7f700bfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f700c139be0 0x7f700c0aade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35420/0 (socket says 172.21.15.107:35420) 2023-10-25T00:15:08.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.360+0000 7f700bfff640 1 -- 172.21.15.107:0/1525479229 learned_addr learned my addr 172.21.15.107:0/1525479229 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:08.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.364+0000 7f700bfff640 1 -- 172.21.15.107:0/1525479229 >> v2:172.21.15.107:3300/0 conn(0x7f700c153fb0 msgr2=0x7f700c154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:08.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.364+0000 7f700bfff640 1 --2- 172.21.15.107:0/1525479229 >> v2:172.21.15.107:3300/0 conn(0x7f700c153fb0 0x7f700c154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.364+0000 7f700bfff640 1 -- 172.21.15.107:0/1525479229 >> v1:172.21.15.110:6789/0 conn(0x7f700c069dd0 legacy=0x7f700c0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:08.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.364+0000 7f700bfff640 1 -- 172.21.15.107:0/1525479229 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c150d60 con 0x7f700c139be0 2023-10-25T00:15:08.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.364+0000 7f700bfff640 1 --2- 172.21.15.107:0/1525479229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700c139be0 0x7f700c0aade0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f700c06e9d0 tx=0x7f6ffc01c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e06dbcae87498e49 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:08.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.364+0000 7f700b7fe640 1 -- 172.21.15.107:0/1525479229 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0ab640 con 0x7f700c139be0 2023-10-25T00:15:08.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.364+0000 7f700b7fe640 1 -- 172.21.15.107:0/1525479229 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc0a7070 con 0x7f700c139be0 2023-10-25T00:15:08.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.364+0000 7f700b7fe640 1 -- 172.21.15.107:0/1525479229 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0ab9c0 con 0x7f700c139be0 2023-10-25T00:15:08.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.368+0000 7f70124f1640 1 -- 172.21.15.107:0/1525479229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700c139be0 msgr2=0x7f700c0aade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:08.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.368+0000 7f70124f1640 1 --2- 172.21.15.107:0/1525479229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700c139be0 0x7f700c0aade0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f700c06e9d0 tx=0x7f6ffc01c440 comp rx=0 tx=0).stop 2023-10-25T00:15:08.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.368+0000 7f70124f1640 1 -- 172.21.15.107:0/1525479229 shutdown_connections 2023-10-25T00:15:08.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.368+0000 7f70124f1640 1 --2- 172.21.15.107:0/1525479229 >> v2:172.21.15.107:3300/0 conn(0x7f700c153fb0 0x7f700c154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.368+0000 7f70124f1640 1 --2- 172.21.15.107:0/1525479229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700c139be0 0x7f700c0aade0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.368+0000 7f70124f1640 1 -- 172.21.15.107:0/1525479229 >> 172.21.15.107:0/1525479229 conn(0x7f700c0c2950 msgr2=0x7f700c0c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:08.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.372+0000 7f70124f1640 1 -- 172.21.15.107:0/1525479229 shutdown_connections 2023-10-25T00:15:08.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.372+0000 7f70124f1640 1 -- 172.21.15.107:0/1525479229 wait complete. 2023-10-25T00:15:08.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f70124f1640 1 Processor -- start 2023-10-25T00:15:08.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f70124f1640 1 -- start start 2023-10-25T00:15:08.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f70124f1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700c069dd0 0x7f700c163ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:08.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f70124f1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700c139be0 0x7f700c15ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:08.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f70124f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f700c153fb0 0x7f700c161c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:08.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f70124f1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c20ecb0 con 0x7f700c153fb0 2023-10-25T00:15:08.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f70124f1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c20ef90 con 0x7f700c069dd0 2023-10-25T00:15:08.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f70124f1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700c06be20 con 0x7f700c139be0 2023-10-25T00:15:08.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f700bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700c069dd0 0x7f700c163ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f700bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700c069dd0 0x7f700c163ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35432/0 (socket says 172.21.15.107:35432) 2023-10-25T00:15:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f700bfff640 1 -- 172.21.15.107:0/3517582034 learned_addr learned my addr 172.21.15.107:0/3517582034 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f700bfff640 1 -- 172.21.15.107:0/3517582034 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700c139be0 msgr2=0x7f700c15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f700bfff640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700c139be0 0x7f700c15ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f700bfff640 1 -- 172.21.15.107:0/3517582034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f700c153fb0 msgr2=0x7f700c161c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:08.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f700bfff640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f700c153fb0 0x7f700c161c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f700bfff640 1 -- 172.21.15.107:0/3517582034 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc0094d0 con 0x7f700c069dd0 2023-10-25T00:15:08.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.376+0000 7f700bfff640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700c069dd0 0x7f700c163ae0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc019050 tx=0x7f6ffc01ac80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:08.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.380+0000 7f70097fa640 1 -- 172.21.15.107:0/3517582034 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0abea0 con 0x7f700c069dd0 2023-10-25T00:15:08.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.380+0000 7f70097fa640 1 -- 172.21.15.107:0/3517582034 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc0be3f0 con 0x7f700c069dd0 2023-10-25T00:15:08.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.380+0000 7f70097fa640 1 -- 172.21.15.107:0/3517582034 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0be7c0 con 0x7f700c069dd0 2023-10-25T00:15:08.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.380+0000 7f70124f1640 1 -- 172.21.15.107:0/3517582034 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700c153b40 con 0x7f700c069dd0 2023-10-25T00:15:08.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.380+0000 7f70124f1640 1 -- 172.21.15.107:0/3517582034 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700c06c100 con 0x7f700c069dd0 2023-10-25T00:15:08.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.380+0000 7f70027fc640 1 -- 172.21.15.107:0/3517582034 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe4005260 con 0x7f700c069dd0 2023-10-25T00:15:08.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.384+0000 7f70097fa640 1 -- 172.21.15.107:0/3517582034 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6ffc0059a0 con 0x7f700c069dd0 2023-10-25T00:15:08.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.384+0000 7f70097fa640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065eb0 0x7f6fec068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:08.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.384+0000 7f70097fa640 1 -- 172.21.15.107:0/3517582034 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7f6ffc11f520 con 0x7f700c069dd0 2023-10-25T00:15:08.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.384+0000 7f7003fff640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065eb0 0x7f6fec068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:08.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.384+0000 7f7003fff640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065eb0 0x7f6fec068370 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4007fd0 tx=0x7f6ff4007960 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:08.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.388+0000 7f70097fa640 1 -- 172.21.15.107:0/3517582034 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6ffc0ebcb0 con 0x7f700c069dd0 2023-10-25T00:15:08.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.716+0000 7f70027fc640 1 -- 172.21.15.107:0/3517582034 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fe40057e0 con 0x7f700c069dd0 2023-10-25T00:15:08.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.720+0000 7f70097fa640 1 -- 172.21.15.107:0/3517582034 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f6ffc0ef2d0 con 0x7f700c069dd0 2023-10-25T00:15:08.718 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:08.718 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:08.719 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.724+0000 7f70027fc640 1 -- 172.21.15.107:0/3517582034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065eb0 msgr2=0x7f6fec068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:08.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.724+0000 7f70027fc640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065eb0 0x7f6fec068370 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4007fd0 tx=0x7f6ff4007960 comp rx=0 tx=0).stop 2023-10-25T00:15:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.724+0000 7f70027fc640 1 -- 172.21.15.107:0/3517582034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700c069dd0 msgr2=0x7f700c163ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:08.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.724+0000 7f70027fc640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700c069dd0 0x7f700c163ae0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc019050 tx=0x7f6ffc01ac80 comp rx=0 tx=0).stop 2023-10-25T00:15:08.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.728+0000 7f70027fc640 1 -- 172.21.15.107:0/3517582034 shutdown_connections 2023-10-25T00:15:08.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.728+0000 7f70027fc640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f700c153fb0 0x7f700c161c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.728+0000 7f70027fc640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6fec065eb0 0x7f6fec068370 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.728+0000 7f70027fc640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f700c139be0 0x7f700c15ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.728+0000 7f70027fc640 1 --2- 172.21.15.107:0/3517582034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f700c069dd0 0x7f700c163ae0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.728+0000 7f70027fc640 1 -- 172.21.15.107:0/3517582034 >> 172.21.15.107:0/3517582034 conn(0x7f700c0c2950 msgr2=0x7f700c137de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.728+0000 7f70027fc640 1 -- 172.21.15.107:0/3517582034 shutdown_connections 2023-10-25T00:15:08.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:08.728+0000 7f70027fc640 1 -- 172.21.15.107:0/3517582034 wait complete. 2023-10-25T00:15:08.763 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:15:09.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d1325f640 1 Processor -- start 2023-10-25T00:15:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d1325f640 1 -- start start 2023-10-25T00:15:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d1325f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5d0c15c200 0x7f5d0c071100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d1325f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c0c5d40 con 0x7f5d0c156220 2023-10-25T00:15:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d1325f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c0c6130 con 0x7f5d0c0c6a30 2023-10-25T00:15:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d1325f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0c158f60 con 0x7f5d0c15c200 2023-10-25T00:15:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d10ffb640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5d0c156220 0x7f5d0c06df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51832/0 (socket says 172.21.15.107:51832) 2023-10-25T00:15:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d10ffb640 1 -- 172.21.15.107:0/3827675683 learned_addr learned my addr 172.21.15.107:0/3827675683 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d0bfff640 1 --2- 172.21.15.107:0/3827675683 >> v2:172.21.15.110:3300/0 conn(0x7f5d0c15c200 0x7f5d0c071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d0b7fe640 1 -- 172.21.15.107:0/3827675683 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720045541 0 0) 0x7f5d0c0c5d40 con 0x7f5d0c156220 2023-10-25T00:15:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d0b7fe640 1 -- 172.21.15.107:0/3827675683 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf8003530 con 0x7f5d0c156220 2023-10-25T00:15:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d0bfff640 1 -- 172.21.15.107:0/3827675683 >> v1:172.21.15.110:6789/0 conn(0x7f5d0c0c6a30 legacy=0x7f5d0c071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d0bfff640 1 -- 172.21.15.107:0/3827675683 >> v1:172.21.15.107:6789/0 conn(0x7f5d0c156220 legacy=0x7f5d0c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.704+0000 7f5d0bfff640 1 -- 172.21.15.107:0/3827675683 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c159240 con 0x7f5d0c15c200 2023-10-25T00:15:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.712+0000 7f5d0bfff640 1 --2- 172.21.15.107:0/3827675683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d0c15c200 0x7f5d0c071100 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0097c0 tx=0x7f5cfc01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ef0d6adcf79cbbf server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.712+0000 7f5d0b7fe640 1 -- 172.21.15.107:0/3827675683 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc0ac620 con 0x7f5d0c15c200 2023-10-25T00:15:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.712+0000 7f5d0b7fe640 1 -- 172.21.15.107:0/3827675683 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cfc0ac780 con 0x7f5d0c15c200 2023-10-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.712+0000 7f5d1325f640 1 -- 172.21.15.107:0/3827675683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d0c15c200 msgr2=0x7f5d0c071100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.712+0000 7f5d1325f640 1 --2- 172.21.15.107:0/3827675683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d0c15c200 0x7f5d0c071100 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc0097c0 tx=0x7f5cfc01c3c0 comp rx=0 tx=0).stop 2023-10-25T00:15:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 -- 172.21.15.107:0/3827675683 shutdown_connections 2023-10-25T00:15:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 --2- 172.21.15.107:0/3827675683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d0c15c200 0x7f5d0c071100 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 -- 172.21.15.107:0/3827675683 >> 172.21.15.107:0/3827675683 conn(0x7f5d0c0c39a0 msgr2=0x7f5d0c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 -- 172.21.15.107:0/3827675683 shutdown_connections 2023-10-25T00:15:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 -- 172.21.15.107:0/3827675683 wait complete. 2023-10-25T00:15:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 Processor -- start 2023-10-25T00:15:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 -- start start 2023-10-25T00:15:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d0c0c6a30 0x7f5d0c0c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d0c156220 0x7f5d0c1501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d0c15c200 0x7f5d0c14b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c15bfa0 con 0x7f5d0c15c200 2023-10-25T00:15:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c14f540 con 0x7f5d0c156220 2023-10-25T00:15:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.716+0000 7f5d1325f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c15b9e0 con 0x7f5d0c0c6a30 2023-10-25T00:15:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d10ffb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d0c0c6a30 0x7f5d0c0c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d10ffb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d0c0c6a30 0x7f5d0c0c4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:57894/0 (socket says 172.21.15.107:57894) 2023-10-25T00:15:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d10ffb640 1 -- 172.21.15.107:0/3578709115 learned_addr learned my addr 172.21.15.107:0/3578709115 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d0bfff640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d0c156220 0x7f5d0c1501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d117fc640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d0c15c200 0x7f5d0c14b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d117fc640 1 -- 172.21.15.107:0/3578709115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d0c0c6a30 msgr2=0x7f5d0c0c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d117fc640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d0c0c6a30 0x7f5d0c0c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d117fc640 1 -- 172.21.15.107:0/3578709115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d0c156220 msgr2=0x7f5d0c1501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d117fc640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d0c156220 0x7f5d0c1501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d117fc640 1 -- 172.21.15.107:0/3578709115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cfc009420 con 0x7f5d0c15c200 2023-10-25T00:15:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d117fc640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d0c15c200 0x7f5d0c14b010 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5cf400c700 tx=0x7f5cf400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d097fa640 1 -- 172.21.15.107:0/3578709115 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4003b80 con 0x7f5d0c15c200 2023-10-25T00:15:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d1325f640 1 -- 172.21.15.107:0/3578709115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0c15bcc0 con 0x7f5d0c15c200 2023-10-25T00:15:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d1325f640 1 -- 172.21.15.107:0/3578709115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0c151770 con 0x7f5d0c15c200 2023-10-25T00:15:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d097fa640 1 -- 172.21.15.107:0/3578709115 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf4003ce0 con 0x7f5d0c15c200 2023-10-25T00:15:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.720+0000 7f5d097fa640 1 -- 172.21.15.107:0/3578709115 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf400aa30 con 0x7f5d0c15c200 2023-10-25T00:15:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.724+0000 7f5d1325f640 1 -- 172.21.15.107:0/3578709115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d0c070480 con 0x7f5d0c15c200 2023-10-25T00:15:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.724+0000 7f5d097fa640 1 -- 172.21.15.107:0/3578709115 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5cf400ab90 con 0x7f5d0c15c200 2023-10-25T00:15:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.724+0000 7f5d097fa640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5cec065e60 0x7f5cec068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.724+0000 7f5d097fa640 1 -- 172.21.15.107:0/3578709115 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7f5cf40843b0 con 0x7f5d0c15c200 2023-10-25T00:15:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.724+0000 7f5d10ffb640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5cec065e60 0x7f5cec068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.724+0000 7f5d10ffb640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5cec065e60 0x7f5cec068320 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f5d04007fd0 tx=0x7f5d04009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:09.728+0000 7f5d097fa640 1 -- 172.21.15.107:0/3578709115 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5cf4050ba0 con 0x7f5d0c15c200 2023-10-25T00:15:10.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.088+0000 7f5d1325f640 1 -- 172.21.15.107:0/3578709115 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d0c158be0 con 0x7f5d0c15c200 2023-10-25T00:15:10.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.088+0000 7f5d097fa640 1 -- 172.21.15.107:0/3578709115 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f5cf40541c0 con 0x7f5d0c15c200 2023-10-25T00:15:10.088 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:10.089 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:10.089 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:10.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 -- 172.21.15.107:0/3578709115 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5cec065e60 msgr2=0x7f5cec068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:10.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5cec065e60 0x7f5cec068320 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f5d04007fd0 tx=0x7f5d04009040 comp rx=0 tx=0).stop 2023-10-25T00:15:10.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 -- 172.21.15.107:0/3578709115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d0c15c200 msgr2=0x7f5d0c14b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:10.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d0c15c200 0x7f5d0c14b010 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5cf400c700 tx=0x7f5cf400cbc0 comp rx=0 tx=0).stop 2023-10-25T00:15:10.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 -- 172.21.15.107:0/3578709115 shutdown_connections 2023-10-25T00:15:10.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d0c15c200 0x7f5d0c14b010 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:10.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d0c156220 0x7f5d0c1501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:10.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5cec065e60 0x7f5cec068320 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:10.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 --2- 172.21.15.107:0/3578709115 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d0c0c6a30 0x7f5d0c0c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:10.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 -- 172.21.15.107:0/3578709115 >> 172.21.15.107:0/3578709115 conn(0x7f5d0c0c39a0 msgr2=0x7f5d0c0c21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:10.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 -- 172.21.15.107:0/3578709115 shutdown_connections 2023-10-25T00:15:10.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.092+0000 7f5d1325f640 1 -- 172.21.15.107:0/3578709115 wait complete. 2023-10-25T00:15:10.136 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:15:10.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:10.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd27e5fb640 1 Processor -- start 2023-10-25T00:15:10.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd27e5fb640 1 -- start start 2023-10-25T00:15:10.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd27e5fb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd27814fd40 0x7fd2780ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:10.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd27e5fb640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd27806ee40 0x7fd2780aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:10.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd27e5fb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd27806f730 con 0x7fd2780a9820 2023-10-25T00:15:10.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd27e5fb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2780aa750 con 0x7fd27814fd40 2023-10-25T00:15:10.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd27e5fb640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd278156a40 con 0x7fd27806ee40 2023-10-25T00:15:10.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd277fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd27814fd40 0x7fd2780ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:10.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd277fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd27814fd40 0x7fd2780ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35678/0 (socket says 172.21.15.107:35678) 2023-10-25T00:15:10.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd277fff640 1 -- 172.21.15.107:0/4008832638 learned_addr learned my addr 172.21.15.107:0/4008832638 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:10.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd27cb98640 1 --2- 172.21.15.107:0/4008832638 >> v2:172.21.15.110:3300/0 conn(0x7fd27806ee40 0x7fd2780aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:10.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd277fff640 1 -- 172.21.15.107:0/4008832638 >> v2:172.21.15.110:3300/0 conn(0x7fd27806ee40 msgr2=0x7fd2780aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:10.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd277fff640 1 --2- 172.21.15.107:0/4008832638 >> v2:172.21.15.110:3300/0 conn(0x7fd27806ee40 0x7fd2780aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:10.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd277fff640 1 -- 172.21.15.107:0/4008832638 >> v1:172.21.15.110:6789/0 conn(0x7fd2780a9820 legacy=0x7fd2780c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:10.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd277fff640 1 -- 172.21.15.107:0/4008832638 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd278156710 con 0x7fd27814fd40 2023-10-25T00:15:10.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd277fff640 1 --2- 172.21.15.107:0/4008832638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27814fd40 0x7fd2780ab090 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fd2781569b0 tx=0x7fd26c01c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39384abc58bd7a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:10.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.896+0000 7fd276ffd640 1 -- 172.21.15.107:0/4008832638 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0ac640 con 0x7fd27814fd40 2023-10-25T00:15:10.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd276ffd640 1 -- 172.21.15.107:0/4008832638 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c0ac7a0 con 0x7fd27814fd40 2023-10-25T00:15:10.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd276ffd640 1 -- 172.21.15.107:0/4008832638 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0acb30 con 0x7fd27814fd40 2023-10-25T00:15:10.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 -- 172.21.15.107:0/4008832638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27814fd40 msgr2=0x7fd2780ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:10.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 --2- 172.21.15.107:0/4008832638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27814fd40 0x7fd2780ab090 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fd2781569b0 tx=0x7fd26c01c370 comp rx=0 tx=0).stop 2023-10-25T00:15:10.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 -- 172.21.15.107:0/4008832638 shutdown_connections 2023-10-25T00:15:10.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 --2- 172.21.15.107:0/4008832638 >> v2:172.21.15.110:3300/0 conn(0x7fd27806ee40 0x7fd2780aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:10.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 --2- 172.21.15.107:0/4008832638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27814fd40 0x7fd2780ab090 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:10.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 -- 172.21.15.107:0/4008832638 >> 172.21.15.107:0/4008832638 conn(0x7fd2780c2950 msgr2=0x7fd2780c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:10.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 -- 172.21.15.107:0/4008832638 shutdown_connections 2023-10-25T00:15:10.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 -- 172.21.15.107:0/4008832638 wait complete. 2023-10-25T00:15:10.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 Processor -- start 2023-10-25T00:15:10.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 -- start start 2023-10-25T00:15:10.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27806ee40 0x7fd2780bb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:10.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2780bfee0 0x7fd27814e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:10.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2780bf990 0x7fd27814ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:10.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2780c5c70 con 0x7fd27806ee40 2023-10-25T00:15:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2781ebe30 con 0x7fd2780bf990 2023-10-25T00:15:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd27815a360 con 0x7fd2780bfee0 2023-10-25T00:15:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd277fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27806ee40 0x7fd2780bb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd277fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27806ee40 0x7fd2780bb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35684/0 (socket says 172.21.15.107:35684) 2023-10-25T00:15:10.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd277fff640 1 -- 172.21.15.107:0/3806882381 learned_addr learned my addr 172.21.15.107:0/3806882381 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27cb98640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2780bf990 0x7fd27814ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd277fff640 1 -- 172.21.15.107:0/3806882381 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2780bfee0 msgr2=0x7fd27814e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd277fff640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2780bfee0 0x7fd27814e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd277fff640 1 -- 172.21.15.107:0/3806882381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2780bf990 msgr2=0x7fd27814ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd277fff640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2780bf990 0x7fd27814ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:10.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd277fff640 1 -- 172.21.15.107:0/3806882381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd26c0094d0 con 0x7fd27806ee40 2023-10-25T00:15:10.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd277fff640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27806ee40 0x7fd2780bb8c0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0af040 tx=0x7fd26c019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:10.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd274ff9640 1 -- 172.21.15.107:0/3806882381 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0ac640 con 0x7fd27806ee40 2023-10-25T00:15:10.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd274ff9640 1 -- 172.21.15.107:0/3806882381 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd26c0b5870 con 0x7fd27806ee40 2023-10-25T00:15:10.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 -- 172.21.15.107:0/3806882381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2781eb870 con 0x7fd27806ee40 2023-10-25T00:15:10.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd274ff9640 1 -- 172.21.15.107:0/3806882381 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c0b5c20 con 0x7fd27806ee40 2023-10-25T00:15:10.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.900+0000 7fd27e5fb640 1 -- 172.21.15.107:0/3806882381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27815a600 con 0x7fd27806ee40 2023-10-25T00:15:10.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.904+0000 7fd27e5fb640 1 -- 172.21.15.107:0/3806882381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2780c24a0 con 0x7fd27806ee40 2023-10-25T00:15:10.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.904+0000 7fd274ff9640 1 -- 172.21.15.107:0/3806882381 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd26c0b3070 con 0x7fd27806ee40 2023-10-25T00:15:10.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.904+0000 7fd274ff9640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd264065e90 0x7fd264068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:10.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.904+0000 7fd274ff9640 1 -- 172.21.15.107:0/3806882381 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7fd26c11e880 con 0x7fd27806ee40 2023-10-25T00:15:10.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.904+0000 7fd2777fe640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd264065e90 0x7fd264068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:10.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.904+0000 7fd2777fe640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd264065e90 0x7fd264068350 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd27815ca00 tx=0x7fd268006bb0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:10.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:10.908+0000 7fd274ff9640 1 -- 172.21.15.107:0/3806882381 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd26c0eb020 con 0x7fd27806ee40 2023-10-25T00:15:11.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.080+0000 7fd27e5fb640 1 -- 172.21.15.107:0/3806882381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd278070840 con 0x7fd27806ee40 2023-10-25T00:15:11.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.080+0000 7fd274ff9640 1 -- 172.21.15.107:0/3806882381 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7fd26c0b9090 con 0x7fd27806ee40 2023-10-25T00:15:11.081 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:11.082 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:11.082 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:11.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 -- 172.21.15.107:0/3806882381 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd264065e90 msgr2=0x7fd264068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd264065e90 0x7fd264068350 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd27815ca00 tx=0x7fd268006bb0 comp rx=0 tx=0).stop 2023-10-25T00:15:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 -- 172.21.15.107:0/3806882381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27806ee40 msgr2=0x7fd2780bb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27806ee40 0x7fd2780bb8c0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fd26c0af040 tx=0x7fd26c019110 comp rx=0 tx=0).stop 2023-10-25T00:15:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 -- 172.21.15.107:0/3806882381 shutdown_connections 2023-10-25T00:15:11.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2780bf990 0x7fd27814ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd264065e90 0x7fd264068350 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2780bfee0 0x7fd27814e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 --2- 172.21.15.107:0/3806882381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd27806ee40 0x7fd2780bb8c0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 -- 172.21.15.107:0/3806882381 >> 172.21.15.107:0/3806882381 conn(0x7fd2780c2950 msgr2=0x7fd2780c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:11.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 -- 172.21.15.107:0/3806882381 shutdown_connections 2023-10-25T00:15:11.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:11.084+0000 7fd27e5fb640 1 -- 172.21.15.107:0/3806882381 wait complete. 2023-10-25T00:15:11.098 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:15:12.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:12.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1d14f640 1 Processor -- start 2023-10-25T00:15:12.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1d14f640 1 -- start start 2023-10-25T00:15:12.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1d14f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180aa140 0x7f9f180aa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:12.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1d14f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9f180c26f0 0x7f9f1806c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:12.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1d14f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f18066000 con 0x7f9f180ac020 2023-10-25T00:15:12.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1d14f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f180aaa60 con 0x7f9f180c26f0 2023-10-25T00:15:12.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1d14f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180c0f40 con 0x7f9f180aa140 2023-10-25T00:15:12.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1759e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9f180c26f0 0x7f9f1806c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:12.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f16d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180aa140 0x7f9f180aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:12.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f16d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180aa140 0x7f9f180aa520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:57926/0 (socket says 172.21.15.107:57926) 2023-10-25T00:15:12.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f0e59c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9f180ac020 0x7f9f180c1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:54782/0 (socket says 172.21.15.107:54782) 2023-10-25T00:15:12.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1759e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9f180c26f0 0x7f9f1806c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35494/0 (socket says 172.21.15.107:35494) 2023-10-25T00:15:12.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f16d9d640 1 -- 172.21.15.107:0/4034055990 learned_addr learned my addr 172.21.15.107:0/4034055990 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:12.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1759e640 1 -- 172.21.15.107:0/4034055990 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180aa140 msgr2=0x7f9f180aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:12.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1759e640 1 --2- 172.21.15.107:0/4034055990 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180aa140 0x7f9f180aa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:12.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1759e640 1 -- 172.21.15.107:0/4034055990 >> v1:172.21.15.110:6789/0 conn(0x7f9f180ac020 legacy=0x7f9f180c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:12.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f1759e640 1 -- 172.21.15.107:0/4034055990 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f180c1220 con 0x7f9f180c26f0 2023-10-25T00:15:12.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.268+0000 7f9f16d9d640 1 --2- 172.21.15.107:0/4034055990 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180aa140 0x7f9f180aa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:15:12.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1759e640 1 --2- 172.21.15.107:0/4034055990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f180c26f0 0x7f9f1806c490 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000bfa0 tx=0x7f9f1001eb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90b1f486e39423fd server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:12.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1659c640 1 -- 172.21.15.107:0/4034055990 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1000b2d0 con 0x7f9f180c26f0 2023-10-25T00:15:12.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1659c640 1 -- 172.21.15.107:0/4034055990 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f100a79f0 con 0x7f9f180c26f0 2023-10-25T00:15:12.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1659c640 1 -- 172.21.15.107:0/4034055990 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f100a7d80 con 0x7f9f180c26f0 2023-10-25T00:15:12.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 -- 172.21.15.107:0/4034055990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f180c26f0 msgr2=0x7f9f1806c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:12.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 --2- 172.21.15.107:0/4034055990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f180c26f0 0x7f9f1806c490 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000bfa0 tx=0x7f9f1001eb20 comp rx=0 tx=0).stop 2023-10-25T00:15:12.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 -- 172.21.15.107:0/4034055990 shutdown_connections 2023-10-25T00:15:12.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 --2- 172.21.15.107:0/4034055990 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f180c26f0 0x7f9f1806c490 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:12.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 --2- 172.21.15.107:0/4034055990 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180aa140 0x7f9f180aa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:12.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 -- 172.21.15.107:0/4034055990 >> 172.21.15.107:0/4034055990 conn(0x7f9f180c65c0 msgr2=0x7f9f180c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:12.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 -- 172.21.15.107:0/4034055990 shutdown_connections 2023-10-25T00:15:12.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 -- 172.21.15.107:0/4034055990 wait complete. 2023-10-25T00:15:12.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 Processor -- start 2023-10-25T00:15:12.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 -- start start 2023-10-25T00:15:12.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f180aa140 0x7f9f181561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:12.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f180ac020 0x7f9f181ebbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:12.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180c26f0 0x7f9f181ec120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:12.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180a7f10 con 0x7f9f180aa140 2023-10-25T00:15:12.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181e8340 con 0x7f9f180ac020 2023-10-25T00:15:12.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1d14f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1814c420 con 0x7f9f180c26f0 2023-10-25T00:15:12.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f16d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f180aa140 0x7f9f181561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:12.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f16d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f180aa140 0x7f9f181561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35688/0 (socket says 172.21.15.107:35688) 2023-10-25T00:15:12.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f16d9d640 1 -- 172.21.15.107:0/20379347 learned_addr learned my addr 172.21.15.107:0/20379347 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:12.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f1759e640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180c26f0 0x7f9f181ec120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:12.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.272+0000 7f9f0e59c640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f180ac020 0x7f9f181ebbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:12.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f16d9d640 1 -- 172.21.15.107:0/20379347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180c26f0 msgr2=0x7f9f181ec120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:12.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f16d9d640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180c26f0 0x7f9f181ec120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:12.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f16d9d640 1 -- 172.21.15.107:0/20379347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f180ac020 msgr2=0x7f9f181ebbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:12.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f16d9d640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f180ac020 0x7f9f181ebbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:12.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f16d9d640 1 -- 172.21.15.107:0/20379347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f080094d0 con 0x7f9f180aa140 2023-10-25T00:15:12.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f16d9d640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f180aa140 0x7f9f181561b0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800e930 tx=0x7f9f0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:12.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f0ffff640 1 -- 172.21.15.107:0/20379347 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0800ca00 con 0x7f9f180aa140 2023-10-25T00:15:12.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f0ffff640 1 -- 172.21.15.107:0/20379347 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0800cb60 con 0x7f9f180aa140 2023-10-25T00:15:12.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f1d14f640 1 -- 172.21.15.107:0/20379347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1000bc80 con 0x7f9f180aa140 2023-10-25T00:15:12.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f0ffff640 1 -- 172.21.15.107:0/20379347 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08013470 con 0x7f9f180aa140 2023-10-25T00:15:12.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f1d14f640 1 -- 172.21.15.107:0/20379347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f181542d0 con 0x7f9f180aa140 2023-10-25T00:15:12.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f1d14f640 1 -- 172.21.15.107:0/20379347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f180beac0 con 0x7f9f180aa140 2023-10-25T00:15:12.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f0ffff640 1 -- 172.21.15.107:0/20379347 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9f080135d0 con 0x7f9f180aa140 2023-10-25T00:15:12.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f0ffff640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ef8065e60 0x7f9ef8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:12.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f0ffff640 1 -- 172.21.15.107:0/20379347 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7f9f08084970 con 0x7f9f180aa140 2023-10-25T00:15:12.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f0e59c640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ef8065e60 0x7f9ef8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:12.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.276+0000 7f9f0e59c640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ef8065e60 0x7f9ef8068320 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9f180c71f0 tx=0x7f9f00007cf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:12.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.280+0000 7f9f0ffff640 1 -- 172.21.15.107:0/20379347 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9f08051110 con 0x7f9f180aa140 2023-10-25T00:15:12.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.464+0000 7f9f1d14f640 1 -- 172.21.15.107:0/20379347 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f18154130 con 0x7f9f180aa140 2023-10-25T00:15:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f0ffff640 1 -- 172.21.15.107:0/20379347 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f9f18154130 con 0x7f9f180aa140 2023-10-25T00:15:12.467 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:12.467 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:12.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 -- 172.21.15.107:0/20379347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ef8065e60 msgr2=0x7f9ef8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:12.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ef8065e60 0x7f9ef8068320 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9f180c71f0 tx=0x7f9f00007cf0 comp rx=0 tx=0).stop 2023-10-25T00:15:12.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 -- 172.21.15.107:0/20379347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f180aa140 msgr2=0x7f9f181561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:12.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f180aa140 0x7f9f181561b0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800e930 tx=0x7f9f0800edf0 comp rx=0 tx=0).stop 2023-10-25T00:15:12.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 -- 172.21.15.107:0/20379347 shutdown_connections 2023-10-25T00:15:12.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9ef8065e60 0x7f9ef8068320 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:12.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9f180c26f0 0x7f9f181ec120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:12.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9f180ac020 0x7f9f181ebbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:12.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 --2- 172.21.15.107:0/20379347 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f180aa140 0x7f9f181561b0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:12.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 -- 172.21.15.107:0/20379347 >> 172.21.15.107:0/20379347 conn(0x7f9f180c65c0 msgr2=0x7f9f180c5080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:12.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 -- 172.21.15.107:0/20379347 shutdown_connections 2023-10-25T00:15:12.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:12.468+0000 7f9f1d14f640 1 -- 172.21.15.107:0/20379347 wait complete. 2023-10-25T00:15:12.485 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:15:13.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6f66c640 1 Processor -- start 2023-10-25T00:15:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6f66c640 1 -- start start 2023-10-25T00:15:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6f66c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce6814fd40 0x7fce680ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6f66c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce680a9820 0x7fce680aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6f66c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce6806ee40 0x7fce680c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:13.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6f66c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce680aa750 con 0x7fce6806ee40 2023-10-25T00:15:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6f66c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6806f730 con 0x7fce6814fd40 2023-10-25T00:15:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6f66c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce68156a40 con 0x7fce680a9820 2023-10-25T00:15:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6dc09640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce6806ee40 0x7fce680c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6dc09640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fce6806ee40 0x7fce680c1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58566/0 (socket says 172.21.15.107:58566) 2023-10-25T00:15:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6dc09640 1 -- 172.21.15.107:0/4086457195 learned_addr learned my addr 172.21.15.107:0/4086457195 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:13.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6d408640 1 --2- 172.21.15.107:0/4086457195 >> v2:172.21.15.110:3300/0 conn(0x7fce6814fd40 0x7fce680ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6cc07640 1 --2- 172.21.15.107:0/4086457195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce680a9820 0x7fce680aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6dc09640 1 -- 172.21.15.107:0/4086457195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce680a9820 msgr2=0x7fce680aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6dc09640 1 --2- 172.21.15.107:0/4086457195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce680a9820 0x7fce680aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6dc09640 1 -- 172.21.15.107:0/4086457195 >> v2:172.21.15.110:3300/0 conn(0x7fce6814fd40 msgr2=0x7fce680ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:13.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6dc09640 1 --2- 172.21.15.107:0/4086457195 >> v2:172.21.15.110:3300/0 conn(0x7fce6814fd40 0x7fce680ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:13.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.232+0000 7fce6dc09640 1 -- 172.21.15.107:0/4086457195 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce68156710 con 0x7fce6806ee40 2023-10-25T00:15:13.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce6dc09640 1 --2- 172.21.15.107:0/4086457195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6806ee40 0x7fce680c1860 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fce680c28b0 tx=0x7fce6001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9fc7b16962f04 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce5ffff640 1 -- 172.21.15.107:0/4086457195 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce600a56a0 con 0x7fce6806ee40 2023-10-25T00:15:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce5ffff640 1 -- 172.21.15.107:0/4086457195 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce600a5800 con 0x7fce6806ee40 2023-10-25T00:15:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce5ffff640 1 -- 172.21.15.107:0/4086457195 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce600a5b90 con 0x7fce6806ee40 2023-10-25T00:15:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce6f66c640 1 -- 172.21.15.107:0/4086457195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6806ee40 msgr2=0x7fce680c1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce6f66c640 1 --2- 172.21.15.107:0/4086457195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6806ee40 0x7fce680c1860 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fce680c28b0 tx=0x7fce6001c880 comp rx=0 tx=0).stop 2023-10-25T00:15:13.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce6f66c640 1 -- 172.21.15.107:0/4086457195 shutdown_connections 2023-10-25T00:15:13.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce6f66c640 1 --2- 172.21.15.107:0/4086457195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce6806ee40 0x7fce680c1860 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:13.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce6f66c640 1 --2- 172.21.15.107:0/4086457195 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce680a9820 0x7fce680aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:13.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce6f66c640 1 --2- 172.21.15.107:0/4086457195 >> v2:172.21.15.110:3300/0 conn(0x7fce6814fd40 0x7fce680ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:13.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce6f66c640 1 -- 172.21.15.107:0/4086457195 >> 172.21.15.107:0/4086457195 conn(0x7fce680c2950 msgr2=0x7fce680c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:13.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce6f66c640 1 -- 172.21.15.107:0/4086457195 shutdown_connections 2023-10-25T00:15:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.236+0000 7fce6f66c640 1 -- 172.21.15.107:0/4086457195 wait complete. 2023-10-25T00:15:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6f66c640 1 Processor -- start 2023-10-25T00:15:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6f66c640 1 -- start start 2023-10-25T00:15:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6f66c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce6806ee40 0x7fce680bb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:13.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6f66c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce6814fd40 0x7fce68155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:13.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6f66c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce680bf0c0 0x7fce680bf570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:13.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6f66c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6814b420 con 0x7fce680bf0c0 2023-10-25T00:15:13.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6f66c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce680bb900 con 0x7fce6806ee40 2023-10-25T00:15:13.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6f66c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce680bfab0 con 0x7fce6814fd40 2023-10-25T00:15:13.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6dc09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce680bf0c0 0x7fce680bf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:13.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6dc09640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce680bf0c0 0x7fce680bf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58582/0 (socket says 172.21.15.107:58582) 2023-10-25T00:15:13.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6dc09640 1 -- 172.21.15.107:0/3096167703 learned_addr learned my addr 172.21.15.107:0/3096167703 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:13.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6cc07640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce6814fd40 0x7fce68155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:13.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6d408640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce6806ee40 0x7fce680bb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:13.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6dc09640 1 -- 172.21.15.107:0/3096167703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce6814fd40 msgr2=0x7fce68155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:13.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6dc09640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce6814fd40 0x7fce68155760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:13.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6dc09640 1 -- 172.21.15.107:0/3096167703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce6806ee40 msgr2=0x7fce680bb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:13.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6dc09640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce6806ee40 0x7fce680bb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:13.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6dc09640 1 -- 172.21.15.107:0/3096167703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce60009b30 con 0x7fce680bf0c0 2023-10-25T00:15:13.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6d408640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce6806ee40 0x7fce680bb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:15:13.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6dc09640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce680bf0c0 0x7fce680bf570 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fce6001d120 tx=0x7fce60019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:13.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce5dffb640 1 -- 172.21.15.107:0/3096167703 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce600b6680 con 0x7fce680bf0c0 2023-10-25T00:15:13.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce5dffb640 1 -- 172.21.15.107:0/3096167703 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce600b6800 con 0x7fce680bf0c0 2023-10-25T00:15:13.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6f66c640 1 -- 172.21.15.107:0/3096167703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce680c0350 con 0x7fce680bf0c0 2023-10-25T00:15:13.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce5dffb640 1 -- 172.21.15.107:0/3096167703 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce600b6bb0 con 0x7fce680bf0c0 2023-10-25T00:15:13.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6f66c640 1 -- 172.21.15.107:0/3096167703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce680c0100 con 0x7fce680bf0c0 2023-10-25T00:15:13.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce6f66c640 1 -- 172.21.15.107:0/3096167703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce680c24a0 con 0x7fce680bf0c0 2023-10-25T00:15:13.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce5dffb640 1 -- 172.21.15.107:0/3096167703 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fce600b6d10 con 0x7fce680bf0c0 2023-10-25T00:15:13.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.240+0000 7fce5dffb640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce4c065e60 0x7fce4c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:13.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.244+0000 7fce5dffb640 1 -- 172.21.15.107:0/3096167703 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7fce601200a0 con 0x7fce680bf0c0 2023-10-25T00:15:13.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.244+0000 7fce6d408640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce4c065e60 0x7fce4c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:13.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.244+0000 7fce6d408640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce4c065e60 0x7fce4c068320 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fce681533c0 tx=0x7fce64008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:13.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.244+0000 7fce5dffb640 1 -- 172.21.15.107:0/3096167703 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fce600ec960 con 0x7fce680bf0c0 2023-10-25T00:15:13.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.432+0000 7fce6f66c640 1 -- 172.21.15.107:0/3096167703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce68070840 con 0x7fce680bf0c0 2023-10-25T00:15:13.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.432+0000 7fce5dffb640 1 -- 172.21.15.107:0/3096167703 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7fce600eff80 con 0x7fce680bf0c0 2023-10-25T00:15:13.433 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:13.433 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 -- 172.21.15.107:0/3096167703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce4c065e60 msgr2=0x7fce4c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce4c065e60 0x7fce4c068320 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fce681533c0 tx=0x7fce64008040 comp rx=0 tx=0).stop 2023-10-25T00:15:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 -- 172.21.15.107:0/3096167703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce680bf0c0 msgr2=0x7fce680bf570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce680bf0c0 0x7fce680bf570 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fce6001d120 tx=0x7fce60019620 comp rx=0 tx=0).stop 2023-10-25T00:15:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 -- 172.21.15.107:0/3096167703 shutdown_connections 2023-10-25T00:15:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce680bf0c0 0x7fce680bf570 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:13.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fce4c065e60 0x7fce4c068320 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce6814fd40 0x7fce68155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 --2- 172.21.15.107:0/3096167703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce6806ee40 0x7fce680bb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:13.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 -- 172.21.15.107:0/3096167703 >> 172.21.15.107:0/3096167703 conn(0x7fce680c2950 msgr2=0x7fce680c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 -- 172.21.15.107:0/3096167703 shutdown_connections 2023-10-25T00:15:13.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:13.436+0000 7fce6f66c640 1 -- 172.21.15.107:0/3096167703 wait complete. 2023-10-25T00:15:13.452 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:15:14.487 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:14.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd310463640 1 Processor -- start 2023-10-25T00:15:14.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd310463640 1 -- start start 2023-10-25T00:15:14.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd310463640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd30813bb00 0x7fd3081531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:14.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd310463640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd3080aa330 0x7fd30806e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd310463640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd308152d40 con 0x7fd3080a9450 2023-10-25T00:15:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd310463640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd308156100 con 0x7fd3080aa330 2023-10-25T00:15:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd310463640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3081527c0 con 0x7fd30813bb00 2023-10-25T00:15:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd30d9fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd3080aa330 0x7fd30806e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:14.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd30d9fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd3080aa330 0x7fd30806e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58592/0 (socket says 172.21.15.107:58592) 2023-10-25T00:15:14.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd30d9fe640 1 -- 172.21.15.107:0/3832611843 learned_addr learned my addr 172.21.15.107:0/3832611843 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:14.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd30e1ff640 1 --2- 172.21.15.107:0/3832611843 >> v2:172.21.15.110:3300/0 conn(0x7fd30813bb00 0x7fd3081531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:14.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd30d9fe640 1 -- 172.21.15.107:0/3832611843 >> v2:172.21.15.110:3300/0 conn(0x7fd30813bb00 msgr2=0x7fd3081531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:14.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd30d9fe640 1 --2- 172.21.15.107:0/3832611843 >> v2:172.21.15.110:3300/0 conn(0x7fd30813bb00 0x7fd3081531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:14.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd30d9fe640 1 -- 172.21.15.107:0/3832611843 >> v1:172.21.15.110:6789/0 conn(0x7fd3080a9450 legacy=0x7fd308071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:14.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.616+0000 7fd30d9fe640 1 -- 172.21.15.107:0/3832611843 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30806d660 con 0x7fd3080aa330 2023-10-25T00:15:14.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd30d9fe640 1 --2- 172.21.15.107:0/3832611843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3080aa330 0x7fd30806e250 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fd30806a6d0 tx=0x7fd2f801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa743383dbf74332 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:14.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd30d1fd640 1 -- 172.21.15.107:0/3832611843 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80ab600 con 0x7fd3080aa330 2023-10-25T00:15:14.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd30d1fd640 1 -- 172.21.15.107:0/3832611843 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f80a7050 con 0x7fd3080aa330 2023-10-25T00:15:14.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd30d1fd640 1 -- 172.21.15.107:0/3832611843 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80ab980 con 0x7fd3080aa330 2023-10-25T00:15:14.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd310463640 1 -- 172.21.15.107:0/3832611843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3080aa330 msgr2=0x7fd30806e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:14.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd310463640 1 --2- 172.21.15.107:0/3832611843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3080aa330 0x7fd30806e250 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fd30806a6d0 tx=0x7fd2f801c3a0 comp rx=0 tx=0).stop 2023-10-25T00:15:14.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd310463640 1 -- 172.21.15.107:0/3832611843 shutdown_connections 2023-10-25T00:15:14.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd310463640 1 --2- 172.21.15.107:0/3832611843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3080aa330 0x7fd30806e250 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:14.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd310463640 1 --2- 172.21.15.107:0/3832611843 >> v2:172.21.15.110:3300/0 conn(0x7fd30813bb00 0x7fd3081531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:14.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd310463640 1 -- 172.21.15.107:0/3832611843 >> 172.21.15.107:0/3832611843 conn(0x7fd308070f60 msgr2=0x7fd308071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:14.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd310463640 1 -- 172.21.15.107:0/3832611843 shutdown_connections 2023-10-25T00:15:14.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd310463640 1 -- 172.21.15.107:0/3832611843 wait complete. 2023-10-25T00:15:14.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd310463640 1 Processor -- start 2023-10-25T00:15:14.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.620+0000 7fd310463640 1 -- start start 2023-10-25T00:15:14.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd310463640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3080a9450 0x7fd3081511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:14.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd310463640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3080aa330 0x7fd308136310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:14.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd310463640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd30813bb00 0x7fd3081386b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:14.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd310463640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3081eeac0 con 0x7fd3080aa330 2023-10-25T00:15:14.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd310463640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3080aad60 con 0x7fd30813bb00 2023-10-25T00:15:14.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd310463640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3080ab040 con 0x7fd3080a9450 2023-10-25T00:15:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30d9fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3080aa330 0x7fd308136310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30d9fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3080aa330 0x7fd308136310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58598/0 (socket says 172.21.15.107:58598) 2023-10-25T00:15:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30d9fe640 1 -- 172.21.15.107:0/2553954744 learned_addr learned my addr 172.21.15.107:0/2553954744 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30ea00640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd30813bb00 0x7fd3081386b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:14.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30e1ff640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3080a9450 0x7fd3081511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:14.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30d9fe640 1 -- 172.21.15.107:0/2553954744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3080a9450 msgr2=0x7fd3081511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:14.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30d9fe640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3080a9450 0x7fd3081511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:14.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30d9fe640 1 -- 172.21.15.107:0/2553954744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd30813bb00 msgr2=0x7fd3081386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:14.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30d9fe640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd30813bb00 0x7fd3081386b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:14.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30d9fe640 1 -- 172.21.15.107:0/2553954744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80094d0 con 0x7fd3080aa330 2023-10-25T00:15:14.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30e1ff640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3080a9450 0x7fd3081511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:15:14.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30ea00640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd30813bb00 0x7fd3081386b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:15:14.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd30d9fe640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3080aa330 0x7fd308136310 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8019fd0 tx=0x7fd2f8019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:14.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd2feffd640 1 -- 172.21.15.107:0/2553954744 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80b37d0 con 0x7fd3080aa330 2023-10-25T00:15:14.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd2feffd640 1 -- 172.21.15.107:0/2553954744 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f80b3930 con 0x7fd3080aa330 2023-10-25T00:15:14.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd310463640 1 -- 172.21.15.107:0/2553954744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3080ab320 con 0x7fd3080aa330 2023-10-25T00:15:14.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd2feffd640 1 -- 172.21.15.107:0/2553954744 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80b3ca0 con 0x7fd3080aa330 2023-10-25T00:15:14.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd310463640 1 -- 172.21.15.107:0/2553954744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3081ea6c0 con 0x7fd3080aa330 2023-10-25T00:15:14.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.624+0000 7fd310463640 1 -- 172.21.15.107:0/2553954744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f0005260 con 0x7fd3080aa330 2023-10-25T00:15:14.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.628+0000 7fd2feffd640 1 -- 172.21.15.107:0/2553954744 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd2f80b2070 con 0x7fd3080aa330 2023-10-25T00:15:14.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.628+0000 7fd2feffd640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2ec065e60 0x7fd2ec068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:14.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.628+0000 7fd2feffd640 1 -- 172.21.15.107:0/2553954744 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7fd2f811f3b0 con 0x7fd3080aa330 2023-10-25T00:15:14.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.628+0000 7fd30e1ff640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2ec065e60 0x7fd2ec068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:14.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.628+0000 7fd30e1ff640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2ec065e60 0x7fd2ec068320 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd3080c22e0 tx=0x7fd304008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:14.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.632+0000 7fd2feffd640 1 -- 172.21.15.107:0/2553954744 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd2f80ebbd0 con 0x7fd3080aa330 2023-10-25T00:15:14.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 -- 172.21.15.107:0/2553954744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2f00057e0 con 0x7fd3080aa330 2023-10-25T00:15:14.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd2feffd640 1 -- 172.21.15.107:0/2553954744 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7fd2f80b0020 con 0x7fd3080aa330 2023-10-25T00:15:14.815 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:14.816 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 -- 172.21.15.107:0/2553954744 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2ec065e60 msgr2=0x7fd2ec068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2ec065e60 0x7fd2ec068320 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd3080c22e0 tx=0x7fd304008040 comp rx=0 tx=0).stop 2023-10-25T00:15:14.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 -- 172.21.15.107:0/2553954744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3080aa330 msgr2=0x7fd308136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3080aa330 0x7fd308136310 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8019fd0 tx=0x7fd2f8019140 comp rx=0 tx=0).stop 2023-10-25T00:15:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 -- 172.21.15.107:0/2553954744 shutdown_connections 2023-10-25T00:15:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd30813bb00 0x7fd3081386b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:14.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3080aa330 0x7fd308136310 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:14.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2ec065e60 0x7fd2ec068320 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:14.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 --2- 172.21.15.107:0/2553954744 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd3080a9450 0x7fd3081511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:14.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 -- 172.21.15.107:0/2553954744 >> 172.21.15.107:0/2553954744 conn(0x7fd308070f60 msgr2=0x7fd308071a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:14.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 -- 172.21.15.107:0/2553954744 shutdown_connections 2023-10-25T00:15:14.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:14.816+0000 7fd310463640 1 -- 172.21.15.107:0/2553954744 wait complete. 2023-10-25T00:15:14.832 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:15:15.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:15.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.580+0000 7f7bafbdc640 1 Processor -- start 2023-10-25T00:15:15.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.580+0000 7f7bafbdc640 1 -- start start 2023-10-25T00:15:15.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bafbdc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7ba806ee40 0x7f7ba80ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:15.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bafbdc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba806f730 con 0x7f7ba814fd40 2023-10-25T00:15:15.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bafbdc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba8156a40 con 0x7f7ba80a9820 2023-10-25T00:15:15.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bafbdc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba8156710 con 0x7f7ba806ee40 2023-10-25T00:15:15.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bad978640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7ba814fd40 0x7f7ba80c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46074/0 (socket says 172.21.15.107:46074) 2023-10-25T00:15:15.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bad978640 1 -- 172.21.15.107:0/2563416402 learned_addr learned my addr 172.21.15.107:0/2563416402 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:15.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bae179640 1 --2- 172.21.15.107:0/2563416402 >> v2:172.21.15.110:3300/0 conn(0x7f7ba806ee40 0x7f7ba80ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:15.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239778376 0 0) 0x7f7ba806f730 con 0x7f7ba814fd40 2023-10-25T00:15:15.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b94003530 con 0x7f7ba814fd40 2023-10-25T00:15:15.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 948641228 0 0) 0x7f7b94003530 con 0x7f7ba814fd40 2023-10-25T00:15:15.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 >> v2:172.21.15.110:3300/0 conn(0x7f7ba806ee40 msgr2=0x7f7ba80ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:15.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 --2- 172.21.15.107:0/2563416402 >> v2:172.21.15.110:3300/0 conn(0x7f7ba806ee40 0x7f7ba80ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:15.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 >> v1:172.21.15.110:6789/0 conn(0x7f7ba80a9820 legacy=0x7f7ba815af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba806f980 con 0x7f7ba814fd40 2023-10-25T00:15:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f7b98001680 con 0x7f7ba814fd40 2023-10-25T00:15:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 >> v1:172.21.15.107:6789/0 conn(0x7f7ba814fd40 legacy=0x7f7ba80c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 --2- 172.21.15.107:0/2563416402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b94003980 0x7f7b94003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 --2- 172.21.15.107:0/2563416402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b940047d0 0x7f7b94004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 --2- 172.21.15.107:0/2563416402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 0x7f7b9400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8156710 con 0x7f7b940052b0 2023-10-25T00:15:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9400c180 con 0x7f7b940047d0 2023-10-25T00:15:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9400c340 con 0x7f7b94003980 2023-10-25T00:15:15.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bae179640 1 --2- 172.21.15.107:0/2563416402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 0x7f7b9400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bae179640 1 -- 172.21.15.107:0/2563416402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b94003980 msgr2=0x7f7b94003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bae179640 1 --2- 172.21.15.107:0/2563416402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b94003980 0x7f7b94003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bae179640 1 -- 172.21.15.107:0/2563416402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b940047d0 msgr2=0x7f7b94004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bae179640 1 --2- 172.21.15.107:0/2563416402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b940047d0 0x7f7b94004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bae179640 1 -- 172.21.15.107:0/2563416402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b94003530 con 0x7f7b940052b0 2023-10-25T00:15:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bae179640 1 -- 172.21.15.107:0/2563416402 reap_dead start 2023-10-25T00:15:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.584+0000 7f7bae179640 1 --2- 172.21.15.107:0/2563416402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 0x7f7b9400b990 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f7ba80725f0 tx=0x7f7ba001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d476cefc389bb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0003ba0 con 0x7f7b940052b0 2023-10-25T00:15:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba000f040 con 0x7f7b940052b0 2023-10-25T00:15:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bac976640 1 -- 172.21.15.107:0/2563416402 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba00a5950 con 0x7f7b940052b0 2023-10-25T00:15:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 -- 172.21.15.107:0/2563416402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 msgr2=0x7f7b9400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 --2- 172.21.15.107:0/2563416402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 0x7f7b9400b990 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f7ba80725f0 tx=0x7f7ba001cc40 comp rx=0 tx=0).stop 2023-10-25T00:15:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 -- 172.21.15.107:0/2563416402 shutdown_connections 2023-10-25T00:15:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 --2- 172.21.15.107:0/2563416402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 0x7f7b9400b990 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 -- 172.21.15.107:0/2563416402 >> 172.21.15.107:0/2563416402 conn(0x7f7ba80c2950 msgr2=0x7f7ba80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 -- 172.21.15.107:0/2563416402 shutdown_connections 2023-10-25T00:15:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 -- 172.21.15.107:0/2563416402 wait complete. 2023-10-25T00:15:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 Processor -- start 2023-10-25T00:15:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 -- start start 2023-10-25T00:15:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 0x7f7ba80bf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ba814ee20 0x7f7ba8141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba8141b80 0x7f7ba806a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8154380 con 0x7f7b940052b0 2023-10-25T00:15:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8154c20 con 0x7f7ba8141b80 2023-10-25T00:15:15.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8142000 con 0x7f7ba814ee20 2023-10-25T00:15:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bad978640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 0x7f7ba80bf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bad978640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 0x7f7ba80bf4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58614/0 (socket says 172.21.15.107:58614) 2023-10-25T00:15:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bad978640 1 -- 172.21.15.107:0/1760100972 learned_addr learned my addr 172.21.15.107:0/1760100972 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bad177640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ba814ee20 0x7f7ba8141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bae179640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba8141b80 0x7f7ba806a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bad978640 1 -- 172.21.15.107:0/1760100972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ba814ee20 msgr2=0x7f7ba8141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bad978640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ba814ee20 0x7f7ba8141640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bad978640 1 -- 172.21.15.107:0/1760100972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba8141b80 msgr2=0x7f7ba806a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bad978640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba8141b80 0x7f7ba806a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bad978640 1 -- 172.21.15.107:0/1760100972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba814f3c0 con 0x7f7b940052b0 2023-10-25T00:15:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bad978640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 0x7f7ba80bf4e0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800c650 tx=0x7f7b9800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7b9e7fc640 1 -- 172.21.15.107:0/1760100972 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b980072d0 con 0x7f7b940052b0 2023-10-25T00:15:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7b9e7fc640 1 -- 172.21.15.107:0/1760100972 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b98007450 con 0x7f7b940052b0 2023-10-25T00:15:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 -- 172.21.15.107:0/1760100972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba81545d0 con 0x7f7b940052b0 2023-10-25T00:15:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7b9e7fc640 1 -- 172.21.15.107:0/1760100972 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b980047f0 con 0x7f7b940052b0 2023-10-25T00:15:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.588+0000 7f7bafbdc640 1 -- 172.21.15.107:0/1760100972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba80c14f0 con 0x7f7b940052b0 2023-10-25T00:15:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.592+0000 7f7bafbdc640 1 -- 172.21.15.107:0/1760100972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba80c24a0 con 0x7f7b940052b0 2023-10-25T00:15:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.592+0000 7f7b9e7fc640 1 -- 172.21.15.107:0/1760100972 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7b98004950 con 0x7f7b940052b0 2023-10-25T00:15:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.592+0000 7f7b9e7fc640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b8c065e60 0x7f7b8c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.592+0000 7f7b9e7fc640 1 -- 172.21.15.107:0/1760100972 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7f7b98084150 con 0x7f7b940052b0 2023-10-25T00:15:15.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.592+0000 7f7bad177640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b8c065e60 0x7f7b8c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:15.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.592+0000 7f7bad177640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b8c065e60 0x7f7b8c068320 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7ba806b6b0 tx=0x7f7ba4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:15.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.596+0000 7f7b9e7fc640 1 -- 172.21.15.107:0/1760100972 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7b98050880 con 0x7f7b940052b0 2023-10-25T00:15:15.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 -- 172.21.15.107:0/1760100972 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ba806ad20 con 0x7f7b940052b0 2023-10-25T00:15:15.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7b9e7fc640 1 -- 172.21.15.107:0/1760100972 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f7b98018070 con 0x7f7b940052b0 2023-10-25T00:15:15.771 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:15.771 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:15.771 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:15.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 -- 172.21.15.107:0/1760100972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b8c065e60 msgr2=0x7f7b8c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:15.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b8c065e60 0x7f7b8c068320 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7ba806b6b0 tx=0x7f7ba4009040 comp rx=0 tx=0).stop 2023-10-25T00:15:15.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 -- 172.21.15.107:0/1760100972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 msgr2=0x7f7ba80bf4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:15.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 0x7f7ba80bf4e0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800c650 tx=0x7f7b9800cb10 comp rx=0 tx=0).stop 2023-10-25T00:15:15.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 -- 172.21.15.107:0/1760100972 shutdown_connections 2023-10-25T00:15:15.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ba8141b80 0x7f7ba806a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:15.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b8c065e60 0x7f7b8c068320 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:15.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ba814ee20 0x7f7ba8141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:15.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 --2- 172.21.15.107:0/1760100972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b940052b0 0x7f7ba80bf4e0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:15.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 -- 172.21.15.107:0/1760100972 >> 172.21.15.107:0/1760100972 conn(0x7f7ba80c2950 msgr2=0x7f7ba80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:15.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 -- 172.21.15.107:0/1760100972 shutdown_connections 2023-10-25T00:15:15.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:15.772+0000 7f7bafbdc640 1 -- 172.21.15.107:0/1760100972 wait complete. 2023-10-25T00:15:15.787 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:15:16.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:16.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.960+0000 7fbd0e306640 1 Processor -- start 2023-10-25T00:15:16.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.960+0000 7fbd0e306640 1 -- start start 2023-10-25T00:15:16.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd0e306640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd0814fd40 0x7fbd080ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:16.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd0e306640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd080a9820 0x7fbd080aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:16.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd0e306640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0806f730 con 0x7fbd0806ee40 2023-10-25T00:15:16.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd0e306640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd080aa750 con 0x7fbd0814fd40 2023-10-25T00:15:16.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd0e306640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd08156a40 con 0x7fbd080a9820 2023-10-25T00:15:16.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd07fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd0814fd40 0x7fbd080ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:16.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd07fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbd0814fd40 0x7fbd080ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58624/0 (socket says 172.21.15.107:58624) 2023-10-25T00:15:16.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd07fff640 1 -- 172.21.15.107:0/1420265955 learned_addr learned my addr 172.21.15.107:0/1420265955 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:16.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd077fe640 1 --2- 172.21.15.107:0/1420265955 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd080a9820 0x7fbd080aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:16.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd07fff640 1 -- 172.21.15.107:0/1420265955 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd080a9820 msgr2=0x7fbd080aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:16.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd07fff640 1 --2- 172.21.15.107:0/1420265955 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd080a9820 0x7fbd080aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:16.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd07fff640 1 -- 172.21.15.107:0/1420265955 >> v1:172.21.15.110:6789/0 conn(0x7fbd0806ee40 legacy=0x7fbd080c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:16.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd07fff640 1 -- 172.21.15.107:0/1420265955 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd08156710 con 0x7fbd0814fd40 2023-10-25T00:15:16.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd07fff640 1 --2- 172.21.15.107:0/1420265955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd0814fd40 0x7fbd080ab090 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009870 tx=0x7fbcf801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=589d71b12eb2134e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:16.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.964+0000 7fbd06ffd640 1 -- 172.21.15.107:0/1420265955 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf80ac640 con 0x7fbd0814fd40 2023-10-25T00:15:16.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd06ffd640 1 -- 172.21.15.107:0/1420265955 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf80ac7a0 con 0x7fbd0814fd40 2023-10-25T00:15:16.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd06ffd640 1 -- 172.21.15.107:0/1420265955 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf80acb30 con 0x7fbd0814fd40 2023-10-25T00:15:16.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 -- 172.21.15.107:0/1420265955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd0814fd40 msgr2=0x7fbd080ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:16.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 --2- 172.21.15.107:0/1420265955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd0814fd40 0x7fbd080ab090 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009870 tx=0x7fbcf801c3b0 comp rx=0 tx=0).stop 2023-10-25T00:15:16.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 -- 172.21.15.107:0/1420265955 shutdown_connections 2023-10-25T00:15:16.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 --2- 172.21.15.107:0/1420265955 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd080a9820 0x7fbd080aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:16.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 --2- 172.21.15.107:0/1420265955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd0814fd40 0x7fbd080ab090 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:16.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 -- 172.21.15.107:0/1420265955 >> 172.21.15.107:0/1420265955 conn(0x7fbd080c2950 msgr2=0x7fbd080c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:16.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 -- 172.21.15.107:0/1420265955 shutdown_connections 2023-10-25T00:15:16.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 -- 172.21.15.107:0/1420265955 wait complete. 2023-10-25T00:15:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 Processor -- start 2023-10-25T00:15:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 -- start start 2023-10-25T00:15:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd0806ee40 0x7fbd081ea720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:16.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd0814fd40 0x7fbd0815b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:16.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd0806ba90 0x7fbd081f0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:16.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0806b400 con 0x7fbd0814fd40 2023-10-25T00:15:16.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd08069e70 con 0x7fbd0806ee40 2023-10-25T00:15:16.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0e306640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd081edf20 con 0x7fbd0806ba90 2023-10-25T00:15:16.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd0814fd40 0x7fbd0815b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:16.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd0814fd40 0x7fbd0815b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58636/0 (socket says 172.21.15.107:58636) 2023-10-25T00:15:16.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd077fe640 1 -- 172.21.15.107:0/2188717312 learned_addr learned my addr 172.21.15.107:0/2188717312 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:16.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd07fff640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd0806ee40 0x7fbd081ea720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:16.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd0c8a3640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd0806ba90 0x7fbd081f0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:16.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd077fe640 1 -- 172.21.15.107:0/2188717312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd0806ba90 msgr2=0x7fbd081f0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd077fe640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd0806ba90 0x7fbd081f0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd077fe640 1 -- 172.21.15.107:0/2188717312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd0806ee40 msgr2=0x7fbd081ea720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd077fe640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd0806ee40 0x7fbd081ea720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd077fe640 1 -- 172.21.15.107:0/2188717312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf80094d0 con 0x7fbd0814fd40 2023-10-25T00:15:16.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd077fe640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd0814fd40 0x7fbd0815b340 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00c7f0 tx=0x7fbcfc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.968+0000 7fbd04ff9640 1 -- 172.21.15.107:0/2188717312 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc004220 con 0x7fbd0814fd40 2023-10-25T00:15:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.972+0000 7fbd04ff9640 1 -- 172.21.15.107:0/2188717312 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcfc0043a0 con 0x7fbd0814fd40 2023-10-25T00:15:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.972+0000 7fbd0e306640 1 -- 172.21.15.107:0/2188717312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0814f020 con 0x7fbd0814fd40 2023-10-25T00:15:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.972+0000 7fbd04ff9640 1 -- 172.21.15.107:0/2188717312 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc004510 con 0x7fbd0814fd40 2023-10-25T00:15:16.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.972+0000 7fbd0e306640 1 -- 172.21.15.107:0/2188717312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0814f300 con 0x7fbd0814fd40 2023-10-25T00:15:16.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.972+0000 7fbd0e306640 1 -- 172.21.15.107:0/2188717312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd080c24a0 con 0x7fbd0814fd40 2023-10-25T00:15:16.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.972+0000 7fbd04ff9640 1 -- 172.21.15.107:0/2188717312 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbcfc003a70 con 0x7fbd0814fd40 2023-10-25T00:15:16.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.972+0000 7fbd04ff9640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbcf4065e90 0x7fbcf4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:16.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.972+0000 7fbd07fff640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbcf4065e90 0x7fbcf4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:16.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.972+0000 7fbd04ff9640 1 -- 172.21.15.107:0/2188717312 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7fbcfc084610 con 0x7fbd0814fd40 2023-10-25T00:15:16.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.972+0000 7fbd07fff640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbcf4065e90 0x7fbcf4068350 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80ae000 tx=0x7fbcf8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:16.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:16.976+0000 7fbd04ff9640 1 -- 172.21.15.107:0/2188717312 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbcfc050db0 con 0x7fbd0814fd40 2023-10-25T00:15:17.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.160+0000 7fbd0e306640 1 -- 172.21.15.107:0/2188717312 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd08070840 con 0x7fbd0814fd40 2023-10-25T00:15:17.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd04ff9640 1 -- 172.21.15.107:0/2188717312 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7fbcfc01e020 con 0x7fbd0814fd40 2023-10-25T00:15:17.163 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:17.164 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:17.165 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:17.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 -- 172.21.15.107:0/2188717312 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbcf4065e90 msgr2=0x7fbcf4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:17.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbcf4065e90 0x7fbcf4068350 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80ae000 tx=0x7fbcf8008040 comp rx=0 tx=0).stop 2023-10-25T00:15:17.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 -- 172.21.15.107:0/2188717312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd0814fd40 msgr2=0x7fbd0815b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:17.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd0814fd40 0x7fbd0815b340 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00c7f0 tx=0x7fbcfc00ccb0 comp rx=0 tx=0).stop 2023-10-25T00:15:17.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 -- 172.21.15.107:0/2188717312 shutdown_connections 2023-10-25T00:15:17.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbcf4065e90 0x7fbcf4068350 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:17.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbd0806ba90 0x7fbd081f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:17.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd0814fd40 0x7fbd0815b340 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:17.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 --2- 172.21.15.107:0/2188717312 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbd0806ee40 0x7fbd081ea720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:17.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 -- 172.21.15.107:0/2188717312 >> 172.21.15.107:0/2188717312 conn(0x7fbd080c2950 msgr2=0x7fbd080c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:17.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 -- 172.21.15.107:0/2188717312 shutdown_connections 2023-10-25T00:15:17.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.164+0000 7fbd0e306640 1 -- 172.21.15.107:0/2188717312 wait complete. 2023-10-25T00:15:17.180 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:15:17.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:17.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fe48e5640 1 Processor -- start 2023-10-25T00:15:17.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fe48e5640 1 -- start start 2023-10-25T00:15:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fe48e5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1fe014c470 0x7f1fe014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fe48e5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fe006f490 0x7f1fe0071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fe48e5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe0155cd0 con 0x7f1fe0071fc0 2023-10-25T00:15:17.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fe48e5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe01526e0 con 0x7f1fe006f490 2023-10-25T00:15:17.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fe48e5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe00c4380 con 0x7f1fe014c470 2023-10-25T00:15:17.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fddd9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fe006f490 0x7f1fe0071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:17.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fddd9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1fe006f490 0x7f1fe0071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58644/0 (socket says 172.21.15.107:58644) 2023-10-25T00:15:17.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fddd9b640 1 -- 172.21.15.107:0/2840875759 learned_addr learned my addr 172.21.15.107:0/2840875759 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:17.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fde59c640 1 --2- 172.21.15.107:0/2840875759 >> v2:172.21.15.110:3300/0 conn(0x7f1fe014c470 0x7f1fe014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:17.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fddd9b640 1 -- 172.21.15.107:0/2840875759 >> v2:172.21.15.110:3300/0 conn(0x7f1fe014c470 msgr2=0x7f1fe014c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:17.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fddd9b640 1 --2- 172.21.15.107:0/2840875759 >> v2:172.21.15.110:3300/0 conn(0x7f1fe014c470 0x7f1fe014c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:17.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fddd9b640 1 -- 172.21.15.107:0/2840875759 >> v1:172.21.15.110:6789/0 conn(0x7f1fe0071fc0 legacy=0x7f1fe0152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:17.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fddd9b640 1 -- 172.21.15.107:0/2840875759 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00c48b0 con 0x7f1fe006f490 2023-10-25T00:15:17.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fde59c640 1 --2- 172.21.15.107:0/2840875759 >> v2:172.21.15.110:3300/0 conn(0x7f1fe014c470 0x7f1fe014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:15:17.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.920+0000 7f1fddd9b640 1 --2- 172.21.15.107:0/2840875759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe006f490 0x7f1fe0071a80 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f1fe006a8e0 tx=0x7f1fd401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7608200846f32a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:17.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fdd59a640 1 -- 172.21.15.107:0/2840875759 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40a5530 con 0x7f1fe006f490 2023-10-25T00:15:17.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fdd59a640 1 -- 172.21.15.107:0/2840875759 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd40a5690 con 0x7f1fe006f490 2023-10-25T00:15:17.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fdd59a640 1 -- 172.21.15.107:0/2840875759 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40a5a20 con 0x7f1fe006f490 2023-10-25T00:15:17.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 -- 172.21.15.107:0/2840875759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe006f490 msgr2=0x7f1fe0071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:17.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 --2- 172.21.15.107:0/2840875759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe006f490 0x7f1fe0071a80 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f1fe006a8e0 tx=0x7f1fd401c870 comp rx=0 tx=0).stop 2023-10-25T00:15:17.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 -- 172.21.15.107:0/2840875759 shutdown_connections 2023-10-25T00:15:17.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 --2- 172.21.15.107:0/2840875759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe006f490 0x7f1fe0071a80 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:17.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 --2- 172.21.15.107:0/2840875759 >> v2:172.21.15.110:3300/0 conn(0x7f1fe014c470 0x7f1fe014c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:17.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 -- 172.21.15.107:0/2840875759 >> 172.21.15.107:0/2840875759 conn(0x7f1fe0070f60 msgr2=0x7f1fe0071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:17.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 -- 172.21.15.107:0/2840875759 shutdown_connections 2023-10-25T00:15:17.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 -- 172.21.15.107:0/2840875759 wait complete. 2023-10-25T00:15:17.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 Processor -- start 2023-10-25T00:15:17.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 -- start start 2023-10-25T00:15:17.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe006f490 0x7f1fe0155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:17.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fe0071fc0 0x7f1fe0140b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:17.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fe014c470 0x7f1fe0141050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:17.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe0142050 con 0x7f1fe006f490 2023-10-25T00:15:17.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe0141590 con 0x7f1fe014c470 2023-10-25T00:15:17.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe006c630 con 0x7f1fe0071fc0 2023-10-25T00:15:17.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fde59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe006f490 0x7f1fe0155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:17.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fde59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe006f490 0x7f1fe0155420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58648/0 (socket says 172.21.15.107:58648) 2023-10-25T00:15:17.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fde59c640 1 -- 172.21.15.107:0/4044991661 learned_addr learned my addr 172.21.15.107:0/4044991661 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:17.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fddd9b640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fe0071fc0 0x7f1fe0140b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:17.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fde59c640 1 -- 172.21.15.107:0/4044991661 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fe0071fc0 msgr2=0x7f1fe0140b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:17.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fded9d640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fe014c470 0x7f1fe0141050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:17.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fde59c640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fe0071fc0 0x7f1fe0140b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:17.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fde59c640 1 -- 172.21.15.107:0/4044991661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fe014c470 msgr2=0x7f1fe0141050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:17.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fde59c640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fe014c470 0x7f1fe0141050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:17.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fde59c640 1 -- 172.21.15.107:0/4044991661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd4009b30 con 0x7f1fe006f490 2023-10-25T00:15:17.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fde59c640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe006f490 0x7f1fe0155420 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c820 tx=0x7f1fd000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:17.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fceffd640 1 -- 172.21.15.107:0/4044991661 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0007a30 con 0x7f1fe006f490 2023-10-25T00:15:17.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fceffd640 1 -- 172.21.15.107:0/4044991661 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd0007b90 con 0x7f1fe006f490 2023-10-25T00:15:17.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 -- 172.21.15.107:0/4044991661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe0141d70 con 0x7f1fe006f490 2023-10-25T00:15:17.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fceffd640 1 -- 172.21.15.107:0/4044991661 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd000ce10 con 0x7f1fe006f490 2023-10-25T00:15:17.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.924+0000 7f1fe48e5640 1 -- 172.21.15.107:0/4044991661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe006c8b0 con 0x7f1fe006f490 2023-10-25T00:15:17.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.928+0000 7f1fe48e5640 1 -- 172.21.15.107:0/4044991661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe0141780 con 0x7f1fe006f490 2023-10-25T00:15:17.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.928+0000 7f1fceffd640 1 -- 172.21.15.107:0/4044991661 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1fd00113f0 con 0x7f1fe006f490 2023-10-25T00:15:17.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.928+0000 7f1fceffd640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fc0065e60 0x7f1fc0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:17.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.928+0000 7f1fddd9b640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fc0065e60 0x7f1fc0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:17.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.928+0000 7f1fceffd640 1 -- 172.21.15.107:0/4044991661 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 9885+0+0 (secure 0 0 0) 0x7f1fd00847c0 con 0x7f1fe006f490 2023-10-25T00:15:17.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.928+0000 7f1fddd9b640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fc0065e60 0x7f1fc0068320 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1fd40ae000 tx=0x7f1fd40a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:17.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:17.932+0000 7f1fceffd640 1 -- 172.21.15.107:0/4044991661 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1fe0141780 con 0x7f1fe006f490 2023-10-25T00:15:18.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.116+0000 7f1fe48e5640 1 -- 172.21.15.107:0/4044991661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fe0141780 con 0x7f1fe006f490 2023-10-25T00:15:18.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fceffd640 1 -- 172.21.15.107:0/4044991661 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f1fe0141780 con 0x7f1fe006f490 2023-10-25T00:15:18.119 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:18.119 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120},{"gid":7030,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:13:21.937931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":171,"state":"up:active","state_seq":13,"addr":"172.21.15.107:6837/175581297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":175581297},{"type":"v1","addr":"172.21.15.107:6837","nonce":175581297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:18.119 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 180 2023-10-25T00:15:18.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 -- 172.21.15.107:0/4044991661 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fc0065e60 msgr2=0x7f1fc0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:18.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fc0065e60 0x7f1fc0068320 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1fd40ae000 tx=0x7f1fd40a7070 comp rx=0 tx=0).stop 2023-10-25T00:15:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 -- 172.21.15.107:0/4044991661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe006f490 msgr2=0x7f1fe0155420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe006f490 0x7f1fe0155420 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000c820 tx=0x7f1fd000cce0 comp rx=0 tx=0).stop 2023-10-25T00:15:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 -- 172.21.15.107:0/4044991661 shutdown_connections 2023-10-25T00:15:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1fe014c470 0x7f1fe0141050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:18.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1fc0065e60 0x7f1fc0068320 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:18.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1fe0071fc0 0x7f1fe0140b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:18.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 --2- 172.21.15.107:0/4044991661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1fe006f490 0x7f1fe0155420 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:18.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 -- 172.21.15.107:0/4044991661 >> 172.21.15.107:0/4044991661 conn(0x7f1fe0070f60 msgr2=0x7f1fe00c20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:18.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 -- 172.21.15.107:0/4044991661 shutdown_connections 2023-10-25T00:15:18.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:18.120+0000 7f1fe48e5640 1 -- 172.21.15.107:0/4044991661 wait complete. 2023-10-25T00:15:18.137 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:15:19.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:19.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.320+0000 7f7ecbd15640 1 Processor -- start 2023-10-25T00:15:19.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.320+0000 7f7ecbd15640 1 -- start start 2023-10-25T00:15:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.320+0000 7f7ecbd15640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7ec4156770 0x7f7ec40ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.320+0000 7f7ecbd15640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7ec40c70e0 0x7f7ec40bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.320+0000 7f7ecbd15640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec40c1ca0 con 0x7f7ec40c2130 2023-10-25T00:15:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ecbd15640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec40c27e0 con 0x7f7ec40c70e0 2023-10-25T00:15:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ecbd15640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec40c10c0 con 0x7f7ec4156770 2023-10-25T00:15:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ec92b0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7ec40c70e0 0x7f7ec40bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:19.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7eca2b2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7ec40c2130 0x7f7ec4069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46076/0 (socket says 172.21.15.107:46076) 2023-10-25T00:15:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7eca2b2640 1 -- 172.21.15.107:0/2545812527 learned_addr learned my addr 172.21.15.107:0/2545812527 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ec9ab1640 1 --2- 172.21.15.107:0/2545812527 >> v2:172.21.15.110:3300/0 conn(0x7f7ec4156770 0x7f7ec40ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ec92b0640 1 -- 172.21.15.107:0/2545812527 >> v2:172.21.15.110:3300/0 conn(0x7f7ec4156770 msgr2=0x7f7ec40ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ec92b0640 1 --2- 172.21.15.107:0/2545812527 >> v2:172.21.15.110:3300/0 conn(0x7f7ec4156770 0x7f7ec40ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ec92b0640 1 -- 172.21.15.107:0/2545812527 >> v1:172.21.15.107:6789/0 conn(0x7f7ec40c2130 legacy=0x7f7ec4069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ec92b0640 1 -- 172.21.15.107:0/2545812527 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec40c13a0 con 0x7f7ec40c70e0 2023-10-25T00:15:19.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ec8aaf640 1 -- 172.21.15.107:0/2545812527 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695835457 0 0) 0x7f7ec40c1ca0 con 0x7f7ec40c2130 2023-10-25T00:15:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ec92b0640 1 --2- 172.21.15.107:0/2545812527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec40c70e0 0x7f7ec40bdfa0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40ac6f0 tx=0x7f7eb401c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73e7b1c111246f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ec8aaf640 1 -- 172.21.15.107:0/2545812527 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb40ab400 con 0x7f7ec40c70e0 2023-10-25T00:15:19.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ec8aaf640 1 -- 172.21.15.107:0/2545812527 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb40a7050 con 0x7f7ec40c70e0 2023-10-25T00:15:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ec8aaf640 1 -- 172.21.15.107:0/2545812527 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb40ab780 con 0x7f7ec40c70e0 2023-10-25T00:15:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ecbd15640 1 -- 172.21.15.107:0/2545812527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec40c70e0 msgr2=0x7f7ec40bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ecbd15640 1 --2- 172.21.15.107:0/2545812527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec40c70e0 0x7f7ec40bdfa0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40ac6f0 tx=0x7f7eb401c310 comp rx=0 tx=0).stop 2023-10-25T00:15:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ecbd15640 1 -- 172.21.15.107:0/2545812527 shutdown_connections 2023-10-25T00:15:19.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ecbd15640 1 --2- 172.21.15.107:0/2545812527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec40c70e0 0x7f7ec40bdfa0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ecbd15640 1 --2- 172.21.15.107:0/2545812527 >> v2:172.21.15.110:3300/0 conn(0x7f7ec4156770 0x7f7ec40ab7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ecbd15640 1 -- 172.21.15.107:0/2545812527 >> 172.21.15.107:0/2545812527 conn(0x7f7ec40c65c0 msgr2=0x7f7ec40c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ecbd15640 1 -- 172.21.15.107:0/2545812527 shutdown_connections 2023-10-25T00:15:19.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.324+0000 7f7ecbd15640 1 -- 172.21.15.107:0/2545812527 wait complete. 2023-10-25T00:15:19.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ecbd15640 1 Processor -- start 2023-10-25T00:15:19.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ecbd15640 1 -- start start 2023-10-25T00:15:19.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ecbd15640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ec40c2130 0x7f7ec409ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ecbd15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ec40c70e0 0x7f7ec409c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ecbd15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec4156770 0x7f7ec40a7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ecbd15640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec40ab5c0 con 0x7f7ec4156770 2023-10-25T00:15:19.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ecbd15640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec40abaf0 con 0x7f7ec40c2130 2023-10-25T00:15:19.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ecbd15640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec4151d00 con 0x7f7ec40c70e0 2023-10-25T00:15:19.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eca2b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec4156770 0x7f7ec40a7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:19.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eca2b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec4156770 0x7f7ec40a7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58668/0 (socket says 172.21.15.107:58668) 2023-10-25T00:15:19.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eca2b2640 1 -- 172.21.15.107:0/1680363104 learned_addr learned my addr 172.21.15.107:0/1680363104 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:19.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ec9ab1640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ec40c2130 0x7f7ec409ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:19.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ec92b0640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ec40c70e0 0x7f7ec409c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:19.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eca2b2640 1 -- 172.21.15.107:0/1680363104 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ec40c70e0 msgr2=0x7f7ec409c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eca2b2640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ec40c70e0 0x7f7ec409c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eca2b2640 1 -- 172.21.15.107:0/1680363104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ec40c2130 msgr2=0x7f7ec409ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eca2b2640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ec40c2130 0x7f7ec409ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eca2b2640 1 -- 172.21.15.107:0/1680363104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4009420 con 0x7f7ec4156770 2023-10-25T00:15:19.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ec9ab1640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ec40c2130 0x7f7ec409ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:15:19.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eca2b2640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec4156770 0x7f7ec40a7790 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc001570 tx=0x7f7ebc00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:19.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eba7fc640 1 -- 172.21.15.107:0/1680363104 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc00a640 con 0x7f7ec4156770 2023-10-25T00:15:19.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eba7fc640 1 -- 172.21.15.107:0/1680363104 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc00a7c0 con 0x7f7ec4156770 2023-10-25T00:15:19.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ecbd15640 1 -- 172.21.15.107:0/1680363104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec40ab810 con 0x7f7ec4156770 2023-10-25T00:15:19.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7eba7fc640 1 -- 172.21.15.107:0/1680363104 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc00ab70 con 0x7f7ec4156770 2023-10-25T00:15:19.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.328+0000 7f7ecbd15640 1 -- 172.21.15.107:0/1680363104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec409b810 con 0x7f7ec4156770 2023-10-25T00:15:19.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.332+0000 7f7ecbd15640 1 -- 172.21.15.107:0/1680363104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec40746e0 con 0x7f7ec4156770 2023-10-25T00:15:19.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.332+0000 7f7eba7fc640 1 -- 172.21.15.107:0/1680363104 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7ebc00acd0 con 0x7f7ec4156770 2023-10-25T00:15:19.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.332+0000 7f7eba7fc640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ea8065e60 0x7f7ea8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.332+0000 7f7eba7fc640 1 -- 172.21.15.107:0/1680363104 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 9971+0+0 (secure 0 0 0) 0x7f7ebc083fc0 con 0x7f7ec4156770 2023-10-25T00:15:19.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.332+0000 7f7ec9ab1640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ea8065e60 0x7f7ea8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:19.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.332+0000 7f7ec9ab1640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ea8065e60 0x7f7ea8068320 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0006fd0 tx=0x7f7ec0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:19.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.336+0000 7f7eba7fc640 1 -- 172.21.15.107:0/1680363104 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7ebc0b5040 con 0x7f7ec4156770 2023-10-25T00:15:19.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.520+0000 7f7ecbd15640 1 -- 172.21.15.107:0/1680363104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ec40659e0 con 0x7f7ec4156770 2023-10-25T00:15:19.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.520+0000 7f7eba7fc640 1 -- 172.21.15.107:0/1680363104 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+5015 (secure 0 0 0) 0x7f7ebc018020 con 0x7f7ec4156770 2023-10-25T00:15:19.521 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:19.521 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:18.922432+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:19.522 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 181 2023-10-25T00:15:19.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 -- 172.21.15.107:0/1680363104 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ea8065e60 msgr2=0x7f7ea8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ea8065e60 0x7f7ea8068320 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0006fd0 tx=0x7f7ec0008040 comp rx=0 tx=0).stop 2023-10-25T00:15:19.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 -- 172.21.15.107:0/1680363104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec4156770 msgr2=0x7f7ec40a7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec4156770 0x7f7ec40a7790 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc001570 tx=0x7f7ebc00cca0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 -- 172.21.15.107:0/1680363104 shutdown_connections 2023-10-25T00:15:19.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ec4156770 0x7f7ec40a7790 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ea8065e60 0x7f7ea8068320 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7ec40c70e0 0x7f7ec409c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 --2- 172.21.15.107:0/1680363104 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ec40c2130 0x7f7ec409ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 -- 172.21.15.107:0/1680363104 >> 172.21.15.107:0/1680363104 conn(0x7f7ec40c65c0 msgr2=0x7f7ec4069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:19.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 -- 172.21.15.107:0/1680363104 shutdown_connections 2023-10-25T00:15:19.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.524+0000 7f7ecbd15640 1 -- 172.21.15.107:0/1680363104 wait complete. 2023-10-25T00:15:19.543 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2023-10-25T00:15:19.543 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:15:19.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:19.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.652+0000 7f4589529640 1 Processor -- start 2023-10-25T00:15:19.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.652+0000 7f4589529640 1 -- start start 2023-10-25T00:15:19.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f45840c8100 0x7f45840abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f458406a4b0 0x7f45840ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4584155680 con 0x7f4584152040 2023-10-25T00:15:19.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4584154a70 con 0x7f458406a4b0 2023-10-25T00:15:19.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4584154d50 con 0x7f45840c8100 2023-10-25T00:15:19.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f45837fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f458406a4b0 0x7f45840ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:19.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4582ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4584152040 0x7f45840c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:46092/0 (socket says 172.21.15.107:46092) 2023-10-25T00:15:19.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4582ffd640 1 -- 172.21.15.107:0/3899524619 learned_addr learned my addr 172.21.15.107:0/3899524619 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:19.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f45837fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f458406a4b0 0x7f45840ac240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58682/0 (socket says 172.21.15.107:58682) 2023-10-25T00:15:19.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f45837fe640 1 -- 172.21.15.107:0/3899524619 >> v2:172.21.15.110:3300/0 conn(0x7f45840c8100 msgr2=0x7f45840abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:19.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f45837fe640 1 --2- 172.21.15.107:0/3899524619 >> v2:172.21.15.110:3300/0 conn(0x7f45840c8100 0x7f45840abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f45837fe640 1 -- 172.21.15.107:0/3899524619 >> v1:172.21.15.107:6789/0 conn(0x7f4584152040 legacy=0x7f45840c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f45837fe640 1 -- 172.21.15.107:0/3899524619 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4584155030 con 0x7f458406a4b0 2023-10-25T00:15:19.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4581ffb640 1 -- 172.21.15.107:0/3899524619 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827706075 0 0) 0x7f4584155680 con 0x7f4584152040 2023-10-25T00:15:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f45837fe640 1 --2- 172.21.15.107:0/3899524619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458406a4b0 0x7f45840ac240 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f45840c4500 tx=0x7f457c01c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94134bfaadaeed7 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4581ffb640 1 -- 172.21.15.107:0/3899524619 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c0a5760 con 0x7f458406a4b0 2023-10-25T00:15:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4581ffb640 1 -- 172.21.15.107:0/3899524619 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c0a58c0 con 0x7f458406a4b0 2023-10-25T00:15:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4581ffb640 1 -- 172.21.15.107:0/3899524619 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c0a5c50 con 0x7f458406a4b0 2023-10-25T00:15:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 -- 172.21.15.107:0/3899524619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458406a4b0 msgr2=0x7f45840ac240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 --2- 172.21.15.107:0/3899524619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458406a4b0 0x7f45840ac240 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f45840c4500 tx=0x7f457c01c880 comp rx=0 tx=0).stop 2023-10-25T00:15:19.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 -- 172.21.15.107:0/3899524619 shutdown_connections 2023-10-25T00:15:19.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 --2- 172.21.15.107:0/3899524619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f458406a4b0 0x7f45840ac240 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 --2- 172.21.15.107:0/3899524619 >> v2:172.21.15.110:3300/0 conn(0x7f45840c8100 0x7f45840abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 -- 172.21.15.107:0/3899524619 >> 172.21.15.107:0/3899524619 conn(0x7f45840ca200 msgr2=0x7f45840ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:19.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 -- 172.21.15.107:0/3899524619 shutdown_connections 2023-10-25T00:15:19.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.656+0000 7f4589529640 1 -- 172.21.15.107:0/3899524619 wait complete. 2023-10-25T00:15:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f4589529640 1 Processor -- start 2023-10-25T00:15:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f4589529640 1 -- start start 2023-10-25T00:15:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f4589529640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f458406a4b0 0x7f45841ed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f4589529640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45840c8100 0x7f458414b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f4589529640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4584152040 0x7f45841433c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f4589529640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4584145ec0 con 0x7f45840c8100 2023-10-25T00:15:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f4589529640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45841ed930 con 0x7f458406a4b0 2023-10-25T00:15:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f4589529640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458414bf20 con 0x7f4584152040 2023-10-25T00:15:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f45827fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45840c8100 0x7f458414b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f45827fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45840c8100 0x7f458414b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58694/0 (socket says 172.21.15.107:58694) 2023-10-25T00:15:19.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f45827fc640 1 -- 172.21.15.107:0/2270478713 learned_addr learned my addr 172.21.15.107:0/2270478713 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:19.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f45827fc640 1 -- 172.21.15.107:0/2270478713 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4584152040 msgr2=0x7f45841433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f45827fc640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4584152040 0x7f45841433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f45827fc640 1 -- 172.21.15.107:0/2270478713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f458406a4b0 msgr2=0x7f45841ed3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f45827fc640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f458406a4b0 0x7f45841ed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f45827fc640 1 -- 172.21.15.107:0/2270478713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c009a70 con 0x7f45840c8100 2023-10-25T00:15:19.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f45827fc640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45840c8100 0x7f458414b9e0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f457400b480 tx=0x7f457400b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:19.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f456b7fe640 1 -- 172.21.15.107:0/2270478713 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574010870 con 0x7f45840c8100 2023-10-25T00:15:19.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f456b7fe640 1 -- 172.21.15.107:0/2270478713 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45740109f0 con 0x7f45840c8100 2023-10-25T00:15:19.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f4589529640 1 -- 172.21.15.107:0/2270478713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458414d370 con 0x7f45840c8100 2023-10-25T00:15:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f456b7fe640 1 -- 172.21.15.107:0/2270478713 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574010da0 con 0x7f45840c8100 2023-10-25T00:15:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f4589529640 1 -- 172.21.15.107:0/2270478713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458414cb00 con 0x7f45840c8100 2023-10-25T00:15:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f4589529640 1 -- 172.21.15.107:0/2270478713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458414ccf0 con 0x7f45840c8100 2023-10-25T00:15:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.660+0000 7f456b7fe640 1 -- 172.21.15.107:0/2270478713 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f457400f3c0 con 0x7f45840c8100 2023-10-25T00:15:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.664+0000 7f456b7fe640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4570065e60 0x7f4570068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.664+0000 7f456b7fe640 1 -- 172.21.15.107:0/2270478713 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 9971+0+0 (secure 0 0 0) 0x7f45740847e0 con 0x7f45840c8100 2023-10-25T00:15:19.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.664+0000 7f4582ffd640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4570065e60 0x7f4570068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:19.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.664+0000 7f4582ffd640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4570065e60 0x7f4570068320 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f45780011f0 tx=0x7f4578007930 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:19.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.668+0000 7f456b7fe640 1 -- 172.21.15.107:0/2270478713 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4574050f20 con 0x7f45840c8100 2023-10-25T00:15:19.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.856+0000 7f4589529640 1 -- 172.21.15.107:0/2270478713 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45840746e0 con 0x7f45840c8100 2023-10-25T00:15:19.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f456b7fe640 1 -- 172.21.15.107:0/2270478713 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+5015 (secure 0 0 0) 0x7f457401e090 con 0x7f45840c8100 2023-10-25T00:15:19.858 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:19.859 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:18.922432+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:19.859 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 181 2023-10-25T00:15:19.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 -- 172.21.15.107:0/2270478713 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4570065e60 msgr2=0x7f4570068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4570065e60 0x7f4570068320 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f45780011f0 tx=0x7f4578007930 comp rx=0 tx=0).stop 2023-10-25T00:15:19.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 -- 172.21.15.107:0/2270478713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45840c8100 msgr2=0x7f458414b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:19.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45840c8100 0x7f458414b9e0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f457400b480 tx=0x7f457400b940 comp rx=0 tx=0).stop 2023-10-25T00:15:19.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 -- 172.21.15.107:0/2270478713 shutdown_connections 2023-10-25T00:15:19.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4570065e60 0x7f4570068320 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4584152040 0x7f45841433c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45840c8100 0x7f458414b9e0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 --2- 172.21.15.107:0/2270478713 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f458406a4b0 0x7f45841ed3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:19.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 -- 172.21.15.107:0/2270478713 >> 172.21.15.107:0/2270478713 conn(0x7f45840ca200 msgr2=0x7f4584154190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:19.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 -- 172.21.15.107:0/2270478713 shutdown_connections 2023-10-25T00:15:19.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:19.860+0000 7f4589529640 1 -- 172.21.15.107:0/2270478713 wait complete. 2023-10-25T00:15:19.878 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 181, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 181, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7030": { "addr": "172.21.15.107:6835/208713198", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 208713198, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 208713198, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7030, "incarnation": 181, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:15:18.922432+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7030 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 179, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6976": { "addr": "172.21.15.110:6835/1619327197", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 1619327197, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 1619327197, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6976, "incarnation": 176, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:13:51.910080+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6976 } } } ], "standbys": [ { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 120, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:15:20.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:20.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.256+0000 7f235b0e0640 1 Processor -- start 2023-10-25T00:15:20.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.256+0000 7f235b0e0640 1 -- start start 2023-10-25T00:15:20.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f235b0e0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f235414fd40 0x7f23540ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:20.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f235b0e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f23540a9820 0x7f23540aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:20.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f235b0e0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f235406ee40 0x7f23540c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f235b0e0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23540aa750 con 0x7f23540a9820 2023-10-25T00:15:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f235b0e0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f235406f730 con 0x7f235414fd40 2023-10-25T00:15:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f235b0e0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2354156a40 con 0x7f235406ee40 2023-10-25T00:15:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f23540a9820 0x7f23540aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:20.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f23540a9820 0x7f23540aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58700/0 (socket says 172.21.15.107:58700) 2023-10-25T00:15:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234bfff640 1 -- 172.21.15.107:0/2906858357 learned_addr learned my addr 172.21.15.107:0/2906858357 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f2358e7c640 1 --2- 172.21.15.107:0/2906858357 >> v2:172.21.15.110:3300/0 conn(0x7f235414fd40 0x7f23540ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f235967d640 1 --2- 172.21.15.107:0/2906858357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f235406ee40 0x7f23540c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234bfff640 1 -- 172.21.15.107:0/2906858357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f235406ee40 msgr2=0x7f23540c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:20.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234bfff640 1 --2- 172.21.15.107:0/2906858357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f235406ee40 0x7f23540c1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234bfff640 1 -- 172.21.15.107:0/2906858357 >> v2:172.21.15.110:3300/0 conn(0x7f235414fd40 msgr2=0x7f23540ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234bfff640 1 --2- 172.21.15.107:0/2906858357 >> v2:172.21.15.110:3300/0 conn(0x7f235414fd40 0x7f23540ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234bfff640 1 -- 172.21.15.107:0/2906858357 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2354156710 con 0x7f23540a9820 2023-10-25T00:15:20.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234bfff640 1 --2- 172.21.15.107:0/2906858357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23540a9820 0x7f23540aa210 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f23540c28b0 tx=0x7f234401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c449d19283b8fa6 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234b7fe640 1 -- 172.21.15.107:0/2906858357 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23440ac640 con 0x7f23540a9820 2023-10-25T00:15:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234b7fe640 1 -- 172.21.15.107:0/2906858357 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23440ac7a0 con 0x7f23540a9820 2023-10-25T00:15:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.260+0000 7f234b7fe640 1 -- 172.21.15.107:0/2906858357 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23440acb30 con 0x7f23540a9820 2023-10-25T00:15:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 -- 172.21.15.107:0/2906858357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23540a9820 msgr2=0x7f23540aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:20.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 --2- 172.21.15.107:0/2906858357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23540a9820 0x7f23540aa210 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f23540c28b0 tx=0x7f234401c3b0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 -- 172.21.15.107:0/2906858357 shutdown_connections 2023-10-25T00:15:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 --2- 172.21.15.107:0/2906858357 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f235406ee40 0x7f23540c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 --2- 172.21.15.107:0/2906858357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f23540a9820 0x7f23540aa210 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 --2- 172.21.15.107:0/2906858357 >> v2:172.21.15.110:3300/0 conn(0x7f235414fd40 0x7f23540ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 -- 172.21.15.107:0/2906858357 >> 172.21.15.107:0/2906858357 conn(0x7f23540c2950 msgr2=0x7f23540c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:20.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 -- 172.21.15.107:0/2906858357 shutdown_connections 2023-10-25T00:15:20.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 -- 172.21.15.107:0/2906858357 wait complete. 2023-10-25T00:15:20.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 Processor -- start 2023-10-25T00:15:20.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 -- start start 2023-10-25T00:15:20.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f235406ee40 0x7f23541eb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:20.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f235414fd40 0x7f235414ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:20.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23541eedf0 0x7f235415af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:20.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2354153b20 con 0x7f235406ee40 2023-10-25T00:15:20.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2354150120 con 0x7f235414fd40 2023-10-25T00:15:20.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23541ee960 con 0x7f23541eedf0 2023-10-25T00:15:20.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f2358e7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f235406ee40 0x7f23541eb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:20.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f2358e7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f235406ee40 0x7f23541eb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58708/0 (socket says 172.21.15.107:58708) 2023-10-25T00:15:20.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f2358e7c640 1 -- 172.21.15.107:0/2848905005 learned_addr learned my addr 172.21.15.107:0/2848905005 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:20.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f234bfff640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f235414fd40 0x7f235414ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:20.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235967d640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23541eedf0 0x7f235415af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:20.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f2358e7c640 1 -- 172.21.15.107:0/2848905005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23541eedf0 msgr2=0x7f235415af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:20.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f2358e7c640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23541eedf0 0x7f235415af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f2358e7c640 1 -- 172.21.15.107:0/2848905005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f235414fd40 msgr2=0x7f235414ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:20.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f2358e7c640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f235414fd40 0x7f235414ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f2358e7c640 1 -- 172.21.15.107:0/2848905005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23440094d0 con 0x7f235406ee40 2023-10-25T00:15:20.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f234bfff640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f235414fd40 0x7f235414ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:15:20.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f2358e7c640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f235406ee40 0x7f23541eb1c0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f235000c820 tx=0x7f235000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:20.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f23497fa640 1 -- 172.21.15.107:0/2848905005 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23500079d0 con 0x7f235406ee40 2023-10-25T00:15:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f23497fa640 1 -- 172.21.15.107:0/2848905005 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2350007b30 con 0x7f235406ee40 2023-10-25T00:15:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 -- 172.21.15.107:0/2848905005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2354150370 con 0x7f235406ee40 2023-10-25T00:15:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f23497fa640 1 -- 172.21.15.107:0/2848905005 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235000ce10 con 0x7f235406ee40 2023-10-25T00:15:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.264+0000 7f235b0e0640 1 -- 172.21.15.107:0/2848905005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235414b150 con 0x7f235406ee40 2023-10-25T00:15:20.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.268+0000 7f235b0e0640 1 -- 172.21.15.107:0/2848905005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23540c24a0 con 0x7f235406ee40 2023-10-25T00:15:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.268+0000 7f23497fa640 1 -- 172.21.15.107:0/2848905005 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f23500113f0 con 0x7f235406ee40 2023-10-25T00:15:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.268+0000 7f23497fa640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2340065e90 0x7f2340068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.268+0000 7f23497fa640 1 -- 172.21.15.107:0/2848905005 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 9971+0+0 (secure 0 0 0) 0x7f2350084670 con 0x7f235406ee40 2023-10-25T00:15:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.268+0000 7f234bfff640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2340065e90 0x7f2340068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:20.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.268+0000 7f234bfff640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2340065e90 0x7f2340068350 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f23440ae000 tx=0x7f2344008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:20.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.272+0000 7f23497fa640 1 -- 172.21.15.107:0/2848905005 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2350050d50 con 0x7f235406ee40 2023-10-25T00:15:20.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.444+0000 7f235b0e0640 1 -- 172.21.15.107:0/2848905005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2354070840 con 0x7f235406ee40 2023-10-25T00:15:20.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.444+0000 7f23497fa640 1 -- 172.21.15.107:0/2848905005 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5019 (secure 0 0 0) 0x7f235001f020 con 0x7f235406ee40 2023-10-25T00:15:20.444 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:20.445 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:19.946554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:20.445 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 182 2023-10-25T00:15:20.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 -- 172.21.15.107:0/2848905005 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2340065e90 msgr2=0x7f2340068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:20.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2340065e90 0x7f2340068350 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f23440ae000 tx=0x7f2344008040 comp rx=0 tx=0).stop 2023-10-25T00:15:20.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 -- 172.21.15.107:0/2848905005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f235406ee40 msgr2=0x7f23541eb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:20.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f235406ee40 0x7f23541eb1c0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f235000c820 tx=0x7f235000cce0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 -- 172.21.15.107:0/2848905005 shutdown_connections 2023-10-25T00:15:20.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2340065e90 0x7f2340068350 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23541eedf0 0x7f235415af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f235414fd40 0x7f235414ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 --2- 172.21.15.107:0/2848905005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f235406ee40 0x7f23541eb1c0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:20.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 -- 172.21.15.107:0/2848905005 >> 172.21.15.107:0/2848905005 conn(0x7f23540c2950 msgr2=0x7f235414d570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:20.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 -- 172.21.15.107:0/2848905005 shutdown_connections 2023-10-25T00:15:20.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:20.448+0000 7f235b0e0640 1 -- 172.21.15.107:0/2848905005 wait complete. 2023-10-25T00:15:20.462 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-25T00:15:21.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:22.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e073dc640 1 Processor -- start 2023-10-25T00:15:22.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e073dc640 1 -- start start 2023-10-25T00:15:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e073dc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0e00152040 0x7f0e000c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e073dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0006a4b0 0x7f0e000ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e073dc640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e00155680 con 0x7f0e000c8100 2023-10-25T00:15:22.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e073dc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e00154a70 con 0x7f0e00152040 2023-10-25T00:15:22.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e073dc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e00154d50 con 0x7f0e0006a4b0 2023-10-25T00:15:22.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e05178640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0e00152040 0x7f0e000c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e05979640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0006a4b0 0x7f0e000ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e04977640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0e000c8100 0x7f0e000abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:37736/0 (socket says 172.21.15.107:37736) 2023-10-25T00:15:22.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e05178640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0e00152040 0x7f0e000c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:45750/0 (socket says 172.21.15.107:45750) 2023-10-25T00:15:22.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.011+0000 7f0e04977640 1 -- 172.21.15.107:0/2068921785 learned_addr learned my addr 172.21.15.107:0/2068921785 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:22.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e05979640 1 -- 172.21.15.107:0/2068921785 >> v2:172.21.15.110:3300/0 conn(0x7f0e00152040 msgr2=0x7f0e000c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:22.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e05979640 1 --2- 172.21.15.107:0/2068921785 >> v2:172.21.15.110:3300/0 conn(0x7f0e00152040 0x7f0e000c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e05979640 1 -- 172.21.15.107:0/2068921785 >> v1:172.21.15.110:6789/0 conn(0x7f0e000c8100 legacy=0x7f0e000abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e05979640 1 -- 172.21.15.107:0/2068921785 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e00155030 con 0x7f0e0006a4b0 2023-10-25T00:15:22.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e05979640 1 --2- 172.21.15.107:0/2068921785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0006a4b0 0x7f0e000ac410 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009d90 tx=0x7f0df801c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0019b5632f7b12e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:22.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0df7fff640 1 -- 172.21.15.107:0/2068921785 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df80a5890 con 0x7f0e0006a4b0 2023-10-25T00:15:22.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0df7fff640 1 -- 172.21.15.107:0/2068921785 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df80a59f0 con 0x7f0e0006a4b0 2023-10-25T00:15:22.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0df7fff640 1 -- 172.21.15.107:0/2068921785 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df80a5d80 con 0x7f0e0006a4b0 2023-10-25T00:15:22.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e073dc640 1 -- 172.21.15.107:0/2068921785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0006a4b0 msgr2=0x7f0e000ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e073dc640 1 --2- 172.21.15.107:0/2068921785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0006a4b0 0x7f0e000ac410 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009d90 tx=0x7f0df801c8c0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e073dc640 1 -- 172.21.15.107:0/2068921785 shutdown_connections 2023-10-25T00:15:22.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e073dc640 1 --2- 172.21.15.107:0/2068921785 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0006a4b0 0x7f0e000ac410 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e073dc640 1 --2- 172.21.15.107:0/2068921785 >> v2:172.21.15.110:3300/0 conn(0x7f0e00152040 0x7f0e000c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e073dc640 1 -- 172.21.15.107:0/2068921785 >> 172.21.15.107:0/2068921785 conn(0x7f0e000ca200 msgr2=0x7f0e000ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:22.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e073dc640 1 -- 172.21.15.107:0/2068921785 shutdown_connections 2023-10-25T00:15:22.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.015+0000 7f0e073dc640 1 -- 172.21.15.107:0/2068921785 wait complete. 2023-10-25T00:15:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e073dc640 1 Processor -- start 2023-10-25T00:15:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e073dc640 1 -- start start 2023-10-25T00:15:22.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e073dc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e0006a4b0 0x7f0e000abc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e073dc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e000c8100 0x7f0e0014de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e073dc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e00152040 0x7f0e001efee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e073dc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e000ac250 con 0x7f0e00152040 2023-10-25T00:15:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e073dc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e00151340 con 0x7f0e0006a4b0 2023-10-25T00:15:22.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e073dc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e00151620 con 0x7f0e000c8100 2023-10-25T00:15:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e05979640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e00152040 0x7f0e001efee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e05979640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e00152040 0x7f0e001efee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58722/0 (socket says 172.21.15.107:58722) 2023-10-25T00:15:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e05979640 1 -- 172.21.15.107:0/896824248 learned_addr learned my addr 172.21.15.107:0/896824248 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e04977640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e000c8100 0x7f0e0014de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e05178640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e0006a4b0 0x7f0e000abc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e05979640 1 -- 172.21.15.107:0/896824248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e000c8100 msgr2=0x7f0e0014de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e05979640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e000c8100 0x7f0e0014de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e05979640 1 -- 172.21.15.107:0/896824248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e0006a4b0 msgr2=0x7f0e000abc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e05979640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e0006a4b0 0x7f0e000abc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e05979640 1 -- 172.21.15.107:0/896824248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8009a70 con 0x7f0e00152040 2023-10-25T00:15:22.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e05979640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e00152040 0x7f0e001efee0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f0df801af70 tx=0x7f0df8019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0df5ffb640 1 -- 172.21.15.107:0/896824248 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df80ad640 con 0x7f0e00152040 2023-10-25T00:15:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0df5ffb640 1 -- 172.21.15.107:0/896824248 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df80ad7a0 con 0x7f0e00152040 2023-10-25T00:15:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e073dc640 1 -- 172.21.15.107:0/896824248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e00151900 con 0x7f0e00152040 2023-10-25T00:15:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0df5ffb640 1 -- 172.21.15.107:0/896824248 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df80adb10 con 0x7f0e00152040 2023-10-25T00:15:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e073dc640 1 -- 172.21.15.107:0/896824248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0014e3c0 con 0x7f0e00152040 2023-10-25T00:15:22.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.019+0000 7f0e073dc640 1 -- 172.21.15.107:0/896824248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e00151e20 con 0x7f0e00152040 2023-10-25T00:15:22.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.023+0000 7f0df5ffb640 1 -- 172.21.15.107:0/896824248 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0df80b4050 con 0x7f0e00152040 2023-10-25T00:15:22.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.023+0000 7f0df5ffb640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0de4065e60 0x7f0de4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.023+0000 7f0df5ffb640 1 -- 172.21.15.107:0/896824248 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 9971+0+0 (secure 0 0 0) 0x7f0df811f9b0 con 0x7f0e00152040 2023-10-25T00:15:22.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.023+0000 7f0e05178640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0de4065e60 0x7f0de4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.023+0000 7f0e05178640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0de4065e60 0x7f0de4068320 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009af0 tx=0x7f0dfc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:22.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.027+0000 7f0df5ffb640 1 -- 172.21.15.107:0/896824248 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0e00151e20 con 0x7f0e00152040 2023-10-25T00:15:22.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 -- 172.21.15.107:0/896824248 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e00151e20 con 0x7f0e00152040 2023-10-25T00:15:22.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0df5ffb640 1 -- 172.21.15.107:0/896824248 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f0e00151e20 con 0x7f0e00152040 2023-10-25T00:15:22.199 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:22.199 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5699,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":120}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:21.646566+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:13:51.910080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6976},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6976":{"gid":6976,"name":"d","rank":0,"incarnation":176,"state":"up:active","state_seq":9,"addr":"172.21.15.110:6835/1619327197","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":1619327197},{"type":"v1","addr":"172.21.15.110:6835","nonce":1619327197}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:22.199 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 183 2023-10-25T00:15:22.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 -- 172.21.15.107:0/896824248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0de4065e60 msgr2=0x7f0de4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0de4065e60 0x7f0de4068320 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009af0 tx=0x7f0dfc008040 comp rx=0 tx=0).stop 2023-10-25T00:15:22.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 -- 172.21.15.107:0/896824248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e00152040 msgr2=0x7f0e001efee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e00152040 0x7f0e001efee0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f0df801af70 tx=0x7f0df8019660 comp rx=0 tx=0).stop 2023-10-25T00:15:22.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 -- 172.21.15.107:0/896824248 shutdown_connections 2023-10-25T00:15:22.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e00152040 0x7f0e001efee0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0de4065e60 0x7f0de4068320 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e000c8100 0x7f0e0014de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 --2- 172.21.15.107:0/896824248 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e0006a4b0 0x7f0e000abc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 -- 172.21.15.107:0/896824248 >> 172.21.15.107:0/896824248 conn(0x7f0e000ca200 msgr2=0x7f0e00153fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 -- 172.21.15.107:0/896824248 shutdown_connections 2023-10-25T00:15:22.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.199+0000 7f0e073dc640 1 -- 172.21.15.107:0/896824248 wait complete. 2023-10-25T00:15:22.217 INFO:tasks.mds_thrash.fs.[a]:no change 2023-10-25T00:15:22.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:22.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.587+0000 7f347f9d8640 1 Processor -- start 2023-10-25T00:15:22.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.587+0000 7f347f9d8640 1 -- start start 2023-10-25T00:15:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347f9d8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34780c8100 0x7f34780abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347f9d8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f347806a4b0 0x7f34780ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347f9d8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3478155680 con 0x7f3478152040 2023-10-25T00:15:22.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347f9d8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3478154a70 con 0x7f347806a4b0 2023-10-25T00:15:22.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347f9d8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3478154d50 con 0x7f34780c8100 2023-10-25T00:15:22.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347df75640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f347806a4b0 0x7f34780ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347df75640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f347806a4b0 0x7f34780ac240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53596/0 (socket says 172.21.15.107:53596) 2023-10-25T00:15:22.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347df75640 1 -- 172.21.15.107:0/1500877057 learned_addr learned my addr 172.21.15.107:0/1500877057 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:22.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347d774640 1 --1- 172.21.15.107:0/1500877057 >> v1:172.21.15.107:6789/0 conn(0x7f3478152040 0x7f34780c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38926/0 (socket says 172.21.15.107:38926) 2023-10-25T00:15:22.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347cf73640 1 --2- 172.21.15.107:0/1500877057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34780c8100 0x7f34780abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347df75640 1 -- 172.21.15.107:0/1500877057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34780c8100 msgr2=0x7f34780abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:22.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347df75640 1 --2- 172.21.15.107:0/1500877057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34780c8100 0x7f34780abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347df75640 1 -- 172.21.15.107:0/1500877057 >> v1:172.21.15.107:6789/0 conn(0x7f3478152040 legacy=0x7f34780c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347df75640 1 -- 172.21.15.107:0/1500877057 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3478155030 con 0x7f347806a4b0 2023-10-25T00:15:22.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347cf73640 1 --2- 172.21.15.107:0/1500877057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34780c8100 0x7f34780abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:15:22.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347df75640 1 --2- 172.21.15.107:0/1500877057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347806a4b0 0x7f34780ac240 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f3470009e50 tx=0x7f347001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6517f587fb934141 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:22.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f346ffff640 1 -- 172.21.15.107:0/1500877057 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700a5660 con 0x7f347806a4b0 2023-10-25T00:15:22.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f346ffff640 1 -- 172.21.15.107:0/1500877057 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34700a57c0 con 0x7f347806a4b0 2023-10-25T00:15:22.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f346ffff640 1 -- 172.21.15.107:0/1500877057 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700a5b50 con 0x7f347806a4b0 2023-10-25T00:15:22.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347f9d8640 1 -- 172.21.15.107:0/1500877057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347806a4b0 msgr2=0x7f34780ac240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.591+0000 7f347f9d8640 1 --2- 172.21.15.107:0/1500877057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347806a4b0 0x7f34780ac240 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f3470009e50 tx=0x7f347001c980 comp rx=0 tx=0).stop 2023-10-25T00:15:22.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 -- 172.21.15.107:0/1500877057 shutdown_connections 2023-10-25T00:15:22.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 --2- 172.21.15.107:0/1500877057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f347806a4b0 0x7f34780ac240 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 --2- 172.21.15.107:0/1500877057 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34780c8100 0x7f34780abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 -- 172.21.15.107:0/1500877057 >> 172.21.15.107:0/1500877057 conn(0x7f34780ca200 msgr2=0x7f34780ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:22.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 -- 172.21.15.107:0/1500877057 shutdown_connections 2023-10-25T00:15:22.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 -- 172.21.15.107:0/1500877057 wait complete. 2023-10-25T00:15:22.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 Processor -- start 2023-10-25T00:15:22.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 -- start start 2023-10-25T00:15:22.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347806a4b0 0x7f34780abc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34780c8100 0x7f3478149830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3478152040 0x7f347814ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780aaf50 con 0x7f3478152040 2023-10-25T00:15:22.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780ab7f0 con 0x7f347806a4b0 2023-10-25T00:15:22.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34780aa740 con 0x7f34780c8100 2023-10-25T00:15:22.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347df75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3478152040 0x7f347814ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347df75640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3478152040 0x7f347814ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53612/0 (socket says 172.21.15.107:53612) 2023-10-25T00:15:22.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347df75640 1 -- 172.21.15.107:0/4215085817 learned_addr learned my addr 172.21.15.107:0/4215085817 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:22.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347cf73640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34780c8100 0x7f3478149830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347d774640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347806a4b0 0x7f34780abc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347df75640 1 -- 172.21.15.107:0/4215085817 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34780c8100 msgr2=0x7f3478149830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347df75640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34780c8100 0x7f3478149830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347df75640 1 -- 172.21.15.107:0/4215085817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347806a4b0 msgr2=0x7f34780abc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347df75640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347806a4b0 0x7f34780abc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347df75640 1 -- 172.21.15.107:0/4215085817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3470009b30 con 0x7f3478152040 2023-10-25T00:15:22.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347d774640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347806a4b0 0x7f34780abc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:15:22.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347df75640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3478152040 0x7f347814ef60 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f347001a5b0 tx=0x7f3470019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:22.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f346dffb640 1 -- 172.21.15.107:0/4215085817 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700b8670 con 0x7f3478152040 2023-10-25T00:15:22.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f346dffb640 1 -- 172.21.15.107:0/4215085817 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34700b87d0 con 0x7f3478152040 2023-10-25T00:15:22.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 -- 172.21.15.107:0/4215085817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34780ab1a0 con 0x7f3478152040 2023-10-25T00:15:22.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f346dffb640 1 -- 172.21.15.107:0/4215085817 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700b8b60 con 0x7f3478152040 2023-10-25T00:15:22.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 -- 172.21.15.107:0/4215085817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3478149d70 con 0x7f3478152040 2023-10-25T00:15:22.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.595+0000 7f347f9d8640 1 -- 172.21.15.107:0/4215085817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34780ab960 con 0x7f3478152040 2023-10-25T00:15:22.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.599+0000 7f346dffb640 1 -- 172.21.15.107:0/4215085817 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f34700b8cc0 con 0x7f3478152040 2023-10-25T00:15:22.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.599+0000 7f346dffb640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f345c065e60 0x7f345c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:22.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.599+0000 7f346dffb640 1 -- 172.21.15.107:0/4215085817 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 9971+0+0 (secure 0 0 0) 0x7f347011fc10 con 0x7f3478152040 2023-10-25T00:15:22.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.599+0000 7f347d774640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f345c065e60 0x7f345c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:22.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.599+0000 7f347d774640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f345c065e60 0x7f345c068320 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3474000f70 tx=0x7f3474009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:22.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.603+0000 7f346dffb640 1 -- 172.21.15.107:0/4215085817 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f34700ec520 con 0x7f3478152040 2023-10-25T00:15:22.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.863+0000 7f347f9d8640 1 -- 172.21.15.107:0/4215085817 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34780746e0 con 0x7f3478152040 2023-10-25T00:15:22.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.863+0000 7f346dffb640 1 -- 172.21.15.107:0/4215085817 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+4256 (secure 0 0 0) 0x7f34700efb40 con 0x7f3478152040 2023-10-25T00:15:22.866 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:22.866 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:22.674820+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:22.866 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 185 2023-10-25T00:15:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 -- 172.21.15.107:0/4215085817 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f345c065e60 msgr2=0x7f345c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f345c065e60 0x7f345c068320 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3474000f70 tx=0x7f3474009040 comp rx=0 tx=0).stop 2023-10-25T00:15:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 -- 172.21.15.107:0/4215085817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3478152040 msgr2=0x7f347814ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3478152040 0x7f347814ef60 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f347001a5b0 tx=0x7f3470019720 comp rx=0 tx=0).stop 2023-10-25T00:15:22.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 -- 172.21.15.107:0/4215085817 shutdown_connections 2023-10-25T00:15:22.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3478152040 0x7f347814ef60 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f345c065e60 0x7f345c068320 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34780c8100 0x7f3478149830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 --2- 172.21.15.107:0/4215085817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f347806a4b0 0x7f34780abc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:22.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 -- 172.21.15.107:0/4215085817 >> 172.21.15.107:0/4215085817 conn(0x7f34780ca200 msgr2=0x7f3478154090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:22.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 -- 172.21.15.107:0/4215085817 shutdown_connections 2023-10-25T00:15:22.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:22.867+0000 7f347f9d8640 1 -- 172.21.15.107:0/4215085817 wait complete. 2023-10-25T00:15:22.888 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2023-10-25T00:15:22.888 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:15:22.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:23.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.011+0000 7ff605984640 1 Processor -- start 2023-10-25T00:15:23.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.015+0000 7ff605984640 1 -- start start 2023-10-25T00:15:23.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.015+0000 7ff605984640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff60006bf90 0x7ff60014e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:23.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.015+0000 7ff605984640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff600151860 0x7ff60006c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:23.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.015+0000 7ff605984640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60006cd50 con 0x7ff600152220 2023-10-25T00:15:23.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.015+0000 7ff605984640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60006d2a0 con 0x7ff60006bf90 2023-10-25T00:15:23.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.015+0000 7ff605984640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff600151ce0 con 0x7ff600151860 2023-10-25T00:15:23.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.019+0000 7ff5fe7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff600151860 0x7ff60006c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:23.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.019+0000 7ff5fe7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff600151860 0x7ff60006c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:45504/0 (socket says 172.21.15.107:45504) 2023-10-25T00:15:23.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.019+0000 7ff5ff7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff600152220 0x7ff600154320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:50224/0 (socket says 172.21.15.107:50224) 2023-10-25T00:15:23.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.019+0000 7ff5ff7fe640 1 -- 172.21.15.107:0/4186409033 learned_addr learned my addr 172.21.15.107:0/4186409033 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:23.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.019+0000 7ff5fe7fc640 1 -- 172.21.15.107:0/4186409033 >> v2:172.21.15.110:3300/0 conn(0x7ff60006bf90 msgr2=0x7ff60014e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:23.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.019+0000 7ff5fe7fc640 1 --2- 172.21.15.107:0/4186409033 >> v2:172.21.15.110:3300/0 conn(0x7ff60006bf90 0x7ff60014e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:23.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.019+0000 7ff5fe7fc640 1 -- 172.21.15.107:0/4186409033 >> v1:172.21.15.110:6789/0 conn(0x7ff600152220 legacy=0x7ff600154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:23.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.019+0000 7ff5fe7fc640 1 -- 172.21.15.107:0/4186409033 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60006b4f0 con 0x7ff600151860 2023-10-25T00:15:23.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff5fe7fc640 1 --2- 172.21.15.107:0/4186409033 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff600151860 0x7ff60006c810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00097c0 tx=0x7ff5f001c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e4c1481b07cd44f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:23.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff5fdffb640 1 -- 172.21.15.107:0/4186409033 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f00ab600 con 0x7ff600151860 2023-10-25T00:15:23.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff5fdffb640 1 -- 172.21.15.107:0/4186409033 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f00a7050 con 0x7ff600151860 2023-10-25T00:15:23.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff5fdffb640 1 -- 172.21.15.107:0/4186409033 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f00ab980 con 0x7ff600151860 2023-10-25T00:15:23.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff605984640 1 -- 172.21.15.107:0/4186409033 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff600151860 msgr2=0x7ff60006c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff605984640 1 --2- 172.21.15.107:0/4186409033 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff600151860 0x7ff60006c810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00097c0 tx=0x7ff5f001c380 comp rx=0 tx=0).stop 2023-10-25T00:15:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff605984640 1 -- 172.21.15.107:0/4186409033 shutdown_connections 2023-10-25T00:15:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff605984640 1 --2- 172.21.15.107:0/4186409033 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff600151860 0x7ff60006c810 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff605984640 1 --2- 172.21.15.107:0/4186409033 >> v2:172.21.15.110:3300/0 conn(0x7ff60006bf90 0x7ff60014e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff605984640 1 -- 172.21.15.107:0/4186409033 >> 172.21.15.107:0/4186409033 conn(0x7ff6000c6270 msgr2=0x7ff6000c6680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:23.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff605984640 1 -- 172.21.15.107:0/4186409033 shutdown_connections 2023-10-25T00:15:23.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.023+0000 7ff605984640 1 -- 172.21.15.107:0/4186409033 wait complete. 2023-10-25T00:15:23.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff605984640 1 Processor -- start 2023-10-25T00:15:23.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff605984640 1 -- start start 2023-10-25T00:15:23.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff605984640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60006bf90 0x7ff6001f0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:23.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff605984640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff600151860 0x7ff6000b8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:23.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff605984640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff600152220 0x7ff6001f0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:23.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff605984640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff600150d60 con 0x7ff600151860 2023-10-25T00:15:23.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff605984640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60014fbd0 con 0x7ff600152220 2023-10-25T00:15:23.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff605984640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6000c0490 con 0x7ff60006bf90 2023-10-25T00:15:23.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff5fe7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff600151860 0x7ff6000b8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:23.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff5fe7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff600151860 0x7ff6000b8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53616/0 (socket says 172.21.15.107:53616) 2023-10-25T00:15:23.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff5fe7fc640 1 -- 172.21.15.107:0/423595897 learned_addr learned my addr 172.21.15.107:0/423595897 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:23.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff5feffd640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60006bf90 0x7ff6001f0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:23.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff5ff7fe640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff600152220 0x7ff6001f0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:23.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff5feffd640 1 -- 172.21.15.107:0/423595897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff600152220 msgr2=0x7ff6001f0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:23.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff5feffd640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff600152220 0x7ff6001f0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:23.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff5feffd640 1 -- 172.21.15.107:0/423595897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff600151860 msgr2=0x7ff6000b8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:23.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff5feffd640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff600151860 0x7ff6000b8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:23.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.027+0000 7ff5feffd640 1 -- 172.21.15.107:0/423595897 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f0009420 con 0x7ff60006bf90 2023-10-25T00:15:23.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.039+0000 7ff5feffd640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60006bf90 0x7ff6001f0db0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400b420 tx=0x7ff5f400b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:23.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.039+0000 7ff5e7fff640 1 -- 172.21.15.107:0/423595897 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4011830 con 0x7ff60006bf90 2023-10-25T00:15:23.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.039+0000 7ff5e7fff640 1 -- 172.21.15.107:0/423595897 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f4011990 con 0x7ff60006bf90 2023-10-25T00:15:23.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.039+0000 7ff5e7fff640 1 -- 172.21.15.107:0/423595897 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f4011d40 con 0x7ff60006bf90 2023-10-25T00:15:23.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.039+0000 7ff605984640 1 -- 172.21.15.107:0/423595897 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff600150a80 con 0x7ff60006bf90 2023-10-25T00:15:23.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.039+0000 7ff605984640 1 -- 172.21.15.107:0/423595897 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60014f860 con 0x7ff60006bf90 2023-10-25T00:15:23.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.039+0000 7ff5e7fff640 1 -- 172.21.15.107:0/423595897 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff5f401b460 con 0x7ff60006bf90 2023-10-25T00:15:23.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.039+0000 7ff5e7fff640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5ec065eb0 0x7ff5ec068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:23.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.039+0000 7ff5e7fff640 1 -- 172.21.15.107:0/423595897 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7ff5f4084470 con 0x7ff60006bf90 2023-10-25T00:15:23.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.043+0000 7ff5fe7fc640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5ec065eb0 0x7ff5ec068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:23.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.043+0000 7ff605984640 1 -- 172.21.15.107:0/423595897 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6000746e0 con 0x7ff60006bf90 2023-10-25T00:15:23.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.043+0000 7ff5fe7fc640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5ec065eb0 0x7ff5ec068370 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff5f001aa30 tx=0x7ff5f00a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:23.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.047+0000 7ff5e7fff640 1 -- 172.21.15.107:0/423595897 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff5f4050be0 con 0x7ff60006bf90 2023-10-25T00:15:23.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff605984640 1 -- 172.21.15.107:0/423595897 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff60014c9f0 con 0x7ff60006bf90 2023-10-25T00:15:23.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff5e7fff640 1 -- 172.21.15.107:0/423595897 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+4261 (secure 0 0 0) 0x7ff5f4054200 con 0x7ff60006bf90 2023-10-25T00:15:23.712 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:23.712 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:23.693824+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:reconnect","state_seq":142,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 186 2023-10-25T00:15:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff605984640 1 -- 172.21.15.107:0/423595897 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5ec065eb0 msgr2=0x7ff5ec068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff605984640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5ec065eb0 0x7ff5ec068370 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7ff5f001aa30 tx=0x7ff5f00a6000 comp rx=0 tx=0).stop 2023-10-25T00:15:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff605984640 1 -- 172.21.15.107:0/423595897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60006bf90 msgr2=0x7ff6001f0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff605984640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60006bf90 0x7ff6001f0db0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400b420 tx=0x7ff5f400b8e0 comp rx=0 tx=0).stop 2023-10-25T00:15:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff605984640 1 -- 172.21.15.107:0/423595897 shutdown_connections 2023-10-25T00:15:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff605984640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff600152220 0x7ff6001f0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff605984640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff600151860 0x7ff6000b8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff605984640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5ec065eb0 0x7ff5ec068370 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff605984640 1 --2- 172.21.15.107:0/423595897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60006bf90 0x7ff6001f0db0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.711+0000 7ff605984640 1 -- 172.21.15.107:0/423595897 >> 172.21.15.107:0/423595897 conn(0x7ff6000c6270 msgr2=0x7ff6000c4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.715+0000 7ff605984640 1 -- 172.21.15.107:0/423595897 shutdown_connections 2023-10-25T00:15:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:23.715+0000 7ff605984640 1 -- 172.21.15.107:0/423595897 wait complete. 2023-10-25T00:15:23.777 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 186, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 184, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7030": { "addr": "172.21.15.107:6835/208713198", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 208713198, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 208713198, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7030, "incarnation": 181, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:15:22.657484+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7030 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 186, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5699": { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 185, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 142 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:15:23.693824+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5699 } } } ], "standbys": [] } 2023-10-25T00:15:24.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:24.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3e2081640 1 Processor -- start 2023-10-25T00:15:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3e2081640 1 -- start start 2023-10-25T00:15:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3e2081640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa3dc156220 0x7fa3dc06df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3e2081640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3dc0c5d40 con 0x7fa3dc15c200 2023-10-25T00:15:24.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3e2081640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3dc0c6130 con 0x7fa3dc0c6a30 2023-10-25T00:15:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3e2081640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3dc158f60 con 0x7fa3dc156220 2023-10-25T00:15:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3daffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa3dc15c200 0x7fa3dc071100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38930/0 (socket says 172.21.15.107:38930) 2023-10-25T00:15:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3dbfff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa3dc0c6a30 0x7fa3dc071810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:50240/0 (socket says 172.21.15.107:50240) 2023-10-25T00:15:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3dbfff640 1 -- 172.21.15.107:0/1265375060 learned_addr learned my addr 172.21.15.107:0/1265375060 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3db7fe640 1 --2- 172.21.15.107:0/1265375060 >> v2:172.21.15.110:3300/0 conn(0x7fa3dc156220 0x7fa3dc06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3da7fc640 1 -- 172.21.15.107:0/1265375060 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845175152 0 0) 0x7fa3dc0c5d40 con 0x7fa3dc15c200 2023-10-25T00:15:24.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3da7fc640 1 -- 172.21.15.107:0/1265375060 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c4003530 con 0x7fa3dc15c200 2023-10-25T00:15:24.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.507+0000 7fa3db7fe640 1 -- 172.21.15.107:0/1265375060 >> v1:172.21.15.110:6789/0 conn(0x7fa3dc0c6a30 legacy=0x7fa3dc071810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:24.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.511+0000 7fa3da7fc640 1 -- 172.21.15.107:0/1265375060 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2218985550 0 0) 0x7fa3c4003530 con 0x7fa3dc15c200 2023-10-25T00:15:24.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.511+0000 7fa3db7fe640 1 -- 172.21.15.107:0/1265375060 >> v1:172.21.15.107:6789/0 conn(0x7fa3dc15c200 legacy=0x7fa3dc071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:24.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.511+0000 7fa3db7fe640 1 -- 172.21.15.107:0/1265375060 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3dc159240 con 0x7fa3dc156220 2023-10-25T00:15:24.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.523+0000 7fa3db7fe640 1 --2- 172.21.15.107:0/1265375060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3dc156220 0x7fa3dc06df30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa3d00097c0 tx=0x7fa3d001c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eae065f0390b0a6f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:24.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.523+0000 7fa3da7fc640 1 -- 172.21.15.107:0/1265375060 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc001680 con 0x7fa3dc156220 2023-10-25T00:15:24.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.527+0000 7fa3da7fc640 1 -- 172.21.15.107:0/1265375060 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d00a7050 con 0x7fa3dc156220 2023-10-25T00:15:24.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.527+0000 7fa3da7fc640 1 -- 172.21.15.107:0/1265375060 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc001680 con 0x7fa3dc156220 2023-10-25T00:15:24.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.527+0000 7fa3e2081640 1 -- 172.21.15.107:0/1265375060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3dc156220 msgr2=0x7fa3dc06df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:24.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.527+0000 7fa3e2081640 1 --2- 172.21.15.107:0/1265375060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3dc156220 0x7fa3dc06df30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa3d00097c0 tx=0x7fa3d001c2e0 comp rx=0 tx=0).stop 2023-10-25T00:15:24.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.527+0000 7fa3e2081640 1 -- 172.21.15.107:0/1265375060 shutdown_connections 2023-10-25T00:15:24.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.527+0000 7fa3e2081640 1 --2- 172.21.15.107:0/1265375060 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3dc156220 0x7fa3dc06df30 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:24.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.527+0000 7fa3e2081640 1 -- 172.21.15.107:0/1265375060 >> 172.21.15.107:0/1265375060 conn(0x7fa3dc0c39a0 msgr2=0x7fa3dc0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:24.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.527+0000 7fa3e2081640 1 -- 172.21.15.107:0/1265375060 shutdown_connections 2023-10-25T00:15:24.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.527+0000 7fa3e2081640 1 -- 172.21.15.107:0/1265375060 wait complete. 2023-10-25T00:15:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.527+0000 7fa3e2081640 1 Processor -- start 2023-10-25T00:15:24.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3e2081640 1 -- start start 2023-10-25T00:15:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3e2081640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3dc0c6a30 0x7fa3dc06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3e2081640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3dc156220 0x7fa3dc06b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3e2081640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3dc15c200 0x7fa3dc1f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:24.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3e2081640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc14bc30 con 0x7fa3dc0c6a30 2023-10-25T00:15:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3e2081640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc0718d0 con 0x7fa3dc15c200 2023-10-25T00:15:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3e2081640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc14a280 con 0x7fa3dc156220 2023-10-25T00:15:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3db7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3dc0c6a30 0x7fa3dc06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3db7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3dc0c6a30 0x7fa3dc06afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53620/0 (socket says 172.21.15.107:53620) 2023-10-25T00:15:24.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3db7fe640 1 -- 172.21.15.107:0/1017242898 learned_addr learned my addr 172.21.15.107:0/1017242898 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3db7fe640 1 -- 172.21.15.107:0/1017242898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3dc156220 msgr2=0x7fa3dc06b4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:15:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3db7fe640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3dc156220 0x7fa3dc06b4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3db7fe640 1 -- 172.21.15.107:0/1017242898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3dc15c200 msgr2=0x7fa3dc1f4b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:15:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3db7fe640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3dc15c200 0x7fa3dc1f4b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3db7fe640 1 -- 172.21.15.107:0/1017242898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d0009420 con 0x7fa3dc0c6a30 2023-10-25T00:15:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3db7fe640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3dc0c6a30 0x7fa3dc06afb0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fa3d00097c0 tx=0x7fa3d0019080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3c3fff640 1 -- 172.21.15.107:0/1017242898 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc001680 con 0x7fa3dc0c6a30 2023-10-25T00:15:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3c3fff640 1 -- 172.21.15.107:0/1017242898 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d00abda0 con 0x7fa3dc0c6a30 2023-10-25T00:15:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3e2081640 1 -- 172.21.15.107:0/1017242898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3dc14bed0 con 0x7fa3dc0c6a30 2023-10-25T00:15:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3e2081640 1 -- 172.21.15.107:0/1017242898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc14a560 con 0x7fa3dc0c6a30 2023-10-25T00:15:24.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3c3fff640 1 -- 172.21.15.107:0/1017242898 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d00b4600 con 0x7fa3dc0c6a30 2023-10-25T00:15:24.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.535+0000 7fa3c3fff640 1 -- 172.21.15.107:0/1017242898 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa3d00b2040 con 0x7fa3dc0c6a30 2023-10-25T00:15:24.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.535+0000 7fa3c3fff640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3c8065f60 0x7fa3c8068420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:24.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.535+0000 7fa3c3fff640 1 -- 172.21.15.107:0/1017242898 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7fa3d011e210 con 0x7fa3dc0c6a30 2023-10-25T00:15:24.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.531+0000 7fa3e2081640 1 -- 172.21.15.107:0/1017242898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3bc005260 con 0x7fa3dc0c6a30 2023-10-25T00:15:24.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.543+0000 7fa3daffd640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3c8065f60 0x7fa3c8068420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:24.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.543+0000 7fa3daffd640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3c8065f60 0x7fa3c8068420 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc001390 tx=0x7fa3cc0010c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:24.543+0000 7fa3c3fff640 1 -- 172.21.15.107:0/1017242898 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa3d00b0080 con 0x7fa3dc0c6a30 2023-10-25T00:15:25.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.027+0000 7fa3e2081640 1 -- 172.21.15.107:0/1017242898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3bc005a90 con 0x7fa3dc0c6a30 2023-10-25T00:15:25.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.027+0000 7fa3c3fff640 1 -- 172.21.15.107:0/1017242898 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+4258 (secure 0 0 0) 0x7fa3d00b9030 con 0x7fa3dc0c6a30 2023-10-25T00:15:25.029 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:25.030 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:24.696837+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:rejoin","state_seq":143,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:25.030 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 187 2023-10-25T00:15:25.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 -- 172.21.15.107:0/1017242898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3c8065f60 msgr2=0x7fa3c8068420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:25.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3c8065f60 0x7fa3c8068420 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc001390 tx=0x7fa3cc0010c0 comp rx=0 tx=0).stop 2023-10-25T00:15:25.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 -- 172.21.15.107:0/1017242898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3dc0c6a30 msgr2=0x7fa3dc06afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:25.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3dc0c6a30 0x7fa3dc06afb0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fa3d00097c0 tx=0x7fa3d0019080 comp rx=0 tx=0).stop 2023-10-25T00:15:25.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 -- 172.21.15.107:0/1017242898 shutdown_connections 2023-10-25T00:15:25.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa3dc15c200 0x7fa3dc1f4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:25.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa3c8065f60 0x7fa3c8068420 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:25.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa3dc156220 0x7fa3dc06b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:25.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 --2- 172.21.15.107:0/1017242898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3dc0c6a30 0x7fa3dc06afb0 secure :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fa3d00097c0 tx=0x7fa3d0019080 comp rx=0 tx=0).stop 2023-10-25T00:15:25.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 -- 172.21.15.107:0/1017242898 >> 172.21.15.107:0/1017242898 conn(0x7fa3dc0c39a0 msgr2=0x7fa3dc150c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:25.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 -- 172.21.15.107:0/1017242898 shutdown_connections 2023-10-25T00:15:25.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:25.039+0000 7fa3e2081640 1 -- 172.21.15.107:0/1017242898 wait complete. 2023-10-25T00:15:25.088 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=8106 2023-10-25T00:15:25.088 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.a 2023-10-25T00:15:25.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:26.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a98a640 1 Processor -- start 2023-10-25T00:15:26.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a98a640 1 -- start start 2023-10-25T00:15:26.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a98a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f798406ba60 0x7f7984071e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:26.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a98a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f79840c6590 0x7f7984152d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:26.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f7989187640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f79840723a0 0x7f7984152630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38944/0 (socket says 172.21.15.107:38944) 2023-10-25T00:15:26.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f7989988640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f798406ba60 0x7f7984071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f7989988640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f798406ba60 0x7f7984071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53624/0 (socket says 172.21.15.107:53624) 2023-10-25T00:15:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f7989187640 1 -- 172.21.15.107:0/1615047421 learned_addr learned my addr 172.21.15.107:0/1615047421 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a98a640 1 -- 172.21.15.107:0/1615047421 --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798406be40 con 0x7f79840723a0 2023-10-25T00:15:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a98a640 1 -- 172.21.15.107:0/1615047421 --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79840c25a0 con 0x7f798406ba60 2023-10-25T00:15:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a189640 1 --2- 172.21.15.107:0/1615047421 >> v2:172.21.15.110:3300/0 conn(0x7f79840c6590 0x7f7984152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:26.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a98a640 1 -- 172.21.15.107:0/1615047421 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7984155ba0 con 0x7f79840c6590 2023-10-25T00:15:26.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f7988986640 1 -- 172.21.15.107:0/1615047421 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144729662 0 0) 0x7f798406be40 con 0x7f79840723a0 2023-10-25T00:15:26.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f7988986640 1 -- 172.21.15.107:0/1615047421 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7970003530 con 0x7f79840723a0 2023-10-25T00:15:26.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a189640 1 -- 172.21.15.107:0/1615047421 >> v2:172.21.15.107:3300/0 conn(0x7f798406ba60 msgr2=0x7f7984071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a189640 1 --2- 172.21.15.107:0/1615047421 >> v2:172.21.15.107:3300/0 conn(0x7f798406ba60 0x7f7984071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a189640 1 -- 172.21.15.107:0/1615047421 >> v1:172.21.15.107:6789/0 conn(0x7f79840723a0 legacy=0x7f7984152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:26.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.095+0000 7f798a189640 1 -- 172.21.15.107:0/1615047421 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7984153280 con 0x7f79840c6590 2023-10-25T00:15:26.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.103+0000 7f798a189640 1 --2- 172.21.15.107:0/1615047421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79840c6590 0x7f7984152d40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f79840a66f0 tx=0x7f797c01c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e5ca9ad719e9e83 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.103+0000 7f7988986640 1 -- 172.21.15.107:0/1615047421 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c0a5960 con 0x7f79840c6590 2023-10-25T00:15:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.103+0000 7f7988986640 1 -- 172.21.15.107:0/1615047421 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c0a5ac0 con 0x7f79840c6590 2023-10-25T00:15:26.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.103+0000 7f7988986640 1 -- 172.21.15.107:0/1615047421 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c0a5e50 con 0x7f79840c6590 2023-10-25T00:15:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.103+0000 7f798a98a640 1 -- 172.21.15.107:0/1615047421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79840c6590 msgr2=0x7f7984152d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:26.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.103+0000 7f798a98a640 1 --2- 172.21.15.107:0/1615047421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79840c6590 0x7f7984152d40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f79840a66f0 tx=0x7f797c01c8a0 comp rx=0 tx=0).stop 2023-10-25T00:15:26.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 -- 172.21.15.107:0/1615047421 shutdown_connections 2023-10-25T00:15:26.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 --2- 172.21.15.107:0/1615047421 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79840c6590 0x7f7984152d40 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:26.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 --2- 172.21.15.107:0/1615047421 >> v2:172.21.15.107:3300/0 conn(0x7f798406ba60 0x7f7984071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:26.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 -- 172.21.15.107:0/1615047421 >> 172.21.15.107:0/1615047421 conn(0x7f79840c6cc0 msgr2=0x7f79840c70d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:26.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 -- 172.21.15.107:0/1615047421 shutdown_connections 2023-10-25T00:15:26.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 -- 172.21.15.107:0/1615047421 wait complete. 2023-10-25T00:15:26.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 Processor -- start 2023-10-25T00:15:26.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 -- start start 2023-10-25T00:15:26.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f798406ba60 0x7f79840a3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:26.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79840723a0 0x7f798414cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:26.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79840c6590 0x7f798414d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:26.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79840a4260 con 0x7f79840723a0 2023-10-25T00:15:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79841f48a0 con 0x7f798406ba60 2023-10-25T00:15:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f798a98a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7984152520 con 0x7f79840c6590 2023-10-25T00:15:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f7989187640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79840723a0 0x7f798414cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.107+0000 7f7989187640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79840723a0 0x7f798414cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53628/0 (socket says 172.21.15.107:53628) 2023-10-25T00:15:26.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f7989187640 1 -- 172.21.15.107:0/381214657 learned_addr learned my addr 172.21.15.107:0/381214657 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:26.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f7989988640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f798406ba60 0x7f79840a3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f7989187640 1 -- 172.21.15.107:0/381214657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79840c6590 msgr2=0x7f798414d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:15:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f7989187640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79840c6590 0x7f798414d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f7989187640 1 -- 172.21.15.107:0/381214657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f798406ba60 msgr2=0x7f79840a3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:26.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f7989187640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f798406ba60 0x7f79840a3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f7989187640 1 -- 172.21.15.107:0/381214657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c009a70 con 0x7f79840723a0 2023-10-25T00:15:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f7989187640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79840723a0 0x7f798414cae0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f797400c520 tx=0x7f797400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f797a7fc640 1 -- 172.21.15.107:0/381214657 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79740074e0 con 0x7f79840723a0 2023-10-25T00:15:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f797a7fc640 1 -- 172.21.15.107:0/381214657 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7974007640 con 0x7f79840723a0 2023-10-25T00:15:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f797a7fc640 1 -- 172.21.15.107:0/381214657 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79740049f0 con 0x7f79840723a0 2023-10-25T00:15:26.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f798a98a640 1 -- 172.21.15.107:0/381214657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79841f4af0 con 0x7f79840723a0 2023-10-25T00:15:26.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f798a98a640 1 -- 172.21.15.107:0/381214657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79841f50b0 con 0x7f79840723a0 2023-10-25T00:15:26.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.111+0000 7f797a7fc640 1 -- 172.21.15.107:0/381214657 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7974004b50 con 0x7f79840723a0 2023-10-25T00:15:26.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.115+0000 7f797a7fc640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7968065e60 0x7f7968068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:26.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.115+0000 7f797a7fc640 1 -- 172.21.15.107:0/381214657 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7f7974083970 con 0x7f79840723a0 2023-10-25T00:15:26.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.115+0000 7f798a98a640 1 -- 172.21.15.107:0/381214657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79841f1330 con 0x7f79840723a0 2023-10-25T00:15:26.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.115+0000 7f7989988640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7968065e60 0x7f7968068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:26.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.119+0000 7f7989988640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7968065e60 0x7f7968068320 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7980009ac0 tx=0x7f7980009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:26.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.131+0000 7f797a7fc640 1 -- 172.21.15.107:0/381214657 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7974018070 con 0x7f79840723a0 2023-10-25T00:15:26.980 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:15:26.979+0000 7f5ea5493640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:15:27.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.999+0000 7f798a98a640 1 -- 172.21.15.107:0/381214657 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f798404e050 con 0x7f79840723a0 2023-10-25T00:15:27.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:26.999+0000 7f797a7fc640 1 -- 172.21.15.107:0/381214657 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+4258 (secure 0 0 0) 0x7f7974050080 con 0x7f79840723a0 2023-10-25T00:15:27.002 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:27.002 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:27.002 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 188 2023-10-25T00:15:27.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.003+0000 7f798a98a640 1 -- 172.21.15.107:0/381214657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7968065e60 msgr2=0x7f7968068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:27.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.003+0000 7f798a98a640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7968065e60 0x7f7968068320 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7980009ac0 tx=0x7f7980009190 comp rx=0 tx=0).stop 2023-10-25T00:15:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.003+0000 7f798a98a640 1 -- 172.21.15.107:0/381214657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79840723a0 msgr2=0x7f798414cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.003+0000 7f798a98a640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79840723a0 0x7f798414cae0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f797400c520 tx=0x7f797400c9e0 comp rx=0 tx=0).stop 2023-10-25T00:15:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.003+0000 7f798a98a640 1 -- 172.21.15.107:0/381214657 shutdown_connections 2023-10-25T00:15:27.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.003+0000 7f798a98a640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7968065e60 0x7f7968068320 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.003+0000 7f798a98a640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79840c6590 0x7f798414d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.007+0000 7f798a98a640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79840723a0 0x7f798414cae0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.007+0000 7f798a98a640 1 --2- 172.21.15.107:0/381214657 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f798406ba60 0x7f79840a3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:27.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.007+0000 7f798a98a640 1 -- 172.21.15.107:0/381214657 >> 172.21.15.107:0/381214657 conn(0x7f79840c6cc0 msgr2=0x7f798415c100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:27.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.011+0000 7f798a98a640 1 -- 172.21.15.107:0/381214657 shutdown_connections 2023-10-25T00:15:27.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:27.011+0000 7f798a98a640 1 -- 172.21.15.107:0/381214657 wait complete. 2023-10-25T00:15:27.049 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=6976 2023-10-25T00:15:27.049 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.d 2023-10-25T00:15:27.679 INFO:tasks.ceph.mds.b.smithi110.stderr:2023-10-25T00:15:27.676+0000 7fdb068dd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:15:31.088 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2023-10-25T00:15:31.089 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:15:31.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:15:31.093 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:15:31.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:31.229 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:15:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.439+0000 7ff14ffff640 1 Processor -- start 2023-10-25T00:15:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.439+0000 7ff14ffff640 1 -- start start 2023-10-25T00:15:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.439+0000 7ff14ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff14800bcc0 0x7ff1480d4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:31.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.439+0000 7ff14ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff148006380 0x7ff1480d4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:31.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.439+0000 7ff14ffff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff148007ee0 con 0x7ff14800c0a0 2023-10-25T00:15:31.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.439+0000 7ff14ffff640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14804c040 con 0x7ff148006380 2023-10-25T00:15:31.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.439+0000 7ff14ffff640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1480078d0 con 0x7ff14800bcc0 2023-10-25T00:15:31.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.443+0000 7ff14e7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff14800c0a0 0x7ff1480d7420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:38964/0 (socket says 172.21.15.107:38964) 2023-10-25T00:15:31.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.443+0000 7ff14e7fc640 1 -- 172.21.15.107:0/3184618021 learned_addr learned my addr 172.21.15.107:0/3184618021 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:31.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.443+0000 7ff14effd640 1 --2- 172.21.15.107:0/3184618021 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff14800bcc0 0x7ff1480d4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:31.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.443+0000 7ff14dffb640 1 -- 172.21.15.107:0/3184618021 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295615732 0 0) 0x7ff148007ee0 con 0x7ff14800c0a0 2023-10-25T00:15:31.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.443+0000 7ff14dffb640 1 -- 172.21.15.107:0/3184618021 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13c003530 con 0x7ff14800c0a0 2023-10-25T00:15:31.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.443+0000 7ff14effd640 1 -- 172.21.15.107:0/3184618021 >> v2:172.21.15.110:3300/0 conn(0x7ff148006380 msgr2=0x7ff1480d4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:15:31.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.443+0000 7ff14effd640 1 --2- 172.21.15.107:0/3184618021 >> v2:172.21.15.110:3300/0 conn(0x7ff148006380 0x7ff1480d4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:31.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.443+0000 7ff14effd640 1 -- 172.21.15.107:0/3184618021 >> v1:172.21.15.107:6789/0 conn(0x7ff14800c0a0 legacy=0x7ff1480d7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:31.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.443+0000 7ff14effd640 1 -- 172.21.15.107:0/3184618021 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14804bd60 con 0x7ff14800bcc0 2023-10-25T00:15:31.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.451+0000 7ff14effd640 1 --2- 172.21.15.107:0/3184618021 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff14800bcc0 0x7ff1480d4810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff1380097c0 tx=0x7ff13801c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c870e352b4f84b70 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:31.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.451+0000 7ff14dffb640 1 -- 172.21.15.107:0/3184618021 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1380ab600 con 0x7ff14800bcc0 2023-10-25T00:15:31.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.451+0000 7ff14dffb640 1 -- 172.21.15.107:0/3184618021 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1380a7050 con 0x7ff14800bcc0 2023-10-25T00:15:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.451+0000 7ff14ffff640 1 -- 172.21.15.107:0/3184618021 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff14800bcc0 msgr2=0x7ff1480d4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:31.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.451+0000 7ff14ffff640 1 --2- 172.21.15.107:0/3184618021 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff14800bcc0 0x7ff1480d4810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff1380097c0 tx=0x7ff13801c370 comp rx=0 tx=0).stop 2023-10-25T00:15:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.455+0000 7ff14ffff640 1 -- 172.21.15.107:0/3184618021 shutdown_connections 2023-10-25T00:15:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.455+0000 7ff14ffff640 1 --2- 172.21.15.107:0/3184618021 >> v2:172.21.15.110:3300/0 conn(0x7ff148006380 0x7ff1480d4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.455+0000 7ff14ffff640 1 --2- 172.21.15.107:0/3184618021 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff14800bcc0 0x7ff1480d4810 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.455+0000 7ff14ffff640 1 -- 172.21.15.107:0/3184618021 >> 172.21.15.107:0/3184618021 conn(0x7ff148006c10 msgr2=0x7ff148007020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.455+0000 7ff14ffff640 1 -- 172.21.15.107:0/3184618021 shutdown_connections 2023-10-25T00:15:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.455+0000 7ff14ffff640 1 -- 172.21.15.107:0/3184618021 wait complete. 2023-10-25T00:15:31.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14ffff640 1 Processor -- start 2023-10-25T00:15:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14ffff640 1 -- start start 2023-10-25T00:15:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff148006380 0x7ff148047e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff14800bcc0 0x7ff1480ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:31.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff14800c0a0 0x7ff1480cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:31.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14ffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1480cc640 con 0x7ff14800bcc0 2023-10-25T00:15:31.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14ffff640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1480cd980 con 0x7ff14800c0a0 2023-10-25T00:15:31.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14ffff640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14804aed0 con 0x7ff148006380 2023-10-25T00:15:31.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff14800c0a0 0x7ff1480cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:31.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff14800c0a0 0x7ff1480cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:41740/0 (socket says 172.21.15.107:41740) 2023-10-25T00:15:31.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14f7fe640 1 -- 172.21.15.107:0/3251624285 learned_addr learned my addr 172.21.15.107:0/3251624285 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:31.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14f7fe640 1 -- 172.21.15.107:0/3251624285 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff148006380 msgr2=0x7ff148047e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-25T00:15:31.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14f7fe640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff148006380 0x7ff148047e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:31.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14f7fe640 1 -- 172.21.15.107:0/3251624285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff14800bcc0 msgr2=0x7ff1480ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:31.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14f7fe640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff14800bcc0 0x7ff1480ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:31.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14f7fe640 1 -- 172.21.15.107:0/3251624285 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff138009420 con 0x7ff14800c0a0 2023-10-25T00:15:31.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.459+0000 7ff14f7fe640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff14800c0a0 0x7ff1480cd440 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff150000b70 tx=0x7ff150069810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:31.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.463+0000 7ff1477fe640 1 -- 172.21.15.107:0/3251624285 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1500723c0 con 0x7ff14800c0a0 2023-10-25T00:15:31.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.463+0000 7ff1477fe640 1 -- 172.21.15.107:0/3251624285 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff150072580 con 0x7ff14800c0a0 2023-10-25T00:15:31.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.463+0000 7ff1477fe640 1 -- 172.21.15.107:0/3251624285 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1500729f0 con 0x7ff14800c0a0 2023-10-25T00:15:31.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.463+0000 7ff14ffff640 1 -- 172.21.15.107:0/3251624285 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1480cbff0 con 0x7ff14800c0a0 2023-10-25T00:15:31.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.463+0000 7ff14ffff640 1 -- 172.21.15.107:0/3251624285 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14802c550 con 0x7ff14800c0a0 2023-10-25T00:15:31.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.463+0000 7ff1477fe640 1 -- 172.21.15.107:0/3251624285 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff150093400 con 0x7ff14800c0a0 2023-10-25T00:15:31.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.467+0000 7ff14ffff640 1 -- 172.21.15.107:0/3251624285 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff134005260 con 0x7ff14800c0a0 2023-10-25T00:15:31.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.467+0000 7ff1477fe640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff130065f30 0x7ff1300683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:31.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.467+0000 7ff1477fe640 1 -- 172.21.15.107:0/3251624285 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7ff1500fe6d0 con 0x7ff14800c0a0 2023-10-25T00:15:31.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.467+0000 7ff14effd640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff130065f30 0x7ff1300683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:31.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.467+0000 7ff14effd640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff130065f30 0x7ff1300683f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff1380ad000 tx=0x7ff1380a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:31.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.475+0000 7ff1477fe640 1 -- 172.21.15.107:0/3251624285 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff1500cae20 con 0x7ff14800c0a0 2023-10-25T00:15:31.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.979+0000 7ff14ffff640 1 -- 172.21.15.107:0/3251624285 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff134005a90 con 0x7ff14800c0a0 2023-10-25T00:15:31.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.983+0000 7ff1477fe640 1 -- 172.21.15.107:0/3251624285 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+5015 (secure 0 0 0) 0x7ff1500ce440 con 0x7ff14800c0a0 2023-10-25T00:15:31.984 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:31.984 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:31.985 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 189 2023-10-25T00:15:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 -- 172.21.15.107:0/3251624285 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff130065f30 msgr2=0x7ff1300683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff130065f30 0x7ff1300683f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff1380ad000 tx=0x7ff1380a6000 comp rx=0 tx=0).stop 2023-10-25T00:15:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 -- 172.21.15.107:0/3251624285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff14800c0a0 msgr2=0x7ff1480cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff14800c0a0 0x7ff1480cd440 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff150000b70 tx=0x7ff150069810 comp rx=0 tx=0).stop 2023-10-25T00:15:31.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 -- 172.21.15.107:0/3251624285 shutdown_connections 2023-10-25T00:15:31.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff14800c0a0 0x7ff1480cd440 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:31.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff14800bcc0 0x7ff1480ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:31.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff130065f30 0x7ff1300683f0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:31.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 --2- 172.21.15.107:0/3251624285 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff148006380 0x7ff148047e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:31.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 -- 172.21.15.107:0/3251624285 >> 172.21.15.107:0/3251624285 conn(0x7ff148006c10 msgr2=0x7ff1480d45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:31.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 -- 172.21.15.107:0/3251624285 shutdown_connections 2023-10-25T00:15:31.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:31.991+0000 7ff14ffff640 1 -- 172.21.15.107:0/3251624285 wait complete. 2023-10-25T00:15:32.032 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2023-10-25T00:15:32.032 INFO:tasks.mds_thrash.fs.[a]:waiting for 95 secs before thrashing 2023-10-25T00:15:33.050 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2023-10-25T00:15:33.050 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-25T00:15:33.050 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-25T00:15:33.053 INFO:tasks.ceph.mds.d:Started 2023-10-25T00:15:33.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:15:33.125 INFO:tasks.ceph.mds.d.smithi110.stdout:starting mds.d at 2023-10-25T00:15:33.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.327+0000 7fb7772be640 1 Processor -- start 2023-10-25T00:15:33.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.327+0000 7fb7772be640 1 -- start start 2023-10-25T00:15:33.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.327+0000 7fb7772be640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb770152040 0x7fb7700c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:33.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.327+0000 7fb7772be640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb770155680 con 0x7fb77006a4b0 2023-10-25T00:15:33.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.327+0000 7fb7772be640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb770154a70 con 0x7fb7700c8100 2023-10-25T00:15:33.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.327+0000 7fb7772be640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb770154d50 con 0x7fb770152040 2023-10-25T00:15:33.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.327+0000 7fb77585b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb77006a4b0 0x7fb7700ac410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35612/0 (socket says 172.21.15.107:35612) 2023-10-25T00:15:33.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.327+0000 7fb77585b640 1 -- 172.21.15.107:0/510497969 learned_addr learned my addr 172.21.15.107:0/510497969 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:15:33.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.331+0000 7fb77505a640 1 --2- 172.21.15.107:0/510497969 >> v2:172.21.15.110:3300/0 conn(0x7fb770152040 0x7fb7700c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:33.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.331+0000 7fb767fff640 1 -- 172.21.15.107:0/510497969 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879011599 0 0) 0x7fb770155680 con 0x7fb77006a4b0 2023-10-25T00:15:33.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.331+0000 7fb767fff640 1 -- 172.21.15.107:0/510497969 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb75c003530 con 0x7fb77006a4b0 2023-10-25T00:15:33.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.331+0000 7fb77505a640 1 -- 172.21.15.107:0/510497969 >> v1:172.21.15.110:6789/0 conn(0x7fb7700c8100 legacy=0x7fb7700abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:33.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.331+0000 7fb77505a640 1 -- 172.21.15.107:0/510497969 >> v1:172.21.15.107:6789/0 conn(0x7fb77006a4b0 legacy=0x7fb7700ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:33.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.331+0000 7fb77505a640 1 -- 172.21.15.107:0/510497969 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb770155030 con 0x7fb770152040 2023-10-25T00:15:33.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.339+0000 7fb77505a640 1 --2- 172.21.15.107:0/510497969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb770152040 0x7fb7700c4b80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009740 tx=0x7fb76c01bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=391fae46395731c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:33.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.343+0000 7fb767fff640 1 -- 172.21.15.107:0/510497969 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0a5640 con 0x7fb770152040 2023-10-25T00:15:33.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.343+0000 7fb767fff640 1 -- 172.21.15.107:0/510497969 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c0a57a0 con 0x7fb770152040 2023-10-25T00:15:33.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.343+0000 7fb767fff640 1 -- 172.21.15.107:0/510497969 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0a5b30 con 0x7fb770152040 2023-10-25T00:15:33.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.347+0000 7fb7772be640 1 -- 172.21.15.107:0/510497969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb770152040 msgr2=0x7fb7700c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:33.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.347+0000 7fb7772be640 1 --2- 172.21.15.107:0/510497969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb770152040 0x7fb7700c4b80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009740 tx=0x7fb76c01bf00 comp rx=0 tx=0).stop 2023-10-25T00:15:33.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.347+0000 7fb7772be640 1 -- 172.21.15.107:0/510497969 shutdown_connections 2023-10-25T00:15:33.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.347+0000 7fb7772be640 1 --2- 172.21.15.107:0/510497969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb770152040 0x7fb7700c4b80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:33.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.347+0000 7fb7772be640 1 -- 172.21.15.107:0/510497969 >> 172.21.15.107:0/510497969 conn(0x7fb7700ca200 msgr2=0x7fb7700ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:33.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.347+0000 7fb7772be640 1 -- 172.21.15.107:0/510497969 shutdown_connections 2023-10-25T00:15:33.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.347+0000 7fb7772be640 1 -- 172.21.15.107:0/510497969 wait complete. 2023-10-25T00:15:33.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb7772be640 1 Processor -- start 2023-10-25T00:15:33.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb7772be640 1 -- start start 2023-10-25T00:15:33.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb7772be640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb77006a4b0 0x7fb770145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:33.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb7772be640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7700c8100 0x7fb7701464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:33.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb77505a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb77006a4b0 0x7fb770145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:33.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb7772be640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb770152040 0x7fb7701f55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:33.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb7772be640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb770160190 con 0x7fb770152040 2023-10-25T00:15:33.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb77505a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb77006a4b0 0x7fb770145f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:35220/0 (socket says 172.21.15.107:35220) 2023-10-25T00:15:33.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb77505a640 1 -- 172.21.15.107:0/74080765 learned_addr learned my addr 172.21.15.107:0/74080765 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:15:33.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb774859640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7700c8100 0x7fb7701464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:33.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb7772be640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb770146a00 con 0x7fb7700c8100 2023-10-25T00:15:33.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb7772be640 1 -- 172.21.15.107:0/74080765 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7701f64c0 con 0x7fb77006a4b0 2023-10-25T00:15:33.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb77585b640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb770152040 0x7fb7701f55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb774859640 1 -- 172.21.15.107:0/74080765 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb77006a4b0 msgr2=0x7fb770145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb774859640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb77006a4b0 0x7fb770145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb774859640 1 -- 172.21.15.107:0/74080765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb770152040 msgr2=0x7fb7701f55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:33.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb774859640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb770152040 0x7fb7701f55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:33.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb774859640 1 -- 172.21.15.107:0/74080765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c009420 con 0x7fb7700c8100 2023-10-25T00:15:33.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.351+0000 7fb774859640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7700c8100 0x7fb7701464c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb7700c8560 tx=0x7fb76000c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:33.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.359+0000 7fb765ffb640 1 -- 172.21.15.107:0/74080765 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb760005480 con 0x7fb7700c8100 2023-10-25T00:15:33.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.359+0000 7fb7772be640 1 -- 172.21.15.107:0/74080765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb77014aa60 con 0x7fb7700c8100 2023-10-25T00:15:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.359+0000 7fb7772be640 1 -- 172.21.15.107:0/74080765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77014c170 con 0x7fb7700c8100 2023-10-25T00:15:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.359+0000 7fb765ffb640 1 -- 172.21.15.107:0/74080765 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb760003600 con 0x7fb7700c8100 2023-10-25T00:15:33.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.359+0000 7fb765ffb640 1 -- 172.21.15.107:0/74080765 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7600039d0 con 0x7fb7700c8100 2023-10-25T00:15:33.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.359+0000 7fb7772be640 1 -- 172.21.15.107:0/74080765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb750005260 con 0x7fb7700c8100 2023-10-25T00:15:33.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.359+0000 7fb765ffb640 1 -- 172.21.15.107:0/74080765 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb760003b30 con 0x7fb7700c8100 2023-10-25T00:15:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.363+0000 7fb765ffb640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb754065f30 0x7fb7540683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:15:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.363+0000 7fb765ffb640 1 -- 172.21.15.107:0/74080765 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7fb7600841b0 con 0x7fb7700c8100 2023-10-25T00:15:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.363+0000 7fb77505a640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb754065f30 0x7fb7540683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:15:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.363+0000 7fb77505a640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb754065f30 0x7fb7540683f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb76c01bf40 tx=0x7fb76c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:15:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.367+0000 7fb765ffb640 1 -- 172.21.15.107:0/74080765 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb760050970 con 0x7fb7700c8100 2023-10-25T00:15:33.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.815+0000 7fb7772be640 1 -- 172.21.15.107:0/74080765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb750005a90 con 0x7fb7700c8100 2023-10-25T00:15:33.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.815+0000 7fb765ffb640 1 -- 172.21.15.107:0/74080765 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5767 (secure 0 0 0) 0x7fb760053f90 con 0x7fb7700c8100 2023-10-25T00:15:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:15:33.816 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:15:33.817 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 190 2023-10-25T00:15:33.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 -- 172.21.15.107:0/74080765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb754065f30 msgr2=0x7fb7540683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:33.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb754065f30 0x7fb7540683f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb76c01bf40 tx=0x7fb76c008040 comp rx=0 tx=0).stop 2023-10-25T00:15:33.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 -- 172.21.15.107:0/74080765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7700c8100 msgr2=0x7fb7701464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:15:33.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7700c8100 0x7fb7701464c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb7700c8560 tx=0x7fb76000c930 comp rx=0 tx=0).stop 2023-10-25T00:15:33.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 -- 172.21.15.107:0/74080765 shutdown_connections 2023-10-25T00:15:33.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb770152040 0x7fb7701f55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:33.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb7700c8100 0x7fb7701464c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:33.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb754065f30 0x7fb7540683f0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:33.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 --2- 172.21.15.107:0/74080765 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb77006a4b0 0x7fb770145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:15:33.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 -- 172.21.15.107:0/74080765 >> 172.21.15.107:0/74080765 conn(0x7fb7700ca200 msgr2=0x7fb7700adde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:15:33.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 -- 172.21.15.107:0/74080765 shutdown_connections 2023-10-25T00:15:33.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:15:33.819+0000 7fb7772be640 1 -- 172.21.15.107:0/74080765 wait complete. 2023-10-25T00:15:33.858 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2023-10-25T00:15:33.858 INFO:tasks.mds_thrash.fs.[b]:waiting for 109 secs before thrashing 2023-10-25T00:16:47.571 INFO:tasks.workunit.client.1.smithi110.stdout:fs setup took 1262 secs 2023-10-25T00:16:51.026 INFO:tasks.workunit.client.1.smithi110.stdout:Syncing()...3 sec 2023-10-25T00:16:51.027 INFO:tasks.workunit.client.1.smithi110.stdout:Starting Actual Benchmark At: Wed Oct 25 00:16:51 2023 2023-10-25T00:16:51.027 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:17:07.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:07.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.973+0000 7fb24a833640 1 Processor -- start 2023-10-25T00:17:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.977+0000 7fb24a833640 1 -- start start 2023-10-25T00:17:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.977+0000 7fb24a833640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2440c5680 0x7fb244154930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:07.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.977+0000 7fb24a833640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb244154e70 0x7fb2441552e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.977+0000 7fb24a833640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2440c4040 con 0x7fb2440c2c60 2023-10-25T00:17:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.977+0000 7fb24a833640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24406c980 con 0x7fb244154e70 2023-10-25T00:17:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.977+0000 7fb24a833640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440c3bd0 con 0x7fb2440c5680 2023-10-25T00:17:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.977+0000 7fb249030640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2440c5680 0x7fb244154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.977+0000 7fb249030640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2440c5680 0x7fb244154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:41870/0 (socket says 172.21.15.107:41870) 2023-10-25T00:17:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.977+0000 7fb249030640 1 -- 172.21.15.107:0/975441008 learned_addr learned my addr 172.21.15.107:0/975441008 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:07.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.977+0000 7fb24a032640 1 --2- 172.21.15.107:0/975441008 >> v2:172.21.15.107:3300/0 conn(0x7fb244154e70 0x7fb2441552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.981+0000 7fb249030640 1 -- 172.21.15.107:0/975441008 >> v2:172.21.15.107:3300/0 conn(0x7fb244154e70 msgr2=0x7fb2441552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.981+0000 7fb249030640 1 --2- 172.21.15.107:0/975441008 >> v2:172.21.15.107:3300/0 conn(0x7fb244154e70 0x7fb2441552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.981+0000 7fb249030640 1 -- 172.21.15.107:0/975441008 >> v1:172.21.15.107:6789/0 conn(0x7fb2440c2c60 legacy=0x7fb2440a8c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:07.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.981+0000 7fb249030640 1 -- 172.21.15.107:0/975441008 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2440c4290 con 0x7fb2440c5680 2023-10-25T00:17:07.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.981+0000 7fb24a032640 1 --2- 172.21.15.107:0/975441008 >> v2:172.21.15.107:3300/0 conn(0x7fb244154e70 0x7fb2441552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:17:07.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:07.981+0000 7fb249030640 1 --2- 172.21.15.107:0/975441008 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2440c5680 0x7fb244154930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb2340091e0 tx=0x7fb23401e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c12d515787bca06f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:08.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.001+0000 7fb24882f640 1 -- 172.21.15.107:0/975441008 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2340a7770 con 0x7fb2440c5680 2023-10-25T00:17:08.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.001+0000 7fb24882f640 1 -- 172.21.15.107:0/975441008 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2340a78d0 con 0x7fb2440c5680 2023-10-25T00:17:08.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.005+0000 7fb24a833640 1 -- 172.21.15.107:0/975441008 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2440c5680 msgr2=0x7fb244154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:08.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.005+0000 7fb24a833640 1 --2- 172.21.15.107:0/975441008 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2440c5680 0x7fb244154930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb2340091e0 tx=0x7fb23401e720 comp rx=0 tx=0).stop 2023-10-25T00:17:08.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.009+0000 7fb24a833640 1 -- 172.21.15.107:0/975441008 shutdown_connections 2023-10-25T00:17:08.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.009+0000 7fb24a833640 1 --2- 172.21.15.107:0/975441008 >> v2:172.21.15.107:3300/0 conn(0x7fb244154e70 0x7fb2441552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:08.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.009+0000 7fb24a833640 1 --2- 172.21.15.107:0/975441008 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2440c5680 0x7fb244154930 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:08.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.009+0000 7fb24a833640 1 -- 172.21.15.107:0/975441008 >> 172.21.15.107:0/975441008 conn(0x7fb2440a77e0 msgr2=0x7fb2440a7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:08.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.013+0000 7fb24a833640 1 -- 172.21.15.107:0/975441008 shutdown_connections 2023-10-25T00:17:08.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.013+0000 7fb24a833640 1 -- 172.21.15.107:0/975441008 wait complete. 2023-10-25T00:17:08.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.017+0000 7fb24a833640 1 Processor -- start 2023-10-25T00:17:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.017+0000 7fb24a833640 1 -- start start 2023-10-25T00:17:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.017+0000 7fb24a833640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2440c2c60 0x7fb2441fa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.017+0000 7fb24a833640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2440c5680 0x7fb2440a8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.017+0000 7fb24a833640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb244154e70 0x7fb2441f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.017+0000 7fb24a833640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2441fadf0 con 0x7fb244154e70 2023-10-25T00:17:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.017+0000 7fb24a833640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440ab900 con 0x7fb2440c5680 2023-10-25T00:17:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.017+0000 7fb24a833640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2440ab620 con 0x7fb2440c2c60 2023-10-25T00:17:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.017+0000 7fb249030640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2440c5680 0x7fb2440a8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.017+0000 7fb249030640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2440c5680 0x7fb2440a8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:36326/0 (socket says 172.21.15.107:36326) 2023-10-25T00:17:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.017+0000 7fb249030640 1 -- 172.21.15.107:0/2811525968 learned_addr learned my addr 172.21.15.107:0/2811525968 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:08.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.021+0000 7fb24a032640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb244154e70 0x7fb2441f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.021+0000 7fb24a032640 1 -- 172.21.15.107:0/2811525968 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2440c2c60 msgr2=0x7fb2441fa8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:17:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.021+0000 7fb24a032640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2440c2c60 0x7fb2441fa8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.021+0000 7fb24a032640 1 -- 172.21.15.107:0/2811525968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2440c5680 msgr2=0x7fb2440a8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.021+0000 7fb24a032640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2440c5680 0x7fb2440a8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:08.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.021+0000 7fb24a032640 1 -- 172.21.15.107:0/2811525968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c009a70 con 0x7fb244154e70 2023-10-25T00:17:08.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.021+0000 7fb24a032640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb244154e70 0x7fb2441f6e20 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00b360 tx=0x7fb23c00b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:08.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.025+0000 7fb23a7fc640 1 -- 172.21.15.107:0/2811525968 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c010040 con 0x7fb244154e70 2023-10-25T00:17:08.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.025+0000 7fb24a833640 1 -- 172.21.15.107:0/2811525968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb234009d60 con 0x7fb244154e70 2023-10-25T00:17:08.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.025+0000 7fb24a833640 1 -- 172.21.15.107:0/2811525968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2440a9950 con 0x7fb244154e70 2023-10-25T00:17:08.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.029+0000 7fb23a7fc640 1 -- 172.21.15.107:0/2811525968 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c009d20 con 0x7fb244154e70 2023-10-25T00:17:08.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.029+0000 7fb23a7fc640 1 -- 172.21.15.107:0/2811525968 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c003be0 con 0x7fb244154e70 2023-10-25T00:17:08.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.029+0000 7fb23a7fc640 1 -- 172.21.15.107:0/2811525968 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb23c01d410 con 0x7fb244154e70 2023-10-25T00:17:08.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.029+0000 7fb23a7fc640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb230066120 0x7fb2300685e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:08.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.029+0000 7fb249831640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb230066120 0x7fb2300685e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:08.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.029+0000 7fb249831640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb230066120 0x7fb2300685e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb244071ad0 tx=0x7fb240006b50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:08.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.029+0000 7fb23a7fc640 1 -- 172.21.15.107:0/2811525968 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7fb23c084ee0 con 0x7fb244154e70 2023-10-25T00:17:08.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.029+0000 7fb24a833640 1 -- 172.21.15.107:0/2811525968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24404e050 con 0x7fb244154e70 2023-10-25T00:17:08.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:08.037+0000 7fb23a7fc640 1 -- 172.21.15.107:0/2811525968 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb23c0515f0 con 0x7fb244154e70 2023-10-25T00:17:09.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.109+0000 7fb24a833640 1 -- 172.21.15.107:0/2811525968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb22c0050f0 con 0x7fb244154e70 2023-10-25T00:17:09.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.109+0000 7fb23a7fc640 1 -- 172.21.15.107:0/2811525968 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5767 (secure 0 0 0) 0x7fb23c018070 con 0x7fb244154e70 2023-10-25T00:17:09.115 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:09.116 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:09.116 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 190 2023-10-25T00:17:09.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.125+0000 7fb227fff640 1 -- 172.21.15.107:0/2811525968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb230066120 msgr2=0x7fb2300685e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:09.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.125+0000 7fb227fff640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb230066120 0x7fb2300685e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb244071ad0 tx=0x7fb240006b50 comp rx=0 tx=0).stop 2023-10-25T00:17:09.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.125+0000 7fb227fff640 1 -- 172.21.15.107:0/2811525968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb244154e70 msgr2=0x7fb2441f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:09.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.125+0000 7fb227fff640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb244154e70 0x7fb2441f6e20 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fb23c00b360 tx=0x7fb23c00b820 comp rx=0 tx=0).stop 2023-10-25T00:17:09.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.125+0000 7fb227fff640 1 -- 172.21.15.107:0/2811525968 shutdown_connections 2023-10-25T00:17:09.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.125+0000 7fb227fff640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb244154e70 0x7fb2441f6e20 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:09.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.125+0000 7fb227fff640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb2440c5680 0x7fb2440a8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:09.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.125+0000 7fb227fff640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb230066120 0x7fb2300685e0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:09.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.125+0000 7fb227fff640 1 --2- 172.21.15.107:0/2811525968 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb2440c2c60 0x7fb2441fa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:09.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.125+0000 7fb227fff640 1 -- 172.21.15.107:0/2811525968 >> 172.21.15.107:0/2811525968 conn(0x7fb2440a77e0 msgr2=0x7fb2440a3040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:09.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.133+0000 7fb227fff640 1 -- 172.21.15.107:0/2811525968 shutdown_connections 2023-10-25T00:17:09.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.133+0000 7fb227fff640 1 -- 172.21.15.107:0/2811525968 wait complete. 2023-10-25T00:17:09.240 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2023-10-25T00:17:09.241 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:17:09.241 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:17:09.267 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:17:09.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.701+0000 7fae151a2640 1 Processor -- start 2023-10-25T00:17:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.701+0000 7fae151a2640 1 -- start start 2023-10-25T00:17:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.701+0000 7fae151a2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fae1014eb70 0x7fae100c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.701+0000 7fae151a2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae10070b00 0x7fae101514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.701+0000 7fae151a2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae100c22f0 con 0x7fae10151a20 2023-10-25T00:17:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.701+0000 7fae151a2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae10071700 con 0x7fae10070b00 2023-10-25T00:17:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.701+0000 7fae151a2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae100c25d0 con 0x7fae1014eb70 2023-10-25T00:17:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.701+0000 7fae0f59e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fae10151a20 0x7fae1006df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:40364/0 (socket says 172.21.15.107:40364) 2023-10-25T00:17:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.701+0000 7fae0f59e640 1 -- 172.21.15.107:0/3753758836 learned_addr learned my addr 172.21.15.107:0/3753758836 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:17:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.705+0000 7fae0e59c640 1 --2- 172.21.15.107:0/3753758836 >> v2:172.21.15.107:3300/0 conn(0x7fae10070b00 0x7fae101514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.705+0000 7fae0dd9b640 1 -- 172.21.15.107:0/3753758836 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186522616 0 0) 0x7fae100c22f0 con 0x7fae10151a20 2023-10-25T00:17:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.705+0000 7fae0dd9b640 1 -- 172.21.15.107:0/3753758836 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadf8003530 con 0x7fae10151a20 2023-10-25T00:17:09.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.705+0000 7fae0e59c640 1 -- 172.21.15.107:0/3753758836 >> v2:172.21.15.110:3300/0 conn(0x7fae1014eb70 msgr2=0x7fae100c33e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.705+0000 7fae0e59c640 1 --2- 172.21.15.107:0/3753758836 >> v2:172.21.15.110:3300/0 conn(0x7fae1014eb70 0x7fae100c33e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.705+0000 7fae0e59c640 1 -- 172.21.15.107:0/3753758836 >> v1:172.21.15.110:6789/0 conn(0x7fae10151a20 legacy=0x7fae1006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:09.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.705+0000 7fae0e59c640 1 -- 172.21.15.107:0/3753758836 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae100c2be0 con 0x7fae10070b00 2023-10-25T00:17:09.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.709+0000 7fae0e59c640 1 --2- 172.21.15.107:0/3753758836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae10070b00 0x7fae101514e0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fae100c6730 tx=0x7fae0001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ef29b9457d693a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.709+0000 7fae0dd9b640 1 -- 172.21.15.107:0/3753758836 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae000ab620 con 0x7fae10070b00 2023-10-25T00:17:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.709+0000 7fae0dd9b640 1 -- 172.21.15.107:0/3753758836 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae000ab780 con 0x7fae10070b00 2023-10-25T00:17:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.709+0000 7fae0dd9b640 1 -- 172.21.15.107:0/3753758836 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae000abb10 con 0x7fae10070b00 2023-10-25T00:17:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.709+0000 7fae151a2640 1 -- 172.21.15.107:0/3753758836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae10070b00 msgr2=0x7fae101514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.709+0000 7fae151a2640 1 --2- 172.21.15.107:0/3753758836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae10070b00 0x7fae101514e0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fae100c6730 tx=0x7fae0001c4a0 comp rx=0 tx=0).stop 2023-10-25T00:17:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.709+0000 7fae151a2640 1 -- 172.21.15.107:0/3753758836 shutdown_connections 2023-10-25T00:17:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.709+0000 7fae151a2640 1 --2- 172.21.15.107:0/3753758836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae10070b00 0x7fae101514e0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.709+0000 7fae151a2640 1 --2- 172.21.15.107:0/3753758836 >> v2:172.21.15.110:3300/0 conn(0x7fae1014eb70 0x7fae100c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.709+0000 7fae151a2640 1 -- 172.21.15.107:0/3753758836 >> 172.21.15.107:0/3753758836 conn(0x7fae100c39a0 msgr2=0x7fae100c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.713+0000 7fae151a2640 1 -- 172.21.15.107:0/3753758836 shutdown_connections 2023-10-25T00:17:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.713+0000 7fae151a2640 1 -- 172.21.15.107:0/3753758836 wait complete. 2023-10-25T00:17:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.713+0000 7fae151a2640 1 Processor -- start 2023-10-25T00:17:09.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae151a2640 1 -- start start 2023-10-25T00:17:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae151a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae10070b00 0x7fae1014ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae151a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae1014eb70 0x7fae1014f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae151a2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae10151a20 0x7fae1013ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae151a2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae10142f70 con 0x7fae10151a20 2023-10-25T00:17:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae151a2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae101428f0 con 0x7fae1014eb70 2023-10-25T00:17:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae151a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae101f4d40 con 0x7fae10070b00 2023-10-25T00:17:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae0f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae10151a20 0x7fae1013ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae0ed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae10070b00 0x7fae1014ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae0f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae10151a20 0x7fae1013ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45690/0 (socket says 172.21.15.107:45690) 2023-10-25T00:17:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae0f59e640 1 -- 172.21.15.107:0/3641727701 learned_addr learned my addr 172.21.15.107:0/3641727701 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae0ed9d640 1 -- 172.21.15.107:0/3641727701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae1014eb70 msgr2=0x7fae1014f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae0ed9d640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae1014eb70 0x7fae1014f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae0ed9d640 1 -- 172.21.15.107:0/3641727701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae10151a20 msgr2=0x7fae1013ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae0ed9d640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae10151a20 0x7fae1013ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae0ed9d640 1 -- 172.21.15.107:0/3641727701 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae00009420 con 0x7fae10070b00 2023-10-25T00:17:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae0f59e640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae10151a20 0x7fae1013ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:17:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae0ed9d640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae10070b00 0x7fae1014ff70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae0400b3c0 tx=0x7fae0400b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fadf77fe640 1 -- 172.21.15.107:0/3641727701 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae04011830 con 0x7fae10070b00 2023-10-25T00:17:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fadf77fe640 1 -- 172.21.15.107:0/3641727701 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae04011990 con 0x7fae10070b00 2023-10-25T00:17:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fadf77fe640 1 -- 172.21.15.107:0/3641727701 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae04011d80 con 0x7fae10070b00 2023-10-25T00:17:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae151a2640 1 -- 172.21.15.107:0/3641727701 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae10142b90 con 0x7fae10070b00 2023-10-25T00:17:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.717+0000 7fae151a2640 1 -- 172.21.15.107:0/3641727701 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1014faf0 con 0x7fae10070b00 2023-10-25T00:17:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.721+0000 7fae151a2640 1 -- 172.21.15.107:0/3641727701 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae100746e0 con 0x7fae10070b00 2023-10-25T00:17:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.721+0000 7fadf77fe640 1 -- 172.21.15.107:0/3641727701 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fae0401b460 con 0x7fae10070b00 2023-10-25T00:17:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.721+0000 7fadf77fe640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fadfc065e60 0x7fadfc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.721+0000 7fadf77fe640 1 -- 172.21.15.107:0/3641727701 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7fae040844e0 con 0x7fae10070b00 2023-10-25T00:17:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.725+0000 7fae0e59c640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fadfc065e60 0x7fadfc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:09.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.725+0000 7fae0e59c640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fadfc065e60 0x7fadfc068320 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fae000af000 tx=0x7fae00008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:09.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:09.745+0000 7fadf77fe640 1 -- 172.21.15.107:0/3641727701 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fae04050b40 con 0x7fae10070b00 2023-10-25T00:17:10.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.637+0000 7fae151a2640 1 -- 172.21.15.107:0/3641727701 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae100c1a50 con 0x7fae10070b00 2023-10-25T00:17:10.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.645+0000 7fadf77fe640 1 -- 172.21.15.107:0/3641727701 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5767 (secure 0 0 0) 0x7fae04054160 con 0x7fae10070b00 2023-10-25T00:17:10.648 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:10.648 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:10.649 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 190 2023-10-25T00:17:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.649+0000 7fae151a2640 1 -- 172.21.15.107:0/3641727701 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fadfc065e60 msgr2=0x7fadfc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.649+0000 7fae151a2640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fadfc065e60 0x7fadfc068320 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fae000af000 tx=0x7fae00008040 comp rx=0 tx=0).stop 2023-10-25T00:17:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.649+0000 7fae151a2640 1 -- 172.21.15.107:0/3641727701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae10070b00 msgr2=0x7fae1014ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:10.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.653+0000 7fae151a2640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae10070b00 0x7fae1014ff70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae0400b3c0 tx=0x7fae0400b880 comp rx=0 tx=0).stop 2023-10-25T00:17:10.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.657+0000 7fae151a2640 1 -- 172.21.15.107:0/3641727701 shutdown_connections 2023-10-25T00:17:10.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.657+0000 7fae151a2640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae10151a20 0x7fae1013ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:10.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.657+0000 7fae151a2640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae1014eb70 0x7fae1014f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:10.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.657+0000 7fae151a2640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fadfc065e60 0x7fadfc068320 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:10.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.657+0000 7fae151a2640 1 --2- 172.21.15.107:0/3641727701 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae10070b00 0x7fae1014ff70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:10.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.657+0000 7fae151a2640 1 -- 172.21.15.107:0/3641727701 >> 172.21.15.107:0/3641727701 conn(0x7fae100c39a0 msgr2=0x7fae1014c360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:10.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.657+0000 7fae151a2640 1 -- 172.21.15.107:0/3641727701 shutdown_connections 2023-10-25T00:17:10.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:10.657+0000 7fae151a2640 1 -- 172.21.15.107:0/3641727701 wait complete. 2023-10-25T00:17:10.779 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:17:12.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.309+0000 7f80882de640 1 Processor -- start 2023-10-25T00:17:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.309+0000 7f80882de640 1 -- start start 2023-10-25T00:17:13.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f80882de640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80780dc100 0x7f807800bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f80882de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f807803ed80 0x7f807803f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:13.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f80882de640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80780075e0 con 0x7f807800b5f0 2023-10-25T00:17:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f80882de640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80780df960 con 0x7f80780dc100 2023-10-25T00:17:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f808607a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80780dc100 0x7f807800bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f808607a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f80780dc100 0x7f807800bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51044/0 (socket says 172.21.15.107:51044) 2023-10-25T00:17:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f808607a640 1 -- 172.21.15.107:0/293568999 learned_addr learned my addr 172.21.15.107:0/293568999 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f808687b640 1 --2- 172.21.15.107:0/293568999 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f807803ed80 0x7f807803f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f80882de640 1 -- 172.21.15.107:0/293568999 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8078047d10 con 0x7f807803ed80 2023-10-25T00:17:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f8085078640 1 -- 172.21.15.107:0/293568999 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985736444 0 0) 0x7f80780075e0 con 0x7f807800b5f0 2023-10-25T00:17:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f8085078640 1 -- 172.21.15.107:0/293568999 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f806c003530 con 0x7f807800b5f0 2023-10-25T00:17:13.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f808607a640 1 -- 172.21.15.107:0/293568999 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f807803ed80 msgr2=0x7f807803f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:13.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f808607a640 1 --2- 172.21.15.107:0/293568999 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f807803ed80 0x7f807803f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f808607a640 1 -- 172.21.15.107:0/293568999 >> v1:172.21.15.107:6789/0 conn(0x7f807800b5f0 legacy=0x7f8078047600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.313+0000 7f808607a640 1 -- 172.21.15.107:0/293568999 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807800c120 con 0x7f80780dc100 2023-10-25T00:17:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.317+0000 7f808607a640 1 --2- 172.21.15.107:0/293568999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80780dc100 0x7f807800bbe0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f807c009870 tx=0x7f807c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a1ec73bece0a4c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.317+0000 7f8085078640 1 -- 172.21.15.107:0/293568999 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c0ab600 con 0x7f80780dc100 2023-10-25T00:17:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.317+0000 7f8085078640 1 -- 172.21.15.107:0/293568999 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807c0a7050 con 0x7f80780dc100 2023-10-25T00:17:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.317+0000 7f8085078640 1 -- 172.21.15.107:0/293568999 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c0ab9c0 con 0x7f80780dc100 2023-10-25T00:17:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.321+0000 7f80882de640 1 -- 172.21.15.107:0/293568999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80780dc100 msgr2=0x7f807800bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:13.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.321+0000 7f80882de640 1 --2- 172.21.15.107:0/293568999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80780dc100 0x7f807800bbe0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f807c009870 tx=0x7f807c01c460 comp rx=0 tx=0).stop 2023-10-25T00:17:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.321+0000 7f80882de640 1 -- 172.21.15.107:0/293568999 shutdown_connections 2023-10-25T00:17:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.321+0000 7f80882de640 1 --2- 172.21.15.107:0/293568999 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f807803ed80 0x7f807803f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.321+0000 7f80882de640 1 --2- 172.21.15.107:0/293568999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80780dc100 0x7f807800bbe0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.321+0000 7f80882de640 1 -- 172.21.15.107:0/293568999 >> 172.21.15.107:0/293568999 conn(0x7f8078006470 msgr2=0x7f8078006880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:13.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.321+0000 7f80882de640 1 -- 172.21.15.107:0/293568999 shutdown_connections 2023-10-25T00:17:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.321+0000 7f80882de640 1 -- 172.21.15.107:0/293568999 wait complete. 2023-10-25T00:17:13.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.321+0000 7f80882de640 1 Processor -- start 2023-10-25T00:17:13.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.321+0000 7f80882de640 1 -- start start 2023-10-25T00:17:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.325+0000 7f80882de640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f807800b5f0 0x7f80780414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:13.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.325+0000 7f80882de640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f807803ed80 0x7f8078041f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:13.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.325+0000 7f80882de640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80780dc100 0x7f80780d3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:13.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.325+0000 7f808607a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f807800b5f0 0x7f80780414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:13.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.325+0000 7f808607a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f807800b5f0 0x7f80780414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51060/0 (socket says 172.21.15.107:51060) 2023-10-25T00:17:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.329+0000 7f808687b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80780dc100 0x7f80780d3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.329+0000 7f808687b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80780dc100 0x7f80780d3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:59406/0 (socket says 172.21.15.107:59406) 2023-10-25T00:17:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.329+0000 7f8085879640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f807803ed80 0x7f8078041f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:13.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.329+0000 7f8085879640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f807803ed80 0x7f8078041f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:42426/0 (socket says 172.21.15.107:42426) 2023-10-25T00:17:13.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.337+0000 7f808607a640 1 -- 172.21.15.107:0/1212593282 learned_addr learned my addr 172.21.15.107:0/1212593282 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:13.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.337+0000 7f80882de640 1 -- 172.21.15.107:0/1212593282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80780de3e0 con 0x7f807800b5f0 2023-10-25T00:17:13.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.337+0000 7f80882de640 1 -- 172.21.15.107:0/1212593282 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80780dec70 con 0x7f807803ed80 2023-10-25T00:17:13.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.337+0000 7f80882de640 1 -- 172.21.15.107:0/1212593282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80780ddab0 con 0x7f80780dc100 2023-10-25T00:17:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f808687b640 1 -- 172.21.15.107:0/1212593282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f807803ed80 msgr2=0x7f8078041f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f808687b640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f807803ed80 0x7f8078041f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f808687b640 1 -- 172.21.15.107:0/1212593282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f807800b5f0 msgr2=0x7f80780414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f808687b640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f807800b5f0 0x7f80780414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f808687b640 1 -- 172.21.15.107:0/1212593282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c0094d0 con 0x7f80780dc100 2023-10-25T00:17:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f808687b640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80780dc100 0x7f80780d3a40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f808004eea0 tx=0x7f8080069810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:13.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f8076ffd640 1 -- 172.21.15.107:0/1212593282 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80800723d0 con 0x7f80780dc100 2023-10-25T00:17:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f8076ffd640 1 -- 172.21.15.107:0/1212593282 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8080072530 con 0x7f80780dc100 2023-10-25T00:17:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f8076ffd640 1 -- 172.21.15.107:0/1212593282 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80800728e0 con 0x7f80780dc100 2023-10-25T00:17:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f80882de640 1 -- 172.21.15.107:0/1212593282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80780ddd90 con 0x7f80780dc100 2023-10-25T00:17:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f80882de640 1 -- 172.21.15.107:0/1212593282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807802ccb0 con 0x7f80780dc100 2023-10-25T00:17:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.345+0000 7f8074ff9640 1 -- 172.21.15.107:0/1212593282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805c005260 con 0x7f80780dc100 2023-10-25T00:17:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.349+0000 7f8076ffd640 1 -- 172.21.15.107:0/1212593282 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8080094440 con 0x7f80780dc100 2023-10-25T00:17:13.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.349+0000 7f8076ffd640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8064065e90 0x7f8064068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:13.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.349+0000 7f8076ffd640 1 -- 172.21.15.107:0/1212593282 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7f80800fee30 con 0x7f80780dc100 2023-10-25T00:17:13.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.349+0000 7f808607a640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8064065e90 0x7f8064068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:13.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.349+0000 7f808607a640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8064065e90 0x7f8064068350 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f807c000c10 tx=0x7f807c0a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:13.365+0000 7f8076ffd640 1 -- 172.21.15.107:0/1212593282 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f80800cb650 con 0x7f80780dc100 2023-10-25T00:17:14.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.373+0000 7f8074ff9640 1 -- 172.21.15.107:0/1212593282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f805c0057e0 con 0x7f80780dc100 2023-10-25T00:17:14.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.377+0000 7f8076ffd640 1 -- 172.21.15.107:0/1212593282 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5767 (secure 0 0 0) 0x7f80800cec70 con 0x7f80780dc100 2023-10-25T00:17:14.380 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:14.381 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:14.381 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 190 2023-10-25T00:17:14.382 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 -- 172.21.15.107:0/1212593282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8064065e90 msgr2=0x7f8064068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:14.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8064065e90 0x7f8064068350 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f807c000c10 tx=0x7f807c0a6000 comp rx=0 tx=0).stop 2023-10-25T00:17:14.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 -- 172.21.15.107:0/1212593282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80780dc100 msgr2=0x7f80780d3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:14.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80780dc100 0x7f80780d3a40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f808004eea0 tx=0x7f8080069810 comp rx=0 tx=0).stop 2023-10-25T00:17:14.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 -- 172.21.15.107:0/1212593282 shutdown_connections 2023-10-25T00:17:14.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8064065e90 0x7f8064068350 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:14.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80780dc100 0x7f80780d3a40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f807803ed80 0x7f8078041f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 --2- 172.21.15.107:0/1212593282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f807800b5f0 0x7f80780414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 -- 172.21.15.107:0/1212593282 >> 172.21.15.107:0/1212593282 conn(0x7f8078006470 msgr2=0x7f8078004ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 -- 172.21.15.107:0/1212593282 shutdown_connections 2023-10-25T00:17:14.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:14.381+0000 7f80882de640 1 -- 172.21.15.107:0/1212593282 wait complete. 2023-10-25T00:17:14.484 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:17:16.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa51c939640 1 Processor -- start 2023-10-25T00:17:16.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa51c939640 1 -- start start 2023-10-25T00:17:16.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa51c939640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa51815eb00 0x7fa51806b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:16.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa51c939640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5180aa990 0x7fa518164a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:16.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa51c939640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5181656c0 con 0x7fa51808b9e0 2023-10-25T00:17:16.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa51c939640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa51815eee0 con 0x7fa51815eb00 2023-10-25T00:17:16.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa51c939640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa51806c850 con 0x7fa5180aa990 2023-10-25T00:17:16.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa50dd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5180aa990 0x7fa518164a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:16.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa50dd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5180aa990 0x7fa518164a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:59410/0 (socket says 172.21.15.107:59410) 2023-10-25T00:17:16.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa50dd9b640 1 -- 172.21.15.107:0/133972520 learned_addr learned my addr 172.21.15.107:0/133972520 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:16.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087659593 0 0) 0x7fa5181656c0 con 0x7fa51808b9e0 2023-10-25T00:17:16.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.621+0000 7fa51659c640 1 --2- 172.21.15.107:0/133972520 >> v2:172.21.15.110:3300/0 conn(0x7fa51815eb00 0x7fa51806b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:16.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.717+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa504003510 con 0x7fa51808b9e0 2023-10-25T00:17:16.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2016426545 0 0) 0x7fa504003510 con 0x7fa51808b9e0 2023-10-25T00:17:16.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5180aa990 msgr2=0x7fa518164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:16.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 --2- 172.21.15.107:0/133972520 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa5180aa990 0x7fa518164a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:16.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 >> v2:172.21.15.110:3300/0 conn(0x7fa51815eb00 msgr2=0x7fa51806b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:16.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 --2- 172.21.15.107:0/133972520 >> v2:172.21.15.110:3300/0 conn(0x7fa51815eb00 0x7fa51806b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:16.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa518161560 con 0x7fa51808b9e0 2023-10-25T00:17:16.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fa510001a00 con 0x7fa51808b9e0 2023-10-25T00:17:16.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 >> v1:172.21.15.110:6789/0 conn(0x7fa51808b9e0 legacy=0x7fa518164fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:16.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 --2- 172.21.15.107:0/133972520 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa504003ab0 0x7fa504005f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:16.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 --2- 172.21.15.107:0/133972520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa504006520 0x7fa504008a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:16.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 --2- 172.21.15.107:0/133972520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa504009250 0x7fa50400b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:16.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51815eee0 con 0x7fa504006520 2023-10-25T00:17:16.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50400bf90 con 0x7fa504009250 2023-10-25T00:17:16.729 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50400c150 con 0x7fa504003ab0 2023-10-25T00:17:16.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa51659c640 1 --2- 172.21.15.107:0/133972520 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa504003ab0 0x7fa504005f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:16.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.721+0000 7fa516d9d640 1 --2- 172.21.15.107:0/133972520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa504009250 0x7fa50400b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:16.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.725+0000 7fa516d9d640 1 -- 172.21.15.107:0/133972520 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa504003ab0 msgr2=0x7fa504005f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:16.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.725+0000 7fa516d9d640 1 --2- 172.21.15.107:0/133972520 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa504003ab0 0x7fa504005f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:16.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.725+0000 7fa516d9d640 1 -- 172.21.15.107:0/133972520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa504006520 msgr2=0x7fa504008a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:16.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.725+0000 7fa516d9d640 1 --2- 172.21.15.107:0/133972520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa504006520 0x7fa504008a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:16.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.725+0000 7fa516d9d640 1 -- 172.21.15.107:0/133972520 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa504003510 con 0x7fa504009250 2023-10-25T00:17:16.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.733+0000 7fa516d9d640 1 --2- 172.21.15.107:0/133972520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa504009250 0x7fa50400b6e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa51000c4c0 tx=0x7fa51001ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f61042b3f2d93291 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:16.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.737+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5100a5d10 con 0x7fa504009250 2023-10-25T00:17:16.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.737+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5100a5e90 con 0x7fa504009250 2023-10-25T00:17:16.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.737+0000 7fa515d9b640 1 -- 172.21.15.107:0/133972520 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5100ad640 con 0x7fa504009250 2023-10-25T00:17:16.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.749+0000 7fa51c939640 1 -- 172.21.15.107:0/133972520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa504009250 msgr2=0x7fa50400b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:16.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.749+0000 7fa51c939640 1 --2- 172.21.15.107:0/133972520 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa504009250 0x7fa50400b6e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa51000c4c0 tx=0x7fa51001ce30 comp rx=0 tx=0).stop 2023-10-25T00:17:16.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.749+0000 7fa516d9d640 1 -- 172.21.15.107:0/133972520 reap_dead start 2023-10-25T00:17:16.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.757+0000 7fa51c939640 1 -- 172.21.15.107:0/133972520 shutdown_connections 2023-10-25T00:17:16.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.757+0000 7fa51c939640 1 -- 172.21.15.107:0/133972520 >> 172.21.15.107:0/133972520 conn(0x7fa5180c2950 msgr2=0x7fa5180c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:16.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.757+0000 7fa51c939640 1 -- 172.21.15.107:0/133972520 shutdown_connections 2023-10-25T00:17:16.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.757+0000 7fa51c939640 1 -- 172.21.15.107:0/133972520 wait complete. 2023-10-25T00:17:16.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.761+0000 7fa51c939640 1 Processor -- start 2023-10-25T00:17:16.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.761+0000 7fa51c939640 1 -- start start 2023-10-25T00:17:16.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.761+0000 7fa51c939640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa504009250 0x7fa5181fcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:16.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.761+0000 7fa51c939640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa51808b310 0x7fa5181fd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:16.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.761+0000 7fa51c939640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5181628c0 0x7fa518162d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:16.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.761+0000 7fa51c939640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518160840 con 0x7fa504009250 2023-10-25T00:17:16.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.761+0000 7fa51c939640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181610e0 con 0x7fa5181628c0 2023-10-25T00:17:16.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.761+0000 7fa51c939640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181fd970 con 0x7fa51808b310 2023-10-25T00:17:16.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa51659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa504009250 0x7fa5181fcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:16.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa51659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa504009250 0x7fa5181fcbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51070/0 (socket says 172.21.15.107:51070) 2023-10-25T00:17:16.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa51659c640 1 -- 172.21.15.107:0/1205610263 learned_addr learned my addr 172.21.15.107:0/1205610263 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:16.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa516d9d640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5181628c0 0x7fa518162d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:16.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa516d9d640 1 -- 172.21.15.107:0/1205610263 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa51808b310 msgr2=0x7fa5181fd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:16.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa516d9d640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa51808b310 0x7fa5181fd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:16.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa516d9d640 1 -- 172.21.15.107:0/1205610263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa504009250 msgr2=0x7fa5181fcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:16.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa516d9d640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa504009250 0x7fa5181fcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:16.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa516d9d640 1 -- 172.21.15.107:0/1205610263 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa518160a90 con 0x7fa5181628c0 2023-10-25T00:17:16.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa516d9d640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5181628c0 0x7fa518162d70 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa5100ac000 tx=0x7fa5040037c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:16.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa50f7fe640 1 -- 172.21.15.107:0/1205610263 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5100019c0 con 0x7fa5181628c0 2023-10-25T00:17:16.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa51c939640 1 -- 172.21.15.107:0/1205610263 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa518160d70 con 0x7fa5181628c0 2023-10-25T00:17:16.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa51c939640 1 -- 172.21.15.107:0/1205610263 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5181632b0 con 0x7fa5181628c0 2023-10-25T00:17:16.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.765+0000 7fa50f7fe640 1 -- 172.21.15.107:0/1205610263 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5100a5d20 con 0x7fa5181628c0 2023-10-25T00:17:16.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.769+0000 7fa50f7fe640 1 -- 172.21.15.107:0/1205610263 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5100052b0 con 0x7fa5181628c0 2023-10-25T00:17:16.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.769+0000 7fa51c939640 1 -- 172.21.15.107:0/1205610263 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4fc005260 con 0x7fa5181628c0 2023-10-25T00:17:16.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.773+0000 7fa50f7fe640 1 -- 172.21.15.107:0/1205610263 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa5100b4040 con 0x7fa5181628c0 2023-10-25T00:17:16.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.773+0000 7fa50f7fe640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa4f8065f60 0x7fa4f8068420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:16.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.773+0000 7fa50f7fe640 1 -- 172.21.15.107:0/1205610263 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7fa51011e930 con 0x7fa5181628c0 2023-10-25T00:17:16.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.773+0000 7fa51659c640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa4f8065f60 0x7fa4f8068420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:16.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.773+0000 7fa51659c640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa4f8065f60 0x7fa4f8068420 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa518164150 tx=0x7fa508010040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:16.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:16.785+0000 7fa50f7fe640 1 -- 172.21.15.107:0/1205610263 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa5100eb100 con 0x7fa5181628c0 2023-10-25T00:17:17.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.073+0000 7fa51c939640 1 -- 172.21.15.107:0/1205610263 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4fc005a90 con 0x7fa5181628c0 2023-10-25T00:17:17.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.073+0000 7fa50f7fe640 1 -- 172.21.15.107:0/1205610263 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5767 (secure 0 0 0) 0x7fa51011e6a0 con 0x7fa5181628c0 2023-10-25T00:17:17.077 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:17.077 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:17.078 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 190 2023-10-25T00:17:17.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 -- 172.21.15.107:0/1205610263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa4f8065f60 msgr2=0x7fa4f8068420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:17.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa4f8065f60 0x7fa4f8068420 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa518164150 tx=0x7fa508010040 comp rx=0 tx=0).stop 2023-10-25T00:17:17.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 -- 172.21.15.107:0/1205610263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5181628c0 msgr2=0x7fa518162d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:17.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5181628c0 0x7fa518162d70 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa5100ac000 tx=0x7fa5040037c0 comp rx=0 tx=0).stop 2023-10-25T00:17:17.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 -- 172.21.15.107:0/1205610263 shutdown_connections 2023-10-25T00:17:17.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5181628c0 0x7fa518162d70 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:17.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa4f8065f60 0x7fa4f8068420 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:17.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa51808b310 0x7fa5181fd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:17.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 --2- 172.21.15.107:0/1205610263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa504009250 0x7fa5181fcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:17.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 -- 172.21.15.107:0/1205610263 >> 172.21.15.107:0/1205610263 conn(0x7fa5180c2950 msgr2=0x7fa5180c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:17.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 -- 172.21.15.107:0/1205610263 shutdown_connections 2023-10-25T00:17:17.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:17.077+0000 7fa51c939640 1 -- 172.21.15.107:0/1205610263 wait complete. 2023-10-25T00:17:17.103 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:17:19.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:19.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f1853bae640 1 Processor -- start 2023-10-25T00:17:19.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f1853bae640 1 -- start start 2023-10-25T00:17:19.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f1853bae640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f184c14c470 0x7f184c14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:19.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f1853bae640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c152b30 0x7f184c152f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:19.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f1853bae640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c155cd0 con 0x7f184c06f490 2023-10-25T00:17:19.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f1853bae640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184c1526e0 con 0x7f184c14c470 2023-10-25T00:17:19.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f1853bae640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0c4380 con 0x7f184c152b30 2023-10-25T00:17:19.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f185214b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c152b30 0x7f184c152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:19.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f185214b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c152b30 0x7f184c152f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:59430/0 (socket says 172.21.15.107:59430) 2023-10-25T00:17:19.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f185214b640 1 -- 172.21.15.107:0/1334519583 learned_addr learned my addr 172.21.15.107:0/1334519583 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:19.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f185214b640 1 -- 172.21.15.107:0/1334519583 >> v2:172.21.15.110:3300/0 conn(0x7f184c14c470 msgr2=0x7f184c14c850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:19.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f185214b640 1 --2- 172.21.15.107:0/1334519583 >> v2:172.21.15.110:3300/0 conn(0x7f184c14c470 0x7f184c14c850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:19.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f185214b640 1 -- 172.21.15.107:0/1334519583 >> v1:172.21.15.107:6789/0 conn(0x7f184c06f490 legacy=0x7f184c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:19.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.541+0000 7f185214b640 1 -- 172.21.15.107:0/1334519583 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c0c48b0 con 0x7f184c152b30 2023-10-25T00:17:19.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.545+0000 7f185214b640 1 --2- 172.21.15.107:0/1334519583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c152b30 0x7f184c152f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1844009e50 tx=0x7f184401c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f87532fecafd0bb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:19.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.545+0000 7f1850948640 1 -- 172.21.15.107:0/1334519583 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18440a56c0 con 0x7f184c152b30 2023-10-25T00:17:19.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.545+0000 7f1850948640 1 -- 172.21.15.107:0/1334519583 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18440a5820 con 0x7f184c152b30 2023-10-25T00:17:19.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.545+0000 7f1850948640 1 -- 172.21.15.107:0/1334519583 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18440a5bb0 con 0x7f184c152b30 2023-10-25T00:17:19.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 -- 172.21.15.107:0/1334519583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c152b30 msgr2=0x7f184c152f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:19.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 --2- 172.21.15.107:0/1334519583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c152b30 0x7f184c152f80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1844009e50 tx=0x7f184401c910 comp rx=0 tx=0).stop 2023-10-25T00:17:19.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 -- 172.21.15.107:0/1334519583 shutdown_connections 2023-10-25T00:17:19.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 --2- 172.21.15.107:0/1334519583 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c152b30 0x7f184c152f80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:19.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 --2- 172.21.15.107:0/1334519583 >> v2:172.21.15.110:3300/0 conn(0x7f184c14c470 0x7f184c14c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:19.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 -- 172.21.15.107:0/1334519583 >> 172.21.15.107:0/1334519583 conn(0x7f184c070f60 msgr2=0x7f184c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:19.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 -- 172.21.15.107:0/1334519583 shutdown_connections 2023-10-25T00:17:19.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 -- 172.21.15.107:0/1334519583 wait complete. 2023-10-25T00:17:19.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 Processor -- start 2023-10-25T00:17:19.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 -- start start 2023-10-25T00:17:19.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c06f490 0x7f184c0aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:19.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f184c14c470 0x7f184c14d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:19.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f184c152b30 0x7f184c0c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:19.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c0c7940 con 0x7f184c14c470 2023-10-25T00:17:19.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c14f4e0 con 0x7f184c152b30 2023-10-25T00:17:19.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1853bae640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c14d300 con 0x7f184c06f490 2023-10-25T00:17:19.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1851149640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f184c14c470 0x7f184c14d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:19.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1851149640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f184c14c470 0x7f184c14d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51086/0 (socket says 172.21.15.107:51086) 2023-10-25T00:17:19.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.549+0000 7f1851149640 1 -- 172.21.15.107:0/4197609958 learned_addr learned my addr 172.21.15.107:0/4197609958 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:19.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f185194a640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c06f490 0x7f184c0aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:19.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f1851149640 1 -- 172.21.15.107:0/4197609958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c06f490 msgr2=0x7f184c0aabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:19.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f1851149640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c06f490 0x7f184c0aabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:19.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f1851149640 1 -- 172.21.15.107:0/4197609958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f184c152b30 msgr2=0x7f184c0c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:19.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f1851149640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f184c152b30 0x7f184c0c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:19.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f1851149640 1 -- 172.21.15.107:0/4197609958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1844009b30 con 0x7f184c14c470 2023-10-25T00:17:19.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f1851149640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f184c14c470 0x7f184c14d820 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f183c00c500 tx=0x7f183c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:19.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f18427fc640 1 -- 172.21.15.107:0/4197609958 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c005220 con 0x7f184c14c470 2023-10-25T00:17:19.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f18427fc640 1 -- 172.21.15.107:0/4197609958 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183c005380 con 0x7f184c14c470 2023-10-25T00:17:19.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f1853bae640 1 -- 172.21.15.107:0/4197609958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c14dd60 con 0x7f184c14c470 2023-10-25T00:17:19.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f1853bae640 1 -- 172.21.15.107:0/4197609958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184c0c75b0 con 0x7f184c14c470 2023-10-25T00:17:19.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f1853bae640 1 -- 172.21.15.107:0/4197609958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184c0746e0 con 0x7f184c14c470 2023-10-25T00:17:19.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.553+0000 7f18427fc640 1 -- 172.21.15.107:0/4197609958 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c007270 con 0x7f184c14c470 2023-10-25T00:17:19.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.557+0000 7f18427fc640 1 -- 172.21.15.107:0/4197609958 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f183c007490 con 0x7f184c14c470 2023-10-25T00:17:19.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.557+0000 7f18427fc640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1830065e60 0x7f1830068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:19.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.557+0000 7f185194a640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1830065e60 0x7f1830068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:19.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.557+0000 7f18427fc640 1 -- 172.21.15.107:0/4197609958 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7f183c084130 con 0x7f184c14c470 2023-10-25T00:17:19.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.557+0000 7f185194a640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1830065e60 0x7f1830068320 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f1848009980 tx=0x7f18480090e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:19.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.561+0000 7f18427fc640 1 -- 172.21.15.107:0/4197609958 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f183c050740 con 0x7f184c14c470 2023-10-25T00:17:19.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 -- 172.21.15.107:0/4197609958 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f184c14d470 con 0x7f184c14c470 2023-10-25T00:17:19.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f18427fc640 1 -- 172.21.15.107:0/4197609958 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5767 (secure 0 0 0) 0x7f183c053d60 con 0x7f184c14c470 2023-10-25T00:17:19.896 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:19.896 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:19.896 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 190 2023-10-25T00:17:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 -- 172.21.15.107:0/4197609958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1830065e60 msgr2=0x7f1830068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1830065e60 0x7f1830068320 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f1848009980 tx=0x7f18480090e0 comp rx=0 tx=0).stop 2023-10-25T00:17:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 -- 172.21.15.107:0/4197609958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f184c14c470 msgr2=0x7f184c14d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f184c14c470 0x7f184c14d820 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f183c00c500 tx=0x7f183c00c9c0 comp rx=0 tx=0).stop 2023-10-25T00:17:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 -- 172.21.15.107:0/4197609958 shutdown_connections 2023-10-25T00:17:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f184c152b30 0x7f184c0c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f184c14c470 0x7f184c14d820 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1830065e60 0x7f1830068320 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:19.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 --2- 172.21.15.107:0/4197609958 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f184c06f490 0x7f184c0aabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 -- 172.21.15.107:0/4197609958 >> 172.21.15.107:0/4197609958 conn(0x7f184c070f60 msgr2=0x7f184c0c3170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 -- 172.21.15.107:0/4197609958 shutdown_connections 2023-10-25T00:17:19.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:19.893+0000 7f1853bae640 1 -- 172.21.15.107:0/4197609958 wait complete. 2023-10-25T00:17:19.923 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:17:21.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:22.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.069+0000 7fe026199640 1 Processor -- start 2023-10-25T00:17:22.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.069+0000 7fe026199640 1 -- start start 2023-10-25T00:17:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.069+0000 7fe026199640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020069d40 0x7fe020158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe026199640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe020065bc0 0x7fe02006a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:22.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe026199640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0200a94b0 0x7fe0200aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:22.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe026199640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0200ac650 con 0x7fe0200a94b0 2023-10-25T00:17:22.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe026199640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0200a9e00 con 0x7fe020065bc0 2023-10-25T00:17:22.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe026199640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02006cea0 con 0x7fe020069d40 2023-10-25T00:17:22.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0200a94b0 0x7fe0200aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:22.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe0200a94b0 0x7fe0200aaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51090/0 (socket says 172.21.15.107:51090) 2023-10-25T00:17:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01ffff640 1 -- 172.21.15.107:0/2089684690 learned_addr learned my addr 172.21.15.107:0/2089684690 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01f7fe640 1 --2- 172.21.15.107:0/2089684690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020069d40 0x7fe020158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01effd640 1 --2- 172.21.15.107:0/2089684690 >> v2:172.21.15.110:3300/0 conn(0x7fe020065bc0 0x7fe02006a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01ffff640 1 -- 172.21.15.107:0/2089684690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020069d40 msgr2=0x7fe020158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:22.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01ffff640 1 --2- 172.21.15.107:0/2089684690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020069d40 0x7fe020158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01ffff640 1 -- 172.21.15.107:0/2089684690 >> v2:172.21.15.110:3300/0 conn(0x7fe020065bc0 msgr2=0x7fe02006a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:22.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01ffff640 1 --2- 172.21.15.107:0/2089684690 >> v2:172.21.15.110:3300/0 conn(0x7fe020065bc0 0x7fe02006a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01ffff640 1 -- 172.21.15.107:0/2089684690 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0201584e0 con 0x7fe0200a94b0 2023-10-25T00:17:22.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01effd640 1 --2- 172.21.15.107:0/2089684690 >> v2:172.21.15.110:3300/0 conn(0x7fe020065bc0 0x7fe02006a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:17:22.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01ffff640 1 --2- 172.21.15.107:0/2089684690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0200a94b0 0x7fe0200aaef0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fe02006e3c0 tx=0x7fe01801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6669d09f40595c41 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:22.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01e7fc640 1 -- 172.21.15.107:0/2089684690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0180a56b0 con 0x7fe0200a94b0 2023-10-25T00:17:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01e7fc640 1 -- 172.21.15.107:0/2089684690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0180a5810 con 0x7fe0200a94b0 2023-10-25T00:17:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe01e7fc640 1 -- 172.21.15.107:0/2089684690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0180a5ba0 con 0x7fe0200a94b0 2023-10-25T00:17:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe026199640 1 -- 172.21.15.107:0/2089684690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0200a94b0 msgr2=0x7fe0200aaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:22.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.073+0000 7fe026199640 1 --2- 172.21.15.107:0/2089684690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0200a94b0 0x7fe0200aaef0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fe02006e3c0 tx=0x7fe01801c940 comp rx=0 tx=0).stop 2023-10-25T00:17:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.077+0000 7fe026199640 1 -- 172.21.15.107:0/2089684690 shutdown_connections 2023-10-25T00:17:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.077+0000 7fe026199640 1 --2- 172.21.15.107:0/2089684690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0200a94b0 0x7fe0200aaef0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.077+0000 7fe026199640 1 --2- 172.21.15.107:0/2089684690 >> v2:172.21.15.110:3300/0 conn(0x7fe020065bc0 0x7fe02006a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.077+0000 7fe026199640 1 --2- 172.21.15.107:0/2089684690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020069d40 0x7fe020158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.077+0000 7fe026199640 1 -- 172.21.15.107:0/2089684690 >> 172.21.15.107:0/2089684690 conn(0x7fe0200c65c0 msgr2=0x7fe0200c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.077+0000 7fe026199640 1 -- 172.21.15.107:0/2089684690 shutdown_connections 2023-10-25T00:17:22.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.077+0000 7fe026199640 1 -- 172.21.15.107:0/2089684690 wait complete. 2023-10-25T00:17:22.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.077+0000 7fe026199640 1 Processor -- start 2023-10-25T00:17:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe026199640 1 -- start start 2023-10-25T00:17:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe026199640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020065bc0 0x7fe0201513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe026199640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe020069d40 0x7fe0201f4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe026199640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0200a94b0 0x7fe0200c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe026199640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200c10e0 con 0x7fe0200a94b0 2023-10-25T00:17:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe026199640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0200bc940 con 0x7fe020069d40 2023-10-25T00:17:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe026199640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0201f52d0 con 0x7fe020065bc0 2023-10-25T00:17:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe01f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020065bc0 0x7fe0201513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:22.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe01f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020065bc0 0x7fe0201513d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:59456/0 (socket says 172.21.15.107:59456) 2023-10-25T00:17:22.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe01f7fe640 1 -- 172.21.15.107:0/299355575 learned_addr learned my addr 172.21.15.107:0/299355575 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe01f7fe640 1 -- 172.21.15.107:0/299355575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe020069d40 msgr2=0x7fe0201f4820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:17:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe01f7fe640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe020069d40 0x7fe0201f4820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe01f7fe640 1 -- 172.21.15.107:0/299355575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0200a94b0 msgr2=0x7fe0200c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe01f7fe640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0200a94b0 0x7fe0200c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe01f7fe640 1 -- 172.21.15.107:0/299355575 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe018009b30 con 0x7fe020065bc0 2023-10-25T00:17:22.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe01f7fe640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020065bc0 0x7fe0201513d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe01000e930 tx=0x7fe01000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:22.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe007fff640 1 -- 172.21.15.107:0/299355575 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe010009690 con 0x7fe020065bc0 2023-10-25T00:17:22.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe007fff640 1 -- 172.21.15.107:0/299355575 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe01000ca00 con 0x7fe020065bc0 2023-10-25T00:17:22.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe007fff640 1 -- 172.21.15.107:0/299355575 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01000cd70 con 0x7fe020065bc0 2023-10-25T00:17:22.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe026199640 1 -- 172.21.15.107:0/299355575 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0201f5570 con 0x7fe020065bc0 2023-10-25T00:17:22.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.081+0000 7fe026199640 1 -- 172.21.15.107:0/299355575 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0200c1570 con 0x7fe020065bc0 2023-10-25T00:17:22.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.085+0000 7fe026199640 1 -- 172.21.15.107:0/299355575 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0200746e0 con 0x7fe020065bc0 2023-10-25T00:17:22.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.085+0000 7fe007fff640 1 -- 172.21.15.107:0/299355575 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe010013470 con 0x7fe020065bc0 2023-10-25T00:17:22.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.085+0000 7fe007fff640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe00c065e60 0x7fe00c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:22.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.085+0000 7fe007fff640 1 -- 172.21.15.107:0/299355575 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7fe010084980 con 0x7fe020065bc0 2023-10-25T00:17:22.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.085+0000 7fe01effd640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe00c065e60 0x7fe00c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:22.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.089+0000 7fe01effd640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe00c065e60 0x7fe00c068320 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe020151c70 tx=0x7fe014008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:22.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.093+0000 7fe007fff640 1 -- 172.21.15.107:0/299355575 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe010051130 con 0x7fe020065bc0 2023-10-25T00:17:22.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.361+0000 7fe026199640 1 -- 172.21.15.107:0/299355575 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe02006acc0 con 0x7fe020065bc0 2023-10-25T00:17:22.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.361+0000 7fe007fff640 1 -- 172.21.15.107:0/299355575 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5767 (secure 0 0 0) 0x7fe010021090 con 0x7fe020065bc0 2023-10-25T00:17:22.366 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:22.366 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 190 2023-10-25T00:17:22.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 -- 172.21.15.107:0/299355575 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe00c065e60 msgr2=0x7fe00c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:22.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe00c065e60 0x7fe00c068320 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fe020151c70 tx=0x7fe014008040 comp rx=0 tx=0).stop 2023-10-25T00:17:22.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 -- 172.21.15.107:0/299355575 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020065bc0 msgr2=0x7fe0201513d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:22.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020065bc0 0x7fe0201513d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe01000e930 tx=0x7fe01000edf0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 -- 172.21.15.107:0/299355575 shutdown_connections 2023-10-25T00:17:22.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0200a94b0 0x7fe0200c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe020069d40 0x7fe0201f4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe00c065e60 0x7fe00c068320 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 --2- 172.21.15.107:0/299355575 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe020065bc0 0x7fe0201513d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:22.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 -- 172.21.15.107:0/299355575 >> 172.21.15.107:0/299355575 conn(0x7fe0200c65c0 msgr2=0x7fe020156bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 -- 172.21.15.107:0/299355575 shutdown_connections 2023-10-25T00:17:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:22.365+0000 7fe026199640 1 -- 172.21.15.107:0/299355575 wait complete. 2023-10-25T00:17:22.391 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:17:22.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:23.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.001+0000 7f38f36fd640 1 Processor -- start 2023-10-25T00:17:23.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.001+0000 7f38f36fd640 1 -- start start 2023-10-25T00:17:23.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.001+0000 7f38f36fd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f38ec152040 0x7f38ec158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.001+0000 7f38f36fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38ec0adb80 0x7f38ec071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.001+0000 7f38f36fd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38ec152420 con 0x7f38ec071fc0 2023-10-25T00:17:23.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.001+0000 7f38f36fd640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38ec1556e0 con 0x7f38ec152040 2023-10-25T00:17:23.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.001+0000 7f38f36fd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec154d40 con 0x7f38ec0adb80 2023-10-25T00:17:23.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.001+0000 7f38f1499640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f38ec152040 0x7f38ec158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:23.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.005+0000 7f38f1499640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f38ec152040 0x7f38ec158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:41726/0 (socket says 172.21.15.107:41726) 2023-10-25T00:17:23.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.005+0000 7f38f1499640 1 -- 172.21.15.107:0/3945411225 learned_addr learned my addr 172.21.15.107:0/3945411225 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:23.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.005+0000 7f38f1499640 1 -- 172.21.15.107:0/3945411225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38ec0adb80 msgr2=0x7f38ec071a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:23.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.005+0000 7f38f1499640 1 --2- 172.21.15.107:0/3945411225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38ec0adb80 0x7f38ec071a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.005+0000 7f38f1499640 1 -- 172.21.15.107:0/3945411225 >> v1:172.21.15.110:6789/0 conn(0x7f38ec071fc0 legacy=0x7f38ec06e800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:23.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.005+0000 7f38e3fff640 1 -- 172.21.15.107:0/3945411225 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505534508 0 0) 0x7f38ec152420 con 0x7f38ec071fc0 2023-10-25T00:17:23.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.005+0000 7f38f1499640 1 -- 172.21.15.107:0/3945411225 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec155020 con 0x7f38ec152040 2023-10-25T00:17:23.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38f1499640 1 --2- 172.21.15.107:0/3945411225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38ec152040 0x7f38ec158940 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009870 tx=0x7f38e801c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=382b98de8bd4c8bb server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:23.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38e3fff640 1 -- 172.21.15.107:0/3945411225 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80ad680 con 0x7f38ec152040 2023-10-25T00:17:23.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38e3fff640 1 -- 172.21.15.107:0/3945411225 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e80ad7e0 con 0x7f38ec152040 2023-10-25T00:17:23.012 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38e3fff640 1 -- 172.21.15.107:0/3945411225 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80adb70 con 0x7f38ec152040 2023-10-25T00:17:23.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38f36fd640 1 -- 172.21.15.107:0/3945411225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38ec152040 msgr2=0x7f38ec158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:23.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38f36fd640 1 --2- 172.21.15.107:0/3945411225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38ec152040 0x7f38ec158940 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009870 tx=0x7f38e801c4b0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38f36fd640 1 -- 172.21.15.107:0/3945411225 shutdown_connections 2023-10-25T00:17:23.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38f36fd640 1 --2- 172.21.15.107:0/3945411225 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38ec0adb80 0x7f38ec071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38f36fd640 1 --2- 172.21.15.107:0/3945411225 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38ec152040 0x7f38ec158940 secure :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009870 tx=0x7f38e801c4b0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38f36fd640 1 -- 172.21.15.107:0/3945411225 >> 172.21.15.107:0/3945411225 conn(0x7f38ec070f60 msgr2=0x7f38ec071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38f36fd640 1 -- 172.21.15.107:0/3945411225 shutdown_connections 2023-10-25T00:17:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.009+0000 7f38f36fd640 1 -- 172.21.15.107:0/3945411225 wait complete. 2023-10-25T00:17:23.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f36fd640 1 Processor -- start 2023-10-25T00:17:23.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f36fd640 1 -- start start 2023-10-25T00:17:23.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f36fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38ec071fc0 0x7f38ec0c6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f36fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38ec0adb80 0x7f38ec14a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f36fd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38ec1500d0 0x7f38ec14ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f36fd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec0c7430 con 0x7f38ec071fc0 2023-10-25T00:17:23.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f36fd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec1edc50 con 0x7f38ec1500d0 2023-10-25T00:17:23.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f36fd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38ec1f0ab0 con 0x7f38ec0adb80 2023-10-25T00:17:23.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f1499640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38ec071fc0 0x7f38ec0c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:23.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f1499640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38ec071fc0 0x7f38ec0c6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39662/0 (socket says 172.21.15.107:39662) 2023-10-25T00:17:23.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f1499640 1 -- 172.21.15.107:0/3272648903 learned_addr learned my addr 172.21.15.107:0/3272648903 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:23.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f0c98640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38ec0adb80 0x7f38ec14a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:23.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f1499640 1 -- 172.21.15.107:0/3272648903 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38ec0adb80 msgr2=0x7f38ec14a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:23.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f1c9a640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38ec1500d0 0x7f38ec14ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:23.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f1499640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38ec0adb80 0x7f38ec14a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f1499640 1 -- 172.21.15.107:0/3272648903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38ec1500d0 msgr2=0x7f38ec14ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:23.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f1499640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38ec1500d0 0x7f38ec14ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f1499640 1 -- 172.21.15.107:0/3272648903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e80094d0 con 0x7f38ec071fc0 2023-10-25T00:17:23.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f1c9a640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38ec1500d0 0x7f38ec14ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:17:23.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f1499640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38ec071fc0 0x7f38ec0c6120 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f38e801a0e0 tx=0x7f38e801acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:23.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38e1ffb640 1 -- 172.21.15.107:0/3272648903 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80ade60 con 0x7f38ec071fc0 2023-10-25T00:17:23.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38e1ffb640 1 -- 172.21.15.107:0/3272648903 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e80b8420 con 0x7f38ec071fc0 2023-10-25T00:17:23.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f36fd640 1 -- 172.21.15.107:0/3272648903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38ec1ede60 con 0x7f38ec071fc0 2023-10-25T00:17:23.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38e1ffb640 1 -- 172.21.15.107:0/3272648903 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e80b87f0 con 0x7f38ec071fc0 2023-10-25T00:17:23.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.013+0000 7f38f36fd640 1 -- 172.21.15.107:0/3272648903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38ec1ee420 con 0x7f38ec071fc0 2023-10-25T00:17:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.017+0000 7f38f36fd640 1 -- 172.21.15.107:0/3272648903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38ec0746e0 con 0x7f38ec071fc0 2023-10-25T00:17:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.017+0000 7f38e1ffb640 1 -- 172.21.15.107:0/3272648903 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f38e80b4040 con 0x7f38ec071fc0 2023-10-25T00:17:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.017+0000 7f38e1ffb640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38d0065e60 0x7f38d0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.017+0000 7f38e1ffb640 1 -- 172.21.15.107:0/3272648903 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7f38e811f0f0 con 0x7f38ec071fc0 2023-10-25T00:17:23.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.017+0000 7f38f0c98640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38d0065e60 0x7f38d0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:23.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.017+0000 7f38f0c98640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38d0065e60 0x7f38d0068320 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f38ec15e490 tx=0x7f38dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:23.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.025+0000 7f38e1ffb640 1 -- 172.21.15.107:0/3272648903 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f38e80eb920 con 0x7f38ec071fc0 2023-10-25T00:17:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.297+0000 7f38f36fd640 1 -- 172.21.15.107:0/3272648903 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38ec150d00 con 0x7f38ec071fc0 2023-10-25T00:17:23.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38e1ffb640 1 -- 172.21.15.107:0/3272648903 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5767 (secure 0 0 0) 0x7f38e80eef40 con 0x7f38ec071fc0 2023-10-25T00:17:23.303 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:23.303 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:23.303 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 190 2023-10-25T00:17:23.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 -- 172.21.15.107:0/3272648903 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38d0065e60 msgr2=0x7f38d0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:23.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38d0065e60 0x7f38d0068320 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f38ec15e490 tx=0x7f38dc008040 comp rx=0 tx=0).stop 2023-10-25T00:17:23.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 -- 172.21.15.107:0/3272648903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38ec071fc0 msgr2=0x7f38ec0c6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:23.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38ec071fc0 0x7f38ec0c6120 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f38e801a0e0 tx=0x7f38e801acf0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 -- 172.21.15.107:0/3272648903 shutdown_connections 2023-10-25T00:17:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38ec1500d0 0x7f38ec14ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38d0065e60 0x7f38d0068320 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38ec0adb80 0x7f38ec14a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 --2- 172.21.15.107:0/3272648903 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38ec071fc0 0x7f38ec0c6120 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 -- 172.21.15.107:0/3272648903 >> 172.21.15.107:0/3272648903 conn(0x7f38ec070f60 msgr2=0x7f38ec071340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 -- 172.21.15.107:0/3272648903 shutdown_connections 2023-10-25T00:17:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.301+0000 7f38f36fd640 1 -- 172.21.15.107:0/3272648903 wait complete. 2023-10-25T00:17:23.328 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2023-10-25T00:17:23.328 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-25T00:17:23.328 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:17:23.342 INFO:tasks.ceph.mds.b:Stopped 2023-10-25T00:17:23.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0ceee2d640 1 Processor -- start 2023-10-25T00:17:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0ceee2d640 1 -- start start 2023-10-25T00:17:23.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0ceee2d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0ce8152040 0x7f0ce8158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0ceee2d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0ce80adb80 0x7f0ce8071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0ceee2d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce8152420 con 0x7f0ce8071fc0 2023-10-25T00:17:23.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0ceee2d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce81556e0 con 0x7f0ce8152040 2023-10-25T00:17:23.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0ceee2d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce8154d40 con 0x7f0ce80adb80 2023-10-25T00:17:23.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0cecbc9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0ce8152040 0x7f0ce8158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:23.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0cecbc9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0ce8152040 0x7f0ce8158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39678/0 (socket says 172.21.15.107:39678) 2023-10-25T00:17:23.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0cecbc9640 1 -- 172.21.15.107:0/3528026657 learned_addr learned my addr 172.21.15.107:0/3528026657 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:23.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0ced3ca640 1 --1- 172.21.15.107:0/3528026657 >> v1:172.21.15.110:6789/0 conn(0x7f0ce8071fc0 0x7f0ce806e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:59844/0 (socket says 172.21.15.107:59844) 2023-10-25T00:17:23.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0ce7fff640 1 --2- 172.21.15.107:0/3528026657 >> v2:172.21.15.110:3300/0 conn(0x7f0ce80adb80 0x7f0ce8071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:23.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0cecbc9640 1 -- 172.21.15.107:0/3528026657 >> v2:172.21.15.110:3300/0 conn(0x7f0ce80adb80 msgr2=0x7f0ce8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:23.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0cecbc9640 1 --2- 172.21.15.107:0/3528026657 >> v2:172.21.15.110:3300/0 conn(0x7f0ce80adb80 0x7f0ce8071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0cecbc9640 1 -- 172.21.15.107:0/3528026657 >> v1:172.21.15.110:6789/0 conn(0x7f0ce8071fc0 legacy=0x7f0ce806e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:23.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.485+0000 7f0cecbc9640 1 -- 172.21.15.107:0/3528026657 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce8155020 con 0x7f0ce8152040 2023-10-25T00:17:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0cecbc9640 1 --2- 172.21.15.107:0/3528026657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ce8152040 0x7f0ce8158940 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8070910 tx=0x7f0cdc01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f503a1fb2ef452a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0ce77fe640 1 -- 172.21.15.107:0/3528026657 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0ab600 con 0x7f0ce8152040 2023-10-25T00:17:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0ce77fe640 1 -- 172.21.15.107:0/3528026657 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cdc0a7050 con 0x7f0ce8152040 2023-10-25T00:17:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0ce77fe640 1 -- 172.21.15.107:0/3528026657 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0ab980 con 0x7f0ce8152040 2023-10-25T00:17:23.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0ceee2d640 1 -- 172.21.15.107:0/3528026657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ce8152040 msgr2=0x7f0ce8158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:23.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0ceee2d640 1 --2- 172.21.15.107:0/3528026657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ce8152040 0x7f0ce8158940 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8070910 tx=0x7f0cdc01c450 comp rx=0 tx=0).stop 2023-10-25T00:17:23.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0ceee2d640 1 -- 172.21.15.107:0/3528026657 shutdown_connections 2023-10-25T00:17:23.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0ceee2d640 1 --2- 172.21.15.107:0/3528026657 >> v2:172.21.15.110:3300/0 conn(0x7f0ce80adb80 0x7f0ce8071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0ceee2d640 1 --2- 172.21.15.107:0/3528026657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ce8152040 0x7f0ce8158940 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0ceee2d640 1 -- 172.21.15.107:0/3528026657 >> 172.21.15.107:0/3528026657 conn(0x7f0ce8070f60 msgr2=0x7f0ce8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:23.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0ceee2d640 1 -- 172.21.15.107:0/3528026657 shutdown_connections 2023-10-25T00:17:23.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.489+0000 7f0ceee2d640 1 -- 172.21.15.107:0/3528026657 wait complete. 2023-10-25T00:17:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ceee2d640 1 Processor -- start 2023-10-25T00:17:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ceee2d640 1 -- start start 2023-10-25T00:17:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ceee2d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ce8071fc0 0x7f0ce8148430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ceee2d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ce80adb80 0x7f0ce8148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ceee2d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ce8152040 0x7f0ce81efcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ceee2d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce80c45e0 con 0x7f0ce8152040 2023-10-25T00:17:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ceee2d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce8154190 con 0x7f0ce80adb80 2023-10-25T00:17:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ceee2d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce8154730 con 0x7f0ce8071fc0 2023-10-25T00:17:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ced3ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ce8152040 0x7f0ce81efcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ced3ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ce8152040 0x7f0ce81efcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39694/0 (socket says 172.21.15.107:39694) 2023-10-25T00:17:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ced3ca640 1 -- 172.21.15.107:0/1783073727 learned_addr learned my addr 172.21.15.107:0/1783073727 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ce7fff640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ce80adb80 0x7f0ce8148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ced3ca640 1 -- 172.21.15.107:0/1783073727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ce8071fc0 msgr2=0x7f0ce8148430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ced3ca640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ce8071fc0 0x7f0ce8148430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ced3ca640 1 -- 172.21.15.107:0/1783073727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ce80adb80 msgr2=0x7f0ce8148970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ced3ca640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ce80adb80 0x7f0ce8148970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ced3ca640 1 -- 172.21.15.107:0/1783073727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc0094d0 con 0x7f0ce8152040 2023-10-25T00:17:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ced3ca640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ce8152040 0x7f0ce81efcf0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000c700 tx=0x7f0ce000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:23.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ce57fa640 1 -- 172.21.15.107:0/1783073727 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce000ce10 con 0x7f0ce8152040 2023-10-25T00:17:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ce57fa640 1 -- 172.21.15.107:0/1783073727 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce000a640 con 0x7f0ce8152040 2023-10-25T00:17:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ceee2d640 1 -- 172.21.15.107:0/1783073727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce81f0880 con 0x7f0ce8152040 2023-10-25T00:17:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ce57fa640 1 -- 172.21.15.107:0/1783073727 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce000a9b0 con 0x7f0ce8152040 2023-10-25T00:17:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.493+0000 7f0ceee2d640 1 -- 172.21.15.107:0/1783073727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce814a280 con 0x7f0ce8152040 2023-10-25T00:17:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.497+0000 7f0ceee2d640 1 -- 172.21.15.107:0/1783073727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce814a470 con 0x7f0ce8152040 2023-10-25T00:17:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.497+0000 7f0ce57fa640 1 -- 172.21.15.107:0/1783073727 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0ce000ab10 con 0x7f0ce8152040 2023-10-25T00:17:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.497+0000 7f0ce57fa640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0ccc065e60 0x7f0ccc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.497+0000 7f0ce57fa640 1 -- 172.21.15.107:0/1783073727 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 10057+0+0 (secure 0 0 0) 0x7f0ce0083f10 con 0x7f0ce8152040 2023-10-25T00:17:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.497+0000 7f0cecbc9640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0ccc065e60 0x7f0ccc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:23.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.497+0000 7f0cecbc9640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0ccc065e60 0x7f0ccc068320 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0ad000 tx=0x7f0cdc004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:23.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.505+0000 7f0ce57fa640 1 -- 172.21.15.107:0/1783073727 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0ce0050630 con 0x7f0ce8152040 2023-10-25T00:17:23.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.773+0000 7f0ceee2d640 1 -- 172.21.15.107:0/1783073727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ce80746e0 con 0x7f0ce8152040 2023-10-25T00:17:23.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.777+0000 7f0ce57fa640 1 -- 172.21.15.107:0/1783073727 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5767 (secure 0 0 0) 0x7f0ce0053c50 con 0x7f0ce8152040 2023-10-25T00:17:23.779 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:23.779 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8418,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:15:22.657484+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7030":{"gid":7030,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":24,"addr":"172.21.15.107:6835/208713198","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":208713198},{"type":"v1","addr":"172.21.15.107:6835","nonce":208713198}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:23.780 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 190 2023-10-25T00:17:23.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 -- 172.21.15.107:0/1783073727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0ccc065e60 msgr2=0x7f0ccc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:23.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0ccc065e60 0x7f0ccc068320 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0ad000 tx=0x7f0cdc004740 comp rx=0 tx=0).stop 2023-10-25T00:17:23.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 -- 172.21.15.107:0/1783073727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ce8152040 msgr2=0x7f0ce81efcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:23.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ce8152040 0x7f0ce81efcf0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f0ce000c700 tx=0x7f0ce000cbc0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 -- 172.21.15.107:0/1783073727 shutdown_connections 2023-10-25T00:17:23.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ce8152040 0x7f0ce81efcf0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ce80adb80 0x7f0ce8148970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0ccc065e60 0x7f0ccc068320 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 --2- 172.21.15.107:0/1783073727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ce8071fc0 0x7f0ce8148430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:23.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 -- 172.21.15.107:0/1783073727 >> 172.21.15.107:0/1783073727 conn(0x7f0ce8070f60 msgr2=0x7f0ce8158770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:23.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 -- 172.21.15.107:0/1783073727 shutdown_connections 2023-10-25T00:17:23.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:23.781+0000 7f0ceee2d640 1 -- 172.21.15.107:0/1783073727 wait complete. 2023-10-25T00:17:23.808 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:17:24.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8cbea640 1 Processor -- start 2023-10-25T00:17:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8cbea640 1 -- start start 2023-10-25T00:17:24.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8cbea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e880c1570 0x7f6e880a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:24.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8cbea640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e880a9b90 con 0x7f6e88150930 2023-10-25T00:17:24.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8cbea640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e880c1390 con 0x7f6e881510c0 2023-10-25T00:17:24.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8cbea640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e88154dd0 con 0x7f6e880c1570 2023-10-25T00:17:24.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6e88150930 0x7f6e88069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37018/0 (socket says 172.21.15.107:37018) 2023-10-25T00:17:24.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8659c640 1 -- 172.21.15.107:0/3326529552 learned_addr learned my addr 172.21.15.107:0/3326529552 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:17:24.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e85d9b640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e880c1570 0x7f6e880a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:24.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372732666 0 0) 0x7f6e880a9b90 con 0x7f6e88150930 2023-10-25T00:17:24.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e70003530 con 0x7f6e88150930 2023-10-25T00:17:24.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038876081 0 0) 0x7f6e880c1390 con 0x7f6e881510c0 2023-10-25T00:17:24.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e880a9b90 con 0x7f6e881510c0 2023-10-25T00:17:24.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 221921149 0 0) 0x7f6e70003530 con 0x7f6e88150930 2023-10-25T00:17:24.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e880c1570 msgr2=0x7f6e880a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:24.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e880c1570 0x7f6e880a94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:24.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 >> v1:172.21.15.110:6789/0 conn(0x7f6e881510c0 legacy=0x7f6e88151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:24.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e881550b0 con 0x7f6e88150930 2023-10-25T00:17:24.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6e78001680 con 0x7f6e88150930 2023-10-25T00:17:24.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 >> v1:172.21.15.107:6789/0 conn(0x7f6e88150930 legacy=0x7f6e88069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:24.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e70003b40 0x7f6e70003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e700047d0 0x7f6e70004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e70005190 0x7f6e7000ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e88154dd0 con 0x7f6e700047d0 2023-10-25T00:17:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e700104e0 con 0x7f6e70003b40 2023-10-25T00:17:24.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e70010620 con 0x7f6e70005190 2023-10-25T00:17:24.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e86d9d640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e70005190 0x7f6e7000ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:24.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e8659c640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e70003b40 0x7f6e70003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:24.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.561+0000 7f6e85d9b640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e700047d0 0x7f6e70004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:24.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.565+0000 7f6e8659c640 1 -- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e70005190 msgr2=0x7f6e7000ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:24.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.565+0000 7f6e8659c640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e70005190 0x7f6e7000ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:24.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.565+0000 7f6e8659c640 1 -- 172.21.15.107:0/3326529552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e700047d0 msgr2=0x7f6e70004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:24.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.565+0000 7f6e8659c640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e700047d0 0x7f6e70004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:24.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.565+0000 7f6e8659c640 1 -- 172.21.15.107:0/3326529552 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e881550b0 con 0x7f6e70003b40 2023-10-25T00:17:24.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.565+0000 7f6e86d9d640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e70005190 0x7f6e7000ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:17:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.565+0000 7f6e8659c640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e70003b40 0x7f6e70003f40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800c290 tx=0x7f6e7801cc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec08f04ccd9373b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.565+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78005ae0 con 0x7f6e70003b40 2023-10-25T00:17:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.565+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e78005c40 con 0x7f6e70003b40 2023-10-25T00:17:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.565+0000 7f6e8559a640 1 -- 172.21.15.107:0/3326529552 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78007070 con 0x7f6e70003b40 2023-10-25T00:17:24.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 -- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e70003b40 msgr2=0x7f6e70003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 --2- 172.21.15.107:0/3326529552 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e70003b40 0x7f6e70003f40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800c290 tx=0x7f6e7801cc00 comp rx=0 tx=0).stop 2023-10-25T00:17:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e86d9d640 1 -- 172.21.15.107:0/3326529552 reap_dead start 2023-10-25T00:17:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 -- 172.21.15.107:0/3326529552 shutdown_connections 2023-10-25T00:17:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 -- 172.21.15.107:0/3326529552 >> 172.21.15.107:0/3326529552 conn(0x7f6e880c65c0 msgr2=0x7f6e880c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:24.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 -- 172.21.15.107:0/3326529552 shutdown_connections 2023-10-25T00:17:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 -- 172.21.15.107:0/3326529552 wait complete. 2023-10-25T00:17:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 Processor -- start 2023-10-25T00:17:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 -- start start 2023-10-25T00:17:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e70003b40 0x7f6e88152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:24.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e88153f80 0x7f6e881ec020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e88071dc0 0x7f6e88149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e881ec900 con 0x7f6e88153f80 2023-10-25T00:17:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8806f510 con 0x7f6e70003b40 2023-10-25T00:17:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.569+0000 7f6e8cbea640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8806f7f0 con 0x7f6e88071dc0 2023-10-25T00:17:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e8659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e70003b40 0x7f6e88152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e86d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e88071dc0 0x7f6e88149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e86d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e88071dc0 0x7f6e88149a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:58900/0 (socket says 172.21.15.107:58900) 2023-10-25T00:17:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e86d9d640 1 -- 172.21.15.107:0/1280286182 learned_addr learned my addr 172.21.15.107:0/1280286182 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e85d9b640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e88153f80 0x7f6e881ec020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e86d9d640 1 -- 172.21.15.107:0/1280286182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e70003b40 msgr2=0x7f6e88152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e86d9d640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e70003b40 0x7f6e88152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e86d9d640 1 -- 172.21.15.107:0/1280286182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e88153f80 msgr2=0x7f6e881ec020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e86d9d640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e88153f80 0x7f6e881ec020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e86d9d640 1 -- 172.21.15.107:0/1280286182 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e881ecbe0 con 0x7f6e88071dc0 2023-10-25T00:17:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e86d9d640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e88071dc0 0x7f6e88149a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6e880a8180 tx=0x7f6e8000ff40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e76ffd640 1 -- 172.21.15.107:0/1280286182 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80003b90 con 0x7f6e88071dc0 2023-10-25T00:17:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e76ffd640 1 -- 172.21.15.107:0/1280286182 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e80003cf0 con 0x7f6e88071dc0 2023-10-25T00:17:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e8cbea640 1 -- 172.21.15.107:0/1280286182 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e881ec560 con 0x7f6e88071dc0 2023-10-25T00:17:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e76ffd640 1 -- 172.21.15.107:0/1280286182 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8000d9c0 con 0x7f6e88071dc0 2023-10-25T00:17:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e8cbea640 1 -- 172.21.15.107:0/1280286182 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e88072250 con 0x7f6e88071dc0 2023-10-25T00:17:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.573+0000 7f6e8cbea640 1 -- 172.21.15.107:0/1280286182 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e8806f9b0 con 0x7f6e88071dc0 2023-10-25T00:17:24.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.577+0000 7f6e76ffd640 1 -- 172.21.15.107:0/1280286182 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6e8000db20 con 0x7f6e88071dc0 2023-10-25T00:17:24.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.577+0000 7f6e76ffd640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e68065e60 0x7f6e68068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:24.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.577+0000 7f6e8659c640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e68065e60 0x7f6e68068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:24.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.577+0000 7f6e8659c640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e68065e60 0x7f6e68068320 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f6e780ab000 tx=0x7f6e780a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:24.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.581+0000 7f6e76ffd640 1 -- 172.21.15.107:0/1280286182 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 10143+0+0 (secure 0 0 0) 0x7f6e80055ce0 con 0x7f6e88071dc0 2023-10-25T00:17:24.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.593+0000 7f6e76ffd640 1 -- 172.21.15.107:0/1280286182 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6e80051920 con 0x7f6e88071dc0 2023-10-25T00:17:24.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 -- 172.21.15.107:0/1280286182 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e880746e0 con 0x7f6e88071dc0 2023-10-25T00:17:24.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e76ffd640 1 -- 172.21.15.107:0/1280286182 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5008 (secure 0 0 0) 0x7f6e80003e60 con 0x7f6e88071dc0 2023-10-25T00:17:24.868 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:24.869 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:24.033550+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:24.870 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 191 2023-10-25T00:17:24.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 -- 172.21.15.107:0/1280286182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e68065e60 msgr2=0x7f6e68068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:24.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e68065e60 0x7f6e68068320 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f6e780ab000 tx=0x7f6e780a7070 comp rx=0 tx=0).stop 2023-10-25T00:17:24.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 -- 172.21.15.107:0/1280286182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e88071dc0 msgr2=0x7f6e88149a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:24.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e88071dc0 0x7f6e88149a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6e880a8180 tx=0x7f6e8000ff40 comp rx=0 tx=0).stop 2023-10-25T00:17:24.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 -- 172.21.15.107:0/1280286182 shutdown_connections 2023-10-25T00:17:24.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6e68065e60 0x7f6e68068320 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:24.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e88071dc0 0x7f6e88149a00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:24.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e88153f80 0x7f6e881ec020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:24.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 --2- 172.21.15.107:0/1280286182 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e70003b40 0x7f6e88152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:24.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 -- 172.21.15.107:0/1280286182 >> 172.21.15.107:0/1280286182 conn(0x7f6e880c65c0 msgr2=0x7f6e881de3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:24.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 -- 172.21.15.107:0/1280286182 shutdown_connections 2023-10-25T00:17:24.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:24.865+0000 7f6e8cbea640 1 -- 172.21.15.107:0/1280286182 wait complete. 2023-10-25T00:17:24.891 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2023-10-25T00:17:24.892 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:17:24.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:25.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213fff8640 1 Processor -- start 2023-10-25T00:17:25.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213fff8640 1 -- start start 2023-10-25T00:17:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213fff8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f213814fc30 0x7f2138153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213fff8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2138072330 con 0x7f21380c5c20 2023-10-25T00:17:25.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213fff8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2138155ca0 con 0x7f21380c18a0 2023-10-25T00:17:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213fff8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2138155f80 con 0x7f213814fc30 2023-10-25T00:17:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213d593640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f21380c5c20 0x7f213806df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37026/0 (socket says 172.21.15.107:37026) 2023-10-25T00:17:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213d593640 1 -- 172.21.15.107:0/686447931 learned_addr learned my addr 172.21.15.107:0/686447931 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:17:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213dd94640 1 --2- 172.21.15.107:0/686447931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f213814fc30 0x7f2138153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:25.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213cd92640 1 -- 172.21.15.107:0/686447931 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830938201 0 0) 0x7f2138072330 con 0x7f21380c5c20 2023-10-25T00:17:25.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213cd92640 1 -- 172.21.15.107:0/686447931 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2124003530 con 0x7f21380c5c20 2023-10-25T00:17:25.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213dd94640 1 -- 172.21.15.107:0/686447931 >> v1:172.21.15.110:6789/0 conn(0x7f21380c18a0 legacy=0x7f21380c1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:25.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213dd94640 1 -- 172.21.15.107:0/686447931 >> v1:172.21.15.107:6789/0 conn(0x7f21380c5c20 legacy=0x7f213806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:25.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213dd94640 1 -- 172.21.15.107:0/686447931 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2138156260 con 0x7f213814fc30 2023-10-25T00:17:25.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.029+0000 7f213dd94640 1 --2- 172.21.15.107:0/686447931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f213814fc30 0x7f2138153190 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f21340097f0 tx=0x7f213401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62e85c25fe53c2af server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:25.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213cd92640 1 -- 172.21.15.107:0/686447931 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21340ab600 con 0x7f213814fc30 2023-10-25T00:17:25.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213cd92640 1 -- 172.21.15.107:0/686447931 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21340ab760 con 0x7f213814fc30 2023-10-25T00:17:25.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213cd92640 1 -- 172.21.15.107:0/686447931 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21340abad0 con 0x7f213814fc30 2023-10-25T00:17:25.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 -- 172.21.15.107:0/686447931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f213814fc30 msgr2=0x7f2138153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:25.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 --2- 172.21.15.107:0/686447931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f213814fc30 0x7f2138153190 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f21340097f0 tx=0x7f213401c400 comp rx=0 tx=0).stop 2023-10-25T00:17:25.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 -- 172.21.15.107:0/686447931 shutdown_connections 2023-10-25T00:17:25.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 --2- 172.21.15.107:0/686447931 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f213814fc30 0x7f2138153190 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 -- 172.21.15.107:0/686447931 >> 172.21.15.107:0/686447931 conn(0x7f21380c39a0 msgr2=0x7f21380c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:25.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 -- 172.21.15.107:0/686447931 shutdown_connections 2023-10-25T00:17:25.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 -- 172.21.15.107:0/686447931 wait complete. 2023-10-25T00:17:25.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 Processor -- start 2023-10-25T00:17:25.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 -- start start 2023-10-25T00:17:25.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f21380c18a0 0x7f21381ed880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21380c5c20 0x7f21381304c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f213814fc30 0x7f2138133290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2138132e00 con 0x7f213814fc30 2023-10-25T00:17:25.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2138131080 con 0x7f21380c5c20 2023-10-25T00:17:25.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213fff8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21381ee560 con 0x7f21380c18a0 2023-10-25T00:17:25.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213e595640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f213814fc30 0x7f2138133290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:25.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213dd94640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f21380c18a0 0x7f21381ed880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:25.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213e595640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f213814fc30 0x7f2138133290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39722/0 (socket says 172.21.15.107:39722) 2023-10-25T00:17:25.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213e595640 1 -- 172.21.15.107:0/2136052360 learned_addr learned my addr 172.21.15.107:0/2136052360 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:25.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213d593640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21380c5c20 0x7f21381304c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:25.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213e595640 1 -- 172.21.15.107:0/2136052360 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f21380c18a0 msgr2=0x7f21381ed880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:25.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213e595640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f21380c18a0 0x7f21381ed880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213e595640 1 -- 172.21.15.107:0/2136052360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21380c5c20 msgr2=0x7f21381304c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:25.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213e595640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21380c5c20 0x7f21381304c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213e595640 1 -- 172.21.15.107:0/2136052360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2134009420 con 0x7f213814fc30 2023-10-25T00:17:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f213e595640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f213814fc30 0x7f2138133290 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f213000c770 tx=0x7f213000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f212e7fc640 1 -- 172.21.15.107:0/2136052360 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2130003b80 con 0x7f213814fc30 2023-10-25T00:17:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f212e7fc640 1 -- 172.21.15.107:0/2136052360 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2130003ce0 con 0x7f213814fc30 2023-10-25T00:17:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.033+0000 7f212e7fc640 1 -- 172.21.15.107:0/2136052360 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213000aa90 con 0x7f213814fc30 2023-10-25T00:17:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.037+0000 7f213fff8640 1 -- 172.21.15.107:0/2136052360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21381ee280 con 0x7f213814fc30 2023-10-25T00:17:25.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.037+0000 7f213fff8640 1 -- 172.21.15.107:0/2136052360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21381556e0 con 0x7f213814fc30 2023-10-25T00:17:25.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.037+0000 7f213fff8640 1 -- 172.21.15.107:0/2136052360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2138132f70 con 0x7f213814fc30 2023-10-25T00:17:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.037+0000 7f212e7fc640 1 -- 172.21.15.107:0/2136052360 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f213000abf0 con 0x7f213814fc30 2023-10-25T00:17:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.037+0000 7f212e7fc640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f211c065e60 0x7f211c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.037+0000 7f212e7fc640 1 -- 172.21.15.107:0/2136052360 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 10143+0+0 (secure 0 0 0) 0x7f2130084ff0 con 0x7f213814fc30 2023-10-25T00:17:25.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.037+0000 7f213dd94640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f211c065e60 0x7f211c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.037+0000 7f213dd94640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f211c065e60 0x7f211c068320 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f21340af000 tx=0x7f21340b6040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:25.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.041+0000 7f212e7fc640 1 -- 172.21.15.107:0/2136052360 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2130051740 con 0x7f213814fc30 2023-10-25T00:17:25.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.297+0000 7f213fff8640 1 -- 172.21.15.107:0/2136052360 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21380746e0 con 0x7f213814fc30 2023-10-25T00:17:25.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.297+0000 7f212e7fc640 1 -- 172.21.15.107:0/2136052360 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+5012 (secure 0 0 0) 0x7f2130054d60 con 0x7f213814fc30 2023-10-25T00:17:25.301 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:25.301 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:25.048123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:25.302 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 192 2023-10-25T00:17:25.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 -- 172.21.15.107:0/2136052360 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f211c065e60 msgr2=0x7f211c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:25.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f211c065e60 0x7f211c068320 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f21340af000 tx=0x7f21340b6040 comp rx=0 tx=0).stop 2023-10-25T00:17:25.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 -- 172.21.15.107:0/2136052360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f213814fc30 msgr2=0x7f2138133290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:25.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f213814fc30 0x7f2138133290 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f213000c770 tx=0x7f213000cc30 comp rx=0 tx=0).stop 2023-10-25T00:17:25.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 -- 172.21.15.107:0/2136052360 shutdown_connections 2023-10-25T00:17:25.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f213814fc30 0x7f2138133290 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f21380c5c20 0x7f21381304c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f211c065e60 0x7f211c068320 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 --2- 172.21.15.107:0/2136052360 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f21380c18a0 0x7f21381ed880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 -- 172.21.15.107:0/2136052360 >> 172.21.15.107:0/2136052360 conn(0x7f21380c39a0 msgr2=0x7f21380c5090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:25.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 -- 172.21.15.107:0/2136052360 shutdown_connections 2023-10-25T00:17:25.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.301+0000 7f213fff8640 1 -- 172.21.15.107:0/2136052360 wait complete. 2023-10-25T00:17:25.323 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 192, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 192, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8418": { "addr": "172.21.15.107:6837/418589140", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 418589140, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 418589140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8418, "incarnation": 191, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 30 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:17:25.048123+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8418 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 188, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5699": { "addr": "172.21.15.110:6834/985135365", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 985135365, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 985135365, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5699, "incarnation": 185, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 144 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:15:25.707726+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5699 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2437148", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2437148, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2437148, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 190, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6443, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:17:25.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:25.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.937+0000 7f2fadb83640 1 Processor -- start 2023-10-25T00:17:25.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.937+0000 7f2fadb83640 1 -- start start 2023-10-25T00:17:25.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fadb83640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2fa814eb70 0x7f2fa80c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fadb83640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2fa8070b00 0x7f2fa81514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fadb83640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fa8151a20 0x7f2fa806df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fadb83640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa8071700 con 0x7f2fa8070b00 2023-10-25T00:17:25.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fadb83640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa80c22f0 con 0x7f2fa814eb70 2023-10-25T00:17:25.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fadb83640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa80c25d0 con 0x7f2fa8151a20 2023-10-25T00:17:25.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2fa8070b00 0x7f2fa81514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:25.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2fa8070b00 0x7f2fa81514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39730/0 (socket says 172.21.15.107:39730) 2023-10-25T00:17:25.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa6ffd640 1 -- 172.21.15.107:0/361766924 learned_addr learned my addr 172.21.15.107:0/361766924 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa7fff640 1 --2- 172.21.15.107:0/361766924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fa8151a20 0x7f2fa806df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa77fe640 1 --2- 172.21.15.107:0/361766924 >> v2:172.21.15.110:3300/0 conn(0x7f2fa814eb70 0x7f2fa80c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa6ffd640 1 -- 172.21.15.107:0/361766924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fa8151a20 msgr2=0x7f2fa806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa6ffd640 1 --2- 172.21.15.107:0/361766924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fa8151a20 0x7f2fa806df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa6ffd640 1 -- 172.21.15.107:0/361766924 >> v2:172.21.15.110:3300/0 conn(0x7f2fa814eb70 msgr2=0x7f2fa80c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:25.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa6ffd640 1 --2- 172.21.15.107:0/361766924 >> v2:172.21.15.110:3300/0 conn(0x7f2fa814eb70 0x7f2fa80c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa6ffd640 1 -- 172.21.15.107:0/361766924 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa80c2be0 con 0x7f2fa8070b00 2023-10-25T00:17:25.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa77fe640 1 --2- 172.21.15.107:0/361766924 >> v2:172.21.15.110:3300/0 conn(0x7f2fa814eb70 0x7f2fa80c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:17:25.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa6ffd640 1 --2- 172.21.15.107:0/361766924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fa8070b00 0x7f2fa81514e0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80c6730 tx=0x7f2f9001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4b2cee31b9757a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:25.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa67fc640 1 -- 172.21.15.107:0/361766924 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f900ab660 con 0x7f2fa8070b00 2023-10-25T00:17:25.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa67fc640 1 -- 172.21.15.107:0/361766924 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f900ab7c0 con 0x7f2fa8070b00 2023-10-25T00:17:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.941+0000 7f2fa67fc640 1 -- 172.21.15.107:0/361766924 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f900abb50 con 0x7f2fa8070b00 2023-10-25T00:17:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 -- 172.21.15.107:0/361766924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fa8070b00 msgr2=0x7f2fa81514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 --2- 172.21.15.107:0/361766924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fa8070b00 0x7f2fa81514e0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80c6730 tx=0x7f2f9001c500 comp rx=0 tx=0).stop 2023-10-25T00:17:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 -- 172.21.15.107:0/361766924 shutdown_connections 2023-10-25T00:17:25.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 --2- 172.21.15.107:0/361766924 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fa8151a20 0x7f2fa806df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 --2- 172.21.15.107:0/361766924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fa8070b00 0x7f2fa81514e0 secure :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80c6730 tx=0x7f2f9001c500 comp rx=0 tx=0).stop 2023-10-25T00:17:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 --2- 172.21.15.107:0/361766924 >> v2:172.21.15.110:3300/0 conn(0x7f2fa814eb70 0x7f2fa80c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 -- 172.21.15.107:0/361766924 >> 172.21.15.107:0/361766924 conn(0x7f2fa80c39a0 msgr2=0x7f2fa80c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 -- 172.21.15.107:0/361766924 shutdown_connections 2023-10-25T00:17:25.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 -- 172.21.15.107:0/361766924 wait complete. 2023-10-25T00:17:25.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 Processor -- start 2023-10-25T00:17:25.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 -- start start 2023-10-25T00:17:25.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fa814eb70 0x7f2fa814fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fa8151a20 0x7f2fa806c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fa8140300 0x7f2fa8140780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa814aed0 con 0x7f2fa8140300 2023-10-25T00:17:25.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa80aa5c0 con 0x7f2fa8151a20 2023-10-25T00:17:25.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fadb83640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa80aa000 con 0x7f2fa814eb70 2023-10-25T00:17:25.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fa7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fa8140300 0x7f2fa8140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:25.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fa7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fa8140300 0x7f2fa8140780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39744/0 (socket says 172.21.15.107:39744) 2023-10-25T00:17:25.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.945+0000 7f2fa7fff640 1 -- 172.21.15.107:0/742292577 learned_addr learned my addr 172.21.15.107:0/742292577 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:25.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fa6ffd640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fa8151a20 0x7f2fa806c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:25.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fa7fff640 1 -- 172.21.15.107:0/742292577 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fa814eb70 msgr2=0x7f2fa814fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fa7fff640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fa814eb70 0x7f2fa814fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fa7fff640 1 -- 172.21.15.107:0/742292577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fa8151a20 msgr2=0x7f2fa806c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fa7fff640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fa8151a20 0x7f2fa806c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fa7fff640 1 -- 172.21.15.107:0/742292577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f900094d0 con 0x7f2fa8140300 2023-10-25T00:17:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fa6ffd640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fa8151a20 0x7f2fa806c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:17:25.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fa7fff640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fa8140300 0x7f2fa8140780 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800e7f0 tx=0x7f2f9800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2f9ffff640 1 -- 172.21.15.107:0/742292577 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f98009530 con 0x7f2fa8140300 2023-10-25T00:17:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2f9ffff640 1 -- 172.21.15.107:0/742292577 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9800eea0 con 0x7f2fa8140300 2023-10-25T00:17:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fadb83640 1 -- 172.21.15.107:0/742292577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa80aa2e0 con 0x7f2fa8140300 2023-10-25T00:17:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2f9ffff640 1 -- 172.21.15.107:0/742292577 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9800cb60 con 0x7f2fa8140300 2023-10-25T00:17:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fadb83640 1 -- 172.21.15.107:0/742292577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa80aab80 con 0x7f2fa8140300 2023-10-25T00:17:25.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fadb83640 1 -- 172.21.15.107:0/742292577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa80746e0 con 0x7f2fa8140300 2023-10-25T00:17:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2f9ffff640 1 -- 172.21.15.107:0/742292577 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2f9800ccc0 con 0x7f2fa8140300 2023-10-25T00:17:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2f9ffff640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f88065e90 0x7f2f88068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fa77fe640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f88065e90 0x7f2f88068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2f9ffff640 1 -- 172.21.15.107:0/742292577 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 10143+0+0 (secure 0 0 0) 0x7f2f980849d0 con 0x7f2fa8140300 2023-10-25T00:17:25.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.949+0000 7f2fa77fe640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f88065e90 0x7f2f88068350 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0007e70 tx=0x7f2fa00073a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:25.953+0000 7f2f9ffff640 1 -- 172.21.15.107:0/742292577 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2f98050f50 con 0x7f2fa8140300 2023-10-25T00:17:26.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.213+0000 7f2fadb83640 1 -- 172.21.15.107:0/742292577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fa80c1ed0 con 0x7f2fa8140300 2023-10-25T00:17:26.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.213+0000 7f2f9ffff640 1 -- 172.21.15.107:0/742292577 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+5012 (secure 0 0 0) 0x7f2f98021020 con 0x7f2fa8140300 2023-10-25T00:17:26.218 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:26.218 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:25.048123+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:26.219 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 192 2023-10-25T00:17:26.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 -- 172.21.15.107:0/742292577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f88065e90 msgr2=0x7f2f88068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:26.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f88065e90 0x7f2f88068350 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0007e70 tx=0x7f2fa00073a0 comp rx=0 tx=0).stop 2023-10-25T00:17:26.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 -- 172.21.15.107:0/742292577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fa8140300 msgr2=0x7f2fa8140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:26.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fa8140300 0x7f2fa8140780 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800e7f0 tx=0x7f2f9800ecb0 comp rx=0 tx=0).stop 2023-10-25T00:17:26.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 -- 172.21.15.107:0/742292577 shutdown_connections 2023-10-25T00:17:26.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fa8140300 0x7f2fa8140780 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:26.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fa8151a20 0x7f2fa806c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:26.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2f88065e90 0x7f2f88068350 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:26.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 --2- 172.21.15.107:0/742292577 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fa814eb70 0x7f2fa814fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:26.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 -- 172.21.15.107:0/742292577 >> 172.21.15.107:0/742292577 conn(0x7f2fa80c39a0 msgr2=0x7f2fa8150670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:26.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 -- 172.21.15.107:0/742292577 shutdown_connections 2023-10-25T00:17:26.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:26.217+0000 7f2fadb83640 1 -- 172.21.15.107:0/742292577 wait complete. 2023-10-25T00:17:26.239 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:17:27.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:27.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11bd656640 1 Processor -- start 2023-10-25T00:17:27.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11bd656640 1 -- start start 2023-10-25T00:17:27.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11bd656640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f11b806f490 0x7f11b814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:27.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11bd656640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b8155cd0 con 0x7f11b8152b30 2023-10-25T00:17:27.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11bd656640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b80c4380 con 0x7f11b814c470 2023-10-25T00:17:27.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11bd656640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b81526e0 con 0x7f11b806f490 2023-10-25T00:17:27.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11affff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f11b806f490 0x7f11b814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:27.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11b77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f11b8152b30 0x7f11b8152f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37040/0 (socket says 172.21.15.107:37040) 2023-10-25T00:17:27.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11b77fe640 1 -- 172.21.15.107:0/2067271147 learned_addr learned my addr 172.21.15.107:0/2067271147 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:17:27.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11affff640 1 -- 172.21.15.107:0/2067271147 >> v1:172.21.15.110:6789/0 conn(0x7f11b814c470 legacy=0x7f11b8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:27.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11affff640 1 -- 172.21.15.107:0/2067271147 >> v1:172.21.15.107:6789/0 conn(0x7f11b8152b30 legacy=0x7f11b8152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.461+0000 7f11affff640 1 -- 172.21.15.107:0/2067271147 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b80c48b0 con 0x7f11b806f490 2023-10-25T00:17:27.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.465+0000 7f11affff640 1 --2- 172.21.15.107:0/2067271147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11b806f490 0x7f11b814c850 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f11a00097c0 tx=0x7f11a001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9042ff67fc126b95 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.465+0000 7f11b67fc640 1 -- 172.21.15.107:0/2067271147 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a00ab600 con 0x7f11b806f490 2023-10-25T00:17:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.465+0000 7f11b67fc640 1 -- 172.21.15.107:0/2067271147 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a00a7050 con 0x7f11b806f490 2023-10-25T00:17:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.465+0000 7f11b67fc640 1 -- 172.21.15.107:0/2067271147 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a00ab980 con 0x7f11b806f490 2023-10-25T00:17:27.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.465+0000 7f11bd656640 1 -- 172.21.15.107:0/2067271147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11b806f490 msgr2=0x7f11b814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:27.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.465+0000 7f11bd656640 1 --2- 172.21.15.107:0/2067271147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11b806f490 0x7f11b814c850 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f11a00097c0 tx=0x7f11a001c320 comp rx=0 tx=0).stop 2023-10-25T00:17:27.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.465+0000 7f11bd656640 1 -- 172.21.15.107:0/2067271147 shutdown_connections 2023-10-25T00:17:27.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.465+0000 7f11bd656640 1 --2- 172.21.15.107:0/2067271147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11b806f490 0x7f11b814c850 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:27.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.465+0000 7f11bd656640 1 -- 172.21.15.107:0/2067271147 >> 172.21.15.107:0/2067271147 conn(0x7f11b8070f60 msgr2=0x7f11b8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:27.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.465+0000 7f11bd656640 1 -- 172.21.15.107:0/2067271147 shutdown_connections 2023-10-25T00:17:27.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.465+0000 7f11bd656640 1 -- 172.21.15.107:0/2067271147 wait complete. 2023-10-25T00:17:27.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11bd656640 1 Processor -- start 2023-10-25T00:17:27.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11bd656640 1 -- start start 2023-10-25T00:17:27.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11bd656640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11b806f490 0x7f11b813c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:27.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11bd656640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11b814c470 0x7f11b8140ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:27.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11bd656640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11b8152b30 0x7f11b81edd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:27.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11bd656640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b81558e0 con 0x7f11b8152b30 2023-10-25T00:17:27.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11bd656640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b80c6330 con 0x7f11b814c470 2023-10-25T00:17:27.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11bd656640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b80c6610 con 0x7f11b806f490 2023-10-25T00:17:27.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11b77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11b8152b30 0x7f11b81edd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:27.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11b6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11b806f490 0x7f11b813c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:27.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11b77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11b8152b30 0x7f11b81edd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39770/0 (socket says 172.21.15.107:39770) 2023-10-25T00:17:27.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11b6ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11b806f490 0x7f11b813c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:58944/0 (socket says 172.21.15.107:58944) 2023-10-25T00:17:27.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11b77fe640 1 -- 172.21.15.107:0/1775142056 learned_addr learned my addr 172.21.15.107:0/1775142056 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:27.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11affff640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11b814c470 0x7f11b8140ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:27.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11b77fe640 1 -- 172.21.15.107:0/1775142056 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11b806f490 msgr2=0x7f11b813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:27.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11b77fe640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11b806f490 0x7f11b813c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:27.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11b77fe640 1 -- 172.21.15.107:0/1775142056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11b814c470 msgr2=0x7f11b8140ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:27.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11b77fe640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11b814c470 0x7f11b8140ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:27.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11b77fe640 1 -- 172.21.15.107:0/1775142056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a0009420 con 0x7f11b8152b30 2023-10-25T00:17:27.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11b77fe640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11b8152b30 0x7f11b81edd60 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f11b000ca80 tx=0x7f11b000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:27.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11af7fe640 1 -- 172.21.15.107:0/1775142056 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b000ac30 con 0x7f11b8152b30 2023-10-25T00:17:27.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11af7fe640 1 -- 172.21.15.107:0/1775142056 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b000ad90 con 0x7f11b8152b30 2023-10-25T00:17:27.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11bd656640 1 -- 172.21.15.107:0/1775142056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b8155290 con 0x7f11b8152b30 2023-10-25T00:17:27.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11af7fe640 1 -- 172.21.15.107:0/1775142056 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0012600 con 0x7f11b8152b30 2023-10-25T00:17:27.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.469+0000 7f11bd656640 1 -- 172.21.15.107:0/1775142056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b813cd80 con 0x7f11b8152b30 2023-10-25T00:17:27.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.473+0000 7f11bd656640 1 -- 172.21.15.107:0/1775142056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b8155a50 con 0x7f11b8152b30 2023-10-25T00:17:27.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.473+0000 7f11af7fe640 1 -- 172.21.15.107:0/1775142056 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f11b0012760 con 0x7f11b8152b30 2023-10-25T00:17:27.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.473+0000 7f11af7fe640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1198065e90 0x7f1198068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.473+0000 7f11af7fe640 1 -- 172.21.15.107:0/1775142056 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 10143+0+0 (secure 0 0 0) 0x7f11b0084800 con 0x7f11b8152b30 2023-10-25T00:17:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.473+0000 7f11b6ffd640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1198065e90 0x7f1198068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:27.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.473+0000 7f11b6ffd640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1198065e90 0x7f1198068350 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f11a8007fd0 tx=0x7f11a8007e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:27.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.477+0000 7f11af7fe640 1 -- 172.21.15.107:0/1775142056 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f11b0050e60 con 0x7f11b8152b30 2023-10-25T00:17:27.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.761+0000 7f11bd656640 1 -- 172.21.15.107:0/1775142056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11b80746e0 con 0x7f11b8152b30 2023-10-25T00:17:27.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.761+0000 7f11af7fe640 1 -- 172.21.15.107:0/1775142056 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+5009 (secure 0 0 0) 0x7f11b0017090 con 0x7f11b8152b30 2023-10-25T00:17:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:27.764 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:27.060010+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:27.765 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 193 2023-10-25T00:17:27.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 -- 172.21.15.107:0/1775142056 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1198065e90 msgr2=0x7f1198068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:27.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1198065e90 0x7f1198068350 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f11a8007fd0 tx=0x7f11a8007e40 comp rx=0 tx=0).stop 2023-10-25T00:17:27.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 -- 172.21.15.107:0/1775142056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11b8152b30 msgr2=0x7f11b81edd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:27.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11b8152b30 0x7f11b81edd60 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f11b000ca80 tx=0x7f11b000cf40 comp rx=0 tx=0).stop 2023-10-25T00:17:27.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 -- 172.21.15.107:0/1775142056 shutdown_connections 2023-10-25T00:17:27.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11b8152b30 0x7f11b81edd60 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:27.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f11b814c470 0x7f11b8140ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:27.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1198065e90 0x7f1198068350 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:27.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 --2- 172.21.15.107:0/1775142056 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f11b806f490 0x7f11b813c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:27.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 -- 172.21.15.107:0/1775142056 >> 172.21.15.107:0/1775142056 conn(0x7f11b8070f60 msgr2=0x7f11b8151a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:27.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 -- 172.21.15.107:0/1775142056 shutdown_connections 2023-10-25T00:17:27.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:27.765+0000 7f11bd656640 1 -- 172.21.15.107:0/1775142056 wait complete. 2023-10-25T00:17:27.794 INFO:tasks.mds_thrash.fs.[a]:no change 2023-10-25T00:17:28.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:28.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecaf190640 1 Processor -- start 2023-10-25T00:17:28.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecaf190640 1 -- start start 2023-10-25T00:17:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecaf190640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feca814c470 0x7feca814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecaf190640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feca8152b30 0x7feca8152f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecaf190640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca8155cd0 con 0x7feca806f490 2023-10-25T00:17:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecaf190640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca81526e0 con 0x7feca814c470 2023-10-25T00:17:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecaf190640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca80c4380 con 0x7feca8152b30 2023-10-25T00:17:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecacf2c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7feca814c470 0x7feca814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7feca7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7feca806f490 0x7feca8071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37046/0 (socket says 172.21.15.107:37046) 2023-10-25T00:17:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7feca7fff640 1 -- 172.21.15.107:0/2982571950 learned_addr learned my addr 172.21.15.107:0/2982571950 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:17:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecad72d640 1 --2- 172.21.15.107:0/2982571950 >> v2:172.21.15.110:3300/0 conn(0x7feca8152b30 0x7feca8152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7feca77fe640 1 -- 172.21.15.107:0/2982571950 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248077982 0 0) 0x7feca8155cd0 con 0x7feca806f490 2023-10-25T00:17:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7feca77fe640 1 -- 172.21.15.107:0/2982571950 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec98003530 con 0x7feca806f490 2023-10-25T00:17:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecacf2c640 1 -- 172.21.15.107:0/2982571950 >> v2:172.21.15.110:3300/0 conn(0x7feca8152b30 msgr2=0x7feca8152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecacf2c640 1 --2- 172.21.15.107:0/2982571950 >> v2:172.21.15.110:3300/0 conn(0x7feca8152b30 0x7feca8152f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecacf2c640 1 -- 172.21.15.107:0/2982571950 >> v1:172.21.15.107:6789/0 conn(0x7feca806f490 legacy=0x7feca8071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.437+0000 7fecacf2c640 1 -- 172.21.15.107:0/2982571950 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca80c48b0 con 0x7feca814c470 2023-10-25T00:17:28.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7fecacf2c640 1 --2- 172.21.15.107:0/2982571950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feca814c470 0x7feca814c850 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7feca806fa90 tx=0x7fec9c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcf0dd1e5f04d3c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7feca77fe640 1 -- 172.21.15.107:0/2982571950 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0ab600 con 0x7feca814c470 2023-10-25T00:17:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7feca77fe640 1 -- 172.21.15.107:0/2982571950 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c0a7050 con 0x7feca814c470 2023-10-25T00:17:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7feca77fe640 1 -- 172.21.15.107:0/2982571950 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0ab980 con 0x7feca814c470 2023-10-25T00:17:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7fecaf190640 1 -- 172.21.15.107:0/2982571950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feca814c470 msgr2=0x7feca814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7fecaf190640 1 --2- 172.21.15.107:0/2982571950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feca814c470 0x7feca814c850 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7feca806fa90 tx=0x7fec9c01c340 comp rx=0 tx=0).stop 2023-10-25T00:17:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7fecaf190640 1 -- 172.21.15.107:0/2982571950 shutdown_connections 2023-10-25T00:17:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7fecaf190640 1 --2- 172.21.15.107:0/2982571950 >> v2:172.21.15.110:3300/0 conn(0x7feca8152b30 0x7feca8152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7fecaf190640 1 --2- 172.21.15.107:0/2982571950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feca814c470 0x7feca814c850 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7fecaf190640 1 -- 172.21.15.107:0/2982571950 >> 172.21.15.107:0/2982571950 conn(0x7feca8070f60 msgr2=0x7feca8071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7fecaf190640 1 -- 172.21.15.107:0/2982571950 shutdown_connections 2023-10-25T00:17:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.441+0000 7fecaf190640 1 -- 172.21.15.107:0/2982571950 wait complete. 2023-10-25T00:17:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecaf190640 1 Processor -- start 2023-10-25T00:17:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecaf190640 1 -- start start 2023-10-25T00:17:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecaf190640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feca806f490 0x7feca814a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecaf190640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feca814c470 0x7feca81454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecaf190640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feca8152b30 0x7feca814a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecaf190640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca8142e70 con 0x7feca8152b30 2023-10-25T00:17:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecaf190640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca814bda0 con 0x7feca814c470 2023-10-25T00:17:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecaf190640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca8155960 con 0x7feca806f490 2023-10-25T00:17:28.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecad72d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feca8152b30 0x7feca814a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:28.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecad72d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feca8152b30 0x7feca814a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39778/0 (socket says 172.21.15.107:39778) 2023-10-25T00:17:28.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecad72d640 1 -- 172.21.15.107:0/1965706170 learned_addr learned my addr 172.21.15.107:0/1965706170 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:28.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecacf2c640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feca806f490 0x7feca814a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:28.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7feca7fff640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feca814c470 0x7feca81454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecad72d640 1 -- 172.21.15.107:0/1965706170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feca806f490 msgr2=0x7feca814a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecad72d640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feca806f490 0x7feca814a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecad72d640 1 -- 172.21.15.107:0/1965706170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feca814c470 msgr2=0x7feca81454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecad72d640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feca814c470 0x7feca81454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecad72d640 1 -- 172.21.15.107:0/1965706170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c009420 con 0x7feca8152b30 2023-10-25T00:17:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecad72d640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feca8152b30 0x7feca814a0d0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7feca0009d30 tx=0x7feca000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7feca57fa640 1 -- 172.21.15.107:0/1965706170 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0009ba0 con 0x7feca8152b30 2023-10-25T00:17:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecaf190640 1 -- 172.21.15.107:0/1965706170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca8155610 con 0x7feca8152b30 2023-10-25T00:17:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7feca57fa640 1 -- 172.21.15.107:0/1965706170 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca000f040 con 0x7feca8152b30 2023-10-25T00:17:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7feca57fa640 1 -- 172.21.15.107:0/1965706170 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca00146c0 con 0x7feca8152b30 2023-10-25T00:17:28.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecaf190640 1 -- 172.21.15.107:0/1965706170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca81eb4f0 con 0x7feca8152b30 2023-10-25T00:17:28.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7fecaf190640 1 -- 172.21.15.107:0/1965706170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca8155af0 con 0x7feca8152b30 2023-10-25T00:17:28.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.445+0000 7feca57fa640 1 -- 172.21.15.107:0/1965706170 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7feca0014820 con 0x7feca8152b30 2023-10-25T00:17:28.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.449+0000 7feca57fa640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fec8c065e90 0x7fec8c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:28.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.449+0000 7feca57fa640 1 -- 172.21.15.107:0/1965706170 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 10143+0+0 (secure 0 0 0) 0x7feca0087070 con 0x7feca8152b30 2023-10-25T00:17:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.449+0000 7fecacf2c640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fec8c065e90 0x7fec8c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.449+0000 7fecacf2c640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fec8c065e90 0x7fec8c068350 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fec9c0ad000 tx=0x7fec9c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:28.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:28.453+0000 7feca57fa640 1 -- 172.21.15.107:0/1965706170 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7feca0050e40 con 0x7feca8152b30 2023-10-25T00:17:29.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.253+0000 7fecaf190640 1 -- 172.21.15.107:0/1965706170 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feca80746e0 con 0x7feca8152b30 2023-10-25T00:17:29.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.257+0000 7feca57fa640 1 -- 172.21.15.107:0/1965706170 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+5009 (secure 0 0 0) 0x7feca0018090 con 0x7feca8152b30 2023-10-25T00:17:29.260 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:29.260 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:29.260 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 194 2023-10-25T00:17:29.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.277+0000 7fecaf190640 1 -- 172.21.15.107:0/1965706170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fec8c065e90 msgr2=0x7fec8c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:29.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.277+0000 7fecaf190640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fec8c065e90 0x7fec8c068350 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fec9c0ad000 tx=0x7fec9c008040 comp rx=0 tx=0).stop 2023-10-25T00:17:29.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.277+0000 7fecaf190640 1 -- 172.21.15.107:0/1965706170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feca8152b30 msgr2=0x7feca814a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:29.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.277+0000 7fecaf190640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feca8152b30 0x7feca814a0d0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7feca0009d30 tx=0x7feca000c490 comp rx=0 tx=0).stop 2023-10-25T00:17:29.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.281+0000 7fecaf190640 1 -- 172.21.15.107:0/1965706170 shutdown_connections 2023-10-25T00:17:29.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.281+0000 7fecaf190640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feca8152b30 0x7feca814a0d0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:29.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.281+0000 7fecaf190640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feca814c470 0x7feca81454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:29.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.281+0000 7fecaf190640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fec8c065e90 0x7fec8c068350 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:29.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.281+0000 7fecaf190640 1 --2- 172.21.15.107:0/1965706170 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7feca806f490 0x7feca814a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:29.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.285+0000 7fecaf190640 1 -- 172.21.15.107:0/1965706170 >> 172.21.15.107:0/1965706170 conn(0x7feca8070f60 msgr2=0x7feca8151c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:29.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.285+0000 7fecaf190640 1 -- 172.21.15.107:0/1965706170 shutdown_connections 2023-10-25T00:17:29.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:29.285+0000 7fecaf190640 1 -- 172.21.15.107:0/1965706170 wait complete. 2023-10-25T00:17:29.379 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:17:29.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:30.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.373+0000 7ff22a624640 1 Processor -- start 2023-10-25T00:17:30.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.373+0000 7ff22a624640 1 -- start start 2023-10-25T00:17:30.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.373+0000 7ff22a624640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2240c2430 0x7ff2240c0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:30.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.373+0000 7ff22a624640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2240c4fb0 0x7ff224165360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:30.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.373+0000 7ff22a624640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2240c2f20 con 0x7ff22408baf0 2023-10-25T00:17:30.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.373+0000 7ff22a624640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22406c990 con 0x7ff2240c4fb0 2023-10-25T00:17:30.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.373+0000 7ff22a624640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff224072580 con 0x7ff2240c2430 2023-10-25T00:17:30.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.377+0000 7ff229e23640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2240c4fb0 0x7ff224165360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:30.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.377+0000 7ff229e23640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff2240c4fb0 0x7ff224165360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39782/0 (socket says 172.21.15.107:39782) 2023-10-25T00:17:30.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.377+0000 7ff229e23640 1 -- 172.21.15.107:0/224236633 learned_addr learned my addr 172.21.15.107:0/224236633 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:30.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.377+0000 7ff229e23640 1 -- 172.21.15.107:0/224236633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2240c2430 msgr2=0x7ff2240c0890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:30.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.377+0000 7ff229e23640 1 --2- 172.21.15.107:0/224236633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2240c2430 0x7ff2240c0890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:30.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.377+0000 7ff229e23640 1 -- 172.21.15.107:0/224236633 >> v1:172.21.15.107:6789/0 conn(0x7ff22408baf0 legacy=0x7ff2240c2810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:30.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.377+0000 7ff229e23640 1 -- 172.21.15.107:0/224236633 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff224072310 con 0x7ff2240c4fb0 2023-10-25T00:17:30.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.401+0000 7ff229e23640 1 --2- 172.21.15.107:0/224236633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2240c4fb0 0x7ff224165360 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7ff2140089f0 tx=0x7ff21401e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5003860af25c4b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.413+0000 7ff21bfff640 1 -- 172.21.15.107:0/224236633 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2140a7830 con 0x7ff2240c4fb0 2023-10-25T00:17:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.413+0000 7ff21bfff640 1 -- 172.21.15.107:0/224236633 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2140ab070 con 0x7ff2240c4fb0 2023-10-25T00:17:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.413+0000 7ff21bfff640 1 -- 172.21.15.107:0/224236633 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2140a7bd0 con 0x7ff2240c4fb0 2023-10-25T00:17:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.413+0000 7ff22a624640 1 -- 172.21.15.107:0/224236633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2240c4fb0 msgr2=0x7ff224165360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.413+0000 7ff22a624640 1 --2- 172.21.15.107:0/224236633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2240c4fb0 0x7ff224165360 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7ff2140089f0 tx=0x7ff21401e8b0 comp rx=0 tx=0).stop 2023-10-25T00:17:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.413+0000 7ff22a624640 1 -- 172.21.15.107:0/224236633 shutdown_connections 2023-10-25T00:17:30.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.413+0000 7ff22a624640 1 --2- 172.21.15.107:0/224236633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2240c4fb0 0x7ff224165360 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:30.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.413+0000 7ff22a624640 1 --2- 172.21.15.107:0/224236633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2240c2430 0x7ff2240c0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:30.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.413+0000 7ff22a624640 1 -- 172.21.15.107:0/224236633 >> 172.21.15.107:0/224236633 conn(0x7ff2240a77e0 msgr2=0x7ff2240a7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:30.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.413+0000 7ff22a624640 1 -- 172.21.15.107:0/224236633 shutdown_connections 2023-10-25T00:17:30.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.413+0000 7ff22a624640 1 -- 172.21.15.107:0/224236633 wait complete. 2023-10-25T00:17:30.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff22a624640 1 Processor -- start 2023-10-25T00:17:30.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff22a624640 1 -- start start 2023-10-25T00:17:30.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff22a624640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff22408baf0 0x7ff224070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:30.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff22a624640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2240c2430 0x7ff22416ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:30.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff22a624640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2240c4fb0 0x7ff22416bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:30.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff22a624640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2240aae50 con 0x7ff2240c2430 2023-10-25T00:17:30.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff22a624640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2240bac70 con 0x7ff2240c4fb0 2023-10-25T00:17:30.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff22a624640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2240baf50 con 0x7ff22408baf0 2023-10-25T00:17:30.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff229622640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff22408baf0 0x7ff224070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:30.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff229622640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff22408baf0 0x7ff224070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:58964/0 (socket says 172.21.15.107:58964) 2023-10-25T00:17:30.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff229622640 1 -- 172.21.15.107:0/3752555682 learned_addr learned my addr 172.21.15.107:0/3752555682 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:30.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff229622640 1 -- 172.21.15.107:0/3752555682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2240c4fb0 msgr2=0x7ff22416bf60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:17:30.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff229622640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2240c4fb0 0x7ff22416bf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:30.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff229622640 1 -- 172.21.15.107:0/3752555682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2240c2430 msgr2=0x7ff22416ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:30.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff229622640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2240c2430 0x7ff22416ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:30.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.417+0000 7ff229622640 1 -- 172.21.15.107:0/3752555682 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140086d0 con 0x7ff22408baf0 2023-10-25T00:17:30.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.421+0000 7ff229622640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff22408baf0 0x7ff224070ff0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00b3f0 tx=0x7ff21c00b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.429+0000 7ff219ffb640 1 -- 172.21.15.107:0/3752555682 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c011830 con 0x7ff22408baf0 2023-10-25T00:17:30.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.429+0000 7ff219ffb640 1 -- 172.21.15.107:0/3752555682 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c0119b0 con 0x7ff22408baf0 2023-10-25T00:17:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.429+0000 7ff219ffb640 1 -- 172.21.15.107:0/3752555682 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c011d60 con 0x7ff22408baf0 2023-10-25T00:17:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.429+0000 7ff22a624640 1 -- 172.21.15.107:0/3752555682 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2240bb230 con 0x7ff22408baf0 2023-10-25T00:17:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.429+0000 7ff22a624640 1 -- 172.21.15.107:0/3752555682 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2240a73b0 con 0x7ff22408baf0 2023-10-25T00:17:30.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.429+0000 7ff22a624640 1 -- 172.21.15.107:0/3752555682 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22404e050 con 0x7ff22408baf0 2023-10-25T00:17:30.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.429+0000 7ff219ffb640 1 -- 172.21.15.107:0/3752555682 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff21c01a400 con 0x7ff22408baf0 2023-10-25T00:17:30.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.429+0000 7ff219ffb640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff208065f30 0x7ff2080683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:30.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.433+0000 7ff219ffb640 1 -- 172.21.15.107:0/3752555682 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 10143+0+0 (secure 0 0 0) 0x7ff21c085000 con 0x7ff22408baf0 2023-10-25T00:17:30.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.437+0000 7ff219ffb640 1 -- 172.21.15.107:0/3752555682 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff21c051770 con 0x7ff22408baf0 2023-10-25T00:17:30.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.437+0000 7ff228e21640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff208065f30 0x7ff2080683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:30.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:30.437+0000 7ff228e21640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff208065f30 0x7ff2080683f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff2240a4930 tx=0x7ff2200015c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:31.341 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:17:31.337+0000 7f4d1ee91640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:17:31.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:31.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.429+0000 7ff22a624640 1 -- 172.21.15.107:0/3752555682 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2240bb090 con 0x7ff22408baf0 2023-10-25T00:17:31.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.429+0000 7ff219ffb640 1 -- 172.21.15.107:0/3752555682 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+5009 (secure 0 0 0) 0x7ff2240bb090 con 0x7ff22408baf0 2023-10-25T00:17:31.434 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:31.434 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:31.435 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 194 2023-10-25T00:17:31.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.437+0000 7ff22a624640 1 -- 172.21.15.107:0/3752555682 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff208065f30 msgr2=0x7ff2080683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:31.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.437+0000 7ff22a624640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff208065f30 0x7ff2080683f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff2240a4930 tx=0x7ff2200015c0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.437+0000 7ff22a624640 1 -- 172.21.15.107:0/3752555682 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff22408baf0 msgr2=0x7ff224070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.437+0000 7ff22a624640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff22408baf0 0x7ff224070ff0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00b3f0 tx=0x7ff21c00b8b0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.437+0000 7ff22a624640 1 -- 172.21.15.107:0/3752555682 shutdown_connections 2023-10-25T00:17:31.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.437+0000 7ff22a624640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff2240c4fb0 0x7ff22416bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.437+0000 7ff22a624640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2240c2430 0x7ff22416ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.437+0000 7ff22a624640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff208065f30 0x7ff2080683f0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.437+0000 7ff22a624640 1 --2- 172.21.15.107:0/3752555682 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff22408baf0 0x7ff224070ff0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.441+0000 7ff22a624640 1 -- 172.21.15.107:0/3752555682 >> 172.21.15.107:0/3752555682 conn(0x7ff2240a77e0 msgr2=0x7ff2240bd960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:31.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.441+0000 7ff22a624640 1 -- 172.21.15.107:0/3752555682 shutdown_connections 2023-10-25T00:17:31.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.441+0000 7ff22a624640 1 -- 172.21.15.107:0/3752555682 wait complete. 2023-10-25T00:17:31.512 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=7030 2023-10-25T00:17:31.513 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.c 2023-10-25T00:17:31.513 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2023-10-25T00:17:31.513 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:17:31.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:17:31.516 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:17:31.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:31.756 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:17:31.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1e149640 1 Processor -- start 2023-10-25T00:17:31.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1e149640 1 -- start start 2023-10-25T00:17:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1e149640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4d1004dbe0 0x7f4d1000e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1e149640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4d1004e0f0 0x7f4d1000e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1e149640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d100da610 con 0x7f4d100d7bf0 2023-10-25T00:17:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1e149640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d100d7fd0 con 0x7f4d1004e0f0 2023-10-25T00:17:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1e149640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100da8b0 con 0x7f4d1004dbe0 2023-10-25T00:17:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1d948640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4d1004e0f0 0x7f4d1000e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1d948640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4d1004e0f0 0x7f4d1000e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39806/0 (socket says 172.21.15.107:39806) 2023-10-25T00:17:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1d948640 1 -- 172.21.15.107:0/1079009735 learned_addr learned my addr 172.21.15.107:0/1079009735 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1d948640 1 -- 172.21.15.107:0/1079009735 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4d1004dbe0 msgr2=0x7f4d1000e360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1d948640 1 --2- 172.21.15.107:0/1079009735 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4d1004dbe0 0x7f4d1000e360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1d948640 1 -- 172.21.15.107:0/1079009735 >> v1:172.21.15.107:6789/0 conn(0x7f4d100d7bf0 legacy=0x7f4d100073b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.849+0000 7f4d1d948640 1 -- 172.21.15.107:0/1079009735 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d100dab90 con 0x7f4d1004e0f0 2023-10-25T00:17:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.853+0000 7f4d1d948640 1 --2- 172.21.15.107:0/1079009735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d1004e0f0 0x7f4d1000e8a0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800ff30 tx=0x7f4d0802def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55086a0062ae4377 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:31.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.853+0000 7f4d0ffff640 1 -- 172.21.15.107:0/1079009735 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d080b6e90 con 0x7f4d1004e0f0 2023-10-25T00:17:31.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.853+0000 7f4d0ffff640 1 -- 172.21.15.107:0/1079009735 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d080b6e90 con 0x7f4d1004e0f0 2023-10-25T00:17:31.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.853+0000 7f4d1e149640 1 -- 172.21.15.107:0/1079009735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d1004e0f0 msgr2=0x7f4d1000e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:31.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.853+0000 7f4d0ffff640 1 -- 172.21.15.107:0/1079009735 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d080be440 con 0x7f4d1004e0f0 2023-10-25T00:17:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.853+0000 7f4d1e149640 1 --2- 172.21.15.107:0/1079009735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d1004e0f0 0x7f4d1000e8a0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f4d0800ff30 tx=0x7f4d0802def0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.865+0000 7f4d1e149640 1 -- 172.21.15.107:0/1079009735 shutdown_connections 2023-10-25T00:17:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.865+0000 7f4d1e149640 1 --2- 172.21.15.107:0/1079009735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d1004e0f0 0x7f4d1000e8a0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.865+0000 7f4d1e149640 1 --2- 172.21.15.107:0/1079009735 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4d1004dbe0 0x7f4d1000e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.865+0000 7f4d1e149640 1 -- 172.21.15.107:0/1079009735 >> 172.21.15.107:0/1079009735 conn(0x7f4d1004c760 msgr2=0x7f4d1004cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:31.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.885+0000 7f4d1e149640 1 -- 172.21.15.107:0/1079009735 shutdown_connections 2023-10-25T00:17:31.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.885+0000 7f4d1e149640 1 -- 172.21.15.107:0/1079009735 wait complete. 2023-10-25T00:17:31.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1e149640 1 Processor -- start 2023-10-25T00:17:31.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1e149640 1 -- start start 2023-10-25T00:17:31.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1e149640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d1004dbe0 0x7f4d10171d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1e149640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d1004e0f0 0x7f4d100d2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1e149640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4d100431f0 0x7f4d100d27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1e149640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d101728f0 con 0x7f4d1004e0f0 2023-10-25T00:17:31.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1e149640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d10172650 con 0x7f4d1004dbe0 2023-10-25T00:17:31.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1e149640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1000e030 con 0x7f4d100431f0 2023-10-25T00:17:31.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1c946640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d1004e0f0 0x7f4d100d2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:31.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1c946640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d1004e0f0 0x7f4d100d2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39812/0 (socket says 172.21.15.107:39812) 2023-10-25T00:17:31.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1c946640 1 -- 172.21.15.107:0/1526103029 learned_addr learned my addr 172.21.15.107:0/1526103029 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1d147640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d1004dbe0 0x7f4d10171d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1d948640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4d100431f0 0x7f4d100d27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1c946640 1 -- 172.21.15.107:0/1526103029 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4d100431f0 msgr2=0x7f4d100d27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1c946640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4d100431f0 0x7f4d100d27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1c946640 1 -- 172.21.15.107:0/1526103029 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d1004dbe0 msgr2=0x7f4d10171d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:31.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1c946640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d1004dbe0 0x7f4d10171d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1c946640 1 -- 172.21.15.107:0/1526103029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d0800fd10 con 0x7f4d1004e0f0 2023-10-25T00:17:31.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1c946640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d1004e0f0 0x7f4d100d2260 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f4d18000b70 tx=0x7f4d18069810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:31.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d0dffb640 1 -- 172.21.15.107:0/1526103029 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d180723c0 con 0x7f4d1004e0f0 2023-10-25T00:17:31.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d0dffb640 1 -- 172.21.15.107:0/1526103029 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d18072560 con 0x7f4d1004e0f0 2023-10-25T00:17:31.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.889+0000 7f4d1e149640 1 -- 172.21.15.107:0/1526103029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1000e310 con 0x7f4d1004e0f0 2023-10-25T00:17:31.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.893+0000 7f4d1e149640 1 -- 172.21.15.107:0/1526103029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d10046db0 con 0x7f4d1004e0f0 2023-10-25T00:17:31.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.893+0000 7f4d0dffb640 1 -- 172.21.15.107:0/1526103029 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18072990 con 0x7f4d1004e0f0 2023-10-25T00:17:31.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.893+0000 7f4d0dffb640 1 -- 172.21.15.107:0/1526103029 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4d18093400 con 0x7f4d1004e0f0 2023-10-25T00:17:31.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.893+0000 7f4d0dffb640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cfc065fe0 0x7f4cfc0684a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.893+0000 7f4d0dffb640 1 -- 172.21.15.107:0/1526103029 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 10143+0+0 (secure 0 0 0) 0x7f4d18101170 con 0x7f4d1004e0f0 2023-10-25T00:17:31.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.897+0000 7f4d1e149640 1 -- 172.21.15.107:0/1526103029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d1000de90 con 0x7f4d1004e0f0 2023-10-25T00:17:31.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.901+0000 7f4d1d147640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cfc065fe0 0x7f4cfc0684a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:31.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.901+0000 7f4d0dffb640 1 -- 172.21.15.107:0/1526103029 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4d1000de90 con 0x7f4d1004e0f0 2023-10-25T00:17:31.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.905+0000 7f4d1d147640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cfc065fe0 0x7f4cfc0684a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4d100118f0 tx=0x7f4d140090e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:31.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77be4f1640 1 Processor -- start 2023-10-25T00:17:31.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77be4f1640 1 -- start start 2023-10-25T00:17:31.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77be4f1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77b00dbfa0 0x7f77b00dfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77be4f1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f77b003e900 0x7f77b003ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77be4f1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b00dede0 con 0x7f77b000b890 2023-10-25T00:17:31.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77be4f1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b000b520 con 0x7f77b003e900 2023-10-25T00:17:31.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77be4f1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b00df980 con 0x7f77b00dbfa0 2023-10-25T00:17:31.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77bdcf0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f77b003e900 0x7f77b003ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:31.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77bd4ef640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77b00dbfa0 0x7f77b00dfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:31.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77bdcf0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f77b003e900 0x7f77b003ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:41866/0 (socket says 172.21.15.107:41866) 2023-10-25T00:17:31.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77bdcf0640 1 -- 172.21.15.107:0/1044038721 learned_addr learned my addr 172.21.15.107:0/1044038721 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77bccee640 1 --1- 172.21.15.107:0/1044038721 >> v1:172.21.15.110:6789/0 conn(0x7f77b000b890 0x7f77b00df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:59900/0 (socket says 172.21.15.107:59900) 2023-10-25T00:17:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77bdcf0640 1 -- 172.21.15.107:0/1044038721 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77b00dbfa0 msgr2=0x7f77b00dfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77bdcf0640 1 --2- 172.21.15.107:0/1044038721 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77b00dbfa0 0x7f77b00dfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77bdcf0640 1 -- 172.21.15.107:0/1044038721 >> v1:172.21.15.110:6789/0 conn(0x7f77b000b890 legacy=0x7f77b00df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:31.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.937+0000 7f77bdcf0640 1 -- 172.21.15.107:0/1044038721 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b0007220 con 0x7f77b003e900 2023-10-25T00:17:31.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.941+0000 7f77bdcf0640 1 --2- 172.21.15.107:0/1044038721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77b003e900 0x7f77b003ed50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f77b802c850 tx=0x7f77b8096de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f58ae63d0e1f37d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:31.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.941+0000 7f77affff640 1 -- 172.21.15.107:0/1044038721 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b811fb90 con 0x7f77b003e900 2023-10-25T00:17:31.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.941+0000 7f77affff640 1 -- 172.21.15.107:0/1044038721 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b811fd30 con 0x7f77b003e900 2023-10-25T00:17:31.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.941+0000 7f77affff640 1 -- 172.21.15.107:0/1044038721 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b8127640 con 0x7f77b003e900 2023-10-25T00:17:31.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.941+0000 7f77be4f1640 1 -- 172.21.15.107:0/1044038721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77b003e900 msgr2=0x7f77b003ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:31.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.941+0000 7f77be4f1640 1 --2- 172.21.15.107:0/1044038721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77b003e900 0x7f77b003ed50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f77b802c850 tx=0x7f77b8096de0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 -- 172.21.15.107:0/1044038721 shutdown_connections 2023-10-25T00:17:31.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 --2- 172.21.15.107:0/1044038721 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77b003e900 0x7f77b003ed50 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 --2- 172.21.15.107:0/1044038721 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77b00dbfa0 0x7f77b00dfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 -- 172.21.15.107:0/1044038721 >> 172.21.15.107:0/1044038721 conn(0x7f77b0005ff0 msgr2=0x7f77b0006400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:31.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 -- 172.21.15.107:0/1044038721 shutdown_connections 2023-10-25T00:17:31.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 -- 172.21.15.107:0/1044038721 wait complete. 2023-10-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 Processor -- start 2023-10-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 -- start start 2023-10-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77b000b890 0x7f77b01940e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77b003e900 0x7f77b003fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77b00dbfa0 0x7f77b00e8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b003fa20 con 0x7f77b000b890 2023-10-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b0194620 con 0x7f77b00dbfa0 2023-10-25T00:17:31.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77be4f1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b003f460 con 0x7f77b003e900 2023-10-25T00:17:31.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77bdcf0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77b00dbfa0 0x7f77b00e8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:31.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77bccee640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77b003e900 0x7f77b003fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:31.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77bd4ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77b000b890 0x7f77b01940e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:31.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77bd4ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77b000b890 0x7f77b01940e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39824/0 (socket says 172.21.15.107:39824) 2023-10-25T00:17:31.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.957+0000 7f77bd4ef640 1 -- 172.21.15.107:0/1417855166 learned_addr learned my addr 172.21.15.107:0/1417855166 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77bd4ef640 1 -- 172.21.15.107:0/1417855166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77b003e900 msgr2=0x7f77b003fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77bd4ef640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77b003e900 0x7f77b003fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77bd4ef640 1 -- 172.21.15.107:0/1417855166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77b00dbfa0 msgr2=0x7f77b00e8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77bd4ef640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77b00dbfa0 0x7f77b00e8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:31.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77bd4ef640 1 -- 172.21.15.107:0/1417855166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b804e5c0 con 0x7f77b000b890 2023-10-25T00:17:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77bd4ef640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77b000b890 0x7f77b01940e0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f77a800e930 tx=0x7f77a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77adffb640 1 -- 172.21.15.107:0/1417855166 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a8009690 con 0x7f77b000b890 2023-10-25T00:17:31.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77adffb640 1 -- 172.21.15.107:0/1417855166 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a800ca00 con 0x7f77b000b890 2023-10-25T00:17:31.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77adffb640 1 -- 172.21.15.107:0/1417855166 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a800cd70 con 0x7f77b000b890 2023-10-25T00:17:31.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77be4f1640 1 -- 172.21.15.107:0/1417855166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b003f740 con 0x7f77b000b890 2023-10-25T00:17:31.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77be4f1640 1 -- 172.21.15.107:0/1417855166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b0193bc0 con 0x7f77b000b890 2023-10-25T00:17:31.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.961+0000 7f77be4f1640 1 -- 172.21.15.107:0/1417855166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b0193740 con 0x7f77b000b890 2023-10-25T00:17:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.973+0000 7f77adffb640 1 -- 172.21.15.107:0/1417855166 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f77a8013470 con 0x7f77b000b890 2023-10-25T00:17:31.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.973+0000 7f77adffb640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f779c065f00 0x7f779c0683c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.973+0000 7f77bccee640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f779c065f00 0x7f779c0683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:31.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.973+0000 7f77bccee640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f779c065f00 0x7f779c0683c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f77b00de040 tx=0x7f77b4007e50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:31.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.973+0000 7f77adffb640 1 -- 172.21.15.107:0/1417855166 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 10143+0+0 (secure 0 0 0) 0x7f77a80849a0 con 0x7f77b000b890 2023-10-25T00:17:31.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:31.993+0000 7f77adffb640 1 -- 172.21.15.107:0/1417855166 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f77b0193740 con 0x7f77b000b890 2023-10-25T00:17:32.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.625+0000 7f4d1e149640 1 -- 172.21.15.107:0/1526103029 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d1000de90 con 0x7f4d1004e0f0 2023-10-25T00:17:32.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.625+0000 7f4d0dffb640 1 -- 172.21.15.107:0/1526103029 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+5770 (secure 0 0 0) 0x7f4d1000de90 con 0x7f4d1004e0f0 2023-10-25T00:17:32.629 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:32.630 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:32.630 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 195 2023-10-25T00:17:32.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 -- 172.21.15.107:0/1526103029 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cfc065fe0 msgr2=0x7f4cfc0684a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:32.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cfc065fe0 0x7f4cfc0684a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4d100118f0 tx=0x7f4d140090e0 comp rx=0 tx=0).stop 2023-10-25T00:17:32.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 -- 172.21.15.107:0/1526103029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d1004e0f0 msgr2=0x7f4d100d2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:32.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d1004e0f0 0x7f4d100d2260 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f4d18000b70 tx=0x7f4d18069810 comp rx=0 tx=0).stop 2023-10-25T00:17:32.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 -- 172.21.15.107:0/1526103029 shutdown_connections 2023-10-25T00:17:32.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cfc065fe0 0x7f4cfc0684a0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:32.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4d100431f0 0x7f4d100d27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:32.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d1004e0f0 0x7f4d100d2260 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:32.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 --2- 172.21.15.107:0/1526103029 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d1004dbe0 0x7f4d10171d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:32.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 -- 172.21.15.107:0/1526103029 >> 172.21.15.107:0/1526103029 conn(0x7f4d1004c760 msgr2=0x7f4d100d5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:32.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 -- 172.21.15.107:0/1526103029 shutdown_connections 2023-10-25T00:17:32.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.629+0000 7f4d1e149640 1 -- 172.21.15.107:0/1526103029 wait complete. 2023-10-25T00:17:32.681 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:17:32.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.861+0000 7f77be4f1640 1 -- 172.21.15.107:0/1417855166 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77b0193740 con 0x7f77b000b890 2023-10-25T00:17:32.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.861+0000 7f77adffb640 1 -- 172.21.15.107:0/1417855166 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+5770 (secure 0 0 0) 0x7f77b0193740 con 0x7f77b000b890 2023-10-25T00:17:32.864 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:32.865 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:32.865 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 195 2023-10-25T00:17:32.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 -- 172.21.15.107:0/1417855166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f779c065f00 msgr2=0x7f779c0683c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:32.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f779c065f00 0x7f779c0683c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f77b00de040 tx=0x7f77b4007e50 comp rx=0 tx=0).stop 2023-10-25T00:17:32.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 -- 172.21.15.107:0/1417855166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77b000b890 msgr2=0x7f77b01940e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:32.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77b000b890 0x7f77b01940e0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f77a800e930 tx=0x7f77a800edf0 comp rx=0 tx=0).stop 2023-10-25T00:17:32.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 -- 172.21.15.107:0/1417855166 shutdown_connections 2023-10-25T00:17:32.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77b00dbfa0 0x7f77b00e8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:32.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f779c065f00 0x7f779c0683c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:32.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f77b003e900 0x7f77b003fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:32.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 --2- 172.21.15.107:0/1417855166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77b000b890 0x7f77b01940e0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:32.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 -- 172.21.15.107:0/1417855166 >> 172.21.15.107:0/1417855166 conn(0x7f77b0005ff0 msgr2=0x7f77b00de2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:32.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 -- 172.21.15.107:0/1417855166 shutdown_connections 2023-10-25T00:17:32.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:32.865+0000 7f77be4f1640 1 -- 172.21.15.107:0/1417855166 wait complete. 2023-10-25T00:17:32.928 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2023-10-25T00:17:32.929 INFO:tasks.mds_thrash.fs.[a]:waiting for 60 secs before thrashing 2023-10-25T00:17:34.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:35.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.085+0000 7f40e7427640 1 Processor -- start 2023-10-25T00:17:35.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.085+0000 7f40e7427640 1 -- start start 2023-10-25T00:17:35.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e7427640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d800bd10 0x7f40d80db9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e7427640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f40d8048720 0x7f40d804a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e7427640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f40d80d5700 0x7f40d80d5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e7427640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d80084e0 con 0x7f40d8048720 2023-10-25T00:17:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e6425640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d800bd10 0x7f40d80db9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e5c24640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f40d8048720 0x7f40d804a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e6425640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d800bd10 0x7f40d80db9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:43760/0 (socket says 172.21.15.107:43760) 2023-10-25T00:17:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e5c24640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f40d8048720 0x7f40d804a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46878/0 (socket says 172.21.15.107:46878) 2023-10-25T00:17:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e5c24640 1 -- 172.21.15.107:0/645890868 learned_addr learned my addr 172.21.15.107:0/645890868 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:35.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e6c26640 1 --2- 172.21.15.107:0/645890868 >> v2:172.21.15.110:3300/0 conn(0x7f40d80d5700 0x7f40d80d5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:35.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e7427640 1 -- 172.21.15.107:0/645890868 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d80063a0 con 0x7f40d80d5700 2023-10-25T00:17:35.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e7427640 1 -- 172.21.15.107:0/645890868 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d80d60d0 con 0x7f40d800bd10 2023-10-25T00:17:35.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e6c26640 1 -- 172.21.15.107:0/645890868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d800bd10 msgr2=0x7f40d80db9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:35.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e6c26640 1 --2- 172.21.15.107:0/645890868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d800bd10 0x7f40d80db9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e6c26640 1 -- 172.21.15.107:0/645890868 >> v2:172.21.15.107:3300/0 conn(0x7f40d8048720 msgr2=0x7f40d804a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:35.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e6c26640 1 --2- 172.21.15.107:0/645890868 >> v2:172.21.15.107:3300/0 conn(0x7f40d8048720 0x7f40d804a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e6c26640 1 -- 172.21.15.107:0/645890868 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d80065f0 con 0x7f40d80d5700 2023-10-25T00:17:35.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e5c24640 1 --2- 172.21.15.107:0/645890868 >> v2:172.21.15.107:3300/0 conn(0x7f40d8048720 0x7f40d804a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:17:35.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.089+0000 7f40e6425640 1 --2- 172.21.15.107:0/645890868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d800bd10 0x7f40d80db9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:17:35.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.093+0000 7f40e6c26640 1 --2- 172.21.15.107:0/645890868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d80d5700 0x7f40d80d5b90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f40e0063dd0 tx=0x7f40e0096d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1a9148f4067d159 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:35.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.093+0000 7f40e5423640 1 -- 172.21.15.107:0/645890868 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e011fc10 con 0x7f40d80d5700 2023-10-25T00:17:35.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.093+0000 7f40e5423640 1 -- 172.21.15.107:0/645890868 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e011fdb0 con 0x7f40d80d5700 2023-10-25T00:17:35.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.093+0000 7f40e5423640 1 -- 172.21.15.107:0/645890868 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e0127640 con 0x7f40d80d5700 2023-10-25T00:17:35.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.097+0000 7f40e7427640 1 -- 172.21.15.107:0/645890868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d80d5700 msgr2=0x7f40d80d5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:35.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.097+0000 7f40e7427640 1 --2- 172.21.15.107:0/645890868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d80d5700 0x7f40d80d5b90 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f40e0063dd0 tx=0x7f40e0096d80 comp rx=0 tx=0).stop 2023-10-25T00:17:35.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.101+0000 7f40e7427640 1 -- 172.21.15.107:0/645890868 shutdown_connections 2023-10-25T00:17:35.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.101+0000 7f40e7427640 1 --2- 172.21.15.107:0/645890868 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d80d5700 0x7f40d80d5b90 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.101+0000 7f40e7427640 1 --2- 172.21.15.107:0/645890868 >> v2:172.21.15.107:3300/0 conn(0x7f40d8048720 0x7f40d804a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.101+0000 7f40e7427640 1 --2- 172.21.15.107:0/645890868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d800bd10 0x7f40d80db9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.101+0000 7f40e7427640 1 -- 172.21.15.107:0/645890868 >> 172.21.15.107:0/645890868 conn(0x7f40d8006c30 msgr2=0x7f40d8007040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:35.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.105+0000 7f40e7427640 1 -- 172.21.15.107:0/645890868 shutdown_connections 2023-10-25T00:17:35.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.113+0000 7f40e7427640 1 -- 172.21.15.107:0/645890868 wait complete. 2023-10-25T00:17:35.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e7427640 1 Processor -- start 2023-10-25T00:17:35.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e7427640 1 -- start start 2023-10-25T00:17:35.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e7427640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d800bd10 0x7f40d80d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:35.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e7427640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d8048720 0x7f40d80450c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:35.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e7427640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d80d5700 0x7f40d8045600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:35.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e5c24640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d8048720 0x7f40d80450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:35.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e6c26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d80d5700 0x7f40d8045600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:35.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e6c26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d80d5700 0x7f40d8045600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46882/0 (socket says 172.21.15.107:46882) 2023-10-25T00:17:35.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e5c24640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d8048720 0x7f40d80450c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:43774/0 (socket says 172.21.15.107:43774) 2023-10-25T00:17:35.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e6c26640 1 -- 172.21.15.107:0/1899100852 learned_addr learned my addr 172.21.15.107:0/1899100852 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:35.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d8045b40 con 0x7f40d80d5700 2023-10-25T00:17:35.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d802f110 con 0x7f40d800bd10 2023-10-25T00:17:35.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d802f3d0 con 0x7f40d8048720 2023-10-25T00:17:35.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e6425640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d800bd10 0x7f40d80d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:35.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e5c24640 1 -- 172.21.15.107:0/1899100852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d800bd10 msgr2=0x7f40d80d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:35.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e5c24640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d800bd10 0x7f40d80d1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e5c24640 1 -- 172.21.15.107:0/1899100852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d80d5700 msgr2=0x7f40d8045600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:35.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e5c24640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d80d5700 0x7f40d8045600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e5c24640 1 -- 172.21.15.107:0/1899100852 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc009420 con 0x7f40d8048720 2023-10-25T00:17:35.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e5c24640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d8048720 0x7f40d80450c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f40d8048b20 tx=0x7f40d000eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:35.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40d6ffd640 1 -- 172.21.15.107:0/1899100852 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0004030 con 0x7f40d8048720 2023-10-25T00:17:35.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40d6ffd640 1 -- 172.21.15.107:0/1899100852 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d0004190 con 0x7f40d8048720 2023-10-25T00:17:35.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d00094d0 con 0x7f40d8048720 2023-10-25T00:17:35.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40d6ffd640 1 -- 172.21.15.107:0/1899100852 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0004520 con 0x7f40d8048720 2023-10-25T00:17:35.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.117+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40e004e5c0 con 0x7f40d8048720 2023-10-25T00:17:35.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.121+0000 7f40d6ffd640 1 -- 172.21.15.107:0/1899100852 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f40d000c960 con 0x7f40d8048720 2023-10-25T00:17:35.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.125+0000 7f40d6ffd640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40c4065e60 0x7f40c4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:35.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.125+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c8005260 con 0x7f40d8048720 2023-10-25T00:17:35.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.125+0000 7f40d6ffd640 1 -- 172.21.15.107:0/1899100852 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 10143+0+0 (secure 0 0 0) 0x7f40d0084840 con 0x7f40d8048720 2023-10-25T00:17:35.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.133+0000 7f40d6ffd640 1 -- 172.21.15.107:0/1899100852 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f40d0050f30 con 0x7f40d8048720 2023-10-25T00:17:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.133+0000 7f40e6425640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40c4065e60 0x7f40c4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:35.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.133+0000 7f40e6425640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40c4065e60 0x7f40c4068320 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0093f0 tx=0x7f40dc009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:35.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.753+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40c8005a90 con 0x7f40d8048720 2023-10-25T00:17:35.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.753+0000 7f40d6ffd640 1 -- 172.21.15.107:0/1899100852 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+5770 (secure 0 0 0) 0x7f40d0020070 con 0x7f40d8048720 2023-10-25T00:17:35.757 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:35.757 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:35.758 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 195 2023-10-25T00:17:35.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.757+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40c4065e60 msgr2=0x7f40c4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:35.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.757+0000 7f40e7427640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40c4065e60 0x7f40c4068320 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f40dc0093f0 tx=0x7f40dc009190 comp rx=0 tx=0).stop 2023-10-25T00:17:35.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.757+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d8048720 msgr2=0x7f40d80450c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:35.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.757+0000 7f40e7427640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d8048720 0x7f40d80450c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f40d8048b20 tx=0x7f40d000eda0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.761+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 shutdown_connections 2023-10-25T00:17:35.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.761+0000 7f40e7427640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40d80d5700 0x7f40d8045600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.761+0000 7f40e7427640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40c4065e60 0x7f40c4068320 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.761+0000 7f40e7427640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f40d8048720 0x7f40d80450c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.761+0000 7f40e7427640 1 --2- 172.21.15.107:0/1899100852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40d800bd10 0x7f40d80d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:35.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.761+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 >> 172.21.15.107:0/1899100852 conn(0x7f40d8006c30 msgr2=0x7f40d802c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:35.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.761+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 shutdown_connections 2023-10-25T00:17:35.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:35.761+0000 7f40e7427640 1 -- 172.21.15.107:0/1899100852 wait complete. 2023-10-25T00:17:35.839 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:17:37.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:38.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.141+0000 7f340f103640 1 Processor -- start 2023-10-25T00:17:38.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.141+0000 7f340f103640 1 -- start start 2023-10-25T00:17:38.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.145+0000 7f340f103640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3400009140 0x7f34000daf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:38.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.145+0000 7f340f103640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3400008730 con 0x7f34000472b0 2023-10-25T00:17:38.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.145+0000 7f340f103640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3400008f40 con 0x7f34000d47b0 2023-10-25T00:17:38.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.145+0000 7f340f103640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34000daae0 con 0x7f3400009140 2023-10-25T00:17:38.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.145+0000 7f340e902640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f34000472b0 0x7f34000e19d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52768/0 (socket says 172.21.15.107:52768) 2023-10-25T00:17:38.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.145+0000 7f340e902640 1 -- 172.21.15.107:0/1191688596 learned_addr learned my addr 172.21.15.107:0/1191688596 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:17:38.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.145+0000 7f340d900640 1 --2- 172.21.15.107:0/1191688596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3400009140 0x7f34000daf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:38.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.145+0000 7f340d900640 1 -- 172.21.15.107:0/1191688596 >> v1:172.21.15.110:6789/0 conn(0x7f34000d47b0 legacy=0x7f34000d8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:38.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.145+0000 7f340d900640 1 -- 172.21.15.107:0/1191688596 >> v1:172.21.15.107:6789/0 conn(0x7f34000472b0 legacy=0x7f34000e19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:38.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.145+0000 7f340d900640 1 -- 172.21.15.107:0/1191688596 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3400008980 con 0x7f3400009140 2023-10-25T00:17:38.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.149+0000 7f340d900640 1 --2- 172.21.15.107:0/1191688596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3400009140 0x7f34000daf70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f33f80097f0 tx=0x7f33f801c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=447029982db61abc server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:38.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.149+0000 7f340d0ff640 1 -- 172.21.15.107:0/1191688596 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f80ab600 con 0x7f3400009140 2023-10-25T00:17:38.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.149+0000 7f340d0ff640 1 -- 172.21.15.107:0/1191688596 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f80a7050 con 0x7f3400009140 2023-10-25T00:17:38.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.149+0000 7f340d0ff640 1 -- 172.21.15.107:0/1191688596 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f80ab980 con 0x7f3400009140 2023-10-25T00:17:38.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.149+0000 7f340f103640 1 -- 172.21.15.107:0/1191688596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3400009140 msgr2=0x7f34000daf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:38.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.149+0000 7f340f103640 1 --2- 172.21.15.107:0/1191688596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3400009140 0x7f34000daf70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f33f80097f0 tx=0x7f33f801c320 comp rx=0 tx=0).stop 2023-10-25T00:17:38.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.149+0000 7f340f103640 1 -- 172.21.15.107:0/1191688596 shutdown_connections 2023-10-25T00:17:38.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.149+0000 7f340f103640 1 --2- 172.21.15.107:0/1191688596 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3400009140 0x7f34000daf70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:38.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.149+0000 7f340f103640 1 -- 172.21.15.107:0/1191688596 >> 172.21.15.107:0/1191688596 conn(0x7f3400007c10 msgr2=0x7f3400008020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:38.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.161+0000 7f340f103640 1 -- 172.21.15.107:0/1191688596 shutdown_connections 2023-10-25T00:17:38.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.161+0000 7f340f103640 1 -- 172.21.15.107:0/1191688596 wait complete. 2023-10-25T00:17:38.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340f103640 1 Processor -- start 2023-10-25T00:17:38.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340f103640 1 -- start start 2023-10-25T00:17:38.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340f103640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3400009140 0x7f340004b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:38.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340f103640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34000472b0 0x7f34000d0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:38.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340f103640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34000d47b0 0x7f34000c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:38.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340f103640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340004aa60 con 0x7f34000d47b0 2023-10-25T00:17:38.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340f103640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34000d05d0 con 0x7f3400009140 2023-10-25T00:17:38.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340f103640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3400177460 con 0x7f34000472b0 2023-10-25T00:17:38.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340e101640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3400009140 0x7f340004b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:38.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340e902640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34000d47b0 0x7f34000c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:38.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340e902640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34000d47b0 0x7f34000c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46888/0 (socket says 172.21.15.107:46888) 2023-10-25T00:17:38.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340e902640 1 -- 172.21.15.107:0/483330380 learned_addr learned my addr 172.21.15.107:0/483330380 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:38.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340e101640 1 -- 172.21.15.107:0/483330380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34000472b0 msgr2=0x7f34000d0090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:17:38.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340e101640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34000472b0 0x7f34000d0090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:38.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340e101640 1 -- 172.21.15.107:0/483330380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34000d47b0 msgr2=0x7f34000c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:38.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340e101640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34000d47b0 0x7f34000c7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:38.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.165+0000 7f340e101640 1 -- 172.21.15.107:0/483330380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f8009420 con 0x7f3400009140 2023-10-25T00:17:38.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.169+0000 7f340e101640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3400009140 0x7f340004b590 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f340400c410 tx=0x7f340400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:38.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.181+0000 7f33feffd640 1 -- 172.21.15.107:0/483330380 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3404011790 con 0x7f3400009140 2023-10-25T00:17:38.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.181+0000 7f340f103640 1 -- 172.21.15.107:0/483330380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3400177740 con 0x7f3400009140 2023-10-25T00:17:38.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.181+0000 7f340f103640 1 -- 172.21.15.107:0/483330380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3400177d00 con 0x7f3400009140 2023-10-25T00:17:38.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.181+0000 7f33feffd640 1 -- 172.21.15.107:0/483330380 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34040118f0 con 0x7f3400009140 2023-10-25T00:17:38.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.181+0000 7f33feffd640 1 -- 172.21.15.107:0/483330380 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3404011ca0 con 0x7f3400009140 2023-10-25T00:17:38.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.181+0000 7f33feffd640 1 -- 172.21.15.107:0/483330380 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f340401c460 con 0x7f3400009140 2023-10-25T00:17:38.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.185+0000 7f340f103640 1 -- 172.21.15.107:0/483330380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34000057a0 con 0x7f3400009140 2023-10-25T00:17:38.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.185+0000 7f33feffd640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f33ec065f30 0x7f33ec0683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:38.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.185+0000 7f33feffd640 1 -- 172.21.15.107:0/483330380 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 10143+0+0 (secure 0 0 0) 0x7f3404084700 con 0x7f3400009140 2023-10-25T00:17:38.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.185+0000 7f340d900640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f33ec065f30 0x7f33ec0683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:38.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.185+0000 7f340d900640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f33ec065f30 0x7f33ec0683f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f33f80ad000 tx=0x7f33f80a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:38.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.193+0000 7f33feffd640 1 -- 172.21.15.107:0/483330380 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3404050d30 con 0x7f3400009140 2023-10-25T00:17:38.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.789+0000 7f340f103640 1 -- 172.21.15.107:0/483330380 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34000c58a0 con 0x7f3400009140 2023-10-25T00:17:38.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.789+0000 7f33feffd640 1 -- 172.21.15.107:0/483330380 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+5770 (secure 0 0 0) 0x7f3404054350 con 0x7f3400009140 2023-10-25T00:17:38.793 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:38.793 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:15:25.707726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5699":{"gid":5699,"name":"b","rank":0,"incarnation":185,"state":"up:active","state_seq":144,"addr":"172.21.15.110:6834/985135365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":985135365},{"type":"v1","addr":"172.21.15.110:6834","nonce":985135365}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:38.793 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 195 2023-10-25T00:17:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.793+0000 7f33fcef9640 1 -- 172.21.15.107:0/483330380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f33ec065f30 msgr2=0x7f33ec0683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:38.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.797+0000 7f33fcef9640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f33ec065f30 0x7f33ec0683f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f33f80ad000 tx=0x7f33f80a6000 comp rx=0 tx=0).stop 2023-10-25T00:17:38.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.797+0000 7f33fcef9640 1 -- 172.21.15.107:0/483330380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3400009140 msgr2=0x7f340004b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:38.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.797+0000 7f33fcef9640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3400009140 0x7f340004b590 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f340400c410 tx=0x7f340400c8d0 comp rx=0 tx=0).stop 2023-10-25T00:17:38.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.797+0000 7f33fcef9640 1 -- 172.21.15.107:0/483330380 shutdown_connections 2023-10-25T00:17:38.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.797+0000 7f33fcef9640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f34000d47b0 0x7f34000c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:38.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.797+0000 7f33fcef9640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f33ec065f30 0x7f33ec0683f0 secure :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f33f80ad000 tx=0x7f33f80a6000 comp rx=0 tx=0).stop 2023-10-25T00:17:38.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.797+0000 7f33fcef9640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34000472b0 0x7f34000d0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:38.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.797+0000 7f33fcef9640 1 --2- 172.21.15.107:0/483330380 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3400009140 0x7f340004b590 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:38.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.797+0000 7f33fcef9640 1 -- 172.21.15.107:0/483330380 >> 172.21.15.107:0/483330380 conn(0x7f3400007c10 msgr2=0x7f34000d8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:38.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.801+0000 7f33fcef9640 1 -- 172.21.15.107:0/483330380 shutdown_connections 2023-10-25T00:17:38.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:38.801+0000 7f33fcef9640 1 -- 172.21.15.107:0/483330380 wait complete. 2023-10-25T00:17:38.944 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:17:40.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:41.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.273+0000 7fcfa3f5a640 1 Processor -- start 2023-10-25T00:17:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.277+0000 7fcfa3f5a640 1 -- start start 2023-10-25T00:17:41.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.277+0000 7fcfa3f5a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcf9c0c24a0 0x7fcf9c154840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.277+0000 7fcfa3f5a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf9c0a9450 0x7fcf9c0c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.277+0000 7fcfa3f5a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c154d80 con 0x7fcf9c151ff0 2023-10-25T00:17:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.277+0000 7fcfa3f5a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf9c06b630 con 0x7fcf9c0c24a0 2023-10-25T00:17:41.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.277+0000 7fcfa3f5a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0aac40 con 0x7fcf9c0a9450 2023-10-25T00:17:41.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.281+0000 7fcfa14f5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcf9c0c24a0 0x7fcf9c154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:41.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.281+0000 7fcfa14f5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcf9c0c24a0 0x7fcf9c154840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:60978/0 (socket says 172.21.15.107:60978) 2023-10-25T00:17:41.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.281+0000 7fcfa14f5640 1 -- 172.21.15.107:0/399201981 learned_addr learned my addr 172.21.15.107:0/399201981 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:41.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.281+0000 7fcfa14f5640 1 -- 172.21.15.107:0/399201981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf9c0a9450 msgr2=0x7fcf9c0c1860 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-25T00:17:41.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.281+0000 7fcfa14f5640 1 --2- 172.21.15.107:0/399201981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf9c0a9450 0x7fcf9c0c1860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:41.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.281+0000 7fcfa14f5640 1 -- 172.21.15.107:0/399201981 >> v1:172.21.15.110:6789/0 conn(0x7fcf9c151ff0 legacy=0x7fcf9c158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:41.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.277+0000 7fcfa1cf6640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcf9c151ff0 0x7fcf9c158760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:45560/0 (socket says 172.21.15.107:45560) 2023-10-25T00:17:41.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.281+0000 7fcfa14f5640 1 -- 172.21.15.107:0/399201981 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c0ab450 con 0x7fcf9c0c24a0 2023-10-25T00:17:41.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.285+0000 7fcfa14f5640 1 --2- 172.21.15.107:0/399201981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf9c0c24a0 0x7fcf9c154840 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009e50 tx=0x7fcf9401c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=368c546793c217ab server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.285+0000 7fcfa0cf4640 1 -- 172.21.15.107:0/399201981 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf940a5550 con 0x7fcf9c0c24a0 2023-10-25T00:17:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.285+0000 7fcfa0cf4640 1 -- 172.21.15.107:0/399201981 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf940a56b0 con 0x7fcf9c0c24a0 2023-10-25T00:17:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.285+0000 7fcfa0cf4640 1 -- 172.21.15.107:0/399201981 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf940a5a40 con 0x7fcf9c0c24a0 2023-10-25T00:17:41.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.285+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/399201981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf9c0c24a0 msgr2=0x7fcf9c154840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:41.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.285+0000 7fcfa3f5a640 1 --2- 172.21.15.107:0/399201981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf9c0c24a0 0x7fcf9c154840 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009e50 tx=0x7fcf9401c8d0 comp rx=0 tx=0).stop 2023-10-25T00:17:41.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.289+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/399201981 shutdown_connections 2023-10-25T00:17:41.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.289+0000 7fcfa3f5a640 1 --2- 172.21.15.107:0/399201981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf9c0a9450 0x7fcf9c0c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:41.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.289+0000 7fcfa3f5a640 1 --2- 172.21.15.107:0/399201981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf9c0c24a0 0x7fcf9c154840 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:41.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.289+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/399201981 >> 172.21.15.107:0/399201981 conn(0x7fcf9c0c2950 msgr2=0x7fcf9c0c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:41.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.289+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/399201981 shutdown_connections 2023-10-25T00:17:41.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.289+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/399201981 wait complete. 2023-10-25T00:17:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.289+0000 7fcfa3f5a640 1 Processor -- start 2023-10-25T00:17:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.289+0000 7fcfa3f5a640 1 -- start start 2023-10-25T00:17:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.289+0000 7fcfa3f5a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf9c0a9450 0x7fcf9c1425c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.289+0000 7fcfa3f5a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf9c0c24a0 0x7fcf9c14c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:41.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.289+0000 7fcfa3f5a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf9c151ff0 0x7fcf9c153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:41.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa3f5a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c3ea0 con 0x7fcf9c0c24a0 2023-10-25T00:17:41.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa3f5a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c1535b0 con 0x7fcf9c0a9450 2023-10-25T00:17:41.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa3f5a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c153fe0 con 0x7fcf9c151ff0 2023-10-25T00:17:41.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa24f7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf9c151ff0 0x7fcf9c153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:41.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa24f7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf9c151ff0 0x7fcf9c153070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:43810/0 (socket says 172.21.15.107:43810) 2023-10-25T00:17:41.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa24f7640 1 -- 172.21.15.107:0/4215918192 learned_addr learned my addr 172.21.15.107:0/4215918192 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:41.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa1cf6640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf9c0a9450 0x7fcf9c1425c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:41.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa14f5640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf9c0c24a0 0x7fcf9c14c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:41.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa24f7640 1 -- 172.21.15.107:0/4215918192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf9c0a9450 msgr2=0x7fcf9c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:41.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa24f7640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf9c0a9450 0x7fcf9c1425c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:41.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa24f7640 1 -- 172.21.15.107:0/4215918192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf9c0c24a0 msgr2=0x7fcf9c14c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:41.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa24f7640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf9c0c24a0 0x7fcf9c14c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:41.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa24f7640 1 -- 172.21.15.107:0/4215918192 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf94009b30 con 0x7fcf9c151ff0 2023-10-25T00:17:41.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa24f7640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf9c151ff0 0x7fcf9c153070 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800b470 tx=0x7fcf9800b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:41.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcf927fc640 1 -- 172.21.15.107:0/4215918192 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98011830 con 0x7fcf9c151ff0 2023-10-25T00:17:41.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcf927fc640 1 -- 172.21.15.107:0/4215918192 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf980119b0 con 0x7fcf9c151ff0 2023-10-25T00:17:41.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcf927fc640 1 -- 172.21.15.107:0/4215918192 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98011d60 con 0x7fcf9c151ff0 2023-10-25T00:17:41.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/4215918192 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9c1f0290 con 0x7fcf9c151ff0 2023-10-25T00:17:41.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.293+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/4215918192 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9c153d00 con 0x7fcf9c151ff0 2023-10-25T00:17:41.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.297+0000 7fcf927fc640 1 -- 172.21.15.107:0/4215918192 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fcf9801a400 con 0x7fcf9c151ff0 2023-10-25T00:17:41.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.297+0000 7fcf927fc640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcf80065f30 0x7fcf800683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:41.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.297+0000 7fcf927fc640 1 -- 172.21.15.107:0/4215918192 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7fcf980849e0 con 0x7fcf9c151ff0 2023-10-25T00:17:41.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.297+0000 7fcfa1cf6640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcf80065f30 0x7fcf800683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:41.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.297+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/4215918192 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9c0746e0 con 0x7fcf9c151ff0 2023-10-25T00:17:41.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.297+0000 7fcfa1cf6640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcf80065f30 0x7fcf800683f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c007240 tx=0x7fcf8c007e70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:41.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.305+0000 7fcf927fc640 1 -- 172.21.15.107:0/4215918192 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fcf98083020 con 0x7fcf9c151ff0 2023-10-25T00:17:41.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.893+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/4215918192 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf9c0aa590 con 0x7fcf9c151ff0 2023-10-25T00:17:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.893+0000 7fcf927fc640 1 -- 172.21.15.107:0/4215918192 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+5013 (secure 0 0 0) 0x7fcf98015070 con 0x7fcf9c151ff0 2023-10-25T00:17:41.898 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:41.898 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:41.353353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:41.898 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 197 2023-10-25T00:17:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.901+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/4215918192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcf80065f30 msgr2=0x7fcf800683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.901+0000 7fcfa3f5a640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcf80065f30 0x7fcf800683f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c007240 tx=0x7fcf8c007e70 comp rx=0 tx=0).stop 2023-10-25T00:17:41.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.905+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/4215918192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf9c151ff0 msgr2=0x7fcf9c153070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:41.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.905+0000 7fcfa3f5a640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf9c151ff0 0x7fcf9c153070 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800b470 tx=0x7fcf9800b930 comp rx=0 tx=0).stop 2023-10-25T00:17:41.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.905+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/4215918192 shutdown_connections 2023-10-25T00:17:41.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.905+0000 7fcfa3f5a640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fcf80065f30 0x7fcf800683f0 secure :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c007240 tx=0x7fcf8c007e70 comp rx=0 tx=0).stop 2023-10-25T00:17:41.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.905+0000 7fcfa3f5a640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcf9c151ff0 0x7fcf9c153070 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:41.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.905+0000 7fcfa3f5a640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf9c0c24a0 0x7fcf9c14c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:41.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.905+0000 7fcfa3f5a640 1 --2- 172.21.15.107:0/4215918192 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcf9c0a9450 0x7fcf9c1425c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:41.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.905+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/4215918192 >> 172.21.15.107:0/4215918192 conn(0x7fcf9c0c2950 msgr2=0x7fcf9c0bf280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:41.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.905+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/4215918192 shutdown_connections 2023-10-25T00:17:41.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:41.905+0000 7fcfa3f5a640 1 -- 172.21.15.107:0/4215918192 wait complete. 2023-10-25T00:17:41.955 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2023-10-25T00:17:41.955 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:17:41.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:42.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.261+0000 7f4e01cae640 1 Processor -- start 2023-10-25T00:17:42.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.261+0000 7f4e01cae640 1 -- start start 2023-10-25T00:17:42.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.261+0000 7f4e01cae640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc0c3fe0 0x7f4dfc0a9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:42.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.261+0000 7f4e01cae640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4dfc0c43c0 0x7f4dfc071f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:42.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.261+0000 7f4e01cae640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4dfc0c5670 0x7f4dfc0c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:42.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.261+0000 7f4e01cae640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dfc0a56f0 con 0x7f4dfc0c43c0 2023-10-25T00:17:42.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.261+0000 7f4e01cae640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dfc0c5af0 con 0x7f4dfc0c5670 2023-10-25T00:17:42.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.261+0000 7f4e01cae640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc0c0ed0 con 0x7f4dfc0c3fe0 2023-10-25T00:17:42.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.261+0000 7f4dfaffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4dfc0c43c0 0x7f4dfc071f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:42.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.261+0000 7f4dfaffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4dfc0c43c0 0x7f4dfc071f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46920/0 (socket says 172.21.15.107:46920) 2023-10-25T00:17:42.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.261+0000 7f4dfaffd640 1 -- 172.21.15.107:0/4237562828 learned_addr learned my addr 172.21.15.107:0/4237562828 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.265+0000 7f4dfaffd640 1 -- 172.21.15.107:0/4237562828 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc0c3fe0 msgr2=0x7f4dfc0a9040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.265+0000 7f4dfaffd640 1 --2- 172.21.15.107:0/4237562828 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc0c3fe0 0x7f4dfc0a9040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.265+0000 7f4dfaffd640 1 -- 172.21.15.107:0/4237562828 >> v2:172.21.15.110:3300/0 conn(0x7f4dfc0c5670 msgr2=0x7f4dfc0c4b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.265+0000 7f4dfaffd640 1 --2- 172.21.15.107:0/4237562828 >> v2:172.21.15.110:3300/0 conn(0x7f4dfc0c5670 0x7f4dfc0c4b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.265+0000 7f4dfaffd640 1 -- 172.21.15.107:0/4237562828 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc0c1120 con 0x7f4dfc0c43c0 2023-10-25T00:17:42.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.269+0000 7f4dfaffd640 1 --2- 172.21.15.107:0/4237562828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0c43c0 0x7f4dfc071f70 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0c6d60 tx=0x7f4df001d880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582437d7830ccbcd server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:42.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.277+0000 7f4dfa7fc640 1 -- 172.21.15.107:0/4237562828 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df00a68f0 con 0x7f4dfc0c43c0 2023-10-25T00:17:42.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.277+0000 7f4dfa7fc640 1 -- 172.21.15.107:0/4237562828 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df00a6a50 con 0x7f4dfc0c43c0 2023-10-25T00:17:42.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.277+0000 7f4dfa7fc640 1 -- 172.21.15.107:0/4237562828 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df00a6de0 con 0x7f4dfc0c43c0 2023-10-25T00:17:42.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.281+0000 7f4e01cae640 1 -- 172.21.15.107:0/4237562828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0c43c0 msgr2=0x7f4dfc071f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:42.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.281+0000 7f4e01cae640 1 --2- 172.21.15.107:0/4237562828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0c43c0 0x7f4dfc071f70 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0c6d60 tx=0x7f4df001d880 comp rx=0 tx=0).stop 2023-10-25T00:17:42.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.281+0000 7f4e01cae640 1 -- 172.21.15.107:0/4237562828 shutdown_connections 2023-10-25T00:17:42.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.281+0000 7f4e01cae640 1 --2- 172.21.15.107:0/4237562828 >> v2:172.21.15.110:3300/0 conn(0x7f4dfc0c5670 0x7f4dfc0c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:42.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.281+0000 7f4e01cae640 1 --2- 172.21.15.107:0/4237562828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0c43c0 0x7f4dfc071f70 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:42.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.281+0000 7f4e01cae640 1 --2- 172.21.15.107:0/4237562828 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc0c3fe0 0x7f4dfc0a9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:42.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.281+0000 7f4e01cae640 1 -- 172.21.15.107:0/4237562828 >> 172.21.15.107:0/4237562828 conn(0x7f4dfc0c5c60 msgr2=0x7f4dfc0c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:42.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.289+0000 7f4e01cae640 1 -- 172.21.15.107:0/4237562828 shutdown_connections 2023-10-25T00:17:42.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4e01cae640 1 -- 172.21.15.107:0/4237562828 wait complete. 2023-10-25T00:17:42.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4e01cae640 1 Processor -- start 2023-10-25T00:17:42.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4e01cae640 1 -- start start 2023-10-25T00:17:42.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4e01cae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0c3fe0 0x7f4dfc155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:42.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4e01cae640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc0c43c0 0x7f4dfc06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:42.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4e01cae640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc0c5670 0x7f4dfc070760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:42.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4e01cae640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc20f040 con 0x7f4dfc0c3fe0 2023-10-25T00:17:42.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4e01cae640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc071700 con 0x7f4dfc0c5670 2023-10-25T00:17:42.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4e01cae640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc20ed60 con 0x7f4dfc0c43c0 2023-10-25T00:17:42.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0c3fe0 0x7f4dfc155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:42.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0c3fe0 0x7f4dfc155ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46930/0 (socket says 172.21.15.107:46930) 2023-10-25T00:17:42.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfb7fe640 1 -- 172.21.15.107:0/802341966 learned_addr learned my addr 172.21.15.107:0/802341966 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:42.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfbfff640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc0c5670 0x7f4dfc070760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:42.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfaffd640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc0c43c0 0x7f4dfc06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:42.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfb7fe640 1 -- 172.21.15.107:0/802341966 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc0c43c0 msgr2=0x7f4dfc06eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:42.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfb7fe640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc0c43c0 0x7f4dfc06eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:42.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfb7fe640 1 -- 172.21.15.107:0/802341966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc0c5670 msgr2=0x7f4dfc070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:42.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfb7fe640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc0c5670 0x7f4dfc070760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:42.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfb7fe640 1 -- 172.21.15.107:0/802341966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0009d60 con 0x7f4dfc0c3fe0 2023-10-25T00:17:42.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfbfff640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc0c5670 0x7f4dfc070760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:17:42.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4dfb7fe640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0c3fe0 0x7f4dfc155ae0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f4df400ba50 tx=0x7f4df400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:42.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4de3fff640 1 -- 172.21.15.107:0/802341966 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df400cbe0 con 0x7f4dfc0c3fe0 2023-10-25T00:17:42.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4de3fff640 1 -- 172.21.15.107:0/802341966 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df400cd80 con 0x7f4dfc0c3fe0 2023-10-25T00:17:42.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4e01cae640 1 -- 172.21.15.107:0/802341966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dfc0719a0 con 0x7f4dfc0c3fe0 2023-10-25T00:17:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.293+0000 7f4e01cae640 1 -- 172.21.15.107:0/802341966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dfc155220 con 0x7f4dfc0c3fe0 2023-10-25T00:17:42.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.297+0000 7f4de3fff640 1 -- 172.21.15.107:0/802341966 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4014680 con 0x7f4dfc0c3fe0 2023-10-25T00:17:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.297+0000 7f4de3fff640 1 -- 172.21.15.107:0/802341966 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4df40147e0 con 0x7f4dfc0c3fe0 2023-10-25T00:17:42.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.297+0000 7f4de3fff640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4de8065e60 0x7f4de8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.297+0000 7f4dfaffd640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4de8065e60 0x7f4de8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.297+0000 7f4de3fff640 1 -- 172.21.15.107:0/802341966 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7f4df4084880 con 0x7f4dfc0c3fe0 2023-10-25T00:17:42.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.297+0000 7f4dfaffd640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4de8065e60 0x7f4de8068320 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4df00b0000 tx=0x7f4df00a8070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:42.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.301+0000 7f4e01cae640 1 -- 172.21.15.107:0/802341966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dfc071be0 con 0x7f4dfc0c3fe0 2023-10-25T00:17:42.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:42.309+0000 7f4de3fff640 1 -- 172.21.15.107:0/802341966 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4dfc071be0 con 0x7f4dfc0c3fe0 2023-10-25T00:17:43.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.125+0000 7f4e01cae640 1 -- 172.21.15.107:0/802341966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dfc071be0 con 0x7f4dfc0c3fe0 2023-10-25T00:17:43.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.125+0000 7f4de3fff640 1 -- 172.21.15.107:0/802341966 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+5010 (secure 0 0 0) 0x7f4dfc071be0 con 0x7f4dfc0c3fe0 2023-10-25T00:17:43.130 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:43.130 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:42.361341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:rejoin","state_seq":5,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:43.130 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 198 2023-10-25T00:17:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 -- 172.21.15.107:0/802341966 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4de8065e60 msgr2=0x7f4de8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4de8065e60 0x7f4de8068320 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4df00b0000 tx=0x7f4df00a8070 comp rx=0 tx=0).stop 2023-10-25T00:17:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 -- 172.21.15.107:0/802341966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0c3fe0 msgr2=0x7f4dfc155ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:43.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0c3fe0 0x7f4dfc155ae0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f4df400ba50 tx=0x7f4df400bf10 comp rx=0 tx=0).stop 2023-10-25T00:17:43.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 -- 172.21.15.107:0/802341966 shutdown_connections 2023-10-25T00:17:43.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dfc0c5670 0x7f4dfc070760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:43.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4de8065e60 0x7f4de8068320 secure :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4df00b0000 tx=0x7f4df00a8070 comp rx=0 tx=0).stop 2023-10-25T00:17:43.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dfc0c43c0 0x7f4dfc06eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 --2- 172.21.15.107:0/802341966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4dfc0c3fe0 0x7f4dfc155ae0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 -- 172.21.15.107:0/802341966 >> 172.21.15.107:0/802341966 conn(0x7f4dfc0c5c60 msgr2=0x7f4dfc1e2950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 -- 172.21.15.107:0/802341966 shutdown_connections 2023-10-25T00:17:43.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:43.141+0000 7f4de1ffb640 1 -- 172.21.15.107:0/802341966 wait complete. 2023-10-25T00:17:43.233 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 198, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 194, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8418": { "addr": "172.21.15.107:6837/418589140", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 418589140, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 418589140, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8418, "incarnation": 191, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 32 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:17:28.068403+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8418 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 198, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7213": { "addr": "172.21.15.107:6835/2970408860", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2970408860, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 2970408860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7213, "incarnation": 196, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:17:42.361341+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7213 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2437148", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2437148, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2437148, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 190, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6443, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:17:43.528 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:17:43.525+0000 7f16c2c5f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:17:45.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:45.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.925+0000 7fc1230e3640 1 Processor -- start 2023-10-25T00:17:45.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.925+0000 7fc1230e3640 1 -- start start 2023-10-25T00:17:45.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.925+0000 7fc1230e3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c08baf0 0x7fc11c0c0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:45.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.925+0000 7fc1230e3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc11c0c4fb0 0x7fc11c165360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.925+0000 7fc1230e3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc11c0c2f20 con 0x7fc11c0c2430 2023-10-25T00:17:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.925+0000 7fc1230e3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc11c06c990 con 0x7fc11c0c4fb0 2023-10-25T00:17:45.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.925+0000 7fc1230e3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c072580 con 0x7fc11c08baf0 2023-10-25T00:17:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.929+0000 7fc1220e1640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc11c0c2430 0x7fc11c0c2810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:34666/0 (socket says 172.21.15.107:34666) 2023-10-25T00:17:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.929+0000 7fc1220e1640 1 -- 172.21.15.107:0/185942543 learned_addr learned my addr 172.21.15.107:0/185942543 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:17:45.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.929+0000 7fc1228e2640 1 --2- 172.21.15.107:0/185942543 >> v2:172.21.15.107:3300/0 conn(0x7fc11c0c4fb0 0x7fc11c165360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.929+0000 7fc1228e2640 1 -- 172.21.15.107:0/185942543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c08baf0 msgr2=0x7fc11c0c0890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-25T00:17:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.929+0000 7fc1228e2640 1 --2- 172.21.15.107:0/185942543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c08baf0 0x7fc11c0c0890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.929+0000 7fc1228e2640 1 -- 172.21.15.107:0/185942543 >> v1:172.21.15.110:6789/0 conn(0x7fc11c0c2430 legacy=0x7fc11c0c2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:17:45.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.929+0000 7fc1228e2640 1 -- 172.21.15.107:0/185942543 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c072310 con 0x7fc11c0c4fb0 2023-10-25T00:17:45.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.937+0000 7fc1228e2640 1 --2- 172.21.15.107:0/185942543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0c4fb0 0x7fc11c165360 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fc10c008aa0 tx=0x7fc10c020ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c90006dd1d9beee server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:45.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.949+0000 7fc1210df640 1 -- 172.21.15.107:0/185942543 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c0a9c80 con 0x7fc11c0c4fb0 2023-10-25T00:17:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.949+0000 7fc1210df640 1 -- 172.21.15.107:0/185942543 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10c0a9de0 con 0x7fc11c0c4fb0 2023-10-25T00:17:45.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.949+0000 7fc1230e3640 1 -- 172.21.15.107:0/185942543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0c4fb0 msgr2=0x7fc11c165360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:45.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.949+0000 7fc1230e3640 1 --2- 172.21.15.107:0/185942543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0c4fb0 0x7fc11c165360 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fc10c008aa0 tx=0x7fc10c020ce0 comp rx=0 tx=0).stop 2023-10-25T00:17:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.957+0000 7fc1230e3640 1 -- 172.21.15.107:0/185942543 shutdown_connections 2023-10-25T00:17:45.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.957+0000 7fc1230e3640 1 --2- 172.21.15.107:0/185942543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0c4fb0 0x7fc11c165360 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.957+0000 7fc1230e3640 1 --2- 172.21.15.107:0/185942543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c08baf0 0x7fc11c0c0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.957+0000 7fc1230e3640 1 -- 172.21.15.107:0/185942543 >> 172.21.15.107:0/185942543 conn(0x7fc11c0a77e0 msgr2=0x7fc11c0a7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:45.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.957+0000 7fc1230e3640 1 -- 172.21.15.107:0/185942543 shutdown_connections 2023-10-25T00:17:45.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.957+0000 7fc1230e3640 1 -- 172.21.15.107:0/185942543 wait complete. 2023-10-25T00:17:45.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1230e3640 1 Processor -- start 2023-10-25T00:17:45.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1230e3640 1 -- start start 2023-10-25T00:17:45.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1230e3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c08baf0 0x7fc11c08a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:45.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1230e3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0c2430 0x7fc11c0a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:45.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1230e3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c0c4fb0 0x7fc11c086d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:45.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1230e3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c089850 con 0x7fc11c0c2430 2023-10-25T00:17:45.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1230e3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c06bd80 con 0x7fc11c0c4fb0 2023-10-25T00:17:45.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1230e3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c08ac50 con 0x7fc11c08baf0 2023-10-25T00:17:45.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1220e1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c08baf0 0x7fc11c08a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:45.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1228e2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c0c4fb0 0x7fc11c086d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:45.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1220e1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c08baf0 0x7fc11c08a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:59254/0 (socket says 172.21.15.107:59254) 2023-10-25T00:17:45.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.961+0000 7fc1220e1640 1 -- 172.21.15.107:0/671799289 learned_addr learned my addr 172.21.15.107:0/671799289 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:45.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.965+0000 7fc1220e1640 1 -- 172.21.15.107:0/671799289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c0c4fb0 msgr2=0x7fc11c086d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:45.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.965+0000 7fc1220e1640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c0c4fb0 0x7fc11c086d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:45.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.965+0000 7fc1220e1640 1 -- 172.21.15.107:0/671799289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0c2430 msgr2=0x7fc11c0a5c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:17:45.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.965+0000 7fc1220e1640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0c2430 0x7fc11c0a5c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:45.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.965+0000 7fc1220e1640 1 -- 172.21.15.107:0/671799289 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c0084a0 con 0x7fc11c08baf0 2023-10-25T00:17:45.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.969+0000 7fc1220e1640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c08baf0 0x7fc11c08a710 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc11800b430 tx=0x7fc11800b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:45.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.973+0000 7fc112ffd640 1 -- 172.21.15.107:0/671799289 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc118011830 con 0x7fc11c08baf0 2023-10-25T00:17:45.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.973+0000 7fc112ffd640 1 -- 172.21.15.107:0/671799289 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc118011990 con 0x7fc11c08baf0 2023-10-25T00:17:45.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.973+0000 7fc112ffd640 1 -- 172.21.15.107:0/671799289 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc118011d20 con 0x7fc11c08baf0 2023-10-25T00:17:45.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.973+0000 7fc1230e3640 1 -- 172.21.15.107:0/671799289 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11c088f20 con 0x7fc11c08baf0 2023-10-25T00:17:45.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.973+0000 7fc1230e3640 1 -- 172.21.15.107:0/671799289 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11c0894e0 con 0x7fc11c08baf0 2023-10-25T00:17:45.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.973+0000 7fc1230e3640 1 -- 172.21.15.107:0/671799289 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc104005260 con 0x7fc11c08baf0 2023-10-25T00:17:45.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.977+0000 7fc112ffd640 1 -- 172.21.15.107:0/671799289 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc11801a400 con 0x7fc11c08baf0 2023-10-25T00:17:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.977+0000 7fc112ffd640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc100065e60 0x7fc100068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:45.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.977+0000 7fc112ffd640 1 -- 172.21.15.107:0/671799289 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7fc118084ff0 con 0x7fc11c08baf0 2023-10-25T00:17:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.981+0000 7fc1218e0640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc100065e60 0x7fc100068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.981+0000 7fc1218e0640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc100065e60 0x7fc100068320 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc11c06f5c0 tx=0x7fc114009250 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:45.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:45.989+0000 7fc112ffd640 1 -- 172.21.15.107:0/671799289 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc1180550e0 con 0x7fc11c08baf0 2023-10-25T00:17:47.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.133+0000 7fc1230e3640 1 -- 172.21.15.107:0/671799289 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1040057e0 con 0x7fc11c08baf0 2023-10-25T00:17:47.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.141+0000 7fc112ffd640 1 -- 172.21.15.107:0/671799289 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5010 (secure 0 0 0) 0x7fc118054c70 con 0x7fc11c08baf0 2023-10-25T00:17:47.149 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:47.150 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:47.150 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 199 2023-10-25T00:17:47.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.169+0000 7fc1230e3640 1 -- 172.21.15.107:0/671799289 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc100065e60 msgr2=0x7fc100068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.169+0000 7fc1230e3640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc100065e60 0x7fc100068320 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc11c06f5c0 tx=0x7fc114009250 comp rx=0 tx=0).stop 2023-10-25T00:17:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.169+0000 7fc1230e3640 1 -- 172.21.15.107:0/671799289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c08baf0 msgr2=0x7fc11c08a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.169+0000 7fc1230e3640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c08baf0 0x7fc11c08a710 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc11800b430 tx=0x7fc11800b8f0 comp rx=0 tx=0).stop 2023-10-25T00:17:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.173+0000 7fc1230e3640 1 -- 172.21.15.107:0/671799289 shutdown_connections 2023-10-25T00:17:47.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.173+0000 7fc1230e3640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc11c0c4fb0 0x7fc11c086d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:47.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.173+0000 7fc1230e3640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc11c0c2430 0x7fc11c0a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:47.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.173+0000 7fc1230e3640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc100065e60 0x7fc100068320 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:47.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.173+0000 7fc1230e3640 1 --2- 172.21.15.107:0/671799289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc11c08baf0 0x7fc11c08a710 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:47.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.173+0000 7fc1230e3640 1 -- 172.21.15.107:0/671799289 >> 172.21.15.107:0/671799289 conn(0x7fc11c0a77e0 msgr2=0x7fc11c0c2810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:47.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.173+0000 7fc1230e3640 1 -- 172.21.15.107:0/671799289 shutdown_connections 2023-10-25T00:17:47.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:47.173+0000 7fc1230e3640 1 -- 172.21.15.107:0/671799289 wait complete. 2023-10-25T00:17:47.345 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5699 2023-10-25T00:17:47.345 INFO:tasks.mds_thrash.fs.[b]:waiting for 2 secs before reviving mds.b 2023-10-25T00:17:49.346 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2023-10-25T00:17:49.347 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-25T00:17:49.347 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-25T00:17:49.352 INFO:tasks.ceph.mds.b:Started 2023-10-25T00:17:49.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:49.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.824+0000 7f898009e640 1 Processor -- start 2023-10-25T00:17:49.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.824+0000 7f898009e640 1 -- start start 2023-10-25T00:17:49.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.824+0000 7f898009e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8970047600 0x7f897000b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:49.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.824+0000 7f898009e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89700dfe30 0x7f897003e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:49.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.824+0000 7f898009e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89700dc470 con 0x7f89700dc090 2023-10-25T00:17:49.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.824+0000 7f898009e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89700e02e0 con 0x7f8970047600 2023-10-25T00:17:49.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.824+0000 7f898009e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8970046d40 con 0x7f89700dfe30 2023-10-25T00:17:49.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.828+0000 7f897e63b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89700dfe30 0x7f897003e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:49.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.828+0000 7f897e63b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89700dfe30 0x7f897003e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:59266/0 (socket says 172.21.15.107:59266) 2023-10-25T00:17:49.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.828+0000 7f897e63b640 1 -- 172.21.15.107:0/3507350914 learned_addr learned my addr 172.21.15.107:0/3507350914 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:49.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.828+0000 7f897e63b640 1 -- 172.21.15.107:0/3507350914 >> v2:172.21.15.110:3300/0 conn(0x7f8970047600 msgr2=0x7f897000b8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:49.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.828+0000 7f897e63b640 1 --2- 172.21.15.107:0/3507350914 >> v2:172.21.15.110:3300/0 conn(0x7f8970047600 0x7f897000b8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:49.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.828+0000 7f897e63b640 1 -- 172.21.15.107:0/3507350914 >> v1:172.21.15.107:6789/0 conn(0x7f89700dc090 legacy=0x7f89700df260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:17:49.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.828+0000 7f897e63b640 1 -- 172.21.15.107:0/3507350914 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89700dedd0 con 0x7f89700dfe30 2023-10-25T00:17:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.832+0000 7f897e63b640 1 --2- 172.21.15.107:0/3507350914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89700dfe30 0x7f897003e930 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f89680098a0 tx=0x7f896801c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f40e2298c7ce17b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.844+0000 7f897ce38640 1 -- 172.21.15.107:0/3507350914 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89680ab600 con 0x7f89700dfe30 2023-10-25T00:17:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.844+0000 7f897ce38640 1 -- 172.21.15.107:0/3507350914 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89680a7050 con 0x7f89700dfe30 2023-10-25T00:17:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.844+0000 7f898009e640 1 -- 172.21.15.107:0/3507350914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89700dfe30 msgr2=0x7f897003e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.844+0000 7f898009e640 1 --2- 172.21.15.107:0/3507350914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89700dfe30 0x7f897003e930 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f89680098a0 tx=0x7f896801c500 comp rx=0 tx=0).stop 2023-10-25T00:17:49.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.852+0000 7f898009e640 1 -- 172.21.15.107:0/3507350914 shutdown_connections 2023-10-25T00:17:49.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.852+0000 7f898009e640 1 --2- 172.21.15.107:0/3507350914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89700dfe30 0x7f897003e930 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:49.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.852+0000 7f898009e640 1 --2- 172.21.15.107:0/3507350914 >> v2:172.21.15.110:3300/0 conn(0x7f8970047600 0x7f897000b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:49.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.852+0000 7f898009e640 1 -- 172.21.15.107:0/3507350914 >> 172.21.15.107:0/3507350914 conn(0x7f8970006020 msgr2=0x7f8970006430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:49.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.864+0000 7f898009e640 1 -- 172.21.15.107:0/3507350914 shutdown_connections 2023-10-25T00:17:49.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.864+0000 7f898009e640 1 -- 172.21.15.107:0/3507350914 wait complete. 2023-10-25T00:17:49.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f898009e640 1 Processor -- start 2023-10-25T00:17:49.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f898009e640 1 -- start start 2023-10-25T00:17:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f898009e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8970047600 0x7f8970040040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f898009e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89700dc090 0x7f8970044c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f898009e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700dfe30 0x7f8970194200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f898009e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8970044490 con 0x7f89700dfe30 2023-10-25T00:17:49.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f898009e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8970044770 con 0x7f89700dc090 2023-10-25T00:17:49.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f898009e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8970045900 con 0x7f8970047600 2023-10-25T00:17:49.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f897d639640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89700dc090 0x7f8970044c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:49.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f897de3a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8970047600 0x7f8970040040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:49.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f897d639640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89700dc090 0x7f8970044c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:38680/0 (socket says 172.21.15.107:38680) 2023-10-25T00:17:49.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.868+0000 7f897d639640 1 -- 172.21.15.107:0/1988448051 learned_addr learned my addr 172.21.15.107:0/1988448051 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:49.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.872+0000 7f897d639640 1 -- 172.21.15.107:0/1988448051 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8970047600 msgr2=0x7f8970040040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:49.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.872+0000 7f897d639640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8970047600 0x7f8970040040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:49.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.872+0000 7f897d639640 1 -- 172.21.15.107:0/1988448051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700dfe30 msgr2=0x7f8970194200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:17:49.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.872+0000 7f897d639640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700dfe30 0x7f8970194200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:49.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.872+0000 7f897d639640 1 -- 172.21.15.107:0/1988448051 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89680094d0 con 0x7f89700dc090 2023-10-25T00:17:49.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.872+0000 7f897d639640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89700dc090 0x7f8970044c00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f897802c850 tx=0x7f8978069810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:49.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.872+0000 7f896e7fc640 1 -- 172.21.15.107:0/1988448051 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89780723c0 con 0x7f89700dc090 2023-10-25T00:17:49.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.872+0000 7f898009e640 1 -- 172.21.15.107:0/1988448051 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8970045be0 con 0x7f89700dc090 2023-10-25T00:17:49.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.872+0000 7f898009e640 1 -- 172.21.15.107:0/1988448051 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8970005a80 con 0x7f89700dc090 2023-10-25T00:17:49.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.872+0000 7f896e7fc640 1 -- 172.21.15.107:0/1988448051 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8978072520 con 0x7f89700dc090 2023-10-25T00:17:49.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.872+0000 7f896e7fc640 1 -- 172.21.15.107:0/1988448051 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8978072990 con 0x7f89700dc090 2023-10-25T00:17:49.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.876+0000 7f896e7fc640 1 -- 172.21.15.107:0/1988448051 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8978093400 con 0x7f89700dc090 2023-10-25T00:17:49.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.876+0000 7f896e7fc640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f895c065f60 0x7f895c068420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:49.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.876+0000 7f897de3a640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f895c065f60 0x7f895c068420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:49.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.876+0000 7f896e7fc640 1 -- 172.21.15.107:0/1988448051 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7f89780fe800 con 0x7f89700dc090 2023-10-25T00:17:49.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.880+0000 7f898009e640 1 -- 172.21.15.107:0/1988448051 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8970005450 con 0x7f89700dc090 2023-10-25T00:17:49.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.880+0000 7f897de3a640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f895c065f60 0x7f895c068420 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8974009980 tx=0x7f89740090e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:49.896+0000 7f896e7fc640 1 -- 172.21.15.107:0/1988448051 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f89780cae60 con 0x7f89700dc090 2023-10-25T00:17:50.737 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.736+0000 7f898009e640 1 -- 172.21.15.107:0/1988448051 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89700448b0 con 0x7f89700dc090 2023-10-25T00:17:50.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.740+0000 7f896e7fc640 1 -- 172.21.15.107:0/1988448051 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5010 (secure 0 0 0) 0x7f89780ce480 con 0x7f89700dc090 2023-10-25T00:17:50.742 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:50.742 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:50.743 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 199 2023-10-25T00:17:50.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.760+0000 7f898009e640 1 -- 172.21.15.107:0/1988448051 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f895c065f60 msgr2=0x7f895c068420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.760+0000 7f898009e640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f895c065f60 0x7f895c068420 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8974009980 tx=0x7f89740090e0 comp rx=0 tx=0).stop 2023-10-25T00:17:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.760+0000 7f898009e640 1 -- 172.21.15.107:0/1988448051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89700dc090 msgr2=0x7f8970044c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.760+0000 7f898009e640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89700dc090 0x7f8970044c00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f897802c850 tx=0x7f8978069810 comp rx=0 tx=0).stop 2023-10-25T00:17:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.760+0000 7f898009e640 1 -- 172.21.15.107:0/1988448051 shutdown_connections 2023-10-25T00:17:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.760+0000 7f898009e640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89700dfe30 0x7f8970194200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.760+0000 7f898009e640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89700dc090 0x7f8970044c00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.760+0000 7f898009e640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f895c065f60 0x7f895c068420 secure :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8974009980 tx=0x7f89740090e0 comp rx=0 tx=0).stop 2023-10-25T00:17:50.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.760+0000 7f898009e640 1 --2- 172.21.15.107:0/1988448051 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8970047600 0x7f8970040040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.760+0000 7f898009e640 1 -- 172.21.15.107:0/1988448051 >> 172.21.15.107:0/1988448051 conn(0x7f8970006020 msgr2=0x7f897000a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.764+0000 7f898009e640 1 -- 172.21.15.107:0/1988448051 shutdown_connections 2023-10-25T00:17:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:50.764+0000 7f898009e640 1 -- 172.21.15.107:0/1988448051 wait complete. 2023-10-25T00:17:50.839 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-25T00:17:51.595 INFO:tasks.ceph.mds.b.smithi110.stdout:starting mds.b at 2023-10-25T00:17:52.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:17:53.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.304+0000 7fd2c0882640 1 Processor -- start 2023-10-25T00:17:53.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.304+0000 7fd2c0882640 1 -- start start 2023-10-25T00:17:53.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.304+0000 7fd2c0882640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c2430 0x7fd2bc0c0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:53.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.304+0000 7fd2c0882640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc0c2f20 con 0x7fd2bc0c4fb0 2023-10-25T00:17:53.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.304+0000 7fd2c0882640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc06c990 con 0x7fd2bc08baf0 2023-10-25T00:17:53.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.304+0000 7fd2c0882640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc072580 con 0x7fd2bc0c2430 2023-10-25T00:17:53.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.308+0000 7fd2baffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd2bc08baf0 0x7fd2bc0c2810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:48354/0 (socket says 172.21.15.107:48354) 2023-10-25T00:17:53.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.308+0000 7fd2baffd640 1 -- 172.21.15.107:0/1523677454 learned_addr learned my addr 172.21.15.107:0/1523677454 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:17:53.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.308+0000 7fd2bb7fe640 1 --2- 172.21.15.107:0/1523677454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c2430 0x7fd2bc0c0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.312+0000 7fd2bb7fe640 1 -- 172.21.15.107:0/1523677454 >> v1:172.21.15.110:6789/0 conn(0x7fd2bc08baf0 legacy=0x7fd2bc0c2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.312+0000 7fd2bb7fe640 1 -- 172.21.15.107:0/1523677454 >> v1:172.21.15.107:6789/0 conn(0x7fd2bc0c4fb0 legacy=0x7fd2bc165360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.312+0000 7fd2bb7fe640 1 -- 172.21.15.107:0/1523677454 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc072310 con 0x7fd2bc0c2430 2023-10-25T00:17:53.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.320+0000 7fd2bb7fe640 1 --2- 172.21.15.107:0/1523677454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c2430 0x7fd2bc0c0890 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00a990 tx=0x7fd2ac01e3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eba2004502fb26c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:53.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.320+0000 7fd2ba7fc640 1 -- 172.21.15.107:0/1523677454 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0a69c0 con 0x7fd2bc0c2430 2023-10-25T00:17:53.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.320+0000 7fd2ba7fc640 1 -- 172.21.15.107:0/1523677454 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac0a6b20 con 0x7fd2bc0c2430 2023-10-25T00:17:53.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.320+0000 7fd2c0882640 1 -- 172.21.15.107:0/1523677454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c2430 msgr2=0x7fd2bc0c0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:53.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.320+0000 7fd2c0882640 1 --2- 172.21.15.107:0/1523677454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c2430 0x7fd2bc0c0890 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00a990 tx=0x7fd2ac01e3c0 comp rx=0 tx=0).stop 2023-10-25T00:17:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.324+0000 7fd2c0882640 1 -- 172.21.15.107:0/1523677454 shutdown_connections 2023-10-25T00:17:53.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.324+0000 7fd2c0882640 1 --2- 172.21.15.107:0/1523677454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c2430 0x7fd2bc0c0890 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00a990 tx=0x7fd2ac01e3c0 comp rx=0 tx=0).stop 2023-10-25T00:17:53.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.324+0000 7fd2c0882640 1 -- 172.21.15.107:0/1523677454 >> 172.21.15.107:0/1523677454 conn(0x7fd2bc0a77e0 msgr2=0x7fd2bc0a7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:53.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.324+0000 7fd2c0882640 1 -- 172.21.15.107:0/1523677454 shutdown_connections 2023-10-25T00:17:53.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.328+0000 7fd2c0882640 1 -- 172.21.15.107:0/1523677454 wait complete. 2023-10-25T00:17:53.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.328+0000 7fd2c0882640 1 Processor -- start 2023-10-25T00:17:53.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.328+0000 7fd2c0882640 1 -- start start 2023-10-25T00:17:53.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.328+0000 7fd2c0882640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2bc08baf0 0x7fd2bc1ff690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:53.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.328+0000 7fd2c0882640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c4fb0 0x7fd2bc1ffbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:53.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.328+0000 7fd2c0882640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc084190 0x7fd2bc0ba460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:53.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.328+0000 7fd2c0882640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc084c50 con 0x7fd2bc084190 2023-10-25T00:17:53.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.328+0000 7fd2c0882640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0b9a50 con 0x7fd2bc08baf0 2023-10-25T00:17:53.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.328+0000 7fd2c0882640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0b9fd0 con 0x7fd2bc0c4fb0 2023-10-25T00:17:53.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.328+0000 7fd2bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc084190 0x7fd2bc0ba460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:53.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.332+0000 7fd2bbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc084190 0x7fd2bc0ba460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36444/0 (socket says 172.21.15.107:36444) 2023-10-25T00:17:53.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.332+0000 7fd2bbfff640 1 -- 172.21.15.107:0/85399585 learned_addr learned my addr 172.21.15.107:0/85399585 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:17:53.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.332+0000 7fd2baffd640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c4fb0 0x7fd2bc1ffbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:53.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.332+0000 7fd2bb7fe640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2bc08baf0 0x7fd2bc1ff690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:53.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.332+0000 7fd2baffd640 1 -- 172.21.15.107:0/85399585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2bc08baf0 msgr2=0x7fd2bc1ff690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:53.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.332+0000 7fd2baffd640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2bc08baf0 0x7fd2bc1ff690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:53.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.332+0000 7fd2baffd640 1 -- 172.21.15.107:0/85399585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc084190 msgr2=0x7fd2bc0ba460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:53.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.336+0000 7fd2baffd640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc084190 0x7fd2bc0ba460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:53.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.336+0000 7fd2baffd640 1 -- 172.21.15.107:0/85399585 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac00a5c0 con 0x7fd2bc0c4fb0 2023-10-25T00:17:53.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.336+0000 7fd2bbfff640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc084190 0x7fd2bc0ba460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:17:53.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.340+0000 7fd2baffd640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c4fb0 0x7fd2bc1ffbd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc08bf50 tx=0x7fd2b400a860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:53.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.344+0000 7fd2a3fff640 1 -- 172.21.15.107:0/85399585 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4014430 con 0x7fd2bc0c4fb0 2023-10-25T00:17:53.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.344+0000 7fd2c0882640 1 -- 172.21.15.107:0/85399585 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc084690 con 0x7fd2bc0c4fb0 2023-10-25T00:17:53.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.344+0000 7fd2c0882640 1 -- 172.21.15.107:0/85399585 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc0b9c60 con 0x7fd2bc0c4fb0 2023-10-25T00:17:53.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.344+0000 7fd2a3fff640 1 -- 172.21.15.107:0/85399585 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4014590 con 0x7fd2bc0c4fb0 2023-10-25T00:17:53.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.344+0000 7fd2a3fff640 1 -- 172.21.15.107:0/85399585 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4014980 con 0x7fd2bc0c4fb0 2023-10-25T00:17:53.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.348+0000 7fd2c0882640 1 -- 172.21.15.107:0/85399585 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29c005260 con 0x7fd2bc0c4fb0 2023-10-25T00:17:53.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.356+0000 7fd2a3fff640 1 -- 172.21.15.107:0/85399585 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd2b400e3e0 con 0x7fd2bc0c4fb0 2023-10-25T00:17:53.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.360+0000 7fd2a3fff640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2a8065f30 0x7fd2a80683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:17:53.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.360+0000 7fd2a3fff640 1 -- 172.21.15.107:0/85399585 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7fd2b4085080 con 0x7fd2bc0c4fb0 2023-10-25T00:17:53.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.364+0000 7fd2bb7fe640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2a8065f30 0x7fd2a80683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:17:53.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.364+0000 7fd2bb7fe640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2a8065f30 0x7fd2a80683f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0b0000 tx=0x7fd2ac0a8070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:17:53.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:53.364+0000 7fd2a3fff640 1 -- 172.21.15.107:0/85399585 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd2b40516e0 con 0x7fd2bc0c4fb0 2023-10-25T00:17:54.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.480+0000 7fd2c0882640 1 -- 172.21.15.107:0/85399585 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd29c005a90 con 0x7fd2bc0c4fb0 2023-10-25T00:17:54.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.488+0000 7fd2a3fff640 1 -- 172.21.15.107:0/85399585 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5771 (secure 0 0 0) 0x7fd2b4054d00 con 0x7fd2bc0c4fb0 2023-10-25T00:17:54.498 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:17:54.498 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:17:54.498 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 200 2023-10-25T00:17:54.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 -- 172.21.15.107:0/85399585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2a8065f30 msgr2=0x7fd2a80683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:54.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2a8065f30 0x7fd2a80683f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0b0000 tx=0x7fd2ac0a8070 comp rx=0 tx=0).stop 2023-10-25T00:17:54.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 -- 172.21.15.107:0/85399585 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c4fb0 msgr2=0x7fd2bc1ffbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:17:54.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c4fb0 0x7fd2bc1ffbd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc08bf50 tx=0x7fd2b400a860 comp rx=0 tx=0).stop 2023-10-25T00:17:54.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 -- 172.21.15.107:0/85399585 shutdown_connections 2023-10-25T00:17:54.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2bc084190 0x7fd2bc0ba460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:54.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd2a8065f30 0x7fd2a80683f0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:54.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd2bc0c4fb0 0x7fd2bc1ffbd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:54.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 --2- 172.21.15.107:0/85399585 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2bc08baf0 0x7fd2bc1ff690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:17:54.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 -- 172.21.15.107:0/85399585 >> 172.21.15.107:0/85399585 conn(0x7fd2bc0a77e0 msgr2=0x7fd2bc0873d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:17:54.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 -- 172.21.15.107:0/85399585 shutdown_connections 2023-10-25T00:17:54.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:17:54.500+0000 7fd2c0882640 1 -- 172.21.15.107:0/85399585 wait complete. 2023-10-25T00:17:54.561 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2023-10-25T00:17:54.561 INFO:tasks.mds_thrash.fs.[b]:waiting for 53 secs before thrashing 2023-10-25T00:18:32.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:33.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af627d640 1 Processor -- start 2023-10-25T00:18:33.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af627d640 1 -- start start 2023-10-25T00:18:33.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af627d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5af00c2430 0x7f5af00c0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:33.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af627d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5af008baf0 0x7f5af00c2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:33.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af627d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af006c990 con 0x7f5af00c4fb0 2023-10-25T00:18:33.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af627d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af0072580 con 0x7f5af00c2430 2023-10-25T00:18:33.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af627d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af0072310 con 0x7f5af008baf0 2023-10-25T00:18:33.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af527b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5af00c2430 0x7f5af00c0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:33.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af527b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5af00c2430 0x7f5af00c0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45984/0 (socket says 172.21.15.107:45984) 2023-10-25T00:18:33.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af527b640 1 -- 172.21.15.107:0/2382018844 learned_addr learned my addr 172.21.15.107:0/2382018844 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:33.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5ae7fff640 1 -- 172.21.15.107:0/2382018844 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109533539 0 0) 0x7f5af006c990 con 0x7f5af00c4fb0 2023-10-25T00:18:33.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5ae7fff640 1 -- 172.21.15.107:0/2382018844 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5adc003510 con 0x7f5af00c4fb0 2023-10-25T00:18:33.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af527b640 1 -- 172.21.15.107:0/2382018844 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5af008baf0 msgr2=0x7f5af00c2810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-25T00:18:33.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af527b640 1 --2- 172.21.15.107:0/2382018844 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5af008baf0 0x7f5af00c2810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:33.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af527b640 1 -- 172.21.15.107:0/2382018844 >> v1:172.21.15.110:6789/0 conn(0x7f5af00c4fb0 legacy=0x7f5af006af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:33.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.568+0000 7f5af527b640 1 -- 172.21.15.107:0/2382018844 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af00b9580 con 0x7f5af00c2430 2023-10-25T00:18:33.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.576+0000 7f5af527b640 1 --2- 172.21.15.107:0/2382018844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5af00c2430 0x7f5af00c0890 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800b250 tx=0x7f5ae801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b802e404aba2712 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:33.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.576+0000 7f5ae7fff640 1 -- 172.21.15.107:0/2382018844 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae80a5d90 con 0x7f5af00c2430 2023-10-25T00:18:33.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.576+0000 7f5ae7fff640 1 -- 172.21.15.107:0/2382018844 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae800a0d0 con 0x7f5af00c2430 2023-10-25T00:18:33.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.576+0000 7f5af627d640 1 -- 172.21.15.107:0/2382018844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5af00c2430 msgr2=0x7f5af00c0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:33.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.576+0000 7f5af627d640 1 --2- 172.21.15.107:0/2382018844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5af00c2430 0x7f5af00c0890 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800b250 tx=0x7f5ae801ccc0 comp rx=0 tx=0).stop 2023-10-25T00:18:33.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.580+0000 7f5af627d640 1 -- 172.21.15.107:0/2382018844 shutdown_connections 2023-10-25T00:18:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.580+0000 7f5af627d640 1 --2- 172.21.15.107:0/2382018844 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5af008baf0 0x7f5af00c2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.580+0000 7f5af627d640 1 --2- 172.21.15.107:0/2382018844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5af00c2430 0x7f5af00c0890 secure :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800b250 tx=0x7f5ae801ccc0 comp rx=0 tx=0).stop 2023-10-25T00:18:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.580+0000 7f5af627d640 1 -- 172.21.15.107:0/2382018844 >> 172.21.15.107:0/2382018844 conn(0x7f5af00a77e0 msgr2=0x7f5af00a7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.588+0000 7f5af627d640 1 -- 172.21.15.107:0/2382018844 shutdown_connections 2023-10-25T00:18:33.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.592+0000 7f5af627d640 1 -- 172.21.15.107:0/2382018844 wait complete. 2023-10-25T00:18:33.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.592+0000 7f5af627d640 1 Processor -- start 2023-10-25T00:18:33.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.604+0000 7f5af627d640 1 -- start start 2023-10-25T00:18:33.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.604+0000 7f5af627d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5af008baf0 0x7f5af00b8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:33.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.604+0000 7f5af627d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af00c4fb0 0x7f5af01fce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:33.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.604+0000 7f5af627d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5af008a830 0x7f5af00ba510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:33.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.604+0000 7f5af627d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af00b99f0 con 0x7f5af008a830 2023-10-25T00:18:33.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.604+0000 7f5af627d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af01fd7d0 con 0x7f5af00c4fb0 2023-10-25T00:18:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.604+0000 7f5af627d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af0089d50 con 0x7f5af008baf0 2023-10-25T00:18:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.608+0000 7f5af4a7a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af00c4fb0 0x7f5af01fce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.608+0000 7f5af5a7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5af008a830 0x7f5af00ba510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.608+0000 7f5af5a7c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5af008a830 0x7f5af00ba510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45996/0 (socket says 172.21.15.107:45996) 2023-10-25T00:18:33.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.608+0000 7f5af5a7c640 1 -- 172.21.15.107:0/11973169 learned_addr learned my addr 172.21.15.107:0/11973169 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.608+0000 7f5af5a7c640 1 -- 172.21.15.107:0/11973169 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5af008baf0 msgr2=0x7f5af00b8d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-25T00:18:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.608+0000 7f5af5a7c640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5af008baf0 0x7f5af00b8d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.608+0000 7f5af5a7c640 1 -- 172.21.15.107:0/11973169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af00c4fb0 msgr2=0x7f5af01fce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.608+0000 7f5af5a7c640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af00c4fb0 0x7f5af01fce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:33.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.608+0000 7f5af5a7c640 1 -- 172.21.15.107:0/11973169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae8008b20 con 0x7f5af008a830 2023-10-25T00:18:33.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.608+0000 7f5af5a7c640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5af008a830 0x7f5af00ba510 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000c3f0 tx=0x7f5ae000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:33.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.612+0000 7f5af4a7a640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af00c4fb0 0x7f5af01fce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:18:33.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.624+0000 7f5ae5ffb640 1 -- 172.21.15.107:0/11973169 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae00052d0 con 0x7f5af008a830 2023-10-25T00:18:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.624+0000 7f5af627d640 1 -- 172.21.15.107:0/11973169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af008a030 con 0x7f5af008a830 2023-10-25T00:18:33.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.624+0000 7f5af627d640 1 -- 172.21.15.107:0/11973169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af01fda70 con 0x7f5af008a830 2023-10-25T00:18:33.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.632+0000 7f5ae5ffb640 1 -- 172.21.15.107:0/11973169 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae0005430 con 0x7f5af008a830 2023-10-25T00:18:33.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.632+0000 7f5ae5ffb640 1 -- 172.21.15.107:0/11973169 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0003800 con 0x7f5af008a830 2023-10-25T00:18:33.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.632+0000 7f5ae5ffb640 1 -- 172.21.15.107:0/11973169 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5ae00039a0 con 0x7f5af008a830 2023-10-25T00:18:33.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.632+0000 7f5ae5ffb640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ad4065fe0 0x7f5ad40684a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:33.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.636+0000 7f5af527b640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ad4065fe0 0x7f5ad40684a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.636+0000 7f5af627d640 1 -- 172.21.15.107:0/11973169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af0089e90 con 0x7f5af008a830 2023-10-25T00:18:33.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.640+0000 7f5af527b640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ad4065fe0 0x7f5ad40684a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80ae000 tx=0x7f5ae8004ac0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:33.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.644+0000 7f5ae5ffb640 1 -- 172.21.15.107:0/11973169 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7f5ae0083dd0 con 0x7f5af008a830 2023-10-25T00:18:33.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:33.644+0000 7f5ae5ffb640 1 -- 172.21.15.107:0/11973169 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5af0089e90 con 0x7f5af008a830 2023-10-25T00:18:34.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.532+0000 7f5af627d640 1 -- 172.21.15.107:0/11973169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5af0089e90 con 0x7f5af008a830 2023-10-25T00:18:34.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.532+0000 7f5ae5ffb640 1 -- 172.21.15.107:0/11973169 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5771 (secure 0 0 0) 0x7f5af0089e90 con 0x7f5af008a830 2023-10-25T00:18:34.534 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:34.534 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:34.534 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 200 2023-10-25T00:18:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 -- 172.21.15.107:0/11973169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ad4065fe0 msgr2=0x7f5ad40684a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ad4065fe0 0x7f5ad40684a0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5ae80ae000 tx=0x7f5ae8004ac0 comp rx=0 tx=0).stop 2023-10-25T00:18:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 -- 172.21.15.107:0/11973169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5af008a830 msgr2=0x7f5af00ba510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:34.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5af008a830 0x7f5af00ba510 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f5ae000c3f0 tx=0x7f5ae000c8b0 comp rx=0 tx=0).stop 2023-10-25T00:18:34.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 -- 172.21.15.107:0/11973169 shutdown_connections 2023-10-25T00:18:34.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5af008a830 0x7f5af00ba510 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:34.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af00c4fb0 0x7f5af01fce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:34.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ad4065fe0 0x7f5ad40684a0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:34.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 --2- 172.21.15.107:0/11973169 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5af008baf0 0x7f5af00b8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:34.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 -- 172.21.15.107:0/11973169 >> 172.21.15.107:0/11973169 conn(0x7f5af00a77e0 msgr2=0x7f5af00c6050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:34.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 -- 172.21.15.107:0/11973169 shutdown_connections 2023-10-25T00:18:34.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.536+0000 7f5af627d640 1 -- 172.21.15.107:0/11973169 wait complete. 2023-10-25T00:18:34.595 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2023-10-25T00:18:34.595 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:18:34.596 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:18:34.608 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:18:34.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:34.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.884+0000 7f2a9972b640 1 Processor -- start 2023-10-25T00:18:34.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.884+0000 7f2a9972b640 1 -- start start 2023-10-25T00:18:34.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a9972b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2a940c49d0 0x7f2a94159550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:34.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a9972b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2a9415c740 0x7f2a9415cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:34.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a9972b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a9415c2b0 con 0x7f2a94156300 2023-10-25T00:18:34.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a9972b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a940c9990 con 0x7f2a9415c740 2023-10-25T00:18:34.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a9972b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a94159a90 con 0x7f2a940c49d0 2023-10-25T00:18:34.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a98f2a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2a9415c740 0x7f2a9415cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:34.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a98f2a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f2a9415c740 0x7f2a9415cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46012/0 (socket says 172.21.15.107:46012) 2023-10-25T00:18:34.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a98f2a640 1 -- 172.21.15.107:0/2537019097 learned_addr learned my addr 172.21.15.107:0/2537019097 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:34.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a98f2a640 1 -- 172.21.15.107:0/2537019097 >> v2:172.21.15.110:3300/0 conn(0x7f2a940c49d0 msgr2=0x7f2a94159550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:18:34.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a98f2a640 1 --2- 172.21.15.107:0/2537019097 >> v2:172.21.15.110:3300/0 conn(0x7f2a940c49d0 0x7f2a94159550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:34.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a98f2a640 1 -- 172.21.15.107:0/2537019097 >> v1:172.21.15.110:6789/0 conn(0x7f2a94156300 legacy=0x7f2a940abc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:18:34.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.888+0000 7f2a98f2a640 1 -- 172.21.15.107:0/2537019097 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a940c8e30 con 0x7f2a9415c740 2023-10-25T00:18:34.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.892+0000 7f2a98f2a640 1 --2- 172.21.15.107:0/2537019097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a9415c740 0x7f2a9415cbf0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009870 tx=0x7f2a8801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e97c150dfffba5 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:34.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.896+0000 7f2a92ffd640 1 -- 172.21.15.107:0/2537019097 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a880ab600 con 0x7f2a9415c740 2023-10-25T00:18:34.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.896+0000 7f2a92ffd640 1 -- 172.21.15.107:0/2537019097 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a880a7050 con 0x7f2a9415c740 2023-10-25T00:18:34.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.896+0000 7f2a92ffd640 1 -- 172.21.15.107:0/2537019097 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a880ab980 con 0x7f2a9415c740 2023-10-25T00:18:34.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.896+0000 7f2a9972b640 1 -- 172.21.15.107:0/2537019097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a9415c740 msgr2=0x7f2a9415cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:34.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.896+0000 7f2a9972b640 1 --2- 172.21.15.107:0/2537019097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a9415c740 0x7f2a9415cbf0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009870 tx=0x7f2a8801c450 comp rx=0 tx=0).stop 2023-10-25T00:18:34.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.896+0000 7f2a9972b640 1 -- 172.21.15.107:0/2537019097 shutdown_connections 2023-10-25T00:18:34.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.896+0000 7f2a9972b640 1 --2- 172.21.15.107:0/2537019097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a9415c740 0x7f2a9415cbf0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:34.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.896+0000 7f2a9972b640 1 --2- 172.21.15.107:0/2537019097 >> v2:172.21.15.110:3300/0 conn(0x7f2a940c49d0 0x7f2a94159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:34.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.896+0000 7f2a9972b640 1 -- 172.21.15.107:0/2537019097 >> 172.21.15.107:0/2537019097 conn(0x7f2a940ca220 msgr2=0x7f2a940ca630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:34.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.904+0000 7f2a9972b640 1 -- 172.21.15.107:0/2537019097 shutdown_connections 2023-10-25T00:18:34.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.904+0000 7f2a9972b640 1 -- 172.21.15.107:0/2537019097 wait complete. 2023-10-25T00:18:34.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.904+0000 7f2a9972b640 1 Processor -- start 2023-10-25T00:18:34.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.904+0000 7f2a9972b640 1 -- start start 2023-10-25T00:18:34.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a9972b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a940c49d0 0x7f2a940c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:34.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a9972b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a94156300 0x7f2a9414c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:34.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a9972b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a9415c740 0x7f2a9414cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:34.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a9972b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a9414d300 con 0x7f2a94156300 2023-10-25T00:18:34.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a9972b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a94155060 con 0x7f2a9415c740 2023-10-25T00:18:34.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a9972b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a941f5340 con 0x7f2a940c49d0 2023-10-25T00:18:34.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a94156300 0x7f2a9414c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:34.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a937fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a94156300 0x7f2a9414c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46024/0 (socket says 172.21.15.107:46024) 2023-10-25T00:18:34.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a937fe640 1 -- 172.21.15.107:0/380962160 learned_addr learned my addr 172.21.15.107:0/380962160 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:34.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a93fff640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a940c49d0 0x7f2a940c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:34.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a98f2a640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a9415c740 0x7f2a9414cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:34.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a937fe640 1 -- 172.21.15.107:0/380962160 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a940c49d0 msgr2=0x7f2a940c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:34.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a937fe640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a940c49d0 0x7f2a940c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:34.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a937fe640 1 -- 172.21.15.107:0/380962160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a9415c740 msgr2=0x7f2a9414cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:34.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a937fe640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a9415c740 0x7f2a9414cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:34.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a937fe640 1 -- 172.21.15.107:0/380962160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a880094d0 con 0x7f2a94156300 2023-10-25T00:18:34.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.908+0000 7f2a937fe640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a94156300 0x7f2a9414c880 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400b450 tx=0x7f2a8400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:34.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.920+0000 7f2a90ff9640 1 -- 172.21.15.107:0/380962160 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84011890 con 0x7f2a94156300 2023-10-25T00:18:34.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.920+0000 7f2a9972b640 1 -- 172.21.15.107:0/380962160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a940c15f0 con 0x7f2a94156300 2023-10-25T00:18:34.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.920+0000 7f2a9972b640 1 -- 172.21.15.107:0/380962160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a941f4860 con 0x7f2a94156300 2023-10-25T00:18:34.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.924+0000 7f2a90ff9640 1 -- 172.21.15.107:0/380962160 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a840119f0 con 0x7f2a94156300 2023-10-25T00:18:34.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.924+0000 7f2a90ff9640 1 -- 172.21.15.107:0/380962160 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84011da0 con 0x7f2a94156300 2023-10-25T00:18:34.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.924+0000 7f2a90ff9640 1 -- 172.21.15.107:0/380962160 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2a8401a400 con 0x7f2a94156300 2023-10-25T00:18:34.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.924+0000 7f2a90ff9640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a80065fb0 0x7f2a80068470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:34.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.924+0000 7f2a90ff9640 1 -- 172.21.15.107:0/380962160 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7f2a84084c00 con 0x7f2a94156300 2023-10-25T00:18:34.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.928+0000 7f2a93fff640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a80065fb0 0x7f2a80068470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:34.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.928+0000 7f2a93fff640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a80065fb0 0x7f2a80068470 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00b450 tx=0x7f2a8c00d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:34.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.932+0000 7f2a9972b640 1 -- 172.21.15.107:0/380962160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a74005260 con 0x7f2a94156300 2023-10-25T00:18:34.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:34.944+0000 7f2a90ff9640 1 -- 172.21.15.107:0/380962160 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2a84015020 con 0x7f2a94156300 2023-10-25T00:18:35.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.496+0000 7f2a9972b640 1 -- 172.21.15.107:0/380962160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a74005a90 con 0x7f2a94156300 2023-10-25T00:18:35.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.496+0000 7f2a90ff9640 1 -- 172.21.15.107:0/380962160 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5771 (secure 0 0 0) 0x7f2a84051280 con 0x7f2a94156300 2023-10-25T00:18:35.497 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:35.497 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:35.498 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 200 2023-10-25T00:18:35.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.500+0000 7f2a9972b640 1 -- 172.21.15.107:0/380962160 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a80065fb0 msgr2=0x7f2a80068470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:35.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.500+0000 7f2a9972b640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a80065fb0 0x7f2a80068470 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00b450 tx=0x7f2a8c00d040 comp rx=0 tx=0).stop 2023-10-25T00:18:35.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.500+0000 7f2a9972b640 1 -- 172.21.15.107:0/380962160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a94156300 msgr2=0x7f2a9414c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:35.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.500+0000 7f2a9972b640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a94156300 0x7f2a9414c880 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400b450 tx=0x7f2a8400b910 comp rx=0 tx=0).stop 2023-10-25T00:18:35.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.500+0000 7f2a9972b640 1 -- 172.21.15.107:0/380962160 shutdown_connections 2023-10-25T00:18:35.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.500+0000 7f2a9972b640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2a9415c740 0x7f2a9414cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:35.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.500+0000 7f2a9972b640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a94156300 0x7f2a9414c880 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:35.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.500+0000 7f2a9972b640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2a80065fb0 0x7f2a80068470 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:35.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.500+0000 7f2a9972b640 1 --2- 172.21.15.107:0/380962160 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2a940c49d0 0x7f2a940c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:35.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.500+0000 7f2a9972b640 1 -- 172.21.15.107:0/380962160 >> 172.21.15.107:0/380962160 conn(0x7f2a940ca220 msgr2=0x7f2a940c5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:35.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.504+0000 7f2a9972b640 1 -- 172.21.15.107:0/380962160 shutdown_connections 2023-10-25T00:18:35.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:35.504+0000 7f2a9972b640 1 -- 172.21.15.107:0/380962160 wait complete. 2023-10-25T00:18:35.557 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:18:37.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:38.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.076+0000 7f380b1e7640 1 Processor -- start 2023-10-25T00:18:38.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.076+0000 7f380b1e7640 1 -- start start 2023-10-25T00:18:38.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.076+0000 7f380b1e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3804152060 0x7f38040aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:38.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.076+0000 7f380b1e7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f38040c2990 0x7f38040a9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:38.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.076+0000 7f380b1e7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38040a9aa0 con 0x7f38040aa470 2023-10-25T00:18:38.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.076+0000 7f380b1e7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3804155480 con 0x7f3804152060 2023-10-25T00:18:38.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.076+0000 7f380b1e7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3804155760 con 0x7f38040c2990 2023-10-25T00:18:38.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.080+0000 7f3808f83640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3804152060 0x7f38040aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:38.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.080+0000 7f3809784640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f38040aa470 0x7f38040c1480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52198/0 (socket says 172.21.15.107:52198) 2023-10-25T00:18:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.080+0000 7f3809784640 1 -- 172.21.15.107:0/2440848299 learned_addr learned my addr 172.21.15.107:0/2440848299 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:18:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.080+0000 7f3808f83640 1 -- 172.21.15.107:0/2440848299 >> v2:172.21.15.110:3300/0 conn(0x7f38040c2990 msgr2=0x7f38040a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.080+0000 7f3808f83640 1 --2- 172.21.15.107:0/2440848299 >> v2:172.21.15.110:3300/0 conn(0x7f38040c2990 0x7f38040a9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.080+0000 7f3808f83640 1 -- 172.21.15.107:0/2440848299 >> v1:172.21.15.107:6789/0 conn(0x7f38040aa470 legacy=0x7f38040c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:38.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.080+0000 7f3808f83640 1 -- 172.21.15.107:0/2440848299 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38040a9450 con 0x7f3804152060 2023-10-25T00:18:38.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.080+0000 7f3808f83640 1 --2- 172.21.15.107:0/2440848299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3804152060 0x7f38040aada0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3804155a40 tx=0x7f380001c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f531c3cf5920ea server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:38.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.084+0000 7f37fb7fe640 1 -- 172.21.15.107:0/2440848299 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38000ab600 con 0x7f3804152060 2023-10-25T00:18:38.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.084+0000 7f37fb7fe640 1 -- 172.21.15.107:0/2440848299 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38000a7050 con 0x7f3804152060 2023-10-25T00:18:38.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.084+0000 7f37fb7fe640 1 -- 172.21.15.107:0/2440848299 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38000ab980 con 0x7f3804152060 2023-10-25T00:18:38.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.084+0000 7f380b1e7640 1 -- 172.21.15.107:0/2440848299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3804152060 msgr2=0x7f38040aada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:38.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.084+0000 7f380b1e7640 1 --2- 172.21.15.107:0/2440848299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3804152060 0x7f38040aada0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3804155a40 tx=0x7f380001c310 comp rx=0 tx=0).stop 2023-10-25T00:18:38.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.084+0000 7f380b1e7640 1 -- 172.21.15.107:0/2440848299 shutdown_connections 2023-10-25T00:18:38.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.084+0000 7f380b1e7640 1 --2- 172.21.15.107:0/2440848299 >> v2:172.21.15.110:3300/0 conn(0x7f38040c2990 0x7f38040a9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:38.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.084+0000 7f380b1e7640 1 --2- 172.21.15.107:0/2440848299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3804152060 0x7f38040aada0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:38.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.084+0000 7f380b1e7640 1 -- 172.21.15.107:0/2440848299 >> 172.21.15.107:0/2440848299 conn(0x7f38040c4a90 msgr2=0x7f38040c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:38.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.084+0000 7f380b1e7640 1 -- 172.21.15.107:0/2440848299 shutdown_connections 2023-10-25T00:18:38.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.084+0000 7f380b1e7640 1 -- 172.21.15.107:0/2440848299 wait complete. 2023-10-25T00:18:38.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f380b1e7640 1 Processor -- start 2023-10-25T00:18:38.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f380b1e7640 1 -- start start 2023-10-25T00:18:38.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f380b1e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38040aa470 0x7f38040c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:38.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f380b1e7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38040c2990 0x7f38040c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:38.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f380b1e7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3804152060 0x7f38040c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:38.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f380b1e7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38041ed4f0 con 0x7f38040aa470 2023-10-25T00:18:38.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f380b1e7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38041ed7d0 con 0x7f3804152060 2023-10-25T00:18:38.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f380b1e7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38041edab0 con 0x7f38040c2990 2023-10-25T00:18:38.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f3808f83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38040aa470 0x7f38040c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:38.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f3808f83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38040aa470 0x7f38040c50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46040/0 (socket says 172.21.15.107:46040) 2023-10-25T00:18:38.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f3808f83640 1 -- 172.21.15.107:0/2278510155 learned_addr learned my addr 172.21.15.107:0/2278510155 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:38.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f3809784640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3804152060 0x7f38040c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:38.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f3808f83640 1 -- 172.21.15.107:0/2278510155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38040c2990 msgr2=0x7f38040c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:38.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f3808f83640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38040c2990 0x7f38040c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:38.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f3808f83640 1 -- 172.21.15.107:0/2278510155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3804152060 msgr2=0x7f38040c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:38.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f3808f83640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3804152060 0x7f38040c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:38.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f3808f83640 1 -- 172.21.15.107:0/2278510155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3800009420 con 0x7f38040aa470 2023-10-25T00:18:38.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f3809784640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3804152060 0x7f38040c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:18:38.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f3808f83640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38040aa470 0x7f38040c50d0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f38000ae040 tx=0x7f38000190b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:38.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f37f97fa640 1 -- 172.21.15.107:0/2278510155 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38000b4840 con 0x7f38040aa470 2023-10-25T00:18:38.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f37f97fa640 1 -- 172.21.15.107:0/2278510155 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38000b49c0 con 0x7f38040aa470 2023-10-25T00:18:38.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f380b1e7640 1 -- 172.21.15.107:0/2278510155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38041edd90 con 0x7f38040aa470 2023-10-25T00:18:38.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f37f97fa640 1 -- 172.21.15.107:0/2278510155 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38000b4d70 con 0x7f38040aa470 2023-10-25T00:18:38.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f380b1e7640 1 -- 172.21.15.107:0/2278510155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38040c6030 con 0x7f38040aa470 2023-10-25T00:18:38.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f380b1e7640 1 -- 172.21.15.107:0/2278510155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38040c6ff0 con 0x7f38040aa470 2023-10-25T00:18:38.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.088+0000 7f37f97fa640 1 -- 172.21.15.107:0/2278510155 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f38000b2040 con 0x7f38040aa470 2023-10-25T00:18:38.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.092+0000 7f37f97fa640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37f0065e60 0x7f37f0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:38.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.092+0000 7f37f97fa640 1 -- 172.21.15.107:0/2278510155 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7f380011ed30 con 0x7f38040aa470 2023-10-25T00:18:38.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.092+0000 7f37fbfff640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37f0065e60 0x7f37f0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:38.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.092+0000 7f37fbfff640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37f0065e60 0x7f37f0068320 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f380415f060 tx=0x7f37f4006ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:38.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.096+0000 7f37f97fa640 1 -- 172.21.15.107:0/2278510155 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f38000eb370 con 0x7f38040aa470 2023-10-25T00:18:38.340 INFO:tasks.workunit.client.0.smithi107.stdout:Syncing()...3 sec 2023-10-25T00:18:38.341 INFO:tasks.workunit.client.0.smithi107.stdout:FFSB benchmark finished at: Wed Oct 25 00:18:38 2023 2023-10-25T00:18:38.341 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:18:38.341 INFO:tasks.workunit.client.0.smithi107.stdout:Results: 2023-10-25T00:18:38.341 INFO:tasks.workunit.client.0.smithi107.stdout:Benchmark took 308.35 sec 2023-10-25T00:18:38.341 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:18:38.342 INFO:tasks.workunit.client.0.smithi107.stdout:Total Results 2023-10-25T00:18:38.342 INFO:tasks.workunit.client.0.smithi107.stdout:=============== 2023-10-25T00:18:38.342 INFO:tasks.workunit.client.0.smithi107.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2023-10-25T00:18:38.342 INFO:tasks.workunit.client.0.smithi107.stdout: ======= ============ ========= ======= =========== ========== 2023-10-25T00:18:38.343 INFO:tasks.workunit.client.0.smithi107.stdout: write : 1671680 5421.43 100.000% 100.000% 21.2MB/sec 2023-10-25T00:18:38.343 INFO:tasks.workunit.client.0.smithi107.stdout:- 2023-10-25T00:18:38.343 INFO:tasks.workunit.client.0.smithi107.stdout:5421.43 Transactions per Second 2023-10-25T00:18:38.343 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:18:38.343 INFO:tasks.workunit.client.0.smithi107.stdout:Throughput Results 2023-10-25T00:18:38.344 INFO:tasks.workunit.client.0.smithi107.stdout:=================== 2023-10-25T00:18:38.344 INFO:tasks.workunit.client.0.smithi107.stdout:Write Throughput: 21.2MB/sec 2023-10-25T00:18:38.351 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:18:38.351 INFO:tasks.workunit.client.0.smithi107.stdout:System Call Latency statistics in millisecs 2023-10-25T00:18:38.351 INFO:tasks.workunit.client.0.smithi107.stdout:===== 2023-10-25T00:18:38.351 INFO:tasks.workunit.client.0.smithi107.stdout: Min Avg Max Total Calls 2023-10-25T00:18:38.352 INFO:tasks.workunit.client.0.smithi107.stdout: ======== ======== ======== ============ 2023-10-25T00:18:38.352 INFO:tasks.workunit.client.0.smithi107.stdout:[ open] 0.329000 775.859901 18939.818359 1306 2023-10-25T00:18:38.352 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2023-10-25T00:18:38.352 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-10-25T00:18:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[2] 1000.000000 - 2000.000000 : 2 2023-10-25T00:18:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[3] 500.000000 - 1000.000000 : 11 2023-10-25T00:18:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[4] 200.000000 - 500.000000 : 13 2023-10-25T00:18:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[5] 100.000000 - 200.000000 : 13 2023-10-25T00:18:38.353 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[6] 50.000000 - 100.000000 : 46 2023-10-25T00:18:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[7] 20.000000 - 50.000000 : 346 2023-10-25T00:18:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[8] 10.000000 - 20.000000 : 522 2023-10-25T00:18:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[9] 5.000000 - 10.000000 : 170 2023-10-25T00:18:38.354 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[10] 2.000000 - 5.000000 : 68 2023-10-25T00:18:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[11] 1.000000 - 2.000000 : 35 2023-10-25T00:18:38.355 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[12] 0.500000 - 1.000000 : 13 2023-10-25T00:18:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[13] 0.200000 - 0.500000 : 3 2023-10-25T00:18:38.356 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2023-10-25T00:18:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2023-10-25T00:18:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2023-10-25T00:18:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2023-10-25T00:18:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2023-10-25T00:18:38.357 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:18:38.358 INFO:tasks.workunit.client.0.smithi107.stdout:[ write] 0.082000 5.173492 720.143005 1671680 2023-10-25T00:18:38.358 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2023-10-25T00:18:38.358 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-10-25T00:18:38.358 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2023-10-25T00:18:38.358 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[3] 500.000000 - 1000.000000 : 6 2023-10-25T00:18:38.359 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[4] 200.000000 - 500.000000 : 19 2023-10-25T00:18:38.359 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[5] 100.000000 - 200.000000 : 211 2023-10-25T00:18:38.359 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[6] 50.000000 - 100.000000 : 5608 2023-10-25T00:18:38.359 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[7] 20.000000 - 50.000000 : 72461 2023-10-25T00:18:38.360 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[8] 10.000000 - 20.000000 : 161442 2023-10-25T00:18:38.360 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[9] 5.000000 - 10.000000 : 310149 2023-10-25T00:18:38.360 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[10] 2.000000 - 5.000000 : 404455 2023-10-25T00:18:38.360 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[11] 1.000000 - 2.000000 : 166980 2023-10-25T00:18:38.360 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[12] 0.500000 - 1.000000 : 71549 2023-10-25T00:18:38.361 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[13] 0.200000 - 0.500000 : 229801 2023-10-25T00:18:38.361 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[14] 0.100000 - 0.200000 : 238674 2023-10-25T00:18:38.361 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[15] 0.050000 - 0.100000 : 10325 2023-10-25T00:18:38.361 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2023-10-25T00:18:38.361 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2023-10-25T00:18:38.361 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2023-10-25T00:18:38.362 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:18:38.362 INFO:tasks.workunit.client.0.smithi107.stdout:[ lseek] 0.000000 0.000977 4.086000 1671680 2023-10-25T00:18:38.362 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2023-10-25T00:18:38.363 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-10-25T00:18:38.363 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2023-10-25T00:18:38.364 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2023-10-25T00:18:38.364 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2023-10-25T00:18:38.364 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2023-10-25T00:18:38.364 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2023-10-25T00:18:38.364 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2023-10-25T00:18:38.365 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2023-10-25T00:18:38.365 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2023-10-25T00:18:38.365 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[10] 2.000000 - 5.000000 : 5 2023-10-25T00:18:38.365 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[11] 1.000000 - 2.000000 : 5 2023-10-25T00:18:38.366 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[12] 0.500000 - 1.000000 : 6 2023-10-25T00:18:38.366 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[13] 0.200000 - 0.500000 : 27 2023-10-25T00:18:38.366 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[14] 0.100000 - 0.200000 : 31 2023-10-25T00:18:38.366 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[15] 0.050000 - 0.100000 : 60 2023-10-25T00:18:38.367 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[16] 0.020000 - 0.050000 : 400 2023-10-25T00:18:38.367 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[17] 0.010000 - 0.020000 : 897 2023-10-25T00:18:38.367 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[18] 0.000000 - 0.010000 : 1670249 2023-10-25T00:18:38.367 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:18:38.368 INFO:tasks.workunit.client.0.smithi107.stdout:[ close] 0.019000 2.516590 64.589996 1306 2023-10-25T00:18:38.368 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2023-10-25T00:18:38.368 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-10-25T00:18:38.368 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2023-10-25T00:18:38.368 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2023-10-25T00:18:38.368 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2023-10-25T00:18:38.369 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2023-10-25T00:18:38.369 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[6] 50.000000 - 100.000000 : 2 2023-10-25T00:18:38.369 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[7] 20.000000 - 50.000000 : 35 2023-10-25T00:18:38.369 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[8] 10.000000 - 20.000000 : 44 2023-10-25T00:18:38.369 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[9] 5.000000 - 10.000000 : 83 2023-10-25T00:18:38.370 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[10] 2.000000 - 5.000000 : 168 2023-10-25T00:18:38.370 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[11] 1.000000 - 2.000000 : 169 2023-10-25T00:18:38.370 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[12] 0.500000 - 1.000000 : 78 2023-10-25T00:18:38.370 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[13] 0.200000 - 0.500000 : 63 2023-10-25T00:18:38.370 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[14] 0.100000 - 0.200000 : 145 2023-10-25T00:18:38.371 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[15] 0.050000 - 0.100000 : 134 2023-10-25T00:18:38.371 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[16] 0.020000 - 0.050000 : 382 2023-10-25T00:18:38.371 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[17] 0.010000 - 0.020000 : 3 2023-10-25T00:18:38.371 INFO:tasks.workunit.client.0.smithi107.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2023-10-25T00:18:38.371 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:18:38.371 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:18:38.372 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:18:38.372 INFO:tasks.workunit.client.0.smithi107.stdout: 2023-10-25T00:18:38.372 INFO:tasks.workunit.client.0.smithi107.stdout:0.7% User Time 2023-10-25T00:18:38.372 INFO:tasks.workunit.client.0.smithi107.stdout:52.2% System Time 2023-10-25T00:18:38.372 INFO:tasks.workunit.client.0.smithi107.stdout:52.8% CPU Utilization 2023-10-25T00:18:38.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.372+0000 7f380b1e7640 1 -- 172.21.15.107:0/2278510155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f380406f170 con 0x7f38040aa470 2023-10-25T00:18:38.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.372+0000 7f37f97fa640 1 -- 172.21.15.107:0/2278510155 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5771 (secure 0 0 0) 0x7f38000b8090 con 0x7f38040aa470 2023-10-25T00:18:38.374 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:38.374 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:38.375 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 200 2023-10-25T00:18:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 -- 172.21.15.107:0/2278510155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37f0065e60 msgr2=0x7f37f0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37f0065e60 0x7f37f0068320 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f380415f060 tx=0x7f37f4006ca0 comp rx=0 tx=0).stop 2023-10-25T00:18:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 -- 172.21.15.107:0/2278510155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38040aa470 msgr2=0x7f38040c50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38040aa470 0x7f38040c50d0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f38000ae040 tx=0x7f38000190b0 comp rx=0 tx=0).stop 2023-10-25T00:18:38.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 -- 172.21.15.107:0/2278510155 shutdown_connections 2023-10-25T00:18:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3804152060 0x7f38040c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f37f0065e60 0x7f37f0068320 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38040c2990 0x7f38040c71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 --2- 172.21.15.107:0/2278510155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38040aa470 0x7f38040c50d0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:38.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 -- 172.21.15.107:0/2278510155 >> 172.21.15.107:0/2278510155 conn(0x7f38040c4a90 msgr2=0x7f38040abdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:38.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 -- 172.21.15.107:0/2278510155 shutdown_connections 2023-10-25T00:18:38.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:38.376+0000 7f380b1e7640 1 -- 172.21.15.107:0/2278510155 wait complete. 2023-10-25T00:18:38.399 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:18:40.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:40.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d4a7a2640 1 Processor -- start 2023-10-25T00:18:40.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d4a7a2640 1 -- start start 2023-10-25T00:18:40.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d4a7a2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7d440aa330 0x7f7d4406e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:40.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d4a7a2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7d440a9450 0x7f7d44071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:40.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d4a7a2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d44152d40 con 0x7f7d4413bb00 2023-10-25T00:18:40.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d4a7a2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d44156100 con 0x7f7d440a9450 2023-10-25T00:18:40.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d4a7a2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d441527c0 con 0x7f7d440aa330 2023-10-25T00:18:40.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d48d3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7d440a9450 0x7f7d44071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:40.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d48d3f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7d440a9450 0x7f7d44071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46054/0 (socket says 172.21.15.107:46054) 2023-10-25T00:18:40.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d48d3f640 1 -- 172.21.15.107:0/2919752375 learned_addr learned my addr 172.21.15.107:0/2919752375 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:40.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d437fe640 1 --2- 172.21.15.107:0/2919752375 >> v2:172.21.15.110:3300/0 conn(0x7f7d440aa330 0x7f7d4406e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:40.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d48d3f640 1 -- 172.21.15.107:0/2919752375 >> v2:172.21.15.110:3300/0 conn(0x7f7d440aa330 msgr2=0x7f7d4406e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:40.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d48d3f640 1 --2- 172.21.15.107:0/2919752375 >> v2:172.21.15.110:3300/0 conn(0x7f7d440aa330 0x7f7d4406e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:40.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d48d3f640 1 -- 172.21.15.107:0/2919752375 >> v1:172.21.15.107:6789/0 conn(0x7f7d4413bb00 legacy=0x7f7d441531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:40.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d48d3f640 1 -- 172.21.15.107:0/2919752375 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4406d660 con 0x7f7d440a9450 2023-10-25T00:18:40.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.544+0000 7f7d42ffd640 1 -- 172.21.15.107:0/2919752375 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456419924 0 0) 0x7f7d44152d40 con 0x7f7d4413bb00 2023-10-25T00:18:40.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d48d3f640 1 --2- 172.21.15.107:0/2919752375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d440a9450 0x7f7d44071a80 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009e50 tx=0x7f7d3c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85048485c34c8b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:40.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d42ffd640 1 -- 172.21.15.107:0/2919752375 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c0a5570 con 0x7f7d440a9450 2023-10-25T00:18:40.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d42ffd640 1 -- 172.21.15.107:0/2919752375 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c0a56d0 con 0x7f7d440a9450 2023-10-25T00:18:40.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d42ffd640 1 -- 172.21.15.107:0/2919752375 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c0a5a60 con 0x7f7d440a9450 2023-10-25T00:18:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/2919752375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d440a9450 msgr2=0x7f7d44071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d4a7a2640 1 --2- 172.21.15.107:0/2919752375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d440a9450 0x7f7d44071a80 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009e50 tx=0x7f7d3c01c8d0 comp rx=0 tx=0).stop 2023-10-25T00:18:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/2919752375 shutdown_connections 2023-10-25T00:18:40.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d4a7a2640 1 --2- 172.21.15.107:0/2919752375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d440a9450 0x7f7d44071a80 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:40.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d4a7a2640 1 --2- 172.21.15.107:0/2919752375 >> v2:172.21.15.110:3300/0 conn(0x7f7d440aa330 0x7f7d4406e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:40.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/2919752375 >> 172.21.15.107:0/2919752375 conn(0x7f7d44070f60 msgr2=0x7f7d44071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:40.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/2919752375 shutdown_connections 2023-10-25T00:18:40.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.548+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/2919752375 wait complete. 2023-10-25T00:18:40.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d4a7a2640 1 Processor -- start 2023-10-25T00:18:40.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d4a7a2640 1 -- start start 2023-10-25T00:18:40.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d4a7a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7d440a9450 0x7f7d440c3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:40.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d4a7a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d440aa330 0x7f7d440c44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:40.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d4a7a2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d4413bb00 0x7f7d44132a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:40.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d4a7a2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4412cc20 con 0x7f7d4413bb00 2023-10-25T00:18:40.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d4a7a2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d441500e0 con 0x7f7d440aa330 2023-10-25T00:18:40.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d4a7a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d440c7310 con 0x7f7d440a9450 2023-10-25T00:18:40.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d48d3f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d4413bb00 0x7f7d44132a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:40.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d48d3f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d4413bb00 0x7f7d44132a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46056/0 (socket says 172.21.15.107:46056) 2023-10-25T00:18:40.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d48d3f640 1 -- 172.21.15.107:0/3190162680 learned_addr learned my addr 172.21.15.107:0/3190162680 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:40.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d437fe640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d440aa330 0x7f7d440c44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:40.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d43fff640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7d440a9450 0x7f7d440c3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:40.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d48d3f640 1 -- 172.21.15.107:0/3190162680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7d440a9450 msgr2=0x7f7d440c3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:40.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d48d3f640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7d440a9450 0x7f7d440c3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:40.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d48d3f640 1 -- 172.21.15.107:0/3190162680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d440aa330 msgr2=0x7f7d440c44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:40.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d48d3f640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d440aa330 0x7f7d440c44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:40.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d48d3f640 1 -- 172.21.15.107:0/3190162680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3c009b30 con 0x7f7d4413bb00 2023-10-25T00:18:40.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d43fff640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7d440a9450 0x7f7d440c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:18:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d48d3f640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d4413bb00 0x7f7d44132a90 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009e50 tx=0x7f7d3c019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d40ff9640 1 -- 172.21.15.107:0/3190162680 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c0a5570 con 0x7f7d4413bb00 2023-10-25T00:18:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d40ff9640 1 -- 172.21.15.107:0/3190162680 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c0b8880 con 0x7f7d4413bb00 2023-10-25T00:18:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/3190162680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d440c2b00 con 0x7f7d4413bb00 2023-10-25T00:18:40.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d40ff9640 1 -- 172.21.15.107:0/3190162680 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c0b8c30 con 0x7f7d4413bb00 2023-10-25T00:18:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/3190162680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d44131fb0 con 0x7f7d4413bb00 2023-10-25T00:18:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.552+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/3190162680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d440746e0 con 0x7f7d4413bb00 2023-10-25T00:18:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.556+0000 7f7d40ff9640 1 -- 172.21.15.107:0/3190162680 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7d3c0b6440 con 0x7f7d4413bb00 2023-10-25T00:18:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.556+0000 7f7d40ff9640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7d30065e60 0x7f7d30068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.556+0000 7f7d40ff9640 1 -- 172.21.15.107:0/3190162680 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7f7d3c11ff80 con 0x7f7d4413bb00 2023-10-25T00:18:40.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.556+0000 7f7d43fff640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7d30065e60 0x7f7d30068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:40.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.556+0000 7f7d43fff640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7d30065e60 0x7f7d30068320 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7d38000f70 tx=0x7f7d38009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:40.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.560+0000 7f7d40ff9640 1 -- 172.21.15.107:0/3190162680 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7d3c151040 con 0x7f7d4413bb00 2023-10-25T00:18:40.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.820+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/3190162680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d4406d160 con 0x7f7d4413bb00 2023-10-25T00:18:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.820+0000 7f7d40ff9640 1 -- 172.21.15.107:0/3190162680 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5771 (secure 0 0 0) 0x7f7d3c0bc090 con 0x7f7d4413bb00 2023-10-25T00:18:40.821 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:40.821 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:40.822 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 200 2023-10-25T00:18:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/3190162680 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7d30065e60 msgr2=0x7f7d30068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7d30065e60 0x7f7d30068320 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f7d38000f70 tx=0x7f7d38009040 comp rx=0 tx=0).stop 2023-10-25T00:18:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/3190162680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d4413bb00 msgr2=0x7f7d44132a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d4413bb00 0x7f7d44132a90 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009e50 tx=0x7f7d3c019670 comp rx=0 tx=0).stop 2023-10-25T00:18:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/3190162680 shutdown_connections 2023-10-25T00:18:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d4413bb00 0x7f7d44132a90 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7d440aa330 0x7f7d440c44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:40.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7d30065e60 0x7f7d30068320 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:40.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 --2- 172.21.15.107:0/3190162680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7d440a9450 0x7f7d440c3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:40.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/3190162680 >> 172.21.15.107:0/3190162680 conn(0x7f7d44070f60 msgr2=0x7f7d44150f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:40.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/3190162680 shutdown_connections 2023-10-25T00:18:40.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:40.824+0000 7f7d4a7a2640 1 -- 172.21.15.107:0/3190162680 wait complete. 2023-10-25T00:18:40.862 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:18:42.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:43.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f3715794640 1 Processor -- start 2023-10-25T00:18:43.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f3715794640 1 -- start start 2023-10-25T00:18:43.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f3715794640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f371013dc20 0x7f37100c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:43.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f3715794640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37100720c0 0x7f371006df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:43.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f3715794640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f371006e470 0x7f3710157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:43.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f3715794640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3710154860 con 0x7f371006e470 2023-10-25T00:18:43.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f3715794640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37100c1f10 con 0x7f371013dc20 2023-10-25T00:18:43.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f3715794640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3710154b20 con 0x7f37100720c0 2023-10-25T00:18:43.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f370f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f371006e470 0x7f3710157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:43.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f370f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f371006e470 0x7f3710157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58352/0 (socket says 172.21.15.107:58352) 2023-10-25T00:18:43.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f370f7fe640 1 -- 172.21.15.107:0/3331280512 learned_addr learned my addr 172.21.15.107:0/3331280512 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:43.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f370e7fc640 1 --2- 172.21.15.107:0/3331280512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37100720c0 0x7f371006df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:43.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f370effd640 1 --2- 172.21.15.107:0/3331280512 >> v2:172.21.15.110:3300/0 conn(0x7f371013dc20 0x7f37100c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:43.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f370f7fe640 1 -- 172.21.15.107:0/3331280512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37100720c0 msgr2=0x7f371006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:43.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f370f7fe640 1 --2- 172.21.15.107:0/3331280512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37100720c0 0x7f371006df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f370f7fe640 1 -- 172.21.15.107:0/3331280512 >> v2:172.21.15.110:3300/0 conn(0x7f371013dc20 msgr2=0x7f37100c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:43.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f370f7fe640 1 --2- 172.21.15.107:0/3331280512 >> v2:172.21.15.110:3300/0 conn(0x7f371013dc20 0x7f37100c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.028+0000 7f370f7fe640 1 -- 172.21.15.107:0/3331280512 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3710154e00 con 0x7f371006e470 2023-10-25T00:18:43.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f370f7fe640 1 --2- 172.21.15.107:0/3331280512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f371006e470 0x7f3710157e50 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f37100c8020 tx=0x7f370801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50012a54390090f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:43.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f370dffb640 1 -- 172.21.15.107:0/3331280512 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37080a5530 con 0x7f371006e470 2023-10-25T00:18:43.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f370dffb640 1 -- 172.21.15.107:0/3331280512 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37080a5690 con 0x7f371006e470 2023-10-25T00:18:43.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f370dffb640 1 -- 172.21.15.107:0/3331280512 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37080a5a20 con 0x7f371006e470 2023-10-25T00:18:43.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f3715794640 1 -- 172.21.15.107:0/3331280512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f371006e470 msgr2=0x7f3710157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:43.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f3715794640 1 --2- 172.21.15.107:0/3331280512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f371006e470 0x7f3710157e50 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f37100c8020 tx=0x7f370801c8b0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f3715794640 1 -- 172.21.15.107:0/3331280512 shutdown_connections 2023-10-25T00:18:43.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f3715794640 1 --2- 172.21.15.107:0/3331280512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f371006e470 0x7f3710157e50 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f3715794640 1 --2- 172.21.15.107:0/3331280512 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f37100720c0 0x7f371006df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f3715794640 1 --2- 172.21.15.107:0/3331280512 >> v2:172.21.15.110:3300/0 conn(0x7f371013dc20 0x7f37100c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f3715794640 1 -- 172.21.15.107:0/3331280512 >> 172.21.15.107:0/3331280512 conn(0x7f37100c39a0 msgr2=0x7f37100c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:43.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f3715794640 1 -- 172.21.15.107:0/3331280512 shutdown_connections 2023-10-25T00:18:43.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f3715794640 1 -- 172.21.15.107:0/3331280512 wait complete. 2023-10-25T00:18:43.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f3715794640 1 Processor -- start 2023-10-25T00:18:43.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.032+0000 7f3715794640 1 -- start start 2023-10-25T00:18:43.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f3715794640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f371006e470 0x7f371013c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:43.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f3715794640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37100720c0 0x7f37101ee300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:43.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f3715794640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f371013dc20 0x7f37101ee840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:43.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f3715794640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37101f0f40 con 0x7f371006e470 2023-10-25T00:18:43.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f3715794640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37101f1360 con 0x7f37100720c0 2023-10-25T00:18:43.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f3715794640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37101f05a0 con 0x7f371013dc20 2023-10-25T00:18:43.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f370effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f371006e470 0x7f371013c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:43.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f370effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f371006e470 0x7f371013c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58354/0 (socket says 172.21.15.107:58354) 2023-10-25T00:18:43.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f370effd640 1 -- 172.21.15.107:0/2320537778 learned_addr learned my addr 172.21.15.107:0/2320537778 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:43.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f370f7fe640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f371013dc20 0x7f37101ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:43.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f370effd640 1 -- 172.21.15.107:0/2320537778 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f371013dc20 msgr2=0x7f37101ee840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:43.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f370effd640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f371013dc20 0x7f37101ee840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f370effd640 1 -- 172.21.15.107:0/2320537778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37100720c0 msgr2=0x7f37101ee300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:43.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f370effd640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37100720c0 0x7f37101ee300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f370effd640 1 -- 172.21.15.107:0/2320537778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3708009b30 con 0x7f371006e470 2023-10-25T00:18:43.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f370effd640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f371006e470 0x7f371013c280 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f370000e960 tx=0x7f370000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:43.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f36f77fe640 1 -- 172.21.15.107:0/2320537778 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370000ca00 con 0x7f371006e470 2023-10-25T00:18:43.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f36f77fe640 1 -- 172.21.15.107:0/2320537778 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370000cb60 con 0x7f371006e470 2023-10-25T00:18:43.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f36f77fe640 1 -- 172.21.15.107:0/2320537778 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370000ced0 con 0x7f371006e470 2023-10-25T00:18:43.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f3715794640 1 -- 172.21.15.107:0/2320537778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37101f0880 con 0x7f371006e470 2023-10-25T00:18:43.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.036+0000 7f3715794640 1 -- 172.21.15.107:0/2320537778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37101eed80 con 0x7f371006e470 2023-10-25T00:18:43.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.040+0000 7f3715794640 1 -- 172.21.15.107:0/2320537778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f0005260 con 0x7f371006e470 2023-10-25T00:18:43.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.044+0000 7f36f77fe640 1 -- 172.21.15.107:0/2320537778 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3700013470 con 0x7f371006e470 2023-10-25T00:18:43.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.044+0000 7f36f77fe640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36fc065ee0 0x7f36fc0683a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:43.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.044+0000 7f36f77fe640 1 -- 172.21.15.107:0/2320537778 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7f3700084ae0 con 0x7f371006e470 2023-10-25T00:18:43.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.044+0000 7f370e7fc640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36fc065ee0 0x7f36fc0683a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:43.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.044+0000 7f370e7fc640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36fc065ee0 0x7f36fc0683a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f371013a030 tx=0x7f3704009480 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:43.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.048+0000 7f36f77fe640 1 -- 172.21.15.107:0/2320537778 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3700051180 con 0x7f371006e470 2023-10-25T00:18:43.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.328+0000 7f3715794640 1 -- 172.21.15.107:0/2320537778 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36f0005a90 con 0x7f371006e470 2023-10-25T00:18:43.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f36f77fe640 1 -- 172.21.15.107:0/2320537778 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5771 (secure 0 0 0) 0x7f3700021090 con 0x7f371006e470 2023-10-25T00:18:43.331 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:43.331 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:43.332 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 200 2023-10-25T00:18:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 -- 172.21.15.107:0/2320537778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36fc065ee0 msgr2=0x7f36fc0683a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36fc065ee0 0x7f36fc0683a0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f371013a030 tx=0x7f3704009480 comp rx=0 tx=0).stop 2023-10-25T00:18:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 -- 172.21.15.107:0/2320537778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f371006e470 msgr2=0x7f371013c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f371006e470 0x7f371013c280 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f370000e960 tx=0x7f370000ee20 comp rx=0 tx=0).stop 2023-10-25T00:18:43.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 -- 172.21.15.107:0/2320537778 shutdown_connections 2023-10-25T00:18:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f36fc065ee0 0x7f36fc0683a0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f371013dc20 0x7f37101ee840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f37100720c0 0x7f37101ee300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 --2- 172.21.15.107:0/2320537778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f371006e470 0x7f371013c280 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 -- 172.21.15.107:0/2320537778 >> 172.21.15.107:0/2320537778 conn(0x7f37100c39a0 msgr2=0x7f371006c410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 -- 172.21.15.107:0/2320537778 shutdown_connections 2023-10-25T00:18:43.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:43.332+0000 7f3715794640 1 -- 172.21.15.107:0/2320537778 wait complete. 2023-10-25T00:18:43.352 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:18:45.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:45.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e8a489640 1 Processor -- start 2023-10-25T00:18:45.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e8a489640 1 -- start start 2023-10-25T00:18:45.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e8a489640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0e841561e0 0x7f0e840c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:45.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e8a489640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e840c1da0 0x7f0e8415c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:45.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e8a489640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e841565c0 con 0x7f0e8406bc30 2023-10-25T00:18:45.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e8a489640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8406a2b0 con 0x7f0e841561e0 2023-10-25T00:18:45.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e8a489640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8406a590 con 0x7f0e840c1da0 2023-10-25T00:18:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e83fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0e841561e0 0x7f0e840c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e837fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f0e8406bc30 0x7f0e8415c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53678/0 (socket says 172.21.15.107:53678) 2023-10-25T00:18:45.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e837fe640 1 -- 172.21.15.107:0/1837557825 learned_addr learned my addr 172.21.15.107:0/1837557825 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:18:45.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e88a26640 1 --2- 172.21.15.107:0/1837557825 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e840c1da0 0x7f0e8415c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:45.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e83fff640 1 -- 172.21.15.107:0/1837557825 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e840c1da0 msgr2=0x7f0e8415c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:45.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e83fff640 1 --2- 172.21.15.107:0/1837557825 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e840c1da0 0x7f0e8415c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:45.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e83fff640 1 -- 172.21.15.107:0/1837557825 >> v1:172.21.15.107:6789/0 conn(0x7f0e8406bc30 legacy=0x7f0e8415c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:45.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e82ffd640 1 -- 172.21.15.107:0/1837557825 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686082985 0 0) 0x7f0e841565c0 con 0x7f0e8406bc30 2023-10-25T00:18:45.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.496+0000 7f0e83fff640 1 -- 172.21.15.107:0/1837557825 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8406a870 con 0x7f0e841561e0 2023-10-25T00:18:45.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e83fff640 1 --2- 172.21.15.107:0/1837557825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e841561e0 0x7f0e840c1860 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f0e740097c0 tx=0x7f0e7401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b4d26635f210cf server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:45.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e82ffd640 1 -- 172.21.15.107:0/1837557825 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e740ab600 con 0x7f0e841561e0 2023-10-25T00:18:45.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e82ffd640 1 -- 172.21.15.107:0/1837557825 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e740a7050 con 0x7f0e841561e0 2023-10-25T00:18:45.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e82ffd640 1 -- 172.21.15.107:0/1837557825 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e740ab980 con 0x7f0e841561e0 2023-10-25T00:18:45.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e8a489640 1 -- 172.21.15.107:0/1837557825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e841561e0 msgr2=0x7f0e840c1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:45.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e8a489640 1 --2- 172.21.15.107:0/1837557825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e841561e0 0x7f0e840c1860 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f0e740097c0 tx=0x7f0e7401c3f0 comp rx=0 tx=0).stop 2023-10-25T00:18:45.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e8a489640 1 -- 172.21.15.107:0/1837557825 shutdown_connections 2023-10-25T00:18:45.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e8a489640 1 --2- 172.21.15.107:0/1837557825 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e840c1da0 0x7f0e8415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:45.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e8a489640 1 --2- 172.21.15.107:0/1837557825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e841561e0 0x7f0e840c1860 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:45.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e8a489640 1 -- 172.21.15.107:0/1837557825 >> 172.21.15.107:0/1837557825 conn(0x7f0e840c2950 msgr2=0x7f0e840c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:45.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e8a489640 1 -- 172.21.15.107:0/1837557825 shutdown_connections 2023-10-25T00:18:45.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e8a489640 1 -- 172.21.15.107:0/1837557825 wait complete. 2023-10-25T00:18:45.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.500+0000 7f0e8a489640 1 Processor -- start 2023-10-25T00:18:45.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e8a489640 1 -- start start 2023-10-25T00:18:45.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e8a489640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e840c1da0 0x7f0e841f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:45.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e8a489640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e841561e0 0x7f0e841f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:45.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e8a489640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e841f46b0 0x7f0e8414ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:45.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e8a489640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e84163320 con 0x7f0e841561e0 2023-10-25T00:18:45.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e8a489640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e841f50b0 con 0x7f0e841f46b0 2023-10-25T00:18:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e8a489640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8406fc90 con 0x7f0e840c1da0 2023-10-25T00:18:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e841561e0 0x7f0e841f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e837fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e841561e0 0x7f0e841f4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58382/0 (socket says 172.21.15.107:58382) 2023-10-25T00:18:45.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e837fe640 1 -- 172.21.15.107:0/3895021834 learned_addr learned my addr 172.21.15.107:0/3895021834 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:45.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e83fff640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e840c1da0 0x7f0e841f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:45.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e88a26640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e841f46b0 0x7f0e8414ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:45.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e837fe640 1 -- 172.21.15.107:0/3895021834 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e840c1da0 msgr2=0x7f0e841f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:45.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e837fe640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e840c1da0 0x7f0e841f3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:45.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e837fe640 1 -- 172.21.15.107:0/3895021834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e841f46b0 msgr2=0x7f0e8414ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:45.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e837fe640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e841f46b0 0x7f0e8414ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:45.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e837fe640 1 -- 172.21.15.107:0/3895021834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e74009420 con 0x7f0e841561e0 2023-10-25T00:18:45.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e837fe640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e841561e0 0x7f0e841f4170 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800c520 tx=0x7f0e7800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:45.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e80ff9640 1 -- 172.21.15.107:0/3895021834 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e780074e0 con 0x7f0e841561e0 2023-10-25T00:18:45.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e80ff9640 1 -- 172.21.15.107:0/3895021834 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e78007640 con 0x7f0e841561e0 2023-10-25T00:18:45.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e8a489640 1 -- 172.21.15.107:0/3895021834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e841f4b40 con 0x7f0e841561e0 2023-10-25T00:18:45.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e80ff9640 1 -- 172.21.15.107:0/3895021834 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e780049f0 con 0x7f0e841561e0 2023-10-25T00:18:45.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e8a489640 1 -- 172.21.15.107:0/3895021834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8406f920 con 0x7f0e841561e0 2023-10-25T00:18:45.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.504+0000 7f0e8a489640 1 -- 172.21.15.107:0/3895021834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8406fe20 con 0x7f0e841561e0 2023-10-25T00:18:45.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.508+0000 7f0e80ff9640 1 -- 172.21.15.107:0/3895021834 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0e78004b50 con 0x7f0e841561e0 2023-10-25T00:18:45.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.508+0000 7f0e80ff9640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e70065e90 0x7f0e70068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:45.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.508+0000 7f0e80ff9640 1 -- 172.21.15.107:0/3895021834 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7f0e78083de0 con 0x7f0e841561e0 2023-10-25T00:18:45.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.508+0000 7f0e83fff640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e70065e90 0x7f0e70068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:45.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.508+0000 7f0e83fff640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e70065e90 0x7f0e70068350 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0e740ad000 tx=0x7f0e740a4f90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:45.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.516+0000 7f0e80ff9640 1 -- 172.21.15.107:0/3895021834 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0e780504d0 con 0x7f0e841561e0 2023-10-25T00:18:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.772+0000 7f0e8a489640 1 -- 172.21.15.107:0/3895021834 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e840746e0 con 0x7f0e841561e0 2023-10-25T00:18:45.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.772+0000 7f0e80ff9640 1 -- 172.21.15.107:0/3895021834 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5771 (secure 0 0 0) 0x7f0e78018070 con 0x7f0e841561e0 2023-10-25T00:18:45.774 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:45.775 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:45.776 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 200 2023-10-25T00:18:45.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 -- 172.21.15.107:0/3895021834 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e70065e90 msgr2=0x7f0e70068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e70065e90 0x7f0e70068350 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0e740ad000 tx=0x7f0e740a4f90 comp rx=0 tx=0).stop 2023-10-25T00:18:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 -- 172.21.15.107:0/3895021834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e841561e0 msgr2=0x7f0e841f4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e841561e0 0x7f0e841f4170 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800c520 tx=0x7f0e7800c9e0 comp rx=0 tx=0).stop 2023-10-25T00:18:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 -- 172.21.15.107:0/3895021834 shutdown_connections 2023-10-25T00:18:45.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e841f46b0 0x7f0e8414ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e841561e0 0x7f0e841f4170 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0e70065e90 0x7f0e70068350 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 --2- 172.21.15.107:0/3895021834 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e840c1da0 0x7f0e841f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 -- 172.21.15.107:0/3895021834 >> 172.21.15.107:0/3895021834 conn(0x7f0e840c2950 msgr2=0x7f0e840c53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:45.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 -- 172.21.15.107:0/3895021834 shutdown_connections 2023-10-25T00:18:45.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:45.776+0000 7f0e8a489640 1 -- 172.21.15.107:0/3895021834 wait complete. 2023-10-25T00:18:45.799 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:18:47.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:47.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.707+0000 7fc237b4b640 1 Processor -- start 2023-10-25T00:18:47.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.707+0000 7fc237b4b640 1 -- start start 2023-10-25T00:18:47.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc237b4b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc2301476e0 0x7fc2301516d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:47.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc237b4b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2300723f0 con 0x7fc23006a810 2023-10-25T00:18:47.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc237b4b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23006a380 con 0x7fc2300c24c0 2023-10-25T00:18:47.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc237b4b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc230151c10 con 0x7fc2301476e0 2023-10-25T00:18:47.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc2358e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc2301476e0 0x7fc2301516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:47.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc2360e8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc23006a810 0x7fc23006ac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53682/0 (socket says 172.21.15.107:53682) 2023-10-25T00:18:47.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc2360e8640 1 -- 172.21.15.107:0/4039217565 learned_addr learned my addr 172.21.15.107:0/4039217565 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:18:47.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc2358e7640 1 -- 172.21.15.107:0/4039217565 >> v1:172.21.15.110:6789/0 conn(0x7fc2300c24c0 legacy=0x7fc23006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:47.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc2358e7640 1 -- 172.21.15.107:0/4039217565 >> v1:172.21.15.107:6789/0 conn(0x7fc23006a810 legacy=0x7fc23006ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:47.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc2348e5640 1 -- 172.21.15.107:0/4039217565 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345081333 0 0) 0x7fc2300723f0 con 0x7fc23006a810 2023-10-25T00:18:47.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc2358e7640 1 -- 172.21.15.107:0/4039217565 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc230069fd0 con 0x7fc2301476e0 2023-10-25T00:18:47.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc2358e7640 1 --2- 172.21.15.107:0/4039217565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301476e0 0x7fc2301516d0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0097c0 tx=0x7fc22c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af7d1689bcca743 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:47.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc2348e5640 1 -- 172.21.15.107:0/4039217565 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0ab600 con 0x7fc2301476e0 2023-10-25T00:18:47.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc2348e5640 1 -- 172.21.15.107:0/4039217565 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c0a7050 con 0x7fc2301476e0 2023-10-25T00:18:47.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.711+0000 7fc2348e5640 1 -- 172.21.15.107:0/4039217565 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0ab980 con 0x7fc2301476e0 2023-10-25T00:18:47.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 -- 172.21.15.107:0/4039217565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301476e0 msgr2=0x7fc2301516d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:47.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 --2- 172.21.15.107:0/4039217565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301476e0 0x7fc2301516d0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0097c0 tx=0x7fc22c01c360 comp rx=0 tx=0).stop 2023-10-25T00:18:47.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 -- 172.21.15.107:0/4039217565 shutdown_connections 2023-10-25T00:18:47.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 --2- 172.21.15.107:0/4039217565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301476e0 0x7fc2301516d0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:47.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 -- 172.21.15.107:0/4039217565 >> 172.21.15.107:0/4039217565 conn(0x7fc2300c2950 msgr2=0x7fc2300c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:47.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 -- 172.21.15.107:0/4039217565 shutdown_connections 2023-10-25T00:18:47.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 -- 172.21.15.107:0/4039217565 wait complete. 2023-10-25T00:18:47.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 Processor -- start 2023-10-25T00:18:47.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 -- start start 2023-10-25T00:18:47.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc23006a810 0x7fc2300c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:47.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2300c24c0 0x7fc2301f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:47.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301476e0 0x7fc23013f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:47.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc230142550 con 0x7fc2301476e0 2023-10-25T00:18:47.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2301f0c90 con 0x7fc23006a810 2023-10-25T00:18:47.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc230154ec0 con 0x7fc2300c24c0 2023-10-25T00:18:47.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2360e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301476e0 0x7fc23013f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:47.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2360e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301476e0 0x7fc23013f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58410/0 (socket says 172.21.15.107:58410) 2023-10-25T00:18:47.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2360e8640 1 -- 172.21.15.107:0/3377605218 learned_addr learned my addr 172.21.15.107:0/3377605218 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:47.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2350e6640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2300c24c0 0x7fc2301f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:47.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2358e7640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc23006a810 0x7fc2300c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:47.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2360e8640 1 -- 172.21.15.107:0/3377605218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2300c24c0 msgr2=0x7fc2301f13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:47.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2360e8640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2300c24c0 0x7fc2301f13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:47.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2360e8640 1 -- 172.21.15.107:0/3377605218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc23006a810 msgr2=0x7fc2300c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:47.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2360e8640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc23006a810 0x7fc2300c50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:47.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2360e8640 1 -- 172.21.15.107:0/3377605218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c009420 con 0x7fc2301476e0 2023-10-25T00:18:47.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2360e8640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301476e0 0x7fc23013f1f0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fc22800c780 tx=0x7fc22800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:47.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc2267fc640 1 -- 172.21.15.107:0/3377605218 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22800ce90 con 0x7fc2301476e0 2023-10-25T00:18:47.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.715+0000 7fc237b4b640 1 -- 172.21.15.107:0/3377605218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc230141f90 con 0x7fc2301476e0 2023-10-25T00:18:47.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.719+0000 7fc237b4b640 1 -- 172.21.15.107:0/3377605218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2301f0ea0 con 0x7fc2301476e0 2023-10-25T00:18:47.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.719+0000 7fc2267fc640 1 -- 172.21.15.107:0/3377605218 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22800a640 con 0x7fc2301476e0 2023-10-25T00:18:47.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.719+0000 7fc2267fc640 1 -- 172.21.15.107:0/3377605218 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22800a9d0 con 0x7fc2301476e0 2023-10-25T00:18:47.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.719+0000 7fc237b4b640 1 -- 172.21.15.107:0/3377605218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2301f10b0 con 0x7fc2301476e0 2023-10-25T00:18:47.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.719+0000 7fc2267fc640 1 -- 172.21.15.107:0/3377605218 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc22800ab30 con 0x7fc2301476e0 2023-10-25T00:18:47.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.719+0000 7fc2267fc640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc214065e60 0x7fc214068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:47.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.719+0000 7fc2358e7640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc214065e60 0x7fc214068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:47.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.719+0000 7fc2267fc640 1 -- 172.21.15.107:0/3377605218 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7fc228083fb0 con 0x7fc2301476e0 2023-10-25T00:18:47.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.719+0000 7fc2358e7640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc214065e60 0x7fc214068320 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0ad000 tx=0x7fc22c0a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:47.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.723+0000 7fc2267fc640 1 -- 172.21.15.107:0/3377605218 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc2280b5040 con 0x7fc2301476e0 2023-10-25T00:18:47.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:47.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.947+0000 7ff5b99d1640 1 Processor -- start 2023-10-25T00:18:47.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.947+0000 7ff5b99d1640 1 -- start start 2023-10-25T00:18:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b99d1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5b414e310 0x7ff5b4152220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b99d1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff5b406bcf0 0x7ff5b40c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:47.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b99d1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b4152760 con 0x7ff5b406c870 2023-10-25T00:18:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b99d1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b40ac230 con 0x7ff5b414e310 2023-10-25T00:18:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b99d1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b406cd50 con 0x7ff5b406bcf0 2023-10-25T00:18:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5b414e310 0x7ff5b4152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b2ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff5b414e310 0x7ff5b4152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58420/0 (socket says 172.21.15.107:58420) 2023-10-25T00:18:47.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b2ffd640 1 -- 172.21.15.107:0/1697943961 learned_addr learned my addr 172.21.15.107:0/1697943961 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b2ffd640 1 -- 172.21.15.107:0/1697943961 >> v2:172.21.15.110:3300/0 conn(0x7ff5b406bcf0 msgr2=0x7ff5b40c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b2ffd640 1 --2- 172.21.15.107:0/1697943961 >> v2:172.21.15.110:3300/0 conn(0x7ff5b406bcf0 0x7ff5b40c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b2ffd640 1 -- 172.21.15.107:0/1697943961 >> v1:172.21.15.110:6789/0 conn(0x7ff5b406c870 legacy=0x7ff5b41517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b2ffd640 1 -- 172.21.15.107:0/1697943961 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b406d030 con 0x7ff5b414e310 2023-10-25T00:18:47.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b2ffd640 1 --2- 172.21.15.107:0/1697943961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b414e310 0x7ff5b4152220 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7ff5b40c2890 tx=0x7ff5ac01eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5adc7cc317957ed0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:47.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b1ffb640 1 -- 172.21.15.107:0/1697943961 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac00b2d0 con 0x7ff5b414e310 2023-10-25T00:18:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b1ffb640 1 -- 172.21.15.107:0/1697943961 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ac00b430 con 0x7ff5b414e310 2023-10-25T00:18:47.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.951+0000 7ff5b1ffb640 1 -- 172.21.15.107:0/1697943961 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac0a7a60 con 0x7ff5b414e310 2023-10-25T00:18:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.955+0000 7ff5b99d1640 1 -- 172.21.15.107:0/1697943961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b414e310 msgr2=0x7ff5b4152220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:47.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.955+0000 7ff5b99d1640 1 --2- 172.21.15.107:0/1697943961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b414e310 0x7ff5b4152220 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7ff5b40c2890 tx=0x7ff5ac01eb00 comp rx=0 tx=0).stop 2023-10-25T00:18:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.959+0000 7ff5b99d1640 1 -- 172.21.15.107:0/1697943961 shutdown_connections 2023-10-25T00:18:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.959+0000 7ff5b99d1640 1 --2- 172.21.15.107:0/1697943961 >> v2:172.21.15.110:3300/0 conn(0x7ff5b406bcf0 0x7ff5b40c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:47.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.959+0000 7ff5b99d1640 1 --2- 172.21.15.107:0/1697943961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b414e310 0x7ff5b4152220 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.959+0000 7ff5b99d1640 1 -- 172.21.15.107:0/1697943961 >> 172.21.15.107:0/1697943961 conn(0x7ff5b40c6270 msgr2=0x7ff5b40c6680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.959+0000 7ff5b99d1640 1 -- 172.21.15.107:0/1697943961 shutdown_connections 2023-10-25T00:18:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.959+0000 7ff5b99d1640 1 -- 172.21.15.107:0/1697943961 wait complete. 2023-10-25T00:18:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b99d1640 1 Processor -- start 2023-10-25T00:18:47.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b99d1640 1 -- start start 2023-10-25T00:18:47.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b99d1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5b406bcf0 0x7ff5b414d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:47.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b99d1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406c870 0x7ff5b4150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:47.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b99d1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5b414e310 0x7ff5b414ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:47.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b99d1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b40713c0 con 0x7ff5b406c870 2023-10-25T00:18:47.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b99d1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b414db20 con 0x7ff5b414e310 2023-10-25T00:18:47.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b99d1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b41e94e0 con 0x7ff5b406bcf0 2023-10-25T00:18:47.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406c870 0x7ff5b4150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:47.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406c870 0x7ff5b4150160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58434/0 (socket says 172.21.15.107:58434) 2023-10-25T00:18:47.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b27fc640 1 -- 172.21.15.107:0/2977348854 learned_addr learned my addr 172.21.15.107:0/2977348854 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:47.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b2ffd640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5b406bcf0 0x7ff5b414d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:47.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b37fe640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5b414e310 0x7ff5b414ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:47.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b27fc640 1 -- 172.21.15.107:0/2977348854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5b406bcf0 msgr2=0x7ff5b414d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:47.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b27fc640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5b406bcf0 0x7ff5b414d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:47.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b27fc640 1 -- 172.21.15.107:0/2977348854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5b414e310 msgr2=0x7ff5b414ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:47.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b27fc640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5b414e310 0x7ff5b414ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:47.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b27fc640 1 -- 172.21.15.107:0/2977348854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5ac00bd40 con 0x7ff5b406c870 2023-10-25T00:18:47.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b27fc640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406c870 0x7ff5b4150160 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7ff5a800c410 tx=0x7ff5a800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:47.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff59bfff640 1 -- 172.21.15.107:0/2977348854 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8011790 con 0x7ff5b406c870 2023-10-25T00:18:47.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff59bfff640 1 -- 172.21.15.107:0/2977348854 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a80118f0 con 0x7ff5b406c870 2023-10-25T00:18:47.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b99d1640 1 -- 172.21.15.107:0/2977348854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b41e8960 con 0x7ff5b406c870 2023-10-25T00:18:47.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff59bfff640 1 -- 172.21.15.107:0/2977348854 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8011ce0 con 0x7ff5b406c870 2023-10-25T00:18:47.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b99d1640 1 -- 172.21.15.107:0/2977348854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b41e8f20 con 0x7ff5b406c870 2023-10-25T00:18:47.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.963+0000 7ff5b99d1640 1 -- 172.21.15.107:0/2977348854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b40badf0 con 0x7ff5b406c870 2023-10-25T00:18:47.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.967+0000 7ff59bfff640 1 -- 172.21.15.107:0/2977348854 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff5a801c460 con 0x7ff5b406c870 2023-10-25T00:18:47.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.967+0000 7ff59bfff640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5a0065e60 0x7ff5a0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:47.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.967+0000 7ff59bfff640 1 -- 172.21.15.107:0/2977348854 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7ff5a8084440 con 0x7ff5b406c870 2023-10-25T00:18:47.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.967+0000 7ff5b2ffd640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5a0065e60 0x7ff5a0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.967+0000 7ff5b2ffd640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5a0065e60 0x7ff5a0068320 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0b0000 tx=0x7ff5ac0a9070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:47.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:47.979+0000 7ff59bfff640 1 -- 172.21.15.107:0/2977348854 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff5b40badf0 con 0x7ff5b406c870 2023-10-25T00:18:48.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.027+0000 7fc237b4b640 1 -- 172.21.15.107:0/3377605218 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2300746e0 con 0x7fc2301476e0 2023-10-25T00:18:48.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.027+0000 7fc2267fc640 1 -- 172.21.15.107:0/3377605218 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5771 (secure 0 0 0) 0x7fc228018020 con 0x7fc2301476e0 2023-10-25T00:18:48.028 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:48.028 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:48.028 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 200 2023-10-25T00:18:48.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.031+0000 7fc237b4b640 1 -- 172.21.15.107:0/3377605218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc214065e60 msgr2=0x7fc214068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:48.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.031+0000 7fc237b4b640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc214065e60 0x7fc214068320 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0ad000 tx=0x7fc22c0a6000 comp rx=0 tx=0).stop 2023-10-25T00:18:48.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.031+0000 7fc237b4b640 1 -- 172.21.15.107:0/3377605218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301476e0 msgr2=0x7fc23013f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:48.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.031+0000 7fc237b4b640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301476e0 0x7fc23013f1f0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fc22800c780 tx=0x7fc22800cc40 comp rx=0 tx=0).stop 2023-10-25T00:18:48.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.031+0000 7fc237b4b640 1 -- 172.21.15.107:0/3377605218 shutdown_connections 2023-10-25T00:18:48.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.031+0000 7fc237b4b640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301476e0 0x7fc23013f1f0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.031+0000 7fc237b4b640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc214065e60 0x7fc214068320 secure :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc22c0ad000 tx=0x7fc22c0a6000 comp rx=0 tx=0).stop 2023-10-25T00:18:48.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.031+0000 7fc237b4b640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2300c24c0 0x7fc2301f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.031+0000 7fc237b4b640 1 --2- 172.21.15.107:0/3377605218 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc23006a810 0x7fc2300c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.031+0000 7fc237b4b640 1 -- 172.21.15.107:0/3377605218 >> 172.21.15.107:0/3377605218 conn(0x7fc2300c2950 msgr2=0x7fc23006bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:48.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.035+0000 7fc237b4b640 1 -- 172.21.15.107:0/3377605218 shutdown_connections 2023-10-25T00:18:48.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.035+0000 7fc237b4b640 1 -- 172.21.15.107:0/3377605218 wait complete. 2023-10-25T00:18:48.064 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2023-10-25T00:18:48.064 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:18:48.064 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:18:48.078 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:18:48.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:48.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.259+0000 7ff5b99d1640 1 -- 172.21.15.107:0/2977348854 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5b40badf0 con 0x7ff5b406c870 2023-10-25T00:18:48.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff59bfff640 1 -- 172.21.15.107:0/2977348854 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5771 (secure 0 0 0) 0x7ff5b40badf0 con 0x7ff5b406c870 2023-10-25T00:18:48.265 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:48.265 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:48.266 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 200 2023-10-25T00:18:48.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 -- 172.21.15.107:0/2977348854 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5a0065e60 msgr2=0x7ff5a0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:48.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5a0065e60 0x7ff5a0068320 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0b0000 tx=0x7ff5ac0a9070 comp rx=0 tx=0).stop 2023-10-25T00:18:48.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 -- 172.21.15.107:0/2977348854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406c870 msgr2=0x7ff5b4150160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:48.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406c870 0x7ff5b4150160 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7ff5a800c410 tx=0x7ff5a800c8d0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 -- 172.21.15.107:0/2977348854 shutdown_connections 2023-10-25T00:18:48.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5b414e310 0x7ff5b414ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5b406c870 0x7ff5b4150160 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5a0065e60 0x7ff5a0068320 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 --2- 172.21.15.107:0/2977348854 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5b406bcf0 0x7ff5b414d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 -- 172.21.15.107:0/2977348854 >> 172.21.15.107:0/2977348854 conn(0x7ff5b40c6270 msgr2=0x7ff5b4152930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:48.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 -- 172.21.15.107:0/2977348854 shutdown_connections 2023-10-25T00:18:48.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.263+0000 7ff5b99d1640 1 -- 172.21.15.107:0/2977348854 wait complete. 2023-10-25T00:18:48.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8b4905640 1 Processor -- start 2023-10-25T00:18:48.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8b4905640 1 -- start start 2023-10-25T00:18:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8b4905640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc8b013dc20 0x7fc8b00c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8b4905640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc8b00720c0 0x7fc8b006df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8b4905640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 0x7fc8b0157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8b4905640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b0154860 con 0x7fc8b013dc20 2023-10-25T00:18:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8b4905640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b00c1f10 con 0x7fc8b00720c0 2023-10-25T00:18:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8b4905640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b0154b20 con 0x7fc8b006e470 2023-10-25T00:18:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8ae59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc8b013dc20 0x7fc8b00c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:48.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8ae59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc8b013dc20 0x7fc8b00c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58450/0 (socket says 172.21.15.107:58450) 2023-10-25T00:18:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8ae59c640 1 -- 172.21.15.107:0/1620855006 learned_addr learned my addr 172.21.15.107:0/1620855006 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8aed9d640 1 --2- 172.21.15.107:0/1620855006 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 0x7fc8b0157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8ae59c640 1 -- 172.21.15.107:0/1620855006 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 msgr2=0x7fc8b0157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8ae59c640 1 --2- 172.21.15.107:0/1620855006 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 0x7fc8b0157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8ae59c640 1 -- 172.21.15.107:0/1620855006 >> v2:172.21.15.110:3300/0 conn(0x7fc8b00720c0 msgr2=0x7fc8b006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8ae59c640 1 --2- 172.21.15.107:0/1620855006 >> v2:172.21.15.110:3300/0 conn(0x7fc8b00720c0 0x7fc8b006df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8ae59c640 1 -- 172.21.15.107:0/1620855006 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b0154e00 con 0x7fc8b013dc20 2023-10-25T00:18:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.279+0000 7fc8aed9d640 1 --2- 172.21.15.107:0/1620855006 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 0x7fc8b0157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:18:48.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.283+0000 7fc8ae59c640 1 --2- 172.21.15.107:0/1620855006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8b013dc20 0x7fc8b00c18a0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00c06a0 tx=0x7fc8a001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b05cbf2c7129dc server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:48.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.283+0000 7fc8ad59a640 1 -- 172.21.15.107:0/1620855006 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a00ab600 con 0x7fc8b013dc20 2023-10-25T00:18:48.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.287+0000 7fc8ad59a640 1 -- 172.21.15.107:0/1620855006 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a00a7050 con 0x7fc8b013dc20 2023-10-25T00:18:48.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.287+0000 7fc8ad59a640 1 -- 172.21.15.107:0/1620855006 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a00ab980 con 0x7fc8b013dc20 2023-10-25T00:18:48.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.287+0000 7fc8b4905640 1 -- 172.21.15.107:0/1620855006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8b013dc20 msgr2=0x7fc8b00c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:48.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.287+0000 7fc8b4905640 1 --2- 172.21.15.107:0/1620855006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8b013dc20 0x7fc8b00c18a0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fc8b00c06a0 tx=0x7fc8a001c3e0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.291 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:18:48.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.291+0000 7fc8b4905640 1 -- 172.21.15.107:0/1620855006 shutdown_connections 2023-10-25T00:18:48.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.291+0000 7fc8b4905640 1 --2- 172.21.15.107:0/1620855006 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 0x7fc8b0157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.291+0000 7fc8b4905640 1 --2- 172.21.15.107:0/1620855006 >> v2:172.21.15.110:3300/0 conn(0x7fc8b00720c0 0x7fc8b006df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.291+0000 7fc8b4905640 1 --2- 172.21.15.107:0/1620855006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8b013dc20 0x7fc8b00c18a0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.291+0000 7fc8b4905640 1 -- 172.21.15.107:0/1620855006 >> 172.21.15.107:0/1620855006 conn(0x7fc8b00c39a0 msgr2=0x7fc8b00c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:48.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.291+0000 7fc8b4905640 1 -- 172.21.15.107:0/1620855006 shutdown_connections 2023-10-25T00:18:48.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.291+0000 7fc8b4905640 1 -- 172.21.15.107:0/1620855006 wait complete. 2023-10-25T00:18:48.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.291+0000 7fc8b4905640 1 Processor -- start 2023-10-25T00:18:48.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.291+0000 7fc8b4905640 1 -- start start 2023-10-25T00:18:48.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8b4905640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 0x7fc8b013c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:48.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8b4905640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b00720c0 0x7fc8b00c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:48.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8b4905640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8b013dc20 0x7fc8b01ec900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:48.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8b4905640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b01571b0 con 0x7fc8b013dc20 2023-10-25T00:18:48.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8b4905640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b0154660 con 0x7fc8b00720c0 2023-10-25T00:18:48.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8b4905640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b0154130 con 0x7fc8b006e470 2023-10-25T00:18:48.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8ae59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 0x7fc8b013c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:48.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8add9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b00720c0 0x7fc8b00c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:48.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8add9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b00720c0 0x7fc8b00c6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:51350/0 (socket says 172.21.15.107:51350) 2023-10-25T00:18:48.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8add9b640 1 -- 172.21.15.107:0/1087551395 learned_addr learned my addr 172.21.15.107:0/1087551395 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:48.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8ae59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 0x7fc8b013c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:36088/0 (socket says 172.21.15.107:36088) 2023-10-25T00:18:48.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8ae59c640 1 -- 172.21.15.107:0/1087551395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b00720c0 msgr2=0x7fc8b00c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:48.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8ae59c640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b00720c0 0x7fc8b00c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8ae59c640 1 -- 172.21.15.107:0/1087551395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8b013dc20 msgr2=0x7fc8b01ec900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:18:48.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8aed9d640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8b013dc20 0x7fc8b01ec900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:48.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8ae59c640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8b013dc20 0x7fc8b01ec900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8ae59c640 1 -- 172.21.15.107:0/1087551395 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a8009a70 con 0x7fc8b006e470 2023-10-25T00:18:48.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8aed9d640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8b013dc20 0x7fc8b01ec900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:18:48.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8add9b640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b00720c0 0x7fc8b00c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:18:48.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8ae59c640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 0x7fc8b013c260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc8a00ab000 tx=0x7fc8a0019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:48.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc89effd640 1 -- 172.21.15.107:0/1087551395 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a00b4660 con 0x7fc8b006e470 2023-10-25T00:18:48.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc89effd640 1 -- 172.21.15.107:0/1087551395 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a00b47c0 con 0x7fc8b006e470 2023-10-25T00:18:48.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8b4905640 1 -- 172.21.15.107:0/1087551395 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a00094d0 con 0x7fc8b006e470 2023-10-25T00:18:48.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8b4905640 1 -- 172.21.15.107:0/1087551395 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b01f0660 con 0x7fc8b006e470 2023-10-25T00:18:48.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc89effd640 1 -- 172.21.15.107:0/1087551395 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a00b4b50 con 0x7fc8b006e470 2023-10-25T00:18:48.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.295+0000 7fc8b4905640 1 -- 172.21.15.107:0/1087551395 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc894005260 con 0x7fc8b006e470 2023-10-25T00:18:48.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.299+0000 7fc89effd640 1 -- 172.21.15.107:0/1087551395 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc8a00b2070 con 0x7fc8b006e470 2023-10-25T00:18:48.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.299+0000 7fc89effd640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc890065eb0 0x7fc890068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:48.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.299+0000 7fc89effd640 1 -- 172.21.15.107:0/1087551395 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 10229+0+0 (secure 0 0 0) 0x7fc8a011e180 con 0x7fc8b006e470 2023-10-25T00:18:48.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.299+0000 7fc8add9b640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc890065eb0 0x7fc890068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:48.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.299+0000 7fc8add9b640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc890065eb0 0x7fc890068370 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc8b01eda30 tx=0x7fc8a4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:48.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.307+0000 7fc89effd640 1 -- 172.21.15.107:0/1087551395 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc8a00ea8f0 con 0x7fc8b006e470 2023-10-25T00:18:48.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.579+0000 7fc8b4905640 1 -- 172.21.15.107:0/1087551395 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc894005510 con 0x7fc8b006e470 2023-10-25T00:18:48.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.579+0000 7fc89effd640 1 -- 172.21.15.107:0/1087551395 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5771 (secure 0 0 0) 0x7fc8a00b0090 con 0x7fc8b006e470 2023-10-25T00:18:48.581 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:48.581 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":200}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:17:28.068403+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"a","rank":0,"incarnation":191,"state":"up:active","state_seq":32,"addr":"172.21.15.107:6837/418589140","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":418589140},{"type":"v1","addr":"172.21.15.107:6837","nonce":418589140}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 200 2023-10-25T00:18:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 -- 172.21.15.107:0/1087551395 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc890065eb0 msgr2=0x7fc890068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc890065eb0 0x7fc890068370 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc8b01eda30 tx=0x7fc8a4009190 comp rx=0 tx=0).stop 2023-10-25T00:18:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 -- 172.21.15.107:0/1087551395 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 msgr2=0x7fc8b013c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 0x7fc8b013c260 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc8a00ab000 tx=0x7fc8a0019180 comp rx=0 tx=0).stop 2023-10-25T00:18:48.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 -- 172.21.15.107:0/1087551395 shutdown_connections 2023-10-25T00:18:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8b013dc20 0x7fc8b01ec900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8b00720c0 0x7fc8b00c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc890065eb0 0x7fc890068370 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 --2- 172.21.15.107:0/1087551395 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc8b006e470 0x7fc8b013c260 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:48.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 -- 172.21.15.107:0/1087551395 >> 172.21.15.107:0/1087551395 conn(0x7fc8b00c39a0 msgr2=0x7fc8b00c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 -- 172.21.15.107:0/1087551395 shutdown_connections 2023-10-25T00:18:48.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:48.583+0000 7fc8b4905640 1 -- 172.21.15.107:0/1087551395 wait complete. 2023-10-25T00:18:48.603 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:18:50.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:50.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.427+0000 7f97580f2640 1 Processor -- start 2023-10-25T00:18:50.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.427+0000 7f97580f2640 1 -- start start 2023-10-25T00:18:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f97580f2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f97500720c0 0x7f97500c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f97580f2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9750154860 con 0x7f9750157e50 2023-10-25T00:18:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f97580f2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97500c1f10 con 0x7f975013dc20 2023-10-25T00:18:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f97580f2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9750154b20 con 0x7f97500720c0 2023-10-25T00:18:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f975668f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f9750157e50 0x7f975015f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:53684/0 (socket says 172.21.15.107:53684) 2023-10-25T00:18:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f975668f640 1 -- 172.21.15.107:0/3696520992 learned_addr learned my addr 172.21.15.107:0/3696520992 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:18:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f975568d640 1 --2- 172.21.15.107:0/3696520992 >> v2:172.21.15.110:3300/0 conn(0x7f97500720c0 0x7f97500c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180457200 0 0) 0x7f9750154860 con 0x7f9750157e50 2023-10-25T00:18:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c003530 con 0x7f9750157e50 2023-10-25T00:18:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2215424907 0 0) 0x7f973c003530 con 0x7f9750157e50 2023-10-25T00:18:50.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 >> v2:172.21.15.110:3300/0 conn(0x7f97500720c0 msgr2=0x7f97500c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 --2- 172.21.15.107:0/3696520992 >> v2:172.21.15.110:3300/0 conn(0x7f97500720c0 0x7f97500c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 >> v1:172.21.15.110:6789/0 conn(0x7f975013dc20 legacy=0x7f975006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9750154e00 con 0x7f9750157e50 2023-10-25T00:18:50.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f9748001a00 con 0x7f9750157e50 2023-10-25T00:18:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 >> v1:172.21.15.107:6789/0 conn(0x7f9750157e50 legacy=0x7f975015f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 --2- 172.21.15.107:0/3696520992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f973c003980 0x7f973c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 --2- 172.21.15.107:0/3696520992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c0047d0 0x7f973c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 --2- 172.21.15.107:0/3696520992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f973c0052b0 0x7f973c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9750154b20 con 0x7f973c0047d0 2023-10-25T00:18:50.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973c00c180 con 0x7f973c003980 2023-10-25T00:18:50.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f973c00c340 con 0x7f973c0052b0 2023-10-25T00:18:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f975568d640 1 --2- 172.21.15.107:0/3696520992 >> v2:172.21.15.110:3300/0 conn(0x7f97500720c0 0x7f97500c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:18:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f975568d640 1 --2- 172.21.15.107:0/3696520992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c0047d0 0x7f973c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f9755e8e640 1 --2- 172.21.15.107:0/3696520992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f973c003980 0x7f973c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f975568d640 1 -- 172.21.15.107:0/3696520992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f973c0052b0 msgr2=0x7f973c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:50.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f975568d640 1 --2- 172.21.15.107:0/3696520992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f973c0052b0 0x7f973c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f975568d640 1 -- 172.21.15.107:0/3696520992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f973c003980 msgr2=0x7f973c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:50.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f975568d640 1 --2- 172.21.15.107:0/3696520992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f973c003980 0x7f973c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f975568d640 1 -- 172.21.15.107:0/3696520992 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c003530 con 0x7f973c0047d0 2023-10-25T00:18:50.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.431+0000 7f975668f640 1 -- 172.21.15.107:0/3696520992 reap_dead start 2023-10-25T00:18:50.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f975568d640 1 --2- 172.21.15.107:0/3696520992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c0047d0 0x7f973c004c50 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f9740009c60 tx=0x7f974001cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49252b15a9cbc657 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:50.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740004020 con 0x7f973c0047d0 2023-10-25T00:18:50.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9740004180 con 0x7f973c0047d0 2023-10-25T00:18:50.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f9754e8c640 1 -- 172.21.15.107:0/3696520992 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740003620 con 0x7f973c0047d0 2023-10-25T00:18:50.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f97580f2640 1 -- 172.21.15.107:0/3696520992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c0047d0 msgr2=0x7f973c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f97580f2640 1 --2- 172.21.15.107:0/3696520992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c0047d0 0x7f973c004c50 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f9740009c60 tx=0x7f974001cb70 comp rx=0 tx=0).stop 2023-10-25T00:18:50.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f97580f2640 1 -- 172.21.15.107:0/3696520992 shutdown_connections 2023-10-25T00:18:50.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f97580f2640 1 --2- 172.21.15.107:0/3696520992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c0047d0 0x7f973c004c50 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f97580f2640 1 -- 172.21.15.107:0/3696520992 >> 172.21.15.107:0/3696520992 conn(0x7f97500c39a0 msgr2=0x7f97500c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:50.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f97580f2640 1 -- 172.21.15.107:0/3696520992 shutdown_connections 2023-10-25T00:18:50.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f97580f2640 1 -- 172.21.15.107:0/3696520992 wait complete. 2023-10-25T00:18:50.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f97580f2640 1 Processor -- start 2023-10-25T00:18:50.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.435+0000 7f97580f2640 1 -- start start 2023-10-25T00:18:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f97580f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c0047d0 0x7f9750155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f97580f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97500c4d90 0x7f9750156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f97580f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9750152b10 0x7f9750152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f97580f2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97501e2b10 con 0x7f973c0047d0 2023-10-25T00:18:50.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f97580f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97501e33b0 con 0x7f9750152b10 2023-10-25T00:18:50.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f97580f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97501534b0 con 0x7f97500c4d90 2023-10-25T00:18:50.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f975568d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97500c4d90 0x7f9750156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f975668f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9750152b10 0x7f9750152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f975568d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97500c4d90 0x7f9750156920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:36092/0 (socket says 172.21.15.107:36092) 2023-10-25T00:18:50.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f975568d640 1 -- 172.21.15.107:0/1871803852 learned_addr learned my addr 172.21.15.107:0/1871803852 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:50.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f9755e8e640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c0047d0 0x7f9750155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f975668f640 1 -- 172.21.15.107:0/1871803852 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97500c4d90 msgr2=0x7f9750156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f975668f640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97500c4d90 0x7f9750156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f975668f640 1 -- 172.21.15.107:0/1871803852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c0047d0 msgr2=0x7f9750155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f975668f640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c0047d0 0x7f9750155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f975668f640 1 -- 172.21.15.107:0/1871803852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9750156310 con 0x7f9750152b10 2023-10-25T00:18:50.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f975668f640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9750152b10 0x7f9750152f70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f975006ade0 tx=0x7f974800ccd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:50.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f97467fc640 1 -- 172.21.15.107:0/1871803852 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974800cec0 con 0x7f9750152b10 2023-10-25T00:18:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f97467fc640 1 -- 172.21.15.107:0/1871803852 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f974800a610 con 0x7f9750152b10 2023-10-25T00:18:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f97580f2640 1 -- 172.21.15.107:0/1871803852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97501e2d60 con 0x7f9750152b10 2023-10-25T00:18:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f97467fc640 1 -- 172.21.15.107:0/1871803852 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974800a9c0 con 0x7f9750152b10 2023-10-25T00:18:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.439+0000 7f97580f2640 1 -- 172.21.15.107:0/1871803852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9750156e60 con 0x7f9750152b10 2023-10-25T00:18:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.443+0000 7f97580f2640 1 -- 172.21.15.107:0/1871803852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97500746e0 con 0x7f9750152b10 2023-10-25T00:18:50.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.443+0000 7f97467fc640 1 -- 172.21.15.107:0/1871803852 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f974800ab20 con 0x7f9750152b10 2023-10-25T00:18:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.443+0000 7f97467fc640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9734065e90 0x7f9734068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.443+0000 7f97467fc640 1 -- 172.21.15.107:0/1871803852 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 10315+0+0 (secure 0 0 0) 0x7f9748084830 con 0x7f9750152b10 2023-10-25T00:18:50.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.443+0000 7f9755e8e640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9734065e90 0x7f9734068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.443+0000 7f9755e8e640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9734065e90 0x7f9734068350 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f974c000c10 tx=0x7f974c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:50.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.451+0000 7f97467fc640 1 -- 172.21.15.107:0/1871803852 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9748050f50 con 0x7f9750152b10 2023-10-25T00:18:50.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:50.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.707+0000 7f97580f2640 1 -- 172.21.15.107:0/1871803852 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97500c1190 con 0x7f9750152b10 2023-10-25T00:18:50.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.715+0000 7f97467fc640 1 -- 172.21.15.107:0/1871803852 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+5012 (secure 0 0 0) 0x7f9748054570 con 0x7f9750152b10 2023-10-25T00:18:50.715 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:50.715 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":201,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:49.119150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:50.716 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 201 2023-10-25T00:18:50.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.715+0000 7f97580f2640 1 -- 172.21.15.107:0/1871803852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9734065e90 msgr2=0x7f9734068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.715+0000 7f97580f2640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9734065e90 0x7f9734068350 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f974c000c10 tx=0x7f974c009040 comp rx=0 tx=0).stop 2023-10-25T00:18:50.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.715+0000 7f97580f2640 1 -- 172.21.15.107:0/1871803852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9750152b10 msgr2=0x7f9750152f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.715+0000 7f97580f2640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9750152b10 0x7f9750152f70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f975006ade0 tx=0x7f974800ccd0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.715+0000 7f97580f2640 1 -- 172.21.15.107:0/1871803852 shutdown_connections 2023-10-25T00:18:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.715+0000 7f97580f2640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9750152b10 0x7f9750152f70 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.715+0000 7f97580f2640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9734065e90 0x7f9734068350 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.715+0000 7f97580f2640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f97500c4d90 0x7f9750156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.715+0000 7f97580f2640 1 --2- 172.21.15.107:0/1871803852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f973c0047d0 0x7f9750155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.715+0000 7f97580f2640 1 -- 172.21.15.107:0/1871803852 >> 172.21.15.107:0/1871803852 conn(0x7f97500c39a0 msgr2=0x7f97500c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:50.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.719+0000 7f97580f2640 1 -- 172.21.15.107:0/1871803852 shutdown_connections 2023-10-25T00:18:50.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.719+0000 7f97580f2640 1 -- 172.21.15.107:0/1871803852 wait complete. 2023-10-25T00:18:50.746 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2023-10-25T00:18:50.746 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:18:50.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7af481a640 1 Processor -- start 2023-10-25T00:18:50.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7af481a640 1 -- start start 2023-10-25T00:18:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7af481a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7af014eb70 0x7f7af00c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7af481a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0070b00 0x7f7af01514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7af481a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7af0151a20 0x7f7af006df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7af481a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af0071700 con 0x7f7af0151a20 2023-10-25T00:18:50.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7af481a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af00c22f0 con 0x7f7af014eb70 2023-10-25T00:18:50.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7af481a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af00c25d0 con 0x7f7af0070b00 2023-10-25T00:18:50.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7aedd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0070b00 0x7f7af01514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7aedd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0070b00 0x7f7af01514e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:36108/0 (socket says 172.21.15.107:36108) 2023-10-25T00:18:50.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7aedd9b640 1 -- 172.21.15.107:0/1530816647 learned_addr learned my addr 172.21.15.107:0/1530816647 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:50.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.747+0000 7f7aee59c640 1 --2- 172.21.15.107:0/1530816647 >> v2:172.21.15.110:3300/0 conn(0x7f7af014eb70 0x7f7af00c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.751+0000 7f7aeed9d640 1 --2- 172.21.15.107:0/1530816647 >> v2:172.21.15.107:3300/0 conn(0x7f7af0151a20 0x7f7af006df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.751+0000 7f7aedd9b640 1 -- 172.21.15.107:0/1530816647 >> v2:172.21.15.110:3300/0 conn(0x7f7af014eb70 msgr2=0x7f7af00c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.751+0000 7f7aedd9b640 1 --2- 172.21.15.107:0/1530816647 >> v2:172.21.15.110:3300/0 conn(0x7f7af014eb70 0x7f7af00c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.751+0000 7f7aedd9b640 1 -- 172.21.15.107:0/1530816647 >> v2:172.21.15.107:3300/0 conn(0x7f7af0151a20 msgr2=0x7f7af006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:50.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.751+0000 7f7aedd9b640 1 --2- 172.21.15.107:0/1530816647 >> v2:172.21.15.107:3300/0 conn(0x7f7af0151a20 0x7f7af006df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.751+0000 7f7aedd9b640 1 -- 172.21.15.107:0/1530816647 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af00c2be0 con 0x7f7af0070b00 2023-10-25T00:18:50.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.751+0000 7f7aeed9d640 1 --2- 172.21.15.107:0/1530816647 >> v2:172.21.15.107:3300/0 conn(0x7f7af0151a20 0x7f7af006df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:18:50.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7aedd9b640 1 --2- 172.21.15.107:0/1530816647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0070b00 0x7f7af01514e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7af00c6730 tx=0x7f7ae401c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8907f284ad683fe7 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:50.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7aed59a640 1 -- 172.21.15.107:0/1530816647 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae40ab660 con 0x7f7af0070b00 2023-10-25T00:18:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7aed59a640 1 -- 172.21.15.107:0/1530816647 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae40ab7c0 con 0x7f7af0070b00 2023-10-25T00:18:50.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7aed59a640 1 -- 172.21.15.107:0/1530816647 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae40abb50 con 0x7f7af0070b00 2023-10-25T00:18:50.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7af481a640 1 -- 172.21.15.107:0/1530816647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0070b00 msgr2=0x7f7af01514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7af481a640 1 --2- 172.21.15.107:0/1530816647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0070b00 0x7f7af01514e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7af00c6730 tx=0x7f7ae401c500 comp rx=0 tx=0).stop 2023-10-25T00:18:50.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7af481a640 1 -- 172.21.15.107:0/1530816647 shutdown_connections 2023-10-25T00:18:50.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7af481a640 1 --2- 172.21.15.107:0/1530816647 >> v2:172.21.15.107:3300/0 conn(0x7f7af0151a20 0x7f7af006df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7af481a640 1 --2- 172.21.15.107:0/1530816647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0070b00 0x7f7af01514e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7af481a640 1 --2- 172.21.15.107:0/1530816647 >> v2:172.21.15.110:3300/0 conn(0x7f7af014eb70 0x7f7af00c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7af481a640 1 -- 172.21.15.107:0/1530816647 >> 172.21.15.107:0/1530816647 conn(0x7f7af00c39a0 msgr2=0x7f7af00c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:50.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7af481a640 1 -- 172.21.15.107:0/1530816647 shutdown_connections 2023-10-25T00:18:50.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.755+0000 7f7af481a640 1 -- 172.21.15.107:0/1530816647 wait complete. 2023-10-25T00:18:50.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7af481a640 1 Processor -- start 2023-10-25T00:18:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7af481a640 1 -- start start 2023-10-25T00:18:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7af481a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af0070b00 0x7f7af00ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7af481a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7af014eb70 0x7f7af00a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7af481a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0151a20 0x7f7af006b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7af481a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af00ab830 con 0x7f7af0070b00 2023-10-25T00:18:50.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7af481a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af006a8f0 con 0x7f7af014eb70 2023-10-25T00:18:50.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7af481a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af006abd0 con 0x7f7af0151a20 2023-10-25T00:18:50.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7aee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af0070b00 0x7f7af00ab2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7aee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af0070b00 0x7f7af00ab2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58484/0 (socket says 172.21.15.107:58484) 2023-10-25T00:18:50.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7aee59c640 1 -- 172.21.15.107:0/524680759 learned_addr learned my addr 172.21.15.107:0/524680759 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7aedd9b640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7af014eb70 0x7f7af00a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7aeed9d640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0151a20 0x7f7af006b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7aee59c640 1 -- 172.21.15.107:0/524680759 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0151a20 msgr2=0x7f7af006b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7aee59c640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0151a20 0x7f7af006b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7aee59c640 1 -- 172.21.15.107:0/524680759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7af014eb70 msgr2=0x7f7af00a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7aee59c640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7af014eb70 0x7f7af00a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7aee59c640 1 -- 172.21.15.107:0/524680759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae8009a70 con 0x7f7af0070b00 2023-10-25T00:18:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7aee59c640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af0070b00 0x7f7af00ab2f0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000c7f0 tx=0x7f7ae000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7adeffd640 1 -- 172.21.15.107:0/524680759 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0004220 con 0x7f7af0070b00 2023-10-25T00:18:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7af481a640 1 -- 172.21.15.107:0/524680759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae40094d0 con 0x7f7af0070b00 2023-10-25T00:18:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7adeffd640 1 -- 172.21.15.107:0/524680759 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae0004380 con 0x7f7af0070b00 2023-10-25T00:18:50.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7adeffd640 1 -- 172.21.15.107:0/524680759 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0003820 con 0x7f7af0070b00 2023-10-25T00:18:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.759+0000 7f7af481a640 1 -- 172.21.15.107:0/524680759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af00abf60 con 0x7f7af0070b00 2023-10-25T00:18:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.763+0000 7f7af481a640 1 -- 172.21.15.107:0/524680759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af00746e0 con 0x7f7af0070b00 2023-10-25T00:18:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.763+0000 7f7adeffd640 1 -- 172.21.15.107:0/524680759 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7ae0003980 con 0x7f7af0070b00 2023-10-25T00:18:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.763+0000 7f7adeffd640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad0065e60 0x7f7ad0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.763+0000 7f7adeffd640 1 -- 172.21.15.107:0/524680759 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 10315+0+0 (secure 0 0 0) 0x7f7ae00849a0 con 0x7f7af0070b00 2023-10-25T00:18:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.763+0000 7f7aedd9b640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad0065e60 0x7f7ad0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.763+0000 7f7aedd9b640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad0065e60 0x7f7ad0068320 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40af000 tx=0x7f7ae40a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:50.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.767+0000 7f7adeffd640 1 -- 172.21.15.107:0/524680759 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7ae0050e80 con 0x7f7af0070b00 2023-10-25T00:18:50.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.899+0000 7f64ad610640 1 Processor -- start 2023-10-25T00:18:50.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.899+0000 7f64ad610640 1 -- start start 2023-10-25T00:18:50.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.899+0000 7f64ad610640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64a814eb70 0x7f64a80c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.899+0000 7f64ad610640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f64a8070b00 0x7f64a81514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.899+0000 7f64ad610640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a80c22f0 con 0x7f64a8151a20 2023-10-25T00:18:50.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.899+0000 7f64ad610640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a8071700 con 0x7f64a814eb70 2023-10-25T00:18:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.899+0000 7f64ad610640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64a80c25d0 con 0x7f64a8070b00 2023-10-25T00:18:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.899+0000 7f64a6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64a814eb70 0x7f64a80c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.899+0000 7f64a6ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f64a814eb70 0x7f64a80c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58500/0 (socket says 172.21.15.107:58500) 2023-10-25T00:18:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.899+0000 7f64a6ffd640 1 -- 172.21.15.107:0/2379615947 learned_addr learned my addr 172.21.15.107:0/2379615947 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:50.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.899+0000 7f64a67fc640 1 --2- 172.21.15.107:0/2379615947 >> v2:172.21.15.110:3300/0 conn(0x7f64a8070b00 0x7f64a81514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64a67fc640 1 -- 172.21.15.107:0/2379615947 >> v2:172.21.15.107:3300/0 conn(0x7f64a814eb70 msgr2=0x7f64a80c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64a67fc640 1 --2- 172.21.15.107:0/2379615947 >> v2:172.21.15.107:3300/0 conn(0x7f64a814eb70 0x7f64a80c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64a67fc640 1 -- 172.21.15.107:0/2379615947 >> v1:172.21.15.110:6789/0 conn(0x7f64a8151a20 legacy=0x7f64a806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64a67fc640 1 -- 172.21.15.107:0/2379615947 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a80c2be0 con 0x7f64a8070b00 2023-10-25T00:18:50.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64a67fc640 1 --2- 172.21.15.107:0/2379615947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64a8070b00 0x7f64a81514e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f64a80c1e70 tx=0x7f649c01c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa2e18a0c7effba5 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:50.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64a5ffb640 1 -- 172.21.15.107:0/2379615947 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c0ab600 con 0x7f64a8070b00 2023-10-25T00:18:50.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64a5ffb640 1 -- 172.21.15.107:0/2379615947 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649c0ab760 con 0x7f64a8070b00 2023-10-25T00:18:50.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64a5ffb640 1 -- 172.21.15.107:0/2379615947 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c0abaf0 con 0x7f64a8070b00 2023-10-25T00:18:50.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64ad610640 1 -- 172.21.15.107:0/2379615947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64a8070b00 msgr2=0x7f64a81514e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64ad610640 1 --2- 172.21.15.107:0/2379615947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64a8070b00 0x7f64a81514e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f64a80c1e70 tx=0x7f649c01c460 comp rx=0 tx=0).stop 2023-10-25T00:18:50.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64ad610640 1 -- 172.21.15.107:0/2379615947 shutdown_connections 2023-10-25T00:18:50.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64ad610640 1 --2- 172.21.15.107:0/2379615947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64a8070b00 0x7f64a81514e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64ad610640 1 --2- 172.21.15.107:0/2379615947 >> v2:172.21.15.107:3300/0 conn(0x7f64a814eb70 0x7f64a80c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64ad610640 1 -- 172.21.15.107:0/2379615947 >> 172.21.15.107:0/2379615947 conn(0x7f64a80c39a0 msgr2=0x7f64a80c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:50.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.903+0000 7f64ad610640 1 -- 172.21.15.107:0/2379615947 shutdown_connections 2023-10-25T00:18:50.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64ad610640 1 -- 172.21.15.107:0/2379615947 wait complete. 2023-10-25T00:18:50.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64ad610640 1 Processor -- start 2023-10-25T00:18:50.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64ad610640 1 -- start start 2023-10-25T00:18:50.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64ad610640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64a8070b00 0x7f64a814b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64ad610640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64a814eb70 0x7f64a8153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64ad610640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64a8151a20 0x7f64a8154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64ad610640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8155900 con 0x7f64a814eb70 2023-10-25T00:18:50.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64ad610640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8153640 con 0x7f64a8070b00 2023-10-25T00:18:50.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64ad610640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a81ee700 con 0x7f64a8151a20 2023-10-25T00:18:50.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64a67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64a814eb70 0x7f64a8153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64a8151a20 0x7f64a8154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64a67fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64a814eb70 0x7f64a8153ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58512/0 (socket says 172.21.15.107:58512) 2023-10-25T00:18:50.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64a8151a20 0x7f64a8154010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:36130/0 (socket says 172.21.15.107:36130) 2023-10-25T00:18:50.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64a67fc640 1 -- 172.21.15.107:0/142932534 learned_addr learned my addr 172.21.15.107:0/142932534 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:50.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.907+0000 7f64a6ffd640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64a8070b00 0x7f64a814b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f64a67fc640 1 -- 172.21.15.107:0/142932534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64a8151a20 msgr2=0x7f64a8154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f64a67fc640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64a8151a20 0x7f64a8154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f64a67fc640 1 -- 172.21.15.107:0/142932534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64a8070b00 msgr2=0x7f64a814b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:50.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f64a67fc640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64a8070b00 0x7f64a814b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:50.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f64a67fc640 1 -- 172.21.15.107:0/142932534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c009420 con 0x7f64a814eb70 2023-10-25T00:18:50.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f64a67fc640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64a814eb70 0x7f64a8153ad0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f649c019070 tx=0x7f649c01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:50.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f648f7fe640 1 -- 172.21.15.107:0/142932534 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c0b6720 con 0x7f64a814eb70 2023-10-25T00:18:50.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f648f7fe640 1 -- 172.21.15.107:0/142932534 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649c0b68a0 con 0x7f64a814eb70 2023-10-25T00:18:50.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f648f7fe640 1 -- 172.21.15.107:0/142932534 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c0b6c50 con 0x7f64a814eb70 2023-10-25T00:18:50.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f64ad610640 1 -- 172.21.15.107:0/142932534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a8153310 con 0x7f64a814eb70 2023-10-25T00:18:50.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f64ad610640 1 -- 172.21.15.107:0/142932534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a806cfb0 con 0x7f64a814eb70 2023-10-25T00:18:50.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f64a77fe640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64a8151a20 0x7f64a8154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:18:50.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f64ad610640 1 -- 172.21.15.107:0/142932534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a81537d0 con 0x7f64a814eb70 2023-10-25T00:18:50.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.911+0000 7f648f7fe640 1 -- 172.21.15.107:0/142932534 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f649c0c83b0 con 0x7f64a814eb70 2023-10-25T00:18:50.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.915+0000 7f648f7fe640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6494065eb0 0x7f6494068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:50.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.915+0000 7f648f7fe640 1 -- 172.21.15.107:0/142932534 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 10315+0+0 (secure 0 0 0) 0x7f649c0f3020 con 0x7f64a814eb70 2023-10-25T00:18:50.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.915+0000 7f64a6ffd640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6494065eb0 0x7f6494068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:50.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.915+0000 7f64a6ffd640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6494065eb0 0x7f6494068370 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6498009ba0 tx=0x7f6498009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:50.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:50.923+0000 7f648f7fe640 1 -- 172.21.15.107:0/142932534 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f649c0efaf0 con 0x7f64a814eb70 2023-10-25T00:18:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.095+0000 7f7af481a640 1 -- 172.21.15.107:0/524680759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7af00c1a50 con 0x7f7af0070b00 2023-10-25T00:18:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.095+0000 7f7adeffd640 1 -- 172.21.15.107:0/524680759 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+5012 (secure 0 0 0) 0x7f7ae001e020 con 0x7f7af0070b00 2023-10-25T00:18:51.102 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:51.103 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":201,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:49.119150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:51.104 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 201 2023-10-25T00:18:51.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 -- 172.21.15.107:0/524680759 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad0065e60 msgr2=0x7f7ad0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:51.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad0065e60 0x7f7ad0068320 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f7ae40af000 tx=0x7f7ae40a7000 comp rx=0 tx=0).stop 2023-10-25T00:18:51.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 -- 172.21.15.107:0/524680759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af0070b00 msgr2=0x7f7af00ab2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:51.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af0070b00 0x7f7af00ab2f0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000c7f0 tx=0x7f7ae000cbf0 comp rx=0 tx=0).stop 2023-10-25T00:18:51.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 -- 172.21.15.107:0/524680759 shutdown_connections 2023-10-25T00:18:51.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7ad0065e60 0x7f7ad0068320 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:51.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7af0151a20 0x7f7af006b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:51.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7af014eb70 0x7f7af00a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:51.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 --2- 172.21.15.107:0/524680759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7af0070b00 0x7f7af00ab2f0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:51.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 -- 172.21.15.107:0/524680759 >> 172.21.15.107:0/524680759 conn(0x7f7af00c39a0 msgr2=0x7f7af014dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 -- 172.21.15.107:0/524680759 shutdown_connections 2023-10-25T00:18:51.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.103+0000 7f7adcff9640 1 -- 172.21.15.107:0/524680759 wait complete. 2023-10-25T00:18:51.128 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:18:51.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.255+0000 7f64ad610640 1 -- 172.21.15.107:0/142932534 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64a80746e0 con 0x7f64a814eb70 2023-10-25T00:18:51.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.255+0000 7f648f7fe640 1 -- 172.21.15.107:0/142932534 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f649c0efc90 con 0x7f64a814eb70 2023-10-25T00:18:51.257 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:51.257 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":202,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:51.137001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:51.258 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 202 2023-10-25T00:18:51.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 -- 172.21.15.107:0/142932534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6494065eb0 msgr2=0x7f6494068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:51.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6494065eb0 0x7f6494068370 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6498009ba0 tx=0x7f6498009240 comp rx=0 tx=0).stop 2023-10-25T00:18:51.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 -- 172.21.15.107:0/142932534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64a814eb70 msgr2=0x7f64a8153ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:51.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64a814eb70 0x7f64a8153ad0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f649c019070 tx=0x7f649c01aca0 comp rx=0 tx=0).stop 2023-10-25T00:18:51.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 -- 172.21.15.107:0/142932534 shutdown_connections 2023-10-25T00:18:51.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6494065eb0 0x7f6494068370 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:51.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f64a8151a20 0x7f64a8154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:51.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64a814eb70 0x7f64a8153ad0 secure :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f649c019070 tx=0x7f649c01aca0 comp rx=0 tx=0).stop 2023-10-25T00:18:51.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 --2- 172.21.15.107:0/142932534 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64a8070b00 0x7f64a814b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:51.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 -- 172.21.15.107:0/142932534 >> 172.21.15.107:0/142932534 conn(0x7f64a80c39a0 msgr2=0x7f64a814ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:51.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 -- 172.21.15.107:0/142932534 shutdown_connections 2023-10-25T00:18:51.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:51.259+0000 7f64ad610640 1 -- 172.21.15.107:0/142932534 wait complete. 2023-10-25T00:18:51.282 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 202, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 202, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7258": { "addr": "172.21.15.110:6834/3771564389", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 3771564389, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 3771564389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7258, "incarnation": 201, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:18:51.137001+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7258 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 199, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7213": { "addr": "172.21.15.107:6835/2970408860", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2970408860, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 2970408860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7213, "incarnation": 196, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:17:43.371756+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7213 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2437148", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2437148, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2437148, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 190, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6443, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:18:53.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:53.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:53.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbebe95640 1 Processor -- start 2023-10-25T00:18:53.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbebe95640 1 -- start start 2023-10-25T00:18:53.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbebe95640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdbe414c470 0x7fdbe414c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbebe95640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdbe406f490 0x7fdbe4071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbe9430640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdbe406f490 0x7fdbe4071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:53.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbe9430640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fdbe406f490 0x7fdbe4071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38016/0 (socket says 172.21.15.107:38016) 2023-10-25T00:18:53.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbebe95640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe4155cd0 con 0x7fdbe4071fc0 2023-10-25T00:18:53.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbebe95640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe41526e0 con 0x7fdbe406f490 2023-10-25T00:18:53.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbebe95640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe40c4380 con 0x7fdbe414c470 2023-10-25T00:18:53.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbe9430640 1 -- 172.21.15.107:0/1643380423 learned_addr learned my addr 172.21.15.107:0/1643380423 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:53.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbe9430640 1 -- 172.21.15.107:0/1643380423 >> v2:172.21.15.110:3300/0 conn(0x7fdbe414c470 msgr2=0x7fdbe414c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:53.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbe9430640 1 --2- 172.21.15.107:0/1643380423 >> v2:172.21.15.110:3300/0 conn(0x7fdbe414c470 0x7fdbe414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbe9430640 1 -- 172.21.15.107:0/1643380423 >> v1:172.21.15.110:6789/0 conn(0x7fdbe4071fc0 legacy=0x7fdbe4152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:53.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.287+0000 7fdbe9430640 1 -- 172.21.15.107:0/1643380423 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe40c48b0 con 0x7fdbe406f490 2023-10-25T00:18:53.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbe9430640 1 --2- 172.21.15.107:0/1643380423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbe406f490 0x7fdbe4071a80 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fdbe406a8e0 tx=0x7fdbd401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d767aa59502677d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:53.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbe8c2f640 1 -- 172.21.15.107:0/1643380423 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40ab600 con 0x7fdbe406f490 2023-10-25T00:18:53.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbe8c2f640 1 -- 172.21.15.107:0/1643380423 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd40a7050 con 0x7fdbe406f490 2023-10-25T00:18:53.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbe8c2f640 1 -- 172.21.15.107:0/1643380423 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40ab980 con 0x7fdbe406f490 2023-10-25T00:18:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbebe95640 1 -- 172.21.15.107:0/1643380423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbe406f490 msgr2=0x7fdbe4071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:53.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbebe95640 1 --2- 172.21.15.107:0/1643380423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbe406f490 0x7fdbe4071a80 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fdbe406a8e0 tx=0x7fdbd401c3a0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbebe95640 1 -- 172.21.15.107:0/1643380423 shutdown_connections 2023-10-25T00:18:53.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbebe95640 1 --2- 172.21.15.107:0/1643380423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbe406f490 0x7fdbe4071a80 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbebe95640 1 --2- 172.21.15.107:0/1643380423 >> v2:172.21.15.110:3300/0 conn(0x7fdbe414c470 0x7fdbe414c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbebe95640 1 -- 172.21.15.107:0/1643380423 >> 172.21.15.107:0/1643380423 conn(0x7fdbe4070f60 msgr2=0x7fdbe4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:53.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbebe95640 1 -- 172.21.15.107:0/1643380423 shutdown_connections 2023-10-25T00:18:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.291+0000 7fdbebe95640 1 -- 172.21.15.107:0/1643380423 wait complete. 2023-10-25T00:18:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbebe95640 1 Processor -- start 2023-10-25T00:18:53.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbebe95640 1 -- start start 2023-10-25T00:18:53.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbebe95640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbe406f490 0x7fdbe40c39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbebe95640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbe4071fc0 0x7fdbe4146d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbebe95640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbe414c470 0x7fdbe4147280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbebe95640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe41eb660 con 0x7fdbe414c470 2023-10-25T00:18:53.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbebe95640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe4147d30 con 0x7fdbe406f490 2023-10-25T00:18:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbebe95640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe41443d0 con 0x7fdbe4071fc0 2023-10-25T00:18:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbe9430640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbe4071fc0 0x7fdbe4146d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbe9430640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbe4071fc0 0x7fdbe4146d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:57906/0 (socket says 172.21.15.107:57906) 2023-10-25T00:18:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbe9430640 1 -- 172.21.15.107:0/2693630994 learned_addr learned my addr 172.21.15.107:0/2693630994 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:53.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbea432640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbe414c470 0x7fdbe4147280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbe9430640 1 -- 172.21.15.107:0/2693630994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbe406f490 msgr2=0x7fdbe40c39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbe9430640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbe406f490 0x7fdbe40c39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbe9430640 1 -- 172.21.15.107:0/2693630994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbe414c470 msgr2=0x7fdbe4147280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbe9430640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbe414c470 0x7fdbe4147280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbe9430640 1 -- 172.21.15.107:0/2693630994 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd40094d0 con 0x7fdbe4071fc0 2023-10-25T00:18:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.295+0000 7fdbe9430640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbe4071fc0 0x7fdbe4146d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdbd401aa50 tx=0x7fdbd401abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.299+0000 7fdbda7fc640 1 -- 172.21.15.107:0/2693630994 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40b6830 con 0x7fdbe4071fc0 2023-10-25T00:18:53.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.299+0000 7fdbda7fc640 1 -- 172.21.15.107:0/2693630994 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd40b69b0 con 0x7fdbe4071fc0 2023-10-25T00:18:53.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.299+0000 7fdbebe95640 1 -- 172.21.15.107:0/2693630994 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe4144670 con 0x7fdbe4071fc0 2023-10-25T00:18:53.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.299+0000 7fdbebe95640 1 -- 172.21.15.107:0/2693630994 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe40c3f00 con 0x7fdbe4071fc0 2023-10-25T00:18:53.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.299+0000 7fdbda7fc640 1 -- 172.21.15.107:0/2693630994 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40b6d60 con 0x7fdbe4071fc0 2023-10-25T00:18:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.299+0000 7fdbda7fc640 1 -- 172.21.15.107:0/2693630994 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fdbd40b2040 con 0x7fdbe4071fc0 2023-10-25T00:18:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.299+0000 7fdbda7fc640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbc8065e60 0x7fdbc8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.299+0000 7fdbda7fc640 1 -- 172.21.15.107:0/2693630994 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 10315+0+0 (secure 0 0 0) 0x7fdbd411f090 con 0x7fdbe4071fc0 2023-10-25T00:18:53.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.299+0000 7fdbe9c31640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbc8065e60 0x7fdbc8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:53.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.303+0000 7fdbe9c31640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbc8065e60 0x7fdbc8068320 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4153b50 tx=0x7fdbe0006c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:53.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.303+0000 7fdbebe95640 1 -- 172.21.15.107:0/2693630994 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbcc005260 con 0x7fdbe4071fc0 2023-10-25T00:18:53.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.311+0000 7fdbda7fc640 1 -- 172.21.15.107:0/2693630994 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fdbd40eb640 con 0x7fdbe4071fc0 2023-10-25T00:18:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.439+0000 7f1867d31640 1 Processor -- start 2023-10-25T00:18:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.439+0000 7f1867d31640 1 -- start start 2023-10-25T00:18:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f1867d31640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f18600aa990 0x7f1860165460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f1867d31640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f186008b9e0 0x7f18600c1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f1867d31640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1860069d40 con 0x7f18600c1860 2023-10-25T00:18:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f1867d31640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186006beb0 con 0x7f186008b9e0 2023-10-25T00:18:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f1867d31640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186006c570 con 0x7f18600aa990 2023-10-25T00:18:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f18652cc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f18600aa990 0x7f1860165460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:53.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f18652cc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f18600aa990 0x7f1860165460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:50312/0 (socket says 172.21.15.107:50312) 2023-10-25T00:18:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f18652cc640 1 -- 172.21.15.107:0/1410180600 learned_addr learned my addr 172.21.15.107:0/1410180600 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:53.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f18652cc640 1 -- 172.21.15.107:0/1410180600 >> v2:172.21.15.107:3300/0 conn(0x7f186008b9e0 msgr2=0x7f18600c1c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-25T00:18:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f18662ce640 1 --2- 172.21.15.107:0/1410180600 >> v2:172.21.15.107:3300/0 conn(0x7f186008b9e0 0x7f18600c1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f18652cc640 1 --2- 172.21.15.107:0/1410180600 >> v2:172.21.15.107:3300/0 conn(0x7f186008b9e0 0x7f18600c1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f18652cc640 1 -- 172.21.15.107:0/1410180600 >> v1:172.21.15.110:6789/0 conn(0x7f18600c1860 legacy=0x7f1860164d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:18:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f18652cc640 1 -- 172.21.15.107:0/1410180600 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1860161f70 con 0x7f18600aa990 2023-10-25T00:18:53.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.443+0000 7f18662ce640 1 --2- 172.21.15.107:0/1410180600 >> v2:172.21.15.107:3300/0 conn(0x7f186008b9e0 0x7f18600c1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:18:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.447+0000 7f18652cc640 1 --2- 172.21.15.107:0/1410180600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18600aa990 0x7f1860165460 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f186006d0a0 tx=0x7f185001c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddf54e72b010fa68 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.447+0000 7f1864acb640 1 -- 172.21.15.107:0/1410180600 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18500a5790 con 0x7f18600aa990 2023-10-25T00:18:53.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.447+0000 7f1864acb640 1 -- 172.21.15.107:0/1410180600 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18500a58f0 con 0x7f18600aa990 2023-10-25T00:18:53.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.447+0000 7f1864acb640 1 -- 172.21.15.107:0/1410180600 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18500a5c80 con 0x7f18600aa990 2023-10-25T00:18:53.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.451+0000 7f1867d31640 1 -- 172.21.15.107:0/1410180600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18600aa990 msgr2=0x7f1860165460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:53.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.451+0000 7f1867d31640 1 --2- 172.21.15.107:0/1410180600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18600aa990 0x7f1860165460 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f186006d0a0 tx=0x7f185001c8d0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.451+0000 7f1867d31640 1 -- 172.21.15.107:0/1410180600 shutdown_connections 2023-10-25T00:18:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.451+0000 7f1867d31640 1 --2- 172.21.15.107:0/1410180600 >> v2:172.21.15.107:3300/0 conn(0x7f186008b9e0 0x7f18600c1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.451+0000 7f1867d31640 1 --2- 172.21.15.107:0/1410180600 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18600aa990 0x7f1860165460 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.451+0000 7f1867d31640 1 -- 172.21.15.107:0/1410180600 >> 172.21.15.107:0/1410180600 conn(0x7f18600c2950 msgr2=0x7f18600c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.451+0000 7f1867d31640 1 -- 172.21.15.107:0/1410180600 shutdown_connections 2023-10-25T00:18:53.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.451+0000 7f1867d31640 1 -- 172.21.15.107:0/1410180600 wait complete. 2023-10-25T00:18:53.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f1867d31640 1 Processor -- start 2023-10-25T00:18:53.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f1867d31640 1 -- start start 2023-10-25T00:18:53.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f1867d31640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f186008b9e0 0x7f1860160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f1867d31640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18600aa990 0x7f186015e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f1867d31640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18600c1860 0x7f18602025a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f1867d31640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18602020a0 con 0x7f18600c1860 2023-10-25T00:18:53.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f1867d31640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1860205b10 con 0x7f18600aa990 2023-10-25T00:18:53.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f1867d31640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1860205df0 con 0x7f186008b9e0 2023-10-25T00:18:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f18662ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18600c1860 0x7f18602025a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f18662ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18600c1860 0x7f18602025a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38046/0 (socket says 172.21.15.107:38046) 2023-10-25T00:18:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f1865acd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f186008b9e0 0x7f1860160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:53.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f18662ce640 1 -- 172.21.15.107:0/3851116977 learned_addr learned my addr 172.21.15.107:0/3851116977 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f18662ce640 1 -- 172.21.15.107:0/3851116977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f186008b9e0 msgr2=0x7f1860160940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:53.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f18662ce640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f186008b9e0 0x7f1860160940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f18662ce640 1 -- 172.21.15.107:0/3851116977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18600aa990 msgr2=0x7f186015e290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:18:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f18662ce640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18600aa990 0x7f186015e290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f18662ce640 1 -- 172.21.15.107:0/3851116977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1850009b30 con 0x7f18600c1860 2023-10-25T00:18:53.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f18662ce640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18600c1860 0x7f18602025a0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f185800c7f0 tx=0x7f185800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f185e7fc640 1 -- 172.21.15.107:0/3851116977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858004230 con 0x7f18600c1860 2023-10-25T00:18:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f1867d31640 1 -- 172.21.15.107:0/3851116977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18602060d0 con 0x7f18600c1860 2023-10-25T00:18:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f1867d31640 1 -- 172.21.15.107:0/3851116977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186015e7d0 con 0x7f18600c1860 2023-10-25T00:18:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f185e7fc640 1 -- 172.21.15.107:0/3851116977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1858004390 con 0x7f18600c1860 2023-10-25T00:18:53.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.455+0000 7f185e7fc640 1 -- 172.21.15.107:0/3851116977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858003860 con 0x7f18600c1860 2023-10-25T00:18:53.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.459+0000 7f1867d31640 1 -- 172.21.15.107:0/3851116977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1840005260 con 0x7f18600c1860 2023-10-25T00:18:53.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.459+0000 7f185e7fc640 1 -- 172.21.15.107:0/3851116977 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1858003c90 con 0x7f18600c1860 2023-10-25T00:18:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.459+0000 7f185e7fc640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f184c065f30 0x7f184c0683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.459+0000 7f185e7fc640 1 -- 172.21.15.107:0/3851116977 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 10315+0+0 (secure 0 0 0) 0x7f1858085370 con 0x7f18600c1860 2023-10-25T00:18:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.459+0000 7f1865acd640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f184c065f30 0x7f184c0683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.463+0000 7f1865acd640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f184c065f30 0x7f184c0683f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1854009980 tx=0x7f18540090e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:53.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.467+0000 7f185e7fc640 1 -- 172.21.15.107:0/3851116977 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f18580518f0 con 0x7f18600c1860 2023-10-25T00:18:53.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.607+0000 7fdbebe95640 1 -- 172.21.15.107:0/2693630994 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbcc005a90 con 0x7fdbe4071fc0 2023-10-25T00:18:53.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.611+0000 7fdbda7fc640 1 -- 172.21.15.107:0/2693630994 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+5013 (secure 0 0 0) 0x7fdbd40b0070 con 0x7fdbe4071fc0 2023-10-25T00:18:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:53.613 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:53.613 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 204 2023-10-25T00:18:53.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.611+0000 7fdbebe95640 1 -- 172.21.15.107:0/2693630994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbc8065e60 msgr2=0x7fdbc8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:53.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.611+0000 7fdbebe95640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbc8065e60 0x7fdbc8068320 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4153b50 tx=0x7fdbe0006c00 comp rx=0 tx=0).stop 2023-10-25T00:18:53.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.611+0000 7fdbebe95640 1 -- 172.21.15.107:0/2693630994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbe4071fc0 msgr2=0x7fdbe4146d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:53.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.611+0000 7fdbebe95640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbe4071fc0 0x7fdbe4146d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdbd401aa50 tx=0x7fdbd401abe0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.611+0000 7fdbebe95640 1 -- 172.21.15.107:0/2693630994 shutdown_connections 2023-10-25T00:18:53.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.611+0000 7fdbebe95640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbe414c470 0x7fdbe4147280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.611+0000 7fdbebe95640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fdbc8065e60 0x7fdbc8068320 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.611+0000 7fdbebe95640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbe4071fc0 0x7fdbe4146d40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.611+0000 7fdbebe95640 1 --2- 172.21.15.107:0/2693630994 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbe406f490 0x7fdbe40c39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.611+0000 7fdbebe95640 1 -- 172.21.15.107:0/2693630994 >> 172.21.15.107:0/2693630994 conn(0x7fdbe4070f60 msgr2=0x7fdbe40c20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.615+0000 7fdbebe95640 1 -- 172.21.15.107:0/2693630994 shutdown_connections 2023-10-25T00:18:53.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.615+0000 7fdbebe95640 1 -- 172.21.15.107:0/2693630994 wait complete. 2023-10-25T00:18:53.636 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:18:53.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.743+0000 7f1867d31640 1 -- 172.21.15.107:0/3851116977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18400057e0 con 0x7f18600c1860 2023-10-25T00:18:53.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.743+0000 7f185e7fc640 1 -- 172.21.15.107:0/3851116977 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f1858015090 con 0x7f18600c1860 2023-10-25T00:18:53.745 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:53.745 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:53.746 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 204 2023-10-25T00:18:53.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 -- 172.21.15.107:0/3851116977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f184c065f30 msgr2=0x7f184c0683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:53.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f184c065f30 0x7f184c0683f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f1854009980 tx=0x7f18540090e0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 -- 172.21.15.107:0/3851116977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18600c1860 msgr2=0x7f18602025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:53.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18600c1860 0x7f18602025a0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f185800c7f0 tx=0x7f185800ccb0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 -- 172.21.15.107:0/3851116977 shutdown_connections 2023-10-25T00:18:53.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18600c1860 0x7f18602025a0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f18600aa990 0x7f186015e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f184c065f30 0x7f184c0683f0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 --2- 172.21.15.107:0/3851116977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f186008b9e0 0x7f1860160940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:53.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 -- 172.21.15.107:0/3851116977 >> 172.21.15.107:0/3851116977 conn(0x7f18600c2950 msgr2=0x7f18600c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:53.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 -- 172.21.15.107:0/3851116977 shutdown_connections 2023-10-25T00:18:53.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:53.747+0000 7f1867d31640 1 -- 172.21.15.107:0/3851116977 wait complete. 2023-10-25T00:18:53.769 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=8418 2023-10-25T00:18:53.769 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.a 2023-10-25T00:18:54.771 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2023-10-25T00:18:54.771 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:18:54.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:18:54.774 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:18:54.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:54.842 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:18:54.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.891+0000 7f5ed6b6c640 1 Processor -- start 2023-10-25T00:18:54.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.891+0000 7f5ed6b6c640 1 -- start start 2023-10-25T00:18:54.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.891+0000 7f5ed6b6c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed00c8100 0x7f5ed00abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:54.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.891+0000 7f5ed6b6c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed0155680 con 0x7f5ed0152040 2023-10-25T00:18:54.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.891+0000 7f5ed6b6c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed0154a70 con 0x7f5ed006a4b0 2023-10-25T00:18:54.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.891+0000 7f5ed6b6c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed0154d50 con 0x7f5ed00c8100 2023-10-25T00:18:54.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.891+0000 7f5ed4908640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5ed0152040 0x7f5ed00c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44770/0 (socket says 172.21.15.107:44770) 2023-10-25T00:18:54.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.891+0000 7f5ed4908640 1 -- 172.21.15.107:0/2527305340 learned_addr learned my addr 172.21.15.107:0/2527305340 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:18:54.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.891+0000 7f5ecffff640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed00c8100 0x7f5ed00abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:54.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.891+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910961673 0 0) 0x7f5ed0155680 con 0x7f5ed0152040 2023-10-25T00:18:54.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.891+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb8003530 con 0x7f5ed0152040 2023-10-25T00:18:54.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 383367438 0 0) 0x7f5eb8003530 con 0x7f5ed0152040 2023-10-25T00:18:54.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed00c8100 msgr2=0x7f5ed00abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:54.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed00c8100 0x7f5ed00abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:54.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 >> v1:172.21.15.110:6789/0 conn(0x7f5ed006a4b0 legacy=0x7f5ed00ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:54.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed0155030 con 0x7f5ed0152040 2023-10-25T00:18:54.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5ec0001680 con 0x7f5ed0152040 2023-10-25T00:18:54.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 >> v1:172.21.15.107:6789/0 conn(0x7f5ed0152040 legacy=0x7f5ed00c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:54.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed00c8100 0x7f5eb80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:54.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5eb8003f30 0x7f5eb80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:54.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 0x7f5eb8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:54.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed0154d50 con 0x7f5eb8004d10 2023-10-25T00:18:54.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb800bb20 con 0x7f5eb8003f30 2023-10-25T00:18:54.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb800bce0 con 0x7f5ed00c8100 2023-10-25T00:18:54.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed5109640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 0x7f5eb8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:54.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecffff640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5eb8003f30 0x7f5eb80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:54.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed4908640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed00c8100 0x7f5eb80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:54.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed5109640 1 -- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed00c8100 msgr2=0x7f5eb80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:54.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed5109640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed00c8100 0x7f5eb80039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:54.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed5109640 1 -- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5eb8003f30 msgr2=0x7f5eb80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:54.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed5109640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5eb8003f30 0x7f5eb80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:54.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed5109640 1 -- 172.21.15.107:0/2527305340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb8003530 con 0x7f5eb8004d10 2023-10-25T00:18:54.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecffff640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5eb8003f30 0x7f5eb80047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:18:54.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed5109640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 0x7f5eb8005190 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8001e70 tx=0x7f5ec801c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746da55b43515e9f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:54.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8003b80 con 0x7f5eb8004d10 2023-10-25T00:18:54.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec8003ce0 con 0x7f5eb8004d10 2023-10-25T00:18:54.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ecf7fe640 1 -- 172.21.15.107:0/2527305340 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec80052c0 con 0x7f5eb8004d10 2023-10-25T00:18:54.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/2527305340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 msgr2=0x7f5eb8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:54.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed6b6c640 1 --2- 172.21.15.107:0/2527305340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 0x7f5eb8005190 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8001e70 tx=0x7f5ec801c760 comp rx=0 tx=0).stop 2023-10-25T00:18:54.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed5109640 1 -- 172.21.15.107:0/2527305340 reap_dead start 2023-10-25T00:18:54.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/2527305340 shutdown_connections 2023-10-25T00:18:54.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/2527305340 >> 172.21.15.107:0/2527305340 conn(0x7f5ed00ca200 msgr2=0x7f5ed00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:54.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/2527305340 shutdown_connections 2023-10-25T00:18:54.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.895+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/2527305340 wait complete. 2023-10-25T00:18:54.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed6b6c640 1 Processor -- start 2023-10-25T00:18:54.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed6b6c640 1 -- start start 2023-10-25T00:18:54.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed6b6c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 0x7f5ed00c3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:54.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed6b6c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed01ddee0 0x7f5ed0153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:54.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed6b6c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed01dea50 0x7f5ed01e9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:54.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed6b6c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00c2220 con 0x7f5eb8004d10 2023-10-25T00:18:54.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed6b6c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed01e9630 con 0x7f5ed01ddee0 2023-10-25T00:18:54.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed6b6c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00abfa0 con 0x7f5ed01dea50 2023-10-25T00:18:54.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed4908640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 0x7f5ed00c3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:54.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed4908640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 0x7f5ed00c3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38058/0 (socket says 172.21.15.107:38058) 2023-10-25T00:18:54.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed4908640 1 -- 172.21.15.107:0/868162044 learned_addr learned my addr 172.21.15.107:0/868162044 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:54.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed4908640 1 -- 172.21.15.107:0/868162044 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed01dea50 msgr2=0x7f5ed01e9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:54.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed5109640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed01dea50 0x7f5ed01e9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:54.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed4908640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed01dea50 0x7f5ed01e9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:54.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed4908640 1 -- 172.21.15.107:0/868162044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed01ddee0 msgr2=0x7f5ed0153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:54.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ecffff640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed01ddee0 0x7f5ed0153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:54.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed4908640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed01ddee0 0x7f5ed0153960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:54.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed4908640 1 -- 172.21.15.107:0/868162044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed00ac280 con 0x7f5eb8004d10 2023-10-25T00:18:54.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed4908640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 0x7f5ed00c3310 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0000f70 tx=0x7f5ec000e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:54.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ecd7fa640 1 -- 172.21.15.107:0/868162044 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0010ea0 con 0x7f5eb8004d10 2023-10-25T00:18:54.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ecd7fa640 1 -- 172.21.15.107:0/868162044 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec00143a0 con 0x7f5eb8004d10 2023-10-25T00:18:54.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/868162044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed00ac560 con 0x7f5eb8004d10 2023-10-25T00:18:54.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ecd7fa640 1 -- 172.21.15.107:0/868162044 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0014710 con 0x7f5eb8004d10 2023-10-25T00:18:54.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/868162044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed0153ea0 con 0x7f5eb8004d10 2023-10-25T00:18:54.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/868162044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed01e9a80 con 0x7f5eb8004d10 2023-10-25T00:18:54.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.899+0000 7f5ecd7fa640 1 -- 172.21.15.107:0/868162044 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5ec0014870 con 0x7f5eb8004d10 2023-10-25T00:18:54.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.903+0000 7f5ecd7fa640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ebc065e60 0x7f5ebc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:54.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.903+0000 7f5ecd7fa640 1 -- 172.21.15.107:0/868162044 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 10315+0+0 (secure 0 0 0) 0x7f5ec0087a40 con 0x7f5eb8004d10 2023-10-25T00:18:54.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.903+0000 7f5ecffff640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ebc065e60 0x7f5ebc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:54.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.903+0000 7f5ecffff640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ebc065e60 0x7f5ebc068320 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00c3850 tx=0x7f5ec4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:54.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:54.907+0000 7f5ecd7fa640 1 -- 172.21.15.107:0/868162044 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5ec0089e60 con 0x7f5eb8004d10 2023-10-25T00:18:55.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.095+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/868162044 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ed00746e0 con 0x7f5eb8004d10 2023-10-25T00:18:55.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.095+0000 7f5ecd7fa640 1 -- 172.21.15.107:0/868162044 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f5ec001b070 con 0x7f5eb8004d10 2023-10-25T00:18:55.098 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:55.098 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:55.098 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 204 2023-10-25T00:18:55.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/868162044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ebc065e60 msgr2=0x7f5ebc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:55.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ebc065e60 0x7f5ebc068320 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00c3850 tx=0x7f5ec4008040 comp rx=0 tx=0).stop 2023-10-25T00:18:55.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/868162044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 msgr2=0x7f5ed00c3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:55.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 0x7f5ed00c3310 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0000f70 tx=0x7f5ec000e4d0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/868162044 shutdown_connections 2023-10-25T00:18:55.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5ebc065e60 0x7f5ebc068320 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ed01dea50 0x7f5ed01e9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ed01ddee0 0x7f5ed0153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 --2- 172.21.15.107:0/868162044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5eb8004d10 0x7f5ed00c3310 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/868162044 >> 172.21.15.107:0/868162044 conn(0x7f5ed00ca200 msgr2=0x7f5ed00ad070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:55.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/868162044 shutdown_connections 2023-10-25T00:18:55.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.099+0000 7f5ed6b6c640 1 -- 172.21.15.107:0/868162044 wait complete. 2023-10-25T00:18:55.117 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-25T00:18:55.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:55.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.767+0000 7faccc690640 1 Processor -- start 2023-10-25T00:18:55.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.767+0000 7faccc690640 1 -- start start 2023-10-25T00:18:55.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.767+0000 7faccc690640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc414eb70 0x7facc40c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:55.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.767+0000 7faccc690640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7facc4070b00 0x7facc41514e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:55.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.767+0000 7faccc690640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc40c22f0 con 0x7facc4151a20 2023-10-25T00:18:55.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.767+0000 7faccc690640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc4071700 con 0x7facc4070b00 2023-10-25T00:18:55.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.767+0000 7faccc690640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc40c25d0 con 0x7facc414eb70 2023-10-25T00:18:55.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.767+0000 7facca42c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc414eb70 0x7facc40c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:55.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.767+0000 7facca42c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc414eb70 0x7facc40c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:57988/0 (socket says 172.21.15.107:57988) 2023-10-25T00:18:55.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.767+0000 7facca42c640 1 -- 172.21.15.107:0/1296831400 learned_addr learned my addr 172.21.15.107:0/1296831400 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:55.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.767+0000 7facc9c2b640 1 --2- 172.21.15.107:0/1296831400 >> v2:172.21.15.110:3300/0 conn(0x7facc4070b00 0x7facc41514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:55.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7facca42c640 1 -- 172.21.15.107:0/1296831400 >> v2:172.21.15.110:3300/0 conn(0x7facc4070b00 msgr2=0x7facc41514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:55.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7facca42c640 1 --2- 172.21.15.107:0/1296831400 >> v2:172.21.15.110:3300/0 conn(0x7facc4070b00 0x7facc41514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7facca42c640 1 -- 172.21.15.107:0/1296831400 >> v1:172.21.15.110:6789/0 conn(0x7facc4151a20 legacy=0x7facc406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:55.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7facca42c640 1 -- 172.21.15.107:0/1296831400 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc40c2be0 con 0x7facc414eb70 2023-10-25T00:18:55.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7facca42c640 1 --2- 172.21.15.107:0/1296831400 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc414eb70 0x7facc40c33e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7facc0009870 tx=0x7facc001c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a25cdbbe73508d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:55.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7facc942a640 1 -- 172.21.15.107:0/1296831400 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc00aa640 con 0x7facc414eb70 2023-10-25T00:18:55.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7facc942a640 1 -- 172.21.15.107:0/1296831400 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc00aa7a0 con 0x7facc414eb70 2023-10-25T00:18:55.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7facc942a640 1 -- 172.21.15.107:0/1296831400 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc00aab30 con 0x7facc414eb70 2023-10-25T00:18:55.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7faccc690640 1 -- 172.21.15.107:0/1296831400 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc414eb70 msgr2=0x7facc40c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:55.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7faccc690640 1 --2- 172.21.15.107:0/1296831400 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc414eb70 0x7facc40c33e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7facc0009870 tx=0x7facc001c540 comp rx=0 tx=0).stop 2023-10-25T00:18:55.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7faccc690640 1 -- 172.21.15.107:0/1296831400 shutdown_connections 2023-10-25T00:18:55.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7faccc690640 1 --2- 172.21.15.107:0/1296831400 >> v2:172.21.15.110:3300/0 conn(0x7facc4070b00 0x7facc41514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7faccc690640 1 --2- 172.21.15.107:0/1296831400 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc414eb70 0x7facc40c33e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7faccc690640 1 -- 172.21.15.107:0/1296831400 >> 172.21.15.107:0/1296831400 conn(0x7facc40c39a0 msgr2=0x7facc40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:55.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7faccc690640 1 -- 172.21.15.107:0/1296831400 shutdown_connections 2023-10-25T00:18:55.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.771+0000 7faccc690640 1 -- 172.21.15.107:0/1296831400 wait complete. 2023-10-25T00:18:55.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccc690640 1 Processor -- start 2023-10-25T00:18:55.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccc690640 1 -- start start 2023-10-25T00:18:55.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccc690640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc4070b00 0x7facc41541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:55.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccc690640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc414eb70 0x7facc414dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:55.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccc690640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc4151a20 0x7facc40a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:55.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccc690640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc41559c0 con 0x7facc4070b00 2023-10-25T00:18:55.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccc690640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc41533a0 con 0x7facc414eb70 2023-10-25T00:18:55.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccc690640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc4155490 con 0x7facc4151a20 2023-10-25T00:18:55.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facca42c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc4070b00 0x7facc41541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:55.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facca42c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc4070b00 0x7facc41541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38062/0 (socket says 172.21.15.107:38062) 2023-10-25T00:18:55.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facca42c640 1 -- 172.21.15.107:0/366687913 learned_addr learned my addr 172.21.15.107:0/366687913 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:55.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccac2d640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc4151a20 0x7facc40a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:55.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facc9c2b640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc414eb70 0x7facc414dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:55.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facca42c640 1 -- 172.21.15.107:0/366687913 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc4151a20 msgr2=0x7facc40a9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:55.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facca42c640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc4151a20 0x7facc40a9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facca42c640 1 -- 172.21.15.107:0/366687913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc414eb70 msgr2=0x7facc414dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:55.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facca42c640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc414eb70 0x7facc414dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facca42c640 1 -- 172.21.15.107:0/366687913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc00094d0 con 0x7facc4070b00 2023-10-25T00:18:55.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facc9c2b640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc414eb70 0x7facc414dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:18:55.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facca42c640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc4070b00 0x7facc41541e0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7facc00ac040 tx=0x7facc001ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:55.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facbaffd640 1 -- 172.21.15.107:0/366687913 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc00bd5f0 con 0x7facc4070b00 2023-10-25T00:18:55.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facbaffd640 1 -- 172.21.15.107:0/366687913 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc00bd750 con 0x7facc4070b00 2023-10-25T00:18:55.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccc690640 1 -- 172.21.15.107:0/366687913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc41556e0 con 0x7facc4070b00 2023-10-25T00:18:55.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7facbaffd640 1 -- 172.21.15.107:0/366687913 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc00bdb00 con 0x7facc4070b00 2023-10-25T00:18:55.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccc690640 1 -- 172.21.15.107:0/366687913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc4152fc0 con 0x7facc4070b00 2023-10-25T00:18:55.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.775+0000 7faccc690640 1 -- 172.21.15.107:0/366687913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc4152b40 con 0x7facc4070b00 2023-10-25T00:18:55.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.779+0000 7facbaffd640 1 -- 172.21.15.107:0/366687913 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7facc00a8050 con 0x7facc4070b00 2023-10-25T00:18:55.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.779+0000 7facbaffd640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faca8065e90 0x7faca8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:55.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.779+0000 7facbaffd640 1 -- 172.21.15.107:0/366687913 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 10315+0+0 (secure 0 0 0) 0x7facc011ef80 con 0x7facc4070b00 2023-10-25T00:18:55.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.779+0000 7facc9c2b640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faca8065e90 0x7faca8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:55.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.779+0000 7facc9c2b640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faca8065e90 0x7faca8068350 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7facb4009ad0 tx=0x7facb4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:55.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.783+0000 7facbaffd640 1 -- 172.21.15.107:0/366687913 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7facc4152b40 con 0x7facc4070b00 2023-10-25T00:18:55.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.975+0000 7faccc690640 1 -- 172.21.15.107:0/366687913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facc4152b40 con 0x7facc4070b00 2023-10-25T00:18:55.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7facbaffd640 1 -- 172.21.15.107:0/366687913 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5774 (secure 0 0 0) 0x7facc4152b40 con 0x7facc4070b00 2023-10-25T00:18:55.980 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:55.980 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:55.980 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 205 2023-10-25T00:18:55.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 -- 172.21.15.107:0/366687913 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faca8065e90 msgr2=0x7faca8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:55.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faca8065e90 0x7faca8068350 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7facb4009ad0 tx=0x7facb4008040 comp rx=0 tx=0).stop 2023-10-25T00:18:55.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 -- 172.21.15.107:0/366687913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc4070b00 msgr2=0x7facc41541e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:55.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc4070b00 0x7facc41541e0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7facc00ac040 tx=0x7facc001ad80 comp rx=0 tx=0).stop 2023-10-25T00:18:55.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 -- 172.21.15.107:0/366687913 shutdown_connections 2023-10-25T00:18:55.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faca8065e90 0x7faca8068350 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7facc4151a20 0x7facc40a9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7facc414eb70 0x7facc414dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 --2- 172.21.15.107:0/366687913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7facc4070b00 0x7facc41541e0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:55.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 -- 172.21.15.107:0/366687913 >> 172.21.15.107:0/366687913 conn(0x7facc40c39a0 msgr2=0x7facc414c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:55.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 -- 172.21.15.107:0/366687913 shutdown_connections 2023-10-25T00:18:55.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:55.979+0000 7faccc690640 1 -- 172.21.15.107:0/366687913 wait complete. 2023-10-25T00:18:55.999 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:18:57.115 INFO:tasks.ceph.mds.b.smithi110.stderr:2023-10-25T00:18:57.104+0000 7f701772d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:18:57.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:57.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f259069b640 1 Processor -- start 2023-10-25T00:18:57.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f259069b640 1 -- start start 2023-10-25T00:18:57.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f259069b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f258814b600 0x7f2588151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:57.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f259069b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258806a230 0x7f25881551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:57.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f259069b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25880c1860 con 0x7f258806af70 2023-10-25T00:18:57.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f259069b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2588154d50 con 0x7f258814b600 2023-10-25T00:18:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f259069b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25881524f0 con 0x7f258806a230 2023-10-25T00:18:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258dc36640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f258806af70 0x7f258806c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44784/0 (socket says 172.21.15.107:44784) 2023-10-25T00:18:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258dc36640 1 -- 172.21.15.107:0/1522582874 learned_addr learned my addr 172.21.15.107:0/1522582874 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:18:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258ec38640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258806a230 0x7f25881551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258e437640 1 --2- 172.21.15.107:0/1522582874 >> v2:172.21.15.110:3300/0 conn(0x7f258814b600 0x7f2588151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651469018 0 0) 0x7f25880c1860 con 0x7f258806af70 2023-10-25T00:18:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2574003530 con 0x7f258806af70 2023-10-25T00:18:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3775521060 0 0) 0x7f2574003530 con 0x7f258806af70 2023-10-25T00:18:57.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258806a230 msgr2=0x7f25881551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258806a230 0x7f25881551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 >> v2:172.21.15.110:3300/0 conn(0x7f258814b600 msgr2=0x7f2588151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 --2- 172.21.15.107:0/1522582874 >> v2:172.21.15.110:3300/0 conn(0x7f258814b600 0x7f2588151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258814b9e0 con 0x7f258806af70 2023-10-25T00:18:57.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f2578001680 con 0x7f258806af70 2023-10-25T00:18:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 >> v1:172.21.15.107:6789/0 conn(0x7f258806af70 legacy=0x7f258806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258806a230 0x7f2574003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 0x7f25740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2574004d10 0x7f2574005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:57.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2588154d50 con 0x7f2574003f00 2023-10-25T00:18:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25881524f0 con 0x7f2574004d10 2023-10-25T00:18:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25740059e0 con 0x7f258806a230 2023-10-25T00:18:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258dc36640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 0x7f25740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258ec38640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2574004d10 0x7f2574005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258e437640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258806a230 0x7f2574003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:57.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258dc36640 1 -- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258806a230 msgr2=0x7f2574003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:57.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258dc36640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258806a230 0x7f2574003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:57.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258dc36640 1 -- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2574004d10 msgr2=0x7f2574005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:57.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258dc36640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2574004d10 0x7f2574005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:57.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258dc36640 1 -- 172.21.15.107:0/1522582874 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2574003530 con 0x7f2574003f00 2023-10-25T00:18:57.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.251+0000 7f258e437640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258806a230 0x7f2574003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:18:57.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f258dc36640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 0x7f25740047d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f257800c190 tx=0x7f257801cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a529c4ee1e005ad server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:57.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25780058e0 con 0x7f2574003f00 2023-10-25T00:18:57.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2578005a60 con 0x7f2574003f00 2023-10-25T00:18:57.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f258d435640 1 -- 172.21.15.107:0/1522582874 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578005e10 con 0x7f2574003f00 2023-10-25T00:18:57.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 -- 172.21.15.107:0/1522582874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 msgr2=0x7f25740047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:57.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 --2- 172.21.15.107:0/1522582874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 0x7f25740047d0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f257800c190 tx=0x7f257801cbf0 comp rx=0 tx=0).stop 2023-10-25T00:18:57.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f258ec38640 1 -- 172.21.15.107:0/1522582874 reap_dead start 2023-10-25T00:18:57.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 -- 172.21.15.107:0/1522582874 shutdown_connections 2023-10-25T00:18:57.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 -- 172.21.15.107:0/1522582874 >> 172.21.15.107:0/1522582874 conn(0x7f25880c2950 msgr2=0x7f25880c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:57.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 -- 172.21.15.107:0/1522582874 shutdown_connections 2023-10-25T00:18:57.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 -- 172.21.15.107:0/1522582874 wait complete. 2023-10-25T00:18:57.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 Processor -- start 2023-10-25T00:18:57.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 -- start start 2023-10-25T00:18:57.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 0x7f2588147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:57.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258814f030 0x7f25880aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:57.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2588144670 0x7f2588144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2588146a80 con 0x7f2574003f00 2023-10-25T00:18:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25881ec2b0 con 0x7f2588144670 2023-10-25T00:18:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.255+0000 7f259069b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25881eb710 con 0x7f258814f030 2023-10-25T00:18:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258e437640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 0x7f2588147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258e437640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 0x7f2588147890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38082/0 (socket says 172.21.15.107:38082) 2023-10-25T00:18:57.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258e437640 1 -- 172.21.15.107:0/1740736745 learned_addr learned my addr 172.21.15.107:0/1740736745 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:57.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258ec38640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2588144670 0x7f2588144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:57.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258dc36640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258814f030 0x7f25880aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:57.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258e437640 1 -- 172.21.15.107:0/1740736745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258814f030 msgr2=0x7f25880aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:57.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258e437640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258814f030 0x7f25880aabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:57.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258e437640 1 -- 172.21.15.107:0/1740736745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2588144670 msgr2=0x7f2588144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:57.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258e437640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2588144670 0x7f2588144af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:57.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258e437640 1 -- 172.21.15.107:0/1740736745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25881eb980 con 0x7f2574003f00 2023-10-25T00:18:57.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258e437640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 0x7f2588147890 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f25840109b0 tx=0x7f2584010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:57.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f257effd640 1 -- 172.21.15.107:0/1740736745 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2584004220 con 0x7f2574003f00 2023-10-25T00:18:57.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f257effd640 1 -- 172.21.15.107:0/1740736745 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2584004380 con 0x7f2574003f00 2023-10-25T00:18:57.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f259069b640 1 -- 172.21.15.107:0/1740736745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25881ebc60 con 0x7f2574003f00 2023-10-25T00:18:57.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f257effd640 1 -- 172.21.15.107:0/1740736745 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258400ebc0 con 0x7f2574003f00 2023-10-25T00:18:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f259069b640 1 -- 172.21.15.107:0/1740736745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2588145030 con 0x7f2574003f00 2023-10-25T00:18:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f259069b640 1 -- 172.21.15.107:0/1740736745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25880746e0 con 0x7f2574003f00 2023-10-25T00:18:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f257effd640 1 -- 172.21.15.107:0/1740736745 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f258400ed20 con 0x7f2574003f00 2023-10-25T00:18:57.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f257effd640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f256c065e90 0x7f256c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:57.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f257effd640 1 -- 172.21.15.107:0/1740736745 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 10315+0+0 (secure 0 0 0) 0x7f2584084800 con 0x7f2574003f00 2023-10-25T00:18:57.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258dc36640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f256c065e90 0x7f256c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:57.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.259+0000 7f258dc36640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f256c065e90 0x7f256c068350 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f25780aa000 tx=0x7f258806f430 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:57.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.263+0000 7f257effd640 1 -- 172.21.15.107:0/1740736745 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2584050df0 con 0x7f2574003f00 2023-10-25T00:18:57.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.479+0000 7f259069b640 1 -- 172.21.15.107:0/1740736745 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2588069fa0 con 0x7f2574003f00 2023-10-25T00:18:57.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.479+0000 7f257effd640 1 -- 172.21.15.107:0/1740736745 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f25840044f0 con 0x7f2574003f00 2023-10-25T00:18:57.481 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:57.481 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:57.481 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 205 2023-10-25T00:18:57.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 -- 172.21.15.107:0/1740736745 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f256c065e90 msgr2=0x7f256c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:57.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f256c065e90 0x7f256c068350 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f25780aa000 tx=0x7f258806f430 comp rx=0 tx=0).stop 2023-10-25T00:18:57.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 -- 172.21.15.107:0/1740736745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 msgr2=0x7f2588147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:57.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 0x7f2588147890 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f25840109b0 tx=0x7f2584010e70 comp rx=0 tx=0).stop 2023-10-25T00:18:57.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 -- 172.21.15.107:0/1740736745 shutdown_connections 2023-10-25T00:18:57.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2588144670 0x7f2588144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:57.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f256c065e90 0x7f256c068350 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:57.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f258814f030 0x7f25880aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:57.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 --2- 172.21.15.107:0/1740736745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2574003f00 0x7f2588147890 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:57.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 -- 172.21.15.107:0/1740736745 >> 172.21.15.107:0/1740736745 conn(0x7f25880c2950 msgr2=0x7f25881494f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:57.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 -- 172.21.15.107:0/1740736745 shutdown_connections 2023-10-25T00:18:57.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:57.483+0000 7f259069b640 1 -- 172.21.15.107:0/1740736745 wait complete. 2023-10-25T00:18:57.511 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2023-10-25T00:18:57.511 INFO:tasks.mds_thrash.fs.[a]:waiting for 65 secs before thrashing 2023-10-25T00:18:58.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:18:58.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.247+0000 7ff36e5f2640 1 Processor -- start 2023-10-25T00:18:58.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.247+0000 7ff36e5f2640 1 -- start start 2023-10-25T00:18:58.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff3680720c0 0x7ff36813d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:58.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff368154860 0x7ff368158230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3680adf70 con 0x7ff368071ce0 2023-10-25T00:18:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff36806d900 con 0x7ff3680720c0 2023-10-25T00:18:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36806e710 con 0x7ff368154860 2023-10-25T00:18:58.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff3677fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff3680720c0 0x7ff36813d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:58.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36cb8f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff368154860 0x7ff368158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:58.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36cb8f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff368154860 0x7ff368158230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:58036/0 (socket says 172.21.15.107:58036) 2023-10-25T00:18:58.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36cb8f640 1 -- 172.21.15.107:0/860511868 learned_addr learned my addr 172.21.15.107:0/860511868 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:58.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff3677fe640 1 -- 172.21.15.107:0/860511868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff368154860 msgr2=0x7ff368158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:58.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff3677fe640 1 --2- 172.21.15.107:0/860511868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff368154860 0x7ff368158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:58.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff3677fe640 1 -- 172.21.15.107:0/860511868 >> v1:172.21.15.110:6789/0 conn(0x7ff368071ce0 legacy=0x7ff3680ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:18:58.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff3677fe640 1 -- 172.21.15.107:0/860511868 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff368154d20 con 0x7ff3680720c0 2023-10-25T00:18:58.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff3677fe640 1 --2- 172.21.15.107:0/860511868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3680720c0 0x7ff36813d970 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7ff35c0097c0 tx=0x7ff35c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1011a90baec6205 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:58.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff366ffd640 1 -- 172.21.15.107:0/860511868 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0ab600 con 0x7ff3680720c0 2023-10-25T00:18:58.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff366ffd640 1 -- 172.21.15.107:0/860511868 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35c0a7050 con 0x7ff3680720c0 2023-10-25T00:18:58.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff366ffd640 1 -- 172.21.15.107:0/860511868 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0ab980 con 0x7ff3680720c0 2023-10-25T00:18:58.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 -- 172.21.15.107:0/860511868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3680720c0 msgr2=0x7ff36813d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:58.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 --2- 172.21.15.107:0/860511868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3680720c0 0x7ff36813d970 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7ff35c0097c0 tx=0x7ff35c01c380 comp rx=0 tx=0).stop 2023-10-25T00:18:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 -- 172.21.15.107:0/860511868 shutdown_connections 2023-10-25T00:18:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 --2- 172.21.15.107:0/860511868 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff368154860 0x7ff368158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 --2- 172.21.15.107:0/860511868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3680720c0 0x7ff36813d970 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 -- 172.21.15.107:0/860511868 >> 172.21.15.107:0/860511868 conn(0x7ff368070f60 msgr2=0x7ff368071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 -- 172.21.15.107:0/860511868 shutdown_connections 2023-10-25T00:18:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.251+0000 7ff36e5f2640 1 -- 172.21.15.107:0/860511868 wait complete. 2023-10-25T00:18:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36e5f2640 1 Processor -- start 2023-10-25T00:18:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36e5f2640 1 -- start start 2023-10-25T00:18:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36e5f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff368071ce0 0x7ff36813c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36e5f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3680720c0 0x7ff368136490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36e5f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff368154860 0x7ff3681369d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36e5f2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3681577e0 con 0x7ff3680720c0 2023-10-25T00:18:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36e5f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff368157c80 con 0x7ff368071ce0 2023-10-25T00:18:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36e5f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff368156710 con 0x7ff368154860 2023-10-25T00:18:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff3677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3680720c0 0x7ff368136490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff3677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3680720c0 0x7ff368136490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38090/0 (socket says 172.21.15.107:38090) 2023-10-25T00:18:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff3677fe640 1 -- 172.21.15.107:0/3113681030 learned_addr learned my addr 172.21.15.107:0/3113681030 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:18:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36cb8f640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff368154860 0x7ff3681369d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff367fff640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff368071ce0 0x7ff36813c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff3677fe640 1 -- 172.21.15.107:0/3113681030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff368154860 msgr2=0x7ff3681369d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff3677fe640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff368154860 0x7ff3681369d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff3677fe640 1 -- 172.21.15.107:0/3113681030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff368071ce0 msgr2=0x7ff36813c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff3677fe640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff368071ce0 0x7ff36813c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff3677fe640 1 -- 172.21.15.107:0/3113681030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c009420 con 0x7ff3680720c0 2023-10-25T00:18:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff3677fe640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3680720c0 0x7ff368136490 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ff35c0097c0 tx=0x7ff35c01abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff364ff9640 1 -- 172.21.15.107:0/3113681030 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0b4420 con 0x7ff3680720c0 2023-10-25T00:18:58.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff364ff9640 1 -- 172.21.15.107:0/3113681030 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35c0b45a0 con 0x7ff3680720c0 2023-10-25T00:18:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36e5f2640 1 -- 172.21.15.107:0/3113681030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3681579e0 con 0x7ff3680720c0 2023-10-25T00:18:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff364ff9640 1 -- 172.21.15.107:0/3113681030 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0b4950 con 0x7ff3680720c0 2023-10-25T00:18:58.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36e5f2640 1 -- 172.21.15.107:0/3113681030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff368135c90 con 0x7ff3680720c0 2023-10-25T00:18:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff364ff9640 1 -- 172.21.15.107:0/3113681030 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff35c0b2070 con 0x7ff3680720c0 2023-10-25T00:18:58.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.255+0000 7ff36e5f2640 1 -- 172.21.15.107:0/3113681030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff368135850 con 0x7ff3680720c0 2023-10-25T00:18:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.259+0000 7ff364ff9640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff354065e60 0x7ff354068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:18:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.259+0000 7ff364ff9640 1 -- 172.21.15.107:0/3113681030 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 10315+0+0 (secure 0 0 0) 0x7ff35c11d510 con 0x7ff3680720c0 2023-10-25T00:18:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.259+0000 7ff367fff640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff354065e60 0x7ff354068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:18:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.259+0000 7ff367fff640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff354065e60 0x7ff354068320 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7ff358010e90 tx=0x7ff358007c60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:18:58.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.263+0000 7ff364ff9640 1 -- 172.21.15.107:0/3113681030 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff368135850 con 0x7ff3680720c0 2023-10-25T00:18:58.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.475+0000 7ff36e5f2640 1 -- 172.21.15.107:0/3113681030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff368135850 con 0x7ff3680720c0 2023-10-25T00:18:58.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.483+0000 7ff364ff9640 1 -- 172.21.15.107:0/3113681030 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5774 (secure 0 0 0) 0x7ff368135850 con 0x7ff3680720c0 2023-10-25T00:18:58.486 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:18:58.486 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:18:58.487 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 205 2023-10-25T00:18:58.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 -- 172.21.15.107:0/3113681030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff354065e60 msgr2=0x7ff354068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:58.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff354065e60 0x7ff354068320 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7ff358010e90 tx=0x7ff358007c60 comp rx=0 tx=0).stop 2023-10-25T00:18:58.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 -- 172.21.15.107:0/3113681030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3680720c0 msgr2=0x7ff368136490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:18:58.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3680720c0 0x7ff368136490 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ff35c0097c0 tx=0x7ff35c01abc0 comp rx=0 tx=0).stop 2023-10-25T00:18:58.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 -- 172.21.15.107:0/3113681030 shutdown_connections 2023-10-25T00:18:58.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff354065e60 0x7ff354068320 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:58.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff368154860 0x7ff3681369d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:58.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3680720c0 0x7ff368136490 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:58.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 --2- 172.21.15.107:0/3113681030 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff368071ce0 0x7ff36813c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:18:58.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 -- 172.21.15.107:0/3113681030 >> 172.21.15.107:0/3113681030 conn(0x7ff368070f60 msgr2=0x7ff36806f080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:18:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 -- 172.21.15.107:0/3113681030 shutdown_connections 2023-10-25T00:18:58.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:18:58.487+0000 7ff34e7fc640 1 -- 172.21.15.107:0/3113681030 wait complete. 2023-10-25T00:18:58.512 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:19:00.260 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-25T00:19:00.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-25T00:19:00.277 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2023-10-25T00:19:00.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-25T00:19:00.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:00.669 DEBUG:teuthology.parallel:result is None 2023-10-25T00:19:00.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa530a5e640 1 Processor -- start 2023-10-25T00:19:00.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa530a5e640 1 -- start start 2023-10-25T00:19:00.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa530a5e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa52c0aa330 0x7fa52c06e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:00.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa530a5e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52c152d40 con 0x7fa52c0a9450 2023-10-25T00:19:00.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa530a5e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52c06d660 con 0x7fa52c13bb00 2023-10-25T00:19:00.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa530a5e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c156100 con 0x7fa52c0aa330 2023-10-25T00:19:00.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa52ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa52c0a9450 0x7fa52c071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44788/0 (socket says 172.21.15.107:44788) 2023-10-25T00:19:00.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa52ad9d640 1 -- 172.21.15.107:0/3700159704 learned_addr learned my addr 172.21.15.107:0/3700159704 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:19:00.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa523fff640 1 --2- 172.21.15.107:0/3700159704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa52c0aa330 0x7fa52c06e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:00.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa529d9b640 1 -- 172.21.15.107:0/3700159704 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515217572 0 0) 0x7fa52c152d40 con 0x7fa52c0a9450 2023-10-25T00:19:00.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa529d9b640 1 -- 172.21.15.107:0/3700159704 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa518003530 con 0x7fa52c0a9450 2023-10-25T00:19:00.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa529d9b640 1 -- 172.21.15.107:0/3700159704 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131660214 0 0) 0x7fa52c06d660 con 0x7fa52c13bb00 2023-10-25T00:19:00.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa529d9b640 1 -- 172.21.15.107:0/3700159704 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa52c152d40 con 0x7fa52c13bb00 2023-10-25T00:19:00.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa523fff640 1 -- 172.21.15.107:0/3700159704 >> v1:172.21.15.110:6789/0 conn(0x7fa52c13bb00 legacy=0x7fa52c1531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:00.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa523fff640 1 -- 172.21.15.107:0/3700159704 >> v1:172.21.15.107:6789/0 conn(0x7fa52c0a9450 legacy=0x7fa52c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:00.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.747+0000 7fa523fff640 1 -- 172.21.15.107:0/3700159704 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c155e60 con 0x7fa52c0aa330 2023-10-25T00:19:00.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa523fff640 1 --2- 172.21.15.107:0/3700159704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa52c0aa330 0x7fa52c06e250 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa514009820 tx=0x7fa51401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f085cbd8fe00c094 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:00.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa529d9b640 1 -- 172.21.15.107:0/3700159704 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5140ab600 con 0x7fa52c0aa330 2023-10-25T00:19:00.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa529d9b640 1 -- 172.21.15.107:0/3700159704 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5140a7050 con 0x7fa52c0aa330 2023-10-25T00:19:00.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa529d9b640 1 -- 172.21.15.107:0/3700159704 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5140ab960 con 0x7fa52c0aa330 2023-10-25T00:19:00.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa530a5e640 1 -- 172.21.15.107:0/3700159704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa52c0aa330 msgr2=0x7fa52c06e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:00.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa530a5e640 1 --2- 172.21.15.107:0/3700159704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa52c0aa330 0x7fa52c06e250 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa514009820 tx=0x7fa51401c380 comp rx=0 tx=0).stop 2023-10-25T00:19:00.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa530a5e640 1 -- 172.21.15.107:0/3700159704 shutdown_connections 2023-10-25T00:19:00.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa530a5e640 1 --2- 172.21.15.107:0/3700159704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa52c0aa330 0x7fa52c06e250 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:00.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa530a5e640 1 -- 172.21.15.107:0/3700159704 >> 172.21.15.107:0/3700159704 conn(0x7fa52c070f60 msgr2=0x7fa52c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:00.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa530a5e640 1 -- 172.21.15.107:0/3700159704 shutdown_connections 2023-10-25T00:19:00.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa530a5e640 1 -- 172.21.15.107:0/3700159704 wait complete. 2023-10-25T00:19:00.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa530a5e640 1 Processor -- start 2023-10-25T00:19:00.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.751+0000 7fa530a5e640 1 -- start start 2023-10-25T00:19:00.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa530a5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa52c0a9450 0x7fa52c155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:00.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa530a5e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa52c0aa330 0x7fa52c135a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:00.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa530a5e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa52c13bb00 0x7fa52c1387e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:00.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa530a5e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c131a90 con 0x7fa52c0a9450 2023-10-25T00:19:00.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa530a5e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c133c70 con 0x7fa52c0aa330 2023-10-25T00:19:00.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa530a5e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c153bd0 con 0x7fa52c13bb00 2023-10-25T00:19:00.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa52a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa52c0a9450 0x7fa52c155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:00.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa52a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa52c0a9450 0x7fa52c155690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38102/0 (socket says 172.21.15.107:38102) 2023-10-25T00:19:00.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa52a59c640 1 -- 172.21.15.107:0/2629193490 learned_addr learned my addr 172.21.15.107:0/2629193490 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:00.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa52ad9d640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa52c13bb00 0x7fa52c1387e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:00.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa523fff640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa52c0aa330 0x7fa52c135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:00.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa52a59c640 1 -- 172.21.15.107:0/2629193490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa52c13bb00 msgr2=0x7fa52c1387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:00.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa52a59c640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa52c13bb00 0x7fa52c1387e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:00.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa52a59c640 1 -- 172.21.15.107:0/2629193490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa52c0aa330 msgr2=0x7fa52c135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:00.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa52a59c640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa52c0aa330 0x7fa52c135a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:00.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa52a59c640 1 -- 172.21.15.107:0/2629193490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa514009420 con 0x7fa52c0a9450 2023-10-25T00:19:00.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa52a59c640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa52c0a9450 0x7fa52c155690 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00c430 tx=0x7fa51c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:00.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa522ffd640 1 -- 172.21.15.107:0/2629193490 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c0052d0 con 0x7fa52c0a9450 2023-10-25T00:19:00.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa522ffd640 1 -- 172.21.15.107:0/2629193490 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51c005450 con 0x7fa52c0a9450 2023-10-25T00:19:00.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa530a5e640 1 -- 172.21.15.107:0/2629193490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52c153eb0 con 0x7fa52c0a9450 2023-10-25T00:19:00.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa522ffd640 1 -- 172.21.15.107:0/2629193490 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51c003840 con 0x7fa52c0a9450 2023-10-25T00:19:00.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa530a5e640 1 -- 172.21.15.107:0/2629193490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52c133f50 con 0x7fa52c0a9450 2023-10-25T00:19:00.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa530a5e640 1 -- 172.21.15.107:0/2629193490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52c0746e0 con 0x7fa52c0a9450 2023-10-25T00:19:00.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa522ffd640 1 -- 172.21.15.107:0/2629193490 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa51c0039a0 con 0x7fa52c0a9450 2023-10-25T00:19:00.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa522ffd640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa50c065e90 0x7fa50c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:00.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa522ffd640 1 -- 172.21.15.107:0/2629193490 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 10315+0+0 (secure 0 0 0) 0x7fa51c083e20 con 0x7fa52c0a9450 2023-10-25T00:19:00.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa523fff640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa50c065e90 0x7fa50c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:00.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.755+0000 7fa523fff640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa50c065e90 0x7fa50c068350 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa5140ad000 tx=0x7fa5140a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:00.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.759+0000 7fa522ffd640 1 -- 172.21.15.107:0/2629193490 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa51c0504c0 con 0x7fa52c0a9450 2023-10-25T00:19:00.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 -- 172.21.15.107:0/2629193490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa52c06d160 con 0x7fa52c0a9450 2023-10-25T00:19:00.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa522ffd640 1 -- 172.21.15.107:0/2629193490 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5774 (secure 0 0 0) 0x7fa51c017090 con 0x7fa52c0a9450 2023-10-25T00:19:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:00.953 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:00.953 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 205 2023-10-25T00:19:00.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 -- 172.21.15.107:0/2629193490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa50c065e90 msgr2=0x7fa50c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:00.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa50c065e90 0x7fa50c068350 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa5140ad000 tx=0x7fa5140a6000 comp rx=0 tx=0).stop 2023-10-25T00:19:00.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 -- 172.21.15.107:0/2629193490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa52c0a9450 msgr2=0x7fa52c155690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:00.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa52c0a9450 0x7fa52c155690 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fa51c00c430 tx=0x7fa51c00c8f0 comp rx=0 tx=0).stop 2023-10-25T00:19:00.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 -- 172.21.15.107:0/2629193490 shutdown_connections 2023-10-25T00:19:00.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa50c065e90 0x7fa50c068350 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:00.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa52c13bb00 0x7fa52c1387e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:00.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa52c0aa330 0x7fa52c135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:00.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 --2- 172.21.15.107:0/2629193490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa52c0a9450 0x7fa52c155690 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:00.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 -- 172.21.15.107:0/2629193490 >> 172.21.15.107:0/2629193490 conn(0x7fa52c070f60 msgr2=0x7fa52c1531d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:00.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 -- 172.21.15.107:0/2629193490 shutdown_connections 2023-10-25T00:19:00.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:00.951+0000 7fa530a5e640 1 -- 172.21.15.107:0/2629193490 wait complete. 2023-10-25T00:19:00.971 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:19:02.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:03.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.111+0000 7f4cf25ba640 1 Processor -- start 2023-10-25T00:19:03.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.111+0000 7f4cf25ba640 1 -- start start 2023-10-25T00:19:03.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf25ba640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4cec0c2200 0x7f4cec154930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:03.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf25ba640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4cec06c920 0x7f4cec154e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:03.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf25ba640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec158da0 con 0x7f4cec0ab3c0 2023-10-25T00:19:03.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf25ba640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cec1588b0 con 0x7f4cec0c2200 2023-10-25T00:19:03.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf25ba640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cec0c06e0 con 0x7f4cec06c920 2023-10-25T00:19:03.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf1db9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4cec06c920 0x7f4cec154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:03.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf1db9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4cec06c920 0x7f4cec154e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39396/0 (socket says 172.21.15.107:39396) 2023-10-25T00:19:03.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf1db9640 1 -- 172.21.15.107:0/2030952040 learned_addr learned my addr 172.21.15.107:0/2030952040 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:03.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf1db9640 1 -- 172.21.15.107:0/2030952040 >> v2:172.21.15.107:3300/0 conn(0x7f4cec0c2200 msgr2=0x7f4cec154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:03.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf1db9640 1 --2- 172.21.15.107:0/2030952040 >> v2:172.21.15.107:3300/0 conn(0x7f4cec0c2200 0x7f4cec154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:03.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf1db9640 1 -- 172.21.15.107:0/2030952040 >> v1:172.21.15.110:6789/0 conn(0x7f4cec0ab3c0 legacy=0x7f4cec0c2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:03.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.115+0000 7f4cf1db9640 1 -- 172.21.15.107:0/2030952040 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec1553b0 con 0x7f4cec06c920 2023-10-25T00:19:03.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.119+0000 7f4cf1db9640 1 --2- 172.21.15.107:0/2030952040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cec06c920 0x7f4cec154e70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0bcc90 tx=0x7f4ce401c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c99a64d2483bf9fc server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:03.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.119+0000 7f4ce3fff640 1 -- 172.21.15.107:0/2030952040 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce40a57c0 con 0x7f4cec06c920 2023-10-25T00:19:03.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.119+0000 7f4ce3fff640 1 -- 172.21.15.107:0/2030952040 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce40a5920 con 0x7f4cec06c920 2023-10-25T00:19:03.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.119+0000 7f4ce3fff640 1 -- 172.21.15.107:0/2030952040 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce40a5cb0 con 0x7f4cec06c920 2023-10-25T00:19:03.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.119+0000 7f4cf25ba640 1 -- 172.21.15.107:0/2030952040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cec06c920 msgr2=0x7f4cec154e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:03.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.119+0000 7f4cf25ba640 1 --2- 172.21.15.107:0/2030952040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cec06c920 0x7f4cec154e70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0bcc90 tx=0x7f4ce401c920 comp rx=0 tx=0).stop 2023-10-25T00:19:03.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.119+0000 7f4cf25ba640 1 -- 172.21.15.107:0/2030952040 shutdown_connections 2023-10-25T00:19:03.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.119+0000 7f4cf25ba640 1 --2- 172.21.15.107:0/2030952040 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cec06c920 0x7f4cec154e70 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:03.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.119+0000 7f4cf25ba640 1 --2- 172.21.15.107:0/2030952040 >> v2:172.21.15.107:3300/0 conn(0x7f4cec0c2200 0x7f4cec154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:03.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.119+0000 7f4cf25ba640 1 -- 172.21.15.107:0/2030952040 >> 172.21.15.107:0/2030952040 conn(0x7f4cec0a77e0 msgr2=0x7f4cec0a7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:03.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.119+0000 7f4cf25ba640 1 -- 172.21.15.107:0/2030952040 shutdown_connections 2023-10-25T00:19:03.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 -- 172.21.15.107:0/2030952040 wait complete. 2023-10-25T00:19:03.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 Processor -- start 2023-10-25T00:19:03.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 -- start start 2023-10-25T00:19:03.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec06c920 0x7f4cec15e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:03.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cec0ab3c0 0x7f4cec0701b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:03.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec0c2200 0x7f4cec0706f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:03.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0bd800 con 0x7f4cec0c2200 2023-10-25T00:19:03.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0bd1d0 con 0x7f4cec0ab3c0 2023-10-25T00:19:03.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec1f0630 con 0x7f4cec06c920 2023-10-25T00:19:03.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf1db9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec0c2200 0x7f4cec0706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:03.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf1db9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec0c2200 0x7f4cec0706f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51604/0 (socket says 172.21.15.107:51604) 2023-10-25T00:19:03.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf1db9640 1 -- 172.21.15.107:0/4275225035 learned_addr learned my addr 172.21.15.107:0/4275225035 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:03.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf0db7640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cec0ab3c0 0x7f4cec0701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:03.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf15b8640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec06c920 0x7f4cec15e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:03.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf1db9640 1 -- 172.21.15.107:0/4275225035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec06c920 msgr2=0x7f4cec15e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:03.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf1db9640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec06c920 0x7f4cec15e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:03.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf1db9640 1 -- 172.21.15.107:0/4275225035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cec0ab3c0 msgr2=0x7f4cec0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:03.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf1db9640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cec0ab3c0 0x7f4cec0701b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:03.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf1db9640 1 -- 172.21.15.107:0/4275225035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce4009b30 con 0x7f4cec0c2200 2023-10-25T00:19:03.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf1db9640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec0c2200 0x7f4cec0706f0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f4ce401a550 tx=0x7f4ce40196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:03.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4ce1ffb640 1 -- 172.21.15.107:0/4275225035 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce40b8880 con 0x7f4cec0c2200 2023-10-25T00:19:03.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4ce1ffb640 1 -- 172.21.15.107:0/4275225035 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce40b89e0 con 0x7f4cec0c2200 2023-10-25T00:19:03.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 -- 172.21.15.107:0/4275225035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cec1f0910 con 0x7f4cec0c2200 2023-10-25T00:19:03.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4ce1ffb640 1 -- 172.21.15.107:0/4275225035 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce40b8d70 con 0x7f4cec0c2200 2023-10-25T00:19:03.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 -- 172.21.15.107:0/4275225035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cec0aaa80 con 0x7f4cec0c2200 2023-10-25T00:19:03.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.123+0000 7f4cf25ba640 1 -- 172.21.15.107:0/4275225035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cec1f0aa0 con 0x7f4cec0c2200 2023-10-25T00:19:03.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.127+0000 7f4ce1ffb640 1 -- 172.21.15.107:0/4275225035 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4ce40b6440 con 0x7f4cec0c2200 2023-10-25T00:19:03.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.127+0000 7f4ce1ffb640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e60 0x7f4cd0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:03.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.127+0000 7f4ce1ffb640 1 -- 172.21.15.107:0/4275225035 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 10315+0+0 (secure 0 0 0) 0x7f4ce4120090 con 0x7f4cec0c2200 2023-10-25T00:19:03.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.127+0000 7f4cf15b8640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e60 0x7f4cd0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:03.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.127+0000 7f4cf15b8640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e60 0x7f4cd0068320 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0bc090 tx=0x7f4cdc009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:03.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.131+0000 7f4ce1ffb640 1 -- 172.21.15.107:0/4275225035 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4ce40ec7d0 con 0x7f4cec0c2200 2023-10-25T00:19:03.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.315+0000 7f4cf25ba640 1 -- 172.21.15.107:0/4275225035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cec04e050 con 0x7f4cec0c2200 2023-10-25T00:19:03.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4ce1ffb640 1 -- 172.21.15.107:0/4275225035 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5774 (secure 0 0 0) 0x7f4ce40efdf0 con 0x7f4cec0c2200 2023-10-25T00:19:03.319 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:03.320 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:17:43.371756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7213},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7213":{"gid":7213,"name":"c","rank":0,"incarnation":196,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/2970408860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2970408860},{"type":"v1","addr":"172.21.15.107:6835","nonce":2970408860}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:03.320 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 205 2023-10-25T00:19:03.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 -- 172.21.15.107:0/4275225035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e60 msgr2=0x7f4cd0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:03.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e60 0x7f4cd0068320 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0bc090 tx=0x7f4cdc009060 comp rx=0 tx=0).stop 2023-10-25T00:19:03.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 -- 172.21.15.107:0/4275225035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec0c2200 msgr2=0x7f4cec0706f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:03.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec0c2200 0x7f4cec0706f0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f4ce401a550 tx=0x7f4ce40196c0 comp rx=0 tx=0).stop 2023-10-25T00:19:03.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 -- 172.21.15.107:0/4275225035 shutdown_connections 2023-10-25T00:19:03.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec0c2200 0x7f4cec0706f0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:03.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4cec0ab3c0 0x7f4cec0701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:03.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4cd0065e60 0x7f4cd0068320 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:03.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 --2- 172.21.15.107:0/4275225035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4cec06c920 0x7f4cec15e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:03.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 -- 172.21.15.107:0/4275225035 >> 172.21.15.107:0/4275225035 conn(0x7f4cec0a77e0 msgr2=0x7f4cec0a7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:03.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 -- 172.21.15.107:0/4275225035 shutdown_connections 2023-10-25T00:19:03.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:03.319+0000 7f4cf25ba640 1 -- 172.21.15.107:0/4275225035 wait complete. 2023-10-25T00:19:03.338 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:19:05.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:05.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.471+0000 7f1904a65640 1 Processor -- start 2023-10-25T00:19:05.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.471+0000 7f1904a65640 1 -- start start 2023-10-25T00:19:05.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f190014fd40 0x7f19000ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f190006ee40 0x7f19000aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190006f730 con 0x7f19000a9820 2023-10-25T00:19:05.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19000aa750 con 0x7f190006ee40 2023-10-25T00:19:05.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1900156a40 con 0x7f190014fd40 2023-10-25T00:19:05.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fdd9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f19000a9820 0x7f19000c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36354/0 (socket says 172.21.15.107:36354) 2023-10-25T00:19:05.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fdd9b640 1 -- 172.21.15.107:0/3611219882 learned_addr learned my addr 172.21.15.107:0/3611219882 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:19:05.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fed9d640 1 --2- 172.21.15.107:0/3611219882 >> v2:172.21.15.107:3300/0 conn(0x7f190006ee40 0x7f19000aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fe59c640 1 --2- 172.21.15.107:0/3611219882 >> v2:172.21.15.110:3300/0 conn(0x7f190014fd40 0x7f19000ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fd59a640 1 -- 172.21.15.107:0/3611219882 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916107903 0 0) 0x7f190006f730 con 0x7f19000a9820 2023-10-25T00:19:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fd59a640 1 -- 172.21.15.107:0/3611219882 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e8003530 con 0x7f19000a9820 2023-10-25T00:19:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fed9d640 1 -- 172.21.15.107:0/3611219882 >> v2:172.21.15.110:3300/0 conn(0x7f190014fd40 msgr2=0x7f19000ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:05.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fed9d640 1 --2- 172.21.15.107:0/3611219882 >> v2:172.21.15.110:3300/0 conn(0x7f190014fd40 0x7f19000ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fed9d640 1 -- 172.21.15.107:0/3611219882 >> v1:172.21.15.107:6789/0 conn(0x7f19000a9820 legacy=0x7f19000c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fed9d640 1 -- 172.21.15.107:0/3611219882 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1900156710 con 0x7f190006ee40 2023-10-25T00:19:05.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fed9d640 1 --2- 172.21.15.107:0/3611219882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f190006ee40 0x7f19000aa210 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f19000c28b0 tx=0x7f18f801c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c3b08a03eb0036 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fd59a640 1 -- 172.21.15.107:0/3611219882 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f80a5820 con 0x7f190006ee40 2023-10-25T00:19:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fd59a640 1 -- 172.21.15.107:0/3611219882 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f80a5980 con 0x7f190006ee40 2023-10-25T00:19:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f18fd59a640 1 -- 172.21.15.107:0/3611219882 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f80a5d10 con 0x7f190006ee40 2023-10-25T00:19:05.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 -- 172.21.15.107:0/3611219882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f190006ee40 msgr2=0x7f19000aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 --2- 172.21.15.107:0/3611219882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f190006ee40 0x7f19000aa210 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f19000c28b0 tx=0x7f18f801c7c0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 -- 172.21.15.107:0/3611219882 shutdown_connections 2023-10-25T00:19:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 --2- 172.21.15.107:0/3611219882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f190006ee40 0x7f19000aa210 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 --2- 172.21.15.107:0/3611219882 >> v2:172.21.15.110:3300/0 conn(0x7f190014fd40 0x7f19000ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 -- 172.21.15.107:0/3611219882 >> 172.21.15.107:0/3611219882 conn(0x7f19000c2950 msgr2=0x7f19000c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:05.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 -- 172.21.15.107:0/3611219882 shutdown_connections 2023-10-25T00:19:05.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.475+0000 7f1904a65640 1 -- 172.21.15.107:0/3611219882 wait complete. 2023-10-25T00:19:05.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f1904a65640 1 Processor -- start 2023-10-25T00:19:05.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f1904a65640 1 -- start start 2023-10-25T00:19:05.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f1904a65640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f190006ee40 0x7f190015b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f1904a65640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f190014fd40 0x7f19000c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f1904a65640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f19000c5610 0x7f19001ecfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f1904a65640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1900069d40 con 0x7f190006ee40 2023-10-25T00:19:05.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f1904a65640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190014c590 con 0x7f190014fd40 2023-10-25T00:19:05.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f1904a65640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f190014cac0 con 0x7f19000c5610 2023-10-25T00:19:05.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f190006ee40 0x7f190015b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:05.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fe59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f190006ee40 0x7f190015b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51630/0 (socket says 172.21.15.107:51630) 2023-10-25T00:19:05.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fe59c640 1 -- 172.21.15.107:0/309369359 learned_addr learned my addr 172.21.15.107:0/309369359 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:05.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fed9d640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f19000c5610 0x7f19001ecfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:05.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fdd9b640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f190014fd40 0x7f19000c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:05.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fe59c640 1 -- 172.21.15.107:0/309369359 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f19000c5610 msgr2=0x7f19001ecfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fe59c640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f19000c5610 0x7f19001ecfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fe59c640 1 -- 172.21.15.107:0/309369359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f190014fd40 msgr2=0x7f19000c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fe59c640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f190014fd40 0x7f19000c49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fe59c640 1 -- 172.21.15.107:0/309369359 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f8009a70 con 0x7f190006ee40 2023-10-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fdd9b640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f190014fd40 0x7f19000c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18fe59c640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f190006ee40 0x7f190015b5a0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f18f000d800 tx=0x7f18f000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18eeffd640 1 -- 172.21.15.107:0/309369359 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f000b660 con 0x7f190006ee40 2023-10-25T00:19:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18eeffd640 1 -- 172.21.15.107:0/309369359 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f000b7c0 con 0x7f190006ee40 2023-10-25T00:19:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f1904a65640 1 -- 172.21.15.107:0/309369359 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1900154390 con 0x7f190006ee40 2023-10-25T00:19:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f18eeffd640 1 -- 172.21.15.107:0/309369359 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f000bb30 con 0x7f190006ee40 2023-10-25T00:19:05.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f1904a65640 1 -- 172.21.15.107:0/309369359 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190014c7e0 con 0x7f190006ee40 2023-10-25T00:19:05.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.479+0000 7f1904a65640 1 -- 172.21.15.107:0/309369359 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19000c24a0 con 0x7f190006ee40 2023-10-25T00:19:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.483+0000 7f18eeffd640 1 -- 172.21.15.107:0/309369359 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f18f000bc90 con 0x7f190006ee40 2023-10-25T00:19:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.483+0000 7f18eeffd640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f18e0065e60 0x7f18e0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.483+0000 7f18fdd9b640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f18e0065e60 0x7f18e0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:05.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.483+0000 7f18eeffd640 1 -- 172.21.15.107:0/309369359 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7f18f0084e60 con 0x7f190006ee40 2023-10-25T00:19:05.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.483+0000 7f18fdd9b640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f18e0065e60 0x7f18e0068320 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f1900073f20 tx=0x7f18f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:05.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.487+0000 7f18eeffd640 1 -- 172.21.15.107:0/309369359 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f18f00513f0 con 0x7f190006ee40 2023-10-25T00:19:05.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.659+0000 7f1904a65640 1 -- 172.21.15.107:0/309369359 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19000c27c0 con 0x7f190006ee40 2023-10-25T00:19:05.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f18eeffd640 1 -- 172.21.15.107:0/309369359 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f18f0054a10 con 0x7f190006ee40 2023-10-25T00:19:05.664 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:05.664 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:05.384643+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":206,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:05.664 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 207 2023-10-25T00:19:05.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 -- 172.21.15.107:0/309369359 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f18e0065e60 msgr2=0x7f18e0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f18e0065e60 0x7f18e0068320 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f1900073f20 tx=0x7f18f4009040 comp rx=0 tx=0).stop 2023-10-25T00:19:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 -- 172.21.15.107:0/309369359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f190006ee40 msgr2=0x7f190015b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f190006ee40 0x7f190015b5a0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f18f000d800 tx=0x7f18f000dcc0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 -- 172.21.15.107:0/309369359 shutdown_connections 2023-10-25T00:19:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f18e0065e60 0x7f18e0068320 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f19000c5610 0x7f19001ecfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f190014fd40 0x7f19000c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 --2- 172.21.15.107:0/309369359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f190006ee40 0x7f190015b5a0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 -- 172.21.15.107:0/309369359 >> 172.21.15.107:0/309369359 conn(0x7f19000c2950 msgr2=0x7f19000c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:05.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 -- 172.21.15.107:0/309369359 shutdown_connections 2023-10-25T00:19:05.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.663+0000 7f1904a65640 1 -- 172.21.15.107:0/309369359 wait complete. 2023-10-25T00:19:05.682 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2023-10-25T00:19:05.682 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:19:05.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:05.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.787+0000 7fa33346d640 1 Processor -- start 2023-10-25T00:19:05.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa33346d640 1 -- start start 2023-10-25T00:19:05.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa33346d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa32c0c1570 0x7fa32c0a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa33346d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa32c069d40 0x7fa32c06a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa33346d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c0a9b90 con 0x7fa32c150930 2023-10-25T00:19:05.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa33346d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa32c066040 con 0x7fa32c069d40 2023-10-25T00:19:05.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa33346d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c0c1390 con 0x7fa32c0c1570 2023-10-25T00:19:05.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa331a0a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa32c069d40 0x7fa32c06a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:05.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa330a08640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa32c150930 0x7fa32c065930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36366/0 (socket says 172.21.15.107:36366) 2023-10-25T00:19:05.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa331a0a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa32c069d40 0x7fa32c06a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51638/0 (socket says 172.21.15.107:51638) 2023-10-25T00:19:05.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa330a08640 1 -- 172.21.15.107:0/615981486 learned_addr learned my addr 172.21.15.107:0/615981486 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:19:05.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa331209640 1 --2- 172.21.15.107:0/615981486 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa32c0c1570 0x7fa32c0a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:05.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa331a0a640 1 -- 172.21.15.107:0/615981486 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa32c0c1570 msgr2=0x7fa32c0a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:05.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa331a0a640 1 --2- 172.21.15.107:0/615981486 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa32c0c1570 0x7fa32c0a94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa331a0a640 1 -- 172.21.15.107:0/615981486 >> v1:172.21.15.107:6789/0 conn(0x7fa32c150930 legacy=0x7fa32c065930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa323fff640 1 -- 172.21.15.107:0/615981486 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852979302 0 0) 0x7fa32c0a9b90 con 0x7fa32c150930 2023-10-25T00:19:05.797 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa331a0a640 1 -- 172.21.15.107:0/615981486 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c154dd0 con 0x7fa32c069d40 2023-10-25T00:19:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.791+0000 7fa331a0a640 1 --2- 172.21.15.107:0/615981486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa32c069d40 0x7fa32c06a1d0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fa324009dc0 tx=0x7fa32401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54dd127ba88b4ba server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa323fff640 1 -- 172.21.15.107:0/615981486 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3240a57e0 con 0x7fa32c069d40 2023-10-25T00:19:05.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa323fff640 1 -- 172.21.15.107:0/615981486 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3240a5940 con 0x7fa32c069d40 2023-10-25T00:19:05.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa323fff640 1 -- 172.21.15.107:0/615981486 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3240a5cd0 con 0x7fa32c069d40 2023-10-25T00:19:05.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 -- 172.21.15.107:0/615981486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa32c069d40 msgr2=0x7fa32c06a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 --2- 172.21.15.107:0/615981486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa32c069d40 0x7fa32c06a1d0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7fa324009dc0 tx=0x7fa32401c810 comp rx=0 tx=0).stop 2023-10-25T00:19:05.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 -- 172.21.15.107:0/615981486 shutdown_connections 2023-10-25T00:19:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 --2- 172.21.15.107:0/615981486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa32c069d40 0x7fa32c06a1d0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 --2- 172.21.15.107:0/615981486 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa32c0c1570 0x7fa32c0a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 -- 172.21.15.107:0/615981486 >> 172.21.15.107:0/615981486 conn(0x7fa32c0c65c0 msgr2=0x7fa32c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:05.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 -- 172.21.15.107:0/615981486 shutdown_connections 2023-10-25T00:19:05.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 -- 172.21.15.107:0/615981486 wait complete. 2023-10-25T00:19:05.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 Processor -- start 2023-10-25T00:19:05.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 -- start start 2023-10-25T00:19:05.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa32c069d40 0x7fa32c145890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa32c0c1570 0x7fa32c0a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa32c150930 0x7fa32c0a6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c149a90 con 0x7fa32c069d40 2023-10-25T00:19:05.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c14a050 con 0x7fa32c0c1570 2023-10-25T00:19:05.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c1ee610 con 0x7fa32c150930 2023-10-25T00:19:05.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa331209640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa32c069d40 0x7fa32c145890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:05.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa331209640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa32c069d40 0x7fa32c145890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51650/0 (socket says 172.21.15.107:51650) 2023-10-25T00:19:05.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa331209640 1 -- 172.21.15.107:0/4141184668 learned_addr learned my addr 172.21.15.107:0/4141184668 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:05.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa331a0a640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa32c150930 0x7fa32c0a6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:05.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa331209640 1 -- 172.21.15.107:0/4141184668 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa32c150930 msgr2=0x7fa32c0a6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa331209640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa32c150930 0x7fa32c0a6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa331209640 1 -- 172.21.15.107:0/4141184668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa32c0c1570 msgr2=0x7fa32c0a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa331209640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa32c0c1570 0x7fa32c0a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa331209640 1 -- 172.21.15.107:0/4141184668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa324009a70 con 0x7fa32c069d40 2023-10-25T00:19:05.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa331a0a640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa32c150930 0x7fa32c0a6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:19:05.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa331209640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa32c069d40 0x7fa32c145890 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fa32800e930 tx=0x7fa32800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:05.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa321ffb640 1 -- 172.21.15.107:0/4141184668 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328009690 con 0x7fa32c069d40 2023-10-25T00:19:05.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa321ffb640 1 -- 172.21.15.107:0/4141184668 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32800ca00 con 0x7fa32c069d40 2023-10-25T00:19:05.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 -- 172.21.15.107:0/4141184668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32c1ee8b0 con 0x7fa32c069d40 2023-10-25T00:19:05.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa321ffb640 1 -- 172.21.15.107:0/4141184668 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32800cd70 con 0x7fa32c069d40 2023-10-25T00:19:05.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.795+0000 7fa33346d640 1 -- 172.21.15.107:0/4141184668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32c145dd0 con 0x7fa32c069d40 2023-10-25T00:19:05.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.799+0000 7fa33346d640 1 -- 172.21.15.107:0/4141184668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32c0746e0 con 0x7fa32c069d40 2023-10-25T00:19:05.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.799+0000 7fa321ffb640 1 -- 172.21.15.107:0/4141184668 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa328013470 con 0x7fa32c069d40 2023-10-25T00:19:05.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.799+0000 7fa321ffb640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa310065e90 0x7fa310068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:05.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.799+0000 7fa321ffb640 1 -- 172.21.15.107:0/4141184668 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7fa328084b50 con 0x7fa32c069d40 2023-10-25T00:19:05.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.799+0000 7fa330a08640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa310065e90 0x7fa310068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:05.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.799+0000 7fa330a08640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa310065e90 0x7fa310068350 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa32c06c600 tx=0x7fa31c007970 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:05.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.803+0000 7fa321ffb640 1 -- 172.21.15.107:0/4141184668 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa3280511a0 con 0x7fa32c069d40 2023-10-25T00:19:05.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.991+0000 7fa33346d640 1 -- 172.21.15.107:0/4141184668 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa32c06acc0 con 0x7fa32c069d40 2023-10-25T00:19:05.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.991+0000 7fa321ffb640 1 -- 172.21.15.107:0/4141184668 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+5016 (secure 0 0 0) 0x7fa328021090 con 0x7fa32c069d40 2023-10-25T00:19:05.993 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:05.994 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:05.384643+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":206,"state":"up:reconnect","state_seq":4,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:05.995 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 207 2023-10-25T00:19:05.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.991+0000 7fa33346d640 1 -- 172.21.15.107:0/4141184668 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa310065e90 msgr2=0x7fa310068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.991+0000 7fa33346d640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa310065e90 0x7fa310068350 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fa32c06c600 tx=0x7fa31c007970 comp rx=0 tx=0).stop 2023-10-25T00:19:05.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.995+0000 7fa33346d640 1 -- 172.21.15.107:0/4141184668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa32c069d40 msgr2=0x7fa32c145890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:05.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.995+0000 7fa33346d640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa32c069d40 0x7fa32c145890 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fa32800e930 tx=0x7fa32800edf0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.995+0000 7fa33346d640 1 -- 172.21.15.107:0/4141184668 shutdown_connections 2023-10-25T00:19:05.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.995+0000 7fa33346d640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa310065e90 0x7fa310068350 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.995+0000 7fa33346d640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa32c150930 0x7fa32c0a6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.995+0000 7fa33346d640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa32c0c1570 0x7fa32c0a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.995+0000 7fa33346d640 1 --2- 172.21.15.107:0/4141184668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa32c069d40 0x7fa32c145890 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:05.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.995+0000 7fa33346d640 1 -- 172.21.15.107:0/4141184668 >> 172.21.15.107:0/4141184668 conn(0x7fa32c0c65c0 msgr2=0x7fa32c0b9570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:05.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.995+0000 7fa33346d640 1 -- 172.21.15.107:0/4141184668 shutdown_connections 2023-10-25T00:19:05.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:05.995+0000 7fa33346d640 1 -- 172.21.15.107:0/4141184668 wait complete. 2023-10-25T00:19:06.012 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 207, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 204, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7258": { "addr": "172.21.15.110:6834/3771564389", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 3771564389, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 3771564389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7258, "incarnation": 201, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:18:53.158376+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7258 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 207, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8649": { "addr": "172.21.15.107:6835/3396644475", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 3396644475, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 3396644475, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8649, "incarnation": 206, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:19:05.384643+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8649 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2437148", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2437148, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2437148, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 190, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6443, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:19:08.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cde1b6640 1 Processor -- start 2023-10-25T00:19:08.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cde1b6640 1 -- start start 2023-10-25T00:19:08.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cde1b6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2cd806ee40 0x7f2cd80ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cde1b6640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd806f730 con 0x7f2cd80a9820 2023-10-25T00:19:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cde1b6640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd8156a40 con 0x7f2cd814fd40 2023-10-25T00:19:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cde1b6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd8156710 con 0x7f2cd806ee40 2023-10-25T00:19:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cd6ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2cd80a9820 0x7f2cd815af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36370/0 (socket says 172.21.15.107:36370) 2023-10-25T00:19:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cd6ffd640 1 -- 172.21.15.107:0/4233168618 learned_addr learned my addr 172.21.15.107:0/4233168618 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:19:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cd7fff640 1 --2- 172.21.15.107:0/4233168618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2cd806ee40 0x7f2cd80ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cd67fc640 1 -- 172.21.15.107:0/4233168618 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508952821 0 0) 0x7f2cd806f730 con 0x7f2cd80a9820 2023-10-25T00:19:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cd67fc640 1 -- 172.21.15.107:0/4233168618 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc0003530 con 0x7f2cd80a9820 2023-10-25T00:19:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cd67fc640 1 -- 172.21.15.107:0/4233168618 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273997769 0 0) 0x7f2cd8156a40 con 0x7f2cd814fd40 2023-10-25T00:19:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cd67fc640 1 -- 172.21.15.107:0/4233168618 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd806f730 con 0x7f2cd814fd40 2023-10-25T00:19:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cd7fff640 1 -- 172.21.15.107:0/4233168618 >> v1:172.21.15.110:6789/0 conn(0x7f2cd814fd40 legacy=0x7f2cd80c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cd7fff640 1 -- 172.21.15.107:0/4233168618 >> v1:172.21.15.107:6789/0 conn(0x7f2cd80a9820 legacy=0x7f2cd815af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.143+0000 7f2cd7fff640 1 -- 172.21.15.107:0/4233168618 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd806f980 con 0x7f2cd806ee40 2023-10-25T00:19:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.147+0000 7f2cd7fff640 1 --2- 172.21.15.107:0/4233168618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2cd806ee40 0x7f2cd80ab090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0009dc0 tx=0x7f2cd001c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5b061db47126382 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:08.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.147+0000 7f2cd67fc640 1 -- 172.21.15.107:0/4233168618 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00a5830 con 0x7f2cd806ee40 2023-10-25T00:19:08.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.147+0000 7f2cd67fc640 1 -- 172.21.15.107:0/4233168618 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd00a5990 con 0x7f2cd806ee40 2023-10-25T00:19:08.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.147+0000 7f2cd67fc640 1 -- 172.21.15.107:0/4233168618 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00a5d20 con 0x7f2cd806ee40 2023-10-25T00:19:08.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.147+0000 7f2cde1b6640 1 -- 172.21.15.107:0/4233168618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2cd806ee40 msgr2=0x7f2cd80ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:08.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.147+0000 7f2cde1b6640 1 --2- 172.21.15.107:0/4233168618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2cd806ee40 0x7f2cd80ab090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0009dc0 tx=0x7f2cd001c850 comp rx=0 tx=0).stop 2023-10-25T00:19:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.147+0000 7f2cde1b6640 1 -- 172.21.15.107:0/4233168618 shutdown_connections 2023-10-25T00:19:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.147+0000 7f2cde1b6640 1 --2- 172.21.15.107:0/4233168618 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2cd806ee40 0x7f2cd80ab090 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.147+0000 7f2cde1b6640 1 -- 172.21.15.107:0/4233168618 >> 172.21.15.107:0/4233168618 conn(0x7f2cd80c2950 msgr2=0x7f2cd80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.147+0000 7f2cde1b6640 1 -- 172.21.15.107:0/4233168618 shutdown_connections 2023-10-25T00:19:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.147+0000 7f2cde1b6640 1 -- 172.21.15.107:0/4233168618 wait complete. 2023-10-25T00:19:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cde1b6640 1 Processor -- start 2023-10-25T00:19:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cde1b6640 1 -- start start 2023-10-25T00:19:08.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cde1b6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cd806ee40 0x7f2cd80bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:08.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cde1b6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2cd814fd40 0x7f2cd814ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:08.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cde1b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd80c4f30 0x7f2cd814ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:08.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cde1b6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd8147ca0 con 0x7f2cd80c4f30 2023-10-25T00:19:08.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cde1b6640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd80c56f0 con 0x7f2cd806ee40 2023-10-25T00:19:08.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cde1b6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd80c0320 con 0x7f2cd814fd40 2023-10-25T00:19:08.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd80c4f30 0x7f2cd814ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:08.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd80c4f30 0x7f2cd814ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51666/0 (socket says 172.21.15.107:51666) 2023-10-25T00:19:08.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd7fff640 1 -- 172.21.15.107:0/3049659532 learned_addr learned my addr 172.21.15.107:0/3049659532 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:08.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd77fe640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cd806ee40 0x7f2cd80bfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:08.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd6ffd640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2cd814fd40 0x7f2cd814ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:08.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd7fff640 1 -- 172.21.15.107:0/3049659532 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2cd814fd40 msgr2=0x7f2cd814ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:08.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd7fff640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2cd814fd40 0x7f2cd814ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:08.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd7fff640 1 -- 172.21.15.107:0/3049659532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cd806ee40 msgr2=0x7f2cd80bfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:08.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd7fff640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cd806ee40 0x7f2cd80bfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:08.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd7fff640 1 -- 172.21.15.107:0/3049659532 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd0009a70 con 0x7f2cd80c4f30 2023-10-25T00:19:08.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd77fe640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cd806ee40 0x7f2cd80bfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:19:08.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd7fff640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd80c4f30 0x7f2cd814ae90 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f2cd001a480 tx=0x7f2cd00195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:08.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cbffff640 1 -- 172.21.15.107:0/3049659532 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00b9650 con 0x7f2cd80c4f30 2023-10-25T00:19:08.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cbffff640 1 -- 172.21.15.107:0/3049659532 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd00b97b0 con 0x7f2cd80c4f30 2023-10-25T00:19:08.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cde1b6640 1 -- 172.21.15.107:0/3049659532 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd80c5410 con 0x7f2cd80c4f30 2023-10-25T00:19:08.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cbffff640 1 -- 172.21.15.107:0/3049659532 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00b9b40 con 0x7f2cd80c4f30 2023-10-25T00:19:08.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cde1b6640 1 -- 172.21.15.107:0/3049659532 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd8149f30 con 0x7f2cd80c4f30 2023-10-25T00:19:08.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cde1b6640 1 -- 172.21.15.107:0/3049659532 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd80c24a0 con 0x7f2cd80c4f30 2023-10-25T00:19:08.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cbffff640 1 -- 172.21.15.107:0/3049659532 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2cd00b9ca0 con 0x7f2cd80c4f30 2023-10-25T00:19:08.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cbffff640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2cc4065e90 0x7f2cc4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:08.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cbffff640 1 -- 172.21.15.107:0/3049659532 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7f2cd011f9f0 con 0x7f2cd80c4f30 2023-10-25T00:19:08.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.151+0000 7f2cd77fe640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2cc4065e90 0x7f2cc4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:08.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.155+0000 7f2cd77fe640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2cc4065e90 0x7f2cc4068350 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0011f0 tx=0x7f2ccc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:08.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.159+0000 7f2cbffff640 1 -- 172.21.15.107:0/3049659532 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2cd00ebfb0 con 0x7f2cd80c4f30 2023-10-25T00:19:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.447+0000 7f2cde1b6640 1 -- 172.21.15.107:0/3049659532 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cd80746e0 con 0x7f2cd80c4f30 2023-10-25T00:19:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.447+0000 7f2cbffff640 1 -- 172.21.15.107:0/3049659532 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5013 (secure 0 0 0) 0x7f2cd00ef5d0 con 0x7f2cd80c4f30 2023-10-25T00:19:08.450 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:08.450 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:07.398158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":206,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:08.451 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 209 2023-10-25T00:19:08.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 -- 172.21.15.107:0/3049659532 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2cc4065e90 msgr2=0x7f2cc4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:08.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2cc4065e90 0x7f2cc4068350 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0011f0 tx=0x7f2ccc009040 comp rx=0 tx=0).stop 2023-10-25T00:19:08.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 -- 172.21.15.107:0/3049659532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd80c4f30 msgr2=0x7f2cd814ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:08.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd80c4f30 0x7f2cd814ae90 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f2cd001a480 tx=0x7f2cd00195f0 comp rx=0 tx=0).stop 2023-10-25T00:19:08.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 -- 172.21.15.107:0/3049659532 shutdown_connections 2023-10-25T00:19:08.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd80c4f30 0x7f2cd814ae90 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:08.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2cc4065e90 0x7f2cc4068350 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:08.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2cd814fd40 0x7f2cd814ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:08.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 --2- 172.21.15.107:0/3049659532 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cd806ee40 0x7f2cd80bfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:08.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 -- 172.21.15.107:0/3049659532 >> 172.21.15.107:0/3049659532 conn(0x7f2cd80c2950 msgr2=0x7f2cd814d390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:08.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 -- 172.21.15.107:0/3049659532 shutdown_connections 2023-10-25T00:19:08.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:08.451+0000 7f2cde1b6640 1 -- 172.21.15.107:0/3049659532 wait complete. 2023-10-25T00:19:08.477 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=7213 2023-10-25T00:19:08.477 INFO:tasks.mds_thrash.fs.[b]:waiting for 9 secs before reviving mds.c 2023-10-25T00:19:10.176 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:19:10.175+0000 7f4c8a28d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:19:17.479 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2023-10-25T00:19:17.479 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:19:17.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:19:17.482 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:19:17.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:17.563 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:19:17.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b4689640 1 Processor -- start 2023-10-25T00:19:17.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b4689640 1 -- start start 2023-10-25T00:19:17.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b4689640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05ac14c470 0x7f05ac14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:17.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b4689640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac155cd0 con 0x7f05ac06f490 2023-10-25T00:19:17.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b4689640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac1536b0 con 0x7f05ac152b30 2023-10-25T00:19:17.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b4689640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac1526e0 con 0x7f05ac14c470 2023-10-25T00:19:17.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b1c24640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f05ac06f490 0x7f05ac071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57668/0 (socket says 172.21.15.107:57668) 2023-10-25T00:19:17.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b1c24640 1 -- 172.21.15.107:0/3653892842 learned_addr learned my addr 172.21.15.107:0/3653892842 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:19:17.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b2425640 1 --2- 172.21.15.107:0/3653892842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05ac14c470 0x7f05ac14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:17.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b1423640 1 -- 172.21.15.107:0/3653892842 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080261110 0 0) 0x7f05ac155cd0 con 0x7f05ac06f490 2023-10-25T00:19:17.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b1423640 1 -- 172.21.15.107:0/3653892842 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0598003530 con 0x7f05ac06f490 2023-10-25T00:19:17.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b1423640 1 -- 172.21.15.107:0/3653892842 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390003620 0 0) 0x7f05a40014a0 con 0x7f05ac152b30 2023-10-25T00:19:17.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b1423640 1 -- 172.21.15.107:0/3653892842 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05ac155cd0 con 0x7f05ac152b30 2023-10-25T00:19:17.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b2425640 1 -- 172.21.15.107:0/3653892842 >> v1:172.21.15.110:6789/0 conn(0x7f05ac152b30 legacy=0x7f05ac152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:17.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b2425640 1 -- 172.21.15.107:0/3653892842 >> v1:172.21.15.107:6789/0 conn(0x7f05ac06f490 legacy=0x7f05ac071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:17.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b1423640 1 -- 172.21.15.107:0/3653892842 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 207586695 0 0) 0x7f0598003530 con 0x7f05ac06f490 2023-10-25T00:19:17.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.627+0000 7f05b2425640 1 -- 172.21.15.107:0/3653892842 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac0c4380 con 0x7f05ac14c470 2023-10-25T00:19:17.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.631+0000 7f05b2425640 1 --2- 172.21.15.107:0/3653892842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05ac14c470 0x7f05ac14c850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f05a80097f0 tx=0x7f05a801c2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a44fa5f3c0f4489e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:17.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.631+0000 7f05b1423640 1 -- 172.21.15.107:0/3653892842 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c001680 con 0x7f05ac14c470 2023-10-25T00:19:17.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.631+0000 7f05b1423640 1 -- 172.21.15.107:0/3653892842 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a80a7050 con 0x7f05ac14c470 2023-10-25T00:19:17.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.631+0000 7f05b1423640 1 -- 172.21.15.107:0/3653892842 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a80ab620 con 0x7f05ac14c470 2023-10-25T00:19:17.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 -- 172.21.15.107:0/3653892842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05ac14c470 msgr2=0x7f05ac14c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:17.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 --2- 172.21.15.107:0/3653892842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05ac14c470 0x7f05ac14c850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f05a80097f0 tx=0x7f05a801c2c0 comp rx=0 tx=0).stop 2023-10-25T00:19:17.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 -- 172.21.15.107:0/3653892842 shutdown_connections 2023-10-25T00:19:17.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 --2- 172.21.15.107:0/3653892842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05ac14c470 0x7f05ac14c850 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:17.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 -- 172.21.15.107:0/3653892842 >> 172.21.15.107:0/3653892842 conn(0x7f05ac070f60 msgr2=0x7f05ac071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:17.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 -- 172.21.15.107:0/3653892842 shutdown_connections 2023-10-25T00:19:17.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 -- 172.21.15.107:0/3653892842 wait complete. 2023-10-25T00:19:17.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 Processor -- start 2023-10-25T00:19:17.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 -- start start 2023-10-25T00:19:17.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05ac06f490 0x7f05ac1eff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05ac14c470 0x7f05ac145200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ac152b30 0x7f05ac0c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:17.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac06ae50 con 0x7f05ac152b30 2023-10-25T00:19:17.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac06b6f0 con 0x7f05ac14c470 2023-10-25T00:19:17.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b4689640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac06ce30 con 0x7f05ac06f490 2023-10-25T00:19:17.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b2c26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ac152b30 0x7f05ac0c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b2c26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ac152b30 0x7f05ac0c2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37398/0 (socket says 172.21.15.107:37398) 2023-10-25T00:19:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b2c26640 1 -- 172.21.15.107:0/2267421658 learned_addr learned my addr 172.21.15.107:0/2267421658 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b2c26640 1 -- 172.21.15.107:0/2267421658 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05ac06f490 msgr2=0x7f05ac1eff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:17.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b2c26640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05ac06f490 0x7f05ac1eff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:17.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b2c26640 1 -- 172.21.15.107:0/2267421658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05ac14c470 msgr2=0x7f05ac145200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:17.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b2c26640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05ac14c470 0x7f05ac145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:17.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b2c26640 1 -- 172.21.15.107:0/2267421658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a8009420 con 0x7f05ac152b30 2023-10-25T00:19:17.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.635+0000 7f05b2c26640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ac152b30 0x7f05ac0c2f60 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f05a400c790 tx=0x7f05a400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.639+0000 7f05a2ffd640 1 -- 172.21.15.107:0/2267421658 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a400cea0 con 0x7f05ac152b30 2023-10-25T00:19:17.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.639+0000 7f05a2ffd640 1 -- 172.21.15.107:0/2267421658 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a4003b80 con 0x7f05ac152b30 2023-10-25T00:19:17.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.639+0000 7f05b4689640 1 -- 172.21.15.107:0/2267421658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac06b030 con 0x7f05ac152b30 2023-10-25T00:19:17.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.639+0000 7f05a2ffd640 1 -- 172.21.15.107:0/2267421658 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a400a850 con 0x7f05ac152b30 2023-10-25T00:19:17.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.639+0000 7f05b4689640 1 -- 172.21.15.107:0/2267421658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac06c7e0 con 0x7f05ac152b30 2023-10-25T00:19:17.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.639+0000 7f05a0ff9640 1 -- 172.21.15.107:0/2267421658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0588005260 con 0x7f05ac152b30 2023-10-25T00:19:17.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.639+0000 7f05a2ffd640 1 -- 172.21.15.107:0/2267421658 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f05a400a9b0 con 0x7f05ac152b30 2023-10-25T00:19:17.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.643+0000 7f05a2ffd640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0590065eb0 0x7f0590068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:17.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.643+0000 7f05a2ffd640 1 -- 172.21.15.107:0/2267421658 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7f05a40845a0 con 0x7f05ac152b30 2023-10-25T00:19:17.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.643+0000 7f05b2425640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0590065eb0 0x7f0590068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:17.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.643+0000 7f05b2425640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0590065eb0 0x7f0590068370 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f05a80ad000 tx=0x7f05a8004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:17.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.647+0000 7f05a2ffd640 1 -- 172.21.15.107:0/2267421658 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f05a4050bb0 con 0x7f05ac152b30 2023-10-25T00:19:17.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.931+0000 7f05a0ff9640 1 -- 172.21.15.107:0/2267421658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0588005510 con 0x7f05ac152b30 2023-10-25T00:19:17.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.931+0000 7f05a2ffd640 1 -- 172.21.15.107:0/2267421658 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+5771 (secure 0 0 0) 0x7f05a40541d0 con 0x7f05ac152b30 2023-10-25T00:19:17.934 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:17.934 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:07.398158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":206,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:17.935 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 210 2023-10-25T00:19:17.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 -- 172.21.15.107:0/2267421658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0590065eb0 msgr2=0x7f0590068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:17.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0590065eb0 0x7f0590068370 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f05a80ad000 tx=0x7f05a8004740 comp rx=0 tx=0).stop 2023-10-25T00:19:17.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 -- 172.21.15.107:0/2267421658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ac152b30 msgr2=0x7f05ac0c2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:17.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ac152b30 0x7f05ac0c2f60 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f05a400c790 tx=0x7f05a400cc50 comp rx=0 tx=0).stop 2023-10-25T00:19:17.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 -- 172.21.15.107:0/2267421658 shutdown_connections 2023-10-25T00:19:17.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05ac152b30 0x7f05ac0c2f60 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:17.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f05ac14c470 0x7f05ac145200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:17.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0590065eb0 0x7f0590068370 secure :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f05a80ad000 tx=0x7f05a8004740 comp rx=0 tx=0).stop 2023-10-25T00:19:17.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 --2- 172.21.15.107:0/2267421658 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f05ac06f490 0x7f05ac1eff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:17.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 -- 172.21.15.107:0/2267421658 >> 172.21.15.107:0/2267421658 conn(0x7f05ac070f60 msgr2=0x7f05ac073410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:17.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 -- 172.21.15.107:0/2267421658 shutdown_connections 2023-10-25T00:19:17.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:17.935+0000 7f05a0ff9640 1 -- 172.21.15.107:0/2267421658 wait complete. 2023-10-25T00:19:17.966 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2023-10-25T00:19:17.967 INFO:tasks.mds_thrash.fs.[b]:waiting for 11 secs before thrashing 2023-10-25T00:19:28.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.115+0000 7ff61157b640 1 Processor -- start 2023-10-25T00:19:29.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.115+0000 7ff61157b640 1 -- start start 2023-10-25T00:19:29.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff61157b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff60c152040 0x7ff60c158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff61157b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff60c0adb80 0x7ff60c071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff61157b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c071fc0 0x7ff60c06e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff61157b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c1556e0 con 0x7ff60c0adb80 2023-10-25T00:19:29.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff61157b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c152420 con 0x7ff60c152040 2023-10-25T00:19:29.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff61157b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c154d40 con 0x7ff60c071fc0 2023-10-25T00:19:29.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff60a7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff60c0adb80 0x7ff60c071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:29.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff60a7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff60c0adb80 0x7ff60c071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48788/0 (socket says 172.21.15.107:48788) 2023-10-25T00:19:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff60a7fc640 1 -- 172.21.15.107:0/930942897 learned_addr learned my addr 172.21.15.107:0/930942897 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff60a7fc640 1 -- 172.21.15.107:0/930942897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c071fc0 msgr2=0x7ff60c06e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff60a7fc640 1 --2- 172.21.15.107:0/930942897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c071fc0 0x7ff60c06e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff60a7fc640 1 -- 172.21.15.107:0/930942897 >> v2:172.21.15.110:3300/0 conn(0x7ff60c152040 msgr2=0x7ff60c158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff60a7fc640 1 --2- 172.21.15.107:0/930942897 >> v2:172.21.15.110:3300/0 conn(0x7ff60c152040 0x7ff60c158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.119+0000 7ff60a7fc640 1 -- 172.21.15.107:0/930942897 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c155020 con 0x7ff60c0adb80 2023-10-25T00:19:29.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff60a7fc640 1 --2- 172.21.15.107:0/930942897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c0adb80 0x7ff60c071a80 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0669f0 tx=0x7ff60001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26fe45a1635785c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:29.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff609ffb640 1 -- 172.21.15.107:0/930942897 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000ab600 con 0x7ff60c0adb80 2023-10-25T00:19:29.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff609ffb640 1 -- 172.21.15.107:0/930942897 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6000a7050 con 0x7ff60c0adb80 2023-10-25T00:19:29.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff609ffb640 1 -- 172.21.15.107:0/930942897 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000ab980 con 0x7ff60c0adb80 2023-10-25T00:19:29.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff61157b640 1 -- 172.21.15.107:0/930942897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c0adb80 msgr2=0x7ff60c071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff61157b640 1 --2- 172.21.15.107:0/930942897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c0adb80 0x7ff60c071a80 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0669f0 tx=0x7ff60001c490 comp rx=0 tx=0).stop 2023-10-25T00:19:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff61157b640 1 -- 172.21.15.107:0/930942897 shutdown_connections 2023-10-25T00:19:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff61157b640 1 --2- 172.21.15.107:0/930942897 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c071fc0 0x7ff60c06e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff61157b640 1 --2- 172.21.15.107:0/930942897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c0adb80 0x7ff60c071a80 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff61157b640 1 --2- 172.21.15.107:0/930942897 >> v2:172.21.15.110:3300/0 conn(0x7ff60c152040 0x7ff60c158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff61157b640 1 -- 172.21.15.107:0/930942897 >> 172.21.15.107:0/930942897 conn(0x7ff60c070f60 msgr2=0x7ff60c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:29.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff61157b640 1 -- 172.21.15.107:0/930942897 shutdown_connections 2023-10-25T00:19:29.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff61157b640 1 -- 172.21.15.107:0/930942897 wait complete. 2023-10-25T00:19:29.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff61157b640 1 Processor -- start 2023-10-25T00:19:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.123+0000 7ff61157b640 1 -- start start 2023-10-25T00:19:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff61157b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c071fc0 0x7ff60c158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff61157b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0adb80 0x7ff60c14a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff61157b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c152040 0x7ff60c156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c152040 0x7ff60c156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c071fc0 0x7ff60c158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c152040 0x7ff60c156fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48790/0 (socket says 172.21.15.107:48790) 2023-10-25T00:19:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c071fc0 0x7ff60c158370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:35734/0 (socket says 172.21.15.107:35734) 2023-10-25T00:19:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60b7fe640 1 -- 172.21.15.107:0/3935080679 learned_addr learned my addr 172.21.15.107:0/3935080679 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:29.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c14aca0 con 0x7ff60c152040 2023-10-25T00:19:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c1f1070 con 0x7ff60c071fc0 2023-10-25T00:19:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c14baf0 con 0x7ff60c0adb80 2023-10-25T00:19:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60affd640 1 -- 172.21.15.107:0/3935080679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0adb80 msgr2=0x7ff60c14a760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:19:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60affd640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0adb80 0x7ff60c14a760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60affd640 1 -- 172.21.15.107:0/3935080679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c152040 msgr2=0x7ff60c156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:29.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60affd640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c152040 0x7ff60c156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60affd640 1 -- 172.21.15.107:0/3935080679 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6000094d0 con 0x7ff60c071fc0 2023-10-25T00:19:29.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60b7fe640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c152040 0x7ff60c156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:19:29.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff60affd640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c071fc0 0x7ff60c158370 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00b410 tx=0x7ff5fc00b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:29.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff5f37fe640 1 -- 172.21.15.107:0/3935080679 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc011830 con 0x7ff60c071fc0 2023-10-25T00:19:29.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff5f37fe640 1 -- 172.21.15.107:0/3935080679 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5fc011990 con 0x7ff60c071fc0 2023-10-25T00:19:29.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff5f37fe640 1 -- 172.21.15.107:0/3935080679 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc011d40 con 0x7ff60c071fc0 2023-10-25T00:19:29.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c1f0af0 con 0x7ff60c071fc0 2023-10-25T00:19:29.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.127+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c14b4a0 con 0x7ff60c071fc0 2023-10-25T00:19:29.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.131+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60c0746e0 con 0x7ff60c071fc0 2023-10-25T00:19:29.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.131+0000 7ff5f37fe640 1 -- 172.21.15.107:0/3935080679 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff5fc01a400 con 0x7ff60c071fc0 2023-10-25T00:19:29.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.131+0000 7ff5f37fe640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5f8065e60 0x7ff5f8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.131+0000 7ff5f37fe640 1 -- 172.21.15.107:0/3935080679 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7ff5fc084820 con 0x7ff60c071fc0 2023-10-25T00:19:29.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.131+0000 7ff60a7fc640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5f8065e60 0x7ff5f8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:29.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.131+0000 7ff60a7fc640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5f8065e60 0x7ff5f8068320 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff6000ad000 tx=0x7ff600008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:29.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.139+0000 7ff5f37fe640 1 -- 172.21.15.107:0/3935080679 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff5fc050e50 con 0x7ff60c071fc0 2023-10-25T00:19:29.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff60c0ad960 con 0x7ff60c071fc0 2023-10-25T00:19:29.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff5f37fe640 1 -- 172.21.15.107:0/3935080679 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+5771 (secure 0 0 0) 0x7ff5fc054470 con 0x7ff60c071fc0 2023-10-25T00:19:29.417 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:29.417 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:07.398158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":206,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:29.418 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 210 2023-10-25T00:19:29.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5f8065e60 msgr2=0x7ff5f8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:29.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5f8065e60 0x7ff5f8068320 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ff6000ad000 tx=0x7ff600008040 comp rx=0 tx=0).stop 2023-10-25T00:19:29.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c071fc0 msgr2=0x7ff60c158370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:29.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c071fc0 0x7ff60c158370 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00b410 tx=0x7ff5fc00b810 comp rx=0 tx=0).stop 2023-10-25T00:19:29.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 shutdown_connections 2023-10-25T00:19:29.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff60c152040 0x7ff60c156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff5f8065e60 0x7ff5f8068320 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff60c0adb80 0x7ff60c14a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 --2- 172.21.15.107:0/3935080679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff60c071fc0 0x7ff60c158370 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 >> 172.21.15.107:0/3935080679 conn(0x7ff60c070f60 msgr2=0x7ff60c14f260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:29.420 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 shutdown_connections 2023-10-25T00:19:29.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.415+0000 7ff61157b640 1 -- 172.21.15.107:0/3935080679 wait complete. 2023-10-25T00:19:29.438 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2023-10-25T00:19:29.438 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:19:29.438 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:19:29.450 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:19:29.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:29.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.623+0000 7fef3b08a640 1 Processor -- start 2023-10-25T00:19:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.623+0000 7fef3b08a640 1 -- start start 2023-10-25T00:19:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef3b08a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fef34158560 0x7fef340c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef3b08a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef34154860 con 0x7fef3413dc20 2023-10-25T00:19:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef3b08a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef340c1f10 con 0x7fef340720c0 2023-10-25T00:19:29.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef3b08a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef34154b20 con 0x7fef34158560 2023-10-25T00:19:29.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef39627640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fef34158560 0x7fef340c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:29.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef38e26640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fef3413dc20 0x7fef3406df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:43364/0 (socket says 172.21.15.107:43364) 2023-10-25T00:19:29.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef38e26640 1 -- 172.21.15.107:0/1813538683 learned_addr learned my addr 172.21.15.107:0/1813538683 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:19:29.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef39627640 1 -- 172.21.15.107:0/1813538683 >> v1:172.21.15.110:6789/0 conn(0x7fef340720c0 legacy=0x7fef34157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:29.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef39627640 1 -- 172.21.15.107:0/1813538683 >> v1:172.21.15.107:6789/0 conn(0x7fef3413dc20 legacy=0x7fef3406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:29.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef337fe640 1 -- 172.21.15.107:0/1813538683 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653361723 0 0) 0x7fef34154860 con 0x7fef3413dc20 2023-10-25T00:19:29.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef39627640 1 -- 172.21.15.107:0/1813538683 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef34154e00 con 0x7fef34158560 2023-10-25T00:19:29.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef39627640 1 --2- 172.21.15.107:0/1813538683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef34158560 0x7fef340c18a0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fef2c0097c0 tx=0x7fef2c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311cc92f6d462be8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:29.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef337fe640 1 -- 172.21.15.107:0/1813538683 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0ab600 con 0x7fef34158560 2023-10-25T00:19:29.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef337fe640 1 -- 172.21.15.107:0/1813538683 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2c0a7050 con 0x7fef34158560 2023-10-25T00:19:29.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.627+0000 7fef337fe640 1 -- 172.21.15.107:0/1813538683 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0ab980 con 0x7fef34158560 2023-10-25T00:19:29.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 -- 172.21.15.107:0/1813538683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef34158560 msgr2=0x7fef340c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:29.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 --2- 172.21.15.107:0/1813538683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef34158560 0x7fef340c18a0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fef2c0097c0 tx=0x7fef2c01c3f0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 -- 172.21.15.107:0/1813538683 shutdown_connections 2023-10-25T00:19:29.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 --2- 172.21.15.107:0/1813538683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef34158560 0x7fef340c18a0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 -- 172.21.15.107:0/1813538683 >> 172.21.15.107:0/1813538683 conn(0x7fef340c39a0 msgr2=0x7fef340c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:29.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 -- 172.21.15.107:0/1813538683 shutdown_connections 2023-10-25T00:19:29.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 -- 172.21.15.107:0/1813538683 wait complete. 2023-10-25T00:19:29.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 Processor -- start 2023-10-25T00:19:29.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 -- start start 2023-10-25T00:19:29.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef340720c0 0x7fef34136260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef3413dc20 0x7fef341535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef341367a0 0x7fef341ee280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef341edd80 con 0x7fef3413dc20 2023-10-25T00:19:29.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef341f03d0 con 0x7fef341367a0 2023-10-25T00:19:29.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef3b08a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef341f06b0 con 0x7fef340720c0 2023-10-25T00:19:29.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef38e26640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef340720c0 0x7fef34136260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:29.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef38e26640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef340720c0 0x7fef34136260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:42186/0 (socket says 172.21.15.107:42186) 2023-10-25T00:19:29.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef38e26640 1 -- 172.21.15.107:0/2904778804 learned_addr learned my addr 172.21.15.107:0/2904778804 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:29.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef38e26640 1 -- 172.21.15.107:0/2904778804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef341367a0 msgr2=0x7fef341ee280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:29.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef38e26640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef341367a0 0x7fef341ee280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef38e26640 1 -- 172.21.15.107:0/2904778804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef3413dc20 msgr2=0x7fef341535b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:19:29.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef38e26640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef3413dc20 0x7fef341535b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef38e26640 1 -- 172.21.15.107:0/2904778804 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c009420 con 0x7fef340720c0 2023-10-25T00:19:29.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.631+0000 7fef38e26640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef340720c0 0x7fef34136260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fef24001570 tx=0x7fef2400cd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:29.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.635+0000 7fef317fa640 1 -- 172.21.15.107:0/2904778804 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2400a640 con 0x7fef340720c0 2023-10-25T00:19:29.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.635+0000 7fef3b08a640 1 -- 172.21.15.107:0/2904778804 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef341f0990 con 0x7fef340720c0 2023-10-25T00:19:29.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.635+0000 7fef3b08a640 1 -- 172.21.15.107:0/2904778804 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef341544b0 con 0x7fef340720c0 2023-10-25T00:19:29.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.635+0000 7fef317fa640 1 -- 172.21.15.107:0/2904778804 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2400a7a0 con 0x7fef340720c0 2023-10-25T00:19:29.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.635+0000 7fef317fa640 1 -- 172.21.15.107:0/2904778804 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2400abd0 con 0x7fef340720c0 2023-10-25T00:19:29.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.635+0000 7fef3b08a640 1 -- 172.21.15.107:0/2904778804 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef18005260 con 0x7fef340720c0 2023-10-25T00:19:29.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.635+0000 7fef317fa640 1 -- 172.21.15.107:0/2904778804 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fef2400ad30 con 0x7fef340720c0 2023-10-25T00:19:29.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.635+0000 7fef317fa640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef14065e90 0x7fef14068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:29.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.635+0000 7fef317fa640 1 -- 172.21.15.107:0/2904778804 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7fef24084130 con 0x7fef340720c0 2023-10-25T00:19:29.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.635+0000 7fef33fff640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef14065e90 0x7fef14068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:29.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.635+0000 7fef33fff640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef14065e90 0x7fef14068350 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fef1c000f70 tx=0x7fef1c0079c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:29.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.647+0000 7fef317fa640 1 -- 172.21.15.107:0/2904778804 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fef24050740 con 0x7fef340720c0 2023-10-25T00:19:29.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.907+0000 7fef3b08a640 1 -- 172.21.15.107:0/2904778804 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef180057e0 con 0x7fef340720c0 2023-10-25T00:19:29.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef317fa640 1 -- 172.21.15.107:0/2904778804 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+5771 (secure 0 0 0) 0x7fef24018020 con 0x7fef340720c0 2023-10-25T00:19:29.911 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:29.911 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:07.398158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":206,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:29.912 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 210 2023-10-25T00:19:29.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 -- 172.21.15.107:0/2904778804 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef14065e90 msgr2=0x7fef14068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:29.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef14065e90 0x7fef14068350 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fef1c000f70 tx=0x7fef1c0079c0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 -- 172.21.15.107:0/2904778804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef340720c0 msgr2=0x7fef34136260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:29.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef340720c0 0x7fef34136260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fef24001570 tx=0x7fef2400cd20 comp rx=0 tx=0).stop 2023-10-25T00:19:29.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 -- 172.21.15.107:0/2904778804 shutdown_connections 2023-10-25T00:19:29.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef341367a0 0x7fef341ee280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef3413dc20 0x7fef341535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fef14065e90 0x7fef14068350 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 --2- 172.21.15.107:0/2904778804 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef340720c0 0x7fef34136260 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:29.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 -- 172.21.15.107:0/2904778804 >> 172.21.15.107:0/2904778804 conn(0x7fef340c39a0 msgr2=0x7fef340c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:29.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 -- 172.21.15.107:0/2904778804 shutdown_connections 2023-10-25T00:19:29.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:29.911+0000 7fef3b08a640 1 -- 172.21.15.107:0/2904778804 wait complete. 2023-10-25T00:19:29.933 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:19:31.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:32.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.083+0000 7f25aabd0640 1 Processor -- start 2023-10-25T00:19:32.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.083+0000 7f25aabd0640 1 -- start start 2023-10-25T00:19:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.083+0000 7f25aabd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a413dc20 0x7f25a40c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.083+0000 7f25aabd0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25a4157e50 0x7f25a41582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.083+0000 7f25aabd0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a4154860 con 0x7f25a40720c0 2023-10-25T00:19:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.083+0000 7f25aabd0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a4158800 con 0x7f25a4157e50 2023-10-25T00:19:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.083+0000 7f25aabd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a40c1f10 con 0x7f25a413dc20 2023-10-25T00:19:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.083+0000 7f25a896c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a413dc20 0x7f25a40c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.083+0000 7f25a916d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f25a4157e50 0x7f25a41582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:32.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.083+0000 7f25a896c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a413dc20 0x7f25a40c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:42198/0 (socket says 172.21.15.107:42198) 2023-10-25T00:19:32.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.083+0000 7f25a896c640 1 -- 172.21.15.107:0/2091021310 learned_addr learned my addr 172.21.15.107:0/2091021310 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:32.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.087+0000 7f25a896c640 1 -- 172.21.15.107:0/2091021310 >> v2:172.21.15.107:3300/0 conn(0x7f25a4157e50 msgr2=0x7f25a41582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:32.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.087+0000 7f25a896c640 1 --2- 172.21.15.107:0/2091021310 >> v2:172.21.15.107:3300/0 conn(0x7f25a4157e50 0x7f25a41582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.087+0000 7f25a896c640 1 -- 172.21.15.107:0/2091021310 >> v1:172.21.15.110:6789/0 conn(0x7f25a40720c0 legacy=0x7f25a406df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.087+0000 7f25a37fe640 1 -- 172.21.15.107:0/2091021310 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245863875 0 0) 0x7f25a4154860 con 0x7f25a40720c0 2023-10-25T00:19:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.087+0000 7f25a896c640 1 -- 172.21.15.107:0/2091021310 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a4154b20 con 0x7f25a413dc20 2023-10-25T00:19:32.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.087+0000 7f25a896c640 1 --2- 172.21.15.107:0/2091021310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a413dc20 0x7f25a40c18a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2598009870 tx=0x7f259801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3eead3105a1df1a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:32.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25a37fe640 1 -- 172.21.15.107:0/2091021310 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25980ac640 con 0x7f25a413dc20 2023-10-25T00:19:32.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25a37fe640 1 -- 172.21.15.107:0/2091021310 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25980ac7a0 con 0x7f25a413dc20 2023-10-25T00:19:32.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25a37fe640 1 -- 172.21.15.107:0/2091021310 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25980acb30 con 0x7f25a413dc20 2023-10-25T00:19:32.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 -- 172.21.15.107:0/2091021310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a413dc20 msgr2=0x7f25a40c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:32.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 --2- 172.21.15.107:0/2091021310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a413dc20 0x7f25a40c18a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2598009870 tx=0x7f259801c420 comp rx=0 tx=0).stop 2023-10-25T00:19:32.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 -- 172.21.15.107:0/2091021310 shutdown_connections 2023-10-25T00:19:32.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 --2- 172.21.15.107:0/2091021310 >> v2:172.21.15.107:3300/0 conn(0x7f25a4157e50 0x7f25a41582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:32.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 --2- 172.21.15.107:0/2091021310 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a413dc20 0x7f25a40c18a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 -- 172.21.15.107:0/2091021310 >> 172.21.15.107:0/2091021310 conn(0x7f25a40c39a0 msgr2=0x7f25a40c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 -- 172.21.15.107:0/2091021310 shutdown_connections 2023-10-25T00:19:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 -- 172.21.15.107:0/2091021310 wait complete. 2023-10-25T00:19:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 Processor -- start 2023-10-25T00:19:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 -- start start 2023-10-25T00:19:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25a40720c0 0x7f25a4135610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a413dc20 0x7f25a41efcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a4157e50 0x7f25a40ab970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a41563b0 con 0x7f25a413dc20 2023-10-25T00:19:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a41f0640 con 0x7f25a40720c0 2023-10-25T00:19:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25aabd0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a4156650 con 0x7f25a4157e50 2023-10-25T00:19:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a413dc20 0x7f25a41efcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a413dc20 0x7f25a41efcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48812/0 (socket says 172.21.15.107:48812) 2023-10-25T00:19:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25a896c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25a40720c0 0x7f25a4135610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:32.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25a3fff640 1 -- 172.21.15.107:0/2417204424 learned_addr learned my addr 172.21.15.107:0/2417204424 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:32.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.091+0000 7f25a916d640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a4157e50 0x7f25a40ab970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:32.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a3fff640 1 -- 172.21.15.107:0/2417204424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a4157e50 msgr2=0x7f25a40ab970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:32.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a3fff640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a4157e50 0x7f25a40ab970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:32.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a3fff640 1 -- 172.21.15.107:0/2417204424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25a40720c0 msgr2=0x7f25a4135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:32.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a3fff640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25a40720c0 0x7f25a4135610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:32.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a3fff640 1 -- 172.21.15.107:0/2417204424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25980094d0 con 0x7f25a413dc20 2023-10-25T00:19:32.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a3fff640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a413dc20 0x7f25a41efcf0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f2590001ec0 tx=0x7f259000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:32.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a17fa640 1 -- 172.21.15.107:0/2417204424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25900054e0 con 0x7f25a413dc20 2023-10-25T00:19:32.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a17fa640 1 -- 172.21.15.107:0/2417204424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2590003600 con 0x7f25a413dc20 2023-10-25T00:19:32.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25aabd0640 1 -- 172.21.15.107:0/2417204424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a4156930 con 0x7f25a413dc20 2023-10-25T00:19:32.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a17fa640 1 -- 172.21.15.107:0/2417204424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25900039b0 con 0x7f25a413dc20 2023-10-25T00:19:32.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25aabd0640 1 -- 172.21.15.107:0/2417204424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a41f0230 con 0x7f25a413dc20 2023-10-25T00:19:32.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25aabd0640 1 -- 172.21.15.107:0/2417204424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a40c4dd0 con 0x7f25a413dc20 2023-10-25T00:19:32.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a17fa640 1 -- 172.21.15.107:0/2417204424 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2590003b10 con 0x7f25a413dc20 2023-10-25T00:19:32.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a17fa640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2588065e60 0x7f2588068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:32.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a17fa640 1 -- 172.21.15.107:0/2417204424 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7f2590083e50 con 0x7f25a413dc20 2023-10-25T00:19:32.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a896c640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2588065e60 0x7f2588068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.095+0000 7f25a896c640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2588065e60 0x7f2588068320 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f25980ae000 tx=0x7f2598008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:32.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.103+0000 7f25a17fa640 1 -- 172.21.15.107:0/2417204424 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2590050490 con 0x7f25a413dc20 2023-10-25T00:19:32.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.367+0000 7f25aabd0640 1 -- 172.21.15.107:0/2417204424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25a4156b20 con 0x7f25a413dc20 2023-10-25T00:19:32.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.367+0000 7f25a17fa640 1 -- 172.21.15.107:0/2417204424 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+5771 (secure 0 0 0) 0x7f2590017070 con 0x7f25a413dc20 2023-10-25T00:19:32.369 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:32.369 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:07.398158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":206,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:32.370 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 210 2023-10-25T00:19:32.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 -- 172.21.15.107:0/2417204424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2588065e60 msgr2=0x7f2588068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:32.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2588065e60 0x7f2588068320 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f25980ae000 tx=0x7f2598008040 comp rx=0 tx=0).stop 2023-10-25T00:19:32.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 -- 172.21.15.107:0/2417204424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a413dc20 msgr2=0x7f25a41efcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:32.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a413dc20 0x7f25a41efcf0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f2590001ec0 tx=0x7f259000c8e0 comp rx=0 tx=0).stop 2023-10-25T00:19:32.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 -- 172.21.15.107:0/2417204424 shutdown_connections 2023-10-25T00:19:32.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2588065e60 0x7f2588068320 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:32.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25a4157e50 0x7f25a40ab970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:32.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25a413dc20 0x7f25a41efcf0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:32.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 --2- 172.21.15.107:0/2417204424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25a40720c0 0x7f25a4135610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:32.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 -- 172.21.15.107:0/2417204424 >> 172.21.15.107:0/2417204424 conn(0x7f25a40c39a0 msgr2=0x7f25a40c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:32.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 -- 172.21.15.107:0/2417204424 shutdown_connections 2023-10-25T00:19:32.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:32.371+0000 7f25aabd0640 1 -- 172.21.15.107:0/2417204424 wait complete. 2023-10-25T00:19:32.394 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:19:34.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:34.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.543+0000 7fa2219e0640 1 Processor -- start 2023-10-25T00:19:34.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.543+0000 7fa2219e0640 1 -- start start 2023-10-25T00:19:34.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa2219e0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa21c0c2990 0x7fa21c0aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:34.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa2219e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa21c0c20b0 0x7fa21c0c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:34.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa2219e0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c0a9aa0 con 0x7fa21c152060 2023-10-25T00:19:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa2219e0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21c155480 con 0x7fa21c0c20b0 2023-10-25T00:19:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa2219e0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa21c155760 con 0x7fa21c0c2990 2023-10-25T00:19:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa21b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa21c0c20b0 0x7fa21c0c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa21affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa21c152060 0x7fa21c0a9f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42536/0 (socket says 172.21.15.107:42536) 2023-10-25T00:19:34.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa21affd640 1 -- 172.21.15.107:0/2450068404 learned_addr learned my addr 172.21.15.107:0/2450068404 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:19:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa21b7fe640 1 -- 172.21.15.107:0/2450068404 >> v2:172.21.15.110:3300/0 conn(0x7fa21c0c2990 msgr2=0x7fa21c0aada0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:19:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa21b7fe640 1 --2- 172.21.15.107:0/2450068404 >> v2:172.21.15.110:3300/0 conn(0x7fa21c0c2990 0x7fa21c0aada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa21b7fe640 1 -- 172.21.15.107:0/2450068404 >> v1:172.21.15.107:6789/0 conn(0x7fa21c152060 legacy=0x7fa21c0a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:34.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.547+0000 7fa21b7fe640 1 -- 172.21.15.107:0/2450068404 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c0a9450 con 0x7fa21c0c20b0 2023-10-25T00:19:34.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa21b7fe640 1 --2- 172.21.15.107:0/2450068404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21c0c20b0 0x7fa21c0c1480 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fa21c155a40 tx=0x7fa21401c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e0221cac3ea0cb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa219ffb640 1 -- 172.21.15.107:0/2450068404 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2140a5860 con 0x7fa21c0c20b0 2023-10-25T00:19:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa219ffb640 1 -- 172.21.15.107:0/2450068404 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2140a59c0 con 0x7fa21c0c20b0 2023-10-25T00:19:34.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa219ffb640 1 -- 172.21.15.107:0/2450068404 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2140a5d50 con 0x7fa21c0c20b0 2023-10-25T00:19:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa2219e0640 1 -- 172.21.15.107:0/2450068404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21c0c20b0 msgr2=0x7fa21c0c1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:34.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa2219e0640 1 --2- 172.21.15.107:0/2450068404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21c0c20b0 0x7fa21c0c1480 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fa21c155a40 tx=0x7fa21401c7e0 comp rx=0 tx=0).stop 2023-10-25T00:19:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa2219e0640 1 -- 172.21.15.107:0/2450068404 shutdown_connections 2023-10-25T00:19:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa2219e0640 1 --2- 172.21.15.107:0/2450068404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21c0c20b0 0x7fa21c0c1480 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa2219e0640 1 --2- 172.21.15.107:0/2450068404 >> v2:172.21.15.110:3300/0 conn(0x7fa21c0c2990 0x7fa21c0aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa2219e0640 1 -- 172.21.15.107:0/2450068404 >> 172.21.15.107:0/2450068404 conn(0x7fa21c0c4a90 msgr2=0x7fa21c0c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:34.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa2219e0640 1 -- 172.21.15.107:0/2450068404 shutdown_connections 2023-10-25T00:19:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa2219e0640 1 -- 172.21.15.107:0/2450068404 wait complete. 2023-10-25T00:19:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa2219e0640 1 Processor -- start 2023-10-25T00:19:34.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.551+0000 7fa2219e0640 1 -- start start 2023-10-25T00:19:34.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa2219e0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa21c0c20b0 0x7fa21c145ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:34.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa2219e0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21c0c2990 0x7fa21c14eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:34.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa2219e0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa21c152060 0x7fa21c14eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:34.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa2219e0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c152a90 con 0x7fa21c0c2990 2023-10-25T00:19:34.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa2219e0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c146ea0 con 0x7fa21c152060 2023-10-25T00:19:34.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa2219e0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c147180 con 0x7fa21c0c20b0 2023-10-25T00:19:34.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa21a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21c0c2990 0x7fa21c14eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:34.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa21a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21c0c2990 0x7fa21c14eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59768/0 (socket says 172.21.15.107:59768) 2023-10-25T00:19:34.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa21a7fc640 1 -- 172.21.15.107:0/2156906932 learned_addr learned my addr 172.21.15.107:0/2156906932 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:34.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa21b7fe640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa21c152060 0x7fa21c14eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:34.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa21a7fc640 1 -- 172.21.15.107:0/2156906932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa21c0c20b0 msgr2=0x7fa21c145ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:34.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa21a7fc640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa21c0c20b0 0x7fa21c145ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:34.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa21a7fc640 1 -- 172.21.15.107:0/2156906932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa21c152060 msgr2=0x7fa21c14eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:34.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa21a7fc640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa21c152060 0x7fa21c14eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:34.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa21a7fc640 1 -- 172.21.15.107:0/2156906932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa214009a70 con 0x7fa21c0c2990 2023-10-25T00:19:34.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa21a7fc640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21c0c2990 0x7fa21c14eab0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fa21000b450 tx=0x7fa21000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:34.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa203fff640 1 -- 172.21.15.107:0/2156906932 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa210011890 con 0x7fa21c0c2990 2023-10-25T00:19:34.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa203fff640 1 -- 172.21.15.107:0/2156906932 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2100119f0 con 0x7fa21c0c2990 2023-10-25T00:19:34.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa2219e0640 1 -- 172.21.15.107:0/2156906932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21c147460 con 0x7fa21c0c2990 2023-10-25T00:19:34.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa2219e0640 1 -- 172.21.15.107:0/2156906932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21c1463e0 con 0x7fa21c0c2990 2023-10-25T00:19:34.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa203fff640 1 -- 172.21.15.107:0/2156906932 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa210011da0 con 0x7fa21c0c2990 2023-10-25T00:19:34.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa2219e0640 1 -- 172.21.15.107:0/2156906932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1fc005260 con 0x7fa21c0c2990 2023-10-25T00:19:34.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.555+0000 7fa203fff640 1 -- 172.21.15.107:0/2156906932 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa21001a400 con 0x7fa21c0c2990 2023-10-25T00:19:34.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.559+0000 7fa203fff640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa208065e60 0x7fa208068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:34.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.559+0000 7fa203fff640 1 -- 172.21.15.107:0/2156906932 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7fa210084c90 con 0x7fa21c0c2990 2023-10-25T00:19:34.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.559+0000 7fa21affd640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa208065e60 0x7fa208068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:34.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.559+0000 7fa21affd640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa208065e60 0x7fa208068320 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa21c15ebc0 tx=0x7fa20c007a50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:34.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.563+0000 7fa203fff640 1 -- 172.21.15.107:0/2156906932 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa210051220 con 0x7fa21c0c2990 2023-10-25T00:19:34.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 -- 172.21.15.107:0/2156906932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1fc005a90 con 0x7fa21c0c2990 2023-10-25T00:19:34.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa203fff640 1 -- 172.21.15.107:0/2156906932 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+5771 (secure 0 0 0) 0x7fa210054840 con 0x7fa21c0c2990 2023-10-25T00:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:07.398158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":206,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:34.820 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 210 2023-10-25T00:19:34.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 -- 172.21.15.107:0/2156906932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa208065e60 msgr2=0x7fa208068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:34.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa208065e60 0x7fa208068320 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa21c15ebc0 tx=0x7fa20c007a50 comp rx=0 tx=0).stop 2023-10-25T00:19:34.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 -- 172.21.15.107:0/2156906932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21c0c2990 msgr2=0x7fa21c14eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:34.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21c0c2990 0x7fa21c14eab0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fa21000b450 tx=0x7fa21000b910 comp rx=0 tx=0).stop 2023-10-25T00:19:34.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 -- 172.21.15.107:0/2156906932 shutdown_connections 2023-10-25T00:19:34.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa21c152060 0x7fa21c14eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:34.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa21c0c2990 0x7fa21c14eab0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:34.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa208065e60 0x7fa208068320 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:34.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 --2- 172.21.15.107:0/2156906932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa21c0c20b0 0x7fa21c145ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:34.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 -- 172.21.15.107:0/2156906932 >> 172.21.15.107:0/2156906932 conn(0x7fa21c0c4a90 msgr2=0x7fa21c0a9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:34.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 -- 172.21.15.107:0/2156906932 shutdown_connections 2023-10-25T00:19:34.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:34.819+0000 7fa2219e0640 1 -- 172.21.15.107:0/2156906932 wait complete. 2023-10-25T00:19:34.839 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:19:36.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:36.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc199bea640 1 Processor -- start 2023-10-25T00:19:36.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc199bea640 1 -- start start 2023-10-25T00:19:36.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc199bea640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc19414b600 0x7fc194151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:36.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc199bea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc19406af70 0x7fc19406c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:36.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc199bea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc19406a230 0x7fc1941551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:36.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc199bea640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1940c1860 con 0x7fc19414b600 2023-10-25T00:19:36.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc199bea640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc194154d50 con 0x7fc19406a230 2023-10-25T00:19:36.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc199bea640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1941524f0 con 0x7fc19406af70 2023-10-25T00:19:36.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc1937fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc19414b600 0x7fc194151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:36.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc1937fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc19414b600 0x7fc194151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59778/0 (socket says 172.21.15.107:59778) 2023-10-25T00:19:36.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc1937fe640 1 -- 172.21.15.107:0/1004523922 learned_addr learned my addr 172.21.15.107:0/1004523922 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:36.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc18bfff640 1 --2- 172.21.15.107:0/1004523922 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc19406af70 0x7fc19406c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:36.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc193fff640 1 --2- 172.21.15.107:0/1004523922 >> v2:172.21.15.110:3300/0 conn(0x7fc19406a230 0x7fc1941551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:36.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc1937fe640 1 -- 172.21.15.107:0/1004523922 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc19406af70 msgr2=0x7fc19406c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:36.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc1937fe640 1 --2- 172.21.15.107:0/1004523922 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc19406af70 0x7fc19406c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:36.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc1937fe640 1 -- 172.21.15.107:0/1004523922 >> v2:172.21.15.110:3300/0 conn(0x7fc19406a230 msgr2=0x7fc1941551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:36.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc1937fe640 1 --2- 172.21.15.107:0/1004523922 >> v2:172.21.15.110:3300/0 conn(0x7fc19406a230 0x7fc1941551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:36.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc1937fe640 1 -- 172.21.15.107:0/1004523922 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19414b9e0 con 0x7fc19414b600 2023-10-25T00:19:36.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc193fff640 1 --2- 172.21.15.107:0/1004523922 >> v2:172.21.15.110:3300/0 conn(0x7fc19406a230 0x7fc1941551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:19:36.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.971+0000 7fc18bfff640 1 --2- 172.21.15.107:0/1004523922 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc19406af70 0x7fc19406c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:19:36.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc1937fe640 1 --2- 172.21.15.107:0/1004523922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc19414b600 0x7fc194151fb0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fc19406d0a0 tx=0x7fc18401e650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1634cc8cefefa030 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:36.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc192ffd640 1 -- 172.21.15.107:0/1004523922 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1840ac670 con 0x7fc19414b600 2023-10-25T00:19:36.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc192ffd640 1 -- 172.21.15.107:0/1004523922 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1840ac7d0 con 0x7fc19414b600 2023-10-25T00:19:36.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc192ffd640 1 -- 172.21.15.107:0/1004523922 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1840acb60 con 0x7fc19414b600 2023-10-25T00:19:36.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc199bea640 1 -- 172.21.15.107:0/1004523922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc19414b600 msgr2=0x7fc194151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:36.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc199bea640 1 --2- 172.21.15.107:0/1004523922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc19414b600 0x7fc194151fb0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fc19406d0a0 tx=0x7fc18401e650 comp rx=0 tx=0).stop 2023-10-25T00:19:36.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc199bea640 1 -- 172.21.15.107:0/1004523922 shutdown_connections 2023-10-25T00:19:36.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc199bea640 1 --2- 172.21.15.107:0/1004523922 >> v2:172.21.15.110:3300/0 conn(0x7fc19406a230 0x7fc1941551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:36.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc199bea640 1 --2- 172.21.15.107:0/1004523922 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc19406af70 0x7fc19406c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:36.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc199bea640 1 --2- 172.21.15.107:0/1004523922 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc19414b600 0x7fc194151fb0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:36.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc199bea640 1 -- 172.21.15.107:0/1004523922 >> 172.21.15.107:0/1004523922 conn(0x7fc1940c2950 msgr2=0x7fc1940c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:36.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc199bea640 1 -- 172.21.15.107:0/1004523922 shutdown_connections 2023-10-25T00:19:36.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.975+0000 7fc199bea640 1 -- 172.21.15.107:0/1004523922 wait complete. 2023-10-25T00:19:36.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc199bea640 1 Processor -- start 2023-10-25T00:19:36.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc199bea640 1 -- start start 2023-10-25T00:19:36.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc199bea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc19406a230 0x7fc19415a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:36.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc199bea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc19406af70 0x7fc19413cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:36.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc199bea640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc19413d8b0 0x7fc19413d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:36.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc199bea640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc194146200 con 0x7fc19406af70 2023-10-25T00:19:36.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc199bea640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19413d6c0 con 0x7fc19413d8b0 2023-10-25T00:19:36.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc199bea640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19413ddb0 con 0x7fc19406a230 2023-10-25T00:19:36.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc18bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc19406af70 0x7fc19413cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:36.985 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc18bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc19406af70 0x7fc19413cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59790/0 (socket says 172.21.15.107:59790) 2023-10-25T00:19:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc18bfff640 1 -- 172.21.15.107:0/337055968 learned_addr learned my addr 172.21.15.107:0/337055968 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc193fff640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc19413d8b0 0x7fc19413d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc1937fe640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc19406a230 0x7fc19415a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc18bfff640 1 -- 172.21.15.107:0/337055968 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc19406a230 msgr2=0x7fc19415a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:36.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc18bfff640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc19406a230 0x7fc19415a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc18bfff640 1 -- 172.21.15.107:0/337055968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc19413d8b0 msgr2=0x7fc19413d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc18bfff640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc19413d8b0 0x7fc19413d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc18bfff640 1 -- 172.21.15.107:0/337055968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1840094d0 con 0x7fc19406af70 2023-10-25T00:19:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc18bfff640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc19406af70 0x7fc19413cc40 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fc17c00c7f0 tx=0x7fc17c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:36.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc190ff9640 1 -- 172.21.15.107:0/337055968 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c004220 con 0x7fc19406af70 2023-10-25T00:19:36.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc190ff9640 1 -- 172.21.15.107:0/337055968 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c004380 con 0x7fc19406af70 2023-10-25T00:19:36.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc199bea640 1 -- 172.21.15.107:0/337055968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19413fcf0 con 0x7fc19406af70 2023-10-25T00:19:36.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc190ff9640 1 -- 172.21.15.107:0/337055968 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c003840 con 0x7fc19406af70 2023-10-25T00:19:36.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc199bea640 1 -- 172.21.15.107:0/337055968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1941402b0 con 0x7fc19406af70 2023-10-25T00:19:36.988 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc199bea640 1 -- 172.21.15.107:0/337055968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1940746e0 con 0x7fc19406af70 2023-10-25T00:19:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc190ff9640 1 -- 172.21.15.107:0/337055968 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc17c0039a0 con 0x7fc19406af70 2023-10-25T00:19:36.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc190ff9640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc174065e90 0x7fc174068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:36.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.979+0000 7fc190ff9640 1 -- 172.21.15.107:0/337055968 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7fc17c084660 con 0x7fc19406af70 2023-10-25T00:19:36.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.983+0000 7fc1937fe640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc174065e90 0x7fc174068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:36.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.983+0000 7fc1937fe640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc174065e90 0x7fc174068350 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc1840b0000 tx=0x7fc1840a9000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:36.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:36.987+0000 7fc190ff9640 1 -- 172.21.15.107:0/337055968 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc17c050b60 con 0x7fc19406af70 2023-10-25T00:19:37.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.171+0000 7fc199bea640 1 -- 172.21.15.107:0/337055968 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc194069fa0 con 0x7fc19406af70 2023-10-25T00:19:37.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc190ff9640 1 -- 172.21.15.107:0/337055968 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+5771 (secure 0 0 0) 0x7fc17c01e020 con 0x7fc19406af70 2023-10-25T00:19:37.175 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:37.175 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:07.398158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":206,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:37.176 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 210 2023-10-25T00:19:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 -- 172.21.15.107:0/337055968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc174065e90 msgr2=0x7fc174068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc174065e90 0x7fc174068350 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc1840b0000 tx=0x7fc1840a9000 comp rx=0 tx=0).stop 2023-10-25T00:19:37.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 -- 172.21.15.107:0/337055968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc19406af70 msgr2=0x7fc19413cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc19406af70 0x7fc19413cc40 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fc17c00c7f0 tx=0x7fc17c00ccb0 comp rx=0 tx=0).stop 2023-10-25T00:19:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 -- 172.21.15.107:0/337055968 shutdown_connections 2023-10-25T00:19:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc19413d8b0 0x7fc19413d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc19406af70 0x7fc19413cc40 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc174065e90 0x7fc174068350 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:37.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 --2- 172.21.15.107:0/337055968 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc19406a230 0x7fc19415a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:37.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 -- 172.21.15.107:0/337055968 >> 172.21.15.107:0/337055968 conn(0x7fc1940c2950 msgr2=0x7fc1940c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:37.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 -- 172.21.15.107:0/337055968 shutdown_connections 2023-10-25T00:19:37.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:37.175+0000 7fc199bea640 1 -- 172.21.15.107:0/337055968 wait complete. 2023-10-25T00:19:37.194 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:19:39.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:39.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f8815e28640 1 Processor -- start 2023-10-25T00:19:39.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f8815e28640 1 -- start start 2023-10-25T00:19:39.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f8815e28640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f881014b600 0x7f8810151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:39.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f8815e28640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f881006a230 0x7f88101551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:39.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f8815e28640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100c1860 con 0x7f881006af70 2023-10-25T00:19:39.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f8815e28640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8810154d50 con 0x7f881014b600 2023-10-25T00:19:39.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f8815e28640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88101524f0 con 0x7f881006a230 2023-10-25T00:19:39.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f880f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f881014b600 0x7f8810151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:39.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f880f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f881014b600 0x7f8810151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59792/0 (socket says 172.21.15.107:59792) 2023-10-25T00:19:39.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f880f7fe640 1 -- 172.21.15.107:0/1517030932 learned_addr learned my addr 172.21.15.107:0/1517030932 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:39.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f880ffff640 1 --2- 172.21.15.107:0/1517030932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f881006a230 0x7f88101551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:39.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f880f7fe640 1 -- 172.21.15.107:0/1517030932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f881006a230 msgr2=0x7f88101551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:39.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f880f7fe640 1 --2- 172.21.15.107:0/1517030932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f881006a230 0x7f88101551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:39.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f880f7fe640 1 -- 172.21.15.107:0/1517030932 >> v1:172.21.15.107:6789/0 conn(0x7f881006af70 legacy=0x7f881006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:39.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f880f7fe640 1 -- 172.21.15.107:0/1517030932 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881014b9e0 con 0x7f881014b600 2023-10-25T00:19:39.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.327+0000 7f880effd640 1 -- 172.21.15.107:0/1517030932 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342812981 0 0) 0x7f88100c1860 con 0x7f881006af70 2023-10-25T00:19:39.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f880f7fe640 1 --2- 172.21.15.107:0/1517030932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881014b600 0x7f8810151fb0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f87f800bb10 tx=0x7f87f801eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=302a926cb53c8d91 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:39.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f880effd640 1 -- 172.21.15.107:0/1517030932 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f80a7a30 con 0x7f881014b600 2023-10-25T00:19:39.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f880effd640 1 -- 172.21.15.107:0/1517030932 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f80a7b90 con 0x7f881014b600 2023-10-25T00:19:39.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f880effd640 1 -- 172.21.15.107:0/1517030932 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f80af440 con 0x7f881014b600 2023-10-25T00:19:39.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 -- 172.21.15.107:0/1517030932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881014b600 msgr2=0x7f8810151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:39.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 --2- 172.21.15.107:0/1517030932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881014b600 0x7f8810151fb0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f87f800bb10 tx=0x7f87f801eb50 comp rx=0 tx=0).stop 2023-10-25T00:19:39.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 -- 172.21.15.107:0/1517030932 shutdown_connections 2023-10-25T00:19:39.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 --2- 172.21.15.107:0/1517030932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f881006a230 0x7f88101551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:39.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 --2- 172.21.15.107:0/1517030932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881014b600 0x7f8810151fb0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:39.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 -- 172.21.15.107:0/1517030932 >> 172.21.15.107:0/1517030932 conn(0x7f88100c2950 msgr2=0x7f88100c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:39.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 -- 172.21.15.107:0/1517030932 shutdown_connections 2023-10-25T00:19:39.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 -- 172.21.15.107:0/1517030932 wait complete. 2023-10-25T00:19:39.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 Processor -- start 2023-10-25T00:19:39.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 -- start start 2023-10-25T00:19:39.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f881006a230 0x7f8810145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:39.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881006af70 0x7f88101538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:39.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f881014b600 0x7f88100c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88101ec170 con 0x7f881006af70 2023-10-25T00:19:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88101e89a0 con 0x7f881006a230 2023-10-25T00:19:39.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88101ec660 con 0x7f881014b600 2023-10-25T00:19:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8806ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881006af70 0x7f88101538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8806ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881006af70 0x7f88101538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59800/0 (socket says 172.21.15.107:59800) 2023-10-25T00:19:39.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8806ffd640 1 -- 172.21.15.107:0/3479051529 learned_addr learned my addr 172.21.15.107:0/3479051529 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:39.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f880ffff640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f881014b600 0x7f88100c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:39.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f880f7fe640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f881006a230 0x7f8810145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:39.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8806ffd640 1 -- 172.21.15.107:0/3479051529 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f881014b600 msgr2=0x7f88100c4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:39.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8806ffd640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f881014b600 0x7f88100c4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:39.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8806ffd640 1 -- 172.21.15.107:0/3479051529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f881006a230 msgr2=0x7f8810145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:39.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8806ffd640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f881006a230 0x7f8810145370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:39.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8806ffd640 1 -- 172.21.15.107:0/3479051529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f800b770 con 0x7f881006af70 2023-10-25T00:19:39.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8806ffd640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881006af70 0x7f88101538c0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f880000c520 tx=0x7f880000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:39.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f880cff9640 1 -- 172.21.15.107:0/3479051529 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800003600 con 0x7f881006af70 2023-10-25T00:19:39.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f880cff9640 1 -- 172.21.15.107:0/3479051529 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8800003780 con 0x7f881006af70 2023-10-25T00:19:39.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f880cff9640 1 -- 172.21.15.107:0/3479051529 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8800003b30 con 0x7f881006af70 2023-10-25T00:19:39.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 -- 172.21.15.107:0/3479051529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88101e8390 con 0x7f881006af70 2023-10-25T00:19:39.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.331+0000 7f8815e28640 1 -- 172.21.15.107:0/3479051529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88101e8630 con 0x7f881006af70 2023-10-25T00:19:39.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.335+0000 7f8815e28640 1 -- 172.21.15.107:0/3479051529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88101e81f0 con 0x7f881006af70 2023-10-25T00:19:39.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.335+0000 7f880cff9640 1 -- 172.21.15.107:0/3479051529 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8800003c90 con 0x7f881006af70 2023-10-25T00:19:39.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.335+0000 7f880cff9640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87f0065e60 0x7f87f0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:39.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.335+0000 7f880cff9640 1 -- 172.21.15.107:0/3479051529 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7f8800083f20 con 0x7f881006af70 2023-10-25T00:19:39.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.335+0000 7f880f7fe640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87f0065e60 0x7f87f0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:39.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.335+0000 7f880f7fe640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87f0065e60 0x7f87f0068320 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f87f80b1000 tx=0x7f87f80a9070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:39.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.339+0000 7f880cff9640 1 -- 172.21.15.107:0/3479051529 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f88101e81f0 con 0x7f881006af70 2023-10-25T00:19:39.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 -- 172.21.15.107:0/3479051529 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88101e81f0 con 0x7f881006af70 2023-10-25T00:19:39.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f880cff9640 1 -- 172.21.15.107:0/3479051529 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+5771 (secure 0 0 0) 0x7f88101e81f0 con 0x7f881006af70 2023-10-25T00:19:39.528 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:39.528 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:07.398158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":206,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:39.529 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 210 2023-10-25T00:19:39.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 -- 172.21.15.107:0/3479051529 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87f0065e60 msgr2=0x7f87f0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:39.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87f0065e60 0x7f87f0068320 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f87f80b1000 tx=0x7f87f80a9070 comp rx=0 tx=0).stop 2023-10-25T00:19:39.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 -- 172.21.15.107:0/3479051529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881006af70 msgr2=0x7f88101538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881006af70 0x7f88101538c0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f880000c520 tx=0x7f880000c9e0 comp rx=0 tx=0).stop 2023-10-25T00:19:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 -- 172.21.15.107:0/3479051529 shutdown_connections 2023-10-25T00:19:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87f0065e60 0x7f87f0068320 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f881014b600 0x7f88100c4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881006af70 0x7f88101538c0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:39.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 --2- 172.21.15.107:0/3479051529 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f881006a230 0x7f8810145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:39.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 -- 172.21.15.107:0/3479051529 >> 172.21.15.107:0/3479051529 conn(0x7f88100c2950 msgr2=0x7f881006b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:39.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 -- 172.21.15.107:0/3479051529 shutdown_connections 2023-10-25T00:19:39.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:39.527+0000 7f8815e28640 1 -- 172.21.15.107:0/3479051529 wait complete. 2023-10-25T00:19:39.548 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:19:41.549 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd7845640 1 Processor -- start 2023-10-25T00:19:41.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd7845640 1 -- start start 2023-10-25T00:19:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd7845640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd014fd40 0x7f5fd00ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd7845640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5fd00a9820 0x7f5fd00aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd7845640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fd006f730 con 0x7f5fd006ee40 2023-10-25T00:19:41.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd7845640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fd00aa750 con 0x7f5fd00a9820 2023-10-25T00:19:41.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd7845640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd0156a40 con 0x7f5fd014fd40 2023-10-25T00:19:41.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd5de2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5fd006ee40 0x7f5fd00c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42552/0 (socket says 172.21.15.107:42552) 2023-10-25T00:19:41.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd5de2640 1 -- 172.21.15.107:0/2975121550 learned_addr learned my addr 172.21.15.107:0/2975121550 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:19:41.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd55e1640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd014fd40 0x7f5fd00ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:41.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd4de0640 1 --2- 172.21.15.107:0/2975121550 >> v2:172.21.15.110:3300/0 conn(0x7f5fd00a9820 0x7f5fd00aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:41.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704837725 0 0) 0x7f5fd006f730 con 0x7f5fd006ee40 2023-10-25T00:19:41.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fbc003530 con 0x7f5fd006ee40 2023-10-25T00:19:41.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336668129 0 0) 0x7f5fbc003530 con 0x7f5fd006ee40 2023-10-25T00:19:41.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd014fd40 msgr2=0x7f5fd00ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:41.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd014fd40 0x7f5fd00ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 >> v2:172.21.15.110:3300/0 conn(0x7f5fd00a9820 msgr2=0x7f5fd00aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 --2- 172.21.15.107:0/2975121550 >> v2:172.21.15.110:3300/0 conn(0x7f5fd00a9820 0x7f5fd00aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd0156710 con 0x7f5fd006ee40 2023-10-25T00:19:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5fc8001a00 con 0x7f5fd006ee40 2023-10-25T00:19:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 >> v1:172.21.15.107:6789/0 conn(0x7f5fd006ee40 legacy=0x7f5fd00c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:41.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd014fd40 0x7f5fbc003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fbc003f00 0x7f5fbc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fbc004d10 0x7f5fbc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd00aa750 con 0x7f5fbc004d10 2023-10-25T00:19:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd0156a40 con 0x7f5fbc003f00 2023-10-25T00:19:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fbc00bb20 con 0x7f5fd014fd40 2023-10-25T00:19:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd5de2640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fbc004d10 0x7f5fbc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd55e1640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd014fd40 0x7f5fbc003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd4de0640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fbc003f00 0x7f5fbc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd5de2640 1 -- 172.21.15.107:0/2975121550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd014fd40 msgr2=0x7f5fbc003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:41.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd5de2640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fd014fd40 0x7f5fbc003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd5de2640 1 -- 172.21.15.107:0/2975121550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fbc003f00 msgr2=0x7f5fbc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd5de2640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fbc003f00 0x7f5fbc0047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:41.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.683+0000 7f5fd5de2640 1 -- 172.21.15.107:0/2975121550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc003530 con 0x7f5fbc004d10 2023-10-25T00:19:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd5de2640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fbc004d10 0x7f5fbc005190 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800a840 tx=0x7f5fc801d310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cbbef90d9ebc839 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc80ad630 con 0x7f5fbc004d10 2023-10-25T00:19:41.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc80a8070 con 0x7f5fbc004d10 2023-10-25T00:19:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fc7fff640 1 -- 172.21.15.107:0/2975121550 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc80ad9f0 con 0x7f5fbc004d10 2023-10-25T00:19:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 -- 172.21.15.107:0/2975121550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fbc004d10 msgr2=0x7f5fbc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 --2- 172.21.15.107:0/2975121550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fbc004d10 0x7f5fbc005190 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800a840 tx=0x7f5fc801d310 comp rx=0 tx=0).stop 2023-10-25T00:19:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd5de2640 1 -- 172.21.15.107:0/2975121550 reap_dead start 2023-10-25T00:19:41.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 -- 172.21.15.107:0/2975121550 shutdown_connections 2023-10-25T00:19:41.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 -- 172.21.15.107:0/2975121550 >> 172.21.15.107:0/2975121550 conn(0x7f5fd00c2950 msgr2=0x7f5fd00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:41.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 -- 172.21.15.107:0/2975121550 shutdown_connections 2023-10-25T00:19:41.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 -- 172.21.15.107:0/2975121550 wait complete. 2023-10-25T00:19:41.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 Processor -- start 2023-10-25T00:19:41.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 -- start start 2023-10-25T00:19:41.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fbc004d10 0x7f5fd00c52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:41.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd015a6b0 0x7f5fd014c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:41.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd015ab30 0x7f5fd014cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:41.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd01eb5c0 con 0x7f5fd015a6b0 2023-10-25T00:19:41.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd01eb8a0 con 0x7f5fd015ab30 2023-10-25T00:19:41.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd7845640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd01ebb80 con 0x7f5fbc004d10 2023-10-25T00:19:41.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd4de0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd015a6b0 0x7f5fd014c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:41.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd4de0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd015a6b0 0x7f5fd014c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59824/0 (socket says 172.21.15.107:59824) 2023-10-25T00:19:41.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd4de0640 1 -- 172.21.15.107:0/1768462935 learned_addr learned my addr 172.21.15.107:0/1768462935 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:41.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd55e1640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fbc004d10 0x7f5fd00c52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:41.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.687+0000 7f5fd5de2640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd015ab30 0x7f5fd014cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:41.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd4de0640 1 -- 172.21.15.107:0/1768462935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fbc004d10 msgr2=0x7f5fd00c52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:41.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd4de0640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fbc004d10 0x7f5fd00c52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:41.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd4de0640 1 -- 172.21.15.107:0/1768462935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd015ab30 msgr2=0x7f5fd014cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:41.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd4de0640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd015ab30 0x7f5fd014cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:41.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd4de0640 1 -- 172.21.15.107:0/1768462935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd01ebe60 con 0x7f5fd015a6b0 2023-10-25T00:19:41.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd55e1640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fbc004d10 0x7f5fd00c52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:19:41.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd4de0640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd015a6b0 0x7f5fd014c7f0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f5fc000e810 tx=0x7f5fc000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:41.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fc5ffb640 1 -- 172.21.15.107:0/1768462935 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc000c6d0 con 0x7f5fd015a6b0 2023-10-25T00:19:41.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fc5ffb640 1 -- 172.21.15.107:0/1768462935 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc000c830 con 0x7f5fd015a6b0 2023-10-25T00:19:41.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd7845640 1 -- 172.21.15.107:0/1768462935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd00c5800 con 0x7f5fd015a6b0 2023-10-25T00:19:41.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fc5ffb640 1 -- 172.21.15.107:0/1768462935 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc000cbe0 con 0x7f5fd015a6b0 2023-10-25T00:19:41.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd7845640 1 -- 172.21.15.107:0/1768462935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd014f420 con 0x7f5fd015a6b0 2023-10-25T00:19:41.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd7845640 1 -- 172.21.15.107:0/1768462935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd00c24a0 con 0x7f5fd015a6b0 2023-10-25T00:19:41.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fc5ffb640 1 -- 172.21.15.107:0/1768462935 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5fc000cd40 con 0x7f5fd015a6b0 2023-10-25T00:19:41.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fc5ffb640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5fb4065e90 0x7f5fb4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:41.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fc5ffb640 1 -- 172.21.15.107:0/1768462935 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7f5fc00846b0 con 0x7f5fd015a6b0 2023-10-25T00:19:41.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd55e1640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5fb4065e90 0x7f5fb4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:41.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.691+0000 7f5fd55e1640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5fb4065e90 0x7f5fb4068350 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009af0 tx=0x7f5fcc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:41.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.695+0000 7f5fc5ffb640 1 -- 172.21.15.107:0/1768462935 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5fc0050c40 con 0x7f5fd015a6b0 2023-10-25T00:19:41.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.899+0000 7f5fd7845640 1 -- 172.21.15.107:0/1768462935 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fd01eb700 con 0x7f5fd015a6b0 2023-10-25T00:19:41.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.899+0000 7f5fc5ffb640 1 -- 172.21.15.107:0/1768462935 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+5771 (secure 0 0 0) 0x7f5fd01eb700 con 0x7f5fd015a6b0 2023-10-25T00:19:41.903 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:41.903 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8712,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:07.398158+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8649},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8649":{"gid":8649,"name":"a","rank":0,"incarnation":206,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6835/3396644475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3396644475},{"type":"v1","addr":"172.21.15.107:6835","nonce":3396644475}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:41.903 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 210 2023-10-25T00:19:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 -- 172.21.15.107:0/1768462935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5fb4065e90 msgr2=0x7f5fb4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5fb4065e90 0x7f5fb4068350 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc009af0 tx=0x7f5fcc008040 comp rx=0 tx=0).stop 2023-10-25T00:19:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 -- 172.21.15.107:0/1768462935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd015a6b0 msgr2=0x7f5fd014c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:41.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd015a6b0 0x7f5fd014c7f0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f5fc000e810 tx=0x7f5fc000ecd0 comp rx=0 tx=0).stop 2023-10-25T00:19:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 -- 172.21.15.107:0/1768462935 shutdown_connections 2023-10-25T00:19:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5fd015ab30 0x7f5fd014cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fd015a6b0 0x7f5fd014c7f0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5fb4065e90 0x7f5fb4068350 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 --2- 172.21.15.107:0/1768462935 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5fbc004d10 0x7f5fd00c52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:41.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 -- 172.21.15.107:0/1768462935 >> 172.21.15.107:0/1768462935 conn(0x7f5fd00c2950 msgr2=0x7f5fd0151140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:41.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 -- 172.21.15.107:0/1768462935 shutdown_connections 2023-10-25T00:19:41.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:41.903+0000 7f5fd7845640 1 -- 172.21.15.107:0/1768462935 wait complete. 2023-10-25T00:19:41.922 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:19:43.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:44.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.055+0000 7fa94554f640 1 Processor -- start 2023-10-25T00:19:44.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.055+0000 7fa94554f640 1 -- start start 2023-10-25T00:19:44.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa94554f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa940071220 0x7fa94006bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa94554f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa940151960 0x7fa940071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa94554f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9400c24c0 0x7fa94006a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa94554f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa94006c7f0 con 0x7fa940151960 2023-10-25T00:19:44.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa94554f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa940150bc0 con 0x7fa940071220 2023-10-25T00:19:44.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa94554f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94006c510 con 0x7fa9400c24c0 2023-10-25T00:19:44.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93e7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa940151960 0x7fa940071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93e7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa940151960 0x7fa940071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44404/0 (socket says 172.21.15.107:44404) 2023-10-25T00:19:44.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93e7fc640 1 -- 172.21.15.107:0/2005284100 learned_addr learned my addr 172.21.15.107:0/2005284100 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:44.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93f7fe640 1 --2- 172.21.15.107:0/2005284100 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9400c24c0 0x7fa94006a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93effd640 1 --2- 172.21.15.107:0/2005284100 >> v2:172.21.15.110:3300/0 conn(0x7fa940071220 0x7fa94006bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93e7fc640 1 -- 172.21.15.107:0/2005284100 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9400c24c0 msgr2=0x7fa94006a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:44.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93e7fc640 1 --2- 172.21.15.107:0/2005284100 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9400c24c0 0x7fa94006a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93e7fc640 1 -- 172.21.15.107:0/2005284100 >> v2:172.21.15.110:3300/0 conn(0x7fa940071220 msgr2=0x7fa94006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:44.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93e7fc640 1 --2- 172.21.15.107:0/2005284100 >> v2:172.21.15.110:3300/0 conn(0x7fa940071220 0x7fa94006bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93e7fc640 1 -- 172.21.15.107:0/2005284100 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa940150e80 con 0x7fa940151960 2023-10-25T00:19:44.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93e7fc640 1 --2- 172.21.15.107:0/2005284100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa940151960 0x7fa940071a90 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fa94006d0a0 tx=0x7fa93401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647982f239235f42 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:44.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93dffb640 1 -- 172.21.15.107:0/2005284100 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9340ab600 con 0x7fa940151960 2023-10-25T00:19:44.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93dffb640 1 -- 172.21.15.107:0/2005284100 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9340a7050 con 0x7fa940151960 2023-10-25T00:19:44.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.059+0000 7fa93dffb640 1 -- 172.21.15.107:0/2005284100 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9340ab980 con 0x7fa940151960 2023-10-25T00:19:44.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 -- 172.21.15.107:0/2005284100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa940151960 msgr2=0x7fa940071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:44.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 --2- 172.21.15.107:0/2005284100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa940151960 0x7fa940071a90 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fa94006d0a0 tx=0x7fa93401c470 comp rx=0 tx=0).stop 2023-10-25T00:19:44.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 -- 172.21.15.107:0/2005284100 shutdown_connections 2023-10-25T00:19:44.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 --2- 172.21.15.107:0/2005284100 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9400c24c0 0x7fa94006a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 --2- 172.21.15.107:0/2005284100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa940151960 0x7fa940071a90 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 --2- 172.21.15.107:0/2005284100 >> v2:172.21.15.110:3300/0 conn(0x7fa940071220 0x7fa94006bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 -- 172.21.15.107:0/2005284100 >> 172.21.15.107:0/2005284100 conn(0x7fa9400c2950 msgr2=0x7fa9400c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:44.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 -- 172.21.15.107:0/2005284100 shutdown_connections 2023-10-25T00:19:44.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 -- 172.21.15.107:0/2005284100 wait complete. 2023-10-25T00:19:44.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 Processor -- start 2023-10-25T00:19:44.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 -- start start 2023-10-25T00:19:44.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa940071220 0x7fa9400bf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9400c24c0 0x7fa940142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa940151960 0x7fa9400c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9401edf40 con 0x7fa940071220 2023-10-25T00:19:44.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9401ed330 con 0x7fa940151960 2023-10-25T00:19:44.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9401ed610 con 0x7fa9400c24c0 2023-10-25T00:19:44.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa940071220 0x7fa9400bf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa940071220 0x7fa9400bf4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44408/0 (socket says 172.21.15.107:44408) 2023-10-25T00:19:44.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93effd640 1 -- 172.21.15.107:0/4164927069 learned_addr learned my addr 172.21.15.107:0/4164927069 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:44.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93f7fe640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa940151960 0x7fa9400c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93e7fc640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9400c24c0 0x7fa940142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93effd640 1 -- 172.21.15.107:0/4164927069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9400c24c0 msgr2=0x7fa940142930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:44.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93effd640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9400c24c0 0x7fa940142930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93effd640 1 -- 172.21.15.107:0/4164927069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa940151960 msgr2=0x7fa9400c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:44.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93effd640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa940151960 0x7fa9400c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93effd640 1 -- 172.21.15.107:0/4164927069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9340094d0 con 0x7fa940071220 2023-10-25T00:19:44.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93f7fe640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa940151960 0x7fa9400c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:19:44.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa93effd640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa940071220 0x7fa9400bf4e0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fa930007ef0 tx=0x7fa93000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:44.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa9277fe640 1 -- 172.21.15.107:0/4164927069 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa930014620 con 0x7fa940071220 2023-10-25T00:19:44.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa9277fe640 1 -- 172.21.15.107:0/4164927069 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa930014780 con 0x7fa940071220 2023-10-25T00:19:44.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 -- 172.21.15.107:0/4164927069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9401ed8f0 con 0x7fa940071220 2023-10-25T00:19:44.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa9277fe640 1 -- 172.21.15.107:0/4164927069 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa930014b30 con 0x7fa940071220 2023-10-25T00:19:44.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.063+0000 7fa94554f640 1 -- 172.21.15.107:0/4164927069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9400bfa20 con 0x7fa940071220 2023-10-25T00:19:44.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.067+0000 7fa94554f640 1 -- 172.21.15.107:0/4164927069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9400746e0 con 0x7fa940071220 2023-10-25T00:19:44.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.067+0000 7fa9277fe640 1 -- 172.21.15.107:0/4164927069 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa930014c90 con 0x7fa940071220 2023-10-25T00:19:44.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.067+0000 7fa9277fe640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa92c065e60 0x7fa92c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.067+0000 7fa9277fe640 1 -- 172.21.15.107:0/4164927069 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 10401+0+0 (secure 0 0 0) 0x7fa930085300 con 0x7fa940071220 2023-10-25T00:19:44.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.067+0000 7fa93e7fc640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa92c065e60 0x7fa92c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.067+0000 7fa93e7fc640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa92c065e60 0x7fa92c068320 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa9340ad000 tx=0x7fa9340a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:44.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.071+0000 7fa9277fe640 1 -- 172.21.15.107:0/4164927069 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa930051890 con 0x7fa940071220 2023-10-25T00:19:44.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.275+0000 7fa94554f640 1 -- 172.21.15.107:0/4164927069 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9400aa910 con 0x7fa940071220 2023-10-25T00:19:44.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.275+0000 7fa9277fe640 1 -- 172.21.15.107:0/4164927069 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+5010 (secure 0 0 0) 0x7fa930054eb0 con 0x7fa940071220 2023-10-25T00:19:44.279 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:44.279 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:44.137763+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 211 2023-10-25T00:19:44.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 -- 172.21.15.107:0/4164927069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa92c065e60 msgr2=0x7fa92c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa92c065e60 0x7fa92c068320 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fa9340ad000 tx=0x7fa9340a6000 comp rx=0 tx=0).stop 2023-10-25T00:19:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 -- 172.21.15.107:0/4164927069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa940071220 msgr2=0x7fa9400bf4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa940071220 0x7fa9400bf4e0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fa930007ef0 tx=0x7fa93000e4b0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 -- 172.21.15.107:0/4164927069 shutdown_connections 2023-10-25T00:19:44.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa940151960 0x7fa9400c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa92c065e60 0x7fa92c068320 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa9400c24c0 0x7fa940142930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 --2- 172.21.15.107:0/4164927069 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa940071220 0x7fa9400bf4e0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 -- 172.21.15.107:0/4164927069 >> 172.21.15.107:0/4164927069 conn(0x7fa9400c2950 msgr2=0x7fa94015be10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:44.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 -- 172.21.15.107:0/4164927069 shutdown_connections 2023-10-25T00:19:44.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.279+0000 7fa94554f640 1 -- 172.21.15.107:0/4164927069 wait complete. 2023-10-25T00:19:44.297 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2023-10-25T00:19:44.297 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:19:44.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:44.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f73fc8640 1 Processor -- start 2023-10-25T00:19:44.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f73fc8640 1 -- start start 2023-10-25T00:19:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f73fc8640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6f6c0c5c20 0x7f6f6c153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f73fc8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f6c0c18a0 0x7f6f6c0c1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f73fc8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f6c072330 con 0x7f6f6c14fc30 2023-10-25T00:19:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f73fc8640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6c155ca0 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f73fc8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c155f80 con 0x7f6f6c0c18a0 2023-10-25T00:19:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f71563640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6f6c0c5c20 0x7f6f6c153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f71d64640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6f6c14fc30 0x7f6f6c06df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50918/0 (socket says 172.21.15.107:50918) 2023-10-25T00:19:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f71d64640 1 -- 172.21.15.107:0/3973856376 learned_addr learned my addr 172.21.15.107:0/3973856376 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:19:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f72565640 1 --2- 172.21.15.107:0/3973856376 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f6c0c18a0 0x7f6f6c0c1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f71563640 1 -- 172.21.15.107:0/3973856376 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f6c0c18a0 msgr2=0x7f6f6c0c1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f71563640 1 --2- 172.21.15.107:0/3973856376 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f6c0c18a0 0x7f6f6c0c1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f71563640 1 -- 172.21.15.107:0/3973856376 >> v1:172.21.15.107:6789/0 conn(0x7f6f6c14fc30 legacy=0x7f6f6c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f71563640 1 -- 172.21.15.107:0/3973856376 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6c156260 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.419+0000 7f6f72565640 1 --2- 172.21.15.107:0/3973856376 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f6c0c18a0 0x7f6f6c0c1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:19:44.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f71563640 1 --2- 172.21.15.107:0/3973856376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f6c0c5c20 0x7f6f6c153190 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0097c0 tx=0x7f6f5c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1300a59c52ef49e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:44.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f70d62640 1 -- 172.21.15.107:0/3973856376 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c0ac620 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f70d62640 1 -- 172.21.15.107:0/3973856376 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5c0ac780 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f70d62640 1 -- 172.21.15.107:0/3973856376 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c0acb10 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 -- 172.21.15.107:0/3973856376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f6c0c5c20 msgr2=0x7f6f6c153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:44.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 --2- 172.21.15.107:0/3973856376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f6c0c5c20 0x7f6f6c153190 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0097c0 tx=0x7f6f5c01c3c0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 -- 172.21.15.107:0/3973856376 shutdown_connections 2023-10-25T00:19:44.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 --2- 172.21.15.107:0/3973856376 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f6c0c18a0 0x7f6f6c0c1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 --2- 172.21.15.107:0/3973856376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f6c0c5c20 0x7f6f6c153190 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 -- 172.21.15.107:0/3973856376 >> 172.21.15.107:0/3973856376 conn(0x7f6f6c0c39a0 msgr2=0x7f6f6c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:44.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 -- 172.21.15.107:0/3973856376 shutdown_connections 2023-10-25T00:19:44.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 -- 172.21.15.107:0/3973856376 wait complete. 2023-10-25T00:19:44.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 Processor -- start 2023-10-25T00:19:44.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 -- start start 2023-10-25T00:19:44.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f6c0c18a0 0x7f6f6c157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f6c0c5c20 0x7f6f6c1f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f6c14fc30 0x7f6f6c1552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1f0900 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1ed1a0 con 0x7f6f6c14fc30 2023-10-25T00:19:44.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f73fc8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6c1ece50 con 0x7f6f6c0c18a0 2023-10-25T00:19:44.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f71563640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f6c0c5c20 0x7f6f6c1f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f71563640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f6c0c5c20 0x7f6f6c1f03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44438/0 (socket says 172.21.15.107:44438) 2023-10-25T00:19:44.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f71563640 1 -- 172.21.15.107:0/594859928 learned_addr learned my addr 172.21.15.107:0/594859928 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:44.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f71d64640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f6c0c18a0 0x7f6f6c157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f72565640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f6c14fc30 0x7f6f6c1552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f71563640 1 -- 172.21.15.107:0/594859928 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f6c0c18a0 msgr2=0x7f6f6c157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:44.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f71563640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f6c0c18a0 0x7f6f6c157000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f71563640 1 -- 172.21.15.107:0/594859928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f6c14fc30 msgr2=0x7f6f6c1552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f71563640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f6c14fc30 0x7f6f6c1552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.423+0000 7f6f71563640 1 -- 172.21.15.107:0/594859928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c009420 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f71563640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f6c0c5c20 0x7f6f6c1f03c0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0af040 tx=0x7f6f5c019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f627fc640 1 -- 172.21.15.107:0/594859928 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c0b5610 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f627fc640 1 -- 172.21.15.107:0/594859928 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5c0b5790 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f627fc640 1 -- 172.21.15.107:0/594859928 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c0b5b40 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f73fc8640 1 -- 172.21.15.107:0/594859928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6c1f0b30 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f73fc8640 1 -- 172.21.15.107:0/594859928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6c1f10f0 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f73fc8640 1 -- 172.21.15.107:0/594859928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6c1ecfc0 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f627fc640 1 -- 172.21.15.107:0/594859928 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6f5c0b3040 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f627fc640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f50065e90 0x7f6f50068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:44.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f627fc640 1 -- 172.21.15.107:0/594859928 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7f6f5c11ebc0 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f71d64640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f50065e90 0x7f6f50068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.427+0000 7f6f71d64640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f50065e90 0x7f6f50068350 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6f68005890 tx=0x7f6f68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:44.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.431+0000 7f6f627fc640 1 -- 172.21.15.107:0/594859928 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6f5c0eb220 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.615+0000 7f6f73fc8640 1 -- 172.21.15.107:0/594859928 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f6c0746e0 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.615+0000 7f6f627fc640 1 -- 172.21.15.107:0/594859928 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+5010 (secure 0 0 0) 0x7f6f5c0b1020 con 0x7f6f6c0c5c20 2023-10-25T00:19:44.618 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:44.619 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:44.137763+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:44.619 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 211 2023-10-25T00:19:44.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 -- 172.21.15.107:0/594859928 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f50065e90 msgr2=0x7f6f50068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:44.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f50065e90 0x7f6f50068350 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6f68005890 tx=0x7f6f68009040 comp rx=0 tx=0).stop 2023-10-25T00:19:44.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 -- 172.21.15.107:0/594859928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f6c0c5c20 msgr2=0x7f6f6c1f03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:44.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f6c0c5c20 0x7f6f6c1f03c0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0af040 tx=0x7f6f5c019160 comp rx=0 tx=0).stop 2023-10-25T00:19:44.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 -- 172.21.15.107:0/594859928 shutdown_connections 2023-10-25T00:19:44.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f6c14fc30 0x7f6f6c1552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f6c0c5c20 0x7f6f6c1f03c0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f50065e90 0x7f6f50068350 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 --2- 172.21.15.107:0/594859928 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f6c0c18a0 0x7f6f6c157000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 -- 172.21.15.107:0/594859928 >> 172.21.15.107:0/594859928 conn(0x7f6f6c0c39a0 msgr2=0x7f6f6c0c5090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 -- 172.21.15.107:0/594859928 shutdown_connections 2023-10-25T00:19:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:44.619+0000 7f6f73fc8640 1 -- 172.21.15.107:0/594859928 wait complete. 2023-10-25T00:19:44.635 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 211, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 204, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7258": { "addr": "172.21.15.110:6834/3771564389", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 3771564389, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 3771564389, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7258, "incarnation": 201, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:18:53.158376+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7258 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 211, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8712": { "addr": "172.21.15.107:6837/657626630", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 657626630, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 657626630, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8712, "incarnation": 211, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:19:44.137763+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8712 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2437148", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2437148, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2437148, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 190, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6443, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:19:46.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:46.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.763+0000 7f63ed447640 1 Processor -- start 2023-10-25T00:19:46.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.763+0000 7f63ed447640 1 -- start start 2023-10-25T00:19:46.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.763+0000 7f63ed447640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f63e814b600 0x7f63e8151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:46.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63ed447640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e80c1860 con 0x7f63e806a230 2023-10-25T00:19:46.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63ed447640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e8154d50 con 0x7f63e806af70 2023-10-25T00:19:46.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63ed447640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e81524f0 con 0x7f63e814b600 2023-10-25T00:19:46.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f63e806a230 0x7f63e81551e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50934/0 (socket says 172.21.15.107:50934) 2023-10-25T00:19:46.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e77fe640 1 -- 172.21.15.107:0/2878357633 learned_addr learned my addr 172.21.15.107:0/2878357633 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:19:46.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e6ffd640 1 --2- 172.21.15.107:0/2878357633 >> v2:172.21.15.110:3300/0 conn(0x7f63e814b600 0x7f63e8151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:46.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313163920 0 0) 0x7f63e80c1860 con 0x7f63e806a230 2023-10-25T00:19:46.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d4003530 con 0x7f63e806a230 2023-10-25T00:19:46.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4201911170 0 0) 0x7f63d4003530 con 0x7f63e806a230 2023-10-25T00:19:46.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 >> v2:172.21.15.110:3300/0 conn(0x7f63e814b600 msgr2=0x7f63e8151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:46.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 --2- 172.21.15.107:0/2878357633 >> v2:172.21.15.110:3300/0 conn(0x7f63e814b600 0x7f63e8151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:46.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 >> v1:172.21.15.110:6789/0 conn(0x7f63e806af70 legacy=0x7f63e806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:46.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e814b9e0 con 0x7f63e806a230 2023-10-25T00:19:46.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f63e0001680 con 0x7f63e806a230 2023-10-25T00:19:46.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 >> v1:172.21.15.107:6789/0 conn(0x7f63e806a230 legacy=0x7f63e81551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:46.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d4003980 0x7f63d4003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:46.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63d40047d0 0x7f63d4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:46.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d4005190 0x7f63d400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:46.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e81524f0 con 0x7f63d4005190 2023-10-25T00:19:46.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d400c0c0 con 0x7f63d4003980 2023-10-25T00:19:46.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d400c280 con 0x7f63d40047d0 2023-10-25T00:19:46.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e77fe640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d4005190 0x7f63d400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e6ffd640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d4003980 0x7f63d4003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e67fc640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63d40047d0 0x7f63d4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e77fe640 1 -- 172.21.15.107:0/2878357633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63d40047d0 msgr2=0x7f63d4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e77fe640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63d40047d0 0x7f63d4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e77fe640 1 -- 172.21.15.107:0/2878357633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d4003980 msgr2=0x7f63d4003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:46.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e77fe640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63d4003980 0x7f63d4003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:46.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e77fe640 1 -- 172.21.15.107:0/2878357633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d4003530 con 0x7f63d4005190 2023-10-25T00:19:46.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.767+0000 7f63e67fc640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63d40047d0 0x7f63d4004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:19:46.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e77fe640 1 -- 172.21.15.107:0/2878357633 reap_dead start 2023-10-25T00:19:46.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e77fe640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d4005190 0x7f63d400b990 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f63e8066c20 tx=0x7f63e0021580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee964167daeae6ee server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:46.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0005ae0 con 0x7f63d4005190 2023-10-25T00:19:46.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e00ad070 con 0x7f63d4005190 2023-10-25T00:19:46.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e5ffb640 1 -- 172.21.15.107:0/2878357633 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0005ea0 con 0x7f63d4005190 2023-10-25T00:19:46.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 -- 172.21.15.107:0/2878357633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d4005190 msgr2=0x7f63d400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:46.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d4005190 0x7f63d400b990 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f63e8066c20 tx=0x7f63e0021580 comp rx=0 tx=0).stop 2023-10-25T00:19:46.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 -- 172.21.15.107:0/2878357633 shutdown_connections 2023-10-25T00:19:46.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 --2- 172.21.15.107:0/2878357633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63d4005190 0x7f63d400b990 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:46.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 -- 172.21.15.107:0/2878357633 >> 172.21.15.107:0/2878357633 conn(0x7f63e80c2950 msgr2=0x7f63e80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:46.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 -- 172.21.15.107:0/2878357633 shutdown_connections 2023-10-25T00:19:46.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 -- 172.21.15.107:0/2878357633 wait complete. 2023-10-25T00:19:46.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 Processor -- start 2023-10-25T00:19:46.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 -- start start 2023-10-25T00:19:46.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63d4005190 0x7f63e8071460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:46.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63e80c56e0 0x7f63e8143760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:46.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e8143ca0 0x7f63e81e9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:46.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e8144120 con 0x7f63e8143ca0 2023-10-25T00:19:46.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e8154ad0 con 0x7f63e80c56e0 2023-10-25T00:19:46.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e81544c0 con 0x7f63d4005190 2023-10-25T00:19:46.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e8143ca0 0x7f63e81e9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:46.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e8143ca0 0x7f63e81e9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44450/0 (socket says 172.21.15.107:44450) 2023-10-25T00:19:46.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e77fe640 1 -- 172.21.15.107:0/256506155 learned_addr learned my addr 172.21.15.107:0/256506155 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:46.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e6ffd640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63d4005190 0x7f63e8071460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:46.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e67fc640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63e80c56e0 0x7f63e8143760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:46.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e77fe640 1 -- 172.21.15.107:0/256506155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63d4005190 msgr2=0x7f63e8071460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:46.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e77fe640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63d4005190 0x7f63e8071460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:46.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e77fe640 1 -- 172.21.15.107:0/256506155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63e80c56e0 msgr2=0x7f63e8143760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:46.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e77fe640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63e80c56e0 0x7f63e8143760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:46.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e77fe640 1 -- 172.21.15.107:0/256506155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e81e6710 con 0x7f63e8143ca0 2023-10-25T00:19:46.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63e77fe640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e8143ca0 0x7f63e81e9320 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f63e0021e20 tx=0x7f63d4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:46.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63df7fe640 1 -- 172.21.15.107:0/256506155 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0001640 con 0x7f63e8143ca0 2023-10-25T00:19:46.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63df7fe640 1 -- 172.21.15.107:0/256506155 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e00b8040 con 0x7f63e8143ca0 2023-10-25T00:19:46.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63ed447640 1 -- 172.21.15.107:0/256506155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e8154760 con 0x7f63e8143ca0 2023-10-25T00:19:46.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.771+0000 7f63df7fe640 1 -- 172.21.15.107:0/256506155 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0004a10 con 0x7f63e8143ca0 2023-10-25T00:19:46.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.775+0000 7f63ed447640 1 -- 172.21.15.107:0/256506155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e81e69f0 con 0x7f63e8143ca0 2023-10-25T00:19:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.775+0000 7f63ed447640 1 -- 172.21.15.107:0/256506155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e80746e0 con 0x7f63e8143ca0 2023-10-25T00:19:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.775+0000 7f63df7fe640 1 -- 172.21.15.107:0/256506155 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f63e00af030 con 0x7f63e8143ca0 2023-10-25T00:19:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.775+0000 7f63df7fe640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c8065e60 0x7f63c8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.775+0000 7f63e6ffd640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c8065e60 0x7f63c8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:46.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.775+0000 7f63df7fe640 1 -- 172.21.15.107:0/256506155 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7f63e0123740 con 0x7f63e8143ca0 2023-10-25T00:19:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.775+0000 7f63e6ffd640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c8065e60 0x7f63c8068320 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f63d80099b0 tx=0x7f63d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:46.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.779+0000 7f63df7fe640 1 -- 172.21.15.107:0/256506155 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f63e00f3710 con 0x7f63e8143ca0 2023-10-25T00:19:46.970 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.967+0000 7f63ed447640 1 -- 172.21.15.107:0/256506155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63e81e9080 con 0x7f63e8143ca0 2023-10-25T00:19:46.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63df7fe640 1 -- 172.21.15.107:0/256506155 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+5010 (secure 0 0 0) 0x7f63e00bd3b0 con 0x7f63e8143ca0 2023-10-25T00:19:46.971 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:46.972 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:46.157036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:46.972 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 213 2023-10-25T00:19:46.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 -- 172.21.15.107:0/256506155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c8065e60 msgr2=0x7f63c8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:46.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c8065e60 0x7f63c8068320 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f63d80099b0 tx=0x7f63d8008040 comp rx=0 tx=0).stop 2023-10-25T00:19:46.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 -- 172.21.15.107:0/256506155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e8143ca0 msgr2=0x7f63e81e9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:46.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e8143ca0 0x7f63e81e9320 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f63e0021e20 tx=0x7f63d4005800 comp rx=0 tx=0).stop 2023-10-25T00:19:46.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 -- 172.21.15.107:0/256506155 shutdown_connections 2023-10-25T00:19:46.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63e8143ca0 0x7f63e81e9320 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:46.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f63e80c56e0 0x7f63e8143760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:46.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f63c8065e60 0x7f63c8068320 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:46.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 --2- 172.21.15.107:0/256506155 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f63d4005190 0x7f63e8071460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:46.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 -- 172.21.15.107:0/256506155 >> 172.21.15.107:0/256506155 conn(0x7f63e80c2950 msgr2=0x7f63e8144a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:46.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 -- 172.21.15.107:0/256506155 shutdown_connections 2023-10-25T00:19:46.975 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:46.971+0000 7f63ed447640 1 -- 172.21.15.107:0/256506155 wait complete. 2023-10-25T00:19:46.989 INFO:tasks.mds_thrash.fs.[b]:no change 2023-10-25T00:19:47.628 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:19:47.627+0000 7f7c21cf1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:19:48.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4e60a9640 1 Processor -- start 2023-10-25T00:19:49.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4e60a9640 1 -- start start 2023-10-25T00:19:49.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4e60a9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe4e0152040 0x7fe4e00c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:49.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4e60a9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e0155680 con 0x7fe4e00c8100 2023-10-25T00:19:49.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4e60a9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4e0154a70 con 0x7fe4e006a4b0 2023-10-25T00:19:49.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4e60a9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4e0154d50 con 0x7fe4e0152040 2023-10-25T00:19:49.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4deffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe4e00c8100 0x7fe4e00abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50940/0 (socket says 172.21.15.107:50940) 2023-10-25T00:19:49.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4deffd640 1 -- 172.21.15.107:0/3143921923 learned_addr learned my addr 172.21.15.107:0/3143921923 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:19:49.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4df7fe640 1 --2- 172.21.15.107:0/3143921923 >> v2:172.21.15.107:3300/0 conn(0x7fe4e0152040 0x7fe4e00c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:49.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4df7fe640 1 -- 172.21.15.107:0/3143921923 >> v1:172.21.15.110:6789/0 conn(0x7fe4e006a4b0 legacy=0x7fe4e00ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:49.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4df7fe640 1 -- 172.21.15.107:0/3143921923 >> v1:172.21.15.107:6789/0 conn(0x7fe4e00c8100 legacy=0x7fe4e00abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:49.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.127+0000 7fe4df7fe640 1 -- 172.21.15.107:0/3143921923 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0155030 con 0x7fe4e0152040 2023-10-25T00:19:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.131+0000 7fe4df7fe640 1 --2- 172.21.15.107:0/3143921923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e0152040 0x7fe4e00c4b80 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40097c0 tx=0x7fe4d401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d10598b24bfdda server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.131+0000 7fe4de7fc640 1 -- 172.21.15.107:0/3143921923 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d40ab600 con 0x7fe4e0152040 2023-10-25T00:19:49.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.131+0000 7fe4de7fc640 1 -- 172.21.15.107:0/3143921923 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d40a7050 con 0x7fe4e0152040 2023-10-25T00:19:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.131+0000 7fe4de7fc640 1 -- 172.21.15.107:0/3143921923 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d40ab980 con 0x7fe4e0152040 2023-10-25T00:19:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.131+0000 7fe4e60a9640 1 -- 172.21.15.107:0/3143921923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e0152040 msgr2=0x7fe4e00c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.131+0000 7fe4e60a9640 1 --2- 172.21.15.107:0/3143921923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e0152040 0x7fe4e00c4b80 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40097c0 tx=0x7fe4d401c320 comp rx=0 tx=0).stop 2023-10-25T00:19:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.131+0000 7fe4e60a9640 1 -- 172.21.15.107:0/3143921923 shutdown_connections 2023-10-25T00:19:49.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.131+0000 7fe4e60a9640 1 --2- 172.21.15.107:0/3143921923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e0152040 0x7fe4e00c4b80 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.131+0000 7fe4e60a9640 1 -- 172.21.15.107:0/3143921923 >> 172.21.15.107:0/3143921923 conn(0x7fe4e00ca200 msgr2=0x7fe4e00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.131+0000 7fe4e60a9640 1 -- 172.21.15.107:0/3143921923 shutdown_connections 2023-10-25T00:19:49.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.131+0000 7fe4e60a9640 1 -- 172.21.15.107:0/3143921923 wait complete. 2023-10-25T00:19:49.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4e60a9640 1 Processor -- start 2023-10-25T00:19:49.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4e60a9640 1 -- start start 2023-10-25T00:19:49.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4e60a9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4e006a4b0 0x7fe4e01488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:49.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4e60a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00c8100 0x7fe4e015e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:49.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4e60a9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4e0152040 0x7fe4e015e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4e60a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e01f51d0 con 0x7fe4e00c8100 2023-10-25T00:19:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4e60a9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e015edc0 con 0x7fe4e006a4b0 2023-10-25T00:19:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4e60a9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e0148490 con 0x7fe4e0152040 2023-10-25T00:19:49.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4deffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00c8100 0x7fe4e015e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4deffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00c8100 0x7fe4e015e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44474/0 (socket says 172.21.15.107:44474) 2023-10-25T00:19:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4deffd640 1 -- 172.21.15.107:0/537651768 learned_addr learned my addr 172.21.15.107:0/537651768 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4df7fe640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4e006a4b0 0x7fe4e01488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4deffd640 1 -- 172.21.15.107:0/537651768 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4e0152040 msgr2=0x7fe4e015e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:49.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4deffd640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4e0152040 0x7fe4e015e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4deffd640 1 -- 172.21.15.107:0/537651768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4e006a4b0 msgr2=0x7fe4e01488d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4deffd640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4e006a4b0 0x7fe4e01488d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4deffd640 1 -- 172.21.15.107:0/537651768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d4009420 con 0x7fe4e00c8100 2023-10-25T00:19:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4deffd640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00c8100 0x7fe4e015e340 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000c4d0 tx=0x7fe4d000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4c7fff640 1 -- 172.21.15.107:0/537651768 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0005cf0 con 0x7fe4e00c8100 2023-10-25T00:19:49.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4c7fff640 1 -- 172.21.15.107:0/537651768 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d0005e50 con 0x7fe4e00c8100 2023-10-25T00:19:49.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.135+0000 7fe4c7fff640 1 -- 172.21.15.107:0/537651768 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0007270 con 0x7fe4e00c8100 2023-10-25T00:19:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.139+0000 7fe4e60a9640 1 -- 172.21.15.107:0/537651768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e01476d0 con 0x7fe4e00c8100 2023-10-25T00:19:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.139+0000 7fe4e60a9640 1 -- 172.21.15.107:0/537651768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e0147c90 con 0x7fe4e00c8100 2023-10-25T00:19:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.139+0000 7fe4e60a9640 1 -- 172.21.15.107:0/537651768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e00746e0 con 0x7fe4e00c8100 2023-10-25T00:19:49.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.139+0000 7fe4c7fff640 1 -- 172.21.15.107:0/537651768 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe4d00073d0 con 0x7fe4e00c8100 2023-10-25T00:19:49.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.139+0000 7fe4c7fff640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4cc065e60 0x7fe4cc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:49.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.139+0000 7fe4c7fff640 1 -- 172.21.15.107:0/537651768 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7fe4d0083ea0 con 0x7fe4e00c8100 2023-10-25T00:19:49.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.139+0000 7fe4df7fe640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4cc065e60 0x7fe4cc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:49.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.143+0000 7fe4df7fe640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4cc065e60 0x7fe4cc068320 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40ad000 tx=0x7fe4d40a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:49.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.147+0000 7fe4c7fff640 1 -- 172.21.15.107:0/537651768 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe4d0050370 con 0x7fe4e00c8100 2023-10-25T00:19:49.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.423+0000 7fe4e60a9640 1 -- 172.21.15.107:0/537651768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4e014d5a0 con 0x7fe4e00c8100 2023-10-25T00:19:49.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.423+0000 7fe4c7fff640 1 -- 172.21.15.107:0/537651768 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 214 v214) v1 ==== 95+0+5011 (secure 0 0 0) 0x7fe4d0017070 con 0x7fe4e00c8100 2023-10-25T00:19:49.426 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:49.426 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":214,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:49.427 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 214 2023-10-25T00:19:49.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.427+0000 7fe4e60a9640 1 -- 172.21.15.107:0/537651768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4cc065e60 msgr2=0x7fe4cc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:49.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.427+0000 7fe4e60a9640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4cc065e60 0x7fe4cc068320 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fe4d40ad000 tx=0x7fe4d40a6000 comp rx=0 tx=0).stop 2023-10-25T00:19:49.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.427+0000 7fe4e60a9640 1 -- 172.21.15.107:0/537651768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00c8100 msgr2=0x7fe4e015e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.427+0000 7fe4e60a9640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00c8100 0x7fe4e015e340 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000c4d0 tx=0x7fe4d000c990 comp rx=0 tx=0).stop 2023-10-25T00:19:49.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.431+0000 7fe4e60a9640 1 -- 172.21.15.107:0/537651768 shutdown_connections 2023-10-25T00:19:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.431+0000 7fe4e60a9640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe4cc065e60 0x7fe4cc068320 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.431+0000 7fe4e60a9640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe4e0152040 0x7fe4e015e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.431+0000 7fe4e60a9640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4e00c8100 0x7fe4e015e340 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.431+0000 7fe4e60a9640 1 --2- 172.21.15.107:0/537651768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe4e006a4b0 0x7fe4e01488d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:49.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.431+0000 7fe4e60a9640 1 -- 172.21.15.107:0/537651768 >> 172.21.15.107:0/537651768 conn(0x7fe4e00ca200 msgr2=0x7fe4e006a920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:49.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.431+0000 7fe4e60a9640 1 -- 172.21.15.107:0/537651768 shutdown_connections 2023-10-25T00:19:49.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:49.431+0000 7fe4e60a9640 1 -- 172.21.15.107:0/537651768 wait complete. 2023-10-25T00:19:49.452 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=8649 2023-10-25T00:19:49.453 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.a 2023-10-25T00:19:56.454 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2023-10-25T00:19:56.454 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:19:56.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:19:56.457 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:19:56.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:56.547 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:19:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.614+0000 7fc393244640 1 Processor -- start 2023-10-25T00:19:56.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.614+0000 7fc393244640 1 -- start start 2023-10-25T00:19:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc393244640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc38c148110 0x7fc38c151f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc393244640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc38c06c1a0 0x7fc38c070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc393244640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c06ae50 con 0x7fc38c071a40 2023-10-25T00:19:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc390fe0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc38c148110 0x7fc38c151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:56.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc393244640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc38c0a9800 con 0x7fc38c06c1a0 2023-10-25T00:19:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc393244640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c06bfb0 con 0x7fc38c148110 2023-10-25T00:19:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc390fe0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc38c148110 0x7fc38c151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:45968/0 (socket says 172.21.15.107:45968) 2023-10-25T00:19:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc390fe0640 1 -- 172.21.15.107:0/193594981 learned_addr learned my addr 172.21.15.107:0/193594981 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:56.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc3917e1640 1 --2- 172.21.15.107:0/193594981 >> v2:172.21.15.110:3300/0 conn(0x7fc38c06c1a0 0x7fc38c070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:56.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc3917e1640 1 -- 172.21.15.107:0/193594981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc38c148110 msgr2=0x7fc38c151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:56.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc3917e1640 1 --2- 172.21.15.107:0/193594981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc38c148110 0x7fc38c151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:56.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc3917e1640 1 -- 172.21.15.107:0/193594981 >> v1:172.21.15.110:6789/0 conn(0x7fc38c071a40 legacy=0x7fc38c06b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:56.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc3917e1640 1 -- 172.21.15.107:0/193594981 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c0c1550 con 0x7fc38c06c1a0 2023-10-25T00:19:56.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.618+0000 7fc3917e1640 1 --2- 172.21.15.107:0/193594981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc38c06c1a0 0x7fc38c070e10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc384009dc0 tx=0x7fc38401c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=257ef2764cc521de server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:56.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc38b7fe640 1 -- 172.21.15.107:0/193594981 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3840a5800 con 0x7fc38c06c1a0 2023-10-25T00:19:56.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc38b7fe640 1 -- 172.21.15.107:0/193594981 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3840a5960 con 0x7fc38c06c1a0 2023-10-25T00:19:56.624 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc38b7fe640 1 -- 172.21.15.107:0/193594981 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3840a5cf0 con 0x7fc38c06c1a0 2023-10-25T00:19:56.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc393244640 1 -- 172.21.15.107:0/193594981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc38c06c1a0 msgr2=0x7fc38c070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:56.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc393244640 1 --2- 172.21.15.107:0/193594981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc38c06c1a0 0x7fc38c070e10 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc384009dc0 tx=0x7fc38401c930 comp rx=0 tx=0).stop 2023-10-25T00:19:56.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc393244640 1 -- 172.21.15.107:0/193594981 shutdown_connections 2023-10-25T00:19:56.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc393244640 1 --2- 172.21.15.107:0/193594981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc38c06c1a0 0x7fc38c070e10 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:56.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc393244640 1 --2- 172.21.15.107:0/193594981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc38c148110 0x7fc38c151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:56.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc393244640 1 -- 172.21.15.107:0/193594981 >> 172.21.15.107:0/193594981 conn(0x7fc38c0c5cd0 msgr2=0x7fc38c0c60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:56.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc393244640 1 -- 172.21.15.107:0/193594981 shutdown_connections 2023-10-25T00:19:56.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc393244640 1 -- 172.21.15.107:0/193594981 wait complete. 2023-10-25T00:19:56.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.622+0000 7fc393244640 1 Processor -- start 2023-10-25T00:19:56.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc393244640 1 -- start start 2023-10-25T00:19:56.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc393244640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc38c06c1a0 0x7fc38c144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:56.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc393244640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc38c071a40 0x7fc38c0aa080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:56.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc393244640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc38c148110 0x7fc38c145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:56.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc393244640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c06b480 con 0x7fc38c06c1a0 2023-10-25T00:19:56.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc393244640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c13fae0 con 0x7fc38c071a40 2023-10-25T00:19:56.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc393244640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38c144380 con 0x7fc38c148110 2023-10-25T00:19:56.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc390fe0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc38c06c1a0 0x7fc38c144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:56.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc390fe0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc38c06c1a0 0x7fc38c144dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38210/0 (socket says 172.21.15.107:38210) 2023-10-25T00:19:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc390fe0640 1 -- 172.21.15.107:0/3039487863 learned_addr learned my addr 172.21.15.107:0/3039487863 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc38bfff640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc38c071a40 0x7fc38c0aa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc390fe0640 1 -- 172.21.15.107:0/3039487863 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc38c148110 msgr2=0x7fc38c145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc390fe0640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc38c148110 0x7fc38c145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:56.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc390fe0640 1 -- 172.21.15.107:0/3039487863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc38c071a40 msgr2=0x7fc38c0aa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:56.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc390fe0640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc38c071a40 0x7fc38c0aa080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:56.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc390fe0640 1 -- 172.21.15.107:0/3039487863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc384009a70 con 0x7fc38c06c1a0 2023-10-25T00:19:56.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc390fe0640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc38c06c1a0 0x7fc38c144dd0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fc380012ad0 tx=0x7fc380012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc3897fa640 1 -- 172.21.15.107:0/3039487863 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380010a80 con 0x7fc38c06c1a0 2023-10-25T00:19:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc3897fa640 1 -- 172.21.15.107:0/3039487863 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc380013040 con 0x7fc38c06c1a0 2023-10-25T00:19:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc393244640 1 -- 172.21.15.107:0/3039487863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c144940 con 0x7fc38c06c1a0 2023-10-25T00:19:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc3897fa640 1 -- 172.21.15.107:0/3039487863 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc380010e40 con 0x7fc38c06c1a0 2023-10-25T00:19:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc393244640 1 -- 172.21.15.107:0/3039487863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c145360 con 0x7fc38c06c1a0 2023-10-25T00:19:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc393244640 1 -- 172.21.15.107:0/3039487863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c0746e0 con 0x7fc38c06c1a0 2023-10-25T00:19:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc3897fa640 1 -- 172.21.15.107:0/3039487863 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc380018480 con 0x7fc38c06c1a0 2023-10-25T00:19:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.626+0000 7fc3897fa640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc370065e90 0x7fc370068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.630+0000 7fc3897fa640 1 -- 172.21.15.107:0/3039487863 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7fc380088740 con 0x7fc38c06c1a0 2023-10-25T00:19:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.630+0000 7fc38bfff640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc370065e90 0x7fc370068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:56.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.630+0000 7fc38bfff640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc370065e90 0x7fc370068350 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc38c152840 tx=0x7fc378009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:56.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.634+0000 7fc3897fa640 1 -- 172.21.15.107:0/3039487863 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc380054c80 con 0x7fc38c06c1a0 2023-10-25T00:19:56.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.906+0000 7fc393244640 1 -- 172.21.15.107:0/3039487863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc38c145360 con 0x7fc38c06c1a0 2023-10-25T00:19:56.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.906+0000 7fc3897fa640 1 -- 172.21.15.107:0/3039487863 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 214 v214) v1 ==== 95+0+5011 (secure 0 0 0) 0x7fc38c145360 con 0x7fc38c06c1a0 2023-10-25T00:19:56.910 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:56.910 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":214,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:56.912 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 214 2023-10-25T00:19:56.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 -- 172.21.15.107:0/3039487863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc370065e90 msgr2=0x7fc370068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:56.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc370065e90 0x7fc370068350 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc38c152840 tx=0x7fc378009040 comp rx=0 tx=0).stop 2023-10-25T00:19:56.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 -- 172.21.15.107:0/3039487863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc38c06c1a0 msgr2=0x7fc38c144dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:56.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc38c06c1a0 0x7fc38c144dd0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fc380012ad0 tx=0x7fc380012f90 comp rx=0 tx=0).stop 2023-10-25T00:19:56.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 -- 172.21.15.107:0/3039487863 shutdown_connections 2023-10-25T00:19:56.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc370065e90 0x7fc370068350 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:56.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc38c148110 0x7fc38c145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:56.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc38c071a40 0x7fc38c0aa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 --2- 172.21.15.107:0/3039487863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc38c06c1a0 0x7fc38c144dd0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 -- 172.21.15.107:0/3039487863 >> 172.21.15.107:0/3039487863 conn(0x7fc38c0c5cd0 msgr2=0x7fc38c0c1710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 -- 172.21.15.107:0/3039487863 shutdown_connections 2023-10-25T00:19:56.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:56.910+0000 7fc393244640 1 -- 172.21.15.107:0/3039487863 wait complete. 2023-10-25T00:19:56.929 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2023-10-25T00:19:58.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:19:59.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.074+0000 7f8a92d01640 1 Processor -- start 2023-10-25T00:19:59.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.074+0000 7f8a92d01640 1 -- start start 2023-10-25T00:19:59.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.074+0000 7f8a92d01640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8a8c14f810 0x7f8a8c1595d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:59.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.074+0000 7f8a92d01640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8a8c0aa990 0x7f8a8c06b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:59.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.074+0000 7f8a92d01640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a8c069d40 con 0x7f8a8c06b9e0 2023-10-25T00:19:59.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.074+0000 7f8a92d01640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8c06c580 con 0x7f8a8c14f810 2023-10-25T00:19:59.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.074+0000 7f8a92d01640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c0c2310 con 0x7f8a8c0aa990 2023-10-25T00:19:59.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.074+0000 7f8a90a9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8a8c14f810 0x7f8a8c1595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:59.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a90a9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8a8c14f810 0x7f8a8c1595d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:43058/0 (socket says 172.21.15.107:43058) 2023-10-25T00:19:59.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a90a9d640 1 -- 172.21.15.107:0/1646244706 learned_addr learned my addr 172.21.15.107:0/1646244706 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:59.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a9129e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8a8c06b9e0 0x7f8a8c15c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:38904/0 (socket says 172.21.15.107:38904) 2023-10-25T00:19:59.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a8bfff640 1 --2- 172.21.15.107:0/1646244706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8a8c0aa990 0x7f8a8c06b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:59.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a90a9d640 1 -- 172.21.15.107:0/1646244706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8a8c0aa990 msgr2=0x7f8a8c06b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:19:59.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a90a9d640 1 --2- 172.21.15.107:0/1646244706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8a8c0aa990 0x7f8a8c06b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:59.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a90a9d640 1 -- 172.21.15.107:0/1646244706 >> v1:172.21.15.110:6789/0 conn(0x7f8a8c06b9e0 legacy=0x7f8a8c15c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:59.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a90a9d640 1 -- 172.21.15.107:0/1646244706 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c0c1a70 con 0x7f8a8c14f810 2023-10-25T00:19:59.082 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a8bfff640 1 --2- 172.21.15.107:0/1646244706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8a8c0aa990 0x7f8a8c06b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:19:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a90a9d640 1 --2- 172.21.15.107:0/1646244706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a8c14f810 0x7f8a8c1595d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8a80009870 tx=0x7f8a8001c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd8b3e75ab3991ca server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a8b7fe640 1 -- 172.21.15.107:0/1646244706 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a800ab600 con 0x7f8a8c14f810 2023-10-25T00:19:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a8b7fe640 1 -- 172.21.15.107:0/1646244706 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a800a7050 con 0x7f8a8c14f810 2023-10-25T00:19:59.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.078+0000 7f8a8b7fe640 1 -- 172.21.15.107:0/1646244706 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a800ab980 con 0x7f8a8c14f810 2023-10-25T00:19:59.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.082+0000 7f8a92d01640 1 -- 172.21.15.107:0/1646244706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a8c14f810 msgr2=0x7f8a8c1595d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:59.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.082+0000 7f8a92d01640 1 --2- 172.21.15.107:0/1646244706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a8c14f810 0x7f8a8c1595d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8a80009870 tx=0x7f8a8001c540 comp rx=0 tx=0).stop 2023-10-25T00:19:59.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.082+0000 7f8a92d01640 1 -- 172.21.15.107:0/1646244706 shutdown_connections 2023-10-25T00:19:59.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.082+0000 7f8a92d01640 1 --2- 172.21.15.107:0/1646244706 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8a8c0aa990 0x7f8a8c06b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:59.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.082+0000 7f8a92d01640 1 --2- 172.21.15.107:0/1646244706 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a8c14f810 0x7f8a8c1595d0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:59.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.082+0000 7f8a92d01640 1 -- 172.21.15.107:0/1646244706 >> 172.21.15.107:0/1646244706 conn(0x7f8a8c0c2950 msgr2=0x7f8a8c0c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:59.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.082+0000 7f8a92d01640 1 -- 172.21.15.107:0/1646244706 shutdown_connections 2023-10-25T00:19:59.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.082+0000 7f8a92d01640 1 -- 172.21.15.107:0/1646244706 wait complete. 2023-10-25T00:19:59.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.082+0000 7f8a92d01640 1 Processor -- start 2023-10-25T00:19:59.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.082+0000 7f8a92d01640 1 -- start start 2023-10-25T00:19:59.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a92d01640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8a8c06b9e0 0x7f8a8c1539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:59.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a92d01640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a8c0aa990 0x7f8a8c14e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:59.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a92d01640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a8c14f810 0x7f8a8c1f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:59.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a92d01640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c06fa10 con 0x7f8a8c0aa990 2023-10-25T00:19:59.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a92d01640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c1f12d0 con 0x7f8a8c14f810 2023-10-25T00:19:59.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a92d01640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c0701e0 con 0x7f8a8c06b9e0 2023-10-25T00:19:59.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a8bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a8c0aa990 0x7f8a8c14e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:59.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a8bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a8c0aa990 0x7f8a8c14e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:38216/0 (socket says 172.21.15.107:38216) 2023-10-25T00:19:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a8bfff640 1 -- 172.21.15.107:0/2099781826 learned_addr learned my addr 172.21.15.107:0/2099781826 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:19:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a9129e640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a8c14f810 0x7f8a8c1f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a90a9d640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8a8c06b9e0 0x7f8a8c1539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a8bfff640 1 -- 172.21.15.107:0/2099781826 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8a8c06b9e0 msgr2=0x7f8a8c1539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:59.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a8bfff640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8a8c06b9e0 0x7f8a8c1539f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:59.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a8bfff640 1 -- 172.21.15.107:0/2099781826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a8c14f810 msgr2=0x7f8a8c1f19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:59.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a8bfff640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a8c14f810 0x7f8a8c1f19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:59.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a8bfff640 1 -- 172.21.15.107:0/2099781826 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a800094d0 con 0x7f8a8c0aa990 2023-10-25T00:19:59.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a9129e640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a8c14f810 0x7f8a8c1f19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:19:59.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a8bfff640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a8c0aa990 0x7f8a8c14e4d0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800c7f0 tx=0x7f8a7800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:59.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a897fa640 1 -- 172.21.15.107:0/2099781826 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7800cea0 con 0x7f8a8c0aa990 2023-10-25T00:19:59.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a92d01640 1 -- 172.21.15.107:0/2099781826 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8c06fc20 con 0x7f8a8c0aa990 2023-10-25T00:19:59.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a92d01640 1 -- 172.21.15.107:0/2099781826 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8c1f14e0 con 0x7f8a8c0aa990 2023-10-25T00:19:59.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a897fa640 1 -- 172.21.15.107:0/2099781826 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a780079d0 con 0x7f8a8c0aa990 2023-10-25T00:19:59.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a897fa640 1 -- 172.21.15.107:0/2099781826 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a78007d80 con 0x7f8a8c0aa990 2023-10-25T00:19:59.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a897fa640 1 -- 172.21.15.107:0/2099781826 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8a780113f0 con 0x7f8a8c0aa990 2023-10-25T00:19:59.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a92d01640 1 -- 172.21.15.107:0/2099781826 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6c005260 con 0x7f8a8c0aa990 2023-10-25T00:19:59.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.086+0000 7f8a897fa640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8a70065e60 0x7f8a70068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:19:59.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.090+0000 7f8a897fa640 1 -- 172.21.15.107:0/2099781826 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7f8a780849c0 con 0x7f8a8c0aa990 2023-10-25T00:19:59.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.090+0000 7f8a90a9d640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8a70065e60 0x7f8a70068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:19:59.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.090+0000 7f8a90a9d640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8a70065e60 0x7f8a70068320 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0c6d00 tx=0x7f8a800a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:19:59.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.094+0000 7f8a897fa640 1 -- 172.21.15.107:0/2099781826 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8a78050f50 con 0x7f8a8c0aa990 2023-10-25T00:19:59.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.358+0000 7f8a92d01640 1 -- 172.21.15.107:0/2099781826 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a6c005a90 con 0x7f8a8c0aa990 2023-10-25T00:19:59.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.358+0000 7f8a897fa640 1 -- 172.21.15.107:0/2099781826 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5772 (secure 0 0 0) 0x7f8a7801f090 con 0x7f8a8c0aa990 2023-10-25T00:19:59.363 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:19:59.363 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":215}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:19:59.364 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 215 2023-10-25T00:19:59.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 -- 172.21.15.107:0/2099781826 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8a70065e60 msgr2=0x7f8a70068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:59.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8a70065e60 0x7f8a70068320 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0c6d00 tx=0x7f8a800a6000 comp rx=0 tx=0).stop 2023-10-25T00:19:59.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 -- 172.21.15.107:0/2099781826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a8c0aa990 msgr2=0x7f8a8c14e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:19:59.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a8c0aa990 0x7f8a8c14e4d0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800c7f0 tx=0x7f8a7800ccb0 comp rx=0 tx=0).stop 2023-10-25T00:19:59.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 -- 172.21.15.107:0/2099781826 shutdown_connections 2023-10-25T00:19:59.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a8c14f810 0x7f8a8c1f19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:59.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a8c0aa990 0x7f8a8c14e4d0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:59.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8a70065e60 0x7f8a70068320 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:59.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 --2- 172.21.15.107:0/2099781826 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8a8c06b9e0 0x7f8a8c1539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:19:59.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 -- 172.21.15.107:0/2099781826 >> 172.21.15.107:0/2099781826 conn(0x7f8a8c0c2950 msgr2=0x7f8a8c157500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:19:59.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 -- 172.21.15.107:0/2099781826 shutdown_connections 2023-10-25T00:19:59.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:19:59.362+0000 7f8a92d01640 1 -- 172.21.15.107:0/2099781826 wait complete. 2023-10-25T00:19:59.384 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2023-10-25T00:19:59.385 INFO:tasks.mds_thrash.fs.[b]:waiting for 65 secs before thrashing 2023-10-25T00:20:02.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:02.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f7309fde640 1 Processor -- start 2023-10-25T00:20:02.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f7309fde640 1 -- start start 2023-10-25T00:20:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f7309fde640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f730414eb70 0x7f73040c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f7309fde640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f730406df30 0x7f730406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:02.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f7309fde640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73040c22f0 con 0x7f7304070b00 2023-10-25T00:20:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f7309fde640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7304071700 con 0x7f730414eb70 2023-10-25T00:20:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f7309fde640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73040c25d0 con 0x7f730406df30 2023-10-25T00:20:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f73037fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f730414eb70 0x7f73040c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:02.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f73037fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f730414eb70 0x7f73040c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33868/0 (socket says 172.21.15.107:33868) 2023-10-25T00:20:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f7302ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7304070b00 0x7f73041514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58760/0 (socket says 172.21.15.107:58760) 2023-10-25T00:20:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f73037fe640 1 -- 172.21.15.107:0/2640712704 learned_addr learned my addr 172.21.15.107:0/2640712704 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f7303fff640 1 --2- 172.21.15.107:0/2640712704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f730406df30 0x7f730406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f73037fe640 1 -- 172.21.15.107:0/2640712704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f730406df30 msgr2=0x7f730406e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f73037fe640 1 --2- 172.21.15.107:0/2640712704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f730406df30 0x7f730406e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f73037fe640 1 -- 172.21.15.107:0/2640712704 >> v1:172.21.15.107:6789/0 conn(0x7f7304070b00 legacy=0x7f73041514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f73037fe640 1 -- 172.21.15.107:0/2640712704 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73040c2be0 con 0x7f730414eb70 2023-10-25T00:20:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f73027fc640 1 -- 172.21.15.107:0/2640712704 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082519999 0 0) 0x7f73040c22f0 con 0x7f7304070b00 2023-10-25T00:20:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.658+0000 7f7303fff640 1 --2- 172.21.15.107:0/2640712704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f730406df30 0x7f730406e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:20:02.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f73037fe640 1 --2- 172.21.15.107:0/2640712704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730414eb70 0x7f73040c33e0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009870 tx=0x7f72f401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85dac77ba5a5be server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f73027fc640 1 -- 172.21.15.107:0/2640712704 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f40a9620 con 0x7f730414eb70 2023-10-25T00:20:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f73027fc640 1 -- 172.21.15.107:0/2640712704 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f40a9780 con 0x7f730414eb70 2023-10-25T00:20:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f73027fc640 1 -- 172.21.15.107:0/2640712704 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f40a9b10 con 0x7f730414eb70 2023-10-25T00:20:02.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f7309fde640 1 -- 172.21.15.107:0/2640712704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730414eb70 msgr2=0x7f73040c33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:02.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f7309fde640 1 --2- 172.21.15.107:0/2640712704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730414eb70 0x7f73040c33e0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f72f4009870 tx=0x7f72f401c540 comp rx=0 tx=0).stop 2023-10-25T00:20:02.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f7309fde640 1 -- 172.21.15.107:0/2640712704 shutdown_connections 2023-10-25T00:20:02.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f7309fde640 1 --2- 172.21.15.107:0/2640712704 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f730406df30 0x7f730406e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:02.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f7309fde640 1 --2- 172.21.15.107:0/2640712704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730414eb70 0x7f73040c33e0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:02.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f7309fde640 1 -- 172.21.15.107:0/2640712704 >> 172.21.15.107:0/2640712704 conn(0x7f73040c39a0 msgr2=0x7f73040c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:02.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f7309fde640 1 -- 172.21.15.107:0/2640712704 shutdown_connections 2023-10-25T00:20:02.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f7309fde640 1 -- 172.21.15.107:0/2640712704 wait complete. 2023-10-25T00:20:02.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f7309fde640 1 Processor -- start 2023-10-25T00:20:02.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.662+0000 7f7309fde640 1 -- start start 2023-10-25T00:20:02.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7309fde640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730406df30 0x7f7304070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:02.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7309fde640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7304070b00 0x7f73040aa830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:02.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7309fde640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f730414eb70 0x7f73041402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:02.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7309fde640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7304069d40 con 0x7f730406df30 2023-10-25T00:20:02.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7309fde640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73040ac1c0 con 0x7f730414eb70 2023-10-25T00:20:02.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7309fde640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73040ac410 con 0x7f7304070b00 2023-10-25T00:20:02.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f73037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730406df30 0x7f7304070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:02.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f73037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730406df30 0x7f7304070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33874/0 (socket says 172.21.15.107:33874) 2023-10-25T00:20:02.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f73037fe640 1 -- 172.21.15.107:0/2569158340 learned_addr learned my addr 172.21.15.107:0/2569158340 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:02.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7302ffd640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7304070b00 0x7f73040aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:02.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7303fff640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f730414eb70 0x7f73041402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:02.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f73037fe640 1 -- 172.21.15.107:0/2569158340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7304070b00 msgr2=0x7f73040aa830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:02.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f73037fe640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7304070b00 0x7f73040aa830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:02.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f73037fe640 1 -- 172.21.15.107:0/2569158340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f730414eb70 msgr2=0x7f73041402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:02.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f73037fe640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f730414eb70 0x7f73041402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:02.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f73037fe640 1 -- 172.21.15.107:0/2569158340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f40094d0 con 0x7f730406df30 2023-10-25T00:20:02.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f73037fe640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730406df30 0x7f7304070140 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f72f401abf0 tx=0x7f72f40192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:02.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f72ebfff640 1 -- 172.21.15.107:0/2569158340 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f40bb5f0 con 0x7f730406df30 2023-10-25T00:20:02.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f72ebfff640 1 -- 172.21.15.107:0/2569158340 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f40bb770 con 0x7f730406df30 2023-10-25T00:20:02.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f72ebfff640 1 -- 172.21.15.107:0/2569158340 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f40bbb20 con 0x7f730406df30 2023-10-25T00:20:02.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7309fde640 1 -- 172.21.15.107:0/2569158340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73040a9e00 con 0x7f730406df30 2023-10-25T00:20:02.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7309fde640 1 -- 172.21.15.107:0/2569158340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730406a200 con 0x7f730406df30 2023-10-25T00:20:02.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7309fde640 1 -- 172.21.15.107:0/2569158340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730406a3f0 con 0x7f730406df30 2023-10-25T00:20:02.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f72ebfff640 1 -- 172.21.15.107:0/2569158340 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f72f40c53f0 con 0x7f730406df30 2023-10-25T00:20:02.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f72ebfff640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f72f0065e90 0x7f72f0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:02.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f72ebfff640 1 -- 172.21.15.107:0/2569158340 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7f72f411eef0 con 0x7f730406df30 2023-10-25T00:20:02.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7302ffd640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f72f0065e90 0x7f72f0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:02.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.666+0000 7f7302ffd640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f72f0065e90 0x7f72f0068350 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f730406ce70 tx=0x7f72f8007d60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:02.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.674+0000 7f72ebfff640 1 -- 172.21.15.107:0/2569158340 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f72f40eb4e0 con 0x7f730406df30 2023-10-25T00:20:02.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.926+0000 7f7309fde640 1 -- 172.21.15.107:0/2569158340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73040746e0 con 0x7f730406df30 2023-10-25T00:20:02.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.926+0000 7f72ebfff640 1 -- 172.21.15.107:0/2569158340 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5772 (secure 0 0 0) 0x7f72f40c0090 con 0x7f730406df30 2023-10-25T00:20:02.930 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:02.930 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":215}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:02.931 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 215 2023-10-25T00:20:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 -- 172.21.15.107:0/2569158340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f72f0065e90 msgr2=0x7f72f0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:02.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f72f0065e90 0x7f72f0068350 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f730406ce70 tx=0x7f72f8007d60 comp rx=0 tx=0).stop 2023-10-25T00:20:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 -- 172.21.15.107:0/2569158340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730406df30 msgr2=0x7f7304070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730406df30 0x7f7304070140 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f72f401abf0 tx=0x7f72f40192e0 comp rx=0 tx=0).stop 2023-10-25T00:20:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 -- 172.21.15.107:0/2569158340 shutdown_connections 2023-10-25T00:20:02.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f730414eb70 0x7f73041402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f72f0065e90 0x7f72f0068350 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7304070b00 0x7f73040aa830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 --2- 172.21.15.107:0/2569158340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f730406df30 0x7f7304070140 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 -- 172.21.15.107:0/2569158340 >> 172.21.15.107:0/2569158340 conn(0x7f73040c39a0 msgr2=0x7f73040c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 -- 172.21.15.107:0/2569158340 shutdown_connections 2023-10-25T00:20:02.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:02.930+0000 7f7309fde640 1 -- 172.21.15.107:0/2569158340 wait complete. 2023-10-25T00:20:02.954 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2023-10-25T00:20:02.955 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-25T00:20:02.955 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:20:03.054 INFO:tasks.ceph.mds.b:Stopped 2023-10-25T00:20:03.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:03.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a7e234640 1 Processor -- start 2023-10-25T00:20:03.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a7e234640 1 -- start start 2023-10-25T00:20:03.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a7e234640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4a780c8100 0x7f4a780abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:03.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a7e234640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4a7806a4b0 0x7f4a780ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:03.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a7e234640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a78155680 con 0x7f4a78152040 2023-10-25T00:20:03.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a7e234640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a78154a70 con 0x7f4a780c8100 2023-10-25T00:20:03.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a7e234640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a78154d50 con 0x7f4a7806a4b0 2023-10-25T00:20:03.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a76ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4a780c8100 0x7f4a780abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:03.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a76ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4a780c8100 0x7f4a780abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33884/0 (socket says 172.21.15.107:33884) 2023-10-25T00:20:03.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a76ffd640 1 -- 172.21.15.107:0/2872949445 learned_addr learned my addr 172.21.15.107:0/2872949445 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:03.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a77fff640 1 --2- 172.21.15.107:0/2872949445 >> v2:172.21.15.110:3300/0 conn(0x7f4a7806a4b0 0x7f4a780ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:03.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a76ffd640 1 -- 172.21.15.107:0/2872949445 >> v2:172.21.15.110:3300/0 conn(0x7f4a7806a4b0 msgr2=0x7f4a780ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:03.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a76ffd640 1 --2- 172.21.15.107:0/2872949445 >> v2:172.21.15.110:3300/0 conn(0x7f4a7806a4b0 0x7f4a780ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:03.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a76ffd640 1 -- 172.21.15.107:0/2872949445 >> v1:172.21.15.107:6789/0 conn(0x7f4a78152040 legacy=0x7f4a780c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:03.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a76ffd640 1 -- 172.21.15.107:0/2872949445 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a78155030 con 0x7f4a780c8100 2023-10-25T00:20:03.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.186+0000 7f4a77fff640 1 --2- 172.21.15.107:0/2872949445 >> v2:172.21.15.110:3300/0 conn(0x7f4a7806a4b0 0x7f4a780ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:20:03.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a76ffd640 1 --2- 172.21.15.107:0/2872949445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a780c8100 0x7f4a780abd00 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f4a7806b590 tx=0x7f4a6c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35cab67fc7fcfb7a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:03.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a767fc640 1 -- 172.21.15.107:0/2872949445 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c0ab600 con 0x7f4a780c8100 2023-10-25T00:20:03.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a767fc640 1 -- 172.21.15.107:0/2872949445 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6c0a7050 con 0x7f4a780c8100 2023-10-25T00:20:03.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a767fc640 1 -- 172.21.15.107:0/2872949445 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c0ab980 con 0x7f4a780c8100 2023-10-25T00:20:03.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 -- 172.21.15.107:0/2872949445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a780c8100 msgr2=0x7f4a780abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:03.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 --2- 172.21.15.107:0/2872949445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a780c8100 0x7f4a780abd00 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f4a7806b590 tx=0x7f4a6c01c470 comp rx=0 tx=0).stop 2023-10-25T00:20:03.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 -- 172.21.15.107:0/2872949445 shutdown_connections 2023-10-25T00:20:03.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 --2- 172.21.15.107:0/2872949445 >> v2:172.21.15.110:3300/0 conn(0x7f4a7806a4b0 0x7f4a780ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:03.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 --2- 172.21.15.107:0/2872949445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a780c8100 0x7f4a780abd00 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:03.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 -- 172.21.15.107:0/2872949445 >> 172.21.15.107:0/2872949445 conn(0x7f4a780ca200 msgr2=0x7f4a780ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:03.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 -- 172.21.15.107:0/2872949445 shutdown_connections 2023-10-25T00:20:03.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 -- 172.21.15.107:0/2872949445 wait complete. 2023-10-25T00:20:03.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 Processor -- start 2023-10-25T00:20:03.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 -- start start 2023-10-25T00:20:03.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a7806a4b0 0x7f4a7814d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:03.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a780c8100 0x7f4a781eceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:03.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a78152040 0x7f4a7814e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:03.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a781eded0 con 0x7f4a7806a4b0 2023-10-25T00:20:03.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780aa440 con 0x7f4a78152040 2023-10-25T00:20:03.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a7e234640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a780aa720 con 0x7f4a780c8100 2023-10-25T00:20:03.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a7806a4b0 0x7f4a7814d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:03.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a777fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a7806a4b0 0x7f4a7814d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33900/0 (socket says 172.21.15.107:33900) 2023-10-25T00:20:03.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a777fe640 1 -- 172.21.15.107:0/3090600679 learned_addr learned my addr 172.21.15.107:0/3090600679 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:03.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a76ffd640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a780c8100 0x7f4a781eceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:03.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a77fff640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a78152040 0x7f4a7814e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:03.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a777fe640 1 -- 172.21.15.107:0/3090600679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a780c8100 msgr2=0x7f4a781eceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:03.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a777fe640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a780c8100 0x7f4a781eceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:03.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a777fe640 1 -- 172.21.15.107:0/3090600679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a78152040 msgr2=0x7f4a7814e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:03.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a777fe640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a78152040 0x7f4a7814e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:03.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.190+0000 7f4a777fe640 1 -- 172.21.15.107:0/3090600679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6c0094d0 con 0x7f4a7806a4b0 2023-10-25T00:20:03.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a777fe640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a7806a4b0 0x7f4a7814d850 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800c3c0 tx=0x7f4a6800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:03.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a5ffff640 1 -- 172.21.15.107:0/3090600679 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a680052d0 con 0x7f4a7806a4b0 2023-10-25T00:20:03.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a5ffff640 1 -- 172.21.15.107:0/3090600679 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a68005430 con 0x7f4a7806a4b0 2023-10-25T00:20:03.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a7e234640 1 -- 172.21.15.107:0/3090600679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a781f04c0 con 0x7f4a7806a4b0 2023-10-25T00:20:03.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a5ffff640 1 -- 172.21.15.107:0/3090600679 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68003840 con 0x7f4a7806a4b0 2023-10-25T00:20:03.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a7e234640 1 -- 172.21.15.107:0/3090600679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a7814e5d0 con 0x7f4a7806a4b0 2023-10-25T00:20:03.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a7e234640 1 -- 172.21.15.107:0/3090600679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a781f09e0 con 0x7f4a7806a4b0 2023-10-25T00:20:03.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a5ffff640 1 -- 172.21.15.107:0/3090600679 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4a680039a0 con 0x7f4a7806a4b0 2023-10-25T00:20:03.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a5ffff640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4a64065e60 0x7f4a64068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:03.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a5ffff640 1 -- 172.21.15.107:0/3090600679 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7f4a68083e90 con 0x7f4a7806a4b0 2023-10-25T00:20:03.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a76ffd640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4a64065e60 0x7f4a64068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:03.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.194+0000 7f4a76ffd640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4a64065e60 0x7f4a64068320 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0ad000 tx=0x7f4a6c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:03.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.198+0000 7f4a5ffff640 1 -- 172.21.15.107:0/3090600679 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4a781f09e0 con 0x7f4a7806a4b0 2023-10-25T00:20:03.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 -- 172.21.15.107:0/3090600679 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a781f09e0 con 0x7f4a7806a4b0 2023-10-25T00:20:03.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a5ffff640 1 -- 172.21.15.107:0/3090600679 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5772 (secure 0 0 0) 0x7f4a781f09e0 con 0x7f4a7806a4b0 2023-10-25T00:20:03.460 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:03.461 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":215}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:03.461 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 215 2023-10-25T00:20:03.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 -- 172.21.15.107:0/3090600679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4a64065e60 msgr2=0x7f4a64068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:03.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4a64065e60 0x7f4a64068320 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0ad000 tx=0x7f4a6c008040 comp rx=0 tx=0).stop 2023-10-25T00:20:03.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 -- 172.21.15.107:0/3090600679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a7806a4b0 msgr2=0x7f4a7814d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:03.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a7806a4b0 0x7f4a7814d850 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800c3c0 tx=0x7f4a6800c880 comp rx=0 tx=0).stop 2023-10-25T00:20:03.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 -- 172.21.15.107:0/3090600679 shutdown_connections 2023-10-25T00:20:03.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a78152040 0x7f4a7814e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:03.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4a64065e60 0x7f4a64068320 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:03.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a780c8100 0x7f4a781eceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:03.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 --2- 172.21.15.107:0/3090600679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a7806a4b0 0x7f4a7814d850 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:03.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 -- 172.21.15.107:0/3090600679 >> 172.21.15.107:0/3090600679 conn(0x7f4a780ca200 msgr2=0x7f4a78154030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:03.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 -- 172.21.15.107:0/3090600679 shutdown_connections 2023-10-25T00:20:03.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:03.458+0000 7f4a7e234640 1 -- 172.21.15.107:0/3090600679 wait complete. 2023-10-25T00:20:03.484 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:20:05.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:05.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f90310b9640 1 Processor -- start 2023-10-25T00:20:05.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f90310b9640 1 -- start start 2023-10-25T00:20:05.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f90310b9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f902c152040 0x7f902c158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:05.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f90310b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 0x7f902c071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:05.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f90310b9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c152420 con 0x7f902c071fc0 2023-10-25T00:20:05.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f90310b9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f902c1556e0 con 0x7f902c152040 2023-10-25T00:20:05.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f90310b9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902c154d40 con 0x7f902c0adb80 2023-10-25T00:20:05.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f902b59e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f902c071fc0 0x7f902c06e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58776/0 (socket says 172.21.15.107:58776) 2023-10-25T00:20:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f902b59e640 1 -- 172.21.15.107:0/1924568660 learned_addr learned my addr 172.21.15.107:0/1924568660 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:20:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f902a59c640 1 --2- 172.21.15.107:0/1924568660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 0x7f902c071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f9029d9b640 1 -- 172.21.15.107:0/1924568660 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996980876 0 0) 0x7f902c152420 con 0x7f902c071fc0 2023-10-25T00:20:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f9029d9b640 1 -- 172.21.15.107:0/1924568660 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9014003530 con 0x7f902c071fc0 2023-10-25T00:20:05.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f902a59c640 1 -- 172.21.15.107:0/1924568660 >> v2:172.21.15.110:3300/0 conn(0x7f902c152040 msgr2=0x7f902c158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:05.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f902a59c640 1 --2- 172.21.15.107:0/1924568660 >> v2:172.21.15.110:3300/0 conn(0x7f902c152040 0x7f902c158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:05.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f902a59c640 1 -- 172.21.15.107:0/1924568660 >> v1:172.21.15.107:6789/0 conn(0x7f902c071fc0 legacy=0x7f902c06e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:05.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.634+0000 7f902a59c640 1 -- 172.21.15.107:0/1924568660 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c155020 con 0x7f902c0adb80 2023-10-25T00:20:05.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f902a59c640 1 --2- 172.21.15.107:0/1924568660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 0x7f902c071a80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f90200097c0 tx=0x7f902001c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a314cc61c89bad27 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:05.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f9029d9b640 1 -- 172.21.15.107:0/1924568660 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90200ab600 con 0x7f902c0adb80 2023-10-25T00:20:05.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f9029d9b640 1 -- 172.21.15.107:0/1924568660 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90200a7050 con 0x7f902c0adb80 2023-10-25T00:20:05.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f9029d9b640 1 -- 172.21.15.107:0/1924568660 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90200ab980 con 0x7f902c0adb80 2023-10-25T00:20:05.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f90310b9640 1 -- 172.21.15.107:0/1924568660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 msgr2=0x7f902c071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:05.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f90310b9640 1 --2- 172.21.15.107:0/1924568660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 0x7f902c071a80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f90200097c0 tx=0x7f902001c3d0 comp rx=0 tx=0).stop 2023-10-25T00:20:05.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f90310b9640 1 -- 172.21.15.107:0/1924568660 shutdown_connections 2023-10-25T00:20:05.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f90310b9640 1 --2- 172.21.15.107:0/1924568660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 0x7f902c071a80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:05.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f90310b9640 1 --2- 172.21.15.107:0/1924568660 >> v2:172.21.15.110:3300/0 conn(0x7f902c152040 0x7f902c158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:05.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f90310b9640 1 -- 172.21.15.107:0/1924568660 >> 172.21.15.107:0/1924568660 conn(0x7f902c070f60 msgr2=0x7f902c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:05.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f90310b9640 1 -- 172.21.15.107:0/1924568660 shutdown_connections 2023-10-25T00:20:05.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.638+0000 7f90310b9640 1 -- 172.21.15.107:0/1924568660 wait complete. 2023-10-25T00:20:05.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90310b9640 1 Processor -- start 2023-10-25T00:20:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90310b9640 1 -- start start 2023-10-25T00:20:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90310b9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c071fc0 0x7f902c145950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90310b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 0x7f902c145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90310b9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902c152040 0x7f902c150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:05.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90310b9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c0c7890 con 0x7f902c071fc0 2023-10-25T00:20:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90310b9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902c14b2d0 con 0x7f902c152040 2023-10-25T00:20:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90310b9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902c149e10 con 0x7f902c0adb80 2023-10-25T00:20:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f902a59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 0x7f902c145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:05.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f902a59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 0x7f902c145e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:55096/0 (socket says 172.21.15.107:55096) 2023-10-25T00:20:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f902a59c640 1 -- 172.21.15.107:0/2488820578 learned_addr learned my addr 172.21.15.107:0/2488820578 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f902a59c640 1 -- 172.21.15.107:0/2488820578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902c152040 msgr2=0x7f902c150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f902a59c640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902c152040 0x7f902c150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f902a59c640 1 -- 172.21.15.107:0/2488820578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c071fc0 msgr2=0x7f902c145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:05.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f902a59c640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c071fc0 0x7f902c145950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:05.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f902a59c640 1 -- 172.21.15.107:0/2488820578 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9020009420 con 0x7f902c0adb80 2023-10-25T00:20:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f902a59c640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 0x7f902c145e90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f90200097c0 tx=0x7f9020019170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90137fe640 1 -- 172.21.15.107:0/2488820578 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90200b4430 con 0x7f902c0adb80 2023-10-25T00:20:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90310b9640 1 -- 172.21.15.107:0/2488820578 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902c14ad10 con 0x7f902c0adb80 2023-10-25T00:20:05.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90137fe640 1 -- 172.21.15.107:0/2488820578 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90200b4590 con 0x7f902c0adb80 2023-10-25T00:20:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90137fe640 1 -- 172.21.15.107:0/2488820578 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90200b4940 con 0x7f902c0adb80 2023-10-25T00:20:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.642+0000 7f90310b9640 1 -- 172.21.15.107:0/2488820578 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902c14a0b0 con 0x7f902c0adb80 2023-10-25T00:20:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.646+0000 7f90137fe640 1 -- 172.21.15.107:0/2488820578 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f90200b2040 con 0x7f902c0adb80 2023-10-25T00:20:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.646+0000 7f90137fe640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9018065e60 0x7f9018068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.646+0000 7f90310b9640 1 -- 172.21.15.107:0/2488820578 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902c14a2a0 con 0x7f902c0adb80 2023-10-25T00:20:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.646+0000 7f902ad9d640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9018065e60 0x7f9018068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.646+0000 7f90137fe640 1 -- 172.21.15.107:0/2488820578 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7f902011e980 con 0x7f902c0adb80 2023-10-25T00:20:05.652 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.646+0000 7f902ad9d640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9018065e60 0x7f9018068320 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f901c009690 tx=0x7f901c006c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:05.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.650+0000 7f90137fe640 1 -- 172.21.15.107:0/2488820578 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f90200ee950 con 0x7f902c0adb80 2023-10-25T00:20:05.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 -- 172.21.15.107:0/2488820578 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f902c0746e0 con 0x7f902c0adb80 2023-10-25T00:20:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90137fe640 1 -- 172.21.15.107:0/2488820578 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5772 (secure 0 0 0) 0x7f90200ee4e0 con 0x7f902c0adb80 2023-10-25T00:20:05.925 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:05.925 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":215}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 215 2023-10-25T00:20:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 -- 172.21.15.107:0/2488820578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9018065e60 msgr2=0x7f9018068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9018065e60 0x7f9018068320 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f901c009690 tx=0x7f901c006c00 comp rx=0 tx=0).stop 2023-10-25T00:20:05.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 -- 172.21.15.107:0/2488820578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 msgr2=0x7f902c145e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:05.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 0x7f902c145e90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f90200097c0 tx=0x7f9020019170 comp rx=0 tx=0).stop 2023-10-25T00:20:05.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 -- 172.21.15.107:0/2488820578 shutdown_connections 2023-10-25T00:20:05.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f902c152040 0x7f902c150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9018065e60 0x7f9018068320 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f902c0adb80 0x7f902c145e90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 --2- 172.21.15.107:0/2488820578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f902c071fc0 0x7f902c145950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 -- 172.21.15.107:0/2488820578 >> 172.21.15.107:0/2488820578 conn(0x7f902c070f60 msgr2=0x7f902c0c2860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 -- 172.21.15.107:0/2488820578 shutdown_connections 2023-10-25T00:20:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:05.922+0000 7f90310b9640 1 -- 172.21.15.107:0/2488820578 wait complete. 2023-10-25T00:20:05.951 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:20:07.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:08.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa832efd640 1 Processor -- start 2023-10-25T00:20:08.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa832efd640 1 -- start start 2023-10-25T00:20:08.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa832efd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa82c13dc20 0x7fa82c0c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:08.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa832efd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa82c157e50 0x7fa82c158260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:08.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa832efd640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c1587a0 con 0x7fa82c0720c0 2023-10-25T00:20:08.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa832efd640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82c154860 con 0x7fa82c157e50 2023-10-25T00:20:08.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa832efd640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82c0c1f10 con 0x7fa82c13dc20 2023-10-25T00:20:08.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa83149a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa82c157e50 0x7fa82c158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:08.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa83149a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa82c157e50 0x7fa82c158260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33930/0 (socket says 172.21.15.107:33930) 2023-10-25T00:20:08.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa83149a640 1 -- 172.21.15.107:0/1705227968 learned_addr learned my addr 172.21.15.107:0/1705227968 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:08.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa82bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa82c0720c0 0x7fa82c06df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:58778/0 (socket says 172.21.15.107:58778) 2023-10-25T00:20:08.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa830c99640 1 --2- 172.21.15.107:0/1705227968 >> v2:172.21.15.110:3300/0 conn(0x7fa82c13dc20 0x7fa82c0c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:08.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa83149a640 1 -- 172.21.15.107:0/1705227968 >> v2:172.21.15.110:3300/0 conn(0x7fa82c13dc20 msgr2=0x7fa82c0c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:08.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa83149a640 1 --2- 172.21.15.107:0/1705227968 >> v2:172.21.15.110:3300/0 conn(0x7fa82c13dc20 0x7fa82c0c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:08.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa83149a640 1 -- 172.21.15.107:0/1705227968 >> v1:172.21.15.107:6789/0 conn(0x7fa82c0720c0 legacy=0x7fa82c06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:08.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa83149a640 1 -- 172.21.15.107:0/1705227968 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c154b20 con 0x7fa82c157e50 2023-10-25T00:20:08.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.082+0000 7fa82b7fe640 1 -- 172.21.15.107:0/1705227968 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591912835 0 0) 0x7fa82c1587a0 con 0x7fa82c0720c0 2023-10-25T00:20:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa83149a640 1 --2- 172.21.15.107:0/1705227968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa82c157e50 0x7fa82c158260 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0c8020 tx=0x7fa82401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc5f09dc9587bf4c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa82b7fe640 1 -- 172.21.15.107:0/1705227968 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8240a5660 con 0x7fa82c157e50 2023-10-25T00:20:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa82b7fe640 1 -- 172.21.15.107:0/1705227968 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8240a57c0 con 0x7fa82c157e50 2023-10-25T00:20:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa82b7fe640 1 -- 172.21.15.107:0/1705227968 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8240a5b50 con 0x7fa82c157e50 2023-10-25T00:20:08.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa832efd640 1 -- 172.21.15.107:0/1705227968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa82c157e50 msgr2=0x7fa82c158260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa832efd640 1 --2- 172.21.15.107:0/1705227968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa82c157e50 0x7fa82c158260 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0c8020 tx=0x7fa82401c8b0 comp rx=0 tx=0).stop 2023-10-25T00:20:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa832efd640 1 -- 172.21.15.107:0/1705227968 shutdown_connections 2023-10-25T00:20:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa832efd640 1 --2- 172.21.15.107:0/1705227968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa82c157e50 0x7fa82c158260 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa832efd640 1 --2- 172.21.15.107:0/1705227968 >> v2:172.21.15.110:3300/0 conn(0x7fa82c13dc20 0x7fa82c0c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:08.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa832efd640 1 -- 172.21.15.107:0/1705227968 >> 172.21.15.107:0/1705227968 conn(0x7fa82c0c39a0 msgr2=0x7fa82c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:08.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa832efd640 1 -- 172.21.15.107:0/1705227968 shutdown_connections 2023-10-25T00:20:08.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa832efd640 1 -- 172.21.15.107:0/1705227968 wait complete. 2023-10-25T00:20:08.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa832efd640 1 Processor -- start 2023-10-25T00:20:08.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.086+0000 7fa832efd640 1 -- start start 2023-10-25T00:20:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa832efd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa82c0720c0 0x7fa82c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa832efd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa82c13dc20 0x7fa82c15aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:08.095 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa832efd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa82c157e50 0x7fa82c15b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa832efd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0ac310 con 0x7fa82c0720c0 2023-10-25T00:20:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa832efd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0aa810 con 0x7fa82c13dc20 2023-10-25T00:20:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa832efd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0aaaf0 con 0x7fa82c157e50 2023-10-25T00:20:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa830c99640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa82c0720c0 0x7fa82c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:08.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa830c99640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa82c0720c0 0x7fa82c06d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33932/0 (socket says 172.21.15.107:33932) 2023-10-25T00:20:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa830c99640 1 -- 172.21.15.107:0/3003460324 learned_addr learned my addr 172.21.15.107:0/3003460324 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa830c99640 1 -- 172.21.15.107:0/3003460324 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa82c157e50 msgr2=0x7fa82c15b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa830c99640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa82c157e50 0x7fa82c15b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa830c99640 1 -- 172.21.15.107:0/3003460324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa82c13dc20 msgr2=0x7fa82c15aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa830c99640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa82c13dc20 0x7fa82c15aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa830c99640 1 -- 172.21.15.107:0/3003460324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa824009b30 con 0x7fa82c0720c0 2023-10-25T00:20:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa830c99640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa82c0720c0 0x7fa82c06d180 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fa82000e930 tx=0x7fa82000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa8297fa640 1 -- 172.21.15.107:0/3003460324 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820009690 con 0x7fa82c0720c0 2023-10-25T00:20:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa8297fa640 1 -- 172.21.15.107:0/3003460324 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82000ca00 con 0x7fa82c0720c0 2023-10-25T00:20:08.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.090+0000 7fa8297fa640 1 -- 172.21.15.107:0/3003460324 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82000cd70 con 0x7fa82c0720c0 2023-10-25T00:20:08.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.094+0000 7fa832efd640 1 -- 172.21.15.107:0/3003460324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82c0aadd0 con 0x7fa82c0720c0 2023-10-25T00:20:08.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.094+0000 7fa832efd640 1 -- 172.21.15.107:0/3003460324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82c0ab390 con 0x7fa82c0720c0 2023-10-25T00:20:08.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.094+0000 7fa832efd640 1 -- 172.21.15.107:0/3003460324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82c0746e0 con 0x7fa82c0720c0 2023-10-25T00:20:08.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.094+0000 7fa8297fa640 1 -- 172.21.15.107:0/3003460324 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa820013470 con 0x7fa82c0720c0 2023-10-25T00:20:08.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.098+0000 7fa8297fa640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa810065e60 0x7fa810068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:08.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.098+0000 7fa8297fa640 1 -- 172.21.15.107:0/3003460324 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7fa8200853f0 con 0x7fa82c0720c0 2023-10-25T00:20:08.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.098+0000 7fa82bfff640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa810065e60 0x7fa810068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:08.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.098+0000 7fa82bfff640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa810065e60 0x7fa810068320 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa82c06ac50 tx=0x7fa818007990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:08.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.102+0000 7fa8297fa640 1 -- 172.21.15.107:0/3003460324 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa8200519b0 con 0x7fa82c0720c0 2023-10-25T00:20:08.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.362+0000 7fa832efd640 1 -- 172.21.15.107:0/3003460324 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa82c13c690 con 0x7fa82c0720c0 2023-10-25T00:20:08.365 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:08.366 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":215}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:08.366 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 215 2023-10-25T00:20:08.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.362+0000 7fa8297fa640 1 -- 172.21.15.107:0/3003460324 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5772 (secure 0 0 0) 0x7fa820021090 con 0x7fa82c0720c0 2023-10-25T00:20:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 -- 172.21.15.107:0/3003460324 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa810065e60 msgr2=0x7fa810068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa810065e60 0x7fa810068320 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa82c06ac50 tx=0x7fa818007990 comp rx=0 tx=0).stop 2023-10-25T00:20:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 -- 172.21.15.107:0/3003460324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa82c0720c0 msgr2=0x7fa82c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa82c0720c0 0x7fa82c06d180 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fa82000e930 tx=0x7fa82000edf0 comp rx=0 tx=0).stop 2023-10-25T00:20:08.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 -- 172.21.15.107:0/3003460324 shutdown_connections 2023-10-25T00:20:08.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa810065e60 0x7fa810068320 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:08.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa82c157e50 0x7fa82c15b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:08.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa82c13dc20 0x7fa82c15aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:08.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 --2- 172.21.15.107:0/3003460324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa82c0720c0 0x7fa82c06d180 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:08.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 -- 172.21.15.107:0/3003460324 >> 172.21.15.107:0/3003460324 conn(0x7fa82c0c39a0 msgr2=0x7fa82c06a490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:08.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 -- 172.21.15.107:0/3003460324 shutdown_connections 2023-10-25T00:20:08.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:08.366+0000 7fa832efd640 1 -- 172.21.15.107:0/3003460324 wait complete. 2023-10-25T00:20:08.432 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:20:10.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:10.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b18699640 1 Processor -- start 2023-10-25T00:20:10.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b18699640 1 -- start start 2023-10-25T00:20:10.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b18699640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3b10071770 0x7f3b10151480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:10.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b18699640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3b100c2790 0x7f3b1006cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:10.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b18699640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3b1006d430 0x7f3b1006fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:10.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b18699640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b10071b50 con 0x7f3b1006d430 2023-10-25T00:20:10.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b18699640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b100c44c0 con 0x7f3b10071770 2023-10-25T00:20:10.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b18699640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b100c38b0 con 0x7f3b100c2790 2023-10-25T00:20:10.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b16c36640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3b1006d430 0x7f3b1006fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:10.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b15c34640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3b100c2790 0x7f3b1006cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:10.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b16c36640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3b1006d430 0x7f3b1006fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33938/0 (socket says 172.21.15.107:33938) 2023-10-25T00:20:10.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b16c36640 1 -- 172.21.15.107:0/2942595973 learned_addr learned my addr 172.21.15.107:0/2942595973 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:10.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b16435640 1 --2- 172.21.15.107:0/2942595973 >> v2:172.21.15.110:3300/0 conn(0x7f3b10071770 0x7f3b10151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:10.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b16c36640 1 -- 172.21.15.107:0/2942595973 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3b100c2790 msgr2=0x7f3b1006cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:10.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b16c36640 1 --2- 172.21.15.107:0/2942595973 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3b100c2790 0x7f3b1006cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b16c36640 1 -- 172.21.15.107:0/2942595973 >> v2:172.21.15.110:3300/0 conn(0x7f3b10071770 msgr2=0x7f3b10151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:10.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b16c36640 1 --2- 172.21.15.107:0/2942595973 >> v2:172.21.15.110:3300/0 conn(0x7f3b10071770 0x7f3b10151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.586+0000 7f3b16c36640 1 -- 172.21.15.107:0/2942595973 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b100c3b90 con 0x7f3b1006d430 2023-10-25T00:20:10.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.590+0000 7f3b16c36640 1 --2- 172.21.15.107:0/2942595973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b1006d430 0x7f3b1006fde0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f3b100c1e60 tx=0x7f3b08020cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2a5897f996c6e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:10.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.590+0000 7f3b15433640 1 -- 172.21.15.107:0/2942595973 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b10070b30 con 0x7f3b1006d430 2023-10-25T00:20:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.590+0000 7f3b15433640 1 -- 172.21.15.107:0/2942595973 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0800b2d0 con 0x7f3b1006d430 2023-10-25T00:20:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.590+0000 7f3b15433640 1 -- 172.21.15.107:0/2942595973 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b080a9c80 con 0x7f3b1006d430 2023-10-25T00:20:10.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.590+0000 7f3b18699640 1 -- 172.21.15.107:0/2942595973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b1006d430 msgr2=0x7f3b1006fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.590+0000 7f3b18699640 1 --2- 172.21.15.107:0/2942595973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b1006d430 0x7f3b1006fde0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f3b100c1e60 tx=0x7f3b08020cc0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.590+0000 7f3b18699640 1 -- 172.21.15.107:0/2942595973 shutdown_connections 2023-10-25T00:20:10.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.590+0000 7f3b18699640 1 --2- 172.21.15.107:0/2942595973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b1006d430 0x7f3b1006fde0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.590+0000 7f3b18699640 1 --2- 172.21.15.107:0/2942595973 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3b100c2790 0x7f3b1006cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.590+0000 7f3b18699640 1 --2- 172.21.15.107:0/2942595973 >> v2:172.21.15.110:3300/0 conn(0x7f3b10071770 0x7f3b10151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.590+0000 7f3b18699640 1 -- 172.21.15.107:0/2942595973 >> 172.21.15.107:0/2942595973 conn(0x7f3b100c6ed0 msgr2=0x7f3b100c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 -- 172.21.15.107:0/2942595973 shutdown_connections 2023-10-25T00:20:10.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 -- 172.21.15.107:0/2942595973 wait complete. 2023-10-25T00:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 Processor -- start 2023-10-25T00:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 -- start start 2023-10-25T00:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b1006d430 0x7f3b10146d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3b10071770 0x7f3b10141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:10.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b100c2790 0x7f3b10141cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b100abb80 con 0x7f3b100c2790 2023-10-25T00:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b100ab8a0 con 0x7f3b1006d430 2023-10-25T00:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1014de40 con 0x7f3b10071770 2023-10-25T00:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b16c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b100c2790 0x7f3b10141cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:10.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b16c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b100c2790 0x7f3b10141cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33944/0 (socket says 172.21.15.107:33944) 2023-10-25T00:20:10.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b16c36640 1 -- 172.21.15.107:0/4202225541 learned_addr learned my addr 172.21.15.107:0/4202225541 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:10.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b15c34640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3b10071770 0x7f3b10141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:10.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b16435640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b1006d430 0x7f3b10146d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:10.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b16c36640 1 -- 172.21.15.107:0/4202225541 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3b10071770 msgr2=0x7f3b10141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b16c36640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3b10071770 0x7f3b10141780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b16c36640 1 -- 172.21.15.107:0/4202225541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b1006d430 msgr2=0x7f3b10146d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b16c36640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b1006d430 0x7f3b10146d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b16c36640 1 -- 172.21.15.107:0/4202225541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0800bd40 con 0x7f3b100c2790 2023-10-25T00:20:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b16c36640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b100c2790 0x7f3b10141cc0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f3b080b3040 tx=0x7f3b0801da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b06ffd640 1 -- 172.21.15.107:0/4202225541 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0800b440 con 0x7f3b100c2790 2023-10-25T00:20:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b06ffd640 1 -- 172.21.15.107:0/4202225541 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b08009810 con 0x7f3b100c2790 2023-10-25T00:20:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 -- 172.21.15.107:0/4202225541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1014e120 con 0x7f3b100c2790 2023-10-25T00:20:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b06ffd640 1 -- 172.21.15.107:0/4202225541 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b08009ba0 con 0x7f3b100c2790 2023-10-25T00:20:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.594+0000 7f3b18699640 1 -- 172.21.15.107:0/4202225541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1014e6e0 con 0x7f3b100c2790 2023-10-25T00:20:10.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.598+0000 7f3b18699640 1 -- 172.21.15.107:0/4202225541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1014e8d0 con 0x7f3b100c2790 2023-10-25T00:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.598+0000 7f3b06ffd640 1 -- 172.21.15.107:0/4202225541 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3b08009d00 con 0x7f3b100c2790 2023-10-25T00:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.598+0000 7f3b06ffd640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3af4065e60 0x7f3af4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.598+0000 7f3b06ffd640 1 -- 172.21.15.107:0/4202225541 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7f3b08124040 con 0x7f3b100c2790 2023-10-25T00:20:10.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.602+0000 7f3b16435640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3af4065e60 0x7f3af4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:10.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.602+0000 7f3b06ffd640 1 -- 172.21.15.107:0/4202225541 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3b08155040 con 0x7f3b100c2790 2023-10-25T00:20:10.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.606+0000 7f3b16435640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3af4065e60 0x7f3af4068320 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009b60 tx=0x7f3b0c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:10.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.886+0000 7f3b18699640 1 -- 172.21.15.107:0/4202225541 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b100746e0 con 0x7f3b100c2790 2023-10-25T00:20:10.892 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:10.892 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":215}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:10.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.890+0000 7f3b06ffd640 1 -- 172.21.15.107:0/4202225541 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5772 (secure 0 0 0) 0x7f3b080c0090 con 0x7f3b100c2790 2023-10-25T00:20:10.893 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 215 2023-10-25T00:20:10.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 -- 172.21.15.107:0/4202225541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3af4065e60 msgr2=0x7f3af4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:10.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3af4065e60 0x7f3af4068320 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009b60 tx=0x7f3b0c009190 comp rx=0 tx=0).stop 2023-10-25T00:20:10.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 -- 172.21.15.107:0/4202225541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b100c2790 msgr2=0x7f3b10141cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:10.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b100c2790 0x7f3b10141cc0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f3b080b3040 tx=0x7f3b0801da60 comp rx=0 tx=0).stop 2023-10-25T00:20:10.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 -- 172.21.15.107:0/4202225541 shutdown_connections 2023-10-25T00:20:10.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b100c2790 0x7f3b10141cc0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3af4065e60 0x7f3af4068320 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3b10071770 0x7f3b10141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 --2- 172.21.15.107:0/4202225541 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3b1006d430 0x7f3b10146d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:10.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 -- 172.21.15.107:0/4202225541 >> 172.21.15.107:0/4202225541 conn(0x7f3b100c6ed0 msgr2=0x7f3b1015b250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:10.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 -- 172.21.15.107:0/4202225541 shutdown_connections 2023-10-25T00:20:10.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:10.894+0000 7f3b18699640 1 -- 172.21.15.107:0/4202225541 wait complete. 2023-10-25T00:20:10.916 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:20:12.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:13.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f60919640 1 Processor -- start 2023-10-25T00:20:13.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f60919640 1 -- start start 2023-10-25T00:20:13.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f60919640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f6f5c152040 0x7f6f5c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:13.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f60919640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c06a4b0 0x7f6f5c0ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:13.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f60919640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5c155680 con 0x7f6f5c0c8100 2023-10-25T00:20:13.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f60919640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f5c154a70 con 0x7f6f5c152040 2023-10-25T00:20:13.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f60919640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c154d50 con 0x7f6f5c06a4b0 2023-10-25T00:20:13.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f53fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6f5c0c8100 0x7f6f5c0abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37174/0 (socket says 172.21.15.107:37174) 2023-10-25T00:20:13.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f53fff640 1 -- 172.21.15.107:0/4003301813 learned_addr learned my addr 172.21.15.107:0/4003301813 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:20:13.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791597937 0 0) 0x7f6f5c155680 con 0x7f6f5c0c8100 2023-10-25T00:20:13.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f48003530 con 0x7f6f5c0c8100 2023-10-25T00:20:13.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3009601421 0 0) 0x7f6f48003530 con 0x7f6f5c0c8100 2023-10-25T00:20:13.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c06a4b0 msgr2=0x7f6f5c0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:13.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 --2- 172.21.15.107:0/4003301813 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c06a4b0 0x7f6f5c0ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 >> v2:172.21.15.107:3300/0 conn(0x7f6f5c152040 msgr2=0x7f6f5c0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:13.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f5a59c640 1 --2- 172.21.15.107:0/4003301813 >> v2:172.21.15.107:3300/0 conn(0x7f6f5c152040 0x7f6f5c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:13.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 --2- 172.21.15.107:0/4003301813 >> v2:172.21.15.107:3300/0 conn(0x7f6f5c152040 0x7f6f5c0c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c155030 con 0x7f6f5c0c8100 2023-10-25T00:20:13.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6f44001680 con 0x7f6f5c0c8100 2023-10-25T00:20:13.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 >> v1:172.21.15.107:6789/0 conn(0x7f6f5c0c8100 legacy=0x7f6f5c0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:13.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 --2- 172.21.15.107:0/4003301813 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f48003980 0x7f6f48003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:13.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 --2- 172.21.15.107:0/4003301813 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c06a4b0 0x7f6f480047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:13.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 --2- 172.21.15.107:0/4003301813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f48004d10 0x7f6f48005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:13.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c154a70 con 0x7f6f48004d10 2023-10-25T00:20:13.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c154d50 con 0x7f6f48003980 2023-10-25T00:20:13.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4800bc70 con 0x7f6f5c06a4b0 2023-10-25T00:20:13.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f5a59c640 1 --2- 172.21.15.107:0/4003301813 >> v2:172.21.15.107:3300/0 conn(0x7f6f5c152040 0x7f6f5c0c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:20:13.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f5ad9d640 1 --2- 172.21.15.107:0/4003301813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f48004d10 0x7f6f48005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:13.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f5ad9d640 1 -- 172.21.15.107:0/4003301813 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c06a4b0 msgr2=0x7f6f480047d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:20:13.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f5ad9d640 1 --2- 172.21.15.107:0/4003301813 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c06a4b0 0x7f6f480047d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.050+0000 7f6f5ad9d640 1 -- 172.21.15.107:0/4003301813 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f48003980 msgr2=0x7f6f48003da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:20:13.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.054+0000 7f6f5ad9d640 1 --2- 172.21.15.107:0/4003301813 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f48003980 0x7f6f48003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.054+0000 7f6f5ad9d640 1 -- 172.21.15.107:0/4003301813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c155030 con 0x7f6f48004d10 2023-10-25T00:20:13.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.054+0000 7f6f5ad9d640 1 --2- 172.21.15.107:0/4003301813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f48004d10 0x7f6f48005190 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009ca0 tx=0x7f6f5401cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6132f1c72e11e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:13.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.058+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f540a5c20 con 0x7f6f48004d10 2023-10-25T00:20:13.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.058+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f540a5da0 con 0x7f6f48004d10 2023-10-25T00:20:13.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.058+0000 7f6f59d9b640 1 -- 172.21.15.107:0/4003301813 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f540ad680 con 0x7f6f48004d10 2023-10-25T00:20:13.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.058+0000 7f6f60919640 1 -- 172.21.15.107:0/4003301813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f48004d10 msgr2=0x7f6f48005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:13.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.058+0000 7f6f60919640 1 --2- 172.21.15.107:0/4003301813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f48004d10 0x7f6f48005190 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009ca0 tx=0x7f6f5401cc10 comp rx=0 tx=0).stop 2023-10-25T00:20:13.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.058+0000 7f6f5ad9d640 1 -- 172.21.15.107:0/4003301813 reap_dead start 2023-10-25T00:20:13.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.058+0000 7f6f60919640 1 -- 172.21.15.107:0/4003301813 shutdown_connections 2023-10-25T00:20:13.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.058+0000 7f6f60919640 1 -- 172.21.15.107:0/4003301813 >> 172.21.15.107:0/4003301813 conn(0x7f6f5c0ca200 msgr2=0x7f6f5c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:13.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.062+0000 7f6f60919640 1 -- 172.21.15.107:0/4003301813 shutdown_connections 2023-10-25T00:20:13.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.062+0000 7f6f60919640 1 -- 172.21.15.107:0/4003301813 wait complete. 2023-10-25T00:20:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.062+0000 7f6f60919640 1 Processor -- start 2023-10-25T00:20:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.062+0000 7f6f60919640 1 -- start start 2023-10-25T00:20:13.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f60919640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5c1df6f0 0x7f6f5c0bd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:13.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f60919640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c0ab7c0 0x7f6f5c0abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:13.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f60919640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f5c1f9b70 0x7f6f5c1fa020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:13.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f60919640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0c2e00 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f60919640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0ac180 con 0x7f6f5c1f9b70 2023-10-25T00:20:13.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f60919640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0ab050 con 0x7f6f5c0ab7c0 2023-10-25T00:20:13.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f5a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5c1df6f0 0x7f6f5c0bd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:13.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f5a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5c1df6f0 0x7f6f5c0bd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60452/0 (socket says 172.21.15.107:60452) 2023-10-25T00:20:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f5a59c640 1 -- 172.21.15.107:0/4223216438 learned_addr learned my addr 172.21.15.107:0/4223216438 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f53fff640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c0ab7c0 0x7f6f5c0abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f5a59c640 1 -- 172.21.15.107:0/4223216438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c0ab7c0 msgr2=0x7f6f5c0abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f5a59c640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c0ab7c0 0x7f6f5c0abc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f5a59c640 1 -- 172.21.15.107:0/4223216438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f5c1f9b70 msgr2=0x7f6f5c1fa020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f5a59c640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f5c1f9b70 0x7f6f5c1fa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f5a59c640 1 -- 172.21.15.107:0/4223216438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c0ac4f0 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.066+0000 7f6f53fff640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c0ab7c0 0x7f6f5c0abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:20:13.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.070+0000 7f6f5a59c640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5c1df6f0 0x7f6f5c0bd270 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00c7f0 tx=0x7f6f4c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:13.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.074+0000 7f6f52ffd640 1 -- 172.21.15.107:0/4223216438 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c0077c0 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.074+0000 7f6f52ffd640 1 -- 172.21.15.107:0/4223216438 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4c007920 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.074+0000 7f6f52ffd640 1 -- 172.21.15.107:0/4223216438 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c007cf0 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.074+0000 7f6f60919640 1 -- 172.21.15.107:0/4223216438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5c0ac7d0 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.074+0000 7f6f60919640 1 -- 172.21.15.107:0/4223216438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5c0ab2a0 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.074+0000 7f6f60919640 1 -- 172.21.15.107:0/4223216438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5c0ab4c0 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.078+0000 7f6f52ffd640 1 -- 172.21.15.107:0/4223216438 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6f4c0113f0 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.078+0000 7f6f52ffd640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f3c065ee0 0x7f6f3c0683a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:13.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.078+0000 7f6f52ffd640 1 -- 172.21.15.107:0/4223216438 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7f6f4c0848b0 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.078+0000 7f6f53fff640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f3c065ee0 0x7f6f3c0683a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:13.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.078+0000 7f6f53fff640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f3c065ee0 0x7f6f3c0683a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6f44000f70 tx=0x7f6f4400aad0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:13.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.082+0000 7f6f52ffd640 1 -- 172.21.15.107:0/4223216438 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6f4c050ec0 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.366+0000 7f6f60919640 1 -- 172.21.15.107:0/4223216438 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f5c0746e0 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.366+0000 7f6f52ffd640 1 -- 172.21.15.107:0/4223216438 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5772 (secure 0 0 0) 0x7f6f4c01f020 con 0x7f6f5c1df6f0 2023-10-25T00:20:13.368 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:13.368 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":215}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:13.369 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 215 2023-10-25T00:20:13.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.366+0000 7f6f60919640 1 -- 172.21.15.107:0/4223216438 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f3c065ee0 msgr2=0x7f6f3c0683a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:13.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.366+0000 7f6f60919640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f3c065ee0 0x7f6f3c0683a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6f44000f70 tx=0x7f6f4400aad0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.366+0000 7f6f60919640 1 -- 172.21.15.107:0/4223216438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5c1df6f0 msgr2=0x7f6f5c0bd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:13.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.366+0000 7f6f60919640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5c1df6f0 0x7f6f5c0bd270 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00c7f0 tx=0x7f6f4c00ccb0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.370+0000 7f6f60919640 1 -- 172.21.15.107:0/4223216438 shutdown_connections 2023-10-25T00:20:13.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.370+0000 7f6f60919640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f5c1f9b70 0x7f6f5c1fa020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.370+0000 7f6f60919640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6f3c065ee0 0x7f6f3c0683a0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.370+0000 7f6f60919640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f5c0ab7c0 0x7f6f5c0abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.370+0000 7f6f60919640 1 --2- 172.21.15.107:0/4223216438 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f5c1df6f0 0x7f6f5c0bd270 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:13.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.370+0000 7f6f60919640 1 -- 172.21.15.107:0/4223216438 >> 172.21.15.107:0/4223216438 conn(0x7f6f5c0ca200 msgr2=0x7f6f5c0ae850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:13.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.370+0000 7f6f60919640 1 -- 172.21.15.107:0/4223216438 shutdown_connections 2023-10-25T00:20:13.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:13.370+0000 7f6f60919640 1 -- 172.21.15.107:0/4223216438 wait complete. 2023-10-25T00:20:13.392 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:20:15.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:15.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c9d1af640 1 Processor -- start 2023-10-25T00:20:15.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c9d1af640 1 -- start start 2023-10-25T00:20:15.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c9d1af640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0c9814c470 0x7f0c9814c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:15.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c9d1af640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0c98152b30 0x7f0c98152f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:15.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c9d1af640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c98155cd0 con 0x7f0c9806f490 2023-10-25T00:20:15.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c9d1af640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c981526e0 con 0x7f0c9814c470 2023-10-25T00:20:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c9d1af640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c980c4380 con 0x7f0c98152b30 2023-10-25T00:20:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c96d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0c9814c470 0x7f0c9814c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c96d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0c9814c470 0x7f0c9814c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60462/0 (socket says 172.21.15.107:60462) 2023-10-25T00:20:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c96d9d640 1 -- 172.21.15.107:0/3436773718 learned_addr learned my addr 172.21.15.107:0/3436773718 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:15.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c9759e640 1 --2- 172.21.15.107:0/3436773718 >> v2:172.21.15.110:3300/0 conn(0x7f0c98152b30 0x7f0c98152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:15.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c96d9d640 1 -- 172.21.15.107:0/3436773718 >> v2:172.21.15.110:3300/0 conn(0x7f0c98152b30 msgr2=0x7f0c98152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:15.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c96d9d640 1 --2- 172.21.15.107:0/3436773718 >> v2:172.21.15.110:3300/0 conn(0x7f0c98152b30 0x7f0c98152f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:15.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c96d9d640 1 -- 172.21.15.107:0/3436773718 >> v1:172.21.15.107:6789/0 conn(0x7f0c9806f490 legacy=0x7f0c98071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:15.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.538+0000 7f0c96d9d640 1 -- 172.21.15.107:0/3436773718 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c980c48b0 con 0x7f0c9814c470 2023-10-25T00:20:15.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c96d9d640 1 --2- 172.21.15.107:0/3436773718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c9814c470 0x7f0c9814c850 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f0c9806fa90 tx=0x7f0c8801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607e2cd4756b1c80 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c95d9b640 1 -- 172.21.15.107:0/3436773718 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880a5650 con 0x7f0c9814c470 2023-10-25T00:20:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c95d9b640 1 -- 172.21.15.107:0/3436773718 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c880a57b0 con 0x7f0c9814c470 2023-10-25T00:20:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c95d9b640 1 -- 172.21.15.107:0/3436773718 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880a5b40 con 0x7f0c9814c470 2023-10-25T00:20:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c9d1af640 1 -- 172.21.15.107:0/3436773718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c9814c470 msgr2=0x7f0c9814c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:15.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c9d1af640 1 --2- 172.21.15.107:0/3436773718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c9814c470 0x7f0c9814c850 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f0c9806fa90 tx=0x7f0c8801ca10 comp rx=0 tx=0).stop 2023-10-25T00:20:15.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c9d1af640 1 -- 172.21.15.107:0/3436773718 shutdown_connections 2023-10-25T00:20:15.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c9d1af640 1 --2- 172.21.15.107:0/3436773718 >> v2:172.21.15.110:3300/0 conn(0x7f0c98152b30 0x7f0c98152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:15.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c9d1af640 1 --2- 172.21.15.107:0/3436773718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c9814c470 0x7f0c9814c850 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:15.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c9d1af640 1 -- 172.21.15.107:0/3436773718 >> 172.21.15.107:0/3436773718 conn(0x7f0c98070f60 msgr2=0x7f0c98071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:15.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c9d1af640 1 -- 172.21.15.107:0/3436773718 shutdown_connections 2023-10-25T00:20:15.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c9d1af640 1 -- 172.21.15.107:0/3436773718 wait complete. 2023-10-25T00:20:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c9d1af640 1 Processor -- start 2023-10-25T00:20:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.542+0000 7f0c9d1af640 1 -- start start 2023-10-25T00:20:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9d1af640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0c9806f490 0x7f0c981eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:15.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9d1af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c9814c470 0x7f0c9814d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9d1af640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c98152b30 0x7f0c980ab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9d1af640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c980bdaa0 con 0x7f0c9814c470 2023-10-25T00:20:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9d1af640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c980ac7d0 con 0x7f0c98152b30 2023-10-25T00:20:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9d1af640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c980acab0 con 0x7f0c9806f490 2023-10-25T00:20:15.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c9814c470 0x7f0c9814d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:15.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c9814c470 0x7f0c9814d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60466/0 (socket says 172.21.15.107:60466) 2023-10-25T00:20:15.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9659c640 1 -- 172.21.15.107:0/358055181 learned_addr learned my addr 172.21.15.107:0/358055181 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:15.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9659c640 1 -- 172.21.15.107:0/358055181 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0c9806f490 msgr2=0x7f0c981eb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:15.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9659c640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0c9806f490 0x7f0c981eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:15.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9659c640 1 -- 172.21.15.107:0/358055181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c98152b30 msgr2=0x7f0c980ab4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:15.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9659c640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c98152b30 0x7f0c980ab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9659c640 1 -- 172.21.15.107:0/358055181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c880094d0 con 0x7f0c9814c470 2023-10-25T00:20:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9659c640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c9814c470 0x7f0c9814d8a0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00c750 tx=0x7f0c8c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:15.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c7f7fe640 1 -- 172.21.15.107:0/358055181 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c005270 con 0x7f0c9814c470 2023-10-25T00:20:15.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c7f7fe640 1 -- 172.21.15.107:0/358055181 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c0053d0 con 0x7f0c9814c470 2023-10-25T00:20:15.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c7f7fe640 1 -- 172.21.15.107:0/358055181 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c003820 con 0x7f0c9814c470 2023-10-25T00:20:15.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9d1af640 1 -- 172.21.15.107:0/358055181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c980acd90 con 0x7f0c9814c470 2023-10-25T00:20:15.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9d1af640 1 -- 172.21.15.107:0/358055181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c980ad350 con 0x7f0c9814c470 2023-10-25T00:20:15.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c9d1af640 1 -- 172.21.15.107:0/358055181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c980ae030 con 0x7f0c9814c470 2023-10-25T00:20:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c7f7fe640 1 -- 172.21.15.107:0/358055181 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0c8c003980 con 0x7f0c9814c470 2023-10-25T00:20:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c7f7fe640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c84065e60 0x7f0c84068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.546+0000 7f0c7f7fe640 1 -- 172.21.15.107:0/358055181 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7f0c8c084000 con 0x7f0c9814c470 2023-10-25T00:20:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.550+0000 7f0c96d9d640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c84065e60 0x7f0c84068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.550+0000 7f0c96d9d640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c84065e60 0x7f0c84068320 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0c880ac000 tx=0x7f0c880046d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.554+0000 7f0c7f7fe640 1 -- 172.21.15.107:0/358055181 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0c8c050540 con 0x7f0c9814c470 2023-10-25T00:20:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.834+0000 7f0c9d1af640 1 -- 172.21.15.107:0/358055181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c980746e0 con 0x7f0c9814c470 2023-10-25T00:20:15.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.834+0000 7f0c7f7fe640 1 -- 172.21.15.107:0/358055181 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5772 (secure 0 0 0) 0x7f0c8c017020 con 0x7f0c9814c470 2023-10-25T00:20:15.838 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:15.838 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":215}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 215 2023-10-25T00:20:15.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.838+0000 7f0c9d1af640 1 -- 172.21.15.107:0/358055181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c84065e60 msgr2=0x7f0c84068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:15.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.838+0000 7f0c9d1af640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c84065e60 0x7f0c84068320 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f0c880ac000 tx=0x7f0c880046d0 comp rx=0 tx=0).stop 2023-10-25T00:20:15.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.838+0000 7f0c9d1af640 1 -- 172.21.15.107:0/358055181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c9814c470 msgr2=0x7f0c9814d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:15.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.838+0000 7f0c9d1af640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c9814c470 0x7f0c9814d8a0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00c750 tx=0x7f0c8c00cc10 comp rx=0 tx=0).stop 2023-10-25T00:20:15.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.838+0000 7f0c9d1af640 1 -- 172.21.15.107:0/358055181 shutdown_connections 2023-10-25T00:20:15.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.838+0000 7f0c9d1af640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0c98152b30 0x7f0c980ab4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:15.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.838+0000 7f0c9d1af640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c9814c470 0x7f0c9814d8a0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:15.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.838+0000 7f0c9d1af640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0c84065e60 0x7f0c84068320 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:15.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.838+0000 7f0c9d1af640 1 --2- 172.21.15.107:0/358055181 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0c9806f490 0x7f0c981eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:15.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.838+0000 7f0c9d1af640 1 -- 172.21.15.107:0/358055181 >> 172.21.15.107:0/358055181 conn(0x7f0c98070f60 msgr2=0x7f0c98152f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:15.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.838+0000 7f0c9d1af640 1 -- 172.21.15.107:0/358055181 shutdown_connections 2023-10-25T00:20:15.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:15.842+0000 7f0c9d1af640 1 -- 172.21.15.107:0/358055181 wait complete. 2023-10-25T00:20:15.864 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:20:17.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdd6fb7640 1 Processor -- start 2023-10-25T00:20:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdd6fb7640 1 -- start start 2023-10-25T00:20:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdd6fb7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbdd015eb00 0x7fbdd006b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdd6fb7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbdd00aa990 0x7fbdd0164a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdd6fb7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd01656c0 con 0x7fbdd008b9e0 2023-10-25T00:20:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdd6fb7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd015eee0 con 0x7fbdd00aa990 2023-10-25T00:20:18.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdd6fb7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd006c850 con 0x7fbdd015eb00 2023-10-25T00:20:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdcffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbdd00aa990 0x7fbdd0164a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdd5554640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbdd008b9e0 0x7fbdd0164fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37186/0 (socket says 172.21.15.107:37186) 2023-10-25T00:20:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdcffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbdd00aa990 0x7fbdd0164a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60474/0 (socket says 172.21.15.107:60474) 2023-10-25T00:20:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdd5554640 1 -- 172.21.15.107:0/114244625 learned_addr learned my addr 172.21.15.107:0/114244625 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:20:18.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdcf7fe640 1 -- 172.21.15.107:0/114244625 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647284982 0 0) 0x7fbdd01656c0 con 0x7fbdd008b9e0 2023-10-25T00:20:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdd4d53640 1 --2- 172.21.15.107:0/114244625 >> v2:172.21.15.110:3300/0 conn(0x7fbdd015eb00 0x7fbdd006b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdcf7fe640 1 -- 172.21.15.107:0/114244625 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc0003530 con 0x7fbdd008b9e0 2023-10-25T00:20:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdcffff640 1 -- 172.21.15.107:0/114244625 >> v2:172.21.15.110:3300/0 conn(0x7fbdd015eb00 msgr2=0x7fbdd006b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdcffff640 1 --2- 172.21.15.107:0/114244625 >> v2:172.21.15.110:3300/0 conn(0x7fbdd015eb00 0x7fbdd006b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:18.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdcffff640 1 -- 172.21.15.107:0/114244625 >> v1:172.21.15.107:6789/0 conn(0x7fbdd008b9e0 legacy=0x7fbdd0164fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:18.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdcffff640 1 -- 172.21.15.107:0/114244625 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd0161560 con 0x7fbdd00aa990 2023-10-25T00:20:18.017 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.010+0000 7fbdd4d53640 1 --2- 172.21.15.107:0/114244625 >> v2:172.21.15.110:3300/0 conn(0x7fbdd015eb00 0x7fbdd006b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:20:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.014+0000 7fbdcffff640 1 --2- 172.21.15.107:0/114244625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd00aa990 0x7fbdd0164a70 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fbdd00697c0 tx=0x7fbdbc01e610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4247526f27cdebab server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.014+0000 7fbdcf7fe640 1 -- 172.21.15.107:0/114244625 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc0a86f0 con 0x7fbdd00aa990 2023-10-25T00:20:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.014+0000 7fbdcf7fe640 1 -- 172.21.15.107:0/114244625 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdbc0a8850 con 0x7fbdd00aa990 2023-10-25T00:20:18.018 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.014+0000 7fbdcf7fe640 1 -- 172.21.15.107:0/114244625 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc0a8be0 con 0x7fbdd00aa990 2023-10-25T00:20:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.014+0000 7fbdd6fb7640 1 -- 172.21.15.107:0/114244625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd00aa990 msgr2=0x7fbdd0164a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:18.019 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.014+0000 7fbdd6fb7640 1 --2- 172.21.15.107:0/114244625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd00aa990 0x7fbdd0164a70 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fbdd00697c0 tx=0x7fbdbc01e610 comp rx=0 tx=0).stop 2023-10-25T00:20:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 -- 172.21.15.107:0/114244625 shutdown_connections 2023-10-25T00:20:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 --2- 172.21.15.107:0/114244625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd00aa990 0x7fbdd0164a70 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:18.020 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 --2- 172.21.15.107:0/114244625 >> v2:172.21.15.110:3300/0 conn(0x7fbdd015eb00 0x7fbdd006b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 -- 172.21.15.107:0/114244625 >> 172.21.15.107:0/114244625 conn(0x7fbdd00c2950 msgr2=0x7fbdd00c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 -- 172.21.15.107:0/114244625 shutdown_connections 2023-10-25T00:20:18.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 -- 172.21.15.107:0/114244625 wait complete. 2023-10-25T00:20:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 Processor -- start 2023-10-25T00:20:18.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 -- start start 2023-10-25T00:20:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbdd008b9e0 0x7fbdd00c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd00aa990 0x7fbdd006f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdd015eb00 0x7fbdd01fda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00c52e0 con 0x7fbdd00aa990 2023-10-25T00:20:18.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0072260 con 0x7fbdd015eb00 2023-10-25T00:20:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd6fb7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0071ca0 con 0x7fbdd008b9e0 2023-10-25T00:20:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd5554640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdd015eb00 0x7fbdd01fda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd5554640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdd015eb00 0x7fbdd01fda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:39578/0 (socket says 172.21.15.107:39578) 2023-10-25T00:20:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdd5554640 1 -- 172.21.15.107:0/3039461992 learned_addr learned my addr 172.21.15.107:0/3039461992 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:18.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.018+0000 7fbdcffff640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd00aa990 0x7fbdd006f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.022+0000 7fbdd5554640 1 -- 172.21.15.107:0/3039461992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbdd008b9e0 msgr2=0x7fbdd00c4da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:20:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.022+0000 7fbdd5554640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbdd008b9e0 0x7fbdd00c4da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:18.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.022+0000 7fbdd5554640 1 -- 172.21.15.107:0/3039461992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd00aa990 msgr2=0x7fbdd006f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.022+0000 7fbdd5554640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd00aa990 0x7fbdd006f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.022+0000 7fbdd5554640 1 -- 172.21.15.107:0/3039461992 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc00b840 con 0x7fbdd015eb00 2023-10-25T00:20:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.022+0000 7fbdd5554640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdd015eb00 0x7fbdd01fda20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c7a0 tx=0x7fbdc800cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:18.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.034+0000 7fbdcd7fa640 1 -- 172.21.15.107:0/3039461992 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc800a840 con 0x7fbdd015eb00 2023-10-25T00:20:18.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.034+0000 7fbdcd7fa640 1 -- 172.21.15.107:0/3039461992 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc800a9a0 con 0x7fbdd015eb00 2023-10-25T00:20:18.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.034+0000 7fbdcd7fa640 1 -- 172.21.15.107:0/3039461992 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc800ad70 con 0x7fbdd015eb00 2023-10-25T00:20:18.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.034+0000 7fbdd6fb7640 1 -- 172.21.15.107:0/3039461992 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd0071f80 con 0x7fbdd015eb00 2023-10-25T00:20:18.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.034+0000 7fbdd6fb7640 1 -- 172.21.15.107:0/3039461992 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd006ff60 con 0x7fbdd015eb00 2023-10-25T00:20:18.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.042+0000 7fbdcd7fa640 1 -- 172.21.15.107:0/3039461992 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbdc8012400 con 0x7fbdd015eb00 2023-10-25T00:20:18.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.046+0000 7fbdcd7fa640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdb4065e90 0x7fbdb4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:18.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.046+0000 7fbdbaffd640 1 -- 172.21.15.107:0/3039461992 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd0070150 con 0x7fbdd015eb00 2023-10-25T00:20:18.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.046+0000 7fbdcd7fa640 1 -- 172.21.15.107:0/3039461992 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 10487+0+0 (secure 0 0 0) 0x7fbdc80841b0 con 0x7fbdd015eb00 2023-10-25T00:20:18.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.046+0000 7fbdd4d53640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdb4065e90 0x7fbdb4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:18.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.046+0000 7fbdd4d53640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdb4065e90 0x7fbdb4068350 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400a830 tx=0x7fbdc4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:18.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.078+0000 7fbdcd7fa640 1 -- 172.21.15.107:0/3039461992 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbdc8017080 con 0x7fbdd015eb00 2023-10-25T00:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.358+0000 7fbdbaffd640 1 -- 172.21.15.107:0/3039461992 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdd00746e0 con 0x7fbdd015eb00 2023-10-25T00:20:18.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.358+0000 7fbdcd7fa640 1 -- 172.21.15.107:0/3039461992 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5772 (secure 0 0 0) 0x7fbdc80506b0 con 0x7fbdd015eb00 2023-10-25T00:20:18.362 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:18.362 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":8802,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":215}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:18:53.158376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7258},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7258":{"gid":7258,"name":"b","rank":0,"incarnation":201,"state":"up:active","state_seq":18,"addr":"172.21.15.110:6834/3771564389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":3771564389},{"type":"v1","addr":"172.21.15.110:6834","nonce":3771564389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:18.363 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 215 2023-10-25T00:20:18.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 -- 172.21.15.107:0/3039461992 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdb4065e90 msgr2=0x7fbdb4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:18.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdb4065e90 0x7fbdb4068350 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fbdc400a830 tx=0x7fbdc4008040 comp rx=0 tx=0).stop 2023-10-25T00:20:18.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 -- 172.21.15.107:0/3039461992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdd015eb00 msgr2=0x7fbdd01fda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:18.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdd015eb00 0x7fbdd01fda20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c7a0 tx=0x7fbdc800cc60 comp rx=0 tx=0).stop 2023-10-25T00:20:18.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 -- 172.21.15.107:0/3039461992 shutdown_connections 2023-10-25T00:20:18.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbdd015eb00 0x7fbdd01fda20 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:18.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdd00aa990 0x7fbdd006f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:18.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbdb4065e90 0x7fbdb4068350 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:18.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 --2- 172.21.15.107:0/3039461992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbdd008b9e0 0x7fbdd00c4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:18.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 -- 172.21.15.107:0/3039461992 >> 172.21.15.107:0/3039461992 conn(0x7fbdd00c2950 msgr2=0x7fbdd0165130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:18.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 -- 172.21.15.107:0/3039461992 shutdown_connections 2023-10-25T00:20:18.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:18.362+0000 7fbdbaffd640 1 -- 172.21.15.107:0/3039461992 wait complete. 2023-10-25T00:20:18.380 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2023-10-25T00:20:20.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:20.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e2e1c2640 1 Processor -- start 2023-10-25T00:20:20.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e2e1c2640 1 -- start start 2023-10-25T00:20:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e2e1c2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3e2814fc30 0x7f3e28153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e2e1c2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 0x7f3e2806df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:20.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e2e1c2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e2806e470 0x7f3e280c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:20.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e2e1c2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e28072330 con 0x7f3e2814fc30 2023-10-25T00:20:20.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e2e1c2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e28155ca0 con 0x7f3e2806e470 2023-10-25T00:20:20.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e2e1c2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e28155f80 con 0x7f3e280c5c20 2023-10-25T00:20:20.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e26ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 0x7f3e2806df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:20.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e27fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e2806e470 0x7f3e280c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:20.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e26ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 0x7f3e2806df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:38466/0 (socket says 172.21.15.107:38466) 2023-10-25T00:20:20.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.518+0000 7f3e26ffd640 1 -- 172.21.15.107:0/1480169134 learned_addr learned my addr 172.21.15.107:0/1480169134 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:20.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.522+0000 7f3e26ffd640 1 -- 172.21.15.107:0/1480169134 >> v2:172.21.15.110:3300/0 conn(0x7f3e2806e470 msgr2=0x7f3e280c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:20.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.522+0000 7f3e26ffd640 1 --2- 172.21.15.107:0/1480169134 >> v2:172.21.15.110:3300/0 conn(0x7f3e2806e470 0x7f3e280c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.522+0000 7f3e26ffd640 1 -- 172.21.15.107:0/1480169134 >> v2:172.21.15.107:3300/0 conn(0x7f3e2814fc30 msgr2=0x7f3e28153190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:20:20.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.522+0000 7f3e26ffd640 1 --2- 172.21.15.107:0/1480169134 >> v2:172.21.15.107:3300/0 conn(0x7f3e2814fc30 0x7f3e28153190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.522+0000 7f3e26ffd640 1 -- 172.21.15.107:0/1480169134 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e28156260 con 0x7f3e280c5c20 2023-10-25T00:20:20.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.522+0000 7f3e26ffd640 1 --2- 172.21.15.107:0/1480169134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 0x7f3e2806df30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3e280a94e0 tx=0x7f3e1c02d3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d088758f9b9dd05f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:20.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.522+0000 7f3e267fc640 1 -- 172.21.15.107:0/1480169134 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0bd410 con 0x7f3e280c5c20 2023-10-25T00:20:20.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.522+0000 7f3e267fc640 1 -- 172.21.15.107:0/1480169134 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c0bd570 con 0x7f3e280c5c20 2023-10-25T00:20:20.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.522+0000 7f3e267fc640 1 -- 172.21.15.107:0/1480169134 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0bd900 con 0x7f3e280c5c20 2023-10-25T00:20:20.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/1480169134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 msgr2=0x7f3e2806df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:20.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 --2- 172.21.15.107:0/1480169134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 0x7f3e2806df30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3e280a94e0 tx=0x7f3e1c02d3d0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/1480169134 shutdown_connections 2023-10-25T00:20:20.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 --2- 172.21.15.107:0/1480169134 >> v2:172.21.15.110:3300/0 conn(0x7f3e2806e470 0x7f3e280c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 --2- 172.21.15.107:0/1480169134 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 0x7f3e2806df30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 --2- 172.21.15.107:0/1480169134 >> v2:172.21.15.107:3300/0 conn(0x7f3e2814fc30 0x7f3e28153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/1480169134 >> 172.21.15.107:0/1480169134 conn(0x7f3e280c39a0 msgr2=0x7f3e280c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:20.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/1480169134 shutdown_connections 2023-10-25T00:20:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/1480169134 wait complete. 2023-10-25T00:20:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 Processor -- start 2023-10-25T00:20:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 -- start start 2023-10-25T00:20:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2806e470 0x7f3e28150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:20.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 0x7f3e2813ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e2814fc30 0x7f3e281ebe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2813a9f0 con 0x7f3e2806e470 2023-10-25T00:20:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e280bd7d0 con 0x7f3e2814fc30 2023-10-25T00:20:20.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e28154230 con 0x7f3e280c5c20 2023-10-25T00:20:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2806e470 0x7f3e28150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2806e470 0x7f3e28150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60486/0 (socket says 172.21.15.107:60486) 2023-10-25T00:20:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e277fe640 1 -- 172.21.15.107:0/3216863577 learned_addr learned my addr 172.21.15.107:0/3216863577 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:20.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e26ffd640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 0x7f3e2813ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e27fff640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e2814fc30 0x7f3e281ebe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e277fe640 1 -- 172.21.15.107:0/3216863577 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 msgr2=0x7f3e2813ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e277fe640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 0x7f3e2813ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e277fe640 1 -- 172.21.15.107:0/3216863577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e2814fc30 msgr2=0x7f3e281ebe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e277fe640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e2814fc30 0x7f3e281ebe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e277fe640 1 -- 172.21.15.107:0/3216863577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1c0094d0 con 0x7f3e2806e470 2023-10-25T00:20:20.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e26ffd640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 0x7f3e2813ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:20:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e277fe640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2806e470 0x7f3e28150b20 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800b3f0 tx=0x7f3e1800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e0ffff640 1 -- 172.21.15.107:0/3216863577 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18011830 con 0x7f3e2806e470 2023-10-25T00:20:20.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e0ffff640 1 -- 172.21.15.107:0/3216863577 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e180119b0 con 0x7f3e2806e470 2023-10-25T00:20:20.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/3216863577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e28152c60 con 0x7f3e2806e470 2023-10-25T00:20:20.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e0ffff640 1 -- 172.21.15.107:0/3216863577 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18011d60 con 0x7f3e2806e470 2023-10-25T00:20:20.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.526+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/3216863577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e281528f0 con 0x7f3e2806e470 2023-10-25T00:20:20.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.534+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/3216863577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e280746e0 con 0x7f3e2806e470 2023-10-25T00:20:20.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.538+0000 7f3e0ffff640 1 -- 172.21.15.107:0/3216863577 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3e1801a400 con 0x7f3e2806e470 2023-10-25T00:20:20.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.538+0000 7f3e0ffff640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e14065e90 0x7f3e14068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:20.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.538+0000 7f3e26ffd640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e14065e90 0x7f3e14068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:20.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.538+0000 7f3e26ffd640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e14065e90 0x7f3e14068350 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c0bf000 tx=0x7f3e1c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:20.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.538+0000 7f3e0ffff640 1 -- 172.21.15.107:0/3216863577 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7f3e18085040 con 0x7f3e2806e470 2023-10-25T00:20:20.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.550+0000 7f3e0ffff640 1 -- 172.21.15.107:0/3216863577 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3e18051570 con 0x7f3e2806e470 2023-10-25T00:20:20.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.822+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/3216863577 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e2814f440 con 0x7f3e2806e470 2023-10-25T00:20:20.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.822+0000 7f3e0ffff640 1 -- 172.21.15.107:0/3216863577 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) v1 ==== 95+0+5010 (secure 0 0 0) 0x7f3e18015090 con 0x7f3e2806e470 2023-10-25T00:20:20.825 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:20.826 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:19.149797+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:20.826 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 216 2023-10-25T00:20:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/3216863577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e14065e90 msgr2=0x7f3e14068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e14065e90 0x7f3e14068350 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c0bf000 tx=0x7f3e1c009040 comp rx=0 tx=0).stop 2023-10-25T00:20:20.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/3216863577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2806e470 msgr2=0x7f3e28150b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:20.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2806e470 0x7f3e28150b20 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800b3f0 tx=0x7f3e1800b8b0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/3216863577 shutdown_connections 2023-10-25T00:20:20.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e2814fc30 0x7f3e281ebe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3e14065e90 0x7f3e14068350 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e280c5c20 0x7f3e2813ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 --2- 172.21.15.107:0/3216863577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e2806e470 0x7f3e28150b20 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/3216863577 >> 172.21.15.107:0/3216863577 conn(0x7f3e280c39a0 msgr2=0x7f3e280c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:20.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/3216863577 shutdown_connections 2023-10-25T00:20:20.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.826+0000 7f3e2e1c2640 1 -- 172.21.15.107:0/3216863577 wait complete. 2023-10-25T00:20:20.856 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2023-10-25T00:20:20.856 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:20:20.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:20.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.990+0000 7fe55d202640 1 Processor -- start 2023-10-25T00:20:20.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.990+0000 7fe55d202640 1 -- start start 2023-10-25T00:20:20.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe55d202640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe558069d40 0x7fe558158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:20.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe55d202640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe55806a120 0x7fe5580aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:20.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe55d202640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55806cea0 con 0x7fe558065bc0 2023-10-25T00:20:20.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe55659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe558065bc0 0x7fe5580a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37198/0 (socket says 172.21.15.107:37198) 2023-10-25T00:20:20.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe55d202640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5580ac650 con 0x7fe55806a120 2023-10-25T00:20:20.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe55d202640 1 -- 172.21.15.107:0/776590588 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5580a9e00 con 0x7fe558069d40 2023-10-25T00:20:20.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe55659c640 1 -- 172.21.15.107:0/776590588 learned_addr learned my addr 172.21.15.107:0/776590588 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:20:20.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe556d9d640 1 --2- 172.21.15.107:0/776590588 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe558069d40 0x7fe558158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:20.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe555d9b640 1 -- 172.21.15.107:0/776590588 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902547378 0 0) 0x7fe55806cea0 con 0x7fe558065bc0 2023-10-25T00:20:20.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe555d9b640 1 -- 172.21.15.107:0/776590588 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe540003530 con 0x7fe558065bc0 2023-10-25T00:20:20.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe556d9d640 1 -- 172.21.15.107:0/776590588 >> v2:172.21.15.110:3300/0 conn(0x7fe55806a120 msgr2=0x7fe5580aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:20.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe556d9d640 1 --2- 172.21.15.107:0/776590588 >> v2:172.21.15.110:3300/0 conn(0x7fe55806a120 0x7fe5580aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:20.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe556d9d640 1 -- 172.21.15.107:0/776590588 >> v1:172.21.15.107:6789/0 conn(0x7fe558065bc0 legacy=0x7fe5580a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:20.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.994+0000 7fe556d9d640 1 -- 172.21.15.107:0/776590588 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5581584e0 con 0x7fe558069d40 2023-10-25T00:20:21.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe556d9d640 1 --2- 172.21.15.107:0/776590588 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe558069d40 0x7fe558158970 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe5480097c0 tx=0x7fe54801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df8affbbbf0e9622 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:21.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe555d9b640 1 -- 172.21.15.107:0/776590588 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5480a4ea0 con 0x7fe558069d40 2023-10-25T00:20:21.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe555d9b640 1 -- 172.21.15.107:0/776590588 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5480a7050 con 0x7fe558069d40 2023-10-25T00:20:21.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe555d9b640 1 -- 172.21.15.107:0/776590588 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5480ab860 con 0x7fe558069d40 2023-10-25T00:20:21.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe55d202640 1 -- 172.21.15.107:0/776590588 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe558069d40 msgr2=0x7fe558158970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:21.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe55d202640 1 --2- 172.21.15.107:0/776590588 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe558069d40 0x7fe558158970 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe5480097c0 tx=0x7fe54801c410 comp rx=0 tx=0).stop 2023-10-25T00:20:21.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe55d202640 1 -- 172.21.15.107:0/776590588 shutdown_connections 2023-10-25T00:20:21.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe55d202640 1 --2- 172.21.15.107:0/776590588 >> v2:172.21.15.110:3300/0 conn(0x7fe55806a120 0x7fe5580aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:21.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe55d202640 1 --2- 172.21.15.107:0/776590588 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe558069d40 0x7fe558158970 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:21.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe55d202640 1 -- 172.21.15.107:0/776590588 >> 172.21.15.107:0/776590588 conn(0x7fe5580c65c0 msgr2=0x7fe5580c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:21.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe55d202640 1 -- 172.21.15.107:0/776590588 shutdown_connections 2023-10-25T00:20:21.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe55d202640 1 -- 172.21.15.107:0/776590588 wait complete. 2023-10-25T00:20:21.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe55d202640 1 Processor -- start 2023-10-25T00:20:21.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:20.998+0000 7fe55d202640 1 -- start start 2023-10-25T00:20:21.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe55d202640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe558065bc0 0x7fe558154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:21.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe55d202640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe558069d40 0x7fe5581f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:21.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe55d202640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe55806a120 0x7fe5581f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:21.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe55d202640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5580bda40 con 0x7fe558065bc0 2023-10-25T00:20:21.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe55d202640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5581f5a10 con 0x7fe55806a120 2023-10-25T00:20:21.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe55d202640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5581f3b10 con 0x7fe558069d40 2023-10-25T00:20:21.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe556d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe558065bc0 0x7fe558154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:21.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe556d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe558065bc0 0x7fe558154c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60496/0 (socket says 172.21.15.107:60496) 2023-10-25T00:20:21.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe556d9d640 1 -- 172.21.15.107:0/222302398 learned_addr learned my addr 172.21.15.107:0/222302398 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:21.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe55659c640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe558069d40 0x7fe5581f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:21.008 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe55759e640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe55806a120 0x7fe5581f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:21.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe556d9d640 1 -- 172.21.15.107:0/222302398 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe558069d40 msgr2=0x7fe5581f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:21.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe556d9d640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe558069d40 0x7fe5581f3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:21.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe556d9d640 1 -- 172.21.15.107:0/222302398 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe55806a120 msgr2=0x7fe5581f44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:21.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe556d9d640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe55806a120 0x7fe5581f44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:21.009 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe556d9d640 1 -- 172.21.15.107:0/222302398 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe548009420 con 0x7fe558065bc0 2023-10-25T00:20:21.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe556d9d640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe558065bc0 0x7fe558154c70 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fe54801a040 tx=0x7fe54801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:21.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe53f7fe640 1 -- 172.21.15.107:0/222302398 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5480a4ea0 con 0x7fe558065bc0 2023-10-25T00:20:21.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe53f7fe640 1 -- 172.21.15.107:0/222302398 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5480b7450 con 0x7fe558065bc0 2023-10-25T00:20:21.010 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe55d202640 1 -- 172.21.15.107:0/222302398 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5581f2f00 con 0x7fe558065bc0 2023-10-25T00:20:21.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe53f7fe640 1 -- 172.21.15.107:0/222302398 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5480b7800 con 0x7fe558065bc0 2023-10-25T00:20:21.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe55d202640 1 -- 172.21.15.107:0/222302398 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5581f34c0 con 0x7fe558065bc0 2023-10-25T00:20:21.011 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe55d202640 1 -- 172.21.15.107:0/222302398 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe558158cc0 con 0x7fe558065bc0 2023-10-25T00:20:21.014 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe53f7fe640 1 -- 172.21.15.107:0/222302398 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe5480b7960 con 0x7fe558065bc0 2023-10-25T00:20:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.002+0000 7fe53f7fe640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe544065e60 0x7fe544068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.006+0000 7fe53f7fe640 1 -- 172.21.15.107:0/222302398 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7fe54811eef0 con 0x7fe558065bc0 2023-10-25T00:20:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.006+0000 7fe55659c640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe544065e60 0x7fe544068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.006+0000 7fe55659c640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe544065e60 0x7fe544068320 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe5580be2e0 tx=0x7fe54c007d40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:21.016 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.010+0000 7fe53f7fe640 1 -- 172.21.15.107:0/222302398 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe5480eb430 con 0x7fe558065bc0 2023-10-25T00:20:21.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.282+0000 7fe55d202640 1 -- 172.21.15.107:0/222302398 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5580746e0 con 0x7fe558065bc0 2023-10-25T00:20:21.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.282+0000 7fe53f7fe640 1 -- 172.21.15.107:0/222302398 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+5013 (secure 0 0 0) 0x7fe5480eea50 con 0x7fe558065bc0 2023-10-25T00:20:21.287 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:21.287 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:21.051579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:21.287 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 217 2023-10-25T00:20:21.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.286+0000 7fe55d202640 1 -- 172.21.15.107:0/222302398 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe544065e60 msgr2=0x7fe544068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:21.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.286+0000 7fe55d202640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe544065e60 0x7fe544068320 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe5580be2e0 tx=0x7fe54c007d40 comp rx=0 tx=0).stop 2023-10-25T00:20:21.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.286+0000 7fe55d202640 1 -- 172.21.15.107:0/222302398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe558065bc0 msgr2=0x7fe558154c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:21.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.286+0000 7fe55d202640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe558065bc0 0x7fe558154c70 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fe54801a040 tx=0x7fe54801ac50 comp rx=0 tx=0).stop 2023-10-25T00:20:21.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.286+0000 7fe55d202640 1 -- 172.21.15.107:0/222302398 shutdown_connections 2023-10-25T00:20:21.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.290+0000 7fe55d202640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe55806a120 0x7fe5581f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:21.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.290+0000 7fe55d202640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe544065e60 0x7fe544068320 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:21.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.290+0000 7fe55d202640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe558069d40 0x7fe5581f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:21.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.290+0000 7fe55d202640 1 --2- 172.21.15.107:0/222302398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe558065bc0 0x7fe558154c70 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:21.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.290+0000 7fe55d202640 1 -- 172.21.15.107:0/222302398 >> 172.21.15.107:0/222302398 conn(0x7fe5580c65c0 msgr2=0x7fe558070160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:21.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.290+0000 7fe55d202640 1 -- 172.21.15.107:0/222302398 shutdown_connections 2023-10-25T00:20:21.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:21.290+0000 7fe55d202640 1 -- 172.21.15.107:0/222302398 wait complete. 2023-10-25T00:20:21.318 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 217, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 217, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8802": { "addr": "172.21.15.107:6835/2187059523", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2187059523, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 2187059523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8802, "incarnation": 216, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:20:21.051579+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8802 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 214, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8712": { "addr": "172.21.15.107:6837/657626630", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 657626630, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 657626630, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8712, "incarnation": 211, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:19:47.167046+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8712 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2437148", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2437148, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2437148, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 190, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6443, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:20:23.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:23.468 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.466+0000 7fee64877640 1 Processor -- start 2023-10-25T00:20:23.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.466+0000 7fee64877640 1 -- start start 2023-10-25T00:20:23.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.466+0000 7fee64877640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee600c70e0 0x7fee600bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:23.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.466+0000 7fee64877640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee600c2130 0x7fee600be4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:23.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.466+0000 7fee64877640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee600c1ca0 con 0x7fee60156770 2023-10-25T00:20:23.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.466+0000 7fee64877640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee600c27e0 con 0x7fee600c2130 2023-10-25T00:20:23.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.466+0000 7fee64877640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee600c10c0 con 0x7fee600c70e0 2023-10-25T00:20:23.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.470+0000 7fee5ed9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee600c2130 0x7fee600be4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:23.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.470+0000 7fee5ed9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fee600c2130 0x7fee600be4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34952/0 (socket says 172.21.15.107:34952) 2023-10-25T00:20:23.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.470+0000 7fee5ed9d640 1 -- 172.21.15.107:0/2724372607 learned_addr learned my addr 172.21.15.107:0/2724372607 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:23.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.470+0000 7fee5ed9d640 1 -- 172.21.15.107:0/2724372607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee600c70e0 msgr2=0x7fee600bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:23.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.470+0000 7fee5ed9d640 1 --2- 172.21.15.107:0/2724372607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee600c70e0 0x7fee600bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:23.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.470+0000 7fee5ed9d640 1 -- 172.21.15.107:0/2724372607 >> v1:172.21.15.110:6789/0 conn(0x7fee60156770 legacy=0x7fee600ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:23.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.470+0000 7fee5ed9d640 1 -- 172.21.15.107:0/2724372607 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee600c13a0 con 0x7fee600c2130 2023-10-25T00:20:23.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.474+0000 7fee5ed9d640 1 --2- 172.21.15.107:0/2724372607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee600c2130 0x7fee600be4e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fee58009e50 tx=0x7fee5801c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31b5f4037a7709cd server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:23.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.474+0000 7fee5d59a640 1 -- 172.21.15.107:0/2724372607 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee580a5530 con 0x7fee600c2130 2023-10-25T00:20:23.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.474+0000 7fee5d59a640 1 -- 172.21.15.107:0/2724372607 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee580a5690 con 0x7fee600c2130 2023-10-25T00:20:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.474+0000 7fee5d59a640 1 -- 172.21.15.107:0/2724372607 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee580a5a20 con 0x7fee600c2130 2023-10-25T00:20:23.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.474+0000 7fee64877640 1 -- 172.21.15.107:0/2724372607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee600c2130 msgr2=0x7fee600be4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.474+0000 7fee64877640 1 --2- 172.21.15.107:0/2724372607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee600c2130 0x7fee600be4e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fee58009e50 tx=0x7fee5801c8b0 comp rx=0 tx=0).stop 2023-10-25T00:20:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.478+0000 7fee64877640 1 -- 172.21.15.107:0/2724372607 shutdown_connections 2023-10-25T00:20:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.478+0000 7fee64877640 1 --2- 172.21.15.107:0/2724372607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee600c2130 0x7fee600be4e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.478+0000 7fee64877640 1 --2- 172.21.15.107:0/2724372607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee600c70e0 0x7fee600bdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:23.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.478+0000 7fee64877640 1 -- 172.21.15.107:0/2724372607 >> 172.21.15.107:0/2724372607 conn(0x7fee600c65c0 msgr2=0x7fee600c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.478+0000 7fee64877640 1 -- 172.21.15.107:0/2724372607 shutdown_connections 2023-10-25T00:20:23.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.478+0000 7fee64877640 1 -- 172.21.15.107:0/2724372607 wait complete. 2023-10-25T00:20:23.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.478+0000 7fee64877640 1 Processor -- start 2023-10-25T00:20:23.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.478+0000 7fee64877640 1 -- start start 2023-10-25T00:20:23.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee64877640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee600c2130 0x7fee6014ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:23.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee64877640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee600c70e0 0x7fee600bbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee64877640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee60156770 0x7fee601526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee64877640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee60152290 con 0x7fee60156770 2023-10-25T00:20:23.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee64877640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee600bcf10 con 0x7fee600c2130 2023-10-25T00:20:23.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee64877640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee601ed6b0 con 0x7fee600c70e0 2023-10-25T00:20:23.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5dd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee600c70e0 0x7fee600bbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:23.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee600c2130 0x7fee6014ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee60156770 0x7fee601526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5e59c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee600c2130 0x7fee6014ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34954/0 (socket says 172.21.15.107:34954) 2023-10-25T00:20:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5dd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee600c70e0 0x7fee600bbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:38320/0 (socket says 172.21.15.107:38320) 2023-10-25T00:20:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5e59c640 1 -- 172.21.15.107:0/2485664660 learned_addr learned my addr 172.21.15.107:0/2485664660 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:23.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee60156770 0x7fee601526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50012/0 (socket says 172.21.15.107:50012) 2023-10-25T00:20:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5ed9d640 1 -- 172.21.15.107:0/2485664660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee600c70e0 msgr2=0x7fee600bbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5ed9d640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee600c70e0 0x7fee600bbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5ed9d640 1 -- 172.21.15.107:0/2485664660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee600c2130 msgr2=0x7fee6014ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5ed9d640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee600c2130 0x7fee6014ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5ed9d640 1 -- 172.21.15.107:0/2485664660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee58009b30 con 0x7fee60156770 2023-10-25T00:20:23.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee5ed9d640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee60156770 0x7fee601526b0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fee580ab000 tx=0x7fee58019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:23.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee4effd640 1 -- 172.21.15.107:0/2485664660 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58009510 con 0x7fee60156770 2023-10-25T00:20:23.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee4effd640 1 -- 172.21.15.107:0/2485664660 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee580b8470 con 0x7fee60156770 2023-10-25T00:20:23.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee64877640 1 -- 172.21.15.107:0/2485664660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee601ed990 con 0x7fee60156770 2023-10-25T00:20:23.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee4effd640 1 -- 172.21.15.107:0/2485664660 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee580b8800 con 0x7fee60156770 2023-10-25T00:20:23.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.482+0000 7fee64877640 1 -- 172.21.15.107:0/2485664660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee600c5410 con 0x7fee60156770 2023-10-25T00:20:23.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.486+0000 7fee64877640 1 -- 172.21.15.107:0/2485664660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee601523d0 con 0x7fee60156770 2023-10-25T00:20:23.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.490+0000 7fee4effd640 1 -- 172.21.15.107:0/2485664660 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fee580b8960 con 0x7fee60156770 2023-10-25T00:20:23.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.490+0000 7fee4effd640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee40065e60 0x7fee40068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.490+0000 7fee4effd640 1 -- 172.21.15.107:0/2485664660 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7fee5811ff80 con 0x7fee60156770 2023-10-25T00:20:23.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.490+0000 7fee5e59c640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee40065e60 0x7fee40068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:23.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.494+0000 7fee4effd640 1 -- 172.21.15.107:0/2485664660 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fee580ec470 con 0x7fee60156770 2023-10-25T00:20:23.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.494+0000 7fee5e59c640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee40065e60 0x7fee40068320 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fee50001780 tx=0x7fee50009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:23.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.746+0000 7fee64877640 1 -- 172.21.15.107:0/2485664660 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee601523d0 con 0x7fee60156770 2023-10-25T00:20:23.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.746+0000 7fee4effd640 1 -- 172.21.15.107:0/2485664660 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+5010 (secure 0 0 0) 0x7fee601523d0 con 0x7fee60156770 2023-10-25T00:20:23.751 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:23.751 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:23.751 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 219 2023-10-25T00:20:23.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 -- 172.21.15.107:0/2485664660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee40065e60 msgr2=0x7fee40068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:23.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee40065e60 0x7fee40068320 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fee50001780 tx=0x7fee50009040 comp rx=0 tx=0).stop 2023-10-25T00:20:23.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 -- 172.21.15.107:0/2485664660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee60156770 msgr2=0x7fee601526b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:23.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee60156770 0x7fee601526b0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fee580ab000 tx=0x7fee58019650 comp rx=0 tx=0).stop 2023-10-25T00:20:23.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 -- 172.21.15.107:0/2485664660 shutdown_connections 2023-10-25T00:20:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee60156770 0x7fee601526b0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fee40065e60 0x7fee40068320 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fee600c70e0 0x7fee600bbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 --2- 172.21.15.107:0/2485664660 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fee600c2130 0x7fee6014ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:23.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 -- 172.21.15.107:0/2485664660 >> 172.21.15.107:0/2485664660 conn(0x7fee600c65c0 msgr2=0x7fee60155630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 -- 172.21.15.107:0/2485664660 shutdown_connections 2023-10-25T00:20:23.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:23.750+0000 7fee64877640 1 -- 172.21.15.107:0/2485664660 wait complete. 2023-10-25T00:20:23.774 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=7258 2023-10-25T00:20:23.774 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.b 2023-10-25T00:20:24.775 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2023-10-25T00:20:24.776 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-25T00:20:24.776 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-25T00:20:24.778 INFO:tasks.ceph.mds.b:Started 2023-10-25T00:20:24.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:24.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f8073c4a640 1 Processor -- start 2023-10-25T00:20:24.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f8073c4a640 1 -- start start 2023-10-25T00:20:24.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f8073c4a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f806c152040 0x7f806c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:24.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f8073c4a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f806c0c8100 0x7f806c0abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:24.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f8073c4a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f806c06a4b0 0x7f806c0ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:24.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f8073c4a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806c155680 con 0x7f806c0c8100 2023-10-25T00:20:24.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f8073c4a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806c154a70 con 0x7f806c152040 2023-10-25T00:20:24.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f8073c4a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806c154d50 con 0x7f806c06a4b0 2023-10-25T00:20:24.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f80711e5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f806c0c8100 0x7f806c0abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f80711e5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f806c0c8100 0x7f806c0abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50028/0 (socket says 172.21.15.107:50028) 2023-10-25T00:20:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f80711e5640 1 -- 172.21.15.107:0/3813301901 learned_addr learned my addr 172.21.15.107:0/3813301901 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f80721e7640 1 --2- 172.21.15.107:0/3813301901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f806c06a4b0 0x7f806c0ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:24.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f80711e5640 1 -- 172.21.15.107:0/3813301901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f806c06a4b0 msgr2=0x7f806c0ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f80711e5640 1 --2- 172.21.15.107:0/3813301901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f806c06a4b0 0x7f806c0ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f80711e5640 1 -- 172.21.15.107:0/3813301901 >> v2:172.21.15.110:3300/0 conn(0x7f806c152040 msgr2=0x7f806c0c4b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:20:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f80711e5640 1 --2- 172.21.15.107:0/3813301901 >> v2:172.21.15.110:3300/0 conn(0x7f806c152040 0x7f806c0c4b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:24.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.902+0000 7f80711e5640 1 -- 172.21.15.107:0/3813301901 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806c155030 con 0x7f806c0c8100 2023-10-25T00:20:24.910 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f80711e5640 1 --2- 172.21.15.107:0/3813301901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c0c8100 0x7f806c0abd00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f806c067ba0 tx=0x7f805c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e89264a62787cc server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f80709e4640 1 -- 172.21.15.107:0/3813301901 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c0ad660 con 0x7f806c0c8100 2023-10-25T00:20:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f80709e4640 1 -- 172.21.15.107:0/3813301901 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c0ad7c0 con 0x7f806c0c8100 2023-10-25T00:20:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f80709e4640 1 -- 172.21.15.107:0/3813301901 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c0adb50 con 0x7f806c0c8100 2023-10-25T00:20:24.911 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f8073c4a640 1 -- 172.21.15.107:0/3813301901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c0c8100 msgr2=0x7f806c0abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f8073c4a640 1 --2- 172.21.15.107:0/3813301901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c0c8100 0x7f806c0abd00 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f806c067ba0 tx=0x7f805c01c490 comp rx=0 tx=0).stop 2023-10-25T00:20:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f8073c4a640 1 -- 172.21.15.107:0/3813301901 shutdown_connections 2023-10-25T00:20:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f8073c4a640 1 --2- 172.21.15.107:0/3813301901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f806c06a4b0 0x7f806c0ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f8073c4a640 1 --2- 172.21.15.107:0/3813301901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c0c8100 0x7f806c0abd00 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:24.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f8073c4a640 1 --2- 172.21.15.107:0/3813301901 >> v2:172.21.15.110:3300/0 conn(0x7f806c152040 0x7f806c0c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f8073c4a640 1 -- 172.21.15.107:0/3813301901 >> 172.21.15.107:0/3813301901 conn(0x7f806c0ca200 msgr2=0x7f806c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f8073c4a640 1 -- 172.21.15.107:0/3813301901 shutdown_connections 2023-10-25T00:20:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.906+0000 7f8073c4a640 1 -- 172.21.15.107:0/3813301901 wait complete. 2023-10-25T00:20:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.910+0000 7f8073c4a640 1 Processor -- start 2023-10-25T00:20:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.910+0000 7f8073c4a640 1 -- start start 2023-10-25T00:20:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f8073c4a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f806c06a4b0 0x7f806c151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:24.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f8073c4a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f806c0c8100 0x7f806c14d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f8073c4a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c152040 0x7f806c14fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f8073c4a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c0ad690 con 0x7f806c152040 2023-10-25T00:20:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f8073c4a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c0aed10 con 0x7f806c06a4b0 2023-10-25T00:20:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f8073c4a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806c14e400 con 0x7f806c0c8100 2023-10-25T00:20:24.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80721e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c152040 0x7f806c14fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80721e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c152040 0x7f806c14fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50044/0 (socket says 172.21.15.107:50044) 2023-10-25T00:20:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80721e7640 1 -- 172.21.15.107:0/2668893634 learned_addr learned my addr 172.21.15.107:0/2668893634 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80711e5640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f806c0c8100 0x7f806c14d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80719e6640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f806c06a4b0 0x7f806c151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:24.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80721e7640 1 -- 172.21.15.107:0/2668893634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f806c0c8100 msgr2=0x7f806c14d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:24.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80721e7640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f806c0c8100 0x7f806c14d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:24.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80721e7640 1 -- 172.21.15.107:0/2668893634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f806c06a4b0 msgr2=0x7f806c151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:24.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80721e7640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f806c06a4b0 0x7f806c151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:24.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80721e7640 1 -- 172.21.15.107:0/2668893634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c0094d0 con 0x7f806c152040 2023-10-25T00:20:24.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80721e7640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c152040 0x7f806c14fc30 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f8064009d30 tx=0x7f806400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:24.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80627fc640 1 -- 172.21.15.107:0/2668893634 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8064009ba0 con 0x7f806c152040 2023-10-25T00:20:24.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80627fc640 1 -- 172.21.15.107:0/2668893634 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806400f040 con 0x7f806c152040 2023-10-25T00:20:24.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f8073c4a640 1 -- 172.21.15.107:0/2668893634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806c14e120 con 0x7f806c152040 2023-10-25T00:20:24.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80627fc640 1 -- 172.21.15.107:0/2668893634 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80640146c0 con 0x7f806c152040 2023-10-25T00:20:24.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f8073c4a640 1 -- 172.21.15.107:0/2668893634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806c0ae930 con 0x7f806c152040 2023-10-25T00:20:24.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f8073c4a640 1 -- 172.21.15.107:0/2668893634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806c0aee50 con 0x7f806c152040 2023-10-25T00:20:24.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80627fc640 1 -- 172.21.15.107:0/2668893634 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8064014820 con 0x7f806c152040 2023-10-25T00:20:24.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80627fc640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8050065e60 0x7f8050068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:24.924 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.914+0000 7f80627fc640 1 -- 172.21.15.107:0/2668893634 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7f8064085410 con 0x7f806c152040 2023-10-25T00:20:24.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.922+0000 7f80719e6640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8050065e60 0x7f8050068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:24.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.922+0000 7f80627fc640 1 -- 172.21.15.107:0/2668893634 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f806c0aee50 con 0x7f806c152040 2023-10-25T00:20:24.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:24.922+0000 7f80719e6640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8050065e60 0x7f8050068320 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f8068009890 tx=0x7f8068008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:25.168 INFO:tasks.ceph.mds.b.smithi110.stdout:starting mds.b at 2023-10-25T00:20:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:25.187 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:25.187 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 219 2023-10-25T00:20:25.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.182+0000 7f8073c4a640 1 -- 172.21.15.107:0/2668893634 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f806c0aee50 con 0x7f806c152040 2023-10-25T00:20:25.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.182+0000 7f80627fc640 1 -- 172.21.15.107:0/2668893634 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+5010 (secure 0 0 0) 0x7f806c0aee50 con 0x7f806c152040 2023-10-25T00:20:25.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 -- 172.21.15.107:0/2668893634 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8050065e60 msgr2=0x7f8050068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:25.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8050065e60 0x7f8050068320 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f8068009890 tx=0x7f8068008040 comp rx=0 tx=0).stop 2023-10-25T00:20:25.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 -- 172.21.15.107:0/2668893634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c152040 msgr2=0x7f806c14fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:25.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c152040 0x7f806c14fc30 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f8064009d30 tx=0x7f806400c490 comp rx=0 tx=0).stop 2023-10-25T00:20:25.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 -- 172.21.15.107:0/2668893634 shutdown_connections 2023-10-25T00:20:25.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f806c152040 0x7f806c14fc30 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:25.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8050065e60 0x7f8050068320 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:25.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f806c0c8100 0x7f806c14d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:25.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 --2- 172.21.15.107:0/2668893634 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f806c06a4b0 0x7f806c151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:25.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 -- 172.21.15.107:0/2668893634 >> 172.21.15.107:0/2668893634 conn(0x7f806c0ca200 msgr2=0x7f806c14b370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:25.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 -- 172.21.15.107:0/2668893634 shutdown_connections 2023-10-25T00:20:25.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:25.186+0000 7f8073c4a640 1 -- 172.21.15.107:0/2668893634 wait complete. 2023-10-25T00:20:25.209 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2023-10-25T00:20:27.039 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:20:27.038+0000 7f150536e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:20:27.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:20:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.350+0000 7f5652108640 1 Processor -- start 2023-10-25T00:20:27.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f5652108640 1 -- start start 2023-10-25T00:20:27.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f5652108640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f564c14eb70 0x7f564c0c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:27.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f5652108640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 0x7f564c06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:27.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f5652108640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f564c0c22f0 con 0x7f564c070b00 2023-10-25T00:20:27.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f5652108640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c071700 con 0x7f564c14eb70 2023-10-25T00:20:27.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f5652108640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0c25d0 con 0x7f564c06df30 2023-10-25T00:20:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f564bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 0x7f564c06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f564b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f564c14eb70 0x7f564c0c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f564bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 0x7f564c06e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:38354/0 (socket says 172.21.15.107:38354) 2023-10-25T00:20:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f564bfff640 1 -- 172.21.15.107:0/1114561311 learned_addr learned my addr 172.21.15.107:0/1114561311 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:27.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f564bfff640 1 -- 172.21.15.107:0/1114561311 >> v2:172.21.15.110:3300/0 conn(0x7f564c14eb70 msgr2=0x7f564c0c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:20:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f564bfff640 1 --2- 172.21.15.107:0/1114561311 >> v2:172.21.15.110:3300/0 conn(0x7f564c14eb70 0x7f564c0c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f564bfff640 1 -- 172.21.15.107:0/1114561311 >> v1:172.21.15.110:6789/0 conn(0x7f564c070b00 legacy=0x7f564c1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.354+0000 7f564bfff640 1 -- 172.21.15.107:0/1114561311 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c0c2be0 con 0x7f564c06df30 2023-10-25T00:20:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.358+0000 7f564bfff640 1 --2- 172.21.15.107:0/1114561311 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 0x7f564c06e3a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5644009e50 tx=0x7f564401ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60e119ba313d4962 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:27.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.358+0000 7f564a7fc640 1 -- 172.21.15.107:0/1114561311 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56440a5900 con 0x7f564c06df30 2023-10-25T00:20:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.358+0000 7f564a7fc640 1 -- 172.21.15.107:0/1114561311 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56440a5a60 con 0x7f564c06df30 2023-10-25T00:20:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.358+0000 7f564a7fc640 1 -- 172.21.15.107:0/1114561311 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56440a5df0 con 0x7f564c06df30 2023-10-25T00:20:27.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.358+0000 7f5652108640 1 -- 172.21.15.107:0/1114561311 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 msgr2=0x7f564c06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:27.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.358+0000 7f5652108640 1 --2- 172.21.15.107:0/1114561311 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 0x7f564c06e3a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5644009e50 tx=0x7f564401ca30 comp rx=0 tx=0).stop 2023-10-25T00:20:27.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 -- 172.21.15.107:0/1114561311 shutdown_connections 2023-10-25T00:20:27.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 --2- 172.21.15.107:0/1114561311 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 0x7f564c06e3a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:27.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 --2- 172.21.15.107:0/1114561311 >> v2:172.21.15.110:3300/0 conn(0x7f564c14eb70 0x7f564c0c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:27.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 -- 172.21.15.107:0/1114561311 >> 172.21.15.107:0/1114561311 conn(0x7f564c0c39a0 msgr2=0x7f564c0c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:27.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 -- 172.21.15.107:0/1114561311 shutdown_connections 2023-10-25T00:20:27.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 -- 172.21.15.107:0/1114561311 wait complete. 2023-10-25T00:20:27.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 Processor -- start 2023-10-25T00:20:27.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 -- start start 2023-10-25T00:20:27.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 0x7f564c1f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:27.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f564c070b00 0x7f564c137e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:27.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f564c14eb70 0x7f564c13d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:27.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0a9ca0 con 0x7f564c070b00 2023-10-25T00:20:27.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c1f1450 con 0x7f564c14eb70 2023-10-25T00:20:27.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c144df0 con 0x7f564c06df30 2023-10-25T00:20:27.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f564affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f564c070b00 0x7f564c137e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:27.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f564affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f564c070b00 0x7f564c137e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50052/0 (socket says 172.21.15.107:50052) 2023-10-25T00:20:27.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f564affd640 1 -- 172.21.15.107:0/2561121399 learned_addr learned my addr 172.21.15.107:0/2561121399 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:20:27.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f564bfff640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f564c14eb70 0x7f564c13d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:27.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f564b7fe640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 0x7f564c1f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:27.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f564affd640 1 -- 172.21.15.107:0/2561121399 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 msgr2=0x7f564c1f0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:27.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f564affd640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 0x7f564c1f0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:27.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f564affd640 1 -- 172.21.15.107:0/2561121399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f564c14eb70 msgr2=0x7f564c13d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:27.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f564affd640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f564c14eb70 0x7f564c13d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:27.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f564affd640 1 -- 172.21.15.107:0/2561121399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5644009b30 con 0x7f564c070b00 2023-10-25T00:20:27.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f564affd640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f564c070b00 0x7f564c137e00 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f564000c420 tx=0x7f564000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:27.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5633fff640 1 -- 172.21.15.107:0/2561121399 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56400052d0 con 0x7f564c070b00 2023-10-25T00:20:27.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5633fff640 1 -- 172.21.15.107:0/2561121399 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5640005430 con 0x7f564c070b00 2023-10-25T00:20:27.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.362+0000 7f5652108640 1 -- 172.21.15.107:0/2561121399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564c1450d0 con 0x7f564c070b00 2023-10-25T00:20:27.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.366+0000 7f5633fff640 1 -- 172.21.15.107:0/2561121399 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5640003820 con 0x7f564c070b00 2023-10-25T00:20:27.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.366+0000 7f5652108640 1 -- 172.21.15.107:0/2561121399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564c0aabd0 con 0x7f564c070b00 2023-10-25T00:20:27.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.366+0000 7f5633fff640 1 -- 172.21.15.107:0/2561121399 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5640003a40 con 0x7f564c070b00 2023-10-25T00:20:27.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.366+0000 7f5652108640 1 -- 172.21.15.107:0/2561121399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564c0746e0 con 0x7f564c070b00 2023-10-25T00:20:27.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.366+0000 7f5633fff640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5638065e60 0x7f5638068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:20:27.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.366+0000 7f5633fff640 1 -- 172.21.15.107:0/2561121399 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7f56400864a0 con 0x7f564c070b00 2023-10-25T00:20:27.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.370+0000 7f564b7fe640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5638065e60 0x7f5638068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:20:27.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.370+0000 7f564b7fe640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5638065e60 0x7f5638068320 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f563c009b60 tx=0x7f563c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:20:27.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.370+0000 7f5633fff640 1 -- 172.21.15.107:0/2561121399 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5640050110 con 0x7f564c070b00 2023-10-25T00:20:27.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.642+0000 7f5652108640 1 -- 172.21.15.107:0/2561121399 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f564c1489b0 con 0x7f564c070b00 2023-10-25T00:20:27.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.642+0000 7f5633fff640 1 -- 172.21.15.107:0/2561121399 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5765 (secure 0 0 0) 0x7f5640017020 con 0x7f564c070b00 2023-10-25T00:20:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:20:27.646 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:20:27.646 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 220 2023-10-25T00:20:27.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 -- 172.21.15.107:0/2561121399 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5638065e60 msgr2=0x7f5638068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:27.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5638065e60 0x7f5638068320 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f563c009b60 tx=0x7f563c009190 comp rx=0 tx=0).stop 2023-10-25T00:20:27.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 -- 172.21.15.107:0/2561121399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f564c070b00 msgr2=0x7f564c137e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:20:27.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f564c070b00 0x7f564c137e00 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f564000c420 tx=0x7f564000c8e0 comp rx=0 tx=0).stop 2023-10-25T00:20:27.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 -- 172.21.15.107:0/2561121399 shutdown_connections 2023-10-25T00:20:27.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f564c14eb70 0x7f564c13d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:27.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f564c070b00 0x7f564c137e00 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:27.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5638065e60 0x7f5638068320 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:27.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 --2- 172.21.15.107:0/2561121399 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f564c06df30 0x7f564c1f0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:20:27.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 -- 172.21.15.107:0/2561121399 >> 172.21.15.107:0/2561121399 conn(0x7f564c0c39a0 msgr2=0x7f564c15b090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:20:27.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 -- 172.21.15.107:0/2561121399 shutdown_connections 2023-10-25T00:20:27.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:20:27.646+0000 7f5652108640 1 -- 172.21.15.107:0/2561121399 wait complete. 2023-10-25T00:20:27.669 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2023-10-25T00:20:27.669 INFO:tasks.mds_thrash.fs.[a]:waiting for 56 secs before thrashing 2023-10-25T00:21:04.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:04.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.537+0000 7f80f5a80640 1 Processor -- start 2023-10-25T00:21:04.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.537+0000 7f80f5a80640 1 -- start start 2023-10-25T00:21:04.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.537+0000 7f80f5a80640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80f00aa990 0x7f80f0164a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:04.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.537+0000 7f80f5a80640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f01656c0 con 0x7f80f015eb00 2023-10-25T00:21:04.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.537+0000 7f80f5a80640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f0161560 con 0x7f80f008b9e0 2023-10-25T00:21:04.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.537+0000 7f80f5a80640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f015eee0 con 0x7f80f00aa990 2023-10-25T00:21:04.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.537+0000 7f80ef7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f80f015eb00 0x7f80f006b4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45524/0 (socket says 172.21.15.107:45524) 2023-10-25T00:21:04.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.537+0000 7f80ef7fe640 1 -- 172.21.15.107:0/4188772479 learned_addr learned my addr 172.21.15.107:0/4188772479 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:04.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.537+0000 7f80eeffd640 1 --2- 172.21.15.107:0/4188772479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80f00aa990 0x7f80f0164a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:04.553 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.537+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443818510 0 0) 0x7f80f01656c0 con 0x7f80f015eb00 2023-10-25T00:21:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d8003530 con 0x7f80f015eb00 2023-10-25T00:21:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525362600 0 0) 0x7f80f0161560 con 0x7f80f008b9e0 2023-10-25T00:21:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f01656c0 con 0x7f80f008b9e0 2023-10-25T00:21:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1665431324 0 0) 0x7f80d8003530 con 0x7f80f015eb00 2023-10-25T00:21:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80f00aa990 msgr2=0x7f80f0164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 --2- 172.21.15.107:0/4188772479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80f00aa990 0x7f80f0164a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 >> v1:172.21.15.110:6789/0 conn(0x7f80f008b9e0 legacy=0x7f80f0164fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f0161840 con 0x7f80f015eb00 2023-10-25T00:21:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f80e8001a00 con 0x7f80f015eb00 2023-10-25T00:21:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 >> v1:172.21.15.107:6789/0 conn(0x7f80f015eb00 legacy=0x7f80f006b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 --2- 172.21.15.107:0/4188772479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80d8003d10 0x7f80d8006120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 --2- 172.21.15.107:0/4188772479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80d8006740 0x7f80d8008bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 --2- 172.21.15.107:0/4188772479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80d8009390 0x7f80d800b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f015eee0 con 0x7f80d8009390 2023-10-25T00:21:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80d800c100 con 0x7f80d8006740 2023-10-25T00:21:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80d800c2c0 con 0x7f80d8003d10 2023-10-25T00:21:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80effff640 1 --2- 172.21.15.107:0/4188772479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80d8009390 0x7f80d800b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80ef7fe640 1 --2- 172.21.15.107:0/4188772479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80d8003d10 0x7f80d8006120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80effff640 1 -- 172.21.15.107:0/4188772479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80d8003d10 msgr2=0x7f80d8006120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80effff640 1 --2- 172.21.15.107:0/4188772479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80d8003d10 0x7f80d8006120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:04.557 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80effff640 1 -- 172.21.15.107:0/4188772479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80d8006740 msgr2=0x7f80d8008bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80effff640 1 --2- 172.21.15.107:0/4188772479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80d8006740 0x7f80d8008bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.541+0000 7f80effff640 1 -- 172.21.15.107:0/4188772479 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d8003530 con 0x7f80d8009390 2023-10-25T00:21:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80effff640 1 --2- 172.21.15.107:0/4188772479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80d8009390 0x7f80d800b820 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f80e0001950 tx=0x7f80e002fd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1dcf7bdfcc2811 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e0005110 con 0x7f80d8009390 2023-10-25T00:21:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e0005270 con 0x7f80d8009390 2023-10-25T00:21:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80ee7fc640 1 -- 172.21.15.107:0/4188772479 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e00b9890 con 0x7f80d8009390 2023-10-25T00:21:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80f5a80640 1 -- 172.21.15.107:0/4188772479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80d8009390 msgr2=0x7f80d800b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80f5a80640 1 --2- 172.21.15.107:0/4188772479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80d8009390 0x7f80d800b820 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f80e0001950 tx=0x7f80e002fd60 comp rx=0 tx=0).stop 2023-10-25T00:21:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80effff640 1 -- 172.21.15.107:0/4188772479 reap_dead start 2023-10-25T00:21:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80f5a80640 1 -- 172.21.15.107:0/4188772479 shutdown_connections 2023-10-25T00:21:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80f5a80640 1 -- 172.21.15.107:0/4188772479 >> 172.21.15.107:0/4188772479 conn(0x7f80f00c2950 msgr2=0x7f80f00c2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80f5a80640 1 -- 172.21.15.107:0/4188772479 shutdown_connections 2023-10-25T00:21:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80f5a80640 1 -- 172.21.15.107:0/4188772479 wait complete. 2023-10-25T00:21:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80f5a80640 1 Processor -- start 2023-10-25T00:21:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.545+0000 7f80f5a80640 1 -- start start 2023-10-25T00:21:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80f5a80640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80d8009390 0x7f80f00c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80f5a80640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80f0070ed0 0x7f80f0071520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80f5a80640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80f006fb50 0x7f80f0071a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80f5a80640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f00c5b20 con 0x7f80f0070ed0 2023-10-25T00:21:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80f5a80640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01f7160 con 0x7f80d8009390 2023-10-25T00:21:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80f5a80640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f0161250 con 0x7f80f006fb50 2023-10-25T00:21:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80eeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80f0070ed0 0x7f80f0071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80eeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80f0070ed0 0x7f80f0071520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32860/0 (socket says 172.21.15.107:32860) 2023-10-25T00:21:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80eeffd640 1 -- 172.21.15.107:0/3235823856 learned_addr learned my addr 172.21.15.107:0/3235823856 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:04.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80effff640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80f006fb50 0x7f80f0071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80ef7fe640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80d8009390 0x7f80f00c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80eeffd640 1 -- 172.21.15.107:0/3235823856 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80f006fb50 msgr2=0x7f80f0071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80eeffd640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80f006fb50 0x7f80f0071a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80eeffd640 1 -- 172.21.15.107:0/3235823856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80d8009390 msgr2=0x7f80f00c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:04.563 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80eeffd640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80d8009390 0x7f80f00c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80eeffd640 1 -- 172.21.15.107:0/3235823856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f01f7400 con 0x7f80f0070ed0 2023-10-25T00:21:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80ef7fe640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80d8009390 0x7f80f00c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:21:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80eeffd640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80f0070ed0 0x7f80f0071520 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f80e40047c0 tx=0x7f80e400d880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80d7fff640 1 -- 172.21.15.107:0/3235823856 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4012760 con 0x7f80f0070ed0 2023-10-25T00:21:04.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80d7fff640 1 -- 172.21.15.107:0/3235823856 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e400f040 con 0x7f80f0070ed0 2023-10-25T00:21:04.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.549+0000 7f80d7fff640 1 -- 172.21.15.107:0/3235823856 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4012b20 con 0x7f80f0070ed0 2023-10-25T00:21:04.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.557+0000 7f80f5a80640 1 -- 172.21.15.107:0/3235823856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f015f3d0 con 0x7f80f0070ed0 2023-10-25T00:21:04.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.557+0000 7f80f5a80640 1 -- 172.21.15.107:0/3235823856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f015e210 con 0x7f80f0070ed0 2023-10-25T00:21:04.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.557+0000 7f80d5ffb640 1 -- 172.21.15.107:0/3235823856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c8005260 con 0x7f80f0070ed0 2023-10-25T00:21:04.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.561+0000 7f80d7fff640 1 -- 172.21.15.107:0/3235823856 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f80e4012c80 con 0x7f80f0070ed0 2023-10-25T00:21:04.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.561+0000 7f80d7fff640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80dc065e90 0x7f80dc068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:04.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.561+0000 7f80d7fff640 1 -- 172.21.15.107:0/3235823856 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7f80e40850c0 con 0x7f80f0070ed0 2023-10-25T00:21:04.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.561+0000 7f80ef7fe640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80dc065e90 0x7f80dc068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:04.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.561+0000 7f80ef7fe640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80dc065e90 0x7f80dc068350 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f80e8002240 tx=0x7f80e8006040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:04.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.565+0000 7f80d7fff640 1 -- 172.21.15.107:0/3235823856 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f80e40515a0 con 0x7f80f0070ed0 2023-10-25T00:21:04.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.845+0000 7f80d5ffb640 1 -- 172.21.15.107:0/3235823856 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80c80057e0 con 0x7f80f0070ed0 2023-10-25T00:21:04.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.845+0000 7f80d7fff640 1 -- 172.21.15.107:0/3235823856 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5765 (secure 0 0 0) 0x7f80e4054bc0 con 0x7f80f0070ed0 2023-10-25T00:21:04.851 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:04.851 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 220 2023-10-25T00:21:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 -- 172.21.15.107:0/3235823856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80dc065e90 msgr2=0x7f80dc068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80dc065e90 0x7f80dc068350 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f80e8002240 tx=0x7f80e8006040 comp rx=0 tx=0).stop 2023-10-25T00:21:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 -- 172.21.15.107:0/3235823856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80f0070ed0 msgr2=0x7f80f0071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:04.853 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80f0070ed0 0x7f80f0071520 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f80e40047c0 tx=0x7f80e400d880 comp rx=0 tx=0).stop 2023-10-25T00:21:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 -- 172.21.15.107:0/3235823856 shutdown_connections 2023-10-25T00:21:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80dc065e90 0x7f80dc068350 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f80f006fb50 0x7f80f0071a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:04.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f80f0070ed0 0x7f80f0071520 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 --2- 172.21.15.107:0/3235823856 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f80d8009390 0x7f80f00c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 -- 172.21.15.107:0/3235823856 >> 172.21.15.107:0/3235823856 conn(0x7f80f00c2950 msgr2=0x7f80f00c10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 -- 172.21.15.107:0/3235823856 shutdown_connections 2023-10-25T00:21:04.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:04.849+0000 7f80d5ffb640 1 -- 172.21.15.107:0/3235823856 wait complete. 2023-10-25T00:21:04.874 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2023-10-25T00:21:04.875 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:21:04.876 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:21:04.885 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:21:04.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:05.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.061+0000 7fa482620640 1 Processor -- start 2023-10-25T00:21:05.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.061+0000 7fa482620640 1 -- start start 2023-10-25T00:21:05.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.061+0000 7fa482620640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa47c0c1c60 0x7fa47c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:05.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.061+0000 7fa482620640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c154cb0 con 0x7fa47c155850 2023-10-25T00:21:05.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.061+0000 7fa480bbd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa47c155850 0x7fa47c15f3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45528/0 (socket says 172.21.15.107:45528) 2023-10-25T00:21:05.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.061+0000 7fa480bbd640 1 -- 172.21.15.107:0/2510686269 learned_addr learned my addr 172.21.15.107:0/2510686269 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:05.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.061+0000 7fa482620640 1 -- 172.21.15.107:0/2510686269 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c152550 con 0x7fa47c0c2040 2023-10-25T00:21:05.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.061+0000 7fa482620640 1 -- 172.21.15.107:0/2510686269 --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa47c0bd290 con 0x7fa47c0c1c60 2023-10-25T00:21:05.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483379524 0 0) 0x7fa47c154cb0 con 0x7fa47c155850 2023-10-25T00:21:05.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa468003530 con 0x7fa47c155850 2023-10-25T00:21:05.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3552303034 0 0) 0x7fa468003530 con 0x7fa47c155850 2023-10-25T00:21:05.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 >> v2:172.21.15.107:3300/0 conn(0x7fa47c0c1c60 msgr2=0x7fa47c06c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:05.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 --2- 172.21.15.107:0/2510686269 >> v2:172.21.15.107:3300/0 conn(0x7fa47c0c1c60 0x7fa47c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:05.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 >> v1:172.21.15.110:6789/0 conn(0x7fa47c0c2040 legacy=0x7fa47c155140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:05.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c06b0e0 con 0x7fa47c155850 2023-10-25T00:21:05.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fa46c001a00 con 0x7fa47c155850 2023-10-25T00:21:05.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 >> v1:172.21.15.107:6789/0 conn(0x7fa47c155850 legacy=0x7fa47c15f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:05.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 --2- 172.21.15.107:0/2510686269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4680039d0 0x7fa468003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 --2- 172.21.15.107:0/2510686269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa4680047d0 0x7fa468004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 --2- 172.21.15.107:0/2510686269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 0x7fa4680141a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c0bd290 con 0x7fa468005390 2023-10-25T00:21:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa468005960 con 0x7fa4680039d0 2023-10-25T00:21:05.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa468005b20 con 0x7fa4680047d0 2023-10-25T00:21:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa480bbd640 1 --2- 172.21.15.107:0/2510686269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 0x7fa4680141a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa480bbd640 1 -- 172.21.15.107:0/2510686269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa4680047d0 msgr2=0x7fa468004c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:21:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa480bbd640 1 --2- 172.21.15.107:0/2510686269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa4680047d0 0x7fa468004c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:05.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa480bbd640 1 -- 172.21.15.107:0/2510686269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4680039d0 msgr2=0x7fa468003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:05.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa480bbd640 1 --2- 172.21.15.107:0/2510686269 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa4680039d0 0x7fa468003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:05.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.065+0000 7fa480bbd640 1 -- 172.21.15.107:0/2510686269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa468003530 con 0x7fa468005390 2023-10-25T00:21:05.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.069+0000 7fa480bbd640 1 -- 172.21.15.107:0/2510686269 reap_dead start 2023-10-25T00:21:05.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.069+0000 7fa480bbd640 1 --2- 172.21.15.107:0/2510686269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 0x7fa4680141a0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00a7e0 tx=0x7fa46c01d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d77f73c0833a8e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.069+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c0ac620 con 0x7fa468005390 2023-10-25T00:21:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.069+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46c0a8070 con 0x7fa468005390 2023-10-25T00:21:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.069+0000 7fa47affd640 1 -- 172.21.15.107:0/2510686269 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46c0ac9c0 con 0x7fa468005390 2023-10-25T00:21:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.069+0000 7fa482620640 1 -- 172.21.15.107:0/2510686269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 msgr2=0x7fa4680141a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.073+0000 7fa482620640 1 --2- 172.21.15.107:0/2510686269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 0x7fa4680141a0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00a7e0 tx=0x7fa46c01d300 comp rx=0 tx=0).stop 2023-10-25T00:21:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.073+0000 7fa482620640 1 -- 172.21.15.107:0/2510686269 shutdown_connections 2023-10-25T00:21:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.073+0000 7fa482620640 1 --2- 172.21.15.107:0/2510686269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 0x7fa4680141a0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.073+0000 7fa482620640 1 -- 172.21.15.107:0/2510686269 >> 172.21.15.107:0/2510686269 conn(0x7fa47c0c2d50 msgr2=0x7fa47c0c3160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 -- 172.21.15.107:0/2510686269 shutdown_connections 2023-10-25T00:21:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 -- 172.21.15.107:0/2510686269 wait complete. 2023-10-25T00:21:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 Processor -- start 2023-10-25T00:21:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 -- start start 2023-10-25T00:21:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 0x7fa47c144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa47c0c5b60 0x7fa47c14bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa47c1538e0 0x7fa47c0bb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:05.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c146070 con 0x7fa468005390 2023-10-25T00:21:05.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c152eb0 con 0x7fa47c0c5b60 2023-10-25T00:21:05.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c146250 con 0x7fa47c1538e0 2023-10-25T00:21:05.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa47bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 0x7fa47c144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa47bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 0x7fa47c144530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32880/0 (socket says 172.21.15.107:32880) 2023-10-25T00:21:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa47bfff640 1 -- 172.21.15.107:0/420841471 learned_addr learned my addr 172.21.15.107:0/420841471 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa47b7fe640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa47c0c5b60 0x7fa47c14bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:05.087 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa480bbd640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa47c1538e0 0x7fa47c0bb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa47bfff640 1 -- 172.21.15.107:0/420841471 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa47c1538e0 msgr2=0x7fa47c0bb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa47bfff640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa47c1538e0 0x7fa47c0bb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa47bfff640 1 -- 172.21.15.107:0/420841471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa47c0c5b60 msgr2=0x7fa47c14bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa47bfff640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa47c0c5b60 0x7fa47c14bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa47bfff640 1 -- 172.21.15.107:0/420841471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c0c4200 con 0x7fa468005390 2023-10-25T00:21:05.088 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa47bfff640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 0x7fa47c144530 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fa47400b3c0 tx=0x7fa47400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:05.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa478ff9640 1 -- 172.21.15.107:0/420841471 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa474011830 con 0x7fa468005390 2023-10-25T00:21:05.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa478ff9640 1 -- 172.21.15.107:0/420841471 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa474011990 con 0x7fa468005390 2023-10-25T00:21:05.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa478ff9640 1 -- 172.21.15.107:0/420841471 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa474011d40 con 0x7fa468005390 2023-10-25T00:21:05.089 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 -- 172.21.15.107:0/420841471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47c1530e0 con 0x7fa468005390 2023-10-25T00:21:05.090 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.077+0000 7fa482620640 1 -- 172.21.15.107:0/420841471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47c1549e0 con 0x7fa468005390 2023-10-25T00:21:05.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.081+0000 7fa482620640 1 -- 172.21.15.107:0/420841471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47c153600 con 0x7fa468005390 2023-10-25T00:21:05.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.081+0000 7fa478ff9640 1 -- 172.21.15.107:0/420841471 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa47401b460 con 0x7fa468005390 2023-10-25T00:21:05.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.081+0000 7fa478ff9640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa45c065e90 0x7fa45c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:05.091 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.081+0000 7fa478ff9640 1 -- 172.21.15.107:0/420841471 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7fa474084ef0 con 0x7fa468005390 2023-10-25T00:21:05.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.081+0000 7fa47b7fe640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa45c065e90 0x7fa45c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:05.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.081+0000 7fa47b7fe640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa45c065e90 0x7fa45c068350 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa47c0bc270 tx=0x7fa464009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:05.092 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.085+0000 7fa478ff9640 1 -- 172.21.15.107:0/420841471 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa474051510 con 0x7fa468005390 2023-10-25T00:21:05.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.357+0000 7fa482620640 1 -- 172.21.15.107:0/420841471 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa47c153600 con 0x7fa468005390 2023-10-25T00:21:05.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.357+0000 7fa478ff9640 1 -- 172.21.15.107:0/420841471 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5765 (secure 0 0 0) 0x7fa47c153600 con 0x7fa468005390 2023-10-25T00:21:05.362 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:05.363 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:05.363 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 220 2023-10-25T00:21:05.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 -- 172.21.15.107:0/420841471 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa45c065e90 msgr2=0x7fa45c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:05.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa45c065e90 0x7fa45c068350 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa47c0bc270 tx=0x7fa464009040 comp rx=0 tx=0).stop 2023-10-25T00:21:05.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 -- 172.21.15.107:0/420841471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 msgr2=0x7fa47c144530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:05.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 0x7fa47c144530 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7fa47400b3c0 tx=0x7fa47400b880 comp rx=0 tx=0).stop 2023-10-25T00:21:05.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 -- 172.21.15.107:0/420841471 shutdown_connections 2023-10-25T00:21:05.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa45c065e90 0x7fa45c068350 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:05.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa47c1538e0 0x7fa47c0bb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:05.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa47c0c5b60 0x7fa47c14bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:05.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 --2- 172.21.15.107:0/420841471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa468005390 0x7fa47c144530 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:05.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 -- 172.21.15.107:0/420841471 >> 172.21.15.107:0/420841471 conn(0x7fa47c0c2d50 msgr2=0x7fa47c149a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:05.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 -- 172.21.15.107:0/420841471 shutdown_connections 2023-10-25T00:21:05.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:05.361+0000 7fa482620640 1 -- 172.21.15.107:0/420841471 wait complete. 2023-10-25T00:21:05.387 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:21:07.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:07.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74ea89640 1 Processor -- start 2023-10-25T00:21:07.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74ea89640 1 -- start start 2023-10-25T00:21:07.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74ea89640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa74814eb70 0x7fa7480c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:07.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74ea89640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa74806df30 0x7fa74806e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74ea89640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7480c22f0 con 0x7fa748070b00 2023-10-25T00:21:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74ea89640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa748071700 con 0x7fa74806df30 2023-10-25T00:21:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74ea89640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7480c25d0 con 0x7fa74814eb70 2023-10-25T00:21:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74d026640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa74806df30 0x7fa74806e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74d026640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa74806df30 0x7fa74806e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32888/0 (socket says 172.21.15.107:32888) 2023-10-25T00:21:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74d026640 1 -- 172.21.15.107:0/2288197550 learned_addr learned my addr 172.21.15.107:0/2288197550 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74c825640 1 --2- 172.21.15.107:0/2288197550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa74814eb70 0x7fa7480c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74d026640 1 -- 172.21.15.107:0/2288197550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa74814eb70 msgr2=0x7fa7480c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74d026640 1 --2- 172.21.15.107:0/2288197550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa74814eb70 0x7fa7480c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74d026640 1 -- 172.21.15.107:0/2288197550 >> v1:172.21.15.110:6789/0 conn(0x7fa748070b00 legacy=0x7fa7481514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74d026640 1 -- 172.21.15.107:0/2288197550 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7480c2be0 con 0x7fa74806df30 2023-10-25T00:21:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74c825640 1 --2- 172.21.15.107:0/2288197550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa74814eb70 0x7fa7480c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:21:07.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.533+0000 7fa74d026640 1 --2- 172.21.15.107:0/2288197550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa74806df30 0x7fa74806e360 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fa740009e50 tx=0x7fa74001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c124f3a7c0a853 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:07.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa7477fe640 1 -- 172.21.15.107:0/2288197550 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7400a5800 con 0x7fa74806df30 2023-10-25T00:21:07.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa7477fe640 1 -- 172.21.15.107:0/2288197550 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7400a5960 con 0x7fa74806df30 2023-10-25T00:21:07.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa7477fe640 1 -- 172.21.15.107:0/2288197550 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7400a5cf0 con 0x7fa74806df30 2023-10-25T00:21:07.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 -- 172.21.15.107:0/2288197550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa74806df30 msgr2=0x7fa74806e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:07.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 --2- 172.21.15.107:0/2288197550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa74806df30 0x7fa74806e360 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fa740009e50 tx=0x7fa74001ca10 comp rx=0 tx=0).stop 2023-10-25T00:21:07.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 -- 172.21.15.107:0/2288197550 shutdown_connections 2023-10-25T00:21:07.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 --2- 172.21.15.107:0/2288197550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa74806df30 0x7fa74806e360 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:07.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 --2- 172.21.15.107:0/2288197550 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa74814eb70 0x7fa7480c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:07.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 -- 172.21.15.107:0/2288197550 >> 172.21.15.107:0/2288197550 conn(0x7fa7480c39a0 msgr2=0x7fa7480c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:07.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 -- 172.21.15.107:0/2288197550 shutdown_connections 2023-10-25T00:21:07.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 -- 172.21.15.107:0/2288197550 wait complete. 2023-10-25T00:21:07.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 Processor -- start 2023-10-25T00:21:07.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 -- start start 2023-10-25T00:21:07.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa74806df30 0x7fa74814d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:07.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa748070b00 0x7fa74813f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:07.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa74814eb70 0x7fa74813f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:07.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74814ce40 con 0x7fa74814eb70 2023-10-25T00:21:07.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7481454e0 con 0x7fa748070b00 2023-10-25T00:21:07.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa748148f00 con 0x7fa74806df30 2023-10-25T00:21:07.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74d026640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa74814eb70 0x7fa74813f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:07.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74d026640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa74814eb70 0x7fa74813f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32896/0 (socket says 172.21.15.107:32896) 2023-10-25T00:21:07.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74d026640 1 -- 172.21.15.107:0/365481379 learned_addr learned my addr 172.21.15.107:0/365481379 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:07.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74c825640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa74806df30 0x7fa74814d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:07.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa747fff640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa748070b00 0x7fa74813f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:07.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74d026640 1 -- 172.21.15.107:0/365481379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa74806df30 msgr2=0x7fa74814d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:07.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74d026640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa74806df30 0x7fa74814d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:07.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74d026640 1 -- 172.21.15.107:0/365481379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa748070b00 msgr2=0x7fa74813f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:07.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74d026640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa748070b00 0x7fa74813f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:07.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74d026640 1 -- 172.21.15.107:0/365481379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa740009b30 con 0x7fa74814eb70 2023-10-25T00:21:07.548 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74d026640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa74814eb70 0x7fa74813f860 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fa74001a640 tx=0x7fa7400197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:07.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa7457fa640 1 -- 172.21.15.107:0/365481379 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7400b8670 con 0x7fa74814eb70 2023-10-25T00:21:07.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa7457fa640 1 -- 172.21.15.107:0/365481379 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7400b87d0 con 0x7fa74814eb70 2023-10-25T00:21:07.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 -- 172.21.15.107:0/365481379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74814cbf0 con 0x7fa74814eb70 2023-10-25T00:21:07.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa7457fa640 1 -- 172.21.15.107:0/365481379 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7400b8b60 con 0x7fa74814eb70 2023-10-25T00:21:07.549 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.537+0000 7fa74ea89640 1 -- 172.21.15.107:0/365481379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa748144fb0 con 0x7fa74814eb70 2023-10-25T00:21:07.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.541+0000 7fa74ea89640 1 -- 172.21.15.107:0/365481379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7480746e0 con 0x7fa74814eb70 2023-10-25T00:21:07.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.541+0000 7fa7457fa640 1 -- 172.21.15.107:0/365481379 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa7400b8cc0 con 0x7fa74814eb70 2023-10-25T00:21:07.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.541+0000 7fa7457fa640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa72c065e60 0x7fa72c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:07.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.541+0000 7fa74c825640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa72c065e60 0x7fa72c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:07.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.541+0000 7fa7457fa640 1 -- 172.21.15.107:0/365481379 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7fa74011fca0 con 0x7fa74814eb70 2023-10-25T00:21:07.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.541+0000 7fa74c825640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa72c065e60 0x7fa72c068320 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009950 tx=0x7fa73c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:07.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.545+0000 7fa7457fa640 1 -- 172.21.15.107:0/365481379 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa7400ec240 con 0x7fa74814eb70 2023-10-25T00:21:07.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.829+0000 7fa74ea89640 1 -- 172.21.15.107:0/365481379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7480c1a50 con 0x7fa74814eb70 2023-10-25T00:21:07.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.829+0000 7fa7457fa640 1 -- 172.21.15.107:0/365481379 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5765 (secure 0 0 0) 0x7fa7400bc090 con 0x7fa74814eb70 2023-10-25T00:21:07.835 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:07.835 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:07.836 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 220 2023-10-25T00:21:07.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 -- 172.21.15.107:0/365481379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa72c065e60 msgr2=0x7fa72c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:07.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa72c065e60 0x7fa72c068320 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa73c009950 tx=0x7fa73c008040 comp rx=0 tx=0).stop 2023-10-25T00:21:07.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 -- 172.21.15.107:0/365481379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa74814eb70 msgr2=0x7fa74813f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:07.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa74814eb70 0x7fa74813f860 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fa74001a640 tx=0x7fa7400197b0 comp rx=0 tx=0).stop 2023-10-25T00:21:07.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 -- 172.21.15.107:0/365481379 shutdown_connections 2023-10-25T00:21:07.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa74814eb70 0x7fa74813f860 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:07.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa748070b00 0x7fa74813f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:07.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa72c065e60 0x7fa72c068320 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:07.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 --2- 172.21.15.107:0/365481379 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa74806df30 0x7fa74814d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:07.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 -- 172.21.15.107:0/365481379 >> 172.21.15.107:0/365481379 conn(0x7fa7480c39a0 msgr2=0x7fa7480c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:07.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 -- 172.21.15.107:0/365481379 shutdown_connections 2023-10-25T00:21:07.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:07.833+0000 7fa74ea89640 1 -- 172.21.15.107:0/365481379 wait complete. 2023-10-25T00:21:07.851 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:21:09.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:09.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.981+0000 7f0127076640 1 Processor -- start 2023-10-25T00:21:09.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.981+0000 7f0127076640 1 -- start start 2023-10-25T00:21:09.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0127076640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f012014fd40 0x7f01200ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:09.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0127076640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012006f730 con 0x7f012006ee40 2023-10-25T00:21:09.989 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0127076640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0120156a40 con 0x7f01200a9820 2023-10-25T00:21:09.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0127076640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0120156710 con 0x7f012014fd40 2023-10-25T00:21:09.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0125613640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f012006ee40 0x7f012015cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45536/0 (socket says 172.21.15.107:45536) 2023-10-25T00:21:09.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0125613640 1 -- 172.21.15.107:0/1309337165 learned_addr learned my addr 172.21.15.107:0/1309337165 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:09.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315153710 0 0) 0x7f012006f730 con 0x7f012006ee40 2023-10-25T00:21:09.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f010c003530 con 0x7f012006ee40 2023-10-25T00:21:09.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0124e12640 1 --2- 172.21.15.107:0/1309337165 >> v2:172.21.15.110:3300/0 conn(0x7f012014fd40 0x7f01200ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:09.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2773505623 0 0) 0x7f010c003530 con 0x7f012006ee40 2023-10-25T00:21:09.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 >> v2:172.21.15.110:3300/0 conn(0x7f012014fd40 msgr2=0x7f01200ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:09.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 --2- 172.21.15.107:0/1309337165 >> v2:172.21.15.110:3300/0 conn(0x7f012014fd40 0x7f01200ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:09.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 >> v1:172.21.15.110:6789/0 conn(0x7f01200a9820 legacy=0x7f01200c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:09.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012006f980 con 0x7f012006ee40 2023-10-25T00:21:09.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f0118001680 con 0x7f012006ee40 2023-10-25T00:21:09.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 >> v1:172.21.15.107:6789/0 conn(0x7f012006ee40 legacy=0x7f012015cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:09.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 --2- 172.21.15.107:0/1309337165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f010c003980 0x7f010c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:09.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 --2- 172.21.15.107:0/1309337165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f010c0047d0 0x7f010c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:09.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 --2- 172.21.15.107:0/1309337165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f010c0052b0 0x7f010c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:09.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0120156710 con 0x7f010c003980 2023-10-25T00:21:09.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010c00c180 con 0x7f010c0052b0 2023-10-25T00:21:09.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010c00c340 con 0x7f010c0047d0 2023-10-25T00:21:09.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0124e12640 1 --2- 172.21.15.107:0/1309337165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f010c003980 0x7f010c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:09.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f011ffff640 1 --2- 172.21.15.107:0/1309337165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f010c0047d0 0x7f010c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:09.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0125613640 1 --2- 172.21.15.107:0/1309337165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f010c0052b0 0x7f010c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:09.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0124e12640 1 -- 172.21.15.107:0/1309337165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f010c0047d0 msgr2=0x7f010c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:09.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0124e12640 1 --2- 172.21.15.107:0/1309337165 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f010c0047d0 0x7f010c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:09.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0124e12640 1 -- 172.21.15.107:0/1309337165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f010c0052b0 msgr2=0x7f010c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:09.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0124e12640 1 --2- 172.21.15.107:0/1309337165 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f010c0052b0 0x7f010c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:09.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0124e12640 1 -- 172.21.15.107:0/1309337165 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010c003530 con 0x7f010c003980 2023-10-25T00:21:09.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0125613640 1 -- 172.21.15.107:0/1309337165 reap_dead start 2023-10-25T00:21:09.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.985+0000 7f0124e12640 1 --2- 172.21.15.107:0/1309337165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f010c003980 0x7f010c003da0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f0110009ab0 tx=0x7f011001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c5a02c04178712 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:09.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100a5520 con 0x7f010c003980 2023-10-25T00:21:09.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01100a5680 con 0x7f010c003980 2023-10-25T00:21:09.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f011f7fe640 1 -- 172.21.15.107:0/1309337165 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01100a5a30 con 0x7f010c003980 2023-10-25T00:21:09.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 -- 172.21.15.107:0/1309337165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f010c003980 msgr2=0x7f010c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:09.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 --2- 172.21.15.107:0/1309337165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f010c003980 0x7f010c003da0 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f0110009ab0 tx=0x7f011001ca30 comp rx=0 tx=0).stop 2023-10-25T00:21:09.999 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 -- 172.21.15.107:0/1309337165 shutdown_connections 2023-10-25T00:21:10.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 --2- 172.21.15.107:0/1309337165 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f010c003980 0x7f010c003da0 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:10.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 -- 172.21.15.107:0/1309337165 >> 172.21.15.107:0/1309337165 conn(0x7f01200c2950 msgr2=0x7f01200c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:10.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 -- 172.21.15.107:0/1309337165 shutdown_connections 2023-10-25T00:21:10.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 -- 172.21.15.107:0/1309337165 wait complete. 2023-10-25T00:21:10.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 Processor -- start 2023-10-25T00:21:10.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 -- start start 2023-10-25T00:21:10.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f010c003980 0x7f01200bf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:10.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f012014c9f0 0x7f012014bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:10.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01201dec60 0x7f012014c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:10.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01201df220 con 0x7f01201dec60 2023-10-25T00:21:10.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0120152190 con 0x7f010c003980 2023-10-25T00:21:10.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012014c870 con 0x7f012014c9f0 2023-10-25T00:21:10.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0125613640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01201dec60 0x7f012014c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:10.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0125613640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01201dec60 0x7f012014c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32916/0 (socket says 172.21.15.107:32916) 2023-10-25T00:21:10.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0125613640 1 -- 172.21.15.107:0/2550276889 learned_addr learned my addr 172.21.15.107:0/2550276889 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:10.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f011ffff640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f012014c9f0 0x7f012014bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0124e12640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f010c003980 0x7f01200bf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0125613640 1 -- 172.21.15.107:0/2550276889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f012014c9f0 msgr2=0x7f012014bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0125613640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f012014c9f0 0x7f012014bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0125613640 1 -- 172.21.15.107:0/2550276889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f010c003980 msgr2=0x7f01200bf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:10.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0125613640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f010c003980 0x7f01200bf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:10.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0125613640 1 -- 172.21.15.107:0/2550276889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01201f13a0 con 0x7f01201dec60 2023-10-25T00:21:10.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f011ffff640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f012014c9f0 0x7f012014bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:21:10.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0125613640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01201dec60 0x7f012014c330 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f0120070360 tx=0x7f011800e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:10.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f011d7fa640 1 -- 172.21.15.107:0/2550276889 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118014480 con 0x7f01201dec60 2023-10-25T00:21:10.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f011d7fa640 1 -- 172.21.15.107:0/2550276889 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01180145e0 con 0x7f01201dec60 2023-10-25T00:21:10.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 -- 172.21.15.107:0/2550276889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01201f18d0 con 0x7f01201dec60 2023-10-25T00:21:10.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f011d7fa640 1 -- 172.21.15.107:0/2550276889 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118014990 con 0x7f01201dec60 2023-10-25T00:21:10.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.989+0000 7f0127076640 1 -- 172.21.15.107:0/2550276889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0120151db0 con 0x7f01201dec60 2023-10-25T00:21:10.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.993+0000 7f0127076640 1 -- 172.21.15.107:0/2550276889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01200c24a0 con 0x7f01201dec60 2023-10-25T00:21:10.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.993+0000 7f011d7fa640 1 -- 172.21.15.107:0/2550276889 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f01180114f0 con 0x7f01201dec60 2023-10-25T00:21:10.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.993+0000 7f011d7fa640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0100065e90 0x7f0100068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:10.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.993+0000 7f011d7fa640 1 -- 172.21.15.107:0/2550276889 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7f0118084030 con 0x7f01201dec60 2023-10-25T00:21:10.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.993+0000 7f0124e12640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0100065e90 0x7f0100068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:10.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.993+0000 7f0124e12640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0100065e90 0x7f0100068350 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f01100af000 tx=0x7f0110008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:10.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:09.997+0000 7f011d7fa640 1 -- 172.21.15.107:0/2550276889 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f01180b5040 con 0x7f01201dec60 2023-10-25T00:21:10.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.185+0000 7f0127076640 1 -- 172.21.15.107:0/2550276889 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01201522d0 con 0x7f01201dec60 2023-10-25T00:21:10.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.185+0000 7f011d7fa640 1 -- 172.21.15.107:0/2550276889 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5765 (secure 0 0 0) 0x7f01201522d0 con 0x7f01201dec60 2023-10-25T00:21:10.190 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:10.191 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:10.191 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 220 2023-10-25T00:21:10.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 -- 172.21.15.107:0/2550276889 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0100065e90 msgr2=0x7f0100068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:10.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0100065e90 0x7f0100068350 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f01100af000 tx=0x7f0110008040 comp rx=0 tx=0).stop 2023-10-25T00:21:10.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 -- 172.21.15.107:0/2550276889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01201dec60 msgr2=0x7f012014c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:10.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01201dec60 0x7f012014c330 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f0120070360 tx=0x7f011800e890 comp rx=0 tx=0).stop 2023-10-25T00:21:10.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 -- 172.21.15.107:0/2550276889 shutdown_connections 2023-10-25T00:21:10.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01201dec60 0x7f012014c330 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:10.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0100065e90 0x7f0100068350 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:10.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f012014c9f0 0x7f012014bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:10.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 --2- 172.21.15.107:0/2550276889 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f010c003980 0x7f01200bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:10.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 -- 172.21.15.107:0/2550276889 >> 172.21.15.107:0/2550276889 conn(0x7f01200c2950 msgr2=0x7f01200c0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:10.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 -- 172.21.15.107:0/2550276889 shutdown_connections 2023-10-25T00:21:10.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:10.189+0000 7f0127076640 1 -- 172.21.15.107:0/2550276889 wait complete. 2023-10-25T00:21:10.209 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:21:12.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:12.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.333+0000 7fde555cc640 1 Processor -- start 2023-10-25T00:21:12.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.333+0000 7fde555cc640 1 -- start start 2023-10-25T00:21:12.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde555cc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fde50151d60 0x7fde500a9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:12.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde555cc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fde500c24b0 0x7fde5015f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:12.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde555cc640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde50154840 con 0x7fde5006a780 2023-10-25T00:21:12.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde555cc640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde500aadc0 con 0x7fde500c24b0 2023-10-25T00:21:12.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde555cc640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde5006afe0 con 0x7fde50151d60 2023-10-25T00:21:12.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fde500c24b0 0x7fde5015f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:12.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4e7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fde5006a780 0x7fde50154d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:45552/0 (socket says 172.21.15.107:45552) 2023-10-25T00:21:12.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fde500c24b0 0x7fde5015f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32918/0 (socket says 172.21.15.107:32918) 2023-10-25T00:21:12.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4e7fc640 1 -- 172.21.15.107:0/3091491549 learned_addr learned my addr 172.21.15.107:0/3091491549 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:12.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4effd640 1 --2- 172.21.15.107:0/3091491549 >> v2:172.21.15.110:3300/0 conn(0x7fde50151d60 0x7fde500a9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:12.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4f7fe640 1 -- 172.21.15.107:0/3091491549 >> v2:172.21.15.110:3300/0 conn(0x7fde50151d60 msgr2=0x7fde500a9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:12.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4f7fe640 1 --2- 172.21.15.107:0/3091491549 >> v2:172.21.15.110:3300/0 conn(0x7fde50151d60 0x7fde500a9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:12.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4f7fe640 1 -- 172.21.15.107:0/3091491549 >> v1:172.21.15.107:6789/0 conn(0x7fde5006a780 legacy=0x7fde50154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:12.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4f7fe640 1 -- 172.21.15.107:0/3091491549 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde500aa640 con 0x7fde500c24b0 2023-10-25T00:21:12.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4dffb640 1 -- 172.21.15.107:0/3091491549 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190148242 0 0) 0x7fde50154840 con 0x7fde5006a780 2023-10-25T00:21:12.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4f7fe640 1 --2- 172.21.15.107:0/3091491549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde500c24b0 0x7fde5015f1c0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fde5006d3e0 tx=0x7fde4001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5025611e4d15ec server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:12.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4dffb640 1 -- 172.21.15.107:0/3091491549 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde400ab600 con 0x7fde500c24b0 2023-10-25T00:21:12.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4dffb640 1 -- 172.21.15.107:0/3091491549 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde400a7050 con 0x7fde500c24b0 2023-10-25T00:21:12.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.337+0000 7fde4dffb640 1 -- 172.21.15.107:0/3091491549 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde400ab980 con 0x7fde500c24b0 2023-10-25T00:21:12.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 -- 172.21.15.107:0/3091491549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde500c24b0 msgr2=0x7fde5015f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:12.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 --2- 172.21.15.107:0/3091491549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde500c24b0 0x7fde5015f1c0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fde5006d3e0 tx=0x7fde4001c460 comp rx=0 tx=0).stop 2023-10-25T00:21:12.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 -- 172.21.15.107:0/3091491549 shutdown_connections 2023-10-25T00:21:12.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 --2- 172.21.15.107:0/3091491549 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde500c24b0 0x7fde5015f1c0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:12.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 --2- 172.21.15.107:0/3091491549 >> v2:172.21.15.110:3300/0 conn(0x7fde50151d60 0x7fde500a9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:12.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 -- 172.21.15.107:0/3091491549 >> 172.21.15.107:0/3091491549 conn(0x7fde500c2950 msgr2=0x7fde500c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:12.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 -- 172.21.15.107:0/3091491549 shutdown_connections 2023-10-25T00:21:12.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 -- 172.21.15.107:0/3091491549 wait complete. 2023-10-25T00:21:12.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 Processor -- start 2023-10-25T00:21:12.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 -- start start 2023-10-25T00:21:12.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fde500c24b0 0x7fde501578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:12.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde50151d60 0x7fde501f0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:12.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde50144de0 0x7fde50154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:12.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5014b520 con 0x7fde50151d60 2023-10-25T00:21:12.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde501f0bd0 con 0x7fde50144de0 2023-10-25T00:21:12.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde501f0eb0 con 0x7fde500c24b0 2023-10-25T00:21:12.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde50151d60 0x7fde501f0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:12.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4e7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde50151d60 0x7fde501f0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:32924/0 (socket says 172.21.15.107:32924) 2023-10-25T00:21:12.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4e7fc640 1 -- 172.21.15.107:0/2932004109 learned_addr learned my addr 172.21.15.107:0/2932004109 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:12.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4f7fe640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde50144de0 0x7fde50154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:12.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4effd640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fde500c24b0 0x7fde501578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:12.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4e7fc640 1 -- 172.21.15.107:0/2932004109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fde500c24b0 msgr2=0x7fde501578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:12.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4e7fc640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fde500c24b0 0x7fde501578b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:12.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4e7fc640 1 -- 172.21.15.107:0/2932004109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde50144de0 msgr2=0x7fde50154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:12.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4e7fc640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde50144de0 0x7fde50154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:12.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4e7fc640 1 -- 172.21.15.107:0/2932004109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde40009420 con 0x7fde50151d60 2023-10-25T00:21:12.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4f7fe640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde50144de0 0x7fde50154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:21:12.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde4e7fc640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde50151d60 0x7fde501f0310 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fde3800c480 tx=0x7fde3800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:12.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde477fe640 1 -- 172.21.15.107:0/2932004109 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde380074e0 con 0x7fde50151d60 2023-10-25T00:21:12.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde477fe640 1 -- 172.21.15.107:0/2932004109 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde38007660 con 0x7fde50151d60 2023-10-25T00:21:12.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 -- 172.21.15.107:0/2932004109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde50148560 con 0x7fde50151d60 2023-10-25T00:21:12.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde477fe640 1 -- 172.21.15.107:0/2932004109 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38004a30 con 0x7fde50151d60 2023-10-25T00:21:12.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.341+0000 7fde555cc640 1 -- 172.21.15.107:0/2932004109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde501f0850 con 0x7fde50151d60 2023-10-25T00:21:12.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.345+0000 7fde555cc640 1 -- 172.21.15.107:0/2932004109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde50157df0 con 0x7fde50151d60 2023-10-25T00:21:12.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.345+0000 7fde477fe640 1 -- 172.21.15.107:0/2932004109 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fde38004b90 con 0x7fde50151d60 2023-10-25T00:21:12.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.345+0000 7fde477fe640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fde30065e90 0x7fde30068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:12.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.345+0000 7fde477fe640 1 -- 172.21.15.107:0/2932004109 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7fde38083e80 con 0x7fde50151d60 2023-10-25T00:21:12.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.345+0000 7fde4effd640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fde30065e90 0x7fde30068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:12.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.345+0000 7fde4effd640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fde30065e90 0x7fde30068350 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fde48003fd0 tx=0x7fde480039a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:12.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.349+0000 7fde477fe640 1 -- 172.21.15.107:0/2932004109 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fde38053e00 con 0x7fde50151d60 2023-10-25T00:21:12.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.533+0000 7fde555cc640 1 -- 172.21.15.107:0/2932004109 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde50148a10 con 0x7fde50151d60 2023-10-25T00:21:12.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde477fe640 1 -- 172.21.15.107:0/2932004109 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5765 (secure 0 0 0) 0x7fde38018020 con 0x7fde50151d60 2023-10-25T00:21:12.539 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:12.539 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:12.539 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 220 2023-10-25T00:21:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 -- 172.21.15.107:0/2932004109 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fde30065e90 msgr2=0x7fde30068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fde30065e90 0x7fde30068350 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fde48003fd0 tx=0x7fde480039a0 comp rx=0 tx=0).stop 2023-10-25T00:21:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 -- 172.21.15.107:0/2932004109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde50151d60 msgr2=0x7fde501f0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde50151d60 0x7fde501f0310 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fde3800c480 tx=0x7fde3800c940 comp rx=0 tx=0).stop 2023-10-25T00:21:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 -- 172.21.15.107:0/2932004109 shutdown_connections 2023-10-25T00:21:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fde50144de0 0x7fde50154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde50151d60 0x7fde501f0310 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fde30065e90 0x7fde30068350 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 --2- 172.21.15.107:0/2932004109 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fde500c24b0 0x7fde501578b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 -- 172.21.15.107:0/2932004109 >> 172.21.15.107:0/2932004109 conn(0x7fde500c2950 msgr2=0x7fde500c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 -- 172.21.15.107:0/2932004109 shutdown_connections 2023-10-25T00:21:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:12.537+0000 7fde555cc640 1 -- 172.21.15.107:0/2932004109 wait complete. 2023-10-25T00:21:12.558 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:21:14.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:14.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f313dcc0640 1 Processor -- start 2023-10-25T00:21:14.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f313dcc0640 1 -- start start 2023-10-25T00:21:14.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f313dcc0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3138065bc0 0x7f3138158970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:14.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f313dcc0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f313806a120 0x7f31380aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:14.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f313dcc0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f313806cea0 con 0x7f3138069d40 2023-10-25T00:21:14.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f313dcc0640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31380ac650 con 0x7f313806a120 2023-10-25T00:21:14.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f313dcc0640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31380a9e00 con 0x7f3138065bc0 2023-10-25T00:21:14.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f3137fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f313806a120 0x7f31380aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:14.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f31377fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f3138069d40 0x7f31380a94b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:54136/0 (socket says 172.21.15.107:54136) 2023-10-25T00:21:14.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f31377fe640 1 -- 172.21.15.107:0/3296543796 learned_addr learned my addr 172.21.15.107:0/3296543796 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:14.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f3136ffd640 1 --2- 172.21.15.107:0/3296543796 >> v2:172.21.15.110:3300/0 conn(0x7f3138065bc0 0x7f3138158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:14.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f3137fff640 1 -- 172.21.15.107:0/3296543796 >> v2:172.21.15.110:3300/0 conn(0x7f3138065bc0 msgr2=0x7f3138158970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:14.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f3137fff640 1 --2- 172.21.15.107:0/3296543796 >> v2:172.21.15.110:3300/0 conn(0x7f3138065bc0 0x7f3138158970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:14.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f3137fff640 1 -- 172.21.15.107:0/3296543796 >> v1:172.21.15.107:6789/0 conn(0x7f3138069d40 legacy=0x7f31380a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:14.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f31367fc640 1 -- 172.21.15.107:0/3296543796 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105783815 0 0) 0x7f313806cea0 con 0x7f3138069d40 2023-10-25T00:21:14.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.689+0000 7f3137fff640 1 -- 172.21.15.107:0/3296543796 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31381584e0 con 0x7f313806a120 2023-10-25T00:21:14.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f3137fff640 1 --2- 172.21.15.107:0/3296543796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f313806a120 0x7f31380aaef0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f3138066170 tx=0x7f312801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac5730546ef6d34 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:14.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f31367fc640 1 -- 172.21.15.107:0/3296543796 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31280ab600 con 0x7f313806a120 2023-10-25T00:21:14.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f31367fc640 1 -- 172.21.15.107:0/3296543796 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31280a7050 con 0x7f313806a120 2023-10-25T00:21:14.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f31367fc640 1 -- 172.21.15.107:0/3296543796 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31280ab980 con 0x7f313806a120 2023-10-25T00:21:14.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f313dcc0640 1 -- 172.21.15.107:0/3296543796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f313806a120 msgr2=0x7f31380aaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:14.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f313dcc0640 1 --2- 172.21.15.107:0/3296543796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f313806a120 0x7f31380aaef0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f3138066170 tx=0x7f312801c3b0 comp rx=0 tx=0).stop 2023-10-25T00:21:14.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f313dcc0640 1 -- 172.21.15.107:0/3296543796 shutdown_connections 2023-10-25T00:21:14.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f313dcc0640 1 --2- 172.21.15.107:0/3296543796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f313806a120 0x7f31380aaef0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:14.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f313dcc0640 1 --2- 172.21.15.107:0/3296543796 >> v2:172.21.15.110:3300/0 conn(0x7f3138065bc0 0x7f3138158970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:14.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f313dcc0640 1 -- 172.21.15.107:0/3296543796 >> 172.21.15.107:0/3296543796 conn(0x7f31380c65c0 msgr2=0x7f31380c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:14.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f313dcc0640 1 -- 172.21.15.107:0/3296543796 shutdown_connections 2023-10-25T00:21:14.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.693+0000 7f313dcc0640 1 -- 172.21.15.107:0/3296543796 wait complete. 2023-10-25T00:21:14.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f313dcc0640 1 Processor -- start 2023-10-25T00:21:14.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f313dcc0640 1 -- start start 2023-10-25T00:21:14.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f313dcc0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3138065bc0 0x7f31381ee260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:14.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f313dcc0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3138069d40 0x7f3138150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:14.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f313dcc0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f313806a120 0x7f31381ed790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:14.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f313dcc0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313815ef30 con 0x7f3138065bc0 2023-10-25T00:21:14.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f313dcc0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f313806fe80 con 0x7f3138069d40 2023-10-25T00:21:14.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f313dcc0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31381ee7a0 con 0x7f313806a120 2023-10-25T00:21:14.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f31377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3138065bc0 0x7f31381ee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:14.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f31377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3138065bc0 0x7f31381ee260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59558/0 (socket says 172.21.15.107:59558) 2023-10-25T00:21:14.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f31377fe640 1 -- 172.21.15.107:0/3746079530 learned_addr learned my addr 172.21.15.107:0/3746079530 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:14.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f3136ffd640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3138069d40 0x7f3138150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:14.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f3137fff640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f313806a120 0x7f31381ed790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:14.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f31377fe640 1 -- 172.21.15.107:0/3746079530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f313806a120 msgr2=0x7f31381ed790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:14.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f31377fe640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f313806a120 0x7f31381ed790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:14.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f31377fe640 1 -- 172.21.15.107:0/3746079530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3138069d40 msgr2=0x7f3138150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:14.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f31377fe640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3138069d40 0x7f3138150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:14.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f31377fe640 1 -- 172.21.15.107:0/3746079530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3128009420 con 0x7f3138065bc0 2023-10-25T00:21:14.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f31377fe640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3138065bc0 0x7f31381ee260 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f313000c890 tx=0x7f313000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:14.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f312ffff640 1 -- 172.21.15.107:0/3746079530 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313000a640 con 0x7f3138065bc0 2023-10-25T00:21:14.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f312ffff640 1 -- 172.21.15.107:0/3746079530 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313000a7c0 con 0x7f3138065bc0 2023-10-25T00:21:14.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f313dcc0640 1 -- 172.21.15.107:0/3746079530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313806f5e0 con 0x7f3138065bc0 2023-10-25T00:21:14.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f312ffff640 1 -- 172.21.15.107:0/3746079530 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313000ab70 con 0x7f3138065bc0 2023-10-25T00:21:14.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f313dcc0640 1 -- 172.21.15.107:0/3746079530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313806fba0 con 0x7f3138065bc0 2023-10-25T00:21:14.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f313dcc0640 1 -- 172.21.15.107:0/3746079530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31380746e0 con 0x7f3138065bc0 2023-10-25T00:21:14.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.697+0000 7f312ffff640 1 -- 172.21.15.107:0/3746079530 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f313000acd0 con 0x7f3138065bc0 2023-10-25T00:21:14.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.701+0000 7f312ffff640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3118065e60 0x7f3118068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:14.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.701+0000 7f312ffff640 1 -- 172.21.15.107:0/3746079530 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7f3130084220 con 0x7f3138065bc0 2023-10-25T00:21:14.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.701+0000 7f3136ffd640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3118065e60 0x7f3118068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:14.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.701+0000 7f3136ffd640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3118065e60 0x7f3118068320 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f31380abea0 tx=0x7f3120008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:14.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.705+0000 7f312ffff640 1 -- 172.21.15.107:0/3746079530 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3130050710 con 0x7f3138065bc0 2023-10-25T00:21:14.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.893+0000 7f313dcc0640 1 -- 172.21.15.107:0/3746079530 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f313806acc0 con 0x7f3138065bc0 2023-10-25T00:21:14.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.893+0000 7f312ffff640 1 -- 172.21.15.107:0/3746079530 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5765 (secure 0 0 0) 0x7f3130018090 con 0x7f3138065bc0 2023-10-25T00:21:14.897 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:14.897 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:14.897 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 220 2023-10-25T00:21:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.893+0000 7f313dcc0640 1 -- 172.21.15.107:0/3746079530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3118065e60 msgr2=0x7f3118068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.893+0000 7f313dcc0640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3118065e60 0x7f3118068320 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f31380abea0 tx=0x7f3120008040 comp rx=0 tx=0).stop 2023-10-25T00:21:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.897+0000 7f313dcc0640 1 -- 172.21.15.107:0/3746079530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3138065bc0 msgr2=0x7f31381ee260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.897+0000 7f313dcc0640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3138065bc0 0x7f31381ee260 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f313000c890 tx=0x7f313000cd50 comp rx=0 tx=0).stop 2023-10-25T00:21:14.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.897+0000 7f313dcc0640 1 -- 172.21.15.107:0/3746079530 shutdown_connections 2023-10-25T00:21:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.897+0000 7f313dcc0640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3118065e60 0x7f3118068320 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.897+0000 7f313dcc0640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f313806a120 0x7f31381ed790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:14.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.897+0000 7f313dcc0640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3138069d40 0x7f3138150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.897+0000 7f313dcc0640 1 --2- 172.21.15.107:0/3746079530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3138065bc0 0x7f31381ee260 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.897+0000 7f313dcc0640 1 -- 172.21.15.107:0/3746079530 >> 172.21.15.107:0/3746079530 conn(0x7f31380c65c0 msgr2=0x7f3138156430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.897+0000 7f313dcc0640 1 -- 172.21.15.107:0/3746079530 shutdown_connections 2023-10-25T00:21:14.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:14.897+0000 7f313dcc0640 1 -- 172.21.15.107:0/3746079530 wait complete. 2023-10-25T00:21:14.916 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:21:16.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:17.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.045+0000 7fae5e4ae640 1 Processor -- start 2023-10-25T00:21:17.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.045+0000 7fae5e4ae640 1 -- start start 2023-10-25T00:21:17.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae5e4ae640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae580720a0 0x7fae58072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:17.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae5e4ae640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae580abaa0 0x7fae5806b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:17.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae5e4ae640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fae5806bd40 0x7fae580c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:17.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae5e4ae640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae580c5bb0 con 0x7fae580abaa0 2023-10-25T00:21:17.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae5e4ae640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae580c15f0 con 0x7fae5806bd40 2023-10-25T00:21:17.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae5e4ae640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae580c2600 con 0x7fae580720a0 2023-10-25T00:21:17.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae580abaa0 0x7fae5806b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:17.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae577fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fae580abaa0 0x7fae5806b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59570/0 (socket says 172.21.15.107:59570) 2023-10-25T00:21:17.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae577fe640 1 -- 172.21.15.107:0/556112301 learned_addr learned my addr 172.21.15.107:0/556112301 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:17.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae57fff640 1 --2- 172.21.15.107:0/556112301 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae580720a0 0x7fae58072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:17.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae5ca4b640 1 --2- 172.21.15.107:0/556112301 >> v2:172.21.15.110:3300/0 conn(0x7fae5806bd40 0x7fae580c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:17.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae577fe640 1 -- 172.21.15.107:0/556112301 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae580720a0 msgr2=0x7fae58072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:17.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae577fe640 1 --2- 172.21.15.107:0/556112301 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae580720a0 0x7fae58072480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae577fe640 1 -- 172.21.15.107:0/556112301 >> v2:172.21.15.110:3300/0 conn(0x7fae5806bd40 msgr2=0x7fae580c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae577fe640 1 --2- 172.21.15.107:0/556112301 >> v2:172.21.15.110:3300/0 conn(0x7fae5806bd40 0x7fae580c3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae577fe640 1 -- 172.21.15.107:0/556112301 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae580729c0 con 0x7fae580abaa0 2023-10-25T00:21:17.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae5ca4b640 1 --2- 172.21.15.107:0/556112301 >> v2:172.21.15.110:3300/0 conn(0x7fae5806bd40 0x7fae580c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:21:17.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae577fe640 1 --2- 172.21.15.107:0/556112301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae580abaa0 0x7fae5806b800 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fae58066b00 tx=0x7fae4c01d840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5787a300b5a001ac server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:17.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae56ffd640 1 -- 172.21.15.107:0/556112301 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c0a6720 con 0x7fae580abaa0 2023-10-25T00:21:17.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae56ffd640 1 -- 172.21.15.107:0/556112301 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4c0a6880 con 0x7fae580abaa0 2023-10-25T00:21:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.049+0000 7fae56ffd640 1 -- 172.21.15.107:0/556112301 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c0a6c10 con 0x7fae580abaa0 2023-10-25T00:21:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- 172.21.15.107:0/556112301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae580abaa0 msgr2=0x7fae5806b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 --2- 172.21.15.107:0/556112301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae580abaa0 0x7fae5806b800 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fae58066b00 tx=0x7fae4c01d840 comp rx=0 tx=0).stop 2023-10-25T00:21:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- 172.21.15.107:0/556112301 shutdown_connections 2023-10-25T00:21:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 --2- 172.21.15.107:0/556112301 >> v2:172.21.15.110:3300/0 conn(0x7fae5806bd40 0x7fae580c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 --2- 172.21.15.107:0/556112301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae580abaa0 0x7fae5806b800 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 --2- 172.21.15.107:0/556112301 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae580720a0 0x7fae58072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- 172.21.15.107:0/556112301 >> 172.21.15.107:0/556112301 conn(0x7fae580c5d50 msgr2=0x7fae580c6160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:17.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- 172.21.15.107:0/556112301 shutdown_connections 2023-10-25T00:21:17.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- 172.21.15.107:0/556112301 wait complete. 2023-10-25T00:21:17.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 Processor -- start 2023-10-25T00:21:17.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- start start 2023-10-25T00:21:17.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae5806bd40 0x7fae580bef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:17.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae580720a0 0x7fae581eb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:17.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae580abaa0 0x7fae581eb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae580bc380 con 0x7fae5806bd40 2023-10-25T00:21:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae581ee710 con 0x7fae580720a0 2023-10-25T00:21:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae581ee9f0 con 0x7fae580abaa0 2023-10-25T00:21:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae5806bd40 0x7fae580bef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:17.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae57fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae5806bd40 0x7fae580bef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59582/0 (socket says 172.21.15.107:59582) 2023-10-25T00:21:17.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae57fff640 1 -- 172.21.15.107:0/3378589641 learned_addr learned my addr 172.21.15.107:0/3378589641 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:17.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5ca4b640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae580abaa0 0x7fae581eb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:17.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae577fe640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae580720a0 0x7fae581eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:17.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae57fff640 1 -- 172.21.15.107:0/3378589641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae580abaa0 msgr2=0x7fae581eb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:17.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae57fff640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae580abaa0 0x7fae581eb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae57fff640 1 -- 172.21.15.107:0/3378589641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae580720a0 msgr2=0x7fae581eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:17.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae57fff640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae580720a0 0x7fae581eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae57fff640 1 -- 172.21.15.107:0/3378589641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c00a670 con 0x7fae5806bd40 2023-10-25T00:21:17.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae57fff640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae5806bd40 0x7fae580bef60 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fae4800d800 tx=0x7fae4800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:17.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae54ff9640 1 -- 172.21.15.107:0/3378589641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4800b6c0 con 0x7fae5806bd40 2023-10-25T00:21:17.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae54ff9640 1 -- 172.21.15.107:0/3378589641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4800b820 con 0x7fae5806bd40 2023-10-25T00:21:17.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- 172.21.15.107:0/3378589641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae581eecd0 con 0x7fae5806bd40 2023-10-25T00:21:17.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae54ff9640 1 -- 172.21.15.107:0/3378589641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4800bb90 con 0x7fae5806bd40 2023-10-25T00:21:17.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- 172.21.15.107:0/3378589641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae580c0920 con 0x7fae5806bd40 2023-10-25T00:21:17.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.053+0000 7fae5e4ae640 1 -- 172.21.15.107:0/3378589641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5804e0c0 con 0x7fae5806bd40 2023-10-25T00:21:17.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.057+0000 7fae54ff9640 1 -- 172.21.15.107:0/3378589641 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fae4800bcf0 con 0x7fae5806bd40 2023-10-25T00:21:17.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.057+0000 7fae54ff9640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae44065e60 0x7fae44068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:17.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.057+0000 7fae54ff9640 1 -- 172.21.15.107:0/3378589641 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 10573+0+0 (secure 0 0 0) 0x7fae480859e0 con 0x7fae5806bd40 2023-10-25T00:21:17.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.057+0000 7fae577fe640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae44065e60 0x7fae44068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:17.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.057+0000 7fae577fe640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae44065e60 0x7fae44068320 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0b0000 tx=0x7fae4c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:17.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.061+0000 7fae54ff9640 1 -- 172.21.15.107:0/3378589641 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fae48051f50 con 0x7fae5806bd40 2023-10-25T00:21:17.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.245+0000 7fae5e4ae640 1 -- 172.21.15.107:0/3378589641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae580c0110 con 0x7fae5806bd40 2023-10-25T00:21:17.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae54ff9640 1 -- 172.21.15.107:0/3378589641 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+5765 (secure 0 0 0) 0x7fae48055570 con 0x7fae5806bd40 2023-10-25T00:21:17.251 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:17.252 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":7483,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:19:47.167046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8712},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8712":{"gid":8712,"name":"c","rank":0,"incarnation":211,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/657626630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":657626630},{"type":"v1","addr":"172.21.15.107:6837","nonce":657626630}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:17.252 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 220 2023-10-25T00:21:17.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 -- 172.21.15.107:0/3378589641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae44065e60 msgr2=0x7fae44068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:17.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae44065e60 0x7fae44068320 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fae4c0b0000 tx=0x7fae4c008040 comp rx=0 tx=0).stop 2023-10-25T00:21:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 -- 172.21.15.107:0/3378589641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae5806bd40 msgr2=0x7fae580bef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae5806bd40 0x7fae580bef60 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fae4800d800 tx=0x7fae4800dcc0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 -- 172.21.15.107:0/3378589641 shutdown_connections 2023-10-25T00:21:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fae44065e60 0x7fae44068320 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fae580abaa0 0x7fae581eb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae580720a0 0x7fae581eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 --2- 172.21.15.107:0/3378589641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae5806bd40 0x7fae580bef60 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 -- 172.21.15.107:0/3378589641 >> 172.21.15.107:0/3378589641 conn(0x7fae580c5d50 msgr2=0x7fae58151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 -- 172.21.15.107:0/3378589641 shutdown_connections 2023-10-25T00:21:17.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:17.249+0000 7fae5e4ae640 1 -- 172.21.15.107:0/3378589641 wait complete. 2023-10-25T00:21:17.270 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:21:19.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:19.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.397+0000 7fe1e3aa3640 1 Processor -- start 2023-10-25T00:21:19.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.397+0000 7fe1e3aa3640 1 -- start start 2023-10-25T00:21:19.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e3aa3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe1dc152040 0x7fe1dc0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e3aa3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe1dc0c8100 0x7fe1dc0abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e3aa3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc155680 con 0x7fe1dc06a4b0 2023-10-25T00:21:19.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e3aa3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc154a70 con 0x7fe1dc0c8100 2023-10-25T00:21:19.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e3aa3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc154d50 con 0x7fe1dc152040 2023-10-25T00:21:19.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e103e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe1dc0c8100 0x7fe1dc0abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e103e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe1dc0c8100 0x7fe1dc0abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59592/0 (socket says 172.21.15.107:59592) 2023-10-25T00:21:19.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e103e640 1 -- 172.21.15.107:0/2235993098 learned_addr learned my addr 172.21.15.107:0/2235993098 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:19.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e183f640 1 --2- 172.21.15.107:0/2235993098 >> v2:172.21.15.110:3300/0 conn(0x7fe1dc152040 0x7fe1dc0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e103e640 1 -- 172.21.15.107:0/2235993098 >> v2:172.21.15.110:3300/0 conn(0x7fe1dc152040 msgr2=0x7fe1dc0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:19.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e103e640 1 --2- 172.21.15.107:0/2235993098 >> v2:172.21.15.110:3300/0 conn(0x7fe1dc152040 0x7fe1dc0c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e103e640 1 -- 172.21.15.107:0/2235993098 >> v1:172.21.15.110:6789/0 conn(0x7fe1dc06a4b0 legacy=0x7fe1dc0ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:19.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e103e640 1 -- 172.21.15.107:0/2235993098 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc155030 con 0x7fe1dc0c8100 2023-10-25T00:21:19.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e103e640 1 --2- 172.21.15.107:0/2235993098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1dc0c8100 0x7fe1dc0abd00 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc067ba0 tx=0x7fe1d801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcaf5b60c675dbc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:19.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e083d640 1 -- 172.21.15.107:0/2235993098 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80a56d0 con 0x7fe1dc0c8100 2023-10-25T00:21:19.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e083d640 1 -- 172.21.15.107:0/2235993098 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d80a5830 con 0x7fe1dc0c8100 2023-10-25T00:21:19.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.401+0000 7fe1e083d640 1 -- 172.21.15.107:0/2235993098 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80a5bc0 con 0x7fe1dc0c8100 2023-10-25T00:21:19.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/2235993098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1dc0c8100 msgr2=0x7fe1dc0abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:19.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 --2- 172.21.15.107:0/2235993098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1dc0c8100 0x7fe1dc0abd00 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc067ba0 tx=0x7fe1d801c900 comp rx=0 tx=0).stop 2023-10-25T00:21:19.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/2235993098 shutdown_connections 2023-10-25T00:21:19.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 --2- 172.21.15.107:0/2235993098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1dc0c8100 0x7fe1dc0abd00 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 --2- 172.21.15.107:0/2235993098 >> v2:172.21.15.110:3300/0 conn(0x7fe1dc152040 0x7fe1dc0c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/2235993098 >> 172.21.15.107:0/2235993098 conn(0x7fe1dc0ca200 msgr2=0x7fe1dc0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:19.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/2235993098 shutdown_connections 2023-10-25T00:21:19.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/2235993098 wait complete. 2023-10-25T00:21:19.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 Processor -- start 2023-10-25T00:21:19.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 -- start start 2023-10-25T00:21:19.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1dc06a4b0 0x7fe1dc144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1dc0c8100 0x7fe1dc153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1dc152040 0x7fe1dc0aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc151910 con 0x7fe1dc0c8100 2023-10-25T00:21:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc1ecfe0 con 0x7fe1dc06a4b0 2023-10-25T00:21:19.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc1ed2c0 con 0x7fe1dc152040 2023-10-25T00:21:19.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e103e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1dc0c8100 0x7fe1dc153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e103e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1dc0c8100 0x7fe1dc153550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59602/0 (socket says 172.21.15.107:59602) 2023-10-25T00:21:19.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e103e640 1 -- 172.21.15.107:0/4189575784 learned_addr learned my addr 172.21.15.107:0/4189575784 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:19.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e183f640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1dc06a4b0 0x7fe1dc144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e2040640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1dc152040 0x7fe1dc0aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e103e640 1 -- 172.21.15.107:0/4189575784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1dc152040 msgr2=0x7fe1dc0aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e103e640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1dc152040 0x7fe1dc0aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e103e640 1 -- 172.21.15.107:0/4189575784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1dc06a4b0 msgr2=0x7fe1dc144f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e103e640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1dc06a4b0 0x7fe1dc144f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e103e640 1 -- 172.21.15.107:0/4189575784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d8009b30 con 0x7fe1dc0c8100 2023-10-25T00:21:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e2040640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1dc152040 0x7fe1dc0aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:21:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e103e640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1dc0c8100 0x7fe1dc153550 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80af040 tx=0x7fe1d80196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:19.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1ce7fc640 1 -- 172.21.15.107:0/4189575784 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80b8670 con 0x7fe1dc0c8100 2023-10-25T00:21:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1ce7fc640 1 -- 172.21.15.107:0/4189575784 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d80b87d0 con 0x7fe1dc0c8100 2023-10-25T00:21:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/4189575784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc1ed5a0 con 0x7fe1dc0c8100 2023-10-25T00:21:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1ce7fc640 1 -- 172.21.15.107:0/4189575784 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80b8b80 con 0x7fe1dc0c8100 2023-10-25T00:21:19.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.405+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/4189575784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc1edb60 con 0x7fe1dc0c8100 2023-10-25T00:21:19.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.409+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/4189575784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc1edd50 con 0x7fe1dc0c8100 2023-10-25T00:21:19.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.409+0000 7fe1ce7fc640 1 -- 172.21.15.107:0/4189575784 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe1d80b8ce0 con 0x7fe1dc0c8100 2023-10-25T00:21:19.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.409+0000 7fe1ce7fc640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1c0065e60 0x7fe1c0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.409+0000 7fe1e183f640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1c0065e60 0x7fe1c0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.409+0000 7fe1ce7fc640 1 -- 172.21.15.107:0/4189575784 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7fe1d811ff70 con 0x7fe1dc0c8100 2023-10-25T00:21:19.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.409+0000 7fe1e183f640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1c0065e60 0x7fe1c0068320 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0aa070 tx=0x7fe1d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:19.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.413+0000 7fe1ce7fc640 1 -- 172.21.15.107:0/4189575784 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe1d8151040 con 0x7fe1dc0c8100 2023-10-25T00:21:19.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/4189575784 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1dc0746e0 con 0x7fe1dc0c8100 2023-10-25T00:21:19.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1ce7fc640 1 -- 172.21.15.107:0/4189575784 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+5006 (secure 0 0 0) 0x7fe1d80ec520 con 0x7fe1dc0c8100 2023-10-25T00:21:19.608 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:19.608 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:19.168298+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:19.609 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 221 2023-10-25T00:21:19.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/4189575784 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1c0065e60 msgr2=0x7fe1c0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1e3aa3640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1c0065e60 0x7fe1c0068320 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc0aa070 tx=0x7fe1d4008040 comp rx=0 tx=0).stop 2023-10-25T00:21:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/4189575784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1dc0c8100 msgr2=0x7fe1dc153550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1e3aa3640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1dc0c8100 0x7fe1dc153550 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80af040 tx=0x7fe1d80196a0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/4189575784 shutdown_connections 2023-10-25T00:21:19.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1e3aa3640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe1c0065e60 0x7fe1c0068320 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1e3aa3640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1dc152040 0x7fe1dc0aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1e3aa3640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1dc0c8100 0x7fe1dc153550 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1e3aa3640 1 --2- 172.21.15.107:0/4189575784 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1dc06a4b0 0x7fe1dc144f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.605+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/4189575784 >> 172.21.15.107:0/4189575784 conn(0x7fe1dc0ca200 msgr2=0x7fe1dc1527d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:19.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.609+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/4189575784 shutdown_connections 2023-10-25T00:21:19.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.609+0000 7fe1e3aa3640 1 -- 172.21.15.107:0/4189575784 wait complete. 2023-10-25T00:21:19.627 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2023-10-25T00:21:19.628 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2023-10-25T00:21:19.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:19.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.745+0000 7fb978a87640 1 Processor -- start 2023-10-25T00:21:19.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.745+0000 7fb978a87640 1 -- start start 2023-10-25T00:21:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb978a87640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb97414b600 0x7fb974151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb978a87640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb97406af70 0x7fb97406c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb978a87640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9740c1860 con 0x7fb97406a230 2023-10-25T00:21:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb978a87640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb974154d50 con 0x7fb97406af70 2023-10-25T00:21:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb978a87640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9741524f0 con 0x7fb97414b600 2023-10-25T00:21:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb96bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb97406af70 0x7fb97406c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb96bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb97406af70 0x7fb97406c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59604/0 (socket says 172.21.15.107:59604) 2023-10-25T00:21:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb96bfff640 1 -- 172.21.15.107:0/3046668270 learned_addr learned my addr 172.21.15.107:0/3046668270 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb97259c640 1 --2- 172.21.15.107:0/3046668270 >> v2:172.21.15.110:3300/0 conn(0x7fb97414b600 0x7fb974151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb96bfff640 1 -- 172.21.15.107:0/3046668270 >> v2:172.21.15.110:3300/0 conn(0x7fb97414b600 msgr2=0x7fb974151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb96bfff640 1 --2- 172.21.15.107:0/3046668270 >> v2:172.21.15.110:3300/0 conn(0x7fb97414b600 0x7fb974151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb96bfff640 1 -- 172.21.15.107:0/3046668270 >> v1:172.21.15.110:6789/0 conn(0x7fb97406a230 legacy=0x7fb9741551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:19.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb96bfff640 1 -- 172.21.15.107:0/3046668270 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97414b9e0 con 0x7fb97406af70 2023-10-25T00:21:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb96bfff640 1 --2- 172.21.15.107:0/3046668270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb97406af70 0x7fb97406c4c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fb97406f860 tx=0x7fb95c01ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8113c0b9eb56f210 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb971d9b640 1 -- 172.21.15.107:0/3046668270 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95c0a7aa0 con 0x7fb97406af70 2023-10-25T00:21:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb971d9b640 1 -- 172.21.15.107:0/3046668270 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb95c0a7c00 con 0x7fb97406af70 2023-10-25T00:21:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.749+0000 7fb971d9b640 1 -- 172.21.15.107:0/3046668270 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95c0af440 con 0x7fb97406af70 2023-10-25T00:21:19.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 -- 172.21.15.107:0/3046668270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb97406af70 msgr2=0x7fb97406c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 --2- 172.21.15.107:0/3046668270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb97406af70 0x7fb97406c4c0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fb97406f860 tx=0x7fb95c01ead0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 -- 172.21.15.107:0/3046668270 shutdown_connections 2023-10-25T00:21:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 --2- 172.21.15.107:0/3046668270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb97406af70 0x7fb97406c4c0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 --2- 172.21.15.107:0/3046668270 >> v2:172.21.15.110:3300/0 conn(0x7fb97414b600 0x7fb974151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 -- 172.21.15.107:0/3046668270 >> 172.21.15.107:0/3046668270 conn(0x7fb9740c2950 msgr2=0x7fb9740c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:19.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 -- 172.21.15.107:0/3046668270 shutdown_connections 2023-10-25T00:21:19.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 -- 172.21.15.107:0/3046668270 wait complete. 2023-10-25T00:21:19.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 Processor -- start 2023-10-25T00:21:19.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 -- start start 2023-10-25T00:21:19.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb97406a230 0x7fb974142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb97406af70 0x7fb97414a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97414b600 0x7fb97406b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97414ac60 con 0x7fb97406a230 2023-10-25T00:21:19.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9741529e0 con 0x7fb97406af70 2023-10-25T00:21:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb974070590 con 0x7fb97414b600 2023-10-25T00:21:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb97259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb97406a230 0x7fb974142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb97259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb97406a230 0x7fb974142390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59606/0 (socket says 172.21.15.107:59606) 2023-10-25T00:21:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb97259c640 1 -- 172.21.15.107:0/2558813746 learned_addr learned my addr 172.21.15.107:0/2558813746 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:19.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb96bfff640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb97406af70 0x7fb97414a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb972d9d640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97414b600 0x7fb97406b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb97259c640 1 -- 172.21.15.107:0/2558813746 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97414b600 msgr2=0x7fb97406b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb97259c640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97414b600 0x7fb97406b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb97259c640 1 -- 172.21.15.107:0/2558813746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb97406af70 msgr2=0x7fb97414a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:19.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb97259c640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb97406af70 0x7fb97414a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb97259c640 1 -- 172.21.15.107:0/2558813746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c00b770 con 0x7fb97406a230 2023-10-25T00:21:19.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb97259c640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb97406a230 0x7fb974142390 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fb96400d5f0 tx=0x7fb96400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:19.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb96affd640 1 -- 172.21.15.107:0/2558813746 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9640125f0 con 0x7fb97406a230 2023-10-25T00:21:19.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb96affd640 1 -- 172.21.15.107:0/2558813746 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb964012750 con 0x7fb97406a230 2023-10-25T00:21:19.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 -- 172.21.15.107:0/2558813746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9741eda50 con 0x7fb97406a230 2023-10-25T00:21:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb96affd640 1 -- 172.21.15.107:0/2558813746 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb964012ae0 con 0x7fb97406a230 2023-10-25T00:21:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.753+0000 7fb978a87640 1 -- 172.21.15.107:0/2558813746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97406bb30 con 0x7fb97406a230 2023-10-25T00:21:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.757+0000 7fb978a87640 1 -- 172.21.15.107:0/2558813746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9741edf70 con 0x7fb97406a230 2023-10-25T00:21:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.757+0000 7fb96affd640 1 -- 172.21.15.107:0/2558813746 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb964012c40 con 0x7fb97406a230 2023-10-25T00:21:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.757+0000 7fb96affd640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb954065e60 0x7fb954068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:19.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.757+0000 7fb96affd640 1 -- 172.21.15.107:0/2558813746 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7fb964084f00 con 0x7fb97406a230 2023-10-25T00:21:19.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.757+0000 7fb96bfff640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb954065e60 0x7fb954068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:19.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.757+0000 7fb96bfff640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb954065e60 0x7fb954068320 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0b1000 tx=0x7fb95c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:19.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.761+0000 7fb96affd640 1 -- 172.21.15.107:0/2558813746 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb964051390 con 0x7fb97406a230 2023-10-25T00:21:19.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.949+0000 7fb978a87640 1 -- 172.21.15.107:0/2558813746 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb974152b50 con 0x7fb97406a230 2023-10-25T00:21:19.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb96affd640 1 -- 172.21.15.107:0/2558813746 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+5006 (secure 0 0 0) 0x7fb9640549b0 con 0x7fb97406a230 2023-10-25T00:21:19.955 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:19.956 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:19.168298+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:replay","state_seq":1,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:19.956 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 221 2023-10-25T00:21:19.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 -- 172.21.15.107:0/2558813746 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb954065e60 msgr2=0x7fb954068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:19.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb954065e60 0x7fb954068320 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0b1000 tx=0x7fb95c008040 comp rx=0 tx=0).stop 2023-10-25T00:21:19.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 -- 172.21.15.107:0/2558813746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb97406a230 msgr2=0x7fb974142390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:19.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb97406a230 0x7fb974142390 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7fb96400d5f0 tx=0x7fb96400dab0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 -- 172.21.15.107:0/2558813746 shutdown_connections 2023-10-25T00:21:19.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb954065e60 0x7fb954068320 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97414b600 0x7fb97406b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb97406af70 0x7fb97414a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 --2- 172.21.15.107:0/2558813746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb97406a230 0x7fb974142390 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:19.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 -- 172.21.15.107:0/2558813746 >> 172.21.15.107:0/2558813746 conn(0x7fb9740c2950 msgr2=0x7fb9741551e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:19.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 -- 172.21.15.107:0/2558813746 shutdown_connections 2023-10-25T00:21:19.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:19.953+0000 7fb978a87640 1 -- 172.21.15.107:0/2558813746 wait complete. 2023-10-25T00:21:19.975 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 221, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 219, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8802": { "addr": "172.21.15.107:6835/2187059523", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2187059523, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 2187059523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8802, "incarnation": 216, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:20:23.056107+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8802 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 221, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7483": { "addr": "172.21.15.110:6834/47497412", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 47497412, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 47497412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7483, "incarnation": 221, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:21:19.168298+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7483 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2437148", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2437148, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2437148, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 190, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6443, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:21:21.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:22.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.105+0000 7f0f8606e640 1 Processor -- start 2023-10-25T00:21:22.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.105+0000 7f0f8606e640 1 -- start start 2023-10-25T00:21:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f8606e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0f8014b600 0x7f0f80151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f8606e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0f8006af70 0x7f0f8006c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:22.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f8606e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8006a230 0x7f0f801551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f8606e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f800c1860 con 0x7f0f8006af70 2023-10-25T00:21:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f8606e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f80154d50 con 0x7f0f8014b600 2023-10-25T00:21:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f8606e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f801524f0 con 0x7f0f8006a230 2023-10-25T00:21:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0f8006af70 0x7f0f8006c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:22.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0f8006af70 0x7f0f8006c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59622/0 (socket says 172.21.15.107:59622) 2023-10-25T00:21:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7effd640 1 -- 172.21.15.107:0/220227373 learned_addr learned my addr 172.21.15.107:0/220227373 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7f7fe640 1 --2- 172.21.15.107:0/220227373 >> v2:172.21.15.110:3300/0 conn(0x7f0f8014b600 0x7f0f80151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7ffff640 1 --2- 172.21.15.107:0/220227373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8006a230 0x7f0f801551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7effd640 1 -- 172.21.15.107:0/220227373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8006a230 msgr2=0x7f0f801551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7effd640 1 --2- 172.21.15.107:0/220227373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8006a230 0x7f0f801551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7effd640 1 -- 172.21.15.107:0/220227373 >> v2:172.21.15.110:3300/0 conn(0x7f0f8014b600 msgr2=0x7f0f80151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7effd640 1 --2- 172.21.15.107:0/220227373 >> v2:172.21.15.110:3300/0 conn(0x7f0f8014b600 0x7f0f80151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7effd640 1 -- 172.21.15.107:0/220227373 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8014b9e0 con 0x7f0f8006af70 2023-10-25T00:21:22.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7f7fe640 1 --2- 172.21.15.107:0/220227373 >> v2:172.21.15.110:3300/0 conn(0x7f0f8014b600 0x7f0f80151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:21:22.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7effd640 1 --2- 172.21.15.107:0/220227373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f8006af70 0x7f0f8006c4c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f0f8006f860 tx=0x7f0f7401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139d489b279ea393 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:22.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7e7fc640 1 -- 172.21.15.107:0/220227373 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f740ab600 con 0x7f0f8006af70 2023-10-25T00:21:22.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7e7fc640 1 -- 172.21.15.107:0/220227373 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f740ab760 con 0x7f0f8006af70 2023-10-25T00:21:22.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f7e7fc640 1 -- 172.21.15.107:0/220227373 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f740abaf0 con 0x7f0f8006af70 2023-10-25T00:21:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.109+0000 7f0f8606e640 1 -- 172.21.15.107:0/220227373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f8006af70 msgr2=0x7f0f8006c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 --2- 172.21.15.107:0/220227373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f8006af70 0x7f0f8006c4c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f0f8006f860 tx=0x7f0f7401c440 comp rx=0 tx=0).stop 2023-10-25T00:21:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 -- 172.21.15.107:0/220227373 shutdown_connections 2023-10-25T00:21:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 --2- 172.21.15.107:0/220227373 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8006a230 0x7f0f801551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 --2- 172.21.15.107:0/220227373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f8006af70 0x7f0f8006c4c0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 --2- 172.21.15.107:0/220227373 >> v2:172.21.15.110:3300/0 conn(0x7f0f8014b600 0x7f0f80151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 -- 172.21.15.107:0/220227373 >> 172.21.15.107:0/220227373 conn(0x7f0f800c2950 msgr2=0x7f0f800c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:22.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 -- 172.21.15.107:0/220227373 shutdown_connections 2023-10-25T00:21:22.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 -- 172.21.15.107:0/220227373 wait complete. 2023-10-25T00:21:22.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 Processor -- start 2023-10-25T00:21:22.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 -- start start 2023-10-25T00:21:22.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f8006a230 0x7f0f800c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:22.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f8006af70 0x7f0f80142ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:22.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8014b600 0x7f0f800bf670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:22.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f800c5980 con 0x7f0f8006a230 2023-10-25T00:21:22.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f800701b0 con 0x7f0f8006af70 2023-10-25T00:21:22.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f800c3a40 con 0x7f0f8014b600 2023-10-25T00:21:22.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f7f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f8006a230 0x7f0f800c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:22.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f7f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f8006a230 0x7f0f800c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59630/0 (socket says 172.21.15.107:59630) 2023-10-25T00:21:22.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f7f7fe640 1 -- 172.21.15.107:0/2955421641 learned_addr learned my addr 172.21.15.107:0/2955421641 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:22.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f7effd640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f8006af70 0x7f0f80142ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:22.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f7ffff640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8014b600 0x7f0f800bf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:22.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f7f7fe640 1 -- 172.21.15.107:0/2955421641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8014b600 msgr2=0x7f0f800bf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:22.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f7f7fe640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8014b600 0x7f0f800bf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f7f7fe640 1 -- 172.21.15.107:0/2955421641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f8006af70 msgr2=0x7f0f80142ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:22.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f7f7fe640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f8006af70 0x7f0f80142ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f7f7fe640 1 -- 172.21.15.107:0/2955421641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f740094d0 con 0x7f0f8006a230 2023-10-25T00:21:22.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f7f7fe640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f8006a230 0x7f0f800c5440 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c7f0 tx=0x7f0f7000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:22.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f67fff640 1 -- 172.21.15.107:0/2955421641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70007820 con 0x7f0f8006a230 2023-10-25T00:21:22.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f67fff640 1 -- 172.21.15.107:0/2955421641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f70007980 con 0x7f0f8006a230 2023-10-25T00:21:22.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 -- 172.21.15.107:0/2955421641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f801f11a0 con 0x7f0f8006a230 2023-10-25T00:21:22.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f67fff640 1 -- 172.21.15.107:0/2955421641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70007d30 con 0x7f0f8006a230 2023-10-25T00:21:22.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 -- 172.21.15.107:0/2955421641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f80070400 con 0x7f0f8006a230 2023-10-25T00:21:22.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.113+0000 7f0f8606e640 1 -- 172.21.15.107:0/2955421641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f800746e0 con 0x7f0f8006a230 2023-10-25T00:21:22.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.117+0000 7f0f67fff640 1 -- 172.21.15.107:0/2955421641 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0f700113f0 con 0x7f0f8006a230 2023-10-25T00:21:22.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.117+0000 7f0f67fff640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f6c065e90 0x7f0f6c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:22.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.117+0000 7f0f67fff640 1 -- 172.21.15.107:0/2955421641 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7f0f70084d90 con 0x7f0f8006a230 2023-10-25T00:21:22.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.117+0000 7f0f7effd640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f6c065e90 0x7f0f6c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:22.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.117+0000 7f0f7effd640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f6c065e90 0x7f0f6c068350 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0f740af000 tx=0x7f0f740a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:22.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.121+0000 7f0f67fff640 1 -- 172.21.15.107:0/2955421641 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0f70051190 con 0x7f0f8006a230 2023-10-25T00:21:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 -- 172.21.15.107:0/2955421641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f801f0f70 con 0x7f0f8006a230 2023-10-25T00:21:22.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f67fff640 1 -- 172.21.15.107:0/2955421641 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+5007 (secure 0 0 0) 0x7f0f7001f020 con 0x7f0f8006a230 2023-10-25T00:21:22.313 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:22.313 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":223,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:21.329179+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 223 2023-10-25T00:21:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 -- 172.21.15.107:0/2955421641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f6c065e90 msgr2=0x7f0f6c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f6c065e90 0x7f0f6c068350 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0f740af000 tx=0x7f0f740a7000 comp rx=0 tx=0).stop 2023-10-25T00:21:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 -- 172.21.15.107:0/2955421641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f8006a230 msgr2=0x7f0f800c5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f8006a230 0x7f0f800c5440 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c7f0 tx=0x7f0f7000ccb0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 -- 172.21.15.107:0/2955421641 shutdown_connections 2023-10-25T00:21:22.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0f6c065e90 0x7f0f6c068350 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f8014b600 0x7f0f800bf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f8006af70 0x7f0f80142ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 --2- 172.21.15.107:0/2955421641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f8006a230 0x7f0f800c5440 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 -- 172.21.15.107:0/2955421641 >> 172.21.15.107:0/2955421641 conn(0x7f0f800c2950 msgr2=0x7f0f801499f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:22.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 -- 172.21.15.107:0/2955421641 shutdown_connections 2023-10-25T00:21:22.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:22.309+0000 7f0f8606e640 1 -- 172.21.15.107:0/2955421641 wait complete. 2023-10-25T00:21:22.331 INFO:tasks.mds_thrash.fs.[b]:no change 2023-10-25T00:21:23.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:23.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f5681090640 1 Processor -- start 2023-10-25T00:21:23.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f5681090640 1 -- start start 2023-10-25T00:21:23.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f5681090640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f567c06f490 0x7f567c14c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:23.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f5681090640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c155cd0 con 0x7f567c152b30 2023-10-25T00:21:23.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f5681090640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c0c4380 con 0x7f567c14c470 2023-10-25T00:21:23.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f5681090640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c1526e0 con 0x7f567c06f490 2023-10-25T00:21:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f567a59c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f567c06f490 0x7f567c14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f567b59e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f567c152b30 0x7f567c152f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44038/0 (socket says 172.21.15.107:44038) 2023-10-25T00:21:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f567b59e640 1 -- 172.21.15.107:0/2997098500 learned_addr learned my addr 172.21.15.107:0/2997098500 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f5679d9b640 1 -- 172.21.15.107:0/2997098500 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180201969 0 0) 0x7f567c155cd0 con 0x7f567c152b30 2023-10-25T00:21:23.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f5679d9b640 1 -- 172.21.15.107:0/2997098500 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5668003530 con 0x7f567c152b30 2023-10-25T00:21:23.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f5679d9b640 1 -- 172.21.15.107:0/2997098500 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280465373 0 0) 0x7f567c0c4380 con 0x7f567c14c470 2023-10-25T00:21:23.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f5679d9b640 1 -- 172.21.15.107:0/2997098500 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5668003850 con 0x7f567c14c470 2023-10-25T00:21:23.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f567a59c640 1 -- 172.21.15.107:0/2997098500 >> v1:172.21.15.110:6789/0 conn(0x7f567c14c470 legacy=0x7f567c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:23.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f567a59c640 1 -- 172.21.15.107:0/2997098500 >> v1:172.21.15.107:6789/0 conn(0x7f567c152b30 legacy=0x7f567c152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:23.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.825+0000 7f567a59c640 1 -- 172.21.15.107:0/2997098500 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c0c48b0 con 0x7f567c06f490 2023-10-25T00:21:23.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.829+0000 7f567a59c640 1 --2- 172.21.15.107:0/2997098500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f567c06f490 0x7f567c14c850 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f566c0097c0 tx=0x7f566c01c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18898b1479ee0c3f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:23.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.829+0000 7f5679d9b640 1 -- 172.21.15.107:0/2997098500 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c0ab600 con 0x7f567c06f490 2023-10-25T00:21:23.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.829+0000 7f5679d9b640 1 -- 172.21.15.107:0/2997098500 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566c0a7050 con 0x7f567c06f490 2023-10-25T00:21:23.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.829+0000 7f5679d9b640 1 -- 172.21.15.107:0/2997098500 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c0ab960 con 0x7f567c06f490 2023-10-25T00:21:23.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.829+0000 7f5681090640 1 -- 172.21.15.107:0/2997098500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f567c06f490 msgr2=0x7f567c14c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:23.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.829+0000 7f5681090640 1 --2- 172.21.15.107:0/2997098500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f567c06f490 0x7f567c14c850 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f566c0097c0 tx=0x7f566c01c360 comp rx=0 tx=0).stop 2023-10-25T00:21:23.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.829+0000 7f5681090640 1 -- 172.21.15.107:0/2997098500 shutdown_connections 2023-10-25T00:21:23.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.829+0000 7f5681090640 1 --2- 172.21.15.107:0/2997098500 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f567c06f490 0x7f567c14c850 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:23.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.829+0000 7f5681090640 1 -- 172.21.15.107:0/2997098500 >> 172.21.15.107:0/2997098500 conn(0x7f567c070f60 msgr2=0x7f567c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:23.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.829+0000 7f5681090640 1 -- 172.21.15.107:0/2997098500 shutdown_connections 2023-10-25T00:21:23.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.829+0000 7f5681090640 1 -- 172.21.15.107:0/2997098500 wait complete. 2023-10-25T00:21:23.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f5681090640 1 Processor -- start 2023-10-25T00:21:23.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f5681090640 1 -- start start 2023-10-25T00:21:23.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f5681090640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f567c06f490 0x7f567c0ab960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:23.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f5681090640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f567c14c470 0x7f567c1427e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:23.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f5681090640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f567c152b30 0x7f567c141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:23.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f5681090640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c1413f0 con 0x7f567c06f490 2023-10-25T00:21:23.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f5681090640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c0abea0 con 0x7f567c152b30 2023-10-25T00:21:23.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f5681090640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c0adac0 con 0x7f567c14c470 2023-10-25T00:21:23.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f567ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f567c06f490 0x7f567c0ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:23.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f567ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f567c06f490 0x7f567c0ab960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45648/0 (socket says 172.21.15.107:45648) 2023-10-25T00:21:23.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f567ad9d640 1 -- 172.21.15.107:0/449079336 learned_addr learned my addr 172.21.15.107:0/449079336 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:23.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f567b59e640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f567c152b30 0x7f567c141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:23.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f567a59c640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f567c14c470 0x7f567c1427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:23.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f567ad9d640 1 -- 172.21.15.107:0/449079336 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f567c14c470 msgr2=0x7f567c1427e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f567ad9d640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f567c14c470 0x7f567c1427e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f567ad9d640 1 -- 172.21.15.107:0/449079336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f567c152b30 msgr2=0x7f567c141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f567ad9d640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f567c152b30 0x7f567c141880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:23.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f567ad9d640 1 -- 172.21.15.107:0/449079336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c009420 con 0x7f567c06f490 2023-10-25T00:21:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f567ad9d640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f567c06f490 0x7f567c0ab960 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f566400c500 tx=0x7f566400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f56737fe640 1 -- 172.21.15.107:0/449079336 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56640052d0 con 0x7f567c06f490 2023-10-25T00:21:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f56737fe640 1 -- 172.21.15.107:0/449079336 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5664005450 con 0x7f567c06f490 2023-10-25T00:21:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f5681090640 1 -- 172.21.15.107:0/449079336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567c0ad590 con 0x7f567c06f490 2023-10-25T00:21:23.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f56737fe640 1 -- 172.21.15.107:0/449079336 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664012870 con 0x7f567c06f490 2023-10-25T00:21:23.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.833+0000 7f5681090640 1 -- 172.21.15.107:0/449079336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567c145ea0 con 0x7f567c06f490 2023-10-25T00:21:23.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.837+0000 7f56737fe640 1 -- 172.21.15.107:0/449079336 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f56640129d0 con 0x7f567c06f490 2023-10-25T00:21:23.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.837+0000 7f56737fe640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f565c065e90 0x7f565c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:23.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.837+0000 7f56737fe640 1 -- 172.21.15.107:0/449079336 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7f56640845a0 con 0x7f567c06f490 2023-10-25T00:21:23.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.837+0000 7f5681090640 1 -- 172.21.15.107:0/449079336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567c0746e0 con 0x7f567c06f490 2023-10-25T00:21:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.837+0000 7f567a59c640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f565c065e90 0x7f565c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.837+0000 7f567a59c640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f565c065e90 0x7f565c068350 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f566c0ad000 tx=0x7f566c0a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:23.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:23.845+0000 7f56737fe640 1 -- 172.21.15.107:0/449079336 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5664050950 con 0x7f567c06f490 2023-10-25T00:21:24.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.129+0000 7f5681090640 1 -- 172.21.15.107:0/449079336 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f567c1411a0 con 0x7f567c06f490 2023-10-25T00:21:24.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f56737fe640 1 -- 172.21.15.107:0/449079336 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+5007 (secure 0 0 0) 0x7f5664053f70 con 0x7f567c06f490 2023-10-25T00:21:24.135 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:24.135 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:24.136 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 224 2023-10-25T00:21:24.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 -- 172.21.15.107:0/449079336 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f565c065e90 msgr2=0x7f565c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f565c065e90 0x7f565c068350 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f566c0ad000 tx=0x7f566c0a6000 comp rx=0 tx=0).stop 2023-10-25T00:21:24.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 -- 172.21.15.107:0/449079336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f567c06f490 msgr2=0x7f567c0ab960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f567c06f490 0x7f567c0ab960 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f566400c500 tx=0x7f566400c9c0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 -- 172.21.15.107:0/449079336 shutdown_connections 2023-10-25T00:21:24.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f567c152b30 0x7f567c141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f565c065e90 0x7f565c068350 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f567c14c470 0x7f567c1427e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 --2- 172.21.15.107:0/449079336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f567c06f490 0x7f567c0ab960 secure :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f566400c500 tx=0x7f566400c9c0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 -- 172.21.15.107:0/449079336 >> 172.21.15.107:0/449079336 conn(0x7f567c070f60 msgr2=0x7f567c0c32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:24.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 -- 172.21.15.107:0/449079336 shutdown_connections 2023-10-25T00:21:24.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.133+0000 7f5681090640 1 -- 172.21.15.107:0/449079336 wait complete. 2023-10-25T00:21:24.160 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2023-10-25T00:21:24.161 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:21:24.161 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:21:24.173 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:21:24.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:24.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:24.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc550ae2640 1 Processor -- start 2023-10-25T00:21:24.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc550ae2640 1 -- start start 2023-10-25T00:21:24.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc550ae2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc54c0c70e0 0x7fc54c0bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:24.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc550ae2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c0c1ca0 con 0x7fc54c0c2130 2023-10-25T00:21:24.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc550ae2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc54c0c10c0 con 0x7fc54c156770 2023-10-25T00:21:24.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc550ae2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc54c0c27e0 con 0x7fc54c0c70e0 2023-10-25T00:21:24.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc54ad9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc54c0c2130 0x7fc54c15d130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44046/0 (socket says 172.21.15.107:44046) 2023-10-25T00:21:24.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc54ad9d640 1 -- 172.21.15.107:0/3388972510 learned_addr learned my addr 172.21.15.107:0/3388972510 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:24.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc549d9b640 1 --2- 172.21.15.107:0/3388972510 >> v2:172.21.15.107:3300/0 conn(0x7fc54c0c70e0 0x7fc54c0bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:24.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc54959a640 1 -- 172.21.15.107:0/3388972510 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840425165 0 0) 0x7fc54c0c1ca0 con 0x7fc54c0c2130 2023-10-25T00:21:24.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc54959a640 1 -- 172.21.15.107:0/3388972510 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc534003530 con 0x7fc54c0c2130 2023-10-25T00:21:24.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc549d9b640 1 -- 172.21.15.107:0/3388972510 >> v1:172.21.15.110:6789/0 conn(0x7fc54c156770 legacy=0x7fc54c0ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc549d9b640 1 -- 172.21.15.107:0/3388972510 >> v1:172.21.15.107:6789/0 conn(0x7fc54c0c2130 legacy=0x7fc54c15d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.373+0000 7fc549d9b640 1 -- 172.21.15.107:0/3388972510 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54c0c13a0 con 0x7fc54c0c70e0 2023-10-25T00:21:24.383 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc549d9b640 1 --2- 172.21.15.107:0/3388972510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0c70e0 0x7fc54c0bdfa0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fc53c013db0 tx=0x7fc53c024a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def8d76fca300b94 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:24.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc54959a640 1 -- 172.21.15.107:0/3388972510 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c0ad8d0 con 0x7fc54c0c70e0 2023-10-25T00:21:24.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc54959a640 1 -- 172.21.15.107:0/3388972510 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c0ada30 con 0x7fc54c0c70e0 2023-10-25T00:21:24.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc54959a640 1 -- 172.21.15.107:0/3388972510 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c0addc0 con 0x7fc54c0c70e0 2023-10-25T00:21:24.384 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc550ae2640 1 -- 172.21.15.107:0/3388972510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0c70e0 msgr2=0x7fc54c0bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc550ae2640 1 --2- 172.21.15.107:0/3388972510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0c70e0 0x7fc54c0bdfa0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fc53c013db0 tx=0x7fc53c024a40 comp rx=0 tx=0).stop 2023-10-25T00:21:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc550ae2640 1 -- 172.21.15.107:0/3388972510 shutdown_connections 2023-10-25T00:21:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc550ae2640 1 --2- 172.21.15.107:0/3388972510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0c70e0 0x7fc54c0bdfa0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc550ae2640 1 -- 172.21.15.107:0/3388972510 >> 172.21.15.107:0/3388972510 conn(0x7fc54c0c65c0 msgr2=0x7fc54c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc550ae2640 1 -- 172.21.15.107:0/3388972510 shutdown_connections 2023-10-25T00:21:24.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc550ae2640 1 -- 172.21.15.107:0/3388972510 wait complete. 2023-10-25T00:21:24.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc550ae2640 1 Processor -- start 2023-10-25T00:21:24.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.377+0000 7fc550ae2640 1 -- start start 2023-10-25T00:21:24.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc550ae2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc54c0c2130 0x7fc54c14d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:24.386 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc550ae2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0c70e0 0x7fc54c1ea700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:24.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc550ae2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc54c156770 0x7fc54c1eac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:24.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc550ae2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c06f900 con 0x7fc54c0c70e0 2023-10-25T00:21:24.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc550ae2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c14a7e0 con 0x7fc54c156770 2023-10-25T00:21:24.387 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc550ae2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54c14aa30 con 0x7fc54c0c2130 2023-10-25T00:21:24.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc549d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0c70e0 0x7fc54c1ea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:24.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc549d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0c70e0 0x7fc54c1ea700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45668/0 (socket says 172.21.15.107:45668) 2023-10-25T00:21:24.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc549d9b640 1 -- 172.21.15.107:0/657643637 learned_addr learned my addr 172.21.15.107:0/657643637 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:24.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc549d9b640 1 -- 172.21.15.107:0/657643637 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc54c0c2130 msgr2=0x7fc54c14d070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:21:24.388 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc549d9b640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc54c0c2130 0x7fc54c14d070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc549d9b640 1 -- 172.21.15.107:0/657643637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc54c156770 msgr2=0x7fc54c1eac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc549d9b640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc54c156770 0x7fc54c1eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc549d9b640 1 -- 172.21.15.107:0/657643637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c013a10 con 0x7fc54c0c70e0 2023-10-25T00:21:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc549d9b640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0c70e0 0x7fc54c1ea700 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fc53c0b8040 tx=0x7fc53c0217e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc53affd640 1 -- 172.21.15.107:0/657643637 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c0b5a90 con 0x7fc54c0c70e0 2023-10-25T00:21:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc53affd640 1 -- 172.21.15.107:0/657643637 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c0b5c10 con 0x7fc54c0c70e0 2023-10-25T00:21:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc53affd640 1 -- 172.21.15.107:0/657643637 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c0be430 con 0x7fc54c0c70e0 2023-10-25T00:21:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc550ae2640 1 -- 172.21.15.107:0/657643637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54c14ad10 con 0x7fc54c0c70e0 2023-10-25T00:21:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.381+0000 7fc550ae2640 1 -- 172.21.15.107:0/657643637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54c14b2d0 con 0x7fc54c0c70e0 2023-10-25T00:21:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.385+0000 7fc550ae2640 1 -- 172.21.15.107:0/657643637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54c14b4f0 con 0x7fc54c0c70e0 2023-10-25T00:21:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.385+0000 7fc53affd640 1 -- 172.21.15.107:0/657643637 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc53c0bc040 con 0x7fc54c0c70e0 2023-10-25T00:21:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.385+0000 7fc53affd640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc52c065e60 0x7fc52c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.385+0000 7fc54a59c640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc52c065e60 0x7fc52c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.389+0000 7fc53affd640 1 -- 172.21.15.107:0/657643637 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7fc53c0b3340 con 0x7fc54c0c70e0 2023-10-25T00:21:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.389+0000 7fc54a59c640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc52c065e60 0x7fc52c068320 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc540000f70 tx=0x7fc5400079c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.389+0000 7fc53affd640 1 -- 172.21.15.107:0/657643637 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc53c0f4bb0 con 0x7fc54c0c70e0 2023-10-25T00:21:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e9110640 1 Processor -- start 2023-10-25T00:21:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e9110640 1 -- start start 2023-10-25T00:21:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e9110640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5e4071ce0 0x7fe5e40ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e9110640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe5e4154860 0x7fe5e4158230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e9110640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e406e710 con 0x7fe5e40720c0 2023-10-25T00:21:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e9110640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e406d900 con 0x7fe5e4071ce0 2023-10-25T00:21:24.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e9110640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e4154d20 con 0x7fe5e4154860 2023-10-25T00:21:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e2d9d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe5e4071ce0 0x7fe5e40ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e259c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe5e40720c0 0x7fe5e413d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44060/0 (socket says 172.21.15.107:44060) 2023-10-25T00:21:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e259c640 1 -- 172.21.15.107:0/1008088378 learned_addr learned my addr 172.21.15.107:0/1008088378 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e359e640 1 --2- 172.21.15.107:0/1008088378 >> v2:172.21.15.110:3300/0 conn(0x7fe5e4154860 0x7fe5e4158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:24.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e1d9b640 1 -- 172.21.15.107:0/1008088378 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774786319 0 0) 0x7fe5e406e710 con 0x7fe5e40720c0 2023-10-25T00:21:24.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e1d9b640 1 -- 172.21.15.107:0/1008088378 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5cc003530 con 0x7fe5e40720c0 2023-10-25T00:21:24.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e359e640 1 -- 172.21.15.107:0/1008088378 >> v2:172.21.15.107:3300/0 conn(0x7fe5e4071ce0 msgr2=0x7fe5e40ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:24.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e359e640 1 --2- 172.21.15.107:0/1008088378 >> v2:172.21.15.107:3300/0 conn(0x7fe5e4071ce0 0x7fe5e40ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e359e640 1 -- 172.21.15.107:0/1008088378 >> v1:172.21.15.107:6789/0 conn(0x7fe5e40720c0 legacy=0x7fe5e413d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.529+0000 7fe5e359e640 1 -- 172.21.15.107:0/1008088378 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e4155000 con 0x7fe5e4154860 2023-10-25T00:21:24.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.533+0000 7fe5e359e640 1 --2- 172.21.15.107:0/1008088378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e4154860 0x7fe5e4158230 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe5e406dc20 tx=0x7fe5dc01c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb778267384a2a70 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:24.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.533+0000 7fe5e1d9b640 1 -- 172.21.15.107:0/1008088378 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc0a5910 con 0x7fe5e4154860 2023-10-25T00:21:24.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.533+0000 7fe5e1d9b640 1 -- 172.21.15.107:0/1008088378 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5dc0a5a70 con 0x7fe5e4154860 2023-10-25T00:21:24.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.533+0000 7fe5e1d9b640 1 -- 172.21.15.107:0/1008088378 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc0a5e00 con 0x7fe5e4154860 2023-10-25T00:21:24.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.533+0000 7fe5e9110640 1 -- 172.21.15.107:0/1008088378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e4154860 msgr2=0x7fe5e4158230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.533+0000 7fe5e9110640 1 --2- 172.21.15.107:0/1008088378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e4154860 0x7fe5e4158230 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe5e406dc20 tx=0x7fe5dc01c870 comp rx=0 tx=0).stop 2023-10-25T00:21:24.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.533+0000 7fe5e9110640 1 -- 172.21.15.107:0/1008088378 shutdown_connections 2023-10-25T00:21:24.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.533+0000 7fe5e9110640 1 --2- 172.21.15.107:0/1008088378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e4154860 0x7fe5e4158230 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.533+0000 7fe5e9110640 1 --2- 172.21.15.107:0/1008088378 >> v2:172.21.15.107:3300/0 conn(0x7fe5e4071ce0 0x7fe5e40ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.533+0000 7fe5e9110640 1 -- 172.21.15.107:0/1008088378 >> 172.21.15.107:0/1008088378 conn(0x7fe5e4070f60 msgr2=0x7fe5e4071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:24.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 -- 172.21.15.107:0/1008088378 shutdown_connections 2023-10-25T00:21:24.541 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 -- 172.21.15.107:0/1008088378 wait complete. 2023-10-25T00:21:24.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 Processor -- start 2023-10-25T00:21:24.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 -- start start 2023-10-25T00:21:24.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5e4071ce0 0x7fe5e4137cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:24.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e40720c0 0x7fe5e41350b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:24.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e4154860 0x7fe5e413c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:24.543 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e414ec80 con 0x7fe5e40720c0 2023-10-25T00:21:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e41355f0 con 0x7fe5e4154860 2023-10-25T00:21:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e41576f0 con 0x7fe5e4071ce0 2023-10-25T00:21:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e40720c0 0x7fe5e41350b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e40720c0 0x7fe5e41350b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45692/0 (socket says 172.21.15.107:45692) 2023-10-25T00:21:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e259c640 1 -- 172.21.15.107:0/3777863085 learned_addr learned my addr 172.21.15.107:0/3777863085 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e2d9d640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5e4071ce0 0x7fe5e4137cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e359e640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e4154860 0x7fe5e413c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e259c640 1 -- 172.21.15.107:0/3777863085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5e4071ce0 msgr2=0x7fe5e4137cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e259c640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5e4071ce0 0x7fe5e4137cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e259c640 1 -- 172.21.15.107:0/3777863085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e4154860 msgr2=0x7fe5e413c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e259c640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e4154860 0x7fe5e413c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e259c640 1 -- 172.21.15.107:0/3777863085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5dc009a70 con 0x7fe5e40720c0 2023-10-25T00:21:24.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e259c640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e40720c0 0x7fe5e41350b0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800c500 tx=0x7fe5d800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:24.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5cb7fe640 1 -- 172.21.15.107:0/3777863085 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d8005cf0 con 0x7fe5e40720c0 2023-10-25T00:21:24.546 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 -- 172.21.15.107:0/3777863085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e413e200 con 0x7fe5e40720c0 2023-10-25T00:21:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.537+0000 7fe5e9110640 1 -- 172.21.15.107:0/3777863085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e414eed0 con 0x7fe5e40720c0 2023-10-25T00:21:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.541+0000 7fe5e9110640 1 -- 172.21.15.107:0/3777863085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e40746e0 con 0x7fe5e40720c0 2023-10-25T00:21:24.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.549+0000 7fe5cb7fe640 1 -- 172.21.15.107:0/3777863085 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d8005e50 con 0x7fe5e40720c0 2023-10-25T00:21:24.551 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.549+0000 7fe5cb7fe640 1 -- 172.21.15.107:0/3777863085 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d80072f0 con 0x7fe5e40720c0 2023-10-25T00:21:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.549+0000 7fe5cb7fe640 1 -- 172.21.15.107:0/3777863085 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe5d80074d0 con 0x7fe5e40720c0 2023-10-25T00:21:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.549+0000 7fe5cb7fe640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe5d0065ee0 0x7fe5d00683a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.549+0000 7fe5cb7fe640 1 -- 172.21.15.107:0/3777863085 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7fe5d80845e0 con 0x7fe5e40720c0 2023-10-25T00:21:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.549+0000 7fe5cb7fe640 1 -- 172.21.15.107:0/3777863085 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe5d8084970 con 0x7fe5e40720c0 2023-10-25T00:21:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.549+0000 7fe5e2d9d640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe5d0065ee0 0x7fe5d00683a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:24.562 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.557+0000 7fe5e2d9d640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe5d0065ee0 0x7fe5d00683a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40c2150 tx=0x7fe5d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:24.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 -- 172.21.15.107:0/657643637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc54c156b50 con 0x7fc54c0c70e0 2023-10-25T00:21:24.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc53affd640 1 -- 172.21.15.107:0/657643637 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+5007 (secure 0 0 0) 0x7fc53c0f4bb0 con 0x7fc54c0c70e0 2023-10-25T00:21:24.700 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:24.700 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:24.700 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 224 2023-10-25T00:21:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 -- 172.21.15.107:0/657643637 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc52c065e60 msgr2=0x7fc52c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc52c065e60 0x7fc52c068320 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc540000f70 tx=0x7fc5400079c0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 -- 172.21.15.107:0/657643637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0c70e0 msgr2=0x7fc54c1ea700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0c70e0 0x7fc54c1ea700 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fc53c0b8040 tx=0x7fc53c0217e0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 -- 172.21.15.107:0/657643637 shutdown_connections 2023-10-25T00:21:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc54c156770 0x7fc54c1eac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc54c0c70e0 0x7fc54c1ea700 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc52c065e60 0x7fc52c068320 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 --2- 172.21.15.107:0/657643637 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc54c0c2130 0x7fc54c14d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 -- 172.21.15.107:0/657643637 >> 172.21.15.107:0/657643637 conn(0x7fc54c0c65c0 msgr2=0x7fc54c0bfca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 -- 172.21.15.107:0/657643637 shutdown_connections 2023-10-25T00:21:24.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.697+0000 7fc550ae2640 1 -- 172.21.15.107:0/657643637 wait complete. 2023-10-25T00:21:24.724 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:21:24.851 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.849+0000 7fe5e9110640 1 -- 172.21.15.107:0/3777863085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5e406f940 con 0x7fe5e40720c0 2023-10-25T00:21:24.852 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.849+0000 7fe5cb7fe640 1 -- 172.21.15.107:0/3777863085 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+5007 (secure 0 0 0) 0x7fe5d8017090 con 0x7fe5e40720c0 2023-10-25T00:21:24.854 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:24.854 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:24.854 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 224 2023-10-25T00:21:24.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 -- 172.21.15.107:0/3777863085 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe5d0065ee0 msgr2=0x7fe5d00683a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe5d0065ee0 0x7fe5d00683a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40c2150 tx=0x7fe5d4008040 comp rx=0 tx=0).stop 2023-10-25T00:21:24.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 -- 172.21.15.107:0/3777863085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e40720c0 msgr2=0x7fe5e41350b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:24.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e40720c0 0x7fe5e41350b0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800c500 tx=0x7fe5d800c9c0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 -- 172.21.15.107:0/3777863085 shutdown_connections 2023-10-25T00:21:24.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5e4154860 0x7fe5e413c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5e40720c0 0x7fe5e41350b0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe5d0065ee0 0x7fe5d00683a0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 --2- 172.21.15.107:0/3777863085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe5e4071ce0 0x7fe5e4137cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:24.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 -- 172.21.15.107:0/3777863085 >> 172.21.15.107:0/3777863085 conn(0x7fe5e4070f60 msgr2=0x7fe5e4152520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:24.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 -- 172.21.15.107:0/3777863085 shutdown_connections 2023-10-25T00:21:24.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:24.853+0000 7fe5e9110640 1 -- 172.21.15.107:0/3777863085 wait complete. 2023-10-25T00:21:24.876 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=8712 2023-10-25T00:21:24.877 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.c 2023-10-25T00:21:26.237 INFO:tasks.ceph.mds.b.smithi110.stderr:2023-10-25T00:21:26.233+0000 7fe4e8ee8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:21:26.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.873+0000 7fa459ef1640 1 Processor -- start 2023-10-25T00:21:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.873+0000 7fa459ef1640 1 -- start start 2023-10-25T00:21:26.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa459ef1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa454156220 0x7fa45406df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:26.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa459ef1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4540c5d40 con 0x7fa4540c6a30 2023-10-25T00:21:26.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa459ef1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4540c6130 con 0x7fa45415c200 2023-10-25T00:21:26.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa459ef1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa454158f60 con 0x7fa454156220 2023-10-25T00:21:26.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa453fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa4540c6a30 0x7fa454071810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:44062/0 (socket says 172.21.15.107:44062) 2023-10-25T00:21:26.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa453fff640 1 -- 172.21.15.107:0/3612084268 learned_addr learned my addr 172.21.15.107:0/3612084268 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:26.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa4537fe640 1 --2- 172.21.15.107:0/3612084268 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa454156220 0x7fa45406df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:26.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa4527fc640 1 -- 172.21.15.107:0/3612084268 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001874457 0 0) 0x7fa4540c5d40 con 0x7fa4540c6a30 2023-10-25T00:21:26.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa4527fc640 1 -- 172.21.15.107:0/3612084268 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa43c003530 con 0x7fa4540c6a30 2023-10-25T00:21:26.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa4527fc640 1 -- 172.21.15.107:0/3612084268 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727861184 0 0) 0x7fa4540c6130 con 0x7fa45415c200 2023-10-25T00:21:26.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa4527fc640 1 -- 172.21.15.107:0/3612084268 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4540c5d40 con 0x7fa45415c200 2023-10-25T00:21:26.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa4537fe640 1 -- 172.21.15.107:0/3612084268 >> v1:172.21.15.110:6789/0 conn(0x7fa45415c200 legacy=0x7fa454071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:26.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa4537fe640 1 -- 172.21.15.107:0/3612084268 >> v1:172.21.15.107:6789/0 conn(0x7fa4540c6a30 legacy=0x7fa454071810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:26.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.877+0000 7fa4537fe640 1 -- 172.21.15.107:0/3612084268 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa454159240 con 0x7fa454156220 2023-10-25T00:21:26.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.881+0000 7fa4537fe640 1 --2- 172.21.15.107:0/3612084268 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa454156220 0x7fa45406df30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009dc0 tx=0x7fa44c01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b20f3f5345d82f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:26.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.881+0000 7fa4527fc640 1 -- 172.21.15.107:0/3612084268 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0a5970 con 0x7fa454156220 2023-10-25T00:21:26.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.881+0000 7fa4527fc640 1 -- 172.21.15.107:0/3612084268 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44c0a5ad0 con 0x7fa454156220 2023-10-25T00:21:26.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.881+0000 7fa459ef1640 1 -- 172.21.15.107:0/3612084268 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa454156220 msgr2=0x7fa45406df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:26.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.881+0000 7fa459ef1640 1 --2- 172.21.15.107:0/3612084268 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa454156220 0x7fa45406df30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009dc0 tx=0x7fa44c01c8b0 comp rx=0 tx=0).stop 2023-10-25T00:21:26.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.881+0000 7fa459ef1640 1 -- 172.21.15.107:0/3612084268 shutdown_connections 2023-10-25T00:21:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.881+0000 7fa459ef1640 1 --2- 172.21.15.107:0/3612084268 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa454156220 0x7fa45406df30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:26.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.885+0000 7fa459ef1640 1 -- 172.21.15.107:0/3612084268 >> 172.21.15.107:0/3612084268 conn(0x7fa4540c39a0 msgr2=0x7fa4540c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:26.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.885+0000 7fa459ef1640 1 -- 172.21.15.107:0/3612084268 shutdown_connections 2023-10-25T00:21:26.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.885+0000 7fa459ef1640 1 -- 172.21.15.107:0/3612084268 wait complete. 2023-10-25T00:21:26.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.885+0000 7fa459ef1640 1 Processor -- start 2023-10-25T00:21:26.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.885+0000 7fa459ef1640 1 -- start start 2023-10-25T00:21:26.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa459ef1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa4540c6a30 0x7fa4540c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:26.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa459ef1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa454156220 0x7fa45414e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:26.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa459ef1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa45415c200 0x7fa45414ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:26.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa459ef1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa454150780 con 0x7fa45415c200 2023-10-25T00:21:26.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa459ef1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4541f1bd0 con 0x7fa454156220 2023-10-25T00:21:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa459ef1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45406ae50 con 0x7fa4540c6a30 2023-10-25T00:21:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa453fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa45415c200 0x7fa45414ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa453fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa45415c200 0x7fa45414ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45700/0 (socket says 172.21.15.107:45700) 2023-10-25T00:21:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa453fff640 1 -- 172.21.15.107:0/3503079537 learned_addr learned my addr 172.21.15.107:0/3503079537 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:26.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa452ffd640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa454156220 0x7fa45414e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:26.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa453fff640 1 -- 172.21.15.107:0/3503079537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa4540c6a30 msgr2=0x7fa4540c1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:26.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa453fff640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa4540c6a30 0x7fa4540c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:26.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa453fff640 1 -- 172.21.15.107:0/3503079537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa454156220 msgr2=0x7fa45414e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:26.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa453fff640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa454156220 0x7fa45414e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:26.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa453fff640 1 -- 172.21.15.107:0/3503079537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c009a70 con 0x7fa45415c200 2023-10-25T00:21:26.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa453fff640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa45415c200 0x7fa45414ef00 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fa44800c450 tx=0x7fa44800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:26.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa43bfff640 1 -- 172.21.15.107:0/3503079537 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448005cf0 con 0x7fa45415c200 2023-10-25T00:21:26.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa43bfff640 1 -- 172.21.15.107:0/3503079537 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa448005e70 con 0x7fa45415c200 2023-10-25T00:21:26.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa459ef1640 1 -- 172.21.15.107:0/3503079537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45406b130 con 0x7fa45415c200 2023-10-25T00:21:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa43bfff640 1 -- 172.21.15.107:0/3503079537 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448007290 con 0x7fa45415c200 2023-10-25T00:21:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa459ef1640 1 -- 172.21.15.107:0/3503079537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4541f1eb0 con 0x7fa45415c200 2023-10-25T00:21:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa459ef1640 1 -- 172.21.15.107:0/3503079537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa454070480 con 0x7fa45415c200 2023-10-25T00:21:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa43bfff640 1 -- 172.21.15.107:0/3503079537 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa4480073f0 con 0x7fa45415c200 2023-10-25T00:21:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa43bfff640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa440065ee0 0x7fa4400683a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:26.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.889+0000 7fa4537fe640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa440065ee0 0x7fa4400683a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.893+0000 7fa43bfff640 1 -- 172.21.15.107:0/3503079537 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7fa448083ea0 con 0x7fa45415c200 2023-10-25T00:21:26.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.893+0000 7fa4537fe640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa440065ee0 0x7fa4400683a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0af000 tx=0x7fa44c004ac0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:26.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:26.897+0000 7fa43bfff640 1 -- 172.21.15.107:0/3503079537 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa448050460 con 0x7fa45415c200 2023-10-25T00:21:27.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.169+0000 7fa459ef1640 1 -- 172.21.15.107:0/3503079537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4540746e0 con 0x7fa45415c200 2023-10-25T00:21:27.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.169+0000 7fa43bfff640 1 -- 172.21.15.107:0/3503079537 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+5007 (secure 0 0 0) 0x7fa448017020 con 0x7fa45415c200 2023-10-25T00:21:27.176 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:27.176 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:27.177 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 224 2023-10-25T00:21:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 -- 172.21.15.107:0/3503079537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa440065ee0 msgr2=0x7fa4400683a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa440065ee0 0x7fa4400683a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa44c0af000 tx=0x7fa44c004ac0 comp rx=0 tx=0).stop 2023-10-25T00:21:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 -- 172.21.15.107:0/3503079537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa45415c200 msgr2=0x7fa45414ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa45415c200 0x7fa45414ef00 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fa44800c450 tx=0x7fa44800c910 comp rx=0 tx=0).stop 2023-10-25T00:21:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 -- 172.21.15.107:0/3503079537 shutdown_connections 2023-10-25T00:21:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa45415c200 0x7fa45414ef00 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa454156220 0x7fa45414e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa440065ee0 0x7fa4400683a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 --2- 172.21.15.107:0/3503079537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa4540c6a30 0x7fa4540c1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:27.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 -- 172.21.15.107:0/3503079537 >> 172.21.15.107:0/3503079537 conn(0x7fa4540c39a0 msgr2=0x7fa4540c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 -- 172.21.15.107:0/3503079537 shutdown_connections 2023-10-25T00:21:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:27.173+0000 7fa459ef1640 1 -- 172.21.15.107:0/3503079537 wait complete. 2023-10-25T00:21:27.196 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:21:29.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:29.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5f1bef640 1 Processor -- start 2023-10-25T00:21:29.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5f1bef640 1 -- start start 2023-10-25T00:21:29.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5f1bef640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5ec0c2790 0x7fc5ec151480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:29.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5f1bef640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5ec06fde0 0x7fc5ec070250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:29.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5f1bef640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec0c44c0 con 0x7fc5ec071770 2023-10-25T00:21:29.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5f1bef640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ec071b50 con 0x7fc5ec0c2790 2023-10-25T00:21:29.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5f1bef640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec0c38b0 con 0x7fc5ec06fde0 2023-10-25T00:21:29.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5eaffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5ec0c2790 0x7fc5ec151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:29.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5eaffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc5ec0c2790 0x7fc5ec151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45708/0 (socket says 172.21.15.107:45708) 2023-10-25T00:21:29.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5eb7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc5ec071770 0x7fc5ec06cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:41064/0 (socket says 172.21.15.107:41064) 2023-10-25T00:21:29.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5eaffd640 1 -- 172.21.15.107:0/919230198 learned_addr learned my addr 172.21.15.107:0/919230198 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:29.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5ebfff640 1 --2- 172.21.15.107:0/919230198 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5ec06fde0 0x7fc5ec070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:29.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5ea7fc640 1 -- 172.21.15.107:0/919230198 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114228264 0 0) 0x7fc5ec0c44c0 con 0x7fc5ec071770 2023-10-25T00:21:29.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5eaffd640 1 -- 172.21.15.107:0/919230198 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5ec06fde0 msgr2=0x7fc5ec070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:29.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5eaffd640 1 --2- 172.21.15.107:0/919230198 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5ec06fde0 0x7fc5ec070250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:29.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5eaffd640 1 -- 172.21.15.107:0/919230198 >> v1:172.21.15.110:6789/0 conn(0x7fc5ec071770 legacy=0x7fc5ec06cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:29.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.341+0000 7fc5eaffd640 1 -- 172.21.15.107:0/919230198 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec0c3b90 con 0x7fc5ec0c2790 2023-10-25T00:21:29.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5eaffd640 1 --2- 172.21.15.107:0/919230198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5ec0c2790 0x7fc5ec151480 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000be50 tx=0x7fc5e001e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3abebd0496155cb server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:29.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5ea7fc640 1 -- 172.21.15.107:0/919230198 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e00a75b0 con 0x7fc5ec0c2790 2023-10-25T00:21:29.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5ea7fc640 1 -- 172.21.15.107:0/919230198 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e00a7710 con 0x7fc5ec0c2790 2023-10-25T00:21:29.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5ea7fc640 1 -- 172.21.15.107:0/919230198 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e00a7aa0 con 0x7fc5ec0c2790 2023-10-25T00:21:29.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5f1bef640 1 -- 172.21.15.107:0/919230198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5ec0c2790 msgr2=0x7fc5ec151480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:29.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5f1bef640 1 --2- 172.21.15.107:0/919230198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5ec0c2790 0x7fc5ec151480 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000be50 tx=0x7fc5e001e8f0 comp rx=0 tx=0).stop 2023-10-25T00:21:29.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5f1bef640 1 -- 172.21.15.107:0/919230198 shutdown_connections 2023-10-25T00:21:29.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5f1bef640 1 --2- 172.21.15.107:0/919230198 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5ec06fde0 0x7fc5ec070250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:29.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5f1bef640 1 --2- 172.21.15.107:0/919230198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5ec0c2790 0x7fc5ec151480 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:29.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5f1bef640 1 -- 172.21.15.107:0/919230198 >> 172.21.15.107:0/919230198 conn(0x7fc5ec0c6ed0 msgr2=0x7fc5ec0c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:29.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5f1bef640 1 -- 172.21.15.107:0/919230198 shutdown_connections 2023-10-25T00:21:29.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.345+0000 7fc5f1bef640 1 -- 172.21.15.107:0/919230198 wait complete. 2023-10-25T00:21:29.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5f1bef640 1 Processor -- start 2023-10-25T00:21:29.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5f1bef640 1 -- start start 2023-10-25T00:21:29.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5f1bef640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5ec06fde0 0x7fc5ec1f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:29.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5f1bef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5ec071770 0x7fc5ec144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:29.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5f1bef640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5ec0c2790 0x7fc5ec147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:29.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5f1bef640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec1f1350 con 0x7fc5ec071770 2023-10-25T00:21:29.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5f1bef640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec1f2320 con 0x7fc5ec06fde0 2023-10-25T00:21:29.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5f1bef640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec1f1560 con 0x7fc5ec0c2790 2023-10-25T00:21:29.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5ec071770 0x7fc5ec144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:29.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5eaffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5ec071770 0x7fc5ec144620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45716/0 (socket says 172.21.15.107:45716) 2023-10-25T00:21:29.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5eaffd640 1 -- 172.21.15.107:0/377074231 learned_addr learned my addr 172.21.15.107:0/377074231 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:29.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5ebfff640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5ec0c2790 0x7fc5ec147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:29.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5eb7fe640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5ec06fde0 0x7fc5ec1f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:29.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5ebfff640 1 -- 172.21.15.107:0/377074231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5ec06fde0 msgr2=0x7fc5ec1f52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:29.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5ebfff640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5ec06fde0 0x7fc5ec1f52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:29.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5ebfff640 1 -- 172.21.15.107:0/377074231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5ec071770 msgr2=0x7fc5ec144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:29.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5ebfff640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5ec071770 0x7fc5ec144620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:29.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5ebfff640 1 -- 172.21.15.107:0/377074231 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e000bb30 con 0x7fc5ec0c2790 2023-10-25T00:21:29.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5ebfff640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5ec0c2790 0x7fc5ec147190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400eba0 tx=0x7fc5e400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:29.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5d3fff640 1 -- 172.21.15.107:0/377074231 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e400cd20 con 0x7fc5ec0c2790 2023-10-25T00:21:29.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5d3fff640 1 -- 172.21.15.107:0/377074231 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e400f040 con 0x7fc5ec0c2790 2023-10-25T00:21:29.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5d3fff640 1 -- 172.21.15.107:0/377074231 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40146c0 con 0x7fc5ec0c2790 2023-10-25T00:21:29.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5f1bef640 1 -- 172.21.15.107:0/377074231 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec1f1840 con 0x7fc5ec0c2790 2023-10-25T00:21:29.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5f1bef640 1 -- 172.21.15.107:0/377074231 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec1f1e00 con 0x7fc5ec0c2790 2023-10-25T00:21:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.349+0000 7fc5f1bef640 1 -- 172.21.15.107:0/377074231 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec0746e0 con 0x7fc5ec0c2790 2023-10-25T00:21:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.353+0000 7fc5d3fff640 1 -- 172.21.15.107:0/377074231 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc5e4014820 con 0x7fc5ec0c2790 2023-10-25T00:21:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.353+0000 7fc5d3fff640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5d8065e60 0x7fc5d8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.353+0000 7fc5d3fff640 1 -- 172.21.15.107:0/377074231 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7fc5e4085bb0 con 0x7fc5ec0c2790 2023-10-25T00:21:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.353+0000 7fc5eb7fe640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5d8065e60 0x7fc5d8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:29.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.353+0000 7fc5eb7fe640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5d8065e60 0x7fc5d8068320 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc000f70 tx=0x7fc5dc007df0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:29.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.361+0000 7fc5d3fff640 1 -- 172.21.15.107:0/377074231 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc5e4052110 con 0x7fc5ec0c2790 2023-10-25T00:21:29.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.625+0000 7fc5f1bef640 1 -- 172.21.15.107:0/377074231 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5ec0aa590 con 0x7fc5ec0c2790 2023-10-25T00:21:29.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5d3fff640 1 -- 172.21.15.107:0/377074231 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+5007 (secure 0 0 0) 0x7fc5e4018020 con 0x7fc5ec0c2790 2023-10-25T00:21:29.632 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:29.632 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:29.633 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 224 2023-10-25T00:21:29.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 -- 172.21.15.107:0/377074231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5d8065e60 msgr2=0x7fc5d8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:29.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5d8065e60 0x7fc5d8068320 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc000f70 tx=0x7fc5dc007df0 comp rx=0 tx=0).stop 2023-10-25T00:21:29.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 -- 172.21.15.107:0/377074231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5ec0c2790 msgr2=0x7fc5ec147190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:29.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5ec0c2790 0x7fc5ec147190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400eba0 tx=0x7fc5e400c490 comp rx=0 tx=0).stop 2023-10-25T00:21:29.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 -- 172.21.15.107:0/377074231 shutdown_connections 2023-10-25T00:21:29.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc5d8065e60 0x7fc5d8068320 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:29.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5ec0c2790 0x7fc5ec147190 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:29.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5ec071770 0x7fc5ec144620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:29.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 --2- 172.21.15.107:0/377074231 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5ec06fde0 0x7fc5ec1f52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:29.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 -- 172.21.15.107:0/377074231 >> 172.21.15.107:0/377074231 conn(0x7fc5ec0c6ed0 msgr2=0x7fc5ec0ac820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:29.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 -- 172.21.15.107:0/377074231 shutdown_connections 2023-10-25T00:21:29.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:29.629+0000 7fc5f1bef640 1 -- 172.21.15.107:0/377074231 wait complete. 2023-10-25T00:21:29.653 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:21:31.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:31.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.789+0000 7ff8ffeb2640 1 Processor -- start 2023-10-25T00:21:31.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.789+0000 7ff8ffeb2640 1 -- start start 2023-10-25T00:21:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8ffeb2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8f814fd40 0x7ff8f80ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8ffeb2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f80a9820 0x7ff8f80aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:31.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8ffeb2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff8f806ee40 0x7ff8f80c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8ffeb2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f80aa750 con 0x7ff8f814fd40 2023-10-25T00:21:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8ffeb2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f806f730 con 0x7ff8f806ee40 2023-10-25T00:21:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8ffeb2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8156a40 con 0x7ff8f80a9820 2023-10-25T00:21:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8fd44d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f80a9820 0x7ff8f80aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8fd44d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f80a9820 0x7ff8f80aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:40592/0 (socket says 172.21.15.107:40592) 2023-10-25T00:21:31.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8fd44d640 1 -- 172.21.15.107:0/3727264527 learned_addr learned my addr 172.21.15.107:0/3727264527 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:31.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8fdc4e640 1 --2- 172.21.15.107:0/3727264527 >> v2:172.21.15.107:3300/0 conn(0x7ff8f814fd40 0x7ff8f80ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:31.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8fdc4e640 1 -- 172.21.15.107:0/3727264527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f80a9820 msgr2=0x7ff8f80aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:31.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8fdc4e640 1 --2- 172.21.15.107:0/3727264527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f80a9820 0x7ff8f80aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:31.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8fdc4e640 1 -- 172.21.15.107:0/3727264527 >> v2:172.21.15.110:3300/0 conn(0x7ff8f806ee40 msgr2=0x7ff8f80c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:31.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8fdc4e640 1 --2- 172.21.15.107:0/3727264527 >> v2:172.21.15.110:3300/0 conn(0x7ff8f806ee40 0x7ff8f80c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:31.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.793+0000 7ff8fdc4e640 1 -- 172.21.15.107:0/3727264527 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f8156710 con 0x7ff8f814fd40 2023-10-25T00:21:31.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.797+0000 7ff8fdc4e640 1 --2- 172.21.15.107:0/3727264527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8f814fd40 0x7ff8f80ab090 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7ff8f806f260 tx=0x7ff8f401c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b90687f1d393a59 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:31.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.797+0000 7ff8fcc4c640 1 -- 172.21.15.107:0/3727264527 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f40ab600 con 0x7ff8f814fd40 2023-10-25T00:21:31.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.797+0000 7ff8fcc4c640 1 -- 172.21.15.107:0/3727264527 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f40a7050 con 0x7ff8f814fd40 2023-10-25T00:21:31.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.797+0000 7ff8fcc4c640 1 -- 172.21.15.107:0/3727264527 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f40ab980 con 0x7ff8f814fd40 2023-10-25T00:21:31.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/3727264527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8f814fd40 msgr2=0x7ff8f80ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:31.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 --2- 172.21.15.107:0/3727264527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8f814fd40 0x7ff8f80ab090 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7ff8f806f260 tx=0x7ff8f401c310 comp rx=0 tx=0).stop 2023-10-25T00:21:31.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/3727264527 shutdown_connections 2023-10-25T00:21:31.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 --2- 172.21.15.107:0/3727264527 >> v2:172.21.15.110:3300/0 conn(0x7ff8f806ee40 0x7ff8f80c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:31.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 --2- 172.21.15.107:0/3727264527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f80a9820 0x7ff8f80aa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:31.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 --2- 172.21.15.107:0/3727264527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8f814fd40 0x7ff8f80ab090 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:31.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/3727264527 >> 172.21.15.107:0/3727264527 conn(0x7ff8f80c2950 msgr2=0x7ff8f80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:31.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/3727264527 shutdown_connections 2023-10-25T00:21:31.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/3727264527 wait complete. 2023-10-25T00:21:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 Processor -- start 2023-10-25T00:21:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 -- start start 2023-10-25T00:21:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f806ee40 0x7ff8f814ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8f814fd40 0x7ff8f81eeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8f81eb070 0x7ff8f81ef0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8146050 con 0x7ff8f81eb070 2023-10-25T00:21:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8151bf0 con 0x7ff8f814fd40 2023-10-25T00:21:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8151ed0 con 0x7ff8f806ee40 2023-10-25T00:21:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fe44f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8f81eb070 0x7ff8f81ef0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:31.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fe44f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8f81eb070 0x7ff8f81ef0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45730/0 (socket says 172.21.15.107:45730) 2023-10-25T00:21:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fe44f640 1 -- 172.21.15.107:0/1113029142 learned_addr learned my addr 172.21.15.107:0/1113029142 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fd44d640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8f814fd40 0x7ff8f81eeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fdc4e640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f806ee40 0x7ff8f814ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fe44f640 1 -- 172.21.15.107:0/1113029142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f806ee40 msgr2=0x7ff8f814ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fe44f640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f806ee40 0x7ff8f814ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:31.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fe44f640 1 -- 172.21.15.107:0/1113029142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8f814fd40 msgr2=0x7ff8f81eeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:31.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fe44f640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8f814fd40 0x7ff8f81eeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:31.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fe44f640 1 -- 172.21.15.107:0/1113029142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f4009420 con 0x7ff8f81eb070 2023-10-25T00:21:31.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fdc4e640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f806ee40 0x7ff8f814ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:21:31.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8fe44f640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8f81eb070 0x7ff8f81ef0e0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000b970 tx=0x7ff8f000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ee7fc640 1 -- 172.21.15.107:0/1113029142 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f000cb40 con 0x7ff8f81eb070 2023-10-25T00:21:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ee7fc640 1 -- 172.21.15.107:0/1113029142 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f000ccc0 con 0x7ff8f81eb070 2023-10-25T00:21:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/1113029142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f81521b0 con 0x7ff8f81eb070 2023-10-25T00:21:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ee7fc640 1 -- 172.21.15.107:0/1113029142 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0014680 con 0x7ff8f81eb070 2023-10-25T00:21:31.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.801+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/1113029142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f814b150 con 0x7ff8f81eb070 2023-10-25T00:21:31.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.805+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/1113029142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f80c24a0 con 0x7ff8f81eb070 2023-10-25T00:21:31.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.805+0000 7ff8ee7fc640 1 -- 172.21.15.107:0/1113029142 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff8f001d3b0 con 0x7ff8f81eb070 2023-10-25T00:21:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.805+0000 7ff8ee7fc640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8dc065e60 0x7ff8dc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.805+0000 7ff8ee7fc640 1 -- 172.21.15.107:0/1113029142 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7ff8f000ce30 con 0x7ff8f81eb070 2023-10-25T00:21:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.809+0000 7ff8fdc4e640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8dc065e60 0x7ff8dc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:31.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.809+0000 7ff8fdc4e640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8dc065e60 0x7ff8dc068320 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff8f40ad000 tx=0x7ff8f40a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:31.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:31.817+0000 7ff8ee7fc640 1 -- 172.21.15.107:0/1113029142 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff8f0051a10 con 0x7ff8f81eb070 2023-10-25T00:21:32.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.093+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/1113029142 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8f8070840 con 0x7ff8f81eb070 2023-10-25T00:21:32.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.093+0000 7ff8ee7fc640 1 -- 172.21.15.107:0/1113029142 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+5007 (secure 0 0 0) 0x7ff8f0019070 con 0x7ff8f81eb070 2023-10-25T00:21:32.098 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:32.098 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:32.099 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 224 2023-10-25T00:21:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/1113029142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8dc065e60 msgr2=0x7ff8dc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:32.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8dc065e60 0x7ff8dc068320 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7ff8f40ad000 tx=0x7ff8f40a6000 comp rx=0 tx=0).stop 2023-10-25T00:21:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/1113029142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8f81eb070 msgr2=0x7ff8f81ef0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8f81eb070 0x7ff8f81ef0e0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000b970 tx=0x7ff8f000be30 comp rx=0 tx=0).stop 2023-10-25T00:21:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/1113029142 shutdown_connections 2023-10-25T00:21:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8f81eb070 0x7ff8f81ef0e0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:32.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8f814fd40 0x7ff8f81eeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff8dc065e60 0x7ff8dc068320 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 --2- 172.21.15.107:0/1113029142 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8f806ee40 0x7ff8f814ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/1113029142 >> 172.21.15.107:0/1113029142 conn(0x7ff8f80c2950 msgr2=0x7ff8f814e080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:32.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/1113029142 shutdown_connections 2023-10-25T00:21:32.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:32.097+0000 7ff8ffeb2640 1 -- 172.21.15.107:0/1113029142 wait complete. 2023-10-25T00:21:32.123 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:21:32.877 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2023-10-25T00:21:32.878 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:21:32.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:21:32.881 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:21:32.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:32.975 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:21:33.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.029+0000 7f36ae86e640 1 Processor -- start 2023-10-25T00:21:33.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ae86e640 1 -- start start 2023-10-25T00:21:33.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ae86e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36a80c8640 0x7f36a806a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:33.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ae86e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36a81553d0 0x7f36a806ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:33.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ae86e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a8152400 con 0x7f36a8152020 2023-10-25T00:21:33.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ae86e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a81558d0 con 0x7f36a81553d0 2023-10-25T00:21:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ae86e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a80c6ce0 con 0x7f36a80c8640 2023-10-25T00:21:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ace0b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36a81553d0 0x7f36a806ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ace0b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f36a81553d0 0x7f36a806ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52554/0 (socket says 172.21.15.107:52554) 2023-10-25T00:21:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ace0b640 1 -- 172.21.15.107:0/1166256586 learned_addr learned my addr 172.21.15.107:0/1166256586 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36a77fe640 1 --2- 172.21.15.107:0/1166256586 >> v2:172.21.15.110:3300/0 conn(0x7f36a80c8640 0x7f36a806a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ace0b640 1 -- 172.21.15.107:0/1166256586 >> v2:172.21.15.110:3300/0 conn(0x7f36a80c8640 msgr2=0x7f36a806a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ace0b640 1 --2- 172.21.15.107:0/1166256586 >> v2:172.21.15.110:3300/0 conn(0x7f36a80c8640 0x7f36a806a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:33.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ace0b640 1 -- 172.21.15.107:0/1166256586 >> v1:172.21.15.110:6789/0 conn(0x7f36a8152020 legacy=0x7f36a80c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:33.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.033+0000 7f36ace0b640 1 -- 172.21.15.107:0/1166256586 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a80c6fc0 con 0x7f36a81553d0 2023-10-25T00:21:33.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.037+0000 7f36ace0b640 1 --2- 172.21.15.107:0/1166256586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36a81553d0 0x7f36a806ac50 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f36a80bdd00 tx=0x7f36a001d370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de422eda1d3ac2d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:33.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.037+0000 7f36a6ffd640 1 -- 172.21.15.107:0/1166256586 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00a57f0 con 0x7f36a81553d0 2023-10-25T00:21:33.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.037+0000 7f36a6ffd640 1 -- 172.21.15.107:0/1166256586 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a00a5950 con 0x7f36a81553d0 2023-10-25T00:21:33.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.037+0000 7f36a6ffd640 1 -- 172.21.15.107:0/1166256586 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00a5ce0 con 0x7f36a81553d0 2023-10-25T00:21:33.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.041+0000 7f36ae86e640 1 -- 172.21.15.107:0/1166256586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36a81553d0 msgr2=0x7f36a806ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:33.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.041+0000 7f36ae86e640 1 --2- 172.21.15.107:0/1166256586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36a81553d0 0x7f36a806ac50 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f36a80bdd00 tx=0x7f36a001d370 comp rx=0 tx=0).stop 2023-10-25T00:21:33.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.041+0000 7f36ae86e640 1 -- 172.21.15.107:0/1166256586 shutdown_connections 2023-10-25T00:21:33.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.041+0000 7f36ae86e640 1 --2- 172.21.15.107:0/1166256586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36a81553d0 0x7f36a806ac50 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:33.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.041+0000 7f36ae86e640 1 --2- 172.21.15.107:0/1166256586 >> v2:172.21.15.110:3300/0 conn(0x7f36a80c8640 0x7f36a806a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:33.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.041+0000 7f36ae86e640 1 -- 172.21.15.107:0/1166256586 >> 172.21.15.107:0/1166256586 conn(0x7f36a80abd60 msgr2=0x7f36a80ac170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:33.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.041+0000 7f36ae86e640 1 -- 172.21.15.107:0/1166256586 shutdown_connections 2023-10-25T00:21:33.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.041+0000 7f36ae86e640 1 -- 172.21.15.107:0/1166256586 wait complete. 2023-10-25T00:21:33.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ae86e640 1 Processor -- start 2023-10-25T00:21:33.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ae86e640 1 -- start start 2023-10-25T00:21:33.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ae86e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36a80c8640 0x7f36a80c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:33.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ae86e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36a8152020 0x7f36a81f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:33.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ae86e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36a81553d0 0x7f36a81f4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:33.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ae86e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a80c2040 con 0x7f36a81553d0 2023-10-25T00:21:33.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ae86e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a80c1430 con 0x7f36a8152020 2023-10-25T00:21:33.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ae86e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a80c1710 con 0x7f36a80c8640 2023-10-25T00:21:33.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ace0b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36a81553d0 0x7f36a81f4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:33.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ace0b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36a81553d0 0x7f36a81f4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52558/0 (socket says 172.21.15.107:52558) 2023-10-25T00:21:33.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ace0b640 1 -- 172.21.15.107:0/1378819403 learned_addr learned my addr 172.21.15.107:0/1378819403 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:33.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36a77fe640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36a8152020 0x7f36a81f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:33.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ace0b640 1 -- 172.21.15.107:0/1378819403 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36a80c8640 msgr2=0x7f36a80c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:33.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ace0b640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36a80c8640 0x7f36a80c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:33.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ace0b640 1 -- 172.21.15.107:0/1378819403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36a8152020 msgr2=0x7f36a81f3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:33.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ace0b640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36a8152020 0x7f36a81f3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:33.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ace0b640 1 -- 172.21.15.107:0/1378819403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a0009b30 con 0x7f36a81553d0 2023-10-25T00:21:33.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.045+0000 7f36ace0b640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36a81553d0 0x7f36a81f4350 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f36a00af040 tx=0x7f36a00196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.053+0000 7f36a4ff9640 1 -- 172.21.15.107:0/1378819403 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00b8470 con 0x7f36a81553d0 2023-10-25T00:21:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.053+0000 7f36a4ff9640 1 -- 172.21.15.107:0/1378819403 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36a00b85d0 con 0x7f36a81553d0 2023-10-25T00:21:33.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.053+0000 7f36a4ff9640 1 -- 172.21.15.107:0/1378819403 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a00b8960 con 0x7f36a81553d0 2023-10-25T00:21:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.053+0000 7f36ae86e640 1 -- 172.21.15.107:0/1378819403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a80c19f0 con 0x7f36a81553d0 2023-10-25T00:21:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.053+0000 7f36ae86e640 1 -- 172.21.15.107:0/1378819403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a8153e30 con 0x7f36a81553d0 2023-10-25T00:21:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.053+0000 7f36a4ff9640 1 -- 172.21.15.107:0/1378819403 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f36a00b8ac0 con 0x7f36a81553d0 2023-10-25T00:21:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.053+0000 7f368e7fc640 1 -- 172.21.15.107:0/1378819403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3680005260 con 0x7f36a81553d0 2023-10-25T00:21:33.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.053+0000 7f36a4ff9640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065eb0 0x7f3694068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.053+0000 7f36a4ff9640 1 -- 172.21.15.107:0/1378819403 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7f36a011fd90 con 0x7f36a81553d0 2023-10-25T00:21:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.053+0000 7f36a7fff640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065eb0 0x7f3694068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:33.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.053+0000 7f36a7fff640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065eb0 0x7f3694068370 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f3698000f70 tx=0x7f36980079c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.061+0000 7f36a4ff9640 1 -- 172.21.15.107:0/1378819403 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f36a0123030 con 0x7f36a81553d0 2023-10-25T00:21:33.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.341+0000 7f368e7fc640 1 -- 172.21.15.107:0/1378819403 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3680005510 con 0x7f36a81553d0 2023-10-25T00:21:33.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.341+0000 7f36a4ff9640 1 -- 172.21.15.107:0/1378819403 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5768 (secure 0 0 0) 0x7f36a00ec380 con 0x7f36a81553d0 2023-10-25T00:21:33.346 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:33.346 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9009,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":225}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:33.347 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 225 2023-10-25T00:21:33.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 -- 172.21.15.107:0/1378819403 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065eb0 msgr2=0x7f3694068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:33.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065eb0 0x7f3694068370 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f3698000f70 tx=0x7f36980079c0 comp rx=0 tx=0).stop 2023-10-25T00:21:33.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 -- 172.21.15.107:0/1378819403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36a81553d0 msgr2=0x7f36a81f4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:33.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36a81553d0 0x7f36a81f4350 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f36a00af040 tx=0x7f36a00196b0 comp rx=0 tx=0).stop 2023-10-25T00:21:33.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 -- 172.21.15.107:0/1378819403 shutdown_connections 2023-10-25T00:21:33.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36a81553d0 0x7f36a81f4350 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:33.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36a8152020 0x7f36a81f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:33.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3694065eb0 0x7f3694068370 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:33.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 --2- 172.21.15.107:0/1378819403 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f36a80c8640 0x7f36a80c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:33.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 -- 172.21.15.107:0/1378819403 >> 172.21.15.107:0/1378819403 conn(0x7f36a80abd60 msgr2=0x7f36a80ab210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:33.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 -- 172.21.15.107:0/1378819403 shutdown_connections 2023-10-25T00:21:33.350 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:33.345+0000 7f368e7fc640 1 -- 172.21.15.107:0/1378819403 wait complete. 2023-10-25T00:21:33.374 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2023-10-25T00:21:33.374 INFO:tasks.mds_thrash.fs.[b]:waiting for 84 secs before thrashing 2023-10-25T00:21:34.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:34.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a9df18640 1 Processor -- start 2023-10-25T00:21:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a9df18640 1 -- start start 2023-10-25T00:21:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a9df18640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a980c2790 0x7f0a98071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a9df18640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0a9806fde0 0x7f0a980701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a9df18640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a980c44c0 con 0x7f0a98151480 2023-10-25T00:21:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a9df18640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a980c38b0 con 0x7f0a9806fde0 2023-10-25T00:21:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a9df18640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980c3b90 con 0x7f0a980c2790 2023-10-25T00:21:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a97fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0a9806fde0 0x7f0a980701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a97fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0a9806fde0 0x7f0a980701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:34968/0 (socket says 172.21.15.107:34968) 2023-10-25T00:21:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a97fff640 1 -- 172.21.15.107:0/799815391 learned_addr learned my addr 172.21.15.107:0/799815391 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a97fff640 1 -- 172.21.15.107:0/799815391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a980c2790 msgr2=0x7f0a98071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a97fff640 1 --2- 172.21.15.107:0/799815391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a980c2790 0x7f0a98071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a97fff640 1 -- 172.21.15.107:0/799815391 >> v1:172.21.15.110:6789/0 conn(0x7f0a98151480 legacy=0x7f0a9806cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.261+0000 7f0a97fff640 1 -- 172.21.15.107:0/799815391 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a980c3e70 con 0x7f0a9806fde0 2023-10-25T00:21:34.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.273+0000 7f0a97fff640 1 --2- 172.21.15.107:0/799815391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a9806fde0 0x7f0a980701e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009e50 tx=0x7f0a9001c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=592ff6aad3183083 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:34.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.281+0000 7f0a96ffd640 1 -- 172.21.15.107:0/799815391 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900a6460 con 0x7f0a9806fde0 2023-10-25T00:21:34.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.281+0000 7f0a96ffd640 1 -- 172.21.15.107:0/799815391 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a900a65c0 con 0x7f0a9806fde0 2023-10-25T00:21:34.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.285+0000 7f0a96ffd640 1 -- 172.21.15.107:0/799815391 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900a6990 con 0x7f0a9806fde0 2023-10-25T00:21:34.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.285+0000 7f0a9df18640 1 -- 172.21.15.107:0/799815391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a9806fde0 msgr2=0x7f0a980701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:34.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.285+0000 7f0a9df18640 1 --2- 172.21.15.107:0/799815391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a9806fde0 0x7f0a980701e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009e50 tx=0x7f0a9001c8d0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 -- 172.21.15.107:0/799815391 shutdown_connections 2023-10-25T00:21:34.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 --2- 172.21.15.107:0/799815391 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a9806fde0 0x7f0a980701e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 --2- 172.21.15.107:0/799815391 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a980c2790 0x7f0a98071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 -- 172.21.15.107:0/799815391 >> 172.21.15.107:0/799815391 conn(0x7f0a980c6ed0 msgr2=0x7f0a980c72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:34.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 -- 172.21.15.107:0/799815391 shutdown_connections 2023-10-25T00:21:34.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 -- 172.21.15.107:0/799815391 wait complete. 2023-10-25T00:21:34.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 Processor -- start 2023-10-25T00:21:34.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 -- start start 2023-10-25T00:21:34.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a9806fde0 0x7f0a98153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:34.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980c2790 0x7f0a980ac370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:34.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a98151480 0x7f0a98138920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:34.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a980c5840 con 0x7f0a980c2790 2023-10-25T00:21:34.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9814e0b0 con 0x7f0a98151480 2023-10-25T00:21:34.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9814e390 con 0x7f0a9806fde0 2023-10-25T00:21:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a8ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980c2790 0x7f0a980ac370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a8ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980c2790 0x7f0a980ac370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52570/0 (socket says 172.21.15.107:52570) 2023-10-25T00:21:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a8ffff640 1 -- 172.21.15.107:0/3081799818 learned_addr learned my addr 172.21.15.107:0/3081799818 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:34.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a97fff640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a98151480 0x7f0a98138920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a8ffff640 1 -- 172.21.15.107:0/3081799818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a9806fde0 msgr2=0x7f0a98153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a8ffff640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a9806fde0 0x7f0a98153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a8ffff640 1 -- 172.21.15.107:0/3081799818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a98151480 msgr2=0x7f0a98138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a8ffff640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a98151480 0x7f0a98138920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a8ffff640 1 -- 172.21.15.107:0/3081799818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a90009b30 con 0x7f0a980c2790 2023-10-25T00:21:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a8ffff640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980c2790 0x7f0a980ac370 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800b3e0 tx=0x7f0a8800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a94ff9640 1 -- 172.21.15.107:0/3081799818 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88011830 con 0x7f0a980c2790 2023-10-25T00:21:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a94ff9640 1 -- 172.21.15.107:0/3081799818 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a880119b0 con 0x7f0a980c2790 2023-10-25T00:21:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 -- 172.21.15.107:0/3081799818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9814e670 con 0x7f0a980c2790 2023-10-25T00:21:34.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a94ff9640 1 -- 172.21.15.107:0/3081799818 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a88011d60 con 0x7f0a980c2790 2023-10-25T00:21:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.289+0000 7f0a9df18640 1 -- 172.21.15.107:0/3081799818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a980c14d0 con 0x7f0a980c2790 2023-10-25T00:21:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.293+0000 7f0a9df18640 1 -- 172.21.15.107:0/3081799818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a980746e0 con 0x7f0a980c2790 2023-10-25T00:21:34.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.293+0000 7f0a94ff9640 1 -- 172.21.15.107:0/3081799818 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0a8801a400 con 0x7f0a980c2790 2023-10-25T00:21:34.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.293+0000 7f0a94ff9640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a7c065e60 0x7f0a7c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:34.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.293+0000 7f0a94ff9640 1 -- 172.21.15.107:0/3081799818 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7f0a88084af0 con 0x7f0a980c2790 2023-10-25T00:21:34.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.297+0000 7f0a977fe640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a7c065e60 0x7f0a7c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:34.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.297+0000 7f0a977fe640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a7c065e60 0x7f0a7c068320 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0a980c6d00 tx=0x7f0a80006ca0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:34.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.301+0000 7f0a94ff9640 1 -- 172.21.15.107:0/3081799818 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0a88050f80 con 0x7f0a980c2790 2023-10-25T00:21:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.569+0000 7f0a9df18640 1 -- 172.21.15.107:0/3081799818 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a980c2570 con 0x7f0a980c2790 2023-10-25T00:21:34.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.569+0000 7f0a94ff9640 1 -- 172.21.15.107:0/3081799818 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5768 (secure 0 0 0) 0x7f0a88015070 con 0x7f0a980c2790 2023-10-25T00:21:34.574 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:34.574 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9009,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":225}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:34.575 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 225 2023-10-25T00:21:34.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 -- 172.21.15.107:0/3081799818 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a7c065e60 msgr2=0x7f0a7c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:34.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a7c065e60 0x7f0a7c068320 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0a980c6d00 tx=0x7f0a80006ca0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 -- 172.21.15.107:0/3081799818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980c2790 msgr2=0x7f0a980ac370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:34.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980c2790 0x7f0a980ac370 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f0a8800b3e0 tx=0x7f0a8800b8a0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 -- 172.21.15.107:0/3081799818 shutdown_connections 2023-10-25T00:21:34.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0a98151480 0x7f0a98138920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a980c2790 0x7f0a980ac370 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0a7c065e60 0x7f0a7c068320 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 --2- 172.21.15.107:0/3081799818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0a9806fde0 0x7f0a98153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:34.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 -- 172.21.15.107:0/3081799818 >> 172.21.15.107:0/3081799818 conn(0x7f0a980c6ed0 msgr2=0x7f0a980c5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:34.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 -- 172.21.15.107:0/3081799818 shutdown_connections 2023-10-25T00:21:34.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:34.573+0000 7f0a9df18640 1 -- 172.21.15.107:0/3081799818 wait complete. 2023-10-25T00:21:34.598 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:21:36.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:36.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60dcb15640 1 Processor -- start 2023-10-25T00:21:36.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60dcb15640 1 -- start start 2023-10-25T00:21:36.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60dcb15640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f60d806a4b0 0x7f60d80ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:36.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60dcb15640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d8155680 con 0x7f60d80c8100 2023-10-25T00:21:36.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60dcb15640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d8154a70 con 0x7f60d8152040 2023-10-25T00:21:36.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60dcb15640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d8154d50 con 0x7f60d806a4b0 2023-10-25T00:21:36.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60d5d9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f60d80c8100 0x7f60d80abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50662/0 (socket says 172.21.15.107:50662) 2023-10-25T00:21:36.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60d5d9b640 1 -- 172.21.15.107:0/654244983 learned_addr learned my addr 172.21.15.107:0/654244983 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:36.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60d6d9d640 1 --2- 172.21.15.107:0/654244983 >> v2:172.21.15.110:3300/0 conn(0x7f60d806a4b0 0x7f60d80ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:36.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60d559a640 1 -- 172.21.15.107:0/654244983 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407569501 0 0) 0x7f60d8155680 con 0x7f60d80c8100 2023-10-25T00:21:36.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60d559a640 1 -- 172.21.15.107:0/654244983 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60c0003530 con 0x7f60d80c8100 2023-10-25T00:21:36.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60d6d9d640 1 -- 172.21.15.107:0/654244983 >> v1:172.21.15.110:6789/0 conn(0x7f60d8152040 legacy=0x7f60d80c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:36.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60d559a640 1 -- 172.21.15.107:0/654244983 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464555979 0 0) 0x7f60d8154a70 con 0x7f60d8152040 2023-10-25T00:21:36.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60d6d9d640 1 -- 172.21.15.107:0/654244983 >> v1:172.21.15.107:6789/0 conn(0x7f60d80c8100 legacy=0x7f60d80abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:36.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.737+0000 7f60d6d9d640 1 -- 172.21.15.107:0/654244983 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d8155030 con 0x7f60d806a4b0 2023-10-25T00:21:36.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60d6d9d640 1 --2- 172.21.15.107:0/654244983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60d806a4b0 0x7f60d80ac410 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009d90 tx=0x7f60d001c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd8e21eb86742033 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:36.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60d559a640 1 -- 172.21.15.107:0/654244983 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d00a5890 con 0x7f60d806a4b0 2023-10-25T00:21:36.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60d559a640 1 -- 172.21.15.107:0/654244983 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d00a59f0 con 0x7f60d806a4b0 2023-10-25T00:21:36.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60d559a640 1 -- 172.21.15.107:0/654244983 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d00a5d80 con 0x7f60d806a4b0 2023-10-25T00:21:36.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60dcb15640 1 -- 172.21.15.107:0/654244983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60d806a4b0 msgr2=0x7f60d80ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:36.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60dcb15640 1 --2- 172.21.15.107:0/654244983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60d806a4b0 0x7f60d80ac410 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009d90 tx=0x7f60d001c8c0 comp rx=0 tx=0).stop 2023-10-25T00:21:36.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60dcb15640 1 -- 172.21.15.107:0/654244983 shutdown_connections 2023-10-25T00:21:36.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60dcb15640 1 --2- 172.21.15.107:0/654244983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60d806a4b0 0x7f60d80ac410 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:36.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60dcb15640 1 -- 172.21.15.107:0/654244983 >> 172.21.15.107:0/654244983 conn(0x7f60d80ca200 msgr2=0x7f60d80ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:36.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60dcb15640 1 -- 172.21.15.107:0/654244983 shutdown_connections 2023-10-25T00:21:36.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60dcb15640 1 -- 172.21.15.107:0/654244983 wait complete. 2023-10-25T00:21:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60dcb15640 1 Processor -- start 2023-10-25T00:21:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.741+0000 7f60dcb15640 1 -- start start 2023-10-25T00:21:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60dcb15640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60d806a4b0 0x7f60d8148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60dcb15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d80c8100 0x7f60d81efb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:36.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60dcb15640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60d8152040 0x7f60d81f0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60dcb15640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d81f05a0 con 0x7f60d80c8100 2023-10-25T00:21:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60dcb15640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d80bf250 con 0x7f60d806a4b0 2023-10-25T00:21:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60dcb15640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d81f07d0 con 0x7f60d8152040 2023-10-25T00:21:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60d5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d80c8100 0x7f60d81efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:36.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60d5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d80c8100 0x7f60d81efb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52574/0 (socket says 172.21.15.107:52574) 2023-10-25T00:21:36.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60d5d9b640 1 -- 172.21.15.107:0/88903181 learned_addr learned my addr 172.21.15.107:0/88903181 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:36.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60d659c640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60d806a4b0 0x7f60d8148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:36.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60d6d9d640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60d8152040 0x7f60d81f0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:36.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60d5d9b640 1 -- 172.21.15.107:0/88903181 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60d8152040 msgr2=0x7f60d81f0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:36.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60d5d9b640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60d8152040 0x7f60d81f0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:36.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60d5d9b640 1 -- 172.21.15.107:0/88903181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60d806a4b0 msgr2=0x7f60d8148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:36.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60d5d9b640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60d806a4b0 0x7f60d8148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:36.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60d5d9b640 1 -- 172.21.15.107:0/88903181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d0009a70 con 0x7f60d80c8100 2023-10-25T00:21:36.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60d5d9b640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d80c8100 0x7f60d81efb20 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00c390 tx=0x7f60cc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:36.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60c6ffd640 1 -- 172.21.15.107:0/88903181 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc005cf0 con 0x7f60d80c8100 2023-10-25T00:21:36.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60c6ffd640 1 -- 172.21.15.107:0/88903181 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60cc005e70 con 0x7f60d80c8100 2023-10-25T00:21:36.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60c6ffd640 1 -- 172.21.15.107:0/88903181 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc0072f0 con 0x7f60d80c8100 2023-10-25T00:21:36.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60dcb15640 1 -- 172.21.15.107:0/88903181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d80bdd40 con 0x7f60d80c8100 2023-10-25T00:21:36.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60dcb15640 1 -- 172.21.15.107:0/88903181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d80c0a70 con 0x7f60d80c8100 2023-10-25T00:21:36.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60dcb15640 1 -- 172.21.15.107:0/88903181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d80c0c60 con 0x7f60d80c8100 2023-10-25T00:21:36.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60c6ffd640 1 -- 172.21.15.107:0/88903181 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f60cc007450 con 0x7f60d80c8100 2023-10-25T00:21:36.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60c6ffd640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60b8065e90 0x7f60b8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:36.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.745+0000 7f60c6ffd640 1 -- 172.21.15.107:0/88903181 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 10659+0+0 (secure 0 0 0) 0x7f60cc083ff0 con 0x7f60d80c8100 2023-10-25T00:21:36.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.749+0000 7f60d659c640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60b8065e90 0x7f60b8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:36.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.749+0000 7f60d659c640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60b8065e90 0x7f60b8068350 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f60c8000f70 tx=0x7f60c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:36.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:36.757+0000 7f60c6ffd640 1 -- 172.21.15.107:0/88903181 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f60cc050560 con 0x7f60d80c8100 2023-10-25T00:21:37.021 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.017+0000 7f60dcb15640 1 -- 172.21.15.107:0/88903181 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d80746e0 con 0x7f60d80c8100 2023-10-25T00:21:37.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.021+0000 7f60c6ffd640 1 -- 172.21.15.107:0/88903181 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5768 (secure 0 0 0) 0x7f60cc017090 con 0x7f60d80c8100 2023-10-25T00:21:37.025 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:37.025 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9009,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":225}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:20:23.056107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8802},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8802":{"gid":8802,"name":"a","rank":0,"incarnation":216,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/2187059523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2187059523},{"type":"v1","addr":"172.21.15.107:6835","nonce":2187059523}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:37.025 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 225 2023-10-25T00:21:37.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 -- 172.21.15.107:0/88903181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60b8065e90 msgr2=0x7f60b8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:37.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60b8065e90 0x7f60b8068350 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f60c8000f70 tx=0x7f60c8009040 comp rx=0 tx=0).stop 2023-10-25T00:21:37.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 -- 172.21.15.107:0/88903181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d80c8100 msgr2=0x7f60d81efb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:37.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d80c8100 0x7f60d81efb20 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00c390 tx=0x7f60cc00c850 comp rx=0 tx=0).stop 2023-10-25T00:21:37.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 -- 172.21.15.107:0/88903181 shutdown_connections 2023-10-25T00:21:37.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60b8065e90 0x7f60b8068350 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:37.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60d8152040 0x7f60d81f0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:37.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60d80c8100 0x7f60d81efb20 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:37.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 --2- 172.21.15.107:0/88903181 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60d806a4b0 0x7f60d8148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:37.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 -- 172.21.15.107:0/88903181 >> 172.21.15.107:0/88903181 conn(0x7f60d80ca200 msgr2=0x7f60d80ad100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:37.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 -- 172.21.15.107:0/88903181 shutdown_connections 2023-10-25T00:21:37.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:37.025+0000 7f60dcb15640 1 -- 172.21.15.107:0/88903181 wait complete. 2023-10-25T00:21:37.047 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:21:39.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:39.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.205+0000 7fd6dffff640 1 Processor -- start 2023-10-25T00:21:39.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.205+0000 7fd6dffff640 1 -- start start 2023-10-25T00:21:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.205+0000 7fd6dffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd6d80d5820 0x7fd6d80483b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.205+0000 7fd6dffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d80410e0 0x7fd6d80d87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.205+0000 7fd6dffff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d804a5e0 con 0x7fd6d804b930 2023-10-25T00:21:39.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.205+0000 7fd6dffff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d80dc690 con 0x7fd6d80d5820 2023-10-25T00:21:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.205+0000 7fd6dffff640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d80dc0d0 con 0x7fd6d80410e0 2023-10-25T00:21:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.205+0000 7fd6deffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd6d80d5820 0x7fd6d80483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.209+0000 7fd6deffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd6d80d5820 0x7fd6d80483b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52578/0 (socket says 172.21.15.107:52578) 2023-10-25T00:21:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.209+0000 7fd6deffd640 1 -- 172.21.15.107:0/1912837680 learned_addr learned my addr 172.21.15.107:0/1912837680 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.209+0000 7fd6df7fe640 1 --2- 172.21.15.107:0/1912837680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d80410e0 0x7fd6d80d87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.209+0000 7fd6df7fe640 1 -- 172.21.15.107:0/1912837680 >> v2:172.21.15.107:3300/0 conn(0x7fd6d80d5820 msgr2=0x7fd6d80483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.209+0000 7fd6df7fe640 1 --2- 172.21.15.107:0/1912837680 >> v2:172.21.15.107:3300/0 conn(0x7fd6d80d5820 0x7fd6d80483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.209+0000 7fd6df7fe640 1 -- 172.21.15.107:0/1912837680 >> v1:172.21.15.107:6789/0 conn(0x7fd6d804b930 legacy=0x7fd6d80d80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:39.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.209+0000 7fd6df7fe640 1 -- 172.21.15.107:0/1912837680 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d80dc3b0 con 0x7fd6d80410e0 2023-10-25T00:21:39.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.213+0000 7fd6df7fe640 1 --2- 172.21.15.107:0/1912837680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d80410e0 0x7fd6d80d87e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00097c0 tx=0x7fd6d001c280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e12bfd1499614bf server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:39.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.221+0000 7fd6ddffb640 1 -- 172.21.15.107:0/1912837680 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00a4c90 con 0x7fd6d80410e0 2023-10-25T00:21:39.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.221+0000 7fd6ddffb640 1 -- 172.21.15.107:0/1912837680 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d00a4df0 con 0x7fd6d80410e0 2023-10-25T00:21:39.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.221+0000 7fd6ddffb640 1 -- 172.21.15.107:0/1912837680 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d00ac660 con 0x7fd6d80410e0 2023-10-25T00:21:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.225+0000 7fd6dffff640 1 -- 172.21.15.107:0/1912837680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d80410e0 msgr2=0x7fd6d80d87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.225+0000 7fd6dffff640 1 --2- 172.21.15.107:0/1912837680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d80410e0 0x7fd6d80d87e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00097c0 tx=0x7fd6d001c280 comp rx=0 tx=0).stop 2023-10-25T00:21:39.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.229+0000 7fd6dffff640 1 -- 172.21.15.107:0/1912837680 shutdown_connections 2023-10-25T00:21:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.229+0000 7fd6dffff640 1 --2- 172.21.15.107:0/1912837680 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d80410e0 0x7fd6d80d87e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.229+0000 7fd6dffff640 1 --2- 172.21.15.107:0/1912837680 >> v2:172.21.15.107:3300/0 conn(0x7fd6d80d5820 0x7fd6d80483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.229+0000 7fd6dffff640 1 -- 172.21.15.107:0/1912837680 >> 172.21.15.107:0/1912837680 conn(0x7fd6d802f600 msgr2=0x7fd6d802fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:39.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.237+0000 7fd6dffff640 1 -- 172.21.15.107:0/1912837680 shutdown_connections 2023-10-25T00:21:39.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.237+0000 7fd6dffff640 1 -- 172.21.15.107:0/1912837680 wait complete. 2023-10-25T00:21:39.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.237+0000 7fd6dffff640 1 Processor -- start 2023-10-25T00:21:39.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.237+0000 7fd6dffff640 1 -- start start 2023-10-25T00:21:39.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6dffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6d80410e0 0x7fd6d8043110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6dffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6d80d5820 0x7fd6d80e2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6dffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d804bf60 0x7fd6d804c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6dffff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8043c00 con 0x7fd6d80d5820 2023-10-25T00:21:39.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6dffff640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d8044180 con 0x7fd6d80410e0 2023-10-25T00:21:39.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6dffff640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d80d7b60 con 0x7fd6d804bf60 2023-10-25T00:21:39.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6df7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d804bf60 0x7fd6d804c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:39.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6df7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d804bf60 0x7fd6d804c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:47080/0 (socket says 172.21.15.107:47080) 2023-10-25T00:21:39.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6df7fe640 1 -- 172.21.15.107:0/1985832581 learned_addr learned my addr 172.21.15.107:0/1985832581 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:39.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6de7fc640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6d80d5820 0x7fd6d80e2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:39.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6de7fc640 1 -- 172.21.15.107:0/1985832581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d804bf60 msgr2=0x7fd6d804c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:39.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6de7fc640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d804bf60 0x7fd6d804c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6de7fc640 1 -- 172.21.15.107:0/1985832581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6d80410e0 msgr2=0x7fd6d8043110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:39.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6de7fc640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6d80410e0 0x7fd6d8043110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6de7fc640 1 -- 172.21.15.107:0/1985832581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d0009420 con 0x7fd6d80d5820 2023-10-25T00:21:39.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6de7fc640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6d80d5820 0x7fd6d80e2090 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400c7b0 tx=0x7fd6d400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:39.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6c77fe640 1 -- 172.21.15.107:0/1985832581 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4005270 con 0x7fd6d80d5820 2023-10-25T00:21:39.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.241+0000 7fd6c77fe640 1 -- 172.21.15.107:0/1985832581 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d40053d0 con 0x7fd6d80d5820 2023-10-25T00:21:39.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.245+0000 7fd6c77fe640 1 -- 172.21.15.107:0/1985832581 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d400a880 con 0x7fd6d80d5820 2023-10-25T00:21:39.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.245+0000 7fd6dffff640 1 -- 172.21.15.107:0/1985832581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d80d7e40 con 0x7fd6d80d5820 2023-10-25T00:21:39.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.245+0000 7fd6dffff640 1 -- 172.21.15.107:0/1985832581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d80e25d0 con 0x7fd6d80d5820 2023-10-25T00:21:39.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.245+0000 7fd6dffff640 1 -- 172.21.15.107:0/1985832581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c0005260 con 0x7fd6d80d5820 2023-10-25T00:21:39.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.253+0000 7fd6c77fe640 1 -- 172.21.15.107:0/1985832581 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd6d400a9e0 con 0x7fd6d80d5820 2023-10-25T00:21:39.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.253+0000 7fd6c77fe640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd6cc065ee0 0x7fd6cc0683a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.253+0000 7fd6deffd640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd6cc065ee0 0x7fd6cc0683a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:39.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.257+0000 7fd6c77fe640 1 -- 172.21.15.107:0/1985832581 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fd6d401ee50 con 0x7fd6d80d5820 2023-10-25T00:21:39.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.257+0000 7fd6deffd640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd6cc065ee0 0x7fd6cc0683a0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd6e002c850 tx=0x7fd6e008cf90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:39.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.261+0000 7fd6c77fe640 1 -- 172.21.15.107:0/1985832581 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd6d4051170 con 0x7fd6d80d5820 2023-10-25T00:21:39.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.609+0000 7fd6dffff640 1 -- 172.21.15.107:0/1985832581 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6c00057e0 con 0x7fd6d80d5820 2023-10-25T00:21:39.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.609+0000 7fd6c77fe640 1 -- 172.21.15.107:0/1985832581 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) v1 ==== 95+0+5007 (secure 0 0 0) 0x7fd6d4051170 con 0x7fd6d80d5820 2023-10-25T00:21:39.614 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:39.614 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":226,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:21:39.173788+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9009":{"gid":9009,"name":"c","rank":0,"incarnation":226,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:39.614 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 226 2023-10-25T00:21:39.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.621+0000 7fd6c57fa640 1 -- 172.21.15.107:0/1985832581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd6cc065ee0 msgr2=0x7fd6cc0683a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:39.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.621+0000 7fd6c57fa640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd6cc065ee0 0x7fd6cc0683a0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fd6e002c850 tx=0x7fd6e008cf90 comp rx=0 tx=0).stop 2023-10-25T00:21:39.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.621+0000 7fd6c57fa640 1 -- 172.21.15.107:0/1985832581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6d80d5820 msgr2=0x7fd6d80e2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:39.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.621+0000 7fd6c57fa640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6d80d5820 0x7fd6d80e2090 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400c7b0 tx=0x7fd6d400cc70 comp rx=0 tx=0).stop 2023-10-25T00:21:39.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.621+0000 7fd6c57fa640 1 -- 172.21.15.107:0/1985832581 shutdown_connections 2023-10-25T00:21:39.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.621+0000 7fd6c57fa640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd6cc065ee0 0x7fd6cc0683a0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.621+0000 7fd6c57fa640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d804bf60 0x7fd6d804c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.627 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.621+0000 7fd6c57fa640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd6d80d5820 0x7fd6d80e2090 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.621+0000 7fd6c57fa640 1 --2- 172.21.15.107:0/1985832581 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6d80410e0 0x7fd6d8043110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.621+0000 7fd6c57fa640 1 -- 172.21.15.107:0/1985832581 >> 172.21.15.107:0/1985832581 conn(0x7fd6d802f600 msgr2=0x7fd6d804e200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:39.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.625+0000 7fd6c57fa640 1 -- 172.21.15.107:0/1985832581 shutdown_connections 2023-10-25T00:21:39.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.625+0000 7fd6c57fa640 1 -- 172.21.15.107:0/1985832581 wait complete. 2023-10-25T00:21:39.653 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2023-10-25T00:21:39.653 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:21:39.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:39.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.821+0000 7f1e6db5e640 1 Processor -- start 2023-10-25T00:21:39.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.821+0000 7f1e6db5e640 1 -- start start 2023-10-25T00:21:39.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e6db5e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1e68070b00 0x7f1e680c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e6db5e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e6806df30 0x7f1e6806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e6db5e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e680c22f0 con 0x7f1e6814eb70 2023-10-25T00:21:39.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e6db5e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e68071700 con 0x7f1e6806df30 2023-10-25T00:21:39.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e6db5e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e680c25d0 con 0x7f1e68070b00 2023-10-25T00:21:39.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e67fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e6806df30 0x7f1e6806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:39.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e677fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1e6814eb70 0x7f1e681514e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50680/0 (socket says 172.21.15.107:50680) 2023-10-25T00:21:39.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e677fe640 1 -- 172.21.15.107:0/2731838701 learned_addr learned my addr 172.21.15.107:0/2731838701 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:39.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e66ffd640 1 --2- 172.21.15.107:0/2731838701 >> v2:172.21.15.110:3300/0 conn(0x7f1e68070b00 0x7f1e680c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:39.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e67fff640 1 -- 172.21.15.107:0/2731838701 >> v2:172.21.15.110:3300/0 conn(0x7f1e68070b00 msgr2=0x7f1e680c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:39.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e67fff640 1 --2- 172.21.15.107:0/2731838701 >> v2:172.21.15.110:3300/0 conn(0x7f1e68070b00 0x7f1e680c33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e67fff640 1 -- 172.21.15.107:0/2731838701 >> v1:172.21.15.107:6789/0 conn(0x7f1e6814eb70 legacy=0x7f1e681514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:39.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e67fff640 1 -- 172.21.15.107:0/2731838701 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e680c2be0 con 0x7f1e6806df30 2023-10-25T00:21:39.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.825+0000 7f1e667fc640 1 -- 172.21.15.107:0/2731838701 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151106562 0 0) 0x7f1e680c22f0 con 0x7f1e6814eb70 2023-10-25T00:21:39.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.829+0000 7f1e67fff640 1 --2- 172.21.15.107:0/2731838701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e6806df30 0x7f1e6806e3a0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f1e680c1e70 tx=0x7f1e6001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1570279796ae4fef server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:39.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.829+0000 7f1e667fc640 1 -- 172.21.15.107:0/2731838701 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e600a59f0 con 0x7f1e6806df30 2023-10-25T00:21:39.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.829+0000 7f1e667fc640 1 -- 172.21.15.107:0/2731838701 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e600a5b50 con 0x7f1e6806df30 2023-10-25T00:21:39.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.829+0000 7f1e667fc640 1 -- 172.21.15.107:0/2731838701 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e600ad440 con 0x7f1e6806df30 2023-10-25T00:21:39.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.829+0000 7f1e6db5e640 1 -- 172.21.15.107:0/2731838701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e6806df30 msgr2=0x7f1e6806e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:39.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.829+0000 7f1e6db5e640 1 --2- 172.21.15.107:0/2731838701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e6806df30 0x7f1e6806e3a0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f1e680c1e70 tx=0x7f1e6001c910 comp rx=0 tx=0).stop 2023-10-25T00:21:39.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.829+0000 7f1e6db5e640 1 -- 172.21.15.107:0/2731838701 shutdown_connections 2023-10-25T00:21:39.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.829+0000 7f1e6db5e640 1 --2- 172.21.15.107:0/2731838701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e6806df30 0x7f1e6806e3a0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.829+0000 7f1e6db5e640 1 --2- 172.21.15.107:0/2731838701 >> v2:172.21.15.110:3300/0 conn(0x7f1e68070b00 0x7f1e680c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.829+0000 7f1e6db5e640 1 -- 172.21.15.107:0/2731838701 >> 172.21.15.107:0/2731838701 conn(0x7f1e680c39a0 msgr2=0x7f1e680c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:39.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.829+0000 7f1e6db5e640 1 -- 172.21.15.107:0/2731838701 shutdown_connections 2023-10-25T00:21:39.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e6db5e640 1 -- 172.21.15.107:0/2731838701 wait complete. 2023-10-25T00:21:39.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e6db5e640 1 Processor -- start 2023-10-25T00:21:39.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e6db5e640 1 -- start start 2023-10-25T00:21:39.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e6db5e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e6806df30 0x7f1e68154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e6db5e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e68070b00 0x7f1e680aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e6db5e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e6814eb70 0x7f1e68144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e6db5e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e68152ae0 con 0x7f1e6806df30 2023-10-25T00:21:39.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e6db5e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6806c470 con 0x7f1e68070b00 2023-10-25T00:21:39.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e6db5e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6806c6e0 con 0x7f1e6814eb70 2023-10-25T00:21:39.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e6806df30 0x7f1e68154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:39.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e6806df30 0x7f1e68154310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52606/0 (socket says 172.21.15.107:52606) 2023-10-25T00:21:39.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e677fe640 1 -- 172.21.15.107:0/1164596490 learned_addr learned my addr 172.21.15.107:0/1164596490 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:39.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e67fff640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e6814eb70 0x7f1e68144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:39.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e66ffd640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e68070b00 0x7f1e680aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:39.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e677fe640 1 -- 172.21.15.107:0/1164596490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e6814eb70 msgr2=0x7f1e68144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:39.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e677fe640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e6814eb70 0x7f1e68144c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e677fe640 1 -- 172.21.15.107:0/1164596490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e68070b00 msgr2=0x7f1e680aa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:39.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e677fe640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e68070b00 0x7f1e680aa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:39.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e677fe640 1 -- 172.21.15.107:0/1164596490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e60009a70 con 0x7f1e6806df30 2023-10-25T00:21:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e677fe640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e6806df30 0x7f1e68154310 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f1e580011f0 tx=0x7f1e5800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e4ffff640 1 -- 172.21.15.107:0/1164596490 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e580074e0 con 0x7f1e6806df30 2023-10-25T00:21:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e4ffff640 1 -- 172.21.15.107:0/1164596490 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e58007640 con 0x7f1e6806df30 2023-10-25T00:21:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e4ffff640 1 -- 172.21.15.107:0/1164596490 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e58004a10 con 0x7f1e6806df30 2023-10-25T00:21:39.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e6db5e640 1 -- 172.21.15.107:0/1164596490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e680abe00 con 0x7f1e6806df30 2023-10-25T00:21:39.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.833+0000 7f1e6db5e640 1 -- 172.21.15.107:0/1164596490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e681451b0 con 0x7f1e6806df30 2023-10-25T00:21:39.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.837+0000 7f1e6db5e640 1 -- 172.21.15.107:0/1164596490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e680ac320 con 0x7f1e6806df30 2023-10-25T00:21:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.845+0000 7f1e4ffff640 1 -- 172.21.15.107:0/1164596490 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1e58004b70 con 0x7f1e6806df30 2023-10-25T00:21:39.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.845+0000 7f1e4ffff640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e54065f30 0x7f1e540683f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:39.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.845+0000 7f1e4ffff640 1 -- 172.21.15.107:0/1164596490 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f1e58084050 con 0x7f1e6806df30 2023-10-25T00:21:39.854 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.849+0000 7f1e66ffd640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e54065f30 0x7f1e540683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:39.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.853+0000 7f1e66ffd640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e54065f30 0x7f1e540683f0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f1e681ee950 tx=0x7f1e5c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:39.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:39.857+0000 7f1e4ffff640 1 -- 172.21.15.107:0/1164596490 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1e680ac320 con 0x7f1e6806df30 2023-10-25T00:21:40.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.169+0000 7f1e6db5e640 1 -- 172.21.15.107:0/1164596490 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e680ac320 con 0x7f1e6806df30 2023-10-25T00:21:40.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.169+0000 7f1e4ffff640 1 -- 172.21.15.107:0/1164596490 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) v1 ==== 95+0+5007 (secure 0 0 0) 0x7f1e680ac320 con 0x7f1e6806df30 2023-10-25T00:21:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:40.174 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":226,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:21:39.173788+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9009":{"gid":9009,"name":"c","rank":0,"incarnation":226,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:40.175 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 226 2023-10-25T00:21:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.173+0000 7f1e6db5e640 1 -- 172.21.15.107:0/1164596490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e54065f30 msgr2=0x7f1e540683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:40.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.173+0000 7f1e6db5e640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e54065f30 0x7f1e540683f0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f1e681ee950 tx=0x7f1e5c009190 comp rx=0 tx=0).stop 2023-10-25T00:21:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.173+0000 7f1e6db5e640 1 -- 172.21.15.107:0/1164596490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e6806df30 msgr2=0x7f1e68154310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.173+0000 7f1e6db5e640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e6806df30 0x7f1e68154310 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f1e580011f0 tx=0x7f1e5800c9b0 comp rx=0 tx=0).stop 2023-10-25T00:21:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.173+0000 7f1e6db5e640 1 -- 172.21.15.107:0/1164596490 shutdown_connections 2023-10-25T00:21:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.173+0000 7f1e6db5e640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1e54065f30 0x7f1e540683f0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.173+0000 7f1e6db5e640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e6814eb70 0x7f1e68144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.173+0000 7f1e6db5e640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e68070b00 0x7f1e680aa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.173+0000 7f1e6db5e640 1 --2- 172.21.15.107:0/1164596490 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e6806df30 0x7f1e68154310 secure :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f1e580011f0 tx=0x7f1e5800c9b0 comp rx=0 tx=0).stop 2023-10-25T00:21:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.173+0000 7f1e6db5e640 1 -- 172.21.15.107:0/1164596490 >> 172.21.15.107:0/1164596490 conn(0x7f1e680c39a0 msgr2=0x7f1e680c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:40.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.177+0000 7f1e6db5e640 1 -- 172.21.15.107:0/1164596490 shutdown_connections 2023-10-25T00:21:40.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:40.177+0000 7f1e6db5e640 1 -- 172.21.15.107:0/1164596490 wait complete. 2023-10-25T00:21:40.204 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 226, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 226, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_9009": { "addr": "172.21.15.107:6835/1528689730", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 1528689730, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 1528689730, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9009, "incarnation": 226, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:21:39.173788+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9009 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 224, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7483": { "addr": "172.21.15.110:6834/47497412", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 47497412, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 47497412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7483, "incarnation": 221, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:21:22.342534+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7483 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2437148", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2437148, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2437148, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 190, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6443, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:21:42.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:42.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960f707640 1 Processor -- start 2023-10-25T00:21:42.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960f707640 1 -- start start 2023-10-25T00:21:42.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960f707640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f960814fd20 0x7f9608071460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:42.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960f707640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9608069d40 0x7f960806a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:42.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960f707640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9608153080 con 0x7f96080c67c0 2023-10-25T00:21:42.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960f707640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9608150100 con 0x7f960814fd20 2023-10-25T00:21:42.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960f707640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9608071040 con 0x7f9608069d40 2023-10-25T00:21:42.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960cca2640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f96080c67c0 0x7f96080719a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50686/0 (socket says 172.21.15.107:50686) 2023-10-25T00:21:42.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960cca2640 1 -- 172.21.15.107:0/1946915350 learned_addr learned my addr 172.21.15.107:0/1946915350 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:21:42.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960d4a3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f960814fd20 0x7f9608071460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:42.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f95fffff640 1 -- 172.21.15.107:0/1946915350 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334965902 0 0) 0x7f9608153080 con 0x7f96080c67c0 2023-10-25T00:21:42.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f95fffff640 1 -- 172.21.15.107:0/1946915350 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f4003530 con 0x7f96080c67c0 2023-10-25T00:21:42.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960d4a3640 1 -- 172.21.15.107:0/1946915350 >> v2:172.21.15.110:3300/0 conn(0x7f9608069d40 msgr2=0x7f960806a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:42.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960d4a3640 1 --2- 172.21.15.107:0/1946915350 >> v2:172.21.15.110:3300/0 conn(0x7f9608069d40 0x7f960806a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:42.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960d4a3640 1 -- 172.21.15.107:0/1946915350 >> v1:172.21.15.107:6789/0 conn(0x7f96080c67c0 legacy=0x7f96080719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:42.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.337+0000 7f960d4a3640 1 -- 172.21.15.107:0/1946915350 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96081535b0 con 0x7f960814fd20 2023-10-25T00:21:42.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f960d4a3640 1 --2- 172.21.15.107:0/1946915350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f960814fd20 0x7f9608071460 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f96080c16c0 tx=0x7f960401c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f26f0efee8ee42b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:42.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f95fffff640 1 -- 172.21.15.107:0/1946915350 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96040ab600 con 0x7f960814fd20 2023-10-25T00:21:42.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f95fffff640 1 -- 172.21.15.107:0/1946915350 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96040a7050 con 0x7f960814fd20 2023-10-25T00:21:42.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f95fffff640 1 -- 172.21.15.107:0/1946915350 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96040ab980 con 0x7f960814fd20 2023-10-25T00:21:42.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f960f707640 1 -- 172.21.15.107:0/1946915350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f960814fd20 msgr2=0x7f9608071460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:42.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f960f707640 1 --2- 172.21.15.107:0/1946915350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f960814fd20 0x7f9608071460 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f96080c16c0 tx=0x7f960401c2f0 comp rx=0 tx=0).stop 2023-10-25T00:21:42.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f960f707640 1 -- 172.21.15.107:0/1946915350 shutdown_connections 2023-10-25T00:21:42.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f960f707640 1 --2- 172.21.15.107:0/1946915350 >> v2:172.21.15.110:3300/0 conn(0x7f9608069d40 0x7f960806a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f960f707640 1 --2- 172.21.15.107:0/1946915350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f960814fd20 0x7f9608071460 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f960f707640 1 -- 172.21.15.107:0/1946915350 >> 172.21.15.107:0/1946915350 conn(0x7f96080c5cd0 msgr2=0x7f96080c60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f960f707640 1 -- 172.21.15.107:0/1946915350 shutdown_connections 2023-10-25T00:21:42.349 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.341+0000 7f960f707640 1 -- 172.21.15.107:0/1946915350 wait complete. 2023-10-25T00:21:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960f707640 1 Processor -- start 2023-10-25T00:21:42.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960f707640 1 -- start start 2023-10-25T00:21:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960f707640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9608069d40 0x7f9608151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960f707640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96080c67c0 0x7f9608143cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:42.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960f707640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f960814fd20 0x7f960813f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:42.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960f707640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9608141a90 con 0x7f960814fd20 2023-10-25T00:21:42.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960f707640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96081409f0 con 0x7f96080c67c0 2023-10-25T00:21:42.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960f707640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9608140cd0 con 0x7f9608069d40 2023-10-25T00:21:42.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960d4a3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9608069d40 0x7f9608151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:42.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960d4a3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9608069d40 0x7f9608151720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:47102/0 (socket says 172.21.15.107:47102) 2023-10-25T00:21:42.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960d4a3640 1 -- 172.21.15.107:0/77535358 learned_addr learned my addr 172.21.15.107:0/77535358 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:42.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960cca2640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96080c67c0 0x7f9608143cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:42.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960d4a3640 1 -- 172.21.15.107:0/77535358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96080c67c0 msgr2=0x7f9608143cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:42.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960d4a3640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96080c67c0 0x7f9608143cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:42.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960d4a3640 1 -- 172.21.15.107:0/77535358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f960814fd20 msgr2=0x7f960813f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:42.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960d4a3640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f960814fd20 0x7f960813f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:42.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960d4a3640 1 -- 172.21.15.107:0/77535358 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9604009420 con 0x7f9608069d40 2023-10-25T00:21:42.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.345+0000 7f960d4a3640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9608069d40 0x7f9608151720 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9604018f00 tx=0x7f9604019090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:42.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.349+0000 7f95fdffb640 1 -- 172.21.15.107:0/77535358 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96040b4410 con 0x7f9608069d40 2023-10-25T00:21:42.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.349+0000 7f95fdffb640 1 -- 172.21.15.107:0/77535358 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96040b4570 con 0x7f9608069d40 2023-10-25T00:21:42.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.349+0000 7f95fdffb640 1 -- 172.21.15.107:0/77535358 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96040b48e0 con 0x7f9608069d40 2023-10-25T00:21:42.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.349+0000 7f960f707640 1 -- 172.21.15.107:0/77535358 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9608140fb0 con 0x7f9608069d40 2023-10-25T00:21:42.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.353+0000 7f960f707640 1 -- 172.21.15.107:0/77535358 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9608141570 con 0x7f9608069d40 2023-10-25T00:21:42.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.353+0000 7f960f707640 1 -- 172.21.15.107:0/77535358 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96080746e0 con 0x7f9608069d40 2023-10-25T00:21:42.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.357+0000 7f95fdffb640 1 -- 172.21.15.107:0/77535358 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f96040b2040 con 0x7f9608069d40 2023-10-25T00:21:42.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.361+0000 7f95fdffb640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f95ec065e60 0x7f95ec068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:42.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.361+0000 7f95fdffb640 1 -- 172.21.15.107:0/77535358 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f960411f080 con 0x7f9608069d40 2023-10-25T00:21:42.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.361+0000 7f960cca2640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f95ec065e60 0x7f95ec068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:42.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.361+0000 7f960cca2640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f95ec065e60 0x7f95ec068320 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f96080c6120 tx=0x7f95f8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:42.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.365+0000 7f95fdffb640 1 -- 172.21.15.107:0/77535358 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f96040eb4e0 con 0x7f9608069d40 2023-10-25T00:21:42.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.637+0000 7f960f707640 1 -- 172.21.15.107:0/77535358 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9608140e10 con 0x7f9608069d40 2023-10-25T00:21:42.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.637+0000 7f95fdffb640 1 -- 172.21.15.107:0/77535358 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+5007 (secure 0 0 0) 0x7f9608140e10 con 0x7f9608069d40 2023-10-25T00:21:42.641 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:42.641 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:21:42.241858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9009":{"gid":9009,"name":"c","rank":0,"incarnation":226,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:42.642 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 229 2023-10-25T00:21:42.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 -- 172.21.15.107:0/77535358 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f95ec065e60 msgr2=0x7f95ec068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:42.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f95ec065e60 0x7f95ec068320 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f96080c6120 tx=0x7f95f8009040 comp rx=0 tx=0).stop 2023-10-25T00:21:42.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 -- 172.21.15.107:0/77535358 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9608069d40 msgr2=0x7f9608151720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:42.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9608069d40 0x7f9608151720 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9604018f00 tx=0x7f9604019090 comp rx=0 tx=0).stop 2023-10-25T00:21:42.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 -- 172.21.15.107:0/77535358 shutdown_connections 2023-10-25T00:21:42.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f960814fd20 0x7f960813f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:42.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96080c67c0 0x7f9608143cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:42.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f95ec065e60 0x7f95ec068320 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:42.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 --2- 172.21.15.107:0/77535358 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9608069d40 0x7f9608151720 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:42.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 -- 172.21.15.107:0/77535358 >> 172.21.15.107:0/77535358 conn(0x7f96080c5cd0 msgr2=0x7f96081498b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:42.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 -- 172.21.15.107:0/77535358 shutdown_connections 2023-10-25T00:21:42.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:42.641+0000 7f960f707640 1 -- 172.21.15.107:0/77535358 wait complete. 2023-10-25T00:21:42.669 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=8802 2023-10-25T00:21:42.669 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.a 2023-10-25T00:21:43.075 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:21:43.073+0000 7fa2b0962640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:21:45.670 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2023-10-25T00:21:45.671 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:21:45.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:21:45.673 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:21:45.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:21:45.788 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:21:45.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.813+0000 7f5b2787c640 1 Processor -- start 2023-10-25T00:21:45.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.813+0000 7f5b2787c640 1 -- start start 2023-10-25T00:21:45.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.813+0000 7f5b2787c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b2014fc30 0x7f5b20153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:45.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.813+0000 7f5b2787c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5b200c5c20 0x7f5b2006df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:45.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.813+0000 7f5b2787c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b2006e470 0x7f5b200c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:45.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.813+0000 7f5b2787c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b20072330 con 0x7f5b2014fc30 2023-10-25T00:21:45.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.813+0000 7f5b2787c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b20155ca0 con 0x7f5b200c5c20 2023-10-25T00:21:45.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.813+0000 7f5b2787c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b20155f80 con 0x7f5b2006e470 2023-10-25T00:21:45.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.813+0000 7f5b25618640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b2014fc30 0x7f5b20153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:45.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b25618640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5b2014fc30 0x7f5b20153190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45604/0 (socket says 172.21.15.107:45604) 2023-10-25T00:21:45.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b25618640 1 -- 172.21.15.107:0/1844040748 learned_addr learned my addr 172.21.15.107:0/1844040748 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:45.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b25e19640 1 --2- 172.21.15.107:0/1844040748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b2006e470 0x7f5b200c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:45.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b25618640 1 -- 172.21.15.107:0/1844040748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b2006e470 msgr2=0x7f5b200c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:45.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b25618640 1 --2- 172.21.15.107:0/1844040748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b2006e470 0x7f5b200c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:45.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b25618640 1 -- 172.21.15.107:0/1844040748 >> v2:172.21.15.110:3300/0 conn(0x7f5b200c5c20 msgr2=0x7f5b2006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:21:45.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b25618640 1 --2- 172.21.15.107:0/1844040748 >> v2:172.21.15.110:3300/0 conn(0x7f5b200c5c20 0x7f5b2006df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:45.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b25618640 1 -- 172.21.15.107:0/1844040748 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b20156260 con 0x7f5b2014fc30 2023-10-25T00:21:45.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b25618640 1 --2- 172.21.15.107:0/1844040748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b2014fc30 0x7f5b20153190 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f5b200c68b0 tx=0x7f5b1c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b74c0890e481d31 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:45.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b17fff640 1 -- 172.21.15.107:0/1844040748 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0ab600 con 0x7f5b2014fc30 2023-10-25T00:21:45.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b17fff640 1 -- 172.21.15.107:0/1844040748 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c0a7050 con 0x7f5b2014fc30 2023-10-25T00:21:45.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b17fff640 1 -- 172.21.15.107:0/1844040748 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0ab980 con 0x7f5b2014fc30 2023-10-25T00:21:45.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b2787c640 1 -- 172.21.15.107:0/1844040748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b2014fc30 msgr2=0x7f5b20153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:45.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b2787c640 1 --2- 172.21.15.107:0/1844040748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b2014fc30 0x7f5b20153190 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f5b200c68b0 tx=0x7f5b1c01c380 comp rx=0 tx=0).stop 2023-10-25T00:21:45.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b2787c640 1 -- 172.21.15.107:0/1844040748 shutdown_connections 2023-10-25T00:21:45.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b2787c640 1 --2- 172.21.15.107:0/1844040748 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b2006e470 0x7f5b200c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:45.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b2787c640 1 --2- 172.21.15.107:0/1844040748 >> v2:172.21.15.110:3300/0 conn(0x7f5b200c5c20 0x7f5b2006df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:45.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b2787c640 1 --2- 172.21.15.107:0/1844040748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b2014fc30 0x7f5b20153190 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:45.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b2787c640 1 -- 172.21.15.107:0/1844040748 >> 172.21.15.107:0/1844040748 conn(0x7f5b200c39a0 msgr2=0x7f5b200c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:45.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b2787c640 1 -- 172.21.15.107:0/1844040748 shutdown_connections 2023-10-25T00:21:45.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.817+0000 7f5b2787c640 1 -- 172.21.15.107:0/1844040748 wait complete. 2023-10-25T00:21:45.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b2787c640 1 Processor -- start 2023-10-25T00:21:45.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b2787c640 1 -- start start 2023-10-25T00:21:45.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b2787c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b2006e470 0x7f5b201518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:45.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b2787c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b200c5c20 0x7f5b201eda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:45.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b2787c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b2014fc30 0x7f5b201edfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:45.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b2787c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b200c15e0 con 0x7f5b2014fc30 2023-10-25T00:21:45.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b2787c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b201ea500 con 0x7f5b2006e470 2023-10-25T00:21:45.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b2787c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b201ead60 con 0x7f5b200c5c20 2023-10-25T00:21:45.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b25e19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b2014fc30 0x7f5b201edfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:45.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b25e19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b2014fc30 0x7f5b201edfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45608/0 (socket says 172.21.15.107:45608) 2023-10-25T00:21:45.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b25e19640 1 -- 172.21.15.107:0/475993038 learned_addr learned my addr 172.21.15.107:0/475993038 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:21:45.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b24e17640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b200c5c20 0x7f5b201eda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:45.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b25e19640 1 -- 172.21.15.107:0/475993038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b200c5c20 msgr2=0x7f5b201eda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:45.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b25e19640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b200c5c20 0x7f5b201eda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:45.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b25e19640 1 -- 172.21.15.107:0/475993038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b2006e470 msgr2=0x7f5b201518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:45.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b25e19640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b2006e470 0x7f5b201518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:45.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b25e19640 1 -- 172.21.15.107:0/475993038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c0094d0 con 0x7f5b2014fc30 2023-10-25T00:21:45.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b25e19640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b2014fc30 0x7f5b201edfb0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800efc0 tx=0x7f5b1800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:45.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b15ffb640 1 -- 172.21.15.107:0/475993038 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18014480 con 0x7f5b2014fc30 2023-10-25T00:21:45.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b15ffb640 1 -- 172.21.15.107:0/475993038 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1800f040 con 0x7f5b2014fc30 2023-10-25T00:21:45.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b2787c640 1 -- 172.21.15.107:0/475993038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2013b090 con 0x7f5b2014fc30 2023-10-25T00:21:45.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b15ffb640 1 -- 172.21.15.107:0/475993038 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18014840 con 0x7f5b2014fc30 2023-10-25T00:21:45.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.821+0000 7f5b2787c640 1 -- 172.21.15.107:0/475993038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b201ea710 con 0x7f5b2014fc30 2023-10-25T00:21:45.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.825+0000 7f5b0b7fe640 1 -- 172.21.15.107:0/475993038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af8005260 con 0x7f5b2014fc30 2023-10-25T00:21:45.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.825+0000 7f5b15ffb640 1 -- 172.21.15.107:0/475993038 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5b180149a0 con 0x7f5b2014fc30 2023-10-25T00:21:45.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.825+0000 7f5b15ffb640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b04065e90 0x7f5b04068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:21:45.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.825+0000 7f5b15ffb640 1 -- 172.21.15.107:0/475993038 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f5b180858b0 con 0x7f5b2014fc30 2023-10-25T00:21:45.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.825+0000 7f5b25618640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b04065e90 0x7f5b04068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:21:45.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.825+0000 7f5b25618640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b04065e90 0x7f5b04068350 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f5b200c5060 tx=0x7f5b1c004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:21:45.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:45.833+0000 7f5b15ffb640 1 -- 172.21.15.107:0/475993038 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5b18051cf0 con 0x7f5b2014fc30 2023-10-25T00:21:46.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.101+0000 7f5b0b7fe640 1 -- 172.21.15.107:0/475993038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5af8005a90 con 0x7f5b2014fc30 2023-10-25T00:21:46.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.101+0000 7f5b15ffb640 1 -- 172.21.15.107:0/475993038 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5768 (secure 0 0 0) 0x7f5b18018070 con 0x7f5b2014fc30 2023-10-25T00:21:46.106 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:21:46.106 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:21:42.241858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9009":{"gid":9009,"name":"c","rank":0,"incarnation":226,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:21:46.106 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 230 2023-10-25T00:21:46.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.105+0000 7f5b0b7fe640 1 -- 172.21.15.107:0/475993038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b04065e90 msgr2=0x7f5b04068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:46.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.105+0000 7f5b0b7fe640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b04065e90 0x7f5b04068350 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f5b200c5060 tx=0x7f5b1c004740 comp rx=0 tx=0).stop 2023-10-25T00:21:46.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.105+0000 7f5b0b7fe640 1 -- 172.21.15.107:0/475993038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b2014fc30 msgr2=0x7f5b201edfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:21:46.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.105+0000 7f5b0b7fe640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b2014fc30 0x7f5b201edfb0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f5b1800efc0 tx=0x7f5b1800c490 comp rx=0 tx=0).stop 2023-10-25T00:21:46.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.105+0000 7f5b0b7fe640 1 -- 172.21.15.107:0/475993038 shutdown_connections 2023-10-25T00:21:46.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.105+0000 7f5b0b7fe640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b2014fc30 0x7f5b201edfb0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:46.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.105+0000 7f5b0b7fe640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b04065e90 0x7f5b04068350 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:46.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.105+0000 7f5b0b7fe640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b200c5c20 0x7f5b201eda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:46.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.105+0000 7f5b0b7fe640 1 --2- 172.21.15.107:0/475993038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b2006e470 0x7f5b201518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:21:46.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.105+0000 7f5b0b7fe640 1 -- 172.21.15.107:0/475993038 >> 172.21.15.107:0/475993038 conn(0x7f5b200c39a0 msgr2=0x7f5b2006b210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:21:46.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.109+0000 7f5b0b7fe640 1 -- 172.21.15.107:0/475993038 shutdown_connections 2023-10-25T00:21:46.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:21:46.109+0000 7f5b0b7fe640 1 -- 172.21.15.107:0/475993038 wait complete. 2023-10-25T00:21:46.133 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2023-10-25T00:21:46.134 INFO:tasks.mds_thrash.fs.[a]:waiting for 49 secs before thrashing 2023-10-25T00:21:59.875 INFO:tasks.workunit.client.1.smithi110.stdout:Syncing()...5 sec 2023-10-25T00:21:59.876 INFO:tasks.workunit.client.1.smithi110.stdout:FFSB benchmark finished at: Wed Oct 25 00:21:59 2023 2023-10-25T00:21:59.876 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:21:59.877 INFO:tasks.workunit.client.1.smithi110.stdout:Results: 2023-10-25T00:21:59.877 INFO:tasks.workunit.client.1.smithi110.stdout:Benchmark took 308.85 sec 2023-10-25T00:21:59.877 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:21:59.877 INFO:tasks.workunit.client.1.smithi110.stdout:Total Results 2023-10-25T00:21:59.878 INFO:tasks.workunit.client.1.smithi110.stdout:=============== 2023-10-25T00:21:59.878 INFO:tasks.workunit.client.1.smithi110.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2023-10-25T00:21:59.878 INFO:tasks.workunit.client.1.smithi110.stdout: ======= ============ ========= ======= =========== ========== 2023-10-25T00:21:59.878 INFO:tasks.workunit.client.1.smithi110.stdout: write : 1493760 4836.55 100.000% 100.000% 18.9MB/sec 2023-10-25T00:21:59.879 INFO:tasks.workunit.client.1.smithi110.stdout:- 2023-10-25T00:21:59.879 INFO:tasks.workunit.client.1.smithi110.stdout:4836.55 Transactions per Second 2023-10-25T00:21:59.879 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:21:59.879 INFO:tasks.workunit.client.1.smithi110.stdout:Throughput Results 2023-10-25T00:21:59.879 INFO:tasks.workunit.client.1.smithi110.stdout:=================== 2023-10-25T00:21:59.880 INFO:tasks.workunit.client.1.smithi110.stdout:Write Throughput: 18.9MB/sec 2023-10-25T00:21:59.884 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:21:59.884 INFO:tasks.workunit.client.1.smithi110.stdout:System Call Latency statistics in millisecs 2023-10-25T00:21:59.885 INFO:tasks.workunit.client.1.smithi110.stdout:===== 2023-10-25T00:21:59.885 INFO:tasks.workunit.client.1.smithi110.stdout: Min Avg Max Total Calls 2023-10-25T00:21:59.885 INFO:tasks.workunit.client.1.smithi110.stdout: ======== ======== ======== ============ 2023-10-25T00:21:59.885 INFO:tasks.workunit.client.1.smithi110.stdout:[ open] 0.364000 1727.767909 19475.904297 1167 2023-10-25T00:21:59.886 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2023-10-25T00:21:59.886 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-10-25T00:21:59.886 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2023-10-25T00:21:59.887 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[3] 500.000000 - 1000.000000 : 1 2023-10-25T00:21:59.887 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[4] 200.000000 - 500.000000 : 13 2023-10-25T00:21:59.887 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[5] 100.000000 - 200.000000 : 17 2023-10-25T00:21:59.887 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[6] 50.000000 - 100.000000 : 28 2023-10-25T00:21:59.887 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[7] 20.000000 - 50.000000 : 333 2023-10-25T00:21:59.887 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[8] 10.000000 - 20.000000 : 399 2023-10-25T00:21:59.888 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[9] 5.000000 - 10.000000 : 130 2023-10-25T00:21:59.888 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[10] 2.000000 - 5.000000 : 64 2023-10-25T00:21:59.888 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[11] 1.000000 - 2.000000 : 40 2023-10-25T00:21:59.888 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[12] 0.500000 - 1.000000 : 11 2023-10-25T00:21:59.888 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[13] 0.200000 - 0.500000 : 4 2023-10-25T00:21:59.889 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2023-10-25T00:21:59.889 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2023-10-25T00:21:59.889 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2023-10-25T00:21:59.889 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2023-10-25T00:21:59.889 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2023-10-25T00:21:59.889 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:21:59.890 INFO:tasks.workunit.client.1.smithi110.stdout:[ write] 0.084000 5.122040 17537.828125 1493760 2023-10-25T00:21:59.890 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2023-10-25T00:21:59.890 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-10-25T00:21:59.890 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2023-10-25T00:21:59.891 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[3] 500.000000 - 1000.000000 : 5 2023-10-25T00:21:59.891 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[4] 200.000000 - 500.000000 : 24 2023-10-25T00:21:59.891 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[5] 100.000000 - 200.000000 : 183 2023-10-25T00:21:59.891 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[6] 50.000000 - 100.000000 : 6358 2023-10-25T00:21:59.891 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[7] 20.000000 - 50.000000 : 66553 2023-10-25T00:21:59.891 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[8] 10.000000 - 20.000000 : 138628 2023-10-25T00:21:59.892 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[9] 5.000000 - 10.000000 : 256724 2023-10-25T00:21:59.892 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[10] 2.000000 - 5.000000 : 338439 2023-10-25T00:21:59.892 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[11] 1.000000 - 2.000000 : 166615 2023-10-25T00:21:59.892 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[12] 0.500000 - 1.000000 : 77411 2023-10-25T00:21:59.892 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[13] 0.200000 - 0.500000 : 202421 2023-10-25T00:21:59.893 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[14] 0.100000 - 0.200000 : 234631 2023-10-25T00:21:59.893 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[15] 0.050000 - 0.100000 : 5767 2023-10-25T00:21:59.893 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2023-10-25T00:21:59.893 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2023-10-25T00:21:59.893 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2023-10-25T00:21:59.893 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:21:59.894 INFO:tasks.workunit.client.1.smithi110.stdout:[ lseek] 0.000000 0.000956 3.734000 1493760 2023-10-25T00:21:59.894 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2023-10-25T00:21:59.894 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-10-25T00:21:59.894 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2023-10-25T00:21:59.894 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2023-10-25T00:21:59.895 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2023-10-25T00:21:59.895 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2023-10-25T00:21:59.895 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2023-10-25T00:21:59.896 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2023-10-25T00:21:59.896 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2023-10-25T00:21:59.896 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2023-10-25T00:21:59.896 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[10] 2.000000 - 5.000000 : 3 2023-10-25T00:21:59.896 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[11] 1.000000 - 2.000000 : 3 2023-10-25T00:21:59.897 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[12] 0.500000 - 1.000000 : 3 2023-10-25T00:21:59.897 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[13] 0.200000 - 0.500000 : 9 2023-10-25T00:21:59.897 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[14] 0.100000 - 0.200000 : 17 2023-10-25T00:21:59.897 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[15] 0.050000 - 0.100000 : 45 2023-10-25T00:21:59.897 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[16] 0.020000 - 0.050000 : 248 2023-10-25T00:21:59.898 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[17] 0.010000 - 0.020000 : 821 2023-10-25T00:21:59.898 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[18] 0.000000 - 0.010000 : 1492611 2023-10-25T00:21:59.898 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:21:59.898 INFO:tasks.workunit.client.1.smithi110.stdout:[ close] 0.020000 2.563697 67.160004 1167 2023-10-25T00:21:59.898 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2023-10-25T00:21:59.899 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2023-10-25T00:21:59.899 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2023-10-25T00:21:59.899 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2023-10-25T00:21:59.899 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2023-10-25T00:21:59.899 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2023-10-25T00:21:59.900 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[6] 50.000000 - 100.000000 : 2 2023-10-25T00:21:59.900 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[7] 20.000000 - 50.000000 : 31 2023-10-25T00:21:59.900 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[8] 10.000000 - 20.000000 : 42 2023-10-25T00:21:59.900 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[9] 5.000000 - 10.000000 : 83 2023-10-25T00:21:59.900 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[10] 2.000000 - 5.000000 : 120 2023-10-25T00:21:59.901 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[11] 1.000000 - 2.000000 : 134 2023-10-25T00:21:59.901 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[12] 0.500000 - 1.000000 : 77 2023-10-25T00:21:59.901 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[13] 0.200000 - 0.500000 : 64 2023-10-25T00:21:59.901 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[14] 0.100000 - 0.200000 : 112 2023-10-25T00:21:59.901 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[15] 0.050000 - 0.100000 : 113 2023-10-25T00:21:59.902 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[16] 0.020000 - 0.050000 : 389 2023-10-25T00:21:59.902 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2023-10-25T00:21:59.902 INFO:tasks.workunit.client.1.smithi110.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2023-10-25T00:21:59.902 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:21:59.902 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:21:59.903 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:21:59.903 INFO:tasks.workunit.client.1.smithi110.stdout: 2023-10-25T00:21:59.903 INFO:tasks.workunit.client.1.smithi110.stdout:0.4% User Time 2023-10-25T00:21:59.903 INFO:tasks.workunit.client.1.smithi110.stdout:56.2% System Time 2023-10-25T00:21:59.903 INFO:tasks.workunit.client.1.smithi110.stdout:56.7% CPU Utilization 2023-10-25T00:22:05.917 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-25T00:22:05.917 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2023-10-25T00:22:06.005 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.1... 2023-10-25T00:22:06.005 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2023-10-25T00:22:06.394 DEBUG:teuthology.parallel:result is None 2023-10-25T00:22:06.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-10-25T00:22:06.412 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-10-25T00:22:06.413 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2023-10-25T00:22:06.458 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2023-10-25T00:22:06.459 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-10-25T00:22:06.473 INFO:tasks.mds_thrash:joining mds_thrasher 2023-10-25T00:22:06.474 INFO:tasks.mds_thrash.fs.[b]:stat['max_mds'] = 0 2023-10-25T00:22:06.474 INFO:tasks.mds_thrash.fs.[b]:stat['deactivate'] = 0 2023-10-25T00:22:06.474 INFO:tasks.mds_thrash.fs.[b]:stat['kill'] = 22 2023-10-25T00:22:06.475 INFO:tasks.mds_thrash:done joining 2023-10-25T00:22:06.475 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-10-25T00:22:06.521 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-10-25T00:22:06.522 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-25T00:22:06.522 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-25T00:22:06.547 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-25T00:22:06.547 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-25T00:22:06.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-25T00:22:06.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98cec1640 1 Processor -- start 2023-10-25T00:22:06.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98cec1640 1 -- start start 2023-10-25T00:22:06.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98cec1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb9880c8100 0x7fb9880abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:06.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98cec1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb988155680 con 0x7fb988152040 2023-10-25T00:22:06.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98cec1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb988154a70 con 0x7fb98806a4b0 2023-10-25T00:22:06.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98cec1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb988154d50 con 0x7fb9880c8100 2023-10-25T00:22:06.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98659c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb988152040 0x7fb9880c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47426/0 (socket says 172.21.15.107:47426) 2023-10-25T00:22:06.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98659c640 1 -- 172.21.15.107:0/281104122 learned_addr learned my addr 172.21.15.107:0/281104122 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:06.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb985d9b640 1 --2- 172.21.15.107:0/281104122 >> v2:172.21.15.110:3300/0 conn(0x7fb9880c8100 0x7fb9880abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:06.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106405605 0 0) 0x7fb988155680 con 0x7fb988152040 2023-10-25T00:22:06.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb970003530 con 0x7fb988152040 2023-10-25T00:22:06.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3709153563 0 0) 0x7fb970003530 con 0x7fb988152040 2023-10-25T00:22:06.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 >> v2:172.21.15.110:3300/0 conn(0x7fb9880c8100 msgr2=0x7fb9880abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:06.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98559a640 1 --2- 172.21.15.107:0/281104122 >> v2:172.21.15.110:3300/0 conn(0x7fb9880c8100 0x7fb9880abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:06.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 >> v1:172.21.15.110:6789/0 conn(0x7fb98806a4b0 legacy=0x7fb9880ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:06.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.800+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb988155030 con 0x7fb988152040 2023-10-25T00:22:06.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb980001680 con 0x7fb988152040 2023-10-25T00:22:06.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 >> v1:172.21.15.107:6789/0 conn(0x7fb988152040 legacy=0x7fb9880c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:06.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98559a640 1 --2- 172.21.15.107:0/281104122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb970003980 0x7fb970003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98559a640 1 --2- 172.21.15.107:0/281104122 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9700047d0 0x7fb970004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98559a640 1 --2- 172.21.15.107:0/281104122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb970005190 0x7fb97000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb988154d50 con 0x7fb970003980 2023-10-25T00:22:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97000c130 con 0x7fb970005190 2023-10-25T00:22:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97000c2f0 con 0x7fb9700047d0 2023-10-25T00:22:06.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98659c640 1 --2- 172.21.15.107:0/281104122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb970003980 0x7fb970003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:06.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb985d9b640 1 --2- 172.21.15.107:0/281104122 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9700047d0 0x7fb970004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:06.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb986d9d640 1 --2- 172.21.15.107:0/281104122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb970005190 0x7fb97000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:06.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98659c640 1 -- 172.21.15.107:0/281104122 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9700047d0 msgr2=0x7fb970004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:06.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98659c640 1 --2- 172.21.15.107:0/281104122 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb9700047d0 0x7fb970004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:06.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98659c640 1 -- 172.21.15.107:0/281104122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb970005190 msgr2=0x7fb97000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:06.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98659c640 1 --2- 172.21.15.107:0/281104122 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb970005190 0x7fb97000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:06.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98659c640 1 -- 172.21.15.107:0/281104122 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb970003530 con 0x7fb970003980 2023-10-25T00:22:06.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb986d9d640 1 -- 172.21.15.107:0/281104122 reap_dead start 2023-10-25T00:22:06.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98659c640 1 --2- 172.21.15.107:0/281104122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb970003980 0x7fb970003da0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fb98000c240 tx=0x7fb98001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2a31a84f9f0895 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:06.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9800058e0 con 0x7fb970003980 2023-10-25T00:22:06.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb980005a40 con 0x7fb970003980 2023-10-25T00:22:06.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98559a640 1 -- 172.21.15.107:0/281104122 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb980005e70 con 0x7fb970003980 2023-10-25T00:22:06.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98cec1640 1 -- 172.21.15.107:0/281104122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb970003980 msgr2=0x7fb970003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:06.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98cec1640 1 --2- 172.21.15.107:0/281104122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb970003980 0x7fb970003da0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fb98000c240 tx=0x7fb98001cc40 comp rx=0 tx=0).stop 2023-10-25T00:22:06.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98cec1640 1 -- 172.21.15.107:0/281104122 shutdown_connections 2023-10-25T00:22:06.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98cec1640 1 --2- 172.21.15.107:0/281104122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb970003980 0x7fb970003da0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:06.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98cec1640 1 -- 172.21.15.107:0/281104122 >> 172.21.15.107:0/281104122 conn(0x7fb9880ca200 msgr2=0x7fb9880ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:06.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98cec1640 1 -- 172.21.15.107:0/281104122 shutdown_connections 2023-10-25T00:22:06.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.804+0000 7fb98cec1640 1 -- 172.21.15.107:0/281104122 wait complete. 2023-10-25T00:22:06.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98cec1640 1 Processor -- start 2023-10-25T00:22:06.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98cec1640 1 -- start start 2023-10-25T00:22:06.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98cec1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb970003980 0x7fb9881ed150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:06.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98cec1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9881e2870 0x7fb9881efad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:06.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98cec1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9881f0010 0x7fb9880c35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:06.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98cec1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9880af1a0 con 0x7fb9881f0010 2023-10-25T00:22:06.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98cec1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9880ad100 con 0x7fb9881e2870 2023-10-25T00:22:06.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98cec1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98814f800 con 0x7fb970003980 2023-10-25T00:22:06.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb986d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9881f0010 0x7fb9880c35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:06.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb986d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9881f0010 0x7fb9880c35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60220/0 (socket says 172.21.15.107:60220) 2023-10-25T00:22:06.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb986d9d640 1 -- 172.21.15.107:0/205487169 learned_addr learned my addr 172.21.15.107:0/205487169 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:06.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb985d9b640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9881e2870 0x7fb9881efad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:06.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98659c640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb970003980 0x7fb9881ed150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:06.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb986d9d640 1 -- 172.21.15.107:0/205487169 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb970003980 msgr2=0x7fb9881ed150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:06.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb986d9d640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb970003980 0x7fb9881ed150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:06.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb986d9d640 1 -- 172.21.15.107:0/205487169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9881e2870 msgr2=0x7fb9881efad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:06.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb986d9d640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9881e2870 0x7fb9881efad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:06.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb986d9d640 1 -- 172.21.15.107:0/205487169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9881ec670 con 0x7fb9881f0010 2023-10-25T00:22:06.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb985d9b640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9881e2870 0x7fb9881efad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:06.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb986d9d640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9881f0010 0x7fb9880c35b0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fb9880c6b30 tx=0x7fb97800e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:06.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb976ffd640 1 -- 172.21.15.107:0/205487169 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb978006e30 con 0x7fb9881f0010 2023-10-25T00:22:06.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb976ffd640 1 -- 172.21.15.107:0/205487169 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb978003980 con 0x7fb9881f0010 2023-10-25T00:22:06.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb976ffd640 1 -- 172.21.15.107:0/205487169 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb978003d30 con 0x7fb9881f0010 2023-10-25T00:22:06.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98cec1640 1 -- 172.21.15.107:0/205487169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9881ec950 con 0x7fb9881f0010 2023-10-25T00:22:06.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98cec1640 1 -- 172.21.15.107:0/205487169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb98814faa0 con 0x7fb9881f0010 2023-10-25T00:22:06.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb98cec1640 1 -- 172.21.15.107:0/205487169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9881ece70 con 0x7fb9881f0010 2023-10-25T00:22:06.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.808+0000 7fb976ffd640 1 -- 172.21.15.107:0/205487169 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb978005110 con 0x7fb9881f0010 2023-10-25T00:22:06.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.812+0000 7fb976ffd640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e60 0x7fb968068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:06.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.812+0000 7fb976ffd640 1 -- 172.21.15.107:0/205487169 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fb978083f40 con 0x7fb9881f0010 2023-10-25T00:22:06.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.812+0000 7fb98659c640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e60 0x7fb968068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:06.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.812+0000 7fb98659c640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e60 0x7fb968068320 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fb9800ab000 tx=0x7fb9800a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:06.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.816+0000 7fb976ffd640 1 -- 172.21.15.107:0/205487169 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb9881ece70 con 0x7fb9881f0010 2023-10-25T00:22:06.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.972+0000 7fb98cec1640 1 -- 172.21.15.107:0/205487169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9881ece70 con 0x7fb9881f0010 2023-10-25T00:22:06.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.976+0000 7fb976ffd640 1 -- 172.21.15.107:0/205487169 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+27395 (secure 0 0 0) 0x7fb9881ece70 con 0x7fb9881f0010 2023-10-25T00:22:06.977 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:06.977 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":72,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-25T00:21:39.173159+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2f","mappings":[{"from":3,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":2}]},{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6835/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6837/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.110:6834/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.110:6832/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.107:6835/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.107:6837/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.110:6834/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.110:6832/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.107:6835/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.107:6834/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.110:6833/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6836/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6836/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.107:6834/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6835/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6836/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.110:6834/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6835/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.110:6835/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6834/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.107:6837/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.107:6837/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6832/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6833/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.110:6833/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6834/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.110:6835/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6836/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6836/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6834/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.110:6835/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.107:6834/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.107:6834/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.110:6833/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.107:6837/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6834/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.110:6833/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.110:6832/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.107:6836/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6837/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.107:6837/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.110:6833/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6837/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.107:6835/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.107:6837/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6837/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6835/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.110:6835/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.107:6836/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.107:6834/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.110:6832/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.107:6835/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6835/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.110:6832/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.110:6833/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6836/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6834/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6836/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.110:6834/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6837/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6836/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.110:6835/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6834/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6835/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6835/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.110:6832/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6835/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.107:6836/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6833/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6835/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.107:6836/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6834/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.107:6836/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6836/1788268787":"2023-10-26T00:10:38.784519+0000","172.21.15.110:6835/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.107:6837/1788268787":"2023-10-26T00:10:38.784519+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-25T00:22:06.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.976+0000 7fb98cec1640 1 -- 172.21.15.107:0/205487169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e60 msgr2=0x7fb968068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:06.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.976+0000 7fb98cec1640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e60 0x7fb968068320 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fb9800ab000 tx=0x7fb9800a7070 comp rx=0 tx=0).stop 2023-10-25T00:22:06.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.976+0000 7fb98cec1640 1 -- 172.21.15.107:0/205487169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9881f0010 msgr2=0x7fb9880c35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:06.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.976+0000 7fb98cec1640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9881f0010 0x7fb9880c35b0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fb9880c6b30 tx=0x7fb97800e960 comp rx=0 tx=0).stop 2023-10-25T00:22:06.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.980+0000 7fb98cec1640 1 -- 172.21.15.107:0/205487169 shutdown_connections 2023-10-25T00:22:06.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.980+0000 7fb98cec1640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9881f0010 0x7fb9880c35b0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:06.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.980+0000 7fb98cec1640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb9881e2870 0x7fb9881efad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:06.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.980+0000 7fb98cec1640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e60 0x7fb968068320 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:06.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.980+0000 7fb98cec1640 1 --2- 172.21.15.107:0/205487169 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb970003980 0x7fb9881ed150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:06.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.980+0000 7fb98cec1640 1 -- 172.21.15.107:0/205487169 >> 172.21.15.107:0/205487169 conn(0x7fb9880ca200 msgr2=0x7fb9880ac090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:06.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.980+0000 7fb98cec1640 1 -- 172.21.15.107:0/205487169 shutdown_connections 2023-10-25T00:22:06.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:06.980+0000 7fb98cec1640 1 -- 172.21.15.107:0/205487169 wait complete. 2023-10-25T00:22:06.996 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-10-24T23:53:09.786122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-10-24T23:53:10.766645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-25T00:22:06.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-25T00:22:07.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.140+0000 7fd15b8e1640 1 Processor -- start 2023-10-25T00:22:07.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.140+0000 7fd15b8e1640 1 -- start start 2023-10-25T00:22:07.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15b8e1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd1540c4f50 0x7fd15406b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15b8e1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540c1b40 0x7fd1540a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15b8e1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15406a750 con 0x7fd1540b9470 2023-10-25T00:22:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15b8e1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15406bca0 con 0x7fd1540c4f50 2023-10-25T00:22:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15b8e1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15406a9a0 con 0x7fd1540c1b40 2023-10-25T00:22:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd159e7e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd1540b9470 0x7fd1540c0760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47436/0 (socket says 172.21.15.107:47436) 2023-10-25T00:22:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd159e7e640 1 -- 172.21.15.107:0/566056739 learned_addr learned my addr 172.21.15.107:0/566056739 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15967d640 1 --2- 172.21.15.107:0/566056739 >> v2:172.21.15.110:3300/0 conn(0x7fd1540c4f50 0x7fd15406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd158e7c640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540c1b40 0x7fd1540a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211108372 0 0) 0x7fd15406a750 con 0x7fd1540b9470 2023-10-25T00:22:07.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd140003530 con 0x7fd1540b9470 2023-10-25T00:22:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1197995130 0 0) 0x7fd140003530 con 0x7fd1540b9470 2023-10-25T00:22:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540c1b40 msgr2=0x7fd1540a91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540c1b40 0x7fd1540a91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 >> v2:172.21.15.110:3300/0 conn(0x7fd1540c4f50 msgr2=0x7fd15406b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:07.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 --2- 172.21.15.107:0/566056739 >> v2:172.21.15.110:3300/0 conn(0x7fd1540c4f50 0x7fd15406b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15406ac80 con 0x7fd1540b9470 2023-10-25T00:22:07.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fd150004e70 con 0x7fd1540b9470 2023-10-25T00:22:07.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 >> v1:172.21.15.107:6789/0 conn(0x7fd1540b9470 legacy=0x7fd1540c0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:07.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd140003980 0x7fd140003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540c1b40 0x7fd1400047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd140004d10 0x7fd140005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15967d640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd140003980 0x7fd140003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15406bca0 con 0x7fd140003980 2023-10-25T00:22:07.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15406a9a0 con 0x7fd140004d10 2023-10-25T00:22:07.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14000dc20 con 0x7fd1540c1b40 2023-10-25T00:22:07.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd158e7c640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540c1b40 0x7fd1400047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd159e7e640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd140004d10 0x7fd140005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15967d640 1 -- 172.21.15.107:0/566056739 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540c1b40 msgr2=0x7fd1400047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15967d640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540c1b40 0x7fd1400047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15967d640 1 -- 172.21.15.107:0/566056739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd140004d10 msgr2=0x7fd140005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15967d640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd140004d10 0x7fd140005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.144+0000 7fd15967d640 1 -- 172.21.15.107:0/566056739 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd140003530 con 0x7fd140003980 2023-10-25T00:22:07.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.148+0000 7fd15967d640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd140003980 0x7fd140003da0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00a170 tx=0x7fd14c01cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e63514efb6d31f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:07.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.148+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0a5a50 con 0x7fd140003980 2023-10-25T00:22:07.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.148+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c0a5bb0 con 0x7fd140003980 2023-10-25T00:22:07.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.148+0000 7fd14bfff640 1 -- 172.21.15.107:0/566056739 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0b0460 con 0x7fd140003980 2023-10-25T00:22:07.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.148+0000 7fd15b8e1640 1 -- 172.21.15.107:0/566056739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd140003980 msgr2=0x7fd140003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:07.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.148+0000 7fd15b8e1640 1 --2- 172.21.15.107:0/566056739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd140003980 0x7fd140003da0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00a170 tx=0x7fd14c01cc70 comp rx=0 tx=0).stop 2023-10-25T00:22:07.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.148+0000 7fd159e7e640 1 -- 172.21.15.107:0/566056739 reap_dead start 2023-10-25T00:22:07.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.148+0000 7fd15b8e1640 1 -- 172.21.15.107:0/566056739 shutdown_connections 2023-10-25T00:22:07.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.148+0000 7fd15b8e1640 1 -- 172.21.15.107:0/566056739 >> 172.21.15.107:0/566056739 conn(0x7fd1540c2860 msgr2=0x7fd1540c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:07.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.148+0000 7fd15b8e1640 1 -- 172.21.15.107:0/566056739 shutdown_connections 2023-10-25T00:22:07.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.148+0000 7fd15b8e1640 1 -- 172.21.15.107:0/566056739 wait complete. 2023-10-25T00:22:07.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd15b8e1640 1 Processor -- start 2023-10-25T00:22:07.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd15b8e1640 1 -- start start 2023-10-25T00:22:07.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd15b8e1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd140003980 0x7fd1540aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd15b8e1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540701c0 0x7fd1541ec8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd15b8e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd154071a20 0x7fd1541ecde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd15b8e1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15406fc50 con 0x7fd154071a20 2023-10-25T00:22:07.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd15b8e1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1541e9760 con 0x7fd140003980 2023-10-25T00:22:07.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd15b8e1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1541e9c90 con 0x7fd1540701c0 2023-10-25T00:22:07.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd159e7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd154071a20 0x7fd1541ecde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd159e7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd154071a20 0x7fd1541ecde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60234/0 (socket says 172.21.15.107:60234) 2023-10-25T00:22:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd159e7e640 1 -- 172.21.15.107:0/2451468962 learned_addr learned my addr 172.21.15.107:0/2451468962 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd15967d640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd140003980 0x7fd1540aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd158e7c640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540701c0 0x7fd1541ec8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd159e7e640 1 -- 172.21.15.107:0/2451468962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540701c0 msgr2=0x7fd1541ec8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd159e7e640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540701c0 0x7fd1541ec8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd159e7e640 1 -- 172.21.15.107:0/2451468962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd140003980 msgr2=0x7fd1540aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd159e7e640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd140003980 0x7fd1540aa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd159e7e640 1 -- 172.21.15.107:0/2451468962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1541ec410 con 0x7fd154071a20 2023-10-25T00:22:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd159e7e640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd154071a20 0x7fd1541ecde0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fd15406e8b0 tx=0x7fd15000ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd149ffb640 1 -- 172.21.15.107:0/2451468962 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150001880 con 0x7fd154071a20 2023-10-25T00:22:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd15b8e1640 1 -- 172.21.15.107:0/2451468962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1540bccb0 con 0x7fd154071a20 2023-10-25T00:22:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd149ffb640 1 -- 172.21.15.107:0/2451468962 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1500019e0 con 0x7fd154071a20 2023-10-25T00:22:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd149ffb640 1 -- 172.21.15.107:0/2451468962 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd150001d90 con 0x7fd154071a20 2023-10-25T00:22:07.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.152+0000 7fd15b8e1640 1 -- 172.21.15.107:0/2451468962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1540bd510 con 0x7fd154071a20 2023-10-25T00:22:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.156+0000 7fd149ffb640 1 -- 172.21.15.107:0/2451468962 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd1500103f0 con 0x7fd154071a20 2023-10-25T00:22:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.156+0000 7fd15b8e1640 1 -- 172.21.15.107:0/2451468962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1541ec5a0 con 0x7fd154071a20 2023-10-25T00:22:07.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.156+0000 7fd149ffb640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd138065e60 0x7fd138068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.156+0000 7fd149ffb640 1 -- 172.21.15.107:0/2451468962 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fd150009040 con 0x7fd154071a20 2023-10-25T00:22:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.156+0000 7fd15967d640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd138065e60 0x7fd138068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.156+0000 7fd15967d640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd138065e60 0x7fd138068320 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0af000 tx=0x7fd14c0b3040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:07.163 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.160+0000 7fd149ffb640 1 -- 172.21.15.107:0/2451468962 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd1500518a0 con 0x7fd154071a20 2023-10-25T00:22:07.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.320+0000 7fd15b8e1640 1 -- 172.21.15.107:0/2451468962 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd15404e130 con 0x7fd154071a20 2023-10-25T00:22:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.320+0000 7fd149ffb640 1 -- 172.21.15.107:0/2451468962 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd15001e0a0 con 0x7fd154071a20 2023-10-25T00:22:07.322 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-25T00:22:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 -- 172.21.15.107:0/2451468962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd138065e60 msgr2=0x7fd138068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd138065e60 0x7fd138068320 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0af000 tx=0x7fd14c0b3040 comp rx=0 tx=0).stop 2023-10-25T00:22:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 -- 172.21.15.107:0/2451468962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd154071a20 msgr2=0x7fd1541ecde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd154071a20 0x7fd1541ecde0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fd15406e8b0 tx=0x7fd15000ca40 comp rx=0 tx=0).stop 2023-10-25T00:22:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 -- 172.21.15.107:0/2451468962 shutdown_connections 2023-10-25T00:22:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd154071a20 0x7fd1541ecde0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd138065e60 0x7fd138068320 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1540701c0 0x7fd1541ec8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 --2- 172.21.15.107:0/2451468962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd140003980 0x7fd1540aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 -- 172.21.15.107:0/2451468962 >> 172.21.15.107:0/2451468962 conn(0x7fd1540c2860 msgr2=0x7fd154154e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 -- 172.21.15.107:0/2451468962 shutdown_connections 2023-10-25T00:22:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.324+0000 7fd15b8e1640 1 -- 172.21.15.107:0/2451468962 wait complete. 2023-10-25T00:22:07.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-25T00:22:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb9827f5640 1 Processor -- start 2023-10-25T00:22:07.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb9827f5640 1 -- start start 2023-10-25T00:22:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb9827f5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97c072760 0x7fb97c0c1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb9827f5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb97c0ab8c0 0x7fb97c0abd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb9827f5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97c06dd70 con 0x7fb97c14fe10 2023-10-25T00:22:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb9827f5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97c0c2e50 con 0x7fb97c0ab8c0 2023-10-25T00:22:07.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb9827f5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c06da40 con 0x7fb97c072760 2023-10-25T00:22:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb97bfff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb97c14fe10 0x7fb97c065a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:58772/0 (socket says 172.21.15.107:58772) 2023-10-25T00:22:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb97bfff640 1 -- 172.21.15.107:0/3275275439 learned_addr learned my addr 172.21.15.107:0/3275275439 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb980d92640 1 --2- 172.21.15.107:0/3275275439 >> v2:172.21.15.110:3300/0 conn(0x7fb97c0ab8c0 0x7fb97c0abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522582745 0 0) 0x7fb97c06dd70 con 0x7fb97c14fe10 2023-10-25T00:22:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.500+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb964003530 con 0x7fb97c14fe10 2023-10-25T00:22:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3631824062 0 0) 0x7fb964003530 con 0x7fb97c14fe10 2023-10-25T00:22:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97c072760 msgr2=0x7fb97c0c1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 --2- 172.21.15.107:0/3275275439 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97c072760 0x7fb97c0c1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 >> v2:172.21.15.110:3300/0 conn(0x7fb97c0ab8c0 msgr2=0x7fb97c0abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 --2- 172.21.15.107:0/3275275439 >> v2:172.21.15.110:3300/0 conn(0x7fb97c0ab8c0 0x7fb97c0abd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c153400 con 0x7fb97c14fe10 2023-10-25T00:22:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb96c001680 con 0x7fb97c14fe10 2023-10-25T00:22:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 >> v1:172.21.15.110:6789/0 conn(0x7fb97c14fe10 legacy=0x7fb97c065a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 --2- 172.21.15.107:0/3275275439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb964003980 0x7fb964003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 --2- 172.21.15.107:0/3275275439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 0x7fb964004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 --2- 172.21.15.107:0/3275275439 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97c072760 0x7fb9640051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c06da40 con 0x7fb9640047d0 2023-10-25T00:22:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96400baa0 con 0x7fb964003980 2023-10-25T00:22:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96400bc60 con 0x7fb97c072760 2023-10-25T00:22:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97b7fe640 1 --2- 172.21.15.107:0/3275275439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 0x7fb964004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97b7fe640 1 -- 172.21.15.107:0/3275275439 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97c072760 msgr2=0x7fb9640051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97b7fe640 1 --2- 172.21.15.107:0/3275275439 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97c072760 0x7fb9640051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97b7fe640 1 -- 172.21.15.107:0/3275275439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb964003980 msgr2=0x7fb964003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97b7fe640 1 --2- 172.21.15.107:0/3275275439 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb964003980 0x7fb964003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.504+0000 7fb97b7fe640 1 -- 172.21.15.107:0/3275275439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb964003530 con 0x7fb9640047d0 2023-10-25T00:22:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb97b7fe640 1 --2- 172.21.15.107:0/3275275439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 0x7fb964004c50 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fb970009890 tx=0x7fb97001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0adca1e80accabc server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:07.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9700ab600 con 0x7fb9640047d0 2023-10-25T00:22:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9700a7050 con 0x7fb9640047d0 2023-10-25T00:22:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb97affd640 1 -- 172.21.15.107:0/3275275439 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9700ab9c0 con 0x7fb9640047d0 2023-10-25T00:22:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb9827f5640 1 -- 172.21.15.107:0/3275275439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 msgr2=0x7fb964004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb9827f5640 1 --2- 172.21.15.107:0/3275275439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 0x7fb964004c50 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fb970009890 tx=0x7fb97001c3a0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb980d92640 1 -- 172.21.15.107:0/3275275439 reap_dead start 2023-10-25T00:22:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb9827f5640 1 -- 172.21.15.107:0/3275275439 shutdown_connections 2023-10-25T00:22:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb9827f5640 1 -- 172.21.15.107:0/3275275439 >> 172.21.15.107:0/3275275439 conn(0x7fb97c0c65e0 msgr2=0x7fb97c0c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb9827f5640 1 -- 172.21.15.107:0/3275275439 shutdown_connections 2023-10-25T00:22:07.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb9827f5640 1 -- 172.21.15.107:0/3275275439 wait complete. 2023-10-25T00:22:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb9827f5640 1 Processor -- start 2023-10-25T00:22:07.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.508+0000 7fb9827f5640 1 -- start start 2023-10-25T00:22:07.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb9827f5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 0x7fb97c149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb9827f5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb97c14f1b0 0x7fb97c1e7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb9827f5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97c1e7d60 0x7fb97c1473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb9827f5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c1e81e0 con 0x7fb9640047d0 2023-10-25T00:22:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb9827f5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c1ebc10 con 0x7fb97c14f1b0 2023-10-25T00:22:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb9827f5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c148210 con 0x7fb97c1e7d60 2023-10-25T00:22:07.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb97bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 0x7fb97c149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb97bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 0x7fb97c149df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60258/0 (socket says 172.21.15.107:60258) 2023-10-25T00:22:07.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb97bfff640 1 -- 172.21.15.107:0/436704436 learned_addr learned my addr 172.21.15.107:0/436704436 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:07.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb97bfff640 1 -- 172.21.15.107:0/436704436 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97c1e7d60 msgr2=0x7fb97c1473b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-25T00:22:07.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb97bfff640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97c1e7d60 0x7fb97c1473b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb97bfff640 1 -- 172.21.15.107:0/436704436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb97c14f1b0 msgr2=0x7fb97c1e7820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-25T00:22:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb97bfff640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb97c14f1b0 0x7fb97c1e7820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb97bfff640 1 -- 172.21.15.107:0/436704436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c1484f0 con 0x7fb9640047d0 2023-10-25T00:22:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb97bfff640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 0x7fb97c149df0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00f460 tx=0x7fb96c00f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:07.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb978ff9640 1 -- 172.21.15.107:0/436704436 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c007d00 con 0x7fb9640047d0 2023-10-25T00:22:07.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb978ff9640 1 -- 172.21.15.107:0/436704436 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96c007e80 con 0x7fb9640047d0 2023-10-25T00:22:07.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb9827f5640 1 -- 172.21.15.107:0/436704436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97c1487d0 con 0x7fb9640047d0 2023-10-25T00:22:07.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb978ff9640 1 -- 172.21.15.107:0/436704436 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96c00a2f0 con 0x7fb9640047d0 2023-10-25T00:22:07.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.516+0000 7fb9827f5640 1 -- 172.21.15.107:0/436704436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97c0ac000 con 0x7fb9640047d0 2023-10-25T00:22:07.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.520+0000 7fb978ff9640 1 -- 172.21.15.107:0/436704436 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb96c00a450 con 0x7fb9640047d0 2023-10-25T00:22:07.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.520+0000 7fb978ff9640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e90 0x7fb968068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:07.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.520+0000 7fb97b7fe640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e90 0x7fb968068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:07.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.520+0000 7fb978ff9640 1 -- 172.21.15.107:0/436704436 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fb96c0536d0 con 0x7fb9640047d0 2023-10-25T00:22:07.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.520+0000 7fb9827f5640 1 -- 172.21.15.107:0/436704436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97c148350 con 0x7fb9640047d0 2023-10-25T00:22:07.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.524+0000 7fb97b7fe640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e90 0x7fb968068350 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb9700ad000 tx=0x7fb970004850 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:07.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.532+0000 7fb978ff9640 1 -- 172.21.15.107:0/436704436 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb97c148350 con 0x7fb9640047d0 2023-10-25T00:22:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.864+0000 7fb9827f5640 1 -- 172.21.15.107:0/436704436 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb97c14e440 con 0x7fb9640047d0 2023-10-25T00:22:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.864+0000 7fb978ff9640 1 -- 172.21.15.107:0/436704436 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v72) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb97c14e440 con 0x7fb9640047d0 2023-10-25T00:22:07.865 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-25T00:22:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.868+0000 7fb9827f5640 1 -- 172.21.15.107:0/436704436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e90 msgr2=0x7fb968068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:07.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.868+0000 7fb9827f5640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e90 0x7fb968068350 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb9700ad000 tx=0x7fb970004850 comp rx=0 tx=0).stop 2023-10-25T00:22:07.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.868+0000 7fb9827f5640 1 -- 172.21.15.107:0/436704436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 msgr2=0x7fb97c149df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:07.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.868+0000 7fb9827f5640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 0x7fb97c149df0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fb96c00f460 tx=0x7fb96c00f920 comp rx=0 tx=0).stop 2023-10-25T00:22:07.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.872+0000 7fb9827f5640 1 -- 172.21.15.107:0/436704436 shutdown_connections 2023-10-25T00:22:07.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.872+0000 7fb9827f5640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb968065e90 0x7fb968068350 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.872+0000 7fb9827f5640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb97c1e7d60 0x7fb97c1473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.872+0000 7fb9827f5640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb97c14f1b0 0x7fb97c1e7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.872+0000 7fb9827f5640 1 --2- 172.21.15.107:0/436704436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb9640047d0 0x7fb97c149df0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:07.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.872+0000 7fb9827f5640 1 -- 172.21.15.107:0/436704436 >> 172.21.15.107:0/436704436 conn(0x7fb97c0c65e0 msgr2=0x7fb97c0c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:07.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.872+0000 7fb9827f5640 1 -- 172.21.15.107:0/436704436 shutdown_connections 2023-10-25T00:22:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:07.872+0000 7fb9827f5640 1 -- 172.21.15.107:0/436704436 wait complete. 2023-10-25T00:22:07.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-25T00:22:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f619ae28640 1 Processor -- start 2023-10-25T00:22:08.140 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f619ae28640 1 -- start start 2023-10-25T00:22:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f619ae28640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f61940c2000 0x7f61940ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f619ae28640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61940c4030 0x7f619415f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f619ae28640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6194071de0 con 0x7f61940adda0 2023-10-25T00:22:08.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f619ae28640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61940bd250 con 0x7f61940c4030 2023-10-25T00:22:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f619ae28640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61940bcf00 con 0x7f61940c2000 2023-10-25T00:22:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f61993c5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61940c4030 0x7f619415f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f61993c5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f61940c4030 0x7f619415f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60268/0 (socket says 172.21.15.107:60268) 2023-10-25T00:22:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f61993c5640 1 -- 172.21.15.107:0/3830100794 learned_addr learned my addr 172.21.15.107:0/3830100794 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:08.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f6198bc4640 1 --2- 172.21.15.107:0/3830100794 >> v2:172.21.15.110:3300/0 conn(0x7f61940c2000 0x7f61940ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f61993c5640 1 -- 172.21.15.107:0/3830100794 >> v2:172.21.15.110:3300/0 conn(0x7f61940c2000 msgr2=0x7f61940ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f61993c5640 1 --2- 172.21.15.107:0/3830100794 >> v2:172.21.15.110:3300/0 conn(0x7f61940c2000 0x7f61940ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f61993c5640 1 -- 172.21.15.107:0/3830100794 >> v1:172.21.15.110:6789/0 conn(0x7f61940adda0 legacy=0x7f61940c3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f61993c5640 1 -- 172.21.15.107:0/3830100794 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619406a740 con 0x7f61940c4030 2023-10-25T00:22:08.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.140+0000 7f6198bc4640 1 --2- 172.21.15.107:0/3830100794 >> v2:172.21.15.110:3300/0 conn(0x7f61940c2000 0x7f61940ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:22:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61993c5640 1 --2- 172.21.15.107:0/3830100794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61940c4030 0x7f619415f080 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f6194070920 tx=0x7f618c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e0e9478a64b5ea server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61937fe640 1 -- 172.21.15.107:0/3830100794 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c0a5860 con 0x7f61940c4030 2023-10-25T00:22:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61937fe640 1 -- 172.21.15.107:0/3830100794 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618c0a59c0 con 0x7f61940c4030 2023-10-25T00:22:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61937fe640 1 -- 172.21.15.107:0/3830100794 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c0a5d50 con 0x7f61940c4030 2023-10-25T00:22:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 -- 172.21.15.107:0/3830100794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61940c4030 msgr2=0x7f619415f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 --2- 172.21.15.107:0/3830100794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61940c4030 0x7f619415f080 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f6194070920 tx=0x7f618c01c9a0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 -- 172.21.15.107:0/3830100794 shutdown_connections 2023-10-25T00:22:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 --2- 172.21.15.107:0/3830100794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61940c4030 0x7f619415f080 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 --2- 172.21.15.107:0/3830100794 >> v2:172.21.15.110:3300/0 conn(0x7f61940c2000 0x7f61940ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 -- 172.21.15.107:0/3830100794 >> 172.21.15.107:0/3830100794 conn(0x7f6194070ef0 msgr2=0x7f6194071300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 -- 172.21.15.107:0/3830100794 shutdown_connections 2023-10-25T00:22:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 -- 172.21.15.107:0/3830100794 wait complete. 2023-10-25T00:22:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 Processor -- start 2023-10-25T00:22:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 -- start start 2023-10-25T00:22:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61940adda0 0x7f6194157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61940c2000 0x7f61941565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61940c4030 0x7f6194156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61941584a0 con 0x7f61940c4030 2023-10-25T00:22:08.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61940c6e60 con 0x7f61940c2000 2023-10-25T00:22:08.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f619ae28640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61940c7100 con 0x7f61940adda0 2023-10-25T00:22:08.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61993c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61940c4030 0x7f6194156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61993c5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61940c4030 0x7f6194156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60276/0 (socket says 172.21.15.107:60276) 2023-10-25T00:22:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61993c5640 1 -- 172.21.15.107:0/478903083 learned_addr learned my addr 172.21.15.107:0/478903083 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f6193fff640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61940c2000 0x7f61941565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f6198bc4640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61940adda0 0x7f6194157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61993c5640 1 -- 172.21.15.107:0/478903083 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61940adda0 msgr2=0x7f6194157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61993c5640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61940adda0 0x7f6194157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61993c5640 1 -- 172.21.15.107:0/478903083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61940c2000 msgr2=0x7f61941565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61993c5640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61940c2000 0x7f61941565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61993c5640 1 -- 172.21.15.107:0/478903083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c009b30 con 0x7f61940c4030 2023-10-25T00:22:08.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.144+0000 7f61993c5640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61940c4030 0x7f6194156b20 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f618c0195b0 tx=0x7f618c019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:08.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.148+0000 7f61917fa640 1 -- 172.21.15.107:0/478903083 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c0b8670 con 0x7f61940c4030 2023-10-25T00:22:08.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.148+0000 7f61917fa640 1 -- 172.21.15.107:0/478903083 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618c0b87d0 con 0x7f61940c4030 2023-10-25T00:22:08.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.148+0000 7f61917fa640 1 -- 172.21.15.107:0/478903083 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c0b8b60 con 0x7f61940c4030 2023-10-25T00:22:08.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.148+0000 7f619ae28640 1 -- 172.21.15.107:0/478903083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61941ed870 con 0x7f61940c4030 2023-10-25T00:22:08.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.148+0000 7f619ae28640 1 -- 172.21.15.107:0/478903083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6194157060 con 0x7f61940c4030 2023-10-25T00:22:08.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.148+0000 7f619ae28640 1 -- 172.21.15.107:0/478903083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6194156290 con 0x7f61940c4030 2023-10-25T00:22:08.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.148+0000 7f61917fa640 1 -- 172.21.15.107:0/478903083 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f618c0b8cc0 con 0x7f61940c4030 2023-10-25T00:22:08.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.148+0000 7f61917fa640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6178065e90 0x7f6178068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.148+0000 7f6198bc4640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6178065e90 0x7f6178068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.148+0000 7f61917fa640 1 -- 172.21.15.107:0/478903083 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f618c120030 con 0x7f61940c4030 2023-10-25T00:22:08.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.148+0000 7f6198bc4640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6178065e90 0x7f6178068350 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f61940c1d10 tx=0x7f6180008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:08.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.152+0000 7f61917fa640 1 -- 172.21.15.107:0/478903083 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f618c0ec520 con 0x7f61940c4030 2023-10-25T00:22:08.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 -- 172.21.15.107:0/478903083 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6194073c30 con 0x7f61940c4030 2023-10-25T00:22:08.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f61917fa640 1 -- 172.21.15.107:0/478903083 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v72) v1 ==== 99+0+11 (secure 0 0 0) 0x7f618c0efb40 con 0x7f61940c4030 2023-10-25T00:22:08.324 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-25T00:22:08.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 -- 172.21.15.107:0/478903083 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6178065e90 msgr2=0x7f6178068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6178065e90 0x7f6178068350 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f61940c1d10 tx=0x7f6180008040 comp rx=0 tx=0).stop 2023-10-25T00:22:08.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 -- 172.21.15.107:0/478903083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61940c4030 msgr2=0x7f6194156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61940c4030 0x7f6194156b20 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f618c0195b0 tx=0x7f618c019740 comp rx=0 tx=0).stop 2023-10-25T00:22:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 -- 172.21.15.107:0/478903083 shutdown_connections 2023-10-25T00:22:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61940c4030 0x7f6194156b20 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61940c2000 0x7f61941565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6178065e90 0x7f6178068350 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 --2- 172.21.15.107:0/478903083 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61940adda0 0x7f6194157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 -- 172.21.15.107:0/478903083 >> 172.21.15.107:0/478903083 conn(0x7f6194070ef0 msgr2=0x7f619406f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:08.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 -- 172.21.15.107:0/478903083 shutdown_connections 2023-10-25T00:22:08.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.324+0000 7f619ae28640 1 -- 172.21.15.107:0/478903083 wait complete. 2023-10-25T00:22:08.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-25T00:22:08.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef1606a640 1 Processor -- start 2023-10-25T00:22:08.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef1606a640 1 -- start start 2023-10-25T00:22:08.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef1606a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fef100af9b0 0x7fef100c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef1606a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef1006b360 0x7fef10069d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef1606a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef100c8180 con 0x7fef100bd930 2023-10-25T00:22:08.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef1606a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef100c5870 con 0x7fef100af9b0 2023-10-25T00:22:08.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef1606a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef10155430 con 0x7fef1006b360 2023-10-25T00:22:08.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef0ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fef100bd930 0x7fef10157f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47444/0 (socket says 172.21.15.107:47444) 2023-10-25T00:22:08.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef0ffff640 1 -- 172.21.15.107:0/719556072 learned_addr learned my addr 172.21.15.107:0/719556072 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:08.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef0effd640 1 --2- 172.21.15.107:0/719556072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef1006b360 0x7fef10069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef0f7fe640 1 --2- 172.21.15.107:0/719556072 >> v2:172.21.15.110:3300/0 conn(0x7fef100af9b0 0x7fef100c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef0e7fc640 1 -- 172.21.15.107:0/719556072 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104033538 0 0) 0x7fef100c8180 con 0x7fef100bd930 2023-10-25T00:22:08.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef0e7fc640 1 -- 172.21.15.107:0/719556072 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef8003530 con 0x7fef100bd930 2023-10-25T00:22:08.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef0effd640 1 -- 172.21.15.107:0/719556072 >> v2:172.21.15.110:3300/0 conn(0x7fef100af9b0 msgr2=0x7fef100c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:08.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef0effd640 1 --2- 172.21.15.107:0/719556072 >> v2:172.21.15.110:3300/0 conn(0x7fef100af9b0 0x7fef100c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef0effd640 1 -- 172.21.15.107:0/719556072 >> v1:172.21.15.107:6789/0 conn(0x7fef100bd930 legacy=0x7fef10157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef0effd640 1 -- 172.21.15.107:0/719556072 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef10155710 con 0x7fef1006b360 2023-10-25T00:22:08.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.492+0000 7fef0f7fe640 1 --2- 172.21.15.107:0/719556072 >> v2:172.21.15.110:3300/0 conn(0x7fef100af9b0 0x7fef100c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:22:08.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.496+0000 7fef0effd640 1 --2- 172.21.15.107:0/719556072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef1006b360 0x7fef10069d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fef040097c0 tx=0x7fef0401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feedbdc058d632c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:08.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.496+0000 7fef0e7fc640 1 -- 172.21.15.107:0/719556072 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04004230 con 0x7fef1006b360 2023-10-25T00:22:08.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.496+0000 7fef0e7fc640 1 -- 172.21.15.107:0/719556072 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef040a7050 con 0x7fef1006b360 2023-10-25T00:22:08.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.496+0000 7fef0e7fc640 1 -- 172.21.15.107:0/719556072 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04003620 con 0x7fef1006b360 2023-10-25T00:22:08.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.504+0000 7fef1606a640 1 -- 172.21.15.107:0/719556072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef1006b360 msgr2=0x7fef10069d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.504+0000 7fef1606a640 1 --2- 172.21.15.107:0/719556072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef1006b360 0x7fef10069d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fef040097c0 tx=0x7fef0401c3c0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.504+0000 7fef1606a640 1 -- 172.21.15.107:0/719556072 shutdown_connections 2023-10-25T00:22:08.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.504+0000 7fef1606a640 1 --2- 172.21.15.107:0/719556072 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef1006b360 0x7fef10069d40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.504+0000 7fef1606a640 1 --2- 172.21.15.107:0/719556072 >> v2:172.21.15.110:3300/0 conn(0x7fef100af9b0 0x7fef100c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.504+0000 7fef1606a640 1 -- 172.21.15.107:0/719556072 >> 172.21.15.107:0/719556072 conn(0x7fef10071b00 msgr2=0x7fef100abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:08.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.512+0000 7fef1606a640 1 -- 172.21.15.107:0/719556072 shutdown_connections 2023-10-25T00:22:08.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.516+0000 7fef1606a640 1 -- 172.21.15.107:0/719556072 wait complete. 2023-10-25T00:22:08.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef1606a640 1 Processor -- start 2023-10-25T00:22:08.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef1606a640 1 -- start start 2023-10-25T00:22:08.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef1606a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef1006b360 0x7fef100c0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef1606a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef100af9b0 0x7fef100c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef1606a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef100bd930 0x7fef10150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef1606a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef100c3b60 con 0x7fef100af9b0 2023-10-25T00:22:08.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef1606a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef100c0540 con 0x7fef100bd930 2023-10-25T00:22:08.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef1606a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef100aef30 con 0x7fef1006b360 2023-10-25T00:22:08.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef0ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef100bd930 0x7fef10150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef0ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef100bd930 0x7fef10150c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:46582/0 (socket says 172.21.15.107:46582) 2023-10-25T00:22:08.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef0ffff640 1 -- 172.21.15.107:0/2907986130 learned_addr learned my addr 172.21.15.107:0/2907986130 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:08.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef0ffff640 1 -- 172.21.15.107:0/2907986130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef1006b360 msgr2=0x7fef100c0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef0ffff640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef1006b360 0x7fef100c0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef0ffff640 1 -- 172.21.15.107:0/2907986130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef100af9b0 msgr2=0x7fef100c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef0ffff640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef100af9b0 0x7fef100c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef0ffff640 1 -- 172.21.15.107:0/2907986130 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef00009420 con 0x7fef100bd930 2023-10-25T00:22:08.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.520+0000 7fef0ffff640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef100bd930 0x7fef10150c00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fef0800ca70 tx=0x7fef0800cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:08.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.524+0000 7feef7fff640 1 -- 172.21.15.107:0/2907986130 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0800abd0 con 0x7fef100bd930 2023-10-25T00:22:08.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.524+0000 7fef1606a640 1 -- 172.21.15.107:0/2907986130 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef04009420 con 0x7fef100bd930 2023-10-25T00:22:08.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.524+0000 7fef1606a640 1 -- 172.21.15.107:0/2907986130 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef100c3db0 con 0x7fef100bd930 2023-10-25T00:22:08.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.524+0000 7feef7fff640 1 -- 172.21.15.107:0/2907986130 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0800ad30 con 0x7fef100bd930 2023-10-25T00:22:08.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.524+0000 7feef7fff640 1 -- 172.21.15.107:0/2907986130 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef08012600 con 0x7fef100bd930 2023-10-25T00:22:08.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.532+0000 7feef7fff640 1 -- 172.21.15.107:0/2907986130 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fef08012760 con 0x7fef100bd930 2023-10-25T00:22:08.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.536+0000 7fef1606a640 1 -- 172.21.15.107:0/2907986130 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef0005260 con 0x7fef100bd930 2023-10-25T00:22:08.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.536+0000 7feef7fff640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feefc065eb0 0x7feefc068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.536+0000 7feef7fff640 1 -- 172.21.15.107:0/2907986130 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fef08084810 con 0x7fef100bd930 2023-10-25T00:22:08.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.536+0000 7fef0f7fe640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feefc065eb0 0x7feefc068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.536+0000 7fef0f7fe640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feefc065eb0 0x7feefc068370 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fef00006e90 tx=0x7fef00006de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:08.542 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.540+0000 7feef7fff640 1 -- 172.21.15.107:0/2907986130 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fef08050d80 con 0x7fef100bd930 2023-10-25T00:22:08.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 -- 172.21.15.107:0/2907986130 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7feef0005a90 con 0x7fef100bd930 2023-10-25T00:22:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7feef7fff640 1 -- 172.21.15.107:0/2907986130 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v72) v1 ==== 95+0+11 (secure 0 0 0) 0x7fef080170a0 con 0x7fef100bd930 2023-10-25T00:22:08.824 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-25T00:22:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 -- 172.21.15.107:0/2907986130 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feefc065eb0 msgr2=0x7feefc068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feefc065eb0 0x7feefc068370 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fef00006e90 tx=0x7fef00006de0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 -- 172.21.15.107:0/2907986130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef100bd930 msgr2=0x7fef10150c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef100bd930 0x7fef10150c00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fef0800ca70 tx=0x7fef0800cf30 comp rx=0 tx=0).stop 2023-10-25T00:22:08.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 -- 172.21.15.107:0/2907986130 shutdown_connections 2023-10-25T00:22:08.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef100bd930 0x7fef10150c00 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef100af9b0 0x7fef100c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feefc065eb0 0x7feefc068370 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 --2- 172.21.15.107:0/2907986130 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fef1006b360 0x7fef100c0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 -- 172.21.15.107:0/2907986130 >> 172.21.15.107:0/2907986130 conn(0x7fef10071b00 msgr2=0x7fef100adda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 -- 172.21.15.107:0/2907986130 shutdown_connections 2023-10-25T00:22:08.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.824+0000 7fef1606a640 1 -- 172.21.15.107:0/2907986130 wait complete. 2023-10-25T00:22:08.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-10-25T00:22:08.950 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.948+0000 7ffa37c16640 1 Processor -- start 2023-10-25T00:22:08.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.948+0000 7ffa37c16640 1 -- start start 2023-10-25T00:22:08.951 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa37c16640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa30071ee0 0x7ffa3006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa37c16640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffa300c7590 0x7ffa300a9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa37c16640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa30150360 con 0x7ffa30155cc0 2023-10-25T00:22:08.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa37c16640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa30156c90 con 0x7ffa300c7590 2023-10-25T00:22:08.952 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa37c16640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa301567b0 con 0x7ffa30071ee0 2023-10-25T00:22:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa351b1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa30071ee0 0x7ffa3006d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa361b3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ffa300c7590 0x7ffa300a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa359b2640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ffa30155cc0 0x7ffa301560a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:58780/0 (socket says 172.21.15.107:58780) 2023-10-25T00:22:08.953 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa359b2640 1 -- 172.21.15.107:0/4212844380 learned_addr learned my addr 172.21.15.107:0/4212844380 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:08.954 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa351b1640 1 -- 172.21.15.107:0/4212844380 >> v2:172.21.15.110:3300/0 conn(0x7ffa300c7590 msgr2=0x7ffa300a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa351b1640 1 --2- 172.21.15.107:0/4212844380 >> v2:172.21.15.110:3300/0 conn(0x7ffa300c7590 0x7ffa300a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa351b1640 1 -- 172.21.15.107:0/4212844380 >> v1:172.21.15.110:6789/0 conn(0x7ffa30155cc0 legacy=0x7ffa301560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.952+0000 7ffa351b1640 1 -- 172.21.15.107:0/4212844380 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa300c7e20 con 0x7ffa30071ee0 2023-10-25T00:22:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa351b1640 1 --2- 172.21.15.107:0/4212844380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa30071ee0 0x7ffa3006d0d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009d90 tx=0x7ffa2401c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=744250998a6b985e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:08.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa349b0640 1 -- 172.21.15.107:0/4212844380 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa240a57e0 con 0x7ffa30071ee0 2023-10-25T00:22:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa349b0640 1 -- 172.21.15.107:0/4212844380 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa240a5940 con 0x7ffa30071ee0 2023-10-25T00:22:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa349b0640 1 -- 172.21.15.107:0/4212844380 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa240a5cd0 con 0x7ffa30071ee0 2023-10-25T00:22:08.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 -- 172.21.15.107:0/4212844380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa30071ee0 msgr2=0x7ffa3006d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 --2- 172.21.15.107:0/4212844380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa30071ee0 0x7ffa3006d0d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009d90 tx=0x7ffa2401c830 comp rx=0 tx=0).stop 2023-10-25T00:22:08.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 -- 172.21.15.107:0/4212844380 shutdown_connections 2023-10-25T00:22:08.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 --2- 172.21.15.107:0/4212844380 >> v2:172.21.15.110:3300/0 conn(0x7ffa300c7590 0x7ffa300a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 --2- 172.21.15.107:0/4212844380 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa30071ee0 0x7ffa3006d0d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 -- 172.21.15.107:0/4212844380 >> 172.21.15.107:0/4212844380 conn(0x7ffa300c8020 msgr2=0x7ffa300c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:08.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 -- 172.21.15.107:0/4212844380 shutdown_connections 2023-10-25T00:22:08.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 -- 172.21.15.107:0/4212844380 wait complete. 2023-10-25T00:22:08.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 Processor -- start 2023-10-25T00:22:08.959 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 -- start start 2023-10-25T00:22:08.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa30071ee0 0x7ffa3014cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa300c7590 0x7ffa301ec320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30155cc0 0x7ffa301ec860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa300aad00 con 0x7ffa30155cc0 2023-10-25T00:22:08.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa301eaf30 con 0x7ffa300c7590 2023-10-25T00:22:08.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa37c16640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa301eb210 con 0x7ffa30071ee0 2023-10-25T00:22:08.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa361b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30155cc0 0x7ffa301ec860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa361b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30155cc0 0x7ffa301ec860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60288/0 (socket says 172.21.15.107:60288) 2023-10-25T00:22:08.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa361b3640 1 -- 172.21.15.107:0/2185601144 learned_addr learned my addr 172.21.15.107:0/2185601144 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:08.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa359b2640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa30071ee0 0x7ffa3014cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa351b1640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa300c7590 0x7ffa301ec320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa361b3640 1 -- 172.21.15.107:0/2185601144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa30071ee0 msgr2=0x7ffa3014cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.956+0000 7ffa361b3640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa30071ee0 0x7ffa3014cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa361b3640 1 -- 172.21.15.107:0/2185601144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa300c7590 msgr2=0x7ffa301ec320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:08.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa361b3640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa300c7590 0x7ffa301ec320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:08.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa361b3640 1 -- 172.21.15.107:0/2185601144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa24009a70 con 0x7ffa30155cc0 2023-10-25T00:22:08.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa351b1640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa300c7590 0x7ffa301ec320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:22:08.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa361b3640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30155cc0 0x7ffa301ec860 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000e7f0 tx=0x7ffa2000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:08.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa2e7fc640 1 -- 172.21.15.107:0/2185601144 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa20009530 con 0x7ffa30155cc0 2023-10-25T00:22:08.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa2e7fc640 1 -- 172.21.15.107:0/2185601144 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa2000eea0 con 0x7ffa30155cc0 2023-10-25T00:22:08.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa37c16640 1 -- 172.21.15.107:0/2185601144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa301eb4f0 con 0x7ffa30155cc0 2023-10-25T00:22:08.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa2e7fc640 1 -- 172.21.15.107:0/2185601144 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2000cb60 con 0x7ffa30155cc0 2023-10-25T00:22:08.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa37c16640 1 -- 172.21.15.107:0/2185601144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa301ebab0 con 0x7ffa30155cc0 2023-10-25T00:22:08.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa37c16640 1 -- 172.21.15.107:0/2185601144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa301ebca0 con 0x7ffa30155cc0 2023-10-25T00:22:08.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa2e7fc640 1 -- 172.21.15.107:0/2185601144 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ffa2000ccc0 con 0x7ffa30155cc0 2023-10-25T00:22:08.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa2e7fc640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa1c065e60 0x7ffa1c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:08.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa2e7fc640 1 -- 172.21.15.107:0/2185601144 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7ffa20084ad0 con 0x7ffa30155cc0 2023-10-25T00:22:08.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa359b2640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa1c065e60 0x7ffa1c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:08.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.960+0000 7ffa359b2640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa1c065e60 0x7ffa1c068320 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ffa30154b10 tx=0x7ffa28007e90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:08.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:08.964+0000 7ffa2e7fc640 1 -- 172.21.15.107:0/2185601144 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ffa20050f40 con 0x7ffa30155cc0 2023-10-25T00:22:09.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.104+0000 7ffa37c16640 1 -- 172.21.15.107:0/2185601144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffa3004e130 con 0x7ffa30155cc0 2023-10-25T00:22:09.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa2e7fc640 1 -- 172.21.15.107:0/2185601144 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v72) v1 ==== 99+0+11 (secure 0 0 0) 0x7ffa200170a0 con 0x7ffa30155cc0 2023-10-25T00:22:09.107 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-25T00:22:09.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 -- 172.21.15.107:0/2185601144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa1c065e60 msgr2=0x7ffa1c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa1c065e60 0x7ffa1c068320 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ffa30154b10 tx=0x7ffa28007e90 comp rx=0 tx=0).stop 2023-10-25T00:22:09.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 -- 172.21.15.107:0/2185601144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30155cc0 msgr2=0x7ffa301ec860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30155cc0 0x7ffa301ec860 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000e7f0 tx=0x7ffa2000ecb0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 -- 172.21.15.107:0/2185601144 shutdown_connections 2023-10-25T00:22:09.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffa30155cc0 0x7ffa301ec860 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffa300c7590 0x7ffa301ec320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ffa1c065e60 0x7ffa1c068320 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 --2- 172.21.15.107:0/2185601144 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffa30071ee0 0x7ffa3014cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 -- 172.21.15.107:0/2185601144 >> 172.21.15.107:0/2185601144 conn(0x7ffa300c8020 msgr2=0x7ffa3014ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:09.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 -- 172.21.15.107:0/2185601144 shutdown_connections 2023-10-25T00:22:09.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.108+0000 7ffa37c16640 1 -- 172.21.15.107:0/2185601144 wait complete. 2023-10-25T00:22:09.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-10-25T00:22:09.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.236+0000 7f410e20a640 1 Processor -- start 2023-10-25T00:22:09.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.236+0000 7f410e20a640 1 -- start start 2023-10-25T00:22:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f410e20a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f41080c1b30 0x7f4108072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f410e20a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f41080aa9f0 0x7f4108155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f410e20a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41080befa0 con 0x7f41080a9300 2023-10-25T00:22:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f410e20a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41080a9b80 con 0x7f41080c1b30 2023-10-25T00:22:09.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f410e20a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41080aa0b0 con 0x7f41080aa9f0 2023-10-25T00:22:09.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f4106ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f41080a9300 0x7f41080aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47454/0 (socket says 172.21.15.107:47454) 2023-10-25T00:22:09.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f4106ffd640 1 -- 172.21.15.107:0/3558914514 learned_addr learned my addr 172.21.15.107:0/3558914514 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:09.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f41077fe640 1 --2- 172.21.15.107:0/3558914514 >> v2:172.21.15.107:3300/0 conn(0x7f41080c1b30 0x7f4108072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f41067fc640 1 -- 172.21.15.107:0/3558914514 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331040208 0 0) 0x7f41080befa0 con 0x7f41080a9300 2023-10-25T00:22:09.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f4107fff640 1 --2- 172.21.15.107:0/3558914514 >> v2:172.21.15.110:3300/0 conn(0x7f41080aa9f0 0x7f4108155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f41067fc640 1 -- 172.21.15.107:0/3558914514 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f0003530 con 0x7f41080a9300 2023-10-25T00:22:09.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f41077fe640 1 -- 172.21.15.107:0/3558914514 >> v2:172.21.15.110:3300/0 conn(0x7f41080aa9f0 msgr2=0x7f4108155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:09.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f41077fe640 1 --2- 172.21.15.107:0/3558914514 >> v2:172.21.15.110:3300/0 conn(0x7f41080aa9f0 0x7f4108155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f41077fe640 1 -- 172.21.15.107:0/3558914514 >> v1:172.21.15.107:6789/0 conn(0x7f41080a9300 legacy=0x7f41080aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f41077fe640 1 -- 172.21.15.107:0/3558914514 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41080bf1f0 con 0x7f41080c1b30 2023-10-25T00:22:09.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f41077fe640 1 --2- 172.21.15.107:0/3558914514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41080c1b30 0x7f4108072210 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f41080c5d90 tx=0x7f40f801c080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c620cd718e9fd307 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:09.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f41067fc640 1 -- 172.21.15.107:0/3558914514 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f80a4d20 con 0x7f41080c1b30 2023-10-25T00:22:09.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f41067fc640 1 -- 172.21.15.107:0/3558914514 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f80a4e80 con 0x7f41080c1b30 2023-10-25T00:22:09.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f41067fc640 1 -- 172.21.15.107:0/3558914514 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f80ac660 con 0x7f41080c1b30 2023-10-25T00:22:09.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f410e20a640 1 -- 172.21.15.107:0/3558914514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41080c1b30 msgr2=0x7f4108072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f410e20a640 1 --2- 172.21.15.107:0/3558914514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41080c1b30 0x7f4108072210 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f41080c5d90 tx=0x7f40f801c080 comp rx=0 tx=0).stop 2023-10-25T00:22:09.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f410e20a640 1 -- 172.21.15.107:0/3558914514 shutdown_connections 2023-10-25T00:22:09.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f410e20a640 1 --2- 172.21.15.107:0/3558914514 >> v2:172.21.15.110:3300/0 conn(0x7f41080aa9f0 0x7f4108155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f410e20a640 1 --2- 172.21.15.107:0/3558914514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41080c1b30 0x7f4108072210 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.240+0000 7f410e20a640 1 -- 172.21.15.107:0/3558914514 >> 172.21.15.107:0/3558914514 conn(0x7f41080c2860 msgr2=0x7f41080c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:09.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 -- 172.21.15.107:0/3558914514 shutdown_connections 2023-10-25T00:22:09.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 -- 172.21.15.107:0/3558914514 wait complete. 2023-10-25T00:22:09.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 Processor -- start 2023-10-25T00:22:09.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 -- start start 2023-10-25T00:22:09.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41080a9300 0x7f41081ee620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41080c1b30 0x7f41081eeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f410814f3c0 0x7f410814f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410814e570 con 0x7f41080a9300 2023-10-25T00:22:09.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410814dfb0 con 0x7f41080c1b30 2023-10-25T00:22:09.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410814e290 con 0x7f410814f3c0 2023-10-25T00:22:09.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f41077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41080a9300 0x7f41081ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f41077fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41080a9300 0x7f41081ee620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60312/0 (socket says 172.21.15.107:60312) 2023-10-25T00:22:09.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f41077fe640 1 -- 172.21.15.107:0/1598411531 learned_addr learned my addr 172.21.15.107:0/1598411531 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:09.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f4106ffd640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41080c1b30 0x7f41081eeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f41077fe640 1 -- 172.21.15.107:0/1598411531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f410814f3c0 msgr2=0x7f410814f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f41077fe640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f410814f3c0 0x7f410814f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f41077fe640 1 -- 172.21.15.107:0/1598411531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41080c1b30 msgr2=0x7f41081eeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f41077fe640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41080c1b30 0x7f41081eeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f41077fe640 1 -- 172.21.15.107:0/1598411531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f8009420 con 0x7f41080a9300 2023-10-25T00:22:09.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f4106ffd640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41080c1b30 0x7f41081eeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:09.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f41077fe640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41080a9300 0x7f41081ee620 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f40f80af040 tx=0x7f40f8018e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:09.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f40effff640 1 -- 172.21.15.107:0/1598411531 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f80acdd0 con 0x7f41080a9300 2023-10-25T00:22:09.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f40effff640 1 -- 172.21.15.107:0/1598411531 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f80b5410 con 0x7f41080a9300 2023-10-25T00:22:09.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 -- 172.21.15.107:0/1598411531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410814ebc0 con 0x7f41080a9300 2023-10-25T00:22:09.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f40effff640 1 -- 172.21.15.107:0/1598411531 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f80b57c0 con 0x7f41080a9300 2023-10-25T00:22:09.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 -- 172.21.15.107:0/1598411531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410814dbf0 con 0x7f41080a9300 2023-10-25T00:22:09.253 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.244+0000 7f410e20a640 1 -- 172.21.15.107:0/1598411531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410804e130 con 0x7f41080a9300 2023-10-25T00:22:09.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.248+0000 7f40effff640 1 -- 172.21.15.107:0/1598411531 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f40f80b3040 con 0x7f41080a9300 2023-10-25T00:22:09.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.248+0000 7f40effff640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40f4065e60 0x7f40f4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.248+0000 7f40effff640 1 -- 172.21.15.107:0/1598411531 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f40f811e5b0 con 0x7f41080a9300 2023-10-25T00:22:09.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.248+0000 7f4106ffd640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40f4065e60 0x7f40f4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.248+0000 7f4106ffd640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40f4065e60 0x7f40f4068320 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f410806ae50 tx=0x7f40fc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:09.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.252+0000 7f40effff640 1 -- 172.21.15.107:0/1598411531 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f40f80eab10 con 0x7f41080a9300 2023-10-25T00:22:09.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f410e20a640 1 -- 172.21.15.107:0/1598411531 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7f41080a90e0 con 0x7f41080a9300 2023-10-25T00:22:09.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f40effff640 1 -- 172.21.15.107:0/1598411531 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v72) v1 ==== 95+0+11 (secure 0 0 0) 0x7f40f80ee130 con 0x7f41080a9300 2023-10-25T00:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-25T00:22:09.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f410e20a640 1 -- 172.21.15.107:0/1598411531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40f4065e60 msgr2=0x7f40f4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f410e20a640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40f4065e60 0x7f40f4068320 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f410806ae50 tx=0x7f40fc009040 comp rx=0 tx=0).stop 2023-10-25T00:22:09.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f410e20a640 1 -- 172.21.15.107:0/1598411531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41080a9300 msgr2=0x7f41081ee620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f410e20a640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41080a9300 0x7f41081ee620 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f40f80af040 tx=0x7f40f8018e20 comp rx=0 tx=0).stop 2023-10-25T00:22:09.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f410e20a640 1 -- 172.21.15.107:0/1598411531 shutdown_connections 2023-10-25T00:22:09.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f410e20a640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f40f4065e60 0x7f40f4068320 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f410e20a640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f410814f3c0 0x7f410814f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f410e20a640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41080c1b30 0x7f41081eeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f410e20a640 1 --2- 172.21.15.107:0/1598411531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41080a9300 0x7f41081ee620 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.392+0000 7f410e20a640 1 -- 172.21.15.107:0/1598411531 >> 172.21.15.107:0/1598411531 conn(0x7f41080c2860 msgr2=0x7f41080c0660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:09.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.396+0000 7f410e20a640 1 -- 172.21.15.107:0/1598411531 shutdown_connections 2023-10-25T00:22:09.397 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.396+0000 7f410e20a640 1 -- 172.21.15.107:0/1598411531 wait complete. 2023-10-25T00:22:09.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-25T00:22:09.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f838be50640 1 Processor -- start 2023-10-25T00:22:09.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f838be50640 1 -- start start 2023-10-25T00:22:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f838be50640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f838414fd10 0x7f83840c7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f838be50640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8384071f70 0x7f83840a9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f838be50640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8384155b70 0x7f8384156000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f838be50640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83840c71a0 con 0x7f838414fd10 2023-10-25T00:22:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f838be50640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83841569f0 con 0x7f8384071f70 2023-10-25T00:22:09.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f838be50640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838406d150 con 0x7f8384155b70 2023-10-25T00:22:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f8389bec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f838414fd10 0x7f83840c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f8389bec640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f838414fd10 0x7f83840c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60322/0 (socket says 172.21.15.107:60322) 2023-10-25T00:22:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f8389bec640 1 -- 172.21.15.107:0/212054906 learned_addr learned my addr 172.21.15.107:0/212054906 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f838a3ed640 1 --2- 172.21.15.107:0/212054906 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8384155b70 0x7f8384156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f83893eb640 1 --2- 172.21.15.107:0/212054906 >> v2:172.21.15.110:3300/0 conn(0x7f8384071f70 0x7f83840a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f8389bec640 1 -- 172.21.15.107:0/212054906 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8384155b70 msgr2=0x7f8384156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f8389bec640 1 --2- 172.21.15.107:0/212054906 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8384155b70 0x7f8384156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f8389bec640 1 -- 172.21.15.107:0/212054906 >> v2:172.21.15.110:3300/0 conn(0x7f8384071f70 msgr2=0x7f83840a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f8389bec640 1 --2- 172.21.15.107:0/212054906 >> v2:172.21.15.110:3300/0 conn(0x7f8384071f70 0x7f83840a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f8389bec640 1 -- 172.21.15.107:0/212054906 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838406d430 con 0x7f838414fd10 2023-10-25T00:22:09.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.516+0000 7f83893eb640 1 --2- 172.21.15.107:0/212054906 >> v2:172.21.15.110:3300/0 conn(0x7f8384071f70 0x7f83840a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:09.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f8389bec640 1 --2- 172.21.15.107:0/212054906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f838414fd10 0x7f83840c7680 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f838406dbc0 tx=0x7f838001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68a168aad7711bc server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:09.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f8388bea640 1 -- 172.21.15.107:0/212054906 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83800a89b0 con 0x7f838414fd10 2023-10-25T00:22:09.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f8388bea640 1 -- 172.21.15.107:0/212054906 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83800a8b10 con 0x7f838414fd10 2023-10-25T00:22:09.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f8388bea640 1 -- 172.21.15.107:0/212054906 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83800a8ea0 con 0x7f838414fd10 2023-10-25T00:22:09.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f838be50640 1 -- 172.21.15.107:0/212054906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f838414fd10 msgr2=0x7f83840c7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f838be50640 1 --2- 172.21.15.107:0/212054906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f838414fd10 0x7f83840c7680 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f838406dbc0 tx=0x7f838001ecf0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f838be50640 1 -- 172.21.15.107:0/212054906 shutdown_connections 2023-10-25T00:22:09.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f838be50640 1 --2- 172.21.15.107:0/212054906 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8384155b70 0x7f8384156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f838be50640 1 --2- 172.21.15.107:0/212054906 >> v2:172.21.15.110:3300/0 conn(0x7f8384071f70 0x7f83840a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f838be50640 1 --2- 172.21.15.107:0/212054906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f838414fd10 0x7f83840c7680 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f838be50640 1 -- 172.21.15.107:0/212054906 >> 172.21.15.107:0/212054906 conn(0x7f83840c8020 msgr2=0x7f83840c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:09.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f838be50640 1 -- 172.21.15.107:0/212054906 shutdown_connections 2023-10-25T00:22:09.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f838be50640 1 -- 172.21.15.107:0/212054906 wait complete. 2023-10-25T00:22:09.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f838be50640 1 Processor -- start 2023-10-25T00:22:09.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.520+0000 7f838be50640 1 -- start start 2023-10-25T00:22:09.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f838be50640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8384071f70 0x7f83841df790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f838be50640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f838414fd10 0x7f83840c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f838be50640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8384155b70 0x7f8384153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.529 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f838be50640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83840c3880 con 0x7f838414fd10 2023-10-25T00:22:09.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f838be50640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8384150f30 con 0x7f8384155b70 2023-10-25T00:22:09.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f838be50640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83840c40c0 con 0x7f8384071f70 2023-10-25T00:22:09.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f83893eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f838414fd10 0x7f83840c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f83893eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f838414fd10 0x7f83840c2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60334/0 (socket says 172.21.15.107:60334) 2023-10-25T00:22:09.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f83893eb640 1 -- 172.21.15.107:0/2228090641 learned_addr learned my addr 172.21.15.107:0/2228090641 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:09.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f8389bec640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8384071f70 0x7f83841df790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f838a3ed640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8384155b70 0x7f8384153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f83893eb640 1 -- 172.21.15.107:0/2228090641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8384071f70 msgr2=0x7f83841df790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f83893eb640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8384071f70 0x7f83841df790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f83893eb640 1 -- 172.21.15.107:0/2228090641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8384155b70 msgr2=0x7f8384153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f83893eb640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8384155b70 0x7f8384153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f83893eb640 1 -- 172.21.15.107:0/2228090641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83800094d0 con 0x7f838414fd10 2023-10-25T00:22:09.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f8389bec640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8384071f70 0x7f83841df790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:09.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f838a3ed640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8384155b70 0x7f8384153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:09.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f83893eb640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f838414fd10 0x7f83840c2ca0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f837400c7f0 tx=0x7f837400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:09.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f837a7fc640 1 -- 172.21.15.107:0/2228090641 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8374007820 con 0x7f838414fd10 2023-10-25T00:22:09.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f837a7fc640 1 -- 172.21.15.107:0/2228090641 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8374007980 con 0x7f838414fd10 2023-10-25T00:22:09.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f838be50640 1 -- 172.21.15.107:0/2228090641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83840c46a0 con 0x7f838414fd10 2023-10-25T00:22:09.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f837a7fc640 1 -- 172.21.15.107:0/2228090641 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8374007d30 con 0x7f838414fd10 2023-10-25T00:22:09.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f838be50640 1 -- 172.21.15.107:0/2228090641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83840c3d50 con 0x7f838414fd10 2023-10-25T00:22:09.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f838be50640 1 -- 172.21.15.107:0/2228090641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83840c29c0 con 0x7f838414fd10 2023-10-25T00:22:09.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f837a7fc640 1 -- 172.21.15.107:0/2228090641 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f83740113f0 con 0x7f838414fd10 2023-10-25T00:22:09.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f837a7fc640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8368065e90 0x7f8368068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f837a7fc640 1 -- 172.21.15.107:0/2228090641 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f8374084d20 con 0x7f838414fd10 2023-10-25T00:22:09.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f8389bec640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8368065e90 0x7f8368068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.524+0000 7f8389bec640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8368065e90 0x7f8368068350 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f83800b2000 tx=0x7f83800aa070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:09.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.528+0000 7f837a7fc640 1 -- 172.21.15.107:0/2228090641 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f83840c29c0 con 0x7f838414fd10 2023-10-25T00:22:09.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.680+0000 7f838be50640 1 -- 172.21.15.107:0/2228090641 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f83840c29c0 con 0x7f838414fd10 2023-10-25T00:22:09.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f837a7fc640 1 -- 172.21.15.107:0/2228090641 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 88 entries v72) v1 ==== 81+0+5426 (secure 0 0 0) 0x7f83840c29c0 con 0x7f838414fd10 2023-10-25T00:22:09.684 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2187059523 2023-10-26T00:21:39.173084+0000 2023-10-25T00:22:09.684 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/657626630 2023-10-26T00:21:19.167088+0000 2023-10-25T00:22:09.684 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/3771564389 2023-10-26T00:20:19.149072+0000 2023-10-25T00:22:09.684 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/3771564389 2023-10-26T00:20:19.149072+0000 2023-10-25T00:22:09.685 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2970408860 2023-10-26T00:19:04.122659+0000 2023-10-25T00:22:09.685 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/418589140 2023-10-26T00:18:49.118366+0000 2023-10-25T00:22:09.685 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/985135365 2023-10-26T00:17:39.037219+0000 2023-10-25T00:22:09.685 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/985135365 2023-10-26T00:17:39.037219+0000 2023-10-25T00:22:09.685 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/208713198 2023-10-26T00:17:24.032902+0000 2023-10-25T00:22:09.685 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/208713198 2023-10-26T00:17:24.032902+0000 2023-10-25T00:22:09.686 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/1619327197 2023-10-26T00:15:22.673615+0000 2023-10-25T00:22:09.686 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/175581297 2023-10-26T00:15:18.921240+0000 2023-10-25T00:22:09.686 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/418589140 2023-10-26T00:18:49.118366+0000 2023-10-25T00:22:09.686 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2856791739 2023-10-26T00:13:48.893880+0000 2023-10-25T00:22:09.686 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/898258574 2023-10-26T00:13:18.883076+0000 2023-10-25T00:22:09.687 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3396644475 2023-10-26T00:19:44.136579+0000 2023-10-25T00:22:09.687 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/3370462806 2023-10-26T00:12:33.862408+0000 2023-10-25T00:22:09.687 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/1101018338 2023-10-25T23:59:48.457105+0000 2023-10-25T00:22:09.687 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/838886697 2023-10-25T23:56:18.381650+0000 2023-10-25T00:22:09.687 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1381726420 2023-10-26T00:01:28.521014+0000 2023-10-25T00:22:09.688 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/1619327197 2023-10-26T00:15:22.673615+0000 2023-10-25T00:22:09.688 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1381726420 2023-10-26T00:01:28.521014+0000 2023-10-25T00:22:09.688 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/3103569740 2023-10-26T00:01:58.532147+0000 2023-10-25T00:22:09.688 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/780830443 2023-10-26T00:06:23.634894+0000 2023-10-25T00:22:09.688 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/1101018338 2023-10-25T23:59:48.457105+0000 2023-10-25T00:22:09.689 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1555840329 2023-10-26T00:05:18.610829+0000 2023-10-25T00:22:09.689 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/3836115666 2023-10-26T00:08:58.700067+0000 2023-10-25T00:22:09.689 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/898258574 2023-10-26T00:13:18.883076+0000 2023-10-25T00:22:09.689 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1508236506 2023-10-25T23:58:58.439244+0000 2023-10-25T00:22:09.689 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2348043270 2023-10-25T23:58:53.436687+0000 2023-10-25T00:22:09.689 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/204494054 2023-10-25T23:58:03.418411+0000 2023-10-25T00:22:09.690 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1555840329 2023-10-26T00:05:18.610829+0000 2023-10-25T00:22:09.690 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/3721101226 2023-10-25T23:58:03.418295+0000 2023-10-25T00:22:09.690 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2480841919 2023-10-25T23:55:58.372958+0000 2023-10-25T00:22:09.690 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/2577776351 2023-10-26T00:04:34.728765+0000 2023-10-25T00:22:09.690 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1246928633 2023-10-26T00:05:13.609231+0000 2023-10-25T00:22:09.691 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/913067382 2023-10-26T00:08:08.675592+0000 2023-10-25T00:22:09.691 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/1031240534 2023-10-25T23:57:23.404313+0000 2023-10-25T00:22:09.691 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/727042497 2023-10-25T23:54:13.340839+0000 2023-10-25T00:22:09.691 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2970408860 2023-10-26T00:19:04.122659+0000 2023-10-25T00:22:09.692 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2480841919 2023-10-25T23:55:58.372958+0000 2023-10-25T00:22:09.692 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/201284779 2023-10-26T00:02:23.541454+0000 2023-10-25T00:22:09.692 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/201284779 2023-10-26T00:02:23.541454+0000 2023-10-25T00:22:09.692 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/838886697 2023-10-25T23:56:18.381650+0000 2023-10-25T00:22:09.692 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1292188015 2023-10-26T00:00:48.479959+0000 2023-10-25T00:22:09.693 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1317796403 2023-10-26T00:08:23.682615+0000 2023-10-25T00:22:09.693 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1220982369 2023-10-25T23:55:23.361145+0000 2023-10-25T00:22:09.693 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2348043270 2023-10-25T23:58:53.436687+0000 2023-10-25T00:22:09.693 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/3370462806 2023-10-26T00:12:33.862408+0000 2023-10-25T00:22:09.693 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1220982369 2023-10-25T23:55:23.361145+0000 2023-10-25T00:22:09.694 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/204494054 2023-10-25T23:58:03.418411+0000 2023-10-25T00:22:09.694 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/175581297 2023-10-26T00:15:18.921240+0000 2023-10-25T00:22:09.694 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2856791739 2023-10-26T00:13:48.893880+0000 2023-10-25T00:22:09.694 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/1031240534 2023-10-25T23:57:23.404313+0000 2023-10-25T00:22:09.694 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/326715537 2023-10-26T00:11:03.824824+0000 2023-10-25T00:22:09.695 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1004926771 2023-10-25T23:57:28.406683+0000 2023-10-25T00:22:09.695 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/657626630 2023-10-26T00:21:19.167088+0000 2023-10-25T00:22:09.695 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1004926771 2023-10-25T23:57:28.406683+0000 2023-10-25T00:22:09.695 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/3721101226 2023-10-25T23:58:03.418295+0000 2023-10-25T00:22:09.695 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3474118649 2023-10-26T00:11:18.831318+0000 2023-10-25T00:22:09.696 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3232233949 2023-10-26T00:03:28.568587+0000 2023-10-25T00:22:09.696 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/2577776351 2023-10-26T00:04:34.728765+0000 2023-10-25T00:22:09.696 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/326715537 2023-10-26T00:11:03.824824+0000 2023-10-25T00:22:09.696 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2276136116 2023-10-26T00:09:18.706706+0000 2023-10-25T00:22:09.697 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/319918260 2023-10-26T00:03:53.579302+0000 2023-10-25T00:22:09.697 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3474118649 2023-10-26T00:11:18.831318+0000 2023-10-25T00:22:09.697 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1292188015 2023-10-26T00:00:48.479959+0000 2023-10-25T00:22:09.697 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/174863522 2023-10-26T00:05:53.622773+0000 2023-10-25T00:22:09.697 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/319918260 2023-10-26T00:03:53.579302+0000 2023-10-25T00:22:09.697 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/3103569740 2023-10-26T00:01:58.532147+0000 2023-10-25T00:22:09.698 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2150827048 2023-10-26T00:07:48.667043+0000 2023-10-25T00:22:09.698 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/4180698155 2023-10-26T00:04:28.591981+0000 2023-10-25T00:22:09.698 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3396644475 2023-10-26T00:19:44.136579+0000 2023-10-25T00:22:09.698 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/4180698155 2023-10-26T00:04:28.591981+0000 2023-10-25T00:22:09.699 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2187059523 2023-10-26T00:21:39.173084+0000 2023-10-25T00:22:09.699 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1246928633 2023-10-26T00:05:13.609231+0000 2023-10-25T00:22:09.699 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/174863522 2023-10-26T00:05:53.622773+0000 2023-10-25T00:22:09.699 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1508236506 2023-10-25T23:58:58.439244+0000 2023-10-25T00:22:09.699 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/780830443 2023-10-26T00:06:23.634894+0000 2023-10-25T00:22:09.700 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2150827048 2023-10-26T00:07:48.667043+0000 2023-10-25T00:22:09.700 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/913067382 2023-10-26T00:08:08.675592+0000 2023-10-25T00:22:09.700 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1317796403 2023-10-26T00:08:23.682615+0000 2023-10-25T00:22:09.700 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2276136116 2023-10-26T00:09:18.706706+0000 2023-10-25T00:22:09.700 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3232233949 2023-10-26T00:03:28.568587+0000 2023-10-25T00:22:09.700 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/727042497 2023-10-25T23:54:13.340839+0000 2023-10-25T00:22:09.701 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1788268787 2023-10-26T00:10:38.784519+0000 2023-10-25T00:22:09.701 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/3836115666 2023-10-26T00:08:58.700067+0000 2023-10-25T00:22:09.701 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1788268787 2023-10-26T00:10:38.784519+0000 2023-10-25T00:22:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:listed 88 entries 2023-10-25T00:22:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 -- 172.21.15.107:0/2228090641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8368065e90 msgr2=0x7f8368068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8368065e90 0x7f8368068350 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f83800b2000 tx=0x7f83800aa070 comp rx=0 tx=0).stop 2023-10-25T00:22:09.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 -- 172.21.15.107:0/2228090641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f838414fd10 msgr2=0x7f83840c2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f838414fd10 0x7f83840c2ca0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f837400c7f0 tx=0x7f837400ccb0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 -- 172.21.15.107:0/2228090641 shutdown_connections 2023-10-25T00:22:09.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8384155b70 0x7f8384153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f838414fd10 0x7f83840c2ca0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8368065e90 0x7f8368068350 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 --2- 172.21.15.107:0/2228090641 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8384071f70 0x7f83841df790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 -- 172.21.15.107:0/2228090641 >> 172.21.15.107:0/2228090641 conn(0x7f83840c8020 msgr2=0x7f8384142e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 -- 172.21.15.107:0/2228090641 shutdown_connections 2023-10-25T00:22:09.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.684+0000 7f838be50640 1 -- 172.21.15.107:0/2228090641 wait complete. 2023-10-25T00:22:09.706 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-10-25T00:22:09.706 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-25T00:22:09.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-25T00:22:09.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5b1846640 1 Processor -- start 2023-10-25T00:22:09.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5b1846640 1 -- start start 2023-10-25T00:22:09.872 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5b1846640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5ac1561e0 0x7fd5ac0c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5b1846640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5ac1565c0 con 0x7fd5ac06bc30 2023-10-25T00:22:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5b1846640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5ac06a2b0 con 0x7fd5ac0c1da0 2023-10-25T00:22:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5b1846640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5ac06a590 con 0x7fd5ac1561e0 2023-10-25T00:22:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5aaffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fd5ac1561e0 0x7fd5ac0c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5aa7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd5ac06bc30 0x7fd5ac15c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47456/0 (socket says 172.21.15.107:47456) 2023-10-25T00:22:09.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5aa7fc640 1 -- 172.21.15.107:0/485864781 learned_addr learned my addr 172.21.15.107:0/485864781 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:09.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5aaffd640 1 -- 172.21.15.107:0/485864781 >> v1:172.21.15.110:6789/0 conn(0x7fd5ac0c1da0 legacy=0x7fd5ac1634d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5aaffd640 1 -- 172.21.15.107:0/485864781 >> v1:172.21.15.107:6789/0 conn(0x7fd5ac06bc30 legacy=0x7fd5ac15c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5a9ffb640 1 -- 172.21.15.107:0/485864781 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130056318 0 0) 0x7fd5ac1565c0 con 0x7fd5ac06bc30 2023-10-25T00:22:09.874 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5aaffd640 1 -- 172.21.15.107:0/485864781 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ac06a870 con 0x7fd5ac1561e0 2023-10-25T00:22:09.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5aaffd640 1 --2- 172.21.15.107:0/485864781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5ac1561e0 0x7fd5ac0c1860 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009d90 tx=0x7fd59c01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d9a46a880c5a3f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:09.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5a9ffb640 1 -- 172.21.15.107:0/485864781 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c0a5890 con 0x7fd5ac1561e0 2023-10-25T00:22:09.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5a9ffb640 1 -- 172.21.15.107:0/485864781 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59c0a59f0 con 0x7fd5ac1561e0 2023-10-25T00:22:09.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.872+0000 7fd5a9ffb640 1 -- 172.21.15.107:0/485864781 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd59c0a5d80 con 0x7fd5ac1561e0 2023-10-25T00:22:09.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- 172.21.15.107:0/485864781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5ac1561e0 msgr2=0x7fd5ac0c1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 --2- 172.21.15.107:0/485864781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5ac1561e0 0x7fd5ac0c1860 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009d90 tx=0x7fd59c01c8c0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- 172.21.15.107:0/485864781 shutdown_connections 2023-10-25T00:22:09.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 --2- 172.21.15.107:0/485864781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5ac1561e0 0x7fd5ac0c1860 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- 172.21.15.107:0/485864781 >> 172.21.15.107:0/485864781 conn(0x7fd5ac0c2950 msgr2=0x7fd5ac0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:09.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- 172.21.15.107:0/485864781 shutdown_connections 2023-10-25T00:22:09.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- 172.21.15.107:0/485864781 wait complete. 2023-10-25T00:22:09.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 Processor -- start 2023-10-25T00:22:09.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- start start 2023-10-25T00:22:09.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5ac0c1da0 0x7fd5ac1583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5ac14c040 0x7fd5ac14fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5ac1521a0 0x7fd5ac152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac1f26a0 con 0x7fd5ac1521a0 2023-10-25T00:22:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac1589a0 con 0x7fd5ac14c040 2023-10-25T00:22:09.881 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac1593e0 con 0x7fd5ac0c1da0 2023-10-25T00:22:09.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5ab7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5ac1521a0 0x7fd5ac152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5ab7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5ac1521a0 0x7fd5ac152620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60350/0 (socket says 172.21.15.107:60350) 2023-10-25T00:22:09.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5ab7fe640 1 -- 172.21.15.107:0/3366943133 learned_addr learned my addr 172.21.15.107:0/3366943133 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:09.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5ab7fe640 1 -- 172.21.15.107:0/3366943133 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5ac0c1da0 msgr2=0x7fd5ac1583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.882 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5ab7fe640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5ac0c1da0 0x7fd5ac1583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5ab7fe640 1 -- 172.21.15.107:0/3366943133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5ac14c040 msgr2=0x7fd5ac14fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5ab7fe640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5ac14c040 0x7fd5ac14fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5ab7fe640 1 -- 172.21.15.107:0/3366943133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c009a70 con 0x7fd5ac1521a0 2023-10-25T00:22:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5ab7fe640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5ac1521a0 0x7fd5ac152620 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400c430 tx=0x7fd5a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd593fff640 1 -- 172.21.15.107:0/3366943133 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4005220 con 0x7fd5ac1521a0 2023-10-25T00:22:09.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd593fff640 1 -- 172.21.15.107:0/3366943133 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a4005380 con 0x7fd5ac1521a0 2023-10-25T00:22:09.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- 172.21.15.107:0/3366943133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5ac1596c0 con 0x7fd5ac1521a0 2023-10-25T00:22:09.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd593fff640 1 -- 172.21.15.107:0/3366943133 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4007270 con 0x7fd5ac1521a0 2023-10-25T00:22:09.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- 172.21.15.107:0/3366943133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5ac14c4c0 con 0x7fd5ac1521a0 2023-10-25T00:22:09.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.876+0000 7fd5b1846640 1 -- 172.21.15.107:0/3366943133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ac0746e0 con 0x7fd5ac1521a0 2023-10-25T00:22:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.880+0000 7fd593fff640 1 -- 172.21.15.107:0/3366943133 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd5a40073d0 con 0x7fd5ac1521a0 2023-10-25T00:22:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.880+0000 7fd593fff640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd598065e60 0x7fd598068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.880+0000 7fd593fff640 1 -- 172.21.15.107:0/3366943133 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fd5a4084060 con 0x7fd5ac1521a0 2023-10-25T00:22:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.880+0000 7fd5aaffd640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd598065e60 0x7fd598068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:09.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.880+0000 7fd5aaffd640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd598065e60 0x7fd598068320 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0af000 tx=0x7fd59c004ac0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:09.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:09.884+0000 7fd593fff640 1 -- 172.21.15.107:0/3366943133 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd5a40504a0 con 0x7fd5ac1521a0 2023-10-25T00:22:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.028+0000 7fd5b1846640 1 -- 172.21.15.107:0/3366943133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5ac154630 con 0x7fd5ac1521a0 2023-10-25T00:22:10.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.032+0000 7fd593fff640 1 -- 172.21.15.107:0/3366943133 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+27395 (secure 0 0 0) 0x7fd5a4053ac0 con 0x7fd5ac1521a0 2023-10-25T00:22:10.032 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:10.033 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":72,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-25T00:21:39.173159+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2f","mappings":[{"from":3,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":2}]},{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6835/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6837/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.110:6834/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.110:6832/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.107:6835/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.107:6837/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.110:6834/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.110:6832/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.107:6835/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.107:6834/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.110:6833/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6836/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6836/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.107:6834/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6835/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6836/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.110:6834/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6835/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.110:6835/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6834/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.107:6837/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.107:6837/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6832/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6833/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.110:6833/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6834/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.110:6835/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6836/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6836/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6834/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.110:6835/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.107:6834/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.107:6834/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.110:6833/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.107:6837/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6834/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.110:6833/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.110:6832/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.107:6836/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6837/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.107:6837/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.110:6833/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6837/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.107:6835/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.107:6837/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6837/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6835/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.110:6835/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.107:6836/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.107:6834/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.110:6832/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.107:6835/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6835/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.110:6832/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.110:6833/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6836/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6834/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6836/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.110:6834/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6837/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6836/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.110:6835/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6834/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6835/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6835/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.110:6832/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6835/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.107:6836/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6833/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6835/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.107:6836/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6834/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.107:6836/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6836/1788268787":"2023-10-26T00:10:38.784519+0000","172.21.15.110:6835/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.107:6837/1788268787":"2023-10-26T00:10:38.784519+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-25T00:22:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.032+0000 7fd5b1846640 1 -- 172.21.15.107:0/3366943133 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd598065e60 msgr2=0x7fd598068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.032+0000 7fd5b1846640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd598065e60 0x7fd598068320 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fd59c0af000 tx=0x7fd59c004ac0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.032+0000 7fd5b1846640 1 -- 172.21.15.107:0/3366943133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5ac1521a0 msgr2=0x7fd5ac152620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.032+0000 7fd5b1846640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5ac1521a0 0x7fd5ac152620 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400c430 tx=0x7fd5a400c8f0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.032+0000 7fd5b1846640 1 -- 172.21.15.107:0/3366943133 shutdown_connections 2023-10-25T00:22:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.032+0000 7fd5b1846640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5ac1521a0 0x7fd5ac152620 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.032+0000 7fd5b1846640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd5ac14c040 0x7fd5ac14fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.032+0000 7fd5b1846640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd598065e60 0x7fd598068320 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.032+0000 7fd5b1846640 1 --2- 172.21.15.107:0/3366943133 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd5ac0c1da0 0x7fd5ac1583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.036+0000 7fd5b1846640 1 -- 172.21.15.107:0/3366943133 >> 172.21.15.107:0/3366943133 conn(0x7fd5ac0c2950 msgr2=0x7fd5ac1550d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.036+0000 7fd5b1846640 1 -- 172.21.15.107:0/3366943133 shutdown_connections 2023-10-25T00:22:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.036+0000 7fd5b1846640 1 -- 172.21.15.107:0/3366943133 wait complete. 2023-10-25T00:22:10.050 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-10-24T23:53:09.786122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-10-24T23:53:10.766645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-25T00:22:10.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-25T00:22:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff850243640 1 Processor -- start 2023-10-25T00:22:10.171 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff850243640 1 -- start start 2023-10-25T00:22:10.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff850243640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8480aa9f0 0x7ff848072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff850243640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff8480a9300 0x7ff848155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff850243640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8480befa0 con 0x7ff8480c1b30 2023-10-25T00:22:10.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff850243640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8480a9b80 con 0x7ff8480aa9f0 2023-10-25T00:22:10.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff850243640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8480aa0b0 con 0x7ff8480a9300 2023-10-25T00:22:10.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff84d7de640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8480aa9f0 0x7ff848072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff84dfdf640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff8480c1b30 0x7ff8480aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47462/0 (socket says 172.21.15.107:47462) 2023-10-25T00:22:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff84d7de640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff8480aa9f0 0x7ff848072210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60362/0 (socket says 172.21.15.107:60362) 2023-10-25T00:22:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff84dfdf640 1 -- 172.21.15.107:0/1975203186 learned_addr learned my addr 172.21.15.107:0/1975203186 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff84d7de640 1 -- 172.21.15.107:0/1975203186 >> v2:172.21.15.110:3300/0 conn(0x7ff8480a9300 msgr2=0x7ff848155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:10.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff84d7de640 1 --2- 172.21.15.107:0/1975203186 >> v2:172.21.15.110:3300/0 conn(0x7ff8480a9300 0x7ff848155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff84d7de640 1 -- 172.21.15.107:0/1975203186 >> v1:172.21.15.107:6789/0 conn(0x7ff8480c1b30 legacy=0x7ff8480aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff84cfdd640 1 -- 172.21.15.107:0/1975203186 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005881068 0 0) 0x7ff8480befa0 con 0x7ff8480c1b30 2023-10-25T00:22:10.175 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff84d7de640 1 -- 172.21.15.107:0/1975203186 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8480bf1f0 con 0x7ff8480aa9f0 2023-10-25T00:22:10.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.172+0000 7ff84d7de640 1 --2- 172.21.15.107:0/1975203186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8480aa9f0 0x7ff848072210 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7ff84806ec00 tx=0x7ff83801c0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3527225a66633f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:10.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84cfdd640 1 -- 172.21.15.107:0/1975203186 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8380a4d40 con 0x7ff8480aa9f0 2023-10-25T00:22:10.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84cfdd640 1 -- 172.21.15.107:0/1975203186 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8380ac440 con 0x7ff8480aa9f0 2023-10-25T00:22:10.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84cfdd640 1 -- 172.21.15.107:0/1975203186 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8380a4ea0 con 0x7ff8480aa9f0 2023-10-25T00:22:10.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 -- 172.21.15.107:0/1975203186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8480aa9f0 msgr2=0x7ff848072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 --2- 172.21.15.107:0/1975203186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8480aa9f0 0x7ff848072210 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7ff84806ec00 tx=0x7ff83801c0a0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 -- 172.21.15.107:0/1975203186 shutdown_connections 2023-10-25T00:22:10.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 --2- 172.21.15.107:0/1975203186 >> v2:172.21.15.110:3300/0 conn(0x7ff8480a9300 0x7ff848155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.179 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 --2- 172.21.15.107:0/1975203186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8480aa9f0 0x7ff848072210 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 -- 172.21.15.107:0/1975203186 >> 172.21.15.107:0/1975203186 conn(0x7ff8480c2860 msgr2=0x7ff8480c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:10.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 -- 172.21.15.107:0/1975203186 shutdown_connections 2023-10-25T00:22:10.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 -- 172.21.15.107:0/1975203186 wait complete. 2023-10-25T00:22:10.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 Processor -- start 2023-10-25T00:22:10.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 -- start start 2023-10-25T00:22:10.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8480a9300 0x7ff84806c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8480c1b30 0x7ff8480c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8480c55b0 0x7ff848150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8481eefc0 con 0x7ff8480a9300 2023-10-25T00:22:10.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8481ee3f0 con 0x7ff8480c55b0 2023-10-25T00:22:10.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8481ee690 con 0x7ff8480c1b30 2023-10-25T00:22:10.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84dfdf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8480a9300 0x7ff84806c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84dfdf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8480a9300 0x7ff84806c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60366/0 (socket says 172.21.15.107:60366) 2023-10-25T00:22:10.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84dfdf640 1 -- 172.21.15.107:0/3154543727 learned_addr learned my addr 172.21.15.107:0/3154543727 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:10.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84dfdf640 1 -- 172.21.15.107:0/3154543727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8480c1b30 msgr2=0x7ff8480c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84dfdf640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8480c1b30 0x7ff8480c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84dfdf640 1 -- 172.21.15.107:0/3154543727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8480c55b0 msgr2=0x7ff848150a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84dfdf640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8480c55b0 0x7ff848150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84dfdf640 1 -- 172.21.15.107:0/3154543727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff838009420 con 0x7ff8480a9300 2023-10-25T00:22:10.183 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff84dfdf640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8480a9300 0x7ff84806c120 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7ff84400c7c0 tx=0x7ff84400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:10.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff83e7fc640 1 -- 172.21.15.107:0/3154543727 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84400a640 con 0x7ff8480a9300 2023-10-25T00:22:10.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff83e7fc640 1 -- 172.21.15.107:0/3154543727 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84400a7c0 con 0x7ff8480a9300 2023-10-25T00:22:10.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 -- 172.21.15.107:0/3154543727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8481ee970 con 0x7ff8480a9300 2023-10-25T00:22:10.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff83e7fc640 1 -- 172.21.15.107:0/3154543727 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84400ab70 con 0x7ff8480a9300 2023-10-25T00:22:10.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.176+0000 7ff850243640 1 -- 172.21.15.107:0/3154543727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8480b9620 con 0x7ff8480a9300 2023-10-25T00:22:10.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.180+0000 7ff850243640 1 -- 172.21.15.107:0/3154543727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84804e130 con 0x7ff8480a9300 2023-10-25T00:22:10.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.180+0000 7ff83e7fc640 1 -- 172.21.15.107:0/3154543727 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff84400acd0 con 0x7ff8480a9300 2023-10-25T00:22:10.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.180+0000 7ff83e7fc640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff82c065e90 0x7ff82c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.180+0000 7ff83e7fc640 1 -- 172.21.15.107:0/3154543727 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7ff844084280 con 0x7ff8480a9300 2023-10-25T00:22:10.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.180+0000 7ff84d7de640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff82c065e90 0x7ff82c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.180+0000 7ff84d7de640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff82c065e90 0x7ff82c068350 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff8380ae000 tx=0x7ff838004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:10.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.184+0000 7ff83e7fc640 1 -- 172.21.15.107:0/3154543727 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff8440506c0 con 0x7ff8480a9300 2023-10-25T00:22:10.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.328+0000 7ff850243640 1 -- 172.21.15.107:0/3154543727 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff8480a90e0 con 0x7ff8480a9300 2023-10-25T00:22:10.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff83e7fc640 1 -- 172.21.15.107:0/3154543727 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff8440180a0 con 0x7ff8480a9300 2023-10-25T00:22:10.331 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-25T00:22:10.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 -- 172.21.15.107:0/3154543727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff82c065e90 msgr2=0x7ff82c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff82c065e90 0x7ff82c068350 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7ff8380ae000 tx=0x7ff838004740 comp rx=0 tx=0).stop 2023-10-25T00:22:10.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 -- 172.21.15.107:0/3154543727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8480a9300 msgr2=0x7ff84806c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8480a9300 0x7ff84806c120 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7ff84400c7c0 tx=0x7ff84400cc80 comp rx=0 tx=0).stop 2023-10-25T00:22:10.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 -- 172.21.15.107:0/3154543727 shutdown_connections 2023-10-25T00:22:10.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff8480c55b0 0x7ff848150a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff82c065e90 0x7ff82c068350 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff8480c1b30 0x7ff8480c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 --2- 172.21.15.107:0/3154543727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8480a9300 0x7ff84806c120 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 -- 172.21.15.107:0/3154543727 >> 172.21.15.107:0/3154543727 conn(0x7ff8480c2860 msgr2=0x7ff8480c43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:10.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 -- 172.21.15.107:0/3154543727 shutdown_connections 2023-10-25T00:22:10.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.332+0000 7ff850243640 1 -- 172.21.15.107:0/3154543727 wait complete. 2023-10-25T00:22:10.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-25T00:22:10.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f181502c640 1 Processor -- start 2023-10-25T00:22:10.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f181502c640 1 -- start start 2023-10-25T00:22:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f181502c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f18100aa9f0 0x7f1810072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.469 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f181502c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f18100a9300 0x7f1810155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f181502c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18100befa0 con 0x7f18100c1b30 2023-10-25T00:22:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f181502c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18100a9b80 con 0x7f18100aa9f0 2023-10-25T00:22:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f181502c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18100aa0b0 con 0x7f18100a9300 2023-10-25T00:22:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f18100c1b30 0x7f18100aa2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47478/0 (socket says 172.21.15.107:47478) 2023-10-25T00:22:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180ed9d640 1 -- 172.21.15.107:0/1815291252 learned_addr learned my addr 172.21.15.107:0/1815291252 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180f59e640 1 --2- 172.21.15.107:0/1815291252 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f18100a9300 0x7f1810155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180e59c640 1 --2- 172.21.15.107:0/1815291252 >> v2:172.21.15.110:3300/0 conn(0x7f18100aa9f0 0x7f1810072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702545822 0 0) 0x7f18100befa0 con 0x7f18100c1b30 2023-10-25T00:22:10.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f8003530 con 0x7f18100c1b30 2023-10-25T00:22:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1587965005 0 0) 0x7f17f8003530 con 0x7f18100c1b30 2023-10-25T00:22:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f18100a9300 msgr2=0x7f1810155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 --2- 172.21.15.107:0/1815291252 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f18100a9300 0x7f1810155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.472 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 >> v2:172.21.15.110:3300/0 conn(0x7f18100aa9f0 msgr2=0x7f1810072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:10.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 --2- 172.21.15.107:0/1815291252 >> v2:172.21.15.110:3300/0 conn(0x7f18100aa9f0 0x7f1810072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18100bf1f0 con 0x7f18100c1b30 2023-10-25T00:22:10.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f1800001680 con 0x7f18100c1b30 2023-10-25T00:22:10.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 >> v1:172.21.15.107:6789/0 conn(0x7f18100c1b30 legacy=0x7f18100aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 --2- 172.21.15.107:0/1815291252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8003980 0x7f17f8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 --2- 172.21.15.107:0/1815291252 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f18100a9300 0x7f17f80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 --2- 172.21.15.107:0/1815291252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f8004d10 0x7f17f8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18100a9b80 con 0x7f17f8003980 2023-10-25T00:22:10.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18100aa0b0 con 0x7f17f8004d10 2023-10-25T00:22:10.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f800bba0 con 0x7f18100a9300 2023-10-25T00:22:10.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180e59c640 1 --2- 172.21.15.107:0/1815291252 >> v2:172.21.15.110:3300/0 conn(0x7f18100aa9f0 0x7f1810072210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:22:10.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180ed9d640 1 --2- 172.21.15.107:0/1815291252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8003980 0x7f17f8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180ed9d640 1 -- 172.21.15.107:0/1815291252 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f18100a9300 msgr2=0x7f17f80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:10.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180ed9d640 1 --2- 172.21.15.107:0/1815291252 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f18100a9300 0x7f17f80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180ed9d640 1 -- 172.21.15.107:0/1815291252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f8004d10 msgr2=0x7f17f8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:10.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180ed9d640 1 --2- 172.21.15.107:0/1815291252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f17f8004d10 0x7f17f8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.468+0000 7f180ed9d640 1 -- 172.21.15.107:0/1815291252 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f8003530 con 0x7f17f8003980 2023-10-25T00:22:10.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180ed9d640 1 --2- 172.21.15.107:0/1815291252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8003980 0x7f17f8003da0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f180000c240 tx=0x7f180001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35ac92fad353193 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:10.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18000058e0 con 0x7f17f8003980 2023-10-25T00:22:10.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1800005a60 con 0x7f17f8003980 2023-10-25T00:22:10.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180dd9b640 1 -- 172.21.15.107:0/1815291252 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800005e10 con 0x7f17f8003980 2023-10-25T00:22:10.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 -- 172.21.15.107:0/1815291252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8003980 msgr2=0x7f17f8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 --2- 172.21.15.107:0/1815291252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17f8003980 0x7f17f8003da0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f180000c240 tx=0x7f180001cc60 comp rx=0 tx=0).stop 2023-10-25T00:22:10.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180f59e640 1 -- 172.21.15.107:0/1815291252 reap_dead start 2023-10-25T00:22:10.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 -- 172.21.15.107:0/1815291252 shutdown_connections 2023-10-25T00:22:10.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 -- 172.21.15.107:0/1815291252 >> 172.21.15.107:0/1815291252 conn(0x7f18100c2860 msgr2=0x7f18100c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:10.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 -- 172.21.15.107:0/1815291252 shutdown_connections 2023-10-25T00:22:10.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 -- 172.21.15.107:0/1815291252 wait complete. 2023-10-25T00:22:10.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 Processor -- start 2023-10-25T00:22:10.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 -- start start 2023-10-25T00:22:10.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1810154a80 0x7f18101595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1810159b20 0x7f181006c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f181006cb70 0x7f18101eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18101ef1f0 con 0x7f181006cb70 2023-10-25T00:22:10.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1810156950 con 0x7f1810154a80 2023-10-25T00:22:10.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f181502c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18100724a0 con 0x7f1810159b20 2023-10-25T00:22:10.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f181006cb70 0x7f18101eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f181006cb70 0x7f18101eb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60392/0 (socket says 172.21.15.107:60392) 2023-10-25T00:22:10.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180f59e640 1 -- 172.21.15.107:0/1743012243 learned_addr learned my addr 172.21.15.107:0/1743012243 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:10.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180e59c640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1810159b20 0x7f181006c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180ed9d640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1810154a80 0x7f18101595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180f59e640 1 -- 172.21.15.107:0/1743012243 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1810159b20 msgr2=0x7f181006c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180f59e640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1810159b20 0x7f181006c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180f59e640 1 -- 172.21.15.107:0/1743012243 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1810154a80 msgr2=0x7f18101595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180f59e640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1810154a80 0x7f18101595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180f59e640 1 -- 172.21.15.107:0/1743012243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18101ee570 con 0x7f181006cb70 2023-10-25T00:22:10.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180ed9d640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1810154a80 0x7f18101595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:10.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f180f59e640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f181006cb70 0x7f18101eb080 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f18100c5db0 tx=0x7f1808009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:10.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.472+0000 7f17f77fe640 1 -- 172.21.15.107:0/1743012243 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1808003db0 con 0x7f181006cb70 2023-10-25T00:22:10.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.476+0000 7f17f77fe640 1 -- 172.21.15.107:0/1743012243 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180800e6f0 con 0x7f181006cb70 2023-10-25T00:22:10.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.476+0000 7f17f77fe640 1 -- 172.21.15.107:0/1743012243 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180800eaa0 con 0x7f181006cb70 2023-10-25T00:22:10.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.476+0000 7f181502c640 1 -- 172.21.15.107:0/1743012243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18101ee850 con 0x7f181006cb70 2023-10-25T00:22:10.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.476+0000 7f181502c640 1 -- 172.21.15.107:0/1743012243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18101eee10 con 0x7f181006cb70 2023-10-25T00:22:10.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.476+0000 7f181502c640 1 -- 172.21.15.107:0/1743012243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1810159f00 con 0x7f181006cb70 2023-10-25T00:22:10.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.476+0000 7f17f77fe640 1 -- 172.21.15.107:0/1743012243 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f180800ec00 con 0x7f181006cb70 2023-10-25T00:22:10.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.476+0000 7f17f77fe640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17fc065e90 0x7f17fc068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.476+0000 7f17f77fe640 1 -- 172.21.15.107:0/1743012243 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f18080848d0 con 0x7f181006cb70 2023-10-25T00:22:10.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.476+0000 7f180ed9d640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17fc065e90 0x7f17fc068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.476+0000 7f180ed9d640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17fc065e90 0x7f17fc068350 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f18000aa000 tx=0x7f1800001a80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:10.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.480+0000 7f17f77fe640 1 -- 172.21.15.107:0/1743012243 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1810159f00 con 0x7f181006cb70 2023-10-25T00:22:10.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 -- 172.21.15.107:0/1743012243 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f18101ef330 con 0x7f181006cb70 2023-10-25T00:22:10.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f17f77fe640 1 -- 172.21.15.107:0/1743012243 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v72) v1 ==== 92+0+10 (secure 0 0 0) 0x7f180801a020 con 0x7f181006cb70 2023-10-25T00:22:10.632 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-25T00:22:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 -- 172.21.15.107:0/1743012243 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17fc065e90 msgr2=0x7f17fc068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17fc065e90 0x7f17fc068350 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f18000aa000 tx=0x7f1800001a80 comp rx=0 tx=0).stop 2023-10-25T00:22:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 -- 172.21.15.107:0/1743012243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f181006cb70 msgr2=0x7f18101eb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f181006cb70 0x7f18101eb080 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f18100c5db0 tx=0x7f1808009e30 comp rx=0 tx=0).stop 2023-10-25T00:22:10.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 -- 172.21.15.107:0/1743012243 shutdown_connections 2023-10-25T00:22:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f181006cb70 0x7f18101eb080 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f17fc065e90 0x7f17fc068350 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1810159b20 0x7f181006c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 --2- 172.21.15.107:0/1743012243 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1810154a80 0x7f18101595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 -- 172.21.15.107:0/1743012243 >> 172.21.15.107:0/1743012243 conn(0x7f18100c2860 msgr2=0x7f18100bb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 -- 172.21.15.107:0/1743012243 shutdown_connections 2023-10-25T00:22:10.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.632+0000 7f181502c640 1 -- 172.21.15.107:0/1743012243 wait complete. 2023-10-25T00:22:10.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-25T00:22:10.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.768+0000 7f32c7bf1640 1 Processor -- start 2023-10-25T00:22:10.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c7bf1640 1 -- start start 2023-10-25T00:22:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c7bf1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f32c00ab1a0 0x7f32c00c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c7bf1640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f32c00bceb0 0x7f32c015c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c7bf1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c00c1ce0 con 0x7f32c0072720 2023-10-25T00:22:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c7bf1640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c00be1d0 con 0x7f32c00bceb0 2023-10-25T00:22:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c7bf1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32c00c0240 con 0x7f32c00ab1a0 2023-10-25T00:22:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c618e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f32c00bceb0 0x7f32c015c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c518c640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f32c0072720 0x7f32c00bfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:47490/0 (socket says 172.21.15.107:47490) 2023-10-25T00:22:10.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c618e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f32c00bceb0 0x7f32c015c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60398/0 (socket says 172.21.15.107:60398) 2023-10-25T00:22:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c518c640 1 -- 172.21.15.107:0/1910915739 learned_addr learned my addr 172.21.15.107:0/1910915739 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c598d640 1 --2- 172.21.15.107:0/1910915739 >> v2:172.21.15.110:3300/0 conn(0x7f32c00ab1a0 0x7f32c00c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c618e640 1 -- 172.21.15.107:0/1910915739 >> v2:172.21.15.110:3300/0 conn(0x7f32c00ab1a0 msgr2=0x7f32c00c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c618e640 1 --2- 172.21.15.107:0/1910915739 >> v2:172.21.15.110:3300/0 conn(0x7f32c00ab1a0 0x7f32c00c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c618e640 1 -- 172.21.15.107:0/1910915739 >> v1:172.21.15.107:6789/0 conn(0x7f32c0072720 legacy=0x7f32c00bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c618e640 1 -- 172.21.15.107:0/1910915739 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c00c0880 con 0x7f32c00bceb0 2023-10-25T00:22:10.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c618e640 1 --2- 172.21.15.107:0/1910915739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32c00bceb0 0x7f32c015c660 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f32c00c3960 tx=0x7f32b401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f0ecbe7ecb4c65 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:10.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c498b640 1 -- 172.21.15.107:0/1910915739 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b40ac600 con 0x7f32c00bceb0 2023-10-25T00:22:10.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c498b640 1 -- 172.21.15.107:0/1910915739 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b40a8050 con 0x7f32c00bceb0 2023-10-25T00:22:10.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.772+0000 7f32c498b640 1 -- 172.21.15.107:0/1910915739 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b40ac980 con 0x7f32c00bceb0 2023-10-25T00:22:10.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 -- 172.21.15.107:0/1910915739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32c00bceb0 msgr2=0x7f32c015c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 --2- 172.21.15.107:0/1910915739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32c00bceb0 0x7f32c015c660 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f32c00c3960 tx=0x7f32b401c3a0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 -- 172.21.15.107:0/1910915739 shutdown_connections 2023-10-25T00:22:10.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 --2- 172.21.15.107:0/1910915739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32c00bceb0 0x7f32c015c660 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 --2- 172.21.15.107:0/1910915739 >> v2:172.21.15.110:3300/0 conn(0x7f32c00ab1a0 0x7f32c00c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 -- 172.21.15.107:0/1910915739 >> 172.21.15.107:0/1910915739 conn(0x7f32c00c2860 msgr2=0x7f32c00c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:10.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 -- 172.21.15.107:0/1910915739 shutdown_connections 2023-10-25T00:22:10.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 -- 172.21.15.107:0/1910915739 wait complete. 2023-10-25T00:22:10.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 Processor -- start 2023-10-25T00:22:10.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 -- start start 2023-10-25T00:22:10.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32c00ab1a0 0x7f32c014d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32c00bceb0 0x7f32c01ed810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32c006ed10 0x7f32c01eac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c0156c60 con 0x7f32c00bceb0 2023-10-25T00:22:10.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c00c5120 con 0x7f32c006ed10 2023-10-25T00:22:10.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c7bf1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c01ea7b0 con 0x7f32c00ab1a0 2023-10-25T00:22:10.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c518c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32c00bceb0 0x7f32c01ed810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c518c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32c00bceb0 0x7f32c01ed810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60410/0 (socket says 172.21.15.107:60410) 2023-10-25T00:22:10.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c518c640 1 -- 172.21.15.107:0/163596406 learned_addr learned my addr 172.21.15.107:0/163596406 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:10.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c618e640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32c006ed10 0x7f32c01eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c598d640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32c00ab1a0 0x7f32c014d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c518c640 1 -- 172.21.15.107:0/163596406 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32c00ab1a0 msgr2=0x7f32c014d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c518c640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32c00ab1a0 0x7f32c014d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c518c640 1 -- 172.21.15.107:0/163596406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32c006ed10 msgr2=0x7f32c01eac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c518c640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32c006ed10 0x7f32c01eac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c518c640 1 -- 172.21.15.107:0/163596406 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b4009420 con 0x7f32c00bceb0 2023-10-25T00:22:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c618e640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32c006ed10 0x7f32c01eac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32c518c640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32c00bceb0 0x7f32c01ed810 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f32b800c450 tx=0x7f32b800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:10.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.776+0000 7f32be7fc640 1 -- 172.21.15.107:0/163596406 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8005cf0 con 0x7f32c00bceb0 2023-10-25T00:22:10.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.780+0000 7f32be7fc640 1 -- 172.21.15.107:0/163596406 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b8005e50 con 0x7f32c00bceb0 2023-10-25T00:22:10.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.780+0000 7f32c7bf1640 1 -- 172.21.15.107:0/163596406 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c01edd50 con 0x7f32c00bceb0 2023-10-25T00:22:10.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.780+0000 7f32be7fc640 1 -- 172.21.15.107:0/163596406 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8007270 con 0x7f32c00bceb0 2023-10-25T00:22:10.785 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.780+0000 7f32c7bf1640 1 -- 172.21.15.107:0/163596406 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c00c4e40 con 0x7f32c00bceb0 2023-10-25T00:22:10.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.780+0000 7f32c7bf1640 1 -- 172.21.15.107:0/163596406 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c00beff0 con 0x7f32c00bceb0 2023-10-25T00:22:10.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.780+0000 7f32be7fc640 1 -- 172.21.15.107:0/163596406 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f32b80073d0 con 0x7f32c00bceb0 2023-10-25T00:22:10.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.780+0000 7f32be7fc640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f32ac065e90 0x7f32ac068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:10.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.780+0000 7f32be7fc640 1 -- 172.21.15.107:0/163596406 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f32b8083e10 con 0x7f32c00bceb0 2023-10-25T00:22:10.786 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.780+0000 7f32c598d640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f32ac065e90 0x7f32ac068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.780+0000 7f32c598d640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f32ac065e90 0x7f32ac068350 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009f10 tx=0x7f32b000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:10.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.784+0000 7f32be7fc640 1 -- 172.21.15.107:0/163596406 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f32b8050300 con 0x7f32c00bceb0 2023-10-25T00:22:10.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.924+0000 7f32c7bf1640 1 -- 172.21.15.107:0/163596406 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f32c0158980 con 0x7f32c00bceb0 2023-10-25T00:22:10.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32be7fc640 1 -- 172.21.15.107:0/163596406 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v72) v1 ==== 99+0+11 (secure 0 0 0) 0x7f32b8017070 con 0x7f32c00bceb0 2023-10-25T00:22:10.927 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-25T00:22:10.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 -- 172.21.15.107:0/163596406 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f32ac065e90 msgr2=0x7f32ac068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f32ac065e90 0x7f32ac068350 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f32b0009f10 tx=0x7f32b000b040 comp rx=0 tx=0).stop 2023-10-25T00:22:10.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 -- 172.21.15.107:0/163596406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32c00bceb0 msgr2=0x7f32c01ed810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:10.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32c00bceb0 0x7f32c01ed810 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f32b800c450 tx=0x7f32b800c910 comp rx=0 tx=0).stop 2023-10-25T00:22:10.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 -- 172.21.15.107:0/163596406 shutdown_connections 2023-10-25T00:22:10.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32c006ed10 0x7f32c01eac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32c00bceb0 0x7f32c01ed810 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f32ac065e90 0x7f32ac068350 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 --2- 172.21.15.107:0/163596406 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f32c00ab1a0 0x7f32c014d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:10.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 -- 172.21.15.107:0/163596406 >> 172.21.15.107:0/163596406 conn(0x7f32c00c2860 msgr2=0x7f32c0071450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:10.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 -- 172.21.15.107:0/163596406 shutdown_connections 2023-10-25T00:22:10.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:10.928+0000 7f32c7bf1640 1 -- 172.21.15.107:0/163596406 wait complete. 2023-10-25T00:22:10.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-25T00:22:11.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd3ccb2640 1 Processor -- start 2023-10-25T00:22:11.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd3ccb2640 1 -- start start 2023-10-25T00:22:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd3ccb2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efd3806b360 0x7efd38069d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd3ccb2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efd380bd930 0x7efd3806a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd3ccb2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3806a7c0 con 0x7efd380af9b0 2023-10-25T00:22:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd3ccb2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd380c5870 con 0x7efd3806b360 2023-10-25T00:22:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd3ccb2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd380c8180 con 0x7efd380bd930 2023-10-25T00:22:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd35d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efd3806b360 0x7efd38069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd36d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7efd380bd930 0x7efd3806a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd35d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7efd3806b360 0x7efd38069d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60422/0 (socket says 172.21.15.107:60422) 2023-10-25T00:22:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd35d9b640 1 -- 172.21.15.107:0/3318296243 learned_addr learned my addr 172.21.15.107:0/3318296243 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd35d9b640 1 -- 172.21.15.107:0/3318296243 >> v2:172.21.15.110:3300/0 conn(0x7efd380bd930 msgr2=0x7efd3806a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd35d9b640 1 --2- 172.21.15.107:0/3318296243 >> v2:172.21.15.110:3300/0 conn(0x7efd380bd930 0x7efd3806a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd35d9b640 1 -- 172.21.15.107:0/3318296243 >> v1:172.21.15.110:6789/0 conn(0x7efd380af9b0 legacy=0x7efd380c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.056+0000 7efd35d9b640 1 -- 172.21.15.107:0/3318296243 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd38155430 con 0x7efd3806b360 2023-10-25T00:22:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd35d9b640 1 --2- 172.21.15.107:0/3318296243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd3806b360 0x7efd38069d40 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7efd38158fa0 tx=0x7efd2c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3250323814e1e318 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3559a640 1 -- 172.21.15.107:0/3318296243 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0ab640 con 0x7efd3806b360 2023-10-25T00:22:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3559a640 1 -- 172.21.15.107:0/3318296243 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2c0a7070 con 0x7efd3806b360 2023-10-25T00:22:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3559a640 1 -- 172.21.15.107:0/3318296243 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0ab9c0 con 0x7efd3806b360 2023-10-25T00:22:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3318296243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd3806b360 msgr2=0x7efd38069d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 --2- 172.21.15.107:0/3318296243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd3806b360 0x7efd38069d40 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7efd38158fa0 tx=0x7efd2c01c440 comp rx=0 tx=0).stop 2023-10-25T00:22:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3318296243 shutdown_connections 2023-10-25T00:22:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 --2- 172.21.15.107:0/3318296243 >> v2:172.21.15.110:3300/0 conn(0x7efd380bd930 0x7efd3806a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 --2- 172.21.15.107:0/3318296243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd3806b360 0x7efd38069d40 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3318296243 >> 172.21.15.107:0/3318296243 conn(0x7efd38071b00 msgr2=0x7efd380abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:11.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3318296243 shutdown_connections 2023-10-25T00:22:11.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3318296243 wait complete. 2023-10-25T00:22:11.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 Processor -- start 2023-10-25T00:22:11.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 -- start start 2023-10-25T00:22:11.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd3806b360 0x7efd381f0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd380af9b0 0x7efd3814f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd380bd930 0x7efd381ed160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd380c37b0 con 0x7efd380af9b0 2023-10-25T00:22:11.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd380ab380 con 0x7efd380bd930 2023-10-25T00:22:11.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3ccb2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd380ab660 con 0x7efd3806b360 2023-10-25T00:22:11.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd35d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd380af9b0 0x7efd3814f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd35d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd380af9b0 0x7efd3814f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60434/0 (socket says 172.21.15.107:60434) 2023-10-25T00:22:11.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd35d9b640 1 -- 172.21.15.107:0/3261866023 learned_addr learned my addr 172.21.15.107:0/3261866023 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:11.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd36d9d640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd380bd930 0x7efd381ed160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd3659c640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd3806b360 0x7efd381f0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd35d9b640 1 -- 172.21.15.107:0/3261866023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd3806b360 msgr2=0x7efd381f0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd35d9b640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd3806b360 0x7efd381f0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd35d9b640 1 -- 172.21.15.107:0/3261866023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd380bd930 msgr2=0x7efd381ed160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd35d9b640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd380bd930 0x7efd381ed160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.060+0000 7efd35d9b640 1 -- 172.21.15.107:0/3261866023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c0094d0 con 0x7efd380af9b0 2023-10-25T00:22:11.068 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd35d9b640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd380af9b0 0x7efd3814f8d0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0ad040 tx=0x7efd2c01ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd26ffd640 1 -- 172.21.15.107:0/3261866023 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0abe20 con 0x7efd380af9b0 2023-10-25T00:22:11.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd26ffd640 1 -- 172.21.15.107:0/3261866023 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2c0be3f0 con 0x7efd380af9b0 2023-10-25T00:22:11.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3261866023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd380ab940 con 0x7efd380af9b0 2023-10-25T00:22:11.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd26ffd640 1 -- 172.21.15.107:0/3261866023 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2c0be7a0 con 0x7efd380af9b0 2023-10-25T00:22:11.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3261866023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd381519a0 con 0x7efd380af9b0 2023-10-25T00:22:11.069 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3261866023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd38066560 con 0x7efd380af9b0 2023-10-25T00:22:11.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd26ffd640 1 -- 172.21.15.107:0/3261866023 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7efd2c0c7440 con 0x7efd380af9b0 2023-10-25T00:22:11.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd26ffd640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd18065e60 0x7efd18068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd3659c640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd18065e60 0x7efd18068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd26ffd640 1 -- 172.21.15.107:0/3261866023 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7efd2c11f390 con 0x7efd380af9b0 2023-10-25T00:22:11.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.064+0000 7efd3659c640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd18065e60 0x7efd18068320 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7efd380c4350 tx=0x7efd28009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.068+0000 7efd26ffd640 1 -- 172.21.15.107:0/3261866023 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7efd2c0eb750 con 0x7efd380af9b0 2023-10-25T00:22:11.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.220+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3261866023 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7efd380ac4c0 con 0x7efd380af9b0 2023-10-25T00:22:11.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.220+0000 7efd26ffd640 1 -- 172.21.15.107:0/3261866023 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v72) v1 ==== 95+0+11 (secure 0 0 0) 0x7efd2c0eed70 con 0x7efd380af9b0 2023-10-25T00:22:11.221 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-25T00:22:11.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3261866023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd18065e60 msgr2=0x7efd18068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd18065e60 0x7efd18068320 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7efd380c4350 tx=0x7efd28009040 comp rx=0 tx=0).stop 2023-10-25T00:22:11.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3261866023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd380af9b0 msgr2=0x7efd3814f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd380af9b0 0x7efd3814f8d0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7efd2c0ad040 tx=0x7efd2c01ac80 comp rx=0 tx=0).stop 2023-10-25T00:22:11.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3261866023 shutdown_connections 2023-10-25T00:22:11.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efd380bd930 0x7efd381ed160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd380af9b0 0x7efd3814f8d0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7efd18065e60 0x7efd18068320 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 --2- 172.21.15.107:0/3261866023 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efd3806b360 0x7efd381f0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3261866023 >> 172.21.15.107:0/3261866023 conn(0x7efd38071b00 msgr2=0x7efd38069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:11.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3261866023 shutdown_connections 2023-10-25T00:22:11.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.224+0000 7efd3ccb2640 1 -- 172.21.15.107:0/3261866023 wait complete. 2023-10-25T00:22:11.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-10-25T00:22:11.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.360+0000 7fe0dd950640 1 Processor -- start 2023-10-25T00:22:11.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.360+0000 7fe0dd950640 1 -- start start 2023-10-25T00:22:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0dd950640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0d80ab1a0 0x7fe0d80c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0dd950640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d80bceb0 0x7fe0d815c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0dd950640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d80c1ce0 con 0x7fe0d8072720 2023-10-25T00:22:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0dd950640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d80be1d0 con 0x7fe0d80ab1a0 2023-10-25T00:22:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0dd950640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d80c0240 con 0x7fe0d80bceb0 2023-10-25T00:22:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0d6ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0d80ab1a0 0x7fe0d80c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0d77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d80bceb0 0x7fe0d815c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0d6ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0d80ab1a0 0x7fe0d80c0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.107:46758/0 (socket says 172.21.15.107:46758) 2023-10-25T00:22:11.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0d6ffd640 1 -- 172.21.15.107:0/1888901345 learned_addr learned my addr 172.21.15.107:0/1888901345 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:11.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0d67fc640 1 --1- 172.21.15.107:0/1888901345 >> v1:172.21.15.110:6789/0 conn(0x7fe0d8072720 0x7fe0d80bfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:58798/0 (socket says 172.21.15.107:58798) 2023-10-25T00:22:11.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0d77fe640 1 -- 172.21.15.107:0/1888901345 >> v2:172.21.15.110:3300/0 conn(0x7fe0d80ab1a0 msgr2=0x7fe0d80c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:11.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0d77fe640 1 --2- 172.21.15.107:0/1888901345 >> v2:172.21.15.110:3300/0 conn(0x7fe0d80ab1a0 0x7fe0d80c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0d77fe640 1 -- 172.21.15.107:0/1888901345 >> v1:172.21.15.110:6789/0 conn(0x7fe0d8072720 legacy=0x7fe0d80bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.364+0000 7fe0d77fe640 1 -- 172.21.15.107:0/1888901345 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d80c0880 con 0x7fe0d80bceb0 2023-10-25T00:22:11.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0d77fe640 1 --2- 172.21.15.107:0/1888901345 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d80bceb0 0x7fe0d815c660 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe0d00097c0 tx=0x7fe0d001c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fa4d78d3ff58294 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0d5ffb640 1 -- 172.21.15.107:0/1888901345 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00ab600 con 0x7fe0d80bceb0 2023-10-25T00:22:11.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0d5ffb640 1 -- 172.21.15.107:0/1888901345 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d00a7050 con 0x7fe0d80bceb0 2023-10-25T00:22:11.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0d5ffb640 1 -- 172.21.15.107:0/1888901345 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00ab980 con 0x7fe0d80bceb0 2023-10-25T00:22:11.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 -- 172.21.15.107:0/1888901345 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d80bceb0 msgr2=0x7fe0d815c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 --2- 172.21.15.107:0/1888901345 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d80bceb0 0x7fe0d815c660 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe0d00097c0 tx=0x7fe0d001c330 comp rx=0 tx=0).stop 2023-10-25T00:22:11.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 -- 172.21.15.107:0/1888901345 shutdown_connections 2023-10-25T00:22:11.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 --2- 172.21.15.107:0/1888901345 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d80bceb0 0x7fe0d815c660 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 --2- 172.21.15.107:0/1888901345 >> v2:172.21.15.110:3300/0 conn(0x7fe0d80ab1a0 0x7fe0d80c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 -- 172.21.15.107:0/1888901345 >> 172.21.15.107:0/1888901345 conn(0x7fe0d80c2860 msgr2=0x7fe0d80c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:11.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 -- 172.21.15.107:0/1888901345 shutdown_connections 2023-10-25T00:22:11.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 -- 172.21.15.107:0/1888901345 wait complete. 2023-10-25T00:22:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 Processor -- start 2023-10-25T00:22:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 -- start start 2023-10-25T00:22:11.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d80ab1a0 0x7fe0d814e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d80bceb0 0x7fe0d8157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d806ffd0 0x7fe0d8070480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d814fd50 con 0x7fe0d806ffd0 2023-10-25T00:22:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d81567d0 con 0x7fe0d80ab1a0 2023-10-25T00:22:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0dd950640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d8156a20 con 0x7fe0d80bceb0 2023-10-25T00:22:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0d77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d806ffd0 0x7fe0d8070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0d77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d806ffd0 0x7fe0d8070480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60444/0 (socket says 172.21.15.107:60444) 2023-10-25T00:22:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.368+0000 7fe0d77fe640 1 -- 172.21.15.107:0/202119799 learned_addr learned my addr 172.21.15.107:0/202119799 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0d6ffd640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d80ab1a0 0x7fe0d814e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.375 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0d67fc640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d80bceb0 0x7fe0d8157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0d77fe640 1 -- 172.21.15.107:0/202119799 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d80bceb0 msgr2=0x7fe0d8157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0d77fe640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d80bceb0 0x7fe0d8157f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0d77fe640 1 -- 172.21.15.107:0/202119799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d80ab1a0 msgr2=0x7fe0d814e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0d77fe640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d80ab1a0 0x7fe0d814e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.376 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0d77fe640 1 -- 172.21.15.107:0/202119799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d0009420 con 0x7fe0d806ffd0 2023-10-25T00:22:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0d77fe640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d806ffd0 0x7fe0d8070480 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0019f60 tx=0x7fe0d00190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0cffff640 1 -- 172.21.15.107:0/202119799 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00b4410 con 0x7fe0d806ffd0 2023-10-25T00:22:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0cffff640 1 -- 172.21.15.107:0/202119799 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d00b4570 con 0x7fe0d806ffd0 2023-10-25T00:22:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0dd950640 1 -- 172.21.15.107:0/202119799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d8156d00 con 0x7fe0d806ffd0 2023-10-25T00:22:11.377 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0cffff640 1 -- 172.21.15.107:0/202119799 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00b4900 con 0x7fe0d806ffd0 2023-10-25T00:22:11.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0dd950640 1 -- 172.21.15.107:0/202119799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d81572c0 con 0x7fe0d806ffd0 2023-10-25T00:22:11.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0dd950640 1 -- 172.21.15.107:0/202119799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d80beff0 con 0x7fe0d806ffd0 2023-10-25T00:22:11.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0cffff640 1 -- 172.21.15.107:0/202119799 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe0d00b2040 con 0x7fe0d806ffd0 2023-10-25T00:22:11.378 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0cffff640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065e90 0x7fe0b8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.379 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0cffff640 1 -- 172.21.15.107:0/202119799 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fe0d011e990 con 0x7fe0d806ffd0 2023-10-25T00:22:11.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0d6ffd640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065e90 0x7fe0b8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.380 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.372+0000 7fe0d6ffd640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065e90 0x7fe0b8068350 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe0d80726f0 tx=0x7fe0c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.376+0000 7fe0cffff640 1 -- 172.21.15.107:0/202119799 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe0d00eadf0 con 0x7fe0d806ffd0 2023-10-25T00:22:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.516+0000 7fe0dd950640 1 -- 172.21.15.107:0/202119799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe0d80ab610 con 0x7fe0d806ffd0 2023-10-25T00:22:11.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0cffff640 1 -- 172.21.15.107:0/202119799 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v72) v1 ==== 99+0+11 (secure 0 0 0) 0x7fe0d00c20a0 con 0x7fe0d806ffd0 2023-10-25T00:22:11.519 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-25T00:22:11.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 -- 172.21.15.107:0/202119799 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065e90 msgr2=0x7fe0b8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065e90 0x7fe0b8068350 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe0d80726f0 tx=0x7fe0c8008040 comp rx=0 tx=0).stop 2023-10-25T00:22:11.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 -- 172.21.15.107:0/202119799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d806ffd0 msgr2=0x7fe0d8070480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d806ffd0 0x7fe0d8070480 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0019f60 tx=0x7fe0d00190d0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 -- 172.21.15.107:0/202119799 shutdown_connections 2023-10-25T00:22:11.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0d806ffd0 0x7fe0d8070480 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0b8065e90 0x7fe0b8068350 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0d80bceb0 0x7fe0d8157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 --2- 172.21.15.107:0/202119799 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d80ab1a0 0x7fe0d814e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 -- 172.21.15.107:0/202119799 >> 172.21.15.107:0/202119799 conn(0x7fe0d80c2860 msgr2=0x7fe0d815c330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:11.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 -- 172.21.15.107:0/202119799 shutdown_connections 2023-10-25T00:22:11.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.520+0000 7fe0dd950640 1 -- 172.21.15.107:0/202119799 wait complete. 2023-10-25T00:22:11.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-10-25T00:22:11.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febcaa72640 1 Processor -- start 2023-10-25T00:22:11.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febcaa72640 1 -- start start 2023-10-25T00:22:11.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febcaa72640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40af9b0 0x7febc40c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febcaa72640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7febc406b360 0x7febc4069d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febcaa72640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40c8180 con 0x7febc40bd930 2023-10-25T00:22:11.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febcaa72640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc40c5870 con 0x7febc406b360 2023-10-25T00:22:11.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febcaa72640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc4155430 con 0x7febc40af9b0 2023-10-25T00:22:11.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febc880e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40af9b0 0x7febc40c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febc3fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7febc406b360 0x7febc4069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febc880e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40af9b0 0x7febc40c6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:36432/0 (socket says 172.21.15.107:36432) 2023-10-25T00:22:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febc880e640 1 -- 172.21.15.107:0/225437711 learned_addr learned my addr 172.21.15.107:0/225437711 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febc880e640 1 -- 172.21.15.107:0/225437711 >> v2:172.21.15.110:3300/0 conn(0x7febc406b360 msgr2=0x7febc4069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:11.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febc880e640 1 --2- 172.21.15.107:0/225437711 >> v2:172.21.15.110:3300/0 conn(0x7febc406b360 0x7febc4069d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febc880e640 1 -- 172.21.15.107:0/225437711 >> v1:172.21.15.110:6789/0 conn(0x7febc40bd930 legacy=0x7febc4157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.660+0000 7febc880e640 1 -- 172.21.15.107:0/225437711 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc4155710 con 0x7febc40af9b0 2023-10-25T00:22:11.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febc880e640 1 --2- 172.21.15.107:0/225437711 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40af9b0 0x7febc40c6c90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7febb8009870 tx=0x7febb801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaedc8fd969994e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febc37fe640 1 -- 172.21.15.107:0/225437711 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb80ab610 con 0x7febc40af9b0 2023-10-25T00:22:11.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febc37fe640 1 -- 172.21.15.107:0/225437711 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb80ab770 con 0x7febc40af9b0 2023-10-25T00:22:11.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febc37fe640 1 -- 172.21.15.107:0/225437711 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb80abb00 con 0x7febc40af9b0 2023-10-25T00:22:11.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febcaa72640 1 -- 172.21.15.107:0/225437711 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40af9b0 msgr2=0x7febc40c6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febcaa72640 1 --2- 172.21.15.107:0/225437711 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40af9b0 0x7febc40c6c90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7febb8009870 tx=0x7febb801c480 comp rx=0 tx=0).stop 2023-10-25T00:22:11.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febcaa72640 1 -- 172.21.15.107:0/225437711 shutdown_connections 2023-10-25T00:22:11.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febcaa72640 1 --2- 172.21.15.107:0/225437711 >> v2:172.21.15.110:3300/0 conn(0x7febc406b360 0x7febc4069d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febcaa72640 1 --2- 172.21.15.107:0/225437711 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40af9b0 0x7febc40c6c90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febcaa72640 1 -- 172.21.15.107:0/225437711 >> 172.21.15.107:0/225437711 conn(0x7febc4071b00 msgr2=0x7febc40abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:11.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febcaa72640 1 -- 172.21.15.107:0/225437711 shutdown_connections 2023-10-25T00:22:11.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febcaa72640 1 -- 172.21.15.107:0/225437711 wait complete. 2023-10-25T00:22:11.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febcaa72640 1 Processor -- start 2023-10-25T00:22:11.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.664+0000 7febcaa72640 1 -- start start 2023-10-25T00:22:11.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febcaa72640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc406b360 0x7febc40c16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febcaa72640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febc40af9b0 0x7febc4153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febcaa72640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40bd930 0x7febc414c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febcaa72640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc41edbd0 con 0x7febc40bd930 2023-10-25T00:22:11.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febcaa72640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc41f0690 con 0x7febc40af9b0 2023-10-25T00:22:11.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febcaa72640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc41edff0 con 0x7febc406b360 2023-10-25T00:22:11.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc900f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40bd930 0x7febc414c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc900f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40bd930 0x7febc414c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60456/0 (socket says 172.21.15.107:60456) 2023-10-25T00:22:11.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc900f640 1 -- 172.21.15.107:0/1024433842 learned_addr learned my addr 172.21.15.107:0/1024433842 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:11.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc3fff640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febc40af9b0 0x7febc4153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc880e640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc406b360 0x7febc40c16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc900f640 1 -- 172.21.15.107:0/1024433842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc406b360 msgr2=0x7febc40c16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc900f640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc406b360 0x7febc40c16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc900f640 1 -- 172.21.15.107:0/1024433842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febc40af9b0 msgr2=0x7febc4153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc900f640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febc40af9b0 0x7febc4153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc900f640 1 -- 172.21.15.107:0/1024433842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb80094d0 con 0x7febc40bd930 2023-10-25T00:22:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc900f640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40bd930 0x7febc414c490 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7febbc00c730 tx=0x7febbc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc17fa640 1 -- 172.21.15.107:0/1024433842 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc006d60 con 0x7febc40bd930 2023-10-25T00:22:11.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc17fa640 1 -- 172.21.15.107:0/1024433842 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc003980 con 0x7febc40bd930 2023-10-25T00:22:11.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febcaa72640 1 -- 172.21.15.107:0/1024433842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc41ed230 con 0x7febc40bd930 2023-10-25T00:22:11.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc17fa640 1 -- 172.21.15.107:0/1024433842 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc003d30 con 0x7febc40bd930 2023-10-25T00:22:11.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febcaa72640 1 -- 172.21.15.107:0/1024433842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc41ed7f0 con 0x7febc40bd930 2023-10-25T00:22:11.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febcaa72640 1 -- 172.21.15.107:0/1024433842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc41edd10 con 0x7febc40bd930 2023-10-25T00:22:11.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc17fa640 1 -- 172.21.15.107:0/1024433842 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7febbc0052e0 con 0x7febc40bd930 2023-10-25T00:22:11.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc17fa640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feba8065e90 0x7feba8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc17fa640 1 -- 172.21.15.107:0/1024433842 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7febbc084800 con 0x7febc40bd930 2023-10-25T00:22:11.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc880e640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feba8065e90 0x7feba8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.668+0000 7febc880e640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feba8065e90 0x7feba8068350 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7febb80af000 tx=0x7febb80a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.672+0000 7febc17fa640 1 -- 172.21.15.107:0/1024433842 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7febc41edd10 con 0x7febc40bd930 2023-10-25T00:22:11.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 -- 172.21.15.107:0/1024433842 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7febc41edd10 con 0x7febc40bd930 2023-10-25T00:22:11.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febc17fa640 1 -- 172.21.15.107:0/1024433842 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v72) v1 ==== 95+0+11 (secure 0 0 0) 0x7febbc016020 con 0x7febc40bd930 2023-10-25T00:22:11.828 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-25T00:22:11.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 -- 172.21.15.107:0/1024433842 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feba8065e90 msgr2=0x7feba8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feba8065e90 0x7feba8068350 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7febb80af000 tx=0x7febb80a7000 comp rx=0 tx=0).stop 2023-10-25T00:22:11.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 -- 172.21.15.107:0/1024433842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40bd930 msgr2=0x7febc414c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40bd930 0x7febc414c490 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7febbc00c730 tx=0x7febbc00cbf0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 -- 172.21.15.107:0/1024433842 shutdown_connections 2023-10-25T00:22:11.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40bd930 0x7febc414c490 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febc40af9b0 0x7febc4153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7feba8065e90 0x7feba8068350 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 --2- 172.21.15.107:0/1024433842 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc406b360 0x7febc40c16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 -- 172.21.15.107:0/1024433842 >> 172.21.15.107:0/1024433842 conn(0x7febc4071b00 msgr2=0x7febc415f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:11.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 -- 172.21.15.107:0/1024433842 shutdown_connections 2023-10-25T00:22:11.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.828+0000 7febcaa72640 1 -- 172.21.15.107:0/1024433842 wait complete. 2023-10-25T00:22:11.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-25T00:22:11.964 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.964+0000 7f2450367640 1 Processor -- start 2023-10-25T00:22:11.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.964+0000 7f2450367640 1 -- start start 2023-10-25T00:22:11.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.964+0000 7f2450367640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24480bdc80 0x7f24480a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.964+0000 7f2450367640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f24480c1770 0x7f24480c1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.964+0000 7f2450367640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24480abdc0 con 0x7f24480a4f20 2023-10-25T00:22:11.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.964+0000 7f2450367640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24480be0a0 con 0x7f24480c1770 2023-10-25T00:22:11.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.964+0000 7f2450367640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24480a8c30 con 0x7f24480bdc80 2023-10-25T00:22:11.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.964+0000 7f244e904640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f24480c1770 0x7f24480c1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.964+0000 7f244e103640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f24480a4f20 0x7f24480a90c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:58818/0 (socket says 172.21.15.107:58818) 2023-10-25T00:22:11.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.964+0000 7f244e103640 1 -- 172.21.15.107:0/537705141 learned_addr learned my addr 172.21.15.107:0/537705141 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:11.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.964+0000 7f244d902640 1 --2- 172.21.15.107:0/537705141 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24480bdc80 0x7f24480a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f244e904640 1 -- 172.21.15.107:0/537705141 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24480bdc80 msgr2=0x7f24480a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:11.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f244e904640 1 --2- 172.21.15.107:0/537705141 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24480bdc80 0x7f24480a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f244e904640 1 -- 172.21.15.107:0/537705141 >> v1:172.21.15.110:6789/0 conn(0x7f24480a4f20 legacy=0x7f24480a90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f244e904640 1 -- 172.21.15.107:0/537705141 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24480bef60 con 0x7f24480c1770 2023-10-25T00:22:11.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f244e904640 1 --2- 172.21.15.107:0/537705141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24480c1770 0x7f24480c1c00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2440009dc0 tx=0x7f244001c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b88c64472fe85da9 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f244d101640 1 -- 172.21.15.107:0/537705141 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24400a5940 con 0x7f24480c1770 2023-10-25T00:22:11.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f244d101640 1 -- 172.21.15.107:0/537705141 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24400a5aa0 con 0x7f24480c1770 2023-10-25T00:22:11.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f244d101640 1 -- 172.21.15.107:0/537705141 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24400a5e30 con 0x7f24480c1770 2023-10-25T00:22:11.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f2450367640 1 -- 172.21.15.107:0/537705141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24480c1770 msgr2=0x7f24480c1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f2450367640 1 --2- 172.21.15.107:0/537705141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24480c1770 0x7f24480c1c00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2440009dc0 tx=0x7f244001c900 comp rx=0 tx=0).stop 2023-10-25T00:22:11.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f2450367640 1 -- 172.21.15.107:0/537705141 shutdown_connections 2023-10-25T00:22:11.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f2450367640 1 --2- 172.21.15.107:0/537705141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24480c1770 0x7f24480c1c00 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f2450367640 1 --2- 172.21.15.107:0/537705141 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24480bdc80 0x7f24480a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f2450367640 1 -- 172.21.15.107:0/537705141 >> 172.21.15.107:0/537705141 conn(0x7f24480c2860 msgr2=0x7f24480c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:11.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f2450367640 1 -- 172.21.15.107:0/537705141 shutdown_connections 2023-10-25T00:22:11.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.968+0000 7f2450367640 1 -- 172.21.15.107:0/537705141 wait complete. 2023-10-25T00:22:11.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f2450367640 1 Processor -- start 2023-10-25T00:22:11.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f2450367640 1 -- start start 2023-10-25T00:22:11.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f2450367640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24480a4f20 0x7f24480bc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f2450367640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24480bdc80 0x7f24481532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f2450367640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24480c1770 0x7f244814e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f2450367640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2448154700 con 0x7f24480c1770 2023-10-25T00:22:11.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f2450367640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2448154f50 con 0x7f24480a4f20 2023-10-25T00:22:11.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f2450367640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244815a3c0 con 0x7f24480bdc80 2023-10-25T00:22:11.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f244e904640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24480c1770 0x7f244814e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f244e904640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24480c1770 0x7f244814e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60464/0 (socket says 172.21.15.107:60464) 2023-10-25T00:22:11.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f244e904640 1 -- 172.21.15.107:0/1669880802 learned_addr learned my addr 172.21.15.107:0/1669880802 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:11.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f244d902640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24480bdc80 0x7f24481532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f244e103640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24480a4f20 0x7f24480bc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f244e904640 1 -- 172.21.15.107:0/1669880802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24480bdc80 msgr2=0x7f24481532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f244e904640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24480bdc80 0x7f24481532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f244e904640 1 -- 172.21.15.107:0/1669880802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24480a4f20 msgr2=0x7f24480bc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f244e904640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24480a4f20 0x7f24480bc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f244e904640 1 -- 172.21.15.107:0/1669880802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2440009a70 con 0x7f24480c1770 2023-10-25T00:22:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f244e904640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24480c1770 0x7f244814e990 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f244001a530 tx=0x7f24400196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f243effd640 1 -- 172.21.15.107:0/1669880802 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2440007000 con 0x7f24480c1770 2023-10-25T00:22:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f243effd640 1 -- 172.21.15.107:0/1669880802 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24400aede0 con 0x7f24480c1770 2023-10-25T00:22:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f2450367640 1 -- 172.21.15.107:0/1669880802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2448155460 con 0x7f24480c1770 2023-10-25T00:22:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f243effd640 1 -- 172.21.15.107:0/1669880802 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24400b7650 con 0x7f24480c1770 2023-10-25T00:22:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.972+0000 7f2450367640 1 -- 172.21.15.107:0/1669880802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2448155180 con 0x7f24480c1770 2023-10-25T00:22:11.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.976+0000 7f2450367640 1 -- 172.21.15.107:0/1669880802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244815a500 con 0x7f24480c1770 2023-10-25T00:22:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.976+0000 7f243effd640 1 -- 172.21.15.107:0/1669880802 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f24400b4050 con 0x7f24480c1770 2023-10-25T00:22:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.976+0000 7f243effd640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f242c065e60 0x7f242c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.976+0000 7f243effd640 1 -- 172.21.15.107:0/1669880802 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f244011fd50 con 0x7f24480c1770 2023-10-25T00:22:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.976+0000 7f244e103640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f242c065e60 0x7f242c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:11.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.976+0000 7f244e103640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f242c065e60 0x7f242c068320 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f24480a6e50 tx=0x7f2438007e90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:11.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:11.980+0000 7f243effd640 1 -- 172.21.15.107:0/1669880802 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2440123030 con 0x7f24480c1770 2023-10-25T00:22:12.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.144+0000 7f2450367640 1 -- 172.21.15.107:0/1669880802 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f244804e130 con 0x7f24480c1770 2023-10-25T00:22:12.145 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.144+0000 7f243effd640 1 -- 172.21.15.107:0/1669880802 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 88 entries v72) v1 ==== 81+0+5426 (secure 0 0 0) 0x7f24400c7020 con 0x7f24480c1770 2023-10-25T00:22:12.145 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2187059523 2023-10-26T00:21:39.173084+0000 2023-10-25T00:22:12.146 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/657626630 2023-10-26T00:21:19.167088+0000 2023-10-25T00:22:12.146 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/3771564389 2023-10-26T00:20:19.149072+0000 2023-10-25T00:22:12.146 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/3771564389 2023-10-26T00:20:19.149072+0000 2023-10-25T00:22:12.146 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2970408860 2023-10-26T00:19:04.122659+0000 2023-10-25T00:22:12.147 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/418589140 2023-10-26T00:18:49.118366+0000 2023-10-25T00:22:12.147 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/985135365 2023-10-26T00:17:39.037219+0000 2023-10-25T00:22:12.147 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/985135365 2023-10-26T00:17:39.037219+0000 2023-10-25T00:22:12.147 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/208713198 2023-10-26T00:17:24.032902+0000 2023-10-25T00:22:12.147 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/208713198 2023-10-26T00:17:24.032902+0000 2023-10-25T00:22:12.147 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/1619327197 2023-10-26T00:15:22.673615+0000 2023-10-25T00:22:12.148 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/175581297 2023-10-26T00:15:18.921240+0000 2023-10-25T00:22:12.148 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/418589140 2023-10-26T00:18:49.118366+0000 2023-10-25T00:22:12.148 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2856791739 2023-10-26T00:13:48.893880+0000 2023-10-25T00:22:12.148 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/898258574 2023-10-26T00:13:18.883076+0000 2023-10-25T00:22:12.148 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3396644475 2023-10-26T00:19:44.136579+0000 2023-10-25T00:22:12.149 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/3370462806 2023-10-26T00:12:33.862408+0000 2023-10-25T00:22:12.149 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/1101018338 2023-10-25T23:59:48.457105+0000 2023-10-25T00:22:12.149 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/838886697 2023-10-25T23:56:18.381650+0000 2023-10-25T00:22:12.149 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1381726420 2023-10-26T00:01:28.521014+0000 2023-10-25T00:22:12.149 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/1619327197 2023-10-26T00:15:22.673615+0000 2023-10-25T00:22:12.149 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1381726420 2023-10-26T00:01:28.521014+0000 2023-10-25T00:22:12.150 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/3103569740 2023-10-26T00:01:58.532147+0000 2023-10-25T00:22:12.150 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/780830443 2023-10-26T00:06:23.634894+0000 2023-10-25T00:22:12.150 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/1101018338 2023-10-25T23:59:48.457105+0000 2023-10-25T00:22:12.150 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1555840329 2023-10-26T00:05:18.610829+0000 2023-10-25T00:22:12.150 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/3836115666 2023-10-26T00:08:58.700067+0000 2023-10-25T00:22:12.151 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/898258574 2023-10-26T00:13:18.883076+0000 2023-10-25T00:22:12.151 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1508236506 2023-10-25T23:58:58.439244+0000 2023-10-25T00:22:12.151 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2348043270 2023-10-25T23:58:53.436687+0000 2023-10-25T00:22:12.151 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/204494054 2023-10-25T23:58:03.418411+0000 2023-10-25T00:22:12.151 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1555840329 2023-10-26T00:05:18.610829+0000 2023-10-25T00:22:12.152 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/3721101226 2023-10-25T23:58:03.418295+0000 2023-10-25T00:22:12.152 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2480841919 2023-10-25T23:55:58.372958+0000 2023-10-25T00:22:12.152 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/2577776351 2023-10-26T00:04:34.728765+0000 2023-10-25T00:22:12.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1246928633 2023-10-26T00:05:13.609231+0000 2023-10-25T00:22:12.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/913067382 2023-10-26T00:08:08.675592+0000 2023-10-25T00:22:12.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/1031240534 2023-10-25T23:57:23.404313+0000 2023-10-25T00:22:12.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/727042497 2023-10-25T23:54:13.340839+0000 2023-10-25T00:22:12.153 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2970408860 2023-10-26T00:19:04.122659+0000 2023-10-25T00:22:12.154 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2480841919 2023-10-25T23:55:58.372958+0000 2023-10-25T00:22:12.154 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/201284779 2023-10-26T00:02:23.541454+0000 2023-10-25T00:22:12.154 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/201284779 2023-10-26T00:02:23.541454+0000 2023-10-25T00:22:12.154 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/838886697 2023-10-25T23:56:18.381650+0000 2023-10-25T00:22:12.154 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1292188015 2023-10-26T00:00:48.479959+0000 2023-10-25T00:22:12.155 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1317796403 2023-10-26T00:08:23.682615+0000 2023-10-25T00:22:12.155 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1220982369 2023-10-25T23:55:23.361145+0000 2023-10-25T00:22:12.155 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2348043270 2023-10-25T23:58:53.436687+0000 2023-10-25T00:22:12.155 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/3370462806 2023-10-26T00:12:33.862408+0000 2023-10-25T00:22:12.155 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1220982369 2023-10-25T23:55:23.361145+0000 2023-10-25T00:22:12.156 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/204494054 2023-10-25T23:58:03.418411+0000 2023-10-25T00:22:12.156 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/175581297 2023-10-26T00:15:18.921240+0000 2023-10-25T00:22:12.156 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2856791739 2023-10-26T00:13:48.893880+0000 2023-10-25T00:22:12.156 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/1031240534 2023-10-25T23:57:23.404313+0000 2023-10-25T00:22:12.156 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/326715537 2023-10-26T00:11:03.824824+0000 2023-10-25T00:22:12.156 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1004926771 2023-10-25T23:57:28.406683+0000 2023-10-25T00:22:12.157 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/657626630 2023-10-26T00:21:19.167088+0000 2023-10-25T00:22:12.157 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1004926771 2023-10-25T23:57:28.406683+0000 2023-10-25T00:22:12.157 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/3721101226 2023-10-25T23:58:03.418295+0000 2023-10-25T00:22:12.157 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3474118649 2023-10-26T00:11:18.831318+0000 2023-10-25T00:22:12.157 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3232233949 2023-10-26T00:03:28.568587+0000 2023-10-25T00:22:12.158 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/2577776351 2023-10-26T00:04:34.728765+0000 2023-10-25T00:22:12.158 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/326715537 2023-10-26T00:11:03.824824+0000 2023-10-25T00:22:12.158 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2276136116 2023-10-26T00:09:18.706706+0000 2023-10-25T00:22:12.158 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/319918260 2023-10-26T00:03:53.579302+0000 2023-10-25T00:22:12.158 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3474118649 2023-10-26T00:11:18.831318+0000 2023-10-25T00:22:12.158 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1292188015 2023-10-26T00:00:48.479959+0000 2023-10-25T00:22:12.159 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/174863522 2023-10-26T00:05:53.622773+0000 2023-10-25T00:22:12.159 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/319918260 2023-10-26T00:03:53.579302+0000 2023-10-25T00:22:12.159 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/3103569740 2023-10-26T00:01:58.532147+0000 2023-10-25T00:22:12.159 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2150827048 2023-10-26T00:07:48.667043+0000 2023-10-25T00:22:12.159 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/4180698155 2023-10-26T00:04:28.591981+0000 2023-10-25T00:22:12.160 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3396644475 2023-10-26T00:19:44.136579+0000 2023-10-25T00:22:12.160 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/4180698155 2023-10-26T00:04:28.591981+0000 2023-10-25T00:22:12.160 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2187059523 2023-10-26T00:21:39.173084+0000 2023-10-25T00:22:12.160 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1246928633 2023-10-26T00:05:13.609231+0000 2023-10-25T00:22:12.160 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/174863522 2023-10-26T00:05:53.622773+0000 2023-10-25T00:22:12.160 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1508236506 2023-10-25T23:58:58.439244+0000 2023-10-25T00:22:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/780830443 2023-10-26T00:06:23.634894+0000 2023-10-25T00:22:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2150827048 2023-10-26T00:07:48.667043+0000 2023-10-25T00:22:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/913067382 2023-10-26T00:08:08.675592+0000 2023-10-25T00:22:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1317796403 2023-10-26T00:08:23.682615+0000 2023-10-25T00:22:12.161 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2276136116 2023-10-26T00:09:18.706706+0000 2023-10-25T00:22:12.162 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3232233949 2023-10-26T00:03:28.568587+0000 2023-10-25T00:22:12.162 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/727042497 2023-10-25T23:54:13.340839+0000 2023-10-25T00:22:12.162 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1788268787 2023-10-26T00:10:38.784519+0000 2023-10-25T00:22:12.162 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/3836115666 2023-10-26T00:08:58.700067+0000 2023-10-25T00:22:12.163 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1788268787 2023-10-26T00:10:38.784519+0000 2023-10-25T00:22:12.164 INFO:teuthology.orchestra.run.smithi107.stderr:listed 88 entries 2023-10-25T00:22:12.164 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 -- 172.21.15.107:0/1669880802 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f242c065e60 msgr2=0x7f242c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:12.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f242c065e60 0x7f242c068320 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f24480a6e50 tx=0x7f2438007e90 comp rx=0 tx=0).stop 2023-10-25T00:22:12.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 -- 172.21.15.107:0/1669880802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24480c1770 msgr2=0x7f244814e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:12.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24480c1770 0x7f244814e990 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f244001a530 tx=0x7f24400196a0 comp rx=0 tx=0).stop 2023-10-25T00:22:12.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 -- 172.21.15.107:0/1669880802 shutdown_connections 2023-10-25T00:22:12.165 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24480c1770 0x7f244814e990 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:12.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f242c065e60 0x7f242c068320 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:12.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f24480bdc80 0x7f24481532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:12.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 --2- 172.21.15.107:0/1669880802 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f24480a4f20 0x7f24480bc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:12.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 -- 172.21.15.107:0/1669880802 >> 172.21.15.107:0/1669880802 conn(0x7f24480c2860 msgr2=0x7f24480a6300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:12.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 -- 172.21.15.107:0/1669880802 shutdown_connections 2023-10-25T00:22:12.167 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:12.148+0000 7f2450367640 1 -- 172.21.15.107:0/1669880802 wait complete. 2023-10-25T00:22:12.167 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi107.front.sepia.ceph.com... 2023-10-25T00:22:12.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-25T00:22:12.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-10-25T00:22:12.255 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:22:12.258 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi107.stderr:ceph-fuse[46369]: fuse finished with error 0 and tester_r 0 2023-10-25T00:22:18.357 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2023-10-25T00:22:18.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-25T00:22:18.358 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-25T00:22:18.388 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-25T00:22:18.388 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-25T00:22:18.470 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-25T00:22:18.470 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-25T00:22:18.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-25T00:22:18.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.652+0000 7f1c91f07640 1 Processor -- start 2023-10-25T00:22:18.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.652+0000 7f1c91f07640 1 -- start start 2023-10-25T00:22:18.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c91f07640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1c8c152040 0x7f1c8c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c91f07640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c155680 con 0x7f1c8c0c8100 2023-10-25T00:22:18.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c91f07640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c154a70 con 0x7f1c8c06a4b0 2023-10-25T00:22:18.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c91f07640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8c154d50 con 0x7f1c8c152040 2023-10-25T00:22:18.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c83fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1c8c0c8100 0x7f1c8c0abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60696/0 (socket says 172.21.15.107:60696) 2023-10-25T00:22:18.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c83fff640 1 -- 172.21.15.107:0/974714119 learned_addr learned my addr 172.21.15.107:0/974714119 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:18.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/974714119 >> v2:172.21.15.110:3300/0 conn(0x7f1c8c152040 0x7f1c8c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94321898 0 0) 0x7f1c8c155680 con 0x7f1c8c0c8100 2023-10-25T00:22:18.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c78003530 con 0x7f1c8c0c8100 2023-10-25T00:22:18.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1312926245 0 0) 0x7f1c78003530 con 0x7f1c8c0c8100 2023-10-25T00:22:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 >> v2:172.21.15.110:3300/0 conn(0x7f1c8c152040 msgr2=0x7f1c8c0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 --2- 172.21.15.107:0/974714119 >> v2:172.21.15.110:3300/0 conn(0x7f1c8c152040 0x7f1c8c0c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 >> v1:172.21.15.110:6789/0 conn(0x7f1c8c06a4b0 legacy=0x7f1c8c0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c155030 con 0x7f1c8c0c8100 2023-10-25T00:22:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f1c74001680 con 0x7f1c8c0c8100 2023-10-25T00:22:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 >> v1:172.21.15.107:6789/0 conn(0x7f1c8c0c8100 legacy=0x7f1c8c0abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:18.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c78003980 0x7f1c78003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c780047d0 0x7f1c78004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c780052b0 0x7f1c7800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c154d50 con 0x7f1c780052b0 2023-10-25T00:22:18.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7800c180 con 0x7f1c780047d0 2023-10-25T00:22:18.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7800c340 con 0x7f1c78003980 2023-10-25T00:22:18.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c780052b0 0x7f1c7800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c83fff640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c780047d0 0x7f1c78004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c78003980 0x7f1c78003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8bfff640 1 -- 172.21.15.107:0/974714119 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c78003980 msgr2=0x7f1c78003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:18.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c78003980 0x7f1c78003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8bfff640 1 -- 172.21.15.107:0/974714119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c780047d0 msgr2=0x7f1c78004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:18.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c780047d0 0x7f1c78004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c8bfff640 1 -- 172.21.15.107:0/974714119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c78003530 con 0x7f1c780052b0 2023-10-25T00:22:18.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.656+0000 7f1c83fff640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c780047d0 0x7f1c78004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:22:18.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8bfff640 1 -- 172.21.15.107:0/974714119 reap_dead start 2023-10-25T00:22:18.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c780052b0 0x7f1c7800b990 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f1c84001e40 tx=0x7f1c8401c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=767f6cb475778b78 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:18.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84003b80 con 0x7f1c780052b0 2023-10-25T00:22:18.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c84003ce0 con 0x7f1c780052b0 2023-10-25T00:22:18.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8affd640 1 -- 172.21.15.107:0/974714119 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84005310 con 0x7f1c780052b0 2023-10-25T00:22:18.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 -- 172.21.15.107:0/974714119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c780052b0 msgr2=0x7f1c7800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:18.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c780052b0 0x7f1c7800b990 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f1c84001e40 tx=0x7f1c8401c740 comp rx=0 tx=0).stop 2023-10-25T00:22:18.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 -- 172.21.15.107:0/974714119 shutdown_connections 2023-10-25T00:22:18.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 --2- 172.21.15.107:0/974714119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c780052b0 0x7f1c7800b990 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 -- 172.21.15.107:0/974714119 >> 172.21.15.107:0/974714119 conn(0x7f1c8c0ca200 msgr2=0x7f1c8c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:18.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 -- 172.21.15.107:0/974714119 shutdown_connections 2023-10-25T00:22:18.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 -- 172.21.15.107:0/974714119 wait complete. 2023-10-25T00:22:18.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 Processor -- start 2023-10-25T00:22:18.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 -- start start 2023-10-25T00:22:18.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c780052b0 0x7f1c8c0c37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c146420 0x7f1c8c144bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c0c95c0 0x7f1c8c1efa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c144170 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c1469b0 con 0x7f1c8c146420 2023-10-25T00:22:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c91f07640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c146c90 con 0x7f1c780052b0 2023-10-25T00:22:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c0c95c0 0x7f1c8c1efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c0c95c0 0x7f1c8c1efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51572/0 (socket says 172.21.15.107:51572) 2023-10-25T00:22:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8bfff640 1 -- 172.21.15.107:0/813874272 learned_addr learned my addr 172.21.15.107:0/813874272 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c780052b0 0x7f1c8c0c37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c83fff640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c146420 0x7f1c8c144bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8bfff640 1 -- 172.21.15.107:0/813874272 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c780052b0 msgr2=0x7f1c8c0c37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:18.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c780052b0 0x7f1c8c0c37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8bfff640 1 -- 172.21.15.107:0/813874272 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c146420 msgr2=0x7f1c8c144bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:18.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c146420 0x7f1c8c144bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.660+0000 7f1c8bfff640 1 -- 172.21.15.107:0/813874272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c1443c0 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c8bfff640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c0c95c0 0x7f1c8c1efa10 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f1c8401cfe0 tx=0x7f1c78005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:18.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c88ff9640 1 -- 172.21.15.107:0/813874272 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84001260 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c88ff9640 1 -- 172.21.15.107:0/813874272 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c84003980 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c91f07640 1 -- 172.21.15.107:0/813874272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8c1446a0 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c88ff9640 1 -- 172.21.15.107:0/813874272 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c840b4690 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c91f07640 1 -- 172.21.15.107:0/813874272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c0ae2b0 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c91f07640 1 -- 172.21.15.107:0/813874272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8c0c9c10 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c88ff9640 1 -- 172.21.15.107:0/813874272 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1c840b33d0 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c88ff9640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c6c065e90 0x7f1c6c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c88ff9640 1 -- 172.21.15.107:0/813874272 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f1c840b0070 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c6c065e90 0x7f1c6c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.664+0000 7f1c8b7fe640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c6c065e90 0x7f1c6c068350 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0bd860 tx=0x7f1c7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:18.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.668+0000 7f1c88ff9640 1 -- 172.21.15.107:0/813874272 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1c840e9e00 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.823 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.824+0000 7f1c91f07640 1 -- 172.21.15.107:0/813874272 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c8c1448b0 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.824+0000 7f1c88ff9640 1 -- 172.21.15.107:0/813874272 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+27395 (secure 0 0 0) 0x7f1c840b9090 con 0x7f1c8c0c95c0 2023-10-25T00:22:18.827 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:18.827 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":72,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-25T00:21:39.173159+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2f","mappings":[{"from":3,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":2}]},{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6835/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6837/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.110:6834/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.110:6832/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.107:6835/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.107:6837/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.110:6834/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.110:6832/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.107:6835/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.107:6834/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.110:6833/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6836/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6836/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.107:6834/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6835/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6836/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.110:6834/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6835/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.110:6835/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6834/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.107:6837/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.107:6837/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6832/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6833/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.110:6833/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6834/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.110:6835/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6836/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6836/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6834/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.110:6835/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.107:6834/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.107:6834/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.110:6833/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.107:6837/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6834/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.110:6833/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.110:6832/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.107:6836/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6837/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.107:6837/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.110:6833/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6837/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.107:6835/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.107:6837/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6837/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6835/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.110:6835/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.107:6836/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.107:6834/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.110:6832/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.107:6835/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6835/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.110:6832/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.110:6833/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6836/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6834/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6836/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.110:6834/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6837/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6836/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.110:6835/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6834/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6835/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6835/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.110:6832/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6835/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.107:6836/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6833/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6835/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.107:6836/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6834/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.107:6836/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6836/1788268787":"2023-10-26T00:10:38.784519+0000","172.21.15.110:6835/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.107:6837/1788268787":"2023-10-26T00:10:38.784519+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-25T00:22:18.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 -- 172.21.15.107:0/813874272 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c6c065e90 msgr2=0x7f1c6c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c6c065e90 0x7f1c6c068350 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0bd860 tx=0x7f1c7c008040 comp rx=0 tx=0).stop 2023-10-25T00:22:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 -- 172.21.15.107:0/813874272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c0c95c0 msgr2=0x7f1c8c1efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c0c95c0 0x7f1c8c1efa10 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f1c8401cfe0 tx=0x7f1c78005800 comp rx=0 tx=0).stop 2023-10-25T00:22:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 -- 172.21.15.107:0/813874272 shutdown_connections 2023-10-25T00:22:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c8c0c95c0 0x7f1c8c1efa10 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c8c146420 0x7f1c8c144bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c6c065e90 0x7f1c6c068350 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 --2- 172.21.15.107:0/813874272 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c780052b0 0x7f1c8c0c37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 -- 172.21.15.107:0/813874272 >> 172.21.15.107:0/813874272 conn(0x7f1c8c0ca200 msgr2=0x7f1c8c0aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:18.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 -- 172.21.15.107:0/813874272 shutdown_connections 2023-10-25T00:22:18.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.828+0000 7f1c91f07640 1 -- 172.21.15.107:0/813874272 wait complete. 2023-10-25T00:22:18.846 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-10-24T23:53:09.786122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-10-24T23:53:10.766645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-25T00:22:18.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-25T00:22:18.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.968+0000 7fc08dba3640 1 Processor -- start 2023-10-25T00:22:18.969 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.968+0000 7fc08dba3640 1 -- start start 2023-10-25T00:22:18.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.968+0000 7fc08dba3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc088152120 0x7fc088158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.968+0000 7fc08dba3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc0880c8350 0x7fc088155300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.968+0000 7fc08dba3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc088155840 0x7fc0880c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.971 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.968+0000 7fc08dba3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc088158650 con 0x7fc088155840 2023-10-25T00:22:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.968+0000 7fc08dba3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc088154b20 con 0x7fc0880c8350 2023-10-25T00:22:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.968+0000 7fc08dba3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881588e0 con 0x7fc088152120 2023-10-25T00:22:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc087fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc088155840 0x7fc0880c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc087fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fc088155840 0x7fc0880c6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51576/0 (socket says 172.21.15.107:51576) 2023-10-25T00:22:18.972 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc087fff640 1 -- 172.21.15.107:0/24316591 learned_addr learned my addr 172.21.15.107:0/24316591 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc0877fe640 1 --2- 172.21.15.107:0/24316591 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc088152120 0x7fc088158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc07effd640 1 --2- 172.21.15.107:0/24316591 >> v2:172.21.15.110:3300/0 conn(0x7fc0880c8350 0x7fc088155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc087fff640 1 -- 172.21.15.107:0/24316591 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc088152120 msgr2=0x7fc088158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc087fff640 1 --2- 172.21.15.107:0/24316591 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc088152120 0x7fc088158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc087fff640 1 -- 172.21.15.107:0/24316591 >> v2:172.21.15.110:3300/0 conn(0x7fc0880c8350 msgr2=0x7fc088155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:18.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc087fff640 1 --2- 172.21.15.107:0/24316591 >> v2:172.21.15.110:3300/0 conn(0x7fc0880c8350 0x7fc088155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc087fff640 1 -- 172.21.15.107:0/24316591 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc088158bc0 con 0x7fc088155840 2023-10-25T00:22:18.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc07effd640 1 --2- 172.21.15.107:0/24316591 >> v2:172.21.15.110:3300/0 conn(0x7fc0880c8350 0x7fc088155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:22:18.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc0877fe640 1 --2- 172.21.15.107:0/24316591 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc088152120 0x7fc088158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:22:18.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc087fff640 1 --2- 172.21.15.107:0/24316591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc088155840 0x7fc0880c6c40 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fc088073b80 tx=0x7fc08001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6102d2f0fcc3f86b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:18.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc086ffd640 1 -- 172.21.15.107:0/24316591 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800a56b0 con 0x7fc088155840 2023-10-25T00:22:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc086ffd640 1 -- 172.21.15.107:0/24316591 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0800a5810 con 0x7fc088155840 2023-10-25T00:22:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc086ffd640 1 -- 172.21.15.107:0/24316591 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800a5ba0 con 0x7fc088155840 2023-10-25T00:22:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc08dba3640 1 -- 172.21.15.107:0/24316591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc088155840 msgr2=0x7fc0880c6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc08dba3640 1 --2- 172.21.15.107:0/24316591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc088155840 0x7fc0880c6c40 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fc088073b80 tx=0x7fc08001c940 comp rx=0 tx=0).stop 2023-10-25T00:22:18.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc08dba3640 1 -- 172.21.15.107:0/24316591 shutdown_connections 2023-10-25T00:22:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc08dba3640 1 --2- 172.21.15.107:0/24316591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc088155840 0x7fc0880c6c40 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc08dba3640 1 --2- 172.21.15.107:0/24316591 >> v2:172.21.15.110:3300/0 conn(0x7fc0880c8350 0x7fc088155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc08dba3640 1 --2- 172.21.15.107:0/24316591 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc088152120 0x7fc088158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc08dba3640 1 -- 172.21.15.107:0/24316591 >> 172.21.15.107:0/24316591 conn(0x7fc0880abdc0 msgr2=0x7fc0880ac1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:18.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc08dba3640 1 -- 172.21.15.107:0/24316591 shutdown_connections 2023-10-25T00:22:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.972+0000 7fc08dba3640 1 -- 172.21.15.107:0/24316591 wait complete. 2023-10-25T00:22:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc08dba3640 1 Processor -- start 2023-10-25T00:22:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc08dba3640 1 -- start start 2023-10-25T00:22:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc08dba3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0880c8350 0x7fc0881f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc08dba3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc088152120 0x7fc088157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc08dba3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc088157130 0x7fc0881ed770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc08dba3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0880be470 con 0x7fc088157130 2023-10-25T00:22:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc08dba3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881f13d0 con 0x7fc088152120 2023-10-25T00:22:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc08dba3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881580b0 con 0x7fc0880c8350 2023-10-25T00:22:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc087fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc088157130 0x7fc0881ed770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc087fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc088157130 0x7fc0881ed770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51588/0 (socket says 172.21.15.107:51588) 2023-10-25T00:22:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc087fff640 1 -- 172.21.15.107:0/2241130932 learned_addr learned my addr 172.21.15.107:0/2241130932 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc0877fe640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0880c8350 0x7fc0881f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc087fff640 1 -- 172.21.15.107:0/2241130932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0880c8350 msgr2=0x7fc0881f1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc087fff640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0880c8350 0x7fc0881f1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc087fff640 1 -- 172.21.15.107:0/2241130932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc088152120 msgr2=0x7fc088157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc07effd640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc088152120 0x7fc088157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc087fff640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc088152120 0x7fc088157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc087fff640 1 -- 172.21.15.107:0/2241130932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc080009b30 con 0x7fc088157130 2023-10-25T00:22:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc0877fe640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0880c8350 0x7fc0881f1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:18.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc087fff640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc088157130 0x7fc0881ed770 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fc0800af040 tx=0x7fc0800196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:18.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc084ff9640 1 -- 172.21.15.107:0/2241130932 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800b8670 con 0x7fc088157130 2023-10-25T00:22:18.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc084ff9640 1 -- 172.21.15.107:0/2241130932 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0800b8810 con 0x7fc088157130 2023-10-25T00:22:18.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc08dba3640 1 -- 172.21.15.107:0/2241130932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0881f0d80 con 0x7fc088157130 2023-10-25T00:22:18.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc084ff9640 1 -- 172.21.15.107:0/2241130932 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800b8bc0 con 0x7fc088157130 2023-10-25T00:22:18.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc08dba3640 1 -- 172.21.15.107:0/2241130932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc088158390 con 0x7fc088157130 2023-10-25T00:22:18.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.976+0000 7fc08dba3640 1 -- 172.21.15.107:0/2241130932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc088073dc0 con 0x7fc088157130 2023-10-25T00:22:18.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.980+0000 7fc084ff9640 1 -- 172.21.15.107:0/2241130932 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc0800b8d20 con 0x7fc088157130 2023-10-25T00:22:18.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.980+0000 7fc084ff9640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc068065e60 0x7fc068068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:18.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.980+0000 7fc084ff9640 1 -- 172.21.15.107:0/2241130932 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fc08011fb70 con 0x7fc088157130 2023-10-25T00:22:18.986 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.980+0000 7fc0877fe640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc068065e60 0x7fc068068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:18.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.980+0000 7fc0877fe640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc068065e60 0x7fc068068320 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc0880bd220 tx=0x7fc078008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:18.987 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:18.984+0000 7fc084ff9640 1 -- 172.21.15.107:0/2241130932 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc0800ec0d0 con 0x7fc088157130 2023-10-25T00:22:19.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.136+0000 7fc08dba3640 1 -- 172.21.15.107:0/2241130932 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0881f1510 con 0x7fc088157130 2023-10-25T00:22:19.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc084ff9640 1 -- 172.21.15.107:0/2241130932 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc080009510 con 0x7fc088157130 2023-10-25T00:22:19.139 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-25T00:22:19.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 -- 172.21.15.107:0/2241130932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc068065e60 msgr2=0x7fc068068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc068065e60 0x7fc068068320 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fc0880bd220 tx=0x7fc078008040 comp rx=0 tx=0).stop 2023-10-25T00:22:19.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 -- 172.21.15.107:0/2241130932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc088157130 msgr2=0x7fc0881ed770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc088157130 0x7fc0881ed770 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fc0800af040 tx=0x7fc0800196e0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 -- 172.21.15.107:0/2241130932 shutdown_connections 2023-10-25T00:22:19.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc088157130 0x7fc0881ed770 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc088152120 0x7fc088157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc068065e60 0x7fc068068320 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 --2- 172.21.15.107:0/2241130932 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc0880c8350 0x7fc0881f1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 -- 172.21.15.107:0/2241130932 >> 172.21.15.107:0/2241130932 conn(0x7fc0880abdc0 msgr2=0x7fc088150ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 -- 172.21.15.107:0/2241130932 shutdown_connections 2023-10-25T00:22:19.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.140+0000 7fc08dba3640 1 -- 172.21.15.107:0/2241130932 wait complete. 2023-10-25T00:22:19.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-25T00:22:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b97911640 1 Processor -- start 2023-10-25T00:22:19.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b97911640 1 -- start start 2023-10-25T00:22:19.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b97911640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b901526d0 0x7f7b9006adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b97911640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b900ac380 con 0x7f7b9014c6b0 2023-10-25T00:22:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b97911640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b90069d40 con 0x7f7b900ab860 2023-10-25T00:22:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b97911640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b900a9b60 con 0x7f7b901526d0 2023-10-25T00:22:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b94eac640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7b901526d0 0x7f7b9006adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b956ad640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7b9014c6b0 0x7f7b900a9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60698/0 (socket says 172.21.15.107:60698) 2023-10-25T00:22:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b956ad640 1 -- 172.21.15.107:0/185103191 learned_addr learned my addr 172.21.15.107:0/185103191 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b87fff640 1 -- 172.21.15.107:0/185103191 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944179156 0 0) 0x7f7b900ac380 con 0x7f7b9014c6b0 2023-10-25T00:22:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b87fff640 1 -- 172.21.15.107:0/185103191 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b7c003530 con 0x7f7b9014c6b0 2023-10-25T00:22:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b94eac640 1 -- 172.21.15.107:0/185103191 >> v1:172.21.15.110:6789/0 conn(0x7f7b900ab860 legacy=0x7f7b900abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:19.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b94eac640 1 -- 172.21.15.107:0/185103191 >> v1:172.21.15.107:6789/0 conn(0x7f7b9014c6b0 legacy=0x7f7b900a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.284+0000 7f7b94eac640 1 -- 172.21.15.107:0/185103191 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9006a000 con 0x7f7b901526d0 2023-10-25T00:22:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b94eac640 1 --2- 172.21.15.107:0/185103191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b901526d0 0x7f7b9006adc0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f7b800097c0 tx=0x7f7b8001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6222d8b432f86cf0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b87fff640 1 -- 172.21.15.107:0/185103191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b800ac620 con 0x7f7b901526d0 2023-10-25T00:22:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b87fff640 1 -- 172.21.15.107:0/185103191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b800ac780 con 0x7f7b901526d0 2023-10-25T00:22:19.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b87fff640 1 -- 172.21.15.107:0/185103191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b800acaf0 con 0x7f7b901526d0 2023-10-25T00:22:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 -- 172.21.15.107:0/185103191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b901526d0 msgr2=0x7f7b9006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 --2- 172.21.15.107:0/185103191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b901526d0 0x7f7b9006adc0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f7b800097c0 tx=0x7f7b8001c3c0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 -- 172.21.15.107:0/185103191 shutdown_connections 2023-10-25T00:22:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 --2- 172.21.15.107:0/185103191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b901526d0 0x7f7b9006adc0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 -- 172.21.15.107:0/185103191 >> 172.21.15.107:0/185103191 conn(0x7f7b900c4e30 msgr2=0x7f7b900c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 -- 172.21.15.107:0/185103191 shutdown_connections 2023-10-25T00:22:19.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 -- 172.21.15.107:0/185103191 wait complete. 2023-10-25T00:22:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 Processor -- start 2023-10-25T00:22:19.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 -- start start 2023-10-25T00:22:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b900ab860 0x7f7b9013f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b9014c6b0 0x7f7b901ea5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b901526d0 0x7f7b901eab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9013ca30 con 0x7f7b900ab860 2023-10-25T00:22:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9013c750 con 0x7f7b9014c6b0 2023-10-25T00:22:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b97911640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b901edc80 con 0x7f7b901526d0 2023-10-25T00:22:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b956ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b900ab860 0x7f7b9013f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b956ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b900ab860 0x7f7b9013f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51616/0 (socket says 172.21.15.107:51616) 2023-10-25T00:22:19.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.288+0000 7f7b956ad640 1 -- 172.21.15.107:0/1543676863 learned_addr learned my addr 172.21.15.107:0/1543676863 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b956ad640 1 -- 172.21.15.107:0/1543676863 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b901526d0 msgr2=0x7f7b901eab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b956ad640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b901526d0 0x7f7b901eab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b956ad640 1 -- 172.21.15.107:0/1543676863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b9014c6b0 msgr2=0x7f7b901ea5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b956ad640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b9014c6b0 0x7f7b901ea5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b956ad640 1 -- 172.21.15.107:0/1543676863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b80009420 con 0x7f7b900ab860 2023-10-25T00:22:19.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b956ad640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b900ab860 0x7f7b9013f930 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00c420 tx=0x7f7b8c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b85ffb640 1 -- 172.21.15.107:0/1543676863 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c005cf0 con 0x7f7b900ab860 2023-10-25T00:22:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b85ffb640 1 -- 172.21.15.107:0/1543676863 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8c005e50 con 0x7f7b900ab860 2023-10-25T00:22:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b97911640 1 -- 172.21.15.107:0/1543676863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b901edf60 con 0x7f7b900ab860 2023-10-25T00:22:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b85ffb640 1 -- 172.21.15.107:0/1543676863 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0072d0 con 0x7f7b900ab860 2023-10-25T00:22:19.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b97911640 1 -- 172.21.15.107:0/1543676863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b901469d0 con 0x7f7b900ab860 2023-10-25T00:22:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b97911640 1 -- 172.21.15.107:0/1543676863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9006b260 con 0x7f7b900ab860 2023-10-25T00:22:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b85ffb640 1 -- 172.21.15.107:0/1543676863 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7b8c007430 con 0x7f7b900ab860 2023-10-25T00:22:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b85ffb640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b74065e60 0x7f7b74068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b94eac640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b74065e60 0x7f7b74068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b85ffb640 1 -- 172.21.15.107:0/1543676863 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f7b8c083f70 con 0x7f7b900ab860 2023-10-25T00:22:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.292+0000 7f7b94eac640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b74065e60 0x7f7b74068320 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7b800ae000 tx=0x7f7b80004740 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:19.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.296+0000 7f7b85ffb640 1 -- 172.21.15.107:0/1543676863 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7b8c0b5040 con 0x7f7b900ab860 2023-10-25T00:22:19.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 -- 172.21.15.107:0/1543676863 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7b90073dc0 con 0x7f7b900ab860 2023-10-25T00:22:19.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b85ffb640 1 -- 172.21.15.107:0/1543676863 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v72) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7b8c017020 con 0x7f7b900ab860 2023-10-25T00:22:19.452 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-25T00:22:19.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 -- 172.21.15.107:0/1543676863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b74065e60 msgr2=0x7f7b74068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b74065e60 0x7f7b74068320 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7b800ae000 tx=0x7f7b80004740 comp rx=0 tx=0).stop 2023-10-25T00:22:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 -- 172.21.15.107:0/1543676863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b900ab860 msgr2=0x7f7b9013f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b900ab860 0x7f7b9013f930 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00c420 tx=0x7f7b8c00c8e0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 -- 172.21.15.107:0/1543676863 shutdown_connections 2023-10-25T00:22:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7b74065e60 0x7f7b74068320 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b901526d0 0x7f7b901eab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b9014c6b0 0x7f7b901ea5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 --2- 172.21.15.107:0/1543676863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b900ab860 0x7f7b9013f930 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 -- 172.21.15.107:0/1543676863 >> 172.21.15.107:0/1543676863 conn(0x7f7b900c4e30 msgr2=0x7f7b900a9450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 -- 172.21.15.107:0/1543676863 shutdown_connections 2023-10-25T00:22:19.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.452+0000 7f7b97911640 1 -- 172.21.15.107:0/1543676863 wait complete. 2023-10-25T00:22:19.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-25T00:22:19.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.596+0000 7f030b97c640 1 Processor -- start 2023-10-25T00:22:19.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.596+0000 7f030b97c640 1 -- start start 2023-10-25T00:22:19.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f030b97c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03040aa9f0 0x7f0304072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f030b97c640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03040befa0 con 0x7f03040a9300 2023-10-25T00:22:19.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f030b97c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03040a9b80 con 0x7f03040c1b30 2023-10-25T00:22:19.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f030b97c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03040aa0b0 con 0x7f03040aa9f0 2023-10-25T00:22:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0309f19640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f03040a9300 0x7f0304155620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60708/0 (socket says 172.21.15.107:60708) 2023-10-25T00:22:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0309f19640 1 -- 172.21.15.107:0/1186725062 learned_addr learned my addr 172.21.15.107:0/1186725062 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0308f17640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03040aa9f0 0x7f0304072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1295275403 0 0) 0x7f03040befa0 con 0x7f03040a9300 2023-10-25T00:22:19.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f0003530 con 0x7f03040a9300 2023-10-25T00:22:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2312260046 0 0) 0x7f02f0003530 con 0x7f03040a9300 2023-10-25T00:22:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03040aa9f0 msgr2=0x7f0304072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03040aa9f0 0x7f0304072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 >> v1:172.21.15.110:6789/0 conn(0x7f03040c1b30 legacy=0x7f03040aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03040bf1f0 con 0x7f03040a9300 2023-10-25T00:22:19.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f02fc001680 con 0x7f03040a9300 2023-10-25T00:22:19.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 >> v1:172.21.15.107:6789/0 conn(0x7f03040a9300 legacy=0x7f0304155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02f00039b0 0x7f02f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f0004310 0x7f02f00047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02f0005090 0x7f02f000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03040aa0b0 con 0x7f02f0004310 2023-10-25T00:22:19.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f000bfb0 con 0x7f02f0005090 2023-10-25T00:22:19.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f000c170 con 0x7f02f00039b0 2023-10-25T00:22:19.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0308f17640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f0004310 0x7f02f00047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0309718640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02f00039b0 0x7f02f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0308f17640 1 -- 172.21.15.107:0/1186725062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02f00039b0 msgr2=0x7f02f0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:19.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0309f19640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02f0005090 0x7f02f000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0308f17640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02f00039b0 0x7f02f0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0308f17640 1 -- 172.21.15.107:0/1186725062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02f0005090 msgr2=0x7f02f000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:19.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0308f17640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f02f0005090 0x7f02f000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0308f17640 1 -- 172.21.15.107:0/1186725062 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f0003530 con 0x7f02f0004310 2023-10-25T00:22:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f0308f17640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f0004310 0x7f02f00047f0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009fc0 tx=0x7f02f801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6886fc724a35026a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f80a5830 con 0x7f02f0004310 2023-10-25T00:22:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f800f040 con 0x7f02f0004310 2023-10-25T00:22:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.600+0000 7f02f7fff640 1 -- 172.21.15.107:0/1186725062 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f80a5bd0 con 0x7f02f0004310 2023-10-25T00:22:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 -- 172.21.15.107:0/1186725062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f0004310 msgr2=0x7f02f00047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 --2- 172.21.15.107:0/1186725062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f0004310 0x7f02f00047f0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f02f8009fc0 tx=0x7f02f801cc60 comp rx=0 tx=0).stop 2023-10-25T00:22:19.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309f19640 1 -- 172.21.15.107:0/1186725062 reap_dead start 2023-10-25T00:22:19.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 -- 172.21.15.107:0/1186725062 shutdown_connections 2023-10-25T00:22:19.612 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 -- 172.21.15.107:0/1186725062 >> 172.21.15.107:0/1186725062 conn(0x7f03040c2860 msgr2=0x7f03040c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:19.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 -- 172.21.15.107:0/1186725062 shutdown_connections 2023-10-25T00:22:19.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 -- 172.21.15.107:0/1186725062 wait complete. 2023-10-25T00:22:19.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 Processor -- start 2023-10-25T00:22:19.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 -- start start 2023-10-25T00:22:19.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02f0004310 0x7f03041522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03040bad10 0x7f030414cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03040bb170 0x7f030414d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030414f0e0 con 0x7f03040bb170 2023-10-25T00:22:19.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03040bbac0 con 0x7f03040bad10 2023-10-25T00:22:19.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03040bbda0 con 0x7f02f0004310 2023-10-25T00:22:19.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309f19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03040bb170 0x7f030414d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309f19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03040bb170 0x7f030414d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51624/0 (socket says 172.21.15.107:51624) 2023-10-25T00:22:19.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309f19640 1 -- 172.21.15.107:0/62240280 learned_addr learned my addr 172.21.15.107:0/62240280 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:19.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0308f17640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03040bad10 0x7f030414cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309718640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02f0004310 0x7f03041522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309f19640 1 -- 172.21.15.107:0/62240280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02f0004310 msgr2=0x7f03041522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309f19640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02f0004310 0x7f03041522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309f19640 1 -- 172.21.15.107:0/62240280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03040bad10 msgr2=0x7f030414cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309f19640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03040bad10 0x7f030414cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309f19640 1 -- 172.21.15.107:0/62240280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030414fd40 con 0x7f03040bb170 2023-10-25T00:22:19.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309718640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02f0004310 0x7f03041522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:22:19.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f0309f19640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03040bb170 0x7f030414d020 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f02fc001210 tx=0x7f02fc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:19.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f02f5ffb640 1 -- 172.21.15.107:0/62240280 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc007230 con 0x7f03040bb170 2023-10-25T00:22:19.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f02f5ffb640 1 -- 172.21.15.107:0/62240280 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc0073b0 con 0x7f03040bb170 2023-10-25T00:22:19.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f02f5ffb640 1 -- 172.21.15.107:0/62240280 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0047f0 con 0x7f03040bb170 2023-10-25T00:22:19.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 -- 172.21.15.107:0/62240280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0304150020 con 0x7f03040bb170 2023-10-25T00:22:19.621 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.604+0000 7f030b97c640 1 -- 172.21.15.107:0/62240280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0304152800 con 0x7f03040bb170 2023-10-25T00:22:19.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.608+0000 7f030b97c640 1 -- 172.21.15.107:0/62240280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03040aba00 con 0x7f03040bb170 2023-10-25T00:22:19.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.608+0000 7f02f5ffb640 1 -- 172.21.15.107:0/62240280 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f02fc004950 con 0x7f03040bb170 2023-10-25T00:22:19.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.608+0000 7f02f5ffb640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f02e8065eb0 0x7f02e8068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.608+0000 7f02f5ffb640 1 -- 172.21.15.107:0/62240280 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f02fc084060 con 0x7f03040bb170 2023-10-25T00:22:19.622 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.608+0000 7f0309718640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f02e8065eb0 0x7f02e8068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.608+0000 7f0309718640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f02e8065eb0 0x7f02e8068370 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0300005630 tx=0x7f0300009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:19.623 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.612+0000 7f02f5ffb640 1 -- 172.21.15.107:0/62240280 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f02fc050490 con 0x7f03040bb170 2023-10-25T00:22:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.764+0000 7f030b97c640 1 -- 172.21.15.107:0/62240280 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f03040c15a0 con 0x7f03040bb170 2023-10-25T00:22:19.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f02f5ffb640 1 -- 172.21.15.107:0/62240280 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v72) v1 ==== 99+0+11 (secure 0 0 0) 0x7f02fc053ab0 con 0x7f03040bb170 2023-10-25T00:22:19.767 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-25T00:22:19.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 -- 172.21.15.107:0/62240280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f02e8065eb0 msgr2=0x7f02e8068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f02e8065eb0 0x7f02e8068370 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0300005630 tx=0x7f0300009040 comp rx=0 tx=0).stop 2023-10-25T00:22:19.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 -- 172.21.15.107:0/62240280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03040bb170 msgr2=0x7f030414d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03040bb170 0x7f030414d020 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f02fc001210 tx=0x7f02fc00dc00 comp rx=0 tx=0).stop 2023-10-25T00:22:19.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 -- 172.21.15.107:0/62240280 shutdown_connections 2023-10-25T00:22:19.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03040bb170 0x7f030414d020 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03040bad10 0x7f030414cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f02e8065eb0 0x7f02e8068370 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 --2- 172.21.15.107:0/62240280 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f02f0004310 0x7f03041522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 -- 172.21.15.107:0/62240280 >> 172.21.15.107:0/62240280 conn(0x7f03040c2860 msgr2=0x7f03040c4520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:19.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 -- 172.21.15.107:0/62240280 shutdown_connections 2023-10-25T00:22:19.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.768+0000 7f030b97c640 1 -- 172.21.15.107:0/62240280 wait complete. 2023-10-25T00:22:19.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-25T00:22:19.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.900+0000 7f874d05d640 1 Processor -- start 2023-10-25T00:22:19.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.900+0000 7f874d05d640 1 -- start start 2023-10-25T00:22:19.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.900+0000 7f874d05d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8748072720 0x7f87480c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.900+0000 7f874d05d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87480bceb0 0x7f874815c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.900+0000 7f874d05d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87480c1ce0 con 0x7f87480ab1a0 2023-10-25T00:22:19.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.900+0000 7f874d05d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87480be1d0 con 0x7f8748072720 2023-10-25T00:22:19.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.900+0000 7f874d05d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87480c0240 con 0x7f87480bceb0 2023-10-25T00:22:19.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8748072720 0x7f87480c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874659c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8748072720 0x7f87480c0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51626/0 (socket says 172.21.15.107:51626) 2023-10-25T00:22:19.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874659c640 1 -- 172.21.15.107:0/3060947724 learned_addr learned my addr 172.21.15.107:0/3060947724 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:19.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874759e640 1 --2- 172.21.15.107:0/3060947724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87480bceb0 0x7f874815c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874659c640 1 -- 172.21.15.107:0/3060947724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87480bceb0 msgr2=0x7f874815c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874659c640 1 --2- 172.21.15.107:0/3060947724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87480bceb0 0x7f874815c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874659c640 1 -- 172.21.15.107:0/3060947724 >> v1:172.21.15.110:6789/0 conn(0x7f87480ab1a0 legacy=0x7f87480bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:19.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874659c640 1 -- 172.21.15.107:0/3060947724 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87480c0880 con 0x7f8748072720 2023-10-25T00:22:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874659c640 1 --2- 172.21.15.107:0/3060947724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8748072720 0x7f87480c0d00 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f873c009870 tx=0x7f873c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d159065b11ffce88 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f8745d9b640 1 -- 172.21.15.107:0/3060947724 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c0ab600 con 0x7f8748072720 2023-10-25T00:22:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f8745d9b640 1 -- 172.21.15.107:0/3060947724 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c0a7050 con 0x7f8748072720 2023-10-25T00:22:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f8745d9b640 1 -- 172.21.15.107:0/3060947724 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c0ab980 con 0x7f8748072720 2023-10-25T00:22:19.907 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874d05d640 1 -- 172.21.15.107:0/3060947724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8748072720 msgr2=0x7f87480c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874d05d640 1 --2- 172.21.15.107:0/3060947724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8748072720 0x7f87480c0d00 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f873c009870 tx=0x7f873c01c3b0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874d05d640 1 -- 172.21.15.107:0/3060947724 shutdown_connections 2023-10-25T00:22:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874d05d640 1 --2- 172.21.15.107:0/3060947724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87480bceb0 0x7f874815c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874d05d640 1 --2- 172.21.15.107:0/3060947724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8748072720 0x7f87480c0d00 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.908 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874d05d640 1 -- 172.21.15.107:0/3060947724 >> 172.21.15.107:0/3060947724 conn(0x7f87480c2860 msgr2=0x7f87480c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:19.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874d05d640 1 -- 172.21.15.107:0/3060947724 shutdown_connections 2023-10-25T00:22:19.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.904+0000 7f874d05d640 1 -- 172.21.15.107:0/3060947724 wait complete. 2023-10-25T00:22:19.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874d05d640 1 Processor -- start 2023-10-25T00:22:19.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874d05d640 1 -- start start 2023-10-25T00:22:19.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874d05d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87480ab1a0 0x7f8748070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874d05d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87480bceb0 0x7f87480c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.912 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874d05d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8748154410 0x7f87480c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874d05d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8748159ec0 con 0x7f87480bceb0 2023-10-25T00:22:19.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874d05d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874815ace0 con 0x7f8748154410 2023-10-25T00:22:19.913 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874d05d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87481ed7e0 con 0x7f87480ab1a0 2023-10-25T00:22:19.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87480bceb0 0x7f87480c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87480bceb0 0x7f87480c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51630/0 (socket says 172.21.15.107:51630) 2023-10-25T00:22:19.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874659c640 1 -- 172.21.15.107:0/2513266835 learned_addr learned my addr 172.21.15.107:0/2513266835 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:19.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874759e640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8748154410 0x7f87480c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f8746d9d640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87480ab1a0 0x7f8748070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874659c640 1 -- 172.21.15.107:0/2513266835 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87480ab1a0 msgr2=0x7f8748070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874659c640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87480ab1a0 0x7f8748070f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874659c640 1 -- 172.21.15.107:0/2513266835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8748154410 msgr2=0x7f87480c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:19.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874659c640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8748154410 0x7f87480c51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:19.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874659c640 1 -- 172.21.15.107:0/2513266835 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c0094d0 con 0x7f87480bceb0 2023-10-25T00:22:19.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874759e640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8748154410 0x7f87480c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:19.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874659c640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87480bceb0 0x7f87480c4c80 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f873c0ae040 tx=0x7f873c01abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:19.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f872f7fe640 1 -- 172.21.15.107:0/2513266835 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c0b6620 con 0x7f87480bceb0 2023-10-25T00:22:19.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f872f7fe640 1 -- 172.21.15.107:0/2513266835 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f873c0abe90 con 0x7f87480bceb0 2023-10-25T00:22:19.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f872f7fe640 1 -- 172.21.15.107:0/2513266835 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873c0b69e0 con 0x7f87480bceb0 2023-10-25T00:22:19.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874d05d640 1 -- 172.21.15.107:0/2513266835 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87481ecc60 con 0x7f87480bceb0 2023-10-25T00:22:19.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874d05d640 1 -- 172.21.15.107:0/2513266835 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87481ed220 con 0x7f87480bceb0 2023-10-25T00:22:19.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f874d05d640 1 -- 172.21.15.107:0/2513266835 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87481efe30 con 0x7f87480bceb0 2023-10-25T00:22:19.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.908+0000 7f872f7fe640 1 -- 172.21.15.107:0/2513266835 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f873c0b2040 con 0x7f87480bceb0 2023-10-25T00:22:19.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.912+0000 7f872f7fe640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8734065e90 0x7f8734068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:19.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.912+0000 7f8746d9d640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8734065e90 0x7f8734068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.912+0000 7f872f7fe640 1 -- 172.21.15.107:0/2513266835 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f873c11efb0 con 0x7f87480bceb0 2023-10-25T00:22:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.912+0000 7f8746d9d640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8734065e90 0x7f8734068350 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f874806ef60 tx=0x7f8738009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:19.916+0000 7f872f7fe640 1 -- 172.21.15.107:0/2513266835 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f873c150040 con 0x7f87480bceb0 2023-10-25T00:22:20.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.056+0000 7f874d05d640 1 -- 172.21.15.107:0/2513266835 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f87480beff0 con 0x7f87480bceb0 2023-10-25T00:22:20.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.056+0000 7f872f7fe640 1 -- 172.21.15.107:0/2513266835 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v72) v1 ==== 95+0+11 (secure 0 0 0) 0x7f873c0ba0a0 con 0x7f87480bceb0 2023-10-25T00:22:20.058 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-25T00:22:20.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 -- 172.21.15.107:0/2513266835 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8734065e90 msgr2=0x7f8734068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8734065e90 0x7f8734068350 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f874806ef60 tx=0x7f8738009040 comp rx=0 tx=0).stop 2023-10-25T00:22:20.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 -- 172.21.15.107:0/2513266835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87480bceb0 msgr2=0x7f87480c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87480bceb0 0x7f87480c4c80 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f873c0ae040 tx=0x7f873c01abf0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 -- 172.21.15.107:0/2513266835 shutdown_connections 2023-10-25T00:22:20.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8748154410 0x7f87480c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87480bceb0 0x7f87480c4c80 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8734065e90 0x7f8734068350 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 --2- 172.21.15.107:0/2513266835 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87480ab1a0 0x7f8748070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 -- 172.21.15.107:0/2513266835 >> 172.21.15.107:0/2513266835 conn(0x7f87480c2860 msgr2=0x7f87480a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:20.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 -- 172.21.15.107:0/2513266835 shutdown_connections 2023-10-25T00:22:20.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.060+0000 7f874d05d640 1 -- 172.21.15.107:0/2513266835 wait complete. 2023-10-25T00:22:20.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-10-25T00:22:20.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f1064ab2640 1 Processor -- start 2023-10-25T00:22:20.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f1064ab2640 1 -- start start 2023-10-25T00:22:20.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f1064ab2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1060072720 0x7f10600c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f1064ab2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10600c1ce0 con 0x7f10600bceb0 2023-10-25T00:22:20.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f1064ab2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10600c0240 con 0x7f10600ab1a0 2023-10-25T00:22:20.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f1064ab2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10600be1d0 con 0x7f1060072720 2023-10-25T00:22:20.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f105ed9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f10600bceb0 0x7f106015c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60710/0 (socket says 172.21.15.107:60710) 2023-10-25T00:22:20.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f1057fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1060072720 0x7f10600c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f105ed9d640 1 -- 172.21.15.107:0/3138099085 learned_addr learned my addr 172.21.15.107:0/3138099085 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:20.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f1057fff640 1 -- 172.21.15.107:0/3138099085 >> v1:172.21.15.110:6789/0 conn(0x7f10600ab1a0 legacy=0x7f10600bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:20.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f1057fff640 1 -- 172.21.15.107:0/3138099085 >> v1:172.21.15.107:6789/0 conn(0x7f10600bceb0 legacy=0x7f106015c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f1057fff640 1 -- 172.21.15.107:0/3138099085 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10600c0880 con 0x7f1060072720 2023-10-25T00:22:20.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.200+0000 7f105dd9b640 1 -- 172.21.15.107:0/3138099085 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316323125 0 0) 0x7f10600c1ce0 con 0x7f10600bceb0 2023-10-25T00:22:20.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1057fff640 1 --2- 172.21.15.107:0/3138099085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1060072720 0x7f10600c0d00 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f10480097c0 tx=0x7f104801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fed5f1bb24b41af server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:20.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105dd9b640 1 -- 172.21.15.107:0/3138099085 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10480ad680 con 0x7f1060072720 2023-10-25T00:22:20.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105dd9b640 1 -- 172.21.15.107:0/3138099085 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10480ad7e0 con 0x7f1060072720 2023-10-25T00:22:20.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105dd9b640 1 -- 172.21.15.107:0/3138099085 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10480adb70 con 0x7f1060072720 2023-10-25T00:22:20.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 -- 172.21.15.107:0/3138099085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1060072720 msgr2=0x7f10600c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 --2- 172.21.15.107:0/3138099085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1060072720 0x7f10600c0d00 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f10480097c0 tx=0x7f104801c380 comp rx=0 tx=0).stop 2023-10-25T00:22:20.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 -- 172.21.15.107:0/3138099085 shutdown_connections 2023-10-25T00:22:20.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 --2- 172.21.15.107:0/3138099085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1060072720 0x7f10600c0d00 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 -- 172.21.15.107:0/3138099085 >> 172.21.15.107:0/3138099085 conn(0x7f10600c2860 msgr2=0x7f10600c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 -- 172.21.15.107:0/3138099085 shutdown_connections 2023-10-25T00:22:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 -- 172.21.15.107:0/3138099085 wait complete. 2023-10-25T00:22:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 Processor -- start 2023-10-25T00:22:20.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 -- start start 2023-10-25T00:22:20.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10600ab1a0 0x7f10601f05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10600bceb0 0x7f10601f0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1060148540 0x7f1060070760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10600bc740 con 0x7f1060148540 2023-10-25T00:22:20.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10600bc9b0 con 0x7f10600bceb0 2023-10-25T00:22:20.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1060146900 con 0x7f10600ab1a0 2023-10-25T00:22:20.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1060148540 0x7f1060070760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1060148540 0x7f1060070760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51652/0 (socket says 172.21.15.107:51652) 2023-10-25T00:22:20.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105ed9d640 1 -- 172.21.15.107:0/4154410199 learned_addr learned my addr 172.21.15.107:0/4154410199 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:20.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105ed9d640 1 -- 172.21.15.107:0/4154410199 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10600ab1a0 msgr2=0x7f10601f05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105ed9d640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10600ab1a0 0x7f10601f05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105ed9d640 1 -- 172.21.15.107:0/4154410199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10600bceb0 msgr2=0x7f10601f0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105ed9d640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10600bceb0 0x7f10601f0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105ed9d640 1 -- 172.21.15.107:0/4154410199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1048009420 con 0x7f1060148540 2023-10-25T00:22:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f105ed9d640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1060148540 0x7f1060070760 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f105800c750 tx=0x7f105800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1056ffd640 1 -- 172.21.15.107:0/4154410199 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800ce60 con 0x7f1060148540 2023-10-25T00:22:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1056ffd640 1 -- 172.21.15.107:0/4154410199 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105800a640 con 0x7f1060148540 2023-10-25T00:22:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 -- 172.21.15.107:0/4154410199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1060146be0 con 0x7f1060148540 2023-10-25T00:22:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1056ffd640 1 -- 172.21.15.107:0/4154410199 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800a9b0 con 0x7f1060148540 2023-10-25T00:22:20.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.204+0000 7f1064ab2640 1 -- 172.21.15.107:0/4154410199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1060146ec0 con 0x7f1060148540 2023-10-25T00:22:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.208+0000 7f1064ab2640 1 -- 172.21.15.107:0/4154410199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1060071b80 con 0x7f1060148540 2023-10-25T00:22:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.208+0000 7f1056ffd640 1 -- 172.21.15.107:0/4154410199 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f105800ab10 con 0x7f1060148540 2023-10-25T00:22:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.208+0000 7f1056ffd640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1040065e60 0x7f1040068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.208+0000 7f1056ffd640 1 -- 172.21.15.107:0/4154410199 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f10580840c0 con 0x7f1060148540 2023-10-25T00:22:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.208+0000 7f105e59c640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1040065e60 0x7f1040068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.208+0000 7f105e59c640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1040065e60 0x7f1040068320 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1050003cf0 tx=0x7f10500079c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:20.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.212+0000 7f1056ffd640 1 -- 172.21.15.107:0/4154410199 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f10580503e0 con 0x7f1060148540 2023-10-25T00:22:20.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1064ab2640 1 -- 172.21.15.107:0/4154410199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7f106004e130 con 0x7f1060148540 2023-10-25T00:22:20.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1056ffd640 1 -- 172.21.15.107:0/4154410199 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v72) v1 ==== 99+0+11 (secure 0 0 0) 0x7f1058018030 con 0x7f1060148540 2023-10-25T00:22:20.357 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-25T00:22:20.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1064ab2640 1 -- 172.21.15.107:0/4154410199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1040065e60 msgr2=0x7f1040068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1064ab2640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1040065e60 0x7f1040068320 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1050003cf0 tx=0x7f10500079c0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1064ab2640 1 -- 172.21.15.107:0/4154410199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1060148540 msgr2=0x7f1060070760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1064ab2640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1060148540 0x7f1060070760 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f105800c750 tx=0x7f105800cc10 comp rx=0 tx=0).stop 2023-10-25T00:22:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1064ab2640 1 -- 172.21.15.107:0/4154410199 shutdown_connections 2023-10-25T00:22:20.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1064ab2640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1060148540 0x7f1060070760 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1064ab2640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10600bceb0 0x7f10601f0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1064ab2640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1040065e60 0x7f1040068320 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1064ab2640 1 --2- 172.21.15.107:0/4154410199 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10600ab1a0 0x7f10601f05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.356+0000 7f1064ab2640 1 -- 172.21.15.107:0/4154410199 >> 172.21.15.107:0/4154410199 conn(0x7f10600c2860 msgr2=0x7f1060155470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:20.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.360+0000 7f1064ab2640 1 -- 172.21.15.107:0/4154410199 shutdown_connections 2023-10-25T00:22:20.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.360+0000 7f1064ab2640 1 -- 172.21.15.107:0/4154410199 wait complete. 2023-10-25T00:22:20.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-10-25T00:22:20.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.496+0000 7f0ef5e73640 1 Processor -- start 2023-10-25T00:22:20.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.496+0000 7f0ef5e73640 1 -- start start 2023-10-25T00:22:20.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0ef00ab1a0 0x7f0ef00c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ef00bceb0 0x7f0ef015c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef00c1ce0 con 0x7f0ef0072720 2023-10-25T00:22:20.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ef00be1d0 con 0x7f0ef00ab1a0 2023-10-25T00:22:20.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef00c0240 con 0x7f0ef00bceb0 2023-10-25T00:22:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eef7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0ef00ab1a0 0x7f0ef00c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eef7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f0ef00ab1a0 0x7f0ef00c0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51664/0 (socket says 172.21.15.107:51664) 2023-10-25T00:22:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eef7fe640 1 -- 172.21.15.107:0/3797546299 learned_addr learned my addr 172.21.15.107:0/3797546299 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eeffff640 1 --2- 172.21.15.107:0/3797546299 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ef00bceb0 0x7f0ef015c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eef7fe640 1 -- 172.21.15.107:0/3797546299 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ef00bceb0 msgr2=0x7f0ef015c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:20.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eef7fe640 1 --2- 172.21.15.107:0/3797546299 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ef00bceb0 0x7f0ef015c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eef7fe640 1 -- 172.21.15.107:0/3797546299 >> v1:172.21.15.110:6789/0 conn(0x7f0ef0072720 legacy=0x7f0ef00bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:20.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eef7fe640 1 -- 172.21.15.107:0/3797546299 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef00c0880 con 0x7f0ef00ab1a0 2023-10-25T00:22:20.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eeffff640 1 --2- 172.21.15.107:0/3797546299 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ef00bceb0 0x7f0ef015c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:20.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eef7fe640 1 --2- 172.21.15.107:0/3797546299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef00ab1a0 0x7f0ef00c0d00 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0009870 tx=0x7f0ee001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feefb73bdc1282fb server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:20.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eee7fc640 1 -- 172.21.15.107:0/3797546299 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee00ab600 con 0x7f0ef00ab1a0 2023-10-25T00:22:20.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eee7fc640 1 -- 172.21.15.107:0/3797546299 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee00a7050 con 0x7f0ef00ab1a0 2023-10-25T00:22:20.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0eee7fc640 1 -- 172.21.15.107:0/3797546299 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee00ab980 con 0x7f0ef00ab1a0 2023-10-25T00:22:20.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 -- 172.21.15.107:0/3797546299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef00ab1a0 msgr2=0x7f0ef00c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 --2- 172.21.15.107:0/3797546299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef00ab1a0 0x7f0ef00c0d00 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0009870 tx=0x7f0ee001c3b0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 -- 172.21.15.107:0/3797546299 shutdown_connections 2023-10-25T00:22:20.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 --2- 172.21.15.107:0/3797546299 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ef00bceb0 0x7f0ef015c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 --2- 172.21.15.107:0/3797546299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef00ab1a0 0x7f0ef00c0d00 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 -- 172.21.15.107:0/3797546299 >> 172.21.15.107:0/3797546299 conn(0x7f0ef00c2860 msgr2=0x7f0ef00c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:20.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 -- 172.21.15.107:0/3797546299 shutdown_connections 2023-10-25T00:22:20.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.500+0000 7f0ef5e73640 1 -- 172.21.15.107:0/3797546299 wait complete. 2023-10-25T00:22:20.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ef5e73640 1 Processor -- start 2023-10-25T00:22:20.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ef5e73640 1 -- start start 2023-10-25T00:22:20.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ef5e73640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ef00ab1a0 0x7f0ef0157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ef5e73640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef00bceb0 0x7f0ef00bbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ef5e73640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ef00bc9f0 0x7f0ef00bc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ef5e73640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef014d140 con 0x7f0ef00bceb0 2023-10-25T00:22:20.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ef5e73640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef014c620 con 0x7f0ef00bc9f0 2023-10-25T00:22:20.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ef5e73640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef01eb380 con 0x7f0ef00ab1a0 2023-10-25T00:22:20.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0eeeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef00bceb0 0x7f0ef00bbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0eeeffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef00bceb0 0x7f0ef00bbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51674/0 (socket says 172.21.15.107:51674) 2023-10-25T00:22:20.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0eeeffd640 1 -- 172.21.15.107:0/1014409333 learned_addr learned my addr 172.21.15.107:0/1014409333 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:20.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0eeffff640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ef00bc9f0 0x7f0ef00bc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0eef7fe640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ef00ab1a0 0x7f0ef0157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0eeeffd640 1 -- 172.21.15.107:0/1014409333 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ef00ab1a0 msgr2=0x7f0ef0157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0eeeffd640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ef00ab1a0 0x7f0ef0157c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0eeeffd640 1 -- 172.21.15.107:0/1014409333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ef00bc9f0 msgr2=0x7f0ef00bc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0eeeffd640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ef00bc9f0 0x7f0ef00bc470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0eeeffd640 1 -- 172.21.15.107:0/1014409333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee00094d0 con 0x7f0ef00bceb0 2023-10-25T00:22:20.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0eeeffd640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef00bceb0 0x7f0ef00bbf30 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400c430 tx=0x7f0ee400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ed7fff640 1 -- 172.21.15.107:0/1014409333 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4016070 con 0x7f0ef00bceb0 2023-10-25T00:22:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ed7fff640 1 -- 172.21.15.107:0/1014409333 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee4007020 con 0x7f0ef00bceb0 2023-10-25T00:22:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ef5e73640 1 -- 172.21.15.107:0/1014409333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef014c870 con 0x7f0ef00bceb0 2023-10-25T00:22:20.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ed7fff640 1 -- 172.21.15.107:0/1014409333 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee40073d0 con 0x7f0ef00bceb0 2023-10-25T00:22:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ef5e73640 1 -- 172.21.15.107:0/1014409333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef01edb20 con 0x7f0ef00bceb0 2023-10-25T00:22:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ef5e73640 1 -- 172.21.15.107:0/1014409333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef014cd90 con 0x7f0ef00bceb0 2023-10-25T00:22:20.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.504+0000 7f0ed7fff640 1 -- 172.21.15.107:0/1014409333 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f0ee4007530 con 0x7f0ef00bceb0 2023-10-25T00:22:20.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.508+0000 7f0ed7fff640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0edc065e60 0x7f0edc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.508+0000 7f0ed7fff640 1 -- 172.21.15.107:0/1014409333 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f0ee4084300 con 0x7f0ef00bceb0 2023-10-25T00:22:20.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.508+0000 7f0eef7fe640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0edc065e60 0x7f0edc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.508+0000 7f0eef7fe640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0edc065e60 0x7f0edc068320 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00ad000 tx=0x7f0ee0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:20.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.512+0000 7f0ed7fff640 1 -- 172.21.15.107:0/1014409333 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f0ef014cd90 con 0x7f0ef00bceb0 2023-10-25T00:22:20.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 -- 172.21.15.107:0/1014409333 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7f0ef014cd90 con 0x7f0ef00bceb0 2023-10-25T00:22:20.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ed7fff640 1 -- 172.21.15.107:0/1014409333 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v72) v1 ==== 95+0+11 (secure 0 0 0) 0x7f0ef014cd90 con 0x7f0ef00bceb0 2023-10-25T00:22:20.664 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-25T00:22:20.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 -- 172.21.15.107:0/1014409333 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0edc065e60 msgr2=0x7f0edc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0edc065e60 0x7f0edc068320 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0ee00ad000 tx=0x7f0ee0008040 comp rx=0 tx=0).stop 2023-10-25T00:22:20.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 -- 172.21.15.107:0/1014409333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef00bceb0 msgr2=0x7f0ef00bbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef00bceb0 0x7f0ef00bbf30 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400c430 tx=0x7f0ee400c8f0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 -- 172.21.15.107:0/1014409333 shutdown_connections 2023-10-25T00:22:20.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0ef00bc9f0 0x7f0ef00bc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ef00bceb0 0x7f0ef00bbf30 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f0edc065e60 0x7f0edc068320 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 --2- 172.21.15.107:0/1014409333 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0ef00ab1a0 0x7f0ef0157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 -- 172.21.15.107:0/1014409333 >> 172.21.15.107:0/1014409333 conn(0x7f0ef00c2860 msgr2=0x7f0ef0155570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:20.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 -- 172.21.15.107:0/1014409333 shutdown_connections 2023-10-25T00:22:20.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.664+0000 7f0ef5e73640 1 -- 172.21.15.107:0/1014409333 wait complete. 2023-10-25T00:22:20.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-25T00:22:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f9e130640 1 Processor -- start 2023-10-25T00:22:20.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f9e130640 1 -- start start 2023-10-25T00:22:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f9e130640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f980ab1a0 0x7f1f980c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f9e130640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f98072720 0x7f1f980bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f9e130640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f980c1ce0 con 0x7f1f980bceb0 2023-10-25T00:22:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f9e130640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f980be1d0 con 0x7f1f98072720 2023-10-25T00:22:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f9e130640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f980c0880 con 0x7f1f980ab1a0 2023-10-25T00:22:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f96ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f98072720 0x7f1f980bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f96ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1f98072720 0x7f1f980bfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51682/0 (socket says 172.21.15.107:51682) 2023-10-25T00:22:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f96ffd640 1 -- 172.21.15.107:0/3423663211 learned_addr learned my addr 172.21.15.107:0/3423663211 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:20.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f977fe640 1 --2- 172.21.15.107:0/3423663211 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f980ab1a0 0x7f1f980c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f96ffd640 1 -- 172.21.15.107:0/3423663211 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f980ab1a0 msgr2=0x7f1f980c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:20.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f96ffd640 1 --2- 172.21.15.107:0/3423663211 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f980ab1a0 0x7f1f980c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f96ffd640 1 -- 172.21.15.107:0/3423663211 >> v1:172.21.15.110:6789/0 conn(0x7f1f980bceb0 legacy=0x7f1f980c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:20.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f96ffd640 1 -- 172.21.15.107:0/3423663211 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f980c1f30 con 0x7f1f98072720 2023-10-25T00:22:20.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.804+0000 7f1f977fe640 1 --2- 172.21.15.107:0/3423663211 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f980ab1a0 0x7f1f980c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:20.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f96ffd640 1 --2- 172.21.15.107:0/3423663211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f98072720 0x7f1f980bfb30 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009870 tx=0x7f1f8c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3641d6135f949717 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:20.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f967fc640 1 -- 172.21.15.107:0/3423663211 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c0ab600 con 0x7f1f98072720 2023-10-25T00:22:20.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f967fc640 1 -- 172.21.15.107:0/3423663211 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c0a7050 con 0x7f1f98072720 2023-10-25T00:22:20.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f967fc640 1 -- 172.21.15.107:0/3423663211 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c0ab980 con 0x7f1f98072720 2023-10-25T00:22:20.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f9e130640 1 -- 172.21.15.107:0/3423663211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f98072720 msgr2=0x7f1f980bfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f9e130640 1 --2- 172.21.15.107:0/3423663211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f98072720 0x7f1f980bfb30 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c009870 tx=0x7f1f8c01c3b0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f9e130640 1 -- 172.21.15.107:0/3423663211 shutdown_connections 2023-10-25T00:22:20.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f9e130640 1 --2- 172.21.15.107:0/3423663211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f98072720 0x7f1f980bfb30 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f9e130640 1 --2- 172.21.15.107:0/3423663211 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f980ab1a0 0x7f1f980c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f9e130640 1 -- 172.21.15.107:0/3423663211 >> 172.21.15.107:0/3423663211 conn(0x7f1f980c2860 msgr2=0x7f1f980c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:20.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f9e130640 1 -- 172.21.15.107:0/3423663211 shutdown_connections 2023-10-25T00:22:20.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f9e130640 1 -- 172.21.15.107:0/3423663211 wait complete. 2023-10-25T00:22:20.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f9e130640 1 Processor -- start 2023-10-25T00:22:20.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.808+0000 7f1f9e130640 1 -- start start 2023-10-25T00:22:20.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f9e130640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f980ab1a0 0x7f1f98158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f9e130640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f980bceb0 0x7f1f98156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f9e130640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f980a42a0 0x7f1f980a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f9e130640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f980a5530 con 0x7f1f980bceb0 2023-10-25T00:22:20.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f9e130640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f980a6e10 con 0x7f1f980a42a0 2023-10-25T00:22:20.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f9e130640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f980a68e0 con 0x7f1f980ab1a0 2023-10-25T00:22:20.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f96ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f980bceb0 0x7f1f98156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f96ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f980bceb0 0x7f1f98156100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51690/0 (socket says 172.21.15.107:51690) 2023-10-25T00:22:20.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f96ffd640 1 -- 172.21.15.107:0/217551038 learned_addr learned my addr 172.21.15.107:0/217551038 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:20.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f96ffd640 1 -- 172.21.15.107:0/217551038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f980ab1a0 msgr2=0x7f1f98158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f96ffd640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f980ab1a0 0x7f1f98158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f96ffd640 1 -- 172.21.15.107:0/217551038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f980a42a0 msgr2=0x7f1f980a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:20.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f96ffd640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f980a42a0 0x7f1f980a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:20.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f96ffd640 1 -- 172.21.15.107:0/217551038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c0094d0 con 0x7f1f980bceb0 2023-10-25T00:22:20.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f96ffd640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f980bceb0 0x7f1f98156100 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c01cc50 tx=0x7f1f8c019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:20.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f7ffff640 1 -- 172.21.15.107:0/217551038 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c0b37d0 con 0x7f1f980bceb0 2023-10-25T00:22:20.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f7ffff640 1 -- 172.21.15.107:0/217551038 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8c0b3950 con 0x7f1f980bceb0 2023-10-25T00:22:20.818 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f9e130640 1 -- 172.21.15.107:0/217551038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f98159070 con 0x7f1f980bceb0 2023-10-25T00:22:20.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f7ffff640 1 -- 172.21.15.107:0/217551038 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8c0b3d00 con 0x7f1f980bceb0 2023-10-25T00:22:20.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f9e130640 1 -- 172.21.15.107:0/217551038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f980aab70 con 0x7f1f980bceb0 2023-10-25T00:22:20.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f9e130640 1 -- 172.21.15.107:0/217551038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f980a5300 con 0x7f1f980bceb0 2023-10-25T00:22:20.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f7ffff640 1 -- 172.21.15.107:0/217551038 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1f8c0b2070 con 0x7f1f980bceb0 2023-10-25T00:22:20.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f7ffff640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f84065e90 0x7f1f84068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:20.819 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f977fe640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f84065e90 0x7f1f84068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:20.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f7ffff640 1 -- 172.21.15.107:0/217551038 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f1f8c11ecb0 con 0x7f1f980bceb0 2023-10-25T00:22:20.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.812+0000 7f1f977fe640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f84065e90 0x7f1f84068350 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f1f88008ea0 tx=0x7f1f880083d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:20.820 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.816+0000 7f1f7ffff640 1 -- 172.21.15.107:0/217551038 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1f980a5300 con 0x7f1f980bceb0 2023-10-25T00:22:20.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.980+0000 7f1f9e130640 1 -- 172.21.15.107:0/217551038 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1f980beff0 con 0x7f1f980bceb0 2023-10-25T00:22:20.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f7ffff640 1 -- 172.21.15.107:0/217551038 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 88 entries v72) v1 ==== 81+0+5426 (secure 0 0 0) 0x7f1f8c0b0020 con 0x7f1f980bceb0 2023-10-25T00:22:20.983 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2187059523 2023-10-26T00:21:39.173084+0000 2023-10-25T00:22:20.984 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/657626630 2023-10-26T00:21:19.167088+0000 2023-10-25T00:22:20.984 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/3771564389 2023-10-26T00:20:19.149072+0000 2023-10-25T00:22:20.984 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/3771564389 2023-10-26T00:20:19.149072+0000 2023-10-25T00:22:20.984 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2970408860 2023-10-26T00:19:04.122659+0000 2023-10-25T00:22:20.984 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/418589140 2023-10-26T00:18:49.118366+0000 2023-10-25T00:22:20.985 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/985135365 2023-10-26T00:17:39.037219+0000 2023-10-25T00:22:20.985 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/985135365 2023-10-26T00:17:39.037219+0000 2023-10-25T00:22:20.985 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/208713198 2023-10-26T00:17:24.032902+0000 2023-10-25T00:22:20.985 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/208713198 2023-10-26T00:17:24.032902+0000 2023-10-25T00:22:20.985 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/1619327197 2023-10-26T00:15:22.673615+0000 2023-10-25T00:22:20.986 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/175581297 2023-10-26T00:15:18.921240+0000 2023-10-25T00:22:20.986 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/418589140 2023-10-26T00:18:49.118366+0000 2023-10-25T00:22:20.986 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2856791739 2023-10-26T00:13:48.893880+0000 2023-10-25T00:22:20.986 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/898258574 2023-10-26T00:13:18.883076+0000 2023-10-25T00:22:20.986 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3396644475 2023-10-26T00:19:44.136579+0000 2023-10-25T00:22:20.987 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/3370462806 2023-10-26T00:12:33.862408+0000 2023-10-25T00:22:20.987 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/1101018338 2023-10-25T23:59:48.457105+0000 2023-10-25T00:22:20.987 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/838886697 2023-10-25T23:56:18.381650+0000 2023-10-25T00:22:20.987 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1381726420 2023-10-26T00:01:28.521014+0000 2023-10-25T00:22:20.988 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/1619327197 2023-10-26T00:15:22.673615+0000 2023-10-25T00:22:20.988 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1381726420 2023-10-26T00:01:28.521014+0000 2023-10-25T00:22:20.988 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/3103569740 2023-10-26T00:01:58.532147+0000 2023-10-25T00:22:20.988 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/780830443 2023-10-26T00:06:23.634894+0000 2023-10-25T00:22:20.988 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/1101018338 2023-10-25T23:59:48.457105+0000 2023-10-25T00:22:20.989 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1555840329 2023-10-26T00:05:18.610829+0000 2023-10-25T00:22:20.989 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/3836115666 2023-10-26T00:08:58.700067+0000 2023-10-25T00:22:20.989 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/898258574 2023-10-26T00:13:18.883076+0000 2023-10-25T00:22:20.989 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1508236506 2023-10-25T23:58:58.439244+0000 2023-10-25T00:22:20.989 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2348043270 2023-10-25T23:58:53.436687+0000 2023-10-25T00:22:20.990 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/204494054 2023-10-25T23:58:03.418411+0000 2023-10-25T00:22:20.990 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1555840329 2023-10-26T00:05:18.610829+0000 2023-10-25T00:22:20.990 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/3721101226 2023-10-25T23:58:03.418295+0000 2023-10-25T00:22:20.990 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2480841919 2023-10-25T23:55:58.372958+0000 2023-10-25T00:22:20.990 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/2577776351 2023-10-26T00:04:34.728765+0000 2023-10-25T00:22:20.990 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1246928633 2023-10-26T00:05:13.609231+0000 2023-10-25T00:22:20.991 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/913067382 2023-10-26T00:08:08.675592+0000 2023-10-25T00:22:20.991 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/1031240534 2023-10-25T23:57:23.404313+0000 2023-10-25T00:22:20.991 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/727042497 2023-10-25T23:54:13.340839+0000 2023-10-25T00:22:20.991 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2970408860 2023-10-26T00:19:04.122659+0000 2023-10-25T00:22:20.991 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2480841919 2023-10-25T23:55:58.372958+0000 2023-10-25T00:22:20.992 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/201284779 2023-10-26T00:02:23.541454+0000 2023-10-25T00:22:20.992 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/201284779 2023-10-26T00:02:23.541454+0000 2023-10-25T00:22:20.992 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/838886697 2023-10-25T23:56:18.381650+0000 2023-10-25T00:22:20.992 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1292188015 2023-10-26T00:00:48.479959+0000 2023-10-25T00:22:20.992 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1317796403 2023-10-26T00:08:23.682615+0000 2023-10-25T00:22:20.993 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1220982369 2023-10-25T23:55:23.361145+0000 2023-10-25T00:22:20.993 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2348043270 2023-10-25T23:58:53.436687+0000 2023-10-25T00:22:20.993 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/3370462806 2023-10-26T00:12:33.862408+0000 2023-10-25T00:22:20.993 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1220982369 2023-10-25T23:55:23.361145+0000 2023-10-25T00:22:20.993 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/204494054 2023-10-25T23:58:03.418411+0000 2023-10-25T00:22:20.994 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/175581297 2023-10-26T00:15:18.921240+0000 2023-10-25T00:22:20.994 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2856791739 2023-10-26T00:13:48.893880+0000 2023-10-25T00:22:20.994 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/1031240534 2023-10-25T23:57:23.404313+0000 2023-10-25T00:22:20.995 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/326715537 2023-10-26T00:11:03.824824+0000 2023-10-25T00:22:20.995 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1004926771 2023-10-25T23:57:28.406683+0000 2023-10-25T00:22:20.995 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/657626630 2023-10-26T00:21:19.167088+0000 2023-10-25T00:22:20.996 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1004926771 2023-10-25T23:57:28.406683+0000 2023-10-25T00:22:20.996 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/3721101226 2023-10-25T23:58:03.418295+0000 2023-10-25T00:22:20.996 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3474118649 2023-10-26T00:11:18.831318+0000 2023-10-25T00:22:20.996 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3232233949 2023-10-26T00:03:28.568587+0000 2023-10-25T00:22:20.997 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/2577776351 2023-10-26T00:04:34.728765+0000 2023-10-25T00:22:20.997 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/326715537 2023-10-26T00:11:03.824824+0000 2023-10-25T00:22:20.997 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2276136116 2023-10-26T00:09:18.706706+0000 2023-10-25T00:22:20.997 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/319918260 2023-10-26T00:03:53.579302+0000 2023-10-25T00:22:20.998 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3474118649 2023-10-26T00:11:18.831318+0000 2023-10-25T00:22:20.998 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1292188015 2023-10-26T00:00:48.479959+0000 2023-10-25T00:22:20.998 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/174863522 2023-10-26T00:05:53.622773+0000 2023-10-25T00:22:20.998 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/319918260 2023-10-26T00:03:53.579302+0000 2023-10-25T00:22:20.998 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/3103569740 2023-10-26T00:01:58.532147+0000 2023-10-25T00:22:20.999 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2150827048 2023-10-26T00:07:48.667043+0000 2023-10-25T00:22:20.999 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/4180698155 2023-10-26T00:04:28.591981+0000 2023-10-25T00:22:20.999 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3396644475 2023-10-26T00:19:44.136579+0000 2023-10-25T00:22:20.999 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/4180698155 2023-10-26T00:04:28.591981+0000 2023-10-25T00:22:21.000 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2187059523 2023-10-26T00:21:39.173084+0000 2023-10-25T00:22:21.000 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1246928633 2023-10-26T00:05:13.609231+0000 2023-10-25T00:22:21.000 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/174863522 2023-10-26T00:05:53.622773+0000 2023-10-25T00:22:21.000 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1508236506 2023-10-25T23:58:58.439244+0000 2023-10-25T00:22:21.000 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/780830443 2023-10-26T00:06:23.634894+0000 2023-10-25T00:22:21.000 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2150827048 2023-10-26T00:07:48.667043+0000 2023-10-25T00:22:21.001 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/913067382 2023-10-26T00:08:08.675592+0000 2023-10-25T00:22:21.001 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1317796403 2023-10-26T00:08:23.682615+0000 2023-10-25T00:22:21.001 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2276136116 2023-10-26T00:09:18.706706+0000 2023-10-25T00:22:21.001 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3232233949 2023-10-26T00:03:28.568587+0000 2023-10-25T00:22:21.001 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/727042497 2023-10-25T23:54:13.340839+0000 2023-10-25T00:22:21.002 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1788268787 2023-10-26T00:10:38.784519+0000 2023-10-25T00:22:21.002 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/3836115666 2023-10-26T00:08:58.700067+0000 2023-10-25T00:22:21.002 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1788268787 2023-10-26T00:10:38.784519+0000 2023-10-25T00:22:21.004 INFO:teuthology.orchestra.run.smithi107.stderr:listed 88 entries 2023-10-25T00:22:21.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 -- 172.21.15.107:0/217551038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f84065e90 msgr2=0x7f1f84068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f84065e90 0x7f1f84068350 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f1f88008ea0 tx=0x7f1f880083d0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 -- 172.21.15.107:0/217551038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f980bceb0 msgr2=0x7f1f98156100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f980bceb0 0x7f1f98156100 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c01cc50 tx=0x7f1f8c019150 comp rx=0 tx=0).stop 2023-10-25T00:22:21.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 -- 172.21.15.107:0/217551038 shutdown_connections 2023-10-25T00:22:21.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f980a42a0 0x7f1f980a8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f980bceb0 0x7f1f98156100 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1f84065e90 0x7f1f84068350 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 --2- 172.21.15.107:0/217551038 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f980ab1a0 0x7f1f98158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 -- 172.21.15.107:0/217551038 >> 172.21.15.107:0/217551038 conn(0x7f1f980c2860 msgr2=0x7f1f980c3990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:21.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 -- 172.21.15.107:0/217551038 shutdown_connections 2023-10-25T00:22:21.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:20.984+0000 7f1f9e130640 1 -- 172.21.15.107:0/217551038 wait complete. 2023-10-25T00:22:21.007 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-10-25T00:22:21.007 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-25T00:22:21.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-25T00:22:21.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.132+0000 7f28acba7640 1 Processor -- start 2023-10-25T00:22:21.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.132+0000 7f28acba7640 1 -- start start 2023-10-25T00:22:21.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28acba7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f28a8071220 0x7f28a806bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28acba7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a80c24c0 0x7f28a806a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28acba7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a8150bc0 con 0x7f28a8151960 2023-10-25T00:22:21.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28acba7640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a806c7f0 con 0x7f28a8071220 2023-10-25T00:22:21.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28acba7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a806c510 con 0x7f28a80c24c0 2023-10-25T00:22:21.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28a6d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a80c24c0 0x7f28a806a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28a659c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f28a8071220 0x7f28a806bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28a5d9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f28a8151960 0x7f28a8071a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:34648/0 (socket says 172.21.15.107:34648) 2023-10-25T00:22:21.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28a5d9b640 1 -- 172.21.15.107:0/1523663065 learned_addr learned my addr 172.21.15.107:0/1523663065 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:21.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28a559a640 1 -- 172.21.15.107:0/1523663065 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578862509 0 0) 0x7f28a8150bc0 con 0x7f28a8151960 2023-10-25T00:22:21.138 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28a559a640 1 -- 172.21.15.107:0/1523663065 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2894005740 con 0x7f28a8151960 2023-10-25T00:22:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28a6d9d640 1 -- 172.21.15.107:0/1523663065 >> v2:172.21.15.110:3300/0 conn(0x7f28a8071220 msgr2=0x7f28a806bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28a6d9d640 1 --2- 172.21.15.107:0/1523663065 >> v2:172.21.15.110:3300/0 conn(0x7f28a8071220 0x7f28a806bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28a6d9d640 1 -- 172.21.15.107:0/1523663065 >> v1:172.21.15.110:6789/0 conn(0x7f28a8151960 legacy=0x7f28a8071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28a6d9d640 1 -- 172.21.15.107:0/1523663065 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a8150e80 con 0x7f28a80c24c0 2023-10-25T00:22:21.139 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.136+0000 7f28a659c640 1 --2- 172.21.15.107:0/1523663065 >> v2:172.21.15.110:3300/0 conn(0x7f28a8071220 0x7f28a806bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:22:21.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a6d9d640 1 --2- 172.21.15.107:0/1523663065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a80c24c0 0x7f28a806a9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f28a000a960 tx=0x7f28a001c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79b95acd090ceb5c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:21.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a559a640 1 -- 172.21.15.107:0/1523663065 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a00a9610 con 0x7f28a80c24c0 2023-10-25T00:22:21.141 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a559a640 1 -- 172.21.15.107:0/1523663065 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a00a9770 con 0x7f28a80c24c0 2023-10-25T00:22:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a559a640 1 -- 172.21.15.107:0/1523663065 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a00a9b00 con 0x7f28a80c24c0 2023-10-25T00:22:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 -- 172.21.15.107:0/1523663065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a80c24c0 msgr2=0x7f28a806a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 --2- 172.21.15.107:0/1523663065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a80c24c0 0x7f28a806a9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f28a000a960 tx=0x7f28a001c6c0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.142 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 -- 172.21.15.107:0/1523663065 shutdown_connections 2023-10-25T00:22:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 --2- 172.21.15.107:0/1523663065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a80c24c0 0x7f28a806a9c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 --2- 172.21.15.107:0/1523663065 >> v2:172.21.15.110:3300/0 conn(0x7f28a8071220 0x7f28a806bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 -- 172.21.15.107:0/1523663065 >> 172.21.15.107:0/1523663065 conn(0x7f28a80c2950 msgr2=0x7f28a80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 -- 172.21.15.107:0/1523663065 shutdown_connections 2023-10-25T00:22:21.143 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 -- 172.21.15.107:0/1523663065 wait complete. 2023-10-25T00:22:21.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 Processor -- start 2023-10-25T00:22:21.144 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 -- start start 2023-10-25T00:22:21.146 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a8071220 0x7f28a81ecda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a80c24c0 0x7f28a80bf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28a8151960 0x7f28a81f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a81f03b0 con 0x7f28a8071220 2023-10-25T00:22:21.147 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a8147840 con 0x7f28a8151960 2023-10-25T00:22:21.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28acba7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a81f0620 con 0x7f28a80c24c0 2023-10-25T00:22:21.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a8071220 0x7f28a81ecda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a8071220 0x7f28a81ecda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51704/0 (socket says 172.21.15.107:51704) 2023-10-25T00:22:21.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a659c640 1 -- 172.21.15.107:0/3630117098 learned_addr learned my addr 172.21.15.107:0/3630117098 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:21.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a6d9d640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28a8151960 0x7f28a81f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.148 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a5d9b640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a80c24c0 0x7f28a80bf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a659c640 1 -- 172.21.15.107:0/3630117098 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a80c24c0 msgr2=0x7f28a80bf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a659c640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a80c24c0 0x7f28a80bf4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a659c640 1 -- 172.21.15.107:0/3630117098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28a8151960 msgr2=0x7f28a81f0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.149 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a659c640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28a8151960 0x7f28a81f0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.140+0000 7f28a659c640 1 -- 172.21.15.107:0/3630117098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2890009a70 con 0x7f28a8071220 2023-10-25T00:22:21.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f28a659c640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a8071220 0x7f28a81ecda0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f289000efc0 tx=0x7f289000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:21.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f289effd640 1 -- 172.21.15.107:0/3630117098 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2890014680 con 0x7f28a8071220 2023-10-25T00:22:21.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f289effd640 1 -- 172.21.15.107:0/3630117098 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289000f040 con 0x7f28a8071220 2023-10-25T00:22:21.150 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f28acba7640 1 -- 172.21.15.107:0/3630117098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a000a5c0 con 0x7f28a8071220 2023-10-25T00:22:21.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f289effd640 1 -- 172.21.15.107:0/3630117098 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2890014a00 con 0x7f28a8071220 2023-10-25T00:22:21.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f28acba7640 1 -- 172.21.15.107:0/3630117098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a8148570 con 0x7f28a8071220 2023-10-25T00:22:21.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f28acba7640 1 -- 172.21.15.107:0/3630117098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a81f07e0 con 0x7f28a8071220 2023-10-25T00:22:21.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f289effd640 1 -- 172.21.15.107:0/3630117098 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2890014b60 con 0x7f28a8071220 2023-10-25T00:22:21.151 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f289effd640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2888065e60 0x7f2888068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f289effd640 1 -- 172.21.15.107:0/3630117098 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f28900859c0 con 0x7f28a8071220 2023-10-25T00:22:21.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f28a5d9b640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2888065e60 0x7f2888068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.152 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.144+0000 7f28a5d9b640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2888065e60 0x7f2888068320 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f28a8141420 tx=0x7f2898009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:21.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.148+0000 7f289effd640 1 -- 172.21.15.107:0/3630117098 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2890051e00 con 0x7f28a8071220 2023-10-25T00:22:21.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.304+0000 7f28acba7640 1 -- 172.21.15.107:0/3630117098 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28a80746e0 con 0x7f28a8071220 2023-10-25T00:22:21.308 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.308+0000 7f289effd640 1 -- 172.21.15.107:0/3630117098 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+27395 (secure 0 0 0) 0x7f2890055420 con 0x7f28a8071220 2023-10-25T00:22:21.309 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:21.309 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":72,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-25T00:21:39.173159+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2f","mappings":[{"from":3,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":2}]},{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6835/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6837/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.110:6834/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.110:6832/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.107:6835/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.107:6837/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.110:6834/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.110:6832/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.107:6835/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.107:6834/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.110:6833/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6836/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6836/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.107:6834/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6835/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6836/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.110:6834/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6835/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.110:6835/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6834/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.107:6837/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.107:6837/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6832/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6833/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.110:6833/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6834/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.110:6835/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6836/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6836/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6834/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.110:6835/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.107:6834/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.107:6834/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.110:6833/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.107:6837/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6834/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.110:6833/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.110:6832/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.107:6836/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6837/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.107:6837/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.110:6833/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6837/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.107:6835/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.107:6837/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6837/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6835/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.110:6835/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.107:6836/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.107:6834/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.110:6832/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.107:6835/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6835/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.110:6832/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.110:6833/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6836/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6834/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6836/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.110:6834/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6837/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6836/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.110:6835/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6834/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6835/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6835/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.110:6832/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6835/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.107:6836/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6833/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6835/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.107:6836/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6834/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.107:6836/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6836/1788268787":"2023-10-26T00:10:38.784519+0000","172.21.15.110:6835/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.107:6837/1788268787":"2023-10-26T00:10:38.784519+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-25T00:22:21.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 -- 172.21.15.107:0/3630117098 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2888065e60 msgr2=0x7f2888068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2888065e60 0x7f2888068320 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f28a8141420 tx=0x7f2898009040 comp rx=0 tx=0).stop 2023-10-25T00:22:21.311 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 -- 172.21.15.107:0/3630117098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a8071220 msgr2=0x7f28a81ecda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a8071220 0x7f28a81ecda0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f289000efc0 tx=0x7f289000c490 comp rx=0 tx=0).stop 2023-10-25T00:22:21.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 -- 172.21.15.107:0/3630117098 shutdown_connections 2023-10-25T00:22:21.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f28a8151960 0x7f28a81f0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2888065e60 0x7f2888068320 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f28a80c24c0 0x7f28a80bf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 --2- 172.21.15.107:0/3630117098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f28a8071220 0x7f28a81ecda0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 -- 172.21.15.107:0/3630117098 >> 172.21.15.107:0/3630117098 conn(0x7f28a80c2950 msgr2=0x7f28a80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:21.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 -- 172.21.15.107:0/3630117098 shutdown_connections 2023-10-25T00:22:21.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.312+0000 7f28acba7640 1 -- 172.21.15.107:0/3630117098 wait complete. 2023-10-25T00:22:21.328 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-24T23:52:51.349622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-10-24T23:52:57.086491+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_metadata', 'create_time': '2023-10-24T23:53:00.990207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'a_data', 'create_time': '2023-10-24T23:53:01.642395+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_metadata', 'create_time': '2023-10-24T23:53:09.786122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'b_data', 'create_time': '2023-10-24T23:53:10.766645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-25T00:22:21.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-25T00:22:21.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa67163b640 1 Processor -- start 2023-10-25T00:22:21.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa67163b640 1 -- start start 2023-10-25T00:22:21.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa67163b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66c158320 0x7fa66c158700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa67163b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa66c0c1cc0 0x7fa66c0c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa67163b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa66c0c5230 0x7fa66c152900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa67163b640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66c1548c0 con 0x7fa66c0c1cc0 2023-10-25T00:22:21.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa67163b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66c155880 con 0x7fa66c0c5230 2023-10-25T00:22:21.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa67163b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c158da0 con 0x7fa66c158320 2023-10-25T00:22:21.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa66a7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa66c0c1cc0 0x7fa66c0c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa66a7fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa66c0c1cc0 0x7fa66c0c4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51714/0 (socket says 172.21.15.107:51714) 2023-10-25T00:22:21.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa66a7fc640 1 -- 172.21.15.107:0/2384624690 learned_addr learned my addr 172.21.15.107:0/2384624690 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:21.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa66b7fe640 1 --2- 172.21.15.107:0/2384624690 >> v2:172.21.15.110:3300/0 conn(0x7fa66c0c5230 0x7fa66c152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa66affd640 1 --2- 172.21.15.107:0/2384624690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66c158320 0x7fa66c158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa66a7fc640 1 -- 172.21.15.107:0/2384624690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66c158320 msgr2=0x7fa66c158700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:21.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa66a7fc640 1 --2- 172.21.15.107:0/2384624690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66c158320 0x7fa66c158700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa66a7fc640 1 -- 172.21.15.107:0/2384624690 >> v2:172.21.15.110:3300/0 conn(0x7fa66c0c5230 msgr2=0x7fa66c152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:21.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa66a7fc640 1 --2- 172.21.15.107:0/2384624690 >> v2:172.21.15.110:3300/0 conn(0x7fa66c0c5230 0x7fa66c152900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.452+0000 7fa66a7fc640 1 -- 172.21.15.107:0/2384624690 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c154ac0 con 0x7fa66c0c1cc0 2023-10-25T00:22:21.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa66a7fc640 1 --2- 172.21.15.107:0/2384624690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66c0c1cc0 0x7fa66c0c4cf0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fa66c06d410 tx=0x7fa65401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2b2c71f03cf564 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:21.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa669ffb640 1 -- 172.21.15.107:0/2384624690 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6540ab600 con 0x7fa66c0c1cc0 2023-10-25T00:22:21.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa669ffb640 1 -- 172.21.15.107:0/2384624690 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6540a7050 con 0x7fa66c0c1cc0 2023-10-25T00:22:21.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa669ffb640 1 -- 172.21.15.107:0/2384624690 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6540ab980 con 0x7fa66c0c1cc0 2023-10-25T00:22:21.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa67163b640 1 -- 172.21.15.107:0/2384624690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66c0c1cc0 msgr2=0x7fa66c0c4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa67163b640 1 --2- 172.21.15.107:0/2384624690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66c0c1cc0 0x7fa66c0c4cf0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fa66c06d410 tx=0x7fa65401c470 comp rx=0 tx=0).stop 2023-10-25T00:22:21.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa67163b640 1 -- 172.21.15.107:0/2384624690 shutdown_connections 2023-10-25T00:22:21.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa67163b640 1 --2- 172.21.15.107:0/2384624690 >> v2:172.21.15.110:3300/0 conn(0x7fa66c0c5230 0x7fa66c152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa67163b640 1 --2- 172.21.15.107:0/2384624690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66c0c1cc0 0x7fa66c0c4cf0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa67163b640 1 --2- 172.21.15.107:0/2384624690 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66c158320 0x7fa66c158700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa67163b640 1 -- 172.21.15.107:0/2384624690 >> 172.21.15.107:0/2384624690 conn(0x7fa66c0c2860 msgr2=0x7fa66c0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:21.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa67163b640 1 -- 172.21.15.107:0/2384624690 shutdown_connections 2023-10-25T00:22:21.459 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa67163b640 1 -- 172.21.15.107:0/2384624690 wait complete. 2023-10-25T00:22:21.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa67163b640 1 Processor -- start 2023-10-25T00:22:21.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.456+0000 7fa67163b640 1 -- start start 2023-10-25T00:22:21.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa67163b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66c0c1cc0 0x7fa66c1eccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa67163b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66c0c5230 0x7fa66c1ee6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa67163b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c158320 0x7fa66c15f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa67163b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c0aae40 con 0x7fa66c0c5230 2023-10-25T00:22:21.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa67163b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c1ebc50 con 0x7fa66c158320 2023-10-25T00:22:21.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa67163b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c1ebf30 con 0x7fa66c0c1cc0 2023-10-25T00:22:21.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66c0c5230 0x7fa66c1ee6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66c0c5230 0x7fa66c1ee6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51726/0 (socket says 172.21.15.107:51726) 2023-10-25T00:22:21.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66a7fc640 1 -- 172.21.15.107:0/1959458522 learned_addr learned my addr 172.21.15.107:0/1959458522 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:21.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66affd640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66c0c1cc0 0x7fa66c1eccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66b7fe640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c158320 0x7fa66c15f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66a7fc640 1 -- 172.21.15.107:0/1959458522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66c0c1cc0 msgr2=0x7fa66c1eccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66a7fc640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66c0c1cc0 0x7fa66c1eccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66a7fc640 1 -- 172.21.15.107:0/1959458522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c158320 msgr2=0x7fa66c15f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66a7fc640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c158320 0x7fa66c15f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66a7fc640 1 -- 172.21.15.107:0/1959458522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6540094d0 con 0x7fa66c0c5230 2023-10-25T00:22:21.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66a7fc640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66c0c5230 0x7fa66c1ee6d0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fa6540ae040 tx=0x7fa654019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:21.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa6637fe640 1 -- 172.21.15.107:0/1959458522 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6540ba070 con 0x7fa66c0c5230 2023-10-25T00:22:21.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa6637fe640 1 -- 172.21.15.107:0/1959458522 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6540abdb0 con 0x7fa66c0c5230 2023-10-25T00:22:21.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa67163b640 1 -- 172.21.15.107:0/1959458522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa66c1ec210 con 0x7fa66c0c5230 2023-10-25T00:22:21.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa6637fe640 1 -- 172.21.15.107:0/1959458522 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6540b66b0 con 0x7fa66c0c5230 2023-10-25T00:22:21.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa67163b640 1 -- 172.21.15.107:0/1959458522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa66c1ec7d0 con 0x7fa66c0c5230 2023-10-25T00:22:21.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa67163b640 1 -- 172.21.15.107:0/1959458522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66c1ec9c0 con 0x7fa66c0c5230 2023-10-25T00:22:21.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa6637fe640 1 -- 172.21.15.107:0/1959458522 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa6540b2070 con 0x7fa66c0c5230 2023-10-25T00:22:21.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa6637fe640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa64c065e60 0x7fa64c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa6637fe640 1 -- 172.21.15.107:0/1959458522 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fa65411e690 con 0x7fa66c0c5230 2023-10-25T00:22:21.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66affd640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa64c065e60 0x7fa64c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.460+0000 7fa66affd640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa64c065e60 0x7fa64c068320 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa664007330 tx=0x7fa66400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:21.467 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.464+0000 7fa6637fe640 1 -- 172.21.15.107:0/1959458522 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa6540eaad0 con 0x7fa66c0c5230 2023-10-25T00:22:21.613 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.612+0000 7fa67163b640 1 -- 172.21.15.107:0/1959458522 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa66c04e130 con 0x7fa66c0c5230 2023-10-25T00:22:21.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.612+0000 7fa6637fe640 1 -- 172.21.15.107:0/1959458522 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa6540ee0f0 con 0x7fa66c0c5230 2023-10-25T00:22:21.615 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2023-10-25T00:22:21.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 -- 172.21.15.107:0/1959458522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa64c065e60 msgr2=0x7fa64c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa64c065e60 0x7fa64c068320 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa664007330 tx=0x7fa66400b040 comp rx=0 tx=0).stop 2023-10-25T00:22:21.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 -- 172.21.15.107:0/1959458522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66c0c5230 msgr2=0x7fa66c1ee6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66c0c5230 0x7fa66c1ee6d0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fa6540ae040 tx=0x7fa654019210 comp rx=0 tx=0).stop 2023-10-25T00:22:21.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 -- 172.21.15.107:0/1959458522 shutdown_connections 2023-10-25T00:22:21.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c158320 0x7fa66c15f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa66c0c5230 0x7fa66c1ee6d0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa64c065e60 0x7fa64c068320 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 --2- 172.21.15.107:0/1959458522 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa66c0c1cc0 0x7fa66c1eccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 -- 172.21.15.107:0/1959458522 >> 172.21.15.107:0/1959458522 conn(0x7fa66c0c2860 msgr2=0x7fa66c0a6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:21.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 -- 172.21.15.107:0/1959458522 shutdown_connections 2023-10-25T00:22:21.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.616+0000 7fa67163b640 1 -- 172.21.15.107:0/1959458522 wait complete. 2023-10-25T00:22:21.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-25T00:22:21.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4649c48640 1 Processor -- start 2023-10-25T00:22:21.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4649c48640 1 -- start start 2023-10-25T00:22:21.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4649c48640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f46440af9b0 0x7f46440c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4649c48640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46440bd930 0x7f464415f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4649c48640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46440c8180 con 0x7f464406b360 2023-10-25T00:22:21.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4649c48640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46440c5870 con 0x7f46440bd930 2023-10-25T00:22:21.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4649c48640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4644155430 con 0x7f46440af9b0 2023-10-25T00:22:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4643fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46440bd930 0x7f464415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4642ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f464406b360 0x7f4644069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60714/0 (socket says 172.21.15.107:60714) 2023-10-25T00:22:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4643fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f46440bd930 0x7f464415f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51728/0 (socket says 172.21.15.107:51728) 2023-10-25T00:22:21.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4642ffd640 1 -- 172.21.15.107:0/2512530018 learned_addr learned my addr 172.21.15.107:0/2512530018 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4643fff640 1 -- 172.21.15.107:0/2512530018 >> v2:172.21.15.110:3300/0 conn(0x7f46440af9b0 msgr2=0x7f46440c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4643fff640 1 --2- 172.21.15.107:0/2512530018 >> v2:172.21.15.110:3300/0 conn(0x7f46440af9b0 0x7f46440c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4643fff640 1 -- 172.21.15.107:0/2512530018 >> v1:172.21.15.107:6789/0 conn(0x7f464406b360 legacy=0x7f4644069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f4643fff640 1 -- 172.21.15.107:0/2512530018 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4644155710 con 0x7f46440bd930 2023-10-25T00:22:21.749 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.744+0000 7f46427fc640 1 -- 172.21.15.107:0/2512530018 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569473758 0 0) 0x7f46440c8180 con 0x7f464406b360 2023-10-25T00:22:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4643fff640 1 --2- 172.21.15.107:0/2512530018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440bd930 0x7f464415f290 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f46440c7dc0 tx=0x7f463c01c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c30d214419ead00 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46427fc640 1 -- 172.21.15.107:0/2512530018 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0a58a0 con 0x7f46440bd930 2023-10-25T00:22:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46427fc640 1 -- 172.21.15.107:0/2512530018 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c0a5a00 con 0x7f46440bd930 2023-10-25T00:22:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46427fc640 1 -- 172.21.15.107:0/2512530018 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0a5d90 con 0x7f46440bd930 2023-10-25T00:22:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 -- 172.21.15.107:0/2512530018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440bd930 msgr2=0x7f464415f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 --2- 172.21.15.107:0/2512530018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440bd930 0x7f464415f290 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f46440c7dc0 tx=0x7f463c01c850 comp rx=0 tx=0).stop 2023-10-25T00:22:21.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 -- 172.21.15.107:0/2512530018 shutdown_connections 2023-10-25T00:22:21.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 --2- 172.21.15.107:0/2512530018 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440bd930 0x7f464415f290 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 --2- 172.21.15.107:0/2512530018 >> v2:172.21.15.110:3300/0 conn(0x7f46440af9b0 0x7f46440c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 -- 172.21.15.107:0/2512530018 >> 172.21.15.107:0/2512530018 conn(0x7f4644071b00 msgr2=0x7f46440abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 -- 172.21.15.107:0/2512530018 shutdown_connections 2023-10-25T00:22:21.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 -- 172.21.15.107:0/2512530018 wait complete. 2023-10-25T00:22:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 Processor -- start 2023-10-25T00:22:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 -- start start 2023-10-25T00:22:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f464406b360 0x7f464414a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f46440af9b0 0x7f4644152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46440bd930 0x7f46441ed610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46441ed180 con 0x7f464406b360 2023-10-25T00:22:21.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4644149df0 con 0x7f46440bd930 2023-10-25T00:22:21.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46441f1300 con 0x7f46440af9b0 2023-10-25T00:22:21.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f464406b360 0x7f464414a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46437fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f464406b360 0x7f464414a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51730/0 (socket says 172.21.15.107:51730) 2023-10-25T00:22:21.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46437fe640 1 -- 172.21.15.107:0/2570770121 learned_addr learned my addr 172.21.15.107:0/2570770121 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:21.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46437fe640 1 -- 172.21.15.107:0/2570770121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f46440af9b0 msgr2=0x7f4644152940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46437fe640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f46440af9b0 0x7f4644152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46437fe640 1 -- 172.21.15.107:0/2570770121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46440bd930 msgr2=0x7f46441ed610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46437fe640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46440bd930 0x7f46441ed610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46437fe640 1 -- 172.21.15.107:0/2570770121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c009a70 con 0x7f464406b360 2023-10-25T00:22:21.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f46437fe640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f464406b360 0x7f464414a250 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f463400b550 tx=0x7f463400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:21.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f462bfff640 1 -- 172.21.15.107:0/2570770121 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634010870 con 0x7f464406b360 2023-10-25T00:22:21.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f462bfff640 1 -- 172.21.15.107:0/2570770121 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46340109f0 con 0x7f464406b360 2023-10-25T00:22:21.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 -- 172.21.15.107:0/2570770121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4644149830 con 0x7f464406b360 2023-10-25T00:22:21.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f462bfff640 1 -- 172.21.15.107:0/2570770121 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4634010da0 con 0x7f464406b360 2023-10-25T00:22:21.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.748+0000 7f4649c48640 1 -- 172.21.15.107:0/2570770121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4644152e80 con 0x7f464406b360 2023-10-25T00:22:21.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.752+0000 7f4649c48640 1 -- 172.21.15.107:0/2570770121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4644066560 con 0x7f464406b360 2023-10-25T00:22:21.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.752+0000 7f462bfff640 1 -- 172.21.15.107:0/2570770121 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f463400f3c0 con 0x7f464406b360 2023-10-25T00:22:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.752+0000 7f462bfff640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4630065e60 0x7f4630068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.752+0000 7f462bfff640 1 -- 172.21.15.107:0/2570770121 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f4634084b00 con 0x7f464406b360 2023-10-25T00:22:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.752+0000 7f4642ffd640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4630065e60 0x7f4630068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:21.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.752+0000 7f4642ffd640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4630065e60 0x7f4630068320 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f46380011f0 tx=0x7f4638007930 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:21.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.756+0000 7f462bfff640 1 -- 172.21.15.107:0/2570770121 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4634050eb0 con 0x7f464406b360 2023-10-25T00:22:21.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 -- 172.21.15.107:0/2570770121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f464404e130 con 0x7f464406b360 2023-10-25T00:22:21.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f462bfff640 1 -- 172.21.15.107:0/2570770121 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v72) v1 ==== 92+0+10 (secure 0 0 0) 0x7f463401e0a0 con 0x7f464406b360 2023-10-25T00:22:21.900 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2023-10-25T00:22:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 -- 172.21.15.107:0/2570770121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4630065e60 msgr2=0x7f4630068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4630065e60 0x7f4630068320 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f46380011f0 tx=0x7f4638007930 comp rx=0 tx=0).stop 2023-10-25T00:22:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 -- 172.21.15.107:0/2570770121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f464406b360 msgr2=0x7f464414a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:21.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f464406b360 0x7f464414a250 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f463400b550 tx=0x7f463400ba10 comp rx=0 tx=0).stop 2023-10-25T00:22:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 -- 172.21.15.107:0/2570770121 shutdown_connections 2023-10-25T00:22:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46440bd930 0x7f46441ed610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4630065e60 0x7f4630068320 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f46440af9b0 0x7f4644152940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 --2- 172.21.15.107:0/2570770121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f464406b360 0x7f464414a250 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 -- 172.21.15.107:0/2570770121 >> 172.21.15.107:0/2570770121 conn(0x7f4644071b00 msgr2=0x7f46440ad0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 -- 172.21.15.107:0/2570770121 shutdown_connections 2023-10-25T00:22:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:21.900+0000 7f4649c48640 1 -- 172.21.15.107:0/2570770121 wait complete. 2023-10-25T00:22:21.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2023-10-25T00:22:22.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.040+0000 7f668e4c3640 1 Processor -- start 2023-10-25T00:22:22.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.040+0000 7f668e4c3640 1 -- start start 2023-10-25T00:22:22.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f668e4c3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f66880ab1a0 0x7f66880c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f668e4c3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66880bceb0 0x7f668815c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f668e4c3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66880c1ce0 con 0x7f6688072720 2023-10-25T00:22:22.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f668e4c3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66880be1d0 con 0x7f66880bceb0 2023-10-25T00:22:22.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f668e4c3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66880c0240 con 0x7f66880ab1a0 2023-10-25T00:22:22.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f668ca60640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f66880bceb0 0x7f668815c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f66877fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6688072720 0x7f66880bfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60720/0 (socket says 172.21.15.107:60720) 2023-10-25T00:22:22.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f66877fe640 1 -- 172.21.15.107:0/1344234774 learned_addr learned my addr 172.21.15.107:0/1344234774 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:22.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f6687fff640 1 --2- 172.21.15.107:0/1344234774 >> v2:172.21.15.110:3300/0 conn(0x7f66880ab1a0 0x7f66880c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f668ca60640 1 -- 172.21.15.107:0/1344234774 >> v2:172.21.15.110:3300/0 conn(0x7f66880ab1a0 msgr2=0x7f66880c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:22.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f668ca60640 1 --2- 172.21.15.107:0/1344234774 >> v2:172.21.15.110:3300/0 conn(0x7f66880ab1a0 0x7f66880c0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f668ca60640 1 -- 172.21.15.107:0/1344234774 >> v1:172.21.15.107:6789/0 conn(0x7f6688072720 legacy=0x7f66880bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f668ca60640 1 -- 172.21.15.107:0/1344234774 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66880c0880 con 0x7f66880bceb0 2023-10-25T00:22:22.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f6686ffd640 1 -- 172.21.15.107:0/1344234774 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043011415 0 0) 0x7f66880c1ce0 con 0x7f6688072720 2023-10-25T00:22:22.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f668ca60640 1 --2- 172.21.15.107:0/1344234774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66880bceb0 0x7f668815c660 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f66880c3960 tx=0x7f668001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57df6734b89d3c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f6686ffd640 1 -- 172.21.15.107:0/1344234774 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800a5910 con 0x7f66880bceb0 2023-10-25T00:22:22.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f6686ffd640 1 -- 172.21.15.107:0/1344234774 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66800a5a70 con 0x7f66880bceb0 2023-10-25T00:22:22.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.044+0000 7f6686ffd640 1 -- 172.21.15.107:0/1344234774 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800a5e00 con 0x7f66880bceb0 2023-10-25T00:22:22.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 -- 172.21.15.107:0/1344234774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66880bceb0 msgr2=0x7f668815c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 --2- 172.21.15.107:0/1344234774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66880bceb0 0x7f668815c660 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f66880c3960 tx=0x7f668001c850 comp rx=0 tx=0).stop 2023-10-25T00:22:22.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 -- 172.21.15.107:0/1344234774 shutdown_connections 2023-10-25T00:22:22.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 --2- 172.21.15.107:0/1344234774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66880bceb0 0x7f668815c660 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 --2- 172.21.15.107:0/1344234774 >> v2:172.21.15.110:3300/0 conn(0x7f66880ab1a0 0x7f66880c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 -- 172.21.15.107:0/1344234774 >> 172.21.15.107:0/1344234774 conn(0x7f66880c2860 msgr2=0x7f66880c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:22.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 -- 172.21.15.107:0/1344234774 shutdown_connections 2023-10-25T00:22:22.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 -- 172.21.15.107:0/1344234774 wait complete. 2023-10-25T00:22:22.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 Processor -- start 2023-10-25T00:22:22.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 -- start start 2023-10-25T00:22:22.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66880ab1a0 0x7f66881ee050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66880bceb0 0x7f6688148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66881ee590 0x7f66881492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6688157700 con 0x7f66880bceb0 2023-10-25T00:22:22.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66881533a0 con 0x7f66881ee590 2023-10-25T00:22:22.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66880c3990 con 0x7f66880ab1a0 2023-10-25T00:22:22.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f66877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66880bceb0 0x7f6688148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f66877fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66880bceb0 0x7f6688148d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51758/0 (socket says 172.21.15.107:51758) 2023-10-25T00:22:22.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f66877fe640 1 -- 172.21.15.107:0/1532422372 learned_addr learned my addr 172.21.15.107:0/1532422372 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:22.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668ca60640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66881ee590 0x7f66881492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f66877fe640 1 -- 172.21.15.107:0/1532422372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66880ab1a0 msgr2=0x7f66881ee050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.057 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f66877fe640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66880ab1a0 0x7f66881ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f66877fe640 1 -- 172.21.15.107:0/1532422372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66881ee590 msgr2=0x7f66881492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f66877fe640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66881ee590 0x7f66881492a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f66877fe640 1 -- 172.21.15.107:0/1532422372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6680009a70 con 0x7f66880bceb0 2023-10-25T00:22:22.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f66877fe640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66880bceb0 0x7f6688148d60 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f667c00c520 tx=0x7f667c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f6684ff9640 1 -- 172.21.15.107:0/1532422372 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c007560 con 0x7f66880bceb0 2023-10-25T00:22:22.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f6684ff9640 1 -- 172.21.15.107:0/1532422372 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667c0047f0 con 0x7f66880bceb0 2023-10-25T00:22:22.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 -- 172.21.15.107:0/1532422372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668814f2c0 con 0x7f66880bceb0 2023-10-25T00:22:22.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f6684ff9640 1 -- 172.21.15.107:0/1532422372 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c004ba0 con 0x7f66880bceb0 2023-10-25T00:22:22.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.048+0000 7f668e4c3640 1 -- 172.21.15.107:0/1532422372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6688154ee0 con 0x7f66880bceb0 2023-10-25T00:22:22.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.052+0000 7f668e4c3640 1 -- 172.21.15.107:0/1532422372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66880beff0 con 0x7f66880bceb0 2023-10-25T00:22:22.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.052+0000 7f6684ff9640 1 -- 172.21.15.107:0/1532422372 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f667c004d00 con 0x7f66880bceb0 2023-10-25T00:22:22.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.052+0000 7f6684ff9640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6674065e60 0x7f6674068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.052+0000 7f6684ff9640 1 -- 172.21.15.107:0/1532422372 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f667c083d20 con 0x7f66880bceb0 2023-10-25T00:22:22.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.052+0000 7f6687fff640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6674065e60 0x7f6674068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.052+0000 7f6687fff640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6674065e60 0x7f6674068320 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f667800a8b0 tx=0x7f6678008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.056+0000 7f6684ff9640 1 -- 172.21.15.107:0/1532422372 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f667c050160 con 0x7f66880bceb0 2023-10-25T00:22:22.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 -- 172.21.15.107:0/1532422372 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) v1 -- 0x7f66881534e0 con 0x7f66880bceb0 2023-10-25T00:22:22.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f6684ff9640 1 -- 172.21.15.107:0/1532422372 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v72) v1 ==== 99+0+11 (secure 0 0 0) 0x7f667c018030 con 0x7f66880bceb0 2023-10-25T00:22:22.200 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-25T00:22:22.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 -- 172.21.15.107:0/1532422372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6674065e60 msgr2=0x7f6674068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6674065e60 0x7f6674068320 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f667800a8b0 tx=0x7f6678008040 comp rx=0 tx=0).stop 2023-10-25T00:22:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 -- 172.21.15.107:0/1532422372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66880bceb0 msgr2=0x7f6688148d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66880bceb0 0x7f6688148d60 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f667c00c520 tx=0x7f667c00c9e0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 -- 172.21.15.107:0/1532422372 shutdown_connections 2023-10-25T00:22:22.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66881ee590 0x7f66881492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66880bceb0 0x7f6688148d60 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6674065e60 0x7f6674068320 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 --2- 172.21.15.107:0/1532422372 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66880ab1a0 0x7f66881ee050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 -- 172.21.15.107:0/1532422372 >> 172.21.15.107:0/1532422372 conn(0x7f66880c2860 msgr2=0x7f66881552b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:22.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 -- 172.21.15.107:0/1532422372 shutdown_connections 2023-10-25T00:22:22.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.200+0000 7f668e4c3640 1 -- 172.21.15.107:0/1532422372 wait complete. 2023-10-25T00:22:22.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2023-10-25T00:22:22.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f62a1eee640 1 Processor -- start 2023-10-25T00:22:22.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f62a1eee640 1 -- start start 2023-10-25T00:22:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f62a1eee640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c0abf70 0x7f629c06adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f62a1eee640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f629c0ac380 con 0x7f629c14c6b0 2023-10-25T00:22:22.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f62a1eee640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f629c069d40 con 0x7f629c1526d0 2023-10-25T00:22:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f62a1eee640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629c0a9b60 con 0x7f629c0abf70 2023-10-25T00:22:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629b7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f629c14c6b0 0x7f629c0a9450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:60730/0 (socket says 172.21.15.107:60730) 2023-10-25T00:22:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629b7fe640 1 -- 172.21.15.107:0/1577519659 learned_addr learned my addr 172.21.15.107:0/1577519659 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629bfff640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c0abf70 0x7f629c06adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147069137 0 0) 0x7f629c0ac380 con 0x7f629c14c6b0 2023-10-25T00:22:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6284003530 con 0x7f629c14c6b0 2023-10-25T00:22:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3931370658 0 0) 0x7f6284003530 con 0x7f629c14c6b0 2023-10-25T00:22:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c0abf70 msgr2=0x7f629c06adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c0abf70 0x7f629c06adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 >> v1:172.21.15.110:6789/0 conn(0x7f629c1526d0 legacy=0x7f629c0ab860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629c06a000 con 0x7f629c14c6b0 2023-10-25T00:22:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f628c001680 con 0x7f629c14c6b0 2023-10-25T00:22:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 >> v1:172.21.15.107:6789/0 conn(0x7f629c14c6b0 legacy=0x7f629c0a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62840039b0 0x7f6284003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62840047d0 0x7f6284004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c0abf70 0x7f62840051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629c0a9b60 con 0x7f62840047d0 2023-10-25T00:22:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628400bac0 con 0x7f62840039b0 2023-10-25T00:22:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628400bc80 con 0x7f629c0abf70 2023-10-25T00:22:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629affd640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62840047d0 0x7f6284004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629b7fe640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62840039b0 0x7f6284003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629bfff640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c0abf70 0x7f62840051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629affd640 1 -- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c0abf70 msgr2=0x7f62840051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:22.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629affd640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c0abf70 0x7f62840051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629affd640 1 -- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62840039b0 msgr2=0x7f6284003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:22.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629affd640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62840039b0 0x7f6284003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629affd640 1 -- 172.21.15.107:0/1577519659 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6284003530 con 0x7f62840047d0 2023-10-25T00:22:22.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629bfff640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c0abf70 0x7f62840051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:22.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.340+0000 7f629b7fe640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62840039b0 0x7f6284003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629affd640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62840047d0 0x7f6284004c50 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f6290001780 tx=0x7f629001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=759f802c9eb39342 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62900052d0 con 0x7f62840047d0 2023-10-25T00:22:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6290005450 con 0x7f62840047d0 2023-10-25T00:22:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629a7fc640 1 -- 172.21.15.107:0/1577519659 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290003840 con 0x7f62840047d0 2023-10-25T00:22:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 -- 172.21.15.107:0/1577519659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62840047d0 msgr2=0x7f6284004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 --2- 172.21.15.107:0/1577519659 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62840047d0 0x7f6284004c50 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f6290001780 tx=0x7f629001c3d0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629bfff640 1 -- 172.21.15.107:0/1577519659 reap_dead start 2023-10-25T00:22:22.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 -- 172.21.15.107:0/1577519659 shutdown_connections 2023-10-25T00:22:22.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 -- 172.21.15.107:0/1577519659 >> 172.21.15.107:0/1577519659 conn(0x7f629c0c4e30 msgr2=0x7f629c0c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:22.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 -- 172.21.15.107:0/1577519659 shutdown_connections 2023-10-25T00:22:22.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 -- 172.21.15.107:0/1577519659 wait complete. 2023-10-25T00:22:22.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 Processor -- start 2023-10-25T00:22:22.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 -- start start 2023-10-25T00:22:22.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62840047d0 0x7f629c148500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c142880 0x7f629c14b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0c04b0 0x7f629c1ecdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629c143060 con 0x7f629c0c04b0 2023-10-25T00:22:22.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629c14a930 con 0x7f62840047d0 2023-10-25T00:22:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629c144270 con 0x7f629c142880 2023-10-25T00:22:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0c04b0 0x7f629c1ecdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0c04b0 0x7f629c1ecdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:51780/0 (socket says 172.21.15.107:51780) 2023-10-25T00:22:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629bfff640 1 -- 172.21.15.107:0/3498722114 learned_addr learned my addr 172.21.15.107:0/3498722114 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:22.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629b7fe640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62840047d0 0x7f629c148500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629affd640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c142880 0x7f629c14b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629bfff640 1 -- 172.21.15.107:0/3498722114 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c142880 msgr2=0x7f629c14b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629bfff640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c142880 0x7f629c14b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629bfff640 1 -- 172.21.15.107:0/3498722114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62840047d0 msgr2=0x7f629c148500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629bfff640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62840047d0 0x7f629c148500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629bfff640 1 -- 172.21.15.107:0/3498722114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629c1f0f10 con 0x7f629c0c04b0 2023-10-25T00:22:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f629bfff640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0c04b0 0x7f629c1ecdf0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f6284004750 tx=0x7f629400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f6283fff640 1 -- 172.21.15.107:0/3498722114 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629400fe60 con 0x7f629c0c04b0 2023-10-25T00:22:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f6283fff640 1 -- 172.21.15.107:0/3498722114 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6294011040 con 0x7f629c0c04b0 2023-10-25T00:22:22.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f62a1eee640 1 -- 172.21.15.107:0/3498722114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629c144510 con 0x7f629c0c04b0 2023-10-25T00:22:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.344+0000 7f6283fff640 1 -- 172.21.15.107:0/3498722114 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6294017600 con 0x7f629c0c04b0 2023-10-25T00:22:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.348+0000 7f62a1eee640 1 -- 172.21.15.107:0/3498722114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629c0c0960 con 0x7f629c0c04b0 2023-10-25T00:22:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.348+0000 7f62a1eee640 1 -- 172.21.15.107:0/3498722114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629c073dc0 con 0x7f629c0c04b0 2023-10-25T00:22:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.348+0000 7f6283fff640 1 -- 172.21.15.107:0/3498722114 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f62940203b0 con 0x7f629c0c04b0 2023-10-25T00:22:22.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.348+0000 7f6283fff640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6288065e90 0x7f6288068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.348+0000 7f6283fff640 1 -- 172.21.15.107:0/3498722114 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f6294084d50 con 0x7f629c0c04b0 2023-10-25T00:22:22.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.348+0000 7f629b7fe640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6288065e90 0x7f6288068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.348+0000 7f629b7fe640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6288065e90 0x7f6288068350 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f628c0018f0 tx=0x7f628c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.352+0000 7f6283fff640 1 -- 172.21.15.107:0/3498722114 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6294088030 con 0x7f629c0c04b0 2023-10-25T00:22:22.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.496+0000 7f62a1eee640 1 -- 172.21.15.107:0/3498722114 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) v1 -- 0x7f629c04e130 con 0x7f629c0c04b0 2023-10-25T00:22:22.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f6283fff640 1 -- 172.21.15.107:0/3498722114 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v72) v1 ==== 95+0+11 (secure 0 0 0) 0x7f629401c020 con 0x7f629c0c04b0 2023-10-25T00:22:22.499 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-25T00:22:22.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 -- 172.21.15.107:0/3498722114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6288065e90 msgr2=0x7f6288068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6288065e90 0x7f6288068350 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f628c0018f0 tx=0x7f628c009040 comp rx=0 tx=0).stop 2023-10-25T00:22:22.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 -- 172.21.15.107:0/3498722114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0c04b0 msgr2=0x7f629c1ecdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0c04b0 0x7f629c1ecdf0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f6284004750 tx=0x7f629400f4a0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 -- 172.21.15.107:0/3498722114 shutdown_connections 2023-10-25T00:22:22.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0c04b0 0x7f629c1ecdf0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6288065e90 0x7f6288068350 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f629c142880 0x7f629c14b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 --2- 172.21.15.107:0/3498722114 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62840047d0 0x7f629c148500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 -- 172.21.15.107:0/3498722114 >> 172.21.15.107:0/3498722114 conn(0x7f629c0c4e30 msgr2=0x7f629c145ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:22.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 -- 172.21.15.107:0/3498722114 shutdown_connections 2023-10-25T00:22:22.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.500+0000 7f62a1eee640 1 -- 172.21.15.107:0/3498722114 wait complete. 2023-10-25T00:22:22.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2023-10-25T00:22:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.628+0000 7ff3f2904640 1 Processor -- start 2023-10-25T00:22:22.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.628+0000 7ff3f2904640 1 -- start start 2023-10-25T00:22:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f2904640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff3ec0ab1a0 0x7ff3ec0c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f2904640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0c1ce0 con 0x7ff3ec072720 2023-10-25T00:22:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f2904640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0c0240 con 0x7ff3ec0bceb0 2023-10-25T00:22:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f2904640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3ec0be1d0 con 0x7ff3ec0ab1a0 2023-10-25T00:22:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eb7fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff3ec072720 0x7ff3ec0bfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36876/0 (socket says 172.21.15.107:36876) 2023-10-25T00:22:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eb7fe640 1 -- 172.21.15.107:0/3487489290 learned_addr learned my addr 172.21.15.107:0/3487489290 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3ebfff640 1 --2- 172.21.15.107:0/3487489290 >> v2:172.21.15.110:3300/0 conn(0x7ff3ec0ab1a0 0x7ff3ec0c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174725733 0 0) 0x7ff3ec0c1ce0 con 0x7ff3ec072720 2023-10-25T00:22:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d4003530 con 0x7ff3ec072720 2023-10-25T00:22:22.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3894023057 0 0) 0x7ff3d4003530 con 0x7ff3ec072720 2023-10-25T00:22:22.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 >> v2:172.21.15.110:3300/0 conn(0x7ff3ec0ab1a0 msgr2=0x7ff3ec0c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:22.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 --2- 172.21.15.107:0/3487489290 >> v2:172.21.15.110:3300/0 conn(0x7ff3ec0ab1a0 0x7ff3ec0c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 >> v1:172.21.15.110:6789/0 conn(0x7ff3ec0bceb0 legacy=0x7ff3ec15c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0c0880 con 0x7ff3ec072720 2023-10-25T00:22:22.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7ff3dc001a00 con 0x7ff3ec072720 2023-10-25T00:22:22.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 >> v1:172.21.15.107:6789/0 conn(0x7ff3ec072720 legacy=0x7ff3ec0bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3d4003980 0x7ff3d4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3d40047d0 0x7ff3d4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d40052b0 0x7ff3d400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d400c150 con 0x7ff3d40052b0 2023-10-25T00:22:22.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d400c2c0 con 0x7ff3d4003980 2023-10-25T00:22:22.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d400c480 con 0x7ff3d40047d0 2023-10-25T00:22:22.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f0ea1640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d40052b0 0x7ff3d400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3ebfff640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3d4003980 0x7ff3d4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eb7fe640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3d40047d0 0x7ff3d4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f0ea1640 1 -- 172.21.15.107:0/3487489290 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3d40047d0 msgr2=0x7ff3d4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:22.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f0ea1640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3d40047d0 0x7ff3d4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f0ea1640 1 -- 172.21.15.107:0/3487489290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3d4003980 msgr2=0x7ff3d4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:22.638 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f0ea1640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3d4003980 0x7ff3d4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f0ea1640 1 -- 172.21.15.107:0/3487489290 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d4003530 con 0x7ff3d40052b0 2023-10-25T00:22:22.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3eb7fe640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3d40047d0 0x7ff3d4004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:22.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3ebfff640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3d4003980 0x7ff3d4003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:22:22.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f0ea1640 1 -- 172.21.15.107:0/3487489290 reap_dead start 2023-10-25T00:22:22.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.632+0000 7ff3f0ea1640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d40052b0 0x7ff3d400b990 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec0abe40 tx=0x7ff3e001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f455900ce9761785 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e00052d0 con 0x7ff3d40052b0 2023-10-25T00:22:22.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e00a7050 con 0x7ff3d40052b0 2023-10-25T00:22:22.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eaffd640 1 -- 172.21.15.107:0/3487489290 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0003600 con 0x7ff3d40052b0 2023-10-25T00:22:22.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 -- 172.21.15.107:0/3487489290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d40052b0 msgr2=0x7ff3d400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d40052b0 0x7ff3d400b990 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec0abe40 tx=0x7ff3e001c3a0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 -- 172.21.15.107:0/3487489290 shutdown_connections 2023-10-25T00:22:22.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 --2- 172.21.15.107:0/3487489290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3d40052b0 0x7ff3d400b990 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 -- 172.21.15.107:0/3487489290 >> 172.21.15.107:0/3487489290 conn(0x7ff3ec0c2860 msgr2=0x7ff3ec0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:22.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 -- 172.21.15.107:0/3487489290 shutdown_connections 2023-10-25T00:22:22.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 -- 172.21.15.107:0/3487489290 wait complete. 2023-10-25T00:22:22.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 Processor -- start 2023-10-25T00:22:22.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 -- start start 2023-10-25T00:22:22.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3d40052b0 0x7ff3ec1d9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ec1d7ae0 0x7ff3ec1d6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3ec1d8030 0x7ff3ec1d72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec06ca30 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0a3d60 con 0x7ff3ec1d8030 2023-10-25T00:22:22.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0a4040 con 0x7ff3d40052b0 2023-10-25T00:22:22.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ec1d7ae0 0x7ff3ec1d6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ec1d7ae0 0x7ff3ec1d6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54040/0 (socket says 172.21.15.107:54040) 2023-10-25T00:22:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eb7fe640 1 -- 172.21.15.107:0/2317319872 learned_addr learned my addr 172.21.15.107:0/2317319872 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3ebfff640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3d40052b0 0x7ff3ec1d9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f0ea1640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3ec1d8030 0x7ff3ec1d72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eb7fe640 1 -- 172.21.15.107:0/2317319872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3d40052b0 msgr2=0x7ff3ec1d9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eb7fe640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3d40052b0 0x7ff3ec1d9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eb7fe640 1 -- 172.21.15.107:0/2317319872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3ec1d8030 msgr2=0x7ff3ec1d72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eb7fe640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3ec1d8030 0x7ff3ec1d72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eb7fe640 1 -- 172.21.15.107:0/2317319872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0a4320 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f0ea1640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3ec1d8030 0x7ff3ec1d72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:22.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3eb7fe640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ec1d7ae0 0x7ff3ec1d6d90 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc004c00 tx=0x7ff3dc010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3e8ff9640 1 -- 172.21.15.107:0/2317319872 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc00ec40 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3e8ff9640 1 -- 172.21.15.107:0/2317319872 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc00eda0 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 -- 172.21.15.107:0/2317319872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec0a4600 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3e8ff9640 1 -- 172.21.15.107:0/2317319872 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc0145a0 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.636+0000 7ff3f2904640 1 -- 172.21.15.107:0/2317319872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec1da380 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.640+0000 7ff3f2904640 1 -- 172.21.15.107:0/2317319872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec04e130 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.640+0000 7ff3e8ff9640 1 -- 172.21.15.107:0/2317319872 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff3dc014700 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.640+0000 7ff3e8ff9640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff3d8065e90 0x7ff3d8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.640+0000 7ff3e8ff9640 1 -- 172.21.15.107:0/2317319872 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7ff3dc087680 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.650 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.640+0000 7ff3ebfff640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff3d8065e90 0x7ff3d8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.640+0000 7ff3ebfff640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff3d8065e90 0x7ff3d8068350 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400d9b0 tx=0x7ff3e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.644+0000 7ff3e8ff9640 1 -- 172.21.15.107:0/2317319872 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff3dc0539a0 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 -- 172.21.15.107:0/2317319872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff3ec0bf630 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3e8ff9640 1 -- 172.21.15.107:0/2317319872 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v72) v1 ==== 99+0+11 (secure 0 0 0) 0x7ff3dc01b070 con 0x7ff3ec1d7ae0 2023-10-25T00:22:22.788 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2023-10-25T00:22:22.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 -- 172.21.15.107:0/2317319872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff3d8065e90 msgr2=0x7ff3d8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff3d8065e90 0x7ff3d8068350 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ff3e400d9b0 tx=0x7ff3e4008040 comp rx=0 tx=0).stop 2023-10-25T00:22:22.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 -- 172.21.15.107:0/2317319872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ec1d7ae0 msgr2=0x7ff3ec1d6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ec1d7ae0 0x7ff3ec1d6d90 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc004c00 tx=0x7ff3dc010ef0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 -- 172.21.15.107:0/2317319872 shutdown_connections 2023-10-25T00:22:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3ec1d8030 0x7ff3ec1d72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff3ec1d7ae0 0x7ff3ec1d6d90 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff3d8065e90 0x7ff3d8068350 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 --2- 172.21.15.107:0/2317319872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff3d40052b0 0x7ff3ec1d9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 -- 172.21.15.107:0/2317319872 >> 172.21.15.107:0/2317319872 conn(0x7ff3ec0c2860 msgr2=0x7ff3ec0a63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 -- 172.21.15.107:0/2317319872 shutdown_connections 2023-10-25T00:22:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.788+0000 7ff3f2904640 1 -- 172.21.15.107:0/2317319872 wait complete. 2023-10-25T00:22:22.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2023-10-25T00:22:22.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f602931c640 1 Processor -- start 2023-10-25T00:22:22.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f602931c640 1 -- start start 2023-10-25T00:22:22.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f602931c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f60240a4f20 0x7f60240bdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f602931c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f60240c1770 0x7f60240c1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f602931c640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60240abdc0 con 0x7f60240a90c0 2023-10-25T00:22:22.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f602931c640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60240a8c30 con 0x7f60240c1770 2023-10-25T00:22:22.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f602931c640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60240bef60 con 0x7f60240a4f20 2023-10-25T00:22:22.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f60237fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f60240c1770 0x7f60240c1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f60237fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f60240c1770 0x7f60240c1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54054/0 (socket says 172.21.15.107:54054) 2023-10-25T00:22:22.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f60237fe640 1 -- 172.21.15.107:0/4102615702 learned_addr learned my addr 172.21.15.107:0/4102615702 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:22.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f6022ffd640 1 --2- 172.21.15.107:0/4102615702 >> v2:172.21.15.110:3300/0 conn(0x7f60240a4f20 0x7f60240bdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f60237fe640 1 -- 172.21.15.107:0/4102615702 >> v2:172.21.15.110:3300/0 conn(0x7f60240a4f20 msgr2=0x7f60240bdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:22.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f60237fe640 1 --2- 172.21.15.107:0/4102615702 >> v2:172.21.15.110:3300/0 conn(0x7f60240a4f20 0x7f60240bdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f60237fe640 1 -- 172.21.15.107:0/4102615702 >> v1:172.21.15.110:6789/0 conn(0x7f60240a90c0 legacy=0x7f60240a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:22.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f60237fe640 1 -- 172.21.15.107:0/4102615702 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60240bf240 con 0x7f60240c1770 2023-10-25T00:22:22.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.932+0000 7f60237fe640 1 --2- 172.21.15.107:0/4102615702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60240c1770 0x7f60240c1be0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f60240c27c0 tx=0x7f601c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a93812a69cf8d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f6021ffb640 1 -- 172.21.15.107:0/4102615702 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c0a5690 con 0x7f60240c1770 2023-10-25T00:22:22.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f6021ffb640 1 -- 172.21.15.107:0/4102615702 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c0a57f0 con 0x7f60240c1770 2023-10-25T00:22:22.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f6021ffb640 1 -- 172.21.15.107:0/4102615702 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c0a5b80 con 0x7f60240c1770 2023-10-25T00:22:22.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 -- 172.21.15.107:0/4102615702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60240c1770 msgr2=0x7f60240c1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 --2- 172.21.15.107:0/4102615702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60240c1770 0x7f60240c1be0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f60240c27c0 tx=0x7f601c01c960 comp rx=0 tx=0).stop 2023-10-25T00:22:22.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 -- 172.21.15.107:0/4102615702 shutdown_connections 2023-10-25T00:22:22.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 --2- 172.21.15.107:0/4102615702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60240c1770 0x7f60240c1be0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 --2- 172.21.15.107:0/4102615702 >> v2:172.21.15.110:3300/0 conn(0x7f60240a4f20 0x7f60240bdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 -- 172.21.15.107:0/4102615702 >> 172.21.15.107:0/4102615702 conn(0x7f60240c2860 msgr2=0x7f60240c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:22.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 -- 172.21.15.107:0/4102615702 shutdown_connections 2023-10-25T00:22:22.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 -- 172.21.15.107:0/4102615702 wait complete. 2023-10-25T00:22:22.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 Processor -- start 2023-10-25T00:22:22.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 -- start start 2023-10-25T00:22:22.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60240a4f20 0x7f6024154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60240a90c0 0x7f602414fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60240c1770 0x7f6024150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60240bdf90 con 0x7f60240c1770 2023-10-25T00:22:22.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6024071530 con 0x7f60240a90c0 2023-10-25T00:22:22.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60241eb690 con 0x7f60240a4f20 2023-10-25T00:22:22.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f60237fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60240c1770 0x7f6024150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f60237fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60240c1770 0x7f6024150280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54070/0 (socket says 172.21.15.107:54070) 2023-10-25T00:22:22.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f60237fe640 1 -- 172.21.15.107:0/3708314993 learned_addr learned my addr 172.21.15.107:0/3708314993 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:22.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f6022ffd640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60240a4f20 0x7f6024154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f60227fc640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60240a90c0 0x7f602414fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f60237fe640 1 -- 172.21.15.107:0/3708314993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60240a4f20 msgr2=0x7f6024154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f60237fe640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60240a4f20 0x7f6024154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f60237fe640 1 -- 172.21.15.107:0/3708314993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60240a90c0 msgr2=0x7f602414fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:22.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f60237fe640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60240a90c0 0x7f602414fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:22.944 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f60237fe640 1 -- 172.21.15.107:0/3708314993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601c009b30 con 0x7f60240c1770 2023-10-25T00:22:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f60227fc640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60240a90c0 0x7f602414fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f60237fe640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60240c1770 0x7f6024150280 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f601c01a590 tx=0x7f601c019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f600b7fe640 1 -- 172.21.15.107:0/3708314993 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c0b8640 con 0x7f60240c1770 2023-10-25T00:22:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f600b7fe640 1 -- 172.21.15.107:0/3708314993 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c0b87a0 con 0x7f60240c1770 2023-10-25T00:22:22.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 -- 172.21.15.107:0/3708314993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60241eb970 con 0x7f60240c1770 2023-10-25T00:22:22.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f600b7fe640 1 -- 172.21.15.107:0/3708314993 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c0b8b10 con 0x7f60240c1770 2023-10-25T00:22:22.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.936+0000 7f602931c640 1 -- 172.21.15.107:0/3708314993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60240711c0 con 0x7f60240c1770 2023-10-25T00:22:22.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.940+0000 7f602931c640 1 -- 172.21.15.107:0/3708314993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602406fbc0 con 0x7f60240c1770 2023-10-25T00:22:22.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.940+0000 7f600b7fe640 1 -- 172.21.15.107:0/3708314993 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f601c0b8c70 con 0x7f60240c1770 2023-10-25T00:22:22.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.940+0000 7f600b7fe640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6010065e90 0x7f6010068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:22.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.940+0000 7f600b7fe640 1 -- 172.21.15.107:0/3708314993 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f601c11f9e0 con 0x7f60240c1770 2023-10-25T00:22:22.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.940+0000 7f6022ffd640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6010065e90 0x7f6010068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:22.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.940+0000 7f6022ffd640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6010065e90 0x7f6010068350 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f60240a5a60 tx=0x7f6014008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:22.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:22.944+0000 7f600b7fe640 1 -- 172.21.15.107:0/3708314993 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f602406fbc0 con 0x7f60240c1770 2023-10-25T00:22:23.093 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.092+0000 7f602931c640 1 -- 172.21.15.107:0/3708314993 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) v1 -- 0x7f602406fbc0 con 0x7f60240c1770 2023-10-25T00:22:23.094 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.092+0000 7f600b7fe640 1 -- 172.21.15.107:0/3708314993 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v72) v1 ==== 95+0+11 (secure 0 0 0) 0x7f602406fbc0 con 0x7f60240c1770 2023-10-25T00:22:23.095 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2023-10-25T00:22:23.096 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 -- 172.21.15.107:0/3708314993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6010065e90 msgr2=0x7f6010068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:23.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6010065e90 0x7f6010068350 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f60240a5a60 tx=0x7f6014008040 comp rx=0 tx=0).stop 2023-10-25T00:22:23.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 -- 172.21.15.107:0/3708314993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60240c1770 msgr2=0x7f6024150280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:23.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60240c1770 0x7f6024150280 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f601c01a590 tx=0x7f601c019700 comp rx=0 tx=0).stop 2023-10-25T00:22:23.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 -- 172.21.15.107:0/3708314993 shutdown_connections 2023-10-25T00:22:23.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60240c1770 0x7f6024150280 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60240a90c0 0x7f602414fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6010065e90 0x7f6010068350 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 --2- 172.21.15.107:0/3708314993 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60240a4f20 0x7f6024154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 -- 172.21.15.107:0/3708314993 >> 172.21.15.107:0/3708314993 conn(0x7f60240c2860 msgr2=0x7f60240c2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:23.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 -- 172.21.15.107:0/3708314993 shutdown_connections 2023-10-25T00:22:23.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.096+0000 7f602931c640 1 -- 172.21.15.107:0/3708314993 wait complete. 2023-10-25T00:22:23.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-25T00:22:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.220+0000 7f7a046cb640 1 Processor -- start 2023-10-25T00:22:23.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.220+0000 7f7a046cb640 1 -- start start 2023-10-25T00:22:23.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.220+0000 7f7a046cb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79fc0aa9f0 0x7f79fc072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:23.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.220+0000 7f7a046cb640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f79fc0a9300 0x7f79fc155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:23.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.220+0000 7f7a046cb640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79fc0befa0 con 0x7f79fc0c1b30 2023-10-25T00:22:23.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.220+0000 7f7a046cb640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79fc0a9b80 con 0x7f79fc0a9300 2023-10-25T00:22:23.224 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.220+0000 7f7a046cb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc0aa0b0 con 0x7f79fc0aa9f0 2023-10-25T00:22:23.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.220+0000 7f7a02c68640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f79fc0a9300 0x7f79fc155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:23.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.220+0000 7f7a02c68640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f79fc0a9300 0x7f79fc155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54086/0 (socket says 172.21.15.107:54086) 2023-10-25T00:22:23.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.220+0000 7f7a02c68640 1 -- 172.21.15.107:0/1736593128 learned_addr learned my addr 172.21.15.107:0/1736593128 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:23.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.220+0000 7f7a01c66640 1 --2- 172.21.15.107:0/1736593128 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79fc0aa9f0 0x7f79fc072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:23.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.224+0000 7f7a02c68640 1 -- 172.21.15.107:0/1736593128 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79fc0aa9f0 msgr2=0x7f79fc072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:23.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.224+0000 7f7a02c68640 1 --2- 172.21.15.107:0/1736593128 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79fc0aa9f0 0x7f79fc072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.224+0000 7f7a02c68640 1 -- 172.21.15.107:0/1736593128 >> v1:172.21.15.110:6789/0 conn(0x7f79fc0c1b30 legacy=0x7f79fc0aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:23.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.224+0000 7f7a02c68640 1 -- 172.21.15.107:0/1736593128 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79fc0bf1f0 con 0x7f79fc0a9300 2023-10-25T00:22:23.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.224+0000 7f7a02c68640 1 --2- 172.21.15.107:0/1736593128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79fc0a9300 0x7f79fc155620 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009e50 tx=0x7f79f401c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946c7b00ea4f7514 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:23.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.224+0000 7f7a01465640 1 -- 172.21.15.107:0/1736593128 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f40a7070 con 0x7f79fc0a9300 2023-10-25T00:22:23.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.224+0000 7f7a01465640 1 -- 172.21.15.107:0/1736593128 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f40090d0 con 0x7f79fc0a9300 2023-10-25T00:22:23.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.224+0000 7f7a01465640 1 -- 172.21.15.107:0/1736593128 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f40ab610 con 0x7f79fc0a9300 2023-10-25T00:22:23.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- 172.21.15.107:0/1736593128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79fc0a9300 msgr2=0x7f79fc155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:23.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 --2- 172.21.15.107:0/1736593128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79fc0a9300 0x7f79fc155620 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009e50 tx=0x7f79f401c630 comp rx=0 tx=0).stop 2023-10-25T00:22:23.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- 172.21.15.107:0/1736593128 shutdown_connections 2023-10-25T00:22:23.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 --2- 172.21.15.107:0/1736593128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79fc0a9300 0x7f79fc155620 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 --2- 172.21.15.107:0/1736593128 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79fc0aa9f0 0x7f79fc072210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- 172.21.15.107:0/1736593128 >> 172.21.15.107:0/1736593128 conn(0x7f79fc0c2860 msgr2=0x7f79fc0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:23.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- 172.21.15.107:0/1736593128 shutdown_connections 2023-10-25T00:22:23.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- 172.21.15.107:0/1736593128 wait complete. 2023-10-25T00:22:23.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 Processor -- start 2023-10-25T00:22:23.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- start start 2023-10-25T00:22:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79fc0a9300 0x7f79fc14b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79fc0c1b30 0x7f79fc149900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79fc0b9050 0x7f79fc151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc151040 con 0x7f79fc0a9300 2023-10-25T00:22:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc06cee0 con 0x7f79fc0c1b30 2023-10-25T00:22:23.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc149e40 con 0x7f79fc0b9050 2023-10-25T00:22:23.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a02467640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79fc0a9300 0x7f79fc14b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:23.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a02467640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79fc0a9300 0x7f79fc14b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54092/0 (socket says 172.21.15.107:54092) 2023-10-25T00:22:23.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a02467640 1 -- 172.21.15.107:0/3730833259 learned_addr learned my addr 172.21.15.107:0/3730833259 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a02467640 1 -- 172.21.15.107:0/3730833259 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79fc0b9050 msgr2=0x7f79fc151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a02467640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79fc0b9050 0x7f79fc151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a02467640 1 -- 172.21.15.107:0/3730833259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79fc0c1b30 msgr2=0x7f79fc149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a01c66640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79fc0c1b30 0x7f79fc149900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:23.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a02467640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79fc0c1b30 0x7f79fc149900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a02467640 1 -- 172.21.15.107:0/3730833259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f4009b30 con 0x7f79fc0a9300 2023-10-25T00:22:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a02467640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79fc0a9300 0x7f79fc14b3e0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f79f800c3f0 tx=0x7f79f800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f79f2ffd640 1 -- 172.21.15.107:0/3730833259 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f80052d0 con 0x7f79fc0a9300 2023-10-25T00:22:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f79f2ffd640 1 -- 172.21.15.107:0/3730833259 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f8005430 con 0x7f79fc0a9300 2023-10-25T00:22:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- 172.21.15.107:0/3730833259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79fc14ac80 con 0x7f79fc0a9300 2023-10-25T00:22:23.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f79f2ffd640 1 -- 172.21.15.107:0/3730833259 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8003840 con 0x7f79fc0a9300 2023-10-25T00:22:23.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- 172.21.15.107:0/3730833259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79fc14a350 con 0x7f79fc0a9300 2023-10-25T00:22:23.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.228+0000 7f7a046cb640 1 -- 172.21.15.107:0/3730833259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79fc0aa500 con 0x7f79fc0a9300 2023-10-25T00:22:23.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.232+0000 7f79f2ffd640 1 -- 172.21.15.107:0/3730833259 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f79f80039a0 con 0x7f79fc0a9300 2023-10-25T00:22:23.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.232+0000 7f79f2ffd640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f79e0065e90 0x7f79e0068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.232+0000 7f79f2ffd640 1 -- 172.21.15.107:0/3730833259 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f79f8084070 con 0x7f79fc0a9300 2023-10-25T00:22:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.232+0000 7f7a01c66640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f79e0065e90 0x7f79e0068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.232+0000 7f7a01c66640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f79e0065e90 0x7f79e0068350 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f79fc0a4d20 tx=0x7f79ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.236+0000 7f79f2ffd640 1 -- 172.21.15.107:0/3730833259 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f79f8050420 con 0x7f79fc0a9300 2023-10-25T00:22:23.390 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.388+0000 7f7a046cb640 1 -- 172.21.15.107:0/3730833259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f79fc04e130 con 0x7f79fc0a9300 2023-10-25T00:22:23.391 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f79f2ffd640 1 -- 172.21.15.107:0/3730833259 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 88 entries v72) v1 ==== 81+0+5426 (secure 0 0 0) 0x7f79f8053a40 con 0x7f79fc0a9300 2023-10-25T00:22:23.391 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2187059523 2023-10-26T00:21:39.173084+0000 2023-10-25T00:22:23.392 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/657626630 2023-10-26T00:21:19.167088+0000 2023-10-25T00:22:23.392 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/3771564389 2023-10-26T00:20:19.149072+0000 2023-10-25T00:22:23.392 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/3771564389 2023-10-26T00:20:19.149072+0000 2023-10-25T00:22:23.392 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2970408860 2023-10-26T00:19:04.122659+0000 2023-10-25T00:22:23.392 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/418589140 2023-10-26T00:18:49.118366+0000 2023-10-25T00:22:23.393 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/985135365 2023-10-26T00:17:39.037219+0000 2023-10-25T00:22:23.393 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/985135365 2023-10-26T00:17:39.037219+0000 2023-10-25T00:22:23.393 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/208713198 2023-10-26T00:17:24.032902+0000 2023-10-25T00:22:23.393 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/208713198 2023-10-26T00:17:24.032902+0000 2023-10-25T00:22:23.393 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/1619327197 2023-10-26T00:15:22.673615+0000 2023-10-25T00:22:23.394 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/175581297 2023-10-26T00:15:18.921240+0000 2023-10-25T00:22:23.394 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/418589140 2023-10-26T00:18:49.118366+0000 2023-10-25T00:22:23.394 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2856791739 2023-10-26T00:13:48.893880+0000 2023-10-25T00:22:23.394 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/898258574 2023-10-26T00:13:18.883076+0000 2023-10-25T00:22:23.394 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3396644475 2023-10-26T00:19:44.136579+0000 2023-10-25T00:22:23.395 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/3370462806 2023-10-26T00:12:33.862408+0000 2023-10-25T00:22:23.395 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/1101018338 2023-10-25T23:59:48.457105+0000 2023-10-25T00:22:23.395 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/838886697 2023-10-25T23:56:18.381650+0000 2023-10-25T00:22:23.395 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1381726420 2023-10-26T00:01:28.521014+0000 2023-10-25T00:22:23.395 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/1619327197 2023-10-26T00:15:22.673615+0000 2023-10-25T00:22:23.396 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1381726420 2023-10-26T00:01:28.521014+0000 2023-10-25T00:22:23.396 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/3103569740 2023-10-26T00:01:58.532147+0000 2023-10-25T00:22:23.396 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/780830443 2023-10-26T00:06:23.634894+0000 2023-10-25T00:22:23.396 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/1101018338 2023-10-25T23:59:48.457105+0000 2023-10-25T00:22:23.396 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1555840329 2023-10-26T00:05:18.610829+0000 2023-10-25T00:22:23.397 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/3836115666 2023-10-26T00:08:58.700067+0000 2023-10-25T00:22:23.397 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/898258574 2023-10-26T00:13:18.883076+0000 2023-10-25T00:22:23.397 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1508236506 2023-10-25T23:58:58.439244+0000 2023-10-25T00:22:23.397 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2348043270 2023-10-25T23:58:53.436687+0000 2023-10-25T00:22:23.397 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/204494054 2023-10-25T23:58:03.418411+0000 2023-10-25T00:22:23.397 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1555840329 2023-10-26T00:05:18.610829+0000 2023-10-25T00:22:23.398 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/3721101226 2023-10-25T23:58:03.418295+0000 2023-10-25T00:22:23.398 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2480841919 2023-10-25T23:55:58.372958+0000 2023-10-25T00:22:23.398 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/2577776351 2023-10-26T00:04:34.728765+0000 2023-10-25T00:22:23.398 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1246928633 2023-10-26T00:05:13.609231+0000 2023-10-25T00:22:23.398 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/913067382 2023-10-26T00:08:08.675592+0000 2023-10-25T00:22:23.399 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/1031240534 2023-10-25T23:57:23.404313+0000 2023-10-25T00:22:23.399 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/727042497 2023-10-25T23:54:13.340839+0000 2023-10-25T00:22:23.399 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2970408860 2023-10-26T00:19:04.122659+0000 2023-10-25T00:22:23.399 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2480841919 2023-10-25T23:55:58.372958+0000 2023-10-25T00:22:23.400 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/201284779 2023-10-26T00:02:23.541454+0000 2023-10-25T00:22:23.400 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/201284779 2023-10-26T00:02:23.541454+0000 2023-10-25T00:22:23.400 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/838886697 2023-10-25T23:56:18.381650+0000 2023-10-25T00:22:23.400 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1292188015 2023-10-26T00:00:48.479959+0000 2023-10-25T00:22:23.400 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1317796403 2023-10-26T00:08:23.682615+0000 2023-10-25T00:22:23.401 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1220982369 2023-10-25T23:55:23.361145+0000 2023-10-25T00:22:23.401 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2348043270 2023-10-25T23:58:53.436687+0000 2023-10-25T00:22:23.401 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/3370462806 2023-10-26T00:12:33.862408+0000 2023-10-25T00:22:23.401 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1220982369 2023-10-25T23:55:23.361145+0000 2023-10-25T00:22:23.401 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/204494054 2023-10-25T23:58:03.418411+0000 2023-10-25T00:22:23.402 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/175581297 2023-10-26T00:15:18.921240+0000 2023-10-25T00:22:23.402 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2856791739 2023-10-26T00:13:48.893880+0000 2023-10-25T00:22:23.402 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/1031240534 2023-10-25T23:57:23.404313+0000 2023-10-25T00:22:23.402 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/326715537 2023-10-26T00:11:03.824824+0000 2023-10-25T00:22:23.402 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1004926771 2023-10-25T23:57:28.406683+0000 2023-10-25T00:22:23.403 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/657626630 2023-10-26T00:21:19.167088+0000 2023-10-25T00:22:23.403 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/1004926771 2023-10-25T23:57:28.406683+0000 2023-10-25T00:22:23.403 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/3721101226 2023-10-25T23:58:03.418295+0000 2023-10-25T00:22:23.403 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3474118649 2023-10-26T00:11:18.831318+0000 2023-10-25T00:22:23.403 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/3232233949 2023-10-26T00:03:28.568587+0000 2023-10-25T00:22:23.404 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/2577776351 2023-10-26T00:04:34.728765+0000 2023-10-25T00:22:23.404 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/326715537 2023-10-26T00:11:03.824824+0000 2023-10-25T00:22:23.404 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/2276136116 2023-10-26T00:09:18.706706+0000 2023-10-25T00:22:23.404 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/319918260 2023-10-26T00:03:53.579302+0000 2023-10-25T00:22:23.404 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3474118649 2023-10-26T00:11:18.831318+0000 2023-10-25T00:22:23.405 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1292188015 2023-10-26T00:00:48.479959+0000 2023-10-25T00:22:23.405 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6834/174863522 2023-10-26T00:05:53.622773+0000 2023-10-25T00:22:23.405 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/319918260 2023-10-26T00:03:53.579302+0000 2023-10-25T00:22:23.405 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/3103569740 2023-10-26T00:01:58.532147+0000 2023-10-25T00:22:23.405 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/2150827048 2023-10-26T00:07:48.667043+0000 2023-10-25T00:22:23.406 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/4180698155 2023-10-26T00:04:28.591981+0000 2023-10-25T00:22:23.406 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3396644475 2023-10-26T00:19:44.136579+0000 2023-10-25T00:22:23.406 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/4180698155 2023-10-26T00:04:28.591981+0000 2023-10-25T00:22:23.406 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2187059523 2023-10-26T00:21:39.173084+0000 2023-10-25T00:22:23.407 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1246928633 2023-10-26T00:05:13.609231+0000 2023-10-25T00:22:23.407 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6832/174863522 2023-10-26T00:05:53.622773+0000 2023-10-25T00:22:23.407 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/1508236506 2023-10-25T23:58:58.439244+0000 2023-10-25T00:22:23.407 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/780830443 2023-10-26T00:06:23.634894+0000 2023-10-25T00:22:23.407 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6833/2150827048 2023-10-26T00:07:48.667043+0000 2023-10-25T00:22:23.407 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6835/913067382 2023-10-26T00:08:08.675592+0000 2023-10-25T00:22:23.408 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1317796403 2023-10-26T00:08:23.682615+0000 2023-10-25T00:22:23.408 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/2276136116 2023-10-26T00:09:18.706706+0000 2023-10-25T00:22:23.408 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6834/3232233949 2023-10-26T00:03:28.568587+0000 2023-10-25T00:22:23.408 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/727042497 2023-10-25T23:54:13.340839+0000 2023-10-25T00:22:23.408 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6836/1788268787 2023-10-26T00:10:38.784519+0000 2023-10-25T00:22:23.409 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.110:6835/3836115666 2023-10-26T00:08:58.700067+0000 2023-10-25T00:22:23.409 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6837/1788268787 2023-10-26T00:10:38.784519+0000 2023-10-25T00:22:23.410 INFO:teuthology.orchestra.run.smithi107.stderr:listed 88 entries 2023-10-25T00:22:23.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 -- 172.21.15.107:0/3730833259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f79e0065e90 msgr2=0x7f79e0068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:23.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f79e0065e90 0x7f79e0068350 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f79fc0a4d20 tx=0x7f79ec008040 comp rx=0 tx=0).stop 2023-10-25T00:22:23.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 -- 172.21.15.107:0/3730833259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79fc0a9300 msgr2=0x7f79fc14b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:23.411 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79fc0a9300 0x7f79fc14b3e0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f79f800c3f0 tx=0x7f79f800c8b0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 -- 172.21.15.107:0/3730833259 shutdown_connections 2023-10-25T00:22:23.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f79e0065e90 0x7f79e0068350 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79fc0b9050 0x7f79fc151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79fc0c1b30 0x7f79fc149900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 --2- 172.21.15.107:0/3730833259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79fc0a9300 0x7f79fc14b3e0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:23.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 -- 172.21.15.107:0/3730833259 >> 172.21.15.107:0/3730833259 conn(0x7f79fc0c2860 msgr2=0x7f79fc0bc670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:23.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 -- 172.21.15.107:0/3730833259 shutdown_connections 2023-10-25T00:22:23.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:23.392+0000 7f7a046cb640 1 -- 172.21.15.107:0/3730833259 wait complete. 2023-10-25T00:22:23.413 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi110.front.sepia.ceph.com... 2023-10-25T00:22:23.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-25T00:22:23.414 DEBUG:teuthology.orchestra.run.smithi110:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-10-25T00:22:23.505 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:22:23.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi110.stderr:ceph-fuse[42968]: fuse finished with error 0 and tester_r 0 2023-10-25T00:22:29.608 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi110.front.sepia.ceph.com 2023-10-25T00:22:29.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-25T00:22:29.609 DEBUG:teuthology.orchestra.run.smithi110:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-25T00:22:29.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-25T00:22:29.618 DEBUG:teuthology.orchestra.run.smithi110:> ip netns list 2023-10-25T00:22:29.936 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-25T00:22:29.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-25T00:22:29.937 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-25T00:22:29.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-25T00:22:29.951 DEBUG:teuthology.orchestra.run.smithi110:> sudo ip link delete ceph-brx 2023-10-25T00:22:30.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-25T00:22:30.165 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2023-10-25T00:22:30.315 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-25T00:22:30.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-25T00:22:30.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-25T00:22:30.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-25T00:22:30.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link delete ceph-brx 2023-10-25T00:22:30.549 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-25T00:22:30.560 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-25T00:22:30.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-25T00:22:30.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.688+0000 7f16c0526640 1 Processor -- start 2023-10-25T00:22:30.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.688+0000 7f16c0526640 1 -- start start 2023-10-25T00:22:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16c0526640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b80c24c0 0x7f16b806a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16c0526640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b8150bc0 con 0x7f16b8151960 2023-10-25T00:22:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16c0526640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b806c510 con 0x7f16b8071220 2023-10-25T00:22:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16c0526640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b806c7f0 con 0x7f16b80c24c0 2023-10-25T00:22:30.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bdac1640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f16b8151960 0x7f16b8071a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:36880/0 (socket says 172.21.15.107:36880) 2023-10-25T00:22:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bdac1640 1 -- 172.21.15.107:0/548240035 learned_addr learned my addr 172.21.15.107:0/548240035 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861269408 0 0) 0x7f16b8150bc0 con 0x7f16b8151960 2023-10-25T00:22:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a4003450 con 0x7f16b8151960 2023-10-25T00:22:30.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16beac3640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b80c24c0 0x7f16b806a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3423527026 0 0) 0x7f16a4003450 con 0x7f16b8151960 2023-10-25T00:22:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b80c24c0 msgr2=0x7f16b806a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b80c24c0 0x7f16b806a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 >> v1:172.21.15.110:6789/0 conn(0x7f16b8071220 legacy=0x7f16b806bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:30.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b8150e80 con 0x7f16b8151960 2023-10-25T00:22:30.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f16a8001680 con 0x7f16b8151960 2023-10-25T00:22:30.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 >> v1:172.21.15.107:6789/0 conn(0x7f16b8151960 legacy=0x7f16b8071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:30.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b80c24c0 0x7f16a4003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:30.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16a40040d0 0x7f16a400a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:30.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16a4004690 0x7f16a4004b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b806c7f0 con 0x7f16a4004690 2023-10-25T00:22:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a40053f0 con 0x7f16a40040d0 2023-10-25T00:22:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a40055b0 con 0x7f16b80c24c0 2023-10-25T00:22:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16beac3640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16a4004690 0x7f16a4004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16be2c2640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b80c24c0 0x7f16a4003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:30.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16bdac1640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16a40040d0 0x7f16a400a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:30.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16beac3640 1 -- 172.21.15.107:0/548240035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b80c24c0 msgr2=0x7f16a4003aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:30.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16beac3640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b80c24c0 0x7f16a4003aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:30.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16beac3640 1 -- 172.21.15.107:0/548240035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16a40040d0 msgr2=0x7f16a400a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:30.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16beac3640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16a40040d0 0x7f16a400a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:30.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.692+0000 7f16beac3640 1 -- 172.21.15.107:0/548240035 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a4003450 con 0x7f16a4004690 2023-10-25T00:22:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16beac3640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16a4004690 0x7f16a4004b10 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f16b0007360 tx=0x7f16b001ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66b83e11f92f9ef server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b00a5c10 con 0x7f16a4004690 2023-10-25T00:22:30.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b00a5d70 con 0x7f16a4004690 2023-10-25T00:22:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16bd2c0640 1 -- 172.21.15.107:0/548240035 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b00ad660 con 0x7f16a4004690 2023-10-25T00:22:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 -- 172.21.15.107:0/548240035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16a4004690 msgr2=0x7f16a4004b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 --2- 172.21.15.107:0/548240035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16a4004690 0x7f16a4004b10 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f16b0007360 tx=0x7f16b001ce20 comp rx=0 tx=0).stop 2023-10-25T00:22:30.704 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16beac3640 1 -- 172.21.15.107:0/548240035 reap_dead start 2023-10-25T00:22:30.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 -- 172.21.15.107:0/548240035 shutdown_connections 2023-10-25T00:22:30.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 -- 172.21.15.107:0/548240035 >> 172.21.15.107:0/548240035 conn(0x7f16b80c2950 msgr2=0x7f16b80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:30.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 -- 172.21.15.107:0/548240035 shutdown_connections 2023-10-25T00:22:30.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 -- 172.21.15.107:0/548240035 wait complete. 2023-10-25T00:22:30.705 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 Processor -- start 2023-10-25T00:22:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 -- start start 2023-10-25T00:22:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b81e8c60 0x7f16b81e9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b8148e80 0x7f16b8147bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16b8148130 0x7f16b81dbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b814a4b0 con 0x7f16b8148e80 2023-10-25T00:22:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8148930 con 0x7f16b8148130 2023-10-25T00:22:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16c0526640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b814a020 con 0x7f16b81e8c60 2023-10-25T00:22:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16bdac1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b8148e80 0x7f16b8147bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16bdac1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b8148e80 0x7f16b8147bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54116/0 (socket says 172.21.15.107:54116) 2023-10-25T00:22:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16bdac1640 1 -- 172.21.15.107:0/1520239715 learned_addr learned my addr 172.21.15.107:0/1520239715 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:30.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16beac3640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16b8148130 0x7f16b81dbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.696+0000 7f16be2c2640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b81e8c60 0x7f16b81e9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16bdac1640 1 -- 172.21.15.107:0/1520239715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b81e8c60 msgr2=0x7f16b81e9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16bdac1640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b81e8c60 0x7f16b81e9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16bdac1640 1 -- 172.21.15.107:0/1520239715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16b8148130 msgr2=0x7f16b81dbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:30.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16bdac1640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16b8148130 0x7f16b81dbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16bdac1640 1 -- 172.21.15.107:0/1520239715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b814e320 con 0x7f16b8148e80 2023-10-25T00:22:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16be2c2640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b81e8c60 0x7f16b81e9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:22:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16bdac1640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b8148e80 0x7f16b8147bf0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f16a80119e0 tx=0x7f16a8011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16aeffd640 1 -- 172.21.15.107:0/1520239715 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a800fd10 con 0x7f16b8148e80 2023-10-25T00:22:30.709 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16aeffd640 1 -- 172.21.15.107:0/1520239715 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a800fe90 con 0x7f16b8148e80 2023-10-25T00:22:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16c0526640 1 -- 172.21.15.107:0/1520239715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b8148b80 con 0x7f16b8148e80 2023-10-25T00:22:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16aeffd640 1 -- 172.21.15.107:0/1520239715 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8015650 con 0x7f16b8148e80 2023-10-25T00:22:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16c0526640 1 -- 172.21.15.107:0/1520239715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b8152ad0 con 0x7f16b8148e80 2023-10-25T00:22:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16c0526640 1 -- 172.21.15.107:0/1520239715 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b81e9580 con 0x7f16b8148e80 2023-10-25T00:22:30.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16aeffd640 1 -- 172.21.15.107:0/1520239715 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f16a80157b0 con 0x7f16b8148e80 2023-10-25T00:22:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16aeffd640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f169c065e90 0x7f169c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16aeffd640 1 -- 172.21.15.107:0/1520239715 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f16a8087ed0 con 0x7f16b8148e80 2023-10-25T00:22:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16be2c2640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f169c065e90 0x7f169c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:30.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.700+0000 7f16be2c2640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f169c065e90 0x7f169c068350 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f16b4007fd0 tx=0x7f16b4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:30.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.704+0000 7f16aeffd640 1 -- 172.21.15.107:0/1520239715 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f16a8054310 con 0x7f16b8148e80 2023-10-25T00:22:30.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.860+0000 7f16c0526640 1 -- 172.21.15.107:0/1520239715 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16b8149e10 con 0x7f169c065e90 2023-10-25T00:22:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.904+0000 7f16aeffd640 1 -- 172.21.15.107:0/1520239715 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+550969 (secure 0 0 0) 0x7f16b4001590 con 0x7f169c065e90 2023-10-25T00:22:30.905 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:30.913 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-25T00:22:30.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 -- 172.21.15.107:0/1520239715 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f169c065e90 msgr2=0x7f169c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:30.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f169c065e90 0x7f169c068350 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f16b4007fd0 tx=0x7f16b4009040 comp rx=0 tx=0).stop 2023-10-25T00:22:30.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 -- 172.21.15.107:0/1520239715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b8148e80 msgr2=0x7f16b8147bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:30.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b8148e80 0x7f16b8147bf0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f16a80119e0 tx=0x7f16a8011ea0 comp rx=0 tx=0).stop 2023-10-25T00:22:30.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 -- 172.21.15.107:0/1520239715 shutdown_connections 2023-10-25T00:22:30.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f16b8148130 0x7f16b81dbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:30.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16b8148e80 0x7f16b8147bf0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:30.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f169c065e90 0x7f169c068350 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:30.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 --2- 172.21.15.107:0/1520239715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f16b81e8c60 0x7f16b81e9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:30.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 -- 172.21.15.107:0/1520239715 >> 172.21.15.107:0/1520239715 conn(0x7f16b80c2950 msgr2=0x7f16b80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:30.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 -- 172.21.15.107:0/1520239715 shutdown_connections 2023-10-25T00:22:30.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:30.908+0000 7f16c0526640 1 -- 172.21.15.107:0/1520239715 wait complete. 2023-10-25T00:22:30.930 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":955,"stamp":"2023-10-25T00:22:30.008225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40122242,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3906,"num_read_kb":457920,"num_write":3153446,"num_write_kb":38880045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":583926,"ondisk_log_size":583926,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7826840,"kb_used_data":86040,"kb_used_omap":0,"kb_used_meta":7740736,"kb_avail":747147880,"statfs":{"total":773094113280,"available":765079429120,"internally_reserved":0,"allocated":88104960,"data_stored":84515236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7926513664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":445,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005534"},"pg_stats":[{"pgid":"6.1a","version":"72'3995","reported_seq":4066,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.672949+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-25T00:22:08.672949+0000","last_peered":"2023-10-25T00:22:08.672949+0000","last_clean":"2023-10-25T00:22:08.672949+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-25T00:22:08.672949+0000","last_undegraded":"2023-10-25T00:22:08.672949+0000","last_fullsized":"2023-10-25T00:22:08.672949+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:51:20.137796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21480,"num_write_kb":282788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.773236+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-25T00:22:08.773236+0000","last_peered":"2023-10-25T00:22:08.773236+0000","last_clean":"2023-10-25T00:22:08.773236+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-25T00:22:08.773236+0000","last_undegraded":"2023-10-25T00:22:08.773236+0000","last_fullsized":"2023-10-25T00:22:08.773236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:00:20.101379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"68'864","reported_seq":988,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.254160+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-25T00:21:42.254160+0000","last_peered":"2023-10-25T00:21:42.254160+0000","last_clean":"2023-10-25T00:21:42.254160+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-25T00:21:42.254160+0000","last_undegraded":"2023-10-25T00:21:42.254160+0000","last_fullsized":"2023-10-25T00:21:42.254160+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":18441,"num_write":841,"num_write_kb":4586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'5616","reported_seq":5688,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195291+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-25T00:21:39.195291+0000","last_peered":"2023-10-25T00:21:39.195291+0000","last_clean":"2023-10-25T00:21:39.195291+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-25T00:21:39.195291+0000","last_undegraded":"2023-10-25T00:21:39.195291+0000","last_fullsized":"2023-10-25T00:21:39.195291+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29775,"num_write_kb":348818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"72'3735","reported_seq":3804,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.701818+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-25T00:22:08.701818+0000","last_peered":"2023-10-25T00:22:08.701818+0000","last_clean":"2023-10-25T00:22:08.701818+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-25T00:22:08.701818+0000","last_undegraded":"2023-10-25T00:22:08.701818+0000","last_fullsized":"2023-10-25T00:22:08.701818+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:59.368156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20090,"num_write_kb":264783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"72'2","reported_seq":70,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.258523+0000","last_change":"2023-10-24T23:53:11.444089+0000","last_active":"2023-10-25T00:22:08.258523+0000","last_peered":"2023-10-25T00:22:08.258523+0000","last_clean":"2023-10-25T00:22:08.258523+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-25T00:22:08.258523+0000","last_undegraded":"2023-10-25T00:22:08.258523+0000","last_fullsized":"2023-10-25T00:22:08.258523+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:54:06.875882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"72'27","reported_seq":101,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186964+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-25T00:21:39.186964+0000","last_peered":"2023-10-25T00:21:39.186964+0000","last_clean":"2023-10-25T00:21:39.186964+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-25T00:21:39.186964+0000","last_undegraded":"2023-10-25T00:21:39.186964+0000","last_fullsized":"2023-10-25T00:21:39.186964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'5447","reported_seq":5519,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.184033+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-25T00:21:39.184033+0000","last_peered":"2023-10-25T00:21:39.184033+0000","last_clean":"2023-10-25T00:21:39.184033+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-25T00:21:39.184033+0000","last_undegraded":"2023-10-25T00:21:39.184033+0000","last_fullsized":"2023-10-25T00:21:39.184033+0000","mapping_epoch":21,"log_start":"61'100","ondisk_log_start":"61'100","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28063,"num_write_kb":350200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"72'4747","reported_seq":4818,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.795259+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-25T00:22:08.795259+0000","last_peered":"2023-10-25T00:22:08.795259+0000","last_clean":"2023-10-25T00:22:08.795259+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-25T00:22:08.795259+0000","last_undegraded":"2023-10-25T00:22:08.795259+0000","last_fullsized":"2023-10-25T00:22:08.795259+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:38:56.643025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":25766,"num_write_kb":328702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"72'753","reported_seq":847,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.712526+0000","last_change":"2023-10-24T23:53:11.438681+0000","last_active":"2023-10-25T00:22:08.712526+0000","last_peered":"2023-10-25T00:22:08.712526+0000","last_clean":"2023-10-25T00:22:08.712526+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-25T00:22:08.712526+0000","last_undegraded":"2023-10-25T00:22:08.712526+0000","last_fullsized":"2023-10-25T00:22:08.712526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:50.139697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":77282,"num_write":740,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"72'48","reported_seq":163,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.487987+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-25T00:21:42.487987+0000","last_peered":"2023-10-25T00:21:42.487987+0000","last_clean":"2023-10-25T00:21:42.487987+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-25T00:21:42.487987+0000","last_undegraded":"2023-10-25T00:21:42.487987+0000","last_fullsized":"2023-10-25T00:21:42.487987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'4792","reported_seq":4864,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205939+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-25T00:21:19.205939+0000","last_peered":"2023-10-25T00:21:19.205939+0000","last_clean":"2023-10-25T00:21:19.205939+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-25T00:21:19.205939+0000","last_undegraded":"2023-10-25T00:21:19.205939+0000","last_fullsized":"2023-10-25T00:21:19.205939+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24997,"num_write_kb":313272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"72'4739","reported_seq":4811,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.785162+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-25T00:22:08.785162+0000","last_peered":"2023-10-25T00:22:08.785162+0000","last_clean":"2023-10-25T00:22:08.785162+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-25T00:22:08.785162+0000","last_undegraded":"2023-10-25T00:22:08.785162+0000","last_fullsized":"2023-10-25T00:22:08.785162+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:22.974333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":22,"num_write":25639,"num_write_kb":332364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"72'357","reported_seq":426,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:15.185838+0000","last_change":"2023-10-24T23:53:11.454537+0000","last_active":"2023-10-25T00:22:15.185838+0000","last_peered":"2023-10-25T00:22:15.185838+0000","last_clean":"2023-10-25T00:22:15.185838+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-25T00:22:15.185838+0000","last_undegraded":"2023-10-25T00:22:15.185838+0000","last_fullsized":"2023-10-25T00:22:15.185838+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:14:06.792651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2627766,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":227,"num_write":329,"num_write_kb":2759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"68'5","reported_seq":122,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.254866+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-25T00:21:42.254866+0000","last_peered":"2023-10-25T00:21:42.254866+0000","last_clean":"2023-10-25T00:21:42.254866+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-25T00:21:42.254866+0000","last_undegraded":"2023-10-25T00:21:42.254866+0000","last_fullsized":"2023-10-25T00:21:42.254866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'3731","reported_seq":3801,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205159+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-25T00:21:19.205159+0000","last_peered":"2023-10-25T00:21:19.205159+0000","last_clean":"2023-10-25T00:21:19.205159+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-25T00:21:19.205159+0000","last_undegraded":"2023-10-25T00:21:19.205159+0000","last_fullsized":"2023-10-25T00:21:19.205159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19492,"num_write_kb":250091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"72'3799","reported_seq":3869,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.756165+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-25T00:22:08.756165+0000","last_peered":"2023-10-25T00:22:08.756165+0000","last_clean":"2023-10-25T00:22:08.756165+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-25T00:22:08.756165+0000","last_undegraded":"2023-10-25T00:22:08.756165+0000","last_fullsized":"2023-10-25T00:22:08.756165+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:35:39.879563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21581,"num_write_kb":262877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"72'45","reported_seq":156,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.747187+0000","last_change":"2023-10-24T23:53:11.444138+0000","last_active":"2023-10-25T00:22:08.747187+0000","last_peered":"2023-10-25T00:22:08.747187+0000","last_clean":"2023-10-25T00:22:08.747187+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-25T00:22:08.747187+0000","last_undegraded":"2023-10-25T00:22:08.747187+0000","last_fullsized":"2023-10-25T00:22:08.747187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:27:38.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"68'714","reported_seq":813,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195222+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-25T00:21:39.195222+0000","last_peered":"2023-10-25T00:21:39.195222+0000","last_clean":"2023-10-25T00:21:39.195222+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-25T00:21:39.195222+0000","last_undegraded":"2023-10-25T00:21:39.195222+0000","last_fullsized":"2023-10-25T00:21:39.195222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":86332,"num_write":703,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'4008","reported_seq":4082,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.208728+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-25T00:21:19.208728+0000","last_peered":"2023-10-25T00:21:19.208728+0000","last_clean":"2023-10-25T00:21:19.208728+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-25T00:21:19.208728+0000","last_undegraded":"2023-10-25T00:21:19.208728+0000","last_fullsized":"2023-10-25T00:21:19.208728+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20791,"num_write_kb":267770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"72'5469","reported_seq":5550,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793934+0000","last_change":"2023-10-24T23:53:11.832809+0000","last_active":"2023-10-25T00:22:08.793934+0000","last_peered":"2023-10-25T00:22:08.793934+0000","last_clean":"2023-10-25T00:22:08.793934+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-25T00:22:08.793934+0000","last_undegraded":"2023-10-25T00:22:08.793934+0000","last_fullsized":"2023-10-25T00:22:08.793934+0000","mapping_epoch":25,"log_start":"64'200","ondisk_log_start":"64'200","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:01.151315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":40,"num_write":30646,"num_write_kb":401481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"72'5","reported_seq":116,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.200649+0000","last_change":"2023-10-24T23:53:11.442035+0000","last_active":"2023-10-25T00:21:39.200649+0000","last_peered":"2023-10-25T00:21:39.200649+0000","last_clean":"2023-10-25T00:21:39.200649+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-25T00:21:39.200649+0000","last_undegraded":"2023-10-25T00:21:39.200649+0000","last_fullsized":"2023-10-25T00:21:39.200649+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:13.717844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"72'532","reported_seq":611,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.992831+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-25T00:22:07.992831+0000","last_peered":"2023-10-25T00:22:07.992831+0000","last_clean":"2023-10-25T00:22:07.992831+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-25T00:22:07.992831+0000","last_undegraded":"2023-10-25T00:22:07.992831+0000","last_fullsized":"2023-10-25T00:22:07.992831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3177545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7524,"num_write":504,"num_write_kb":3406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'6069","reported_seq":6142,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183010+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-25T00:21:19.183010+0000","last_peered":"2023-10-25T00:21:19.183010+0000","last_clean":"2023-10-25T00:21:19.183010+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-25T00:21:19.183010+0000","last_undegraded":"2023-10-25T00:21:19.183010+0000","last_fullsized":"2023-10-25T00:21:19.183010+0000","mapping_epoch":21,"log_start":"62'800","ondisk_log_start":"62'800","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":33187,"num_write_kb":387130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'3606","reported_seq":3680,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.451770+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-25T00:22:08.451770+0000","last_peered":"2023-10-25T00:22:08.451770+0000","last_clean":"2023-10-25T00:22:08.451770+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-25T00:22:08.451770+0000","last_undegraded":"2023-10-25T00:22:08.451770+0000","last_fullsized":"2023-10-25T00:22:08.451770+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:26.012711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19493,"num_write_kb":254342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'1061","reported_seq":1182,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.761652+0000","last_change":"2023-10-24T23:53:11.448032+0000","last_active":"2023-10-25T00:22:08.761652+0000","last_peered":"2023-10-25T00:22:08.761652+0000","last_clean":"2023-10-25T00:22:08.761652+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-25T00:22:08.761652+0000","last_undegraded":"2023-10-25T00:22:08.761652+0000","last_fullsized":"2023-10-25T00:22:08.761652+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1061,"log_dups_size":0,"ondisk_log_size":1061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:50:32.490810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":9402,"num_write":1035,"num_write_kb":4717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'3","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.185352+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-25T00:21:39.185352+0000","last_peered":"2023-10-25T00:21:39.185352+0000","last_clean":"2023-10-25T00:21:39.185352+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-25T00:21:39.185352+0000","last_undegraded":"2023-10-25T00:21:39.185352+0000","last_fullsized":"2023-10-25T00:21:39.185352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'5139","reported_seq":5212,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.216678+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-25T00:21:19.216678+0000","last_peered":"2023-10-25T00:21:19.216678+0000","last_clean":"2023-10-25T00:21:19.216678+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-25T00:21:19.216678+0000","last_undegraded":"2023-10-25T00:21:19.216678+0000","last_fullsized":"2023-10-25T00:21:19.216678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27772,"num_write_kb":328469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"72'4843","reported_seq":4913,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.750475+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-25T00:22:08.750475+0000","last_peered":"2023-10-25T00:22:08.750475+0000","last_clean":"2023-10-25T00:22:08.750475+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-25T00:22:08.750475+0000","last_undegraded":"2023-10-25T00:22:08.750475+0000","last_fullsized":"2023-10-25T00:22:08.750475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:10.820967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":27290,"num_write_kb":330906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"72'27","reported_seq":97,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.749463+0000","last_change":"2023-10-24T23:53:11.446925+0000","last_active":"2023-10-25T00:22:08.749463+0000","last_peered":"2023-10-25T00:22:08.749463+0000","last_clean":"2023-10-25T00:22:08.749463+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-25T00:22:08.749463+0000","last_undegraded":"2023-10-25T00:22:08.749463+0000","last_fullsized":"2023-10-25T00:22:08.749463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:59:52.485469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'2","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182801+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-25T00:21:39.182801+0000","last_peered":"2023-10-25T00:21:39.182801+0000","last_clean":"2023-10-25T00:21:39.182801+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-25T00:21:39.182801+0000","last_undegraded":"2023-10-25T00:21:39.182801+0000","last_fullsized":"2023-10-25T00:21:39.182801+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'5814","reported_seq":5888,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189609+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-25T00:21:39.189609+0000","last_peered":"2023-10-25T00:21:39.189609+0000","last_clean":"2023-10-25T00:21:39.189609+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-25T00:21:39.189609+0000","last_undegraded":"2023-10-25T00:21:39.189609+0000","last_fullsized":"2023-10-25T00:21:39.189609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31996,"num_write_kb":382397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"72'3827","reported_seq":3898,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.365063+0000","last_change":"2023-10-24T23:53:11.849056+0000","last_active":"2023-10-25T00:22:08.365063+0000","last_peered":"2023-10-25T00:22:08.365063+0000","last_clean":"2023-10-25T00:22:08.365063+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-25T00:22:08.365063+0000","last_undegraded":"2023-10-25T00:22:08.365063+0000","last_fullsized":"2023-10-25T00:22:08.365063+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:08.932942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":21199,"num_write_kb":277730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"68'4820","reported_seq":4893,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.181692+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-25T00:21:19.181692+0000","last_peered":"2023-10-25T00:21:19.181692+0000","last_clean":"2023-10-25T00:21:19.181692+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-25T00:21:19.181692+0000","last_undegraded":"2023-10-25T00:21:19.181692+0000","last_fullsized":"2023-10-25T00:21:19.181692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25163,"num_write_kb":285293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"72'5","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:06.452266+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-25T00:22:06.452266+0000","last_peered":"2023-10-25T00:22:06.452266+0000","last_clean":"2023-10-25T00:22:06.452266+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-25T00:22:06.452266+0000","last_undegraded":"2023-10-25T00:22:06.452266+0000","last_fullsized":"2023-10-25T00:22:06.452266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"72'5","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.744190+0000","last_change":"2023-10-24T23:53:11.447167+0000","last_active":"2023-10-25T00:22:08.744190+0000","last_peered":"2023-10-25T00:22:08.744190+0000","last_clean":"2023-10-25T00:22:08.744190+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-25T00:22:08.744190+0000","last_undegraded":"2023-10-25T00:22:08.744190+0000","last_fullsized":"2023-10-25T00:22:08.744190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:06:25.006728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"72'4957","reported_seq":5031,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.762994+0000","last_change":"2023-10-24T23:53:11.813704+0000","last_active":"2023-10-25T00:22:08.762994+0000","last_peered":"2023-10-25T00:22:08.762994+0000","last_clean":"2023-10-25T00:22:08.762994+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-25T00:22:08.762994+0000","last_undegraded":"2023-10-25T00:22:08.762994+0000","last_fullsized":"2023-10-25T00:22:08.762994+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:39:09.996039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28093,"num_write_kb":358545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'4782","reported_seq":4860,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.185276+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-25T00:21:19.185276+0000","last_peered":"2023-10-25T00:21:19.185276+0000","last_clean":"2023-10-25T00:21:19.185276+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-25T00:21:19.185276+0000","last_undegraded":"2023-10-25T00:21:19.185276+0000","last_fullsized":"2023-10-25T00:21:19.185276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26432,"num_write_kb":298639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"68'3","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193250+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-25T00:21:39.193250+0000","last_peered":"2023-10-25T00:21:39.193250+0000","last_clean":"2023-10-25T00:21:39.193250+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-25T00:21:39.193250+0000","last_undegraded":"2023-10-25T00:21:39.193250+0000","last_fullsized":"2023-10-25T00:21:39.193250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"72'14","reported_seq":197,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.746812+0000","last_change":"2023-10-24T23:53:11.450974+0000","last_active":"2023-10-25T00:22:08.746812+0000","last_peered":"2023-10-25T00:22:08.746812+0000","last_clean":"2023-10-25T00:22:08.746812+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-25T00:22:08.746812+0000","last_undegraded":"2023-10-25T00:22:08.746812+0000","last_fullsized":"2023-10-25T00:22:08.746812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:51:05.443664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":345,"num_read_kb":345,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"72'2954","reported_seq":3023,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.717448+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-25T00:22:08.717448+0000","last_peered":"2023-10-25T00:22:08.717448+0000","last_clean":"2023-10-25T00:22:08.717448+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-25T00:22:08.717448+0000","last_undegraded":"2023-10-25T00:22:08.717448+0000","last_fullsized":"2023-10-25T00:22:08.717448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:37:30.278659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15961,"num_write_kb":219669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"72'36","reported_seq":152,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.212150+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-25T00:22:10.212150+0000","last_peered":"2023-10-25T00:22:10.212150+0000","last_clean":"2023-10-25T00:22:10.212150+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-25T00:22:10.212150+0000","last_undegraded":"2023-10-25T00:22:10.212150+0000","last_fullsized":"2023-10-25T00:22:10.212150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'4690","reported_seq":4764,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178387+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-25T00:21:19.178387+0000","last_peered":"2023-10-25T00:21:19.178387+0000","last_clean":"2023-10-25T00:21:19.178387+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-25T00:21:19.178387+0000","last_undegraded":"2023-10-25T00:21:19.178387+0000","last_fullsized":"2023-10-25T00:21:19.178387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25172,"num_write_kb":305786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"72'8","reported_seq":120,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.777365+0000","last_change":"2023-10-24T23:53:11.429694+0000","last_active":"2023-10-25T00:22:08.777365+0000","last_peered":"2023-10-25T00:22:08.777365+0000","last_clean":"2023-10-25T00:22:08.777365+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-25T00:22:08.777365+0000","last_undegraded":"2023-10-25T00:22:08.777365+0000","last_fullsized":"2023-10-25T00:22:08.777365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:15.964994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"72'4200","reported_seq":4277,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.728522+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-25T00:22:08.728522+0000","last_peered":"2023-10-25T00:22:08.728522+0000","last_clean":"2023-10-25T00:22:08.728522+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-25T00:22:08.728522+0000","last_undegraded":"2023-10-25T00:22:08.728522+0000","last_fullsized":"2023-10-25T00:22:08.728522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:38:18.678087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":24169,"num_write_kb":293796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"72'389","reported_seq":572,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.983790+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-25T00:22:07.983790+0000","last_peered":"2023-10-25T00:22:07.983790+0000","last_clean":"2023-10-25T00:22:07.983790+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-25T00:22:07.983790+0000","last_undegraded":"2023-10-25T00:22:07.983790+0000","last_fullsized":"2023-10-25T00:22:07.983790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":201,"num_read_kb":90291,"num_write":386,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'5730","reported_seq":5803,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205335+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-25T00:21:19.205335+0000","last_peered":"2023-10-25T00:21:19.205335+0000","last_clean":"2023-10-25T00:21:19.205335+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-25T00:21:19.205335+0000","last_undegraded":"2023-10-25T00:21:19.205335+0000","last_fullsized":"2023-10-25T00:21:19.205335+0000","mapping_epoch":21,"log_start":"61'500","ondisk_log_start":"61'500","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":31652,"num_write_kb":356481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760659+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-25T00:22:08.760659+0000","last_peered":"2023-10-25T00:22:08.760659+0000","last_clean":"2023-10-25T00:22:08.760659+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-25T00:22:08.760659+0000","last_undegraded":"2023-10-25T00:22:08.760659+0000","last_fullsized":"2023-10-25T00:22:08.760659+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:30.163913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"72'4168","reported_seq":4240,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.466973+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-25T00:22:08.466973+0000","last_peered":"2023-10-25T00:22:08.466973+0000","last_clean":"2023-10-25T00:22:08.466973+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-25T00:22:08.466973+0000","last_undegraded":"2023-10-25T00:22:08.466973+0000","last_fullsized":"2023-10-25T00:22:08.466973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:46.000330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23458,"num_write_kb":294850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"68'4339","reported_seq":4413,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197911+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-25T00:21:39.197911+0000","last_peered":"2023-10-25T00:21:39.197911+0000","last_clean":"2023-10-25T00:21:39.197911+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-25T00:21:39.197911+0000","last_undegraded":"2023-10-25T00:21:39.197911+0000","last_fullsized":"2023-10-25T00:21:39.197911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23267,"num_write_kb":265268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.222903+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-25T00:21:19.222903+0000","last_peered":"2023-10-25T00:21:19.222903+0000","last_clean":"2023-10-25T00:21:19.222903+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-25T00:21:19.222903+0000","last_undegraded":"2023-10-25T00:21:19.222903+0000","last_fullsized":"2023-10-25T00:21:19.222903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"72'230","reported_seq":326,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.211353+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-25T00:22:10.211353+0000","last_peered":"2023-10-25T00:22:10.211353+0000","last_clean":"2023-10-25T00:22:10.211353+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-25T00:22:10.211353+0000","last_undegraded":"2023-10-25T00:22:10.211353+0000","last_fullsized":"2023-10-25T00:22:10.211353+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":61,"num_write":589,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"72'11","reported_seq":81,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.756187+0000","last_change":"2023-10-24T23:53:11.433306+0000","last_active":"2023-10-25T00:22:08.756187+0000","last_peered":"2023-10-25T00:22:08.756187+0000","last_clean":"2023-10-25T00:22:08.756187+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-25T00:22:08.756187+0000","last_undegraded":"2023-10-25T00:22:08.756187+0000","last_fullsized":"2023-10-25T00:22:08.756187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:44:38.728345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"72'3993","reported_seq":4067,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793391+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-25T00:22:08.793391+0000","last_peered":"2023-10-25T00:22:08.793391+0000","last_clean":"2023-10-25T00:22:08.793391+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-25T00:22:08.793391+0000","last_undegraded":"2023-10-25T00:22:08.793391+0000","last_fullsized":"2023-10-25T00:22:08.793391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:30:24.805861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21723,"num_write_kb":283850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'4270","reported_seq":4344,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.181969+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-25T00:21:19.181969+0000","last_peered":"2023-10-25T00:21:19.181969+0000","last_clean":"2023-10-25T00:21:19.181969+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-25T00:21:19.181969+0000","last_undegraded":"2023-10-25T00:21:19.181969+0000","last_fullsized":"2023-10-25T00:21:19.181969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23049,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180537+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-25T00:21:19.180537+0000","last_peered":"2023-10-25T00:21:19.180537+0000","last_clean":"2023-10-25T00:21:19.180537+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-25T00:21:19.180537+0000","last_undegraded":"2023-10-25T00:21:19.180537+0000","last_fullsized":"2023-10-25T00:21:19.180537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'9","reported_seq":128,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.260782+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-25T00:21:42.260782+0000","last_peered":"2023-10-25T00:21:42.260782+0000","last_clean":"2023-10-25T00:21:42.260782+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-25T00:21:42.260782+0000","last_undegraded":"2023-10-25T00:21:42.260782+0000","last_fullsized":"2023-10-25T00:21:42.260782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"72'64","reported_seq":214,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.752186+0000","last_change":"2023-10-24T23:53:11.441983+0000","last_active":"2023-10-25T00:22:08.752186+0000","last_peered":"2023-10-25T00:22:08.752186+0000","last_clean":"2023-10-25T00:22:08.752186+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-25T00:22:08.752186+0000","last_undegraded":"2023-10-25T00:22:08.752186+0000","last_fullsized":"2023-10-25T00:22:08.752186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:30.643019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":123,"num_write":59,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"72'3769","reported_seq":3843,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.609361+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-25T00:22:08.609361+0000","last_peered":"2023-10-25T00:22:08.609361+0000","last_clean":"2023-10-25T00:22:08.609361+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-25T00:22:08.609361+0000","last_undegraded":"2023-10-25T00:22:08.609361+0000","last_fullsized":"2023-10-25T00:22:08.609361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:50:53.467058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20807,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'5752","reported_seq":5826,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194706+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-25T00:21:39.194706+0000","last_peered":"2023-10-25T00:21:39.194706+0000","last_clean":"2023-10-25T00:21:39.194706+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-25T00:21:39.194706+0000","last_undegraded":"2023-10-25T00:21:39.194706+0000","last_fullsized":"2023-10-25T00:21:39.194706+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31231,"num_write_kb":354718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199135+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-25T00:21:39.199135+0000","last_peered":"2023-10-25T00:21:39.199135+0000","last_clean":"2023-10-25T00:21:39.199135+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-25T00:21:39.199135+0000","last_undegraded":"2023-10-25T00:21:39.199135+0000","last_fullsized":"2023-10-25T00:21:39.199135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'2","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189313+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-25T00:21:39.189313+0000","last_peered":"2023-10-25T00:21:39.189313+0000","last_clean":"2023-10-25T00:21:39.189313+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-25T00:21:39.189313+0000","last_undegraded":"2023-10-25T00:21:39.189313+0000","last_fullsized":"2023-10-25T00:21:39.189313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"72'4","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.259221+0000","last_change":"2023-10-24T23:53:11.426287+0000","last_active":"2023-10-25T00:22:08.259221+0000","last_peered":"2023-10-25T00:22:08.259221+0000","last_clean":"2023-10-25T00:22:08.259221+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-25T00:22:08.259221+0000","last_undegraded":"2023-10-25T00:22:08.259221+0000","last_fullsized":"2023-10-25T00:22:08.259221+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:02:39.399866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"72'3939","reported_seq":4008,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.780603+0000","last_change":"2023-10-24T23:53:11.811139+0000","last_active":"2023-10-25T00:22:08.780603+0000","last_peered":"2023-10-25T00:22:08.780603+0000","last_clean":"2023-10-25T00:22:08.780603+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-25T00:22:08.780603+0000","last_undegraded":"2023-10-25T00:22:08.780603+0000","last_fullsized":"2023-10-25T00:22:08.780603+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:05.507633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22393,"num_write_kb":262004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'4305","reported_seq":4379,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186001+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-25T00:21:39.186001+0000","last_peered":"2023-10-25T00:21:39.186001+0000","last_clean":"2023-10-25T00:21:39.186001+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-25T00:21:39.186001+0000","last_undegraded":"2023-10-25T00:21:39.186001+0000","last_fullsized":"2023-10-25T00:21:39.186001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22696,"num_write_kb":287574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187237+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-25T00:21:39.187237+0000","last_peered":"2023-10-25T00:21:39.187237+0000","last_clean":"2023-10-25T00:21:39.187237+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-25T00:21:39.187237+0000","last_undegraded":"2023-10-25T00:21:39.187237+0000","last_fullsized":"2023-10-25T00:21:39.187237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.188607+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-25T00:21:39.188607+0000","last_peered":"2023-10-25T00:21:39.188607+0000","last_clean":"2023-10-25T00:21:39.188607+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-25T00:21:39.188607+0000","last_undegraded":"2023-10-25T00:21:39.188607+0000","last_fullsized":"2023-10-25T00:21:39.188607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"72'3","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.892690+0000","last_change":"2023-10-24T23:53:11.426199+0000","last_active":"2023-10-25T00:22:07.892690+0000","last_peered":"2023-10-25T00:22:07.892690+0000","last_clean":"2023-10-25T00:22:07.892690+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-25T00:22:07.892690+0000","last_undegraded":"2023-10-25T00:22:07.892690+0000","last_fullsized":"2023-10-25T00:22:07.892690+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:57:51.018816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"72'5003","reported_seq":5078,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.705186+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-25T00:22:08.705186+0000","last_peered":"2023-10-25T00:22:08.705186+0000","last_clean":"2023-10-25T00:22:08.705186+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-25T00:22:08.705186+0000","last_undegraded":"2023-10-25T00:22:08.705186+0000","last_fullsized":"2023-10-25T00:22:08.705186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:20:16.596993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":13,"num_write":27120,"num_write_kb":350355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'5635","reported_seq":5708,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186387+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-25T00:21:39.186387+0000","last_peered":"2023-10-25T00:21:39.186387+0000","last_clean":"2023-10-25T00:21:39.186387+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-25T00:21:39.186387+0000","last_undegraded":"2023-10-25T00:21:39.186387+0000","last_fullsized":"2023-10-25T00:21:39.186387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30556,"num_write_kb":356140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"68'3","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.196592+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-25T00:21:39.196592+0000","last_peered":"2023-10-25T00:21:39.196592+0000","last_clean":"2023-10-25T00:21:39.196592+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-25T00:21:39.196592+0000","last_undegraded":"2023-10-25T00:21:39.196592+0000","last_fullsized":"2023-10-25T00:21:39.196592+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183369+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-25T00:21:39.183369+0000","last_peered":"2023-10-25T00:21:39.183369+0000","last_clean":"2023-10-25T00:21:39.183369+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-25T00:21:39.183369+0000","last_undegraded":"2023-10-25T00:21:39.183369+0000","last_fullsized":"2023-10-25T00:21:39.183369+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"72'2","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193585+0000","last_change":"2023-10-24T23:53:11.432666+0000","last_active":"2023-10-25T00:21:39.193585+0000","last_peered":"2023-10-25T00:21:39.193585+0000","last_clean":"2023-10-25T00:21:39.193585+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-25T00:21:39.193585+0000","last_undegraded":"2023-10-25T00:21:39.193585+0000","last_fullsized":"2023-10-25T00:21:39.193585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:17.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"72'3753","reported_seq":3826,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:09.504989+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-25T00:22:09.504989+0000","last_peered":"2023-10-25T00:22:09.504989+0000","last_clean":"2023-10-25T00:22:09.504989+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-25T00:22:09.504989+0000","last_undegraded":"2023-10-25T00:22:09.504989+0000","last_fullsized":"2023-10-25T00:22:09.504989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:42.131555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":21045,"num_write_kb":276986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'4101","reported_seq":4173,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.179395+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-25T00:21:19.179395+0000","last_peered":"2023-10-25T00:21:19.179395+0000","last_clean":"2023-10-25T00:21:19.179395+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-25T00:21:19.179395+0000","last_undegraded":"2023-10-25T00:21:19.179395+0000","last_fullsized":"2023-10-25T00:21:19.179395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22383,"num_write_kb":270023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"72'12","reported_seq":86,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.184571+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-25T00:21:39.184571+0000","last_peered":"2023-10-25T00:21:39.184571+0000","last_clean":"2023-10-25T00:21:39.184571+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-25T00:21:39.184571+0000","last_undegraded":"2023-10-25T00:21:39.184571+0000","last_fullsized":"2023-10-25T00:21:39.184571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193953+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-25T00:21:39.193953+0000","last_peered":"2023-10-25T00:21:39.193953+0000","last_clean":"2023-10-25T00:21:39.193953+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-25T00:21:39.193953+0000","last_undegraded":"2023-10-25T00:21:39.193953+0000","last_fullsized":"2023-10-25T00:21:39.193953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"72'248","reported_seq":339,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.188700+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-25T00:22:10.188700+0000","last_peered":"2023-10-25T00:22:10.188700+0000","last_clean":"2023-10-25T00:22:10.188700+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-25T00:22:10.188700+0000","last_undegraded":"2023-10-25T00:22:10.188700+0000","last_fullsized":"2023-10-25T00:22:10.188700+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:55:47.548432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":62,"num_write":607,"num_write_kb":223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"72'5236","reported_seq":5311,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.753672+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-25T00:22:08.753672+0000","last_peered":"2023-10-25T00:22:08.753672+0000","last_clean":"2023-10-25T00:22:08.753672+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-25T00:22:08.753672+0000","last_undegraded":"2023-10-25T00:22:08.753672+0000","last_fullsized":"2023-10-25T00:22:08.753672+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:11:14.431984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":29554,"num_write_kb":368820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'5444","reported_seq":5520,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.216863+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-25T00:21:19.216863+0000","last_peered":"2023-10-25T00:21:19.216863+0000","last_clean":"2023-10-25T00:21:19.216863+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-25T00:21:19.216863+0000","last_undegraded":"2023-10-25T00:21:19.216863+0000","last_fullsized":"2023-10-25T00:21:19.216863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28342,"num_write_kb":342931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"72'66","reported_seq":223,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.219689+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-25T00:22:10.219689+0000","last_peered":"2023-10-25T00:22:10.219689+0000","last_clean":"2023-10-25T00:22:10.219689+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-25T00:22:10.219689+0000","last_undegraded":"2023-10-25T00:22:10.219689+0000","last_fullsized":"2023-10-25T00:22:10.219689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":125,"num_write":61,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193375+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-25T00:21:19.193375+0000","last_peered":"2023-10-25T00:21:19.193375+0000","last_clean":"2023-10-25T00:21:19.193375+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-25T00:21:19.193375+0000","last_undegraded":"2023-10-25T00:21:19.193375+0000","last_fullsized":"2023-10-25T00:21:19.193375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"72'9","reported_seq":123,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760947+0000","last_change":"2023-10-24T23:53:11.436603+0000","last_active":"2023-10-25T00:22:08.760947+0000","last_peered":"2023-10-25T00:22:08.760947+0000","last_clean":"2023-10-25T00:22:08.760947+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-25T00:22:08.760947+0000","last_undegraded":"2023-10-25T00:22:08.760947+0000","last_fullsized":"2023-10-25T00:22:08.760947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:09:05.948928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":135,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"72'3666","reported_seq":3735,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.792399+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-25T00:22:08.792399+0000","last_peered":"2023-10-25T00:22:08.792399+0000","last_clean":"2023-10-25T00:22:08.792399+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-25T00:22:08.792399+0000","last_undegraded":"2023-10-25T00:22:08.792399+0000","last_fullsized":"2023-10-25T00:22:08.792399+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:01:59.639080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20519,"num_write_kb":254391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"68'4182","reported_seq":4256,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187475+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-25T00:21:39.187475+0000","last_peered":"2023-10-25T00:21:39.187475+0000","last_clean":"2023-10-25T00:21:39.187475+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-25T00:21:39.187475+0000","last_undegraded":"2023-10-25T00:21:39.187475+0000","last_fullsized":"2023-10-25T00:21:39.187475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22839,"num_write_kb":259586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'4","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182378+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-25T00:21:39.182378+0000","last_peered":"2023-10-25T00:21:39.182378+0000","last_clean":"2023-10-25T00:21:39.182378+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-25T00:21:39.182378+0000","last_undegraded":"2023-10-25T00:21:39.182378+0000","last_fullsized":"2023-10-25T00:21:39.182378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183752+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-25T00:21:39.183752+0000","last_peered":"2023-10-25T00:21:39.183752+0000","last_clean":"2023-10-25T00:21:39.183752+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-25T00:21:39.183752+0000","last_undegraded":"2023-10-25T00:21:39.183752+0000","last_fullsized":"2023-10-25T00:21:39.183752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"43'161","reported_seq":283,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193737+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-25T00:21:19.193737+0000","last_peered":"2023-10-25T00:21:19.193737+0000","last_clean":"2023-10-25T00:21:19.193737+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-25T00:21:19.193737+0000","last_undegraded":"2023-10-25T00:21:19.193737+0000","last_fullsized":"2023-10-25T00:21:19.193737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"72'2","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784850+0000","last_change":"2023-10-24T23:53:11.451073+0000","last_active":"2023-10-25T00:22:08.784850+0000","last_peered":"2023-10-25T00:22:08.784850+0000","last_clean":"2023-10-25T00:22:08.784850+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-25T00:22:08.784850+0000","last_undegraded":"2023-10-25T00:22:08.784850+0000","last_fullsized":"2023-10-25T00:22:08.784850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:32.324710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"72'4105","reported_seq":4177,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793407+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-25T00:22:08.793407+0000","last_peered":"2023-10-25T00:22:08.793407+0000","last_clean":"2023-10-25T00:22:08.793407+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-25T00:22:08.793407+0000","last_undegraded":"2023-10-25T00:22:08.793407+0000","last_fullsized":"2023-10-25T00:22:08.793407+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:04.125835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":19,"num_write":23147,"num_write_kb":297959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'5705","reported_seq":5779,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189167+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-25T00:21:39.189167+0000","last_peered":"2023-10-25T00:21:39.189167+0000","last_clean":"2023-10-25T00:21:39.189167+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-25T00:21:39.189167+0000","last_undegraded":"2023-10-25T00:21:39.189167+0000","last_fullsized":"2023-10-25T00:21:39.189167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":29845,"num_write_kb":357286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"68'5","reported_seq":78,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183009+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-25T00:21:19.183009+0000","last_peered":"2023-10-25T00:21:19.183009+0000","last_clean":"2023-10-25T00:21:19.183009+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-25T00:21:19.183009+0000","last_undegraded":"2023-10-25T00:21:19.183009+0000","last_fullsized":"2023-10-25T00:21:19.183009+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"72'7","reported_seq":121,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.761062+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-25T00:22:08.761062+0000","last_peered":"2023-10-25T00:22:08.761062+0000","last_clean":"2023-10-25T00:22:08.761062+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-25T00:22:08.761062+0000","last_undegraded":"2023-10-25T00:22:08.761062+0000","last_fullsized":"2023-10-25T00:22:08.761062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:06:53.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"72'4234","reported_seq":4305,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.780516+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-25T00:22:08.780516+0000","last_peered":"2023-10-25T00:22:08.780516+0000","last_clean":"2023-10-25T00:22:08.780516+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-25T00:22:08.780516+0000","last_undegraded":"2023-10-25T00:22:08.780516+0000","last_fullsized":"2023-10-25T00:22:08.780516+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:41.905772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":15,"num_write":23173,"num_write_kb":302022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"68'4052","reported_seq":4125,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197631+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-25T00:21:39.197631+0000","last_peered":"2023-10-25T00:21:39.197631+0000","last_clean":"2023-10-25T00:21:39.197631+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-25T00:21:39.197631+0000","last_undegraded":"2023-10-25T00:21:39.197631+0000","last_fullsized":"2023-10-25T00:21:39.197631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21486,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"68'2","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.204255+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-25T00:21:19.204255+0000","last_peered":"2023-10-25T00:21:19.204255+0000","last_clean":"2023-10-25T00:21:19.204255+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-25T00:21:19.204255+0000","last_undegraded":"2023-10-25T00:21:19.204255+0000","last_fullsized":"2023-10-25T00:21:19.204255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"72'4","reported_seq":94,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.751351+0000","last_change":"2023-10-24T23:53:11.458463+0000","last_active":"2023-10-25T00:22:08.751351+0000","last_peered":"2023-10-25T00:22:08.751351+0000","last_clean":"2023-10-25T00:22:08.751351+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-25T00:22:08.751351+0000","last_undegraded":"2023-10-25T00:22:08.751351+0000","last_fullsized":"2023-10-25T00:22:08.751351+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:11:45.040156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"72'4690","reported_seq":4761,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.556236+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-25T00:22:08.556236+0000","last_peered":"2023-10-25T00:22:08.556236+0000","last_clean":"2023-10-25T00:22:08.556236+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-25T00:22:08.556236+0000","last_undegraded":"2023-10-25T00:22:08.556236+0000","last_fullsized":"2023-10-25T00:22:08.556236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:51.707178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24994,"num_write_kb":341798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'4780","reported_seq":4855,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194222+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-25T00:21:39.194222+0000","last_peered":"2023-10-25T00:21:39.194222+0000","last_clean":"2023-10-25T00:21:39.194222+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-25T00:21:39.194222+0000","last_undegraded":"2023-10-25T00:21:39.194222+0000","last_fullsized":"2023-10-25T00:21:39.194222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25083,"num_write_kb":293163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'3","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:20.638748+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-25T00:21:20.638748+0000","last_peered":"2023-10-25T00:21:20.638748+0000","last_clean":"2023-10-25T00:21:20.638748+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-25T00:21:20.638748+0000","last_undegraded":"2023-10-25T00:21:20.638748+0000","last_fullsized":"2023-10-25T00:21:20.638748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"72'5","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.745978+0000","last_change":"2023-10-24T23:53:11.448675+0000","last_active":"2023-10-25T00:22:08.745978+0000","last_peered":"2023-10-25T00:22:08.745978+0000","last_clean":"2023-10-25T00:22:08.745978+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-25T00:22:08.745978+0000","last_undegraded":"2023-10-25T00:22:08.745978+0000","last_fullsized":"2023-10-25T00:22:08.745978+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:01.502343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"72'3596","reported_seq":3668,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.775009+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-25T00:22:08.775009+0000","last_peered":"2023-10-25T00:22:08.775009+0000","last_clean":"2023-10-25T00:22:08.775009+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-25T00:22:08.775009+0000","last_undegraded":"2023-10-25T00:22:08.775009+0000","last_fullsized":"2023-10-25T00:22:08.775009+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:54.478897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":19820,"num_write_kb":259784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"68'4970","reported_seq":5044,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.201300+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-25T00:21:39.201300+0000","last_peered":"2023-10-25T00:21:39.201300+0000","last_clean":"2023-10-25T00:21:39.201300+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-25T00:21:39.201300+0000","last_undegraded":"2023-10-25T00:21:39.201300+0000","last_fullsized":"2023-10-25T00:21:39.201300+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26721,"num_write_kb":311966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"72'1464","reported_seq":1588,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.213604+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-25T00:22:10.213604+0000","last_peered":"2023-10-25T00:22:10.213604+0000","last_clean":"2023-10-25T00:22:10.213604+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-25T00:22:10.213604+0000","last_undegraded":"2023-10-25T00:22:10.213604+0000","last_fullsized":"2023-10-25T00:22:10.213604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1464,"log_dups_size":0,"ondisk_log_size":1464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":43700,"num_write":1439,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"72'6","reported_seq":96,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.262467+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-25T00:22:08.262467+0000","last_peered":"2023-10-25T00:22:08.262467+0000","last_clean":"2023-10-25T00:22:08.262467+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-25T00:22:08.262467+0000","last_undegraded":"2023-10-25T00:22:08.262467+0000","last_fullsized":"2023-10-25T00:22:08.262467+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:58.545886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"72'3548","reported_seq":3620,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.608884+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-25T00:22:08.608884+0000","last_peered":"2023-10-25T00:22:08.608884+0000","last_clean":"2023-10-25T00:22:08.608884+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-25T00:22:08.608884+0000","last_undegraded":"2023-10-25T00:22:08.608884+0000","last_fullsized":"2023-10-25T00:22:08.608884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:05.595887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20243,"num_write_kb":261539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"68'3393","reported_seq":3469,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194393+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-25T00:21:39.194393+0000","last_peered":"2023-10-25T00:21:39.194393+0000","last_clean":"2023-10-25T00:21:39.194393+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-25T00:21:39.194393+0000","last_undegraded":"2023-10-25T00:21:39.194393+0000","last_fullsized":"2023-10-25T00:21:39.194393+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17500,"num_write_kb":217638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"68'5","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187988+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-25T00:21:39.187988+0000","last_peered":"2023-10-25T00:21:39.187988+0000","last_clean":"2023-10-25T00:21:39.187988+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-25T00:21:39.187988+0000","last_undegraded":"2023-10-25T00:21:39.187988+0000","last_fullsized":"2023-10-25T00:21:39.187988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"72'3","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.738581+0000","last_change":"2023-10-24T23:53:11.454658+0000","last_active":"2023-10-25T00:22:08.738581+0000","last_peered":"2023-10-25T00:22:08.738581+0000","last_clean":"2023-10-25T00:22:08.738581+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-25T00:22:08.738581+0000","last_undegraded":"2023-10-25T00:22:08.738581+0000","last_fullsized":"2023-10-25T00:22:08.738581+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:43:43.869069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"72'3584","reported_seq":3656,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.616239+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-25T00:22:08.616239+0000","last_peered":"2023-10-25T00:22:08.616239+0000","last_clean":"2023-10-25T00:22:08.616239+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-25T00:22:08.616239+0000","last_undegraded":"2023-10-25T00:22:08.616239+0000","last_fullsized":"2023-10-25T00:22:08.616239+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:09:40.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20504,"num_write_kb":254394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'5094","reported_seq":5172,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:20.638687+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-25T00:21:20.638687+0000","last_peered":"2023-10-25T00:21:20.638687+0000","last_clean":"2023-10-25T00:21:20.638687+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-25T00:21:20.638687+0000","last_undegraded":"2023-10-25T00:21:20.638687+0000","last_fullsized":"2023-10-25T00:21:20.638687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27763,"num_write_kb":328490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'6","reported_seq":101,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199601+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-25T00:21:39.199601+0000","last_peered":"2023-10-25T00:21:39.199601+0000","last_clean":"2023-10-25T00:21:39.199601+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-25T00:21:39.199601+0000","last_undegraded":"2023-10-25T00:21:39.199601+0000","last_fullsized":"2023-10-25T00:21:39.199601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"72'1438","reported_seq":1554,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.763905+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-25T00:22:08.763905+0000","last_peered":"2023-10-25T00:22:08.763905+0000","last_clean":"2023-10-25T00:22:08.763905+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-25T00:22:08.763905+0000","last_undegraded":"2023-10-25T00:22:08.763905+0000","last_fullsized":"2023-10-25T00:22:08.763905+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1438,"log_dups_size":0,"ondisk_log_size":1438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:02.591627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":35275,"num_write":1412,"num_write_kb":4487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"72'3292","reported_seq":3364,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.791717+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-25T00:22:08.791717+0000","last_peered":"2023-10-25T00:22:08.791717+0000","last_clean":"2023-10-25T00:22:08.791717+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-25T00:22:08.791717+0000","last_undegraded":"2023-10-25T00:22:08.791717+0000","last_fullsized":"2023-10-25T00:22:08.791717+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:43.015050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17853,"num_write_kb":243602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'4242","reported_seq":4315,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.203827+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-25T00:21:19.203827+0000","last_peered":"2023-10-25T00:21:19.203827+0000","last_clean":"2023-10-25T00:21:19.203827+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-25T00:21:19.203827+0000","last_undegraded":"2023-10-25T00:21:19.203827+0000","last_fullsized":"2023-10-25T00:21:19.203827+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22687,"num_write_kb":275346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'7","reported_seq":124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.498042+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-25T00:21:42.498042+0000","last_peered":"2023-10-25T00:21:42.498042+0000","last_clean":"2023-10-25T00:21:42.498042+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-25T00:21:42.498042+0000","last_undegraded":"2023-10-25T00:21:42.498042+0000","last_fullsized":"2023-10-25T00:21:42.498042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"72'5","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.259848+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-25T00:22:08.259848+0000","last_peered":"2023-10-25T00:22:08.259848+0000","last_clean":"2023-10-25T00:22:08.259848+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-25T00:22:08.259848+0000","last_undegraded":"2023-10-25T00:22:08.259848+0000","last_fullsized":"2023-10-25T00:22:08.259848+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:59.068359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"72'4091","reported_seq":4163,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.758376+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-25T00:22:08.758376+0000","last_peered":"2023-10-25T00:22:08.758376+0000","last_clean":"2023-10-25T00:22:08.758376+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-25T00:22:08.758376+0000","last_undegraded":"2023-10-25T00:22:08.758376+0000","last_fullsized":"2023-10-25T00:22:08.758376+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:13:03.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22779,"num_write_kb":288098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'4310","reported_seq":4383,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.176404+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-25T00:21:19.176404+0000","last_peered":"2023-10-25T00:21:19.176404+0000","last_clean":"2023-10-25T00:21:19.176404+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-25T00:21:19.176404+0000","last_undegraded":"2023-10-25T00:21:19.176404+0000","last_fullsized":"2023-10-25T00:21:19.176404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23791,"num_write_kb":283914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'4","reported_seq":99,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199642+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-25T00:21:39.199642+0000","last_peered":"2023-10-25T00:21:39.199642+0000","last_clean":"2023-10-25T00:21:39.199642+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-25T00:21:39.199642+0000","last_undegraded":"2023-10-25T00:21:39.199642+0000","last_fullsized":"2023-10-25T00:21:39.199642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"72'2","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.702572+0000","last_change":"2023-10-24T23:53:11.451013+0000","last_active":"2023-10-25T00:22:08.702572+0000","last_peered":"2023-10-25T00:22:08.702572+0000","last_clean":"2023-10-25T00:22:08.702572+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-25T00:22:08.702572+0000","last_undegraded":"2023-10-25T00:22:08.702572+0000","last_fullsized":"2023-10-25T00:22:08.702572+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:52:22.503023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"72'3695","reported_seq":3768,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.635013+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-25T00:22:08.635013+0000","last_peered":"2023-10-25T00:22:08.635013+0000","last_clean":"2023-10-25T00:22:08.635013+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-25T00:22:08.635013+0000","last_undegraded":"2023-10-25T00:22:08.635013+0000","last_fullsized":"2023-10-25T00:22:08.635013+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:05:30.149693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21426,"num_write_kb":254037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.776347+0000","last_change":"2023-10-24T23:53:11.434538+0000","last_active":"2023-10-25T00:22:08.776347+0000","last_peered":"2023-10-25T00:22:08.776347+0000","last_clean":"2023-10-25T00:22:08.776347+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-25T00:22:08.776347+0000","last_undegraded":"2023-10-25T00:22:08.776347+0000","last_fullsized":"2023-10-25T00:22:08.776347+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:55.179253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"72'15","reported_seq":196,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.481650+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-25T00:21:42.481650+0000","last_peered":"2023-10-25T00:21:42.481650+0000","last_clean":"2023-10-25T00:21:42.481650+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-25T00:21:42.481650+0000","last_undegraded":"2023-10-25T00:21:42.481650+0000","last_fullsized":"2023-10-25T00:21:42.481650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":327,"num_read_kb":327,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'5249","reported_seq":5329,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183983+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-25T00:21:39.183983+0000","last_peered":"2023-10-25T00:21:39.183983+0000","last_clean":"2023-10-25T00:21:39.183983+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-25T00:21:39.183983+0000","last_undegraded":"2023-10-25T00:21:39.183983+0000","last_fullsized":"2023-10-25T00:21:39.183983+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":27646,"num_write_kb":332285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"72'4380","reported_seq":4450,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.769070+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-25T00:22:08.769070+0000","last_peered":"2023-10-25T00:22:08.769070+0000","last_clean":"2023-10-25T00:22:08.769070+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-25T00:22:08.769070+0000","last_undegraded":"2023-10-25T00:22:08.769070+0000","last_fullsized":"2023-10-25T00:22:08.769070+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:33:08.373963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23662,"num_write_kb":324338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"72'5","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.785854+0000","last_change":"2023-10-24T23:53:11.433387+0000","last_active":"2023-10-25T00:22:08.785854+0000","last_peered":"2023-10-25T00:22:08.785854+0000","last_clean":"2023-10-25T00:22:08.785854+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-25T00:22:08.785854+0000","last_undegraded":"2023-10-25T00:22:08.785854+0000","last_fullsized":"2023-10-25T00:22:08.785854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:29.148450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"68'5","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.194080+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-25T00:21:19.194080+0000","last_peered":"2023-10-25T00:21:19.194080+0000","last_clean":"2023-10-25T00:21:19.194080+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-25T00:21:19.194080+0000","last_undegraded":"2023-10-25T00:21:19.194080+0000","last_fullsized":"2023-10-25T00:21:19.194080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'4442","reported_seq":4514,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195509+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-25T00:21:39.195509+0000","last_peered":"2023-10-25T00:21:39.195509+0000","last_clean":"2023-10-25T00:21:39.195509+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-25T00:21:39.195509+0000","last_undegraded":"2023-10-25T00:21:39.195509+0000","last_fullsized":"2023-10-25T00:21:39.195509+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22810,"num_write_kb":296225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"72'4076","reported_seq":4148,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784809+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-25T00:22:08.784809+0000","last_peered":"2023-10-25T00:22:08.784809+0000","last_clean":"2023-10-25T00:22:08.784809+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-25T00:22:08.784809+0000","last_undegraded":"2023-10-25T00:22:08.784809+0000","last_fullsized":"2023-10-25T00:22:08.784809+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:33:06.172412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22055,"num_write_kb":297609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"68'3","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.211351+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-25T00:21:19.211351+0000","last_peered":"2023-10-25T00:21:19.211351+0000","last_clean":"2023-10-25T00:21:19.211351+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-25T00:21:19.211351+0000","last_undegraded":"2023-10-25T00:21:19.211351+0000","last_fullsized":"2023-10-25T00:21:19.211351+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'4101","reported_seq":4174,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.188446+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-25T00:21:39.188446+0000","last_peered":"2023-10-25T00:21:39.188446+0000","last_clean":"2023-10-25T00:21:39.188446+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-25T00:21:39.188446+0000","last_undegraded":"2023-10-25T00:21:39.188446+0000","last_fullsized":"2023-10-25T00:21:39.188446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22343,"num_write_kb":253486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"72'393","reported_seq":573,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:02.239640+0000","last_change":"2023-10-24T23:53:11.446979+0000","last_active":"2023-10-25T00:22:02.239640+0000","last_peered":"2023-10-25T00:22:02.239640+0000","last_clean":"2023-10-25T00:22:02.239640+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-25T00:22:02.239640+0000","last_undegraded":"2023-10-25T00:22:02.239640+0000","last_fullsized":"2023-10-25T00:22:02.239640+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:54:52.843631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":86197,"num_write":389,"num_write_kb":4306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"72'4606","reported_seq":4676,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784571+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-25T00:22:08.784571+0000","last_peered":"2023-10-25T00:22:08.784571+0000","last_clean":"2023-10-25T00:22:08.784571+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-25T00:22:08.784571+0000","last_undegraded":"2023-10-25T00:22:08.784571+0000","last_fullsized":"2023-10-25T00:22:08.784571+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:18:48.679682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25763,"num_write_kb":316477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"72'11","reported_seq":127,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:40.179752+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-25T00:21:40.179752+0000","last_peered":"2023-10-25T00:21:40.179752+0000","last_clean":"2023-10-25T00:21:40.179752+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-25T00:21:40.179752+0000","last_undegraded":"2023-10-25T00:21:40.179752+0000","last_fullsized":"2023-10-25T00:21:40.179752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'5680","reported_seq":5756,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183466+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-25T00:21:19.183466+0000","last_peered":"2023-10-25T00:21:19.183466+0000","last_clean":"2023-10-25T00:21:19.183466+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-25T00:21:19.183466+0000","last_undegraded":"2023-10-25T00:21:19.183466+0000","last_fullsized":"2023-10-25T00:21:19.183466+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29749,"num_write_kb":360962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"72'38","reported_seq":172,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:09.508467+0000","last_change":"2023-10-24T23:53:11.436964+0000","last_active":"2023-10-25T00:22:09.508467+0000","last_peered":"2023-10-25T00:22:09.508467+0000","last_clean":"2023-10-25T00:22:09.508467+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-25T00:22:09.508467+0000","last_undegraded":"2023-10-25T00:22:09.508467+0000","last_fullsized":"2023-10-25T00:22:09.508467+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:04:15.366966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":39,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"72'4808","reported_seq":4884,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.778926+0000","last_change":"2023-10-24T23:53:11.814304+0000","last_active":"2023-10-25T00:22:08.778926+0000","last_peered":"2023-10-25T00:22:08.778926+0000","last_clean":"2023-10-25T00:22:08.778926+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-25T00:22:08.778926+0000","last_undegraded":"2023-10-25T00:22:08.778926+0000","last_fullsized":"2023-10-25T00:22:08.778926+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:16.635178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26552,"num_write_kb":336125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"68'4747","reported_seq":4822,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182709+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-25T00:21:39.182709+0000","last_peered":"2023-10-25T00:21:39.182709+0000","last_clean":"2023-10-25T00:21:39.182709+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-25T00:21:39.182709+0000","last_undegraded":"2023-10-25T00:21:39.182709+0000","last_fullsized":"2023-10-25T00:21:39.182709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24847,"num_write_kb":316685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"72'4283","reported_seq":4356,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.589765+0000","last_change":"2023-10-24T23:53:11.807096+0000","last_active":"2023-10-25T00:22:08.589765+0000","last_peered":"2023-10-25T00:22:08.589765+0000","last_clean":"2023-10-25T00:22:08.589765+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-25T00:22:08.589765+0000","last_undegraded":"2023-10-25T00:22:08.589765+0000","last_fullsized":"2023-10-25T00:22:08.589765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:25.144578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23853,"num_write_kb":296504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"68'4036","reported_seq":4111,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183566+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-25T00:21:39.183566+0000","last_peered":"2023-10-25T00:21:39.183566+0000","last_clean":"2023-10-25T00:21:39.183566+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-25T00:21:39.183566+0000","last_undegraded":"2023-10-25T00:21:39.183566+0000","last_fullsized":"2023-10-25T00:21:39.183566+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21233,"num_write_kb":273467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"72'4276","reported_seq":4349,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.743483+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-25T00:22:08.743483+0000","last_peered":"2023-10-25T00:22:08.743483+0000","last_clean":"2023-10-25T00:22:08.743483+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-25T00:22:08.743483+0000","last_undegraded":"2023-10-25T00:22:08.743483+0000","last_fullsized":"2023-10-25T00:22:08.743483+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:38.946585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23362,"num_write_kb":310817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"68'6201","reported_seq":6278,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.198522+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-25T00:21:39.198522+0000","last_peered":"2023-10-25T00:21:39.198522+0000","last_clean":"2023-10-25T00:21:39.198522+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-25T00:21:39.198522+0000","last_undegraded":"2023-10-25T00:21:39.198522+0000","last_fullsized":"2023-10-25T00:21:39.198522+0000","mapping_epoch":21,"log_start":"62'1000","ondisk_log_start":"62'1000","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33486,"num_write_kb":384441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"72'4018","reported_seq":4092,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.764145+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-25T00:22:08.764145+0000","last_peered":"2023-10-25T00:22:08.764145+0000","last_clean":"2023-10-25T00:22:08.764145+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-25T00:22:08.764145+0000","last_undegraded":"2023-10-25T00:22:08.764145+0000","last_fullsized":"2023-10-25T00:22:08.764145+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:48:24.911279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":21920,"num_write_kb":288651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"68'4522","reported_seq":4598,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186452+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-25T00:21:39.186452+0000","last_peered":"2023-10-25T00:21:39.186452+0000","last_clean":"2023-10-25T00:21:39.186452+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-25T00:21:39.186452+0000","last_undegraded":"2023-10-25T00:21:39.186452+0000","last_fullsized":"2023-10-25T00:21:39.186452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24476,"num_write_kb":282500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"72'4209","reported_seq":4279,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760824+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-25T00:22:08.760824+0000","last_peered":"2023-10-25T00:22:08.760824+0000","last_clean":"2023-10-25T00:22:08.760824+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-25T00:22:08.760824+0000","last_undegraded":"2023-10-25T00:22:08.760824+0000","last_fullsized":"2023-10-25T00:22:08.760824+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:21.165755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24141,"num_write_kb":281325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"68'4915","reported_seq":4988,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180147+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-25T00:21:19.180147+0000","last_peered":"2023-10-25T00:21:19.180147+0000","last_clean":"2023-10-25T00:21:19.180147+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-25T00:21:19.180147+0000","last_undegraded":"2023-10-25T00:21:19.180147+0000","last_fullsized":"2023-10-25T00:21:19.180147+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26161,"num_write_kb":314084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"72'4933","reported_seq":5003,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.718975+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-25T00:22:08.718975+0000","last_peered":"2023-10-25T00:22:08.718975+0000","last_clean":"2023-10-25T00:22:08.718975+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-25T00:22:08.718975+0000","last_undegraded":"2023-10-25T00:22:08.718975+0000","last_fullsized":"2023-10-25T00:22:08.718975+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26695,"num_write_kb":352905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"68'5055","reported_seq":5129,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180248+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-25T00:21:19.180248+0000","last_peered":"2023-10-25T00:21:19.180248+0000","last_clean":"2023-10-25T00:21:19.180248+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-25T00:21:19.180248+0000","last_undegraded":"2023-10-25T00:21:19.180248+0000","last_fullsized":"2023-10-25T00:21:19.180248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27516,"num_write_kb":327406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"72'4083","reported_seq":4157,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.731484+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-25T00:22:08.731484+0000","last_peered":"2023-10-25T00:22:08.731484+0000","last_clean":"2023-10-25T00:22:08.731484+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-25T00:22:08.731484+0000","last_undegraded":"2023-10-25T00:22:08.731484+0000","last_fullsized":"2023-10-25T00:22:08.731484+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:54.422365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":22892,"num_write_kb":296872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"68'4777","reported_seq":4852,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178413+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-25T00:21:19.178413+0000","last_peered":"2023-10-25T00:21:19.178413+0000","last_clean":"2023-10-25T00:21:19.178413+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-25T00:21:19.178413+0000","last_undegraded":"2023-10-25T00:21:19.178413+0000","last_fullsized":"2023-10-25T00:21:19.178413+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25812,"num_write_kb":300181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"72'4618","reported_seq":4693,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.741877+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-25T00:22:08.741877+0000","last_peered":"2023-10-25T00:22:08.741877+0000","last_clean":"2023-10-25T00:22:08.741877+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-25T00:22:08.741877+0000","last_undegraded":"2023-10-25T00:22:08.741877+0000","last_fullsized":"2023-10-25T00:22:08.741877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:01:11.838035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":26180,"num_write_kb":326342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"68'5340","reported_seq":5413,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178233+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-25T00:21:19.178233+0000","last_peered":"2023-10-25T00:21:19.178233+0000","last_clean":"2023-10-25T00:21:19.178233+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-25T00:21:19.178233+0000","last_undegraded":"2023-10-25T00:21:19.178233+0000","last_fullsized":"2023-10-25T00:21:19.178233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28701,"num_write_kb":348521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"72'4171","reported_seq":4242,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.497360+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-25T00:22:08.497360+0000","last_peered":"2023-10-25T00:22:08.497360+0000","last_clean":"2023-10-25T00:22:08.497360+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-25T00:22:08.497360+0000","last_undegraded":"2023-10-25T00:22:08.497360+0000","last_fullsized":"2023-10-25T00:22:08.497360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:09:25.424344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22551,"num_write_kb":303498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"68'4493","reported_seq":4567,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.192644+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-25T00:21:19.192644+0000","last_peered":"2023-10-25T00:21:19.192644+0000","last_clean":"2023-10-25T00:21:19.192644+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-25T00:21:19.192644+0000","last_undegraded":"2023-10-25T00:21:19.192644+0000","last_fullsized":"2023-10-25T00:21:19.192644+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23797,"num_write_kb":300260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"72'4763","reported_seq":4833,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.772760+0000","last_change":"2023-10-24T23:53:44.474203+0000","last_active":"2023-10-25T00:22:08.772760+0000","last_peered":"2023-10-25T00:22:08.772760+0000","last_clean":"2023-10-25T00:22:08.772760+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-25T00:22:08.772760+0000","last_undegraded":"2023-10-25T00:22:08.772760+0000","last_fullsized":"2023-10-25T00:22:08.772760+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:16:50.292761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":25751,"num_write_kb":345225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"68'4946","reported_seq":5019,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183160+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-25T00:21:39.183160+0000","last_peered":"2023-10-25T00:21:39.183160+0000","last_clean":"2023-10-25T00:21:39.183160+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-25T00:21:39.183160+0000","last_undegraded":"2023-10-25T00:21:39.183160+0000","last_fullsized":"2023-10-25T00:21:39.183160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26583,"num_write_kb":315633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"72'4352","reported_seq":4423,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.762501+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-25T00:22:08.762501+0000","last_peered":"2023-10-25T00:22:08.762501+0000","last_clean":"2023-10-25T00:22:08.762501+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-25T00:22:08.762501+0000","last_undegraded":"2023-10-25T00:22:08.762501+0000","last_fullsized":"2023-10-25T00:22:08.762501+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:10:20.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23957,"num_write_kb":301052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"68'5026","reported_seq":5099,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193252+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-25T00:21:39.193252+0000","last_peered":"2023-10-25T00:21:39.193252+0000","last_clean":"2023-10-25T00:21:39.193252+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-25T00:21:39.193252+0000","last_undegraded":"2023-10-25T00:21:39.193252+0000","last_fullsized":"2023-10-25T00:21:39.193252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26401,"num_write_kb":310586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"72'4393","reported_seq":4468,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.745975+0000","last_change":"2023-10-24T23:53:11.860133+0000","last_active":"2023-10-25T00:22:08.745975+0000","last_peered":"2023-10-25T00:22:08.745975+0000","last_clean":"2023-10-25T00:22:08.745975+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-25T00:22:08.745975+0000","last_undegraded":"2023-10-25T00:22:08.745975+0000","last_fullsized":"2023-10-25T00:22:08.745975+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:43.051362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":24752,"num_write_kb":296109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"68'5620","reported_seq":5697,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197727+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-25T00:21:39.197727+0000","last_peered":"2023-10-25T00:21:39.197727+0000","last_clean":"2023-10-25T00:21:39.197727+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-25T00:21:39.197727+0000","last_undegraded":"2023-10-25T00:21:39.197727+0000","last_fullsized":"2023-10-25T00:21:39.197727+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29863,"num_write_kb":337169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"72'4340","reported_seq":4414,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.730151+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-25T00:22:08.730151+0000","last_peered":"2023-10-25T00:22:08.730151+0000","last_clean":"2023-10-25T00:22:08.730151+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-25T00:22:08.730151+0000","last_undegraded":"2023-10-25T00:22:08.730151+0000","last_fullsized":"2023-10-25T00:22:08.730151+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:39:14.947372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":36,"num_write":23367,"num_write_kb":302669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"68'4464","reported_seq":4536,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.200961+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-25T00:21:39.200961+0000","last_peered":"2023-10-25T00:21:39.200961+0000","last_clean":"2023-10-25T00:21:39.200961+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-25T00:21:39.200961+0000","last_undegraded":"2023-10-25T00:21:39.200961+0000","last_fullsized":"2023-10-25T00:21:39.200961+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24372,"num_write_kb":278074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"72'5589","reported_seq":5659,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.768377+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-25T00:22:08.768377+0000","last_peered":"2023-10-25T00:22:08.768377+0000","last_clean":"2023-10-25T00:22:08.768377+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-25T00:22:08.768377+0000","last_undegraded":"2023-10-25T00:22:08.768377+0000","last_fullsized":"2023-10-25T00:22:08.768377+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:37.413259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":30206,"num_write_kb":399557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"68'4542","reported_seq":4615,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.196542+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-25T00:21:39.196542+0000","last_peered":"2023-10-25T00:21:39.196542+0000","last_clean":"2023-10-25T00:21:39.196542+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-25T00:21:39.196542+0000","last_undegraded":"2023-10-25T00:21:39.196542+0000","last_fullsized":"2023-10-25T00:21:39.196542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23889,"num_write_kb":292323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"72'3430","reported_seq":3501,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.718898+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-25T00:22:08.718898+0000","last_peered":"2023-10-25T00:22:08.718898+0000","last_clean":"2023-10-25T00:22:08.718898+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-25T00:22:08.718898+0000","last_undegraded":"2023-10-25T00:22:08.718898+0000","last_fullsized":"2023-10-25T00:22:08.718898+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:31.344724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19411,"num_write_kb":241862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"68'4719","reported_seq":4792,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.207948+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-25T00:21:19.207948+0000","last_peered":"2023-10-25T00:21:19.207948+0000","last_clean":"2023-10-25T00:21:19.207948+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-25T00:21:19.207948+0000","last_undegraded":"2023-10-25T00:21:19.207948+0000","last_fullsized":"2023-10-25T00:21:19.207948+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25020,"num_write_kb":305197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"72'4169","reported_seq":4237,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.618436+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-25T00:22:08.618436+0000","last_peered":"2023-10-25T00:22:08.618436+0000","last_clean":"2023-10-25T00:22:08.618436+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-25T00:22:08.618436+0000","last_undegraded":"2023-10-25T00:22:08.618436+0000","last_fullsized":"2023-10-25T00:22:08.618436+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23235,"num_write_kb":306175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"68'5952","reported_seq":6033,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.215357+0000","last_change":"2023-10-24T23:53:44.629959+0000","last_active":"2023-10-25T00:21:19.215357+0000","last_peered":"2023-10-25T00:21:19.215357+0000","last_clean":"2023-10-25T00:21:19.215357+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-25T00:21:19.215357+0000","last_undegraded":"2023-10-25T00:21:19.215357+0000","last_fullsized":"2023-10-25T00:21:19.215357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:59:36.474350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32341,"num_write_kb":375441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"72'3717","reported_seq":3786,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.786783+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-25T00:22:08.786783+0000","last_peered":"2023-10-25T00:22:08.786783+0000","last_clean":"2023-10-25T00:22:08.786783+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-25T00:22:08.786783+0000","last_undegraded":"2023-10-25T00:22:08.786783+0000","last_fullsized":"2023-10-25T00:22:08.786783+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:32:05.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21101,"num_write_kb":269108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"68'4825","reported_seq":4899,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.204063+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-25T00:21:19.204063+0000","last_peered":"2023-10-25T00:21:19.204063+0000","last_clean":"2023-10-25T00:21:19.204063+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-25T00:21:19.204063+0000","last_undegraded":"2023-10-25T00:21:19.204063+0000","last_fullsized":"2023-10-25T00:21:19.204063+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25319,"num_write_kb":294113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"72'3714","reported_seq":3786,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.739265+0000","last_change":"2023-10-24T23:53:11.847814+0000","last_active":"2023-10-25T00:22:08.739265+0000","last_peered":"2023-10-25T00:22:08.739265+0000","last_clean":"2023-10-25T00:22:08.739265+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-25T00:22:08.739265+0000","last_undegraded":"2023-10-25T00:22:08.739265+0000","last_fullsized":"2023-10-25T00:22:08.739265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:59.075858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21148,"num_write_kb":265324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"68'4878","reported_seq":4953,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.194001+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-25T00:21:19.194001+0000","last_peered":"2023-10-25T00:21:19.194001+0000","last_clean":"2023-10-25T00:21:19.194001+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-25T00:21:19.194001+0000","last_undegraded":"2023-10-25T00:21:19.194001+0000","last_fullsized":"2023-10-25T00:21:19.194001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26154,"num_write_kb":305558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"72'3434","reported_seq":3504,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.726527+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-25T00:22:08.726527+0000","last_peered":"2023-10-25T00:22:08.726527+0000","last_clean":"2023-10-25T00:22:08.726527+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-25T00:22:08.726527+0000","last_undegraded":"2023-10-25T00:22:08.726527+0000","last_fullsized":"2023-10-25T00:22:08.726527+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:25:12.525273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18449,"num_write_kb":262373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"68'4672","reported_seq":4746,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178513+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-25T00:21:19.178513+0000","last_peered":"2023-10-25T00:21:19.178513+0000","last_clean":"2023-10-25T00:21:19.178513+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-25T00:21:19.178513+0000","last_undegraded":"2023-10-25T00:21:19.178513+0000","last_fullsized":"2023-10-25T00:21:19.178513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25223,"num_write_kb":293792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"72'4013","reported_seq":4088,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.743804+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-25T00:22:08.743804+0000","last_peered":"2023-10-25T00:22:08.743804+0000","last_clean":"2023-10-25T00:22:08.743804+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-25T00:22:08.743804+0000","last_undegraded":"2023-10-25T00:22:08.743804+0000","last_fullsized":"2023-10-25T00:22:08.743804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:12:21.026996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":22066,"num_write_kb":297298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"68'3918","reported_seq":3990,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194748+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-25T00:21:39.194748+0000","last_peered":"2023-10-25T00:21:39.194748+0000","last_clean":"2023-10-25T00:21:39.194748+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-25T00:21:39.194748+0000","last_undegraded":"2023-10-25T00:21:39.194748+0000","last_fullsized":"2023-10-25T00:21:39.194748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21675,"num_write_kb":247000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"72'4401","reported_seq":4472,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.740526+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-25T00:22:08.740526+0000","last_peered":"2023-10-25T00:22:08.740526+0000","last_clean":"2023-10-25T00:22:08.740526+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-25T00:22:08.740526+0000","last_undegraded":"2023-10-25T00:22:08.740526+0000","last_fullsized":"2023-10-25T00:22:08.740526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:07.909021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":24294,"num_write_kb":326749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"68'3970","reported_seq":4042,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194610+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-25T00:21:39.194610+0000","last_peered":"2023-10-25T00:21:39.194610+0000","last_clean":"2023-10-25T00:21:39.194610+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-25T00:21:39.194610+0000","last_undegraded":"2023-10-25T00:21:39.194610+0000","last_fullsized":"2023-10-25T00:21:39.194610+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21202,"num_write_kb":257085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"72'5559","reported_seq":5636,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.789325+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-25T00:22:08.789325+0000","last_peered":"2023-10-25T00:22:08.789325+0000","last_clean":"2023-10-25T00:22:08.789325+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-25T00:22:08.789325+0000","last_undegraded":"2023-10-25T00:22:08.789325+0000","last_fullsized":"2023-10-25T00:22:08.789325+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:48.217889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":31202,"num_write_kb":391572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"68'4936","reported_seq":5009,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.185286+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-25T00:21:39.185286+0000","last_peered":"2023-10-25T00:21:39.185286+0000","last_clean":"2023-10-25T00:21:39.185286+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-25T00:21:39.185286+0000","last_undegraded":"2023-10-25T00:21:39.185286+0000","last_fullsized":"2023-10-25T00:21:39.185286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26134,"num_write_kb":309652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"72'3558","reported_seq":3632,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.781338+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-25T00:22:08.781338+0000","last_peered":"2023-10-25T00:22:08.781338+0000","last_clean":"2023-10-25T00:22:08.781338+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-25T00:22:08.781338+0000","last_undegraded":"2023-10-25T00:22:08.781338+0000","last_fullsized":"2023-10-25T00:22:08.781338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:10:18.973606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":34,"num_write":19715,"num_write_kb":243025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"68'5282","reported_seq":5355,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.192868+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-25T00:21:39.192868+0000","last_peered":"2023-10-25T00:21:39.192868+0000","last_clean":"2023-10-25T00:21:39.192868+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-25T00:21:39.192868+0000","last_undegraded":"2023-10-25T00:21:39.192868+0000","last_fullsized":"2023-10-25T00:21:39.192868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27848,"num_write_kb":324561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"72'3914","reported_seq":3989,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.624743+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-25T00:22:08.624743+0000","last_peered":"2023-10-25T00:22:08.624743+0000","last_clean":"2023-10-25T00:22:08.624743+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-25T00:22:08.624743+0000","last_undegraded":"2023-10-25T00:22:08.624743+0000","last_fullsized":"2023-10-25T00:22:08.624743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:24:45.536141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":20938,"num_write_kb":288796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"68'6616","reported_seq":6693,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194923+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-25T00:21:39.194923+0000","last_peered":"2023-10-25T00:21:39.194923+0000","last_clean":"2023-10-25T00:21:39.194923+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-25T00:21:39.194923+0000","last_undegraded":"2023-10-25T00:21:39.194923+0000","last_fullsized":"2023-10-25T00:21:39.194923+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36221,"num_write_kb":423916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"72'3536","reported_seq":3606,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.700035+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-25T00:22:08.700035+0000","last_peered":"2023-10-25T00:22:08.700035+0000","last_clean":"2023-10-25T00:22:08.700035+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-25T00:22:08.700035+0000","last_undegraded":"2023-10-25T00:22:08.700035+0000","last_fullsized":"2023-10-25T00:22:08.700035+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:06.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19276,"num_write_kb":253506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"68'4980","reported_seq":5050,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.176264+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-25T00:21:19.176264+0000","last_peered":"2023-10-25T00:21:19.176264+0000","last_clean":"2023-10-25T00:21:19.176264+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-25T00:21:19.176264+0000","last_undegraded":"2023-10-25T00:21:19.176264+0000","last_fullsized":"2023-10-25T00:21:19.176264+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26525,"num_write_kb":306963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"72'4160","reported_seq":4232,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.763999+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-25T00:22:08.763999+0000","last_peered":"2023-10-25T00:22:08.763999+0000","last_clean":"2023-10-25T00:22:08.763999+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-25T00:22:08.763999+0000","last_undegraded":"2023-10-25T00:22:08.763999+0000","last_fullsized":"2023-10-25T00:22:08.763999+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:56:02.798009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":21819,"num_write_kb":288439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"68'4434","reported_seq":4506,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193726+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-25T00:21:19.193726+0000","last_peered":"2023-10-25T00:21:19.193726+0000","last_clean":"2023-10-25T00:21:19.193726+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-25T00:21:19.193726+0000","last_undegraded":"2023-10-25T00:21:19.193726+0000","last_fullsized":"2023-10-25T00:21:19.193726+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23791,"num_write_kb":292031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"72'4195","reported_seq":4262,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.728890+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-25T00:22:08.728890+0000","last_peered":"2023-10-25T00:22:08.728890+0000","last_clean":"2023-10-25T00:22:08.728890+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-25T00:22:08.728890+0000","last_undegraded":"2023-10-25T00:22:08.728890+0000","last_fullsized":"2023-10-25T00:22:08.728890+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:40.168016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23000,"num_write_kb":285019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"68'4243","reported_seq":4314,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205516+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-25T00:21:19.205516+0000","last_peered":"2023-10-25T00:21:19.205516+0000","last_clean":"2023-10-25T00:21:19.205516+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-25T00:21:19.205516+0000","last_undegraded":"2023-10-25T00:21:19.205516+0000","last_fullsized":"2023-10-25T00:21:19.205516+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22439,"num_write_kb":266094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"72'4017","reported_seq":4085,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.783880+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-25T00:22:08.783880+0000","last_peered":"2023-10-25T00:22:08.783880+0000","last_clean":"2023-10-25T00:22:08.783880+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-25T00:22:08.783880+0000","last_undegraded":"2023-10-25T00:22:08.783880+0000","last_fullsized":"2023-10-25T00:22:08.783880+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:39.720375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22665,"num_write_kb":291707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"68'4509","reported_seq":4583,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.213920+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-25T00:21:19.213920+0000","last_peered":"2023-10-25T00:21:19.213920+0000","last_clean":"2023-10-25T00:21:19.213920+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-25T00:21:19.213920+0000","last_undegraded":"2023-10-25T00:21:19.213920+0000","last_fullsized":"2023-10-25T00:21:19.213920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23158,"num_write_kb":297883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"72'5235","reported_seq":5312,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.789212+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-25T00:22:08.789212+0000","last_peered":"2023-10-25T00:22:08.789212+0000","last_clean":"2023-10-25T00:22:08.789212+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-25T00:22:08.789212+0000","last_undegraded":"2023-10-25T00:22:08.789212+0000","last_fullsized":"2023-10-25T00:22:08.789212+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:51:40.960630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":15,"num_write":29235,"num_write_kb":367391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"68'4260","reported_seq":4335,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.221539+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-25T00:21:19.221539+0000","last_peered":"2023-10-25T00:21:19.221539+0000","last_clean":"2023-10-25T00:21:19.221539+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-25T00:21:19.221539+0000","last_undegraded":"2023-10-25T00:21:19.221539+0000","last_fullsized":"2023-10-25T00:21:19.221539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23357,"num_write_kb":265837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"72'4656","reported_seq":4735,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.794660+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-25T00:22:08.794660+0000","last_peered":"2023-10-25T00:22:08.794660+0000","last_clean":"2023-10-25T00:22:08.794660+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-25T00:22:08.794660+0000","last_undegraded":"2023-10-25T00:22:08.794660+0000","last_fullsized":"2023-10-25T00:22:08.794660+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:22:42.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":26109,"num_write_kb":317781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"68'4636","reported_seq":4713,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194609+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-25T00:21:39.194609+0000","last_peered":"2023-10-25T00:21:39.194609+0000","last_clean":"2023-10-25T00:21:39.194609+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-25T00:21:39.194609+0000","last_undegraded":"2023-10-25T00:21:39.194609+0000","last_fullsized":"2023-10-25T00:21:39.194609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25124,"num_write_kb":285314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"72'3117","reported_seq":3190,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.574961+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-25T00:22:08.574961+0000","last_peered":"2023-10-25T00:22:08.574961+0000","last_clean":"2023-10-25T00:22:08.574961+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-25T00:22:08.574961+0000","last_undegraded":"2023-10-25T00:22:08.574961+0000","last_fullsized":"2023-10-25T00:22:08.574961+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:47.658717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17240,"num_write_kb":232937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"68'5520","reported_seq":5604,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.192356+0000","last_change":"2023-10-24T23:53:44.630867+0000","last_active":"2023-10-25T00:21:39.192356+0000","last_peered":"2023-10-25T00:21:39.192356+0000","last_clean":"2023-10-25T00:21:39.192356+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-25T00:21:39.192356+0000","last_undegraded":"2023-10-25T00:21:39.192356+0000","last_fullsized":"2023-10-25T00:21:39.192356+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:49.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29665,"num_write_kb":368823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"72'3962","reported_seq":4034,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.769841+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-25T00:22:08.769841+0000","last_peered":"2023-10-25T00:22:08.769841+0000","last_clean":"2023-10-25T00:22:08.769841+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-25T00:22:08.769841+0000","last_undegraded":"2023-10-25T00:22:08.769841+0000","last_fullsized":"2023-10-25T00:22:08.769841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:50.994593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21989,"num_write_kb":280762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"68'4908","reported_seq":4987,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.198947+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-25T00:21:39.198947+0000","last_peered":"2023-10-25T00:21:39.198947+0000","last_clean":"2023-10-25T00:21:39.198947+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-25T00:21:39.198947+0000","last_undegraded":"2023-10-25T00:21:39.198947+0000","last_fullsized":"2023-10-25T00:21:39.198947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26829,"num_write_kb":320697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":320,"num_write":1482516,"num_write_kb":19060243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267718,"ondisk_log_size":267718,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":19490997,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1686,"num_read_kb":209824,"num_write":4665,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39043072,"data_stored":38981994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4535,"ondisk_log_size":4535,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":0,"num_write":1661441,"num_write_kb":19773626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307026,"ondisk_log_size":307026,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":20040877,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1620,"num_read_kb":247649,"num_write":4615,"num_write_kb":21609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40140800,"data_stored":40081754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4486,"ondisk_log_size":4486,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575206,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919080,"kb_used_data":12320,"kb_used_omap":0,"kb_used_meta":906752,"kb_avail":93452760,"statfs":{"total":96636764160,"available":95695626240,"internally_reserved":0,"allocated":12615680,"data_stored":12142216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":928514048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.304,"15min":0.93799999999999994},"min":{"1min":0.22,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.6349999999999998,"5min":13.929,"15min":13.929},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.3,"15min":0.92900000000000005},"min":{"1min":0.29699999999999999,"5min":0.27000000000000002,"15min":0.20999999999999999},"max":{"1min":3.6659999999999999,"5min":11.315,"15min":11.315},"last":1.02}]},{"osd":1,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.714,"15min":1.1639999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.215},"max":{"1min":3.6949999999999998,"5min":28.308,"15min":28.308},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.782,"15min":1.2290000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":3.7490000000000001,"5min":28.346,"15min":28.346},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.3779999999999999,"15min":0.99399999999999999},"min":{"1min":0.28499999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":5.71,"5min":11.430999999999999,"15min":11.430999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.139,"15min":0.94599999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.8799999999999999,"5min":7.9710000000000001,"15min":7.9710000000000001},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":2.3180000000000001,"15min":1.363},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.6150000000000002,"5min":93.664000000000001,"15min":93.664000000000001},"last":1.081},{"interface":"front","average":{"1min":1.2030000000000001,"5min":2.379,"15min":1.3740000000000001},"min":{"1min":0.34499999999999997,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":2.8809999999999998,"5min":93.073999999999998,"15min":93.073999999999998},"last":0.91400000000000003}]},{"osd":4,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0840000000000001,"15min":0.877},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.249},"max":{"1min":2.9089999999999998,"5min":7.6280000000000001,"15min":7.6280000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.714,"15min":1.1339999999999999},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.27200000000000002},"max":{"1min":4.9640000000000004,"5min":27.936,"15min":27.936},"last":0.98299999999999998}]},{"osd":5,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":2.1459999999999999,"5min":1.573,"15min":1.05},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.8170000000000002,"5min":21.126999999999999,"15min":21.126999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.995,"15min":0.94699999999999995},"min":{"1min":0.39800000000000002,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":2.5910000000000002,"5min":11.49,"15min":11.49},"last":0.878}]},{"osd":6,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.4770000000000001,"15min":1.077},"min":{"1min":0.41999999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.8140000000000001,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7290000000000001,"15min":1.1799999999999999},"min":{"1min":0.49399999999999999,"5min":0.38100000000000001,"15min":0.35499999999999998},"max":{"1min":4.8639999999999999,"5min":20.651,"15min":20.651},"last":1.115}]}]},{"osd":6,"up_from":13,"seq":55834575206,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859004,"kb_used_data":5044,"kb_used_omap":0,"kb_used_meta":853952,"kb_avail":93512836,"statfs":{"total":96636764160,"available":95757144064,"internally_reserved":0,"allocated":5165056,"data_stored":4728730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":874446848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.397,"15min":0.93899999999999995},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.9009999999999998,"5min":15.974,"15min":15.974},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.4199999999999999,"15min":0.96799999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":8.2400000000000002,"5min":8.4789999999999992,"15min":8.4789999999999992},"last":0.89500000000000002}]},{"osd":1,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.042,"15min":0.878},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.221},"max":{"1min":3.5630000000000002,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.2549999999999999,"15min":0.98599999999999999},"min":{"1min":0.28699999999999998,"5min":0.254,"15min":0.249},"max":{"1min":3.5840000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.92200000000000004}]},{"osd":2,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.617,"15min":1.0600000000000001},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.6779999999999999,"5min":12.615,"15min":12.615},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.228,"5min":1.591,"15min":1.105},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.702,"5min":12.667999999999999,"15min":12.667999999999999},"last":0.72399999999999998}]},{"osd":3,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.9690000000000001,"15min":1.2350000000000001},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.7350000000000003,"5min":38.649999999999999,"15min":38.649999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":2.0249999999999999,"15min":1.3260000000000001},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.766,"5min":38.683,"15min":38.683},"last":1.014}]},{"osd":4,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.7589999999999999,"15min":1.0960000000000001},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.738,"5min":12.682,"15min":12.682},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4350000000000001,"15min":1.079},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.694,"5min":10.686,"15min":10.686},"last":1.1279999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.1990000000000001,"15min":0.94799999999999995},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.24299999999999999},"max":{"1min":3.6379999999999999,"5min":8.0039999999999996,"15min":8.0039999999999996},"last":1.091},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.3700000000000001,"15min":1.046},"min":{"1min":0.496,"5min":0.33400000000000002,"15min":0.30299999999999999},"max":{"1min":3.903,"5min":11.766999999999999,"15min":11.766999999999999},"last":1.0429999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.907,"5min":1.306,"15min":1.018},"min":{"1min":0.32200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":8.2720000000000002,"5min":12.926,"15min":12.926},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.258,"15min":1.026},"min":{"1min":0.38200000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.1100000000000003,"5min":7.6619999999999999,"15min":7.6619999999999999},"last":1.335}]}]},{"osd":5,"up_from":13,"seq":55834575206,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":958856,"kb_used_data":9152,"kb_used_omap":0,"kb_used_meta":949696,"kb_avail":93412984,"statfs":{"total":96636764160,"available":95654895616,"internally_reserved":0,"allocated":9371648,"data_stored":8923216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":972488704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":2.4649999999999999,"5min":1.4690000000000001,"15min":1.032},"min":{"1min":0.23599999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.837999999999999,"5min":14.837999999999999,"15min":14.837999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.1599999999999999,"15min":0.95999999999999996},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.536,"5min":7.423,"15min":7.423},"last":0.73999999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":1.9990000000000001,"5min":1.6000000000000001,"15min":1.0840000000000001},"min":{"1min":0.27500000000000002,"5min":0.24299999999999999,"15min":0.23400000000000001},"max":{"1min":12.141,"5min":12.141,"15min":12.141},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.518,"15min":1.0820000000000001},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.252},"max":{"1min":3.5950000000000002,"5min":24.648,"15min":24.648},"last":1.008}]},{"osd":2,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":1.9750000000000001,"5min":1.841,"15min":1.236},"min":{"1min":0.29899999999999999,"5min":0.25,"15min":0.25},"max":{"1min":12.175000000000001,"5min":20.760999999999999,"15min":20.760999999999999},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.655,"5min":1.46,"15min":1.0800000000000001},"min":{"1min":0.317,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":12.109999999999999,"5min":20.552,"15min":20.552},"last":0.84599999999999997}]},{"osd":3,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":1.5569999999999999,"15min":1.224},"min":{"1min":0.35599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":12.013,"5min":12.013,"15min":12.013},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.6819999999999999,"15min":1.2549999999999999},"min":{"1min":0.34399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.919,"5min":12.805,"15min":12.805},"last":0.97399999999999998}]},{"osd":4,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":2.3919999999999999,"5min":1.8240000000000001,"15min":1.204},"min":{"1min":0.38900000000000001,"5min":0.26000000000000001,"15min":0.23999999999999999},"max":{"1min":12.201000000000001,"5min":12.728,"15min":12.728},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.131,"5min":1.486,"15min":1.141},"min":{"1min":0.36599999999999999,"5min":0.27600000000000002,"15min":0.27200000000000002},"max":{"1min":3.6179999999999999,"5min":10.943,"15min":10.943},"last":1.081}]},{"osd":6,"last update":"Wed Oct 25 00:21:58 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.369,"15min":1.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.33500000000000002,"15min":0.26400000000000001},"max":{"1min":9.6489999999999991,"5min":17.936,"15min":17.936},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.96,"15min":1.224},"min":{"1min":0.312,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":12.042999999999999,"5min":14.766999999999999,"15min":14.766999999999999},"last":0.872}]},{"osd":7,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":2.984,"5min":1.7709999999999999,"15min":1.26},"min":{"1min":0.373,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":28.178999999999998},"last":1.107},{"interface":"front","average":{"1min":3.23,"5min":2.1659999999999999,"15min":1.3540000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":28.222999999999999,"5min":28.222999999999999,"15min":28.222999999999999},"last":1.1439999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575205,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946272,"kb_used_data":19096,"kb_used_omap":0,"kb_used_meta":927168,"kb_avail":93425568,"statfs":{"total":96636764160,"available":95667781632,"internally_reserved":0,"allocated":19554304,"data_stored":19092590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":949420032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":1.1910000000000001,"15min":0.93000000000000005},"min":{"1min":0.21199999999999999,"5min":0.19700000000000001,"15min":0.187},"max":{"1min":3.6120000000000001,"5min":20.922999999999998,"15min":20.922999999999998},"last":1.081},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.1499999999999999,"15min":0.96199999999999997},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.217},"max":{"1min":1.224,"5min":20.532,"15min":20.532},"last":0.76000000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.3560000000000001,"15min":0.97699999999999998},"min":{"1min":0.224,"5min":0.22,"15min":0.22},"max":{"1min":2.8140000000000001,"5min":20.914000000000001,"15min":20.914000000000001},"last":0.999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.738,"15min":1.157},"min":{"1min":0.28699999999999998,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":2.8399999999999999,"5min":23.265999999999998,"15min":23.265999999999998},"last":0.86099999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":1.5309999999999999,"15min":1.0920000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.3719999999999999,"5min":20.774000000000001,"15min":20.774000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.7509999999999999,"15min":1.2190000000000001},"min":{"1min":0.30099999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.9009999999999998,"5min":20.817,"15min":20.817},"last":1.115}]},{"osd":3,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.575,"15min":1.131},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.1440000000000001,"5min":24.613,"15min":24.613},"last":1.1639999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.3300000000000001,"15min":1.113},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.3650000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.1850000000000001}]},{"osd":5,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.04,"15min":0.88500000000000001},"min":{"1min":0.35599999999999998,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":4.7759999999999998,"5min":6.1459999999999999,"15min":6.1459999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.0640000000000001,"15min":0.94799999999999995},"min":{"1min":0.246,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.5960000000000001,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.92800000000000005}]},{"osd":6,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.337,"15min":1.0740000000000001},"min":{"1min":0.36899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.048,"5min":6.5220000000000002,"15min":6.5220000000000002},"last":1.143},{"interface":"front","average":{"1min":1.2130000000000001,"5min":0.97599999999999998,"15min":0.94299999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":4.3819999999999997,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":1.2529999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:21:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.2450000000000001,"15min":1.0469999999999999},"min":{"1min":0.47399999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":6.3330000000000002,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.4279999999999999,"15min":1.1020000000000001},"min":{"1min":0.51000000000000001,"5min":0.309,"15min":0.309},"max":{"1min":6.3550000000000004,"5min":6.3550000000000004,"15min":6.3550000000000004},"last":1.2130000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575206,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160732,"kb_used_data":9236,"kb_used_omap":0,"kb_used_meta":1151488,"kb_avail":93211108,"statfs":{"total":96636764160,"available":95448174592,"internally_reserved":0,"allocated":9457664,"data_stored":9007793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1179123712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.996,"15min":0.78900000000000003},"min":{"1min":0.20000000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.018,"5min":13.327999999999999,"15min":13.327999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.61299999999999999,"15min":0.73399999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.0489999999999999,"5min":3.556,"15min":5.4580000000000002},"last":0.72699999999999998}]},{"osd":1,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.2290000000000001,"15min":0.86099999999999999},"min":{"1min":0.21199999999999999,"5min":0.20599999999999999,"15min":0.19900000000000001},"max":{"1min":1.796,"5min":13.831,"15min":13.831},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.80900000000000005,"15min":0.76200000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.9300000000000002,"5min":4.4500000000000002,"15min":4.4500000000000002},"last":0.79600000000000004}]},{"osd":2,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.92200000000000004,"15min":0.877},"min":{"1min":0.23799999999999999,"5min":0.223,"15min":0.221},"max":{"1min":2.7890000000000001,"5min":8.2520000000000007,"15min":8.2520000000000007},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.075,"15min":0.90700000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.7360000000000002,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.83799999999999997}]},{"osd":4,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.9630000000000001,"15min":1.2010000000000001},"min":{"1min":0.32100000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.0339999999999998,"5min":39.070999999999998,"15min":39.070999999999998},"last":1.208},{"interface":"front","average":{"1min":1.5389999999999999,"5min":2.0870000000000002,"15min":1.2509999999999999},"min":{"1min":0.34599999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.0300000000000002,"5min":55.104999999999997,"15min":55.104999999999997},"last":0.88400000000000001}]},{"osd":5,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.6379999999999999,"15min":1.117},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.1849999999999996,"5min":31.481999999999999,"15min":31.481999999999999},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.681,"15min":1.1970000000000001},"min":{"1min":0.42499999999999999,"5min":0.35299999999999998,"15min":0.32500000000000001},"max":{"1min":4.2060000000000004,"5min":31.512,"15min":31.512},"last":1.4179999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.9179999999999999,"5min":1.617,"15min":1.1910000000000001},"min":{"1min":0.311,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.7080000000000002,"5min":25.809999999999999,"15min":25.809999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.9790000000000001,"5min":1.4530000000000001,"15min":1.175},"min":{"1min":0.47299999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":4.7779999999999996,"5min":24.036999999999999,"15min":24.036999999999999},"last":0.91900000000000004}]},{"osd":7,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.704,"15min":1.226},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.8529999999999998,"5min":27.581,"15min":27.581},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.6379999999999999,"15min":1.24},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":8.5259999999999998,"5min":24.332000000000001,"15min":24.332000000000001},"last":1.349}]}]},{"osd":2,"up_from":13,"seq":55834575206,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946044,"kb_used_data":5044,"kb_used_omap":0,"kb_used_meta":940992,"kb_avail":93425796,"statfs":{"total":96636764160,"available":95668015104,"internally_reserved":0,"allocated":5165056,"data_stored":4728218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":963575808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.651,"15min":0.98399999999999999},"min":{"1min":0.188,"5min":0.16,"15min":0.16},"max":{"1min":2.9329999999999998,"5min":29.344000000000001,"15min":29.344000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.0800000000000001,"15min":0.82499999999999996},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.0539999999999998,"5min":10.755000000000001,"15min":10.755000000000001},"last":0.49099999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.3640000000000001,"15min":0.92200000000000004},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":2.8860000000000001,"5min":18.617999999999999,"15min":18.617999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":1.099,"15min":0.86099999999999999},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":2.1669999999999998,"5min":15.064,"15min":15.064},"last":0.56699999999999995}]},{"osd":3,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.1259999999999999,"15min":0.88800000000000001},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.192},"max":{"1min":1.766,"5min":13.034000000000001,"15min":13.034000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.5700000000000001,"15min":0.98099999999999998},"min":{"1min":0.26200000000000001,"5min":0.25,"15min":0.25},"max":{"1min":2.964,"5min":18.547999999999998,"15min":18.547999999999998},"last":0.38400000000000001}]},{"osd":4,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.45,"15min":1.1659999999999999},"min":{"1min":0.35099999999999998,"5min":0.252,"15min":0.252},"max":{"1min":2.839,"5min":22.997,"15min":33.378999999999998},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.5,"15min":1.2689999999999999},"min":{"1min":0.38500000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.8690000000000002,"5min":23.030999999999999,"15min":33.558999999999997},"last":0.78700000000000003}]},{"osd":5,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":2.2589999999999999,"15min":1.45},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":4.6440000000000001,"5min":25.183,"15min":34.545999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.038,"5min":1.3879999999999999,"15min":1.169},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.4829999999999997,"5min":11.685,"15min":36.104999999999997},"last":0.65700000000000003}]},{"osd":6,"last update":"Wed Oct 25 00:21:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.208,"15min":1.1100000000000001},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":3.5739999999999998,"5min":9.7149999999999999,"15min":33.661999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.26,"15min":1.2},"min":{"1min":0.45700000000000002,"5min":0.371,"15min":0.371},"max":{"1min":3.5979999999999999,"5min":9.6799999999999997,"15min":34.043999999999997},"last":1.0149999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.3,"15min":1.214},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.395,"5min":11.503,"15min":33.307000000000002},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.3420000000000001,"15min":1.276},"min":{"1min":0.51200000000000001,"5min":0.371,"15min":0.371},"max":{"1min":2.419,"5min":11.468999999999999,"15min":33.344999999999999},"last":0.89400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575207,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1070492,"kb_used_data":4052,"kb_used_omap":0,"kb_used_meta":1066432,"kb_avail":93301348,"statfs":{"total":96636764160,"available":95540580352,"internally_reserved":0,"allocated":4149248,"data_stored":3711479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1092026368},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.1739999999999999,"15min":0.81699999999999995},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.81000000000000005,"5min":21.334,"15min":21.334},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.0449999999999999,"15min":0.84099999999999997},"min":{"1min":0.20499999999999999,"5min":0.19,"15min":0.19},"max":{"1min":3.141,"5min":21.276,"15min":21.276},"last":0.84799999999999998}]},{"osd":2,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.071,"15min":0.80200000000000005},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":8.1600000000000001,"5min":11.420999999999999,"15min":11.420999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.90100000000000002,"15min":0.83399999999999996},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.1760000000000002,"5min":11.648,"15min":11.648},"last":0.78300000000000003}]},{"osd":3,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.22,"15min":0.91500000000000004},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":3.46,"5min":14.346,"15min":14.346},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.2789999999999999,"15min":1.016},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.248},"max":{"1min":3.4969999999999999,"5min":14.379,"15min":14.379},"last":0.99399999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.3959999999999999,"15min":1.0569999999999999},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.6500000000000004,"5min":10.582000000000001,"15min":10.582000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.49,"5min":1.3009999999999999,"15min":1.04},"min":{"1min":0.32600000000000001,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":4.6630000000000003,"5min":11.173,"15min":11.173},"last":1.0429999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.244,"15min":1.054},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.0129999999999999,"5min":11.577999999999999,"15min":11.577999999999999},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.0760000000000001,"15min":0.98099999999999998},"min":{"1min":0.29899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.3220000000000001,"5min":10.018000000000001,"15min":10.018000000000001},"last":1.0820000000000001}]},{"osd":6,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.1140000000000001,"15min":1.026},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.3610000000000002,"5min":9.2989999999999995,"15min":11.845000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.1699999999999999,"15min":1.109},"min":{"1min":0.38300000000000001,"5min":0.379,"15min":0.33800000000000002},"max":{"1min":2.3809999999999998,"5min":9.3350000000000009,"15min":11.919},"last":1.115}]},{"osd":7,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.498,"15min":1.125},"min":{"1min":0.38200000000000001,"5min":0.36199999999999999,"15min":0.35599999999999998},"max":{"1min":5.077,"5min":9.407,"15min":11.881},"last":1.02},{"interface":"front","average":{"1min":0.874,"5min":1.3089999999999999,"15min":1.137},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.9750000000000001,"5min":10.478,"15min":10.478},"last":1.1439999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575207,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":966360,"kb_used_data":22096,"kb_used_omap":0,"kb_used_meta":944256,"kb_avail":93405480,"statfs":{"total":96636764160,"available":95647211520,"internally_reserved":0,"allocated":22626304,"data_stored":22180994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":966918144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.4159999999999999,"15min":0.98399999999999999},"min":{"1min":0.19700000000000001,"5min":0.18099999999999999,"15min":0.16900000000000001},"max":{"1min":1.7290000000000001,"5min":30.065999999999999,"15min":30.065999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.2330000000000001,"15min":0.98799999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.182},"max":{"1min":3.0030000000000001,"5min":19.757999999999999,"15min":19.757999999999999},"last":0.76300000000000001}]},{"osd":2,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.96099999999999997,"15min":0.78200000000000003},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.9740000000000002,"5min":15.324999999999999,"15min":15.324999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.504,"5min":1.2050000000000001,"15min":0.94299999999999995},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.95699999999999996,"5min":30.283999999999999,"15min":30.283999999999999},"last":1.014}]},{"osd":3,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.5209999999999999,"15min":1.101},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.191},"max":{"1min":6.3570000000000002,"5min":22.966999999999999,"15min":22.966999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":1.417,"15min":1.048},"min":{"1min":0.29499999999999998,"5min":0.24299999999999999,"15min":0.23599999999999999},"max":{"1min":1.1000000000000001,"5min":19.460999999999999,"15min":19.460999999999999},"last":1.0629999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.2090000000000001,"15min":1.121},"min":{"1min":0.33700000000000002,"5min":0.318,"15min":0.26600000000000001},"max":{"1min":3.0720000000000001,"5min":5.5990000000000002,"15min":8.3670000000000009},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.375,"15min":1.1850000000000001},"min":{"1min":0.29499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.1749999999999998,"5min":13.023999999999999,"15min":13.023999999999999},"last":0.98899999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.23,"15min":1.006},"min":{"1min":0.33000000000000002,"5min":0.30399999999999999,"15min":0.29099999999999998},"max":{"1min":2.9710000000000001,"5min":10.683999999999999,"15min":10.683999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.089,"15min":0.96999999999999997},"min":{"1min":0.378,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.5779999999999998,"5min":15.436999999999999,"15min":15.436999999999999},"last":1.173}]},{"osd":6,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.393,"15min":1.125},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.7469999999999999,"5min":8.7260000000000009,"15min":8.7260000000000009},"last":1.085},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.3500000000000001,"15min":1.248},"min":{"1min":0.38200000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.9039999999999999,"5min":15.375,"15min":15.375},"last":1.2250000000000001}]},{"osd":7,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.367,"15min":1.113},"min":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":5.0979999999999999,"5min":10.81,"15min":10.81},"last":1.036},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.3220000000000001,"15min":1.25},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":3.1499999999999999,"5min":11.417,"15min":11.417},"last":1.157}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12673024,"data_stored":12668140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3178496,"data_stored":3177545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11669504,"data_stored":11651527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4195192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":8388642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4279555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6918144,"data_stored":6907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6836224,"data_stored":6822812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-25T00:22:30.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-25T00:22:31.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.096+0000 7f9ba245d640 1 Processor -- start 2023-10-25T00:22:31.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.096+0000 7f9ba245d640 1 -- start start 2023-10-25T00:22:31.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9b9c14fd40 0x7f9b9c0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9b9c0a9820 0x7f9b9c0aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9b9c06ee40 0x7f9b9c0c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9c0aa750 con 0x7f9b9c0a9820 2023-10-25T00:22:31.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9c06f730 con 0x7f9b9c06ee40 2023-10-25T00:22:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c156a40 con 0x7f9b9c14fd40 2023-10-25T00:22:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9b9c0a9820 0x7f9b9c0aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f9b9c0a9820 0x7f9b9c0aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54130/0 (socket says 172.21.15.107:54130) 2023-10-25T00:22:31.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9b7fe640 1 -- 172.21.15.107:0/3107676185 learned_addr learned my addr 172.21.15.107:0/3107676185 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:31.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba09fa640 1 --2- 172.21.15.107:0/3107676185 >> v2:172.21.15.110:3300/0 conn(0x7f9b9c06ee40 0x7f9b9c0c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9bfff640 1 --2- 172.21.15.107:0/3107676185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9b9c14fd40 0x7f9b9c0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9b7fe640 1 -- 172.21.15.107:0/3107676185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9b9c14fd40 msgr2=0x7f9b9c0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:31.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9b7fe640 1 --2- 172.21.15.107:0/3107676185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9b9c14fd40 0x7f9b9c0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9b7fe640 1 -- 172.21.15.107:0/3107676185 >> v2:172.21.15.110:3300/0 conn(0x7f9b9c06ee40 msgr2=0x7f9b9c0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9b7fe640 1 --2- 172.21.15.107:0/3107676185 >> v2:172.21.15.110:3300/0 conn(0x7f9b9c06ee40 0x7f9b9c0c1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9b7fe640 1 -- 172.21.15.107:0/3107676185 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c156710 con 0x7f9b9c0a9820 2023-10-25T00:22:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9b7fe640 1 --2- 172.21.15.107:0/3107676185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b9c0a9820 0x7f9b9c0aa210 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c06f260 tx=0x7f9b8401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24f48749d015344 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9affd640 1 -- 172.21.15.107:0/3107676185 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b840ac640 con 0x7f9b9c0a9820 2023-10-25T00:22:31.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9affd640 1 -- 172.21.15.107:0/3107676185 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b840ac7a0 con 0x7f9b9c0a9820 2023-10-25T00:22:31.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9b9affd640 1 -- 172.21.15.107:0/3107676185 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b840acb30 con 0x7f9b9c0a9820 2023-10-25T00:22:31.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 -- 172.21.15.107:0/3107676185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b9c0a9820 msgr2=0x7f9b9c0aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 --2- 172.21.15.107:0/3107676185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b9c0a9820 0x7f9b9c0aa210 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c06f260 tx=0x7f9b8401c3b0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 -- 172.21.15.107:0/3107676185 shutdown_connections 2023-10-25T00:22:31.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 --2- 172.21.15.107:0/3107676185 >> v2:172.21.15.110:3300/0 conn(0x7f9b9c06ee40 0x7f9b9c0c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 --2- 172.21.15.107:0/3107676185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b9c0a9820 0x7f9b9c0aa210 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 --2- 172.21.15.107:0/3107676185 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9b9c14fd40 0x7f9b9c0ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 -- 172.21.15.107:0/3107676185 >> 172.21.15.107:0/3107676185 conn(0x7f9b9c0c2950 msgr2=0x7f9b9c0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:31.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 -- 172.21.15.107:0/3107676185 shutdown_connections 2023-10-25T00:22:31.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.100+0000 7f9ba245d640 1 -- 172.21.15.107:0/3107676185 wait complete. 2023-10-25T00:22:31.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba245d640 1 Processor -- start 2023-10-25T00:22:31.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba245d640 1 -- start start 2023-10-25T00:22:31.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba245d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9b9c06ee40 0x7f9b9c14a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba245d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b9c14fd40 0x7f9b9c14e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba245d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b9c14a7c0 0x7f9b9c1ed650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba245d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c1f0b50 con 0x7f9b9c14a7c0 2023-10-25T00:22:31.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba245d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c1f0e30 con 0x7f9b9c14fd40 2023-10-25T00:22:31.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba245d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c1f1110 con 0x7f9b9c06ee40 2023-10-25T00:22:31.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba09fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b9c14a7c0 0x7f9b9c1ed650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba09fa640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b9c14a7c0 0x7f9b9c1ed650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54140/0 (socket says 172.21.15.107:54140) 2023-10-25T00:22:31.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba09fa640 1 -- 172.21.15.107:0/1049482703 learned_addr learned my addr 172.21.15.107:0/1049482703 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:31.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9b9bfff640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9b9c06ee40 0x7f9b9c14a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9b9b7fe640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b9c14fd40 0x7f9b9c14e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba09fa640 1 -- 172.21.15.107:0/1049482703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9b9c06ee40 msgr2=0x7f9b9c14a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba09fa640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9b9c06ee40 0x7f9b9c14a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba09fa640 1 -- 172.21.15.107:0/1049482703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b9c14fd40 msgr2=0x7f9b9c14e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba09fa640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b9c14fd40 0x7f9b9c14e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba09fa640 1 -- 172.21.15.107:0/1049482703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b840094d0 con 0x7f9b9c14a7c0 2023-10-25T00:22:31.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba09fa640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b9c14a7c0 0x7f9b9c1ed650 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00d800 tx=0x7f9b8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:31.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9b98ff9640 1 -- 172.21.15.107:0/1049482703 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c00b6c0 con 0x7f9b9c14a7c0 2023-10-25T00:22:31.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9b98ff9640 1 -- 172.21.15.107:0/1049482703 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c00b820 con 0x7f9b9c14a7c0 2023-10-25T00:22:31.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba245d640 1 -- 172.21.15.107:0/1049482703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c1ed130 con 0x7f9b9c14a7c0 2023-10-25T00:22:31.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9b98ff9640 1 -- 172.21.15.107:0/1049482703 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c00bbd0 con 0x7f9b9c14a7c0 2023-10-25T00:22:31.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.104+0000 7f9ba245d640 1 -- 172.21.15.107:0/1049482703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c0bfc50 con 0x7f9b9c14a7c0 2023-10-25T00:22:31.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.108+0000 7f9ba245d640 1 -- 172.21.15.107:0/1049482703 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9c0c24a0 con 0x7f9b9c14a7c0 2023-10-25T00:22:31.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.108+0000 7f9b98ff9640 1 -- 172.21.15.107:0/1049482703 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f9b8c00bd30 con 0x7f9b9c14a7c0 2023-10-25T00:22:31.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.108+0000 7f9b98ff9640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9b7c065e60 0x7f9b7c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.108+0000 7f9b98ff9640 1 -- 172.21.15.107:0/1049482703 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f9b8c084fe0 con 0x7f9b9c14a7c0 2023-10-25T00:22:31.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.108+0000 7f9b9bfff640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9b7c065e60 0x7f9b7c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.108+0000 7f9b9bfff640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9b7c065e60 0x7f9b7c068320 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f9b94003fd0 tx=0x7f9b94003ce0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:31.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.112+0000 7f9b98ff9640 1 -- 172.21.15.107:0/1049482703 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f9b8c0513e0 con 0x7f9b9c14a7c0 2023-10-25T00:22:31.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.260+0000 7f9ba245d640 1 -- 172.21.15.107:0/1049482703 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9b9c1ecfc0 con 0x7f9b7c065e60 2023-10-25T00:22:31.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.304+0000 7f9b98ff9640 1 -- 172.21.15.107:0/1049482703 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+550969 (secure 0 0 0) 0x7f9b9c1ecfc0 con 0x7f9b7c065e60 2023-10-25T00:22:31.309 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:31.313 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-25T00:22:31.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 -- 172.21.15.107:0/1049482703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9b7c065e60 msgr2=0x7f9b7c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9b7c065e60 0x7f9b7c068320 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f9b94003fd0 tx=0x7f9b94003ce0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 -- 172.21.15.107:0/1049482703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b9c14a7c0 msgr2=0x7f9b9c1ed650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b9c14a7c0 0x7f9b9c1ed650 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00d800 tx=0x7f9b8c00dcc0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 -- 172.21.15.107:0/1049482703 shutdown_connections 2023-10-25T00:22:31.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b9c14a7c0 0x7f9b9c1ed650 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9b9c14fd40 0x7f9b9c14e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9b7c065e60 0x7f9b7c068320 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 --2- 172.21.15.107:0/1049482703 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9b9c06ee40 0x7f9b9c14a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 -- 172.21.15.107:0/1049482703 >> 172.21.15.107:0/1049482703 conn(0x7f9b9c0c2950 msgr2=0x7f9b9c14c970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:31.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 -- 172.21.15.107:0/1049482703 shutdown_connections 2023-10-25T00:22:31.317 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.308+0000 7f9ba245d640 1 -- 172.21.15.107:0/1049482703 wait complete. 2023-10-25T00:22:31.331 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":955,"stamp":"2023-10-25T00:22:30.008225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40122242,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3906,"num_read_kb":457920,"num_write":3153446,"num_write_kb":38880045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":583926,"ondisk_log_size":583926,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7826840,"kb_used_data":86040,"kb_used_omap":0,"kb_used_meta":7740736,"kb_avail":747147880,"statfs":{"total":773094113280,"available":765079429120,"internally_reserved":0,"allocated":88104960,"data_stored":84515236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7926513664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":445,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005534"},"pg_stats":[{"pgid":"6.1a","version":"72'3995","reported_seq":4066,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.672949+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-25T00:22:08.672949+0000","last_peered":"2023-10-25T00:22:08.672949+0000","last_clean":"2023-10-25T00:22:08.672949+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-25T00:22:08.672949+0000","last_undegraded":"2023-10-25T00:22:08.672949+0000","last_fullsized":"2023-10-25T00:22:08.672949+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:51:20.137796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21480,"num_write_kb":282788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.773236+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-25T00:22:08.773236+0000","last_peered":"2023-10-25T00:22:08.773236+0000","last_clean":"2023-10-25T00:22:08.773236+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-25T00:22:08.773236+0000","last_undegraded":"2023-10-25T00:22:08.773236+0000","last_fullsized":"2023-10-25T00:22:08.773236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:00:20.101379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"68'864","reported_seq":988,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.254160+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-25T00:21:42.254160+0000","last_peered":"2023-10-25T00:21:42.254160+0000","last_clean":"2023-10-25T00:21:42.254160+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-25T00:21:42.254160+0000","last_undegraded":"2023-10-25T00:21:42.254160+0000","last_fullsized":"2023-10-25T00:21:42.254160+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":18441,"num_write":841,"num_write_kb":4586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'5616","reported_seq":5688,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195291+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-25T00:21:39.195291+0000","last_peered":"2023-10-25T00:21:39.195291+0000","last_clean":"2023-10-25T00:21:39.195291+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-25T00:21:39.195291+0000","last_undegraded":"2023-10-25T00:21:39.195291+0000","last_fullsized":"2023-10-25T00:21:39.195291+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29775,"num_write_kb":348818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"72'3735","reported_seq":3804,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.701818+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-25T00:22:08.701818+0000","last_peered":"2023-10-25T00:22:08.701818+0000","last_clean":"2023-10-25T00:22:08.701818+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-25T00:22:08.701818+0000","last_undegraded":"2023-10-25T00:22:08.701818+0000","last_fullsized":"2023-10-25T00:22:08.701818+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:59.368156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20090,"num_write_kb":264783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"72'2","reported_seq":70,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.258523+0000","last_change":"2023-10-24T23:53:11.444089+0000","last_active":"2023-10-25T00:22:08.258523+0000","last_peered":"2023-10-25T00:22:08.258523+0000","last_clean":"2023-10-25T00:22:08.258523+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-25T00:22:08.258523+0000","last_undegraded":"2023-10-25T00:22:08.258523+0000","last_fullsized":"2023-10-25T00:22:08.258523+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:54:06.875882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"72'27","reported_seq":101,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186964+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-25T00:21:39.186964+0000","last_peered":"2023-10-25T00:21:39.186964+0000","last_clean":"2023-10-25T00:21:39.186964+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-25T00:21:39.186964+0000","last_undegraded":"2023-10-25T00:21:39.186964+0000","last_fullsized":"2023-10-25T00:21:39.186964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'5447","reported_seq":5519,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.184033+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-25T00:21:39.184033+0000","last_peered":"2023-10-25T00:21:39.184033+0000","last_clean":"2023-10-25T00:21:39.184033+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-25T00:21:39.184033+0000","last_undegraded":"2023-10-25T00:21:39.184033+0000","last_fullsized":"2023-10-25T00:21:39.184033+0000","mapping_epoch":21,"log_start":"61'100","ondisk_log_start":"61'100","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28063,"num_write_kb":350200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"72'4747","reported_seq":4818,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.795259+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-25T00:22:08.795259+0000","last_peered":"2023-10-25T00:22:08.795259+0000","last_clean":"2023-10-25T00:22:08.795259+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-25T00:22:08.795259+0000","last_undegraded":"2023-10-25T00:22:08.795259+0000","last_fullsized":"2023-10-25T00:22:08.795259+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:38:56.643025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":25766,"num_write_kb":328702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"72'753","reported_seq":847,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.712526+0000","last_change":"2023-10-24T23:53:11.438681+0000","last_active":"2023-10-25T00:22:08.712526+0000","last_peered":"2023-10-25T00:22:08.712526+0000","last_clean":"2023-10-25T00:22:08.712526+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-25T00:22:08.712526+0000","last_undegraded":"2023-10-25T00:22:08.712526+0000","last_fullsized":"2023-10-25T00:22:08.712526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:50.139697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":77282,"num_write":740,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"72'48","reported_seq":163,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.487987+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-25T00:21:42.487987+0000","last_peered":"2023-10-25T00:21:42.487987+0000","last_clean":"2023-10-25T00:21:42.487987+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-25T00:21:42.487987+0000","last_undegraded":"2023-10-25T00:21:42.487987+0000","last_fullsized":"2023-10-25T00:21:42.487987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'4792","reported_seq":4864,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205939+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-25T00:21:19.205939+0000","last_peered":"2023-10-25T00:21:19.205939+0000","last_clean":"2023-10-25T00:21:19.205939+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-25T00:21:19.205939+0000","last_undegraded":"2023-10-25T00:21:19.205939+0000","last_fullsized":"2023-10-25T00:21:19.205939+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24997,"num_write_kb":313272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"72'4739","reported_seq":4811,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.785162+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-25T00:22:08.785162+0000","last_peered":"2023-10-25T00:22:08.785162+0000","last_clean":"2023-10-25T00:22:08.785162+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-25T00:22:08.785162+0000","last_undegraded":"2023-10-25T00:22:08.785162+0000","last_fullsized":"2023-10-25T00:22:08.785162+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:22.974333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":22,"num_write":25639,"num_write_kb":332364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"72'357","reported_seq":426,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:15.185838+0000","last_change":"2023-10-24T23:53:11.454537+0000","last_active":"2023-10-25T00:22:15.185838+0000","last_peered":"2023-10-25T00:22:15.185838+0000","last_clean":"2023-10-25T00:22:15.185838+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-25T00:22:15.185838+0000","last_undegraded":"2023-10-25T00:22:15.185838+0000","last_fullsized":"2023-10-25T00:22:15.185838+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:14:06.792651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2627766,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":227,"num_write":329,"num_write_kb":2759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"68'5","reported_seq":122,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.254866+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-25T00:21:42.254866+0000","last_peered":"2023-10-25T00:21:42.254866+0000","last_clean":"2023-10-25T00:21:42.254866+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-25T00:21:42.254866+0000","last_undegraded":"2023-10-25T00:21:42.254866+0000","last_fullsized":"2023-10-25T00:21:42.254866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'3731","reported_seq":3801,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205159+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-25T00:21:19.205159+0000","last_peered":"2023-10-25T00:21:19.205159+0000","last_clean":"2023-10-25T00:21:19.205159+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-25T00:21:19.205159+0000","last_undegraded":"2023-10-25T00:21:19.205159+0000","last_fullsized":"2023-10-25T00:21:19.205159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19492,"num_write_kb":250091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"72'3799","reported_seq":3869,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.756165+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-25T00:22:08.756165+0000","last_peered":"2023-10-25T00:22:08.756165+0000","last_clean":"2023-10-25T00:22:08.756165+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-25T00:22:08.756165+0000","last_undegraded":"2023-10-25T00:22:08.756165+0000","last_fullsized":"2023-10-25T00:22:08.756165+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:35:39.879563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21581,"num_write_kb":262877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"72'45","reported_seq":156,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.747187+0000","last_change":"2023-10-24T23:53:11.444138+0000","last_active":"2023-10-25T00:22:08.747187+0000","last_peered":"2023-10-25T00:22:08.747187+0000","last_clean":"2023-10-25T00:22:08.747187+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-25T00:22:08.747187+0000","last_undegraded":"2023-10-25T00:22:08.747187+0000","last_fullsized":"2023-10-25T00:22:08.747187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:27:38.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"68'714","reported_seq":813,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195222+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-25T00:21:39.195222+0000","last_peered":"2023-10-25T00:21:39.195222+0000","last_clean":"2023-10-25T00:21:39.195222+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-25T00:21:39.195222+0000","last_undegraded":"2023-10-25T00:21:39.195222+0000","last_fullsized":"2023-10-25T00:21:39.195222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":86332,"num_write":703,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'4008","reported_seq":4082,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.208728+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-25T00:21:19.208728+0000","last_peered":"2023-10-25T00:21:19.208728+0000","last_clean":"2023-10-25T00:21:19.208728+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-25T00:21:19.208728+0000","last_undegraded":"2023-10-25T00:21:19.208728+0000","last_fullsized":"2023-10-25T00:21:19.208728+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20791,"num_write_kb":267770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"72'5469","reported_seq":5550,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793934+0000","last_change":"2023-10-24T23:53:11.832809+0000","last_active":"2023-10-25T00:22:08.793934+0000","last_peered":"2023-10-25T00:22:08.793934+0000","last_clean":"2023-10-25T00:22:08.793934+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-25T00:22:08.793934+0000","last_undegraded":"2023-10-25T00:22:08.793934+0000","last_fullsized":"2023-10-25T00:22:08.793934+0000","mapping_epoch":25,"log_start":"64'200","ondisk_log_start":"64'200","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:01.151315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":40,"num_write":30646,"num_write_kb":401481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"72'5","reported_seq":116,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.200649+0000","last_change":"2023-10-24T23:53:11.442035+0000","last_active":"2023-10-25T00:21:39.200649+0000","last_peered":"2023-10-25T00:21:39.200649+0000","last_clean":"2023-10-25T00:21:39.200649+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-25T00:21:39.200649+0000","last_undegraded":"2023-10-25T00:21:39.200649+0000","last_fullsized":"2023-10-25T00:21:39.200649+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:13.717844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"72'532","reported_seq":611,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.992831+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-25T00:22:07.992831+0000","last_peered":"2023-10-25T00:22:07.992831+0000","last_clean":"2023-10-25T00:22:07.992831+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-25T00:22:07.992831+0000","last_undegraded":"2023-10-25T00:22:07.992831+0000","last_fullsized":"2023-10-25T00:22:07.992831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3177545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7524,"num_write":504,"num_write_kb":3406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'6069","reported_seq":6142,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183010+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-25T00:21:19.183010+0000","last_peered":"2023-10-25T00:21:19.183010+0000","last_clean":"2023-10-25T00:21:19.183010+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-25T00:21:19.183010+0000","last_undegraded":"2023-10-25T00:21:19.183010+0000","last_fullsized":"2023-10-25T00:21:19.183010+0000","mapping_epoch":21,"log_start":"62'800","ondisk_log_start":"62'800","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":33187,"num_write_kb":387130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'3606","reported_seq":3680,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.451770+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-25T00:22:08.451770+0000","last_peered":"2023-10-25T00:22:08.451770+0000","last_clean":"2023-10-25T00:22:08.451770+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-25T00:22:08.451770+0000","last_undegraded":"2023-10-25T00:22:08.451770+0000","last_fullsized":"2023-10-25T00:22:08.451770+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:26.012711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19493,"num_write_kb":254342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'1061","reported_seq":1182,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.761652+0000","last_change":"2023-10-24T23:53:11.448032+0000","last_active":"2023-10-25T00:22:08.761652+0000","last_peered":"2023-10-25T00:22:08.761652+0000","last_clean":"2023-10-25T00:22:08.761652+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-25T00:22:08.761652+0000","last_undegraded":"2023-10-25T00:22:08.761652+0000","last_fullsized":"2023-10-25T00:22:08.761652+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1061,"log_dups_size":0,"ondisk_log_size":1061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:50:32.490810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":9402,"num_write":1035,"num_write_kb":4717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'3","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.185352+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-25T00:21:39.185352+0000","last_peered":"2023-10-25T00:21:39.185352+0000","last_clean":"2023-10-25T00:21:39.185352+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-25T00:21:39.185352+0000","last_undegraded":"2023-10-25T00:21:39.185352+0000","last_fullsized":"2023-10-25T00:21:39.185352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'5139","reported_seq":5212,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.216678+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-25T00:21:19.216678+0000","last_peered":"2023-10-25T00:21:19.216678+0000","last_clean":"2023-10-25T00:21:19.216678+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-25T00:21:19.216678+0000","last_undegraded":"2023-10-25T00:21:19.216678+0000","last_fullsized":"2023-10-25T00:21:19.216678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27772,"num_write_kb":328469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"72'4843","reported_seq":4913,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.750475+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-25T00:22:08.750475+0000","last_peered":"2023-10-25T00:22:08.750475+0000","last_clean":"2023-10-25T00:22:08.750475+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-25T00:22:08.750475+0000","last_undegraded":"2023-10-25T00:22:08.750475+0000","last_fullsized":"2023-10-25T00:22:08.750475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:10.820967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":27290,"num_write_kb":330906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"72'27","reported_seq":97,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.749463+0000","last_change":"2023-10-24T23:53:11.446925+0000","last_active":"2023-10-25T00:22:08.749463+0000","last_peered":"2023-10-25T00:22:08.749463+0000","last_clean":"2023-10-25T00:22:08.749463+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-25T00:22:08.749463+0000","last_undegraded":"2023-10-25T00:22:08.749463+0000","last_fullsized":"2023-10-25T00:22:08.749463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:59:52.485469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'2","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182801+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-25T00:21:39.182801+0000","last_peered":"2023-10-25T00:21:39.182801+0000","last_clean":"2023-10-25T00:21:39.182801+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-25T00:21:39.182801+0000","last_undegraded":"2023-10-25T00:21:39.182801+0000","last_fullsized":"2023-10-25T00:21:39.182801+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'5814","reported_seq":5888,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189609+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-25T00:21:39.189609+0000","last_peered":"2023-10-25T00:21:39.189609+0000","last_clean":"2023-10-25T00:21:39.189609+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-25T00:21:39.189609+0000","last_undegraded":"2023-10-25T00:21:39.189609+0000","last_fullsized":"2023-10-25T00:21:39.189609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31996,"num_write_kb":382397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"72'3827","reported_seq":3898,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.365063+0000","last_change":"2023-10-24T23:53:11.849056+0000","last_active":"2023-10-25T00:22:08.365063+0000","last_peered":"2023-10-25T00:22:08.365063+0000","last_clean":"2023-10-25T00:22:08.365063+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-25T00:22:08.365063+0000","last_undegraded":"2023-10-25T00:22:08.365063+0000","last_fullsized":"2023-10-25T00:22:08.365063+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:08.932942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":21199,"num_write_kb":277730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"68'4820","reported_seq":4893,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.181692+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-25T00:21:19.181692+0000","last_peered":"2023-10-25T00:21:19.181692+0000","last_clean":"2023-10-25T00:21:19.181692+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-25T00:21:19.181692+0000","last_undegraded":"2023-10-25T00:21:19.181692+0000","last_fullsized":"2023-10-25T00:21:19.181692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25163,"num_write_kb":285293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"72'5","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:06.452266+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-25T00:22:06.452266+0000","last_peered":"2023-10-25T00:22:06.452266+0000","last_clean":"2023-10-25T00:22:06.452266+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-25T00:22:06.452266+0000","last_undegraded":"2023-10-25T00:22:06.452266+0000","last_fullsized":"2023-10-25T00:22:06.452266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"72'5","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.744190+0000","last_change":"2023-10-24T23:53:11.447167+0000","last_active":"2023-10-25T00:22:08.744190+0000","last_peered":"2023-10-25T00:22:08.744190+0000","last_clean":"2023-10-25T00:22:08.744190+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-25T00:22:08.744190+0000","last_undegraded":"2023-10-25T00:22:08.744190+0000","last_fullsized":"2023-10-25T00:22:08.744190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:06:25.006728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"72'4957","reported_seq":5031,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.762994+0000","last_change":"2023-10-24T23:53:11.813704+0000","last_active":"2023-10-25T00:22:08.762994+0000","last_peered":"2023-10-25T00:22:08.762994+0000","last_clean":"2023-10-25T00:22:08.762994+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-25T00:22:08.762994+0000","last_undegraded":"2023-10-25T00:22:08.762994+0000","last_fullsized":"2023-10-25T00:22:08.762994+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:39:09.996039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28093,"num_write_kb":358545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'4782","reported_seq":4860,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.185276+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-25T00:21:19.185276+0000","last_peered":"2023-10-25T00:21:19.185276+0000","last_clean":"2023-10-25T00:21:19.185276+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-25T00:21:19.185276+0000","last_undegraded":"2023-10-25T00:21:19.185276+0000","last_fullsized":"2023-10-25T00:21:19.185276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26432,"num_write_kb":298639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"68'3","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193250+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-25T00:21:39.193250+0000","last_peered":"2023-10-25T00:21:39.193250+0000","last_clean":"2023-10-25T00:21:39.193250+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-25T00:21:39.193250+0000","last_undegraded":"2023-10-25T00:21:39.193250+0000","last_fullsized":"2023-10-25T00:21:39.193250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"72'14","reported_seq":197,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.746812+0000","last_change":"2023-10-24T23:53:11.450974+0000","last_active":"2023-10-25T00:22:08.746812+0000","last_peered":"2023-10-25T00:22:08.746812+0000","last_clean":"2023-10-25T00:22:08.746812+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-25T00:22:08.746812+0000","last_undegraded":"2023-10-25T00:22:08.746812+0000","last_fullsized":"2023-10-25T00:22:08.746812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:51:05.443664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":345,"num_read_kb":345,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"72'2954","reported_seq":3023,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.717448+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-25T00:22:08.717448+0000","last_peered":"2023-10-25T00:22:08.717448+0000","last_clean":"2023-10-25T00:22:08.717448+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-25T00:22:08.717448+0000","last_undegraded":"2023-10-25T00:22:08.717448+0000","last_fullsized":"2023-10-25T00:22:08.717448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:37:30.278659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15961,"num_write_kb":219669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"72'36","reported_seq":152,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.212150+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-25T00:22:10.212150+0000","last_peered":"2023-10-25T00:22:10.212150+0000","last_clean":"2023-10-25T00:22:10.212150+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-25T00:22:10.212150+0000","last_undegraded":"2023-10-25T00:22:10.212150+0000","last_fullsized":"2023-10-25T00:22:10.212150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'4690","reported_seq":4764,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178387+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-25T00:21:19.178387+0000","last_peered":"2023-10-25T00:21:19.178387+0000","last_clean":"2023-10-25T00:21:19.178387+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-25T00:21:19.178387+0000","last_undegraded":"2023-10-25T00:21:19.178387+0000","last_fullsized":"2023-10-25T00:21:19.178387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25172,"num_write_kb":305786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"72'8","reported_seq":120,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.777365+0000","last_change":"2023-10-24T23:53:11.429694+0000","last_active":"2023-10-25T00:22:08.777365+0000","last_peered":"2023-10-25T00:22:08.777365+0000","last_clean":"2023-10-25T00:22:08.777365+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-25T00:22:08.777365+0000","last_undegraded":"2023-10-25T00:22:08.777365+0000","last_fullsized":"2023-10-25T00:22:08.777365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:15.964994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"72'4200","reported_seq":4277,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.728522+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-25T00:22:08.728522+0000","last_peered":"2023-10-25T00:22:08.728522+0000","last_clean":"2023-10-25T00:22:08.728522+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-25T00:22:08.728522+0000","last_undegraded":"2023-10-25T00:22:08.728522+0000","last_fullsized":"2023-10-25T00:22:08.728522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:38:18.678087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":24169,"num_write_kb":293796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"72'389","reported_seq":572,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.983790+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-25T00:22:07.983790+0000","last_peered":"2023-10-25T00:22:07.983790+0000","last_clean":"2023-10-25T00:22:07.983790+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-25T00:22:07.983790+0000","last_undegraded":"2023-10-25T00:22:07.983790+0000","last_fullsized":"2023-10-25T00:22:07.983790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":201,"num_read_kb":90291,"num_write":386,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'5730","reported_seq":5803,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205335+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-25T00:21:19.205335+0000","last_peered":"2023-10-25T00:21:19.205335+0000","last_clean":"2023-10-25T00:21:19.205335+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-25T00:21:19.205335+0000","last_undegraded":"2023-10-25T00:21:19.205335+0000","last_fullsized":"2023-10-25T00:21:19.205335+0000","mapping_epoch":21,"log_start":"61'500","ondisk_log_start":"61'500","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":31652,"num_write_kb":356481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760659+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-25T00:22:08.760659+0000","last_peered":"2023-10-25T00:22:08.760659+0000","last_clean":"2023-10-25T00:22:08.760659+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-25T00:22:08.760659+0000","last_undegraded":"2023-10-25T00:22:08.760659+0000","last_fullsized":"2023-10-25T00:22:08.760659+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:30.163913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"72'4168","reported_seq":4240,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.466973+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-25T00:22:08.466973+0000","last_peered":"2023-10-25T00:22:08.466973+0000","last_clean":"2023-10-25T00:22:08.466973+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-25T00:22:08.466973+0000","last_undegraded":"2023-10-25T00:22:08.466973+0000","last_fullsized":"2023-10-25T00:22:08.466973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:46.000330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23458,"num_write_kb":294850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"68'4339","reported_seq":4413,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197911+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-25T00:21:39.197911+0000","last_peered":"2023-10-25T00:21:39.197911+0000","last_clean":"2023-10-25T00:21:39.197911+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-25T00:21:39.197911+0000","last_undegraded":"2023-10-25T00:21:39.197911+0000","last_fullsized":"2023-10-25T00:21:39.197911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23267,"num_write_kb":265268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.222903+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-25T00:21:19.222903+0000","last_peered":"2023-10-25T00:21:19.222903+0000","last_clean":"2023-10-25T00:21:19.222903+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-25T00:21:19.222903+0000","last_undegraded":"2023-10-25T00:21:19.222903+0000","last_fullsized":"2023-10-25T00:21:19.222903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"72'230","reported_seq":326,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.211353+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-25T00:22:10.211353+0000","last_peered":"2023-10-25T00:22:10.211353+0000","last_clean":"2023-10-25T00:22:10.211353+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-25T00:22:10.211353+0000","last_undegraded":"2023-10-25T00:22:10.211353+0000","last_fullsized":"2023-10-25T00:22:10.211353+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":61,"num_write":589,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"72'11","reported_seq":81,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.756187+0000","last_change":"2023-10-24T23:53:11.433306+0000","last_active":"2023-10-25T00:22:08.756187+0000","last_peered":"2023-10-25T00:22:08.756187+0000","last_clean":"2023-10-25T00:22:08.756187+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-25T00:22:08.756187+0000","last_undegraded":"2023-10-25T00:22:08.756187+0000","last_fullsized":"2023-10-25T00:22:08.756187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:44:38.728345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"72'3993","reported_seq":4067,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793391+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-25T00:22:08.793391+0000","last_peered":"2023-10-25T00:22:08.793391+0000","last_clean":"2023-10-25T00:22:08.793391+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-25T00:22:08.793391+0000","last_undegraded":"2023-10-25T00:22:08.793391+0000","last_fullsized":"2023-10-25T00:22:08.793391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:30:24.805861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21723,"num_write_kb":283850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'4270","reported_seq":4344,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.181969+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-25T00:21:19.181969+0000","last_peered":"2023-10-25T00:21:19.181969+0000","last_clean":"2023-10-25T00:21:19.181969+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-25T00:21:19.181969+0000","last_undegraded":"2023-10-25T00:21:19.181969+0000","last_fullsized":"2023-10-25T00:21:19.181969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23049,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180537+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-25T00:21:19.180537+0000","last_peered":"2023-10-25T00:21:19.180537+0000","last_clean":"2023-10-25T00:21:19.180537+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-25T00:21:19.180537+0000","last_undegraded":"2023-10-25T00:21:19.180537+0000","last_fullsized":"2023-10-25T00:21:19.180537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'9","reported_seq":128,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.260782+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-25T00:21:42.260782+0000","last_peered":"2023-10-25T00:21:42.260782+0000","last_clean":"2023-10-25T00:21:42.260782+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-25T00:21:42.260782+0000","last_undegraded":"2023-10-25T00:21:42.260782+0000","last_fullsized":"2023-10-25T00:21:42.260782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"72'64","reported_seq":214,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.752186+0000","last_change":"2023-10-24T23:53:11.441983+0000","last_active":"2023-10-25T00:22:08.752186+0000","last_peered":"2023-10-25T00:22:08.752186+0000","last_clean":"2023-10-25T00:22:08.752186+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-25T00:22:08.752186+0000","last_undegraded":"2023-10-25T00:22:08.752186+0000","last_fullsized":"2023-10-25T00:22:08.752186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:30.643019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":123,"num_write":59,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"72'3769","reported_seq":3843,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.609361+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-25T00:22:08.609361+0000","last_peered":"2023-10-25T00:22:08.609361+0000","last_clean":"2023-10-25T00:22:08.609361+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-25T00:22:08.609361+0000","last_undegraded":"2023-10-25T00:22:08.609361+0000","last_fullsized":"2023-10-25T00:22:08.609361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:50:53.467058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20807,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'5752","reported_seq":5826,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194706+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-25T00:21:39.194706+0000","last_peered":"2023-10-25T00:21:39.194706+0000","last_clean":"2023-10-25T00:21:39.194706+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-25T00:21:39.194706+0000","last_undegraded":"2023-10-25T00:21:39.194706+0000","last_fullsized":"2023-10-25T00:21:39.194706+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31231,"num_write_kb":354718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199135+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-25T00:21:39.199135+0000","last_peered":"2023-10-25T00:21:39.199135+0000","last_clean":"2023-10-25T00:21:39.199135+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-25T00:21:39.199135+0000","last_undegraded":"2023-10-25T00:21:39.199135+0000","last_fullsized":"2023-10-25T00:21:39.199135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'2","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189313+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-25T00:21:39.189313+0000","last_peered":"2023-10-25T00:21:39.189313+0000","last_clean":"2023-10-25T00:21:39.189313+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-25T00:21:39.189313+0000","last_undegraded":"2023-10-25T00:21:39.189313+0000","last_fullsized":"2023-10-25T00:21:39.189313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"72'4","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.259221+0000","last_change":"2023-10-24T23:53:11.426287+0000","last_active":"2023-10-25T00:22:08.259221+0000","last_peered":"2023-10-25T00:22:08.259221+0000","last_clean":"2023-10-25T00:22:08.259221+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-25T00:22:08.259221+0000","last_undegraded":"2023-10-25T00:22:08.259221+0000","last_fullsized":"2023-10-25T00:22:08.259221+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:02:39.399866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"72'3939","reported_seq":4008,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.780603+0000","last_change":"2023-10-24T23:53:11.811139+0000","last_active":"2023-10-25T00:22:08.780603+0000","last_peered":"2023-10-25T00:22:08.780603+0000","last_clean":"2023-10-25T00:22:08.780603+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-25T00:22:08.780603+0000","last_undegraded":"2023-10-25T00:22:08.780603+0000","last_fullsized":"2023-10-25T00:22:08.780603+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:05.507633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22393,"num_write_kb":262004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'4305","reported_seq":4379,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186001+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-25T00:21:39.186001+0000","last_peered":"2023-10-25T00:21:39.186001+0000","last_clean":"2023-10-25T00:21:39.186001+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-25T00:21:39.186001+0000","last_undegraded":"2023-10-25T00:21:39.186001+0000","last_fullsized":"2023-10-25T00:21:39.186001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22696,"num_write_kb":287574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187237+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-25T00:21:39.187237+0000","last_peered":"2023-10-25T00:21:39.187237+0000","last_clean":"2023-10-25T00:21:39.187237+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-25T00:21:39.187237+0000","last_undegraded":"2023-10-25T00:21:39.187237+0000","last_fullsized":"2023-10-25T00:21:39.187237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.188607+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-25T00:21:39.188607+0000","last_peered":"2023-10-25T00:21:39.188607+0000","last_clean":"2023-10-25T00:21:39.188607+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-25T00:21:39.188607+0000","last_undegraded":"2023-10-25T00:21:39.188607+0000","last_fullsized":"2023-10-25T00:21:39.188607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"72'3","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.892690+0000","last_change":"2023-10-24T23:53:11.426199+0000","last_active":"2023-10-25T00:22:07.892690+0000","last_peered":"2023-10-25T00:22:07.892690+0000","last_clean":"2023-10-25T00:22:07.892690+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-25T00:22:07.892690+0000","last_undegraded":"2023-10-25T00:22:07.892690+0000","last_fullsized":"2023-10-25T00:22:07.892690+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:57:51.018816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"72'5003","reported_seq":5078,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.705186+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-25T00:22:08.705186+0000","last_peered":"2023-10-25T00:22:08.705186+0000","last_clean":"2023-10-25T00:22:08.705186+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-25T00:22:08.705186+0000","last_undegraded":"2023-10-25T00:22:08.705186+0000","last_fullsized":"2023-10-25T00:22:08.705186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:20:16.596993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":13,"num_write":27120,"num_write_kb":350355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'5635","reported_seq":5708,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186387+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-25T00:21:39.186387+0000","last_peered":"2023-10-25T00:21:39.186387+0000","last_clean":"2023-10-25T00:21:39.186387+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-25T00:21:39.186387+0000","last_undegraded":"2023-10-25T00:21:39.186387+0000","last_fullsized":"2023-10-25T00:21:39.186387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30556,"num_write_kb":356140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"68'3","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.196592+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-25T00:21:39.196592+0000","last_peered":"2023-10-25T00:21:39.196592+0000","last_clean":"2023-10-25T00:21:39.196592+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-25T00:21:39.196592+0000","last_undegraded":"2023-10-25T00:21:39.196592+0000","last_fullsized":"2023-10-25T00:21:39.196592+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183369+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-25T00:21:39.183369+0000","last_peered":"2023-10-25T00:21:39.183369+0000","last_clean":"2023-10-25T00:21:39.183369+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-25T00:21:39.183369+0000","last_undegraded":"2023-10-25T00:21:39.183369+0000","last_fullsized":"2023-10-25T00:21:39.183369+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"72'2","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193585+0000","last_change":"2023-10-24T23:53:11.432666+0000","last_active":"2023-10-25T00:21:39.193585+0000","last_peered":"2023-10-25T00:21:39.193585+0000","last_clean":"2023-10-25T00:21:39.193585+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-25T00:21:39.193585+0000","last_undegraded":"2023-10-25T00:21:39.193585+0000","last_fullsized":"2023-10-25T00:21:39.193585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:17.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"72'3753","reported_seq":3826,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:09.504989+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-25T00:22:09.504989+0000","last_peered":"2023-10-25T00:22:09.504989+0000","last_clean":"2023-10-25T00:22:09.504989+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-25T00:22:09.504989+0000","last_undegraded":"2023-10-25T00:22:09.504989+0000","last_fullsized":"2023-10-25T00:22:09.504989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:42.131555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":21045,"num_write_kb":276986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'4101","reported_seq":4173,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.179395+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-25T00:21:19.179395+0000","last_peered":"2023-10-25T00:21:19.179395+0000","last_clean":"2023-10-25T00:21:19.179395+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-25T00:21:19.179395+0000","last_undegraded":"2023-10-25T00:21:19.179395+0000","last_fullsized":"2023-10-25T00:21:19.179395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22383,"num_write_kb":270023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"72'12","reported_seq":86,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.184571+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-25T00:21:39.184571+0000","last_peered":"2023-10-25T00:21:39.184571+0000","last_clean":"2023-10-25T00:21:39.184571+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-25T00:21:39.184571+0000","last_undegraded":"2023-10-25T00:21:39.184571+0000","last_fullsized":"2023-10-25T00:21:39.184571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193953+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-25T00:21:39.193953+0000","last_peered":"2023-10-25T00:21:39.193953+0000","last_clean":"2023-10-25T00:21:39.193953+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-25T00:21:39.193953+0000","last_undegraded":"2023-10-25T00:21:39.193953+0000","last_fullsized":"2023-10-25T00:21:39.193953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"72'248","reported_seq":339,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.188700+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-25T00:22:10.188700+0000","last_peered":"2023-10-25T00:22:10.188700+0000","last_clean":"2023-10-25T00:22:10.188700+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-25T00:22:10.188700+0000","last_undegraded":"2023-10-25T00:22:10.188700+0000","last_fullsized":"2023-10-25T00:22:10.188700+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:55:47.548432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":62,"num_write":607,"num_write_kb":223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"72'5236","reported_seq":5311,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.753672+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-25T00:22:08.753672+0000","last_peered":"2023-10-25T00:22:08.753672+0000","last_clean":"2023-10-25T00:22:08.753672+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-25T00:22:08.753672+0000","last_undegraded":"2023-10-25T00:22:08.753672+0000","last_fullsized":"2023-10-25T00:22:08.753672+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:11:14.431984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":29554,"num_write_kb":368820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'5444","reported_seq":5520,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.216863+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-25T00:21:19.216863+0000","last_peered":"2023-10-25T00:21:19.216863+0000","last_clean":"2023-10-25T00:21:19.216863+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-25T00:21:19.216863+0000","last_undegraded":"2023-10-25T00:21:19.216863+0000","last_fullsized":"2023-10-25T00:21:19.216863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28342,"num_write_kb":342931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"72'66","reported_seq":223,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.219689+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-25T00:22:10.219689+0000","last_peered":"2023-10-25T00:22:10.219689+0000","last_clean":"2023-10-25T00:22:10.219689+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-25T00:22:10.219689+0000","last_undegraded":"2023-10-25T00:22:10.219689+0000","last_fullsized":"2023-10-25T00:22:10.219689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":125,"num_write":61,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193375+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-25T00:21:19.193375+0000","last_peered":"2023-10-25T00:21:19.193375+0000","last_clean":"2023-10-25T00:21:19.193375+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-25T00:21:19.193375+0000","last_undegraded":"2023-10-25T00:21:19.193375+0000","last_fullsized":"2023-10-25T00:21:19.193375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"72'9","reported_seq":123,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760947+0000","last_change":"2023-10-24T23:53:11.436603+0000","last_active":"2023-10-25T00:22:08.760947+0000","last_peered":"2023-10-25T00:22:08.760947+0000","last_clean":"2023-10-25T00:22:08.760947+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-25T00:22:08.760947+0000","last_undegraded":"2023-10-25T00:22:08.760947+0000","last_fullsized":"2023-10-25T00:22:08.760947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:09:05.948928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":135,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"72'3666","reported_seq":3735,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.792399+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-25T00:22:08.792399+0000","last_peered":"2023-10-25T00:22:08.792399+0000","last_clean":"2023-10-25T00:22:08.792399+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-25T00:22:08.792399+0000","last_undegraded":"2023-10-25T00:22:08.792399+0000","last_fullsized":"2023-10-25T00:22:08.792399+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:01:59.639080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20519,"num_write_kb":254391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"68'4182","reported_seq":4256,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187475+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-25T00:21:39.187475+0000","last_peered":"2023-10-25T00:21:39.187475+0000","last_clean":"2023-10-25T00:21:39.187475+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-25T00:21:39.187475+0000","last_undegraded":"2023-10-25T00:21:39.187475+0000","last_fullsized":"2023-10-25T00:21:39.187475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22839,"num_write_kb":259586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'4","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182378+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-25T00:21:39.182378+0000","last_peered":"2023-10-25T00:21:39.182378+0000","last_clean":"2023-10-25T00:21:39.182378+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-25T00:21:39.182378+0000","last_undegraded":"2023-10-25T00:21:39.182378+0000","last_fullsized":"2023-10-25T00:21:39.182378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183752+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-25T00:21:39.183752+0000","last_peered":"2023-10-25T00:21:39.183752+0000","last_clean":"2023-10-25T00:21:39.183752+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-25T00:21:39.183752+0000","last_undegraded":"2023-10-25T00:21:39.183752+0000","last_fullsized":"2023-10-25T00:21:39.183752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"43'161","reported_seq":283,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193737+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-25T00:21:19.193737+0000","last_peered":"2023-10-25T00:21:19.193737+0000","last_clean":"2023-10-25T00:21:19.193737+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-25T00:21:19.193737+0000","last_undegraded":"2023-10-25T00:21:19.193737+0000","last_fullsized":"2023-10-25T00:21:19.193737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"72'2","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784850+0000","last_change":"2023-10-24T23:53:11.451073+0000","last_active":"2023-10-25T00:22:08.784850+0000","last_peered":"2023-10-25T00:22:08.784850+0000","last_clean":"2023-10-25T00:22:08.784850+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-25T00:22:08.784850+0000","last_undegraded":"2023-10-25T00:22:08.784850+0000","last_fullsized":"2023-10-25T00:22:08.784850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:32.324710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"72'4105","reported_seq":4177,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793407+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-25T00:22:08.793407+0000","last_peered":"2023-10-25T00:22:08.793407+0000","last_clean":"2023-10-25T00:22:08.793407+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-25T00:22:08.793407+0000","last_undegraded":"2023-10-25T00:22:08.793407+0000","last_fullsized":"2023-10-25T00:22:08.793407+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:04.125835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":19,"num_write":23147,"num_write_kb":297959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'5705","reported_seq":5779,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189167+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-25T00:21:39.189167+0000","last_peered":"2023-10-25T00:21:39.189167+0000","last_clean":"2023-10-25T00:21:39.189167+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-25T00:21:39.189167+0000","last_undegraded":"2023-10-25T00:21:39.189167+0000","last_fullsized":"2023-10-25T00:21:39.189167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":29845,"num_write_kb":357286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"68'5","reported_seq":78,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183009+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-25T00:21:19.183009+0000","last_peered":"2023-10-25T00:21:19.183009+0000","last_clean":"2023-10-25T00:21:19.183009+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-25T00:21:19.183009+0000","last_undegraded":"2023-10-25T00:21:19.183009+0000","last_fullsized":"2023-10-25T00:21:19.183009+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"72'7","reported_seq":121,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.761062+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-25T00:22:08.761062+0000","last_peered":"2023-10-25T00:22:08.761062+0000","last_clean":"2023-10-25T00:22:08.761062+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-25T00:22:08.761062+0000","last_undegraded":"2023-10-25T00:22:08.761062+0000","last_fullsized":"2023-10-25T00:22:08.761062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:06:53.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"72'4234","reported_seq":4305,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.780516+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-25T00:22:08.780516+0000","last_peered":"2023-10-25T00:22:08.780516+0000","last_clean":"2023-10-25T00:22:08.780516+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-25T00:22:08.780516+0000","last_undegraded":"2023-10-25T00:22:08.780516+0000","last_fullsized":"2023-10-25T00:22:08.780516+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:41.905772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":15,"num_write":23173,"num_write_kb":302022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"68'4052","reported_seq":4125,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197631+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-25T00:21:39.197631+0000","last_peered":"2023-10-25T00:21:39.197631+0000","last_clean":"2023-10-25T00:21:39.197631+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-25T00:21:39.197631+0000","last_undegraded":"2023-10-25T00:21:39.197631+0000","last_fullsized":"2023-10-25T00:21:39.197631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21486,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"68'2","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.204255+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-25T00:21:19.204255+0000","last_peered":"2023-10-25T00:21:19.204255+0000","last_clean":"2023-10-25T00:21:19.204255+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-25T00:21:19.204255+0000","last_undegraded":"2023-10-25T00:21:19.204255+0000","last_fullsized":"2023-10-25T00:21:19.204255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"72'4","reported_seq":94,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.751351+0000","last_change":"2023-10-24T23:53:11.458463+0000","last_active":"2023-10-25T00:22:08.751351+0000","last_peered":"2023-10-25T00:22:08.751351+0000","last_clean":"2023-10-25T00:22:08.751351+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-25T00:22:08.751351+0000","last_undegraded":"2023-10-25T00:22:08.751351+0000","last_fullsized":"2023-10-25T00:22:08.751351+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:11:45.040156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"72'4690","reported_seq":4761,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.556236+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-25T00:22:08.556236+0000","last_peered":"2023-10-25T00:22:08.556236+0000","last_clean":"2023-10-25T00:22:08.556236+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-25T00:22:08.556236+0000","last_undegraded":"2023-10-25T00:22:08.556236+0000","last_fullsized":"2023-10-25T00:22:08.556236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:51.707178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24994,"num_write_kb":341798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'4780","reported_seq":4855,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194222+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-25T00:21:39.194222+0000","last_peered":"2023-10-25T00:21:39.194222+0000","last_clean":"2023-10-25T00:21:39.194222+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-25T00:21:39.194222+0000","last_undegraded":"2023-10-25T00:21:39.194222+0000","last_fullsized":"2023-10-25T00:21:39.194222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25083,"num_write_kb":293163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'3","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:20.638748+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-25T00:21:20.638748+0000","last_peered":"2023-10-25T00:21:20.638748+0000","last_clean":"2023-10-25T00:21:20.638748+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-25T00:21:20.638748+0000","last_undegraded":"2023-10-25T00:21:20.638748+0000","last_fullsized":"2023-10-25T00:21:20.638748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"72'5","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.745978+0000","last_change":"2023-10-24T23:53:11.448675+0000","last_active":"2023-10-25T00:22:08.745978+0000","last_peered":"2023-10-25T00:22:08.745978+0000","last_clean":"2023-10-25T00:22:08.745978+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-25T00:22:08.745978+0000","last_undegraded":"2023-10-25T00:22:08.745978+0000","last_fullsized":"2023-10-25T00:22:08.745978+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:01.502343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"72'3596","reported_seq":3668,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.775009+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-25T00:22:08.775009+0000","last_peered":"2023-10-25T00:22:08.775009+0000","last_clean":"2023-10-25T00:22:08.775009+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-25T00:22:08.775009+0000","last_undegraded":"2023-10-25T00:22:08.775009+0000","last_fullsized":"2023-10-25T00:22:08.775009+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:54.478897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":19820,"num_write_kb":259784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"68'4970","reported_seq":5044,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.201300+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-25T00:21:39.201300+0000","last_peered":"2023-10-25T00:21:39.201300+0000","last_clean":"2023-10-25T00:21:39.201300+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-25T00:21:39.201300+0000","last_undegraded":"2023-10-25T00:21:39.201300+0000","last_fullsized":"2023-10-25T00:21:39.201300+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26721,"num_write_kb":311966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"72'1464","reported_seq":1588,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.213604+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-25T00:22:10.213604+0000","last_peered":"2023-10-25T00:22:10.213604+0000","last_clean":"2023-10-25T00:22:10.213604+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-25T00:22:10.213604+0000","last_undegraded":"2023-10-25T00:22:10.213604+0000","last_fullsized":"2023-10-25T00:22:10.213604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1464,"log_dups_size":0,"ondisk_log_size":1464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":43700,"num_write":1439,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"72'6","reported_seq":96,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.262467+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-25T00:22:08.262467+0000","last_peered":"2023-10-25T00:22:08.262467+0000","last_clean":"2023-10-25T00:22:08.262467+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-25T00:22:08.262467+0000","last_undegraded":"2023-10-25T00:22:08.262467+0000","last_fullsized":"2023-10-25T00:22:08.262467+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:58.545886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"72'3548","reported_seq":3620,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.608884+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-25T00:22:08.608884+0000","last_peered":"2023-10-25T00:22:08.608884+0000","last_clean":"2023-10-25T00:22:08.608884+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-25T00:22:08.608884+0000","last_undegraded":"2023-10-25T00:22:08.608884+0000","last_fullsized":"2023-10-25T00:22:08.608884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:05.595887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20243,"num_write_kb":261539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"68'3393","reported_seq":3469,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194393+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-25T00:21:39.194393+0000","last_peered":"2023-10-25T00:21:39.194393+0000","last_clean":"2023-10-25T00:21:39.194393+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-25T00:21:39.194393+0000","last_undegraded":"2023-10-25T00:21:39.194393+0000","last_fullsized":"2023-10-25T00:21:39.194393+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17500,"num_write_kb":217638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"68'5","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187988+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-25T00:21:39.187988+0000","last_peered":"2023-10-25T00:21:39.187988+0000","last_clean":"2023-10-25T00:21:39.187988+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-25T00:21:39.187988+0000","last_undegraded":"2023-10-25T00:21:39.187988+0000","last_fullsized":"2023-10-25T00:21:39.187988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"72'3","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.738581+0000","last_change":"2023-10-24T23:53:11.454658+0000","last_active":"2023-10-25T00:22:08.738581+0000","last_peered":"2023-10-25T00:22:08.738581+0000","last_clean":"2023-10-25T00:22:08.738581+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-25T00:22:08.738581+0000","last_undegraded":"2023-10-25T00:22:08.738581+0000","last_fullsized":"2023-10-25T00:22:08.738581+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:43:43.869069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"72'3584","reported_seq":3656,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.616239+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-25T00:22:08.616239+0000","last_peered":"2023-10-25T00:22:08.616239+0000","last_clean":"2023-10-25T00:22:08.616239+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-25T00:22:08.616239+0000","last_undegraded":"2023-10-25T00:22:08.616239+0000","last_fullsized":"2023-10-25T00:22:08.616239+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:09:40.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20504,"num_write_kb":254394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'5094","reported_seq":5172,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:20.638687+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-25T00:21:20.638687+0000","last_peered":"2023-10-25T00:21:20.638687+0000","last_clean":"2023-10-25T00:21:20.638687+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-25T00:21:20.638687+0000","last_undegraded":"2023-10-25T00:21:20.638687+0000","last_fullsized":"2023-10-25T00:21:20.638687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27763,"num_write_kb":328490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'6","reported_seq":101,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199601+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-25T00:21:39.199601+0000","last_peered":"2023-10-25T00:21:39.199601+0000","last_clean":"2023-10-25T00:21:39.199601+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-25T00:21:39.199601+0000","last_undegraded":"2023-10-25T00:21:39.199601+0000","last_fullsized":"2023-10-25T00:21:39.199601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"72'1438","reported_seq":1554,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.763905+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-25T00:22:08.763905+0000","last_peered":"2023-10-25T00:22:08.763905+0000","last_clean":"2023-10-25T00:22:08.763905+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-25T00:22:08.763905+0000","last_undegraded":"2023-10-25T00:22:08.763905+0000","last_fullsized":"2023-10-25T00:22:08.763905+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1438,"log_dups_size":0,"ondisk_log_size":1438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:02.591627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":35275,"num_write":1412,"num_write_kb":4487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"72'3292","reported_seq":3364,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.791717+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-25T00:22:08.791717+0000","last_peered":"2023-10-25T00:22:08.791717+0000","last_clean":"2023-10-25T00:22:08.791717+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-25T00:22:08.791717+0000","last_undegraded":"2023-10-25T00:22:08.791717+0000","last_fullsized":"2023-10-25T00:22:08.791717+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:43.015050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17853,"num_write_kb":243602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'4242","reported_seq":4315,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.203827+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-25T00:21:19.203827+0000","last_peered":"2023-10-25T00:21:19.203827+0000","last_clean":"2023-10-25T00:21:19.203827+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-25T00:21:19.203827+0000","last_undegraded":"2023-10-25T00:21:19.203827+0000","last_fullsized":"2023-10-25T00:21:19.203827+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22687,"num_write_kb":275346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'7","reported_seq":124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.498042+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-25T00:21:42.498042+0000","last_peered":"2023-10-25T00:21:42.498042+0000","last_clean":"2023-10-25T00:21:42.498042+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-25T00:21:42.498042+0000","last_undegraded":"2023-10-25T00:21:42.498042+0000","last_fullsized":"2023-10-25T00:21:42.498042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"72'5","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.259848+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-25T00:22:08.259848+0000","last_peered":"2023-10-25T00:22:08.259848+0000","last_clean":"2023-10-25T00:22:08.259848+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-25T00:22:08.259848+0000","last_undegraded":"2023-10-25T00:22:08.259848+0000","last_fullsized":"2023-10-25T00:22:08.259848+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:59.068359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"72'4091","reported_seq":4163,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.758376+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-25T00:22:08.758376+0000","last_peered":"2023-10-25T00:22:08.758376+0000","last_clean":"2023-10-25T00:22:08.758376+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-25T00:22:08.758376+0000","last_undegraded":"2023-10-25T00:22:08.758376+0000","last_fullsized":"2023-10-25T00:22:08.758376+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:13:03.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22779,"num_write_kb":288098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'4310","reported_seq":4383,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.176404+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-25T00:21:19.176404+0000","last_peered":"2023-10-25T00:21:19.176404+0000","last_clean":"2023-10-25T00:21:19.176404+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-25T00:21:19.176404+0000","last_undegraded":"2023-10-25T00:21:19.176404+0000","last_fullsized":"2023-10-25T00:21:19.176404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23791,"num_write_kb":283914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'4","reported_seq":99,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199642+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-25T00:21:39.199642+0000","last_peered":"2023-10-25T00:21:39.199642+0000","last_clean":"2023-10-25T00:21:39.199642+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-25T00:21:39.199642+0000","last_undegraded":"2023-10-25T00:21:39.199642+0000","last_fullsized":"2023-10-25T00:21:39.199642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"72'2","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.702572+0000","last_change":"2023-10-24T23:53:11.451013+0000","last_active":"2023-10-25T00:22:08.702572+0000","last_peered":"2023-10-25T00:22:08.702572+0000","last_clean":"2023-10-25T00:22:08.702572+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-25T00:22:08.702572+0000","last_undegraded":"2023-10-25T00:22:08.702572+0000","last_fullsized":"2023-10-25T00:22:08.702572+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:52:22.503023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"72'3695","reported_seq":3768,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.635013+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-25T00:22:08.635013+0000","last_peered":"2023-10-25T00:22:08.635013+0000","last_clean":"2023-10-25T00:22:08.635013+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-25T00:22:08.635013+0000","last_undegraded":"2023-10-25T00:22:08.635013+0000","last_fullsized":"2023-10-25T00:22:08.635013+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:05:30.149693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21426,"num_write_kb":254037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.776347+0000","last_change":"2023-10-24T23:53:11.434538+0000","last_active":"2023-10-25T00:22:08.776347+0000","last_peered":"2023-10-25T00:22:08.776347+0000","last_clean":"2023-10-25T00:22:08.776347+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-25T00:22:08.776347+0000","last_undegraded":"2023-10-25T00:22:08.776347+0000","last_fullsized":"2023-10-25T00:22:08.776347+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:55.179253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"72'15","reported_seq":196,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.481650+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-25T00:21:42.481650+0000","last_peered":"2023-10-25T00:21:42.481650+0000","last_clean":"2023-10-25T00:21:42.481650+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-25T00:21:42.481650+0000","last_undegraded":"2023-10-25T00:21:42.481650+0000","last_fullsized":"2023-10-25T00:21:42.481650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":327,"num_read_kb":327,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'5249","reported_seq":5329,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183983+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-25T00:21:39.183983+0000","last_peered":"2023-10-25T00:21:39.183983+0000","last_clean":"2023-10-25T00:21:39.183983+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-25T00:21:39.183983+0000","last_undegraded":"2023-10-25T00:21:39.183983+0000","last_fullsized":"2023-10-25T00:21:39.183983+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":27646,"num_write_kb":332285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"72'4380","reported_seq":4450,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.769070+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-25T00:22:08.769070+0000","last_peered":"2023-10-25T00:22:08.769070+0000","last_clean":"2023-10-25T00:22:08.769070+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-25T00:22:08.769070+0000","last_undegraded":"2023-10-25T00:22:08.769070+0000","last_fullsized":"2023-10-25T00:22:08.769070+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:33:08.373963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23662,"num_write_kb":324338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"72'5","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.785854+0000","last_change":"2023-10-24T23:53:11.433387+0000","last_active":"2023-10-25T00:22:08.785854+0000","last_peered":"2023-10-25T00:22:08.785854+0000","last_clean":"2023-10-25T00:22:08.785854+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-25T00:22:08.785854+0000","last_undegraded":"2023-10-25T00:22:08.785854+0000","last_fullsized":"2023-10-25T00:22:08.785854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:29.148450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"68'5","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.194080+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-25T00:21:19.194080+0000","last_peered":"2023-10-25T00:21:19.194080+0000","last_clean":"2023-10-25T00:21:19.194080+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-25T00:21:19.194080+0000","last_undegraded":"2023-10-25T00:21:19.194080+0000","last_fullsized":"2023-10-25T00:21:19.194080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'4442","reported_seq":4514,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195509+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-25T00:21:39.195509+0000","last_peered":"2023-10-25T00:21:39.195509+0000","last_clean":"2023-10-25T00:21:39.195509+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-25T00:21:39.195509+0000","last_undegraded":"2023-10-25T00:21:39.195509+0000","last_fullsized":"2023-10-25T00:21:39.195509+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22810,"num_write_kb":296225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"72'4076","reported_seq":4148,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784809+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-25T00:22:08.784809+0000","last_peered":"2023-10-25T00:22:08.784809+0000","last_clean":"2023-10-25T00:22:08.784809+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-25T00:22:08.784809+0000","last_undegraded":"2023-10-25T00:22:08.784809+0000","last_fullsized":"2023-10-25T00:22:08.784809+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:33:06.172412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22055,"num_write_kb":297609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"68'3","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.211351+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-25T00:21:19.211351+0000","last_peered":"2023-10-25T00:21:19.211351+0000","last_clean":"2023-10-25T00:21:19.211351+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-25T00:21:19.211351+0000","last_undegraded":"2023-10-25T00:21:19.211351+0000","last_fullsized":"2023-10-25T00:21:19.211351+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'4101","reported_seq":4174,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.188446+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-25T00:21:39.188446+0000","last_peered":"2023-10-25T00:21:39.188446+0000","last_clean":"2023-10-25T00:21:39.188446+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-25T00:21:39.188446+0000","last_undegraded":"2023-10-25T00:21:39.188446+0000","last_fullsized":"2023-10-25T00:21:39.188446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22343,"num_write_kb":253486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"72'393","reported_seq":573,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:02.239640+0000","last_change":"2023-10-24T23:53:11.446979+0000","last_active":"2023-10-25T00:22:02.239640+0000","last_peered":"2023-10-25T00:22:02.239640+0000","last_clean":"2023-10-25T00:22:02.239640+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-25T00:22:02.239640+0000","last_undegraded":"2023-10-25T00:22:02.239640+0000","last_fullsized":"2023-10-25T00:22:02.239640+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:54:52.843631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":86197,"num_write":389,"num_write_kb":4306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"72'4606","reported_seq":4676,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784571+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-25T00:22:08.784571+0000","last_peered":"2023-10-25T00:22:08.784571+0000","last_clean":"2023-10-25T00:22:08.784571+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-25T00:22:08.784571+0000","last_undegraded":"2023-10-25T00:22:08.784571+0000","last_fullsized":"2023-10-25T00:22:08.784571+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:18:48.679682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25763,"num_write_kb":316477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"72'11","reported_seq":127,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:40.179752+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-25T00:21:40.179752+0000","last_peered":"2023-10-25T00:21:40.179752+0000","last_clean":"2023-10-25T00:21:40.179752+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-25T00:21:40.179752+0000","last_undegraded":"2023-10-25T00:21:40.179752+0000","last_fullsized":"2023-10-25T00:21:40.179752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'5680","reported_seq":5756,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183466+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-25T00:21:19.183466+0000","last_peered":"2023-10-25T00:21:19.183466+0000","last_clean":"2023-10-25T00:21:19.183466+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-25T00:21:19.183466+0000","last_undegraded":"2023-10-25T00:21:19.183466+0000","last_fullsized":"2023-10-25T00:21:19.183466+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29749,"num_write_kb":360962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"72'38","reported_seq":172,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:09.508467+0000","last_change":"2023-10-24T23:53:11.436964+0000","last_active":"2023-10-25T00:22:09.508467+0000","last_peered":"2023-10-25T00:22:09.508467+0000","last_clean":"2023-10-25T00:22:09.508467+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-25T00:22:09.508467+0000","last_undegraded":"2023-10-25T00:22:09.508467+0000","last_fullsized":"2023-10-25T00:22:09.508467+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:04:15.366966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":39,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"72'4808","reported_seq":4884,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.778926+0000","last_change":"2023-10-24T23:53:11.814304+0000","last_active":"2023-10-25T00:22:08.778926+0000","last_peered":"2023-10-25T00:22:08.778926+0000","last_clean":"2023-10-25T00:22:08.778926+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-25T00:22:08.778926+0000","last_undegraded":"2023-10-25T00:22:08.778926+0000","last_fullsized":"2023-10-25T00:22:08.778926+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:16.635178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26552,"num_write_kb":336125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"68'4747","reported_seq":4822,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182709+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-25T00:21:39.182709+0000","last_peered":"2023-10-25T00:21:39.182709+0000","last_clean":"2023-10-25T00:21:39.182709+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-25T00:21:39.182709+0000","last_undegraded":"2023-10-25T00:21:39.182709+0000","last_fullsized":"2023-10-25T00:21:39.182709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24847,"num_write_kb":316685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"72'4283","reported_seq":4356,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.589765+0000","last_change":"2023-10-24T23:53:11.807096+0000","last_active":"2023-10-25T00:22:08.589765+0000","last_peered":"2023-10-25T00:22:08.589765+0000","last_clean":"2023-10-25T00:22:08.589765+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-25T00:22:08.589765+0000","last_undegraded":"2023-10-25T00:22:08.589765+0000","last_fullsized":"2023-10-25T00:22:08.589765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:25.144578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23853,"num_write_kb":296504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"68'4036","reported_seq":4111,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183566+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-25T00:21:39.183566+0000","last_peered":"2023-10-25T00:21:39.183566+0000","last_clean":"2023-10-25T00:21:39.183566+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-25T00:21:39.183566+0000","last_undegraded":"2023-10-25T00:21:39.183566+0000","last_fullsized":"2023-10-25T00:21:39.183566+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21233,"num_write_kb":273467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"72'4276","reported_seq":4349,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.743483+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-25T00:22:08.743483+0000","last_peered":"2023-10-25T00:22:08.743483+0000","last_clean":"2023-10-25T00:22:08.743483+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-25T00:22:08.743483+0000","last_undegraded":"2023-10-25T00:22:08.743483+0000","last_fullsized":"2023-10-25T00:22:08.743483+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:38.946585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23362,"num_write_kb":310817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"68'6201","reported_seq":6278,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.198522+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-25T00:21:39.198522+0000","last_peered":"2023-10-25T00:21:39.198522+0000","last_clean":"2023-10-25T00:21:39.198522+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-25T00:21:39.198522+0000","last_undegraded":"2023-10-25T00:21:39.198522+0000","last_fullsized":"2023-10-25T00:21:39.198522+0000","mapping_epoch":21,"log_start":"62'1000","ondisk_log_start":"62'1000","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33486,"num_write_kb":384441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"72'4018","reported_seq":4092,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.764145+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-25T00:22:08.764145+0000","last_peered":"2023-10-25T00:22:08.764145+0000","last_clean":"2023-10-25T00:22:08.764145+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-25T00:22:08.764145+0000","last_undegraded":"2023-10-25T00:22:08.764145+0000","last_fullsized":"2023-10-25T00:22:08.764145+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:48:24.911279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":21920,"num_write_kb":288651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"68'4522","reported_seq":4598,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186452+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-25T00:21:39.186452+0000","last_peered":"2023-10-25T00:21:39.186452+0000","last_clean":"2023-10-25T00:21:39.186452+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-25T00:21:39.186452+0000","last_undegraded":"2023-10-25T00:21:39.186452+0000","last_fullsized":"2023-10-25T00:21:39.186452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24476,"num_write_kb":282500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"72'4209","reported_seq":4279,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760824+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-25T00:22:08.760824+0000","last_peered":"2023-10-25T00:22:08.760824+0000","last_clean":"2023-10-25T00:22:08.760824+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-25T00:22:08.760824+0000","last_undegraded":"2023-10-25T00:22:08.760824+0000","last_fullsized":"2023-10-25T00:22:08.760824+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:21.165755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24141,"num_write_kb":281325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"68'4915","reported_seq":4988,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180147+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-25T00:21:19.180147+0000","last_peered":"2023-10-25T00:21:19.180147+0000","last_clean":"2023-10-25T00:21:19.180147+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-25T00:21:19.180147+0000","last_undegraded":"2023-10-25T00:21:19.180147+0000","last_fullsized":"2023-10-25T00:21:19.180147+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26161,"num_write_kb":314084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"72'4933","reported_seq":5003,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.718975+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-25T00:22:08.718975+0000","last_peered":"2023-10-25T00:22:08.718975+0000","last_clean":"2023-10-25T00:22:08.718975+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-25T00:22:08.718975+0000","last_undegraded":"2023-10-25T00:22:08.718975+0000","last_fullsized":"2023-10-25T00:22:08.718975+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26695,"num_write_kb":352905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"68'5055","reported_seq":5129,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180248+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-25T00:21:19.180248+0000","last_peered":"2023-10-25T00:21:19.180248+0000","last_clean":"2023-10-25T00:21:19.180248+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-25T00:21:19.180248+0000","last_undegraded":"2023-10-25T00:21:19.180248+0000","last_fullsized":"2023-10-25T00:21:19.180248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27516,"num_write_kb":327406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"72'4083","reported_seq":4157,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.731484+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-25T00:22:08.731484+0000","last_peered":"2023-10-25T00:22:08.731484+0000","last_clean":"2023-10-25T00:22:08.731484+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-25T00:22:08.731484+0000","last_undegraded":"2023-10-25T00:22:08.731484+0000","last_fullsized":"2023-10-25T00:22:08.731484+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:54.422365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":22892,"num_write_kb":296872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"68'4777","reported_seq":4852,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178413+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-25T00:21:19.178413+0000","last_peered":"2023-10-25T00:21:19.178413+0000","last_clean":"2023-10-25T00:21:19.178413+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-25T00:21:19.178413+0000","last_undegraded":"2023-10-25T00:21:19.178413+0000","last_fullsized":"2023-10-25T00:21:19.178413+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25812,"num_write_kb":300181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"72'4618","reported_seq":4693,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.741877+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-25T00:22:08.741877+0000","last_peered":"2023-10-25T00:22:08.741877+0000","last_clean":"2023-10-25T00:22:08.741877+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-25T00:22:08.741877+0000","last_undegraded":"2023-10-25T00:22:08.741877+0000","last_fullsized":"2023-10-25T00:22:08.741877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:01:11.838035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":26180,"num_write_kb":326342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"68'5340","reported_seq":5413,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178233+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-25T00:21:19.178233+0000","last_peered":"2023-10-25T00:21:19.178233+0000","last_clean":"2023-10-25T00:21:19.178233+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-25T00:21:19.178233+0000","last_undegraded":"2023-10-25T00:21:19.178233+0000","last_fullsized":"2023-10-25T00:21:19.178233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28701,"num_write_kb":348521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"72'4171","reported_seq":4242,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.497360+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-25T00:22:08.497360+0000","last_peered":"2023-10-25T00:22:08.497360+0000","last_clean":"2023-10-25T00:22:08.497360+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-25T00:22:08.497360+0000","last_undegraded":"2023-10-25T00:22:08.497360+0000","last_fullsized":"2023-10-25T00:22:08.497360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:09:25.424344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22551,"num_write_kb":303498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"68'4493","reported_seq":4567,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.192644+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-25T00:21:19.192644+0000","last_peered":"2023-10-25T00:21:19.192644+0000","last_clean":"2023-10-25T00:21:19.192644+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-25T00:21:19.192644+0000","last_undegraded":"2023-10-25T00:21:19.192644+0000","last_fullsized":"2023-10-25T00:21:19.192644+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23797,"num_write_kb":300260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"72'4763","reported_seq":4833,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.772760+0000","last_change":"2023-10-24T23:53:44.474203+0000","last_active":"2023-10-25T00:22:08.772760+0000","last_peered":"2023-10-25T00:22:08.772760+0000","last_clean":"2023-10-25T00:22:08.772760+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-25T00:22:08.772760+0000","last_undegraded":"2023-10-25T00:22:08.772760+0000","last_fullsized":"2023-10-25T00:22:08.772760+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:16:50.292761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":25751,"num_write_kb":345225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"68'4946","reported_seq":5019,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183160+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-25T00:21:39.183160+0000","last_peered":"2023-10-25T00:21:39.183160+0000","last_clean":"2023-10-25T00:21:39.183160+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-25T00:21:39.183160+0000","last_undegraded":"2023-10-25T00:21:39.183160+0000","last_fullsized":"2023-10-25T00:21:39.183160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26583,"num_write_kb":315633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"72'4352","reported_seq":4423,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.762501+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-25T00:22:08.762501+0000","last_peered":"2023-10-25T00:22:08.762501+0000","last_clean":"2023-10-25T00:22:08.762501+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-25T00:22:08.762501+0000","last_undegraded":"2023-10-25T00:22:08.762501+0000","last_fullsized":"2023-10-25T00:22:08.762501+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:10:20.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23957,"num_write_kb":301052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"68'5026","reported_seq":5099,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193252+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-25T00:21:39.193252+0000","last_peered":"2023-10-25T00:21:39.193252+0000","last_clean":"2023-10-25T00:21:39.193252+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-25T00:21:39.193252+0000","last_undegraded":"2023-10-25T00:21:39.193252+0000","last_fullsized":"2023-10-25T00:21:39.193252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26401,"num_write_kb":310586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"72'4393","reported_seq":4468,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.745975+0000","last_change":"2023-10-24T23:53:11.860133+0000","last_active":"2023-10-25T00:22:08.745975+0000","last_peered":"2023-10-25T00:22:08.745975+0000","last_clean":"2023-10-25T00:22:08.745975+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-25T00:22:08.745975+0000","last_undegraded":"2023-10-25T00:22:08.745975+0000","last_fullsized":"2023-10-25T00:22:08.745975+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:43.051362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":24752,"num_write_kb":296109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"68'5620","reported_seq":5697,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197727+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-25T00:21:39.197727+0000","last_peered":"2023-10-25T00:21:39.197727+0000","last_clean":"2023-10-25T00:21:39.197727+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-25T00:21:39.197727+0000","last_undegraded":"2023-10-25T00:21:39.197727+0000","last_fullsized":"2023-10-25T00:21:39.197727+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29863,"num_write_kb":337169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"72'4340","reported_seq":4414,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.730151+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-25T00:22:08.730151+0000","last_peered":"2023-10-25T00:22:08.730151+0000","last_clean":"2023-10-25T00:22:08.730151+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-25T00:22:08.730151+0000","last_undegraded":"2023-10-25T00:22:08.730151+0000","last_fullsized":"2023-10-25T00:22:08.730151+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:39:14.947372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":36,"num_write":23367,"num_write_kb":302669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"68'4464","reported_seq":4536,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.200961+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-25T00:21:39.200961+0000","last_peered":"2023-10-25T00:21:39.200961+0000","last_clean":"2023-10-25T00:21:39.200961+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-25T00:21:39.200961+0000","last_undegraded":"2023-10-25T00:21:39.200961+0000","last_fullsized":"2023-10-25T00:21:39.200961+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24372,"num_write_kb":278074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"72'5589","reported_seq":5659,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.768377+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-25T00:22:08.768377+0000","last_peered":"2023-10-25T00:22:08.768377+0000","last_clean":"2023-10-25T00:22:08.768377+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-25T00:22:08.768377+0000","last_undegraded":"2023-10-25T00:22:08.768377+0000","last_fullsized":"2023-10-25T00:22:08.768377+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:37.413259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":30206,"num_write_kb":399557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"68'4542","reported_seq":4615,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.196542+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-25T00:21:39.196542+0000","last_peered":"2023-10-25T00:21:39.196542+0000","last_clean":"2023-10-25T00:21:39.196542+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-25T00:21:39.196542+0000","last_undegraded":"2023-10-25T00:21:39.196542+0000","last_fullsized":"2023-10-25T00:21:39.196542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23889,"num_write_kb":292323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"72'3430","reported_seq":3501,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.718898+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-25T00:22:08.718898+0000","last_peered":"2023-10-25T00:22:08.718898+0000","last_clean":"2023-10-25T00:22:08.718898+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-25T00:22:08.718898+0000","last_undegraded":"2023-10-25T00:22:08.718898+0000","last_fullsized":"2023-10-25T00:22:08.718898+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:31.344724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19411,"num_write_kb":241862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"68'4719","reported_seq":4792,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.207948+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-25T00:21:19.207948+0000","last_peered":"2023-10-25T00:21:19.207948+0000","last_clean":"2023-10-25T00:21:19.207948+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-25T00:21:19.207948+0000","last_undegraded":"2023-10-25T00:21:19.207948+0000","last_fullsized":"2023-10-25T00:21:19.207948+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25020,"num_write_kb":305197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"72'4169","reported_seq":4237,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.618436+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-25T00:22:08.618436+0000","last_peered":"2023-10-25T00:22:08.618436+0000","last_clean":"2023-10-25T00:22:08.618436+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-25T00:22:08.618436+0000","last_undegraded":"2023-10-25T00:22:08.618436+0000","last_fullsized":"2023-10-25T00:22:08.618436+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23235,"num_write_kb":306175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"68'5952","reported_seq":6033,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.215357+0000","last_change":"2023-10-24T23:53:44.629959+0000","last_active":"2023-10-25T00:21:19.215357+0000","last_peered":"2023-10-25T00:21:19.215357+0000","last_clean":"2023-10-25T00:21:19.215357+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-25T00:21:19.215357+0000","last_undegraded":"2023-10-25T00:21:19.215357+0000","last_fullsized":"2023-10-25T00:21:19.215357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:59:36.474350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32341,"num_write_kb":375441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"72'3717","reported_seq":3786,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.786783+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-25T00:22:08.786783+0000","last_peered":"2023-10-25T00:22:08.786783+0000","last_clean":"2023-10-25T00:22:08.786783+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-25T00:22:08.786783+0000","last_undegraded":"2023-10-25T00:22:08.786783+0000","last_fullsized":"2023-10-25T00:22:08.786783+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:32:05.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21101,"num_write_kb":269108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"68'4825","reported_seq":4899,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.204063+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-25T00:21:19.204063+0000","last_peered":"2023-10-25T00:21:19.204063+0000","last_clean":"2023-10-25T00:21:19.204063+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-25T00:21:19.204063+0000","last_undegraded":"2023-10-25T00:21:19.204063+0000","last_fullsized":"2023-10-25T00:21:19.204063+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25319,"num_write_kb":294113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"72'3714","reported_seq":3786,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.739265+0000","last_change":"2023-10-24T23:53:11.847814+0000","last_active":"2023-10-25T00:22:08.739265+0000","last_peered":"2023-10-25T00:22:08.739265+0000","last_clean":"2023-10-25T00:22:08.739265+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-25T00:22:08.739265+0000","last_undegraded":"2023-10-25T00:22:08.739265+0000","last_fullsized":"2023-10-25T00:22:08.739265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:59.075858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21148,"num_write_kb":265324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"68'4878","reported_seq":4953,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.194001+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-25T00:21:19.194001+0000","last_peered":"2023-10-25T00:21:19.194001+0000","last_clean":"2023-10-25T00:21:19.194001+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-25T00:21:19.194001+0000","last_undegraded":"2023-10-25T00:21:19.194001+0000","last_fullsized":"2023-10-25T00:21:19.194001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26154,"num_write_kb":305558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"72'3434","reported_seq":3504,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.726527+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-25T00:22:08.726527+0000","last_peered":"2023-10-25T00:22:08.726527+0000","last_clean":"2023-10-25T00:22:08.726527+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-25T00:22:08.726527+0000","last_undegraded":"2023-10-25T00:22:08.726527+0000","last_fullsized":"2023-10-25T00:22:08.726527+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:25:12.525273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18449,"num_write_kb":262373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"68'4672","reported_seq":4746,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178513+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-25T00:21:19.178513+0000","last_peered":"2023-10-25T00:21:19.178513+0000","last_clean":"2023-10-25T00:21:19.178513+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-25T00:21:19.178513+0000","last_undegraded":"2023-10-25T00:21:19.178513+0000","last_fullsized":"2023-10-25T00:21:19.178513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25223,"num_write_kb":293792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"72'4013","reported_seq":4088,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.743804+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-25T00:22:08.743804+0000","last_peered":"2023-10-25T00:22:08.743804+0000","last_clean":"2023-10-25T00:22:08.743804+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-25T00:22:08.743804+0000","last_undegraded":"2023-10-25T00:22:08.743804+0000","last_fullsized":"2023-10-25T00:22:08.743804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:12:21.026996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":22066,"num_write_kb":297298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"68'3918","reported_seq":3990,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194748+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-25T00:21:39.194748+0000","last_peered":"2023-10-25T00:21:39.194748+0000","last_clean":"2023-10-25T00:21:39.194748+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-25T00:21:39.194748+0000","last_undegraded":"2023-10-25T00:21:39.194748+0000","last_fullsized":"2023-10-25T00:21:39.194748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21675,"num_write_kb":247000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"72'4401","reported_seq":4472,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.740526+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-25T00:22:08.740526+0000","last_peered":"2023-10-25T00:22:08.740526+0000","last_clean":"2023-10-25T00:22:08.740526+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-25T00:22:08.740526+0000","last_undegraded":"2023-10-25T00:22:08.740526+0000","last_fullsized":"2023-10-25T00:22:08.740526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:07.909021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":24294,"num_write_kb":326749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"68'3970","reported_seq":4042,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194610+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-25T00:21:39.194610+0000","last_peered":"2023-10-25T00:21:39.194610+0000","last_clean":"2023-10-25T00:21:39.194610+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-25T00:21:39.194610+0000","last_undegraded":"2023-10-25T00:21:39.194610+0000","last_fullsized":"2023-10-25T00:21:39.194610+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21202,"num_write_kb":257085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"72'5559","reported_seq":5636,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.789325+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-25T00:22:08.789325+0000","last_peered":"2023-10-25T00:22:08.789325+0000","last_clean":"2023-10-25T00:22:08.789325+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-25T00:22:08.789325+0000","last_undegraded":"2023-10-25T00:22:08.789325+0000","last_fullsized":"2023-10-25T00:22:08.789325+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:48.217889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":31202,"num_write_kb":391572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"68'4936","reported_seq":5009,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.185286+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-25T00:21:39.185286+0000","last_peered":"2023-10-25T00:21:39.185286+0000","last_clean":"2023-10-25T00:21:39.185286+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-25T00:21:39.185286+0000","last_undegraded":"2023-10-25T00:21:39.185286+0000","last_fullsized":"2023-10-25T00:21:39.185286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26134,"num_write_kb":309652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"72'3558","reported_seq":3632,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.781338+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-25T00:22:08.781338+0000","last_peered":"2023-10-25T00:22:08.781338+0000","last_clean":"2023-10-25T00:22:08.781338+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-25T00:22:08.781338+0000","last_undegraded":"2023-10-25T00:22:08.781338+0000","last_fullsized":"2023-10-25T00:22:08.781338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:10:18.973606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":34,"num_write":19715,"num_write_kb":243025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"68'5282","reported_seq":5355,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.192868+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-25T00:21:39.192868+0000","last_peered":"2023-10-25T00:21:39.192868+0000","last_clean":"2023-10-25T00:21:39.192868+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-25T00:21:39.192868+0000","last_undegraded":"2023-10-25T00:21:39.192868+0000","last_fullsized":"2023-10-25T00:21:39.192868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27848,"num_write_kb":324561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"72'3914","reported_seq":3989,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.624743+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-25T00:22:08.624743+0000","last_peered":"2023-10-25T00:22:08.624743+0000","last_clean":"2023-10-25T00:22:08.624743+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-25T00:22:08.624743+0000","last_undegraded":"2023-10-25T00:22:08.624743+0000","last_fullsized":"2023-10-25T00:22:08.624743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:24:45.536141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":20938,"num_write_kb":288796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"68'6616","reported_seq":6693,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194923+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-25T00:21:39.194923+0000","last_peered":"2023-10-25T00:21:39.194923+0000","last_clean":"2023-10-25T00:21:39.194923+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-25T00:21:39.194923+0000","last_undegraded":"2023-10-25T00:21:39.194923+0000","last_fullsized":"2023-10-25T00:21:39.194923+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36221,"num_write_kb":423916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"72'3536","reported_seq":3606,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.700035+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-25T00:22:08.700035+0000","last_peered":"2023-10-25T00:22:08.700035+0000","last_clean":"2023-10-25T00:22:08.700035+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-25T00:22:08.700035+0000","last_undegraded":"2023-10-25T00:22:08.700035+0000","last_fullsized":"2023-10-25T00:22:08.700035+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:06.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19276,"num_write_kb":253506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"68'4980","reported_seq":5050,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.176264+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-25T00:21:19.176264+0000","last_peered":"2023-10-25T00:21:19.176264+0000","last_clean":"2023-10-25T00:21:19.176264+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-25T00:21:19.176264+0000","last_undegraded":"2023-10-25T00:21:19.176264+0000","last_fullsized":"2023-10-25T00:21:19.176264+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26525,"num_write_kb":306963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"72'4160","reported_seq":4232,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.763999+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-25T00:22:08.763999+0000","last_peered":"2023-10-25T00:22:08.763999+0000","last_clean":"2023-10-25T00:22:08.763999+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-25T00:22:08.763999+0000","last_undegraded":"2023-10-25T00:22:08.763999+0000","last_fullsized":"2023-10-25T00:22:08.763999+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:56:02.798009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":21819,"num_write_kb":288439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"68'4434","reported_seq":4506,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193726+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-25T00:21:19.193726+0000","last_peered":"2023-10-25T00:21:19.193726+0000","last_clean":"2023-10-25T00:21:19.193726+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-25T00:21:19.193726+0000","last_undegraded":"2023-10-25T00:21:19.193726+0000","last_fullsized":"2023-10-25T00:21:19.193726+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23791,"num_write_kb":292031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"72'4195","reported_seq":4262,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.728890+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-25T00:22:08.728890+0000","last_peered":"2023-10-25T00:22:08.728890+0000","last_clean":"2023-10-25T00:22:08.728890+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-25T00:22:08.728890+0000","last_undegraded":"2023-10-25T00:22:08.728890+0000","last_fullsized":"2023-10-25T00:22:08.728890+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:40.168016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23000,"num_write_kb":285019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"68'4243","reported_seq":4314,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205516+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-25T00:21:19.205516+0000","last_peered":"2023-10-25T00:21:19.205516+0000","last_clean":"2023-10-25T00:21:19.205516+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-25T00:21:19.205516+0000","last_undegraded":"2023-10-25T00:21:19.205516+0000","last_fullsized":"2023-10-25T00:21:19.205516+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22439,"num_write_kb":266094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"72'4017","reported_seq":4085,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.783880+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-25T00:22:08.783880+0000","last_peered":"2023-10-25T00:22:08.783880+0000","last_clean":"2023-10-25T00:22:08.783880+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-25T00:22:08.783880+0000","last_undegraded":"2023-10-25T00:22:08.783880+0000","last_fullsized":"2023-10-25T00:22:08.783880+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:39.720375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22665,"num_write_kb":291707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"68'4509","reported_seq":4583,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.213920+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-25T00:21:19.213920+0000","last_peered":"2023-10-25T00:21:19.213920+0000","last_clean":"2023-10-25T00:21:19.213920+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-25T00:21:19.213920+0000","last_undegraded":"2023-10-25T00:21:19.213920+0000","last_fullsized":"2023-10-25T00:21:19.213920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23158,"num_write_kb":297883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"72'5235","reported_seq":5312,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.789212+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-25T00:22:08.789212+0000","last_peered":"2023-10-25T00:22:08.789212+0000","last_clean":"2023-10-25T00:22:08.789212+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-25T00:22:08.789212+0000","last_undegraded":"2023-10-25T00:22:08.789212+0000","last_fullsized":"2023-10-25T00:22:08.789212+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:51:40.960630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":15,"num_write":29235,"num_write_kb":367391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"68'4260","reported_seq":4335,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.221539+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-25T00:21:19.221539+0000","last_peered":"2023-10-25T00:21:19.221539+0000","last_clean":"2023-10-25T00:21:19.221539+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-25T00:21:19.221539+0000","last_undegraded":"2023-10-25T00:21:19.221539+0000","last_fullsized":"2023-10-25T00:21:19.221539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23357,"num_write_kb":265837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"72'4656","reported_seq":4735,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.794660+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-25T00:22:08.794660+0000","last_peered":"2023-10-25T00:22:08.794660+0000","last_clean":"2023-10-25T00:22:08.794660+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-25T00:22:08.794660+0000","last_undegraded":"2023-10-25T00:22:08.794660+0000","last_fullsized":"2023-10-25T00:22:08.794660+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:22:42.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":26109,"num_write_kb":317781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"68'4636","reported_seq":4713,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194609+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-25T00:21:39.194609+0000","last_peered":"2023-10-25T00:21:39.194609+0000","last_clean":"2023-10-25T00:21:39.194609+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-25T00:21:39.194609+0000","last_undegraded":"2023-10-25T00:21:39.194609+0000","last_fullsized":"2023-10-25T00:21:39.194609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25124,"num_write_kb":285314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"72'3117","reported_seq":3190,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.574961+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-25T00:22:08.574961+0000","last_peered":"2023-10-25T00:22:08.574961+0000","last_clean":"2023-10-25T00:22:08.574961+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-25T00:22:08.574961+0000","last_undegraded":"2023-10-25T00:22:08.574961+0000","last_fullsized":"2023-10-25T00:22:08.574961+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:47.658717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17240,"num_write_kb":232937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"68'5520","reported_seq":5604,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.192356+0000","last_change":"2023-10-24T23:53:44.630867+0000","last_active":"2023-10-25T00:21:39.192356+0000","last_peered":"2023-10-25T00:21:39.192356+0000","last_clean":"2023-10-25T00:21:39.192356+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-25T00:21:39.192356+0000","last_undegraded":"2023-10-25T00:21:39.192356+0000","last_fullsized":"2023-10-25T00:21:39.192356+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:49.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29665,"num_write_kb":368823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"72'3962","reported_seq":4034,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.769841+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-25T00:22:08.769841+0000","last_peered":"2023-10-25T00:22:08.769841+0000","last_clean":"2023-10-25T00:22:08.769841+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-25T00:22:08.769841+0000","last_undegraded":"2023-10-25T00:22:08.769841+0000","last_fullsized":"2023-10-25T00:22:08.769841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:50.994593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21989,"num_write_kb":280762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"68'4908","reported_seq":4987,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.198947+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-25T00:21:39.198947+0000","last_peered":"2023-10-25T00:21:39.198947+0000","last_clean":"2023-10-25T00:21:39.198947+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-25T00:21:39.198947+0000","last_undegraded":"2023-10-25T00:21:39.198947+0000","last_fullsized":"2023-10-25T00:21:39.198947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26829,"num_write_kb":320697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":320,"num_write":1482516,"num_write_kb":19060243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267718,"ondisk_log_size":267718,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":19490997,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1686,"num_read_kb":209824,"num_write":4665,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39043072,"data_stored":38981994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4535,"ondisk_log_size":4535,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":0,"num_write":1661441,"num_write_kb":19773626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307026,"ondisk_log_size":307026,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":20040877,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1620,"num_read_kb":247649,"num_write":4615,"num_write_kb":21609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40140800,"data_stored":40081754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4486,"ondisk_log_size":4486,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575206,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919080,"kb_used_data":12320,"kb_used_omap":0,"kb_used_meta":906752,"kb_avail":93452760,"statfs":{"total":96636764160,"available":95695626240,"internally_reserved":0,"allocated":12615680,"data_stored":12142216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":928514048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.304,"15min":0.93799999999999994},"min":{"1min":0.22,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.6349999999999998,"5min":13.929,"15min":13.929},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.3,"15min":0.92900000000000005},"min":{"1min":0.29699999999999999,"5min":0.27000000000000002,"15min":0.20999999999999999},"max":{"1min":3.6659999999999999,"5min":11.315,"15min":11.315},"last":1.02}]},{"osd":1,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.714,"15min":1.1639999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.215},"max":{"1min":3.6949999999999998,"5min":28.308,"15min":28.308},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.782,"15min":1.2290000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":3.7490000000000001,"5min":28.346,"15min":28.346},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.3779999999999999,"15min":0.99399999999999999},"min":{"1min":0.28499999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":5.71,"5min":11.430999999999999,"15min":11.430999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.139,"15min":0.94599999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.8799999999999999,"5min":7.9710000000000001,"15min":7.9710000000000001},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":2.3180000000000001,"15min":1.363},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.6150000000000002,"5min":93.664000000000001,"15min":93.664000000000001},"last":1.081},{"interface":"front","average":{"1min":1.2030000000000001,"5min":2.379,"15min":1.3740000000000001},"min":{"1min":0.34499999999999997,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":2.8809999999999998,"5min":93.073999999999998,"15min":93.073999999999998},"last":0.91400000000000003}]},{"osd":4,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0840000000000001,"15min":0.877},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.249},"max":{"1min":2.9089999999999998,"5min":7.6280000000000001,"15min":7.6280000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.714,"15min":1.1339999999999999},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.27200000000000002},"max":{"1min":4.9640000000000004,"5min":27.936,"15min":27.936},"last":0.98299999999999998}]},{"osd":5,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":2.1459999999999999,"5min":1.573,"15min":1.05},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.8170000000000002,"5min":21.126999999999999,"15min":21.126999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.995,"15min":0.94699999999999995},"min":{"1min":0.39800000000000002,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":2.5910000000000002,"5min":11.49,"15min":11.49},"last":0.878}]},{"osd":6,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.4770000000000001,"15min":1.077},"min":{"1min":0.41999999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.8140000000000001,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7290000000000001,"15min":1.1799999999999999},"min":{"1min":0.49399999999999999,"5min":0.38100000000000001,"15min":0.35499999999999998},"max":{"1min":4.8639999999999999,"5min":20.651,"15min":20.651},"last":1.115}]}]},{"osd":6,"up_from":13,"seq":55834575206,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859004,"kb_used_data":5044,"kb_used_omap":0,"kb_used_meta":853952,"kb_avail":93512836,"statfs":{"total":96636764160,"available":95757144064,"internally_reserved":0,"allocated":5165056,"data_stored":4728730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":874446848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.397,"15min":0.93899999999999995},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.9009999999999998,"5min":15.974,"15min":15.974},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.4199999999999999,"15min":0.96799999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":8.2400000000000002,"5min":8.4789999999999992,"15min":8.4789999999999992},"last":0.89500000000000002}]},{"osd":1,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.042,"15min":0.878},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.221},"max":{"1min":3.5630000000000002,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.2549999999999999,"15min":0.98599999999999999},"min":{"1min":0.28699999999999998,"5min":0.254,"15min":0.249},"max":{"1min":3.5840000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.92200000000000004}]},{"osd":2,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.617,"15min":1.0600000000000001},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.6779999999999999,"5min":12.615,"15min":12.615},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.228,"5min":1.591,"15min":1.105},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.702,"5min":12.667999999999999,"15min":12.667999999999999},"last":0.72399999999999998}]},{"osd":3,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.9690000000000001,"15min":1.2350000000000001},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.7350000000000003,"5min":38.649999999999999,"15min":38.649999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":2.0249999999999999,"15min":1.3260000000000001},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.766,"5min":38.683,"15min":38.683},"last":1.014}]},{"osd":4,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.7589999999999999,"15min":1.0960000000000001},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.738,"5min":12.682,"15min":12.682},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4350000000000001,"15min":1.079},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.694,"5min":10.686,"15min":10.686},"last":1.1279999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.1990000000000001,"15min":0.94799999999999995},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.24299999999999999},"max":{"1min":3.6379999999999999,"5min":8.0039999999999996,"15min":8.0039999999999996},"last":1.091},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.3700000000000001,"15min":1.046},"min":{"1min":0.496,"5min":0.33400000000000002,"15min":0.30299999999999999},"max":{"1min":3.903,"5min":11.766999999999999,"15min":11.766999999999999},"last":1.0429999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.907,"5min":1.306,"15min":1.018},"min":{"1min":0.32200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":8.2720000000000002,"5min":12.926,"15min":12.926},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.258,"15min":1.026},"min":{"1min":0.38200000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.1100000000000003,"5min":7.6619999999999999,"15min":7.6619999999999999},"last":1.335}]}]},{"osd":5,"up_from":13,"seq":55834575206,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":958856,"kb_used_data":9152,"kb_used_omap":0,"kb_used_meta":949696,"kb_avail":93412984,"statfs":{"total":96636764160,"available":95654895616,"internally_reserved":0,"allocated":9371648,"data_stored":8923216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":972488704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":2.4649999999999999,"5min":1.4690000000000001,"15min":1.032},"min":{"1min":0.23599999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.837999999999999,"5min":14.837999999999999,"15min":14.837999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.1599999999999999,"15min":0.95999999999999996},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.536,"5min":7.423,"15min":7.423},"last":0.73999999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":1.9990000000000001,"5min":1.6000000000000001,"15min":1.0840000000000001},"min":{"1min":0.27500000000000002,"5min":0.24299999999999999,"15min":0.23400000000000001},"max":{"1min":12.141,"5min":12.141,"15min":12.141},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.518,"15min":1.0820000000000001},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.252},"max":{"1min":3.5950000000000002,"5min":24.648,"15min":24.648},"last":1.008}]},{"osd":2,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":1.9750000000000001,"5min":1.841,"15min":1.236},"min":{"1min":0.29899999999999999,"5min":0.25,"15min":0.25},"max":{"1min":12.175000000000001,"5min":20.760999999999999,"15min":20.760999999999999},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.655,"5min":1.46,"15min":1.0800000000000001},"min":{"1min":0.317,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":12.109999999999999,"5min":20.552,"15min":20.552},"last":0.84599999999999997}]},{"osd":3,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":1.5569999999999999,"15min":1.224},"min":{"1min":0.35599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":12.013,"5min":12.013,"15min":12.013},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.6819999999999999,"15min":1.2549999999999999},"min":{"1min":0.34399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.919,"5min":12.805,"15min":12.805},"last":0.97399999999999998}]},{"osd":4,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":2.3919999999999999,"5min":1.8240000000000001,"15min":1.204},"min":{"1min":0.38900000000000001,"5min":0.26000000000000001,"15min":0.23999999999999999},"max":{"1min":12.201000000000001,"5min":12.728,"15min":12.728},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.131,"5min":1.486,"15min":1.141},"min":{"1min":0.36599999999999999,"5min":0.27600000000000002,"15min":0.27200000000000002},"max":{"1min":3.6179999999999999,"5min":10.943,"15min":10.943},"last":1.081}]},{"osd":6,"last update":"Wed Oct 25 00:21:58 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.369,"15min":1.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.33500000000000002,"15min":0.26400000000000001},"max":{"1min":9.6489999999999991,"5min":17.936,"15min":17.936},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.96,"15min":1.224},"min":{"1min":0.312,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":12.042999999999999,"5min":14.766999999999999,"15min":14.766999999999999},"last":0.872}]},{"osd":7,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":2.984,"5min":1.7709999999999999,"15min":1.26},"min":{"1min":0.373,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":28.178999999999998},"last":1.107},{"interface":"front","average":{"1min":3.23,"5min":2.1659999999999999,"15min":1.3540000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":28.222999999999999,"5min":28.222999999999999,"15min":28.222999999999999},"last":1.1439999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575205,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946272,"kb_used_data":19096,"kb_used_omap":0,"kb_used_meta":927168,"kb_avail":93425568,"statfs":{"total":96636764160,"available":95667781632,"internally_reserved":0,"allocated":19554304,"data_stored":19092590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":949420032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":1.1910000000000001,"15min":0.93000000000000005},"min":{"1min":0.21199999999999999,"5min":0.19700000000000001,"15min":0.187},"max":{"1min":3.6120000000000001,"5min":20.922999999999998,"15min":20.922999999999998},"last":1.081},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.1499999999999999,"15min":0.96199999999999997},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.217},"max":{"1min":1.224,"5min":20.532,"15min":20.532},"last":0.76000000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.3560000000000001,"15min":0.97699999999999998},"min":{"1min":0.224,"5min":0.22,"15min":0.22},"max":{"1min":2.8140000000000001,"5min":20.914000000000001,"15min":20.914000000000001},"last":0.999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.738,"15min":1.157},"min":{"1min":0.28699999999999998,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":2.8399999999999999,"5min":23.265999999999998,"15min":23.265999999999998},"last":0.86099999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":1.5309999999999999,"15min":1.0920000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.3719999999999999,"5min":20.774000000000001,"15min":20.774000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.7509999999999999,"15min":1.2190000000000001},"min":{"1min":0.30099999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.9009999999999998,"5min":20.817,"15min":20.817},"last":1.115}]},{"osd":3,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.575,"15min":1.131},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.1440000000000001,"5min":24.613,"15min":24.613},"last":1.1639999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.3300000000000001,"15min":1.113},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.3650000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.1850000000000001}]},{"osd":5,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.04,"15min":0.88500000000000001},"min":{"1min":0.35599999999999998,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":4.7759999999999998,"5min":6.1459999999999999,"15min":6.1459999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.0640000000000001,"15min":0.94799999999999995},"min":{"1min":0.246,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.5960000000000001,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.92800000000000005}]},{"osd":6,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.337,"15min":1.0740000000000001},"min":{"1min":0.36899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.048,"5min":6.5220000000000002,"15min":6.5220000000000002},"last":1.143},{"interface":"front","average":{"1min":1.2130000000000001,"5min":0.97599999999999998,"15min":0.94299999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":4.3819999999999997,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":1.2529999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:21:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.2450000000000001,"15min":1.0469999999999999},"min":{"1min":0.47399999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":6.3330000000000002,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.4279999999999999,"15min":1.1020000000000001},"min":{"1min":0.51000000000000001,"5min":0.309,"15min":0.309},"max":{"1min":6.3550000000000004,"5min":6.3550000000000004,"15min":6.3550000000000004},"last":1.2130000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575206,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160732,"kb_used_data":9236,"kb_used_omap":0,"kb_used_meta":1151488,"kb_avail":93211108,"statfs":{"total":96636764160,"available":95448174592,"internally_reserved":0,"allocated":9457664,"data_stored":9007793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1179123712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.996,"15min":0.78900000000000003},"min":{"1min":0.20000000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.018,"5min":13.327999999999999,"15min":13.327999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.61299999999999999,"15min":0.73399999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.0489999999999999,"5min":3.556,"15min":5.4580000000000002},"last":0.72699999999999998}]},{"osd":1,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.2290000000000001,"15min":0.86099999999999999},"min":{"1min":0.21199999999999999,"5min":0.20599999999999999,"15min":0.19900000000000001},"max":{"1min":1.796,"5min":13.831,"15min":13.831},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.80900000000000005,"15min":0.76200000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.9300000000000002,"5min":4.4500000000000002,"15min":4.4500000000000002},"last":0.79600000000000004}]},{"osd":2,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.92200000000000004,"15min":0.877},"min":{"1min":0.23799999999999999,"5min":0.223,"15min":0.221},"max":{"1min":2.7890000000000001,"5min":8.2520000000000007,"15min":8.2520000000000007},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.075,"15min":0.90700000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.7360000000000002,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.83799999999999997}]},{"osd":4,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.9630000000000001,"15min":1.2010000000000001},"min":{"1min":0.32100000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.0339999999999998,"5min":39.070999999999998,"15min":39.070999999999998},"last":1.208},{"interface":"front","average":{"1min":1.5389999999999999,"5min":2.0870000000000002,"15min":1.2509999999999999},"min":{"1min":0.34599999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.0300000000000002,"5min":55.104999999999997,"15min":55.104999999999997},"last":0.88400000000000001}]},{"osd":5,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.6379999999999999,"15min":1.117},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.1849999999999996,"5min":31.481999999999999,"15min":31.481999999999999},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.681,"15min":1.1970000000000001},"min":{"1min":0.42499999999999999,"5min":0.35299999999999998,"15min":0.32500000000000001},"max":{"1min":4.2060000000000004,"5min":31.512,"15min":31.512},"last":1.4179999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.9179999999999999,"5min":1.617,"15min":1.1910000000000001},"min":{"1min":0.311,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.7080000000000002,"5min":25.809999999999999,"15min":25.809999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.9790000000000001,"5min":1.4530000000000001,"15min":1.175},"min":{"1min":0.47299999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":4.7779999999999996,"5min":24.036999999999999,"15min":24.036999999999999},"last":0.91900000000000004}]},{"osd":7,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.704,"15min":1.226},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.8529999999999998,"5min":27.581,"15min":27.581},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.6379999999999999,"15min":1.24},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":8.5259999999999998,"5min":24.332000000000001,"15min":24.332000000000001},"last":1.349}]}]},{"osd":2,"up_from":13,"seq":55834575206,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946044,"kb_used_data":5044,"kb_used_omap":0,"kb_used_meta":940992,"kb_avail":93425796,"statfs":{"total":96636764160,"available":95668015104,"internally_reserved":0,"allocated":5165056,"data_stored":4728218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":963575808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.651,"15min":0.98399999999999999},"min":{"1min":0.188,"5min":0.16,"15min":0.16},"max":{"1min":2.9329999999999998,"5min":29.344000000000001,"15min":29.344000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.0800000000000001,"15min":0.82499999999999996},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.0539999999999998,"5min":10.755000000000001,"15min":10.755000000000001},"last":0.49099999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.3640000000000001,"15min":0.92200000000000004},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":2.8860000000000001,"5min":18.617999999999999,"15min":18.617999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":1.099,"15min":0.86099999999999999},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":2.1669999999999998,"5min":15.064,"15min":15.064},"last":0.56699999999999995}]},{"osd":3,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.1259999999999999,"15min":0.88800000000000001},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.192},"max":{"1min":1.766,"5min":13.034000000000001,"15min":13.034000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.5700000000000001,"15min":0.98099999999999998},"min":{"1min":0.26200000000000001,"5min":0.25,"15min":0.25},"max":{"1min":2.964,"5min":18.547999999999998,"15min":18.547999999999998},"last":0.38400000000000001}]},{"osd":4,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.45,"15min":1.1659999999999999},"min":{"1min":0.35099999999999998,"5min":0.252,"15min":0.252},"max":{"1min":2.839,"5min":22.997,"15min":33.378999999999998},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.5,"15min":1.2689999999999999},"min":{"1min":0.38500000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.8690000000000002,"5min":23.030999999999999,"15min":33.558999999999997},"last":0.78700000000000003}]},{"osd":5,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":2.2589999999999999,"15min":1.45},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":4.6440000000000001,"5min":25.183,"15min":34.545999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.038,"5min":1.3879999999999999,"15min":1.169},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.4829999999999997,"5min":11.685,"15min":36.104999999999997},"last":0.65700000000000003}]},{"osd":6,"last update":"Wed Oct 25 00:21:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.208,"15min":1.1100000000000001},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":3.5739999999999998,"5min":9.7149999999999999,"15min":33.661999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.26,"15min":1.2},"min":{"1min":0.45700000000000002,"5min":0.371,"15min":0.371},"max":{"1min":3.5979999999999999,"5min":9.6799999999999997,"15min":34.043999999999997},"last":1.0149999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.3,"15min":1.214},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.395,"5min":11.503,"15min":33.307000000000002},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.3420000000000001,"15min":1.276},"min":{"1min":0.51200000000000001,"5min":0.371,"15min":0.371},"max":{"1min":2.419,"5min":11.468999999999999,"15min":33.344999999999999},"last":0.89400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575207,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1070492,"kb_used_data":4052,"kb_used_omap":0,"kb_used_meta":1066432,"kb_avail":93301348,"statfs":{"total":96636764160,"available":95540580352,"internally_reserved":0,"allocated":4149248,"data_stored":3711479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1092026368},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.1739999999999999,"15min":0.81699999999999995},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.81000000000000005,"5min":21.334,"15min":21.334},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.0449999999999999,"15min":0.84099999999999997},"min":{"1min":0.20499999999999999,"5min":0.19,"15min":0.19},"max":{"1min":3.141,"5min":21.276,"15min":21.276},"last":0.84799999999999998}]},{"osd":2,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.071,"15min":0.80200000000000005},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":8.1600000000000001,"5min":11.420999999999999,"15min":11.420999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.90100000000000002,"15min":0.83399999999999996},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.1760000000000002,"5min":11.648,"15min":11.648},"last":0.78300000000000003}]},{"osd":3,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.22,"15min":0.91500000000000004},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":3.46,"5min":14.346,"15min":14.346},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.2789999999999999,"15min":1.016},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.248},"max":{"1min":3.4969999999999999,"5min":14.379,"15min":14.379},"last":0.99399999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.3959999999999999,"15min":1.0569999999999999},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.6500000000000004,"5min":10.582000000000001,"15min":10.582000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.49,"5min":1.3009999999999999,"15min":1.04},"min":{"1min":0.32600000000000001,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":4.6630000000000003,"5min":11.173,"15min":11.173},"last":1.0429999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.244,"15min":1.054},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.0129999999999999,"5min":11.577999999999999,"15min":11.577999999999999},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.0760000000000001,"15min":0.98099999999999998},"min":{"1min":0.29899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.3220000000000001,"5min":10.018000000000001,"15min":10.018000000000001},"last":1.0820000000000001}]},{"osd":6,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.1140000000000001,"15min":1.026},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.3610000000000002,"5min":9.2989999999999995,"15min":11.845000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.1699999999999999,"15min":1.109},"min":{"1min":0.38300000000000001,"5min":0.379,"15min":0.33800000000000002},"max":{"1min":2.3809999999999998,"5min":9.3350000000000009,"15min":11.919},"last":1.115}]},{"osd":7,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.498,"15min":1.125},"min":{"1min":0.38200000000000001,"5min":0.36199999999999999,"15min":0.35599999999999998},"max":{"1min":5.077,"5min":9.407,"15min":11.881},"last":1.02},{"interface":"front","average":{"1min":0.874,"5min":1.3089999999999999,"15min":1.137},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.9750000000000001,"5min":10.478,"15min":10.478},"last":1.1439999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575207,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":966360,"kb_used_data":22096,"kb_used_omap":0,"kb_used_meta":944256,"kb_avail":93405480,"statfs":{"total":96636764160,"available":95647211520,"internally_reserved":0,"allocated":22626304,"data_stored":22180994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":966918144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.4159999999999999,"15min":0.98399999999999999},"min":{"1min":0.19700000000000001,"5min":0.18099999999999999,"15min":0.16900000000000001},"max":{"1min":1.7290000000000001,"5min":30.065999999999999,"15min":30.065999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.2330000000000001,"15min":0.98799999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.182},"max":{"1min":3.0030000000000001,"5min":19.757999999999999,"15min":19.757999999999999},"last":0.76300000000000001}]},{"osd":2,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.96099999999999997,"15min":0.78200000000000003},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.9740000000000002,"5min":15.324999999999999,"15min":15.324999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.504,"5min":1.2050000000000001,"15min":0.94299999999999995},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.95699999999999996,"5min":30.283999999999999,"15min":30.283999999999999},"last":1.014}]},{"osd":3,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.5209999999999999,"15min":1.101},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.191},"max":{"1min":6.3570000000000002,"5min":22.966999999999999,"15min":22.966999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":1.417,"15min":1.048},"min":{"1min":0.29499999999999998,"5min":0.24299999999999999,"15min":0.23599999999999999},"max":{"1min":1.1000000000000001,"5min":19.460999999999999,"15min":19.460999999999999},"last":1.0629999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.2090000000000001,"15min":1.121},"min":{"1min":0.33700000000000002,"5min":0.318,"15min":0.26600000000000001},"max":{"1min":3.0720000000000001,"5min":5.5990000000000002,"15min":8.3670000000000009},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.375,"15min":1.1850000000000001},"min":{"1min":0.29499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.1749999999999998,"5min":13.023999999999999,"15min":13.023999999999999},"last":0.98899999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.23,"15min":1.006},"min":{"1min":0.33000000000000002,"5min":0.30399999999999999,"15min":0.29099999999999998},"max":{"1min":2.9710000000000001,"5min":10.683999999999999,"15min":10.683999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.089,"15min":0.96999999999999997},"min":{"1min":0.378,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.5779999999999998,"5min":15.436999999999999,"15min":15.436999999999999},"last":1.173}]},{"osd":6,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.393,"15min":1.125},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.7469999999999999,"5min":8.7260000000000009,"15min":8.7260000000000009},"last":1.085},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.3500000000000001,"15min":1.248},"min":{"1min":0.38200000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.9039999999999999,"5min":15.375,"15min":15.375},"last":1.2250000000000001}]},{"osd":7,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.367,"15min":1.113},"min":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":5.0979999999999999,"5min":10.81,"15min":10.81},"last":1.036},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.3220000000000001,"15min":1.25},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":3.1499999999999999,"5min":11.417,"15min":11.417},"last":1.157}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12673024,"data_stored":12668140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3178496,"data_stored":3177545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11669504,"data_stored":11651527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4195192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":8388642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4279555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6918144,"data_stored":6907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6836224,"data_stored":6822812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-25T00:22:31.343 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-25T00:22:31.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-25T00:22:31.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.496+0000 7fa20677a640 1 Processor -- start 2023-10-25T00:22:31.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.496+0000 7fa20677a640 1 -- start start 2023-10-25T00:22:31.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa20677a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa200152040 0x7fa2000c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa20677a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa2000c8100 0x7fa2000abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa20677a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa20006a4b0 0x7fa2000ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa20677a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa200155680 con 0x7fa200152040 2023-10-25T00:22:31.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa20677a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa200154a70 con 0x7fa2000c8100 2023-10-25T00:22:31.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa20677a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa200154d50 con 0x7fa20006a4b0 2023-10-25T00:22:31.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1fffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa200152040 0x7fa2000c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1fffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fa200152040 0x7fa2000c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54144/0 (socket says 172.21.15.107:54144) 2023-10-25T00:22:31.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1fffff640 1 -- 172.21.15.107:0/2618261291 learned_addr learned my addr 172.21.15.107:0/2618261291 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:31.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa204d17640 1 --2- 172.21.15.107:0/2618261291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa20006a4b0 0x7fa2000ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1ff7fe640 1 --2- 172.21.15.107:0/2618261291 >> v2:172.21.15.110:3300/0 conn(0x7fa2000c8100 0x7fa2000abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1fffff640 1 -- 172.21.15.107:0/2618261291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa20006a4b0 msgr2=0x7fa2000ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:31.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1fffff640 1 --2- 172.21.15.107:0/2618261291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa20006a4b0 0x7fa2000ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1fffff640 1 -- 172.21.15.107:0/2618261291 >> v2:172.21.15.110:3300/0 conn(0x7fa2000c8100 msgr2=0x7fa2000abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:31.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1fffff640 1 --2- 172.21.15.107:0/2618261291 >> v2:172.21.15.110:3300/0 conn(0x7fa2000c8100 0x7fa2000abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1fffff640 1 -- 172.21.15.107:0/2618261291 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa200155030 con 0x7fa200152040 2023-10-25T00:22:31.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1ff7fe640 1 --2- 172.21.15.107:0/2618261291 >> v2:172.21.15.110:3300/0 conn(0x7fa2000c8100 0x7fa2000abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:31.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1fffff640 1 --2- 172.21.15.107:0/2618261291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa200152040 0x7fa2000c4b80 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fa2000c4500 tx=0x7fa1f401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd9758543ff59597 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:31.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1feffd640 1 -- 172.21.15.107:0/2618261291 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f40ab600 con 0x7fa200152040 2023-10-25T00:22:31.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1feffd640 1 -- 172.21.15.107:0/2618261291 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f40a7050 con 0x7fa200152040 2023-10-25T00:22:31.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.500+0000 7fa1feffd640 1 -- 172.21.15.107:0/2618261291 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f40ab980 con 0x7fa200152040 2023-10-25T00:22:31.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 -- 172.21.15.107:0/2618261291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa200152040 msgr2=0x7fa2000c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 --2- 172.21.15.107:0/2618261291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa200152040 0x7fa2000c4b80 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fa2000c4500 tx=0x7fa1f401c470 comp rx=0 tx=0).stop 2023-10-25T00:22:31.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 -- 172.21.15.107:0/2618261291 shutdown_connections 2023-10-25T00:22:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 --2- 172.21.15.107:0/2618261291 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa20006a4b0 0x7fa2000ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 --2- 172.21.15.107:0/2618261291 >> v2:172.21.15.110:3300/0 conn(0x7fa2000c8100 0x7fa2000abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 --2- 172.21.15.107:0/2618261291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa200152040 0x7fa2000c4b80 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 -- 172.21.15.107:0/2618261291 >> 172.21.15.107:0/2618261291 conn(0x7fa2000ca200 msgr2=0x7fa2000ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 -- 172.21.15.107:0/2618261291 shutdown_connections 2023-10-25T00:22:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 -- 172.21.15.107:0/2618261291 wait complete. 2023-10-25T00:22:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 Processor -- start 2023-10-25T00:22:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 -- start start 2023-10-25T00:22:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa20006a4b0 0x7fa200151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2000c8100 0x7fa20014fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa200152040 0x7fa2000c2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa200158be0 con 0x7fa20006a4b0 2023-10-25T00:22:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2000bde30 con 0x7fa2000c8100 2023-10-25T00:22:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2000be640 con 0x7fa200152040 2023-10-25T00:22:31.510 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa20006a4b0 0x7fa200151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa20006a4b0 0x7fa200151830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54156/0 (socket says 172.21.15.107:54156) 2023-10-25T00:22:31.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fffff640 1 -- 172.21.15.107:0/936480768 learned_addr learned my addr 172.21.15.107:0/936480768 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:31.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa204d17640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa200152040 0x7fa2000c2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1ff7fe640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2000c8100 0x7fa20014fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fffff640 1 -- 172.21.15.107:0/936480768 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa200152040 msgr2=0x7fa2000c2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fffff640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa200152040 0x7fa2000c2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fffff640 1 -- 172.21.15.107:0/936480768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2000c8100 msgr2=0x7fa20014fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fffff640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2000c8100 0x7fa20014fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fffff640 1 -- 172.21.15.107:0/936480768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f40094d0 con 0x7fa20006a4b0 2023-10-25T00:22:31.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa204d17640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa200152040 0x7fa2000c2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:22:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fffff640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa20006a4b0 0x7fa200151830 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fa1f401cd10 tx=0x7fa1f4019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fcff9640 1 -- 172.21.15.107:0/936480768 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f40b6640 con 0x7fa20006a4b0 2023-10-25T00:22:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fcff9640 1 -- 172.21.15.107:0/936480768 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f40b67a0 con 0x7fa20006a4b0 2023-10-25T00:22:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 -- 172.21.15.107:0/936480768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa200151e40 con 0x7fa20006a4b0 2023-10-25T00:22:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa1fcff9640 1 -- 172.21.15.107:0/936480768 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f40b6b10 con 0x7fa20006a4b0 2023-10-25T00:22:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.504+0000 7fa20677a640 1 -- 172.21.15.107:0/936480768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2000be360 con 0x7fa20006a4b0 2023-10-25T00:22:31.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.508+0000 7fa20677a640 1 -- 172.21.15.107:0/936480768 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2000746e0 con 0x7fa20006a4b0 2023-10-25T00:22:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.508+0000 7fa1fcff9640 1 -- 172.21.15.107:0/936480768 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa1f40b2040 con 0x7fa20006a4b0 2023-10-25T00:22:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.508+0000 7fa1fcff9640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa1ec065e60 0x7fa1ec068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.508+0000 7fa1fcff9640 1 -- 172.21.15.107:0/936480768 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fa1f411eaf0 con 0x7fa20006a4b0 2023-10-25T00:22:31.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.508+0000 7fa1ff7fe640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa1ec065e60 0x7fa1ec068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.508+0000 7fa1ff7fe640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa1ec065e60 0x7fa1ec068320 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa2000c1170 tx=0x7fa1f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:31.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.512+0000 7fa1fcff9640 1 -- 172.21.15.107:0/936480768 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa1f40eaf30 con 0x7fa20006a4b0 2023-10-25T00:22:31.653 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.652+0000 7fa20677a640 1 -- 172.21.15.107:0/936480768 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa200158760 con 0x7fa1ec065e60 2023-10-25T00:22:31.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.668+0000 7fa1fcff9640 1 -- 172.21.15.107:0/936480768 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+550969 (secure 0 0 0) 0x7fa1f000a750 con 0x7fa1ec065e60 2023-10-25T00:22:31.670 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:31.677 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-25T00:22:31.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 -- 172.21.15.107:0/936480768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa1ec065e60 msgr2=0x7fa1ec068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa1ec065e60 0x7fa1ec068320 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa2000c1170 tx=0x7fa1f0008040 comp rx=0 tx=0).stop 2023-10-25T00:22:31.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 -- 172.21.15.107:0/936480768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa20006a4b0 msgr2=0x7fa200151830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa20006a4b0 0x7fa200151830 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fa1f401cd10 tx=0x7fa1f4019210 comp rx=0 tx=0).stop 2023-10-25T00:22:31.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 -- 172.21.15.107:0/936480768 shutdown_connections 2023-10-25T00:22:31.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa1ec065e60 0x7fa1ec068320 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa200152040 0x7fa2000c2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa2000c8100 0x7fa20014fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 --2- 172.21.15.107:0/936480768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa20006a4b0 0x7fa200151830 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 -- 172.21.15.107:0/936480768 >> 172.21.15.107:0/936480768 conn(0x7fa2000ca200 msgr2=0x7fa2000c8820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:31.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 -- 172.21.15.107:0/936480768 shutdown_connections 2023-10-25T00:22:31.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.676+0000 7fa20677a640 1 -- 172.21.15.107:0/936480768 wait complete. 2023-10-25T00:22:31.692 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":955,"stamp":"2023-10-25T00:22:30.008225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40122242,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3906,"num_read_kb":457920,"num_write":3153446,"num_write_kb":38880045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":583926,"ondisk_log_size":583926,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7826840,"kb_used_data":86040,"kb_used_omap":0,"kb_used_meta":7740736,"kb_avail":747147880,"statfs":{"total":773094113280,"available":765079429120,"internally_reserved":0,"allocated":88104960,"data_stored":84515236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7926513664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":445,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005534"},"pg_stats":[{"pgid":"6.1a","version":"72'3995","reported_seq":4066,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.672949+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-25T00:22:08.672949+0000","last_peered":"2023-10-25T00:22:08.672949+0000","last_clean":"2023-10-25T00:22:08.672949+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-25T00:22:08.672949+0000","last_undegraded":"2023-10-25T00:22:08.672949+0000","last_fullsized":"2023-10-25T00:22:08.672949+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:51:20.137796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21480,"num_write_kb":282788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.773236+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-25T00:22:08.773236+0000","last_peered":"2023-10-25T00:22:08.773236+0000","last_clean":"2023-10-25T00:22:08.773236+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-25T00:22:08.773236+0000","last_undegraded":"2023-10-25T00:22:08.773236+0000","last_fullsized":"2023-10-25T00:22:08.773236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:00:20.101379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"68'864","reported_seq":988,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.254160+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-25T00:21:42.254160+0000","last_peered":"2023-10-25T00:21:42.254160+0000","last_clean":"2023-10-25T00:21:42.254160+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-25T00:21:42.254160+0000","last_undegraded":"2023-10-25T00:21:42.254160+0000","last_fullsized":"2023-10-25T00:21:42.254160+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":18441,"num_write":841,"num_write_kb":4586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'5616","reported_seq":5688,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195291+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-25T00:21:39.195291+0000","last_peered":"2023-10-25T00:21:39.195291+0000","last_clean":"2023-10-25T00:21:39.195291+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-25T00:21:39.195291+0000","last_undegraded":"2023-10-25T00:21:39.195291+0000","last_fullsized":"2023-10-25T00:21:39.195291+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29775,"num_write_kb":348818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"72'3735","reported_seq":3804,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.701818+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-25T00:22:08.701818+0000","last_peered":"2023-10-25T00:22:08.701818+0000","last_clean":"2023-10-25T00:22:08.701818+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-25T00:22:08.701818+0000","last_undegraded":"2023-10-25T00:22:08.701818+0000","last_fullsized":"2023-10-25T00:22:08.701818+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:59.368156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20090,"num_write_kb":264783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"72'2","reported_seq":70,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.258523+0000","last_change":"2023-10-24T23:53:11.444089+0000","last_active":"2023-10-25T00:22:08.258523+0000","last_peered":"2023-10-25T00:22:08.258523+0000","last_clean":"2023-10-25T00:22:08.258523+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-25T00:22:08.258523+0000","last_undegraded":"2023-10-25T00:22:08.258523+0000","last_fullsized":"2023-10-25T00:22:08.258523+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:54:06.875882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"72'27","reported_seq":101,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186964+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-25T00:21:39.186964+0000","last_peered":"2023-10-25T00:21:39.186964+0000","last_clean":"2023-10-25T00:21:39.186964+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-25T00:21:39.186964+0000","last_undegraded":"2023-10-25T00:21:39.186964+0000","last_fullsized":"2023-10-25T00:21:39.186964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'5447","reported_seq":5519,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.184033+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-25T00:21:39.184033+0000","last_peered":"2023-10-25T00:21:39.184033+0000","last_clean":"2023-10-25T00:21:39.184033+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-25T00:21:39.184033+0000","last_undegraded":"2023-10-25T00:21:39.184033+0000","last_fullsized":"2023-10-25T00:21:39.184033+0000","mapping_epoch":21,"log_start":"61'100","ondisk_log_start":"61'100","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28063,"num_write_kb":350200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"72'4747","reported_seq":4818,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.795259+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-25T00:22:08.795259+0000","last_peered":"2023-10-25T00:22:08.795259+0000","last_clean":"2023-10-25T00:22:08.795259+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-25T00:22:08.795259+0000","last_undegraded":"2023-10-25T00:22:08.795259+0000","last_fullsized":"2023-10-25T00:22:08.795259+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:38:56.643025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":25766,"num_write_kb":328702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"72'753","reported_seq":847,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.712526+0000","last_change":"2023-10-24T23:53:11.438681+0000","last_active":"2023-10-25T00:22:08.712526+0000","last_peered":"2023-10-25T00:22:08.712526+0000","last_clean":"2023-10-25T00:22:08.712526+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-25T00:22:08.712526+0000","last_undegraded":"2023-10-25T00:22:08.712526+0000","last_fullsized":"2023-10-25T00:22:08.712526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:50.139697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":77282,"num_write":740,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"72'48","reported_seq":163,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.487987+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-25T00:21:42.487987+0000","last_peered":"2023-10-25T00:21:42.487987+0000","last_clean":"2023-10-25T00:21:42.487987+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-25T00:21:42.487987+0000","last_undegraded":"2023-10-25T00:21:42.487987+0000","last_fullsized":"2023-10-25T00:21:42.487987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'4792","reported_seq":4864,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205939+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-25T00:21:19.205939+0000","last_peered":"2023-10-25T00:21:19.205939+0000","last_clean":"2023-10-25T00:21:19.205939+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-25T00:21:19.205939+0000","last_undegraded":"2023-10-25T00:21:19.205939+0000","last_fullsized":"2023-10-25T00:21:19.205939+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24997,"num_write_kb":313272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"72'4739","reported_seq":4811,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.785162+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-25T00:22:08.785162+0000","last_peered":"2023-10-25T00:22:08.785162+0000","last_clean":"2023-10-25T00:22:08.785162+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-25T00:22:08.785162+0000","last_undegraded":"2023-10-25T00:22:08.785162+0000","last_fullsized":"2023-10-25T00:22:08.785162+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:22.974333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":22,"num_write":25639,"num_write_kb":332364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"72'357","reported_seq":426,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:15.185838+0000","last_change":"2023-10-24T23:53:11.454537+0000","last_active":"2023-10-25T00:22:15.185838+0000","last_peered":"2023-10-25T00:22:15.185838+0000","last_clean":"2023-10-25T00:22:15.185838+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-25T00:22:15.185838+0000","last_undegraded":"2023-10-25T00:22:15.185838+0000","last_fullsized":"2023-10-25T00:22:15.185838+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:14:06.792651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2627766,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":227,"num_write":329,"num_write_kb":2759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"68'5","reported_seq":122,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.254866+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-25T00:21:42.254866+0000","last_peered":"2023-10-25T00:21:42.254866+0000","last_clean":"2023-10-25T00:21:42.254866+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-25T00:21:42.254866+0000","last_undegraded":"2023-10-25T00:21:42.254866+0000","last_fullsized":"2023-10-25T00:21:42.254866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'3731","reported_seq":3801,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205159+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-25T00:21:19.205159+0000","last_peered":"2023-10-25T00:21:19.205159+0000","last_clean":"2023-10-25T00:21:19.205159+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-25T00:21:19.205159+0000","last_undegraded":"2023-10-25T00:21:19.205159+0000","last_fullsized":"2023-10-25T00:21:19.205159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19492,"num_write_kb":250091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"72'3799","reported_seq":3869,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.756165+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-25T00:22:08.756165+0000","last_peered":"2023-10-25T00:22:08.756165+0000","last_clean":"2023-10-25T00:22:08.756165+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-25T00:22:08.756165+0000","last_undegraded":"2023-10-25T00:22:08.756165+0000","last_fullsized":"2023-10-25T00:22:08.756165+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:35:39.879563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21581,"num_write_kb":262877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"72'45","reported_seq":156,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.747187+0000","last_change":"2023-10-24T23:53:11.444138+0000","last_active":"2023-10-25T00:22:08.747187+0000","last_peered":"2023-10-25T00:22:08.747187+0000","last_clean":"2023-10-25T00:22:08.747187+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-25T00:22:08.747187+0000","last_undegraded":"2023-10-25T00:22:08.747187+0000","last_fullsized":"2023-10-25T00:22:08.747187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:27:38.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"68'714","reported_seq":813,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195222+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-25T00:21:39.195222+0000","last_peered":"2023-10-25T00:21:39.195222+0000","last_clean":"2023-10-25T00:21:39.195222+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-25T00:21:39.195222+0000","last_undegraded":"2023-10-25T00:21:39.195222+0000","last_fullsized":"2023-10-25T00:21:39.195222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":86332,"num_write":703,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'4008","reported_seq":4082,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.208728+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-25T00:21:19.208728+0000","last_peered":"2023-10-25T00:21:19.208728+0000","last_clean":"2023-10-25T00:21:19.208728+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-25T00:21:19.208728+0000","last_undegraded":"2023-10-25T00:21:19.208728+0000","last_fullsized":"2023-10-25T00:21:19.208728+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:48.110295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20791,"num_write_kb":267770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"72'5469","reported_seq":5550,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793934+0000","last_change":"2023-10-24T23:53:11.832809+0000","last_active":"2023-10-25T00:22:08.793934+0000","last_peered":"2023-10-25T00:22:08.793934+0000","last_clean":"2023-10-25T00:22:08.793934+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-25T00:22:08.793934+0000","last_undegraded":"2023-10-25T00:22:08.793934+0000","last_fullsized":"2023-10-25T00:22:08.793934+0000","mapping_epoch":25,"log_start":"64'200","ondisk_log_start":"64'200","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:01.151315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":40,"num_write":30646,"num_write_kb":401481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"72'5","reported_seq":116,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.200649+0000","last_change":"2023-10-24T23:53:11.442035+0000","last_active":"2023-10-25T00:21:39.200649+0000","last_peered":"2023-10-25T00:21:39.200649+0000","last_clean":"2023-10-25T00:21:39.200649+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-25T00:21:39.200649+0000","last_undegraded":"2023-10-25T00:21:39.200649+0000","last_fullsized":"2023-10-25T00:21:39.200649+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:13.717844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"72'532","reported_seq":611,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.992831+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-25T00:22:07.992831+0000","last_peered":"2023-10-25T00:22:07.992831+0000","last_clean":"2023-10-25T00:22:07.992831+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-25T00:22:07.992831+0000","last_undegraded":"2023-10-25T00:22:07.992831+0000","last_fullsized":"2023-10-25T00:22:07.992831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3177545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7524,"num_write":504,"num_write_kb":3406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'6069","reported_seq":6142,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183010+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-25T00:21:19.183010+0000","last_peered":"2023-10-25T00:21:19.183010+0000","last_clean":"2023-10-25T00:21:19.183010+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-25T00:21:19.183010+0000","last_undegraded":"2023-10-25T00:21:19.183010+0000","last_fullsized":"2023-10-25T00:21:19.183010+0000","mapping_epoch":21,"log_start":"62'800","ondisk_log_start":"62'800","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":33187,"num_write_kb":387130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'3606","reported_seq":3680,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.451770+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-25T00:22:08.451770+0000","last_peered":"2023-10-25T00:22:08.451770+0000","last_clean":"2023-10-25T00:22:08.451770+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-25T00:22:08.451770+0000","last_undegraded":"2023-10-25T00:22:08.451770+0000","last_fullsized":"2023-10-25T00:22:08.451770+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:26.012711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19493,"num_write_kb":254342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'1061","reported_seq":1182,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.761652+0000","last_change":"2023-10-24T23:53:11.448032+0000","last_active":"2023-10-25T00:22:08.761652+0000","last_peered":"2023-10-25T00:22:08.761652+0000","last_clean":"2023-10-25T00:22:08.761652+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-25T00:22:08.761652+0000","last_undegraded":"2023-10-25T00:22:08.761652+0000","last_fullsized":"2023-10-25T00:22:08.761652+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1061,"log_dups_size":0,"ondisk_log_size":1061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:50:32.490810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":9402,"num_write":1035,"num_write_kb":4717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'3","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.185352+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-25T00:21:39.185352+0000","last_peered":"2023-10-25T00:21:39.185352+0000","last_clean":"2023-10-25T00:21:39.185352+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-25T00:21:39.185352+0000","last_undegraded":"2023-10-25T00:21:39.185352+0000","last_fullsized":"2023-10-25T00:21:39.185352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'5139","reported_seq":5212,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.216678+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-25T00:21:19.216678+0000","last_peered":"2023-10-25T00:21:19.216678+0000","last_clean":"2023-10-25T00:21:19.216678+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-25T00:21:19.216678+0000","last_undegraded":"2023-10-25T00:21:19.216678+0000","last_fullsized":"2023-10-25T00:21:19.216678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27772,"num_write_kb":328469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"72'4843","reported_seq":4913,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.750475+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-25T00:22:08.750475+0000","last_peered":"2023-10-25T00:22:08.750475+0000","last_clean":"2023-10-25T00:22:08.750475+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-25T00:22:08.750475+0000","last_undegraded":"2023-10-25T00:22:08.750475+0000","last_fullsized":"2023-10-25T00:22:08.750475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:10.820967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":27290,"num_write_kb":330906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"72'27","reported_seq":97,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.749463+0000","last_change":"2023-10-24T23:53:11.446925+0000","last_active":"2023-10-25T00:22:08.749463+0000","last_peered":"2023-10-25T00:22:08.749463+0000","last_clean":"2023-10-25T00:22:08.749463+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-25T00:22:08.749463+0000","last_undegraded":"2023-10-25T00:22:08.749463+0000","last_fullsized":"2023-10-25T00:22:08.749463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:59:52.485469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'2","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182801+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-25T00:21:39.182801+0000","last_peered":"2023-10-25T00:21:39.182801+0000","last_clean":"2023-10-25T00:21:39.182801+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-25T00:21:39.182801+0000","last_undegraded":"2023-10-25T00:21:39.182801+0000","last_fullsized":"2023-10-25T00:21:39.182801+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'5814","reported_seq":5888,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189609+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-25T00:21:39.189609+0000","last_peered":"2023-10-25T00:21:39.189609+0000","last_clean":"2023-10-25T00:21:39.189609+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-25T00:21:39.189609+0000","last_undegraded":"2023-10-25T00:21:39.189609+0000","last_fullsized":"2023-10-25T00:21:39.189609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31996,"num_write_kb":382397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"72'3827","reported_seq":3898,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.365063+0000","last_change":"2023-10-24T23:53:11.849056+0000","last_active":"2023-10-25T00:22:08.365063+0000","last_peered":"2023-10-25T00:22:08.365063+0000","last_clean":"2023-10-25T00:22:08.365063+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-25T00:22:08.365063+0000","last_undegraded":"2023-10-25T00:22:08.365063+0000","last_fullsized":"2023-10-25T00:22:08.365063+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:08.932942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":21199,"num_write_kb":277730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"68'4820","reported_seq":4893,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.181692+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-25T00:21:19.181692+0000","last_peered":"2023-10-25T00:21:19.181692+0000","last_clean":"2023-10-25T00:21:19.181692+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-25T00:21:19.181692+0000","last_undegraded":"2023-10-25T00:21:19.181692+0000","last_fullsized":"2023-10-25T00:21:19.181692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25163,"num_write_kb":285293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"72'5","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:06.452266+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-25T00:22:06.452266+0000","last_peered":"2023-10-25T00:22:06.452266+0000","last_clean":"2023-10-25T00:22:06.452266+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-25T00:22:06.452266+0000","last_undegraded":"2023-10-25T00:22:06.452266+0000","last_fullsized":"2023-10-25T00:22:06.452266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"72'5","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.744190+0000","last_change":"2023-10-24T23:53:11.447167+0000","last_active":"2023-10-25T00:22:08.744190+0000","last_peered":"2023-10-25T00:22:08.744190+0000","last_clean":"2023-10-25T00:22:08.744190+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-25T00:22:08.744190+0000","last_undegraded":"2023-10-25T00:22:08.744190+0000","last_fullsized":"2023-10-25T00:22:08.744190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:06:25.006728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"72'4957","reported_seq":5031,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.762994+0000","last_change":"2023-10-24T23:53:11.813704+0000","last_active":"2023-10-25T00:22:08.762994+0000","last_peered":"2023-10-25T00:22:08.762994+0000","last_clean":"2023-10-25T00:22:08.762994+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-25T00:22:08.762994+0000","last_undegraded":"2023-10-25T00:22:08.762994+0000","last_fullsized":"2023-10-25T00:22:08.762994+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:39:09.996039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28093,"num_write_kb":358545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'4782","reported_seq":4860,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.185276+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-25T00:21:19.185276+0000","last_peered":"2023-10-25T00:21:19.185276+0000","last_clean":"2023-10-25T00:21:19.185276+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-25T00:21:19.185276+0000","last_undegraded":"2023-10-25T00:21:19.185276+0000","last_fullsized":"2023-10-25T00:21:19.185276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26432,"num_write_kb":298639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"68'3","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193250+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-25T00:21:39.193250+0000","last_peered":"2023-10-25T00:21:39.193250+0000","last_clean":"2023-10-25T00:21:39.193250+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-25T00:21:39.193250+0000","last_undegraded":"2023-10-25T00:21:39.193250+0000","last_fullsized":"2023-10-25T00:21:39.193250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"72'14","reported_seq":197,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.746812+0000","last_change":"2023-10-24T23:53:11.450974+0000","last_active":"2023-10-25T00:22:08.746812+0000","last_peered":"2023-10-25T00:22:08.746812+0000","last_clean":"2023-10-25T00:22:08.746812+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-25T00:22:08.746812+0000","last_undegraded":"2023-10-25T00:22:08.746812+0000","last_fullsized":"2023-10-25T00:22:08.746812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:51:05.443664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":345,"num_read_kb":345,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"72'2954","reported_seq":3023,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.717448+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-25T00:22:08.717448+0000","last_peered":"2023-10-25T00:22:08.717448+0000","last_clean":"2023-10-25T00:22:08.717448+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-25T00:22:08.717448+0000","last_undegraded":"2023-10-25T00:22:08.717448+0000","last_fullsized":"2023-10-25T00:22:08.717448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:37:30.278659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15961,"num_write_kb":219669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"72'36","reported_seq":152,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.212150+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-25T00:22:10.212150+0000","last_peered":"2023-10-25T00:22:10.212150+0000","last_clean":"2023-10-25T00:22:10.212150+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-25T00:22:10.212150+0000","last_undegraded":"2023-10-25T00:22:10.212150+0000","last_fullsized":"2023-10-25T00:22:10.212150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'4690","reported_seq":4764,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178387+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-25T00:21:19.178387+0000","last_peered":"2023-10-25T00:21:19.178387+0000","last_clean":"2023-10-25T00:21:19.178387+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-25T00:21:19.178387+0000","last_undegraded":"2023-10-25T00:21:19.178387+0000","last_fullsized":"2023-10-25T00:21:19.178387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25172,"num_write_kb":305786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"72'8","reported_seq":120,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.777365+0000","last_change":"2023-10-24T23:53:11.429694+0000","last_active":"2023-10-25T00:22:08.777365+0000","last_peered":"2023-10-25T00:22:08.777365+0000","last_clean":"2023-10-25T00:22:08.777365+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-25T00:22:08.777365+0000","last_undegraded":"2023-10-25T00:22:08.777365+0000","last_fullsized":"2023-10-25T00:22:08.777365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:15.964994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"72'4200","reported_seq":4277,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.728522+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-25T00:22:08.728522+0000","last_peered":"2023-10-25T00:22:08.728522+0000","last_clean":"2023-10-25T00:22:08.728522+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-25T00:22:08.728522+0000","last_undegraded":"2023-10-25T00:22:08.728522+0000","last_fullsized":"2023-10-25T00:22:08.728522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:38:18.678087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":24169,"num_write_kb":293796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"72'389","reported_seq":572,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.983790+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-25T00:22:07.983790+0000","last_peered":"2023-10-25T00:22:07.983790+0000","last_clean":"2023-10-25T00:22:07.983790+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-25T00:22:07.983790+0000","last_undegraded":"2023-10-25T00:22:07.983790+0000","last_fullsized":"2023-10-25T00:22:07.983790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":201,"num_read_kb":90291,"num_write":386,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'5730","reported_seq":5803,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205335+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-25T00:21:19.205335+0000","last_peered":"2023-10-25T00:21:19.205335+0000","last_clean":"2023-10-25T00:21:19.205335+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-25T00:21:19.205335+0000","last_undegraded":"2023-10-25T00:21:19.205335+0000","last_fullsized":"2023-10-25T00:21:19.205335+0000","mapping_epoch":21,"log_start":"61'500","ondisk_log_start":"61'500","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":31652,"num_write_kb":356481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760659+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-25T00:22:08.760659+0000","last_peered":"2023-10-25T00:22:08.760659+0000","last_clean":"2023-10-25T00:22:08.760659+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-25T00:22:08.760659+0000","last_undegraded":"2023-10-25T00:22:08.760659+0000","last_fullsized":"2023-10-25T00:22:08.760659+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:30.163913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"72'4168","reported_seq":4240,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.466973+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-25T00:22:08.466973+0000","last_peered":"2023-10-25T00:22:08.466973+0000","last_clean":"2023-10-25T00:22:08.466973+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-25T00:22:08.466973+0000","last_undegraded":"2023-10-25T00:22:08.466973+0000","last_fullsized":"2023-10-25T00:22:08.466973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:46.000330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23458,"num_write_kb":294850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"68'4339","reported_seq":4413,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197911+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-25T00:21:39.197911+0000","last_peered":"2023-10-25T00:21:39.197911+0000","last_clean":"2023-10-25T00:21:39.197911+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-25T00:21:39.197911+0000","last_undegraded":"2023-10-25T00:21:39.197911+0000","last_fullsized":"2023-10-25T00:21:39.197911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23267,"num_write_kb":265268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.222903+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-25T00:21:19.222903+0000","last_peered":"2023-10-25T00:21:19.222903+0000","last_clean":"2023-10-25T00:21:19.222903+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-25T00:21:19.222903+0000","last_undegraded":"2023-10-25T00:21:19.222903+0000","last_fullsized":"2023-10-25T00:21:19.222903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"72'230","reported_seq":326,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.211353+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-25T00:22:10.211353+0000","last_peered":"2023-10-25T00:22:10.211353+0000","last_clean":"2023-10-25T00:22:10.211353+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-25T00:22:10.211353+0000","last_undegraded":"2023-10-25T00:22:10.211353+0000","last_fullsized":"2023-10-25T00:22:10.211353+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":61,"num_write":589,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"72'11","reported_seq":81,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.756187+0000","last_change":"2023-10-24T23:53:11.433306+0000","last_active":"2023-10-25T00:22:08.756187+0000","last_peered":"2023-10-25T00:22:08.756187+0000","last_clean":"2023-10-25T00:22:08.756187+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-25T00:22:08.756187+0000","last_undegraded":"2023-10-25T00:22:08.756187+0000","last_fullsized":"2023-10-25T00:22:08.756187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:44:38.728345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"72'3993","reported_seq":4067,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793391+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-25T00:22:08.793391+0000","last_peered":"2023-10-25T00:22:08.793391+0000","last_clean":"2023-10-25T00:22:08.793391+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-25T00:22:08.793391+0000","last_undegraded":"2023-10-25T00:22:08.793391+0000","last_fullsized":"2023-10-25T00:22:08.793391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:30:24.805861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21723,"num_write_kb":283850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'4270","reported_seq":4344,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.181969+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-25T00:21:19.181969+0000","last_peered":"2023-10-25T00:21:19.181969+0000","last_clean":"2023-10-25T00:21:19.181969+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-25T00:21:19.181969+0000","last_undegraded":"2023-10-25T00:21:19.181969+0000","last_fullsized":"2023-10-25T00:21:19.181969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23049,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180537+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-25T00:21:19.180537+0000","last_peered":"2023-10-25T00:21:19.180537+0000","last_clean":"2023-10-25T00:21:19.180537+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-25T00:21:19.180537+0000","last_undegraded":"2023-10-25T00:21:19.180537+0000","last_fullsized":"2023-10-25T00:21:19.180537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'9","reported_seq":128,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.260782+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-25T00:21:42.260782+0000","last_peered":"2023-10-25T00:21:42.260782+0000","last_clean":"2023-10-25T00:21:42.260782+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-25T00:21:42.260782+0000","last_undegraded":"2023-10-25T00:21:42.260782+0000","last_fullsized":"2023-10-25T00:21:42.260782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"72'64","reported_seq":214,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.752186+0000","last_change":"2023-10-24T23:53:11.441983+0000","last_active":"2023-10-25T00:22:08.752186+0000","last_peered":"2023-10-25T00:22:08.752186+0000","last_clean":"2023-10-25T00:22:08.752186+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-25T00:22:08.752186+0000","last_undegraded":"2023-10-25T00:22:08.752186+0000","last_fullsized":"2023-10-25T00:22:08.752186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:30.643019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":123,"num_write":59,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"72'3769","reported_seq":3843,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.609361+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-25T00:22:08.609361+0000","last_peered":"2023-10-25T00:22:08.609361+0000","last_clean":"2023-10-25T00:22:08.609361+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-25T00:22:08.609361+0000","last_undegraded":"2023-10-25T00:22:08.609361+0000","last_fullsized":"2023-10-25T00:22:08.609361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:50:53.467058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20807,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'5752","reported_seq":5826,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194706+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-25T00:21:39.194706+0000","last_peered":"2023-10-25T00:21:39.194706+0000","last_clean":"2023-10-25T00:21:39.194706+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-25T00:21:39.194706+0000","last_undegraded":"2023-10-25T00:21:39.194706+0000","last_fullsized":"2023-10-25T00:21:39.194706+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31231,"num_write_kb":354718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199135+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-25T00:21:39.199135+0000","last_peered":"2023-10-25T00:21:39.199135+0000","last_clean":"2023-10-25T00:21:39.199135+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-25T00:21:39.199135+0000","last_undegraded":"2023-10-25T00:21:39.199135+0000","last_fullsized":"2023-10-25T00:21:39.199135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'2","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189313+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-25T00:21:39.189313+0000","last_peered":"2023-10-25T00:21:39.189313+0000","last_clean":"2023-10-25T00:21:39.189313+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-25T00:21:39.189313+0000","last_undegraded":"2023-10-25T00:21:39.189313+0000","last_fullsized":"2023-10-25T00:21:39.189313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"72'4","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.259221+0000","last_change":"2023-10-24T23:53:11.426287+0000","last_active":"2023-10-25T00:22:08.259221+0000","last_peered":"2023-10-25T00:22:08.259221+0000","last_clean":"2023-10-25T00:22:08.259221+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-25T00:22:08.259221+0000","last_undegraded":"2023-10-25T00:22:08.259221+0000","last_fullsized":"2023-10-25T00:22:08.259221+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:02:39.399866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"72'3939","reported_seq":4008,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.780603+0000","last_change":"2023-10-24T23:53:11.811139+0000","last_active":"2023-10-25T00:22:08.780603+0000","last_peered":"2023-10-25T00:22:08.780603+0000","last_clean":"2023-10-25T00:22:08.780603+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-25T00:22:08.780603+0000","last_undegraded":"2023-10-25T00:22:08.780603+0000","last_fullsized":"2023-10-25T00:22:08.780603+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:05.507633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22393,"num_write_kb":262004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'4305","reported_seq":4379,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186001+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-25T00:21:39.186001+0000","last_peered":"2023-10-25T00:21:39.186001+0000","last_clean":"2023-10-25T00:21:39.186001+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-25T00:21:39.186001+0000","last_undegraded":"2023-10-25T00:21:39.186001+0000","last_fullsized":"2023-10-25T00:21:39.186001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22696,"num_write_kb":287574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187237+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-25T00:21:39.187237+0000","last_peered":"2023-10-25T00:21:39.187237+0000","last_clean":"2023-10-25T00:21:39.187237+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-25T00:21:39.187237+0000","last_undegraded":"2023-10-25T00:21:39.187237+0000","last_fullsized":"2023-10-25T00:21:39.187237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.188607+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-25T00:21:39.188607+0000","last_peered":"2023-10-25T00:21:39.188607+0000","last_clean":"2023-10-25T00:21:39.188607+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-25T00:21:39.188607+0000","last_undegraded":"2023-10-25T00:21:39.188607+0000","last_fullsized":"2023-10-25T00:21:39.188607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"72'3","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.892690+0000","last_change":"2023-10-24T23:53:11.426199+0000","last_active":"2023-10-25T00:22:07.892690+0000","last_peered":"2023-10-25T00:22:07.892690+0000","last_clean":"2023-10-25T00:22:07.892690+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-25T00:22:07.892690+0000","last_undegraded":"2023-10-25T00:22:07.892690+0000","last_fullsized":"2023-10-25T00:22:07.892690+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:57:51.018816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"72'5003","reported_seq":5078,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.705186+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-25T00:22:08.705186+0000","last_peered":"2023-10-25T00:22:08.705186+0000","last_clean":"2023-10-25T00:22:08.705186+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-25T00:22:08.705186+0000","last_undegraded":"2023-10-25T00:22:08.705186+0000","last_fullsized":"2023-10-25T00:22:08.705186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:20:16.596993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":13,"num_write":27120,"num_write_kb":350355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'5635","reported_seq":5708,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186387+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-25T00:21:39.186387+0000","last_peered":"2023-10-25T00:21:39.186387+0000","last_clean":"2023-10-25T00:21:39.186387+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-25T00:21:39.186387+0000","last_undegraded":"2023-10-25T00:21:39.186387+0000","last_fullsized":"2023-10-25T00:21:39.186387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30556,"num_write_kb":356140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"68'3","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.196592+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-25T00:21:39.196592+0000","last_peered":"2023-10-25T00:21:39.196592+0000","last_clean":"2023-10-25T00:21:39.196592+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-25T00:21:39.196592+0000","last_undegraded":"2023-10-25T00:21:39.196592+0000","last_fullsized":"2023-10-25T00:21:39.196592+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183369+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-25T00:21:39.183369+0000","last_peered":"2023-10-25T00:21:39.183369+0000","last_clean":"2023-10-25T00:21:39.183369+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-25T00:21:39.183369+0000","last_undegraded":"2023-10-25T00:21:39.183369+0000","last_fullsized":"2023-10-25T00:21:39.183369+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"72'2","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193585+0000","last_change":"2023-10-24T23:53:11.432666+0000","last_active":"2023-10-25T00:21:39.193585+0000","last_peered":"2023-10-25T00:21:39.193585+0000","last_clean":"2023-10-25T00:21:39.193585+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-25T00:21:39.193585+0000","last_undegraded":"2023-10-25T00:21:39.193585+0000","last_fullsized":"2023-10-25T00:21:39.193585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:17.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"72'3753","reported_seq":3826,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:09.504989+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-25T00:22:09.504989+0000","last_peered":"2023-10-25T00:22:09.504989+0000","last_clean":"2023-10-25T00:22:09.504989+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-25T00:22:09.504989+0000","last_undegraded":"2023-10-25T00:22:09.504989+0000","last_fullsized":"2023-10-25T00:22:09.504989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:42.131555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":21045,"num_write_kb":276986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'4101","reported_seq":4173,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.179395+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-25T00:21:19.179395+0000","last_peered":"2023-10-25T00:21:19.179395+0000","last_clean":"2023-10-25T00:21:19.179395+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-25T00:21:19.179395+0000","last_undegraded":"2023-10-25T00:21:19.179395+0000","last_fullsized":"2023-10-25T00:21:19.179395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22383,"num_write_kb":270023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"72'12","reported_seq":86,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.184571+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-25T00:21:39.184571+0000","last_peered":"2023-10-25T00:21:39.184571+0000","last_clean":"2023-10-25T00:21:39.184571+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-25T00:21:39.184571+0000","last_undegraded":"2023-10-25T00:21:39.184571+0000","last_fullsized":"2023-10-25T00:21:39.184571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193953+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-25T00:21:39.193953+0000","last_peered":"2023-10-25T00:21:39.193953+0000","last_clean":"2023-10-25T00:21:39.193953+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-25T00:21:39.193953+0000","last_undegraded":"2023-10-25T00:21:39.193953+0000","last_fullsized":"2023-10-25T00:21:39.193953+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:22:38.665718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"72'248","reported_seq":339,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.188700+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-25T00:22:10.188700+0000","last_peered":"2023-10-25T00:22:10.188700+0000","last_clean":"2023-10-25T00:22:10.188700+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-25T00:22:10.188700+0000","last_undegraded":"2023-10-25T00:22:10.188700+0000","last_fullsized":"2023-10-25T00:22:10.188700+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:55:47.548432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":62,"num_write":607,"num_write_kb":223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"72'5236","reported_seq":5311,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.753672+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-25T00:22:08.753672+0000","last_peered":"2023-10-25T00:22:08.753672+0000","last_clean":"2023-10-25T00:22:08.753672+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-25T00:22:08.753672+0000","last_undegraded":"2023-10-25T00:22:08.753672+0000","last_fullsized":"2023-10-25T00:22:08.753672+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:11:14.431984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":29554,"num_write_kb":368820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'5444","reported_seq":5520,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.216863+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-25T00:21:19.216863+0000","last_peered":"2023-10-25T00:21:19.216863+0000","last_clean":"2023-10-25T00:21:19.216863+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-25T00:21:19.216863+0000","last_undegraded":"2023-10-25T00:21:19.216863+0000","last_fullsized":"2023-10-25T00:21:19.216863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28342,"num_write_kb":342931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"72'66","reported_seq":223,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.219689+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-25T00:22:10.219689+0000","last_peered":"2023-10-25T00:22:10.219689+0000","last_clean":"2023-10-25T00:22:10.219689+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-25T00:22:10.219689+0000","last_undegraded":"2023-10-25T00:22:10.219689+0000","last_fullsized":"2023-10-25T00:22:10.219689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":125,"num_write":61,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193375+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-25T00:21:19.193375+0000","last_peered":"2023-10-25T00:21:19.193375+0000","last_clean":"2023-10-25T00:21:19.193375+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-25T00:21:19.193375+0000","last_undegraded":"2023-10-25T00:21:19.193375+0000","last_fullsized":"2023-10-25T00:21:19.193375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"72'9","reported_seq":123,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760947+0000","last_change":"2023-10-24T23:53:11.436603+0000","last_active":"2023-10-25T00:22:08.760947+0000","last_peered":"2023-10-25T00:22:08.760947+0000","last_clean":"2023-10-25T00:22:08.760947+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-25T00:22:08.760947+0000","last_undegraded":"2023-10-25T00:22:08.760947+0000","last_fullsized":"2023-10-25T00:22:08.760947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:09:05.948928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":135,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"72'3666","reported_seq":3735,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.792399+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-25T00:22:08.792399+0000","last_peered":"2023-10-25T00:22:08.792399+0000","last_clean":"2023-10-25T00:22:08.792399+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-25T00:22:08.792399+0000","last_undegraded":"2023-10-25T00:22:08.792399+0000","last_fullsized":"2023-10-25T00:22:08.792399+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:01:59.639080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20519,"num_write_kb":254391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"68'4182","reported_seq":4256,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187475+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-25T00:21:39.187475+0000","last_peered":"2023-10-25T00:21:39.187475+0000","last_clean":"2023-10-25T00:21:39.187475+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-25T00:21:39.187475+0000","last_undegraded":"2023-10-25T00:21:39.187475+0000","last_fullsized":"2023-10-25T00:21:39.187475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22839,"num_write_kb":259586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'4","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182378+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-25T00:21:39.182378+0000","last_peered":"2023-10-25T00:21:39.182378+0000","last_clean":"2023-10-25T00:21:39.182378+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-25T00:21:39.182378+0000","last_undegraded":"2023-10-25T00:21:39.182378+0000","last_fullsized":"2023-10-25T00:21:39.182378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183752+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-25T00:21:39.183752+0000","last_peered":"2023-10-25T00:21:39.183752+0000","last_clean":"2023-10-25T00:21:39.183752+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-25T00:21:39.183752+0000","last_undegraded":"2023-10-25T00:21:39.183752+0000","last_fullsized":"2023-10-25T00:21:39.183752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"43'161","reported_seq":283,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193737+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-25T00:21:19.193737+0000","last_peered":"2023-10-25T00:21:19.193737+0000","last_clean":"2023-10-25T00:21:19.193737+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-25T00:21:19.193737+0000","last_undegraded":"2023-10-25T00:21:19.193737+0000","last_fullsized":"2023-10-25T00:21:19.193737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"72'2","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784850+0000","last_change":"2023-10-24T23:53:11.451073+0000","last_active":"2023-10-25T00:22:08.784850+0000","last_peered":"2023-10-25T00:22:08.784850+0000","last_clean":"2023-10-25T00:22:08.784850+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-25T00:22:08.784850+0000","last_undegraded":"2023-10-25T00:22:08.784850+0000","last_fullsized":"2023-10-25T00:22:08.784850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:32.324710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"72'4105","reported_seq":4177,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793407+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-25T00:22:08.793407+0000","last_peered":"2023-10-25T00:22:08.793407+0000","last_clean":"2023-10-25T00:22:08.793407+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-25T00:22:08.793407+0000","last_undegraded":"2023-10-25T00:22:08.793407+0000","last_fullsized":"2023-10-25T00:22:08.793407+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:04.125835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":19,"num_write":23147,"num_write_kb":297959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'5705","reported_seq":5779,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189167+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-25T00:21:39.189167+0000","last_peered":"2023-10-25T00:21:39.189167+0000","last_clean":"2023-10-25T00:21:39.189167+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-25T00:21:39.189167+0000","last_undegraded":"2023-10-25T00:21:39.189167+0000","last_fullsized":"2023-10-25T00:21:39.189167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":29845,"num_write_kb":357286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"68'5","reported_seq":78,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183009+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-25T00:21:19.183009+0000","last_peered":"2023-10-25T00:21:19.183009+0000","last_clean":"2023-10-25T00:21:19.183009+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-25T00:21:19.183009+0000","last_undegraded":"2023-10-25T00:21:19.183009+0000","last_fullsized":"2023-10-25T00:21:19.183009+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"72'7","reported_seq":121,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.761062+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-25T00:22:08.761062+0000","last_peered":"2023-10-25T00:22:08.761062+0000","last_clean":"2023-10-25T00:22:08.761062+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-25T00:22:08.761062+0000","last_undegraded":"2023-10-25T00:22:08.761062+0000","last_fullsized":"2023-10-25T00:22:08.761062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:06:53.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"72'4234","reported_seq":4305,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.780516+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-25T00:22:08.780516+0000","last_peered":"2023-10-25T00:22:08.780516+0000","last_clean":"2023-10-25T00:22:08.780516+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-25T00:22:08.780516+0000","last_undegraded":"2023-10-25T00:22:08.780516+0000","last_fullsized":"2023-10-25T00:22:08.780516+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:41.905772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":15,"num_write":23173,"num_write_kb":302022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"68'4052","reported_seq":4125,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197631+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-25T00:21:39.197631+0000","last_peered":"2023-10-25T00:21:39.197631+0000","last_clean":"2023-10-25T00:21:39.197631+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-25T00:21:39.197631+0000","last_undegraded":"2023-10-25T00:21:39.197631+0000","last_fullsized":"2023-10-25T00:21:39.197631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21486,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"68'2","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.204255+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-25T00:21:19.204255+0000","last_peered":"2023-10-25T00:21:19.204255+0000","last_clean":"2023-10-25T00:21:19.204255+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-25T00:21:19.204255+0000","last_undegraded":"2023-10-25T00:21:19.204255+0000","last_fullsized":"2023-10-25T00:21:19.204255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"72'4","reported_seq":94,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.751351+0000","last_change":"2023-10-24T23:53:11.458463+0000","last_active":"2023-10-25T00:22:08.751351+0000","last_peered":"2023-10-25T00:22:08.751351+0000","last_clean":"2023-10-25T00:22:08.751351+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-25T00:22:08.751351+0000","last_undegraded":"2023-10-25T00:22:08.751351+0000","last_fullsized":"2023-10-25T00:22:08.751351+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:11:45.040156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"72'4690","reported_seq":4761,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.556236+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-25T00:22:08.556236+0000","last_peered":"2023-10-25T00:22:08.556236+0000","last_clean":"2023-10-25T00:22:08.556236+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-25T00:22:08.556236+0000","last_undegraded":"2023-10-25T00:22:08.556236+0000","last_fullsized":"2023-10-25T00:22:08.556236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:51.707178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24994,"num_write_kb":341798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'4780","reported_seq":4855,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194222+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-25T00:21:39.194222+0000","last_peered":"2023-10-25T00:21:39.194222+0000","last_clean":"2023-10-25T00:21:39.194222+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-25T00:21:39.194222+0000","last_undegraded":"2023-10-25T00:21:39.194222+0000","last_fullsized":"2023-10-25T00:21:39.194222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25083,"num_write_kb":293163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'3","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:20.638748+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-25T00:21:20.638748+0000","last_peered":"2023-10-25T00:21:20.638748+0000","last_clean":"2023-10-25T00:21:20.638748+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-25T00:21:20.638748+0000","last_undegraded":"2023-10-25T00:21:20.638748+0000","last_fullsized":"2023-10-25T00:21:20.638748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"72'5","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.745978+0000","last_change":"2023-10-24T23:53:11.448675+0000","last_active":"2023-10-25T00:22:08.745978+0000","last_peered":"2023-10-25T00:22:08.745978+0000","last_clean":"2023-10-25T00:22:08.745978+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-25T00:22:08.745978+0000","last_undegraded":"2023-10-25T00:22:08.745978+0000","last_fullsized":"2023-10-25T00:22:08.745978+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:01.502343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"72'3596","reported_seq":3668,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.775009+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-25T00:22:08.775009+0000","last_peered":"2023-10-25T00:22:08.775009+0000","last_clean":"2023-10-25T00:22:08.775009+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-25T00:22:08.775009+0000","last_undegraded":"2023-10-25T00:22:08.775009+0000","last_fullsized":"2023-10-25T00:22:08.775009+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:54.478897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":19820,"num_write_kb":259784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"68'4970","reported_seq":5044,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.201300+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-25T00:21:39.201300+0000","last_peered":"2023-10-25T00:21:39.201300+0000","last_clean":"2023-10-25T00:21:39.201300+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-25T00:21:39.201300+0000","last_undegraded":"2023-10-25T00:21:39.201300+0000","last_fullsized":"2023-10-25T00:21:39.201300+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26721,"num_write_kb":311966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"72'1464","reported_seq":1588,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.213604+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-25T00:22:10.213604+0000","last_peered":"2023-10-25T00:22:10.213604+0000","last_clean":"2023-10-25T00:22:10.213604+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-25T00:22:10.213604+0000","last_undegraded":"2023-10-25T00:22:10.213604+0000","last_fullsized":"2023-10-25T00:22:10.213604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1464,"log_dups_size":0,"ondisk_log_size":1464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":43700,"num_write":1439,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"72'6","reported_seq":96,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.262467+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-25T00:22:08.262467+0000","last_peered":"2023-10-25T00:22:08.262467+0000","last_clean":"2023-10-25T00:22:08.262467+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-25T00:22:08.262467+0000","last_undegraded":"2023-10-25T00:22:08.262467+0000","last_fullsized":"2023-10-25T00:22:08.262467+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:58.545886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"72'3548","reported_seq":3620,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.608884+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-25T00:22:08.608884+0000","last_peered":"2023-10-25T00:22:08.608884+0000","last_clean":"2023-10-25T00:22:08.608884+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-25T00:22:08.608884+0000","last_undegraded":"2023-10-25T00:22:08.608884+0000","last_fullsized":"2023-10-25T00:22:08.608884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:05.595887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20243,"num_write_kb":261539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"68'3393","reported_seq":3469,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194393+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-25T00:21:39.194393+0000","last_peered":"2023-10-25T00:21:39.194393+0000","last_clean":"2023-10-25T00:21:39.194393+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-25T00:21:39.194393+0000","last_undegraded":"2023-10-25T00:21:39.194393+0000","last_fullsized":"2023-10-25T00:21:39.194393+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17500,"num_write_kb":217638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"68'5","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187988+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-25T00:21:39.187988+0000","last_peered":"2023-10-25T00:21:39.187988+0000","last_clean":"2023-10-25T00:21:39.187988+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-25T00:21:39.187988+0000","last_undegraded":"2023-10-25T00:21:39.187988+0000","last_fullsized":"2023-10-25T00:21:39.187988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"72'3","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.738581+0000","last_change":"2023-10-24T23:53:11.454658+0000","last_active":"2023-10-25T00:22:08.738581+0000","last_peered":"2023-10-25T00:22:08.738581+0000","last_clean":"2023-10-25T00:22:08.738581+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-25T00:22:08.738581+0000","last_undegraded":"2023-10-25T00:22:08.738581+0000","last_fullsized":"2023-10-25T00:22:08.738581+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:43:43.869069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"72'3584","reported_seq":3656,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.616239+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-25T00:22:08.616239+0000","last_peered":"2023-10-25T00:22:08.616239+0000","last_clean":"2023-10-25T00:22:08.616239+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-25T00:22:08.616239+0000","last_undegraded":"2023-10-25T00:22:08.616239+0000","last_fullsized":"2023-10-25T00:22:08.616239+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:09:40.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20504,"num_write_kb":254394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'5094","reported_seq":5172,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:20.638687+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-25T00:21:20.638687+0000","last_peered":"2023-10-25T00:21:20.638687+0000","last_clean":"2023-10-25T00:21:20.638687+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-25T00:21:20.638687+0000","last_undegraded":"2023-10-25T00:21:20.638687+0000","last_fullsized":"2023-10-25T00:21:20.638687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27763,"num_write_kb":328490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'6","reported_seq":101,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199601+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-25T00:21:39.199601+0000","last_peered":"2023-10-25T00:21:39.199601+0000","last_clean":"2023-10-25T00:21:39.199601+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-25T00:21:39.199601+0000","last_undegraded":"2023-10-25T00:21:39.199601+0000","last_fullsized":"2023-10-25T00:21:39.199601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"72'1438","reported_seq":1554,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.763905+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-25T00:22:08.763905+0000","last_peered":"2023-10-25T00:22:08.763905+0000","last_clean":"2023-10-25T00:22:08.763905+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-25T00:22:08.763905+0000","last_undegraded":"2023-10-25T00:22:08.763905+0000","last_fullsized":"2023-10-25T00:22:08.763905+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1438,"log_dups_size":0,"ondisk_log_size":1438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:02.591627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":35275,"num_write":1412,"num_write_kb":4487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"72'3292","reported_seq":3364,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.791717+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-25T00:22:08.791717+0000","last_peered":"2023-10-25T00:22:08.791717+0000","last_clean":"2023-10-25T00:22:08.791717+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-25T00:22:08.791717+0000","last_undegraded":"2023-10-25T00:22:08.791717+0000","last_fullsized":"2023-10-25T00:22:08.791717+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:43.015050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17853,"num_write_kb":243602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'4242","reported_seq":4315,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.203827+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-25T00:21:19.203827+0000","last_peered":"2023-10-25T00:21:19.203827+0000","last_clean":"2023-10-25T00:21:19.203827+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-25T00:21:19.203827+0000","last_undegraded":"2023-10-25T00:21:19.203827+0000","last_fullsized":"2023-10-25T00:21:19.203827+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22687,"num_write_kb":275346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'7","reported_seq":124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.498042+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-25T00:21:42.498042+0000","last_peered":"2023-10-25T00:21:42.498042+0000","last_clean":"2023-10-25T00:21:42.498042+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-25T00:21:42.498042+0000","last_undegraded":"2023-10-25T00:21:42.498042+0000","last_fullsized":"2023-10-25T00:21:42.498042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"72'5","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.259848+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-25T00:22:08.259848+0000","last_peered":"2023-10-25T00:22:08.259848+0000","last_clean":"2023-10-25T00:22:08.259848+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-25T00:22:08.259848+0000","last_undegraded":"2023-10-25T00:22:08.259848+0000","last_fullsized":"2023-10-25T00:22:08.259848+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:59.068359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"72'4091","reported_seq":4163,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.758376+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-25T00:22:08.758376+0000","last_peered":"2023-10-25T00:22:08.758376+0000","last_clean":"2023-10-25T00:22:08.758376+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-25T00:22:08.758376+0000","last_undegraded":"2023-10-25T00:22:08.758376+0000","last_fullsized":"2023-10-25T00:22:08.758376+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:13:03.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22779,"num_write_kb":288098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'4310","reported_seq":4383,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.176404+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-25T00:21:19.176404+0000","last_peered":"2023-10-25T00:21:19.176404+0000","last_clean":"2023-10-25T00:21:19.176404+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-25T00:21:19.176404+0000","last_undegraded":"2023-10-25T00:21:19.176404+0000","last_fullsized":"2023-10-25T00:21:19.176404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23791,"num_write_kb":283914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'4","reported_seq":99,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199642+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-25T00:21:39.199642+0000","last_peered":"2023-10-25T00:21:39.199642+0000","last_clean":"2023-10-25T00:21:39.199642+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-25T00:21:39.199642+0000","last_undegraded":"2023-10-25T00:21:39.199642+0000","last_fullsized":"2023-10-25T00:21:39.199642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"72'2","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.702572+0000","last_change":"2023-10-24T23:53:11.451013+0000","last_active":"2023-10-25T00:22:08.702572+0000","last_peered":"2023-10-25T00:22:08.702572+0000","last_clean":"2023-10-25T00:22:08.702572+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-25T00:22:08.702572+0000","last_undegraded":"2023-10-25T00:22:08.702572+0000","last_fullsized":"2023-10-25T00:22:08.702572+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:52:22.503023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"72'3695","reported_seq":3768,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.635013+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-25T00:22:08.635013+0000","last_peered":"2023-10-25T00:22:08.635013+0000","last_clean":"2023-10-25T00:22:08.635013+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-25T00:22:08.635013+0000","last_undegraded":"2023-10-25T00:22:08.635013+0000","last_fullsized":"2023-10-25T00:22:08.635013+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:05:30.149693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21426,"num_write_kb":254037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.776347+0000","last_change":"2023-10-24T23:53:11.434538+0000","last_active":"2023-10-25T00:22:08.776347+0000","last_peered":"2023-10-25T00:22:08.776347+0000","last_clean":"2023-10-25T00:22:08.776347+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-25T00:22:08.776347+0000","last_undegraded":"2023-10-25T00:22:08.776347+0000","last_fullsized":"2023-10-25T00:22:08.776347+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:55.179253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"72'15","reported_seq":196,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.481650+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-25T00:21:42.481650+0000","last_peered":"2023-10-25T00:21:42.481650+0000","last_clean":"2023-10-25T00:21:42.481650+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-25T00:21:42.481650+0000","last_undegraded":"2023-10-25T00:21:42.481650+0000","last_fullsized":"2023-10-25T00:21:42.481650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":327,"num_read_kb":327,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'5249","reported_seq":5329,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183983+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-25T00:21:39.183983+0000","last_peered":"2023-10-25T00:21:39.183983+0000","last_clean":"2023-10-25T00:21:39.183983+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-25T00:21:39.183983+0000","last_undegraded":"2023-10-25T00:21:39.183983+0000","last_fullsized":"2023-10-25T00:21:39.183983+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":27646,"num_write_kb":332285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"72'4380","reported_seq":4450,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.769070+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-25T00:22:08.769070+0000","last_peered":"2023-10-25T00:22:08.769070+0000","last_clean":"2023-10-25T00:22:08.769070+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-25T00:22:08.769070+0000","last_undegraded":"2023-10-25T00:22:08.769070+0000","last_fullsized":"2023-10-25T00:22:08.769070+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:33:08.373963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23662,"num_write_kb":324338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"72'5","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.785854+0000","last_change":"2023-10-24T23:53:11.433387+0000","last_active":"2023-10-25T00:22:08.785854+0000","last_peered":"2023-10-25T00:22:08.785854+0000","last_clean":"2023-10-25T00:22:08.785854+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-25T00:22:08.785854+0000","last_undegraded":"2023-10-25T00:22:08.785854+0000","last_fullsized":"2023-10-25T00:22:08.785854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:29.148450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"68'5","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.194080+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-25T00:21:19.194080+0000","last_peered":"2023-10-25T00:21:19.194080+0000","last_clean":"2023-10-25T00:21:19.194080+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-25T00:21:19.194080+0000","last_undegraded":"2023-10-25T00:21:19.194080+0000","last_fullsized":"2023-10-25T00:21:19.194080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'4442","reported_seq":4514,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195509+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-25T00:21:39.195509+0000","last_peered":"2023-10-25T00:21:39.195509+0000","last_clean":"2023-10-25T00:21:39.195509+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-25T00:21:39.195509+0000","last_undegraded":"2023-10-25T00:21:39.195509+0000","last_fullsized":"2023-10-25T00:21:39.195509+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22810,"num_write_kb":296225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"72'4076","reported_seq":4148,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784809+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-25T00:22:08.784809+0000","last_peered":"2023-10-25T00:22:08.784809+0000","last_clean":"2023-10-25T00:22:08.784809+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-25T00:22:08.784809+0000","last_undegraded":"2023-10-25T00:22:08.784809+0000","last_fullsized":"2023-10-25T00:22:08.784809+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:33:06.172412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22055,"num_write_kb":297609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"68'3","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.211351+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-25T00:21:19.211351+0000","last_peered":"2023-10-25T00:21:19.211351+0000","last_clean":"2023-10-25T00:21:19.211351+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-25T00:21:19.211351+0000","last_undegraded":"2023-10-25T00:21:19.211351+0000","last_fullsized":"2023-10-25T00:21:19.211351+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'4101","reported_seq":4174,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.188446+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-25T00:21:39.188446+0000","last_peered":"2023-10-25T00:21:39.188446+0000","last_clean":"2023-10-25T00:21:39.188446+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-25T00:21:39.188446+0000","last_undegraded":"2023-10-25T00:21:39.188446+0000","last_fullsized":"2023-10-25T00:21:39.188446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22343,"num_write_kb":253486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"72'393","reported_seq":573,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:02.239640+0000","last_change":"2023-10-24T23:53:11.446979+0000","last_active":"2023-10-25T00:22:02.239640+0000","last_peered":"2023-10-25T00:22:02.239640+0000","last_clean":"2023-10-25T00:22:02.239640+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-25T00:22:02.239640+0000","last_undegraded":"2023-10-25T00:22:02.239640+0000","last_fullsized":"2023-10-25T00:22:02.239640+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:54:52.843631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":86197,"num_write":389,"num_write_kb":4306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"72'4606","reported_seq":4676,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784571+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-25T00:22:08.784571+0000","last_peered":"2023-10-25T00:22:08.784571+0000","last_clean":"2023-10-25T00:22:08.784571+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-25T00:22:08.784571+0000","last_undegraded":"2023-10-25T00:22:08.784571+0000","last_fullsized":"2023-10-25T00:22:08.784571+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:18:48.679682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25763,"num_write_kb":316477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"72'11","reported_seq":127,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:40.179752+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-25T00:21:40.179752+0000","last_peered":"2023-10-25T00:21:40.179752+0000","last_clean":"2023-10-25T00:21:40.179752+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-25T00:21:40.179752+0000","last_undegraded":"2023-10-25T00:21:40.179752+0000","last_fullsized":"2023-10-25T00:21:40.179752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'5680","reported_seq":5756,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183466+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-25T00:21:19.183466+0000","last_peered":"2023-10-25T00:21:19.183466+0000","last_clean":"2023-10-25T00:21:19.183466+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-25T00:21:19.183466+0000","last_undegraded":"2023-10-25T00:21:19.183466+0000","last_fullsized":"2023-10-25T00:21:19.183466+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29749,"num_write_kb":360962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"72'38","reported_seq":172,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:09.508467+0000","last_change":"2023-10-24T23:53:11.436964+0000","last_active":"2023-10-25T00:22:09.508467+0000","last_peered":"2023-10-25T00:22:09.508467+0000","last_clean":"2023-10-25T00:22:09.508467+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-25T00:22:09.508467+0000","last_undegraded":"2023-10-25T00:22:09.508467+0000","last_fullsized":"2023-10-25T00:22:09.508467+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:04:15.366966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":39,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"72'4808","reported_seq":4884,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.778926+0000","last_change":"2023-10-24T23:53:11.814304+0000","last_active":"2023-10-25T00:22:08.778926+0000","last_peered":"2023-10-25T00:22:08.778926+0000","last_clean":"2023-10-25T00:22:08.778926+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-25T00:22:08.778926+0000","last_undegraded":"2023-10-25T00:22:08.778926+0000","last_fullsized":"2023-10-25T00:22:08.778926+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:16.635178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26552,"num_write_kb":336125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"68'4747","reported_seq":4822,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182709+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-25T00:21:39.182709+0000","last_peered":"2023-10-25T00:21:39.182709+0000","last_clean":"2023-10-25T00:21:39.182709+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-25T00:21:39.182709+0000","last_undegraded":"2023-10-25T00:21:39.182709+0000","last_fullsized":"2023-10-25T00:21:39.182709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24847,"num_write_kb":316685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"72'4283","reported_seq":4356,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.589765+0000","last_change":"2023-10-24T23:53:11.807096+0000","last_active":"2023-10-25T00:22:08.589765+0000","last_peered":"2023-10-25T00:22:08.589765+0000","last_clean":"2023-10-25T00:22:08.589765+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-25T00:22:08.589765+0000","last_undegraded":"2023-10-25T00:22:08.589765+0000","last_fullsized":"2023-10-25T00:22:08.589765+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:25.144578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23853,"num_write_kb":296504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"68'4036","reported_seq":4111,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183566+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-25T00:21:39.183566+0000","last_peered":"2023-10-25T00:21:39.183566+0000","last_clean":"2023-10-25T00:21:39.183566+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-25T00:21:39.183566+0000","last_undegraded":"2023-10-25T00:21:39.183566+0000","last_fullsized":"2023-10-25T00:21:39.183566+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21233,"num_write_kb":273467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"72'4276","reported_seq":4349,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.743483+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-25T00:22:08.743483+0000","last_peered":"2023-10-25T00:22:08.743483+0000","last_clean":"2023-10-25T00:22:08.743483+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-25T00:22:08.743483+0000","last_undegraded":"2023-10-25T00:22:08.743483+0000","last_fullsized":"2023-10-25T00:22:08.743483+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:38.946585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23362,"num_write_kb":310817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"68'6201","reported_seq":6278,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.198522+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-25T00:21:39.198522+0000","last_peered":"2023-10-25T00:21:39.198522+0000","last_clean":"2023-10-25T00:21:39.198522+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-25T00:21:39.198522+0000","last_undegraded":"2023-10-25T00:21:39.198522+0000","last_fullsized":"2023-10-25T00:21:39.198522+0000","mapping_epoch":21,"log_start":"62'1000","ondisk_log_start":"62'1000","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33486,"num_write_kb":384441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"72'4018","reported_seq":4092,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.764145+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-25T00:22:08.764145+0000","last_peered":"2023-10-25T00:22:08.764145+0000","last_clean":"2023-10-25T00:22:08.764145+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-25T00:22:08.764145+0000","last_undegraded":"2023-10-25T00:22:08.764145+0000","last_fullsized":"2023-10-25T00:22:08.764145+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:48:24.911279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":21920,"num_write_kb":288651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"68'4522","reported_seq":4598,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186452+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-25T00:21:39.186452+0000","last_peered":"2023-10-25T00:21:39.186452+0000","last_clean":"2023-10-25T00:21:39.186452+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-25T00:21:39.186452+0000","last_undegraded":"2023-10-25T00:21:39.186452+0000","last_fullsized":"2023-10-25T00:21:39.186452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24476,"num_write_kb":282500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"72'4209","reported_seq":4279,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760824+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-25T00:22:08.760824+0000","last_peered":"2023-10-25T00:22:08.760824+0000","last_clean":"2023-10-25T00:22:08.760824+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-25T00:22:08.760824+0000","last_undegraded":"2023-10-25T00:22:08.760824+0000","last_fullsized":"2023-10-25T00:22:08.760824+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:21.165755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24141,"num_write_kb":281325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"68'4915","reported_seq":4988,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180147+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-25T00:21:19.180147+0000","last_peered":"2023-10-25T00:21:19.180147+0000","last_clean":"2023-10-25T00:21:19.180147+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-25T00:21:19.180147+0000","last_undegraded":"2023-10-25T00:21:19.180147+0000","last_fullsized":"2023-10-25T00:21:19.180147+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26161,"num_write_kb":314084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"72'4933","reported_seq":5003,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.718975+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-25T00:22:08.718975+0000","last_peered":"2023-10-25T00:22:08.718975+0000","last_clean":"2023-10-25T00:22:08.718975+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-25T00:22:08.718975+0000","last_undegraded":"2023-10-25T00:22:08.718975+0000","last_fullsized":"2023-10-25T00:22:08.718975+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26695,"num_write_kb":352905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"68'5055","reported_seq":5129,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180248+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-25T00:21:19.180248+0000","last_peered":"2023-10-25T00:21:19.180248+0000","last_clean":"2023-10-25T00:21:19.180248+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-25T00:21:19.180248+0000","last_undegraded":"2023-10-25T00:21:19.180248+0000","last_fullsized":"2023-10-25T00:21:19.180248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27516,"num_write_kb":327406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"72'4083","reported_seq":4157,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.731484+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-25T00:22:08.731484+0000","last_peered":"2023-10-25T00:22:08.731484+0000","last_clean":"2023-10-25T00:22:08.731484+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-25T00:22:08.731484+0000","last_undegraded":"2023-10-25T00:22:08.731484+0000","last_fullsized":"2023-10-25T00:22:08.731484+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:54.422365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":22892,"num_write_kb":296872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"68'4777","reported_seq":4852,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178413+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-25T00:21:19.178413+0000","last_peered":"2023-10-25T00:21:19.178413+0000","last_clean":"2023-10-25T00:21:19.178413+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-25T00:21:19.178413+0000","last_undegraded":"2023-10-25T00:21:19.178413+0000","last_fullsized":"2023-10-25T00:21:19.178413+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25812,"num_write_kb":300181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"72'4618","reported_seq":4693,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.741877+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-25T00:22:08.741877+0000","last_peered":"2023-10-25T00:22:08.741877+0000","last_clean":"2023-10-25T00:22:08.741877+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-25T00:22:08.741877+0000","last_undegraded":"2023-10-25T00:22:08.741877+0000","last_fullsized":"2023-10-25T00:22:08.741877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:01:11.838035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":26180,"num_write_kb":326342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"68'5340","reported_seq":5413,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178233+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-25T00:21:19.178233+0000","last_peered":"2023-10-25T00:21:19.178233+0000","last_clean":"2023-10-25T00:21:19.178233+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-25T00:21:19.178233+0000","last_undegraded":"2023-10-25T00:21:19.178233+0000","last_fullsized":"2023-10-25T00:21:19.178233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28701,"num_write_kb":348521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"72'4171","reported_seq":4242,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.497360+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-25T00:22:08.497360+0000","last_peered":"2023-10-25T00:22:08.497360+0000","last_clean":"2023-10-25T00:22:08.497360+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-25T00:22:08.497360+0000","last_undegraded":"2023-10-25T00:22:08.497360+0000","last_fullsized":"2023-10-25T00:22:08.497360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:09:25.424344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22551,"num_write_kb":303498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"68'4493","reported_seq":4567,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.192644+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-25T00:21:19.192644+0000","last_peered":"2023-10-25T00:21:19.192644+0000","last_clean":"2023-10-25T00:21:19.192644+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-25T00:21:19.192644+0000","last_undegraded":"2023-10-25T00:21:19.192644+0000","last_fullsized":"2023-10-25T00:21:19.192644+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23797,"num_write_kb":300260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"72'4763","reported_seq":4833,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.772760+0000","last_change":"2023-10-24T23:53:44.474203+0000","last_active":"2023-10-25T00:22:08.772760+0000","last_peered":"2023-10-25T00:22:08.772760+0000","last_clean":"2023-10-25T00:22:08.772760+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-25T00:22:08.772760+0000","last_undegraded":"2023-10-25T00:22:08.772760+0000","last_fullsized":"2023-10-25T00:22:08.772760+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:16:50.292761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":25751,"num_write_kb":345225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"68'4946","reported_seq":5019,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183160+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-25T00:21:39.183160+0000","last_peered":"2023-10-25T00:21:39.183160+0000","last_clean":"2023-10-25T00:21:39.183160+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-25T00:21:39.183160+0000","last_undegraded":"2023-10-25T00:21:39.183160+0000","last_fullsized":"2023-10-25T00:21:39.183160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26583,"num_write_kb":315633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"72'4352","reported_seq":4423,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.762501+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-25T00:22:08.762501+0000","last_peered":"2023-10-25T00:22:08.762501+0000","last_clean":"2023-10-25T00:22:08.762501+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-25T00:22:08.762501+0000","last_undegraded":"2023-10-25T00:22:08.762501+0000","last_fullsized":"2023-10-25T00:22:08.762501+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:10:20.503622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23957,"num_write_kb":301052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"68'5026","reported_seq":5099,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193252+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-25T00:21:39.193252+0000","last_peered":"2023-10-25T00:21:39.193252+0000","last_clean":"2023-10-25T00:21:39.193252+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-25T00:21:39.193252+0000","last_undegraded":"2023-10-25T00:21:39.193252+0000","last_fullsized":"2023-10-25T00:21:39.193252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26401,"num_write_kb":310586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"72'4393","reported_seq":4468,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.745975+0000","last_change":"2023-10-24T23:53:11.860133+0000","last_active":"2023-10-25T00:22:08.745975+0000","last_peered":"2023-10-25T00:22:08.745975+0000","last_clean":"2023-10-25T00:22:08.745975+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-25T00:22:08.745975+0000","last_undegraded":"2023-10-25T00:22:08.745975+0000","last_fullsized":"2023-10-25T00:22:08.745975+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:43.051362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":24752,"num_write_kb":296109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"68'5620","reported_seq":5697,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197727+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-25T00:21:39.197727+0000","last_peered":"2023-10-25T00:21:39.197727+0000","last_clean":"2023-10-25T00:21:39.197727+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-25T00:21:39.197727+0000","last_undegraded":"2023-10-25T00:21:39.197727+0000","last_fullsized":"2023-10-25T00:21:39.197727+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29863,"num_write_kb":337169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"72'4340","reported_seq":4414,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.730151+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-25T00:22:08.730151+0000","last_peered":"2023-10-25T00:22:08.730151+0000","last_clean":"2023-10-25T00:22:08.730151+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-25T00:22:08.730151+0000","last_undegraded":"2023-10-25T00:22:08.730151+0000","last_fullsized":"2023-10-25T00:22:08.730151+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:39:14.947372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":36,"num_write":23367,"num_write_kb":302669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"68'4464","reported_seq":4536,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.200961+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-25T00:21:39.200961+0000","last_peered":"2023-10-25T00:21:39.200961+0000","last_clean":"2023-10-25T00:21:39.200961+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-25T00:21:39.200961+0000","last_undegraded":"2023-10-25T00:21:39.200961+0000","last_fullsized":"2023-10-25T00:21:39.200961+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24372,"num_write_kb":278074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"72'5589","reported_seq":5659,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.768377+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-25T00:22:08.768377+0000","last_peered":"2023-10-25T00:22:08.768377+0000","last_clean":"2023-10-25T00:22:08.768377+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-25T00:22:08.768377+0000","last_undegraded":"2023-10-25T00:22:08.768377+0000","last_fullsized":"2023-10-25T00:22:08.768377+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:37.413259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":30206,"num_write_kb":399557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"68'4542","reported_seq":4615,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.196542+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-25T00:21:39.196542+0000","last_peered":"2023-10-25T00:21:39.196542+0000","last_clean":"2023-10-25T00:21:39.196542+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-25T00:21:39.196542+0000","last_undegraded":"2023-10-25T00:21:39.196542+0000","last_fullsized":"2023-10-25T00:21:39.196542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:26:18.928304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23889,"num_write_kb":292323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"72'3430","reported_seq":3501,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.718898+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-25T00:22:08.718898+0000","last_peered":"2023-10-25T00:22:08.718898+0000","last_clean":"2023-10-25T00:22:08.718898+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-25T00:22:08.718898+0000","last_undegraded":"2023-10-25T00:22:08.718898+0000","last_fullsized":"2023-10-25T00:22:08.718898+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:31.344724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19411,"num_write_kb":241862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"68'4719","reported_seq":4792,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.207948+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-25T00:21:19.207948+0000","last_peered":"2023-10-25T00:21:19.207948+0000","last_clean":"2023-10-25T00:21:19.207948+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-25T00:21:19.207948+0000","last_undegraded":"2023-10-25T00:21:19.207948+0000","last_fullsized":"2023-10-25T00:21:19.207948+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25020,"num_write_kb":305197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"72'4169","reported_seq":4237,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.618436+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-25T00:22:08.618436+0000","last_peered":"2023-10-25T00:22:08.618436+0000","last_clean":"2023-10-25T00:22:08.618436+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-25T00:22:08.618436+0000","last_undegraded":"2023-10-25T00:22:08.618436+0000","last_fullsized":"2023-10-25T00:22:08.618436+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23235,"num_write_kb":306175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"68'5952","reported_seq":6033,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.215357+0000","last_change":"2023-10-24T23:53:44.629959+0000","last_active":"2023-10-25T00:21:19.215357+0000","last_peered":"2023-10-25T00:21:19.215357+0000","last_clean":"2023-10-25T00:21:19.215357+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-25T00:21:19.215357+0000","last_undegraded":"2023-10-25T00:21:19.215357+0000","last_fullsized":"2023-10-25T00:21:19.215357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:59:36.474350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32341,"num_write_kb":375441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"72'3717","reported_seq":3786,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.786783+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-25T00:22:08.786783+0000","last_peered":"2023-10-25T00:22:08.786783+0000","last_clean":"2023-10-25T00:22:08.786783+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-25T00:22:08.786783+0000","last_undegraded":"2023-10-25T00:22:08.786783+0000","last_fullsized":"2023-10-25T00:22:08.786783+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:32:05.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21101,"num_write_kb":269108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"68'4825","reported_seq":4899,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.204063+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-25T00:21:19.204063+0000","last_peered":"2023-10-25T00:21:19.204063+0000","last_clean":"2023-10-25T00:21:19.204063+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-25T00:21:19.204063+0000","last_undegraded":"2023-10-25T00:21:19.204063+0000","last_fullsized":"2023-10-25T00:21:19.204063+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25319,"num_write_kb":294113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"72'3714","reported_seq":3786,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.739265+0000","last_change":"2023-10-24T23:53:11.847814+0000","last_active":"2023-10-25T00:22:08.739265+0000","last_peered":"2023-10-25T00:22:08.739265+0000","last_clean":"2023-10-25T00:22:08.739265+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-25T00:22:08.739265+0000","last_undegraded":"2023-10-25T00:22:08.739265+0000","last_fullsized":"2023-10-25T00:22:08.739265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:59.075858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21148,"num_write_kb":265324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"68'4878","reported_seq":4953,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.194001+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-25T00:21:19.194001+0000","last_peered":"2023-10-25T00:21:19.194001+0000","last_clean":"2023-10-25T00:21:19.194001+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-25T00:21:19.194001+0000","last_undegraded":"2023-10-25T00:21:19.194001+0000","last_fullsized":"2023-10-25T00:21:19.194001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26154,"num_write_kb":305558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"72'3434","reported_seq":3504,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.726527+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-25T00:22:08.726527+0000","last_peered":"2023-10-25T00:22:08.726527+0000","last_clean":"2023-10-25T00:22:08.726527+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-25T00:22:08.726527+0000","last_undegraded":"2023-10-25T00:22:08.726527+0000","last_fullsized":"2023-10-25T00:22:08.726527+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:25:12.525273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18449,"num_write_kb":262373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"68'4672","reported_seq":4746,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178513+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-25T00:21:19.178513+0000","last_peered":"2023-10-25T00:21:19.178513+0000","last_clean":"2023-10-25T00:21:19.178513+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-25T00:21:19.178513+0000","last_undegraded":"2023-10-25T00:21:19.178513+0000","last_fullsized":"2023-10-25T00:21:19.178513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25223,"num_write_kb":293792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"72'4013","reported_seq":4088,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.743804+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-25T00:22:08.743804+0000","last_peered":"2023-10-25T00:22:08.743804+0000","last_clean":"2023-10-25T00:22:08.743804+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-25T00:22:08.743804+0000","last_undegraded":"2023-10-25T00:22:08.743804+0000","last_fullsized":"2023-10-25T00:22:08.743804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:12:21.026996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":22066,"num_write_kb":297298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"68'3918","reported_seq":3990,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194748+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-25T00:21:39.194748+0000","last_peered":"2023-10-25T00:21:39.194748+0000","last_clean":"2023-10-25T00:21:39.194748+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-25T00:21:39.194748+0000","last_undegraded":"2023-10-25T00:21:39.194748+0000","last_fullsized":"2023-10-25T00:21:39.194748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:32:34.789544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21675,"num_write_kb":247000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"72'4401","reported_seq":4472,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.740526+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-25T00:22:08.740526+0000","last_peered":"2023-10-25T00:22:08.740526+0000","last_clean":"2023-10-25T00:22:08.740526+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-25T00:22:08.740526+0000","last_undegraded":"2023-10-25T00:22:08.740526+0000","last_fullsized":"2023-10-25T00:22:08.740526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:07.909021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":24294,"num_write_kb":326749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"68'3970","reported_seq":4042,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194610+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-25T00:21:39.194610+0000","last_peered":"2023-10-25T00:21:39.194610+0000","last_clean":"2023-10-25T00:21:39.194610+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-25T00:21:39.194610+0000","last_undegraded":"2023-10-25T00:21:39.194610+0000","last_fullsized":"2023-10-25T00:21:39.194610+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:26:46.775764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21202,"num_write_kb":257085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"72'5559","reported_seq":5636,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.789325+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-25T00:22:08.789325+0000","last_peered":"2023-10-25T00:22:08.789325+0000","last_clean":"2023-10-25T00:22:08.789325+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-25T00:22:08.789325+0000","last_undegraded":"2023-10-25T00:22:08.789325+0000","last_fullsized":"2023-10-25T00:22:08.789325+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:48.217889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":31202,"num_write_kb":391572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"68'4936","reported_seq":5009,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.185286+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-25T00:21:39.185286+0000","last_peered":"2023-10-25T00:21:39.185286+0000","last_clean":"2023-10-25T00:21:39.185286+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-25T00:21:39.185286+0000","last_undegraded":"2023-10-25T00:21:39.185286+0000","last_fullsized":"2023-10-25T00:21:39.185286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26134,"num_write_kb":309652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"72'3558","reported_seq":3632,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.781338+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-25T00:22:08.781338+0000","last_peered":"2023-10-25T00:22:08.781338+0000","last_clean":"2023-10-25T00:22:08.781338+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-25T00:22:08.781338+0000","last_undegraded":"2023-10-25T00:22:08.781338+0000","last_fullsized":"2023-10-25T00:22:08.781338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:10:18.973606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":34,"num_write":19715,"num_write_kb":243025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"68'5282","reported_seq":5355,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.192868+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-25T00:21:39.192868+0000","last_peered":"2023-10-25T00:21:39.192868+0000","last_clean":"2023-10-25T00:21:39.192868+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-25T00:21:39.192868+0000","last_undegraded":"2023-10-25T00:21:39.192868+0000","last_fullsized":"2023-10-25T00:21:39.192868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27848,"num_write_kb":324561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"72'3914","reported_seq":3989,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.624743+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-25T00:22:08.624743+0000","last_peered":"2023-10-25T00:22:08.624743+0000","last_clean":"2023-10-25T00:22:08.624743+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-25T00:22:08.624743+0000","last_undegraded":"2023-10-25T00:22:08.624743+0000","last_fullsized":"2023-10-25T00:22:08.624743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:24:45.536141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":20938,"num_write_kb":288796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"68'6616","reported_seq":6693,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194923+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-25T00:21:39.194923+0000","last_peered":"2023-10-25T00:21:39.194923+0000","last_clean":"2023-10-25T00:21:39.194923+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-25T00:21:39.194923+0000","last_undegraded":"2023-10-25T00:21:39.194923+0000","last_fullsized":"2023-10-25T00:21:39.194923+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36221,"num_write_kb":423916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"72'3536","reported_seq":3606,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.700035+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-25T00:22:08.700035+0000","last_peered":"2023-10-25T00:22:08.700035+0000","last_clean":"2023-10-25T00:22:08.700035+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-25T00:22:08.700035+0000","last_undegraded":"2023-10-25T00:22:08.700035+0000","last_fullsized":"2023-10-25T00:22:08.700035+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:06.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19276,"num_write_kb":253506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"68'4980","reported_seq":5050,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.176264+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-25T00:21:19.176264+0000","last_peered":"2023-10-25T00:21:19.176264+0000","last_clean":"2023-10-25T00:21:19.176264+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-25T00:21:19.176264+0000","last_undegraded":"2023-10-25T00:21:19.176264+0000","last_fullsized":"2023-10-25T00:21:19.176264+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26525,"num_write_kb":306963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"72'4160","reported_seq":4232,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.763999+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-25T00:22:08.763999+0000","last_peered":"2023-10-25T00:22:08.763999+0000","last_clean":"2023-10-25T00:22:08.763999+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-25T00:22:08.763999+0000","last_undegraded":"2023-10-25T00:22:08.763999+0000","last_fullsized":"2023-10-25T00:22:08.763999+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:56:02.798009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":21819,"num_write_kb":288439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"68'4434","reported_seq":4506,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193726+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-25T00:21:19.193726+0000","last_peered":"2023-10-25T00:21:19.193726+0000","last_clean":"2023-10-25T00:21:19.193726+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-25T00:21:19.193726+0000","last_undegraded":"2023-10-25T00:21:19.193726+0000","last_fullsized":"2023-10-25T00:21:19.193726+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23791,"num_write_kb":292031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"72'4195","reported_seq":4262,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.728890+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-25T00:22:08.728890+0000","last_peered":"2023-10-25T00:22:08.728890+0000","last_clean":"2023-10-25T00:22:08.728890+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-25T00:22:08.728890+0000","last_undegraded":"2023-10-25T00:22:08.728890+0000","last_fullsized":"2023-10-25T00:22:08.728890+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:40.168016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23000,"num_write_kb":285019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"68'4243","reported_seq":4314,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205516+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-25T00:21:19.205516+0000","last_peered":"2023-10-25T00:21:19.205516+0000","last_clean":"2023-10-25T00:21:19.205516+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-25T00:21:19.205516+0000","last_undegraded":"2023-10-25T00:21:19.205516+0000","last_fullsized":"2023-10-25T00:21:19.205516+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22439,"num_write_kb":266094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"72'4017","reported_seq":4085,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.783880+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-25T00:22:08.783880+0000","last_peered":"2023-10-25T00:22:08.783880+0000","last_clean":"2023-10-25T00:22:08.783880+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-25T00:22:08.783880+0000","last_undegraded":"2023-10-25T00:22:08.783880+0000","last_fullsized":"2023-10-25T00:22:08.783880+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:39.720375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22665,"num_write_kb":291707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"68'4509","reported_seq":4583,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.213920+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-25T00:21:19.213920+0000","last_peered":"2023-10-25T00:21:19.213920+0000","last_clean":"2023-10-25T00:21:19.213920+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-25T00:21:19.213920+0000","last_undegraded":"2023-10-25T00:21:19.213920+0000","last_fullsized":"2023-10-25T00:21:19.213920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23158,"num_write_kb":297883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"72'5235","reported_seq":5312,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.789212+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-25T00:22:08.789212+0000","last_peered":"2023-10-25T00:22:08.789212+0000","last_clean":"2023-10-25T00:22:08.789212+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-25T00:22:08.789212+0000","last_undegraded":"2023-10-25T00:22:08.789212+0000","last_fullsized":"2023-10-25T00:22:08.789212+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:51:40.960630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":15,"num_write":29235,"num_write_kb":367391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"68'4260","reported_seq":4335,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.221539+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-25T00:21:19.221539+0000","last_peered":"2023-10-25T00:21:19.221539+0000","last_clean":"2023-10-25T00:21:19.221539+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-25T00:21:19.221539+0000","last_undegraded":"2023-10-25T00:21:19.221539+0000","last_fullsized":"2023-10-25T00:21:19.221539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23357,"num_write_kb":265837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"72'4656","reported_seq":4735,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.794660+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-25T00:22:08.794660+0000","last_peered":"2023-10-25T00:22:08.794660+0000","last_clean":"2023-10-25T00:22:08.794660+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-25T00:22:08.794660+0000","last_undegraded":"2023-10-25T00:22:08.794660+0000","last_fullsized":"2023-10-25T00:22:08.794660+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:22:42.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":26109,"num_write_kb":317781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"68'4636","reported_seq":4713,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194609+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-25T00:21:39.194609+0000","last_peered":"2023-10-25T00:21:39.194609+0000","last_clean":"2023-10-25T00:21:39.194609+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-25T00:21:39.194609+0000","last_undegraded":"2023-10-25T00:21:39.194609+0000","last_fullsized":"2023-10-25T00:21:39.194609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25124,"num_write_kb":285314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"72'3117","reported_seq":3190,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.574961+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-25T00:22:08.574961+0000","last_peered":"2023-10-25T00:22:08.574961+0000","last_clean":"2023-10-25T00:22:08.574961+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-25T00:22:08.574961+0000","last_undegraded":"2023-10-25T00:22:08.574961+0000","last_fullsized":"2023-10-25T00:22:08.574961+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:47.658717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17240,"num_write_kb":232937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"68'5520","reported_seq":5604,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.192356+0000","last_change":"2023-10-24T23:53:44.630867+0000","last_active":"2023-10-25T00:21:39.192356+0000","last_peered":"2023-10-25T00:21:39.192356+0000","last_clean":"2023-10-25T00:21:39.192356+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-25T00:21:39.192356+0000","last_undegraded":"2023-10-25T00:21:39.192356+0000","last_fullsized":"2023-10-25T00:21:39.192356+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:49.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29665,"num_write_kb":368823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"72'3962","reported_seq":4034,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.769841+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-25T00:22:08.769841+0000","last_peered":"2023-10-25T00:22:08.769841+0000","last_clean":"2023-10-25T00:22:08.769841+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-25T00:22:08.769841+0000","last_undegraded":"2023-10-25T00:22:08.769841+0000","last_fullsized":"2023-10-25T00:22:08.769841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:50.994593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21989,"num_write_kb":280762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"68'4908","reported_seq":4987,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.198947+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-25T00:21:39.198947+0000","last_peered":"2023-10-25T00:21:39.198947+0000","last_clean":"2023-10-25T00:21:39.198947+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-25T00:21:39.198947+0000","last_undegraded":"2023-10-25T00:21:39.198947+0000","last_fullsized":"2023-10-25T00:21:39.198947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:13:03.558050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26829,"num_write_kb":320697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":320,"num_write":1482516,"num_write_kb":19060243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267718,"ondisk_log_size":267718,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":19490997,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1686,"num_read_kb":209824,"num_write":4665,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39043072,"data_stored":38981994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4535,"ondisk_log_size":4535,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":0,"num_write":1661441,"num_write_kb":19773626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307026,"ondisk_log_size":307026,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":20040877,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1620,"num_read_kb":247649,"num_write":4615,"num_write_kb":21609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40140800,"data_stored":40081754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4486,"ondisk_log_size":4486,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575206,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919080,"kb_used_data":12320,"kb_used_omap":0,"kb_used_meta":906752,"kb_avail":93452760,"statfs":{"total":96636764160,"available":95695626240,"internally_reserved":0,"allocated":12615680,"data_stored":12142216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":928514048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.304,"15min":0.93799999999999994},"min":{"1min":0.22,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.6349999999999998,"5min":13.929,"15min":13.929},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.3,"15min":0.92900000000000005},"min":{"1min":0.29699999999999999,"5min":0.27000000000000002,"15min":0.20999999999999999},"max":{"1min":3.6659999999999999,"5min":11.315,"15min":11.315},"last":1.02}]},{"osd":1,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.714,"15min":1.1639999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.215},"max":{"1min":3.6949999999999998,"5min":28.308,"15min":28.308},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.782,"15min":1.2290000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":3.7490000000000001,"5min":28.346,"15min":28.346},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.3779999999999999,"15min":0.99399999999999999},"min":{"1min":0.28499999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":5.71,"5min":11.430999999999999,"15min":11.430999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.139,"15min":0.94599999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.8799999999999999,"5min":7.9710000000000001,"15min":7.9710000000000001},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":2.3180000000000001,"15min":1.363},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.6150000000000002,"5min":93.664000000000001,"15min":93.664000000000001},"last":1.081},{"interface":"front","average":{"1min":1.2030000000000001,"5min":2.379,"15min":1.3740000000000001},"min":{"1min":0.34499999999999997,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":2.8809999999999998,"5min":93.073999999999998,"15min":93.073999999999998},"last":0.91400000000000003}]},{"osd":4,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0840000000000001,"15min":0.877},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.249},"max":{"1min":2.9089999999999998,"5min":7.6280000000000001,"15min":7.6280000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.714,"15min":1.1339999999999999},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.27200000000000002},"max":{"1min":4.9640000000000004,"5min":27.936,"15min":27.936},"last":0.98299999999999998}]},{"osd":5,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":2.1459999999999999,"5min":1.573,"15min":1.05},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.8170000000000002,"5min":21.126999999999999,"15min":21.126999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.995,"15min":0.94699999999999995},"min":{"1min":0.39800000000000002,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":2.5910000000000002,"5min":11.49,"15min":11.49},"last":0.878}]},{"osd":6,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.4770000000000001,"15min":1.077},"min":{"1min":0.41999999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.8140000000000001,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7290000000000001,"15min":1.1799999999999999},"min":{"1min":0.49399999999999999,"5min":0.38100000000000001,"15min":0.35499999999999998},"max":{"1min":4.8639999999999999,"5min":20.651,"15min":20.651},"last":1.115}]}]},{"osd":6,"up_from":13,"seq":55834575206,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859004,"kb_used_data":5044,"kb_used_omap":0,"kb_used_meta":853952,"kb_avail":93512836,"statfs":{"total":96636764160,"available":95757144064,"internally_reserved":0,"allocated":5165056,"data_stored":4728730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":874446848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.397,"15min":0.93899999999999995},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.9009999999999998,"5min":15.974,"15min":15.974},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.4199999999999999,"15min":0.96799999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":8.2400000000000002,"5min":8.4789999999999992,"15min":8.4789999999999992},"last":0.89500000000000002}]},{"osd":1,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.042,"15min":0.878},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.221},"max":{"1min":3.5630000000000002,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.2549999999999999,"15min":0.98599999999999999},"min":{"1min":0.28699999999999998,"5min":0.254,"15min":0.249},"max":{"1min":3.5840000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.92200000000000004}]},{"osd":2,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.617,"15min":1.0600000000000001},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.6779999999999999,"5min":12.615,"15min":12.615},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.228,"5min":1.591,"15min":1.105},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.702,"5min":12.667999999999999,"15min":12.667999999999999},"last":0.72399999999999998}]},{"osd":3,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.9690000000000001,"15min":1.2350000000000001},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.7350000000000003,"5min":38.649999999999999,"15min":38.649999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":2.0249999999999999,"15min":1.3260000000000001},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.766,"5min":38.683,"15min":38.683},"last":1.014}]},{"osd":4,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.7589999999999999,"15min":1.0960000000000001},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.738,"5min":12.682,"15min":12.682},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4350000000000001,"15min":1.079},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.694,"5min":10.686,"15min":10.686},"last":1.1279999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.1990000000000001,"15min":0.94799999999999995},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.24299999999999999},"max":{"1min":3.6379999999999999,"5min":8.0039999999999996,"15min":8.0039999999999996},"last":1.091},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.3700000000000001,"15min":1.046},"min":{"1min":0.496,"5min":0.33400000000000002,"15min":0.30299999999999999},"max":{"1min":3.903,"5min":11.766999999999999,"15min":11.766999999999999},"last":1.0429999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.907,"5min":1.306,"15min":1.018},"min":{"1min":0.32200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":8.2720000000000002,"5min":12.926,"15min":12.926},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.258,"15min":1.026},"min":{"1min":0.38200000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.1100000000000003,"5min":7.6619999999999999,"15min":7.6619999999999999},"last":1.335}]}]},{"osd":5,"up_from":13,"seq":55834575206,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":958856,"kb_used_data":9152,"kb_used_omap":0,"kb_used_meta":949696,"kb_avail":93412984,"statfs":{"total":96636764160,"available":95654895616,"internally_reserved":0,"allocated":9371648,"data_stored":8923216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":972488704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":2.4649999999999999,"5min":1.4690000000000001,"15min":1.032},"min":{"1min":0.23599999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.837999999999999,"5min":14.837999999999999,"15min":14.837999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.1599999999999999,"15min":0.95999999999999996},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.536,"5min":7.423,"15min":7.423},"last":0.73999999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":1.9990000000000001,"5min":1.6000000000000001,"15min":1.0840000000000001},"min":{"1min":0.27500000000000002,"5min":0.24299999999999999,"15min":0.23400000000000001},"max":{"1min":12.141,"5min":12.141,"15min":12.141},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.518,"15min":1.0820000000000001},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.252},"max":{"1min":3.5950000000000002,"5min":24.648,"15min":24.648},"last":1.008}]},{"osd":2,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":1.9750000000000001,"5min":1.841,"15min":1.236},"min":{"1min":0.29899999999999999,"5min":0.25,"15min":0.25},"max":{"1min":12.175000000000001,"5min":20.760999999999999,"15min":20.760999999999999},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.655,"5min":1.46,"15min":1.0800000000000001},"min":{"1min":0.317,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":12.109999999999999,"5min":20.552,"15min":20.552},"last":0.84599999999999997}]},{"osd":3,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":1.879,"5min":1.5569999999999999,"15min":1.224},"min":{"1min":0.35599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":12.013,"5min":12.013,"15min":12.013},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.6819999999999999,"15min":1.2549999999999999},"min":{"1min":0.34399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.919,"5min":12.805,"15min":12.805},"last":0.97399999999999998}]},{"osd":4,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":2.3919999999999999,"5min":1.8240000000000001,"15min":1.204},"min":{"1min":0.38900000000000001,"5min":0.26000000000000001,"15min":0.23999999999999999},"max":{"1min":12.201000000000001,"5min":12.728,"15min":12.728},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.131,"5min":1.486,"15min":1.141},"min":{"1min":0.36599999999999999,"5min":0.27600000000000002,"15min":0.27200000000000002},"max":{"1min":3.6179999999999999,"5min":10.943,"15min":10.943},"last":1.081}]},{"osd":6,"last update":"Wed Oct 25 00:21:58 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.369,"15min":1.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.33500000000000002,"15min":0.26400000000000001},"max":{"1min":9.6489999999999991,"5min":17.936,"15min":17.936},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.96,"15min":1.224},"min":{"1min":0.312,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":12.042999999999999,"5min":14.766999999999999,"15min":14.766999999999999},"last":0.872}]},{"osd":7,"last update":"Wed Oct 25 00:21:35 2023","interfaces":[{"interface":"back","average":{"1min":2.984,"5min":1.7709999999999999,"15min":1.26},"min":{"1min":0.373,"5min":0.316,"15min":0.29499999999999998},"max":{"1min":28.178999999999998,"5min":28.178999999999998,"15min":28.178999999999998},"last":1.107},{"interface":"front","average":{"1min":3.23,"5min":2.1659999999999999,"15min":1.3540000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":28.222999999999999,"5min":28.222999999999999,"15min":28.222999999999999},"last":1.1439999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575205,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946272,"kb_used_data":19096,"kb_used_omap":0,"kb_used_meta":927168,"kb_avail":93425568,"statfs":{"total":96636764160,"available":95667781632,"internally_reserved":0,"allocated":19554304,"data_stored":19092590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":949420032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":1.1910000000000001,"15min":0.93000000000000005},"min":{"1min":0.21199999999999999,"5min":0.19700000000000001,"15min":0.187},"max":{"1min":3.6120000000000001,"5min":20.922999999999998,"15min":20.922999999999998},"last":1.081},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.1499999999999999,"15min":0.96199999999999997},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.217},"max":{"1min":1.224,"5min":20.532,"15min":20.532},"last":0.76000000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.3560000000000001,"15min":0.97699999999999998},"min":{"1min":0.224,"5min":0.22,"15min":0.22},"max":{"1min":2.8140000000000001,"5min":20.914000000000001,"15min":20.914000000000001},"last":0.999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.738,"15min":1.157},"min":{"1min":0.28699999999999998,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":2.8399999999999999,"5min":23.265999999999998,"15min":23.265999999999998},"last":0.86099999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":1.5309999999999999,"15min":1.0920000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.3719999999999999,"5min":20.774000000000001,"15min":20.774000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.7509999999999999,"15min":1.2190000000000001},"min":{"1min":0.30099999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.9009999999999998,"5min":20.817,"15min":20.817},"last":1.115}]},{"osd":3,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.575,"15min":1.131},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.1440000000000001,"5min":24.613,"15min":24.613},"last":1.1639999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.3300000000000001,"15min":1.113},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.3650000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":1.1850000000000001}]},{"osd":5,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.04,"15min":0.88500000000000001},"min":{"1min":0.35599999999999998,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":4.7759999999999998,"5min":6.1459999999999999,"15min":6.1459999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.0640000000000001,"15min":0.94799999999999995},"min":{"1min":0.246,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.5960000000000001,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.92800000000000005}]},{"osd":6,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.337,"15min":1.0740000000000001},"min":{"1min":0.36899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.048,"5min":6.5220000000000002,"15min":6.5220000000000002},"last":1.143},{"interface":"front","average":{"1min":1.2130000000000001,"5min":0.97599999999999998,"15min":0.94299999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":4.3819999999999997,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":1.2529999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:21:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.2450000000000001,"15min":1.0469999999999999},"min":{"1min":0.47399999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":6.3330000000000002,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.4279999999999999,"15min":1.1020000000000001},"min":{"1min":0.51000000000000001,"5min":0.309,"15min":0.309},"max":{"1min":6.3550000000000004,"5min":6.3550000000000004,"15min":6.3550000000000004},"last":1.2130000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575206,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160732,"kb_used_data":9236,"kb_used_omap":0,"kb_used_meta":1151488,"kb_avail":93211108,"statfs":{"total":96636764160,"available":95448174592,"internally_reserved":0,"allocated":9457664,"data_stored":9007793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1179123712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.996,"15min":0.78900000000000003},"min":{"1min":0.20000000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.018,"5min":13.327999999999999,"15min":13.327999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.61299999999999999,"15min":0.73399999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.0489999999999999,"5min":3.556,"15min":5.4580000000000002},"last":0.72699999999999998}]},{"osd":1,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.2290000000000001,"15min":0.86099999999999999},"min":{"1min":0.21199999999999999,"5min":0.20599999999999999,"15min":0.19900000000000001},"max":{"1min":1.796,"5min":13.831,"15min":13.831},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.80900000000000005,"15min":0.76200000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.9300000000000002,"5min":4.4500000000000002,"15min":4.4500000000000002},"last":0.79600000000000004}]},{"osd":2,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.92200000000000004,"15min":0.877},"min":{"1min":0.23799999999999999,"5min":0.223,"15min":0.221},"max":{"1min":2.7890000000000001,"5min":8.2520000000000007,"15min":8.2520000000000007},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.075,"15min":0.90700000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.7360000000000002,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.83799999999999997}]},{"osd":4,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.9630000000000001,"15min":1.2010000000000001},"min":{"1min":0.32100000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.0339999999999998,"5min":39.070999999999998,"15min":39.070999999999998},"last":1.208},{"interface":"front","average":{"1min":1.5389999999999999,"5min":2.0870000000000002,"15min":1.2509999999999999},"min":{"1min":0.34599999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.0300000000000002,"5min":55.104999999999997,"15min":55.104999999999997},"last":0.88400000000000001}]},{"osd":5,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.6379999999999999,"15min":1.117},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.1849999999999996,"5min":31.481999999999999,"15min":31.481999999999999},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.681,"15min":1.1970000000000001},"min":{"1min":0.42499999999999999,"5min":0.35299999999999998,"15min":0.32500000000000001},"max":{"1min":4.2060000000000004,"5min":31.512,"15min":31.512},"last":1.4179999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.9179999999999999,"5min":1.617,"15min":1.1910000000000001},"min":{"1min":0.311,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.7080000000000002,"5min":25.809999999999999,"15min":25.809999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.9790000000000001,"5min":1.4530000000000001,"15min":1.175},"min":{"1min":0.47299999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":4.7779999999999996,"5min":24.036999999999999,"15min":24.036999999999999},"last":0.91900000000000004}]},{"osd":7,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.704,"15min":1.226},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.8529999999999998,"5min":27.581,"15min":27.581},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.6379999999999999,"15min":1.24},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":8.5259999999999998,"5min":24.332000000000001,"15min":24.332000000000001},"last":1.349}]}]},{"osd":2,"up_from":13,"seq":55834575206,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946044,"kb_used_data":5044,"kb_used_omap":0,"kb_used_meta":940992,"kb_avail":93425796,"statfs":{"total":96636764160,"available":95668015104,"internally_reserved":0,"allocated":5165056,"data_stored":4728218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":963575808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.651,"15min":0.98399999999999999},"min":{"1min":0.188,"5min":0.16,"15min":0.16},"max":{"1min":2.9329999999999998,"5min":29.344000000000001,"15min":29.344000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.0800000000000001,"15min":0.82499999999999996},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.0539999999999998,"5min":10.755000000000001,"15min":10.755000000000001},"last":0.49099999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.3640000000000001,"15min":0.92200000000000004},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":2.8860000000000001,"5min":18.617999999999999,"15min":18.617999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":1.099,"15min":0.86099999999999999},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":2.1669999999999998,"5min":15.064,"15min":15.064},"last":0.56699999999999995}]},{"osd":3,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.1259999999999999,"15min":0.88800000000000001},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.192},"max":{"1min":1.766,"5min":13.034000000000001,"15min":13.034000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.5700000000000001,"15min":0.98099999999999998},"min":{"1min":0.26200000000000001,"5min":0.25,"15min":0.25},"max":{"1min":2.964,"5min":18.547999999999998,"15min":18.547999999999998},"last":0.38400000000000001}]},{"osd":4,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.45,"15min":1.1659999999999999},"min":{"1min":0.35099999999999998,"5min":0.252,"15min":0.252},"max":{"1min":2.839,"5min":22.997,"15min":33.378999999999998},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.5,"15min":1.2689999999999999},"min":{"1min":0.38500000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.8690000000000002,"5min":23.030999999999999,"15min":33.558999999999997},"last":0.78700000000000003}]},{"osd":5,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":2.2589999999999999,"15min":1.45},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":4.6440000000000001,"5min":25.183,"15min":34.545999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.038,"5min":1.3879999999999999,"15min":1.169},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.4829999999999997,"5min":11.685,"15min":36.104999999999997},"last":0.65700000000000003}]},{"osd":6,"last update":"Wed Oct 25 00:21:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.208,"15min":1.1100000000000001},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":3.5739999999999998,"5min":9.7149999999999999,"15min":33.661999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.26,"15min":1.2},"min":{"1min":0.45700000000000002,"5min":0.371,"15min":0.371},"max":{"1min":3.5979999999999999,"5min":9.6799999999999997,"15min":34.043999999999997},"last":1.0149999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.3,"15min":1.214},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.395,"5min":11.503,"15min":33.307000000000002},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.3420000000000001,"15min":1.276},"min":{"1min":0.51200000000000001,"5min":0.371,"15min":0.371},"max":{"1min":2.419,"5min":11.468999999999999,"15min":33.344999999999999},"last":0.89400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575207,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1070492,"kb_used_data":4052,"kb_used_omap":0,"kb_used_meta":1066432,"kb_avail":93301348,"statfs":{"total":96636764160,"available":95540580352,"internally_reserved":0,"allocated":4149248,"data_stored":3711479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1092026368},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.1739999999999999,"15min":0.81699999999999995},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.81000000000000005,"5min":21.334,"15min":21.334},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.0449999999999999,"15min":0.84099999999999997},"min":{"1min":0.20499999999999999,"5min":0.19,"15min":0.19},"max":{"1min":3.141,"5min":21.276,"15min":21.276},"last":0.84799999999999998}]},{"osd":2,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.071,"15min":0.80200000000000005},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":8.1600000000000001,"5min":11.420999999999999,"15min":11.420999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.90100000000000002,"15min":0.83399999999999996},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.1760000000000002,"5min":11.648,"15min":11.648},"last":0.78300000000000003}]},{"osd":3,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.22,"15min":0.91500000000000004},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":3.46,"5min":14.346,"15min":14.346},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.2789999999999999,"15min":1.016},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.248},"max":{"1min":3.4969999999999999,"5min":14.379,"15min":14.379},"last":0.99399999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.3959999999999999,"15min":1.0569999999999999},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.6500000000000004,"5min":10.582000000000001,"15min":10.582000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.49,"5min":1.3009999999999999,"15min":1.04},"min":{"1min":0.32600000000000001,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":4.6630000000000003,"5min":11.173,"15min":11.173},"last":1.0429999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.244,"15min":1.054},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.0129999999999999,"5min":11.577999999999999,"15min":11.577999999999999},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.0760000000000001,"15min":0.98099999999999998},"min":{"1min":0.29899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.3220000000000001,"5min":10.018000000000001,"15min":10.018000000000001},"last":1.0820000000000001}]},{"osd":6,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.1140000000000001,"15min":1.026},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.3610000000000002,"5min":9.2989999999999995,"15min":11.845000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.1699999999999999,"15min":1.109},"min":{"1min":0.38300000000000001,"5min":0.379,"15min":0.33800000000000002},"max":{"1min":2.3809999999999998,"5min":9.3350000000000009,"15min":11.919},"last":1.115}]},{"osd":7,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.498,"15min":1.125},"min":{"1min":0.38200000000000001,"5min":0.36199999999999999,"15min":0.35599999999999998},"max":{"1min":5.077,"5min":9.407,"15min":11.881},"last":1.02},{"interface":"front","average":{"1min":0.874,"5min":1.3089999999999999,"15min":1.137},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.9750000000000001,"5min":10.478,"15min":10.478},"last":1.1439999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575207,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":966360,"kb_used_data":22096,"kb_used_omap":0,"kb_used_meta":944256,"kb_avail":93405480,"statfs":{"total":96636764160,"available":95647211520,"internally_reserved":0,"allocated":22626304,"data_stored":22180994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":966918144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.4159999999999999,"15min":0.98399999999999999},"min":{"1min":0.19700000000000001,"5min":0.18099999999999999,"15min":0.16900000000000001},"max":{"1min":1.7290000000000001,"5min":30.065999999999999,"15min":30.065999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.2330000000000001,"15min":0.98799999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.182},"max":{"1min":3.0030000000000001,"5min":19.757999999999999,"15min":19.757999999999999},"last":0.76300000000000001}]},{"osd":2,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.96099999999999997,"15min":0.78200000000000003},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.9740000000000002,"5min":15.324999999999999,"15min":15.324999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.504,"5min":1.2050000000000001,"15min":0.94299999999999995},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.95699999999999996,"5min":30.283999999999999,"15min":30.283999999999999},"last":1.014}]},{"osd":3,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.5209999999999999,"15min":1.101},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.191},"max":{"1min":6.3570000000000002,"5min":22.966999999999999,"15min":22.966999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":1.417,"15min":1.048},"min":{"1min":0.29499999999999998,"5min":0.24299999999999999,"15min":0.23599999999999999},"max":{"1min":1.1000000000000001,"5min":19.460999999999999,"15min":19.460999999999999},"last":1.0629999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.2090000000000001,"15min":1.121},"min":{"1min":0.33700000000000002,"5min":0.318,"15min":0.26600000000000001},"max":{"1min":3.0720000000000001,"5min":5.5990000000000002,"15min":8.3670000000000009},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.375,"15min":1.1850000000000001},"min":{"1min":0.29499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.1749999999999998,"5min":13.023999999999999,"15min":13.023999999999999},"last":0.98899999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.23,"15min":1.006},"min":{"1min":0.33000000000000002,"5min":0.30399999999999999,"15min":0.29099999999999998},"max":{"1min":2.9710000000000001,"5min":10.683999999999999,"15min":10.683999999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.089,"15min":0.96999999999999997},"min":{"1min":0.378,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.5779999999999998,"5min":15.436999999999999,"15min":15.436999999999999},"last":1.173}]},{"osd":6,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.393,"15min":1.125},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.7469999999999999,"5min":8.7260000000000009,"15min":8.7260000000000009},"last":1.085},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.3500000000000001,"15min":1.248},"min":{"1min":0.38200000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.9039999999999999,"5min":15.375,"15min":15.375},"last":1.2250000000000001}]},{"osd":7,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.367,"15min":1.113},"min":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":5.0979999999999999,"5min":10.81,"15min":10.81},"last":1.036},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.3220000000000001,"15min":1.25},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":3.1499999999999999,"5min":11.417,"15min":11.417},"last":1.157}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12673024,"data_stored":12668140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3178496,"data_stored":3177545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11669504,"data_stored":11651527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4195192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":8388642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4279555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6918144,"data_stored":6907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6836224,"data_stored":6822812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-25T00:22:31.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-25T00:22:31.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e7b94640 1 Processor -- start 2023-10-25T00:22:31.855 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e7b94640 1 -- start start 2023-10-25T00:22:31.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e7b94640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f60e006af70 0x7f60e006c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.856 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e7b94640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f60e006a230 0x7f60e01551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e7b94640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e00c1860 con 0x7f60e014b600 2023-10-25T00:22:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e7b94640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e0154d50 con 0x7f60e006a230 2023-10-25T00:22:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e7b94640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e014b9e0 con 0x7f60e006af70 2023-10-25T00:22:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e6131640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f60e006a230 0x7f60e01551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.857 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e6131640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f60e006a230 0x7f60e01551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54160/0 (socket says 172.21.15.107:54160) 2023-10-25T00:22:31.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e6131640 1 -- 172.21.15.107:0/524988849 learned_addr learned my addr 172.21.15.107:0/524988849 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:31.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e512f640 1 --2- 172.21.15.107:0/524988849 >> v2:172.21.15.110:3300/0 conn(0x7f60e006af70 0x7f60e006c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e6131640 1 -- 172.21.15.107:0/524988849 >> v2:172.21.15.110:3300/0 conn(0x7f60e006af70 msgr2=0x7f60e006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:31.858 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e6131640 1 --2- 172.21.15.107:0/524988849 >> v2:172.21.15.110:3300/0 conn(0x7f60e006af70 0x7f60e006c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e6131640 1 -- 172.21.15.107:0/524988849 >> v1:172.21.15.110:6789/0 conn(0x7f60e014b600 legacy=0x7f60e0151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:31.859 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e6131640 1 -- 172.21.15.107:0/524988849 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e00c1ab0 con 0x7f60e006a230 2023-10-25T00:22:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.856+0000 7f60e6131640 1 --2- 172.21.15.107:0/524988849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60e006a230 0x7f60e01551e0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f60e00c2870 tx=0x7f60d801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27615cf1b4b51e00 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e492e640 1 -- 172.21.15.107:0/524988849 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d80a5790 con 0x7f60e006a230 2023-10-25T00:22:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e492e640 1 -- 172.21.15.107:0/524988849 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d80a58f0 con 0x7f60e006a230 2023-10-25T00:22:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e492e640 1 -- 172.21.15.107:0/524988849 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d80a5c80 con 0x7f60e006a230 2023-10-25T00:22:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 -- 172.21.15.107:0/524988849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60e006a230 msgr2=0x7f60e01551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 --2- 172.21.15.107:0/524988849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60e006a230 0x7f60e01551e0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f60e00c2870 tx=0x7f60d801c8d0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 -- 172.21.15.107:0/524988849 shutdown_connections 2023-10-25T00:22:31.861 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 --2- 172.21.15.107:0/524988849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60e006a230 0x7f60e01551e0 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 --2- 172.21.15.107:0/524988849 >> v2:172.21.15.110:3300/0 conn(0x7f60e006af70 0x7f60e006c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 -- 172.21.15.107:0/524988849 >> 172.21.15.107:0/524988849 conn(0x7f60e00c2950 msgr2=0x7f60e00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:31.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 -- 172.21.15.107:0/524988849 shutdown_connections 2023-10-25T00:22:31.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 -- 172.21.15.107:0/524988849 wait complete. 2023-10-25T00:22:31.862 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 Processor -- start 2023-10-25T00:22:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 -- start start 2023-10-25T00:22:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60e006a230 0x7f60e01edee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60e006af70 0x7f60e01f0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60e014b600 0x7f60e0151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e0072220 con 0x7f60e006a230 2023-10-25T00:22:31.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e0070cf0 con 0x7f60e014b600 2023-10-25T00:22:31.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e0070fd0 con 0x7f60e006af70 2023-10-25T00:22:31.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e5930640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60e006a230 0x7f60e01edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.866 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e5930640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60e006a230 0x7f60e01edee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54164/0 (socket says 172.21.15.107:54164) 2023-10-25T00:22:31.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e5930640 1 -- 172.21.15.107:0/3429524407 learned_addr learned my addr 172.21.15.107:0/3429524407 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:31.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e6131640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60e014b600 0x7f60e0151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e512f640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60e006af70 0x7f60e01f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e5930640 1 -- 172.21.15.107:0/3429524407 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60e006af70 msgr2=0x7f60e01f0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.867 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e5930640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60e006af70 0x7f60e01f0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e5930640 1 -- 172.21.15.107:0/3429524407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60e014b600 msgr2=0x7f60e0151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e5930640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60e014b600 0x7f60e0151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e5930640 1 -- 172.21.15.107:0/3429524407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d8009b30 con 0x7f60e006a230 2023-10-25T00:22:31.868 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e512f640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60e006af70 0x7f60e01f0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:31.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e6131640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60e014b600 0x7f60e0151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:31.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e5930640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60e006a230 0x7f60e01edee0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00b410 tx=0x7f60dc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:31.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60d67fc640 1 -- 172.21.15.107:0/3429524407 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc004020 con 0x7f60e006a230 2023-10-25T00:22:31.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60d67fc640 1 -- 172.21.15.107:0/3429524407 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60dc004180 con 0x7f60e006a230 2023-10-25T00:22:31.869 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 -- 172.21.15.107:0/3429524407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e01ed6e0 con 0x7f60e006a230 2023-10-25T00:22:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60d67fc640 1 -- 172.21.15.107:0/3429524407 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc011630 con 0x7f60e006a230 2023-10-25T00:22:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.860+0000 7f60e7b94640 1 -- 172.21.15.107:0/3429524407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e01ed390 con 0x7f60e006a230 2023-10-25T00:22:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.864+0000 7f60e7b94640 1 -- 172.21.15.107:0/3429524407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e00746e0 con 0x7f60e006a230 2023-10-25T00:22:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.864+0000 7f60d67fc640 1 -- 172.21.15.107:0/3429524407 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f60dc011790 con 0x7f60e006a230 2023-10-25T00:22:31.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.864+0000 7f60d67fc640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60c4065e60 0x7f60c4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:31.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.864+0000 7f60d67fc640 1 -- 172.21.15.107:0/3429524407 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f60dc0847c0 con 0x7f60e006a230 2023-10-25T00:22:31.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.864+0000 7f60e512f640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60c4065e60 0x7f60c4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:31.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.864+0000 7f60e512f640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60c4065e60 0x7f60c4068320 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f60d0006fd0 tx=0x7f60d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:31.871 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:31.868+0000 7f60d67fc640 1 -- 172.21.15.107:0/3429524407 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f60dc050c00 con 0x7f60e006a230 2023-10-25T00:22:32.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.024+0000 7f60e7b94640 1 -- 172.21.15.107:0/3429524407 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60e00c0310 con 0x7f60e006a230 2023-10-25T00:22:32.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.028+0000 7f60d67fc640 1 -- 172.21.15.107:0/3429524407 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+27395 (secure 0 0 0) 0x7f60dc015090 con 0x7f60e006a230 2023-10-25T00:22:32.029 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:32.029 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":72,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-25T00:21:39.173159+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2f","mappings":[{"from":3,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":2}]},{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6835/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6837/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.110:6834/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.110:6832/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.107:6835/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.107:6837/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.110:6834/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.110:6832/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.107:6835/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.107:6834/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.110:6833/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6836/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6836/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.107:6834/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6835/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6836/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.110:6834/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6835/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.110:6835/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6834/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.107:6837/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.107:6837/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6832/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6833/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.110:6833/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6834/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.110:6835/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6836/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6836/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6834/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.110:6835/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.107:6834/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.107:6834/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.110:6833/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.107:6837/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6834/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.110:6833/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.110:6832/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.107:6836/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6837/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.107:6837/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.110:6833/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6837/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.107:6835/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.107:6837/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6837/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6835/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.110:6835/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.107:6836/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.107:6834/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.110:6832/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.107:6835/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6835/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.110:6832/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.110:6833/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6836/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6834/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6836/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.110:6834/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6837/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6836/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.110:6835/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6834/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6835/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6835/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.110:6832/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6835/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.107:6836/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6833/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6835/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.107:6836/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6834/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.107:6836/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6836/1788268787":"2023-10-26T00:10:38.784519+0000","172.21.15.110:6835/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.107:6837/1788268787":"2023-10-26T00:10:38.784519+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-25T00:22:32.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 -- 172.21.15.107:0/3429524407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60c4065e60 msgr2=0x7f60c4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:32.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60c4065e60 0x7f60c4068320 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f60d0006fd0 tx=0x7f60d0008040 comp rx=0 tx=0).stop 2023-10-25T00:22:32.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 -- 172.21.15.107:0/3429524407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60e006a230 msgr2=0x7f60e01edee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:32.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60e006a230 0x7f60e01edee0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00b410 tx=0x7f60dc00b8d0 comp rx=0 tx=0).stop 2023-10-25T00:22:32.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 -- 172.21.15.107:0/3429524407 shutdown_connections 2023-10-25T00:22:32.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f60e014b600 0x7f60e0151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:32.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f60c4065e60 0x7f60c4068320 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:32.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f60e006af70 0x7f60e01f0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:32.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 --2- 172.21.15.107:0/3429524407 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60e006a230 0x7f60e01edee0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:32.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 -- 172.21.15.107:0/3429524407 >> 172.21.15.107:0/3429524407 conn(0x7f60e00c2950 msgr2=0x7f60e014a070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:32.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 -- 172.21.15.107:0/3429524407 shutdown_connections 2023-10-25T00:22:32.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:32.032+0000 7f60e7b94640 1 -- 172.21.15.107:0/3429524407 wait complete. 2023-10-25T00:22:33.050 INFO:tasks.ceph:Scrubbing osd.0 2023-10-25T00:22:33.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-25T00:22:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f616752e640 1 Processor -- start 2023-10-25T00:22:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f616752e640 1 -- start start 2023-10-25T00:22:33.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f616752e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f616014f5e0 0x7f6160154b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f616752e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61600c1d10 con 0x7f61600c56e0 2023-10-25T00:22:33.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f616752e640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61600a8e60 con 0x7f61600a5300 2023-10-25T00:22:33.191 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f616752e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6160069f90 con 0x7f616014f5e0 2023-10-25T00:22:33.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6164ac9640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f61600c56e0 0x7f6160154460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35236/0 (socket says 172.21.15.107:35236) 2023-10-25T00:22:33.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6164ac9640 1 -- 172.21.15.107:0/3943055526 learned_addr learned my addr 172.21.15.107:0/3943055526 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:33.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6165acb640 1 --2- 172.21.15.107:0/3943055526 >> v2:172.21.15.110:3300/0 conn(0x7f616014f5e0 0x7f6160154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197592194 0 0) 0x7f61600c1d10 con 0x7f61600c56e0 2023-10-25T00:22:33.192 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f614c004640 con 0x7f61600c56e0 2023-10-25T00:22:33.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 654063234 0 0) 0x7f614c004640 con 0x7f61600c56e0 2023-10-25T00:22:33.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 >> v2:172.21.15.110:3300/0 conn(0x7f616014f5e0 msgr2=0x7f6160154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:33.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 --2- 172.21.15.107:0/3943055526 >> v2:172.21.15.110:3300/0 conn(0x7f616014f5e0 0x7f6160154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 >> v1:172.21.15.110:6789/0 conn(0x7f61600a5300 legacy=0x7f616014eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.193 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616014fa30 con 0x7f61600c56e0 2023-10-25T00:22:33.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6150001680 con 0x7f61600c56e0 2023-10-25T00:22:33.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 >> v1:172.21.15.107:6789/0 conn(0x7f61600c56e0 legacy=0x7f6160154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 --2- 172.21.15.107:0/3943055526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f614c004a90 0x7f614c004eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.194 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 --2- 172.21.15.107:0/3943055526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f614c0058e0 0x7f614c005d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 --2- 172.21.15.107:0/3943055526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 0x7f614c00caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6160069f90 con 0x7f614c0062a0 2023-10-25T00:22:33.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c00d240 con 0x7f614c004a90 2023-10-25T00:22:33.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614c0035b0 con 0x7f614c0058e0 2023-10-25T00:22:33.195 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6165acb640 1 --2- 172.21.15.107:0/3943055526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 0x7f614c00caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f61652ca640 1 --2- 172.21.15.107:0/3943055526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f614c004a90 0x7f614c004eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.188+0000 7f6164ac9640 1 --2- 172.21.15.107:0/3943055526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f614c0058e0 0x7f614c005d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.196 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f6165acb640 1 -- 172.21.15.107:0/3943055526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f614c0058e0 msgr2=0x7f614c005d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:33.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f6165acb640 1 --2- 172.21.15.107:0/3943055526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f614c0058e0 0x7f614c005d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f6165acb640 1 -- 172.21.15.107:0/3943055526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f614c004a90 msgr2=0x7f614c004eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:33.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f6165acb640 1 --2- 172.21.15.107:0/3943055526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f614c004a90 0x7f614c004eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.197 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f6165acb640 1 -- 172.21.15.107:0/3943055526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c004640 con 0x7f614c0062a0 2023-10-25T00:22:33.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f6165acb640 1 -- 172.21.15.107:0/3943055526 reap_dead start 2023-10-25T00:22:33.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f6165acb640 1 --2- 172.21.15.107:0/3943055526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 0x7f614c00caa0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f6158009fc0 tx=0x7f615801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea034550516dbb4c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6158003ba0 con 0x7f614c0062a0 2023-10-25T00:22:33.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615800f040 con 0x7f614c0062a0 2023-10-25T00:22:33.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f6157fff640 1 -- 172.21.15.107:0/3943055526 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61580a59a0 con 0x7f614c0062a0 2023-10-25T00:22:33.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f616752e640 1 -- 172.21.15.107:0/3943055526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 msgr2=0x7f614c00caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f616752e640 1 --2- 172.21.15.107:0/3943055526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 0x7f614c00caa0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f6158009fc0 tx=0x7f615801cca0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f616752e640 1 -- 172.21.15.107:0/3943055526 shutdown_connections 2023-10-25T00:22:33.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f616752e640 1 --2- 172.21.15.107:0/3943055526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 0x7f614c00caa0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f616752e640 1 -- 172.21.15.107:0/3943055526 >> 172.21.15.107:0/3943055526 conn(0x7f61600c27f0 msgr2=0x7f61600c2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:33.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f616752e640 1 -- 172.21.15.107:0/3943055526 shutdown_connections 2023-10-25T00:22:33.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.192+0000 7f616752e640 1 -- 172.21.15.107:0/3943055526 wait complete. 2023-10-25T00:22:33.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f616752e640 1 Processor -- start 2023-10-25T00:22:33.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f616752e640 1 -- start start 2023-10-25T00:22:33.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f616752e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 0x7f61601e5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f616752e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f616006ec60 0x7f6160153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f616752e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6160071110 0x7f61601dc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f616752e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61601e4e90 con 0x7f614c0062a0 2023-10-25T00:22:33.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f616752e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6160071b60 con 0x7f616006ec60 2023-10-25T00:22:33.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f616752e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6160071e40 con 0x7f6160071110 2023-10-25T00:22:33.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f61652ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 0x7f61601e5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f61652ca640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 0x7f61601e5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45360/0 (socket says 172.21.15.107:45360) 2023-10-25T00:22:33.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f61652ca640 1 -- 172.21.15.107:0/4009134996 learned_addr learned my addr 172.21.15.107:0/4009134996 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:33.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f6165acb640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6160071110 0x7f61601dc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f6164ac9640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f616006ec60 0x7f6160153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f61652ca640 1 -- 172.21.15.107:0/4009134996 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6160071110 msgr2=0x7f61601dc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f61652ca640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6160071110 0x7f61601dc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f61652ca640 1 -- 172.21.15.107:0/4009134996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f616006ec60 msgr2=0x7f6160153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f61652ca640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f616006ec60 0x7f6160153b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f61652ca640 1 -- 172.21.15.107:0/4009134996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61601e5150 con 0x7f614c0062a0 2023-10-25T00:22:33.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f61652ca640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 0x7f61601e5950 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f615c00e440 tx=0x7f615c00e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f6155ffb640 1 -- 172.21.15.107:0/4009134996 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c0052e0 con 0x7f614c0062a0 2023-10-25T00:22:33.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f6155ffb640 1 -- 172.21.15.107:0/4009134996 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c005440 con 0x7f614c0062a0 2023-10-25T00:22:33.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f616752e640 1 -- 172.21.15.107:0/4009134996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61601e5430 con 0x7f614c0062a0 2023-10-25T00:22:33.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f6155ffb640 1 -- 172.21.15.107:0/4009134996 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c003820 con 0x7f614c0062a0 2023-10-25T00:22:33.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f616752e640 1 -- 172.21.15.107:0/4009134996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6160154060 con 0x7f614c0062a0 2023-10-25T00:22:33.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f616752e640 1 -- 172.21.15.107:0/4009134996 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f61600c13e0 con 0x7f614c0062a0 2023-10-25T00:22:33.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f6155ffb640 1 -- 172.21.15.107:0/4009134996 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f615c003980 con 0x7f614c0062a0 2023-10-25T00:22:33.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f6155ffb640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6144065e60 0x7f6144068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f6155ffb640 1 -- 172.21.15.107:0/4009134996 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f615c085120 con 0x7f614c0062a0 2023-10-25T00:22:33.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.196+0000 7f6164ac9640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6144065e60 0x7f6144068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.200+0000 7f6155ffb640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] conn(0x7f6144073100 0x7f6144075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.200+0000 7f6155ffb640 1 -- 172.21.15.107:0/4009134996 --> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6144075c30 con 0x7f6144073100 2023-10-25T00:22:33.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.200+0000 7f6155ffb640 1 -- 172.21.15.107:0/4009134996 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f615c085530 con 0x7f614c0062a0 2023-10-25T00:22:33.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.200+0000 7f6165acb640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] conn(0x7f6144073100 0x7f6144075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.200+0000 7f6164ac9640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6144065e60 0x7f6144068320 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6150001210 tx=0x7f6150001160 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.200+0000 7f6165acb640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] conn(0x7f6144073100 0x7f6144075560 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.200+0000 7f6155ffb640 1 -- 172.21.15.107:0/4009134996 <== osd.0 v2:172.21.15.107:6810/37540 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6144075c30 con 0x7f6144073100 2023-10-25T00:22:33.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f616752e640 1 -- 172.21.15.107:0/4009134996 --> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f61601dcb20 con 0x7f6144073100 2023-10-25T00:22:33.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f6155ffb640 1 -- 172.21.15.107:0/4009134996 <== osd.0 v2:172.21.15.107:6810/37540 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f61601dcb20 con 0x7f6144073100 2023-10-25T00:22:33.214 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-10-25T00:22:33.214 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-25T00:22:33.214 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-10-25T00:22:33.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f616752e640 1 -- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] conn(0x7f6144073100 msgr2=0x7f6144075560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f616752e640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:6810/37540,v1:172.21.15.107:6813/37540] conn(0x7f6144073100 0x7f6144075560 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f616752e640 1 -- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6144065e60 msgr2=0x7f6144068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f616752e640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6144065e60 0x7f6144068320 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6150001210 tx=0x7f6150001160 comp rx=0 tx=0).stop 2023-10-25T00:22:33.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f616752e640 1 -- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 msgr2=0x7f61601e5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f616752e640 1 --2- 172.21.15.107:0/4009134996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f614c0062a0 0x7f61601e5950 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f615c00e440 tx=0x7f615c00e900 comp rx=0 tx=0).stop 2023-10-25T00:22:33.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f6165acb640 1 -- 172.21.15.107:0/4009134996 reap_dead start 2023-10-25T00:22:33.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f616752e640 1 -- 172.21.15.107:0/4009134996 shutdown_connections 2023-10-25T00:22:33.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f616752e640 1 -- 172.21.15.107:0/4009134996 >> 172.21.15.107:0/4009134996 conn(0x7f61600c27f0 msgr2=0x7f61600c2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:33.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f616752e640 1 -- 172.21.15.107:0/4009134996 shutdown_connections 2023-10-25T00:22:33.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.212+0000 7f616752e640 1 -- 172.21.15.107:0/4009134996 wait complete. 2023-10-25T00:22:33.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-25T00:22:33.351 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff37d2ff640 1 Processor -- start 2023-10-25T00:22:33.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff37d2ff640 1 -- start start 2023-10-25T00:22:33.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff37d2ff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff378071540 0x7ff3780a9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff37d2ff640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff378154cb0 con 0x7ff378157f50 2023-10-25T00:22:33.353 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff37d2ff640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff378154f90 con 0x7ff37813dc60 2023-10-25T00:22:33.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff37d2ff640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff378155270 con 0x7ff378071540 2023-10-25T00:22:33.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff3767fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff378071540 0x7ff3780a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff3767fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff378071540 0x7ff3780a9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45370/0 (socket says 172.21.15.107:45370) 2023-10-25T00:22:33.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff3767fc640 1 -- 172.21.15.107:0/2387506298 learned_addr learned my addr 172.21.15.107:0/2387506298 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:33.354 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff3767fc640 1 -- 172.21.15.107:0/2387506298 >> v1:172.21.15.110:6789/0 conn(0x7ff37813dc60 legacy=0x7ff3780c28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff3767fc640 1 -- 172.21.15.107:0/2387506298 >> v1:172.21.15.107:6789/0 conn(0x7ff378157f50 legacy=0x7ff3780a9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff3767fc640 1 -- 172.21.15.107:0/2387506298 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff378155550 con 0x7ff378071540 2023-10-25T00:22:33.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.352+0000 7ff375ffb640 1 -- 172.21.15.107:0/2387506298 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428153122 0 0) 0x7ff378154cb0 con 0x7ff378157f50 2023-10-25T00:22:33.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff3767fc640 1 --2- 172.21.15.107:0/2387506298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff378071540 0x7ff3780a9450 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009e50 tx=0x7ff36c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d634d24e72e66267 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff375ffb640 1 -- 172.21.15.107:0/2387506298 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c0a5900 con 0x7ff378071540 2023-10-25T00:22:33.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff375ffb640 1 -- 172.21.15.107:0/2387506298 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36c0a5a60 con 0x7ff378071540 2023-10-25T00:22:33.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff375ffb640 1 -- 172.21.15.107:0/2387506298 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c0a5df0 con 0x7ff378071540 2023-10-25T00:22:33.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 -- 172.21.15.107:0/2387506298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff378071540 msgr2=0x7ff3780a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 --2- 172.21.15.107:0/2387506298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff378071540 0x7ff3780a9450 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009e50 tx=0x7ff36c01ca10 comp rx=0 tx=0).stop 2023-10-25T00:22:33.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 -- 172.21.15.107:0/2387506298 shutdown_connections 2023-10-25T00:22:33.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 --2- 172.21.15.107:0/2387506298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff378071540 0x7ff3780a9450 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 -- 172.21.15.107:0/2387506298 >> 172.21.15.107:0/2387506298 conn(0x7ff3780c39e0 msgr2=0x7ff3780c3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:33.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 -- 172.21.15.107:0/2387506298 shutdown_connections 2023-10-25T00:22:33.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 -- 172.21.15.107:0/2387506298 wait complete. 2023-10-25T00:22:33.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 Processor -- start 2023-10-25T00:22:33.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 -- start start 2023-10-25T00:22:33.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff378071540 0x7ff3781edbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37813dc60 0x7ff378153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff378157f50 0x7ff3781375e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3780be2f0 con 0x7ff37813dc60 2023-10-25T00:22:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3780bd330 con 0x7ff378071540 2023-10-25T00:22:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff37d2ff640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3780bd050 con 0x7ff378157f50 2023-10-25T00:22:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff3767fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37813dc60 0x7ff378153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff3767fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37813dc60 0x7ff378153180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45384/0 (socket says 172.21.15.107:45384) 2023-10-25T00:22:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff3767fc640 1 -- 172.21.15.107:0/4218702656 learned_addr learned my addr 172.21.15.107:0/4218702656 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff376ffd640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff378071540 0x7ff3781edbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff3767fc640 1 -- 172.21.15.107:0/4218702656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff378157f50 msgr2=0x7ff3781375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff3777fe640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff378157f50 0x7ff3781375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff3767fc640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff378157f50 0x7ff3781375e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff3767fc640 1 -- 172.21.15.107:0/4218702656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff378071540 msgr2=0x7ff3781edbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff3767fc640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff378071540 0x7ff3781edbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.356+0000 7ff3767fc640 1 -- 172.21.15.107:0/4218702656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c009b30 con 0x7ff37813dc60 2023-10-25T00:22:33.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff376ffd640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff378071540 0x7ff3781edbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:33.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff3767fc640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37813dc60 0x7ff378153180 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7ff36c0af040 tx=0x7ff36c0197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff35f7fe640 1 -- 172.21.15.107:0/4218702656 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c0b8880 con 0x7ff37813dc60 2023-10-25T00:22:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff35f7fe640 1 -- 172.21.15.107:0/4218702656 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36c0b8a20 con 0x7ff37813dc60 2023-10-25T00:22:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff37d2ff640 1 -- 172.21.15.107:0/4218702656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3781ed0d0 con 0x7ff37813dc60 2023-10-25T00:22:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff35f7fe640 1 -- 172.21.15.107:0/4218702656 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36c0b8dd0 con 0x7ff37813dc60 2023-10-25T00:22:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff37d2ff640 1 -- 172.21.15.107:0/4218702656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3781ed690 con 0x7ff37813dc60 2023-10-25T00:22:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff37d2ff640 1 -- 172.21.15.107:0/4218702656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff378073dc0 con 0x7ff37813dc60 2023-10-25T00:22:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff35f7fe640 1 -- 172.21.15.107:0/4218702656 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff36c0b6440 con 0x7ff37813dc60 2023-10-25T00:22:33.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff35f7fe640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff364065e60 0x7ff364068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff376ffd640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff364065e60 0x7ff364068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff35f7fe640 1 -- 172.21.15.107:0/4218702656 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7ff36c1202d0 con 0x7ff37813dc60 2023-10-25T00:22:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.360+0000 7ff376ffd640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff364065e60 0x7ff364068320 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ff370000f70 tx=0x7ff370009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.364+0000 7ff35f7fe640 1 -- 172.21.15.107:0/4218702656 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff36c0ec710 con 0x7ff37813dc60 2023-10-25T00:22:33.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.512+0000 7ff37d2ff640 1 -- 172.21.15.107:0/4218702656 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7ff37813e040 con 0x7ff364065e60 2023-10-25T00:22:33.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.512+0000 7ff35f7fe640 1 -- 172.21.15.107:0/4218702656 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff370001cc0 con 0x7ff364065e60 2023-10-25T00:22:33.514 INFO:teuthology.orchestra.run.smithi107.stderr:instructed osd(s) 0 to deep-scrub 2023-10-25T00:22:33.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 -- 172.21.15.107:0/4218702656 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff364065e60 msgr2=0x7ff364068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff364065e60 0x7ff364068320 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ff370000f70 tx=0x7ff370009040 comp rx=0 tx=0).stop 2023-10-25T00:22:33.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 -- 172.21.15.107:0/4218702656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37813dc60 msgr2=0x7ff378153180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37813dc60 0x7ff378153180 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7ff36c0af040 tx=0x7ff36c0197b0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 -- 172.21.15.107:0/4218702656 shutdown_connections 2023-10-25T00:22:33.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff364065e60 0x7ff364068320 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff378157f50 0x7ff3781375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37813dc60 0x7ff378153180 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 --2- 172.21.15.107:0/4218702656 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff378071540 0x7ff3781edbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 -- 172.21.15.107:0/4218702656 >> 172.21.15.107:0/4218702656 conn(0x7ff3780c39e0 msgr2=0x7ff3780c28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:33.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 -- 172.21.15.107:0/4218702656 shutdown_connections 2023-10-25T00:22:33.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.516+0000 7ff37d2ff640 1 -- 172.21.15.107:0/4218702656 wait complete. 2023-10-25T00:22:33.532 INFO:tasks.ceph:Scrubbing osd.1 2023-10-25T00:22:33.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-25T00:22:33.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b3e958640 1 Processor -- start 2023-10-25T00:22:33.651 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b3e958640 1 -- start start 2023-10-25T00:22:33.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b3e958640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b380a9370 0x7f5b380c1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b3e958640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b38151960 con 0x7f5b380a8d60 2023-10-25T00:22:33.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b3e958640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b38151c40 con 0x7f5b380c1800 2023-10-25T00:22:33.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b3e958640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b38151f20 con 0x7f5b380a9370 2023-10-25T00:22:33.654 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b377fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f5b380a8d60 0x7f5b38159e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35254/0 (socket says 172.21.15.107:35254) 2023-10-25T00:22:33.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b377fe640 1 -- 172.21.15.107:0/2254978384 learned_addr learned my addr 172.21.15.107:0/2254978384 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:33.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b3cef5640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b380a9370 0x7f5b380c1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997676081 0 0) 0x7f5b38151960 con 0x7f5b380a8d60 2023-10-25T00:22:33.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b24003530 con 0x7f5b380a8d60 2023-10-25T00:22:33.655 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2575334622 0 0) 0x7f5b24003530 con 0x7f5b380a8d60 2023-10-25T00:22:33.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b380a9370 msgr2=0x7f5b380c1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:33.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b380a9370 0x7f5b380c1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 >> v1:172.21.15.110:6789/0 conn(0x7f5b380c1800 legacy=0x7f5b38154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b38152200 con 0x7f5b380a8d60 2023-10-25T00:22:33.656 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f5b20001680 con 0x7f5b380a8d60 2023-10-25T00:22:33.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b3cef5640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b380a9370 0x7f5b380c1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:22:33.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 >> v1:172.21.15.107:6789/0 conn(0x7f5b380a8d60 legacy=0x7f5b38159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b240039b0 0x7f5b24003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b380a9370 0x7f5b240047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b24004d10 0x7f5b24005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b38151f20 con 0x7f5b240039b0 2023-10-25T00:22:33.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2400bb20 con 0x7f5b24004d10 2023-10-25T00:22:33.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2400bce0 con 0x7f5b380a9370 2023-10-25T00:22:33.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b37fff640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b240039b0 0x7f5b24003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b377fe640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b380a9370 0x7f5b240047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b3cef5640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b24004d10 0x7f5b24005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b37fff640 1 -- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b380a9370 msgr2=0x7f5b240047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b37fff640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b380a9370 0x7f5b240047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.659 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b37fff640 1 -- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b24004d10 msgr2=0x7f5b24005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b37fff640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b24004d10 0x7f5b24005190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.652+0000 7f5b37fff640 1 -- 172.21.15.107:0/2254978384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b28009420 con 0x7f5b240039b0 2023-10-25T00:22:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b37fff640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b240039b0 0x7f5b24003dd0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f5b30001e70 tx=0x7f5b3001c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b691b56dd5f88442 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b30003b80 con 0x7f5b240039b0 2023-10-25T00:22:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b30003ce0 con 0x7f5b240039b0 2023-10-25T00:22:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b36ffd640 1 -- 172.21.15.107:0/2254978384 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b30005310 con 0x7f5b240039b0 2023-10-25T00:22:33.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 -- 172.21.15.107:0/2254978384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b240039b0 msgr2=0x7f5b24003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 --2- 172.21.15.107:0/2254978384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b240039b0 0x7f5b24003dd0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f5b30001e70 tx=0x7f5b3001c760 comp rx=0 tx=0).stop 2023-10-25T00:22:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3cef5640 1 -- 172.21.15.107:0/2254978384 reap_dead start 2023-10-25T00:22:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 -- 172.21.15.107:0/2254978384 shutdown_connections 2023-10-25T00:22:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 -- 172.21.15.107:0/2254978384 >> 172.21.15.107:0/2254978384 conn(0x7f5b380c27f0 msgr2=0x7f5b380c2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 -- 172.21.15.107:0/2254978384 shutdown_connections 2023-10-25T00:22:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 -- 172.21.15.107:0/2254978384 wait complete. 2023-10-25T00:22:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 Processor -- start 2023-10-25T00:22:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 -- start start 2023-10-25T00:22:33.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b380714c0 0x7f5b381ee990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b380718a0 0x7f5b38155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b381ebcc0 0x7f5b3814e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b38072520 con 0x7f5b380718a0 2023-10-25T00:22:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b380bdad0 con 0x7f5b380714c0 2023-10-25T00:22:33.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3e958640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b380bddb0 con 0x7f5b381ebcc0 2023-10-25T00:22:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b380718a0 0x7f5b38155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b380718a0 0x7f5b38155b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45398/0 (socket says 172.21.15.107:45398) 2023-10-25T00:22:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b377fe640 1 -- 172.21.15.107:0/4946121 learned_addr learned my addr 172.21.15.107:0/4946121 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b37fff640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b380714c0 0x7f5b381ee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3cef5640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b381ebcc0 0x7f5b3814e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b377fe640 1 -- 172.21.15.107:0/4946121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b381ebcc0 msgr2=0x7f5b3814e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b377fe640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b381ebcc0 0x7f5b3814e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b377fe640 1 -- 172.21.15.107:0/4946121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b380714c0 msgr2=0x7f5b381ee990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b377fe640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b380714c0 0x7f5b381ee990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b377fe640 1 -- 172.21.15.107:0/4946121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b380be090 con 0x7f5b380718a0 2023-10-25T00:22:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b3cef5640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5b381ebcc0 0x7f5b3814e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b37fff640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b380714c0 0x7f5b381ee990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.656+0000 7f5b377fe640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b380718a0 0x7f5b38155b00 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000eb20 tx=0x7f5b20001600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b34ff9640 1 -- 172.21.15.107:0/4946121 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2000c8d0 con 0x7f5b380718a0 2023-10-25T00:22:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b34ff9640 1 -- 172.21.15.107:0/4946121 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2000ca50 con 0x7f5b380718a0 2023-10-25T00:22:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b3e958640 1 -- 172.21.15.107:0/4946121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b381eb7a0 con 0x7f5b380718a0 2023-10-25T00:22:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b34ff9640 1 -- 172.21.15.107:0/4946121 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2000ce00 con 0x7f5b380718a0 2023-10-25T00:22:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b3e958640 1 -- 172.21.15.107:0/4946121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b380bd710 con 0x7f5b380718a0 2023-10-25T00:22:33.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b3e958640 1 -- 172.21.15.107:0/4946121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5b38155980 con 0x7f5b380718a0 2023-10-25T00:22:33.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b34ff9640 1 -- 172.21.15.107:0/4946121 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5b20026430 con 0x7f5b380718a0 2023-10-25T00:22:33.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b34ff9640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b18065e90 0x7f5b18068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b34ff9640 1 -- 172.21.15.107:0/4946121 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f5b20085150 con 0x7f5b380718a0 2023-10-25T00:22:33.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b37fff640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b18065e90 0x7f5b18068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b34ff9640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] conn(0x7f5b18073130 0x7f5b18075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b34ff9640 1 -- 172.21.15.107:0/4946121 --> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b18075c60 con 0x7f5b18073130 2023-10-25T00:22:33.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b34ff9640 1 -- 172.21.15.107:0/4946121 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b200854c0 con 0x7f5b380718a0 2023-10-25T00:22:33.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b3cef5640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] conn(0x7f5b18073130 0x7f5b18075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b37fff640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b18065e90 0x7f5b18068350 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5b300ad000 tx=0x7f5b30001520 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b3cef5640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] conn(0x7f5b18073130 0x7f5b18075590 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.660+0000 7f5b34ff9640 1 -- 172.21.15.107:0/4946121 <== osd.1 v2:172.21.15.107:6800/37537 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f5b18075c60 con 0x7f5b18073130 2023-10-25T00:22:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3e958640 1 -- 172.21.15.107:0/4946121 --> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5b380be220 con 0x7f5b18073130 2023-10-25T00:22:33.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b34ff9640 1 -- 172.21.15.107:0/4946121 <== osd.1 v2:172.21.15.107:6800/37537 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5b380be220 con 0x7f5b18073130 2023-10-25T00:22:33.675 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-10-25T00:22:33.675 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-25T00:22:33.675 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-10-25T00:22:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3e958640 1 -- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] conn(0x7f5b18073130 msgr2=0x7f5b18075590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3e958640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:6800/37537,v1:172.21.15.107:6801/37537] conn(0x7f5b18073130 0x7f5b18075590 crc :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3e958640 1 -- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b18065e90 msgr2=0x7f5b18068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3e958640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b18065e90 0x7f5b18068350 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5b300ad000 tx=0x7f5b30001520 comp rx=0 tx=0).stop 2023-10-25T00:22:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3e958640 1 -- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b380718a0 msgr2=0x7f5b38155b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3e958640 1 --2- 172.21.15.107:0/4946121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b380718a0 0x7f5b38155b00 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000eb20 tx=0x7f5b20001600 comp rx=0 tx=0).stop 2023-10-25T00:22:33.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3cef5640 1 -- 172.21.15.107:0/4946121 reap_dead start 2023-10-25T00:22:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3e958640 1 -- 172.21.15.107:0/4946121 shutdown_connections 2023-10-25T00:22:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3e958640 1 -- 172.21.15.107:0/4946121 >> 172.21.15.107:0/4946121 conn(0x7f5b380c27f0 msgr2=0x7f5b380aaf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3e958640 1 -- 172.21.15.107:0/4946121 shutdown_connections 2023-10-25T00:22:33.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.672+0000 7f5b3e958640 1 -- 172.21.15.107:0/4946121 wait complete. 2023-10-25T00:22:33.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-25T00:22:33.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.796+0000 7f35a2825640 1 Processor -- start 2023-10-25T00:22:33.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.796+0000 7f35a2825640 1 -- start start 2023-10-25T00:22:33.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f359c072720 0x7f359c0c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f359c0bceb0 0x7f359c15c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359c0c1ce0 con 0x7f359c0ab1a0 2023-10-25T00:22:33.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359c0be1d0 con 0x7f359c072720 2023-10-25T00:22:33.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c0c0240 con 0x7f359c0bceb0 2023-10-25T00:22:33.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f359c072720 0x7f359c0c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f359c072720 0x7f359c0c0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45404/0 (socket says 172.21.15.107:45404) 2023-10-25T00:22:33.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359b7fe640 1 -- 172.21.15.107:0/634415005 learned_addr learned my addr 172.21.15.107:0/634415005 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:33.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a0dc2640 1 --2- 172.21.15.107:0/634415005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f359c0bceb0 0x7f359c15c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359b7fe640 1 -- 172.21.15.107:0/634415005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f359c0bceb0 msgr2=0x7f359c15c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:33.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359b7fe640 1 --2- 172.21.15.107:0/634415005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f359c0bceb0 0x7f359c15c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359b7fe640 1 -- 172.21.15.107:0/634415005 >> v1:172.21.15.107:6789/0 conn(0x7f359c0ab1a0 legacy=0x7f359c0bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359affd640 1 -- 172.21.15.107:0/634415005 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821287378 0 0) 0x7f359c0c1ce0 con 0x7f359c0ab1a0 2023-10-25T00:22:33.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359b7fe640 1 -- 172.21.15.107:0/634415005 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c0c0880 con 0x7f359c072720 2023-10-25T00:22:33.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a0dc2640 1 --2- 172.21.15.107:0/634415005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f359c0bceb0 0x7f359c15c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:33.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359b7fe640 1 --2- 172.21.15.107:0/634415005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f359c072720 0x7f359c0c0d00 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f3590009870 tx=0x7f359001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8b6ae66e049de0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359affd640 1 -- 172.21.15.107:0/634415005 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35900ab5f0 con 0x7f359c072720 2023-10-25T00:22:33.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359affd640 1 -- 172.21.15.107:0/634415005 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35900ab750 con 0x7f359c072720 2023-10-25T00:22:33.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f359affd640 1 -- 172.21.15.107:0/634415005 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35900abae0 con 0x7f359c072720 2023-10-25T00:22:33.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 -- 172.21.15.107:0/634415005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f359c072720 msgr2=0x7f359c0c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 --2- 172.21.15.107:0/634415005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f359c072720 0x7f359c0c0d00 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f3590009870 tx=0x7f359001c3f0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 -- 172.21.15.107:0/634415005 shutdown_connections 2023-10-25T00:22:33.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 --2- 172.21.15.107:0/634415005 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f359c0bceb0 0x7f359c15c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 --2- 172.21.15.107:0/634415005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f359c072720 0x7f359c0c0d00 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 -- 172.21.15.107:0/634415005 >> 172.21.15.107:0/634415005 conn(0x7f359c0c2860 msgr2=0x7f359c0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:33.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 -- 172.21.15.107:0/634415005 shutdown_connections 2023-10-25T00:22:33.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.800+0000 7f35a2825640 1 -- 172.21.15.107:0/634415005 wait complete. 2023-10-25T00:22:33.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a2825640 1 Processor -- start 2023-10-25T00:22:33.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a2825640 1 -- start start 2023-10-25T00:22:33.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a2825640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f359c0ab1a0 0x7f359c1efef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a2825640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f359c0bceb0 0x7f359c15c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a2825640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f359c0701b0 0x7f359c14eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a2825640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c1f0c40 con 0x7f359c0701b0 2023-10-25T00:22:33.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a2825640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c06fbe0 con 0x7f359c0ab1a0 2023-10-25T00:22:33.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a2825640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359c06f460 con 0x7f359c0bceb0 2023-10-25T00:22:33.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a0dc2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f359c0701b0 0x7f359c14eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a0dc2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f359c0701b0 0x7f359c14eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45410/0 (socket says 172.21.15.107:45410) 2023-10-25T00:22:33.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a0dc2640 1 -- 172.21.15.107:0/3130903849 learned_addr learned my addr 172.21.15.107:0/3130903849 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:33.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f359bfff640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f359c0ab1a0 0x7f359c1efef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a0dc2640 1 -- 172.21.15.107:0/3130903849 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f359c0bceb0 msgr2=0x7f359c15c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f359b7fe640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f359c0bceb0 0x7f359c15c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a0dc2640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f359c0bceb0 0x7f359c15c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a0dc2640 1 -- 172.21.15.107:0/3130903849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f359c0ab1a0 msgr2=0x7f359c1efef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a0dc2640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f359c0ab1a0 0x7f359c1efef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a0dc2640 1 -- 172.21.15.107:0/3130903849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35900094d0 con 0x7f359c0701b0 2023-10-25T00:22:33.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a0dc2640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f359c0701b0 0x7f359c14eb00 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f359400cd50 tx=0x7f359400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f3598ff9640 1 -- 172.21.15.107:0/3130903849 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359400f680 con 0x7f359c0701b0 2023-10-25T00:22:33.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f3598ff9640 1 -- 172.21.15.107:0/3130903849 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359400f800 con 0x7f359c0701b0 2023-10-25T00:22:33.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a2825640 1 -- 172.21.15.107:0/3130903849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c06fdf0 con 0x7f359c0701b0 2023-10-25T00:22:33.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f3598ff9640 1 -- 172.21.15.107:0/3130903849 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359400fbb0 con 0x7f359c0701b0 2023-10-25T00:22:33.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a2825640 1 -- 172.21.15.107:0/3130903849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c154ee0 con 0x7f359c0701b0 2023-10-25T00:22:33.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f35a2825640 1 -- 172.21.15.107:0/3130903849 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c0beff0 con 0x7f359c0701b0 2023-10-25T00:22:33.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.804+0000 7f3598ff9640 1 -- 172.21.15.107:0/3130903849 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f359400b040 con 0x7f359c0701b0 2023-10-25T00:22:33.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.808+0000 7f3598ff9640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3588065e90 0x7f3588068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:33.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.808+0000 7f3598ff9640 1 -- 172.21.15.107:0/3130903849 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f3594085460 con 0x7f359c0701b0 2023-10-25T00:22:33.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.808+0000 7f359bfff640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3588065e90 0x7f3588068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:33.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.808+0000 7f359bfff640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3588065e90 0x7f3588068350 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f358c000f70 tx=0x7f358c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:33.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.812+0000 7f3598ff9640 1 -- 172.21.15.107:0/3130903849 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3594051810 con 0x7f359c0701b0 2023-10-25T00:22:33.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.960+0000 7f35a2825640 1 -- 172.21.15.107:0/3130903849 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f359c0bf5a0 con 0x7f3588065e90 2023-10-25T00:22:33.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f3598ff9640 1 -- 172.21.15.107:0/3130903849 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f358c001c80 con 0x7f3588065e90 2023-10-25T00:22:33.962 INFO:teuthology.orchestra.run.smithi107.stderr:instructed osd(s) 1 to deep-scrub 2023-10-25T00:22:33.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 -- 172.21.15.107:0/3130903849 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3588065e90 msgr2=0x7f3588068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3588065e90 0x7f3588068350 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f358c000f70 tx=0x7f358c009040 comp rx=0 tx=0).stop 2023-10-25T00:22:33.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 -- 172.21.15.107:0/3130903849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f359c0701b0 msgr2=0x7f359c14eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:33.965 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f359c0701b0 0x7f359c14eb00 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f359400cd50 tx=0x7f359400a650 comp rx=0 tx=0).stop 2023-10-25T00:22:33.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 -- 172.21.15.107:0/3130903849 shutdown_connections 2023-10-25T00:22:33.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f359c0701b0 0x7f359c14eb00 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.966 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3588065e90 0x7f3588068350 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f359c0bceb0 0x7f359c15c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 --2- 172.21.15.107:0/3130903849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f359c0ab1a0 0x7f359c1efef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:33.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 -- 172.21.15.107:0/3130903849 >> 172.21.15.107:0/3130903849 conn(0x7f359c0c2860 msgr2=0x7f359c0bc510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:33.967 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 -- 172.21.15.107:0/3130903849 shutdown_connections 2023-10-25T00:22:33.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:33.964+0000 7f35a2825640 1 -- 172.21.15.107:0/3130903849 wait complete. 2023-10-25T00:22:33.981 INFO:tasks.ceph:Scrubbing osd.2 2023-10-25T00:22:33.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-25T00:22:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.104+0000 7f3547443640 1 Processor -- start 2023-10-25T00:22:34.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.104+0000 7f3547443640 1 -- start start 2023-10-25T00:22:34.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35400ab2d0 0x7f35400a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f354006cc30 0x7f354015d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3540156880 con 0x7f354006aac0 2023-10-25T00:22:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3540155f50 con 0x7f354006cc30 2023-10-25T00:22:34.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3540156230 con 0x7f35400ab2d0 2023-10-25T00:22:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f35459e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f354006cc30 0x7f354015d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f35459e0640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f354006cc30 0x7f354015d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45418/0 (socket says 172.21.15.107:45418) 2023-10-25T00:22:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f35459e0640 1 -- 172.21.15.107:0/750745209 learned_addr learned my addr 172.21.15.107:0/750745209 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f35459e0640 1 -- 172.21.15.107:0/750745209 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35400ab2d0 msgr2=0x7f35400a4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:34.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f35459e0640 1 --2- 172.21.15.107:0/750745209 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35400ab2d0 0x7f35400a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f35459e0640 1 -- 172.21.15.107:0/750745209 >> v1:172.21.15.110:6789/0 conn(0x7f354006aac0 legacy=0x7f35400c0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:34.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f35459e0640 1 -- 172.21.15.107:0/750745209 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3540156510 con 0x7f354006cc30 2023-10-25T00:22:34.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f35459e0640 1 --2- 172.21.15.107:0/750745209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354006cc30 0x7f354015d1e0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f3538009e80 tx=0x7f353801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=854e72b22d6091bc server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3537fff640 1 -- 172.21.15.107:0/750745209 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35380a5740 con 0x7f354006cc30 2023-10-25T00:22:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3537fff640 1 -- 172.21.15.107:0/750745209 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35380a58a0 con 0x7f354006cc30 2023-10-25T00:22:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3537fff640 1 -- 172.21.15.107:0/750745209 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35380a5c30 con 0x7f354006cc30 2023-10-25T00:22:34.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 -- 172.21.15.107:0/750745209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354006cc30 msgr2=0x7f354015d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 --2- 172.21.15.107:0/750745209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354006cc30 0x7f354015d1e0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f3538009e80 tx=0x7f353801c9b0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 -- 172.21.15.107:0/750745209 shutdown_connections 2023-10-25T00:22:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 --2- 172.21.15.107:0/750745209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354006cc30 0x7f354015d1e0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 --2- 172.21.15.107:0/750745209 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35400ab2d0 0x7f35400a4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 -- 172.21.15.107:0/750745209 >> 172.21.15.107:0/750745209 conn(0x7f35400c27f0 msgr2=0x7f35400c2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 -- 172.21.15.107:0/750745209 shutdown_connections 2023-10-25T00:22:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.108+0000 7f3547443640 1 -- 172.21.15.107:0/750745209 wait complete. 2023-10-25T00:22:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3547443640 1 Processor -- start 2023-10-25T00:22:34.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3547443640 1 -- start start 2023-10-25T00:22:34.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3547443640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354006aac0 0x7f354006f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3547443640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f354006cc30 0x7f35401508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3547443640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35400ab2d0 0x7f3540150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3547443640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35401eaf20 con 0x7f354006aac0 2023-10-25T00:22:34.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3547443640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3540151a10 con 0x7f354006cc30 2023-10-25T00:22:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3547443640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35401ef210 con 0x7f35400ab2d0 2023-10-25T00:22:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f35451df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354006aac0 0x7f354006f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f35451df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354006aac0 0x7f354006f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45426/0 (socket says 172.21.15.107:45426) 2023-10-25T00:22:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f35451df640 1 -- 172.21.15.107:0/1524613258 learned_addr learned my addr 172.21.15.107:0/1524613258 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f35449de640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f354006cc30 0x7f35401508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f35459e0640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35400ab2d0 0x7f3540150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f35451df640 1 -- 172.21.15.107:0/1524613258 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35400ab2d0 msgr2=0x7f3540150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f35451df640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f35400ab2d0 0x7f3540150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f35451df640 1 -- 172.21.15.107:0/1524613258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f354006cc30 msgr2=0x7f35401508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f35451df640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f354006cc30 0x7f35401508a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f35451df640 1 -- 172.21.15.107:0/1524613258 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3538009b30 con 0x7f354006aac0 2023-10-25T00:22:34.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f35451df640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354006aac0 0x7f354006f410 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f353000b410 tx=0x7f353000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:34.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3535ffb640 1 -- 172.21.15.107:0/1524613258 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530011830 con 0x7f354006aac0 2023-10-25T00:22:34.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3535ffb640 1 -- 172.21.15.107:0/1524613258 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3530011990 con 0x7f354006aac0 2023-10-25T00:22:34.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3535ffb640 1 -- 172.21.15.107:0/1524613258 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3530011d40 con 0x7f354006aac0 2023-10-25T00:22:34.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3547443640 1 -- 172.21.15.107:0/1524613258 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35400bc7e0 con 0x7f354006aac0 2023-10-25T00:22:34.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3547443640 1 -- 172.21.15.107:0/1524613258 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35400bcda0 con 0x7f354006aac0 2023-10-25T00:22:34.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3547443640 1 -- 172.21.15.107:0/1524613258 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f35400a54b0 con 0x7f354006aac0 2023-10-25T00:22:34.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.112+0000 7f3535ffb640 1 -- 172.21.15.107:0/1524613258 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f353001a400 con 0x7f354006aac0 2023-10-25T00:22:34.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.116+0000 7f3535ffb640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3524065e60 0x7f3524068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.116+0000 7f35449de640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3524065e60 0x7f3524068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.116+0000 7f3535ffb640 1 -- 172.21.15.107:0/1524613258 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f3530084c30 con 0x7f354006aac0 2023-10-25T00:22:34.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.116+0000 7f3535ffb640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] conn(0x7f3524073100 0x7f3524075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.116+0000 7f3535ffb640 1 -- 172.21.15.107:0/1524613258 --> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3524075c30 con 0x7f3524073100 2023-10-25T00:22:34.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.116+0000 7f3535ffb640 1 -- 172.21.15.107:0/1524613258 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f353008a240 con 0x7f354006aac0 2023-10-25T00:22:34.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.116+0000 7f35459e0640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] conn(0x7f3524073100 0x7f3524075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.116+0000 7f35449de640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3524065e60 0x7f3524068320 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f354015d0d0 tx=0x7f353c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:34.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.116+0000 7f35459e0640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] conn(0x7f3524073100 0x7f3524075560 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:34.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.116+0000 7f3535ffb640 1 -- 172.21.15.107:0/1524613258 <== osd.2 v2:172.21.15.107:6808/37539 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3524075c30 con 0x7f3524073100 2023-10-25T00:22:34.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3547443640 1 -- 172.21.15.107:0/1524613258 --> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f35400746e0 con 0x7f3524073100 2023-10-25T00:22:34.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3535ffb640 1 -- 172.21.15.107:0/1524613258 <== osd.2 v2:172.21.15.107:6808/37539 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f35400746e0 con 0x7f3524073100 2023-10-25T00:22:34.130 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-10-25T00:22:34.130 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-25T00:22:34.130 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-10-25T00:22:34.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3547443640 1 -- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] conn(0x7f3524073100 msgr2=0x7f3524075560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3547443640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:6808/37539,v1:172.21.15.107:6809/37539] conn(0x7f3524073100 0x7f3524075560 crc :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3547443640 1 -- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3524065e60 msgr2=0x7f3524068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3547443640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3524065e60 0x7f3524068320 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f354015d0d0 tx=0x7f353c009040 comp rx=0 tx=0).stop 2023-10-25T00:22:34.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3547443640 1 -- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354006aac0 msgr2=0x7f354006f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3547443640 1 --2- 172.21.15.107:0/1524613258 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f354006aac0 0x7f354006f410 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f353000b410 tx=0x7f353000b8d0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f35459e0640 1 -- 172.21.15.107:0/1524613258 reap_dead start 2023-10-25T00:22:34.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3547443640 1 -- 172.21.15.107:0/1524613258 shutdown_connections 2023-10-25T00:22:34.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3547443640 1 -- 172.21.15.107:0/1524613258 >> 172.21.15.107:0/1524613258 conn(0x7f35400c27f0 msgr2=0x7f35400c0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:34.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3547443640 1 -- 172.21.15.107:0/1524613258 shutdown_connections 2023-10-25T00:22:34.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.128+0000 7f3547443640 1 -- 172.21.15.107:0/1524613258 wait complete. 2023-10-25T00:22:34.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-25T00:22:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1e23c3640 1 Processor -- start 2023-10-25T00:22:34.257 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1e23c3640 1 -- start start 2023-10-25T00:22:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1e23c3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff1dc0c1b30 0x7ff1dc072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1e23c3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1dc0a9300 0x7ff1dc0aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1e23c3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc0befa0 con 0x7ff1dc0aa820 2023-10-25T00:22:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1e23c3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1dc0aac70 con 0x7ff1dc0a9300 2023-10-25T00:22:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1e23c3640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1dc0a9b80 con 0x7ff1dc0c1b30 2023-10-25T00:22:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1db7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff1dc0a9300 0x7ff1dc0aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1e0960640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7ff1dc0aa820 0x7ff1dc0c4cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35278/0 (socket says 172.21.15.107:35278) 2023-10-25T00:22:34.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1e0960640 1 -- 172.21.15.107:0/351699975 learned_addr learned my addr 172.21.15.107:0/351699975 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1dbfff640 1 --2- 172.21.15.107:0/351699975 >> v2:172.21.15.110:3300/0 conn(0x7ff1dc0c1b30 0x7ff1dc072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1daffd640 1 -- 172.21.15.107:0/351699975 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810936000 0 0) 0x7ff1dc0befa0 con 0x7ff1dc0aa820 2023-10-25T00:22:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1daffd640 1 -- 172.21.15.107:0/351699975 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c4003530 con 0x7ff1dc0aa820 2023-10-25T00:22:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1db7fe640 1 -- 172.21.15.107:0/351699975 >> v2:172.21.15.110:3300/0 conn(0x7ff1dc0c1b30 msgr2=0x7ff1dc072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:34.261 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1db7fe640 1 --2- 172.21.15.107:0/351699975 >> v2:172.21.15.110:3300/0 conn(0x7ff1dc0c1b30 0x7ff1dc072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1daffd640 1 -- 172.21.15.107:0/351699975 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2539482007 0 0) 0x7ff1c4003530 con 0x7ff1dc0aa820 2023-10-25T00:22:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1db7fe640 1 -- 172.21.15.107:0/351699975 >> v1:172.21.15.107:6789/0 conn(0x7ff1dc0aa820 legacy=0x7ff1dc0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.262 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.256+0000 7ff1db7fe640 1 -- 172.21.15.107:0/351699975 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc0aa0b0 con 0x7ff1dc0a9300 2023-10-25T00:22:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1db7fe640 1 --2- 172.21.15.107:0/351699975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1dc0a9300 0x7ff1dc0aa2e0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0c26e0 tx=0x7ff1d001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a5c2240a46577d server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1daffd640 1 -- 172.21.15.107:0/351699975 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4001a00 con 0x7ff1dc0a9300 2023-10-25T00:22:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1daffd640 1 -- 172.21.15.107:0/351699975 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d00a4b80 con 0x7ff1dc0a9300 2023-10-25T00:22:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1daffd640 1 -- 172.21.15.107:0/351699975 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00ac440 con 0x7ff1dc0a9300 2023-10-25T00:22:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1e23c3640 1 -- 172.21.15.107:0/351699975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1dc0a9300 msgr2=0x7ff1dc0aa2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1e23c3640 1 --2- 172.21.15.107:0/351699975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1dc0a9300 0x7ff1dc0aa2e0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0c26e0 tx=0x7ff1d001c260 comp rx=0 tx=0).stop 2023-10-25T00:22:34.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1e23c3640 1 -- 172.21.15.107:0/351699975 shutdown_connections 2023-10-25T00:22:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1e23c3640 1 --2- 172.21.15.107:0/351699975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1dc0a9300 0x7ff1dc0aa2e0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1e23c3640 1 --2- 172.21.15.107:0/351699975 >> v2:172.21.15.110:3300/0 conn(0x7ff1dc0c1b30 0x7ff1dc072210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1e23c3640 1 -- 172.21.15.107:0/351699975 >> 172.21.15.107:0/351699975 conn(0x7ff1dc0c2860 msgr2=0x7ff1dc0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1e23c3640 1 -- 172.21.15.107:0/351699975 shutdown_connections 2023-10-25T00:22:34.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.260+0000 7ff1e23c3640 1 -- 172.21.15.107:0/351699975 wait complete. 2023-10-25T00:22:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e23c3640 1 Processor -- start 2023-10-25T00:22:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e23c3640 1 -- start start 2023-10-25T00:22:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e23c3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1dc0a9300 0x7ff1dc1eb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e23c3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1dc0aa820 0x7ff1dc06c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e23c3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1dc0c1b30 0x7ff1dc071000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e23c3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc1edef0 con 0x7ff1dc0aa820 2023-10-25T00:22:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e23c3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc1ee100 con 0x7ff1dc0c1b30 2023-10-25T00:22:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e23c3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc1eacb0 con 0x7ff1dc0a9300 2023-10-25T00:22:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1db7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1dc0aa820 0x7ff1dc06c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1db7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1dc0aa820 0x7ff1dc06c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45444/0 (socket says 172.21.15.107:45444) 2023-10-25T00:22:34.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1db7fe640 1 -- 172.21.15.107:0/221171366 learned_addr learned my addr 172.21.15.107:0/221171366 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1dbfff640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1dc0a9300 0x7ff1dc1eb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e0960640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1dc0c1b30 0x7ff1dc071000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1db7fe640 1 -- 172.21.15.107:0/221171366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1dc0a9300 msgr2=0x7ff1dc1eb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1db7fe640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1dc0a9300 0x7ff1dc1eb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1db7fe640 1 -- 172.21.15.107:0/221171366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1dc0c1b30 msgr2=0x7ff1dc071000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1db7fe640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1dc0c1b30 0x7ff1dc071000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1db7fe640 1 -- 172.21.15.107:0/221171366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d0009420 con 0x7ff1dc0aa820 2023-10-25T00:22:34.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1db7fe640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1dc0aa820 0x7ff1dc06c770 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00af040 tx=0x7ff1d0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1d8ff9640 1 -- 172.21.15.107:0/221171366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00b5430 con 0x7ff1dc0aa820 2023-10-25T00:22:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1d8ff9640 1 -- 172.21.15.107:0/221171366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d00b55b0 con 0x7ff1dc0aa820 2023-10-25T00:22:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e23c3640 1 -- 172.21.15.107:0/221171366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1dc1eaf90 con 0x7ff1dc0aa820 2023-10-25T00:22:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1d8ff9640 1 -- 172.21.15.107:0/221171366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00b5960 con 0x7ff1dc0aa820 2023-10-25T00:22:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e23c3640 1 -- 172.21.15.107:0/221171366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1dc1ea980 con 0x7ff1dc0aa820 2023-10-25T00:22:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1e23c3640 1 -- 172.21.15.107:0/221171366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1dc04e130 con 0x7ff1dc0aa820 2023-10-25T00:22:34.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1d8ff9640 1 -- 172.21.15.107:0/221171366 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff1d00b3040 con 0x7ff1dc0aa820 2023-10-25T00:22:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.264+0000 7ff1d8ff9640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1c8065e60 0x7ff1c8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.268+0000 7ff1d8ff9640 1 -- 172.21.15.107:0/221171366 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7ff1d011e970 con 0x7ff1dc0aa820 2023-10-25T00:22:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.268+0000 7ff1dbfff640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1c8065e60 0x7ff1c8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.268+0000 7ff1dbfff640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1c8065e60 0x7ff1c8068320 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc009a50 tx=0x7ff1cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.272+0000 7ff1d8ff9640 1 -- 172.21.15.107:0/221171366 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff1d00eadb0 con 0x7ff1dc0aa820 2023-10-25T00:22:34.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.424+0000 7ff1e23c3640 1 -- 172.21.15.107:0/221171366 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1dc0c1f10 con 0x7ff1c8065e60 2023-10-25T00:22:34.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.424+0000 7ff1d8ff9640 1 -- 172.21.15.107:0/221171366 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff1cc0119a0 con 0x7ff1c8065e60 2023-10-25T00:22:34.425 INFO:teuthology.orchestra.run.smithi107.stderr:instructed osd(s) 2 to deep-scrub 2023-10-25T00:22:34.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 -- 172.21.15.107:0/221171366 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1c8065e60 msgr2=0x7ff1c8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1c8065e60 0x7ff1c8068320 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc009a50 tx=0x7ff1cc008040 comp rx=0 tx=0).stop 2023-10-25T00:22:34.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 -- 172.21.15.107:0/221171366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1dc0aa820 msgr2=0x7ff1dc06c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1dc0aa820 0x7ff1dc06c770 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00af040 tx=0x7ff1d0019000 comp rx=0 tx=0).stop 2023-10-25T00:22:34.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 -- 172.21.15.107:0/221171366 shutdown_connections 2023-10-25T00:22:34.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1dc0c1b30 0x7ff1dc071000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1dc0aa820 0x7ff1dc06c770 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1c8065e60 0x7ff1c8068320 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 --2- 172.21.15.107:0/221171366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff1dc0a9300 0x7ff1dc1eb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 -- 172.21.15.107:0/221171366 >> 172.21.15.107:0/221171366 conn(0x7ff1dc0c2860 msgr2=0x7ff1dc15c010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:34.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 -- 172.21.15.107:0/221171366 shutdown_connections 2023-10-25T00:22:34.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.428+0000 7ff1e23c3640 1 -- 172.21.15.107:0/221171366 wait complete. 2023-10-25T00:22:34.444 INFO:tasks.ceph:Scrubbing osd.3 2023-10-25T00:22:34.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-25T00:22:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39ee3e2640 1 Processor -- start 2023-10-25T00:22:34.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39ee3e2640 1 -- start start 2023-10-25T00:22:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39ee3e2640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f39e8151dd0 0x7f39e80c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39ee3e2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f39e8155310 0x7f39e8158150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39ee3e2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e8155820 con 0x7f39e80c8100 2023-10-25T00:22:34.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39ee3e2640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e80c6cb0 con 0x7f39e8151dd0 2023-10-25T00:22:34.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39ee3e2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e80c6f90 con 0x7f39e8155310 2023-10-25T00:22:34.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39e7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f39e8151dd0 0x7f39e80c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39e7fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f39e8151dd0 0x7f39e80c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45460/0 (socket says 172.21.15.107:45460) 2023-10-25T00:22:34.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39e7fff640 1 -- 172.21.15.107:0/3974937245 learned_addr learned my addr 172.21.15.107:0/3974937245 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:34.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39e77fe640 1 --2- 172.21.15.107:0/3974937245 >> v2:172.21.15.110:3300/0 conn(0x7f39e8155310 0x7f39e8158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39e7fff640 1 -- 172.21.15.107:0/3974937245 >> v2:172.21.15.110:3300/0 conn(0x7f39e8155310 msgr2=0x7f39e8158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:34.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39e7fff640 1 --2- 172.21.15.107:0/3974937245 >> v2:172.21.15.110:3300/0 conn(0x7f39e8155310 0x7f39e8158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39e7fff640 1 -- 172.21.15.107:0/3974937245 >> v1:172.21.15.110:6789/0 conn(0x7f39e80c8100 legacy=0x7f39e8158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:34.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39e7fff640 1 -- 172.21.15.107:0/3974937245 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e80c7270 con 0x7f39e8151dd0 2023-10-25T00:22:34.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.572+0000 7f39e77fe640 1 --2- 172.21.15.107:0/3974937245 >> v2:172.21.15.110:3300/0 conn(0x7f39e8155310 0x7f39e8158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:34.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39e7fff640 1 --2- 172.21.15.107:0/3974937245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39e8151dd0 0x7f39e80c4b80 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f39e806cdd0 tx=0x7f39d801ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d698125a595d0a45 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:34.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39e6ffd640 1 -- 172.21.15.107:0/3974937245 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d80aa5f0 con 0x7f39e8151dd0 2023-10-25T00:22:34.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39e6ffd640 1 -- 172.21.15.107:0/3974937245 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d80aa750 con 0x7f39e8151dd0 2023-10-25T00:22:34.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39e6ffd640 1 -- 172.21.15.107:0/3974937245 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d80aaae0 con 0x7f39e8151dd0 2023-10-25T00:22:34.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39ee3e2640 1 -- 172.21.15.107:0/3974937245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39e8151dd0 msgr2=0x7f39e80c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39ee3e2640 1 --2- 172.21.15.107:0/3974937245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39e8151dd0 0x7f39e80c4b80 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f39e806cdd0 tx=0x7f39d801ced0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39ee3e2640 1 -- 172.21.15.107:0/3974937245 shutdown_connections 2023-10-25T00:22:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39ee3e2640 1 --2- 172.21.15.107:0/3974937245 >> v2:172.21.15.110:3300/0 conn(0x7f39e8155310 0x7f39e8158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39ee3e2640 1 --2- 172.21.15.107:0/3974937245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39e8151dd0 0x7f39e80c4b80 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39ee3e2640 1 -- 172.21.15.107:0/3974937245 >> 172.21.15.107:0/3974937245 conn(0x7f39e80abdb0 msgr2=0x7f39e80ac1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39ee3e2640 1 -- 172.21.15.107:0/3974937245 shutdown_connections 2023-10-25T00:22:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39ee3e2640 1 -- 172.21.15.107:0/3974937245 wait complete. 2023-10-25T00:22:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39ee3e2640 1 Processor -- start 2023-10-25T00:22:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.576+0000 7f39ee3e2640 1 -- start start 2023-10-25T00:22:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ee3e2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39e80c8100 0x7f39e80aa550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ee3e2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39e8155310 0x7f39e814dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ee3e2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39e80c09e0 0x7f39e81472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ee3e2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e814c770 con 0x7f39e80c09e0 2023-10-25T00:22:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ee3e2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e80c0f60 con 0x7f39e80c8100 2023-10-25T00:22:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ee3e2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e8157140 con 0x7f39e8155310 2023-10-25T00:22:34.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ec97f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39e80c09e0 0x7f39e81472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ec97f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39e80c09e0 0x7f39e81472e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45472/0 (socket says 172.21.15.107:45472) 2023-10-25T00:22:34.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ec97f640 1 -- 172.21.15.107:0/2961772722 learned_addr learned my addr 172.21.15.107:0/2961772722 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:34.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e7fff640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39e80c8100 0x7f39e80aa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e77fe640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39e8155310 0x7f39e814dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ec97f640 1 -- 172.21.15.107:0/2961772722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39e8155310 msgr2=0x7f39e814dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ec97f640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f39e8155310 0x7f39e814dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ec97f640 1 -- 172.21.15.107:0/2961772722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39e80c8100 msgr2=0x7f39e80aa550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:34.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ec97f640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39e80c8100 0x7f39e80aa550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:34.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ec97f640 1 -- 172.21.15.107:0/2961772722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d80094d0 con 0x7f39e80c09e0 2023-10-25T00:22:34.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ec97f640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39e80c09e0 0x7f39e81472e0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f39e000c770 tx=0x7f39e000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:34.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e4ff9640 1 -- 172.21.15.107:0/2961772722 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e0003b80 con 0x7f39e80c09e0 2023-10-25T00:22:34.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e4ff9640 1 -- 172.21.15.107:0/2961772722 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39e0003ce0 con 0x7f39e80c09e0 2023-10-25T00:22:34.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ee3e2640 1 -- 172.21.15.107:0/2961772722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e81573e0 con 0x7f39e80c09e0 2023-10-25T00:22:34.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e4ff9640 1 -- 172.21.15.107:0/2961772722 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39e000aa90 con 0x7f39e80c09e0 2023-10-25T00:22:34.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ee3e2640 1 -- 172.21.15.107:0/2961772722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e80aaa90 con 0x7f39e80c09e0 2023-10-25T00:22:34.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39ee3e2640 1 -- 172.21.15.107:0/2961772722 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f39e814c0d0 con 0x7f39e80c09e0 2023-10-25T00:22:34.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e4ff9640 1 -- 172.21.15.107:0/2961772722 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f39e000abf0 con 0x7f39e80c09e0 2023-10-25T00:22:34.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e4ff9640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39d4065e90 0x7f39d4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e4ff9640 1 -- 172.21.15.107:0/2961772722 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f39e0084890 con 0x7f39e80c09e0 2023-10-25T00:22:34.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e7fff640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39d4065e90 0x7f39d4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e4ff9640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] conn(0x7f39d4073130 0x7f39d4075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:34.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e4ff9640 1 -- 172.21.15.107:0/2961772722 --> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39d4075c60 con 0x7f39d4073130 2023-10-25T00:22:34.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e4ff9640 1 -- 172.21.15.107:0/2961772722 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39e00892a0 con 0x7f39e80c09e0 2023-10-25T00:22:34.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e77fe640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] conn(0x7f39d4073130 0x7f39d4075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:34.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e7fff640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39d4065e90 0x7f39d4068350 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f39d80ae000 tx=0x7f39d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:34.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.580+0000 7f39e77fe640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] conn(0x7f39d4073130 0x7f39d4075590 crc :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:34.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.584+0000 7f39e4ff9640 1 -- 172.21.15.107:0/2961772722 <== osd.3 v2:172.21.15.107:6812/37538 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f39d4075c60 con 0x7f39d4073130 2023-10-25T00:22:34.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:34.596+0000 7f39ee3e2640 1 -- 172.21.15.107:0/2961772722 --> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f39e814c490 con 0x7f39d4073130 2023-10-25T00:22:35.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:22:35.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39e4ff9640 1 -- 172.21.15.107:0/2961772722 <== osd.3 v2:172.21.15.107:6812/37538 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f39e814c490 con 0x7f39d4073130 2023-10-25T00:22:35.155 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-10-25T00:22:35.156 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-25T00:22:35.156 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-10-25T00:22:35.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39ee3e2640 1 -- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] conn(0x7f39d4073130 msgr2=0x7f39d4075590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39ee3e2640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:6812/37538,v1:172.21.15.107:6815/37538] conn(0x7f39d4073130 0x7f39d4075590 crc :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39ee3e2640 1 -- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39d4065e90 msgr2=0x7f39d4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39ee3e2640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f39d4065e90 0x7f39d4068350 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f39d80ae000 tx=0x7f39d8008040 comp rx=0 tx=0).stop 2023-10-25T00:22:35.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39ee3e2640 1 -- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39e80c09e0 msgr2=0x7f39e81472e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39ee3e2640 1 --2- 172.21.15.107:0/2961772722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39e80c09e0 0x7f39e81472e0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f39e000c770 tx=0x7f39e000cc30 comp rx=0 tx=0).stop 2023-10-25T00:22:35.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39ec97f640 1 -- 172.21.15.107:0/2961772722 reap_dead start 2023-10-25T00:22:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39ee3e2640 1 -- 172.21.15.107:0/2961772722 shutdown_connections 2023-10-25T00:22:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39ee3e2640 1 -- 172.21.15.107:0/2961772722 >> 172.21.15.107:0/2961772722 conn(0x7f39e80abdb0 msgr2=0x7f39e806ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39ee3e2640 1 -- 172.21.15.107:0/2961772722 shutdown_connections 2023-10-25T00:22:35.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.156+0000 7f39ee3e2640 1 -- 172.21.15.107:0/2961772722 wait complete. 2023-10-25T00:22:35.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-25T00:22:35.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.264+0000 7f8c56aa5640 1 Processor -- start 2023-10-25T00:22:35.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.264+0000 7f8c56aa5640 1 -- start start 2023-10-25T00:22:35.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c56aa5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8c5006f490 0x7f8c5014c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c56aa5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8c50152b30 0x7f8c50152f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.268 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c56aa5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c50155cd0 con 0x7f8c5014c470 2023-10-25T00:22:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c56aa5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c501526e0 con 0x7f8c5006f490 2023-10-25T00:22:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c56aa5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c500c4380 con 0x7f8c50152b30 2023-10-25T00:22:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c4ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8c5006f490 0x7f8c5014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c54841640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8c5014c470 0x7f8c50071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35288/0 (socket says 172.21.15.107:35288) 2023-10-25T00:22:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c54841640 1 -- 172.21.15.107:0/694829713 learned_addr learned my addr 172.21.15.107:0/694829713 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:35.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c55042640 1 --2- 172.21.15.107:0/694829713 >> v2:172.21.15.110:3300/0 conn(0x7f8c50152b30 0x7f8c50152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c4ffff640 1 -- 172.21.15.107:0/694829713 >> v2:172.21.15.110:3300/0 conn(0x7f8c50152b30 msgr2=0x7f8c50152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:35.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c4ffff640 1 --2- 172.21.15.107:0/694829713 >> v2:172.21.15.110:3300/0 conn(0x7f8c50152b30 0x7f8c50152f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c4ffff640 1 -- 172.21.15.107:0/694829713 >> v1:172.21.15.107:6789/0 conn(0x7f8c5014c470 legacy=0x7f8c50071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c4ffff640 1 -- 172.21.15.107:0/694829713 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c500c48b0 con 0x7f8c5006f490 2023-10-25T00:22:35.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c4f7fe640 1 -- 172.21.15.107:0/694829713 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557820650 0 0) 0x7f8c50155cd0 con 0x7f8c5014c470 2023-10-25T00:22:35.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c55042640 1 --2- 172.21.15.107:0/694829713 >> v2:172.21.15.110:3300/0 conn(0x7f8c50152b30 0x7f8c50152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:35.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c4ffff640 1 --2- 172.21.15.107:0/694829713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c5006f490 0x7f8c5014c850 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f8c5006fa90 tx=0x7f8c4401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f0507968a62625 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c4f7fe640 1 -- 172.21.15.107:0/694829713 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c440ab600 con 0x7f8c5006f490 2023-10-25T00:22:35.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c4f7fe640 1 -- 172.21.15.107:0/694829713 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c440a7050 con 0x7f8c5006f490 2023-10-25T00:22:35.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c4f7fe640 1 -- 172.21.15.107:0/694829713 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c440ab980 con 0x7f8c5006f490 2023-10-25T00:22:35.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.268+0000 7f8c56aa5640 1 -- 172.21.15.107:0/694829713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c5006f490 msgr2=0x7f8c5014c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 --2- 172.21.15.107:0/694829713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c5006f490 0x7f8c5014c850 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f8c5006fa90 tx=0x7f8c4401c340 comp rx=0 tx=0).stop 2023-10-25T00:22:35.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 -- 172.21.15.107:0/694829713 shutdown_connections 2023-10-25T00:22:35.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 --2- 172.21.15.107:0/694829713 >> v2:172.21.15.110:3300/0 conn(0x7f8c50152b30 0x7f8c50152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 --2- 172.21.15.107:0/694829713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c5006f490 0x7f8c5014c850 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 -- 172.21.15.107:0/694829713 >> 172.21.15.107:0/694829713 conn(0x7f8c50070f60 msgr2=0x7f8c50071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:35.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 -- 172.21.15.107:0/694829713 shutdown_connections 2023-10-25T00:22:35.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 -- 172.21.15.107:0/694829713 wait complete. 2023-10-25T00:22:35.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 Processor -- start 2023-10-25T00:22:35.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 -- start start 2023-10-25T00:22:35.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c5006f490 0x7f8c500c1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c5014c470 0x7f8c500aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c50152b30 0x7f8c500ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c501f0010 con 0x7f8c5014c470 2023-10-25T00:22:35.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c500adef0 con 0x7f8c5006f490 2023-10-25T00:22:35.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c501449f0 con 0x7f8c50152b30 2023-10-25T00:22:35.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c5014c470 0x7f8c500aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c5014c470 0x7f8c500aabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45494/0 (socket says 172.21.15.107:45494) 2023-10-25T00:22:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4ffff640 1 -- 172.21.15.107:0/3448846385 learned_addr learned my addr 172.21.15.107:0/3448846385 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c54841640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c5006f490 0x7f8c500c1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c55042640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c50152b30 0x7f8c500ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4ffff640 1 -- 172.21.15.107:0/3448846385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c50152b30 msgr2=0x7f8c500ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4ffff640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c50152b30 0x7f8c500ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4ffff640 1 -- 172.21.15.107:0/3448846385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c5006f490 msgr2=0x7f8c500c1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4ffff640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c5006f490 0x7f8c500c1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4ffff640 1 -- 172.21.15.107:0/3448846385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c44009420 con 0x7f8c5014c470 2023-10-25T00:22:35.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4ffff640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c5014c470 0x7f8c500aabf0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f8c44018f50 tx=0x7f8c440190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4d7fa640 1 -- 172.21.15.107:0/3448846385 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c440b48d0 con 0x7f8c5014c470 2023-10-25T00:22:35.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4d7fa640 1 -- 172.21.15.107:0/3448846385 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c440b4a30 con 0x7f8c5014c470 2023-10-25T00:22:35.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c4d7fa640 1 -- 172.21.15.107:0/3448846385 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c440b4da0 con 0x7f8c5014c470 2023-10-25T00:22:35.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 -- 172.21.15.107:0/3448846385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c501ef950 con 0x7f8c5014c470 2023-10-25T00:22:35.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.272+0000 7f8c56aa5640 1 -- 172.21.15.107:0/3448846385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c500c1ab0 con 0x7f8c5014c470 2023-10-25T00:22:35.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.276+0000 7f8c56aa5640 1 -- 172.21.15.107:0/3448846385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c501ef7b0 con 0x7f8c5014c470 2023-10-25T00:22:35.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.276+0000 7f8c4d7fa640 1 -- 172.21.15.107:0/3448846385 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8c440b2040 con 0x7f8c5014c470 2023-10-25T00:22:35.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.276+0000 7f8c4d7fa640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c3c065e60 0x7f8c3c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.276+0000 7f8c4d7fa640 1 -- 172.21.15.107:0/3448846385 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f8c4411f3b0 con 0x7f8c5014c470 2023-10-25T00:22:35.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.276+0000 7f8c54841640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c3c065e60 0x7f8c3c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.276+0000 7f8c54841640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c3c065e60 0x7f8c3c068320 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8c48007fd0 tx=0x7f8c48007ee0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.280+0000 7f8c4d7fa640 1 -- 172.21.15.107:0/3448846385 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8c440eb7f0 con 0x7f8c5014c470 2023-10-25T00:22:35.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf547a4640 1 Processor -- start 2023-10-25T00:22:35.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf547a4640 1 -- start start 2023-10-25T00:22:35.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf547a4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faf4c1521e0 0x7faf4c1554f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf547a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c071ee0 0x7faf4c0a9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf547a4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c0c8dd0 con 0x7faf4c06e360 2023-10-25T00:22:35.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf547a4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf4c0c2430 con 0x7faf4c1521e0 2023-10-25T00:22:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf547a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c1525c0 con 0x7faf4c071ee0 2023-10-25T00:22:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf52540640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faf4c1521e0 0x7faf4c1554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf52d41640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7faf4c06e360 0x7faf4c0c1480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:42416/0 (socket says 172.21.15.107:42416) 2023-10-25T00:22:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf51d3f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c071ee0 0x7faf4c0a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf52d41640 1 -- 172.21.15.107:0/2335846016 learned_addr learned my addr 172.21.15.107:0/2335846016 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf5153e640 1 -- 172.21.15.107:0/2335846016 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163039138 0 0) 0x7faf440014a0 con 0x7faf4c06e360 2023-10-25T00:22:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf5153e640 1 -- 172.21.15.107:0/2335846016 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf38003530 con 0x7faf4c06e360 2023-10-25T00:22:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf51d3f640 1 -- 172.21.15.107:0/2335846016 >> v2:172.21.15.110:3300/0 conn(0x7faf4c1521e0 msgr2=0x7faf4c1554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:35.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf51d3f640 1 --2- 172.21.15.107:0/2335846016 >> v2:172.21.15.110:3300/0 conn(0x7faf4c1521e0 0x7faf4c1554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf51d3f640 1 -- 172.21.15.107:0/2335846016 >> v1:172.21.15.110:6789/0 conn(0x7faf4c06e360 legacy=0x7faf4c0c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.356+0000 7faf51d3f640 1 -- 172.21.15.107:0/2335846016 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c06ed10 con 0x7faf4c071ee0 2023-10-25T00:22:35.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf51d3f640 1 --2- 172.21.15.107:0/2335846016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c071ee0 0x7faf4c0a9450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7faf480097c0 tx=0x7faf4801c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89d1916613fd9265 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf5153e640 1 -- 172.21.15.107:0/2335846016 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf480ab600 con 0x7faf4c071ee0 2023-10-25T00:22:35.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf5153e640 1 -- 172.21.15.107:0/2335846016 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf480a7050 con 0x7faf4c071ee0 2023-10-25T00:22:35.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf5153e640 1 -- 172.21.15.107:0/2335846016 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf480ab960 con 0x7faf4c071ee0 2023-10-25T00:22:35.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf547a4640 1 -- 172.21.15.107:0/2335846016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c071ee0 msgr2=0x7faf4c0a9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf547a4640 1 --2- 172.21.15.107:0/2335846016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c071ee0 0x7faf4c0a9450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7faf480097c0 tx=0x7faf4801c3f0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.362 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf547a4640 1 -- 172.21.15.107:0/2335846016 shutdown_connections 2023-10-25T00:22:35.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf547a4640 1 --2- 172.21.15.107:0/2335846016 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c071ee0 0x7faf4c0a9450 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf547a4640 1 --2- 172.21.15.107:0/2335846016 >> v2:172.21.15.110:3300/0 conn(0x7faf4c1521e0 0x7faf4c1554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf547a4640 1 -- 172.21.15.107:0/2335846016 >> 172.21.15.107:0/2335846016 conn(0x7faf4c0c8020 msgr2=0x7faf4c0c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:35.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf547a4640 1 -- 172.21.15.107:0/2335846016 shutdown_connections 2023-10-25T00:22:35.363 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf547a4640 1 -- 172.21.15.107:0/2335846016 wait complete. 2023-10-25T00:22:35.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf547a4640 1 Processor -- start 2023-10-25T00:22:35.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.360+0000 7faf547a4640 1 -- start start 2023-10-25T00:22:35.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf547a4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c06e360 0x7faf4c145f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf547a4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf4c071ee0 0x7faf4c1534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf547a4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c1521e0 0x7faf4c143830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf547a4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c14b540 con 0x7faf4c1521e0 2023-10-25T00:22:35.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf547a4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c146620 con 0x7faf4c071ee0 2023-10-25T00:22:35.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf547a4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c14a340 con 0x7faf4c06e360 2023-10-25T00:22:35.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52d41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c1521e0 0x7faf4c143830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52d41640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c1521e0 0x7faf4c143830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45504/0 (socket says 172.21.15.107:45504) 2023-10-25T00:22:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52d41640 1 -- 172.21.15.107:0/442394947 learned_addr learned my addr 172.21.15.107:0/442394947 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52540640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c06e360 0x7faf4c145f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf51d3f640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf4c071ee0 0x7faf4c1534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52d41640 1 -- 172.21.15.107:0/442394947 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c06e360 msgr2=0x7faf4c145f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52d41640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c06e360 0x7faf4c145f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52d41640 1 -- 172.21.15.107:0/442394947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf4c071ee0 msgr2=0x7faf4c1534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52d41640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf4c071ee0 0x7faf4c1534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52d41640 1 -- 172.21.15.107:0/442394947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf48009420 con 0x7faf4c1521e0 2023-10-25T00:22:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52540640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c06e360 0x7faf4c145f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:22:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52d41640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c1521e0 0x7faf4c143830 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7faf4400c9f0 tx=0x7faf4400ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf3effd640 1 -- 172.21.15.107:0/442394947 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4400ab70 con 0x7faf4c1521e0 2023-10-25T00:22:35.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf3effd640 1 -- 172.21.15.107:0/442394947 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4400acd0 con 0x7faf4c1521e0 2023-10-25T00:22:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf547a4640 1 -- 172.21.15.107:0/442394947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4c14a620 con 0x7faf4c1521e0 2023-10-25T00:22:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf3effd640 1 -- 172.21.15.107:0/442394947 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf44012600 con 0x7faf4c1521e0 2023-10-25T00:22:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf547a4640 1 -- 172.21.15.107:0/442394947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4c14abe0 con 0x7faf4c1521e0 2023-10-25T00:22:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf547a4640 1 -- 172.21.15.107:0/442394947 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4c14add0 con 0x7faf4c1521e0 2023-10-25T00:22:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf3effd640 1 -- 172.21.15.107:0/442394947 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7faf44007070 con 0x7faf4c1521e0 2023-10-25T00:22:35.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf3effd640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf30065e60 0x7faf30068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52540640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf30065e60 0x7faf30068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf3effd640 1 -- 172.21.15.107:0/442394947 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7faf44084740 con 0x7faf4c1521e0 2023-10-25T00:22:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.364+0000 7faf52540640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf30065e60 0x7faf30068320 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7faf4c1eac20 tx=0x7faf40008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.374 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.372+0000 7faf3effd640 1 -- 172.21.15.107:0/442394947 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7faf44050bb0 con 0x7faf4c1521e0 2023-10-25T00:22:35.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.476+0000 7f8c56aa5640 1 -- 172.21.15.107:0/3448846385 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c501f0150 con 0x7f8c5014c470 2023-10-25T00:22:35.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.476+0000 7f8c4d7fa640 1 -- 172.21.15.107:0/3448846385 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5768 (secure 0 0 0) 0x7f8c501f0150 con 0x7f8c5014c470 2023-10-25T00:22:35.478 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:35.479 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:21:42.241858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9009":{"gid":9009,"name":"c","rank":0,"incarnation":226,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:22:35.479 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 230 2023-10-25T00:22:35.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 -- 172.21.15.107:0/3448846385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c3c065e60 msgr2=0x7f8c3c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c3c065e60 0x7f8c3c068320 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8c48007fd0 tx=0x7f8c48007ee0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 -- 172.21.15.107:0/3448846385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c5014c470 msgr2=0x7f8c500aabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c5014c470 0x7f8c500aabf0 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f8c44018f50 tx=0x7f8c440190e0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 -- 172.21.15.107:0/3448846385 shutdown_connections 2023-10-25T00:22:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c3c065e60 0x7f8c3c068320 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c50152b30 0x7f8c500ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c5014c470 0x7f8c500aabf0 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 --2- 172.21.15.107:0/3448846385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c5006f490 0x7f8c500c1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 -- 172.21.15.107:0/3448846385 >> 172.21.15.107:0/3448846385 conn(0x7f8c50070f60 msgr2=0x7f8c500c3150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:35.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 -- 172.21.15.107:0/3448846385 shutdown_connections 2023-10-25T00:22:35.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.480+0000 7f8c56aa5640 1 -- 172.21.15.107:0/3448846385 wait complete. 2023-10-25T00:22:35.498 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2023-10-25T00:22:35.498 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:22:35.499 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:22:35.530 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.528+0000 7faf547a4640 1 -- 172.21.15.107:0/442394947 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7faf4c14a4b0 con 0x7faf30065e60 2023-10-25T00:22:35.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf3effd640 1 -- 172.21.15.107:0/442394947 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7faf4c14a4b0 con 0x7faf30065e60 2023-10-25T00:22:35.531 INFO:teuthology.orchestra.run.smithi107.stderr:instructed osd(s) 3 to deep-scrub 2023-10-25T00:22:35.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 -- 172.21.15.107:0/442394947 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf30065e60 msgr2=0x7faf30068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf30065e60 0x7faf30068320 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7faf4c1eac20 tx=0x7faf40008040 comp rx=0 tx=0).stop 2023-10-25T00:22:35.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 -- 172.21.15.107:0/442394947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c1521e0 msgr2=0x7faf4c143830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c1521e0 0x7faf4c143830 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7faf4400c9f0 tx=0x7faf4400ceb0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 -- 172.21.15.107:0/442394947 shutdown_connections 2023-10-25T00:22:35.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf4c1521e0 0x7faf4c143830 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf4c071ee0 0x7faf4c1534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf30065e60 0x7faf30068320 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 --2- 172.21.15.107:0/442394947 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf4c06e360 0x7faf4c145f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 -- 172.21.15.107:0/442394947 >> 172.21.15.107:0/442394947 conn(0x7faf4c0c8020 msgr2=0x7faf4c0c5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:35.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 -- 172.21.15.107:0/442394947 shutdown_connections 2023-10-25T00:22:35.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.532+0000 7faf547a4640 1 -- 172.21.15.107:0/442394947 wait complete. 2023-10-25T00:22:35.545 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:22:35.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:22:35.554 INFO:tasks.ceph:Scrubbing osd.4 2023-10-25T00:22:35.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-25T00:22:35.707 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b71a2640 1 Processor -- start 2023-10-25T00:22:35.708 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b71a2640 1 -- start start 2023-10-25T00:22:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b71a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b00720c0 0x7fd8b00c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b71a2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd8b0157e50 0x7fd8b01582c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b71a2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b0154860 con 0x7fd8b013dc20 2023-10-25T00:22:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b71a2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b0158800 con 0x7fd8b0157e50 2023-10-25T00:22:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b71a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00c1f10 con 0x7fd8b00720c0 2023-10-25T00:22:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b4f3e640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd8b013dc20 0x7fd8b006df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:42432/0 (socket says 172.21.15.107:42432) 2023-10-25T00:22:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b4f3e640 1 -- 172.21.15.107:0/1790320458 learned_addr learned my addr 172.21.15.107:0/1790320458 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b573f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd8b0157e50 0x7fd8b01582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8affff640 1 --2- 172.21.15.107:0/1790320458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b00720c0 0x7fd8b00c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8af7fe640 1 -- 172.21.15.107:0/1790320458 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039914876 0 0) 0x7fd8b0154860 con 0x7fd8b013dc20 2023-10-25T00:22:35.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8af7fe640 1 -- 172.21.15.107:0/1790320458 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd89c003530 con 0x7fd8b013dc20 2023-10-25T00:22:35.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b573f640 1 -- 172.21.15.107:0/1790320458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b00720c0 msgr2=0x7fd8b00c18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:35.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b573f640 1 --2- 172.21.15.107:0/1790320458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b00720c0 0x7fd8b00c18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b573f640 1 -- 172.21.15.107:0/1790320458 >> v1:172.21.15.110:6789/0 conn(0x7fd8b013dc20 legacy=0x7fd8b006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8b573f640 1 -- 172.21.15.107:0/1790320458 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b0154b20 con 0x7fd8b0157e50 2023-10-25T00:22:35.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.708+0000 7fd8affff640 1 --2- 172.21.15.107:0/1790320458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b00720c0 0x7fd8b00c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:22:35.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8b573f640 1 --2- 172.21.15.107:0/1790320458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8b0157e50 0x7fd8b01582c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009d90 tx=0x7fd8a801c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3d83aa1228521cc server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8af7fe640 1 -- 172.21.15.107:0/1790320458 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a80a57e0 con 0x7fd8b0157e50 2023-10-25T00:22:35.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8af7fe640 1 -- 172.21.15.107:0/1790320458 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a80a5940 con 0x7fd8b0157e50 2023-10-25T00:22:35.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8af7fe640 1 -- 172.21.15.107:0/1790320458 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a80a5cb0 con 0x7fd8b0157e50 2023-10-25T00:22:35.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8b71a2640 1 -- 172.21.15.107:0/1790320458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8b0157e50 msgr2=0x7fd8b01582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8b71a2640 1 --2- 172.21.15.107:0/1790320458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8b0157e50 0x7fd8b01582c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009d90 tx=0x7fd8a801c830 comp rx=0 tx=0).stop 2023-10-25T00:22:35.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8b71a2640 1 -- 172.21.15.107:0/1790320458 shutdown_connections 2023-10-25T00:22:35.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8b71a2640 1 --2- 172.21.15.107:0/1790320458 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8b0157e50 0x7fd8b01582c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8b71a2640 1 --2- 172.21.15.107:0/1790320458 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b00720c0 0x7fd8b00c18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8b71a2640 1 -- 172.21.15.107:0/1790320458 >> 172.21.15.107:0/1790320458 conn(0x7fd8b00c39a0 msgr2=0x7fd8b00c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:35.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8b71a2640 1 -- 172.21.15.107:0/1790320458 shutdown_connections 2023-10-25T00:22:35.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8b71a2640 1 -- 172.21.15.107:0/1790320458 wait complete. 2023-10-25T00:22:35.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7f1e0ed82640 1 Processor -- start 2023-10-25T00:22:35.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8b71a2640 1 Processor -- start 2023-10-25T00:22:35.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7f1e0ed82640 1 -- start start 2023-10-25T00:22:35.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.712+0000 7fd8b71a2640 1 -- start start 2023-10-25T00:22:35.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b71a2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b00720c0 0x7fd8b01ed500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b71a2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8b013dc20 0x7fd8b00c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b71a2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b0157e50 0x7fd8b00aab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b71a2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01eda40 con 0x7fd8b00720c0 2023-10-25T00:22:35.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b71a2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b015b700 con 0x7fd8b013dc20 2023-10-25T00:22:35.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b71a2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00ab8a0 con 0x7fd8b0157e50 2023-10-25T00:22:35.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0ed82640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e080a9450 0x7f1e08156310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0ed82640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e0806ddf0 0x7f1e0806e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0ed82640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e080c4310 con 0x7f1e0814c850 2023-10-25T00:22:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0ed82640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e080c3460 con 0x7f1e0806ddf0 2023-10-25T00:22:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0ed82640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080c3740 con 0x7f1e080a9450 2023-10-25T00:22:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b4f3e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b00720c0 0x7fd8b01ed500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b4f3e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b00720c0 0x7fd8b01ed500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45518/0 (socket says 172.21.15.107:45518) 2023-10-25T00:22:35.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b4f3e640 1 -- 172.21.15.107:0/499576620 learned_addr learned my addr 172.21.15.107:0/499576620 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:35.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0d31f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e0806ddf0 0x7f1e0806e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0d31f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1e0806ddf0 0x7f1e0806e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45534/0 (socket says 172.21.15.107:45534) 2023-10-25T00:22:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0d31f640 1 -- 172.21.15.107:0/2625340722 learned_addr learned my addr 172.21.15.107:0/2625340722 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:35.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8affff640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8b013dc20 0x7fd8b00c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b573f640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b0157e50 0x7fd8b00aab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e07fff640 1 --2- 172.21.15.107:0/2625340722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e080a9450 0x7f1e08156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b4f3e640 1 -- 172.21.15.107:0/499576620 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b0157e50 msgr2=0x7fd8b00aab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b4f3e640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b0157e50 0x7fd8b00aab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b4f3e640 1 -- 172.21.15.107:0/499576620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8b013dc20 msgr2=0x7fd8b00c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b4f3e640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8b013dc20 0x7fd8b00c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b4f3e640 1 -- 172.21.15.107:0/499576620 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd898009420 con 0x7fd8b00720c0 2023-10-25T00:22:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8affff640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8b013dc20 0x7fd8b00c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:35.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0d31f640 1 -- 172.21.15.107:0/2625340722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e080a9450 msgr2=0x7f1e08156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0d31f640 1 --2- 172.21.15.107:0/2625340722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e080a9450 0x7f1e08156310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0d31f640 1 -- 172.21.15.107:0/2625340722 >> v1:172.21.15.110:6789/0 conn(0x7f1e0814c850 legacy=0x7f1e08155c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0d31f640 1 -- 172.21.15.107:0/2625340722 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e080c3a20 con 0x7f1e0806ddf0 2023-10-25T00:22:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b573f640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b0157e50 0x7fd8b00aab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b4f3e640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b00720c0 0x7fd8b01ed500 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c3c0 tx=0x7fd8a000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8ad7fa640 1 -- 172.21.15.107:0/499576620 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0005cf0 con 0x7fd8b00720c0 2023-10-25T00:22:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8ad7fa640 1 -- 172.21.15.107:0/499576620 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a0005e50 con 0x7fd8b00720c0 2023-10-25T00:22:35.730 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b71a2640 1 -- 172.21.15.107:0/499576620 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a8009a70 con 0x7fd8b00720c0 2023-10-25T00:22:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8ad7fa640 1 -- 172.21.15.107:0/499576620 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a00072f0 con 0x7fd8b00720c0 2023-10-25T00:22:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b71a2640 1 -- 172.21.15.107:0/499576620 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b00abdd0 con 0x7fd8b00720c0 2023-10-25T00:22:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8b71a2640 1 -- 172.21.15.107:0/499576620 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b00746e0 con 0x7fd8b00720c0 2023-10-25T00:22:35.731 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8ad7fa640 1 -- 172.21.15.107:0/499576620 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd8a0007450 con 0x7fd8b00720c0 2023-10-25T00:22:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8ad7fa640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd890065e90 0x7fd890068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8ad7fa640 1 -- 172.21.15.107:0/499576620 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fd8a0084000 con 0x7fd8b00720c0 2023-10-25T00:22:35.732 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7fd8affff640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd890065e90 0x7fd890068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.716+0000 7f1e0d31f640 1 --2- 172.21.15.107:0/2625340722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e0806ddf0 0x7f1e0806e280 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009e80 tx=0x7f1e0001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea7b1388c2c884c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.733 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.720+0000 7fd8affff640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd890065e90 0x7fd890068350 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00c43b0 tx=0x7fd898008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.720+0000 7f1e077fe640 1 -- 172.21.15.107:0/2625340722 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e000a5730 con 0x7f1e0806ddf0 2023-10-25T00:22:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.720+0000 7f1e077fe640 1 -- 172.21.15.107:0/2625340722 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e000a5890 con 0x7f1e0806ddf0 2023-10-25T00:22:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.720+0000 7f1e077fe640 1 -- 172.21.15.107:0/2625340722 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e000a5c20 con 0x7f1e0806ddf0 2023-10-25T00:22:35.735 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.720+0000 7fd8ad7fa640 1 -- 172.21.15.107:0/499576620 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd8a0050440 con 0x7fd8b00720c0 2023-10-25T00:22:35.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 -- 172.21.15.107:0/2625340722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e0806ddf0 msgr2=0x7f1e0806e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 --2- 172.21.15.107:0/2625340722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e0806ddf0 0x7f1e0806e280 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009e80 tx=0x7f1e0001c960 comp rx=0 tx=0).stop 2023-10-25T00:22:35.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 -- 172.21.15.107:0/2625340722 shutdown_connections 2023-10-25T00:22:35.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 --2- 172.21.15.107:0/2625340722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e0806ddf0 0x7f1e0806e280 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 --2- 172.21.15.107:0/2625340722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e080a9450 0x7f1e08156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 -- 172.21.15.107:0/2625340722 >> 172.21.15.107:0/2625340722 conn(0x7f1e080c49b0 msgr2=0x7f1e080c4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:35.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 -- 172.21.15.107:0/2625340722 shutdown_connections 2023-10-25T00:22:35.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 -- 172.21.15.107:0/2625340722 wait complete. 2023-10-25T00:22:35.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 Processor -- start 2023-10-25T00:22:35.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 -- start start 2023-10-25T00:22:35.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e0806ddf0 0x7f1e08145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e080a9450 0x7f1e080c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e0814c850 0x7f1e080acb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080ac020 con 0x7f1e080a9450 2023-10-25T00:22:35.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e08145bc0 con 0x7f1e0806ddf0 2023-10-25T00:22:35.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e081ed9a0 con 0x7f1e0814c850 2023-10-25T00:22:35.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e080a9450 0x7f1e080c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e07fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e080a9450 0x7f1e080c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45550/0 (socket says 172.21.15.107:45550) 2023-10-25T00:22:35.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e07fff640 1 -- 172.21.15.107:0/1256104480 learned_addr learned my addr 172.21.15.107:0/1256104480 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:35.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0d31f640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e0814c850 0x7f1e080acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0cb1e640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e0806ddf0 0x7f1e08145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e07fff640 1 -- 172.21.15.107:0/1256104480 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e0814c850 msgr2=0x7f1e080acb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e07fff640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1e0814c850 0x7f1e080acb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e07fff640 1 -- 172.21.15.107:0/1256104480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e0806ddf0 msgr2=0x7f1e08145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e07fff640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e0806ddf0 0x7f1e08145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e07fff640 1 -- 172.21.15.107:0/1256104480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e00009b30 con 0x7f1e080a9450 2023-10-25T00:22:35.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e07fff640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e080a9450 0x7f1e080c12a0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f1df400d800 tx=0x7f1df400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e057fa640 1 -- 172.21.15.107:0/1256104480 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df400b6c0 con 0x7f1e080a9450 2023-10-25T00:22:35.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e057fa640 1 -- 172.21.15.107:0/1256104480 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df400b820 con 0x7f1e080a9450 2023-10-25T00:22:35.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 -- 172.21.15.107:0/1256104480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e08152d10 con 0x7f1e080a9450 2023-10-25T00:22:35.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e057fa640 1 -- 172.21.15.107:0/1256104480 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df400bb90 con 0x7f1e080a9450 2023-10-25T00:22:35.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.724+0000 7f1e0ed82640 1 -- 172.21.15.107:0/1256104480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e081ed350 con 0x7f1e080a9450 2023-10-25T00:22:35.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.728+0000 7f1e0ed82640 1 -- 172.21.15.107:0/1256104480 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1e080746e0 con 0x7f1e080a9450 2023-10-25T00:22:35.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.728+0000 7f1e057fa640 1 -- 172.21.15.107:0/1256104480 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1df400bcf0 con 0x7f1e080a9450 2023-10-25T00:22:35.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.728+0000 7f1e057fa640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1dec065e60 0x7f1dec068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.745 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.728+0000 7f1e057fa640 1 -- 172.21.15.107:0/1256104480 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f1df4085ae0 con 0x7f1e080a9450 2023-10-25T00:22:35.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.728+0000 7f1e0cb1e640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1dec065e60 0x7f1dec068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.728+0000 7f1e057fa640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] conn(0x7f1dec073100 0x7f1dec075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.728+0000 7f1e057fa640 1 -- 172.21.15.107:0/1256104480 --> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1dec075c30 con 0x7f1dec073100 2023-10-25T00:22:35.746 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.728+0000 7f1e057fa640 1 -- 172.21.15.107:0/1256104480 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1df4085ef0 con 0x7f1e080a9450 2023-10-25T00:22:35.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.728+0000 7f1e0cb1e640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1dec065e60 0x7f1dec068320 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc000f70 tx=0x7f1dfc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.728+0000 7f1e0d31f640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] conn(0x7f1dec073100 0x7f1dec075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.728+0000 7f1e0d31f640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] conn(0x7f1dec073100 0x7f1dec075560 crc :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.747 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.732+0000 7f1e057fa640 1 -- 172.21.15.107:0/1256104480 <== osd.4 v2:172.21.15.110:6801/37764 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1dec075c30 con 0x7f1dec073100 2023-10-25T00:22:35.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0ed82640 1 -- 172.21.15.107:0/1256104480 --> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1e081edb10 con 0x7f1dec073100 2023-10-25T00:22:35.748 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e057fa640 1 -- 172.21.15.107:0/1256104480 <== osd.4 v2:172.21.15.110:6801/37764 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1e081edb10 con 0x7f1dec073100 2023-10-25T00:22:35.748 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-10-25T00:22:35.749 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-25T00:22:35.749 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-10-25T00:22:35.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0ed82640 1 -- 172.21.15.107:0/1256104480 >> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] conn(0x7f1dec073100 msgr2=0x7f1dec075560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0ed82640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.110:6801/37764,v1:172.21.15.110:6803/37764] conn(0x7f1dec073100 0x7f1dec075560 crc :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0ed82640 1 -- 172.21.15.107:0/1256104480 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1dec065e60 msgr2=0x7f1dec068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0ed82640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1dec065e60 0x7f1dec068320 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc000f70 tx=0x7f1dfc009040 comp rx=0 tx=0).stop 2023-10-25T00:22:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0ed82640 1 -- 172.21.15.107:0/1256104480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e080a9450 msgr2=0x7f1e080c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0ed82640 1 --2- 172.21.15.107:0/1256104480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e080a9450 0x7f1e080c12a0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f1df400d800 tx=0x7f1df400dcc0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0d31f640 1 -- 172.21.15.107:0/1256104480 reap_dead start 2023-10-25T00:22:35.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0ed82640 1 -- 172.21.15.107:0/1256104480 shutdown_connections 2023-10-25T00:22:35.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0ed82640 1 -- 172.21.15.107:0/1256104480 >> 172.21.15.107:0/1256104480 conn(0x7f1e080c49b0 msgr2=0x7f1e081550c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:35.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0ed82640 1 -- 172.21.15.107:0/1256104480 shutdown_connections 2023-10-25T00:22:35.752 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.744+0000 7f1e0ed82640 1 -- 172.21.15.107:0/1256104480 wait complete. 2023-10-25T00:22:35.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-25T00:22:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7fafa2632640 1 Processor -- start 2023-10-25T00:22:35.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7fafa2632640 1 -- start start 2023-10-25T00:22:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7fafa2632640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faf9c0abfe0 0x7faf9c152670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7fafa2632640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faf9c0c3450 0x7faf9c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7fafa2632640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9c0aa680 con 0x7faf9c06a3d0 2023-10-25T00:22:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7fafa2632640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9c152c80 con 0x7faf9c0c3450 2023-10-25T00:22:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7fafa2632640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9c152f60 con 0x7faf9c0abfe0 2023-10-25T00:22:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7faf9b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faf9c0c3450 0x7faf9c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7faf9b7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7faf9c0c3450 0x7faf9c069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45566/0 (socket says 172.21.15.107:45566) 2023-10-25T00:22:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7faf9b7fe640 1 -- 172.21.15.107:0/1263324131 learned_addr learned my addr 172.21.15.107:0/1263324131 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7faf9bfff640 1 --2- 172.21.15.107:0/1263324131 >> v2:172.21.15.110:3300/0 conn(0x7faf9c0abfe0 0x7faf9c152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7faf9b7fe640 1 -- 172.21.15.107:0/1263324131 >> v2:172.21.15.110:3300/0 conn(0x7faf9c0abfe0 msgr2=0x7faf9c152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7faf9b7fe640 1 --2- 172.21.15.107:0/1263324131 >> v2:172.21.15.110:3300/0 conn(0x7faf9c0abfe0 0x7faf9c152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7faf9b7fe640 1 -- 172.21.15.107:0/1263324131 >> v1:172.21.15.110:6789/0 conn(0x7faf9c06a3d0 legacy=0x7faf9c156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.891 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.888+0000 7faf9b7fe640 1 -- 172.21.15.107:0/1263324131 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9c153240 con 0x7faf9c0c3450 2023-10-25T00:22:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7faf9b7fe640 1 --2- 172.21.15.107:0/1263324131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf9c0c3450 0x7faf9c069e90 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7faf9c069790 tx=0x7faf8c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6941e74d26a196a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7faf9affd640 1 -- 172.21.15.107:0/1263324131 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c004230 con 0x7faf9c0c3450 2023-10-25T00:22:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7faf9affd640 1 -- 172.21.15.107:0/1263324131 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8c0a7050 con 0x7faf9c0c3450 2023-10-25T00:22:35.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7faf9affd640 1 -- 172.21.15.107:0/1263324131 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8c003620 con 0x7faf9c0c3450 2023-10-25T00:22:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7fafa2632640 1 -- 172.21.15.107:0/1263324131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf9c0c3450 msgr2=0x7faf9c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7fafa2632640 1 --2- 172.21.15.107:0/1263324131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf9c0c3450 0x7faf9c069e90 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7faf9c069790 tx=0x7faf8c01c440 comp rx=0 tx=0).stop 2023-10-25T00:22:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7fafa2632640 1 -- 172.21.15.107:0/1263324131 shutdown_connections 2023-10-25T00:22:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7fafa2632640 1 --2- 172.21.15.107:0/1263324131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf9c0c3450 0x7faf9c069e90 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7fafa2632640 1 --2- 172.21.15.107:0/1263324131 >> v2:172.21.15.110:3300/0 conn(0x7faf9c0abfe0 0x7faf9c152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7fafa2632640 1 -- 172.21.15.107:0/1263324131 >> 172.21.15.107:0/1263324131 conn(0x7faf9c0c3960 msgr2=0x7faf9c0c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7fafa2632640 1 -- 172.21.15.107:0/1263324131 shutdown_connections 2023-10-25T00:22:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7fafa2632640 1 -- 172.21.15.107:0/1263324131 wait complete. 2023-10-25T00:22:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7fafa2632640 1 Processor -- start 2023-10-25T00:22:35.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.892+0000 7fafa2632640 1 -- start start 2023-10-25T00:22:35.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7fafa2632640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf9c06a3d0 0x7faf9c1eee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7fafa2632640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf9c0abfe0 0x7faf9c151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7fafa2632640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf9c0c3450 0x7faf9c1eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7fafa2632640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9c1eea40 con 0x7faf9c06a3d0 2023-10-25T00:22:35.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7fafa2632640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9c0c2570 con 0x7faf9c0abfe0 2023-10-25T00:22:35.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7fafa2632640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9c144e50 con 0x7faf9c0c3450 2023-10-25T00:22:35.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf9c06a3d0 0x7faf9c1eee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf9c06a3d0 0x7faf9c1eee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45572/0 (socket says 172.21.15.107:45572) 2023-10-25T00:22:35.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9bfff640 1 -- 172.21.15.107:0/1518132496 learned_addr learned my addr 172.21.15.107:0/1518132496 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:35.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9bfff640 1 -- 172.21.15.107:0/1518132496 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf9c0c3450 msgr2=0x7faf9c1eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7fafa0bcf640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf9c0c3450 0x7faf9c1eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9bfff640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf9c0c3450 0x7faf9c1eaf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9bfff640 1 -- 172.21.15.107:0/1518132496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf9c0abfe0 msgr2=0x7faf9c151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9b7fe640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf9c0abfe0 0x7faf9c151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9bfff640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf9c0abfe0 0x7faf9c151020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9bfff640 1 -- 172.21.15.107:0/1518132496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c0094d0 con 0x7faf9c06a3d0 2023-10-25T00:22:35.902 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9bfff640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf9c06a3d0 0x7faf9c1eee60 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7faf8400d5f0 tx=0x7faf8400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf98ff9640 1 -- 172.21.15.107:0/1518132496 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf840125f0 con 0x7faf9c06a3d0 2023-10-25T00:22:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf98ff9640 1 -- 172.21.15.107:0/1518132496 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf84012750 con 0x7faf9c06a3d0 2023-10-25T00:22:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7fafa2632640 1 -- 172.21.15.107:0/1518132496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9c1ef3a0 con 0x7faf9c06a3d0 2023-10-25T00:22:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf98ff9640 1 -- 172.21.15.107:0/1518132496 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf84012b00 con 0x7faf9c06a3d0 2023-10-25T00:22:35.903 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7fafa2632640 1 -- 172.21.15.107:0/1518132496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf9c0c2290 con 0x7faf9c06a3d0 2023-10-25T00:22:35.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7fafa2632640 1 -- 172.21.15.107:0/1518132496 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9c1eecb0 con 0x7faf9c06a3d0 2023-10-25T00:22:35.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf98ff9640 1 -- 172.21.15.107:0/1518132496 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7faf84012c60 con 0x7faf9c06a3d0 2023-10-25T00:22:35.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf98ff9640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf7c065e90 0x7faf7c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf98ff9640 1 -- 172.21.15.107:0/1518132496 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7faf84085970 con 0x7faf9c06a3d0 2023-10-25T00:22:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9b7fe640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf7c065e90 0x7faf7c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:35.905 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.896+0000 7faf9b7fe640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf7c065e90 0x7faf7c068350 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0ad000 tx=0x7faf8c0a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:35.906 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.900+0000 7faf98ff9640 1 -- 172.21.15.107:0/1518132496 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7faf84051ed0 con 0x7faf9c06a3d0 2023-10-25T00:22:35.915 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.916+0000 7fd8b71a2640 1 -- 172.21.15.107:0/499576620 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8b00c1bc0 con 0x7fd8b00720c0 2023-10-25T00:22:35.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.916+0000 7fd8ad7fa640 1 -- 172.21.15.107:0/499576620 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5768 (secure 0 0 0) 0x7fd8a0017020 con 0x7fd8b00720c0 2023-10-25T00:22:35.918 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:35.918 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:21:42.241858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9009":{"gid":9009,"name":"c","rank":0,"incarnation":226,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:22:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 230 2023-10-25T00:22:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 -- 172.21.15.107:0/499576620 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd890065e90 msgr2=0x7fd890068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.919 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd890065e90 0x7fd890068350 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00c43b0 tx=0x7fd898008040 comp rx=0 tx=0).stop 2023-10-25T00:22:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 -- 172.21.15.107:0/499576620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b00720c0 msgr2=0x7fd8b01ed500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b00720c0 0x7fd8b01ed500 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c3c0 tx=0x7fd8a000c880 comp rx=0 tx=0).stop 2023-10-25T00:22:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 -- 172.21.15.107:0/499576620 shutdown_connections 2023-10-25T00:22:35.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd890065e90 0x7fd890068350 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd8b0157e50 0x7fd8b00aab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd8b013dc20 0x7fd8b00c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 --2- 172.21.15.107:0/499576620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8b00720c0 0x7fd8b01ed500 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 -- 172.21.15.107:0/499576620 >> 172.21.15.107:0/499576620 conn(0x7fd8b00c39a0 msgr2=0x7fd8b015e860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:35.921 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 -- 172.21.15.107:0/499576620 shutdown_connections 2023-10-25T00:22:35.922 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:35.920+0000 7fd8b71a2640 1 -- 172.21.15.107:0/499576620 wait complete. 2023-10-25T00:22:35.935 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:22:36.058 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.056+0000 7fafa2632640 1 -- 172.21.15.107:0/1518132496 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7faf9c0ac3c0 con 0x7faf7c065e90 2023-10-25T00:22:36.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7faf98ff9640 1 -- 172.21.15.107:0/1518132496 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7faf8c004250 con 0x7faf7c065e90 2023-10-25T00:22:36.059 INFO:teuthology.orchestra.run.smithi107.stderr:instructed osd(s) 4 to deep-scrub 2023-10-25T00:22:36.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 -- 172.21.15.107:0/1518132496 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf7c065e90 msgr2=0x7faf7c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf7c065e90 0x7faf7c068350 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0ad000 tx=0x7faf8c0a6000 comp rx=0 tx=0).stop 2023-10-25T00:22:36.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 -- 172.21.15.107:0/1518132496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf9c06a3d0 msgr2=0x7faf9c1eee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf9c06a3d0 0x7faf9c1eee60 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7faf8400d5f0 tx=0x7faf8400dab0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 -- 172.21.15.107:0/1518132496 shutdown_connections 2023-10-25T00:22:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7faf7c065e90 0x7faf7c068350 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7faf9c0c3450 0x7faf9c1eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faf9c0abfe0 0x7faf9c151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 --2- 172.21.15.107:0/1518132496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf9c06a3d0 0x7faf9c1eee60 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 -- 172.21.15.107:0/1518132496 >> 172.21.15.107:0/1518132496 conn(0x7faf9c0c3960 msgr2=0x7faf9c0aa990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 -- 172.21.15.107:0/1518132496 shutdown_connections 2023-10-25T00:22:36.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.060+0000 7fafa2632640 1 -- 172.21.15.107:0/1518132496 wait complete. 2023-10-25T00:22:36.078 INFO:tasks.ceph:Scrubbing osd.5 2023-10-25T00:22:36.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-25T00:22:36.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.204+0000 7fec60652640 1 Processor -- start 2023-10-25T00:22:36.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.204+0000 7fec60652640 1 -- start start 2023-10-25T00:22:36.206 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.204+0000 7fec60652640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fec5814fdc0 0x7fec5806d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.204+0000 7fec60652640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fec5806d5a0 0x7fec5806edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.204+0000 7fec60652640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec580ae000 con 0x7fec5806f310 2023-10-25T00:22:36.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.204+0000 7fec60652640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec581501a0 con 0x7fec5814fdc0 2023-10-25T00:22:36.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.204+0000 7fec60652640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec580bce20 con 0x7fec5806d5a0 2023-10-25T00:22:36.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.204+0000 7fec5e3ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fec5814fdc0 0x7fec5806d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.204+0000 7fec5e3ee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fec5814fdc0 0x7fec5806d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45584/0 (socket says 172.21.15.107:45584) 2023-10-25T00:22:36.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.204+0000 7fec5e3ee640 1 -- 172.21.15.107:0/2251975696 learned_addr learned my addr 172.21.15.107:0/2251975696 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:36.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec5dbed640 1 --2- 172.21.15.107:0/2251975696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fec5806d5a0 0x7fec5806edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec5e3ee640 1 -- 172.21.15.107:0/2251975696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fec5806d5a0 msgr2=0x7fec5806edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:36.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec5e3ee640 1 --2- 172.21.15.107:0/2251975696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fec5806d5a0 0x7fec5806edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec5e3ee640 1 -- 172.21.15.107:0/2251975696 >> v1:172.21.15.110:6789/0 conn(0x7fec5806f310 legacy=0x7fec58152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec5e3ee640 1 -- 172.21.15.107:0/2251975696 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec580bd070 con 0x7fec5814fdc0 2023-10-25T00:22:36.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec5e3ee640 1 --2- 172.21.15.107:0/2251975696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec5814fdc0 0x7fec5806d060 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fec54009e50 tx=0x7fec5401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a14578312e3b64 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec5d3ec640 1 -- 172.21.15.107:0/2251975696 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec540a56e0 con 0x7fec5814fdc0 2023-10-25T00:22:36.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec5d3ec640 1 -- 172.21.15.107:0/2251975696 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec540a5840 con 0x7fec5814fdc0 2023-10-25T00:22:36.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec5d3ec640 1 -- 172.21.15.107:0/2251975696 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec540a5bd0 con 0x7fec5814fdc0 2023-10-25T00:22:36.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec60652640 1 -- 172.21.15.107:0/2251975696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec5814fdc0 msgr2=0x7fec5806d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec60652640 1 --2- 172.21.15.107:0/2251975696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec5814fdc0 0x7fec5806d060 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fec54009e50 tx=0x7fec5401ca10 comp rx=0 tx=0).stop 2023-10-25T00:22:36.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec60652640 1 -- 172.21.15.107:0/2251975696 shutdown_connections 2023-10-25T00:22:36.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec60652640 1 --2- 172.21.15.107:0/2251975696 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fec5806d5a0 0x7fec5806edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec60652640 1 --2- 172.21.15.107:0/2251975696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec5814fdc0 0x7fec5806d060 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec60652640 1 -- 172.21.15.107:0/2251975696 >> 172.21.15.107:0/2251975696 conn(0x7fec580c49a0 msgr2=0x7fec580c4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:36.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec60652640 1 -- 172.21.15.107:0/2251975696 shutdown_connections 2023-10-25T00:22:36.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.208+0000 7fec60652640 1 -- 172.21.15.107:0/2251975696 wait complete. 2023-10-25T00:22:36.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec60652640 1 Processor -- start 2023-10-25T00:22:36.215 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec60652640 1 -- start start 2023-10-25T00:22:36.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec60652640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fec5806d5a0 0x7fec5814d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec60652640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5806f310 0x7fec581ece30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec60652640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec5814fdc0 0x7fec581ed370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec60652640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec581f0e00 con 0x7fec5814fdc0 2023-10-25T00:22:36.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec60652640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5814a3f0 con 0x7fec5806f310 2023-10-25T00:22:36.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec60652640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5806b0f0 con 0x7fec5806d5a0 2023-10-25T00:22:36.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec5ebef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec5814fdc0 0x7fec581ed370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec5ebef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec5814fdc0 0x7fec581ed370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45598/0 (socket says 172.21.15.107:45598) 2023-10-25T00:22:36.217 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec5ebef640 1 -- 172.21.15.107:0/895130053 learned_addr learned my addr 172.21.15.107:0/895130053 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:36.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec5e3ee640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fec5806d5a0 0x7fec5814d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec5dbed640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5806f310 0x7fec581ece30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec5ebef640 1 -- 172.21.15.107:0/895130053 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fec5806d5a0 msgr2=0x7fec5814d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec5ebef640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fec5806d5a0 0x7fec5814d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec5ebef640 1 -- 172.21.15.107:0/895130053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5806f310 msgr2=0x7fec581ece30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec5ebef640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec5806f310 0x7fec581ece30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec5ebef640 1 -- 172.21.15.107:0/895130053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec54009b30 con 0x7fec5814fdc0 2023-10-25T00:22:36.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec5ebef640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec5814fdc0 0x7fec581ed370 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fec5000c430 tx=0x7fec5000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec4effd640 1 -- 172.21.15.107:0/895130053 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec50005220 con 0x7fec5814fdc0 2023-10-25T00:22:36.219 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec4effd640 1 -- 172.21.15.107:0/895130053 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec50005380 con 0x7fec5814fdc0 2023-10-25T00:22:36.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec60652640 1 -- 172.21.15.107:0/895130053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec581f1290 con 0x7fec5814fdc0 2023-10-25T00:22:36.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec4effd640 1 -- 172.21.15.107:0/895130053 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec50007270 con 0x7fec5814fdc0 2023-10-25T00:22:36.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec60652640 1 -- 172.21.15.107:0/895130053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec581f07b0 con 0x7fec5814fdc0 2023-10-25T00:22:36.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec60652640 1 -- 172.21.15.107:0/895130053 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec580bdd60 con 0x7fec5814fdc0 2023-10-25T00:22:36.220 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.212+0000 7fec4effd640 1 -- 172.21.15.107:0/895130053 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fec500073d0 con 0x7fec5814fdc0 2023-10-25T00:22:36.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.216+0000 7fec4effd640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fec3c065e60 0x7fec3c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.216+0000 7fec4effd640 1 -- 172.21.15.107:0/895130053 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fec500845e0 con 0x7fec5814fdc0 2023-10-25T00:22:36.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.216+0000 7fec5e3ee640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fec3c065e60 0x7fec3c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.216+0000 7fec4effd640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] conn(0x7fec3c073100 0x7fec3c075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.221 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.216+0000 7fec4effd640 1 -- 172.21.15.107:0/895130053 --> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec3c075c30 con 0x7fec3c073100 2023-10-25T00:22:36.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.216+0000 7fec4effd640 1 -- 172.21.15.107:0/895130053 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec500849f0 con 0x7fec5814fdc0 2023-10-25T00:22:36.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.216+0000 7fec5dbed640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] conn(0x7fec3c073100 0x7fec3c075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.216+0000 7fec5e3ee640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fec3c065e60 0x7fec3c068320 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fec540ae000 tx=0x7fec540a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.216+0000 7fec5dbed640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] conn(0x7fec3c073100 0x7fec3c075560 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.216+0000 7fec4effd640 1 -- 172.21.15.107:0/895130053 <== osd.5 v2:172.21.15.110:6816/37763 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fec3c075c30 con 0x7fec3c073100 2023-10-25T00:22:36.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.228+0000 7fec60652640 1 -- 172.21.15.107:0/895130053 --> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fec580bd910 con 0x7fec3c073100 2023-10-25T00:22:36.230 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec4effd640 1 -- 172.21.15.107:0/895130053 <== osd.5 v2:172.21.15.110:6816/37763 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fec4800bb00 con 0x7fec3c073100 2023-10-25T00:22:36.231 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-10-25T00:22:36.231 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-25T00:22:36.231 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-10-25T00:22:36.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec60652640 1 -- 172.21.15.107:0/895130053 >> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] conn(0x7fec3c073100 msgr2=0x7fec3c075560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec60652640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.110:6816/37763,v1:172.21.15.110:6817/37763] conn(0x7fec3c073100 0x7fec3c075560 crc :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.232 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec60652640 1 -- 172.21.15.107:0/895130053 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fec3c065e60 msgr2=0x7fec3c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec60652640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fec3c065e60 0x7fec3c068320 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fec540ae000 tx=0x7fec540a7070 comp rx=0 tx=0).stop 2023-10-25T00:22:36.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec60652640 1 -- 172.21.15.107:0/895130053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec5814fdc0 msgr2=0x7fec581ed370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec60652640 1 --2- 172.21.15.107:0/895130053 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec5814fdc0 0x7fec581ed370 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fec5000c430 tx=0x7fec5000c8f0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec5ebef640 1 -- 172.21.15.107:0/895130053 reap_dead start 2023-10-25T00:22:36.233 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec60652640 1 -- 172.21.15.107:0/895130053 shutdown_connections 2023-10-25T00:22:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec60652640 1 -- 172.21.15.107:0/895130053 >> 172.21.15.107:0/895130053 conn(0x7fec580c49a0 msgr2=0x7fec58151610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec60652640 1 -- 172.21.15.107:0/895130053 shutdown_connections 2023-10-25T00:22:36.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.232+0000 7fec60652640 1 -- 172.21.15.107:0/895130053 wait complete. 2023-10-25T00:22:36.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-25T00:22:36.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febde695640 1 Processor -- start 2023-10-25T00:22:36.352 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febde695640 1 -- start start 2023-10-25T00:22:36.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febde695640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febd8072720 0x7febd80c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febde695640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd80c1ce0 con 0x7febd80ab1a0 2023-10-25T00:22:36.355 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febde695640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febd80c0240 con 0x7febd80bceb0 2023-10-25T00:22:36.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febde695640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd80be1d0 con 0x7febd8072720 2023-10-25T00:22:36.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd7fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7febd80ab1a0 0x7febd80bfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35290/0 (socket says 172.21.15.107:35290) 2023-10-25T00:22:36.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd7fff640 1 -- 172.21.15.107:0/4181577853 learned_addr learned my addr 172.21.15.107:0/4181577853 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:36.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febcf7fe640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febd8072720 0x7febd80c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.356 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624813950 0 0) 0x7febd80c1ce0 con 0x7febd80ab1a0 2023-10-25T00:22:36.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc4003530 con 0x7febd80ab1a0 2023-10-25T00:22:36.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2210020975 0 0) 0x7febc4003530 con 0x7febd80ab1a0 2023-10-25T00:22:36.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febd8072720 msgr2=0x7febd80c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:36.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febd8072720 0x7febd80c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.357 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 >> v1:172.21.15.110:6789/0 conn(0x7febd80bceb0 legacy=0x7febd815c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd80c0880 con 0x7febd80ab1a0 2023-10-25T00:22:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7febc8001680 con 0x7febd80ab1a0 2023-10-25T00:22:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 >> v1:172.21.15.107:6789/0 conn(0x7febd80ab1a0 legacy=0x7febd80bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40039b0 0x7febc4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.358 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40047d0 0x7febc4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febc4005150 0x7febc400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc400bf80 con 0x7febc40039b0 2023-10-25T00:22:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc400c0c0 con 0x7febc4005150 2023-10-25T00:22:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc400c250 con 0x7febc40047d0 2023-10-25T00:22:36.359 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd7fff640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40039b0 0x7febc4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febcf7fe640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40047d0 0x7febc4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd7fff640 1 -- 172.21.15.107:0/4181577853 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40047d0 msgr2=0x7febc4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febdcc32640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febc4005150 0x7febc400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd7fff640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40047d0 0x7febc4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd7fff640 1 -- 172.21.15.107:0/4181577853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febc4005150 msgr2=0x7febc400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:36.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd7fff640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febc4005150 0x7febc400b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.361 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.352+0000 7febd7fff640 1 -- 172.21.15.107:0/4181577853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc4003530 con 0x7febc40039b0 2023-10-25T00:22:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febd7fff640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40039b0 0x7febc4003dd0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7febc800c270 tx=0x7febc801cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b23835a447c079f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.364 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8005ae0 con 0x7febc40039b0 2023-10-25T00:22:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc8005c40 con 0x7febc40039b0 2023-10-25T00:22:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febd77fe640 1 -- 172.21.15.107:0/4181577853 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc80a5800 con 0x7febc40039b0 2023-10-25T00:22:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febde695640 1 -- 172.21.15.107:0/4181577853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40039b0 msgr2=0x7febc4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febde695640 1 --2- 172.21.15.107:0/4181577853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febc40039b0 0x7febc4003dd0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7febc800c270 tx=0x7febc801cc10 comp rx=0 tx=0).stop 2023-10-25T00:22:36.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febdcc32640 1 -- 172.21.15.107:0/4181577853 reap_dead start 2023-10-25T00:22:36.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febde695640 1 -- 172.21.15.107:0/4181577853 shutdown_connections 2023-10-25T00:22:36.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febde695640 1 -- 172.21.15.107:0/4181577853 >> 172.21.15.107:0/4181577853 conn(0x7febd80c2860 msgr2=0x7febd80c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:36.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febde695640 1 -- 172.21.15.107:0/4181577853 shutdown_connections 2023-10-25T00:22:36.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febde695640 1 -- 172.21.15.107:0/4181577853 wait complete. 2023-10-25T00:22:36.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febde695640 1 Processor -- start 2023-10-25T00:22:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.356+0000 7febde695640 1 -- start start 2023-10-25T00:22:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febde695640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40039b0 0x7febd81542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febde695640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febd80a8d30 0x7febd81f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febde695640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febd81555e0 0x7febd81f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.367 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febde695640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd80a44c0 con 0x7febd81555e0 2023-10-25T00:22:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febde695640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd80aa6b0 con 0x7febd80a8d30 2023-10-25T00:22:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febde695640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd80aa990 con 0x7febc40039b0 2023-10-25T00:22:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febdcc32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febd81555e0 0x7febd81f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febdcc32640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febd81555e0 0x7febd81f0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45612/0 (socket says 172.21.15.107:45612) 2023-10-25T00:22:36.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febdcc32640 1 -- 172.21.15.107:0/1664753563 learned_addr learned my addr 172.21.15.107:0/1664753563 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febcf7fe640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febd80a8d30 0x7febd81f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febd7fff640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40039b0 0x7febd81542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febdcc32640 1 -- 172.21.15.107:0/1664753563 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40039b0 msgr2=0x7febd81542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febdcc32640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40039b0 0x7febd81542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febdcc32640 1 -- 172.21.15.107:0/1664753563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febd80a8d30 msgr2=0x7febd81f0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febdcc32640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febd80a8d30 0x7febd81f0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febdcc32640 1 -- 172.21.15.107:0/1664753563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd81f0120 con 0x7febd81555e0 2023-10-25T00:22:36.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febdcc32640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febd81555e0 0x7febd81f0b80 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7febd806e610 tx=0x7febd000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febd57fa640 1 -- 172.21.15.107:0/1664753563 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0003b80 con 0x7febd81555e0 2023-10-25T00:22:36.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febd57fa640 1 -- 172.21.15.107:0/1664753563 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd0003ce0 con 0x7febd81555e0 2023-10-25T00:22:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febde695640 1 -- 172.21.15.107:0/1664753563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd8154810 con 0x7febd81555e0 2023-10-25T00:22:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febd57fa640 1 -- 172.21.15.107:0/1664753563 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0005310 con 0x7febd81555e0 2023-10-25T00:22:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febde695640 1 -- 172.21.15.107:0/1664753563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd80a4710 con 0x7febd81555e0 2023-10-25T00:22:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febde695640 1 -- 172.21.15.107:0/1664753563 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd8156ba0 con 0x7febd81555e0 2023-10-25T00:22:36.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febd57fa640 1 -- 172.21.15.107:0/1664753563 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7febd0005470 con 0x7febd81555e0 2023-10-25T00:22:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febd57fa640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7febb8065e60 0x7febb8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febd57fa640 1 -- 172.21.15.107:0/1664753563 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7febd0084c80 con 0x7febd81555e0 2023-10-25T00:22:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.360+0000 7febd7fff640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7febb8065e60 0x7febb8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.364+0000 7febd7fff640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7febb8065e60 0x7febb8068320 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7febc80ab000 tx=0x7febc80a7070 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.373 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.368+0000 7febd57fa640 1 -- 172.21.15.107:0/1664753563 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7febd00510c0 con 0x7febd81555e0 2023-10-25T00:22:36.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.520+0000 7febde695640 1 -- 172.21.15.107:0/1664753563 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7febd80bf5a0 con 0x7febb8065e60 2023-10-25T00:22:36.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.520+0000 7febd57fa640 1 -- 172.21.15.107:0/1664753563 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7febc80b5040 con 0x7febb8065e60 2023-10-25T00:22:36.521 INFO:teuthology.orchestra.run.smithi107.stderr:instructed osd(s) 5 to deep-scrub 2023-10-25T00:22:36.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 -- 172.21.15.107:0/1664753563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7febb8065e60 msgr2=0x7febb8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7febb8065e60 0x7febb8068320 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7febc80ab000 tx=0x7febc80a7070 comp rx=0 tx=0).stop 2023-10-25T00:22:36.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 -- 172.21.15.107:0/1664753563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febd81555e0 msgr2=0x7febd81f0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febd81555e0 0x7febd81f0b80 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7febd806e610 tx=0x7febd000dc00 comp rx=0 tx=0).stop 2023-10-25T00:22:36.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 -- 172.21.15.107:0/1664753563 shutdown_connections 2023-10-25T00:22:36.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7febd81555e0 0x7febd81f0b80 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7febd80a8d30 0x7febd81f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7febb8065e60 0x7febb8068320 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 --2- 172.21.15.107:0/1664753563 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7febc40039b0 0x7febd81542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 -- 172.21.15.107:0/1664753563 >> 172.21.15.107:0/1664753563 conn(0x7febd80c2860 msgr2=0x7febd80a7120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:36.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 -- 172.21.15.107:0/1664753563 shutdown_connections 2023-10-25T00:22:36.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.524+0000 7febde695640 1 -- 172.21.15.107:0/1664753563 wait complete. 2023-10-25T00:22:36.538 INFO:tasks.ceph:Scrubbing osd.6 2023-10-25T00:22:36.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-25T00:22:36.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50bc0f2640 1 Processor -- start 2023-10-25T00:22:36.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50bc0f2640 1 -- start start 2023-10-25T00:22:36.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50bc0f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b40af9a0 0x7f50b40ca200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50bc0f2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f50b40ca740 0x7f50b40c7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50bc0f2640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b40afd80 con 0x7f50b40b0140 2023-10-25T00:22:36.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50bc0f2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b40c75d0 con 0x7f50b40ca740 2023-10-25T00:22:36.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50bc0f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4155310 con 0x7f50b40af9a0 2023-10-25T00:22:36.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50b9e8e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b40af9a0 0x7f50b40ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50b968d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f50b40b0140 0x7f50b40c6980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:42474/0 (socket says 172.21.15.107:42474) 2023-10-25T00:22:36.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50ba68f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f50b40ca740 0x7f50b40c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50b968d640 1 -- 172.21.15.107:0/2458674754 learned_addr learned my addr 172.21.15.107:0/2458674754 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:36.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50b8e8c640 1 -- 172.21.15.107:0/2458674754 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199956332 0 0) 0x7f50b40afd80 con 0x7f50b40b0140 2023-10-25T00:22:36.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50b8e8c640 1 -- 172.21.15.107:0/2458674754 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50a0003530 con 0x7f50b40b0140 2023-10-25T00:22:36.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50b9e8e640 1 -- 172.21.15.107:0/2458674754 >> v2:172.21.15.110:3300/0 conn(0x7f50b40ca740 msgr2=0x7f50b40c7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:36.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50b9e8e640 1 --2- 172.21.15.107:0/2458674754 >> v2:172.21.15.110:3300/0 conn(0x7f50b40ca740 0x7f50b40c7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50b9e8e640 1 -- 172.21.15.107:0/2458674754 >> v1:172.21.15.110:6789/0 conn(0x7f50b40b0140 legacy=0x7f50b40c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.664+0000 7f50b9e8e640 1 -- 172.21.15.107:0/2458674754 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b4155580 con 0x7f50b40af9a0 2023-10-25T00:22:36.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50b9e8e640 1 --2- 172.21.15.107:0/2458674754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b40af9a0 0x7f50b40ca200 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f50b40c7710 tx=0x7f50b001ceb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fb3fef925837dec server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50b8e8c640 1 -- 172.21.15.107:0/2458674754 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00ab670 con 0x7f50b40af9a0 2023-10-25T00:22:36.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50b8e8c640 1 -- 172.21.15.107:0/2458674754 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b00ab7d0 con 0x7f50b40af9a0 2023-10-25T00:22:36.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50b8e8c640 1 -- 172.21.15.107:0/2458674754 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00abb40 con 0x7f50b40af9a0 2023-10-25T00:22:36.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2458674754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b40af9a0 msgr2=0x7f50b40ca200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50bc0f2640 1 --2- 172.21.15.107:0/2458674754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b40af9a0 0x7f50b40ca200 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f50b40c7710 tx=0x7f50b001ceb0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2458674754 shutdown_connections 2023-10-25T00:22:36.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50bc0f2640 1 --2- 172.21.15.107:0/2458674754 >> v2:172.21.15.110:3300/0 conn(0x7f50b40ca740 0x7f50b40c7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50bc0f2640 1 --2- 172.21.15.107:0/2458674754 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b40af9a0 0x7f50b40ca200 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2458674754 >> 172.21.15.107:0/2458674754 conn(0x7f50b4071470 msgr2=0x7f50b4071880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:36.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2458674754 shutdown_connections 2023-10-25T00:22:36.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2458674754 wait complete. 2023-10-25T00:22:36.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.668+0000 7f50bc0f2640 1 Processor -- start 2023-10-25T00:22:36.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50bc0f2640 1 -- start start 2023-10-25T00:22:36.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50bc0f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50b40af9a0 0x7f50b41495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50bc0f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50b40b0140 0x7f50b4150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50bc0f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b40ca740 0x7f50b4150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50bc0f2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b40adf90 con 0x7f50b40af9a0 2023-10-25T00:22:36.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50bc0f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4149d60 con 0x7f50b40b0140 2023-10-25T00:22:36.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50bc0f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b414ce10 con 0x7f50b40ca740 2023-10-25T00:22:36.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50b9e8e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50b40af9a0 0x7f50b41495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50b9e8e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50b40af9a0 0x7f50b41495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45616/0 (socket says 172.21.15.107:45616) 2023-10-25T00:22:36.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50b9e8e640 1 -- 172.21.15.107:0/2438107188 learned_addr learned my addr 172.21.15.107:0/2438107188 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:36.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50b968d640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50b40b0140 0x7f50b4150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50ba68f640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b40ca740 0x7f50b4150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50b9e8e640 1 -- 172.21.15.107:0/2438107188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b40ca740 msgr2=0x7f50b4150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50b9e8e640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b40ca740 0x7f50b4150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50b9e8e640 1 -- 172.21.15.107:0/2438107188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50b40b0140 msgr2=0x7f50b4150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50b9e8e640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50b40b0140 0x7f50b4150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50b9e8e640 1 -- 172.21.15.107:0/2438107188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b0009420 con 0x7f50b40af9a0 2023-10-25T00:22:36.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50b9e8e640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50b40af9a0 0x7f50b41495d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f50b0000c10 tx=0x7f50b00191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50aa7fc640 1 -- 172.21.15.107:0/2438107188 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0004450 con 0x7f50b40af9a0 2023-10-25T00:22:36.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50aa7fc640 1 -- 172.21.15.107:0/2438107188 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b00ab940 con 0x7f50b40af9a0 2023-10-25T00:22:36.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2438107188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b40ada10 con 0x7f50b40af9a0 2023-10-25T00:22:36.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50aa7fc640 1 -- 172.21.15.107:0/2438107188 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00abcb0 con 0x7f50b40af9a0 2023-10-25T00:22:36.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2438107188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b40adcb0 con 0x7f50b40af9a0 2023-10-25T00:22:36.682 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2438107188 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f50b406b6c0 con 0x7f50b40af9a0 2023-10-25T00:22:36.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50aa7fc640 1 -- 172.21.15.107:0/2438107188 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f50b0003620 con 0x7f50b40af9a0 2023-10-25T00:22:36.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50aa7fc640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5098065e60 0x7f5098068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50b968d640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5098065e60 0x7f5098068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.683 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.672+0000 7f50aa7fc640 1 -- 172.21.15.107:0/2438107188 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f50b011ee50 con 0x7f50b40af9a0 2023-10-25T00:22:36.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.676+0000 7f50aa7fc640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] conn(0x7f5098073100 0x7f5098075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.676+0000 7f50aa7fc640 1 -- 172.21.15.107:0/2438107188 --> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5098075c30 con 0x7f5098073100 2023-10-25T00:22:36.684 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.676+0000 7f50aa7fc640 1 -- 172.21.15.107:0/2438107188 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50b011f260 con 0x7f50b40af9a0 2023-10-25T00:22:36.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.676+0000 7f50b968d640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5098065e60 0x7f5098068320 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f50a4005890 tx=0x7f50a4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.676+0000 7f50ba68f640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] conn(0x7f5098073100 0x7f5098075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.676+0000 7f50ba68f640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] conn(0x7f5098073100 0x7f5098075560 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.676+0000 7f50aa7fc640 1 -- 172.21.15.107:0/2438107188 <== osd.6 v2:172.21.15.110:6800/37765 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f5098075c30 con 0x7f5098073100 2023-10-25T00:22:36.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.688+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2438107188 --> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f50b4161990 con 0x7f5098073100 2023-10-25T00:22:36.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50aa7fc640 1 -- 172.21.15.107:0/2438107188 <== osd.6 v2:172.21.15.110:6800/37765 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f50ac00f040 con 0x7f5098073100 2023-10-25T00:22:36.691 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-10-25T00:22:36.691 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-25T00:22:36.691 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-10-25T00:22:36.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2438107188 >> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] conn(0x7f5098073100 msgr2=0x7f5098075560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50bc0f2640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.110:6800/37765,v1:172.21.15.110:6802/37765] conn(0x7f5098073100 0x7f5098075560 crc :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2438107188 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5098065e60 msgr2=0x7f5098068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50bc0f2640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5098065e60 0x7f5098068320 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f50a4005890 tx=0x7f50a4009040 comp rx=0 tx=0).stop 2023-10-25T00:22:36.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2438107188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50b40af9a0 msgr2=0x7f50b41495d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50bc0f2640 1 --2- 172.21.15.107:0/2438107188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50b40af9a0 0x7f50b41495d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f50b0000c10 tx=0x7f50b00191f0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50ba68f640 1 -- 172.21.15.107:0/2438107188 reap_dead start 2023-10-25T00:22:36.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2438107188 shutdown_connections 2023-10-25T00:22:36.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2438107188 >> 172.21.15.107:0/2438107188 conn(0x7f50b4071470 msgr2=0x7f50b40a9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:36.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2438107188 shutdown_connections 2023-10-25T00:22:36.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.692+0000 7f50bc0f2640 1 -- 172.21.15.107:0/2438107188 wait complete. 2023-10-25T00:22:36.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-25T00:22:36.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66c88d640 1 Processor -- start 2023-10-25T00:22:36.817 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66c88d640 1 -- start start 2023-10-25T00:22:36.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66c88d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb6680aa9f0 0x7fb668072210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66c88d640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6680befa0 con 0x7fb6680a9300 2023-10-25T00:22:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66c88d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6680a9b80 con 0x7fb6680c1b30 2023-10-25T00:22:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66c88d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6680aa0b0 con 0x7fb6680aa9f0 2023-10-25T00:22:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb666d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb6680a9300 0x7fb668155620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35304/0 (socket says 172.21.15.107:35304) 2023-10-25T00:22:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb666d9d640 1 -- 172.21.15.107:0/777578118 learned_addr learned my addr 172.21.15.107:0/777578118 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb665d9b640 1 --2- 172.21.15.107:0/777578118 >> v2:172.21.15.110:3300/0 conn(0x7fb6680aa9f0 0x7fb668072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270496047 0 0) 0x7fb6680befa0 con 0x7fb6680a9300 2023-10-25T00:22:36.829 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb650003530 con 0x7fb6680a9300 2023-10-25T00:22:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2329972969 0 0) 0x7fb650003530 con 0x7fb6680a9300 2023-10-25T00:22:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 >> v2:172.21.15.110:3300/0 conn(0x7fb6680aa9f0 msgr2=0x7fb668072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66559a640 1 --2- 172.21.15.107:0/777578118 >> v2:172.21.15.110:3300/0 conn(0x7fb6680aa9f0 0x7fb668072210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 >> v1:172.21.15.110:6789/0 conn(0x7fb6680c1b30 legacy=0x7fb6680aa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6680bf1f0 con 0x7fb6680a9300 2023-10-25T00:22:36.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fb660001a00 con 0x7fb6680a9300 2023-10-25T00:22:36.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.816+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 >> v1:172.21.15.107:6789/0 conn(0x7fb6680a9300 legacy=0x7fb668155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66559a640 1 --2- 172.21.15.107:0/777578118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb650003980 0x7fb650003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66559a640 1 --2- 172.21.15.107:0/777578118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 0x7fb650004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66559a640 1 --2- 172.21.15.107:0/777578118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6500052b0 0x7fb65000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6680aa0b0 con 0x7fb6500047d0 2023-10-25T00:22:36.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65000c180 con 0x7fb650003980 2023-10-25T00:22:36.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65000c340 con 0x7fb6500052b0 2023-10-25T00:22:36.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb665d9b640 1 --2- 172.21.15.107:0/777578118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 0x7fb650004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66659c640 1 --2- 172.21.15.107:0/777578118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb650003980 0x7fb650003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb666d9d640 1 --2- 172.21.15.107:0/777578118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6500052b0 0x7fb65000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb665d9b640 1 -- 172.21.15.107:0/777578118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6500052b0 msgr2=0x7fb65000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:36.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb665d9b640 1 --2- 172.21.15.107:0/777578118 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6500052b0 0x7fb65000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb665d9b640 1 -- 172.21.15.107:0/777578118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb650003980 msgr2=0x7fb650003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:36.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb665d9b640 1 --2- 172.21.15.107:0/777578118 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb650003980 0x7fb650003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb665d9b640 1 -- 172.21.15.107:0/777578118 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb650003530 con 0x7fb6500047d0 2023-10-25T00:22:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb666d9d640 1 -- 172.21.15.107:0/777578118 reap_dead start 2023-10-25T00:22:36.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb665d9b640 1 --2- 172.21.15.107:0/777578118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 0x7fb650004c50 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009670 tx=0x7fb65c01ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d210f4cf6855e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0aa450 con 0x7fb6500047d0 2023-10-25T00:22:36.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65c0aa5b0 con 0x7fb6500047d0 2023-10-25T00:22:36.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66559a640 1 -- 172.21.15.107:0/777578118 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0aa960 con 0x7fb6500047d0 2023-10-25T00:22:36.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66c88d640 1 -- 172.21.15.107:0/777578118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 msgr2=0x7fb650004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66c88d640 1 --2- 172.21.15.107:0/777578118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 0x7fb650004c50 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009670 tx=0x7fb65c01ec20 comp rx=0 tx=0).stop 2023-10-25T00:22:36.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66c88d640 1 -- 172.21.15.107:0/777578118 shutdown_connections 2023-10-25T00:22:36.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66c88d640 1 --2- 172.21.15.107:0/777578118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 0x7fb650004c50 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66c88d640 1 -- 172.21.15.107:0/777578118 >> 172.21.15.107:0/777578118 conn(0x7fb6680c2860 msgr2=0x7fb6680c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:36.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66c88d640 1 -- 172.21.15.107:0/777578118 shutdown_connections 2023-10-25T00:22:36.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.820+0000 7fb66c88d640 1 -- 172.21.15.107:0/777578118 wait complete. 2023-10-25T00:22:36.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66c88d640 1 Processor -- start 2023-10-25T00:22:36.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66c88d640 1 -- start start 2023-10-25T00:22:36.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66c88d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 0x7fb6681eea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66c88d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb668069d40 0x7fb6681eef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66c88d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6681529b0 0x7fb668152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66c88d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66806a290 con 0x7fb6500047d0 2023-10-25T00:22:36.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66c88d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6680706b0 con 0x7fb668069d40 2023-10-25T00:22:36.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66c88d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb668070990 con 0x7fb6681529b0 2023-10-25T00:22:36.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 0x7fb6681eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 0x7fb6681eea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45634/0 (socket says 172.21.15.107:45634) 2023-10-25T00:22:36.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66659c640 1 -- 172.21.15.107:0/3875613263 learned_addr learned my addr 172.21.15.107:0/3875613263 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66659c640 1 -- 172.21.15.107:0/3875613263 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6681529b0 msgr2=0x7fb668152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66659c640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6681529b0 0x7fb668152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66659c640 1 -- 172.21.15.107:0/3875613263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb668069d40 msgr2=0x7fb6681eef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66659c640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb668069d40 0x7fb6681eef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66659c640 1 -- 172.21.15.107:0/3875613263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb668070c70 con 0x7fb6500047d0 2023-10-25T00:22:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66659c640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 0x7fb6681eea50 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fb65800e550 tx=0x7fb65800ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb656ffd640 1 -- 172.21.15.107:0/3875613263 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6580052e0 con 0x7fb6500047d0 2023-10-25T00:22:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb656ffd640 1 -- 172.21.15.107:0/3875613263 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb658005440 con 0x7fb6500047d0 2023-10-25T00:22:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb656ffd640 1 -- 172.21.15.107:0/3875613263 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb658003820 con 0x7fb6500047d0 2023-10-25T00:22:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66c88d640 1 -- 172.21.15.107:0/3875613263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb668070f50 con 0x7fb6500047d0 2023-10-25T00:22:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66c88d640 1 -- 172.21.15.107:0/3875613263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb668153370 con 0x7fb6500047d0 2023-10-25T00:22:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb66c88d640 1 -- 172.21.15.107:0/3875613263 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb66804e130 con 0x7fb6500047d0 2023-10-25T00:22:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.824+0000 7fb656ffd640 1 -- 172.21.15.107:0/3875613263 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb658003980 con 0x7fb6500047d0 2023-10-25T00:22:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.828+0000 7fb656ffd640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb648065e60 0x7fb648068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:36.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.828+0000 7fb656ffd640 1 -- 172.21.15.107:0/3875613263 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fb6580848e0 con 0x7fb6500047d0 2023-10-25T00:22:36.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.828+0000 7fb665d9b640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb648065e60 0x7fb648068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:36.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.828+0000 7fb665d9b640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb648065e60 0x7fb648068320 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0b1000 tx=0x7fb65c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.832+0000 7fb656ffd640 1 -- 172.21.15.107:0/3875613263 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb658050c90 con 0x7fb6500047d0 2023-10-25T00:22:36.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.976+0000 7fb66c88d640 1 -- 172.21.15.107:0/3875613263 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fb668070820 con 0x7fb648065e60 2023-10-25T00:22:36.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.976+0000 7fb656ffd640 1 -- 172.21.15.107:0/3875613263 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb668070820 con 0x7fb648065e60 2023-10-25T00:22:36.977 INFO:teuthology.orchestra.run.smithi107.stderr:instructed osd(s) 6 to deep-scrub 2023-10-25T00:22:36.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 -- 172.21.15.107:0/3875613263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb648065e60 msgr2=0x7fb648068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb648065e60 0x7fb648068320 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0b1000 tx=0x7fb65c008040 comp rx=0 tx=0).stop 2023-10-25T00:22:36.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 -- 172.21.15.107:0/3875613263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 msgr2=0x7fb6681eea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:36.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 0x7fb6681eea50 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fb65800e550 tx=0x7fb65800ea10 comp rx=0 tx=0).stop 2023-10-25T00:22:36.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 -- 172.21.15.107:0/3875613263 shutdown_connections 2023-10-25T00:22:36.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb648065e60 0x7fb648068320 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6681529b0 0x7fb668152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb668069d40 0x7fb6681eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 --2- 172.21.15.107:0/3875613263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6500047d0 0x7fb6681eea50 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:36.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 -- 172.21.15.107:0/3875613263 >> 172.21.15.107:0/3875613263 conn(0x7fb6680c2860 msgr2=0x7fb6680c55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:36.981 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 -- 172.21.15.107:0/3875613263 shutdown_connections 2023-10-25T00:22:36.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:36.980+0000 7fb66c88d640 1 -- 172.21.15.107:0/3875613263 wait complete. 2023-10-25T00:22:36.995 INFO:tasks.ceph:Scrubbing osd.7 2023-10-25T00:22:36.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-25T00:22:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3e00e88640 1 Processor -- start 2023-10-25T00:22:37.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3e00e88640 1 -- start start 2023-10-25T00:22:37.109 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3e00e88640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3dfc151dd0 0x7f3dfc0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3e00e88640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3dfc155310 0x7f3dfc158150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3e00e88640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3dfc0c8100 0x7f3dfc158690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3e00e88640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc155820 con 0x7f3dfc155310 2023-10-25T00:22:37.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3e00e88640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc0c6cb0 con 0x7f3dfc151dd0 2023-10-25T00:22:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3e00e88640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0c6f90 con 0x7f3dfc0c8100 2023-10-25T00:22:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3df9d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3dfc155310 0x7f3dfc158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3df9d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f3dfc155310 0x7f3dfc158150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45646/0 (socket says 172.21.15.107:45646) 2023-10-25T00:22:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3df9d9b640 1 -- 172.21.15.107:0/2177167341 learned_addr learned my addr 172.21.15.107:0/2177167341 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:37.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3dfad9d640 1 --2- 172.21.15.107:0/2177167341 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3dfc0c8100 0x7f3dfc158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3df9d9b640 1 -- 172.21.15.107:0/2177167341 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3dfc0c8100 msgr2=0x7f3dfc158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3dfa59c640 1 --2- 172.21.15.107:0/2177167341 >> v2:172.21.15.110:3300/0 conn(0x7f3dfc151dd0 0x7f3dfc0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3df9d9b640 1 --2- 172.21.15.107:0/2177167341 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3dfc0c8100 0x7f3dfc158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3df9d9b640 1 -- 172.21.15.107:0/2177167341 >> v2:172.21.15.110:3300/0 conn(0x7f3dfc151dd0 msgr2=0x7f3dfc0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:37.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3df9d9b640 1 --2- 172.21.15.107:0/2177167341 >> v2:172.21.15.110:3300/0 conn(0x7f3dfc151dd0 0x7f3dfc0c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.108+0000 7f3df9d9b640 1 -- 172.21.15.107:0/2177167341 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc0c7270 con 0x7f3dfc155310 2023-10-25T00:22:37.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3df9d9b640 1 --2- 172.21.15.107:0/2177167341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dfc155310 0x7f3dfc158150 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc06cd00 tx=0x7f3dec01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02d85a9b8912166 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:37.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3df959a640 1 -- 172.21.15.107:0/2177167341 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec0ab650 con 0x7f3dfc155310 2023-10-25T00:22:37.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3df959a640 1 -- 172.21.15.107:0/2177167341 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dec0ab7b0 con 0x7f3dfc155310 2023-10-25T00:22:37.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3df959a640 1 -- 172.21.15.107:0/2177167341 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec0abb40 con 0x7f3dfc155310 2023-10-25T00:22:37.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3e00e88640 1 -- 172.21.15.107:0/2177167341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dfc155310 msgr2=0x7f3dfc158150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3e00e88640 1 --2- 172.21.15.107:0/2177167341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dfc155310 0x7f3dfc158150 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc06cd00 tx=0x7f3dec01c470 comp rx=0 tx=0).stop 2023-10-25T00:22:37.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3e00e88640 1 -- 172.21.15.107:0/2177167341 shutdown_connections 2023-10-25T00:22:37.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3e00e88640 1 --2- 172.21.15.107:0/2177167341 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3dfc0c8100 0x7f3dfc158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3e00e88640 1 --2- 172.21.15.107:0/2177167341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dfc155310 0x7f3dfc158150 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3e00e88640 1 --2- 172.21.15.107:0/2177167341 >> v2:172.21.15.110:3300/0 conn(0x7f3dfc151dd0 0x7f3dfc0c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3e00e88640 1 -- 172.21.15.107:0/2177167341 >> 172.21.15.107:0/2177167341 conn(0x7f3dfc0abdb0 msgr2=0x7f3dfc0ac1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3e00e88640 1 -- 172.21.15.107:0/2177167341 shutdown_connections 2023-10-25T00:22:37.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3e00e88640 1 -- 172.21.15.107:0/2177167341 wait complete. 2023-10-25T00:22:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3e00e88640 1 Processor -- start 2023-10-25T00:22:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.112+0000 7f3e00e88640 1 -- start start 2023-10-25T00:22:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3e00e88640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3dfc0c8100 0x7f3dfc1edc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3e00e88640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dfc151dd0 0x7f3dfc0bf410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3e00e88640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dfc155310 0x7f3dfc15e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3e00e88640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc153f30 con 0x7f3dfc155310 2023-10-25T00:22:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3e00e88640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0bf950 con 0x7f3dfc151dd0 2023-10-25T00:22:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3e00e88640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc152aa0 con 0x7f3dfc0c8100 2023-10-25T00:22:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dfc155310 0x7f3dfc15e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dfc155310 0x7f3dfc15e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45650/0 (socket says 172.21.15.107:45650) 2023-10-25T00:22:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfad9d640 1 -- 172.21.15.107:0/3463366191 learned_addr learned my addr 172.21.15.107:0/3463366191 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfa59c640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3dfc0c8100 0x7f3dfc1edc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3df9d9b640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dfc151dd0 0x7f3dfc0bf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfad9d640 1 -- 172.21.15.107:0/3463366191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3dfc0c8100 msgr2=0x7f3dfc1edc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfad9d640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3dfc0c8100 0x7f3dfc1edc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfad9d640 1 -- 172.21.15.107:0/3463366191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dfc151dd0 msgr2=0x7f3dfc0bf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfad9d640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3dfc151dd0 0x7f3dfc0bf410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfad9d640 1 -- 172.21.15.107:0/3463366191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dec0094d0 con 0x7f3dfc155310 2023-10-25T00:22:37.122 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfad9d640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dfc155310 0x7f3dfc15e6b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f3df400ccf0 tx=0x7f3df400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3deaffd640 1 -- 172.21.15.107:0/3463366191 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4003ba0 con 0x7f3dfc155310 2023-10-25T00:22:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3deaffd640 1 -- 172.21.15.107:0/3463366191 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df4003d00 con 0x7f3dfc155310 2023-10-25T00:22:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3deaffd640 1 -- 172.21.15.107:0/3463366191 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4004240 con 0x7f3dfc155310 2023-10-25T00:22:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3e00e88640 1 -- 172.21.15.107:0/3463366191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dfc1542f0 con 0x7f3dfc155310 2023-10-25T00:22:37.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3e00e88640 1 -- 172.21.15.107:0/3463366191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dfc152730 con 0x7f3dfc155310 2023-10-25T00:22:37.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3e00e88640 1 -- 172.21.15.107:0/3463366191 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3dfc06aa70 con 0x7f3dfc155310 2023-10-25T00:22:37.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3deaffd640 1 -- 172.21.15.107:0/3463366191 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f3df40043a0 con 0x7f3dfc155310 2023-10-25T00:22:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3deaffd640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3ddc065e60 0x7f3ddc068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfa59c640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3ddc065e60 0x7f3ddc068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3deaffd640 1 -- 172.21.15.107:0/3463366191 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f3df4012070 con 0x7f3dfc155310 2023-10-25T00:22:37.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3deaffd640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] conn(0x7f3ddc073100 0x7f3ddc075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3deaffd640 1 -- 172.21.15.107:0/3463366191 --> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ddc075c30 con 0x7f3ddc073100 2023-10-25T00:22:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3deaffd640 1 -- 172.21.15.107:0/3463366191 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3df4087850 con 0x7f3dfc155310 2023-10-25T00:22:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3dfa59c640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3ddc065e60 0x7f3ddc068320 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f3df000a970 tx=0x7f3df0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.116+0000 7f3df9d9b640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] conn(0x7f3ddc073100 0x7f3ddc075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.120+0000 7f3df9d9b640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] conn(0x7f3ddc073100 0x7f3ddc075560 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:37.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.120+0000 7f3deaffd640 1 -- 172.21.15.107:0/3463366191 <== osd.7 v2:172.21.15.110:6824/37766 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3ddc075c30 con 0x7f3ddc073100 2023-10-25T00:22:37.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3e00e88640 1 -- 172.21.15.107:0/3463366191 --> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3dfc154120 con 0x7f3ddc073100 2023-10-25T00:22:37.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3deaffd640 1 -- 172.21.15.107:0/3463366191 <== osd.7 v2:172.21.15.110:6824/37766 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f3dfc154120 con 0x7f3ddc073100 2023-10-25T00:22:37.133 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-10-25T00:22:37.134 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-25T00:22:37.134 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-10-25T00:22:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3e00e88640 1 -- 172.21.15.107:0/3463366191 >> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] conn(0x7f3ddc073100 msgr2=0x7f3ddc075560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3e00e88640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.110:6824/37766,v1:172.21.15.110:6825/37766] conn(0x7f3ddc073100 0x7f3ddc075560 crc :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3e00e88640 1 -- 172.21.15.107:0/3463366191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3ddc065e60 msgr2=0x7f3ddc068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3e00e88640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3ddc065e60 0x7f3ddc068320 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f3df000a970 tx=0x7f3df0008040 comp rx=0 tx=0).stop 2023-10-25T00:22:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3e00e88640 1 -- 172.21.15.107:0/3463366191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dfc155310 msgr2=0x7f3dfc15e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3e00e88640 1 --2- 172.21.15.107:0/3463366191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3dfc155310 0x7f3dfc15e6b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f3df400ccf0 tx=0x7f3df400a660 comp rx=0 tx=0).stop 2023-10-25T00:22:37.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3dfad9d640 1 -- 172.21.15.107:0/3463366191 reap_dead start 2023-10-25T00:22:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3e00e88640 1 -- 172.21.15.107:0/3463366191 shutdown_connections 2023-10-25T00:22:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3e00e88640 1 -- 172.21.15.107:0/3463366191 >> 172.21.15.107:0/3463366191 conn(0x7f3dfc0abdb0 msgr2=0x7f3dfc0c86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3e00e88640 1 -- 172.21.15.107:0/3463366191 shutdown_connections 2023-10-25T00:22:37.137 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.132+0000 7f3e00e88640 1 -- 172.21.15.107:0/3463366191 wait complete. 2023-10-25T00:22:37.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-25T00:22:37.263 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f588157a640 1 Processor -- start 2023-10-25T00:22:37.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f588157a640 1 -- start start 2023-10-25T00:22:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f588157a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f587c154c30 0x7f587c0a9550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f588157a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f587c071330 0x7f587c0a9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f588157a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c14f0f0 con 0x7f587c14ed10 2023-10-25T00:22:37.265 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f588157a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f587c1524e0 con 0x7f587c071330 2023-10-25T00:22:37.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f588157a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f587c151860 con 0x7f587c154c30 2023-10-25T00:22:37.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f587affd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f587c14ed10 0x7f587c0c2870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35308/0 (socket says 172.21.15.107:35308) 2023-10-25T00:22:37.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f587affd640 1 -- 172.21.15.107:0/311341021 learned_addr learned my addr 172.21.15.107:0/311341021 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:37.266 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f587b7fe640 1 --2- 172.21.15.107:0/311341021 >> v2:172.21.15.107:3300/0 conn(0x7f587c071330 0x7f587c0a9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f587a7fc640 1 --2- 172.21.15.107:0/311341021 >> v2:172.21.15.110:3300/0 conn(0x7f587c154c30 0x7f587c0a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f587b7fe640 1 -- 172.21.15.107:0/311341021 >> v2:172.21.15.110:3300/0 conn(0x7f587c154c30 msgr2=0x7f587c0a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f587b7fe640 1 --2- 172.21.15.107:0/311341021 >> v2:172.21.15.110:3300/0 conn(0x7f587c154c30 0x7f587c0a9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f587b7fe640 1 -- 172.21.15.107:0/311341021 >> v1:172.21.15.107:6789/0 conn(0x7f587c14ed10 legacy=0x7f587c0c2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.267 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.264+0000 7f587b7fe640 1 -- 172.21.15.107:0/311341021 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c151b40 con 0x7f587c071330 2023-10-25T00:22:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f587b7fe640 1 --2- 172.21.15.107:0/311341021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f587c071330 0x7f587c0a9a90 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f587c155cf0 tx=0x7f587401c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f05a29c79d8d0dd server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f5879ffb640 1 -- 172.21.15.107:0/311341021 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58740a5910 con 0x7f587c071330 2023-10-25T00:22:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f5879ffb640 1 -- 172.21.15.107:0/311341021 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58740a5a70 con 0x7f587c071330 2023-10-25T00:22:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f5879ffb640 1 -- 172.21.15.107:0/311341021 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58740a5e00 con 0x7f587c071330 2023-10-25T00:22:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 -- 172.21.15.107:0/311341021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f587c071330 msgr2=0x7f587c0a9a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 --2- 172.21.15.107:0/311341021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f587c071330 0x7f587c0a9a90 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f587c155cf0 tx=0x7f587401c850 comp rx=0 tx=0).stop 2023-10-25T00:22:37.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 -- 172.21.15.107:0/311341021 shutdown_connections 2023-10-25T00:22:37.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 --2- 172.21.15.107:0/311341021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f587c071330 0x7f587c0a9a90 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.270 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 --2- 172.21.15.107:0/311341021 >> v2:172.21.15.110:3300/0 conn(0x7f587c154c30 0x7f587c0a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 -- 172.21.15.107:0/311341021 >> 172.21.15.107:0/311341021 conn(0x7f587c0c3960 msgr2=0x7f587c0c3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 -- 172.21.15.107:0/311341021 shutdown_connections 2023-10-25T00:22:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 -- 172.21.15.107:0/311341021 wait complete. 2023-10-25T00:22:37.273 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 Processor -- start 2023-10-25T00:22:37.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 -- start start 2023-10-25T00:22:37.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f587c071330 0x7f587c150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f587c14ed10 0x7f587c1eb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f587c154c30 0x7f587c1eb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587c1500e0 con 0x7f587c14ed10 2023-10-25T00:22:37.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587c151430 con 0x7f587c071330 2023-10-25T00:22:37.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f588157a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f587c14f530 con 0x7f587c154c30 2023-10-25T00:22:37.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f587a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f587c14ed10 0x7f587c1eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f587a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f587c14ed10 0x7f587c1eb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45666/0 (socket says 172.21.15.107:45666) 2023-10-25T00:22:37.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f587a7fc640 1 -- 172.21.15.107:0/2043553329 learned_addr learned my addr 172.21.15.107:0/2043553329 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:37.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f587affd640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f587c071330 0x7f587c150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f587a7fc640 1 -- 172.21.15.107:0/2043553329 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f587c154c30 msgr2=0x7f587c1eb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f587a7fc640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f587c154c30 0x7f587c1eb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.268+0000 7f587a7fc640 1 -- 172.21.15.107:0/2043553329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f587c071330 msgr2=0x7f587c150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f587a7fc640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f587c071330 0x7f587c150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f587a7fc640 1 -- 172.21.15.107:0/2043553329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5874009a70 con 0x7f587c14ed10 2023-10-25T00:22:37.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f587affd640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f587c071330 0x7f587c150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:37.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f587a7fc640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f587c14ed10 0x7f587c1eb020 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f586c00e960 tx=0x7f586c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:37.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f58637fe640 1 -- 172.21.15.107:0/2043553329 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c009690 con 0x7f587c14ed10 2023-10-25T00:22:37.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f58637fe640 1 -- 172.21.15.107:0/2043553329 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f586c00c9a0 con 0x7f587c14ed10 2023-10-25T00:22:37.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f588157a640 1 -- 172.21.15.107:0/2043553329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587c14fb20 con 0x7f587c14ed10 2023-10-25T00:22:37.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f58637fe640 1 -- 172.21.15.107:0/2043553329 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c00cd10 con 0x7f587c14ed10 2023-10-25T00:22:37.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f588157a640 1 -- 172.21.15.107:0/2043553329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587c1eab00 con 0x7f587c14ed10 2023-10-25T00:22:37.278 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f588157a640 1 -- 172.21.15.107:0/2043553329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587c1ea960 con 0x7f587c14ed10 2023-10-25T00:22:37.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f58637fe640 1 -- 172.21.15.107:0/2043553329 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f586c013470 con 0x7f587c14ed10 2023-10-25T00:22:37.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f58637fe640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5868065e60 0x7f5868068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f58637fe640 1 -- 172.21.15.107:0/2043553329 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f586c084cd0 con 0x7f587c14ed10 2023-10-25T00:22:37.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f587affd640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5868065e60 0x7f5868068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.272+0000 7f587affd640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5868065e60 0x7f5868068320 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5870005890 tx=0x7f5870009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:37.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.276+0000 7f58637fe640 1 -- 172.21.15.107:0/2043553329 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f587c1ea960 con 0x7f587c14ed10 2023-10-25T00:22:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 -- 172.21.15.107:0/2043553329 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f587c070eb0 con 0x7f5868065e60 2023-10-25T00:22:37.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f58637fe640 1 -- 172.21.15.107:0/2043553329 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5870001340 con 0x7f5868065e60 2023-10-25T00:22:37.436 INFO:teuthology.orchestra.run.smithi107.stderr:instructed osd(s) 7 to deep-scrub 2023-10-25T00:22:37.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 -- 172.21.15.107:0/2043553329 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5868065e60 msgr2=0x7f5868068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5868065e60 0x7f5868068320 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f5870005890 tx=0x7f5870009040 comp rx=0 tx=0).stop 2023-10-25T00:22:37.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 -- 172.21.15.107:0/2043553329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f587c14ed10 msgr2=0x7f587c1eb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f587c14ed10 0x7f587c1eb020 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f586c00e960 tx=0x7f586c00ee20 comp rx=0 tx=0).stop 2023-10-25T00:22:37.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 -- 172.21.15.107:0/2043553329 shutdown_connections 2023-10-25T00:22:37.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5868065e60 0x7f5868068320 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f587c154c30 0x7f587c1eb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f587c14ed10 0x7f587c1eb020 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 --2- 172.21.15.107:0/2043553329 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f587c071330 0x7f587c150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 -- 172.21.15.107:0/2043553329 >> 172.21.15.107:0/2043553329 conn(0x7f587c0c3960 msgr2=0x7f587c155480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:37.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 -- 172.21.15.107:0/2043553329 shutdown_connections 2023-10-25T00:22:37.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.436+0000 7f588157a640 1 -- 172.21.15.107:0/2043553329 wait complete. 2023-10-25T00:22:37.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-25T00:22:37.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.568+0000 7fe113711640 1 Processor -- start 2023-10-25T00:22:37.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.568+0000 7fe113711640 1 -- start start 2023-10-25T00:22:37.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe113711640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe10c0c70e0 0x7fe10c0bdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe113711640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0c1ca0 con 0x7fe10c156770 2023-10-25T00:22:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe113711640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0c10c0 con 0x7fe10c0c2130 2023-10-25T00:22:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe113711640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c0c27e0 con 0x7fe10c0c70e0 2023-10-25T00:22:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe1114ad640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe10c156770 0x7fe10c0ab7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:35318/0 (socket says 172.21.15.107:35318) 2023-10-25T00:22:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe1114ad640 1 -- 172.21.15.107:0/641262720 learned_addr learned my addr 172.21.15.107:0/641262720 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe110cac640 1 --2- 172.21.15.107:0/641262720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe10c0c70e0 0x7fe10c0bdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe103fff640 1 -- 172.21.15.107:0/641262720 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224849261 0 0) 0x7fe10c0c1ca0 con 0x7fe10c156770 2023-10-25T00:22:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe103fff640 1 -- 172.21.15.107:0/641262720 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f8003530 con 0x7fe10c156770 2023-10-25T00:22:37.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe103fff640 1 -- 172.21.15.107:0/641262720 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070667477 0 0) 0x7fe10c0c10c0 con 0x7fe10c0c2130 2023-10-25T00:22:37.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe103fff640 1 -- 172.21.15.107:0/641262720 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe10c0c1ca0 con 0x7fe10c0c2130 2023-10-25T00:22:37.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe110cac640 1 -- 172.21.15.107:0/641262720 >> v1:172.21.15.110:6789/0 conn(0x7fe10c0c2130 legacy=0x7fe10c15d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe110cac640 1 -- 172.21.15.107:0/641262720 >> v1:172.21.15.107:6789/0 conn(0x7fe10c156770 legacy=0x7fe10c0ab7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.572+0000 7fe110cac640 1 -- 172.21.15.107:0/641262720 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c0c13a0 con 0x7fe10c0c70e0 2023-10-25T00:22:37.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe110cac640 1 --2- 172.21.15.107:0/641262720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe10c0c70e0 0x7fe10c0bdfa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0097f0 tx=0x7fe0fc01c1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a62038b68271e31 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:37.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe103fff640 1 -- 172.21.15.107:0/641262720 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe108001680 con 0x7fe10c0c70e0 2023-10-25T00:22:37.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe103fff640 1 -- 172.21.15.107:0/641262720 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc0a7050 con 0x7fe10c0c70e0 2023-10-25T00:22:37.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe103fff640 1 -- 172.21.15.107:0/641262720 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc0ab600 con 0x7fe10c0c70e0 2023-10-25T00:22:37.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 -- 172.21.15.107:0/641262720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe10c0c70e0 msgr2=0x7fe10c0bdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 --2- 172.21.15.107:0/641262720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe10c0c70e0 0x7fe10c0bdfa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0097f0 tx=0x7fe0fc01c1b0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 -- 172.21.15.107:0/641262720 shutdown_connections 2023-10-25T00:22:37.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 --2- 172.21.15.107:0/641262720 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe10c0c70e0 0x7fe10c0bdfa0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 -- 172.21.15.107:0/641262720 >> 172.21.15.107:0/641262720 conn(0x7fe10c0c65c0 msgr2=0x7fe10c0c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:37.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 -- 172.21.15.107:0/641262720 shutdown_connections 2023-10-25T00:22:37.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 -- 172.21.15.107:0/641262720 wait complete. 2023-10-25T00:22:37.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 Processor -- start 2023-10-25T00:22:37.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 -- start start 2023-10-25T00:22:37.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe10c0c2130 0x7fe10c154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe10c0c70e0 0x7fe10c158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe10c156770 0x7fe10c150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c15dc10 con 0x7fe10c0c2130 2023-10-25T00:22:37.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c15d040 con 0x7fe10c156770 2023-10-25T00:22:37.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c15d2e0 con 0x7fe10c0c70e0 2023-10-25T00:22:37.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe1114ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe10c0c2130 0x7fe10c154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe1114ad640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe10c0c2130 0x7fe10c154ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45680/0 (socket says 172.21.15.107:45680) 2023-10-25T00:22:37.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe1114ad640 1 -- 172.21.15.107:0/2699486075 learned_addr learned my addr 172.21.15.107:0/2699486075 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:37.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe110cac640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe10c0c70e0 0x7fe10c158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe111cae640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe10c156770 0x7fe10c150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe1114ad640 1 -- 172.21.15.107:0/2699486075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe10c0c70e0 msgr2=0x7fe10c158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe1114ad640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe10c0c70e0 0x7fe10c158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe1114ad640 1 -- 172.21.15.107:0/2699486075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe10c156770 msgr2=0x7fe10c150540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe1114ad640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe10c156770 0x7fe10c150540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe1114ad640 1 -- 172.21.15.107:0/2699486075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc009420 con 0x7fe10c0c2130 2023-10-25T00:22:37.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe111cae640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe10c156770 0x7fe10c150540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:37.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe1114ad640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe10c0c2130 0x7fe10c154ad0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fe1080011f0 tx=0x7fe10800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:37.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe101ffb640 1 -- 172.21.15.107:0/2699486075 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1080074e0 con 0x7fe10c0c2130 2023-10-25T00:22:37.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 -- 172.21.15.107:0/2699486075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10c15d5c0 con 0x7fe10c0c2130 2023-10-25T00:22:37.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe101ffb640 1 -- 172.21.15.107:0/2699486075 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe108007640 con 0x7fe10c0c2130 2023-10-25T00:22:37.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe113711640 1 -- 172.21.15.107:0/2699486075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10c155010 con 0x7fe10c0c2130 2023-10-25T00:22:37.587 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.576+0000 7fe101ffb640 1 -- 172.21.15.107:0/2699486075 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1080049f0 con 0x7fe10c0c2130 2023-10-25T00:22:37.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.580+0000 7fe113711640 1 -- 172.21.15.107:0/2699486075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10c0746e0 con 0x7fe10c0c2130 2023-10-25T00:22:37.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.580+0000 7fe101ffb640 1 -- 172.21.15.107:0/2699486075 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe10800a640 con 0x7fe10c0c2130 2023-10-25T00:22:37.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.580+0000 7fe101ffb640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0f0065e60 0x7fe0f0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:37.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.580+0000 7fe101ffb640 1 -- 172.21.15.107:0/2699486075 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fe108084070 con 0x7fe10c0c2130 2023-10-25T00:22:37.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.580+0000 7fe110cac640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0f0065e60 0x7fe0f0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:37.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.580+0000 7fe110cac640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0f0065e60 0x7fe0f0068320 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0ad000 tx=0x7fe0fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:37.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.584+0000 7fe101ffb640 1 -- 172.21.15.107:0/2699486075 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe108050430 con 0x7fe10c0c2130 2023-10-25T00:22:37.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.736+0000 7fe113711640 1 -- 172.21.15.107:0/2699486075 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe10c15d450 con 0x7fe0f0065e60 2023-10-25T00:22:37.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.752+0000 7fe101ffb640 1 -- 172.21.15.107:0/2699486075 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+550308 (secure 0 0 0) 0x7fe10c15d450 con 0x7fe0f0065e60 2023-10-25T00:22:37.753 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:37.760 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-25T00:22:37.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 -- 172.21.15.107:0/2699486075 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0f0065e60 msgr2=0x7fe0f0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0f0065e60 0x7fe0f0068320 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0ad000 tx=0x7fe0fc008040 comp rx=0 tx=0).stop 2023-10-25T00:22:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 -- 172.21.15.107:0/2699486075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe10c0c2130 msgr2=0x7fe10c154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe10c0c2130 0x7fe10c154ad0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fe1080011f0 tx=0x7fe10800cc20 comp rx=0 tx=0).stop 2023-10-25T00:22:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 -- 172.21.15.107:0/2699486075 shutdown_connections 2023-10-25T00:22:37.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe10c156770 0x7fe10c150540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe0f0065e60 0x7fe0f0068320 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe10c0c70e0 0x7fe10c158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 --2- 172.21.15.107:0/2699486075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe10c0c2130 0x7fe10c154ad0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 -- 172.21.15.107:0/2699486075 >> 172.21.15.107:0/2699486075 conn(0x7fe10c0c65c0 msgr2=0x7fe10c0c69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 -- 172.21.15.107:0/2699486075 shutdown_connections 2023-10-25T00:22:37.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:37.756+0000 7fe113711640 1 -- 172.21.15.107:0/2699486075 wait complete. 2023-10-25T00:22:37.775 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":958,"stamp":"2023-10-25T00:22:36.010567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40122242,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3906,"num_read_kb":457920,"num_write":3153446,"num_write_kb":38880045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":583926,"ondisk_log_size":583926,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7826840,"kb_used_data":86040,"kb_used_omap":0,"kb_used_meta":7740736,"kb_avail":747147880,"statfs":{"total":773094113280,"available":765079429120,"internally_reserved":0,"allocated":88104960,"data_stored":84515236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7926513664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":445,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005205"},"pg_stats":[{"pgid":"6.1a","version":"72'3995","reported_seq":4066,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.672949+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-25T00:22:08.672949+0000","last_peered":"2023-10-25T00:22:08.672949+0000","last_clean":"2023-10-25T00:22:08.672949+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-25T00:22:08.672949+0000","last_undegraded":"2023-10-25T00:22:08.672949+0000","last_fullsized":"2023-10-25T00:22:08.672949+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:51:20.137796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21480,"num_write_kb":282788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.773236+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-25T00:22:08.773236+0000","last_peered":"2023-10-25T00:22:08.773236+0000","last_clean":"2023-10-25T00:22:08.773236+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-25T00:22:08.773236+0000","last_undegraded":"2023-10-25T00:22:08.773236+0000","last_fullsized":"2023-10-25T00:22:08.773236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:00:20.101379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"68'864","reported_seq":988,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.254160+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-25T00:21:42.254160+0000","last_peered":"2023-10-25T00:21:42.254160+0000","last_clean":"2023-10-25T00:21:42.254160+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-25T00:21:42.254160+0000","last_undegraded":"2023-10-25T00:21:42.254160+0000","last_fullsized":"2023-10-25T00:21:42.254160+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:08.487170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":18441,"num_write":841,"num_write_kb":4586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'5616","reported_seq":5688,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195291+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-25T00:21:39.195291+0000","last_peered":"2023-10-25T00:21:39.195291+0000","last_clean":"2023-10-25T00:21:39.195291+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-25T00:21:39.195291+0000","last_undegraded":"2023-10-25T00:21:39.195291+0000","last_fullsized":"2023-10-25T00:21:39.195291+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:22:02.751840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29775,"num_write_kb":348818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"72'3735","reported_seq":3804,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.701818+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-25T00:22:08.701818+0000","last_peered":"2023-10-25T00:22:08.701818+0000","last_clean":"2023-10-25T00:22:08.701818+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-25T00:22:08.701818+0000","last_undegraded":"2023-10-25T00:22:08.701818+0000","last_fullsized":"2023-10-25T00:22:08.701818+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:59.368156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20090,"num_write_kb":264783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"72'2","reported_seq":70,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.258523+0000","last_change":"2023-10-24T23:53:11.444089+0000","last_active":"2023-10-25T00:22:08.258523+0000","last_peered":"2023-10-25T00:22:08.258523+0000","last_clean":"2023-10-25T00:22:08.258523+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-25T00:22:08.258523+0000","last_undegraded":"2023-10-25T00:22:08.258523+0000","last_fullsized":"2023-10-25T00:22:08.258523+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:54:06.875882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"72'27","reported_seq":101,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186964+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-25T00:21:39.186964+0000","last_peered":"2023-10-25T00:21:39.186964+0000","last_clean":"2023-10-25T00:21:39.186964+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-25T00:21:39.186964+0000","last_undegraded":"2023-10-25T00:21:39.186964+0000","last_fullsized":"2023-10-25T00:21:39.186964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:44:34.776830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'5447","reported_seq":5519,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.184033+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-25T00:21:39.184033+0000","last_peered":"2023-10-25T00:21:39.184033+0000","last_clean":"2023-10-25T00:21:39.184033+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-25T00:21:39.184033+0000","last_undegraded":"2023-10-25T00:21:39.184033+0000","last_fullsized":"2023-10-25T00:21:39.184033+0000","mapping_epoch":21,"log_start":"61'100","ondisk_log_start":"61'100","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:02:32.285292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28063,"num_write_kb":350200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"72'4747","reported_seq":4818,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.795259+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-25T00:22:08.795259+0000","last_peered":"2023-10-25T00:22:08.795259+0000","last_clean":"2023-10-25T00:22:08.795259+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-25T00:22:08.795259+0000","last_undegraded":"2023-10-25T00:22:08.795259+0000","last_fullsized":"2023-10-25T00:22:08.795259+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:38:56.643025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":25766,"num_write_kb":328702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"72'753","reported_seq":847,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.712526+0000","last_change":"2023-10-24T23:53:11.438681+0000","last_active":"2023-10-25T00:22:08.712526+0000","last_peered":"2023-10-25T00:22:08.712526+0000","last_clean":"2023-10-25T00:22:08.712526+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-25T00:22:08.712526+0000","last_undegraded":"2023-10-25T00:22:08.712526+0000","last_fullsized":"2023-10-25T00:22:08.712526+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:50.139697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":77282,"num_write":740,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"72'48","reported_seq":163,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.487987+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-25T00:21:42.487987+0000","last_peered":"2023-10-25T00:21:42.487987+0000","last_clean":"2023-10-25T00:21:42.487987+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-25T00:21:42.487987+0000","last_undegraded":"2023-10-25T00:21:42.487987+0000","last_fullsized":"2023-10-25T00:21:42.487987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:24:32.052062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'4792","reported_seq":4864,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205939+0000","last_change":"2023-10-24T23:53:02.718248+0000","last_active":"2023-10-25T00:21:19.205939+0000","last_peered":"2023-10-25T00:21:19.205939+0000","last_clean":"2023-10-25T00:21:19.205939+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-25T00:21:19.205939+0000","last_undegraded":"2023-10-25T00:21:19.205939+0000","last_fullsized":"2023-10-25T00:21:19.205939+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:44.724921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24997,"num_write_kb":313272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"72'4739","reported_seq":4811,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.785162+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-25T00:22:08.785162+0000","last_peered":"2023-10-25T00:22:08.785162+0000","last_clean":"2023-10-25T00:22:08.785162+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-25T00:22:08.785162+0000","last_undegraded":"2023-10-25T00:22:08.785162+0000","last_fullsized":"2023-10-25T00:22:08.785162+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:22.974333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":22,"num_write":25639,"num_write_kb":332364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"72'357","reported_seq":426,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:15.185838+0000","last_change":"2023-10-24T23:53:11.454537+0000","last_active":"2023-10-25T00:22:15.185838+0000","last_peered":"2023-10-25T00:22:15.185838+0000","last_clean":"2023-10-25T00:22:15.185838+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-25T00:22:15.185838+0000","last_undegraded":"2023-10-25T00:22:15.185838+0000","last_fullsized":"2023-10-25T00:22:15.185838+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:14:06.792651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2627766,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":227,"num_write":329,"num_write_kb":2759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"68'5","reported_seq":122,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.254866+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-25T00:21:42.254866+0000","last_peered":"2023-10-25T00:21:42.254866+0000","last_clean":"2023-10-25T00:21:42.254866+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-25T00:21:42.254866+0000","last_undegraded":"2023-10-25T00:21:42.254866+0000","last_fullsized":"2023-10-25T00:21:42.254866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:43:49.763090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'3731","reported_seq":3801,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205159+0000","last_change":"2023-10-24T23:53:02.724451+0000","last_active":"2023-10-25T00:21:19.205159+0000","last_peered":"2023-10-25T00:21:19.205159+0000","last_clean":"2023-10-25T00:21:19.205159+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-25T00:21:19.205159+0000","last_undegraded":"2023-10-25T00:21:19.205159+0000","last_fullsized":"2023-10-25T00:21:19.205159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:33.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19492,"num_write_kb":250091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"72'3799","reported_seq":3869,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.756165+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-25T00:22:08.756165+0000","last_peered":"2023-10-25T00:22:08.756165+0000","last_clean":"2023-10-25T00:22:08.756165+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-25T00:22:08.756165+0000","last_undegraded":"2023-10-25T00:22:08.756165+0000","last_fullsized":"2023-10-25T00:22:08.756165+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:35:39.879563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21581,"num_write_kb":262877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"72'45","reported_seq":156,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.747187+0000","last_change":"2023-10-24T23:53:11.444138+0000","last_active":"2023-10-25T00:22:08.747187+0000","last_peered":"2023-10-25T00:22:08.747187+0000","last_clean":"2023-10-25T00:22:08.747187+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-25T00:22:08.747187+0000","last_undegraded":"2023-10-25T00:22:08.747187+0000","last_fullsized":"2023-10-25T00:22:08.747187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:27:38.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"68'714","reported_seq":813,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195222+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-25T00:21:39.195222+0000","last_peered":"2023-10-25T00:21:39.195222+0000","last_clean":"2023-10-25T00:21:39.195222+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-25T00:21:39.195222+0000","last_undegraded":"2023-10-25T00:21:39.195222+0000","last_fullsized":"2023-10-25T00:21:39.195222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:41.133668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":86332,"num_write":703,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'4008","reported_seq":4083,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195693+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-25T00:21:39.195693+0000","last_peered":"2023-10-25T00:21:39.195693+0000","last_clean":"2023-10-25T00:21:39.195693+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-25T00:21:39.195693+0000","last_undegraded":"2023-10-25T00:21:39.195693+0000","last_fullsized":"2023-10-25T00:21:39.195693+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20791,"num_write_kb":267770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"72'5469","reported_seq":5550,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793934+0000","last_change":"2023-10-24T23:53:11.832809+0000","last_active":"2023-10-25T00:22:08.793934+0000","last_peered":"2023-10-25T00:22:08.793934+0000","last_clean":"2023-10-25T00:22:08.793934+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-25T00:22:08.793934+0000","last_undegraded":"2023-10-25T00:22:08.793934+0000","last_fullsized":"2023-10-25T00:22:08.793934+0000","mapping_epoch":25,"log_start":"64'200","ondisk_log_start":"64'200","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:01.151315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":40,"num_write":30646,"num_write_kb":401481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"72'5","reported_seq":116,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.200649+0000","last_change":"2023-10-24T23:53:11.442035+0000","last_active":"2023-10-25T00:21:39.200649+0000","last_peered":"2023-10-25T00:21:39.200649+0000","last_clean":"2023-10-25T00:21:39.200649+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-25T00:21:39.200649+0000","last_undegraded":"2023-10-25T00:21:39.200649+0000","last_fullsized":"2023-10-25T00:21:39.200649+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:13.717844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"72'532","reported_seq":611,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.992831+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-25T00:22:07.992831+0000","last_peered":"2023-10-25T00:22:07.992831+0000","last_clean":"2023-10-25T00:22:07.992831+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-25T00:22:07.992831+0000","last_undegraded":"2023-10-25T00:22:07.992831+0000","last_fullsized":"2023-10-25T00:22:07.992831+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:08.939184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3177545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7524,"num_write":504,"num_write_kb":3406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'6069","reported_seq":6142,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183010+0000","last_change":"2023-10-24T23:53:02.718952+0000","last_active":"2023-10-25T00:21:19.183010+0000","last_peered":"2023-10-25T00:21:19.183010+0000","last_clean":"2023-10-25T00:21:19.183010+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-25T00:21:19.183010+0000","last_undegraded":"2023-10-25T00:21:19.183010+0000","last_fullsized":"2023-10-25T00:21:19.183010+0000","mapping_epoch":21,"log_start":"62'800","ondisk_log_start":"62'800","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:34.494842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":33187,"num_write_kb":387130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'3606","reported_seq":3680,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.451770+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-25T00:22:08.451770+0000","last_peered":"2023-10-25T00:22:08.451770+0000","last_clean":"2023-10-25T00:22:08.451770+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-25T00:22:08.451770+0000","last_undegraded":"2023-10-25T00:22:08.451770+0000","last_fullsized":"2023-10-25T00:22:08.451770+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:26.012711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19493,"num_write_kb":254342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'1061","reported_seq":1182,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.761652+0000","last_change":"2023-10-24T23:53:11.448032+0000","last_active":"2023-10-25T00:22:08.761652+0000","last_peered":"2023-10-25T00:22:08.761652+0000","last_clean":"2023-10-25T00:22:08.761652+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-25T00:22:08.761652+0000","last_undegraded":"2023-10-25T00:22:08.761652+0000","last_fullsized":"2023-10-25T00:22:08.761652+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1061,"log_dups_size":0,"ondisk_log_size":1061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:50:32.490810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":9402,"num_write":1035,"num_write_kb":4717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'3","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.185352+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-25T00:21:39.185352+0000","last_peered":"2023-10-25T00:21:39.185352+0000","last_clean":"2023-10-25T00:21:39.185352+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-25T00:21:39.185352+0000","last_undegraded":"2023-10-25T00:21:39.185352+0000","last_fullsized":"2023-10-25T00:21:39.185352+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:35:46.751031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'5139","reported_seq":5212,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.216678+0000","last_change":"2023-10-24T23:53:02.727702+0000","last_active":"2023-10-25T00:21:19.216678+0000","last_peered":"2023-10-25T00:21:19.216678+0000","last_clean":"2023-10-25T00:21:19.216678+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-25T00:21:19.216678+0000","last_undegraded":"2023-10-25T00:21:19.216678+0000","last_fullsized":"2023-10-25T00:21:19.216678+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:00:45.686429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27772,"num_write_kb":328469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"72'4843","reported_seq":4913,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.750475+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-25T00:22:08.750475+0000","last_peered":"2023-10-25T00:22:08.750475+0000","last_clean":"2023-10-25T00:22:08.750475+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-25T00:22:08.750475+0000","last_undegraded":"2023-10-25T00:22:08.750475+0000","last_fullsized":"2023-10-25T00:22:08.750475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:10.820967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":27290,"num_write_kb":330906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"72'27","reported_seq":97,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.749463+0000","last_change":"2023-10-24T23:53:11.446925+0000","last_active":"2023-10-25T00:22:08.749463+0000","last_peered":"2023-10-25T00:22:08.749463+0000","last_clean":"2023-10-25T00:22:08.749463+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-25T00:22:08.749463+0000","last_undegraded":"2023-10-25T00:22:08.749463+0000","last_fullsized":"2023-10-25T00:22:08.749463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:59:52.485469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'2","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182801+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-25T00:21:39.182801+0000","last_peered":"2023-10-25T00:21:39.182801+0000","last_clean":"2023-10-25T00:21:39.182801+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-25T00:21:39.182801+0000","last_undegraded":"2023-10-25T00:21:39.182801+0000","last_fullsized":"2023-10-25T00:21:39.182801+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:27.961240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'5814","reported_seq":5888,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189609+0000","last_change":"2023-10-24T23:53:02.698217+0000","last_active":"2023-10-25T00:21:39.189609+0000","last_peered":"2023-10-25T00:21:39.189609+0000","last_clean":"2023-10-25T00:21:39.189609+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-25T00:21:39.189609+0000","last_undegraded":"2023-10-25T00:21:39.189609+0000","last_fullsized":"2023-10-25T00:21:39.189609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31996,"num_write_kb":382397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"72'3827","reported_seq":3898,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.365063+0000","last_change":"2023-10-24T23:53:11.849056+0000","last_active":"2023-10-25T00:22:08.365063+0000","last_peered":"2023-10-25T00:22:08.365063+0000","last_clean":"2023-10-25T00:22:08.365063+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-25T00:22:08.365063+0000","last_undegraded":"2023-10-25T00:22:08.365063+0000","last_fullsized":"2023-10-25T00:22:08.365063+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:08.932942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":21199,"num_write_kb":277730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"68'4820","reported_seq":4893,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.181692+0000","last_change":"2023-10-24T23:53:02.718501+0000","last_active":"2023-10-25T00:21:19.181692+0000","last_peered":"2023-10-25T00:21:19.181692+0000","last_clean":"2023-10-25T00:21:19.181692+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-25T00:21:19.181692+0000","last_undegraded":"2023-10-25T00:21:19.181692+0000","last_fullsized":"2023-10-25T00:21:19.181692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:28:33.342768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25163,"num_write_kb":285293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"72'5","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:06.452266+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-25T00:22:06.452266+0000","last_peered":"2023-10-25T00:22:06.452266+0000","last_clean":"2023-10-25T00:22:06.452266+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-25T00:22:06.452266+0000","last_undegraded":"2023-10-25T00:22:06.452266+0000","last_fullsized":"2023-10-25T00:22:06.452266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:49:38.685175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"72'5","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.744190+0000","last_change":"2023-10-24T23:53:11.447167+0000","last_active":"2023-10-25T00:22:08.744190+0000","last_peered":"2023-10-25T00:22:08.744190+0000","last_clean":"2023-10-25T00:22:08.744190+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-25T00:22:08.744190+0000","last_undegraded":"2023-10-25T00:22:08.744190+0000","last_fullsized":"2023-10-25T00:22:08.744190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:06:25.006728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"72'4957","reported_seq":5031,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.762994+0000","last_change":"2023-10-24T23:53:11.813704+0000","last_active":"2023-10-25T00:22:08.762994+0000","last_peered":"2023-10-25T00:22:08.762994+0000","last_clean":"2023-10-25T00:22:08.762994+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-25T00:22:08.762994+0000","last_undegraded":"2023-10-25T00:22:08.762994+0000","last_fullsized":"2023-10-25T00:22:08.762994+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:39:09.996039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28093,"num_write_kb":358545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'4782","reported_seq":4860,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.185276+0000","last_change":"2023-10-24T23:53:02.714089+0000","last_active":"2023-10-25T00:21:19.185276+0000","last_peered":"2023-10-25T00:21:19.185276+0000","last_clean":"2023-10-25T00:21:19.185276+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-25T00:21:19.185276+0000","last_undegraded":"2023-10-25T00:21:19.185276+0000","last_fullsized":"2023-10-25T00:21:19.185276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:16:26.315537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26432,"num_write_kb":298639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"68'3","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193250+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-25T00:21:39.193250+0000","last_peered":"2023-10-25T00:21:39.193250+0000","last_clean":"2023-10-25T00:21:39.193250+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-25T00:21:39.193250+0000","last_undegraded":"2023-10-25T00:21:39.193250+0000","last_fullsized":"2023-10-25T00:21:39.193250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:30:01.317970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"72'14","reported_seq":197,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.746812+0000","last_change":"2023-10-24T23:53:11.450974+0000","last_active":"2023-10-25T00:22:08.746812+0000","last_peered":"2023-10-25T00:22:08.746812+0000","last_clean":"2023-10-25T00:22:08.746812+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-25T00:22:08.746812+0000","last_undegraded":"2023-10-25T00:22:08.746812+0000","last_fullsized":"2023-10-25T00:22:08.746812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:51:05.443664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":345,"num_read_kb":345,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"72'2954","reported_seq":3023,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.717448+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-25T00:22:08.717448+0000","last_peered":"2023-10-25T00:22:08.717448+0000","last_clean":"2023-10-25T00:22:08.717448+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-25T00:22:08.717448+0000","last_undegraded":"2023-10-25T00:22:08.717448+0000","last_fullsized":"2023-10-25T00:22:08.717448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:37:30.278659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15961,"num_write_kb":219669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"72'36","reported_seq":152,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.212150+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-25T00:22:10.212150+0000","last_peered":"2023-10-25T00:22:10.212150+0000","last_clean":"2023-10-25T00:22:10.212150+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-25T00:22:10.212150+0000","last_undegraded":"2023-10-25T00:22:10.212150+0000","last_fullsized":"2023-10-25T00:22:10.212150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:22.877533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'4690","reported_seq":4764,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178387+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-25T00:21:19.178387+0000","last_peered":"2023-10-25T00:21:19.178387+0000","last_clean":"2023-10-25T00:21:19.178387+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-25T00:21:19.178387+0000","last_undegraded":"2023-10-25T00:21:19.178387+0000","last_fullsized":"2023-10-25T00:21:19.178387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:58:32.792574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25172,"num_write_kb":305786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"72'8","reported_seq":120,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.777365+0000","last_change":"2023-10-24T23:53:11.429694+0000","last_active":"2023-10-25T00:22:08.777365+0000","last_peered":"2023-10-25T00:22:08.777365+0000","last_clean":"2023-10-25T00:22:08.777365+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-25T00:22:08.777365+0000","last_undegraded":"2023-10-25T00:22:08.777365+0000","last_fullsized":"2023-10-25T00:22:08.777365+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:15.964994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"72'4200","reported_seq":4277,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.728522+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-25T00:22:08.728522+0000","last_peered":"2023-10-25T00:22:08.728522+0000","last_clean":"2023-10-25T00:22:08.728522+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-25T00:22:08.728522+0000","last_undegraded":"2023-10-25T00:22:08.728522+0000","last_fullsized":"2023-10-25T00:22:08.728522+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:38:18.678087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":24169,"num_write_kb":293796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"72'389","reported_seq":572,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.983790+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-25T00:22:07.983790+0000","last_peered":"2023-10-25T00:22:07.983790+0000","last_clean":"2023-10-25T00:22:07.983790+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-25T00:22:07.983790+0000","last_undegraded":"2023-10-25T00:22:07.983790+0000","last_fullsized":"2023-10-25T00:22:07.983790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":201,"num_read_kb":90291,"num_write":386,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'5730","reported_seq":5803,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205335+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-25T00:21:19.205335+0000","last_peered":"2023-10-25T00:21:19.205335+0000","last_clean":"2023-10-25T00:21:19.205335+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-25T00:21:19.205335+0000","last_undegraded":"2023-10-25T00:21:19.205335+0000","last_fullsized":"2023-10-25T00:21:19.205335+0000","mapping_epoch":21,"log_start":"61'500","ondisk_log_start":"61'500","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:21.840353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":31652,"num_write_kb":356481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760659+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-25T00:22:08.760659+0000","last_peered":"2023-10-25T00:22:08.760659+0000","last_clean":"2023-10-25T00:22:08.760659+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-25T00:22:08.760659+0000","last_undegraded":"2023-10-25T00:22:08.760659+0000","last_fullsized":"2023-10-25T00:22:08.760659+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:30.163913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"72'4168","reported_seq":4240,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.466973+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-25T00:22:08.466973+0000","last_peered":"2023-10-25T00:22:08.466973+0000","last_clean":"2023-10-25T00:22:08.466973+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-25T00:22:08.466973+0000","last_undegraded":"2023-10-25T00:22:08.466973+0000","last_fullsized":"2023-10-25T00:22:08.466973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:46.000330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23458,"num_write_kb":294850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"68'4339","reported_seq":4413,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197911+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-25T00:21:39.197911+0000","last_peered":"2023-10-25T00:21:39.197911+0000","last_clean":"2023-10-25T00:21:39.197911+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-25T00:21:39.197911+0000","last_undegraded":"2023-10-25T00:21:39.197911+0000","last_fullsized":"2023-10-25T00:21:39.197911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:09.326135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23267,"num_write_kb":265268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.222903+0000","last_change":"2023-10-24T23:52:58.165256+0000","last_active":"2023-10-25T00:21:19.222903+0000","last_peered":"2023-10-25T00:21:19.222903+0000","last_clean":"2023-10-25T00:21:19.222903+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-25T00:21:19.222903+0000","last_undegraded":"2023-10-25T00:21:19.222903+0000","last_fullsized":"2023-10-25T00:21:19.222903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:38.707455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"72'230","reported_seq":326,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.211353+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-25T00:22:10.211353+0000","last_peered":"2023-10-25T00:22:10.211353+0000","last_clean":"2023-10-25T00:22:10.211353+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-25T00:22:10.211353+0000","last_undegraded":"2023-10-25T00:22:10.211353+0000","last_fullsized":"2023-10-25T00:22:10.211353+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:54:41.340098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":61,"num_write":589,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"72'11","reported_seq":81,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.756187+0000","last_change":"2023-10-24T23:53:11.433306+0000","last_active":"2023-10-25T00:22:08.756187+0000","last_peered":"2023-10-25T00:22:08.756187+0000","last_clean":"2023-10-25T00:22:08.756187+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-25T00:22:08.756187+0000","last_undegraded":"2023-10-25T00:22:08.756187+0000","last_fullsized":"2023-10-25T00:22:08.756187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:44:38.728345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"72'3993","reported_seq":4067,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793391+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-25T00:22:08.793391+0000","last_peered":"2023-10-25T00:22:08.793391+0000","last_clean":"2023-10-25T00:22:08.793391+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-25T00:22:08.793391+0000","last_undegraded":"2023-10-25T00:22:08.793391+0000","last_fullsized":"2023-10-25T00:22:08.793391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:30:24.805861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21723,"num_write_kb":283850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'4270","reported_seq":4344,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.181969+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-25T00:21:19.181969+0000","last_peered":"2023-10-25T00:21:19.181969+0000","last_clean":"2023-10-25T00:21:19.181969+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-25T00:21:19.181969+0000","last_undegraded":"2023-10-25T00:21:19.181969+0000","last_fullsized":"2023-10-25T00:21:19.181969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:21:06.910103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23049,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180537+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-25T00:21:19.180537+0000","last_peered":"2023-10-25T00:21:19.180537+0000","last_clean":"2023-10-25T00:21:19.180537+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-25T00:21:19.180537+0000","last_undegraded":"2023-10-25T00:21:19.180537+0000","last_fullsized":"2023-10-25T00:21:19.180537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:12.201390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'9","reported_seq":128,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.260782+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-25T00:21:42.260782+0000","last_peered":"2023-10-25T00:21:42.260782+0000","last_clean":"2023-10-25T00:21:42.260782+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-25T00:21:42.260782+0000","last_undegraded":"2023-10-25T00:21:42.260782+0000","last_fullsized":"2023-10-25T00:21:42.260782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:43:16.517986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"72'64","reported_seq":214,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.752186+0000","last_change":"2023-10-24T23:53:11.441983+0000","last_active":"2023-10-25T00:22:08.752186+0000","last_peered":"2023-10-25T00:22:08.752186+0000","last_clean":"2023-10-25T00:22:08.752186+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-25T00:22:08.752186+0000","last_undegraded":"2023-10-25T00:22:08.752186+0000","last_fullsized":"2023-10-25T00:22:08.752186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:30.643019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":123,"num_write":59,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"72'3769","reported_seq":3843,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.609361+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-25T00:22:08.609361+0000","last_peered":"2023-10-25T00:22:08.609361+0000","last_clean":"2023-10-25T00:22:08.609361+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-25T00:22:08.609361+0000","last_undegraded":"2023-10-25T00:22:08.609361+0000","last_fullsized":"2023-10-25T00:22:08.609361+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:50:53.467058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20807,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'5752","reported_seq":5826,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194706+0000","last_change":"2023-10-24T23:53:02.728801+0000","last_active":"2023-10-25T00:21:39.194706+0000","last_peered":"2023-10-25T00:21:39.194706+0000","last_clean":"2023-10-25T00:21:39.194706+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-25T00:21:39.194706+0000","last_undegraded":"2023-10-25T00:21:39.194706+0000","last_fullsized":"2023-10-25T00:21:39.194706+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:57.865002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31231,"num_write_kb":354718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199135+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-25T00:21:39.199135+0000","last_peered":"2023-10-25T00:21:39.199135+0000","last_clean":"2023-10-25T00:21:39.199135+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-25T00:21:39.199135+0000","last_undegraded":"2023-10-25T00:21:39.199135+0000","last_fullsized":"2023-10-25T00:21:39.199135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:59.406201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'2","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189313+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-25T00:21:39.189313+0000","last_peered":"2023-10-25T00:21:39.189313+0000","last_clean":"2023-10-25T00:21:39.189313+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-25T00:21:39.189313+0000","last_undegraded":"2023-10-25T00:21:39.189313+0000","last_fullsized":"2023-10-25T00:21:39.189313+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:08:02.180025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"72'4","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.259221+0000","last_change":"2023-10-24T23:53:11.426287+0000","last_active":"2023-10-25T00:22:08.259221+0000","last_peered":"2023-10-25T00:22:08.259221+0000","last_clean":"2023-10-25T00:22:08.259221+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-25T00:22:08.259221+0000","last_undegraded":"2023-10-25T00:22:08.259221+0000","last_fullsized":"2023-10-25T00:22:08.259221+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:02:39.399866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"72'3939","reported_seq":4008,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.780603+0000","last_change":"2023-10-24T23:53:11.811139+0000","last_active":"2023-10-25T00:22:08.780603+0000","last_peered":"2023-10-25T00:22:08.780603+0000","last_clean":"2023-10-25T00:22:08.780603+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-25T00:22:08.780603+0000","last_undegraded":"2023-10-25T00:22:08.780603+0000","last_fullsized":"2023-10-25T00:22:08.780603+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:05.507633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22393,"num_write_kb":262004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'4305","reported_seq":4379,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186001+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-25T00:21:39.186001+0000","last_peered":"2023-10-25T00:21:39.186001+0000","last_clean":"2023-10-25T00:21:39.186001+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-25T00:21:39.186001+0000","last_undegraded":"2023-10-25T00:21:39.186001+0000","last_fullsized":"2023-10-25T00:21:39.186001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:12:04.084045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22696,"num_write_kb":287574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187237+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-25T00:21:39.187237+0000","last_peered":"2023-10-25T00:21:39.187237+0000","last_clean":"2023-10-25T00:21:39.187237+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-25T00:21:39.187237+0000","last_undegraded":"2023-10-25T00:21:39.187237+0000","last_fullsized":"2023-10-25T00:21:39.187237+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:36.250300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.188607+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-25T00:21:39.188607+0000","last_peered":"2023-10-25T00:21:39.188607+0000","last_clean":"2023-10-25T00:21:39.188607+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-25T00:21:39.188607+0000","last_undegraded":"2023-10-25T00:21:39.188607+0000","last_fullsized":"2023-10-25T00:21:39.188607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:38.327928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"72'3","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:07.892690+0000","last_change":"2023-10-24T23:53:11.426199+0000","last_active":"2023-10-25T00:22:07.892690+0000","last_peered":"2023-10-25T00:22:07.892690+0000","last_clean":"2023-10-25T00:22:07.892690+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-25T00:22:07.892690+0000","last_undegraded":"2023-10-25T00:22:07.892690+0000","last_fullsized":"2023-10-25T00:22:07.892690+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:57:51.018816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"72'5003","reported_seq":5078,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.705186+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-25T00:22:08.705186+0000","last_peered":"2023-10-25T00:22:08.705186+0000","last_clean":"2023-10-25T00:22:08.705186+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-25T00:22:08.705186+0000","last_undegraded":"2023-10-25T00:22:08.705186+0000","last_fullsized":"2023-10-25T00:22:08.705186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:20:16.596993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":13,"num_write":27120,"num_write_kb":350355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'5635","reported_seq":5708,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186387+0000","last_change":"2023-10-24T23:53:02.718597+0000","last_active":"2023-10-25T00:21:39.186387+0000","last_peered":"2023-10-25T00:21:39.186387+0000","last_clean":"2023-10-25T00:21:39.186387+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-25T00:21:39.186387+0000","last_undegraded":"2023-10-25T00:21:39.186387+0000","last_fullsized":"2023-10-25T00:21:39.186387+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:11:02.843780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30556,"num_write_kb":356140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"68'3","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.196592+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-25T00:21:39.196592+0000","last_peered":"2023-10-25T00:21:39.196592+0000","last_clean":"2023-10-25T00:21:39.196592+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-25T00:21:39.196592+0000","last_undegraded":"2023-10-25T00:21:39.196592+0000","last_fullsized":"2023-10-25T00:21:39.196592+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:03:24.001058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183369+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-25T00:21:39.183369+0000","last_peered":"2023-10-25T00:21:39.183369+0000","last_clean":"2023-10-25T00:21:39.183369+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-25T00:21:39.183369+0000","last_undegraded":"2023-10-25T00:21:39.183369+0000","last_fullsized":"2023-10-25T00:21:39.183369+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:45:33.347007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"72'2","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.259391+0000","last_change":"2023-10-24T23:53:11.432666+0000","last_active":"2023-10-25T00:22:08.259391+0000","last_peered":"2023-10-25T00:22:08.259391+0000","last_clean":"2023-10-25T00:22:08.259391+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-25T00:22:08.259391+0000","last_undegraded":"2023-10-25T00:22:08.259391+0000","last_fullsized":"2023-10-25T00:22:08.259391+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:17.903430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"72'3753","reported_seq":3826,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:09.504989+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-25T00:22:09.504989+0000","last_peered":"2023-10-25T00:22:09.504989+0000","last_clean":"2023-10-25T00:22:09.504989+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-25T00:22:09.504989+0000","last_undegraded":"2023-10-25T00:22:09.504989+0000","last_fullsized":"2023-10-25T00:22:09.504989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:42.131555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":21045,"num_write_kb":276986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'4101","reported_seq":4173,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.179395+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-25T00:21:19.179395+0000","last_peered":"2023-10-25T00:21:19.179395+0000","last_clean":"2023-10-25T00:21:19.179395+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-25T00:21:19.179395+0000","last_undegraded":"2023-10-25T00:21:19.179395+0000","last_fullsized":"2023-10-25T00:21:19.179395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22383,"num_write_kb":270023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"72'12","reported_seq":86,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.184571+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-25T00:21:39.184571+0000","last_peered":"2023-10-25T00:21:39.184571+0000","last_clean":"2023-10-25T00:21:39.184571+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-25T00:21:39.184571+0000","last_undegraded":"2023-10-25T00:21:39.184571+0000","last_fullsized":"2023-10-25T00:21:39.184571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:46:04.300376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.804406+0000","last_change":"2023-10-24T23:52:58.162746+0000","last_active":"2023-10-25T00:22:10.804406+0000","last_peered":"2023-10-25T00:22:10.804406+0000","last_clean":"2023-10-25T00:22:10.804406+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-25T00:22:10.804406+0000","last_undegraded":"2023-10-25T00:22:10.804406+0000","last_fullsized":"2023-10-25T00:22:10.804406+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"72'248","reported_seq":340,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:15.187644+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-25T00:22:15.187644+0000","last_peered":"2023-10-25T00:22:15.187644+0000","last_clean":"2023-10-25T00:22:15.187644+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-25T00:22:15.187644+0000","last_undegraded":"2023-10-25T00:22:15.187644+0000","last_fullsized":"2023-10-25T00:22:15.187644+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":62,"num_write":607,"num_write_kb":223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"72'5236","reported_seq":5311,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.753672+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-25T00:22:08.753672+0000","last_peered":"2023-10-25T00:22:08.753672+0000","last_clean":"2023-10-25T00:22:08.753672+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-25T00:22:08.753672+0000","last_undegraded":"2023-10-25T00:22:08.753672+0000","last_fullsized":"2023-10-25T00:22:08.753672+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:11:14.431984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":29554,"num_write_kb":368820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'5444","reported_seq":5520,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.216863+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-25T00:21:19.216863+0000","last_peered":"2023-10-25T00:21:19.216863+0000","last_clean":"2023-10-25T00:21:19.216863+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-25T00:21:19.216863+0000","last_undegraded":"2023-10-25T00:21:19.216863+0000","last_fullsized":"2023-10-25T00:21:19.216863+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:27.575357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28342,"num_write_kb":342931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"72'66","reported_seq":223,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.219689+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-25T00:22:10.219689+0000","last_peered":"2023-10-25T00:22:10.219689+0000","last_clean":"2023-10-25T00:22:10.219689+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-25T00:22:10.219689+0000","last_undegraded":"2023-10-25T00:22:10.219689+0000","last_fullsized":"2023-10-25T00:22:10.219689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:40:38.233439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":125,"num_write":61,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193375+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-25T00:21:19.193375+0000","last_peered":"2023-10-25T00:21:19.193375+0000","last_clean":"2023-10-25T00:21:19.193375+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-25T00:21:19.193375+0000","last_undegraded":"2023-10-25T00:21:19.193375+0000","last_fullsized":"2023-10-25T00:21:19.193375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:26:39.612096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"72'9","reported_seq":123,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760947+0000","last_change":"2023-10-24T23:53:11.436603+0000","last_active":"2023-10-25T00:22:08.760947+0000","last_peered":"2023-10-25T00:22:08.760947+0000","last_clean":"2023-10-25T00:22:08.760947+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-25T00:22:08.760947+0000","last_undegraded":"2023-10-25T00:22:08.760947+0000","last_fullsized":"2023-10-25T00:22:08.760947+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:09:05.948928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":135,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"72'3666","reported_seq":3735,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.792399+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-25T00:22:08.792399+0000","last_peered":"2023-10-25T00:22:08.792399+0000","last_clean":"2023-10-25T00:22:08.792399+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-25T00:22:08.792399+0000","last_undegraded":"2023-10-25T00:22:08.792399+0000","last_fullsized":"2023-10-25T00:22:08.792399+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:01:59.639080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20519,"num_write_kb":254391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"68'4182","reported_seq":4256,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187475+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-25T00:21:39.187475+0000","last_peered":"2023-10-25T00:21:39.187475+0000","last_clean":"2023-10-25T00:21:39.187475+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-25T00:21:39.187475+0000","last_undegraded":"2023-10-25T00:21:39.187475+0000","last_fullsized":"2023-10-25T00:21:39.187475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:51.314680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22839,"num_write_kb":259586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'4","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182378+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-25T00:21:39.182378+0000","last_peered":"2023-10-25T00:21:39.182378+0000","last_clean":"2023-10-25T00:21:39.182378+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-25T00:21:39.182378+0000","last_undegraded":"2023-10-25T00:21:39.182378+0000","last_fullsized":"2023-10-25T00:21:39.182378+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:08.362962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183752+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-25T00:21:39.183752+0000","last_peered":"2023-10-25T00:21:39.183752+0000","last_clean":"2023-10-25T00:21:39.183752+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-25T00:21:39.183752+0000","last_undegraded":"2023-10-25T00:21:39.183752+0000","last_fullsized":"2023-10-25T00:21:39.183752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:47:14.151120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"43'161","reported_seq":283,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193737+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-25T00:21:19.193737+0000","last_peered":"2023-10-25T00:21:19.193737+0000","last_clean":"2023-10-25T00:21:19.193737+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-25T00:21:19.193737+0000","last_undegraded":"2023-10-25T00:21:19.193737+0000","last_fullsized":"2023-10-25T00:21:19.193737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:19:47.531466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"72'2","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784850+0000","last_change":"2023-10-24T23:53:11.451073+0000","last_active":"2023-10-25T00:22:08.784850+0000","last_peered":"2023-10-25T00:22:08.784850+0000","last_clean":"2023-10-25T00:22:08.784850+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-25T00:22:08.784850+0000","last_undegraded":"2023-10-25T00:22:08.784850+0000","last_fullsized":"2023-10-25T00:22:08.784850+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:32.324710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"72'4105","reported_seq":4178,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.793839+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-25T00:22:08.793839+0000","last_peered":"2023-10-25T00:22:08.793839+0000","last_clean":"2023-10-25T00:22:08.793839+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-25T00:22:08.793839+0000","last_undegraded":"2023-10-25T00:22:08.793839+0000","last_fullsized":"2023-10-25T00:22:08.793839+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:04.125835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":19,"num_write":23147,"num_write_kb":297959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'5705","reported_seq":5779,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.189167+0000","last_change":"2023-10-24T23:53:02.716613+0000","last_active":"2023-10-25T00:21:39.189167+0000","last_peered":"2023-10-25T00:21:39.189167+0000","last_clean":"2023-10-25T00:21:39.189167+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-25T00:21:39.189167+0000","last_undegraded":"2023-10-25T00:21:39.189167+0000","last_fullsized":"2023-10-25T00:21:39.189167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:35:42.805030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":29845,"num_write_kb":357286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"68'5","reported_seq":78,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183009+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-25T00:21:19.183009+0000","last_peered":"2023-10-25T00:21:19.183009+0000","last_clean":"2023-10-25T00:21:19.183009+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-25T00:21:19.183009+0000","last_undegraded":"2023-10-25T00:21:19.183009+0000","last_fullsized":"2023-10-25T00:21:19.183009+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:57:42.857234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"72'7","reported_seq":122,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.761173+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-25T00:22:08.761173+0000","last_peered":"2023-10-25T00:22:08.761173+0000","last_clean":"2023-10-25T00:22:08.761173+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-25T00:22:08.761173+0000","last_undegraded":"2023-10-25T00:22:08.761173+0000","last_fullsized":"2023-10-25T00:22:08.761173+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:06:53.291510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"72'4234","reported_seq":4306,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.788961+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-25T00:22:08.788961+0000","last_peered":"2023-10-25T00:22:08.788961+0000","last_clean":"2023-10-25T00:22:08.788961+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-25T00:22:08.788961+0000","last_undegraded":"2023-10-25T00:22:08.788961+0000","last_fullsized":"2023-10-25T00:22:08.788961+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:41.905772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":15,"num_write":23173,"num_write_kb":302022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"68'4052","reported_seq":4125,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197631+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-25T00:21:39.197631+0000","last_peered":"2023-10-25T00:21:39.197631+0000","last_clean":"2023-10-25T00:21:39.197631+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-25T00:21:39.197631+0000","last_undegraded":"2023-10-25T00:21:39.197631+0000","last_fullsized":"2023-10-25T00:21:39.197631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:27:01.348283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21486,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"68'2","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.204255+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-25T00:21:19.204255+0000","last_peered":"2023-10-25T00:21:19.204255+0000","last_clean":"2023-10-25T00:21:19.204255+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-25T00:21:19.204255+0000","last_undegraded":"2023-10-25T00:21:19.204255+0000","last_fullsized":"2023-10-25T00:21:19.204255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:45:26.335828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"72'4","reported_seq":95,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.768028+0000","last_change":"2023-10-24T23:53:11.458463+0000","last_active":"2023-10-25T00:22:08.768028+0000","last_peered":"2023-10-25T00:22:08.768028+0000","last_clean":"2023-10-25T00:22:08.768028+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-25T00:22:08.768028+0000","last_undegraded":"2023-10-25T00:22:08.768028+0000","last_fullsized":"2023-10-25T00:22:08.768028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:11:45.040156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"72'4690","reported_seq":4763,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:30.808154+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-25T00:22:30.808154+0000","last_peered":"2023-10-25T00:22:30.808154+0000","last_clean":"2023-10-25T00:22:30.808154+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-25T00:22:30.808154+0000","last_undegraded":"2023-10-25T00:22:30.808154+0000","last_fullsized":"2023-10-25T00:22:30.808154+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24994,"num_write_kb":341798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'4780","reported_seq":4855,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194222+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-25T00:21:39.194222+0000","last_peered":"2023-10-25T00:21:39.194222+0000","last_clean":"2023-10-25T00:21:39.194222+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-25T00:21:39.194222+0000","last_undegraded":"2023-10-25T00:21:39.194222+0000","last_fullsized":"2023-10-25T00:21:39.194222+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:51:48.894740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25083,"num_write_kb":293163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'3","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:20.638748+0000","last_change":"2023-10-24T23:53:02.347713+0000","last_active":"2023-10-25T00:21:20.638748+0000","last_peered":"2023-10-25T00:21:20.638748+0000","last_clean":"2023-10-25T00:21:20.638748+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-25T00:21:20.638748+0000","last_undegraded":"2023-10-25T00:21:20.638748+0000","last_fullsized":"2023-10-25T00:21:20.638748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:37.611771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"72'5","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.753449+0000","last_change":"2023-10-24T23:53:11.448675+0000","last_active":"2023-10-25T00:22:08.753449+0000","last_peered":"2023-10-25T00:22:08.753449+0000","last_clean":"2023-10-25T00:22:08.753449+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-25T00:22:08.753449+0000","last_undegraded":"2023-10-25T00:22:08.753449+0000","last_fullsized":"2023-10-25T00:22:08.753449+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:05:01.502343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"72'3596","reported_seq":3669,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.779443+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-25T00:22:08.779443+0000","last_peered":"2023-10-25T00:22:08.779443+0000","last_clean":"2023-10-25T00:22:08.779443+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-25T00:22:08.779443+0000","last_undegraded":"2023-10-25T00:22:08.779443+0000","last_fullsized":"2023-10-25T00:22:08.779443+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:54.478897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":19820,"num_write_kb":259784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"68'4970","reported_seq":5044,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.201300+0000","last_change":"2023-10-24T23:53:02.724222+0000","last_active":"2023-10-25T00:21:39.201300+0000","last_peered":"2023-10-25T00:21:39.201300+0000","last_clean":"2023-10-25T00:21:39.201300+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-25T00:21:39.201300+0000","last_undegraded":"2023-10-25T00:21:39.201300+0000","last_fullsized":"2023-10-25T00:21:39.201300+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.081635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26721,"num_write_kb":311966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"72'1464","reported_seq":1588,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:10.213604+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-25T00:22:10.213604+0000","last_peered":"2023-10-25T00:22:10.213604+0000","last_clean":"2023-10-25T00:22:10.213604+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-25T00:22:10.213604+0000","last_undegraded":"2023-10-25T00:22:10.213604+0000","last_fullsized":"2023-10-25T00:22:10.213604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1464,"log_dups_size":0,"ondisk_log_size":1464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:48:43.088546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":43700,"num_write":1439,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"72'6","reported_seq":98,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:30.808048+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-25T00:22:30.808048+0000","last_peered":"2023-10-25T00:22:30.808048+0000","last_clean":"2023-10-25T00:22:30.808048+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-25T00:22:30.808048+0000","last_undegraded":"2023-10-25T00:22:30.808048+0000","last_fullsized":"2023-10-25T00:22:30.808048+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"72'3548","reported_seq":3620,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.608884+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-25T00:22:08.608884+0000","last_peered":"2023-10-25T00:22:08.608884+0000","last_clean":"2023-10-25T00:22:08.608884+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-25T00:22:08.608884+0000","last_undegraded":"2023-10-25T00:22:08.608884+0000","last_fullsized":"2023-10-25T00:22:08.608884+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:05.595887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20243,"num_write_kb":261539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"68'3393","reported_seq":3469,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194393+0000","last_change":"2023-10-24T23:53:02.729378+0000","last_active":"2023-10-25T00:21:39.194393+0000","last_peered":"2023-10-25T00:21:39.194393+0000","last_clean":"2023-10-25T00:21:39.194393+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-25T00:21:39.194393+0000","last_undegraded":"2023-10-25T00:21:39.194393+0000","last_fullsized":"2023-10-25T00:21:39.194393+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:57:09.056909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17500,"num_write_kb":217638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"68'5","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.187988+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-25T00:21:39.187988+0000","last_peered":"2023-10-25T00:21:39.187988+0000","last_clean":"2023-10-25T00:21:39.187988+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-25T00:21:39.187988+0000","last_undegraded":"2023-10-25T00:21:39.187988+0000","last_fullsized":"2023-10-25T00:21:39.187988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:24.678967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"72'3","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.764814+0000","last_change":"2023-10-24T23:53:11.454658+0000","last_active":"2023-10-25T00:22:08.764814+0000","last_peered":"2023-10-25T00:22:08.764814+0000","last_clean":"2023-10-25T00:22:08.764814+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-25T00:22:08.764814+0000","last_undegraded":"2023-10-25T00:22:08.764814+0000","last_fullsized":"2023-10-25T00:22:08.764814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"72'3584","reported_seq":3656,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.616239+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-25T00:22:08.616239+0000","last_peered":"2023-10-25T00:22:08.616239+0000","last_clean":"2023-10-25T00:22:08.616239+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-25T00:22:08.616239+0000","last_undegraded":"2023-10-25T00:22:08.616239+0000","last_fullsized":"2023-10-25T00:22:08.616239+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:09:40.887960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20504,"num_write_kb":254394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'5094","reported_seq":5172,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:20.638687+0000","last_change":"2023-10-24T23:53:02.719782+0000","last_active":"2023-10-25T00:21:20.638687+0000","last_peered":"2023-10-25T00:21:20.638687+0000","last_clean":"2023-10-25T00:21:20.638687+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-25T00:21:20.638687+0000","last_undegraded":"2023-10-25T00:21:20.638687+0000","last_fullsized":"2023-10-25T00:21:20.638687+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:05:21.568888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27763,"num_write_kb":328490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'6","reported_seq":101,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199601+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-25T00:21:39.199601+0000","last_peered":"2023-10-25T00:21:39.199601+0000","last_clean":"2023-10-25T00:21:39.199601+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-25T00:21:39.199601+0000","last_undegraded":"2023-10-25T00:21:39.199601+0000","last_fullsized":"2023-10-25T00:21:39.199601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:05:02.006416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"72'1438","reported_seq":1556,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:30.807836+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-25T00:22:30.807836+0000","last_peered":"2023-10-25T00:22:30.807836+0000","last_clean":"2023-10-25T00:22:30.807836+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-25T00:22:30.807836+0000","last_undegraded":"2023-10-25T00:22:30.807836+0000","last_fullsized":"2023-10-25T00:22:30.807836+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1438,"log_dups_size":0,"ondisk_log_size":1438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":35275,"num_write":1412,"num_write_kb":4487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"72'3292","reported_seq":3365,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.794538+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-25T00:22:08.794538+0000","last_peered":"2023-10-25T00:22:08.794538+0000","last_clean":"2023-10-25T00:22:08.794538+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-25T00:22:08.794538+0000","last_undegraded":"2023-10-25T00:22:08.794538+0000","last_fullsized":"2023-10-25T00:22:08.794538+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:43.015050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17853,"num_write_kb":243602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'4242","reported_seq":4315,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.203827+0000","last_change":"2023-10-24T23:53:02.717893+0000","last_active":"2023-10-25T00:21:19.203827+0000","last_peered":"2023-10-25T00:21:19.203827+0000","last_clean":"2023-10-25T00:21:19.203827+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-25T00:21:19.203827+0000","last_undegraded":"2023-10-25T00:21:19.203827+0000","last_fullsized":"2023-10-25T00:21:19.203827+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:06.850357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22687,"num_write_kb":275346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'7","reported_seq":124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.498042+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-25T00:21:42.498042+0000","last_peered":"2023-10-25T00:21:42.498042+0000","last_clean":"2023-10-25T00:21:42.498042+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-25T00:21:42.498042+0000","last_undegraded":"2023-10-25T00:21:42.498042+0000","last_fullsized":"2023-10-25T00:21:42.498042+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:23:29.515819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"72'5","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760678+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-25T00:22:08.760678+0000","last_peered":"2023-10-25T00:22:08.760678+0000","last_clean":"2023-10-25T00:22:08.760678+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-25T00:22:08.760678+0000","last_undegraded":"2023-10-25T00:22:08.760678+0000","last_fullsized":"2023-10-25T00:22:08.760678+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"72'4091","reported_seq":4163,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.758376+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-25T00:22:08.758376+0000","last_peered":"2023-10-25T00:22:08.758376+0000","last_clean":"2023-10-25T00:22:08.758376+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-25T00:22:08.758376+0000","last_undegraded":"2023-10-25T00:22:08.758376+0000","last_fullsized":"2023-10-25T00:22:08.758376+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:13:03.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22779,"num_write_kb":288098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'4310","reported_seq":4383,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.176404+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-25T00:21:19.176404+0000","last_peered":"2023-10-25T00:21:19.176404+0000","last_clean":"2023-10-25T00:21:19.176404+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-25T00:21:19.176404+0000","last_undegraded":"2023-10-25T00:21:19.176404+0000","last_fullsized":"2023-10-25T00:21:19.176404+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:55:07.756549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23791,"num_write_kb":283914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'4","reported_seq":99,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.199642+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-25T00:21:39.199642+0000","last_peered":"2023-10-25T00:21:39.199642+0000","last_clean":"2023-10-25T00:21:39.199642+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-25T00:21:39.199642+0000","last_undegraded":"2023-10-25T00:21:39.199642+0000","last_fullsized":"2023-10-25T00:21:39.199642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:15:42.051040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"72'2","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.702572+0000","last_change":"2023-10-24T23:53:11.451013+0000","last_active":"2023-10-25T00:22:08.702572+0000","last_peered":"2023-10-25T00:22:08.702572+0000","last_clean":"2023-10-25T00:22:08.702572+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-25T00:22:08.702572+0000","last_undegraded":"2023-10-25T00:22:08.702572+0000","last_fullsized":"2023-10-25T00:22:08.702572+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:52:22.503023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"72'3695","reported_seq":3768,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.635013+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-25T00:22:08.635013+0000","last_peered":"2023-10-25T00:22:08.635013+0000","last_clean":"2023-10-25T00:22:08.635013+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-25T00:22:08.635013+0000","last_undegraded":"2023-10-25T00:22:08.635013+0000","last_fullsized":"2023-10-25T00:22:08.635013+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:05:30.149693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21426,"num_write_kb":254037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"72'3","reported_seq":71,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.776347+0000","last_change":"2023-10-24T23:53:11.434538+0000","last_active":"2023-10-25T00:22:08.776347+0000","last_peered":"2023-10-25T00:22:08.776347+0000","last_clean":"2023-10-25T00:22:08.776347+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-25T00:22:08.776347+0000","last_undegraded":"2023-10-25T00:22:08.776347+0000","last_fullsized":"2023-10-25T00:22:08.776347+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:55.179253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"72'15","reported_seq":196,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:42.481650+0000","last_change":"2023-10-24T23:53:02.338984+0000","last_active":"2023-10-25T00:21:42.481650+0000","last_peered":"2023-10-25T00:21:42.481650+0000","last_clean":"2023-10-25T00:21:42.481650+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-25T00:21:42.481650+0000","last_undegraded":"2023-10-25T00:21:42.481650+0000","last_fullsized":"2023-10-25T00:21:42.481650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:58:33.072634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":327,"num_read_kb":327,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'5249","reported_seq":5329,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183983+0000","last_change":"2023-10-24T23:53:02.713462+0000","last_active":"2023-10-25T00:21:39.183983+0000","last_peered":"2023-10-25T00:21:39.183983+0000","last_clean":"2023-10-25T00:21:39.183983+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-25T00:21:39.183983+0000","last_undegraded":"2023-10-25T00:21:39.183983+0000","last_fullsized":"2023-10-25T00:21:39.183983+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:05:52.793147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":27646,"num_write_kb":332285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"72'4380","reported_seq":4451,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.773294+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-25T00:22:08.773294+0000","last_peered":"2023-10-25T00:22:08.773294+0000","last_clean":"2023-10-25T00:22:08.773294+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-25T00:22:08.773294+0000","last_undegraded":"2023-10-25T00:22:08.773294+0000","last_fullsized":"2023-10-25T00:22:08.773294+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23662,"num_write_kb":324338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"72'5","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.785854+0000","last_change":"2023-10-24T23:53:11.433387+0000","last_active":"2023-10-25T00:22:08.785854+0000","last_peered":"2023-10-25T00:22:08.785854+0000","last_clean":"2023-10-25T00:22:08.785854+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-25T00:22:08.785854+0000","last_undegraded":"2023-10-25T00:22:08.785854+0000","last_fullsized":"2023-10-25T00:22:08.785854+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:29.148450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"68'5","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.194080+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-25T00:21:19.194080+0000","last_peered":"2023-10-25T00:21:19.194080+0000","last_clean":"2023-10-25T00:21:19.194080+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-25T00:21:19.194080+0000","last_undegraded":"2023-10-25T00:21:19.194080+0000","last_fullsized":"2023-10-25T00:21:19.194080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:12.060535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'4442","reported_seq":4514,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.195509+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-25T00:21:39.195509+0000","last_peered":"2023-10-25T00:21:39.195509+0000","last_clean":"2023-10-25T00:21:39.195509+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-25T00:21:39.195509+0000","last_undegraded":"2023-10-25T00:21:39.195509+0000","last_fullsized":"2023-10-25T00:21:39.195509+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:54:53.195135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22810,"num_write_kb":296225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"72'4076","reported_seq":4148,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784809+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-25T00:22:08.784809+0000","last_peered":"2023-10-25T00:22:08.784809+0000","last_clean":"2023-10-25T00:22:08.784809+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-25T00:22:08.784809+0000","last_undegraded":"2023-10-25T00:22:08.784809+0000","last_fullsized":"2023-10-25T00:22:08.784809+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:33:06.172412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22055,"num_write_kb":297609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"68'3","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.211351+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-25T00:21:19.211351+0000","last_peered":"2023-10-25T00:21:19.211351+0000","last_clean":"2023-10-25T00:21:19.211351+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-25T00:21:19.211351+0000","last_undegraded":"2023-10-25T00:21:19.211351+0000","last_fullsized":"2023-10-25T00:21:19.211351+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:13:44.072648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'4101","reported_seq":4174,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.188446+0000","last_change":"2023-10-24T23:53:02.718835+0000","last_active":"2023-10-25T00:21:39.188446+0000","last_peered":"2023-10-25T00:21:39.188446+0000","last_clean":"2023-10-25T00:21:39.188446+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-25T00:21:39.188446+0000","last_undegraded":"2023-10-25T00:21:39.188446+0000","last_fullsized":"2023-10-25T00:21:39.188446+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:01:13.121234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22343,"num_write_kb":253486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"72'393","reported_seq":573,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:02.239640+0000","last_change":"2023-10-24T23:53:11.446979+0000","last_active":"2023-10-25T00:22:02.239640+0000","last_peered":"2023-10-25T00:22:02.239640+0000","last_clean":"2023-10-25T00:22:02.239640+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-25T00:22:02.239640+0000","last_undegraded":"2023-10-25T00:22:02.239640+0000","last_fullsized":"2023-10-25T00:22:02.239640+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:54:52.843631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":86197,"num_write":389,"num_write_kb":4306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"72'4606","reported_seq":4676,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.784571+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-25T00:22:08.784571+0000","last_peered":"2023-10-25T00:22:08.784571+0000","last_clean":"2023-10-25T00:22:08.784571+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-25T00:22:08.784571+0000","last_undegraded":"2023-10-25T00:22:08.784571+0000","last_fullsized":"2023-10-25T00:22:08.784571+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:18:48.679682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25763,"num_write_kb":316477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"72'11","reported_seq":127,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:40.179752+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-25T00:21:40.179752+0000","last_peered":"2023-10-25T00:21:40.179752+0000","last_clean":"2023-10-25T00:21:40.179752+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-25T00:21:40.179752+0000","last_undegraded":"2023-10-25T00:21:40.179752+0000","last_fullsized":"2023-10-25T00:21:40.179752+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:52:53.098641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'5680","reported_seq":5756,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.183466+0000","last_change":"2023-10-24T23:53:02.721618+0000","last_active":"2023-10-25T00:21:19.183466+0000","last_peered":"2023-10-25T00:21:19.183466+0000","last_clean":"2023-10-25T00:21:19.183466+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-25T00:21:19.183466+0000","last_undegraded":"2023-10-25T00:21:19.183466+0000","last_fullsized":"2023-10-25T00:21:19.183466+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:32:36.955782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29749,"num_write_kb":360962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"72'38","reported_seq":172,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:09.508467+0000","last_change":"2023-10-24T23:53:11.436964+0000","last_active":"2023-10-25T00:22:09.508467+0000","last_peered":"2023-10-25T00:22:09.508467+0000","last_clean":"2023-10-25T00:22:09.508467+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-25T00:22:09.508467+0000","last_undegraded":"2023-10-25T00:22:09.508467+0000","last_fullsized":"2023-10-25T00:22:09.508467+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:04:15.366966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":39,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"72'4808","reported_seq":4884,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.778926+0000","last_change":"2023-10-24T23:53:11.814304+0000","last_active":"2023-10-25T00:22:08.778926+0000","last_peered":"2023-10-25T00:22:08.778926+0000","last_clean":"2023-10-25T00:22:08.778926+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-25T00:22:08.778926+0000","last_undegraded":"2023-10-25T00:22:08.778926+0000","last_fullsized":"2023-10-25T00:22:08.778926+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:16.635178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26552,"num_write_kb":336125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"68'4747","reported_seq":4822,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.182709+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-25T00:21:39.182709+0000","last_peered":"2023-10-25T00:21:39.182709+0000","last_clean":"2023-10-25T00:21:39.182709+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-25T00:21:39.182709+0000","last_undegraded":"2023-10-25T00:21:39.182709+0000","last_fullsized":"2023-10-25T00:21:39.182709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:53:34.390444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24847,"num_write_kb":316685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"72'4283","reported_seq":4357,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.741347+0000","last_change":"2023-10-24T23:53:11.807096+0000","last_active":"2023-10-25T00:22:08.741347+0000","last_peered":"2023-10-25T00:22:08.741347+0000","last_clean":"2023-10-25T00:22:08.741347+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-25T00:22:08.741347+0000","last_undegraded":"2023-10-25T00:22:08.741347+0000","last_fullsized":"2023-10-25T00:22:08.741347+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:03:25.144578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23853,"num_write_kb":296504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"68'4036","reported_seq":4111,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183566+0000","last_change":"2023-10-24T23:53:02.720803+0000","last_active":"2023-10-25T00:21:39.183566+0000","last_peered":"2023-10-25T00:21:39.183566+0000","last_clean":"2023-10-25T00:21:39.183566+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-25T00:21:39.183566+0000","last_undegraded":"2023-10-25T00:21:39.183566+0000","last_fullsized":"2023-10-25T00:21:39.183566+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:15:16.950197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21233,"num_write_kb":273467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"72'4276","reported_seq":4349,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.743483+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-25T00:22:08.743483+0000","last_peered":"2023-10-25T00:22:08.743483+0000","last_clean":"2023-10-25T00:22:08.743483+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-25T00:22:08.743483+0000","last_undegraded":"2023-10-25T00:22:08.743483+0000","last_fullsized":"2023-10-25T00:22:08.743483+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:38.946585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23362,"num_write_kb":310817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"68'6201","reported_seq":6278,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.198522+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-25T00:21:39.198522+0000","last_peered":"2023-10-25T00:21:39.198522+0000","last_clean":"2023-10-25T00:21:39.198522+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-25T00:21:39.198522+0000","last_undegraded":"2023-10-25T00:21:39.198522+0000","last_fullsized":"2023-10-25T00:21:39.198522+0000","mapping_epoch":21,"log_start":"62'1000","ondisk_log_start":"62'1000","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:44:40.070552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33486,"num_write_kb":384441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"72'4018","reported_seq":4092,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.764145+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-25T00:22:08.764145+0000","last_peered":"2023-10-25T00:22:08.764145+0000","last_clean":"2023-10-25T00:22:08.764145+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-25T00:22:08.764145+0000","last_undegraded":"2023-10-25T00:22:08.764145+0000","last_fullsized":"2023-10-25T00:22:08.764145+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:48:24.911279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":21920,"num_write_kb":288651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"68'4522","reported_seq":4598,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.186452+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-25T00:21:39.186452+0000","last_peered":"2023-10-25T00:21:39.186452+0000","last_clean":"2023-10-25T00:21:39.186452+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-25T00:21:39.186452+0000","last_undegraded":"2023-10-25T00:21:39.186452+0000","last_fullsized":"2023-10-25T00:21:39.186452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:46:53.119739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24476,"num_write_kb":282500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"72'4209","reported_seq":4279,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.760824+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-25T00:22:08.760824+0000","last_peered":"2023-10-25T00:22:08.760824+0000","last_clean":"2023-10-25T00:22:08.760824+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-25T00:22:08.760824+0000","last_undegraded":"2023-10-25T00:22:08.760824+0000","last_fullsized":"2023-10-25T00:22:08.760824+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:21.165755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24141,"num_write_kb":281325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"68'4915","reported_seq":4988,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180147+0000","last_change":"2023-10-24T23:53:02.718226+0000","last_active":"2023-10-25T00:21:19.180147+0000","last_peered":"2023-10-25T00:21:19.180147+0000","last_clean":"2023-10-25T00:21:19.180147+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-25T00:21:19.180147+0000","last_undegraded":"2023-10-25T00:21:19.180147+0000","last_fullsized":"2023-10-25T00:21:19.180147+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26161,"num_write_kb":314084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"72'4933","reported_seq":5003,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.718975+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-25T00:22:08.718975+0000","last_peered":"2023-10-25T00:22:08.718975+0000","last_clean":"2023-10-25T00:22:08.718975+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-25T00:22:08.718975+0000","last_undegraded":"2023-10-25T00:22:08.718975+0000","last_fullsized":"2023-10-25T00:22:08.718975+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-25T23:54:29.274088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26695,"num_write_kb":352905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"68'5055","reported_seq":5129,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.180248+0000","last_change":"2023-10-24T23:53:02.718343+0000","last_active":"2023-10-25T00:21:19.180248+0000","last_peered":"2023-10-25T00:21:19.180248+0000","last_clean":"2023-10-25T00:21:19.180248+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-25T00:21:19.180248+0000","last_undegraded":"2023-10-25T00:21:19.180248+0000","last_fullsized":"2023-10-25T00:21:19.180248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:29:45.123152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27516,"num_write_kb":327406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"72'4083","reported_seq":4157,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.731484+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-25T00:22:08.731484+0000","last_peered":"2023-10-25T00:22:08.731484+0000","last_clean":"2023-10-25T00:22:08.731484+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-25T00:22:08.731484+0000","last_undegraded":"2023-10-25T00:22:08.731484+0000","last_fullsized":"2023-10-25T00:22:08.731484+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:01:54.422365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":22892,"num_write_kb":296872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"68'4777","reported_seq":4852,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178413+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-25T00:21:19.178413+0000","last_peered":"2023-10-25T00:21:19.178413+0000","last_clean":"2023-10-25T00:21:19.178413+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-25T00:21:19.178413+0000","last_undegraded":"2023-10-25T00:21:19.178413+0000","last_fullsized":"2023-10-25T00:21:19.178413+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:16.377025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25812,"num_write_kb":300181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"72'4618","reported_seq":4693,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.741877+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-25T00:22:08.741877+0000","last_peered":"2023-10-25T00:22:08.741877+0000","last_clean":"2023-10-25T00:22:08.741877+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-25T00:22:08.741877+0000","last_undegraded":"2023-10-25T00:22:08.741877+0000","last_fullsized":"2023-10-25T00:22:08.741877+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:01:11.838035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":26180,"num_write_kb":326342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"68'5340","reported_seq":5413,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178233+0000","last_change":"2023-10-24T23:53:02.716519+0000","last_active":"2023-10-25T00:21:19.178233+0000","last_peered":"2023-10-25T00:21:19.178233+0000","last_clean":"2023-10-25T00:21:19.178233+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-25T00:21:19.178233+0000","last_undegraded":"2023-10-25T00:21:19.178233+0000","last_fullsized":"2023-10-25T00:21:19.178233+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:01:51.397876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28701,"num_write_kb":348521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"72'4171","reported_seq":4243,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.721251+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-25T00:22:08.721251+0000","last_peered":"2023-10-25T00:22:08.721251+0000","last_clean":"2023-10-25T00:22:08.721251+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-25T00:22:08.721251+0000","last_undegraded":"2023-10-25T00:22:08.721251+0000","last_fullsized":"2023-10-25T00:22:08.721251+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:09:25.424344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22551,"num_write_kb":303498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"68'4493","reported_seq":4567,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.192644+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-25T00:21:19.192644+0000","last_peered":"2023-10-25T00:21:19.192644+0000","last_clean":"2023-10-25T00:21:19.192644+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-25T00:21:19.192644+0000","last_undegraded":"2023-10-25T00:21:19.192644+0000","last_fullsized":"2023-10-25T00:21:19.192644+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:24:53.192415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23797,"num_write_kb":300260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"72'4763","reported_seq":4833,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.772760+0000","last_change":"2023-10-24T23:53:44.474203+0000","last_active":"2023-10-25T00:22:08.772760+0000","last_peered":"2023-10-25T00:22:08.772760+0000","last_clean":"2023-10-25T00:22:08.772760+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-25T00:22:08.772760+0000","last_undegraded":"2023-10-25T00:22:08.772760+0000","last_fullsized":"2023-10-25T00:22:08.772760+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:16:50.292761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":25751,"num_write_kb":345225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"68'4946","reported_seq":5019,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.183160+0000","last_change":"2023-10-24T23:53:02.730536+0000","last_active":"2023-10-25T00:21:39.183160+0000","last_peered":"2023-10-25T00:21:39.183160+0000","last_clean":"2023-10-25T00:21:39.183160+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-25T00:21:39.183160+0000","last_undegraded":"2023-10-25T00:21:39.183160+0000","last_fullsized":"2023-10-25T00:21:39.183160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:55.104648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26583,"num_write_kb":315633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"72'4352","reported_seq":4424,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.767254+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-25T00:22:08.767254+0000","last_peered":"2023-10-25T00:22:08.767254+0000","last_clean":"2023-10-25T00:22:08.767254+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-25T00:22:08.767254+0000","last_undegraded":"2023-10-25T00:22:08.767254+0000","last_fullsized":"2023-10-25T00:22:08.767254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23957,"num_write_kb":301052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"68'5026","reported_seq":5099,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.193252+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-25T00:21:39.193252+0000","last_peered":"2023-10-25T00:21:39.193252+0000","last_clean":"2023-10-25T00:21:39.193252+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-25T00:21:39.193252+0000","last_undegraded":"2023-10-25T00:21:39.193252+0000","last_fullsized":"2023-10-25T00:21:39.193252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:17:21.610394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26401,"num_write_kb":310586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"72'4393","reported_seq":4469,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.746849+0000","last_change":"2023-10-24T23:53:11.860133+0000","last_active":"2023-10-25T00:22:08.746849+0000","last_peered":"2023-10-25T00:22:08.746849+0000","last_clean":"2023-10-25T00:22:08.746849+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-25T00:22:08.746849+0000","last_undegraded":"2023-10-25T00:22:08.746849+0000","last_fullsized":"2023-10-25T00:22:08.746849+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:57:43.051362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":24752,"num_write_kb":296109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"68'5620","reported_seq":5697,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.197727+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-25T00:21:39.197727+0000","last_peered":"2023-10-25T00:21:39.197727+0000","last_clean":"2023-10-25T00:21:39.197727+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-25T00:21:39.197727+0000","last_undegraded":"2023-10-25T00:21:39.197727+0000","last_fullsized":"2023-10-25T00:21:39.197727+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:25.491517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29863,"num_write_kb":337169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"72'4340","reported_seq":4414,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.730151+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-25T00:22:08.730151+0000","last_peered":"2023-10-25T00:22:08.730151+0000","last_clean":"2023-10-25T00:22:08.730151+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-25T00:22:08.730151+0000","last_undegraded":"2023-10-25T00:22:08.730151+0000","last_fullsized":"2023-10-25T00:22:08.730151+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:39:14.947372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":36,"num_write":23367,"num_write_kb":302669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"68'4464","reported_seq":4536,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.200961+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-25T00:21:39.200961+0000","last_peered":"2023-10-25T00:21:39.200961+0000","last_clean":"2023-10-25T00:21:39.200961+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-25T00:21:39.200961+0000","last_undegraded":"2023-10-25T00:21:39.200961+0000","last_fullsized":"2023-10-25T00:21:39.200961+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:59:20.689205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24372,"num_write_kb":278074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"72'5589","reported_seq":5659,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.768377+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-25T00:22:08.768377+0000","last_peered":"2023-10-25T00:22:08.768377+0000","last_clean":"2023-10-25T00:22:08.768377+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-25T00:22:08.768377+0000","last_undegraded":"2023-10-25T00:22:08.768377+0000","last_fullsized":"2023-10-25T00:22:08.768377+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:37.413259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":30206,"num_write_kb":399557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"68'4542","reported_seq":4616,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:20.806044+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-25T00:22:20.806044+0000","last_peered":"2023-10-25T00:22:20.806044+0000","last_clean":"2023-10-25T00:22:20.806044+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-25T00:22:20.806044+0000","last_undegraded":"2023-10-25T00:22:20.806044+0000","last_fullsized":"2023-10-25T00:22:20.806044+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23889,"num_write_kb":292323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"72'3430","reported_seq":3502,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.771571+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-25T00:22:08.771571+0000","last_peered":"2023-10-25T00:22:08.771571+0000","last_clean":"2023-10-25T00:22:08.771571+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-25T00:22:08.771571+0000","last_undegraded":"2023-10-25T00:22:08.771571+0000","last_fullsized":"2023-10-25T00:22:08.771571+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:43:31.344724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19411,"num_write_kb":241862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"68'4719","reported_seq":4792,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.207948+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-25T00:21:19.207948+0000","last_peered":"2023-10-25T00:21:19.207948+0000","last_clean":"2023-10-25T00:21:19.207948+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-25T00:21:19.207948+0000","last_undegraded":"2023-10-25T00:21:19.207948+0000","last_fullsized":"2023-10-25T00:21:19.207948+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:18:21.184541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25020,"num_write_kb":305197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"72'4169","reported_seq":4239,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:30.807993+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-25T00:22:30.807993+0000","last_peered":"2023-10-25T00:22:30.807993+0000","last_clean":"2023-10-25T00:22:30.807993+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-25T00:22:30.807993+0000","last_undegraded":"2023-10-25T00:22:30.807993+0000","last_fullsized":"2023-10-25T00:22:30.807993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23235,"num_write_kb":306175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"68'5952","reported_seq":6033,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.215357+0000","last_change":"2023-10-24T23:53:44.629959+0000","last_active":"2023-10-25T00:21:19.215357+0000","last_peered":"2023-10-25T00:21:19.215357+0000","last_clean":"2023-10-25T00:21:19.215357+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-25T00:21:19.215357+0000","last_undegraded":"2023-10-25T00:21:19.215357+0000","last_fullsized":"2023-10-25T00:21:19.215357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:59:36.474350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32341,"num_write_kb":375441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"72'3717","reported_seq":3786,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.786783+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-25T00:22:08.786783+0000","last_peered":"2023-10-25T00:22:08.786783+0000","last_clean":"2023-10-25T00:22:08.786783+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-25T00:22:08.786783+0000","last_undegraded":"2023-10-25T00:22:08.786783+0000","last_fullsized":"2023-10-25T00:22:08.786783+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:32:05.547590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21101,"num_write_kb":269108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"68'4825","reported_seq":4899,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.204063+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-25T00:21:19.204063+0000","last_peered":"2023-10-25T00:21:19.204063+0000","last_clean":"2023-10-25T00:21:19.204063+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-25T00:21:19.204063+0000","last_undegraded":"2023-10-25T00:21:19.204063+0000","last_fullsized":"2023-10-25T00:21:19.204063+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:33.959217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25319,"num_write_kb":294113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"72'3714","reported_seq":3786,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.739265+0000","last_change":"2023-10-24T23:53:11.847814+0000","last_active":"2023-10-25T00:22:08.739265+0000","last_peered":"2023-10-25T00:22:08.739265+0000","last_clean":"2023-10-25T00:22:08.739265+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-25T00:22:08.739265+0000","last_undegraded":"2023-10-25T00:22:08.739265+0000","last_fullsized":"2023-10-25T00:22:08.739265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:59.075858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21148,"num_write_kb":265324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"68'4878","reported_seq":4953,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.194001+0000","last_change":"2023-10-24T23:53:02.721570+0000","last_active":"2023-10-25T00:21:19.194001+0000","last_peered":"2023-10-25T00:21:19.194001+0000","last_clean":"2023-10-25T00:21:19.194001+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-25T00:21:19.194001+0000","last_undegraded":"2023-10-25T00:21:19.194001+0000","last_fullsized":"2023-10-25T00:21:19.194001+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:30:31.695829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26154,"num_write_kb":305558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"72'3434","reported_seq":3504,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.726527+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-25T00:22:08.726527+0000","last_peered":"2023-10-25T00:22:08.726527+0000","last_clean":"2023-10-25T00:22:08.726527+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-25T00:22:08.726527+0000","last_undegraded":"2023-10-25T00:22:08.726527+0000","last_fullsized":"2023-10-25T00:22:08.726527+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:25:12.525273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18449,"num_write_kb":262373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"68'4672","reported_seq":4746,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.178513+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-25T00:21:19.178513+0000","last_peered":"2023-10-25T00:21:19.178513+0000","last_clean":"2023-10-25T00:21:19.178513+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-25T00:21:19.178513+0000","last_undegraded":"2023-10-25T00:21:19.178513+0000","last_fullsized":"2023-10-25T00:21:19.178513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:49.937685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25223,"num_write_kb":293792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"72'4013","reported_seq":4088,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.743804+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-25T00:22:08.743804+0000","last_peered":"2023-10-25T00:22:08.743804+0000","last_clean":"2023-10-25T00:22:08.743804+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-25T00:22:08.743804+0000","last_undegraded":"2023-10-25T00:22:08.743804+0000","last_fullsized":"2023-10-25T00:22:08.743804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:12:21.026996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":22066,"num_write_kb":297298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"68'3918","reported_seq":3991,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:05.803866+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-25T00:22:05.803866+0000","last_peered":"2023-10-25T00:22:05.803866+0000","last_clean":"2023-10-25T00:22:05.803866+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-25T00:22:05.803866+0000","last_undegraded":"2023-10-25T00:22:05.803866+0000","last_fullsized":"2023-10-25T00:22:05.803866+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21675,"num_write_kb":247000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"72'4401","reported_seq":4472,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.740526+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-25T00:22:08.740526+0000","last_peered":"2023-10-25T00:22:08.740526+0000","last_clean":"2023-10-25T00:22:08.740526+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-25T00:22:08.740526+0000","last_undegraded":"2023-10-25T00:22:08.740526+0000","last_fullsized":"2023-10-25T00:22:08.740526+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:07.909021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":24294,"num_write_kb":326749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"68'3970","reported_seq":4043,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:05.803915+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-25T00:22:05.803915+0000","last_peered":"2023-10-25T00:22:05.803915+0000","last_clean":"2023-10-25T00:22:05.803915+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-25T00:22:05.803915+0000","last_undegraded":"2023-10-25T00:22:05.803915+0000","last_fullsized":"2023-10-25T00:22:05.803915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21202,"num_write_kb":257085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"72'5559","reported_seq":5636,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.789325+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-25T00:22:08.789325+0000","last_peered":"2023-10-25T00:22:08.789325+0000","last_clean":"2023-10-25T00:22:08.789325+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-25T00:22:08.789325+0000","last_undegraded":"2023-10-25T00:22:08.789325+0000","last_fullsized":"2023-10-25T00:22:08.789325+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:48.217889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":31202,"num_write_kb":391572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"68'4936","reported_seq":5009,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.185286+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-25T00:21:39.185286+0000","last_peered":"2023-10-25T00:21:39.185286+0000","last_clean":"2023-10-25T00:21:39.185286+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-25T00:21:39.185286+0000","last_undegraded":"2023-10-25T00:21:39.185286+0000","last_fullsized":"2023-10-25T00:21:39.185286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:22:14.156331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26134,"num_write_kb":309652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"72'3558","reported_seq":3632,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.781338+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-25T00:22:08.781338+0000","last_peered":"2023-10-25T00:22:08.781338+0000","last_clean":"2023-10-25T00:22:08.781338+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-25T00:22:08.781338+0000","last_undegraded":"2023-10-25T00:22:08.781338+0000","last_fullsized":"2023-10-25T00:22:08.781338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:10:18.973606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":34,"num_write":19715,"num_write_kb":243025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"68'5282","reported_seq":5355,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.192868+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-25T00:21:39.192868+0000","last_peered":"2023-10-25T00:21:39.192868+0000","last_clean":"2023-10-25T00:21:39.192868+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-25T00:21:39.192868+0000","last_undegraded":"2023-10-25T00:21:39.192868+0000","last_fullsized":"2023-10-25T00:21:39.192868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:47.265216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27848,"num_write_kb":324561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"72'3914","reported_seq":3989,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.624743+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-25T00:22:08.624743+0000","last_peered":"2023-10-25T00:22:08.624743+0000","last_clean":"2023-10-25T00:22:08.624743+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-25T00:22:08.624743+0000","last_undegraded":"2023-10-25T00:22:08.624743+0000","last_fullsized":"2023-10-25T00:22:08.624743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:24:45.536141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":20938,"num_write_kb":288796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"68'6616","reported_seq":6693,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194923+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-25T00:21:39.194923+0000","last_peered":"2023-10-25T00:21:39.194923+0000","last_clean":"2023-10-25T00:21:39.194923+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-25T00:21:39.194923+0000","last_undegraded":"2023-10-25T00:21:39.194923+0000","last_fullsized":"2023-10-25T00:21:39.194923+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:33:11.764625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36221,"num_write_kb":423916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"72'3536","reported_seq":3606,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.700035+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-25T00:22:08.700035+0000","last_peered":"2023-10-25T00:22:08.700035+0000","last_clean":"2023-10-25T00:22:08.700035+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-25T00:22:08.700035+0000","last_undegraded":"2023-10-25T00:22:08.700035+0000","last_fullsized":"2023-10-25T00:22:08.700035+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:06.568271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19276,"num_write_kb":253506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"68'4980","reported_seq":5050,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.176264+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-25T00:21:19.176264+0000","last_peered":"2023-10-25T00:21:19.176264+0000","last_clean":"2023-10-25T00:21:19.176264+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-25T00:21:19.176264+0000","last_undegraded":"2023-10-25T00:21:19.176264+0000","last_fullsized":"2023-10-25T00:21:19.176264+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:53:20.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26525,"num_write_kb":306963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"72'4160","reported_seq":4233,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.786968+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-25T00:22:08.786968+0000","last_peered":"2023-10-25T00:22:08.786968+0000","last_clean":"2023-10-25T00:22:08.786968+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-25T00:22:08.786968+0000","last_undegraded":"2023-10-25T00:22:08.786968+0000","last_fullsized":"2023-10-25T00:22:08.786968+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":21819,"num_write_kb":288439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"68'4434","reported_seq":4506,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.193726+0000","last_change":"2023-10-24T23:53:02.721317+0000","last_active":"2023-10-25T00:21:19.193726+0000","last_peered":"2023-10-25T00:21:19.193726+0000","last_clean":"2023-10-25T00:21:19.193726+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-25T00:21:19.193726+0000","last_undegraded":"2023-10-25T00:21:19.193726+0000","last_fullsized":"2023-10-25T00:21:19.193726+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:00:04.535311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23791,"num_write_kb":292031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"72'4195","reported_seq":4262,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.728890+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-25T00:22:08.728890+0000","last_peered":"2023-10-25T00:22:08.728890+0000","last_clean":"2023-10-25T00:22:08.728890+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-25T00:22:08.728890+0000","last_undegraded":"2023-10-25T00:22:08.728890+0000","last_fullsized":"2023-10-25T00:22:08.728890+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:18:40.168016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23000,"num_write_kb":285019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"68'4243","reported_seq":4314,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.205516+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-25T00:21:19.205516+0000","last_peered":"2023-10-25T00:21:19.205516+0000","last_clean":"2023-10-25T00:21:19.205516+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-25T00:21:19.205516+0000","last_undegraded":"2023-10-25T00:21:19.205516+0000","last_fullsized":"2023-10-25T00:21:19.205516+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:46:14.393139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22439,"num_write_kb":266094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"72'4017","reported_seq":4085,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.783880+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-25T00:22:08.783880+0000","last_peered":"2023-10-25T00:22:08.783880+0000","last_clean":"2023-10-25T00:22:08.783880+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-25T00:22:08.783880+0000","last_undegraded":"2023-10-25T00:22:08.783880+0000","last_fullsized":"2023-10-25T00:22:08.783880+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:39.720375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22665,"num_write_kb":291707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"68'4509","reported_seq":4583,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.213920+0000","last_change":"2023-10-24T23:53:02.730102+0000","last_active":"2023-10-25T00:21:19.213920+0000","last_peered":"2023-10-25T00:21:19.213920+0000","last_clean":"2023-10-25T00:21:19.213920+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-25T00:21:19.213920+0000","last_undegraded":"2023-10-25T00:21:19.213920+0000","last_fullsized":"2023-10-25T00:21:19.213920+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:27:07.539172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23158,"num_write_kb":297883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"72'5235","reported_seq":5312,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.789212+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-25T00:22:08.789212+0000","last_peered":"2023-10-25T00:22:08.789212+0000","last_clean":"2023-10-25T00:22:08.789212+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-25T00:22:08.789212+0000","last_undegraded":"2023-10-25T00:22:08.789212+0000","last_fullsized":"2023-10-25T00:22:08.789212+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:51:40.960630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":15,"num_write":29235,"num_write_kb":367391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"68'4260","reported_seq":4335,"reported_epoch":71,"state":"active+clean","last_fresh":"2023-10-25T00:21:19.221539+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-25T00:21:19.221539+0000","last_peered":"2023-10-25T00:21:19.221539+0000","last_clean":"2023-10-25T00:21:19.221539+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-25T00:21:19.221539+0000","last_undegraded":"2023-10-25T00:21:19.221539+0000","last_fullsized":"2023-10-25T00:21:19.221539+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:02:21.146118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23357,"num_write_kb":265837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"72'4656","reported_seq":4735,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.794660+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-25T00:22:08.794660+0000","last_peered":"2023-10-25T00:22:08.794660+0000","last_clean":"2023-10-25T00:22:08.794660+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-25T00:22:08.794660+0000","last_undegraded":"2023-10-25T00:22:08.794660+0000","last_fullsized":"2023-10-25T00:22:08.794660+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:22:42.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":26109,"num_write_kb":317781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"68'4636","reported_seq":4713,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.194609+0000","last_change":"2023-10-24T23:53:02.726938+0000","last_active":"2023-10-25T00:21:39.194609+0000","last_peered":"2023-10-25T00:21:39.194609+0000","last_clean":"2023-10-25T00:21:39.194609+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-25T00:21:39.194609+0000","last_undegraded":"2023-10-25T00:21:39.194609+0000","last_fullsized":"2023-10-25T00:21:39.194609+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:38:19.879911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25124,"num_write_kb":285314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"72'3117","reported_seq":3191,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.674704+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-25T00:22:08.674704+0000","last_peered":"2023-10-25T00:22:08.674704+0000","last_clean":"2023-10-25T00:22:08.674704+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-25T00:22:08.674704+0000","last_undegraded":"2023-10-25T00:22:08.674704+0000","last_fullsized":"2023-10-25T00:22:08.674704+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17240,"num_write_kb":232937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"68'5520","reported_seq":5604,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:21:39.192356+0000","last_change":"2023-10-24T23:53:44.630867+0000","last_active":"2023-10-25T00:21:39.192356+0000","last_peered":"2023-10-25T00:21:39.192356+0000","last_clean":"2023-10-25T00:21:39.192356+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-25T00:21:39.192356+0000","last_undegraded":"2023-10-25T00:21:39.192356+0000","last_fullsized":"2023-10-25T00:21:39.192356+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:37:49.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29665,"num_write_kb":368823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"72'3962","reported_seq":4034,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:08.769841+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-25T00:22:08.769841+0000","last_peered":"2023-10-25T00:22:08.769841+0000","last_clean":"2023-10-25T00:22:08.769841+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-25T00:22:08.769841+0000","last_undegraded":"2023-10-25T00:22:08.769841+0000","last_fullsized":"2023-10-25T00:22:08.769841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:31:50.994593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21989,"num_write_kb":280762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"68'4908","reported_seq":4988,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:15.805517+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-25T00:22:15.805517+0000","last_peered":"2023-10-25T00:22:15.805517+0000","last_clean":"2023-10-25T00:22:15.805517+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-25T00:22:15.805517+0000","last_undegraded":"2023-10-25T00:22:15.805517+0000","last_fullsized":"2023-10-25T00:22:15.805517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26829,"num_write_kb":320697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":320,"num_write":1482516,"num_write_kb":19060243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267718,"ondisk_log_size":267718,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":19490997,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1686,"num_read_kb":209824,"num_write":4665,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39043072,"data_stored":38981994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4535,"ondisk_log_size":4535,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":0,"num_write":1661441,"num_write_kb":19773626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307026,"ondisk_log_size":307026,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":20040877,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1620,"num_read_kb":247649,"num_write":4615,"num_write_kb":21609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40140800,"data_stored":40081754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4486,"ondisk_log_size":4486,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575207,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919080,"kb_used_data":12320,"kb_used_omap":0,"kb_used_meta":906752,"kb_avail":93452760,"statfs":{"total":96636764160,"available":95695626240,"internally_reserved":0,"allocated":12615680,"data_stored":12142216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":928514048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.304,"15min":0.93799999999999994},"min":{"1min":0.22,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.6349999999999998,"5min":13.929,"15min":13.929},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.3,"15min":0.92900000000000005},"min":{"1min":0.29699999999999999,"5min":0.27000000000000002,"15min":0.20999999999999999},"max":{"1min":3.6659999999999999,"5min":11.315,"15min":11.315},"last":0.96999999999999997}]},{"osd":1,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.714,"15min":1.1639999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.215},"max":{"1min":3.6949999999999998,"5min":28.308,"15min":28.308},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.94099999999999995,"5min":1.782,"15min":1.2290000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":3.7490000000000001,"5min":28.346,"15min":28.346},"last":1.044}]},{"osd":2,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":1.3779999999999999,"15min":0.99399999999999999},"min":{"1min":0.28499999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":5.71,"5min":11.430999999999999,"15min":11.430999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.333,"5min":1.139,"15min":0.94599999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.8799999999999999,"5min":7.9710000000000001,"15min":7.9710000000000001},"last":1.157}]},{"osd":3,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0720000000000001,"5min":2.3180000000000001,"15min":1.363},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.6150000000000002,"5min":93.664000000000001,"15min":93.664000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":2.379,"15min":1.3740000000000001},"min":{"1min":0.34499999999999997,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":2.8809999999999998,"5min":93.073999999999998,"15min":93.073999999999998},"last":1.0860000000000001}]},{"osd":4,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0840000000000001,"15min":0.877},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.249},"max":{"1min":2.9089999999999998,"5min":7.6280000000000001,"15min":7.6280000000000001},"last":1.002},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.714,"15min":1.1339999999999999},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.27200000000000002},"max":{"1min":4.9640000000000004,"5min":27.936,"15min":27.936},"last":0.72199999999999998}]},{"osd":5,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":2.1459999999999999,"5min":1.573,"15min":1.05},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.8170000000000002,"5min":21.126999999999999,"15min":21.126999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.995,"15min":0.94699999999999995},"min":{"1min":0.39800000000000002,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":2.5910000000000002,"5min":11.49,"15min":11.49},"last":1.117}]},{"osd":6,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":1.677,"5min":1.4770000000000001,"15min":1.077},"min":{"1min":0.41999999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.8140000000000001,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.7290000000000001,"15min":1.1799999999999999},"min":{"1min":0.49399999999999999,"5min":0.38100000000000001,"15min":0.35499999999999998},"max":{"1min":4.8639999999999999,"5min":20.651,"15min":20.651},"last":1.2310000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575208,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859004,"kb_used_data":5044,"kb_used_omap":0,"kb_used_meta":853952,"kb_avail":93512836,"statfs":{"total":96636764160,"available":95757144064,"internally_reserved":0,"allocated":5165056,"data_stored":4728730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":874446848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.397,"15min":0.93899999999999995},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.9009999999999998,"5min":15.974,"15min":15.974},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.4199999999999999,"15min":0.96799999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":8.2400000000000002,"5min":8.4789999999999992,"15min":8.4789999999999992},"last":0.81200000000000006}]},{"osd":1,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.042,"15min":0.878},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.221},"max":{"1min":3.5630000000000002,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.21,"5min":1.2549999999999999,"15min":0.98599999999999999},"min":{"1min":0.28699999999999998,"5min":0.254,"15min":0.249},"max":{"1min":3.5840000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.77900000000000003}]},{"osd":2,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.617,"15min":1.0600000000000001},"min":{"1min":0.27600000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.6779999999999999,"5min":12.615,"15min":12.615},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.228,"5min":1.591,"15min":1.105},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.702,"5min":12.667999999999999,"15min":12.667999999999999},"last":1.1000000000000001}]},{"osd":3,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.9690000000000001,"15min":1.2350000000000001},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.7350000000000003,"5min":38.649999999999999,"15min":38.649999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":2.0249999999999999,"15min":1.3260000000000001},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.766,"5min":38.683,"15min":38.683},"last":0.68400000000000005}]},{"osd":4,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.7589999999999999,"15min":1.0960000000000001},"min":{"1min":0.28399999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":3.738,"5min":12.682,"15min":12.682},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.4350000000000001,"15min":1.079},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.694,"5min":10.686,"15min":10.686},"last":1.246}]},{"osd":5,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.1990000000000001,"15min":0.94799999999999995},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.24299999999999999},"max":{"1min":3.6379999999999999,"5min":8.0039999999999996,"15min":8.0039999999999996},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.3700000000000001,"15min":1.046},"min":{"1min":0.496,"5min":0.33400000000000002,"15min":0.30299999999999999},"max":{"1min":3.903,"5min":11.766999999999999,"15min":11.766999999999999},"last":1.173}]},{"osd":7,"last update":"Wed Oct 25 00:21:43 2023","interfaces":[{"interface":"back","average":{"1min":1.907,"5min":1.306,"15min":1.018},"min":{"1min":0.32200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":8.2720000000000002,"5min":12.926,"15min":12.926},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.258,"15min":1.026},"min":{"1min":0.38200000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.1100000000000003,"5min":7.6619999999999999,"15min":7.6619999999999999},"last":1.196}]}]},{"osd":5,"up_from":13,"seq":55834575208,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":958856,"kb_used_data":9152,"kb_used_omap":0,"kb_used_meta":949696,"kb_avail":93412984,"statfs":{"total":96636764160,"available":95654895616,"internally_reserved":0,"allocated":9371648,"data_stored":8923216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":972488704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:44 2023","interfaces":[{"interface":"back","average":{"1min":2.4649999999999999,"5min":1.4690000000000001,"15min":1.032},"min":{"1min":0.23599999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.837999999999999,"5min":14.837999999999999,"15min":14.837999999999999},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.1599999999999999,"15min":0.95999999999999996},"min":{"1min":0.254,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.536,"5min":7.423,"15min":7.423},"last":0.63500000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.415,"15min":1.085},"min":{"1min":0.312,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":2.0419999999999998,"5min":12.141,"15min":12.141},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.4199999999999999,"15min":1.0860000000000001},"min":{"1min":0.314,"5min":0.26400000000000001,"15min":0.252},"max":{"1min":3.714,"5min":24.648,"15min":24.648},"last":0.71999999999999997}]},{"osd":2,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.46,"15min":1.2390000000000001},"min":{"1min":0.376,"5min":0.25,"15min":0.25},"max":{"1min":1.925,"5min":20.760999999999999,"15min":20.760999999999999},"last":1.016},{"interface":"front","average":{"1min":0.76700000000000002,"5min":1.274,"15min":1.0700000000000001},"min":{"1min":0.39800000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.1779999999999999,"5min":20.552,"15min":20.552},"last":0.78600000000000003}]},{"osd":3,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.429,"15min":1.2210000000000001},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.28000000000000003},"max":{"1min":3.6139999999999999,"5min":12.013,"15min":12.013},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.359,"15min":1.246},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.5779999999999998,"5min":12.504,"15min":12.805},"last":0.83099999999999996}]},{"osd":4,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.708,"15min":1.202},"min":{"1min":0.47099999999999997,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":2.3290000000000002,"5min":12.728,"15min":12.728},"last":1.159},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.532,"15min":1.1419999999999999},"min":{"1min":0.39200000000000002,"5min":0.27600000000000002,"15min":0.27200000000000002},"max":{"1min":3.6699999999999999,"5min":10.943,"15min":10.943},"last":1.05}]},{"osd":6,"last update":"Wed Oct 25 00:21:58 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.369,"15min":1.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.33500000000000002,"15min":0.26400000000000001},"max":{"1min":9.6489999999999991,"5min":17.936,"15min":17.936},"last":1.127},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.96,"15min":1.224},"min":{"1min":0.312,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":12.042999999999999,"5min":14.766999999999999,"15min":14.766999999999999},"last":0.98199999999999998}]},{"osd":7,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.75,"15min":1.2470000000000001},"min":{"1min":0.46500000000000002,"5min":0.33200000000000002,"15min":0.29499999999999998},"max":{"1min":1.2869999999999999,"5min":28.178999999999998,"15min":28.178999999999998},"last":1.099},{"interface":"front","average":{"1min":1.0740000000000001,"5min":2.133,"15min":1.357},"min":{"1min":0.59699999999999998,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":3.6429999999999998,"5min":28.222999999999999,"15min":28.222999999999999},"last":1.1899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575206,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946272,"kb_used_data":19096,"kb_used_omap":0,"kb_used_meta":927168,"kb_avail":93425568,"statfs":{"total":96636764160,"available":95667781632,"internally_reserved":0,"allocated":19554304,"data_stored":19092590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":949420032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":1.1910000000000001,"15min":0.93000000000000005},"min":{"1min":0.21199999999999999,"5min":0.19700000000000001,"15min":0.187},"max":{"1min":3.6120000000000001,"5min":20.922999999999998,"15min":20.922999999999998},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.1499999999999999,"15min":0.96199999999999997},"min":{"1min":0.26200000000000001,"5min":0.247,"15min":0.217},"max":{"1min":1.224,"5min":20.532,"15min":20.532},"last":0.751}]},{"osd":1,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.3560000000000001,"15min":0.97699999999999998},"min":{"1min":0.224,"5min":0.22,"15min":0.22},"max":{"1min":2.8140000000000001,"5min":20.914000000000001,"15min":20.914000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.738,"15min":1.157},"min":{"1min":0.28699999999999998,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":2.8399999999999999,"5min":23.265999999999998,"15min":23.265999999999998},"last":1.0109999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":1.5309999999999999,"15min":1.0920000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.3719999999999999,"5min":20.774000000000001,"15min":20.774000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.7509999999999999,"15min":1.2190000000000001},"min":{"1min":0.30099999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.9009999999999998,"5min":20.817,"15min":20.817},"last":1.0840000000000001}]},{"osd":3,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.575,"15min":1.131},"min":{"1min":0.32900000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.1440000000000001,"5min":24.613,"15min":24.613},"last":1.036},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.3300000000000001,"15min":1.113},"min":{"1min":0.307,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.3650000000000002,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.85199999999999998}]},{"osd":5,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.04,"15min":0.88500000000000001},"min":{"1min":0.35599999999999998,"5min":0.25600000000000001,"15min":0.23999999999999999},"max":{"1min":4.7759999999999998,"5min":6.1459999999999999,"15min":6.1459999999999999},"last":0.874},{"interface":"front","average":{"1min":1.149,"5min":1.0640000000000001,"15min":0.94799999999999995},"min":{"1min":0.246,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.5960000000000001,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.71999999999999997}]},{"osd":6,"last update":"Wed Oct 25 00:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.337,"15min":1.0740000000000001},"min":{"1min":0.36899999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.048,"5min":6.5220000000000002,"15min":6.5220000000000002},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.2130000000000001,"5min":0.97599999999999998,"15min":0.94299999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":4.3819999999999997,"5min":4.6609999999999996,"15min":4.6609999999999996},"last":1.103}]},{"osd":7,"last update":"Wed Oct 25 00:21:52 2023","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.2450000000000001,"15min":1.0469999999999999},"min":{"1min":0.47399999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":6.3330000000000002,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.4279999999999999,"15min":1.1020000000000001},"min":{"1min":0.51000000000000001,"5min":0.309,"15min":0.309},"max":{"1min":6.3550000000000004,"5min":6.3550000000000004,"15min":6.3550000000000004},"last":1.1279999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575207,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160732,"kb_used_data":9236,"kb_used_omap":0,"kb_used_meta":1151488,"kb_avail":93211108,"statfs":{"total":96636764160,"available":95448174592,"internally_reserved":0,"allocated":9457664,"data_stored":9007793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1179123712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.996,"15min":0.78900000000000003},"min":{"1min":0.20000000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.018,"5min":13.327999999999999,"15min":13.327999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.497,"5min":0.61299999999999999,"15min":0.73399999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.0489999999999999,"5min":3.556,"15min":5.4580000000000002},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.2290000000000001,"15min":0.86099999999999999},"min":{"1min":0.21199999999999999,"5min":0.20599999999999999,"15min":0.19900000000000001},"max":{"1min":1.796,"5min":13.831,"15min":13.831},"last":0.75},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.80900000000000005,"15min":0.76200000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.9300000000000002,"5min":4.4500000000000002,"15min":4.4500000000000002},"last":0.91200000000000003}]},{"osd":2,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.92200000000000004,"15min":0.877},"min":{"1min":0.23799999999999999,"5min":0.223,"15min":0.221},"max":{"1min":2.7890000000000001,"5min":8.2520000000000007,"15min":8.2520000000000007},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.075,"15min":0.90700000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.7360000000000002,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.79300000000000004}]},{"osd":4,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.9630000000000001,"15min":1.2010000000000001},"min":{"1min":0.32100000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.0339999999999998,"5min":39.070999999999998,"15min":39.070999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.5389999999999999,"5min":2.0870000000000002,"15min":1.2509999999999999},"min":{"1min":0.34599999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.0300000000000002,"5min":55.104999999999997,"15min":55.104999999999997},"last":1.018}]},{"osd":5,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.6379999999999999,"15min":1.117},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":4.1849999999999996,"5min":31.481999999999999,"15min":31.481999999999999},"last":1.052},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.681,"15min":1.1970000000000001},"min":{"1min":0.42499999999999999,"5min":0.35299999999999998,"15min":0.32500000000000001},"max":{"1min":4.2060000000000004,"5min":31.512,"15min":31.512},"last":1.115}]},{"osd":6,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.9179999999999999,"5min":1.617,"15min":1.1910000000000001},"min":{"1min":0.311,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":8.7080000000000002,"5min":25.809999999999999,"15min":25.809999999999999},"last":1.155},{"interface":"front","average":{"1min":1.9790000000000001,"5min":1.4530000000000001,"15min":1.175},"min":{"1min":0.47299999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":4.7779999999999996,"5min":24.036999999999999,"15min":24.036999999999999},"last":1.093}]},{"osd":7,"last update":"Wed Oct 25 00:21:48 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.704,"15min":1.226},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.8529999999999998,"5min":27.581,"15min":27.581},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.6379999999999999,"15min":1.24},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":8.5259999999999998,"5min":24.332000000000001,"15min":24.332000000000001},"last":1.198}]}]},{"osd":2,"up_from":13,"seq":55834575208,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946044,"kb_used_data":5044,"kb_used_omap":0,"kb_used_meta":940992,"kb_avail":93425796,"statfs":{"total":96636764160,"available":95668015104,"internally_reserved":0,"allocated":5165056,"data_stored":4728218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":963575808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.651,"15min":0.98399999999999999},"min":{"1min":0.188,"5min":0.16,"15min":0.16},"max":{"1min":2.9329999999999998,"5min":29.344000000000001,"15min":29.344000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":1.0800000000000001,"15min":0.82499999999999996},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.0539999999999998,"5min":10.755000000000001,"15min":10.755000000000001},"last":0.38700000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.3640000000000001,"15min":0.92200000000000004},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":2.8860000000000001,"5min":18.617999999999999,"15min":18.617999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":1.099,"15min":0.86099999999999999},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23100000000000001},"max":{"1min":2.1669999999999998,"5min":15.064,"15min":15.064},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.1259999999999999,"15min":0.88800000000000001},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.192},"max":{"1min":1.766,"5min":13.034000000000001,"15min":13.034000000000001},"last":0.373},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.5700000000000001,"15min":0.98099999999999998},"min":{"1min":0.26200000000000001,"5min":0.25,"15min":0.25},"max":{"1min":2.964,"5min":18.547999999999998,"15min":18.547999999999998},"last":0.41599999999999998}]},{"osd":4,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.45,"15min":1.1659999999999999},"min":{"1min":0.35099999999999998,"5min":0.252,"15min":0.252},"max":{"1min":2.839,"5min":22.997,"15min":33.378999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.5,"15min":1.2689999999999999},"min":{"1min":0.38500000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.8690000000000002,"5min":23.030999999999999,"15min":33.558999999999997},"last":0.82699999999999996}]},{"osd":5,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":2.2589999999999999,"15min":1.45},"min":{"1min":0.34699999999999998,"5min":0.32400000000000001,"15min":0.29099999999999998},"max":{"1min":4.6440000000000001,"5min":25.183,"15min":34.545999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.038,"5min":1.3879999999999999,"15min":1.169},"min":{"1min":0.34599999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":4.4829999999999997,"5min":11.685,"15min":36.104999999999997},"last":0.86299999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:21:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.208,"15min":1.1100000000000001},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":3.5739999999999998,"5min":9.7149999999999999,"15min":33.661999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.26,"15min":1.2},"min":{"1min":0.45700000000000002,"5min":0.371,"15min":0.371},"max":{"1min":3.5979999999999999,"5min":9.6799999999999997,"15min":34.043999999999997},"last":0.77400000000000002}]},{"osd":7,"last update":"Wed Oct 25 00:21:36 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":1.3,"15min":1.214},"min":{"1min":0.40500000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.395,"5min":11.503,"15min":33.307000000000002},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.3420000000000001,"15min":1.276},"min":{"1min":0.51200000000000001,"5min":0.371,"15min":0.371},"max":{"1min":2.419,"5min":11.468999999999999,"15min":33.344999999999999},"last":0.91300000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575208,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1070492,"kb_used_data":4052,"kb_used_omap":0,"kb_used_meta":1066432,"kb_avail":93301348,"statfs":{"total":96636764160,"available":95540580352,"internally_reserved":0,"allocated":4149248,"data_stored":3711479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1092026368},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":1.1739999999999999,"15min":0.81699999999999995},"min":{"1min":0.14299999999999999,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.81000000000000005,"5min":21.334,"15min":21.334},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.0449999999999999,"15min":0.84099999999999997},"min":{"1min":0.20499999999999999,"5min":0.19,"15min":0.19},"max":{"1min":3.141,"5min":21.276,"15min":21.276},"last":0.72399999999999998}]},{"osd":2,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":1.071,"15min":0.80200000000000005},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":8.1600000000000001,"5min":11.420999999999999,"15min":11.420999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.90100000000000002,"15min":0.83399999999999996},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.1760000000000002,"5min":11.648,"15min":11.648},"last":0.76900000000000002}]},{"osd":3,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.22,"15min":0.91500000000000004},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":3.46,"5min":14.346,"15min":14.346},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.2789999999999999,"15min":1.016},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.248},"max":{"1min":3.4969999999999999,"5min":14.379,"15min":14.379},"last":0.93500000000000005}]},{"osd":4,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.3959999999999999,"15min":1.0569999999999999},"min":{"1min":0.28000000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.6500000000000004,"5min":10.582000000000001,"15min":10.582000000000001},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.49,"5min":1.3009999999999999,"15min":1.04},"min":{"1min":0.32600000000000001,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":4.6630000000000003,"5min":11.173,"15min":11.173},"last":1.02}]},{"osd":5,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.244,"15min":1.054},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.0129999999999999,"5min":11.577999999999999,"15min":11.577999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.0760000000000001,"15min":0.98099999999999998},"min":{"1min":0.29899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.3220000000000001,"5min":10.018000000000001,"15min":10.018000000000001},"last":0.92000000000000004}]},{"osd":6,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":1.1140000000000001,"15min":1.026},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.3610000000000002,"5min":9.2989999999999995,"15min":11.845000000000001},"last":1.244},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.1699999999999999,"15min":1.109},"min":{"1min":0.38300000000000001,"5min":0.379,"15min":0.33800000000000002},"max":{"1min":2.3809999999999998,"5min":9.3350000000000009,"15min":11.919},"last":1.3200000000000001}]},{"osd":7,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.498,"15min":1.125},"min":{"1min":0.38200000000000001,"5min":0.36199999999999999,"15min":0.35599999999999998},"max":{"1min":5.077,"5min":9.407,"15min":11.881},"last":1.167},{"interface":"front","average":{"1min":0.874,"5min":1.3089999999999999,"15min":1.137},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.9750000000000001,"5min":10.478,"15min":10.478},"last":1.0800000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575208,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":966360,"kb_used_data":22096,"kb_used_omap":0,"kb_used_meta":944256,"kb_avail":93405480,"statfs":{"total":96636764160,"available":95647211520,"internally_reserved":0,"allocated":22626304,"data_stored":22180994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":966918144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":1.4159999999999999,"15min":0.98399999999999999},"min":{"1min":0.19700000000000001,"5min":0.18099999999999999,"15min":0.16900000000000001},"max":{"1min":1.7290000000000001,"5min":30.065999999999999,"15min":30.065999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.2330000000000001,"15min":0.98799999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.182},"max":{"1min":3.0030000000000001,"5min":19.757999999999999,"15min":19.757999999999999},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.96099999999999997,"15min":0.78200000000000003},"min":{"1min":0.22900000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.9740000000000002,"5min":15.324999999999999,"15min":15.324999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.2050000000000001,"15min":0.94299999999999995},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.95699999999999996,"5min":30.283999999999999,"15min":30.283999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.5209999999999999,"15min":1.101},"min":{"1min":0.20999999999999999,"5min":0.191,"15min":0.191},"max":{"1min":6.3570000000000002,"5min":22.966999999999999,"15min":22.966999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":1.417,"15min":1.048},"min":{"1min":0.29499999999999998,"5min":0.24299999999999999,"15min":0.23599999999999999},"max":{"1min":1.1000000000000001,"5min":19.460999999999999,"15min":19.460999999999999},"last":0.51700000000000002}]},{"osd":4,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.2090000000000001,"15min":1.121},"min":{"1min":0.33700000000000002,"5min":0.318,"15min":0.26600000000000001},"max":{"1min":3.0720000000000001,"5min":5.5990000000000002,"15min":8.3670000000000009},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.375,"15min":1.1850000000000001},"min":{"1min":0.29499999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.1749999999999998,"5min":13.023999999999999,"15min":13.023999999999999},"last":0.70199999999999996}]},{"osd":5,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.23,"15min":1.006},"min":{"1min":0.33000000000000002,"5min":0.30399999999999999,"15min":0.29099999999999998},"max":{"1min":2.9710000000000001,"5min":10.683999999999999,"15min":10.683999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.089,"15min":0.96999999999999997},"min":{"1min":0.378,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.5779999999999998,"5min":15.436999999999999,"15min":15.436999999999999},"last":0.69399999999999995}]},{"osd":6,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.393,"15min":1.125},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.7469999999999999,"5min":8.7260000000000009,"15min":8.7260000000000009},"last":0.628},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.3500000000000001,"15min":1.248},"min":{"1min":0.38200000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":2.9039999999999999,"5min":15.375,"15min":15.375},"last":0.66000000000000003}]},{"osd":7,"last update":"Wed Oct 25 00:21:45 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.367,"15min":1.113},"min":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":5.0979999999999999,"5min":10.81,"15min":10.81},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.3220000000000001,"15min":1.25},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":3.1499999999999999,"5min":11.417,"15min":11.417},"last":0.67600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12673024,"data_stored":12668140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3178496,"data_stored":3177545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11669504,"data_stored":11651527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4195192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":8388642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4279555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6918144,"data_stored":6907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6836224,"data_stored":6822812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-25T00:22:37.836 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.837 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.837 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.837 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.838 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.838 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.838 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.839 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.839 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.839 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.840 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.840 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.840 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.841 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.841 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.841 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.841 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.842 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.842 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.842 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.843 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.843 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.844 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.844 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.845 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.845 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.845 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.845 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.846 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.846 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.846 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.847 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.847 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.847 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.848 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.848 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.848 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.849 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.849 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.849 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.849 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.850 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.850 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.850 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.851 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.851 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.851 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.852 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.852 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.852 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.853 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.853 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.853 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.854 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.854 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.854 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.855 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.855 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.855 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.855 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.856 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.856 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.856 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.857 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.857 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.857 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.858 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.858 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.858 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.859 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.859 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.859 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.860 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.860 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.860 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.860 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.861 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.861 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.861 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.862 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.862 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.862 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.863 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.863 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.863 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.864 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.864 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.864 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-24T23:52:51.349804+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=51, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.865 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.865 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.865 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.866 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.866 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.866 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.867 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.867 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.867 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.868 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.868 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.868 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.869 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.869 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.869 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.870 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.870 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.870 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.871 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.871 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.871 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.872 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.872 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.872 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.873 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.873 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.873 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.874 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.874 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.874 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.874 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.875 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.875 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.875 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.876 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.876 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.876 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.877 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.877 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.877 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.878 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.878 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.878 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.879 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.879 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.879 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.880 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.880 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.880 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.880 INFO:tasks.ceph:pgid 6.22 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.881 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.881 INFO:tasks.ceph:pgid 6.23 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.881 INFO:tasks.ceph:pgid 4.21 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.882 INFO:tasks.ceph:pgid 6.20 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.882 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.883 INFO:tasks.ceph:pgid 6.21 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.883 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.883 INFO:tasks.ceph:pgid 6.26 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.883 INFO:tasks.ceph:pgid 4.24 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.884 INFO:tasks.ceph:pgid 6.27 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.884 INFO:tasks.ceph:pgid 4.25 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.884 INFO:tasks.ceph:pgid 6.24 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.885 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.885 INFO:tasks.ceph:pgid 6.25 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.885 INFO:tasks.ceph:pgid 4.27 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.886 INFO:tasks.ceph:pgid 6.2a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.886 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.886 INFO:tasks.ceph:pgid 6.2b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.887 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.887 INFO:tasks.ceph:pgid 6.28 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.887 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.887 INFO:tasks.ceph:pgid 6.29 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.888 INFO:tasks.ceph:pgid 4.2b last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.888 INFO:tasks.ceph:pgid 6.2e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.888 INFO:tasks.ceph:pgid 4.2c last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.889 INFO:tasks.ceph:pgid 6.2f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.889 INFO:tasks.ceph:pgid 4.2d last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.889 INFO:tasks.ceph:pgid 6.2c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.889 INFO:tasks.ceph:pgid 4.2e last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.890 INFO:tasks.ceph:pgid 6.2d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.890 INFO:tasks.ceph:pgid 4.2f last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.890 INFO:tasks.ceph:pgid 6.32 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.891 INFO:tasks.ceph:pgid 4.30 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.891 INFO:tasks.ceph:pgid 6.33 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.891 INFO:tasks.ceph:pgid 4.31 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.892 INFO:tasks.ceph:pgid 6.30 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.892 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.892 INFO:tasks.ceph:pgid 6.31 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.892 INFO:tasks.ceph:pgid 4.33 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.893 INFO:tasks.ceph:pgid 6.36 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.893 INFO:tasks.ceph:pgid 4.34 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.893 INFO:tasks.ceph:pgid 6.37 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.894 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.894 INFO:tasks.ceph:pgid 6.34 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.894 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.895 INFO:tasks.ceph:pgid 6.35 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.895 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.895 INFO:tasks.ceph:pgid 6.3a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.896 INFO:tasks.ceph:pgid 4.38 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.896 INFO:tasks.ceph:pgid 6.3b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.896 INFO:tasks.ceph:pgid 4.39 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.897 INFO:tasks.ceph:pgid 6.38 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.898 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.898 INFO:tasks.ceph:pgid 6.39 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.898 INFO:tasks.ceph:pgid 4.3b last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.899 INFO:tasks.ceph:pgid 6.3e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.899 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.899 INFO:tasks.ceph:pgid 6.3f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.900 INFO:tasks.ceph:pgid 4.3d last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.900 INFO:tasks.ceph:pgid 6.3c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.900 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.900 INFO:tasks.ceph:pgid 6.3d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.901 INFO:tasks.ceph:pgid 4.3f last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:37.901 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-25T00:22:37.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:22:38.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f237660d640 1 Processor -- start 2023-10-25T00:22:38.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f237660d640 1 -- start start 2023-10-25T00:22:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f237660d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23700c8100 0x7f23700abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f237660d640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f237006a4b0 0x7f23700ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f237660d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2370155680 con 0x7f2370152040 2023-10-25T00:22:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f237660d640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2370154a70 con 0x7f237006a4b0 2023-10-25T00:22:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f237660d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2370154d50 con 0x7f23700c8100 2023-10-25T00:22:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f2374baa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f237006a4b0 0x7f23700ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f2374baa640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f237006a4b0 0x7f23700ac240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45690/0 (socket says 172.21.15.107:45690) 2023-10-25T00:22:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f2374baa640 1 -- 172.21.15.107:0/2250963774 learned_addr learned my addr 172.21.15.107:0/2250963774 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f236f7fe640 1 --2- 172.21.15.107:0/2250963774 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23700c8100 0x7f23700abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f2374baa640 1 -- 172.21.15.107:0/2250963774 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23700c8100 msgr2=0x7f23700abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f2374baa640 1 --2- 172.21.15.107:0/2250963774 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23700c8100 0x7f23700abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f2374baa640 1 -- 172.21.15.107:0/2250963774 >> v1:172.21.15.110:6789/0 conn(0x7f2370152040 legacy=0x7f23700c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f2374baa640 1 -- 172.21.15.107:0/2250963774 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2370155030 con 0x7f237006a4b0 2023-10-25T00:22:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.064+0000 7f236f7fe640 1 --2- 172.21.15.107:0/2250963774 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23700c8100 0x7f23700abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:38.070 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f2374baa640 1 --2- 172.21.15.107:0/2250963774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237006a4b0 0x7f23700ac240 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f2368009e50 tx=0x7f236801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368ec882d5e1ba21 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:38.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236effd640 1 -- 172.21.15.107:0/2250963774 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23680a55e0 con 0x7f237006a4b0 2023-10-25T00:22:38.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236effd640 1 -- 172.21.15.107:0/2250963774 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23680a5740 con 0x7f237006a4b0 2023-10-25T00:22:38.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236effd640 1 -- 172.21.15.107:0/2250963774 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23680a5ad0 con 0x7f237006a4b0 2023-10-25T00:22:38.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 -- 172.21.15.107:0/2250963774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237006a4b0 msgr2=0x7f23700ac240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:38.071 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 --2- 172.21.15.107:0/2250963774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237006a4b0 0x7f23700ac240 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f2368009e50 tx=0x7f236801c940 comp rx=0 tx=0).stop 2023-10-25T00:22:38.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 -- 172.21.15.107:0/2250963774 shutdown_connections 2023-10-25T00:22:38.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 --2- 172.21.15.107:0/2250963774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237006a4b0 0x7f23700ac240 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:38.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 --2- 172.21.15.107:0/2250963774 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23700c8100 0x7f23700abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:38.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 -- 172.21.15.107:0/2250963774 >> 172.21.15.107:0/2250963774 conn(0x7f23700ca200 msgr2=0x7f23700ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:38.072 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 -- 172.21.15.107:0/2250963774 shutdown_connections 2023-10-25T00:22:38.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 -- 172.21.15.107:0/2250963774 wait complete. 2023-10-25T00:22:38.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 Processor -- start 2023-10-25T00:22:38.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 -- start start 2023-10-25T00:22:38.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237006a4b0 0x7f237014c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:38.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23700c8100 0x7f237014bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:38.073 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2370152040 0x7f23700a9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:38.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23700a98e0 con 0x7f237006a4b0 2023-10-25T00:22:38.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237014ce50 con 0x7f2370152040 2023-10-25T00:22:38.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237014d130 con 0x7f23700c8100 2023-10-25T00:22:38.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237006a4b0 0x7f237014c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:38.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237006a4b0 0x7f237014c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45698/0 (socket says 172.21.15.107:45698) 2023-10-25T00:22:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236ffff640 1 -- 172.21.15.107:0/2987062609 learned_addr learned my addr 172.21.15.107:0/2987062609 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236ffff640 1 -- 172.21.15.107:0/2987062609 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23700c8100 msgr2=0x7f237014bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236ffff640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23700c8100 0x7f237014bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236ffff640 1 -- 172.21.15.107:0/2987062609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2370152040 msgr2=0x7f23700a9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:38.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236ffff640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2370152040 0x7f23700a9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:38.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236ffff640 1 -- 172.21.15.107:0/2987062609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2368009b30 con 0x7f237006a4b0 2023-10-25T00:22:38.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236ffff640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237006a4b0 0x7f237014c620 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f236000c450 tx=0x7f236000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:38.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236cff9640 1 -- 172.21.15.107:0/2987062609 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360012830 con 0x7f237006a4b0 2023-10-25T00:22:38.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236cff9640 1 -- 172.21.15.107:0/2987062609 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2360012990 con 0x7f237006a4b0 2023-10-25T00:22:38.076 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 -- 172.21.15.107:0/2987062609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237014d410 con 0x7f237006a4b0 2023-10-25T00:22:38.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f236cff9640 1 -- 172.21.15.107:0/2987062609 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360012d40 con 0x7f237006a4b0 2023-10-25T00:22:38.077 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.068+0000 7f237660d640 1 -- 172.21.15.107:0/2987062609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237014cb60 con 0x7f237006a4b0 2023-10-25T00:22:38.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.072+0000 7f237660d640 1 -- 172.21.15.107:0/2987062609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23700746e0 con 0x7f237006a4b0 2023-10-25T00:22:38.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.072+0000 7f236cff9640 1 -- 172.21.15.107:0/2987062609 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f236001c460 con 0x7f237006a4b0 2023-10-25T00:22:38.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.072+0000 7f236cff9640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f235c065e60 0x7f235c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:38.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.072+0000 7f236cff9640 1 -- 172.21.15.107:0/2987062609 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f2360084950 con 0x7f237006a4b0 2023-10-25T00:22:38.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.072+0000 7f236f7fe640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f235c065e60 0x7f235c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:38.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.072+0000 7f236f7fe640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f235c065e60 0x7f235c068320 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f23700c66a0 tx=0x7f2364008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:38.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.076+0000 7f236cff9640 1 -- 172.21.15.107:0/2987062609 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2360050d90 con 0x7f237006a4b0 2023-10-25T00:22:38.264 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.264+0000 7f237660d640 1 -- 172.21.15.107:0/2987062609 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f237014eae0 con 0x7f237006a4b0 2023-10-25T00:22:38.272 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.264+0000 7f236cff9640 1 -- 172.21.15.107:0/2987062609 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5768 (secure 0 0 0) 0x7f23600543b0 con 0x7f237006a4b0 2023-10-25T00:22:38.272 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:38.273 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:21:42.241858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9009":{"gid":9009,"name":"c","rank":0,"incarnation":226,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:22:38.274 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 230 2023-10-25T00:22:38.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 -- 172.21.15.107:0/2987062609 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f235c065e60 msgr2=0x7f235c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:38.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f235c065e60 0x7f235c068320 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f23700c66a0 tx=0x7f2364008040 comp rx=0 tx=0).stop 2023-10-25T00:22:38.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 -- 172.21.15.107:0/2987062609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237006a4b0 msgr2=0x7f237014c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:38.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237006a4b0 0x7f237014c620 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f236000c450 tx=0x7f236000c910 comp rx=0 tx=0).stop 2023-10-25T00:22:38.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 -- 172.21.15.107:0/2987062609 shutdown_connections 2023-10-25T00:22:38.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2370152040 0x7f23700a9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:38.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f235c065e60 0x7f235c068320 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:38.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f23700c8100 0x7f237014bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:38.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 --2- 172.21.15.107:0/2987062609 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f237006a4b0 0x7f237014c620 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:38.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 -- 172.21.15.107:0/2987062609 >> 172.21.15.107:0/2987062609 conn(0x7f23700ca200 msgr2=0x7f23700c4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:38.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 -- 172.21.15.107:0/2987062609 shutdown_connections 2023-10-25T00:22:38.277 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:38.268+0000 7f237660d640 1 -- 172.21.15.107:0/2987062609 wait complete. 2023-10-25T00:22:38.285 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:22:40.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:22:40.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.416+0000 7f7de6ac5640 1 Processor -- start 2023-10-25T00:22:40.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.416+0000 7f7de6ac5640 1 -- start start 2023-10-25T00:22:40.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7de6ac5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de00a9820 0x7f7de00ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:40.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7de6ac5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7de006ee40 0x7f7de00aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:40.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7de6ac5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de006f730 con 0x7f7de014fd40 2023-10-25T00:22:40.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7de6ac5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de00aa750 con 0x7f7de006ee40 2023-10-25T00:22:40.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7de6ac5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de0156a40 con 0x7f7de00a9820 2023-10-25T00:22:40.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7ddffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de00a9820 0x7f7de00ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:40.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7de5062640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7de006ee40 0x7f7de00aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:40.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7ddffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de00a9820 0x7f7de00ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:45124/0 (socket says 172.21.15.107:45124) 2023-10-25T00:22:40.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7ddffff640 1 -- 172.21.15.107:0/3688375157 learned_addr learned my addr 172.21.15.107:0/3688375157 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:40.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7de5062640 1 -- 172.21.15.107:0/3688375157 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de00a9820 msgr2=0x7f7de00ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:40.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7de5062640 1 --2- 172.21.15.107:0/3688375157 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de00a9820 0x7f7de00ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:40.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7de5062640 1 -- 172.21.15.107:0/3688375157 >> v1:172.21.15.110:6789/0 conn(0x7f7de014fd40 legacy=0x7f7de00c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:40.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.420+0000 7f7de5062640 1 -- 172.21.15.107:0/3688375157 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de0156710 con 0x7f7de006ee40 2023-10-25T00:22:40.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7de5062640 1 --2- 172.21.15.107:0/3688375157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7de006ee40 0x7f7de00aa210 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009d90 tx=0x7f7dd801c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=942051275ae39ff server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7ddf7fe640 1 -- 172.21.15.107:0/3688375157 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd80a5860 con 0x7f7de006ee40 2023-10-25T00:22:40.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7ddf7fe640 1 -- 172.21.15.107:0/3688375157 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd80a59c0 con 0x7f7de006ee40 2023-10-25T00:22:40.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7ddf7fe640 1 -- 172.21.15.107:0/3688375157 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd80a5d50 con 0x7f7de006ee40 2023-10-25T00:22:40.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3688375157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7de006ee40 msgr2=0x7f7de00aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:40.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7de6ac5640 1 --2- 172.21.15.107:0/3688375157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7de006ee40 0x7f7de00aa210 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8009d90 tx=0x7f7dd801c800 comp rx=0 tx=0).stop 2023-10-25T00:22:40.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3688375157 shutdown_connections 2023-10-25T00:22:40.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7de6ac5640 1 --2- 172.21.15.107:0/3688375157 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7de006ee40 0x7f7de00aa210 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:40.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7de6ac5640 1 --2- 172.21.15.107:0/3688375157 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de00a9820 0x7f7de00ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:40.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3688375157 >> 172.21.15.107:0/3688375157 conn(0x7f7de00c2950 msgr2=0x7f7de00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:40.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3688375157 shutdown_connections 2023-10-25T00:22:40.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3688375157 wait complete. 2023-10-25T00:22:40.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7de6ac5640 1 Processor -- start 2023-10-25T00:22:40.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.424+0000 7f7de6ac5640 1 -- start start 2023-10-25T00:22:40.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de6ac5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de006ee40 0x7f7de015b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:40.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de6ac5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7de014fd40 0x7f7de01511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:40.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de6ac5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de0151730 0x7f7de01469c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:40.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de6ac5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de0153f50 con 0x7f7de006ee40 2023-10-25T00:22:40.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de6ac5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de0148030 con 0x7f7de014fd40 2023-10-25T00:22:40.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de6ac5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de0150d60 con 0x7f7de0151730 2023-10-25T00:22:40.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de4861640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de006ee40 0x7f7de015b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:40.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de4861640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de006ee40 0x7f7de015b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45702/0 (socket says 172.21.15.107:45702) 2023-10-25T00:22:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de4861640 1 -- 172.21.15.107:0/3943421415 learned_addr learned my addr 172.21.15.107:0/3943421415 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7ddffff640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7de014fd40 0x7f7de01511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:40.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de5062640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de0151730 0x7f7de01469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:40.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de4861640 1 -- 172.21.15.107:0/3943421415 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de0151730 msgr2=0x7f7de01469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:40.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de4861640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de0151730 0x7f7de01469c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:40.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de4861640 1 -- 172.21.15.107:0/3943421415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7de014fd40 msgr2=0x7f7de01511f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:40.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de4861640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7de014fd40 0x7f7de01511f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:40.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de4861640 1 -- 172.21.15.107:0/3943421415 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd8009a70 con 0x7f7de006ee40 2023-10-25T00:22:40.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7ddffff640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7de014fd40 0x7f7de01511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:40.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de5062640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de0151730 0x7f7de01469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:40.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de4861640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de006ee40 0x7f7de015b5a0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400c370 tx=0x7f7dd400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:40.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7ddd7fa640 1 -- 172.21.15.107:0/3943421415 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40052d0 con 0x7f7de006ee40 2023-10-25T00:22:40.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7ddd7fa640 1 -- 172.21.15.107:0/3943421415 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd4005430 con 0x7f7de006ee40 2023-10-25T00:22:40.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3943421415 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de0150450 con 0x7f7de006ee40 2023-10-25T00:22:40.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7ddd7fa640 1 -- 172.21.15.107:0/3943421415 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4007260 con 0x7f7de006ee40 2023-10-25T00:22:40.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3943421415 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de0150710 con 0x7f7de006ee40 2023-10-25T00:22:40.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3943421415 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de00c24a0 con 0x7f7de006ee40 2023-10-25T00:22:40.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7ddd7fa640 1 -- 172.21.15.107:0/3943421415 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7dd40073c0 con 0x7f7de006ee40 2023-10-25T00:22:40.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.428+0000 7f7ddd7fa640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7dc4065e90 0x7f7dc4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:40.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.432+0000 7f7ddffff640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7dc4065e90 0x7f7dc4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:40.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.432+0000 7f7ddd7fa640 1 -- 172.21.15.107:0/3943421415 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f7dd4083e60 con 0x7f7de006ee40 2023-10-25T00:22:40.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.432+0000 7f7ddffff640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7dc4065e90 0x7f7dc4068350 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7de015bdc0 tx=0x7f7dcc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:40.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.436+0000 7f7ddd7fa640 1 -- 172.21.15.107:0/3943421415 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7dd4050210 con 0x7f7de006ee40 2023-10-25T00:22:40.625 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.624+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3943421415 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7de0070840 con 0x7f7de006ee40 2023-10-25T00:22:40.626 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.624+0000 7f7ddd7fa640 1 -- 172.21.15.107:0/3943421415 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5768 (secure 0 0 0) 0x7f7dd4053830 con 0x7f7de006ee40 2023-10-25T00:22:40.626 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:40.626 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:21:42.241858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9009":{"gid":9009,"name":"c","rank":0,"incarnation":226,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:22:40.627 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 230 2023-10-25T00:22:40.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3943421415 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7dc4065e90 msgr2=0x7f7dc4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:40.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7dc4065e90 0x7f7dc4068350 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f7de015bdc0 tx=0x7f7dcc008040 comp rx=0 tx=0).stop 2023-10-25T00:22:40.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3943421415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de006ee40 msgr2=0x7f7de015b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:40.628 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de006ee40 0x7f7de015b5a0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400c370 tx=0x7f7dd400c830 comp rx=0 tx=0).stop 2023-10-25T00:22:40.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3943421415 shutdown_connections 2023-10-25T00:22:40.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7dc4065e90 0x7f7dc4068350 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:40.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7de0151730 0x7f7de01469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:40.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7de014fd40 0x7f7de01511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:40.629 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 --2- 172.21.15.107:0/3943421415 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7de006ee40 0x7f7de015b5a0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:40.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3943421415 >> 172.21.15.107:0/3943421415 conn(0x7f7de00c2950 msgr2=0x7f7de00c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:40.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3943421415 shutdown_connections 2023-10-25T00:22:40.630 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:40.628+0000 7f7de6ac5640 1 -- 172.21.15.107:0/3943421415 wait complete. 2023-10-25T00:22:40.645 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:22:42.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:22:42.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c291e5640 1 Processor -- start 2023-10-25T00:22:42.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c291e5640 1 -- start start 2023-10-25T00:22:42.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c291e5640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6c240c8100 0x7f6c240abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:42.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c291e5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c24155680 con 0x7f6c2406a4b0 2023-10-25T00:22:42.778 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c291e5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c24154a70 con 0x7f6c24152040 2023-10-25T00:22:42.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c291e5640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c24154d50 con 0x7f6c240c8100 2023-10-25T00:22:42.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2359e640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6c2406a4b0 0x7f6c240ac240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42268/0 (socket says 172.21.15.107:42268) 2023-10-25T00:22:42.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2359e640 1 -- 172.21.15.107:0/3049171322 learned_addr learned my addr 172.21.15.107:0/3049171322 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:42.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c1bfff640 1 --2- 172.21.15.107:0/3049171322 >> v2:172.21.15.110:3300/0 conn(0x7f6c240c8100 0x7f6c240abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:42.779 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696679278 0 0) 0x7f6c24155680 con 0x7f6c2406a4b0 2023-10-25T00:22:42.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c10003530 con 0x7f6c2406a4b0 2023-10-25T00:22:42.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 770375532 0 0) 0x7f6c10003530 con 0x7f6c2406a4b0 2023-10-25T00:22:42.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 >> v2:172.21.15.110:3300/0 conn(0x7f6c240c8100 msgr2=0x7f6c240abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:42.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 --2- 172.21.15.107:0/3049171322 >> v2:172.21.15.110:3300/0 conn(0x7f6c240c8100 0x7f6c240abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.780 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 >> v1:172.21.15.110:6789/0 conn(0x7f6c24152040 legacy=0x7f6c240c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c24155030 con 0x7f6c2406a4b0 2023-10-25T00:22:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f6c1c001a00 con 0x7f6c2406a4b0 2023-10-25T00:22:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 >> v1:172.21.15.107:6789/0 conn(0x7f6c2406a4b0 legacy=0x7f6c240ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 --2- 172.21.15.107:0/3049171322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c10003980 0x7f6c10003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:42.781 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 --2- 172.21.15.107:0/3049171322 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c100047d0 0x7f6c10004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:42.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 --2- 172.21.15.107:0/3049171322 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c10005190 0x7f6c1000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:42.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c24154d50 con 0x7f6c10003980 2023-10-25T00:22:42.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1000c130 con 0x7f6c10005190 2023-10-25T00:22:42.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c1000c2f0 con 0x7f6c100047d0 2023-10-25T00:22:42.782 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c22d9d640 1 --2- 172.21.15.107:0/3049171322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c10003980 0x7f6c10003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:42.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c1bfff640 1 --2- 172.21.15.107:0/3049171322 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c100047d0 0x7f6c10004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:42.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2359e640 1 --2- 172.21.15.107:0/3049171322 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c10005190 0x7f6c1000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:42.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c22d9d640 1 -- 172.21.15.107:0/3049171322 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c100047d0 msgr2=0x7f6c10004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:42.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c22d9d640 1 --2- 172.21.15.107:0/3049171322 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c100047d0 0x7f6c10004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c22d9d640 1 -- 172.21.15.107:0/3049171322 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c10005190 msgr2=0x7f6c1000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:42.783 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c22d9d640 1 --2- 172.21.15.107:0/3049171322 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c10005190 0x7f6c1000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c2359e640 1 -- 172.21.15.107:0/3049171322 reap_dead start 2023-10-25T00:22:42.784 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.776+0000 7f6c22d9d640 1 -- 172.21.15.107:0/3049171322 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c10003530 con 0x7f6c10003980 2023-10-25T00:22:42.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c22d9d640 1 --2- 172.21.15.107:0/3049171322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c10003980 0x7f6c10003da0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c001780 tx=0x7f6c0c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72dc4543eb1f8fb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:42.787 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c0a4ea0 con 0x7f6c10003980 2023-10-25T00:22:42.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c0a7050 con 0x7f6c10003980 2023-10-25T00:22:42.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c2259c640 1 -- 172.21.15.107:0/3049171322 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c0052d0 con 0x7f6c10003980 2023-10-25T00:22:42.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c291e5640 1 -- 172.21.15.107:0/3049171322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c10003980 msgr2=0x7f6c10003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:42.788 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c291e5640 1 --2- 172.21.15.107:0/3049171322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c10003980 0x7f6c10003da0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c001780 tx=0x7f6c0c01c3e0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c291e5640 1 -- 172.21.15.107:0/3049171322 shutdown_connections 2023-10-25T00:22:42.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c291e5640 1 --2- 172.21.15.107:0/3049171322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c10003980 0x7f6c10003da0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c291e5640 1 -- 172.21.15.107:0/3049171322 >> 172.21.15.107:0/3049171322 conn(0x7f6c240ca200 msgr2=0x7f6c240ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:42.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c291e5640 1 -- 172.21.15.107:0/3049171322 shutdown_connections 2023-10-25T00:22:42.789 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c291e5640 1 -- 172.21.15.107:0/3049171322 wait complete. 2023-10-25T00:22:42.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.780+0000 7f6c291e5640 1 Processor -- start 2023-10-25T00:22:42.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c291e5640 1 -- start start 2023-10-25T00:22:42.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c291e5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c10003980 0x7f6c241ed570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:42.790 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c291e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c241f0ad0 0x7f6c24148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:42.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c291e5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c241e1620 0x7f6c2414b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:42.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c291e5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c241edab0 con 0x7f6c241f0ad0 2023-10-25T00:22:42.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c291e5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c241f0080 con 0x7f6c241e1620 2023-10-25T00:22:42.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c291e5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c241f0640 con 0x7f6c10003980 2023-10-25T00:22:42.791 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c241f0ad0 0x7f6c24148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:42.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c241f0ad0 0x7f6c24148910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45278/0 (socket says 172.21.15.107:45278) 2023-10-25T00:22:42.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1bfff640 1 -- 172.21.15.107:0/1881210424 learned_addr learned my addr 172.21.15.107:0/1881210424 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:42.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c22d9d640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c10003980 0x7f6c241ed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:42.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c2359e640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c241e1620 0x7f6c2414b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:42.792 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1bfff640 1 -- 172.21.15.107:0/1881210424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c10003980 msgr2=0x7f6c241ed570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:42.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1bfff640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c10003980 0x7f6c241ed570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1bfff640 1 -- 172.21.15.107:0/1881210424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c241e1620 msgr2=0x7f6c2414b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:42.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1bfff640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c241e1620 0x7f6c2414b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1bfff640 1 -- 172.21.15.107:0/1881210424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2414aa90 con 0x7f6c241f0ad0 2023-10-25T00:22:42.793 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1bfff640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c241f0ad0 0x7f6c24148910 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f6c14011700 tx=0x7f6c14011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:42.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1b7fe640 1 -- 172.21.15.107:0/1881210424 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1400f960 con 0x7f6c241f0ad0 2023-10-25T00:22:42.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1b7fe640 1 -- 172.21.15.107:0/1881210424 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1400fac0 con 0x7f6c241f0ad0 2023-10-25T00:22:42.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c291e5640 1 -- 172.21.15.107:0/1881210424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2414ad70 con 0x7f6c241f0ad0 2023-10-25T00:22:42.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1b7fe640 1 -- 172.21.15.107:0/1881210424 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1400fe50 con 0x7f6c241f0ad0 2023-10-25T00:22:42.794 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c291e5640 1 -- 172.21.15.107:0/1881210424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c24148e50 con 0x7f6c241f0ad0 2023-10-25T00:22:42.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c291e5640 1 -- 172.21.15.107:0/1881210424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c241f07f0 con 0x7f6c241f0ad0 2023-10-25T00:22:42.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1b7fe640 1 -- 172.21.15.107:0/1881210424 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6c14015460 con 0x7f6c241f0ad0 2023-10-25T00:22:42.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1b7fe640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c04065e60 0x7f6c04068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:42.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c1b7fe640 1 -- 172.21.15.107:0/1881210424 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f6c140852a0 con 0x7f6c241f0ad0 2023-10-25T00:22:42.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.784+0000 7f6c22d9d640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c04065e60 0x7f6c04068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:42.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.788+0000 7f6c22d9d640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c04065e60 0x7f6c04068320 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c001750 tx=0x7f6c0c0011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:42.796 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.788+0000 7f6c1b7fe640 1 -- 172.21.15.107:0/1881210424 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6c241f07f0 con 0x7f6c241f0ad0 2023-10-25T00:22:42.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 -- 172.21.15.107:0/1881210424 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c240746e0 con 0x7f6c241f0ad0 2023-10-25T00:22:42.980 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c1b7fe640 1 -- 172.21.15.107:0/1881210424 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5768 (secure 0 0 0) 0x7f6c14051660 con 0x7f6c241f0ad0 2023-10-25T00:22:42.980 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:42.981 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:21:42.241858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9009":{"gid":9009,"name":"c","rank":0,"incarnation":226,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:22:42.981 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 230 2023-10-25T00:22:42.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 -- 172.21.15.107:0/1881210424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c04065e60 msgr2=0x7f6c04068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:42.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c04065e60 0x7f6c04068320 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c001750 tx=0x7f6c0c0011a0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.982 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 -- 172.21.15.107:0/1881210424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c241f0ad0 msgr2=0x7f6c24148910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c241f0ad0 0x7f6c24148910 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f6c14011700 tx=0x7f6c14011bc0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 -- 172.21.15.107:0/1881210424 shutdown_connections 2023-10-25T00:22:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6c241e1620 0x7f6c2414b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c241f0ad0 0x7f6c24148910 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6c04065e60 0x7f6c04068320 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 --2- 172.21.15.107:0/1881210424 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6c10003980 0x7f6c241ed570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:42.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 -- 172.21.15.107:0/1881210424 >> 172.21.15.107:0/1881210424 conn(0x7f6c240ca200 msgr2=0x7f6c2406a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:42.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 -- 172.21.15.107:0/1881210424 shutdown_connections 2023-10-25T00:22:42.984 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:42.980+0000 7f6c291e5640 1 -- 172.21.15.107:0/1881210424 wait complete. 2023-10-25T00:22:42.999 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:22:45.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:22:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.128+0000 7f5346862640 1 Processor -- start 2023-10-25T00:22:45.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.128+0000 7f5346862640 1 -- start start 2023-10-25T00:22:45.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5346862640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f534014b600 0x7f5340151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:45.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5346862640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f534006af70 0x7f534006c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:45.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5346862640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f534006a230 0x7f53401551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5346862640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53400c1860 con 0x7f534006af70 2023-10-25T00:22:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5346862640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5340154d50 con 0x7f534006a230 2023-10-25T00:22:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5346862640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53401524f0 con 0x7f534014b600 2023-10-25T00:22:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5337fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f534006af70 0x7f534006c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5337fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f534006af70 0x7f534006c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45290/0 (socket says 172.21.15.107:45290) 2023-10-25T00:22:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5337fff640 1 -- 172.21.15.107:0/896782722 learned_addr learned my addr 172.21.15.107:0/896782722 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f533ffff640 1 --2- 172.21.15.107:0/896782722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f534014b600 0x7f5340151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5344dff640 1 --2- 172.21.15.107:0/896782722 >> v2:172.21.15.110:3300/0 conn(0x7f534006a230 0x7f53401551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5337fff640 1 -- 172.21.15.107:0/896782722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f534014b600 msgr2=0x7f5340151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5337fff640 1 --2- 172.21.15.107:0/896782722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f534014b600 0x7f5340151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5337fff640 1 -- 172.21.15.107:0/896782722 >> v2:172.21.15.110:3300/0 conn(0x7f534006a230 msgr2=0x7f53401551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5337fff640 1 --2- 172.21.15.107:0/896782722 >> v2:172.21.15.110:3300/0 conn(0x7f534006a230 0x7f53401551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5337fff640 1 -- 172.21.15.107:0/896782722 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534014b9e0 con 0x7f534006af70 2023-10-25T00:22:45.153 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f5337fff640 1 --2- 172.21.15.107:0/896782722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f534006af70 0x7f534006c4c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f534006a7a0 tx=0x7f532801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1941f1ec0bdc0dc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:45.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f533f7fe640 1 -- 172.21.15.107:0/896782722 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53280ab600 con 0x7f534006af70 2023-10-25T00:22:45.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f533f7fe640 1 -- 172.21.15.107:0/896782722 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53280a7050 con 0x7f534006af70 2023-10-25T00:22:45.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.132+0000 7f533f7fe640 1 -- 172.21.15.107:0/896782722 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53280ab980 con 0x7f534006af70 2023-10-25T00:22:45.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 -- 172.21.15.107:0/896782722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f534006af70 msgr2=0x7f534006c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:45.154 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 --2- 172.21.15.107:0/896782722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f534006af70 0x7f534006c4c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f534006a7a0 tx=0x7f532801c440 comp rx=0 tx=0).stop 2023-10-25T00:22:45.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 -- 172.21.15.107:0/896782722 shutdown_connections 2023-10-25T00:22:45.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 --2- 172.21.15.107:0/896782722 >> v2:172.21.15.110:3300/0 conn(0x7f534006a230 0x7f53401551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 --2- 172.21.15.107:0/896782722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f534006af70 0x7f534006c4c0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.155 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 --2- 172.21.15.107:0/896782722 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f534014b600 0x7f5340151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 -- 172.21.15.107:0/896782722 >> 172.21.15.107:0/896782722 conn(0x7f53400c2950 msgr2=0x7f53400c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:45.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 -- 172.21.15.107:0/896782722 shutdown_connections 2023-10-25T00:22:45.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 -- 172.21.15.107:0/896782722 wait complete. 2023-10-25T00:22:45.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 Processor -- start 2023-10-25T00:22:45.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 -- start start 2023-10-25T00:22:45.156 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f534006a230 0x7f53401ef120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:45.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f534006af70 0x7f5340153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:45.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f534014b600 0x7f534014df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:45.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534014c9a0 con 0x7f534006af70 2023-10-25T00:22:45.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53401ef660 con 0x7f534006a230 2023-10-25T00:22:45.157 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5346862640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53401eb2b0 con 0x7f534014b600 2023-10-25T00:22:45.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5337fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f534006af70 0x7f5340153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:45.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5337fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f534006af70 0x7f5340153bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45296/0 (socket says 172.21.15.107:45296) 2023-10-25T00:22:45.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5337fff640 1 -- 172.21.15.107:0/3499399101 learned_addr learned my addr 172.21.15.107:0/3499399101 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:45.158 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5344dff640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f534014b600 0x7f534014df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:45.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f533ffff640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f534006a230 0x7f53401ef120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:45.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5337fff640 1 -- 172.21.15.107:0/3499399101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f534014b600 msgr2=0x7f534014df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:45.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5337fff640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f534014b600 0x7f534014df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5337fff640 1 -- 172.21.15.107:0/3499399101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f534006a230 msgr2=0x7f53401ef120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:45.159 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5337fff640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f534006a230 0x7f53401ef120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5337fff640 1 -- 172.21.15.107:0/3499399101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53280094d0 con 0x7f534006af70 2023-10-25T00:22:45.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f533ffff640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f534006a230 0x7f53401ef120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:45.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f5337fff640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f534006af70 0x7f5340153bd0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f53280ae040 tx=0x7f53280191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:45.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.136+0000 7f533d7fa640 1 -- 172.21.15.107:0/3499399101 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53280b4420 con 0x7f534006af70 2023-10-25T00:22:45.160 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.140+0000 7f533d7fa640 1 -- 172.21.15.107:0/3499399101 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53280b4580 con 0x7f534006af70 2023-10-25T00:22:45.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.140+0000 7f5346862640 1 -- 172.21.15.107:0/3499399101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53401eb590 con 0x7f534006af70 2023-10-25T00:22:45.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.140+0000 7f533d7fa640 1 -- 172.21.15.107:0/3499399101 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53280b4930 con 0x7f534006af70 2023-10-25T00:22:45.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.140+0000 7f5346862640 1 -- 172.21.15.107:0/3499399101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53401ebb50 con 0x7f534006af70 2023-10-25T00:22:45.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.140+0000 7f5346862640 1 -- 172.21.15.107:0/3499399101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53400746e0 con 0x7f534006af70 2023-10-25T00:22:45.161 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.140+0000 7f533d7fa640 1 -- 172.21.15.107:0/3499399101 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f53280b2070 con 0x7f534006af70 2023-10-25T00:22:45.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.140+0000 7f533d7fa640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e60 0x7f5320068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:45.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.140+0000 7f533d7fa640 1 -- 172.21.15.107:0/3499399101 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7f532811f0d0 con 0x7f534006af70 2023-10-25T00:22:45.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.140+0000 7f533ffff640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e60 0x7f5320068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:45.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.140+0000 7f533ffff640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e60 0x7f5320068320 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f53300097a0 tx=0x7f5330008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:45.162 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.144+0000 7f533d7fa640 1 -- 172.21.15.107:0/3499399101 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f53280eb590 con 0x7f534006af70 2023-10-25T00:22:45.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.332+0000 7f5346862640 1 -- 172.21.15.107:0/3499399101 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53400c3260 con 0x7f534006af70 2023-10-25T00:22:45.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.332+0000 7f533d7fa640 1 -- 172.21.15.107:0/3499399101 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5768 (secure 0 0 0) 0x7f53280eebb0 con 0x7f534006af70 2023-10-25T00:22:45.334 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:45.334 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:21:42.241858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9009":{"gid":9009,"name":"c","rank":0,"incarnation":226,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:22:45.335 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 230 2023-10-25T00:22:45.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 -- 172.21.15.107:0/3499399101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e60 msgr2=0x7f5320068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:45.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e60 0x7f5320068320 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f53300097a0 tx=0x7f5330008040 comp rx=0 tx=0).stop 2023-10-25T00:22:45.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 -- 172.21.15.107:0/3499399101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f534006af70 msgr2=0x7f5340153bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:45.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f534006af70 0x7f5340153bd0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f53280ae040 tx=0x7f53280191e0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 -- 172.21.15.107:0/3499399101 shutdown_connections 2023-10-25T00:22:45.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5320065e60 0x7f5320068320 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f534014b600 0x7f534014df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f534006af70 0x7f5340153bd0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 --2- 172.21.15.107:0/3499399101 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f534006a230 0x7f53401ef120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:45.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 -- 172.21.15.107:0/3499399101 >> 172.21.15.107:0/3499399101 conn(0x7f53400c2950 msgr2=0x7f53400c0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:45.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 -- 172.21.15.107:0/3499399101 shutdown_connections 2023-10-25T00:22:45.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:45.336+0000 7f5346862640 1 -- 172.21.15.107:0/3499399101 wait complete. 2023-10-25T00:22:45.353 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:22:47.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:22:47.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd164ff8640 1 Processor -- start 2023-10-25T00:22:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd164ff8640 1 -- start start 2023-10-25T00:22:47.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd164ff8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd16014fd40 0x7fd1600ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:47.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd164ff8640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16006f730 con 0x7fd1600a9820 2023-10-25T00:22:47.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd164ff8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd160156a40 con 0x7fd16006ee40 2023-10-25T00:22:47.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd164ff8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd160156710 con 0x7fd16014fd40 2023-10-25T00:22:47.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15dd9b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fd1600a9820 0x7fd1600c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42284/0 (socket says 172.21.15.107:42284) 2023-10-25T00:22:47.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15dd9b640 1 -- 172.21.15.107:0/3954200366 learned_addr learned my addr 172.21.15.107:0/3954200366 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:47.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15e59c640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd16014fd40 0x7fd1600ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:47.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323520058 0 0) 0x7fd16006f730 con 0x7fd1600a9820 2023-10-25T00:22:47.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd148003530 con 0x7fd1600a9820 2023-10-25T00:22:47.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3778934073 0 0) 0x7fd148003530 con 0x7fd1600a9820 2023-10-25T00:22:47.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd16014fd40 msgr2=0x7fd1600ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:47.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd16014fd40 0x7fd1600ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:47.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 >> v1:172.21.15.110:6789/0 conn(0x7fd16006ee40 legacy=0x7fd16015cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:47.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16006f980 con 0x7fd1600a9820 2023-10-25T00:22:47.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fd154001680 con 0x7fd1600a9820 2023-10-25T00:22:47.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 >> v1:172.21.15.107:6789/0 conn(0x7fd1600a9820 legacy=0x7fd1600c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:47.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1480039b0 0x7fd148003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:47.493 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd16014fd40 0x7fd1480047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:47.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 0x7fd1480051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:47.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd160156710 con 0x7fd148004d60 2023-10-25T00:22:47.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800bc10 con 0x7fd1480039b0 2023-10-25T00:22:47.494 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14800bdd0 con 0x7fd16014fd40 2023-10-25T00:22:47.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15ed9d640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 0x7fd1480051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:47.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15dd9b640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd16014fd40 0x7fd1480047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:47.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15e59c640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1480039b0 0x7fd148003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:47.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15ed9d640 1 -- 172.21.15.107:0/3954200366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd16014fd40 msgr2=0x7fd1480047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:47.495 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15ed9d640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd16014fd40 0x7fd1480047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:47.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15ed9d640 1 -- 172.21.15.107:0/3954200366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1480039b0 msgr2=0x7fd148003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:47.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15ed9d640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1480039b0 0x7fd148003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:47.496 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.488+0000 7fd15ed9d640 1 -- 172.21.15.107:0/3954200366 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148003530 con 0x7fd148004d60 2023-10-25T00:22:47.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd15ed9d640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 0x7fd1480051e0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fd158001e40 tx=0x7fd15801c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec62bc3a5ae01d05 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:47.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158003b80 con 0x7fd148004d60 2023-10-25T00:22:47.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd158003ce0 con 0x7fd148004d60 2023-10-25T00:22:47.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd15d59a640 1 -- 172.21.15.107:0/3954200366 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1580052c0 con 0x7fd148004d60 2023-10-25T00:22:47.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 -- 172.21.15.107:0/3954200366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 msgr2=0x7fd1480051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:47.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 --2- 172.21.15.107:0/3954200366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 0x7fd1480051e0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fd158001e40 tx=0x7fd15801c720 comp rx=0 tx=0).stop 2023-10-25T00:22:47.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd15ed9d640 1 -- 172.21.15.107:0/3954200366 reap_dead start 2023-10-25T00:22:47.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 -- 172.21.15.107:0/3954200366 shutdown_connections 2023-10-25T00:22:47.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 -- 172.21.15.107:0/3954200366 >> 172.21.15.107:0/3954200366 conn(0x7fd1600c2950 msgr2=0x7fd1600c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:47.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 -- 172.21.15.107:0/3954200366 shutdown_connections 2023-10-25T00:22:47.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 -- 172.21.15.107:0/3954200366 wait complete. 2023-10-25T00:22:47.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 Processor -- start 2023-10-25T00:22:47.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 -- start start 2023-10-25T00:22:47.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 0x7fd160143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:47.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1600c00d0 0x7fd16015d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:47.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd160143750 0x7fd1601ed8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:47.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601ed4a0 con 0x7fd148004d60 2023-10-25T00:22:47.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16006af30 con 0x7fd1600c00d0 2023-10-25T00:22:47.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd164ff8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd16006bd00 con 0x7fd160143750 2023-10-25T00:22:47.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd15e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 0x7fd160143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:47.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd15e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 0x7fd160143210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45320/0 (socket says 172.21.15.107:45320) 2023-10-25T00:22:47.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.492+0000 7fd15e59c640 1 -- 172.21.15.107:0/1138822896 learned_addr learned my addr 172.21.15.107:0/1138822896 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:47.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd15ed9d640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd160143750 0x7fd1601ed8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:47.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd15dd9b640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1600c00d0 0x7fd16015d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:47.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd15e59c640 1 -- 172.21.15.107:0/1138822896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd160143750 msgr2=0x7fd1601ed8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:47.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd15e59c640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd160143750 0x7fd1601ed8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:47.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd15e59c640 1 -- 172.21.15.107:0/1138822896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1600c00d0 msgr2=0x7fd16015d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:47.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd15e59c640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1600c00d0 0x7fd16015d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:47.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd15e59c640 1 -- 172.21.15.107:0/1138822896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd160069d40 con 0x7fd148004d60 2023-10-25T00:22:47.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd15e59c640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 0x7fd160143210 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fd15000e840 tx=0x7fd15000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:47.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd14effd640 1 -- 172.21.15.107:0/1138822896 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15000c6d0 con 0x7fd148004d60 2023-10-25T00:22:47.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd14effd640 1 -- 172.21.15.107:0/1138822896 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15000c830 con 0x7fd148004d60 2023-10-25T00:22:47.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd164ff8640 1 -- 172.21.15.107:0/1138822896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16006b460 con 0x7fd148004d60 2023-10-25T00:22:47.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd14effd640 1 -- 172.21.15.107:0/1138822896 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15000cbe0 con 0x7fd148004d60 2023-10-25T00:22:47.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd164ff8640 1 -- 172.21.15.107:0/1138822896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16006ba20 con 0x7fd148004d60 2023-10-25T00:22:47.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd164ff8640 1 -- 172.21.15.107:0/1138822896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1600c24a0 con 0x7fd148004d60 2023-10-25T00:22:47.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd14effd640 1 -- 172.21.15.107:0/1138822896 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fd15000cd40 con 0x7fd148004d60 2023-10-25T00:22:47.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd14effd640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd140065e90 0x7fd140068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:47.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd14effd640 1 -- 172.21.15.107:0/1138822896 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 10745+0+0 (secure 0 0 0) 0x7fd1500847f0 con 0x7fd148004d60 2023-10-25T00:22:47.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd15dd9b640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd140065e90 0x7fd140068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:47.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.496+0000 7fd15dd9b640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd140065e90 0x7fd140068350 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd154001ec0 tx=0x7fd154009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:47.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.500+0000 7fd14effd640 1 -- 172.21.15.107:0/1138822896 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fd150050ba0 con 0x7fd148004d60 2023-10-25T00:22:47.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.688+0000 7fd164ff8640 1 -- 172.21.15.107:0/1138822896 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1601ed270 con 0x7fd148004d60 2023-10-25T00:22:47.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.688+0000 7fd14effd640 1 -- 172.21.15.107:0/1138822896 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5768 (secure 0 0 0) 0x7fd1601ed270 con 0x7fd148004d60 2023-10-25T00:22:47.690 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:47.691 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:21:42.241858+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9009},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9009":{"gid":9009,"name":"c","rank":0,"incarnation":226,"state":"up:active","state_seq":5,"addr":"172.21.15.107:6835/1528689730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":1528689730},{"type":"v1","addr":"172.21.15.107:6835","nonce":1528689730}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:22:47.691 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 230 2023-10-25T00:22:47.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 -- 172.21.15.107:0/1138822896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd140065e90 msgr2=0x7fd140068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:47.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd140065e90 0x7fd140068350 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd154001ec0 tx=0x7fd154009040 comp rx=0 tx=0).stop 2023-10-25T00:22:47.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 -- 172.21.15.107:0/1138822896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 msgr2=0x7fd160143210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:47.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 0x7fd160143210 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fd15000e840 tx=0x7fd15000ed00 comp rx=0 tx=0).stop 2023-10-25T00:22:47.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 -- 172.21.15.107:0/1138822896 shutdown_connections 2023-10-25T00:22:47.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fd140065e90 0x7fd140068350 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:47.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd160143750 0x7fd1601ed8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:47.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1600c00d0 0x7fd16015d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:47.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 --2- 172.21.15.107:0/1138822896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148004d60 0x7fd160143210 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:47.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 -- 172.21.15.107:0/1138822896 >> 172.21.15.107:0/1138822896 conn(0x7fd1600c2950 msgr2=0x7fd160151c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:47.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 -- 172.21.15.107:0/1138822896 shutdown_connections 2023-10-25T00:22:47.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:47.692+0000 7fd164ff8640 1 -- 172.21.15.107:0/1138822896 wait complete. 2023-10-25T00:22:47.710 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-25T00:22:49.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:22:49.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713fe52640 1 Processor -- start 2023-10-25T00:22:49.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713fe52640 1 -- start start 2023-10-25T00:22:49.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713fe52640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7138150930 0x7f71380a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:49.830 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713fe52640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71380a9b90 con 0x7f71380c1570 2023-10-25T00:22:49.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713fe52640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71380c1390 con 0x7f71381510c0 2023-10-25T00:22:49.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713fe52640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7138154dd0 con 0x7f7138150930 2023-10-25T00:22:49.831 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713d3ed640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f71380c1570 0x7f7138069d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42300/0 (socket says 172.21.15.107:42300) 2023-10-25T00:22:49.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713dbee640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7138150930 0x7f71380a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:49.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713d3ed640 1 -- 172.21.15.107:0/1985397576 learned_addr learned my addr 172.21.15.107:0/1985397576 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:49.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713dbee640 1 -- 172.21.15.107:0/1985397576 >> v1:172.21.15.110:6789/0 conn(0x7f71381510c0 legacy=0x7f7138151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:49.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713dbee640 1 -- 172.21.15.107:0/1985397576 >> v1:172.21.15.107:6789/0 conn(0x7f71380c1570 legacy=0x7f7138069d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:49.833 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713cbec640 1 -- 172.21.15.107:0/1985397576 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008325825 0 0) 0x7f71380a9b90 con 0x7f71380c1570 2023-10-25T00:22:49.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.824+0000 7f713dbee640 1 -- 172.21.15.107:0/1985397576 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71381550b0 con 0x7f7138150930 2023-10-25T00:22:49.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713dbee640 1 --2- 172.21.15.107:0/1985397576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7138150930 0x7f71380a94b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f71280097c0 tx=0x7f712801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97ca5d3c6e5e4ad server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:49.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713cbec640 1 -- 172.21.15.107:0/1985397576 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71280ab600 con 0x7f7138150930 2023-10-25T00:22:49.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713cbec640 1 -- 172.21.15.107:0/1985397576 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71280a7050 con 0x7f7138150930 2023-10-25T00:22:49.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713cbec640 1 -- 172.21.15.107:0/1985397576 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71280ab980 con 0x7f7138150930 2023-10-25T00:22:49.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713fe52640 1 -- 172.21.15.107:0/1985397576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7138150930 msgr2=0x7f71380a94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713fe52640 1 --2- 172.21.15.107:0/1985397576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7138150930 0x7f71380a94b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f71280097c0 tx=0x7f712801c3b0 comp rx=0 tx=0).stop 2023-10-25T00:22:49.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713fe52640 1 -- 172.21.15.107:0/1985397576 shutdown_connections 2023-10-25T00:22:49.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713fe52640 1 --2- 172.21.15.107:0/1985397576 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7138150930 0x7f71380a94b0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:49.837 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713fe52640 1 -- 172.21.15.107:0/1985397576 >> 172.21.15.107:0/1985397576 conn(0x7f71380c65c0 msgr2=0x7f71380c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:49.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713fe52640 1 -- 172.21.15.107:0/1985397576 shutdown_connections 2023-10-25T00:22:49.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713fe52640 1 -- 172.21.15.107:0/1985397576 wait complete. 2023-10-25T00:22:49.838 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713fe52640 1 Processor -- start 2023-10-25T00:22:49.839 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.828+0000 7f713fe52640 1 -- start start 2023-10-25T00:22:49.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713fe52640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71380c1570 0x7f7138072060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713fe52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7138150930 0x7f713814c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713fe52640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f71381510c0 0x7f71381eac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713fe52640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71380ba0f0 con 0x7f7138150930 2023-10-25T00:22:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713fe52640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71381ee9e0 con 0x7f71380c1570 2023-10-25T00:22:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713fe52640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71381eec80 con 0x7f71381510c0 2023-10-25T00:22:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713d3ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7138150930 0x7f713814c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713d3ed640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7138150930 0x7f713814c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45332/0 (socket says 172.21.15.107:45332) 2023-10-25T00:22:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713d3ed640 1 -- 172.21.15.107:0/3409146034 learned_addr learned my addr 172.21.15.107:0/3409146034 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:49.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713e3ef640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f71381510c0 0x7f71381eac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713d3ed640 1 -- 172.21.15.107:0/3409146034 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f71381510c0 msgr2=0x7f71381eac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713d3ed640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f71381510c0 0x7f71381eac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713d3ed640 1 -- 172.21.15.107:0/3409146034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71380c1570 msgr2=0x7f7138072060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713d3ed640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71380c1570 0x7f7138072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713d3ed640 1 -- 172.21.15.107:0/3409146034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7128009420 con 0x7f7138150930 2023-10-25T00:22:49.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713d3ed640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7138150930 0x7f713814c190 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f71340011f0 tx=0x7f713400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:49.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f712e7fc640 1 -- 172.21.15.107:0/3409146034 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71340074e0 con 0x7f7138150930 2023-10-25T00:22:49.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f712e7fc640 1 -- 172.21.15.107:0/3409146034 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7134007640 con 0x7f7138150930 2023-10-25T00:22:49.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713fe52640 1 -- 172.21.15.107:0/3409146034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71380ba340 con 0x7f7138150930 2023-10-25T00:22:49.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f712e7fc640 1 -- 172.21.15.107:0/3409146034 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134004a10 con 0x7f7138150930 2023-10-25T00:22:49.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713fe52640 1 -- 172.21.15.107:0/3409146034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71380725a0 con 0x7f7138150930 2023-10-25T00:22:49.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713fe52640 1 -- 172.21.15.107:0/3409146034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71380ba860 con 0x7f7138150930 2023-10-25T00:22:49.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f712e7fc640 1 -- 172.21.15.107:0/3409146034 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7134004b70 con 0x7f7138150930 2023-10-25T00:22:49.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f712e7fc640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f711c065e90 0x7f711c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:49.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f712e7fc640 1 -- 172.21.15.107:0/3409146034 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f7134083fa0 con 0x7f7138150930 2023-10-25T00:22:49.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713dbee640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f711c065e90 0x7f711c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:49.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.832+0000 7f713dbee640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f711c065e90 0x7f711c068350 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f71280ad000 tx=0x7f7128008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:49.850 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:49.836+0000 7f712e7fc640 1 -- 172.21.15.107:0/3409146034 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f71340b5040 con 0x7f7138150930 2023-10-25T00:22:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.024+0000 7f713fe52640 1 -- 172.21.15.107:0/3409146034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71381eb970 con 0x7f7138150930 2023-10-25T00:22:50.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.024+0000 7f712e7fc640 1 -- 172.21.15.107:0/3409146034 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+5007 (secure 0 0 0) 0x7f7134018020 con 0x7f7138150930 2023-10-25T00:22:50.025 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:49.199320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:22:50.026 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 231 2023-10-25T00:22:50.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.024+0000 7f713fe52640 1 -- 172.21.15.107:0/3409146034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f711c065e90 msgr2=0x7f711c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:50.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.028+0000 7f713fe52640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f711c065e90 0x7f711c068350 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f71280ad000 tx=0x7f7128008040 comp rx=0 tx=0).stop 2023-10-25T00:22:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.028+0000 7f713fe52640 1 -- 172.21.15.107:0/3409146034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7138150930 msgr2=0x7f713814c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.028+0000 7f713fe52640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7138150930 0x7f713814c190 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f71340011f0 tx=0x7f713400c9b0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.028+0000 7f713fe52640 1 -- 172.21.15.107:0/3409146034 shutdown_connections 2023-10-25T00:22:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.028+0000 7f713fe52640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f711c065e90 0x7f711c068350 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.028+0000 7f713fe52640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f71381510c0 0x7f71381eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.028+0000 7f713fe52640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7138150930 0x7f713814c190 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.028+0000 7f713fe52640 1 --2- 172.21.15.107:0/3409146034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f71380c1570 0x7f7138072060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.028+0000 7f713fe52640 1 -- 172.21.15.107:0/3409146034 >> 172.21.15.107:0/3409146034 conn(0x7f71380c65c0 msgr2=0x7f713815b890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.028+0000 7f713fe52640 1 -- 172.21.15.107:0/3409146034 shutdown_connections 2023-10-25T00:22:50.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.028+0000 7f713fe52640 1 -- 172.21.15.107:0/3409146034 wait complete. 2023-10-25T00:22:50.044 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2023-10-25T00:22:50.044 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:22:50.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:22:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.164+0000 7fbe8dbdf640 1 Processor -- start 2023-10-25T00:22:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.164+0000 7fbe8dbdf640 1 -- start start 2023-10-25T00:22:50.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe8dbdf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbe8814b600 0x7fbe88151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:50.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe8dbdf640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe8806af70 0x7fbe8806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:50.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe8dbdf640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe880c1860 con 0x7fbe8806a230 2023-10-25T00:22:50.168 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe8dbdf640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe88154d50 con 0x7fbe8806af70 2023-10-25T00:22:50.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe8dbdf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe881524f0 con 0x7fbe8814b600 2023-10-25T00:22:50.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe86ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe8806af70 0x7fbe8806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:50.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe86ffd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbe8806af70 0x7fbe8806c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45344/0 (socket says 172.21.15.107:45344) 2023-10-25T00:22:50.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe86ffd640 1 -- 172.21.15.107:0/1678057097 learned_addr learned my addr 172.21.15.107:0/1678057097 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:50.169 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe877fe640 1 --2- 172.21.15.107:0/1678057097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbe8814b600 0x7fbe88151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:50.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe86ffd640 1 -- 172.21.15.107:0/1678057097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbe8814b600 msgr2=0x7fbe88151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:50.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe86ffd640 1 --2- 172.21.15.107:0/1678057097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbe8814b600 0x7fbe88151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe86ffd640 1 -- 172.21.15.107:0/1678057097 >> v1:172.21.15.110:6789/0 conn(0x7fbe8806a230 legacy=0x7fbe881551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:50.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe86ffd640 1 -- 172.21.15.107:0/1678057097 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8814b9e0 con 0x7fbe8806af70 2023-10-25T00:22:50.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe86ffd640 1 --2- 172.21.15.107:0/1678057097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe8806af70 0x7fbe8806c4c0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009870 tx=0x7fbe7c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b9731c6583e41b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:50.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe867fc640 1 -- 172.21.15.107:0/1678057097 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c0ab680 con 0x7fbe8806af70 2023-10-25T00:22:50.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe867fc640 1 -- 172.21.15.107:0/1678057097 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7c0ab7e0 con 0x7fbe8806af70 2023-10-25T00:22:50.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.168+0000 7fbe867fc640 1 -- 172.21.15.107:0/1678057097 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c0abb70 con 0x7fbe8806af70 2023-10-25T00:22:50.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/1678057097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe8806af70 msgr2=0x7fbe8806c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:50.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 --2- 172.21.15.107:0/1678057097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe8806af70 0x7fbe8806c4c0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009870 tx=0x7fbe7c01c480 comp rx=0 tx=0).stop 2023-10-25T00:22:50.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/1678057097 shutdown_connections 2023-10-25T00:22:50.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 --2- 172.21.15.107:0/1678057097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe8806af70 0x7fbe8806c4c0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.173 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 --2- 172.21.15.107:0/1678057097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbe8814b600 0x7fbe88151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/1678057097 >> 172.21.15.107:0/1678057097 conn(0x7fbe880c2950 msgr2=0x7fbe880c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:50.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/1678057097 shutdown_connections 2023-10-25T00:22:50.174 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/1678057097 wait complete. 2023-10-25T00:22:50.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 Processor -- start 2023-10-25T00:22:50.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 -- start start 2023-10-25T00:22:50.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbe8806a230 0x7fbe881eb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:50.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe8806af70 0x7fbe881ebe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:50.184 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe8814b600 0x7fbe8806b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:50.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8813ee00 con 0x7fbe8806af70 2023-10-25T00:22:50.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe880c4da0 con 0x7fbe8814b600 2023-10-25T00:22:50.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe881ea990 con 0x7fbe8806a230 2023-10-25T00:22:50.185 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe86ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe8806af70 0x7fbe881ebe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:50.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe86ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe8806af70 0x7fbe881ebe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45362/0 (socket says 172.21.15.107:45362) 2023-10-25T00:22:50.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe86ffd640 1 -- 172.21.15.107:0/4142789061 learned_addr learned my addr 172.21.15.107:0/4142789061 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:50.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe87fff640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe8814b600 0x7fbe8806b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:50.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe877fe640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbe8806a230 0x7fbe881eb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:50.186 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe86ffd640 1 -- 172.21.15.107:0/4142789061 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbe8806a230 msgr2=0x7fbe881eb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe86ffd640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbe8806a230 0x7fbe881eb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe86ffd640 1 -- 172.21.15.107:0/4142789061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe8814b600 msgr2=0x7fbe8806b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe86ffd640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe8814b600 0x7fbe8806b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe86ffd640 1 -- 172.21.15.107:0/4142789061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7c0094d0 con 0x7fbe8806af70 2023-10-25T00:22:50.187 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe877fe640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbe8806a230 0x7fbe881eb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe87fff640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe8814b600 0x7fbe8806b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe86ffd640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe8806af70 0x7fbe881ebe00 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0ad040 tx=0x7fbe7c01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe6ffff640 1 -- 172.21.15.107:0/4142789061 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c0b4800 con 0x7fbe8806af70 2023-10-25T00:22:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe6ffff640 1 -- 172.21.15.107:0/4142789061 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7c0b4960 con 0x7fbe8806af70 2023-10-25T00:22:50.188 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/4142789061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe881eaf10 con 0x7fbe8806af70 2023-10-25T00:22:50.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe6ffff640 1 -- 172.21.15.107:0/4142789061 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c0b4d10 con 0x7fbe8806af70 2023-10-25T00:22:50.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.172+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/4142789061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe881ec340 con 0x7fbe8806af70 2023-10-25T00:22:50.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.176+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/4142789061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8813e8a0 con 0x7fbe8806af70 2023-10-25T00:22:50.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.176+0000 7fbe6ffff640 1 -- 172.21.15.107:0/4142789061 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbe7c0b3380 con 0x7fbe8806af70 2023-10-25T00:22:50.189 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.176+0000 7fbe6ffff640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbe74065e60 0x7fbe74068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:50.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.176+0000 7fbe6ffff640 1 -- 172.21.15.107:0/4142789061 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7fbe7c11ef50 con 0x7fbe8806af70 2023-10-25T00:22:50.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.176+0000 7fbe877fe640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbe74065e60 0x7fbe74068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:50.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.176+0000 7fbe877fe640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbe74065e60 0x7fbe74068320 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fbe780097a0 tx=0x7fbe78008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:50.190 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.180+0000 7fbe6ffff640 1 -- 172.21.15.107:0/4142789061 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbe7c0eb2b0 con 0x7fbe8806af70 2023-10-25T00:22:50.365 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.364+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/4142789061 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe880746e0 con 0x7fbe8806af70 2023-10-25T00:22:50.366 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.364+0000 7fbe6ffff640 1 -- 172.21.15.107:0/4142789061 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+5007 (secure 0 0 0) 0x7fbe7c0ee8d0 con 0x7fbe8806af70 2023-10-25T00:22:50.366 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:50.367 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:49.199320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:22:50.367 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 231 2023-10-25T00:22:50.368 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/4142789061 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbe74065e60 msgr2=0x7fbe74068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbe74065e60 0x7fbe74068320 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fbe780097a0 tx=0x7fbe78008040 comp rx=0 tx=0).stop 2023-10-25T00:22:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/4142789061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe8806af70 msgr2=0x7fbe881ebe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe8806af70 0x7fbe881ebe00 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0ad040 tx=0x7fbe7c01acc0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/4142789061 shutdown_connections 2023-10-25T00:22:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe8814b600 0x7fbe8806b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbe8806af70 0x7fbe881ebe00 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbe74065e60 0x7fbe74068320 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 --2- 172.21.15.107:0/4142789061 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbe8806a230 0x7fbe881eb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/4142789061 >> 172.21.15.107:0/4142789061 conn(0x7fbe880c2950 msgr2=0x7fbe880c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/4142789061 shutdown_connections 2023-10-25T00:22:50.371 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:50.368+0000 7fbe8dbdf640 1 -- 172.21.15.107:0/4142789061 wait complete. 2023-10-25T00:22:50.388 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 231, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 231, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_9054": { "addr": "172.21.15.107:6837/3527905219", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6836", "nonce": 3527905219, "type": "v2" }, { "addr": "172.21.15.107:6837", "nonce": 3527905219, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9054, "incarnation": 231, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:22:49.199320+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9054 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 224, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7483": { "addr": "172.21.15.110:6834/47497412", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 47497412, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 47497412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7483, "incarnation": 221, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:21:22.342534+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7483 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2437148", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2437148, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2437148, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 190, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6443, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:22:52.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:22:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa62d9f0640 1 Processor -- start 2023-10-25T00:22:52.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa62d9f0640 1 -- start start 2023-10-25T00:22:52.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa62d9f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62806a230 0x7fa6281551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:52.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa62d9f0640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6280c1860 con 0x7fa62806af70 2023-10-25T00:22:52.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa62d9f0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa628154d50 con 0x7fa62814b600 2023-10-25T00:22:52.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa62d9f0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62814b9e0 con 0x7fa62806a230 2023-10-25T00:22:52.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa6267fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fa62806af70 0x7fa62806c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:42304/0 (socket says 172.21.15.107:42304) 2023-10-25T00:22:52.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa6267fc640 1 -- 172.21.15.107:0/3599867592 learned_addr learned my addr 172.21.15.107:0/3599867592 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:52.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa6277fe640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62806a230 0x7fa6281551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:52.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861498583 0 0) 0x7fa6280c1860 con 0x7fa62806af70 2023-10-25T00:22:52.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa610005740 con 0x7fa62806af70 2023-10-25T00:22:52.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1807937719 0 0) 0x7fa610005740 con 0x7fa62806af70 2023-10-25T00:22:52.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62806a230 msgr2=0x7fa6281551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:52.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62806a230 0x7fa6281551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:52.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 >> v1:172.21.15.110:6789/0 conn(0x7fa62814b600 legacy=0x7fa628151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:52.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6280c1ab0 con 0x7fa62806af70 2023-10-25T00:22:52.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fa61c001680 con 0x7fa62806af70 2023-10-25T00:22:52.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 >> v1:172.21.15.107:6789/0 conn(0x7fa62806af70 legacy=0x7fa62806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:52.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa610005bc0 0x7fa610005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:52.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6100069e0 0x7fa610006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:52.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62806a230 0x7fa6100073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:52.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62814b9e0 con 0x7fa610005bc0 2023-10-25T00:22:52.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61000dcd0 con 0x7fa6100069e0 2023-10-25T00:22:52.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61000de90 con 0x7fa62806a230 2023-10-25T00:22:52.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa626ffd640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa610005bc0 0x7fa610005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:52.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa6277fe640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62806a230 0x7fa6100073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:52.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa6267fc640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6100069e0 0x7fa610006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:52.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa626ffd640 1 -- 172.21.15.107:0/3599867592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62806a230 msgr2=0x7fa6100073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:52.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa626ffd640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa62806a230 0x7fa6100073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:52.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa626ffd640 1 -- 172.21.15.107:0/3599867592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6100069e0 msgr2=0x7fa610006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:52.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa626ffd640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6100069e0 0x7fa610006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:52.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.519+0000 7fa626ffd640 1 -- 172.21.15.107:0/3599867592 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa610005740 con 0x7fa610005bc0 2023-10-25T00:22:52.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa626ffd640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa610005bc0 0x7fa610005fe0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fa618001780 tx=0x7fa61801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2509ba4caf63763 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:52.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6180052d0 con 0x7fa610005bc0 2023-10-25T00:22:52.531 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6180a7050 con 0x7fa610005bc0 2023-10-25T00:22:52.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa625ffb640 1 -- 172.21.15.107:0/3599867592 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618003600 con 0x7fa610005bc0 2023-10-25T00:22:52.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 -- 172.21.15.107:0/3599867592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa610005bc0 msgr2=0x7fa610005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:52.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 --2- 172.21.15.107:0/3599867592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa610005bc0 0x7fa610005fe0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fa618001780 tx=0x7fa61801c3d0 comp rx=0 tx=0).stop 2023-10-25T00:22:52.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa6277fe640 1 -- 172.21.15.107:0/3599867592 reap_dead start 2023-10-25T00:22:52.532 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 -- 172.21.15.107:0/3599867592 shutdown_connections 2023-10-25T00:22:52.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 -- 172.21.15.107:0/3599867592 >> 172.21.15.107:0/3599867592 conn(0x7fa6280c2950 msgr2=0x7fa6280c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:52.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 -- 172.21.15.107:0/3599867592 shutdown_connections 2023-10-25T00:22:52.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 -- 172.21.15.107:0/3599867592 wait complete. 2023-10-25T00:22:52.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 Processor -- start 2023-10-25T00:22:52.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 -- start start 2023-10-25T00:22:52.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa610005bc0 0x7fa6280a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:52.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6281462c0 0x7fa62814dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:52.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6280bfbe0 0x7fa62814e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:52.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6281484c0 con 0x7fa6281462c0 2023-10-25T00:22:52.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62814e640 con 0x7fa610005bc0 2023-10-25T00:22:52.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa62d9f0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6281ec790 con 0x7fa6280bfbe0 2023-10-25T00:22:52.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa6267fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6281462c0 0x7fa62814dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:52.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa6267fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6281462c0 0x7fa62814dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45388/0 (socket says 172.21.15.107:45388) 2023-10-25T00:22:52.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.523+0000 7fa6267fc640 1 -- 172.21.15.107:0/2250411183 learned_addr learned my addr 172.21.15.107:0/2250411183 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:52.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa626ffd640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa610005bc0 0x7fa6280a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:52.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa6277fe640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6280bfbe0 0x7fa62814e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:52.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa6267fc640 1 -- 172.21.15.107:0/2250411183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6280bfbe0 msgr2=0x7fa62814e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:52.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa6267fc640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6280bfbe0 0x7fa62814e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:52.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa6267fc640 1 -- 172.21.15.107:0/2250411183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa610005bc0 msgr2=0x7fa6280a9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:52.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa6267fc640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa610005bc0 0x7fa6280a9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:52.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa6267fc640 1 -- 172.21.15.107:0/2250411183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6281eca70 con 0x7fa6281462c0 2023-10-25T00:22:52.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa626ffd640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa610005bc0 0x7fa6280a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:52.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa6277fe640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6280bfbe0 0x7fa62814e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:52.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa6267fc640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6281462c0 0x7fa62814dbc0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00e780 tx=0x7fa61c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:52.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa60ffff640 1 -- 172.21.15.107:0/2250411183 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c00ee30 con 0x7fa6281462c0 2023-10-25T00:22:52.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa60ffff640 1 -- 172.21.15.107:0/2250411183 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61c00c8d0 con 0x7fa6281462c0 2023-10-25T00:22:52.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa62d9f0640 1 -- 172.21.15.107:0/2250411183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6281ecd50 con 0x7fa6281462c0 2023-10-25T00:22:52.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa60ffff640 1 -- 172.21.15.107:0/2250411183 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c00cc80 con 0x7fa6281462c0 2023-10-25T00:22:52.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa62d9f0640 1 -- 172.21.15.107:0/2250411183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6281ed310 con 0x7fa6281462c0 2023-10-25T00:22:52.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa62d9f0640 1 -- 172.21.15.107:0/2250411183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6281ed530 con 0x7fa6281462c0 2023-10-25T00:22:52.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa60ffff640 1 -- 172.21.15.107:0/2250411183 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fa61c01d460 con 0x7fa6281462c0 2023-10-25T00:22:52.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa60ffff640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa614065e60 0x7fa614068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:52.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa60ffff640 1 -- 172.21.15.107:0/2250411183 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7fa61c084960 con 0x7fa6281462c0 2023-10-25T00:22:52.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa626ffd640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa614065e60 0x7fa614068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:52.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.527+0000 7fa626ffd640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa614065e60 0x7fa614068320 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa618001750 tx=0x7fa6180011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:52.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.531+0000 7fa60ffff640 1 -- 172.21.15.107:0/2250411183 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fa61c050d50 con 0x7fa6281462c0 2023-10-25T00:22:52.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.719+0000 7fa62d9f0640 1 -- 172.21.15.107:0/2250411183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6280746e0 con 0x7fa6281462c0 2023-10-25T00:22:52.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.719+0000 7fa60ffff640 1 -- 172.21.15.107:0/2250411183 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+5008 (secure 0 0 0) 0x7fa61c054370 con 0x7fa6281462c0 2023-10-25T00:22:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:52.723 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:22:52.723 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 234 2023-10-25T00:22:52.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 -- 172.21.15.107:0/2250411183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa614065e60 msgr2=0x7fa614068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:52.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa614065e60 0x7fa614068320 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa618001750 tx=0x7fa6180011a0 comp rx=0 tx=0).stop 2023-10-25T00:22:52.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 -- 172.21.15.107:0/2250411183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6281462c0 msgr2=0x7fa62814dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:52.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6281462c0 0x7fa62814dbc0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00e780 tx=0x7fa61c00ec40 comp rx=0 tx=0).stop 2023-10-25T00:22:52.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 -- 172.21.15.107:0/2250411183 shutdown_connections 2023-10-25T00:22:52.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fa614065e60 0x7fa614068320 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:52.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa6280bfbe0 0x7fa62814e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:52.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6281462c0 0x7fa62814dbc0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:52.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 --2- 172.21.15.107:0/2250411183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa610005bc0 0x7fa6280a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:52.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 -- 172.21.15.107:0/2250411183 >> 172.21.15.107:0/2250411183 conn(0x7fa6280c2950 msgr2=0x7fa628071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:52.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 -- 172.21.15.107:0/2250411183 shutdown_connections 2023-10-25T00:22:52.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:52.723+0000 7fa62d9f0640 1 -- 172.21.15.107:0/2250411183 wait complete. 2023-10-25T00:22:52.740 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=9009 2023-10-25T00:22:52.741 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before reviving mds.c 2023-10-25T00:22:55.877 INFO:tasks.ceph.mds.a.smithi107.stderr:2023-10-25T00:22:55.875+0000 7feeb7930640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:22:57.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-25T00:22:58.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f5026b30640 1 Processor -- start 2023-10-25T00:22:58.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f5026b30640 1 -- start start 2023-10-25T00:22:58.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f5026b30640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f502006a230 0x7f50201551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:58.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f5026b30640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50200c1860 con 0x7f502006af70 2023-10-25T00:22:58.037 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f5026b30640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5020154d50 con 0x7f502014b600 2023-10-25T00:22:58.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f5026b30640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f502014b9e0 con 0x7f502006a230 2023-10-25T00:22:58.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f50250cd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f502006a230 0x7f50201551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:58.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f501ffff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f502006af70 0x7f502006c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56304/0 (socket says 172.21.15.107:56304) 2023-10-25T00:22:58.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f501ffff640 1 -- 172.21.15.107:0/3772820196 learned_addr learned my addr 172.21.15.107:0/3772820196 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:22:58.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f50250cd640 1 -- 172.21.15.107:0/3772820196 >> v1:172.21.15.110:6789/0 conn(0x7f502014b600 legacy=0x7f5020151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:22:58.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f50250cd640 1 -- 172.21.15.107:0/3772820196 >> v1:172.21.15.107:6789/0 conn(0x7f502006af70 legacy=0x7f502006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:58.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f50250cd640 1 -- 172.21.15.107:0/3772820196 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50200c1ab0 con 0x7f502006a230 2023-10-25T00:22:58.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.035+0000 7f501f7fe640 1 -- 172.21.15.107:0/3772820196 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031197400 0 0) 0x7f50200c1860 con 0x7f502006af70 2023-10-25T00:22:58.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f50250cd640 1 --2- 172.21.15.107:0/3772820196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f502006a230 0x7f50201551e0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f5018009d90 tx=0x7f501801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74a9f22eb045671 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:58.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f501f7fe640 1 -- 172.21.15.107:0/3772820196 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50180a5820 con 0x7f502006a230 2023-10-25T00:22:58.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f501f7fe640 1 -- 172.21.15.107:0/3772820196 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50180a5980 con 0x7f502006a230 2023-10-25T00:22:58.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f501f7fe640 1 -- 172.21.15.107:0/3772820196 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50180a5d10 con 0x7f502006a230 2023-10-25T00:22:58.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 -- 172.21.15.107:0/3772820196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f502006a230 msgr2=0x7f50201551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:58.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 --2- 172.21.15.107:0/3772820196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f502006a230 0x7f50201551e0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f5018009d90 tx=0x7f501801c850 comp rx=0 tx=0).stop 2023-10-25T00:22:58.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 -- 172.21.15.107:0/3772820196 shutdown_connections 2023-10-25T00:22:58.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 --2- 172.21.15.107:0/3772820196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f502006a230 0x7f50201551e0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:58.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 -- 172.21.15.107:0/3772820196 >> 172.21.15.107:0/3772820196 conn(0x7f50200c2950 msgr2=0x7f50200c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:58.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 -- 172.21.15.107:0/3772820196 shutdown_connections 2023-10-25T00:22:58.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 -- 172.21.15.107:0/3772820196 wait complete. 2023-10-25T00:22:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 Processor -- start 2023-10-25T00:22:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 -- start start 2023-10-25T00:22:58.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f502006a230 0x7f5020144990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f502006af70 0x7f50201edd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502014b600 0x7f50200be2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50200bf590 con 0x7f502006a230 2023-10-25T00:22:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50201477e0 con 0x7f502006af70 2023-10-25T00:22:58.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f5026b30640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50201ee660 con 0x7f502014b600 2023-10-25T00:22:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f50248cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f502006a230 0x7f5020144990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f50248cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f502006a230 0x7f5020144990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52892/0 (socket says 172.21.15.107:52892) 2023-10-25T00:22:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f50248cc640 1 -- 172.21.15.107:0/3331784402 learned_addr learned my addr 172.21.15.107:0/3331784402 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:22:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f501ffff640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f502006af70 0x7f50201edd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:58.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f50250cd640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502014b600 0x7f50200be2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f50248cc640 1 -- 172.21.15.107:0/3331784402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502014b600 msgr2=0x7f50200be2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f50248cc640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502014b600 0x7f50200be2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f50248cc640 1 -- 172.21.15.107:0/3331784402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f502006af70 msgr2=0x7f50201edd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f50248cc640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f502006af70 0x7f50201edd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:58.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f50248cc640 1 -- 172.21.15.107:0/3331784402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5018009a70 con 0x7f502006a230 2023-10-25T00:22:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.039+0000 7f50250cd640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502014b600 0x7f50200be2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:22:58.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f50248cc640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f502006a230 0x7f5020144990 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f501400b410 tx=0x7f501400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f501d7fa640 1 -- 172.21.15.107:0/3331784402 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014004020 con 0x7f502006a230 2023-10-25T00:22:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f501d7fa640 1 -- 172.21.15.107:0/3331784402 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5014004180 con 0x7f502006a230 2023-10-25T00:22:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f5026b30640 1 -- 172.21.15.107:0/3331784402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50201ee940 con 0x7f502006a230 2023-10-25T00:22:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f501d7fa640 1 -- 172.21.15.107:0/3331784402 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014011630 con 0x7f502006a230 2023-10-25T00:22:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f5026b30640 1 -- 172.21.15.107:0/3331784402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50201524e0 con 0x7f502006a230 2023-10-25T00:22:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f5026b30640 1 -- 172.21.15.107:0/3331784402 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50200746e0 con 0x7f502006a230 2023-10-25T00:22:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f501d7fa640 1 -- 172.21.15.107:0/3331784402 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5014011790 con 0x7f502006a230 2023-10-25T00:22:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f501d7fa640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5004065e90 0x7f5004068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:22:58.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f501ffff640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5004065e90 0x7f5004068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:22:58.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f501d7fa640 1 -- 172.21.15.107:0/3331784402 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f5014084760 con 0x7f502006a230 2023-10-25T00:22:58.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.043+0000 7f501ffff640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5004065e90 0x7f5004068350 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f502014bc60 tx=0x7f500c007b60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:22:58.053 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.047+0000 7f501d7fa640 1 -- 172.21.15.107:0/3331784402 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5014050b50 con 0x7f502006a230 2023-10-25T00:22:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.199+0000 7f5026b30640 1 -- 172.21.15.107:0/3331784402 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f502015f6e0 con 0x7f5004065e90 2023-10-25T00:22:58.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.239+0000 7f501d7fa640 1 -- 172.21.15.107:0/3331784402 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+546687 (secure 0 0 0) 0x7f500c0071f0 con 0x7f5004065e90 2023-10-25T00:22:58.243 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:22:58.252 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-25T00:22:58.254 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.243+0000 7f5026b30640 1 -- 172.21.15.107:0/3331784402 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5004065e90 msgr2=0x7f5004068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:58.255 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.243+0000 7f5026b30640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5004065e90 0x7f5004068350 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f502014bc60 tx=0x7f500c007b60 comp rx=0 tx=0).stop 2023-10-25T00:22:58.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.243+0000 7f5026b30640 1 -- 172.21.15.107:0/3331784402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f502006a230 msgr2=0x7f5020144990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:22:58.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.243+0000 7f5026b30640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f502006a230 0x7f5020144990 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f501400b410 tx=0x7f501400b8d0 comp rx=0 tx=0).stop 2023-10-25T00:22:58.256 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.243+0000 7f5026b30640 1 -- 172.21.15.107:0/3331784402 shutdown_connections 2023-10-25T00:22:58.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.243+0000 7f5026b30640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5004065e90 0x7f5004068350 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:58.258 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.243+0000 7f5026b30640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f502014b600 0x7f50200be2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.243+0000 7f5026b30640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f502006af70 0x7f50201edd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.243+0000 7f5026b30640 1 --2- 172.21.15.107:0/3331784402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f502006a230 0x7f5020144990 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:22:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.243+0000 7f5026b30640 1 -- 172.21.15.107:0/3331784402 >> 172.21.15.107:0/3331784402 conn(0x7f50200c2950 msgr2=0x7f50200c10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:22:58.259 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.247+0000 7f5026b30640 1 -- 172.21.15.107:0/3331784402 shutdown_connections 2023-10-25T00:22:58.260 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:22:58.247+0000 7f5026b30640 1 -- 172.21.15.107:0/3331784402 wait complete. 2023-10-25T00:22:58.267 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":970,"stamp":"2023-10-25T00:22:58.020755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40142898,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3975,"num_read_kb":477474,"num_write":3153461,"num_write_kb":38880074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":583965,"ondisk_log_size":583965,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7826988,"kb_used_data":86188,"kb_used_omap":0,"kb_used_meta":7740736,"kb_avail":747147732,"statfs":{"total":773094113280,"available":765079277568,"internally_reserved":0,"allocated":88256512,"data_stored":84624386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7926513664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20656,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":69,"num_read_kb":19554,"num_write":15,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004653"},"pg_stats":[{"pgid":"6.1a","version":"72'3995","reported_seq":4068,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060892+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-25T00:22:36.060892+0000","last_peered":"2023-10-25T00:22:36.060892+0000","last_clean":"2023-10-25T00:22:36.060892+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-25T00:22:36.060892+0000","last_undegraded":"2023-10-25T00:22:36.060892+0000","last_fullsized":"2023-10-25T00:22:36.060892+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21480,"num_write_kb":282788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"72'3","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965652+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-25T00:22:33.965652+0000","last_peered":"2023-10-25T00:22:33.965652+0000","last_clean":"2023-10-25T00:22:33.965652+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-25T00:22:33.965652+0000","last_undegraded":"2023-10-25T00:22:33.965652+0000","last_fullsized":"2023-10-25T00:22:33.965652+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"68'864","reported_seq":993,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.255909+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-25T00:22:49.255909+0000","last_peered":"2023-10-25T00:22:49.255909+0000","last_clean":"2023-10-25T00:22:49.255909+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-25T00:22:49.255909+0000","last_undegraded":"2023-10-25T00:22:49.255909+0000","last_fullsized":"2023-10-25T00:22:49.255909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":22541,"num_write":841,"num_write_kb":4586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'5616","reported_seq":5690,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060413+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-25T00:22:36.060413+0000","last_peered":"2023-10-25T00:22:36.060413+0000","last_clean":"2023-10-25T00:22:36.060413+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-25T00:22:36.060413+0000","last_undegraded":"2023-10-25T00:22:36.060413+0000","last_fullsized":"2023-10-25T00:22:36.060413+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29775,"num_write_kb":348818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"72'3735","reported_seq":3806,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534249+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-25T00:22:35.534249+0000","last_peered":"2023-10-25T00:22:35.534249+0000","last_clean":"2023-10-25T00:22:35.534249+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-25T00:22:35.534249+0000","last_undegraded":"2023-10-25T00:22:35.534249+0000","last_fullsized":"2023-10-25T00:22:35.534249+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20090,"num_write_kb":264783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"72'2","reported_seq":72,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060934+0000","last_change":"2023-10-24T23:53:11.444089+0000","last_active":"2023-10-25T00:22:36.060934+0000","last_peered":"2023-10-25T00:22:36.060934+0000","last_clean":"2023-10-25T00:22:36.060934+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-25T00:22:36.060934+0000","last_undegraded":"2023-10-25T00:22:36.060934+0000","last_fullsized":"2023-10-25T00:22:36.060934+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"73'28","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.235955+0000","last_change":"2023-10-24T23:53:02.332632+0000","last_active":"2023-10-25T00:22:49.235955+0000","last_peered":"2023-10-25T00:22:49.235955+0000","last_clean":"2023-10-25T00:22:49.235955+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-25T00:22:49.235955+0000","last_undegraded":"2023-10-25T00:22:49.235955+0000","last_fullsized":"2023-10-25T00:22:49.235955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'5447","reported_seq":5521,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.533893+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-25T00:22:35.533893+0000","last_peered":"2023-10-25T00:22:35.533893+0000","last_clean":"2023-10-25T00:22:35.533893+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-25T00:22:35.533893+0000","last_undegraded":"2023-10-25T00:22:35.533893+0000","last_fullsized":"2023-10-25T00:22:35.533893+0000","mapping_epoch":21,"log_start":"61'100","ondisk_log_start":"61'100","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28063,"num_write_kb":350200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"72'4747","reported_seq":4820,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517509+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-25T00:22:33.517509+0000","last_peered":"2023-10-25T00:22:33.517509+0000","last_clean":"2023-10-25T00:22:33.517509+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-25T00:22:33.517509+0000","last_undegraded":"2023-10-25T00:22:33.517509+0000","last_fullsized":"2023-10-25T00:22:33.517509+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":25766,"num_write_kb":328702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"72'754","reported_seq":861,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.433948+0000","last_change":"2023-10-25T00:22:37.433948+0000","last_active":"2023-10-25T00:22:37.433948+0000","last_peered":"2023-10-25T00:22:37.433948+0000","last_clean":"2023-10-25T00:22:37.433948+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-25T00:22:37.433948+0000","last_undegraded":"2023-10-25T00:22:37.433948+0000","last_fullsized":"2023-10-25T00:22:37.433948+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'754","last_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_deep_scrub":"72'754","last_deep_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.433382+0000","objects_scrubbed":1,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:43.601946+0000","scrub_duration":0.10905978199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":77282,"num_write":740,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"73'50","reported_seq":169,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.424235+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-25T00:22:52.424235+0000","last_peered":"2023-10-25T00:22:52.424235+0000","last_clean":"2023-10-25T00:22:52.424235+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-25T00:22:52.424235+0000","last_undegraded":"2023-10-25T00:22:52.424235+0000","last_fullsized":"2023-10-25T00:22:52.424235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'4792","reported_seq":4877,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:44.881494+0000","last_change":"2023-10-25T00:22:44.881494+0000","last_active":"2023-10-25T00:22:44.881494+0000","last_peered":"2023-10-25T00:22:44.881494+0000","last_clean":"2023-10-25T00:22:44.881494+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-25T00:22:44.881494+0000","last_undegraded":"2023-10-25T00:22:44.881494+0000","last_fullsized":"2023-10-25T00:22:44.881494+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4792","last_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_deep_scrub":"68'4792","last_deep_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.881344+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:23:57.562121+0000","scrub_duration":0.193670646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24997,"num_write_kb":313272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"72'4739","reported_seq":4813,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523335+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-25T00:22:36.523335+0000","last_peered":"2023-10-25T00:22:36.523335+0000","last_clean":"2023-10-25T00:22:36.523335+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-25T00:22:36.523335+0000","last_undegraded":"2023-10-25T00:22:36.523335+0000","last_fullsized":"2023-10-25T00:22:36.523335+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":22,"num_write":25639,"num_write_kb":332364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"72'357","reported_seq":429,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.208814+0000","last_change":"2023-10-25T00:22:55.998388+0000","last_active":"2023-10-25T00:22:49.208814+0000","last_peered":"2023-10-25T00:22:49.208814+0000","last_clean":"2023-10-25T00:22:49.208814+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-25T00:22:49.208814+0000","last_undegraded":"2023-10-25T00:22:49.208814+0000","last_fullsized":"2023-10-25T00:22:49.208814+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2627766,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":227,"num_write":329,"num_write_kb":2759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"68'5","reported_seq":126,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.426569+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-25T00:22:52.426569+0000","last_peered":"2023-10-25T00:22:52.426569+0000","last_clean":"2023-10-25T00:22:52.426569+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-25T00:22:52.426569+0000","last_undegraded":"2023-10-25T00:22:52.426569+0000","last_fullsized":"2023-10-25T00:22:52.426569+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'3731","reported_seq":3814,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.622543+0000","last_change":"2023-10-25T00:22:37.622543+0000","last_active":"2023-10-25T00:22:37.622543+0000","last_peered":"2023-10-25T00:22:37.622543+0000","last_clean":"2023-10-25T00:22:37.622543+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-25T00:22:37.622543+0000","last_undegraded":"2023-10-25T00:22:37.622543+0000","last_fullsized":"2023-10-25T00:22:37.622543+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3731","last_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_deep_scrub":"68'3731","last_deep_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.622304+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:51:52.171234+0000","scrub_duration":0.090187284000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19492,"num_write_kb":250091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"72'3799","reported_seq":3871,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060885+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-25T00:22:36.060885+0000","last_peered":"2023-10-25T00:22:36.060885+0000","last_clean":"2023-10-25T00:22:36.060885+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-25T00:22:36.060885+0000","last_undegraded":"2023-10-25T00:22:36.060885+0000","last_fullsized":"2023-10-25T00:22:36.060885+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21581,"num_write_kb":262877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"73'46","reported_seq":170,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.664107+0000","last_change":"2023-10-25T00:22:52.664257+0000","last_active":"2023-10-25T00:22:52.664107+0000","last_peered":"2023-10-25T00:22:52.664107+0000","last_clean":"2023-10-25T00:22:52.664107+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-25T00:22:52.664107+0000","last_undegraded":"2023-10-25T00:22:52.664107+0000","last_fullsized":"2023-10-25T00:22:52.664107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'46","last_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_deep_scrub":"73'46","last_deep_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.664064+0000","objects_scrubbed":1,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:44:09.108291+0000","scrub_duration":0.106079285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"68'714","reported_seq":816,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.227415+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-25T00:22:49.227415+0000","last_peered":"2023-10-25T00:22:49.227415+0000","last_clean":"2023-10-25T00:22:49.227415+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-25T00:22:49.227415+0000","last_undegraded":"2023-10-25T00:22:49.227415+0000","last_fullsized":"2023-10-25T00:22:49.227415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":90428,"num_write":703,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'4008","reported_seq":4084,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427307+0000","last_change":"2023-10-24T23:53:02.720197+0000","last_active":"2023-10-25T00:22:34.427307+0000","last_peered":"2023-10-25T00:22:34.427307+0000","last_clean":"2023-10-25T00:22:34.427307+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-25T00:22:34.427307+0000","last_undegraded":"2023-10-25T00:22:34.427307+0000","last_fullsized":"2023-10-25T00:22:34.427307+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20791,"num_write_kb":267770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"72'5469","reported_seq":5557,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:57.381099+0000","last_change":"2023-10-25T00:22:57.379534+0000","last_active":"2023-10-25T00:22:57.381099+0000","last_peered":"2023-10-25T00:22:57.381099+0000","last_clean":"2023-10-25T00:22:57.381099+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-25T00:22:57.381099+0000","last_undegraded":"2023-10-25T00:22:57.381099+0000","last_fullsized":"2023-10-25T00:22:57.381099+0000","mapping_epoch":25,"log_start":"64'200","ondisk_log_start":"64'200","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":40,"num_write":30646,"num_write_kb":401481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"72'5","reported_seq":118,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060966+0000","last_change":"2023-10-24T23:53:11.442035+0000","last_active":"2023-10-25T00:22:36.060966+0000","last_peered":"2023-10-25T00:22:36.060966+0000","last_clean":"2023-10-25T00:22:36.060966+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-25T00:22:36.060966+0000","last_undegraded":"2023-10-25T00:22:36.060966+0000","last_fullsized":"2023-10-25T00:22:36.060966+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"73'546","reported_seq":629,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.821701+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-25T00:22:55.821701+0000","last_peered":"2023-10-25T00:22:55.821701+0000","last_clean":"2023-10-25T00:22:55.821701+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-25T00:22:55.821701+0000","last_undegraded":"2023-10-25T00:22:55.821701+0000","last_fullsized":"2023-10-25T00:22:55.821701+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3198201,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10628,"num_write":517,"num_write_kb":3434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'6069","reported_seq":6155,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:46.608504+0000","last_change":"2023-10-25T00:22:46.608504+0000","last_active":"2023-10-25T00:22:46.608504+0000","last_peered":"2023-10-25T00:22:46.608504+0000","last_clean":"2023-10-25T00:22:46.608504+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-25T00:22:46.608504+0000","last_undegraded":"2023-10-25T00:22:46.608504+0000","last_fullsized":"2023-10-25T00:22:46.608504+0000","mapping_epoch":21,"log_start":"62'800","ondisk_log_start":"62'800","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6069","last_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_deep_scrub":"68'6069","last_deep_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.608198+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:37.101860+0000","scrub_duration":0.19917779799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":33187,"num_write_kb":387130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'3606","reported_seq":3682,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437504+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-25T00:22:37.437504+0000","last_peered":"2023-10-25T00:22:37.437504+0000","last_clean":"2023-10-25T00:22:37.437504+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-25T00:22:37.437504+0000","last_undegraded":"2023-10-25T00:22:37.437504+0000","last_fullsized":"2023-10-25T00:22:37.437504+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19493,"num_write_kb":254342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'1061","reported_seq":1184,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978699+0000","last_change":"2023-10-24T23:53:11.448032+0000","last_active":"2023-10-25T00:22:36.978699+0000","last_peered":"2023-10-25T00:22:36.978699+0000","last_clean":"2023-10-25T00:22:36.978699+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-25T00:22:36.978699+0000","last_undegraded":"2023-10-25T00:22:36.978699+0000","last_fullsized":"2023-10-25T00:22:36.978699+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1061,"log_dups_size":0,"ondisk_log_size":1061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":9402,"num_write":1035,"num_write_kb":4717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'3","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965433+0000","last_change":"2023-10-24T23:53:02.337037+0000","last_active":"2023-10-25T00:22:33.965433+0000","last_peered":"2023-10-25T00:22:33.965433+0000","last_clean":"2023-10-25T00:22:33.965433+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-25T00:22:33.965433+0000","last_undegraded":"2023-10-25T00:22:33.965433+0000","last_fullsized":"2023-10-25T00:22:33.965433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'5139","reported_seq":5225,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:54.030808+0000","last_change":"2023-10-25T00:22:54.030976+0000","last_active":"2023-10-25T00:22:54.030808+0000","last_peered":"2023-10-25T00:22:54.030808+0000","last_clean":"2023-10-25T00:22:54.030808+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-25T00:22:54.030808+0000","last_undegraded":"2023-10-25T00:22:54.030808+0000","last_fullsized":"2023-10-25T00:22:54.030808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5139","last_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_deep_scrub":"68'5139","last_deep_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.030782+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:30.182861+0000","scrub_duration":0.103857617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27772,"num_write_kb":328469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"72'4843","reported_seq":4915,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965801+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-25T00:22:33.965801+0000","last_peered":"2023-10-25T00:22:33.965801+0000","last_clean":"2023-10-25T00:22:33.965801+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-25T00:22:33.965801+0000","last_undegraded":"2023-10-25T00:22:33.965801+0000","last_fullsized":"2023-10-25T00:22:33.965801+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":27290,"num_write_kb":330906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"72'27","reported_seq":110,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.986728+0000","last_change":"2023-10-25T00:22:55.987117+0000","last_active":"2023-10-25T00:22:55.986728+0000","last_peered":"2023-10-25T00:22:55.986728+0000","last_clean":"2023-10-25T00:22:55.986728+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-25T00:22:55.986728+0000","last_undegraded":"2023-10-25T00:22:55.986728+0000","last_fullsized":"2023-10-25T00:22:55.986728+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'27","last_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_deep_scrub":"72'27","last_deep_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.986674+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:33.089164+0000","scrub_duration":0.100730242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'2","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.533782+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-25T00:22:35.533782+0000","last_peered":"2023-10-25T00:22:35.533782+0000","last_clean":"2023-10-25T00:22:35.533782+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-25T00:22:35.533782+0000","last_undegraded":"2023-10-25T00:22:35.533782+0000","last_fullsized":"2023-10-25T00:22:35.533782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'5814","reported_seq":5901,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.320438+0000","last_change":"2023-10-25T00:22:52.320696+0000","last_active":"2023-10-25T00:22:52.320438+0000","last_peered":"2023-10-25T00:22:52.320438+0000","last_clean":"2023-10-25T00:22:52.320438+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-25T00:22:52.320438+0000","last_undegraded":"2023-10-25T00:22:52.320438+0000","last_fullsized":"2023-10-25T00:22:52.320438+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5814","last_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_deep_scrub":"68'5814","last_deep_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.320378+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:26.511615+0000","scrub_duration":0.033495337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31996,"num_write_kb":382397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"72'3827","reported_seq":3900,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437501+0000","last_change":"2023-10-24T23:53:11.849056+0000","last_active":"2023-10-25T00:22:37.437501+0000","last_peered":"2023-10-25T00:22:37.437501+0000","last_clean":"2023-10-25T00:22:37.437501+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-25T00:22:37.437501+0000","last_undegraded":"2023-10-25T00:22:37.437501+0000","last_fullsized":"2023-10-25T00:22:37.437501+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":21199,"num_write_kb":277730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"68'4820","reported_seq":4906,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.429763+0000","last_change":"2023-10-25T00:22:36.429763+0000","last_active":"2023-10-25T00:22:36.429763+0000","last_peered":"2023-10-25T00:22:36.429763+0000","last_clean":"2023-10-25T00:22:36.429763+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-25T00:22:36.429763+0000","last_undegraded":"2023-10-25T00:22:36.429763+0000","last_fullsized":"2023-10-25T00:22:36.429763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4820","last_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_deep_scrub":"68'4820","last_deep_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_clean_scrub_stamp":"2023-10-25T00:22:36.429607+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:35.883183+0000","scrub_duration":0.10613655600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25163,"num_write_kb":285293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"72'5","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.516883+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-25T00:22:33.516883+0000","last_peered":"2023-10-25T00:22:33.516883+0000","last_clean":"2023-10-25T00:22:33.516883+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-25T00:22:33.516883+0000","last_undegraded":"2023-10-25T00:22:33.516883+0000","last_fullsized":"2023-10-25T00:22:33.516883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"72'5","reported_seq":86,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:40.197550+0000","last_change":"2023-10-25T00:22:40.197550+0000","last_active":"2023-10-25T00:22:40.197550+0000","last_peered":"2023-10-25T00:22:40.197550+0000","last_clean":"2023-10-25T00:22:40.197550+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-25T00:22:40.197550+0000","last_undegraded":"2023-10-25T00:22:40.197550+0000","last_fullsized":"2023-10-25T00:22:40.197550+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.197213+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:07:28.509914+0000","scrub_duration":0.0047704979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"72'4957","reported_seq":5044,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:53.667769+0000","last_change":"2023-10-25T00:22:53.667906+0000","last_active":"2023-10-25T00:22:53.667769+0000","last_peered":"2023-10-25T00:22:53.667769+0000","last_clean":"2023-10-25T00:22:53.667769+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-25T00:22:53.667769+0000","last_undegraded":"2023-10-25T00:22:53.667769+0000","last_fullsized":"2023-10-25T00:22:53.667769+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4957","last_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_deep_scrub":"72'4957","last_deep_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.667748+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:58.307897+0000","scrub_duration":0.30812842099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28093,"num_write_kb":358545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'4782","reported_seq":4873,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.562198+0000","last_change":"2023-10-25T00:22:42.562198+0000","last_active":"2023-10-25T00:22:42.562198+0000","last_peered":"2023-10-25T00:22:42.562198+0000","last_clean":"2023-10-25T00:22:42.562198+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-25T00:22:42.562198+0000","last_undegraded":"2023-10-25T00:22:42.562198+0000","last_fullsized":"2023-10-25T00:22:42.562198+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4782","last_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_deep_scrub":"68'4782","last_deep_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.562057+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:39.415960+0000","scrub_duration":0.116720554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26432,"num_write_kb":298639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"68'3","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522666+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-25T00:22:36.522666+0000","last_peered":"2023-10-25T00:22:36.522666+0000","last_clean":"2023-10-25T00:22:36.522666+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-25T00:22:36.522666+0000","last_undegraded":"2023-10-25T00:22:36.522666+0000","last_fullsized":"2023-10-25T00:22:36.522666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"72'18","reported_seq":214,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:46.074083+0000","last_change":"2023-10-25T00:22:46.074083+0000","last_active":"2023-10-25T00:22:46.074083+0000","last_peered":"2023-10-25T00:22:46.074083+0000","last_clean":"2023-10-25T00:22:46.074083+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-25T00:22:46.074083+0000","last_undegraded":"2023-10-25T00:22:46.074083+0000","last_fullsized":"2023-10-25T00:22:46.074083+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_deep_scrub":"72'18","last_deep_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.073758+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:08:25.787090+0000","scrub_duration":0.012558368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":345,"num_read_kb":345,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"72'2954","reported_seq":3025,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517454+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-25T00:22:33.517454+0000","last_peered":"2023-10-25T00:22:33.517454+0000","last_clean":"2023-10-25T00:22:33.517454+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-25T00:22:33.517454+0000","last_undegraded":"2023-10-25T00:22:33.517454+0000","last_fullsized":"2023-10-25T00:22:33.517454+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15961,"num_write_kb":219669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"72'36","reported_seq":156,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.225469+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-25T00:22:49.225469+0000","last_peered":"2023-10-25T00:22:49.225469+0000","last_clean":"2023-10-25T00:22:49.225469+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-25T00:22:49.225469+0000","last_undegraded":"2023-10-25T00:22:49.225469+0000","last_fullsized":"2023-10-25T00:22:49.225469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'4690","reported_seq":4766,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965581+0000","last_change":"2023-10-24T23:53:02.716337+0000","last_active":"2023-10-25T00:22:33.965581+0000","last_peered":"2023-10-25T00:22:33.965581+0000","last_clean":"2023-10-25T00:22:33.965581+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-25T00:22:33.965581+0000","last_undegraded":"2023-10-25T00:22:33.965581+0000","last_fullsized":"2023-10-25T00:22:33.965581+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25172,"num_write_kb":305786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"72'10","reported_seq":135,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:40.512549+0000","last_change":"2023-10-25T00:22:40.512549+0000","last_active":"2023-10-25T00:22:40.512549+0000","last_peered":"2023-10-25T00:22:40.512549+0000","last_clean":"2023-10-25T00:22:40.512549+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-25T00:22:40.512549+0000","last_undegraded":"2023-10-25T00:22:40.512549+0000","last_fullsized":"2023-10-25T00:22:40.512549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.512409+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:07:23.705506+0000","scrub_duration":0.091817919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"72'4200","reported_seq":4279,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534173+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-25T00:22:35.534173+0000","last_peered":"2023-10-25T00:22:35.534173+0000","last_clean":"2023-10-25T00:22:35.534173+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-25T00:22:35.534173+0000","last_undegraded":"2023-10-25T00:22:35.534173+0000","last_fullsized":"2023-10-25T00:22:35.534173+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":24169,"num_write_kb":293796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"73'390","reported_seq":580,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.648256+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-25T00:22:52.648256+0000","last_peered":"2023-10-25T00:22:52.648256+0000","last_clean":"2023-10-25T00:22:52.648256+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-25T00:22:52.648256+0000","last_undegraded":"2023-10-25T00:22:52.648256+0000","last_fullsized":"2023-10-25T00:22:52.648256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":210,"num_read_kb":94395,"num_write":387,"num_write_kb":4304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'5730","reported_seq":5805,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060329+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-25T00:22:36.060329+0000","last_peered":"2023-10-25T00:22:36.060329+0000","last_clean":"2023-10-25T00:22:36.060329+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-25T00:22:36.060329+0000","last_undegraded":"2023-10-25T00:22:36.060329+0000","last_fullsized":"2023-10-25T00:22:36.060329+0000","mapping_epoch":21,"log_start":"61'500","ondisk_log_start":"61'500","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":31652,"num_write_kb":356481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"72'3","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965700+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-25T00:22:33.965700+0000","last_peered":"2023-10-25T00:22:33.965700+0000","last_clean":"2023-10-25T00:22:33.965700+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-25T00:22:33.965700+0000","last_undegraded":"2023-10-25T00:22:33.965700+0000","last_fullsized":"2023-10-25T00:22:33.965700+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"72'4168","reported_seq":4242,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437378+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-25T00:22:37.437378+0000","last_peered":"2023-10-25T00:22:37.437378+0000","last_clean":"2023-10-25T00:22:37.437378+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-25T00:22:37.437378+0000","last_undegraded":"2023-10-25T00:22:37.437378+0000","last_fullsized":"2023-10-25T00:22:37.437378+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23458,"num_write_kb":294850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"68'4339","reported_seq":4415,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060019+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-25T00:22:36.060019+0000","last_peered":"2023-10-25T00:22:36.060019+0000","last_clean":"2023-10-25T00:22:36.060019+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-25T00:22:36.060019+0000","last_undegraded":"2023-10-25T00:22:36.060019+0000","last_fullsized":"2023-10-25T00:22:36.060019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23267,"num_write_kb":265268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":89,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.199286+0000","last_change":"2023-10-25T00:22:42.199286+0000","last_active":"2023-10-25T00:22:42.199286+0000","last_peered":"2023-10-25T00:22:42.199286+0000","last_clean":"2023-10-25T00:22:42.199286+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-25T00:22:42.199286+0000","last_undegraded":"2023-10-25T00:22:42.199286+0000","last_fullsized":"2023-10-25T00:22:42.199286+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.198947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:11:17.807377+0000","scrub_duration":0.0041955760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"73'231","reported_seq":330,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.233867+0000","last_change":"2023-10-24T23:53:02.339180+0000","last_active":"2023-10-25T00:22:49.233867+0000","last_peered":"2023-10-25T00:22:49.233867+0000","last_clean":"2023-10-25T00:22:49.233867+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-25T00:22:49.233867+0000","last_undegraded":"2023-10-25T00:22:49.233867+0000","last_fullsized":"2023-10-25T00:22:49.233867+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":63,"num_write":590,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"72'11","reported_seq":94,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:38.409520+0000","last_change":"2023-10-25T00:22:38.409520+0000","last_active":"2023-10-25T00:22:38.409520+0000","last_peered":"2023-10-25T00:22:38.409520+0000","last_clean":"2023-10-25T00:22:38.409520+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-25T00:22:38.409520+0000","last_undegraded":"2023-10-25T00:22:38.409520+0000","last_fullsized":"2023-10-25T00:22:38.409520+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_deep_scrub":"72'11","last_deep_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.409235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:21:24.708709+0000","scrub_duration":0.003592329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"72'3993","reported_seq":4069,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060792+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-25T00:22:36.060792+0000","last_peered":"2023-10-25T00:22:36.060792+0000","last_clean":"2023-10-25T00:22:36.060792+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-25T00:22:36.060792+0000","last_undegraded":"2023-10-25T00:22:36.060792+0000","last_fullsized":"2023-10-25T00:22:36.060792+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21723,"num_write_kb":283850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'4270","reported_seq":4346,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.533905+0000","last_change":"2023-10-24T23:53:02.714081+0000","last_active":"2023-10-25T00:22:35.533905+0000","last_peered":"2023-10-25T00:22:35.533905+0000","last_clean":"2023-10-25T00:22:35.533905+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-25T00:22:35.533905+0000","last_undegraded":"2023-10-25T00:22:35.533905+0000","last_fullsized":"2023-10-25T00:22:35.533905+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23049,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965402+0000","last_change":"2023-10-24T23:52:58.164014+0000","last_active":"2023-10-25T00:22:33.965402+0000","last_peered":"2023-10-25T00:22:33.965402+0000","last_clean":"2023-10-25T00:22:33.965402+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-25T00:22:33.965402+0000","last_undegraded":"2023-10-25T00:22:33.965402+0000","last_fullsized":"2023-10-25T00:22:33.965402+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'9","reported_seq":132,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.438809+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-25T00:22:52.438809+0000","last_peered":"2023-10-25T00:22:52.438809+0000","last_clean":"2023-10-25T00:22:52.438809+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-25T00:22:52.438809+0000","last_undegraded":"2023-10-25T00:22:52.438809+0000","last_fullsized":"2023-10-25T00:22:52.438809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":8,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"72'64","reported_seq":216,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060681+0000","last_change":"2023-10-24T23:53:11.441983+0000","last_active":"2023-10-25T00:22:36.060681+0000","last_peered":"2023-10-25T00:22:36.060681+0000","last_clean":"2023-10-25T00:22:36.060681+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-25T00:22:36.060681+0000","last_undegraded":"2023-10-25T00:22:36.060681+0000","last_fullsized":"2023-10-25T00:22:36.060681+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":123,"num_write":59,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"72'3769","reported_seq":3845,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965701+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-25T00:22:33.965701+0000","last_peered":"2023-10-25T00:22:33.965701+0000","last_clean":"2023-10-25T00:22:33.965701+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-25T00:22:33.965701+0000","last_undegraded":"2023-10-25T00:22:33.965701+0000","last_fullsized":"2023-10-25T00:22:33.965701+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20807,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'5752","reported_seq":5839,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.937946+0000","last_change":"2023-10-25T00:22:33.937946+0000","last_active":"2023-10-25T00:22:33.937946+0000","last_peered":"2023-10-25T00:22:33.937946+0000","last_clean":"2023-10-25T00:22:33.937946+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-25T00:22:33.937946+0000","last_undegraded":"2023-10-25T00:22:33.937946+0000","last_fullsized":"2023-10-25T00:22:33.937946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5752","last_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_deep_scrub":"68'5752","last_deep_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_clean_scrub_stamp":"2023-10-25T00:22:33.937564+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:43:34.779084+0000","scrub_duration":0.080678784000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31231,"num_write_kb":354718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.436957+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-25T00:22:37.436957+0000","last_peered":"2023-10-25T00:22:37.436957+0000","last_clean":"2023-10-25T00:22:37.436957+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-25T00:22:37.436957+0000","last_undegraded":"2023-10-25T00:22:37.436957+0000","last_fullsized":"2023-10-25T00:22:37.436957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'2","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965341+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-25T00:22:33.965341+0000","last_peered":"2023-10-25T00:22:33.965341+0000","last_clean":"2023-10-25T00:22:33.965341+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-25T00:22:33.965341+0000","last_undegraded":"2023-10-25T00:22:33.965341+0000","last_fullsized":"2023-10-25T00:22:33.965341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"72'4","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978692+0000","last_change":"2023-10-24T23:53:11.426287+0000","last_active":"2023-10-25T00:22:36.978692+0000","last_peered":"2023-10-25T00:22:36.978692+0000","last_clean":"2023-10-25T00:22:36.978692+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-25T00:22:36.978692+0000","last_undegraded":"2023-10-25T00:22:36.978692+0000","last_fullsized":"2023-10-25T00:22:36.978692+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"72'3939","reported_seq":4010,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517360+0000","last_change":"2023-10-24T23:53:11.811139+0000","last_active":"2023-10-25T00:22:33.517360+0000","last_peered":"2023-10-25T00:22:33.517360+0000","last_clean":"2023-10-25T00:22:33.517360+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-25T00:22:33.517360+0000","last_undegraded":"2023-10-25T00:22:33.517360+0000","last_fullsized":"2023-10-25T00:22:33.517360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22393,"num_write_kb":262004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'4305","reported_seq":4381,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965378+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-25T00:22:33.965378+0000","last_peered":"2023-10-25T00:22:33.965378+0000","last_clean":"2023-10-25T00:22:33.965378+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-25T00:22:33.965378+0000","last_undegraded":"2023-10-25T00:22:33.965378+0000","last_fullsized":"2023-10-25T00:22:33.965378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22696,"num_write_kb":287574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965279+0000","last_change":"2023-10-24T23:52:58.592524+0000","last_active":"2023-10-25T00:22:33.965279+0000","last_peered":"2023-10-25T00:22:33.965279+0000","last_clean":"2023-10-25T00:22:33.965279+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-25T00:22:33.965279+0000","last_undegraded":"2023-10-25T00:22:33.965279+0000","last_fullsized":"2023-10-25T00:22:33.965279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522611+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-25T00:22:36.522611+0000","last_peered":"2023-10-25T00:22:36.522611+0000","last_clean":"2023-10-25T00:22:36.522611+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-25T00:22:36.522611+0000","last_undegraded":"2023-10-25T00:22:36.522611+0000","last_fullsized":"2023-10-25T00:22:36.522611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"72'3","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.952645+0000","last_change":"2023-10-25T00:22:52.952897+0000","last_active":"2023-10-25T00:22:52.952645+0000","last_peered":"2023-10-25T00:22:52.952645+0000","last_clean":"2023-10-25T00:22:52.952645+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-25T00:22:52.952645+0000","last_undegraded":"2023-10-25T00:22:52.952645+0000","last_fullsized":"2023-10-25T00:22:52.952645+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.952595+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:14:20.890322+0000","scrub_duration":0.0044650949999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"72'5003","reported_seq":5080,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534073+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-25T00:22:35.534073+0000","last_peered":"2023-10-25T00:22:35.534073+0000","last_clean":"2023-10-25T00:22:35.534073+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-25T00:22:35.534073+0000","last_undegraded":"2023-10-25T00:22:35.534073+0000","last_fullsized":"2023-10-25T00:22:35.534073+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":13,"num_write":27120,"num_write_kb":350355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'5635","reported_seq":5721,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:44.547139+0000","last_change":"2023-10-25T00:22:44.547139+0000","last_active":"2023-10-25T00:22:44.547139+0000","last_peered":"2023-10-25T00:22:44.547139+0000","last_clean":"2023-10-25T00:22:44.547139+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-25T00:22:44.547139+0000","last_undegraded":"2023-10-25T00:22:44.547139+0000","last_fullsized":"2023-10-25T00:22:44.547139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5635","last_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_deep_scrub":"68'5635","last_deep_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.546984+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:51:54.474684+0000","scrub_duration":0.15865177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30556,"num_write_kb":356140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"68'3","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060015+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-25T00:22:36.060015+0000","last_peered":"2023-10-25T00:22:36.060015+0000","last_clean":"2023-10-25T00:22:36.060015+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-25T00:22:36.060015+0000","last_undegraded":"2023-10-25T00:22:36.060015+0000","last_fullsized":"2023-10-25T00:22:36.060015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522602+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-25T00:22:36.522602+0000","last_peered":"2023-10-25T00:22:36.522602+0000","last_clean":"2023-10-25T00:22:36.522602+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-25T00:22:36.522602+0000","last_undegraded":"2023-10-25T00:22:36.522602+0000","last_fullsized":"2023-10-25T00:22:36.522602+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"72'2","reported_seq":74,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517267+0000","last_change":"2023-10-24T23:53:11.432666+0000","last_active":"2023-10-25T00:22:33.517267+0000","last_peered":"2023-10-25T00:22:33.517267+0000","last_clean":"2023-10-25T00:22:33.517267+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-25T00:22:33.517267+0000","last_undegraded":"2023-10-25T00:22:33.517267+0000","last_fullsized":"2023-10-25T00:22:33.517267+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"72'3753","reported_seq":3828,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437502+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-25T00:22:37.437502+0000","last_peered":"2023-10-25T00:22:37.437502+0000","last_clean":"2023-10-25T00:22:37.437502+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-25T00:22:37.437502+0000","last_undegraded":"2023-10-25T00:22:37.437502+0000","last_fullsized":"2023-10-25T00:22:37.437502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":21045,"num_write_kb":276986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'4101","reported_seq":4175,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965518+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-25T00:22:33.965518+0000","last_peered":"2023-10-25T00:22:33.965518+0000","last_clean":"2023-10-25T00:22:33.965518+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-25T00:22:33.965518+0000","last_undegraded":"2023-10-25T00:22:33.965518+0000","last_fullsized":"2023-10-25T00:22:33.965518+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22383,"num_write_kb":270023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"73'13","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.230901+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-25T00:22:49.230901+0000","last_peered":"2023-10-25T00:22:49.230901+0000","last_clean":"2023-10-25T00:22:49.230901+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-25T00:22:49.230901+0000","last_undegraded":"2023-10-25T00:22:49.230901+0000","last_fullsized":"2023-10-25T00:22:49.230901+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:51.317510+0000","last_change":"2023-10-25T00:22:51.317813+0000","last_active":"2023-10-25T00:22:51.317510+0000","last_peered":"2023-10-25T00:22:51.317510+0000","last_clean":"2023-10-25T00:22:51.317510+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-25T00:22:51.317510+0000","last_undegraded":"2023-10-25T00:22:51.317510+0000","last_fullsized":"2023-10-25T00:22:51.317510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.317462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:03:14.150576+0000","scrub_duration":0.0044508079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"72'248","reported_seq":341,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427421+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-25T00:22:34.427421+0000","last_peered":"2023-10-25T00:22:34.427421+0000","last_clean":"2023-10-25T00:22:34.427421+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-25T00:22:34.427421+0000","last_undegraded":"2023-10-25T00:22:34.427421+0000","last_fullsized":"2023-10-25T00:22:34.427421+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":62,"num_write":607,"num_write_kb":223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"72'5236","reported_seq":5313,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965779+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-25T00:22:33.965779+0000","last_peered":"2023-10-25T00:22:33.965779+0000","last_clean":"2023-10-25T00:22:33.965779+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-25T00:22:33.965779+0000","last_undegraded":"2023-10-25T00:22:33.965779+0000","last_fullsized":"2023-10-25T00:22:33.965779+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":29554,"num_write_kb":368820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'5444","reported_seq":5522,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517060+0000","last_change":"2023-10-24T23:53:02.727126+0000","last_active":"2023-10-25T00:22:33.517060+0000","last_peered":"2023-10-25T00:22:33.517060+0000","last_clean":"2023-10-25T00:22:33.517060+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-25T00:22:33.517060+0000","last_undegraded":"2023-10-25T00:22:33.517060+0000","last_fullsized":"2023-10-25T00:22:33.517060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28342,"num_write_kb":342931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"72'66","reported_seq":226,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.226645+0000","last_change":"2023-10-24T23:53:02.358378+0000","last_active":"2023-10-25T00:22:49.226645+0000","last_peered":"2023-10-25T00:22:49.226645+0000","last_clean":"2023-10-25T00:22:49.226645+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-25T00:22:49.226645+0000","last_undegraded":"2023-10-25T00:22:49.226645+0000","last_fullsized":"2023-10-25T00:22:49.226645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":125,"num_write":61,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.436968+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-25T00:22:37.436968+0000","last_peered":"2023-10-25T00:22:37.436968+0000","last_clean":"2023-10-25T00:22:37.436968+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-25T00:22:37.436968+0000","last_undegraded":"2023-10-25T00:22:37.436968+0000","last_fullsized":"2023-10-25T00:22:37.436968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"72'10","reported_seq":137,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:45.425308+0000","last_change":"2023-10-25T00:22:45.425308+0000","last_active":"2023-10-25T00:22:45.425308+0000","last_peered":"2023-10-25T00:22:45.425308+0000","last_clean":"2023-10-25T00:22:45.425308+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-25T00:22:45.425308+0000","last_undegraded":"2023-10-25T00:22:45.425308+0000","last_fullsized":"2023-10-25T00:22:45.425308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.425052+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:36:38.527058+0000","scrub_duration":0.14014134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":135,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"72'3666","reported_seq":3737,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523179+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-25T00:22:36.523179+0000","last_peered":"2023-10-25T00:22:36.523179+0000","last_clean":"2023-10-25T00:22:36.523179+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-25T00:22:36.523179+0000","last_undegraded":"2023-10-25T00:22:36.523179+0000","last_fullsized":"2023-10-25T00:22:36.523179+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20519,"num_write_kb":254391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"68'4182","reported_seq":4258,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978808+0000","last_change":"2023-10-24T23:53:02.725803+0000","last_active":"2023-10-25T00:22:36.978808+0000","last_peered":"2023-10-25T00:22:36.978808+0000","last_clean":"2023-10-25T00:22:36.978808+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-25T00:22:36.978808+0000","last_undegraded":"2023-10-25T00:22:36.978808+0000","last_fullsized":"2023-10-25T00:22:36.978808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22839,"num_write_kb":259586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'4","reported_seq":80,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.533686+0000","last_change":"2023-10-24T23:53:02.326735+0000","last_active":"2023-10-25T00:22:35.533686+0000","last_peered":"2023-10-25T00:22:35.533686+0000","last_clean":"2023-10-25T00:22:35.533686+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-25T00:22:35.533686+0000","last_undegraded":"2023-10-25T00:22:35.533686+0000","last_fullsized":"2023-10-25T00:22:35.533686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522626+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-25T00:22:36.522626+0000","last_peered":"2023-10-25T00:22:36.522626+0000","last_clean":"2023-10-25T00:22:36.522626+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-25T00:22:36.522626+0000","last_undegraded":"2023-10-25T00:22:36.522626+0000","last_fullsized":"2023-10-25T00:22:36.522626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"43'161","reported_seq":286,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.210661+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-25T00:22:49.210661+0000","last_peered":"2023-10-25T00:22:49.210661+0000","last_clean":"2023-10-25T00:22:49.210661+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-25T00:22:49.210661+0000","last_undegraded":"2023-10-25T00:22:49.210661+0000","last_fullsized":"2023-10-25T00:22:49.210661+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"72'2","reported_seq":85,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:48.067150+0000","last_change":"2023-10-25T00:22:48.067150+0000","last_active":"2023-10-25T00:22:48.067150+0000","last_peered":"2023-10-25T00:22:48.067150+0000","last_clean":"2023-10-25T00:22:48.067150+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-25T00:22:48.067150+0000","last_undegraded":"2023-10-25T00:22:48.067150+0000","last_fullsized":"2023-10-25T00:22:48.067150+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.066810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:55:20.112674+0000","scrub_duration":0.0050122709999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"72'4105","reported_seq":4180,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523286+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-25T00:22:36.523286+0000","last_peered":"2023-10-25T00:22:36.523286+0000","last_clean":"2023-10-25T00:22:36.523286+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-25T00:22:36.523286+0000","last_undegraded":"2023-10-25T00:22:36.523286+0000","last_fullsized":"2023-10-25T00:22:36.523286+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":19,"num_write":23147,"num_write_kb":297959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'5705","reported_seq":5782,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.231359+0000","last_change":"2023-10-25T00:22:53.576536+0000","last_active":"2023-10-25T00:22:49.231359+0000","last_peered":"2023-10-25T00:22:49.231359+0000","last_clean":"2023-10-25T00:22:49.231359+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-25T00:22:49.231359+0000","last_undegraded":"2023-10-25T00:22:49.231359+0000","last_fullsized":"2023-10-25T00:22:49.231359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":29845,"num_write_kb":357286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"68'5","reported_seq":80,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.533697+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-25T00:22:35.533697+0000","last_peered":"2023-10-25T00:22:35.533697+0000","last_clean":"2023-10-25T00:22:35.533697+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-25T00:22:35.533697+0000","last_undegraded":"2023-10-25T00:22:35.533697+0000","last_fullsized":"2023-10-25T00:22:35.533697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"72'7","reported_seq":124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060732+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-25T00:22:36.060732+0000","last_peered":"2023-10-25T00:22:36.060732+0000","last_clean":"2023-10-25T00:22:36.060732+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-25T00:22:36.060732+0000","last_undegraded":"2023-10-25T00:22:36.060732+0000","last_fullsized":"2023-10-25T00:22:36.060732+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"72'4234","reported_seq":4308,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534135+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-25T00:22:35.534135+0000","last_peered":"2023-10-25T00:22:35.534135+0000","last_clean":"2023-10-25T00:22:35.534135+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-25T00:22:35.534135+0000","last_undegraded":"2023-10-25T00:22:35.534135+0000","last_fullsized":"2023-10-25T00:22:35.534135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":15,"num_write":23173,"num_write_kb":302022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"68'4052","reported_seq":4127,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060267+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-25T00:22:36.060267+0000","last_peered":"2023-10-25T00:22:36.060267+0000","last_clean":"2023-10-25T00:22:36.060267+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-25T00:22:36.060267+0000","last_undegraded":"2023-10-25T00:22:36.060267+0000","last_fullsized":"2023-10-25T00:22:36.060267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21486,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"68'2","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.059972+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-25T00:22:36.059972+0000","last_peered":"2023-10-25T00:22:36.059972+0000","last_clean":"2023-10-25T00:22:36.059972+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-25T00:22:36.059972+0000","last_undegraded":"2023-10-25T00:22:36.059972+0000","last_fullsized":"2023-10-25T00:22:36.059972+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"73'5","reported_seq":109,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:53.546236+0000","last_change":"2023-10-25T00:22:53.546332+0000","last_active":"2023-10-25T00:22:53.546236+0000","last_peered":"2023-10-25T00:22:53.546236+0000","last_clean":"2023-10-25T00:22:53.546236+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-25T00:22:53.546236+0000","last_undegraded":"2023-10-25T00:22:53.546236+0000","last_fullsized":"2023-10-25T00:22:53.546236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.546215+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:16:55.756192+0000","scrub_duration":0.27862564699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"72'4690","reported_seq":4764,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427502+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-25T00:22:34.427502+0000","last_peered":"2023-10-25T00:22:34.427502+0000","last_clean":"2023-10-25T00:22:34.427502+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-25T00:22:34.427502+0000","last_undegraded":"2023-10-25T00:22:34.427502+0000","last_fullsized":"2023-10-25T00:22:34.427502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24994,"num_write_kb":341798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'4780","reported_seq":4857,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060242+0000","last_change":"2023-10-24T23:53:02.721447+0000","last_active":"2023-10-25T00:22:36.060242+0000","last_peered":"2023-10-25T00:22:36.060242+0000","last_clean":"2023-10-25T00:22:36.060242+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-25T00:22:36.060242+0000","last_undegraded":"2023-10-25T00:22:36.060242+0000","last_fullsized":"2023-10-25T00:22:36.060242+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25083,"num_write_kb":293163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'3","reported_seq":89,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:48.328017+0000","last_change":"2023-10-25T00:22:48.328017+0000","last_active":"2023-10-25T00:22:48.328017+0000","last_peered":"2023-10-25T00:22:48.328017+0000","last_clean":"2023-10-25T00:22:48.328017+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-25T00:22:48.328017+0000","last_undegraded":"2023-10-25T00:22:48.328017+0000","last_fullsized":"2023-10-25T00:22:48.328017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.327778+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:58.116941+0000","scrub_duration":0.061328572999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"72'5","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.576377+0000","last_change":"2023-10-25T00:22:55.576688+0000","last_active":"2023-10-25T00:22:55.576377+0000","last_peered":"2023-10-25T00:22:55.576377+0000","last_clean":"2023-10-25T00:22:55.576377+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-25T00:22:55.576377+0000","last_undegraded":"2023-10-25T00:22:55.576377+0000","last_fullsized":"2023-10-25T00:22:55.576377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.576324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:43.752897+0000","scrub_duration":0.0045481469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"72'3596","reported_seq":3671,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060821+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-25T00:22:36.060821+0000","last_peered":"2023-10-25T00:22:36.060821+0000","last_clean":"2023-10-25T00:22:36.060821+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-25T00:22:36.060821+0000","last_undegraded":"2023-10-25T00:22:36.060821+0000","last_fullsized":"2023-10-25T00:22:36.060821+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":19820,"num_write_kb":259784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"68'4970","reported_seq":5057,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.999669+0000","last_change":"2023-10-25T00:22:55.999798+0000","last_active":"2023-10-25T00:22:55.999669+0000","last_peered":"2023-10-25T00:22:55.999669+0000","last_clean":"2023-10-25T00:22:55.999669+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-25T00:22:55.999669+0000","last_undegraded":"2023-10-25T00:22:55.999669+0000","last_fullsized":"2023-10-25T00:22:55.999669+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4970","last_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_deep_scrub":"68'4970","last_deep_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.999644+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:15:47.681358+0000","scrub_duration":0.177095113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26721,"num_write_kb":311966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"72'1464","reported_seq":1592,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.236014+0000","last_change":"2023-10-24T23:53:02.339660+0000","last_active":"2023-10-25T00:22:49.236014+0000","last_peered":"2023-10-25T00:22:49.236014+0000","last_clean":"2023-10-25T00:22:49.236014+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-25T00:22:49.236014+0000","last_undegraded":"2023-10-25T00:22:49.236014+0000","last_fullsized":"2023-10-25T00:22:49.236014+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":1464,"log_dups_size":0,"ondisk_log_size":1464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":47798,"num_write":1439,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"72'6","reported_seq":99,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427548+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-25T00:22:34.427548+0000","last_peered":"2023-10-25T00:22:34.427548+0000","last_clean":"2023-10-25T00:22:34.427548+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-25T00:22:34.427548+0000","last_undegraded":"2023-10-25T00:22:34.427548+0000","last_fullsized":"2023-10-25T00:22:34.427548+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"72'3548","reported_seq":3622,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517358+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-25T00:22:33.517358+0000","last_peered":"2023-10-25T00:22:33.517358+0000","last_clean":"2023-10-25T00:22:33.517358+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-25T00:22:33.517358+0000","last_undegraded":"2023-10-25T00:22:33.517358+0000","last_fullsized":"2023-10-25T00:22:33.517358+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20243,"num_write_kb":261539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"68'3393","reported_seq":3482,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.098602+0000","last_change":"2023-10-25T00:22:42.098602+0000","last_active":"2023-10-25T00:22:42.098602+0000","last_peered":"2023-10-25T00:22:42.098602+0000","last_clean":"2023-10-25T00:22:42.098602+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-25T00:22:42.098602+0000","last_undegraded":"2023-10-25T00:22:42.098602+0000","last_fullsized":"2023-10-25T00:22:42.098602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3393","last_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_deep_scrub":"68'3393","last_deep_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.098371+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:12.736136+0000","scrub_duration":0.153676797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17500,"num_write_kb":217638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"68'5","reported_seq":80,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522611+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-25T00:22:36.522611+0000","last_peered":"2023-10-25T00:22:36.522611+0000","last_clean":"2023-10-25T00:22:36.522611+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-25T00:22:36.522611+0000","last_undegraded":"2023-10-25T00:22:36.522611+0000","last_fullsized":"2023-10-25T00:22:36.522611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"72'3","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:48.308685+0000","last_change":"2023-10-25T00:22:49.292612+0000","last_active":"2023-10-25T00:22:48.308685+0000","last_peered":"2023-10-25T00:22:48.308685+0000","last_clean":"2023-10-25T00:22:48.308685+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-25T00:22:48.308685+0000","last_undegraded":"2023-10-25T00:22:48.308685+0000","last_fullsized":"2023-10-25T00:22:48.308685+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"72'3584","reported_seq":3658,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437442+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-25T00:22:37.437442+0000","last_peered":"2023-10-25T00:22:37.437442+0000","last_clean":"2023-10-25T00:22:37.437442+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-25T00:22:37.437442+0000","last_undegraded":"2023-10-25T00:22:37.437442+0000","last_fullsized":"2023-10-25T00:22:37.437442+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20504,"num_write_kb":254394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'5094","reported_seq":5185,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:51.484970+0000","last_change":"2023-10-25T00:22:51.485282+0000","last_active":"2023-10-25T00:22:51.484970+0000","last_peered":"2023-10-25T00:22:51.484970+0000","last_clean":"2023-10-25T00:22:51.484970+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-25T00:22:51.484970+0000","last_undegraded":"2023-10-25T00:22:51.484970+0000","last_fullsized":"2023-10-25T00:22:51.484970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5094","last_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_deep_scrub":"68'5094","last_deep_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.484917+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:22:43.899134+0000","scrub_duration":0.19299439099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27763,"num_write_kb":328490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'6","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.213667+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-25T00:22:49.213667+0000","last_peered":"2023-10-25T00:22:49.213667+0000","last_clean":"2023-10-25T00:22:49.213667+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-25T00:22:49.213667+0000","last_undegraded":"2023-10-25T00:22:49.213667+0000","last_fullsized":"2023-10-25T00:22:49.213667+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"72'1438","reported_seq":1557,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427567+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-25T00:22:34.427567+0000","last_peered":"2023-10-25T00:22:34.427567+0000","last_clean":"2023-10-25T00:22:34.427567+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-25T00:22:34.427567+0000","last_undegraded":"2023-10-25T00:22:34.427567+0000","last_fullsized":"2023-10-25T00:22:34.427567+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1438,"log_dups_size":0,"ondisk_log_size":1438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":35275,"num_write":1412,"num_write_kb":4487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"72'3292","reported_seq":3367,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534103+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-25T00:22:35.534103+0000","last_peered":"2023-10-25T00:22:35.534103+0000","last_clean":"2023-10-25T00:22:35.534103+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-25T00:22:35.534103+0000","last_undegraded":"2023-10-25T00:22:35.534103+0000","last_fullsized":"2023-10-25T00:22:35.534103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17853,"num_write_kb":243602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'4242","reported_seq":4328,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:45.893397+0000","last_change":"2023-10-25T00:22:45.893397+0000","last_active":"2023-10-25T00:22:45.893397+0000","last_peered":"2023-10-25T00:22:45.893397+0000","last_clean":"2023-10-25T00:22:45.893397+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-25T00:22:45.893397+0000","last_undegraded":"2023-10-25T00:22:45.893397+0000","last_fullsized":"2023-10-25T00:22:45.893397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4242","last_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_deep_scrub":"67'4242","last_deep_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.893150+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:24:28.749675+0000","scrub_duration":0.21189509300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22687,"num_write_kb":275346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'7","reported_seq":128,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.427815+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-25T00:22:52.427815+0000","last_peered":"2023-10-25T00:22:52.427815+0000","last_clean":"2023-10-25T00:22:52.427815+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-25T00:22:52.427815+0000","last_undegraded":"2023-10-25T00:22:52.427815+0000","last_fullsized":"2023-10-25T00:22:52.427815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"72'5","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427402+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-25T00:22:34.427402+0000","last_peered":"2023-10-25T00:22:34.427402+0000","last_clean":"2023-10-25T00:22:34.427402+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-25T00:22:34.427402+0000","last_undegraded":"2023-10-25T00:22:34.427402+0000","last_fullsized":"2023-10-25T00:22:34.427402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"72'4091","reported_seq":4165,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523278+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-25T00:22:36.523278+0000","last_peered":"2023-10-25T00:22:36.523278+0000","last_clean":"2023-10-25T00:22:36.523278+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-25T00:22:36.523278+0000","last_undegraded":"2023-10-25T00:22:36.523278+0000","last_fullsized":"2023-10-25T00:22:36.523278+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22779,"num_write_kb":288098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'4310","reported_seq":4385,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522869+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-25T00:22:36.522869+0000","last_peered":"2023-10-25T00:22:36.522869+0000","last_clean":"2023-10-25T00:22:36.522869+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-25T00:22:36.522869+0000","last_undegraded":"2023-10-25T00:22:36.522869+0000","last_fullsized":"2023-10-25T00:22:36.522869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23791,"num_write_kb":283914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'4","reported_seq":102,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.212618+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-25T00:22:49.212618+0000","last_peered":"2023-10-25T00:22:49.212618+0000","last_clean":"2023-10-25T00:22:49.212618+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-25T00:22:49.212618+0000","last_undegraded":"2023-10-25T00:22:49.212618+0000","last_fullsized":"2023-10-25T00:22:49.212618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"72'2","reported_seq":84,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:38.034628+0000","last_change":"2023-10-25T00:22:38.034628+0000","last_active":"2023-10-25T00:22:38.034628+0000","last_peered":"2023-10-25T00:22:38.034628+0000","last_clean":"2023-10-25T00:22:38.034628+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-25T00:22:38.034628+0000","last_undegraded":"2023-10-25T00:22:38.034628+0000","last_fullsized":"2023-10-25T00:22:38.034628+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.034499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:09.450121+0000","scrub_duration":0.14675892500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"72'3695","reported_seq":3770,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437533+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-25T00:22:37.437533+0000","last_peered":"2023-10-25T00:22:37.437533+0000","last_clean":"2023-10-25T00:22:37.437533+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-25T00:22:37.437533+0000","last_undegraded":"2023-10-25T00:22:37.437533+0000","last_fullsized":"2023-10-25T00:22:37.437533+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21426,"num_write_kb":254037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"72'3","reported_seq":84,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.331112+0000","last_change":"2023-10-25T00:22:42.331112+0000","last_active":"2023-10-25T00:22:42.331112+0000","last_peered":"2023-10-25T00:22:42.331112+0000","last_clean":"2023-10-25T00:22:42.331112+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-25T00:22:42.331112+0000","last_undegraded":"2023-10-25T00:22:42.331112+0000","last_fullsized":"2023-10-25T00:22:42.331112+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.330786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:23:56.573805+0000","scrub_duration":0.0043674530000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"72'19","reported_seq":217,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.643945+0000","last_change":"2023-10-25T00:22:47.253320+0000","last_active":"2023-10-25T00:22:52.643945+0000","last_peered":"2023-10-25T00:22:52.643945+0000","last_clean":"2023-10-25T00:22:52.643945+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-25T00:22:52.643945+0000","last_undegraded":"2023-10-25T00:22:52.643945+0000","last_fullsized":"2023-10-25T00:22:52.643945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_deep_scrub":"72'19","last_deep_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.252995+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:03.720456+0000","scrub_duration":0.018599613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":339,"num_read_kb":339,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'5249","reported_seq":5342,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.439199+0000","last_change":"2023-10-25T00:22:52.439461+0000","last_active":"2023-10-25T00:22:52.439199+0000","last_peered":"2023-10-25T00:22:52.439199+0000","last_clean":"2023-10-25T00:22:52.439199+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-25T00:22:52.439199+0000","last_undegraded":"2023-10-25T00:22:52.439199+0000","last_fullsized":"2023-10-25T00:22:52.439199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5249","last_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_deep_scrub":"68'5249","last_deep_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.439150+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:19:03.730037+0000","scrub_duration":0.077069795999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":27646,"num_write_kb":332285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"72'4380","reported_seq":4452,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427518+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-25T00:22:34.427518+0000","last_peered":"2023-10-25T00:22:34.427518+0000","last_clean":"2023-10-25T00:22:34.427518+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-25T00:22:34.427518+0000","last_undegraded":"2023-10-25T00:22:34.427518+0000","last_fullsized":"2023-10-25T00:22:34.427518+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23662,"num_write_kb":324338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"72'5","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:50.617951+0000","last_change":"2023-10-25T00:22:50.618067+0000","last_active":"2023-10-25T00:22:50.617951+0000","last_peered":"2023-10-25T00:22:50.617951+0000","last_clean":"2023-10-25T00:22:50.617951+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-25T00:22:50.617951+0000","last_undegraded":"2023-10-25T00:22:50.617951+0000","last_fullsized":"2023-10-25T00:22:50.617951+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.617931+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:24.381368+0000","scrub_duration":0.234626104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"68'5","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437000+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-25T00:22:37.437000+0000","last_peered":"2023-10-25T00:22:37.437000+0000","last_clean":"2023-10-25T00:22:37.437000+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-25T00:22:37.437000+0000","last_undegraded":"2023-10-25T00:22:37.437000+0000","last_fullsized":"2023-10-25T00:22:37.437000+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'4442","reported_seq":4516,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517147+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-25T00:22:33.517147+0000","last_peered":"2023-10-25T00:22:33.517147+0000","last_clean":"2023-10-25T00:22:33.517147+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-25T00:22:33.517147+0000","last_undegraded":"2023-10-25T00:22:33.517147+0000","last_fullsized":"2023-10-25T00:22:33.517147+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22810,"num_write_kb":296225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"72'4076","reported_seq":4150,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060841+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-25T00:22:36.060841+0000","last_peered":"2023-10-25T00:22:36.060841+0000","last_clean":"2023-10-25T00:22:36.060841+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-25T00:22:36.060841+0000","last_undegraded":"2023-10-25T00:22:36.060841+0000","last_fullsized":"2023-10-25T00:22:36.060841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22055,"num_write_kb":297609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"68'3","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.516763+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-25T00:22:33.516763+0000","last_peered":"2023-10-25T00:22:33.516763+0000","last_clean":"2023-10-25T00:22:33.516763+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-25T00:22:33.516763+0000","last_undegraded":"2023-10-25T00:22:33.516763+0000","last_fullsized":"2023-10-25T00:22:33.516763+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'4101","reported_seq":4187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:50.337590+0000","last_change":"2023-10-25T00:22:50.337815+0000","last_active":"2023-10-25T00:22:50.337590+0000","last_peered":"2023-10-25T00:22:50.337590+0000","last_clean":"2023-10-25T00:22:50.337590+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-25T00:22:50.337590+0000","last_undegraded":"2023-10-25T00:22:50.337590+0000","last_fullsized":"2023-10-25T00:22:50.337590+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.337536+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:01.275651+0000","scrub_duration":0.055303021000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22343,"num_write_kb":253486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"72'397","reported_seq":590,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:45.554898+0000","last_change":"2023-10-25T00:22:45.554898+0000","last_active":"2023-10-25T00:22:45.554898+0000","last_peered":"2023-10-25T00:22:45.554898+0000","last_clean":"2023-10-25T00:22:45.554898+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-25T00:22:45.554898+0000","last_undegraded":"2023-10-25T00:22:45.554898+0000","last_fullsized":"2023-10-25T00:22:45.554898+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'397","last_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_deep_scrub":"72'397","last_deep_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.554526+0000","objects_scrubbed":4,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:41:32.056818+0000","scrub_duration":0.10248053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":86197,"num_write":389,"num_write_kb":4306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"72'4606","reported_seq":4678,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517449+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-25T00:22:33.517449+0000","last_peered":"2023-10-25T00:22:33.517449+0000","last_clean":"2023-10-25T00:22:33.517449+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-25T00:22:33.517449+0000","last_undegraded":"2023-10-25T00:22:33.517449+0000","last_fullsized":"2023-10-25T00:22:33.517449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25763,"num_write_kb":316477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"73'12","reported_seq":132,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.803776+0000","last_change":"2023-10-24T23:53:02.359393+0000","last_active":"2023-10-25T00:22:49.803776+0000","last_peered":"2023-10-25T00:22:49.803776+0000","last_clean":"2023-10-25T00:22:49.803776+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-25T00:22:49.803776+0000","last_undegraded":"2023-10-25T00:22:49.803776+0000","last_fullsized":"2023-10-25T00:22:49.803776+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'5680","reported_seq":5769,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:54.585888+0000","last_change":"2023-10-25T00:22:54.586176+0000","last_active":"2023-10-25T00:22:54.585888+0000","last_peered":"2023-10-25T00:22:54.585888+0000","last_clean":"2023-10-25T00:22:54.585888+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-25T00:22:54.585888+0000","last_undegraded":"2023-10-25T00:22:54.585888+0000","last_fullsized":"2023-10-25T00:22:54.585888+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5680","last_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_deep_scrub":"68'5680","last_deep_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.585866+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:49.342364+0000","scrub_duration":0.20835251299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29749,"num_write_kb":360962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"72'38","reported_seq":174,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523199+0000","last_change":"2023-10-24T23:53:11.436964+0000","last_active":"2023-10-25T00:22:36.523199+0000","last_peered":"2023-10-25T00:22:36.523199+0000","last_clean":"2023-10-25T00:22:36.523199+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-25T00:22:36.523199+0000","last_undegraded":"2023-10-25T00:22:36.523199+0000","last_fullsized":"2023-10-25T00:22:36.523199+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":39,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"72'4808","reported_seq":4886,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978914+0000","last_change":"2023-10-24T23:53:11.814304+0000","last_active":"2023-10-25T00:22:36.978914+0000","last_peered":"2023-10-25T00:22:36.978914+0000","last_clean":"2023-10-25T00:22:36.978914+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-25T00:22:36.978914+0000","last_undegraded":"2023-10-25T00:22:36.978914+0000","last_fullsized":"2023-10-25T00:22:36.978914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26552,"num_write_kb":336125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"68'4747","reported_seq":4824,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978701+0000","last_change":"2023-10-24T23:53:02.703619+0000","last_active":"2023-10-25T00:22:36.978701+0000","last_peered":"2023-10-25T00:22:36.978701+0000","last_clean":"2023-10-25T00:22:36.978701+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-25T00:22:36.978701+0000","last_undegraded":"2023-10-25T00:22:36.978701+0000","last_fullsized":"2023-10-25T00:22:36.978701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24847,"num_write_kb":316685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"72'4283","reported_seq":4359,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978634+0000","last_change":"2023-10-24T23:53:11.807096+0000","last_active":"2023-10-25T00:22:36.978634+0000","last_peered":"2023-10-25T00:22:36.978634+0000","last_clean":"2023-10-25T00:22:36.978634+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-25T00:22:36.978634+0000","last_undegraded":"2023-10-25T00:22:36.978634+0000","last_fullsized":"2023-10-25T00:22:36.978634+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23853,"num_write_kb":296504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"68'4036","reported_seq":4124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:39.642537+0000","last_change":"2023-10-25T00:22:39.642537+0000","last_active":"2023-10-25T00:22:39.642537+0000","last_peered":"2023-10-25T00:22:39.642537+0000","last_clean":"2023-10-25T00:22:39.642537+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-25T00:22:39.642537+0000","last_undegraded":"2023-10-25T00:22:39.642537+0000","last_fullsized":"2023-10-25T00:22:39.642537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4036","last_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_deep_scrub":"68'4036","last_deep_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.642342+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:02.362976+0000","scrub_duration":0.227585228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21233,"num_write_kb":273467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"72'4276","reported_seq":4351,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965799+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-25T00:22:33.965799+0000","last_peered":"2023-10-25T00:22:33.965799+0000","last_clean":"2023-10-25T00:22:33.965799+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-25T00:22:33.965799+0000","last_undegraded":"2023-10-25T00:22:33.965799+0000","last_fullsized":"2023-10-25T00:22:33.965799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23362,"num_write_kb":310817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"68'6201","reported_seq":6280,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060597+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-25T00:22:36.060597+0000","last_peered":"2023-10-25T00:22:36.060597+0000","last_clean":"2023-10-25T00:22:36.060597+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-25T00:22:36.060597+0000","last_undegraded":"2023-10-25T00:22:36.060597+0000","last_fullsized":"2023-10-25T00:22:36.060597+0000","mapping_epoch":21,"log_start":"62'1000","ondisk_log_start":"62'1000","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33486,"num_write_kb":384441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"72'4018","reported_seq":4094,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437698+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-25T00:22:37.437698+0000","last_peered":"2023-10-25T00:22:37.437698+0000","last_clean":"2023-10-25T00:22:37.437698+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-25T00:22:37.437698+0000","last_undegraded":"2023-10-25T00:22:37.437698+0000","last_fullsized":"2023-10-25T00:22:37.437698+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":21920,"num_write_kb":288651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"68'4522","reported_seq":4600,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534002+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-25T00:22:35.534002+0000","last_peered":"2023-10-25T00:22:35.534002+0000","last_clean":"2023-10-25T00:22:35.534002+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-25T00:22:35.534002+0000","last_undegraded":"2023-10-25T00:22:35.534002+0000","last_fullsized":"2023-10-25T00:22:35.534002+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24476,"num_write_kb":282500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"72'4209","reported_seq":4281,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965838+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-25T00:22:33.965838+0000","last_peered":"2023-10-25T00:22:33.965838+0000","last_clean":"2023-10-25T00:22:33.965838+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-25T00:22:33.965838+0000","last_undegraded":"2023-10-25T00:22:33.965838+0000","last_fullsized":"2023-10-25T00:22:33.965838+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24141,"num_write_kb":281325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"68'4915","reported_seq":5001,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:38.600278+0000","last_change":"2023-10-25T00:22:38.600278+0000","last_active":"2023-10-25T00:22:38.600278+0000","last_peered":"2023-10-25T00:22:38.600278+0000","last_clean":"2023-10-25T00:22:38.600278+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-25T00:22:38.600278+0000","last_undegraded":"2023-10-25T00:22:38.600278+0000","last_fullsized":"2023-10-25T00:22:38.600278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4915","last_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_deep_scrub":"68'4915","last_deep_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.600135+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:42:30.671003+0000","scrub_duration":0.18827047799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26161,"num_write_kb":314084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"72'4933","reported_seq":5005,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978871+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-25T00:22:36.978871+0000","last_peered":"2023-10-25T00:22:36.978871+0000","last_clean":"2023-10-25T00:22:36.978871+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-25T00:22:36.978871+0000","last_undegraded":"2023-10-25T00:22:36.978871+0000","last_fullsized":"2023-10-25T00:22:36.978871+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26695,"num_write_kb":352905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"68'5055","reported_seq":5142,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:43.444464+0000","last_change":"2023-10-25T00:22:43.444464+0000","last_active":"2023-10-25T00:22:43.444464+0000","last_peered":"2023-10-25T00:22:43.444464+0000","last_clean":"2023-10-25T00:22:43.444464+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-25T00:22:43.444464+0000","last_undegraded":"2023-10-25T00:22:43.444464+0000","last_fullsized":"2023-10-25T00:22:43.444464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5055","last_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_deep_scrub":"68'5055","last_deep_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_clean_scrub_stamp":"2023-10-25T00:22:43.444210+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:13:34.310177+0000","scrub_duration":0.081291211000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27516,"num_write_kb":327406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"72'4083","reported_seq":4159,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523384+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-25T00:22:36.523384+0000","last_peered":"2023-10-25T00:22:36.523384+0000","last_clean":"2023-10-25T00:22:36.523384+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-25T00:22:36.523384+0000","last_undegraded":"2023-10-25T00:22:36.523384+0000","last_fullsized":"2023-10-25T00:22:36.523384+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":22892,"num_write_kb":296872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"68'4777","reported_seq":4854,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965628+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-25T00:22:33.965628+0000","last_peered":"2023-10-25T00:22:33.965628+0000","last_clean":"2023-10-25T00:22:33.965628+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-25T00:22:33.965628+0000","last_undegraded":"2023-10-25T00:22:33.965628+0000","last_fullsized":"2023-10-25T00:22:33.965628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25812,"num_write_kb":300181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"72'4618","reported_seq":4695,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.979240+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-25T00:22:36.979240+0000","last_peered":"2023-10-25T00:22:36.979240+0000","last_clean":"2023-10-25T00:22:36.979240+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-25T00:22:36.979240+0000","last_undegraded":"2023-10-25T00:22:36.979240+0000","last_fullsized":"2023-10-25T00:22:36.979240+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":26180,"num_write_kb":326342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"68'5340","reported_seq":5426,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.520048+0000","last_change":"2023-10-25T00:22:42.520048+0000","last_active":"2023-10-25T00:22:42.520048+0000","last_peered":"2023-10-25T00:22:42.520048+0000","last_clean":"2023-10-25T00:22:42.520048+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-25T00:22:42.520048+0000","last_undegraded":"2023-10-25T00:22:42.520048+0000","last_fullsized":"2023-10-25T00:22:42.520048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5340","last_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_deep_scrub":"68'5340","last_deep_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.519886+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.832889+0000","scrub_duration":0.129134375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28701,"num_write_kb":348521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"72'4171","reported_seq":4245,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060881+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-25T00:22:36.060881+0000","last_peered":"2023-10-25T00:22:36.060881+0000","last_clean":"2023-10-25T00:22:36.060881+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-25T00:22:36.060881+0000","last_undegraded":"2023-10-25T00:22:36.060881+0000","last_fullsized":"2023-10-25T00:22:36.060881+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22551,"num_write_kb":303498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"68'4493","reported_seq":4569,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437184+0000","last_change":"2023-10-24T23:53:02.726386+0000","last_active":"2023-10-25T00:22:37.437184+0000","last_peered":"2023-10-25T00:22:37.437184+0000","last_clean":"2023-10-25T00:22:37.437184+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-25T00:22:37.437184+0000","last_undegraded":"2023-10-25T00:22:37.437184+0000","last_fullsized":"2023-10-25T00:22:37.437184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23797,"num_write_kb":300260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"72'4763","reported_seq":4835,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517387+0000","last_change":"2023-10-24T23:53:44.474203+0000","last_active":"2023-10-25T00:22:33.517387+0000","last_peered":"2023-10-25T00:22:33.517387+0000","last_clean":"2023-10-25T00:22:33.517387+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-25T00:22:33.517387+0000","last_undegraded":"2023-10-25T00:22:33.517387+0000","last_fullsized":"2023-10-25T00:22:33.517387+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":25751,"num_write_kb":345225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"68'4946","reported_seq":5022,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.358533+0000","last_change":"2023-10-25T00:22:49.234045+0000","last_active":"2023-10-25T00:22:37.358533+0000","last_peered":"2023-10-25T00:22:37.358533+0000","last_clean":"2023-10-25T00:22:37.358533+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-25T00:22:37.358533+0000","last_undegraded":"2023-10-25T00:22:37.358533+0000","last_fullsized":"2023-10-25T00:22:37.358533+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26583,"num_write_kb":315633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"72'4352","reported_seq":4425,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427628+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-25T00:22:34.427628+0000","last_peered":"2023-10-25T00:22:34.427628+0000","last_clean":"2023-10-25T00:22:34.427628+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-25T00:22:34.427628+0000","last_undegraded":"2023-10-25T00:22:34.427628+0000","last_fullsized":"2023-10-25T00:22:34.427628+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23957,"num_write_kb":301052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"68'5026","reported_seq":5101,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517220+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-25T00:22:33.517220+0000","last_peered":"2023-10-25T00:22:33.517220+0000","last_clean":"2023-10-25T00:22:33.517220+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-25T00:22:33.517220+0000","last_undegraded":"2023-10-25T00:22:33.517220+0000","last_fullsized":"2023-10-25T00:22:33.517220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26401,"num_write_kb":310586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"72'4393","reported_seq":4471,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437748+0000","last_change":"2023-10-24T23:53:11.860133+0000","last_active":"2023-10-25T00:22:37.437748+0000","last_peered":"2023-10-25T00:22:37.437748+0000","last_clean":"2023-10-25T00:22:37.437748+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-25T00:22:37.437748+0000","last_undegraded":"2023-10-25T00:22:37.437748+0000","last_fullsized":"2023-10-25T00:22:37.437748+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":24752,"num_write_kb":296109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"68'5620","reported_seq":5699,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437209+0000","last_change":"2023-10-24T23:53:02.729152+0000","last_active":"2023-10-25T00:22:37.437209+0000","last_peered":"2023-10-25T00:22:37.437209+0000","last_clean":"2023-10-25T00:22:37.437209+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-25T00:22:37.437209+0000","last_undegraded":"2023-10-25T00:22:37.437209+0000","last_fullsized":"2023-10-25T00:22:37.437209+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29863,"num_write_kb":337169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"72'4340","reported_seq":4416,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978854+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-25T00:22:36.978854+0000","last_peered":"2023-10-25T00:22:36.978854+0000","last_clean":"2023-10-25T00:22:36.978854+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-25T00:22:36.978854+0000","last_undegraded":"2023-10-25T00:22:36.978854+0000","last_fullsized":"2023-10-25T00:22:36.978854+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":36,"num_write":23367,"num_write_kb":302669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"68'4464","reported_seq":4538,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060463+0000","last_change":"2023-10-24T23:53:02.724814+0000","last_active":"2023-10-25T00:22:36.060463+0000","last_peered":"2023-10-25T00:22:36.060463+0000","last_clean":"2023-10-25T00:22:36.060463+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-25T00:22:36.060463+0000","last_undegraded":"2023-10-25T00:22:36.060463+0000","last_fullsized":"2023-10-25T00:22:36.060463+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24372,"num_write_kb":278074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"72'5589","reported_seq":5661,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437759+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-25T00:22:37.437759+0000","last_peered":"2023-10-25T00:22:37.437759+0000","last_clean":"2023-10-25T00:22:37.437759+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-25T00:22:37.437759+0000","last_undegraded":"2023-10-25T00:22:37.437759+0000","last_fullsized":"2023-10-25T00:22:37.437759+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":30206,"num_write_kb":399557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"68'4542","reported_seq":4617,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427296+0000","last_change":"2023-10-24T23:53:02.703851+0000","last_active":"2023-10-25T00:22:34.427296+0000","last_peered":"2023-10-25T00:22:34.427296+0000","last_clean":"2023-10-25T00:22:34.427296+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-25T00:22:34.427296+0000","last_undegraded":"2023-10-25T00:22:34.427296+0000","last_fullsized":"2023-10-25T00:22:34.427296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23889,"num_write_kb":292323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"72'3430","reported_seq":3504,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978912+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-25T00:22:36.978912+0000","last_peered":"2023-10-25T00:22:36.978912+0000","last_clean":"2023-10-25T00:22:36.978912+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-25T00:22:36.978912+0000","last_undegraded":"2023-10-25T00:22:36.978912+0000","last_fullsized":"2023-10-25T00:22:36.978912+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19411,"num_write_kb":241862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"68'4719","reported_seq":4794,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060426+0000","last_change":"2023-10-24T23:53:02.723645+0000","last_active":"2023-10-25T00:22:36.060426+0000","last_peered":"2023-10-25T00:22:36.060426+0000","last_clean":"2023-10-25T00:22:36.060426+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-25T00:22:36.060426+0000","last_undegraded":"2023-10-25T00:22:36.060426+0000","last_fullsized":"2023-10-25T00:22:36.060426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25020,"num_write_kb":305197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"72'4169","reported_seq":4240,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427758+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-25T00:22:34.427758+0000","last_peered":"2023-10-25T00:22:34.427758+0000","last_clean":"2023-10-25T00:22:34.427758+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-25T00:22:34.427758+0000","last_undegraded":"2023-10-25T00:22:34.427758+0000","last_fullsized":"2023-10-25T00:22:34.427758+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23235,"num_write_kb":306175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"68'5952","reported_seq":6035,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517423+0000","last_change":"2023-10-24T23:53:44.629959+0000","last_active":"2023-10-25T00:22:33.517423+0000","last_peered":"2023-10-25T00:22:33.517423+0000","last_clean":"2023-10-25T00:22:33.517423+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-25T00:22:33.517423+0000","last_undegraded":"2023-10-25T00:22:33.517423+0000","last_fullsized":"2023-10-25T00:22:33.517423+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32341,"num_write_kb":375441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"72'3717","reported_seq":3788,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060966+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-25T00:22:36.060966+0000","last_peered":"2023-10-25T00:22:36.060966+0000","last_clean":"2023-10-25T00:22:36.060966+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-25T00:22:36.060966+0000","last_undegraded":"2023-10-25T00:22:36.060966+0000","last_fullsized":"2023-10-25T00:22:36.060966+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21101,"num_write_kb":269108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"68'4825","reported_seq":4901,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060533+0000","last_change":"2023-10-24T23:53:02.731105+0000","last_active":"2023-10-25T00:22:36.060533+0000","last_peered":"2023-10-25T00:22:36.060533+0000","last_clean":"2023-10-25T00:22:36.060533+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-25T00:22:36.060533+0000","last_undegraded":"2023-10-25T00:22:36.060533+0000","last_fullsized":"2023-10-25T00:22:36.060533+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25319,"num_write_kb":294113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"72'3714","reported_seq":3788,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523581+0000","last_change":"2023-10-24T23:53:11.847814+0000","last_active":"2023-10-25T00:22:36.523581+0000","last_peered":"2023-10-25T00:22:36.523581+0000","last_clean":"2023-10-25T00:22:36.523581+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-25T00:22:36.523581+0000","last_undegraded":"2023-10-25T00:22:36.523581+0000","last_fullsized":"2023-10-25T00:22:36.523581+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21148,"num_write_kb":265324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"68'4878","reported_seq":4966,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.227110+0000","last_change":"2023-10-25T00:22:49.227455+0000","last_active":"2023-10-25T00:22:49.227110+0000","last_peered":"2023-10-25T00:22:49.227110+0000","last_clean":"2023-10-25T00:22:49.227110+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-25T00:22:49.227110+0000","last_undegraded":"2023-10-25T00:22:49.227110+0000","last_fullsized":"2023-10-25T00:22:49.227110+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4878","last_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_deep_scrub":"68'4878","last_deep_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_clean_scrub_stamp":"2023-10-25T00:22:49.227012+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:58:21.157778+0000","scrub_duration":0.13340163999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26154,"num_write_kb":305558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"72'3434","reported_seq":3506,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437787+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-25T00:22:37.437787+0000","last_peered":"2023-10-25T00:22:37.437787+0000","last_clean":"2023-10-25T00:22:37.437787+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-25T00:22:37.437787+0000","last_undegraded":"2023-10-25T00:22:37.437787+0000","last_fullsized":"2023-10-25T00:22:37.437787+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18449,"num_write_kb":262373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"68'4672","reported_seq":4748,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522905+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-25T00:22:36.522905+0000","last_peered":"2023-10-25T00:22:36.522905+0000","last_clean":"2023-10-25T00:22:36.522905+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-25T00:22:36.522905+0000","last_undegraded":"2023-10-25T00:22:36.522905+0000","last_fullsized":"2023-10-25T00:22:36.522905+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25223,"num_write_kb":293792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"72'4013","reported_seq":4090,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523380+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-25T00:22:36.523380+0000","last_peered":"2023-10-25T00:22:36.523380+0000","last_clean":"2023-10-25T00:22:36.523380+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-25T00:22:36.523380+0000","last_undegraded":"2023-10-25T00:22:36.523380+0000","last_fullsized":"2023-10-25T00:22:36.523380+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":22066,"num_write_kb":297298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"68'3918","reported_seq":3992,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427405+0000","last_change":"2023-10-24T23:53:02.706435+0000","last_active":"2023-10-25T00:22:34.427405+0000","last_peered":"2023-10-25T00:22:34.427405+0000","last_clean":"2023-10-25T00:22:34.427405+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-25T00:22:34.427405+0000","last_undegraded":"2023-10-25T00:22:34.427405+0000","last_fullsized":"2023-10-25T00:22:34.427405+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21675,"num_write_kb":247000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"72'4401","reported_seq":4474,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523617+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-25T00:22:36.523617+0000","last_peered":"2023-10-25T00:22:36.523617+0000","last_clean":"2023-10-25T00:22:36.523617+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-25T00:22:36.523617+0000","last_undegraded":"2023-10-25T00:22:36.523617+0000","last_fullsized":"2023-10-25T00:22:36.523617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":24294,"num_write_kb":326749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"68'3970","reported_seq":4044,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427381+0000","last_change":"2023-10-24T23:53:02.726284+0000","last_active":"2023-10-25T00:22:34.427381+0000","last_peered":"2023-10-25T00:22:34.427381+0000","last_clean":"2023-10-25T00:22:34.427381+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-25T00:22:34.427381+0000","last_undegraded":"2023-10-25T00:22:34.427381+0000","last_fullsized":"2023-10-25T00:22:34.427381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21202,"num_write_kb":257085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"72'5559","reported_seq":5638,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517467+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-25T00:22:33.517467+0000","last_peered":"2023-10-25T00:22:33.517467+0000","last_clean":"2023-10-25T00:22:33.517467+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-25T00:22:33.517467+0000","last_undegraded":"2023-10-25T00:22:33.517467+0000","last_fullsized":"2023-10-25T00:22:33.517467+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":31202,"num_write_kb":391572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"68'4936","reported_seq":5011,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060547+0000","last_change":"2023-10-24T23:53:02.724261+0000","last_active":"2023-10-25T00:22:36.060547+0000","last_peered":"2023-10-25T00:22:36.060547+0000","last_clean":"2023-10-25T00:22:36.060547+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-25T00:22:36.060547+0000","last_undegraded":"2023-10-25T00:22:36.060547+0000","last_fullsized":"2023-10-25T00:22:36.060547+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26134,"num_write_kb":309652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"72'3558","reported_seq":3634,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517421+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-25T00:22:33.517421+0000","last_peered":"2023-10-25T00:22:33.517421+0000","last_clean":"2023-10-25T00:22:33.517421+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-25T00:22:33.517421+0000","last_undegraded":"2023-10-25T00:22:33.517421+0000","last_fullsized":"2023-10-25T00:22:33.517421+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":34,"num_write":19715,"num_write_kb":243025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"68'5282","reported_seq":5357,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522931+0000","last_change":"2023-10-24T23:53:02.726497+0000","last_active":"2023-10-25T00:22:36.522931+0000","last_peered":"2023-10-25T00:22:36.522931+0000","last_clean":"2023-10-25T00:22:36.522931+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-25T00:22:36.522931+0000","last_undegraded":"2023-10-25T00:22:36.522931+0000","last_fullsized":"2023-10-25T00:22:36.522931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27848,"num_write_kb":324561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"72'3914","reported_seq":3991,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523611+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-25T00:22:36.523611+0000","last_peered":"2023-10-25T00:22:36.523611+0000","last_clean":"2023-10-25T00:22:36.523611+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-25T00:22:36.523611+0000","last_undegraded":"2023-10-25T00:22:36.523611+0000","last_fullsized":"2023-10-25T00:22:36.523611+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":20938,"num_write_kb":288796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"68'6616","reported_seq":6695,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517263+0000","last_change":"2023-10-24T23:53:02.727179+0000","last_active":"2023-10-25T00:22:33.517263+0000","last_peered":"2023-10-25T00:22:33.517263+0000","last_clean":"2023-10-25T00:22:33.517263+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-25T00:22:33.517263+0000","last_undegraded":"2023-10-25T00:22:33.517263+0000","last_fullsized":"2023-10-25T00:22:33.517263+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36221,"num_write_kb":423916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"72'3536","reported_seq":3608,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965810+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-25T00:22:33.965810+0000","last_peered":"2023-10-25T00:22:33.965810+0000","last_clean":"2023-10-25T00:22:33.965810+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-25T00:22:33.965810+0000","last_undegraded":"2023-10-25T00:22:33.965810+0000","last_fullsized":"2023-10-25T00:22:33.965810+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19276,"num_write_kb":253506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"68'4980","reported_seq":5052,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522964+0000","last_change":"2023-10-24T23:53:02.720045+0000","last_active":"2023-10-25T00:22:36.522964+0000","last_peered":"2023-10-25T00:22:36.522964+0000","last_clean":"2023-10-25T00:22:36.522964+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-25T00:22:36.522964+0000","last_undegraded":"2023-10-25T00:22:36.522964+0000","last_fullsized":"2023-10-25T00:22:36.522964+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26525,"num_write_kb":306963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"72'4160","reported_seq":4234,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427795+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-25T00:22:34.427795+0000","last_peered":"2023-10-25T00:22:34.427795+0000","last_clean":"2023-10-25T00:22:34.427795+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-25T00:22:34.427795+0000","last_undegraded":"2023-10-25T00:22:34.427795+0000","last_fullsized":"2023-10-25T00:22:34.427795+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":21819,"num_write_kb":288439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"68'4434","reported_seq":4519,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:47.255974+0000","last_change":"2023-10-25T00:22:47.255974+0000","last_active":"2023-10-25T00:22:47.255974+0000","last_peered":"2023-10-25T00:22:47.255974+0000","last_clean":"2023-10-25T00:22:47.255974+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-25T00:22:47.255974+0000","last_undegraded":"2023-10-25T00:22:47.255974+0000","last_fullsized":"2023-10-25T00:22:47.255974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4434","last_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_deep_scrub":"68'4434","last_deep_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.255646+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:35:52.723567+0000","scrub_duration":0.14955275700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23791,"num_write_kb":292031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"72'4195","reported_seq":4264,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534145+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-25T00:22:35.534145+0000","last_peered":"2023-10-25T00:22:35.534145+0000","last_clean":"2023-10-25T00:22:35.534145+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-25T00:22:35.534145+0000","last_undegraded":"2023-10-25T00:22:35.534145+0000","last_fullsized":"2023-10-25T00:22:35.534145+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23000,"num_write_kb":285019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"68'4243","reported_seq":4316,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060720+0000","last_change":"2023-10-24T23:53:02.730963+0000","last_active":"2023-10-25T00:22:36.060720+0000","last_peered":"2023-10-25T00:22:36.060720+0000","last_clean":"2023-10-25T00:22:36.060720+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-25T00:22:36.060720+0000","last_undegraded":"2023-10-25T00:22:36.060720+0000","last_fullsized":"2023-10-25T00:22:36.060720+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22439,"num_write_kb":266094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"72'4017","reported_seq":4087,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523651+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-25T00:22:36.523651+0000","last_peered":"2023-10-25T00:22:36.523651+0000","last_clean":"2023-10-25T00:22:36.523651+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-25T00:22:36.523651+0000","last_undegraded":"2023-10-25T00:22:36.523651+0000","last_fullsized":"2023-10-25T00:22:36.523651+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22665,"num_write_kb":291707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"68'4509","reported_seq":4596,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:44.022792+0000","last_change":"2023-10-25T00:22:44.022792+0000","last_active":"2023-10-25T00:22:44.022792+0000","last_peered":"2023-10-25T00:22:44.022792+0000","last_clean":"2023-10-25T00:22:44.022792+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-25T00:22:44.022792+0000","last_undegraded":"2023-10-25T00:22:44.022792+0000","last_fullsized":"2023-10-25T00:22:44.022792+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4509","last_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_deep_scrub":"68'4509","last_deep_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.022573+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:54:30.651361+0000","scrub_duration":0.11145192700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23158,"num_write_kb":297883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"72'5235","reported_seq":5314,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534214+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-25T00:22:35.534214+0000","last_peered":"2023-10-25T00:22:35.534214+0000","last_clean":"2023-10-25T00:22:35.534214+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-25T00:22:35.534214+0000","last_undegraded":"2023-10-25T00:22:35.534214+0000","last_fullsized":"2023-10-25T00:22:35.534214+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":15,"num_write":29235,"num_write_kb":367391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"68'4260","reported_seq":4337,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978732+0000","last_change":"2023-10-24T23:53:02.727426+0000","last_active":"2023-10-25T00:22:36.978732+0000","last_peered":"2023-10-25T00:22:36.978732+0000","last_clean":"2023-10-25T00:22:36.978732+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-25T00:22:36.978732+0000","last_undegraded":"2023-10-25T00:22:36.978732+0000","last_fullsized":"2023-10-25T00:22:36.978732+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23357,"num_write_kb":265837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"72'4656","reported_seq":4737,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534295+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-25T00:22:35.534295+0000","last_peered":"2023-10-25T00:22:35.534295+0000","last_clean":"2023-10-25T00:22:35.534295+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-25T00:22:35.534295+0000","last_undegraded":"2023-10-25T00:22:35.534295+0000","last_fullsized":"2023-10-25T00:22:35.534295+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":26109,"num_write_kb":317781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"68'4636","reported_seq":4726,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:39.672588+0000","last_change":"2023-10-25T00:22:39.672588+0000","last_active":"2023-10-25T00:22:39.672588+0000","last_peered":"2023-10-25T00:22:39.672588+0000","last_clean":"2023-10-25T00:22:39.672588+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-25T00:22:39.672588+0000","last_undegraded":"2023-10-25T00:22:39.672588+0000","last_fullsized":"2023-10-25T00:22:39.672588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4636","last_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_deep_scrub":"68'4636","last_deep_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.672438+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:47:15.433547+0000","scrub_duration":0.124153799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25124,"num_write_kb":285314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"72'3117","reported_seq":3192,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427804+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-25T00:22:34.427804+0000","last_peered":"2023-10-25T00:22:34.427804+0000","last_clean":"2023-10-25T00:22:34.427804+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-25T00:22:34.427804+0000","last_undegraded":"2023-10-25T00:22:34.427804+0000","last_fullsized":"2023-10-25T00:22:34.427804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17240,"num_write_kb":232937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"68'5520","reported_seq":5606,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522985+0000","last_change":"2023-10-24T23:53:44.630867+0000","last_active":"2023-10-25T00:22:36.522985+0000","last_peered":"2023-10-25T00:22:36.522985+0000","last_clean":"2023-10-25T00:22:36.522985+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-25T00:22:36.522985+0000","last_undegraded":"2023-10-25T00:22:36.522985+0000","last_fullsized":"2023-10-25T00:22:36.522985+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29665,"num_write_kb":368823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"72'3962","reported_seq":4036,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517532+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-25T00:22:33.517532+0000","last_peered":"2023-10-25T00:22:33.517532+0000","last_clean":"2023-10-25T00:22:33.517532+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-25T00:22:33.517532+0000","last_undegraded":"2023-10-25T00:22:33.517532+0000","last_fullsized":"2023-10-25T00:22:33.517532+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21989,"num_write_kb":280762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"68'4908","reported_seq":4989,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427311+0000","last_change":"2023-10-24T23:53:02.706543+0000","last_active":"2023-10-25T00:22:34.427311+0000","last_peered":"2023-10-25T00:22:34.427311+0000","last_clean":"2023-10-25T00:22:34.427311+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-25T00:22:34.427311+0000","last_undegraded":"2023-10-25T00:22:34.427311+0000","last_fullsized":"2023-10-25T00:22:34.427311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26829,"num_write_kb":320697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":320,"num_write":1482516,"num_write_kb":19060243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267718,"ondisk_log_size":267718,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":19490997,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1686,"num_read_kb":209824,"num_write":4665,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39043072,"data_stored":38981994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4549,"ondisk_log_size":4549,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":0,"num_write":1661441,"num_write_kb":19773626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307026,"ondisk_log_size":307026,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":20061533,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1689,"num_read_kb":267203,"num_write":4630,"num_write_kb":21638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40181760,"data_stored":40123066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4511,"ondisk_log_size":4511,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575212,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919096,"kb_used_data":12336,"kb_used_omap":0,"kb_used_meta":906752,"kb_avail":93452744,"statfs":{"total":96636764160,"available":95695609856,"internally_reserved":0,"allocated":12632064,"data_stored":12153420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":928514048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.96199999999999997,"15min":0.94399999999999995},"min":{"1min":0.23699999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.2490000000000001,"5min":6.1879999999999997,"15min":13.929},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.93999999999999995,"15min":0.93899999999999995},"min":{"1min":0.29899999999999999,"5min":0.27600000000000002,"15min":0.20999999999999999},"max":{"1min":1.3149999999999999,"5min":6.8010000000000002,"15min":11.315},"last":1.1020000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":1.3200000000000001,"15min":1.1619999999999999},"min":{"1min":0.32300000000000001,"5min":0.23599999999999999,"15min":0.215},"max":{"1min":1.016,"5min":28.308,"15min":28.308},"last":0.626},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.421,"15min":1.2330000000000001},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":1.232,"5min":28.346,"15min":28.346},"last":1.0429999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":1.0780000000000001,"15min":0.98899999999999999},"min":{"1min":0.312,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.94599999999999995,"5min":5.8620000000000001,"15min":11.430999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.012,"15min":0.95299999999999996},"min":{"1min":0.57499999999999996,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.244,"5min":5.9870000000000001,"15min":7.9710000000000001},"last":0.91100000000000003}]},{"osd":3,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":2.1699999999999999,"15min":1.3600000000000001},"min":{"1min":0.41299999999999998,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":1.135,"5min":93.664000000000001,"15min":93.664000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.91500000000000004,"5min":2.29,"15min":1.3799999999999999},"min":{"1min":0.48699999999999999,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":1.2569999999999999,"5min":93.073999999999998,"15min":93.073999999999998},"last":1.119}]},{"osd":4,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.075,"15min":0.877},"min":{"1min":0.51500000000000001,"5min":0.26500000000000001,"15min":0.249},"max":{"1min":1.0129999999999999,"5min":7.6280000000000001,"15min":7.6280000000000001},"last":1.014},{"interface":"front","average":{"1min":1.085,"5min":1.716,"15min":1.1479999999999999},"min":{"1min":0.68000000000000005,"5min":0.33300000000000002,"15min":0.27200000000000002},"max":{"1min":2.6309999999999998,"5min":27.936,"15min":27.936},"last":1.171}]},{"osd":5,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.47,"15min":1.042},"min":{"1min":0.29599999999999999,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":1.087,"5min":21.126999999999999,"15min":21.126999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.1140000000000001,"15min":0.97199999999999998},"min":{"1min":0.65700000000000003,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":3.8069999999999999,"5min":11.49,"15min":11.49},"last":1.1459999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.556,"15min":1.107},"min":{"1min":0.55200000000000005,"5min":0.311,"15min":0.23599999999999999},"max":{"1min":3.8849999999999998,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":1.075},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.827,"15min":1.2150000000000001},"min":{"1min":0.58299999999999996,"5min":0.45500000000000002,"15min":0.35499999999999998},"max":{"1min":4.3630000000000004,"5min":20.651,"15min":20.651},"last":1.216}]}]},{"osd":6,"up_from":13,"seq":55834575212,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859020,"kb_used_data":5060,"kb_used_omap":0,"kb_used_meta":853952,"kb_avail":93512820,"statfs":{"total":96636764160,"available":95757127680,"internally_reserved":0,"allocated":5181440,"data_stored":4739934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":874446848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.2869999999999999,"15min":0.95199999999999996},"min":{"1min":0.315,"5min":0.185,"15min":0.185},"max":{"1min":5.3490000000000002,"5min":15.974,"15min":15.974},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.331,"15min":0.97399999999999998},"min":{"1min":0.372,"5min":0.218,"15min":0.218},"max":{"1min":1.722,"5min":8.4789999999999992,"15min":8.4789999999999992},"last":0.54400000000000004}]},{"osd":1,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.92500000000000004,"15min":0.88300000000000001},"min":{"1min":0.38900000000000001,"5min":0.221,"15min":0.221},"max":{"1min":1.3919999999999999,"5min":5.556,"15min":7.6360000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.175,"15min":0.98699999999999999},"min":{"1min":0.42099999999999999,"5min":0.254,"15min":0.249},"max":{"1min":1.411,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.81699999999999995}]},{"osd":2,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.5189999999999999,"15min":1.0640000000000001},"min":{"1min":0.318,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.7450000000000001,"5min":12.615,"15min":12.615},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.4610000000000001,"15min":1.109},"min":{"1min":0.38100000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.7669999999999999,"5min":12.667999999999999,"15min":12.667999999999999},"last":0.50900000000000001}]},{"osd":3,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.4770000000000001,"15min":1.2490000000000001},"min":{"1min":0.34300000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.3769999999999998,"5min":18.106999999999999,"15min":38.649999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.107,"5min":1.534,"15min":1.3320000000000001},"min":{"1min":0.39800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.3979999999999997,"5min":18.143000000000001,"15min":38.683},"last":0.67000000000000004}]},{"osd":4,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.7410000000000001,"15min":1.125},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.23799999999999999},"max":{"1min":8.9930000000000003,"5min":12.682,"15min":12.682},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.49,"15min":1.0900000000000001},"min":{"1min":0.36599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.3179999999999996,"5min":10.686,"15min":10.686},"last":0.72099999999999997}]},{"osd":5,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.1559999999999999,"15min":0.96199999999999997},"min":{"1min":0.44500000000000001,"5min":0.28399999999999997,"15min":0.24299999999999999},"max":{"1min":2.1339999999999999,"5min":8.0039999999999996,"15min":8.0039999999999996},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.3660000000000001,"15min":1.069},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.30299999999999999},"max":{"1min":4.0679999999999996,"5min":11.766999999999999,"15min":11.766999999999999},"last":0.69799999999999995}]},{"osd":7,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2969999999999999,"15min":1.0369999999999999},"min":{"1min":0.40999999999999998,"5min":0.251,"15min":0.251},"max":{"1min":5.4219999999999997,"5min":8.2720000000000002,"15min":12.926},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.341,"5min":1.3100000000000001,"15min":1.05},"min":{"1min":0.504,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.4420000000000002,"5min":6.1100000000000003,"15min":7.6619999999999999},"last":0.84899999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575212,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":958872,"kb_used_data":9168,"kb_used_omap":0,"kb_used_meta":949696,"kb_avail":93412968,"statfs":{"total":96636764160,"available":95654879232,"internally_reserved":0,"allocated":9388032,"data_stored":8934420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":972488704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:48 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.2869999999999999,"15min":1.038},"min":{"1min":0.371,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.5569999999999999,"5min":14.837999999999999,"15min":14.837999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.86299999999999999,"15min":0.96799999999999997},"min":{"1min":0.27000000000000002,"5min":0.20699999999999999,"15min":0.20100000000000001},"max":{"1min":3.5779999999999998,"5min":5.8289999999999997,"15min":7.423},"last":1.113}]},{"osd":1,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.415,"15min":1.085},"min":{"1min":0.312,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":2.0419999999999998,"5min":12.141,"15min":12.141},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.4199999999999999,"15min":1.0860000000000001},"min":{"1min":0.314,"5min":0.26400000000000001,"15min":0.252},"max":{"1min":3.714,"5min":24.648,"15min":24.648},"last":0.98799999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.46,"15min":1.2390000000000001},"min":{"1min":0.376,"5min":0.25,"15min":0.25},"max":{"1min":1.925,"5min":20.760999999999999,"15min":20.760999999999999},"last":0.873},{"interface":"front","average":{"1min":0.76700000000000002,"5min":1.274,"15min":1.0700000000000001},"min":{"1min":0.39800000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.1779999999999999,"5min":20.552,"15min":20.552},"last":1.075}]},{"osd":3,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.429,"15min":1.2210000000000001},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.28000000000000003},"max":{"1min":3.6139999999999999,"5min":12.013,"15min":12.013},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.359,"15min":1.246},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.5779999999999998,"5min":12.504,"15min":12.805},"last":0.91100000000000003}]},{"osd":4,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.708,"15min":1.202},"min":{"1min":0.47099999999999997,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":2.3290000000000002,"5min":12.728,"15min":12.728},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.532,"15min":1.1419999999999999},"min":{"1min":0.39200000000000002,"5min":0.27600000000000002,"15min":0.27200000000000002},"max":{"1min":3.6699999999999999,"5min":10.943,"15min":10.943},"last":1.2130000000000001}]},{"osd":6,"last update":"Wed Oct 25 00:21:58 2023","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.369,"15min":1.0569999999999999},"min":{"1min":0.42799999999999999,"5min":0.33500000000000002,"15min":0.26400000000000001},"max":{"1min":9.6489999999999991,"5min":17.936,"15min":17.936},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.96,"15min":1.224},"min":{"1min":0.312,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":12.042999999999999,"5min":14.766999999999999,"15min":14.766999999999999},"last":1.0229999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.75,"15min":1.2470000000000001},"min":{"1min":0.46500000000000002,"5min":0.33200000000000002,"15min":0.29499999999999998},"max":{"1min":1.2869999999999999,"5min":28.178999999999998,"15min":28.178999999999998},"last":1.181},{"interface":"front","average":{"1min":1.0740000000000001,"5min":2.133,"15min":1.357},"min":{"1min":0.59699999999999998,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":3.6429999999999998,"5min":28.222999999999999,"15min":28.222999999999999},"last":1.157}]}]},{"osd":4,"up_from":13,"seq":55834575211,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946308,"kb_used_data":19132,"kb_used_omap":0,"kb_used_meta":927168,"kb_avail":93425532,"statfs":{"total":96636764160,"available":95667744768,"internally_reserved":0,"allocated":19591168,"data_stored":19123312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":949420032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.3320000000000001,"15min":0.98099999999999998},"min":{"1min":0.317,"5min":0.19700000000000001,"15min":0.187},"max":{"1min":11.629,"5min":20.922999999999998,"15min":20.922999999999998},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.101,"5min":1.157,"15min":0.98199999999999998},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.217},"max":{"1min":7.0830000000000002,"5min":20.532,"15min":20.532},"last":0.54700000000000004}]},{"osd":1,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":1.2410000000000001,"15min":0.97699999999999998},"min":{"1min":0.33800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":2.6110000000000002,"5min":20.914000000000001,"15min":20.914000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.3779999999999999,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":6.1890000000000001,"5min":23.265999999999998,"15min":23.265999999999998},"last":0.72899999999999998}]},{"osd":2,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.528,"15min":1.1080000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.1550000000000002,"5min":20.774000000000001,"15min":20.774000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.802,"15min":1.252},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":7.2750000000000004,"5min":20.817,"15min":20.817},"last":0.82499999999999996}]},{"osd":3,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.623,"15min":1.1699999999999999},"min":{"1min":0.44800000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":7.3620000000000001,"5min":24.613,"15min":24.613},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.28,"5min":1.3400000000000001,"15min":1.145},"min":{"1min":0.29299999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.2009999999999996,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.874}]},{"osd":5,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.1679999999999999,"15min":0.93300000000000005},"min":{"1min":0.5,"5min":0.28599999999999998,"15min":0.23999999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.079,"15min":0.95099999999999996},"min":{"1min":0.57899999999999996,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.7549999999999999,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.81200000000000006}]},{"osd":6,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.367,"15min":1.0860000000000001},"min":{"1min":0.627,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":2.794,"5min":6.5220000000000002,"15min":6.5220000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.357,"5min":1.1040000000000001,"15min":0.97199999999999998},"min":{"1min":0.38700000000000001,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"last":0.94699999999999995}]},{"osd":7,"last update":"Wed Oct 25 00:22:52 2023","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.3520000000000001,"15min":1.087},"min":{"1min":0.56499999999999995,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.4020000000000001,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.7290000000000001,"5min":1.5649999999999999,"15min":1.1579999999999999},"min":{"1min":0.58899999999999997,"5min":0.309,"15min":0.309},"max":{"1min":7.444,"5min":7.444,"15min":7.444},"last":0.98799999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575212,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160748,"kb_used_data":9252,"kb_used_omap":0,"kb_used_meta":1151488,"kb_avail":93211092,"statfs":{"total":96636764160,"available":95448158208,"internally_reserved":0,"allocated":9474048,"data_stored":9018997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1179123712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.67500000000000004,"15min":0.78300000000000003},"min":{"1min":0.18099999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.72599999999999998,"5min":12.215999999999999,"15min":13.327999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.55700000000000005,"15min":0.72899999999999998},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.95399999999999996,"5min":2.2770000000000001,"15min":5.4580000000000002},"last":0.72099999999999997}]},{"osd":1,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.86699999999999999,"15min":0.85599999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19900000000000001},"max":{"1min":0.96799999999999997,"5min":11.65,"15min":13.831},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.71099999999999997,"15min":0.76100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.0940000000000001,"5min":4.2130000000000001,"15min":4.4500000000000002},"last":0.875}]},{"osd":2,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.82199999999999995,"15min":0.87},"min":{"1min":0.24099999999999999,"5min":0.223,"15min":0.221},"max":{"1min":1.0129999999999999,"5min":8.2520000000000007,"15min":8.2520000000000007},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.92000000000000004,"15min":0.90300000000000002},"min":{"1min":0.26700000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0349999999999999,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.78200000000000003}]},{"osd":4,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.7,"15min":1.204},"min":{"1min":0.57599999999999996,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.208,"5min":39.070999999999998,"15min":39.070999999999998},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":1.8899999999999999,"15min":1.248},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.25700000000000001},"max":{"1min":1.1080000000000001,"5min":55.104999999999997,"15min":55.104999999999997},"last":1.0369999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.556,"15min":1.1160000000000001},"min":{"1min":0.33300000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1759999999999999,"5min":31.481999999999999,"15min":31.481999999999999},"last":1.175},{"interface":"front","average":{"1min":0.997,"5min":1.617,"15min":1.2010000000000001},"min":{"1min":0.40699999999999997,"5min":0.35299999999999998,"15min":0.32500000000000001},"max":{"1min":1.4470000000000001,"5min":31.512,"15min":31.512},"last":1.446}]},{"osd":6,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.6180000000000001,"15min":1.2370000000000001},"min":{"1min":0.33600000000000002,"5min":0.311,"15min":0.27800000000000002},"max":{"1min":15.824,"5min":25.809999999999999,"15min":25.809999999999999},"last":1.254},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.363,"15min":1.171},"min":{"1min":0.41399999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1910000000000001,"5min":24.036999999999999,"15min":24.036999999999999},"last":1.4079999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.5069999999999999,"15min":1.2250000000000001},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3200000000000001,"5min":27.581,"15min":27.581},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.413,"15min":1.2430000000000001},"min":{"1min":0.503,"5min":0.374,"15min":0.374},"max":{"1min":1.3819999999999999,"5min":24.332000000000001,"15min":24.332000000000001},"last":1.3759999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575212,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946060,"kb_used_data":5060,"kb_used_omap":0,"kb_used_meta":940992,"kb_avail":93425780,"statfs":{"total":96636764160,"available":95667998720,"internally_reserved":0,"allocated":5181440,"data_stored":4739422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":963575808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.96699999999999997,"15min":0.97999999999999998},"min":{"1min":0.185,"5min":0.16,"15min":0.16},"max":{"1min":2.3010000000000002,"5min":5.9710000000000001,"15min":29.344000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83699999999999997,"15min":0.83499999999999996},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.7000000000000002,"5min":6.4290000000000003,"15min":10.755000000000001},"last":0.63100000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.98899999999999999,"15min":0.92900000000000005},"min":{"1min":0.24199999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":3.0409999999999999,"5min":17.192,"15min":18.617999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.86699999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.649,"5min":5.8890000000000002,"15min":15.064},"last":0.75700000000000001}]},{"osd":3,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.014,"15min":0.89700000000000002},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.9220000000000002,"5min":13.034000000000001,"15min":13.034000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.1930000000000001,"15min":0.99099999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.9929999999999999,"5min":9.9890000000000008,"15min":18.547999999999998},"last":0.84199999999999997}]},{"osd":4,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.4670000000000001,"15min":1.1850000000000001},"min":{"1min":0.26900000000000002,"5min":0.252,"15min":0.252},"max":{"1min":5.1070000000000002,"5min":22.997,"15min":33.378999999999998},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.194,"5min":1.5189999999999999,"15min":1.278},"min":{"1min":0.309,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.2290000000000001,"5min":23.030999999999999,"15min":33.558999999999997},"last":1.105}]},{"osd":5,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.887,"15min":1.464},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29099999999999998},"max":{"1min":3.1400000000000001,"5min":25.183,"15min":34.545999999999999},"last":1.089},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.2450000000000001,"15min":1.1679999999999999},"min":{"1min":0.33600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.4870000000000001,"5min":11.685,"15min":36.104999999999997},"last":0.876}]},{"osd":6,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.1339999999999999,"15min":1.103},"min":{"1min":0.501,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.482,"5min":9.7149999999999999,"15min":33.661999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.1930000000000001,"15min":1.1910000000000001},"min":{"1min":0.53100000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.464,"5min":9.6799999999999997,"15min":34.043999999999997},"last":1.1539999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3009999999999999,"15min":1.226},"min":{"1min":0.41499999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":4.8369999999999997,"5min":11.503,"15min":33.307000000000002},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.341,"15min":1.2889999999999999},"min":{"1min":0.44900000000000001,"5min":0.371,"15min":0.371},"max":{"1min":4.8620000000000001,"5min":11.468999999999999,"15min":33.344999999999999},"last":1.1930000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575213,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1070508,"kb_used_data":4068,"kb_used_omap":0,"kb_used_meta":1066432,"kb_avail":93301332,"statfs":{"total":96636764160,"available":95540563968,"internally_reserved":0,"allocated":4165632,"data_stored":3722683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1092026368},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.98099999999999998,"15min":0.81299999999999994},"min":{"1min":0.249,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.85699999999999998,"5min":21.334,"15min":21.334},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.96899999999999997,"15min":0.84199999999999997},"min":{"1min":0.38800000000000001,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":1.0449999999999999,"5min":21.276,"15min":21.276},"last":1.0449999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.93400000000000005,"15min":0.80000000000000004},"min":{"1min":0.34999999999999998,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.92900000000000005,"5min":11.420999999999999,"15min":11.420999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.878,"15min":0.83099999999999996},"min":{"1min":0.27100000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.002,"5min":11.648,"15min":11.648},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.022,"15min":0.91100000000000003},"min":{"1min":0.318,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":1.1140000000000001,"5min":9.0449999999999999,"15min":14.346},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.84899999999999998,"5min":1.1040000000000001,"15min":1.01},"min":{"1min":0.35099999999999998,"5min":0.26300000000000001,"15min":0.248},"max":{"1min":1.1859999999999999,"5min":9.0800000000000001,"15min":14.379},"last":0.97299999999999998}]},{"osd":4,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.2569999999999999,"15min":1.0600000000000001},"min":{"1min":0.50600000000000001,"5min":0.28000000000000003,"15min":0.27200000000000002},"max":{"1min":1.353,"5min":10.582000000000001,"15min":10.582000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.2210000000000001,"15min":1.0449999999999999},"min":{"1min":0.39600000000000002,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":1.272,"5min":11.173,"15min":11.173},"last":1.079}]},{"osd":5,"last update":"Wed Oct 25 00:21:51 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.244,"15min":1.054},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.0129999999999999,"5min":11.577999999999999,"15min":11.577999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.0760000000000001,"15min":0.98099999999999998},"min":{"1min":0.29899999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.3220000000000001,"5min":10.018000000000001,"15min":10.018000000000001},"last":0.98699999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0680000000000001,"15min":1.0229999999999999},"min":{"1min":0.55800000000000005,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.357,"5min":9.2989999999999995,"15min":11.845000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.141,"15min":1.1140000000000001},"min":{"1min":0.62,"5min":0.379,"15min":0.33800000000000002},"max":{"1min":1.4870000000000001,"5min":9.3350000000000009,"15min":11.919},"last":1.012}]},{"osd":7,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.387,"15min":1.1200000000000001},"min":{"1min":0.57499999999999996,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":1.407,"5min":9.407,"15min":11.881},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.22,"15min":1.1359999999999999},"min":{"1min":0.54700000000000004,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3080000000000001,"5min":10.478,"15min":10.478},"last":1.3080000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575213,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":966376,"kb_used_data":22112,"kb_used_omap":0,"kb_used_meta":944256,"kb_avail":93405464,"statfs":{"total":96636764160,"available":95647195136,"internally_reserved":0,"allocated":22642688,"data_stored":22192198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":966918144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.1719999999999999,"15min":0.99099999999999999},"min":{"1min":0.23000000000000001,"5min":0.18099999999999999,"15min":0.16900000000000001},"max":{"1min":1.6779999999999999,"5min":30.065999999999999,"15min":30.065999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.94799999999999995,"15min":0.99199999999999999},"min":{"1min":0.247,"5min":0.184,"15min":0.182},"max":{"1min":0.98999999999999999,"5min":8.8529999999999998,"15min":19.757999999999999},"last":0.76700000000000002}]},{"osd":2,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.93500000000000005,"15min":0.78600000000000003},"min":{"1min":0.42199999999999999,"5min":0.22900000000000001,"15min":0.20399999999999999},"max":{"1min":1.806,"5min":15.324999999999999,"15min":15.324999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.169,"15min":0.95499999999999996},"min":{"1min":0.42899999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.8440000000000001,"5min":30.283999999999999,"15min":30.283999999999999},"last":0.626}]},{"osd":3,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":1.4490000000000001,"15min":1.1020000000000001},"min":{"1min":0.40000000000000002,"5min":0.191,"15min":0.191},"max":{"1min":2.3420000000000001,"5min":22.966999999999999,"15min":22.966999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.2529999999999999,"15min":1.0609999999999999},"min":{"1min":0.32800000000000001,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":1.915,"5min":19.460999999999999,"15min":19.460999999999999},"last":1.0489999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":1.056,"15min":1.121},"min":{"1min":0.35299999999999998,"5min":0.318,"15min":0.26600000000000001},"max":{"1min":1.157,"5min":4.4710000000000001,"15min":8.3670000000000009},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.3799999999999999,"15min":1.222},"min":{"1min":0.59799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.3719999999999999,"5min":13.023999999999999,"15min":13.023999999999999},"last":0.67800000000000005}]},{"osd":5,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.103,"15min":1.0169999999999999},"min":{"1min":0.60999999999999999,"5min":0.30399999999999999,"15min":0.29099999999999998},"max":{"1min":2.391,"5min":6.0149999999999997,"15min":10.683999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1799999999999999,"15min":0.98799999999999999},"min":{"1min":0.52800000000000002,"5min":0.33700000000000002,"15min":0.29899999999999999},"max":{"1min":4.8860000000000001,"5min":15.436999999999999,"15min":15.436999999999999},"last":0.98299999999999998}]},{"osd":6,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.266,"15min":1.125},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.28100000000000003},"max":{"1min":1.3069999999999999,"5min":8.7260000000000009,"15min":8.7260000000000009},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.3700000000000001,"15min":1.2829999999999999},"min":{"1min":0.56299999999999994,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":7.8970000000000002,"5min":15.375,"15min":15.375},"last":0.86899999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.3380000000000001,"15min":1.151},"min":{"1min":0.64800000000000002,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":5.4269999999999996,"5min":6.0640000000000001,"15min":10.81},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.206,"15min":1.256},"min":{"1min":0.45600000000000002,"5min":0.313,"15min":0.313},"max":{"1min":1.8660000000000001,"5min":9.3699999999999992,"15min":11.417},"last":0.97199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12673024,"data_stored":12668140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3198976,"data_stored":3198201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11689984,"data_stored":11672183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4195192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":8388642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4279555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6918144,"data_stored":6907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6836224,"data_stored":6822812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-25T00:22:58.283 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.284 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.284 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.285 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.285 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.285 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.286 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.286 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.286 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.287 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.287 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.287 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.288 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.288 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.288 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.289 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.289 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.289 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.289 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.290 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.290 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.290 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.291 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.291 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.291 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.292 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.292 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.292 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.293 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.293 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.293 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.293 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.294 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.294 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.294 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.295 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.295 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.295 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.296 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.296 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.296 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.297 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.297 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.297 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.297 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.298 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.298 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.298 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.299 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.299 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.299 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.300 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.300 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.300 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.300 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.301 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.301 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.302 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.302 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.303 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.303 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.304 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.304 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.305 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.305 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.306 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.306 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-24T23:52:51.349804+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=51, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.307 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.307 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.307 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.308 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.308 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.308 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.309 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.309 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.309 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.309 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.310 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.310 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.310 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.311 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.311 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.311 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.312 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.312 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.312 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.312 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.313 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.313 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.314 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.314 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.314 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.315 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.315 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.315 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.316 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.316 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.316 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.317 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.317 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.317 INFO:tasks.ceph:pgid 6.22 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.318 INFO:tasks.ceph:pgid 4.20 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.318 INFO:tasks.ceph:pgid 6.23 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.318 INFO:tasks.ceph:pgid 6.20 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.319 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.319 INFO:tasks.ceph:pgid 6.21 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.319 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.319 INFO:tasks.ceph:pgid 6.26 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.320 INFO:tasks.ceph:pgid 6.27 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.320 INFO:tasks.ceph:pgid 6.24 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.321 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.321 INFO:tasks.ceph:pgid 6.25 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.321 INFO:tasks.ceph:pgid 6.2a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.322 INFO:tasks.ceph:pgid 4.28 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.322 INFO:tasks.ceph:pgid 6.2b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.323 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.323 INFO:tasks.ceph:pgid 6.28 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.323 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.324 INFO:tasks.ceph:pgid 6.29 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.324 INFO:tasks.ceph:pgid 4.2b last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.325 INFO:tasks.ceph:pgid 6.2e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.325 INFO:tasks.ceph:pgid 4.2c last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.326 INFO:tasks.ceph:pgid 6.2f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.326 INFO:tasks.ceph:pgid 4.2d last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.327 INFO:tasks.ceph:pgid 6.2c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.327 INFO:tasks.ceph:pgid 4.2e last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.327 INFO:tasks.ceph:pgid 6.2d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.328 INFO:tasks.ceph:pgid 4.2f last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.328 INFO:tasks.ceph:pgid 6.32 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.329 INFO:tasks.ceph:pgid 4.30 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.329 INFO:tasks.ceph:pgid 6.33 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.330 INFO:tasks.ceph:pgid 6.30 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.330 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.330 INFO:tasks.ceph:pgid 6.31 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.331 INFO:tasks.ceph:pgid 4.33 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.331 INFO:tasks.ceph:pgid 6.36 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.332 INFO:tasks.ceph:pgid 4.34 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.332 INFO:tasks.ceph:pgid 6.37 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.332 INFO:tasks.ceph:pgid 4.35 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.332 INFO:tasks.ceph:pgid 6.34 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.333 INFO:tasks.ceph:pgid 4.36 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.333 INFO:tasks.ceph:pgid 6.35 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.333 INFO:tasks.ceph:pgid 4.37 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.334 INFO:tasks.ceph:pgid 6.3a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.334 INFO:tasks.ceph:pgid 4.38 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.334 INFO:tasks.ceph:pgid 6.3b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.334 INFO:tasks.ceph:pgid 6.38 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.335 INFO:tasks.ceph:pgid 4.3a last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.335 INFO:tasks.ceph:pgid 6.39 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.335 INFO:tasks.ceph:pgid 6.3e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.336 INFO:tasks.ceph:pgid 4.3c last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.336 INFO:tasks.ceph:pgid 6.3f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.336 INFO:tasks.ceph:pgid 6.3c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.337 INFO:tasks.ceph:pgid 4.3e last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.337 INFO:tasks.ceph:pgid 6.3d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.337 INFO:tasks.ceph:pgid 4.3f last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:22:58.337 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-25T00:23:01.741 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2023-10-25T00:23:01.742 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-25T00:23:01.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-25T00:23:01.745 INFO:tasks.ceph.mds.c:Started 2023-10-25T00:23:01.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:23:01.815 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:23:01.870 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.867+0000 7f4c124b7640 1 Processor -- start 2023-10-25T00:23:01.873 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.867+0000 7f4c124b7640 1 -- start start 2023-10-25T00:23:01.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c124b7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f4c0c0c3e00 0x7f4c0c0c4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:01.875 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c124b7640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c0c59d0 con 0x7f4c0c155ea0 2023-10-25T00:23:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c124b7640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c0c0c2990 con 0x7f4c0c06ec90 2023-10-25T00:23:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c124b7640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c0c0c2ec0 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c0bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f4c0c155ea0 0x7f4c0c156280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:56306/0 (socket says 172.21.15.107:56306) 2023-10-25T00:23:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c0bfff640 1 -- 172.21.15.107:0/2737728437 learned_addr learned my addr 172.21.15.107:0/2737728437 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:23:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c10a54640 1 --2- 172.21.15.107:0/2737728437 >> v2:172.21.15.107:3300/0 conn(0x7f4c0c0c3e00 0x7f4c0c0c4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c0affd640 1 -- 172.21.15.107:0/2737728437 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409579351 0 0) 0x7f4c0c0c59d0 con 0x7f4c0c155ea0 2023-10-25T00:23:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c0affd640 1 -- 172.21.15.107:0/2737728437 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf4003530 con 0x7f4c0c155ea0 2023-10-25T00:23:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c10a54640 1 -- 172.21.15.107:0/2737728437 >> v1:172.21.15.110:6789/0 conn(0x7f4c0c06ec90 legacy=0x7f4c0c14c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c0affd640 1 -- 172.21.15.107:0/2737728437 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1343921963 0 0) 0x7f4bf4003530 con 0x7f4c0c155ea0 2023-10-25T00:23:01.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c10a54640 1 -- 172.21.15.107:0/2737728437 >> v1:172.21.15.107:6789/0 conn(0x7f4c0c155ea0 legacy=0x7f4c0c156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c10a54640 1 -- 172.21.15.107:0/2737728437 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c06d500 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c10a54640 1 --2- 172.21.15.107:0/2737728437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0c0c3e00 0x7f4c0c0c4220 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009d90 tx=0x7f4c0401c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e56cc378f98c57f server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c0affd640 1 -- 172.21.15.107:0/2737728437 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c00001680 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c0affd640 1 -- 172.21.15.107:0/2737728437 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c040a64b0 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.871+0000 7f4c0affd640 1 -- 172.21.15.107:0/2737728437 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c040a6840 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 -- 172.21.15.107:0/2737728437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0c0c3e00 msgr2=0x7f4c0c0c4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 --2- 172.21.15.107:0/2737728437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0c0c3e00 0x7f4c0c0c4220 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009d90 tx=0x7f4c0401c750 comp rx=0 tx=0).stop 2023-10-25T00:23:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 -- 172.21.15.107:0/2737728437 shutdown_connections 2023-10-25T00:23:01.879 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 --2- 172.21.15.107:0/2737728437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0c0c3e00 0x7f4c0c0c4220 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 -- 172.21.15.107:0/2737728437 >> 172.21.15.107:0/2737728437 conn(0x7f4c0c0c4a90 msgr2=0x7f4c0c0c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:23:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 -- 172.21.15.107:0/2737728437 shutdown_connections 2023-10-25T00:23:01.880 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 -- 172.21.15.107:0/2737728437 wait complete. 2023-10-25T00:23:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 Processor -- start 2023-10-25T00:23:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 -- start start 2023-10-25T00:23:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c0c06ec90 0x7f4c0c13db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0c0c3e00 0x7f4c0c13bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:01.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c0c155ea0 0x7f4c0c13c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c1461c0 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c13fad0 con 0x7f4c0c06ec90 2023-10-25T00:23:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c0c1456e0 con 0x7f4c0c155ea0 2023-10-25T00:23:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c0b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0c0c3e00 0x7f4c0c13bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:01.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c0b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0c0c3e00 0x7f4c0c13bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:52930/0 (socket says 172.21.15.107:52930) 2023-10-25T00:23:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c0b7fe640 1 -- 172.21.15.107:0/356131186 learned_addr learned my addr 172.21.15.107:0/356131186 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:23:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c0bfff640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c0c06ec90 0x7f4c0c13db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c10a54640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c0c155ea0 0x7f4c0c13c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c0b7fe640 1 -- 172.21.15.107:0/356131186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c0c155ea0 msgr2=0x7f4c0c13c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c0b7fe640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c0c155ea0 0x7f4c0c13c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c0b7fe640 1 -- 172.21.15.107:0/356131186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c0c06ec90 msgr2=0x7f4c0c13db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c0b7fe640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c0c06ec90 0x7f4c0c13db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c0b7fe640 1 -- 172.21.15.107:0/356131186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c04009a70 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c0b7fe640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0c0c3e00 0x7f4c0c13bb30 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00c3c0 tx=0x7f4bfc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:01.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c08ff9640 1 -- 172.21.15.107:0/356131186 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc0052d0 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c08ff9640 1 -- 172.21.15.107:0/356131186 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc005430 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 -- 172.21.15.107:0/356131186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c0c1459c0 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c08ff9640 1 -- 172.21.15.107:0/356131186 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bfc003840 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.875+0000 7f4c124b7640 1 -- 172.21.15.107:0/356131186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c0c145ca0 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.879+0000 7f4c124b7640 1 -- 172.21.15.107:0/356131186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c0c13fcd0 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.879+0000 7f4c08ff9640 1 -- 172.21.15.107:0/356131186 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f4bfc0039a0 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.879+0000 7f4c08ff9640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4bf8065e90 0x7f4bf8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.879+0000 7f4c08ff9640 1 -- 172.21.15.107:0/356131186 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f4bfc083f90 con 0x7f4c0c0c3e00 2023-10-25T00:23:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.879+0000 7f4c0bfff640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4bf8065e90 0x7f4bf8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:01.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.879+0000 7f4c0bfff640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4bf8065e90 0x7f4bf8068350 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f4c00000f70 tx=0x7f4c00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:01.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:01.883+0000 7f4c08ff9640 1 -- 172.21.15.107:0/356131186 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f4bfc0b5040 con 0x7f4c0c0c3e00 2023-10-25T00:23:02.074 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.071+0000 7f4c124b7640 1 -- 172.21.15.107:0/356131186 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c0c0746e0 con 0x7f4c0c0c3e00 2023-10-25T00:23:02.075 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c08ff9640 1 -- 172.21.15.107:0/356131186 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+5008 (secure 0 0 0) 0x7f4bfc017090 con 0x7f4c0c0c3e00 2023-10-25T00:23:02.075 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:23:02.075 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:23:02.076 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 234 2023-10-25T00:23:02.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 -- 172.21.15.107:0/356131186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4bf8065e90 msgr2=0x7f4bf8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:02.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4bf8065e90 0x7f4bf8068350 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f4c00000f70 tx=0x7f4c00009040 comp rx=0 tx=0).stop 2023-10-25T00:23:02.078 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 -- 172.21.15.107:0/356131186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0c0c3e00 msgr2=0x7f4c0c13bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0c0c3e00 0x7f4c0c13bb30 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc00c3c0 tx=0x7f4bfc00c880 comp rx=0 tx=0).stop 2023-10-25T00:23:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 -- 172.21.15.107:0/356131186 shutdown_connections 2023-10-25T00:23:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4bf8065e90 0x7f4bf8068350 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:02.079 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4c0c155ea0 0x7f4c0c13c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c0c0c3e00 0x7f4c0c13bb30 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 --2- 172.21.15.107:0/356131186 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c0c06ec90 0x7f4c0c13db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 -- 172.21.15.107:0/356131186 >> 172.21.15.107:0/356131186 conn(0x7f4c0c0c4a90 msgr2=0x7f4c0c0ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:23:02.080 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 -- 172.21.15.107:0/356131186 shutdown_connections 2023-10-25T00:23:02.081 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:02.075+0000 7f4c124b7640 1 -- 172.21.15.107:0/356131186 wait complete. 2023-10-25T00:23:02.100 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-25T00:23:04.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:23:04.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f2016076640 1 Processor -- start 2023-10-25T00:23:04.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f2016076640 1 -- start start 2023-10-25T00:23:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f2016076640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f201013dc20 0x7f20100c18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:04.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f2016076640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f20100720c0 0x7f201006df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f2016076640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2010154860 con 0x7f201006e470 2023-10-25T00:23:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f2016076640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20100c1f10 con 0x7f201013dc20 2023-10-25T00:23:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f2016076640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2010154b20 con 0x7f20100720c0 2023-10-25T00:23:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f200f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f201013dc20 0x7f20100c18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:04.237 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f200f7fe640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f201013dc20 0x7f20100c18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41730/0 (socket says 172.21.15.107:41730) 2023-10-25T00:23:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f200f7fe640 1 -- 172.21.15.107:0/1874474623 learned_addr learned my addr 172.21.15.107:0/1874474623 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:23:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f2007fff640 1 --2- 172.21.15.107:0/1874474623 >> v2:172.21.15.110:3300/0 conn(0x7f20100720c0 0x7f201006df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f200f7fe640 1 -- 172.21.15.107:0/1874474623 >> v2:172.21.15.110:3300/0 conn(0x7f20100720c0 msgr2=0x7f201006df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:23:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f200f7fe640 1 --2- 172.21.15.107:0/1874474623 >> v2:172.21.15.110:3300/0 conn(0x7f20100720c0 0x7f201006df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f200f7fe640 1 -- 172.21.15.107:0/1874474623 >> v1:172.21.15.110:6789/0 conn(0x7f201006e470 legacy=0x7f2010157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:23:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.235+0000 7f200f7fe640 1 -- 172.21.15.107:0/1874474623 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2010154e00 con 0x7f201013dc20 2023-10-25T00:23:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200f7fe640 1 --2- 172.21.15.107:0/1874474623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f201013dc20 0x7f20100c18a0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f20100c06a0 tx=0x7f200001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2483f0d1125f382b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200effd640 1 -- 172.21.15.107:0/1874474623 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20000ab600 con 0x7f201013dc20 2023-10-25T00:23:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200effd640 1 -- 172.21.15.107:0/1874474623 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20000a7050 con 0x7f201013dc20 2023-10-25T00:23:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200effd640 1 -- 172.21.15.107:0/1874474623 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20000ab980 con 0x7f201013dc20 2023-10-25T00:23:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 -- 172.21.15.107:0/1874474623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f201013dc20 msgr2=0x7f20100c18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 --2- 172.21.15.107:0/1874474623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f201013dc20 0x7f20100c18a0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f20100c06a0 tx=0x7f200001c3a0 comp rx=0 tx=0).stop 2023-10-25T00:23:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 -- 172.21.15.107:0/1874474623 shutdown_connections 2023-10-25T00:23:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 --2- 172.21.15.107:0/1874474623 >> v2:172.21.15.110:3300/0 conn(0x7f20100720c0 0x7f201006df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 --2- 172.21.15.107:0/1874474623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f201013dc20 0x7f20100c18a0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 -- 172.21.15.107:0/1874474623 >> 172.21.15.107:0/1874474623 conn(0x7f20100c39a0 msgr2=0x7f20100c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:23:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 -- 172.21.15.107:0/1874474623 shutdown_connections 2023-10-25T00:23:04.242 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 -- 172.21.15.107:0/1874474623 wait complete. 2023-10-25T00:23:04.244 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 Processor -- start 2023-10-25T00:23:04.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 -- start start 2023-10-25T00:23:04.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f201006e470 0x7f20100aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:04.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20100720c0 0x7f20101564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:04.245 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f201013dc20 0x7f201013b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20101573b0 con 0x7f201013dc20 2023-10-25T00:23:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201014fa70 con 0x7f201006e470 2023-10-25T00:23:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20100ac3c0 con 0x7f20100720c0 2023-10-25T00:23:04.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f201013dc20 0x7f201013b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:04.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f201013dc20 0x7f201013b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41736/0 (socket says 172.21.15.107:41736) 2023-10-25T00:23:04.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200ffff640 1 -- 172.21.15.107:0/1889471938 learned_addr learned my addr 172.21.15.107:0/1889471938 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:23:04.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200f7fe640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f201006e470 0x7f20100aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:04.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2007fff640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20100720c0 0x7f20101564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:04.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200ffff640 1 -- 172.21.15.107:0/1889471938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20100720c0 msgr2=0x7f20101564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:04.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200ffff640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20100720c0 0x7f20101564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:04.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200ffff640 1 -- 172.21.15.107:0/1889471938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f201006e470 msgr2=0x7f20100aaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:04.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200ffff640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f201006e470 0x7f20100aaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:04.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200ffff640 1 -- 172.21.15.107:0/1889471938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20000094d0 con 0x7f201013dc20 2023-10-25T00:23:04.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200ffff640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f201013dc20 0x7f201013b3c0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f200800c780 tx=0x7f200800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:04.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200cff9640 1 -- 172.21.15.107:0/1889471938 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2008004b70 con 0x7f201013dc20 2023-10-25T00:23:04.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200cff9640 1 -- 172.21.15.107:0/1889471938 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200800ce90 con 0x7f201013dc20 2023-10-25T00:23:04.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 -- 172.21.15.107:0/1889471938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20100ab3f0 con 0x7f201013dc20 2023-10-25T00:23:04.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f200cff9640 1 -- 172.21.15.107:0/1889471938 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200800a850 con 0x7f201013dc20 2023-10-25T00:23:04.249 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.239+0000 7f2016076640 1 -- 172.21.15.107:0/1889471938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20100ab900 con 0x7f201013dc20 2023-10-25T00:23:04.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.243+0000 7f2016076640 1 -- 172.21.15.107:0/1889471938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20101574f0 con 0x7f201013dc20 2023-10-25T00:23:04.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.243+0000 7f200cff9640 1 -- 172.21.15.107:0/1889471938 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f200800a9b0 con 0x7f201013dc20 2023-10-25T00:23:04.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.243+0000 7f200cff9640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ff0065e60 0x7f1ff0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:04.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.243+0000 7f200f7fe640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ff0065e60 0x7f1ff0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:04.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.243+0000 7f200cff9640 1 -- 172.21.15.107:0/1889471938 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f2008084c90 con 0x7f201013dc20 2023-10-25T00:23:04.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.243+0000 7f200f7fe640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ff0065e60 0x7f1ff0068320 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f20000ad000 tx=0x7f2000008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:04.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.247+0000 7f200cff9640 1 -- 172.21.15.107:0/1889471938 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f20101574f0 con 0x7f201013dc20 2023-10-25T00:23:04.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.435+0000 7f2016076640 1 -- 172.21.15.107:0/1889471938 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20101574f0 con 0x7f201013dc20 2023-10-25T00:23:04.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.435+0000 7f200cff9640 1 -- 172.21.15.107:0/1889471938 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5769 (secure 0 0 0) 0x7f20101574f0 con 0x7f201013dc20 2023-10-25T00:23:04.437 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:23:04.438 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:23:04.438 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 235 2023-10-25T00:23:04.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 -- 172.21.15.107:0/1889471938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ff0065e60 msgr2=0x7f1ff0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:04.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ff0065e60 0x7f1ff0068320 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f20000ad000 tx=0x7f2000008040 comp rx=0 tx=0).stop 2023-10-25T00:23:04.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 -- 172.21.15.107:0/1889471938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f201013dc20 msgr2=0x7f201013b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:04.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f201013dc20 0x7f201013b3c0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f200800c780 tx=0x7f200800cc40 comp rx=0 tx=0).stop 2023-10-25T00:23:04.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 -- 172.21.15.107:0/1889471938 shutdown_connections 2023-10-25T00:23:04.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f201013dc20 0x7f201013b3c0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:04.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1ff0065e60 0x7f1ff0068320 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:04.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f20100720c0 0x7f20101564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:04.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 --2- 172.21.15.107:0/1889471938 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f201006e470 0x7f20100aaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:04.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 -- 172.21.15.107:0/1889471938 >> 172.21.15.107:0/1889471938 conn(0x7f20100c39a0 msgr2=0x7f2010153450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:23:04.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 -- 172.21.15.107:0/1889471938 shutdown_connections 2023-10-25T00:23:04.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:04.439+0000 7f2016076640 1 -- 172.21.15.107:0/1889471938 wait complete. 2023-10-25T00:23:04.457 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2023-10-25T00:23:04.457 INFO:tasks.mds_thrash.fs.[a]:waiting for 71 secs before thrashing 2023-10-25T00:23:18.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-25T00:23:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.467+0000 7f784d823640 1 Processor -- start 2023-10-25T00:23:18.471 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.467+0000 7f784d823640 1 -- start start 2023-10-25T00:23:18.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f784d823640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f784806a780 0x7f7848154d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f784d823640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78480c24b0 0x7f7848155280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f784d823640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78481557c0 con 0x7f7848151d60 2023-10-25T00:23:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f784d823640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78480aadc0 con 0x7f784806a780 2023-10-25T00:23:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f784d823640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784806afe0 con 0x7f78480c24b0 2023-10-25T00:23:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7846ffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7848151d60 0x7f78480a9a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:52666/0 (socket says 172.21.15.107:52666) 2023-10-25T00:23:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7846ffd640 1 -- 172.21.15.107:0/1436598687 learned_addr learned my addr 172.21.15.107:0/1436598687 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:23:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f78477fe640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78480c24b0 0x7f7848155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f78467fc640 1 --2- 172.21.15.107:0/1436598687 >> v2:172.21.15.110:3300/0 conn(0x7f784806a780 0x7f7848154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176598372 0 0) 0x7f78481557c0 con 0x7f7848151d60 2023-10-25T00:23:18.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7830003530 con 0x7f7848151d60 2023-10-25T00:23:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4024281694 0 0) 0x7f7830003530 con 0x7f7848151d60 2023-10-25T00:23:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78480c24b0 msgr2=0x7f7848155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:23:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78480c24b0 0x7f7848155280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 >> v2:172.21.15.110:3300/0 conn(0x7f784806a780 msgr2=0x7f7848154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:23:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 --2- 172.21.15.107:0/1436598687 >> v2:172.21.15.110:3300/0 conn(0x7f784806a780 0x7f7848154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7848154840 con 0x7f7848151d60 2023-10-25T00:23:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f7838001680 con 0x7f7848151d60 2023-10-25T00:23:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 >> v1:172.21.15.107:6789/0 conn(0x7f7848151d60 legacy=0x7f78480a9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7830003980 0x7f7830003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78480c24b0 0x7f78300047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7830004d10 0x7f7830005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78480aadc0 con 0x7f7830003980 2023-10-25T00:23:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783000b990 con 0x7f7830004d10 2023-10-25T00:23:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783000bad0 con 0x7f78480c24b0 2023-10-25T00:23:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7846ffd640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7830003980 0x7f7830003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:18.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f78467fc640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78480c24b0 0x7f78300047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:18.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f78477fe640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7830004d10 0x7f7830005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:18.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7846ffd640 1 -- 172.21.15.107:0/1436598687 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78480c24b0 msgr2=0x7f78300047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:23:18.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7846ffd640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78480c24b0 0x7f78300047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7846ffd640 1 -- 172.21.15.107:0/1436598687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7830004d10 msgr2=0x7f7830005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:23:18.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7846ffd640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7830004d10 0x7f7830005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.471+0000 7f7846ffd640 1 -- 172.21.15.107:0/1436598687 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7830003530 con 0x7f7830003980 2023-10-25T00:23:18.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f7846ffd640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7830003980 0x7f7830003da0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f783800c210 tx=0x7f783801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648bd2089ce94fe server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78380058e0 con 0x7f7830003980 2023-10-25T00:23:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7838005a40 con 0x7f7830003980 2023-10-25T00:23:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f7845ffb640 1 -- 172.21.15.107:0/1436598687 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838005df0 con 0x7f7830003980 2023-10-25T00:23:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 -- 172.21.15.107:0/1436598687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7830003980 msgr2=0x7f7830003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:18.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 --2- 172.21.15.107:0/1436598687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7830003980 0x7f7830003da0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f783800c210 tx=0x7f783801cc40 comp rx=0 tx=0).stop 2023-10-25T00:23:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f78477fe640 1 -- 172.21.15.107:0/1436598687 reap_dead start 2023-10-25T00:23:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 -- 172.21.15.107:0/1436598687 shutdown_connections 2023-10-25T00:23:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 -- 172.21.15.107:0/1436598687 >> 172.21.15.107:0/1436598687 conn(0x7f78480c2950 msgr2=0x7f78480c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:23:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 -- 172.21.15.107:0/1436598687 shutdown_connections 2023-10-25T00:23:18.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 -- 172.21.15.107:0/1436598687 wait complete. 2023-10-25T00:23:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 Processor -- start 2023-10-25T00:23:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 -- start start 2023-10-25T00:23:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7830003980 0x7f78480c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7848143af0 0x7f78481effe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:18.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78481f0520 0x7f78480c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7848156100 con 0x7f78481f0520 2023-10-25T00:23:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78480c5bc0 con 0x7f7848143af0 2023-10-25T00:23:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f784d823640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78480c48a0 con 0x7f7830003980 2023-10-25T00:23:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.475+0000 7f78477fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78481f0520 0x7f78480c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:18.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f78477fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78481f0520 0x7f78480c3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41556/0 (socket says 172.21.15.107:41556) 2023-10-25T00:23:18.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f78477fe640 1 -- 172.21.15.107:0/3100833571 learned_addr learned my addr 172.21.15.107:0/3100833571 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:23:18.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f7846ffd640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7830003980 0x7f78480c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:18.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f78467fc640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7848143af0 0x7f78481effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:18.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f78477fe640 1 -- 172.21.15.107:0/3100833571 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7830003980 msgr2=0x7f78480c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:18.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f78477fe640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7830003980 0x7f78480c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f78477fe640 1 -- 172.21.15.107:0/3100833571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7848143af0 msgr2=0x7f78481effe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:18.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f78477fe640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7848143af0 0x7f78481effe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f78477fe640 1 -- 172.21.15.107:0/3100833571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78480c53a0 con 0x7f78481f0520 2023-10-25T00:23:18.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f78477fe640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78481f0520 0x7f78480c3bd0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f78480c2920 tx=0x7f78400115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:18.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f782ffff640 1 -- 172.21.15.107:0/3100833571 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784000e070 con 0x7f78481f0520 2023-10-25T00:23:18.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f782ffff640 1 -- 172.21.15.107:0/3100833571 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7840011e10 con 0x7f78481f0520 2023-10-25T00:23:18.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f784d823640 1 -- 172.21.15.107:0/3100833571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78480c4b60 con 0x7f78481f0520 2023-10-25T00:23:18.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f782ffff640 1 -- 172.21.15.107:0/3100833571 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78400039a0 con 0x7f78481f0520 2023-10-25T00:23:18.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f784d823640 1 -- 172.21.15.107:0/3100833571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7848158930 con 0x7f78481f0520 2023-10-25T00:23:18.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f784d823640 1 -- 172.21.15.107:0/3100833571 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78480746e0 con 0x7f78481f0520 2023-10-25T00:23:18.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f782ffff640 1 -- 172.21.15.107:0/3100833571 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7840003b00 con 0x7f78481f0520 2023-10-25T00:23:18.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f782ffff640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7834065e90 0x7f7834068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:18.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f782ffff640 1 -- 172.21.15.107:0/3100833571 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f7840085160 con 0x7f78481f0520 2023-10-25T00:23:18.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f7846ffd640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7834065e90 0x7f7834068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:18.491 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.479+0000 7f7846ffd640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7834065e90 0x7f7834068350 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f78380aa000 tx=0x7f7838000ff0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:18.492 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.483+0000 7f782ffff640 1 -- 172.21.15.107:0/3100833571 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f78400514c0 con 0x7f78481f0520 2023-10-25T00:23:18.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.639+0000 7f784d823640 1 -- 172.21.15.107:0/3100833571 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7848152140 con 0x7f7834065e90 2023-10-25T00:23:18.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.679+0000 7f782ffff640 1 -- 172.21.15.107:0/3100833571 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+549276 (secure 0 0 0) 0x7f7838005c30 con 0x7f7834065e90 2023-10-25T00:23:18.682 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:23:18.690 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-25T00:23:18.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 -- 172.21.15.107:0/3100833571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7834065e90 msgr2=0x7f7834068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:18.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7834065e90 0x7f7834068350 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f78380aa000 tx=0x7f7838000ff0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 -- 172.21.15.107:0/3100833571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78481f0520 msgr2=0x7f78480c3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:18.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78481f0520 0x7f78480c3bd0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f78480c2920 tx=0x7f78400115a0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 -- 172.21.15.107:0/3100833571 shutdown_connections 2023-10-25T00:23:18.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78481f0520 0x7f78480c3bd0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7848143af0 0x7f78481effe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7834065e90 0x7f7834068350 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 --2- 172.21.15.107:0/3100833571 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7830003980 0x7f78480c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:18.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 -- 172.21.15.107:0/3100833571 >> 172.21.15.107:0/3100833571 conn(0x7f78480c2950 msgr2=0x7f784815c560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:23:18.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 -- 172.21.15.107:0/3100833571 shutdown_connections 2023-10-25T00:23:18.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:18.687+0000 7f784d823640 1 -- 172.21.15.107:0/3100833571 wait complete. 2023-10-25T00:23:18.707 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":980,"stamp":"2023-10-25T00:23:18.030181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40142898,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3975,"num_read_kb":477474,"num_write":3153461,"num_write_kb":38880074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":583977,"ondisk_log_size":583977,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7810624,"kb_used_data":86208,"kb_used_omap":0,"kb_used_meta":7724352,"kb_avail":747164096,"statfs":{"total":773094113280,"available":765096034304,"internally_reserved":0,"allocated":88276992,"data_stored":84646180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7909736448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005943"},"pg_stats":[{"pgid":"6.1a","version":"72'3995","reported_seq":4069,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.233446+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-25T00:22:49.233446+0000","last_peered":"2023-10-25T00:22:49.233446+0000","last_clean":"2023-10-25T00:22:49.233446+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-25T00:22:49.233446+0000","last_undegraded":"2023-10-25T00:22:49.233446+0000","last_fullsized":"2023-10-25T00:22:49.233446+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21480,"num_write_kb":282788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"72'3","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965652+0000","last_change":"2023-10-24T23:53:11.443861+0000","last_active":"2023-10-25T00:22:33.965652+0000","last_peered":"2023-10-25T00:22:33.965652+0000","last_clean":"2023-10-25T00:22:33.965652+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-25T00:22:33.965652+0000","last_undegraded":"2023-10-25T00:22:33.965652+0000","last_fullsized":"2023-10-25T00:22:33.965652+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"68'864","reported_seq":993,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.255909+0000","last_change":"2023-10-24T23:53:02.350586+0000","last_active":"2023-10-25T00:22:49.255909+0000","last_peered":"2023-10-25T00:22:49.255909+0000","last_clean":"2023-10-25T00:22:49.255909+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-25T00:22:49.255909+0000","last_undegraded":"2023-10-25T00:22:49.255909+0000","last_fullsized":"2023-10-25T00:22:49.255909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":22541,"num_write":841,"num_write_kb":4586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'5616","reported_seq":5690,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060413+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-25T00:22:36.060413+0000","last_peered":"2023-10-25T00:22:36.060413+0000","last_clean":"2023-10-25T00:22:36.060413+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-25T00:22:36.060413+0000","last_undegraded":"2023-10-25T00:22:36.060413+0000","last_fullsized":"2023-10-25T00:22:36.060413+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29775,"num_write_kb":348818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"72'3735","reported_seq":3807,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234391+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-25T00:22:49.234391+0000","last_peered":"2023-10-25T00:22:49.234391+0000","last_clean":"2023-10-25T00:22:49.234391+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-25T00:22:49.234391+0000","last_undegraded":"2023-10-25T00:22:49.234391+0000","last_fullsized":"2023-10-25T00:22:49.234391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20090,"num_write_kb":264783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"72'2","reported_seq":83,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.880273+0000","last_change":"2023-10-25T00:23:02.880442+0000","last_active":"2023-10-25T00:23:02.880273+0000","last_peered":"2023-10-25T00:23:02.880273+0000","last_clean":"2023-10-25T00:23:02.880273+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-25T00:23:02.880273+0000","last_undegraded":"2023-10-25T00:23:02.880273+0000","last_fullsized":"2023-10-25T00:23:02.880273+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.880202+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:45.433003+0000","scrub_duration":0.0048172340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"73'28","reported_seq":116,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.443056+0000","last_change":"2023-10-25T00:23:05.443056+0000","last_active":"2023-10-25T00:23:05.443056+0000","last_peered":"2023-10-25T00:23:05.443056+0000","last_clean":"2023-10-25T00:23:05.443056+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-25T00:23:05.443056+0000","last_undegraded":"2023-10-25T00:23:05.443056+0000","last_fullsized":"2023-10-25T00:23:05.443056+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_deep_scrub":"73'28","last_deep_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.442750+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:29:04.499904+0000","scrub_duration":0.036465213000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'5447","reported_seq":5521,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.533893+0000","last_change":"2023-10-24T23:53:02.702286+0000","last_active":"2023-10-25T00:22:35.533893+0000","last_peered":"2023-10-25T00:22:35.533893+0000","last_clean":"2023-10-25T00:22:35.533893+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-25T00:22:35.533893+0000","last_undegraded":"2023-10-25T00:22:35.533893+0000","last_fullsized":"2023-10-25T00:22:35.533893+0000","mapping_epoch":21,"log_start":"61'100","ondisk_log_start":"61'100","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28063,"num_write_kb":350200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"72'4747","reported_seq":4820,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517509+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-25T00:22:33.517509+0000","last_peered":"2023-10-25T00:22:33.517509+0000","last_clean":"2023-10-25T00:22:33.517509+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-25T00:22:33.517509+0000","last_undegraded":"2023-10-25T00:22:33.517509+0000","last_fullsized":"2023-10-25T00:22:33.517509+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":25766,"num_write_kb":328702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"72'754","reported_seq":862,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.214995+0000","last_change":"2023-10-25T00:22:37.433948+0000","last_active":"2023-10-25T00:22:49.214995+0000","last_peered":"2023-10-25T00:22:49.214995+0000","last_clean":"2023-10-25T00:22:49.214995+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-25T00:22:49.214995+0000","last_undegraded":"2023-10-25T00:22:49.214995+0000","last_fullsized":"2023-10-25T00:22:49.214995+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'754","last_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_deep_scrub":"72'754","last_deep_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.433382+0000","objects_scrubbed":1,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:43.601946+0000","scrub_duration":0.10905978199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":77282,"num_write":740,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"73'50","reported_seq":170,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.647282+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-25T00:22:52.647282+0000","last_peered":"2023-10-25T00:22:52.647282+0000","last_clean":"2023-10-25T00:22:52.647282+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-25T00:22:52.647282+0000","last_undegraded":"2023-10-25T00:22:52.647282+0000","last_fullsized":"2023-10-25T00:22:52.647282+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'4792","reported_seq":4878,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234675+0000","last_change":"2023-10-25T00:22:44.881494+0000","last_active":"2023-10-25T00:22:49.234675+0000","last_peered":"2023-10-25T00:22:49.234675+0000","last_clean":"2023-10-25T00:22:49.234675+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-25T00:22:49.234675+0000","last_undegraded":"2023-10-25T00:22:49.234675+0000","last_fullsized":"2023-10-25T00:22:49.234675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4792","last_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_deep_scrub":"68'4792","last_deep_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.881344+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:23:57.562121+0000","scrub_duration":0.193670646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24997,"num_write_kb":313272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"72'4739","reported_seq":4813,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523335+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-25T00:22:36.523335+0000","last_peered":"2023-10-25T00:22:36.523335+0000","last_clean":"2023-10-25T00:22:36.523335+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-25T00:22:36.523335+0000","last_undegraded":"2023-10-25T00:22:36.523335+0000","last_fullsized":"2023-10-25T00:22:36.523335+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":22,"num_write":25639,"num_write_kb":332364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"72'357","reported_seq":430,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.998388+0000","last_change":"2023-10-25T00:23:16.269311+0000","last_active":"2023-10-25T00:22:55.998388+0000","last_peered":"2023-10-25T00:22:55.998388+0000","last_clean":"2023-10-25T00:22:55.998388+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-25T00:22:55.998388+0000","last_undegraded":"2023-10-25T00:22:55.998388+0000","last_fullsized":"2023-10-25T00:22:55.998388+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2627766,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":227,"num_write":329,"num_write_kb":2759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"68'5","reported_seq":127,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.649341+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-25T00:22:52.649341+0000","last_peered":"2023-10-25T00:22:52.649341+0000","last_clean":"2023-10-25T00:22:52.649341+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-25T00:22:52.649341+0000","last_undegraded":"2023-10-25T00:22:52.649341+0000","last_fullsized":"2023-10-25T00:22:52.649341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'3731","reported_seq":3815,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234039+0000","last_change":"2023-10-25T00:22:37.622543+0000","last_active":"2023-10-25T00:22:49.234039+0000","last_peered":"2023-10-25T00:22:49.234039+0000","last_clean":"2023-10-25T00:22:49.234039+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-25T00:22:49.234039+0000","last_undegraded":"2023-10-25T00:22:49.234039+0000","last_fullsized":"2023-10-25T00:22:49.234039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3731","last_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_deep_scrub":"68'3731","last_deep_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.622304+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:51:52.171234+0000","scrub_duration":0.090187284000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19492,"num_write_kb":250091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"72'3799","reported_seq":3872,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.236389+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-25T00:22:49.236389+0000","last_peered":"2023-10-25T00:22:49.236389+0000","last_clean":"2023-10-25T00:22:49.236389+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-25T00:22:49.236389+0000","last_undegraded":"2023-10-25T00:22:49.236389+0000","last_fullsized":"2023-10-25T00:22:49.236389+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21581,"num_write_kb":262877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"73'46","reported_seq":171,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.664257+0000","last_change":"2023-10-25T00:22:52.664257+0000","last_active":"2023-10-25T00:22:52.664257+0000","last_peered":"2023-10-25T00:22:52.664257+0000","last_clean":"2023-10-25T00:22:52.664257+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-25T00:22:52.664257+0000","last_undegraded":"2023-10-25T00:22:52.664257+0000","last_fullsized":"2023-10-25T00:22:52.664257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'46","last_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_deep_scrub":"73'46","last_deep_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.664064+0000","objects_scrubbed":1,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:44:09.108291+0000","scrub_duration":0.106079285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"68'714","reported_seq":817,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.253373+0000","last_change":"2023-10-24T23:53:02.357003+0000","last_active":"2023-10-25T00:22:49.253373+0000","last_peered":"2023-10-25T00:22:49.253373+0000","last_clean":"2023-10-25T00:22:49.253373+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-25T00:22:49.253373+0000","last_undegraded":"2023-10-25T00:22:49.253373+0000","last_fullsized":"2023-10-25T00:22:49.253373+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":90428,"num_write":703,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'4008","reported_seq":4096,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.488463+0000","last_change":"2023-10-25T00:23:05.488463+0000","last_active":"2023-10-25T00:23:05.488463+0000","last_peered":"2023-10-25T00:23:05.488463+0000","last_clean":"2023-10-25T00:23:05.488463+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-25T00:23:05.488463+0000","last_undegraded":"2023-10-25T00:23:05.488463+0000","last_fullsized":"2023-10-25T00:23:05.488463+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4008","last_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_deep_scrub":"68'4008","last_deep_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.488248+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:51:16.627187+0000","scrub_duration":0.15813021899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20791,"num_write_kb":267770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"72'5469","reported_seq":5564,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.571255+0000","last_change":"2023-10-25T00:22:57.571401+0000","last_active":"2023-10-25T00:22:57.571255+0000","last_peered":"2023-10-25T00:22:57.571255+0000","last_clean":"2023-10-25T00:22:57.571255+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-25T00:22:57.571255+0000","last_undegraded":"2023-10-25T00:22:57.571255+0000","last_fullsized":"2023-10-25T00:22:57.571255+0000","mapping_epoch":25,"log_start":"64'200","ondisk_log_start":"64'200","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5469","last_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_deep_scrub":"72'5469","last_deep_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.571234+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:24:13.836466+0000","scrub_duration":0.19144520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":40,"num_write":30646,"num_write_kb":401481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"73'6","reported_seq":131,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.991436+0000","last_change":"2023-10-25T00:23:14.991436+0000","last_active":"2023-10-25T00:23:14.991436+0000","last_peered":"2023-10-25T00:23:14.991436+0000","last_clean":"2023-10-25T00:23:14.991436+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-25T00:23:14.991436+0000","last_undegraded":"2023-10-25T00:23:14.991436+0000","last_fullsized":"2023-10-25T00:23:14.991436+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_deep_scrub":"73'6","last_deep_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.991238+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:25:08.545014+0000","scrub_duration":0.0072276010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"73'546","reported_seq":630,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.826017+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-25T00:22:55.826017+0000","last_peered":"2023-10-25T00:22:55.826017+0000","last_clean":"2023-10-25T00:22:55.826017+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-25T00:22:55.826017+0000","last_undegraded":"2023-10-25T00:22:55.826017+0000","last_fullsized":"2023-10-25T00:22:55.826017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3198201,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10628,"num_write":517,"num_write_kb":3434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'6069","reported_seq":6156,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.235476+0000","last_change":"2023-10-25T00:22:46.608504+0000","last_active":"2023-10-25T00:22:49.235476+0000","last_peered":"2023-10-25T00:22:49.235476+0000","last_clean":"2023-10-25T00:22:49.235476+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-25T00:22:49.235476+0000","last_undegraded":"2023-10-25T00:22:49.235476+0000","last_fullsized":"2023-10-25T00:22:49.235476+0000","mapping_epoch":21,"log_start":"62'800","ondisk_log_start":"62'800","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6069","last_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_deep_scrub":"68'6069","last_deep_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.608198+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:37.101860+0000","scrub_duration":0.19917779799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":33187,"num_write_kb":387130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'3606","reported_seq":3683,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.207557+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-25T00:22:49.207557+0000","last_peered":"2023-10-25T00:22:49.207557+0000","last_clean":"2023-10-25T00:22:49.207557+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-25T00:22:49.207557+0000","last_undegraded":"2023-10-25T00:22:49.207557+0000","last_fullsized":"2023-10-25T00:22:49.207557+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19493,"num_write_kb":254342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'1064","reported_seq":1198,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:03.701358+0000","last_change":"2023-10-25T00:23:03.701535+0000","last_active":"2023-10-25T00:23:03.701358+0000","last_peered":"2023-10-25T00:23:03.701358+0000","last_clean":"2023-10-25T00:23:03.701358+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-25T00:23:03.701358+0000","last_undegraded":"2023-10-25T00:23:03.701358+0000","last_fullsized":"2023-10-25T00:23:03.701358+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1064","last_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_deep_scrub":"73'1064","last_deep_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_clean_scrub_stamp":"2023-10-25T00:23:03.701304+0000","objects_scrubbed":3,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:08:18.178955+0000","scrub_duration":0.28236572100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":9402,"num_write":1035,"num_write_kb":4717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'3","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.838808+0000","last_change":"2023-10-25T00:23:12.838945+0000","last_active":"2023-10-25T00:23:12.838808+0000","last_peered":"2023-10-25T00:23:12.838808+0000","last_clean":"2023-10-25T00:23:12.838808+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-25T00:23:12.838808+0000","last_undegraded":"2023-10-25T00:23:12.838808+0000","last_fullsized":"2023-10-25T00:23:12.838808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.838787+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:33:35.778931+0000","scrub_duration":0.133446072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'5139","reported_seq":5226,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:54.030976+0000","last_change":"2023-10-25T00:22:54.030976+0000","last_active":"2023-10-25T00:22:54.030976+0000","last_peered":"2023-10-25T00:22:54.030976+0000","last_clean":"2023-10-25T00:22:54.030976+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-25T00:22:54.030976+0000","last_undegraded":"2023-10-25T00:22:54.030976+0000","last_fullsized":"2023-10-25T00:22:54.030976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5139","last_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_deep_scrub":"68'5139","last_deep_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.030782+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:30.182861+0000","scrub_duration":0.103857617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27772,"num_write_kb":328469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"72'4843","reported_seq":4916,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.231448+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-25T00:22:49.231448+0000","last_peered":"2023-10-25T00:22:49.231448+0000","last_clean":"2023-10-25T00:22:49.231448+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-25T00:22:49.231448+0000","last_undegraded":"2023-10-25T00:22:49.231448+0000","last_fullsized":"2023-10-25T00:22:49.231448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":27290,"num_write_kb":330906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"72'27","reported_seq":110,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.986728+0000","last_change":"2023-10-25T00:22:55.987117+0000","last_active":"2023-10-25T00:22:55.986728+0000","last_peered":"2023-10-25T00:22:55.986728+0000","last_clean":"2023-10-25T00:22:55.986728+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-25T00:22:55.986728+0000","last_undegraded":"2023-10-25T00:22:55.986728+0000","last_fullsized":"2023-10-25T00:22:55.986728+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'27","last_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_deep_scrub":"72'27","last_deep_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.986674+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:33.089164+0000","scrub_duration":0.100730242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'2","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.533782+0000","last_change":"2023-10-24T23:53:02.331940+0000","last_active":"2023-10-25T00:22:35.533782+0000","last_peered":"2023-10-25T00:22:35.533782+0000","last_clean":"2023-10-25T00:22:35.533782+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-25T00:22:35.533782+0000","last_undegraded":"2023-10-25T00:22:35.533782+0000","last_fullsized":"2023-10-25T00:22:35.533782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'5814","reported_seq":5901,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.320438+0000","last_change":"2023-10-25T00:22:52.320696+0000","last_active":"2023-10-25T00:22:52.320438+0000","last_peered":"2023-10-25T00:22:52.320438+0000","last_clean":"2023-10-25T00:22:52.320438+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-25T00:22:52.320438+0000","last_undegraded":"2023-10-25T00:22:52.320438+0000","last_fullsized":"2023-10-25T00:22:52.320438+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5814","last_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_deep_scrub":"68'5814","last_deep_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.320378+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:26.511615+0000","scrub_duration":0.033495337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31996,"num_write_kb":382397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"72'3827","reported_seq":3911,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.436201+0000","last_change":"2023-10-25T00:23:14.436458+0000","last_active":"2023-10-25T00:23:14.436201+0000","last_peered":"2023-10-25T00:23:14.436201+0000","last_clean":"2023-10-25T00:23:14.436201+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-25T00:23:14.436201+0000","last_undegraded":"2023-10-25T00:23:14.436201+0000","last_fullsized":"2023-10-25T00:23:14.436201+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3827","last_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_deep_scrub":"72'3827","last_deep_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.436150+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:27:32.423180+0000","scrub_duration":0.30547422699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":21199,"num_write_kb":277730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"68'4820","reported_seq":4906,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.429763+0000","last_change":"2023-10-25T00:22:36.429763+0000","last_active":"2023-10-25T00:22:36.429763+0000","last_peered":"2023-10-25T00:22:36.429763+0000","last_clean":"2023-10-25T00:22:36.429763+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-25T00:22:36.429763+0000","last_undegraded":"2023-10-25T00:22:36.429763+0000","last_fullsized":"2023-10-25T00:22:36.429763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4820","last_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_deep_scrub":"68'4820","last_deep_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_clean_scrub_stamp":"2023-10-25T00:22:36.429607+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:35.883183+0000","scrub_duration":0.10613655600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25163,"num_write_kb":285293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"72'5","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.516883+0000","last_change":"2023-10-24T23:53:02.349959+0000","last_active":"2023-10-25T00:22:33.516883+0000","last_peered":"2023-10-25T00:22:33.516883+0000","last_clean":"2023-10-25T00:22:33.516883+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-25T00:22:33.516883+0000","last_undegraded":"2023-10-25T00:22:33.516883+0000","last_fullsized":"2023-10-25T00:22:33.516883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"72'5","reported_seq":86,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:40.197550+0000","last_change":"2023-10-25T00:22:40.197550+0000","last_active":"2023-10-25T00:22:40.197550+0000","last_peered":"2023-10-25T00:22:40.197550+0000","last_clean":"2023-10-25T00:22:40.197550+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-25T00:22:40.197550+0000","last_undegraded":"2023-10-25T00:22:40.197550+0000","last_fullsized":"2023-10-25T00:22:40.197550+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.197213+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:07:28.509914+0000","scrub_duration":0.0047704979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"72'4957","reported_seq":5044,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:53.667769+0000","last_change":"2023-10-25T00:22:53.667906+0000","last_active":"2023-10-25T00:22:53.667769+0000","last_peered":"2023-10-25T00:22:53.667769+0000","last_clean":"2023-10-25T00:22:53.667769+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-25T00:22:53.667769+0000","last_undegraded":"2023-10-25T00:22:53.667769+0000","last_fullsized":"2023-10-25T00:22:53.667769+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4957","last_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_deep_scrub":"72'4957","last_deep_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.667748+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:58.307897+0000","scrub_duration":0.30812842099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28093,"num_write_kb":358545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'4782","reported_seq":4873,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.562198+0000","last_change":"2023-10-25T00:22:42.562198+0000","last_active":"2023-10-25T00:22:42.562198+0000","last_peered":"2023-10-25T00:22:42.562198+0000","last_clean":"2023-10-25T00:22:42.562198+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-25T00:22:42.562198+0000","last_undegraded":"2023-10-25T00:22:42.562198+0000","last_fullsized":"2023-10-25T00:22:42.562198+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4782","last_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_deep_scrub":"68'4782","last_deep_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.562057+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:39.415960+0000","scrub_duration":0.116720554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26432,"num_write_kb":298639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"68'3","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522666+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-25T00:22:36.522666+0000","last_peered":"2023-10-25T00:22:36.522666+0000","last_clean":"2023-10-25T00:22:36.522666+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-25T00:22:36.522666+0000","last_undegraded":"2023-10-25T00:22:36.522666+0000","last_fullsized":"2023-10-25T00:22:36.522666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"72'18","reported_seq":214,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:46.074083+0000","last_change":"2023-10-25T00:22:46.074083+0000","last_active":"2023-10-25T00:22:46.074083+0000","last_peered":"2023-10-25T00:22:46.074083+0000","last_clean":"2023-10-25T00:22:46.074083+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-25T00:22:46.074083+0000","last_undegraded":"2023-10-25T00:22:46.074083+0000","last_fullsized":"2023-10-25T00:22:46.074083+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_deep_scrub":"72'18","last_deep_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.073758+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:08:25.787090+0000","scrub_duration":0.012558368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":345,"num_read_kb":345,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"72'2954","reported_seq":3025,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517454+0000","last_change":"2023-10-24T23:53:11.798852+0000","last_active":"2023-10-25T00:22:33.517454+0000","last_peered":"2023-10-25T00:22:33.517454+0000","last_clean":"2023-10-25T00:22:33.517454+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-25T00:22:33.517454+0000","last_undegraded":"2023-10-25T00:22:33.517454+0000","last_fullsized":"2023-10-25T00:22:33.517454+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15961,"num_write_kb":219669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"72'36","reported_seq":156,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.225469+0000","last_change":"2023-10-24T23:53:02.355797+0000","last_active":"2023-10-25T00:22:49.225469+0000","last_peered":"2023-10-25T00:22:49.225469+0000","last_clean":"2023-10-25T00:22:49.225469+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-25T00:22:49.225469+0000","last_undegraded":"2023-10-25T00:22:49.225469+0000","last_fullsized":"2023-10-25T00:22:49.225469+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'4690","reported_seq":4777,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.792610+0000","last_change":"2023-10-25T00:23:05.792729+0000","last_active":"2023-10-25T00:23:05.792610+0000","last_peered":"2023-10-25T00:23:05.792610+0000","last_clean":"2023-10-25T00:23:05.792610+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-25T00:23:05.792610+0000","last_undegraded":"2023-10-25T00:23:05.792610+0000","last_fullsized":"2023-10-25T00:23:05.792610+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4690","last_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_deep_scrub":"68'4690","last_deep_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.792588+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:24:53.476824+0000","scrub_duration":0.15089322399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25172,"num_write_kb":305786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"72'10","reported_seq":135,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:40.512549+0000","last_change":"2023-10-25T00:22:40.512549+0000","last_active":"2023-10-25T00:22:40.512549+0000","last_peered":"2023-10-25T00:22:40.512549+0000","last_clean":"2023-10-25T00:22:40.512549+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-25T00:22:40.512549+0000","last_undegraded":"2023-10-25T00:22:40.512549+0000","last_fullsized":"2023-10-25T00:22:40.512549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.512409+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:07:23.705506+0000","scrub_duration":0.091817919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"72'4200","reported_seq":4279,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534173+0000","last_change":"2023-10-24T23:53:11.812783+0000","last_active":"2023-10-25T00:22:35.534173+0000","last_peered":"2023-10-25T00:22:35.534173+0000","last_clean":"2023-10-25T00:22:35.534173+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-25T00:22:35.534173+0000","last_undegraded":"2023-10-25T00:22:35.534173+0000","last_fullsized":"2023-10-25T00:22:35.534173+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":24169,"num_write_kb":293796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"73'390","reported_seq":580,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.648256+0000","last_change":"2023-10-24T23:53:02.354424+0000","last_active":"2023-10-25T00:22:52.648256+0000","last_peered":"2023-10-25T00:22:52.648256+0000","last_clean":"2023-10-25T00:22:52.648256+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-25T00:22:52.648256+0000","last_undegraded":"2023-10-25T00:22:52.648256+0000","last_fullsized":"2023-10-25T00:22:52.648256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":210,"num_read_kb":94395,"num_write":387,"num_write_kb":4304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'5730","reported_seq":5805,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060329+0000","last_change":"2023-10-24T23:53:02.721112+0000","last_active":"2023-10-25T00:22:36.060329+0000","last_peered":"2023-10-25T00:22:36.060329+0000","last_clean":"2023-10-25T00:22:36.060329+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-25T00:22:36.060329+0000","last_undegraded":"2023-10-25T00:22:36.060329+0000","last_fullsized":"2023-10-25T00:22:36.060329+0000","mapping_epoch":21,"log_start":"61'500","ondisk_log_start":"61'500","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":31652,"num_write_kb":356481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"72'3","reported_seq":73,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965700+0000","last_change":"2023-10-24T23:53:11.443951+0000","last_active":"2023-10-25T00:22:33.965700+0000","last_peered":"2023-10-25T00:22:33.965700+0000","last_clean":"2023-10-25T00:22:33.965700+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-25T00:22:33.965700+0000","last_undegraded":"2023-10-25T00:22:33.965700+0000","last_fullsized":"2023-10-25T00:22:33.965700+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"72'4168","reported_seq":4242,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437378+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-25T00:22:37.437378+0000","last_peered":"2023-10-25T00:22:37.437378+0000","last_clean":"2023-10-25T00:22:37.437378+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-25T00:22:37.437378+0000","last_undegraded":"2023-10-25T00:22:37.437378+0000","last_fullsized":"2023-10-25T00:22:37.437378+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23458,"num_write_kb":294850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"68'4339","reported_seq":4415,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060019+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-25T00:22:36.060019+0000","last_peered":"2023-10-25T00:22:36.060019+0000","last_clean":"2023-10-25T00:22:36.060019+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-25T00:22:36.060019+0000","last_undegraded":"2023-10-25T00:22:36.060019+0000","last_fullsized":"2023-10-25T00:22:36.060019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23267,"num_write_kb":265268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":89,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.199286+0000","last_change":"2023-10-25T00:22:42.199286+0000","last_active":"2023-10-25T00:22:42.199286+0000","last_peered":"2023-10-25T00:22:42.199286+0000","last_clean":"2023-10-25T00:22:42.199286+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-25T00:22:42.199286+0000","last_undegraded":"2023-10-25T00:22:42.199286+0000","last_fullsized":"2023-10-25T00:22:42.199286+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.198947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:11:17.807377+0000","scrub_duration":0.0041955760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"73'231","reported_seq":341,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.009873+0000","last_change":"2023-10-25T00:23:16.010034+0000","last_active":"2023-10-25T00:23:16.009873+0000","last_peered":"2023-10-25T00:23:16.009873+0000","last_clean":"2023-10-25T00:23:16.009873+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-25T00:23:16.009873+0000","last_undegraded":"2023-10-25T00:23:16.009873+0000","last_fullsized":"2023-10-25T00:23:16.009873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'231","last_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_deep_scrub":"73'231","last_deep_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.009842+0000","objects_scrubbed":1,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:27:21.881029+0000","scrub_duration":0.076880452000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":63,"num_write":590,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"72'11","reported_seq":95,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.231986+0000","last_change":"2023-10-25T00:22:38.409520+0000","last_active":"2023-10-25T00:22:49.231986+0000","last_peered":"2023-10-25T00:22:49.231986+0000","last_clean":"2023-10-25T00:22:49.231986+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-25T00:22:49.231986+0000","last_undegraded":"2023-10-25T00:22:49.231986+0000","last_fullsized":"2023-10-25T00:22:49.231986+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_deep_scrub":"72'11","last_deep_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.409235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:21:24.708709+0000","scrub_duration":0.003592329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"72'3993","reported_seq":4069,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060792+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-25T00:22:36.060792+0000","last_peered":"2023-10-25T00:22:36.060792+0000","last_clean":"2023-10-25T00:22:36.060792+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-25T00:22:36.060792+0000","last_undegraded":"2023-10-25T00:22:36.060792+0000","last_fullsized":"2023-10-25T00:22:36.060792+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21723,"num_write_kb":283850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'4270","reported_seq":4358,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:59.570791+0000","last_change":"2023-10-25T00:22:59.570791+0000","last_active":"2023-10-25T00:22:59.570791+0000","last_peered":"2023-10-25T00:22:59.570791+0000","last_clean":"2023-10-25T00:22:59.570791+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-25T00:22:59.570791+0000","last_undegraded":"2023-10-25T00:22:59.570791+0000","last_fullsized":"2023-10-25T00:22:59.570791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4270","last_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_deep_scrub":"68'4270","last_deep_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.570628+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:36:56.294621+0000","scrub_duration":0.14092118100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23049,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.643204+0000","last_change":"2023-10-25T00:23:06.643510+0000","last_active":"2023-10-25T00:23:06.643204+0000","last_peered":"2023-10-25T00:23:06.643204+0000","last_clean":"2023-10-25T00:23:06.643204+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-25T00:23:06.643204+0000","last_undegraded":"2023-10-25T00:23:06.643204+0000","last_fullsized":"2023-10-25T00:23:06.643204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.643156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:34:24.773001+0000","scrub_duration":0.0045495850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"68'9","reported_seq":132,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.438809+0000","last_change":"2023-10-24T23:53:02.354720+0000","last_active":"2023-10-25T00:22:52.438809+0000","last_peered":"2023-10-25T00:22:52.438809+0000","last_clean":"2023-10-25T00:22:52.438809+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-25T00:22:52.438809+0000","last_undegraded":"2023-10-25T00:22:52.438809+0000","last_fullsized":"2023-10-25T00:22:52.438809+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":8,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"73'65","reported_seq":228,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.930205+0000","last_change":"2023-10-25T00:23:07.930465+0000","last_active":"2023-10-25T00:23:07.930205+0000","last_peered":"2023-10-25T00:23:07.930205+0000","last_clean":"2023-10-25T00:23:07.930205+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-25T00:23:07.930205+0000","last_undegraded":"2023-10-25T00:23:07.930205+0000","last_fullsized":"2023-10-25T00:23:07.930205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'65","last_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_deep_scrub":"73'65","last_deep_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.930143+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:41:49.528670+0000","scrub_duration":0.020886851000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":123,"num_write":59,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"72'3769","reported_seq":3845,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965701+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-25T00:22:33.965701+0000","last_peered":"2023-10-25T00:22:33.965701+0000","last_clean":"2023-10-25T00:22:33.965701+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-25T00:22:33.965701+0000","last_undegraded":"2023-10-25T00:22:33.965701+0000","last_fullsized":"2023-10-25T00:22:33.965701+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20807,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'5752","reported_seq":5839,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.937946+0000","last_change":"2023-10-25T00:22:33.937946+0000","last_active":"2023-10-25T00:22:33.937946+0000","last_peered":"2023-10-25T00:22:33.937946+0000","last_clean":"2023-10-25T00:22:33.937946+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-25T00:22:33.937946+0000","last_undegraded":"2023-10-25T00:22:33.937946+0000","last_fullsized":"2023-10-25T00:22:33.937946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5752","last_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_deep_scrub":"68'5752","last_deep_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_clean_scrub_stamp":"2023-10-25T00:22:33.937564+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:43:34.779084+0000","scrub_duration":0.080678784000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31231,"num_write_kb":354718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.436957+0000","last_change":"2023-10-24T23:52:58.591522+0000","last_active":"2023-10-25T00:22:37.436957+0000","last_peered":"2023-10-25T00:22:37.436957+0000","last_clean":"2023-10-25T00:22:37.436957+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-25T00:22:37.436957+0000","last_undegraded":"2023-10-25T00:22:37.436957+0000","last_fullsized":"2023-10-25T00:22:37.436957+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'2","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965341+0000","last_change":"2023-10-24T23:53:02.332171+0000","last_active":"2023-10-25T00:22:33.965341+0000","last_peered":"2023-10-25T00:22:33.965341+0000","last_clean":"2023-10-25T00:22:33.965341+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-25T00:22:33.965341+0000","last_undegraded":"2023-10-25T00:22:33.965341+0000","last_fullsized":"2023-10-25T00:22:33.965341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"72'4","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.496631+0000","last_change":"2023-10-25T00:23:06.496904+0000","last_active":"2023-10-25T00:23:06.496631+0000","last_peered":"2023-10-25T00:23:06.496631+0000","last_clean":"2023-10-25T00:23:06.496631+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-25T00:23:06.496631+0000","last_undegraded":"2023-10-25T00:23:06.496631+0000","last_fullsized":"2023-10-25T00:23:06.496631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_deep_scrub":"72'4","last_deep_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.496575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:32:18.665160+0000","scrub_duration":0.0046312189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"72'3939","reported_seq":4021,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.160115+0000","last_change":"2023-10-25T00:23:02.160279+0000","last_active":"2023-10-25T00:23:02.160115+0000","last_peered":"2023-10-25T00:23:02.160115+0000","last_clean":"2023-10-25T00:23:02.160115+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-25T00:23:02.160115+0000","last_undegraded":"2023-10-25T00:23:02.160115+0000","last_fullsized":"2023-10-25T00:23:02.160115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3939","last_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_deep_scrub":"72'3939","last_deep_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.160086+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:37:29.878702+0000","scrub_duration":0.252775894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22393,"num_write_kb":262004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'4305","reported_seq":4381,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965378+0000","last_change":"2023-10-24T23:53:02.716755+0000","last_active":"2023-10-25T00:22:33.965378+0000","last_peered":"2023-10-25T00:22:33.965378+0000","last_clean":"2023-10-25T00:22:33.965378+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-25T00:22:33.965378+0000","last_undegraded":"2023-10-25T00:22:33.965378+0000","last_fullsized":"2023-10-25T00:22:33.965378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22696,"num_write_kb":287574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:11.717097+0000","last_change":"2023-10-25T00:23:11.717353+0000","last_active":"2023-10-25T00:23:11.717097+0000","last_peered":"2023-10-25T00:23:11.717097+0000","last_clean":"2023-10-25T00:23:11.717097+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-25T00:23:11.717097+0000","last_undegraded":"2023-10-25T00:23:11.717097+0000","last_fullsized":"2023-10-25T00:23:11.717097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_clean_scrub_stamp":"2023-10-25T00:23:11.717027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:17:42.982457+0000","scrub_duration":0.0039434270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2","reported_seq":78,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522611+0000","last_change":"2023-10-24T23:53:02.337819+0000","last_active":"2023-10-25T00:22:36.522611+0000","last_peered":"2023-10-25T00:22:36.522611+0000","last_clean":"2023-10-25T00:22:36.522611+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-25T00:22:36.522611+0000","last_undegraded":"2023-10-25T00:22:36.522611+0000","last_fullsized":"2023-10-25T00:22:36.522611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"72'3","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.952645+0000","last_change":"2023-10-25T00:22:52.952897+0000","last_active":"2023-10-25T00:22:52.952645+0000","last_peered":"2023-10-25T00:22:52.952645+0000","last_clean":"2023-10-25T00:22:52.952645+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-25T00:22:52.952645+0000","last_undegraded":"2023-10-25T00:22:52.952645+0000","last_fullsized":"2023-10-25T00:22:52.952645+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.952595+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:14:20.890322+0000","scrub_duration":0.0044650949999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"72'5003","reported_seq":5080,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534073+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-25T00:22:35.534073+0000","last_peered":"2023-10-25T00:22:35.534073+0000","last_clean":"2023-10-25T00:22:35.534073+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-25T00:22:35.534073+0000","last_undegraded":"2023-10-25T00:22:35.534073+0000","last_fullsized":"2023-10-25T00:22:35.534073+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":13,"num_write":27120,"num_write_kb":350355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'5635","reported_seq":5721,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:44.547139+0000","last_change":"2023-10-25T00:22:44.547139+0000","last_active":"2023-10-25T00:22:44.547139+0000","last_peered":"2023-10-25T00:22:44.547139+0000","last_clean":"2023-10-25T00:22:44.547139+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-25T00:22:44.547139+0000","last_undegraded":"2023-10-25T00:22:44.547139+0000","last_fullsized":"2023-10-25T00:22:44.547139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5635","last_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_deep_scrub":"68'5635","last_deep_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.546984+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:51:54.474684+0000","scrub_duration":0.15865177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30556,"num_write_kb":356140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"68'3","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060015+0000","last_change":"2023-10-24T23:53:02.344811+0000","last_active":"2023-10-25T00:22:36.060015+0000","last_peered":"2023-10-25T00:22:36.060015+0000","last_clean":"2023-10-25T00:22:36.060015+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-25T00:22:36.060015+0000","last_undegraded":"2023-10-25T00:22:36.060015+0000","last_fullsized":"2023-10-25T00:22:36.060015+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522602+0000","last_change":"2023-10-24T23:52:58.159262+0000","last_active":"2023-10-25T00:22:36.522602+0000","last_peered":"2023-10-25T00:22:36.522602+0000","last_clean":"2023-10-25T00:22:36.522602+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-25T00:22:36.522602+0000","last_undegraded":"2023-10-25T00:22:36.522602+0000","last_fullsized":"2023-10-25T00:22:36.522602+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"72'2","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:58.925373+0000","last_change":"2023-10-25T00:22:58.925685+0000","last_active":"2023-10-25T00:22:58.925373+0000","last_peered":"2023-10-25T00:22:58.925373+0000","last_clean":"2023-10-25T00:22:58.925373+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-25T00:22:58.925373+0000","last_undegraded":"2023-10-25T00:22:58.925373+0000","last_fullsized":"2023-10-25T00:22:58.925373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.925318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:31:45.562398+0000","scrub_duration":0.0043101720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"72'3753","reported_seq":3828,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437502+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-25T00:22:37.437502+0000","last_peered":"2023-10-25T00:22:37.437502+0000","last_clean":"2023-10-25T00:22:37.437502+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-25T00:22:37.437502+0000","last_undegraded":"2023-10-25T00:22:37.437502+0000","last_fullsized":"2023-10-25T00:22:37.437502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":21045,"num_write_kb":276986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'4101","reported_seq":4175,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965518+0000","last_change":"2023-10-24T23:53:02.715207+0000","last_active":"2023-10-25T00:22:33.965518+0000","last_peered":"2023-10-25T00:22:33.965518+0000","last_clean":"2023-10-25T00:22:33.965518+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-25T00:22:33.965518+0000","last_undegraded":"2023-10-25T00:22:33.965518+0000","last_fullsized":"2023-10-25T00:22:33.965518+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22383,"num_write_kb":270023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"73'13","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.803351+0000","last_change":"2023-10-24T23:53:02.333063+0000","last_active":"2023-10-25T00:22:49.803351+0000","last_peered":"2023-10-25T00:22:49.803351+0000","last_clean":"2023-10-25T00:22:49.803351+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-25T00:22:49.803351+0000","last_undegraded":"2023-10-25T00:22:49.803351+0000","last_fullsized":"2023-10-25T00:22:49.803351+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:51.317510+0000","last_change":"2023-10-25T00:22:51.317813+0000","last_active":"2023-10-25T00:22:51.317510+0000","last_peered":"2023-10-25T00:22:51.317510+0000","last_clean":"2023-10-25T00:22:51.317510+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-25T00:22:51.317510+0000","last_undegraded":"2023-10-25T00:22:51.317510+0000","last_fullsized":"2023-10-25T00:22:51.317510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.317462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:03:14.150576+0000","scrub_duration":0.0044508079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"72'248","reported_seq":341,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427421+0000","last_change":"2023-10-24T23:53:11.443750+0000","last_active":"2023-10-25T00:22:34.427421+0000","last_peered":"2023-10-25T00:22:34.427421+0000","last_clean":"2023-10-25T00:22:34.427421+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-25T00:22:34.427421+0000","last_undegraded":"2023-10-25T00:22:34.427421+0000","last_fullsized":"2023-10-25T00:22:34.427421+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":62,"num_write":607,"num_write_kb":223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"72'5236","reported_seq":5313,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965779+0000","last_change":"2023-10-24T23:53:11.807546+0000","last_active":"2023-10-25T00:22:33.965779+0000","last_peered":"2023-10-25T00:22:33.965779+0000","last_clean":"2023-10-25T00:22:33.965779+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-25T00:22:33.965779+0000","last_undegraded":"2023-10-25T00:22:33.965779+0000","last_fullsized":"2023-10-25T00:22:33.965779+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":29554,"num_write_kb":368820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'5444","reported_seq":5533,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.141480+0000","last_change":"2023-10-25T00:23:00.141618+0000","last_active":"2023-10-25T00:23:00.141480+0000","last_peered":"2023-10-25T00:23:00.141480+0000","last_clean":"2023-10-25T00:23:00.141480+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-25T00:23:00.141480+0000","last_undegraded":"2023-10-25T00:23:00.141480+0000","last_fullsized":"2023-10-25T00:23:00.141480+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5444","last_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_deep_scrub":"68'5444","last_deep_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.141458+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:23:18.614380+0000","scrub_duration":0.197741424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28342,"num_write_kb":342931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"73'67","reported_seq":238,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.941625+0000","last_change":"2023-10-25T00:23:14.941868+0000","last_active":"2023-10-25T00:23:14.941625+0000","last_peered":"2023-10-25T00:23:14.941625+0000","last_clean":"2023-10-25T00:23:14.941625+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-25T00:23:14.941625+0000","last_undegraded":"2023-10-25T00:23:14.941625+0000","last_fullsized":"2023-10-25T00:23:14.941625+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'67","last_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_deep_scrub":"73'67","last_deep_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.941564+0000","objects_scrubbed":1,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:11:54.084201+0000","scrub_duration":0.0084463549999999991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":125,"num_write":61,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.209420+0000","last_change":"2023-10-24T23:52:58.163164+0000","last_active":"2023-10-25T00:22:49.209420+0000","last_peered":"2023-10-25T00:22:49.209420+0000","last_clean":"2023-10-25T00:22:49.209420+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-25T00:22:49.209420+0000","last_undegraded":"2023-10-25T00:22:49.209420+0000","last_fullsized":"2023-10-25T00:22:49.209420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"72'10","reported_seq":137,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:45.425308+0000","last_change":"2023-10-25T00:22:45.425308+0000","last_active":"2023-10-25T00:22:45.425308+0000","last_peered":"2023-10-25T00:22:45.425308+0000","last_clean":"2023-10-25T00:22:45.425308+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-25T00:22:45.425308+0000","last_undegraded":"2023-10-25T00:22:45.425308+0000","last_fullsized":"2023-10-25T00:22:45.425308+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.425052+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:36:38.527058+0000","scrub_duration":0.14014134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":135,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"72'3666","reported_seq":3737,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523179+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-25T00:22:36.523179+0000","last_peered":"2023-10-25T00:22:36.523179+0000","last_clean":"2023-10-25T00:22:36.523179+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-25T00:22:36.523179+0000","last_undegraded":"2023-10-25T00:22:36.523179+0000","last_fullsized":"2023-10-25T00:22:36.523179+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20519,"num_write_kb":254391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"68'4182","reported_seq":4269,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.489530+0000","last_change":"2023-10-25T00:23:02.489861+0000","last_active":"2023-10-25T00:23:02.489530+0000","last_peered":"2023-10-25T00:23:02.489530+0000","last_clean":"2023-10-25T00:23:02.489530+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-25T00:23:02.489530+0000","last_undegraded":"2023-10-25T00:23:02.489530+0000","last_fullsized":"2023-10-25T00:23:02.489530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4182","last_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_deep_scrub":"68'4182","last_deep_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.489471+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:14:08.666419+0000","scrub_duration":0.092197608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22839,"num_write_kb":259586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'4","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:15.544038+0000","last_change":"2023-10-25T00:23:15.544275+0000","last_active":"2023-10-25T00:23:15.544038+0000","last_peered":"2023-10-25T00:23:15.544038+0000","last_clean":"2023-10-25T00:23:15.544038+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-25T00:23:15.544038+0000","last_undegraded":"2023-10-25T00:23:15.544038+0000","last_fullsized":"2023-10-25T00:23:15.544038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_deep_scrub":"67'4","last_deep_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:15.543988+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:33:51.332032+0000","scrub_duration":0.004459345,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522626+0000","last_change":"2023-10-24T23:52:58.163586+0000","last_active":"2023-10-25T00:22:36.522626+0000","last_peered":"2023-10-25T00:22:36.522626+0000","last_clean":"2023-10-25T00:22:36.522626+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-25T00:22:36.522626+0000","last_undegraded":"2023-10-25T00:22:36.522626+0000","last_fullsized":"2023-10-25T00:22:36.522626+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:57.136697+0000","last_clean_scrub_stamp":"2023-10-24T23:52:57.136697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"43'161","reported_seq":286,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.210661+0000","last_change":"2023-10-24T23:52:52.443253+0000","last_active":"2023-10-25T00:22:49.210661+0000","last_peered":"2023-10-25T00:22:49.210661+0000","last_clean":"2023-10-25T00:22:49.210661+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-25T00:22:49.210661+0000","last_undegraded":"2023-10-25T00:22:49.210661+0000","last_fullsized":"2023-10-25T00:22:49.210661+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:52:51.349804+0000","last_clean_scrub_stamp":"2023-10-24T23:52:51.349804+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"72'2","reported_seq":85,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:48.067150+0000","last_change":"2023-10-25T00:22:48.067150+0000","last_active":"2023-10-25T00:22:48.067150+0000","last_peered":"2023-10-25T00:22:48.067150+0000","last_clean":"2023-10-25T00:22:48.067150+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-25T00:22:48.067150+0000","last_undegraded":"2023-10-25T00:22:48.067150+0000","last_fullsized":"2023-10-25T00:22:48.067150+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.066810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:55:20.112674+0000","scrub_duration":0.0050122709999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"72'4105","reported_seq":4180,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523286+0000","last_change":"2023-10-24T23:53:11.831184+0000","last_active":"2023-10-25T00:22:36.523286+0000","last_peered":"2023-10-25T00:22:36.523286+0000","last_clean":"2023-10-25T00:22:36.523286+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-25T00:22:36.523286+0000","last_undegraded":"2023-10-25T00:22:36.523286+0000","last_fullsized":"2023-10-25T00:22:36.523286+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":19,"num_write":23147,"num_write_kb":297959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'5705","reported_seq":5782,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.231359+0000","last_change":"2023-10-25T00:22:53.576536+0000","last_active":"2023-10-25T00:22:49.231359+0000","last_peered":"2023-10-25T00:22:49.231359+0000","last_clean":"2023-10-25T00:22:49.231359+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-25T00:22:49.231359+0000","last_undegraded":"2023-10-25T00:22:49.231359+0000","last_fullsized":"2023-10-25T00:22:49.231359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":29845,"num_write_kb":357286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"68'5","reported_seq":80,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.533697+0000","last_change":"2023-10-24T23:53:02.338221+0000","last_active":"2023-10-25T00:22:35.533697+0000","last_peered":"2023-10-25T00:22:35.533697+0000","last_clean":"2023-10-25T00:22:35.533697+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-25T00:22:35.533697+0000","last_undegraded":"2023-10-25T00:22:35.533697+0000","last_fullsized":"2023-10-25T00:22:35.533697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"72'7","reported_seq":124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060732+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-25T00:22:36.060732+0000","last_peered":"2023-10-25T00:22:36.060732+0000","last_clean":"2023-10-25T00:22:36.060732+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-25T00:22:36.060732+0000","last_undegraded":"2023-10-25T00:22:36.060732+0000","last_fullsized":"2023-10-25T00:22:36.060732+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"72'4234","reported_seq":4308,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534135+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-25T00:22:35.534135+0000","last_peered":"2023-10-25T00:22:35.534135+0000","last_clean":"2023-10-25T00:22:35.534135+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-25T00:22:35.534135+0000","last_undegraded":"2023-10-25T00:22:35.534135+0000","last_fullsized":"2023-10-25T00:22:35.534135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":15,"num_write":23173,"num_write_kb":302022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"68'4052","reported_seq":4127,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060267+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-25T00:22:36.060267+0000","last_peered":"2023-10-25T00:22:36.060267+0000","last_clean":"2023-10-25T00:22:36.060267+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-25T00:22:36.060267+0000","last_undegraded":"2023-10-25T00:22:36.060267+0000","last_fullsized":"2023-10-25T00:22:36.060267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21486,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"68'2","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.059972+0000","last_change":"2023-10-24T23:53:02.347979+0000","last_active":"2023-10-25T00:22:36.059972+0000","last_peered":"2023-10-25T00:22:36.059972+0000","last_clean":"2023-10-25T00:22:36.059972+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-25T00:22:36.059972+0000","last_undegraded":"2023-10-25T00:22:36.059972+0000","last_fullsized":"2023-10-25T00:22:36.059972+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"73'5","reported_seq":109,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:53.546236+0000","last_change":"2023-10-25T00:22:53.546332+0000","last_active":"2023-10-25T00:22:53.546236+0000","last_peered":"2023-10-25T00:22:53.546236+0000","last_clean":"2023-10-25T00:22:53.546236+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-25T00:22:53.546236+0000","last_undegraded":"2023-10-25T00:22:53.546236+0000","last_fullsized":"2023-10-25T00:22:53.546236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.546215+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:16:55.756192+0000","scrub_duration":0.27862564699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"72'4690","reported_seq":4764,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427502+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-25T00:22:34.427502+0000","last_peered":"2023-10-25T00:22:34.427502+0000","last_clean":"2023-10-25T00:22:34.427502+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-25T00:22:34.427502+0000","last_undegraded":"2023-10-25T00:22:34.427502+0000","last_fullsized":"2023-10-25T00:22:34.427502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24994,"num_write_kb":341798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'4780","reported_seq":4868,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:09.092847+0000","last_change":"2023-10-25T00:23:09.093139+0000","last_active":"2023-10-25T00:23:09.092847+0000","last_peered":"2023-10-25T00:23:09.092847+0000","last_clean":"2023-10-25T00:23:09.092847+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-25T00:23:09.092847+0000","last_undegraded":"2023-10-25T00:23:09.092847+0000","last_fullsized":"2023-10-25T00:23:09.092847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4780","last_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_deep_scrub":"68'4780","last_deep_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.092796+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:58:49.347709+0000","scrub_duration":0.152359684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25083,"num_write_kb":293163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'3","reported_seq":89,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:48.328017+0000","last_change":"2023-10-25T00:22:48.328017+0000","last_active":"2023-10-25T00:22:48.328017+0000","last_peered":"2023-10-25T00:22:48.328017+0000","last_clean":"2023-10-25T00:22:48.328017+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-25T00:22:48.328017+0000","last_undegraded":"2023-10-25T00:22:48.328017+0000","last_fullsized":"2023-10-25T00:22:48.328017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.327778+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:58.116941+0000","scrub_duration":0.061328572999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"72'5","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.576377+0000","last_change":"2023-10-25T00:22:55.576688+0000","last_active":"2023-10-25T00:22:55.576377+0000","last_peered":"2023-10-25T00:22:55.576377+0000","last_clean":"2023-10-25T00:22:55.576377+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-25T00:22:55.576377+0000","last_undegraded":"2023-10-25T00:22:55.576377+0000","last_fullsized":"2023-10-25T00:22:55.576377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.576324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:43.752897+0000","scrub_duration":0.0045481469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"72'3596","reported_seq":3671,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060821+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-25T00:22:36.060821+0000","last_peered":"2023-10-25T00:22:36.060821+0000","last_clean":"2023-10-25T00:22:36.060821+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-25T00:22:36.060821+0000","last_undegraded":"2023-10-25T00:22:36.060821+0000","last_fullsized":"2023-10-25T00:22:36.060821+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":19820,"num_write_kb":259784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"68'4970","reported_seq":5057,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.999669+0000","last_change":"2023-10-25T00:22:55.999798+0000","last_active":"2023-10-25T00:22:55.999669+0000","last_peered":"2023-10-25T00:22:55.999669+0000","last_clean":"2023-10-25T00:22:55.999669+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-25T00:22:55.999669+0000","last_undegraded":"2023-10-25T00:22:55.999669+0000","last_fullsized":"2023-10-25T00:22:55.999669+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4970","last_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_deep_scrub":"68'4970","last_deep_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.999644+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:15:47.681358+0000","scrub_duration":0.177095113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26721,"num_write_kb":311966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"73'1466","reported_seq":1605,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.466683+0000","last_change":"2023-10-25T00:23:06.467052+0000","last_active":"2023-10-25T00:23:06.466683+0000","last_peered":"2023-10-25T00:23:06.466683+0000","last_clean":"2023-10-25T00:23:06.466683+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-25T00:23:06.466683+0000","last_undegraded":"2023-10-25T00:23:06.466683+0000","last_fullsized":"2023-10-25T00:23:06.466683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1466","last_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_deep_scrub":"73'1466","last_deep_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.466569+0000","objects_scrubbed":2,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:21.696239+0000","scrub_duration":0.039946036999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":47798,"num_write":1439,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"72'6","reported_seq":99,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427548+0000","last_change":"2023-10-24T23:53:11.446430+0000","last_active":"2023-10-25T00:22:34.427548+0000","last_peered":"2023-10-25T00:22:34.427548+0000","last_clean":"2023-10-25T00:22:34.427548+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-25T00:22:34.427548+0000","last_undegraded":"2023-10-25T00:22:34.427548+0000","last_fullsized":"2023-10-25T00:22:34.427548+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"72'3548","reported_seq":3622,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517358+0000","last_change":"2023-10-24T23:53:11.847110+0000","last_active":"2023-10-25T00:22:33.517358+0000","last_peered":"2023-10-25T00:22:33.517358+0000","last_clean":"2023-10-25T00:22:33.517358+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-25T00:22:33.517358+0000","last_undegraded":"2023-10-25T00:22:33.517358+0000","last_fullsized":"2023-10-25T00:22:33.517358+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20243,"num_write_kb":261539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"68'3393","reported_seq":3482,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.098602+0000","last_change":"2023-10-25T00:22:42.098602+0000","last_active":"2023-10-25T00:22:42.098602+0000","last_peered":"2023-10-25T00:22:42.098602+0000","last_clean":"2023-10-25T00:22:42.098602+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-25T00:22:42.098602+0000","last_undegraded":"2023-10-25T00:22:42.098602+0000","last_fullsized":"2023-10-25T00:22:42.098602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3393","last_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_deep_scrub":"68'3393","last_deep_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.098371+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:12.736136+0000","scrub_duration":0.153676797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17500,"num_write_kb":217638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"68'5","reported_seq":80,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522611+0000","last_change":"2023-10-24T23:53:02.336564+0000","last_active":"2023-10-25T00:22:36.522611+0000","last_peered":"2023-10-25T00:22:36.522611+0000","last_clean":"2023-10-25T00:22:36.522611+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-25T00:22:36.522611+0000","last_undegraded":"2023-10-25T00:22:36.522611+0000","last_fullsized":"2023-10-25T00:22:36.522611+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"72'3","reported_seq":75,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:48.308685+0000","last_change":"2023-10-25T00:22:49.292612+0000","last_active":"2023-10-25T00:22:48.308685+0000","last_peered":"2023-10-25T00:22:48.308685+0000","last_clean":"2023-10-25T00:22:48.308685+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-25T00:22:48.308685+0000","last_undegraded":"2023-10-25T00:22:48.308685+0000","last_fullsized":"2023-10-25T00:22:48.308685+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"72'3584","reported_seq":3658,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437442+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-25T00:22:37.437442+0000","last_peered":"2023-10-25T00:22:37.437442+0000","last_clean":"2023-10-25T00:22:37.437442+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-25T00:22:37.437442+0000","last_undegraded":"2023-10-25T00:22:37.437442+0000","last_fullsized":"2023-10-25T00:22:37.437442+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20504,"num_write_kb":254394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'5094","reported_seq":5185,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:51.484970+0000","last_change":"2023-10-25T00:22:51.485282+0000","last_active":"2023-10-25T00:22:51.484970+0000","last_peered":"2023-10-25T00:22:51.484970+0000","last_clean":"2023-10-25T00:22:51.484970+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-25T00:22:51.484970+0000","last_undegraded":"2023-10-25T00:22:51.484970+0000","last_fullsized":"2023-10-25T00:22:51.484970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5094","last_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_deep_scrub":"68'5094","last_deep_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.484917+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:22:43.899134+0000","scrub_duration":0.19299439099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27763,"num_write_kb":328490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"68'6","reported_seq":104,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.213667+0000","last_change":"2023-10-24T23:53:02.355262+0000","last_active":"2023-10-25T00:22:49.213667+0000","last_peered":"2023-10-25T00:22:49.213667+0000","last_clean":"2023-10-25T00:22:49.213667+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-25T00:22:49.213667+0000","last_undegraded":"2023-10-25T00:22:49.213667+0000","last_fullsized":"2023-10-25T00:22:49.213667+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"72'1438","reported_seq":1557,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427567+0000","last_change":"2023-10-24T23:53:11.423509+0000","last_active":"2023-10-25T00:22:34.427567+0000","last_peered":"2023-10-25T00:22:34.427567+0000","last_clean":"2023-10-25T00:22:34.427567+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-25T00:22:34.427567+0000","last_undegraded":"2023-10-25T00:22:34.427567+0000","last_fullsized":"2023-10-25T00:22:34.427567+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":1438,"log_dups_size":0,"ondisk_log_size":1438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":35275,"num_write":1412,"num_write_kb":4487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"72'3292","reported_seq":3367,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534103+0000","last_change":"2023-10-24T23:53:11.847179+0000","last_active":"2023-10-25T00:22:35.534103+0000","last_peered":"2023-10-25T00:22:35.534103+0000","last_clean":"2023-10-25T00:22:35.534103+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-25T00:22:35.534103+0000","last_undegraded":"2023-10-25T00:22:35.534103+0000","last_fullsized":"2023-10-25T00:22:35.534103+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17853,"num_write_kb":243602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'4242","reported_seq":4328,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:45.893397+0000","last_change":"2023-10-25T00:22:45.893397+0000","last_active":"2023-10-25T00:22:45.893397+0000","last_peered":"2023-10-25T00:22:45.893397+0000","last_clean":"2023-10-25T00:22:45.893397+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-25T00:22:45.893397+0000","last_undegraded":"2023-10-25T00:22:45.893397+0000","last_fullsized":"2023-10-25T00:22:45.893397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4242","last_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_deep_scrub":"67'4242","last_deep_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.893150+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:24:28.749675+0000","scrub_duration":0.21189509300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22687,"num_write_kb":275346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"68'7","reported_seq":128,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.427815+0000","last_change":"2023-10-24T23:53:02.359479+0000","last_active":"2023-10-25T00:22:52.427815+0000","last_peered":"2023-10-25T00:22:52.427815+0000","last_clean":"2023-10-25T00:22:52.427815+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-25T00:22:52.427815+0000","last_undegraded":"2023-10-25T00:22:52.427815+0000","last_fullsized":"2023-10-25T00:22:52.427815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"72'5","reported_seq":76,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427402+0000","last_change":"2023-10-24T23:53:11.423305+0000","last_active":"2023-10-25T00:22:34.427402+0000","last_peered":"2023-10-25T00:22:34.427402+0000","last_clean":"2023-10-25T00:22:34.427402+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-25T00:22:34.427402+0000","last_undegraded":"2023-10-25T00:22:34.427402+0000","last_fullsized":"2023-10-25T00:22:34.427402+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"72'4091","reported_seq":4165,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523278+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-25T00:22:36.523278+0000","last_peered":"2023-10-25T00:22:36.523278+0000","last_clean":"2023-10-25T00:22:36.523278+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-25T00:22:36.523278+0000","last_undegraded":"2023-10-25T00:22:36.523278+0000","last_fullsized":"2023-10-25T00:22:36.523278+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22779,"num_write_kb":288098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'4310","reported_seq":4385,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522869+0000","last_change":"2023-10-24T23:53:02.719413+0000","last_active":"2023-10-25T00:22:36.522869+0000","last_peered":"2023-10-25T00:22:36.522869+0000","last_clean":"2023-10-25T00:22:36.522869+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-25T00:22:36.522869+0000","last_undegraded":"2023-10-25T00:22:36.522869+0000","last_fullsized":"2023-10-25T00:22:36.522869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23791,"num_write_kb":283914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'4","reported_seq":102,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.212618+0000","last_change":"2023-10-24T23:53:02.358216+0000","last_active":"2023-10-25T00:22:49.212618+0000","last_peered":"2023-10-25T00:22:49.212618+0000","last_clean":"2023-10-25T00:22:49.212618+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-25T00:22:49.212618+0000","last_undegraded":"2023-10-25T00:22:49.212618+0000","last_fullsized":"2023-10-25T00:22:49.212618+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"72'2","reported_seq":84,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:38.034628+0000","last_change":"2023-10-25T00:22:38.034628+0000","last_active":"2023-10-25T00:22:38.034628+0000","last_peered":"2023-10-25T00:22:38.034628+0000","last_clean":"2023-10-25T00:22:38.034628+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-25T00:22:38.034628+0000","last_undegraded":"2023-10-25T00:22:38.034628+0000","last_fullsized":"2023-10-25T00:22:38.034628+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.034499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:09.450121+0000","scrub_duration":0.14675892500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"72'3695","reported_seq":3770,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437533+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-25T00:22:37.437533+0000","last_peered":"2023-10-25T00:22:37.437533+0000","last_clean":"2023-10-25T00:22:37.437533+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-25T00:22:37.437533+0000","last_undegraded":"2023-10-25T00:22:37.437533+0000","last_fullsized":"2023-10-25T00:22:37.437533+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21426,"num_write_kb":254037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"72'3","reported_seq":84,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.331112+0000","last_change":"2023-10-25T00:22:42.331112+0000","last_active":"2023-10-25T00:22:42.331112+0000","last_peered":"2023-10-25T00:22:42.331112+0000","last_clean":"2023-10-25T00:22:42.331112+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-25T00:22:42.331112+0000","last_undegraded":"2023-10-25T00:22:42.331112+0000","last_fullsized":"2023-10-25T00:22:42.331112+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.330786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:23:56.573805+0000","scrub_duration":0.0043674530000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"72'19","reported_seq":217,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.643945+0000","last_change":"2023-10-25T00:22:47.253320+0000","last_active":"2023-10-25T00:22:52.643945+0000","last_peered":"2023-10-25T00:22:52.643945+0000","last_clean":"2023-10-25T00:22:52.643945+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-25T00:22:52.643945+0000","last_undegraded":"2023-10-25T00:22:52.643945+0000","last_fullsized":"2023-10-25T00:22:52.643945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_deep_scrub":"72'19","last_deep_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.252995+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:03.720456+0000","scrub_duration":0.018599613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":339,"num_read_kb":339,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'5249","reported_seq":5342,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.439199+0000","last_change":"2023-10-25T00:22:52.439461+0000","last_active":"2023-10-25T00:22:52.439199+0000","last_peered":"2023-10-25T00:22:52.439199+0000","last_clean":"2023-10-25T00:22:52.439199+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-25T00:22:52.439199+0000","last_undegraded":"2023-10-25T00:22:52.439199+0000","last_fullsized":"2023-10-25T00:22:52.439199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5249","last_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_deep_scrub":"68'5249","last_deep_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.439150+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:19:03.730037+0000","scrub_duration":0.077069795999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":27646,"num_write_kb":332285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"72'4380","reported_seq":4452,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427518+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-25T00:22:34.427518+0000","last_peered":"2023-10-25T00:22:34.427518+0000","last_clean":"2023-10-25T00:22:34.427518+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-25T00:22:34.427518+0000","last_undegraded":"2023-10-25T00:22:34.427518+0000","last_fullsized":"2023-10-25T00:22:34.427518+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23662,"num_write_kb":324338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"72'5","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:50.617951+0000","last_change":"2023-10-25T00:22:50.618067+0000","last_active":"2023-10-25T00:22:50.617951+0000","last_peered":"2023-10-25T00:22:50.617951+0000","last_clean":"2023-10-25T00:22:50.617951+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-25T00:22:50.617951+0000","last_undegraded":"2023-10-25T00:22:50.617951+0000","last_fullsized":"2023-10-25T00:22:50.617951+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.617931+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:24.381368+0000","scrub_duration":0.234626104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"68'5","reported_seq":79,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437000+0000","last_change":"2023-10-24T23:53:02.358377+0000","last_active":"2023-10-25T00:22:37.437000+0000","last_peered":"2023-10-25T00:22:37.437000+0000","last_clean":"2023-10-25T00:22:37.437000+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-25T00:22:37.437000+0000","last_undegraded":"2023-10-25T00:22:37.437000+0000","last_fullsized":"2023-10-25T00:22:37.437000+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'4442","reported_seq":4516,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517147+0000","last_change":"2023-10-24T23:53:02.727579+0000","last_active":"2023-10-25T00:22:33.517147+0000","last_peered":"2023-10-25T00:22:33.517147+0000","last_clean":"2023-10-25T00:22:33.517147+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-25T00:22:33.517147+0000","last_undegraded":"2023-10-25T00:22:33.517147+0000","last_fullsized":"2023-10-25T00:22:33.517147+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22810,"num_write_kb":296225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"72'4076","reported_seq":4150,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060841+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-25T00:22:36.060841+0000","last_peered":"2023-10-25T00:22:36.060841+0000","last_clean":"2023-10-25T00:22:36.060841+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-25T00:22:36.060841+0000","last_undegraded":"2023-10-25T00:22:36.060841+0000","last_fullsized":"2023-10-25T00:22:36.060841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22055,"num_write_kb":297609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"68'3","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.516763+0000","last_change":"2023-10-24T23:53:02.354771+0000","last_active":"2023-10-25T00:22:33.516763+0000","last_peered":"2023-10-25T00:22:33.516763+0000","last_clean":"2023-10-25T00:22:33.516763+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-25T00:22:33.516763+0000","last_undegraded":"2023-10-25T00:22:33.516763+0000","last_fullsized":"2023-10-25T00:22:33.516763+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'4101","reported_seq":4187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:50.337590+0000","last_change":"2023-10-25T00:22:50.337815+0000","last_active":"2023-10-25T00:22:50.337590+0000","last_peered":"2023-10-25T00:22:50.337590+0000","last_clean":"2023-10-25T00:22:50.337590+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-25T00:22:50.337590+0000","last_undegraded":"2023-10-25T00:22:50.337590+0000","last_fullsized":"2023-10-25T00:22:50.337590+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.337536+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:01.275651+0000","scrub_duration":0.055303021000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22343,"num_write_kb":253486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"72'397","reported_seq":590,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:45.554898+0000","last_change":"2023-10-25T00:22:45.554898+0000","last_active":"2023-10-25T00:22:45.554898+0000","last_peered":"2023-10-25T00:22:45.554898+0000","last_clean":"2023-10-25T00:22:45.554898+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-25T00:22:45.554898+0000","last_undegraded":"2023-10-25T00:22:45.554898+0000","last_fullsized":"2023-10-25T00:22:45.554898+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'397","last_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_deep_scrub":"72'397","last_deep_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.554526+0000","objects_scrubbed":4,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:41:32.056818+0000","scrub_duration":0.10248053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":86197,"num_write":389,"num_write_kb":4306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"72'4606","reported_seq":4678,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517449+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-25T00:22:33.517449+0000","last_peered":"2023-10-25T00:22:33.517449+0000","last_clean":"2023-10-25T00:22:33.517449+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-25T00:22:33.517449+0000","last_undegraded":"2023-10-25T00:22:33.517449+0000","last_fullsized":"2023-10-25T00:22:33.517449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25763,"num_write_kb":316477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"73'14","reported_seq":145,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.152781+0000","last_change":"2023-10-25T00:23:16.153043+0000","last_active":"2023-10-25T00:23:16.152781+0000","last_peered":"2023-10-25T00:23:16.152781+0000","last_clean":"2023-10-25T00:23:16.152781+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-25T00:23:16.152781+0000","last_undegraded":"2023-10-25T00:23:16.152781+0000","last_fullsized":"2023-10-25T00:23:16.152781+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_deep_scrub":"73'14","last_deep_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.152723+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:32:30.128331+0000","scrub_duration":0.010186763999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'5680","reported_seq":5769,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:54.585888+0000","last_change":"2023-10-25T00:22:54.586176+0000","last_active":"2023-10-25T00:22:54.585888+0000","last_peered":"2023-10-25T00:22:54.585888+0000","last_clean":"2023-10-25T00:22:54.585888+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-25T00:22:54.585888+0000","last_undegraded":"2023-10-25T00:22:54.585888+0000","last_fullsized":"2023-10-25T00:22:54.585888+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5680","last_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_deep_scrub":"68'5680","last_deep_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.585866+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:49.342364+0000","scrub_duration":0.20835251299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29749,"num_write_kb":360962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"73'40","reported_seq":187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:59.279339+0000","last_change":"2023-10-25T00:22:59.279527+0000","last_active":"2023-10-25T00:22:59.279339+0000","last_peered":"2023-10-25T00:22:59.279339+0000","last_clean":"2023-10-25T00:22:59.279339+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-25T00:22:59.279339+0000","last_undegraded":"2023-10-25T00:22:59.279339+0000","last_fullsized":"2023-10-25T00:22:59.279339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'40","last_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_deep_scrub":"73'40","last_deep_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.279288+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:52.911132+0000","scrub_duration":0.0094949800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":39,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"72'4808","reported_seq":4897,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.790951+0000","last_change":"2023-10-25T00:23:14.791099+0000","last_active":"2023-10-25T00:23:14.790951+0000","last_peered":"2023-10-25T00:23:14.790951+0000","last_clean":"2023-10-25T00:23:14.790951+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-25T00:23:14.790951+0000","last_undegraded":"2023-10-25T00:23:14.790951+0000","last_fullsized":"2023-10-25T00:23:14.790951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4808","last_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_deep_scrub":"72'4808","last_deep_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.790929+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:56:36.982903+0000","scrub_duration":0.29119922999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26552,"num_write_kb":336125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"68'4747","reported_seq":4835,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.496098+0000","last_change":"2023-10-25T00:23:00.496309+0000","last_active":"2023-10-25T00:23:00.496098+0000","last_peered":"2023-10-25T00:23:00.496098+0000","last_clean":"2023-10-25T00:23:00.496098+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-25T00:23:00.496098+0000","last_undegraded":"2023-10-25T00:23:00.496098+0000","last_fullsized":"2023-10-25T00:23:00.496098+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4747","last_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_deep_scrub":"68'4747","last_deep_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.496060+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:56.531125+0000","scrub_duration":0.089432630999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24847,"num_write_kb":316685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"72'4283","reported_seq":4370,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:08.728587+0000","last_change":"2023-10-25T00:23:08.728718+0000","last_active":"2023-10-25T00:23:08.728587+0000","last_peered":"2023-10-25T00:23:08.728587+0000","last_clean":"2023-10-25T00:23:08.728587+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-25T00:23:08.728587+0000","last_undegraded":"2023-10-25T00:23:08.728587+0000","last_fullsized":"2023-10-25T00:23:08.728587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4283","last_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_deep_scrub":"72'4283","last_deep_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_clean_scrub_stamp":"2023-10-25T00:23:08.728567+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:33.655445+0000","scrub_duration":0.26846236600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23853,"num_write_kb":296504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"68'4036","reported_seq":4124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:39.642537+0000","last_change":"2023-10-25T00:22:39.642537+0000","last_active":"2023-10-25T00:22:39.642537+0000","last_peered":"2023-10-25T00:22:39.642537+0000","last_clean":"2023-10-25T00:22:39.642537+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-25T00:22:39.642537+0000","last_undegraded":"2023-10-25T00:22:39.642537+0000","last_fullsized":"2023-10-25T00:22:39.642537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4036","last_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_deep_scrub":"68'4036","last_deep_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.642342+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:02.362976+0000","scrub_duration":0.227585228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21233,"num_write_kb":273467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"72'4276","reported_seq":4351,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965799+0000","last_change":"2023-10-24T23:53:11.805752+0000","last_active":"2023-10-25T00:22:33.965799+0000","last_peered":"2023-10-25T00:22:33.965799+0000","last_clean":"2023-10-25T00:22:33.965799+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-25T00:22:33.965799+0000","last_undegraded":"2023-10-25T00:22:33.965799+0000","last_fullsized":"2023-10-25T00:22:33.965799+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23362,"num_write_kb":310817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"68'6201","reported_seq":6280,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060597+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-25T00:22:36.060597+0000","last_peered":"2023-10-25T00:22:36.060597+0000","last_clean":"2023-10-25T00:22:36.060597+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-25T00:22:36.060597+0000","last_undegraded":"2023-10-25T00:22:36.060597+0000","last_fullsized":"2023-10-25T00:22:36.060597+0000","mapping_epoch":21,"log_start":"62'1000","ondisk_log_start":"62'1000","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33486,"num_write_kb":384441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"72'4018","reported_seq":4094,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437698+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-25T00:22:37.437698+0000","last_peered":"2023-10-25T00:22:37.437698+0000","last_clean":"2023-10-25T00:22:37.437698+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-25T00:22:37.437698+0000","last_undegraded":"2023-10-25T00:22:37.437698+0000","last_fullsized":"2023-10-25T00:22:37.437698+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":21920,"num_write_kb":288651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"68'4522","reported_seq":4600,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534002+0000","last_change":"2023-10-24T23:53:02.714677+0000","last_active":"2023-10-25T00:22:35.534002+0000","last_peered":"2023-10-25T00:22:35.534002+0000","last_clean":"2023-10-25T00:22:35.534002+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-25T00:22:35.534002+0000","last_undegraded":"2023-10-25T00:22:35.534002+0000","last_fullsized":"2023-10-25T00:22:35.534002+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24476,"num_write_kb":282500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"72'4209","reported_seq":4281,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965838+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-25T00:22:33.965838+0000","last_peered":"2023-10-25T00:22:33.965838+0000","last_clean":"2023-10-25T00:22:33.965838+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-25T00:22:33.965838+0000","last_undegraded":"2023-10-25T00:22:33.965838+0000","last_fullsized":"2023-10-25T00:22:33.965838+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24141,"num_write_kb":281325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"68'4915","reported_seq":5001,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:38.600278+0000","last_change":"2023-10-25T00:22:38.600278+0000","last_active":"2023-10-25T00:22:38.600278+0000","last_peered":"2023-10-25T00:22:38.600278+0000","last_clean":"2023-10-25T00:22:38.600278+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-25T00:22:38.600278+0000","last_undegraded":"2023-10-25T00:22:38.600278+0000","last_fullsized":"2023-10-25T00:22:38.600278+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4915","last_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_deep_scrub":"68'4915","last_deep_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.600135+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:42:30.671003+0000","scrub_duration":0.18827047799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26161,"num_write_kb":314084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"72'4933","reported_seq":5005,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978871+0000","last_change":"2023-10-24T23:53:11.814368+0000","last_active":"2023-10-25T00:22:36.978871+0000","last_peered":"2023-10-25T00:22:36.978871+0000","last_clean":"2023-10-25T00:22:36.978871+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-25T00:22:36.978871+0000","last_undegraded":"2023-10-25T00:22:36.978871+0000","last_fullsized":"2023-10-25T00:22:36.978871+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26695,"num_write_kb":352905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"68'5055","reported_seq":5142,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:43.444464+0000","last_change":"2023-10-25T00:22:43.444464+0000","last_active":"2023-10-25T00:22:43.444464+0000","last_peered":"2023-10-25T00:22:43.444464+0000","last_clean":"2023-10-25T00:22:43.444464+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-25T00:22:43.444464+0000","last_undegraded":"2023-10-25T00:22:43.444464+0000","last_fullsized":"2023-10-25T00:22:43.444464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5055","last_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_deep_scrub":"68'5055","last_deep_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_clean_scrub_stamp":"2023-10-25T00:22:43.444210+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:13:34.310177+0000","scrub_duration":0.081291211000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27516,"num_write_kb":327406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"72'4083","reported_seq":4159,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523384+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-25T00:22:36.523384+0000","last_peered":"2023-10-25T00:22:36.523384+0000","last_clean":"2023-10-25T00:22:36.523384+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-25T00:22:36.523384+0000","last_undegraded":"2023-10-25T00:22:36.523384+0000","last_fullsized":"2023-10-25T00:22:36.523384+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":22892,"num_write_kb":296872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"68'4777","reported_seq":4854,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965628+0000","last_change":"2023-10-24T23:53:02.716414+0000","last_active":"2023-10-25T00:22:33.965628+0000","last_peered":"2023-10-25T00:22:33.965628+0000","last_clean":"2023-10-25T00:22:33.965628+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-25T00:22:33.965628+0000","last_undegraded":"2023-10-25T00:22:33.965628+0000","last_fullsized":"2023-10-25T00:22:33.965628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25812,"num_write_kb":300181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"72'4618","reported_seq":4695,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.979240+0000","last_change":"2023-10-24T23:53:11.812552+0000","last_active":"2023-10-25T00:22:36.979240+0000","last_peered":"2023-10-25T00:22:36.979240+0000","last_clean":"2023-10-25T00:22:36.979240+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-25T00:22:36.979240+0000","last_undegraded":"2023-10-25T00:22:36.979240+0000","last_fullsized":"2023-10-25T00:22:36.979240+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":26180,"num_write_kb":326342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"68'5340","reported_seq":5426,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.520048+0000","last_change":"2023-10-25T00:22:42.520048+0000","last_active":"2023-10-25T00:22:42.520048+0000","last_peered":"2023-10-25T00:22:42.520048+0000","last_clean":"2023-10-25T00:22:42.520048+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-25T00:22:42.520048+0000","last_undegraded":"2023-10-25T00:22:42.520048+0000","last_fullsized":"2023-10-25T00:22:42.520048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5340","last_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_deep_scrub":"68'5340","last_deep_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.519886+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.832889+0000","scrub_duration":0.129134375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28701,"num_write_kb":348521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"72'4171","reported_seq":4245,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060881+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-25T00:22:36.060881+0000","last_peered":"2023-10-25T00:22:36.060881+0000","last_clean":"2023-10-25T00:22:36.060881+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-25T00:22:36.060881+0000","last_undegraded":"2023-10-25T00:22:36.060881+0000","last_fullsized":"2023-10-25T00:22:36.060881+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22551,"num_write_kb":303498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"68'4493","reported_seq":4581,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:01.322975+0000","last_change":"2023-10-25T00:23:01.323167+0000","last_active":"2023-10-25T00:23:01.322975+0000","last_peered":"2023-10-25T00:23:01.322975+0000","last_clean":"2023-10-25T00:23:01.322975+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-25T00:23:01.322975+0000","last_undegraded":"2023-10-25T00:23:01.322975+0000","last_fullsized":"2023-10-25T00:23:01.322975+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4493","last_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_deep_scrub":"68'4493","last_deep_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_clean_scrub_stamp":"2023-10-25T00:23:01.322941+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:49.029022+0000","scrub_duration":0.13616751799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23797,"num_write_kb":300260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"72'4763","reported_seq":4836,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.226383+0000","last_change":"2023-10-24T23:53:44.474203+0000","last_active":"2023-10-25T00:22:49.226383+0000","last_peered":"2023-10-25T00:22:49.226383+0000","last_clean":"2023-10-25T00:22:49.226383+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-25T00:22:49.226383+0000","last_undegraded":"2023-10-25T00:22:49.226383+0000","last_fullsized":"2023-10-25T00:22:49.226383+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":25751,"num_write_kb":345225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"68'4946","reported_seq":5022,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.358533+0000","last_change":"2023-10-25T00:22:49.234045+0000","last_active":"2023-10-25T00:22:37.358533+0000","last_peered":"2023-10-25T00:22:37.358533+0000","last_clean":"2023-10-25T00:22:37.358533+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-25T00:22:37.358533+0000","last_undegraded":"2023-10-25T00:22:37.358533+0000","last_fullsized":"2023-10-25T00:22:37.358533+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26583,"num_write_kb":315633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"72'4352","reported_seq":4425,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427628+0000","last_change":"2023-10-24T23:53:11.857576+0000","last_active":"2023-10-25T00:22:34.427628+0000","last_peered":"2023-10-25T00:22:34.427628+0000","last_clean":"2023-10-25T00:22:34.427628+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-25T00:22:34.427628+0000","last_undegraded":"2023-10-25T00:22:34.427628+0000","last_fullsized":"2023-10-25T00:22:34.427628+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23957,"num_write_kb":301052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"68'5026","reported_seq":5101,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517220+0000","last_change":"2023-10-24T23:53:02.726357+0000","last_active":"2023-10-25T00:22:33.517220+0000","last_peered":"2023-10-25T00:22:33.517220+0000","last_clean":"2023-10-25T00:22:33.517220+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-25T00:22:33.517220+0000","last_undegraded":"2023-10-25T00:22:33.517220+0000","last_fullsized":"2023-10-25T00:22:33.517220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26401,"num_write_kb":310586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"72'4393","reported_seq":4482,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.556819+0000","last_change":"2023-10-25T00:23:07.556990+0000","last_active":"2023-10-25T00:23:07.556819+0000","last_peered":"2023-10-25T00:23:07.556819+0000","last_clean":"2023-10-25T00:23:07.556819+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-25T00:23:07.556819+0000","last_undegraded":"2023-10-25T00:23:07.556819+0000","last_fullsized":"2023-10-25T00:23:07.556819+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4393","last_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_deep_scrub":"72'4393","last_deep_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.556784+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:21:00.928852+0000","scrub_duration":0.35410455800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":24752,"num_write_kb":296109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"68'5620","reported_seq":5710,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.194060+0000","last_change":"2023-10-25T00:23:12.194324+0000","last_active":"2023-10-25T00:23:12.194060+0000","last_peered":"2023-10-25T00:23:12.194060+0000","last_clean":"2023-10-25T00:23:12.194060+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-25T00:23:12.194060+0000","last_undegraded":"2023-10-25T00:23:12.194060+0000","last_fullsized":"2023-10-25T00:23:12.194060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5620","last_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_deep_scrub":"68'5620","last_deep_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.194008+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:33.868259+0000","scrub_duration":0.086061930999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29863,"num_write_kb":337169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"72'4340","reported_seq":4416,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978854+0000","last_change":"2023-10-24T23:53:11.839281+0000","last_active":"2023-10-25T00:22:36.978854+0000","last_peered":"2023-10-25T00:22:36.978854+0000","last_clean":"2023-10-25T00:22:36.978854+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-25T00:22:36.978854+0000","last_undegraded":"2023-10-25T00:22:36.978854+0000","last_fullsized":"2023-10-25T00:22:36.978854+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":36,"num_write":23367,"num_write_kb":302669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"68'4464","reported_seq":4549,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.904552+0000","last_change":"2023-10-25T00:22:57.904669+0000","last_active":"2023-10-25T00:22:57.904552+0000","last_peered":"2023-10-25T00:22:57.904552+0000","last_clean":"2023-10-25T00:22:57.904552+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-25T00:22:57.904552+0000","last_undegraded":"2023-10-25T00:22:57.904552+0000","last_fullsized":"2023-10-25T00:22:57.904552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4464","last_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_deep_scrub":"68'4464","last_deep_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.904532+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:17:44.026155+0000","scrub_duration":0.14132308199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24372,"num_write_kb":278074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"72'5589","reported_seq":5661,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437759+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-25T00:22:37.437759+0000","last_peered":"2023-10-25T00:22:37.437759+0000","last_clean":"2023-10-25T00:22:37.437759+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-25T00:22:37.437759+0000","last_undegraded":"2023-10-25T00:22:37.437759+0000","last_fullsized":"2023-10-25T00:22:37.437759+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":30206,"num_write_kb":399557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"68'4542","reported_seq":4628,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:09.344932+0000","last_change":"2023-10-25T00:23:09.345212+0000","last_active":"2023-10-25T00:23:09.344932+0000","last_peered":"2023-10-25T00:23:09.344932+0000","last_clean":"2023-10-25T00:23:09.344932+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-25T00:23:09.344932+0000","last_undegraded":"2023-10-25T00:23:09.344932+0000","last_fullsized":"2023-10-25T00:23:09.344932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4542","last_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_deep_scrub":"68'4542","last_deep_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.344860+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:16:09.325372+0000","scrub_duration":0.066583699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23889,"num_write_kb":292323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"72'3430","reported_seq":3504,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.978912+0000","last_change":"2023-10-24T23:53:11.839515+0000","last_active":"2023-10-25T00:22:36.978912+0000","last_peered":"2023-10-25T00:22:36.978912+0000","last_clean":"2023-10-25T00:22:36.978912+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-25T00:22:36.978912+0000","last_undegraded":"2023-10-25T00:22:36.978912+0000","last_fullsized":"2023-10-25T00:22:36.978912+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19411,"num_write_kb":241862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"68'4719","reported_seq":4805,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:58.902820+0000","last_change":"2023-10-25T00:22:58.903089+0000","last_active":"2023-10-25T00:22:58.902820+0000","last_peered":"2023-10-25T00:22:58.902820+0000","last_clean":"2023-10-25T00:22:58.902820+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-25T00:22:58.902820+0000","last_undegraded":"2023-10-25T00:22:58.902820+0000","last_fullsized":"2023-10-25T00:22:58.902820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4719","last_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_deep_scrub":"68'4719","last_deep_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.902769+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:01.236787+0000","scrub_duration":0.136999537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25020,"num_write_kb":305197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"72'4169","reported_seq":4240,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427758+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-25T00:22:34.427758+0000","last_peered":"2023-10-25T00:22:34.427758+0000","last_clean":"2023-10-25T00:22:34.427758+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-25T00:22:34.427758+0000","last_undegraded":"2023-10-25T00:22:34.427758+0000","last_fullsized":"2023-10-25T00:22:34.427758+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23235,"num_write_kb":306175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"68'5952","reported_seq":6046,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.017394+0000","last_change":"2023-10-25T00:23:05.017693+0000","last_active":"2023-10-25T00:23:05.017394+0000","last_peered":"2023-10-25T00:23:05.017394+0000","last_clean":"2023-10-25T00:23:05.017394+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-25T00:23:05.017394+0000","last_undegraded":"2023-10-25T00:23:05.017394+0000","last_fullsized":"2023-10-25T00:23:05.017394+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5952","last_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_deep_scrub":"68'5952","last_deep_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.017340+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:31.140541+0000","scrub_duration":0.055624675999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32341,"num_write_kb":375441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"72'3717","reported_seq":3788,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060966+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-25T00:22:36.060966+0000","last_peered":"2023-10-25T00:22:36.060966+0000","last_clean":"2023-10-25T00:22:36.060966+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-25T00:22:36.060966+0000","last_undegraded":"2023-10-25T00:22:36.060966+0000","last_fullsized":"2023-10-25T00:22:36.060966+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21101,"num_write_kb":269108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"68'4825","reported_seq":4912,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.089727+0000","last_change":"2023-10-25T00:23:06.089847+0000","last_active":"2023-10-25T00:23:06.089727+0000","last_peered":"2023-10-25T00:23:06.089727+0000","last_clean":"2023-10-25T00:23:06.089727+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-25T00:23:06.089727+0000","last_undegraded":"2023-10-25T00:23:06.089727+0000","last_fullsized":"2023-10-25T00:23:06.089727+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4825","last_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_deep_scrub":"68'4825","last_deep_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.089705+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:04:51.412522+0000","scrub_duration":0.15071393499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25319,"num_write_kb":294113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"72'3714","reported_seq":3789,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.215543+0000","last_change":"2023-10-25T00:23:12.182256+0000","last_active":"2023-10-25T00:22:49.215543+0000","last_peered":"2023-10-25T00:22:49.215543+0000","last_clean":"2023-10-25T00:22:49.215543+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-25T00:22:49.215543+0000","last_undegraded":"2023-10-25T00:22:49.215543+0000","last_fullsized":"2023-10-25T00:22:49.215543+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21148,"num_write_kb":265324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"68'4878","reported_seq":4966,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.227110+0000","last_change":"2023-10-25T00:22:49.227455+0000","last_active":"2023-10-25T00:22:49.227110+0000","last_peered":"2023-10-25T00:22:49.227110+0000","last_clean":"2023-10-25T00:22:49.227110+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-25T00:22:49.227110+0000","last_undegraded":"2023-10-25T00:22:49.227110+0000","last_fullsized":"2023-10-25T00:22:49.227110+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4878","last_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_deep_scrub":"68'4878","last_deep_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_clean_scrub_stamp":"2023-10-25T00:22:49.227012+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:58:21.157778+0000","scrub_duration":0.13340163999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26154,"num_write_kb":305558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"72'3434","reported_seq":3506,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437787+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-25T00:22:37.437787+0000","last_peered":"2023-10-25T00:22:37.437787+0000","last_clean":"2023-10-25T00:22:37.437787+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-25T00:22:37.437787+0000","last_undegraded":"2023-10-25T00:22:37.437787+0000","last_fullsized":"2023-10-25T00:22:37.437787+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18449,"num_write_kb":262373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"68'4672","reported_seq":4748,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522905+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-25T00:22:36.522905+0000","last_peered":"2023-10-25T00:22:36.522905+0000","last_clean":"2023-10-25T00:22:36.522905+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-25T00:22:36.522905+0000","last_undegraded":"2023-10-25T00:22:36.522905+0000","last_fullsized":"2023-10-25T00:22:36.522905+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25223,"num_write_kb":293792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"72'4013","reported_seq":4090,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523380+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-25T00:22:36.523380+0000","last_peered":"2023-10-25T00:22:36.523380+0000","last_clean":"2023-10-25T00:22:36.523380+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-25T00:22:36.523380+0000","last_undegraded":"2023-10-25T00:22:36.523380+0000","last_fullsized":"2023-10-25T00:22:36.523380+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":22066,"num_write_kb":297298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"68'3918","reported_seq":3993,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.292751+0000","last_change":"2023-10-25T00:22:57.448743+0000","last_active":"2023-10-25T00:22:49.292751+0000","last_peered":"2023-10-25T00:22:49.292751+0000","last_clean":"2023-10-25T00:22:49.292751+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-25T00:22:49.292751+0000","last_undegraded":"2023-10-25T00:22:49.292751+0000","last_fullsized":"2023-10-25T00:22:49.292751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21675,"num_write_kb":247000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"72'4401","reported_seq":4474,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523617+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-25T00:22:36.523617+0000","last_peered":"2023-10-25T00:22:36.523617+0000","last_clean":"2023-10-25T00:22:36.523617+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-25T00:22:36.523617+0000","last_undegraded":"2023-10-25T00:22:36.523617+0000","last_fullsized":"2023-10-25T00:22:36.523617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":24294,"num_write_kb":326749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"68'3970","reported_seq":4055,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.428569+0000","last_change":"2023-10-25T00:23:12.428867+0000","last_active":"2023-10-25T00:23:12.428569+0000","last_peered":"2023-10-25T00:23:12.428569+0000","last_clean":"2023-10-25T00:23:12.428569+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-25T00:23:12.428569+0000","last_undegraded":"2023-10-25T00:23:12.428569+0000","last_fullsized":"2023-10-25T00:23:12.428569+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3970","last_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_deep_scrub":"68'3970","last_deep_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.428494+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:07.899437+0000","scrub_duration":0.13106953599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21202,"num_write_kb":257085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"72'5559","reported_seq":5638,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517467+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-25T00:22:33.517467+0000","last_peered":"2023-10-25T00:22:33.517467+0000","last_clean":"2023-10-25T00:22:33.517467+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-25T00:22:33.517467+0000","last_undegraded":"2023-10-25T00:22:33.517467+0000","last_fullsized":"2023-10-25T00:22:33.517467+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":31202,"num_write_kb":391572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"68'4936","reported_seq":5022,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.021130+0000","last_change":"2023-10-25T00:23:02.021272+0000","last_active":"2023-10-25T00:23:02.021130+0000","last_peered":"2023-10-25T00:23:02.021130+0000","last_clean":"2023-10-25T00:23:02.021130+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-25T00:23:02.021130+0000","last_undegraded":"2023-10-25T00:23:02.021130+0000","last_fullsized":"2023-10-25T00:23:02.021130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4936","last_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_deep_scrub":"68'4936","last_deep_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.021108+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:22.652380+0000","scrub_duration":0.17348896599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26134,"num_write_kb":309652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"72'3558","reported_seq":3634,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517421+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-25T00:22:33.517421+0000","last_peered":"2023-10-25T00:22:33.517421+0000","last_clean":"2023-10-25T00:22:33.517421+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-25T00:22:33.517421+0000","last_undegraded":"2023-10-25T00:22:33.517421+0000","last_fullsized":"2023-10-25T00:22:33.517421+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":34,"num_write":19715,"num_write_kb":243025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"68'5282","reported_seq":5368,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.290128+0000","last_change":"2023-10-25T00:23:14.290293+0000","last_active":"2023-10-25T00:23:14.290128+0000","last_peered":"2023-10-25T00:23:14.290128+0000","last_clean":"2023-10-25T00:23:14.290128+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-25T00:23:14.290128+0000","last_undegraded":"2023-10-25T00:23:14.290128+0000","last_fullsized":"2023-10-25T00:23:14.290128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5282","last_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_deep_scrub":"68'5282","last_deep_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.290099+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:44:38.139530+0000","scrub_duration":0.10140234200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27848,"num_write_kb":324561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"72'3914","reported_seq":3991,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523611+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-25T00:22:36.523611+0000","last_peered":"2023-10-25T00:22:36.523611+0000","last_clean":"2023-10-25T00:22:36.523611+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-25T00:22:36.523611+0000","last_undegraded":"2023-10-25T00:22:36.523611+0000","last_fullsized":"2023-10-25T00:22:36.523611+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":20938,"num_write_kb":288796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"68'6616","reported_seq":6706,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.985123+0000","last_change":"2023-10-25T00:22:57.985278+0000","last_active":"2023-10-25T00:22:57.985123+0000","last_peered":"2023-10-25T00:22:57.985123+0000","last_clean":"2023-10-25T00:22:57.985123+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-25T00:22:57.985123+0000","last_undegraded":"2023-10-25T00:22:57.985123+0000","last_fullsized":"2023-10-25T00:22:57.985123+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6616","last_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_deep_scrub":"68'6616","last_deep_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.985096+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:14:23.491729+0000","scrub_duration":0.050039131000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36221,"num_write_kb":423916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"72'3536","reported_seq":3609,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.233558+0000","last_change":"2023-10-24T23:53:11.811405+0000","last_active":"2023-10-25T00:22:49.233558+0000","last_peered":"2023-10-25T00:22:49.233558+0000","last_clean":"2023-10-25T00:22:49.233558+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-25T00:22:49.233558+0000","last_undegraded":"2023-10-25T00:22:49.233558+0000","last_fullsized":"2023-10-25T00:22:49.233558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19276,"num_write_kb":253506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"68'4980","reported_seq":5064,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:10.321019+0000","last_change":"2023-10-25T00:23:10.321202+0000","last_active":"2023-10-25T00:23:10.321019+0000","last_peered":"2023-10-25T00:23:10.321019+0000","last_clean":"2023-10-25T00:23:10.321019+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-25T00:23:10.321019+0000","last_undegraded":"2023-10-25T00:23:10.321019+0000","last_fullsized":"2023-10-25T00:23:10.321019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4980","last_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_deep_scrub":"68'4980","last_deep_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:10.320988+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:34.745429+0000","scrub_duration":0.13750111800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26525,"num_write_kb":306963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"72'4160","reported_seq":4235,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.292308+0000","last_change":"2023-10-24T23:53:11.837892+0000","last_active":"2023-10-25T00:22:49.292308+0000","last_peered":"2023-10-25T00:22:49.292308+0000","last_clean":"2023-10-25T00:22:49.292308+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-25T00:22:49.292308+0000","last_undegraded":"2023-10-25T00:22:49.292308+0000","last_fullsized":"2023-10-25T00:22:49.292308+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":21819,"num_write_kb":288439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"68'4434","reported_seq":4520,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.208506+0000","last_change":"2023-10-25T00:22:47.255974+0000","last_active":"2023-10-25T00:22:49.208506+0000","last_peered":"2023-10-25T00:22:49.208506+0000","last_clean":"2023-10-25T00:22:49.208506+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-25T00:22:49.208506+0000","last_undegraded":"2023-10-25T00:22:49.208506+0000","last_fullsized":"2023-10-25T00:22:49.208506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4434","last_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_deep_scrub":"68'4434","last_deep_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.255646+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:35:52.723567+0000","scrub_duration":0.14955275700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23791,"num_write_kb":292031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"72'4195","reported_seq":4265,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.231604+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-25T00:22:49.231604+0000","last_peered":"2023-10-25T00:22:49.231604+0000","last_clean":"2023-10-25T00:22:49.231604+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-25T00:22:49.231604+0000","last_undegraded":"2023-10-25T00:22:49.231604+0000","last_fullsized":"2023-10-25T00:22:49.231604+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23000,"num_write_kb":285019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"68'4243","reported_seq":4328,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.969067+0000","last_change":"2023-10-25T00:23:00.969067+0000","last_active":"2023-10-25T00:23:00.969067+0000","last_peered":"2023-10-25T00:23:00.969067+0000","last_clean":"2023-10-25T00:23:00.969067+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-25T00:23:00.969067+0000","last_undegraded":"2023-10-25T00:23:00.969067+0000","last_fullsized":"2023-10-25T00:23:00.969067+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4243","last_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_deep_scrub":"68'4243","last_deep_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.968692+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:45.877638+0000","scrub_duration":0.15459757700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22439,"num_write_kb":266094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"72'4017","reported_seq":4088,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.212449+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-25T00:22:49.212449+0000","last_peered":"2023-10-25T00:22:49.212449+0000","last_clean":"2023-10-25T00:22:49.212449+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-25T00:22:49.212449+0000","last_undegraded":"2023-10-25T00:22:49.212449+0000","last_fullsized":"2023-10-25T00:22:49.212449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22665,"num_write_kb":291707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"68'4509","reported_seq":4597,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.227093+0000","last_change":"2023-10-25T00:22:44.022792+0000","last_active":"2023-10-25T00:22:49.227093+0000","last_peered":"2023-10-25T00:22:49.227093+0000","last_clean":"2023-10-25T00:22:49.227093+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-25T00:22:49.227093+0000","last_undegraded":"2023-10-25T00:22:49.227093+0000","last_fullsized":"2023-10-25T00:22:49.227093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4509","last_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_deep_scrub":"68'4509","last_deep_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.022573+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:54:30.651361+0000","scrub_duration":0.11145192700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23158,"num_write_kb":297883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"72'5235","reported_seq":5314,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534214+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-25T00:22:35.534214+0000","last_peered":"2023-10-25T00:22:35.534214+0000","last_clean":"2023-10-25T00:22:35.534214+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-25T00:22:35.534214+0000","last_undegraded":"2023-10-25T00:22:35.534214+0000","last_fullsized":"2023-10-25T00:22:35.534214+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":15,"num_write":29235,"num_write_kb":367391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"68'4260","reported_seq":4349,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:56.486314+0000","last_change":"2023-10-25T00:22:56.486314+0000","last_active":"2023-10-25T00:22:56.486314+0000","last_peered":"2023-10-25T00:22:56.486314+0000","last_clean":"2023-10-25T00:22:56.486314+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-25T00:22:56.486314+0000","last_undegraded":"2023-10-25T00:22:56.486314+0000","last_fullsized":"2023-10-25T00:22:56.486314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4260","last_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_deep_scrub":"68'4260","last_deep_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.485966+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:51.396462+0000","scrub_duration":0.116368769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23357,"num_write_kb":265837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"72'4656","reported_seq":4737,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534295+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-25T00:22:35.534295+0000","last_peered":"2023-10-25T00:22:35.534295+0000","last_clean":"2023-10-25T00:22:35.534295+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-25T00:22:35.534295+0000","last_undegraded":"2023-10-25T00:22:35.534295+0000","last_fullsized":"2023-10-25T00:22:35.534295+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":26109,"num_write_kb":317781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"68'4636","reported_seq":4726,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:39.672588+0000","last_change":"2023-10-25T00:22:39.672588+0000","last_active":"2023-10-25T00:22:39.672588+0000","last_peered":"2023-10-25T00:22:39.672588+0000","last_clean":"2023-10-25T00:22:39.672588+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-25T00:22:39.672588+0000","last_undegraded":"2023-10-25T00:22:39.672588+0000","last_fullsized":"2023-10-25T00:22:39.672588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4636","last_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_deep_scrub":"68'4636","last_deep_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.672438+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:47:15.433547+0000","scrub_duration":0.124153799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25124,"num_write_kb":285314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"72'3117","reported_seq":3193,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.292881+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-25T00:22:49.292881+0000","last_peered":"2023-10-25T00:22:49.292881+0000","last_clean":"2023-10-25T00:22:49.292881+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-25T00:22:49.292881+0000","last_undegraded":"2023-10-25T00:22:49.292881+0000","last_fullsized":"2023-10-25T00:22:49.292881+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17240,"num_write_kb":232937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"68'5520","reported_seq":5617,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:56.366454+0000","last_change":"2023-10-25T00:22:56.366641+0000","last_active":"2023-10-25T00:22:56.366454+0000","last_peered":"2023-10-25T00:22:56.366454+0000","last_clean":"2023-10-25T00:22:56.366454+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-25T00:22:56.366454+0000","last_undegraded":"2023-10-25T00:22:56.366454+0000","last_fullsized":"2023-10-25T00:22:56.366454+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5520","last_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_deep_scrub":"68'5520","last_deep_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.366419+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:47:59.889167+0000","scrub_duration":0.104594514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29665,"num_write_kb":368823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"72'3962","reported_seq":4036,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517532+0000","last_change":"2023-10-24T23:53:11.832627+0000","last_active":"2023-10-25T00:22:33.517532+0000","last_peered":"2023-10-25T00:22:33.517532+0000","last_clean":"2023-10-25T00:22:33.517532+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-25T00:22:33.517532+0000","last_undegraded":"2023-10-25T00:22:33.517532+0000","last_fullsized":"2023-10-25T00:22:33.517532+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21989,"num_write_kb":280762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"68'4908","reported_seq":5000,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.407703+0000","last_change":"2023-10-25T00:23:00.408006+0000","last_active":"2023-10-25T00:23:00.407703+0000","last_peered":"2023-10-25T00:23:00.407703+0000","last_clean":"2023-10-25T00:23:00.407703+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-25T00:23:00.407703+0000","last_undegraded":"2023-10-25T00:23:00.407703+0000","last_fullsized":"2023-10-25T00:23:00.407703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4908","last_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_deep_scrub":"68'4908","last_deep_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.407650+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:48:18.113931+0000","scrub_duration":0.051805808000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26829,"num_write_kb":320697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":320,"num_write":1482516,"num_write_kb":19060243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267718,"ondisk_log_size":267718,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":19490997,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1686,"num_read_kb":209824,"num_write":4665,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39043072,"data_stored":38981994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4556,"ondisk_log_size":4556,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":0,"num_write":1661441,"num_write_kb":19773626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307026,"ondisk_log_size":307026,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":20061533,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1689,"num_read_kb":267203,"num_write":4630,"num_write_kb":21638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40181760,"data_stored":40123066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4516,"ondisk_log_size":4516,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575216,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919096,"kb_used_data":12336,"kb_used_omap":0,"kb_used_meta":906752,"kb_avail":93452744,"statfs":{"total":96636764160,"available":95695609856,"internally_reserved":0,"allocated":12632064,"data_stored":12153420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":928514048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.96199999999999997,"15min":0.94399999999999995},"min":{"1min":0.23699999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.2490000000000001,"5min":6.1879999999999997,"15min":13.929},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.93999999999999995,"15min":0.93899999999999995},"min":{"1min":0.29899999999999999,"5min":0.27600000000000002,"15min":0.20999999999999999},"max":{"1min":1.3149999999999999,"5min":6.8010000000000002,"15min":11.315},"last":1.0800000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":1.3200000000000001,"15min":1.1619999999999999},"min":{"1min":0.32300000000000001,"5min":0.23599999999999999,"15min":0.215},"max":{"1min":1.016,"5min":28.308,"15min":28.308},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.421,"15min":1.2330000000000001},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":1.232,"5min":28.346,"15min":28.346},"last":1.026}]},{"osd":2,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":1.0780000000000001,"15min":0.98899999999999999},"min":{"1min":0.312,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.94599999999999995,"5min":5.8620000000000001,"15min":11.430999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.012,"15min":0.95299999999999996},"min":{"1min":0.57499999999999996,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.244,"5min":5.9870000000000001,"15min":7.9710000000000001},"last":0.92300000000000004}]},{"osd":3,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":2.1699999999999999,"15min":1.3600000000000001},"min":{"1min":0.41299999999999998,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":1.135,"5min":93.664000000000001,"15min":93.664000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.91500000000000004,"5min":2.29,"15min":1.3799999999999999},"min":{"1min":0.48699999999999999,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":1.2569999999999999,"5min":93.073999999999998,"15min":93.073999999999998},"last":0.998}]},{"osd":4,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.075,"15min":0.877},"min":{"1min":0.51500000000000001,"5min":0.26500000000000001,"15min":0.249},"max":{"1min":1.0129999999999999,"5min":7.6280000000000001,"15min":7.6280000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":1.085,"5min":1.716,"15min":1.1479999999999999},"min":{"1min":0.68000000000000005,"5min":0.33300000000000002,"15min":0.27200000000000002},"max":{"1min":2.6309999999999998,"5min":27.936,"15min":27.936},"last":0.81599999999999995}]},{"osd":5,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.47,"15min":1.042},"min":{"1min":0.29599999999999999,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":1.087,"5min":21.126999999999999,"15min":21.126999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.1140000000000001,"15min":0.97199999999999998},"min":{"1min":0.65700000000000003,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":3.8069999999999999,"5min":11.49,"15min":11.49},"last":0.69499999999999995}]},{"osd":6,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.556,"15min":1.107},"min":{"1min":0.55200000000000005,"5min":0.311,"15min":0.23599999999999999},"max":{"1min":3.8849999999999998,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.827,"15min":1.2150000000000001},"min":{"1min":0.58299999999999996,"5min":0.45500000000000002,"15min":0.35499999999999998},"max":{"1min":4.3630000000000004,"5min":20.651,"15min":20.651},"last":1.103}]}]},{"osd":6,"up_from":13,"seq":55834575216,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859020,"kb_used_data":5060,"kb_used_omap":0,"kb_used_meta":853952,"kb_avail":93512820,"statfs":{"total":96636764160,"available":95757127680,"internally_reserved":0,"allocated":5181440,"data_stored":4739934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":874446848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.2869999999999999,"15min":0.95199999999999996},"min":{"1min":0.315,"5min":0.185,"15min":0.185},"max":{"1min":5.3490000000000002,"5min":15.974,"15min":15.974},"last":1.077},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.331,"15min":0.97399999999999998},"min":{"1min":0.372,"5min":0.218,"15min":0.218},"max":{"1min":1.722,"5min":8.4789999999999992,"15min":8.4789999999999992},"last":0.63100000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.92500000000000004,"15min":0.88300000000000001},"min":{"1min":0.38900000000000001,"5min":0.221,"15min":0.221},"max":{"1min":1.3919999999999999,"5min":5.556,"15min":7.6360000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.175,"15min":0.98699999999999999},"min":{"1min":0.42099999999999999,"5min":0.254,"15min":0.249},"max":{"1min":1.411,"5min":21.600999999999999,"15min":21.600999999999999},"last":1.002}]},{"osd":2,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.5189999999999999,"15min":1.0640000000000001},"min":{"1min":0.318,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.7450000000000001,"5min":12.615,"15min":12.615},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.4610000000000001,"15min":1.109},"min":{"1min":0.38100000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.7669999999999999,"5min":12.667999999999999,"15min":12.667999999999999},"last":0.94099999999999995}]},{"osd":3,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.4770000000000001,"15min":1.2490000000000001},"min":{"1min":0.34300000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.3769999999999998,"5min":18.106999999999999,"15min":38.649999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.107,"5min":1.534,"15min":1.3320000000000001},"min":{"1min":0.39800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.3979999999999997,"5min":18.143000000000001,"15min":38.683},"last":1.1100000000000001}]},{"osd":4,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.7410000000000001,"15min":1.125},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.23799999999999999},"max":{"1min":8.9930000000000003,"5min":12.682,"15min":12.682},"last":0.70899999999999996},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.49,"15min":1.0900000000000001},"min":{"1min":0.36599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.3179999999999996,"5min":10.686,"15min":10.686},"last":0.89900000000000002}]},{"osd":5,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.1559999999999999,"15min":0.96199999999999997},"min":{"1min":0.44500000000000001,"5min":0.28399999999999997,"15min":0.24299999999999999},"max":{"1min":2.1339999999999999,"5min":8.0039999999999996,"15min":8.0039999999999996},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.3660000000000001,"15min":1.069},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.30299999999999999},"max":{"1min":4.0679999999999996,"5min":11.766999999999999,"15min":11.766999999999999},"last":1.181}]},{"osd":7,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2969999999999999,"15min":1.0369999999999999},"min":{"1min":0.40999999999999998,"5min":0.251,"15min":0.251},"max":{"1min":5.4219999999999997,"5min":8.2720000000000002,"15min":12.926},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.341,"5min":1.3100000000000001,"15min":1.05},"min":{"1min":0.504,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.4420000000000002,"5min":6.1100000000000003,"15min":7.6619999999999999},"last":1.046}]}]},{"osd":5,"up_from":13,"seq":55834575216,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":958872,"kb_used_data":9168,"kb_used_omap":0,"kb_used_meta":949696,"kb_avail":93412968,"statfs":{"total":96636764160,"available":95654879232,"internally_reserved":0,"allocated":9388032,"data_stored":8934420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":972488704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:48 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.2869999999999999,"15min":1.038},"min":{"1min":0.371,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.5569999999999999,"5min":14.837999999999999,"15min":14.837999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.86299999999999999,"15min":0.96799999999999997},"min":{"1min":0.27000000000000002,"5min":0.20699999999999999,"15min":0.20100000000000001},"max":{"1min":3.5779999999999998,"5min":5.8289999999999997,"15min":7.423},"last":1.125}]},{"osd":1,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.415,"15min":1.085},"min":{"1min":0.312,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":2.0419999999999998,"5min":12.141,"15min":12.141},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.4199999999999999,"15min":1.0860000000000001},"min":{"1min":0.314,"5min":0.26400000000000001,"15min":0.252},"max":{"1min":3.714,"5min":24.648,"15min":24.648},"last":0.999}]},{"osd":2,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.46,"15min":1.2390000000000001},"min":{"1min":0.376,"5min":0.25,"15min":0.25},"max":{"1min":1.925,"5min":20.760999999999999,"15min":20.760999999999999},"last":1.155},{"interface":"front","average":{"1min":0.76700000000000002,"5min":1.274,"15min":1.0700000000000001},"min":{"1min":0.39800000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.1779999999999999,"5min":20.552,"15min":20.552},"last":0.94299999999999995}]},{"osd":3,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.429,"15min":1.2210000000000001},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.28000000000000003},"max":{"1min":3.6139999999999999,"5min":12.013,"15min":12.013},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.359,"15min":1.246},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.5779999999999998,"5min":12.504,"15min":12.805},"last":0.874}]},{"osd":4,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.708,"15min":1.202},"min":{"1min":0.47099999999999997,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":2.3290000000000002,"5min":12.728,"15min":12.728},"last":1.034},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.532,"15min":1.1419999999999999},"min":{"1min":0.39200000000000002,"5min":0.27600000000000002,"15min":0.27200000000000002},"max":{"1min":3.6699999999999999,"5min":10.943,"15min":10.943},"last":0.91500000000000004}]},{"osd":6,"last update":"Wed Oct 25 00:23:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.385,"15min":1.0609999999999999},"min":{"1min":0.44400000000000001,"5min":0.34399999999999997,"15min":0.26400000000000001},"max":{"1min":3.6930000000000001,"5min":17.936,"15min":17.936},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.869,"15min":1.212},"min":{"1min":0.31,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.0229999999999999,"5min":14.766999999999999,"15min":14.766999999999999},"last":0.96899999999999997}]},{"osd":7,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.75,"15min":1.2470000000000001},"min":{"1min":0.46500000000000002,"5min":0.33200000000000002,"15min":0.29499999999999998},"max":{"1min":1.2869999999999999,"5min":28.178999999999998,"15min":28.178999999999998},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.0740000000000001,"5min":2.133,"15min":1.357},"min":{"1min":0.59699999999999998,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":3.6429999999999998,"5min":28.222999999999999,"15min":28.222999999999999},"last":1.1910000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575215,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946308,"kb_used_data":19132,"kb_used_omap":0,"kb_used_meta":927168,"kb_avail":93425532,"statfs":{"total":96636764160,"available":95667744768,"internally_reserved":0,"allocated":19591168,"data_stored":19124450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":949420032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.3320000000000001,"15min":0.98099999999999998},"min":{"1min":0.317,"5min":0.19700000000000001,"15min":0.187},"max":{"1min":11.629,"5min":20.922999999999998,"15min":20.922999999999998},"last":0.88},{"interface":"front","average":{"1min":1.101,"5min":1.157,"15min":0.98199999999999998},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.217},"max":{"1min":7.0830000000000002,"5min":20.532,"15min":20.532},"last":1.0920000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":1.2410000000000001,"15min":0.97699999999999998},"min":{"1min":0.33800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":2.6110000000000002,"5min":20.914000000000001,"15min":20.914000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.3779999999999999,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":6.1890000000000001,"5min":23.265999999999998,"15min":23.265999999999998},"last":0.90200000000000002}]},{"osd":2,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.528,"15min":1.1080000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.1550000000000002,"5min":20.774000000000001,"15min":20.774000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.802,"15min":1.252},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":7.2750000000000004,"5min":20.817,"15min":20.817},"last":1.008}]},{"osd":3,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.623,"15min":1.1699999999999999},"min":{"1min":0.44800000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":7.3620000000000001,"5min":24.613,"15min":24.613},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.28,"5min":1.3400000000000001,"15min":1.145},"min":{"1min":0.29299999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.2009999999999996,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.92800000000000005}]},{"osd":5,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.1679999999999999,"15min":0.93300000000000005},"min":{"1min":0.5,"5min":0.28599999999999998,"15min":0.23999999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.079,"15min":0.95099999999999996},"min":{"1min":0.57899999999999996,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.7549999999999999,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.67200000000000004}]},{"osd":6,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.367,"15min":1.0860000000000001},"min":{"1min":0.627,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":2.794,"5min":6.5220000000000002,"15min":6.5220000000000002},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.357,"5min":1.1040000000000001,"15min":0.97199999999999998},"min":{"1min":0.38700000000000001,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"last":1.0289999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:22:52 2023","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.3520000000000001,"15min":1.087},"min":{"1min":0.56499999999999995,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.4020000000000001,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.7290000000000001,"5min":1.5649999999999999,"15min":1.1579999999999999},"min":{"1min":0.58899999999999997,"5min":0.309,"15min":0.309},"max":{"1min":7.444,"5min":7.444,"15min":7.444},"last":1.0629999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575216,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160748,"kb_used_data":9252,"kb_used_omap":0,"kb_used_meta":1151488,"kb_avail":93211092,"statfs":{"total":96636764160,"available":95448158208,"internally_reserved":0,"allocated":9474048,"data_stored":9018997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1179123712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.67500000000000004,"15min":0.78300000000000003},"min":{"1min":0.18099999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.72599999999999998,"5min":12.215999999999999,"15min":13.327999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.55700000000000005,"15min":0.72899999999999998},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.95399999999999996,"5min":2.2770000000000001,"15min":5.4580000000000002},"last":0.83499999999999996}]},{"osd":1,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.86699999999999999,"15min":0.85599999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19900000000000001},"max":{"1min":0.96799999999999997,"5min":11.65,"15min":13.831},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.71099999999999997,"15min":0.76100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.0940000000000001,"5min":4.2130000000000001,"15min":4.4500000000000002},"last":0.752}]},{"osd":2,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.82199999999999995,"15min":0.87},"min":{"1min":0.24099999999999999,"5min":0.223,"15min":0.221},"max":{"1min":1.0129999999999999,"5min":8.2520000000000007,"15min":8.2520000000000007},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.92000000000000004,"15min":0.90300000000000002},"min":{"1min":0.26700000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0349999999999999,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.79200000000000004}]},{"osd":4,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.7,"15min":1.204},"min":{"1min":0.57599999999999996,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.208,"5min":39.070999999999998,"15min":39.070999999999998},"last":0.876},{"interface":"front","average":{"1min":0.82999999999999996,"5min":1.8899999999999999,"15min":1.248},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.25700000000000001},"max":{"1min":1.1080000000000001,"5min":55.104999999999997,"15min":55.104999999999997},"last":1.1579999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.556,"15min":1.1160000000000001},"min":{"1min":0.33300000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1759999999999999,"5min":31.481999999999999,"15min":31.481999999999999},"last":1.05},{"interface":"front","average":{"1min":0.997,"5min":1.617,"15min":1.2010000000000001},"min":{"1min":0.40699999999999997,"5min":0.35299999999999998,"15min":0.32500000000000001},"max":{"1min":1.4470000000000001,"5min":31.512,"15min":31.512},"last":1.109}]},{"osd":6,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.6180000000000001,"15min":1.2370000000000001},"min":{"1min":0.33600000000000002,"5min":0.311,"15min":0.27800000000000002},"max":{"1min":15.824,"5min":25.809999999999999,"15min":25.809999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.363,"15min":1.171},"min":{"1min":0.41399999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1910000000000001,"5min":24.036999999999999,"15min":24.036999999999999},"last":1.085}]},{"osd":7,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.5069999999999999,"15min":1.2250000000000001},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3200000000000001,"5min":27.581,"15min":27.581},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.413,"15min":1.2430000000000001},"min":{"1min":0.503,"5min":0.374,"15min":0.374},"max":{"1min":1.3819999999999999,"5min":24.332000000000001,"15min":24.332000000000001},"last":1.1899999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575216,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946060,"kb_used_data":5060,"kb_used_omap":0,"kb_used_meta":940992,"kb_avail":93425780,"statfs":{"total":96636764160,"available":95667998720,"internally_reserved":0,"allocated":5181440,"data_stored":4739422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":963575808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.96699999999999997,"15min":0.97999999999999998},"min":{"1min":0.185,"5min":0.16,"15min":0.16},"max":{"1min":2.3010000000000002,"5min":5.9710000000000001,"15min":29.344000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83699999999999997,"15min":0.83499999999999996},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.7000000000000002,"5min":6.4290000000000003,"15min":10.755000000000001},"last":0.76000000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.98899999999999999,"15min":0.92900000000000005},"min":{"1min":0.24199999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":3.0409999999999999,"5min":17.192,"15min":18.617999999999999},"last":0.495},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.86699999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.649,"5min":5.8890000000000002,"15min":15.064},"last":0.45900000000000002}]},{"osd":3,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.014,"15min":0.89700000000000002},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.9220000000000002,"5min":13.034000000000001,"15min":13.034000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.1930000000000001,"15min":0.99099999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.9929999999999999,"5min":9.9890000000000008,"15min":18.547999999999998},"last":0.71799999999999997}]},{"osd":4,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.4670000000000001,"15min":1.1850000000000001},"min":{"1min":0.26900000000000002,"5min":0.252,"15min":0.252},"max":{"1min":5.1070000000000002,"5min":22.997,"15min":33.378999999999998},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.194,"5min":1.5189999999999999,"15min":1.278},"min":{"1min":0.309,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.2290000000000001,"5min":23.030999999999999,"15min":33.558999999999997},"last":1.119}]},{"osd":5,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.887,"15min":1.464},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29099999999999998},"max":{"1min":3.1400000000000001,"5min":25.183,"15min":34.545999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.2450000000000001,"15min":1.1679999999999999},"min":{"1min":0.33600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.4870000000000001,"5min":11.685,"15min":36.104999999999997},"last":1.012}]},{"osd":6,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.1339999999999999,"15min":1.103},"min":{"1min":0.501,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.482,"5min":9.7149999999999999,"15min":33.661999999999999},"last":1.202},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.1930000000000001,"15min":1.1910000000000001},"min":{"1min":0.53100000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.464,"5min":9.6799999999999997,"15min":34.043999999999997},"last":1.246}]},{"osd":7,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3009999999999999,"15min":1.226},"min":{"1min":0.41499999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":4.8369999999999997,"5min":11.503,"15min":33.307000000000002},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.341,"15min":1.2889999999999999},"min":{"1min":0.44900000000000001,"5min":0.371,"15min":0.371},"max":{"1min":4.8620000000000001,"5min":11.468999999999999,"15min":33.344999999999999},"last":1.1619999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575217,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054144,"kb_used_data":4088,"kb_used_omap":0,"kb_used_meta":1050048,"kb_avail":93317696,"statfs":{"total":96636764160,"available":95557320704,"internally_reserved":0,"allocated":4186112,"data_stored":3743339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1075249152},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.98099999999999998,"15min":0.81299999999999994},"min":{"1min":0.249,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.85699999999999998,"5min":21.334,"15min":21.334},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.96899999999999997,"15min":0.84199999999999997},"min":{"1min":0.38800000000000001,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":1.0449999999999999,"5min":21.276,"15min":21.276},"last":0.98899999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.93400000000000005,"15min":0.80000000000000004},"min":{"1min":0.34999999999999998,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.92900000000000005,"5min":11.420999999999999,"15min":11.420999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.878,"15min":0.83099999999999996},"min":{"1min":0.27100000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.002,"5min":11.648,"15min":11.648},"last":0.94699999999999995}]},{"osd":3,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.022,"15min":0.91100000000000003},"min":{"1min":0.318,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":1.1140000000000001,"5min":9.0449999999999999,"15min":14.346},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.84899999999999998,"5min":1.1040000000000001,"15min":1.01},"min":{"1min":0.35099999999999998,"5min":0.26300000000000001,"15min":0.248},"max":{"1min":1.1859999999999999,"5min":9.0800000000000001,"15min":14.379},"last":1.1459999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.2569999999999999,"15min":1.0600000000000001},"min":{"1min":0.50600000000000001,"5min":0.28000000000000003,"15min":0.27200000000000002},"max":{"1min":1.353,"5min":10.582000000000001,"15min":10.582000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.2210000000000001,"15min":1.0449999999999999},"min":{"1min":0.39600000000000002,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":1.272,"5min":11.173,"15min":11.173},"last":1.246}]},{"osd":5,"last update":"Wed Oct 25 00:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.0860000000000001,"15min":1.042},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.248,"5min":10.446,"15min":11.577999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.93700000000000006,"15min":0.96799999999999997},"min":{"1min":0.52100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.2869999999999999,"5min":5.3209999999999997,"15min":10.018000000000001},"last":1.21}]},{"osd":6,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0680000000000001,"15min":1.0229999999999999},"min":{"1min":0.55800000000000005,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.357,"5min":9.2989999999999995,"15min":11.845000000000001},"last":1.032},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.141,"15min":1.1140000000000001},"min":{"1min":0.62,"5min":0.379,"15min":0.33800000000000002},"max":{"1min":1.4870000000000001,"5min":9.3350000000000009,"15min":11.919},"last":1.0900000000000001}]},{"osd":7,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.387,"15min":1.1200000000000001},"min":{"1min":0.57499999999999996,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":1.407,"5min":9.407,"15min":11.881},"last":1.2749999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.22,"15min":1.1359999999999999},"min":{"1min":0.54700000000000004,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3080000000000001,"5min":10.478,"15min":10.478},"last":1.304}]}]},{"osd":0,"up_from":13,"seq":55834575217,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":966376,"kb_used_data":22112,"kb_used_omap":0,"kb_used_meta":944256,"kb_avail":93405464,"statfs":{"total":96636764160,"available":95647195136,"internally_reserved":0,"allocated":22642688,"data_stored":22192198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":966918144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.1719999999999999,"15min":0.99099999999999999},"min":{"1min":0.23000000000000001,"5min":0.18099999999999999,"15min":0.16900000000000001},"max":{"1min":1.6779999999999999,"5min":30.065999999999999,"15min":30.065999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.94799999999999995,"15min":0.99199999999999999},"min":{"1min":0.247,"5min":0.184,"15min":0.182},"max":{"1min":0.98999999999999999,"5min":8.8529999999999998,"15min":19.757999999999999},"last":0.83999999999999997}]},{"osd":2,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.93500000000000005,"15min":0.78600000000000003},"min":{"1min":0.42199999999999999,"5min":0.22900000000000001,"15min":0.20399999999999999},"max":{"1min":1.806,"5min":15.324999999999999,"15min":15.324999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.169,"15min":0.95499999999999996},"min":{"1min":0.42899999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.8440000000000001,"5min":30.283999999999999,"15min":30.283999999999999},"last":1.0169999999999999}]},{"osd":3,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":1.4490000000000001,"15min":1.1020000000000001},"min":{"1min":0.40000000000000002,"5min":0.191,"15min":0.191},"max":{"1min":2.3420000000000001,"5min":22.966999999999999,"15min":22.966999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.2529999999999999,"15min":1.0609999999999999},"min":{"1min":0.32800000000000001,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":1.915,"5min":19.460999999999999,"15min":19.460999999999999},"last":0.63600000000000001}]},{"osd":4,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":1.056,"15min":1.121},"min":{"1min":0.35299999999999998,"5min":0.318,"15min":0.26600000000000001},"max":{"1min":1.157,"5min":4.4710000000000001,"15min":8.3670000000000009},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.3799999999999999,"15min":1.222},"min":{"1min":0.59799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.3719999999999999,"5min":13.023999999999999,"15min":13.023999999999999},"last":1.1799999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.103,"15min":1.0169999999999999},"min":{"1min":0.60999999999999999,"5min":0.30399999999999999,"15min":0.29099999999999998},"max":{"1min":2.391,"5min":6.0149999999999997,"15min":10.683999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1799999999999999,"15min":0.98799999999999999},"min":{"1min":0.52800000000000002,"5min":0.33700000000000002,"15min":0.29899999999999999},"max":{"1min":4.8860000000000001,"5min":15.436999999999999,"15min":15.436999999999999},"last":0.93400000000000005}]},{"osd":6,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.266,"15min":1.125},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.28100000000000003},"max":{"1min":1.3069999999999999,"5min":8.7260000000000009,"15min":8.7260000000000009},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.3700000000000001,"15min":1.2829999999999999},"min":{"1min":0.56299999999999994,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":7.8970000000000002,"5min":15.375,"15min":15.375},"last":1.095}]},{"osd":7,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.3380000000000001,"15min":1.151},"min":{"1min":0.64800000000000002,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":5.4269999999999996,"5min":6.0640000000000001,"15min":10.81},"last":1.002},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.206,"15min":1.256},"min":{"1min":0.45600000000000002,"5min":0.313,"15min":0.313},"max":{"1min":1.8660000000000001,"5min":9.3699999999999992,"15min":11.417},"last":1.117}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12673024,"data_stored":12668140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3198976,"data_stored":3198201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11689984,"data_stored":11672183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4195192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":8388642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4279555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6918144,"data_stored":6907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6836224,"data_stored":6822812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-25T00:23:18.719 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.719 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.719 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.720 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.720 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.720 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.721 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.721 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.721 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.721 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.722 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.722 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.722 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.723 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.723 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.724 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.724 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.724 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.724 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.725 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.725 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.725 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.726 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.726 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.726 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.727 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.727 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.727 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.728 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.728 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.728 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.728 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.729 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.729 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.729 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.730 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.730 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.730 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.731 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.731 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.731 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.732 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.732 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.732 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.732 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-24T23:52:57.136697+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=57, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.733 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-24T23:52:51.349804+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=52, tm_sec=51, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.733 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.733 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.734 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.734 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.734 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.734 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.735 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.735 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.735 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.736 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.736 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.736 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.737 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.737 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.737 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.737 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.738 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.738 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.738 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.739 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.739 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.739 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.739 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.740 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.740 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.740 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.741 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.741 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.741 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.742 INFO:tasks.ceph:pgid 6.20 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.742 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.742 INFO:tasks.ceph:pgid 6.21 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.742 INFO:tasks.ceph:pgid 4.23 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.743 INFO:tasks.ceph:pgid 6.26 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.743 INFO:tasks.ceph:pgid 6.27 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.743 INFO:tasks.ceph:pgid 6.24 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.744 INFO:tasks.ceph:pgid 4.26 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.744 INFO:tasks.ceph:pgid 6.25 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.744 INFO:tasks.ceph:pgid 6.2a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.745 INFO:tasks.ceph:pgid 6.2b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.745 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.745 INFO:tasks.ceph:pgid 6.28 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.745 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.746 INFO:tasks.ceph:pgid 6.2e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.746 INFO:tasks.ceph:pgid 6.2f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.746 INFO:tasks.ceph:pgid 6.2c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.747 INFO:tasks.ceph:pgid 6.2d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.747 INFO:tasks.ceph:pgid 6.32 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.747 INFO:tasks.ceph:pgid 6.33 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.748 INFO:tasks.ceph:pgid 6.30 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.748 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.748 INFO:tasks.ceph:pgid 6.31 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.748 INFO:tasks.ceph:pgid 4.33 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.749 INFO:tasks.ceph:pgid 6.36 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.749 INFO:tasks.ceph:pgid 6.37 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.749 INFO:tasks.ceph:pgid 6.34 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.750 INFO:tasks.ceph:pgid 6.35 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.750 INFO:tasks.ceph:pgid 6.3a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.750 INFO:tasks.ceph:pgid 6.3b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.750 INFO:tasks.ceph:pgid 6.38 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.751 INFO:tasks.ceph:pgid 6.39 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.751 INFO:tasks.ceph:pgid 6.3e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.751 INFO:tasks.ceph:pgid 6.3f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.752 INFO:tasks.ceph:pgid 6.3c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.752 INFO:tasks.ceph:pgid 6.3d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:18.752 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-25T00:23:38.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-25T00:23:38.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f626bcb3640 1 Processor -- start 2023-10-25T00:23:38.883 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f626bcb3640 1 -- start start 2023-10-25T00:23:38.884 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f626bcb3640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f626414c470 0x7f626414c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:38.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f626bcb3640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6264155cd0 con 0x7f6264152b30 2023-10-25T00:23:38.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f626bcb3640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62640c4380 con 0x7f626406f490 2023-10-25T00:23:38.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f626bcb3640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62641526e0 con 0x7f626414c470 2023-10-25T00:23:38.885 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f626a250640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6264152b30 0x7f6264152f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:48714/0 (socket says 172.21.15.107:48714) 2023-10-25T00:23:38.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f6269a4f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f626414c470 0x7f626414c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:38.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f626a250640 1 -- 172.21.15.107:0/840945242 learned_addr learned my addr 172.21.15.107:0/840945242 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:23:38.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f6269a4f640 1 -- 172.21.15.107:0/840945242 >> v1:172.21.15.110:6789/0 conn(0x7f626406f490 legacy=0x7f6264071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:23:38.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f6269a4f640 1 -- 172.21.15.107:0/840945242 >> v1:172.21.15.107:6789/0 conn(0x7f6264152b30 legacy=0x7f6264152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:38.886 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f6269a4f640 1 -- 172.21.15.107:0/840945242 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62640c48b0 con 0x7f626414c470 2023-10-25T00:23:38.887 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f6269a4f640 1 --2- 172.21.15.107:0/840945242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f626414c470 0x7f626414c850 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f62600097c0 tx=0x7f626001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a3c01616a4977c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:38.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f6268a4d640 1 -- 172.21.15.107:0/840945242 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62600ab600 con 0x7f626414c470 2023-10-25T00:23:38.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f6268a4d640 1 -- 172.21.15.107:0/840945242 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62600a7050 con 0x7f626414c470 2023-10-25T00:23:38.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.882+0000 7f6268a4d640 1 -- 172.21.15.107:0/840945242 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62600ab980 con 0x7f626414c470 2023-10-25T00:23:38.888 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 -- 172.21.15.107:0/840945242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f626414c470 msgr2=0x7f626414c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:38.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 --2- 172.21.15.107:0/840945242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f626414c470 0x7f626414c850 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f62600097c0 tx=0x7f626001c320 comp rx=0 tx=0).stop 2023-10-25T00:23:38.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 -- 172.21.15.107:0/840945242 shutdown_connections 2023-10-25T00:23:38.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 --2- 172.21.15.107:0/840945242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f626414c470 0x7f626414c850 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:38.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 -- 172.21.15.107:0/840945242 >> 172.21.15.107:0/840945242 conn(0x7f6264070f60 msgr2=0x7f6264071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:23:38.889 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 -- 172.21.15.107:0/840945242 shutdown_connections 2023-10-25T00:23:38.890 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 -- 172.21.15.107:0/840945242 wait complete. 2023-10-25T00:23:38.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 Processor -- start 2023-10-25T00:23:38.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 -- start start 2023-10-25T00:23:38.892 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f626406f490 0x7f62641eda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:38.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f626414c470 0x7f626414b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:38.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6264152b30 0x7f62641ecf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:38.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62641f06c0 con 0x7f626406f490 2023-10-25T00:23:38.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62641ec720 con 0x7f6264152b30 2023-10-25T00:23:38.893 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62641eca00 con 0x7f626414c470 2023-10-25T00:23:38.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f6269a4f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f626406f490 0x7f62641eda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:38.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f6269a4f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f626406f490 0x7f62641eda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33624/0 (socket says 172.21.15.107:33624) 2023-10-25T00:23:38.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f6269a4f640 1 -- 172.21.15.107:0/2139233578 learned_addr learned my addr 172.21.15.107:0/2139233578 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:23:38.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f6269a4f640 1 -- 172.21.15.107:0/2139233578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f626414c470 msgr2=0x7f626414b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:38.894 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626924e640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f626414c470 0x7f626414b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:38.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626a250640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6264152b30 0x7f62641ecf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:38.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f6269a4f640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f626414c470 0x7f626414b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:38.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f6269a4f640 1 -- 172.21.15.107:0/2139233578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6264152b30 msgr2=0x7f62641ecf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:38.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f6269a4f640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6264152b30 0x7f62641ecf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:38.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f6269a4f640 1 -- 172.21.15.107:0/2139233578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6260009420 con 0x7f626406f490 2023-10-25T00:23:38.895 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f6269a4f640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f626406f490 0x7f62641eda80 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f6260018f30 tx=0x7f62600190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:38.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f625a7fc640 1 -- 172.21.15.107:0/2139233578 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62600b4610 con 0x7f626406f490 2023-10-25T00:23:38.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f625a7fc640 1 -- 172.21.15.107:0/2139233578 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62600abe90 con 0x7f626406f490 2023-10-25T00:23:38.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f625a7fc640 1 -- 172.21.15.107:0/2139233578 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62600b49d0 con 0x7f626406f490 2023-10-25T00:23:38.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 -- 172.21.15.107:0/2139233578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626414be00 con 0x7f626406f490 2023-10-25T00:23:38.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.886+0000 7f626bcb3640 1 -- 172.21.15.107:0/2139233578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62641f09a0 con 0x7f626406f490 2023-10-25T00:23:38.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.890+0000 7f626bcb3640 1 -- 172.21.15.107:0/2139233578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626406d450 con 0x7f626406f490 2023-10-25T00:23:38.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.890+0000 7f625a7fc640 1 -- 172.21.15.107:0/2139233578 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f62600b2040 con 0x7f626406f490 2023-10-25T00:23:38.897 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.890+0000 7f625a7fc640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6248065e90 0x7f6248068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:38.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.890+0000 7f626924e640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6248065e90 0x7f6248068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:38.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.890+0000 7f625a7fc640 1 -- 172.21.15.107:0/2139233578 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f626011ecc0 con 0x7f626406f490 2023-10-25T00:23:38.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.890+0000 7f626924e640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6248065e90 0x7f6248068350 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f62640732f0 tx=0x7f6254009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:38.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:38.894+0000 7f625a7fc640 1 -- 172.21.15.107:0/2139233578 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f626406d450 con 0x7f626406f490 2023-10-25T00:23:39.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.042+0000 7f626bcb3640 1 -- 172.21.15.107:0/2139233578 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62641ec890 con 0x7f6248065e90 2023-10-25T00:23:39.086 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.082+0000 7f625a7fc640 1 -- 172.21.15.107:0/2139233578 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+552453 (secure 0 0 0) 0x7f62641ec890 con 0x7f6248065e90 2023-10-25T00:23:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:23:39.096 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-25T00:23:39.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 -- 172.21.15.107:0/2139233578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6248065e90 msgr2=0x7f6248068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:39.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6248065e90 0x7f6248068350 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f62640732f0 tx=0x7f6254009040 comp rx=0 tx=0).stop 2023-10-25T00:23:39.097 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 -- 172.21.15.107:0/2139233578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f626406f490 msgr2=0x7f62641eda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:39.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f626406f490 0x7f62641eda80 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f6260018f30 tx=0x7f62600190c0 comp rx=0 tx=0).stop 2023-10-25T00:23:39.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 -- 172.21.15.107:0/2139233578 shutdown_connections 2023-10-25T00:23:39.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6264152b30 0x7f62641ecf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:39.098 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6248065e90 0x7f6248068350 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:39.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f626414c470 0x7f626414b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:39.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 --2- 172.21.15.107:0/2139233578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f626406f490 0x7f62641eda80 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:39.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 -- 172.21.15.107:0/2139233578 >> 172.21.15.107:0/2139233578 conn(0x7f6264070f60 msgr2=0x7f62640c3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:23:39.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 -- 172.21.15.107:0/2139233578 shutdown_connections 2023-10-25T00:23:39.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:39.090+0000 7f626bcb3640 1 -- 172.21.15.107:0/2139233578 wait complete. 2023-10-25T00:23:39.110 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":990,"stamp":"2023-10-25T00:23:38.039093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40142898,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3975,"num_read_kb":477474,"num_write":3153461,"num_write_kb":38880074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":583996,"ondisk_log_size":583996,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7810624,"kb_used_data":86208,"kb_used_omap":0,"kb_used_meta":7724352,"kb_avail":747164096,"statfs":{"total":773094113280,"available":765096034304,"internally_reserved":0,"allocated":88276992,"data_stored":84646180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7909736448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004997"},"pg_stats":[{"pgid":"6.1a","version":"72'3995","reported_seq":4069,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.233446+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-25T00:22:49.233446+0000","last_peered":"2023-10-25T00:22:49.233446+0000","last_clean":"2023-10-25T00:22:49.233446+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-25T00:22:49.233446+0000","last_undegraded":"2023-10-25T00:22:49.233446+0000","last_fullsized":"2023-10-25T00:22:49.233446+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21480,"num_write_kb":282788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"72'3","reported_seq":84,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:29.637360+0000","last_change":"2023-10-25T00:23:29.637684+0000","last_active":"2023-10-25T00:23:29.637360+0000","last_peered":"2023-10-25T00:23:29.637360+0000","last_clean":"2023-10-25T00:23:29.637360+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-25T00:23:29.637360+0000","last_undegraded":"2023-10-25T00:23:29.637360+0000","last_fullsized":"2023-10-25T00:23:29.637360+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:29.637307+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:29.637307+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.637307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:55.311760+0000","scrub_duration":0.0045847350000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"73'867","reported_seq":1007,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.026995+0000","last_change":"2023-10-25T00:23:27.027247+0000","last_active":"2023-10-25T00:23:27.026995+0000","last_peered":"2023-10-25T00:23:27.026995+0000","last_clean":"2023-10-25T00:23:27.026995+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-25T00:23:27.026995+0000","last_undegraded":"2023-10-25T00:23:27.026995+0000","last_fullsized":"2023-10-25T00:23:27.026995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'867","last_scrub_stamp":"2023-10-25T00:23:27.026943+0000","last_deep_scrub":"73'867","last_deep_scrub_stamp":"2023-10-25T00:23:27.026943+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.026943+0000","objects_scrubbed":3,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:41.065198+0000","scrub_duration":0.082266929000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":22541,"num_write":841,"num_write_kb":4586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'5616","reported_seq":5690,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060413+0000","last_change":"2023-10-24T23:53:02.721489+0000","last_active":"2023-10-25T00:22:36.060413+0000","last_peered":"2023-10-25T00:22:36.060413+0000","last_clean":"2023-10-25T00:22:36.060413+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-25T00:22:36.060413+0000","last_undegraded":"2023-10-25T00:22:36.060413+0000","last_fullsized":"2023-10-25T00:22:36.060413+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29775,"num_write_kb":348818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"72'3735","reported_seq":3807,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234391+0000","last_change":"2023-10-24T23:53:11.847034+0000","last_active":"2023-10-25T00:22:49.234391+0000","last_peered":"2023-10-25T00:22:49.234391+0000","last_clean":"2023-10-25T00:22:49.234391+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-25T00:22:49.234391+0000","last_undegraded":"2023-10-25T00:22:49.234391+0000","last_fullsized":"2023-10-25T00:22:49.234391+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20090,"num_write_kb":264783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"72'2","reported_seq":83,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.880273+0000","last_change":"2023-10-25T00:23:02.880442+0000","last_active":"2023-10-25T00:23:02.880273+0000","last_peered":"2023-10-25T00:23:02.880273+0000","last_clean":"2023-10-25T00:23:02.880273+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-25T00:23:02.880273+0000","last_undegraded":"2023-10-25T00:23:02.880273+0000","last_fullsized":"2023-10-25T00:23:02.880273+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.880202+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:45.433003+0000","scrub_duration":0.0048172340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"73'28","reported_seq":116,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.443056+0000","last_change":"2023-10-25T00:23:05.443056+0000","last_active":"2023-10-25T00:23:05.443056+0000","last_peered":"2023-10-25T00:23:05.443056+0000","last_clean":"2023-10-25T00:23:05.443056+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-25T00:23:05.443056+0000","last_undegraded":"2023-10-25T00:23:05.443056+0000","last_fullsized":"2023-10-25T00:23:05.443056+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_deep_scrub":"73'28","last_deep_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.442750+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:29:04.499904+0000","scrub_duration":0.036465213000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'5447","reported_seq":5532,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:28.801261+0000","last_change":"2023-10-25T00:23:28.801391+0000","last_active":"2023-10-25T00:23:28.801261+0000","last_peered":"2023-10-25T00:23:28.801261+0000","last_clean":"2023-10-25T00:23:28.801261+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-25T00:23:28.801261+0000","last_undegraded":"2023-10-25T00:23:28.801261+0000","last_fullsized":"2023-10-25T00:23:28.801261+0000","mapping_epoch":21,"log_start":"61'100","ondisk_log_start":"61'100","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5447","last_scrub_stamp":"2023-10-25T00:23:28.801240+0000","last_deep_scrub":"68'5447","last_deep_scrub_stamp":"2023-10-25T00:23:28.801240+0000","last_clean_scrub_stamp":"2023-10-25T00:23:28.801240+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:44:02.442972+0000","scrub_duration":0.154979636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28063,"num_write_kb":350200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"72'4747","reported_seq":4820,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517509+0000","last_change":"2023-10-24T23:53:11.793762+0000","last_active":"2023-10-25T00:22:33.517509+0000","last_peered":"2023-10-25T00:22:33.517509+0000","last_clean":"2023-10-25T00:22:33.517509+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-25T00:22:33.517509+0000","last_undegraded":"2023-10-25T00:22:33.517509+0000","last_fullsized":"2023-10-25T00:22:33.517509+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":25766,"num_write_kb":328702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"72'754","reported_seq":862,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.214995+0000","last_change":"2023-10-25T00:22:37.433948+0000","last_active":"2023-10-25T00:22:49.214995+0000","last_peered":"2023-10-25T00:22:49.214995+0000","last_clean":"2023-10-25T00:22:49.214995+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-25T00:22:49.214995+0000","last_undegraded":"2023-10-25T00:22:49.214995+0000","last_fullsized":"2023-10-25T00:22:49.214995+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'754","last_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_deep_scrub":"72'754","last_deep_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.433382+0000","objects_scrubbed":1,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:43.601946+0000","scrub_duration":0.10905978199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":77282,"num_write":740,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"73'50","reported_seq":170,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.647282+0000","last_change":"2023-10-24T23:53:02.343592+0000","last_active":"2023-10-25T00:22:52.647282+0000","last_peered":"2023-10-25T00:22:52.647282+0000","last_clean":"2023-10-25T00:22:52.647282+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-25T00:22:52.647282+0000","last_undegraded":"2023-10-25T00:22:52.647282+0000","last_fullsized":"2023-10-25T00:22:52.647282+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'4792","reported_seq":4878,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234675+0000","last_change":"2023-10-25T00:22:44.881494+0000","last_active":"2023-10-25T00:22:49.234675+0000","last_peered":"2023-10-25T00:22:49.234675+0000","last_clean":"2023-10-25T00:22:49.234675+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-25T00:22:49.234675+0000","last_undegraded":"2023-10-25T00:22:49.234675+0000","last_fullsized":"2023-10-25T00:22:49.234675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4792","last_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_deep_scrub":"68'4792","last_deep_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.881344+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:23:57.562121+0000","scrub_duration":0.193670646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24997,"num_write_kb":313272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"72'4739","reported_seq":4813,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523335+0000","last_change":"2023-10-24T23:53:11.825916+0000","last_active":"2023-10-25T00:22:36.523335+0000","last_peered":"2023-10-25T00:22:36.523335+0000","last_clean":"2023-10-25T00:22:36.523335+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-25T00:22:36.523335+0000","last_undegraded":"2023-10-25T00:22:36.523335+0000","last_fullsized":"2023-10-25T00:22:36.523335+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":22,"num_write":25639,"num_write_kb":332364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"72'357","reported_seq":430,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.998388+0000","last_change":"2023-10-25T00:23:16.269311+0000","last_active":"2023-10-25T00:22:55.998388+0000","last_peered":"2023-10-25T00:22:55.998388+0000","last_clean":"2023-10-25T00:22:55.998388+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-25T00:22:55.998388+0000","last_undegraded":"2023-10-25T00:22:55.998388+0000","last_fullsized":"2023-10-25T00:22:55.998388+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2627766,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":227,"num_write":329,"num_write_kb":2759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"68'5","reported_seq":127,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.649341+0000","last_change":"2023-10-24T23:53:02.349154+0000","last_active":"2023-10-25T00:22:52.649341+0000","last_peered":"2023-10-25T00:22:52.649341+0000","last_clean":"2023-10-25T00:22:52.649341+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-25T00:22:52.649341+0000","last_undegraded":"2023-10-25T00:22:52.649341+0000","last_fullsized":"2023-10-25T00:22:52.649341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'3731","reported_seq":3815,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234039+0000","last_change":"2023-10-25T00:22:37.622543+0000","last_active":"2023-10-25T00:22:49.234039+0000","last_peered":"2023-10-25T00:22:49.234039+0000","last_clean":"2023-10-25T00:22:49.234039+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-25T00:22:49.234039+0000","last_undegraded":"2023-10-25T00:22:49.234039+0000","last_fullsized":"2023-10-25T00:22:49.234039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3731","last_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_deep_scrub":"68'3731","last_deep_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.622304+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:51:52.171234+0000","scrub_duration":0.090187284000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19492,"num_write_kb":250091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"72'3799","reported_seq":3872,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.236389+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-25T00:22:49.236389+0000","last_peered":"2023-10-25T00:22:49.236389+0000","last_clean":"2023-10-25T00:22:49.236389+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-25T00:22:49.236389+0000","last_undegraded":"2023-10-25T00:22:49.236389+0000","last_fullsized":"2023-10-25T00:22:49.236389+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21581,"num_write_kb":262877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"73'46","reported_seq":171,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.664257+0000","last_change":"2023-10-25T00:22:52.664257+0000","last_active":"2023-10-25T00:22:52.664257+0000","last_peered":"2023-10-25T00:22:52.664257+0000","last_clean":"2023-10-25T00:22:52.664257+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-25T00:22:52.664257+0000","last_undegraded":"2023-10-25T00:22:52.664257+0000","last_fullsized":"2023-10-25T00:22:52.664257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'46","last_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_deep_scrub":"73'46","last_deep_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.664064+0000","objects_scrubbed":1,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:44:09.108291+0000","scrub_duration":0.106079285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"73'715","reported_seq":829,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.967748+0000","last_change":"2023-10-25T00:23:21.967998+0000","last_active":"2023-10-25T00:23:21.967748+0000","last_peered":"2023-10-25T00:23:21.967748+0000","last_clean":"2023-10-25T00:23:21.967748+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-25T00:23:21.967748+0000","last_undegraded":"2023-10-25T00:23:21.967748+0000","last_fullsized":"2023-10-25T00:23:21.967748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'715","last_scrub_stamp":"2023-10-25T00:23:21.967694+0000","last_deep_scrub":"73'715","last_deep_scrub_stamp":"2023-10-25T00:23:21.967694+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.967694+0000","objects_scrubbed":1,"log_size":715,"log_dups_size":0,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:56:40.443702+0000","scrub_duration":0.065018907000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":90428,"num_write":703,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'4008","reported_seq":4096,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.488463+0000","last_change":"2023-10-25T00:23:05.488463+0000","last_active":"2023-10-25T00:23:05.488463+0000","last_peered":"2023-10-25T00:23:05.488463+0000","last_clean":"2023-10-25T00:23:05.488463+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-25T00:23:05.488463+0000","last_undegraded":"2023-10-25T00:23:05.488463+0000","last_fullsized":"2023-10-25T00:23:05.488463+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4008","last_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_deep_scrub":"68'4008","last_deep_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.488248+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:51:16.627187+0000","scrub_duration":0.15813021899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20791,"num_write_kb":267770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"72'5469","reported_seq":5564,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.571255+0000","last_change":"2023-10-25T00:22:57.571401+0000","last_active":"2023-10-25T00:22:57.571255+0000","last_peered":"2023-10-25T00:22:57.571255+0000","last_clean":"2023-10-25T00:22:57.571255+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-25T00:22:57.571255+0000","last_undegraded":"2023-10-25T00:22:57.571255+0000","last_fullsized":"2023-10-25T00:22:57.571255+0000","mapping_epoch":25,"log_start":"64'200","ondisk_log_start":"64'200","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5469","last_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_deep_scrub":"72'5469","last_deep_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.571234+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:24:13.836466+0000","scrub_duration":0.19144520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":40,"num_write":30646,"num_write_kb":401481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"73'6","reported_seq":131,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.991436+0000","last_change":"2023-10-25T00:23:14.991436+0000","last_active":"2023-10-25T00:23:14.991436+0000","last_peered":"2023-10-25T00:23:14.991436+0000","last_clean":"2023-10-25T00:23:14.991436+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-25T00:23:14.991436+0000","last_undegraded":"2023-10-25T00:23:14.991436+0000","last_fullsized":"2023-10-25T00:23:14.991436+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_deep_scrub":"73'6","last_deep_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.991238+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:25:08.545014+0000","scrub_duration":0.0072276010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"73'546","reported_seq":630,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.826017+0000","last_change":"2023-10-24T23:53:02.340681+0000","last_active":"2023-10-25T00:22:55.826017+0000","last_peered":"2023-10-25T00:22:55.826017+0000","last_clean":"2023-10-25T00:22:55.826017+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-25T00:22:55.826017+0000","last_undegraded":"2023-10-25T00:22:55.826017+0000","last_fullsized":"2023-10-25T00:22:55.826017+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3198201,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10628,"num_write":517,"num_write_kb":3434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'6069","reported_seq":6156,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.235476+0000","last_change":"2023-10-25T00:22:46.608504+0000","last_active":"2023-10-25T00:22:49.235476+0000","last_peered":"2023-10-25T00:22:49.235476+0000","last_clean":"2023-10-25T00:22:49.235476+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-25T00:22:49.235476+0000","last_undegraded":"2023-10-25T00:22:49.235476+0000","last_fullsized":"2023-10-25T00:22:49.235476+0000","mapping_epoch":21,"log_start":"62'800","ondisk_log_start":"62'800","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6069","last_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_deep_scrub":"68'6069","last_deep_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.608198+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:37.101860+0000","scrub_duration":0.19917779799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":33187,"num_write_kb":387130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'3606","reported_seq":3683,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.207557+0000","last_change":"2023-10-24T23:53:11.849078+0000","last_active":"2023-10-25T00:22:49.207557+0000","last_peered":"2023-10-25T00:22:49.207557+0000","last_clean":"2023-10-25T00:22:49.207557+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-25T00:22:49.207557+0000","last_undegraded":"2023-10-25T00:22:49.207557+0000","last_fullsized":"2023-10-25T00:22:49.207557+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19493,"num_write_kb":254342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'1064","reported_seq":1198,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:03.701358+0000","last_change":"2023-10-25T00:23:03.701535+0000","last_active":"2023-10-25T00:23:03.701358+0000","last_peered":"2023-10-25T00:23:03.701358+0000","last_clean":"2023-10-25T00:23:03.701358+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-25T00:23:03.701358+0000","last_undegraded":"2023-10-25T00:23:03.701358+0000","last_fullsized":"2023-10-25T00:23:03.701358+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1064","last_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_deep_scrub":"73'1064","last_deep_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_clean_scrub_stamp":"2023-10-25T00:23:03.701304+0000","objects_scrubbed":3,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:08:18.178955+0000","scrub_duration":0.28236572100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":9402,"num_write":1035,"num_write_kb":4717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'3","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.838808+0000","last_change":"2023-10-25T00:23:12.838945+0000","last_active":"2023-10-25T00:23:12.838808+0000","last_peered":"2023-10-25T00:23:12.838808+0000","last_clean":"2023-10-25T00:23:12.838808+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-25T00:23:12.838808+0000","last_undegraded":"2023-10-25T00:23:12.838808+0000","last_fullsized":"2023-10-25T00:23:12.838808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.838787+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:33:35.778931+0000","scrub_duration":0.133446072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'5139","reported_seq":5226,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:54.030976+0000","last_change":"2023-10-25T00:22:54.030976+0000","last_active":"2023-10-25T00:22:54.030976+0000","last_peered":"2023-10-25T00:22:54.030976+0000","last_clean":"2023-10-25T00:22:54.030976+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-25T00:22:54.030976+0000","last_undegraded":"2023-10-25T00:22:54.030976+0000","last_fullsized":"2023-10-25T00:22:54.030976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5139","last_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_deep_scrub":"68'5139","last_deep_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.030782+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:30.182861+0000","scrub_duration":0.103857617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27772,"num_write_kb":328469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"72'4843","reported_seq":4916,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.231448+0000","last_change":"2023-10-24T23:53:11.855226+0000","last_active":"2023-10-25T00:22:49.231448+0000","last_peered":"2023-10-25T00:22:49.231448+0000","last_clean":"2023-10-25T00:22:49.231448+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-25T00:22:49.231448+0000","last_undegraded":"2023-10-25T00:22:49.231448+0000","last_fullsized":"2023-10-25T00:22:49.231448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":27290,"num_write_kb":330906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"72'27","reported_seq":111,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.987117+0000","last_change":"2023-10-25T00:22:55.987117+0000","last_active":"2023-10-25T00:22:55.987117+0000","last_peered":"2023-10-25T00:22:55.987117+0000","last_clean":"2023-10-25T00:22:55.987117+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-25T00:22:55.987117+0000","last_undegraded":"2023-10-25T00:22:55.987117+0000","last_fullsized":"2023-10-25T00:22:55.987117+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'27","last_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_deep_scrub":"72'27","last_deep_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.986674+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:33.089164+0000","scrub_duration":0.100730242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'2","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.503834+0000","last_change":"2023-10-25T00:23:21.504086+0000","last_active":"2023-10-25T00:23:21.503834+0000","last_peered":"2023-10-25T00:23:21.503834+0000","last_clean":"2023-10-25T00:23:21.503834+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-25T00:23:21.503834+0000","last_undegraded":"2023-10-25T00:23:21.503834+0000","last_fullsized":"2023-10-25T00:23:21.503834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2023-10-25T00:23:21.503783+0000","last_deep_scrub":"67'2","last_deep_scrub_stamp":"2023-10-25T00:23:21.503783+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.503783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:13.653289+0000","scrub_duration":0.0039280959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'5814","reported_seq":5901,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.320438+0000","last_change":"2023-10-25T00:22:52.320696+0000","last_active":"2023-10-25T00:22:52.320438+0000","last_peered":"2023-10-25T00:22:52.320438+0000","last_clean":"2023-10-25T00:22:52.320438+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-25T00:22:52.320438+0000","last_undegraded":"2023-10-25T00:22:52.320438+0000","last_fullsized":"2023-10-25T00:22:52.320438+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5814","last_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_deep_scrub":"68'5814","last_deep_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.320378+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:26.511615+0000","scrub_duration":0.033495337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31996,"num_write_kb":382397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"72'3827","reported_seq":3912,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.436458+0000","last_change":"2023-10-25T00:23:14.436458+0000","last_active":"2023-10-25T00:23:14.436458+0000","last_peered":"2023-10-25T00:23:14.436458+0000","last_clean":"2023-10-25T00:23:14.436458+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-25T00:23:14.436458+0000","last_undegraded":"2023-10-25T00:23:14.436458+0000","last_fullsized":"2023-10-25T00:23:14.436458+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3827","last_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_deep_scrub":"72'3827","last_deep_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.436150+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:27:32.423180+0000","scrub_duration":0.30547422699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":21199,"num_write_kb":277730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"68'4820","reported_seq":4907,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232177+0000","last_change":"2023-10-25T00:22:36.429763+0000","last_active":"2023-10-25T00:22:49.232177+0000","last_peered":"2023-10-25T00:22:49.232177+0000","last_clean":"2023-10-25T00:22:49.232177+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-25T00:22:49.232177+0000","last_undegraded":"2023-10-25T00:22:49.232177+0000","last_fullsized":"2023-10-25T00:22:49.232177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4820","last_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_deep_scrub":"68'4820","last_deep_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_clean_scrub_stamp":"2023-10-25T00:22:36.429607+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:35.883183+0000","scrub_duration":0.10613655600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25163,"num_write_kb":285293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"72'5","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.946770+0000","last_change":"2023-10-25T00:23:20.947028+0000","last_active":"2023-10-25T00:23:20.946770+0000","last_peered":"2023-10-25T00:23:20.946770+0000","last_clean":"2023-10-25T00:23:20.946770+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-25T00:23:20.946770+0000","last_undegraded":"2023-10-25T00:23:20.946770+0000","last_fullsized":"2023-10-25T00:23:20.946770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:23:20.946713+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:23:20.946713+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.946713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:12:40.769613+0000","scrub_duration":0.0045247550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"72'5","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.215935+0000","last_change":"2023-10-25T00:22:40.197550+0000","last_active":"2023-10-25T00:22:49.215935+0000","last_peered":"2023-10-25T00:22:49.215935+0000","last_clean":"2023-10-25T00:22:49.215935+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-25T00:22:49.215935+0000","last_undegraded":"2023-10-25T00:22:49.215935+0000","last_fullsized":"2023-10-25T00:22:49.215935+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.197213+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:07:28.509914+0000","scrub_duration":0.0047704979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"72'4957","reported_seq":5045,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:53.667906+0000","last_change":"2023-10-25T00:22:53.667906+0000","last_active":"2023-10-25T00:22:53.667906+0000","last_peered":"2023-10-25T00:22:53.667906+0000","last_clean":"2023-10-25T00:22:53.667906+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-25T00:22:53.667906+0000","last_undegraded":"2023-10-25T00:22:53.667906+0000","last_fullsized":"2023-10-25T00:22:53.667906+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4957","last_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_deep_scrub":"72'4957","last_deep_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.667748+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:58.307897+0000","scrub_duration":0.30812842099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28093,"num_write_kb":358545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'4782","reported_seq":4874,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234575+0000","last_change":"2023-10-25T00:22:42.562198+0000","last_active":"2023-10-25T00:22:49.234575+0000","last_peered":"2023-10-25T00:22:49.234575+0000","last_clean":"2023-10-25T00:22:49.234575+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-25T00:22:49.234575+0000","last_undegraded":"2023-10-25T00:22:49.234575+0000","last_fullsized":"2023-10-25T00:22:49.234575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4782","last_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_deep_scrub":"68'4782","last_deep_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.562057+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:39.415960+0000","scrub_duration":0.116720554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26432,"num_write_kb":298639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"68'3","reported_seq":77,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.522666+0000","last_change":"2023-10-24T23:53:02.354912+0000","last_active":"2023-10-25T00:22:36.522666+0000","last_peered":"2023-10-25T00:22:36.522666+0000","last_clean":"2023-10-25T00:22:36.522666+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-25T00:22:36.522666+0000","last_undegraded":"2023-10-25T00:22:36.522666+0000","last_fullsized":"2023-10-25T00:22:36.522666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"72'18","reported_seq":215,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.214421+0000","last_change":"2023-10-25T00:22:46.074083+0000","last_active":"2023-10-25T00:22:49.214421+0000","last_peered":"2023-10-25T00:22:49.214421+0000","last_clean":"2023-10-25T00:22:49.214421+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-25T00:22:49.214421+0000","last_undegraded":"2023-10-25T00:22:49.214421+0000","last_fullsized":"2023-10-25T00:22:49.214421+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_deep_scrub":"72'18","last_deep_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.073758+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:08:25.787090+0000","scrub_duration":0.012558368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":345,"num_read_kb":345,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"72'2954","reported_seq":3037,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.099045+0000","last_change":"2023-10-25T00:23:36.099045+0000","last_active":"2023-10-25T00:23:36.099045+0000","last_peered":"2023-10-25T00:23:36.099045+0000","last_clean":"2023-10-25T00:23:36.099045+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-25T00:23:36.099045+0000","last_undegraded":"2023-10-25T00:23:36.099045+0000","last_fullsized":"2023-10-25T00:23:36.099045+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2954","last_scrub_stamp":"2023-10-25T00:23:36.098826+0000","last_deep_scrub":"72'2954","last_deep_scrub_stamp":"2023-10-25T00:23:36.098826+0000","last_clean_scrub_stamp":"2023-10-25T00:23:36.098826+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:16.696480+0000","scrub_duration":0.32992982700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15961,"num_write_kb":219669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"73'38","reported_seq":169,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.052563+0000","last_change":"2023-10-25T00:23:35.052803+0000","last_active":"2023-10-25T00:23:35.052563+0000","last_peered":"2023-10-25T00:23:35.052563+0000","last_clean":"2023-10-25T00:23:35.052563+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-25T00:23:35.052563+0000","last_undegraded":"2023-10-25T00:23:35.052563+0000","last_fullsized":"2023-10-25T00:23:35.052563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'38","last_scrub_stamp":"2023-10-25T00:23:35.052506+0000","last_deep_scrub":"73'38","last_deep_scrub_stamp":"2023-10-25T00:23:35.052506+0000","last_clean_scrub_stamp":"2023-10-25T00:23:35.052506+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:23:31.370767+0000","scrub_duration":0.0095825770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'4690","reported_seq":4778,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.792729+0000","last_change":"2023-10-25T00:23:05.792729+0000","last_active":"2023-10-25T00:23:05.792729+0000","last_peered":"2023-10-25T00:23:05.792729+0000","last_clean":"2023-10-25T00:23:05.792729+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-25T00:23:05.792729+0000","last_undegraded":"2023-10-25T00:23:05.792729+0000","last_fullsized":"2023-10-25T00:23:05.792729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4690","last_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_deep_scrub":"68'4690","last_deep_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.792588+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:24:53.476824+0000","scrub_duration":0.15089322399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25172,"num_write_kb":305786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"72'10","reported_seq":136,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234044+0000","last_change":"2023-10-25T00:22:40.512549+0000","last_active":"2023-10-25T00:22:49.234044+0000","last_peered":"2023-10-25T00:22:49.234044+0000","last_clean":"2023-10-25T00:22:49.234044+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-25T00:22:49.234044+0000","last_undegraded":"2023-10-25T00:22:49.234044+0000","last_fullsized":"2023-10-25T00:22:49.234044+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.512409+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:07:23.705506+0000","scrub_duration":0.091817919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"72'4200","reported_seq":4291,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.809510+0000","last_change":"2023-10-25T00:23:34.809510+0000","last_active":"2023-10-25T00:23:34.809510+0000","last_peered":"2023-10-25T00:23:34.809510+0000","last_clean":"2023-10-25T00:23:34.809510+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-25T00:23:34.809510+0000","last_undegraded":"2023-10-25T00:23:34.809510+0000","last_fullsized":"2023-10-25T00:23:34.809510+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4200","last_scrub_stamp":"2023-10-25T00:23:34.809362+0000","last_deep_scrub":"72'4200","last_deep_scrub_stamp":"2023-10-25T00:23:34.809362+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.809362+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:04:30.830862+0000","scrub_duration":0.23894108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":24169,"num_write_kb":293796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"73'394","reported_seq":596,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:31.115748+0000","last_change":"2023-10-25T00:23:31.116025+0000","last_active":"2023-10-25T00:23:31.115748+0000","last_peered":"2023-10-25T00:23:31.115748+0000","last_clean":"2023-10-25T00:23:31.115748+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-25T00:23:31.115748+0000","last_undegraded":"2023-10-25T00:23:31.115748+0000","last_fullsized":"2023-10-25T00:23:31.115748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'394","last_scrub_stamp":"2023-10-25T00:23:31.115692+0000","last_deep_scrub":"73'394","last_deep_scrub_stamp":"2023-10-25T00:23:31.115692+0000","last_clean_scrub_stamp":"2023-10-25T00:23:31.115692+0000","objects_scrubbed":4,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:38.234847+0000","scrub_duration":0.026138159000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":210,"num_read_kb":94395,"num_write":387,"num_write_kb":4304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'5730","reported_seq":5817,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.204004+0000","last_change":"2023-10-25T00:23:30.204139+0000","last_active":"2023-10-25T00:23:30.204004+0000","last_peered":"2023-10-25T00:23:30.204004+0000","last_clean":"2023-10-25T00:23:30.204004+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-25T00:23:30.204004+0000","last_undegraded":"2023-10-25T00:23:30.204004+0000","last_fullsized":"2023-10-25T00:23:30.204004+0000","mapping_epoch":21,"log_start":"61'500","ondisk_log_start":"61'500","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5730","last_scrub_stamp":"2023-10-25T00:23:30.203981+0000","last_deep_scrub":"68'5730","last_deep_scrub_stamp":"2023-10-25T00:23:30.203981+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.203981+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:38:35.578976+0000","scrub_duration":0.10293530200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":31652,"num_write_kb":356481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"72'3","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.639911+0000","last_change":"2023-10-25T00:23:27.639911+0000","last_active":"2023-10-25T00:23:27.639911+0000","last_peered":"2023-10-25T00:23:27.639911+0000","last_clean":"2023-10-25T00:23:27.639911+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-25T00:23:27.639911+0000","last_undegraded":"2023-10-25T00:23:27.639911+0000","last_fullsized":"2023-10-25T00:23:27.639911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:27.639604+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:27.639604+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.639604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:09:33.666294+0000","scrub_duration":0.0047758339999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"72'4168","reported_seq":4242,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437378+0000","last_change":"2023-10-24T23:53:11.848037+0000","last_active":"2023-10-25T00:22:37.437378+0000","last_peered":"2023-10-25T00:22:37.437378+0000","last_clean":"2023-10-25T00:22:37.437378+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-25T00:22:37.437378+0000","last_undegraded":"2023-10-25T00:22:37.437378+0000","last_fullsized":"2023-10-25T00:22:37.437378+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23458,"num_write_kb":294850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"68'4339","reported_seq":4415,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060019+0000","last_change":"2023-10-24T23:53:02.723007+0000","last_active":"2023-10-25T00:22:36.060019+0000","last_peered":"2023-10-25T00:22:36.060019+0000","last_clean":"2023-10-25T00:22:36.060019+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-25T00:22:36.060019+0000","last_undegraded":"2023-10-25T00:22:36.060019+0000","last_fullsized":"2023-10-25T00:22:36.060019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23267,"num_write_kb":265268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.223439+0000","last_change":"2023-10-25T00:22:42.199286+0000","last_active":"2023-10-25T00:22:49.223439+0000","last_peered":"2023-10-25T00:22:49.223439+0000","last_clean":"2023-10-25T00:22:49.223439+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-25T00:22:49.223439+0000","last_undegraded":"2023-10-25T00:22:49.223439+0000","last_fullsized":"2023-10-25T00:22:49.223439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.198947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:11:17.807377+0000","scrub_duration":0.0041955760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"73'231","reported_seq":342,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.010034+0000","last_change":"2023-10-25T00:23:16.010034+0000","last_active":"2023-10-25T00:23:16.010034+0000","last_peered":"2023-10-25T00:23:16.010034+0000","last_clean":"2023-10-25T00:23:16.010034+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-25T00:23:16.010034+0000","last_undegraded":"2023-10-25T00:23:16.010034+0000","last_fullsized":"2023-10-25T00:23:16.010034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'231","last_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_deep_scrub":"73'231","last_deep_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.009842+0000","objects_scrubbed":1,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:27:21.881029+0000","scrub_duration":0.076880452000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":63,"num_write":590,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"72'11","reported_seq":95,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.231986+0000","last_change":"2023-10-25T00:22:38.409520+0000","last_active":"2023-10-25T00:22:49.231986+0000","last_peered":"2023-10-25T00:22:49.231986+0000","last_clean":"2023-10-25T00:22:49.231986+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-25T00:22:49.231986+0000","last_undegraded":"2023-10-25T00:22:49.231986+0000","last_fullsized":"2023-10-25T00:22:49.231986+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_deep_scrub":"72'11","last_deep_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.409235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:21:24.708709+0000","scrub_duration":0.003592329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"72'3993","reported_seq":4069,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060792+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-25T00:22:36.060792+0000","last_peered":"2023-10-25T00:22:36.060792+0000","last_clean":"2023-10-25T00:22:36.060792+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-25T00:22:36.060792+0000","last_undegraded":"2023-10-25T00:22:36.060792+0000","last_fullsized":"2023-10-25T00:22:36.060792+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21723,"num_write_kb":283850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'4270","reported_seq":4358,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:59.570791+0000","last_change":"2023-10-25T00:22:59.570791+0000","last_active":"2023-10-25T00:22:59.570791+0000","last_peered":"2023-10-25T00:22:59.570791+0000","last_clean":"2023-10-25T00:22:59.570791+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-25T00:22:59.570791+0000","last_undegraded":"2023-10-25T00:22:59.570791+0000","last_fullsized":"2023-10-25T00:22:59.570791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4270","last_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_deep_scrub":"68'4270","last_deep_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.570628+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:36:56.294621+0000","scrub_duration":0.14092118100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23049,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.643510+0000","last_change":"2023-10-25T00:23:06.643510+0000","last_active":"2023-10-25T00:23:06.643510+0000","last_peered":"2023-10-25T00:23:06.643510+0000","last_clean":"2023-10-25T00:23:06.643510+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-25T00:23:06.643510+0000","last_undegraded":"2023-10-25T00:23:06.643510+0000","last_fullsized":"2023-10-25T00:23:06.643510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.643156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:34:24.773001+0000","scrub_duration":0.0045495850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"73'10","reported_seq":145,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.618489+0000","last_change":"2023-10-25T00:23:20.618489+0000","last_active":"2023-10-25T00:23:20.618489+0000","last_peered":"2023-10-25T00:23:20.618489+0000","last_clean":"2023-10-25T00:23:20.618489+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-25T00:23:20.618489+0000","last_undegraded":"2023-10-25T00:23:20.618489+0000","last_fullsized":"2023-10-25T00:23:20.618489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2023-10-25T00:23:20.618371+0000","last_deep_scrub":"73'10","last_deep_scrub_stamp":"2023-10-25T00:23:20.618371+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.618371+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:30:10.455022+0000","scrub_duration":0.085107656000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":8,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"73'65","reported_seq":228,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.930205+0000","last_change":"2023-10-25T00:23:07.930465+0000","last_active":"2023-10-25T00:23:07.930205+0000","last_peered":"2023-10-25T00:23:07.930205+0000","last_clean":"2023-10-25T00:23:07.930205+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-25T00:23:07.930205+0000","last_undegraded":"2023-10-25T00:23:07.930205+0000","last_fullsized":"2023-10-25T00:23:07.930205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'65","last_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_deep_scrub":"73'65","last_deep_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.930143+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:41:49.528670+0000","scrub_duration":0.020886851000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":123,"num_write":59,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"72'3769","reported_seq":3845,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.965701+0000","last_change":"2023-10-24T23:53:11.821845+0000","last_active":"2023-10-25T00:22:33.965701+0000","last_peered":"2023-10-25T00:22:33.965701+0000","last_clean":"2023-10-25T00:22:33.965701+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-25T00:22:33.965701+0000","last_undegraded":"2023-10-25T00:22:33.965701+0000","last_fullsized":"2023-10-25T00:22:33.965701+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20807,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'5752","reported_seq":5839,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.937946+0000","last_change":"2023-10-25T00:22:33.937946+0000","last_active":"2023-10-25T00:22:33.937946+0000","last_peered":"2023-10-25T00:22:33.937946+0000","last_clean":"2023-10-25T00:22:33.937946+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-25T00:22:33.937946+0000","last_undegraded":"2023-10-25T00:22:33.937946+0000","last_fullsized":"2023-10-25T00:22:33.937946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5752","last_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_deep_scrub":"68'5752","last_deep_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_clean_scrub_stamp":"2023-10-25T00:22:33.937564+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:43:34.779084+0000","scrub_duration":0.080678784000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31231,"num_write_kb":354718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:24.046769+0000","last_change":"2023-10-25T00:23:24.047030+0000","last_active":"2023-10-25T00:23:24.046769+0000","last_peered":"2023-10-25T00:23:24.046769+0000","last_clean":"2023-10-25T00:23:24.046769+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-25T00:23:24.046769+0000","last_undegraded":"2023-10-25T00:23:24.046769+0000","last_fullsized":"2023-10-25T00:23:24.046769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:24.046716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:24.046716+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.046716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:53:42.506468+0000","scrub_duration":0.0043444529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'2","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:18.683846+0000","last_change":"2023-10-25T00:23:18.684046+0000","last_active":"2023-10-25T00:23:18.683846+0000","last_peered":"2023-10-25T00:23:18.683846+0000","last_clean":"2023-10-25T00:23:18.683846+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-25T00:23:18.683846+0000","last_undegraded":"2023-10-25T00:23:18.683846+0000","last_fullsized":"2023-10-25T00:23:18.683846+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2023-10-25T00:23:18.683799+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2023-10-25T00:23:18.683799+0000","last_clean_scrub_stamp":"2023-10-25T00:23:18.683799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:26:44.716117+0000","scrub_duration":0.013956036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"72'4","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.496631+0000","last_change":"2023-10-25T00:23:06.496904+0000","last_active":"2023-10-25T00:23:06.496631+0000","last_peered":"2023-10-25T00:23:06.496631+0000","last_clean":"2023-10-25T00:23:06.496631+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-25T00:23:06.496631+0000","last_undegraded":"2023-10-25T00:23:06.496631+0000","last_fullsized":"2023-10-25T00:23:06.496631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_deep_scrub":"72'4","last_deep_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.496575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:32:18.665160+0000","scrub_duration":0.0046312189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"72'3939","reported_seq":4022,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.160279+0000","last_change":"2023-10-25T00:23:02.160279+0000","last_active":"2023-10-25T00:23:02.160279+0000","last_peered":"2023-10-25T00:23:02.160279+0000","last_clean":"2023-10-25T00:23:02.160279+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-25T00:23:02.160279+0000","last_undegraded":"2023-10-25T00:23:02.160279+0000","last_fullsized":"2023-10-25T00:23:02.160279+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3939","last_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_deep_scrub":"72'3939","last_deep_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.160086+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:37:29.878702+0000","scrub_duration":0.252775894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22393,"num_write_kb":262004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'4305","reported_seq":4392,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.858696+0000","last_change":"2023-10-25T00:23:26.858862+0000","last_active":"2023-10-25T00:23:26.858696+0000","last_peered":"2023-10-25T00:23:26.858696+0000","last_clean":"2023-10-25T00:23:26.858696+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-25T00:23:26.858696+0000","last_undegraded":"2023-10-25T00:23:26.858696+0000","last_fullsized":"2023-10-25T00:23:26.858696+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4305","last_scrub_stamp":"2023-10-25T00:23:26.858657+0000","last_deep_scrub":"68'4305","last_deep_scrub_stamp":"2023-10-25T00:23:26.858657+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.858657+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:10:03.352691+0000","scrub_duration":0.18050687100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22696,"num_write_kb":287574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:11.717097+0000","last_change":"2023-10-25T00:23:11.717353+0000","last_active":"2023-10-25T00:23:11.717097+0000","last_peered":"2023-10-25T00:23:11.717097+0000","last_clean":"2023-10-25T00:23:11.717097+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-25T00:23:11.717097+0000","last_undegraded":"2023-10-25T00:23:11.717097+0000","last_fullsized":"2023-10-25T00:23:11.717097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_clean_scrub_stamp":"2023-10-25T00:23:11.717027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:17:42.982457+0000","scrub_duration":0.0039434270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.354835+0000","last_change":"2023-10-25T00:23:30.355099+0000","last_active":"2023-10-25T00:23:30.354835+0000","last_peered":"2023-10-25T00:23:30.354835+0000","last_clean":"2023-10-25T00:23:30.354835+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-25T00:23:30.354835+0000","last_undegraded":"2023-10-25T00:23:30.354835+0000","last_fullsized":"2023-10-25T00:23:30.354835+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2023-10-25T00:23:30.354780+0000","last_deep_scrub":"67'2","last_deep_scrub_stamp":"2023-10-25T00:23:30.354780+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.354780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:07:40.743916+0000","scrub_duration":0.0044225610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"72'3","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.952645+0000","last_change":"2023-10-25T00:22:52.952897+0000","last_active":"2023-10-25T00:22:52.952645+0000","last_peered":"2023-10-25T00:22:52.952645+0000","last_clean":"2023-10-25T00:22:52.952645+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-25T00:22:52.952645+0000","last_undegraded":"2023-10-25T00:22:52.952645+0000","last_fullsized":"2023-10-25T00:22:52.952645+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.952595+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:14:20.890322+0000","scrub_duration":0.0044650949999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"72'5003","reported_seq":5081,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.235501+0000","last_change":"2023-10-24T23:53:11.828083+0000","last_active":"2023-10-25T00:22:49.235501+0000","last_peered":"2023-10-25T00:22:49.235501+0000","last_clean":"2023-10-25T00:22:49.235501+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-25T00:22:49.235501+0000","last_undegraded":"2023-10-25T00:22:49.235501+0000","last_fullsized":"2023-10-25T00:22:49.235501+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":13,"num_write":27120,"num_write_kb":350355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'5635","reported_seq":5721,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:44.547139+0000","last_change":"2023-10-25T00:22:44.547139+0000","last_active":"2023-10-25T00:22:44.547139+0000","last_peered":"2023-10-25T00:22:44.547139+0000","last_clean":"2023-10-25T00:22:44.547139+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-25T00:22:44.547139+0000","last_undegraded":"2023-10-25T00:22:44.547139+0000","last_fullsized":"2023-10-25T00:22:44.547139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5635","last_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_deep_scrub":"68'5635","last_deep_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.546984+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:51:54.474684+0000","scrub_duration":0.15865177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30556,"num_write_kb":356140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"68'3","reported_seq":80,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.235441+0000","last_change":"2023-10-25T00:23:35.979143+0000","last_active":"2023-10-25T00:22:49.235441+0000","last_peered":"2023-10-25T00:22:49.235441+0000","last_clean":"2023-10-25T00:22:49.235441+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-25T00:22:49.235441+0000","last_undegraded":"2023-10-25T00:22:49.235441+0000","last_fullsized":"2023-10-25T00:22:49.235441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:29.305074+0000","last_change":"2023-10-25T00:23:29.305381+0000","last_active":"2023-10-25T00:23:29.305074+0000","last_peered":"2023-10-25T00:23:29.305074+0000","last_clean":"2023-10-25T00:23:29.305074+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-25T00:23:29.305074+0000","last_undegraded":"2023-10-25T00:23:29.305074+0000","last_fullsized":"2023-10-25T00:23:29.305074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:29.305022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:29.305022+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.305022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:52:12.477955+0000","scrub_duration":0.0044947839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"72'2","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:58.925373+0000","last_change":"2023-10-25T00:22:58.925685+0000","last_active":"2023-10-25T00:22:58.925373+0000","last_peered":"2023-10-25T00:22:58.925373+0000","last_clean":"2023-10-25T00:22:58.925373+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-25T00:22:58.925373+0000","last_undegraded":"2023-10-25T00:22:58.925373+0000","last_fullsized":"2023-10-25T00:22:58.925373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.925318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:31:45.562398+0000","scrub_duration":0.0043101720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"72'3753","reported_seq":3828,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437502+0000","last_change":"2023-10-24T23:53:11.860212+0000","last_active":"2023-10-25T00:22:37.437502+0000","last_peered":"2023-10-25T00:22:37.437502+0000","last_clean":"2023-10-25T00:22:37.437502+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-25T00:22:37.437502+0000","last_undegraded":"2023-10-25T00:22:37.437502+0000","last_fullsized":"2023-10-25T00:22:37.437502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":21045,"num_write_kb":276986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'4101","reported_seq":4187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:25.811837+0000","last_change":"2023-10-25T00:23:25.811837+0000","last_active":"2023-10-25T00:23:25.811837+0000","last_peered":"2023-10-25T00:23:25.811837+0000","last_clean":"2023-10-25T00:23:25.811837+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-25T00:23:25.811837+0000","last_undegraded":"2023-10-25T00:23:25.811837+0000","last_fullsized":"2023-10-25T00:23:25.811837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:23:25.811682+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:23:25.811682+0000","last_clean_scrub_stamp":"2023-10-25T00:23:25.811682+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:39:30.752063+0000","scrub_duration":0.14318265299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22383,"num_write_kb":270023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"73'13","reported_seq":101,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:17.651156+0000","last_change":"2023-10-25T00:23:17.651450+0000","last_active":"2023-10-25T00:23:17.651156+0000","last_peered":"2023-10-25T00:23:17.651156+0000","last_clean":"2023-10-25T00:23:17.651156+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-25T00:23:17.651156+0000","last_undegraded":"2023-10-25T00:23:17.651156+0000","last_fullsized":"2023-10-25T00:23:17.651156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2023-10-25T00:23:17.651103+0000","last_deep_scrub":"73'13","last_deep_scrub_stamp":"2023-10-25T00:23:17.651103+0000","last_clean_scrub_stamp":"2023-10-25T00:23:17.651103+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:54.986553+0000","scrub_duration":0.014684019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:51.317510+0000","last_change":"2023-10-25T00:22:51.317813+0000","last_active":"2023-10-25T00:22:51.317510+0000","last_peered":"2023-10-25T00:22:51.317510+0000","last_clean":"2023-10-25T00:22:51.317510+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-25T00:22:51.317510+0000","last_undegraded":"2023-10-25T00:22:51.317510+0000","last_fullsized":"2023-10-25T00:22:51.317510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.317462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:03:14.150576+0000","scrub_duration":0.0044508079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"73'249","reported_seq":354,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.397705+0000","last_change":"2023-10-25T00:23:21.397705+0000","last_active":"2023-10-25T00:23:21.397705+0000","last_peered":"2023-10-25T00:23:21.397705+0000","last_clean":"2023-10-25T00:23:21.397705+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-25T00:23:21.397705+0000","last_undegraded":"2023-10-25T00:23:21.397705+0000","last_fullsized":"2023-10-25T00:23:21.397705+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'249","last_scrub_stamp":"2023-10-25T00:23:21.397410+0000","last_deep_scrub":"73'249","last_deep_scrub_stamp":"2023-10-25T00:23:21.397410+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.397410+0000","objects_scrubbed":1,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:19:14.548554+0000","scrub_duration":0.017693341000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":62,"num_write":607,"num_write_kb":223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"72'5236","reported_seq":5324,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.899914+0000","last_change":"2023-10-25T00:23:34.900038+0000","last_active":"2023-10-25T00:23:34.899914+0000","last_peered":"2023-10-25T00:23:34.899914+0000","last_clean":"2023-10-25T00:23:34.899914+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-25T00:23:34.899914+0000","last_undegraded":"2023-10-25T00:23:34.899914+0000","last_fullsized":"2023-10-25T00:23:34.899914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5236","last_scrub_stamp":"2023-10-25T00:23:34.899894+0000","last_deep_scrub":"72'5236","last_deep_scrub_stamp":"2023-10-25T00:23:34.899894+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.899894+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:10:20.497931+0000","scrub_duration":0.29917875999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":29554,"num_write_kb":368820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'5444","reported_seq":5534,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.141618+0000","last_change":"2023-10-25T00:23:00.141618+0000","last_active":"2023-10-25T00:23:00.141618+0000","last_peered":"2023-10-25T00:23:00.141618+0000","last_clean":"2023-10-25T00:23:00.141618+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-25T00:23:00.141618+0000","last_undegraded":"2023-10-25T00:23:00.141618+0000","last_fullsized":"2023-10-25T00:23:00.141618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5444","last_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_deep_scrub":"68'5444","last_deep_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.141458+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:23:18.614380+0000","scrub_duration":0.197741424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28342,"num_write_kb":342931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"73'67","reported_seq":238,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.941625+0000","last_change":"2023-10-25T00:23:14.941868+0000","last_active":"2023-10-25T00:23:14.941625+0000","last_peered":"2023-10-25T00:23:14.941625+0000","last_clean":"2023-10-25T00:23:14.941625+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-25T00:23:14.941625+0000","last_undegraded":"2023-10-25T00:23:14.941625+0000","last_fullsized":"2023-10-25T00:23:14.941625+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'67","last_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_deep_scrub":"73'67","last_deep_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.941564+0000","objects_scrubbed":1,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:11:54.084201+0000","scrub_duration":0.0084463549999999991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":125,"num_write":61,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:22.074139+0000","last_change":"2023-10-25T00:23:22.074393+0000","last_active":"2023-10-25T00:23:22.074139+0000","last_peered":"2023-10-25T00:23:22.074139+0000","last_clean":"2023-10-25T00:23:22.074139+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-25T00:23:22.074139+0000","last_undegraded":"2023-10-25T00:23:22.074139+0000","last_fullsized":"2023-10-25T00:23:22.074139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:22.074087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:22.074087+0000","last_clean_scrub_stamp":"2023-10-25T00:23:22.074087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:59:48.682336+0000","scrub_duration":0.0044663139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"72'10","reported_seq":138,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.225873+0000","last_change":"2023-10-25T00:22:45.425308+0000","last_active":"2023-10-25T00:22:49.225873+0000","last_peered":"2023-10-25T00:22:49.225873+0000","last_clean":"2023-10-25T00:22:49.225873+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-25T00:22:49.225873+0000","last_undegraded":"2023-10-25T00:22:49.225873+0000","last_fullsized":"2023-10-25T00:22:49.225873+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.425052+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:36:38.527058+0000","scrub_duration":0.14014134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":135,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"72'3666","reported_seq":3738,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.224676+0000","last_change":"2023-10-24T23:53:11.828445+0000","last_active":"2023-10-25T00:22:49.224676+0000","last_peered":"2023-10-25T00:22:49.224676+0000","last_clean":"2023-10-25T00:22:49.224676+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-25T00:22:49.224676+0000","last_undegraded":"2023-10-25T00:22:49.224676+0000","last_fullsized":"2023-10-25T00:22:49.224676+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20519,"num_write_kb":254391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"68'4182","reported_seq":4269,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.489530+0000","last_change":"2023-10-25T00:23:02.489861+0000","last_active":"2023-10-25T00:23:02.489530+0000","last_peered":"2023-10-25T00:23:02.489530+0000","last_clean":"2023-10-25T00:23:02.489530+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-25T00:23:02.489530+0000","last_undegraded":"2023-10-25T00:23:02.489530+0000","last_fullsized":"2023-10-25T00:23:02.489530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4182","last_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_deep_scrub":"68'4182","last_deep_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.489471+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:14:08.666419+0000","scrub_duration":0.092197608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22839,"num_write_kb":259586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'4","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:15.544038+0000","last_change":"2023-10-25T00:23:15.544275+0000","last_active":"2023-10-25T00:23:15.544038+0000","last_peered":"2023-10-25T00:23:15.544038+0000","last_clean":"2023-10-25T00:23:15.544038+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-25T00:23:15.544038+0000","last_undegraded":"2023-10-25T00:23:15.544038+0000","last_fullsized":"2023-10-25T00:23:15.544038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_deep_scrub":"67'4","last_deep_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:15.543988+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:33:51.332032+0000","scrub_duration":0.004459345,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:28.260459+0000","last_change":"2023-10-25T00:23:28.260731+0000","last_active":"2023-10-25T00:23:28.260459+0000","last_peered":"2023-10-25T00:23:28.260459+0000","last_clean":"2023-10-25T00:23:28.260459+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-25T00:23:28.260459+0000","last_undegraded":"2023-10-25T00:23:28.260459+0000","last_fullsized":"2023-10-25T00:23:28.260459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:28.260409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:28.260409+0000","last_clean_scrub_stamp":"2023-10-25T00:23:28.260409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:10:44.862291+0000","scrub_duration":0.0042298140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'163","reported_seq":299,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:17.149545+0000","last_change":"2023-10-25T00:23:17.149791+0000","last_active":"2023-10-25T00:23:17.149545+0000","last_peered":"2023-10-25T00:23:17.149545+0000","last_clean":"2023-10-25T00:23:17.149545+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-25T00:23:17.149545+0000","last_undegraded":"2023-10-25T00:23:17.149545+0000","last_fullsized":"2023-10-25T00:23:17.149545+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'163","last_scrub_stamp":"2023-10-25T00:23:17.149492+0000","last_deep_scrub":"73'163","last_deep_scrub_stamp":"2023-10-25T00:23:17.149492+0000","last_clean_scrub_stamp":"2023-10-25T00:23:17.149492+0000","objects_scrubbed":2,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:59:05.394868+0000","scrub_duration":0.012985958000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"72'2","reported_seq":85,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:48.067150+0000","last_change":"2023-10-25T00:22:48.067150+0000","last_active":"2023-10-25T00:22:48.067150+0000","last_peered":"2023-10-25T00:22:48.067150+0000","last_clean":"2023-10-25T00:22:48.067150+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-25T00:22:48.067150+0000","last_undegraded":"2023-10-25T00:22:48.067150+0000","last_fullsized":"2023-10-25T00:22:48.067150+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.066810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:55:20.112674+0000","scrub_duration":0.0050122709999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"72'4105","reported_seq":4191,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.535400+0000","last_change":"2023-10-25T00:23:20.535530+0000","last_active":"2023-10-25T00:23:20.535400+0000","last_peered":"2023-10-25T00:23:20.535400+0000","last_clean":"2023-10-25T00:23:20.535400+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-25T00:23:20.535400+0000","last_undegraded":"2023-10-25T00:23:20.535400+0000","last_fullsized":"2023-10-25T00:23:20.535400+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4105","last_scrub_stamp":"2023-10-25T00:23:20.535378+0000","last_deep_scrub":"72'4105","last_deep_scrub_stamp":"2023-10-25T00:23:20.535378+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.535378+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:20:52.287445+0000","scrub_duration":0.31792611900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":19,"num_write":23147,"num_write_kb":297959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'5705","reported_seq":5782,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.231359+0000","last_change":"2023-10-25T00:22:53.576536+0000","last_active":"2023-10-25T00:22:49.231359+0000","last_peered":"2023-10-25T00:22:49.231359+0000","last_clean":"2023-10-25T00:22:49.231359+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-25T00:22:49.231359+0000","last_undegraded":"2023-10-25T00:22:49.231359+0000","last_fullsized":"2023-10-25T00:22:49.231359+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":29845,"num_write_kb":357286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"68'5","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:18.535332+0000","last_change":"2023-10-25T00:23:18.535332+0000","last_active":"2023-10-25T00:23:18.535332+0000","last_peered":"2023-10-25T00:23:18.535332+0000","last_clean":"2023-10-25T00:23:18.535332+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-25T00:23:18.535332+0000","last_undegraded":"2023-10-25T00:23:18.535332+0000","last_fullsized":"2023-10-25T00:23:18.535332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:18.534951+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:18.534951+0000","last_clean_scrub_stamp":"2023-10-25T00:23:18.534951+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:37.756970+0000","scrub_duration":0.0037857049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"72'7","reported_seq":124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060732+0000","last_change":"2023-10-24T23:53:11.450376+0000","last_active":"2023-10-25T00:22:36.060732+0000","last_peered":"2023-10-25T00:22:36.060732+0000","last_clean":"2023-10-25T00:22:36.060732+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-25T00:22:36.060732+0000","last_undegraded":"2023-10-25T00:22:36.060732+0000","last_fullsized":"2023-10-25T00:22:36.060732+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"72'4234","reported_seq":4308,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534135+0000","last_change":"2023-10-24T23:53:11.830114+0000","last_active":"2023-10-25T00:22:35.534135+0000","last_peered":"2023-10-25T00:22:35.534135+0000","last_clean":"2023-10-25T00:22:35.534135+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-25T00:22:35.534135+0000","last_undegraded":"2023-10-25T00:22:35.534135+0000","last_fullsized":"2023-10-25T00:22:35.534135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":15,"num_write":23173,"num_write_kb":302022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"68'4052","reported_seq":4127,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060267+0000","last_change":"2023-10-24T23:53:02.721637+0000","last_active":"2023-10-25T00:22:36.060267+0000","last_peered":"2023-10-25T00:22:36.060267+0000","last_clean":"2023-10-25T00:22:36.060267+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-25T00:22:36.060267+0000","last_undegraded":"2023-10-25T00:22:36.060267+0000","last_fullsized":"2023-10-25T00:22:36.060267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21486,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"68'2","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.046955+0000","last_change":"2023-10-25T00:23:34.047108+0000","last_active":"2023-10-25T00:23:34.046955+0000","last_peered":"2023-10-25T00:23:34.046955+0000","last_clean":"2023-10-25T00:23:34.046955+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-25T00:23:34.046955+0000","last_undegraded":"2023-10-25T00:23:34.046955+0000","last_fullsized":"2023-10-25T00:23:34.046955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2023-10-25T00:23:34.046917+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2023-10-25T00:23:34.046917+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.046917+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:50:34.912538+0000","scrub_duration":0.034858717999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"73'5","reported_seq":109,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:53.546236+0000","last_change":"2023-10-25T00:22:53.546332+0000","last_active":"2023-10-25T00:22:53.546236+0000","last_peered":"2023-10-25T00:22:53.546236+0000","last_clean":"2023-10-25T00:22:53.546236+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-25T00:22:53.546236+0000","last_undegraded":"2023-10-25T00:22:53.546236+0000","last_fullsized":"2023-10-25T00:22:53.546236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.546215+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:16:55.756192+0000","scrub_duration":0.27862564699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"72'4690","reported_seq":4764,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427502+0000","last_change":"2023-10-24T23:53:11.815749+0000","last_active":"2023-10-25T00:22:34.427502+0000","last_peered":"2023-10-25T00:22:34.427502+0000","last_clean":"2023-10-25T00:22:34.427502+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-25T00:22:34.427502+0000","last_undegraded":"2023-10-25T00:22:34.427502+0000","last_fullsized":"2023-10-25T00:22:34.427502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24994,"num_write_kb":341798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'4780","reported_seq":4868,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:09.092847+0000","last_change":"2023-10-25T00:23:09.093139+0000","last_active":"2023-10-25T00:23:09.092847+0000","last_peered":"2023-10-25T00:23:09.092847+0000","last_clean":"2023-10-25T00:23:09.092847+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-25T00:23:09.092847+0000","last_undegraded":"2023-10-25T00:23:09.092847+0000","last_fullsized":"2023-10-25T00:23:09.092847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4780","last_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_deep_scrub":"68'4780","last_deep_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.092796+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:58:49.347709+0000","scrub_duration":0.152359684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25083,"num_write_kb":293163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'3","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.218767+0000","last_change":"2023-10-25T00:22:48.328017+0000","last_active":"2023-10-25T00:22:49.218767+0000","last_peered":"2023-10-25T00:22:49.218767+0000","last_clean":"2023-10-25T00:22:49.218767+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-25T00:22:49.218767+0000","last_undegraded":"2023-10-25T00:22:49.218767+0000","last_fullsized":"2023-10-25T00:22:49.218767+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.327778+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:58.116941+0000","scrub_duration":0.061328572999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"72'5","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.576377+0000","last_change":"2023-10-25T00:22:55.576688+0000","last_active":"2023-10-25T00:22:55.576377+0000","last_peered":"2023-10-25T00:22:55.576377+0000","last_clean":"2023-10-25T00:22:55.576377+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-25T00:22:55.576377+0000","last_undegraded":"2023-10-25T00:22:55.576377+0000","last_fullsized":"2023-10-25T00:22:55.576377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.576324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:43.752897+0000","scrub_duration":0.0045481469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"72'3596","reported_seq":3671,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060821+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-25T00:22:36.060821+0000","last_peered":"2023-10-25T00:22:36.060821+0000","last_clean":"2023-10-25T00:22:36.060821+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-25T00:22:36.060821+0000","last_undegraded":"2023-10-25T00:22:36.060821+0000","last_fullsized":"2023-10-25T00:22:36.060821+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":19820,"num_write_kb":259784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"68'4970","reported_seq":5057,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.999669+0000","last_change":"2023-10-25T00:22:55.999798+0000","last_active":"2023-10-25T00:22:55.999669+0000","last_peered":"2023-10-25T00:22:55.999669+0000","last_clean":"2023-10-25T00:22:55.999669+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-25T00:22:55.999669+0000","last_undegraded":"2023-10-25T00:22:55.999669+0000","last_fullsized":"2023-10-25T00:22:55.999669+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4970","last_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_deep_scrub":"68'4970","last_deep_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.999644+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:15:47.681358+0000","scrub_duration":0.177095113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26721,"num_write_kb":311966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"73'1466","reported_seq":1605,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.466683+0000","last_change":"2023-10-25T00:23:06.467052+0000","last_active":"2023-10-25T00:23:06.466683+0000","last_peered":"2023-10-25T00:23:06.466683+0000","last_clean":"2023-10-25T00:23:06.466683+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-25T00:23:06.466683+0000","last_undegraded":"2023-10-25T00:23:06.466683+0000","last_fullsized":"2023-10-25T00:23:06.466683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1466","last_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_deep_scrub":"73'1466","last_deep_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.466569+0000","objects_scrubbed":2,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:21.696239+0000","scrub_duration":0.039946036999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":47798,"num_write":1439,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"72'6","reported_seq":100,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.292895+0000","last_change":"2023-10-25T00:23:22.330800+0000","last_active":"2023-10-25T00:22:49.292895+0000","last_peered":"2023-10-25T00:22:49.292895+0000","last_clean":"2023-10-25T00:22:49.292895+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-25T00:22:49.292895+0000","last_undegraded":"2023-10-25T00:22:49.292895+0000","last_fullsized":"2023-10-25T00:22:49.292895+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"72'3548","reported_seq":3634,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:32.093782+0000","last_change":"2023-10-25T00:23:32.093782+0000","last_active":"2023-10-25T00:23:32.093782+0000","last_peered":"2023-10-25T00:23:32.093782+0000","last_clean":"2023-10-25T00:23:32.093782+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-25T00:23:32.093782+0000","last_undegraded":"2023-10-25T00:23:32.093782+0000","last_fullsized":"2023-10-25T00:23:32.093782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3548","last_scrub_stamp":"2023-10-25T00:23:32.093615+0000","last_deep_scrub":"72'3548","last_deep_scrub_stamp":"2023-10-25T00:23:32.093615+0000","last_clean_scrub_stamp":"2023-10-25T00:23:32.093615+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:02:03.851334+0000","scrub_duration":0.28610911100000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20243,"num_write_kb":261539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"68'3393","reported_seq":3482,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.098602+0000","last_change":"2023-10-25T00:22:42.098602+0000","last_active":"2023-10-25T00:22:42.098602+0000","last_peered":"2023-10-25T00:22:42.098602+0000","last_clean":"2023-10-25T00:22:42.098602+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-25T00:22:42.098602+0000","last_undegraded":"2023-10-25T00:22:42.098602+0000","last_fullsized":"2023-10-25T00:22:42.098602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3393","last_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_deep_scrub":"68'3393","last_deep_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.098371+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:12.736136+0000","scrub_duration":0.153676797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17500,"num_write_kb":217638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"68'5","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:33.402669+0000","last_change":"2023-10-25T00:23:33.402936+0000","last_active":"2023-10-25T00:23:33.402669+0000","last_peered":"2023-10-25T00:23:33.402669+0000","last_clean":"2023-10-25T00:23:33.402669+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-25T00:23:33.402669+0000","last_undegraded":"2023-10-25T00:23:33.402669+0000","last_fullsized":"2023-10-25T00:23:33.402669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:33.402617+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:33.402617+0000","last_clean_scrub_stamp":"2023-10-25T00:23:33.402617+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:10:14.715542+0000","scrub_duration":0.0048941419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"72'3","reported_seq":76,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.292612+0000","last_change":"2023-10-25T00:22:49.292612+0000","last_active":"2023-10-25T00:22:49.292612+0000","last_peered":"2023-10-25T00:22:49.292612+0000","last_clean":"2023-10-25T00:22:49.292612+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-25T00:22:49.292612+0000","last_undegraded":"2023-10-25T00:22:49.292612+0000","last_fullsized":"2023-10-25T00:22:49.292612+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"72'3584","reported_seq":3659,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.209677+0000","last_change":"2023-10-24T23:53:11.860207+0000","last_active":"2023-10-25T00:22:49.209677+0000","last_peered":"2023-10-25T00:22:49.209677+0000","last_clean":"2023-10-25T00:22:49.209677+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-25T00:22:49.209677+0000","last_undegraded":"2023-10-25T00:22:49.209677+0000","last_fullsized":"2023-10-25T00:22:49.209677+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20504,"num_write_kb":254394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'5094","reported_seq":5186,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:51.485282+0000","last_change":"2023-10-25T00:22:51.485282+0000","last_active":"2023-10-25T00:22:51.485282+0000","last_peered":"2023-10-25T00:22:51.485282+0000","last_clean":"2023-10-25T00:22:51.485282+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-25T00:22:51.485282+0000","last_undegraded":"2023-10-25T00:22:51.485282+0000","last_fullsized":"2023-10-25T00:22:51.485282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5094","last_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_deep_scrub":"68'5094","last_deep_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.484917+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:22:43.899134+0000","scrub_duration":0.19299439099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27763,"num_write_kb":328490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"73'7","reported_seq":116,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.094834+0000","last_change":"2023-10-25T00:23:27.095106+0000","last_active":"2023-10-25T00:23:27.094834+0000","last_peered":"2023-10-25T00:23:27.094834+0000","last_clean":"2023-10-25T00:23:27.094834+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-25T00:23:27.094834+0000","last_undegraded":"2023-10-25T00:23:27.094834+0000","last_fullsized":"2023-10-25T00:23:27.094834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'7","last_scrub_stamp":"2023-10-25T00:23:27.094773+0000","last_deep_scrub":"73'7","last_deep_scrub_stamp":"2023-10-25T00:23:27.094773+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.094773+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:52:00.838368+0000","scrub_duration":0.007318275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"73'1440","reported_seq":1570,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.527480+0000","last_change":"2023-10-25T00:23:16.527658+0000","last_active":"2023-10-25T00:23:16.527480+0000","last_peered":"2023-10-25T00:23:16.527480+0000","last_clean":"2023-10-25T00:23:16.527480+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-25T00:23:16.527480+0000","last_undegraded":"2023-10-25T00:23:16.527480+0000","last_fullsized":"2023-10-25T00:23:16.527480+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1440","last_scrub_stamp":"2023-10-25T00:23:16.527441+0000","last_deep_scrub":"73'1440","last_deep_scrub_stamp":"2023-10-25T00:23:16.527441+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.527441+0000","objects_scrubbed":2,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:47:53.223550+0000","scrub_duration":0.234060301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":35275,"num_write":1412,"num_write_kb":4487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"72'3292","reported_seq":3378,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.971739+0000","last_change":"2023-10-25T00:23:30.971878+0000","last_active":"2023-10-25T00:23:30.971739+0000","last_peered":"2023-10-25T00:23:30.971739+0000","last_clean":"2023-10-25T00:23:30.971739+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-25T00:23:30.971739+0000","last_undegraded":"2023-10-25T00:23:30.971739+0000","last_fullsized":"2023-10-25T00:23:30.971739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3292","last_scrub_stamp":"2023-10-25T00:23:30.971718+0000","last_deep_scrub":"72'3292","last_deep_scrub_stamp":"2023-10-25T00:23:30.971718+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.971718+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:45:07.787712+0000","scrub_duration":0.347700487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17853,"num_write_kb":243602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'4242","reported_seq":4329,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232419+0000","last_change":"2023-10-25T00:22:45.893397+0000","last_active":"2023-10-25T00:22:49.232419+0000","last_peered":"2023-10-25T00:22:49.232419+0000","last_clean":"2023-10-25T00:22:49.232419+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-25T00:22:49.232419+0000","last_undegraded":"2023-10-25T00:22:49.232419+0000","last_fullsized":"2023-10-25T00:22:49.232419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4242","last_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_deep_scrub":"67'4242","last_deep_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.893150+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:24:28.749675+0000","scrub_duration":0.21189509300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22687,"num_write_kb":275346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"73'8","reported_seq":140,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.063186+0000","last_change":"2023-10-25T00:23:30.063493+0000","last_active":"2023-10-25T00:23:30.063186+0000","last_peered":"2023-10-25T00:23:30.063186+0000","last_clean":"2023-10-25T00:23:30.063186+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-25T00:23:30.063186+0000","last_undegraded":"2023-10-25T00:23:30.063186+0000","last_fullsized":"2023-10-25T00:23:30.063186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2023-10-25T00:23:30.063132+0000","last_deep_scrub":"73'8","last_deep_scrub_stamp":"2023-10-25T00:23:30.063132+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.063132+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:12:23.891399+0000","scrub_duration":0.0076496660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"72'5","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:19.320206+0000","last_change":"2023-10-25T00:23:19.320206+0000","last_active":"2023-10-25T00:23:19.320206+0000","last_peered":"2023-10-25T00:23:19.320206+0000","last_clean":"2023-10-25T00:23:19.320206+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-25T00:23:19.320206+0000","last_undegraded":"2023-10-25T00:23:19.320206+0000","last_fullsized":"2023-10-25T00:23:19.320206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:23:19.319840+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:23:19.319840+0000","last_clean_scrub_stamp":"2023-10-25T00:23:19.319840+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:18:41.713579+0000","scrub_duration":0.0042872409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"72'4091","reported_seq":4165,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523278+0000","last_change":"2023-10-24T23:53:11.829574+0000","last_active":"2023-10-25T00:22:36.523278+0000","last_peered":"2023-10-25T00:22:36.523278+0000","last_clean":"2023-10-25T00:22:36.523278+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-25T00:22:36.523278+0000","last_undegraded":"2023-10-25T00:22:36.523278+0000","last_fullsized":"2023-10-25T00:22:36.523278+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22779,"num_write_kb":288098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'4310","reported_seq":4397,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:22.394323+0000","last_change":"2023-10-25T00:23:22.394323+0000","last_active":"2023-10-25T00:23:22.394323+0000","last_peered":"2023-10-25T00:23:22.394323+0000","last_clean":"2023-10-25T00:23:22.394323+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-25T00:23:22.394323+0000","last_undegraded":"2023-10-25T00:23:22.394323+0000","last_fullsized":"2023-10-25T00:23:22.394323+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4310","last_scrub_stamp":"2023-10-25T00:23:22.393940+0000","last_deep_scrub":"67'4310","last_deep_scrub_stamp":"2023-10-25T00:23:22.393940+0000","last_clean_scrub_stamp":"2023-10-25T00:23:22.393940+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:24:46.673127+0000","scrub_duration":0.085301248999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23791,"num_write_kb":283914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"73'5","reported_seq":114,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.183468+0000","last_change":"2023-10-25T00:23:26.183662+0000","last_active":"2023-10-25T00:23:26.183468+0000","last_peered":"2023-10-25T00:23:26.183468+0000","last_clean":"2023-10-25T00:23:26.183468+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-25T00:23:26.183468+0000","last_undegraded":"2023-10-25T00:23:26.183468+0000","last_fullsized":"2023-10-25T00:23:26.183468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:23:26.183430+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:23:26.183430+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.183430+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:28:22.710805+0000","scrub_duration":0.110908062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"72'2","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.206968+0000","last_change":"2023-10-25T00:22:38.034628+0000","last_active":"2023-10-25T00:22:49.206968+0000","last_peered":"2023-10-25T00:22:49.206968+0000","last_clean":"2023-10-25T00:22:49.206968+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-25T00:22:49.206968+0000","last_undegraded":"2023-10-25T00:22:49.206968+0000","last_fullsized":"2023-10-25T00:22:49.206968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.034499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:09.450121+0000","scrub_duration":0.14675892500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"72'3695","reported_seq":3770,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437533+0000","last_change":"2023-10-24T23:53:11.860533+0000","last_active":"2023-10-25T00:22:37.437533+0000","last_peered":"2023-10-25T00:22:37.437533+0000","last_clean":"2023-10-25T00:22:37.437533+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-25T00:22:37.437533+0000","last_undegraded":"2023-10-25T00:22:37.437533+0000","last_fullsized":"2023-10-25T00:22:37.437533+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21426,"num_write_kb":254037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"72'3","reported_seq":84,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.331112+0000","last_change":"2023-10-25T00:22:42.331112+0000","last_active":"2023-10-25T00:22:42.331112+0000","last_peered":"2023-10-25T00:22:42.331112+0000","last_clean":"2023-10-25T00:22:42.331112+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-25T00:22:42.331112+0000","last_undegraded":"2023-10-25T00:22:42.331112+0000","last_fullsized":"2023-10-25T00:22:42.331112+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.330786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:23:56.573805+0000","scrub_duration":0.0043674530000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"72'19","reported_seq":218,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.645016+0000","last_change":"2023-10-25T00:22:47.253320+0000","last_active":"2023-10-25T00:22:52.645016+0000","last_peered":"2023-10-25T00:22:52.645016+0000","last_clean":"2023-10-25T00:22:52.645016+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-25T00:22:52.645016+0000","last_undegraded":"2023-10-25T00:22:52.645016+0000","last_fullsized":"2023-10-25T00:22:52.645016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_deep_scrub":"72'19","last_deep_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.252995+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:03.720456+0000","scrub_duration":0.018599613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":339,"num_read_kb":339,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'5249","reported_seq":5342,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.439199+0000","last_change":"2023-10-25T00:22:52.439461+0000","last_active":"2023-10-25T00:22:52.439199+0000","last_peered":"2023-10-25T00:22:52.439199+0000","last_clean":"2023-10-25T00:22:52.439199+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-25T00:22:52.439199+0000","last_undegraded":"2023-10-25T00:22:52.439199+0000","last_fullsized":"2023-10-25T00:22:52.439199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5249","last_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_deep_scrub":"68'5249","last_deep_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.439150+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:19:03.730037+0000","scrub_duration":0.077069795999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":27646,"num_write_kb":332285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"72'4380","reported_seq":4453,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.293442+0000","last_change":"2023-10-24T23:53:11.845359+0000","last_active":"2023-10-25T00:22:49.293442+0000","last_peered":"2023-10-25T00:22:49.293442+0000","last_clean":"2023-10-25T00:22:49.293442+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-25T00:22:49.293442+0000","last_undegraded":"2023-10-25T00:22:49.293442+0000","last_fullsized":"2023-10-25T00:22:49.293442+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23662,"num_write_kb":324338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"72'5","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:50.617951+0000","last_change":"2023-10-25T00:22:50.618067+0000","last_active":"2023-10-25T00:22:50.617951+0000","last_peered":"2023-10-25T00:22:50.617951+0000","last_clean":"2023-10-25T00:22:50.617951+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-25T00:22:50.617951+0000","last_undegraded":"2023-10-25T00:22:50.617951+0000","last_fullsized":"2023-10-25T00:22:50.617951+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.617931+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:24.381368+0000","scrub_duration":0.234626104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"68'5","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:25.067489+0000","last_change":"2023-10-25T00:23:25.067489+0000","last_active":"2023-10-25T00:23:25.067489+0000","last_peered":"2023-10-25T00:23:25.067489+0000","last_clean":"2023-10-25T00:23:25.067489+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-25T00:23:25.067489+0000","last_undegraded":"2023-10-25T00:23:25.067489+0000","last_fullsized":"2023-10-25T00:23:25.067489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:25.067146+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:25.067146+0000","last_clean_scrub_stamp":"2023-10-25T00:23:25.067146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:32.009959+0000","scrub_duration":0.0041696520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'4442","reported_seq":4527,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:29.981270+0000","last_change":"2023-10-25T00:23:29.981423+0000","last_active":"2023-10-25T00:23:29.981270+0000","last_peered":"2023-10-25T00:23:29.981270+0000","last_clean":"2023-10-25T00:23:29.981270+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-25T00:23:29.981270+0000","last_undegraded":"2023-10-25T00:23:29.981270+0000","last_fullsized":"2023-10-25T00:23:29.981270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4442","last_scrub_stamp":"2023-10-25T00:23:29.981245+0000","last_deep_scrub":"68'4442","last_deep_scrub_stamp":"2023-10-25T00:23:29.981245+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.981245+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:11:38.194434+0000","scrub_duration":0.109448791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22810,"num_write_kb":296225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"72'4076","reported_seq":4150,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060841+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-25T00:22:36.060841+0000","last_peered":"2023-10-25T00:22:36.060841+0000","last_clean":"2023-10-25T00:22:36.060841+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-25T00:22:36.060841+0000","last_undegraded":"2023-10-25T00:22:36.060841+0000","last_fullsized":"2023-10-25T00:22:36.060841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22055,"num_write_kb":297609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"68'3","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:23.928686+0000","last_change":"2023-10-25T00:23:23.928686+0000","last_active":"2023-10-25T00:23:23.928686+0000","last_peered":"2023-10-25T00:23:23.928686+0000","last_clean":"2023-10-25T00:23:23.928686+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-25T00:23:23.928686+0000","last_undegraded":"2023-10-25T00:23:23.928686+0000","last_fullsized":"2023-10-25T00:23:23.928686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:23.928352+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:23.928352+0000","last_clean_scrub_stamp":"2023-10-25T00:23:23.928352+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:41:04.304281+0000","scrub_duration":0.0043703589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'4101","reported_seq":4187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:50.337590+0000","last_change":"2023-10-25T00:22:50.337815+0000","last_active":"2023-10-25T00:22:50.337590+0000","last_peered":"2023-10-25T00:22:50.337590+0000","last_clean":"2023-10-25T00:22:50.337590+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-25T00:22:50.337590+0000","last_undegraded":"2023-10-25T00:22:50.337590+0000","last_fullsized":"2023-10-25T00:22:50.337590+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.337536+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:01.275651+0000","scrub_duration":0.055303021000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22343,"num_write_kb":253486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"72'397","reported_seq":591,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234775+0000","last_change":"2023-10-25T00:22:45.554898+0000","last_active":"2023-10-25T00:22:49.234775+0000","last_peered":"2023-10-25T00:22:49.234775+0000","last_clean":"2023-10-25T00:22:49.234775+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-25T00:22:49.234775+0000","last_undegraded":"2023-10-25T00:22:49.234775+0000","last_fullsized":"2023-10-25T00:22:49.234775+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'397","last_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_deep_scrub":"72'397","last_deep_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.554526+0000","objects_scrubbed":4,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:41:32.056818+0000","scrub_duration":0.10248053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":86197,"num_write":389,"num_write_kb":4306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"72'4606","reported_seq":4678,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517449+0000","last_change":"2023-10-24T23:53:11.847103+0000","last_active":"2023-10-25T00:22:33.517449+0000","last_peered":"2023-10-25T00:22:33.517449+0000","last_clean":"2023-10-25T00:22:33.517449+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-25T00:22:33.517449+0000","last_undegraded":"2023-10-25T00:22:33.517449+0000","last_fullsized":"2023-10-25T00:22:33.517449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25763,"num_write_kb":316477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"73'14","reported_seq":146,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.153043+0000","last_change":"2023-10-25T00:23:16.153043+0000","last_active":"2023-10-25T00:23:16.153043+0000","last_peered":"2023-10-25T00:23:16.153043+0000","last_clean":"2023-10-25T00:23:16.153043+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-25T00:23:16.153043+0000","last_undegraded":"2023-10-25T00:23:16.153043+0000","last_fullsized":"2023-10-25T00:23:16.153043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_deep_scrub":"73'14","last_deep_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.152723+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:32:30.128331+0000","scrub_duration":0.010186763999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'5680","reported_seq":5770,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:54.586176+0000","last_change":"2023-10-25T00:22:54.586176+0000","last_active":"2023-10-25T00:22:54.586176+0000","last_peered":"2023-10-25T00:22:54.586176+0000","last_clean":"2023-10-25T00:22:54.586176+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-25T00:22:54.586176+0000","last_undegraded":"2023-10-25T00:22:54.586176+0000","last_fullsized":"2023-10-25T00:22:54.586176+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5680","last_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_deep_scrub":"68'5680","last_deep_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.585866+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:49.342364+0000","scrub_duration":0.20835251299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29749,"num_write_kb":360962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"73'40","reported_seq":187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:59.279339+0000","last_change":"2023-10-25T00:22:59.279527+0000","last_active":"2023-10-25T00:22:59.279339+0000","last_peered":"2023-10-25T00:22:59.279339+0000","last_clean":"2023-10-25T00:22:59.279339+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-25T00:22:59.279339+0000","last_undegraded":"2023-10-25T00:22:59.279339+0000","last_fullsized":"2023-10-25T00:22:59.279339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'40","last_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_deep_scrub":"73'40","last_deep_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.279288+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:52.911132+0000","scrub_duration":0.0094949800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":39,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"72'4808","reported_seq":4897,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.790951+0000","last_change":"2023-10-25T00:23:14.791099+0000","last_active":"2023-10-25T00:23:14.790951+0000","last_peered":"2023-10-25T00:23:14.790951+0000","last_clean":"2023-10-25T00:23:14.790951+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-25T00:23:14.790951+0000","last_undegraded":"2023-10-25T00:23:14.790951+0000","last_fullsized":"2023-10-25T00:23:14.790951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4808","last_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_deep_scrub":"72'4808","last_deep_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.790929+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:56:36.982903+0000","scrub_duration":0.29119922999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26552,"num_write_kb":336125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"68'4747","reported_seq":4835,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.496098+0000","last_change":"2023-10-25T00:23:00.496309+0000","last_active":"2023-10-25T00:23:00.496098+0000","last_peered":"2023-10-25T00:23:00.496098+0000","last_clean":"2023-10-25T00:23:00.496098+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-25T00:23:00.496098+0000","last_undegraded":"2023-10-25T00:23:00.496098+0000","last_fullsized":"2023-10-25T00:23:00.496098+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4747","last_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_deep_scrub":"68'4747","last_deep_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.496060+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:56.531125+0000","scrub_duration":0.089432630999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24847,"num_write_kb":316685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"72'4283","reported_seq":4370,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:08.728587+0000","last_change":"2023-10-25T00:23:08.728718+0000","last_active":"2023-10-25T00:23:08.728587+0000","last_peered":"2023-10-25T00:23:08.728587+0000","last_clean":"2023-10-25T00:23:08.728587+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-25T00:23:08.728587+0000","last_undegraded":"2023-10-25T00:23:08.728587+0000","last_fullsized":"2023-10-25T00:23:08.728587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4283","last_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_deep_scrub":"72'4283","last_deep_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_clean_scrub_stamp":"2023-10-25T00:23:08.728567+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:33.655445+0000","scrub_duration":0.26846236600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23853,"num_write_kb":296504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"68'4036","reported_seq":4124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:39.642537+0000","last_change":"2023-10-25T00:22:39.642537+0000","last_active":"2023-10-25T00:22:39.642537+0000","last_peered":"2023-10-25T00:22:39.642537+0000","last_clean":"2023-10-25T00:22:39.642537+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-25T00:22:39.642537+0000","last_undegraded":"2023-10-25T00:22:39.642537+0000","last_fullsized":"2023-10-25T00:22:39.642537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4036","last_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_deep_scrub":"68'4036","last_deep_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.642342+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:02.362976+0000","scrub_duration":0.227585228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21233,"num_write_kb":273467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"72'4276","reported_seq":4362,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.997835+0000","last_change":"2023-10-25T00:23:30.997946+0000","last_active":"2023-10-25T00:23:30.997835+0000","last_peered":"2023-10-25T00:23:30.997835+0000","last_clean":"2023-10-25T00:23:30.997835+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-25T00:23:30.997835+0000","last_undegraded":"2023-10-25T00:23:30.997835+0000","last_fullsized":"2023-10-25T00:23:30.997835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4276","last_scrub_stamp":"2023-10-25T00:23:30.997816+0000","last_deep_scrub":"72'4276","last_deep_scrub_stamp":"2023-10-25T00:23:30.997816+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.997816+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:08.863577+0000","scrub_duration":0.39360271099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23362,"num_write_kb":310817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"68'6201","reported_seq":6280,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060597+0000","last_change":"2023-10-24T23:53:02.717589+0000","last_active":"2023-10-25T00:22:36.060597+0000","last_peered":"2023-10-25T00:22:36.060597+0000","last_clean":"2023-10-25T00:22:36.060597+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-25T00:22:36.060597+0000","last_undegraded":"2023-10-25T00:22:36.060597+0000","last_fullsized":"2023-10-25T00:22:36.060597+0000","mapping_epoch":21,"log_start":"62'1000","ondisk_log_start":"62'1000","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33486,"num_write_kb":384441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"72'4018","reported_seq":4094,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.437698+0000","last_change":"2023-10-24T23:53:11.848269+0000","last_active":"2023-10-25T00:22:37.437698+0000","last_peered":"2023-10-25T00:22:37.437698+0000","last_clean":"2023-10-25T00:22:37.437698+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-25T00:22:37.437698+0000","last_undegraded":"2023-10-25T00:22:37.437698+0000","last_fullsized":"2023-10-25T00:22:37.437698+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":21920,"num_write_kb":288651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"68'4522","reported_seq":4611,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.751456+0000","last_change":"2023-10-25T00:23:26.751600+0000","last_active":"2023-10-25T00:23:26.751456+0000","last_peered":"2023-10-25T00:23:26.751456+0000","last_clean":"2023-10-25T00:23:26.751456+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-25T00:23:26.751456+0000","last_undegraded":"2023-10-25T00:23:26.751456+0000","last_fullsized":"2023-10-25T00:23:26.751456+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4522","last_scrub_stamp":"2023-10-25T00:23:26.751435+0000","last_deep_scrub":"68'4522","last_deep_scrub_stamp":"2023-10-25T00:23:26.751435+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.751435+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:40:36.338773+0000","scrub_duration":0.110684227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24476,"num_write_kb":282500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"72'4209","reported_seq":4282,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.233036+0000","last_change":"2023-10-24T23:53:11.805870+0000","last_active":"2023-10-25T00:22:49.233036+0000","last_peered":"2023-10-25T00:22:49.233036+0000","last_clean":"2023-10-25T00:22:49.233036+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-25T00:22:49.233036+0000","last_undegraded":"2023-10-25T00:22:49.233036+0000","last_fullsized":"2023-10-25T00:22:49.233036+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24141,"num_write_kb":281325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"68'4915","reported_seq":5002,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232375+0000","last_change":"2023-10-25T00:22:38.600278+0000","last_active":"2023-10-25T00:22:49.232375+0000","last_peered":"2023-10-25T00:22:49.232375+0000","last_clean":"2023-10-25T00:22:49.232375+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-25T00:22:49.232375+0000","last_undegraded":"2023-10-25T00:22:49.232375+0000","last_fullsized":"2023-10-25T00:22:49.232375+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4915","last_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_deep_scrub":"68'4915","last_deep_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.600135+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:42:30.671003+0000","scrub_duration":0.18827047799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26161,"num_write_kb":314084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"72'4933","reported_seq":5017,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:31.791243+0000","last_change":"2023-10-25T00:23:31.791243+0000","last_active":"2023-10-25T00:23:31.791243+0000","last_peered":"2023-10-25T00:23:31.791243+0000","last_clean":"2023-10-25T00:23:31.791243+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-25T00:23:31.791243+0000","last_undegraded":"2023-10-25T00:23:31.791243+0000","last_fullsized":"2023-10-25T00:23:31.791243+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4933","last_scrub_stamp":"2023-10-25T00:23:31.791069+0000","last_deep_scrub":"72'4933","last_deep_scrub_stamp":"2023-10-25T00:23:31.791069+0000","last_clean_scrub_stamp":"2023-10-25T00:23:31.791069+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:53:40.465688+0000","scrub_duration":0.269140133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26695,"num_write_kb":352905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"68'5055","reported_seq":5143,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.217349+0000","last_change":"2023-10-25T00:22:43.444464+0000","last_active":"2023-10-25T00:22:49.217349+0000","last_peered":"2023-10-25T00:22:49.217349+0000","last_clean":"2023-10-25T00:22:49.217349+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-25T00:22:49.217349+0000","last_undegraded":"2023-10-25T00:22:49.217349+0000","last_fullsized":"2023-10-25T00:22:49.217349+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5055","last_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_deep_scrub":"68'5055","last_deep_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_clean_scrub_stamp":"2023-10-25T00:22:43.444210+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:13:34.310177+0000","scrub_duration":0.081291211000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27516,"num_write_kb":327406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"72'4083","reported_seq":4160,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.215412+0000","last_change":"2023-10-24T23:53:11.823828+0000","last_active":"2023-10-25T00:22:49.215412+0000","last_peered":"2023-10-25T00:22:49.215412+0000","last_clean":"2023-10-25T00:22:49.215412+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-25T00:22:49.215412+0000","last_undegraded":"2023-10-25T00:22:49.215412+0000","last_fullsized":"2023-10-25T00:22:49.215412+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":22892,"num_write_kb":296872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"68'4777","reported_seq":4866,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:24.846143+0000","last_change":"2023-10-25T00:23:24.846143+0000","last_active":"2023-10-25T00:23:24.846143+0000","last_peered":"2023-10-25T00:23:24.846143+0000","last_clean":"2023-10-25T00:23:24.846143+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-25T00:23:24.846143+0000","last_undegraded":"2023-10-25T00:23:24.846143+0000","last_fullsized":"2023-10-25T00:23:24.846143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4777","last_scrub_stamp":"2023-10-25T00:23:24.846002+0000","last_deep_scrub":"68'4777","last_deep_scrub_stamp":"2023-10-25T00:23:24.846002+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.846002+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:41:16.418283+0000","scrub_duration":0.19199595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25812,"num_write_kb":300181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"72'4618","reported_seq":4708,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:33.839487+0000","last_change":"2023-10-25T00:23:33.839487+0000","last_active":"2023-10-25T00:23:33.839487+0000","last_peered":"2023-10-25T00:23:33.839487+0000","last_clean":"2023-10-25T00:23:33.839487+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-25T00:23:33.839487+0000","last_undegraded":"2023-10-25T00:23:33.839487+0000","last_fullsized":"2023-10-25T00:23:33.839487+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4618","last_scrub_stamp":"2023-10-25T00:23:33.839138+0000","last_deep_scrub":"72'4618","last_deep_scrub_stamp":"2023-10-25T00:23:33.839138+0000","last_clean_scrub_stamp":"2023-10-25T00:23:33.839138+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:25:32.655697+0000","scrub_duration":0.330454572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":26180,"num_write_kb":326342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"68'5340","reported_seq":5427,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.231448+0000","last_change":"2023-10-25T00:22:42.520048+0000","last_active":"2023-10-25T00:22:49.231448+0000","last_peered":"2023-10-25T00:22:49.231448+0000","last_clean":"2023-10-25T00:22:49.231448+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-25T00:22:49.231448+0000","last_undegraded":"2023-10-25T00:22:49.231448+0000","last_fullsized":"2023-10-25T00:22:49.231448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5340","last_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_deep_scrub":"68'5340","last_deep_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.519886+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.832889+0000","scrub_duration":0.129134375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28701,"num_write_kb":348521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"72'4171","reported_seq":4245,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060881+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-25T00:22:36.060881+0000","last_peered":"2023-10-25T00:22:36.060881+0000","last_clean":"2023-10-25T00:22:36.060881+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-25T00:22:36.060881+0000","last_undegraded":"2023-10-25T00:22:36.060881+0000","last_fullsized":"2023-10-25T00:22:36.060881+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22551,"num_write_kb":303498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"68'4493","reported_seq":4581,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:01.322975+0000","last_change":"2023-10-25T00:23:01.323167+0000","last_active":"2023-10-25T00:23:01.322975+0000","last_peered":"2023-10-25T00:23:01.322975+0000","last_clean":"2023-10-25T00:23:01.322975+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-25T00:23:01.322975+0000","last_undegraded":"2023-10-25T00:23:01.322975+0000","last_fullsized":"2023-10-25T00:23:01.322975+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4493","last_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_deep_scrub":"68'4493","last_deep_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_clean_scrub_stamp":"2023-10-25T00:23:01.322941+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:49.029022+0000","scrub_duration":0.13616751799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23797,"num_write_kb":300260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"72'4763","reported_seq":4836,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.226383+0000","last_change":"2023-10-24T23:53:44.474203+0000","last_active":"2023-10-25T00:22:49.226383+0000","last_peered":"2023-10-25T00:22:49.226383+0000","last_clean":"2023-10-25T00:22:49.226383+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-25T00:22:49.226383+0000","last_undegraded":"2023-10-25T00:22:49.226383+0000","last_fullsized":"2023-10-25T00:22:49.226383+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":25751,"num_write_kb":345225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"68'4946","reported_seq":5022,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:37.358533+0000","last_change":"2023-10-25T00:22:49.234045+0000","last_active":"2023-10-25T00:22:37.358533+0000","last_peered":"2023-10-25T00:22:37.358533+0000","last_clean":"2023-10-25T00:22:37.358533+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-25T00:22:37.358533+0000","last_undegraded":"2023-10-25T00:22:37.358533+0000","last_fullsized":"2023-10-25T00:22:37.358533+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26583,"num_write_kb":315633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"72'4352","reported_seq":4437,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.629523+0000","last_change":"2023-10-25T00:23:34.629523+0000","last_active":"2023-10-25T00:23:34.629523+0000","last_peered":"2023-10-25T00:23:34.629523+0000","last_clean":"2023-10-25T00:23:34.629523+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-25T00:23:34.629523+0000","last_undegraded":"2023-10-25T00:23:34.629523+0000","last_fullsized":"2023-10-25T00:23:34.629523+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4352","last_scrub_stamp":"2023-10-25T00:23:34.629323+0000","last_deep_scrub":"72'4352","last_deep_scrub_stamp":"2023-10-25T00:23:34.629323+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.629323+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:38.589150+0000","scrub_duration":0.29560794600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23957,"num_write_kb":301052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"68'5026","reported_seq":5102,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.226252+0000","last_change":"2023-10-25T00:23:30.833489+0000","last_active":"2023-10-25T00:22:49.226252+0000","last_peered":"2023-10-25T00:22:49.226252+0000","last_clean":"2023-10-25T00:22:49.226252+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-25T00:22:49.226252+0000","last_undegraded":"2023-10-25T00:22:49.226252+0000","last_fullsized":"2023-10-25T00:22:49.226252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26401,"num_write_kb":310586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"72'4393","reported_seq":4482,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.556819+0000","last_change":"2023-10-25T00:23:07.556990+0000","last_active":"2023-10-25T00:23:07.556819+0000","last_peered":"2023-10-25T00:23:07.556819+0000","last_clean":"2023-10-25T00:23:07.556819+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-25T00:23:07.556819+0000","last_undegraded":"2023-10-25T00:23:07.556819+0000","last_fullsized":"2023-10-25T00:23:07.556819+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4393","last_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_deep_scrub":"72'4393","last_deep_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.556784+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:21:00.928852+0000","scrub_duration":0.35410455800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":24752,"num_write_kb":296109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"68'5620","reported_seq":5710,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.194060+0000","last_change":"2023-10-25T00:23:12.194324+0000","last_active":"2023-10-25T00:23:12.194060+0000","last_peered":"2023-10-25T00:23:12.194060+0000","last_clean":"2023-10-25T00:23:12.194060+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-25T00:23:12.194060+0000","last_undegraded":"2023-10-25T00:23:12.194060+0000","last_fullsized":"2023-10-25T00:23:12.194060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5620","last_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_deep_scrub":"68'5620","last_deep_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.194008+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:33.868259+0000","scrub_duration":0.086061930999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29863,"num_write_kb":337169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"72'4340","reported_seq":4428,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:23.851965+0000","last_change":"2023-10-25T00:23:23.851965+0000","last_active":"2023-10-25T00:23:23.851965+0000","last_peered":"2023-10-25T00:23:23.851965+0000","last_clean":"2023-10-25T00:23:23.851965+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-25T00:23:23.851965+0000","last_undegraded":"2023-10-25T00:23:23.851965+0000","last_fullsized":"2023-10-25T00:23:23.851965+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4340","last_scrub_stamp":"2023-10-25T00:23:23.851809+0000","last_deep_scrub":"72'4340","last_deep_scrub_stamp":"2023-10-25T00:23:23.851809+0000","last_clean_scrub_stamp":"2023-10-25T00:23:23.851809+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:00.223998+0000","scrub_duration":0.37196321399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":36,"num_write":23367,"num_write_kb":302669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"68'4464","reported_seq":4549,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.904552+0000","last_change":"2023-10-25T00:22:57.904669+0000","last_active":"2023-10-25T00:22:57.904552+0000","last_peered":"2023-10-25T00:22:57.904552+0000","last_clean":"2023-10-25T00:22:57.904552+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-25T00:22:57.904552+0000","last_undegraded":"2023-10-25T00:22:57.904552+0000","last_fullsized":"2023-10-25T00:22:57.904552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4464","last_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_deep_scrub":"68'4464","last_deep_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.904532+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:17:44.026155+0000","scrub_duration":0.14132308199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24372,"num_write_kb":278074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"72'5589","reported_seq":5662,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.211056+0000","last_change":"2023-10-24T23:53:11.859473+0000","last_active":"2023-10-25T00:22:49.211056+0000","last_peered":"2023-10-25T00:22:49.211056+0000","last_clean":"2023-10-25T00:22:49.211056+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-25T00:22:49.211056+0000","last_undegraded":"2023-10-25T00:22:49.211056+0000","last_fullsized":"2023-10-25T00:22:49.211056+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":30206,"num_write_kb":399557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"68'4542","reported_seq":4628,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:09.344932+0000","last_change":"2023-10-25T00:23:09.345212+0000","last_active":"2023-10-25T00:23:09.344932+0000","last_peered":"2023-10-25T00:23:09.344932+0000","last_clean":"2023-10-25T00:23:09.344932+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-25T00:23:09.344932+0000","last_undegraded":"2023-10-25T00:23:09.344932+0000","last_fullsized":"2023-10-25T00:23:09.344932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4542","last_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_deep_scrub":"68'4542","last_deep_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.344860+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:16:09.325372+0000","scrub_duration":0.066583699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23889,"num_write_kb":292323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"72'3430","reported_seq":3515,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.797777+0000","last_change":"2023-10-25T00:23:21.798045+0000","last_active":"2023-10-25T00:23:21.797777+0000","last_peered":"2023-10-25T00:23:21.797777+0000","last_clean":"2023-10-25T00:23:21.797777+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-25T00:23:21.797777+0000","last_undegraded":"2023-10-25T00:23:21.797777+0000","last_fullsized":"2023-10-25T00:23:21.797777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3430","last_scrub_stamp":"2023-10-25T00:23:21.797727+0000","last_deep_scrub":"72'3430","last_deep_scrub_stamp":"2023-10-25T00:23:21.797727+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.797727+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:15:30.844796+0000","scrub_duration":0.29875561899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19411,"num_write_kb":241862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"68'4719","reported_seq":4806,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:58.903089+0000","last_change":"2023-10-25T00:22:58.903089+0000","last_active":"2023-10-25T00:22:58.903089+0000","last_peered":"2023-10-25T00:22:58.903089+0000","last_clean":"2023-10-25T00:22:58.903089+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-25T00:22:58.903089+0000","last_undegraded":"2023-10-25T00:22:58.903089+0000","last_fullsized":"2023-10-25T00:22:58.903089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4719","last_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_deep_scrub":"68'4719","last_deep_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.902769+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:01.236787+0000","scrub_duration":0.136999537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25020,"num_write_kb":305197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"72'4169","reported_seq":4240,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:34.427758+0000","last_change":"2023-10-24T23:53:11.809625+0000","last_active":"2023-10-25T00:22:34.427758+0000","last_peered":"2023-10-25T00:22:34.427758+0000","last_clean":"2023-10-25T00:22:34.427758+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-25T00:22:34.427758+0000","last_undegraded":"2023-10-25T00:22:34.427758+0000","last_fullsized":"2023-10-25T00:22:34.427758+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23235,"num_write_kb":306175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"68'5952","reported_seq":6047,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.017693+0000","last_change":"2023-10-25T00:23:05.017693+0000","last_active":"2023-10-25T00:23:05.017693+0000","last_peered":"2023-10-25T00:23:05.017693+0000","last_clean":"2023-10-25T00:23:05.017693+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-25T00:23:05.017693+0000","last_undegraded":"2023-10-25T00:23:05.017693+0000","last_fullsized":"2023-10-25T00:23:05.017693+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5952","last_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_deep_scrub":"68'5952","last_deep_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.017340+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:31.140541+0000","scrub_duration":0.055624675999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32341,"num_write_kb":375441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"72'3717","reported_seq":3789,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232259+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-25T00:22:49.232259+0000","last_peered":"2023-10-25T00:22:49.232259+0000","last_clean":"2023-10-25T00:22:49.232259+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-25T00:22:49.232259+0000","last_undegraded":"2023-10-25T00:22:49.232259+0000","last_fullsized":"2023-10-25T00:22:49.232259+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21101,"num_write_kb":269108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"68'4825","reported_seq":4913,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.089847+0000","last_change":"2023-10-25T00:23:06.089847+0000","last_active":"2023-10-25T00:23:06.089847+0000","last_peered":"2023-10-25T00:23:06.089847+0000","last_clean":"2023-10-25T00:23:06.089847+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-25T00:23:06.089847+0000","last_undegraded":"2023-10-25T00:23:06.089847+0000","last_fullsized":"2023-10-25T00:23:06.089847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4825","last_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_deep_scrub":"68'4825","last_deep_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.089705+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:04:51.412522+0000","scrub_duration":0.15071393499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25319,"num_write_kb":294113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"72'3714","reported_seq":3789,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.215543+0000","last_change":"2023-10-25T00:23:12.182256+0000","last_active":"2023-10-25T00:22:49.215543+0000","last_peered":"2023-10-25T00:22:49.215543+0000","last_clean":"2023-10-25T00:22:49.215543+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-25T00:22:49.215543+0000","last_undegraded":"2023-10-25T00:22:49.215543+0000","last_fullsized":"2023-10-25T00:22:49.215543+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21148,"num_write_kb":265324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"68'4878","reported_seq":4967,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.227455+0000","last_change":"2023-10-25T00:22:49.227455+0000","last_active":"2023-10-25T00:22:49.227455+0000","last_peered":"2023-10-25T00:22:49.227455+0000","last_clean":"2023-10-25T00:22:49.227455+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-25T00:22:49.227455+0000","last_undegraded":"2023-10-25T00:22:49.227455+0000","last_fullsized":"2023-10-25T00:22:49.227455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4878","last_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_deep_scrub":"68'4878","last_deep_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_clean_scrub_stamp":"2023-10-25T00:22:49.227012+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:58:21.157778+0000","scrub_duration":0.13340163999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26154,"num_write_kb":305558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"72'3434","reported_seq":3507,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.208392+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-25T00:22:49.208392+0000","last_peered":"2023-10-25T00:22:49.208392+0000","last_clean":"2023-10-25T00:22:49.208392+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-25T00:22:49.208392+0000","last_undegraded":"2023-10-25T00:22:49.208392+0000","last_fullsized":"2023-10-25T00:22:49.208392+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18449,"num_write_kb":262373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"68'4672","reported_seq":4749,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.216565+0000","last_change":"2023-10-24T23:53:02.694496+0000","last_active":"2023-10-25T00:22:49.216565+0000","last_peered":"2023-10-25T00:22:49.216565+0000","last_clean":"2023-10-25T00:22:49.216565+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-25T00:22:49.216565+0000","last_undegraded":"2023-10-25T00:22:49.216565+0000","last_fullsized":"2023-10-25T00:22:49.216565+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25223,"num_write_kb":293792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"72'4013","reported_seq":4091,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.213129+0000","last_change":"2023-10-24T23:53:11.821851+0000","last_active":"2023-10-25T00:22:49.213129+0000","last_peered":"2023-10-25T00:22:49.213129+0000","last_clean":"2023-10-25T00:22:49.213129+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-25T00:22:49.213129+0000","last_undegraded":"2023-10-25T00:22:49.213129+0000","last_fullsized":"2023-10-25T00:22:49.213129+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":22066,"num_write_kb":297298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"68'3918","reported_seq":3993,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.292751+0000","last_change":"2023-10-25T00:22:57.448743+0000","last_active":"2023-10-25T00:22:49.292751+0000","last_peered":"2023-10-25T00:22:49.292751+0000","last_clean":"2023-10-25T00:22:49.292751+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-25T00:22:49.292751+0000","last_undegraded":"2023-10-25T00:22:49.292751+0000","last_fullsized":"2023-10-25T00:22:49.292751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21675,"num_write_kb":247000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"72'4401","reported_seq":4474,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523617+0000","last_change":"2023-10-24T23:53:11.847308+0000","last_active":"2023-10-25T00:22:36.523617+0000","last_peered":"2023-10-25T00:22:36.523617+0000","last_clean":"2023-10-25T00:22:36.523617+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-25T00:22:36.523617+0000","last_undegraded":"2023-10-25T00:22:36.523617+0000","last_fullsized":"2023-10-25T00:22:36.523617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":24294,"num_write_kb":326749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"68'3970","reported_seq":4055,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.428569+0000","last_change":"2023-10-25T00:23:12.428867+0000","last_active":"2023-10-25T00:23:12.428569+0000","last_peered":"2023-10-25T00:23:12.428569+0000","last_clean":"2023-10-25T00:23:12.428569+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-25T00:23:12.428569+0000","last_undegraded":"2023-10-25T00:23:12.428569+0000","last_fullsized":"2023-10-25T00:23:12.428569+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3970","last_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_deep_scrub":"68'3970","last_deep_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.428494+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:07.899437+0000","scrub_duration":0.13106953599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21202,"num_write_kb":257085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"72'5559","reported_seq":5638,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517467+0000","last_change":"2023-10-24T23:53:11.827522+0000","last_active":"2023-10-25T00:22:33.517467+0000","last_peered":"2023-10-25T00:22:33.517467+0000","last_clean":"2023-10-25T00:22:33.517467+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-25T00:22:33.517467+0000","last_undegraded":"2023-10-25T00:22:33.517467+0000","last_fullsized":"2023-10-25T00:22:33.517467+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":31202,"num_write_kb":391572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"68'4936","reported_seq":5022,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.021130+0000","last_change":"2023-10-25T00:23:02.021272+0000","last_active":"2023-10-25T00:23:02.021130+0000","last_peered":"2023-10-25T00:23:02.021130+0000","last_clean":"2023-10-25T00:23:02.021130+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-25T00:23:02.021130+0000","last_undegraded":"2023-10-25T00:23:02.021130+0000","last_fullsized":"2023-10-25T00:23:02.021130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4936","last_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_deep_scrub":"68'4936","last_deep_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.021108+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:22.652380+0000","scrub_duration":0.17348896599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26134,"num_write_kb":309652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"72'3558","reported_seq":3634,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.517421+0000","last_change":"2023-10-24T23:53:11.807074+0000","last_active":"2023-10-25T00:22:33.517421+0000","last_peered":"2023-10-25T00:22:33.517421+0000","last_clean":"2023-10-25T00:22:33.517421+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-25T00:22:33.517421+0000","last_undegraded":"2023-10-25T00:22:33.517421+0000","last_fullsized":"2023-10-25T00:22:33.517421+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":34,"num_write":19715,"num_write_kb":243025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"68'5282","reported_seq":5368,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.290128+0000","last_change":"2023-10-25T00:23:14.290293+0000","last_active":"2023-10-25T00:23:14.290128+0000","last_peered":"2023-10-25T00:23:14.290128+0000","last_clean":"2023-10-25T00:23:14.290128+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-25T00:23:14.290128+0000","last_undegraded":"2023-10-25T00:23:14.290128+0000","last_fullsized":"2023-10-25T00:23:14.290128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5282","last_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_deep_scrub":"68'5282","last_deep_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.290099+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:44:38.139530+0000","scrub_duration":0.10140234200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27848,"num_write_kb":324561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"72'3914","reported_seq":3991,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.523611+0000","last_change":"2023-10-24T23:53:11.847385+0000","last_active":"2023-10-25T00:22:36.523611+0000","last_peered":"2023-10-25T00:22:36.523611+0000","last_clean":"2023-10-25T00:22:36.523611+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-25T00:22:36.523611+0000","last_undegraded":"2023-10-25T00:22:36.523611+0000","last_fullsized":"2023-10-25T00:22:36.523611+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":20938,"num_write_kb":288796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"68'6616","reported_seq":6706,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.985123+0000","last_change":"2023-10-25T00:22:57.985278+0000","last_active":"2023-10-25T00:22:57.985123+0000","last_peered":"2023-10-25T00:22:57.985123+0000","last_clean":"2023-10-25T00:22:57.985123+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-25T00:22:57.985123+0000","last_undegraded":"2023-10-25T00:22:57.985123+0000","last_fullsized":"2023-10-25T00:22:57.985123+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6616","last_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_deep_scrub":"68'6616","last_deep_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.985096+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:14:23.491729+0000","scrub_duration":0.050039131000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36221,"num_write_kb":423916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"72'3536","reported_seq":3620,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:32.908497+0000","last_change":"2023-10-25T00:23:32.908624+0000","last_active":"2023-10-25T00:23:32.908497+0000","last_peered":"2023-10-25T00:23:32.908497+0000","last_clean":"2023-10-25T00:23:32.908497+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-25T00:23:32.908497+0000","last_undegraded":"2023-10-25T00:23:32.908497+0000","last_fullsized":"2023-10-25T00:23:32.908497+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3536","last_scrub_stamp":"2023-10-25T00:23:32.908476+0000","last_deep_scrub":"72'3536","last_deep_scrub_stamp":"2023-10-25T00:23:32.908476+0000","last_clean_scrub_stamp":"2023-10-25T00:23:32.908476+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:33:20.337271+0000","scrub_duration":0.30729883400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19276,"num_write_kb":253506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"68'4980","reported_seq":5064,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:10.321019+0000","last_change":"2023-10-25T00:23:10.321202+0000","last_active":"2023-10-25T00:23:10.321019+0000","last_peered":"2023-10-25T00:23:10.321019+0000","last_clean":"2023-10-25T00:23:10.321019+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-25T00:23:10.321019+0000","last_undegraded":"2023-10-25T00:23:10.321019+0000","last_fullsized":"2023-10-25T00:23:10.321019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4980","last_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_deep_scrub":"68'4980","last_deep_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:10.320988+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:34.745429+0000","scrub_duration":0.13750111800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26525,"num_write_kb":306963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"72'4160","reported_seq":4246,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:24.541581+0000","last_change":"2023-10-25T00:23:24.541711+0000","last_active":"2023-10-25T00:23:24.541581+0000","last_peered":"2023-10-25T00:23:24.541581+0000","last_clean":"2023-10-25T00:23:24.541581+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-25T00:23:24.541581+0000","last_undegraded":"2023-10-25T00:23:24.541581+0000","last_fullsized":"2023-10-25T00:23:24.541581+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4160","last_scrub_stamp":"2023-10-25T00:23:24.541561+0000","last_deep_scrub":"72'4160","last_deep_scrub_stamp":"2023-10-25T00:23:24.541561+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.541561+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:52:13.060866+0000","scrub_duration":0.264226874,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":21819,"num_write_kb":288439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"68'4434","reported_seq":4520,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.208506+0000","last_change":"2023-10-25T00:22:47.255974+0000","last_active":"2023-10-25T00:22:49.208506+0000","last_peered":"2023-10-25T00:22:49.208506+0000","last_clean":"2023-10-25T00:22:49.208506+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-25T00:22:49.208506+0000","last_undegraded":"2023-10-25T00:22:49.208506+0000","last_fullsized":"2023-10-25T00:22:49.208506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4434","last_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_deep_scrub":"68'4434","last_deep_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.255646+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:35:52.723567+0000","scrub_duration":0.14955275700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23791,"num_write_kb":292031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"72'4195","reported_seq":4265,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.231604+0000","last_change":"2023-10-24T23:53:11.843272+0000","last_active":"2023-10-25T00:22:49.231604+0000","last_peered":"2023-10-25T00:22:49.231604+0000","last_clean":"2023-10-25T00:22:49.231604+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-25T00:22:49.231604+0000","last_undegraded":"2023-10-25T00:22:49.231604+0000","last_fullsized":"2023-10-25T00:22:49.231604+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23000,"num_write_kb":285019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"68'4243","reported_seq":4328,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.969067+0000","last_change":"2023-10-25T00:23:00.969067+0000","last_active":"2023-10-25T00:23:00.969067+0000","last_peered":"2023-10-25T00:23:00.969067+0000","last_clean":"2023-10-25T00:23:00.969067+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-25T00:23:00.969067+0000","last_undegraded":"2023-10-25T00:23:00.969067+0000","last_fullsized":"2023-10-25T00:23:00.969067+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4243","last_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_deep_scrub":"68'4243","last_deep_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.968692+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:45.877638+0000","scrub_duration":0.15459757700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22439,"num_write_kb":266094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"72'4017","reported_seq":4088,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.212449+0000","last_change":"2023-10-24T23:53:11.825405+0000","last_active":"2023-10-25T00:22:49.212449+0000","last_peered":"2023-10-25T00:22:49.212449+0000","last_clean":"2023-10-25T00:22:49.212449+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-25T00:22:49.212449+0000","last_undegraded":"2023-10-25T00:22:49.212449+0000","last_fullsized":"2023-10-25T00:22:49.212449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22665,"num_write_kb":291707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"68'4509","reported_seq":4597,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.227093+0000","last_change":"2023-10-25T00:22:44.022792+0000","last_active":"2023-10-25T00:22:49.227093+0000","last_peered":"2023-10-25T00:22:49.227093+0000","last_clean":"2023-10-25T00:22:49.227093+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-25T00:22:49.227093+0000","last_undegraded":"2023-10-25T00:22:49.227093+0000","last_fullsized":"2023-10-25T00:22:49.227093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4509","last_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_deep_scrub":"68'4509","last_deep_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.022573+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:54:30.651361+0000","scrub_duration":0.11145192700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23158,"num_write_kb":297883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"72'5235","reported_seq":5314,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534214+0000","last_change":"2023-10-24T23:53:11.811042+0000","last_active":"2023-10-25T00:22:35.534214+0000","last_peered":"2023-10-25T00:22:35.534214+0000","last_clean":"2023-10-25T00:22:35.534214+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-25T00:22:35.534214+0000","last_undegraded":"2023-10-25T00:22:35.534214+0000","last_fullsized":"2023-10-25T00:22:35.534214+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":15,"num_write":29235,"num_write_kb":367391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"68'4260","reported_seq":4349,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:56.486314+0000","last_change":"2023-10-25T00:22:56.486314+0000","last_active":"2023-10-25T00:22:56.486314+0000","last_peered":"2023-10-25T00:22:56.486314+0000","last_clean":"2023-10-25T00:22:56.486314+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-25T00:22:56.486314+0000","last_undegraded":"2023-10-25T00:22:56.486314+0000","last_fullsized":"2023-10-25T00:22:56.486314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4260","last_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_deep_scrub":"68'4260","last_deep_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.485966+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:51.396462+0000","scrub_duration":0.116368769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23357,"num_write_kb":265837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"72'4656","reported_seq":4737,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:35.534295+0000","last_change":"2023-10-24T23:53:11.815246+0000","last_active":"2023-10-25T00:22:35.534295+0000","last_peered":"2023-10-25T00:22:35.534295+0000","last_clean":"2023-10-25T00:22:35.534295+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-25T00:22:35.534295+0000","last_undegraded":"2023-10-25T00:22:35.534295+0000","last_fullsized":"2023-10-25T00:22:35.534295+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":26109,"num_write_kb":317781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"68'4636","reported_seq":4726,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:39.672588+0000","last_change":"2023-10-25T00:22:39.672588+0000","last_active":"2023-10-25T00:22:39.672588+0000","last_peered":"2023-10-25T00:22:39.672588+0000","last_clean":"2023-10-25T00:22:39.672588+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-25T00:22:39.672588+0000","last_undegraded":"2023-10-25T00:22:39.672588+0000","last_fullsized":"2023-10-25T00:22:39.672588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4636","last_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_deep_scrub":"68'4636","last_deep_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.672438+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:47:15.433547+0000","scrub_duration":0.124153799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25124,"num_write_kb":285314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"72'3117","reported_seq":3193,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.292881+0000","last_change":"2023-10-24T23:53:11.845469+0000","last_active":"2023-10-25T00:22:49.292881+0000","last_peered":"2023-10-25T00:22:49.292881+0000","last_clean":"2023-10-25T00:22:49.292881+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-25T00:22:49.292881+0000","last_undegraded":"2023-10-25T00:22:49.292881+0000","last_fullsized":"2023-10-25T00:22:49.292881+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17240,"num_write_kb":232937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"68'5520","reported_seq":5617,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:56.366454+0000","last_change":"2023-10-25T00:22:56.366641+0000","last_active":"2023-10-25T00:22:56.366454+0000","last_peered":"2023-10-25T00:22:56.366454+0000","last_clean":"2023-10-25T00:22:56.366454+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-25T00:22:56.366454+0000","last_undegraded":"2023-10-25T00:22:56.366454+0000","last_fullsized":"2023-10-25T00:22:56.366454+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5520","last_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_deep_scrub":"68'5520","last_deep_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.366419+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:47:59.889167+0000","scrub_duration":0.104594514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29665,"num_write_kb":368823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"72'3962","reported_seq":4047,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.105781+0000","last_change":"2023-10-25T00:23:34.105899+0000","last_active":"2023-10-25T00:23:34.105781+0000","last_peered":"2023-10-25T00:23:34.105781+0000","last_clean":"2023-10-25T00:23:34.105781+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-25T00:23:34.105781+0000","last_undegraded":"2023-10-25T00:23:34.105781+0000","last_fullsized":"2023-10-25T00:23:34.105781+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3962","last_scrub_stamp":"2023-10-25T00:23:34.105758+0000","last_deep_scrub":"72'3962","last_deep_scrub_stamp":"2023-10-25T00:23:34.105758+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.105758+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:35:57.960582+0000","scrub_duration":0.32704070499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21989,"num_write_kb":280762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"68'4908","reported_seq":5000,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.407703+0000","last_change":"2023-10-25T00:23:00.408006+0000","last_active":"2023-10-25T00:23:00.407703+0000","last_peered":"2023-10-25T00:23:00.407703+0000","last_clean":"2023-10-25T00:23:00.407703+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-25T00:23:00.407703+0000","last_undegraded":"2023-10-25T00:23:00.407703+0000","last_fullsized":"2023-10-25T00:23:00.407703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4908","last_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_deep_scrub":"68'4908","last_deep_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.407650+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:48:18.113931+0000","scrub_duration":0.051805808000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26829,"num_write_kb":320697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":320,"num_write":1482516,"num_write_kb":19060243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267718,"ondisk_log_size":267718,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":19490997,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1686,"num_read_kb":209824,"num_write":4665,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39043072,"data_stored":38981994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4559,"ondisk_log_size":4559,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":0,"num_write":1661441,"num_write_kb":19773626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307026,"ondisk_log_size":307026,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":20061533,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1689,"num_read_kb":267203,"num_write":4630,"num_write_kb":21638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40181760,"data_stored":40123066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4530,"ondisk_log_size":4530,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163,"ondisk_log_size":163,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575220,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919096,"kb_used_data":12336,"kb_used_omap":0,"kb_used_meta":906752,"kb_avail":93452744,"statfs":{"total":96636764160,"available":95695609856,"internally_reserved":0,"allocated":12632064,"data_stored":12153420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":928514048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.96199999999999997,"15min":0.94399999999999995},"min":{"1min":0.23699999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.2490000000000001,"5min":6.1879999999999997,"15min":13.929},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.93999999999999995,"15min":0.93899999999999995},"min":{"1min":0.29899999999999999,"5min":0.27600000000000002,"15min":0.20999999999999999},"max":{"1min":1.3149999999999999,"5min":6.8010000000000002,"15min":11.315},"last":0.98799999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":1.3200000000000001,"15min":1.1619999999999999},"min":{"1min":0.32300000000000001,"5min":0.23599999999999999,"15min":0.215},"max":{"1min":1.016,"5min":28.308,"15min":28.308},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.421,"15min":1.2330000000000001},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":1.232,"5min":28.346,"15min":28.346},"last":0.78200000000000003}]},{"osd":2,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":1.0780000000000001,"15min":0.98899999999999999},"min":{"1min":0.312,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.94599999999999995,"5min":5.8620000000000001,"15min":11.430999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.012,"15min":0.95299999999999996},"min":{"1min":0.57499999999999996,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.244,"5min":5.9870000000000001,"15min":7.9710000000000001},"last":1.147}]},{"osd":3,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":2.1699999999999999,"15min":1.3600000000000001},"min":{"1min":0.41299999999999998,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":1.135,"5min":93.664000000000001,"15min":93.664000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.91500000000000004,"5min":2.29,"15min":1.3799999999999999},"min":{"1min":0.48699999999999999,"5min":0.28799999999999998,"15min":0.25700000000000001},"max":{"1min":1.2569999999999999,"5min":93.073999999999998,"15min":93.073999999999998},"last":0.94799999999999995}]},{"osd":4,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":1.075,"15min":0.877},"min":{"1min":0.51500000000000001,"5min":0.26500000000000001,"15min":0.249},"max":{"1min":1.0129999999999999,"5min":7.6280000000000001,"15min":7.6280000000000001},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.085,"5min":1.716,"15min":1.1479999999999999},"min":{"1min":0.68000000000000005,"5min":0.33300000000000002,"15min":0.27200000000000002},"max":{"1min":2.6309999999999998,"5min":27.936,"15min":27.936},"last":1.089}]},{"osd":5,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.47,"15min":1.042},"min":{"1min":0.29599999999999999,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":1.087,"5min":21.126999999999999,"15min":21.126999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.1140000000000001,"15min":0.97199999999999998},"min":{"1min":0.65700000000000003,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":3.8069999999999999,"5min":11.49,"15min":11.49},"last":0.91500000000000004}]},{"osd":6,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.556,"15min":1.107},"min":{"1min":0.55200000000000005,"5min":0.311,"15min":0.23599999999999999},"max":{"1min":3.8849999999999998,"5min":8.2959999999999994,"15min":8.2959999999999994},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.827,"15min":1.2150000000000001},"min":{"1min":0.58299999999999996,"5min":0.45500000000000002,"15min":0.35499999999999998},"max":{"1min":4.3630000000000004,"5min":20.651,"15min":20.651},"last":1.1930000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575220,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859020,"kb_used_data":5060,"kb_used_omap":0,"kb_used_meta":853952,"kb_avail":93512820,"statfs":{"total":96636764160,"available":95757127680,"internally_reserved":0,"allocated":5181440,"data_stored":4739934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":874446848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.2869999999999999,"15min":0.95199999999999996},"min":{"1min":0.315,"5min":0.185,"15min":0.185},"max":{"1min":5.3490000000000002,"5min":15.974,"15min":15.974},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.331,"15min":0.97399999999999998},"min":{"1min":0.372,"5min":0.218,"15min":0.218},"max":{"1min":1.722,"5min":8.4789999999999992,"15min":8.4789999999999992},"last":0.84599999999999997}]},{"osd":1,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.92500000000000004,"15min":0.88300000000000001},"min":{"1min":0.38900000000000001,"5min":0.221,"15min":0.221},"max":{"1min":1.3919999999999999,"5min":5.556,"15min":7.6360000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.175,"15min":0.98699999999999999},"min":{"1min":0.42099999999999999,"5min":0.254,"15min":0.249},"max":{"1min":1.411,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.91400000000000003}]},{"osd":2,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.5189999999999999,"15min":1.0640000000000001},"min":{"1min":0.318,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.7450000000000001,"5min":12.615,"15min":12.615},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.4610000000000001,"15min":1.109},"min":{"1min":0.38100000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.7669999999999999,"5min":12.667999999999999,"15min":12.667999999999999},"last":1.1870000000000001}]},{"osd":3,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.4770000000000001,"15min":1.2490000000000001},"min":{"1min":0.34300000000000003,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.3769999999999998,"5min":18.106999999999999,"15min":38.649999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.107,"5min":1.534,"15min":1.3320000000000001},"min":{"1min":0.39800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.3979999999999997,"5min":18.143000000000001,"15min":38.683},"last":0.872}]},{"osd":4,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.7410000000000001,"15min":1.125},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.23799999999999999},"max":{"1min":8.9930000000000003,"5min":12.682,"15min":12.682},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.49,"15min":1.0900000000000001},"min":{"1min":0.36599999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.3179999999999996,"5min":10.686,"15min":10.686},"last":0.90200000000000002}]},{"osd":5,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.1559999999999999,"15min":0.96199999999999997},"min":{"1min":0.44500000000000001,"5min":0.28399999999999997,"15min":0.24299999999999999},"max":{"1min":2.1339999999999999,"5min":8.0039999999999996,"15min":8.0039999999999996},"last":0.622},{"interface":"front","average":{"1min":1.298,"5min":1.3660000000000001,"15min":1.069},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.30299999999999999},"max":{"1min":4.0679999999999996,"5min":11.766999999999999,"15min":11.766999999999999},"last":1.032}]},{"osd":7,"last update":"Wed Oct 25 00:22:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2969999999999999,"15min":1.0369999999999999},"min":{"1min":0.40999999999999998,"5min":0.251,"15min":0.251},"max":{"1min":5.4219999999999997,"5min":8.2720000000000002,"15min":12.926},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.341,"5min":1.3100000000000001,"15min":1.05},"min":{"1min":0.504,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.4420000000000002,"5min":6.1100000000000003,"15min":7.6619999999999999},"last":1.073}]}]},{"osd":5,"up_from":13,"seq":55834575220,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":958872,"kb_used_data":9168,"kb_used_omap":0,"kb_used_meta":949696,"kb_avail":93412968,"statfs":{"total":96636764160,"available":95654879232,"internally_reserved":0,"allocated":9388032,"data_stored":8934420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":972488704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:48 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":1.2869999999999999,"15min":1.038},"min":{"1min":0.371,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.5569999999999999,"5min":14.837999999999999,"15min":14.837999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.86299999999999999,"15min":0.96799999999999997},"min":{"1min":0.27000000000000002,"5min":0.20699999999999999,"15min":0.20100000000000001},"max":{"1min":3.5779999999999998,"5min":5.8289999999999997,"15min":7.423},"last":0.94599999999999995}]},{"osd":1,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.415,"15min":1.085},"min":{"1min":0.312,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":2.0419999999999998,"5min":12.141,"15min":12.141},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.4199999999999999,"15min":1.0860000000000001},"min":{"1min":0.314,"5min":0.26400000000000001,"15min":0.252},"max":{"1min":3.714,"5min":24.648,"15min":24.648},"last":0.98499999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.46,"15min":1.2390000000000001},"min":{"1min":0.376,"5min":0.25,"15min":0.25},"max":{"1min":1.925,"5min":20.760999999999999,"15min":20.760999999999999},"last":1.008},{"interface":"front","average":{"1min":0.76700000000000002,"5min":1.274,"15min":1.0700000000000001},"min":{"1min":0.39800000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.1779999999999999,"5min":20.552,"15min":20.552},"last":0.82599999999999996}]},{"osd":3,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.429,"15min":1.2210000000000001},"min":{"1min":0.33300000000000002,"5min":0.30299999999999999,"15min":0.28000000000000003},"max":{"1min":3.6139999999999999,"5min":12.013,"15min":12.013},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.359,"15min":1.246},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.5779999999999998,"5min":12.504,"15min":12.805},"last":0.85799999999999998}]},{"osd":4,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":1.708,"15min":1.202},"min":{"1min":0.47099999999999997,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":2.3290000000000002,"5min":12.728,"15min":12.728},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.532,"15min":1.1419999999999999},"min":{"1min":0.39200000000000002,"5min":0.27600000000000002,"15min":0.27200000000000002},"max":{"1min":3.6699999999999999,"5min":10.943,"15min":10.943},"last":0.90300000000000002}]},{"osd":6,"last update":"Wed Oct 25 00:23:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.385,"15min":1.0609999999999999},"min":{"1min":0.44400000000000001,"5min":0.34399999999999997,"15min":0.26400000000000001},"max":{"1min":3.6930000000000001,"5min":17.936,"15min":17.936},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.869,"15min":1.212},"min":{"1min":0.31,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.0229999999999999,"5min":14.766999999999999,"15min":14.766999999999999},"last":0.752}]},{"osd":7,"last update":"Wed Oct 25 00:22:35 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.75,"15min":1.2470000000000001},"min":{"1min":0.46500000000000002,"5min":0.33200000000000002,"15min":0.29499999999999998},"max":{"1min":1.2869999999999999,"5min":28.178999999999998,"15min":28.178999999999998},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.0740000000000001,"5min":2.133,"15min":1.357},"min":{"1min":0.59699999999999998,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":3.6429999999999998,"5min":28.222999999999999,"15min":28.222999999999999},"last":1.0629999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575219,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946308,"kb_used_data":19132,"kb_used_omap":0,"kb_used_meta":927168,"kb_avail":93425532,"statfs":{"total":96636764160,"available":95667744768,"internally_reserved":0,"allocated":19591168,"data_stored":19124450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":949420032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.3320000000000001,"15min":0.98099999999999998},"min":{"1min":0.317,"5min":0.19700000000000001,"15min":0.187},"max":{"1min":11.629,"5min":20.922999999999998,"15min":20.922999999999998},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.101,"5min":1.157,"15min":0.98199999999999998},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.217},"max":{"1min":7.0830000000000002,"5min":20.532,"15min":20.532},"last":0.68600000000000005}]},{"osd":1,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":1.2410000000000001,"15min":0.97699999999999998},"min":{"1min":0.33800000000000002,"5min":0.22,"15min":0.22},"max":{"1min":2.6110000000000002,"5min":20.914000000000001,"15min":20.914000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.3779999999999999,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":6.1890000000000001,"5min":23.265999999999998,"15min":23.265999999999998},"last":0.66300000000000003}]},{"osd":2,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.528,"15min":1.1080000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.1550000000000002,"5min":20.774000000000001,"15min":20.774000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.802,"15min":1.252},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":7.2750000000000004,"5min":20.817,"15min":20.817},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.623,"15min":1.1699999999999999},"min":{"1min":0.44800000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":7.3620000000000001,"5min":24.613,"15min":24.613},"last":0.87},{"interface":"front","average":{"1min":1.28,"5min":1.3400000000000001,"15min":1.145},"min":{"1min":0.29299999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":7.2009999999999996,"5min":24.071000000000002,"15min":24.071000000000002},"last":0.82499999999999996}]},{"osd":5,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.1679999999999999,"15min":0.93300000000000005},"min":{"1min":0.5,"5min":0.28599999999999998,"15min":0.23999999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.623},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.079,"15min":0.95099999999999996},"min":{"1min":0.57899999999999996,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.7549999999999999,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.83999999999999997}]},{"osd":6,"last update":"Wed Oct 25 00:22:40 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.367,"15min":1.0860000000000001},"min":{"1min":0.627,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":2.794,"5min":6.5220000000000002,"15min":6.5220000000000002},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.1040000000000001,"15min":0.97199999999999998},"min":{"1min":0.38700000000000001,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":7.2389999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"last":0.96399999999999997}]},{"osd":7,"last update":"Wed Oct 25 00:22:52 2023","interfaces":[{"interface":"back","average":{"1min":1.597,"5min":1.3520000000000001,"15min":1.087},"min":{"1min":0.56499999999999995,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":7.4020000000000001,"5min":7.9850000000000003,"15min":7.9850000000000003},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.7290000000000001,"5min":1.5649999999999999,"15min":1.1579999999999999},"min":{"1min":0.58899999999999997,"5min":0.309,"15min":0.309},"max":{"1min":7.444,"5min":7.444,"15min":7.444},"last":0.99199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575220,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160748,"kb_used_data":9252,"kb_used_omap":0,"kb_used_meta":1151488,"kb_avail":93211092,"statfs":{"total":96636764160,"available":95448158208,"internally_reserved":0,"allocated":9474048,"data_stored":9018997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1179123712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.67500000000000004,"15min":0.78300000000000003},"min":{"1min":0.18099999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.72599999999999998,"5min":12.215999999999999,"15min":13.327999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.55700000000000005,"15min":0.72899999999999998},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.95399999999999996,"5min":2.2770000000000001,"15min":5.4580000000000002},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.86699999999999999,"15min":0.85599999999999998},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19900000000000001},"max":{"1min":0.96799999999999997,"5min":11.65,"15min":13.831},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.71099999999999997,"15min":0.76100000000000001},"min":{"1min":0.20999999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.0940000000000001,"5min":4.2130000000000001,"15min":4.4500000000000002},"last":0.86399999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.82199999999999995,"15min":0.87},"min":{"1min":0.24099999999999999,"5min":0.223,"15min":0.221},"max":{"1min":1.0129999999999999,"5min":8.2520000000000007,"15min":8.2520000000000007},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.92000000000000004,"15min":0.90300000000000002},"min":{"1min":0.26700000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0349999999999999,"5min":6.6840000000000002,"15min":6.6840000000000002},"last":0.751}]},{"osd":4,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.7,"15min":1.204},"min":{"1min":0.57599999999999996,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.208,"5min":39.070999999999998,"15min":39.070999999999998},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.82999999999999996,"5min":1.8899999999999999,"15min":1.248},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.25700000000000001},"max":{"1min":1.1080000000000001,"5min":55.104999999999997,"15min":55.104999999999997},"last":0.97999999999999998}]},{"osd":5,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.556,"15min":1.1160000000000001},"min":{"1min":0.33300000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1759999999999999,"5min":31.481999999999999,"15min":31.481999999999999},"last":1.161},{"interface":"front","average":{"1min":0.997,"5min":1.617,"15min":1.2010000000000001},"min":{"1min":0.40699999999999997,"5min":0.35299999999999998,"15min":0.32500000000000001},"max":{"1min":1.4470000000000001,"5min":31.512,"15min":31.512},"last":1.3300000000000001}]},{"osd":6,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.6180000000000001,"15min":1.2370000000000001},"min":{"1min":0.33600000000000002,"5min":0.311,"15min":0.27800000000000002},"max":{"1min":15.824,"5min":25.809999999999999,"15min":25.809999999999999},"last":1.234},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.363,"15min":1.171},"min":{"1min":0.41399999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1910000000000001,"5min":24.036999999999999,"15min":24.036999999999999},"last":1.115}]},{"osd":7,"last update":"Wed Oct 25 00:22:49 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.5069999999999999,"15min":1.2250000000000001},"min":{"1min":0.35099999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3200000000000001,"5min":27.581,"15min":27.581},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.413,"15min":1.2430000000000001},"min":{"1min":0.503,"5min":0.374,"15min":0.374},"max":{"1min":1.3819999999999999,"5min":24.332000000000001,"15min":24.332000000000001},"last":1.2889999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575220,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946060,"kb_used_data":5060,"kb_used_omap":0,"kb_used_meta":940992,"kb_avail":93425780,"statfs":{"total":96636764160,"available":95667998720,"internally_reserved":0,"allocated":5181440,"data_stored":4739422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":963575808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.96699999999999997,"15min":0.97999999999999998},"min":{"1min":0.185,"5min":0.16,"15min":0.16},"max":{"1min":2.3010000000000002,"5min":5.9710000000000001,"15min":29.344000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.83699999999999997,"15min":0.83499999999999996},"min":{"1min":0.222,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.7000000000000002,"5min":6.4290000000000003,"15min":10.755000000000001},"last":0.69799999999999995}]},{"osd":1,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.98899999999999999,"15min":0.92900000000000005},"min":{"1min":0.24199999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":3.0409999999999999,"5min":17.192,"15min":18.617999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.86699999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.649,"5min":5.8890000000000002,"15min":15.064},"last":0.58599999999999997}]},{"osd":3,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.014,"15min":0.89700000000000002},"min":{"1min":0.20699999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.9220000000000002,"5min":13.034000000000001,"15min":13.034000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.1930000000000001,"15min":0.99099999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.9929999999999999,"5min":9.9890000000000008,"15min":18.547999999999998},"last":0.96199999999999997}]},{"osd":4,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.4670000000000001,"15min":1.1850000000000001},"min":{"1min":0.26900000000000002,"5min":0.252,"15min":0.252},"max":{"1min":5.1070000000000002,"5min":22.997,"15min":33.378999999999998},"last":1.377},{"interface":"front","average":{"1min":1.194,"5min":1.5189999999999999,"15min":1.278},"min":{"1min":0.309,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.2290000000000001,"5min":23.030999999999999,"15min":33.558999999999997},"last":1.464}]},{"osd":5,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.887,"15min":1.464},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.29099999999999998},"max":{"1min":3.1400000000000001,"5min":25.183,"15min":34.545999999999999},"last":1.056},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.2450000000000001,"15min":1.1679999999999999},"min":{"1min":0.33600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.4870000000000001,"5min":11.685,"15min":36.104999999999997},"last":1.2749999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.1339999999999999,"15min":1.103},"min":{"1min":0.501,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.482,"5min":9.7149999999999999,"15min":33.661999999999999},"last":1.1910000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.1930000000000001,"15min":1.1910000000000001},"min":{"1min":0.53100000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.464,"5min":9.6799999999999997,"15min":34.043999999999997},"last":1.417}]},{"osd":7,"last update":"Wed Oct 25 00:22:36 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3009999999999999,"15min":1.226},"min":{"1min":0.41499999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":4.8369999999999997,"5min":11.503,"15min":33.307000000000002},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.341,"15min":1.2889999999999999},"min":{"1min":0.44900000000000001,"5min":0.371,"15min":0.371},"max":{"1min":4.8620000000000001,"5min":11.468999999999999,"15min":33.344999999999999},"last":1.508}]}]},{"osd":1,"up_from":13,"seq":55834575221,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054144,"kb_used_data":4088,"kb_used_omap":0,"kb_used_meta":1050048,"kb_avail":93317696,"statfs":{"total":96636764160,"available":95557320704,"internally_reserved":0,"allocated":4186112,"data_stored":3743339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1075249152},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.98099999999999998,"15min":0.81299999999999994},"min":{"1min":0.249,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":0.85699999999999998,"5min":21.334,"15min":21.334},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.96899999999999997,"15min":0.84199999999999997},"min":{"1min":0.38800000000000001,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":1.0449999999999999,"5min":21.276,"15min":21.276},"last":0.26200000000000001}]},{"osd":2,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.93400000000000005,"15min":0.80000000000000004},"min":{"1min":0.34999999999999998,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.92900000000000005,"5min":11.420999999999999,"15min":11.420999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.878,"15min":0.83099999999999996},"min":{"1min":0.27100000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.002,"5min":11.648,"15min":11.648},"last":0.53500000000000003}]},{"osd":3,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":1.022,"15min":0.91100000000000003},"min":{"1min":0.318,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":1.1140000000000001,"5min":9.0449999999999999,"15min":14.346},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.84899999999999998,"5min":1.1040000000000001,"15min":1.01},"min":{"1min":0.35099999999999998,"5min":0.26300000000000001,"15min":0.248},"max":{"1min":1.1859999999999999,"5min":9.0800000000000001,"15min":14.379},"last":0.73999999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.2569999999999999,"15min":1.0600000000000001},"min":{"1min":0.50600000000000001,"5min":0.28000000000000003,"15min":0.27200000000000002},"max":{"1min":1.353,"5min":10.582000000000001,"15min":10.582000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.2210000000000001,"15min":1.0449999999999999},"min":{"1min":0.39600000000000002,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":1.272,"5min":11.173,"15min":11.173},"last":0.82899999999999996}]},{"osd":5,"last update":"Wed Oct 25 00:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.0860000000000001,"15min":1.042},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.248,"5min":10.446,"15min":11.577999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.93700000000000006,"15min":0.96799999999999997},"min":{"1min":0.52100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.2869999999999999,"5min":5.3209999999999997,"15min":10.018000000000001},"last":0.69699999999999995}]},{"osd":6,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.0680000000000001,"15min":1.0229999999999999},"min":{"1min":0.55800000000000005,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.357,"5min":9.2989999999999995,"15min":11.845000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.141,"15min":1.1140000000000001},"min":{"1min":0.62,"5min":0.379,"15min":0.33800000000000002},"max":{"1min":1.4870000000000001,"5min":9.3350000000000009,"15min":11.919},"last":0.77100000000000002}]},{"osd":7,"last update":"Wed Oct 25 00:22:54 2023","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.387,"15min":1.1200000000000001},"min":{"1min":0.57499999999999996,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":1.407,"5min":9.407,"15min":11.881},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.22,"15min":1.1359999999999999},"min":{"1min":0.54700000000000004,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3080000000000001,"5min":10.478,"15min":10.478},"last":0.56899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834575221,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":966376,"kb_used_data":22112,"kb_used_omap":0,"kb_used_meta":944256,"kb_avail":93405464,"statfs":{"total":96636764160,"available":95647195136,"internally_reserved":0,"allocated":22642688,"data_stored":22192198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":966918144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.1719999999999999,"15min":0.99099999999999999},"min":{"1min":0.23000000000000001,"5min":0.18099999999999999,"15min":0.16900000000000001},"max":{"1min":1.6779999999999999,"5min":30.065999999999999,"15min":30.065999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.94799999999999995,"15min":0.99199999999999999},"min":{"1min":0.247,"5min":0.184,"15min":0.182},"max":{"1min":0.98999999999999999,"5min":8.8529999999999998,"15min":19.757999999999999},"last":0.503}]},{"osd":2,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.93500000000000005,"15min":0.78600000000000003},"min":{"1min":0.42199999999999999,"5min":0.22900000000000001,"15min":0.20399999999999999},"max":{"1min":1.806,"5min":15.324999999999999,"15min":15.324999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.88200000000000001,"5min":1.169,"15min":0.95499999999999996},"min":{"1min":0.42899999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.8440000000000001,"5min":30.283999999999999,"15min":30.283999999999999},"last":0.88800000000000001}]},{"osd":3,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":1.4490000000000001,"15min":1.1020000000000001},"min":{"1min":0.40000000000000002,"5min":0.191,"15min":0.191},"max":{"1min":2.3420000000000001,"5min":22.966999999999999,"15min":22.966999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.2529999999999999,"15min":1.0609999999999999},"min":{"1min":0.32800000000000001,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":1.915,"5min":19.460999999999999,"15min":19.460999999999999},"last":0.99199999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":1.056,"15min":1.121},"min":{"1min":0.35299999999999998,"5min":0.318,"15min":0.26600000000000001},"max":{"1min":1.157,"5min":4.4710000000000001,"15min":8.3670000000000009},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.3799999999999999,"15min":1.222},"min":{"1min":0.59799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":5.3719999999999999,"5min":13.023999999999999,"15min":13.023999999999999},"last":1.1160000000000001}]},{"osd":5,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.103,"15min":1.0169999999999999},"min":{"1min":0.60999999999999999,"5min":0.30399999999999999,"15min":0.29099999999999998},"max":{"1min":2.391,"5min":6.0149999999999997,"15min":10.683999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1799999999999999,"15min":0.98799999999999999},"min":{"1min":0.52800000000000002,"5min":0.33700000000000002,"15min":0.29899999999999999},"max":{"1min":4.8860000000000001,"5min":15.436999999999999,"15min":15.436999999999999},"last":1.048}]},{"osd":6,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":1.266,"15min":1.125},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.28100000000000003},"max":{"1min":1.3069999999999999,"5min":8.7260000000000009,"15min":8.7260000000000009},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.3700000000000001,"15min":1.2829999999999999},"min":{"1min":0.56299999999999994,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":7.8970000000000002,"5min":15.375,"15min":15.375},"last":1.0960000000000001}]},{"osd":7,"last update":"Wed Oct 25 00:22:45 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.3380000000000001,"15min":1.151},"min":{"1min":0.64800000000000002,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":5.4269999999999996,"5min":6.0640000000000001,"15min":10.81},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.206,"15min":1.256},"min":{"1min":0.45600000000000002,"5min":0.313,"15min":0.313},"max":{"1min":1.8660000000000001,"5min":9.3699999999999992,"15min":11.417},"last":0.94299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12673024,"data_stored":12668140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3198976,"data_stored":3198201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11689984,"data_stored":11672183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4195192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":8388642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4279555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6918144,"data_stored":6907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6836224,"data_stored":6822812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-25T00:23:39.122 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.123 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.123 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.123 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.124 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.124 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.124 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.125 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.125 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.126 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.126 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.126 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.127 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.127 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.127 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.128 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.128 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.129 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.129 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.129 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.130 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.130 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.130 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.131 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.131 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.131 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.132 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.132 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.132 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.133 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.133 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.133 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.134 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.134 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.134 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.135 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.135 INFO:tasks.ceph:pgid 6.21 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.135 INFO:tasks.ceph:pgid 6.26 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.136 INFO:tasks.ceph:pgid 6.24 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.136 INFO:tasks.ceph:pgid 6.2a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.136 INFO:tasks.ceph:pgid 6.2b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.137 INFO:tasks.ceph:pgid 4.29 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.137 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.137 INFO:tasks.ceph:pgid 6.2f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.138 INFO:tasks.ceph:pgid 6.2d last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.138 INFO:tasks.ceph:pgid 6.32 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.138 INFO:tasks.ceph:pgid 6.33 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.139 INFO:tasks.ceph:pgid 6.30 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.139 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.139 INFO:tasks.ceph:pgid 6.31 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.140 INFO:tasks.ceph:pgid 4.33 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.140 INFO:tasks.ceph:pgid 6.36 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.140 INFO:tasks.ceph:pgid 6.37 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.140 INFO:tasks.ceph:pgid 6.34 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.141 INFO:tasks.ceph:pgid 6.35 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.141 INFO:tasks.ceph:pgid 6.38 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.141 INFO:tasks.ceph:pgid 6.39 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.142 INFO:tasks.ceph:pgid 6.3e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.142 INFO:tasks.ceph:pgid 6.3f last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.142 INFO:tasks.ceph:pgid 6.3c last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:39.143 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-25T00:23:59.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-25T00:23:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.274+0000 7fe52e64a640 1 Processor -- start 2023-10-25T00:23:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.274+0000 7fe52e64a640 1 -- start start 2023-10-25T00:23:59.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52e64a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe528152040 0x7fe5280c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:59.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52e64a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52806a4b0 0x7fe5280ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:59.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52e64a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe528155680 con 0x7fe5280c8100 2023-10-25T00:23:59.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52e64a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe528154a70 con 0x7fe528152040 2023-10-25T00:23:59.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52e64a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe528154d50 con 0x7fe52806a4b0 2023-10-25T00:23:59.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe5277fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe5280c8100 0x7fe5280abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:51758/0 (socket says 172.21.15.107:51758) 2023-10-25T00:23:59.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe5277fe640 1 -- 172.21.15.107:0/2735286377 learned_addr learned my addr 172.21.15.107:0/2735286377 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:23:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe527fff640 1 --2- 172.21.15.107:0/2735286377 >> v2:172.21.15.110:3300/0 conn(0x7fe528152040 0x7fe5280c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52cbe7640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52806a4b0 0x7fe5280ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267670659 0 0) 0x7fe528155680 con 0x7fe5280c8100 2023-10-25T00:23:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe510003530 con 0x7fe5280c8100 2023-10-25T00:23:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 571846465 0 0) 0x7fe510003530 con 0x7fe5280c8100 2023-10-25T00:23:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52806a4b0 msgr2=0x7fe5280ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:23:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52806a4b0 0x7fe5280ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 >> v2:172.21.15.110:3300/0 conn(0x7fe528152040 msgr2=0x7fe5280c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:23:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 --2- 172.21.15.107:0/2735286377 >> v2:172.21.15.110:3300/0 conn(0x7fe528152040 0x7fe5280c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528155030 con 0x7fe5280c8100 2023-10-25T00:23:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fe51c001680 con 0x7fe5280c8100 2023-10-25T00:23:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 >> v1:172.21.15.107:6789/0 conn(0x7fe5280c8100 legacy=0x7fe5280abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:59.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52806a4b0 0x7fe510003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:59.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe510003f00 0x7fe5100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:59.288 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510004d10 0x7fe510005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:59.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528154a70 con 0x7fe510004d10 2023-10-25T00:23:59.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528154d50 con 0x7fe510003f00 2023-10-25T00:23:59.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51000b9e0 con 0x7fe52806a4b0 2023-10-25T00:23:59.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52cbe7640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510004d10 0x7fe510005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:59.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe5277fe640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe510003f00 0x7fe5100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:59.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe527fff640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52806a4b0 0x7fe510003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:59.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52cbe7640 1 -- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52806a4b0 msgr2=0x7fe510003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:23:59.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52cbe7640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52806a4b0 0x7fe510003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:59.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52cbe7640 1 -- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe510003f00 msgr2=0x7fe5100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:23:59.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52cbe7640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe510003f00 0x7fe5100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:59.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe52cbe7640 1 -- 172.21.15.107:0/2735286377 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe510003530 con 0x7fe510004d10 2023-10-25T00:23:59.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.278+0000 7fe5277fe640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe510003f00 0x7fe5100047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:23:59.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52cbe7640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510004d10 0x7fe510005190 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fe52000e170 tx=0x7fe52001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a4459faf1d5b2e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:59.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5200a5c50 con 0x7fe510004d10 2023-10-25T00:23:59.297 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52000f040 con 0x7fe510004d10 2023-10-25T00:23:59.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe526ffd640 1 -- 172.21.15.107:0/2735286377 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5200ad440 con 0x7fe510004d10 2023-10-25T00:23:59.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 -- 172.21.15.107:0/2735286377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510004d10 msgr2=0x7fe510005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:59.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 --2- 172.21.15.107:0/2735286377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe510004d10 0x7fe510005190 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7fe52000e170 tx=0x7fe52001cc40 comp rx=0 tx=0).stop 2023-10-25T00:23:59.298 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52cbe7640 1 -- 172.21.15.107:0/2735286377 reap_dead start 2023-10-25T00:23:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 -- 172.21.15.107:0/2735286377 shutdown_connections 2023-10-25T00:23:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 -- 172.21.15.107:0/2735286377 >> 172.21.15.107:0/2735286377 conn(0x7fe5280ca200 msgr2=0x7fe5280ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:23:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 -- 172.21.15.107:0/2735286377 shutdown_connections 2023-10-25T00:23:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 -- 172.21.15.107:0/2735286377 wait complete. 2023-10-25T00:23:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 Processor -- start 2023-10-25T00:23:59.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 -- start start 2023-10-25T00:23:59.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5280ac420 0x7fe5280ac800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:59.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52814db30 0x7fe5281ed8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:59.300 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52814fc50 0x7fe5281500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:59.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52814f8a0 con 0x7fe52814db30 2023-10-25T00:23:59.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5281ed410 con 0x7fe5280ac420 2023-10-25T00:23:59.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52814e1c0 con 0x7fe52814fc50 2023-10-25T00:23:59.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe5277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52814db30 0x7fe5281ed8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:59.301 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe5277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52814db30 0x7fe5281ed8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40962/0 (socket says 172.21.15.107:40962) 2023-10-25T00:23:59.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe5277fe640 1 -- 172.21.15.107:0/3458083502 learned_addr learned my addr 172.21.15.107:0/3458083502 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:23:59.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe527fff640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5280ac420 0x7fe5280ac800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:59.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52cbe7640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52814fc50 0x7fe5281500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:59.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe5277fe640 1 -- 172.21.15.107:0/3458083502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52814fc50 msgr2=0x7fe5281500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:59.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe5277fe640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52814fc50 0x7fe5281500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:59.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe5277fe640 1 -- 172.21.15.107:0/3458083502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5280ac420 msgr2=0x7fe5280ac800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:59.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe5277fe640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5280ac420 0x7fe5280ac800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:59.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe5277fe640 1 -- 172.21.15.107:0/3458083502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5280abfd0 con 0x7fe52814db30 2023-10-25T00:23:59.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe5277fe640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52814db30 0x7fe5281ed8a0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fe51c004850 tx=0x7fe51c011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:59.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe524ff9640 1 -- 172.21.15.107:0/3458083502 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c00fb30 con 0x7fe52814db30 2023-10-25T00:23:59.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe524ff9640 1 -- 172.21.15.107:0/3458083502 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c00fc90 con 0x7fe52814db30 2023-10-25T00:23:59.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe52e64a640 1 -- 172.21.15.107:0/3458083502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5280ab860 con 0x7fe52814db30 2023-10-25T00:23:59.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.282+0000 7fe524ff9640 1 -- 172.21.15.107:0/3458083502 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c015410 con 0x7fe52814db30 2023-10-25T00:23:59.304 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.286+0000 7fe52e64a640 1 -- 172.21.15.107:0/3458083502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52814e460 con 0x7fe52814db30 2023-10-25T00:23:59.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.286+0000 7fe52e64a640 1 -- 172.21.15.107:0/3458083502 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5280746e0 con 0x7fe52814db30 2023-10-25T00:23:59.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.286+0000 7fe524ff9640 1 -- 172.21.15.107:0/3458083502 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe51c015570 con 0x7fe52814db30 2023-10-25T00:23:59.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.286+0000 7fe524ff9640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe514065e60 0x7fe514068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:23:59.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.286+0000 7fe524ff9640 1 -- 172.21.15.107:0/3458083502 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7fe51c087cd0 con 0x7fe52814db30 2023-10-25T00:23:59.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.286+0000 7fe527fff640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe514065e60 0x7fe514068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:23:59.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.286+0000 7fe527fff640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe514065e60 0x7fe514068320 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe528150b90 tx=0x7fe518008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:23:59.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.290+0000 7fe524ff9640 1 -- 172.21.15.107:0/3458083502 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe51c054170 con 0x7fe52814db30 2023-10-25T00:23:59.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.442+0000 7fe52e64a640 1 -- 172.21.15.107:0/3458083502 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe528155310 con 0x7fe514065e60 2023-10-25T00:23:59.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.482+0000 7fe524ff9640 1 -- 172.21.15.107:0/3458083502 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+555047 (secure 0 0 0) 0x7fe518014af0 con 0x7fe514065e60 2023-10-25T00:23:59.488 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:23:59.496 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-25T00:23:59.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 -- 172.21.15.107:0/3458083502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe514065e60 msgr2=0x7fe514068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:59.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe514065e60 0x7fe514068320 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fe528150b90 tx=0x7fe518008040 comp rx=0 tx=0).stop 2023-10-25T00:23:59.497 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 -- 172.21.15.107:0/3458083502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52814db30 msgr2=0x7fe5281ed8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:23:59.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52814db30 0x7fe5281ed8a0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fe51c004850 tx=0x7fe51c011d10 comp rx=0 tx=0).stop 2023-10-25T00:23:59.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 -- 172.21.15.107:0/3458083502 shutdown_connections 2023-10-25T00:23:59.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe514065e60 0x7fe514068320 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:59.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe52814fc50 0x7fe5281500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:59.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52814db30 0x7fe5281ed8a0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:59.498 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 --2- 172.21.15.107:0/3458083502 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe5280ac420 0x7fe5280ac800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:23:59.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 -- 172.21.15.107:0/3458083502 >> 172.21.15.107:0/3458083502 conn(0x7fe5280ca200 msgr2=0x7fe5280ad100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:23:59.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 -- 172.21.15.107:0/3458083502 shutdown_connections 2023-10-25T00:23:59.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:23:59.486+0000 7fe52e64a640 1 -- 172.21.15.107:0/3458083502 wait complete. 2023-10-25T00:23:59.508 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1000,"stamp":"2023-10-25T00:23:58.048705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40142898,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3975,"num_read_kb":477474,"num_write":3153461,"num_write_kb":38880074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":583999,"ondisk_log_size":583999,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7810624,"kb_used_data":86208,"kb_used_omap":0,"kb_used_meta":7724352,"kb_avail":747164096,"statfs":{"total":773094113280,"available":765096034304,"internally_reserved":0,"allocated":88276992,"data_stored":84646180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7909736448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005717"},"pg_stats":[{"pgid":"6.1a","version":"72'3995","reported_seq":4069,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.233446+0000","last_change":"2023-10-24T23:53:11.855077+0000","last_active":"2023-10-25T00:22:49.233446+0000","last_peered":"2023-10-25T00:22:49.233446+0000","last_clean":"2023-10-25T00:22:49.233446+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-25T00:22:49.233446+0000","last_undegraded":"2023-10-25T00:22:49.233446+0000","last_fullsized":"2023-10-25T00:22:49.233446+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21480,"num_write_kb":282788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"72'3","reported_seq":84,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:29.637360+0000","last_change":"2023-10-25T00:23:29.637684+0000","last_active":"2023-10-25T00:23:29.637360+0000","last_peered":"2023-10-25T00:23:29.637360+0000","last_clean":"2023-10-25T00:23:29.637360+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-25T00:23:29.637360+0000","last_undegraded":"2023-10-25T00:23:29.637360+0000","last_fullsized":"2023-10-25T00:23:29.637360+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:29.637307+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:29.637307+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.637307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:55.311760+0000","scrub_duration":0.0045847350000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"73'867","reported_seq":1007,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.026995+0000","last_change":"2023-10-25T00:23:27.027247+0000","last_active":"2023-10-25T00:23:27.026995+0000","last_peered":"2023-10-25T00:23:27.026995+0000","last_clean":"2023-10-25T00:23:27.026995+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-25T00:23:27.026995+0000","last_undegraded":"2023-10-25T00:23:27.026995+0000","last_fullsized":"2023-10-25T00:23:27.026995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'867","last_scrub_stamp":"2023-10-25T00:23:27.026943+0000","last_deep_scrub":"73'867","last_deep_scrub_stamp":"2023-10-25T00:23:27.026943+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.026943+0000","objects_scrubbed":3,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:41.065198+0000","scrub_duration":0.082266929000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":22541,"num_write":841,"num_write_kb":4586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'5616","reported_seq":5701,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:46.131046+0000","last_change":"2023-10-25T00:23:46.131251+0000","last_active":"2023-10-25T00:23:46.131046+0000","last_peered":"2023-10-25T00:23:46.131046+0000","last_clean":"2023-10-25T00:23:46.131046+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-25T00:23:46.131046+0000","last_undegraded":"2023-10-25T00:23:46.131046+0000","last_fullsized":"2023-10-25T00:23:46.131046+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5616","last_scrub_stamp":"2023-10-25T00:23:46.131002+0000","last_deep_scrub":"68'5616","last_deep_scrub_stamp":"2023-10-25T00:23:46.131002+0000","last_clean_scrub_stamp":"2023-10-25T00:23:46.131002+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:29:47.372296+0000","scrub_duration":0.13911974099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29775,"num_write_kb":348818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"72'3735","reported_seq":3808,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:23:17.537367+0000","last_change":"2023-10-25T00:23:53.537815+0000","last_active":"2023-10-25T00:23:17.537367+0000","last_peered":"2023-10-25T00:23:17.537367+0000","last_clean":"2023-10-25T00:23:17.537367+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-25T00:23:17.537367+0000","last_undegraded":"2023-10-25T00:23:17.537367+0000","last_fullsized":"2023-10-25T00:23:17.537367+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20090,"num_write_kb":264783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"72'2","reported_seq":83,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.880273+0000","last_change":"2023-10-25T00:23:02.880442+0000","last_active":"2023-10-25T00:23:02.880273+0000","last_peered":"2023-10-25T00:23:02.880273+0000","last_clean":"2023-10-25T00:23:02.880273+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-25T00:23:02.880273+0000","last_undegraded":"2023-10-25T00:23:02.880273+0000","last_fullsized":"2023-10-25T00:23:02.880273+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.880202+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:45.433003+0000","scrub_duration":0.0048172340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"73'28","reported_seq":116,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.443056+0000","last_change":"2023-10-25T00:23:05.443056+0000","last_active":"2023-10-25T00:23:05.443056+0000","last_peered":"2023-10-25T00:23:05.443056+0000","last_clean":"2023-10-25T00:23:05.443056+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-25T00:23:05.443056+0000","last_undegraded":"2023-10-25T00:23:05.443056+0000","last_fullsized":"2023-10-25T00:23:05.443056+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_deep_scrub":"73'28","last_deep_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.442750+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:29:04.499904+0000","scrub_duration":0.036465213000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'5447","reported_seq":5532,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:28.801261+0000","last_change":"2023-10-25T00:23:28.801391+0000","last_active":"2023-10-25T00:23:28.801261+0000","last_peered":"2023-10-25T00:23:28.801261+0000","last_clean":"2023-10-25T00:23:28.801261+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-25T00:23:28.801261+0000","last_undegraded":"2023-10-25T00:23:28.801261+0000","last_fullsized":"2023-10-25T00:23:28.801261+0000","mapping_epoch":21,"log_start":"61'100","ondisk_log_start":"61'100","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5447","last_scrub_stamp":"2023-10-25T00:23:28.801240+0000","last_deep_scrub":"68'5447","last_deep_scrub_stamp":"2023-10-25T00:23:28.801240+0000","last_clean_scrub_stamp":"2023-10-25T00:23:28.801240+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:44:02.442972+0000","scrub_duration":0.154979636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28063,"num_write_kb":350200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"72'4747","reported_seq":4831,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.173769+0000","last_change":"2023-10-25T00:23:40.173944+0000","last_active":"2023-10-25T00:23:40.173769+0000","last_peered":"2023-10-25T00:23:40.173769+0000","last_clean":"2023-10-25T00:23:40.173769+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-25T00:23:40.173769+0000","last_undegraded":"2023-10-25T00:23:40.173769+0000","last_fullsized":"2023-10-25T00:23:40.173769+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4747","last_scrub_stamp":"2023-10-25T00:23:40.173729+0000","last_deep_scrub":"72'4747","last_deep_scrub_stamp":"2023-10-25T00:23:40.173729+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.173729+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:39:14.413546+0000","scrub_duration":0.31868588399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":25766,"num_write_kb":328702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"72'754","reported_seq":862,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.214995+0000","last_change":"2023-10-25T00:22:37.433948+0000","last_active":"2023-10-25T00:22:49.214995+0000","last_peered":"2023-10-25T00:22:49.214995+0000","last_clean":"2023-10-25T00:22:49.214995+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-25T00:22:49.214995+0000","last_undegraded":"2023-10-25T00:22:49.214995+0000","last_fullsized":"2023-10-25T00:22:49.214995+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'754","last_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_deep_scrub":"72'754","last_deep_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.433382+0000","objects_scrubbed":1,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:43.601946+0000","scrub_duration":0.10905978199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":77282,"num_write":740,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"73'51","reported_seq":182,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.477614+0000","last_change":"2023-10-25T00:23:37.477885+0000","last_active":"2023-10-25T00:23:37.477614+0000","last_peered":"2023-10-25T00:23:37.477614+0000","last_clean":"2023-10-25T00:23:37.477614+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-25T00:23:37.477614+0000","last_undegraded":"2023-10-25T00:23:37.477614+0000","last_fullsized":"2023-10-25T00:23:37.477614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'51","last_scrub_stamp":"2023-10-25T00:23:37.477557+0000","last_deep_scrub":"73'51","last_deep_scrub_stamp":"2023-10-25T00:23:37.477557+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.477557+0000","objects_scrubbed":1,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:39.045780+0000","scrub_duration":0.008112401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'4792","reported_seq":4878,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234675+0000","last_change":"2023-10-25T00:22:44.881494+0000","last_active":"2023-10-25T00:22:49.234675+0000","last_peered":"2023-10-25T00:22:49.234675+0000","last_clean":"2023-10-25T00:22:49.234675+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-25T00:22:49.234675+0000","last_undegraded":"2023-10-25T00:22:49.234675+0000","last_fullsized":"2023-10-25T00:22:49.234675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4792","last_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_deep_scrub":"68'4792","last_deep_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.881344+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:23:57.562121+0000","scrub_duration":0.193670646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24997,"num_write_kb":313272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"72'4739","reported_seq":4824,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:49.802909+0000","last_change":"2023-10-25T00:23:49.803089+0000","last_active":"2023-10-25T00:23:49.802909+0000","last_peered":"2023-10-25T00:23:49.802909+0000","last_clean":"2023-10-25T00:23:49.802909+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-25T00:23:49.802909+0000","last_undegraded":"2023-10-25T00:23:49.802909+0000","last_fullsized":"2023-10-25T00:23:49.802909+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4739","last_scrub_stamp":"2023-10-25T00:23:49.802870+0000","last_deep_scrub":"72'4739","last_deep_scrub_stamp":"2023-10-25T00:23:49.802870+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.802870+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:56:59.101994+0000","scrub_duration":0.33146283399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":22,"num_write":25639,"num_write_kb":332364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"72'357","reported_seq":430,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.998388+0000","last_change":"2023-10-25T00:23:16.269311+0000","last_active":"2023-10-25T00:22:55.998388+0000","last_peered":"2023-10-25T00:22:55.998388+0000","last_clean":"2023-10-25T00:22:55.998388+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-25T00:22:55.998388+0000","last_undegraded":"2023-10-25T00:22:55.998388+0000","last_fullsized":"2023-10-25T00:22:55.998388+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2627766,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":227,"num_write":329,"num_write_kb":2759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"73'6","reported_seq":139,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:39.493024+0000","last_change":"2023-10-25T00:23:39.493290+0000","last_active":"2023-10-25T00:23:39.493024+0000","last_peered":"2023-10-25T00:23:39.493024+0000","last_clean":"2023-10-25T00:23:39.493024+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-25T00:23:39.493024+0000","last_undegraded":"2023-10-25T00:23:39.493024+0000","last_fullsized":"2023-10-25T00:23:39.493024+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2023-10-25T00:23:39.492964+0000","last_deep_scrub":"73'6","last_deep_scrub_stamp":"2023-10-25T00:23:39.492964+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.492964+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:34.589742+0000","scrub_duration":0.0071686809999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'3731","reported_seq":3815,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234039+0000","last_change":"2023-10-25T00:22:37.622543+0000","last_active":"2023-10-25T00:22:49.234039+0000","last_peered":"2023-10-25T00:22:49.234039+0000","last_clean":"2023-10-25T00:22:49.234039+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-25T00:22:49.234039+0000","last_undegraded":"2023-10-25T00:22:49.234039+0000","last_fullsized":"2023-10-25T00:22:49.234039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3731","last_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_deep_scrub":"68'3731","last_deep_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.622304+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:51:52.171234+0000","scrub_duration":0.090187284000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19492,"num_write_kb":250091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"72'3799","reported_seq":3872,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.236389+0000","last_change":"2023-10-24T23:53:11.837583+0000","last_active":"2023-10-25T00:22:49.236389+0000","last_peered":"2023-10-25T00:22:49.236389+0000","last_clean":"2023-10-25T00:22:49.236389+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-25T00:22:49.236389+0000","last_undegraded":"2023-10-25T00:22:49.236389+0000","last_fullsized":"2023-10-25T00:22:49.236389+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21581,"num_write_kb":262877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"73'46","reported_seq":171,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.664257+0000","last_change":"2023-10-25T00:22:52.664257+0000","last_active":"2023-10-25T00:22:52.664257+0000","last_peered":"2023-10-25T00:22:52.664257+0000","last_clean":"2023-10-25T00:22:52.664257+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-25T00:22:52.664257+0000","last_undegraded":"2023-10-25T00:22:52.664257+0000","last_fullsized":"2023-10-25T00:22:52.664257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'46","last_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_deep_scrub":"73'46","last_deep_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.664064+0000","objects_scrubbed":1,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:44:09.108291+0000","scrub_duration":0.106079285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"73'715","reported_seq":829,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.967748+0000","last_change":"2023-10-25T00:23:21.967998+0000","last_active":"2023-10-25T00:23:21.967748+0000","last_peered":"2023-10-25T00:23:21.967748+0000","last_clean":"2023-10-25T00:23:21.967748+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-25T00:23:21.967748+0000","last_undegraded":"2023-10-25T00:23:21.967748+0000","last_fullsized":"2023-10-25T00:23:21.967748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'715","last_scrub_stamp":"2023-10-25T00:23:21.967694+0000","last_deep_scrub":"73'715","last_deep_scrub_stamp":"2023-10-25T00:23:21.967694+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.967694+0000","objects_scrubbed":1,"log_size":715,"log_dups_size":0,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:56:40.443702+0000","scrub_duration":0.065018907000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":90428,"num_write":703,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'4008","reported_seq":4096,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.488463+0000","last_change":"2023-10-25T00:23:05.488463+0000","last_active":"2023-10-25T00:23:05.488463+0000","last_peered":"2023-10-25T00:23:05.488463+0000","last_clean":"2023-10-25T00:23:05.488463+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-25T00:23:05.488463+0000","last_undegraded":"2023-10-25T00:23:05.488463+0000","last_fullsized":"2023-10-25T00:23:05.488463+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4008","last_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_deep_scrub":"68'4008","last_deep_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.488248+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:51:16.627187+0000","scrub_duration":0.15813021899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20791,"num_write_kb":267770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"72'5469","reported_seq":5564,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.571255+0000","last_change":"2023-10-25T00:22:57.571401+0000","last_active":"2023-10-25T00:22:57.571255+0000","last_peered":"2023-10-25T00:22:57.571255+0000","last_clean":"2023-10-25T00:22:57.571255+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-25T00:22:57.571255+0000","last_undegraded":"2023-10-25T00:22:57.571255+0000","last_fullsized":"2023-10-25T00:22:57.571255+0000","mapping_epoch":25,"log_start":"64'200","ondisk_log_start":"64'200","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5469","last_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_deep_scrub":"72'5469","last_deep_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.571234+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:24:13.836466+0000","scrub_duration":0.19144520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":40,"num_write":30646,"num_write_kb":401481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"73'6","reported_seq":131,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.991436+0000","last_change":"2023-10-25T00:23:14.991436+0000","last_active":"2023-10-25T00:23:14.991436+0000","last_peered":"2023-10-25T00:23:14.991436+0000","last_clean":"2023-10-25T00:23:14.991436+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-25T00:23:14.991436+0000","last_undegraded":"2023-10-25T00:23:14.991436+0000","last_fullsized":"2023-10-25T00:23:14.991436+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_deep_scrub":"73'6","last_deep_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.991238+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:25:08.545014+0000","scrub_duration":0.0072276010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"73'546","reported_seq":631,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:23:06.302701+0000","last_change":"2023-10-25T00:23:36.987182+0000","last_active":"2023-10-25T00:23:06.302701+0000","last_peered":"2023-10-25T00:23:06.302701+0000","last_clean":"2023-10-25T00:23:06.302701+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-25T00:23:06.302701+0000","last_undegraded":"2023-10-25T00:23:06.302701+0000","last_fullsized":"2023-10-25T00:23:06.302701+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3198201,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10628,"num_write":517,"num_write_kb":3434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'6069","reported_seq":6156,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.235476+0000","last_change":"2023-10-25T00:22:46.608504+0000","last_active":"2023-10-25T00:22:49.235476+0000","last_peered":"2023-10-25T00:22:49.235476+0000","last_clean":"2023-10-25T00:22:49.235476+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-25T00:22:49.235476+0000","last_undegraded":"2023-10-25T00:22:49.235476+0000","last_fullsized":"2023-10-25T00:22:49.235476+0000","mapping_epoch":21,"log_start":"62'800","ondisk_log_start":"62'800","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6069","last_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_deep_scrub":"68'6069","last_deep_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.608198+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:37.101860+0000","scrub_duration":0.19917779799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":33187,"num_write_kb":387130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'3606","reported_seq":3694,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:38.186285+0000","last_change":"2023-10-25T00:23:38.186420+0000","last_active":"2023-10-25T00:23:38.186285+0000","last_peered":"2023-10-25T00:23:38.186285+0000","last_clean":"2023-10-25T00:23:38.186285+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-25T00:23:38.186285+0000","last_undegraded":"2023-10-25T00:23:38.186285+0000","last_fullsized":"2023-10-25T00:23:38.186285+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3606","last_scrub_stamp":"2023-10-25T00:23:38.186264+0000","last_deep_scrub":"72'3606","last_deep_scrub_stamp":"2023-10-25T00:23:38.186264+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.186264+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:15.619690+0000","scrub_duration":0.23143245800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19493,"num_write_kb":254342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'1064","reported_seq":1198,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:03.701358+0000","last_change":"2023-10-25T00:23:03.701535+0000","last_active":"2023-10-25T00:23:03.701358+0000","last_peered":"2023-10-25T00:23:03.701358+0000","last_clean":"2023-10-25T00:23:03.701358+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-25T00:23:03.701358+0000","last_undegraded":"2023-10-25T00:23:03.701358+0000","last_fullsized":"2023-10-25T00:23:03.701358+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1064","last_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_deep_scrub":"73'1064","last_deep_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_clean_scrub_stamp":"2023-10-25T00:23:03.701304+0000","objects_scrubbed":3,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:08:18.178955+0000","scrub_duration":0.28236572100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":9402,"num_write":1035,"num_write_kb":4717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'3","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.838808+0000","last_change":"2023-10-25T00:23:12.838945+0000","last_active":"2023-10-25T00:23:12.838808+0000","last_peered":"2023-10-25T00:23:12.838808+0000","last_clean":"2023-10-25T00:23:12.838808+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-25T00:23:12.838808+0000","last_undegraded":"2023-10-25T00:23:12.838808+0000","last_fullsized":"2023-10-25T00:23:12.838808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.838787+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:33:35.778931+0000","scrub_duration":0.133446072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'5139","reported_seq":5226,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:54.030976+0000","last_change":"2023-10-25T00:22:54.030976+0000","last_active":"2023-10-25T00:22:54.030976+0000","last_peered":"2023-10-25T00:22:54.030976+0000","last_clean":"2023-10-25T00:22:54.030976+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-25T00:22:54.030976+0000","last_undegraded":"2023-10-25T00:22:54.030976+0000","last_fullsized":"2023-10-25T00:22:54.030976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5139","last_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_deep_scrub":"68'5139","last_deep_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.030782+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:30.182861+0000","scrub_duration":0.103857617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27772,"num_write_kb":328469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"72'4843","reported_seq":4927,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:39.041680+0000","last_change":"2023-10-25T00:23:39.041980+0000","last_active":"2023-10-25T00:23:39.041680+0000","last_peered":"2023-10-25T00:23:39.041680+0000","last_clean":"2023-10-25T00:23:39.041680+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-25T00:23:39.041680+0000","last_undegraded":"2023-10-25T00:23:39.041680+0000","last_fullsized":"2023-10-25T00:23:39.041680+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4843","last_scrub_stamp":"2023-10-25T00:23:39.041623+0000","last_deep_scrub":"72'4843","last_deep_scrub_stamp":"2023-10-25T00:23:39.041623+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.041623+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:34:30.360915+0000","scrub_duration":0.33452469800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":27290,"num_write_kb":330906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"72'27","reported_seq":111,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.987117+0000","last_change":"2023-10-25T00:22:55.987117+0000","last_active":"2023-10-25T00:22:55.987117+0000","last_peered":"2023-10-25T00:22:55.987117+0000","last_clean":"2023-10-25T00:22:55.987117+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-25T00:22:55.987117+0000","last_undegraded":"2023-10-25T00:22:55.987117+0000","last_fullsized":"2023-10-25T00:22:55.987117+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'27","last_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_deep_scrub":"72'27","last_deep_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.986674+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:33.089164+0000","scrub_duration":0.100730242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'2","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.503834+0000","last_change":"2023-10-25T00:23:21.504086+0000","last_active":"2023-10-25T00:23:21.503834+0000","last_peered":"2023-10-25T00:23:21.503834+0000","last_clean":"2023-10-25T00:23:21.503834+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-25T00:23:21.503834+0000","last_undegraded":"2023-10-25T00:23:21.503834+0000","last_fullsized":"2023-10-25T00:23:21.503834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2023-10-25T00:23:21.503783+0000","last_deep_scrub":"67'2","last_deep_scrub_stamp":"2023-10-25T00:23:21.503783+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.503783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:13.653289+0000","scrub_duration":0.0039280959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'5814","reported_seq":5901,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.320438+0000","last_change":"2023-10-25T00:22:52.320696+0000","last_active":"2023-10-25T00:22:52.320438+0000","last_peered":"2023-10-25T00:22:52.320438+0000","last_clean":"2023-10-25T00:22:52.320438+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-25T00:22:52.320438+0000","last_undegraded":"2023-10-25T00:22:52.320438+0000","last_fullsized":"2023-10-25T00:22:52.320438+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5814","last_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_deep_scrub":"68'5814","last_deep_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.320378+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:26.511615+0000","scrub_duration":0.033495337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31996,"num_write_kb":382397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"72'3827","reported_seq":3912,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.436458+0000","last_change":"2023-10-25T00:23:14.436458+0000","last_active":"2023-10-25T00:23:14.436458+0000","last_peered":"2023-10-25T00:23:14.436458+0000","last_clean":"2023-10-25T00:23:14.436458+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-25T00:23:14.436458+0000","last_undegraded":"2023-10-25T00:23:14.436458+0000","last_fullsized":"2023-10-25T00:23:14.436458+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3827","last_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_deep_scrub":"72'3827","last_deep_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.436150+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:27:32.423180+0000","scrub_duration":0.30547422699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":21199,"num_write_kb":277730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"68'4820","reported_seq":4907,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232177+0000","last_change":"2023-10-25T00:22:36.429763+0000","last_active":"2023-10-25T00:22:49.232177+0000","last_peered":"2023-10-25T00:22:49.232177+0000","last_clean":"2023-10-25T00:22:49.232177+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-25T00:22:49.232177+0000","last_undegraded":"2023-10-25T00:22:49.232177+0000","last_fullsized":"2023-10-25T00:22:49.232177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4820","last_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_deep_scrub":"68'4820","last_deep_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_clean_scrub_stamp":"2023-10-25T00:22:36.429607+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:35.883183+0000","scrub_duration":0.10613655600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25163,"num_write_kb":285293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"72'5","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.946770+0000","last_change":"2023-10-25T00:23:20.947028+0000","last_active":"2023-10-25T00:23:20.946770+0000","last_peered":"2023-10-25T00:23:20.946770+0000","last_clean":"2023-10-25T00:23:20.946770+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-25T00:23:20.946770+0000","last_undegraded":"2023-10-25T00:23:20.946770+0000","last_fullsized":"2023-10-25T00:23:20.946770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:23:20.946713+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:23:20.946713+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.946713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:12:40.769613+0000","scrub_duration":0.0045247550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"72'5","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.215935+0000","last_change":"2023-10-25T00:22:40.197550+0000","last_active":"2023-10-25T00:22:49.215935+0000","last_peered":"2023-10-25T00:22:49.215935+0000","last_clean":"2023-10-25T00:22:49.215935+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-25T00:22:49.215935+0000","last_undegraded":"2023-10-25T00:22:49.215935+0000","last_fullsized":"2023-10-25T00:22:49.215935+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.197213+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:07:28.509914+0000","scrub_duration":0.0047704979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"72'4957","reported_seq":5045,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:53.667906+0000","last_change":"2023-10-25T00:22:53.667906+0000","last_active":"2023-10-25T00:22:53.667906+0000","last_peered":"2023-10-25T00:22:53.667906+0000","last_clean":"2023-10-25T00:22:53.667906+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-25T00:22:53.667906+0000","last_undegraded":"2023-10-25T00:22:53.667906+0000","last_fullsized":"2023-10-25T00:22:53.667906+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4957","last_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_deep_scrub":"72'4957","last_deep_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.667748+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:58.307897+0000","scrub_duration":0.30812842099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28093,"num_write_kb":358545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'4782","reported_seq":4874,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234575+0000","last_change":"2023-10-25T00:22:42.562198+0000","last_active":"2023-10-25T00:22:49.234575+0000","last_peered":"2023-10-25T00:22:49.234575+0000","last_clean":"2023-10-25T00:22:49.234575+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-25T00:22:49.234575+0000","last_undegraded":"2023-10-25T00:22:49.234575+0000","last_fullsized":"2023-10-25T00:22:49.234575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4782","last_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_deep_scrub":"68'4782","last_deep_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.562057+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:39.415960+0000","scrub_duration":0.116720554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26432,"num_write_kb":298639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"68'3","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.433942+0000","last_change":"2023-10-25T00:23:36.434199+0000","last_active":"2023-10-25T00:23:36.433942+0000","last_peered":"2023-10-25T00:23:36.433942+0000","last_clean":"2023-10-25T00:23:36.433942+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-25T00:23:36.433942+0000","last_undegraded":"2023-10-25T00:23:36.433942+0000","last_fullsized":"2023-10-25T00:23:36.433942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:36.433882+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:36.433882+0000","last_clean_scrub_stamp":"2023-10-25T00:23:36.433882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:00:44.210403+0000","scrub_duration":0.0040752599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"72'18","reported_seq":215,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.214421+0000","last_change":"2023-10-25T00:22:46.074083+0000","last_active":"2023-10-25T00:22:49.214421+0000","last_peered":"2023-10-25T00:22:49.214421+0000","last_clean":"2023-10-25T00:22:49.214421+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-25T00:22:49.214421+0000","last_undegraded":"2023-10-25T00:22:49.214421+0000","last_fullsized":"2023-10-25T00:22:49.214421+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_deep_scrub":"72'18","last_deep_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.073758+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:08:25.787090+0000","scrub_duration":0.012558368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":345,"num_read_kb":345,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"72'2954","reported_seq":3037,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.099045+0000","last_change":"2023-10-25T00:23:36.099045+0000","last_active":"2023-10-25T00:23:36.099045+0000","last_peered":"2023-10-25T00:23:36.099045+0000","last_clean":"2023-10-25T00:23:36.099045+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-25T00:23:36.099045+0000","last_undegraded":"2023-10-25T00:23:36.099045+0000","last_fullsized":"2023-10-25T00:23:36.099045+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2954","last_scrub_stamp":"2023-10-25T00:23:36.098826+0000","last_deep_scrub":"72'2954","last_deep_scrub_stamp":"2023-10-25T00:23:36.098826+0000","last_clean_scrub_stamp":"2023-10-25T00:23:36.098826+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:16.696480+0000","scrub_duration":0.32992982700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15961,"num_write_kb":219669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"73'38","reported_seq":169,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.052563+0000","last_change":"2023-10-25T00:23:35.052803+0000","last_active":"2023-10-25T00:23:35.052563+0000","last_peered":"2023-10-25T00:23:35.052563+0000","last_clean":"2023-10-25T00:23:35.052563+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-25T00:23:35.052563+0000","last_undegraded":"2023-10-25T00:23:35.052563+0000","last_fullsized":"2023-10-25T00:23:35.052563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'38","last_scrub_stamp":"2023-10-25T00:23:35.052506+0000","last_deep_scrub":"73'38","last_deep_scrub_stamp":"2023-10-25T00:23:35.052506+0000","last_clean_scrub_stamp":"2023-10-25T00:23:35.052506+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:23:31.370767+0000","scrub_duration":0.0095825770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'4690","reported_seq":4778,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.792729+0000","last_change":"2023-10-25T00:23:05.792729+0000","last_active":"2023-10-25T00:23:05.792729+0000","last_peered":"2023-10-25T00:23:05.792729+0000","last_clean":"2023-10-25T00:23:05.792729+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-25T00:23:05.792729+0000","last_undegraded":"2023-10-25T00:23:05.792729+0000","last_fullsized":"2023-10-25T00:23:05.792729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4690","last_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_deep_scrub":"68'4690","last_deep_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.792588+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:24:53.476824+0000","scrub_duration":0.15089322399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25172,"num_write_kb":305786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"72'10","reported_seq":136,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234044+0000","last_change":"2023-10-25T00:22:40.512549+0000","last_active":"2023-10-25T00:22:49.234044+0000","last_peered":"2023-10-25T00:22:49.234044+0000","last_clean":"2023-10-25T00:22:49.234044+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-25T00:22:49.234044+0000","last_undegraded":"2023-10-25T00:22:49.234044+0000","last_fullsized":"2023-10-25T00:22:49.234044+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.512409+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:07:23.705506+0000","scrub_duration":0.091817919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"72'4200","reported_seq":4291,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.809510+0000","last_change":"2023-10-25T00:23:34.809510+0000","last_active":"2023-10-25T00:23:34.809510+0000","last_peered":"2023-10-25T00:23:34.809510+0000","last_clean":"2023-10-25T00:23:34.809510+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-25T00:23:34.809510+0000","last_undegraded":"2023-10-25T00:23:34.809510+0000","last_fullsized":"2023-10-25T00:23:34.809510+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4200","last_scrub_stamp":"2023-10-25T00:23:34.809362+0000","last_deep_scrub":"72'4200","last_deep_scrub_stamp":"2023-10-25T00:23:34.809362+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.809362+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:04:30.830862+0000","scrub_duration":0.23894108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":24169,"num_write_kb":293796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"73'394","reported_seq":596,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:31.115748+0000","last_change":"2023-10-25T00:23:31.116025+0000","last_active":"2023-10-25T00:23:31.115748+0000","last_peered":"2023-10-25T00:23:31.115748+0000","last_clean":"2023-10-25T00:23:31.115748+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-25T00:23:31.115748+0000","last_undegraded":"2023-10-25T00:23:31.115748+0000","last_fullsized":"2023-10-25T00:23:31.115748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'394","last_scrub_stamp":"2023-10-25T00:23:31.115692+0000","last_deep_scrub":"73'394","last_deep_scrub_stamp":"2023-10-25T00:23:31.115692+0000","last_clean_scrub_stamp":"2023-10-25T00:23:31.115692+0000","objects_scrubbed":4,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:38.234847+0000","scrub_duration":0.026138159000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":210,"num_read_kb":94395,"num_write":387,"num_write_kb":4304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'5730","reported_seq":5817,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.204004+0000","last_change":"2023-10-25T00:23:30.204139+0000","last_active":"2023-10-25T00:23:30.204004+0000","last_peered":"2023-10-25T00:23:30.204004+0000","last_clean":"2023-10-25T00:23:30.204004+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-25T00:23:30.204004+0000","last_undegraded":"2023-10-25T00:23:30.204004+0000","last_fullsized":"2023-10-25T00:23:30.204004+0000","mapping_epoch":21,"log_start":"61'500","ondisk_log_start":"61'500","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5730","last_scrub_stamp":"2023-10-25T00:23:30.203981+0000","last_deep_scrub":"68'5730","last_deep_scrub_stamp":"2023-10-25T00:23:30.203981+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.203981+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:38:35.578976+0000","scrub_duration":0.10293530200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":31652,"num_write_kb":356481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"72'3","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.639911+0000","last_change":"2023-10-25T00:23:27.639911+0000","last_active":"2023-10-25T00:23:27.639911+0000","last_peered":"2023-10-25T00:23:27.639911+0000","last_clean":"2023-10-25T00:23:27.639911+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-25T00:23:27.639911+0000","last_undegraded":"2023-10-25T00:23:27.639911+0000","last_fullsized":"2023-10-25T00:23:27.639911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:27.639604+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:27.639604+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.639604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:09:33.666294+0000","scrub_duration":0.0047758339999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"72'4168","reported_seq":4243,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.209592+0000","last_change":"2023-10-25T00:23:46.041659+0000","last_active":"2023-10-25T00:22:49.209592+0000","last_peered":"2023-10-25T00:22:49.209592+0000","last_clean":"2023-10-25T00:22:49.209592+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-25T00:22:49.209592+0000","last_undegraded":"2023-10-25T00:22:49.209592+0000","last_fullsized":"2023-10-25T00:22:49.209592+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23458,"num_write_kb":294850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"68'4339","reported_seq":4426,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:47.099364+0000","last_change":"2023-10-25T00:23:47.099492+0000","last_active":"2023-10-25T00:23:47.099364+0000","last_peered":"2023-10-25T00:23:47.099364+0000","last_clean":"2023-10-25T00:23:47.099364+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-25T00:23:47.099364+0000","last_undegraded":"2023-10-25T00:23:47.099364+0000","last_fullsized":"2023-10-25T00:23:47.099364+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4339","last_scrub_stamp":"2023-10-25T00:23:47.099343+0000","last_deep_scrub":"68'4339","last_deep_scrub_stamp":"2023-10-25T00:23:47.099343+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.099343+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:27.913875+0000","scrub_duration":0.12538819900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23267,"num_write_kb":265268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.223439+0000","last_change":"2023-10-25T00:22:42.199286+0000","last_active":"2023-10-25T00:22:49.223439+0000","last_peered":"2023-10-25T00:22:49.223439+0000","last_clean":"2023-10-25T00:22:49.223439+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-25T00:22:49.223439+0000","last_undegraded":"2023-10-25T00:22:49.223439+0000","last_fullsized":"2023-10-25T00:22:49.223439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.198947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:11:17.807377+0000","scrub_duration":0.0041955760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"73'231","reported_seq":342,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.010034+0000","last_change":"2023-10-25T00:23:16.010034+0000","last_active":"2023-10-25T00:23:16.010034+0000","last_peered":"2023-10-25T00:23:16.010034+0000","last_clean":"2023-10-25T00:23:16.010034+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-25T00:23:16.010034+0000","last_undegraded":"2023-10-25T00:23:16.010034+0000","last_fullsized":"2023-10-25T00:23:16.010034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'231","last_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_deep_scrub":"73'231","last_deep_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.009842+0000","objects_scrubbed":1,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:27:21.881029+0000","scrub_duration":0.076880452000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":63,"num_write":590,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"72'11","reported_seq":95,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.231986+0000","last_change":"2023-10-25T00:22:38.409520+0000","last_active":"2023-10-25T00:22:49.231986+0000","last_peered":"2023-10-25T00:22:49.231986+0000","last_clean":"2023-10-25T00:22:49.231986+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-25T00:22:49.231986+0000","last_undegraded":"2023-10-25T00:22:49.231986+0000","last_fullsized":"2023-10-25T00:22:49.231986+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_deep_scrub":"72'11","last_deep_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.409235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:21:24.708709+0000","scrub_duration":0.003592329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"72'3993","reported_seq":4069,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060792+0000","last_change":"2023-10-24T23:53:11.827821+0000","last_active":"2023-10-25T00:22:36.060792+0000","last_peered":"2023-10-25T00:22:36.060792+0000","last_clean":"2023-10-25T00:22:36.060792+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-25T00:22:36.060792+0000","last_undegraded":"2023-10-25T00:22:36.060792+0000","last_fullsized":"2023-10-25T00:22:36.060792+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21723,"num_write_kb":283850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'4270","reported_seq":4358,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:59.570791+0000","last_change":"2023-10-25T00:22:59.570791+0000","last_active":"2023-10-25T00:22:59.570791+0000","last_peered":"2023-10-25T00:22:59.570791+0000","last_clean":"2023-10-25T00:22:59.570791+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-25T00:22:59.570791+0000","last_undegraded":"2023-10-25T00:22:59.570791+0000","last_fullsized":"2023-10-25T00:22:59.570791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4270","last_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_deep_scrub":"68'4270","last_deep_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.570628+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:36:56.294621+0000","scrub_duration":0.14092118100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23049,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.643510+0000","last_change":"2023-10-25T00:23:06.643510+0000","last_active":"2023-10-25T00:23:06.643510+0000","last_peered":"2023-10-25T00:23:06.643510+0000","last_clean":"2023-10-25T00:23:06.643510+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-25T00:23:06.643510+0000","last_undegraded":"2023-10-25T00:23:06.643510+0000","last_fullsized":"2023-10-25T00:23:06.643510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.643156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:34:24.773001+0000","scrub_duration":0.0045495850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"73'10","reported_seq":145,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.618489+0000","last_change":"2023-10-25T00:23:20.618489+0000","last_active":"2023-10-25T00:23:20.618489+0000","last_peered":"2023-10-25T00:23:20.618489+0000","last_clean":"2023-10-25T00:23:20.618489+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-25T00:23:20.618489+0000","last_undegraded":"2023-10-25T00:23:20.618489+0000","last_fullsized":"2023-10-25T00:23:20.618489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2023-10-25T00:23:20.618371+0000","last_deep_scrub":"73'10","last_deep_scrub_stamp":"2023-10-25T00:23:20.618371+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.618371+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:30:10.455022+0000","scrub_duration":0.085107656000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":8,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"73'65","reported_seq":228,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.930205+0000","last_change":"2023-10-25T00:23:07.930465+0000","last_active":"2023-10-25T00:23:07.930205+0000","last_peered":"2023-10-25T00:23:07.930205+0000","last_clean":"2023-10-25T00:23:07.930205+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-25T00:23:07.930205+0000","last_undegraded":"2023-10-25T00:23:07.930205+0000","last_fullsized":"2023-10-25T00:23:07.930205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'65","last_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_deep_scrub":"73'65","last_deep_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.930143+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:41:49.528670+0000","scrub_duration":0.020886851000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":123,"num_write":59,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"72'3769","reported_seq":3856,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:38.049931+0000","last_change":"2023-10-25T00:23:38.050117+0000","last_active":"2023-10-25T00:23:38.049931+0000","last_peered":"2023-10-25T00:23:38.049931+0000","last_clean":"2023-10-25T00:23:38.049931+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-25T00:23:38.049931+0000","last_undegraded":"2023-10-25T00:23:38.049931+0000","last_fullsized":"2023-10-25T00:23:38.049931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3769","last_scrub_stamp":"2023-10-25T00:23:38.049889+0000","last_deep_scrub":"72'3769","last_deep_scrub_stamp":"2023-10-25T00:23:38.049889+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.049889+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:23:14.574715+0000","scrub_duration":0.37571605800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20807,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'5752","reported_seq":5839,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.937946+0000","last_change":"2023-10-25T00:22:33.937946+0000","last_active":"2023-10-25T00:22:33.937946+0000","last_peered":"2023-10-25T00:22:33.937946+0000","last_clean":"2023-10-25T00:22:33.937946+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-25T00:22:33.937946+0000","last_undegraded":"2023-10-25T00:22:33.937946+0000","last_fullsized":"2023-10-25T00:22:33.937946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5752","last_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_deep_scrub":"68'5752","last_deep_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_clean_scrub_stamp":"2023-10-25T00:22:33.937564+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:43:34.779084+0000","scrub_duration":0.080678784000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31231,"num_write_kb":354718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:24.046769+0000","last_change":"2023-10-25T00:23:24.047030+0000","last_active":"2023-10-25T00:23:24.046769+0000","last_peered":"2023-10-25T00:23:24.046769+0000","last_clean":"2023-10-25T00:23:24.046769+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-25T00:23:24.046769+0000","last_undegraded":"2023-10-25T00:23:24.046769+0000","last_fullsized":"2023-10-25T00:23:24.046769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:24.046716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:24.046716+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.046716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:53:42.506468+0000","scrub_duration":0.0043444529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'2","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:18.683846+0000","last_change":"2023-10-25T00:23:18.684046+0000","last_active":"2023-10-25T00:23:18.683846+0000","last_peered":"2023-10-25T00:23:18.683846+0000","last_clean":"2023-10-25T00:23:18.683846+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-25T00:23:18.683846+0000","last_undegraded":"2023-10-25T00:23:18.683846+0000","last_fullsized":"2023-10-25T00:23:18.683846+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2023-10-25T00:23:18.683799+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2023-10-25T00:23:18.683799+0000","last_clean_scrub_stamp":"2023-10-25T00:23:18.683799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:26:44.716117+0000","scrub_duration":0.013956036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"72'4","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.496631+0000","last_change":"2023-10-25T00:23:06.496904+0000","last_active":"2023-10-25T00:23:06.496631+0000","last_peered":"2023-10-25T00:23:06.496631+0000","last_clean":"2023-10-25T00:23:06.496631+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-25T00:23:06.496631+0000","last_undegraded":"2023-10-25T00:23:06.496631+0000","last_fullsized":"2023-10-25T00:23:06.496631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_deep_scrub":"72'4","last_deep_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.496575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:32:18.665160+0000","scrub_duration":0.0046312189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"72'3939","reported_seq":4022,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.160279+0000","last_change":"2023-10-25T00:23:02.160279+0000","last_active":"2023-10-25T00:23:02.160279+0000","last_peered":"2023-10-25T00:23:02.160279+0000","last_clean":"2023-10-25T00:23:02.160279+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-25T00:23:02.160279+0000","last_undegraded":"2023-10-25T00:23:02.160279+0000","last_fullsized":"2023-10-25T00:23:02.160279+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3939","last_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_deep_scrub":"72'3939","last_deep_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.160086+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:37:29.878702+0000","scrub_duration":0.252775894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22393,"num_write_kb":262004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'4305","reported_seq":4392,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.858696+0000","last_change":"2023-10-25T00:23:26.858862+0000","last_active":"2023-10-25T00:23:26.858696+0000","last_peered":"2023-10-25T00:23:26.858696+0000","last_clean":"2023-10-25T00:23:26.858696+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-25T00:23:26.858696+0000","last_undegraded":"2023-10-25T00:23:26.858696+0000","last_fullsized":"2023-10-25T00:23:26.858696+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4305","last_scrub_stamp":"2023-10-25T00:23:26.858657+0000","last_deep_scrub":"68'4305","last_deep_scrub_stamp":"2023-10-25T00:23:26.858657+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.858657+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:10:03.352691+0000","scrub_duration":0.18050687100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22696,"num_write_kb":287574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:11.717097+0000","last_change":"2023-10-25T00:23:11.717353+0000","last_active":"2023-10-25T00:23:11.717097+0000","last_peered":"2023-10-25T00:23:11.717097+0000","last_clean":"2023-10-25T00:23:11.717097+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-25T00:23:11.717097+0000","last_undegraded":"2023-10-25T00:23:11.717097+0000","last_fullsized":"2023-10-25T00:23:11.717097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_clean_scrub_stamp":"2023-10-25T00:23:11.717027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:17:42.982457+0000","scrub_duration":0.0039434270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.354835+0000","last_change":"2023-10-25T00:23:30.355099+0000","last_active":"2023-10-25T00:23:30.354835+0000","last_peered":"2023-10-25T00:23:30.354835+0000","last_clean":"2023-10-25T00:23:30.354835+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-25T00:23:30.354835+0000","last_undegraded":"2023-10-25T00:23:30.354835+0000","last_fullsized":"2023-10-25T00:23:30.354835+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2023-10-25T00:23:30.354780+0000","last_deep_scrub":"67'2","last_deep_scrub_stamp":"2023-10-25T00:23:30.354780+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.354780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:07:40.743916+0000","scrub_duration":0.0044225610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"72'3","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.952645+0000","last_change":"2023-10-25T00:22:52.952897+0000","last_active":"2023-10-25T00:22:52.952645+0000","last_peered":"2023-10-25T00:22:52.952645+0000","last_clean":"2023-10-25T00:22:52.952645+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-25T00:22:52.952645+0000","last_undegraded":"2023-10-25T00:22:52.952645+0000","last_fullsized":"2023-10-25T00:22:52.952645+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.952595+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:14:20.890322+0000","scrub_duration":0.0044650949999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"72'5003","reported_seq":5092,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:44.774558+0000","last_change":"2023-10-25T00:23:44.774832+0000","last_active":"2023-10-25T00:23:44.774558+0000","last_peered":"2023-10-25T00:23:44.774558+0000","last_clean":"2023-10-25T00:23:44.774558+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-25T00:23:44.774558+0000","last_undegraded":"2023-10-25T00:23:44.774558+0000","last_fullsized":"2023-10-25T00:23:44.774558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5003","last_scrub_stamp":"2023-10-25T00:23:44.774508+0000","last_deep_scrub":"72'5003","last_deep_scrub_stamp":"2023-10-25T00:23:44.774508+0000","last_clean_scrub_stamp":"2023-10-25T00:23:44.774508+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:52.739672+0000","scrub_duration":0.33716653400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":13,"num_write":27120,"num_write_kb":350355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'5635","reported_seq":5721,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:44.547139+0000","last_change":"2023-10-25T00:22:44.547139+0000","last_active":"2023-10-25T00:22:44.547139+0000","last_peered":"2023-10-25T00:22:44.547139+0000","last_clean":"2023-10-25T00:22:44.547139+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-25T00:22:44.547139+0000","last_undegraded":"2023-10-25T00:22:44.547139+0000","last_fullsized":"2023-10-25T00:22:44.547139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5635","last_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_deep_scrub":"68'5635","last_deep_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.546984+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:51:54.474684+0000","scrub_duration":0.15865177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30556,"num_write_kb":356140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"68'3","reported_seq":80,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.235441+0000","last_change":"2023-10-25T00:23:35.979143+0000","last_active":"2023-10-25T00:22:49.235441+0000","last_peered":"2023-10-25T00:22:49.235441+0000","last_clean":"2023-10-25T00:22:49.235441+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-25T00:22:49.235441+0000","last_undegraded":"2023-10-25T00:22:49.235441+0000","last_fullsized":"2023-10-25T00:22:49.235441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:29.305074+0000","last_change":"2023-10-25T00:23:29.305381+0000","last_active":"2023-10-25T00:23:29.305074+0000","last_peered":"2023-10-25T00:23:29.305074+0000","last_clean":"2023-10-25T00:23:29.305074+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-25T00:23:29.305074+0000","last_undegraded":"2023-10-25T00:23:29.305074+0000","last_fullsized":"2023-10-25T00:23:29.305074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:29.305022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:29.305022+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.305022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:52:12.477955+0000","scrub_duration":0.0044947839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"72'2","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:58.925373+0000","last_change":"2023-10-25T00:22:58.925685+0000","last_active":"2023-10-25T00:22:58.925373+0000","last_peered":"2023-10-25T00:22:58.925373+0000","last_clean":"2023-10-25T00:22:58.925373+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-25T00:22:58.925373+0000","last_undegraded":"2023-10-25T00:22:58.925373+0000","last_fullsized":"2023-10-25T00:22:58.925373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.925318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:31:45.562398+0000","scrub_duration":0.0043101720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"72'3753","reported_seq":3839,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:44.352105+0000","last_change":"2023-10-25T00:23:44.352225+0000","last_active":"2023-10-25T00:23:44.352105+0000","last_peered":"2023-10-25T00:23:44.352105+0000","last_clean":"2023-10-25T00:23:44.352105+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-25T00:23:44.352105+0000","last_undegraded":"2023-10-25T00:23:44.352105+0000","last_fullsized":"2023-10-25T00:23:44.352105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3753","last_scrub_stamp":"2023-10-25T00:23:44.352083+0000","last_deep_scrub":"72'3753","last_deep_scrub_stamp":"2023-10-25T00:23:44.352083+0000","last_clean_scrub_stamp":"2023-10-25T00:23:44.352083+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:13:26.541612+0000","scrub_duration":0.328578174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":21045,"num_write_kb":276986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'4101","reported_seq":4187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:25.811837+0000","last_change":"2023-10-25T00:23:25.811837+0000","last_active":"2023-10-25T00:23:25.811837+0000","last_peered":"2023-10-25T00:23:25.811837+0000","last_clean":"2023-10-25T00:23:25.811837+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-25T00:23:25.811837+0000","last_undegraded":"2023-10-25T00:23:25.811837+0000","last_fullsized":"2023-10-25T00:23:25.811837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:23:25.811682+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:23:25.811682+0000","last_clean_scrub_stamp":"2023-10-25T00:23:25.811682+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:39:30.752063+0000","scrub_duration":0.14318265299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22383,"num_write_kb":270023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"73'13","reported_seq":101,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:17.651156+0000","last_change":"2023-10-25T00:23:17.651450+0000","last_active":"2023-10-25T00:23:17.651156+0000","last_peered":"2023-10-25T00:23:17.651156+0000","last_clean":"2023-10-25T00:23:17.651156+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-25T00:23:17.651156+0000","last_undegraded":"2023-10-25T00:23:17.651156+0000","last_fullsized":"2023-10-25T00:23:17.651156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2023-10-25T00:23:17.651103+0000","last_deep_scrub":"73'13","last_deep_scrub_stamp":"2023-10-25T00:23:17.651103+0000","last_clean_scrub_stamp":"2023-10-25T00:23:17.651103+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:54.986553+0000","scrub_duration":0.014684019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:51.317510+0000","last_change":"2023-10-25T00:22:51.317813+0000","last_active":"2023-10-25T00:22:51.317510+0000","last_peered":"2023-10-25T00:22:51.317510+0000","last_clean":"2023-10-25T00:22:51.317510+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-25T00:22:51.317510+0000","last_undegraded":"2023-10-25T00:22:51.317510+0000","last_fullsized":"2023-10-25T00:22:51.317510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.317462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:03:14.150576+0000","scrub_duration":0.0044508079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"73'249","reported_seq":354,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.397705+0000","last_change":"2023-10-25T00:23:21.397705+0000","last_active":"2023-10-25T00:23:21.397705+0000","last_peered":"2023-10-25T00:23:21.397705+0000","last_clean":"2023-10-25T00:23:21.397705+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-25T00:23:21.397705+0000","last_undegraded":"2023-10-25T00:23:21.397705+0000","last_fullsized":"2023-10-25T00:23:21.397705+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'249","last_scrub_stamp":"2023-10-25T00:23:21.397410+0000","last_deep_scrub":"73'249","last_deep_scrub_stamp":"2023-10-25T00:23:21.397410+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.397410+0000","objects_scrubbed":1,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:19:14.548554+0000","scrub_duration":0.017693341000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":62,"num_write":607,"num_write_kb":223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"72'5236","reported_seq":5324,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.899914+0000","last_change":"2023-10-25T00:23:34.900038+0000","last_active":"2023-10-25T00:23:34.899914+0000","last_peered":"2023-10-25T00:23:34.899914+0000","last_clean":"2023-10-25T00:23:34.899914+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-25T00:23:34.899914+0000","last_undegraded":"2023-10-25T00:23:34.899914+0000","last_fullsized":"2023-10-25T00:23:34.899914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5236","last_scrub_stamp":"2023-10-25T00:23:34.899894+0000","last_deep_scrub":"72'5236","last_deep_scrub_stamp":"2023-10-25T00:23:34.899894+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.899894+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:10:20.497931+0000","scrub_duration":0.29917875999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":29554,"num_write_kb":368820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'5444","reported_seq":5534,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.141618+0000","last_change":"2023-10-25T00:23:00.141618+0000","last_active":"2023-10-25T00:23:00.141618+0000","last_peered":"2023-10-25T00:23:00.141618+0000","last_clean":"2023-10-25T00:23:00.141618+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-25T00:23:00.141618+0000","last_undegraded":"2023-10-25T00:23:00.141618+0000","last_fullsized":"2023-10-25T00:23:00.141618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5444","last_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_deep_scrub":"68'5444","last_deep_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.141458+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:23:18.614380+0000","scrub_duration":0.197741424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28342,"num_write_kb":342931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"73'67","reported_seq":238,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.941625+0000","last_change":"2023-10-25T00:23:14.941868+0000","last_active":"2023-10-25T00:23:14.941625+0000","last_peered":"2023-10-25T00:23:14.941625+0000","last_clean":"2023-10-25T00:23:14.941625+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-25T00:23:14.941625+0000","last_undegraded":"2023-10-25T00:23:14.941625+0000","last_fullsized":"2023-10-25T00:23:14.941625+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'67","last_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_deep_scrub":"73'67","last_deep_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.941564+0000","objects_scrubbed":1,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:11:54.084201+0000","scrub_duration":0.0084463549999999991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":125,"num_write":61,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:22.074139+0000","last_change":"2023-10-25T00:23:22.074393+0000","last_active":"2023-10-25T00:23:22.074139+0000","last_peered":"2023-10-25T00:23:22.074139+0000","last_clean":"2023-10-25T00:23:22.074139+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-25T00:23:22.074139+0000","last_undegraded":"2023-10-25T00:23:22.074139+0000","last_fullsized":"2023-10-25T00:23:22.074139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:22.074087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:22.074087+0000","last_clean_scrub_stamp":"2023-10-25T00:23:22.074087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:59:48.682336+0000","scrub_duration":0.0044663139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"72'10","reported_seq":138,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.225873+0000","last_change":"2023-10-25T00:22:45.425308+0000","last_active":"2023-10-25T00:22:49.225873+0000","last_peered":"2023-10-25T00:22:49.225873+0000","last_clean":"2023-10-25T00:22:49.225873+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-25T00:22:49.225873+0000","last_undegraded":"2023-10-25T00:22:49.225873+0000","last_fullsized":"2023-10-25T00:22:49.225873+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.425052+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:36:38.527058+0000","scrub_duration":0.14014134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":135,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"72'3666","reported_seq":3749,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.807801+0000","last_change":"2023-10-25T00:23:45.807972+0000","last_active":"2023-10-25T00:23:45.807801+0000","last_peered":"2023-10-25T00:23:45.807801+0000","last_clean":"2023-10-25T00:23:45.807801+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-25T00:23:45.807801+0000","last_undegraded":"2023-10-25T00:23:45.807801+0000","last_fullsized":"2023-10-25T00:23:45.807801+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3666","last_scrub_stamp":"2023-10-25T00:23:45.807770+0000","last_deep_scrub":"72'3666","last_deep_scrub_stamp":"2023-10-25T00:23:45.807770+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.807770+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:26:55.156976+0000","scrub_duration":0.28592742900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20519,"num_write_kb":254391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"68'4182","reported_seq":4269,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.489530+0000","last_change":"2023-10-25T00:23:02.489861+0000","last_active":"2023-10-25T00:23:02.489530+0000","last_peered":"2023-10-25T00:23:02.489530+0000","last_clean":"2023-10-25T00:23:02.489530+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-25T00:23:02.489530+0000","last_undegraded":"2023-10-25T00:23:02.489530+0000","last_fullsized":"2023-10-25T00:23:02.489530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4182","last_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_deep_scrub":"68'4182","last_deep_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.489471+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:14:08.666419+0000","scrub_duration":0.092197608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22839,"num_write_kb":259586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'4","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:15.544038+0000","last_change":"2023-10-25T00:23:15.544275+0000","last_active":"2023-10-25T00:23:15.544038+0000","last_peered":"2023-10-25T00:23:15.544038+0000","last_clean":"2023-10-25T00:23:15.544038+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-25T00:23:15.544038+0000","last_undegraded":"2023-10-25T00:23:15.544038+0000","last_fullsized":"2023-10-25T00:23:15.544038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_deep_scrub":"67'4","last_deep_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:15.543988+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:33:51.332032+0000","scrub_duration":0.004459345,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:28.260459+0000","last_change":"2023-10-25T00:23:28.260731+0000","last_active":"2023-10-25T00:23:28.260459+0000","last_peered":"2023-10-25T00:23:28.260459+0000","last_clean":"2023-10-25T00:23:28.260459+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-25T00:23:28.260459+0000","last_undegraded":"2023-10-25T00:23:28.260459+0000","last_fullsized":"2023-10-25T00:23:28.260459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:28.260409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:28.260409+0000","last_clean_scrub_stamp":"2023-10-25T00:23:28.260409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:10:44.862291+0000","scrub_duration":0.0042298140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'163","reported_seq":299,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:17.149545+0000","last_change":"2023-10-25T00:23:17.149791+0000","last_active":"2023-10-25T00:23:17.149545+0000","last_peered":"2023-10-25T00:23:17.149545+0000","last_clean":"2023-10-25T00:23:17.149545+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-25T00:23:17.149545+0000","last_undegraded":"2023-10-25T00:23:17.149545+0000","last_fullsized":"2023-10-25T00:23:17.149545+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'163","last_scrub_stamp":"2023-10-25T00:23:17.149492+0000","last_deep_scrub":"73'163","last_deep_scrub_stamp":"2023-10-25T00:23:17.149492+0000","last_clean_scrub_stamp":"2023-10-25T00:23:17.149492+0000","objects_scrubbed":2,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:59:05.394868+0000","scrub_duration":0.012985958000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"72'2","reported_seq":85,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:48.067150+0000","last_change":"2023-10-25T00:22:48.067150+0000","last_active":"2023-10-25T00:22:48.067150+0000","last_peered":"2023-10-25T00:22:48.067150+0000","last_clean":"2023-10-25T00:22:48.067150+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-25T00:22:48.067150+0000","last_undegraded":"2023-10-25T00:22:48.067150+0000","last_fullsized":"2023-10-25T00:22:48.067150+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.066810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:55:20.112674+0000","scrub_duration":0.0050122709999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"72'4105","reported_seq":4191,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.535400+0000","last_change":"2023-10-25T00:23:20.535530+0000","last_active":"2023-10-25T00:23:20.535400+0000","last_peered":"2023-10-25T00:23:20.535400+0000","last_clean":"2023-10-25T00:23:20.535400+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-25T00:23:20.535400+0000","last_undegraded":"2023-10-25T00:23:20.535400+0000","last_fullsized":"2023-10-25T00:23:20.535400+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4105","last_scrub_stamp":"2023-10-25T00:23:20.535378+0000","last_deep_scrub":"72'4105","last_deep_scrub_stamp":"2023-10-25T00:23:20.535378+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.535378+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:20:52.287445+0000","scrub_duration":0.31792611900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":19,"num_write":23147,"num_write_kb":297959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'5705","reported_seq":5793,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:39.836227+0000","last_change":"2023-10-25T00:23:39.836379+0000","last_active":"2023-10-25T00:23:39.836227+0000","last_peered":"2023-10-25T00:23:39.836227+0000","last_clean":"2023-10-25T00:23:39.836227+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-25T00:23:39.836227+0000","last_undegraded":"2023-10-25T00:23:39.836227+0000","last_fullsized":"2023-10-25T00:23:39.836227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5705","last_scrub_stamp":"2023-10-25T00:23:39.836205+0000","last_deep_scrub":"68'5705","last_deep_scrub_stamp":"2023-10-25T00:23:39.836205+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.836205+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:38:02.740862+0000","scrub_duration":0.105332388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":29845,"num_write_kb":357286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"68'5","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:18.535332+0000","last_change":"2023-10-25T00:23:18.535332+0000","last_active":"2023-10-25T00:23:18.535332+0000","last_peered":"2023-10-25T00:23:18.535332+0000","last_clean":"2023-10-25T00:23:18.535332+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-25T00:23:18.535332+0000","last_undegraded":"2023-10-25T00:23:18.535332+0000","last_fullsized":"2023-10-25T00:23:18.535332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:18.534951+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:18.534951+0000","last_clean_scrub_stamp":"2023-10-25T00:23:18.534951+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:37.756970+0000","scrub_duration":0.0037857049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"72'7","reported_seq":125,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.237196+0000","last_change":"2023-10-25T00:23:51.873544+0000","last_active":"2023-10-25T00:22:49.237196+0000","last_peered":"2023-10-25T00:22:49.237196+0000","last_clean":"2023-10-25T00:22:49.237196+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-25T00:22:49.237196+0000","last_undegraded":"2023-10-25T00:22:49.237196+0000","last_fullsized":"2023-10-25T00:22:49.237196+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"72'4234","reported_seq":4319,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:41.751577+0000","last_change":"2023-10-25T00:23:41.751679+0000","last_active":"2023-10-25T00:23:41.751577+0000","last_peered":"2023-10-25T00:23:41.751577+0000","last_clean":"2023-10-25T00:23:41.751577+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-25T00:23:41.751577+0000","last_undegraded":"2023-10-25T00:23:41.751577+0000","last_fullsized":"2023-10-25T00:23:41.751577+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4234","last_scrub_stamp":"2023-10-25T00:23:41.751558+0000","last_deep_scrub":"72'4234","last_deep_scrub_stamp":"2023-10-25T00:23:41.751558+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.751558+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:50:10.142078+0000","scrub_duration":0.256161054,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":15,"num_write":23173,"num_write_kb":302022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"68'4052","reported_seq":4138,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:49.042550+0000","last_change":"2023-10-25T00:23:49.042663+0000","last_active":"2023-10-25T00:23:49.042550+0000","last_peered":"2023-10-25T00:23:49.042550+0000","last_clean":"2023-10-25T00:23:49.042550+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-25T00:23:49.042550+0000","last_undegraded":"2023-10-25T00:23:49.042550+0000","last_fullsized":"2023-10-25T00:23:49.042550+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4052","last_scrub_stamp":"2023-10-25T00:23:49.042529+0000","last_deep_scrub":"68'4052","last_deep_scrub_stamp":"2023-10-25T00:23:49.042529+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.042529+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:40:32.211760+0000","scrub_duration":0.105715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21486,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"68'2","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.046955+0000","last_change":"2023-10-25T00:23:34.047108+0000","last_active":"2023-10-25T00:23:34.046955+0000","last_peered":"2023-10-25T00:23:34.046955+0000","last_clean":"2023-10-25T00:23:34.046955+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-25T00:23:34.046955+0000","last_undegraded":"2023-10-25T00:23:34.046955+0000","last_fullsized":"2023-10-25T00:23:34.046955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2023-10-25T00:23:34.046917+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2023-10-25T00:23:34.046917+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.046917+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:50:34.912538+0000","scrub_duration":0.034858717999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"73'5","reported_seq":109,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:53.546236+0000","last_change":"2023-10-25T00:22:53.546332+0000","last_active":"2023-10-25T00:22:53.546236+0000","last_peered":"2023-10-25T00:22:53.546236+0000","last_clean":"2023-10-25T00:22:53.546236+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-25T00:22:53.546236+0000","last_undegraded":"2023-10-25T00:22:53.546236+0000","last_fullsized":"2023-10-25T00:22:53.546236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.546215+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:16:55.756192+0000","scrub_duration":0.27862564699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"72'4690","reported_seq":4775,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:42.556562+0000","last_change":"2023-10-25T00:23:42.556703+0000","last_active":"2023-10-25T00:23:42.556562+0000","last_peered":"2023-10-25T00:23:42.556562+0000","last_clean":"2023-10-25T00:23:42.556562+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-25T00:23:42.556562+0000","last_undegraded":"2023-10-25T00:23:42.556562+0000","last_fullsized":"2023-10-25T00:23:42.556562+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4690","last_scrub_stamp":"2023-10-25T00:23:42.556538+0000","last_deep_scrub":"72'4690","last_deep_scrub_stamp":"2023-10-25T00:23:42.556538+0000","last_clean_scrub_stamp":"2023-10-25T00:23:42.556538+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:36:54.839713+0000","scrub_duration":0.199820896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24994,"num_write_kb":341798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'4780","reported_seq":4868,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:09.092847+0000","last_change":"2023-10-25T00:23:09.093139+0000","last_active":"2023-10-25T00:23:09.092847+0000","last_peered":"2023-10-25T00:23:09.092847+0000","last_clean":"2023-10-25T00:23:09.092847+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-25T00:23:09.092847+0000","last_undegraded":"2023-10-25T00:23:09.092847+0000","last_fullsized":"2023-10-25T00:23:09.092847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4780","last_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_deep_scrub":"68'4780","last_deep_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.092796+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:58:49.347709+0000","scrub_duration":0.152359684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25083,"num_write_kb":293163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'3","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.218767+0000","last_change":"2023-10-25T00:22:48.328017+0000","last_active":"2023-10-25T00:22:49.218767+0000","last_peered":"2023-10-25T00:22:49.218767+0000","last_clean":"2023-10-25T00:22:49.218767+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-25T00:22:49.218767+0000","last_undegraded":"2023-10-25T00:22:49.218767+0000","last_fullsized":"2023-10-25T00:22:49.218767+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.327778+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:58.116941+0000","scrub_duration":0.061328572999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"72'5","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.576377+0000","last_change":"2023-10-25T00:22:55.576688+0000","last_active":"2023-10-25T00:22:55.576377+0000","last_peered":"2023-10-25T00:22:55.576377+0000","last_clean":"2023-10-25T00:22:55.576377+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-25T00:22:55.576377+0000","last_undegraded":"2023-10-25T00:22:55.576377+0000","last_fullsized":"2023-10-25T00:22:55.576377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.576324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:43.752897+0000","scrub_duration":0.0045481469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"72'3596","reported_seq":3671,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060821+0000","last_change":"2023-10-24T23:53:11.816959+0000","last_active":"2023-10-25T00:22:36.060821+0000","last_peered":"2023-10-25T00:22:36.060821+0000","last_clean":"2023-10-25T00:22:36.060821+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-25T00:22:36.060821+0000","last_undegraded":"2023-10-25T00:22:36.060821+0000","last_fullsized":"2023-10-25T00:22:36.060821+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":19820,"num_write_kb":259784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"68'4970","reported_seq":5057,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.999669+0000","last_change":"2023-10-25T00:22:55.999798+0000","last_active":"2023-10-25T00:22:55.999669+0000","last_peered":"2023-10-25T00:22:55.999669+0000","last_clean":"2023-10-25T00:22:55.999669+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-25T00:22:55.999669+0000","last_undegraded":"2023-10-25T00:22:55.999669+0000","last_fullsized":"2023-10-25T00:22:55.999669+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4970","last_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_deep_scrub":"68'4970","last_deep_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.999644+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:15:47.681358+0000","scrub_duration":0.177095113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26721,"num_write_kb":311966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"73'1466","reported_seq":1605,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.466683+0000","last_change":"2023-10-25T00:23:06.467052+0000","last_active":"2023-10-25T00:23:06.466683+0000","last_peered":"2023-10-25T00:23:06.466683+0000","last_clean":"2023-10-25T00:23:06.466683+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-25T00:23:06.466683+0000","last_undegraded":"2023-10-25T00:23:06.466683+0000","last_fullsized":"2023-10-25T00:23:06.466683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1466","last_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_deep_scrub":"73'1466","last_deep_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.466569+0000","objects_scrubbed":2,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:21.696239+0000","scrub_duration":0.039946036999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":47798,"num_write":1439,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'7","reported_seq":112,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:53.616510+0000","last_change":"2023-10-25T00:23:53.616774+0000","last_active":"2023-10-25T00:23:53.616510+0000","last_peered":"2023-10-25T00:23:53.616510+0000","last_clean":"2023-10-25T00:23:53.616510+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-25T00:23:53.616510+0000","last_undegraded":"2023-10-25T00:23:53.616510+0000","last_fullsized":"2023-10-25T00:23:53.616510+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'7","last_scrub_stamp":"2023-10-25T00:23:53.616451+0000","last_deep_scrub":"73'7","last_deep_scrub_stamp":"2023-10-25T00:23:53.616451+0000","last_clean_scrub_stamp":"2023-10-25T00:23:53.616451+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:31:57.298978+0000","scrub_duration":0.13500319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"72'3548","reported_seq":3634,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:32.093782+0000","last_change":"2023-10-25T00:23:32.093782+0000","last_active":"2023-10-25T00:23:32.093782+0000","last_peered":"2023-10-25T00:23:32.093782+0000","last_clean":"2023-10-25T00:23:32.093782+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-25T00:23:32.093782+0000","last_undegraded":"2023-10-25T00:23:32.093782+0000","last_fullsized":"2023-10-25T00:23:32.093782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3548","last_scrub_stamp":"2023-10-25T00:23:32.093615+0000","last_deep_scrub":"72'3548","last_deep_scrub_stamp":"2023-10-25T00:23:32.093615+0000","last_clean_scrub_stamp":"2023-10-25T00:23:32.093615+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:02:03.851334+0000","scrub_duration":0.28610911100000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20243,"num_write_kb":261539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"68'3393","reported_seq":3482,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.098602+0000","last_change":"2023-10-25T00:22:42.098602+0000","last_active":"2023-10-25T00:22:42.098602+0000","last_peered":"2023-10-25T00:22:42.098602+0000","last_clean":"2023-10-25T00:22:42.098602+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-25T00:22:42.098602+0000","last_undegraded":"2023-10-25T00:22:42.098602+0000","last_fullsized":"2023-10-25T00:22:42.098602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3393","last_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_deep_scrub":"68'3393","last_deep_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.098371+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:12.736136+0000","scrub_duration":0.153676797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17500,"num_write_kb":217638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"68'5","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:33.402669+0000","last_change":"2023-10-25T00:23:33.402936+0000","last_active":"2023-10-25T00:23:33.402669+0000","last_peered":"2023-10-25T00:23:33.402669+0000","last_clean":"2023-10-25T00:23:33.402669+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-25T00:23:33.402669+0000","last_undegraded":"2023-10-25T00:23:33.402669+0000","last_fullsized":"2023-10-25T00:23:33.402669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:33.402617+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:33.402617+0000","last_clean_scrub_stamp":"2023-10-25T00:23:33.402617+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:10:14.715542+0000","scrub_duration":0.0048941419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"72'3","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.419583+0000","last_change":"2023-10-25T00:23:45.419862+0000","last_active":"2023-10-25T00:23:45.419583+0000","last_peered":"2023-10-25T00:23:45.419583+0000","last_clean":"2023-10-25T00:23:45.419583+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-25T00:23:45.419583+0000","last_undegraded":"2023-10-25T00:23:45.419583+0000","last_fullsized":"2023-10-25T00:23:45.419583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:45.419528+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:45.419528+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.419528+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:07:59.973530+0000","scrub_duration":0.0045258490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"72'3584","reported_seq":3670,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.285501+0000","last_change":"2023-10-25T00:23:40.285625+0000","last_active":"2023-10-25T00:23:40.285501+0000","last_peered":"2023-10-25T00:23:40.285501+0000","last_clean":"2023-10-25T00:23:40.285501+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-25T00:23:40.285501+0000","last_undegraded":"2023-10-25T00:23:40.285501+0000","last_fullsized":"2023-10-25T00:23:40.285501+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3584","last_scrub_stamp":"2023-10-25T00:23:40.285478+0000","last_deep_scrub":"72'3584","last_deep_scrub_stamp":"2023-10-25T00:23:40.285478+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.285478+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:11:39.447639+0000","scrub_duration":0.26816689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20504,"num_write_kb":254394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'5094","reported_seq":5186,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:51.485282+0000","last_change":"2023-10-25T00:22:51.485282+0000","last_active":"2023-10-25T00:22:51.485282+0000","last_peered":"2023-10-25T00:22:51.485282+0000","last_clean":"2023-10-25T00:22:51.485282+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-25T00:22:51.485282+0000","last_undegraded":"2023-10-25T00:22:51.485282+0000","last_fullsized":"2023-10-25T00:22:51.485282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5094","last_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_deep_scrub":"68'5094","last_deep_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.484917+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:22:43.899134+0000","scrub_duration":0.19299439099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27763,"num_write_kb":328490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"73'7","reported_seq":116,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.094834+0000","last_change":"2023-10-25T00:23:27.095106+0000","last_active":"2023-10-25T00:23:27.094834+0000","last_peered":"2023-10-25T00:23:27.094834+0000","last_clean":"2023-10-25T00:23:27.094834+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-25T00:23:27.094834+0000","last_undegraded":"2023-10-25T00:23:27.094834+0000","last_fullsized":"2023-10-25T00:23:27.094834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'7","last_scrub_stamp":"2023-10-25T00:23:27.094773+0000","last_deep_scrub":"73'7","last_deep_scrub_stamp":"2023-10-25T00:23:27.094773+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.094773+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:52:00.838368+0000","scrub_duration":0.007318275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"73'1440","reported_seq":1570,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.527480+0000","last_change":"2023-10-25T00:23:16.527658+0000","last_active":"2023-10-25T00:23:16.527480+0000","last_peered":"2023-10-25T00:23:16.527480+0000","last_clean":"2023-10-25T00:23:16.527480+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-25T00:23:16.527480+0000","last_undegraded":"2023-10-25T00:23:16.527480+0000","last_fullsized":"2023-10-25T00:23:16.527480+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1440","last_scrub_stamp":"2023-10-25T00:23:16.527441+0000","last_deep_scrub":"73'1440","last_deep_scrub_stamp":"2023-10-25T00:23:16.527441+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.527441+0000","objects_scrubbed":2,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:47:53.223550+0000","scrub_duration":0.234060301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":35275,"num_write":1412,"num_write_kb":4487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"72'3292","reported_seq":3378,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.971739+0000","last_change":"2023-10-25T00:23:30.971878+0000","last_active":"2023-10-25T00:23:30.971739+0000","last_peered":"2023-10-25T00:23:30.971739+0000","last_clean":"2023-10-25T00:23:30.971739+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-25T00:23:30.971739+0000","last_undegraded":"2023-10-25T00:23:30.971739+0000","last_fullsized":"2023-10-25T00:23:30.971739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3292","last_scrub_stamp":"2023-10-25T00:23:30.971718+0000","last_deep_scrub":"72'3292","last_deep_scrub_stamp":"2023-10-25T00:23:30.971718+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.971718+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:45:07.787712+0000","scrub_duration":0.347700487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17853,"num_write_kb":243602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'4242","reported_seq":4329,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232419+0000","last_change":"2023-10-25T00:22:45.893397+0000","last_active":"2023-10-25T00:22:49.232419+0000","last_peered":"2023-10-25T00:22:49.232419+0000","last_clean":"2023-10-25T00:22:49.232419+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-25T00:22:49.232419+0000","last_undegraded":"2023-10-25T00:22:49.232419+0000","last_fullsized":"2023-10-25T00:22:49.232419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4242","last_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_deep_scrub":"67'4242","last_deep_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.893150+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:24:28.749675+0000","scrub_duration":0.21189509300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22687,"num_write_kb":275346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"73'8","reported_seq":140,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.063186+0000","last_change":"2023-10-25T00:23:30.063493+0000","last_active":"2023-10-25T00:23:30.063186+0000","last_peered":"2023-10-25T00:23:30.063186+0000","last_clean":"2023-10-25T00:23:30.063186+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-25T00:23:30.063186+0000","last_undegraded":"2023-10-25T00:23:30.063186+0000","last_fullsized":"2023-10-25T00:23:30.063186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2023-10-25T00:23:30.063132+0000","last_deep_scrub":"73'8","last_deep_scrub_stamp":"2023-10-25T00:23:30.063132+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.063132+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:12:23.891399+0000","scrub_duration":0.0076496660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"72'5","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:19.320206+0000","last_change":"2023-10-25T00:23:19.320206+0000","last_active":"2023-10-25T00:23:19.320206+0000","last_peered":"2023-10-25T00:23:19.320206+0000","last_clean":"2023-10-25T00:23:19.320206+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-25T00:23:19.320206+0000","last_undegraded":"2023-10-25T00:23:19.320206+0000","last_fullsized":"2023-10-25T00:23:19.320206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:23:19.319840+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:23:19.319840+0000","last_clean_scrub_stamp":"2023-10-25T00:23:19.319840+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:18:41.713579+0000","scrub_duration":0.0042872409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"72'4091","reported_seq":4177,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:50.884272+0000","last_change":"2023-10-25T00:23:50.884393+0000","last_active":"2023-10-25T00:23:50.884272+0000","last_peered":"2023-10-25T00:23:50.884272+0000","last_clean":"2023-10-25T00:23:50.884272+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-25T00:23:50.884272+0000","last_undegraded":"2023-10-25T00:23:50.884272+0000","last_fullsized":"2023-10-25T00:23:50.884272+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4091","last_scrub_stamp":"2023-10-25T00:23:50.884251+0000","last_deep_scrub":"72'4091","last_deep_scrub_stamp":"2023-10-25T00:23:50.884251+0000","last_clean_scrub_stamp":"2023-10-25T00:23:50.884251+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:21:51.075072+0000","scrub_duration":0.36544204200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22779,"num_write_kb":288098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'4310","reported_seq":4397,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:22.394323+0000","last_change":"2023-10-25T00:23:22.394323+0000","last_active":"2023-10-25T00:23:22.394323+0000","last_peered":"2023-10-25T00:23:22.394323+0000","last_clean":"2023-10-25T00:23:22.394323+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-25T00:23:22.394323+0000","last_undegraded":"2023-10-25T00:23:22.394323+0000","last_fullsized":"2023-10-25T00:23:22.394323+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4310","last_scrub_stamp":"2023-10-25T00:23:22.393940+0000","last_deep_scrub":"67'4310","last_deep_scrub_stamp":"2023-10-25T00:23:22.393940+0000","last_clean_scrub_stamp":"2023-10-25T00:23:22.393940+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:24:46.673127+0000","scrub_duration":0.085301248999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23791,"num_write_kb":283914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"73'5","reported_seq":114,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.183468+0000","last_change":"2023-10-25T00:23:26.183662+0000","last_active":"2023-10-25T00:23:26.183468+0000","last_peered":"2023-10-25T00:23:26.183468+0000","last_clean":"2023-10-25T00:23:26.183468+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-25T00:23:26.183468+0000","last_undegraded":"2023-10-25T00:23:26.183468+0000","last_fullsized":"2023-10-25T00:23:26.183468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:23:26.183430+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:23:26.183430+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.183430+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:28:22.710805+0000","scrub_duration":0.110908062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"72'2","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.206968+0000","last_change":"2023-10-25T00:22:38.034628+0000","last_active":"2023-10-25T00:22:49.206968+0000","last_peered":"2023-10-25T00:22:49.206968+0000","last_clean":"2023-10-25T00:22:49.206968+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-25T00:22:49.206968+0000","last_undegraded":"2023-10-25T00:22:49.206968+0000","last_fullsized":"2023-10-25T00:22:49.206968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.034499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:09.450121+0000","scrub_duration":0.14675892500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"72'3695","reported_seq":3781,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:51.251436+0000","last_change":"2023-10-25T00:23:51.251559+0000","last_active":"2023-10-25T00:23:51.251436+0000","last_peered":"2023-10-25T00:23:51.251436+0000","last_clean":"2023-10-25T00:23:51.251436+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-25T00:23:51.251436+0000","last_undegraded":"2023-10-25T00:23:51.251436+0000","last_fullsized":"2023-10-25T00:23:51.251436+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3695","last_scrub_stamp":"2023-10-25T00:23:51.251416+0000","last_deep_scrub":"72'3695","last_deep_scrub_stamp":"2023-10-25T00:23:51.251416+0000","last_clean_scrub_stamp":"2023-10-25T00:23:51.251416+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:44:06.748969+0000","scrub_duration":0.274463921,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21426,"num_write_kb":254037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"72'3","reported_seq":84,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.331112+0000","last_change":"2023-10-25T00:22:42.331112+0000","last_active":"2023-10-25T00:22:42.331112+0000","last_peered":"2023-10-25T00:22:42.331112+0000","last_clean":"2023-10-25T00:22:42.331112+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-25T00:22:42.331112+0000","last_undegraded":"2023-10-25T00:22:42.331112+0000","last_fullsized":"2023-10-25T00:22:42.331112+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.330786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:23:56.573805+0000","scrub_duration":0.0043674530000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"72'19","reported_seq":218,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.645016+0000","last_change":"2023-10-25T00:22:47.253320+0000","last_active":"2023-10-25T00:22:52.645016+0000","last_peered":"2023-10-25T00:22:52.645016+0000","last_clean":"2023-10-25T00:22:52.645016+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-25T00:22:52.645016+0000","last_undegraded":"2023-10-25T00:22:52.645016+0000","last_fullsized":"2023-10-25T00:22:52.645016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_deep_scrub":"72'19","last_deep_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.252995+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:03.720456+0000","scrub_duration":0.018599613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":339,"num_read_kb":339,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'5249","reported_seq":5342,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.439199+0000","last_change":"2023-10-25T00:22:52.439461+0000","last_active":"2023-10-25T00:22:52.439199+0000","last_peered":"2023-10-25T00:22:52.439199+0000","last_clean":"2023-10-25T00:22:52.439199+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-25T00:22:52.439199+0000","last_undegraded":"2023-10-25T00:22:52.439199+0000","last_fullsized":"2023-10-25T00:22:52.439199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5249","last_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_deep_scrub":"68'5249","last_deep_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.439150+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:19:03.730037+0000","scrub_duration":0.077069795999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":27646,"num_write_kb":332285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"72'4380","reported_seq":4464,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:38.646492+0000","last_change":"2023-10-25T00:23:38.646626+0000","last_active":"2023-10-25T00:23:38.646492+0000","last_peered":"2023-10-25T00:23:38.646492+0000","last_clean":"2023-10-25T00:23:38.646492+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-25T00:23:38.646492+0000","last_undegraded":"2023-10-25T00:23:38.646492+0000","last_fullsized":"2023-10-25T00:23:38.646492+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4380","last_scrub_stamp":"2023-10-25T00:23:38.646472+0000","last_deep_scrub":"72'4380","last_deep_scrub_stamp":"2023-10-25T00:23:38.646472+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.646472+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:38:56.172654+0000","scrub_duration":0.29662724200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23662,"num_write_kb":324338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"72'5","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:50.617951+0000","last_change":"2023-10-25T00:22:50.618067+0000","last_active":"2023-10-25T00:22:50.617951+0000","last_peered":"2023-10-25T00:22:50.617951+0000","last_clean":"2023-10-25T00:22:50.617951+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-25T00:22:50.617951+0000","last_undegraded":"2023-10-25T00:22:50.617951+0000","last_fullsized":"2023-10-25T00:22:50.617951+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.617931+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:24.381368+0000","scrub_duration":0.234626104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"68'5","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:25.067489+0000","last_change":"2023-10-25T00:23:25.067489+0000","last_active":"2023-10-25T00:23:25.067489+0000","last_peered":"2023-10-25T00:23:25.067489+0000","last_clean":"2023-10-25T00:23:25.067489+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-25T00:23:25.067489+0000","last_undegraded":"2023-10-25T00:23:25.067489+0000","last_fullsized":"2023-10-25T00:23:25.067489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:25.067146+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:25.067146+0000","last_clean_scrub_stamp":"2023-10-25T00:23:25.067146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:32.009959+0000","scrub_duration":0.0041696520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'4442","reported_seq":4527,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:29.981270+0000","last_change":"2023-10-25T00:23:29.981423+0000","last_active":"2023-10-25T00:23:29.981270+0000","last_peered":"2023-10-25T00:23:29.981270+0000","last_clean":"2023-10-25T00:23:29.981270+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-25T00:23:29.981270+0000","last_undegraded":"2023-10-25T00:23:29.981270+0000","last_fullsized":"2023-10-25T00:23:29.981270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4442","last_scrub_stamp":"2023-10-25T00:23:29.981245+0000","last_deep_scrub":"68'4442","last_deep_scrub_stamp":"2023-10-25T00:23:29.981245+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.981245+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:11:38.194434+0000","scrub_duration":0.109448791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22810,"num_write_kb":296225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"72'4076","reported_seq":4150,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060841+0000","last_change":"2023-10-24T23:53:11.834049+0000","last_active":"2023-10-25T00:22:36.060841+0000","last_peered":"2023-10-25T00:22:36.060841+0000","last_clean":"2023-10-25T00:22:36.060841+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-25T00:22:36.060841+0000","last_undegraded":"2023-10-25T00:22:36.060841+0000","last_fullsized":"2023-10-25T00:22:36.060841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22055,"num_write_kb":297609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"68'3","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:23.928686+0000","last_change":"2023-10-25T00:23:23.928686+0000","last_active":"2023-10-25T00:23:23.928686+0000","last_peered":"2023-10-25T00:23:23.928686+0000","last_clean":"2023-10-25T00:23:23.928686+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-25T00:23:23.928686+0000","last_undegraded":"2023-10-25T00:23:23.928686+0000","last_fullsized":"2023-10-25T00:23:23.928686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:23.928352+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:23.928352+0000","last_clean_scrub_stamp":"2023-10-25T00:23:23.928352+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:41:04.304281+0000","scrub_duration":0.0043703589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'4101","reported_seq":4187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:50.337590+0000","last_change":"2023-10-25T00:22:50.337815+0000","last_active":"2023-10-25T00:22:50.337590+0000","last_peered":"2023-10-25T00:22:50.337590+0000","last_clean":"2023-10-25T00:22:50.337590+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-25T00:22:50.337590+0000","last_undegraded":"2023-10-25T00:22:50.337590+0000","last_fullsized":"2023-10-25T00:22:50.337590+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.337536+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:01.275651+0000","scrub_duration":0.055303021000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22343,"num_write_kb":253486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"72'397","reported_seq":591,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234775+0000","last_change":"2023-10-25T00:22:45.554898+0000","last_active":"2023-10-25T00:22:49.234775+0000","last_peered":"2023-10-25T00:22:49.234775+0000","last_clean":"2023-10-25T00:22:49.234775+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-25T00:22:49.234775+0000","last_undegraded":"2023-10-25T00:22:49.234775+0000","last_fullsized":"2023-10-25T00:22:49.234775+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'397","last_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_deep_scrub":"72'397","last_deep_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.554526+0000","objects_scrubbed":4,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:41:32.056818+0000","scrub_duration":0.10248053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":86197,"num_write":389,"num_write_kb":4306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"72'4606","reported_seq":4689,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.057352+0000","last_change":"2023-10-25T00:23:37.057459+0000","last_active":"2023-10-25T00:23:37.057352+0000","last_peered":"2023-10-25T00:23:37.057352+0000","last_clean":"2023-10-25T00:23:37.057352+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-25T00:23:37.057352+0000","last_undegraded":"2023-10-25T00:23:37.057352+0000","last_fullsized":"2023-10-25T00:23:37.057352+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4606","last_scrub_stamp":"2023-10-25T00:23:37.057332+0000","last_deep_scrub":"72'4606","last_deep_scrub_stamp":"2023-10-25T00:23:37.057332+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.057332+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:16:39.313888+0000","scrub_duration":0.28055820399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25763,"num_write_kb":316477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"73'14","reported_seq":146,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.153043+0000","last_change":"2023-10-25T00:23:16.153043+0000","last_active":"2023-10-25T00:23:16.153043+0000","last_peered":"2023-10-25T00:23:16.153043+0000","last_clean":"2023-10-25T00:23:16.153043+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-25T00:23:16.153043+0000","last_undegraded":"2023-10-25T00:23:16.153043+0000","last_fullsized":"2023-10-25T00:23:16.153043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_deep_scrub":"73'14","last_deep_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.152723+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:32:30.128331+0000","scrub_duration":0.010186763999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'5680","reported_seq":5770,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:54.586176+0000","last_change":"2023-10-25T00:22:54.586176+0000","last_active":"2023-10-25T00:22:54.586176+0000","last_peered":"2023-10-25T00:22:54.586176+0000","last_clean":"2023-10-25T00:22:54.586176+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-25T00:22:54.586176+0000","last_undegraded":"2023-10-25T00:22:54.586176+0000","last_fullsized":"2023-10-25T00:22:54.586176+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5680","last_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_deep_scrub":"68'5680","last_deep_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.585866+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:49.342364+0000","scrub_duration":0.20835251299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29749,"num_write_kb":360962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"73'40","reported_seq":187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:59.279339+0000","last_change":"2023-10-25T00:22:59.279527+0000","last_active":"2023-10-25T00:22:59.279339+0000","last_peered":"2023-10-25T00:22:59.279339+0000","last_clean":"2023-10-25T00:22:59.279339+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-25T00:22:59.279339+0000","last_undegraded":"2023-10-25T00:22:59.279339+0000","last_fullsized":"2023-10-25T00:22:59.279339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'40","last_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_deep_scrub":"73'40","last_deep_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.279288+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:52.911132+0000","scrub_duration":0.0094949800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":39,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"72'4808","reported_seq":4897,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.790951+0000","last_change":"2023-10-25T00:23:14.791099+0000","last_active":"2023-10-25T00:23:14.790951+0000","last_peered":"2023-10-25T00:23:14.790951+0000","last_clean":"2023-10-25T00:23:14.790951+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-25T00:23:14.790951+0000","last_undegraded":"2023-10-25T00:23:14.790951+0000","last_fullsized":"2023-10-25T00:23:14.790951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4808","last_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_deep_scrub":"72'4808","last_deep_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.790929+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:56:36.982903+0000","scrub_duration":0.29119922999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26552,"num_write_kb":336125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"68'4747","reported_seq":4835,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.496098+0000","last_change":"2023-10-25T00:23:00.496309+0000","last_active":"2023-10-25T00:23:00.496098+0000","last_peered":"2023-10-25T00:23:00.496098+0000","last_clean":"2023-10-25T00:23:00.496098+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-25T00:23:00.496098+0000","last_undegraded":"2023-10-25T00:23:00.496098+0000","last_fullsized":"2023-10-25T00:23:00.496098+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4747","last_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_deep_scrub":"68'4747","last_deep_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.496060+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:56.531125+0000","scrub_duration":0.089432630999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24847,"num_write_kb":316685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"72'4283","reported_seq":4370,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:08.728587+0000","last_change":"2023-10-25T00:23:08.728718+0000","last_active":"2023-10-25T00:23:08.728587+0000","last_peered":"2023-10-25T00:23:08.728587+0000","last_clean":"2023-10-25T00:23:08.728587+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-25T00:23:08.728587+0000","last_undegraded":"2023-10-25T00:23:08.728587+0000","last_fullsized":"2023-10-25T00:23:08.728587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4283","last_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_deep_scrub":"72'4283","last_deep_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_clean_scrub_stamp":"2023-10-25T00:23:08.728567+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:33.655445+0000","scrub_duration":0.26846236600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23853,"num_write_kb":296504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"68'4036","reported_seq":4124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:39.642537+0000","last_change":"2023-10-25T00:22:39.642537+0000","last_active":"2023-10-25T00:22:39.642537+0000","last_peered":"2023-10-25T00:22:39.642537+0000","last_clean":"2023-10-25T00:22:39.642537+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-25T00:22:39.642537+0000","last_undegraded":"2023-10-25T00:22:39.642537+0000","last_fullsized":"2023-10-25T00:22:39.642537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4036","last_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_deep_scrub":"68'4036","last_deep_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.642342+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:02.362976+0000","scrub_duration":0.227585228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21233,"num_write_kb":273467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"72'4276","reported_seq":4362,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.997835+0000","last_change":"2023-10-25T00:23:30.997946+0000","last_active":"2023-10-25T00:23:30.997835+0000","last_peered":"2023-10-25T00:23:30.997835+0000","last_clean":"2023-10-25T00:23:30.997835+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-25T00:23:30.997835+0000","last_undegraded":"2023-10-25T00:23:30.997835+0000","last_fullsized":"2023-10-25T00:23:30.997835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4276","last_scrub_stamp":"2023-10-25T00:23:30.997816+0000","last_deep_scrub":"72'4276","last_deep_scrub_stamp":"2023-10-25T00:23:30.997816+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.997816+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:08.863577+0000","scrub_duration":0.39360271099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23362,"num_write_kb":310817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"68'6201","reported_seq":6281,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.235390+0000","last_change":"2023-10-25T00:23:50.848970+0000","last_active":"2023-10-25T00:22:49.235390+0000","last_peered":"2023-10-25T00:22:49.235390+0000","last_clean":"2023-10-25T00:22:49.235390+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-25T00:22:49.235390+0000","last_undegraded":"2023-10-25T00:22:49.235390+0000","last_fullsized":"2023-10-25T00:22:49.235390+0000","mapping_epoch":21,"log_start":"62'1000","ondisk_log_start":"62'1000","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33486,"num_write_kb":384441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"72'4018","reported_seq":4105,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:52.269664+0000","last_change":"2023-10-25T00:23:52.269793+0000","last_active":"2023-10-25T00:23:52.269664+0000","last_peered":"2023-10-25T00:23:52.269664+0000","last_clean":"2023-10-25T00:23:52.269664+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-25T00:23:52.269664+0000","last_undegraded":"2023-10-25T00:23:52.269664+0000","last_fullsized":"2023-10-25T00:23:52.269664+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4018","last_scrub_stamp":"2023-10-25T00:23:52.269641+0000","last_deep_scrub":"72'4018","last_deep_scrub_stamp":"2023-10-25T00:23:52.269641+0000","last_clean_scrub_stamp":"2023-10-25T00:23:52.269641+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:58:28.387402+0000","scrub_duration":0.26794800299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":21920,"num_write_kb":288651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"68'4522","reported_seq":4611,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.751456+0000","last_change":"2023-10-25T00:23:26.751600+0000","last_active":"2023-10-25T00:23:26.751456+0000","last_peered":"2023-10-25T00:23:26.751456+0000","last_clean":"2023-10-25T00:23:26.751456+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-25T00:23:26.751456+0000","last_undegraded":"2023-10-25T00:23:26.751456+0000","last_fullsized":"2023-10-25T00:23:26.751456+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4522","last_scrub_stamp":"2023-10-25T00:23:26.751435+0000","last_deep_scrub":"68'4522","last_deep_scrub_stamp":"2023-10-25T00:23:26.751435+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.751435+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:40:36.338773+0000","scrub_duration":0.110684227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24476,"num_write_kb":282500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"72'4209","reported_seq":4293,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.054561+0000","last_change":"2023-10-25T00:23:37.054692+0000","last_active":"2023-10-25T00:23:37.054561+0000","last_peered":"2023-10-25T00:23:37.054561+0000","last_clean":"2023-10-25T00:23:37.054561+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-25T00:23:37.054561+0000","last_undegraded":"2023-10-25T00:23:37.054561+0000","last_fullsized":"2023-10-25T00:23:37.054561+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4209","last_scrub_stamp":"2023-10-25T00:23:37.054538+0000","last_deep_scrub":"72'4209","last_deep_scrub_stamp":"2023-10-25T00:23:37.054538+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.054538+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:39:40.159884+0000","scrub_duration":0.40759778099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24141,"num_write_kb":281325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"68'4915","reported_seq":5002,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232375+0000","last_change":"2023-10-25T00:22:38.600278+0000","last_active":"2023-10-25T00:22:49.232375+0000","last_peered":"2023-10-25T00:22:49.232375+0000","last_clean":"2023-10-25T00:22:49.232375+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-25T00:22:49.232375+0000","last_undegraded":"2023-10-25T00:22:49.232375+0000","last_fullsized":"2023-10-25T00:22:49.232375+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4915","last_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_deep_scrub":"68'4915","last_deep_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.600135+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:42:30.671003+0000","scrub_duration":0.18827047799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26161,"num_write_kb":314084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"72'4933","reported_seq":5017,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:31.791243+0000","last_change":"2023-10-25T00:23:31.791243+0000","last_active":"2023-10-25T00:23:31.791243+0000","last_peered":"2023-10-25T00:23:31.791243+0000","last_clean":"2023-10-25T00:23:31.791243+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-25T00:23:31.791243+0000","last_undegraded":"2023-10-25T00:23:31.791243+0000","last_fullsized":"2023-10-25T00:23:31.791243+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4933","last_scrub_stamp":"2023-10-25T00:23:31.791069+0000","last_deep_scrub":"72'4933","last_deep_scrub_stamp":"2023-10-25T00:23:31.791069+0000","last_clean_scrub_stamp":"2023-10-25T00:23:31.791069+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:53:40.465688+0000","scrub_duration":0.269140133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26695,"num_write_kb":352905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"68'5055","reported_seq":5143,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.217349+0000","last_change":"2023-10-25T00:22:43.444464+0000","last_active":"2023-10-25T00:22:49.217349+0000","last_peered":"2023-10-25T00:22:49.217349+0000","last_clean":"2023-10-25T00:22:49.217349+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-25T00:22:49.217349+0000","last_undegraded":"2023-10-25T00:22:49.217349+0000","last_fullsized":"2023-10-25T00:22:49.217349+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5055","last_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_deep_scrub":"68'5055","last_deep_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_clean_scrub_stamp":"2023-10-25T00:22:43.444210+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:13:34.310177+0000","scrub_duration":0.081291211000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27516,"num_write_kb":327406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"72'4083","reported_seq":4171,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:47.749612+0000","last_change":"2023-10-25T00:23:47.749735+0000","last_active":"2023-10-25T00:23:47.749612+0000","last_peered":"2023-10-25T00:23:47.749612+0000","last_clean":"2023-10-25T00:23:47.749612+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-25T00:23:47.749612+0000","last_undegraded":"2023-10-25T00:23:47.749612+0000","last_fullsized":"2023-10-25T00:23:47.749612+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4083","last_scrub_stamp":"2023-10-25T00:23:47.749591+0000","last_deep_scrub":"72'4083","last_deep_scrub_stamp":"2023-10-25T00:23:47.749591+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.749591+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:43:07.403801+0000","scrub_duration":0.25918773899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":22892,"num_write_kb":296872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"68'4777","reported_seq":4866,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:24.846143+0000","last_change":"2023-10-25T00:23:24.846143+0000","last_active":"2023-10-25T00:23:24.846143+0000","last_peered":"2023-10-25T00:23:24.846143+0000","last_clean":"2023-10-25T00:23:24.846143+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-25T00:23:24.846143+0000","last_undegraded":"2023-10-25T00:23:24.846143+0000","last_fullsized":"2023-10-25T00:23:24.846143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4777","last_scrub_stamp":"2023-10-25T00:23:24.846002+0000","last_deep_scrub":"68'4777","last_deep_scrub_stamp":"2023-10-25T00:23:24.846002+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.846002+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:41:16.418283+0000","scrub_duration":0.19199595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25812,"num_write_kb":300181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"72'4618","reported_seq":4708,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:33.839487+0000","last_change":"2023-10-25T00:23:33.839487+0000","last_active":"2023-10-25T00:23:33.839487+0000","last_peered":"2023-10-25T00:23:33.839487+0000","last_clean":"2023-10-25T00:23:33.839487+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-25T00:23:33.839487+0000","last_undegraded":"2023-10-25T00:23:33.839487+0000","last_fullsized":"2023-10-25T00:23:33.839487+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4618","last_scrub_stamp":"2023-10-25T00:23:33.839138+0000","last_deep_scrub":"72'4618","last_deep_scrub_stamp":"2023-10-25T00:23:33.839138+0000","last_clean_scrub_stamp":"2023-10-25T00:23:33.839138+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:25:32.655697+0000","scrub_duration":0.330454572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":26180,"num_write_kb":326342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"68'5340","reported_seq":5427,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.231448+0000","last_change":"2023-10-25T00:22:42.520048+0000","last_active":"2023-10-25T00:22:49.231448+0000","last_peered":"2023-10-25T00:22:49.231448+0000","last_clean":"2023-10-25T00:22:49.231448+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-25T00:22:49.231448+0000","last_undegraded":"2023-10-25T00:22:49.231448+0000","last_fullsized":"2023-10-25T00:22:49.231448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5340","last_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_deep_scrub":"68'5340","last_deep_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.519886+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.832889+0000","scrub_duration":0.129134375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28701,"num_write_kb":348521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"72'4171","reported_seq":4245,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060881+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-25T00:22:36.060881+0000","last_peered":"2023-10-25T00:22:36.060881+0000","last_clean":"2023-10-25T00:22:36.060881+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-25T00:22:36.060881+0000","last_undegraded":"2023-10-25T00:22:36.060881+0000","last_fullsized":"2023-10-25T00:22:36.060881+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22551,"num_write_kb":303498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"68'4493","reported_seq":4581,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:01.322975+0000","last_change":"2023-10-25T00:23:01.323167+0000","last_active":"2023-10-25T00:23:01.322975+0000","last_peered":"2023-10-25T00:23:01.322975+0000","last_clean":"2023-10-25T00:23:01.322975+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-25T00:23:01.322975+0000","last_undegraded":"2023-10-25T00:23:01.322975+0000","last_fullsized":"2023-10-25T00:23:01.322975+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4493","last_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_deep_scrub":"68'4493","last_deep_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_clean_scrub_stamp":"2023-10-25T00:23:01.322941+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:49.029022+0000","scrub_duration":0.13616751799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23797,"num_write_kb":300260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"72'4763","reported_seq":4836,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.226383+0000","last_change":"2023-10-24T23:53:44.474203+0000","last_active":"2023-10-25T00:22:49.226383+0000","last_peered":"2023-10-25T00:22:49.226383+0000","last_clean":"2023-10-25T00:22:49.226383+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-25T00:22:49.226383+0000","last_undegraded":"2023-10-25T00:22:49.226383+0000","last_fullsized":"2023-10-25T00:22:49.226383+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":25751,"num_write_kb":345225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"68'4946","reported_seq":5033,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:55.632611+0000","last_change":"2023-10-25T00:23:55.632753+0000","last_active":"2023-10-25T00:23:55.632611+0000","last_peered":"2023-10-25T00:23:55.632611+0000","last_clean":"2023-10-25T00:23:55.632611+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-25T00:23:55.632611+0000","last_undegraded":"2023-10-25T00:23:55.632611+0000","last_fullsized":"2023-10-25T00:23:55.632611+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4946","last_scrub_stamp":"2023-10-25T00:23:55.632589+0000","last_deep_scrub":"68'4946","last_deep_scrub_stamp":"2023-10-25T00:23:55.632589+0000","last_clean_scrub_stamp":"2023-10-25T00:23:55.632589+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:34:04.162109+0000","scrub_duration":0.103749073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26583,"num_write_kb":315633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"72'4352","reported_seq":4437,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.629523+0000","last_change":"2023-10-25T00:23:34.629523+0000","last_active":"2023-10-25T00:23:34.629523+0000","last_peered":"2023-10-25T00:23:34.629523+0000","last_clean":"2023-10-25T00:23:34.629523+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-25T00:23:34.629523+0000","last_undegraded":"2023-10-25T00:23:34.629523+0000","last_fullsized":"2023-10-25T00:23:34.629523+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4352","last_scrub_stamp":"2023-10-25T00:23:34.629323+0000","last_deep_scrub":"72'4352","last_deep_scrub_stamp":"2023-10-25T00:23:34.629323+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.629323+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:38.589150+0000","scrub_duration":0.29560794600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23957,"num_write_kb":301052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"68'5026","reported_seq":5102,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.226252+0000","last_change":"2023-10-25T00:23:30.833489+0000","last_active":"2023-10-25T00:22:49.226252+0000","last_peered":"2023-10-25T00:22:49.226252+0000","last_clean":"2023-10-25T00:22:49.226252+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-25T00:22:49.226252+0000","last_undegraded":"2023-10-25T00:22:49.226252+0000","last_fullsized":"2023-10-25T00:22:49.226252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26401,"num_write_kb":310586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"72'4393","reported_seq":4482,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.556819+0000","last_change":"2023-10-25T00:23:07.556990+0000","last_active":"2023-10-25T00:23:07.556819+0000","last_peered":"2023-10-25T00:23:07.556819+0000","last_clean":"2023-10-25T00:23:07.556819+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-25T00:23:07.556819+0000","last_undegraded":"2023-10-25T00:23:07.556819+0000","last_fullsized":"2023-10-25T00:23:07.556819+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4393","last_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_deep_scrub":"72'4393","last_deep_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.556784+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:21:00.928852+0000","scrub_duration":0.35410455800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":24752,"num_write_kb":296109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"68'5620","reported_seq":5710,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.194060+0000","last_change":"2023-10-25T00:23:12.194324+0000","last_active":"2023-10-25T00:23:12.194060+0000","last_peered":"2023-10-25T00:23:12.194060+0000","last_clean":"2023-10-25T00:23:12.194060+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-25T00:23:12.194060+0000","last_undegraded":"2023-10-25T00:23:12.194060+0000","last_fullsized":"2023-10-25T00:23:12.194060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5620","last_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_deep_scrub":"68'5620","last_deep_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.194008+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:33.868259+0000","scrub_duration":0.086061930999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29863,"num_write_kb":337169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"72'4340","reported_seq":4428,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:23.851965+0000","last_change":"2023-10-25T00:23:23.851965+0000","last_active":"2023-10-25T00:23:23.851965+0000","last_peered":"2023-10-25T00:23:23.851965+0000","last_clean":"2023-10-25T00:23:23.851965+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-25T00:23:23.851965+0000","last_undegraded":"2023-10-25T00:23:23.851965+0000","last_fullsized":"2023-10-25T00:23:23.851965+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4340","last_scrub_stamp":"2023-10-25T00:23:23.851809+0000","last_deep_scrub":"72'4340","last_deep_scrub_stamp":"2023-10-25T00:23:23.851809+0000","last_clean_scrub_stamp":"2023-10-25T00:23:23.851809+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:00.223998+0000","scrub_duration":0.37196321399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":36,"num_write":23367,"num_write_kb":302669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"68'4464","reported_seq":4549,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.904552+0000","last_change":"2023-10-25T00:22:57.904669+0000","last_active":"2023-10-25T00:22:57.904552+0000","last_peered":"2023-10-25T00:22:57.904552+0000","last_clean":"2023-10-25T00:22:57.904552+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-25T00:22:57.904552+0000","last_undegraded":"2023-10-25T00:22:57.904552+0000","last_fullsized":"2023-10-25T00:22:57.904552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4464","last_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_deep_scrub":"68'4464","last_deep_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.904532+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:17:44.026155+0000","scrub_duration":0.14132308199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24372,"num_write_kb":278074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"72'5589","reported_seq":5673,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.174468+0000","last_change":"2023-10-25T00:23:45.174602+0000","last_active":"2023-10-25T00:23:45.174468+0000","last_peered":"2023-10-25T00:23:45.174468+0000","last_clean":"2023-10-25T00:23:45.174468+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-25T00:23:45.174468+0000","last_undegraded":"2023-10-25T00:23:45.174468+0000","last_fullsized":"2023-10-25T00:23:45.174468+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5589","last_scrub_stamp":"2023-10-25T00:23:45.174448+0000","last_deep_scrub":"72'5589","last_deep_scrub_stamp":"2023-10-25T00:23:45.174448+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.174448+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:47:45.550696+0000","scrub_duration":0.18148444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":30206,"num_write_kb":399557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"68'4542","reported_seq":4628,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:09.344932+0000","last_change":"2023-10-25T00:23:09.345212+0000","last_active":"2023-10-25T00:23:09.344932+0000","last_peered":"2023-10-25T00:23:09.344932+0000","last_clean":"2023-10-25T00:23:09.344932+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-25T00:23:09.344932+0000","last_undegraded":"2023-10-25T00:23:09.344932+0000","last_fullsized":"2023-10-25T00:23:09.344932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4542","last_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_deep_scrub":"68'4542","last_deep_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.344860+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:16:09.325372+0000","scrub_duration":0.066583699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23889,"num_write_kb":292323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"72'3430","reported_seq":3515,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.797777+0000","last_change":"2023-10-25T00:23:21.798045+0000","last_active":"2023-10-25T00:23:21.797777+0000","last_peered":"2023-10-25T00:23:21.797777+0000","last_clean":"2023-10-25T00:23:21.797777+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-25T00:23:21.797777+0000","last_undegraded":"2023-10-25T00:23:21.797777+0000","last_fullsized":"2023-10-25T00:23:21.797777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3430","last_scrub_stamp":"2023-10-25T00:23:21.797727+0000","last_deep_scrub":"72'3430","last_deep_scrub_stamp":"2023-10-25T00:23:21.797727+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.797727+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:15:30.844796+0000","scrub_duration":0.29875561899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19411,"num_write_kb":241862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"68'4719","reported_seq":4806,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:58.903089+0000","last_change":"2023-10-25T00:22:58.903089+0000","last_active":"2023-10-25T00:22:58.903089+0000","last_peered":"2023-10-25T00:22:58.903089+0000","last_clean":"2023-10-25T00:22:58.903089+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-25T00:22:58.903089+0000","last_undegraded":"2023-10-25T00:22:58.903089+0000","last_fullsized":"2023-10-25T00:22:58.903089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4719","last_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_deep_scrub":"68'4719","last_deep_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.902769+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:01.236787+0000","scrub_duration":0.136999537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25020,"num_write_kb":305197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"72'4169","reported_seq":4251,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.640667+0000","last_change":"2023-10-25T00:23:40.640863+0000","last_active":"2023-10-25T00:23:40.640667+0000","last_peered":"2023-10-25T00:23:40.640667+0000","last_clean":"2023-10-25T00:23:40.640667+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-25T00:23:40.640667+0000","last_undegraded":"2023-10-25T00:23:40.640667+0000","last_fullsized":"2023-10-25T00:23:40.640667+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4169","last_scrub_stamp":"2023-10-25T00:23:40.640645+0000","last_deep_scrub":"72'4169","last_deep_scrub_stamp":"2023-10-25T00:23:40.640645+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.640645+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:06.110573+0000","scrub_duration":0.28697117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23235,"num_write_kb":306175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"68'5952","reported_seq":6047,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.017693+0000","last_change":"2023-10-25T00:23:05.017693+0000","last_active":"2023-10-25T00:23:05.017693+0000","last_peered":"2023-10-25T00:23:05.017693+0000","last_clean":"2023-10-25T00:23:05.017693+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-25T00:23:05.017693+0000","last_undegraded":"2023-10-25T00:23:05.017693+0000","last_fullsized":"2023-10-25T00:23:05.017693+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5952","last_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_deep_scrub":"68'5952","last_deep_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.017340+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:31.140541+0000","scrub_duration":0.055624675999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32341,"num_write_kb":375441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"72'3717","reported_seq":3789,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232259+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-25T00:22:49.232259+0000","last_peered":"2023-10-25T00:22:49.232259+0000","last_clean":"2023-10-25T00:22:49.232259+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-25T00:22:49.232259+0000","last_undegraded":"2023-10-25T00:22:49.232259+0000","last_fullsized":"2023-10-25T00:22:49.232259+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21101,"num_write_kb":269108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"68'4825","reported_seq":4913,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.089847+0000","last_change":"2023-10-25T00:23:06.089847+0000","last_active":"2023-10-25T00:23:06.089847+0000","last_peered":"2023-10-25T00:23:06.089847+0000","last_clean":"2023-10-25T00:23:06.089847+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-25T00:23:06.089847+0000","last_undegraded":"2023-10-25T00:23:06.089847+0000","last_fullsized":"2023-10-25T00:23:06.089847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4825","last_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_deep_scrub":"68'4825","last_deep_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.089705+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:04:51.412522+0000","scrub_duration":0.15071393499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25319,"num_write_kb":294113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"72'3714","reported_seq":3789,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.215543+0000","last_change":"2023-10-25T00:23:12.182256+0000","last_active":"2023-10-25T00:22:49.215543+0000","last_peered":"2023-10-25T00:22:49.215543+0000","last_clean":"2023-10-25T00:22:49.215543+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-25T00:22:49.215543+0000","last_undegraded":"2023-10-25T00:22:49.215543+0000","last_fullsized":"2023-10-25T00:22:49.215543+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21148,"num_write_kb":265324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"68'4878","reported_seq":4967,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.227455+0000","last_change":"2023-10-25T00:22:49.227455+0000","last_active":"2023-10-25T00:22:49.227455+0000","last_peered":"2023-10-25T00:22:49.227455+0000","last_clean":"2023-10-25T00:22:49.227455+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-25T00:22:49.227455+0000","last_undegraded":"2023-10-25T00:22:49.227455+0000","last_fullsized":"2023-10-25T00:22:49.227455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4878","last_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_deep_scrub":"68'4878","last_deep_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_clean_scrub_stamp":"2023-10-25T00:22:49.227012+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:58:21.157778+0000","scrub_duration":0.13340163999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26154,"num_write_kb":305558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"72'3434","reported_seq":3507,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.208392+0000","last_change":"2023-10-24T23:53:11.859609+0000","last_active":"2023-10-25T00:22:49.208392+0000","last_peered":"2023-10-25T00:22:49.208392+0000","last_clean":"2023-10-25T00:22:49.208392+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-25T00:22:49.208392+0000","last_undegraded":"2023-10-25T00:22:49.208392+0000","last_fullsized":"2023-10-25T00:22:49.208392+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18449,"num_write_kb":262373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"68'4672","reported_seq":4750,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:23:30.929700+0000","last_change":"2023-10-25T00:23:41.459981+0000","last_active":"2023-10-25T00:23:30.929700+0000","last_peered":"2023-10-25T00:23:30.929700+0000","last_clean":"2023-10-25T00:23:30.929700+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-25T00:23:30.929700+0000","last_undegraded":"2023-10-25T00:23:30.929700+0000","last_fullsized":"2023-10-25T00:23:30.929700+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25223,"num_write_kb":293792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"72'4013","reported_seq":4102,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:51.895004+0000","last_change":"2023-10-25T00:23:51.895151+0000","last_active":"2023-10-25T00:23:51.895004+0000","last_peered":"2023-10-25T00:23:51.895004+0000","last_clean":"2023-10-25T00:23:51.895004+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-25T00:23:51.895004+0000","last_undegraded":"2023-10-25T00:23:51.895004+0000","last_fullsized":"2023-10-25T00:23:51.895004+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4013","last_scrub_stamp":"2023-10-25T00:23:51.894968+0000","last_deep_scrub":"72'4013","last_deep_scrub_stamp":"2023-10-25T00:23:51.894968+0000","last_clean_scrub_stamp":"2023-10-25T00:23:51.894968+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:20:31.752935+0000","scrub_duration":0.32625951199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":22066,"num_write_kb":297298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"68'3918","reported_seq":4004,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:49.563660+0000","last_change":"2023-10-25T00:23:49.563900+0000","last_active":"2023-10-25T00:23:49.563660+0000","last_peered":"2023-10-25T00:23:49.563660+0000","last_clean":"2023-10-25T00:23:49.563660+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-25T00:23:49.563660+0000","last_undegraded":"2023-10-25T00:23:49.563660+0000","last_fullsized":"2023-10-25T00:23:49.563660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3918","last_scrub_stamp":"2023-10-25T00:23:49.563617+0000","last_deep_scrub":"68'3918","last_deep_scrub_stamp":"2023-10-25T00:23:49.563617+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.563617+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:50:13.550847+0000","scrub_duration":0.092172528000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21675,"num_write_kb":247000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"72'4401","reported_seq":4485,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:42.768643+0000","last_change":"2023-10-25T00:23:42.768767+0000","last_active":"2023-10-25T00:23:42.768643+0000","last_peered":"2023-10-25T00:23:42.768643+0000","last_clean":"2023-10-25T00:23:42.768643+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-25T00:23:42.768643+0000","last_undegraded":"2023-10-25T00:23:42.768643+0000","last_fullsized":"2023-10-25T00:23:42.768643+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4401","last_scrub_stamp":"2023-10-25T00:23:42.768621+0000","last_deep_scrub":"72'4401","last_deep_scrub_stamp":"2023-10-25T00:23:42.768621+0000","last_clean_scrub_stamp":"2023-10-25T00:23:42.768621+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:23:18.678304+0000","scrub_duration":0.26783985300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":24294,"num_write_kb":326749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"68'3970","reported_seq":4055,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.428569+0000","last_change":"2023-10-25T00:23:12.428867+0000","last_active":"2023-10-25T00:23:12.428569+0000","last_peered":"2023-10-25T00:23:12.428569+0000","last_clean":"2023-10-25T00:23:12.428569+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-25T00:23:12.428569+0000","last_undegraded":"2023-10-25T00:23:12.428569+0000","last_fullsized":"2023-10-25T00:23:12.428569+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3970","last_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_deep_scrub":"68'3970","last_deep_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.428494+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:07.899437+0000","scrub_duration":0.13106953599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21202,"num_write_kb":257085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"72'5559","reported_seq":5649,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:41.090126+0000","last_change":"2023-10-25T00:23:41.090253+0000","last_active":"2023-10-25T00:23:41.090126+0000","last_peered":"2023-10-25T00:23:41.090126+0000","last_clean":"2023-10-25T00:23:41.090126+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-25T00:23:41.090126+0000","last_undegraded":"2023-10-25T00:23:41.090126+0000","last_fullsized":"2023-10-25T00:23:41.090126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5559","last_scrub_stamp":"2023-10-25T00:23:41.090104+0000","last_deep_scrub":"72'5559","last_deep_scrub_stamp":"2023-10-25T00:23:41.090104+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.090104+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:40:20.680440+0000","scrub_duration":0.241945666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":31202,"num_write_kb":391572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"68'4936","reported_seq":5022,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.021130+0000","last_change":"2023-10-25T00:23:02.021272+0000","last_active":"2023-10-25T00:23:02.021130+0000","last_peered":"2023-10-25T00:23:02.021130+0000","last_clean":"2023-10-25T00:23:02.021130+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-25T00:23:02.021130+0000","last_undegraded":"2023-10-25T00:23:02.021130+0000","last_fullsized":"2023-10-25T00:23:02.021130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4936","last_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_deep_scrub":"68'4936","last_deep_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.021108+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:22.652380+0000","scrub_duration":0.17348896599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26134,"num_write_kb":309652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"72'3558","reported_seq":3645,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:47.072662+0000","last_change":"2023-10-25T00:23:47.072846+0000","last_active":"2023-10-25T00:23:47.072662+0000","last_peered":"2023-10-25T00:23:47.072662+0000","last_clean":"2023-10-25T00:23:47.072662+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-25T00:23:47.072662+0000","last_undegraded":"2023-10-25T00:23:47.072662+0000","last_fullsized":"2023-10-25T00:23:47.072662+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3558","last_scrub_stamp":"2023-10-25T00:23:47.072625+0000","last_deep_scrub":"72'3558","last_deep_scrub_stamp":"2023-10-25T00:23:47.072625+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.072625+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:59:13.347978+0000","scrub_duration":0.34108986800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":34,"num_write":19715,"num_write_kb":243025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"68'5282","reported_seq":5368,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.290128+0000","last_change":"2023-10-25T00:23:14.290293+0000","last_active":"2023-10-25T00:23:14.290128+0000","last_peered":"2023-10-25T00:23:14.290128+0000","last_clean":"2023-10-25T00:23:14.290128+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-25T00:23:14.290128+0000","last_undegraded":"2023-10-25T00:23:14.290128+0000","last_fullsized":"2023-10-25T00:23:14.290128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5282","last_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_deep_scrub":"68'5282","last_deep_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.290099+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:44:38.139530+0000","scrub_duration":0.10140234200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27848,"num_write_kb":324561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"72'3914","reported_seq":4002,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:46.833316+0000","last_change":"2023-10-25T00:23:46.833446+0000","last_active":"2023-10-25T00:23:46.833316+0000","last_peered":"2023-10-25T00:23:46.833316+0000","last_clean":"2023-10-25T00:23:46.833316+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-25T00:23:46.833316+0000","last_undegraded":"2023-10-25T00:23:46.833316+0000","last_fullsized":"2023-10-25T00:23:46.833316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3914","last_scrub_stamp":"2023-10-25T00:23:46.833294+0000","last_deep_scrub":"72'3914","last_deep_scrub_stamp":"2023-10-25T00:23:46.833294+0000","last_clean_scrub_stamp":"2023-10-25T00:23:46.833294+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:59.194933+0000","scrub_duration":0.30262013700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":20938,"num_write_kb":288796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"68'6616","reported_seq":6706,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.985123+0000","last_change":"2023-10-25T00:22:57.985278+0000","last_active":"2023-10-25T00:22:57.985123+0000","last_peered":"2023-10-25T00:22:57.985123+0000","last_clean":"2023-10-25T00:22:57.985123+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-25T00:22:57.985123+0000","last_undegraded":"2023-10-25T00:22:57.985123+0000","last_fullsized":"2023-10-25T00:22:57.985123+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6616","last_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_deep_scrub":"68'6616","last_deep_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.985096+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:14:23.491729+0000","scrub_duration":0.050039131000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36221,"num_write_kb":423916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"72'3536","reported_seq":3620,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:32.908497+0000","last_change":"2023-10-25T00:23:32.908624+0000","last_active":"2023-10-25T00:23:32.908497+0000","last_peered":"2023-10-25T00:23:32.908497+0000","last_clean":"2023-10-25T00:23:32.908497+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-25T00:23:32.908497+0000","last_undegraded":"2023-10-25T00:23:32.908497+0000","last_fullsized":"2023-10-25T00:23:32.908497+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3536","last_scrub_stamp":"2023-10-25T00:23:32.908476+0000","last_deep_scrub":"72'3536","last_deep_scrub_stamp":"2023-10-25T00:23:32.908476+0000","last_clean_scrub_stamp":"2023-10-25T00:23:32.908476+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:33:20.337271+0000","scrub_duration":0.30729883400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19276,"num_write_kb":253506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"68'4980","reported_seq":5064,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:10.321019+0000","last_change":"2023-10-25T00:23:10.321202+0000","last_active":"2023-10-25T00:23:10.321019+0000","last_peered":"2023-10-25T00:23:10.321019+0000","last_clean":"2023-10-25T00:23:10.321019+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-25T00:23:10.321019+0000","last_undegraded":"2023-10-25T00:23:10.321019+0000","last_fullsized":"2023-10-25T00:23:10.321019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4980","last_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_deep_scrub":"68'4980","last_deep_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:10.320988+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:34.745429+0000","scrub_duration":0.13750111800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26525,"num_write_kb":306963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"72'4160","reported_seq":4246,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:24.541581+0000","last_change":"2023-10-25T00:23:24.541711+0000","last_active":"2023-10-25T00:23:24.541581+0000","last_peered":"2023-10-25T00:23:24.541581+0000","last_clean":"2023-10-25T00:23:24.541581+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-25T00:23:24.541581+0000","last_undegraded":"2023-10-25T00:23:24.541581+0000","last_fullsized":"2023-10-25T00:23:24.541581+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4160","last_scrub_stamp":"2023-10-25T00:23:24.541561+0000","last_deep_scrub":"72'4160","last_deep_scrub_stamp":"2023-10-25T00:23:24.541561+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.541561+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:52:13.060866+0000","scrub_duration":0.264226874,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":21819,"num_write_kb":288439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"68'4434","reported_seq":4520,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.208506+0000","last_change":"2023-10-25T00:22:47.255974+0000","last_active":"2023-10-25T00:22:49.208506+0000","last_peered":"2023-10-25T00:22:49.208506+0000","last_clean":"2023-10-25T00:22:49.208506+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-25T00:22:49.208506+0000","last_undegraded":"2023-10-25T00:22:49.208506+0000","last_fullsized":"2023-10-25T00:22:49.208506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4434","last_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_deep_scrub":"68'4434","last_deep_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.255646+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:35:52.723567+0000","scrub_duration":0.14955275700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23791,"num_write_kb":292031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"72'4195","reported_seq":4276,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.836868+0000","last_change":"2023-10-25T00:23:37.837002+0000","last_active":"2023-10-25T00:23:37.836868+0000","last_peered":"2023-10-25T00:23:37.836868+0000","last_clean":"2023-10-25T00:23:37.836868+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-25T00:23:37.836868+0000","last_undegraded":"2023-10-25T00:23:37.836868+0000","last_fullsized":"2023-10-25T00:23:37.836868+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4195","last_scrub_stamp":"2023-10-25T00:23:37.836848+0000","last_deep_scrub":"72'4195","last_deep_scrub_stamp":"2023-10-25T00:23:37.836848+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.836848+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:49:10.177488+0000","scrub_duration":0.28694403499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23000,"num_write_kb":285019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"68'4243","reported_seq":4328,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.969067+0000","last_change":"2023-10-25T00:23:00.969067+0000","last_active":"2023-10-25T00:23:00.969067+0000","last_peered":"2023-10-25T00:23:00.969067+0000","last_clean":"2023-10-25T00:23:00.969067+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-25T00:23:00.969067+0000","last_undegraded":"2023-10-25T00:23:00.969067+0000","last_fullsized":"2023-10-25T00:23:00.969067+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4243","last_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_deep_scrub":"68'4243","last_deep_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.968692+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:45.877638+0000","scrub_duration":0.15459757700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22439,"num_write_kb":266094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"72'4017","reported_seq":4099,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:54.962933+0000","last_change":"2023-10-25T00:23:54.963064+0000","last_active":"2023-10-25T00:23:54.962933+0000","last_peered":"2023-10-25T00:23:54.962933+0000","last_clean":"2023-10-25T00:23:54.962933+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-25T00:23:54.962933+0000","last_undegraded":"2023-10-25T00:23:54.962933+0000","last_fullsized":"2023-10-25T00:23:54.962933+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4017","last_scrub_stamp":"2023-10-25T00:23:54.962911+0000","last_deep_scrub":"72'4017","last_deep_scrub_stamp":"2023-10-25T00:23:54.962911+0000","last_clean_scrub_stamp":"2023-10-25T00:23:54.962911+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:49:43.769503+0000","scrub_duration":0.29685359700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22665,"num_write_kb":291707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"68'4509","reported_seq":4597,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.227093+0000","last_change":"2023-10-25T00:22:44.022792+0000","last_active":"2023-10-25T00:22:49.227093+0000","last_peered":"2023-10-25T00:22:49.227093+0000","last_clean":"2023-10-25T00:22:49.227093+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-25T00:22:49.227093+0000","last_undegraded":"2023-10-25T00:22:49.227093+0000","last_fullsized":"2023-10-25T00:22:49.227093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4509","last_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_deep_scrub":"68'4509","last_deep_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.022573+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:54:30.651361+0000","scrub_duration":0.11145192700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23158,"num_write_kb":297883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"72'5235","reported_seq":5325,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:50.831818+0000","last_change":"2023-10-25T00:23:50.831958+0000","last_active":"2023-10-25T00:23:50.831818+0000","last_peered":"2023-10-25T00:23:50.831818+0000","last_clean":"2023-10-25T00:23:50.831818+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-25T00:23:50.831818+0000","last_undegraded":"2023-10-25T00:23:50.831818+0000","last_fullsized":"2023-10-25T00:23:50.831818+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5235","last_scrub_stamp":"2023-10-25T00:23:50.831798+0000","last_deep_scrub":"72'5235","last_deep_scrub_stamp":"2023-10-25T00:23:50.831798+0000","last_clean_scrub_stamp":"2023-10-25T00:23:50.831798+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:34:32.587906+0000","scrub_duration":0.30673944400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":15,"num_write":29235,"num_write_kb":367391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"68'4260","reported_seq":4349,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:56.486314+0000","last_change":"2023-10-25T00:22:56.486314+0000","last_active":"2023-10-25T00:22:56.486314+0000","last_peered":"2023-10-25T00:22:56.486314+0000","last_clean":"2023-10-25T00:22:56.486314+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-25T00:22:56.486314+0000","last_undegraded":"2023-10-25T00:22:56.486314+0000","last_fullsized":"2023-10-25T00:22:56.486314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4260","last_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_deep_scrub":"68'4260","last_deep_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.485966+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:51.396462+0000","scrub_duration":0.116368769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23357,"num_write_kb":265837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"72'4656","reported_seq":4748,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.632181+0000","last_change":"2023-10-25T00:23:40.632314+0000","last_active":"2023-10-25T00:23:40.632181+0000","last_peered":"2023-10-25T00:23:40.632181+0000","last_clean":"2023-10-25T00:23:40.632181+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-25T00:23:40.632181+0000","last_undegraded":"2023-10-25T00:23:40.632181+0000","last_fullsized":"2023-10-25T00:23:40.632181+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4656","last_scrub_stamp":"2023-10-25T00:23:40.632160+0000","last_deep_scrub":"72'4656","last_deep_scrub_stamp":"2023-10-25T00:23:40.632160+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.632160+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:23:41.024594+0000","scrub_duration":0.095096767999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":26109,"num_write_kb":317781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"68'4636","reported_seq":4726,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:39.672588+0000","last_change":"2023-10-25T00:22:39.672588+0000","last_active":"2023-10-25T00:22:39.672588+0000","last_peered":"2023-10-25T00:22:39.672588+0000","last_clean":"2023-10-25T00:22:39.672588+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-25T00:22:39.672588+0000","last_undegraded":"2023-10-25T00:22:39.672588+0000","last_fullsized":"2023-10-25T00:22:39.672588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4636","last_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_deep_scrub":"68'4636","last_deep_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.672438+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:47:15.433547+0000","scrub_duration":0.124153799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25124,"num_write_kb":285314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"72'3117","reported_seq":3204,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:41.679337+0000","last_change":"2023-10-25T00:23:41.679657+0000","last_active":"2023-10-25T00:23:41.679337+0000","last_peered":"2023-10-25T00:23:41.679337+0000","last_clean":"2023-10-25T00:23:41.679337+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-25T00:23:41.679337+0000","last_undegraded":"2023-10-25T00:23:41.679337+0000","last_fullsized":"2023-10-25T00:23:41.679337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3117","last_scrub_stamp":"2023-10-25T00:23:41.679268+0000","last_deep_scrub":"72'3117","last_deep_scrub_stamp":"2023-10-25T00:23:41.679268+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.679268+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:35.283156+0000","scrub_duration":0.30581881399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17240,"num_write_kb":232937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"68'5520","reported_seq":5617,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:56.366454+0000","last_change":"2023-10-25T00:22:56.366641+0000","last_active":"2023-10-25T00:22:56.366454+0000","last_peered":"2023-10-25T00:22:56.366454+0000","last_clean":"2023-10-25T00:22:56.366454+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-25T00:22:56.366454+0000","last_undegraded":"2023-10-25T00:22:56.366454+0000","last_fullsized":"2023-10-25T00:22:56.366454+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5520","last_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_deep_scrub":"68'5520","last_deep_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.366419+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:47:59.889167+0000","scrub_duration":0.104594514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29665,"num_write_kb":368823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"72'3962","reported_seq":4047,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.105781+0000","last_change":"2023-10-25T00:23:34.105899+0000","last_active":"2023-10-25T00:23:34.105781+0000","last_peered":"2023-10-25T00:23:34.105781+0000","last_clean":"2023-10-25T00:23:34.105781+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-25T00:23:34.105781+0000","last_undegraded":"2023-10-25T00:23:34.105781+0000","last_fullsized":"2023-10-25T00:23:34.105781+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3962","last_scrub_stamp":"2023-10-25T00:23:34.105758+0000","last_deep_scrub":"72'3962","last_deep_scrub_stamp":"2023-10-25T00:23:34.105758+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.105758+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:35:57.960582+0000","scrub_duration":0.32704070499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21989,"num_write_kb":280762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"68'4908","reported_seq":5000,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.407703+0000","last_change":"2023-10-25T00:23:00.408006+0000","last_active":"2023-10-25T00:23:00.407703+0000","last_peered":"2023-10-25T00:23:00.407703+0000","last_clean":"2023-10-25T00:23:00.407703+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-25T00:23:00.407703+0000","last_undegraded":"2023-10-25T00:23:00.407703+0000","last_fullsized":"2023-10-25T00:23:00.407703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4908","last_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_deep_scrub":"68'4908","last_deep_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.407650+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:48:18.113931+0000","scrub_duration":0.051805808000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26829,"num_write_kb":320697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":320,"num_write":1482516,"num_write_kb":19060243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267718,"ondisk_log_size":267718,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":19490997,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1686,"num_read_kb":209824,"num_write":4665,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39043072,"data_stored":38981994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4560,"ondisk_log_size":4560,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":0,"num_write":1661441,"num_write_kb":19773626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307026,"ondisk_log_size":307026,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":20061533,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1689,"num_read_kb":267203,"num_write":4630,"num_write_kb":21638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40181760,"data_stored":40123066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4532,"ondisk_log_size":4532,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163,"ondisk_log_size":163,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575224,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919096,"kb_used_data":12336,"kb_used_omap":0,"kb_used_meta":906752,"kb_avail":93452744,"statfs":{"total":96636764160,"available":95695609856,"internally_reserved":0,"allocated":12632064,"data_stored":12153420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":928514048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.79600000000000004,"15min":0.94599999999999995},"min":{"1min":0.50600000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.232,"5min":3.6349999999999998,"15min":13.929},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.008,"5min":0.86599999999999999,"15min":0.94699999999999995},"min":{"1min":0.59299999999999997,"5min":0.27600000000000002,"15min":0.20999999999999999},"max":{"1min":1.3200000000000001,"5min":3.6659999999999999,"15min":11.315},"last":1.0169999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.80900000000000005,"15min":1.161},"min":{"1min":0.54900000000000004,"5min":0.23599999999999999,"15min":0.215},"max":{"1min":1.173,"5min":3.6949999999999998,"15min":28.308},"last":0.755},{"interface":"front","average":{"1min":1.0740000000000001,"5min":0.94399999999999995,"15min":1.24},"min":{"1min":0.78200000000000003,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":1.244,"5min":3.7490000000000001,"15min":28.346},"last":1.0680000000000001}]},{"osd":2,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":1.0089999999999999,"15min":0.99299999999999999},"min":{"1min":0.58099999999999996,"5min":0.28499999999999998,"15min":0.22800000000000001},"max":{"1min":1.448,"5min":5.71,"15min":11.430999999999999},"last":0.88},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.93400000000000005,"15min":0.96099999999999997},"min":{"1min":0.58699999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.1990000000000001,"5min":3.8799999999999999,"15min":7.9710000000000001},"last":0.93500000000000005}]},{"osd":3,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.89200000000000002,"15min":1.3620000000000001},"min":{"1min":0.748,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":1.2749999999999999,"5min":3.6150000000000002,"15min":93.664000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.042,"15min":1.3879999999999999},"min":{"1min":0.72199999999999998,"5min":0.32700000000000001,"15min":0.25700000000000001},"max":{"1min":1.2729999999999999,"5min":4.6630000000000003,"15min":93.073999999999998},"last":0.70699999999999996}]},{"osd":4,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.91600000000000004,"15min":0.88400000000000001},"min":{"1min":0.66000000000000003,"5min":0.314,"15min":0.249},"max":{"1min":1.274,"5min":3.0569999999999999,"15min":7.6280000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.05,"5min":1.0940000000000001,"15min":1.157},"min":{"1min":0.81000000000000005,"5min":0.36599999999999999,"15min":0.27200000000000002},"max":{"1min":1.3640000000000001,"5min":4.9640000000000004,"15min":27.936},"last":0.84699999999999998}]},{"osd":5,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.3480000000000001,"15min":1.0449999999999999},"min":{"1min":0.59499999999999997,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":1.5,"5min":21.126999999999999,"15min":21.126999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.95399999999999996,"15min":0.96799999999999997},"min":{"1min":0.65200000000000002,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":1.3740000000000001,"5min":3.8069999999999999,"15min":11.49},"last":0.82499999999999996}]},{"osd":6,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.3280000000000001,"15min":1.1140000000000001},"min":{"1min":0.71099999999999997,"5min":0.311,"15min":0.23599999999999999},"max":{"1min":1.2989999999999999,"5min":4.8140000000000001,"15min":8.2959999999999994},"last":0.996},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.4419999999999999,"15min":1.228},"min":{"1min":1.0189999999999999,"5min":0.45500000000000002,"15min":0.35499999999999998},"max":{"1min":1.4219999999999999,"5min":4.8639999999999999,"15min":20.651},"last":1.087}]}]},{"osd":6,"up_from":13,"seq":55834575224,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859020,"kb_used_data":5060,"kb_used_omap":0,"kb_used_meta":853952,"kb_avail":93512820,"statfs":{"total":96636764160,"available":95757127680,"internally_reserved":0,"allocated":5181440,"data_stored":4739934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":874446848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.91800000000000004,"15min":0.95399999999999996},"min":{"1min":0.39700000000000002,"5min":0.185,"15min":0.185},"max":{"1min":1.194,"5min":5.3490000000000002,"15min":15.974},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.036,"15min":0.97999999999999998},"min":{"1min":0.54400000000000004,"5min":0.218,"15min":0.218},"max":{"1min":1.115,"5min":8.2400000000000002,"15min":8.4789999999999992},"last":1.1799999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.81899999999999995,"15min":0.88300000000000001},"min":{"1min":0.57699999999999996,"5min":0.221,"15min":0.221},"max":{"1min":1.0469999999999999,"5min":3.5630000000000002,"15min":7.6360000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.878,"15min":0.98899999999999999},"min":{"1min":0.54000000000000004,"5min":0.254,"15min":0.249},"max":{"1min":1.111,"5min":3.5840000000000001,"15min":21.600999999999999},"last":1.0329999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.92700000000000005,"15min":1.0620000000000001},"min":{"1min":0.59999999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.198,"5min":3.6779999999999999,"15min":12.615},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.93400000000000005,"15min":1.1060000000000001},"min":{"1min":0.47599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.2370000000000001,"5min":3.702,"15min":12.667999999999999},"last":1.0009999999999999}]},{"osd":3,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.0009999999999999,"15min":1.248},"min":{"1min":0.68600000000000005,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.071,"5min":5.7350000000000003,"15min":38.649999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.0580000000000001,"15min":1.325},"min":{"1min":0.67000000000000004,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.1699999999999999,"5min":5.766,"15min":38.683},"last":0.93899999999999995}]},{"osd":4,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.3460000000000001,"15min":1.1240000000000001},"min":{"1min":0.56999999999999995,"5min":0.28399999999999997,"15min":0.23799999999999999},"max":{"1min":1.284,"5min":8.9930000000000003,"15min":12.682},"last":1.157},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.2609999999999999,"15min":1.081},"min":{"1min":0.63600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2190000000000001,"5min":5.694,"15min":10.686},"last":1.347}]},{"osd":5,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.0309999999999999,"15min":0.95699999999999996},"min":{"1min":0.622,"5min":0.28399999999999997,"15min":0.24299999999999999},"max":{"1min":1.1619999999999999,"5min":3.6379999999999999,"15min":8.0039999999999996},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.1759999999999999,"15min":1.0649999999999999},"min":{"1min":0.69799999999999995,"5min":0.34499999999999997,"15min":0.30299999999999999},"max":{"1min":1.244,"5min":4.0679999999999996,"15min":11.766999999999999},"last":1.1000000000000001}]},{"osd":7,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.177,"15min":1.0249999999999999},"min":{"1min":0.64200000000000002,"5min":0.27000000000000002,"15min":0.251},"max":{"1min":1.159,"5min":8.2720000000000002,"15min":12.926},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.234,"15min":1.0489999999999999},"min":{"1min":0.79400000000000004,"5min":0.34200000000000003,"15min":0.28799999999999998},"max":{"1min":1.2230000000000001,"5min":6.1100000000000003,"15min":7.6619999999999999},"last":1.2190000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575224,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":958872,"kb_used_data":9168,"kb_used_omap":0,"kb_used_meta":949696,"kb_avail":93412968,"statfs":{"total":96636764160,"available":95654879232,"internally_reserved":0,"allocated":9388032,"data_stored":8934420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":972488704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.133,"15min":1.038},"min":{"1min":0.505,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.97499999999999998,"5min":14.837999999999999,"15min":14.837999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.83899999999999997,"15min":0.97399999999999998},"min":{"1min":0.56799999999999995,"5min":0.20699999999999999,"15min":0.20100000000000001},"max":{"1min":1.427,"5min":4.016,"15min":7.423},"last":0.76100000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":1.0489999999999999,"15min":1.0880000000000001},"min":{"1min":0.51600000000000001,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":1.1879999999999999,"5min":12.141,"15min":12.141},"last":1.22},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.94399999999999995,"15min":1.0940000000000001},"min":{"1min":0.746,"5min":0.26400000000000001,"15min":0.252},"max":{"1min":0.999,"5min":9.0549999999999997,"15min":24.648},"last":1.3149999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.0840000000000001,"15min":1.2490000000000001},"min":{"1min":0.67200000000000004,"5min":0.25,"15min":0.25},"max":{"1min":1.258,"5min":12.175000000000001,"15min":20.760999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.117,"15min":1.073},"min":{"1min":0.68899999999999995,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2909999999999999,"5min":12.109999999999999,"15min":20.552},"last":1.099}]},{"osd":3,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.3109999999999999,"15min":1.2130000000000001},"min":{"1min":0.67300000000000004,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":1.1599999999999999,"5min":12.013,"15min":12.013},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.036,"15min":1.244},"min":{"1min":0.65500000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0960000000000001,"5min":4.0490000000000004,"15min":12.805},"last":1.1359999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.3520000000000001,"15min":1.2030000000000001},"min":{"1min":0.61199999999999999,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":1.278,"5min":12.201000000000001,"15min":12.728},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.997,"5min":1.22,"15min":1.137},"min":{"1min":0.64600000000000002,"5min":0.29299999999999998,"15min":0.27200000000000002},"max":{"1min":1.3100000000000001,"5min":7.6470000000000002,"15min":10.943},"last":1.3919999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:23:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.385,"15min":1.0609999999999999},"min":{"1min":0.44400000000000001,"5min":0.34399999999999997,"15min":0.26400000000000001},"max":{"1min":3.6930000000000001,"5min":17.936,"15min":17.936},"last":1.1799999999999999},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.869,"15min":1.212},"min":{"1min":0.31,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.0229999999999999,"5min":14.766999999999999,"15min":14.766999999999999},"last":0.96199999999999997}]},{"osd":7,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.617,"15min":1.246},"min":{"1min":0.81699999999999995,"5min":0.33700000000000002,"15min":0.29499999999999998},"max":{"1min":1.6339999999999999,"5min":28.178999999999998,"15min":28.178999999999998},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.72,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":1.391,"5min":28.222999999999999,"15min":28.222999999999999},"last":1.4690000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575223,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946308,"kb_used_data":19132,"kb_used_omap":0,"kb_used_meta":927168,"kb_avail":93425532,"statfs":{"total":96636764160,"available":95667744768,"internally_reserved":0,"allocated":19591168,"data_stored":19124450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":949420032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.88900000000000001,"15min":0.98299999999999998},"min":{"1min":0.503,"5min":0.19700000000000001,"15min":0.187},"max":{"1min":1.1020000000000001,"5min":11.629,"15min":20.922999999999998},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73599999999999999,"15min":0.97799999999999998},"min":{"1min":0.52400000000000002,"5min":0.247,"15min":0.217},"max":{"1min":1.0920000000000001,"5min":7.0830000000000002,"15min":20.532},"last":0.52400000000000002}]},{"osd":1,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.879,"15min":0.98399999999999999},"min":{"1min":0.61399999999999999,"5min":0.22,"15min":0.22},"max":{"1min":1.1819999999999999,"5min":3.0169999999999999,"15min":20.914000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.92700000000000005,"15min":1.167},"min":{"1min":0.64600000000000002,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":1.216,"5min":6.1890000000000001,"15min":23.265999999999998},"last":0.80500000000000005}]},{"osd":2,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.84799999999999998,"15min":1.117},"min":{"1min":0.63900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.181,"5min":7.1550000000000002,"15min":20.774000000000001},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.1899999999999999,"15min":1.2529999999999999},"min":{"1min":0.65400000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.179,"5min":7.9009999999999998,"15min":20.817},"last":1.1950000000000001}]},{"osd":3,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1779999999999999,"15min":1.169},"min":{"1min":0.64800000000000002,"5min":0.314,"15min":0.28799999999999998},"max":{"1min":1.1879999999999999,"5min":7.3620000000000001,"15min":24.613},"last":1.149},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.92900000000000005,"15min":1.1499999999999999},"min":{"1min":0.69699999999999995,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":1.2,"5min":7.2009999999999996,"15min":24.071000000000002},"last":0.84399999999999997}]},{"osd":5,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.153,"15min":0.93000000000000005},"min":{"1min":0.623,"5min":0.28599999999999998,"15min":0.23999999999999999},"max":{"1min":1.2170000000000001,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.0189999999999999,"15min":0.94099999999999995},"min":{"1min":0.65000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0669999999999999,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.97699999999999998}]},{"osd":6,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.1839999999999999,"15min":1.0800000000000001},"min":{"1min":0.73599999999999999,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":1.175,"5min":4.6550000000000002,"15min":6.5220000000000002},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0329999999999999,"15min":0.97199999999999998},"min":{"1min":0.81599999999999995,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":1.234,"5min":7.2389999999999999,"15min":7.2389999999999999},"last":1.0660000000000001}]},{"osd":7,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.3009999999999999,"15min":1.0700000000000001},"min":{"1min":0.72599999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.1990000000000001,"5min":7.4020000000000001,"15min":7.9850000000000003},"last":0.874},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.4910000000000001,"15min":1.163},"min":{"1min":0.98799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.2909999999999999,"5min":7.444,"15min":7.444},"last":1.099}]}]},{"osd":3,"up_from":13,"seq":55834575224,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160748,"kb_used_data":9252,"kb_used_omap":0,"kb_used_meta":1151488,"kb_avail":93211092,"statfs":{"total":96636764160,"available":95448158208,"internally_reserved":0,"allocated":9474048,"data_stored":9018997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1179123712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.46400000000000002,"15min":0.78800000000000003},"min":{"1min":0.54000000000000004,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.82599999999999996,"5min":1.3899999999999999,"15min":13.327999999999999},"last":0.441},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.58899999999999997,"15min":0.73699999999999999},"min":{"1min":0.48199999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.2170000000000001,"5min":1.444,"15min":5.4580000000000002},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.67700000000000005,"15min":0.86799999999999999},"min":{"1min":0.41499999999999998,"5min":0.20200000000000001,"15min":0.19900000000000001},"max":{"1min":0.93400000000000005,"5min":4.4459999999999997,"15min":13.831},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.749,"5min":0.72299999999999998,"15min":0.76700000000000002},"min":{"1min":0.441,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.92200000000000004,"5min":4.2130000000000001,"15min":4.4500000000000002},"last":0.67700000000000005}]},{"osd":2,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.66300000000000003,"15min":0.875},"min":{"1min":0.504,"5min":0.223,"15min":0.221},"max":{"1min":0.97699999999999998,"5min":2.7890000000000001,"15min":8.2520000000000007},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.71699999999999997,"15min":0.91000000000000003},"min":{"1min":0.35899999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0720000000000001,"5min":2.7360000000000002,"15min":6.6840000000000002},"last":0.53300000000000003}]},{"osd":4,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90700000000000003,"15min":1.206},"min":{"1min":0.56200000000000006,"5min":0.316,"15min":0.26800000000000002},"max":{"1min":1.1419999999999999,"5min":3.9449999999999998,"15min":39.070999999999998},"last":1.149},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.2210000000000001,"15min":1.2589999999999999},"min":{"1min":0.65100000000000002,"5min":0.28599999999999998,"15min":0.25700000000000001},"max":{"1min":1.175,"5min":7.6180000000000003,"15min":55.104999999999997},"last":1.077}]},{"osd":5,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":0.94999999999999996,"15min":1.1299999999999999},"min":{"1min":0.68100000000000005,"5min":0.312,"15min":0.28399999999999997},"max":{"1min":1.597,"5min":4.1849999999999996,"15min":31.481999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.028,"15min":1.2210000000000001},"min":{"1min":0.81999999999999995,"5min":0.38300000000000001,"15min":0.32500000000000001},"max":{"1min":1.5580000000000001,"5min":4.2060000000000004,"15min":31.512},"last":1.1839999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.3140000000000001,"15min":1.252},"min":{"1min":0.72199999999999998,"5min":0.311,"15min":0.27800000000000002},"max":{"1min":1.5109999999999999,"5min":15.824,"15min":25.809999999999999},"last":1.105},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.117,"15min":1.1830000000000001},"min":{"1min":0.873,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.518,"5min":4.7779999999999996,"15min":24.036999999999999},"last":1.004}]},{"osd":7,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.0089999999999999,"15min":1.2330000000000001},"min":{"1min":0.83599999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.5840000000000001,"5min":4.8529999999999998,"15min":27.581},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.077,"15min":1.252},"min":{"1min":0.76500000000000001,"5min":0.374,"15min":0.374},"max":{"1min":1.5600000000000001,"5min":8.5259999999999998,"15min":24.332000000000001},"last":1.2310000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575224,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946060,"kb_used_data":5060,"kb_used_omap":0,"kb_used_meta":940992,"kb_avail":93425780,"statfs":{"total":96636764160,"available":95667998720,"internally_reserved":0,"allocated":5181440,"data_stored":4739422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":963575808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61399999999999999,"15min":0.98999999999999999},"min":{"1min":0.41199999999999998,"5min":0.16,"15min":0.16},"max":{"1min":1.204,"5min":5.29,"15min":29.344000000000001},"last":0.627},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.68000000000000005,"15min":0.84599999999999997},"min":{"1min":0.51800000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.0589999999999999,"5min":6.4290000000000003,"15min":10.755000000000001},"last":0.71899999999999997}]},{"osd":1,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.66600000000000004,"15min":0.93799999999999994},"min":{"1min":0.49299999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":1.032,"5min":3.0409999999999999,"15min":18.617999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.69799999999999995,"15min":0.879},"min":{"1min":0.44700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.2909999999999999,"5min":3.649,"15min":15.064},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.76100000000000001,"15min":0.90200000000000002},"min":{"1min":0.45100000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.93899999999999995,"5min":13.034000000000001,"15min":13.034000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.78600000000000003,"15min":1.006},"min":{"1min":0.59199999999999997,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3,"5min":3.1339999999999999,"15min":18.547999999999998},"last":0.95599999999999996}]},{"osd":4,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0700000000000001,"15min":1.1950000000000001},"min":{"1min":0.69499999999999995,"5min":0.252,"15min":0.252},"max":{"1min":1.377,"5min":7.6500000000000004,"15min":33.378999999999998},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.135,"15min":1.2889999999999999},"min":{"1min":0.79000000000000004,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.464,"5min":7.6779999999999999,"15min":33.558999999999997},"last":1.2310000000000001}]},{"osd":5,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.069,"15min":1.472},"min":{"1min":0.71699999999999997,"5min":0.32000000000000001,"15min":0.29099999999999998},"max":{"1min":1.3460000000000001,"5min":6.3979999999999997,"15min":34.545999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.94099999999999995,"15min":1.177},"min":{"1min":0.64600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.343,"5min":4.4829999999999997,"15min":36.104999999999997},"last":1.089}]},{"osd":6,"last update":"Wed Oct 25 00:23:48 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.03,"15min":1.1140000000000001},"min":{"1min":0.81499999999999995,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.224,"5min":7.5030000000000001,"15min":33.661999999999999},"last":1.006},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.105,"15min":1.198},"min":{"1min":0.97999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":1.417,"5min":7.5490000000000004,"15min":34.043999999999997},"last":1.1519999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.1319999999999999,"15min":1.234},"min":{"1min":0.80400000000000005,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4239999999999999,"5min":7.5899999999999999,"15min":33.307000000000002},"last":1.282},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2,"15min":1.306},"min":{"1min":0.752,"5min":0.371,"15min":0.371},"max":{"1min":1.6579999999999999,"5min":7.6189999999999998,"15min":33.344999999999999},"last":1.323}]}]},{"osd":1,"up_from":13,"seq":55834575225,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054144,"kb_used_data":4088,"kb_used_omap":0,"kb_used_meta":1050048,"kb_avail":93317696,"statfs":{"total":96636764160,"available":95557320704,"internally_reserved":0,"allocated":4186112,"data_stored":3743339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1075249152},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.59099999999999997,"15min":0.82699999999999996},"min":{"1min":0.40799999999999997,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.0129999999999999,"5min":3.6110000000000002,"15min":21.334},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.71599999999999997,"15min":0.85899999999999999},"min":{"1min":0.26200000000000001,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":1.0549999999999999,"5min":3.4809999999999999,"15min":21.276},"last":1.0549999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.67400000000000004,"15min":0.81000000000000005},"min":{"1min":0.48599999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.95199999999999996,"5min":8.1600000000000001,"15min":11.420999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.68400000000000005,"15min":0.83999999999999997},"min":{"1min":0.51600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.0009999999999999,"5min":3.1760000000000002,"15min":11.648},"last":0.89600000000000002}]},{"osd":3,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78600000000000003,"15min":0.92500000000000004},"min":{"1min":0.58699999999999997,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":1.1299999999999999,"5min":3.46,"15min":14.346},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.88800000000000001,"15min":1.026},"min":{"1min":0.71099999999999997,"5min":0.26300000000000001,"15min":0.248},"max":{"1min":1.2090000000000001,"5min":3.4969999999999999,"15min":14.379},"last":1.2090000000000001}]},{"osd":4,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.038,"15min":1.0589999999999999},"min":{"1min":0.51000000000000001,"5min":0.28000000000000003,"15min":0.27200000000000002},"max":{"1min":1.093,"5min":5.2409999999999997,"15min":10.582000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.038,"5min":1.0800000000000001,"15min":1.0580000000000001},"min":{"1min":0.82899999999999996,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":1.246,"5min":4.6630000000000003,"15min":11.173},"last":0.97999999999999998}]},{"osd":5,"last update":"Wed Oct 25 00:22:57 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.0860000000000001,"15min":1.042},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.248,"5min":10.446,"15min":11.577999999999999},"last":1.159},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.93700000000000006,"15min":0.96799999999999997},"min":{"1min":0.52100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.2869999999999999,"5min":5.3209999999999997,"15min":10.018000000000001},"last":1.2290000000000001}]},{"osd":6,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.999,"15min":1.028},"min":{"1min":0.71999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.1739999999999999,"5min":7.4880000000000004,"15min":11.845000000000001},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.095,"5min":1.0860000000000001,"15min":1.125},"min":{"1min":0.77100000000000002,"5min":0.38300000000000001,"15min":0.33800000000000002},"max":{"1min":1.262,"5min":7.5220000000000002,"15min":11.919},"last":1.262}]},{"osd":7,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.1839999999999999,"15min":1.1279999999999999},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":1.2749999999999999,"5min":7.9909999999999997,"15min":11.881},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.95799999999999996,"15min":1.1439999999999999},"min":{"1min":0.56899999999999995,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4610000000000001,"5min":4.2300000000000004,"15min":10.478},"last":1.3120000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575225,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":966376,"kb_used_data":22112,"kb_used_omap":0,"kb_used_meta":944256,"kb_avail":93405464,"statfs":{"total":96636764160,"available":95647195136,"internally_reserved":0,"allocated":22642688,"data_stored":22192198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":966918144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60799999999999998,"15min":1},"min":{"1min":0.40200000000000002,"5min":0.18099999999999999,"15min":0.16900000000000001},"max":{"1min":0.90800000000000003,"5min":2.544,"15min":30.065999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.63500000000000001,"15min":0.997},"min":{"1min":0.503,"5min":0.184,"15min":0.182},"max":{"1min":0.90700000000000003,"5min":3.0030000000000001,"15min":19.757999999999999},"last":0.623}]},{"osd":2,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73099999999999998,"15min":0.79700000000000004},"min":{"1min":0.46800000000000003,"5min":0.22900000000000001,"15min":0.20399999999999999},"max":{"1min":0.996,"5min":5.9740000000000002,"15min":15.324999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.72299999999999998,"15min":0.96999999999999997},"min":{"1min":0.626,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.1040000000000001,"5min":4.2549999999999999,"15min":30.283999999999999},"last":0.74399999999999999}]},{"osd":3,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.84399999999999997,"15min":1.1040000000000001},"min":{"1min":0.59699999999999998,"5min":0.20999999999999999,"15min":0.191},"max":{"1min":1.1060000000000001,"5min":6.3570000000000002,"15min":22.966999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.76400000000000001,"15min":1.083},"min":{"1min":0.58899999999999997,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":1.2090000000000001,"5min":2.5619999999999998,"15min":19.460999999999999},"last":0.86699999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.89100000000000001,"15min":1.1279999999999999},"min":{"1min":0.63200000000000001,"5min":0.33700000000000002,"15min":0.26600000000000001},"max":{"1min":1.2370000000000001,"5min":3.758,"15min":8.3670000000000009},"last":1.337},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.016,"15min":1.2310000000000001},"min":{"1min":0.67800000000000005,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.286,"5min":5.3719999999999999,"15min":13.023999999999999},"last":1.3779999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.96999999999999997,"15min":1.0229999999999999},"min":{"1min":0.751,"5min":0.30399999999999999,"15min":0.29099999999999998},"max":{"1min":1.04,"5min":2.9710000000000001,"15min":10.683999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.097,"5min":1.01,"15min":1},"min":{"1min":0.79100000000000004,"5min":0.34999999999999998,"15min":0.29899999999999999},"max":{"1min":1.5469999999999999,"5min":4.8860000000000001,"15min":15.436999999999999},"last":0.95699999999999996}]},{"osd":6,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.98899999999999999,"15min":1.1339999999999999},"min":{"1min":0.78200000000000003,"5min":0.33700000000000002,"15min":0.28100000000000003},"max":{"1min":1.7969999999999999,"5min":6.1559999999999997,"15min":8.7260000000000009},"last":1.155},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.125,"15min":1.294},"min":{"1min":0.86599999999999999,"5min":0.38200000000000001,"15min":0.33600000000000002},"max":{"1min":1.6379999999999999,"5min":7.8970000000000002,"15min":15.375},"last":1.2909999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.2509999999999999,"15min":1.159},"min":{"1min":0.80000000000000004,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":1.202,"5min":5.5090000000000003,"15min":10.81},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.038,"15min":1.27},"min":{"1min":0.91800000000000004,"5min":0.313,"15min":0.313},"max":{"1min":1.506,"5min":4.2629999999999999,"15min":11.417},"last":1.226}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12673024,"data_stored":12668140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3198976,"data_stored":3198201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11689984,"data_stored":11672183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4195192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":8388642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4279555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6918144,"data_stored":6907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6836224,"data_stored":6822812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-25T00:23:59.520 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.520 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.521 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.521 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.522 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.522 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.523 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.523 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.524 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.524 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.525 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.525 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.525 INFO:tasks.ceph:pgid 6.2a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.526 INFO:tasks.ceph:pgid 6.2b last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.526 INFO:tasks.ceph:pgid 4.2a last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.527 INFO:tasks.ceph:pgid 6.32 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.527 INFO:tasks.ceph:pgid 6.33 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.527 INFO:tasks.ceph:pgid 6.30 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.528 INFO:tasks.ceph:pgid 4.32 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:23:59.528 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-25T00:24:15.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5baff4a640 1 Processor -- start 2023-10-25T00:24:15.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5baff4a640 1 -- start start 2023-10-25T00:24:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5baff4a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5ba8151d60 0x7f5ba80a9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5baff4a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ba806a780 0x7f5ba8154d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5baff4a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ba80c24b0 0x7f5ba8155280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5baff4a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba80aadc0 con 0x7f5ba80c24b0 2023-10-25T00:24:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5baff4a640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba806afe0 con 0x7f5ba8151d60 2023-10-25T00:24:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5baff4a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba806b330 con 0x7f5ba806a780 2023-10-25T00:24:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5bae4e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ba80c24b0 0x7f5ba8155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5bae4e7640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f5ba80c24b0 0x7f5ba8155280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39902/0 (socket says 172.21.15.107:39902) 2023-10-25T00:24:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5bae4e7640 1 -- 172.21.15.107:0/3455062727 learned_addr learned my addr 172.21.15.107:0/3455062727 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5bad4e5640 1 --2- 172.21.15.107:0/3455062727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ba806a780 0x7f5ba8154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5badce6640 1 --2- 172.21.15.107:0/3455062727 >> v2:172.21.15.110:3300/0 conn(0x7f5ba8151d60 0x7f5ba80a9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5bae4e7640 1 -- 172.21.15.107:0/3455062727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ba806a780 msgr2=0x7f5ba8154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5bae4e7640 1 --2- 172.21.15.107:0/3455062727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ba806a780 0x7f5ba8154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5bae4e7640 1 -- 172.21.15.107:0/3455062727 >> v2:172.21.15.110:3300/0 conn(0x7f5ba8151d60 msgr2=0x7f5ba80a9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5bae4e7640 1 --2- 172.21.15.107:0/3455062727 >> v2:172.21.15.110:3300/0 conn(0x7f5ba8151d60 0x7f5ba80a9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.590+0000 7f5bae4e7640 1 -- 172.21.15.107:0/3455062727 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba81557c0 con 0x7f5ba80c24b0 2023-10-25T00:24:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bae4e7640 1 --2- 172.21.15.107:0/3455062727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ba80c24b0 0x7f5ba8155280 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f5ba806d3e0 tx=0x7f5ba001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987ef5580c415b67 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bacce4640 1 -- 172.21.15.107:0/3455062727 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba00a5650 con 0x7f5ba80c24b0 2023-10-25T00:24:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bacce4640 1 -- 172.21.15.107:0/3455062727 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba00a57b0 con 0x7f5ba80c24b0 2023-10-25T00:24:15.598 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bacce4640 1 -- 172.21.15.107:0/3455062727 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba00a5b40 con 0x7f5ba80c24b0 2023-10-25T00:24:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 -- 172.21.15.107:0/3455062727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ba80c24b0 msgr2=0x7f5ba8155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 --2- 172.21.15.107:0/3455062727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ba80c24b0 0x7f5ba8155280 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f5ba806d3e0 tx=0x7f5ba001c9d0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 -- 172.21.15.107:0/3455062727 shutdown_connections 2023-10-25T00:24:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 --2- 172.21.15.107:0/3455062727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ba80c24b0 0x7f5ba8155280 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 --2- 172.21.15.107:0/3455062727 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ba806a780 0x7f5ba8154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 --2- 172.21.15.107:0/3455062727 >> v2:172.21.15.110:3300/0 conn(0x7f5ba8151d60 0x7f5ba80a9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 -- 172.21.15.107:0/3455062727 >> 172.21.15.107:0/3455062727 conn(0x7f5ba80c2950 msgr2=0x7f5ba80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 -- 172.21.15.107:0/3455062727 shutdown_connections 2023-10-25T00:24:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 -- 172.21.15.107:0/3455062727 wait complete. 2023-10-25T00:24:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 Processor -- start 2023-10-25T00:24:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 -- start start 2023-10-25T00:24:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ba80c24b0 0x7f5ba8146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ba8151d60 0x7f5ba8148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ba81486c0 0x7f5ba80715c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:15.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba8156bb0 con 0x7f5ba8151d60 2023-10-25T00:24:15.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba8157a20 con 0x7f5ba80c24b0 2023-10-25T00:24:15.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5baff4a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba8155fa0 con 0x7f5ba81486c0 2023-10-25T00:24:15.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bad4e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ba8151d60 0x7f5ba8148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:15.604 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bad4e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ba8151d60 0x7f5ba8148180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39906/0 (socket says 172.21.15.107:39906) 2023-10-25T00:24:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bad4e5640 1 -- 172.21.15.107:0/1561707199 learned_addr learned my addr 172.21.15.107:0/1561707199 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bae4e7640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ba81486c0 0x7f5ba80715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:15.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5badce6640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ba80c24b0 0x7f5ba8146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bad4e5640 1 -- 172.21.15.107:0/1561707199 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ba81486c0 msgr2=0x7f5ba80715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bad4e5640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ba81486c0 0x7f5ba80715c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bad4e5640 1 -- 172.21.15.107:0/1561707199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ba80c24b0 msgr2=0x7f5ba8146ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bad4e5640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ba80c24b0 0x7f5ba8146ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5bad4e5640 1 -- 172.21.15.107:0/1561707199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba0009b30 con 0x7f5ba8151d60 2023-10-25T00:24:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.594+0000 7f5badce6640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ba80c24b0 0x7f5ba8146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:24:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5bad4e5640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ba8151d60 0x7f5ba8148180 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800d830 tx=0x7f5b9800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5b9e7fc640 1 -- 172.21.15.107:0/1561707199 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b980096a0 con 0x7f5ba8151d60 2023-10-25T00:24:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5b9e7fc640 1 -- 172.21.15.107:0/1561707199 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b98009800 con 0x7f5ba8151d60 2023-10-25T00:24:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5b9e7fc640 1 -- 172.21.15.107:0/1561707199 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b98009b70 con 0x7f5ba8151d60 2023-10-25T00:24:15.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5baff4a640 1 -- 172.21.15.107:0/1561707199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba8156280 con 0x7f5ba8151d60 2023-10-25T00:24:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5baff4a640 1 -- 172.21.15.107:0/1561707199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba8156840 con 0x7f5ba8151d60 2023-10-25T00:24:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5baff4a640 1 -- 172.21.15.107:0/1561707199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba8156d60 con 0x7f5ba8151d60 2023-10-25T00:24:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5b9e7fc640 1 -- 172.21.15.107:0/1561707199 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f5b98014410 con 0x7f5ba8151d60 2023-10-25T00:24:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5b9e7fc640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b8c065e90 0x7f5b8c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:15.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5badce6640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b8c065e90 0x7f5b8c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:15.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5b9e7fc640 1 -- 172.21.15.107:0/1561707199 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f5b98085390 con 0x7f5ba8151d60 2023-10-25T00:24:15.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.598+0000 7f5badce6640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b8c065e90 0x7f5b8c068350 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8147650 tx=0x7f5ba4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:15.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.602+0000 7f5b9e7fc640 1 -- 172.21.15.107:0/1561707199 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f5ba8156d60 con 0x7f5ba8151d60 2023-10-25T00:24:15.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.790+0000 7f5baff4a640 1 -- 172.21.15.107:0/1561707199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ba8156d60 con 0x7f5ba8151d60 2023-10-25T00:24:15.795 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5b9e7fc640 1 -- 172.21.15.107:0/1561707199 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5769 (secure 0 0 0) 0x7f5ba8156d60 con 0x7f5ba8151d60 2023-10-25T00:24:15.796 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:15.796 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:15.796 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 235 2023-10-25T00:24:15.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 -- 172.21.15.107:0/1561707199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b8c065e90 msgr2=0x7f5b8c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:15.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b8c065e90 0x7f5b8c068350 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8147650 tx=0x7f5ba4008040 comp rx=0 tx=0).stop 2023-10-25T00:24:15.798 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 -- 172.21.15.107:0/1561707199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ba8151d60 msgr2=0x7f5ba8148180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:15.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ba8151d60 0x7f5ba8148180 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f5b9800d830 tx=0x7f5b9800dcf0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 -- 172.21.15.107:0/1561707199 shutdown_connections 2023-10-25T00:24:15.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f5b8c065e90 0x7f5b8c068350 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5ba81486c0 0x7f5ba80715c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ba8151d60 0x7f5ba8148180 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 --2- 172.21.15.107:0/1561707199 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5ba80c24b0 0x7f5ba8146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 -- 172.21.15.107:0/1561707199 >> 172.21.15.107:0/1561707199 conn(0x7f5ba80c2950 msgr2=0x7f5ba8152ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:15.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 -- 172.21.15.107:0/1561707199 shutdown_connections 2023-10-25T00:24:15.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.794+0000 7f5baff4a640 1 -- 172.21.15.107:0/1561707199 wait complete. 2023-10-25T00:24:15.815 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2023-10-25T00:24:15.815 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:24:15.815 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:24:15.826 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:24:15.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:15.990 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.986+0000 7f38c5097640 1 Processor -- start 2023-10-25T00:24:15.991 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.986+0000 7f38c5097640 1 -- start start 2023-10-25T00:24:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38c5097640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c0152040 0x7f38c00c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38c5097640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f38c006a4b0 0x7f38c00ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38c5097640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c0155680 con 0x7f38c00c8100 2023-10-25T00:24:15.992 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38c5097640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c0154a70 con 0x7f38c006a4b0 2023-10-25T00:24:15.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38c5097640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c0154d50 con 0x7f38c0152040 2023-10-25T00:24:15.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c0152040 0x7f38c00c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:15.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bf59e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f38c006a4b0 0x7f38c00ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:15.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c0152040 0x7f38c00c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.107:37244/0 (socket says 172.21.15.107:37244) 2023-10-25T00:24:15.993 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bed9d640 1 -- 172.21.15.107:0/533290604 learned_addr learned my addr 172.21.15.107:0/533290604 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:15.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bed9d640 1 -- 172.21.15.107:0/533290604 >> v2:172.21.15.110:3300/0 conn(0x7f38c006a4b0 msgr2=0x7f38c00ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:15.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bed9d640 1 --2- 172.21.15.107:0/533290604 >> v2:172.21.15.110:3300/0 conn(0x7f38c006a4b0 0x7f38c00ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bed9d640 1 -- 172.21.15.107:0/533290604 >> v1:172.21.15.110:6789/0 conn(0x7f38c00c8100 legacy=0x7f38c00abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:15.994 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bed9d640 1 -- 172.21.15.107:0/533290604 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c0155030 con 0x7f38c0152040 2023-10-25T00:24:15.995 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bed9d640 1 --2- 172.21.15.107:0/533290604 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c0152040 0x7f38c00c4b80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f38b0009870 tx=0x7f38b001c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e35cbae72c75f720 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bdd9b640 1 -- 172.21.15.107:0/533290604 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b00ac620 con 0x7f38c0152040 2023-10-25T00:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bdd9b640 1 -- 172.21.15.107:0/533290604 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b00ac780 con 0x7f38c0152040 2023-10-25T00:24:15.996 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38bdd9b640 1 -- 172.21.15.107:0/533290604 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b00acb10 con 0x7f38c0152040 2023-10-25T00:24:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38c5097640 1 -- 172.21.15.107:0/533290604 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c0152040 msgr2=0x7f38c00c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.990+0000 7f38c5097640 1 --2- 172.21.15.107:0/533290604 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c0152040 0x7f38c00c4b80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f38b0009870 tx=0x7f38b001c4b0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 -- 172.21.15.107:0/533290604 shutdown_connections 2023-10-25T00:24:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 --2- 172.21.15.107:0/533290604 >> v2:172.21.15.110:3300/0 conn(0x7f38c006a4b0 0x7f38c00ac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 --2- 172.21.15.107:0/533290604 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c0152040 0x7f38c00c4b80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 -- 172.21.15.107:0/533290604 >> 172.21.15.107:0/533290604 conn(0x7f38c00ca200 msgr2=0x7f38c00ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 -- 172.21.15.107:0/533290604 shutdown_connections 2023-10-25T00:24:15.998 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 -- 172.21.15.107:0/533290604 wait complete. 2023-10-25T00:24:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 Processor -- start 2023-10-25T00:24:16.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 -- start start 2023-10-25T00:24:16.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c006a4b0 0x7f38c0151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:16.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c00c8100 0x7f38c01f0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:16.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38c0152040 0x7f38c01f0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:16.001 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c014f160 con 0x7f38c0152040 2023-10-25T00:24:16.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c01510f0 con 0x7f38c00c8100 2023-10-25T00:24:16.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c014d8d0 con 0x7f38c006a4b0 2023-10-25T00:24:16.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38bf59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38c0152040 0x7f38c01f0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:16.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38bf59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38c0152040 0x7f38c01f0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39918/0 (socket says 172.21.15.107:39918) 2023-10-25T00:24:16.002 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38bf59e640 1 -- 172.21.15.107:0/640336433 learned_addr learned my addr 172.21.15.107:0/640336433 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38bed9d640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c006a4b0 0x7f38c0151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38be59c640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c00c8100 0x7f38c01f0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38bf59e640 1 -- 172.21.15.107:0/640336433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c006a4b0 msgr2=0x7f38c0151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38bf59e640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c006a4b0 0x7f38c0151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38bf59e640 1 -- 172.21.15.107:0/640336433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c00c8100 msgr2=0x7f38c01f0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:16.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38bf59e640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c00c8100 0x7f38c01f0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:16.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38bf59e640 1 -- 172.21.15.107:0/640336433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b00094d0 con 0x7f38c0152040 2023-10-25T00:24:16.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38be59c640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c00c8100 0x7f38c01f0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:24:16.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38bf59e640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38c0152040 0x7f38c01f0b50 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f38b800efc0 tx=0x7f38b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:16.004 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38a77fe640 1 -- 172.21.15.107:0/640336433 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b8009ba0 con 0x7f38c0152040 2023-10-25T00:24:16.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38a77fe640 1 -- 172.21.15.107:0/640336433 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b800f040 con 0x7f38c0152040 2023-10-25T00:24:16.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 -- 172.21.15.107:0/640336433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c00ac150 con 0x7f38c0152040 2023-10-25T00:24:16.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38a77fe640 1 -- 172.21.15.107:0/640336433 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b80146c0 con 0x7f38c0152040 2023-10-25T00:24:16.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 -- 172.21.15.107:0/640336433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c01f11a0 con 0x7f38c0152040 2023-10-25T00:24:16.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.994+0000 7f38c5097640 1 -- 172.21.15.107:0/640336433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c01512a0 con 0x7f38c0152040 2023-10-25T00:24:16.005 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.998+0000 7f38a77fe640 1 -- 172.21.15.107:0/640336433 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f38b8014820 con 0x7f38c0152040 2023-10-25T00:24:16.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.998+0000 7f38a77fe640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ac065e60 0x7f38ac068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:16.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.998+0000 7f38bed9d640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ac065e60 0x7f38ac068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:16.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.998+0000 7f38a77fe640 1 -- 172.21.15.107:0/640336433 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f38b80858e0 con 0x7f38c0152040 2023-10-25T00:24:16.006 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:15.998+0000 7f38bed9d640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ac065e60 0x7f38ac068320 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f38b00ae000 tx=0x7f38b00a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:16.007 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.002+0000 7f38a77fe640 1 -- 172.21.15.107:0/640336433 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f38c01512a0 con 0x7f38c0152040 2023-10-25T00:24:16.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.174+0000 7f38c5097640 1 -- 172.21.15.107:0/640336433 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38c01512a0 con 0x7f38c0152040 2023-10-25T00:24:16.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.174+0000 7f38a77fe640 1 -- 172.21.15.107:0/640336433 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5769 (secure 0 0 0) 0x7f38c01512a0 con 0x7f38c0152040 2023-10-25T00:24:16.178 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:16.178 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:16.179 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 235 2023-10-25T00:24:16.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 -- 172.21.15.107:0/640336433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ac065e60 msgr2=0x7f38ac068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:16.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ac065e60 0x7f38ac068320 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f38b00ae000 tx=0x7f38b00a7000 comp rx=0 tx=0).stop 2023-10-25T00:24:16.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 -- 172.21.15.107:0/640336433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38c0152040 msgr2=0x7f38c01f0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:16.180 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38c0152040 0x7f38c01f0b50 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f38b800efc0 tx=0x7f38b800c490 comp rx=0 tx=0).stop 2023-10-25T00:24:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 -- 172.21.15.107:0/640336433 shutdown_connections 2023-10-25T00:24:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38c0152040 0x7f38c01f0b50 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38c00c8100 0x7f38c01f0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f38ac065e60 0x7f38ac068320 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:16.181 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 --2- 172.21.15.107:0/640336433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f38c006a4b0 0x7f38c0151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:16.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 -- 172.21.15.107:0/640336433 >> 172.21.15.107:0/640336433 conn(0x7f38c00ca200 msgr2=0x7f38c0154180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:16.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 -- 172.21.15.107:0/640336433 shutdown_connections 2023-10-25T00:24:16.182 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:16.178+0000 7f38c5097640 1 -- 172.21.15.107:0/640336433 wait complete. 2023-10-25T00:24:16.195 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:24:18.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:18.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.325+0000 7fb838e57640 1 Processor -- start 2023-10-25T00:24:18.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.325+0000 7fb838e57640 1 -- start start 2023-10-25T00:24:18.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb838e57640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb834152060 0x7fb8340aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:18.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb838e57640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8340c2990 0x7fb8340a9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:18.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb838e57640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8340a9aa0 con 0x7fb8340aa470 2023-10-25T00:24:18.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb838e57640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb834155480 con 0x7fb8340c2990 2023-10-25T00:24:18.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb838e57640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb834155760 con 0x7fb834152060 2023-10-25T00:24:18.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb831d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8340c2990 0x7fb8340a9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:18.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb832d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fb8340aa470 0x7fb8340c1480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57008/0 (socket says 172.21.15.107:57008) 2023-10-25T00:24:18.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb831d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fb8340c2990 0x7fb8340a9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39920/0 (socket says 172.21.15.107:39920) 2023-10-25T00:24:18.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb832d9d640 1 -- 172.21.15.107:0/1329683704 learned_addr learned my addr 172.21.15.107:0/1329683704 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:24:18.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb83259c640 1 --2- 172.21.15.107:0/1329683704 >> v2:172.21.15.110:3300/0 conn(0x7fb834152060 0x7fb8340aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:18.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb831d9b640 1 -- 172.21.15.107:0/1329683704 >> v2:172.21.15.110:3300/0 conn(0x7fb834152060 msgr2=0x7fb8340aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:18.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb831d9b640 1 --2- 172.21.15.107:0/1329683704 >> v2:172.21.15.110:3300/0 conn(0x7fb834152060 0x7fb8340aada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:18.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb831d9b640 1 -- 172.21.15.107:0/1329683704 >> v1:172.21.15.107:6789/0 conn(0x7fb8340aa470 legacy=0x7fb8340c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:18.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb831d9b640 1 -- 172.21.15.107:0/1329683704 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8340a9450 con 0x7fb8340c2990 2023-10-25T00:24:18.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb83259c640 1 --2- 172.21.15.107:0/1329683704 >> v2:172.21.15.110:3300/0 conn(0x7fb834152060 0x7fb8340aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:24:18.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb831d9b640 1 --2- 172.21.15.107:0/1329683704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8340c2990 0x7fb8340a9f30 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fb834073d70 tx=0x7fb82801c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f25644f63257825 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:18.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb83159a640 1 -- 172.21.15.107:0/1329683704 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8280ab600 con 0x7fb8340c2990 2023-10-25T00:24:18.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb83159a640 1 -- 172.21.15.107:0/1329683704 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8280a7050 con 0x7fb8340c2990 2023-10-25T00:24:18.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb83159a640 1 -- 172.21.15.107:0/1329683704 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8280ab980 con 0x7fb8340c2990 2023-10-25T00:24:18.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb838e57640 1 -- 172.21.15.107:0/1329683704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8340c2990 msgr2=0x7fb8340a9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:18.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.329+0000 7fb838e57640 1 --2- 172.21.15.107:0/1329683704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8340c2990 0x7fb8340a9f30 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fb834073d70 tx=0x7fb82801c310 comp rx=0 tx=0).stop 2023-10-25T00:24:18.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 -- 172.21.15.107:0/1329683704 shutdown_connections 2023-10-25T00:24:18.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 --2- 172.21.15.107:0/1329683704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8340c2990 0x7fb8340a9f30 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:18.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 --2- 172.21.15.107:0/1329683704 >> v2:172.21.15.110:3300/0 conn(0x7fb834152060 0x7fb8340aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:18.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 -- 172.21.15.107:0/1329683704 >> 172.21.15.107:0/1329683704 conn(0x7fb8340c4a90 msgr2=0x7fb8340c4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:18.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 -- 172.21.15.107:0/1329683704 shutdown_connections 2023-10-25T00:24:18.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 -- 172.21.15.107:0/1329683704 wait complete. 2023-10-25T00:24:18.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 Processor -- start 2023-10-25T00:24:18.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 -- start start 2023-10-25T00:24:18.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340aa470 0x7fb83414feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:18.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8340c2990 0x7fb834149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:18.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834152060 0x7fb83414a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:18.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8340aa240 con 0x7fb834152060 2023-10-25T00:24:18.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8341ed3b0 con 0x7fb8340aa470 2023-10-25T00:24:18.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb83414dbe0 con 0x7fb8340c2990 2023-10-25T00:24:18.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb832d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834152060 0x7fb83414a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:18.342 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb832d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834152060 0x7fb83414a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39922/0 (socket says 172.21.15.107:39922) 2023-10-25T00:24:18.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb832d9d640 1 -- 172.21.15.107:0/96149846 learned_addr learned my addr 172.21.15.107:0/96149846 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:18.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb83259c640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340aa470 0x7fb83414feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:18.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb832d9d640 1 -- 172.21.15.107:0/96149846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8340c2990 msgr2=0x7fb834149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:18.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb831d9b640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8340c2990 0x7fb834149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:18.343 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb832d9d640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8340c2990 0x7fb834149fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:18.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb832d9d640 1 -- 172.21.15.107:0/96149846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340aa470 msgr2=0x7fb83414feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:18.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb832d9d640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340aa470 0x7fb83414feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:18.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb832d9d640 1 -- 172.21.15.107:0/96149846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb828009420 con 0x7fb834152060 2023-10-25T00:24:18.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb832d9d640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834152060 0x7fb83414a4f0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00caa0 tx=0x7fb82c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:18.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb822ffd640 1 -- 172.21.15.107:0/96149846 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c00ab40 con 0x7fb834152060 2023-10-25T00:24:18.344 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb822ffd640 1 -- 172.21.15.107:0/96149846 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c00aca0 con 0x7fb834152060 2023-10-25T00:24:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 -- 172.21.15.107:0/96149846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83414dec0 con 0x7fb834152060 2023-10-25T00:24:18.345 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb822ffd640 1 -- 172.21.15.107:0/96149846 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c012600 con 0x7fb834152060 2023-10-25T00:24:18.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 -- 172.21.15.107:0/96149846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8341503f0 con 0x7fb834152060 2023-10-25T00:24:18.346 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.333+0000 7fb838e57640 1 -- 172.21.15.107:0/96149846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8341ed830 con 0x7fb834152060 2023-10-25T00:24:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.337+0000 7fb822ffd640 1 -- 172.21.15.107:0/96149846 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fb82c007070 con 0x7fb834152060 2023-10-25T00:24:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.337+0000 7fb822ffd640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 0x7fb814068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.337+0000 7fb822ffd640 1 -- 172.21.15.107:0/96149846 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7fb82c0847e0 con 0x7fb834152060 2023-10-25T00:24:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.337+0000 7fb83259c640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 0x7fb814068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:18.347 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.337+0000 7fb83259c640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 0x7fb814068320 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fb824006fd0 tx=0x7fb824008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.341+0000 7fb822ffd640 1 -- 172.21.15.107:0/96149846 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fb8341ed830 con 0x7fb834152060 2023-10-25T00:24:18.533 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.529+0000 7fb838e57640 1 -- 172.21.15.107:0/96149846 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8341ed830 con 0x7fb834152060 2023-10-25T00:24:18.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.529+0000 7fb822ffd640 1 -- 172.21.15.107:0/96149846 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5769 (secure 0 0 0) 0x7fb8341ed830 con 0x7fb834152060 2023-10-25T00:24:18.534 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:18.535 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:18.535 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 235 2023-10-25T00:24:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 -- 172.21.15.107:0/96149846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 msgr2=0x7fb814068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:18.536 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 0x7fb814068320 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fb824006fd0 tx=0x7fb824008040 comp rx=0 tx=0).stop 2023-10-25T00:24:18.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 -- 172.21.15.107:0/96149846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834152060 msgr2=0x7fb83414a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:18.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834152060 0x7fb83414a4f0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00caa0 tx=0x7fb82c00cf60 comp rx=0 tx=0).stop 2023-10-25T00:24:18.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 -- 172.21.15.107:0/96149846 shutdown_connections 2023-10-25T00:24:18.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb834152060 0x7fb83414a4f0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:18.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fb814065e60 0x7fb814068320 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:18.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb8340c2990 0x7fb834149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:18.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 --2- 172.21.15.107:0/96149846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8340aa470 0x7fb83414feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:18.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 -- 172.21.15.107:0/96149846 >> 172.21.15.107:0/96149846 conn(0x7fb8340c4a90 msgr2=0x7fb8341545d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:18.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 -- 172.21.15.107:0/96149846 shutdown_connections 2023-10-25T00:24:18.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:18.533+0000 7fb838e57640 1 -- 172.21.15.107:0/96149846 wait complete. 2023-10-25T00:24:18.554 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:24:19.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-25T00:24:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b3965640 1 Processor -- start 2023-10-25T00:24:19.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b3965640 1 -- start start 2023-10-25T00:24:19.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b3965640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f79ac06ee40 0x7f79ac0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:19.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b3965640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac06f730 con 0x7f79ac0a9820 2023-10-25T00:24:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b3965640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79ac156a40 con 0x7f79ac14fd40 2023-10-25T00:24:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b3965640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79ac156710 con 0x7f79ac06ee40 2023-10-25T00:24:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b0f00640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f79ac0a9820 0x7f79ac15af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57014/0 (socket says 172.21.15.107:57014) 2023-10-25T00:24:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b0f00640 1 -- 172.21.15.107:0/6936691 learned_addr learned my addr 172.21.15.107:0/6936691 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:24:19.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b1f02640 1 --2- 172.21.15.107:0/6936691 >> v2:172.21.15.110:3300/0 conn(0x7f79ac06ee40 0x7f79ac0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786670326 0 0) 0x7f79ac06f730 con 0x7f79ac0a9820 2023-10-25T00:24:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7994003530 con 0x7f79ac0a9820 2023-10-25T00:24:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1813449001 0 0) 0x7f7994003530 con 0x7f79ac0a9820 2023-10-25T00:24:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 >> v2:172.21.15.110:3300/0 conn(0x7f79ac06ee40 msgr2=0x7f79ac0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:19.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 --2- 172.21.15.107:0/6936691 >> v2:172.21.15.110:3300/0 conn(0x7f79ac06ee40 0x7f79ac0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 >> v1:172.21.15.110:6789/0 conn(0x7f79ac14fd40 legacy=0x7f79ac0c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:19.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ac06f980 con 0x7f79ac0a9820 2023-10-25T00:24:19.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f799c001a00 con 0x7f79ac0a9820 2023-10-25T00:24:19.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 >> v1:172.21.15.107:6789/0 conn(0x7f79ac0a9820 legacy=0x7f79ac15af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:19.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 --2- 172.21.15.107:0/6936691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7994003980 0x7f7994003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:19.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 --2- 172.21.15.107:0/6936691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79940047d0 0x7f7994004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 --2- 172.21.15.107:0/6936691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79940052b0 0x7f799400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac156710 con 0x7f7994003980 2023-10-25T00:24:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799400c180 con 0x7f79940052b0 2023-10-25T00:24:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799400c340 con 0x7f79940047d0 2023-10-25T00:24:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b1701640 1 --2- 172.21.15.107:0/6936691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7994003980 0x7f7994003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b1f02640 1 --2- 172.21.15.107:0/6936691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79940052b0 0x7f799400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b1701640 1 -- 172.21.15.107:0/6936691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79940047d0 msgr2=0x7f7994004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b1701640 1 --2- 172.21.15.107:0/6936691 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f79940047d0 0x7f7994004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.668 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b1701640 1 -- 172.21.15.107:0/6936691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79940052b0 msgr2=0x7f799400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:19.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b1701640 1 --2- 172.21.15.107:0/6936691 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79940052b0 0x7f799400b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b1f02640 1 -- 172.21.15.107:0/6936691 reap_dead start 2023-10-25T00:24:19.669 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.657+0000 7f79b1701640 1 -- 172.21.15.107:0/6936691 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7994003530 con 0x7f7994003980 2023-10-25T00:24:19.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79b1701640 1 --2- 172.21.15.107:0/6936691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7994003980 0x7f7994003da0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f79a0001750 tx=0x7f79a001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed82280a4237496 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:19.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a00052d0 con 0x7f7994003980 2023-10-25T00:24:19.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a00a7050 con 0x7f7994003980 2023-10-25T00:24:19.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79abfff640 1 -- 172.21.15.107:0/6936691 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0003600 con 0x7f7994003980 2023-10-25T00:24:19.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79b3965640 1 -- 172.21.15.107:0/6936691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7994003980 msgr2=0x7f7994003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:19.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79b3965640 1 --2- 172.21.15.107:0/6936691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7994003980 0x7f7994003da0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f79a0001750 tx=0x7f79a001c3a0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79b3965640 1 -- 172.21.15.107:0/6936691 shutdown_connections 2023-10-25T00:24:19.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79b3965640 1 --2- 172.21.15.107:0/6936691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7994003980 0x7f7994003da0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79b3965640 1 -- 172.21.15.107:0/6936691 >> 172.21.15.107:0/6936691 conn(0x7f79ac0c2950 msgr2=0x7f79ac0c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:19.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79b3965640 1 -- 172.21.15.107:0/6936691 shutdown_connections 2023-10-25T00:24:19.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79b3965640 1 -- 172.21.15.107:0/6936691 wait complete. 2023-10-25T00:24:19.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79b3965640 1 Processor -- start 2023-10-25T00:24:19.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.661+0000 7f79b3965640 1 -- start start 2023-10-25T00:24:19.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b3965640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7994003980 0x7f79ac0bf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:19.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b3965640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79ac14d5d0 0x7f79ac155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:19.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b3965640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79ac0bf7c0 0x7f79ac155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:19.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b3965640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac06a1c0 con 0x7f79ac14d5d0 2023-10-25T00:24:19.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b3965640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac143c30 con 0x7f79ac0bf7c0 2023-10-25T00:24:19.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b3965640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac1edd10 con 0x7f7994003980 2023-10-25T00:24:19.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b0f00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79ac14d5d0 0x7f79ac155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:19.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b0f00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79ac14d5d0 0x7f79ac155780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39936/0 (socket says 172.21.15.107:39936) 2023-10-25T00:24:19.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b0f00640 1 -- 172.21.15.107:0/1846826881 learned_addr learned my addr 172.21.15.107:0/1846826881 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:19.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b1701640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7994003980 0x7f79ac0bf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:19.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b1f02640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79ac0bf7c0 0x7f79ac155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:19.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b0f00640 1 -- 172.21.15.107:0/1846826881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7994003980 msgr2=0x7f79ac0bf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:19.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b0f00640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7994003980 0x7f79ac0bf280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b0f00640 1 -- 172.21.15.107:0/1846826881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79ac0bf7c0 msgr2=0x7f79ac155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:19.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b0f00640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79ac0bf7c0 0x7f79ac155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b0f00640 1 -- 172.21.15.107:0/1846826881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ac1edff0 con 0x7f79ac14d5d0 2023-10-25T00:24:19.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b0f00640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79ac14d5d0 0x7f79ac155780 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f799c00c9c0 tx=0x7f799c00ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:19.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79a9ffb640 1 -- 172.21.15.107:0/1846826881 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c00a6a0 con 0x7f79ac14d5d0 2023-10-25T00:24:19.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79a9ffb640 1 -- 172.21.15.107:0/1846826881 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799c00a800 con 0x7f79ac14d5d0 2023-10-25T00:24:19.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b3965640 1 -- 172.21.15.107:0/1846826881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79ac069de0 con 0x7f79ac14d5d0 2023-10-25T00:24:19.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79a9ffb640 1 -- 172.21.15.107:0/1846826881 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c00abb0 con 0x7f79ac14d5d0 2023-10-25T00:24:19.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b3965640 1 -- 172.21.15.107:0/1846826881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79ac143e30 con 0x7f79ac14d5d0 2023-10-25T00:24:19.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b3965640 1 -- 172.21.15.107:0/1846826881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79ac0c24a0 con 0x7f79ac14d5d0 2023-10-25T00:24:19.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79a9ffb640 1 -- 172.21.15.107:0/1846826881 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f799c00ad10 con 0x7f79ac14d5d0 2023-10-25T00:24:19.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79a9ffb640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7998065e90 0x7f7998068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:19.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b1701640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7998065e90 0x7f7998068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:19.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79a9ffb640 1 -- 172.21.15.107:0/1846826881 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f799c084370 con 0x7f79ac14d5d0 2023-10-25T00:24:19.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.665+0000 7f79b1701640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7998065e90 0x7f7998068350 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f79940057b0 tx=0x7f79a00011a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:19.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.669+0000 7f79a9ffb640 1 -- 172.21.15.107:0/1846826881 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f799c050760 con 0x7f79ac14d5d0 2023-10-25T00:24:19.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.821+0000 7f79b3965640 1 -- 172.21.15.107:0/1846826881 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f79ac0c2070 con 0x7f7998065e90 2023-10-25T00:24:19.865 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.861+0000 7f79a9ffb640 1 -- 172.21.15.107:0/1846826881 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+555771 (secure 0 0 0) 0x7f79a00053b0 con 0x7f7998065e90 2023-10-25T00:24:19.868 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:19.875 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-25T00:24:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 -- 172.21.15.107:0/1846826881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7998065e90 msgr2=0x7f7998068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7998065e90 0x7f7998068350 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f79940057b0 tx=0x7f79a00011a0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 -- 172.21.15.107:0/1846826881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79ac14d5d0 msgr2=0x7f79ac155780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79ac14d5d0 0x7f79ac155780 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f799c00c9c0 tx=0x7f799c00ce80 comp rx=0 tx=0).stop 2023-10-25T00:24:19.876 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 -- 172.21.15.107:0/1846826881 shutdown_connections 2023-10-25T00:24:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79ac0bf7c0 0x7f79ac155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79ac14d5d0 0x7f79ac155780 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7998065e90 0x7f7998068350 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 --2- 172.21.15.107:0/1846826881 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7994003980 0x7f79ac0bf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:19.877 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 -- 172.21.15.107:0/1846826881 >> 172.21.15.107:0/1846826881 conn(0x7f79ac0c2950 msgr2=0x7f79ac0c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 -- 172.21.15.107:0/1846826881 shutdown_connections 2023-10-25T00:24:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:19.869+0000 7f79b3965640 1 -- 172.21.15.107:0/1846826881 wait complete. 2023-10-25T00:24:19.895 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1010,"stamp":"2023-10-25T00:24:18.058244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40142898,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3975,"num_read_kb":477474,"num_write":3153461,"num_write_kb":38880074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":584000,"ondisk_log_size":584000,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7810624,"kb_used_data":86208,"kb_used_omap":0,"kb_used_meta":7724352,"kb_avail":747164096,"statfs":{"total":773094113280,"available":765096034304,"internally_reserved":0,"allocated":88276992,"data_stored":84646180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7909736448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005585"},"pg_stats":[{"pgid":"6.1a","version":"72'3995","reported_seq":4074,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:24:16.051453+0000","last_change":"2023-10-25T00:24:16.049904+0000","last_active":"2023-10-25T00:24:16.051453+0000","last_peered":"2023-10-25T00:24:16.051453+0000","last_clean":"2023-10-25T00:24:16.051453+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-25T00:24:16.051453+0000","last_undegraded":"2023-10-25T00:24:16.051453+0000","last_fullsized":"2023-10-25T00:24:16.051453+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21480,"num_write_kb":282788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"72'3","reported_seq":84,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:29.637360+0000","last_change":"2023-10-25T00:23:29.637684+0000","last_active":"2023-10-25T00:23:29.637360+0000","last_peered":"2023-10-25T00:23:29.637360+0000","last_clean":"2023-10-25T00:23:29.637360+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-25T00:23:29.637360+0000","last_undegraded":"2023-10-25T00:23:29.637360+0000","last_fullsized":"2023-10-25T00:23:29.637360+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:29.637307+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:29.637307+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.637307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:55.311760+0000","scrub_duration":0.0045847350000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"73'867","reported_seq":1007,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.026995+0000","last_change":"2023-10-25T00:23:27.027247+0000","last_active":"2023-10-25T00:23:27.026995+0000","last_peered":"2023-10-25T00:23:27.026995+0000","last_clean":"2023-10-25T00:23:27.026995+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-25T00:23:27.026995+0000","last_undegraded":"2023-10-25T00:23:27.026995+0000","last_fullsized":"2023-10-25T00:23:27.026995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'867","last_scrub_stamp":"2023-10-25T00:23:27.026943+0000","last_deep_scrub":"73'867","last_deep_scrub_stamp":"2023-10-25T00:23:27.026943+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.026943+0000","objects_scrubbed":3,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:41.065198+0000","scrub_duration":0.082266929000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":22541,"num_write":841,"num_write_kb":4586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'5616","reported_seq":5701,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:46.131046+0000","last_change":"2023-10-25T00:23:46.131251+0000","last_active":"2023-10-25T00:23:46.131046+0000","last_peered":"2023-10-25T00:23:46.131046+0000","last_clean":"2023-10-25T00:23:46.131046+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-25T00:23:46.131046+0000","last_undegraded":"2023-10-25T00:23:46.131046+0000","last_fullsized":"2023-10-25T00:23:46.131046+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5616","last_scrub_stamp":"2023-10-25T00:23:46.131002+0000","last_deep_scrub":"68'5616","last_deep_scrub_stamp":"2023-10-25T00:23:46.131002+0000","last_clean_scrub_stamp":"2023-10-25T00:23:46.131002+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:29:47.372296+0000","scrub_duration":0.13911974099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29775,"num_write_kb":348818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"72'3735","reported_seq":3819,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:59.753433+0000","last_change":"2023-10-25T00:23:59.753562+0000","last_active":"2023-10-25T00:23:59.753433+0000","last_peered":"2023-10-25T00:23:59.753433+0000","last_clean":"2023-10-25T00:23:59.753433+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-25T00:23:59.753433+0000","last_undegraded":"2023-10-25T00:23:59.753433+0000","last_fullsized":"2023-10-25T00:23:59.753433+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3735","last_scrub_stamp":"2023-10-25T00:23:59.753413+0000","last_deep_scrub":"72'3735","last_deep_scrub_stamp":"2023-10-25T00:23:59.753413+0000","last_clean_scrub_stamp":"2023-10-25T00:23:59.753413+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:57:23.808102+0000","scrub_duration":0.20248877100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20090,"num_write_kb":264783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"72'2","reported_seq":83,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.880273+0000","last_change":"2023-10-25T00:23:02.880442+0000","last_active":"2023-10-25T00:23:02.880273+0000","last_peered":"2023-10-25T00:23:02.880273+0000","last_clean":"2023-10-25T00:23:02.880273+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-25T00:23:02.880273+0000","last_undegraded":"2023-10-25T00:23:02.880273+0000","last_fullsized":"2023-10-25T00:23:02.880273+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.880202+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:45.433003+0000","scrub_duration":0.0048172340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"73'28","reported_seq":116,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.443056+0000","last_change":"2023-10-25T00:23:05.443056+0000","last_active":"2023-10-25T00:23:05.443056+0000","last_peered":"2023-10-25T00:23:05.443056+0000","last_clean":"2023-10-25T00:23:05.443056+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-25T00:23:05.443056+0000","last_undegraded":"2023-10-25T00:23:05.443056+0000","last_fullsized":"2023-10-25T00:23:05.443056+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_deep_scrub":"73'28","last_deep_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.442750+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:29:04.499904+0000","scrub_duration":0.036465213000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'5447","reported_seq":5532,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:28.801261+0000","last_change":"2023-10-25T00:23:28.801391+0000","last_active":"2023-10-25T00:23:28.801261+0000","last_peered":"2023-10-25T00:23:28.801261+0000","last_clean":"2023-10-25T00:23:28.801261+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-25T00:23:28.801261+0000","last_undegraded":"2023-10-25T00:23:28.801261+0000","last_fullsized":"2023-10-25T00:23:28.801261+0000","mapping_epoch":21,"log_start":"61'100","ondisk_log_start":"61'100","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5447","last_scrub_stamp":"2023-10-25T00:23:28.801240+0000","last_deep_scrub":"68'5447","last_deep_scrub_stamp":"2023-10-25T00:23:28.801240+0000","last_clean_scrub_stamp":"2023-10-25T00:23:28.801240+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:44:02.442972+0000","scrub_duration":0.154979636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28063,"num_write_kb":350200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"72'4747","reported_seq":4831,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.173769+0000","last_change":"2023-10-25T00:23:40.173944+0000","last_active":"2023-10-25T00:23:40.173769+0000","last_peered":"2023-10-25T00:23:40.173769+0000","last_clean":"2023-10-25T00:23:40.173769+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-25T00:23:40.173769+0000","last_undegraded":"2023-10-25T00:23:40.173769+0000","last_fullsized":"2023-10-25T00:23:40.173769+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4747","last_scrub_stamp":"2023-10-25T00:23:40.173729+0000","last_deep_scrub":"72'4747","last_deep_scrub_stamp":"2023-10-25T00:23:40.173729+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.173729+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:39:14.413546+0000","scrub_duration":0.31868588399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":25766,"num_write_kb":328702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"72'754","reported_seq":862,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.214995+0000","last_change":"2023-10-25T00:22:37.433948+0000","last_active":"2023-10-25T00:22:49.214995+0000","last_peered":"2023-10-25T00:22:49.214995+0000","last_clean":"2023-10-25T00:22:49.214995+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-25T00:22:49.214995+0000","last_undegraded":"2023-10-25T00:22:49.214995+0000","last_fullsized":"2023-10-25T00:22:49.214995+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'754","last_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_deep_scrub":"72'754","last_deep_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.433382+0000","objects_scrubbed":1,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:43.601946+0000","scrub_duration":0.10905978199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":77282,"num_write":740,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"73'51","reported_seq":182,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.477614+0000","last_change":"2023-10-25T00:23:37.477885+0000","last_active":"2023-10-25T00:23:37.477614+0000","last_peered":"2023-10-25T00:23:37.477614+0000","last_clean":"2023-10-25T00:23:37.477614+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-25T00:23:37.477614+0000","last_undegraded":"2023-10-25T00:23:37.477614+0000","last_fullsized":"2023-10-25T00:23:37.477614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'51","last_scrub_stamp":"2023-10-25T00:23:37.477557+0000","last_deep_scrub":"73'51","last_deep_scrub_stamp":"2023-10-25T00:23:37.477557+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.477557+0000","objects_scrubbed":1,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:39.045780+0000","scrub_duration":0.008112401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'4792","reported_seq":4878,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234675+0000","last_change":"2023-10-25T00:22:44.881494+0000","last_active":"2023-10-25T00:22:49.234675+0000","last_peered":"2023-10-25T00:22:49.234675+0000","last_clean":"2023-10-25T00:22:49.234675+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-25T00:22:49.234675+0000","last_undegraded":"2023-10-25T00:22:49.234675+0000","last_fullsized":"2023-10-25T00:22:49.234675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4792","last_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_deep_scrub":"68'4792","last_deep_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.881344+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:23:57.562121+0000","scrub_duration":0.193670646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24997,"num_write_kb":313272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"72'4739","reported_seq":4824,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:49.802909+0000","last_change":"2023-10-25T00:23:49.803089+0000","last_active":"2023-10-25T00:23:49.802909+0000","last_peered":"2023-10-25T00:23:49.802909+0000","last_clean":"2023-10-25T00:23:49.802909+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-25T00:23:49.802909+0000","last_undegraded":"2023-10-25T00:23:49.802909+0000","last_fullsized":"2023-10-25T00:23:49.802909+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4739","last_scrub_stamp":"2023-10-25T00:23:49.802870+0000","last_deep_scrub":"72'4739","last_deep_scrub_stamp":"2023-10-25T00:23:49.802870+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.802870+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:56:59.101994+0000","scrub_duration":0.33146283399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":22,"num_write":25639,"num_write_kb":332364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"73'358","reported_seq":443,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:04.146397+0000","last_change":"2023-10-25T00:24:04.146618+0000","last_active":"2023-10-25T00:24:04.146397+0000","last_peered":"2023-10-25T00:24:04.146397+0000","last_clean":"2023-10-25T00:24:04.146397+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-25T00:24:04.146397+0000","last_undegraded":"2023-10-25T00:24:04.146397+0000","last_fullsized":"2023-10-25T00:24:04.146397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'358","last_scrub_stamp":"2023-10-25T00:24:04.146337+0000","last_deep_scrub":"73'358","last_deep_scrub_stamp":"2023-10-25T00:24:04.146337+0000","last_clean_scrub_stamp":"2023-10-25T00:24:04.146337+0000","objects_scrubbed":1,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:40:21.104237+0000","scrub_duration":0.154937403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2627766,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":227,"num_write":329,"num_write_kb":2759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"73'6","reported_seq":139,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:39.493024+0000","last_change":"2023-10-25T00:23:39.493290+0000","last_active":"2023-10-25T00:23:39.493024+0000","last_peered":"2023-10-25T00:23:39.493024+0000","last_clean":"2023-10-25T00:23:39.493024+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-25T00:23:39.493024+0000","last_undegraded":"2023-10-25T00:23:39.493024+0000","last_fullsized":"2023-10-25T00:23:39.493024+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2023-10-25T00:23:39.492964+0000","last_deep_scrub":"73'6","last_deep_scrub_stamp":"2023-10-25T00:23:39.492964+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.492964+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:34.589742+0000","scrub_duration":0.0071686809999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'3731","reported_seq":3815,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234039+0000","last_change":"2023-10-25T00:22:37.622543+0000","last_active":"2023-10-25T00:22:49.234039+0000","last_peered":"2023-10-25T00:22:49.234039+0000","last_clean":"2023-10-25T00:22:49.234039+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-25T00:22:49.234039+0000","last_undegraded":"2023-10-25T00:22:49.234039+0000","last_fullsized":"2023-10-25T00:22:49.234039+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3731","last_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_deep_scrub":"68'3731","last_deep_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.622304+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:51:52.171234+0000","scrub_duration":0.090187284000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19492,"num_write_kb":250091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"72'3799","reported_seq":3883,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:15.367310+0000","last_change":"2023-10-25T00:24:15.367485+0000","last_active":"2023-10-25T00:24:15.367310+0000","last_peered":"2023-10-25T00:24:15.367310+0000","last_clean":"2023-10-25T00:24:15.367310+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-25T00:24:15.367310+0000","last_undegraded":"2023-10-25T00:24:15.367310+0000","last_fullsized":"2023-10-25T00:24:15.367310+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3799","last_scrub_stamp":"2023-10-25T00:24:15.367271+0000","last_deep_scrub":"72'3799","last_deep_scrub_stamp":"2023-10-25T00:24:15.367271+0000","last_clean_scrub_stamp":"2023-10-25T00:24:15.367271+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:57:37.963942+0000","scrub_duration":0.31526191399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21581,"num_write_kb":262877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"73'46","reported_seq":171,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.664257+0000","last_change":"2023-10-25T00:22:52.664257+0000","last_active":"2023-10-25T00:22:52.664257+0000","last_peered":"2023-10-25T00:22:52.664257+0000","last_clean":"2023-10-25T00:22:52.664257+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-25T00:22:52.664257+0000","last_undegraded":"2023-10-25T00:22:52.664257+0000","last_fullsized":"2023-10-25T00:22:52.664257+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'46","last_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_deep_scrub":"73'46","last_deep_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.664064+0000","objects_scrubbed":1,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:44:09.108291+0000","scrub_duration":0.106079285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"73'715","reported_seq":829,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.967748+0000","last_change":"2023-10-25T00:23:21.967998+0000","last_active":"2023-10-25T00:23:21.967748+0000","last_peered":"2023-10-25T00:23:21.967748+0000","last_clean":"2023-10-25T00:23:21.967748+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-25T00:23:21.967748+0000","last_undegraded":"2023-10-25T00:23:21.967748+0000","last_fullsized":"2023-10-25T00:23:21.967748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'715","last_scrub_stamp":"2023-10-25T00:23:21.967694+0000","last_deep_scrub":"73'715","last_deep_scrub_stamp":"2023-10-25T00:23:21.967694+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.967694+0000","objects_scrubbed":1,"log_size":715,"log_dups_size":0,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:56:40.443702+0000","scrub_duration":0.065018907000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":90428,"num_write":703,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'4008","reported_seq":4096,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.488463+0000","last_change":"2023-10-25T00:23:05.488463+0000","last_active":"2023-10-25T00:23:05.488463+0000","last_peered":"2023-10-25T00:23:05.488463+0000","last_clean":"2023-10-25T00:23:05.488463+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-25T00:23:05.488463+0000","last_undegraded":"2023-10-25T00:23:05.488463+0000","last_fullsized":"2023-10-25T00:23:05.488463+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4008","last_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_deep_scrub":"68'4008","last_deep_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.488248+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:51:16.627187+0000","scrub_duration":0.15813021899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20791,"num_write_kb":267770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"72'5469","reported_seq":5564,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.571255+0000","last_change":"2023-10-25T00:22:57.571401+0000","last_active":"2023-10-25T00:22:57.571255+0000","last_peered":"2023-10-25T00:22:57.571255+0000","last_clean":"2023-10-25T00:22:57.571255+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-25T00:22:57.571255+0000","last_undegraded":"2023-10-25T00:22:57.571255+0000","last_fullsized":"2023-10-25T00:22:57.571255+0000","mapping_epoch":25,"log_start":"64'200","ondisk_log_start":"64'200","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5469","last_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_deep_scrub":"72'5469","last_deep_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.571234+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:24:13.836466+0000","scrub_duration":0.19144520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":40,"num_write":30646,"num_write_kb":401481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"73'6","reported_seq":131,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.991436+0000","last_change":"2023-10-25T00:23:14.991436+0000","last_active":"2023-10-25T00:23:14.991436+0000","last_peered":"2023-10-25T00:23:14.991436+0000","last_clean":"2023-10-25T00:23:14.991436+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-25T00:23:14.991436+0000","last_undegraded":"2023-10-25T00:23:14.991436+0000","last_fullsized":"2023-10-25T00:23:14.991436+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_deep_scrub":"73'6","last_deep_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.991238+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:25:08.545014+0000","scrub_duration":0.0072276010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"73'546","reported_seq":631,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:23:06.302701+0000","last_change":"2023-10-25T00:23:36.987182+0000","last_active":"2023-10-25T00:23:06.302701+0000","last_peered":"2023-10-25T00:23:06.302701+0000","last_clean":"2023-10-25T00:23:06.302701+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-25T00:23:06.302701+0000","last_undegraded":"2023-10-25T00:23:06.302701+0000","last_fullsized":"2023-10-25T00:23:06.302701+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3198201,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10628,"num_write":517,"num_write_kb":3434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'6069","reported_seq":6156,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.235476+0000","last_change":"2023-10-25T00:22:46.608504+0000","last_active":"2023-10-25T00:22:49.235476+0000","last_peered":"2023-10-25T00:22:49.235476+0000","last_clean":"2023-10-25T00:22:49.235476+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-25T00:22:49.235476+0000","last_undegraded":"2023-10-25T00:22:49.235476+0000","last_fullsized":"2023-10-25T00:22:49.235476+0000","mapping_epoch":21,"log_start":"62'800","ondisk_log_start":"62'800","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6069","last_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_deep_scrub":"68'6069","last_deep_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.608198+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:37.101860+0000","scrub_duration":0.19917779799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":33187,"num_write_kb":387130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'3606","reported_seq":3694,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:38.186285+0000","last_change":"2023-10-25T00:23:38.186420+0000","last_active":"2023-10-25T00:23:38.186285+0000","last_peered":"2023-10-25T00:23:38.186285+0000","last_clean":"2023-10-25T00:23:38.186285+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-25T00:23:38.186285+0000","last_undegraded":"2023-10-25T00:23:38.186285+0000","last_fullsized":"2023-10-25T00:23:38.186285+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3606","last_scrub_stamp":"2023-10-25T00:23:38.186264+0000","last_deep_scrub":"72'3606","last_deep_scrub_stamp":"2023-10-25T00:23:38.186264+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.186264+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:15.619690+0000","scrub_duration":0.23143245800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19493,"num_write_kb":254342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'1064","reported_seq":1198,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:03.701358+0000","last_change":"2023-10-25T00:23:03.701535+0000","last_active":"2023-10-25T00:23:03.701358+0000","last_peered":"2023-10-25T00:23:03.701358+0000","last_clean":"2023-10-25T00:23:03.701358+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-25T00:23:03.701358+0000","last_undegraded":"2023-10-25T00:23:03.701358+0000","last_fullsized":"2023-10-25T00:23:03.701358+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1064","last_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_deep_scrub":"73'1064","last_deep_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_clean_scrub_stamp":"2023-10-25T00:23:03.701304+0000","objects_scrubbed":3,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:08:18.178955+0000","scrub_duration":0.28236572100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":9402,"num_write":1035,"num_write_kb":4717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'3","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.838808+0000","last_change":"2023-10-25T00:23:12.838945+0000","last_active":"2023-10-25T00:23:12.838808+0000","last_peered":"2023-10-25T00:23:12.838808+0000","last_clean":"2023-10-25T00:23:12.838808+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-25T00:23:12.838808+0000","last_undegraded":"2023-10-25T00:23:12.838808+0000","last_fullsized":"2023-10-25T00:23:12.838808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.838787+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:33:35.778931+0000","scrub_duration":0.133446072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'5139","reported_seq":5226,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:54.030976+0000","last_change":"2023-10-25T00:22:54.030976+0000","last_active":"2023-10-25T00:22:54.030976+0000","last_peered":"2023-10-25T00:22:54.030976+0000","last_clean":"2023-10-25T00:22:54.030976+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-25T00:22:54.030976+0000","last_undegraded":"2023-10-25T00:22:54.030976+0000","last_fullsized":"2023-10-25T00:22:54.030976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5139","last_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_deep_scrub":"68'5139","last_deep_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.030782+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:30.182861+0000","scrub_duration":0.103857617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27772,"num_write_kb":328469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"72'4843","reported_seq":4927,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:39.041680+0000","last_change":"2023-10-25T00:23:39.041980+0000","last_active":"2023-10-25T00:23:39.041680+0000","last_peered":"2023-10-25T00:23:39.041680+0000","last_clean":"2023-10-25T00:23:39.041680+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-25T00:23:39.041680+0000","last_undegraded":"2023-10-25T00:23:39.041680+0000","last_fullsized":"2023-10-25T00:23:39.041680+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4843","last_scrub_stamp":"2023-10-25T00:23:39.041623+0000","last_deep_scrub":"72'4843","last_deep_scrub_stamp":"2023-10-25T00:23:39.041623+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.041623+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:34:30.360915+0000","scrub_duration":0.33452469800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":27290,"num_write_kb":330906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"72'27","reported_seq":111,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.987117+0000","last_change":"2023-10-25T00:22:55.987117+0000","last_active":"2023-10-25T00:22:55.987117+0000","last_peered":"2023-10-25T00:22:55.987117+0000","last_clean":"2023-10-25T00:22:55.987117+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-25T00:22:55.987117+0000","last_undegraded":"2023-10-25T00:22:55.987117+0000","last_fullsized":"2023-10-25T00:22:55.987117+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'27","last_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_deep_scrub":"72'27","last_deep_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.986674+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:33.089164+0000","scrub_duration":0.100730242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'2","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.503834+0000","last_change":"2023-10-25T00:23:21.504086+0000","last_active":"2023-10-25T00:23:21.503834+0000","last_peered":"2023-10-25T00:23:21.503834+0000","last_clean":"2023-10-25T00:23:21.503834+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-25T00:23:21.503834+0000","last_undegraded":"2023-10-25T00:23:21.503834+0000","last_fullsized":"2023-10-25T00:23:21.503834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2023-10-25T00:23:21.503783+0000","last_deep_scrub":"67'2","last_deep_scrub_stamp":"2023-10-25T00:23:21.503783+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.503783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:13.653289+0000","scrub_duration":0.0039280959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'5814","reported_seq":5901,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.320438+0000","last_change":"2023-10-25T00:22:52.320696+0000","last_active":"2023-10-25T00:22:52.320438+0000","last_peered":"2023-10-25T00:22:52.320438+0000","last_clean":"2023-10-25T00:22:52.320438+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-25T00:22:52.320438+0000","last_undegraded":"2023-10-25T00:22:52.320438+0000","last_fullsized":"2023-10-25T00:22:52.320438+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5814","last_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_deep_scrub":"68'5814","last_deep_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.320378+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:26.511615+0000","scrub_duration":0.033495337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31996,"num_write_kb":382397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"72'3827","reported_seq":3912,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.436458+0000","last_change":"2023-10-25T00:23:14.436458+0000","last_active":"2023-10-25T00:23:14.436458+0000","last_peered":"2023-10-25T00:23:14.436458+0000","last_clean":"2023-10-25T00:23:14.436458+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-25T00:23:14.436458+0000","last_undegraded":"2023-10-25T00:23:14.436458+0000","last_fullsized":"2023-10-25T00:23:14.436458+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3827","last_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_deep_scrub":"72'3827","last_deep_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.436150+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:27:32.423180+0000","scrub_duration":0.30547422699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":21199,"num_write_kb":277730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"68'4820","reported_seq":4907,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232177+0000","last_change":"2023-10-25T00:22:36.429763+0000","last_active":"2023-10-25T00:22:49.232177+0000","last_peered":"2023-10-25T00:22:49.232177+0000","last_clean":"2023-10-25T00:22:49.232177+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-25T00:22:49.232177+0000","last_undegraded":"2023-10-25T00:22:49.232177+0000","last_fullsized":"2023-10-25T00:22:49.232177+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4820","last_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_deep_scrub":"68'4820","last_deep_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_clean_scrub_stamp":"2023-10-25T00:22:36.429607+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:35.883183+0000","scrub_duration":0.10613655600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25163,"num_write_kb":285293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"72'5","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.946770+0000","last_change":"2023-10-25T00:23:20.947028+0000","last_active":"2023-10-25T00:23:20.946770+0000","last_peered":"2023-10-25T00:23:20.946770+0000","last_clean":"2023-10-25T00:23:20.946770+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-25T00:23:20.946770+0000","last_undegraded":"2023-10-25T00:23:20.946770+0000","last_fullsized":"2023-10-25T00:23:20.946770+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:23:20.946713+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:23:20.946713+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.946713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:12:40.769613+0000","scrub_duration":0.0045247550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"72'5","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.215935+0000","last_change":"2023-10-25T00:22:40.197550+0000","last_active":"2023-10-25T00:22:49.215935+0000","last_peered":"2023-10-25T00:22:49.215935+0000","last_clean":"2023-10-25T00:22:49.215935+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-25T00:22:49.215935+0000","last_undegraded":"2023-10-25T00:22:49.215935+0000","last_fullsized":"2023-10-25T00:22:49.215935+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.197213+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:07:28.509914+0000","scrub_duration":0.0047704979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"72'4957","reported_seq":5045,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:53.667906+0000","last_change":"2023-10-25T00:22:53.667906+0000","last_active":"2023-10-25T00:22:53.667906+0000","last_peered":"2023-10-25T00:22:53.667906+0000","last_clean":"2023-10-25T00:22:53.667906+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-25T00:22:53.667906+0000","last_undegraded":"2023-10-25T00:22:53.667906+0000","last_fullsized":"2023-10-25T00:22:53.667906+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4957","last_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_deep_scrub":"72'4957","last_deep_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.667748+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:58.307897+0000","scrub_duration":0.30812842099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28093,"num_write_kb":358545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'4782","reported_seq":4874,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234575+0000","last_change":"2023-10-25T00:22:42.562198+0000","last_active":"2023-10-25T00:22:49.234575+0000","last_peered":"2023-10-25T00:22:49.234575+0000","last_clean":"2023-10-25T00:22:49.234575+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-25T00:22:49.234575+0000","last_undegraded":"2023-10-25T00:22:49.234575+0000","last_fullsized":"2023-10-25T00:22:49.234575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4782","last_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_deep_scrub":"68'4782","last_deep_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.562057+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:39.415960+0000","scrub_duration":0.116720554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26432,"num_write_kb":298639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"68'3","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.433942+0000","last_change":"2023-10-25T00:23:36.434199+0000","last_active":"2023-10-25T00:23:36.433942+0000","last_peered":"2023-10-25T00:23:36.433942+0000","last_clean":"2023-10-25T00:23:36.433942+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-25T00:23:36.433942+0000","last_undegraded":"2023-10-25T00:23:36.433942+0000","last_fullsized":"2023-10-25T00:23:36.433942+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:36.433882+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:36.433882+0000","last_clean_scrub_stamp":"2023-10-25T00:23:36.433882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:00:44.210403+0000","scrub_duration":0.0040752599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"72'18","reported_seq":215,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.214421+0000","last_change":"2023-10-25T00:22:46.074083+0000","last_active":"2023-10-25T00:22:49.214421+0000","last_peered":"2023-10-25T00:22:49.214421+0000","last_clean":"2023-10-25T00:22:49.214421+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-25T00:22:49.214421+0000","last_undegraded":"2023-10-25T00:22:49.214421+0000","last_fullsized":"2023-10-25T00:22:49.214421+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_deep_scrub":"72'18","last_deep_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.073758+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:08:25.787090+0000","scrub_duration":0.012558368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":345,"num_read_kb":345,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"72'2954","reported_seq":3037,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.099045+0000","last_change":"2023-10-25T00:23:36.099045+0000","last_active":"2023-10-25T00:23:36.099045+0000","last_peered":"2023-10-25T00:23:36.099045+0000","last_clean":"2023-10-25T00:23:36.099045+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-25T00:23:36.099045+0000","last_undegraded":"2023-10-25T00:23:36.099045+0000","last_fullsized":"2023-10-25T00:23:36.099045+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2954","last_scrub_stamp":"2023-10-25T00:23:36.098826+0000","last_deep_scrub":"72'2954","last_deep_scrub_stamp":"2023-10-25T00:23:36.098826+0000","last_clean_scrub_stamp":"2023-10-25T00:23:36.098826+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:16.696480+0000","scrub_duration":0.32992982700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15961,"num_write_kb":219669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"73'38","reported_seq":169,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.052563+0000","last_change":"2023-10-25T00:23:35.052803+0000","last_active":"2023-10-25T00:23:35.052563+0000","last_peered":"2023-10-25T00:23:35.052563+0000","last_clean":"2023-10-25T00:23:35.052563+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-25T00:23:35.052563+0000","last_undegraded":"2023-10-25T00:23:35.052563+0000","last_fullsized":"2023-10-25T00:23:35.052563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'38","last_scrub_stamp":"2023-10-25T00:23:35.052506+0000","last_deep_scrub":"73'38","last_deep_scrub_stamp":"2023-10-25T00:23:35.052506+0000","last_clean_scrub_stamp":"2023-10-25T00:23:35.052506+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:23:31.370767+0000","scrub_duration":0.0095825770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'4690","reported_seq":4778,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.792729+0000","last_change":"2023-10-25T00:23:05.792729+0000","last_active":"2023-10-25T00:23:05.792729+0000","last_peered":"2023-10-25T00:23:05.792729+0000","last_clean":"2023-10-25T00:23:05.792729+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-25T00:23:05.792729+0000","last_undegraded":"2023-10-25T00:23:05.792729+0000","last_fullsized":"2023-10-25T00:23:05.792729+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4690","last_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_deep_scrub":"68'4690","last_deep_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.792588+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:24:53.476824+0000","scrub_duration":0.15089322399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25172,"num_write_kb":305786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"72'10","reported_seq":136,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234044+0000","last_change":"2023-10-25T00:22:40.512549+0000","last_active":"2023-10-25T00:22:49.234044+0000","last_peered":"2023-10-25T00:22:49.234044+0000","last_clean":"2023-10-25T00:22:49.234044+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-25T00:22:49.234044+0000","last_undegraded":"2023-10-25T00:22:49.234044+0000","last_fullsized":"2023-10-25T00:22:49.234044+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.512409+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:07:23.705506+0000","scrub_duration":0.091817919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"72'4200","reported_seq":4291,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.809510+0000","last_change":"2023-10-25T00:23:34.809510+0000","last_active":"2023-10-25T00:23:34.809510+0000","last_peered":"2023-10-25T00:23:34.809510+0000","last_clean":"2023-10-25T00:23:34.809510+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-25T00:23:34.809510+0000","last_undegraded":"2023-10-25T00:23:34.809510+0000","last_fullsized":"2023-10-25T00:23:34.809510+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4200","last_scrub_stamp":"2023-10-25T00:23:34.809362+0000","last_deep_scrub":"72'4200","last_deep_scrub_stamp":"2023-10-25T00:23:34.809362+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.809362+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:04:30.830862+0000","scrub_duration":0.23894108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":24169,"num_write_kb":293796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"73'394","reported_seq":596,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:31.115748+0000","last_change":"2023-10-25T00:23:31.116025+0000","last_active":"2023-10-25T00:23:31.115748+0000","last_peered":"2023-10-25T00:23:31.115748+0000","last_clean":"2023-10-25T00:23:31.115748+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-25T00:23:31.115748+0000","last_undegraded":"2023-10-25T00:23:31.115748+0000","last_fullsized":"2023-10-25T00:23:31.115748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'394","last_scrub_stamp":"2023-10-25T00:23:31.115692+0000","last_deep_scrub":"73'394","last_deep_scrub_stamp":"2023-10-25T00:23:31.115692+0000","last_clean_scrub_stamp":"2023-10-25T00:23:31.115692+0000","objects_scrubbed":4,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:38.234847+0000","scrub_duration":0.026138159000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":210,"num_read_kb":94395,"num_write":387,"num_write_kb":4304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'5730","reported_seq":5817,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.204004+0000","last_change":"2023-10-25T00:23:30.204139+0000","last_active":"2023-10-25T00:23:30.204004+0000","last_peered":"2023-10-25T00:23:30.204004+0000","last_clean":"2023-10-25T00:23:30.204004+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-25T00:23:30.204004+0000","last_undegraded":"2023-10-25T00:23:30.204004+0000","last_fullsized":"2023-10-25T00:23:30.204004+0000","mapping_epoch":21,"log_start":"61'500","ondisk_log_start":"61'500","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5730","last_scrub_stamp":"2023-10-25T00:23:30.203981+0000","last_deep_scrub":"68'5730","last_deep_scrub_stamp":"2023-10-25T00:23:30.203981+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.203981+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:38:35.578976+0000","scrub_duration":0.10293530200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":31652,"num_write_kb":356481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"72'3","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.639911+0000","last_change":"2023-10-25T00:23:27.639911+0000","last_active":"2023-10-25T00:23:27.639911+0000","last_peered":"2023-10-25T00:23:27.639911+0000","last_clean":"2023-10-25T00:23:27.639911+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-25T00:23:27.639911+0000","last_undegraded":"2023-10-25T00:23:27.639911+0000","last_fullsized":"2023-10-25T00:23:27.639911+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:27.639604+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:27.639604+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.639604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:09:33.666294+0000","scrub_duration":0.0047758339999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"72'4168","reported_seq":4254,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:59.283911+0000","last_change":"2023-10-25T00:23:59.284060+0000","last_active":"2023-10-25T00:23:59.283911+0000","last_peered":"2023-10-25T00:23:59.283911+0000","last_clean":"2023-10-25T00:23:59.283911+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-25T00:23:59.283911+0000","last_undegraded":"2023-10-25T00:23:59.283911+0000","last_fullsized":"2023-10-25T00:23:59.283911+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4168","last_scrub_stamp":"2023-10-25T00:23:59.283890+0000","last_deep_scrub":"72'4168","last_deep_scrub_stamp":"2023-10-25T00:23:59.283890+0000","last_clean_scrub_stamp":"2023-10-25T00:23:59.283890+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:19:04.711112+0000","scrub_duration":0.25867606399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23458,"num_write_kb":294850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"68'4339","reported_seq":4426,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:47.099364+0000","last_change":"2023-10-25T00:23:47.099492+0000","last_active":"2023-10-25T00:23:47.099364+0000","last_peered":"2023-10-25T00:23:47.099364+0000","last_clean":"2023-10-25T00:23:47.099364+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-25T00:23:47.099364+0000","last_undegraded":"2023-10-25T00:23:47.099364+0000","last_fullsized":"2023-10-25T00:23:47.099364+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4339","last_scrub_stamp":"2023-10-25T00:23:47.099343+0000","last_deep_scrub":"68'4339","last_deep_scrub_stamp":"2023-10-25T00:23:47.099343+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.099343+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:27.913875+0000","scrub_duration":0.12538819900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23267,"num_write_kb":265268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.223439+0000","last_change":"2023-10-25T00:22:42.199286+0000","last_active":"2023-10-25T00:22:49.223439+0000","last_peered":"2023-10-25T00:22:49.223439+0000","last_clean":"2023-10-25T00:22:49.223439+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-25T00:22:49.223439+0000","last_undegraded":"2023-10-25T00:22:49.223439+0000","last_fullsized":"2023-10-25T00:22:49.223439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.198947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:11:17.807377+0000","scrub_duration":0.0041955760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"73'231","reported_seq":342,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.010034+0000","last_change":"2023-10-25T00:23:16.010034+0000","last_active":"2023-10-25T00:23:16.010034+0000","last_peered":"2023-10-25T00:23:16.010034+0000","last_clean":"2023-10-25T00:23:16.010034+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-25T00:23:16.010034+0000","last_undegraded":"2023-10-25T00:23:16.010034+0000","last_fullsized":"2023-10-25T00:23:16.010034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'231","last_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_deep_scrub":"73'231","last_deep_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.009842+0000","objects_scrubbed":1,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:27:21.881029+0000","scrub_duration":0.076880452000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":63,"num_write":590,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"72'11","reported_seq":95,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.231986+0000","last_change":"2023-10-25T00:22:38.409520+0000","last_active":"2023-10-25T00:22:49.231986+0000","last_peered":"2023-10-25T00:22:49.231986+0000","last_clean":"2023-10-25T00:22:49.231986+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-25T00:22:49.231986+0000","last_undegraded":"2023-10-25T00:22:49.231986+0000","last_fullsized":"2023-10-25T00:22:49.231986+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_deep_scrub":"72'11","last_deep_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.409235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:21:24.708709+0000","scrub_duration":0.003592329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"72'3993","reported_seq":4080,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:58.292731+0000","last_change":"2023-10-25T00:23:58.292855+0000","last_active":"2023-10-25T00:23:58.292731+0000","last_peered":"2023-10-25T00:23:58.292731+0000","last_clean":"2023-10-25T00:23:58.292731+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-25T00:23:58.292731+0000","last_undegraded":"2023-10-25T00:23:58.292731+0000","last_fullsized":"2023-10-25T00:23:58.292731+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3993","last_scrub_stamp":"2023-10-25T00:23:58.292710+0000","last_deep_scrub":"72'3993","last_deep_scrub_stamp":"2023-10-25T00:23:58.292710+0000","last_clean_scrub_stamp":"2023-10-25T00:23:58.292710+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:23:08.798165+0000","scrub_duration":0.310815756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21723,"num_write_kb":283850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'4270","reported_seq":4358,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:59.570791+0000","last_change":"2023-10-25T00:22:59.570791+0000","last_active":"2023-10-25T00:22:59.570791+0000","last_peered":"2023-10-25T00:22:59.570791+0000","last_clean":"2023-10-25T00:22:59.570791+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-25T00:22:59.570791+0000","last_undegraded":"2023-10-25T00:22:59.570791+0000","last_fullsized":"2023-10-25T00:22:59.570791+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4270","last_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_deep_scrub":"68'4270","last_deep_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.570628+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:36:56.294621+0000","scrub_duration":0.14092118100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23049,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.643510+0000","last_change":"2023-10-25T00:23:06.643510+0000","last_active":"2023-10-25T00:23:06.643510+0000","last_peered":"2023-10-25T00:23:06.643510+0000","last_clean":"2023-10-25T00:23:06.643510+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-25T00:23:06.643510+0000","last_undegraded":"2023-10-25T00:23:06.643510+0000","last_fullsized":"2023-10-25T00:23:06.643510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.643156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:34:24.773001+0000","scrub_duration":0.0045495850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"73'10","reported_seq":145,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.618489+0000","last_change":"2023-10-25T00:23:20.618489+0000","last_active":"2023-10-25T00:23:20.618489+0000","last_peered":"2023-10-25T00:23:20.618489+0000","last_clean":"2023-10-25T00:23:20.618489+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-25T00:23:20.618489+0000","last_undegraded":"2023-10-25T00:23:20.618489+0000","last_fullsized":"2023-10-25T00:23:20.618489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2023-10-25T00:23:20.618371+0000","last_deep_scrub":"73'10","last_deep_scrub_stamp":"2023-10-25T00:23:20.618371+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.618371+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:30:10.455022+0000","scrub_duration":0.085107656000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":8,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"73'65","reported_seq":228,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.930205+0000","last_change":"2023-10-25T00:23:07.930465+0000","last_active":"2023-10-25T00:23:07.930205+0000","last_peered":"2023-10-25T00:23:07.930205+0000","last_clean":"2023-10-25T00:23:07.930205+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-25T00:23:07.930205+0000","last_undegraded":"2023-10-25T00:23:07.930205+0000","last_fullsized":"2023-10-25T00:23:07.930205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'65","last_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_deep_scrub":"73'65","last_deep_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.930143+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:41:49.528670+0000","scrub_duration":0.020886851000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":123,"num_write":59,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"72'3769","reported_seq":3856,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:38.049931+0000","last_change":"2023-10-25T00:23:38.050117+0000","last_active":"2023-10-25T00:23:38.049931+0000","last_peered":"2023-10-25T00:23:38.049931+0000","last_clean":"2023-10-25T00:23:38.049931+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-25T00:23:38.049931+0000","last_undegraded":"2023-10-25T00:23:38.049931+0000","last_fullsized":"2023-10-25T00:23:38.049931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3769","last_scrub_stamp":"2023-10-25T00:23:38.049889+0000","last_deep_scrub":"72'3769","last_deep_scrub_stamp":"2023-10-25T00:23:38.049889+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.049889+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:23:14.574715+0000","scrub_duration":0.37571605800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20807,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'5752","reported_seq":5839,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:33.937946+0000","last_change":"2023-10-25T00:22:33.937946+0000","last_active":"2023-10-25T00:22:33.937946+0000","last_peered":"2023-10-25T00:22:33.937946+0000","last_clean":"2023-10-25T00:22:33.937946+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-25T00:22:33.937946+0000","last_undegraded":"2023-10-25T00:22:33.937946+0000","last_fullsized":"2023-10-25T00:22:33.937946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5752","last_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_deep_scrub":"68'5752","last_deep_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_clean_scrub_stamp":"2023-10-25T00:22:33.937564+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:43:34.779084+0000","scrub_duration":0.080678784000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31231,"num_write_kb":354718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:24.047030+0000","last_change":"2023-10-25T00:23:24.047030+0000","last_active":"2023-10-25T00:23:24.047030+0000","last_peered":"2023-10-25T00:23:24.047030+0000","last_clean":"2023-10-25T00:23:24.047030+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-25T00:23:24.047030+0000","last_undegraded":"2023-10-25T00:23:24.047030+0000","last_fullsized":"2023-10-25T00:23:24.047030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:24.046716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:24.046716+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.046716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:53:42.506468+0000","scrub_duration":0.0043444529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'2","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:18.683846+0000","last_change":"2023-10-25T00:23:18.684046+0000","last_active":"2023-10-25T00:23:18.683846+0000","last_peered":"2023-10-25T00:23:18.683846+0000","last_clean":"2023-10-25T00:23:18.683846+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-25T00:23:18.683846+0000","last_undegraded":"2023-10-25T00:23:18.683846+0000","last_fullsized":"2023-10-25T00:23:18.683846+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2023-10-25T00:23:18.683799+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2023-10-25T00:23:18.683799+0000","last_clean_scrub_stamp":"2023-10-25T00:23:18.683799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:26:44.716117+0000","scrub_duration":0.013956036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"72'4","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.496631+0000","last_change":"2023-10-25T00:23:06.496904+0000","last_active":"2023-10-25T00:23:06.496631+0000","last_peered":"2023-10-25T00:23:06.496631+0000","last_clean":"2023-10-25T00:23:06.496631+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-25T00:23:06.496631+0000","last_undegraded":"2023-10-25T00:23:06.496631+0000","last_fullsized":"2023-10-25T00:23:06.496631+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_deep_scrub":"72'4","last_deep_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.496575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:32:18.665160+0000","scrub_duration":0.0046312189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"72'3939","reported_seq":4022,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.160279+0000","last_change":"2023-10-25T00:23:02.160279+0000","last_active":"2023-10-25T00:23:02.160279+0000","last_peered":"2023-10-25T00:23:02.160279+0000","last_clean":"2023-10-25T00:23:02.160279+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-25T00:23:02.160279+0000","last_undegraded":"2023-10-25T00:23:02.160279+0000","last_fullsized":"2023-10-25T00:23:02.160279+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3939","last_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_deep_scrub":"72'3939","last_deep_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.160086+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:37:29.878702+0000","scrub_duration":0.252775894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22393,"num_write_kb":262004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'4305","reported_seq":4392,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.858696+0000","last_change":"2023-10-25T00:23:26.858862+0000","last_active":"2023-10-25T00:23:26.858696+0000","last_peered":"2023-10-25T00:23:26.858696+0000","last_clean":"2023-10-25T00:23:26.858696+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-25T00:23:26.858696+0000","last_undegraded":"2023-10-25T00:23:26.858696+0000","last_fullsized":"2023-10-25T00:23:26.858696+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4305","last_scrub_stamp":"2023-10-25T00:23:26.858657+0000","last_deep_scrub":"68'4305","last_deep_scrub_stamp":"2023-10-25T00:23:26.858657+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.858657+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:10:03.352691+0000","scrub_duration":0.18050687100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22696,"num_write_kb":287574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:11.717353+0000","last_change":"2023-10-25T00:23:11.717353+0000","last_active":"2023-10-25T00:23:11.717353+0000","last_peered":"2023-10-25T00:23:11.717353+0000","last_clean":"2023-10-25T00:23:11.717353+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-25T00:23:11.717353+0000","last_undegraded":"2023-10-25T00:23:11.717353+0000","last_fullsized":"2023-10-25T00:23:11.717353+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_clean_scrub_stamp":"2023-10-25T00:23:11.717027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:17:42.982457+0000","scrub_duration":0.0039434270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.354835+0000","last_change":"2023-10-25T00:23:30.355099+0000","last_active":"2023-10-25T00:23:30.354835+0000","last_peered":"2023-10-25T00:23:30.354835+0000","last_clean":"2023-10-25T00:23:30.354835+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-25T00:23:30.354835+0000","last_undegraded":"2023-10-25T00:23:30.354835+0000","last_fullsized":"2023-10-25T00:23:30.354835+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2023-10-25T00:23:30.354780+0000","last_deep_scrub":"67'2","last_deep_scrub_stamp":"2023-10-25T00:23:30.354780+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.354780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:07:40.743916+0000","scrub_duration":0.0044225610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"72'3","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.952645+0000","last_change":"2023-10-25T00:22:52.952897+0000","last_active":"2023-10-25T00:22:52.952645+0000","last_peered":"2023-10-25T00:22:52.952645+0000","last_clean":"2023-10-25T00:22:52.952645+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-25T00:22:52.952645+0000","last_undegraded":"2023-10-25T00:22:52.952645+0000","last_fullsized":"2023-10-25T00:22:52.952645+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.952595+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:14:20.890322+0000","scrub_duration":0.0044650949999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"72'5003","reported_seq":5092,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:44.774558+0000","last_change":"2023-10-25T00:23:44.774832+0000","last_active":"2023-10-25T00:23:44.774558+0000","last_peered":"2023-10-25T00:23:44.774558+0000","last_clean":"2023-10-25T00:23:44.774558+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-25T00:23:44.774558+0000","last_undegraded":"2023-10-25T00:23:44.774558+0000","last_fullsized":"2023-10-25T00:23:44.774558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5003","last_scrub_stamp":"2023-10-25T00:23:44.774508+0000","last_deep_scrub":"72'5003","last_deep_scrub_stamp":"2023-10-25T00:23:44.774508+0000","last_clean_scrub_stamp":"2023-10-25T00:23:44.774508+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:52.739672+0000","scrub_duration":0.33716653400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":13,"num_write":27120,"num_write_kb":350355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'5635","reported_seq":5721,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:44.547139+0000","last_change":"2023-10-25T00:22:44.547139+0000","last_active":"2023-10-25T00:22:44.547139+0000","last_peered":"2023-10-25T00:22:44.547139+0000","last_clean":"2023-10-25T00:22:44.547139+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-25T00:22:44.547139+0000","last_undegraded":"2023-10-25T00:22:44.547139+0000","last_fullsized":"2023-10-25T00:22:44.547139+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5635","last_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_deep_scrub":"68'5635","last_deep_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.546984+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:51:54.474684+0000","scrub_duration":0.15865177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30556,"num_write_kb":356140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"68'3","reported_seq":80,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.235441+0000","last_change":"2023-10-25T00:23:35.979143+0000","last_active":"2023-10-25T00:22:49.235441+0000","last_peered":"2023-10-25T00:22:49.235441+0000","last_clean":"2023-10-25T00:22:49.235441+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-25T00:22:49.235441+0000","last_undegraded":"2023-10-25T00:22:49.235441+0000","last_fullsized":"2023-10-25T00:22:49.235441+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:01.277969+0000","last_clean_scrub_stamp":"2023-10-24T23:53:01.277969+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:29.305381+0000","last_change":"2023-10-25T00:23:29.305381+0000","last_active":"2023-10-25T00:23:29.305381+0000","last_peered":"2023-10-25T00:23:29.305381+0000","last_clean":"2023-10-25T00:23:29.305381+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-25T00:23:29.305381+0000","last_undegraded":"2023-10-25T00:23:29.305381+0000","last_fullsized":"2023-10-25T00:23:29.305381+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:29.305022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:29.305022+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.305022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:52:12.477955+0000","scrub_duration":0.0044947839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"72'2","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:58.925373+0000","last_change":"2023-10-25T00:22:58.925685+0000","last_active":"2023-10-25T00:22:58.925373+0000","last_peered":"2023-10-25T00:22:58.925373+0000","last_clean":"2023-10-25T00:22:58.925373+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-25T00:22:58.925373+0000","last_undegraded":"2023-10-25T00:22:58.925373+0000","last_fullsized":"2023-10-25T00:22:58.925373+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.925318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:31:45.562398+0000","scrub_duration":0.0043101720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"72'3753","reported_seq":3839,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:44.352105+0000","last_change":"2023-10-25T00:23:44.352225+0000","last_active":"2023-10-25T00:23:44.352105+0000","last_peered":"2023-10-25T00:23:44.352105+0000","last_clean":"2023-10-25T00:23:44.352105+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-25T00:23:44.352105+0000","last_undegraded":"2023-10-25T00:23:44.352105+0000","last_fullsized":"2023-10-25T00:23:44.352105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3753","last_scrub_stamp":"2023-10-25T00:23:44.352083+0000","last_deep_scrub":"72'3753","last_deep_scrub_stamp":"2023-10-25T00:23:44.352083+0000","last_clean_scrub_stamp":"2023-10-25T00:23:44.352083+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:13:26.541612+0000","scrub_duration":0.328578174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":21045,"num_write_kb":276986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'4101","reported_seq":4187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:25.811837+0000","last_change":"2023-10-25T00:23:25.811837+0000","last_active":"2023-10-25T00:23:25.811837+0000","last_peered":"2023-10-25T00:23:25.811837+0000","last_clean":"2023-10-25T00:23:25.811837+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-25T00:23:25.811837+0000","last_undegraded":"2023-10-25T00:23:25.811837+0000","last_fullsized":"2023-10-25T00:23:25.811837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:23:25.811682+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:23:25.811682+0000","last_clean_scrub_stamp":"2023-10-25T00:23:25.811682+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:39:30.752063+0000","scrub_duration":0.14318265299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22383,"num_write_kb":270023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"73'13","reported_seq":101,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:17.651156+0000","last_change":"2023-10-25T00:23:17.651450+0000","last_active":"2023-10-25T00:23:17.651156+0000","last_peered":"2023-10-25T00:23:17.651156+0000","last_clean":"2023-10-25T00:23:17.651156+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-25T00:23:17.651156+0000","last_undegraded":"2023-10-25T00:23:17.651156+0000","last_fullsized":"2023-10-25T00:23:17.651156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2023-10-25T00:23:17.651103+0000","last_deep_scrub":"73'13","last_deep_scrub_stamp":"2023-10-25T00:23:17.651103+0000","last_clean_scrub_stamp":"2023-10-25T00:23:17.651103+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:54.986553+0000","scrub_duration":0.014684019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:51.317813+0000","last_change":"2023-10-25T00:22:51.317813+0000","last_active":"2023-10-25T00:22:51.317813+0000","last_peered":"2023-10-25T00:22:51.317813+0000","last_clean":"2023-10-25T00:22:51.317813+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-25T00:22:51.317813+0000","last_undegraded":"2023-10-25T00:22:51.317813+0000","last_fullsized":"2023-10-25T00:22:51.317813+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.317462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:03:14.150576+0000","scrub_duration":0.0044508079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"73'249","reported_seq":354,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.397705+0000","last_change":"2023-10-25T00:23:21.397705+0000","last_active":"2023-10-25T00:23:21.397705+0000","last_peered":"2023-10-25T00:23:21.397705+0000","last_clean":"2023-10-25T00:23:21.397705+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-25T00:23:21.397705+0000","last_undegraded":"2023-10-25T00:23:21.397705+0000","last_fullsized":"2023-10-25T00:23:21.397705+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'249","last_scrub_stamp":"2023-10-25T00:23:21.397410+0000","last_deep_scrub":"73'249","last_deep_scrub_stamp":"2023-10-25T00:23:21.397410+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.397410+0000","objects_scrubbed":1,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:19:14.548554+0000","scrub_duration":0.017693341000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":62,"num_write":607,"num_write_kb":223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"72'5236","reported_seq":5324,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.899914+0000","last_change":"2023-10-25T00:23:34.900038+0000","last_active":"2023-10-25T00:23:34.899914+0000","last_peered":"2023-10-25T00:23:34.899914+0000","last_clean":"2023-10-25T00:23:34.899914+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-25T00:23:34.899914+0000","last_undegraded":"2023-10-25T00:23:34.899914+0000","last_fullsized":"2023-10-25T00:23:34.899914+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5236","last_scrub_stamp":"2023-10-25T00:23:34.899894+0000","last_deep_scrub":"72'5236","last_deep_scrub_stamp":"2023-10-25T00:23:34.899894+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.899894+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:10:20.497931+0000","scrub_duration":0.29917875999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":29554,"num_write_kb":368820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'5444","reported_seq":5534,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.141618+0000","last_change":"2023-10-25T00:23:00.141618+0000","last_active":"2023-10-25T00:23:00.141618+0000","last_peered":"2023-10-25T00:23:00.141618+0000","last_clean":"2023-10-25T00:23:00.141618+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-25T00:23:00.141618+0000","last_undegraded":"2023-10-25T00:23:00.141618+0000","last_fullsized":"2023-10-25T00:23:00.141618+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5444","last_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_deep_scrub":"68'5444","last_deep_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.141458+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:23:18.614380+0000","scrub_duration":0.197741424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28342,"num_write_kb":342931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"73'67","reported_seq":238,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.941625+0000","last_change":"2023-10-25T00:23:14.941868+0000","last_active":"2023-10-25T00:23:14.941625+0000","last_peered":"2023-10-25T00:23:14.941625+0000","last_clean":"2023-10-25T00:23:14.941625+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-25T00:23:14.941625+0000","last_undegraded":"2023-10-25T00:23:14.941625+0000","last_fullsized":"2023-10-25T00:23:14.941625+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'67","last_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_deep_scrub":"73'67","last_deep_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.941564+0000","objects_scrubbed":1,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:11:54.084201+0000","scrub_duration":0.0084463549999999991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":125,"num_write":61,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:22.074139+0000","last_change":"2023-10-25T00:23:22.074393+0000","last_active":"2023-10-25T00:23:22.074139+0000","last_peered":"2023-10-25T00:23:22.074139+0000","last_clean":"2023-10-25T00:23:22.074139+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-25T00:23:22.074139+0000","last_undegraded":"2023-10-25T00:23:22.074139+0000","last_fullsized":"2023-10-25T00:23:22.074139+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:22.074087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:22.074087+0000","last_clean_scrub_stamp":"2023-10-25T00:23:22.074087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:59:48.682336+0000","scrub_duration":0.0044663139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"72'10","reported_seq":138,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.225873+0000","last_change":"2023-10-25T00:22:45.425308+0000","last_active":"2023-10-25T00:22:49.225873+0000","last_peered":"2023-10-25T00:22:49.225873+0000","last_clean":"2023-10-25T00:22:49.225873+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-25T00:22:49.225873+0000","last_undegraded":"2023-10-25T00:22:49.225873+0000","last_fullsized":"2023-10-25T00:22:49.225873+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.425052+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:36:38.527058+0000","scrub_duration":0.14014134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":135,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"72'3666","reported_seq":3749,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.807801+0000","last_change":"2023-10-25T00:23:45.807972+0000","last_active":"2023-10-25T00:23:45.807801+0000","last_peered":"2023-10-25T00:23:45.807801+0000","last_clean":"2023-10-25T00:23:45.807801+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-25T00:23:45.807801+0000","last_undegraded":"2023-10-25T00:23:45.807801+0000","last_fullsized":"2023-10-25T00:23:45.807801+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3666","last_scrub_stamp":"2023-10-25T00:23:45.807770+0000","last_deep_scrub":"72'3666","last_deep_scrub_stamp":"2023-10-25T00:23:45.807770+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.807770+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:26:55.156976+0000","scrub_duration":0.28592742900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20519,"num_write_kb":254391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"68'4182","reported_seq":4269,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.489530+0000","last_change":"2023-10-25T00:23:02.489861+0000","last_active":"2023-10-25T00:23:02.489530+0000","last_peered":"2023-10-25T00:23:02.489530+0000","last_clean":"2023-10-25T00:23:02.489530+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-25T00:23:02.489530+0000","last_undegraded":"2023-10-25T00:23:02.489530+0000","last_fullsized":"2023-10-25T00:23:02.489530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4182","last_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_deep_scrub":"68'4182","last_deep_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.489471+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:14:08.666419+0000","scrub_duration":0.092197608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22839,"num_write_kb":259586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'4","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:15.544038+0000","last_change":"2023-10-25T00:23:15.544275+0000","last_active":"2023-10-25T00:23:15.544038+0000","last_peered":"2023-10-25T00:23:15.544038+0000","last_clean":"2023-10-25T00:23:15.544038+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-25T00:23:15.544038+0000","last_undegraded":"2023-10-25T00:23:15.544038+0000","last_fullsized":"2023-10-25T00:23:15.544038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_deep_scrub":"67'4","last_deep_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:15.543988+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:33:51.332032+0000","scrub_duration":0.004459345,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:28.260731+0000","last_change":"2023-10-25T00:23:28.260731+0000","last_active":"2023-10-25T00:23:28.260731+0000","last_peered":"2023-10-25T00:23:28.260731+0000","last_clean":"2023-10-25T00:23:28.260731+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-25T00:23:28.260731+0000","last_undegraded":"2023-10-25T00:23:28.260731+0000","last_fullsized":"2023-10-25T00:23:28.260731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:28.260409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:28.260409+0000","last_clean_scrub_stamp":"2023-10-25T00:23:28.260409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:10:44.862291+0000","scrub_duration":0.0042298140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'163","reported_seq":299,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:17.149545+0000","last_change":"2023-10-25T00:23:17.149791+0000","last_active":"2023-10-25T00:23:17.149545+0000","last_peered":"2023-10-25T00:23:17.149545+0000","last_clean":"2023-10-25T00:23:17.149545+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-25T00:23:17.149545+0000","last_undegraded":"2023-10-25T00:23:17.149545+0000","last_fullsized":"2023-10-25T00:23:17.149545+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'163","last_scrub_stamp":"2023-10-25T00:23:17.149492+0000","last_deep_scrub":"73'163","last_deep_scrub_stamp":"2023-10-25T00:23:17.149492+0000","last_clean_scrub_stamp":"2023-10-25T00:23:17.149492+0000","objects_scrubbed":2,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:59:05.394868+0000","scrub_duration":0.012985958000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"72'2","reported_seq":85,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:48.067150+0000","last_change":"2023-10-25T00:22:48.067150+0000","last_active":"2023-10-25T00:22:48.067150+0000","last_peered":"2023-10-25T00:22:48.067150+0000","last_clean":"2023-10-25T00:22:48.067150+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-25T00:22:48.067150+0000","last_undegraded":"2023-10-25T00:22:48.067150+0000","last_fullsized":"2023-10-25T00:22:48.067150+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.066810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:55:20.112674+0000","scrub_duration":0.0050122709999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"72'4105","reported_seq":4191,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.535400+0000","last_change":"2023-10-25T00:23:20.535530+0000","last_active":"2023-10-25T00:23:20.535400+0000","last_peered":"2023-10-25T00:23:20.535400+0000","last_clean":"2023-10-25T00:23:20.535400+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-25T00:23:20.535400+0000","last_undegraded":"2023-10-25T00:23:20.535400+0000","last_fullsized":"2023-10-25T00:23:20.535400+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4105","last_scrub_stamp":"2023-10-25T00:23:20.535378+0000","last_deep_scrub":"72'4105","last_deep_scrub_stamp":"2023-10-25T00:23:20.535378+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.535378+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:20:52.287445+0000","scrub_duration":0.31792611900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":19,"num_write":23147,"num_write_kb":297959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'5705","reported_seq":5793,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:39.836227+0000","last_change":"2023-10-25T00:23:39.836379+0000","last_active":"2023-10-25T00:23:39.836227+0000","last_peered":"2023-10-25T00:23:39.836227+0000","last_clean":"2023-10-25T00:23:39.836227+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-25T00:23:39.836227+0000","last_undegraded":"2023-10-25T00:23:39.836227+0000","last_fullsized":"2023-10-25T00:23:39.836227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5705","last_scrub_stamp":"2023-10-25T00:23:39.836205+0000","last_deep_scrub":"68'5705","last_deep_scrub_stamp":"2023-10-25T00:23:39.836205+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.836205+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:38:02.740862+0000","scrub_duration":0.105332388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":29845,"num_write_kb":357286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"68'5","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:18.535332+0000","last_change":"2023-10-25T00:23:18.535332+0000","last_active":"2023-10-25T00:23:18.535332+0000","last_peered":"2023-10-25T00:23:18.535332+0000","last_clean":"2023-10-25T00:23:18.535332+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-25T00:23:18.535332+0000","last_undegraded":"2023-10-25T00:23:18.535332+0000","last_fullsized":"2023-10-25T00:23:18.535332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:18.534951+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:18.534951+0000","last_clean_scrub_stamp":"2023-10-25T00:23:18.534951+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:37.756970+0000","scrub_duration":0.0037857049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"72'7","reported_seq":125,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.237196+0000","last_change":"2023-10-25T00:23:51.873544+0000","last_active":"2023-10-25T00:22:49.237196+0000","last_peered":"2023-10-25T00:22:49.237196+0000","last_clean":"2023-10-25T00:22:49.237196+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-25T00:22:49.237196+0000","last_undegraded":"2023-10-25T00:22:49.237196+0000","last_fullsized":"2023-10-25T00:22:49.237196+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:10.374501+0000","last_clean_scrub_stamp":"2023-10-24T23:53:10.374501+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"72'4234","reported_seq":4319,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:41.751577+0000","last_change":"2023-10-25T00:23:41.751679+0000","last_active":"2023-10-25T00:23:41.751577+0000","last_peered":"2023-10-25T00:23:41.751577+0000","last_clean":"2023-10-25T00:23:41.751577+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-25T00:23:41.751577+0000","last_undegraded":"2023-10-25T00:23:41.751577+0000","last_fullsized":"2023-10-25T00:23:41.751577+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4234","last_scrub_stamp":"2023-10-25T00:23:41.751558+0000","last_deep_scrub":"72'4234","last_deep_scrub_stamp":"2023-10-25T00:23:41.751558+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.751558+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:50:10.142078+0000","scrub_duration":0.256161054,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":15,"num_write":23173,"num_write_kb":302022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"68'4052","reported_seq":4138,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:49.042550+0000","last_change":"2023-10-25T00:23:49.042663+0000","last_active":"2023-10-25T00:23:49.042550+0000","last_peered":"2023-10-25T00:23:49.042550+0000","last_clean":"2023-10-25T00:23:49.042550+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-25T00:23:49.042550+0000","last_undegraded":"2023-10-25T00:23:49.042550+0000","last_fullsized":"2023-10-25T00:23:49.042550+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4052","last_scrub_stamp":"2023-10-25T00:23:49.042529+0000","last_deep_scrub":"68'4052","last_deep_scrub_stamp":"2023-10-25T00:23:49.042529+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.042529+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:40:32.211760+0000","scrub_duration":0.105715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21486,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"68'2","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.046955+0000","last_change":"2023-10-25T00:23:34.047108+0000","last_active":"2023-10-25T00:23:34.046955+0000","last_peered":"2023-10-25T00:23:34.046955+0000","last_clean":"2023-10-25T00:23:34.046955+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-25T00:23:34.046955+0000","last_undegraded":"2023-10-25T00:23:34.046955+0000","last_fullsized":"2023-10-25T00:23:34.046955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2023-10-25T00:23:34.046917+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2023-10-25T00:23:34.046917+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.046917+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:50:34.912538+0000","scrub_duration":0.034858717999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"73'5","reported_seq":109,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:53.546236+0000","last_change":"2023-10-25T00:22:53.546332+0000","last_active":"2023-10-25T00:22:53.546236+0000","last_peered":"2023-10-25T00:22:53.546236+0000","last_clean":"2023-10-25T00:22:53.546236+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-25T00:22:53.546236+0000","last_undegraded":"2023-10-25T00:22:53.546236+0000","last_fullsized":"2023-10-25T00:22:53.546236+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.546215+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:16:55.756192+0000","scrub_duration":0.27862564699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"72'4690","reported_seq":4775,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:42.556562+0000","last_change":"2023-10-25T00:23:42.556703+0000","last_active":"2023-10-25T00:23:42.556562+0000","last_peered":"2023-10-25T00:23:42.556562+0000","last_clean":"2023-10-25T00:23:42.556562+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-25T00:23:42.556562+0000","last_undegraded":"2023-10-25T00:23:42.556562+0000","last_fullsized":"2023-10-25T00:23:42.556562+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4690","last_scrub_stamp":"2023-10-25T00:23:42.556538+0000","last_deep_scrub":"72'4690","last_deep_scrub_stamp":"2023-10-25T00:23:42.556538+0000","last_clean_scrub_stamp":"2023-10-25T00:23:42.556538+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:36:54.839713+0000","scrub_duration":0.199820896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24994,"num_write_kb":341798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'4780","reported_seq":4868,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:09.092847+0000","last_change":"2023-10-25T00:23:09.093139+0000","last_active":"2023-10-25T00:23:09.092847+0000","last_peered":"2023-10-25T00:23:09.092847+0000","last_clean":"2023-10-25T00:23:09.092847+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-25T00:23:09.092847+0000","last_undegraded":"2023-10-25T00:23:09.092847+0000","last_fullsized":"2023-10-25T00:23:09.092847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4780","last_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_deep_scrub":"68'4780","last_deep_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.092796+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:58:49.347709+0000","scrub_duration":0.152359684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25083,"num_write_kb":293163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'3","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.218767+0000","last_change":"2023-10-25T00:22:48.328017+0000","last_active":"2023-10-25T00:22:49.218767+0000","last_peered":"2023-10-25T00:22:49.218767+0000","last_clean":"2023-10-25T00:22:49.218767+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-25T00:22:49.218767+0000","last_undegraded":"2023-10-25T00:22:49.218767+0000","last_fullsized":"2023-10-25T00:22:49.218767+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.327778+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:58.116941+0000","scrub_duration":0.061328572999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"72'5","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.576377+0000","last_change":"2023-10-25T00:22:55.576688+0000","last_active":"2023-10-25T00:22:55.576377+0000","last_peered":"2023-10-25T00:22:55.576377+0000","last_clean":"2023-10-25T00:22:55.576377+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-25T00:22:55.576377+0000","last_undegraded":"2023-10-25T00:22:55.576377+0000","last_fullsized":"2023-10-25T00:22:55.576377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.576324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:43.752897+0000","scrub_duration":0.0045481469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"72'3596","reported_seq":3672,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.236177+0000","last_change":"2023-10-25T00:23:56.936002+0000","last_active":"2023-10-25T00:22:49.236177+0000","last_peered":"2023-10-25T00:22:49.236177+0000","last_clean":"2023-10-25T00:22:49.236177+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-25T00:22:49.236177+0000","last_undegraded":"2023-10-25T00:22:49.236177+0000","last_fullsized":"2023-10-25T00:22:49.236177+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":19820,"num_write_kb":259784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"68'4970","reported_seq":5057,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.999669+0000","last_change":"2023-10-25T00:22:55.999798+0000","last_active":"2023-10-25T00:22:55.999669+0000","last_peered":"2023-10-25T00:22:55.999669+0000","last_clean":"2023-10-25T00:22:55.999669+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-25T00:22:55.999669+0000","last_undegraded":"2023-10-25T00:22:55.999669+0000","last_fullsized":"2023-10-25T00:22:55.999669+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4970","last_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_deep_scrub":"68'4970","last_deep_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.999644+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:15:47.681358+0000","scrub_duration":0.177095113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26721,"num_write_kb":311966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"73'1466","reported_seq":1605,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.466683+0000","last_change":"2023-10-25T00:23:06.467052+0000","last_active":"2023-10-25T00:23:06.466683+0000","last_peered":"2023-10-25T00:23:06.466683+0000","last_clean":"2023-10-25T00:23:06.466683+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-25T00:23:06.466683+0000","last_undegraded":"2023-10-25T00:23:06.466683+0000","last_fullsized":"2023-10-25T00:23:06.466683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1466","last_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_deep_scrub":"73'1466","last_deep_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.466569+0000","objects_scrubbed":2,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:21.696239+0000","scrub_duration":0.039946036999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":47798,"num_write":1439,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'7","reported_seq":112,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:53.616510+0000","last_change":"2023-10-25T00:23:53.616774+0000","last_active":"2023-10-25T00:23:53.616510+0000","last_peered":"2023-10-25T00:23:53.616510+0000","last_clean":"2023-10-25T00:23:53.616510+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-25T00:23:53.616510+0000","last_undegraded":"2023-10-25T00:23:53.616510+0000","last_fullsized":"2023-10-25T00:23:53.616510+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'7","last_scrub_stamp":"2023-10-25T00:23:53.616451+0000","last_deep_scrub":"73'7","last_deep_scrub_stamp":"2023-10-25T00:23:53.616451+0000","last_clean_scrub_stamp":"2023-10-25T00:23:53.616451+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:31:57.298978+0000","scrub_duration":0.13500319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"72'3548","reported_seq":3634,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:32.093782+0000","last_change":"2023-10-25T00:23:32.093782+0000","last_active":"2023-10-25T00:23:32.093782+0000","last_peered":"2023-10-25T00:23:32.093782+0000","last_clean":"2023-10-25T00:23:32.093782+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-25T00:23:32.093782+0000","last_undegraded":"2023-10-25T00:23:32.093782+0000","last_fullsized":"2023-10-25T00:23:32.093782+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3548","last_scrub_stamp":"2023-10-25T00:23:32.093615+0000","last_deep_scrub":"72'3548","last_deep_scrub_stamp":"2023-10-25T00:23:32.093615+0000","last_clean_scrub_stamp":"2023-10-25T00:23:32.093615+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:02:03.851334+0000","scrub_duration":0.28610911100000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20243,"num_write_kb":261539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"68'3393","reported_seq":3482,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.098602+0000","last_change":"2023-10-25T00:22:42.098602+0000","last_active":"2023-10-25T00:22:42.098602+0000","last_peered":"2023-10-25T00:22:42.098602+0000","last_clean":"2023-10-25T00:22:42.098602+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-25T00:22:42.098602+0000","last_undegraded":"2023-10-25T00:22:42.098602+0000","last_fullsized":"2023-10-25T00:22:42.098602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3393","last_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_deep_scrub":"68'3393","last_deep_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.098371+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:12.736136+0000","scrub_duration":0.153676797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17500,"num_write_kb":217638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"68'5","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:33.402669+0000","last_change":"2023-10-25T00:23:33.402936+0000","last_active":"2023-10-25T00:23:33.402669+0000","last_peered":"2023-10-25T00:23:33.402669+0000","last_clean":"2023-10-25T00:23:33.402669+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-25T00:23:33.402669+0000","last_undegraded":"2023-10-25T00:23:33.402669+0000","last_fullsized":"2023-10-25T00:23:33.402669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:33.402617+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:33.402617+0000","last_clean_scrub_stamp":"2023-10-25T00:23:33.402617+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:10:14.715542+0000","scrub_duration":0.0048941419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"72'3","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.419583+0000","last_change":"2023-10-25T00:23:45.419862+0000","last_active":"2023-10-25T00:23:45.419583+0000","last_peered":"2023-10-25T00:23:45.419583+0000","last_clean":"2023-10-25T00:23:45.419583+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-25T00:23:45.419583+0000","last_undegraded":"2023-10-25T00:23:45.419583+0000","last_fullsized":"2023-10-25T00:23:45.419583+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:45.419528+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:45.419528+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.419528+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:07:59.973530+0000","scrub_duration":0.0045258490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"72'3584","reported_seq":3670,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.285501+0000","last_change":"2023-10-25T00:23:40.285625+0000","last_active":"2023-10-25T00:23:40.285501+0000","last_peered":"2023-10-25T00:23:40.285501+0000","last_clean":"2023-10-25T00:23:40.285501+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-25T00:23:40.285501+0000","last_undegraded":"2023-10-25T00:23:40.285501+0000","last_fullsized":"2023-10-25T00:23:40.285501+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3584","last_scrub_stamp":"2023-10-25T00:23:40.285478+0000","last_deep_scrub":"72'3584","last_deep_scrub_stamp":"2023-10-25T00:23:40.285478+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.285478+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:11:39.447639+0000","scrub_duration":0.26816689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20504,"num_write_kb":254394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'5094","reported_seq":5186,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:51.485282+0000","last_change":"2023-10-25T00:22:51.485282+0000","last_active":"2023-10-25T00:22:51.485282+0000","last_peered":"2023-10-25T00:22:51.485282+0000","last_clean":"2023-10-25T00:22:51.485282+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-25T00:22:51.485282+0000","last_undegraded":"2023-10-25T00:22:51.485282+0000","last_fullsized":"2023-10-25T00:22:51.485282+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5094","last_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_deep_scrub":"68'5094","last_deep_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.484917+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:22:43.899134+0000","scrub_duration":0.19299439099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27763,"num_write_kb":328490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"73'7","reported_seq":116,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.094834+0000","last_change":"2023-10-25T00:23:27.095106+0000","last_active":"2023-10-25T00:23:27.094834+0000","last_peered":"2023-10-25T00:23:27.094834+0000","last_clean":"2023-10-25T00:23:27.094834+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-25T00:23:27.094834+0000","last_undegraded":"2023-10-25T00:23:27.094834+0000","last_fullsized":"2023-10-25T00:23:27.094834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'7","last_scrub_stamp":"2023-10-25T00:23:27.094773+0000","last_deep_scrub":"73'7","last_deep_scrub_stamp":"2023-10-25T00:23:27.094773+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.094773+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:52:00.838368+0000","scrub_duration":0.007318275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"73'1440","reported_seq":1570,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.527480+0000","last_change":"2023-10-25T00:23:16.527658+0000","last_active":"2023-10-25T00:23:16.527480+0000","last_peered":"2023-10-25T00:23:16.527480+0000","last_clean":"2023-10-25T00:23:16.527480+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-25T00:23:16.527480+0000","last_undegraded":"2023-10-25T00:23:16.527480+0000","last_fullsized":"2023-10-25T00:23:16.527480+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1440","last_scrub_stamp":"2023-10-25T00:23:16.527441+0000","last_deep_scrub":"73'1440","last_deep_scrub_stamp":"2023-10-25T00:23:16.527441+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.527441+0000","objects_scrubbed":2,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:47:53.223550+0000","scrub_duration":0.234060301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":35275,"num_write":1412,"num_write_kb":4487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"72'3292","reported_seq":3378,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.971739+0000","last_change":"2023-10-25T00:23:30.971878+0000","last_active":"2023-10-25T00:23:30.971739+0000","last_peered":"2023-10-25T00:23:30.971739+0000","last_clean":"2023-10-25T00:23:30.971739+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-25T00:23:30.971739+0000","last_undegraded":"2023-10-25T00:23:30.971739+0000","last_fullsized":"2023-10-25T00:23:30.971739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3292","last_scrub_stamp":"2023-10-25T00:23:30.971718+0000","last_deep_scrub":"72'3292","last_deep_scrub_stamp":"2023-10-25T00:23:30.971718+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.971718+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:45:07.787712+0000","scrub_duration":0.347700487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17853,"num_write_kb":243602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'4242","reported_seq":4329,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232419+0000","last_change":"2023-10-25T00:22:45.893397+0000","last_active":"2023-10-25T00:22:49.232419+0000","last_peered":"2023-10-25T00:22:49.232419+0000","last_clean":"2023-10-25T00:22:49.232419+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-25T00:22:49.232419+0000","last_undegraded":"2023-10-25T00:22:49.232419+0000","last_fullsized":"2023-10-25T00:22:49.232419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4242","last_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_deep_scrub":"67'4242","last_deep_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.893150+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:24:28.749675+0000","scrub_duration":0.21189509300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22687,"num_write_kb":275346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"73'8","reported_seq":140,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.063186+0000","last_change":"2023-10-25T00:23:30.063493+0000","last_active":"2023-10-25T00:23:30.063186+0000","last_peered":"2023-10-25T00:23:30.063186+0000","last_clean":"2023-10-25T00:23:30.063186+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-25T00:23:30.063186+0000","last_undegraded":"2023-10-25T00:23:30.063186+0000","last_fullsized":"2023-10-25T00:23:30.063186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2023-10-25T00:23:30.063132+0000","last_deep_scrub":"73'8","last_deep_scrub_stamp":"2023-10-25T00:23:30.063132+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.063132+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:12:23.891399+0000","scrub_duration":0.0076496660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"72'5","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:19.320206+0000","last_change":"2023-10-25T00:23:19.320206+0000","last_active":"2023-10-25T00:23:19.320206+0000","last_peered":"2023-10-25T00:23:19.320206+0000","last_clean":"2023-10-25T00:23:19.320206+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-25T00:23:19.320206+0000","last_undegraded":"2023-10-25T00:23:19.320206+0000","last_fullsized":"2023-10-25T00:23:19.320206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:23:19.319840+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:23:19.319840+0000","last_clean_scrub_stamp":"2023-10-25T00:23:19.319840+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:18:41.713579+0000","scrub_duration":0.0042872409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"72'4091","reported_seq":4177,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:50.884272+0000","last_change":"2023-10-25T00:23:50.884393+0000","last_active":"2023-10-25T00:23:50.884272+0000","last_peered":"2023-10-25T00:23:50.884272+0000","last_clean":"2023-10-25T00:23:50.884272+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-25T00:23:50.884272+0000","last_undegraded":"2023-10-25T00:23:50.884272+0000","last_fullsized":"2023-10-25T00:23:50.884272+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4091","last_scrub_stamp":"2023-10-25T00:23:50.884251+0000","last_deep_scrub":"72'4091","last_deep_scrub_stamp":"2023-10-25T00:23:50.884251+0000","last_clean_scrub_stamp":"2023-10-25T00:23:50.884251+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:21:51.075072+0000","scrub_duration":0.36544204200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22779,"num_write_kb":288098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'4310","reported_seq":4397,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:22.394323+0000","last_change":"2023-10-25T00:23:22.394323+0000","last_active":"2023-10-25T00:23:22.394323+0000","last_peered":"2023-10-25T00:23:22.394323+0000","last_clean":"2023-10-25T00:23:22.394323+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-25T00:23:22.394323+0000","last_undegraded":"2023-10-25T00:23:22.394323+0000","last_fullsized":"2023-10-25T00:23:22.394323+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4310","last_scrub_stamp":"2023-10-25T00:23:22.393940+0000","last_deep_scrub":"67'4310","last_deep_scrub_stamp":"2023-10-25T00:23:22.393940+0000","last_clean_scrub_stamp":"2023-10-25T00:23:22.393940+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:24:46.673127+0000","scrub_duration":0.085301248999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23791,"num_write_kb":283914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"73'5","reported_seq":114,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.183468+0000","last_change":"2023-10-25T00:23:26.183662+0000","last_active":"2023-10-25T00:23:26.183468+0000","last_peered":"2023-10-25T00:23:26.183468+0000","last_clean":"2023-10-25T00:23:26.183468+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-25T00:23:26.183468+0000","last_undegraded":"2023-10-25T00:23:26.183468+0000","last_fullsized":"2023-10-25T00:23:26.183468+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:23:26.183430+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:23:26.183430+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.183430+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:28:22.710805+0000","scrub_duration":0.110908062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"72'2","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.206968+0000","last_change":"2023-10-25T00:22:38.034628+0000","last_active":"2023-10-25T00:22:49.206968+0000","last_peered":"2023-10-25T00:22:49.206968+0000","last_clean":"2023-10-25T00:22:49.206968+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-25T00:22:49.206968+0000","last_undegraded":"2023-10-25T00:22:49.206968+0000","last_fullsized":"2023-10-25T00:22:49.206968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.034499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:09.450121+0000","scrub_duration":0.14675892500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"72'3695","reported_seq":3781,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:51.251436+0000","last_change":"2023-10-25T00:23:51.251559+0000","last_active":"2023-10-25T00:23:51.251436+0000","last_peered":"2023-10-25T00:23:51.251436+0000","last_clean":"2023-10-25T00:23:51.251436+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-25T00:23:51.251436+0000","last_undegraded":"2023-10-25T00:23:51.251436+0000","last_fullsized":"2023-10-25T00:23:51.251436+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3695","last_scrub_stamp":"2023-10-25T00:23:51.251416+0000","last_deep_scrub":"72'3695","last_deep_scrub_stamp":"2023-10-25T00:23:51.251416+0000","last_clean_scrub_stamp":"2023-10-25T00:23:51.251416+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:44:06.748969+0000","scrub_duration":0.274463921,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21426,"num_write_kb":254037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"72'3","reported_seq":84,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:42.331112+0000","last_change":"2023-10-25T00:22:42.331112+0000","last_active":"2023-10-25T00:22:42.331112+0000","last_peered":"2023-10-25T00:22:42.331112+0000","last_clean":"2023-10-25T00:22:42.331112+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-25T00:22:42.331112+0000","last_undegraded":"2023-10-25T00:22:42.331112+0000","last_fullsized":"2023-10-25T00:22:42.331112+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.330786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:23:56.573805+0000","scrub_duration":0.0043674530000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"72'19","reported_seq":218,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.645016+0000","last_change":"2023-10-25T00:22:47.253320+0000","last_active":"2023-10-25T00:22:52.645016+0000","last_peered":"2023-10-25T00:22:52.645016+0000","last_clean":"2023-10-25T00:22:52.645016+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-25T00:22:52.645016+0000","last_undegraded":"2023-10-25T00:22:52.645016+0000","last_fullsized":"2023-10-25T00:22:52.645016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_deep_scrub":"72'19","last_deep_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.252995+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:03.720456+0000","scrub_duration":0.018599613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":339,"num_read_kb":339,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'5249","reported_seq":5342,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:52.439199+0000","last_change":"2023-10-25T00:22:52.439461+0000","last_active":"2023-10-25T00:22:52.439199+0000","last_peered":"2023-10-25T00:22:52.439199+0000","last_clean":"2023-10-25T00:22:52.439199+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-25T00:22:52.439199+0000","last_undegraded":"2023-10-25T00:22:52.439199+0000","last_fullsized":"2023-10-25T00:22:52.439199+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5249","last_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_deep_scrub":"68'5249","last_deep_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.439150+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:19:03.730037+0000","scrub_duration":0.077069795999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":27646,"num_write_kb":332285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"72'4380","reported_seq":4464,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:38.646492+0000","last_change":"2023-10-25T00:23:38.646626+0000","last_active":"2023-10-25T00:23:38.646492+0000","last_peered":"2023-10-25T00:23:38.646492+0000","last_clean":"2023-10-25T00:23:38.646492+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-25T00:23:38.646492+0000","last_undegraded":"2023-10-25T00:23:38.646492+0000","last_fullsized":"2023-10-25T00:23:38.646492+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4380","last_scrub_stamp":"2023-10-25T00:23:38.646472+0000","last_deep_scrub":"72'4380","last_deep_scrub_stamp":"2023-10-25T00:23:38.646472+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.646472+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:38:56.172654+0000","scrub_duration":0.29662724200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23662,"num_write_kb":324338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"72'5","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:50.617951+0000","last_change":"2023-10-25T00:22:50.618067+0000","last_active":"2023-10-25T00:22:50.617951+0000","last_peered":"2023-10-25T00:22:50.617951+0000","last_clean":"2023-10-25T00:22:50.617951+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-25T00:22:50.617951+0000","last_undegraded":"2023-10-25T00:22:50.617951+0000","last_fullsized":"2023-10-25T00:22:50.617951+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.617931+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:24.381368+0000","scrub_duration":0.234626104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"68'5","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:25.067489+0000","last_change":"2023-10-25T00:23:25.067489+0000","last_active":"2023-10-25T00:23:25.067489+0000","last_peered":"2023-10-25T00:23:25.067489+0000","last_clean":"2023-10-25T00:23:25.067489+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-25T00:23:25.067489+0000","last_undegraded":"2023-10-25T00:23:25.067489+0000","last_fullsized":"2023-10-25T00:23:25.067489+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:25.067146+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:25.067146+0000","last_clean_scrub_stamp":"2023-10-25T00:23:25.067146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:32.009959+0000","scrub_duration":0.0041696520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'4442","reported_seq":4527,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:29.981270+0000","last_change":"2023-10-25T00:23:29.981423+0000","last_active":"2023-10-25T00:23:29.981270+0000","last_peered":"2023-10-25T00:23:29.981270+0000","last_clean":"2023-10-25T00:23:29.981270+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-25T00:23:29.981270+0000","last_undegraded":"2023-10-25T00:23:29.981270+0000","last_fullsized":"2023-10-25T00:23:29.981270+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4442","last_scrub_stamp":"2023-10-25T00:23:29.981245+0000","last_deep_scrub":"68'4442","last_deep_scrub_stamp":"2023-10-25T00:23:29.981245+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.981245+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:11:38.194434+0000","scrub_duration":0.109448791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22810,"num_write_kb":296225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"72'4076","reported_seq":4161,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:07.280492+0000","last_change":"2023-10-25T00:24:07.280653+0000","last_active":"2023-10-25T00:24:07.280492+0000","last_peered":"2023-10-25T00:24:07.280492+0000","last_clean":"2023-10-25T00:24:07.280492+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-25T00:24:07.280492+0000","last_undegraded":"2023-10-25T00:24:07.280492+0000","last_fullsized":"2023-10-25T00:24:07.280492+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4076","last_scrub_stamp":"2023-10-25T00:24:07.280471+0000","last_deep_scrub":"72'4076","last_deep_scrub_stamp":"2023-10-25T00:24:07.280471+0000","last_clean_scrub_stamp":"2023-10-25T00:24:07.280471+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:36:18.063026+0000","scrub_duration":0.198538093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22055,"num_write_kb":297609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"68'3","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:23.928686+0000","last_change":"2023-10-25T00:23:23.928686+0000","last_active":"2023-10-25T00:23:23.928686+0000","last_peered":"2023-10-25T00:23:23.928686+0000","last_clean":"2023-10-25T00:23:23.928686+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-25T00:23:23.928686+0000","last_undegraded":"2023-10-25T00:23:23.928686+0000","last_fullsized":"2023-10-25T00:23:23.928686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:23.928352+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:23.928352+0000","last_clean_scrub_stamp":"2023-10-25T00:23:23.928352+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:41:04.304281+0000","scrub_duration":0.0043703589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'4101","reported_seq":4187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:50.337590+0000","last_change":"2023-10-25T00:22:50.337815+0000","last_active":"2023-10-25T00:22:50.337590+0000","last_peered":"2023-10-25T00:22:50.337590+0000","last_clean":"2023-10-25T00:22:50.337590+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-25T00:22:50.337590+0000","last_undegraded":"2023-10-25T00:22:50.337590+0000","last_fullsized":"2023-10-25T00:22:50.337590+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.337536+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:01.275651+0000","scrub_duration":0.055303021000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22343,"num_write_kb":253486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"72'397","reported_seq":591,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.234775+0000","last_change":"2023-10-25T00:22:45.554898+0000","last_active":"2023-10-25T00:22:49.234775+0000","last_peered":"2023-10-25T00:22:49.234775+0000","last_clean":"2023-10-25T00:22:49.234775+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-25T00:22:49.234775+0000","last_undegraded":"2023-10-25T00:22:49.234775+0000","last_fullsized":"2023-10-25T00:22:49.234775+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'397","last_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_deep_scrub":"72'397","last_deep_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.554526+0000","objects_scrubbed":4,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:41:32.056818+0000","scrub_duration":0.10248053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":86197,"num_write":389,"num_write_kb":4306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"72'4606","reported_seq":4689,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.057352+0000","last_change":"2023-10-25T00:23:37.057459+0000","last_active":"2023-10-25T00:23:37.057352+0000","last_peered":"2023-10-25T00:23:37.057352+0000","last_clean":"2023-10-25T00:23:37.057352+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-25T00:23:37.057352+0000","last_undegraded":"2023-10-25T00:23:37.057352+0000","last_fullsized":"2023-10-25T00:23:37.057352+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4606","last_scrub_stamp":"2023-10-25T00:23:37.057332+0000","last_deep_scrub":"72'4606","last_deep_scrub_stamp":"2023-10-25T00:23:37.057332+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.057332+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:16:39.313888+0000","scrub_duration":0.28055820399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25763,"num_write_kb":316477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"73'14","reported_seq":146,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.153043+0000","last_change":"2023-10-25T00:23:16.153043+0000","last_active":"2023-10-25T00:23:16.153043+0000","last_peered":"2023-10-25T00:23:16.153043+0000","last_clean":"2023-10-25T00:23:16.153043+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-25T00:23:16.153043+0000","last_undegraded":"2023-10-25T00:23:16.153043+0000","last_fullsized":"2023-10-25T00:23:16.153043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_deep_scrub":"73'14","last_deep_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.152723+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:32:30.128331+0000","scrub_duration":0.010186763999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'5680","reported_seq":5770,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:54.586176+0000","last_change":"2023-10-25T00:22:54.586176+0000","last_active":"2023-10-25T00:22:54.586176+0000","last_peered":"2023-10-25T00:22:54.586176+0000","last_clean":"2023-10-25T00:22:54.586176+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-25T00:22:54.586176+0000","last_undegraded":"2023-10-25T00:22:54.586176+0000","last_fullsized":"2023-10-25T00:22:54.586176+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5680","last_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_deep_scrub":"68'5680","last_deep_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.585866+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:49.342364+0000","scrub_duration":0.20835251299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29749,"num_write_kb":360962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"73'40","reported_seq":187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:59.279339+0000","last_change":"2023-10-25T00:22:59.279527+0000","last_active":"2023-10-25T00:22:59.279339+0000","last_peered":"2023-10-25T00:22:59.279339+0000","last_clean":"2023-10-25T00:22:59.279339+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-25T00:22:59.279339+0000","last_undegraded":"2023-10-25T00:22:59.279339+0000","last_fullsized":"2023-10-25T00:22:59.279339+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'40","last_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_deep_scrub":"73'40","last_deep_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.279288+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:52.911132+0000","scrub_duration":0.0094949800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":39,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"72'4808","reported_seq":4897,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.790951+0000","last_change":"2023-10-25T00:23:14.791099+0000","last_active":"2023-10-25T00:23:14.790951+0000","last_peered":"2023-10-25T00:23:14.790951+0000","last_clean":"2023-10-25T00:23:14.790951+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-25T00:23:14.790951+0000","last_undegraded":"2023-10-25T00:23:14.790951+0000","last_fullsized":"2023-10-25T00:23:14.790951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4808","last_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_deep_scrub":"72'4808","last_deep_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.790929+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:56:36.982903+0000","scrub_duration":0.29119922999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26552,"num_write_kb":336125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"68'4747","reported_seq":4835,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.496098+0000","last_change":"2023-10-25T00:23:00.496309+0000","last_active":"2023-10-25T00:23:00.496098+0000","last_peered":"2023-10-25T00:23:00.496098+0000","last_clean":"2023-10-25T00:23:00.496098+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-25T00:23:00.496098+0000","last_undegraded":"2023-10-25T00:23:00.496098+0000","last_fullsized":"2023-10-25T00:23:00.496098+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4747","last_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_deep_scrub":"68'4747","last_deep_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.496060+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:56.531125+0000","scrub_duration":0.089432630999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24847,"num_write_kb":316685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"72'4283","reported_seq":4370,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:08.728587+0000","last_change":"2023-10-25T00:23:08.728718+0000","last_active":"2023-10-25T00:23:08.728587+0000","last_peered":"2023-10-25T00:23:08.728587+0000","last_clean":"2023-10-25T00:23:08.728587+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-25T00:23:08.728587+0000","last_undegraded":"2023-10-25T00:23:08.728587+0000","last_fullsized":"2023-10-25T00:23:08.728587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4283","last_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_deep_scrub":"72'4283","last_deep_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_clean_scrub_stamp":"2023-10-25T00:23:08.728567+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:33.655445+0000","scrub_duration":0.26846236600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23853,"num_write_kb":296504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"68'4036","reported_seq":4124,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:39.642537+0000","last_change":"2023-10-25T00:22:39.642537+0000","last_active":"2023-10-25T00:22:39.642537+0000","last_peered":"2023-10-25T00:22:39.642537+0000","last_clean":"2023-10-25T00:22:39.642537+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-25T00:22:39.642537+0000","last_undegraded":"2023-10-25T00:22:39.642537+0000","last_fullsized":"2023-10-25T00:22:39.642537+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4036","last_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_deep_scrub":"68'4036","last_deep_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.642342+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:02.362976+0000","scrub_duration":0.227585228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21233,"num_write_kb":273467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"72'4276","reported_seq":4362,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.997835+0000","last_change":"2023-10-25T00:23:30.997946+0000","last_active":"2023-10-25T00:23:30.997835+0000","last_peered":"2023-10-25T00:23:30.997835+0000","last_clean":"2023-10-25T00:23:30.997835+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-25T00:23:30.997835+0000","last_undegraded":"2023-10-25T00:23:30.997835+0000","last_fullsized":"2023-10-25T00:23:30.997835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4276","last_scrub_stamp":"2023-10-25T00:23:30.997816+0000","last_deep_scrub":"72'4276","last_deep_scrub_stamp":"2023-10-25T00:23:30.997816+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.997816+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:08.863577+0000","scrub_duration":0.39360271099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23362,"num_write_kb":310817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"68'6201","reported_seq":6281,"reported_epoch":73,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-25T00:22:49.235390+0000","last_change":"2023-10-25T00:23:50.848970+0000","last_active":"2023-10-25T00:22:49.235390+0000","last_peered":"2023-10-25T00:22:49.235390+0000","last_clean":"2023-10-25T00:22:49.235390+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-25T00:22:49.235390+0000","last_undegraded":"2023-10-25T00:22:49.235390+0000","last_fullsized":"2023-10-25T00:22:49.235390+0000","mapping_epoch":21,"log_start":"62'1000","ondisk_log_start":"62'1000","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:02.290575+0000","last_clean_scrub_stamp":"2023-10-24T23:53:02.290575+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33486,"num_write_kb":384441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"72'4018","reported_seq":4105,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:52.269664+0000","last_change":"2023-10-25T00:23:52.269793+0000","last_active":"2023-10-25T00:23:52.269664+0000","last_peered":"2023-10-25T00:23:52.269664+0000","last_clean":"2023-10-25T00:23:52.269664+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-25T00:23:52.269664+0000","last_undegraded":"2023-10-25T00:23:52.269664+0000","last_fullsized":"2023-10-25T00:23:52.269664+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4018","last_scrub_stamp":"2023-10-25T00:23:52.269641+0000","last_deep_scrub":"72'4018","last_deep_scrub_stamp":"2023-10-25T00:23:52.269641+0000","last_clean_scrub_stamp":"2023-10-25T00:23:52.269641+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:58:28.387402+0000","scrub_duration":0.26794800299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":21920,"num_write_kb":288651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"68'4522","reported_seq":4611,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.751456+0000","last_change":"2023-10-25T00:23:26.751600+0000","last_active":"2023-10-25T00:23:26.751456+0000","last_peered":"2023-10-25T00:23:26.751456+0000","last_clean":"2023-10-25T00:23:26.751456+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-25T00:23:26.751456+0000","last_undegraded":"2023-10-25T00:23:26.751456+0000","last_fullsized":"2023-10-25T00:23:26.751456+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4522","last_scrub_stamp":"2023-10-25T00:23:26.751435+0000","last_deep_scrub":"68'4522","last_deep_scrub_stamp":"2023-10-25T00:23:26.751435+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.751435+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:40:36.338773+0000","scrub_duration":0.110684227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24476,"num_write_kb":282500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"72'4209","reported_seq":4293,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.054561+0000","last_change":"2023-10-25T00:23:37.054692+0000","last_active":"2023-10-25T00:23:37.054561+0000","last_peered":"2023-10-25T00:23:37.054561+0000","last_clean":"2023-10-25T00:23:37.054561+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-25T00:23:37.054561+0000","last_undegraded":"2023-10-25T00:23:37.054561+0000","last_fullsized":"2023-10-25T00:23:37.054561+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4209","last_scrub_stamp":"2023-10-25T00:23:37.054538+0000","last_deep_scrub":"72'4209","last_deep_scrub_stamp":"2023-10-25T00:23:37.054538+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.054538+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:39:40.159884+0000","scrub_duration":0.40759778099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24141,"num_write_kb":281325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"68'4915","reported_seq":5002,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232375+0000","last_change":"2023-10-25T00:22:38.600278+0000","last_active":"2023-10-25T00:22:49.232375+0000","last_peered":"2023-10-25T00:22:49.232375+0000","last_clean":"2023-10-25T00:22:49.232375+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-25T00:22:49.232375+0000","last_undegraded":"2023-10-25T00:22:49.232375+0000","last_fullsized":"2023-10-25T00:22:49.232375+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4915","last_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_deep_scrub":"68'4915","last_deep_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.600135+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:42:30.671003+0000","scrub_duration":0.18827047799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26161,"num_write_kb":314084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"72'4933","reported_seq":5017,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:31.791243+0000","last_change":"2023-10-25T00:23:31.791243+0000","last_active":"2023-10-25T00:23:31.791243+0000","last_peered":"2023-10-25T00:23:31.791243+0000","last_clean":"2023-10-25T00:23:31.791243+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-25T00:23:31.791243+0000","last_undegraded":"2023-10-25T00:23:31.791243+0000","last_fullsized":"2023-10-25T00:23:31.791243+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4933","last_scrub_stamp":"2023-10-25T00:23:31.791069+0000","last_deep_scrub":"72'4933","last_deep_scrub_stamp":"2023-10-25T00:23:31.791069+0000","last_clean_scrub_stamp":"2023-10-25T00:23:31.791069+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:53:40.465688+0000","scrub_duration":0.269140133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26695,"num_write_kb":352905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"68'5055","reported_seq":5143,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.217349+0000","last_change":"2023-10-25T00:22:43.444464+0000","last_active":"2023-10-25T00:22:49.217349+0000","last_peered":"2023-10-25T00:22:49.217349+0000","last_clean":"2023-10-25T00:22:49.217349+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-25T00:22:49.217349+0000","last_undegraded":"2023-10-25T00:22:49.217349+0000","last_fullsized":"2023-10-25T00:22:49.217349+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5055","last_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_deep_scrub":"68'5055","last_deep_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_clean_scrub_stamp":"2023-10-25T00:22:43.444210+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:13:34.310177+0000","scrub_duration":0.081291211000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27516,"num_write_kb":327406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"72'4083","reported_seq":4171,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:47.749612+0000","last_change":"2023-10-25T00:23:47.749735+0000","last_active":"2023-10-25T00:23:47.749612+0000","last_peered":"2023-10-25T00:23:47.749612+0000","last_clean":"2023-10-25T00:23:47.749612+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-25T00:23:47.749612+0000","last_undegraded":"2023-10-25T00:23:47.749612+0000","last_fullsized":"2023-10-25T00:23:47.749612+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4083","last_scrub_stamp":"2023-10-25T00:23:47.749591+0000","last_deep_scrub":"72'4083","last_deep_scrub_stamp":"2023-10-25T00:23:47.749591+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.749591+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:43:07.403801+0000","scrub_duration":0.25918773899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":22892,"num_write_kb":296872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"68'4777","reported_seq":4866,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:24.846143+0000","last_change":"2023-10-25T00:23:24.846143+0000","last_active":"2023-10-25T00:23:24.846143+0000","last_peered":"2023-10-25T00:23:24.846143+0000","last_clean":"2023-10-25T00:23:24.846143+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-25T00:23:24.846143+0000","last_undegraded":"2023-10-25T00:23:24.846143+0000","last_fullsized":"2023-10-25T00:23:24.846143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4777","last_scrub_stamp":"2023-10-25T00:23:24.846002+0000","last_deep_scrub":"68'4777","last_deep_scrub_stamp":"2023-10-25T00:23:24.846002+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.846002+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:41:16.418283+0000","scrub_duration":0.19199595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25812,"num_write_kb":300181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"72'4618","reported_seq":4708,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:33.839487+0000","last_change":"2023-10-25T00:23:33.839487+0000","last_active":"2023-10-25T00:23:33.839487+0000","last_peered":"2023-10-25T00:23:33.839487+0000","last_clean":"2023-10-25T00:23:33.839487+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-25T00:23:33.839487+0000","last_undegraded":"2023-10-25T00:23:33.839487+0000","last_fullsized":"2023-10-25T00:23:33.839487+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4618","last_scrub_stamp":"2023-10-25T00:23:33.839138+0000","last_deep_scrub":"72'4618","last_deep_scrub_stamp":"2023-10-25T00:23:33.839138+0000","last_clean_scrub_stamp":"2023-10-25T00:23:33.839138+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:25:32.655697+0000","scrub_duration":0.330454572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":26180,"num_write_kb":326342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"68'5340","reported_seq":5427,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.231448+0000","last_change":"2023-10-25T00:22:42.520048+0000","last_active":"2023-10-25T00:22:49.231448+0000","last_peered":"2023-10-25T00:22:49.231448+0000","last_clean":"2023-10-25T00:22:49.231448+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-25T00:22:49.231448+0000","last_undegraded":"2023-10-25T00:22:49.231448+0000","last_fullsized":"2023-10-25T00:22:49.231448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5340","last_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_deep_scrub":"68'5340","last_deep_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.519886+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.832889+0000","scrub_duration":0.129134375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28701,"num_write_kb":348521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"72'4171","reported_seq":4245,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:36.060881+0000","last_change":"2023-10-24T23:53:11.835710+0000","last_active":"2023-10-25T00:22:36.060881+0000","last_peered":"2023-10-25T00:22:36.060881+0000","last_clean":"2023-10-25T00:22:36.060881+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-25T00:22:36.060881+0000","last_undegraded":"2023-10-25T00:22:36.060881+0000","last_fullsized":"2023-10-25T00:22:36.060881+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22551,"num_write_kb":303498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"68'4493","reported_seq":4581,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:01.322975+0000","last_change":"2023-10-25T00:23:01.323167+0000","last_active":"2023-10-25T00:23:01.322975+0000","last_peered":"2023-10-25T00:23:01.322975+0000","last_clean":"2023-10-25T00:23:01.322975+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-25T00:23:01.322975+0000","last_undegraded":"2023-10-25T00:23:01.322975+0000","last_fullsized":"2023-10-25T00:23:01.322975+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4493","last_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_deep_scrub":"68'4493","last_deep_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_clean_scrub_stamp":"2023-10-25T00:23:01.322941+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:49.029022+0000","scrub_duration":0.13616751799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23797,"num_write_kb":300260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"72'4763","reported_seq":4847,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:57.009552+0000","last_change":"2023-10-25T00:23:57.009674+0000","last_active":"2023-10-25T00:23:57.009552+0000","last_peered":"2023-10-25T00:23:57.009552+0000","last_clean":"2023-10-25T00:23:57.009552+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-25T00:23:57.009552+0000","last_undegraded":"2023-10-25T00:23:57.009552+0000","last_fullsized":"2023-10-25T00:23:57.009552+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4763","last_scrub_stamp":"2023-10-25T00:23:57.009531+0000","last_deep_scrub":"72'4763","last_deep_scrub_stamp":"2023-10-25T00:23:57.009531+0000","last_clean_scrub_stamp":"2023-10-25T00:23:57.009531+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:34:10.319674+0000","scrub_duration":0.25795965799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":25751,"num_write_kb":345225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"68'4946","reported_seq":5033,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:55.632611+0000","last_change":"2023-10-25T00:23:55.632753+0000","last_active":"2023-10-25T00:23:55.632611+0000","last_peered":"2023-10-25T00:23:55.632611+0000","last_clean":"2023-10-25T00:23:55.632611+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-25T00:23:55.632611+0000","last_undegraded":"2023-10-25T00:23:55.632611+0000","last_fullsized":"2023-10-25T00:23:55.632611+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4946","last_scrub_stamp":"2023-10-25T00:23:55.632589+0000","last_deep_scrub":"68'4946","last_deep_scrub_stamp":"2023-10-25T00:23:55.632589+0000","last_clean_scrub_stamp":"2023-10-25T00:23:55.632589+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:34:04.162109+0000","scrub_duration":0.103749073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26583,"num_write_kb":315633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"72'4352","reported_seq":4437,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.629523+0000","last_change":"2023-10-25T00:23:34.629523+0000","last_active":"2023-10-25T00:23:34.629523+0000","last_peered":"2023-10-25T00:23:34.629523+0000","last_clean":"2023-10-25T00:23:34.629523+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-25T00:23:34.629523+0000","last_undegraded":"2023-10-25T00:23:34.629523+0000","last_fullsized":"2023-10-25T00:23:34.629523+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4352","last_scrub_stamp":"2023-10-25T00:23:34.629323+0000","last_deep_scrub":"72'4352","last_deep_scrub_stamp":"2023-10-25T00:23:34.629323+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.629323+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:38.589150+0000","scrub_duration":0.29560794600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23957,"num_write_kb":301052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"68'5026","reported_seq":5113,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:00.036402+0000","last_change":"2023-10-25T00:24:00.036586+0000","last_active":"2023-10-25T00:24:00.036402+0000","last_peered":"2023-10-25T00:24:00.036402+0000","last_clean":"2023-10-25T00:24:00.036402+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-25T00:24:00.036402+0000","last_undegraded":"2023-10-25T00:24:00.036402+0000","last_fullsized":"2023-10-25T00:24:00.036402+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5026","last_scrub_stamp":"2023-10-25T00:24:00.036334+0000","last_deep_scrub":"68'5026","last_deep_scrub_stamp":"2023-10-25T00:24:00.036334+0000","last_clean_scrub_stamp":"2023-10-25T00:24:00.036334+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:22.913142+0000","scrub_duration":0.209434338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26401,"num_write_kb":310586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"72'4393","reported_seq":4482,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.556819+0000","last_change":"2023-10-25T00:23:07.556990+0000","last_active":"2023-10-25T00:23:07.556819+0000","last_peered":"2023-10-25T00:23:07.556819+0000","last_clean":"2023-10-25T00:23:07.556819+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-25T00:23:07.556819+0000","last_undegraded":"2023-10-25T00:23:07.556819+0000","last_fullsized":"2023-10-25T00:23:07.556819+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4393","last_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_deep_scrub":"72'4393","last_deep_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.556784+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:21:00.928852+0000","scrub_duration":0.35410455800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":24752,"num_write_kb":296109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"68'5620","reported_seq":5710,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.194060+0000","last_change":"2023-10-25T00:23:12.194324+0000","last_active":"2023-10-25T00:23:12.194060+0000","last_peered":"2023-10-25T00:23:12.194060+0000","last_clean":"2023-10-25T00:23:12.194060+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-25T00:23:12.194060+0000","last_undegraded":"2023-10-25T00:23:12.194060+0000","last_fullsized":"2023-10-25T00:23:12.194060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5620","last_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_deep_scrub":"68'5620","last_deep_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.194008+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:33.868259+0000","scrub_duration":0.086061930999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29863,"num_write_kb":337169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"72'4340","reported_seq":4428,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:23.851965+0000","last_change":"2023-10-25T00:23:23.851965+0000","last_active":"2023-10-25T00:23:23.851965+0000","last_peered":"2023-10-25T00:23:23.851965+0000","last_clean":"2023-10-25T00:23:23.851965+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-25T00:23:23.851965+0000","last_undegraded":"2023-10-25T00:23:23.851965+0000","last_fullsized":"2023-10-25T00:23:23.851965+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4340","last_scrub_stamp":"2023-10-25T00:23:23.851809+0000","last_deep_scrub":"72'4340","last_deep_scrub_stamp":"2023-10-25T00:23:23.851809+0000","last_clean_scrub_stamp":"2023-10-25T00:23:23.851809+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:00.223998+0000","scrub_duration":0.37196321399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":36,"num_write":23367,"num_write_kb":302669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"68'4464","reported_seq":4549,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.904552+0000","last_change":"2023-10-25T00:22:57.904669+0000","last_active":"2023-10-25T00:22:57.904552+0000","last_peered":"2023-10-25T00:22:57.904552+0000","last_clean":"2023-10-25T00:22:57.904552+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-25T00:22:57.904552+0000","last_undegraded":"2023-10-25T00:22:57.904552+0000","last_fullsized":"2023-10-25T00:22:57.904552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4464","last_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_deep_scrub":"68'4464","last_deep_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.904532+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:17:44.026155+0000","scrub_duration":0.14132308199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24372,"num_write_kb":278074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"72'5589","reported_seq":5673,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.174468+0000","last_change":"2023-10-25T00:23:45.174602+0000","last_active":"2023-10-25T00:23:45.174468+0000","last_peered":"2023-10-25T00:23:45.174468+0000","last_clean":"2023-10-25T00:23:45.174468+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-25T00:23:45.174468+0000","last_undegraded":"2023-10-25T00:23:45.174468+0000","last_fullsized":"2023-10-25T00:23:45.174468+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5589","last_scrub_stamp":"2023-10-25T00:23:45.174448+0000","last_deep_scrub":"72'5589","last_deep_scrub_stamp":"2023-10-25T00:23:45.174448+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.174448+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:47:45.550696+0000","scrub_duration":0.18148444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":30206,"num_write_kb":399557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"68'4542","reported_seq":4628,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:09.344932+0000","last_change":"2023-10-25T00:23:09.345212+0000","last_active":"2023-10-25T00:23:09.344932+0000","last_peered":"2023-10-25T00:23:09.344932+0000","last_clean":"2023-10-25T00:23:09.344932+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-25T00:23:09.344932+0000","last_undegraded":"2023-10-25T00:23:09.344932+0000","last_fullsized":"2023-10-25T00:23:09.344932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4542","last_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_deep_scrub":"68'4542","last_deep_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.344860+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:16:09.325372+0000","scrub_duration":0.066583699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23889,"num_write_kb":292323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"72'3430","reported_seq":3515,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.797777+0000","last_change":"2023-10-25T00:23:21.798045+0000","last_active":"2023-10-25T00:23:21.797777+0000","last_peered":"2023-10-25T00:23:21.797777+0000","last_clean":"2023-10-25T00:23:21.797777+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-25T00:23:21.797777+0000","last_undegraded":"2023-10-25T00:23:21.797777+0000","last_fullsized":"2023-10-25T00:23:21.797777+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3430","last_scrub_stamp":"2023-10-25T00:23:21.797727+0000","last_deep_scrub":"72'3430","last_deep_scrub_stamp":"2023-10-25T00:23:21.797727+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.797727+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:15:30.844796+0000","scrub_duration":0.29875561899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19411,"num_write_kb":241862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"68'4719","reported_seq":4806,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:58.903089+0000","last_change":"2023-10-25T00:22:58.903089+0000","last_active":"2023-10-25T00:22:58.903089+0000","last_peered":"2023-10-25T00:22:58.903089+0000","last_clean":"2023-10-25T00:22:58.903089+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-25T00:22:58.903089+0000","last_undegraded":"2023-10-25T00:22:58.903089+0000","last_fullsized":"2023-10-25T00:22:58.903089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4719","last_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_deep_scrub":"68'4719","last_deep_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.902769+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:01.236787+0000","scrub_duration":0.136999537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25020,"num_write_kb":305197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"72'4169","reported_seq":4251,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.640667+0000","last_change":"2023-10-25T00:23:40.640863+0000","last_active":"2023-10-25T00:23:40.640667+0000","last_peered":"2023-10-25T00:23:40.640667+0000","last_clean":"2023-10-25T00:23:40.640667+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-25T00:23:40.640667+0000","last_undegraded":"2023-10-25T00:23:40.640667+0000","last_fullsized":"2023-10-25T00:23:40.640667+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4169","last_scrub_stamp":"2023-10-25T00:23:40.640645+0000","last_deep_scrub":"72'4169","last_deep_scrub_stamp":"2023-10-25T00:23:40.640645+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.640645+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:06.110573+0000","scrub_duration":0.28697117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23235,"num_write_kb":306175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"68'5952","reported_seq":6047,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.017693+0000","last_change":"2023-10-25T00:23:05.017693+0000","last_active":"2023-10-25T00:23:05.017693+0000","last_peered":"2023-10-25T00:23:05.017693+0000","last_clean":"2023-10-25T00:23:05.017693+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-25T00:23:05.017693+0000","last_undegraded":"2023-10-25T00:23:05.017693+0000","last_fullsized":"2023-10-25T00:23:05.017693+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5952","last_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_deep_scrub":"68'5952","last_deep_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.017340+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:31.140541+0000","scrub_duration":0.055624675999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32341,"num_write_kb":375441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"72'3717","reported_seq":3789,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.232259+0000","last_change":"2023-10-24T23:53:11.837335+0000","last_active":"2023-10-25T00:22:49.232259+0000","last_peered":"2023-10-25T00:22:49.232259+0000","last_clean":"2023-10-25T00:22:49.232259+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-25T00:22:49.232259+0000","last_undegraded":"2023-10-25T00:22:49.232259+0000","last_fullsized":"2023-10-25T00:22:49.232259+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21101,"num_write_kb":269108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"68'4825","reported_seq":4913,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.089847+0000","last_change":"2023-10-25T00:23:06.089847+0000","last_active":"2023-10-25T00:23:06.089847+0000","last_peered":"2023-10-25T00:23:06.089847+0000","last_clean":"2023-10-25T00:23:06.089847+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-25T00:23:06.089847+0000","last_undegraded":"2023-10-25T00:23:06.089847+0000","last_fullsized":"2023-10-25T00:23:06.089847+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4825","last_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_deep_scrub":"68'4825","last_deep_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.089705+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:04:51.412522+0000","scrub_duration":0.15071393499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25319,"num_write_kb":294113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"72'3714","reported_seq":3800,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:00.873483+0000","last_change":"2023-10-25T00:24:00.873759+0000","last_active":"2023-10-25T00:24:00.873483+0000","last_peered":"2023-10-25T00:24:00.873483+0000","last_clean":"2023-10-25T00:24:00.873483+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-25T00:24:00.873483+0000","last_undegraded":"2023-10-25T00:24:00.873483+0000","last_fullsized":"2023-10-25T00:24:00.873483+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3714","last_scrub_stamp":"2023-10-25T00:24:00.873431+0000","last_deep_scrub":"72'3714","last_deep_scrub_stamp":"2023-10-25T00:24:00.873431+0000","last_clean_scrub_stamp":"2023-10-25T00:24:00.873431+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:48.699043+0000","scrub_duration":0.31231597300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21148,"num_write_kb":265324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"68'4878","reported_seq":4967,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.227455+0000","last_change":"2023-10-25T00:22:49.227455+0000","last_active":"2023-10-25T00:22:49.227455+0000","last_peered":"2023-10-25T00:22:49.227455+0000","last_clean":"2023-10-25T00:22:49.227455+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-25T00:22:49.227455+0000","last_undegraded":"2023-10-25T00:22:49.227455+0000","last_fullsized":"2023-10-25T00:22:49.227455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4878","last_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_deep_scrub":"68'4878","last_deep_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_clean_scrub_stamp":"2023-10-25T00:22:49.227012+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:58:21.157778+0000","scrub_duration":0.13340163999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26154,"num_write_kb":305558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"72'3434","reported_seq":3518,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:57.337505+0000","last_change":"2023-10-25T00:23:57.337823+0000","last_active":"2023-10-25T00:23:57.337505+0000","last_peered":"2023-10-25T00:23:57.337505+0000","last_clean":"2023-10-25T00:23:57.337505+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-25T00:23:57.337505+0000","last_undegraded":"2023-10-25T00:23:57.337505+0000","last_fullsized":"2023-10-25T00:23:57.337505+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3434","last_scrub_stamp":"2023-10-25T00:23:57.337452+0000","last_deep_scrub":"72'3434","last_deep_scrub_stamp":"2023-10-25T00:23:57.337452+0000","last_clean_scrub_stamp":"2023-10-25T00:23:57.337452+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:11.559971+0000","scrub_duration":0.315725692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18449,"num_write_kb":262373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"68'4672","reported_seq":4761,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:03.653180+0000","last_change":"2023-10-25T00:24:03.653382+0000","last_active":"2023-10-25T00:24:03.653180+0000","last_peered":"2023-10-25T00:24:03.653180+0000","last_clean":"2023-10-25T00:24:03.653180+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-25T00:24:03.653180+0000","last_undegraded":"2023-10-25T00:24:03.653180+0000","last_fullsized":"2023-10-25T00:24:03.653180+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4672","last_scrub_stamp":"2023-10-25T00:24:03.653141+0000","last_deep_scrub":"68'4672","last_deep_scrub_stamp":"2023-10-25T00:24:03.653141+0000","last_clean_scrub_stamp":"2023-10-25T00:24:03.653141+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:19:24.224599+0000","scrub_duration":0.077062947000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25223,"num_write_kb":293792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"72'4013","reported_seq":4102,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:51.895004+0000","last_change":"2023-10-25T00:23:51.895151+0000","last_active":"2023-10-25T00:23:51.895004+0000","last_peered":"2023-10-25T00:23:51.895004+0000","last_clean":"2023-10-25T00:23:51.895004+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-25T00:23:51.895004+0000","last_undegraded":"2023-10-25T00:23:51.895004+0000","last_fullsized":"2023-10-25T00:23:51.895004+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4013","last_scrub_stamp":"2023-10-25T00:23:51.894968+0000","last_deep_scrub":"72'4013","last_deep_scrub_stamp":"2023-10-25T00:23:51.894968+0000","last_clean_scrub_stamp":"2023-10-25T00:23:51.894968+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:20:31.752935+0000","scrub_duration":0.32625951199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":22066,"num_write_kb":297298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"68'3918","reported_seq":4004,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:49.563660+0000","last_change":"2023-10-25T00:23:49.563900+0000","last_active":"2023-10-25T00:23:49.563660+0000","last_peered":"2023-10-25T00:23:49.563660+0000","last_clean":"2023-10-25T00:23:49.563660+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-25T00:23:49.563660+0000","last_undegraded":"2023-10-25T00:23:49.563660+0000","last_fullsized":"2023-10-25T00:23:49.563660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3918","last_scrub_stamp":"2023-10-25T00:23:49.563617+0000","last_deep_scrub":"68'3918","last_deep_scrub_stamp":"2023-10-25T00:23:49.563617+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.563617+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:50:13.550847+0000","scrub_duration":0.092172528000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21675,"num_write_kb":247000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"72'4401","reported_seq":4485,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:42.768643+0000","last_change":"2023-10-25T00:23:42.768767+0000","last_active":"2023-10-25T00:23:42.768643+0000","last_peered":"2023-10-25T00:23:42.768643+0000","last_clean":"2023-10-25T00:23:42.768643+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-25T00:23:42.768643+0000","last_undegraded":"2023-10-25T00:23:42.768643+0000","last_fullsized":"2023-10-25T00:23:42.768643+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4401","last_scrub_stamp":"2023-10-25T00:23:42.768621+0000","last_deep_scrub":"72'4401","last_deep_scrub_stamp":"2023-10-25T00:23:42.768621+0000","last_clean_scrub_stamp":"2023-10-25T00:23:42.768621+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:23:18.678304+0000","scrub_duration":0.26783985300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":24294,"num_write_kb":326749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"68'3970","reported_seq":4055,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:12.428569+0000","last_change":"2023-10-25T00:23:12.428867+0000","last_active":"2023-10-25T00:23:12.428569+0000","last_peered":"2023-10-25T00:23:12.428569+0000","last_clean":"2023-10-25T00:23:12.428569+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-25T00:23:12.428569+0000","last_undegraded":"2023-10-25T00:23:12.428569+0000","last_fullsized":"2023-10-25T00:23:12.428569+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3970","last_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_deep_scrub":"68'3970","last_deep_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.428494+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:07.899437+0000","scrub_duration":0.13106953599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21202,"num_write_kb":257085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"72'5559","reported_seq":5649,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:41.090126+0000","last_change":"2023-10-25T00:23:41.090253+0000","last_active":"2023-10-25T00:23:41.090126+0000","last_peered":"2023-10-25T00:23:41.090126+0000","last_clean":"2023-10-25T00:23:41.090126+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-25T00:23:41.090126+0000","last_undegraded":"2023-10-25T00:23:41.090126+0000","last_fullsized":"2023-10-25T00:23:41.090126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5559","last_scrub_stamp":"2023-10-25T00:23:41.090104+0000","last_deep_scrub":"72'5559","last_deep_scrub_stamp":"2023-10-25T00:23:41.090104+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.090104+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:40:20.680440+0000","scrub_duration":0.241945666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":31202,"num_write_kb":391572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"68'4936","reported_seq":5022,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.021130+0000","last_change":"2023-10-25T00:23:02.021272+0000","last_active":"2023-10-25T00:23:02.021130+0000","last_peered":"2023-10-25T00:23:02.021130+0000","last_clean":"2023-10-25T00:23:02.021130+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-25T00:23:02.021130+0000","last_undegraded":"2023-10-25T00:23:02.021130+0000","last_fullsized":"2023-10-25T00:23:02.021130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4936","last_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_deep_scrub":"68'4936","last_deep_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.021108+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:22.652380+0000","scrub_duration":0.17348896599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26134,"num_write_kb":309652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"72'3558","reported_seq":3645,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:47.072662+0000","last_change":"2023-10-25T00:23:47.072846+0000","last_active":"2023-10-25T00:23:47.072662+0000","last_peered":"2023-10-25T00:23:47.072662+0000","last_clean":"2023-10-25T00:23:47.072662+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-25T00:23:47.072662+0000","last_undegraded":"2023-10-25T00:23:47.072662+0000","last_fullsized":"2023-10-25T00:23:47.072662+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3558","last_scrub_stamp":"2023-10-25T00:23:47.072625+0000","last_deep_scrub":"72'3558","last_deep_scrub_stamp":"2023-10-25T00:23:47.072625+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.072625+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:59:13.347978+0000","scrub_duration":0.34108986800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":34,"num_write":19715,"num_write_kb":243025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"68'5282","reported_seq":5368,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:14.290128+0000","last_change":"2023-10-25T00:23:14.290293+0000","last_active":"2023-10-25T00:23:14.290128+0000","last_peered":"2023-10-25T00:23:14.290128+0000","last_clean":"2023-10-25T00:23:14.290128+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-25T00:23:14.290128+0000","last_undegraded":"2023-10-25T00:23:14.290128+0000","last_fullsized":"2023-10-25T00:23:14.290128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5282","last_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_deep_scrub":"68'5282","last_deep_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.290099+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:44:38.139530+0000","scrub_duration":0.10140234200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27848,"num_write_kb":324561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"72'3914","reported_seq":4002,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:46.833316+0000","last_change":"2023-10-25T00:23:46.833446+0000","last_active":"2023-10-25T00:23:46.833316+0000","last_peered":"2023-10-25T00:23:46.833316+0000","last_clean":"2023-10-25T00:23:46.833316+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-25T00:23:46.833316+0000","last_undegraded":"2023-10-25T00:23:46.833316+0000","last_fullsized":"2023-10-25T00:23:46.833316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3914","last_scrub_stamp":"2023-10-25T00:23:46.833294+0000","last_deep_scrub":"72'3914","last_deep_scrub_stamp":"2023-10-25T00:23:46.833294+0000","last_clean_scrub_stamp":"2023-10-25T00:23:46.833294+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:59.194933+0000","scrub_duration":0.30262013700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":20938,"num_write_kb":288796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"68'6616","reported_seq":6706,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.985123+0000","last_change":"2023-10-25T00:22:57.985278+0000","last_active":"2023-10-25T00:22:57.985123+0000","last_peered":"2023-10-25T00:22:57.985123+0000","last_clean":"2023-10-25T00:22:57.985123+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-25T00:22:57.985123+0000","last_undegraded":"2023-10-25T00:22:57.985123+0000","last_fullsized":"2023-10-25T00:22:57.985123+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6616","last_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_deep_scrub":"68'6616","last_deep_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.985096+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:14:23.491729+0000","scrub_duration":0.050039131000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36221,"num_write_kb":423916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"72'3536","reported_seq":3620,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:32.908497+0000","last_change":"2023-10-25T00:23:32.908624+0000","last_active":"2023-10-25T00:23:32.908497+0000","last_peered":"2023-10-25T00:23:32.908497+0000","last_clean":"2023-10-25T00:23:32.908497+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-25T00:23:32.908497+0000","last_undegraded":"2023-10-25T00:23:32.908497+0000","last_fullsized":"2023-10-25T00:23:32.908497+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3536","last_scrub_stamp":"2023-10-25T00:23:32.908476+0000","last_deep_scrub":"72'3536","last_deep_scrub_stamp":"2023-10-25T00:23:32.908476+0000","last_clean_scrub_stamp":"2023-10-25T00:23:32.908476+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:33:20.337271+0000","scrub_duration":0.30729883400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19276,"num_write_kb":253506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"68'4980","reported_seq":5064,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:10.321019+0000","last_change":"2023-10-25T00:23:10.321202+0000","last_active":"2023-10-25T00:23:10.321019+0000","last_peered":"2023-10-25T00:23:10.321019+0000","last_clean":"2023-10-25T00:23:10.321019+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-25T00:23:10.321019+0000","last_undegraded":"2023-10-25T00:23:10.321019+0000","last_fullsized":"2023-10-25T00:23:10.321019+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4980","last_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_deep_scrub":"68'4980","last_deep_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:10.320988+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:34.745429+0000","scrub_duration":0.13750111800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26525,"num_write_kb":306963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"72'4160","reported_seq":4246,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:24.541581+0000","last_change":"2023-10-25T00:23:24.541711+0000","last_active":"2023-10-25T00:23:24.541581+0000","last_peered":"2023-10-25T00:23:24.541581+0000","last_clean":"2023-10-25T00:23:24.541581+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-25T00:23:24.541581+0000","last_undegraded":"2023-10-25T00:23:24.541581+0000","last_fullsized":"2023-10-25T00:23:24.541581+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4160","last_scrub_stamp":"2023-10-25T00:23:24.541561+0000","last_deep_scrub":"72'4160","last_deep_scrub_stamp":"2023-10-25T00:23:24.541561+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.541561+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:52:13.060866+0000","scrub_duration":0.264226874,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":21819,"num_write_kb":288439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"68'4434","reported_seq":4520,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.208506+0000","last_change":"2023-10-25T00:22:47.255974+0000","last_active":"2023-10-25T00:22:49.208506+0000","last_peered":"2023-10-25T00:22:49.208506+0000","last_clean":"2023-10-25T00:22:49.208506+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-25T00:22:49.208506+0000","last_undegraded":"2023-10-25T00:22:49.208506+0000","last_fullsized":"2023-10-25T00:22:49.208506+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4434","last_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_deep_scrub":"68'4434","last_deep_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.255646+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:35:52.723567+0000","scrub_duration":0.14955275700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23791,"num_write_kb":292031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"72'4195","reported_seq":4276,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.836868+0000","last_change":"2023-10-25T00:23:37.837002+0000","last_active":"2023-10-25T00:23:37.836868+0000","last_peered":"2023-10-25T00:23:37.836868+0000","last_clean":"2023-10-25T00:23:37.836868+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-25T00:23:37.836868+0000","last_undegraded":"2023-10-25T00:23:37.836868+0000","last_fullsized":"2023-10-25T00:23:37.836868+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4195","last_scrub_stamp":"2023-10-25T00:23:37.836848+0000","last_deep_scrub":"72'4195","last_deep_scrub_stamp":"2023-10-25T00:23:37.836848+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.836848+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:49:10.177488+0000","scrub_duration":0.28694403499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23000,"num_write_kb":285019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"68'4243","reported_seq":4328,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.969067+0000","last_change":"2023-10-25T00:23:00.969067+0000","last_active":"2023-10-25T00:23:00.969067+0000","last_peered":"2023-10-25T00:23:00.969067+0000","last_clean":"2023-10-25T00:23:00.969067+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-25T00:23:00.969067+0000","last_undegraded":"2023-10-25T00:23:00.969067+0000","last_fullsized":"2023-10-25T00:23:00.969067+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4243","last_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_deep_scrub":"68'4243","last_deep_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.968692+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:45.877638+0000","scrub_duration":0.15459757700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22439,"num_write_kb":266094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"72'4017","reported_seq":4099,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:54.962933+0000","last_change":"2023-10-25T00:23:54.963064+0000","last_active":"2023-10-25T00:23:54.962933+0000","last_peered":"2023-10-25T00:23:54.962933+0000","last_clean":"2023-10-25T00:23:54.962933+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-25T00:23:54.962933+0000","last_undegraded":"2023-10-25T00:23:54.962933+0000","last_fullsized":"2023-10-25T00:23:54.962933+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4017","last_scrub_stamp":"2023-10-25T00:23:54.962911+0000","last_deep_scrub":"72'4017","last_deep_scrub_stamp":"2023-10-25T00:23:54.962911+0000","last_clean_scrub_stamp":"2023-10-25T00:23:54.962911+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:49:43.769503+0000","scrub_duration":0.29685359700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22665,"num_write_kb":291707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"68'4509","reported_seq":4597,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.227093+0000","last_change":"2023-10-25T00:22:44.022792+0000","last_active":"2023-10-25T00:22:49.227093+0000","last_peered":"2023-10-25T00:22:49.227093+0000","last_clean":"2023-10-25T00:22:49.227093+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-25T00:22:49.227093+0000","last_undegraded":"2023-10-25T00:22:49.227093+0000","last_fullsized":"2023-10-25T00:22:49.227093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4509","last_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_deep_scrub":"68'4509","last_deep_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.022573+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:54:30.651361+0000","scrub_duration":0.11145192700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23158,"num_write_kb":297883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"72'5235","reported_seq":5325,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:50.831818+0000","last_change":"2023-10-25T00:23:50.831958+0000","last_active":"2023-10-25T00:23:50.831818+0000","last_peered":"2023-10-25T00:23:50.831818+0000","last_clean":"2023-10-25T00:23:50.831818+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-25T00:23:50.831818+0000","last_undegraded":"2023-10-25T00:23:50.831818+0000","last_fullsized":"2023-10-25T00:23:50.831818+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5235","last_scrub_stamp":"2023-10-25T00:23:50.831798+0000","last_deep_scrub":"72'5235","last_deep_scrub_stamp":"2023-10-25T00:23:50.831798+0000","last_clean_scrub_stamp":"2023-10-25T00:23:50.831798+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:34:32.587906+0000","scrub_duration":0.30673944400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":15,"num_write":29235,"num_write_kb":367391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"68'4260","reported_seq":4349,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:56.486314+0000","last_change":"2023-10-25T00:22:56.486314+0000","last_active":"2023-10-25T00:22:56.486314+0000","last_peered":"2023-10-25T00:22:56.486314+0000","last_clean":"2023-10-25T00:22:56.486314+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-25T00:22:56.486314+0000","last_undegraded":"2023-10-25T00:22:56.486314+0000","last_fullsized":"2023-10-25T00:22:56.486314+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4260","last_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_deep_scrub":"68'4260","last_deep_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.485966+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:51.396462+0000","scrub_duration":0.116368769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23357,"num_write_kb":265837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"72'4656","reported_seq":4748,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.632181+0000","last_change":"2023-10-25T00:23:40.632314+0000","last_active":"2023-10-25T00:23:40.632181+0000","last_peered":"2023-10-25T00:23:40.632181+0000","last_clean":"2023-10-25T00:23:40.632181+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-25T00:23:40.632181+0000","last_undegraded":"2023-10-25T00:23:40.632181+0000","last_fullsized":"2023-10-25T00:23:40.632181+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4656","last_scrub_stamp":"2023-10-25T00:23:40.632160+0000","last_deep_scrub":"72'4656","last_deep_scrub_stamp":"2023-10-25T00:23:40.632160+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.632160+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:23:41.024594+0000","scrub_duration":0.095096767999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":26109,"num_write_kb":317781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"68'4636","reported_seq":4726,"reported_epoch":72,"state":"active+clean","last_fresh":"2023-10-25T00:22:39.672588+0000","last_change":"2023-10-25T00:22:39.672588+0000","last_active":"2023-10-25T00:22:39.672588+0000","last_peered":"2023-10-25T00:22:39.672588+0000","last_clean":"2023-10-25T00:22:39.672588+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-25T00:22:39.672588+0000","last_undegraded":"2023-10-25T00:22:39.672588+0000","last_fullsized":"2023-10-25T00:22:39.672588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4636","last_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_deep_scrub":"68'4636","last_deep_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.672438+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:47:15.433547+0000","scrub_duration":0.124153799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25124,"num_write_kb":285314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"72'3117","reported_seq":3204,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:41.679337+0000","last_change":"2023-10-25T00:23:41.679657+0000","last_active":"2023-10-25T00:23:41.679337+0000","last_peered":"2023-10-25T00:23:41.679337+0000","last_clean":"2023-10-25T00:23:41.679337+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-25T00:23:41.679337+0000","last_undegraded":"2023-10-25T00:23:41.679337+0000","last_fullsized":"2023-10-25T00:23:41.679337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3117","last_scrub_stamp":"2023-10-25T00:23:41.679268+0000","last_deep_scrub":"72'3117","last_deep_scrub_stamp":"2023-10-25T00:23:41.679268+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.679268+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:35.283156+0000","scrub_duration":0.30581881399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17240,"num_write_kb":232937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"68'5520","reported_seq":5617,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:56.366454+0000","last_change":"2023-10-25T00:22:56.366641+0000","last_active":"2023-10-25T00:22:56.366454+0000","last_peered":"2023-10-25T00:22:56.366454+0000","last_clean":"2023-10-25T00:22:56.366454+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-25T00:22:56.366454+0000","last_undegraded":"2023-10-25T00:22:56.366454+0000","last_fullsized":"2023-10-25T00:22:56.366454+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5520","last_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_deep_scrub":"68'5520","last_deep_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.366419+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:47:59.889167+0000","scrub_duration":0.104594514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29665,"num_write_kb":368823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"72'3962","reported_seq":4047,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.105781+0000","last_change":"2023-10-25T00:23:34.105899+0000","last_active":"2023-10-25T00:23:34.105781+0000","last_peered":"2023-10-25T00:23:34.105781+0000","last_clean":"2023-10-25T00:23:34.105781+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-25T00:23:34.105781+0000","last_undegraded":"2023-10-25T00:23:34.105781+0000","last_fullsized":"2023-10-25T00:23:34.105781+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3962","last_scrub_stamp":"2023-10-25T00:23:34.105758+0000","last_deep_scrub":"72'3962","last_deep_scrub_stamp":"2023-10-25T00:23:34.105758+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.105758+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:35:57.960582+0000","scrub_duration":0.32704070499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21989,"num_write_kb":280762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"68'4908","reported_seq":5000,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:00.407703+0000","last_change":"2023-10-25T00:23:00.408006+0000","last_active":"2023-10-25T00:23:00.407703+0000","last_peered":"2023-10-25T00:23:00.407703+0000","last_clean":"2023-10-25T00:23:00.407703+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-25T00:23:00.407703+0000","last_undegraded":"2023-10-25T00:23:00.407703+0000","last_fullsized":"2023-10-25T00:23:00.407703+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4908","last_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_deep_scrub":"68'4908","last_deep_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.407650+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:48:18.113931+0000","scrub_duration":0.051805808000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26829,"num_write_kb":320697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":320,"num_write":1482516,"num_write_kb":19060243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267718,"ondisk_log_size":267718,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":19490997,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1686,"num_read_kb":209824,"num_write":4665,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39043072,"data_stored":38981994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4561,"ondisk_log_size":4561,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":0,"num_write":1661441,"num_write_kb":19773626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307026,"ondisk_log_size":307026,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":20061533,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1689,"num_read_kb":267203,"num_write":4630,"num_write_kb":21638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40181760,"data_stored":40123066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4532,"ondisk_log_size":4532,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163,"ondisk_log_size":163,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575228,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919096,"kb_used_data":12336,"kb_used_omap":0,"kb_used_meta":906752,"kb_avail":93452744,"statfs":{"total":96636764160,"available":95695609856,"internally_reserved":0,"allocated":12632064,"data_stored":12153420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":928514048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.79600000000000004,"15min":0.94599999999999995},"min":{"1min":0.50600000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.232,"5min":3.6349999999999998,"15min":13.929},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.008,"5min":0.86599999999999999,"15min":0.94699999999999995},"min":{"1min":0.59299999999999997,"5min":0.27600000000000002,"15min":0.20999999999999999},"max":{"1min":1.3200000000000001,"5min":3.6659999999999999,"15min":11.315},"last":0.98599999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.80900000000000005,"15min":1.161},"min":{"1min":0.54900000000000004,"5min":0.23599999999999999,"15min":0.215},"max":{"1min":1.173,"5min":3.6949999999999998,"15min":28.308},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.0740000000000001,"5min":0.94399999999999995,"15min":1.24},"min":{"1min":0.78200000000000003,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":1.244,"5min":3.7490000000000001,"15min":28.346},"last":0.91200000000000003}]},{"osd":2,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":1.0089999999999999,"15min":0.99299999999999999},"min":{"1min":0.58099999999999996,"5min":0.28499999999999998,"15min":0.22800000000000001},"max":{"1min":1.448,"5min":5.71,"15min":11.430999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.93400000000000005,"15min":0.96099999999999997},"min":{"1min":0.58699999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.1990000000000001,"5min":3.8799999999999999,"15min":7.9710000000000001},"last":0.95599999999999996}]},{"osd":3,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.89200000000000002,"15min":1.3620000000000001},"min":{"1min":0.748,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":1.2749999999999999,"5min":3.6150000000000002,"15min":93.664000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.042,"15min":1.3879999999999999},"min":{"1min":0.72199999999999998,"5min":0.32700000000000001,"15min":0.25700000000000001},"max":{"1min":1.2729999999999999,"5min":4.6630000000000003,"15min":93.073999999999998},"last":1.1359999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.91600000000000004,"15min":0.88400000000000001},"min":{"1min":0.66000000000000003,"5min":0.314,"15min":0.249},"max":{"1min":1.274,"5min":3.0569999999999999,"15min":7.6280000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.05,"5min":1.0940000000000001,"15min":1.157},"min":{"1min":0.81000000000000005,"5min":0.36599999999999999,"15min":0.27200000000000002},"max":{"1min":1.3640000000000001,"5min":4.9640000000000004,"15min":27.936},"last":1.02}]},{"osd":5,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.3480000000000001,"15min":1.0449999999999999},"min":{"1min":0.59499999999999997,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":1.5,"5min":21.126999999999999,"15min":21.126999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.95399999999999996,"15min":0.96799999999999997},"min":{"1min":0.65200000000000002,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":1.3740000000000001,"5min":3.8069999999999999,"15min":11.49},"last":0.67300000000000004}]},{"osd":6,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.3280000000000001,"15min":1.1140000000000001},"min":{"1min":0.71099999999999997,"5min":0.311,"15min":0.23599999999999999},"max":{"1min":1.2989999999999999,"5min":4.8140000000000001,"15min":8.2959999999999994},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.4419999999999999,"15min":1.228},"min":{"1min":1.0189999999999999,"5min":0.45500000000000002,"15min":0.35499999999999998},"max":{"1min":1.4219999999999999,"5min":4.8639999999999999,"15min":20.651},"last":1.1919999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575228,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859020,"kb_used_data":5060,"kb_used_omap":0,"kb_used_meta":853952,"kb_avail":93512820,"statfs":{"total":96636764160,"available":95757127680,"internally_reserved":0,"allocated":5181440,"data_stored":4739934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":874446848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.91800000000000004,"15min":0.95399999999999996},"min":{"1min":0.39700000000000002,"5min":0.185,"15min":0.185},"max":{"1min":1.194,"5min":5.3490000000000002,"15min":15.974},"last":1.0720000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.036,"15min":0.97999999999999998},"min":{"1min":0.54400000000000004,"5min":0.218,"15min":0.218},"max":{"1min":1.115,"5min":8.2400000000000002,"15min":8.4789999999999992},"last":0.79100000000000004}]},{"osd":1,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.81899999999999995,"15min":0.88300000000000001},"min":{"1min":0.57699999999999996,"5min":0.221,"15min":0.221},"max":{"1min":1.0469999999999999,"5min":3.5630000000000002,"15min":7.6360000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.878,"15min":0.98899999999999999},"min":{"1min":0.54000000000000004,"5min":0.254,"15min":0.249},"max":{"1min":1.111,"5min":3.5840000000000001,"15min":21.600999999999999},"last":1.1120000000000001}]},{"osd":2,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.92700000000000005,"15min":1.0620000000000001},"min":{"1min":0.59999999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.198,"5min":3.6779999999999999,"15min":12.615},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.93400000000000005,"15min":1.1060000000000001},"min":{"1min":0.47599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.2370000000000001,"5min":3.702,"15min":12.667999999999999},"last":0.93700000000000006}]},{"osd":3,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.0009999999999999,"15min":1.248},"min":{"1min":0.68600000000000005,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.071,"5min":5.7350000000000003,"15min":38.649999999999999},"last":1.018},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.0580000000000001,"15min":1.325},"min":{"1min":0.67000000000000004,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.1699999999999999,"5min":5.766,"15min":38.683},"last":1.1459999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.3460000000000001,"15min":1.1240000000000001},"min":{"1min":0.56999999999999995,"5min":0.28399999999999997,"15min":0.23799999999999999},"max":{"1min":1.284,"5min":8.9930000000000003,"15min":12.682},"last":0.754},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.2609999999999999,"15min":1.081},"min":{"1min":0.63600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2190000000000001,"5min":5.694,"15min":10.686},"last":1.1759999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.0309999999999999,"15min":0.95699999999999996},"min":{"1min":0.622,"5min":0.28399999999999997,"15min":0.24299999999999999},"max":{"1min":1.1619999999999999,"5min":3.6379999999999999,"15min":8.0039999999999996},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.1759999999999999,"15min":1.0649999999999999},"min":{"1min":0.69799999999999995,"5min":0.34499999999999997,"15min":0.30299999999999999},"max":{"1min":1.244,"5min":4.0679999999999996,"15min":11.766999999999999},"last":1.226}]},{"osd":7,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.177,"15min":1.0249999999999999},"min":{"1min":0.64200000000000002,"5min":0.27000000000000002,"15min":0.251},"max":{"1min":1.159,"5min":8.2720000000000002,"15min":12.926},"last":1.208},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.234,"15min":1.0489999999999999},"min":{"1min":0.79400000000000004,"5min":0.34200000000000003,"15min":0.28799999999999998},"max":{"1min":1.2230000000000001,"5min":6.1100000000000003,"15min":7.6619999999999999},"last":1.2769999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575228,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":958872,"kb_used_data":9168,"kb_used_omap":0,"kb_used_meta":949696,"kb_avail":93412968,"statfs":{"total":96636764160,"available":95654879232,"internally_reserved":0,"allocated":9388032,"data_stored":8934420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":972488704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.133,"15min":1.038},"min":{"1min":0.505,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.97499999999999998,"5min":14.837999999999999,"15min":14.837999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.83899999999999997,"15min":0.97399999999999998},"min":{"1min":0.56799999999999995,"5min":0.20699999999999999,"15min":0.20100000000000001},"max":{"1min":1.427,"5min":4.016,"15min":7.423},"last":0.83599999999999997}]},{"osd":1,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":1.0489999999999999,"15min":1.0880000000000001},"min":{"1min":0.51600000000000001,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":1.1879999999999999,"5min":12.141,"15min":12.141},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.94399999999999995,"15min":1.0940000000000001},"min":{"1min":0.746,"5min":0.26400000000000001,"15min":0.252},"max":{"1min":0.999,"5min":9.0549999999999997,"15min":24.648},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.0840000000000001,"15min":1.2490000000000001},"min":{"1min":0.67200000000000004,"5min":0.25,"15min":0.25},"max":{"1min":1.258,"5min":12.175000000000001,"15min":20.760999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.117,"15min":1.073},"min":{"1min":0.68899999999999995,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2909999999999999,"5min":12.109999999999999,"15min":20.552},"last":0.77200000000000002}]},{"osd":3,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.3109999999999999,"15min":1.2130000000000001},"min":{"1min":0.67300000000000004,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":1.1599999999999999,"5min":12.013,"15min":12.013},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.036,"15min":1.244},"min":{"1min":0.65500000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0960000000000001,"5min":4.0490000000000004,"15min":12.805},"last":0.63500000000000001}]},{"osd":4,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.3520000000000001,"15min":1.2030000000000001},"min":{"1min":0.61199999999999999,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":1.278,"5min":12.201000000000001,"15min":12.728},"last":0.88},{"interface":"front","average":{"1min":0.997,"5min":1.22,"15min":1.137},"min":{"1min":0.64600000000000002,"5min":0.29299999999999998,"15min":0.27200000000000002},"max":{"1min":1.3100000000000001,"5min":7.6470000000000002,"15min":10.943},"last":0.81100000000000005}]},{"osd":6,"last update":"Wed Oct 25 00:24:04 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.1919999999999999,"15min":1.0609999999999999},"min":{"1min":0.78600000000000003,"5min":0.34399999999999997,"15min":0.26400000000000001},"max":{"1min":1.4079999999999999,"5min":9.6489999999999991,"15min":17.936},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.3680000000000001,"15min":1.216},"min":{"1min":0.71399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.387,"5min":13.029999999999999,"15min":14.766999999999999},"last":0.50800000000000001}]},{"osd":7,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.617,"15min":1.246},"min":{"1min":0.81699999999999995,"5min":0.33700000000000002,"15min":0.29499999999999998},"max":{"1min":1.6339999999999999,"5min":28.178999999999998,"15min":28.178999999999998},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.163,"5min":1.72,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":1.391,"5min":28.222999999999999,"15min":28.222999999999999},"last":0.749}]}]},{"osd":4,"up_from":13,"seq":55834575227,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946308,"kb_used_data":19132,"kb_used_omap":0,"kb_used_meta":927168,"kb_avail":93425532,"statfs":{"total":96636764160,"available":95667744768,"internally_reserved":0,"allocated":19591168,"data_stored":19124450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":949420032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.88900000000000001,"15min":0.98299999999999998},"min":{"1min":0.503,"5min":0.19700000000000001,"15min":0.187},"max":{"1min":1.1020000000000001,"5min":11.629,"15min":20.922999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73599999999999999,"15min":0.97799999999999998},"min":{"1min":0.52400000000000002,"5min":0.247,"15min":0.217},"max":{"1min":1.0920000000000001,"5min":7.0830000000000002,"15min":20.532},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.879,"15min":0.98399999999999999},"min":{"1min":0.61399999999999999,"5min":0.22,"15min":0.22},"max":{"1min":1.1819999999999999,"5min":3.0169999999999999,"15min":20.914000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.92700000000000005,"15min":1.167},"min":{"1min":0.64600000000000002,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":1.216,"5min":6.1890000000000001,"15min":23.265999999999998},"last":1.167}]},{"osd":2,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.84799999999999998,"15min":1.117},"min":{"1min":0.63900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.181,"5min":7.1550000000000002,"15min":20.774000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.1899999999999999,"15min":1.2529999999999999},"min":{"1min":0.65400000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.179,"5min":7.9009999999999998,"15min":20.817},"last":1.0509999999999999}]},{"osd":3,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1779999999999999,"15min":1.169},"min":{"1min":0.64800000000000002,"5min":0.314,"15min":0.28799999999999998},"max":{"1min":1.1879999999999999,"5min":7.3620000000000001,"15min":24.613},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.92900000000000005,"15min":1.1499999999999999},"min":{"1min":0.69699999999999995,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":1.2,"5min":7.2009999999999996,"15min":24.071000000000002},"last":1.1120000000000001}]},{"osd":5,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.153,"15min":0.93000000000000005},"min":{"1min":0.623,"5min":0.28599999999999998,"15min":0.23999999999999999},"max":{"1min":1.2170000000000001,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.0189999999999999,"15min":0.94099999999999995},"min":{"1min":0.65000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0669999999999999,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.52700000000000002}]},{"osd":6,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.1839999999999999,"15min":1.0800000000000001},"min":{"1min":0.73599999999999999,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":1.175,"5min":4.6550000000000002,"15min":6.5220000000000002},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0329999999999999,"15min":0.97199999999999998},"min":{"1min":0.81599999999999995,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":1.234,"5min":7.2389999999999999,"15min":7.2389999999999999},"last":0.94099999999999995}]},{"osd":7,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.3009999999999999,"15min":1.0700000000000001},"min":{"1min":0.72599999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.1990000000000001,"5min":7.4020000000000001,"15min":7.9850000000000003},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.4910000000000001,"15min":1.163},"min":{"1min":0.98799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.2909999999999999,"5min":7.444,"15min":7.444},"last":1.1459999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575228,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160748,"kb_used_data":9252,"kb_used_omap":0,"kb_used_meta":1151488,"kb_avail":93211092,"statfs":{"total":96636764160,"available":95448158208,"internally_reserved":0,"allocated":9474048,"data_stored":9018997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1179123712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.46400000000000002,"15min":0.78800000000000003},"min":{"1min":0.54000000000000004,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.82599999999999996,"5min":1.3899999999999999,"15min":13.327999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.58899999999999997,"15min":0.73699999999999999},"min":{"1min":0.48199999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.2170000000000001,"5min":1.444,"15min":5.4580000000000002},"last":0.78600000000000003}]},{"osd":1,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.67700000000000005,"15min":0.86799999999999999},"min":{"1min":0.41499999999999998,"5min":0.20200000000000001,"15min":0.19900000000000001},"max":{"1min":0.93400000000000005,"5min":4.4459999999999997,"15min":13.831},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.72299999999999998,"15min":0.76700000000000002},"min":{"1min":0.441,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.92200000000000004,"5min":4.2130000000000001,"15min":4.4500000000000002},"last":0.69299999999999995}]},{"osd":2,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.66300000000000003,"15min":0.875},"min":{"1min":0.504,"5min":0.223,"15min":0.221},"max":{"1min":0.97699999999999998,"5min":2.7890000000000001,"15min":8.2520000000000007},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.71699999999999997,"15min":0.91000000000000003},"min":{"1min":0.35899999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0720000000000001,"5min":2.7360000000000002,"15min":6.6840000000000002},"last":0.752}]},{"osd":4,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90700000000000003,"15min":1.206},"min":{"1min":0.56200000000000006,"5min":0.316,"15min":0.26800000000000002},"max":{"1min":1.1419999999999999,"5min":3.9449999999999998,"15min":39.070999999999998},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.2210000000000001,"15min":1.2589999999999999},"min":{"1min":0.65100000000000002,"5min":0.28599999999999998,"15min":0.25700000000000001},"max":{"1min":1.175,"5min":7.6180000000000003,"15min":55.104999999999997},"last":0.878}]},{"osd":5,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":0.94999999999999996,"15min":1.1299999999999999},"min":{"1min":0.68100000000000005,"5min":0.312,"15min":0.28399999999999997},"max":{"1min":1.597,"5min":4.1849999999999996,"15min":31.481999999999999},"last":1.002},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.028,"15min":1.2210000000000001},"min":{"1min":0.81999999999999995,"5min":0.38300000000000001,"15min":0.32500000000000001},"max":{"1min":1.5580000000000001,"5min":4.2060000000000004,"15min":31.512},"last":0.95699999999999996}]},{"osd":6,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.3140000000000001,"15min":1.252},"min":{"1min":0.72199999999999998,"5min":0.311,"15min":0.27800000000000002},"max":{"1min":1.5109999999999999,"5min":15.824,"15min":25.809999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.117,"15min":1.1830000000000001},"min":{"1min":0.873,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.518,"5min":4.7779999999999996,"15min":24.036999999999999},"last":0.97999999999999998}]},{"osd":7,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.0089999999999999,"15min":1.2330000000000001},"min":{"1min":0.83599999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.5840000000000001,"5min":4.8529999999999998,"15min":27.581},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.077,"15min":1.252},"min":{"1min":0.76500000000000001,"5min":0.374,"15min":0.374},"max":{"1min":1.5600000000000001,"5min":8.5259999999999998,"15min":24.332000000000001},"last":1.0800000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575228,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946060,"kb_used_data":5060,"kb_used_omap":0,"kb_used_meta":940992,"kb_avail":93425780,"statfs":{"total":96636764160,"available":95667998720,"internally_reserved":0,"allocated":5181440,"data_stored":4739422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":963575808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61399999999999999,"15min":0.98999999999999999},"min":{"1min":0.41199999999999998,"5min":0.16,"15min":0.16},"max":{"1min":1.204,"5min":5.29,"15min":29.344000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.68000000000000005,"15min":0.84599999999999997},"min":{"1min":0.51800000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.0589999999999999,"5min":6.4290000000000003,"15min":10.755000000000001},"last":0.69299999999999995}]},{"osd":1,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.66600000000000004,"15min":0.93799999999999994},"min":{"1min":0.49299999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":1.032,"5min":3.0409999999999999,"15min":18.617999999999999},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.69799999999999995,"15min":0.879},"min":{"1min":0.44700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.2909999999999999,"5min":3.649,"15min":15.064},"last":0.73699999999999999}]},{"osd":3,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.76100000000000001,"15min":0.90200000000000002},"min":{"1min":0.45100000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.93899999999999995,"5min":13.034000000000001,"15min":13.034000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.78600000000000003,"15min":1.006},"min":{"1min":0.59199999999999997,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3,"5min":3.1339999999999999,"15min":18.547999999999998},"last":0.95799999999999996}]},{"osd":4,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0700000000000001,"15min":1.1950000000000001},"min":{"1min":0.69499999999999995,"5min":0.252,"15min":0.252},"max":{"1min":1.377,"5min":7.6500000000000004,"15min":33.378999999999998},"last":1.137},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.135,"15min":1.2889999999999999},"min":{"1min":0.79000000000000004,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.464,"5min":7.6779999999999999,"15min":33.558999999999997},"last":1.181}]},{"osd":5,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.069,"15min":1.472},"min":{"1min":0.71699999999999997,"5min":0.32000000000000001,"15min":0.29099999999999998},"max":{"1min":1.3460000000000001,"5min":6.3979999999999997,"15min":34.545999999999999},"last":0.877},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.94099999999999995,"15min":1.177},"min":{"1min":0.64600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.343,"5min":4.4829999999999997,"15min":36.104999999999997},"last":1.0389999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:23:48 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.03,"15min":1.1140000000000001},"min":{"1min":0.81499999999999995,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.224,"5min":7.5030000000000001,"15min":33.661999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.105,"15min":1.198},"min":{"1min":0.97999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":1.417,"5min":7.5490000000000004,"15min":34.043999999999997},"last":0.78700000000000003}]},{"osd":7,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.1319999999999999,"15min":1.234},"min":{"1min":0.80400000000000005,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4239999999999999,"5min":7.5899999999999999,"15min":33.307000000000002},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2,"15min":1.306},"min":{"1min":0.752,"5min":0.371,"15min":0.371},"max":{"1min":1.6579999999999999,"5min":7.6189999999999998,"15min":33.344999999999999},"last":1.085}]}]},{"osd":1,"up_from":13,"seq":55834575229,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054144,"kb_used_data":4088,"kb_used_omap":0,"kb_used_meta":1050048,"kb_avail":93317696,"statfs":{"total":96636764160,"available":95557320704,"internally_reserved":0,"allocated":4186112,"data_stored":3743339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1075249152},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.59099999999999997,"15min":0.82699999999999996},"min":{"1min":0.40799999999999997,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.0129999999999999,"5min":3.6110000000000002,"15min":21.334},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.71599999999999997,"15min":0.85899999999999999},"min":{"1min":0.26200000000000001,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":1.0549999999999999,"5min":3.4809999999999999,"15min":21.276},"last":0.85599999999999998}]},{"osd":2,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.67400000000000004,"15min":0.81000000000000005},"min":{"1min":0.48599999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.95199999999999996,"5min":8.1600000000000001,"15min":11.420999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.746,"5min":0.68400000000000005,"15min":0.83999999999999997},"min":{"1min":0.51600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.0009999999999999,"5min":3.1760000000000002,"15min":11.648},"last":0.71499999999999997}]},{"osd":3,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78600000000000003,"15min":0.92500000000000004},"min":{"1min":0.58699999999999997,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":1.1299999999999999,"5min":3.46,"15min":14.346},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.88800000000000001,"15min":1.026},"min":{"1min":0.71099999999999997,"5min":0.26300000000000001,"15min":0.248},"max":{"1min":1.2090000000000001,"5min":3.4969999999999999,"15min":14.379},"last":1.02}]},{"osd":4,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.038,"15min":1.0589999999999999},"min":{"1min":0.51000000000000001,"5min":0.28000000000000003,"15min":0.27200000000000002},"max":{"1min":1.093,"5min":5.2409999999999997,"15min":10.582000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.038,"5min":1.0800000000000001,"15min":1.0580000000000001},"min":{"1min":0.82899999999999996,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":1.246,"5min":4.6630000000000003,"15min":11.173},"last":1.109}]},{"osd":5,"last update":"Wed Oct 25 00:23:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.92400000000000004,"15min":1.0529999999999999},"min":{"1min":0.79500000000000004,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.228,"5min":5.0309999999999997,"15min":11.577999999999999},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":0.92600000000000005,"15min":0.97799999999999998},"min":{"1min":0.69699999999999995,"5min":0.29899999999999999,"15min":0.29399999999999998},"max":{"1min":1.3480000000000001,"5min":2.7469999999999999,"15min":10.018000000000001},"last":1.145}]},{"osd":6,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.999,"15min":1.028},"min":{"1min":0.71999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.1739999999999999,"5min":7.4880000000000004,"15min":11.845000000000001},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.095,"5min":1.0860000000000001,"15min":1.125},"min":{"1min":0.77100000000000002,"5min":0.38300000000000001,"15min":0.33800000000000002},"max":{"1min":1.262,"5min":7.5220000000000002,"15min":11.919},"last":1.071}]},{"osd":7,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.1839999999999999,"15min":1.1279999999999999},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":1.2749999999999999,"5min":7.9909999999999997,"15min":11.881},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.95799999999999996,"15min":1.1439999999999999},"min":{"1min":0.56899999999999995,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4610000000000001,"5min":4.2300000000000004,"15min":10.478},"last":1.2110000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575229,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":966376,"kb_used_data":22112,"kb_used_omap":0,"kb_used_meta":944256,"kb_avail":93405464,"statfs":{"total":96636764160,"available":95647195136,"internally_reserved":0,"allocated":22642688,"data_stored":22192198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":966918144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60799999999999998,"15min":1},"min":{"1min":0.40200000000000002,"5min":0.18099999999999999,"15min":0.16900000000000001},"max":{"1min":0.90800000000000003,"5min":2.544,"15min":30.065999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.63500000000000001,"15min":0.997},"min":{"1min":0.503,"5min":0.184,"15min":0.182},"max":{"1min":0.90700000000000003,"5min":3.0030000000000001,"15min":19.757999999999999},"last":0.68300000000000005}]},{"osd":2,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73099999999999998,"15min":0.79700000000000004},"min":{"1min":0.46800000000000003,"5min":0.22900000000000001,"15min":0.20399999999999999},"max":{"1min":0.996,"5min":5.9740000000000002,"15min":15.324999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.72299999999999998,"15min":0.96999999999999997},"min":{"1min":0.626,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.1040000000000001,"5min":4.2549999999999999,"15min":30.283999999999999},"last":0.65100000000000002}]},{"osd":3,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.84399999999999997,"15min":1.1040000000000001},"min":{"1min":0.59699999999999998,"5min":0.20999999999999999,"15min":0.191},"max":{"1min":1.1060000000000001,"5min":6.3570000000000002,"15min":22.966999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.76400000000000001,"15min":1.083},"min":{"1min":0.58899999999999997,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":1.2090000000000001,"5min":2.5619999999999998,"15min":19.460999999999999},"last":0.93500000000000005}]},{"osd":4,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.89100000000000001,"15min":1.1279999999999999},"min":{"1min":0.63200000000000001,"5min":0.33700000000000002,"15min":0.26600000000000001},"max":{"1min":1.2370000000000001,"5min":3.758,"15min":8.3670000000000009},"last":0.76600000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.016,"15min":1.2310000000000001},"min":{"1min":0.67800000000000005,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.286,"5min":5.3719999999999999,"15min":13.023999999999999},"last":0.91900000000000004}]},{"osd":5,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.96999999999999997,"15min":1.0229999999999999},"min":{"1min":0.751,"5min":0.30399999999999999,"15min":0.29099999999999998},"max":{"1min":1.04,"5min":2.9710000000000001,"15min":10.683999999999999},"last":1.018},{"interface":"front","average":{"1min":1.097,"5min":1.01,"15min":1},"min":{"1min":0.79100000000000004,"5min":0.34999999999999998,"15min":0.29899999999999999},"max":{"1min":1.5469999999999999,"5min":4.8860000000000001,"15min":15.436999999999999},"last":0.97399999999999998}]},{"osd":6,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.98899999999999999,"15min":1.1339999999999999},"min":{"1min":0.78200000000000003,"5min":0.33700000000000002,"15min":0.28100000000000003},"max":{"1min":1.7969999999999999,"5min":6.1559999999999997,"15min":8.7260000000000009},"last":0.999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.125,"15min":1.294},"min":{"1min":0.86599999999999999,"5min":0.38200000000000001,"15min":0.33600000000000002},"max":{"1min":1.6379999999999999,"5min":7.8970000000000002,"15min":15.375},"last":0.80800000000000005}]},{"osd":7,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.2509999999999999,"15min":1.159},"min":{"1min":0.80000000000000004,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":1.202,"5min":5.5090000000000003,"15min":10.81},"last":1.089},{"interface":"front","average":{"1min":1.155,"5min":1.038,"15min":1.27},"min":{"1min":0.91800000000000004,"5min":0.313,"15min":0.313},"max":{"1min":1.506,"5min":4.2629999999999999,"15min":11.417},"last":0.872}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12673024,"data_stored":12668140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3198976,"data_stored":3198201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11689984,"data_stored":11672183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4195192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":8388642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4279555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6918144,"data_stored":6907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6836224,"data_stored":6822812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-25T00:24:19.907 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:24:19.908 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:24:19.908 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-24T23:53:01.277969+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=1, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:24:19.909 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2023-10-24T23:53:10.374501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=10, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:24:19.909 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:24:19.910 INFO:tasks.ceph:pgid 4.22 last_scrub_stamp 2023-10-24T23:53:02.290575+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=2, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:24:19.911 INFO:tasks.ceph:pgid 6.2a last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:24:19.911 INFO:tasks.ceph:pgid 6.32 last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:24:19.912 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-25T00:24:20.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:20.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.681+0000 7f6cfc897640 1 Processor -- start 2023-10-25T00:24:20.686 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.681+0000 7f6cfc897640 1 -- start start 2023-10-25T00:24:20.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.681+0000 7f6cfc897640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf806df30 0x7f6cf806e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:20.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.681+0000 7f6cfc897640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf8155350 con 0x7f6cf8157e20 2023-10-25T00:24:20.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.681+0000 7f6cfc897640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf81557e0 con 0x7f6cf814e900 2023-10-25T00:24:20.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.681+0000 7f6cfc897640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf8154830 con 0x7f6cf806df30 2023-10-25T00:24:20.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.681+0000 7f6cf6d9d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f6cf8157e20 0x7f6cf8158290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:57022/0 (socket says 172.21.15.107:57022) 2023-10-25T00:24:20.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.681+0000 7f6cf6d9d640 1 -- 172.21.15.107:0/3452405345 learned_addr learned my addr 172.21.15.107:0/3452405345 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:24:20.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.685+0000 7f6cf5d9b640 1 --2- 172.21.15.107:0/3452405345 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf806df30 0x7f6cf806e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:20.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.685+0000 7f6cf559a640 1 -- 172.21.15.107:0/3452405345 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685408832 0 0) 0x7f6cf8155350 con 0x7f6cf8157e20 2023-10-25T00:24:20.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.685+0000 7f6cf559a640 1 -- 172.21.15.107:0/3452405345 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ce0003530 con 0x7f6cf8157e20 2023-10-25T00:24:20.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.685+0000 7f6cf559a640 1 -- 172.21.15.107:0/3452405345 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063811464 0 0) 0x7f6cf81557e0 con 0x7f6cf814e900 2023-10-25T00:24:20.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.685+0000 7f6cf5d9b640 1 -- 172.21.15.107:0/3452405345 >> v1:172.21.15.110:6789/0 conn(0x7f6cf814e900 legacy=0x7f6cf80c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:20.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.685+0000 7f6cf5d9b640 1 -- 172.21.15.107:0/3452405345 >> v1:172.21.15.107:6789/0 conn(0x7f6cf8157e20 legacy=0x7f6cf8158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:20.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.685+0000 7f6cf5d9b640 1 -- 172.21.15.107:0/3452405345 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf814ece0 con 0x7f6cf806df30 2023-10-25T00:24:20.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.685+0000 7f6cf5d9b640 1 --2- 172.21.15.107:0/3452405345 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf806df30 0x7f6cf806e3b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80097f0 tx=0x7f6ce801c1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff697b6a4ea90469 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:20.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.685+0000 7f6cf559a640 1 -- 172.21.15.107:0/3452405345 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0001a00 con 0x7f6cf806df30 2023-10-25T00:24:20.691 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.685+0000 7f6cf559a640 1 -- 172.21.15.107:0/3452405345 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce80a7050 con 0x7f6cf806df30 2023-10-25T00:24:20.692 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.685+0000 7f6cf559a640 1 -- 172.21.15.107:0/3452405345 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce80ab600 con 0x7f6cf806df30 2023-10-25T00:24:20.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 -- 172.21.15.107:0/3452405345 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf806df30 msgr2=0x7f6cf806e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:20.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 --2- 172.21.15.107:0/3452405345 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf806df30 0x7f6cf806e3b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80097f0 tx=0x7f6ce801c1e0 comp rx=0 tx=0).stop 2023-10-25T00:24:20.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 -- 172.21.15.107:0/3452405345 shutdown_connections 2023-10-25T00:24:20.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 --2- 172.21.15.107:0/3452405345 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf806df30 0x7f6cf806e3b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:20.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 -- 172.21.15.107:0/3452405345 >> 172.21.15.107:0/3452405345 conn(0x7f6cf80c39a0 msgr2=0x7f6cf80c3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:20.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 -- 172.21.15.107:0/3452405345 shutdown_connections 2023-10-25T00:24:20.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 -- 172.21.15.107:0/3452405345 wait complete. 2023-10-25T00:24:20.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 Processor -- start 2023-10-25T00:24:20.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 -- start start 2023-10-25T00:24:20.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf814e900 0x7f6cf81495e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:20.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf8157e20 0x7f6cf81efd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:20.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf81f0280 0x7f6cf81526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:20.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf814c620 con 0x7f6cf814e900 2023-10-25T00:24:20.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf81577f0 con 0x7f6cf81f0280 2023-10-25T00:24:20.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf814de80 con 0x7f6cf8157e20 2023-10-25T00:24:20.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf814e900 0x7f6cf81495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:20.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf814e900 0x7f6cf81495e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39950/0 (socket says 172.21.15.107:39950) 2023-10-25T00:24:20.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf659c640 1 -- 172.21.15.107:0/1551556977 learned_addr learned my addr 172.21.15.107:0/1551556977 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:20.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf6d9d640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf81f0280 0x7f6cf81526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:20.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf5d9b640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf8157e20 0x7f6cf81efd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:20.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf659c640 1 -- 172.21.15.107:0/1551556977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf8157e20 msgr2=0x7f6cf81efd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:20.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf659c640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf8157e20 0x7f6cf81efd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:20.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf659c640 1 -- 172.21.15.107:0/1551556977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf81f0280 msgr2=0x7f6cf81526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:20.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf659c640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf81f0280 0x7f6cf81526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:20.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf659c640 1 -- 172.21.15.107:0/1551556977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce8009420 con 0x7f6cf814e900 2023-10-25T00:24:20.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf5d9b640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf8157e20 0x7f6cf81efd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:24:20.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cf659c640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf814e900 0x7f6cf81495e0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c440 tx=0x7f6cec00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6ce6ffd640 1 -- 172.21.15.107:0/1551556977 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec005480 con 0x7f6cf814e900 2023-10-25T00:24:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6ce6ffd640 1 -- 172.21.15.107:0/1551556977 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cec003600 con 0x7f6cf814e900 2023-10-25T00:24:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 -- 172.21.15.107:0/1551556977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf814d2e0 con 0x7f6cf814e900 2023-10-25T00:24:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6ce6ffd640 1 -- 172.21.15.107:0/1551556977 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec0039b0 con 0x7f6cf814e900 2023-10-25T00:24:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.689+0000 7f6cfc897640 1 -- 172.21.15.107:0/1551556977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf814db10 con 0x7f6cf814e900 2023-10-25T00:24:20.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.693+0000 7f6cfc897640 1 -- 172.21.15.107:0/1551556977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf814e220 con 0x7f6cf814e900 2023-10-25T00:24:20.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.693+0000 7f6ce6ffd640 1 -- 172.21.15.107:0/1551556977 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f6cec003b10 con 0x7f6cf814e900 2023-10-25T00:24:20.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.693+0000 7f6ce6ffd640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6cd8065e90 0x7f6cd8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:20.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.693+0000 7f6ce6ffd640 1 -- 172.21.15.107:0/1551556977 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f6cec0844f0 con 0x7f6cf814e900 2023-10-25T00:24:20.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.693+0000 7f6cf5d9b640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6cd8065e90 0x7f6cd8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:20.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.693+0000 7f6cf5d9b640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6cd8065e90 0x7f6cd8068350 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80ad000 tx=0x7f6ce8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:20.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.697+0000 7f6ce6ffd640 1 -- 172.21.15.107:0/1551556977 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f6cec054370 con 0x7f6cf814e900 2023-10-25T00:24:20.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 -- 172.21.15.107:0/1551556977 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cf80746e0 con 0x7f6cf814e900 2023-10-25T00:24:20.896 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6ce6ffd640 1 -- 172.21.15.107:0/1551556977 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5769 (secure 0 0 0) 0x7f6cec053f00 con 0x7f6cf814e900 2023-10-25T00:24:20.897 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:20.897 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:20.897 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 235 2023-10-25T00:24:20.898 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 -- 172.21.15.107:0/1551556977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6cd8065e90 msgr2=0x7f6cd8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:20.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6cd8065e90 0x7f6cd8068350 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f6ce80ad000 tx=0x7f6ce8008040 comp rx=0 tx=0).stop 2023-10-25T00:24:20.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 -- 172.21.15.107:0/1551556977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf814e900 msgr2=0x7f6cf81495e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:20.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf814e900 0x7f6cf81495e0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c440 tx=0x7f6cec00c900 comp rx=0 tx=0).stop 2023-10-25T00:24:20.899 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 -- 172.21.15.107:0/1551556977 shutdown_connections 2023-10-25T00:24:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6cf81f0280 0x7f6cf81526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f6cd8065e90 0x7f6cd8068350 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6cf8157e20 0x7f6cf81efd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 --2- 172.21.15.107:0/1551556977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6cf814e900 0x7f6cf81495e0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 -- 172.21.15.107:0/1551556977 >> 172.21.15.107:0/1551556977 conn(0x7f6cf80c39a0 msgr2=0x7f6cf80c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:20.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 -- 172.21.15.107:0/1551556977 shutdown_connections 2023-10-25T00:24:20.901 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:20.893+0000 7f6cfc897640 1 -- 172.21.15.107:0/1551556977 wait complete. 2023-10-25T00:24:20.916 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:24:22.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:23.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50fd68f640 1 Processor -- start 2023-10-25T00:24:23.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50fd68f640 1 -- start start 2023-10-25T00:24:23.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50fd68f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f50f8152040 0x7f50f80c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:23.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50fd68f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50f80c8100 0x7f50f80abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:23.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50fd68f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f8155680 con 0x7f50f806a4b0 2023-10-25T00:24:23.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50fd68f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50f8154a70 con 0x7f50f80c8100 2023-10-25T00:24:23.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50fd68f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50f8154d50 con 0x7f50f8152040 2023-10-25T00:24:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50f67fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50f80c8100 0x7f50f80abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50f67fc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f50f80c8100 0x7f50f80abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59228/0 (socket says 172.21.15.107:59228) 2023-10-25T00:24:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50f77fe640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f50f806a4b0 0x7f50f80ac240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37932/0 (socket says 172.21.15.107:37932) 2023-10-25T00:24:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50f67fc640 1 -- 172.21.15.107:0/627185432 learned_addr learned my addr 172.21.15.107:0/627185432 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:23.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50f6ffd640 1 --2- 172.21.15.107:0/627185432 >> v2:172.21.15.110:3300/0 conn(0x7f50f8152040 0x7f50f80c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:23.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50f67fc640 1 -- 172.21.15.107:0/627185432 >> v2:172.21.15.110:3300/0 conn(0x7f50f8152040 msgr2=0x7f50f80c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:23.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50f67fc640 1 --2- 172.21.15.107:0/627185432 >> v2:172.21.15.110:3300/0 conn(0x7f50f8152040 0x7f50f80c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:23.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50f67fc640 1 -- 172.21.15.107:0/627185432 >> v1:172.21.15.107:6789/0 conn(0x7f50f806a4b0 legacy=0x7f50f80ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:23.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50f67fc640 1 -- 172.21.15.107:0/627185432 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f8155030 con 0x7f50f80c8100 2023-10-25T00:24:23.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.045+0000 7f50f6ffd640 1 --2- 172.21.15.107:0/627185432 >> v2:172.21.15.110:3300/0 conn(0x7f50f8152040 0x7f50f80c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:24:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f67fc640 1 --2- 172.21.15.107:0/627185432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f80c8100 0x7f50f80abd00 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f50f8067ba0 tx=0x7f50ec01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6f16ecd0dbbc94 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f5ffb640 1 -- 172.21.15.107:0/627185432 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ec0ad680 con 0x7f50f80c8100 2023-10-25T00:24:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f5ffb640 1 -- 172.21.15.107:0/627185432 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ec0ad7e0 con 0x7f50f80c8100 2023-10-25T00:24:23.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f5ffb640 1 -- 172.21.15.107:0/627185432 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ec0adb70 con 0x7f50f80c8100 2023-10-25T00:24:23.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 -- 172.21.15.107:0/627185432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f80c8100 msgr2=0x7f50f80abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:23.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 --2- 172.21.15.107:0/627185432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f80c8100 0x7f50f80abd00 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f50f8067ba0 tx=0x7f50ec01c470 comp rx=0 tx=0).stop 2023-10-25T00:24:23.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 -- 172.21.15.107:0/627185432 shutdown_connections 2023-10-25T00:24:23.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 --2- 172.21.15.107:0/627185432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f80c8100 0x7f50f80abd00 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:23.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 --2- 172.21.15.107:0/627185432 >> v2:172.21.15.110:3300/0 conn(0x7f50f8152040 0x7f50f80c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:23.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 -- 172.21.15.107:0/627185432 >> 172.21.15.107:0/627185432 conn(0x7f50f80ca200 msgr2=0x7f50f80ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:23.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 -- 172.21.15.107:0/627185432 shutdown_connections 2023-10-25T00:24:23.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 -- 172.21.15.107:0/627185432 wait complete. 2023-10-25T00:24:23.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 Processor -- start 2023-10-25T00:24:23.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 -- start start 2023-10-25T00:24:23.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f806a4b0 0x7f50f814aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:23.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50f80c8100 0x7f50f80a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:23.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50f8152040 0x7f50f80aa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:23.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80ab3a0 con 0x7f50f806a4b0 2023-10-25T00:24:23.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80aaa40 con 0x7f50f8152040 2023-10-25T00:24:23.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50fd68f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80c41b0 con 0x7f50f80c8100 2023-10-25T00:24:23.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f806a4b0 0x7f50f814aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:23.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f806a4b0 0x7f50f814aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59242/0 (socket says 172.21.15.107:59242) 2023-10-25T00:24:23.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f6ffd640 1 -- 172.21.15.107:0/1956745143 learned_addr learned my addr 172.21.15.107:0/1956745143 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:23.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f67fc640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50f80c8100 0x7f50f80a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:23.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f77fe640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50f8152040 0x7f50f80aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:23.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f6ffd640 1 -- 172.21.15.107:0/1956745143 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50f80c8100 msgr2=0x7f50f80a9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:23.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f6ffd640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50f80c8100 0x7f50f80a9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:23.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f6ffd640 1 -- 172.21.15.107:0/1956745143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50f8152040 msgr2=0x7f50f80aa500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:23.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f6ffd640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50f8152040 0x7f50f80aa500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:23.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f6ffd640 1 -- 172.21.15.107:0/1956745143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0094d0 con 0x7f50f806a4b0 2023-10-25T00:24:23.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.049+0000 7f50f67fc640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50f80c8100 0x7f50f80a9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:24:23.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50f6ffd640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f806a4b0 0x7f50f814aed0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f50e800c7f0 tx=0x7f50e800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:23.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50df7fe640 1 -- 172.21.15.107:0/1956745143 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8007820 con 0x7f50f806a4b0 2023-10-25T00:24:23.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50df7fe640 1 -- 172.21.15.107:0/1956745143 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e8007980 con 0x7f50f806a4b0 2023-10-25T00:24:23.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50fd68f640 1 -- 172.21.15.107:0/1956745143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f8149f20 con 0x7f50f806a4b0 2023-10-25T00:24:23.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50df7fe640 1 -- 172.21.15.107:0/1956745143 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e8007d30 con 0x7f50f806a4b0 2023-10-25T00:24:23.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50fd68f640 1 -- 172.21.15.107:0/1956745143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f814a4e0 con 0x7f50f806a4b0 2023-10-25T00:24:23.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50fd68f640 1 -- 172.21.15.107:0/1956745143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f80c4ac0 con 0x7f50f806a4b0 2023-10-25T00:24:23.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50df7fe640 1 -- 172.21.15.107:0/1956745143 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f50e80113f0 con 0x7f50f806a4b0 2023-10-25T00:24:23.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50df7fe640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50e4065e60 0x7f50e4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:23.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50df7fe640 1 -- 172.21.15.107:0/1956745143 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f50e8084c30 con 0x7f50f806a4b0 2023-10-25T00:24:23.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50f67fc640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50e4065e60 0x7f50e4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:23.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.053+0000 7f50f67fc640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50e4065e60 0x7f50e4068320 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0af000 tx=0x7f50ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:23.065 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.057+0000 7f50df7fe640 1 -- 172.21.15.107:0/1956745143 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f50e8050f90 con 0x7f50f806a4b0 2023-10-25T00:24:23.246 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.241+0000 7f50fd68f640 1 -- 172.21.15.107:0/1956745143 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50f80746e0 con 0x7f50f806a4b0 2023-10-25T00:24:23.247 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50df7fe640 1 -- 172.21.15.107:0/1956745143 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5769 (secure 0 0 0) 0x7f50e801f020 con 0x7f50f806a4b0 2023-10-25T00:24:23.248 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:23.248 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:23.248 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 235 2023-10-25T00:24:23.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 -- 172.21.15.107:0/1956745143 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50e4065e60 msgr2=0x7f50e4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:23.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50e4065e60 0x7f50e4068320 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0af000 tx=0x7f50ec008040 comp rx=0 tx=0).stop 2023-10-25T00:24:23.250 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 -- 172.21.15.107:0/1956745143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f806a4b0 msgr2=0x7f50f814aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:23.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f806a4b0 0x7f50f814aed0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f50e800c7f0 tx=0x7f50e800ccb0 comp rx=0 tx=0).stop 2023-10-25T00:24:23.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 -- 172.21.15.107:0/1956745143 shutdown_connections 2023-10-25T00:24:23.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50f8152040 0x7f50f80aa500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:23.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f50e4065e60 0x7f50e4068320 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:23.251 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50f80c8100 0x7f50f80a9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:23.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 --2- 172.21.15.107:0/1956745143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50f806a4b0 0x7f50f814aed0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:23.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 -- 172.21.15.107:0/1956745143 >> 172.21.15.107:0/1956745143 conn(0x7f50f80ca200 msgr2=0x7f50f81517d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:23.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 -- 172.21.15.107:0/1956745143 shutdown_connections 2023-10-25T00:24:23.252 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:23.245+0000 7f50fd68f640 1 -- 172.21.15.107:0/1956745143 wait complete. 2023-10-25T00:24:23.267 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:24:25.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70aabd5640 1 Processor -- start 2023-10-25T00:24:25.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70aabd5640 1 -- start start 2023-10-25T00:24:25.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70aabd5640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f70a4151960 0x7f70a4071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:25.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70aabd5640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a4150bc0 con 0x7f70a4071220 2023-10-25T00:24:25.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70aabd5640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a406c510 con 0x7f70a40c24c0 2023-10-25T00:24:25.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70aabd5640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a406c7f0 con 0x7f70a4151960 2023-10-25T00:24:25.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70a3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f70a4151960 0x7f70a4071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:25.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70a3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f70a4151960 0x7f70a4071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59252/0 (socket says 172.21.15.107:59252) 2023-10-25T00:24:25.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70a3fff640 1 -- 172.21.15.107:0/1612925788 learned_addr learned my addr 172.21.15.107:0/1612925788 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:25.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70a3fff640 1 -- 172.21.15.107:0/1612925788 >> v1:172.21.15.110:6789/0 conn(0x7f70a40c24c0 legacy=0x7f70a406a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:25.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70a3fff640 1 -- 172.21.15.107:0/1612925788 >> v1:172.21.15.107:6789/0 conn(0x7f70a4071220 legacy=0x7f70a406bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:25.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.397+0000 7f70a3fff640 1 -- 172.21.15.107:0/1612925788 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a4150e80 con 0x7f70a4151960 2023-10-25T00:24:25.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70a3fff640 1 --2- 172.21.15.107:0/1612925788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70a4151960 0x7f70a4071a90 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f709800aa10 tx=0x7f709801d840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b70f3261b140ef7 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:25.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70a37fe640 1 -- 172.21.15.107:0/1612925788 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70980a6760 con 0x7f70a4151960 2023-10-25T00:24:25.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70a37fe640 1 -- 172.21.15.107:0/1612925788 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70980a68c0 con 0x7f70a4151960 2023-10-25T00:24:25.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70a37fe640 1 -- 172.21.15.107:0/1612925788 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70980a6c50 con 0x7f70a4151960 2023-10-25T00:24:25.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70aabd5640 1 -- 172.21.15.107:0/1612925788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70a4151960 msgr2=0x7f70a4071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:25.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70aabd5640 1 --2- 172.21.15.107:0/1612925788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70a4151960 0x7f70a4071a90 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f709800aa10 tx=0x7f709801d840 comp rx=0 tx=0).stop 2023-10-25T00:24:25.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70aabd5640 1 -- 172.21.15.107:0/1612925788 shutdown_connections 2023-10-25T00:24:25.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70aabd5640 1 --2- 172.21.15.107:0/1612925788 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70a4151960 0x7f70a4071a90 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:25.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70aabd5640 1 -- 172.21.15.107:0/1612925788 >> 172.21.15.107:0/1612925788 conn(0x7f70a40c2950 msgr2=0x7f70a40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:25.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70aabd5640 1 -- 172.21.15.107:0/1612925788 shutdown_connections 2023-10-25T00:24:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70aabd5640 1 -- 172.21.15.107:0/1612925788 wait complete. 2023-10-25T00:24:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70aabd5640 1 Processor -- start 2023-10-25T00:24:25.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.401+0000 7f70aabd5640 1 -- start start 2023-10-25T00:24:25.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70aabd5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4071220 0x7f70a40c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70aabd5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70a40c24c0 0x7f70a414de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70aabd5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70a4151960 0x7f70a4145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70aabd5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a41eed90 con 0x7f70a4151960 2023-10-25T00:24:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70aabd5640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a41ee1c0 con 0x7f70a4071220 2023-10-25T00:24:25.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70aabd5640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a40c5ae0 con 0x7f70a40c24c0 2023-10-25T00:24:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a9172640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70a4151960 0x7f70a4145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a9172640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70a4151960 0x7f70a4145cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59268/0 (socket says 172.21.15.107:59268) 2023-10-25T00:24:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a9172640 1 -- 172.21.15.107:0/873508983 learned_addr learned my addr 172.21.15.107:0/873508983 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a8971640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4071220 0x7f70a40c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:25.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a3fff640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70a40c24c0 0x7f70a414de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a9172640 1 -- 172.21.15.107:0/873508983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70a40c24c0 msgr2=0x7f70a414de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a9172640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70a40c24c0 0x7f70a414de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a9172640 1 -- 172.21.15.107:0/873508983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4071220 msgr2=0x7f70a40c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a9172640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4071220 0x7f70a40c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a9172640 1 -- 172.21.15.107:0/873508983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709800a670 con 0x7f70a4151960 2023-10-25T00:24:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a8971640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4071220 0x7f70a40c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:24:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a9172640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70a4151960 0x7f70a4145cd0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c780 tx=0x7f709c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a17fa640 1 -- 172.21.15.107:0/873508983 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c004b70 con 0x7f70a4151960 2023-10-25T00:24:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a17fa640 1 -- 172.21.15.107:0/873508983 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c00ce90 con 0x7f70a4151960 2023-10-25T00:24:25.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70aabd5640 1 -- 172.21.15.107:0/873508983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a41ee460 con 0x7f70a4151960 2023-10-25T00:24:25.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a17fa640 1 -- 172.21.15.107:0/873508983 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c00a850 con 0x7f70a4151960 2023-10-25T00:24:25.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70aabd5640 1 -- 172.21.15.107:0/873508983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a41eea20 con 0x7f70a4151960 2023-10-25T00:24:25.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70aabd5640 1 -- 172.21.15.107:0/873508983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a41eef00 con 0x7f70a4151960 2023-10-25T00:24:25.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a17fa640 1 -- 172.21.15.107:0/873508983 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f709c00a9b0 con 0x7f70a4151960 2023-10-25T00:24:25.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a17fa640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7090065e60 0x7f7090068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:25.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a17fa640 1 -- 172.21.15.107:0/873508983 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f709c084cc0 con 0x7f70a4151960 2023-10-25T00:24:25.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a8971640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7090065e60 0x7f7090068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:25.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.405+0000 7f70a8971640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7090065e60 0x7f7090068320 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f70a406b080 tx=0x7f7094009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:25.419 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.409+0000 7f70a17fa640 1 -- 172.21.15.107:0/873508983 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f709c08b080 con 0x7f70a4151960 2023-10-25T00:24:25.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.597+0000 7f70aabd5640 1 -- 172.21.15.107:0/873508983 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70a40746e0 con 0x7f70a4151960 2023-10-25T00:24:25.603 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70a17fa640 1 -- 172.21.15.107:0/873508983 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5769 (secure 0 0 0) 0x7f709c051050 con 0x7f70a4151960 2023-10-25T00:24:25.604 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:25.604 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:25.604 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 235 2023-10-25T00:24:25.605 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 -- 172.21.15.107:0/873508983 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7090065e60 msgr2=0x7f7090068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:25.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7090065e60 0x7f7090068320 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f70a406b080 tx=0x7f7094009040 comp rx=0 tx=0).stop 2023-10-25T00:24:25.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 -- 172.21.15.107:0/873508983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70a4151960 msgr2=0x7f70a4145cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:25.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70a4151960 0x7f70a4145cd0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c780 tx=0x7f709c00cc40 comp rx=0 tx=0).stop 2023-10-25T00:24:25.606 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 -- 172.21.15.107:0/873508983 shutdown_connections 2023-10-25T00:24:25.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70a4151960 0x7f70a4145cd0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:25.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7090065e60 0x7f7090068320 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:25.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f70a40c24c0 0x7f70a414de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:25.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 --2- 172.21.15.107:0/873508983 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f70a4071220 0x7f70a40c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:25.607 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 -- 172.21.15.107:0/873508983 >> 172.21.15.107:0/873508983 conn(0x7f70a40c2950 msgr2=0x7f70a406bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:25.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 -- 172.21.15.107:0/873508983 shutdown_connections 2023-10-25T00:24:25.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:25.601+0000 7f70aabd5640 1 -- 172.21.15.107:0/873508983 wait complete. 2023-10-25T00:24:25.624 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:24:27.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:27.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.753+0000 7f4527b77640 1 Processor -- start 2023-10-25T00:24:27.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.753+0000 7f4527b77640 1 -- start start 2023-10-25T00:24:27.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.753+0000 7f4527b77640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4520152040 0x7f45200c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:27.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.753+0000 7f4527b77640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f45200c8100 0x7f45200abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:27.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.753+0000 7f4527b77640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f452006a4b0 0x7f45200ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:27.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.753+0000 7f4527b77640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4520155680 con 0x7f452006a4b0 2023-10-25T00:24:27.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.753+0000 7f4527b77640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4520154a70 con 0x7f45200c8100 2023-10-25T00:24:27.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.753+0000 7f4527b77640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4520154d50 con 0x7f4520152040 2023-10-25T00:24:27.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4526114640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f452006a4b0 0x7f45200ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:27.761 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4526114640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f452006a4b0 0x7f45200ac240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59270/0 (socket says 172.21.15.107:59270) 2023-10-25T00:24:27.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4526114640 1 -- 172.21.15.107:0/1481671369 learned_addr learned my addr 172.21.15.107:0/1481671369 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:27.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4525913640 1 --2- 172.21.15.107:0/1481671369 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4520152040 0x7f45200c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:27.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4525112640 1 --2- 172.21.15.107:0/1481671369 >> v2:172.21.15.110:3300/0 conn(0x7f45200c8100 0x7f45200abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:27.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4526114640 1 -- 172.21.15.107:0/1481671369 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4520152040 msgr2=0x7f45200c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:27.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4526114640 1 --2- 172.21.15.107:0/1481671369 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4520152040 0x7f45200c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:27.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4526114640 1 -- 172.21.15.107:0/1481671369 >> v2:172.21.15.110:3300/0 conn(0x7f45200c8100 msgr2=0x7f45200abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:27.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4526114640 1 --2- 172.21.15.107:0/1481671369 >> v2:172.21.15.110:3300/0 conn(0x7f45200c8100 0x7f45200abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:27.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4526114640 1 -- 172.21.15.107:0/1481671369 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4520155030 con 0x7f452006a4b0 2023-10-25T00:24:27.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4526114640 1 --2- 172.21.15.107:0/1481671369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f452006a4b0 0x7f45200ac240 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f4520069840 tx=0x7f451801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56753d307c34e0cd server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:27.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4524911640 1 -- 172.21.15.107:0/1481671369 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45180a56d0 con 0x7f452006a4b0 2023-10-25T00:24:27.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4524911640 1 -- 172.21.15.107:0/1481671369 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45180a5830 con 0x7f452006a4b0 2023-10-25T00:24:27.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4524911640 1 -- 172.21.15.107:0/1481671369 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45180a5bc0 con 0x7f452006a4b0 2023-10-25T00:24:27.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4527b77640 1 -- 172.21.15.107:0/1481671369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f452006a4b0 msgr2=0x7f45200ac240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:27.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4527b77640 1 --2- 172.21.15.107:0/1481671369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f452006a4b0 0x7f45200ac240 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f4520069840 tx=0x7f451801c940 comp rx=0 tx=0).stop 2023-10-25T00:24:27.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4527b77640 1 -- 172.21.15.107:0/1481671369 shutdown_connections 2023-10-25T00:24:27.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4527b77640 1 --2- 172.21.15.107:0/1481671369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f452006a4b0 0x7f45200ac240 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:27.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4527b77640 1 --2- 172.21.15.107:0/1481671369 >> v2:172.21.15.110:3300/0 conn(0x7f45200c8100 0x7f45200abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:27.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4527b77640 1 --2- 172.21.15.107:0/1481671369 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4520152040 0x7f45200c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:27.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4527b77640 1 -- 172.21.15.107:0/1481671369 >> 172.21.15.107:0/1481671369 conn(0x7f45200ca200 msgr2=0x7f45200ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:27.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4527b77640 1 -- 172.21.15.107:0/1481671369 shutdown_connections 2023-10-25T00:24:27.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.757+0000 7f4527b77640 1 -- 172.21.15.107:0/1481671369 wait complete. 2023-10-25T00:24:27.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4527b77640 1 Processor -- start 2023-10-25T00:24:27.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4527b77640 1 -- start start 2023-10-25T00:24:27.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4527b77640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f452006a4b0 0x7f45201edef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:27.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4527b77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45200c8100 0x7f45200c3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:27.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4527b77640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4520152040 0x7f45200c2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:27.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4527b77640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45201efde0 con 0x7f45200c8100 2023-10-25T00:24:27.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4527b77640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45201ee430 con 0x7f4520152040 2023-10-25T00:24:27.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4527b77640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45200c3a80 con 0x7f452006a4b0 2023-10-25T00:24:27.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4525112640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45200c8100 0x7f45200c3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:27.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4525112640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45200c8100 0x7f45200c3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59278/0 (socket says 172.21.15.107:59278) 2023-10-25T00:24:27.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4525112640 1 -- 172.21.15.107:0/1297826904 learned_addr learned my addr 172.21.15.107:0/1297826904 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:27.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4525913640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f452006a4b0 0x7f45201edef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:27.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4526114640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4520152040 0x7f45200c2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:27.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4525112640 1 -- 172.21.15.107:0/1297826904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f452006a4b0 msgr2=0x7f45201edef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:27.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4525112640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f452006a4b0 0x7f45201edef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:27.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4525112640 1 -- 172.21.15.107:0/1297826904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4520152040 msgr2=0x7f45200c2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:27.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4525112640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4520152040 0x7f45200c2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:27.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4525112640 1 -- 172.21.15.107:0/1297826904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4518009b30 con 0x7f45200c8100 2023-10-25T00:24:27.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4525112640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45200c8100 0x7f45200c3540 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f451000d7d0 tx=0x7f451000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:27.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f45167fc640 1 -- 172.21.15.107:0/1297826904 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510004020 con 0x7f45200c8100 2023-10-25T00:24:27.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f45167fc640 1 -- 172.21.15.107:0/1297826904 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4510004180 con 0x7f45200c8100 2023-10-25T00:24:27.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4527b77640 1 -- 172.21.15.107:0/1297826904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45200bdef0 con 0x7f45200c8100 2023-10-25T00:24:27.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f45167fc640 1 -- 172.21.15.107:0/1297826904 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4510004330 con 0x7f45200c8100 2023-10-25T00:24:27.773 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4527b77640 1 -- 172.21.15.107:0/1297826904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45200be4b0 con 0x7f45200c8100 2023-10-25T00:24:27.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f4527b77640 1 -- 172.21.15.107:0/1297826904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45200aeaa0 con 0x7f45200c8100 2023-10-25T00:24:27.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.761+0000 7f45167fc640 1 -- 172.21.15.107:0/1297826904 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f45100096a0 con 0x7f45200c8100 2023-10-25T00:24:27.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.765+0000 7f45167fc640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4504065e60 0x7f4504068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:27.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.765+0000 7f45167fc640 1 -- 172.21.15.107:0/1297826904 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f4510084dc0 con 0x7f45200c8100 2023-10-25T00:24:27.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.765+0000 7f4525913640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4504065e60 0x7f4504068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:27.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.765+0000 7f4525913640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4504065e60 0x7f4504068320 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f451c00b9d0 tx=0x7f451c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:27.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.769+0000 7f45167fc640 1 -- 172.21.15.107:0/1297826904 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f45100511b0 con 0x7f45200c8100 2023-10-25T00:24:27.957 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.953+0000 7f4527b77640 1 -- 172.21.15.107:0/1297826904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45200746e0 con 0x7f45200c8100 2023-10-25T00:24:27.958 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.953+0000 7f45167fc640 1 -- 172.21.15.107:0/1297826904 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5769 (secure 0 0 0) 0x7f4510018090 con 0x7f45200c8100 2023-10-25T00:24:27.958 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:27.959 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:27.959 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 235 2023-10-25T00:24:27.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 -- 172.21.15.107:0/1297826904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4504065e60 msgr2=0x7f4504068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:27.960 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4504065e60 0x7f4504068320 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f451c00b9d0 tx=0x7f451c008040 comp rx=0 tx=0).stop 2023-10-25T00:24:27.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 -- 172.21.15.107:0/1297826904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45200c8100 msgr2=0x7f45200c3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:27.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45200c8100 0x7f45200c3540 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f451000d7d0 tx=0x7f451000dc90 comp rx=0 tx=0).stop 2023-10-25T00:24:27.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 -- 172.21.15.107:0/1297826904 shutdown_connections 2023-10-25T00:24:27.961 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4520152040 0x7f45200c2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:27.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f45200c8100 0x7f45200c3540 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:27.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4504065e60 0x7f4504068320 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:27.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 --2- 172.21.15.107:0/1297826904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f452006a4b0 0x7f45201edef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:27.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 -- 172.21.15.107:0/1297826904 >> 172.21.15.107:0/1297826904 conn(0x7f45200ca200 msgr2=0x7f452006a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:27.962 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 -- 172.21.15.107:0/1297826904 shutdown_connections 2023-10-25T00:24:27.963 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:27.957+0000 7f4527b77640 1 -- 172.21.15.107:0/1297826904 wait complete. 2023-10-25T00:24:27.978 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:24:29.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.105+0000 7fc5780e1640 1 Processor -- start 2023-10-25T00:24:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.105+0000 7fc5780e1640 1 -- start start 2023-10-25T00:24:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc5780e1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc5700a9820 0x7fc5700ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:30.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc5780e1640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc57006f730 con 0x7fc57014fd40 2023-10-25T00:24:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc5780e1640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc570156a40 con 0x7fc57006ee40 2023-10-25T00:24:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc5780e1640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc570156710 con 0x7fc5700a9820 2023-10-25T00:24:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc575e7d640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fc57014fd40 0x7fc5700c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:37944/0 (socket says 172.21.15.107:37944) 2023-10-25T00:24:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc575e7d640 1 -- 172.21.15.107:0/923738288 learned_addr learned my addr 172.21.15.107:0/923738288 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:24:30.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611521520 0 0) 0x7fc57006f730 con 0x7fc57014fd40 2023-10-25T00:24:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc558003530 con 0x7fc57014fd40 2023-10-25T00:24:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc57567c640 1 --2- 172.21.15.107:0/923738288 >> v2:172.21.15.110:3300/0 conn(0x7fc5700a9820 0x7fc5700ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 168285474 0 0) 0x7fc558003530 con 0x7fc57014fd40 2023-10-25T00:24:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 >> v2:172.21.15.110:3300/0 conn(0x7fc5700a9820 msgr2=0x7fc5700ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 --2- 172.21.15.107:0/923738288 >> v2:172.21.15.110:3300/0 conn(0x7fc5700a9820 0x7fc5700ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:30.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 >> v1:172.21.15.110:6789/0 conn(0x7fc57006ee40 legacy=0x7fc57015cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57006f980 con 0x7fc57014fd40 2023-10-25T00:24:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fc568001680 con 0x7fc57014fd40 2023-10-25T00:24:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 >> v1:172.21.15.107:6789/0 conn(0x7fc57014fd40 legacy=0x7fc5700c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 --2- 172.21.15.107:0/923738288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 0x7fc558003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 --2- 172.21.15.107:0/923738288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5580047d0 0x7fc558004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 --2- 172.21.15.107:0/923738288 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5580052b0 0x7fc55800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc570156710 con 0x7fc558003980 2023-10-25T00:24:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55800c180 con 0x7fc5580047d0 2023-10-25T00:24:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55800c340 con 0x7fc5580052b0 2023-10-25T00:24:30.118 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc575e7d640 1 --2- 172.21.15.107:0/923738288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 0x7fc558003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc57567c640 1 --2- 172.21.15.107:0/923738288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5580047d0 0x7fc558004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc575e7d640 1 -- 172.21.15.107:0/923738288 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5580052b0 msgr2=0x7fc55800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc575e7d640 1 --2- 172.21.15.107:0/923738288 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5580052b0 0x7fc55800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc575e7d640 1 -- 172.21.15.107:0/923738288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5580047d0 msgr2=0x7fc558004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:30.119 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc575e7d640 1 --2- 172.21.15.107:0/923738288 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5580047d0 0x7fc558004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc575e7d640 1 -- 172.21.15.107:0/923738288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc558003530 con 0x7fc558003980 2023-10-25T00:24:30.120 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.109+0000 7fc57667e640 1 -- 172.21.15.107:0/923738288 reap_dead start 2023-10-25T00:24:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc575e7d640 1 --2- 172.21.15.107:0/923738288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 0x7fc558003da0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc56800c210 tx=0x7fc56801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8d54f21d536809 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:30.123 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568005ae0 con 0x7fc558003980 2023-10-25T00:24:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc568005c40 con 0x7fc558003980 2023-10-25T00:24:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc574e7b640 1 -- 172.21.15.107:0/923738288 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5680a5a40 con 0x7fc558003980 2023-10-25T00:24:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 -- 172.21.15.107:0/923738288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 msgr2=0x7fc558003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 --2- 172.21.15.107:0/923738288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 0x7fc558003da0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc56800c210 tx=0x7fc56801cc00 comp rx=0 tx=0).stop 2023-10-25T00:24:30.124 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 -- 172.21.15.107:0/923738288 shutdown_connections 2023-10-25T00:24:30.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 --2- 172.21.15.107:0/923738288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 0x7fc558003da0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:30.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 -- 172.21.15.107:0/923738288 >> 172.21.15.107:0/923738288 conn(0x7fc5700c2950 msgr2=0x7fc5700c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:30.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 -- 172.21.15.107:0/923738288 shutdown_connections 2023-10-25T00:24:30.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 -- 172.21.15.107:0/923738288 wait complete. 2023-10-25T00:24:30.125 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 Processor -- start 2023-10-25T00:24:30.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 -- start start 2023-10-25T00:24:30.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 0x7fc5700c5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:30.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5701eea10 0x7fc5701dff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:30.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701eee90 0x7fc5701e04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:30.126 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5700bb530 con 0x7fc558003980 2023-10-25T00:24:30.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5701dec60 con 0x7fc5701eee90 2023-10-25T00:24:30.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5701ef6f0 con 0x7fc5701eea10 2023-10-25T00:24:30.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc575e7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 0x7fc5700c5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:30.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc575e7d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 0x7fc5700c5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59304/0 (socket says 172.21.15.107:59304) 2023-10-25T00:24:30.127 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc575e7d640 1 -- 172.21.15.107:0/2613310981 learned_addr learned my addr 172.21.15.107:0/2613310981 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:30.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc57667e640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701eee90 0x7fc5701e04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:30.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc57567c640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5701eea10 0x7fc5701dff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:30.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc575e7d640 1 -- 172.21.15.107:0/2613310981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5701eea10 msgr2=0x7fc5701dff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:30.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc575e7d640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5701eea10 0x7fc5701dff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:30.128 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc575e7d640 1 -- 172.21.15.107:0/2613310981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701eee90 msgr2=0x7fc5701e04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:30.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc575e7d640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701eee90 0x7fc5701e04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:30.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc575e7d640 1 -- 172.21.15.107:0/2613310981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5701deeb0 con 0x7fc558003980 2023-10-25T00:24:30.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc575e7d640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 0x7fc5700c5780 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc56800c210 tx=0x7fc558005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:30.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc56e7fc640 1 -- 172.21.15.107:0/2613310981 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5680b4070 con 0x7fc558003980 2023-10-25T00:24:30.129 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc56e7fc640 1 -- 172.21.15.107:0/2613310981 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc568001480 con 0x7fc558003980 2023-10-25T00:24:30.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 -- 172.21.15.107:0/2613310981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5701ef410 con 0x7fc558003980 2023-10-25T00:24:30.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc56e7fc640 1 -- 172.21.15.107:0/2613310981 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5680a5a40 con 0x7fc558003980 2023-10-25T00:24:30.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.113+0000 7fc5780e1640 1 -- 172.21.15.107:0/2613310981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5701e0a10 con 0x7fc558003980 2023-10-25T00:24:30.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.117+0000 7fc5780e1640 1 -- 172.21.15.107:0/2613310981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5700c24a0 con 0x7fc558003980 2023-10-25T00:24:30.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.117+0000 7fc56e7fc640 1 -- 172.21.15.107:0/2613310981 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fc5680a5ba0 con 0x7fc558003980 2023-10-25T00:24:30.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.117+0000 7fc56e7fc640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc55c065e90 0x7fc55c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:30.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.117+0000 7fc56e7fc640 1 -- 172.21.15.107:0/2613310981 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7fc56811e730 con 0x7fc558003980 2023-10-25T00:24:30.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.117+0000 7fc57567c640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc55c065e90 0x7fc55c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:30.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.117+0000 7fc57567c640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc55c065e90 0x7fc55c068350 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc57006f870 tx=0x7fc564008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:30.132 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.121+0000 7fc56e7fc640 1 -- 172.21.15.107:0/2613310981 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fc5680eab20 con 0x7fc558003980 2023-10-25T00:24:30.315 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.309+0000 7fc5780e1640 1 -- 172.21.15.107:0/2613310981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5700746e0 con 0x7fc558003980 2023-10-25T00:24:30.316 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc56e7fc640 1 -- 172.21.15.107:0/2613310981 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5769 (secure 0 0 0) 0x7fc5680ee140 con 0x7fc558003980 2023-10-25T00:24:30.316 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:30.316 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:30.317 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 235 2023-10-25T00:24:30.318 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 -- 172.21.15.107:0/2613310981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc55c065e90 msgr2=0x7fc55c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:30.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc55c065e90 0x7fc55c068350 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc57006f870 tx=0x7fc564008040 comp rx=0 tx=0).stop 2023-10-25T00:24:30.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 -- 172.21.15.107:0/2613310981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 msgr2=0x7fc5700c5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:30.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 0x7fc5700c5780 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc56800c210 tx=0x7fc558005800 comp rx=0 tx=0).stop 2023-10-25T00:24:30.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 -- 172.21.15.107:0/2613310981 shutdown_connections 2023-10-25T00:24:30.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc5701eee90 0x7fc5701e04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:30.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fc55c065e90 0x7fc55c068350 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:30.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc5701eea10 0x7fc5701dff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:30.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 --2- 172.21.15.107:0/2613310981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc558003980 0x7fc5700c5780 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:30.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 -- 172.21.15.107:0/2613310981 >> 172.21.15.107:0/2613310981 conn(0x7fc5700c2950 msgr2=0x7fc5700c0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 -- 172.21.15.107:0/2613310981 shutdown_connections 2023-10-25T00:24:30.321 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:30.313+0000 7fc5780e1640 1 -- 172.21.15.107:0/2613310981 wait complete. 2023-10-25T00:24:30.336 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:24:32.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:32.473 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a35e9640 1 Processor -- start 2023-10-25T00:24:32.474 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a35e9640 1 -- start start 2023-10-25T00:24:32.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a35e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f929c152040 0x7f929c0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:32.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a35e9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f929c0c8100 0x7f929c0abd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:32.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a35e9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f929c06a4b0 0x7f929c0ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:32.475 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a35e9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929c155680 con 0x7f929c0c8100 2023-10-25T00:24:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a35e9640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929c154a70 con 0x7f929c06a4b0 2023-10-25T00:24:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a35e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c154d50 con 0x7f929c152040 2023-10-25T00:24:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a0b84640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f929c0c8100 0x7f929c0abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a0b84640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f929c0c8100 0x7f929c0abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59314/0 (socket says 172.21.15.107:59314) 2023-10-25T00:24:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a0b84640 1 -- 172.21.15.107:0/2122380610 learned_addr learned my addr 172.21.15.107:0/2122380610 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:32.476 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a1385640 1 --2- 172.21.15.107:0/2122380610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f929c152040 0x7f929c0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a1b86640 1 --2- 172.21.15.107:0/2122380610 >> v2:172.21.15.110:3300/0 conn(0x7f929c06a4b0 0x7f929c0ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a0b84640 1 -- 172.21.15.107:0/2122380610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f929c152040 msgr2=0x7f929c0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a0b84640 1 --2- 172.21.15.107:0/2122380610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f929c152040 0x7f929c0c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a0b84640 1 -- 172.21.15.107:0/2122380610 >> v2:172.21.15.110:3300/0 conn(0x7f929c06a4b0 msgr2=0x7f929c0ac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a0b84640 1 --2- 172.21.15.107:0/2122380610 >> v2:172.21.15.110:3300/0 conn(0x7f929c06a4b0 0x7f929c0ac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:32.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.469+0000 7f92a0b84640 1 -- 172.21.15.107:0/2122380610 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c155030 con 0x7f929c0c8100 2023-10-25T00:24:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f92a0b84640 1 --2- 172.21.15.107:0/2122380610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c0c8100 0x7f929c0abd00 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f929c0c6380 tx=0x7f928c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db211123435c6f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f9293fff640 1 -- 172.21.15.107:0/2122380610 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c0ab600 con 0x7f929c0c8100 2023-10-25T00:24:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f9293fff640 1 -- 172.21.15.107:0/2122380610 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928c0a7050 con 0x7f929c0c8100 2023-10-25T00:24:32.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f9293fff640 1 -- 172.21.15.107:0/2122380610 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c0ab980 con 0x7f929c0c8100 2023-10-25T00:24:32.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f92a35e9640 1 -- 172.21.15.107:0/2122380610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c0c8100 msgr2=0x7f929c0abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:32.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f92a35e9640 1 --2- 172.21.15.107:0/2122380610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c0c8100 0x7f929c0abd00 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f929c0c6380 tx=0x7f928c01c470 comp rx=0 tx=0).stop 2023-10-25T00:24:32.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f92a35e9640 1 -- 172.21.15.107:0/2122380610 shutdown_connections 2023-10-25T00:24:32.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f92a35e9640 1 --2- 172.21.15.107:0/2122380610 >> v2:172.21.15.110:3300/0 conn(0x7f929c06a4b0 0x7f929c0ac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:32.480 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f92a35e9640 1 --2- 172.21.15.107:0/2122380610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c0c8100 0x7f929c0abd00 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f92a35e9640 1 --2- 172.21.15.107:0/2122380610 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f929c152040 0x7f929c0c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f92a35e9640 1 -- 172.21.15.107:0/2122380610 >> 172.21.15.107:0/2122380610 conn(0x7f929c0ca200 msgr2=0x7f929c0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f92a35e9640 1 -- 172.21.15.107:0/2122380610 shutdown_connections 2023-10-25T00:24:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f92a35e9640 1 -- 172.21.15.107:0/2122380610 wait complete. 2023-10-25T00:24:32.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.473+0000 7f92a35e9640 1 Processor -- start 2023-10-25T00:24:32.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a35e9640 1 -- start start 2023-10-25T00:24:32.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a35e9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f929c06a4b0 0x7f929c0aa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:32.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a35e9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c0c8100 0x7f929c151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:32.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a35e9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f929c152040 0x7f929c1f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:32.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a35e9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c0aaef0 con 0x7f929c0c8100 2023-10-25T00:24:32.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a35e9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c1f0a00 con 0x7f929c06a4b0 2023-10-25T00:24:32.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a35e9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c157580 con 0x7f929c152040 2023-10-25T00:24:32.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a0b84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c0c8100 0x7f929c151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:32.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a0b84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c0c8100 0x7f929c151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59324/0 (socket says 172.21.15.107:59324) 2023-10-25T00:24:32.485 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a0b84640 1 -- 172.21.15.107:0/2851606216 learned_addr learned my addr 172.21.15.107:0/2851606216 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:32.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a1b86640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f929c152040 0x7f929c1f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:32.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a0b84640 1 -- 172.21.15.107:0/2851606216 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f929c152040 msgr2=0x7f929c1f0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:32.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a0b84640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f929c152040 0x7f929c1f0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:32.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a0b84640 1 -- 172.21.15.107:0/2851606216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f929c06a4b0 msgr2=0x7f929c0aa6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:32.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a1385640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f929c06a4b0 0x7f929c0aa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:32.486 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a0b84640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f929c06a4b0 0x7f929c0aa6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:32.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a0b84640 1 -- 172.21.15.107:0/2851606216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928c0094d0 con 0x7f929c0c8100 2023-10-25T00:24:32.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a1385640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f929c06a4b0 0x7f929c0aa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:24:32.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a0b84640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c0c8100 0x7f929c151770 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f928c01cd10 tx=0x7f928c019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:32.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f9291ffb640 1 -- 172.21.15.107:0/2851606216 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c0ac220 con 0x7f929c0c8100 2023-10-25T00:24:32.487 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f9291ffb640 1 -- 172.21.15.107:0/2851606216 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928c0aa5b0 con 0x7f929c0c8100 2023-10-25T00:24:32.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a35e9640 1 -- 172.21.15.107:0/2851606216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c157860 con 0x7f929c0c8100 2023-10-25T00:24:32.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f9291ffb640 1 -- 172.21.15.107:0/2851606216 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c0aa960 con 0x7f929c0c8100 2023-10-25T00:24:32.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a35e9640 1 -- 172.21.15.107:0/2851606216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c151cb0 con 0x7f929c0c8100 2023-10-25T00:24:32.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a35e9640 1 -- 172.21.15.107:0/2851606216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c157d80 con 0x7f929c0c8100 2023-10-25T00:24:32.488 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f9291ffb640 1 -- 172.21.15.107:0/2851606216 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f928c0b2070 con 0x7f929c0c8100 2023-10-25T00:24:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f9291ffb640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9280065e60 0x7f9280068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f9291ffb640 1 -- 172.21.15.107:0/2851606216 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 10831+0+0 (secure 0 0 0) 0x7f928c11e970 con 0x7f929c0c8100 2023-10-25T00:24:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.477+0000 7f92a1385640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9280065e60 0x7f9280068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:32.489 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.481+0000 7f92a1385640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9280065e60 0x7f9280068320 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f9298009840 tx=0x7f9298008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:32.490 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.481+0000 7f9291ffb640 1 -- 172.21.15.107:0/2851606216 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f929c157d80 con 0x7f929c0c8100 2023-10-25T00:24:32.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 -- 172.21.15.107:0/2851606216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f929c157d80 con 0x7f929c0c8100 2023-10-25T00:24:32.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f9291ffb640 1 -- 172.21.15.107:0/2851606216 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+5769 (secure 0 0 0) 0x7f929c157d80 con 0x7f929c0c8100 2023-10-25T00:24:32.677 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:32.677 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9462,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:22:52.419094+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"a","rank":0,"incarnation":231,"state":"up:active","state_seq":20,"addr":"172.21.15.107:6837/3527905219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3527905219},{"type":"v1","addr":"172.21.15.107:6837","nonce":3527905219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:32.678 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 235 2023-10-25T00:24:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 -- 172.21.15.107:0/2851606216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9280065e60 msgr2=0x7f9280068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9280065e60 0x7f9280068320 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f9298009840 tx=0x7f9298008040 comp rx=0 tx=0).stop 2023-10-25T00:24:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 -- 172.21.15.107:0/2851606216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c0c8100 msgr2=0x7f929c151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c0c8100 0x7f929c151770 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f928c01cd10 tx=0x7f928c019210 comp rx=0 tx=0).stop 2023-10-25T00:24:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 -- 172.21.15.107:0/2851606216 shutdown_connections 2023-10-25T00:24:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9280065e60 0x7f9280068320 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f929c152040 0x7f929c1f0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f929c0c8100 0x7f929c151770 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 --2- 172.21.15.107:0/2851606216 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f929c06a4b0 0x7f929c0aa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 -- 172.21.15.107:0/2851606216 >> 172.21.15.107:0/2851606216 conn(0x7f929c0ca200 msgr2=0x7f929c153080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 -- 172.21.15.107:0/2851606216 shutdown_connections 2023-10-25T00:24:32.681 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:32.673+0000 7f92a35e9640 1 -- 172.21.15.107:0/2851606216 wait complete. 2023-10-25T00:24:32.697 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2023-10-25T00:24:34.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:34.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c4ae1e640 1 Processor -- start 2023-10-25T00:24:34.832 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c4ae1e640 1 -- start start 2023-10-25T00:24:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c4ae1e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8c44156770 0x7f8c440ab7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c4ae1e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c440be6b0 0x7f8c4415d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c4ae1e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c440c1ca0 con 0x7f8c440c70e0 2023-10-25T00:24:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c4ae1e640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c440c27e0 con 0x7f8c44156770 2023-10-25T00:24:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c4ae1e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c440c10c0 con 0x7f8c440be6b0 2023-10-25T00:24:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c48bba640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8c44156770 0x7f8c440ab7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c48bba640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f8c44156770 0x7f8c440ab7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54236/0 (socket says 172.21.15.107:54236) 2023-10-25T00:24:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c43fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f8c440c70e0 0x7f8c440bdfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41418/0 (socket says 172.21.15.107:41418) 2023-10-25T00:24:34.835 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c48bba640 1 -- 172.21.15.107:0/1036471815 learned_addr learned my addr 172.21.15.107:0/1036471815 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:34.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c48bba640 1 -- 172.21.15.107:0/1036471815 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c440be6b0 msgr2=0x7f8c4415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:34.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c493bb640 1 --2- 172.21.15.107:0/1036471815 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c440be6b0 0x7f8c4415d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:34.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c48bba640 1 --2- 172.21.15.107:0/1036471815 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c440be6b0 0x7f8c4415d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:34.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c48bba640 1 -- 172.21.15.107:0/1036471815 >> v1:172.21.15.107:6789/0 conn(0x7f8c440c70e0 legacy=0x7f8c440bdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:34.836 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.829+0000 7f8c48bba640 1 -- 172.21.15.107:0/1036471815 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c440c13a0 con 0x7f8c44156770 2023-10-25T00:24:34.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c48bba640 1 --2- 172.21.15.107:0/1036471815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c44156770 0x7f8c440ab7c0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009870 tx=0x7f8c3801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f7e6743e99040c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:34.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c437fe640 1 -- 172.21.15.107:0/1036471815 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c380ac410 con 0x7f8c44156770 2023-10-25T00:24:34.840 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c437fe640 1 -- 172.21.15.107:0/1036471815 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c380ac570 con 0x7f8c44156770 2023-10-25T00:24:34.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c437fe640 1 -- 172.21.15.107:0/1036471815 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c380ac900 con 0x7f8c44156770 2023-10-25T00:24:34.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/1036471815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c44156770 msgr2=0x7f8c440ab7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:34.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 --2- 172.21.15.107:0/1036471815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c44156770 0x7f8c440ab7c0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009870 tx=0x7f8c3801c3f0 comp rx=0 tx=0).stop 2023-10-25T00:24:34.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/1036471815 shutdown_connections 2023-10-25T00:24:34.841 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 --2- 172.21.15.107:0/1036471815 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c440be6b0 0x7f8c4415d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:34.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 --2- 172.21.15.107:0/1036471815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c44156770 0x7f8c440ab7c0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:34.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/1036471815 >> 172.21.15.107:0/1036471815 conn(0x7f8c440c65c0 msgr2=0x7f8c440c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:34.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/1036471815 shutdown_connections 2023-10-25T00:24:34.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/1036471815 wait complete. 2023-10-25T00:24:34.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 Processor -- start 2023-10-25T00:24:34.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 -- start start 2023-10-25T00:24:34.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c440be6b0 0x7f8c441f1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:34.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c440c70e0 0x7f8c441ed260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:34.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c44156770 0x7f8c4414fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:34.843 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4415cd30 con 0x7f8c44156770 2023-10-25T00:24:34.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c441f1690 con 0x7f8c440c70e0 2023-10-25T00:24:34.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4415cfd0 con 0x7f8c440be6b0 2023-10-25T00:24:34.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c493bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c44156770 0x7f8c4414fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:34.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c493bb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c44156770 0x7f8c4414fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54244/0 (socket says 172.21.15.107:54244) 2023-10-25T00:24:34.844 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c493bb640 1 -- 172.21.15.107:0/2497722454 learned_addr learned my addr 172.21.15.107:0/2497722454 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:34.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c48bba640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c440be6b0 0x7f8c441f1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:34.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c43fff640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c440c70e0 0x7f8c441ed260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:34.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c493bb640 1 -- 172.21.15.107:0/2497722454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c440be6b0 msgr2=0x7f8c441f1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:34.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c493bb640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c440be6b0 0x7f8c441f1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:34.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c493bb640 1 -- 172.21.15.107:0/2497722454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c440c70e0 msgr2=0x7f8c441ed260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:34.845 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c493bb640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c440c70e0 0x7f8c441ed260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:34.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c493bb640 1 -- 172.21.15.107:0/2497722454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c380094d0 con 0x7f8c44156770 2023-10-25T00:24:34.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c48bba640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c440be6b0 0x7f8c441f1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:24:34.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c43fff640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c440c70e0 0x7f8c441ed260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:24:34.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c493bb640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c44156770 0x7f8c4414fda0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00cd90 tx=0x7f8c3c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:34.846 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c417fa640 1 -- 172.21.15.107:0/2497722454 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c003ba0 con 0x7f8c44156770 2023-10-25T00:24:34.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c417fa640 1 -- 172.21.15.107:0/2497722454 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3c003d20 con 0x7f8c44156770 2023-10-25T00:24:34.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/2497722454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c441f0950 con 0x7f8c44156770 2023-10-25T00:24:34.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c417fa640 1 -- 172.21.15.107:0/2497722454 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c004260 con 0x7f8c44156770 2023-10-25T00:24:34.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.833+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/2497722454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c441ed7a0 con 0x7f8c44156770 2023-10-25T00:24:34.847 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.837+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/2497722454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c441f0e70 con 0x7f8c44156770 2023-10-25T00:24:34.848 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.837+0000 7f8c417fa640 1 -- 172.21.15.107:0/2497722454 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f8c3c0043c0 con 0x7f8c44156770 2023-10-25T00:24:34.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.837+0000 7f8c417fa640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c28065e60 0x7f8c28068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:34.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.837+0000 7f8c417fa640 1 -- 172.21.15.107:0/2497722454 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7f8c3c012070 con 0x7f8c44156770 2023-10-25T00:24:34.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.837+0000 7f8c48bba640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c28065e60 0x7f8c28068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:34.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.837+0000 7f8c48bba640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c28065e60 0x7f8c28068320 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8c380ae000 tx=0x7f8c380a6040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:34.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:34.841+0000 7f8c417fa640 1 -- 172.21.15.107:0/2497722454 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f8c3c0518b0 con 0x7f8c44156770 2023-10-25T00:24:35.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/2497722454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c440746e0 con 0x7f8c44156770 2023-10-25T00:24:35.036 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c417fa640 1 -- 172.21.15.107:0/2497722454 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 236 v236) v1 ==== 95+0+5007 (secure 0 0 0) 0x7f8c3c054ed0 con 0x7f8c44156770 2023-10-25T00:24:35.037 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:35.037 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":236,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:24:34.269240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"c","rank":0,"incarnation":236,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:35.037 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 236 2023-10-25T00:24:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/2497722454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c28065e60 msgr2=0x7f8c28068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c28065e60 0x7f8c28068320 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8c380ae000 tx=0x7f8c380a6040 comp rx=0 tx=0).stop 2023-10-25T00:24:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/2497722454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c44156770 msgr2=0x7f8c4414fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:35.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c44156770 0x7f8c4414fda0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00cd90 tx=0x7f8c3c00a660 comp rx=0 tx=0).stop 2023-10-25T00:24:35.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/2497722454 shutdown_connections 2023-10-25T00:24:35.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c44156770 0x7f8c4414fda0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c440c70e0 0x7f8c441ed260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f8c28065e60 0x7f8c28068320 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 --2- 172.21.15.107:0/2497722454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c440be6b0 0x7f8c441f1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/2497722454 >> 172.21.15.107:0/2497722454 conn(0x7f8c440c65c0 msgr2=0x7f8c440c5870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:35.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/2497722454 shutdown_connections 2023-10-25T00:24:35.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.033+0000 7f8c4ae1e640 1 -- 172.21.15.107:0/2497722454 wait complete. 2023-10-25T00:24:35.056 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2023-10-25T00:24:35.056 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2023-10-25T00:24:35.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:35.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.193+0000 7fe76852f640 1 Processor -- start 2023-10-25T00:24:35.199 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.193+0000 7fe76852f640 1 -- start start 2023-10-25T00:24:35.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe76852f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe76014c850 0x7fe760153240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:35.200 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe76852f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe760155cd0 con 0x7fe76014c470 2023-10-25T00:24:35.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe76852f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7600c4380 con 0x7fe76006f490 2023-10-25T00:24:35.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe76852f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe76014cc60 con 0x7fe76014c850 2023-10-25T00:24:35.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe766acc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe76014c850 0x7fe760153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:35.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe766acc640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fe76014c850 0x7fe760153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54248/0 (socket says 172.21.15.107:54248) 2023-10-25T00:24:35.201 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe766acc640 1 -- 172.21.15.107:0/4228204302 learned_addr learned my addr 172.21.15.107:0/4228204302 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:35.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe766acc640 1 -- 172.21.15.107:0/4228204302 >> v1:172.21.15.110:6789/0 conn(0x7fe76006f490 legacy=0x7fe760152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:35.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe766acc640 1 -- 172.21.15.107:0/4228204302 >> v1:172.21.15.107:6789/0 conn(0x7fe76014c470 legacy=0x7fe760071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:35.202 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe766acc640 1 -- 172.21.15.107:0/4228204302 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600c48b0 con 0x7fe76014c850 2023-10-25T00:24:35.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe766acc640 1 --2- 172.21.15.107:0/4228204302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76014c850 0x7fe760153240 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fe758009e50 tx=0x7fe75801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d761b2494093e93 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:35.203 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe7652c9640 1 -- 172.21.15.107:0/4228204302 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7580a55b0 con 0x7fe76014c850 2023-10-25T00:24:35.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe7652c9640 1 -- 172.21.15.107:0/4228204302 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7580a5710 con 0x7fe76014c850 2023-10-25T00:24:35.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe7652c9640 1 -- 172.21.15.107:0/4228204302 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7580a5aa0 con 0x7fe76014c850 2023-10-25T00:24:35.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe76852f640 1 -- 172.21.15.107:0/4228204302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76014c850 msgr2=0x7fe760153240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:35.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.197+0000 7fe76852f640 1 --2- 172.21.15.107:0/4228204302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76014c850 0x7fe760153240 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fe758009e50 tx=0x7fe75801c8f0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.204 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 -- 172.21.15.107:0/4228204302 shutdown_connections 2023-10-25T00:24:35.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 --2- 172.21.15.107:0/4228204302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76014c850 0x7fe760153240 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 -- 172.21.15.107:0/4228204302 >> 172.21.15.107:0/4228204302 conn(0x7fe760070f60 msgr2=0x7fe760071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:35.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 -- 172.21.15.107:0/4228204302 shutdown_connections 2023-10-25T00:24:35.205 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 -- 172.21.15.107:0/4228204302 wait complete. 2023-10-25T00:24:35.207 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 Processor -- start 2023-10-25T00:24:35.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 -- start start 2023-10-25T00:24:35.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76006f490 0x7fe760145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:35.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe76014c470 0x7fe7601ea5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:35.208 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76014c850 0x7fe7601eab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:35.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760137b10 con 0x7fe76006f490 2023-10-25T00:24:35.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760136a80 con 0x7fe76014c850 2023-10-25T00:24:35.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7601edb60 con 0x7fe76014c470 2023-10-25T00:24:35.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe7662cb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76006f490 0x7fe760145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:35.209 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe7662cb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76006f490 0x7fe760145bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54254/0 (socket says 172.21.15.107:54254) 2023-10-25T00:24:35.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe7662cb640 1 -- 172.21.15.107:0/3138372526 learned_addr learned my addr 172.21.15.107:0/3138372526 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:35.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe766acc640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76014c850 0x7fe7601eab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:35.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe765aca640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe76014c470 0x7fe7601ea5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:35.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe7662cb640 1 -- 172.21.15.107:0/3138372526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe76014c470 msgr2=0x7fe7601ea5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:35.210 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe7662cb640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe76014c470 0x7fe7601ea5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe7662cb640 1 -- 172.21.15.107:0/3138372526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76014c850 msgr2=0x7fe7601eab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:35.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe7662cb640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76014c850 0x7fe7601eab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe7662cb640 1 -- 172.21.15.107:0/3138372526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe758009b30 con 0x7fe76006f490 2023-10-25T00:24:35.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe766acc640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76014c850 0x7fe7601eab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:24:35.211 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe7662cb640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76006f490 0x7fe760145bd0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00c530 tx=0x7fe75c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:35.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe756ffd640 1 -- 172.21.15.107:0/3138372526 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c005220 con 0x7fe76006f490 2023-10-25T00:24:35.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe756ffd640 1 -- 172.21.15.107:0/3138372526 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c005380 con 0x7fe76006f490 2023-10-25T00:24:35.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe756ffd640 1 -- 172.21.15.107:0/3138372526 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c007270 con 0x7fe76006f490 2023-10-25T00:24:35.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 -- 172.21.15.107:0/3138372526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7601ee400 con 0x7fe76006f490 2023-10-25T00:24:35.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.201+0000 7fe76852f640 1 -- 172.21.15.107:0/3138372526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe760136cb0 con 0x7fe76006f490 2023-10-25T00:24:35.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.205+0000 7fe76852f640 1 -- 172.21.15.107:0/3138372526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7600746e0 con 0x7fe76006f490 2023-10-25T00:24:35.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.205+0000 7fe756ffd640 1 -- 172.21.15.107:0/3138372526 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe75c0073d0 con 0x7fe76006f490 2023-10-25T00:24:35.213 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.205+0000 7fe756ffd640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe744065e60 0x7fe744068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:35.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.205+0000 7fe756ffd640 1 -- 172.21.15.107:0/3138372526 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7fe75c0843a0 con 0x7fe76006f490 2023-10-25T00:24:35.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.205+0000 7fe765aca640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe744065e60 0x7fe744068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:35.214 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.205+0000 7fe765aca640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe744065e60 0x7fe744068320 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe7601423f0 tx=0x7fe750009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:35.216 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.213+0000 7fe756ffd640 1 -- 172.21.15.107:0/3138372526 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe75c050730 con 0x7fe76006f490 2023-10-25T00:24:35.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.397+0000 7fe76852f640 1 -- 172.21.15.107:0/3138372526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe760152270 con 0x7fe76006f490 2023-10-25T00:24:35.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe756ffd640 1 -- 172.21.15.107:0/3138372526 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 236 v236) v1 ==== 95+0+5007 (secure 0 0 0) 0x7fe75c053d50 con 0x7fe76006f490 2023-10-25T00:24:35.404 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:35.405 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":236,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:24:34.269240+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"c","rank":0,"incarnation":236,"state":"up:replay","state_seq":1,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:35.405 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 236 2023-10-25T00:24:35.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 -- 172.21.15.107:0/3138372526 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe744065e60 msgr2=0x7fe744068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:35.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe744065e60 0x7fe744068320 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fe7601423f0 tx=0x7fe750009040 comp rx=0 tx=0).stop 2023-10-25T00:24:35.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 -- 172.21.15.107:0/3138372526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76006f490 msgr2=0x7fe760145bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:35.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76006f490 0x7fe760145bd0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00c530 tx=0x7fe75c00c9f0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 -- 172.21.15.107:0/3138372526 shutdown_connections 2023-10-25T00:24:35.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe76014c850 0x7fe7601eab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe744065e60 0x7fe744068320 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe76014c470 0x7fe7601ea5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 --2- 172.21.15.107:0/3138372526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe76006f490 0x7fe760145bd0 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:35.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 -- 172.21.15.107:0/3138372526 >> 172.21.15.107:0/3138372526 conn(0x7fe760070f60 msgr2=0x7fe7600c20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:35.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 -- 172.21.15.107:0/3138372526 shutdown_connections 2023-10-25T00:24:35.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:35.401+0000 7fe76852f640 1 -- 172.21.15.107:0/3138372526 wait complete. 2023-10-25T00:24:35.428 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 236, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:02.679785+0000", "damaged": [], "data_pools": [ 4 ], "enabled": true, "epoch": 236, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_9462": { "addr": "172.21.15.107:6835/2963448958", "addrs": { "addrvec": [ { "addr": "172.21.15.107:6834", "nonce": 2963448958, "type": "v2" }, { "addr": "172.21.15.107:6835", "nonce": 2963448958, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9462, "incarnation": 236, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 3, "modified": "2023-10-25T00:24:34.269240+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9462 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-24T23:53:11.777452+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 224, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7483": { "addr": "172.21.15.110:6834/47497412", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6832", "nonce": 47497412, "type": "v2" }, { "addr": "172.21.15.110:6834", "nonce": 47497412, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7483, "incarnation": 221, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 5, "modified": "2023-10-25T00:21:22.342534+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7483 } } } ], "standbys": [ { "addr": "172.21.15.110:6835/2437148", "addrs": { "addrvec": [ { "addr": "172.21.15.110:6833", "nonce": 2437148, "type": "v2" }, { "addr": "172.21.15.110:6835", "nonce": 2437148, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 190, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6443, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-25T00:24:37.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:37.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.561+0000 7f109be50640 1 Processor -- start 2023-10-25T00:24:37.567 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.561+0000 7f109be50640 1 -- start start 2023-10-25T00:24:37.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f109be50640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10940ac020 0x7f10940c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:37.568 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f109be50640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f10940c2520 0x7f109406c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:37.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f109be50640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1094066000 con 0x7f10940aa140 2023-10-25T00:24:37.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f109be50640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10940c0f40 con 0x7f10940c2520 2023-10-25T00:24:37.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f109be50640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10940c1220 con 0x7f10940ac020 2023-10-25T00:24:37.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f1099bec640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f10940aa140 0x7f10940aa520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:41428/0 (socket says 172.21.15.107:41428) 2023-10-25T00:24:37.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f1099bec640 1 -- 172.21.15.107:0/2456634256 learned_addr learned my addr 172.21.15.107:0/2456634256 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:24:37.569 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f109a3ed640 1 --2- 172.21.15.107:0/2456634256 >> v2:172.21.15.107:3300/0 conn(0x7f10940c2520 0x7f109406c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:37.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f10993eb640 1 --2- 172.21.15.107:0/2456634256 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10940ac020 0x7f10940c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:37.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f109a3ed640 1 -- 172.21.15.107:0/2456634256 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10940ac020 msgr2=0x7f10940c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:37.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f109a3ed640 1 --2- 172.21.15.107:0/2456634256 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10940ac020 0x7f10940c1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:37.570 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f109a3ed640 1 -- 172.21.15.107:0/2456634256 >> v1:172.21.15.107:6789/0 conn(0x7f10940aa140 legacy=0x7f10940aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:37.571 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f109a3ed640 1 -- 172.21.15.107:0/2456634256 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10940c1500 con 0x7f10940c2520 2023-10-25T00:24:37.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f109a3ed640 1 --2- 172.21.15.107:0/2456634256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940c2520 0x7f109406c490 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f10900097c0 tx=0x7f109001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=519b2b9bfe2cc966 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:37.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f1098bea640 1 -- 172.21.15.107:0/2456634256 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10900ab650 con 0x7f10940c2520 2023-10-25T00:24:37.572 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f1098bea640 1 -- 172.21.15.107:0/2456634256 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10900ab7b0 con 0x7f10940c2520 2023-10-25T00:24:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.565+0000 7f1098bea640 1 -- 172.21.15.107:0/2456634256 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10900abb40 con 0x7f10940c2520 2023-10-25T00:24:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- 172.21.15.107:0/2456634256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940c2520 msgr2=0x7f109406c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 --2- 172.21.15.107:0/2456634256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940c2520 0x7f109406c490 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f10900097c0 tx=0x7f109001c450 comp rx=0 tx=0).stop 2023-10-25T00:24:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- 172.21.15.107:0/2456634256 shutdown_connections 2023-10-25T00:24:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 --2- 172.21.15.107:0/2456634256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940c2520 0x7f109406c490 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 --2- 172.21.15.107:0/2456634256 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10940ac020 0x7f10940c1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- 172.21.15.107:0/2456634256 >> 172.21.15.107:0/2456634256 conn(0x7f10940c65c0 msgr2=0x7f10940c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:37.574 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- 172.21.15.107:0/2456634256 shutdown_connections 2023-10-25T00:24:37.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- 172.21.15.107:0/2456634256 wait complete. 2023-10-25T00:24:37.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 Processor -- start 2023-10-25T00:24:37.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- start start 2023-10-25T00:24:37.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10940aa140 0x7f1094070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:37.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940ac020 0x7f10940bb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:37.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10940c2520 0x7f10940bba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:37.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10941ea2c0 con 0x7f10940ac020 2023-10-25T00:24:37.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10941e93c0 con 0x7f10940c2520 2023-10-25T00:24:37.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10941e9640 con 0x7f10940aa140 2023-10-25T00:24:37.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f10993eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940ac020 0x7f10940bb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:37.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f10993eb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940ac020 0x7f10940bb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54270/0 (socket says 172.21.15.107:54270) 2023-10-25T00:24:37.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f10993eb640 1 -- 172.21.15.107:0/3097698728 learned_addr learned my addr 172.21.15.107:0/3097698728 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:37.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109a3ed640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10940c2520 0x7f10940bba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:37.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f10993eb640 1 -- 172.21.15.107:0/3097698728 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10940aa140 msgr2=0x7f1094070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:37.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f1099bec640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10940aa140 0x7f1094070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:37.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f10993eb640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10940aa140 0x7f1094070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:37.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f10993eb640 1 -- 172.21.15.107:0/3097698728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10940c2520 msgr2=0x7f10940bba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:37.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f10993eb640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10940c2520 0x7f10940bba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:37.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f10993eb640 1 -- 172.21.15.107:0/3097698728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1090009420 con 0x7f10940ac020 2023-10-25T00:24:37.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f10993eb640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940ac020 0x7f10940bb500 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f108400eac0 tx=0x7f108400ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:37.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f108a7fc640 1 -- 172.21.15.107:0/3097698728 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084007c10 con 0x7f10940ac020 2023-10-25T00:24:37.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f108a7fc640 1 -- 172.21.15.107:0/3097698728 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1084007d70 con 0x7f10940ac020 2023-10-25T00:24:37.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- 172.21.15.107:0/3097698728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10941e9920 con 0x7f10940ac020 2023-10-25T00:24:37.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f108a7fc640 1 -- 172.21.15.107:0/3097698728 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1084004240 con 0x7f10940ac020 2023-10-25T00:24:37.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- 172.21.15.107:0/3097698728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10941e9ee0 con 0x7f10940ac020 2023-10-25T00:24:37.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.569+0000 7f109be50640 1 -- 172.21.15.107:0/3097698728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10940746e0 con 0x7f10940ac020 2023-10-25T00:24:37.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.573+0000 7f108a7fc640 1 -- 172.21.15.107:0/3097698728 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f10840043a0 con 0x7f10940ac020 2023-10-25T00:24:37.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.573+0000 7f108a7fc640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1078065e90 0x7f1078068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:37.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.573+0000 7f1099bec640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1078065e90 0x7f1078068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:37.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.573+0000 7f108a7fc640 1 -- 172.21.15.107:0/3097698728 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7f1084014070 con 0x7f10940ac020 2023-10-25T00:24:37.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.573+0000 7f1099bec640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1078065e90 0x7f1078068350 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f108c005c10 tx=0x7f108c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:37.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.577+0000 7f108a7fc640 1 -- 172.21.15.107:0/3097698728 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f10840533d0 con 0x7f10940ac020 2023-10-25T00:24:37.771 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.765+0000 7f109be50640 1 -- 172.21.15.107:0/3097698728 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10940bf850 con 0x7f10940ac020 2023-10-25T00:24:37.772 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f108a7fc640 1 -- 172.21.15.107:0/3097698728 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 238 v238) v1 ==== 95+0+5008 (secure 0 0 0) 0x7f10840569f0 con 0x7f10940ac020 2023-10-25T00:24:37.772 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:37.772 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":238,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":238,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:24:37.435339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"c","rank":0,"incarnation":236,"state":"up:rejoin","state_seq":26,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:37.773 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 238 2023-10-25T00:24:37.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 -- 172.21.15.107:0/3097698728 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1078065e90 msgr2=0x7f1078068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:37.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1078065e90 0x7f1078068350 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f108c005c10 tx=0x7f108c009040 comp rx=0 tx=0).stop 2023-10-25T00:24:37.774 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 -- 172.21.15.107:0/3097698728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940ac020 msgr2=0x7f10940bb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:37.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940ac020 0x7f10940bb500 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f108400eac0 tx=0x7f108400ef80 comp rx=0 tx=0).stop 2023-10-25T00:24:37.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 -- 172.21.15.107:0/3097698728 shutdown_connections 2023-10-25T00:24:37.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10940c2520 0x7f10940bba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:37.775 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10940ac020 0x7f10940bb500 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1078065e90 0x7f1078068350 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 --2- 172.21.15.107:0/3097698728 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10940aa140 0x7f1094070800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 -- 172.21.15.107:0/3097698728 >> 172.21.15.107:0/3097698728 conn(0x7f10940c65c0 msgr2=0x7f1094071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 -- 172.21.15.107:0/3097698728 shutdown_connections 2023-10-25T00:24:37.777 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:37.769+0000 7f109be50640 1 -- 172.21.15.107:0/3097698728 wait complete. 2023-10-25T00:24:37.791 INFO:tasks.mds_thrash.fs.[a]:no change 2023-10-25T00:24:39.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:39.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-25T00:24:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.917+0000 7ff66914a640 1 Processor -- start 2023-10-25T00:24:39.923 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff66914a640 1 -- start start 2023-10-25T00:24:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff66914a640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff6640c2cf0 0x7ff6640ab070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff66914a640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff664073800 con 0x7ff6640ac400 2023-10-25T00:24:39.925 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff66914a640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6640a9810 con 0x7ff66406d8d0 2023-10-25T00:24:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff66914a640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6640aa550 con 0x7ff6640c2cf0 2023-10-25T00:24:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff66259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff6640c2cf0 0x7ff6640ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff66259c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff6640c2cf0 0x7ff6640ab070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54280/0 (socket says 172.21.15.107:54280) 2023-10-25T00:24:39.926 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff66259c640 1 -- 172.21.15.107:0/33234947 learned_addr learned my addr 172.21.15.107:0/33234947 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:39.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff66259c640 1 -- 172.21.15.107:0/33234947 >> v1:172.21.15.110:6789/0 conn(0x7ff66406d8d0 legacy=0x7ff664152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:39.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff66259c640 1 -- 172.21.15.107:0/33234947 >> v1:172.21.15.107:6789/0 conn(0x7ff6640ac400 legacy=0x7ff66414ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:39.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff661d9b640 1 -- 172.21.15.107:0/33234947 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727595206 0 0) 0x7ff664073800 con 0x7ff6640ac400 2023-10-25T00:24:39.927 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.921+0000 7ff66259c640 1 -- 172.21.15.107:0/33234947 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6640a8e90 con 0x7ff6640c2cf0 2023-10-25T00:24:39.929 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66259c640 1 --2- 172.21.15.107:0/33234947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6640c2cf0 0x7ff6640ab070 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7ff658009870 tx=0x7ff65801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9e8855039b018e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff661d9b640 1 -- 172.21.15.107:0/33234947 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6580a6040 con 0x7ff6640c2cf0 2023-10-25T00:24:39.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff661d9b640 1 -- 172.21.15.107:0/33234947 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6580aa600 con 0x7ff6640c2cf0 2023-10-25T00:24:39.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff661d9b640 1 -- 172.21.15.107:0/33234947 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6580aa990 con 0x7ff6640c2cf0 2023-10-25T00:24:39.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 -- 172.21.15.107:0/33234947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6640c2cf0 msgr2=0x7ff6640ab070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:39.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 --2- 172.21.15.107:0/33234947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6640c2cf0 0x7ff6640ab070 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7ff658009870 tx=0x7ff65801c480 comp rx=0 tx=0).stop 2023-10-25T00:24:39.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 -- 172.21.15.107:0/33234947 shutdown_connections 2023-10-25T00:24:39.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 --2- 172.21.15.107:0/33234947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6640c2cf0 0x7ff6640ab070 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:39.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 -- 172.21.15.107:0/33234947 >> 172.21.15.107:0/33234947 conn(0x7ff664069d40 msgr2=0x7ff66406a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:39.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 -- 172.21.15.107:0/33234947 shutdown_connections 2023-10-25T00:24:39.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 -- 172.21.15.107:0/33234947 wait complete. 2023-10-25T00:24:39.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 Processor -- start 2023-10-25T00:24:39.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 -- start start 2023-10-25T00:24:39.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66406d8d0 0x7ff6641ea6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:39.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6640ac400 0x7ff664146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:39.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6640c2cf0 0x7ff6641ee4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:39.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640a9170 con 0x7ff66406d8d0 2023-10-25T00:24:39.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6640c52e0 con 0x7ff6640ac400 2023-10-25T00:24:39.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66914a640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6641eabe0 con 0x7ff6640c2cf0 2023-10-25T00:24:39.936 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff662d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66406d8d0 0x7ff6641ea6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:39.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff662d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66406d8d0 0x7ff6641ea6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54292/0 (socket says 172.21.15.107:54292) 2023-10-25T00:24:39.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff662d9d640 1 -- 172.21.15.107:0/3092351851 learned_addr learned my addr 172.21.15.107:0/3092351851 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:39.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66259c640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6640ac400 0x7ff664146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:39.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66359e640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6640c2cf0 0x7ff6641ee4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:39.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff662d9d640 1 -- 172.21.15.107:0/3092351851 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6640c2cf0 msgr2=0x7ff6641ee4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:39.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff662d9d640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6640c2cf0 0x7ff6641ee4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:39.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff662d9d640 1 -- 172.21.15.107:0/3092351851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6640ac400 msgr2=0x7ff664146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:39.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff662d9d640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6640ac400 0x7ff664146370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:39.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff662d9d640 1 -- 172.21.15.107:0/3092351851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6580094d0 con 0x7ff66406d8d0 2023-10-25T00:24:39.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff66359e640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6640c2cf0 0x7ff6641ee4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:24:39.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.925+0000 7ff662d9d640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66406d8d0 0x7ff6641ea6a0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7ff65400c530 tx=0x7ff65400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:39.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.929+0000 7ff64b7fe640 1 -- 172.21.15.107:0/3092351851 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654005220 con 0x7ff66406d8d0 2023-10-25T00:24:39.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.929+0000 7ff64b7fe640 1 -- 172.21.15.107:0/3092351851 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff654005380 con 0x7ff66406d8d0 2023-10-25T00:24:39.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.929+0000 7ff64b7fe640 1 -- 172.21.15.107:0/3092351851 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654007270 con 0x7ff66406d8d0 2023-10-25T00:24:39.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.929+0000 7ff66914a640 1 -- 172.21.15.107:0/3092351851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6641ee020 con 0x7ff66406d8d0 2023-10-25T00:24:39.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.929+0000 7ff66914a640 1 -- 172.21.15.107:0/3092351851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6640c5000 con 0x7ff66406d8d0 2023-10-25T00:24:39.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.929+0000 7ff66914a640 1 -- 172.21.15.107:0/3092351851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6640746e0 con 0x7ff66406d8d0 2023-10-25T00:24:39.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.929+0000 7ff64b7fe640 1 -- 172.21.15.107:0/3092351851 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff6540073d0 con 0x7ff66406d8d0 2023-10-25T00:24:39.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.929+0000 7ff64b7fe640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff650065e30 0x7ff6500682f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:39.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.929+0000 7ff64b7fe640 1 -- 172.21.15.107:0/3092351851 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7ff654084440 con 0x7ff66406d8d0 2023-10-25T00:24:39.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.929+0000 7ff66259c640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff650065e30 0x7ff6500682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:39.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.929+0000 7ff66259c640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff650065e30 0x7ff6500682f0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff6580ae000 tx=0x7ff6580a7000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:39.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:39.933+0000 7ff64b7fe640 1 -- 172.21.15.107:0/3092351851 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff6540507d0 con 0x7ff66406d8d0 2023-10-25T00:24:40.031 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.025+0000 7f9391cba640 1 Processor -- start 2023-10-25T00:24:40.032 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.025+0000 7f9391cba640 1 -- start start 2023-10-25T00:24:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f9391cba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f938c06e310 0x7f938c0aa910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f9391cba640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f938c0ba4d0 0x7f938c15bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f9391cba640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f938c0c17c0 con 0x7f938c151fd0 2023-10-25T00:24:40.033 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f9391cba640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f938c0aa0c0 con 0x7f938c0ba4d0 2023-10-25T00:24:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f9391cba640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938c0c2060 con 0x7f938c06e310 2023-10-25T00:24:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f938c0ba4d0 0x7f938c15bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938bfff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f938c0ba4d0 0x7f938c15bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54302/0 (socket says 172.21.15.107:54302) 2023-10-25T00:24:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938bfff640 1 -- 172.21.15.107:0/638506095 learned_addr learned my addr 172.21.15.107:0/638506095 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:40.034 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938b7fe640 1 --2- 172.21.15.107:0/638506095 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f938c06e310 0x7f938c0aa910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938bfff640 1 -- 172.21.15.107:0/638506095 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f938c06e310 msgr2=0x7f938c0aa910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938bfff640 1 --2- 172.21.15.107:0/638506095 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f938c06e310 0x7f938c0aa910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938bfff640 1 -- 172.21.15.107:0/638506095 >> v1:172.21.15.110:6789/0 conn(0x7f938c151fd0 legacy=0x7f938c0c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:40.035 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938bfff640 1 -- 172.21.15.107:0/638506095 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c06e6f0 con 0x7f938c0ba4d0 2023-10-25T00:24:40.038 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938bfff640 1 --2- 172.21.15.107:0/638506095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c0ba4d0 0x7f938c15bf40 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f9384009e50 tx=0x7f938401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be29ae9f4c6ae8db server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938a7fc640 1 -- 172.21.15.107:0/638506095 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93840a5790 con 0x7f938c0ba4d0 2023-10-25T00:24:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938a7fc640 1 -- 172.21.15.107:0/638506095 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93840a58f0 con 0x7f938c0ba4d0 2023-10-25T00:24:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f938a7fc640 1 -- 172.21.15.107:0/638506095 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93840a5c80 con 0x7f938c0ba4d0 2023-10-25T00:24:40.039 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f9391cba640 1 -- 172.21.15.107:0/638506095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c0ba4d0 msgr2=0x7f938c15bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.029+0000 7f9391cba640 1 --2- 172.21.15.107:0/638506095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c0ba4d0 0x7f938c15bf40 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f9384009e50 tx=0x7f938401c910 comp rx=0 tx=0).stop 2023-10-25T00:24:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 -- 172.21.15.107:0/638506095 shutdown_connections 2023-10-25T00:24:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 --2- 172.21.15.107:0/638506095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c0ba4d0 0x7f938c15bf40 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 --2- 172.21.15.107:0/638506095 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f938c06e310 0x7f938c0aa910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 -- 172.21.15.107:0/638506095 >> 172.21.15.107:0/638506095 conn(0x7f938c0c38c0 msgr2=0x7f938c0c3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 -- 172.21.15.107:0/638506095 shutdown_connections 2023-10-25T00:24:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 -- 172.21.15.107:0/638506095 wait complete. 2023-10-25T00:24:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 Processor -- start 2023-10-25T00:24:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 -- start start 2023-10-25T00:24:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c06e310 0x7f938c0bc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f938c0ba4d0 0x7f938c0bce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f938c151fd0 0x7f938c1ede40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c0b9b20 con 0x7f938c06e310 2023-10-25T00:24:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938c0bf250 con 0x7f938c0ba4d0 2023-10-25T00:24:40.042 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938c151440 con 0x7f938c151fd0 2023-10-25T00:24:40.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f938b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c06e310 0x7f938c0bc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:40.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f938b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c06e310 0x7f938c0bc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54314/0 (socket says 172.21.15.107:54314) 2023-10-25T00:24:40.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f938b7fe640 1 -- 172.21.15.107:0/2270948904 learned_addr learned my addr 172.21.15.107:0/2270948904 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:40.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f938affd640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f938c0ba4d0 0x7f938c0bce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:40.043 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f938bfff640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f938c151fd0 0x7f938c1ede40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:40.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f938b7fe640 1 -- 172.21.15.107:0/2270948904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f938c151fd0 msgr2=0x7f938c1ede40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:40.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f938b7fe640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f938c151fd0 0x7f938c1ede40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f938b7fe640 1 -- 172.21.15.107:0/2270948904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f938c0ba4d0 msgr2=0x7f938c0bce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:40.044 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f938b7fe640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f938c0ba4d0 0x7f938c0bce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f938b7fe640 1 -- 172.21.15.107:0/2270948904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9384009b30 con 0x7f938c06e310 2023-10-25T00:24:40.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f938b7fe640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c06e310 0x7f938c0bc8d0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f937c00c820 tx=0x7f937c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:40.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9373fff640 1 -- 172.21.15.107:0/2270948904 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937c007a30 con 0x7f938c06e310 2023-10-25T00:24:40.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9373fff640 1 -- 172.21.15.107:0/2270948904 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937c007b90 con 0x7f938c06e310 2023-10-25T00:24:40.045 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 -- 172.21.15.107:0/2270948904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938c0b9560 con 0x7f938c06e310 2023-10-25T00:24:40.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9373fff640 1 -- 172.21.15.107:0/2270948904 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937c00ce10 con 0x7f938c06e310 2023-10-25T00:24:40.046 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 -- 172.21.15.107:0/2270948904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f938c0b9200 con 0x7f938c06e310 2023-10-25T00:24:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.033+0000 7f9391cba640 1 -- 172.21.15.107:0/2270948904 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938c04e0c0 con 0x7f938c06e310 2023-10-25T00:24:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.037+0000 7f9373fff640 1 -- 172.21.15.107:0/2270948904 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f937c024480 con 0x7f938c06e310 2023-10-25T00:24:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.037+0000 7f9373fff640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9378065e60 0x7f9378068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.037+0000 7f9373fff640 1 -- 172.21.15.107:0/2270948904 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7f937c085020 con 0x7f938c06e310 2023-10-25T00:24:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.037+0000 7f938affd640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9378065e60 0x7f9378068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.037+0000 7f938affd640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9378065e60 0x7f9378068320 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f938c15c230 tx=0x7f9380009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:40.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.041+0000 7f9373fff640 1 -- 172.21.15.107:0/2270948904 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f937c051430 con 0x7f938c06e310 2023-10-25T00:24:40.130 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.125+0000 7ff66914a640 1 -- 172.21.15.107:0/3092351851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6640c2310 con 0x7ff66406d8d0 2023-10-25T00:24:40.131 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff64b7fe640 1 -- 172.21.15.107:0/3092351851 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+5008 (secure 0 0 0) 0x7ff654053df0 con 0x7ff66406d8d0 2023-10-25T00:24:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:40.132 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:24:38.446814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"c","rank":0,"incarnation":236,"state":"up:active","state_seq":27,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:40.132 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 239 2023-10-25T00:24:40.133 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 -- 172.21.15.107:0/3092351851 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff650065e30 msgr2=0x7ff6500682f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:40.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff650065e30 0x7ff6500682f0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff6580ae000 tx=0x7ff6580a7000 comp rx=0 tx=0).stop 2023-10-25T00:24:40.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 -- 172.21.15.107:0/3092351851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66406d8d0 msgr2=0x7ff6641ea6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:40.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66406d8d0 0x7ff6641ea6a0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7ff65400c530 tx=0x7ff65400c9f0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.134 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 -- 172.21.15.107:0/3092351851 shutdown_connections 2023-10-25T00:24:40.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff650065e30 0x7ff6500682f0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff6640c2cf0 0x7ff6641ee4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff6640ac400 0x7ff664146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 --2- 172.21.15.107:0/3092351851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff66406d8d0 0x7ff6641ea6a0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.135 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 -- 172.21.15.107:0/3092351851 >> 172.21.15.107:0/3092351851 conn(0x7ff664069d40 msgr2=0x7ff66414e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:40.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 -- 172.21.15.107:0/3092351851 shutdown_connections 2023-10-25T00:24:40.136 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.129+0000 7ff66914a640 1 -- 172.21.15.107:0/3092351851 wait complete. 2023-10-25T00:24:40.150 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=9054 2023-10-25T00:24:40.150 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.a 2023-10-25T00:24:40.198 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.193+0000 7f9391cba640 1 -- 172.21.15.107:0/2270948904 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f938c0c1cf0 con 0x7f9378065e60 2023-10-25T00:24:40.212 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.209+0000 7f9373fff640 1 -- 172.21.15.107:0/2270948904 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+556103 (secure 0 0 0) 0x7f9380005380 con 0x7f9378065e60 2023-10-25T00:24:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:40.224 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-25T00:24:40.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 -- 172.21.15.107:0/2270948904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9378065e60 msgr2=0x7f9378068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:40.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9378065e60 0x7f9378068320 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f938c15c230 tx=0x7f9380009040 comp rx=0 tx=0).stop 2023-10-25T00:24:40.225 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 -- 172.21.15.107:0/2270948904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c06e310 msgr2=0x7f938c0bc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:40.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c06e310 0x7f938c0bc8d0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f937c00c820 tx=0x7f937c00cce0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 -- 172.21.15.107:0/2270948904 shutdown_connections 2023-10-25T00:24:40.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9378065e60 0x7f9378068320 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f938c151fd0 0x7f938c1ede40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.226 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f938c0ba4d0 0x7f938c0bce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 --2- 172.21.15.107:0/2270948904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f938c06e310 0x7f938c0bc8d0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:40.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 -- 172.21.15.107:0/2270948904 >> 172.21.15.107:0/2270948904 conn(0x7f938c0c38c0 msgr2=0x7f938c0c5ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:40.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 -- 172.21.15.107:0/2270948904 shutdown_connections 2023-10-25T00:24:40.227 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:40.217+0000 7f9391cba640 1 -- 172.21.15.107:0/2270948904 wait complete. 2023-10-25T00:24:40.237 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1022,"stamp":"2023-10-25T00:24:40.068896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40142898,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3993,"num_read_kb":496993,"num_write":3153461,"num_write_kb":38880074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":584008,"ondisk_log_size":584008,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7810752,"kb_used_data":86336,"kb_used_omap":0,"kb_used_meta":7724352,"kb_avail":747163968,"statfs":{"total":773094113280,"available":765095903232,"internally_reserved":0,"allocated":88408064,"data_stored":84736500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7909736448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":19519,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004865"},"pg_stats":[{"pgid":"6.1a","version":"72'3995","reported_seq":4082,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:16.475475+0000","last_change":"2023-10-25T00:24:16.475475+0000","last_active":"2023-10-25T00:24:16.475475+0000","last_peered":"2023-10-25T00:24:16.475475+0000","last_clean":"2023-10-25T00:24:16.475475+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-25T00:24:16.475475+0000","last_undegraded":"2023-10-25T00:24:16.475475+0000","last_fullsized":"2023-10-25T00:24:16.475475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3995","last_scrub_stamp":"2023-10-25T00:24:16.475258+0000","last_deep_scrub":"72'3995","last_deep_scrub_stamp":"2023-10-25T00:24:16.475258+0000","last_clean_scrub_stamp":"2023-10-25T00:24:16.475258+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:52:25.511844+0000","scrub_duration":0.42505980700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21480,"num_write_kb":282788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"72'3","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:29.637684+0000","last_change":"2023-10-25T00:23:29.637684+0000","last_active":"2023-10-25T00:23:29.637684+0000","last_peered":"2023-10-25T00:23:29.637684+0000","last_clean":"2023-10-25T00:23:29.637684+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-25T00:23:29.637684+0000","last_undegraded":"2023-10-25T00:23:29.637684+0000","last_fullsized":"2023-10-25T00:23:29.637684+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:29.637307+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:29.637307+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.637307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:55.311760+0000","scrub_duration":0.0045847350000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"73'867","reported_seq":1011,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.121717+0000","last_change":"2023-10-25T00:23:27.027247+0000","last_active":"2023-10-25T00:24:35.121717+0000","last_peered":"2023-10-25T00:24:35.121717+0000","last_clean":"2023-10-25T00:24:35.121717+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-25T00:24:35.121717+0000","last_undegraded":"2023-10-25T00:24:35.121717+0000","last_fullsized":"2023-10-25T00:24:35.121717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'867","last_scrub_stamp":"2023-10-25T00:23:27.026943+0000","last_deep_scrub":"73'867","last_deep_scrub_stamp":"2023-10-25T00:23:27.026943+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.026943+0000","objects_scrubbed":3,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:41.065198+0000","scrub_duration":0.082266929000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":101,"num_read_kb":26638,"num_write":841,"num_write_kb":4586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'5616","reported_seq":5703,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.429483+0000","last_change":"2023-10-25T00:23:46.131251+0000","last_active":"2023-10-25T00:24:31.429483+0000","last_peered":"2023-10-25T00:24:31.429483+0000","last_clean":"2023-10-25T00:24:31.429483+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-25T00:24:31.429483+0000","last_undegraded":"2023-10-25T00:24:31.429483+0000","last_fullsized":"2023-10-25T00:24:31.429483+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5616","last_scrub_stamp":"2023-10-25T00:23:46.131002+0000","last_deep_scrub":"68'5616","last_deep_scrub_stamp":"2023-10-25T00:23:46.131002+0000","last_clean_scrub_stamp":"2023-10-25T00:23:46.131002+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:29:47.372296+0000","scrub_duration":0.13911974099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29775,"num_write_kb":348818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"72'3735","reported_seq":3820,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:59.753562+0000","last_change":"2023-10-25T00:23:59.753562+0000","last_active":"2023-10-25T00:23:59.753562+0000","last_peered":"2023-10-25T00:23:59.753562+0000","last_clean":"2023-10-25T00:23:59.753562+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-25T00:23:59.753562+0000","last_undegraded":"2023-10-25T00:23:59.753562+0000","last_fullsized":"2023-10-25T00:23:59.753562+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3735","last_scrub_stamp":"2023-10-25T00:23:59.753413+0000","last_deep_scrub":"72'3735","last_deep_scrub_stamp":"2023-10-25T00:23:59.753413+0000","last_clean_scrub_stamp":"2023-10-25T00:23:59.753413+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:57:23.808102+0000","scrub_duration":0.20248877100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20090,"num_write_kb":264783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"72'2","reported_seq":84,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:02.880442+0000","last_change":"2023-10-25T00:23:02.880442+0000","last_active":"2023-10-25T00:23:02.880442+0000","last_peered":"2023-10-25T00:23:02.880442+0000","last_clean":"2023-10-25T00:23:02.880442+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-25T00:23:02.880442+0000","last_undegraded":"2023-10-25T00:23:02.880442+0000","last_fullsized":"2023-10-25T00:23:02.880442+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.880202+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:45.433003+0000","scrub_duration":0.0048172340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"74'29","reported_seq":118,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.314049+0000","last_change":"2023-10-25T00:23:05.443056+0000","last_active":"2023-10-25T00:24:34.314049+0000","last_peered":"2023-10-25T00:24:34.314049+0000","last_clean":"2023-10-25T00:24:34.314049+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-25T00:24:34.314049+0000","last_undegraded":"2023-10-25T00:24:34.314049+0000","last_fullsized":"2023-10-25T00:24:34.314049+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_deep_scrub":"73'28","last_deep_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.442750+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:29:04.499904+0000","scrub_duration":0.036465213000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'5447","reported_seq":5534,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.569763+0000","last_change":"2023-10-25T00:23:28.801391+0000","last_active":"2023-10-25T00:24:32.569763+0000","last_peered":"2023-10-25T00:24:32.569763+0000","last_clean":"2023-10-25T00:24:32.569763+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-25T00:24:32.569763+0000","last_undegraded":"2023-10-25T00:24:32.569763+0000","last_fullsized":"2023-10-25T00:24:32.569763+0000","mapping_epoch":21,"log_start":"61'100","ondisk_log_start":"61'100","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5447","last_scrub_stamp":"2023-10-25T00:23:28.801240+0000","last_deep_scrub":"68'5447","last_deep_scrub_stamp":"2023-10-25T00:23:28.801240+0000","last_clean_scrub_stamp":"2023-10-25T00:23:28.801240+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:44:02.442972+0000","scrub_duration":0.154979636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28063,"num_write_kb":350200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"72'4747","reported_seq":4832,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.173944+0000","last_change":"2023-10-25T00:23:40.173944+0000","last_active":"2023-10-25T00:23:40.173944+0000","last_peered":"2023-10-25T00:23:40.173944+0000","last_clean":"2023-10-25T00:23:40.173944+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-25T00:23:40.173944+0000","last_undegraded":"2023-10-25T00:23:40.173944+0000","last_fullsized":"2023-10-25T00:23:40.173944+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4747","last_scrub_stamp":"2023-10-25T00:23:40.173729+0000","last_deep_scrub":"72'4747","last_deep_scrub_stamp":"2023-10-25T00:23:40.173729+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.173729+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:39:14.413546+0000","scrub_duration":0.31868588399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":25766,"num_write_kb":328702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"72'754","reported_seq":863,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:15.925869+0000","last_change":"2023-10-25T00:22:37.433948+0000","last_active":"2023-10-25T00:23:15.925869+0000","last_peered":"2023-10-25T00:23:15.925869+0000","last_clean":"2023-10-25T00:23:15.925869+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-25T00:23:15.925869+0000","last_undegraded":"2023-10-25T00:23:15.925869+0000","last_fullsized":"2023-10-25T00:23:15.925869+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'754","last_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_deep_scrub":"72'754","last_deep_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.433382+0000","objects_scrubbed":1,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:43.601946+0000","scrub_duration":0.10905978199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":77282,"num_write":740,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"74'53","reported_seq":185,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.687802+0000","last_change":"2023-10-25T00:23:37.477885+0000","last_active":"2023-10-25T00:24:35.687802+0000","last_peered":"2023-10-25T00:24:35.687802+0000","last_clean":"2023-10-25T00:24:35.687802+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-25T00:24:35.687802+0000","last_undegraded":"2023-10-25T00:24:35.687802+0000","last_fullsized":"2023-10-25T00:24:35.687802+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'51","last_scrub_stamp":"2023-10-25T00:23:37.477557+0000","last_deep_scrub":"73'51","last_deep_scrub_stamp":"2023-10-25T00:23:37.477557+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.477557+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:39.045780+0000","scrub_duration":0.008112401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'4792","reported_seq":4879,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.302620+0000","last_change":"2023-10-25T00:22:44.881494+0000","last_active":"2023-10-25T00:23:06.302620+0000","last_peered":"2023-10-25T00:23:06.302620+0000","last_clean":"2023-10-25T00:23:06.302620+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-25T00:23:06.302620+0000","last_undegraded":"2023-10-25T00:23:06.302620+0000","last_fullsized":"2023-10-25T00:23:06.302620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4792","last_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_deep_scrub":"68'4792","last_deep_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.881344+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:23:57.562121+0000","scrub_duration":0.193670646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24997,"num_write_kb":313272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"72'4739","reported_seq":4825,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:49.803089+0000","last_change":"2023-10-25T00:23:49.803089+0000","last_active":"2023-10-25T00:23:49.803089+0000","last_peered":"2023-10-25T00:23:49.803089+0000","last_clean":"2023-10-25T00:23:49.803089+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-25T00:23:49.803089+0000","last_undegraded":"2023-10-25T00:23:49.803089+0000","last_fullsized":"2023-10-25T00:23:49.803089+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4739","last_scrub_stamp":"2023-10-25T00:23:49.802870+0000","last_deep_scrub":"72'4739","last_deep_scrub_stamp":"2023-10-25T00:23:49.802870+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.802870+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:56:59.101994+0000","scrub_duration":0.33146283399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":22,"num_write":25639,"num_write_kb":332364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"73'358","reported_seq":444,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:04.146618+0000","last_change":"2023-10-25T00:24:04.146618+0000","last_active":"2023-10-25T00:24:04.146618+0000","last_peered":"2023-10-25T00:24:04.146618+0000","last_clean":"2023-10-25T00:24:04.146618+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-25T00:24:04.146618+0000","last_undegraded":"2023-10-25T00:24:04.146618+0000","last_fullsized":"2023-10-25T00:24:04.146618+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'358","last_scrub_stamp":"2023-10-25T00:24:04.146337+0000","last_deep_scrub":"73'358","last_deep_scrub_stamp":"2023-10-25T00:24:04.146337+0000","last_clean_scrub_stamp":"2023-10-25T00:24:04.146337+0000","objects_scrubbed":1,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:40:21.104237+0000","scrub_duration":0.154937403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2627766,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":227,"num_write":329,"num_write_kb":2759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"73'6","reported_seq":140,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:39.493290+0000","last_change":"2023-10-25T00:23:39.493290+0000","last_active":"2023-10-25T00:23:39.493290+0000","last_peered":"2023-10-25T00:23:39.493290+0000","last_clean":"2023-10-25T00:23:39.493290+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-25T00:23:39.493290+0000","last_undegraded":"2023-10-25T00:23:39.493290+0000","last_fullsized":"2023-10-25T00:23:39.493290+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2023-10-25T00:23:39.492964+0000","last_deep_scrub":"73'6","last_deep_scrub_stamp":"2023-10-25T00:23:39.492964+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.492964+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:34.589742+0000","scrub_duration":0.0071686809999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'3731","reported_seq":3816,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.302763+0000","last_change":"2023-10-25T00:22:37.622543+0000","last_active":"2023-10-25T00:23:06.302763+0000","last_peered":"2023-10-25T00:23:06.302763+0000","last_clean":"2023-10-25T00:23:06.302763+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-25T00:23:06.302763+0000","last_undegraded":"2023-10-25T00:23:06.302763+0000","last_fullsized":"2023-10-25T00:23:06.302763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3731","last_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_deep_scrub":"68'3731","last_deep_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.622304+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:51:52.171234+0000","scrub_duration":0.090187284000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19492,"num_write_kb":250091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"72'3799","reported_seq":3884,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:15.367485+0000","last_change":"2023-10-25T00:24:15.367485+0000","last_active":"2023-10-25T00:24:15.367485+0000","last_peered":"2023-10-25T00:24:15.367485+0000","last_clean":"2023-10-25T00:24:15.367485+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-25T00:24:15.367485+0000","last_undegraded":"2023-10-25T00:24:15.367485+0000","last_fullsized":"2023-10-25T00:24:15.367485+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3799","last_scrub_stamp":"2023-10-25T00:24:15.367271+0000","last_deep_scrub":"72'3799","last_deep_scrub_stamp":"2023-10-25T00:24:15.367271+0000","last_clean_scrub_stamp":"2023-10-25T00:24:15.367271+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:57:37.963942+0000","scrub_duration":0.31526191399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21581,"num_write_kb":262877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"73'46","reported_seq":172,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.323666+0000","last_change":"2023-10-25T00:22:52.664257+0000","last_active":"2023-10-25T00:23:16.323666+0000","last_peered":"2023-10-25T00:23:16.323666+0000","last_clean":"2023-10-25T00:23:16.323666+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-25T00:23:16.323666+0000","last_undegraded":"2023-10-25T00:23:16.323666+0000","last_fullsized":"2023-10-25T00:23:16.323666+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'46","last_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_deep_scrub":"73'46","last_deep_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.664064+0000","objects_scrubbed":1,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:44:09.108291+0000","scrub_duration":0.106079285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"73'715","reported_seq":831,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.317061+0000","last_change":"2023-10-25T00:23:21.967998+0000","last_active":"2023-10-25T00:24:34.317061+0000","last_peered":"2023-10-25T00:24:34.317061+0000","last_clean":"2023-10-25T00:24:34.317061+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-25T00:24:34.317061+0000","last_undegraded":"2023-10-25T00:24:34.317061+0000","last_fullsized":"2023-10-25T00:24:34.317061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'715","last_scrub_stamp":"2023-10-25T00:23:21.967694+0000","last_deep_scrub":"73'715","last_deep_scrub_stamp":"2023-10-25T00:23:21.967694+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.967694+0000","objects_scrubbed":1,"log_size":715,"log_dups_size":0,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:56:40.443702+0000","scrub_duration":0.065018907000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":94524,"num_write":703,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'4008","reported_seq":4097,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.814881+0000","last_change":"2023-10-25T00:23:05.488463+0000","last_active":"2023-10-25T00:23:05.814881+0000","last_peered":"2023-10-25T00:23:05.814881+0000","last_clean":"2023-10-25T00:23:05.814881+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-25T00:23:05.814881+0000","last_undegraded":"2023-10-25T00:23:05.814881+0000","last_fullsized":"2023-10-25T00:23:05.814881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4008","last_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_deep_scrub":"68'4008","last_deep_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.488248+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:51:16.627187+0000","scrub_duration":0.15813021899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20791,"num_write_kb":267770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"72'5469","reported_seq":5565,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.571401+0000","last_change":"2023-10-25T00:22:57.571401+0000","last_active":"2023-10-25T00:22:57.571401+0000","last_peered":"2023-10-25T00:22:57.571401+0000","last_clean":"2023-10-25T00:22:57.571401+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-25T00:22:57.571401+0000","last_undegraded":"2023-10-25T00:22:57.571401+0000","last_fullsized":"2023-10-25T00:22:57.571401+0000","mapping_epoch":25,"log_start":"64'200","ondisk_log_start":"64'200","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5469","last_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_deep_scrub":"72'5469","last_deep_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.571234+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:24:13.836466+0000","scrub_duration":0.19144520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":40,"num_write":30646,"num_write_kb":401481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"73'6","reported_seq":132,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.305295+0000","last_change":"2023-10-25T00:23:14.991436+0000","last_active":"2023-10-25T00:23:16.305295+0000","last_peered":"2023-10-25T00:23:16.305295+0000","last_clean":"2023-10-25T00:23:16.305295+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-25T00:23:16.305295+0000","last_undegraded":"2023-10-25T00:23:16.305295+0000","last_fullsized":"2023-10-25T00:23:16.305295+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_deep_scrub":"73'6","last_deep_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.991238+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:25:08.545014+0000","scrub_duration":0.0072276010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"74'548","reported_seq":647,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.137162+0000","last_change":"2023-10-25T00:24:31.287913+0000","last_active":"2023-10-25T00:24:35.137162+0000","last_peered":"2023-10-25T00:24:35.137162+0000","last_clean":"2023-10-25T00:24:35.137162+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-25T00:24:35.137162+0000","last_undegraded":"2023-10-25T00:24:35.137162+0000","last_fullsized":"2023-10-25T00:24:35.137162+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'547","last_scrub_stamp":"2023-10-25T00:24:31.287462+0000","last_deep_scrub":"73'547","last_deep_scrub_stamp":"2023-10-25T00:24:31.287462+0000","last_clean_scrub_stamp":"2023-10-25T00:24:31.287462+0000","objects_scrubbed":1,"log_size":548,"log_dups_size":0,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:31:39.750665+0000","scrub_duration":0.029858290999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3198201,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":13752,"num_write":517,"num_write_kb":3434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'6069","reported_seq":6157,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.535460+0000","last_change":"2023-10-25T00:22:46.608504+0000","last_active":"2023-10-25T00:23:07.535460+0000","last_peered":"2023-10-25T00:23:07.535460+0000","last_clean":"2023-10-25T00:23:07.535460+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-25T00:23:07.535460+0000","last_undegraded":"2023-10-25T00:23:07.535460+0000","last_fullsized":"2023-10-25T00:23:07.535460+0000","mapping_epoch":21,"log_start":"62'800","ondisk_log_start":"62'800","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6069","last_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_deep_scrub":"68'6069","last_deep_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.608198+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:37.101860+0000","scrub_duration":0.19917779799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":33187,"num_write_kb":387130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'3606","reported_seq":3695,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:38.186420+0000","last_change":"2023-10-25T00:23:38.186420+0000","last_active":"2023-10-25T00:23:38.186420+0000","last_peered":"2023-10-25T00:23:38.186420+0000","last_clean":"2023-10-25T00:23:38.186420+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-25T00:23:38.186420+0000","last_undegraded":"2023-10-25T00:23:38.186420+0000","last_fullsized":"2023-10-25T00:23:38.186420+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3606","last_scrub_stamp":"2023-10-25T00:23:38.186264+0000","last_deep_scrub":"72'3606","last_deep_scrub_stamp":"2023-10-25T00:23:38.186264+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.186264+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:15.619690+0000","scrub_duration":0.23143245800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19493,"num_write_kb":254342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'1064","reported_seq":1200,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:30.690441+0000","last_change":"2023-10-25T00:23:03.701535+0000","last_active":"2023-10-25T00:24:30.690441+0000","last_peered":"2023-10-25T00:24:30.690441+0000","last_clean":"2023-10-25T00:24:30.690441+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-25T00:24:30.690441+0000","last_undegraded":"2023-10-25T00:24:30.690441+0000","last_fullsized":"2023-10-25T00:24:30.690441+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1064","last_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_deep_scrub":"73'1064","last_deep_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_clean_scrub_stamp":"2023-10-25T00:23:03.701304+0000","objects_scrubbed":3,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:08:18.178955+0000","scrub_duration":0.28236572100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":9402,"num_write":1035,"num_write_kb":4717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'3","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.341650+0000","last_change":"2023-10-25T00:23:12.838945+0000","last_active":"2023-10-25T00:24:31.341650+0000","last_peered":"2023-10-25T00:24:31.341650+0000","last_clean":"2023-10-25T00:24:31.341650+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-25T00:24:31.341650+0000","last_undegraded":"2023-10-25T00:24:31.341650+0000","last_fullsized":"2023-10-25T00:24:31.341650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.838787+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:33:35.778931+0000","scrub_duration":0.133446072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'5139","reported_seq":5227,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:11.490339+0000","last_change":"2023-10-25T00:22:54.030976+0000","last_active":"2023-10-25T00:23:11.490339+0000","last_peered":"2023-10-25T00:23:11.490339+0000","last_clean":"2023-10-25T00:23:11.490339+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-25T00:23:11.490339+0000","last_undegraded":"2023-10-25T00:23:11.490339+0000","last_fullsized":"2023-10-25T00:23:11.490339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5139","last_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_deep_scrub":"68'5139","last_deep_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.030782+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:30.182861+0000","scrub_duration":0.103857617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27772,"num_write_kb":328469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"72'4843","reported_seq":4928,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:39.041980+0000","last_change":"2023-10-25T00:23:39.041980+0000","last_active":"2023-10-25T00:23:39.041980+0000","last_peered":"2023-10-25T00:23:39.041980+0000","last_clean":"2023-10-25T00:23:39.041980+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-25T00:23:39.041980+0000","last_undegraded":"2023-10-25T00:23:39.041980+0000","last_fullsized":"2023-10-25T00:23:39.041980+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4843","last_scrub_stamp":"2023-10-25T00:23:39.041623+0000","last_deep_scrub":"72'4843","last_deep_scrub_stamp":"2023-10-25T00:23:39.041623+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.041623+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:34:30.360915+0000","scrub_duration":0.33452469800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":27290,"num_write_kb":330906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"72'27","reported_seq":112,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.493119+0000","last_change":"2023-10-25T00:22:55.987117+0000","last_active":"2023-10-25T00:23:21.493119+0000","last_peered":"2023-10-25T00:23:21.493119+0000","last_clean":"2023-10-25T00:23:21.493119+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-25T00:23:21.493119+0000","last_undegraded":"2023-10-25T00:23:21.493119+0000","last_fullsized":"2023-10-25T00:23:21.493119+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'27","last_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_deep_scrub":"72'27","last_deep_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.986674+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:33.089164+0000","scrub_duration":0.100730242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'2","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.569646+0000","last_change":"2023-10-25T00:23:21.504086+0000","last_active":"2023-10-25T00:24:32.569646+0000","last_peered":"2023-10-25T00:24:32.569646+0000","last_clean":"2023-10-25T00:24:32.569646+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-25T00:24:32.569646+0000","last_undegraded":"2023-10-25T00:24:32.569646+0000","last_fullsized":"2023-10-25T00:24:32.569646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2023-10-25T00:23:21.503783+0000","last_deep_scrub":"67'2","last_deep_scrub_stamp":"2023-10-25T00:23:21.503783+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.503783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:13.653289+0000","scrub_duration":0.0039280959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'5814","reported_seq":5903,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:20.688378+0000","last_change":"2023-10-25T00:22:52.320696+0000","last_active":"2023-10-25T00:24:20.688378+0000","last_peered":"2023-10-25T00:24:20.688378+0000","last_clean":"2023-10-25T00:24:20.688378+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-25T00:24:20.688378+0000","last_undegraded":"2023-10-25T00:24:20.688378+0000","last_fullsized":"2023-10-25T00:24:20.688378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5814","last_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_deep_scrub":"68'5814","last_deep_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.320378+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:26.511615+0000","scrub_duration":0.033495337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31996,"num_write_kb":382397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"72'3827","reported_seq":3913,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.273982+0000","last_change":"2023-10-25T00:23:14.436458+0000","last_active":"2023-10-25T00:23:36.273982+0000","last_peered":"2023-10-25T00:23:36.273982+0000","last_clean":"2023-10-25T00:23:36.273982+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-25T00:23:36.273982+0000","last_undegraded":"2023-10-25T00:23:36.273982+0000","last_fullsized":"2023-10-25T00:23:36.273982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3827","last_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_deep_scrub":"72'3827","last_deep_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.436150+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:27:32.423180+0000","scrub_duration":0.30547422699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":21199,"num_write_kb":277730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"68'4820","reported_seq":4908,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:22.538415+0000","last_change":"2023-10-25T00:22:36.429763+0000","last_active":"2023-10-25T00:23:22.538415+0000","last_peered":"2023-10-25T00:23:22.538415+0000","last_clean":"2023-10-25T00:23:22.538415+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-25T00:23:22.538415+0000","last_undegraded":"2023-10-25T00:23:22.538415+0000","last_fullsized":"2023-10-25T00:23:22.538415+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4820","last_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_deep_scrub":"68'4820","last_deep_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_clean_scrub_stamp":"2023-10-25T00:22:36.429607+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:35.883183+0000","scrub_duration":0.10613655600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25163,"num_write_kb":285293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"72'5","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.510020+0000","last_change":"2023-10-25T00:23:20.947028+0000","last_active":"2023-10-25T00:24:31.510020+0000","last_peered":"2023-10-25T00:24:31.510020+0000","last_clean":"2023-10-25T00:24:31.510020+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-25T00:24:31.510020+0000","last_undegraded":"2023-10-25T00:24:31.510020+0000","last_fullsized":"2023-10-25T00:24:31.510020+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:23:20.946713+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:23:20.946713+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.946713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:12:40.769613+0000","scrub_duration":0.0045247550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"72'5","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.679346+0000","last_change":"2023-10-25T00:22:40.197550+0000","last_active":"2023-10-25T00:23:35.679346+0000","last_peered":"2023-10-25T00:23:35.679346+0000","last_clean":"2023-10-25T00:23:35.679346+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-25T00:23:35.679346+0000","last_undegraded":"2023-10-25T00:23:35.679346+0000","last_fullsized":"2023-10-25T00:23:35.679346+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.197213+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:07:28.509914+0000","scrub_duration":0.0047704979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"72'4957","reported_seq":5046,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.542830+0000","last_change":"2023-10-25T00:22:53.667906+0000","last_active":"2023-10-25T00:23:37.542830+0000","last_peered":"2023-10-25T00:23:37.542830+0000","last_clean":"2023-10-25T00:23:37.542830+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-25T00:23:37.542830+0000","last_undegraded":"2023-10-25T00:23:37.542830+0000","last_fullsized":"2023-10-25T00:23:37.542830+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4957","last_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_deep_scrub":"72'4957","last_deep_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.667748+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:58.307897+0000","scrub_duration":0.30812842099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28093,"num_write_kb":358545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'4782","reported_seq":4875,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.539783+0000","last_change":"2023-10-25T00:22:42.562198+0000","last_active":"2023-10-25T00:23:27.539783+0000","last_peered":"2023-10-25T00:23:27.539783+0000","last_clean":"2023-10-25T00:23:27.539783+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-25T00:23:27.539783+0000","last_undegraded":"2023-10-25T00:23:27.539783+0000","last_fullsized":"2023-10-25T00:23:27.539783+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4782","last_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_deep_scrub":"68'4782","last_deep_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.562057+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:39.415960+0000","scrub_duration":0.116720554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26432,"num_write_kb":298639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"68'3","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.110759+0000","last_change":"2023-10-25T00:23:36.434199+0000","last_active":"2023-10-25T00:24:31.110759+0000","last_peered":"2023-10-25T00:24:31.110759+0000","last_clean":"2023-10-25T00:24:31.110759+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-25T00:24:31.110759+0000","last_undegraded":"2023-10-25T00:24:31.110759+0000","last_fullsized":"2023-10-25T00:24:31.110759+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:36.433882+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:36.433882+0000","last_clean_scrub_stamp":"2023-10-25T00:23:36.433882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:00:44.210403+0000","scrub_duration":0.0040752599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"72'18","reported_seq":216,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.273751+0000","last_change":"2023-10-25T00:22:46.074083+0000","last_active":"2023-10-25T00:23:36.273751+0000","last_peered":"2023-10-25T00:23:36.273751+0000","last_clean":"2023-10-25T00:23:36.273751+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-25T00:23:36.273751+0000","last_undegraded":"2023-10-25T00:23:36.273751+0000","last_fullsized":"2023-10-25T00:23:36.273751+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_deep_scrub":"72'18","last_deep_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.073758+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:08:25.787090+0000","scrub_duration":0.012558368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":345,"num_read_kb":345,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"72'2954","reported_seq":3038,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.496407+0000","last_change":"2023-10-25T00:23:36.099045+0000","last_active":"2023-10-25T00:23:36.496407+0000","last_peered":"2023-10-25T00:23:36.496407+0000","last_clean":"2023-10-25T00:23:36.496407+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-25T00:23:36.496407+0000","last_undegraded":"2023-10-25T00:23:36.496407+0000","last_fullsized":"2023-10-25T00:23:36.496407+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2954","last_scrub_stamp":"2023-10-25T00:23:36.098826+0000","last_deep_scrub":"72'2954","last_deep_scrub_stamp":"2023-10-25T00:23:36.098826+0000","last_clean_scrub_stamp":"2023-10-25T00:23:36.098826+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:16.696480+0000","scrub_duration":0.32992982700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15961,"num_write_kb":219669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"73'38","reported_seq":172,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.316727+0000","last_change":"2023-10-25T00:23:35.052803+0000","last_active":"2023-10-25T00:24:34.316727+0000","last_peered":"2023-10-25T00:24:34.316727+0000","last_clean":"2023-10-25T00:24:34.316727+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-25T00:24:34.316727+0000","last_undegraded":"2023-10-25T00:24:34.316727+0000","last_fullsized":"2023-10-25T00:24:34.316727+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'38","last_scrub_stamp":"2023-10-25T00:23:35.052506+0000","last_deep_scrub":"73'38","last_deep_scrub_stamp":"2023-10-25T00:23:35.052506+0000","last_clean_scrub_stamp":"2023-10-25T00:23:35.052506+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:23:31.370767+0000","scrub_duration":0.0095825770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":93,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'4690","reported_seq":4779,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.325718+0000","last_change":"2023-10-25T00:23:05.792729+0000","last_active":"2023-10-25T00:23:26.325718+0000","last_peered":"2023-10-25T00:23:26.325718+0000","last_clean":"2023-10-25T00:23:26.325718+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-25T00:23:26.325718+0000","last_undegraded":"2023-10-25T00:23:26.325718+0000","last_fullsized":"2023-10-25T00:23:26.325718+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4690","last_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_deep_scrub":"68'4690","last_deep_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.792588+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:24:53.476824+0000","scrub_duration":0.15089322399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25172,"num_write_kb":305786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"72'10","reported_seq":137,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.542701+0000","last_change":"2023-10-25T00:22:40.512549+0000","last_active":"2023-10-25T00:23:37.542701+0000","last_peered":"2023-10-25T00:23:37.542701+0000","last_clean":"2023-10-25T00:23:37.542701+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-25T00:23:37.542701+0000","last_undegraded":"2023-10-25T00:23:37.542701+0000","last_fullsized":"2023-10-25T00:23:37.542701+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.512409+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:07:23.705506+0000","scrub_duration":0.091817919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"72'4200","reported_seq":4292,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.542502+0000","last_change":"2023-10-25T00:23:34.809510+0000","last_active":"2023-10-25T00:23:37.542502+0000","last_peered":"2023-10-25T00:23:37.542502+0000","last_clean":"2023-10-25T00:23:37.542502+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-25T00:23:37.542502+0000","last_undegraded":"2023-10-25T00:23:37.542502+0000","last_fullsized":"2023-10-25T00:23:37.542502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4200","last_scrub_stamp":"2023-10-25T00:23:34.809362+0000","last_deep_scrub":"72'4200","last_deep_scrub_stamp":"2023-10-25T00:23:34.809362+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.809362+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:04:30.830862+0000","scrub_duration":0.23894108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":24169,"num_write_kb":293796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"73'394","reported_seq":600,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.121743+0000","last_change":"2023-10-25T00:23:31.116025+0000","last_active":"2023-10-25T00:24:35.121743+0000","last_peered":"2023-10-25T00:24:35.121743+0000","last_clean":"2023-10-25T00:24:35.121743+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-25T00:24:35.121743+0000","last_undegraded":"2023-10-25T00:24:35.121743+0000","last_fullsized":"2023-10-25T00:24:35.121743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'394","last_scrub_stamp":"2023-10-25T00:23:31.115692+0000","last_deep_scrub":"73'394","last_deep_scrub_stamp":"2023-10-25T00:23:31.115692+0000","last_clean_scrub_stamp":"2023-10-25T00:23:31.115692+0000","objects_scrubbed":4,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:38.234847+0000","scrub_duration":0.026138159000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":213,"num_read_kb":98493,"num_write":387,"num_write_kb":4304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'5730","reported_seq":5818,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.204139+0000","last_change":"2023-10-25T00:23:30.204139+0000","last_active":"2023-10-25T00:23:30.204139+0000","last_peered":"2023-10-25T00:23:30.204139+0000","last_clean":"2023-10-25T00:23:30.204139+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-25T00:23:30.204139+0000","last_undegraded":"2023-10-25T00:23:30.204139+0000","last_fullsized":"2023-10-25T00:23:30.204139+0000","mapping_epoch":21,"log_start":"61'500","ondisk_log_start":"61'500","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5730","last_scrub_stamp":"2023-10-25T00:23:30.203981+0000","last_deep_scrub":"68'5730","last_deep_scrub_stamp":"2023-10-25T00:23:30.203981+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.203981+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:38:35.578976+0000","scrub_duration":0.10293530200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":31652,"num_write_kb":356481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"72'3","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.328042+0000","last_change":"2023-10-25T00:23:27.639911+0000","last_active":"2023-10-25T00:23:36.328042+0000","last_peered":"2023-10-25T00:23:36.328042+0000","last_clean":"2023-10-25T00:23:36.328042+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-25T00:23:36.328042+0000","last_undegraded":"2023-10-25T00:23:36.328042+0000","last_fullsized":"2023-10-25T00:23:36.328042+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:27.639604+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:27.639604+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.639604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:09:33.666294+0000","scrub_duration":0.0047758339999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"72'4168","reported_seq":4256,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.287459+0000","last_change":"2023-10-25T00:23:59.284060+0000","last_active":"2023-10-25T00:24:31.287459+0000","last_peered":"2023-10-25T00:24:31.287459+0000","last_clean":"2023-10-25T00:24:31.287459+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-25T00:24:31.287459+0000","last_undegraded":"2023-10-25T00:24:31.287459+0000","last_fullsized":"2023-10-25T00:24:31.287459+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4168","last_scrub_stamp":"2023-10-25T00:23:59.283890+0000","last_deep_scrub":"72'4168","last_deep_scrub_stamp":"2023-10-25T00:23:59.283890+0000","last_clean_scrub_stamp":"2023-10-25T00:23:59.283890+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:19:04.711112+0000","scrub_duration":0.25867606399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23458,"num_write_kb":294850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"68'4339","reported_seq":4428,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.427145+0000","last_change":"2023-10-25T00:23:47.099492+0000","last_active":"2023-10-25T00:24:21.427145+0000","last_peered":"2023-10-25T00:24:21.427145+0000","last_clean":"2023-10-25T00:24:21.427145+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-25T00:24:21.427145+0000","last_undegraded":"2023-10-25T00:24:21.427145+0000","last_fullsized":"2023-10-25T00:24:21.427145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4339","last_scrub_stamp":"2023-10-25T00:23:47.099343+0000","last_deep_scrub":"68'4339","last_deep_scrub_stamp":"2023-10-25T00:23:47.099343+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.099343+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:27.913875+0000","scrub_duration":0.12538819900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23267,"num_write_kb":265268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.676577+0000","last_change":"2023-10-25T00:22:42.199286+0000","last_active":"2023-10-25T00:23:20.676577+0000","last_peered":"2023-10-25T00:23:20.676577+0000","last_clean":"2023-10-25T00:23:20.676577+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-25T00:23:20.676577+0000","last_undegraded":"2023-10-25T00:23:20.676577+0000","last_fullsized":"2023-10-25T00:23:20.676577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.198947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:11:17.807377+0000","scrub_duration":0.0041955760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"73'231","reported_seq":344,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.316986+0000","last_change":"2023-10-25T00:23:16.010034+0000","last_active":"2023-10-25T00:24:34.316986+0000","last_peered":"2023-10-25T00:24:34.316986+0000","last_clean":"2023-10-25T00:24:34.316986+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-25T00:24:34.316986+0000","last_undegraded":"2023-10-25T00:24:34.316986+0000","last_fullsized":"2023-10-25T00:24:34.316986+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'231","last_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_deep_scrub":"73'231","last_deep_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.009842+0000","objects_scrubbed":1,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:27:21.881029+0000","scrub_duration":0.076880452000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":65,"num_write":590,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"72'11","reported_seq":96,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:17.537123+0000","last_change":"2023-10-25T00:22:38.409520+0000","last_active":"2023-10-25T00:23:17.537123+0000","last_peered":"2023-10-25T00:23:17.537123+0000","last_clean":"2023-10-25T00:23:17.537123+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-25T00:23:17.537123+0000","last_undegraded":"2023-10-25T00:23:17.537123+0000","last_fullsized":"2023-10-25T00:23:17.537123+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_deep_scrub":"72'11","last_deep_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.409235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:21:24.708709+0000","scrub_duration":0.003592329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"72'3993","reported_seq":4082,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.429639+0000","last_change":"2023-10-25T00:23:58.292855+0000","last_active":"2023-10-25T00:24:31.429639+0000","last_peered":"2023-10-25T00:24:31.429639+0000","last_clean":"2023-10-25T00:24:31.429639+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-25T00:24:31.429639+0000","last_undegraded":"2023-10-25T00:24:31.429639+0000","last_fullsized":"2023-10-25T00:24:31.429639+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3993","last_scrub_stamp":"2023-10-25T00:23:58.292710+0000","last_deep_scrub":"72'3993","last_deep_scrub_stamp":"2023-10-25T00:23:58.292710+0000","last_clean_scrub_stamp":"2023-10-25T00:23:58.292710+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:23:08.798165+0000","scrub_duration":0.310815756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21723,"num_write_kb":283850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'4270","reported_seq":4359,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.535306+0000","last_change":"2023-10-25T00:22:59.570791+0000","last_active":"2023-10-25T00:23:07.535306+0000","last_peered":"2023-10-25T00:23:07.535306+0000","last_clean":"2023-10-25T00:23:07.535306+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-25T00:23:07.535306+0000","last_undegraded":"2023-10-25T00:23:07.535306+0000","last_fullsized":"2023-10-25T00:23:07.535306+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4270","last_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_deep_scrub":"68'4270","last_deep_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.570628+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:36:56.294621+0000","scrub_duration":0.14092118100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23049,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.324802+0000","last_change":"2023-10-25T00:23:06.643510+0000","last_active":"2023-10-25T00:23:21.324802+0000","last_peered":"2023-10-25T00:23:21.324802+0000","last_clean":"2023-10-25T00:23:21.324802+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-25T00:23:21.324802+0000","last_undegraded":"2023-10-25T00:23:21.324802+0000","last_fullsized":"2023-10-25T00:23:21.324802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.643156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:34:24.773001+0000","scrub_duration":0.0045495850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"73'10","reported_seq":146,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.540370+0000","last_change":"2023-10-25T00:23:20.618489+0000","last_active":"2023-10-25T00:23:27.540370+0000","last_peered":"2023-10-25T00:23:27.540370+0000","last_clean":"2023-10-25T00:23:27.540370+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-25T00:23:27.540370+0000","last_undegraded":"2023-10-25T00:23:27.540370+0000","last_fullsized":"2023-10-25T00:23:27.540370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2023-10-25T00:23:20.618371+0000","last_deep_scrub":"73'10","last_deep_scrub_stamp":"2023-10-25T00:23:20.618371+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.618371+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:30:10.455022+0000","scrub_duration":0.085107656000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":8,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"73'65","reported_seq":230,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.429515+0000","last_change":"2023-10-25T00:23:07.930465+0000","last_active":"2023-10-25T00:24:31.429515+0000","last_peered":"2023-10-25T00:24:31.429515+0000","last_clean":"2023-10-25T00:24:31.429515+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-25T00:24:31.429515+0000","last_undegraded":"2023-10-25T00:24:31.429515+0000","last_fullsized":"2023-10-25T00:24:31.429515+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'65","last_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_deep_scrub":"73'65","last_deep_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.930143+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:41:49.528670+0000","scrub_duration":0.020886851000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":123,"num_write":59,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"72'3769","reported_seq":3858,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.341574+0000","last_change":"2023-10-25T00:23:38.050117+0000","last_active":"2023-10-25T00:24:31.341574+0000","last_peered":"2023-10-25T00:24:31.341574+0000","last_clean":"2023-10-25T00:24:31.341574+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-25T00:24:31.341574+0000","last_undegraded":"2023-10-25T00:24:31.341574+0000","last_fullsized":"2023-10-25T00:24:31.341574+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3769","last_scrub_stamp":"2023-10-25T00:23:38.049889+0000","last_deep_scrub":"72'3769","last_deep_scrub_stamp":"2023-10-25T00:23:38.049889+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.049889+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:23:14.574715+0000","scrub_duration":0.37571605800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20807,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'5752","reported_seq":5841,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.507272+0000","last_change":"2023-10-25T00:22:33.937946+0000","last_active":"2023-10-25T00:24:21.507272+0000","last_peered":"2023-10-25T00:24:21.507272+0000","last_clean":"2023-10-25T00:24:21.507272+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-25T00:24:21.507272+0000","last_undegraded":"2023-10-25T00:24:21.507272+0000","last_fullsized":"2023-10-25T00:24:21.507272+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5752","last_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_deep_scrub":"68'5752","last_deep_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_clean_scrub_stamp":"2023-10-25T00:22:33.937564+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:43:34.779084+0000","scrub_duration":0.080678784000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31231,"num_write_kb":354718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:16.283897+0000","last_change":"2023-10-25T00:23:24.047030+0000","last_active":"2023-10-25T00:24:16.283897+0000","last_peered":"2023-10-25T00:24:16.283897+0000","last_clean":"2023-10-25T00:24:16.283897+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-25T00:24:16.283897+0000","last_undegraded":"2023-10-25T00:24:16.283897+0000","last_fullsized":"2023-10-25T00:24:16.283897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:24.046716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:24.046716+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.046716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:53:42.506468+0000","scrub_duration":0.0043444529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'2","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.339157+0000","last_change":"2023-10-25T00:23:18.684046+0000","last_active":"2023-10-25T00:24:21.339157+0000","last_peered":"2023-10-25T00:24:21.339157+0000","last_clean":"2023-10-25T00:24:21.339157+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-25T00:24:21.339157+0000","last_undegraded":"2023-10-25T00:24:21.339157+0000","last_fullsized":"2023-10-25T00:24:21.339157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2023-10-25T00:23:18.683799+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2023-10-25T00:23:18.683799+0000","last_clean_scrub_stamp":"2023-10-25T00:23:18.683799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:26:44.716117+0000","scrub_duration":0.013956036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"72'4","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:30.690283+0000","last_change":"2023-10-25T00:23:06.496904+0000","last_active":"2023-10-25T00:24:30.690283+0000","last_peered":"2023-10-25T00:24:30.690283+0000","last_clean":"2023-10-25T00:24:30.690283+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-25T00:24:30.690283+0000","last_undegraded":"2023-10-25T00:24:30.690283+0000","last_fullsized":"2023-10-25T00:24:30.690283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_deep_scrub":"72'4","last_deep_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.496575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:32:18.665160+0000","scrub_duration":0.0046312189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"72'3939","reported_seq":4023,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.492760+0000","last_change":"2023-10-25T00:23:02.160279+0000","last_active":"2023-10-25T00:23:21.492760+0000","last_peered":"2023-10-25T00:23:21.492760+0000","last_clean":"2023-10-25T00:23:21.492760+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-25T00:23:21.492760+0000","last_undegraded":"2023-10-25T00:23:21.492760+0000","last_fullsized":"2023-10-25T00:23:21.492760+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3939","last_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_deep_scrub":"72'3939","last_deep_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.160086+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:37:29.878702+0000","scrub_duration":0.252775894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22393,"num_write_kb":262004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'4305","reported_seq":4394,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.339050+0000","last_change":"2023-10-25T00:23:26.858862+0000","last_active":"2023-10-25T00:24:21.339050+0000","last_peered":"2023-10-25T00:24:21.339050+0000","last_clean":"2023-10-25T00:24:21.339050+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-25T00:24:21.339050+0000","last_undegraded":"2023-10-25T00:24:21.339050+0000","last_fullsized":"2023-10-25T00:24:21.339050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4305","last_scrub_stamp":"2023-10-25T00:23:26.858657+0000","last_deep_scrub":"68'4305","last_deep_scrub_stamp":"2023-10-25T00:23:26.858657+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.858657+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:10:03.352691+0000","scrub_duration":0.18050687100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22696,"num_write_kb":287574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:16.338023+0000","last_change":"2023-10-25T00:23:11.717353+0000","last_active":"2023-10-25T00:24:16.338023+0000","last_peered":"2023-10-25T00:24:16.338023+0000","last_clean":"2023-10-25T00:24:16.338023+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-25T00:24:16.338023+0000","last_undegraded":"2023-10-25T00:24:16.338023+0000","last_fullsized":"2023-10-25T00:24:16.338023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_clean_scrub_stamp":"2023-10-25T00:23:11.717027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:17:42.982457+0000","scrub_duration":0.0039434270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:26.109514+0000","last_change":"2023-10-25T00:23:30.355099+0000","last_active":"2023-10-25T00:24:26.109514+0000","last_peered":"2023-10-25T00:24:26.109514+0000","last_clean":"2023-10-25T00:24:26.109514+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-25T00:24:26.109514+0000","last_undegraded":"2023-10-25T00:24:26.109514+0000","last_fullsized":"2023-10-25T00:24:26.109514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2023-10-25T00:23:30.354780+0000","last_deep_scrub":"67'2","last_deep_scrub_stamp":"2023-10-25T00:23:30.354780+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.354780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:07:40.743916+0000","scrub_duration":0.0044225610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"72'3","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.509696+0000","last_change":"2023-10-25T00:22:52.952897+0000","last_active":"2023-10-25T00:24:31.509696+0000","last_peered":"2023-10-25T00:24:31.509696+0000","last_clean":"2023-10-25T00:24:31.509696+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-25T00:24:31.509696+0000","last_undegraded":"2023-10-25T00:24:31.509696+0000","last_fullsized":"2023-10-25T00:24:31.509696+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.952595+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:14:20.890322+0000","scrub_duration":0.0044650949999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"72'5003","reported_seq":5093,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:44.774832+0000","last_change":"2023-10-25T00:23:44.774832+0000","last_active":"2023-10-25T00:23:44.774832+0000","last_peered":"2023-10-25T00:23:44.774832+0000","last_clean":"2023-10-25T00:23:44.774832+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-25T00:23:44.774832+0000","last_undegraded":"2023-10-25T00:23:44.774832+0000","last_fullsized":"2023-10-25T00:23:44.774832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5003","last_scrub_stamp":"2023-10-25T00:23:44.774508+0000","last_deep_scrub":"72'5003","last_deep_scrub_stamp":"2023-10-25T00:23:44.774508+0000","last_clean_scrub_stamp":"2023-10-25T00:23:44.774508+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:52.739672+0000","scrub_duration":0.33716653400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":13,"num_write":27120,"num_write_kb":350355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'5635","reported_seq":5723,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.339129+0000","last_change":"2023-10-25T00:22:44.547139+0000","last_active":"2023-10-25T00:24:21.339129+0000","last_peered":"2023-10-25T00:24:21.339129+0000","last_clean":"2023-10-25T00:24:21.339129+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-25T00:24:21.339129+0000","last_undegraded":"2023-10-25T00:24:21.339129+0000","last_fullsized":"2023-10-25T00:24:21.339129+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5635","last_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_deep_scrub":"68'5635","last_deep_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.546984+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:51:54.474684+0000","scrub_duration":0.15865177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30556,"num_write_kb":356140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"68'3","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:25.100690+0000","last_change":"2023-10-25T00:24:25.100690+0000","last_active":"2023-10-25T00:24:25.100690+0000","last_peered":"2023-10-25T00:24:25.100690+0000","last_clean":"2023-10-25T00:24:25.100690+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-25T00:24:25.100690+0000","last_undegraded":"2023-10-25T00:24:25.100690+0000","last_fullsized":"2023-10-25T00:24:25.100690+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:24:25.100343+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:24:25.100343+0000","last_clean_scrub_stamp":"2023-10-25T00:24:25.100343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:00:54.991882+0000","scrub_duration":0.0050738520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:16.107164+0000","last_change":"2023-10-25T00:23:29.305381+0000","last_active":"2023-10-25T00:24:16.107164+0000","last_peered":"2023-10-25T00:24:16.107164+0000","last_clean":"2023-10-25T00:24:16.107164+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-25T00:24:16.107164+0000","last_undegraded":"2023-10-25T00:24:16.107164+0000","last_fullsized":"2023-10-25T00:24:16.107164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:29.305022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:29.305022+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.305022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:52:12.477955+0000","scrub_duration":0.0044947839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"72'2","reported_seq":87,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315751+0000","last_change":"2023-10-25T00:22:58.925685+0000","last_active":"2023-10-25T00:24:34.315751+0000","last_peered":"2023-10-25T00:24:34.315751+0000","last_clean":"2023-10-25T00:24:34.315751+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-25T00:24:34.315751+0000","last_undegraded":"2023-10-25T00:24:34.315751+0000","last_fullsized":"2023-10-25T00:24:34.315751+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.925318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:31:45.562398+0000","scrub_duration":0.0043101720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"72'3753","reported_seq":3841,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.287558+0000","last_change":"2023-10-25T00:23:44.352225+0000","last_active":"2023-10-25T00:24:31.287558+0000","last_peered":"2023-10-25T00:24:31.287558+0000","last_clean":"2023-10-25T00:24:31.287558+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-25T00:24:31.287558+0000","last_undegraded":"2023-10-25T00:24:31.287558+0000","last_fullsized":"2023-10-25T00:24:31.287558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3753","last_scrub_stamp":"2023-10-25T00:23:44.352083+0000","last_deep_scrub":"72'3753","last_deep_scrub_stamp":"2023-10-25T00:23:44.352083+0000","last_clean_scrub_stamp":"2023-10-25T00:23:44.352083+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:13:26.541612+0000","scrub_duration":0.328578174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":21045,"num_write_kb":276986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'4101","reported_seq":4188,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.325983+0000","last_change":"2023-10-25T00:23:25.811837+0000","last_active":"2023-10-25T00:23:26.325983+0000","last_peered":"2023-10-25T00:23:26.325983+0000","last_clean":"2023-10-25T00:23:26.325983+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-25T00:23:26.325983+0000","last_undegraded":"2023-10-25T00:23:26.325983+0000","last_fullsized":"2023-10-25T00:23:26.325983+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:23:25.811682+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:23:25.811682+0000","last_clean_scrub_stamp":"2023-10-25T00:23:25.811682+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:39:30.752063+0000","scrub_duration":0.14318265299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22383,"num_write_kb":270023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"74'14","reported_seq":103,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.322235+0000","last_change":"2023-10-25T00:23:17.651450+0000","last_active":"2023-10-25T00:24:35.322235+0000","last_peered":"2023-10-25T00:24:35.322235+0000","last_clean":"2023-10-25T00:24:35.322235+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-25T00:24:35.322235+0000","last_undegraded":"2023-10-25T00:24:35.322235+0000","last_fullsized":"2023-10-25T00:24:35.322235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2023-10-25T00:23:17.651103+0000","last_deep_scrub":"73'13","last_deep_scrub_stamp":"2023-10-25T00:23:17.651103+0000","last_clean_scrub_stamp":"2023-10-25T00:23:17.651103+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:54.986553+0000","scrub_duration":0.014684019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:15.836497+0000","last_change":"2023-10-25T00:22:51.317813+0000","last_active":"2023-10-25T00:24:15.836497+0000","last_peered":"2023-10-25T00:24:15.836497+0000","last_clean":"2023-10-25T00:24:15.836497+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-25T00:24:15.836497+0000","last_undegraded":"2023-10-25T00:24:15.836497+0000","last_fullsized":"2023-10-25T00:24:15.836497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.317462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:03:14.150576+0000","scrub_duration":0.0044508079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"73'249","reported_seq":355,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.820915+0000","last_change":"2023-10-25T00:23:21.397705+0000","last_active":"2023-10-25T00:23:35.820915+0000","last_peered":"2023-10-25T00:23:35.820915+0000","last_clean":"2023-10-25T00:23:35.820915+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-25T00:23:35.820915+0000","last_undegraded":"2023-10-25T00:23:35.820915+0000","last_fullsized":"2023-10-25T00:23:35.820915+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'249","last_scrub_stamp":"2023-10-25T00:23:21.397410+0000","last_deep_scrub":"73'249","last_deep_scrub_stamp":"2023-10-25T00:23:21.397410+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.397410+0000","objects_scrubbed":1,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:19:14.548554+0000","scrub_duration":0.017693341000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":62,"num_write":607,"num_write_kb":223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"72'5236","reported_seq":5326,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.341814+0000","last_change":"2023-10-25T00:23:34.900038+0000","last_active":"2023-10-25T00:24:31.341814+0000","last_peered":"2023-10-25T00:24:31.341814+0000","last_clean":"2023-10-25T00:24:31.341814+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-25T00:24:31.341814+0000","last_undegraded":"2023-10-25T00:24:31.341814+0000","last_fullsized":"2023-10-25T00:24:31.341814+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5236","last_scrub_stamp":"2023-10-25T00:23:34.899894+0000","last_deep_scrub":"72'5236","last_deep_scrub_stamp":"2023-10-25T00:23:34.899894+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.899894+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:10:20.497931+0000","scrub_duration":0.29917875999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":29554,"num_write_kb":368820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'5444","reported_seq":5535,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.494343+0000","last_change":"2023-10-25T00:23:00.141618+0000","last_active":"2023-10-25T00:23:26.494343+0000","last_peered":"2023-10-25T00:23:26.494343+0000","last_clean":"2023-10-25T00:23:26.494343+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-25T00:23:26.494343+0000","last_undegraded":"2023-10-25T00:23:26.494343+0000","last_fullsized":"2023-10-25T00:23:26.494343+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5444","last_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_deep_scrub":"68'5444","last_deep_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.141458+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:23:18.614380+0000","scrub_duration":0.197741424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28342,"num_write_kb":342931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"73'67","reported_seq":241,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315086+0000","last_change":"2023-10-25T00:23:14.941868+0000","last_active":"2023-10-25T00:24:34.315086+0000","last_peered":"2023-10-25T00:24:34.315086+0000","last_clean":"2023-10-25T00:24:34.315086+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-25T00:24:34.315086+0000","last_undegraded":"2023-10-25T00:24:34.315086+0000","last_fullsized":"2023-10-25T00:24:34.315086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'67","last_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_deep_scrub":"73'67","last_deep_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.941564+0000","objects_scrubbed":1,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:11:54.084201+0000","scrub_duration":0.0084463549999999991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":125,"num_write":61,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:22.074393+0000","last_change":"2023-10-25T00:23:22.074393+0000","last_active":"2023-10-25T00:23:22.074393+0000","last_peered":"2023-10-25T00:23:22.074393+0000","last_clean":"2023-10-25T00:23:22.074393+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-25T00:23:22.074393+0000","last_undegraded":"2023-10-25T00:23:22.074393+0000","last_fullsized":"2023-10-25T00:23:22.074393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:22.074087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:22.074087+0000","last_clean_scrub_stamp":"2023-10-25T00:23:22.074087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:59:48.682336+0000","scrub_duration":0.0044663139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"72'10","reported_seq":139,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.097599+0000","last_change":"2023-10-25T00:22:45.425308+0000","last_active":"2023-10-25T00:23:36.097599+0000","last_peered":"2023-10-25T00:23:36.097599+0000","last_clean":"2023-10-25T00:23:36.097599+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-25T00:23:36.097599+0000","last_undegraded":"2023-10-25T00:23:36.097599+0000","last_fullsized":"2023-10-25T00:23:36.097599+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.425052+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:36:38.527058+0000","scrub_duration":0.14014134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":135,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"72'3666","reported_seq":3750,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.807972+0000","last_change":"2023-10-25T00:23:45.807972+0000","last_active":"2023-10-25T00:23:45.807972+0000","last_peered":"2023-10-25T00:23:45.807972+0000","last_clean":"2023-10-25T00:23:45.807972+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-25T00:23:45.807972+0000","last_undegraded":"2023-10-25T00:23:45.807972+0000","last_fullsized":"2023-10-25T00:23:45.807972+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3666","last_scrub_stamp":"2023-10-25T00:23:45.807770+0000","last_deep_scrub":"72'3666","last_deep_scrub_stamp":"2023-10-25T00:23:45.807770+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.807770+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:26:55.156976+0000","scrub_duration":0.28592742900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20519,"num_write_kb":254391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"68'4182","reported_seq":4271,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:25.689389+0000","last_change":"2023-10-25T00:23:02.489861+0000","last_active":"2023-10-25T00:24:25.689389+0000","last_peered":"2023-10-25T00:24:25.689389+0000","last_clean":"2023-10-25T00:24:25.689389+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-25T00:24:25.689389+0000","last_undegraded":"2023-10-25T00:24:25.689389+0000","last_fullsized":"2023-10-25T00:24:25.689389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4182","last_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_deep_scrub":"68'4182","last_deep_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.489471+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:14:08.666419+0000","scrub_duration":0.092197608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22839,"num_write_kb":259586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'4","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:22.567237+0000","last_change":"2023-10-25T00:23:15.544275+0000","last_active":"2023-10-25T00:24:22.567237+0000","last_peered":"2023-10-25T00:24:22.567237+0000","last_clean":"2023-10-25T00:24:22.567237+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-25T00:24:22.567237+0000","last_undegraded":"2023-10-25T00:24:22.567237+0000","last_fullsized":"2023-10-25T00:24:22.567237+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_deep_scrub":"67'4","last_deep_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:15.543988+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:33:51.332032+0000","scrub_duration":0.004459345,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:16.107245+0000","last_change":"2023-10-25T00:23:28.260731+0000","last_active":"2023-10-25T00:24:16.107245+0000","last_peered":"2023-10-25T00:24:16.107245+0000","last_clean":"2023-10-25T00:24:16.107245+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-25T00:24:16.107245+0000","last_undegraded":"2023-10-25T00:24:16.107245+0000","last_fullsized":"2023-10-25T00:24:16.107245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:28.260409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:28.260409+0000","last_clean_scrub_stamp":"2023-10-25T00:23:28.260409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:10:44.862291+0000","scrub_duration":0.0042298140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'163","reported_seq":300,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:17.149791+0000","last_change":"2023-10-25T00:23:17.149791+0000","last_active":"2023-10-25T00:23:17.149791+0000","last_peered":"2023-10-25T00:23:17.149791+0000","last_clean":"2023-10-25T00:23:17.149791+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-25T00:23:17.149791+0000","last_undegraded":"2023-10-25T00:23:17.149791+0000","last_fullsized":"2023-10-25T00:23:17.149791+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'163","last_scrub_stamp":"2023-10-25T00:23:17.149492+0000","last_deep_scrub":"73'163","last_deep_scrub_stamp":"2023-10-25T00:23:17.149492+0000","last_clean_scrub_stamp":"2023-10-25T00:23:17.149492+0000","objects_scrubbed":2,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:59:05.394868+0000","scrub_duration":0.012985958000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"72'2","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.287521+0000","last_change":"2023-10-25T00:22:48.067150+0000","last_active":"2023-10-25T00:24:31.287521+0000","last_peered":"2023-10-25T00:24:31.287521+0000","last_clean":"2023-10-25T00:24:31.287521+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-25T00:24:31.287521+0000","last_undegraded":"2023-10-25T00:24:31.287521+0000","last_fullsized":"2023-10-25T00:24:31.287521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.066810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:55:20.112674+0000","scrub_duration":0.0050122709999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"72'4105","reported_seq":4192,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.535530+0000","last_change":"2023-10-25T00:23:20.535530+0000","last_active":"2023-10-25T00:23:20.535530+0000","last_peered":"2023-10-25T00:23:20.535530+0000","last_clean":"2023-10-25T00:23:20.535530+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-25T00:23:20.535530+0000","last_undegraded":"2023-10-25T00:23:20.535530+0000","last_fullsized":"2023-10-25T00:23:20.535530+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4105","last_scrub_stamp":"2023-10-25T00:23:20.535378+0000","last_deep_scrub":"72'4105","last_deep_scrub_stamp":"2023-10-25T00:23:20.535378+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.535378+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:20:52.287445+0000","scrub_duration":0.31792611900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":19,"num_write":23147,"num_write_kb":297959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'5705","reported_seq":5795,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.324915+0000","last_change":"2023-10-25T00:23:39.836379+0000","last_active":"2023-10-25T00:24:35.324915+0000","last_peered":"2023-10-25T00:24:35.324915+0000","last_clean":"2023-10-25T00:24:35.324915+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-25T00:24:35.324915+0000","last_undegraded":"2023-10-25T00:24:35.324915+0000","last_fullsized":"2023-10-25T00:24:35.324915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5705","last_scrub_stamp":"2023-10-25T00:23:39.836205+0000","last_deep_scrub":"68'5705","last_deep_scrub_stamp":"2023-10-25T00:23:39.836205+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.836205+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:38:02.740862+0000","scrub_duration":0.105332388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":29845,"num_write_kb":357286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"68'5","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.539642+0000","last_change":"2023-10-25T00:23:18.535332+0000","last_active":"2023-10-25T00:23:27.539642+0000","last_peered":"2023-10-25T00:23:27.539642+0000","last_clean":"2023-10-25T00:23:27.539642+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-25T00:23:27.539642+0000","last_undegraded":"2023-10-25T00:23:27.539642+0000","last_fullsized":"2023-10-25T00:23:27.539642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:18.534951+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:18.534951+0000","last_clean_scrub_stamp":"2023-10-25T00:23:18.534951+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:37.756970+0000","scrub_duration":0.0037857049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":138,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.270870+0000","last_change":"2023-10-25T00:24:35.271123+0000","last_active":"2023-10-25T00:24:35.270870+0000","last_peered":"2023-10-25T00:24:35.270870+0000","last_clean":"2023-10-25T00:24:35.270870+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-25T00:24:35.270870+0000","last_undegraded":"2023-10-25T00:24:35.270870+0000","last_fullsized":"2023-10-25T00:24:35.270870+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2023-10-25T00:24:35.270780+0000","last_deep_scrub":"74'8","last_deep_scrub_stamp":"2023-10-25T00:24:35.270780+0000","last_clean_scrub_stamp":"2023-10-25T00:24:35.270780+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:15:45.057328+0000","scrub_duration":0.0084190350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"72'4234","reported_seq":4320,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:41.751679+0000","last_change":"2023-10-25T00:23:41.751679+0000","last_active":"2023-10-25T00:23:41.751679+0000","last_peered":"2023-10-25T00:23:41.751679+0000","last_clean":"2023-10-25T00:23:41.751679+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-25T00:23:41.751679+0000","last_undegraded":"2023-10-25T00:23:41.751679+0000","last_fullsized":"2023-10-25T00:23:41.751679+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4234","last_scrub_stamp":"2023-10-25T00:23:41.751558+0000","last_deep_scrub":"72'4234","last_deep_scrub_stamp":"2023-10-25T00:23:41.751558+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.751558+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:50:10.142078+0000","scrub_duration":0.256161054,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":15,"num_write":23173,"num_write_kb":302022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"68'4052","reported_seq":4140,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.288416+0000","last_change":"2023-10-25T00:23:49.042663+0000","last_active":"2023-10-25T00:24:34.288416+0000","last_peered":"2023-10-25T00:24:34.288416+0000","last_clean":"2023-10-25T00:24:34.288416+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-25T00:24:34.288416+0000","last_undegraded":"2023-10-25T00:24:34.288416+0000","last_fullsized":"2023-10-25T00:24:34.288416+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4052","last_scrub_stamp":"2023-10-25T00:23:49.042529+0000","last_deep_scrub":"68'4052","last_deep_scrub_stamp":"2023-10-25T00:23:49.042529+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.042529+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:40:32.211760+0000","scrub_duration":0.105715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21486,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"68'2","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.047108+0000","last_change":"2023-10-25T00:23:34.047108+0000","last_active":"2023-10-25T00:23:34.047108+0000","last_peered":"2023-10-25T00:23:34.047108+0000","last_clean":"2023-10-25T00:23:34.047108+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-25T00:23:34.047108+0000","last_undegraded":"2023-10-25T00:23:34.047108+0000","last_fullsized":"2023-10-25T00:23:34.047108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2023-10-25T00:23:34.046917+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2023-10-25T00:23:34.046917+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.046917+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:50:34.912538+0000","scrub_duration":0.034858717999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"73'5","reported_seq":110,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:53.546332+0000","last_change":"2023-10-25T00:22:53.546332+0000","last_active":"2023-10-25T00:22:53.546332+0000","last_peered":"2023-10-25T00:22:53.546332+0000","last_clean":"2023-10-25T00:22:53.546332+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-25T00:22:53.546332+0000","last_undegraded":"2023-10-25T00:22:53.546332+0000","last_fullsized":"2023-10-25T00:22:53.546332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.546215+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:16:55.756192+0000","scrub_duration":0.27862564699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"72'4690","reported_seq":4776,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:42.556703+0000","last_change":"2023-10-25T00:23:42.556703+0000","last_active":"2023-10-25T00:23:42.556703+0000","last_peered":"2023-10-25T00:23:42.556703+0000","last_clean":"2023-10-25T00:23:42.556703+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-25T00:23:42.556703+0000","last_undegraded":"2023-10-25T00:23:42.556703+0000","last_fullsized":"2023-10-25T00:23:42.556703+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4690","last_scrub_stamp":"2023-10-25T00:23:42.556538+0000","last_deep_scrub":"72'4690","last_deep_scrub_stamp":"2023-10-25T00:23:42.556538+0000","last_clean_scrub_stamp":"2023-10-25T00:23:42.556538+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:36:54.839713+0000","scrub_duration":0.199820896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24994,"num_write_kb":341798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'4780","reported_seq":4870,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.286287+0000","last_change":"2023-10-25T00:23:09.093139+0000","last_active":"2023-10-25T00:24:34.286287+0000","last_peered":"2023-10-25T00:24:34.286287+0000","last_clean":"2023-10-25T00:24:34.286287+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-25T00:24:34.286287+0000","last_undegraded":"2023-10-25T00:24:34.286287+0000","last_fullsized":"2023-10-25T00:24:34.286287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4780","last_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_deep_scrub":"68'4780","last_deep_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.092796+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:58:49.347709+0000","scrub_duration":0.152359684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25083,"num_write_kb":293163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'3","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.678451+0000","last_change":"2023-10-25T00:22:48.328017+0000","last_active":"2023-10-25T00:23:30.678451+0000","last_peered":"2023-10-25T00:23:30.678451+0000","last_clean":"2023-10-25T00:23:30.678451+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-25T00:23:30.678451+0000","last_undegraded":"2023-10-25T00:23:30.678451+0000","last_fullsized":"2023-10-25T00:23:30.678451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.327778+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:58.116941+0000","scrub_duration":0.061328572999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"72'5","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:55.576688+0000","last_change":"2023-10-25T00:22:55.576688+0000","last_active":"2023-10-25T00:22:55.576688+0000","last_peered":"2023-10-25T00:22:55.576688+0000","last_clean":"2023-10-25T00:22:55.576688+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-25T00:22:55.576688+0000","last_undegraded":"2023-10-25T00:22:55.576688+0000","last_fullsized":"2023-10-25T00:22:55.576688+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.576324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:43.752897+0000","scrub_duration":0.0045481469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"72'3596","reported_seq":3673,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:56.936002+0000","last_change":"2023-10-25T00:24:34.284728+0000","last_active":"2023-10-25T00:23:56.936002+0000","last_peered":"2023-10-25T00:23:56.936002+0000","last_clean":"2023-10-25T00:23:56.936002+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-25T00:23:56.936002+0000","last_undegraded":"2023-10-25T00:23:56.936002+0000","last_fullsized":"2023-10-25T00:23:56.936002+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-24T23:53:11.386581+0000","last_clean_scrub_stamp":"2023-10-24T23:53:11.386581+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":19820,"num_write_kb":259784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"68'4970","reported_seq":5059,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.288619+0000","last_change":"2023-10-25T00:22:55.999798+0000","last_active":"2023-10-25T00:24:34.288619+0000","last_peered":"2023-10-25T00:24:34.288619+0000","last_clean":"2023-10-25T00:24:34.288619+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-25T00:24:34.288619+0000","last_undegraded":"2023-10-25T00:24:34.288619+0000","last_fullsized":"2023-10-25T00:24:34.288619+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4970","last_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_deep_scrub":"68'4970","last_deep_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.999644+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:15:47.681358+0000","scrub_duration":0.177095113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26721,"num_write_kb":311966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"73'1466","reported_seq":1609,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315010+0000","last_change":"2023-10-25T00:23:06.467052+0000","last_active":"2023-10-25T00:24:34.315010+0000","last_peered":"2023-10-25T00:24:34.315010+0000","last_clean":"2023-10-25T00:24:34.315010+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-25T00:24:34.315010+0000","last_undegraded":"2023-10-25T00:24:34.315010+0000","last_fullsized":"2023-10-25T00:24:34.315010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1466","last_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_deep_scrub":"73'1466","last_deep_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.466569+0000","objects_scrubbed":2,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:21.696239+0000","scrub_duration":0.039946036999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":51896,"num_write":1439,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'7","reported_seq":113,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:53.616774+0000","last_change":"2023-10-25T00:23:53.616774+0000","last_active":"2023-10-25T00:23:53.616774+0000","last_peered":"2023-10-25T00:23:53.616774+0000","last_clean":"2023-10-25T00:23:53.616774+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-25T00:23:53.616774+0000","last_undegraded":"2023-10-25T00:23:53.616774+0000","last_fullsized":"2023-10-25T00:23:53.616774+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'7","last_scrub_stamp":"2023-10-25T00:23:53.616451+0000","last_deep_scrub":"73'7","last_deep_scrub_stamp":"2023-10-25T00:23:53.616451+0000","last_clean_scrub_stamp":"2023-10-25T00:23:53.616451+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:31:57.298978+0000","scrub_duration":0.13500319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"72'3548","reported_seq":3635,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.496463+0000","last_change":"2023-10-25T00:23:32.093782+0000","last_active":"2023-10-25T00:23:36.496463+0000","last_peered":"2023-10-25T00:23:36.496463+0000","last_clean":"2023-10-25T00:23:36.496463+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-25T00:23:36.496463+0000","last_undegraded":"2023-10-25T00:23:36.496463+0000","last_fullsized":"2023-10-25T00:23:36.496463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3548","last_scrub_stamp":"2023-10-25T00:23:32.093615+0000","last_deep_scrub":"72'3548","last_deep_scrub_stamp":"2023-10-25T00:23:32.093615+0000","last_clean_scrub_stamp":"2023-10-25T00:23:32.093615+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:02:03.851334+0000","scrub_duration":0.28610911100000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20243,"num_write_kb":261539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"68'3393","reported_seq":3484,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315870+0000","last_change":"2023-10-25T00:22:42.098602+0000","last_active":"2023-10-25T00:24:34.315870+0000","last_peered":"2023-10-25T00:24:34.315870+0000","last_clean":"2023-10-25T00:24:34.315870+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-25T00:24:34.315870+0000","last_undegraded":"2023-10-25T00:24:34.315870+0000","last_fullsized":"2023-10-25T00:24:34.315870+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3393","last_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_deep_scrub":"68'3393","last_deep_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.098371+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:12.736136+0000","scrub_duration":0.153676797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17500,"num_write_kb":217638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"68'5","reported_seq":93,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.297563+0000","last_change":"2023-10-25T00:23:33.402936+0000","last_active":"2023-10-25T00:24:34.297563+0000","last_peered":"2023-10-25T00:24:34.297563+0000","last_clean":"2023-10-25T00:24:34.297563+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-25T00:24:34.297563+0000","last_undegraded":"2023-10-25T00:24:34.297563+0000","last_fullsized":"2023-10-25T00:24:34.297563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:33.402617+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:33.402617+0000","last_clean_scrub_stamp":"2023-10-25T00:23:33.402617+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:10:14.715542+0000","scrub_duration":0.0048941419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"72'3","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.419862+0000","last_change":"2023-10-25T00:23:45.419862+0000","last_active":"2023-10-25T00:23:45.419862+0000","last_peered":"2023-10-25T00:23:45.419862+0000","last_clean":"2023-10-25T00:23:45.419862+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-25T00:23:45.419862+0000","last_undegraded":"2023-10-25T00:23:45.419862+0000","last_fullsized":"2023-10-25T00:23:45.419862+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:45.419528+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:45.419528+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.419528+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:07:59.973530+0000","scrub_duration":0.0045258490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"72'3584","reported_seq":3671,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.285625+0000","last_change":"2023-10-25T00:23:40.285625+0000","last_active":"2023-10-25T00:23:40.285625+0000","last_peered":"2023-10-25T00:23:40.285625+0000","last_clean":"2023-10-25T00:23:40.285625+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-25T00:23:40.285625+0000","last_undegraded":"2023-10-25T00:23:40.285625+0000","last_fullsized":"2023-10-25T00:23:40.285625+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3584","last_scrub_stamp":"2023-10-25T00:23:40.285478+0000","last_deep_scrub":"72'3584","last_deep_scrub_stamp":"2023-10-25T00:23:40.285478+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.285478+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:11:39.447639+0000","scrub_duration":0.26816689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20504,"num_write_kb":254394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'5094","reported_seq":5187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.678409+0000","last_change":"2023-10-25T00:22:51.485282+0000","last_active":"2023-10-25T00:23:30.678409+0000","last_peered":"2023-10-25T00:23:30.678409+0000","last_clean":"2023-10-25T00:23:30.678409+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-25T00:23:30.678409+0000","last_undegraded":"2023-10-25T00:23:30.678409+0000","last_fullsized":"2023-10-25T00:23:30.678409+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5094","last_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_deep_scrub":"68'5094","last_deep_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.484917+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:22:43.899134+0000","scrub_duration":0.19299439099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27763,"num_write_kb":328490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"73'7","reported_seq":119,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.317526+0000","last_change":"2023-10-25T00:23:27.095106+0000","last_active":"2023-10-25T00:24:34.317526+0000","last_peered":"2023-10-25T00:24:34.317526+0000","last_clean":"2023-10-25T00:24:34.317526+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-25T00:24:34.317526+0000","last_undegraded":"2023-10-25T00:24:34.317526+0000","last_fullsized":"2023-10-25T00:24:34.317526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'7","last_scrub_stamp":"2023-10-25T00:23:27.094773+0000","last_deep_scrub":"73'7","last_deep_scrub_stamp":"2023-10-25T00:23:27.094773+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.094773+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:52:00.838368+0000","scrub_duration":0.007318275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"73'1440","reported_seq":1571,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.527658+0000","last_change":"2023-10-25T00:23:16.527658+0000","last_active":"2023-10-25T00:23:16.527658+0000","last_peered":"2023-10-25T00:23:16.527658+0000","last_clean":"2023-10-25T00:23:16.527658+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-25T00:23:16.527658+0000","last_undegraded":"2023-10-25T00:23:16.527658+0000","last_fullsized":"2023-10-25T00:23:16.527658+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1440","last_scrub_stamp":"2023-10-25T00:23:16.527441+0000","last_deep_scrub":"73'1440","last_deep_scrub_stamp":"2023-10-25T00:23:16.527441+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.527441+0000","objects_scrubbed":2,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:47:53.223550+0000","scrub_duration":0.234060301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":35275,"num_write":1412,"num_write_kb":4487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"72'3292","reported_seq":3379,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.971878+0000","last_change":"2023-10-25T00:23:30.971878+0000","last_active":"2023-10-25T00:23:30.971878+0000","last_peered":"2023-10-25T00:23:30.971878+0000","last_clean":"2023-10-25T00:23:30.971878+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-25T00:23:30.971878+0000","last_undegraded":"2023-10-25T00:23:30.971878+0000","last_fullsized":"2023-10-25T00:23:30.971878+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3292","last_scrub_stamp":"2023-10-25T00:23:30.971718+0000","last_deep_scrub":"72'3292","last_deep_scrub_stamp":"2023-10-25T00:23:30.971718+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.971718+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:45:07.787712+0000","scrub_duration":0.347700487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17853,"num_write_kb":243602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'4242","reported_seq":4330,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.307868+0000","last_change":"2023-10-25T00:22:45.893397+0000","last_active":"2023-10-25T00:23:26.307868+0000","last_peered":"2023-10-25T00:23:26.307868+0000","last_clean":"2023-10-25T00:23:26.307868+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-25T00:23:26.307868+0000","last_undegraded":"2023-10-25T00:23:26.307868+0000","last_fullsized":"2023-10-25T00:23:26.307868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4242","last_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_deep_scrub":"67'4242","last_deep_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.893150+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:24:28.749675+0000","scrub_duration":0.21189509300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22687,"num_write_kb":275346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"73'8","reported_seq":142,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.316433+0000","last_change":"2023-10-25T00:23:30.063493+0000","last_active":"2023-10-25T00:24:34.316433+0000","last_peered":"2023-10-25T00:24:34.316433+0000","last_clean":"2023-10-25T00:24:34.316433+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-25T00:24:34.316433+0000","last_undegraded":"2023-10-25T00:24:34.316433+0000","last_fullsized":"2023-10-25T00:24:34.316433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2023-10-25T00:23:30.063132+0000","last_deep_scrub":"73'8","last_deep_scrub_stamp":"2023-10-25T00:23:30.063132+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.063132+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:12:23.891399+0000","scrub_duration":0.0076496660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"72'5","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.820644+0000","last_change":"2023-10-25T00:23:19.320206+0000","last_active":"2023-10-25T00:23:35.820644+0000","last_peered":"2023-10-25T00:23:35.820644+0000","last_clean":"2023-10-25T00:23:35.820644+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-25T00:23:35.820644+0000","last_undegraded":"2023-10-25T00:23:35.820644+0000","last_fullsized":"2023-10-25T00:23:35.820644+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:23:19.319840+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:23:19.319840+0000","last_clean_scrub_stamp":"2023-10-25T00:23:19.319840+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:18:41.713579+0000","scrub_duration":0.0042872409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"72'4091","reported_seq":4178,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:50.884393+0000","last_change":"2023-10-25T00:23:50.884393+0000","last_active":"2023-10-25T00:23:50.884393+0000","last_peered":"2023-10-25T00:23:50.884393+0000","last_clean":"2023-10-25T00:23:50.884393+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-25T00:23:50.884393+0000","last_undegraded":"2023-10-25T00:23:50.884393+0000","last_fullsized":"2023-10-25T00:23:50.884393+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4091","last_scrub_stamp":"2023-10-25T00:23:50.884251+0000","last_deep_scrub":"72'4091","last_deep_scrub_stamp":"2023-10-25T00:23:50.884251+0000","last_clean_scrub_stamp":"2023-10-25T00:23:50.884251+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:21:51.075072+0000","scrub_duration":0.36544204200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22779,"num_write_kb":288098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'4310","reported_seq":4398,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:25.928208+0000","last_change":"2023-10-25T00:23:22.394323+0000","last_active":"2023-10-25T00:23:25.928208+0000","last_peered":"2023-10-25T00:23:25.928208+0000","last_clean":"2023-10-25T00:23:25.928208+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-25T00:23:25.928208+0000","last_undegraded":"2023-10-25T00:23:25.928208+0000","last_fullsized":"2023-10-25T00:23:25.928208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4310","last_scrub_stamp":"2023-10-25T00:23:22.393940+0000","last_deep_scrub":"67'4310","last_deep_scrub_stamp":"2023-10-25T00:23:22.393940+0000","last_clean_scrub_stamp":"2023-10-25T00:23:22.393940+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:24:46.673127+0000","scrub_duration":0.085301248999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23791,"num_write_kb":283914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"73'5","reported_seq":117,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.317432+0000","last_change":"2023-10-25T00:23:26.183662+0000","last_active":"2023-10-25T00:24:34.317432+0000","last_peered":"2023-10-25T00:24:34.317432+0000","last_clean":"2023-10-25T00:24:34.317432+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-25T00:24:34.317432+0000","last_undegraded":"2023-10-25T00:24:34.317432+0000","last_fullsized":"2023-10-25T00:24:34.317432+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:23:26.183430+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:23:26.183430+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.183430+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:28:22.710805+0000","scrub_duration":0.110908062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"72'2","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.273901+0000","last_change":"2023-10-25T00:22:38.034628+0000","last_active":"2023-10-25T00:23:36.273901+0000","last_peered":"2023-10-25T00:23:36.273901+0000","last_clean":"2023-10-25T00:23:36.273901+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-25T00:23:36.273901+0000","last_undegraded":"2023-10-25T00:23:36.273901+0000","last_fullsized":"2023-10-25T00:23:36.273901+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.034499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:09.450121+0000","scrub_duration":0.14675892500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"72'3695","reported_seq":3782,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:51.251559+0000","last_change":"2023-10-25T00:23:51.251559+0000","last_active":"2023-10-25T00:23:51.251559+0000","last_peered":"2023-10-25T00:23:51.251559+0000","last_clean":"2023-10-25T00:23:51.251559+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-25T00:23:51.251559+0000","last_undegraded":"2023-10-25T00:23:51.251559+0000","last_fullsized":"2023-10-25T00:23:51.251559+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3695","last_scrub_stamp":"2023-10-25T00:23:51.251416+0000","last_deep_scrub":"72'3695","last_deep_scrub_stamp":"2023-10-25T00:23:51.251416+0000","last_clean_scrub_stamp":"2023-10-25T00:23:51.251416+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:44:06.748969+0000","scrub_duration":0.274463921,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21426,"num_write_kb":254037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"72'3","reported_seq":85,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:49.222508+0000","last_change":"2023-10-25T00:22:42.331112+0000","last_active":"2023-10-25T00:22:49.222508+0000","last_peered":"2023-10-25T00:22:49.222508+0000","last_clean":"2023-10-25T00:22:49.222508+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-25T00:22:49.222508+0000","last_undegraded":"2023-10-25T00:22:49.222508+0000","last_fullsized":"2023-10-25T00:22:49.222508+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.330786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:23:56.573805+0000","scrub_duration":0.0043674530000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"72'19","reported_seq":219,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:25.677354+0000","last_change":"2023-10-25T00:22:47.253320+0000","last_active":"2023-10-25T00:23:25.677354+0000","last_peered":"2023-10-25T00:23:25.677354+0000","last_clean":"2023-10-25T00:23:25.677354+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-25T00:23:25.677354+0000","last_undegraded":"2023-10-25T00:23:25.677354+0000","last_fullsized":"2023-10-25T00:23:25.677354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_deep_scrub":"72'19","last_deep_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.252995+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:03.720456+0000","scrub_duration":0.018599613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":339,"num_read_kb":339,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'5249","reported_seq":5344,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.570074+0000","last_change":"2023-10-25T00:22:52.439461+0000","last_active":"2023-10-25T00:24:32.570074+0000","last_peered":"2023-10-25T00:24:32.570074+0000","last_clean":"2023-10-25T00:24:32.570074+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-25T00:24:32.570074+0000","last_undegraded":"2023-10-25T00:24:32.570074+0000","last_fullsized":"2023-10-25T00:24:32.570074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5249","last_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_deep_scrub":"68'5249","last_deep_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.439150+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:19:03.730037+0000","scrub_duration":0.077069795999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":27646,"num_write_kb":332285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"72'4380","reported_seq":4465,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:38.646626+0000","last_change":"2023-10-25T00:23:38.646626+0000","last_active":"2023-10-25T00:23:38.646626+0000","last_peered":"2023-10-25T00:23:38.646626+0000","last_clean":"2023-10-25T00:23:38.646626+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-25T00:23:38.646626+0000","last_undegraded":"2023-10-25T00:23:38.646626+0000","last_fullsized":"2023-10-25T00:23:38.646626+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4380","last_scrub_stamp":"2023-10-25T00:23:38.646472+0000","last_deep_scrub":"72'4380","last_deep_scrub_stamp":"2023-10-25T00:23:38.646472+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.646472+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:38:56.172654+0000","scrub_duration":0.29662724200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23662,"num_write_kb":324338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"72'5","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:50.618067+0000","last_change":"2023-10-25T00:22:50.618067+0000","last_active":"2023-10-25T00:22:50.618067+0000","last_peered":"2023-10-25T00:22:50.618067+0000","last_clean":"2023-10-25T00:22:50.618067+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-25T00:22:50.618067+0000","last_undegraded":"2023-10-25T00:22:50.618067+0000","last_fullsized":"2023-10-25T00:22:50.618067+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.617931+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:24.381368+0000","scrub_duration":0.234626104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"68'5","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.271669+0000","last_change":"2023-10-25T00:23:25.067489+0000","last_active":"2023-10-25T00:23:26.271669+0000","last_peered":"2023-10-25T00:23:26.271669+0000","last_clean":"2023-10-25T00:23:26.271669+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-25T00:23:26.271669+0000","last_undegraded":"2023-10-25T00:23:26.271669+0000","last_fullsized":"2023-10-25T00:23:26.271669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:25.067146+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:25.067146+0000","last_clean_scrub_stamp":"2023-10-25T00:23:25.067146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:32.009959+0000","scrub_duration":0.0041696520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'4442","reported_seq":4529,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.509925+0000","last_change":"2023-10-25T00:23:29.981423+0000","last_active":"2023-10-25T00:24:31.509925+0000","last_peered":"2023-10-25T00:24:31.509925+0000","last_clean":"2023-10-25T00:24:31.509925+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-25T00:24:31.509925+0000","last_undegraded":"2023-10-25T00:24:31.509925+0000","last_fullsized":"2023-10-25T00:24:31.509925+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4442","last_scrub_stamp":"2023-10-25T00:23:29.981245+0000","last_deep_scrub":"68'4442","last_deep_scrub_stamp":"2023-10-25T00:23:29.981245+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.981245+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:11:38.194434+0000","scrub_duration":0.109448791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22810,"num_write_kb":296225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"72'4076","reported_seq":4162,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:07.280653+0000","last_change":"2023-10-25T00:24:07.280653+0000","last_active":"2023-10-25T00:24:07.280653+0000","last_peered":"2023-10-25T00:24:07.280653+0000","last_clean":"2023-10-25T00:24:07.280653+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-25T00:24:07.280653+0000","last_undegraded":"2023-10-25T00:24:07.280653+0000","last_fullsized":"2023-10-25T00:24:07.280653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4076","last_scrub_stamp":"2023-10-25T00:24:07.280471+0000","last_deep_scrub":"72'4076","last_deep_scrub_stamp":"2023-10-25T00:24:07.280471+0000","last_clean_scrub_stamp":"2023-10-25T00:24:07.280471+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:36:18.063026+0000","scrub_duration":0.198538093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22055,"num_write_kb":297609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"68'3","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.494095+0000","last_change":"2023-10-25T00:23:23.928686+0000","last_active":"2023-10-25T00:23:26.494095+0000","last_peered":"2023-10-25T00:23:26.494095+0000","last_clean":"2023-10-25T00:23:26.494095+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-25T00:23:26.494095+0000","last_undegraded":"2023-10-25T00:23:26.494095+0000","last_fullsized":"2023-10-25T00:23:26.494095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:23.928352+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:23.928352+0000","last_clean_scrub_stamp":"2023-10-25T00:23:23.928352+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:41:04.304281+0000","scrub_duration":0.0043703589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'4101","reported_seq":4189,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.110625+0000","last_change":"2023-10-25T00:22:50.337815+0000","last_active":"2023-10-25T00:24:31.110625+0000","last_peered":"2023-10-25T00:24:31.110625+0000","last_clean":"2023-10-25T00:24:31.110625+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-25T00:24:31.110625+0000","last_undegraded":"2023-10-25T00:24:31.110625+0000","last_fullsized":"2023-10-25T00:24:31.110625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.337536+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:01.275651+0000","scrub_duration":0.055303021000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22343,"num_write_kb":253486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"72'397","reported_seq":592,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.542939+0000","last_change":"2023-10-25T00:22:45.554898+0000","last_active":"2023-10-25T00:23:37.542939+0000","last_peered":"2023-10-25T00:23:37.542939+0000","last_clean":"2023-10-25T00:23:37.542939+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-25T00:23:37.542939+0000","last_undegraded":"2023-10-25T00:23:37.542939+0000","last_fullsized":"2023-10-25T00:23:37.542939+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'397","last_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_deep_scrub":"72'397","last_deep_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.554526+0000","objects_scrubbed":4,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:41:32.056818+0000","scrub_duration":0.10248053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":86197,"num_write":389,"num_write_kb":4306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"72'4606","reported_seq":4690,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.057459+0000","last_change":"2023-10-25T00:23:37.057459+0000","last_active":"2023-10-25T00:23:37.057459+0000","last_peered":"2023-10-25T00:23:37.057459+0000","last_clean":"2023-10-25T00:23:37.057459+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-25T00:23:37.057459+0000","last_undegraded":"2023-10-25T00:23:37.057459+0000","last_fullsized":"2023-10-25T00:23:37.057459+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4606","last_scrub_stamp":"2023-10-25T00:23:37.057332+0000","last_deep_scrub":"72'4606","last_deep_scrub_stamp":"2023-10-25T00:23:37.057332+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.057332+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:16:39.313888+0000","scrub_duration":0.28055820399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25763,"num_write_kb":316477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"74'15","reported_seq":149,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.318046+0000","last_change":"2023-10-25T00:23:16.153043+0000","last_active":"2023-10-25T00:24:34.318046+0000","last_peered":"2023-10-25T00:24:34.318046+0000","last_clean":"2023-10-25T00:24:34.318046+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-25T00:24:34.318046+0000","last_undegraded":"2023-10-25T00:24:34.318046+0000","last_fullsized":"2023-10-25T00:24:34.318046+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_deep_scrub":"73'14","last_deep_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.152723+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:32:30.128331+0000","scrub_duration":0.010186763999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":185,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'5680","reported_seq":5771,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.540112+0000","last_change":"2023-10-25T00:22:54.586176+0000","last_active":"2023-10-25T00:23:27.540112+0000","last_peered":"2023-10-25T00:23:27.540112+0000","last_clean":"2023-10-25T00:23:27.540112+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-25T00:23:27.540112+0000","last_undegraded":"2023-10-25T00:23:27.540112+0000","last_fullsized":"2023-10-25T00:23:27.540112+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5680","last_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_deep_scrub":"68'5680","last_deep_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.585866+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:49.342364+0000","scrub_duration":0.20835251299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29749,"num_write_kb":360962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"73'40","reported_seq":188,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:59.279527+0000","last_change":"2023-10-25T00:22:59.279527+0000","last_active":"2023-10-25T00:22:59.279527+0000","last_peered":"2023-10-25T00:22:59.279527+0000","last_clean":"2023-10-25T00:22:59.279527+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-25T00:22:59.279527+0000","last_undegraded":"2023-10-25T00:22:59.279527+0000","last_fullsized":"2023-10-25T00:22:59.279527+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'40","last_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_deep_scrub":"73'40","last_deep_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.279288+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:52.911132+0000","scrub_duration":0.0094949800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":39,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"72'4808","reported_seq":4899,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:30.690247+0000","last_change":"2023-10-25T00:23:14.791099+0000","last_active":"2023-10-25T00:24:30.690247+0000","last_peered":"2023-10-25T00:24:30.690247+0000","last_clean":"2023-10-25T00:24:30.690247+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-25T00:24:30.690247+0000","last_undegraded":"2023-10-25T00:24:30.690247+0000","last_fullsized":"2023-10-25T00:24:30.690247+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4808","last_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_deep_scrub":"72'4808","last_deep_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.790929+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:56:36.982903+0000","scrub_duration":0.29119922999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26552,"num_write_kb":336125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"68'4747","reported_seq":4837,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:20.688044+0000","last_change":"2023-10-25T00:23:00.496309+0000","last_active":"2023-10-25T00:24:20.688044+0000","last_peered":"2023-10-25T00:24:20.688044+0000","last_clean":"2023-10-25T00:24:20.688044+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-25T00:24:20.688044+0000","last_undegraded":"2023-10-25T00:24:20.688044+0000","last_fullsized":"2023-10-25T00:24:20.688044+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4747","last_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_deep_scrub":"68'4747","last_deep_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.496060+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:56.531125+0000","scrub_duration":0.089432630999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24847,"num_write_kb":316685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"72'4283","reported_seq":4372,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.286177+0000","last_change":"2023-10-25T00:23:08.728718+0000","last_active":"2023-10-25T00:24:34.286177+0000","last_peered":"2023-10-25T00:24:34.286177+0000","last_clean":"2023-10-25T00:24:34.286177+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-25T00:24:34.286177+0000","last_undegraded":"2023-10-25T00:24:34.286177+0000","last_fullsized":"2023-10-25T00:24:34.286177+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4283","last_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_deep_scrub":"72'4283","last_deep_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_clean_scrub_stamp":"2023-10-25T00:23:08.728567+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:33.655445+0000","scrub_duration":0.26846236600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23853,"num_write_kb":296504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"68'4036","reported_seq":4126,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:22.567176+0000","last_change":"2023-10-25T00:22:39.642537+0000","last_active":"2023-10-25T00:24:22.567176+0000","last_peered":"2023-10-25T00:24:22.567176+0000","last_clean":"2023-10-25T00:24:22.567176+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-25T00:24:22.567176+0000","last_undegraded":"2023-10-25T00:24:22.567176+0000","last_fullsized":"2023-10-25T00:24:22.567176+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4036","last_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_deep_scrub":"68'4036","last_deep_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.642342+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:02.362976+0000","scrub_duration":0.227585228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21233,"num_write_kb":273467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"72'4276","reported_seq":4364,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.341479+0000","last_change":"2023-10-25T00:23:30.997946+0000","last_active":"2023-10-25T00:24:31.341479+0000","last_peered":"2023-10-25T00:24:31.341479+0000","last_clean":"2023-10-25T00:24:31.341479+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-25T00:24:31.341479+0000","last_undegraded":"2023-10-25T00:24:31.341479+0000","last_fullsized":"2023-10-25T00:24:31.341479+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4276","last_scrub_stamp":"2023-10-25T00:23:30.997816+0000","last_deep_scrub":"72'4276","last_deep_scrub_stamp":"2023-10-25T00:23:30.997816+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.997816+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:08.863577+0000","scrub_duration":0.39360271099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23362,"num_write_kb":310817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"68'6201","reported_seq":6294,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.450739+0000","last_change":"2023-10-25T00:24:34.450866+0000","last_active":"2023-10-25T00:24:34.450739+0000","last_peered":"2023-10-25T00:24:34.450739+0000","last_clean":"2023-10-25T00:24:34.450739+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-25T00:24:34.450739+0000","last_undegraded":"2023-10-25T00:24:34.450739+0000","last_fullsized":"2023-10-25T00:24:34.450739+0000","mapping_epoch":21,"log_start":"62'1000","ondisk_log_start":"62'1000","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6201","last_scrub_stamp":"2023-10-25T00:24:34.450718+0000","last_deep_scrub":"68'6201","last_deep_scrub_stamp":"2023-10-25T00:24:34.450718+0000","last_clean_scrub_stamp":"2023-10-25T00:24:34.450718+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:58:02.692663+0000","scrub_duration":0.18173869100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33486,"num_write_kb":384441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"72'4018","reported_seq":4107,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.287337+0000","last_change":"2023-10-25T00:23:52.269793+0000","last_active":"2023-10-25T00:24:31.287337+0000","last_peered":"2023-10-25T00:24:31.287337+0000","last_clean":"2023-10-25T00:24:31.287337+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-25T00:24:31.287337+0000","last_undegraded":"2023-10-25T00:24:31.287337+0000","last_fullsized":"2023-10-25T00:24:31.287337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4018","last_scrub_stamp":"2023-10-25T00:23:52.269641+0000","last_deep_scrub":"72'4018","last_deep_scrub_stamp":"2023-10-25T00:23:52.269641+0000","last_clean_scrub_stamp":"2023-10-25T00:23:52.269641+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:58:28.387402+0000","scrub_duration":0.26794800299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":21920,"num_write_kb":288651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"68'4522","reported_seq":4613,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:22.567304+0000","last_change":"2023-10-25T00:23:26.751600+0000","last_active":"2023-10-25T00:24:22.567304+0000","last_peered":"2023-10-25T00:24:22.567304+0000","last_clean":"2023-10-25T00:24:22.567304+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-25T00:24:22.567304+0000","last_undegraded":"2023-10-25T00:24:22.567304+0000","last_fullsized":"2023-10-25T00:24:22.567304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4522","last_scrub_stamp":"2023-10-25T00:23:26.751435+0000","last_deep_scrub":"68'4522","last_deep_scrub_stamp":"2023-10-25T00:23:26.751435+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.751435+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:40:36.338773+0000","scrub_duration":0.110684227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24476,"num_write_kb":282500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"72'4209","reported_seq":4294,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.054692+0000","last_change":"2023-10-25T00:23:37.054692+0000","last_active":"2023-10-25T00:23:37.054692+0000","last_peered":"2023-10-25T00:23:37.054692+0000","last_clean":"2023-10-25T00:23:37.054692+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-25T00:23:37.054692+0000","last_undegraded":"2023-10-25T00:23:37.054692+0000","last_fullsized":"2023-10-25T00:23:37.054692+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4209","last_scrub_stamp":"2023-10-25T00:23:37.054538+0000","last_deep_scrub":"72'4209","last_deep_scrub_stamp":"2023-10-25T00:23:37.054538+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.054538+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:39:40.159884+0000","scrub_duration":0.40759778099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24141,"num_write_kb":281325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"68'4915","reported_seq":5003,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.325819+0000","last_change":"2023-10-25T00:22:38.600278+0000","last_active":"2023-10-25T00:23:26.325819+0000","last_peered":"2023-10-25T00:23:26.325819+0000","last_clean":"2023-10-25T00:23:26.325819+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-25T00:23:26.325819+0000","last_undegraded":"2023-10-25T00:23:26.325819+0000","last_fullsized":"2023-10-25T00:23:26.325819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4915","last_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_deep_scrub":"68'4915","last_deep_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.600135+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:42:30.671003+0000","scrub_duration":0.18827047799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26161,"num_write_kb":314084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"72'4933","reported_seq":5018,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.679539+0000","last_change":"2023-10-25T00:23:31.791243+0000","last_active":"2023-10-25T00:23:35.679539+0000","last_peered":"2023-10-25T00:23:35.679539+0000","last_clean":"2023-10-25T00:23:35.679539+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-25T00:23:35.679539+0000","last_undegraded":"2023-10-25T00:23:35.679539+0000","last_fullsized":"2023-10-25T00:23:35.679539+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4933","last_scrub_stamp":"2023-10-25T00:23:31.791069+0000","last_deep_scrub":"72'4933","last_deep_scrub_stamp":"2023-10-25T00:23:31.791069+0000","last_clean_scrub_stamp":"2023-10-25T00:23:31.791069+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:53:40.465688+0000","scrub_duration":0.269140133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26695,"num_write_kb":352905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"68'5055","reported_seq":5144,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:25.928638+0000","last_change":"2023-10-25T00:22:43.444464+0000","last_active":"2023-10-25T00:23:25.928638+0000","last_peered":"2023-10-25T00:23:25.928638+0000","last_clean":"2023-10-25T00:23:25.928638+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-25T00:23:25.928638+0000","last_undegraded":"2023-10-25T00:23:25.928638+0000","last_fullsized":"2023-10-25T00:23:25.928638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5055","last_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_deep_scrub":"68'5055","last_deep_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_clean_scrub_stamp":"2023-10-25T00:22:43.444210+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:13:34.310177+0000","scrub_duration":0.081291211000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27516,"num_write_kb":327406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"72'4083","reported_seq":4172,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:47.749735+0000","last_change":"2023-10-25T00:23:47.749735+0000","last_active":"2023-10-25T00:23:47.749735+0000","last_peered":"2023-10-25T00:23:47.749735+0000","last_clean":"2023-10-25T00:23:47.749735+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-25T00:23:47.749735+0000","last_undegraded":"2023-10-25T00:23:47.749735+0000","last_fullsized":"2023-10-25T00:23:47.749735+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4083","last_scrub_stamp":"2023-10-25T00:23:47.749591+0000","last_deep_scrub":"72'4083","last_deep_scrub_stamp":"2023-10-25T00:23:47.749591+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.749591+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:43:07.403801+0000","scrub_duration":0.25918773899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":22892,"num_write_kb":296872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"68'4777","reported_seq":4867,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.325889+0000","last_change":"2023-10-25T00:23:24.846143+0000","last_active":"2023-10-25T00:23:26.325889+0000","last_peered":"2023-10-25T00:23:26.325889+0000","last_clean":"2023-10-25T00:23:26.325889+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-25T00:23:26.325889+0000","last_undegraded":"2023-10-25T00:23:26.325889+0000","last_fullsized":"2023-10-25T00:23:26.325889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4777","last_scrub_stamp":"2023-10-25T00:23:24.846002+0000","last_deep_scrub":"68'4777","last_deep_scrub_stamp":"2023-10-25T00:23:24.846002+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.846002+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:41:16.418283+0000","scrub_duration":0.19199595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25812,"num_write_kb":300181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"72'4618","reported_seq":4709,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.679474+0000","last_change":"2023-10-25T00:23:33.839487+0000","last_active":"2023-10-25T00:23:35.679474+0000","last_peered":"2023-10-25T00:23:35.679474+0000","last_clean":"2023-10-25T00:23:35.679474+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-25T00:23:35.679474+0000","last_undegraded":"2023-10-25T00:23:35.679474+0000","last_fullsized":"2023-10-25T00:23:35.679474+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4618","last_scrub_stamp":"2023-10-25T00:23:33.839138+0000","last_deep_scrub":"72'4618","last_deep_scrub_stamp":"2023-10-25T00:23:33.839138+0000","last_clean_scrub_stamp":"2023-10-25T00:23:33.839138+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:25:32.655697+0000","scrub_duration":0.330454572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":26180,"num_write_kb":326342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"68'5340","reported_seq":5428,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.325944+0000","last_change":"2023-10-25T00:22:42.520048+0000","last_active":"2023-10-25T00:23:26.325944+0000","last_peered":"2023-10-25T00:23:26.325944+0000","last_clean":"2023-10-25T00:23:26.325944+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-25T00:23:26.325944+0000","last_undegraded":"2023-10-25T00:23:26.325944+0000","last_fullsized":"2023-10-25T00:23:26.325944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5340","last_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_deep_scrub":"68'5340","last_deep_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.519886+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.832889+0000","scrub_duration":0.129134375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28701,"num_write_kb":348521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"72'4171","reported_seq":4257,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:17.419254+0000","last_change":"2023-10-25T00:24:17.419254+0000","last_active":"2023-10-25T00:24:17.419254+0000","last_peered":"2023-10-25T00:24:17.419254+0000","last_clean":"2023-10-25T00:24:17.419254+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-25T00:24:17.419254+0000","last_undegraded":"2023-10-25T00:24:17.419254+0000","last_fullsized":"2023-10-25T00:24:17.419254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4171","last_scrub_stamp":"2023-10-25T00:24:17.419103+0000","last_deep_scrub":"72'4171","last_deep_scrub_stamp":"2023-10-25T00:24:17.419103+0000","last_clean_scrub_stamp":"2023-10-25T00:24:17.419103+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:59.723424+0000","scrub_duration":0.37988864300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22551,"num_write_kb":303498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"68'4493","reported_seq":4582,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:01.323167+0000","last_change":"2023-10-25T00:23:01.323167+0000","last_active":"2023-10-25T00:23:01.323167+0000","last_peered":"2023-10-25T00:23:01.323167+0000","last_clean":"2023-10-25T00:23:01.323167+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-25T00:23:01.323167+0000","last_undegraded":"2023-10-25T00:23:01.323167+0000","last_fullsized":"2023-10-25T00:23:01.323167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4493","last_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_deep_scrub":"68'4493","last_deep_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_clean_scrub_stamp":"2023-10-25T00:23:01.322941+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:49.029022+0000","scrub_duration":0.13616751799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23797,"num_write_kb":300260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"72'4763","reported_seq":4848,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:57.009674+0000","last_change":"2023-10-25T00:23:57.009674+0000","last_active":"2023-10-25T00:23:57.009674+0000","last_peered":"2023-10-25T00:23:57.009674+0000","last_clean":"2023-10-25T00:23:57.009674+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-25T00:23:57.009674+0000","last_undegraded":"2023-10-25T00:23:57.009674+0000","last_fullsized":"2023-10-25T00:23:57.009674+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4763","last_scrub_stamp":"2023-10-25T00:23:57.009531+0000","last_deep_scrub":"72'4763","last_deep_scrub_stamp":"2023-10-25T00:23:57.009531+0000","last_clean_scrub_stamp":"2023-10-25T00:23:57.009531+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:34:10.319674+0000","scrub_duration":0.25795965799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":25751,"num_write_kb":345225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"68'4946","reported_seq":5035,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.569710+0000","last_change":"2023-10-25T00:23:55.632753+0000","last_active":"2023-10-25T00:24:32.569710+0000","last_peered":"2023-10-25T00:24:32.569710+0000","last_clean":"2023-10-25T00:24:32.569710+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-25T00:24:32.569710+0000","last_undegraded":"2023-10-25T00:24:32.569710+0000","last_fullsized":"2023-10-25T00:24:32.569710+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4946","last_scrub_stamp":"2023-10-25T00:23:55.632589+0000","last_deep_scrub":"68'4946","last_deep_scrub_stamp":"2023-10-25T00:23:55.632589+0000","last_clean_scrub_stamp":"2023-10-25T00:23:55.632589+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:34:04.162109+0000","scrub_duration":0.103749073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26583,"num_write_kb":315633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"72'4352","reported_seq":4438,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.820605+0000","last_change":"2023-10-25T00:23:34.629523+0000","last_active":"2023-10-25T00:23:35.820605+0000","last_peered":"2023-10-25T00:23:35.820605+0000","last_clean":"2023-10-25T00:23:35.820605+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-25T00:23:35.820605+0000","last_undegraded":"2023-10-25T00:23:35.820605+0000","last_fullsized":"2023-10-25T00:23:35.820605+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4352","last_scrub_stamp":"2023-10-25T00:23:34.629323+0000","last_deep_scrub":"72'4352","last_deep_scrub_stamp":"2023-10-25T00:23:34.629323+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.629323+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:38.589150+0000","scrub_duration":0.29560794600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23957,"num_write_kb":301052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"68'5026","reported_seq":5115,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315212+0000","last_change":"2023-10-25T00:24:00.036586+0000","last_active":"2023-10-25T00:24:34.315212+0000","last_peered":"2023-10-25T00:24:34.315212+0000","last_clean":"2023-10-25T00:24:34.315212+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-25T00:24:34.315212+0000","last_undegraded":"2023-10-25T00:24:34.315212+0000","last_fullsized":"2023-10-25T00:24:34.315212+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5026","last_scrub_stamp":"2023-10-25T00:24:00.036334+0000","last_deep_scrub":"68'5026","last_deep_scrub_stamp":"2023-10-25T00:24:00.036334+0000","last_clean_scrub_stamp":"2023-10-25T00:24:00.036334+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:22.913142+0000","scrub_duration":0.209434338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26401,"num_write_kb":310586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"72'4393","reported_seq":4483,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.556990+0000","last_change":"2023-10-25T00:23:07.556990+0000","last_active":"2023-10-25T00:23:07.556990+0000","last_peered":"2023-10-25T00:23:07.556990+0000","last_clean":"2023-10-25T00:23:07.556990+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-25T00:23:07.556990+0000","last_undegraded":"2023-10-25T00:23:07.556990+0000","last_fullsized":"2023-10-25T00:23:07.556990+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4393","last_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_deep_scrub":"72'4393","last_deep_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.556784+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:21:00.928852+0000","scrub_duration":0.35410455800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":24752,"num_write_kb":296109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"68'5620","reported_seq":5712,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.316699+0000","last_change":"2023-10-25T00:23:12.194324+0000","last_active":"2023-10-25T00:24:34.316699+0000","last_peered":"2023-10-25T00:24:34.316699+0000","last_clean":"2023-10-25T00:24:34.316699+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-25T00:24:34.316699+0000","last_undegraded":"2023-10-25T00:24:34.316699+0000","last_fullsized":"2023-10-25T00:24:34.316699+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5620","last_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_deep_scrub":"68'5620","last_deep_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.194008+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:33.868259+0000","scrub_duration":0.086061930999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29863,"num_write_kb":337169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"72'4340","reported_seq":4429,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.679514+0000","last_change":"2023-10-25T00:23:23.851965+0000","last_active":"2023-10-25T00:23:35.679514+0000","last_peered":"2023-10-25T00:23:35.679514+0000","last_clean":"2023-10-25T00:23:35.679514+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-25T00:23:35.679514+0000","last_undegraded":"2023-10-25T00:23:35.679514+0000","last_fullsized":"2023-10-25T00:23:35.679514+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4340","last_scrub_stamp":"2023-10-25T00:23:23.851809+0000","last_deep_scrub":"72'4340","last_deep_scrub_stamp":"2023-10-25T00:23:23.851809+0000","last_clean_scrub_stamp":"2023-10-25T00:23:23.851809+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:00.223998+0000","scrub_duration":0.37196321399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":36,"num_write":23367,"num_write_kb":302669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"68'4464","reported_seq":4551,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.287660+0000","last_change":"2023-10-25T00:22:57.904669+0000","last_active":"2023-10-25T00:24:34.287660+0000","last_peered":"2023-10-25T00:24:34.287660+0000","last_clean":"2023-10-25T00:24:34.287660+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-25T00:24:34.287660+0000","last_undegraded":"2023-10-25T00:24:34.287660+0000","last_fullsized":"2023-10-25T00:24:34.287660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4464","last_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_deep_scrub":"68'4464","last_deep_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.904532+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:17:44.026155+0000","scrub_duration":0.14132308199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24372,"num_write_kb":278074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"72'5589","reported_seq":5674,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.174602+0000","last_change":"2023-10-25T00:23:45.174602+0000","last_active":"2023-10-25T00:23:45.174602+0000","last_peered":"2023-10-25T00:23:45.174602+0000","last_clean":"2023-10-25T00:23:45.174602+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-25T00:23:45.174602+0000","last_undegraded":"2023-10-25T00:23:45.174602+0000","last_fullsized":"2023-10-25T00:23:45.174602+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5589","last_scrub_stamp":"2023-10-25T00:23:45.174448+0000","last_deep_scrub":"72'5589","last_deep_scrub_stamp":"2023-10-25T00:23:45.174448+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.174448+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:47:45.550696+0000","scrub_duration":0.18148444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":30206,"num_write_kb":399557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"68'4542","reported_seq":4630,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.447005+0000","last_change":"2023-10-25T00:23:09.345212+0000","last_active":"2023-10-25T00:24:34.447005+0000","last_peered":"2023-10-25T00:24:34.447005+0000","last_clean":"2023-10-25T00:24:34.447005+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-25T00:24:34.447005+0000","last_undegraded":"2023-10-25T00:24:34.447005+0000","last_fullsized":"2023-10-25T00:24:34.447005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4542","last_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_deep_scrub":"68'4542","last_deep_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.344860+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:16:09.325372+0000","scrub_duration":0.066583699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23889,"num_write_kb":292323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"72'3430","reported_seq":3516,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.798045+0000","last_change":"2023-10-25T00:23:21.798045+0000","last_active":"2023-10-25T00:23:21.798045+0000","last_peered":"2023-10-25T00:23:21.798045+0000","last_clean":"2023-10-25T00:23:21.798045+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-25T00:23:21.798045+0000","last_undegraded":"2023-10-25T00:23:21.798045+0000","last_fullsized":"2023-10-25T00:23:21.798045+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3430","last_scrub_stamp":"2023-10-25T00:23:21.797727+0000","last_deep_scrub":"72'3430","last_deep_scrub_stamp":"2023-10-25T00:23:21.797727+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.797727+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:15:30.844796+0000","scrub_duration":0.29875561899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19411,"num_write_kb":241862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"68'4719","reported_seq":4807,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.308252+0000","last_change":"2023-10-25T00:22:58.903089+0000","last_active":"2023-10-25T00:23:26.308252+0000","last_peered":"2023-10-25T00:23:26.308252+0000","last_clean":"2023-10-25T00:23:26.308252+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-25T00:23:26.308252+0000","last_undegraded":"2023-10-25T00:23:26.308252+0000","last_fullsized":"2023-10-25T00:23:26.308252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4719","last_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_deep_scrub":"68'4719","last_deep_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.902769+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:01.236787+0000","scrub_duration":0.136999537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25020,"num_write_kb":305197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"72'4169","reported_seq":4252,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.640863+0000","last_change":"2023-10-25T00:23:40.640863+0000","last_active":"2023-10-25T00:23:40.640863+0000","last_peered":"2023-10-25T00:23:40.640863+0000","last_clean":"2023-10-25T00:23:40.640863+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-25T00:23:40.640863+0000","last_undegraded":"2023-10-25T00:23:40.640863+0000","last_fullsized":"2023-10-25T00:23:40.640863+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4169","last_scrub_stamp":"2023-10-25T00:23:40.640645+0000","last_deep_scrub":"72'4169","last_deep_scrub_stamp":"2023-10-25T00:23:40.640645+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.640645+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:06.110573+0000","scrub_duration":0.28697117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23235,"num_write_kb":306175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"68'5952","reported_seq":6048,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.494437+0000","last_change":"2023-10-25T00:23:05.017693+0000","last_active":"2023-10-25T00:23:26.494437+0000","last_peered":"2023-10-25T00:23:26.494437+0000","last_clean":"2023-10-25T00:23:26.494437+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-25T00:23:26.494437+0000","last_undegraded":"2023-10-25T00:23:26.494437+0000","last_fullsized":"2023-10-25T00:23:26.494437+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5952","last_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_deep_scrub":"68'5952","last_deep_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.017340+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:31.140541+0000","scrub_duration":0.055624675999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32341,"num_write_kb":375441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"72'3717","reported_seq":3801,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:23.237030+0000","last_change":"2023-10-25T00:24:23.237030+0000","last_active":"2023-10-25T00:24:23.237030+0000","last_peered":"2023-10-25T00:24:23.237030+0000","last_clean":"2023-10-25T00:24:23.237030+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-25T00:24:23.237030+0000","last_undegraded":"2023-10-25T00:24:23.237030+0000","last_fullsized":"2023-10-25T00:24:23.237030+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3717","last_scrub_stamp":"2023-10-25T00:24:23.236882+0000","last_deep_scrub":"72'3717","last_deep_scrub_stamp":"2023-10-25T00:24:23.236882+0000","last_clean_scrub_stamp":"2023-10-25T00:24:23.236882+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:09:23.115312+0000","scrub_duration":0.223121614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21101,"num_write_kb":269108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"68'4825","reported_seq":4914,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.307799+0000","last_change":"2023-10-25T00:23:06.089847+0000","last_active":"2023-10-25T00:23:26.307799+0000","last_peered":"2023-10-25T00:23:26.307799+0000","last_clean":"2023-10-25T00:23:26.307799+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-25T00:23:26.307799+0000","last_undegraded":"2023-10-25T00:23:26.307799+0000","last_fullsized":"2023-10-25T00:23:26.307799+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4825","last_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_deep_scrub":"68'4825","last_deep_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.089705+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:04:51.412522+0000","scrub_duration":0.15071393499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25319,"num_write_kb":294113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"72'3714","reported_seq":3801,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:00.873759+0000","last_change":"2023-10-25T00:24:00.873759+0000","last_active":"2023-10-25T00:24:00.873759+0000","last_peered":"2023-10-25T00:24:00.873759+0000","last_clean":"2023-10-25T00:24:00.873759+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-25T00:24:00.873759+0000","last_undegraded":"2023-10-25T00:24:00.873759+0000","last_fullsized":"2023-10-25T00:24:00.873759+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3714","last_scrub_stamp":"2023-10-25T00:24:00.873431+0000","last_deep_scrub":"72'3714","last_deep_scrub_stamp":"2023-10-25T00:24:00.873431+0000","last_clean_scrub_stamp":"2023-10-25T00:24:00.873431+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:48.699043+0000","scrub_duration":0.31231597300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21148,"num_write_kb":265324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"68'4878","reported_seq":4968,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.271604+0000","last_change":"2023-10-25T00:22:49.227455+0000","last_active":"2023-10-25T00:23:26.271604+0000","last_peered":"2023-10-25T00:23:26.271604+0000","last_clean":"2023-10-25T00:23:26.271604+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-25T00:23:26.271604+0000","last_undegraded":"2023-10-25T00:23:26.271604+0000","last_fullsized":"2023-10-25T00:23:26.271604+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4878","last_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_deep_scrub":"68'4878","last_deep_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_clean_scrub_stamp":"2023-10-25T00:22:49.227012+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:58:21.157778+0000","scrub_duration":0.13340163999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26154,"num_write_kb":305558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"72'3434","reported_seq":3519,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:57.337823+0000","last_change":"2023-10-25T00:23:57.337823+0000","last_active":"2023-10-25T00:23:57.337823+0000","last_peered":"2023-10-25T00:23:57.337823+0000","last_clean":"2023-10-25T00:23:57.337823+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-25T00:23:57.337823+0000","last_undegraded":"2023-10-25T00:23:57.337823+0000","last_fullsized":"2023-10-25T00:23:57.337823+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3434","last_scrub_stamp":"2023-10-25T00:23:57.337452+0000","last_deep_scrub":"72'3434","last_deep_scrub_stamp":"2023-10-25T00:23:57.337452+0000","last_clean_scrub_stamp":"2023-10-25T00:23:57.337452+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:11.559971+0000","scrub_duration":0.315725692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18449,"num_write_kb":262373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"68'4672","reported_seq":4762,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:03.653382+0000","last_change":"2023-10-25T00:24:03.653382+0000","last_active":"2023-10-25T00:24:03.653382+0000","last_peered":"2023-10-25T00:24:03.653382+0000","last_clean":"2023-10-25T00:24:03.653382+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-25T00:24:03.653382+0000","last_undegraded":"2023-10-25T00:24:03.653382+0000","last_fullsized":"2023-10-25T00:24:03.653382+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4672","last_scrub_stamp":"2023-10-25T00:24:03.653141+0000","last_deep_scrub":"68'4672","last_deep_scrub_stamp":"2023-10-25T00:24:03.653141+0000","last_clean_scrub_stamp":"2023-10-25T00:24:03.653141+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:19:24.224599+0000","scrub_duration":0.077062947000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25223,"num_write_kb":293792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"72'4013","reported_seq":4103,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:51.895151+0000","last_change":"2023-10-25T00:23:51.895151+0000","last_active":"2023-10-25T00:23:51.895151+0000","last_peered":"2023-10-25T00:23:51.895151+0000","last_clean":"2023-10-25T00:23:51.895151+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-25T00:23:51.895151+0000","last_undegraded":"2023-10-25T00:23:51.895151+0000","last_fullsized":"2023-10-25T00:23:51.895151+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4013","last_scrub_stamp":"2023-10-25T00:23:51.894968+0000","last_deep_scrub":"72'4013","last_deep_scrub_stamp":"2023-10-25T00:23:51.894968+0000","last_clean_scrub_stamp":"2023-10-25T00:23:51.894968+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:20:31.752935+0000","scrub_duration":0.32625951199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":22066,"num_write_kb":297298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"68'3918","reported_seq":4006,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:30.839354+0000","last_change":"2023-10-25T00:23:49.563900+0000","last_active":"2023-10-25T00:24:30.839354+0000","last_peered":"2023-10-25T00:24:30.839354+0000","last_clean":"2023-10-25T00:24:30.839354+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-25T00:24:30.839354+0000","last_undegraded":"2023-10-25T00:24:30.839354+0000","last_fullsized":"2023-10-25T00:24:30.839354+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3918","last_scrub_stamp":"2023-10-25T00:23:49.563617+0000","last_deep_scrub":"68'3918","last_deep_scrub_stamp":"2023-10-25T00:23:49.563617+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.563617+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:50:13.550847+0000","scrub_duration":0.092172528000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21675,"num_write_kb":247000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"72'4401","reported_seq":4486,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:42.768767+0000","last_change":"2023-10-25T00:23:42.768767+0000","last_active":"2023-10-25T00:23:42.768767+0000","last_peered":"2023-10-25T00:23:42.768767+0000","last_clean":"2023-10-25T00:23:42.768767+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-25T00:23:42.768767+0000","last_undegraded":"2023-10-25T00:23:42.768767+0000","last_fullsized":"2023-10-25T00:23:42.768767+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4401","last_scrub_stamp":"2023-10-25T00:23:42.768621+0000","last_deep_scrub":"72'4401","last_deep_scrub_stamp":"2023-10-25T00:23:42.768621+0000","last_clean_scrub_stamp":"2023-10-25T00:23:42.768621+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:23:18.678304+0000","scrub_duration":0.26783985300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":24294,"num_write_kb":326749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"68'3970","reported_seq":4057,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:30.839420+0000","last_change":"2023-10-25T00:23:12.428867+0000","last_active":"2023-10-25T00:24:30.839420+0000","last_peered":"2023-10-25T00:24:30.839420+0000","last_clean":"2023-10-25T00:24:30.839420+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-25T00:24:30.839420+0000","last_undegraded":"2023-10-25T00:24:30.839420+0000","last_fullsized":"2023-10-25T00:24:30.839420+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3970","last_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_deep_scrub":"68'3970","last_deep_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.428494+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:07.899437+0000","scrub_duration":0.13106953599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21202,"num_write_kb":257085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"72'5559","reported_seq":5650,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:41.090253+0000","last_change":"2023-10-25T00:23:41.090253+0000","last_active":"2023-10-25T00:23:41.090253+0000","last_peered":"2023-10-25T00:23:41.090253+0000","last_clean":"2023-10-25T00:23:41.090253+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-25T00:23:41.090253+0000","last_undegraded":"2023-10-25T00:23:41.090253+0000","last_fullsized":"2023-10-25T00:23:41.090253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5559","last_scrub_stamp":"2023-10-25T00:23:41.090104+0000","last_deep_scrub":"72'5559","last_deep_scrub_stamp":"2023-10-25T00:23:41.090104+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.090104+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:40:20.680440+0000","scrub_duration":0.241945666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":31202,"num_write_kb":391572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"68'4936","reported_seq":5024,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.429817+0000","last_change":"2023-10-25T00:23:02.021272+0000","last_active":"2023-10-25T00:24:31.429817+0000","last_peered":"2023-10-25T00:24:31.429817+0000","last_clean":"2023-10-25T00:24:31.429817+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-25T00:24:31.429817+0000","last_undegraded":"2023-10-25T00:24:31.429817+0000","last_fullsized":"2023-10-25T00:24:31.429817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4936","last_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_deep_scrub":"68'4936","last_deep_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.021108+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:22.652380+0000","scrub_duration":0.17348896599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26134,"num_write_kb":309652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"72'3558","reported_seq":3646,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:47.072846+0000","last_change":"2023-10-25T00:23:47.072846+0000","last_active":"2023-10-25T00:23:47.072846+0000","last_peered":"2023-10-25T00:23:47.072846+0000","last_clean":"2023-10-25T00:23:47.072846+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-25T00:23:47.072846+0000","last_undegraded":"2023-10-25T00:23:47.072846+0000","last_fullsized":"2023-10-25T00:23:47.072846+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3558","last_scrub_stamp":"2023-10-25T00:23:47.072625+0000","last_deep_scrub":"72'3558","last_deep_scrub_stamp":"2023-10-25T00:23:47.072625+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.072625+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:59:13.347978+0000","scrub_duration":0.34108986800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":34,"num_write":19715,"num_write_kb":243025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"68'5282","reported_seq":5370,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.110730+0000","last_change":"2023-10-25T00:23:14.290293+0000","last_active":"2023-10-25T00:24:31.110730+0000","last_peered":"2023-10-25T00:24:31.110730+0000","last_clean":"2023-10-25T00:24:31.110730+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-25T00:24:31.110730+0000","last_undegraded":"2023-10-25T00:24:31.110730+0000","last_fullsized":"2023-10-25T00:24:31.110730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5282","last_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_deep_scrub":"68'5282","last_deep_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.290099+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:44:38.139530+0000","scrub_duration":0.10140234200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27848,"num_write_kb":324561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"72'3914","reported_seq":4003,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:46.833446+0000","last_change":"2023-10-25T00:23:46.833446+0000","last_active":"2023-10-25T00:23:46.833446+0000","last_peered":"2023-10-25T00:23:46.833446+0000","last_clean":"2023-10-25T00:23:46.833446+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-25T00:23:46.833446+0000","last_undegraded":"2023-10-25T00:23:46.833446+0000","last_fullsized":"2023-10-25T00:23:46.833446+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3914","last_scrub_stamp":"2023-10-25T00:23:46.833294+0000","last_deep_scrub":"72'3914","last_deep_scrub_stamp":"2023-10-25T00:23:46.833294+0000","last_clean_scrub_stamp":"2023-10-25T00:23:46.833294+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:59.194933+0000","scrub_duration":0.30262013700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":20938,"num_write_kb":288796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"68'6616","reported_seq":6708,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.316585+0000","last_change":"2023-10-25T00:22:57.985278+0000","last_active":"2023-10-25T00:24:34.316585+0000","last_peered":"2023-10-25T00:24:34.316585+0000","last_clean":"2023-10-25T00:24:34.316585+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-25T00:24:34.316585+0000","last_undegraded":"2023-10-25T00:24:34.316585+0000","last_fullsized":"2023-10-25T00:24:34.316585+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6616","last_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_deep_scrub":"68'6616","last_deep_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.985096+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:14:23.491729+0000","scrub_duration":0.050039131000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36221,"num_write_kb":423916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"72'3536","reported_seq":3621,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:32.908624+0000","last_change":"2023-10-25T00:23:32.908624+0000","last_active":"2023-10-25T00:23:32.908624+0000","last_peered":"2023-10-25T00:23:32.908624+0000","last_clean":"2023-10-25T00:23:32.908624+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-25T00:23:32.908624+0000","last_undegraded":"2023-10-25T00:23:32.908624+0000","last_fullsized":"2023-10-25T00:23:32.908624+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3536","last_scrub_stamp":"2023-10-25T00:23:32.908476+0000","last_deep_scrub":"72'3536","last_deep_scrub_stamp":"2023-10-25T00:23:32.908476+0000","last_clean_scrub_stamp":"2023-10-25T00:23:32.908476+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:33:20.337271+0000","scrub_duration":0.30729883400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19276,"num_write_kb":253506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"68'4980","reported_seq":5065,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:10.321202+0000","last_change":"2023-10-25T00:23:10.321202+0000","last_active":"2023-10-25T00:23:10.321202+0000","last_peered":"2023-10-25T00:23:10.321202+0000","last_clean":"2023-10-25T00:23:10.321202+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-25T00:23:10.321202+0000","last_undegraded":"2023-10-25T00:23:10.321202+0000","last_fullsized":"2023-10-25T00:23:10.321202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4980","last_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_deep_scrub":"68'4980","last_deep_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:10.320988+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:34.745429+0000","scrub_duration":0.13750111800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26525,"num_write_kb":306963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"72'4160","reported_seq":4247,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:24.541711+0000","last_change":"2023-10-25T00:23:24.541711+0000","last_active":"2023-10-25T00:23:24.541711+0000","last_peered":"2023-10-25T00:23:24.541711+0000","last_clean":"2023-10-25T00:23:24.541711+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-25T00:23:24.541711+0000","last_undegraded":"2023-10-25T00:23:24.541711+0000","last_fullsized":"2023-10-25T00:23:24.541711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4160","last_scrub_stamp":"2023-10-25T00:23:24.541561+0000","last_deep_scrub":"72'4160","last_deep_scrub_stamp":"2023-10-25T00:23:24.541561+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.541561+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:52:13.060866+0000","scrub_duration":0.264226874,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":21819,"num_write_kb":288439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"68'4434","reported_seq":4521,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.266956+0000","last_change":"2023-10-25T00:22:47.255974+0000","last_active":"2023-10-25T00:23:06.266956+0000","last_peered":"2023-10-25T00:23:06.266956+0000","last_clean":"2023-10-25T00:23:06.266956+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-25T00:23:06.266956+0000","last_undegraded":"2023-10-25T00:23:06.266956+0000","last_fullsized":"2023-10-25T00:23:06.266956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4434","last_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_deep_scrub":"68'4434","last_deep_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.255646+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:35:52.723567+0000","scrub_duration":0.14955275700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23791,"num_write_kb":292031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"72'4195","reported_seq":4277,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.837002+0000","last_change":"2023-10-25T00:23:37.837002+0000","last_active":"2023-10-25T00:23:37.837002+0000","last_peered":"2023-10-25T00:23:37.837002+0000","last_clean":"2023-10-25T00:23:37.837002+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-25T00:23:37.837002+0000","last_undegraded":"2023-10-25T00:23:37.837002+0000","last_fullsized":"2023-10-25T00:23:37.837002+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4195","last_scrub_stamp":"2023-10-25T00:23:37.836848+0000","last_deep_scrub":"72'4195","last_deep_scrub_stamp":"2023-10-25T00:23:37.836848+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.836848+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:49:10.177488+0000","scrub_duration":0.28694403499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23000,"num_write_kb":285019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"68'4243","reported_seq":4329,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.302534+0000","last_change":"2023-10-25T00:23:00.969067+0000","last_active":"2023-10-25T00:23:06.302534+0000","last_peered":"2023-10-25T00:23:06.302534+0000","last_clean":"2023-10-25T00:23:06.302534+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-25T00:23:06.302534+0000","last_undegraded":"2023-10-25T00:23:06.302534+0000","last_fullsized":"2023-10-25T00:23:06.302534+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4243","last_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_deep_scrub":"68'4243","last_deep_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.968692+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:45.877638+0000","scrub_duration":0.15459757700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22439,"num_write_kb":266094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"72'4017","reported_seq":4100,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:54.963064+0000","last_change":"2023-10-25T00:23:54.963064+0000","last_active":"2023-10-25T00:23:54.963064+0000","last_peered":"2023-10-25T00:23:54.963064+0000","last_clean":"2023-10-25T00:23:54.963064+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-25T00:23:54.963064+0000","last_undegraded":"2023-10-25T00:23:54.963064+0000","last_fullsized":"2023-10-25T00:23:54.963064+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4017","last_scrub_stamp":"2023-10-25T00:23:54.962911+0000","last_deep_scrub":"72'4017","last_deep_scrub_stamp":"2023-10-25T00:23:54.962911+0000","last_clean_scrub_stamp":"2023-10-25T00:23:54.962911+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:49:43.769503+0000","scrub_duration":0.29685359700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22665,"num_write_kb":291707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"68'4509","reported_seq":4598,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.489031+0000","last_change":"2023-10-25T00:22:44.022792+0000","last_active":"2023-10-25T00:23:06.489031+0000","last_peered":"2023-10-25T00:23:06.489031+0000","last_clean":"2023-10-25T00:23:06.489031+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-25T00:23:06.489031+0000","last_undegraded":"2023-10-25T00:23:06.489031+0000","last_fullsized":"2023-10-25T00:23:06.489031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4509","last_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_deep_scrub":"68'4509","last_deep_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.022573+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:54:30.651361+0000","scrub_duration":0.11145192700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23158,"num_write_kb":297883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"72'5235","reported_seq":5327,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.569964+0000","last_change":"2023-10-25T00:23:50.831958+0000","last_active":"2023-10-25T00:24:32.569964+0000","last_peered":"2023-10-25T00:24:32.569964+0000","last_clean":"2023-10-25T00:24:32.569964+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-25T00:24:32.569964+0000","last_undegraded":"2023-10-25T00:24:32.569964+0000","last_fullsized":"2023-10-25T00:24:32.569964+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5235","last_scrub_stamp":"2023-10-25T00:23:50.831798+0000","last_deep_scrub":"72'5235","last_deep_scrub_stamp":"2023-10-25T00:23:50.831798+0000","last_clean_scrub_stamp":"2023-10-25T00:23:50.831798+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:34:32.587906+0000","scrub_duration":0.30673944400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":15,"num_write":29235,"num_write_kb":367391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"68'4260","reported_seq":4350,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:10.674279+0000","last_change":"2023-10-25T00:22:56.486314+0000","last_active":"2023-10-25T00:23:10.674279+0000","last_peered":"2023-10-25T00:23:10.674279+0000","last_clean":"2023-10-25T00:23:10.674279+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-25T00:23:10.674279+0000","last_undegraded":"2023-10-25T00:23:10.674279+0000","last_fullsized":"2023-10-25T00:23:10.674279+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4260","last_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_deep_scrub":"68'4260","last_deep_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.485966+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:51.396462+0000","scrub_duration":0.116368769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23357,"num_write_kb":265837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"72'4656","reported_seq":4750,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.570019+0000","last_change":"2023-10-25T00:23:40.632314+0000","last_active":"2023-10-25T00:24:32.570019+0000","last_peered":"2023-10-25T00:24:32.570019+0000","last_clean":"2023-10-25T00:24:32.570019+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-25T00:24:32.570019+0000","last_undegraded":"2023-10-25T00:24:32.570019+0000","last_fullsized":"2023-10-25T00:24:32.570019+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4656","last_scrub_stamp":"2023-10-25T00:23:40.632160+0000","last_deep_scrub":"72'4656","last_deep_scrub_stamp":"2023-10-25T00:23:40.632160+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.632160+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:23:41.024594+0000","scrub_duration":0.095096767999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":26109,"num_write_kb":317781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"68'4636","reported_seq":4728,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.427464+0000","last_change":"2023-10-25T00:22:39.672588+0000","last_active":"2023-10-25T00:24:21.427464+0000","last_peered":"2023-10-25T00:24:21.427464+0000","last_clean":"2023-10-25T00:24:21.427464+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-25T00:24:21.427464+0000","last_undegraded":"2023-10-25T00:24:21.427464+0000","last_fullsized":"2023-10-25T00:24:21.427464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4636","last_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_deep_scrub":"68'4636","last_deep_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.672438+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:47:15.433547+0000","scrub_duration":0.124153799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25124,"num_write_kb":285314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"72'3117","reported_seq":3205,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:41.679657+0000","last_change":"2023-10-25T00:23:41.679657+0000","last_active":"2023-10-25T00:23:41.679657+0000","last_peered":"2023-10-25T00:23:41.679657+0000","last_clean":"2023-10-25T00:23:41.679657+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-25T00:23:41.679657+0000","last_undegraded":"2023-10-25T00:23:41.679657+0000","last_fullsized":"2023-10-25T00:23:41.679657+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3117","last_scrub_stamp":"2023-10-25T00:23:41.679268+0000","last_deep_scrub":"72'3117","last_deep_scrub_stamp":"2023-10-25T00:23:41.679268+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.679268+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:35.283156+0000","scrub_duration":0.30581881399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17240,"num_write_kb":232937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"68'5520","reported_seq":5619,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.108493+0000","last_change":"2023-10-25T00:22:56.366641+0000","last_active":"2023-10-25T00:24:21.108493+0000","last_peered":"2023-10-25T00:24:21.108493+0000","last_clean":"2023-10-25T00:24:21.108493+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-25T00:24:21.108493+0000","last_undegraded":"2023-10-25T00:24:21.108493+0000","last_fullsized":"2023-10-25T00:24:21.108493+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5520","last_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_deep_scrub":"68'5520","last_deep_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.366419+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:47:59.889167+0000","scrub_duration":0.104594514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29665,"num_write_kb":368823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"72'3962","reported_seq":4049,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.509842+0000","last_change":"2023-10-25T00:23:34.105899+0000","last_active":"2023-10-25T00:24:31.509842+0000","last_peered":"2023-10-25T00:24:31.509842+0000","last_clean":"2023-10-25T00:24:31.509842+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-25T00:24:31.509842+0000","last_undegraded":"2023-10-25T00:24:31.509842+0000","last_fullsized":"2023-10-25T00:24:31.509842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3962","last_scrub_stamp":"2023-10-25T00:23:34.105758+0000","last_deep_scrub":"72'3962","last_deep_scrub_stamp":"2023-10-25T00:23:34.105758+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.105758+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:35:57.960582+0000","scrub_duration":0.32704070499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21989,"num_write_kb":280762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"68'4908","reported_seq":5002,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:20.837646+0000","last_change":"2023-10-25T00:23:00.408006+0000","last_active":"2023-10-25T00:24:20.837646+0000","last_peered":"2023-10-25T00:24:20.837646+0000","last_clean":"2023-10-25T00:24:20.837646+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-25T00:24:20.837646+0000","last_undegraded":"2023-10-25T00:24:20.837646+0000","last_fullsized":"2023-10-25T00:24:20.837646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4908","last_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_deep_scrub":"68'4908","last_deep_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.407650+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:48:18.113931+0000","scrub_duration":0.051805808000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26829,"num_write_kb":320697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":320,"num_write":1482516,"num_write_kb":19060243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267718,"ondisk_log_size":267718,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":19490997,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1686,"num_read_kb":209824,"num_write":4665,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39043072,"data_stored":38981994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4562,"ondisk_log_size":4562,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":0,"num_write":1661441,"num_write_kb":19773626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307026,"ondisk_log_size":307026,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":20061533,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1707,"num_read_kb":286722,"num_write":4630,"num_write_kb":21638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40181760,"data_stored":40123066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4539,"ondisk_log_size":4539,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163,"ondisk_log_size":163,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575232,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919112,"kb_used_data":12352,"kb_used_omap":0,"kb_used_meta":906752,"kb_avail":93452728,"statfs":{"total":96636764160,"available":95695593472,"internally_reserved":0,"allocated":12648448,"data_stored":12164710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":928514048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.79600000000000004,"15min":0.94599999999999995},"min":{"1min":0.50600000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.232,"5min":3.6349999999999998,"15min":13.929},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.008,"5min":0.86599999999999999,"15min":0.94699999999999995},"min":{"1min":0.59299999999999997,"5min":0.27600000000000002,"15min":0.20999999999999999},"max":{"1min":1.3200000000000001,"5min":3.6659999999999999,"15min":11.315},"last":0.97799999999999998}]},{"osd":1,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.80900000000000005,"15min":1.161},"min":{"1min":0.54900000000000004,"5min":0.23599999999999999,"15min":0.215},"max":{"1min":1.173,"5min":3.6949999999999998,"15min":28.308},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.0740000000000001,"5min":0.94399999999999995,"15min":1.24},"min":{"1min":0.78200000000000003,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":1.244,"5min":3.7490000000000001,"15min":28.346},"last":1.0980000000000001}]},{"osd":2,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":1.0089999999999999,"15min":0.99299999999999999},"min":{"1min":0.58099999999999996,"5min":0.28499999999999998,"15min":0.22800000000000001},"max":{"1min":1.448,"5min":5.71,"15min":11.430999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.93400000000000005,"15min":0.96099999999999997},"min":{"1min":0.58699999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.1990000000000001,"5min":3.8799999999999999,"15min":7.9710000000000001},"last":0.73999999999999999}]},{"osd":3,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.89200000000000002,"15min":1.3620000000000001},"min":{"1min":0.748,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":1.2749999999999999,"5min":3.6150000000000002,"15min":93.664000000000001},"last":0.871},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.042,"15min":1.3879999999999999},"min":{"1min":0.72199999999999998,"5min":0.32700000000000001,"15min":0.25700000000000001},"max":{"1min":1.2729999999999999,"5min":4.6630000000000003,"15min":93.073999999999998},"last":1.05}]},{"osd":4,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.91600000000000004,"15min":0.88400000000000001},"min":{"1min":0.66000000000000003,"5min":0.314,"15min":0.249},"max":{"1min":1.274,"5min":3.0569999999999999,"15min":7.6280000000000001},"last":1.141},{"interface":"front","average":{"1min":1.05,"5min":1.0940000000000001,"15min":1.157},"min":{"1min":0.81000000000000005,"5min":0.36599999999999999,"15min":0.27200000000000002},"max":{"1min":1.3640000000000001,"5min":4.9640000000000004,"15min":27.936},"last":0.83899999999999997}]},{"osd":5,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.3480000000000001,"15min":1.0449999999999999},"min":{"1min":0.59499999999999997,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":1.5,"5min":21.126999999999999,"15min":21.126999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.0369999999999999,"5min":0.95399999999999996,"15min":0.96799999999999997},"min":{"1min":0.65200000000000002,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":1.3740000000000001,"5min":3.8069999999999999,"15min":11.49},"last":1.1679999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:23:45 2023","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.3280000000000001,"15min":1.1140000000000001},"min":{"1min":0.71099999999999997,"5min":0.311,"15min":0.23599999999999999},"max":{"1min":1.2989999999999999,"5min":4.8140000000000001,"15min":8.2959999999999994},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.4419999999999999,"15min":1.228},"min":{"1min":1.0189999999999999,"5min":0.45500000000000002,"15min":0.35499999999999998},"max":{"1min":1.4219999999999999,"5min":4.8639999999999999,"15min":20.651},"last":1.214}]}]},{"osd":6,"up_from":13,"seq":55834575232,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859036,"kb_used_data":5076,"kb_used_omap":0,"kb_used_meta":853952,"kb_avail":93512804,"statfs":{"total":96636764160,"available":95757111296,"internally_reserved":0,"allocated":5197824,"data_stored":4751224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":874446848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.91800000000000004,"15min":0.95399999999999996},"min":{"1min":0.39700000000000002,"5min":0.185,"15min":0.185},"max":{"1min":1.194,"5min":5.3490000000000002,"15min":15.974},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.036,"15min":0.97999999999999998},"min":{"1min":0.54400000000000004,"5min":0.218,"15min":0.218},"max":{"1min":1.115,"5min":8.2400000000000002,"15min":8.4789999999999992},"last":0.92900000000000005}]},{"osd":1,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.81899999999999995,"15min":0.88300000000000001},"min":{"1min":0.57699999999999996,"5min":0.221,"15min":0.221},"max":{"1min":1.0469999999999999,"5min":3.5630000000000002,"15min":7.6360000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.878,"15min":0.98899999999999999},"min":{"1min":0.54000000000000004,"5min":0.254,"15min":0.249},"max":{"1min":1.111,"5min":3.5840000000000001,"15min":21.600999999999999},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.92700000000000005,"15min":1.0620000000000001},"min":{"1min":0.59999999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.198,"5min":3.6779999999999999,"15min":12.615},"last":1.224},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.93400000000000005,"15min":1.1060000000000001},"min":{"1min":0.47599999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.2370000000000001,"5min":3.702,"15min":12.667999999999999},"last":0.78300000000000003}]},{"osd":3,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":1.0009999999999999,"15min":1.248},"min":{"1min":0.68600000000000005,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.071,"5min":5.7350000000000003,"15min":38.649999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.0580000000000001,"15min":1.325},"min":{"1min":0.67000000000000004,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.1699999999999999,"5min":5.766,"15min":38.683},"last":1.2490000000000001}]},{"osd":4,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":1.3460000000000001,"15min":1.1240000000000001},"min":{"1min":0.56999999999999995,"5min":0.28399999999999997,"15min":0.23799999999999999},"max":{"1min":1.284,"5min":8.9930000000000003,"15min":12.682},"last":1.175},{"interface":"front","average":{"1min":0.93200000000000005,"5min":1.2609999999999999,"15min":1.081},"min":{"1min":0.63600000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.2190000000000001,"5min":5.694,"15min":10.686},"last":1.2829999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.0309999999999999,"15min":0.95699999999999996},"min":{"1min":0.622,"5min":0.28399999999999997,"15min":0.24299999999999999},"max":{"1min":1.1619999999999999,"5min":3.6379999999999999,"15min":8.0039999999999996},"last":1.306},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.1759999999999999,"15min":1.0649999999999999},"min":{"1min":0.69799999999999995,"5min":0.34499999999999997,"15min":0.30299999999999999},"max":{"1min":1.244,"5min":4.0679999999999996,"15min":11.766999999999999},"last":1.3320000000000001}]},{"osd":7,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.177,"15min":1.0249999999999999},"min":{"1min":0.64200000000000002,"5min":0.27000000000000002,"15min":0.251},"max":{"1min":1.159,"5min":8.2720000000000002,"15min":12.926},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.234,"15min":1.0489999999999999},"min":{"1min":0.79400000000000004,"5min":0.34200000000000003,"15min":0.28799999999999998},"max":{"1min":1.2230000000000001,"5min":6.1100000000000003,"15min":7.6619999999999999},"last":1.381}]}]},{"osd":5,"up_from":13,"seq":55834575232,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":958888,"kb_used_data":9184,"kb_used_omap":0,"kb_used_meta":949696,"kb_avail":93412952,"statfs":{"total":96636764160,"available":95654862848,"internally_reserved":0,"allocated":9404416,"data_stored":8945710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":972488704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":1.133,"15min":1.038},"min":{"1min":0.505,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.97499999999999998,"5min":14.837999999999999,"15min":14.837999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.83899999999999997,"15min":0.97399999999999998},"min":{"1min":0.56799999999999995,"5min":0.20699999999999999,"15min":0.20100000000000001},"max":{"1min":1.427,"5min":4.016,"15min":7.423},"last":0.93200000000000005}]},{"osd":1,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":1.0489999999999999,"15min":1.0880000000000001},"min":{"1min":0.51600000000000001,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":1.1879999999999999,"5min":12.141,"15min":12.141},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.94399999999999995,"15min":1.0940000000000001},"min":{"1min":0.746,"5min":0.26400000000000001,"15min":0.252},"max":{"1min":0.999,"5min":9.0549999999999997,"15min":24.648},"last":1.1040000000000001}]},{"osd":2,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.0840000000000001,"15min":1.2490000000000001},"min":{"1min":0.67200000000000004,"5min":0.25,"15min":0.25},"max":{"1min":1.258,"5min":12.175000000000001,"15min":20.760999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.117,"15min":1.073},"min":{"1min":0.68899999999999995,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.2909999999999999,"5min":12.109999999999999,"15min":20.552},"last":1.0509999999999999}]},{"osd":3,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.3109999999999999,"15min":1.2130000000000001},"min":{"1min":0.67300000000000004,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":1.1599999999999999,"5min":12.013,"15min":12.013},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.036,"15min":1.244},"min":{"1min":0.65500000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0960000000000001,"5min":4.0490000000000004,"15min":12.805},"last":0.97899999999999998}]},{"osd":4,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.3520000000000001,"15min":1.2030000000000001},"min":{"1min":0.61199999999999999,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":1.278,"5min":12.201000000000001,"15min":12.728},"last":1.2250000000000001},{"interface":"front","average":{"1min":0.997,"5min":1.22,"15min":1.137},"min":{"1min":0.64600000000000002,"5min":0.29299999999999998,"15min":0.27200000000000002},"max":{"1min":1.3100000000000001,"5min":7.6470000000000002,"15min":10.943},"last":1.1180000000000001}]},{"osd":6,"last update":"Wed Oct 25 00:24:04 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.1919999999999999,"15min":1.0609999999999999},"min":{"1min":0.78600000000000003,"5min":0.34399999999999997,"15min":0.26400000000000001},"max":{"1min":1.4079999999999999,"5min":9.6489999999999991,"15min":17.936},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.3680000000000001,"15min":1.216},"min":{"1min":0.71399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.387,"5min":13.029999999999999,"15min":14.766999999999999},"last":0.875}]},{"osd":7,"last update":"Wed Oct 25 00:23:36 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.617,"15min":1.246},"min":{"1min":0.81699999999999995,"5min":0.33700000000000002,"15min":0.29499999999999998},"max":{"1min":1.6339999999999999,"5min":28.178999999999998,"15min":28.178999999999998},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.72,"15min":1.3700000000000001},"min":{"1min":1.0169999999999999,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":1.391,"5min":28.222999999999999,"15min":28.222999999999999},"last":1.1779999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575231,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946324,"kb_used_data":19148,"kb_used_omap":0,"kb_used_meta":927168,"kb_avail":93425516,"statfs":{"total":96636764160,"available":95667728384,"internally_reserved":0,"allocated":19607552,"data_stored":19135740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":949420032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.88900000000000001,"15min":0.98299999999999998},"min":{"1min":0.503,"5min":0.19700000000000001,"15min":0.187},"max":{"1min":1.1020000000000001,"5min":11.629,"15min":20.922999999999998},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.73599999999999999,"15min":0.97799999999999998},"min":{"1min":0.52400000000000002,"5min":0.247,"15min":0.217},"max":{"1min":1.0920000000000001,"5min":7.0830000000000002,"15min":20.532},"last":0.73899999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.879,"15min":0.98399999999999999},"min":{"1min":0.61399999999999999,"5min":0.22,"15min":0.22},"max":{"1min":1.1819999999999999,"5min":3.0169999999999999,"15min":20.914000000000001},"last":1.3100000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.92700000000000005,"15min":1.167},"min":{"1min":0.64600000000000002,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":1.216,"5min":6.1890000000000001,"15min":23.265999999999998},"last":0.99399999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.84799999999999998,"15min":1.117},"min":{"1min":0.63900000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.181,"5min":7.1550000000000002,"15min":20.774000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.1899999999999999,"15min":1.2529999999999999},"min":{"1min":0.65400000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.179,"5min":7.9009999999999998,"15min":20.817},"last":0.83999999999999997}]},{"osd":3,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.1779999999999999,"15min":1.169},"min":{"1min":0.64800000000000002,"5min":0.314,"15min":0.28799999999999998},"max":{"1min":1.1879999999999999,"5min":7.3620000000000001,"15min":24.613},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.96099999999999997,"5min":0.92900000000000005,"15min":1.1499999999999999},"min":{"1min":0.69699999999999995,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":1.2,"5min":7.2009999999999996,"15min":24.071000000000002},"last":1.079}]},{"osd":5,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":1.153,"15min":0.93000000000000005},"min":{"1min":0.623,"5min":0.28599999999999998,"15min":0.23999999999999999},"max":{"1min":1.2170000000000001,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.0189999999999999,"15min":0.94099999999999995},"min":{"1min":0.65000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.0669999999999999,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":1.208}]},{"osd":6,"last update":"Wed Oct 25 00:23:41 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.1839999999999999,"15min":1.0800000000000001},"min":{"1min":0.73599999999999999,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":1.175,"5min":4.6550000000000002,"15min":6.5220000000000002},"last":1.266},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.0329999999999999,"15min":0.97199999999999998},"min":{"1min":0.81599999999999995,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":1.234,"5min":7.2389999999999999,"15min":7.2389999999999999},"last":1.153}]},{"osd":7,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.3009999999999999,"15min":1.0700000000000001},"min":{"1min":0.72599999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.1990000000000001,"5min":7.4020000000000001,"15min":7.9850000000000003},"last":1.107},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.4910000000000001,"15min":1.163},"min":{"1min":0.98799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.2909999999999999,"5min":7.444,"15min":7.444},"last":1.2330000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575232,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160764,"kb_used_data":9268,"kb_used_omap":0,"kb_used_meta":1151488,"kb_avail":93211076,"statfs":{"total":96636764160,"available":95448141824,"internally_reserved":0,"allocated":9490432,"data_stored":9030287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1179123712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.46400000000000002,"15min":0.78800000000000003},"min":{"1min":0.54000000000000004,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.82599999999999996,"5min":1.3899999999999999,"15min":13.327999999999999},"last":0.872},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.58899999999999997,"15min":0.73699999999999999},"min":{"1min":0.48199999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.2170000000000001,"5min":1.444,"15min":5.4580000000000002},"last":0.78800000000000003}]},{"osd":1,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.67700000000000005,"15min":0.86799999999999999},"min":{"1min":0.41499999999999998,"5min":0.20200000000000001,"15min":0.19900000000000001},"max":{"1min":0.93400000000000005,"5min":4.4459999999999997,"15min":13.831},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.72299999999999998,"15min":0.76700000000000002},"min":{"1min":0.441,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.92200000000000004,"5min":4.2130000000000001,"15min":4.4500000000000002},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.66300000000000003,"15min":0.875},"min":{"1min":0.504,"5min":0.223,"15min":0.221},"max":{"1min":0.97699999999999998,"5min":2.7890000000000001,"15min":8.2520000000000007},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.71699999999999997,"15min":0.91000000000000003},"min":{"1min":0.35899999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0720000000000001,"5min":2.7360000000000002,"15min":6.6840000000000002},"last":0.69499999999999995}]},{"osd":4,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90700000000000003,"15min":1.206},"min":{"1min":0.56200000000000006,"5min":0.316,"15min":0.26800000000000002},"max":{"1min":1.1419999999999999,"5min":3.9449999999999998,"15min":39.070999999999998},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.2210000000000001,"15min":1.2589999999999999},"min":{"1min":0.65100000000000002,"5min":0.28599999999999998,"15min":0.25700000000000001},"max":{"1min":1.175,"5min":7.6180000000000003,"15min":55.104999999999997},"last":0.73999999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":0.94999999999999996,"15min":1.1299999999999999},"min":{"1min":0.68100000000000005,"5min":0.312,"15min":0.28399999999999997},"max":{"1min":1.597,"5min":4.1849999999999996,"15min":31.481999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.028,"15min":1.2210000000000001},"min":{"1min":0.81999999999999995,"5min":0.38300000000000001,"15min":0.32500000000000001},"max":{"1min":1.5580000000000001,"5min":4.2060000000000004,"15min":31.512},"last":1.119}]},{"osd":6,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.3140000000000001,"15min":1.252},"min":{"1min":0.72199999999999998,"5min":0.311,"15min":0.27800000000000002},"max":{"1min":1.5109999999999999,"5min":15.824,"15min":25.809999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.117,"15min":1.1830000000000001},"min":{"1min":0.873,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.518,"5min":4.7779999999999996,"15min":24.036999999999999},"last":0.93400000000000005}]},{"osd":7,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.0089999999999999,"15min":1.2330000000000001},"min":{"1min":0.83599999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.5840000000000001,"5min":4.8529999999999998,"15min":27.581},"last":1.073},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.077,"15min":1.252},"min":{"1min":0.76500000000000001,"5min":0.374,"15min":0.374},"max":{"1min":1.5600000000000001,"5min":8.5259999999999998,"15min":24.332000000000001},"last":1.0049999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575232,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946076,"kb_used_data":5076,"kb_used_omap":0,"kb_used_meta":940992,"kb_avail":93425764,"statfs":{"total":96636764160,"available":95667982336,"internally_reserved":0,"allocated":5197824,"data_stored":4750712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":963575808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61399999999999999,"15min":0.98999999999999999},"min":{"1min":0.41199999999999998,"5min":0.16,"15min":0.16},"max":{"1min":1.204,"5min":5.29,"15min":29.344000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.68000000000000005,"15min":0.84599999999999997},"min":{"1min":0.51800000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.0589999999999999,"5min":6.4290000000000003,"15min":10.755000000000001},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.66600000000000004,"15min":0.93799999999999994},"min":{"1min":0.49299999999999999,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":1.032,"5min":3.0409999999999999,"15min":18.617999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.69799999999999995,"15min":0.879},"min":{"1min":0.44700000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.2909999999999999,"5min":3.649,"15min":15.064},"last":0.624}]},{"osd":3,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.76100000000000001,"15min":0.90200000000000002},"min":{"1min":0.45100000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.93899999999999995,"5min":13.034000000000001,"15min":13.034000000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.78600000000000003,"15min":1.006},"min":{"1min":0.59199999999999997,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3,"5min":3.1339999999999999,"15min":18.547999999999998},"last":0.96499999999999997}]},{"osd":4,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.0700000000000001,"15min":1.1950000000000001},"min":{"1min":0.69499999999999995,"5min":0.252,"15min":0.252},"max":{"1min":1.377,"5min":7.6500000000000004,"15min":33.378999999999998},"last":1.26},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.135,"15min":1.2889999999999999},"min":{"1min":0.79000000000000004,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.464,"5min":7.6779999999999999,"15min":33.558999999999997},"last":1.0820000000000001}]},{"osd":5,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.069,"15min":1.472},"min":{"1min":0.71699999999999997,"5min":0.32000000000000001,"15min":0.29099999999999998},"max":{"1min":1.3460000000000001,"5min":6.3979999999999997,"15min":34.545999999999999},"last":1.115},{"interface":"front","average":{"1min":0.98499999999999999,"5min":0.94099999999999995,"15min":1.177},"min":{"1min":0.64600000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.343,"5min":4.4829999999999997,"15min":36.104999999999997},"last":1.337}]},{"osd":6,"last update":"Wed Oct 25 00:23:48 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.03,"15min":1.1140000000000001},"min":{"1min":0.81499999999999995,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.224,"5min":7.5030000000000001,"15min":33.661999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.105,"15min":1.198},"min":{"1min":0.97999999999999998,"5min":0.371,"15min":0.371},"max":{"1min":1.417,"5min":7.5490000000000004,"15min":34.043999999999997},"last":1.0289999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:23:38 2023","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.1319999999999999,"15min":1.234},"min":{"1min":0.80400000000000005,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.4239999999999999,"5min":7.5899999999999999,"15min":33.307000000000002},"last":1.381},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2,"15min":1.306},"min":{"1min":0.752,"5min":0.371,"15min":0.371},"max":{"1min":1.6579999999999999,"5min":7.6189999999999998,"15min":33.344999999999999},"last":1.425}]}]},{"osd":1,"up_from":13,"seq":55834575233,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054160,"kb_used_data":4104,"kb_used_omap":0,"kb_used_meta":1050048,"kb_avail":93317680,"statfs":{"total":96636764160,"available":95557304320,"internally_reserved":0,"allocated":4202496,"data_stored":3754629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1075249152},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.59099999999999997,"15min":0.82699999999999996},"min":{"1min":0.40799999999999997,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.0129999999999999,"5min":3.6110000000000002,"15min":21.334},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.71599999999999997,"15min":0.85899999999999999},"min":{"1min":0.26200000000000001,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":1.0549999999999999,"5min":3.4809999999999999,"15min":21.276},"last":0.79800000000000004}]},{"osd":2,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.67400000000000004,"15min":0.81000000000000005},"min":{"1min":0.48599999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.95199999999999996,"5min":8.1600000000000001,"15min":11.420999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.68400000000000005,"15min":0.83999999999999997},"min":{"1min":0.51600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.0009999999999999,"5min":3.1760000000000002,"15min":11.648},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78600000000000003,"15min":0.92500000000000004},"min":{"1min":0.58699999999999997,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":1.1299999999999999,"5min":3.46,"15min":14.346},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.88800000000000001,"15min":1.026},"min":{"1min":0.71099999999999997,"5min":0.26300000000000001,"15min":0.248},"max":{"1min":1.2090000000000001,"5min":3.4969999999999999,"15min":14.379},"last":1.0249999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.038,"15min":1.0589999999999999},"min":{"1min":0.51000000000000001,"5min":0.28000000000000003,"15min":0.27200000000000002},"max":{"1min":1.093,"5min":5.2409999999999997,"15min":10.582000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.038,"5min":1.0800000000000001,"15min":1.0580000000000001},"min":{"1min":0.82899999999999996,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":1.246,"5min":4.6630000000000003,"15min":11.173},"last":0.90600000000000003}]},{"osd":5,"last update":"Wed Oct 25 00:23:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.92400000000000004,"15min":1.0529999999999999},"min":{"1min":0.79500000000000004,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.228,"5min":5.0309999999999997,"15min":11.577999999999999},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.1120000000000001,"5min":0.92600000000000005,"15min":0.97799999999999998},"min":{"1min":0.69699999999999995,"5min":0.29899999999999999,"15min":0.29399999999999998},"max":{"1min":1.3480000000000001,"5min":2.7469999999999999,"15min":10.018000000000001},"last":0.93500000000000005}]},{"osd":6,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.999,"15min":1.028},"min":{"1min":0.71999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.1739999999999999,"5min":7.4880000000000004,"15min":11.845000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.095,"5min":1.0860000000000001,"15min":1.125},"min":{"1min":0.77100000000000002,"5min":0.38300000000000001,"15min":0.33800000000000002},"max":{"1min":1.262,"5min":7.5220000000000002,"15min":11.919},"last":1.0509999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.1839999999999999,"15min":1.1279999999999999},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":1.2749999999999999,"5min":7.9909999999999997,"15min":11.881},"last":1},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.95799999999999996,"15min":1.1439999999999999},"min":{"1min":0.56899999999999995,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4610000000000001,"5min":4.2300000000000004,"15min":10.478},"last":1.087}]}]},{"osd":0,"up_from":13,"seq":55834575233,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":966392,"kb_used_data":22128,"kb_used_omap":0,"kb_used_meta":944256,"kb_avail":93405448,"statfs":{"total":96636764160,"available":95647178752,"internally_reserved":0,"allocated":22659072,"data_stored":22203488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":966918144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60799999999999998,"15min":1},"min":{"1min":0.40200000000000002,"5min":0.18099999999999999,"15min":0.16900000000000001},"max":{"1min":0.90800000000000003,"5min":2.544,"15min":30.065999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.63500000000000001,"15min":0.997},"min":{"1min":0.503,"5min":0.184,"15min":0.182},"max":{"1min":0.90700000000000003,"5min":3.0030000000000001,"15min":19.757999999999999},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73099999999999998,"15min":0.79700000000000004},"min":{"1min":0.46800000000000003,"5min":0.22900000000000001,"15min":0.20399999999999999},"max":{"1min":0.996,"5min":5.9740000000000002,"15min":15.324999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.72299999999999998,"15min":0.96999999999999997},"min":{"1min":0.626,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.1040000000000001,"5min":4.2549999999999999,"15min":30.283999999999999},"last":0.497}]},{"osd":3,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.84399999999999997,"15min":1.1040000000000001},"min":{"1min":0.59699999999999998,"5min":0.20999999999999999,"15min":0.191},"max":{"1min":1.1060000000000001,"5min":6.3570000000000002,"15min":22.966999999999999},"last":0.374},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.76400000000000001,"15min":1.083},"min":{"1min":0.58899999999999997,"5min":0.25800000000000001,"15min":0.23599999999999999},"max":{"1min":1.2090000000000001,"5min":2.5619999999999998,"15min":19.460999999999999},"last":0.45400000000000001}]},{"osd":4,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.89100000000000001,"15min":1.1279999999999999},"min":{"1min":0.63200000000000001,"5min":0.33700000000000002,"15min":0.26600000000000001},"max":{"1min":1.2370000000000001,"5min":3.758,"15min":8.3670000000000009},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.016,"15min":1.2310000000000001},"min":{"1min":0.67800000000000005,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.286,"5min":5.3719999999999999,"15min":13.023999999999999},"last":0.96599999999999997}]},{"osd":5,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.96999999999999997,"15min":1.0229999999999999},"min":{"1min":0.751,"5min":0.30399999999999999,"15min":0.29099999999999998},"max":{"1min":1.04,"5min":2.9710000000000001,"15min":10.683999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.097,"5min":1.01,"15min":1},"min":{"1min":0.79100000000000004,"5min":0.34999999999999998,"15min":0.29899999999999999},"max":{"1min":1.5469999999999999,"5min":4.8860000000000001,"15min":15.436999999999999},"last":0.72399999999999998}]},{"osd":6,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.98899999999999999,"15min":1.1339999999999999},"min":{"1min":0.78200000000000003,"5min":0.33700000000000002,"15min":0.28100000000000003},"max":{"1min":1.7969999999999999,"5min":6.1559999999999997,"15min":8.7260000000000009},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.125,"15min":1.294},"min":{"1min":0.86599999999999999,"5min":0.38200000000000001,"15min":0.33600000000000002},"max":{"1min":1.6379999999999999,"5min":7.8970000000000002,"15min":15.375},"last":0.66600000000000004}]},{"osd":7,"last update":"Wed Oct 25 00:23:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.2509999999999999,"15min":1.159},"min":{"1min":0.80000000000000004,"5min":0.34499999999999997,"15min":0.33700000000000002},"max":{"1min":1.202,"5min":5.5090000000000003,"15min":10.81},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.155,"5min":1.038,"15min":1.27},"min":{"1min":0.91800000000000004,"5min":0.313,"15min":0.313},"max":{"1min":1.506,"5min":4.2629999999999999,"15min":11.417},"last":0.75800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12673024,"data_stored":12668140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3198976,"data_stored":3198201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11689984,"data_stored":11672183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4195192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":8388642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4279555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6918144,"data_stored":6907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6836224,"data_stored":6822812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-25T00:24:40.258 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2023-10-24T23:53:11.386581+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=24, tm_hour=23, tm_min=53, tm_sec=11, tm_wday=1, tm_yday=297, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=25, tm_hour=0, tm_min=22, tm_sec=32, tm_wday=2, tm_yday=298, tm_isdst=0) 2023-10-25T00:24:40.260 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-25T00:24:42.535 INFO:tasks.ceph.mds.c.smithi107.stderr:2023-10-25T00:24:42.529+0000 7f0e2b592640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:24:43.151 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2023-10-25T00:24:43.151 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-25T00:24:43.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-25T00:24:43.154 INFO:tasks.ceph.mds.a:Started 2023-10-25T00:24:43.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:24:43.227 INFO:tasks.ceph.mds.a.smithi107.stdout:starting mds.a at 2023-10-25T00:24:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f7085aa9640 1 Processor -- start 2023-10-25T00:24:43.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f7085aa9640 1 -- start start 2023-10-25T00:24:43.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f7085aa9640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7080158970 0x7f708006a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:43.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f7085aa9640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f708006cea0 con 0x7f7080065bc0 2023-10-25T00:24:43.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f7085aa9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70801584e0 con 0x7f7080069d40 2023-10-25T00:24:43.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f7085aa9640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70800ac650 con 0x7f7080158970 2023-10-25T00:24:43.282 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f707ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7080158970 0x7f708006a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f707ffff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f7080158970 0x7f708006a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46298/0 (socket says 172.21.15.107:46298) 2023-10-25T00:24:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f707ffff640 1 -- 172.21.15.107:0/3106354816 learned_addr learned my addr 172.21.15.107:0/3106354816 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f707ffff640 1 -- 172.21.15.107:0/3106354816 >> v1:172.21.15.110:6789/0 conn(0x7f7080069d40 legacy=0x7f70800a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:24:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f707ffff640 1 -- 172.21.15.107:0/3106354816 >> v1:172.21.15.107:6789/0 conn(0x7f7080065bc0 legacy=0x7f70800aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f707e7fc640 1 -- 172.21.15.107:0/3106354816 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741165057 0 0) 0x7f708006cea0 con 0x7f7080065bc0 2023-10-25T00:24:43.284 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.277+0000 7f707ffff640 1 -- 172.21.15.107:0/3106354816 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708006d3d0 con 0x7f7080158970 2023-10-25T00:24:43.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707ffff640 1 --2- 172.21.15.107:0/3106354816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7080158970 0x7f708006a120 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7078009e50 tx=0x7f707801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2148c989de60526 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:43.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707e7fc640 1 -- 172.21.15.107:0/3106354816 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70780a5600 con 0x7f7080158970 2023-10-25T00:24:43.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707e7fc640 1 -- 172.21.15.107:0/3106354816 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70780a5760 con 0x7f7080158970 2023-10-25T00:24:43.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707e7fc640 1 -- 172.21.15.107:0/3106354816 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70780a5af0 con 0x7f7080158970 2023-10-25T00:24:43.285 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 -- 172.21.15.107:0/3106354816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7080158970 msgr2=0x7f708006a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 --2- 172.21.15.107:0/3106354816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7080158970 0x7f708006a120 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7078009e50 tx=0x7f707801c940 comp rx=0 tx=0).stop 2023-10-25T00:24:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 -- 172.21.15.107:0/3106354816 shutdown_connections 2023-10-25T00:24:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 --2- 172.21.15.107:0/3106354816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7080158970 0x7f708006a120 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 -- 172.21.15.107:0/3106354816 >> 172.21.15.107:0/3106354816 conn(0x7f70800c65c0 msgr2=0x7f70800c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:43.286 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 -- 172.21.15.107:0/3106354816 shutdown_connections 2023-10-25T00:24:43.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 -- 172.21.15.107:0/3106354816 wait complete. 2023-10-25T00:24:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 Processor -- start 2023-10-25T00:24:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 -- start start 2023-10-25T00:24:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7080065bc0 0x7f70801f0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7080069d40 0x7f70800bf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:43.289 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7080158970 0x7f70801486d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70800be5b0 con 0x7f7080158970 2023-10-25T00:24:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70801f12b0 con 0x7f7080065bc0 2023-10-25T00:24:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70801ed270 con 0x7f7080069d40 2023-10-25T00:24:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7080158970 0x7f70801486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:43.290 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7080158970 0x7f70801486d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46304/0 (socket says 172.21.15.107:46304) 2023-10-25T00:24:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707ffff640 1 -- 172.21.15.107:0/2175264734 learned_addr learned my addr 172.21.15.107:0/2175264734 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:24:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707ffff640 1 -- 172.21.15.107:0/2175264734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7080069d40 msgr2=0x7f70800bf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707ffff640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7080069d40 0x7f70800bf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707ffff640 1 -- 172.21.15.107:0/2175264734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7080065bc0 msgr2=0x7f70801f0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707ffff640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7080065bc0 0x7f70801f0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:43.291 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707ffff640 1 -- 172.21.15.107:0/2175264734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078009b30 con 0x7f7080158970 2023-10-25T00:24:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f707ffff640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7080158970 0x7f70801486d0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f70780af040 tx=0x7f70780196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7067fff640 1 -- 172.21.15.107:0/2175264734 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70780b8670 con 0x7f7080158970 2023-10-25T00:24:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7067fff640 1 -- 172.21.15.107:0/2175264734 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70780b87d0 con 0x7f7080158970 2023-10-25T00:24:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 -- 172.21.15.107:0/2175264734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70800bf900 con 0x7f7080158970 2023-10-25T00:24:43.292 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7067fff640 1 -- 172.21.15.107:0/2175264734 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70780b8b60 con 0x7f7080158970 2023-10-25T00:24:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.281+0000 7f7085aa9640 1 -- 172.21.15.107:0/2175264734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70800bdb10 con 0x7f7080158970 2023-10-25T00:24:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.285+0000 7f7085aa9640 1 -- 172.21.15.107:0/2175264734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70800746e0 con 0x7f7080158970 2023-10-25T00:24:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.285+0000 7f7067fff640 1 -- 172.21.15.107:0/2175264734 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f70780b8cc0 con 0x7f7080158970 2023-10-25T00:24:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.285+0000 7f7067fff640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f706c065e60 0x7f706c068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:24:43.293 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.285+0000 7f7067fff640 1 -- 172.21.15.107:0/2175264734 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7f7078120060 con 0x7f7080158970 2023-10-25T00:24:43.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.285+0000 7f707f7fe640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f706c065e60 0x7f706c068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:24:43.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.285+0000 7f707f7fe640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f706c065e60 0x7f706c068320 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7080149160 tx=0x7f70740079c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:24:43.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.289+0000 7f7067fff640 1 -- 172.21.15.107:0/2175264734 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f70780ec510 con 0x7f7080158970 2023-10-25T00:24:43.478 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.473+0000 7f7085aa9640 1 -- 172.21.15.107:0/2175264734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70801ed3b0 con 0x7f7080158970 2023-10-25T00:24:43.479 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.473+0000 7f7067fff640 1 -- 172.21.15.107:0/2175264734 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+5769 (secure 0 0 0) 0x7f70780efb30 con 0x7f7080158970 2023-10-25T00:24:43.479 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:24:43.479 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9606,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2839787144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2839787144},{"type":"v1","addr":"172.21.15.107:6837","nonce":2839787144}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":240}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:24:38.446814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"c","rank":0,"incarnation":236,"state":"up:active","state_seq":27,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:24:43.480 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 240 2023-10-25T00:24:43.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 -- 172.21.15.107:0/2175264734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f706c065e60 msgr2=0x7f706c068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:43.481 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f706c065e60 0x7f706c068320 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f7080149160 tx=0x7f70740079c0 comp rx=0 tx=0).stop 2023-10-25T00:24:43.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 -- 172.21.15.107:0/2175264734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7080158970 msgr2=0x7f70801486d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:24:43.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7080158970 0x7f70801486d0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f70780af040 tx=0x7f70780196e0 comp rx=0 tx=0).stop 2023-10-25T00:24:43.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 -- 172.21.15.107:0/2175264734 shutdown_connections 2023-10-25T00:24:43.482 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7080158970 0x7f70801486d0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:43.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f706c065e60 0x7f706c068320 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:43.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7080069d40 0x7f70800bf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:43.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 --2- 172.21.15.107:0/2175264734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7080065bc0 0x7f70801f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:24:43.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 -- 172.21.15.107:0/2175264734 >> 172.21.15.107:0/2175264734 conn(0x7f70800c65c0 msgr2=0x7f708006c490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:24:43.483 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 -- 172.21.15.107:0/2175264734 shutdown_connections 2023-10-25T00:24:43.484 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:24:43.477+0000 7f7085aa9640 1 -- 172.21.15.107:0/2175264734 wait complete. 2023-10-25T00:24:43.497 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2023-10-25T00:24:43.497 INFO:tasks.mds_thrash.fs.[a]:waiting for 55 secs before thrashing 2023-10-25T00:25:00.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-25T00:25:00.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c453ea640 1 Processor -- start 2023-10-25T00:25:00.392 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c453ea640 1 -- start start 2023-10-25T00:25:00.393 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c453ea640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c4014fd40 0x7f1c400ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:00.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c453ea640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c4006f730 con 0x7f1c400a9820 2023-10-25T00:25:00.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c453ea640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c40156a40 con 0x7f1c4006ee40 2023-10-25T00:25:00.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c453ea640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c40156710 con 0x7f1c4014fd40 2023-10-25T00:25:00.394 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c3effd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f1c4014fd40 0x7f1c400ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:00.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c37fff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f1c400a9820 0x7f1c400c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34404/0 (socket says 172.21.15.107:34404) 2023-10-25T00:25:00.395 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c37fff640 1 -- 172.21.15.107:0/4078649090 learned_addr learned my addr 172.21.15.107:0/4078649090 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:25:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c3effd640 1 -- 172.21.15.107:0/4078649090 >> v1:172.21.15.110:6789/0 conn(0x7f1c4006ee40 legacy=0x7f1c4015cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c3effd640 1 -- 172.21.15.107:0/4078649090 >> v1:172.21.15.107:6789/0 conn(0x7f1c400a9820 legacy=0x7f1c400c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.389+0000 7f1c3effd640 1 -- 172.21.15.107:0/4078649090 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4006f980 con 0x7f1c4014fd40 2023-10-25T00:25:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3effd640 1 --2- 172.21.15.107:0/4078649090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c4014fd40 0x7f1c400ab090 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f1c300097c0 tx=0x7f1c3001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f683fc72cdace7 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3e7fc640 1 -- 172.21.15.107:0/4078649090 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c300ab600 con 0x7f1c4014fd40 2023-10-25T00:25:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3e7fc640 1 -- 172.21.15.107:0/4078649090 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c300a7050 con 0x7f1c4014fd40 2023-10-25T00:25:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3e7fc640 1 -- 172.21.15.107:0/4078649090 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c300ab980 con 0x7f1c4014fd40 2023-10-25T00:25:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 -- 172.21.15.107:0/4078649090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c4014fd40 msgr2=0x7f1c400ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 --2- 172.21.15.107:0/4078649090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c4014fd40 0x7f1c400ab090 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f1c300097c0 tx=0x7f1c3001c320 comp rx=0 tx=0).stop 2023-10-25T00:25:00.400 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 -- 172.21.15.107:0/4078649090 shutdown_connections 2023-10-25T00:25:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 --2- 172.21.15.107:0/4078649090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c4014fd40 0x7f1c400ab090 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 -- 172.21.15.107:0/4078649090 >> 172.21.15.107:0/4078649090 conn(0x7f1c400c2950 msgr2=0x7f1c400c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 -- 172.21.15.107:0/4078649090 shutdown_connections 2023-10-25T00:25:00.401 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 -- 172.21.15.107:0/4078649090 wait complete. 2023-10-25T00:25:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 Processor -- start 2023-10-25T00:25:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 -- start start 2023-10-25T00:25:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c4006ee40 0x7f1c40143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:00.402 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c4014fd40 0x7f1c40154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c40155b20 0x7f1c4014b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c40144d60 con 0x7f1c4006ee40 2023-10-25T00:25:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c400c03c0 con 0x7f1c4014fd40 2023-10-25T00:25:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4014afc0 con 0x7f1c40155b20 2023-10-25T00:25:00.403 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c4006ee40 0x7f1c40143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c4006ee40 0x7f1c40143bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48742/0 (socket says 172.21.15.107:48742) 2023-10-25T00:25:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3effd640 1 -- 172.21.15.107:0/1032277823 learned_addr learned my addr 172.21.15.107:0/1032277823 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c37fff640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c4014fd40 0x7f1c40154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3f7fe640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c40155b20 0x7f1c4014b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:00.404 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3effd640 1 -- 172.21.15.107:0/1032277823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c40155b20 msgr2=0x7f1c4014b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3effd640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c40155b20 0x7f1c4014b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3effd640 1 -- 172.21.15.107:0/1032277823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c4014fd40 msgr2=0x7f1c40154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3effd640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c4014fd40 0x7f1c40154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3effd640 1 -- 172.21.15.107:0/1032277823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c30009420 con 0x7f1c4006ee40 2023-10-25T00:25:00.405 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3f7fe640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c40155b20 0x7f1c4014b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:00.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c3effd640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c4006ee40 0x7f1c40143bf0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f1c300ae040 tx=0x7f1c3001ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:00.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c377fe640 1 -- 172.21.15.107:0/1032277823 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c300b7650 con 0x7f1c4006ee40 2023-10-25T00:25:00.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c377fe640 1 -- 172.21.15.107:0/1032277823 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c300b77d0 con 0x7f1c4006ee40 2023-10-25T00:25:00.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c377fe640 1 -- 172.21.15.107:0/1032277823 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c300b7b80 con 0x7f1c4006ee40 2023-10-25T00:25:00.406 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 -- 172.21.15.107:0/1032277823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c4014cbd0 con 0x7f1c4006ee40 2023-10-25T00:25:00.407 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.393+0000 7f1c453ea640 1 -- 172.21.15.107:0/1032277823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c401ebcf0 con 0x7f1c4006ee40 2023-10-25T00:25:00.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.397+0000 7f1c453ea640 1 -- 172.21.15.107:0/1032277823 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c400c24a0 con 0x7f1c4006ee40 2023-10-25T00:25:00.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.397+0000 7f1c377fe640 1 -- 172.21.15.107:0/1032277823 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f1c300b7ce0 con 0x7f1c4006ee40 2023-10-25T00:25:00.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.397+0000 7f1c377fe640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c20065e60 0x7f1c20068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:00.408 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.397+0000 7f1c377fe640 1 -- 172.21.15.107:0/1032277823 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7f1c3011f4c0 con 0x7f1c4006ee40 2023-10-25T00:25:00.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.397+0000 7f1c37fff640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c20065e60 0x7f1c20068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:00.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.397+0000 7f1c37fff640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c20065e60 0x7f1c20068320 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f1c401e9010 tx=0x7f1c28009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:00.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.401+0000 7f1c377fe640 1 -- 172.21.15.107:0/1032277823 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f1c300eb970 con 0x7f1c4006ee40 2023-10-25T00:25:00.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.553+0000 7f1c453ea640 1 -- 172.21.15.107:0/1032277823 --> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c400aa620 con 0x7f1c20065e60 2023-10-25T00:25:00.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.593+0000 7f1c377fe640 1 -- 172.21.15.107:0/1032277823 <== mgr.4110 v2:172.21.15.107:6832/37466 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+556391 (secure 0 0 0) 0x7f1c28001c80 con 0x7f1c20065e60 2023-10-25T00:25:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:25:00.607 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2023-10-25T00:25:00.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 -- 172.21.15.107:0/1032277823 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c20065e60 msgr2=0x7f1c20068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:00.608 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c20065e60 0x7f1c20068320 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f1c401e9010 tx=0x7f1c28009040 comp rx=0 tx=0).stop 2023-10-25T00:25:00.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 -- 172.21.15.107:0/1032277823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c4006ee40 msgr2=0x7f1c40143bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:00.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c4006ee40 0x7f1c40143bf0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f1c300ae040 tx=0x7f1c3001ab60 comp rx=0 tx=0).stop 2023-10-25T00:25:00.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 -- 172.21.15.107:0/1032277823 shutdown_connections 2023-10-25T00:25:00.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1c20065e60 0x7f1c20068320 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.609 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1c40155b20 0x7f1c4014b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1c4014fd40 0x7f1c40154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 --2- 172.21.15.107:0/1032277823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c4006ee40 0x7f1c40143bf0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 -- 172.21.15.107:0/1032277823 >> 172.21.15.107:0/1032277823 conn(0x7f1c400c2950 msgr2=0x7f1c400c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:00.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 -- 172.21.15.107:0/1032277823 shutdown_connections 2023-10-25T00:25:00.610 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.601+0000 7f1c453ea640 1 -- 172.21.15.107:0/1032277823 wait complete. 2023-10-25T00:25:00.628 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1032,"stamp":"2023-10-25T00:25:00.078819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40163554,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":4044,"num_read_kb":497048,"num_write":3153476,"num_write_kb":38880103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":584023,"ondisk_log_size":584023,"up":402,"acting":402,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":402,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7810792,"kb_used_data":86376,"kb_used_omap":0,"kb_used_meta":7724352,"kb_avail":747163928,"statfs":{"total":773094113280,"available":765095862272,"internally_reserved":0,"allocated":88449024,"data_stored":84777812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7909736448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006411"},"pg_stats":[{"pgid":"6.1a","version":"72'3995","reported_seq":4082,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:16.475475+0000","last_change":"2023-10-25T00:24:16.475475+0000","last_active":"2023-10-25T00:24:16.475475+0000","last_peered":"2023-10-25T00:24:16.475475+0000","last_clean":"2023-10-25T00:24:16.475475+0000","last_became_active":"2023-10-24T23:53:11.854445+0000","last_became_peered":"2023-10-24T23:53:11.854445+0000","last_unstale":"2023-10-25T00:24:16.475475+0000","last_undegraded":"2023-10-25T00:24:16.475475+0000","last_fullsized":"2023-10-25T00:24:16.475475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3995","last_scrub_stamp":"2023-10-25T00:24:16.475258+0000","last_deep_scrub":"72'3995","last_deep_scrub_stamp":"2023-10-25T00:24:16.475258+0000","last_clean_scrub_stamp":"2023-10-25T00:24:16.475258+0000","objects_scrubbed":0,"log_size":3995,"log_dups_size":0,"ondisk_log_size":3995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:52:25.511844+0000","scrub_duration":0.42505980700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21480,"num_write_kb":282788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"72'3","reported_seq":86,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.322487+0000","last_change":"2023-10-25T00:23:29.637684+0000","last_active":"2023-10-25T00:24:35.322487+0000","last_peered":"2023-10-25T00:24:35.322487+0000","last_clean":"2023-10-25T00:24:35.322487+0000","last_became_active":"2023-10-24T23:53:11.443569+0000","last_became_peered":"2023-10-24T23:53:11.443569+0000","last_unstale":"2023-10-25T00:24:35.322487+0000","last_undegraded":"2023-10-25T00:24:35.322487+0000","last_fullsized":"2023-10-25T00:24:35.322487+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:29.637307+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:29.637307+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.637307+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:19:55.311760+0000","scrub_duration":0.0045847350000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"73'867","reported_seq":1012,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.138889+0000","last_change":"2023-10-25T00:23:27.027247+0000","last_active":"2023-10-25T00:24:35.138889+0000","last_peered":"2023-10-25T00:24:35.138889+0000","last_clean":"2023-10-25T00:24:35.138889+0000","last_became_active":"2023-10-24T23:53:02.349894+0000","last_became_peered":"2023-10-24T23:53:02.349894+0000","last_unstale":"2023-10-25T00:24:35.138889+0000","last_undegraded":"2023-10-25T00:24:35.138889+0000","last_fullsized":"2023-10-25T00:24:35.138889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'867","last_scrub_stamp":"2023-10-25T00:23:27.026943+0000","last_deep_scrub":"73'867","last_deep_scrub_stamp":"2023-10-25T00:23:27.026943+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.026943+0000","objects_scrubbed":3,"log_size":867,"log_dups_size":0,"ondisk_log_size":867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:41.065198+0000","scrub_duration":0.082266929000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":26641,"num_write":841,"num_write_kb":4586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'5616","reported_seq":5703,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.429483+0000","last_change":"2023-10-25T00:23:46.131251+0000","last_active":"2023-10-25T00:24:31.429483+0000","last_peered":"2023-10-25T00:24:31.429483+0000","last_clean":"2023-10-25T00:24:31.429483+0000","last_became_active":"2023-10-24T23:53:02.717687+0000","last_became_peered":"2023-10-24T23:53:02.717687+0000","last_unstale":"2023-10-25T00:24:31.429483+0000","last_undegraded":"2023-10-25T00:24:31.429483+0000","last_fullsized":"2023-10-25T00:24:31.429483+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5616","last_scrub_stamp":"2023-10-25T00:23:46.131002+0000","last_deep_scrub":"68'5616","last_deep_scrub_stamp":"2023-10-25T00:23:46.131002+0000","last_clean_scrub_stamp":"2023-10-25T00:23:46.131002+0000","objects_scrubbed":0,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:29:47.372296+0000","scrub_duration":0.13911974099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":29775,"num_write_kb":348818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"72'3735","reported_seq":3820,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:59.753562+0000","last_change":"2023-10-25T00:23:59.753562+0000","last_active":"2023-10-25T00:23:59.753562+0000","last_peered":"2023-10-25T00:23:59.753562+0000","last_clean":"2023-10-25T00:23:59.753562+0000","last_became_active":"2023-10-24T23:53:11.846763+0000","last_became_peered":"2023-10-24T23:53:11.846763+0000","last_unstale":"2023-10-25T00:23:59.753562+0000","last_undegraded":"2023-10-25T00:23:59.753562+0000","last_fullsized":"2023-10-25T00:23:59.753562+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3735","last_scrub_stamp":"2023-10-25T00:23:59.753413+0000","last_deep_scrub":"72'3735","last_deep_scrub_stamp":"2023-10-25T00:23:59.753413+0000","last_clean_scrub_stamp":"2023-10-25T00:23:59.753413+0000","objects_scrubbed":0,"log_size":3735,"log_dups_size":0,"ondisk_log_size":3735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:57:23.808102+0000","scrub_duration":0.20248877100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20090,"num_write_kb":264783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"72'2","reported_seq":85,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.278757+0000","last_change":"2023-10-25T00:23:02.880442+0000","last_active":"2023-10-25T00:24:34.278757+0000","last_peered":"2023-10-25T00:24:34.278757+0000","last_clean":"2023-10-25T00:24:34.278757+0000","last_became_active":"2023-10-24T23:53:11.443543+0000","last_became_peered":"2023-10-24T23:53:11.443543+0000","last_unstale":"2023-10-25T00:24:34.278757+0000","last_undegraded":"2023-10-25T00:24:34.278757+0000","last_fullsized":"2023-10-25T00:24:34.278757+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:23:02.880202+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.880202+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:45.433003+0000","scrub_duration":0.0048172340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"74'29","reported_seq":118,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.314049+0000","last_change":"2023-10-25T00:23:05.443056+0000","last_active":"2023-10-25T00:24:34.314049+0000","last_peered":"2023-10-25T00:24:34.314049+0000","last_clean":"2023-10-25T00:24:34.314049+0000","last_became_active":"2023-10-24T23:53:02.331586+0000","last_became_peered":"2023-10-24T23:53:02.331586+0000","last_unstale":"2023-10-25T00:24:34.314049+0000","last_undegraded":"2023-10-25T00:24:34.314049+0000","last_fullsized":"2023-10-25T00:24:34.314049+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_deep_scrub":"73'28","last_deep_scrub_stamp":"2023-10-25T00:23:05.442750+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.442750+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:29:04.499904+0000","scrub_duration":0.036465213000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'5447","reported_seq":5534,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.569763+0000","last_change":"2023-10-25T00:23:28.801391+0000","last_active":"2023-10-25T00:24:32.569763+0000","last_peered":"2023-10-25T00:24:32.569763+0000","last_clean":"2023-10-25T00:24:32.569763+0000","last_became_active":"2023-10-24T23:53:02.701906+0000","last_became_peered":"2023-10-24T23:53:02.701906+0000","last_unstale":"2023-10-25T00:24:32.569763+0000","last_undegraded":"2023-10-25T00:24:32.569763+0000","last_fullsized":"2023-10-25T00:24:32.569763+0000","mapping_epoch":21,"log_start":"61'100","ondisk_log_start":"61'100","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5447","last_scrub_stamp":"2023-10-25T00:23:28.801240+0000","last_deep_scrub":"68'5447","last_deep_scrub_stamp":"2023-10-25T00:23:28.801240+0000","last_clean_scrub_stamp":"2023-10-25T00:23:28.801240+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:44:02.442972+0000","scrub_duration":0.154979636,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28063,"num_write_kb":350200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"72'4747","reported_seq":4833,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.313091+0000","last_change":"2023-10-25T00:23:40.173944+0000","last_active":"2023-10-25T00:24:34.313091+0000","last_peered":"2023-10-25T00:24:34.313091+0000","last_clean":"2023-10-25T00:24:34.313091+0000","last_became_active":"2023-10-24T23:53:11.793505+0000","last_became_peered":"2023-10-24T23:53:11.793505+0000","last_unstale":"2023-10-25T00:24:34.313091+0000","last_undegraded":"2023-10-25T00:24:34.313091+0000","last_fullsized":"2023-10-25T00:24:34.313091+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4747","last_scrub_stamp":"2023-10-25T00:23:40.173729+0000","last_deep_scrub":"72'4747","last_deep_scrub_stamp":"2023-10-25T00:23:40.173729+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.173729+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:39:14.413546+0000","scrub_duration":0.31868588399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":12,"num_write":25766,"num_write_kb":328702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"72'754","reported_seq":863,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:15.925869+0000","last_change":"2023-10-25T00:22:37.433948+0000","last_active":"2023-10-25T00:23:15.925869+0000","last_peered":"2023-10-25T00:23:15.925869+0000","last_clean":"2023-10-25T00:23:15.925869+0000","last_became_active":"2023-10-24T23:53:11.437936+0000","last_became_peered":"2023-10-24T23:53:11.437936+0000","last_unstale":"2023-10-25T00:23:15.925869+0000","last_undegraded":"2023-10-25T00:23:15.925869+0000","last_fullsized":"2023-10-25T00:23:15.925869+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'754","last_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_deep_scrub":"72'754","last_deep_scrub_stamp":"2023-10-25T00:22:37.433382+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.433382+0000","objects_scrubbed":1,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:04:43.601946+0000","scrub_duration":0.10905978199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":77282,"num_write":740,"num_write_kb":4340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"74'53","reported_seq":187,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:38.461955+0000","last_change":"2023-10-25T00:23:37.477885+0000","last_active":"2023-10-25T00:24:38.461955+0000","last_peered":"2023-10-25T00:24:38.461955+0000","last_clean":"2023-10-25T00:24:38.461955+0000","last_became_active":"2023-10-24T23:53:02.342683+0000","last_became_peered":"2023-10-24T23:53:02.342683+0000","last_unstale":"2023-10-25T00:24:38.461955+0000","last_undegraded":"2023-10-25T00:24:38.461955+0000","last_fullsized":"2023-10-25T00:24:38.461955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'51","last_scrub_stamp":"2023-10-25T00:23:37.477557+0000","last_deep_scrub":"73'51","last_deep_scrub_stamp":"2023-10-25T00:23:37.477557+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.477557+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:04:39.045780+0000","scrub_duration":0.008112401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'4792","reported_seq":4879,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.302620+0000","last_change":"2023-10-25T00:22:44.881494+0000","last_active":"2023-10-25T00:23:06.302620+0000","last_peered":"2023-10-25T00:23:06.302620+0000","last_clean":"2023-10-25T00:23:06.302620+0000","last_became_active":"2023-10-24T23:53:02.717763+0000","last_became_peered":"2023-10-24T23:53:02.717763+0000","last_unstale":"2023-10-25T00:23:06.302620+0000","last_undegraded":"2023-10-25T00:23:06.302620+0000","last_fullsized":"2023-10-25T00:23:06.302620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4792","last_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_deep_scrub":"68'4792","last_deep_scrub_stamp":"2023-10-25T00:22:44.881344+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.881344+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:23:57.562121+0000","scrub_duration":0.193670646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24997,"num_write_kb":313272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"72'4739","reported_seq":4826,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.292214+0000","last_change":"2023-10-25T00:23:49.803089+0000","last_active":"2023-10-25T00:24:34.292214+0000","last_peered":"2023-10-25T00:24:34.292214+0000","last_clean":"2023-10-25T00:24:34.292214+0000","last_became_active":"2023-10-24T23:53:11.824729+0000","last_became_peered":"2023-10-24T23:53:11.824729+0000","last_unstale":"2023-10-25T00:24:34.292214+0000","last_undegraded":"2023-10-25T00:24:34.292214+0000","last_fullsized":"2023-10-25T00:24:34.292214+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4739","last_scrub_stamp":"2023-10-25T00:23:49.802870+0000","last_deep_scrub":"72'4739","last_deep_scrub_stamp":"2023-10-25T00:23:49.802870+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.802870+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:56:59.101994+0000","scrub_duration":0.33146283399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":22,"num_write":25639,"num_write_kb":332364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"73'358","reported_seq":444,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:04.146618+0000","last_change":"2023-10-25T00:24:04.146618+0000","last_active":"2023-10-25T00:24:04.146618+0000","last_peered":"2023-10-25T00:24:04.146618+0000","last_clean":"2023-10-25T00:24:04.146618+0000","last_became_active":"2023-10-24T23:53:11.453959+0000","last_became_peered":"2023-10-24T23:53:11.453959+0000","last_unstale":"2023-10-25T00:24:04.146618+0000","last_undegraded":"2023-10-25T00:24:04.146618+0000","last_fullsized":"2023-10-25T00:24:04.146618+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'358","last_scrub_stamp":"2023-10-25T00:24:04.146337+0000","last_deep_scrub":"73'358","last_deep_scrub_stamp":"2023-10-25T00:24:04.146337+0000","last_clean_scrub_stamp":"2023-10-25T00:24:04.146337+0000","objects_scrubbed":1,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:40:21.104237+0000","scrub_duration":0.154937403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2627766,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":227,"num_write":329,"num_write_kb":2759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"73'6","reported_seq":142,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:38.463910+0000","last_change":"2023-10-25T00:23:39.493290+0000","last_active":"2023-10-25T00:24:38.463910+0000","last_peered":"2023-10-25T00:24:38.463910+0000","last_clean":"2023-10-25T00:24:38.463910+0000","last_became_active":"2023-10-24T23:53:02.348153+0000","last_became_peered":"2023-10-24T23:53:02.348153+0000","last_unstale":"2023-10-25T00:24:38.463910+0000","last_undegraded":"2023-10-25T00:24:38.463910+0000","last_fullsized":"2023-10-25T00:24:38.463910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2023-10-25T00:23:39.492964+0000","last_deep_scrub":"73'6","last_deep_scrub_stamp":"2023-10-25T00:23:39.492964+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.492964+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:15:34.589742+0000","scrub_duration":0.0071686809999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":150,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'3731","reported_seq":3816,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.302763+0000","last_change":"2023-10-25T00:22:37.622543+0000","last_active":"2023-10-25T00:23:06.302763+0000","last_peered":"2023-10-25T00:23:06.302763+0000","last_clean":"2023-10-25T00:23:06.302763+0000","last_became_active":"2023-10-24T23:53:02.724097+0000","last_became_peered":"2023-10-24T23:53:02.724097+0000","last_unstale":"2023-10-25T00:23:06.302763+0000","last_undegraded":"2023-10-25T00:23:06.302763+0000","last_fullsized":"2023-10-25T00:23:06.302763+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3731","last_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_deep_scrub":"68'3731","last_deep_scrub_stamp":"2023-10-25T00:22:37.622304+0000","last_clean_scrub_stamp":"2023-10-25T00:22:37.622304+0000","objects_scrubbed":0,"log_size":3731,"log_dups_size":0,"ondisk_log_size":3731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:51:52.171234+0000","scrub_duration":0.090187284000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19492,"num_write_kb":250091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"72'3799","reported_seq":3884,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:15.367485+0000","last_change":"2023-10-25T00:24:15.367485+0000","last_active":"2023-10-25T00:24:15.367485+0000","last_peered":"2023-10-25T00:24:15.367485+0000","last_clean":"2023-10-25T00:24:15.367485+0000","last_became_active":"2023-10-24T23:53:11.836718+0000","last_became_peered":"2023-10-24T23:53:11.836718+0000","last_unstale":"2023-10-25T00:24:15.367485+0000","last_undegraded":"2023-10-25T00:24:15.367485+0000","last_fullsized":"2023-10-25T00:24:15.367485+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3799","last_scrub_stamp":"2023-10-25T00:24:15.367271+0000","last_deep_scrub":"72'3799","last_deep_scrub_stamp":"2023-10-25T00:24:15.367271+0000","last_clean_scrub_stamp":"2023-10-25T00:24:15.367271+0000","objects_scrubbed":0,"log_size":3799,"log_dups_size":0,"ondisk_log_size":3799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:57:37.963942+0000","scrub_duration":0.31526191399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21581,"num_write_kb":262877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"73'46","reported_seq":172,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.323666+0000","last_change":"2023-10-25T00:22:52.664257+0000","last_active":"2023-10-25T00:23:16.323666+0000","last_peered":"2023-10-25T00:23:16.323666+0000","last_clean":"2023-10-25T00:23:16.323666+0000","last_became_active":"2023-10-24T23:53:11.443782+0000","last_became_peered":"2023-10-24T23:53:11.443782+0000","last_unstale":"2023-10-25T00:23:16.323666+0000","last_undegraded":"2023-10-25T00:23:16.323666+0000","last_fullsized":"2023-10-25T00:23:16.323666+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'46","last_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_deep_scrub":"73'46","last_deep_scrub_stamp":"2023-10-25T00:22:52.664064+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.664064+0000","objects_scrubbed":1,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:44:09.108291+0000","scrub_duration":0.106079285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"73'715","reported_seq":831,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.317061+0000","last_change":"2023-10-25T00:23:21.967998+0000","last_active":"2023-10-25T00:24:34.317061+0000","last_peered":"2023-10-25T00:24:34.317061+0000","last_clean":"2023-10-25T00:24:34.317061+0000","last_became_active":"2023-10-24T23:53:02.356674+0000","last_became_peered":"2023-10-24T23:53:02.356674+0000","last_unstale":"2023-10-25T00:24:34.317061+0000","last_undegraded":"2023-10-25T00:24:34.317061+0000","last_fullsized":"2023-10-25T00:24:34.317061+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'715","last_scrub_stamp":"2023-10-25T00:23:21.967694+0000","last_deep_scrub":"73'715","last_deep_scrub_stamp":"2023-10-25T00:23:21.967694+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.967694+0000","objects_scrubbed":1,"log_size":715,"log_dups_size":0,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:56:40.443702+0000","scrub_duration":0.065018907000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":94524,"num_write":703,"num_write_kb":4303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'4008","reported_seq":4097,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:05.814881+0000","last_change":"2023-10-25T00:23:05.488463+0000","last_active":"2023-10-25T00:23:05.814881+0000","last_peered":"2023-10-25T00:23:05.814881+0000","last_clean":"2023-10-25T00:23:05.814881+0000","last_became_active":"2023-10-24T23:53:02.719755+0000","last_became_peered":"2023-10-24T23:53:02.719755+0000","last_unstale":"2023-10-25T00:23:05.814881+0000","last_undegraded":"2023-10-25T00:23:05.814881+0000","last_fullsized":"2023-10-25T00:23:05.814881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4008","last_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_deep_scrub":"68'4008","last_deep_scrub_stamp":"2023-10-25T00:23:05.488248+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.488248+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:51:16.627187+0000","scrub_duration":0.15813021899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20791,"num_write_kb":267770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"72'5469","reported_seq":5565,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:22:57.571401+0000","last_change":"2023-10-25T00:22:57.571401+0000","last_active":"2023-10-25T00:22:57.571401+0000","last_peered":"2023-10-25T00:22:57.571401+0000","last_clean":"2023-10-25T00:22:57.571401+0000","last_became_active":"2023-10-24T23:53:11.832520+0000","last_became_peered":"2023-10-24T23:53:11.832520+0000","last_unstale":"2023-10-25T00:22:57.571401+0000","last_undegraded":"2023-10-25T00:22:57.571401+0000","last_fullsized":"2023-10-25T00:22:57.571401+0000","mapping_epoch":25,"log_start":"64'200","ondisk_log_start":"64'200","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5469","last_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_deep_scrub":"72'5469","last_deep_scrub_stamp":"2023-10-25T00:22:57.571234+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.571234+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:24:13.836466+0000","scrub_duration":0.19144520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":40,"num_write":30646,"num_write_kb":401481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"73'6","reported_seq":132,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:16.305295+0000","last_change":"2023-10-25T00:23:14.991436+0000","last_active":"2023-10-25T00:23:16.305295+0000","last_peered":"2023-10-25T00:23:16.305295+0000","last_clean":"2023-10-25T00:23:16.305295+0000","last_became_active":"2023-10-24T23:53:11.441458+0000","last_became_peered":"2023-10-24T23:53:11.441458+0000","last_unstale":"2023-10-25T00:23:16.305295+0000","last_undegraded":"2023-10-25T00:23:16.305295+0000","last_fullsized":"2023-10-25T00:23:16.305295+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_deep_scrub":"73'6","last_deep_scrub_stamp":"2023-10-25T00:23:14.991238+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.991238+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:25:08.545014+0000","scrub_duration":0.0072276010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":132,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"74'561","reported_seq":660,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:41.852327+0000","last_change":"2023-10-25T00:24:31.287913+0000","last_active":"2023-10-25T00:24:41.852327+0000","last_peered":"2023-10-25T00:24:41.852327+0000","last_clean":"2023-10-25T00:24:41.852327+0000","last_became_active":"2023-10-24T23:53:02.340025+0000","last_became_peered":"2023-10-24T23:53:02.340025+0000","last_unstale":"2023-10-25T00:24:41.852327+0000","last_undegraded":"2023-10-25T00:24:41.852327+0000","last_fullsized":"2023-10-25T00:24:41.852327+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'547","last_scrub_stamp":"2023-10-25T00:24:31.287462+0000","last_deep_scrub":"73'547","last_deep_scrub_stamp":"2023-10-25T00:24:31.287462+0000","last_clean_scrub_stamp":"2023-10-25T00:24:31.287462+0000","objects_scrubbed":1,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:31:39.750665+0000","scrub_duration":0.029858290999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3218857,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":13752,"num_write":530,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'6069","reported_seq":6157,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.535460+0000","last_change":"2023-10-25T00:22:46.608504+0000","last_active":"2023-10-25T00:23:07.535460+0000","last_peered":"2023-10-25T00:23:07.535460+0000","last_clean":"2023-10-25T00:23:07.535460+0000","last_became_active":"2023-10-24T23:53:02.718457+0000","last_became_peered":"2023-10-24T23:53:02.718457+0000","last_unstale":"2023-10-25T00:23:07.535460+0000","last_undegraded":"2023-10-25T00:23:07.535460+0000","last_fullsized":"2023-10-25T00:23:07.535460+0000","mapping_epoch":21,"log_start":"62'800","ondisk_log_start":"62'800","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6069","last_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_deep_scrub":"68'6069","last_deep_scrub_stamp":"2023-10-25T00:22:46.608198+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.608198+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:53:37.101860+0000","scrub_duration":0.19917779799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":33187,"num_write_kb":387130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'3606","reported_seq":3695,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:38.186420+0000","last_change":"2023-10-25T00:23:38.186420+0000","last_active":"2023-10-25T00:23:38.186420+0000","last_peered":"2023-10-25T00:23:38.186420+0000","last_clean":"2023-10-25T00:23:38.186420+0000","last_became_active":"2023-10-24T23:53:11.845185+0000","last_became_peered":"2023-10-24T23:53:11.845185+0000","last_unstale":"2023-10-25T00:23:38.186420+0000","last_undegraded":"2023-10-25T00:23:38.186420+0000","last_fullsized":"2023-10-25T00:23:38.186420+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3606","last_scrub_stamp":"2023-10-25T00:23:38.186264+0000","last_deep_scrub":"72'3606","last_deep_scrub_stamp":"2023-10-25T00:23:38.186264+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.186264+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:15.619690+0000","scrub_duration":0.23143245800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19493,"num_write_kb":254342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'1064","reported_seq":1200,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:30.690441+0000","last_change":"2023-10-25T00:23:03.701535+0000","last_active":"2023-10-25T00:24:30.690441+0000","last_peered":"2023-10-25T00:24:30.690441+0000","last_clean":"2023-10-25T00:24:30.690441+0000","last_became_active":"2023-10-24T23:53:11.447469+0000","last_became_peered":"2023-10-24T23:53:11.447469+0000","last_unstale":"2023-10-25T00:24:30.690441+0000","last_undegraded":"2023-10-25T00:24:30.690441+0000","last_fullsized":"2023-10-25T00:24:30.690441+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1064","last_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_deep_scrub":"73'1064","last_deep_scrub_stamp":"2023-10-25T00:23:03.701304+0000","last_clean_scrub_stamp":"2023-10-25T00:23:03.701304+0000","objects_scrubbed":3,"log_size":1064,"log_dups_size":0,"ondisk_log_size":1064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:08:18.178955+0000","scrub_duration":0.28236572100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":9402,"num_write":1035,"num_write_kb":4717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'3","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.341650+0000","last_change":"2023-10-25T00:23:12.838945+0000","last_active":"2023-10-25T00:24:31.341650+0000","last_peered":"2023-10-25T00:24:31.341650+0000","last_clean":"2023-10-25T00:24:31.341650+0000","last_became_active":"2023-10-24T23:53:02.336365+0000","last_became_peered":"2023-10-24T23:53:02.336365+0000","last_unstale":"2023-10-25T00:24:31.341650+0000","last_undegraded":"2023-10-25T00:24:31.341650+0000","last_fullsized":"2023-10-25T00:24:31.341650+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:12.838787+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.838787+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:33:35.778931+0000","scrub_duration":0.133446072,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'5139","reported_seq":5227,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:11.490339+0000","last_change":"2023-10-25T00:22:54.030976+0000","last_active":"2023-10-25T00:23:11.490339+0000","last_peered":"2023-10-25T00:23:11.490339+0000","last_clean":"2023-10-25T00:23:11.490339+0000","last_became_active":"2023-10-24T23:53:02.726598+0000","last_became_peered":"2023-10-24T23:53:02.726598+0000","last_unstale":"2023-10-25T00:23:11.490339+0000","last_undegraded":"2023-10-25T00:23:11.490339+0000","last_fullsized":"2023-10-25T00:23:11.490339+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5139","last_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_deep_scrub":"68'5139","last_deep_scrub_stamp":"2023-10-25T00:22:54.030782+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.030782+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:30:30.182861+0000","scrub_duration":0.103857617,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27772,"num_write_kb":328469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"72'4843","reported_seq":4928,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:39.041980+0000","last_change":"2023-10-25T00:23:39.041980+0000","last_active":"2023-10-25T00:23:39.041980+0000","last_peered":"2023-10-25T00:23:39.041980+0000","last_clean":"2023-10-25T00:23:39.041980+0000","last_became_active":"2023-10-24T23:53:11.854904+0000","last_became_peered":"2023-10-24T23:53:11.854904+0000","last_unstale":"2023-10-25T00:23:39.041980+0000","last_undegraded":"2023-10-25T00:23:39.041980+0000","last_fullsized":"2023-10-25T00:23:39.041980+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4843","last_scrub_stamp":"2023-10-25T00:23:39.041623+0000","last_deep_scrub":"72'4843","last_deep_scrub_stamp":"2023-10-25T00:23:39.041623+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.041623+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:34:30.360915+0000","scrub_duration":0.33452469800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":27290,"num_write_kb":330906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"72'27","reported_seq":112,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.493119+0000","last_change":"2023-10-25T00:22:55.987117+0000","last_active":"2023-10-25T00:23:21.493119+0000","last_peered":"2023-10-25T00:23:21.493119+0000","last_clean":"2023-10-25T00:23:21.493119+0000","last_became_active":"2023-10-24T23:53:11.446610+0000","last_became_peered":"2023-10-24T23:53:11.446610+0000","last_unstale":"2023-10-25T00:23:21.493119+0000","last_undegraded":"2023-10-25T00:23:21.493119+0000","last_fullsized":"2023-10-25T00:23:21.493119+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'27","last_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_deep_scrub":"72'27","last_deep_scrub_stamp":"2023-10-25T00:22:55.986674+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.986674+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:03:33.089164+0000","scrub_duration":0.100730242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"67'2","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.569646+0000","last_change":"2023-10-25T00:23:21.504086+0000","last_active":"2023-10-25T00:24:32.569646+0000","last_peered":"2023-10-25T00:24:32.569646+0000","last_clean":"2023-10-25T00:24:32.569646+0000","last_became_active":"2023-10-24T23:53:02.331287+0000","last_became_peered":"2023-10-24T23:53:02.331287+0000","last_unstale":"2023-10-25T00:24:32.569646+0000","last_undegraded":"2023-10-25T00:24:32.569646+0000","last_fullsized":"2023-10-25T00:24:32.569646+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2023-10-25T00:23:21.503783+0000","last_deep_scrub":"67'2","last_deep_scrub_stamp":"2023-10-25T00:23:21.503783+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.503783+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:13.653289+0000","scrub_duration":0.0039280959999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'5814","reported_seq":5903,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:20.688378+0000","last_change":"2023-10-25T00:22:52.320696+0000","last_active":"2023-10-25T00:24:20.688378+0000","last_peered":"2023-10-25T00:24:20.688378+0000","last_clean":"2023-10-25T00:24:20.688378+0000","last_became_active":"2023-10-24T23:53:02.697934+0000","last_became_peered":"2023-10-24T23:53:02.697934+0000","last_unstale":"2023-10-25T00:24:20.688378+0000","last_undegraded":"2023-10-25T00:24:20.688378+0000","last_fullsized":"2023-10-25T00:24:20.688378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5814","last_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_deep_scrub":"68'5814","last_deep_scrub_stamp":"2023-10-25T00:22:52.320378+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.320378+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:23:26.511615+0000","scrub_duration":0.033495337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31996,"num_write_kb":382397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"72'3827","reported_seq":3913,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.273982+0000","last_change":"2023-10-25T00:23:14.436458+0000","last_active":"2023-10-25T00:23:36.273982+0000","last_peered":"2023-10-25T00:23:36.273982+0000","last_clean":"2023-10-25T00:23:36.273982+0000","last_became_active":"2023-10-24T23:53:11.848390+0000","last_became_peered":"2023-10-24T23:53:11.848390+0000","last_unstale":"2023-10-25T00:23:36.273982+0000","last_undegraded":"2023-10-25T00:23:36.273982+0000","last_fullsized":"2023-10-25T00:23:36.273982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3827","last_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_deep_scrub":"72'3827","last_deep_scrub_stamp":"2023-10-25T00:23:14.436150+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.436150+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:27:32.423180+0000","scrub_duration":0.30547422699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":21199,"num_write_kb":277730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"68'4820","reported_seq":4908,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:22.538415+0000","last_change":"2023-10-25T00:22:36.429763+0000","last_active":"2023-10-25T00:23:22.538415+0000","last_peered":"2023-10-25T00:23:22.538415+0000","last_clean":"2023-10-25T00:23:22.538415+0000","last_became_active":"2023-10-24T23:53:02.718235+0000","last_became_peered":"2023-10-24T23:53:02.718235+0000","last_unstale":"2023-10-25T00:23:22.538415+0000","last_undegraded":"2023-10-25T00:23:22.538415+0000","last_fullsized":"2023-10-25T00:23:22.538415+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4820","last_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_deep_scrub":"68'4820","last_deep_scrub_stamp":"2023-10-25T00:22:36.429607+0000","last_clean_scrub_stamp":"2023-10-25T00:22:36.429607+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:35.883183+0000","scrub_duration":0.10613655600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25163,"num_write_kb":285293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"72'5","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.510020+0000","last_change":"2023-10-25T00:23:20.947028+0000","last_active":"2023-10-25T00:24:31.510020+0000","last_peered":"2023-10-25T00:24:31.510020+0000","last_clean":"2023-10-25T00:24:31.510020+0000","last_became_active":"2023-10-24T23:53:02.349618+0000","last_became_peered":"2023-10-24T23:53:02.349618+0000","last_unstale":"2023-10-25T00:24:31.510020+0000","last_undegraded":"2023-10-25T00:24:31.510020+0000","last_fullsized":"2023-10-25T00:24:31.510020+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:23:20.946713+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:23:20.946713+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.946713+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:12:40.769613+0000","scrub_duration":0.0045247550000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"72'5","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.679346+0000","last_change":"2023-10-25T00:22:40.197550+0000","last_active":"2023-10-25T00:23:35.679346+0000","last_peered":"2023-10-25T00:23:35.679346+0000","last_clean":"2023-10-25T00:23:35.679346+0000","last_became_active":"2023-10-24T23:53:11.446388+0000","last_became_peered":"2023-10-24T23:53:11.446388+0000","last_unstale":"2023-10-25T00:23:35.679346+0000","last_undegraded":"2023-10-25T00:23:35.679346+0000","last_fullsized":"2023-10-25T00:23:35.679346+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:40.197213+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.197213+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:07:28.509914+0000","scrub_duration":0.0047704979999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"72'4957","reported_seq":5046,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.542830+0000","last_change":"2023-10-25T00:22:53.667906+0000","last_active":"2023-10-25T00:23:37.542830+0000","last_peered":"2023-10-25T00:23:37.542830+0000","last_clean":"2023-10-25T00:23:37.542830+0000","last_became_active":"2023-10-24T23:53:11.812543+0000","last_became_peered":"2023-10-24T23:53:11.812543+0000","last_unstale":"2023-10-25T00:23:37.542830+0000","last_undegraded":"2023-10-25T00:23:37.542830+0000","last_fullsized":"2023-10-25T00:23:37.542830+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4957","last_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_deep_scrub":"72'4957","last_deep_scrub_stamp":"2023-10-25T00:22:53.667748+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.667748+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:42:58.307897+0000","scrub_duration":0.30812842099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28093,"num_write_kb":358545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"68'4782","reported_seq":4875,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.539783+0000","last_change":"2023-10-25T00:22:42.562198+0000","last_active":"2023-10-25T00:23:27.539783+0000","last_peered":"2023-10-25T00:23:27.539783+0000","last_clean":"2023-10-25T00:23:27.539783+0000","last_became_active":"2023-10-24T23:53:02.713684+0000","last_became_peered":"2023-10-24T23:53:02.713684+0000","last_unstale":"2023-10-25T00:23:27.539783+0000","last_undegraded":"2023-10-25T00:23:27.539783+0000","last_fullsized":"2023-10-25T00:23:27.539783+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4782","last_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_deep_scrub":"68'4782","last_deep_scrub_stamp":"2023-10-25T00:22:42.562057+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.562057+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:57:39.415960+0000","scrub_duration":0.116720554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26432,"num_write_kb":298639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"68'3","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.110759+0000","last_change":"2023-10-25T00:23:36.434199+0000","last_active":"2023-10-25T00:24:31.110759+0000","last_peered":"2023-10-25T00:24:31.110759+0000","last_clean":"2023-10-25T00:24:31.110759+0000","last_became_active":"2023-10-24T23:53:02.354233+0000","last_became_peered":"2023-10-24T23:53:02.354233+0000","last_unstale":"2023-10-25T00:24:31.110759+0000","last_undegraded":"2023-10-25T00:24:31.110759+0000","last_fullsized":"2023-10-25T00:24:31.110759+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:36.433882+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:36.433882+0000","last_clean_scrub_stamp":"2023-10-25T00:23:36.433882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:00:44.210403+0000","scrub_duration":0.0040752599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"72'18","reported_seq":216,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.273751+0000","last_change":"2023-10-25T00:22:46.074083+0000","last_active":"2023-10-25T00:23:36.273751+0000","last_peered":"2023-10-25T00:23:36.273751+0000","last_clean":"2023-10-25T00:23:36.273751+0000","last_became_active":"2023-10-24T23:53:11.449928+0000","last_became_peered":"2023-10-24T23:53:11.449928+0000","last_unstale":"2023-10-25T00:23:36.273751+0000","last_undegraded":"2023-10-25T00:23:36.273751+0000","last_fullsized":"2023-10-25T00:23:36.273751+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_deep_scrub":"72'18","last_deep_scrub_stamp":"2023-10-25T00:22:46.073758+0000","last_clean_scrub_stamp":"2023-10-25T00:22:46.073758+0000","objects_scrubbed":4,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:08:25.787090+0000","scrub_duration":0.012558368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":345,"num_read_kb":345,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"72'2954","reported_seq":3038,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.496407+0000","last_change":"2023-10-25T00:23:36.099045+0000","last_active":"2023-10-25T00:23:36.496407+0000","last_peered":"2023-10-25T00:23:36.496407+0000","last_clean":"2023-10-25T00:23:36.496407+0000","last_became_active":"2023-10-24T23:53:11.798591+0000","last_became_peered":"2023-10-24T23:53:11.798591+0000","last_unstale":"2023-10-25T00:23:36.496407+0000","last_undegraded":"2023-10-25T00:23:36.496407+0000","last_fullsized":"2023-10-25T00:23:36.496407+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2954","last_scrub_stamp":"2023-10-25T00:23:36.098826+0000","last_deep_scrub":"72'2954","last_deep_scrub_stamp":"2023-10-25T00:23:36.098826+0000","last_clean_scrub_stamp":"2023-10-25T00:23:36.098826+0000","objects_scrubbed":0,"log_size":2954,"log_dups_size":0,"ondisk_log_size":2954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:36:16.696480+0000","scrub_duration":0.32992982700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15961,"num_write_kb":219669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"73'38","reported_seq":173,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.317527+0000","last_change":"2023-10-25T00:23:35.052803+0000","last_active":"2023-10-25T00:24:34.317527+0000","last_peered":"2023-10-25T00:24:34.317527+0000","last_clean":"2023-10-25T00:24:34.317527+0000","last_became_active":"2023-10-24T23:53:02.354557+0000","last_became_peered":"2023-10-24T23:53:02.354557+0000","last_unstale":"2023-10-25T00:24:34.317527+0000","last_undegraded":"2023-10-25T00:24:34.317527+0000","last_fullsized":"2023-10-25T00:24:34.317527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'38","last_scrub_stamp":"2023-10-25T00:23:35.052506+0000","last_deep_scrub":"73'38","last_deep_scrub_stamp":"2023-10-25T00:23:35.052506+0000","last_clean_scrub_stamp":"2023-10-25T00:23:35.052506+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:23:31.370767+0000","scrub_duration":0.0095825770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":96,"num_write":37,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'4690","reported_seq":4779,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.325718+0000","last_change":"2023-10-25T00:23:05.792729+0000","last_active":"2023-10-25T00:23:26.325718+0000","last_peered":"2023-10-25T00:23:26.325718+0000","last_clean":"2023-10-25T00:23:26.325718+0000","last_became_active":"2023-10-24T23:53:02.715504+0000","last_became_peered":"2023-10-24T23:53:02.715504+0000","last_unstale":"2023-10-25T00:23:26.325718+0000","last_undegraded":"2023-10-25T00:23:26.325718+0000","last_fullsized":"2023-10-25T00:23:26.325718+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4690","last_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_deep_scrub":"68'4690","last_deep_scrub_stamp":"2023-10-25T00:23:05.792588+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.792588+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:24:53.476824+0000","scrub_duration":0.15089322399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25172,"num_write_kb":305786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"72'10","reported_seq":137,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.542701+0000","last_change":"2023-10-25T00:22:40.512549+0000","last_active":"2023-10-25T00:23:37.542701+0000","last_peered":"2023-10-25T00:23:37.542701+0000","last_clean":"2023-10-25T00:23:37.542701+0000","last_became_active":"2023-10-24T23:53:11.429261+0000","last_became_peered":"2023-10-24T23:53:11.429261+0000","last_unstale":"2023-10-25T00:23:37.542701+0000","last_undegraded":"2023-10-25T00:23:37.542701+0000","last_fullsized":"2023-10-25T00:23:37.542701+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:40.512409+0000","last_clean_scrub_stamp":"2023-10-25T00:22:40.512409+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:07:23.705506+0000","scrub_duration":0.091817919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":176,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"72'4200","reported_seq":4292,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.542502+0000","last_change":"2023-10-25T00:23:34.809510+0000","last_active":"2023-10-25T00:23:37.542502+0000","last_peered":"2023-10-25T00:23:37.542502+0000","last_clean":"2023-10-25T00:23:37.542502+0000","last_became_active":"2023-10-24T23:53:11.812477+0000","last_became_peered":"2023-10-24T23:53:11.812477+0000","last_unstale":"2023-10-25T00:23:37.542502+0000","last_undegraded":"2023-10-25T00:23:37.542502+0000","last_fullsized":"2023-10-25T00:23:37.542502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4200","last_scrub_stamp":"2023-10-25T00:23:34.809362+0000","last_deep_scrub":"72'4200","last_deep_scrub_stamp":"2023-10-25T00:23:34.809362+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.809362+0000","objects_scrubbed":0,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:04:30.830862+0000","scrub_duration":0.23894108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":2,"num_write":24169,"num_write_kb":293796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"74'395","reported_seq":603,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:38.678550+0000","last_change":"2023-10-25T00:23:31.116025+0000","last_active":"2023-10-25T00:24:38.678550+0000","last_peered":"2023-10-25T00:24:38.678550+0000","last_clean":"2023-10-25T00:24:38.678550+0000","last_became_active":"2023-10-24T23:53:02.353772+0000","last_became_peered":"2023-10-24T23:53:02.353772+0000","last_unstale":"2023-10-25T00:24:38.678550+0000","last_undegraded":"2023-10-25T00:24:38.678550+0000","last_fullsized":"2023-10-25T00:24:38.678550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'394","last_scrub_stamp":"2023-10-25T00:23:31.115692+0000","last_deep_scrub":"73'394","last_deep_scrub_stamp":"2023-10-25T00:23:31.115692+0000","last_clean_scrub_stamp":"2023-10-25T00:23:31.115692+0000","objects_scrubbed":4,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:26:38.234847+0000","scrub_duration":0.026138159000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":219,"num_read_kb":98499,"num_write":388,"num_write_kb":4305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'5730","reported_seq":5818,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.204139+0000","last_change":"2023-10-25T00:23:30.204139+0000","last_active":"2023-10-25T00:23:30.204139+0000","last_peered":"2023-10-25T00:23:30.204139+0000","last_clean":"2023-10-25T00:23:30.204139+0000","last_became_active":"2023-10-24T23:53:02.718037+0000","last_became_peered":"2023-10-24T23:53:02.718037+0000","last_unstale":"2023-10-25T00:23:30.204139+0000","last_undegraded":"2023-10-25T00:23:30.204139+0000","last_fullsized":"2023-10-25T00:23:30.204139+0000","mapping_epoch":21,"log_start":"61'500","ondisk_log_start":"61'500","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5730","last_scrub_stamp":"2023-10-25T00:23:30.203981+0000","last_deep_scrub":"68'5730","last_deep_scrub_stamp":"2023-10-25T00:23:30.203981+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.203981+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:38:35.578976+0000","scrub_duration":0.10293530200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":31652,"num_write_kb":356481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"72'3","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.328042+0000","last_change":"2023-10-25T00:23:27.639911+0000","last_active":"2023-10-25T00:23:36.328042+0000","last_peered":"2023-10-25T00:23:36.328042+0000","last_clean":"2023-10-25T00:23:36.328042+0000","last_became_active":"2023-10-24T23:53:11.443441+0000","last_became_peered":"2023-10-24T23:53:11.443441+0000","last_unstale":"2023-10-25T00:23:36.328042+0000","last_undegraded":"2023-10-25T00:23:36.328042+0000","last_fullsized":"2023-10-25T00:23:36.328042+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:27.639604+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:27.639604+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.639604+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:09:33.666294+0000","scrub_duration":0.0047758339999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"72'4168","reported_seq":4256,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.287459+0000","last_change":"2023-10-25T00:23:59.284060+0000","last_active":"2023-10-25T00:24:31.287459+0000","last_peered":"2023-10-25T00:24:31.287459+0000","last_clean":"2023-10-25T00:24:31.287459+0000","last_became_active":"2023-10-24T23:53:11.845269+0000","last_became_peered":"2023-10-24T23:53:11.845269+0000","last_unstale":"2023-10-25T00:24:31.287459+0000","last_undegraded":"2023-10-25T00:24:31.287459+0000","last_fullsized":"2023-10-25T00:24:31.287459+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4168","last_scrub_stamp":"2023-10-25T00:23:59.283890+0000","last_deep_scrub":"72'4168","last_deep_scrub_stamp":"2023-10-25T00:23:59.283890+0000","last_clean_scrub_stamp":"2023-10-25T00:23:59.283890+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:19:04.711112+0000","scrub_duration":0.25867606399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23458,"num_write_kb":294850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"68'4339","reported_seq":4428,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.427145+0000","last_change":"2023-10-25T00:23:47.099492+0000","last_active":"2023-10-25T00:24:21.427145+0000","last_peered":"2023-10-25T00:24:21.427145+0000","last_clean":"2023-10-25T00:24:21.427145+0000","last_became_active":"2023-10-24T23:53:02.722699+0000","last_became_peered":"2023-10-24T23:53:02.722699+0000","last_unstale":"2023-10-25T00:24:21.427145+0000","last_undegraded":"2023-10-25T00:24:21.427145+0000","last_fullsized":"2023-10-25T00:24:21.427145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4339","last_scrub_stamp":"2023-10-25T00:23:47.099343+0000","last_deep_scrub":"68'4339","last_deep_scrub_stamp":"2023-10-25T00:23:47.099343+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.099343+0000","objects_scrubbed":0,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:10:27.913875+0000","scrub_duration":0.12538819900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23267,"num_write_kb":265268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:20.676577+0000","last_change":"2023-10-25T00:22:42.199286+0000","last_active":"2023-10-25T00:23:20.676577+0000","last_peered":"2023-10-25T00:23:20.676577+0000","last_clean":"2023-10-25T00:23:20.676577+0000","last_became_active":"2023-10-24T23:52:58.164542+0000","last_became_peered":"2023-10-24T23:52:58.164542+0000","last_unstale":"2023-10-25T00:23:20.676577+0000","last_undegraded":"2023-10-25T00:23:20.676577+0000","last_fullsized":"2023-10-25T00:23:20.676577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:42.198947+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.198947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:11:17.807377+0000","scrub_duration":0.0041955760000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"74'232","reported_seq":345,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.321697+0000","last_change":"2023-10-25T00:23:16.010034+0000","last_active":"2023-10-25T00:24:34.321697+0000","last_peered":"2023-10-25T00:24:34.321697+0000","last_clean":"2023-10-25T00:24:34.321697+0000","last_became_active":"2023-10-24T23:53:02.338908+0000","last_became_peered":"2023-10-24T23:53:02.338908+0000","last_unstale":"2023-10-25T00:24:34.321697+0000","last_undegraded":"2023-10-25T00:24:34.321697+0000","last_fullsized":"2023-10-25T00:24:34.321697+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'231","last_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_deep_scrub":"73'231","last_deep_scrub_stamp":"2023-10-25T00:23:16.009842+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.009842+0000","objects_scrubbed":1,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:27:21.881029+0000","scrub_duration":0.076880452000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":65,"num_write":591,"num_write_kb":232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"72'11","reported_seq":96,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:17.537123+0000","last_change":"2023-10-25T00:22:38.409520+0000","last_active":"2023-10-25T00:23:17.537123+0000","last_peered":"2023-10-25T00:23:17.537123+0000","last_clean":"2023-10-25T00:23:17.537123+0000","last_became_active":"2023-10-24T23:53:11.432994+0000","last_became_peered":"2023-10-24T23:53:11.432994+0000","last_unstale":"2023-10-25T00:23:17.537123+0000","last_undegraded":"2023-10-25T00:23:17.537123+0000","last_fullsized":"2023-10-25T00:23:17.537123+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_deep_scrub":"72'11","last_deep_scrub_stamp":"2023-10-25T00:22:38.409235+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.409235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:21:24.708709+0000","scrub_duration":0.003592329,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"72'3993","reported_seq":4082,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.429639+0000","last_change":"2023-10-25T00:23:58.292855+0000","last_active":"2023-10-25T00:24:31.429639+0000","last_peered":"2023-10-25T00:24:31.429639+0000","last_clean":"2023-10-25T00:24:31.429639+0000","last_became_active":"2023-10-24T23:53:11.824574+0000","last_became_peered":"2023-10-24T23:53:11.824574+0000","last_unstale":"2023-10-25T00:24:31.429639+0000","last_undegraded":"2023-10-25T00:24:31.429639+0000","last_fullsized":"2023-10-25T00:24:31.429639+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3993","last_scrub_stamp":"2023-10-25T00:23:58.292710+0000","last_deep_scrub":"72'3993","last_deep_scrub_stamp":"2023-10-25T00:23:58.292710+0000","last_clean_scrub_stamp":"2023-10-25T00:23:58.292710+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:23:08.798165+0000","scrub_duration":0.310815756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21723,"num_write_kb":283850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"68'4270","reported_seq":4359,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:07.535306+0000","last_change":"2023-10-25T00:22:59.570791+0000","last_active":"2023-10-25T00:23:07.535306+0000","last_peered":"2023-10-25T00:23:07.535306+0000","last_clean":"2023-10-25T00:23:07.535306+0000","last_became_active":"2023-10-24T23:53:02.713544+0000","last_became_peered":"2023-10-24T23:53:02.713544+0000","last_unstale":"2023-10-25T00:23:07.535306+0000","last_undegraded":"2023-10-25T00:23:07.535306+0000","last_fullsized":"2023-10-25T00:23:07.535306+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4270","last_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_deep_scrub":"68'4270","last_deep_scrub_stamp":"2023-10-25T00:22:59.570628+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.570628+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:36:56.294621+0000","scrub_duration":0.14092118100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23049,"num_write_kb":276864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.324802+0000","last_change":"2023-10-25T00:23:06.643510+0000","last_active":"2023-10-25T00:23:21.324802+0000","last_peered":"2023-10-25T00:23:21.324802+0000","last_clean":"2023-10-25T00:23:21.324802+0000","last_became_active":"2023-10-24T23:52:58.163648+0000","last_became_peered":"2023-10-24T23:52:58.163648+0000","last_unstale":"2023-10-25T00:23:21.324802+0000","last_undegraded":"2023-10-25T00:23:21.324802+0000","last_fullsized":"2023-10-25T00:23:21.324802+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:06.643156+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.643156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:34:24.773001+0000","scrub_duration":0.0045495850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"73'10","reported_seq":148,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:38.466387+0000","last_change":"2023-10-25T00:23:20.618489+0000","last_active":"2023-10-25T00:24:38.466387+0000","last_peered":"2023-10-25T00:24:38.466387+0000","last_clean":"2023-10-25T00:24:38.466387+0000","last_became_active":"2023-10-24T23:53:02.354425+0000","last_became_peered":"2023-10-24T23:53:02.354425+0000","last_unstale":"2023-10-25T00:24:38.466387+0000","last_undegraded":"2023-10-25T00:24:38.466387+0000","last_fullsized":"2023-10-25T00:24:38.466387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2023-10-25T00:23:20.618371+0000","last_deep_scrub":"73'10","last_deep_scrub_stamp":"2023-10-25T00:23:20.618371+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.618371+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:30:10.455022+0000","scrub_duration":0.085107656000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":150,"num_read_kb":150,"num_write":8,"num_write_kb":53,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"73'65","reported_seq":230,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.429515+0000","last_change":"2023-10-25T00:23:07.930465+0000","last_active":"2023-10-25T00:24:31.429515+0000","last_peered":"2023-10-25T00:24:31.429515+0000","last_clean":"2023-10-25T00:24:31.429515+0000","last_became_active":"2023-10-24T23:53:11.441537+0000","last_became_peered":"2023-10-24T23:53:11.441537+0000","last_unstale":"2023-10-25T00:24:31.429515+0000","last_undegraded":"2023-10-25T00:24:31.429515+0000","last_fullsized":"2023-10-25T00:24:31.429515+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'65","last_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_deep_scrub":"73'65","last_deep_scrub_stamp":"2023-10-25T00:23:07.930143+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.930143+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:41:49.528670+0000","scrub_duration":0.020886851000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84545,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":123,"num_write":59,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"72'3769","reported_seq":3858,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.341574+0000","last_change":"2023-10-25T00:23:38.050117+0000","last_active":"2023-10-25T00:24:31.341574+0000","last_peered":"2023-10-25T00:24:31.341574+0000","last_clean":"2023-10-25T00:24:31.341574+0000","last_became_active":"2023-10-24T23:53:11.821536+0000","last_became_peered":"2023-10-24T23:53:11.821536+0000","last_unstale":"2023-10-25T00:24:31.341574+0000","last_undegraded":"2023-10-25T00:24:31.341574+0000","last_fullsized":"2023-10-25T00:24:31.341574+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3769","last_scrub_stamp":"2023-10-25T00:23:38.049889+0000","last_deep_scrub":"72'3769","last_deep_scrub_stamp":"2023-10-25T00:23:38.049889+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.049889+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:23:14.574715+0000","scrub_duration":0.37571605800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20807,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'5752","reported_seq":5841,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.507272+0000","last_change":"2023-10-25T00:22:33.937946+0000","last_active":"2023-10-25T00:24:21.507272+0000","last_peered":"2023-10-25T00:24:21.507272+0000","last_clean":"2023-10-25T00:24:21.507272+0000","last_became_active":"2023-10-24T23:53:02.725778+0000","last_became_peered":"2023-10-24T23:53:02.725778+0000","last_unstale":"2023-10-25T00:24:21.507272+0000","last_undegraded":"2023-10-25T00:24:21.507272+0000","last_fullsized":"2023-10-25T00:24:21.507272+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5752","last_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_deep_scrub":"68'5752","last_deep_scrub_stamp":"2023-10-25T00:22:33.937564+0000","last_clean_scrub_stamp":"2023-10-25T00:22:33.937564+0000","objects_scrubbed":0,"log_size":5752,"log_dups_size":0,"ondisk_log_size":5752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:43:34.779084+0000","scrub_duration":0.080678784000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31231,"num_write_kb":354718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:16.283897+0000","last_change":"2023-10-25T00:23:24.047030+0000","last_active":"2023-10-25T00:24:16.283897+0000","last_peered":"2023-10-25T00:24:16.283897+0000","last_clean":"2023-10-25T00:24:16.283897+0000","last_became_active":"2023-10-24T23:52:58.590726+0000","last_became_peered":"2023-10-24T23:52:58.590726+0000","last_unstale":"2023-10-25T00:24:16.283897+0000","last_undegraded":"2023-10-25T00:24:16.283897+0000","last_fullsized":"2023-10-25T00:24:16.283897+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:24.046716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:24.046716+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.046716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:53:42.506468+0000","scrub_duration":0.0043444529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'2","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.339157+0000","last_change":"2023-10-25T00:23:18.684046+0000","last_active":"2023-10-25T00:24:21.339157+0000","last_peered":"2023-10-25T00:24:21.339157+0000","last_clean":"2023-10-25T00:24:21.339157+0000","last_became_active":"2023-10-24T23:53:02.331084+0000","last_became_peered":"2023-10-24T23:53:02.331084+0000","last_unstale":"2023-10-25T00:24:21.339157+0000","last_undegraded":"2023-10-25T00:24:21.339157+0000","last_fullsized":"2023-10-25T00:24:21.339157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2023-10-25T00:23:18.683799+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2023-10-25T00:23:18.683799+0000","last_clean_scrub_stamp":"2023-10-25T00:23:18.683799+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:26:44.716117+0000","scrub_duration":0.013956036,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"72'4","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:30.690283+0000","last_change":"2023-10-25T00:23:06.496904+0000","last_active":"2023-10-25T00:24:30.690283+0000","last_peered":"2023-10-25T00:24:30.690283+0000","last_clean":"2023-10-25T00:24:30.690283+0000","last_became_active":"2023-10-24T23:53:11.425420+0000","last_became_peered":"2023-10-24T23:53:11.425420+0000","last_unstale":"2023-10-25T00:24:30.690283+0000","last_undegraded":"2023-10-25T00:24:30.690283+0000","last_fullsized":"2023-10-25T00:24:30.690283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_deep_scrub":"72'4","last_deep_scrub_stamp":"2023-10-25T00:23:06.496575+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.496575+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:32:18.665160+0000","scrub_duration":0.0046312189999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"72'3939","reported_seq":4023,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:21.492760+0000","last_change":"2023-10-25T00:23:02.160279+0000","last_active":"2023-10-25T00:23:21.492760+0000","last_peered":"2023-10-25T00:23:21.492760+0000","last_clean":"2023-10-25T00:23:21.492760+0000","last_became_active":"2023-10-24T23:53:11.810863+0000","last_became_peered":"2023-10-24T23:53:11.810863+0000","last_unstale":"2023-10-25T00:23:21.492760+0000","last_undegraded":"2023-10-25T00:23:21.492760+0000","last_fullsized":"2023-10-25T00:23:21.492760+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3939","last_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_deep_scrub":"72'3939","last_deep_scrub_stamp":"2023-10-25T00:23:02.160086+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.160086+0000","objects_scrubbed":0,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:37:29.878702+0000","scrub_duration":0.252775894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22393,"num_write_kb":262004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"68'4305","reported_seq":4394,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.339050+0000","last_change":"2023-10-25T00:23:26.858862+0000","last_active":"2023-10-25T00:24:21.339050+0000","last_peered":"2023-10-25T00:24:21.339050+0000","last_clean":"2023-10-25T00:24:21.339050+0000","last_became_active":"2023-10-24T23:53:02.714913+0000","last_became_peered":"2023-10-24T23:53:02.714913+0000","last_unstale":"2023-10-25T00:24:21.339050+0000","last_undegraded":"2023-10-25T00:24:21.339050+0000","last_fullsized":"2023-10-25T00:24:21.339050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4305","last_scrub_stamp":"2023-10-25T00:23:26.858657+0000","last_deep_scrub":"68'4305","last_deep_scrub_stamp":"2023-10-25T00:23:26.858657+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.858657+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:10:03.352691+0000","scrub_duration":0.18050687100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22696,"num_write_kb":287574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:16.338023+0000","last_change":"2023-10-25T00:23:11.717353+0000","last_active":"2023-10-25T00:24:16.338023+0000","last_peered":"2023-10-25T00:24:16.338023+0000","last_clean":"2023-10-25T00:24:16.338023+0000","last_became_active":"2023-10-24T23:52:58.591865+0000","last_became_peered":"2023-10-24T23:52:58.591865+0000","last_unstale":"2023-10-25T00:24:16.338023+0000","last_undegraded":"2023-10-25T00:24:16.338023+0000","last_fullsized":"2023-10-25T00:24:16.338023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:11.717027+0000","last_clean_scrub_stamp":"2023-10-25T00:23:11.717027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:17:42.982457+0000","scrub_duration":0.0039434270000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"67'2","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:26.109514+0000","last_change":"2023-10-25T00:23:30.355099+0000","last_active":"2023-10-25T00:24:26.109514+0000","last_peered":"2023-10-25T00:24:26.109514+0000","last_clean":"2023-10-25T00:24:26.109514+0000","last_became_active":"2023-10-24T23:53:02.336835+0000","last_became_peered":"2023-10-24T23:53:02.336835+0000","last_unstale":"2023-10-25T00:24:26.109514+0000","last_undegraded":"2023-10-25T00:24:26.109514+0000","last_fullsized":"2023-10-25T00:24:26.109514+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2023-10-25T00:23:30.354780+0000","last_deep_scrub":"67'2","last_deep_scrub_stamp":"2023-10-25T00:23:30.354780+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.354780+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:07:40.743916+0000","scrub_duration":0.0044225610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"72'3","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.509696+0000","last_change":"2023-10-25T00:22:52.952897+0000","last_active":"2023-10-25T00:24:31.509696+0000","last_peered":"2023-10-25T00:24:31.509696+0000","last_clean":"2023-10-25T00:24:31.509696+0000","last_became_active":"2023-10-24T23:53:11.425292+0000","last_became_peered":"2023-10-24T23:53:11.425292+0000","last_unstale":"2023-10-25T00:24:31.509696+0000","last_undegraded":"2023-10-25T00:24:31.509696+0000","last_fullsized":"2023-10-25T00:24:31.509696+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:52.952595+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.952595+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:14:20.890322+0000","scrub_duration":0.0044650949999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"72'5003","reported_seq":5093,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:44.774832+0000","last_change":"2023-10-25T00:23:44.774832+0000","last_active":"2023-10-25T00:23:44.774832+0000","last_peered":"2023-10-25T00:23:44.774832+0000","last_clean":"2023-10-25T00:23:44.774832+0000","last_became_active":"2023-10-24T23:53:11.827891+0000","last_became_peered":"2023-10-24T23:53:11.827891+0000","last_unstale":"2023-10-25T00:23:44.774832+0000","last_undegraded":"2023-10-25T00:23:44.774832+0000","last_fullsized":"2023-10-25T00:23:44.774832+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5003","last_scrub_stamp":"2023-10-25T00:23:44.774508+0000","last_deep_scrub":"72'5003","last_deep_scrub_stamp":"2023-10-25T00:23:44.774508+0000","last_clean_scrub_stamp":"2023-10-25T00:23:44.774508+0000","objects_scrubbed":0,"log_size":5003,"log_dups_size":0,"ondisk_log_size":5003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:26:52.739672+0000","scrub_duration":0.33716653400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":13,"num_write":27120,"num_write_kb":350355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"68'5635","reported_seq":5723,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.339129+0000","last_change":"2023-10-25T00:22:44.547139+0000","last_active":"2023-10-25T00:24:21.339129+0000","last_peered":"2023-10-25T00:24:21.339129+0000","last_clean":"2023-10-25T00:24:21.339129+0000","last_became_active":"2023-10-24T23:53:02.717543+0000","last_became_peered":"2023-10-24T23:53:02.717543+0000","last_unstale":"2023-10-25T00:24:21.339129+0000","last_undegraded":"2023-10-25T00:24:21.339129+0000","last_fullsized":"2023-10-25T00:24:21.339129+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5635","last_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_deep_scrub":"68'5635","last_deep_scrub_stamp":"2023-10-25T00:22:44.546984+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.546984+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:51:54.474684+0000","scrub_duration":0.15865177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30556,"num_write_kb":356140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"68'3","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:25.100690+0000","last_change":"2023-10-25T00:24:25.100690+0000","last_active":"2023-10-25T00:24:25.100690+0000","last_peered":"2023-10-25T00:24:25.100690+0000","last_clean":"2023-10-25T00:24:25.100690+0000","last_became_active":"2023-10-24T23:53:02.342018+0000","last_became_peered":"2023-10-24T23:53:02.342018+0000","last_unstale":"2023-10-25T00:24:25.100690+0000","last_undegraded":"2023-10-25T00:24:25.100690+0000","last_fullsized":"2023-10-25T00:24:25.100690+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:24:25.100343+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:24:25.100343+0000","last_clean_scrub_stamp":"2023-10-25T00:24:25.100343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:00:54.991882+0000","scrub_duration":0.0050738520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:16.107164+0000","last_change":"2023-10-25T00:23:29.305381+0000","last_active":"2023-10-25T00:24:16.107164+0000","last_peered":"2023-10-25T00:24:16.107164+0000","last_clean":"2023-10-25T00:24:16.107164+0000","last_became_active":"2023-10-24T23:52:58.158440+0000","last_became_peered":"2023-10-24T23:52:58.158440+0000","last_unstale":"2023-10-25T00:24:16.107164+0000","last_undegraded":"2023-10-25T00:24:16.107164+0000","last_fullsized":"2023-10-25T00:24:16.107164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:29.305022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:29.305022+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.305022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:52:12.477955+0000","scrub_duration":0.0044947839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"72'2","reported_seq":87,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315751+0000","last_change":"2023-10-25T00:22:58.925685+0000","last_active":"2023-10-25T00:24:34.315751+0000","last_peered":"2023-10-25T00:24:34.315751+0000","last_clean":"2023-10-25T00:24:34.315751+0000","last_became_active":"2023-10-24T23:53:11.431805+0000","last_became_peered":"2023-10-24T23:53:11.431805+0000","last_unstale":"2023-10-25T00:24:34.315751+0000","last_undegraded":"2023-10-25T00:24:34.315751+0000","last_fullsized":"2023-10-25T00:24:34.315751+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:58.925318+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.925318+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:31:45.562398+0000","scrub_duration":0.0043101720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"72'3753","reported_seq":3841,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.287558+0000","last_change":"2023-10-25T00:23:44.352225+0000","last_active":"2023-10-25T00:24:31.287558+0000","last_peered":"2023-10-25T00:24:31.287558+0000","last_clean":"2023-10-25T00:24:31.287558+0000","last_became_active":"2023-10-24T23:53:11.859758+0000","last_became_peered":"2023-10-24T23:53:11.859758+0000","last_unstale":"2023-10-25T00:24:31.287558+0000","last_undegraded":"2023-10-25T00:24:31.287558+0000","last_fullsized":"2023-10-25T00:24:31.287558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3753","last_scrub_stamp":"2023-10-25T00:23:44.352083+0000","last_deep_scrub":"72'3753","last_deep_scrub_stamp":"2023-10-25T00:23:44.352083+0000","last_clean_scrub_stamp":"2023-10-25T00:23:44.352083+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:13:26.541612+0000","scrub_duration":0.328578174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":21045,"num_write_kb":276986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"68'4101","reported_seq":4188,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.325983+0000","last_change":"2023-10-25T00:23:25.811837+0000","last_active":"2023-10-25T00:23:26.325983+0000","last_peered":"2023-10-25T00:23:26.325983+0000","last_clean":"2023-10-25T00:23:26.325983+0000","last_became_active":"2023-10-24T23:53:02.714842+0000","last_became_peered":"2023-10-24T23:53:02.714842+0000","last_unstale":"2023-10-25T00:23:26.325983+0000","last_undegraded":"2023-10-25T00:23:26.325983+0000","last_fullsized":"2023-10-25T00:23:26.325983+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:23:25.811682+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:23:25.811682+0000","last_clean_scrub_stamp":"2023-10-25T00:23:25.811682+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:39:30.752063+0000","scrub_duration":0.14318265299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22383,"num_write_kb":270023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"74'14","reported_seq":103,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.322235+0000","last_change":"2023-10-25T00:23:17.651450+0000","last_active":"2023-10-25T00:24:35.322235+0000","last_peered":"2023-10-25T00:24:35.322235+0000","last_clean":"2023-10-25T00:24:35.322235+0000","last_became_active":"2023-10-24T23:53:02.332780+0000","last_became_peered":"2023-10-24T23:53:02.332780+0000","last_unstale":"2023-10-25T00:24:35.322235+0000","last_undegraded":"2023-10-25T00:24:35.322235+0000","last_fullsized":"2023-10-25T00:24:35.322235+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2023-10-25T00:23:17.651103+0000","last_deep_scrub":"73'13","last_deep_scrub_stamp":"2023-10-25T00:23:17.651103+0000","last_clean_scrub_stamp":"2023-10-25T00:23:17.651103+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:05:54.986553+0000","scrub_duration":0.014684019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:15.836497+0000","last_change":"2023-10-25T00:22:51.317813+0000","last_active":"2023-10-25T00:24:15.836497+0000","last_peered":"2023-10-25T00:24:15.836497+0000","last_clean":"2023-10-25T00:24:15.836497+0000","last_became_active":"2023-10-24T23:52:58.162469+0000","last_became_peered":"2023-10-24T23:52:58.162469+0000","last_unstale":"2023-10-25T00:24:15.836497+0000","last_undegraded":"2023-10-25T00:24:15.836497+0000","last_fullsized":"2023-10-25T00:24:15.836497+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:22:51.317462+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.317462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:03:14.150576+0000","scrub_duration":0.0044508079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"73'249","reported_seq":355,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.820915+0000","last_change":"2023-10-25T00:23:21.397705+0000","last_active":"2023-10-25T00:23:35.820915+0000","last_peered":"2023-10-25T00:23:35.820915+0000","last_clean":"2023-10-25T00:23:35.820915+0000","last_became_active":"2023-10-24T23:53:11.443310+0000","last_became_peered":"2023-10-24T23:53:11.443310+0000","last_unstale":"2023-10-25T00:23:35.820915+0000","last_undegraded":"2023-10-25T00:23:35.820915+0000","last_fullsized":"2023-10-25T00:23:35.820915+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'249","last_scrub_stamp":"2023-10-25T00:23:21.397410+0000","last_deep_scrub":"73'249","last_deep_scrub_stamp":"2023-10-25T00:23:21.397410+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.397410+0000","objects_scrubbed":1,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:19:14.548554+0000","scrub_duration":0.017693341000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":62,"num_write":607,"num_write_kb":223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"72'5236","reported_seq":5326,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.341814+0000","last_change":"2023-10-25T00:23:34.900038+0000","last_active":"2023-10-25T00:24:31.341814+0000","last_peered":"2023-10-25T00:24:31.341814+0000","last_clean":"2023-10-25T00:24:31.341814+0000","last_became_active":"2023-10-24T23:53:11.806001+0000","last_became_peered":"2023-10-24T23:53:11.806001+0000","last_unstale":"2023-10-25T00:24:31.341814+0000","last_undegraded":"2023-10-25T00:24:31.341814+0000","last_fullsized":"2023-10-25T00:24:31.341814+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5236","last_scrub_stamp":"2023-10-25T00:23:34.899894+0000","last_deep_scrub":"72'5236","last_deep_scrub_stamp":"2023-10-25T00:23:34.899894+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.899894+0000","objects_scrubbed":0,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:10:20.497931+0000","scrub_duration":0.29917875999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":14,"num_write":29554,"num_write_kb":368820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"68'5444","reported_seq":5535,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.494343+0000","last_change":"2023-10-25T00:23:00.141618+0000","last_active":"2023-10-25T00:23:26.494343+0000","last_peered":"2023-10-25T00:23:26.494343+0000","last_clean":"2023-10-25T00:23:26.494343+0000","last_became_active":"2023-10-24T23:53:02.726041+0000","last_became_peered":"2023-10-24T23:53:02.726041+0000","last_unstale":"2023-10-25T00:23:26.494343+0000","last_undegraded":"2023-10-25T00:23:26.494343+0000","last_fullsized":"2023-10-25T00:23:26.494343+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5444","last_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_deep_scrub":"68'5444","last_deep_scrub_stamp":"2023-10-25T00:23:00.141458+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.141458+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:23:18.614380+0000","scrub_duration":0.197741424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28342,"num_write_kb":342931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"73'67","reported_seq":241,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315086+0000","last_change":"2023-10-25T00:23:14.941868+0000","last_active":"2023-10-25T00:24:34.315086+0000","last_peered":"2023-10-25T00:24:34.315086+0000","last_clean":"2023-10-25T00:24:34.315086+0000","last_became_active":"2023-10-24T23:53:02.358056+0000","last_became_peered":"2023-10-24T23:53:02.358056+0000","last_unstale":"2023-10-25T00:24:34.315086+0000","last_undegraded":"2023-10-25T00:24:34.315086+0000","last_fullsized":"2023-10-25T00:24:34.315086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'67","last_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_deep_scrub":"73'67","last_deep_scrub_stamp":"2023-10-25T00:23:14.941564+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.941564+0000","objects_scrubbed":1,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:11:54.084201+0000","scrub_duration":0.0084463549999999991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":125,"num_write":61,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:22.074393+0000","last_change":"2023-10-25T00:23:22.074393+0000","last_active":"2023-10-25T00:23:22.074393+0000","last_peered":"2023-10-25T00:23:22.074393+0000","last_clean":"2023-10-25T00:23:22.074393+0000","last_became_active":"2023-10-24T23:52:58.162360+0000","last_became_peered":"2023-10-24T23:52:58.162360+0000","last_unstale":"2023-10-25T00:23:22.074393+0000","last_undegraded":"2023-10-25T00:23:22.074393+0000","last_fullsized":"2023-10-25T00:23:22.074393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:22.074087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:22.074087+0000","last_clean_scrub_stamp":"2023-10-25T00:23:22.074087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:59:48.682336+0000","scrub_duration":0.0044663139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"72'10","reported_seq":139,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.097599+0000","last_change":"2023-10-25T00:22:45.425308+0000","last_active":"2023-10-25T00:23:36.097599+0000","last_peered":"2023-10-25T00:23:36.097599+0000","last_clean":"2023-10-25T00:23:36.097599+0000","last_became_active":"2023-10-24T23:53:11.435835+0000","last_became_peered":"2023-10-24T23:53:11.435835+0000","last_unstale":"2023-10-25T00:23:36.097599+0000","last_undegraded":"2023-10-25T00:23:36.097599+0000","last_fullsized":"2023-10-25T00:23:36.097599+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_deep_scrub":"72'10","last_deep_scrub_stamp":"2023-10-25T00:22:45.425052+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.425052+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:36:38.527058+0000","scrub_duration":0.14014134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":135,"num_read_kb":135,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"72'3666","reported_seq":3750,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.807972+0000","last_change":"2023-10-25T00:23:45.807972+0000","last_active":"2023-10-25T00:23:45.807972+0000","last_peered":"2023-10-25T00:23:45.807972+0000","last_clean":"2023-10-25T00:23:45.807972+0000","last_became_active":"2023-10-24T23:53:11.822733+0000","last_became_peered":"2023-10-24T23:53:11.822733+0000","last_unstale":"2023-10-25T00:23:45.807972+0000","last_undegraded":"2023-10-25T00:23:45.807972+0000","last_fullsized":"2023-10-25T00:23:45.807972+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3666","last_scrub_stamp":"2023-10-25T00:23:45.807770+0000","last_deep_scrub":"72'3666","last_deep_scrub_stamp":"2023-10-25T00:23:45.807770+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.807770+0000","objects_scrubbed":0,"log_size":3666,"log_dups_size":0,"ondisk_log_size":3666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:26:55.156976+0000","scrub_duration":0.28592742900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20519,"num_write_kb":254391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"68'4182","reported_seq":4271,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:25.689389+0000","last_change":"2023-10-25T00:23:02.489861+0000","last_active":"2023-10-25T00:24:25.689389+0000","last_peered":"2023-10-25T00:24:25.689389+0000","last_clean":"2023-10-25T00:24:25.689389+0000","last_became_active":"2023-10-24T23:53:02.725310+0000","last_became_peered":"2023-10-24T23:53:02.725310+0000","last_unstale":"2023-10-25T00:24:25.689389+0000","last_undegraded":"2023-10-25T00:24:25.689389+0000","last_fullsized":"2023-10-25T00:24:25.689389+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4182","last_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_deep_scrub":"68'4182","last_deep_scrub_stamp":"2023-10-25T00:23:02.489471+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.489471+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:14:08.666419+0000","scrub_duration":0.092197608,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22839,"num_write_kb":259586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'4","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:22.567237+0000","last_change":"2023-10-25T00:23:15.544275+0000","last_active":"2023-10-25T00:24:22.567237+0000","last_peered":"2023-10-25T00:24:22.567237+0000","last_clean":"2023-10-25T00:24:22.567237+0000","last_became_active":"2023-10-24T23:53:02.326268+0000","last_became_peered":"2023-10-24T23:53:02.326268+0000","last_unstale":"2023-10-25T00:24:22.567237+0000","last_undegraded":"2023-10-25T00:24:22.567237+0000","last_fullsized":"2023-10-25T00:24:22.567237+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4","last_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_deep_scrub":"67'4","last_deep_scrub_stamp":"2023-10-25T00:23:15.543988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:15.543988+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:33:51.332032+0000","scrub_duration":0.004459345,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:16.107245+0000","last_change":"2023-10-25T00:23:28.260731+0000","last_active":"2023-10-25T00:24:16.107245+0000","last_peered":"2023-10-25T00:24:16.107245+0000","last_clean":"2023-10-25T00:24:16.107245+0000","last_became_active":"2023-10-24T23:52:58.162739+0000","last_became_peered":"2023-10-24T23:52:58.162739+0000","last_unstale":"2023-10-25T00:24:16.107245+0000","last_undegraded":"2023-10-25T00:24:16.107245+0000","last_fullsized":"2023-10-25T00:24:16.107245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-25T00:23:28.260409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-25T00:23:28.260409+0000","last_clean_scrub_stamp":"2023-10-25T00:23:28.260409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:10:44.862291+0000","scrub_duration":0.0042298140000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"73'163","reported_seq":301,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.314950+0000","last_change":"2023-10-25T00:23:17.149791+0000","last_active":"2023-10-25T00:24:34.314950+0000","last_peered":"2023-10-25T00:24:34.314950+0000","last_clean":"2023-10-25T00:24:34.314950+0000","last_became_active":"2023-10-24T23:52:52.441206+0000","last_became_peered":"2023-10-24T23:52:52.441206+0000","last_unstale":"2023-10-25T00:24:34.314950+0000","last_undegraded":"2023-10-25T00:24:34.314950+0000","last_fullsized":"2023-10-25T00:24:34.314950+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'163","last_scrub_stamp":"2023-10-25T00:23:17.149492+0000","last_deep_scrub":"73'163","last_deep_scrub_stamp":"2023-10-25T00:23:17.149492+0000","last_clean_scrub_stamp":"2023-10-25T00:23:17.149492+0000","objects_scrubbed":2,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:59:05.394868+0000","scrub_duration":0.012985958000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"72'2","reported_seq":87,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.287521+0000","last_change":"2023-10-25T00:22:48.067150+0000","last_active":"2023-10-25T00:24:31.287521+0000","last_peered":"2023-10-25T00:24:31.287521+0000","last_clean":"2023-10-25T00:24:31.287521+0000","last_became_active":"2023-10-24T23:53:11.450574+0000","last_became_peered":"2023-10-24T23:53:11.450574+0000","last_unstale":"2023-10-25T00:24:31.287521+0000","last_undegraded":"2023-10-25T00:24:31.287521+0000","last_fullsized":"2023-10-25T00:24:31.287521+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:48.066810+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.066810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:55:20.112674+0000","scrub_duration":0.0050122709999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"72'4105","reported_seq":4193,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.297244+0000","last_change":"2023-10-25T00:23:20.535530+0000","last_active":"2023-10-25T00:24:34.297244+0000","last_peered":"2023-10-25T00:24:34.297244+0000","last_clean":"2023-10-25T00:24:34.297244+0000","last_became_active":"2023-10-24T23:53:11.830587+0000","last_became_peered":"2023-10-24T23:53:11.830587+0000","last_unstale":"2023-10-25T00:24:34.297244+0000","last_undegraded":"2023-10-25T00:24:34.297244+0000","last_fullsized":"2023-10-25T00:24:34.297244+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4105","last_scrub_stamp":"2023-10-25T00:23:20.535378+0000","last_deep_scrub":"72'4105","last_deep_scrub_stamp":"2023-10-25T00:23:20.535378+0000","last_clean_scrub_stamp":"2023-10-25T00:23:20.535378+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:20:52.287445+0000","scrub_duration":0.31792611900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":19,"num_write":23147,"num_write_kb":297959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"68'5705","reported_seq":5795,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.324915+0000","last_change":"2023-10-25T00:23:39.836379+0000","last_active":"2023-10-25T00:24:35.324915+0000","last_peered":"2023-10-25T00:24:35.324915+0000","last_clean":"2023-10-25T00:24:35.324915+0000","last_became_active":"2023-10-24T23:53:02.716343+0000","last_became_peered":"2023-10-24T23:53:02.716343+0000","last_unstale":"2023-10-25T00:24:35.324915+0000","last_undegraded":"2023-10-25T00:24:35.324915+0000","last_fullsized":"2023-10-25T00:24:35.324915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5705","last_scrub_stamp":"2023-10-25T00:23:39.836205+0000","last_deep_scrub":"68'5705","last_deep_scrub_stamp":"2023-10-25T00:23:39.836205+0000","last_clean_scrub_stamp":"2023-10-25T00:23:39.836205+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:38:02.740862+0000","scrub_duration":0.105332388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":29845,"num_write_kb":357286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"68'5","reported_seq":93,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.539642+0000","last_change":"2023-10-25T00:23:18.535332+0000","last_active":"2023-10-25T00:23:27.539642+0000","last_peered":"2023-10-25T00:23:27.539642+0000","last_clean":"2023-10-25T00:23:27.539642+0000","last_became_active":"2023-10-24T23:53:02.337601+0000","last_became_peered":"2023-10-24T23:53:02.337601+0000","last_unstale":"2023-10-25T00:23:27.539642+0000","last_undegraded":"2023-10-25T00:23:27.539642+0000","last_fullsized":"2023-10-25T00:23:27.539642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:18.534951+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:18.534951+0000","last_clean_scrub_stamp":"2023-10-25T00:23:18.534951+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:55:37.756970+0000","scrub_duration":0.0037857049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":139,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.271123+0000","last_change":"2023-10-25T00:24:35.271123+0000","last_active":"2023-10-25T00:24:35.271123+0000","last_peered":"2023-10-25T00:24:35.271123+0000","last_clean":"2023-10-25T00:24:35.271123+0000","last_became_active":"2023-10-24T23:53:11.449715+0000","last_became_peered":"2023-10-24T23:53:11.449715+0000","last_unstale":"2023-10-25T00:24:35.271123+0000","last_undegraded":"2023-10-25T00:24:35.271123+0000","last_fullsized":"2023-10-25T00:24:35.271123+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2023-10-25T00:24:35.270780+0000","last_deep_scrub":"74'8","last_deep_scrub_stamp":"2023-10-25T00:24:35.270780+0000","last_clean_scrub_stamp":"2023-10-25T00:24:35.270780+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:15:45.057328+0000","scrub_duration":0.0084190350000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"72'4234","reported_seq":4321,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.312107+0000","last_change":"2023-10-25T00:23:41.751679+0000","last_active":"2023-10-25T00:24:34.312107+0000","last_peered":"2023-10-25T00:24:34.312107+0000","last_clean":"2023-10-25T00:24:34.312107+0000","last_became_active":"2023-10-24T23:53:11.829812+0000","last_became_peered":"2023-10-24T23:53:11.829812+0000","last_unstale":"2023-10-25T00:24:34.312107+0000","last_undegraded":"2023-10-25T00:24:34.312107+0000","last_fullsized":"2023-10-25T00:24:34.312107+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4234","last_scrub_stamp":"2023-10-25T00:23:41.751558+0000","last_deep_scrub":"72'4234","last_deep_scrub_stamp":"2023-10-25T00:23:41.751558+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.751558+0000","objects_scrubbed":0,"log_size":4234,"log_dups_size":0,"ondisk_log_size":4234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:50:10.142078+0000","scrub_duration":0.256161054,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":15,"num_write":23173,"num_write_kb":302022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"68'4052","reported_seq":4140,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.288416+0000","last_change":"2023-10-25T00:23:49.042663+0000","last_active":"2023-10-25T00:24:34.288416+0000","last_peered":"2023-10-25T00:24:34.288416+0000","last_clean":"2023-10-25T00:24:34.288416+0000","last_became_active":"2023-10-24T23:53:02.718128+0000","last_became_peered":"2023-10-24T23:53:02.718128+0000","last_unstale":"2023-10-25T00:24:34.288416+0000","last_undegraded":"2023-10-25T00:24:34.288416+0000","last_fullsized":"2023-10-25T00:24:34.288416+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4052","last_scrub_stamp":"2023-10-25T00:23:49.042529+0000","last_deep_scrub":"68'4052","last_deep_scrub_stamp":"2023-10-25T00:23:49.042529+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.042529+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:40:32.211760+0000","scrub_duration":0.105715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21486,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"68'2","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:34.047108+0000","last_change":"2023-10-25T00:23:34.047108+0000","last_active":"2023-10-25T00:23:34.047108+0000","last_peered":"2023-10-25T00:23:34.047108+0000","last_clean":"2023-10-25T00:23:34.047108+0000","last_became_active":"2023-10-24T23:53:02.346161+0000","last_became_peered":"2023-10-24T23:53:02.346161+0000","last_unstale":"2023-10-25T00:23:34.047108+0000","last_undegraded":"2023-10-25T00:23:34.047108+0000","last_fullsized":"2023-10-25T00:23:34.047108+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2023-10-25T00:23:34.046917+0000","last_deep_scrub":"68'2","last_deep_scrub_stamp":"2023-10-25T00:23:34.046917+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.046917+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:50:34.912538+0000","scrub_duration":0.034858717999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"73'5","reported_seq":111,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.296722+0000","last_change":"2023-10-25T00:22:53.546332+0000","last_active":"2023-10-25T00:24:34.296722+0000","last_peered":"2023-10-25T00:24:34.296722+0000","last_clean":"2023-10-25T00:24:34.296722+0000","last_became_active":"2023-10-24T23:53:11.457753+0000","last_became_peered":"2023-10-24T23:53:11.457753+0000","last_unstale":"2023-10-25T00:24:34.296722+0000","last_undegraded":"2023-10-25T00:24:34.296722+0000","last_fullsized":"2023-10-25T00:24:34.296722+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:22:53.546215+0000","last_clean_scrub_stamp":"2023-10-25T00:22:53.546215+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:16:55.756192+0000","scrub_duration":0.27862564699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"72'4690","reported_seq":4777,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.446475+0000","last_change":"2023-10-25T00:23:42.556703+0000","last_active":"2023-10-25T00:24:34.446475+0000","last_peered":"2023-10-25T00:24:34.446475+0000","last_clean":"2023-10-25T00:24:34.446475+0000","last_became_active":"2023-10-24T23:53:11.815371+0000","last_became_peered":"2023-10-24T23:53:11.815371+0000","last_unstale":"2023-10-25T00:24:34.446475+0000","last_undegraded":"2023-10-25T00:24:34.446475+0000","last_fullsized":"2023-10-25T00:24:34.446475+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4690","last_scrub_stamp":"2023-10-25T00:23:42.556538+0000","last_deep_scrub":"72'4690","last_deep_scrub_stamp":"2023-10-25T00:23:42.556538+0000","last_clean_scrub_stamp":"2023-10-25T00:23:42.556538+0000","objects_scrubbed":0,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:36:54.839713+0000","scrub_duration":0.199820896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24994,"num_write_kb":341798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"68'4780","reported_seq":4870,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.286287+0000","last_change":"2023-10-25T00:23:09.093139+0000","last_active":"2023-10-25T00:24:34.286287+0000","last_peered":"2023-10-25T00:24:34.286287+0000","last_clean":"2023-10-25T00:24:34.286287+0000","last_became_active":"2023-10-24T23:53:02.717980+0000","last_became_peered":"2023-10-24T23:53:02.717980+0000","last_unstale":"2023-10-25T00:24:34.286287+0000","last_undegraded":"2023-10-25T00:24:34.286287+0000","last_fullsized":"2023-10-25T00:24:34.286287+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4780","last_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_deep_scrub":"68'4780","last_deep_scrub_stamp":"2023-10-25T00:23:09.092796+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.092796+0000","objects_scrubbed":0,"log_size":4780,"log_dups_size":0,"ondisk_log_size":4780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:58:49.347709+0000","scrub_duration":0.152359684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25083,"num_write_kb":293163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"68'3","reported_seq":91,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.678451+0000","last_change":"2023-10-25T00:22:48.328017+0000","last_active":"2023-10-25T00:23:30.678451+0000","last_peered":"2023-10-25T00:23:30.678451+0000","last_clean":"2023-10-25T00:23:30.678451+0000","last_became_active":"2023-10-24T23:53:02.346929+0000","last_became_peered":"2023-10-24T23:53:02.346929+0000","last_unstale":"2023-10-25T00:23:30.678451+0000","last_undegraded":"2023-10-25T00:23:30.678451+0000","last_fullsized":"2023-10-25T00:23:30.678451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:22:48.327778+0000","last_clean_scrub_stamp":"2023-10-25T00:22:48.327778+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:58.116941+0000","scrub_duration":0.061328572999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"72'5","reported_seq":89,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.323588+0000","last_change":"2023-10-25T00:22:55.576688+0000","last_active":"2023-10-25T00:24:35.323588+0000","last_peered":"2023-10-25T00:24:35.323588+0000","last_clean":"2023-10-25T00:24:35.323588+0000","last_became_active":"2023-10-24T23:53:11.448375+0000","last_became_peered":"2023-10-24T23:53:11.448375+0000","last_unstale":"2023-10-25T00:24:35.323588+0000","last_undegraded":"2023-10-25T00:24:35.323588+0000","last_fullsized":"2023-10-25T00:24:35.323588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:55.576324+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.576324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:49:43.752897+0000","scrub_duration":0.0045481469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"72'3596","reported_seq":3685,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:37.613637+0000","last_change":"2023-10-25T00:24:37.613637+0000","last_active":"2023-10-25T00:24:37.613637+0000","last_peered":"2023-10-25T00:24:37.613637+0000","last_clean":"2023-10-25T00:24:37.613637+0000","last_became_active":"2023-10-24T23:53:11.815292+0000","last_became_peered":"2023-10-24T23:53:11.815292+0000","last_unstale":"2023-10-25T00:24:37.613637+0000","last_undegraded":"2023-10-25T00:24:37.613637+0000","last_fullsized":"2023-10-25T00:24:37.613637+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3596","last_scrub_stamp":"2023-10-25T00:24:37.613422+0000","last_deep_scrub":"72'3596","last_deep_scrub_stamp":"2023-10-25T00:24:37.613422+0000","last_clean_scrub_stamp":"2023-10-25T00:24:37.613422+0000","objects_scrubbed":0,"log_size":3596,"log_dups_size":0,"ondisk_log_size":3596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:25:28.509580+0000","scrub_duration":0.359469288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":19820,"num_write_kb":259784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"68'4970","reported_seq":5059,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.288619+0000","last_change":"2023-10-25T00:22:55.999798+0000","last_active":"2023-10-25T00:24:34.288619+0000","last_peered":"2023-10-25T00:24:34.288619+0000","last_clean":"2023-10-25T00:24:34.288619+0000","last_became_active":"2023-10-24T23:53:02.723695+0000","last_became_peered":"2023-10-24T23:53:02.723695+0000","last_unstale":"2023-10-25T00:24:34.288619+0000","last_undegraded":"2023-10-25T00:24:34.288619+0000","last_fullsized":"2023-10-25T00:24:34.288619+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4970","last_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_deep_scrub":"68'4970","last_deep_scrub_stamp":"2023-10-25T00:22:55.999644+0000","last_clean_scrub_stamp":"2023-10-25T00:22:55.999644+0000","objects_scrubbed":0,"log_size":4970,"log_dups_size":0,"ondisk_log_size":4970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:15:47.681358+0000","scrub_duration":0.177095113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26721,"num_write_kb":311966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"73'1466","reported_seq":1609,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315010+0000","last_change":"2023-10-25T00:23:06.467052+0000","last_active":"2023-10-25T00:24:34.315010+0000","last_peered":"2023-10-25T00:24:34.315010+0000","last_clean":"2023-10-25T00:24:34.315010+0000","last_became_active":"2023-10-24T23:53:02.339074+0000","last_became_peered":"2023-10-24T23:53:02.339074+0000","last_unstale":"2023-10-25T00:24:34.315010+0000","last_undegraded":"2023-10-25T00:24:34.315010+0000","last_fullsized":"2023-10-25T00:24:34.315010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1466","last_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_deep_scrub":"73'1466","last_deep_scrub_stamp":"2023-10-25T00:23:06.466569+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.466569+0000","objects_scrubbed":2,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:21:21.696239+0000","scrub_duration":0.039946036999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":51896,"num_write":1439,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'7","reported_seq":114,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.446111+0000","last_change":"2023-10-25T00:23:53.616774+0000","last_active":"2023-10-25T00:24:34.446111+0000","last_peered":"2023-10-25T00:24:34.446111+0000","last_clean":"2023-10-25T00:24:34.446111+0000","last_became_active":"2023-10-24T23:53:11.445926+0000","last_became_peered":"2023-10-24T23:53:11.445926+0000","last_unstale":"2023-10-25T00:24:34.446111+0000","last_undegraded":"2023-10-25T00:24:34.446111+0000","last_fullsized":"2023-10-25T00:24:34.446111+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'7","last_scrub_stamp":"2023-10-25T00:23:53.616451+0000","last_deep_scrub":"73'7","last_deep_scrub_stamp":"2023-10-25T00:23:53.616451+0000","last_clean_scrub_stamp":"2023-10-25T00:23:53.616451+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:31:57.298978+0000","scrub_duration":0.13500319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"72'3548","reported_seq":3635,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.496463+0000","last_change":"2023-10-25T00:23:32.093782+0000","last_active":"2023-10-25T00:23:36.496463+0000","last_peered":"2023-10-25T00:23:36.496463+0000","last_clean":"2023-10-25T00:23:36.496463+0000","last_became_active":"2023-10-24T23:53:11.846778+0000","last_became_peered":"2023-10-24T23:53:11.846778+0000","last_unstale":"2023-10-25T00:23:36.496463+0000","last_undegraded":"2023-10-25T00:23:36.496463+0000","last_fullsized":"2023-10-25T00:23:36.496463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3548","last_scrub_stamp":"2023-10-25T00:23:32.093615+0000","last_deep_scrub":"72'3548","last_deep_scrub_stamp":"2023-10-25T00:23:32.093615+0000","last_clean_scrub_stamp":"2023-10-25T00:23:32.093615+0000","objects_scrubbed":0,"log_size":3548,"log_dups_size":0,"ondisk_log_size":3548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:02:03.851334+0000","scrub_duration":0.28610911100000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20243,"num_write_kb":261539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"68'3393","reported_seq":3484,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315870+0000","last_change":"2023-10-25T00:22:42.098602+0000","last_active":"2023-10-25T00:24:34.315870+0000","last_peered":"2023-10-25T00:24:34.315870+0000","last_clean":"2023-10-25T00:24:34.315870+0000","last_became_active":"2023-10-24T23:53:02.726897+0000","last_became_peered":"2023-10-24T23:53:02.726897+0000","last_unstale":"2023-10-25T00:24:34.315870+0000","last_undegraded":"2023-10-25T00:24:34.315870+0000","last_fullsized":"2023-10-25T00:24:34.315870+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3393","last_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_deep_scrub":"68'3393","last_deep_scrub_stamp":"2023-10-25T00:22:42.098371+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.098371+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:12.736136+0000","scrub_duration":0.153676797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17500,"num_write_kb":217638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"68'5","reported_seq":93,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.297563+0000","last_change":"2023-10-25T00:23:33.402936+0000","last_active":"2023-10-25T00:24:34.297563+0000","last_peered":"2023-10-25T00:24:34.297563+0000","last_clean":"2023-10-25T00:24:34.297563+0000","last_became_active":"2023-10-24T23:53:02.335918+0000","last_became_peered":"2023-10-24T23:53:02.335918+0000","last_unstale":"2023-10-25T00:24:34.297563+0000","last_undegraded":"2023-10-25T00:24:34.297563+0000","last_fullsized":"2023-10-25T00:24:34.297563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:33.402617+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:33.402617+0000","last_clean_scrub_stamp":"2023-10-25T00:23:33.402617+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:10:14.715542+0000","scrub_duration":0.0048941419999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"72'3","reported_seq":88,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.419862+0000","last_change":"2023-10-25T00:23:45.419862+0000","last_active":"2023-10-25T00:23:45.419862+0000","last_peered":"2023-10-25T00:23:45.419862+0000","last_clean":"2023-10-25T00:23:45.419862+0000","last_became_active":"2023-10-24T23:53:11.454335+0000","last_became_peered":"2023-10-24T23:53:11.454335+0000","last_unstale":"2023-10-25T00:23:45.419862+0000","last_undegraded":"2023-10-25T00:23:45.419862+0000","last_fullsized":"2023-10-25T00:23:45.419862+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:23:45.419528+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:23:45.419528+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.419528+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:07:59.973530+0000","scrub_duration":0.0045258490000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"72'3584","reported_seq":3671,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:40.285625+0000","last_change":"2023-10-25T00:23:40.285625+0000","last_active":"2023-10-25T00:23:40.285625+0000","last_peered":"2023-10-25T00:23:40.285625+0000","last_clean":"2023-10-25T00:23:40.285625+0000","last_became_active":"2023-10-24T23:53:11.859778+0000","last_became_peered":"2023-10-24T23:53:11.859778+0000","last_unstale":"2023-10-25T00:23:40.285625+0000","last_undegraded":"2023-10-25T00:23:40.285625+0000","last_fullsized":"2023-10-25T00:23:40.285625+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3584","last_scrub_stamp":"2023-10-25T00:23:40.285478+0000","last_deep_scrub":"72'3584","last_deep_scrub_stamp":"2023-10-25T00:23:40.285478+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.285478+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:11:39.447639+0000","scrub_duration":0.26816689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20504,"num_write_kb":254394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'5094","reported_seq":5187,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:30.678409+0000","last_change":"2023-10-25T00:22:51.485282+0000","last_active":"2023-10-25T00:23:30.678409+0000","last_peered":"2023-10-25T00:23:30.678409+0000","last_clean":"2023-10-25T00:23:30.678409+0000","last_became_active":"2023-10-24T23:53:02.719500+0000","last_became_peered":"2023-10-24T23:53:02.719500+0000","last_unstale":"2023-10-25T00:23:30.678409+0000","last_undegraded":"2023-10-25T00:23:30.678409+0000","last_fullsized":"2023-10-25T00:23:30.678409+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5094","last_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_deep_scrub":"68'5094","last_deep_scrub_stamp":"2023-10-25T00:22:51.484917+0000","last_clean_scrub_stamp":"2023-10-25T00:22:51.484917+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:22:43.899134+0000","scrub_duration":0.19299439099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":27763,"num_write_kb":328490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"73'7","reported_seq":119,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.317526+0000","last_change":"2023-10-25T00:23:27.095106+0000","last_active":"2023-10-25T00:24:34.317526+0000","last_peered":"2023-10-25T00:24:34.317526+0000","last_clean":"2023-10-25T00:24:34.317526+0000","last_became_active":"2023-10-24T23:53:02.353717+0000","last_became_peered":"2023-10-24T23:53:02.353717+0000","last_unstale":"2023-10-25T00:24:34.317526+0000","last_undegraded":"2023-10-25T00:24:34.317526+0000","last_fullsized":"2023-10-25T00:24:34.317526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'7","last_scrub_stamp":"2023-10-25T00:23:27.094773+0000","last_deep_scrub":"73'7","last_deep_scrub_stamp":"2023-10-25T00:23:27.094773+0000","last_clean_scrub_stamp":"2023-10-25T00:23:27.094773+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:52:00.838368+0000","scrub_duration":0.007318275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"73'1440","reported_seq":1572,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.446248+0000","last_change":"2023-10-25T00:23:16.527658+0000","last_active":"2023-10-25T00:24:34.446248+0000","last_peered":"2023-10-25T00:24:34.446248+0000","last_clean":"2023-10-25T00:24:34.446248+0000","last_became_active":"2023-10-24T23:53:11.423120+0000","last_became_peered":"2023-10-24T23:53:11.423120+0000","last_unstale":"2023-10-25T00:24:34.446248+0000","last_undegraded":"2023-10-25T00:24:34.446248+0000","last_fullsized":"2023-10-25T00:24:34.446248+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1440","last_scrub_stamp":"2023-10-25T00:23:16.527441+0000","last_deep_scrub":"73'1440","last_deep_scrub_stamp":"2023-10-25T00:23:16.527441+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.527441+0000","objects_scrubbed":2,"log_size":1440,"log_dups_size":0,"ondisk_log_size":1440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:47:53.223550+0000","scrub_duration":0.234060301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":35275,"num_write":1412,"num_write_kb":4487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"72'3292","reported_seq":3380,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.314595+0000","last_change":"2023-10-25T00:23:30.971878+0000","last_active":"2023-10-25T00:24:34.314595+0000","last_peered":"2023-10-25T00:24:34.314595+0000","last_clean":"2023-10-25T00:24:34.314595+0000","last_became_active":"2023-10-24T23:53:11.846904+0000","last_became_peered":"2023-10-24T23:53:11.846904+0000","last_unstale":"2023-10-25T00:24:34.314595+0000","last_undegraded":"2023-10-25T00:24:34.314595+0000","last_fullsized":"2023-10-25T00:24:34.314595+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3292","last_scrub_stamp":"2023-10-25T00:23:30.971718+0000","last_deep_scrub":"72'3292","last_deep_scrub_stamp":"2023-10-25T00:23:30.971718+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.971718+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:45:07.787712+0000","scrub_duration":0.347700487,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17853,"num_write_kb":243602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"67'4242","reported_seq":4330,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.307868+0000","last_change":"2023-10-25T00:22:45.893397+0000","last_active":"2023-10-25T00:23:26.307868+0000","last_peered":"2023-10-25T00:23:26.307868+0000","last_clean":"2023-10-25T00:23:26.307868+0000","last_became_active":"2023-10-24T23:53:02.717530+0000","last_became_peered":"2023-10-24T23:53:02.717530+0000","last_unstale":"2023-10-25T00:23:26.307868+0000","last_undegraded":"2023-10-25T00:23:26.307868+0000","last_fullsized":"2023-10-25T00:23:26.307868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4242","last_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_deep_scrub":"67'4242","last_deep_scrub_stamp":"2023-10-25T00:22:45.893150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.893150+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:24:28.749675+0000","scrub_duration":0.21189509300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22687,"num_write_kb":275346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"73'8","reported_seq":144,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:38.464871+0000","last_change":"2023-10-25T00:23:30.063493+0000","last_active":"2023-10-25T00:24:38.464871+0000","last_peered":"2023-10-25T00:24:38.464871+0000","last_clean":"2023-10-25T00:24:38.464871+0000","last_became_active":"2023-10-24T23:53:02.358948+0000","last_became_peered":"2023-10-24T23:53:02.358948+0000","last_unstale":"2023-10-25T00:24:38.464871+0000","last_undegraded":"2023-10-25T00:24:38.464871+0000","last_fullsized":"2023-10-25T00:24:38.464871+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2023-10-25T00:23:30.063132+0000","last_deep_scrub":"73'8","last_deep_scrub_stamp":"2023-10-25T00:23:30.063132+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.063132+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:12:23.891399+0000","scrub_duration":0.0076496660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"72'5","reported_seq":89,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.820644+0000","last_change":"2023-10-25T00:23:19.320206+0000","last_active":"2023-10-25T00:23:35.820644+0000","last_peered":"2023-10-25T00:23:35.820644+0000","last_clean":"2023-10-25T00:23:35.820644+0000","last_became_active":"2023-10-24T23:53:11.422991+0000","last_became_peered":"2023-10-24T23:53:11.422991+0000","last_unstale":"2023-10-25T00:23:35.820644+0000","last_undegraded":"2023-10-25T00:23:35.820644+0000","last_fullsized":"2023-10-25T00:23:35.820644+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:23:19.319840+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:23:19.319840+0000","last_clean_scrub_stamp":"2023-10-25T00:23:19.319840+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:18:41.713579+0000","scrub_duration":0.0042872409999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"72'4091","reported_seq":4178,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:50.884393+0000","last_change":"2023-10-25T00:23:50.884393+0000","last_active":"2023-10-25T00:23:50.884393+0000","last_peered":"2023-10-25T00:23:50.884393+0000","last_clean":"2023-10-25T00:23:50.884393+0000","last_became_active":"2023-10-24T23:53:11.828869+0000","last_became_peered":"2023-10-24T23:53:11.828869+0000","last_unstale":"2023-10-25T00:23:50.884393+0000","last_undegraded":"2023-10-25T00:23:50.884393+0000","last_fullsized":"2023-10-25T00:23:50.884393+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4091","last_scrub_stamp":"2023-10-25T00:23:50.884251+0000","last_deep_scrub":"72'4091","last_deep_scrub_stamp":"2023-10-25T00:23:50.884251+0000","last_clean_scrub_stamp":"2023-10-25T00:23:50.884251+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:21:51.075072+0000","scrub_duration":0.36544204200000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22779,"num_write_kb":288098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"67'4310","reported_seq":4398,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:25.928208+0000","last_change":"2023-10-25T00:23:22.394323+0000","last_active":"2023-10-25T00:23:25.928208+0000","last_peered":"2023-10-25T00:23:25.928208+0000","last_clean":"2023-10-25T00:23:25.928208+0000","last_became_active":"2023-10-24T23:53:02.719131+0000","last_became_peered":"2023-10-24T23:53:02.719131+0000","last_unstale":"2023-10-25T00:23:25.928208+0000","last_undegraded":"2023-10-25T00:23:25.928208+0000","last_fullsized":"2023-10-25T00:23:25.928208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'4310","last_scrub_stamp":"2023-10-25T00:23:22.393940+0000","last_deep_scrub":"67'4310","last_deep_scrub_stamp":"2023-10-25T00:23:22.393940+0000","last_clean_scrub_stamp":"2023-10-25T00:23:22.393940+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:24:46.673127+0000","scrub_duration":0.085301248999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23791,"num_write_kb":283914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"73'5","reported_seq":117,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.317432+0000","last_change":"2023-10-25T00:23:26.183662+0000","last_active":"2023-10-25T00:24:34.317432+0000","last_peered":"2023-10-25T00:24:34.317432+0000","last_clean":"2023-10-25T00:24:34.317432+0000","last_became_active":"2023-10-24T23:53:02.357412+0000","last_became_peered":"2023-10-24T23:53:02.357412+0000","last_unstale":"2023-10-25T00:24:34.317432+0000","last_undegraded":"2023-10-25T00:24:34.317432+0000","last_fullsized":"2023-10-25T00:24:34.317432+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2023-10-25T00:23:26.183430+0000","last_deep_scrub":"73'5","last_deep_scrub_stamp":"2023-10-25T00:23:26.183430+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.183430+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:28:22.710805+0000","scrub_duration":0.110908062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"72'2","reported_seq":86,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:36.273901+0000","last_change":"2023-10-25T00:22:38.034628+0000","last_active":"2023-10-25T00:23:36.273901+0000","last_peered":"2023-10-25T00:23:36.273901+0000","last_clean":"2023-10-25T00:23:36.273901+0000","last_became_active":"2023-10-24T23:53:11.450345+0000","last_became_peered":"2023-10-24T23:53:11.450345+0000","last_unstale":"2023-10-25T00:23:36.273901+0000","last_undegraded":"2023-10-25T00:23:36.273901+0000","last_fullsized":"2023-10-25T00:23:36.273901+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'2","last_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_deep_scrub":"72'2","last_deep_scrub_stamp":"2023-10-25T00:22:38.034499+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.034499+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:09.450121+0000","scrub_duration":0.14675892500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"72'3695","reported_seq":3783,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315305+0000","last_change":"2023-10-25T00:23:51.251559+0000","last_active":"2023-10-25T00:24:34.315305+0000","last_peered":"2023-10-25T00:24:34.315305+0000","last_clean":"2023-10-25T00:24:34.315305+0000","last_became_active":"2023-10-24T23:53:11.858241+0000","last_became_peered":"2023-10-24T23:53:11.858241+0000","last_unstale":"2023-10-25T00:24:34.315305+0000","last_undegraded":"2023-10-25T00:24:34.315305+0000","last_fullsized":"2023-10-25T00:24:34.315305+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3695","last_scrub_stamp":"2023-10-25T00:23:51.251416+0000","last_deep_scrub":"72'3695","last_deep_scrub_stamp":"2023-10-25T00:23:51.251416+0000","last_clean_scrub_stamp":"2023-10-25T00:23:51.251416+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:44:06.748969+0000","scrub_duration":0.274463921,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":21426,"num_write_kb":254037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"72'3","reported_seq":86,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.297938+0000","last_change":"2023-10-25T00:22:42.331112+0000","last_active":"2023-10-25T00:24:34.297938+0000","last_peered":"2023-10-25T00:24:34.297938+0000","last_clean":"2023-10-25T00:24:34.297938+0000","last_became_active":"2023-10-24T23:53:11.433844+0000","last_became_peered":"2023-10-24T23:53:11.433844+0000","last_unstale":"2023-10-25T00:24:34.297938+0000","last_undegraded":"2023-10-25T00:24:34.297938+0000","last_fullsized":"2023-10-25T00:24:34.297938+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3","last_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_deep_scrub":"72'3","last_deep_scrub_stamp":"2023-10-25T00:22:42.330786+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.330786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:23:56.573805+0000","scrub_duration":0.0043674530000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"72'19","reported_seq":223,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:38.673260+0000","last_change":"2023-10-25T00:22:47.253320+0000","last_active":"2023-10-25T00:24:38.673260+0000","last_peered":"2023-10-25T00:24:38.673260+0000","last_clean":"2023-10-25T00:24:38.673260+0000","last_became_active":"2023-10-24T23:53:02.336952+0000","last_became_peered":"2023-10-24T23:53:02.336952+0000","last_unstale":"2023-10-25T00:24:38.673260+0000","last_undegraded":"2023-10-25T00:24:38.673260+0000","last_fullsized":"2023-10-25T00:24:38.673260+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_deep_scrub":"72'19","last_deep_scrub_stamp":"2023-10-25T00:22:47.252995+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.252995+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:07:03.720456+0000","scrub_duration":0.018599613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":351,"num_read_kb":351,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'5249","reported_seq":5344,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.570074+0000","last_change":"2023-10-25T00:22:52.439461+0000","last_active":"2023-10-25T00:24:32.570074+0000","last_peered":"2023-10-25T00:24:32.570074+0000","last_clean":"2023-10-25T00:24:32.570074+0000","last_became_active":"2023-10-24T23:53:02.713155+0000","last_became_peered":"2023-10-24T23:53:02.713155+0000","last_unstale":"2023-10-25T00:24:32.570074+0000","last_undegraded":"2023-10-25T00:24:32.570074+0000","last_fullsized":"2023-10-25T00:24:32.570074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5249","last_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_deep_scrub":"68'5249","last_deep_scrub_stamp":"2023-10-25T00:22:52.439150+0000","last_clean_scrub_stamp":"2023-10-25T00:22:52.439150+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:19:03.730037+0000","scrub_duration":0.077069795999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":0,"num_write":27646,"num_write_kb":332285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"72'4380","reported_seq":4465,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:38.646626+0000","last_change":"2023-10-25T00:23:38.646626+0000","last_active":"2023-10-25T00:23:38.646626+0000","last_peered":"2023-10-25T00:23:38.646626+0000","last_clean":"2023-10-25T00:23:38.646626+0000","last_became_active":"2023-10-24T23:53:11.845155+0000","last_became_peered":"2023-10-24T23:53:11.845155+0000","last_unstale":"2023-10-25T00:23:38.646626+0000","last_undegraded":"2023-10-25T00:23:38.646626+0000","last_fullsized":"2023-10-25T00:23:38.646626+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4380","last_scrub_stamp":"2023-10-25T00:23:38.646472+0000","last_deep_scrub":"72'4380","last_deep_scrub_stamp":"2023-10-25T00:23:38.646472+0000","last_clean_scrub_stamp":"2023-10-25T00:23:38.646472+0000","objects_scrubbed":0,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:38:56.172654+0000","scrub_duration":0.29662724200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23662,"num_write_kb":324338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"72'5","reported_seq":88,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.313629+0000","last_change":"2023-10-25T00:22:50.618067+0000","last_active":"2023-10-25T00:24:34.313629+0000","last_peered":"2023-10-25T00:24:34.313629+0000","last_clean":"2023-10-25T00:24:34.313629+0000","last_became_active":"2023-10-24T23:53:11.433043+0000","last_became_peered":"2023-10-24T23:53:11.433043+0000","last_unstale":"2023-10-25T00:24:34.313629+0000","last_undegraded":"2023-10-25T00:24:34.313629+0000","last_fullsized":"2023-10-25T00:24:34.313629+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_deep_scrub":"72'5","last_deep_scrub_stamp":"2023-10-25T00:22:50.617931+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.617931+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:02:24.381368+0000","scrub_duration":0.234626104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"68'5","reported_seq":92,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.271669+0000","last_change":"2023-10-25T00:23:25.067489+0000","last_active":"2023-10-25T00:23:26.271669+0000","last_peered":"2023-10-25T00:23:26.271669+0000","last_clean":"2023-10-25T00:23:26.271669+0000","last_became_active":"2023-10-24T23:53:02.357715+0000","last_became_peered":"2023-10-24T23:53:02.357715+0000","last_unstale":"2023-10-25T00:23:26.271669+0000","last_undegraded":"2023-10-25T00:23:26.271669+0000","last_fullsized":"2023-10-25T00:23:26.271669+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5","last_scrub_stamp":"2023-10-25T00:23:25.067146+0000","last_deep_scrub":"68'5","last_deep_scrub_stamp":"2023-10-25T00:23:25.067146+0000","last_clean_scrub_stamp":"2023-10-25T00:23:25.067146+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:46:32.009959+0000","scrub_duration":0.0041696520000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'4442","reported_seq":4529,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.509925+0000","last_change":"2023-10-25T00:23:29.981423+0000","last_active":"2023-10-25T00:24:31.509925+0000","last_peered":"2023-10-25T00:24:31.509925+0000","last_clean":"2023-10-25T00:24:31.509925+0000","last_became_active":"2023-10-24T23:53:02.726573+0000","last_became_peered":"2023-10-24T23:53:02.726573+0000","last_unstale":"2023-10-25T00:24:31.509925+0000","last_undegraded":"2023-10-25T00:24:31.509925+0000","last_fullsized":"2023-10-25T00:24:31.509925+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4442","last_scrub_stamp":"2023-10-25T00:23:29.981245+0000","last_deep_scrub":"68'4442","last_deep_scrub_stamp":"2023-10-25T00:23:29.981245+0000","last_clean_scrub_stamp":"2023-10-25T00:23:29.981245+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:11:38.194434+0000","scrub_duration":0.109448791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22810,"num_write_kb":296225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"72'4076","reported_seq":4163,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.292048+0000","last_change":"2023-10-25T00:24:07.280653+0000","last_active":"2023-10-25T00:24:34.292048+0000","last_peered":"2023-10-25T00:24:34.292048+0000","last_clean":"2023-10-25T00:24:34.292048+0000","last_became_active":"2023-10-24T23:53:11.833300+0000","last_became_peered":"2023-10-24T23:53:11.833300+0000","last_unstale":"2023-10-25T00:24:34.292048+0000","last_undegraded":"2023-10-25T00:24:34.292048+0000","last_fullsized":"2023-10-25T00:24:34.292048+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4076","last_scrub_stamp":"2023-10-25T00:24:07.280471+0000","last_deep_scrub":"72'4076","last_deep_scrub_stamp":"2023-10-25T00:24:07.280471+0000","last_clean_scrub_stamp":"2023-10-25T00:24:07.280471+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:36:18.063026+0000","scrub_duration":0.198538093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22055,"num_write_kb":297609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"68'3","reported_seq":90,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.494095+0000","last_change":"2023-10-25T00:23:23.928686+0000","last_active":"2023-10-25T00:23:26.494095+0000","last_peered":"2023-10-25T00:23:26.494095+0000","last_clean":"2023-10-25T00:23:26.494095+0000","last_became_active":"2023-10-24T23:53:02.354470+0000","last_became_peered":"2023-10-24T23:53:02.354470+0000","last_unstale":"2023-10-25T00:23:26.494095+0000","last_undegraded":"2023-10-25T00:23:26.494095+0000","last_fullsized":"2023-10-25T00:23:26.494095+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3","last_scrub_stamp":"2023-10-25T00:23:23.928352+0000","last_deep_scrub":"68'3","last_deep_scrub_stamp":"2023-10-25T00:23:23.928352+0000","last_clean_scrub_stamp":"2023-10-25T00:23:23.928352+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:41:04.304281+0000","scrub_duration":0.0043703589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'4101","reported_seq":4189,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.110625+0000","last_change":"2023-10-25T00:22:50.337815+0000","last_active":"2023-10-25T00:24:31.110625+0000","last_peered":"2023-10-25T00:24:31.110625+0000","last_clean":"2023-10-25T00:24:31.110625+0000","last_became_active":"2023-10-24T23:53:02.718552+0000","last_became_peered":"2023-10-24T23:53:02.718552+0000","last_unstale":"2023-10-25T00:24:31.110625+0000","last_undegraded":"2023-10-25T00:24:31.110625+0000","last_fullsized":"2023-10-25T00:24:31.110625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4101","last_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_deep_scrub":"68'4101","last_deep_scrub_stamp":"2023-10-25T00:22:50.337536+0000","last_clean_scrub_stamp":"2023-10-25T00:22:50.337536+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:20:01.275651+0000","scrub_duration":0.055303021000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22343,"num_write_kb":253486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.14","version":"72'397","reported_seq":592,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.542939+0000","last_change":"2023-10-25T00:22:45.554898+0000","last_active":"2023-10-25T00:23:37.542939+0000","last_peered":"2023-10-25T00:23:37.542939+0000","last_clean":"2023-10-25T00:23:37.542939+0000","last_became_active":"2023-10-24T23:53:11.446635+0000","last_became_peered":"2023-10-24T23:53:11.446635+0000","last_unstale":"2023-10-25T00:23:37.542939+0000","last_undegraded":"2023-10-25T00:23:37.542939+0000","last_fullsized":"2023-10-25T00:23:37.542939+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'397","last_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_deep_scrub":"72'397","last_deep_scrub_stamp":"2023-10-25T00:22:45.554526+0000","last_clean_scrub_stamp":"2023-10-25T00:22:45.554526+0000","objects_scrubbed":4,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:41:32.056818+0000","scrub_duration":0.10248053,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":200,"num_read_kb":86197,"num_write":389,"num_write_kb":4306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.16","version":"72'4606","reported_seq":4691,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315589+0000","last_change":"2023-10-25T00:23:37.057459+0000","last_active":"2023-10-25T00:24:34.315589+0000","last_peered":"2023-10-25T00:24:34.315589+0000","last_clean":"2023-10-25T00:24:34.315589+0000","last_became_active":"2023-10-24T23:53:11.846784+0000","last_became_peered":"2023-10-24T23:53:11.846784+0000","last_unstale":"2023-10-25T00:24:34.315589+0000","last_undegraded":"2023-10-25T00:24:34.315589+0000","last_fullsized":"2023-10-25T00:24:34.315589+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4606","last_scrub_stamp":"2023-10-25T00:23:37.057332+0000","last_deep_scrub":"72'4606","last_deep_scrub_stamp":"2023-10-25T00:23:37.057332+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.057332+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:16:39.313888+0000","scrub_duration":0.28055820399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25763,"num_write_kb":316477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"74'15","reported_seq":150,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:35.686811+0000","last_change":"2023-10-25T00:23:16.153043+0000","last_active":"2023-10-25T00:24:35.686811+0000","last_peered":"2023-10-25T00:24:35.686811+0000","last_clean":"2023-10-25T00:24:35.686811+0000","last_became_active":"2023-10-24T23:53:02.358674+0000","last_became_peered":"2023-10-24T23:53:02.358674+0000","last_unstale":"2023-10-25T00:24:35.686811+0000","last_undegraded":"2023-10-25T00:24:35.686811+0000","last_fullsized":"2023-10-25T00:24:35.686811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_deep_scrub":"73'14","last_deep_scrub_stamp":"2023-10-25T00:23:16.152723+0000","last_clean_scrub_stamp":"2023-10-25T00:23:16.152723+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:32:30.128331+0000","scrub_duration":0.010186763999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":192,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'5680","reported_seq":5771,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:27.540112+0000","last_change":"2023-10-25T00:22:54.586176+0000","last_active":"2023-10-25T00:23:27.540112+0000","last_peered":"2023-10-25T00:23:27.540112+0000","last_clean":"2023-10-25T00:23:27.540112+0000","last_became_active":"2023-10-24T23:53:02.721314+0000","last_became_peered":"2023-10-24T23:53:02.721314+0000","last_unstale":"2023-10-25T00:23:27.540112+0000","last_undegraded":"2023-10-25T00:23:27.540112+0000","last_fullsized":"2023-10-25T00:23:27.540112+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5680","last_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_deep_scrub":"68'5680","last_deep_scrub_stamp":"2023-10-25T00:22:54.585866+0000","last_clean_scrub_stamp":"2023-10-25T00:22:54.585866+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:19:49.342364+0000","scrub_duration":0.20835251299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29749,"num_write_kb":360962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"73'40","reported_seq":189,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.298329+0000","last_change":"2023-10-25T00:22:59.279527+0000","last_active":"2023-10-25T00:24:34.298329+0000","last_peered":"2023-10-25T00:24:34.298329+0000","last_clean":"2023-10-25T00:24:34.298329+0000","last_became_active":"2023-10-24T23:53:11.435809+0000","last_became_peered":"2023-10-24T23:53:11.435809+0000","last_unstale":"2023-10-25T00:24:34.298329+0000","last_undegraded":"2023-10-25T00:24:34.298329+0000","last_fullsized":"2023-10-25T00:24:34.298329+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'40","last_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_deep_scrub":"73'40","last_deep_scrub_stamp":"2023-10-25T00:22:59.279288+0000","last_clean_scrub_stamp":"2023-10-25T00:22:59.279288+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:29:52.911132+0000","scrub_duration":0.0094949800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":39,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.22","version":"72'4808","reported_seq":4899,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:30.690247+0000","last_change":"2023-10-25T00:23:14.791099+0000","last_active":"2023-10-25T00:24:30.690247+0000","last_peered":"2023-10-25T00:24:30.690247+0000","last_clean":"2023-10-25T00:24:30.690247+0000","last_became_active":"2023-10-24T23:53:11.813413+0000","last_became_peered":"2023-10-24T23:53:11.813413+0000","last_unstale":"2023-10-25T00:24:30.690247+0000","last_undegraded":"2023-10-25T00:24:30.690247+0000","last_fullsized":"2023-10-25T00:24:30.690247+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4808","last_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_deep_scrub":"72'4808","last_deep_scrub_stamp":"2023-10-25T00:23:14.790929+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.790929+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:56:36.982903+0000","scrub_duration":0.29119922999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":26552,"num_write_kb":336125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.20","version":"68'4747","reported_seq":4837,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:20.688044+0000","last_change":"2023-10-25T00:23:00.496309+0000","last_active":"2023-10-25T00:24:20.688044+0000","last_peered":"2023-10-25T00:24:20.688044+0000","last_clean":"2023-10-25T00:24:20.688044+0000","last_became_active":"2023-10-24T23:53:02.703342+0000","last_became_peered":"2023-10-24T23:53:02.703342+0000","last_unstale":"2023-10-25T00:24:20.688044+0000","last_undegraded":"2023-10-25T00:24:20.688044+0000","last_fullsized":"2023-10-25T00:24:20.688044+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4747","last_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_deep_scrub":"68'4747","last_deep_scrub_stamp":"2023-10-25T00:23:00.496060+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.496060+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:38:56.531125+0000","scrub_duration":0.089432630999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24847,"num_write_kb":316685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.23","version":"72'4283","reported_seq":4372,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.286177+0000","last_change":"2023-10-25T00:23:08.728718+0000","last_active":"2023-10-25T00:24:34.286177+0000","last_peered":"2023-10-25T00:24:34.286177+0000","last_clean":"2023-10-25T00:24:34.286177+0000","last_became_active":"2023-10-24T23:53:11.804529+0000","last_became_peered":"2023-10-24T23:53:11.804529+0000","last_unstale":"2023-10-25T00:24:34.286177+0000","last_undegraded":"2023-10-25T00:24:34.286177+0000","last_fullsized":"2023-10-25T00:24:34.286177+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4283","last_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_deep_scrub":"72'4283","last_deep_scrub_stamp":"2023-10-25T00:23:08.728567+0000","last_clean_scrub_stamp":"2023-10-25T00:23:08.728567+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:59:33.655445+0000","scrub_duration":0.26846236600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23853,"num_write_kb":296504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"68'4036","reported_seq":4126,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:22.567176+0000","last_change":"2023-10-25T00:22:39.642537+0000","last_active":"2023-10-25T00:24:22.567176+0000","last_peered":"2023-10-25T00:24:22.567176+0000","last_clean":"2023-10-25T00:24:22.567176+0000","last_became_active":"2023-10-24T23:53:02.719539+0000","last_became_peered":"2023-10-24T23:53:02.719539+0000","last_unstale":"2023-10-25T00:24:22.567176+0000","last_undegraded":"2023-10-25T00:24:22.567176+0000","last_fullsized":"2023-10-25T00:24:22.567176+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4036","last_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_deep_scrub":"68'4036","last_deep_scrub_stamp":"2023-10-25T00:22:39.642342+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.642342+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:20:02.362976+0000","scrub_duration":0.227585228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21233,"num_write_kb":273467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.20","version":"72'4276","reported_seq":4364,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.341479+0000","last_change":"2023-10-25T00:23:30.997946+0000","last_active":"2023-10-25T00:24:31.341479+0000","last_peered":"2023-10-25T00:24:31.341479+0000","last_clean":"2023-10-25T00:24:31.341479+0000","last_became_active":"2023-10-24T23:53:11.805465+0000","last_became_peered":"2023-10-24T23:53:11.805465+0000","last_unstale":"2023-10-25T00:24:31.341479+0000","last_undegraded":"2023-10-25T00:24:31.341479+0000","last_fullsized":"2023-10-25T00:24:31.341479+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4276","last_scrub_stamp":"2023-10-25T00:23:30.997816+0000","last_deep_scrub":"72'4276","last_deep_scrub_stamp":"2023-10-25T00:23:30.997816+0000","last_clean_scrub_stamp":"2023-10-25T00:23:30.997816+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:27:08.863577+0000","scrub_duration":0.39360271099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23362,"num_write_kb":310817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.22","version":"68'6201","reported_seq":6294,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.450739+0000","last_change":"2023-10-25T00:24:34.450866+0000","last_active":"2023-10-25T00:24:34.450739+0000","last_peered":"2023-10-25T00:24:34.450739+0000","last_clean":"2023-10-25T00:24:34.450739+0000","last_became_active":"2023-10-24T23:53:02.717290+0000","last_became_peered":"2023-10-24T23:53:02.717290+0000","last_unstale":"2023-10-25T00:24:34.450739+0000","last_undegraded":"2023-10-25T00:24:34.450739+0000","last_fullsized":"2023-10-25T00:24:34.450739+0000","mapping_epoch":21,"log_start":"62'1000","ondisk_log_start":"62'1000","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6201","last_scrub_stamp":"2023-10-25T00:24:34.450718+0000","last_deep_scrub":"68'6201","last_deep_scrub_stamp":"2023-10-25T00:24:34.450718+0000","last_clean_scrub_stamp":"2023-10-25T00:24:34.450718+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:58:02.692663+0000","scrub_duration":0.18173869100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":33486,"num_write_kb":384441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"72'4018","reported_seq":4107,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.287337+0000","last_change":"2023-10-25T00:23:52.269793+0000","last_active":"2023-10-25T00:24:31.287337+0000","last_peered":"2023-10-25T00:24:31.287337+0000","last_clean":"2023-10-25T00:24:31.287337+0000","last_became_active":"2023-10-24T23:53:11.845200+0000","last_became_peered":"2023-10-24T23:53:11.845200+0000","last_unstale":"2023-10-25T00:24:31.287337+0000","last_undegraded":"2023-10-25T00:24:31.287337+0000","last_fullsized":"2023-10-25T00:24:31.287337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4018","last_scrub_stamp":"2023-10-25T00:23:52.269641+0000","last_deep_scrub":"72'4018","last_deep_scrub_stamp":"2023-10-25T00:23:52.269641+0000","last_clean_scrub_stamp":"2023-10-25T00:23:52.269641+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:58:28.387402+0000","scrub_duration":0.26794800299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":21920,"num_write_kb":288651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.23","version":"68'4522","reported_seq":4613,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:22.567304+0000","last_change":"2023-10-25T00:23:26.751600+0000","last_active":"2023-10-25T00:24:22.567304+0000","last_peered":"2023-10-25T00:24:22.567304+0000","last_clean":"2023-10-25T00:24:22.567304+0000","last_became_active":"2023-10-24T23:53:02.713315+0000","last_became_peered":"2023-10-24T23:53:02.713315+0000","last_unstale":"2023-10-25T00:24:22.567304+0000","last_undegraded":"2023-10-25T00:24:22.567304+0000","last_fullsized":"2023-10-25T00:24:22.567304+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4522","last_scrub_stamp":"2023-10-25T00:23:26.751435+0000","last_deep_scrub":"68'4522","last_deep_scrub_stamp":"2023-10-25T00:23:26.751435+0000","last_clean_scrub_stamp":"2023-10-25T00:23:26.751435+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:40:36.338773+0000","scrub_duration":0.110684227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":24476,"num_write_kb":282500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.26","version":"72'4209","reported_seq":4294,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.054692+0000","last_change":"2023-10-25T00:23:37.054692+0000","last_active":"2023-10-25T00:23:37.054692+0000","last_peered":"2023-10-25T00:23:37.054692+0000","last_clean":"2023-10-25T00:23:37.054692+0000","last_became_active":"2023-10-24T23:53:11.805597+0000","last_became_peered":"2023-10-24T23:53:11.805597+0000","last_unstale":"2023-10-25T00:23:37.054692+0000","last_undegraded":"2023-10-25T00:23:37.054692+0000","last_fullsized":"2023-10-25T00:23:37.054692+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4209","last_scrub_stamp":"2023-10-25T00:23:37.054538+0000","last_deep_scrub":"72'4209","last_deep_scrub_stamp":"2023-10-25T00:23:37.054538+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.054538+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:39:40.159884+0000","scrub_duration":0.40759778099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":24141,"num_write_kb":281325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.24","version":"68'4915","reported_seq":5003,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.325819+0000","last_change":"2023-10-25T00:22:38.600278+0000","last_active":"2023-10-25T00:23:26.325819+0000","last_peered":"2023-10-25T00:23:26.325819+0000","last_clean":"2023-10-25T00:23:26.325819+0000","last_became_active":"2023-10-24T23:53:02.717002+0000","last_became_peered":"2023-10-24T23:53:02.717002+0000","last_unstale":"2023-10-25T00:23:26.325819+0000","last_undegraded":"2023-10-25T00:23:26.325819+0000","last_fullsized":"2023-10-25T00:23:26.325819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4915","last_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_deep_scrub":"68'4915","last_deep_scrub_stamp":"2023-10-25T00:22:38.600135+0000","last_clean_scrub_stamp":"2023-10-25T00:22:38.600135+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:42:30.671003+0000","scrub_duration":0.18827047799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26161,"num_write_kb":314084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.27","version":"72'4933","reported_seq":5018,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.679539+0000","last_change":"2023-10-25T00:23:31.791243+0000","last_active":"2023-10-25T00:23:35.679539+0000","last_peered":"2023-10-25T00:23:35.679539+0000","last_clean":"2023-10-25T00:23:35.679539+0000","last_became_active":"2023-10-24T23:53:11.813648+0000","last_became_peered":"2023-10-24T23:53:11.813648+0000","last_unstale":"2023-10-25T00:23:35.679539+0000","last_undegraded":"2023-10-25T00:23:35.679539+0000","last_fullsized":"2023-10-25T00:23:35.679539+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4933","last_scrub_stamp":"2023-10-25T00:23:31.791069+0000","last_deep_scrub":"72'4933","last_deep_scrub_stamp":"2023-10-25T00:23:31.791069+0000","last_clean_scrub_stamp":"2023-10-25T00:23:31.791069+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T07:53:40.465688+0000","scrub_duration":0.269140133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26695,"num_write_kb":352905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.25","version":"68'5055","reported_seq":5144,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:25.928638+0000","last_change":"2023-10-25T00:22:43.444464+0000","last_active":"2023-10-25T00:23:25.928638+0000","last_peered":"2023-10-25T00:23:25.928638+0000","last_clean":"2023-10-25T00:23:25.928638+0000","last_became_active":"2023-10-24T23:53:02.717692+0000","last_became_peered":"2023-10-24T23:53:02.717692+0000","last_unstale":"2023-10-25T00:23:25.928638+0000","last_undegraded":"2023-10-25T00:23:25.928638+0000","last_fullsized":"2023-10-25T00:23:25.928638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5055","last_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_deep_scrub":"68'5055","last_deep_scrub_stamp":"2023-10-25T00:22:43.444210+0000","last_clean_scrub_stamp":"2023-10-25T00:22:43.444210+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:13:34.310177+0000","scrub_duration":0.081291211000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27516,"num_write_kb":327406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.24","version":"72'4083","reported_seq":4172,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:47.749735+0000","last_change":"2023-10-25T00:23:47.749735+0000","last_active":"2023-10-25T00:23:47.749735+0000","last_peered":"2023-10-25T00:23:47.749735+0000","last_clean":"2023-10-25T00:23:47.749735+0000","last_became_active":"2023-10-24T23:53:11.823100+0000","last_became_peered":"2023-10-24T23:53:11.823100+0000","last_unstale":"2023-10-25T00:23:47.749735+0000","last_undegraded":"2023-10-25T00:23:47.749735+0000","last_fullsized":"2023-10-25T00:23:47.749735+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4083","last_scrub_stamp":"2023-10-25T00:23:47.749591+0000","last_deep_scrub":"72'4083","last_deep_scrub_stamp":"2023-10-25T00:23:47.749591+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.749591+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:43:07.403801+0000","scrub_duration":0.25918773899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":22892,"num_write_kb":296872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.26","version":"68'4777","reported_seq":4867,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.325889+0000","last_change":"2023-10-25T00:23:24.846143+0000","last_active":"2023-10-25T00:23:26.325889+0000","last_peered":"2023-10-25T00:23:26.325889+0000","last_clean":"2023-10-25T00:23:26.325889+0000","last_became_active":"2023-10-24T23:53:02.715913+0000","last_became_peered":"2023-10-24T23:53:02.715913+0000","last_unstale":"2023-10-25T00:23:26.325889+0000","last_undegraded":"2023-10-25T00:23:26.325889+0000","last_fullsized":"2023-10-25T00:23:26.325889+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4777","last_scrub_stamp":"2023-10-25T00:23:24.846002+0000","last_deep_scrub":"68'4777","last_deep_scrub_stamp":"2023-10-25T00:23:24.846002+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.846002+0000","objects_scrubbed":0,"log_size":4777,"log_dups_size":0,"ondisk_log_size":4777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:41:16.418283+0000","scrub_duration":0.19199595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25812,"num_write_kb":300181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"72'4618","reported_seq":4709,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.679474+0000","last_change":"2023-10-25T00:23:33.839487+0000","last_active":"2023-10-25T00:23:35.679474+0000","last_peered":"2023-10-25T00:23:35.679474+0000","last_clean":"2023-10-25T00:23:35.679474+0000","last_became_active":"2023-10-24T23:53:11.807070+0000","last_became_peered":"2023-10-24T23:53:11.807070+0000","last_unstale":"2023-10-25T00:23:35.679474+0000","last_undegraded":"2023-10-25T00:23:35.679474+0000","last_fullsized":"2023-10-25T00:23:35.679474+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4618","last_scrub_stamp":"2023-10-25T00:23:33.839138+0000","last_deep_scrub":"72'4618","last_deep_scrub_stamp":"2023-10-25T00:23:33.839138+0000","last_clean_scrub_stamp":"2023-10-25T00:23:33.839138+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:25:32.655697+0000","scrub_duration":0.330454572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":26180,"num_write_kb":326342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"68'5340","reported_seq":5428,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.325944+0000","last_change":"2023-10-25T00:22:42.520048+0000","last_active":"2023-10-25T00:23:26.325944+0000","last_peered":"2023-10-25T00:23:26.325944+0000","last_clean":"2023-10-25T00:23:26.325944+0000","last_became_active":"2023-10-24T23:53:02.715750+0000","last_became_peered":"2023-10-24T23:53:02.715750+0000","last_unstale":"2023-10-25T00:23:26.325944+0000","last_undegraded":"2023-10-25T00:23:26.325944+0000","last_fullsized":"2023-10-25T00:23:26.325944+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5340","last_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_deep_scrub":"68'5340","last_deep_scrub_stamp":"2023-10-25T00:22:42.519886+0000","last_clean_scrub_stamp":"2023-10-25T00:22:42.519886+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:21:49.832889+0000","scrub_duration":0.129134375,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28701,"num_write_kb":348521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2a","version":"72'4171","reported_seq":4258,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.283494+0000","last_change":"2023-10-25T00:24:17.419254+0000","last_active":"2023-10-25T00:24:34.283494+0000","last_peered":"2023-10-25T00:24:34.283494+0000","last_clean":"2023-10-25T00:24:34.283494+0000","last_became_active":"2023-10-24T23:53:11.834769+0000","last_became_peered":"2023-10-24T23:53:11.834769+0000","last_unstale":"2023-10-25T00:24:34.283494+0000","last_undegraded":"2023-10-25T00:24:34.283494+0000","last_fullsized":"2023-10-25T00:24:34.283494+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4171","last_scrub_stamp":"2023-10-25T00:24:17.419103+0000","last_deep_scrub":"72'4171","last_deep_scrub_stamp":"2023-10-25T00:24:17.419103+0000","last_clean_scrub_stamp":"2023-10-25T00:24:17.419103+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:44:59.723424+0000","scrub_duration":0.37988864300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":22551,"num_write_kb":303498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.28","version":"68'4493","reported_seq":4582,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:01.323167+0000","last_change":"2023-10-25T00:23:01.323167+0000","last_active":"2023-10-25T00:23:01.323167+0000","last_peered":"2023-10-25T00:23:01.323167+0000","last_clean":"2023-10-25T00:23:01.323167+0000","last_became_active":"2023-10-24T23:53:02.725846+0000","last_became_peered":"2023-10-24T23:53:02.725846+0000","last_unstale":"2023-10-25T00:23:01.323167+0000","last_undegraded":"2023-10-25T00:23:01.323167+0000","last_fullsized":"2023-10-25T00:23:01.323167+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4493","last_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_deep_scrub":"68'4493","last_deep_scrub_stamp":"2023-10-25T00:23:01.322941+0000","last_clean_scrub_stamp":"2023-10-25T00:23:01.322941+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:28:49.029022+0000","scrub_duration":0.13616751799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23797,"num_write_kb":300260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2b","version":"72'4763","reported_seq":4848,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:57.009674+0000","last_change":"2023-10-25T00:23:57.009674+0000","last_active":"2023-10-25T00:23:57.009674+0000","last_peered":"2023-10-25T00:23:57.009674+0000","last_clean":"2023-10-25T00:23:57.009674+0000","last_became_active":"2023-10-24T23:53:44.473873+0000","last_became_peered":"2023-10-24T23:53:44.473873+0000","last_unstale":"2023-10-25T00:23:57.009674+0000","last_undegraded":"2023-10-25T00:23:57.009674+0000","last_fullsized":"2023-10-25T00:23:57.009674+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4763","last_scrub_stamp":"2023-10-25T00:23:57.009531+0000","last_deep_scrub":"72'4763","last_deep_scrub_stamp":"2023-10-25T00:23:57.009531+0000","last_clean_scrub_stamp":"2023-10-25T00:23:57.009531+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:34:10.319674+0000","scrub_duration":0.25795965799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":25751,"num_write_kb":345225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"68'4946","reported_seq":5035,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.569710+0000","last_change":"2023-10-25T00:23:55.632753+0000","last_active":"2023-10-25T00:24:32.569710+0000","last_peered":"2023-10-25T00:24:32.569710+0000","last_clean":"2023-10-25T00:24:32.569710+0000","last_became_active":"2023-10-24T23:53:02.730184+0000","last_became_peered":"2023-10-24T23:53:02.730184+0000","last_unstale":"2023-10-25T00:24:32.569710+0000","last_undegraded":"2023-10-25T00:24:32.569710+0000","last_fullsized":"2023-10-25T00:24:32.569710+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4946","last_scrub_stamp":"2023-10-25T00:23:55.632589+0000","last_deep_scrub":"68'4946","last_deep_scrub_stamp":"2023-10-25T00:23:55.632589+0000","last_clean_scrub_stamp":"2023-10-25T00:23:55.632589+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T06:34:04.162109+0000","scrub_duration":0.103749073,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26583,"num_write_kb":315633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.28","version":"72'4352","reported_seq":4438,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.820605+0000","last_change":"2023-10-25T00:23:34.629523+0000","last_active":"2023-10-25T00:23:35.820605+0000","last_peered":"2023-10-25T00:23:35.820605+0000","last_clean":"2023-10-25T00:23:35.820605+0000","last_became_active":"2023-10-24T23:53:11.857364+0000","last_became_peered":"2023-10-24T23:53:11.857364+0000","last_unstale":"2023-10-25T00:23:35.820605+0000","last_undegraded":"2023-10-25T00:23:35.820605+0000","last_fullsized":"2023-10-25T00:23:35.820605+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4352","last_scrub_stamp":"2023-10-25T00:23:34.629323+0000","last_deep_scrub":"72'4352","last_deep_scrub_stamp":"2023-10-25T00:23:34.629323+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.629323+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:42:38.589150+0000","scrub_duration":0.29560794600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":23957,"num_write_kb":301052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2a","version":"68'5026","reported_seq":5115,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.315212+0000","last_change":"2023-10-25T00:24:00.036586+0000","last_active":"2023-10-25T00:24:34.315212+0000","last_peered":"2023-10-25T00:24:34.315212+0000","last_clean":"2023-10-25T00:24:34.315212+0000","last_became_active":"2023-10-24T23:53:02.725862+0000","last_became_peered":"2023-10-24T23:53:02.725862+0000","last_unstale":"2023-10-25T00:24:34.315212+0000","last_undegraded":"2023-10-25T00:24:34.315212+0000","last_fullsized":"2023-10-25T00:24:34.315212+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5026","last_scrub_stamp":"2023-10-25T00:24:00.036334+0000","last_deep_scrub":"68'5026","last_deep_scrub_stamp":"2023-10-25T00:24:00.036334+0000","last_clean_scrub_stamp":"2023-10-25T00:24:00.036334+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:01:22.913142+0000","scrub_duration":0.209434338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":26401,"num_write_kb":310586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.29","version":"72'4393","reported_seq":4484,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.313751+0000","last_change":"2023-10-25T00:23:07.556990+0000","last_active":"2023-10-25T00:24:34.313751+0000","last_peered":"2023-10-25T00:24:34.313751+0000","last_clean":"2023-10-25T00:24:34.313751+0000","last_became_active":"2023-10-24T23:53:11.858209+0000","last_became_peered":"2023-10-24T23:53:11.858209+0000","last_unstale":"2023-10-25T00:24:34.313751+0000","last_undegraded":"2023-10-25T00:24:34.313751+0000","last_fullsized":"2023-10-25T00:24:34.313751+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4393","last_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_deep_scrub":"72'4393","last_deep_scrub_stamp":"2023-10-25T00:23:07.556784+0000","last_clean_scrub_stamp":"2023-10-25T00:23:07.556784+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:21:00.928852+0000","scrub_duration":0.35410455800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":24752,"num_write_kb":296109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2b","version":"68'5620","reported_seq":5712,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.316699+0000","last_change":"2023-10-25T00:23:12.194324+0000","last_active":"2023-10-25T00:24:34.316699+0000","last_peered":"2023-10-25T00:24:34.316699+0000","last_clean":"2023-10-25T00:24:34.316699+0000","last_became_active":"2023-10-24T23:53:02.728870+0000","last_became_peered":"2023-10-24T23:53:02.728870+0000","last_unstale":"2023-10-25T00:24:34.316699+0000","last_undegraded":"2023-10-25T00:24:34.316699+0000","last_fullsized":"2023-10-25T00:24:34.316699+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5620","last_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_deep_scrub":"68'5620","last_deep_scrub_stamp":"2023-10-25T00:23:12.194008+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.194008+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:26:33.868259+0000","scrub_duration":0.086061930999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":29863,"num_write_kb":337169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"72'4340","reported_seq":4429,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:35.679514+0000","last_change":"2023-10-25T00:23:23.851965+0000","last_active":"2023-10-25T00:23:35.679514+0000","last_peered":"2023-10-25T00:23:35.679514+0000","last_clean":"2023-10-25T00:23:35.679514+0000","last_became_active":"2023-10-24T23:53:11.838092+0000","last_became_peered":"2023-10-24T23:53:11.838092+0000","last_unstale":"2023-10-25T00:23:35.679514+0000","last_undegraded":"2023-10-25T00:23:35.679514+0000","last_fullsized":"2023-10-25T00:23:35.679514+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4340","last_scrub_stamp":"2023-10-25T00:23:23.851809+0000","last_deep_scrub":"72'4340","last_deep_scrub_stamp":"2023-10-25T00:23:23.851809+0000","last_clean_scrub_stamp":"2023-10-25T00:23:23.851809+0000","objects_scrubbed":0,"log_size":4340,"log_dups_size":0,"ondisk_log_size":4340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:29:00.223998+0000","scrub_duration":0.37196321399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":36,"num_write":23367,"num_write_kb":302669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2c","version":"68'4464","reported_seq":4551,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.287660+0000","last_change":"2023-10-25T00:22:57.904669+0000","last_active":"2023-10-25T00:24:34.287660+0000","last_peered":"2023-10-25T00:24:34.287660+0000","last_clean":"2023-10-25T00:24:34.287660+0000","last_became_active":"2023-10-24T23:53:02.723864+0000","last_became_peered":"2023-10-24T23:53:02.723864+0000","last_unstale":"2023-10-25T00:24:34.287660+0000","last_undegraded":"2023-10-25T00:24:34.287660+0000","last_fullsized":"2023-10-25T00:24:34.287660+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4464","last_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_deep_scrub":"68'4464","last_deep_scrub_stamp":"2023-10-25T00:22:57.904532+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.904532+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:17:44.026155+0000","scrub_duration":0.14132308199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24372,"num_write_kb":278074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2f","version":"72'5589","reported_seq":5674,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:45.174602+0000","last_change":"2023-10-25T00:23:45.174602+0000","last_active":"2023-10-25T00:23:45.174602+0000","last_peered":"2023-10-25T00:23:45.174602+0000","last_clean":"2023-10-25T00:23:45.174602+0000","last_became_active":"2023-10-24T23:53:11.858984+0000","last_became_peered":"2023-10-24T23:53:11.858984+0000","last_unstale":"2023-10-25T00:23:45.174602+0000","last_undegraded":"2023-10-25T00:23:45.174602+0000","last_fullsized":"2023-10-25T00:23:45.174602+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5589","last_scrub_stamp":"2023-10-25T00:23:45.174448+0000","last_deep_scrub":"72'5589","last_deep_scrub_stamp":"2023-10-25T00:23:45.174448+0000","last_clean_scrub_stamp":"2023-10-25T00:23:45.174448+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:47:45.550696+0000","scrub_duration":0.18148444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":30206,"num_write_kb":399557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2d","version":"68'4542","reported_seq":4630,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.447005+0000","last_change":"2023-10-25T00:23:09.345212+0000","last_active":"2023-10-25T00:24:34.447005+0000","last_peered":"2023-10-25T00:24:34.447005+0000","last_clean":"2023-10-25T00:24:34.447005+0000","last_became_active":"2023-10-24T23:53:02.703479+0000","last_became_peered":"2023-10-24T23:53:02.703479+0000","last_unstale":"2023-10-25T00:24:34.447005+0000","last_undegraded":"2023-10-25T00:24:34.447005+0000","last_fullsized":"2023-10-25T00:24:34.447005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4542","last_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_deep_scrub":"68'4542","last_deep_scrub_stamp":"2023-10-25T00:23:09.344860+0000","last_clean_scrub_stamp":"2023-10-25T00:23:09.344860+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:16:09.325372+0000","scrub_duration":0.066583699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23889,"num_write_kb":292323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"72'3430","reported_seq":3517,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.287730+0000","last_change":"2023-10-25T00:23:21.798045+0000","last_active":"2023-10-25T00:24:34.287730+0000","last_peered":"2023-10-25T00:24:34.287730+0000","last_clean":"2023-10-25T00:24:34.287730+0000","last_became_active":"2023-10-24T23:53:11.838498+0000","last_became_peered":"2023-10-24T23:53:11.838498+0000","last_unstale":"2023-10-25T00:24:34.287730+0000","last_undegraded":"2023-10-25T00:24:34.287730+0000","last_fullsized":"2023-10-25T00:24:34.287730+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3430","last_scrub_stamp":"2023-10-25T00:23:21.797727+0000","last_deep_scrub":"72'3430","last_deep_scrub_stamp":"2023-10-25T00:23:21.797727+0000","last_clean_scrub_stamp":"2023-10-25T00:23:21.797727+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:15:30.844796+0000","scrub_duration":0.29875561899999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19411,"num_write_kb":241862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2e","version":"68'4719","reported_seq":4807,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.308252+0000","last_change":"2023-10-25T00:22:58.903089+0000","last_active":"2023-10-25T00:23:26.308252+0000","last_peered":"2023-10-25T00:23:26.308252+0000","last_clean":"2023-10-25T00:23:26.308252+0000","last_became_active":"2023-10-24T23:53:02.723107+0000","last_became_peered":"2023-10-24T23:53:02.723107+0000","last_unstale":"2023-10-25T00:23:26.308252+0000","last_undegraded":"2023-10-25T00:23:26.308252+0000","last_fullsized":"2023-10-25T00:23:26.308252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4719","last_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_deep_scrub":"68'4719","last_deep_scrub_stamp":"2023-10-25T00:22:58.902769+0000","last_clean_scrub_stamp":"2023-10-25T00:22:58.902769+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:17:01.236787+0000","scrub_duration":0.136999537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":25020,"num_write_kb":305197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2d","version":"72'4169","reported_seq":4253,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.445471+0000","last_change":"2023-10-25T00:23:40.640863+0000","last_active":"2023-10-25T00:24:34.445471+0000","last_peered":"2023-10-25T00:24:34.445471+0000","last_clean":"2023-10-25T00:24:34.445471+0000","last_became_active":"2023-10-24T23:53:11.809161+0000","last_became_peered":"2023-10-24T23:53:11.809161+0000","last_unstale":"2023-10-25T00:24:34.445471+0000","last_undegraded":"2023-10-25T00:24:34.445471+0000","last_fullsized":"2023-10-25T00:24:34.445471+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4169","last_scrub_stamp":"2023-10-25T00:23:40.640645+0000","last_deep_scrub":"72'4169","last_deep_scrub_stamp":"2023-10-25T00:23:40.640645+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.640645+0000","objects_scrubbed":0,"log_size":4169,"log_dups_size":0,"ondisk_log_size":4169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:33:06.110573+0000","scrub_duration":0.28697117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23235,"num_write_kb":306175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2f","version":"68'5952","reported_seq":6048,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.494437+0000","last_change":"2023-10-25T00:23:05.017693+0000","last_active":"2023-10-25T00:23:26.494437+0000","last_peered":"2023-10-25T00:23:26.494437+0000","last_clean":"2023-10-25T00:23:26.494437+0000","last_became_active":"2023-10-24T23:53:44.629578+0000","last_became_peered":"2023-10-24T23:53:44.629578+0000","last_unstale":"2023-10-25T00:23:26.494437+0000","last_undegraded":"2023-10-25T00:23:26.494437+0000","last_fullsized":"2023-10-25T00:23:26.494437+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5952","last_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_deep_scrub":"68'5952","last_deep_scrub_stamp":"2023-10-25T00:23:05.017340+0000","last_clean_scrub_stamp":"2023-10-25T00:23:05.017340+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:37:31.140541+0000","scrub_duration":0.055624675999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":32341,"num_write_kb":375441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.32","version":"72'3717","reported_seq":3801,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:23.237030+0000","last_change":"2023-10-25T00:24:23.237030+0000","last_active":"2023-10-25T00:24:23.237030+0000","last_peered":"2023-10-25T00:24:23.237030+0000","last_clean":"2023-10-25T00:24:23.237030+0000","last_became_active":"2023-10-24T23:53:11.836667+0000","last_became_peered":"2023-10-24T23:53:11.836667+0000","last_unstale":"2023-10-25T00:24:23.237030+0000","last_undegraded":"2023-10-25T00:24:23.237030+0000","last_fullsized":"2023-10-25T00:24:23.237030+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3717","last_scrub_stamp":"2023-10-25T00:24:23.236882+0000","last_deep_scrub":"72'3717","last_deep_scrub_stamp":"2023-10-25T00:24:23.236882+0000","last_clean_scrub_stamp":"2023-10-25T00:24:23.236882+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:09:23.115312+0000","scrub_duration":0.223121614,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21101,"num_write_kb":269108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.30","version":"68'4825","reported_seq":4914,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.307799+0000","last_change":"2023-10-25T00:23:06.089847+0000","last_active":"2023-10-25T00:23:26.307799+0000","last_peered":"2023-10-25T00:23:26.307799+0000","last_clean":"2023-10-25T00:23:26.307799+0000","last_became_active":"2023-10-24T23:53:02.730759+0000","last_became_peered":"2023-10-24T23:53:02.730759+0000","last_unstale":"2023-10-25T00:23:26.307799+0000","last_undegraded":"2023-10-25T00:23:26.307799+0000","last_fullsized":"2023-10-25T00:23:26.307799+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4825","last_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_deep_scrub":"68'4825","last_deep_scrub_stamp":"2023-10-25T00:23:06.089705+0000","last_clean_scrub_stamp":"2023-10-25T00:23:06.089705+0000","objects_scrubbed":0,"log_size":4825,"log_dups_size":0,"ondisk_log_size":4825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:04:51.412522+0000","scrub_duration":0.15071393499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25319,"num_write_kb":294113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.33","version":"72'3714","reported_seq":3802,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.296748+0000","last_change":"2023-10-25T00:24:00.873759+0000","last_active":"2023-10-25T00:24:34.296748+0000","last_peered":"2023-10-25T00:24:34.296748+0000","last_clean":"2023-10-25T00:24:34.296748+0000","last_became_active":"2023-10-24T23:53:11.847121+0000","last_became_peered":"2023-10-24T23:53:11.847121+0000","last_unstale":"2023-10-25T00:24:34.296748+0000","last_undegraded":"2023-10-25T00:24:34.296748+0000","last_fullsized":"2023-10-25T00:24:34.296748+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3714","last_scrub_stamp":"2023-10-25T00:24:00.873431+0000","last_deep_scrub":"72'3714","last_deep_scrub_stamp":"2023-10-25T00:24:00.873431+0000","last_clean_scrub_stamp":"2023-10-25T00:24:00.873431+0000","objects_scrubbed":0,"log_size":3714,"log_dups_size":0,"ondisk_log_size":3714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:03:48.699043+0000","scrub_duration":0.31231597300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":21148,"num_write_kb":265324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"68'4878","reported_seq":4968,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:26.271604+0000","last_change":"2023-10-25T00:22:49.227455+0000","last_active":"2023-10-25T00:23:26.271604+0000","last_peered":"2023-10-25T00:23:26.271604+0000","last_clean":"2023-10-25T00:23:26.271604+0000","last_became_active":"2023-10-24T23:53:02.721279+0000","last_became_peered":"2023-10-24T23:53:02.721279+0000","last_unstale":"2023-10-25T00:23:26.271604+0000","last_undegraded":"2023-10-25T00:23:26.271604+0000","last_fullsized":"2023-10-25T00:23:26.271604+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4878","last_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_deep_scrub":"68'4878","last_deep_scrub_stamp":"2023-10-25T00:22:49.227012+0000","last_clean_scrub_stamp":"2023-10-25T00:22:49.227012+0000","objects_scrubbed":0,"log_size":4878,"log_dups_size":0,"ondisk_log_size":4878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:58:21.157778+0000","scrub_duration":0.13340163999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26154,"num_write_kb":305558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.30","version":"72'3434","reported_seq":3519,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:57.337823+0000","last_change":"2023-10-25T00:23:57.337823+0000","last_active":"2023-10-25T00:23:57.337823+0000","last_peered":"2023-10-25T00:23:57.337823+0000","last_clean":"2023-10-25T00:23:57.337823+0000","last_became_active":"2023-10-24T23:53:11.858794+0000","last_became_peered":"2023-10-24T23:53:11.858794+0000","last_unstale":"2023-10-25T00:23:57.337823+0000","last_undegraded":"2023-10-25T00:23:57.337823+0000","last_fullsized":"2023-10-25T00:23:57.337823+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3434","last_scrub_stamp":"2023-10-25T00:23:57.337452+0000","last_deep_scrub":"72'3434","last_deep_scrub_stamp":"2023-10-25T00:23:57.337452+0000","last_clean_scrub_stamp":"2023-10-25T00:23:57.337452+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:08:11.559971+0000","scrub_duration":0.315725692,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":18449,"num_write_kb":262373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.32","version":"68'4672","reported_seq":4762,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:03.653382+0000","last_change":"2023-10-25T00:24:03.653382+0000","last_active":"2023-10-25T00:24:03.653382+0000","last_peered":"2023-10-25T00:24:03.653382+0000","last_clean":"2023-10-25T00:24:03.653382+0000","last_became_active":"2023-10-24T23:53:02.694123+0000","last_became_peered":"2023-10-24T23:53:02.694123+0000","last_unstale":"2023-10-25T00:24:03.653382+0000","last_undegraded":"2023-10-25T00:24:03.653382+0000","last_fullsized":"2023-10-25T00:24:03.653382+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4672","last_scrub_stamp":"2023-10-25T00:24:03.653141+0000","last_deep_scrub":"68'4672","last_deep_scrub_stamp":"2023-10-25T00:24:03.653141+0000","last_clean_scrub_stamp":"2023-10-25T00:24:03.653141+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:19:24.224599+0000","scrub_duration":0.077062947000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25223,"num_write_kb":293792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.31","version":"72'4013","reported_seq":4103,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:51.895151+0000","last_change":"2023-10-25T00:23:51.895151+0000","last_active":"2023-10-25T00:23:51.895151+0000","last_peered":"2023-10-25T00:23:51.895151+0000","last_clean":"2023-10-25T00:23:51.895151+0000","last_became_active":"2023-10-24T23:53:11.821045+0000","last_became_peered":"2023-10-24T23:53:11.821045+0000","last_unstale":"2023-10-25T00:23:51.895151+0000","last_undegraded":"2023-10-25T00:23:51.895151+0000","last_fullsized":"2023-10-25T00:23:51.895151+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4013","last_scrub_stamp":"2023-10-25T00:23:51.894968+0000","last_deep_scrub":"72'4013","last_deep_scrub_stamp":"2023-10-25T00:23:51.894968+0000","last_clean_scrub_stamp":"2023-10-25T00:23:51.894968+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:20:31.752935+0000","scrub_duration":0.32625951199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":1,"num_write":22066,"num_write_kb":297298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.33","version":"68'3918","reported_seq":4006,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:30.839354+0000","last_change":"2023-10-25T00:23:49.563900+0000","last_active":"2023-10-25T00:24:30.839354+0000","last_peered":"2023-10-25T00:24:30.839354+0000","last_clean":"2023-10-25T00:24:30.839354+0000","last_became_active":"2023-10-24T23:53:02.705505+0000","last_became_peered":"2023-10-24T23:53:02.705505+0000","last_unstale":"2023-10-25T00:24:30.839354+0000","last_undegraded":"2023-10-25T00:24:30.839354+0000","last_fullsized":"2023-10-25T00:24:30.839354+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3918","last_scrub_stamp":"2023-10-25T00:23:49.563617+0000","last_deep_scrub":"68'3918","last_deep_scrub_stamp":"2023-10-25T00:23:49.563617+0000","last_clean_scrub_stamp":"2023-10-25T00:23:49.563617+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:50:13.550847+0000","scrub_duration":0.092172528000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21675,"num_write_kb":247000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.36","version":"72'4401","reported_seq":4487,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.296902+0000","last_change":"2023-10-25T00:23:42.768767+0000","last_active":"2023-10-25T00:24:34.296902+0000","last_peered":"2023-10-25T00:24:34.296902+0000","last_clean":"2023-10-25T00:24:34.296902+0000","last_became_active":"2023-10-24T23:53:11.846635+0000","last_became_peered":"2023-10-24T23:53:11.846635+0000","last_unstale":"2023-10-25T00:24:34.296902+0000","last_undegraded":"2023-10-25T00:24:34.296902+0000","last_fullsized":"2023-10-25T00:24:34.296902+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4401","last_scrub_stamp":"2023-10-25T00:23:42.768621+0000","last_deep_scrub":"72'4401","last_deep_scrub_stamp":"2023-10-25T00:23:42.768621+0000","last_clean_scrub_stamp":"2023-10-25T00:23:42.768621+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:23:18.678304+0000","scrub_duration":0.26783985300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":7,"num_write":24294,"num_write_kb":326749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.34","version":"68'3970","reported_seq":4057,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:30.839420+0000","last_change":"2023-10-25T00:23:12.428867+0000","last_active":"2023-10-25T00:24:30.839420+0000","last_peered":"2023-10-25T00:24:30.839420+0000","last_clean":"2023-10-25T00:24:30.839420+0000","last_became_active":"2023-10-24T23:53:02.725990+0000","last_became_peered":"2023-10-24T23:53:02.725990+0000","last_unstale":"2023-10-25T00:24:30.839420+0000","last_undegraded":"2023-10-25T00:24:30.839420+0000","last_fullsized":"2023-10-25T00:24:30.839420+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'3970","last_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_deep_scrub":"68'3970","last_deep_scrub_stamp":"2023-10-25T00:23:12.428494+0000","last_clean_scrub_stamp":"2023-10-25T00:23:12.428494+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:28:07.899437+0000","scrub_duration":0.13106953599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21202,"num_write_kb":257085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.37","version":"72'5559","reported_seq":5651,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.316582+0000","last_change":"2023-10-25T00:23:41.090253+0000","last_active":"2023-10-25T00:24:34.316582+0000","last_peered":"2023-10-25T00:24:34.316582+0000","last_clean":"2023-10-25T00:24:34.316582+0000","last_became_active":"2023-10-24T23:53:11.827200+0000","last_became_peered":"2023-10-24T23:53:11.827200+0000","last_unstale":"2023-10-25T00:24:34.316582+0000","last_undegraded":"2023-10-25T00:24:34.316582+0000","last_fullsized":"2023-10-25T00:24:34.316582+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5559","last_scrub_stamp":"2023-10-25T00:23:41.090104+0000","last_deep_scrub":"72'5559","last_deep_scrub_stamp":"2023-10-25T00:23:41.090104+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.090104+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:40:20.680440+0000","scrub_duration":0.241945666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":31202,"num_write_kb":391572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.35","version":"68'4936","reported_seq":5024,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.429817+0000","last_change":"2023-10-25T00:23:02.021272+0000","last_active":"2023-10-25T00:24:31.429817+0000","last_peered":"2023-10-25T00:24:31.429817+0000","last_clean":"2023-10-25T00:24:31.429817+0000","last_became_active":"2023-10-24T23:53:02.722318+0000","last_became_peered":"2023-10-24T23:53:02.722318+0000","last_unstale":"2023-10-25T00:24:31.429817+0000","last_undegraded":"2023-10-25T00:24:31.429817+0000","last_fullsized":"2023-10-25T00:24:31.429817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4936","last_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_deep_scrub":"68'4936","last_deep_scrub_stamp":"2023-10-25T00:23:02.021108+0000","last_clean_scrub_stamp":"2023-10-25T00:23:02.021108+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:25:22.652380+0000","scrub_duration":0.17348896599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":26134,"num_write_kb":309652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.34","version":"72'3558","reported_seq":3647,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.316185+0000","last_change":"2023-10-25T00:23:47.072846+0000","last_active":"2023-10-25T00:24:34.316185+0000","last_peered":"2023-10-25T00:24:34.316185+0000","last_clean":"2023-10-25T00:24:34.316185+0000","last_became_active":"2023-10-24T23:53:11.806665+0000","last_became_peered":"2023-10-24T23:53:11.806665+0000","last_unstale":"2023-10-25T00:24:34.316185+0000","last_undegraded":"2023-10-25T00:24:34.316185+0000","last_fullsized":"2023-10-25T00:24:34.316185+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3558","last_scrub_stamp":"2023-10-25T00:23:47.072625+0000","last_deep_scrub":"72'3558","last_deep_scrub_stamp":"2023-10-25T00:23:47.072625+0000","last_clean_scrub_stamp":"2023-10-25T00:23:47.072625+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:59:13.347978+0000","scrub_duration":0.34108986800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":34,"num_write":19715,"num_write_kb":243025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"68'5282","reported_seq":5370,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.110730+0000","last_change":"2023-10-25T00:23:14.290293+0000","last_active":"2023-10-25T00:24:31.110730+0000","last_peered":"2023-10-25T00:24:31.110730+0000","last_clean":"2023-10-25T00:24:31.110730+0000","last_became_active":"2023-10-24T23:53:02.726092+0000","last_became_peered":"2023-10-24T23:53:02.726092+0000","last_unstale":"2023-10-25T00:24:31.110730+0000","last_undegraded":"2023-10-25T00:24:31.110730+0000","last_fullsized":"2023-10-25T00:24:31.110730+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5282","last_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_deep_scrub":"68'5282","last_deep_scrub_stamp":"2023-10-25T00:23:14.290099+0000","last_clean_scrub_stamp":"2023-10-25T00:23:14.290099+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:44:38.139530+0000","scrub_duration":0.10140234200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":27848,"num_write_kb":324561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"72'3914","reported_seq":4004,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.296299+0000","last_change":"2023-10-25T00:23:46.833446+0000","last_active":"2023-10-25T00:24:34.296299+0000","last_peered":"2023-10-25T00:24:34.296299+0000","last_clean":"2023-10-25T00:24:34.296299+0000","last_became_active":"2023-10-24T23:53:11.845608+0000","last_became_peered":"2023-10-24T23:53:11.845608+0000","last_unstale":"2023-10-25T00:24:34.296299+0000","last_undegraded":"2023-10-25T00:24:34.296299+0000","last_fullsized":"2023-10-25T00:24:34.296299+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3914","last_scrub_stamp":"2023-10-25T00:23:46.833294+0000","last_deep_scrub":"72'3914","last_deep_scrub_stamp":"2023-10-25T00:23:46.833294+0000","last_clean_scrub_stamp":"2023-10-25T00:23:46.833294+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:02:59.194933+0000","scrub_duration":0.30262013700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":20938,"num_write_kb":288796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"68'6616","reported_seq":6708,"reported_epoch":74,"state":"active+clean","last_fresh":"2023-10-25T00:24:34.316585+0000","last_change":"2023-10-25T00:22:57.985278+0000","last_active":"2023-10-25T00:24:34.316585+0000","last_peered":"2023-10-25T00:24:34.316585+0000","last_clean":"2023-10-25T00:24:34.316585+0000","last_became_active":"2023-10-24T23:53:02.726660+0000","last_became_peered":"2023-10-24T23:53:02.726660+0000","last_unstale":"2023-10-25T00:24:34.316585+0000","last_undegraded":"2023-10-25T00:24:34.316585+0000","last_fullsized":"2023-10-25T00:24:34.316585+0000","mapping_epoch":21,"log_start":"61'400","ondisk_log_start":"61'400","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'6616","last_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_deep_scrub":"68'6616","last_deep_scrub_stamp":"2023-10-25T00:22:57.985096+0000","last_clean_scrub_stamp":"2023-10-25T00:22:57.985096+0000","objects_scrubbed":0,"log_size":6216,"log_dups_size":0,"ondisk_log_size":6216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T12:14:23.491729+0000","scrub_duration":0.050039131000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":36221,"num_write_kb":423916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3a","version":"72'3536","reported_seq":3621,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:32.908624+0000","last_change":"2023-10-25T00:23:32.908624+0000","last_active":"2023-10-25T00:23:32.908624+0000","last_peered":"2023-10-25T00:23:32.908624+0000","last_clean":"2023-10-25T00:23:32.908624+0000","last_became_active":"2023-10-24T23:53:11.811037+0000","last_became_peered":"2023-10-24T23:53:11.811037+0000","last_unstale":"2023-10-25T00:23:32.908624+0000","last_undegraded":"2023-10-25T00:23:32.908624+0000","last_fullsized":"2023-10-25T00:23:32.908624+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3536","last_scrub_stamp":"2023-10-25T00:23:32.908476+0000","last_deep_scrub":"72'3536","last_deep_scrub_stamp":"2023-10-25T00:23:32.908476+0000","last_clean_scrub_stamp":"2023-10-25T00:23:32.908476+0000","objects_scrubbed":0,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T09:33:20.337271+0000","scrub_duration":0.30729883400000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19276,"num_write_kb":253506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.38","version":"68'4980","reported_seq":5065,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:10.321202+0000","last_change":"2023-10-25T00:23:10.321202+0000","last_active":"2023-10-25T00:23:10.321202+0000","last_peered":"2023-10-25T00:23:10.321202+0000","last_clean":"2023-10-25T00:23:10.321202+0000","last_became_active":"2023-10-24T23:53:02.719518+0000","last_became_peered":"2023-10-24T23:53:02.719518+0000","last_unstale":"2023-10-25T00:23:10.321202+0000","last_undegraded":"2023-10-25T00:23:10.321202+0000","last_fullsized":"2023-10-25T00:23:10.321202+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4980","last_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_deep_scrub":"68'4980","last_deep_scrub_stamp":"2023-10-25T00:23:10.320988+0000","last_clean_scrub_stamp":"2023-10-25T00:23:10.320988+0000","objects_scrubbed":0,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:27:34.745429+0000","scrub_duration":0.13750111800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26525,"num_write_kb":306963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3b","version":"72'4160","reported_seq":4247,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:24.541711+0000","last_change":"2023-10-25T00:23:24.541711+0000","last_active":"2023-10-25T00:23:24.541711+0000","last_peered":"2023-10-25T00:23:24.541711+0000","last_clean":"2023-10-25T00:23:24.541711+0000","last_became_active":"2023-10-24T23:53:11.837613+0000","last_became_peered":"2023-10-24T23:53:11.837613+0000","last_unstale":"2023-10-25T00:23:24.541711+0000","last_undegraded":"2023-10-25T00:23:24.541711+0000","last_fullsized":"2023-10-25T00:23:24.541711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4160","last_scrub_stamp":"2023-10-25T00:23:24.541561+0000","last_deep_scrub":"72'4160","last_deep_scrub_stamp":"2023-10-25T00:23:24.541561+0000","last_clean_scrub_stamp":"2023-10-25T00:23:24.541561+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T04:52:13.060866+0000","scrub_duration":0.264226874,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":21819,"num_write_kb":288439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.39","version":"68'4434","reported_seq":4521,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.266956+0000","last_change":"2023-10-25T00:22:47.255974+0000","last_active":"2023-10-25T00:23:06.266956+0000","last_peered":"2023-10-25T00:23:06.266956+0000","last_clean":"2023-10-25T00:23:06.266956+0000","last_became_active":"2023-10-24T23:53:02.720887+0000","last_became_peered":"2023-10-24T23:53:02.720887+0000","last_unstale":"2023-10-25T00:23:06.266956+0000","last_undegraded":"2023-10-25T00:23:06.266956+0000","last_fullsized":"2023-10-25T00:23:06.266956+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4434","last_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_deep_scrub":"68'4434","last_deep_scrub_stamp":"2023-10-25T00:22:47.255646+0000","last_clean_scrub_stamp":"2023-10-25T00:22:47.255646+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:35:52.723567+0000","scrub_duration":0.14955275700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23791,"num_write_kb":292031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.38","version":"72'4195","reported_seq":4277,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:37.837002+0000","last_change":"2023-10-25T00:23:37.837002+0000","last_active":"2023-10-25T00:23:37.837002+0000","last_peered":"2023-10-25T00:23:37.837002+0000","last_clean":"2023-10-25T00:23:37.837002+0000","last_became_active":"2023-10-24T23:53:11.843077+0000","last_became_peered":"2023-10-24T23:53:11.843077+0000","last_unstale":"2023-10-25T00:23:37.837002+0000","last_undegraded":"2023-10-25T00:23:37.837002+0000","last_fullsized":"2023-10-25T00:23:37.837002+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4195","last_scrub_stamp":"2023-10-25T00:23:37.836848+0000","last_deep_scrub":"72'4195","last_deep_scrub_stamp":"2023-10-25T00:23:37.836848+0000","last_clean_scrub_stamp":"2023-10-25T00:23:37.836848+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:49:10.177488+0000","scrub_duration":0.28694403499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":23000,"num_write_kb":285019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3a","version":"68'4243","reported_seq":4329,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.302534+0000","last_change":"2023-10-25T00:23:00.969067+0000","last_active":"2023-10-25T00:23:06.302534+0000","last_peered":"2023-10-25T00:23:06.302534+0000","last_clean":"2023-10-25T00:23:06.302534+0000","last_became_active":"2023-10-24T23:53:02.730746+0000","last_became_peered":"2023-10-24T23:53:02.730746+0000","last_unstale":"2023-10-25T00:23:06.302534+0000","last_undegraded":"2023-10-25T00:23:06.302534+0000","last_fullsized":"2023-10-25T00:23:06.302534+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4243","last_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_deep_scrub":"68'4243","last_deep_scrub_stamp":"2023-10-25T00:23:00.968692+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.968692+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:08:45.877638+0000","scrub_duration":0.15459757700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22439,"num_write_kb":266094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.39","version":"72'4017","reported_seq":4100,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:54.963064+0000","last_change":"2023-10-25T00:23:54.963064+0000","last_active":"2023-10-25T00:23:54.963064+0000","last_peered":"2023-10-25T00:23:54.963064+0000","last_clean":"2023-10-25T00:23:54.963064+0000","last_became_active":"2023-10-24T23:53:11.823553+0000","last_became_peered":"2023-10-24T23:53:11.823553+0000","last_unstale":"2023-10-25T00:23:54.963064+0000","last_undegraded":"2023-10-25T00:23:54.963064+0000","last_fullsized":"2023-10-25T00:23:54.963064+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4017","last_scrub_stamp":"2023-10-25T00:23:54.962911+0000","last_deep_scrub":"72'4017","last_deep_scrub_stamp":"2023-10-25T00:23:54.962911+0000","last_clean_scrub_stamp":"2023-10-25T00:23:54.962911+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T03:49:43.769503+0000","scrub_duration":0.29685359700000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22665,"num_write_kb":291707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3b","version":"68'4509","reported_seq":4598,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:06.489031+0000","last_change":"2023-10-25T00:22:44.022792+0000","last_active":"2023-10-25T00:23:06.489031+0000","last_peered":"2023-10-25T00:23:06.489031+0000","last_clean":"2023-10-25T00:23:06.489031+0000","last_became_active":"2023-10-24T23:53:02.729790+0000","last_became_peered":"2023-10-24T23:53:02.729790+0000","last_unstale":"2023-10-25T00:23:06.489031+0000","last_undegraded":"2023-10-25T00:23:06.489031+0000","last_fullsized":"2023-10-25T00:23:06.489031+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4509","last_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_deep_scrub":"68'4509","last_deep_scrub_stamp":"2023-10-25T00:22:44.022573+0000","last_clean_scrub_stamp":"2023-10-25T00:22:44.022573+0000","objects_scrubbed":0,"log_size":4509,"log_dups_size":0,"ondisk_log_size":4509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T05:54:30.651361+0000","scrub_duration":0.11145192700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23158,"num_write_kb":297883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3e","version":"72'5235","reported_seq":5327,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.569964+0000","last_change":"2023-10-25T00:23:50.831958+0000","last_active":"2023-10-25T00:24:32.569964+0000","last_peered":"2023-10-25T00:24:32.569964+0000","last_clean":"2023-10-25T00:24:32.569964+0000","last_became_active":"2023-10-24T23:53:11.810686+0000","last_became_peered":"2023-10-24T23:53:11.810686+0000","last_unstale":"2023-10-25T00:24:32.569964+0000","last_undegraded":"2023-10-25T00:24:32.569964+0000","last_fullsized":"2023-10-25T00:24:32.569964+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5235","last_scrub_stamp":"2023-10-25T00:23:50.831798+0000","last_deep_scrub":"72'5235","last_deep_scrub_stamp":"2023-10-25T00:23:50.831798+0000","last_clean_scrub_stamp":"2023-10-25T00:23:50.831798+0000","objects_scrubbed":0,"log_size":5235,"log_dups_size":0,"ondisk_log_size":5235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:34:32.587906+0000","scrub_duration":0.30673944400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":15,"num_write":29235,"num_write_kb":367391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3c","version":"68'4260","reported_seq":4350,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:10.674279+0000","last_change":"2023-10-25T00:22:56.486314+0000","last_active":"2023-10-25T00:23:10.674279+0000","last_peered":"2023-10-25T00:23:10.674279+0000","last_clean":"2023-10-25T00:23:10.674279+0000","last_became_active":"2023-10-24T23:53:02.727047+0000","last_became_peered":"2023-10-24T23:53:02.727047+0000","last_unstale":"2023-10-25T00:23:10.674279+0000","last_undegraded":"2023-10-25T00:23:10.674279+0000","last_fullsized":"2023-10-25T00:23:10.674279+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4260","last_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_deep_scrub":"68'4260","last_deep_scrub_stamp":"2023-10-25T00:22:56.485966+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.485966+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T08:16:51.396462+0000","scrub_duration":0.116368769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23357,"num_write_kb":265837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3f","version":"72'4656","reported_seq":4750,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:32.570019+0000","last_change":"2023-10-25T00:23:40.632314+0000","last_active":"2023-10-25T00:24:32.570019+0000","last_peered":"2023-10-25T00:24:32.570019+0000","last_clean":"2023-10-25T00:24:32.570019+0000","last_became_active":"2023-10-24T23:53:11.814977+0000","last_became_peered":"2023-10-24T23:53:11.814977+0000","last_unstale":"2023-10-25T00:24:32.570019+0000","last_undegraded":"2023-10-25T00:24:32.570019+0000","last_fullsized":"2023-10-25T00:24:32.570019+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4656","last_scrub_stamp":"2023-10-25T00:23:40.632160+0000","last_deep_scrub":"72'4656","last_deep_scrub_stamp":"2023-10-25T00:23:40.632160+0000","last_clean_scrub_stamp":"2023-10-25T00:23:40.632160+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:23:41.024594+0000","scrub_duration":0.095096767999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":8,"num_write":26109,"num_write_kb":317781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3d","version":"68'4636","reported_seq":4728,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.427464+0000","last_change":"2023-10-25T00:22:39.672588+0000","last_active":"2023-10-25T00:24:21.427464+0000","last_peered":"2023-10-25T00:24:21.427464+0000","last_clean":"2023-10-25T00:24:21.427464+0000","last_became_active":"2023-10-24T23:53:02.726446+0000","last_became_peered":"2023-10-24T23:53:02.726446+0000","last_unstale":"2023-10-25T00:24:21.427464+0000","last_undegraded":"2023-10-25T00:24:21.427464+0000","last_fullsized":"2023-10-25T00:24:21.427464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4636","last_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_deep_scrub":"68'4636","last_deep_scrub_stamp":"2023-10-25T00:22:39.672438+0000","last_clean_scrub_stamp":"2023-10-25T00:22:39.672438+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T11:47:15.433547+0000","scrub_duration":0.124153799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25124,"num_write_kb":285314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3c","version":"72'3117","reported_seq":3205,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:23:41.679657+0000","last_change":"2023-10-25T00:23:41.679657+0000","last_active":"2023-10-25T00:23:41.679657+0000","last_peered":"2023-10-25T00:23:41.679657+0000","last_clean":"2023-10-25T00:23:41.679657+0000","last_became_active":"2023-10-24T23:53:11.845222+0000","last_became_peered":"2023-10-24T23:53:11.845222+0000","last_unstale":"2023-10-25T00:23:41.679657+0000","last_undegraded":"2023-10-25T00:23:41.679657+0000","last_fullsized":"2023-10-25T00:23:41.679657+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3117","last_scrub_stamp":"2023-10-25T00:23:41.679268+0000","last_deep_scrub":"72'3117","last_deep_scrub_stamp":"2023-10-25T00:23:41.679268+0000","last_clean_scrub_stamp":"2023-10-25T00:23:41.679268+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T00:37:35.283156+0000","scrub_duration":0.30581881399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17240,"num_write_kb":232937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"68'5520","reported_seq":5619,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:21.108493+0000","last_change":"2023-10-25T00:22:56.366641+0000","last_active":"2023-10-25T00:24:21.108493+0000","last_peered":"2023-10-25T00:24:21.108493+0000","last_clean":"2023-10-25T00:24:21.108493+0000","last_became_active":"2023-10-24T23:53:44.630229+0000","last_became_peered":"2023-10-24T23:53:44.630229+0000","last_unstale":"2023-10-25T00:24:21.108493+0000","last_undegraded":"2023-10-25T00:24:21.108493+0000","last_fullsized":"2023-10-25T00:24:21.108493+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'5520","last_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_deep_scrub":"68'5520","last_deep_scrub_stamp":"2023-10-25T00:22:56.366419+0000","last_clean_scrub_stamp":"2023-10-25T00:22:56.366419+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T10:47:59.889167+0000","scrub_duration":0.104594514,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":29665,"num_write_kb":368823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3d","version":"72'3962","reported_seq":4049,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:31.509842+0000","last_change":"2023-10-25T00:23:34.105899+0000","last_active":"2023-10-25T00:24:31.509842+0000","last_peered":"2023-10-25T00:24:31.509842+0000","last_clean":"2023-10-25T00:24:31.509842+0000","last_became_active":"2023-10-24T23:53:11.832349+0000","last_became_peered":"2023-10-24T23:53:11.832349+0000","last_unstale":"2023-10-25T00:24:31.509842+0000","last_undegraded":"2023-10-25T00:24:31.509842+0000","last_fullsized":"2023-10-25T00:24:31.509842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'3962","last_scrub_stamp":"2023-10-25T00:23:34.105758+0000","last_deep_scrub":"72'3962","last_deep_scrub_stamp":"2023-10-25T00:23:34.105758+0000","last_clean_scrub_stamp":"2023-10-25T00:23:34.105758+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T02:35:57.960582+0000","scrub_duration":0.32704070499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21989,"num_write_kb":280762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3f","version":"68'4908","reported_seq":5002,"reported_epoch":73,"state":"active+clean","last_fresh":"2023-10-25T00:24:20.837646+0000","last_change":"2023-10-25T00:23:00.408006+0000","last_active":"2023-10-25T00:24:20.837646+0000","last_peered":"2023-10-25T00:24:20.837646+0000","last_clean":"2023-10-25T00:24:20.837646+0000","last_became_active":"2023-10-24T23:53:02.706063+0000","last_became_peered":"2023-10-24T23:53:02.706063+0000","last_unstale":"2023-10-25T00:24:20.837646+0000","last_undegraded":"2023-10-25T00:24:20.837646+0000","last_fullsized":"2023-10-25T00:24:20.837646+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'4908","last_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_deep_scrub":"68'4908","last_deep_scrub_stamp":"2023-10-25T00:23:00.407650+0000","last_clean_scrub_stamp":"2023-10-25T00:23:00.407650+0000","objects_scrubbed":0,"log_size":4908,"log_dups_size":0,"ondisk_log_size":4908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-26T01:48:18.113931+0000","scrub_duration":0.051805808000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":26829,"num_write_kb":320697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":306,"num_read_kb":320,"num_write":1482516,"num_write_kb":19060243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":267718,"ondisk_log_size":267718,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":19490997,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1686,"num_read_kb":209824,"num_write":4665,"num_write_kb":21105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39043072,"data_stored":38981994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4562,"ondisk_log_size":4562,"up":64,"acting":64,"num_store_stats":8},{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":148,"num_read_kb":0,"num_write":1661441,"num_write_kb":19773626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307026,"ondisk_log_size":307026,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":20082189,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1758,"num_read_kb":286777,"num_write":4645,"num_write_kb":21667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40222720,"data_stored":40164378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4554,"ondisk_log_size":4554,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":127,"num_write":209,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163,"ondisk_log_size":163,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575236,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919112,"kb_used_data":12352,"kb_used_omap":0,"kb_used_meta":906752,"kb_avail":93452728,"statfs":{"total":96636764160,"available":95695593472,"internally_reserved":0,"allocated":12648448,"data_stored":12164710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":928514048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.81499999999999995,"15min":0.95499999999999996},"min":{"1min":0.45500000000000002,"5min":0.22,"15min":0.19900000000000001},"max":{"1min":1.1519999999999999,"5min":3.6349999999999998,"15min":13.929},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.92000000000000004,"15min":0.95699999999999996},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.20999999999999999},"max":{"1min":1.3939999999999999,"5min":3.6659999999999999,"15min":11.315},"last":0.97699999999999998}]},{"osd":1,"last update":"Wed Oct 25 00:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.82999999999999996,"15min":1.1539999999999999},"min":{"1min":0.55700000000000005,"5min":0.23599999999999999,"15min":0.215},"max":{"1min":1.165,"5min":3.6949999999999998,"15min":28.308},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.004,"5min":0.97999999999999998,"15min":1.2470000000000001},"min":{"1min":0.65300000000000002,"5min":0.32800000000000001,"15min":0.251},"max":{"1min":1.2729999999999999,"5min":3.7490000000000001,"15min":28.346},"last":1.1379999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.004,"15min":0.99299999999999999},"min":{"1min":0.61199999999999999,"5min":0.28499999999999998,"15min":0.22800000000000001},"max":{"1min":1.0269999999999999,"5min":5.71,"15min":11.430999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.93300000000000005,"15min":0.96299999999999997},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.139,"5min":3.8799999999999999,"15min":7.9710000000000001},"last":1.0029999999999999}]},{"osd":3,"last update":"Wed Oct 25 00:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.90000000000000002,"15min":1.357},"min":{"1min":0.40999999999999998,"5min":0.26800000000000002,"15min":0.26700000000000002},"max":{"1min":1.331,"5min":3.6150000000000002,"15min":93.664000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.999,"15min":1.389},"min":{"1min":0.51400000000000001,"5min":0.32700000000000001,"15min":0.25700000000000001},"max":{"1min":1.1599999999999999,"5min":4.6630000000000003,"15min":93.073999999999998},"last":1.0660000000000001}]},{"osd":4,"last update":"Wed Oct 25 00:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.91000000000000003,"15min":0.88300000000000001},"min":{"1min":0.35799999999999998,"5min":0.31900000000000001,"15min":0.249},"max":{"1min":1.1859999999999999,"5min":3.0569999999999999,"15min":7.6280000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.0760000000000001,"15min":1.151},"min":{"1min":0.57199999999999995,"5min":0.36599999999999999,"15min":0.27200000000000002},"max":{"1min":1.2609999999999999,"5min":4.9640000000000004,"15min":27.936},"last":0.753}]},{"osd":5,"last update":"Wed Oct 25 00:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.095,"15min":1.0389999999999999},"min":{"1min":0.47799999999999998,"5min":0.29299999999999998,"15min":0.26400000000000001},"max":{"1min":0.97299999999999998,"5min":5.8170000000000002,"15min":21.126999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.996,"15min":0.96099999999999997},"min":{"1min":0.67300000000000004,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":1.2649999999999999,"5min":3.8069999999999999,"15min":11.49},"last":1.0389999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:24:47 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.278,"15min":1.1180000000000001},"min":{"1min":0.435,"5min":0.35599999999999998,"15min":0.23599999999999999},"max":{"1min":1.236,"5min":4.8140000000000001,"15min":8.2959999999999994},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.3899999999999999,"15min":1.2350000000000001},"min":{"1min":0.46600000000000003,"5min":0.45500000000000002,"15min":0.35499999999999998},"max":{"1min":1.3149999999999999,"5min":4.8639999999999999,"15min":20.651},"last":1.111}]}]},{"osd":6,"up_from":13,"seq":55834575236,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859036,"kb_used_data":5076,"kb_used_omap":0,"kb_used_meta":853952,"kb_avail":93512804,"statfs":{"total":96636764160,"available":95757111296,"internally_reserved":0,"allocated":5197824,"data_stored":4751224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":874446848},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91200000000000003,"15min":0.96299999999999997},"min":{"1min":0.63900000000000001,"5min":0.185,"15min":0.185},"max":{"1min":1.2689999999999999,"5min":5.3490000000000002,"15min":15.974},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.018,"15min":0.99199999999999999},"min":{"1min":0.63300000000000001,"5min":0.218,"15min":0.218},"max":{"1min":1.2569999999999999,"5min":8.2400000000000002,"15min":8.4789999999999992},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84799999999999998,"15min":0.88700000000000001},"min":{"1min":0.66100000000000003,"5min":0.221,"15min":0.221},"max":{"1min":1.1799999999999999,"5min":3.5630000000000002,"15min":7.6360000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.92300000000000004,"15min":0.998},"min":{"1min":0.67000000000000004,"5min":0.254,"15min":0.249},"max":{"1min":1.115,"5min":3.5840000000000001,"15min":21.600999999999999},"last":0.90900000000000003}]},{"osd":2,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.94099999999999995,"15min":1.0680000000000001},"min":{"1min":0.66500000000000004,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.224,"5min":3.6779999999999999,"15min":12.615},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.96999999999999997,"15min":1.107},"min":{"1min":0.65500000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.1479999999999999,"5min":3.702,"15min":12.667999999999999},"last":1.127}]},{"osd":3,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.98999999999999999,"15min":1.2490000000000001},"min":{"1min":0.68899999999999995,"5min":0.23899999999999999,"15min":0.22600000000000001},"max":{"1min":1.2509999999999999,"5min":5.7350000000000003,"15min":38.649999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.044,"15min":1.323},"min":{"1min":0.77600000000000002,"5min":0.32300000000000001,"15min":0.30599999999999999},"max":{"1min":1.2490000000000001,"5min":5.766,"15min":38.683},"last":0.873}]},{"osd":4,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.268,"15min":1.123},"min":{"1min":0.53100000000000003,"5min":0.28399999999999997,"15min":0.23799999999999999},"max":{"1min":1.175,"5min":8.9930000000000003,"15min":12.682},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.026,"5min":1.1719999999999999,"15min":1.077},"min":{"1min":0.501,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.347,"5min":5.694,"15min":10.686},"last":1.256}]},{"osd":5,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":1.0700000000000001,"15min":0.96299999999999997},"min":{"1min":0.60099999999999998,"5min":0.28399999999999997,"15min":0.24299999999999999},"max":{"1min":1.306,"5min":3.6379999999999999,"15min":8.0039999999999996},"last":1.167},{"interface":"front","average":{"1min":1.115,"5min":1.2230000000000001,"15min":1.0740000000000001},"min":{"1min":0.72899999999999998,"5min":0.34499999999999997,"15min":0.30299999999999999},"max":{"1min":1.3320000000000001,"5min":4.0679999999999996,"15min":11.766999999999999},"last":1.2030000000000001}]},{"osd":7,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.21,"15min":1.0289999999999999},"min":{"1min":0.64100000000000001,"5min":0.27000000000000002,"15min":0.251},"max":{"1min":1.669,"5min":8.2720000000000002,"15min":12.926},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.266,"15min":1.0529999999999999},"min":{"1min":0.76500000000000001,"5min":0.376,"15min":0.28799999999999998},"max":{"1min":1.381,"5min":6.1100000000000003,"15min":7.6619999999999999},"last":1.2989999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575236,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":958888,"kb_used_data":9184,"kb_used_omap":0,"kb_used_meta":949696,"kb_avail":93412952,"statfs":{"total":96636764160,"available":95654862848,"internally_reserved":0,"allocated":9404416,"data_stored":8945710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":972488704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:24:52 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.1870000000000001,"15min":1.0409999999999999},"min":{"1min":0.38100000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.1839999999999999,"5min":14.837999999999999,"15min":14.837999999999999},"last":1.02},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.88700000000000001,"15min":0.97999999999999998},"min":{"1min":0.67300000000000004,"5min":0.20699999999999999,"15min":0.20100000000000001},"max":{"1min":1.125,"5min":4.016,"15min":7.423},"last":1.0469999999999999}]},{"osd":1,"last update":"Wed Oct 25 00:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.087,"15min":1.099},"min":{"1min":0.53900000000000003,"5min":0.246,"15min":0.23400000000000001},"max":{"1min":1.339,"5min":12.141,"15min":12.141},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.88900000000000001,"15min":1.0960000000000001},"min":{"1min":0.47299999999999998,"5min":0.26400000000000001,"15min":0.252},"max":{"1min":1.3149999999999999,"5min":4.1180000000000003,"15min":24.648},"last":1.1200000000000001}]},{"osd":2,"last update":"Wed Oct 25 00:24:39 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.177,"15min":1.2669999999999999},"min":{"1min":0.70399999999999996,"5min":0.27600000000000002,"15min":0.25},"max":{"1min":1.8220000000000001,"5min":12.175000000000001,"15min":20.760999999999999},"last":1.167},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.129,"15min":1.0860000000000001},"min":{"1min":0.77200000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6659999999999999,"5min":12.109999999999999,"15min":20.552},"last":0.82299999999999995}]},{"osd":3,"last update":"Wed Oct 25 00:24:39 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.2509999999999999,"15min":1.2110000000000001},"min":{"1min":0.57999999999999996,"5min":0.32100000000000001,"15min":0.28000000000000003},"max":{"1min":1.3400000000000001,"5min":12.013,"15min":12.013},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.98699999999999999,"15min":1.2470000000000001},"min":{"1min":0.63500000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.304,"5min":4.0490000000000004,"15min":12.805},"last":0.90100000000000002}]},{"osd":4,"last update":"Wed Oct 25 00:24:39 2023","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.393,"15min":1.212},"min":{"1min":0.624,"5min":0.27900000000000003,"15min":0.23999999999999999},"max":{"1min":1.6100000000000001,"5min":12.201000000000001,"15min":12.728},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.1639999999999999,"15min":1.135},"min":{"1min":0.63200000000000001,"5min":0.29299999999999998,"15min":0.27200000000000002},"max":{"1min":1.55,"5min":7.6470000000000002,"15min":10.943},"last":0.87}]},{"osd":6,"last update":"Wed Oct 25 00:24:04 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.1919999999999999,"15min":1.0609999999999999},"min":{"1min":0.78600000000000003,"5min":0.34399999999999997,"15min":0.26400000000000001},"max":{"1min":1.4079999999999999,"5min":9.6489999999999991,"15min":17.936},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":1.3680000000000001,"15min":1.216},"min":{"1min":0.71399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.387,"5min":13.029999999999999,"15min":14.766999999999999},"last":0.77800000000000002}]},{"osd":7,"last update":"Wed Oct 25 00:24:39 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.5509999999999999,"15min":1.2529999999999999},"min":{"1min":0.83599999999999997,"5min":0.33700000000000002,"15min":0.29499999999999998},"max":{"1min":1.839,"5min":28.178999999999998,"15min":28.178999999999998},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.7350000000000001,"15min":1.3839999999999999},"min":{"1min":0.749,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":1.974,"5min":28.222999999999999,"15min":28.222999999999999},"last":1.0900000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575235,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946344,"kb_used_data":19168,"kb_used_omap":0,"kb_used_meta":927168,"kb_avail":93425496,"statfs":{"total":96636764160,"available":95667707904,"internally_reserved":0,"allocated":19628032,"data_stored":19156396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":949420032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:24:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.93200000000000005,"15min":0.98899999999999999},"min":{"1min":0.57099999999999995,"5min":0.19700000000000001,"15min":0.187},"max":{"1min":1.1739999999999999,"5min":11.629,"15min":20.922999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.76300000000000001,"15min":0.96799999999999997},"min":{"1min":0.50700000000000001,"5min":0.25800000000000001,"15min":0.217},"max":{"1min":1.014,"5min":7.0830000000000002,"15min":20.532},"last":0.58599999999999997}]},{"osd":1,"last update":"Wed Oct 25 00:24:43 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.89000000000000001,"15min":0.98599999999999999},"min":{"1min":0.69899999999999995,"5min":0.22,"15min":0.22},"max":{"1min":1.3100000000000001,"5min":2.879,"15min":20.914000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.004,"5min":0.95599999999999996,"15min":1.1659999999999999},"min":{"1min":0.77500000000000002,"5min":0.26700000000000002,"15min":0.26100000000000001},"max":{"1min":1.22,"5min":6.1890000000000001,"15min":23.265999999999998},"last":0.88500000000000001}]},{"osd":2,"last update":"Wed Oct 25 00:24:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.90100000000000002,"15min":1.125},"min":{"1min":0.60299999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.175,"5min":7.1550000000000002,"15min":20.774000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.2390000000000001,"15min":1.2549999999999999},"min":{"1min":0.59199999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.1950000000000001,"5min":7.9009999999999998,"15min":20.817},"last":1.0329999999999999}]},{"osd":3,"last update":"Wed Oct 25 00:24:43 2023","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.2150000000000001,"15min":1.1799999999999999},"min":{"1min":0.63900000000000001,"5min":0.314,"15min":0.28799999999999998},"max":{"1min":1.258,"5min":7.3620000000000001,"15min":24.613},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.97399999999999998,"15min":1.153},"min":{"1min":0.77900000000000003,"5min":0.29299999999999998,"15min":0.28199999999999997},"max":{"1min":1.2809999999999999,"5min":7.2009999999999996,"15min":24.071000000000002},"last":1.0600000000000001}]},{"osd":5,"last update":"Wed Oct 25 00:24:43 2023","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.1599999999999999,"15min":0.91400000000000003},"min":{"1min":0.64200000000000002,"5min":0.28599999999999998,"15min":0.23999999999999999},"max":{"1min":1.1060000000000001,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.98199999999999998,"15min":0.92500000000000004},"min":{"1min":0.52700000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.2370000000000001,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.999}]},{"osd":6,"last update":"Wed Oct 25 00:24:43 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.1850000000000001,"15min":1.0629999999999999},"min":{"1min":0.60999999999999999,"5min":0.29299999999999998,"15min":0.27000000000000002},"max":{"1min":1.325,"5min":4.6550000000000002,"15min":6.5220000000000002},"last":1.115},{"interface":"front","average":{"1min":0.97099999999999997,"5min":1.0740000000000001,"15min":0.95199999999999996},"min":{"1min":0.68100000000000005,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":1.1799999999999999,"5min":7.2389999999999999,"15min":7.2389999999999999},"last":0.81999999999999995}]},{"osd":7,"last update":"Wed Oct 25 00:23:52 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":1.3009999999999999,"15min":1.0700000000000001},"min":{"1min":0.72599999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.1990000000000001,"5min":7.4020000000000001,"15min":7.9850000000000003},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.4910000000000001,"15min":1.163},"min":{"1min":0.98799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.2909999999999999,"5min":7.444,"15min":7.444},"last":1.1319999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575236,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1160764,"kb_used_data":9268,"kb_used_omap":0,"kb_used_meta":1151488,"kb_avail":93211076,"statfs":{"total":96636764160,"available":95448141824,"internally_reserved":0,"allocated":9490432,"data_stored":9030287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1179123712},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.505,"15min":0.79600000000000004},"min":{"1min":0.27300000000000002,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.88400000000000001,"5min":1.3899999999999999,"15min":13.327999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.61799999999999999,"15min":0.74399999999999999},"min":{"1min":0.34899999999999998,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.996,"5min":1.444,"15min":5.4580000000000002},"last":0.90000000000000002}]},{"osd":1,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.60999999999999999,"15min":0.877},"min":{"1min":0.375,"5min":0.20200000000000001,"15min":0.19900000000000001},"max":{"1min":0.90900000000000003,"5min":2.0739999999999998,"15min":13.831},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.747,"5min":0.69799999999999995,"15min":0.77600000000000002},"min":{"1min":0.38800000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.0609999999999999,"5min":2.9300000000000002,"15min":4.4500000000000002},"last":0.73899999999999999}]},{"osd":2,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.66300000000000003,"15min":0.88100000000000001},"min":{"1min":0.313,"5min":0.223,"15min":0.221},"max":{"1min":0.95799999999999996,"5min":2.7890000000000001,"15min":8.2520000000000007},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.72599999999999998,"15min":0.91800000000000004},"min":{"1min":0.33500000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0209999999999999,"5min":2.7360000000000002,"15min":6.6840000000000002},"last":0.78500000000000003}]},{"osd":4,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.874,"15min":1.218},"min":{"1min":0.53100000000000003,"5min":0.316,"15min":0.26800000000000002},"max":{"1min":1.149,"5min":3.7320000000000002,"15min":39.070999999999998},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.2010000000000001,"15min":1.2729999999999999},"min":{"1min":0.68999999999999995,"5min":0.28599999999999998,"15min":0.25700000000000001},"max":{"1min":1.2629999999999999,"5min":7.6180000000000003,"15min":55.104999999999997},"last":1.1739999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0069999999999999,"15min":1.1459999999999999},"min":{"1min":0.71099999999999997,"5min":0.312,"15min":0.28399999999999997},"max":{"1min":1.2949999999999999,"5min":4.1849999999999996,"15min":31.481999999999999},"last":1.042},{"interface":"front","average":{"1min":1.103,"5min":1.093,"15min":1.2370000000000001},"min":{"1min":0.73999999999999999,"5min":0.38300000000000001,"15min":0.32500000000000001},"max":{"1min":1.3799999999999999,"5min":4.2060000000000004,"15min":31.512},"last":1.113}]},{"osd":6,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.3149999999999999,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.311,"15min":0.27800000000000002},"max":{"1min":1.21,"5min":15.824,"15min":25.809999999999999},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.996,"5min":1.1419999999999999,"15min":1.1930000000000001},"min":{"1min":0.57799999999999996,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2649999999999999,"5min":4.7779999999999996,"15min":24.036999999999999},"last":1.0089999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:24:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.0289999999999999,"15min":1.2470000000000001},"min":{"1min":0.67500000000000004,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.3169999999999999,"5min":4.8529999999999998,"15min":27.581},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.085,"5min":1.1160000000000001,"15min":1.2629999999999999},"min":{"1min":0.57099999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.498,"5min":8.5259999999999998,"15min":24.332000000000001},"last":1.1439999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575236,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":946076,"kb_used_data":5076,"kb_used_omap":0,"kb_used_meta":940992,"kb_avail":93425764,"statfs":{"total":96636764160,"available":95667982336,"internally_reserved":0,"allocated":5197824,"data_stored":4750712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":963575808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:24:40 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.60999999999999999,"15min":1.0009999999999999},"min":{"1min":0.52900000000000003,"5min":0.16,"15min":0.16},"max":{"1min":0.99299999999999999,"5min":5.29,"15min":29.344000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.72199999999999998,"15min":0.84999999999999998},"min":{"1min":0.55700000000000005,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.88300000000000001,"5min":6.4290000000000003,"15min":10.755000000000001},"last":0.96799999999999997}]},{"osd":1,"last update":"Wed Oct 25 00:24:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.67400000000000004,"15min":0.94899999999999995},"min":{"1min":0.60699999999999998,"5min":0.19800000000000001,"15min":0.188},"max":{"1min":1.044,"5min":3.0409999999999999,"15min":18.617999999999999},"last":1.2050000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.73599999999999999,"15min":0.88900000000000001},"min":{"1min":0.61199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.0800000000000001,"5min":3.649,"15min":15.064},"last":0.64200000000000002}]},{"osd":3,"last update":"Wed Oct 25 00:24:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.67500000000000004,"15min":0.90800000000000003},"min":{"1min":0.47799999999999998,"5min":0.192,"15min":0.192},"max":{"1min":0.91500000000000004,"5min":2.9220000000000002,"15min":13.034000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.81299999999999994,"15min":1.0229999999999999},"min":{"1min":0.67700000000000005,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.1479999999999999,"5min":3.1339999999999999,"15min":18.547999999999998},"last":0.89300000000000002}]},{"osd":4,"last update":"Wed Oct 25 00:24:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.0109999999999999,"15min":1.2030000000000001},"min":{"1min":0.82899999999999996,"5min":0.252,"15min":0.252},"max":{"1min":1.26,"5min":5.1070000000000002,"15min":33.378999999999998},"last":1.107},{"interface":"front","average":{"1min":1.091,"5min":1.0840000000000001,"15min":1.294},"min":{"1min":0.90800000000000003,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.2310000000000001,"5min":5.2290000000000001,"15min":33.558999999999997},"last":1.5529999999999999}]},{"osd":5,"last update":"Wed Oct 25 00:24:40 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.1020000000000001,"15min":1.484},"min":{"1min":0.75900000000000001,"5min":0.32000000000000001,"15min":0.29099999999999998},"max":{"1min":1.4850000000000001,"5min":6.3979999999999997,"15min":34.545999999999999},"last":1.131},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.97999999999999998,"15min":1.1850000000000001},"min":{"1min":0.85699999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.337,"5min":4.4829999999999997,"15min":36.104999999999997},"last":1.5960000000000001}]},{"osd":6,"last update":"Wed Oct 25 00:24:48 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.0189999999999999,"15min":1.1200000000000001},"min":{"1min":0.76700000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":1.262,"5min":3.5739999999999998,"15min":33.661999999999999},"last":1.49},{"interface":"front","average":{"1min":1.075,"5min":1.0900000000000001,"15min":1.1990000000000001},"min":{"1min":0.78700000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.633,"5min":3.5979999999999999,"15min":34.043999999999997},"last":1.0669999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:24:40 2023","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.0720000000000001,"15min":1.248},"min":{"1min":0.81100000000000005,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.51,"5min":4.8369999999999997,"15min":33.307000000000002},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.1499999999999999,"15min":1.3149999999999999},"min":{"1min":1.0029999999999999,"5min":0.371,"15min":0.371},"max":{"1min":1.5780000000000001,"5min":4.8620000000000001,"15min":33.344999999999999},"last":1.153}]}]},{"osd":1,"up_from":13,"seq":55834575237,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1054180,"kb_used_data":4124,"kb_used_omap":0,"kb_used_meta":1050048,"kb_avail":93317660,"statfs":{"total":96636764160,"available":95557283840,"internally_reserved":0,"allocated":4222976,"data_stored":3775285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1075249152},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.59099999999999997,"15min":0.82699999999999996},"min":{"1min":0.40799999999999997,"5min":0.14299999999999999,"15min":0.14299999999999999},"max":{"1min":1.0129999999999999,"5min":3.6110000000000002,"15min":21.334},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.71599999999999997,"15min":0.85899999999999999},"min":{"1min":0.26200000000000001,"5min":0.19800000000000001,"15min":0.19},"max":{"1min":1.0549999999999999,"5min":3.4809999999999999,"15min":21.276},"last":0.69699999999999995}]},{"osd":2,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.67400000000000004,"15min":0.81000000000000005},"min":{"1min":0.48599999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.95199999999999996,"5min":8.1600000000000001,"15min":11.420999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.746,"5min":0.68400000000000005,"15min":0.83999999999999997},"min":{"1min":0.51600000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.0009999999999999,"5min":3.1760000000000002,"15min":11.648},"last":0.625}]},{"osd":3,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78600000000000003,"15min":0.92500000000000004},"min":{"1min":0.58699999999999997,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":1.1299999999999999,"5min":3.46,"15min":14.346},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.88800000000000001,"15min":1.026},"min":{"1min":0.71099999999999997,"5min":0.26300000000000001,"15min":0.248},"max":{"1min":1.2090000000000001,"5min":3.4969999999999999,"15min":14.379},"last":0.57299999999999995}]},{"osd":4,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.038,"15min":1.0589999999999999},"min":{"1min":0.51000000000000001,"5min":0.28000000000000003,"15min":0.27200000000000002},"max":{"1min":1.093,"5min":5.2409999999999997,"15min":10.582000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.038,"5min":1.0800000000000001,"15min":1.0580000000000001},"min":{"1min":0.82899999999999996,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":1.246,"5min":4.6630000000000003,"15min":11.173},"last":0.93500000000000005}]},{"osd":5,"last update":"Wed Oct 25 00:23:59 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.92400000000000004,"15min":1.0529999999999999},"min":{"1min":0.79500000000000004,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.228,"5min":5.0309999999999997,"15min":11.577999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.1120000000000001,"5min":0.92600000000000005,"15min":0.97799999999999998},"min":{"1min":0.69699999999999995,"5min":0.29899999999999999,"15min":0.29399999999999998},"max":{"1min":1.3480000000000001,"5min":2.7469999999999999,"15min":10.018000000000001},"last":0.99099999999999999}]},{"osd":6,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.999,"15min":1.028},"min":{"1min":0.71999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.1739999999999999,"5min":7.4880000000000004,"15min":11.845000000000001},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.095,"5min":1.0860000000000001,"15min":1.125},"min":{"1min":0.77100000000000002,"5min":0.38300000000000001,"15min":0.33800000000000002},"max":{"1min":1.262,"5min":7.5220000000000002,"15min":11.919},"last":1.0169999999999999}]},{"osd":7,"last update":"Wed Oct 25 00:23:55 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.1839999999999999,"15min":1.1279999999999999},"min":{"1min":0.45000000000000001,"5min":0.36599999999999999,"15min":0.35599999999999998},"max":{"1min":1.2749999999999999,"5min":7.9909999999999997,"15min":11.881},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.95799999999999996,"15min":1.1439999999999999},"min":{"1min":0.56899999999999995,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.4610000000000001,"5min":4.2300000000000004,"15min":10.478},"last":1.0469999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575237,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":966392,"kb_used_data":22128,"kb_used_omap":0,"kb_used_meta":944256,"kb_avail":93405448,"statfs":{"total":96636764160,"available":95647178752,"internally_reserved":0,"allocated":22659072,"data_stored":22203488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":966918144},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Oct 25 00:24:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59299999999999997,"15min":1.006},"min":{"1min":0.34899999999999998,"5min":0.18099999999999999,"15min":0.16900000000000001},"max":{"1min":1.006,"5min":2.3050000000000002,"15min":30.065999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.63200000000000001,"15min":1.0049999999999999},"min":{"1min":0.38800000000000001,"5min":0.184,"15min":0.182},"max":{"1min":0.97199999999999998,"5min":3.0030000000000001,"15min":19.757999999999999},"last":0.70699999999999996}]},{"osd":2,"last update":"Wed Oct 25 00:24:50 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68999999999999995,"15min":0.80300000000000005},"min":{"1min":0.39800000000000002,"5min":0.22900000000000001,"15min":0.20399999999999999},"max":{"1min":1.1819999999999999,"5min":5.9740000000000002,"15min":15.324999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.72399999999999998,"15min":0.97599999999999998},"min":{"1min":0.46700000000000003,"5min":0.247,"15min":0.23999999999999999},"max":{"1min":1.256,"5min":1.8440000000000001,"15min":30.283999999999999},"last":0.89600000000000002}]},{"osd":3,"last update":"Wed Oct 25 00:24:50 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.85799999999999998,"15min":1.095},"min":{"1min":0.35799999999999998,"5min":0.20999999999999999,"15min":0.191},"max":{"1min":0.89700000000000002,"5min":6.3570000000000002,"15min":22.966999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.82499999999999996,"15min":1.095},"min":{"1min":0.41099999999999998,"5min":0.29499999999999998,"15min":0.23599999999999999},"max":{"1min":1.169,"5min":2.5619999999999998,"15min":19.460999999999999},"last":0.86699999999999999}]},{"osd":4,"last update":"Wed Oct 25 00:24:50 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.90100000000000002,"15min":1.123},"min":{"1min":0.64100000000000001,"5min":0.33700000000000002,"15min":0.26600000000000001},"max":{"1min":1.337,"5min":3.758,"15min":8.3670000000000009},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0640000000000001,"15min":1.246},"min":{"1min":0.80000000000000004,"5min":0.29499999999999998,"15min":0.28799999999999998},"max":{"1min":1.3779999999999999,"5min":5.3719999999999999,"15min":13.023999999999999},"last":1.145}]},{"osd":5,"last update":"Wed Oct 25 00:24:50 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.0049999999999999,"15min":1.0329999999999999},"min":{"1min":0.60399999999999998,"5min":0.30399999999999999,"15min":0.29099999999999998},"max":{"1min":1.2,"5min":2.9710000000000001,"15min":10.683999999999999},"last":0.751},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.073,"15min":1.0049999999999999},"min":{"1min":0.60099999999999998,"5min":0.34999999999999998,"15min":0.29899999999999999},"max":{"1min":1.4019999999999999,"5min":4.8860000000000001,"15min":15.436999999999999},"last":0.96299999999999997}]},{"osd":6,"last update":"Wed Oct 25 00:24:50 2023","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.0309999999999999,"15min":1.1439999999999999},"min":{"1min":0.623,"5min":0.35999999999999999,"15min":0.28100000000000003},"max":{"1min":1.3340000000000001,"5min":6.1559999999999997,"15min":8.7260000000000009},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.1950000000000001,"15min":1.3100000000000001},"min":{"1min":0.66600000000000004,"5min":0.38200000000000001,"15min":0.33600000000000002},"max":{"1min":1.3540000000000001,"5min":7.8970000000000002,"15min":15.375},"last":1.071}]},{"osd":7,"last update":"Wed Oct 25 00:24:50 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.2090000000000001,"15min":1.169},"min":{"1min":0.65500000000000003,"5min":0.36399999999999999,"15min":0.33700000000000002},"max":{"1min":1.3089999999999999,"5min":5.5090000000000003,"15min":10.81},"last":0.79800000000000004},{"interface":"front","average":{"1min":1.091,"5min":1.04,"15min":1.27},"min":{"1min":0.75800000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.339,"5min":4.2629999999999999,"15min":11.417},"last":0.92900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12673024,"data_stored":12668140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3219456,"data_stored":3218857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11710464,"data_stored":11692839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4218880,"data_stored":4195192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":8388642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4279555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6918144,"data_stored":6907219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4202496,"data_stored":4194416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6836224,"data_stored":6822812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-25T00:25:00.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-25T00:25:00.799 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f3621521640 1 Processor -- start 2023-10-25T00:25:00.800 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f3621521640 1 -- start start 2023-10-25T00:25:00.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f3621521640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f361c0ab1a0 0x7f361c0c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:00.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f3621521640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f361c072720 0x7f361c0bfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:00.801 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f3621521640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c0c1ce0 con 0x7f361c0bceb0 2023-10-25T00:25:00.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f3621521640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361c0be1d0 con 0x7f361c0ab1a0 2023-10-25T00:25:00.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f3621521640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361c0c0880 con 0x7f361c072720 2023-10-25T00:25:00.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f361affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f361c0ab1a0 0x7f361c0c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:00.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f361affd640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f361c0ab1a0 0x7f361c0c0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48752/0 (socket says 172.21.15.107:48752) 2023-10-25T00:25:00.802 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f361affd640 1 -- 172.21.15.107:0/2622458455 learned_addr learned my addr 172.21.15.107:0/2622458455 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:00.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f361a7fc640 1 --2- 172.21.15.107:0/2622458455 >> v2:172.21.15.110:3300/0 conn(0x7f361c072720 0x7f361c0bfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:00.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f361affd640 1 -- 172.21.15.107:0/2622458455 >> v2:172.21.15.110:3300/0 conn(0x7f361c072720 msgr2=0x7f361c0bfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:00.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f361affd640 1 --2- 172.21.15.107:0/2622458455 >> v2:172.21.15.110:3300/0 conn(0x7f361c072720 0x7f361c0bfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f361affd640 1 -- 172.21.15.107:0/2622458455 >> v1:172.21.15.110:6789/0 conn(0x7f361c0bceb0 legacy=0x7f361c0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:00.803 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f361affd640 1 -- 172.21.15.107:0/2622458455 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c0c1f30 con 0x7f361c0ab1a0 2023-10-25T00:25:00.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f361affd640 1 --2- 172.21.15.107:0/2622458455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f361c0ab1a0 0x7f361c0c0d00 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f361c071e30 tx=0x7f360c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726f064be4527398 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:00.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f3619ffb640 1 -- 172.21.15.107:0/2622458455 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c0ab600 con 0x7f361c0ab1a0 2023-10-25T00:25:00.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f3619ffb640 1 -- 172.21.15.107:0/2622458455 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360c0a7050 con 0x7f361c0ab1a0 2023-10-25T00:25:00.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.797+0000 7f3619ffb640 1 -- 172.21.15.107:0/2622458455 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c0ab980 con 0x7f361c0ab1a0 2023-10-25T00:25:00.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 -- 172.21.15.107:0/2622458455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f361c0ab1a0 msgr2=0x7f361c0c0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:00.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 --2- 172.21.15.107:0/2622458455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f361c0ab1a0 0x7f361c0c0d00 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f361c071e30 tx=0x7f360c01c400 comp rx=0 tx=0).stop 2023-10-25T00:25:00.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 -- 172.21.15.107:0/2622458455 shutdown_connections 2023-10-25T00:25:00.806 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 --2- 172.21.15.107:0/2622458455 >> v2:172.21.15.110:3300/0 conn(0x7f361c072720 0x7f361c0bfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 --2- 172.21.15.107:0/2622458455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f361c0ab1a0 0x7f361c0c0d00 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 -- 172.21.15.107:0/2622458455 >> 172.21.15.107:0/2622458455 conn(0x7f361c0c2860 msgr2=0x7f361c0c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:00.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 -- 172.21.15.107:0/2622458455 shutdown_connections 2023-10-25T00:25:00.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 -- 172.21.15.107:0/2622458455 wait complete. 2023-10-25T00:25:00.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 Processor -- start 2023-10-25T00:25:00.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 -- start start 2023-10-25T00:25:00.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f361c0ab1a0 0x7f361c14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:00.810 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f361c0bceb0 0x7f361c157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:00.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f361c157ec0 0x7f361c14c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:00.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c0aa780 con 0x7f361c157ec0 2023-10-25T00:25:00.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c0aa510 con 0x7f361c0bceb0 2023-10-25T00:25:00.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361c150b80 con 0x7f361c0ab1a0 2023-10-25T00:25:00.811 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f361c157ec0 0x7f361c14c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:00.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f361c157ec0 0x7f361c14c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48758/0 (socket says 172.21.15.107:48758) 2023-10-25T00:25:00.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361b7fe640 1 -- 172.21.15.107:0/2366381102 learned_addr learned my addr 172.21.15.107:0/2366381102 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:00.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361affd640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f361c0ab1a0 0x7f361c14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:00.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361a7fc640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f361c0bceb0 0x7f361c157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:00.812 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361b7fe640 1 -- 172.21.15.107:0/2366381102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f361c0ab1a0 msgr2=0x7f361c14fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:00.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361b7fe640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f361c0ab1a0 0x7f361c14fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361b7fe640 1 -- 172.21.15.107:0/2366381102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f361c0bceb0 msgr2=0x7f361c157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:00.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361b7fe640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f361c0bceb0 0x7f361c157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361b7fe640 1 -- 172.21.15.107:0/2366381102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f360c0094d0 con 0x7f361c157ec0 2023-10-25T00:25:00.813 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361affd640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f361c0ab1a0 0x7f361c14fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:00.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f361b7fe640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f361c157ec0 0x7f361c14c620 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f361400b9e0 tx=0x7f361400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:00.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f36037fe640 1 -- 172.21.15.107:0/2366381102 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3614003b80 con 0x7f361c157ec0 2023-10-25T00:25:00.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f36037fe640 1 -- 172.21.15.107:0/2366381102 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3614003ce0 con 0x7f361c157ec0 2023-10-25T00:25:00.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 -- 172.21.15.107:0/2366381102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361c151600 con 0x7f361c157ec0 2023-10-25T00:25:00.814 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f36037fe640 1 -- 172.21.15.107:0/2366381102 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361400cb60 con 0x7f361c157ec0 2023-10-25T00:25:00.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.801+0000 7f3621521640 1 -- 172.21.15.107:0/2366381102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361c1510e0 con 0x7f361c157ec0 2023-10-25T00:25:00.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.805+0000 7f3621521640 1 -- 172.21.15.107:0/2366381102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361c0beff0 con 0x7f361c157ec0 2023-10-25T00:25:00.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.805+0000 7f36037fe640 1 -- 172.21.15.107:0/2366381102 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f361400ccc0 con 0x7f361c157ec0 2023-10-25T00:25:00.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.805+0000 7f36037fe640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3608065e60 0x7f3608068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:00.815 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.805+0000 7f361affd640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3608065e60 0x7f3608068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:00.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.805+0000 7f36037fe640 1 -- 172.21.15.107:0/2366381102 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7f3614084870 con 0x7f361c157ec0 2023-10-25T00:25:00.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.805+0000 7f361affd640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3608065e60 0x7f3608068320 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f361c0abdf0 tx=0x7f360c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:00.816 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.809+0000 7f36037fe640 1 -- 172.21.15.107:0/2366381102 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f3614050ba0 con 0x7f361c157ec0 2023-10-25T00:25:00.968 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.965+0000 7f3621521640 1 -- 172.21.15.107:0/2366381102 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f361c04e130 con 0x7f361c157ec0 2023-10-25T00:25:00.973 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.969+0000 7f36037fe640 1 -- 172.21.15.107:0/2366381102 <== mon.0 v2:172.21.15.107:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f3614009880 con 0x7f361c157ec0 2023-10-25T00:25:00.974 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.969+0000 7f36037fe640 1 -- 172.21.15.107:0/2366381102 <== mon.0 v2:172.21.15.107:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7f36140541c0 con 0x7f361c157ec0 2023-10-25T00:25:00.976 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 -- 172.21.15.107:0/2366381102 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3608065e60 msgr2=0x7f3608068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:00.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3608065e60 0x7f3608068320 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f361c0abdf0 tx=0x7f360c008040 comp rx=0 tx=0).stop 2023-10-25T00:25:00.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 -- 172.21.15.107:0/2366381102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f361c157ec0 msgr2=0x7f361c14c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:00.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f361c157ec0 0x7f361c14c620 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f361400b9e0 tx=0x7f361400bea0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 -- 172.21.15.107:0/2366381102 shutdown_connections 2023-10-25T00:25:00.977 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f361c157ec0 0x7f361c14c620 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f361c0bceb0 0x7f361c157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f3608065e60 0x7f3608068320 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 --2- 172.21.15.107:0/2366381102 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f361c0ab1a0 0x7f361c14fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:00.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 -- 172.21.15.107:0/2366381102 >> 172.21.15.107:0/2366381102 conn(0x7f361c0c2860 msgr2=0x7f361c06f600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:00.978 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 -- 172.21.15.107:0/2366381102 shutdown_connections 2023-10-25T00:25:00.979 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:00.973+0000 7f3621521640 1 -- 172.21.15.107:0/2366381102 wait complete. 2023-10-25T00:25:00.993 INFO:tasks.cephfs.filesystem:Destroying file system a and related pools 2023-10-25T00:25:00.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:25:01.099 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.093+0000 7f81064fd640 1 Processor -- start 2023-10-25T00:25:01.100 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f81064fd640 1 -- start start 2023-10-25T00:25:01.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f81064fd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8100151d60 0x7f81000a9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f81064fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f810006a780 0x7f8100154d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.101 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f81064fd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8100154840 con 0x7f81000c24b0 2023-10-25T00:25:01.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f81064fd640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81000aadc0 con 0x7f8100151d60 2023-10-25T00:25:01.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f81064fd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810006afe0 con 0x7f810006a780 2023-10-25T00:25:01.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f80f7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f810006a780 0x7f8100154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.102 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f80fffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8100151d60 0x7f81000a9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f8104a9a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f81000c24b0 0x7f81000bf280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.107:45798/0 (socket says 172.21.15.107:45798) 2023-10-25T00:25:01.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f8104a9a640 1 -- 172.21.15.107:0/2219359279 learned_addr learned my addr 172.21.15.107:0/2219359279 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:25:01.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f80f7fff640 1 -- 172.21.15.107:0/2219359279 >> v2:172.21.15.110:3300/0 conn(0x7f8100151d60 msgr2=0x7f81000a9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:01.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f80f7fff640 1 --2- 172.21.15.107:0/2219359279 >> v2:172.21.15.110:3300/0 conn(0x7f8100151d60 0x7f81000a9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.103 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f80f7fff640 1 -- 172.21.15.107:0/2219359279 >> v1:172.21.15.110:6789/0 conn(0x7f81000c24b0 legacy=0x7f81000bf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f80f7fff640 1 -- 172.21.15.107:0/2219359279 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81000aa640 con 0x7f810006a780 2023-10-25T00:25:01.104 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.097+0000 7f80fffff640 1 --2- 172.21.15.107:0/2219359279 >> v2:172.21.15.110:3300/0 conn(0x7f8100151d60 0x7f81000a9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:25:01.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80f7fff640 1 --2- 172.21.15.107:0/2219359279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f810006a780 0x7f8100154d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f80e80097c0 tx=0x7f80e801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21ce2b4045bae679 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:01.105 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80ff7fe640 1 -- 172.21.15.107:0/2219359279 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e80ab600 con 0x7f810006a780 2023-10-25T00:25:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80ff7fe640 1 -- 172.21.15.107:0/2219359279 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f80e80a7050 con 0x7f810006a780 2023-10-25T00:25:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80ff7fe640 1 -- 172.21.15.107:0/2219359279 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e80ab9a0 con 0x7f810006a780 2023-10-25T00:25:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- 172.21.15.107:0/2219359279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f810006a780 msgr2=0x7f8100154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 --2- 172.21.15.107:0/2219359279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f810006a780 0x7f8100154d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f80e80097c0 tx=0x7f80e801c480 comp rx=0 tx=0).stop 2023-10-25T00:25:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- 172.21.15.107:0/2219359279 shutdown_connections 2023-10-25T00:25:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 --2- 172.21.15.107:0/2219359279 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f810006a780 0x7f8100154d40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 --2- 172.21.15.107:0/2219359279 >> v2:172.21.15.110:3300/0 conn(0x7f8100151d60 0x7f81000a9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- 172.21.15.107:0/2219359279 >> 172.21.15.107:0/2219359279 conn(0x7f81000c2950 msgr2=0x7f81000c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:01.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- 172.21.15.107:0/2219359279 shutdown_connections 2023-10-25T00:25:01.108 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- 172.21.15.107:0/2219359279 wait complete. 2023-10-25T00:25:01.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 Processor -- start 2023-10-25T00:25:01.110 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- start start 2023-10-25T00:25:01.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81000c24b0 0x7f8100156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8100151d60 0x7f81000c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81000715d0 0x7f8100071a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100070f90 con 0x7f81000715d0 2023-10-25T00:25:01.111 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100156950 con 0x7f81000c24b0 2023-10-25T00:25:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8100157320 con 0x7f8100151d60 2023-10-25T00:25:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f8104a9a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81000715d0 0x7f8100071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f8104a9a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81000715d0 0x7f8100071a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48764/0 (socket says 172.21.15.107:48764) 2023-10-25T00:25:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f8104a9a640 1 -- 172.21.15.107:0/2259512454 learned_addr learned my addr 172.21.15.107:0/2259512454 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80fffff640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81000c24b0 0x7f8100156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80f7fff640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8100151d60 0x7f81000c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f8104a9a640 1 -- 172.21.15.107:0/2259512454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8100151d60 msgr2=0x7f81000c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f8104a9a640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8100151d60 0x7f81000c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f8104a9a640 1 -- 172.21.15.107:0/2259512454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81000c24b0 msgr2=0x7f8100156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.113 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f8104a9a640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81000c24b0 0x7f8100156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f8104a9a640 1 -- 172.21.15.107:0/2259512454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f0009420 con 0x7f81000715d0 2023-10-25T00:25:01.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f8104a9a640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81000715d0 0x7f8100071a80 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f80f800cb20 tx=0x7f80f800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:01.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80fd7fa640 1 -- 172.21.15.107:0/2259512454 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f800ac40 con 0x7f81000715d0 2023-10-25T00:25:01.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- 172.21.15.107:0/2259512454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e8009420 con 0x7f81000715d0 2023-10-25T00:25:01.114 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80fd7fa640 1 -- 172.21.15.107:0/2259512454 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f80f800ada0 con 0x7f81000715d0 2023-10-25T00:25:01.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80fd7fa640 1 -- 172.21.15.107:0/2259512454 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8012600 con 0x7f81000715d0 2023-10-25T00:25:01.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- 172.21.15.107:0/2259512454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810006fe00 con 0x7f81000715d0 2023-10-25T00:25:01.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f81064fd640 1 -- 172.21.15.107:0/2259512454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81000746e0 con 0x7f81000715d0 2023-10-25T00:25:01.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80fd7fa640 1 -- 172.21.15.107:0/2259512454 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f80f8007070 con 0x7f81000715d0 2023-10-25T00:25:01.115 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80fd7fa640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80e0065e60 0x7f80e0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80fd7fa640 1 -- 172.21.15.107:0/2259512454 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7f80f80849a0 con 0x7f81000715d0 2023-10-25T00:25:01.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80fffff640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80e0065e60 0x7f80e0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.105+0000 7f80fffff640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80e0065e60 0x7f80e0068320 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009be0 tx=0x7f80f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:01.116 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.109+0000 7f80fd7fa640 1 -- 172.21.15.107:0/2259512454 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f80f8050d00 con 0x7f81000715d0 2023-10-25T00:25:01.302 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.301+0000 7f81064fd640 1 -- 172.21.15.107:0/2259512454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81000a9520 con 0x7f81000715d0 2023-10-25T00:25:01.303 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.301+0000 7f80fd7fa640 1 -- 172.21.15.107:0/2259512454 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+5769 (secure 0 0 0) 0x7f80f8017020 con 0x7f81000715d0 2023-10-25T00:25:01.303 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:25:01.304 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9606,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2839787144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2839787144},{"type":"v1","addr":"172.21.15.107:6837","nonce":2839787144}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":240}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:24:38.446814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"c","rank":0,"incarnation":236,"state":"up:active","state_seq":27,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:25:01.304 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 240 2023-10-25T00:25:01.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 -- 172.21.15.107:0/2259512454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80e0065e60 msgr2=0x7f80e0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.305 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80e0065e60 0x7f80e0068320 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009be0 tx=0x7f80f0008040 comp rx=0 tx=0).stop 2023-10-25T00:25:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 -- 172.21.15.107:0/2259512454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81000715d0 msgr2=0x7f8100071a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81000715d0 0x7f8100071a80 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f80f800cb20 tx=0x7f80f800a060 comp rx=0 tx=0).stop 2023-10-25T00:25:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 -- 172.21.15.107:0/2259512454 shutdown_connections 2023-10-25T00:25:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f81000715d0 0x7f8100071a80 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.306 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f80e0065e60 0x7f80e0068320 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8100151d60 0x7f81000c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 --2- 172.21.15.107:0/2259512454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81000c24b0 0x7f8100156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 -- 172.21.15.107:0/2259512454 >> 172.21.15.107:0/2259512454 conn(0x7f81000c2950 msgr2=0x7f81000bf280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 -- 172.21.15.107:0/2259512454 shutdown_connections 2023-10-25T00:25:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.305+0000 7f81064fd640 1 -- 172.21.15.107:0/2259512454 wait complete. 2023-10-25T00:25:01.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:25:01.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.425+0000 7fbb69754640 1 Processor -- start 2023-10-25T00:25:01.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.425+0000 7fbb69754640 1 -- start start 2023-10-25T00:25:01.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb69754640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb64152040 0x7fbb640c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb69754640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb64155680 con 0x7fbb640c8100 2023-10-25T00:25:01.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb69754640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb64154a70 con 0x7fbb6406a4b0 2023-10-25T00:25:01.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb69754640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb64154d50 con 0x7fbb64152040 2023-10-25T00:25:01.430 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb5a7fc640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbb640c8100 0x7fbb640abd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34420/0 (socket says 172.21.15.107:34420) 2023-10-25T00:25:01.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb5a7fc640 1 -- 172.21.15.107:0/1626536633 learned_addr learned my addr 172.21.15.107:0/1626536633 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:25:01.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211358882 0 0) 0x7fbb64155680 con 0x7fbb640c8100 2023-10-25T00:25:01.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb62ffd640 1 --2- 172.21.15.107:0/1626536633 >> v2:172.21.15.110:3300/0 conn(0x7fbb64152040 0x7fbb640c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50003530 con 0x7fbb640c8100 2023-10-25T00:25:01.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1662084176 0 0) 0x7fbb50003530 con 0x7fbb640c8100 2023-10-25T00:25:01.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 >> v2:172.21.15.110:3300/0 conn(0x7fbb64152040 msgr2=0x7fbb640c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:01.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 --2- 172.21.15.107:0/1626536633 >> v2:172.21.15.110:3300/0 conn(0x7fbb64152040 0x7fbb640c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 >> v1:172.21.15.110:6789/0 conn(0x7fbb6406a4b0 legacy=0x7fbb640ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb64155030 con 0x7fbb640c8100 2023-10-25T00:25:01.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fbb4c001680 con 0x7fbb640c8100 2023-10-25T00:25:01.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 >> v1:172.21.15.107:6789/0 conn(0x7fbb640c8100 legacy=0x7fbb640abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb50003980 0x7fbb50003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb500047d0 0x7fbb50004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb500052b0 0x7fbb5000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb64154d50 con 0x7fbb500047d0 2023-10-25T00:25:01.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5000c180 con 0x7fbb500052b0 2023-10-25T00:25:01.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5000c340 con 0x7fbb50003980 2023-10-25T00:25:01.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb5a7fc640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb500047d0 0x7fbb50004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb62ffd640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb50003980 0x7fbb50003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb637fe640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb500052b0 0x7fbb5000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb5a7fc640 1 -- 172.21.15.107:0/1626536633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb50003980 msgr2=0x7fbb50003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:01.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb5a7fc640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb50003980 0x7fbb50003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb5a7fc640 1 -- 172.21.15.107:0/1626536633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb500052b0 msgr2=0x7fbb5000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:01.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb5a7fc640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb500052b0 0x7fbb5000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb5a7fc640 1 -- 172.21.15.107:0/1626536633 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb50003530 con 0x7fbb500047d0 2023-10-25T00:25:01.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb637fe640 1 -- 172.21.15.107:0/1626536633 reap_dead start 2023-10-25T00:25:01.436 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.429+0000 7fbb62ffd640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb50003980 0x7fbb50003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:01.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5a7fc640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb500047d0 0x7fbb50004c50 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c001ec0 tx=0x7fbb4c01cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72d62320c61b7b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:01.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0058e0 con 0x7fbb500047d0 2023-10-25T00:25:01.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbb4c005a40 con 0x7fbb500047d0 2023-10-25T00:25:01.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb627fc640 1 -- 172.21.15.107:0/1626536633 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c005c90 con 0x7fbb500047d0 2023-10-25T00:25:01.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 -- 172.21.15.107:0/1626536633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb500047d0 msgr2=0x7fbb50004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb500047d0 0x7fbb50004c50 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c001ec0 tx=0x7fbb4c01cba0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 -- 172.21.15.107:0/1626536633 shutdown_connections 2023-10-25T00:25:01.440 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 --2- 172.21.15.107:0/1626536633 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb500047d0 0x7fbb50004c50 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 -- 172.21.15.107:0/1626536633 >> 172.21.15.107:0/1626536633 conn(0x7fbb640ca200 msgr2=0x7fbb640ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:01.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 -- 172.21.15.107:0/1626536633 shutdown_connections 2023-10-25T00:25:01.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 -- 172.21.15.107:0/1626536633 wait complete. 2023-10-25T00:25:01.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 Processor -- start 2023-10-25T00:25:01.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 -- start start 2023-10-25T00:25:01.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb500047d0 0x7fbb640ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb640ae5f0 0x7fbb641efcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb641ed960 0x7fbb641f0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb641e06f0 con 0x7fbb640ae5f0 2023-10-25T00:25:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb641e0150 con 0x7fbb500047d0 2023-10-25T00:25:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb641e0410 con 0x7fbb641ed960 2023-10-25T00:25:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb640ae5f0 0x7fbb641efcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb640ae5f0 0x7fbb641efcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48772/0 (socket says 172.21.15.107:48772) 2023-10-25T00:25:01.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5a7fc640 1 -- 172.21.15.107:0/705085393 learned_addr learned my addr 172.21.15.107:0/705085393 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:01.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb637fe640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb641ed960 0x7fbb641f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb62ffd640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb500047d0 0x7fbb640ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5a7fc640 1 -- 172.21.15.107:0/705085393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb641ed960 msgr2=0x7fbb641f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5a7fc640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb641ed960 0x7fbb641f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5a7fc640 1 -- 172.21.15.107:0/705085393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb500047d0 msgr2=0x7fbb640ae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5a7fc640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb500047d0 0x7fbb640ae0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5a7fc640 1 -- 172.21.15.107:0/705085393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb640adbb0 con 0x7fbb640ae5f0 2023-10-25T00:25:01.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb637fe640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb641ed960 0x7fbb641f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:01.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb62ffd640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb500047d0 0x7fbb640ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:01.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5a7fc640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb640ae5f0 0x7fbb641efcc0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c001ec0 tx=0x7fbb50005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:01.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5bfff640 1 -- 172.21.15.107:0/705085393 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0047f0 con 0x7fbb640ae5f0 2023-10-25T00:25:01.448 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5bfff640 1 -- 172.21.15.107:0/705085393 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbb4c004950 con 0x7fbb640ae5f0 2023-10-25T00:25:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 -- 172.21.15.107:0/705085393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb641f0740 con 0x7fbb640ae5f0 2023-10-25T00:25:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb5bfff640 1 -- 172.21.15.107:0/705085393 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c004ce0 con 0x7fbb640ae5f0 2023-10-25T00:25:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.433+0000 7fbb69754640 1 -- 172.21.15.107:0/705085393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb641ec4f0 con 0x7fbb640ae5f0 2023-10-25T00:25:01.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.437+0000 7fbb69754640 1 -- 172.21.15.107:0/705085393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb640ab350 con 0x7fbb640ae5f0 2023-10-25T00:25:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.437+0000 7fbb5bfff640 1 -- 172.21.15.107:0/705085393 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbb4c0a5790 con 0x7fbb640ae5f0 2023-10-25T00:25:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.437+0000 7fbb5bfff640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb44065e60 0x7fbb44068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.437+0000 7fbb5bfff640 1 -- 172.21.15.107:0/705085393 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7fbb4c11e630 con 0x7fbb640ae5f0 2023-10-25T00:25:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.437+0000 7fbb62ffd640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb44065e60 0x7fbb44068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.437+0000 7fbb62ffd640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb44065e60 0x7fbb44068320 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbb641e30b0 tx=0x7fbb54008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:01.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.441+0000 7fbb5bfff640 1 -- 172.21.15.107:0/705085393 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbb4c0ea9c0 con 0x7fbb640ae5f0 2023-10-25T00:25:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.613+0000 7fbb69754640 1 -- 172.21.15.107:0/705085393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb640746e0 con 0x7fbb640ae5f0 2023-10-25T00:25:01.615 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.613+0000 7fbb5bfff640 1 -- 172.21.15.107:0/705085393 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+5769 (secure 0 0 0) 0x7fbb4c0edfe0 con 0x7fbb640ae5f0 2023-10-25T00:25:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:25:01.615 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9606,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2839787144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2839787144},{"type":"v1","addr":"172.21.15.107:6837","nonce":2839787144}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":240}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:02.679785+0000","modified":"2023-10-25T00:24:38.446814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9462":{"gid":9462,"name":"c","rank":0,"incarnation":236,"state":"up:active","state_seq":27,"addr":"172.21.15.107:6835/2963448958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":2963448958},{"type":"v1","addr":"172.21.15.107:6835","nonce":2963448958}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1},{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:25:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 240 2023-10-25T00:25:01.616 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 -- 172.21.15.107:0/705085393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb44065e60 msgr2=0x7fbb44068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb44065e60 0x7fbb44068320 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbb641e30b0 tx=0x7fbb54008040 comp rx=0 tx=0).stop 2023-10-25T00:25:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 -- 172.21.15.107:0/705085393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb640ae5f0 msgr2=0x7fbb641efcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb640ae5f0 0x7fbb641efcc0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c001ec0 tx=0x7fbb50005800 comp rx=0 tx=0).stop 2023-10-25T00:25:01.617 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 -- 172.21.15.107:0/705085393 shutdown_connections 2023-10-25T00:25:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb44065e60 0x7fbb44068320 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.618 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb641ed960 0x7fbb641f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.619 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb640ae5f0 0x7fbb641efcc0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 --2- 172.21.15.107:0/705085393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb500047d0 0x7fbb640ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 -- 172.21.15.107:0/705085393 >> 172.21.15.107:0/705085393 conn(0x7fbb640ca200 msgr2=0x7fbb640aac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:01.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 -- 172.21.15.107:0/705085393 shutdown_connections 2023-10-25T00:25:01.620 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.617+0000 7fbb69754640 1 -- 172.21.15.107:0/705085393 wait complete. 2023-10-25T00:25:01.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-25T00:25:01.750 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.749+0000 7f2fe04f2640 1 Processor -- start 2023-10-25T00:25:01.751 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.749+0000 7f2fe04f2640 1 -- start start 2023-10-25T00:25:01.753 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fe04f2640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2fd814fd40 0x7f2fd80ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fe04f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fd80a9820 0x7f2fd80aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fe04f2640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd806f730 con 0x7f2fd806ee40 2023-10-25T00:25:01.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fe04f2640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd80aa750 con 0x7f2fd814fd40 2023-10-25T00:25:01.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fe04f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd8156a40 con 0x7f2fd80a9820 2023-10-25T00:25:01.754 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdea8f640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f2fd806ee40 0x7f2fd80c1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34430/0 (socket says 172.21.15.107:34430) 2023-10-25T00:25:01.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdea8f640 1 -- 172.21.15.107:0/220279270 learned_addr learned my addr 172.21.15.107:0/220279270 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:25:01.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdda8d640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fd80a9820 0x7f2fd80aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fde28e640 1 --2- 172.21.15.107:0/220279270 >> v2:172.21.15.110:3300/0 conn(0x7f2fd814fd40 0x7f2fd80ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494705769 0 0) 0x7f2fd806f730 con 0x7f2fd806ee40 2023-10-25T00:25:01.755 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc0003530 con 0x7f2fd806ee40 2023-10-25T00:25:01.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926724814 0 0) 0x7f2fc0003530 con 0x7f2fd806ee40 2023-10-25T00:25:01.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fd80a9820 msgr2=0x7f2fd80aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:01.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fd80a9820 0x7f2fd80aa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 >> v2:172.21.15.110:3300/0 conn(0x7f2fd814fd40 msgr2=0x7f2fd80ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:01.756 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 --2- 172.21.15.107:0/220279270 >> v2:172.21.15.110:3300/0 conn(0x7f2fd814fd40 0x7f2fd80ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd8156710 con 0x7f2fd806ee40 2023-10-25T00:25:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f2fd0001a00 con 0x7f2fd806ee40 2023-10-25T00:25:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 >> v1:172.21.15.107:6789/0 conn(0x7f2fd806ee40 legacy=0x7f2fd80c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 0x7f2fc0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fc00047d0 0x7f2fc0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.757 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fc0005190 0x7f2fc000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd80aa750 con 0x7f2fc0003980 2023-10-25T00:25:01.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd8156a40 con 0x7f2fc00047d0 2023-10-25T00:25:01.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc000bed0 con 0x7f2fc0005190 2023-10-25T00:25:01.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fde28e640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 0x7f2fc0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.758 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fde28e640 1 -- 172.21.15.107:0/220279270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fc0005190 msgr2=0x7f2fc000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:01.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdea8f640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fc0005190 0x7f2fc000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fde28e640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fc0005190 0x7f2fc000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdda8d640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fc00047d0 0x7f2fc0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fde28e640 1 -- 172.21.15.107:0/220279270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fc00047d0 msgr2=0x7f2fc0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:01.759 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fde28e640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fc00047d0 0x7f2fc0004c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.760 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fde28e640 1 -- 172.21.15.107:0/220279270 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc0003530 con 0x7f2fc0003980 2023-10-25T00:25:01.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fde28e640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 0x7f2fc0003da0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8009ae0 tx=0x7f2fc801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b60bce19a9e3a971 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:01.762 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80a55d0 con 0x7f2fc0003980 2023-10-25T00:25:01.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2fc80a5730 con 0x7f2fc0003980 2023-10-25T00:25:01.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.753+0000 7f2fdd28c640 1 -- 172.21.15.107:0/220279270 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80a5b00 con 0x7f2fc0003980 2023-10-25T00:25:01.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- 172.21.15.107:0/220279270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 msgr2=0x7f2fc0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 --2- 172.21.15.107:0/220279270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 0x7f2fc0003da0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8009ae0 tx=0x7f2fc801ca60 comp rx=0 tx=0).stop 2023-10-25T00:25:01.763 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fdea8f640 1 -- 172.21.15.107:0/220279270 reap_dead start 2023-10-25T00:25:01.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- 172.21.15.107:0/220279270 shutdown_connections 2023-10-25T00:25:01.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- 172.21.15.107:0/220279270 >> 172.21.15.107:0/220279270 conn(0x7f2fd80c2950 msgr2=0x7f2fd80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:01.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- 172.21.15.107:0/220279270 shutdown_connections 2023-10-25T00:25:01.764 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- 172.21.15.107:0/220279270 wait complete. 2023-10-25T00:25:01.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 Processor -- start 2023-10-25T00:25:01.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- start start 2023-10-25T00:25:01.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 0x7f2fd8154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fd81ee610 0x7f2fd81eb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fd814efd0 0x7f2fd81ebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.765 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd81ef590 con 0x7f2fc0003980 2023-10-25T00:25:01.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd814f520 con 0x7f2fd814efd0 2023-10-25T00:25:01.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd814e570 con 0x7f2fd81ee610 2023-10-25T00:25:01.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fde28e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 0x7f2fd8154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fde28e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 0x7f2fd8154a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48786/0 (socket says 172.21.15.107:48786) 2023-10-25T00:25:01.766 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fde28e640 1 -- 172.21.15.107:0/3155895097 learned_addr learned my addr 172.21.15.107:0/3155895097 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:01.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fde28e640 1 -- 172.21.15.107:0/3155895097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fd81ee610 msgr2=0x7f2fd81eb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fde28e640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fd81ee610 0x7f2fd81eb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fde28e640 1 -- 172.21.15.107:0/3155895097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fd814efd0 msgr2=0x7f2fd81ebbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fde28e640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fd814efd0 0x7f2fd81ebbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.767 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fde28e640 1 -- 172.21.15.107:0/3155895097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd814e850 con 0x7f2fc0003980 2023-10-25T00:25:01.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fde28e640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 0x7f2fd8154a20 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f2fc801c890 tx=0x7f2fc0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:01.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fd6ffd640 1 -- 172.21.15.107:0/3155895097 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8004020 con 0x7f2fc0003980 2023-10-25T00:25:01.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fd6ffd640 1 -- 172.21.15.107:0/3155895097 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2fc80041c0 con 0x7f2fc0003980 2023-10-25T00:25:01.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- 172.21.15.107:0/3155895097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd814eb30 con 0x7f2fc0003980 2023-10-25T00:25:01.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fd6ffd640 1 -- 172.21.15.107:0/3155895097 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8003620 con 0x7f2fc0003980 2023-10-25T00:25:01.768 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- 172.21.15.107:0/3155895097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd8154f60 con 0x7f2fc0003980 2023-10-25T00:25:01.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.757+0000 7f2fe04f2640 1 -- 172.21.15.107:0/3155895097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd80c24a0 con 0x7f2fc0003980 2023-10-25T00:25:01.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.761+0000 7f2fd6ffd640 1 -- 172.21.15.107:0/3155895097 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f2fc80b4040 con 0x7f2fc0003980 2023-10-25T00:25:01.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.761+0000 7f2fd6ffd640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fc4065e90 0x7f2fc4068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:01.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.761+0000 7f2fd6ffd640 1 -- 172.21.15.107:0/3155895097 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7f2fc811ede0 con 0x7f2fc0003980 2023-10-25T00:25:01.769 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.761+0000 7f2fdda8d640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fc4065e90 0x7f2fc4068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:01.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.761+0000 7f2fdda8d640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fc4065e90 0x7f2fc4068350 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2fd81e8e80 tx=0x7f2fcc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:01.770 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.765+0000 7f2fd6ffd640 1 -- 172.21.15.107:0/3155895097 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f2fc80eb190 con 0x7f2fc0003980 2023-10-25T00:25:01.909 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.909+0000 7f2fe04f2640 1 -- 172.21.15.107:0/3155895097 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fd80746e0 con 0x7f2fc0003980 2023-10-25T00:25:01.914 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.913+0000 7f2fd6ffd640 1 -- 172.21.15.107:0/3155895097 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+27659 (secure 0 0 0) 0x7f2fc810ce60 con 0x7f2fc0003980 2023-10-25T00:25:01.915 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:25:01.915 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":74,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-25T00:24:34.268077+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_metadata","create_time":"2023-10-24T23:53:00.990207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"a_data","create_time":"2023-10-24T23:53:01.642395+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2f","mappings":[{"from":3,"to":2}]},{"pgid":"4.3e","mappings":[{"from":1,"to":2}]},{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6837/3527905219":"2023-10-26T00:24:34.267980+0000","172.21.15.107:6836/3527905219":"2023-10-26T00:24:34.267980+0000","172.21.15.107:6835/1528689730":"2023-10-26T00:22:49.197964+0000","172.21.15.107:6834/1528689730":"2023-10-26T00:22:49.197964+0000","172.21.15.107:6835/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6837/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.110:6834/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.110:6832/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.107:6835/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.107:6837/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.110:6834/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.110:6832/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.107:6835/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.107:6834/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.110:6833/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6836/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6836/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.107:6834/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6835/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6836/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.110:6834/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6835/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.110:6835/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6834/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.107:6837/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.107:6837/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6832/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6833/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.110:6833/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6834/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.110:6835/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6836/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6836/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6834/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.110:6835/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.107:6834/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.107:6834/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.110:6833/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.107:6837/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6834/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.110:6833/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.110:6832/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.107:6836/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6837/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.107:6837/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.110:6833/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6837/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.107:6835/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.107:6837/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6837/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6835/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.110:6835/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.107:6836/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.107:6834/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.110:6832/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.107:6835/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6835/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.110:6832/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.110:6833/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6836/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6834/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6836/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.110:6834/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6837/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6836/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.110:6835/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6834/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6835/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6835/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.110:6832/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6835/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.107:6836/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6833/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6835/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.107:6836/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6834/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.107:6836/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6836/1788268787":"2023-10-26T00:10:38.784519+0000","172.21.15.110:6835/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.107:6837/1788268787":"2023-10-26T00:10:38.784519+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-25T00:25:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 -- 172.21.15.107:0/3155895097 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fc4065e90 msgr2=0x7f2fc4068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.916 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fc4065e90 0x7f2fc4068350 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2fd81e8e80 tx=0x7f2fcc008040 comp rx=0 tx=0).stop 2023-10-25T00:25:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 -- 172.21.15.107:0/3155895097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 msgr2=0x7f2fd8154a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 0x7f2fd8154a20 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f2fc801c890 tx=0x7f2fc0005800 comp rx=0 tx=0).stop 2023-10-25T00:25:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 -- 172.21.15.107:0/3155895097 shutdown_connections 2023-10-25T00:25:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2fd814efd0 0x7f2fd81ebbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2fc4065e90 0x7f2fc4068350 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.917 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2fd81ee610 0x7f2fd81eb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 --2- 172.21.15.107:0/3155895097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2fc0003980 0x7f2fd8154a20 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 -- 172.21.15.107:0/3155895097 >> 172.21.15.107:0/3155895097 conn(0x7f2fd80c2950 msgr2=0x7f2fd80c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 -- 172.21.15.107:0/3155895097 shutdown_connections 2023-10-25T00:25:01.918 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:01.917+0000 7f2fe04f2640 1 -- 172.21.15.107:0/3155895097 wait complete. 2023-10-25T00:25:01.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail a 2023-10-25T00:25:02.048 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3ec0f640 1 Processor -- start 2023-10-25T00:25:02.049 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3ec0f640 1 -- start start 2023-10-25T00:25:02.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3ec0f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb3806e6d0 0x7fbb38158890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3ec0f640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbb380ad860 0x7fbb3813d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3ec0f640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb380adf50 con 0x7fbb38071cb0 2023-10-25T00:25:02.050 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3ec0f640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb3806d8c0 con 0x7fbb380ad860 2023-10-25T00:25:02.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3ec0f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb381548e0 con 0x7fbb3806e6d0 2023-10-25T00:25:02.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3d1ac640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7fbb380ad860 0x7fbb3813d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3c9ab640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fbb38071cb0 0x7fbb38154fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34432/0 (socket says 172.21.15.107:34432) 2023-10-25T00:25:02.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3c9ab640 1 -- 172.21.15.107:0/1184911968 learned_addr learned my addr 172.21.15.107:0/1184911968 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:25:02.051 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb37fff640 1 --2- 172.21.15.107:0/1184911968 >> v2:172.21.15.110:3300/0 conn(0x7fbb3806e6d0 0x7fbb38158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3d1ac640 1 -- 172.21.15.107:0/1184911968 >> v2:172.21.15.110:3300/0 conn(0x7fbb3806e6d0 msgr2=0x7fbb38158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:02.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3d1ac640 1 --2- 172.21.15.107:0/1184911968 >> v2:172.21.15.110:3300/0 conn(0x7fbb3806e6d0 0x7fbb38158890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3d1ac640 1 -- 172.21.15.107:0/1184911968 >> v1:172.21.15.107:6789/0 conn(0x7fbb38071cb0 legacy=0x7fbb38154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:02.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb377fe640 1 -- 172.21.15.107:0/1184911968 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662709416 0 0) 0x7fbb380adf50 con 0x7fbb38071cb0 2023-10-25T00:25:02.052 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.049+0000 7fbb3d1ac640 1 -- 172.21.15.107:0/1184911968 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3806c960 con 0x7fbb380ad860 2023-10-25T00:25:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3d1ac640 1 --2- 172.21.15.107:0/1184911968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb380ad860 0x7fbb3813d840 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fbb380c24a0 tx=0x7fbb2801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a46e3ed9f2230e server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb377fe640 1 -- 172.21.15.107:0/1184911968 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280ab600 con 0x7fbb380ad860 2023-10-25T00:25:02.054 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb377fe640 1 -- 172.21.15.107:0/1184911968 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbb280a7050 con 0x7fbb380ad860 2023-10-25T00:25:02.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb377fe640 1 -- 172.21.15.107:0/1184911968 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb280ab9a0 con 0x7fbb380ad860 2023-10-25T00:25:02.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1184911968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb380ad860 msgr2=0x7fbb3813d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:02.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 --2- 172.21.15.107:0/1184911968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb380ad860 0x7fbb3813d840 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fbb380c24a0 tx=0x7fbb2801c460 comp rx=0 tx=0).stop 2023-10-25T00:25:02.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1184911968 shutdown_connections 2023-10-25T00:25:02.055 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 --2- 172.21.15.107:0/1184911968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb380ad860 0x7fbb3813d840 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 --2- 172.21.15.107:0/1184911968 >> v2:172.21.15.110:3300/0 conn(0x7fbb3806e6d0 0x7fbb38158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1184911968 >> 172.21.15.107:0/1184911968 conn(0x7fbb38070ed0 msgr2=0x7fbb380712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:02.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1184911968 shutdown_connections 2023-10-25T00:25:02.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1184911968 wait complete. 2023-10-25T00:25:02.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 Processor -- start 2023-10-25T00:25:02.056 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 -- start start 2023-10-25T00:25:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb3806e6d0 0x7fbb38155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb38071cb0 0x7fbb381ed580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb380ad860 0x7fbb3813bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38156c30 con 0x7fbb38071cb0 2023-10-25T00:25:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb381edac0 con 0x7fbb3806e6d0 2023-10-25T00:25:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3ec0f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38157db0 con 0x7fbb380ad860 2023-10-25T00:25:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb38071cb0 0x7fbb381ed580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb38071cb0 0x7fbb381ed580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48810/0 (socket says 172.21.15.107:48810) 2023-10-25T00:25:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb37fff640 1 -- 172.21.15.107:0/1808568340 learned_addr learned my addr 172.21.15.107:0/1808568340 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:02.060 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb37fff640 1 -- 172.21.15.107:0/1808568340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb380ad860 msgr2=0x7fbb3813bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:02.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb37fff640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb380ad860 0x7fbb3813bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb3c9ab640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb3806e6d0 0x7fbb38155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb37fff640 1 -- 172.21.15.107:0/1808568340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb3806e6d0 msgr2=0x7fbb38155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:02.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb37fff640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb3806e6d0 0x7fbb38155f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.061 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb37fff640 1 -- 172.21.15.107:0/1808568340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb28009420 con 0x7fbb38071cb0 2023-10-25T00:25:02.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb37fff640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb38071cb0 0x7fbb381ed580 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000c5e0 tx=0x7fbb2000caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:02.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.053+0000 7fbb357fa640 1 -- 172.21.15.107:0/1808568340 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20004220 con 0x7fbb38071cb0 2023-10-25T00:25:02.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.057+0000 7fbb357fa640 1 -- 172.21.15.107:0/1808568340 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbb20004380 con 0x7fbb38071cb0 2023-10-25T00:25:02.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.057+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1808568340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb38157ad0 con 0x7fbb38071cb0 2023-10-25T00:25:02.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.057+0000 7fbb357fa640 1 -- 172.21.15.107:0/1808568340 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20003880 con 0x7fbb38071cb0 2023-10-25T00:25:02.062 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.057+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1808568340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb380c7ad0 con 0x7fbb38071cb0 2023-10-25T00:25:02.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.057+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1808568340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3804e130 con 0x7fbb38071cb0 2023-10-25T00:25:02.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.057+0000 7fbb357fa640 1 -- 172.21.15.107:0/1808568340 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fbb200039e0 con 0x7fbb38071cb0 2023-10-25T00:25:02.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.057+0000 7fbb357fa640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 0x7fbb18068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.057+0000 7fbb357fa640 1 -- 172.21.15.107:0/1808568340 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 10917+0+0 (secure 0 0 0) 0x7fbb20084c50 con 0x7fbb38071cb0 2023-10-25T00:25:02.063 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.057+0000 7fbb3c9ab640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 0x7fbb18068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.057+0000 7fbb3c9ab640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 0x7fbb18068320 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fbb300012f0 tx=0x7fbb30009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:02.064 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.061+0000 7fbb357fa640 1 -- 172.21.15.107:0/1808568340 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fbb2008a020 con 0x7fbb38071cb0 2023-10-25T00:25:02.248 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.249+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1808568340 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "a"} v 0) v1 -- 0x7fbb381529f0 con 0x7fbb38071cb0 2023-10-25T00:25:02.271 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.269+0000 7fbb357fa640 1 -- 172.21.15.107:0/1808568340 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "a"}]=0 a marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v241) v1 ==== 151+0+0 (secure 0 0 0) 0x7fbb2001e070 con 0x7fbb38071cb0 2023-10-25T00:25:02.272 INFO:teuthology.orchestra.run.smithi107.stderr:a marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-25T00:25:02.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1808568340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 msgr2=0x7fbb18068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:02.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 0x7fbb18068320 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fbb300012f0 tx=0x7fbb30009040 comp rx=0 tx=0).stop 2023-10-25T00:25:02.274 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1808568340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb38071cb0 msgr2=0x7fbb381ed580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:02.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb38071cb0 0x7fbb381ed580 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000c5e0 tx=0x7fbb2000caa0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1808568340 shutdown_connections 2023-10-25T00:25:02.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fbb18065e60 0x7fbb18068320 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbb380ad860 0x7fbb3813bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb38071cb0 0x7fbb381ed580 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 --2- 172.21.15.107:0/1808568340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb3806e6d0 0x7fbb38155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1808568340 >> 172.21.15.107:0/1808568340 conn(0x7fbb38070ed0 msgr2=0x7fbb381541a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:02.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1808568340 shutdown_connections 2023-10-25T00:25:02.276 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.273+0000 7fbb3ec0f640 1 -- 172.21.15.107:0/1808568340 wait complete. 2023-10-25T00:25:02.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm a --yes-i-really-mean-it 2023-10-25T00:25:02.294 INFO:tasks.ceph.mds.c.smithi107.stderr: -465> 2023-10-25T00:24:42.529+0000 7f0e2b592640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-25T00:25:02.336 INFO:tasks.ceph.mds.c.smithi107.stdout:starting mds.c at 2023-10-25T00:25:02.409 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe66863b640 1 Processor -- start 2023-10-25T00:25:02.410 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe66863b640 1 -- start start 2023-10-25T00:25:02.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe66863b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe6600a4f20 0x7fe6600bdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.412 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe66863b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6600a90c0 0x7fe6600a94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe66863b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6600abdc0 con 0x7fe6600a9a20 2023-10-25T00:25:02.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe66863b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6600a8c30 con 0x7fe6600a4f20 2023-10-25T00:25:02.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe66863b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6600bef60 con 0x7fe6600a90c0 2023-10-25T00:25:02.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe666bd8640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7fe6600a9a20 0x7fe6600c0760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:34442/0 (socket says 172.21.15.107:34442) 2023-10-25T00:25:02.413 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe666bd8640 1 -- 172.21.15.107:0/429769586 learned_addr learned my addr 172.21.15.107:0/429769586 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:25:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6663d7640 1 --2- 172.21.15.107:0/429769586 >> v2:172.21.15.110:3300/0 conn(0x7fe6600a4f20 0x7fe6600bdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835133645 0 0) 0x7fe6600abdc0 con 0x7fe6600a9a20 2023-10-25T00:25:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe64c003530 con 0x7fe6600a9a20 2023-10-25T00:25:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe665bd6640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6600a90c0 0x7fe6600a94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4048157723 0 0) 0x7fe64c003530 con 0x7fe6600a9a20 2023-10-25T00:25:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6600a90c0 msgr2=0x7fe6600a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6600a90c0 0x7fe6600a94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 >> v2:172.21.15.110:3300/0 conn(0x7fe6600a4f20 msgr2=0x7fe6600bdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 --2- 172.21.15.107:0/429769586 >> v2:172.21.15.110:3300/0 conn(0x7fe6600a4f20 0x7fe6600bdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6600bf240 con 0x7fe6600a9a20 2023-10-25T00:25:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7fe658001a00 con 0x7fe6600a9a20 2023-10-25T00:25:02.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 >> v1:172.21.15.107:6789/0 conn(0x7fe6600a9a20 legacy=0x7fe6600c0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:02.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe64c003980 0x7fe64c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe64c0047d0 0x7fe64c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6600a90c0 0x7fe64c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6600a8c30 con 0x7fe64c003980 2023-10-25T00:25:02.416 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6600bef60 con 0x7fe64c0047d0 2023-10-25T00:25:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.409+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c00b990 con 0x7fe6600a90c0 2023-10-25T00:25:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe6663d7640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe64c003980 0x7fe64c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe666bd8640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6600a90c0 0x7fe64c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe665bd6640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe64c0047d0 0x7fe64c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe6663d7640 1 -- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6600a90c0 msgr2=0x7fe64c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe6663d7640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe6600a90c0 0x7fe64c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe6663d7640 1 -- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe64c0047d0 msgr2=0x7fe64c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe6663d7640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe64c0047d0 0x7fe64c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe6663d7640 1 -- 172.21.15.107:0/429769586 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c003530 con 0x7fe64c003980 2023-10-25T00:25:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe665bd6640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe64c0047d0 0x7fe64c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:02.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe6663d7640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe64c003980 0x7fe64c003da0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00aae0 tx=0x7fe65c01dfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72124e86577b43a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:02.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0a6e50 con 0x7fe64c003980 2023-10-25T00:25:02.421 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe65c0ae440 con 0x7fe64c003980 2023-10-25T00:25:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe6653d5640 1 -- 172.21.15.107:0/429769586 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0ae810 con 0x7fe64c003980 2023-10-25T00:25:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe66863b640 1 -- 172.21.15.107:0/429769586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe64c003980 msgr2=0x7fe64c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe66863b640 1 --2- 172.21.15.107:0/429769586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe64c003980 0x7fe64c003da0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00aae0 tx=0x7fe65c01dfb0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe666bd8640 1 -- 172.21.15.107:0/429769586 reap_dead start 2023-10-25T00:25:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe66863b640 1 -- 172.21.15.107:0/429769586 shutdown_connections 2023-10-25T00:25:02.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe66863b640 1 -- 172.21.15.107:0/429769586 >> 172.21.15.107:0/429769586 conn(0x7fe6600c2860 msgr2=0x7fe6600c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:02.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe66863b640 1 -- 172.21.15.107:0/429769586 shutdown_connections 2023-10-25T00:25:02.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.413+0000 7fe66863b640 1 -- 172.21.15.107:0/429769586 wait complete. 2023-10-25T00:25:02.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe66863b640 1 Processor -- start 2023-10-25T00:25:02.423 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe66863b640 1 -- start start 2023-10-25T00:25:02.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe66863b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe64c003980 0x7fe6601f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe66863b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66006eb70 0x7fe66006f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe66863b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe66006ce00 0x7fe660154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.424 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe66863b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6601f0ed0 con 0x7fe66006eb70 2023-10-25T00:25:02.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe66863b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66006bde0 con 0x7fe66006ce00 2023-10-25T00:25:02.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe66863b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66006c0c0 con 0x7fe64c003980 2023-10-25T00:25:02.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe665bd6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66006eb70 0x7fe66006f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe665bd6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66006eb70 0x7fe66006f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48854/0 (socket says 172.21.15.107:48854) 2023-10-25T00:25:02.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe665bd6640 1 -- 172.21.15.107:0/2121596595 learned_addr learned my addr 172.21.15.107:0/2121596595 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:02.425 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe666bd8640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe66006ce00 0x7fe660154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe665bd6640 1 -- 172.21.15.107:0/2121596595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe64c003980 msgr2=0x7fe6601f05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:02.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe665bd6640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe64c003980 0x7fe6601f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe665bd6640 1 -- 172.21.15.107:0/2121596595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe66006ce00 msgr2=0x7fe660154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:02.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe665bd6640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe66006ce00 0x7fe660154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:02.426 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe665bd6640 1 -- 172.21.15.107:0/2121596595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66006c3a0 con 0x7fe66006eb70 2023-10-25T00:25:02.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe665bd6640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66006eb70 0x7fe66006f790 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fe65000e7e0 tx=0x7fe65000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:02.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe656ffd640 1 -- 172.21.15.107:0/2121596595 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65000c6d0 con 0x7fe66006eb70 2023-10-25T00:25:02.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe656ffd640 1 -- 172.21.15.107:0/2121596595 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fe65000c830 con 0x7fe66006eb70 2023-10-25T00:25:02.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe66863b640 1 -- 172.21.15.107:0/2121596595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6601f11b0 con 0x7fe66006eb70 2023-10-25T00:25:02.427 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe656ffd640 1 -- 172.21.15.107:0/2121596595 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65000cc00 con 0x7fe66006eb70 2023-10-25T00:25:02.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe66863b640 1 -- 172.21.15.107:0/2121596595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe660156ac0 con 0x7fe66006eb70 2023-10-25T00:25:02.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe66863b640 1 -- 172.21.15.107:0/2121596595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66006c530 con 0x7fe66006eb70 2023-10-25T00:25:02.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe656ffd640 1 -- 172.21.15.107:0/2121596595 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7fe650013410 con 0x7fe66006eb70 2023-10-25T00:25:02.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe656ffd640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe644065e90 0x7fe644068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:02.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe6663d7640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe644065e90 0x7fe644068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:02.428 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.417+0000 7fe656ffd640 1 -- 172.21.15.107:0/2121596595 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 11003+0+0 (secure 0 0 0) 0x7fe650084a10 con 0x7fe66006eb70 2023-10-25T00:25:02.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.421+0000 7fe6663d7640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe644065e90 0x7fe644068350 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0b0000 tx=0x7fe65c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:02.429 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.425+0000 7fe656ffd640 1 -- 172.21.15.107:0/2121596595 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7fe650050c30 con 0x7fe66006eb70 2023-10-25T00:25:02.601 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:02.601+0000 7fe66863b640 1 -- 172.21.15.107:0/2121596595 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "a", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe660155b60 con 0x7fe66006eb70 2023-10-25T00:25:03.275 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.273+0000 7fe656ffd640 1 -- 172.21.15.107:0/2121596595 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "a", "yes_i_really_mean_it": true}]=0 v242) v1 ==== 99+0+0 (secure 0 0 0) 0x7fe650021070 con 0x7fe66006eb70 2023-10-25T00:25:03.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.277+0000 7fe66863b640 1 -- 172.21.15.107:0/2121596595 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe644065e90 msgr2=0x7fe644068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:03.279 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.277+0000 7fe66863b640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe644065e90 0x7fe644068350 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0b0000 tx=0x7fe65c008040 comp rx=0 tx=0).stop 2023-10-25T00:25:03.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.277+0000 7fe66863b640 1 -- 172.21.15.107:0/2121596595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66006eb70 msgr2=0x7fe66006f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:03.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.277+0000 7fe66863b640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66006eb70 0x7fe66006f790 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fe65000e7e0 tx=0x7fe65000eca0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.277+0000 7fe66863b640 1 -- 172.21.15.107:0/2121596595 shutdown_connections 2023-10-25T00:25:03.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.277+0000 7fe66863b640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe66006ce00 0x7fe660154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.280 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.277+0000 7fe66863b640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe66006eb70 0x7fe66006f790 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.277+0000 7fe66863b640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7fe644065e90 0x7fe644068350 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.277+0000 7fe66863b640 1 --2- 172.21.15.107:0/2121596595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe64c003980 0x7fe6601f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.277+0000 7fe66863b640 1 -- 172.21.15.107:0/2121596595 >> 172.21.15.107:0/2121596595 conn(0x7fe6600c2860 msgr2=0x7fe6600a5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:03.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.281+0000 7fe66863b640 1 -- 172.21.15.107:0/2121596595 shutdown_connections 2023-10-25T00:25:03.281 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.281+0000 7fe66863b640 1 -- 172.21.15.107:0/2121596595 wait complete. 2023-10-25T00:25:03.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm a_metadata a_metadata --yes-i-really-really-mean-it 2023-10-25T00:25:03.431 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.429+0000 7ff218b90640 1 Processor -- start 2023-10-25T00:25:03.432 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.429+0000 7ff218b90640 1 -- start start 2023-10-25T00:25:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff218b90640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff2140bd760 0x7ff214154d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff218b90640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff214071dd0 0x7ff214155270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff218b90640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff21406cd70 0x7ff21406d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:03.433 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff218b90640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2140c2a60 con 0x7ff214071dd0 2023-10-25T00:25:03.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff218b90640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2140c23c0 con 0x7ff2140bd760 2023-10-25T00:25:03.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff218b90640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2141557b0 con 0x7ff21406cd70 2023-10-25T00:25:03.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff211d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff214071dd0 0x7ff214155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:03.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff211d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7ff214071dd0 0x7ff214155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55240/0 (socket says 172.21.15.107:55240) 2023-10-25T00:25:03.434 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff211d9b640 1 -- 172.21.15.107:0/2457667551 learned_addr learned my addr 172.21.15.107:0/2457667551 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:03.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff211d9b640 1 -- 172.21.15.107:0/2457667551 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff21406cd70 msgr2=0x7ff21406d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:03.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff211d9b640 1 --2- 172.21.15.107:0/2457667551 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff21406cd70 0x7ff21406d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff21259c640 1 --2- 172.21.15.107:0/2457667551 >> v2:172.21.15.110:3300/0 conn(0x7ff2140bd760 0x7ff214154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:03.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff211d9b640 1 -- 172.21.15.107:0/2457667551 >> v2:172.21.15.110:3300/0 conn(0x7ff2140bd760 msgr2=0x7ff214154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:03.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff211d9b640 1 --2- 172.21.15.107:0/2457667551 >> v2:172.21.15.110:3300/0 conn(0x7ff2140bd760 0x7ff214154d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.435 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff211d9b640 1 -- 172.21.15.107:0/2457667551 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140c6b80 con 0x7ff214071dd0 2023-10-25T00:25:03.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.433+0000 7ff211d9b640 1 --2- 172.21.15.107:0/2457667551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff214071dd0 0x7ff214155270 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7ff2140c2070 tx=0x7ff20401eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55fccb63b72855b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:03.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff21159a640 1 -- 172.21.15.107:0/2457667551 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2040a7a40 con 0x7ff214071dd0 2023-10-25T00:25:03.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff21159a640 1 -- 172.21.15.107:0/2457667551 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7ff2040a7ba0 con 0x7ff214071dd0 2023-10-25T00:25:03.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff21159a640 1 -- 172.21.15.107:0/2457667551 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2040af440 con 0x7ff214071dd0 2023-10-25T00:25:03.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 -- 172.21.15.107:0/2457667551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff214071dd0 msgr2=0x7ff214155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:03.437 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 --2- 172.21.15.107:0/2457667551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff214071dd0 0x7ff214155270 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7ff2140c2070 tx=0x7ff20401eab0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 -- 172.21.15.107:0/2457667551 shutdown_connections 2023-10-25T00:25:03.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 --2- 172.21.15.107:0/2457667551 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff21406cd70 0x7ff21406d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 --2- 172.21.15.107:0/2457667551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff214071dd0 0x7ff214155270 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 --2- 172.21.15.107:0/2457667551 >> v2:172.21.15.110:3300/0 conn(0x7ff2140bd760 0x7ff214154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.438 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 -- 172.21.15.107:0/2457667551 >> 172.21.15.107:0/2457667551 conn(0x7ff2140a8a70 msgr2=0x7ff2140a8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:03.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 -- 172.21.15.107:0/2457667551 shutdown_connections 2023-10-25T00:25:03.439 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 -- 172.21.15.107:0/2457667551 wait complete. 2023-10-25T00:25:03.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 Processor -- start 2023-10-25T00:25:03.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 -- start start 2023-10-25T00:25:03.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff21406cd70 0x7ff214071400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:03.441 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff214071dd0 0x7ff2141ea940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:03.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2140bd760 0x7ff2141eae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:03.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2140c1260 con 0x7ff214071dd0 2023-10-25T00:25:03.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21414f590 con 0x7ff21406cd70 2023-10-25T00:25:03.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2141e9ef0 con 0x7ff2140bd760 2023-10-25T00:25:03.442 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff211d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff214071dd0 0x7ff2141ea940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:03.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff211d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff214071dd0 0x7ff2141ea940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55254/0 (socket says 172.21.15.107:55254) 2023-10-25T00:25:03.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff211d9b640 1 -- 172.21.15.107:0/451879119 learned_addr learned my addr 172.21.15.107:0/451879119 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:03.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff21259c640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff21406cd70 0x7ff214071400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:03.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff212d9d640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2140bd760 0x7ff2141eae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:03.443 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff211d9b640 1 -- 172.21.15.107:0/451879119 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2140bd760 msgr2=0x7ff2141eae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:03.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff211d9b640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2140bd760 0x7ff2141eae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff211d9b640 1 -- 172.21.15.107:0/451879119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff21406cd70 msgr2=0x7ff214071400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:03.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff211d9b640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff21406cd70 0x7ff214071400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:03.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff211d9b640 1 -- 172.21.15.107:0/451879119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20400b770 con 0x7ff214071dd0 2023-10-25T00:25:03.444 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff211d9b640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff214071dd0 0x7ff2141ea940 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ff20400b2c0 tx=0x7ff20401b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff20affd640 1 -- 172.21.15.107:0/451879119 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2040a7a40 con 0x7ff214071dd0 2023-10-25T00:25:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff20affd640 1 -- 172.21.15.107:0/451879119 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7ff2040ba420 con 0x7ff214071dd0 2023-10-25T00:25:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 -- 172.21.15.107:0/451879119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2141eca30 con 0x7ff214071dd0 2023-10-25T00:25:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff20affd640 1 -- 172.21.15.107:0/451879119 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2040ba7f0 con 0x7ff214071dd0 2023-10-25T00:25:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.437+0000 7ff218b90640 1 -- 172.21.15.107:0/451879119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2141ea140 con 0x7ff214071dd0 2023-10-25T00:25:03.445 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.441+0000 7ff218b90640 1 -- 172.21.15.107:0/451879119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21406d280 con 0x7ff214071dd0 2023-10-25T00:25:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.441+0000 7ff20affd640 1 -- 172.21.15.107:0/451879119 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7ff2040b6040 con 0x7ff214071dd0 2023-10-25T00:25:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.441+0000 7ff20affd640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1f4065e60 0x7ff1f4068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.441+0000 7ff20affd640 1 -- 172.21.15.107:0/451879119 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 11003+0+0 (secure 0 0 0) 0x7ff204122080 con 0x7ff214071dd0 2023-10-25T00:25:03.446 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.441+0000 7ff21259c640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1f4065e60 0x7ff1f4068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:03.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.441+0000 7ff21259c640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1f4065e60 0x7ff1f4068320 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00a8d0 tx=0x7ff1fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:03.447 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.445+0000 7ff20affd640 1 -- 172.21.15.107:0/451879119 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7ff2040ee330 con 0x7ff214071dd0 2023-10-25T00:25:03.602 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:03.601+0000 7ff218b90640 1 -- 172.21.15.107:0/451879119 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "a_metadata", "pool2": "a_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff2140746e0 con 0x7ff214071dd0 2023-10-25T00:25:04.287 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.285+0000 7ff20affd640 1 -- 172.21.15.107:0/451879119 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "a_metadata", "pool2": "a_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'a_metadata' removed v76) v1 ==== 166+0+0 (secure 0 0 0) 0x7ff2040f1950 con 0x7ff214071dd0 2023-10-25T00:25:04.288 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'a_metadata' removed 2023-10-25T00:25:04.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 -- 172.21.15.107:0/451879119 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1f4065e60 msgr2=0x7ff1f4068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:04.294 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1f4065e60 0x7ff1f4068320 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00a8d0 tx=0x7ff1fc008040 comp rx=0 tx=0).stop 2023-10-25T00:25:04.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 -- 172.21.15.107:0/451879119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff214071dd0 msgr2=0x7ff2141ea940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:04.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff214071dd0 0x7ff2141ea940 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ff20400b2c0 tx=0x7ff20401b850 comp rx=0 tx=0).stop 2023-10-25T00:25:04.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 -- 172.21.15.107:0/451879119 shutdown_connections 2023-10-25T00:25:04.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7ff1f4065e60 0x7ff1f4068320 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:04.295 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff2140bd760 0x7ff2141eae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:04.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff214071dd0 0x7ff2141ea940 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:04.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 --2- 172.21.15.107:0/451879119 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff21406cd70 0x7ff214071400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:04.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 -- 172.21.15.107:0/451879119 >> 172.21.15.107:0/451879119 conn(0x7ff2140a8a70 msgr2=0x7ff2140bc670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:04.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 -- 172.21.15.107:0/451879119 shutdown_connections 2023-10-25T00:25:04.296 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.293+0000 7ff218b90640 1 -- 172.21.15.107:0/451879119 wait complete. 2023-10-25T00:25:04.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm a_data a_data --yes-i-really-really-mean-it 2023-10-25T00:25:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f10a4b05640 1 Processor -- start 2023-10-25T00:25:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f10a4b05640 1 -- start start 2023-10-25T00:25:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f10a4b05640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f10a00ab2d0 0x7f10a00a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f10a4b05640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f10a006aac0 0x7f10a00c0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f10a4b05640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10a006cc30 0x7f10a00c0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f10a4b05640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a00c1390 con 0x7f10a00ab2d0 2023-10-25T00:25:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f10a4b05640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a0156880 con 0x7f10a006aac0 2023-10-25T00:25:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f10a4b05640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a0155f50 con 0x7f10a006cc30 2023-10-25T00:25:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f109e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f10a00ab2d0 0x7f10a00a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f109e59c640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f10a00ab2d0 0x7f10a00a4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55270/0 (socket says 172.21.15.107:55270) 2023-10-25T00:25:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f109e59c640 1 -- 172.21.15.107:0/2832364714 learned_addr learned my addr 172.21.15.107:0/2832364714 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:04.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f109ed9d640 1 --2- 172.21.15.107:0/2832364714 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10a006cc30 0x7f10a00c0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:04.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f109dd9b640 1 --2- 172.21.15.107:0/2832364714 >> v2:172.21.15.110:3300/0 conn(0x7f10a006aac0 0x7f10a00c0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:04.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f109e59c640 1 -- 172.21.15.107:0/2832364714 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10a006cc30 msgr2=0x7f10a00c0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:04.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f109e59c640 1 --2- 172.21.15.107:0/2832364714 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10a006cc30 0x7f10a00c0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:04.453 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f109e59c640 1 -- 172.21.15.107:0/2832364714 >> v2:172.21.15.110:3300/0 conn(0x7f10a006aac0 msgr2=0x7f10a00c0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:04.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f109e59c640 1 --2- 172.21.15.107:0/2832364714 >> v2:172.21.15.110:3300/0 conn(0x7f10a006aac0 0x7f10a00c0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:04.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f109e59c640 1 -- 172.21.15.107:0/2832364714 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a0156230 con 0x7f10a00ab2d0 2023-10-25T00:25:04.454 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.449+0000 7f109dd9b640 1 --2- 172.21.15.107:0/2832364714 >> v2:172.21.15.110:3300/0 conn(0x7f10a006aac0 0x7f10a00c0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:04.455 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f109e59c640 1 --2- 172.21.15.107:0/2832364714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10a00ab2d0 0x7f10a00a4f70 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f10a00abec0 tx=0x7f109001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8160e707fc68809 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:04.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f109d59a640 1 -- 172.21.15.107:0/2832364714 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1090004230 con 0x7f10a00ab2d0 2023-10-25T00:25:04.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f109d59a640 1 -- 172.21.15.107:0/2832364714 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f10900a7050 con 0x7f10a00ab2d0 2023-10-25T00:25:04.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f109d59a640 1 -- 172.21.15.107:0/2832364714 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1090003620 con 0x7f10a00ab2d0 2023-10-25T00:25:04.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f10a4b05640 1 -- 172.21.15.107:0/2832364714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10a00ab2d0 msgr2=0x7f10a00a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:04.456 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f10a4b05640 1 --2- 172.21.15.107:0/2832364714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10a00ab2d0 0x7f10a00a4f70 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f10a00abec0 tx=0x7f109001c500 comp rx=0 tx=0).stop 2023-10-25T00:25:04.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f10a4b05640 1 -- 172.21.15.107:0/2832364714 shutdown_connections 2023-10-25T00:25:04.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f10a4b05640 1 --2- 172.21.15.107:0/2832364714 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10a006cc30 0x7f10a00c0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:04.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f10a4b05640 1 --2- 172.21.15.107:0/2832364714 >> v2:172.21.15.110:3300/0 conn(0x7f10a006aac0 0x7f10a00c0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:04.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f10a4b05640 1 --2- 172.21.15.107:0/2832364714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10a00ab2d0 0x7f10a00a4f70 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:04.457 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f10a4b05640 1 -- 172.21.15.107:0/2832364714 >> 172.21.15.107:0/2832364714 conn(0x7f10a00c27f0 msgr2=0x7f10a00c2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:04.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f10a4b05640 1 -- 172.21.15.107:0/2832364714 shutdown_connections 2023-10-25T00:25:04.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.453+0000 7f10a4b05640 1 -- 172.21.15.107:0/2832364714 wait complete. 2023-10-25T00:25:04.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f10a4b05640 1 Processor -- start 2023-10-25T00:25:04.458 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f10a4b05640 1 -- start start 2023-10-25T00:25:04.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f10a4b05640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10a006aac0 0x7f10a00aa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:04.460 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f10a4b05640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10a006cc30 0x7f10a006f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:04.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f10a4b05640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10a00ab2d0 0x7f10a01f0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:04.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f10a4b05640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a00a9370 con 0x7f10a006aac0 2023-10-25T00:25:04.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f10a4b05640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a01f1360 con 0x7f10a006cc30 2023-10-25T00:25:04.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f10a4b05640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a014f300 con 0x7f10a00ab2d0 2023-10-25T00:25:04.461 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10a006aac0 0x7f10a00aa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:04.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10a006aac0 0x7f10a00aa3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55276/0 (socket says 172.21.15.107:55276) 2023-10-25T00:25:04.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109e59c640 1 -- 172.21.15.107:0/4212790201 learned_addr learned my addr 172.21.15.107:0/4212790201 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:04.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109dd9b640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10a006cc30 0x7f10a006f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:04.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109e59c640 1 -- 172.21.15.107:0/4212790201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10a00ab2d0 msgr2=0x7f10a01f0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:04.462 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109ed9d640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10a00ab2d0 0x7f10a01f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:04.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109e59c640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10a00ab2d0 0x7f10a01f0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:04.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109e59c640 1 -- 172.21.15.107:0/4212790201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10a006cc30 msgr2=0x7f10a006f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:04.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109e59c640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10a006cc30 0x7f10a006f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:04.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109e59c640 1 -- 172.21.15.107:0/4212790201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10900094d0 con 0x7f10a006aac0 2023-10-25T00:25:04.463 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109e59c640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10a006aac0 0x7f10a00aa3e0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f109001cda0 tx=0x7f10900192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:04.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f1096ffd640 1 -- 172.21.15.107:0/4212790201 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1090003d10 con 0x7f10a006aac0 2023-10-25T00:25:04.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f1096ffd640 1 -- 172.21.15.107:0/4212790201 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1090003e70 con 0x7f10a006aac0 2023-10-25T00:25:04.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f1096ffd640 1 -- 172.21.15.107:0/4212790201 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10900ab680 con 0x7f10a006aac0 2023-10-25T00:25:04.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f10a4b05640 1 -- 172.21.15.107:0/4212790201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a01f1080 con 0x7f10a006aac0 2023-10-25T00:25:04.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f10a4b05640 1 -- 172.21.15.107:0/4212790201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a006ec60 con 0x7f10a006aac0 2023-10-25T00:25:04.464 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f10a4b05640 1 -- 172.21.15.107:0/4212790201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a00aa100 con 0x7f10a006aac0 2023-10-25T00:25:04.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f1096ffd640 1 -- 172.21.15.107:0/4212790201 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f10900b2040 con 0x7f10a006aac0 2023-10-25T00:25:04.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f1096ffd640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1080065e60 0x7f1080068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:04.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f1096ffd640 1 -- 172.21.15.107:0/4212790201 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 10575+0+0 (secure 0 0 0) 0x7f109011f010 con 0x7f10a006aac0 2023-10-25T00:25:04.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.457+0000 7f109dd9b640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1080065e60 0x7f1080068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:04.465 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.461+0000 7f109dd9b640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1080065e60 0x7f1080068320 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f10a00c1d30 tx=0x7f1088008460 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:04.466 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.461+0000 7f1096ffd640 1 -- 172.21.15.107:0/4212790201 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f10a00aa100 con 0x7f10a006aac0 2023-10-25T00:25:04.637 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:04.637+0000 7f10a4b05640 1 -- 172.21.15.107:0/4212790201 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "a_data", "pool2": "a_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f10a00aa100 con 0x7f10a006aac0 2023-10-25T00:25:05.309 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.308+0000 7f1096ffd640 1 -- 172.21.15.107:0/4212790201 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "a_data", "pool2": "a_data", "yes_i_really_really_mean_it": true}]=0 pool 'a_data' removed v77) v1 ==== 154+0+0 (secure 0 0 0) 0x7f10900eb500 con 0x7f10a006aac0 2023-10-25T00:25:05.309 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'a_data' removed 2023-10-25T00:25:05.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.312+0000 7f10a4b05640 1 -- 172.21.15.107:0/4212790201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1080065e60 msgr2=0x7f1080068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:05.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.312+0000 7f10a4b05640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1080065e60 0x7f1080068320 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f10a00c1d30 tx=0x7f1088008460 comp rx=0 tx=0).stop 2023-10-25T00:25:05.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.312+0000 7f10a4b05640 1 -- 172.21.15.107:0/4212790201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10a006aac0 msgr2=0x7f10a00aa3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:05.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.312+0000 7f10a4b05640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10a006aac0 0x7f10a00aa3e0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f109001cda0 tx=0x7f10900192a0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.328+0000 7f10a4b05640 1 -- 172.21.15.107:0/4212790201 shutdown_connections 2023-10-25T00:25:05.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.328+0000 7f10a4b05640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f1080065e60 0x7f1080068320 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.329 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.328+0000 7f10a4b05640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10a00ab2d0 0x7f10a01f0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.328+0000 7f10a4b05640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10a006cc30 0x7f10a006f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.328+0000 7f10a4b05640 1 --2- 172.21.15.107:0/4212790201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f10a006aac0 0x7f10a00aa3e0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.328+0000 7f10a4b05640 1 -- 172.21.15.107:0/4212790201 >> 172.21.15.107:0/4212790201 conn(0x7f10a00c27f0 msgr2=0x7f10a00c2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:05.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.328+0000 7f10a4b05640 1 -- 172.21.15.107:0/4212790201 shutdown_connections 2023-10-25T00:25:05.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.328+0000 7f10a4b05640 1 -- 172.21.15.107:0/4212790201 wait complete. 2023-10-25T00:25:05.363 INFO:tasks.cephfs.filesystem:Destroying file system b and related pools 2023-10-25T00:25:05.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:25:05.576 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff61d60640 1 Processor -- start 2023-10-25T00:25:05.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff61d60640 1 -- start start 2023-10-25T00:25:05.577 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff61d60640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7eff5c0aa330 0x7eff5c06e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:05.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff61d60640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c152d40 con 0x7eff5c0a9450 2023-10-25T00:25:05.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff61d60640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c06d660 con 0x7eff5c13bb00 2023-10-25T00:25:05.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff61d60640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff5c156100 con 0x7eff5c0aa330 2023-10-25T00:25:05.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5bfff640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7eff5c0a9450 0x7eff5c071a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50742/0 (socket says 172.21.15.107:50742) 2023-10-25T00:25:05.578 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5bfff640 1 -- 172.21.15.107:0/3924733327 learned_addr learned my addr 172.21.15.107:0/3924733327 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:25:05.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff53fff640 1 --2- 172.21.15.107:0/3924733327 >> v2:172.21.15.110:3300/0 conn(0x7eff5c0aa330 0x7eff5c06e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:05.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022081943 0 0) 0x7eff5c152d40 con 0x7eff5c0a9450 2023-10-25T00:25:05.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff48003530 con 0x7eff5c0a9450 2023-10-25T00:25:05.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896958092 0 0) 0x7eff48003530 con 0x7eff5c0a9450 2023-10-25T00:25:05.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 >> v2:172.21.15.110:3300/0 conn(0x7eff5c0aa330 msgr2=0x7eff5c06e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:05.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 --2- 172.21.15.107:0/3924733327 >> v2:172.21.15.110:3300/0 conn(0x7eff5c0aa330 0x7eff5c06e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 >> v1:172.21.15.110:6789/0 conn(0x7eff5c13bb00 legacy=0x7eff5c1531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:05.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c155e60 con 0x7eff5c0a9450 2023-10-25T00:25:05.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7eff54001a00 con 0x7eff5c0a9450 2023-10-25T00:25:05.582 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 >> v1:172.21.15.107:6789/0 conn(0x7eff5c0a9450 legacy=0x7eff5c071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:05.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 --2- 172.21.15.107:0/3924733327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 0x7eff48003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:05.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 --2- 172.21.15.107:0/3924733327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff480047d0 0x7eff48004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:05.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 --2- 172.21.15.107:0/3924733327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff48005190 0x7eff4800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:05.583 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c156100 con 0x7eff48003980 2023-10-25T00:25:05.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4800c0e0 con 0x7eff48005190 2023-10-25T00:25:05.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4800c2a0 con 0x7eff480047d0 2023-10-25T00:25:05.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5b7fe640 1 --2- 172.21.15.107:0/3924733327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 0x7eff48003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:05.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5bfff640 1 --2- 172.21.15.107:0/3924733327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff48005190 0x7eff4800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:05.584 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff53fff640 1 --2- 172.21.15.107:0/3924733327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff480047d0 0x7eff48004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:05.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5b7fe640 1 -- 172.21.15.107:0/3924733327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff480047d0 msgr2=0x7eff48004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:05.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5b7fe640 1 --2- 172.21.15.107:0/3924733327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff480047d0 0x7eff48004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5b7fe640 1 -- 172.21.15.107:0/3924733327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff48005190 msgr2=0x7eff4800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:05.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5b7fe640 1 --2- 172.21.15.107:0/3924733327 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff48005190 0x7eff4800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.585 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5b7fe640 1 -- 172.21.15.107:0/3924733327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff48003530 con 0x7eff48003980 2023-10-25T00:25:05.586 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.576+0000 7eff5bfff640 1 -- 172.21.15.107:0/3924733327 reap_dead start 2023-10-25T00:25:05.588 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.580+0000 7eff5b7fe640 1 --2- 172.21.15.107:0/3924733327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 0x7eff48003d60 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7eff4c001860 tx=0x7eff4c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79de9a29233d91c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.580+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c0ab600 con 0x7eff48003980 2023-10-25T00:25:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.580+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7eff4c0a7050 con 0x7eff48003980 2023-10-25T00:25:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.580+0000 7eff5affd640 1 -- 172.21.15.107:0/3924733327 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c0aba00 con 0x7eff48003980 2023-10-25T00:25:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.580+0000 7eff61d60640 1 -- 172.21.15.107:0/3924733327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 msgr2=0x7eff48003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:05.589 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.580+0000 7eff61d60640 1 --2- 172.21.15.107:0/3924733327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 0x7eff48003d60 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7eff4c001860 tx=0x7eff4c01c4e0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.580+0000 7eff61d60640 1 -- 172.21.15.107:0/3924733327 shutdown_connections 2023-10-25T00:25:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.580+0000 7eff61d60640 1 --2- 172.21.15.107:0/3924733327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 0x7eff48003d60 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.580+0000 7eff61d60640 1 -- 172.21.15.107:0/3924733327 >> 172.21.15.107:0/3924733327 conn(0x7eff5c070f60 msgr2=0x7eff5c071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.580+0000 7eff61d60640 1 -- 172.21.15.107:0/3924733327 shutdown_connections 2023-10-25T00:25:05.590 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.580+0000 7eff61d60640 1 -- 172.21.15.107:0/3924733327 wait complete. 2023-10-25T00:25:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff61d60640 1 Processor -- start 2023-10-25T00:25:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff61d60640 1 -- start start 2023-10-25T00:25:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff61d60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 0x7eff5c0abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff61d60640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff5c1357a0 0x7eff5c0ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff61d60640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff5c154890 0x7eff5c0ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:05.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff61d60640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c1525c0 con 0x7eff48003980 2023-10-25T00:25:05.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff61d60640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c0abff0 con 0x7eff5c154890 2023-10-25T00:25:05.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff61d60640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c1de130 con 0x7eff5c1357a0 2023-10-25T00:25:05.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff5b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 0x7eff5c0abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:05.592 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff5b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 0x7eff5c0abab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55304/0 (socket says 172.21.15.107:55304) 2023-10-25T00:25:05.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff5b7fe640 1 -- 172.21.15.107:0/3169452034 learned_addr learned my addr 172.21.15.107:0/3169452034 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:05.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff5bfff640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff5c154890 0x7eff5c0ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:05.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff53fff640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff5c1357a0 0x7eff5c0ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:05.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff5b7fe640 1 -- 172.21.15.107:0/3169452034 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff5c1357a0 msgr2=0x7eff5c0ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:05.593 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff5b7fe640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff5c1357a0 0x7eff5c0ac6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff5b7fe640 1 -- 172.21.15.107:0/3169452034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff5c154890 msgr2=0x7eff5c0ad0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:05.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff5b7fe640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff5c154890 0x7eff5c0ad0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:05.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff5b7fe640 1 -- 172.21.15.107:0/3169452034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c1555b0 con 0x7eff48003980 2023-10-25T00:25:05.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff5b7fe640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 0x7eff5c0abab0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0ae040 tx=0x7eff48005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:05.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff58ff9640 1 -- 172.21.15.107:0/3169452034 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c007240 con 0x7eff48003980 2023-10-25T00:25:05.594 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff61d60640 1 -- 172.21.15.107:0/3169452034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5c155890 con 0x7eff48003980 2023-10-25T00:25:05.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff58ff9640 1 -- 172.21.15.107:0/3169452034 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7eff4c0073a0 con 0x7eff48003980 2023-10-25T00:25:05.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff58ff9640 1 -- 172.21.15.107:0/3169452034 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c0b4400 con 0x7eff48003980 2023-10-25T00:25:05.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff61d60640 1 -- 172.21.15.107:0/3169452034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5c1de410 con 0x7eff48003980 2023-10-25T00:25:05.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.584+0000 7eff61d60640 1 -- 172.21.15.107:0/3169452034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5c155a20 con 0x7eff48003980 2023-10-25T00:25:05.595 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.588+0000 7eff58ff9640 1 -- 172.21.15.107:0/3169452034 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7eff4c0b2040 con 0x7eff48003980 2023-10-25T00:25:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.588+0000 7eff58ff9640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff3c065e90 0x7eff3c068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.588+0000 7eff53fff640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff3c065e90 0x7eff3c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:05.596 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.588+0000 7eff58ff9640 1 -- 172.21.15.107:0/3169452034 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 10113+0+0 (secure 0 0 0) 0x7eff4c11e6a0 con 0x7eff48003980 2023-10-25T00:25:05.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.588+0000 7eff53fff640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff3c065e90 0x7eff3c068350 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7eff5c139200 tx=0x7eff44008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:05.597 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:05.592+0000 7eff58ff9640 1 -- 172.21.15.107:0/3169452034 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7eff4c0eaeb0 con 0x7eff48003980 2023-10-25T00:25:06.218 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.216+0000 7eff61d60640 1 -- 172.21.15.107:0/3169452034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff5c0746e0 con 0x7eff48003980 2023-10-25T00:25:06.222 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.220+0000 7eff58ff9640 1 -- 172.21.15.107:0/3169452034 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) v1 ==== 95+0+4647 (secure 0 0 0) 0x7eff4c0b0090 con 0x7eff48003980 2023-10-25T00:25:06.223 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:25:06.223 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":242,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9606,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2839787144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2839787144},{"type":"v1","addr":"172.21.15.107:6837","nonce":2839787144}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":240},{"gid":9660,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/434729603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":434729603},{"type":"v1","addr":"172.21.15.107:6835","nonce":434729603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:25:06.223 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 242 2023-10-25T00:25:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 -- 172.21.15.107:0/3169452034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff3c065e90 msgr2=0x7eff3c068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff3c065e90 0x7eff3c068350 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7eff5c139200 tx=0x7eff44008040 comp rx=0 tx=0).stop 2023-10-25T00:25:06.234 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 -- 172.21.15.107:0/3169452034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 msgr2=0x7eff5c0abab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 0x7eff5c0abab0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7eff4c0ae040 tx=0x7eff48005800 comp rx=0 tx=0).stop 2023-10-25T00:25:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 -- 172.21.15.107:0/3169452034 shutdown_connections 2023-10-25T00:25:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7eff5c154890 0x7eff5c0ad0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7eff3c065e90 0x7eff3c068350 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7eff5c1357a0 0x7eff5c0ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.235 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 --2- 172.21.15.107:0/3169452034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff48003980 0x7eff5c0abab0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 -- 172.21.15.107:0/3169452034 >> 172.21.15.107:0/3169452034 conn(0x7eff5c070f60 msgr2=0x7eff5c0c2860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:06.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 -- 172.21.15.107:0/3169452034 shutdown_connections 2023-10-25T00:25:06.236 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.232+0000 7eff61d60640 1 -- 172.21.15.107:0/3169452034 wait complete. 2023-10-25T00:25:06.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:25:06.511 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.508+0000 7f409e5f6640 1 Processor -- start 2023-10-25T00:25:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.508+0000 7f409e5f6640 1 -- start start 2023-10-25T00:25:06.512 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f409e5f6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4098150930 0x7f40980a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f409e5f6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f40980c1570 0x7f4098069d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f409e5f6640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f409806a280 0x7f40981510c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f409e5f6640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40980a9b90 con 0x7f409806a280 2023-10-25T00:25:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f409e5f6640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40980c1390 con 0x7f40980c1570 2023-10-25T00:25:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f409e5f6640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4098154dd0 con 0x7f4098150930 2023-10-25T00:25:06.514 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f409cb93640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f409806a280 0x7f40981510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f409cb93640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f409806a280 0x7f40981510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55310/0 (socket says 172.21.15.107:55310) 2023-10-25T00:25:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f409cb93640 1 -- 172.21.15.107:0/2544026513 learned_addr learned my addr 172.21.15.107:0/2544026513 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f4097fff640 1 --2- 172.21.15.107:0/2544026513 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4098150930 0x7f40980a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f4097fff640 1 -- 172.21.15.107:0/2544026513 >> v2:172.21.15.110:3300/0 conn(0x7f40980c1570 msgr2=0x7f4098069d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-25T00:25:06.515 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f4097fff640 1 --2- 172.21.15.107:0/2544026513 >> v2:172.21.15.110:3300/0 conn(0x7f40980c1570 0x7f4098069d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f4097fff640 1 -- 172.21.15.107:0/2544026513 >> v2:172.21.15.107:3300/0 conn(0x7f409806a280 msgr2=0x7f40981510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:06.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f4097fff640 1 --2- 172.21.15.107:0/2544026513 >> v2:172.21.15.107:3300/0 conn(0x7f409806a280 0x7f40981510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f4097fff640 1 -- 172.21.15.107:0/2544026513 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40981550b0 con 0x7f4098150930 2023-10-25T00:25:06.516 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.512+0000 7f409cb93640 1 --2- 172.21.15.107:0/2544026513 >> v2:172.21.15.107:3300/0 conn(0x7f409806a280 0x7f40981510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-25T00:25:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f4097fff640 1 --2- 172.21.15.107:0/2544026513 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4098150930 0x7f40980a94b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40880097c0 tx=0x7f408801c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32503a3518faa93a server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f4096ffd640 1 -- 172.21.15.107:0/2544026513 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40880ab600 con 0x7f4098150930 2023-10-25T00:25:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f4096ffd640 1 -- 172.21.15.107:0/2544026513 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f40880a7050 con 0x7f4098150930 2023-10-25T00:25:06.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f4096ffd640 1 -- 172.21.15.107:0/2544026513 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40880ab9a0 con 0x7f4098150930 2023-10-25T00:25:06.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f409e5f6640 1 -- 172.21.15.107:0/2544026513 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4098150930 msgr2=0x7f40980a94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:06.518 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f409e5f6640 1 --2- 172.21.15.107:0/2544026513 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4098150930 0x7f40980a94b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40880097c0 tx=0x7f408801c460 comp rx=0 tx=0).stop 2023-10-25T00:25:06.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f409e5f6640 1 -- 172.21.15.107:0/2544026513 shutdown_connections 2023-10-25T00:25:06.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f409e5f6640 1 --2- 172.21.15.107:0/2544026513 >> v2:172.21.15.107:3300/0 conn(0x7f409806a280 0x7f40981510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f409e5f6640 1 --2- 172.21.15.107:0/2544026513 >> v2:172.21.15.110:3300/0 conn(0x7f40980c1570 0x7f4098069d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.519 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f409e5f6640 1 --2- 172.21.15.107:0/2544026513 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4098150930 0x7f40980a94b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f409e5f6640 1 -- 172.21.15.107:0/2544026513 >> 172.21.15.107:0/2544026513 conn(0x7f40980c65c0 msgr2=0x7f40980c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f409e5f6640 1 -- 172.21.15.107:0/2544026513 shutdown_connections 2023-10-25T00:25:06.520 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.516+0000 7f409e5f6640 1 -- 172.21.15.107:0/2544026513 wait complete. 2023-10-25T00:25:06.521 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f409e5f6640 1 Processor -- start 2023-10-25T00:25:06.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f409e5f6640 1 -- start start 2023-10-25T00:25:06.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f409e5f6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f409806a280 0x7f40980c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f409e5f6640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40980c1570 0x7f40981ec4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4097fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f409806a280 0x7f40980c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.522 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4097fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f409806a280 0x7f40980c5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55318/0 (socket says 172.21.15.107:55318) 2023-10-25T00:25:06.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4097fff640 1 -- 172.21.15.107:0/1586635295 learned_addr learned my addr 172.21.15.107:0/1586635295 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:06.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f409e5f6640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4098150930 0x7f40980b8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f409e5f6640 1 -- 172.21.15.107:0/1586635295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40980ba9b0 con 0x7f409806a280 2023-10-25T00:25:06.523 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f409e5f6640 1 -- 172.21.15.107:0/1586635295 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40981e8c90 con 0x7f40980c1570 2023-10-25T00:25:06.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f409e5f6640 1 -- 172.21.15.107:0/1586635295 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40981ec040 con 0x7f4098150930 2023-10-25T00:25:06.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f40977fe640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40980c1570 0x7f40981ec4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.524 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f409cb93640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4098150930 0x7f40980b8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4097fff640 1 -- 172.21.15.107:0/1586635295 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4098150930 msgr2=0x7f40980b8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:06.525 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4097fff640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4098150930 0x7f40980b8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4097fff640 1 -- 172.21.15.107:0/1586635295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40980c1570 msgr2=0x7f40981ec4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:06.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4097fff640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40980c1570 0x7f40981ec4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4097fff640 1 -- 172.21.15.107:0/1586635295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409000bd40 con 0x7f409806a280 2023-10-25T00:25:06.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4097fff640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f409806a280 0x7f40980c5a40 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f408801ab10 tx=0x7f4088019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:06.526 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4094ff9640 1 -- 172.21.15.107:0/1586635295 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40880b6bd0 con 0x7f409806a280 2023-10-25T00:25:06.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4094ff9640 1 -- 172.21.15.107:0/1586635295 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f40880ba070 con 0x7f409806a280 2023-10-25T00:25:06.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f4094ff9640 1 -- 172.21.15.107:0/1586635295 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40880c1480 con 0x7f409806a280 2023-10-25T00:25:06.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f409e5f6640 1 -- 172.21.15.107:0/1586635295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4088009420 con 0x7f409806a280 2023-10-25T00:25:06.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.520+0000 7f409e5f6640 1 -- 172.21.15.107:0/1586635295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40980c1cf0 con 0x7f409806a280 2023-10-25T00:25:06.527 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.524+0000 7f407e7fc640 1 -- 172.21.15.107:0/1586635295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4070005260 con 0x7f409806a280 2023-10-25T00:25:06.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.532+0000 7f4094ff9640 1 -- 172.21.15.107:0/1586635295 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f40880b2040 con 0x7f409806a280 2023-10-25T00:25:06.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.532+0000 7f4094ff9640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4084065e90 0x7f4084068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.534 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.532+0000 7f4094ff9640 1 -- 172.21.15.107:0/1586635295 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 10113+0+0 (secure 0 0 0) 0x7f40880bc070 con 0x7f409806a280 2023-10-25T00:25:06.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.532+0000 7f40977fe640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4084065e90 0x7f4084068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.532+0000 7f4094ff9640 1 -- 172.21.15.107:0/1586635295 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f408814e040 con 0x7f409806a280 2023-10-25T00:25:06.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.532+0000 7f40977fe640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4084065e90 0x7f4084068350 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f40980c6310 tx=0x7f408c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:06.804 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 -- 172.21.15.107:0/1586635295 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40700057e0 con 0x7f409806a280 2023-10-25T00:25:06.805 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f4094ff9640 1 -- 172.21.15.107:0/1586635295 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) v1 ==== 95+0+4647 (secure 0 0 0) 0x7f40880b0030 con 0x7f409806a280 2023-10-25T00:25:06.805 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:25:06.805 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":242,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":190},{"gid":9606,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2839787144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2839787144},{"type":"v1","addr":"172.21.15.107:6837","nonce":2839787144}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":240},{"gid":9660,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/434729603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":434729603},{"type":"v1","addr":"172.21.15.107:6835","nonce":434729603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-24T23:53:11.777452+0000","modified":"2023-10-25T00:21:22.342534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7483":{"gid":7483,"name":"b","rank":0,"incarnation":221,"state":"up:active","state_seq":17,"addr":"172.21.15.110:6834/47497412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":47497412},{"type":"v1","addr":"172.21.15.110:6834","nonce":47497412}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-25T00:25:06.806 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 242 2023-10-25T00:25:06.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 -- 172.21.15.107:0/1586635295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4084065e90 msgr2=0x7f4084068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:06.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4084065e90 0x7f4084068350 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f40980c6310 tx=0x7f408c008040 comp rx=0 tx=0).stop 2023-10-25T00:25:06.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 -- 172.21.15.107:0/1586635295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f409806a280 msgr2=0x7f40980c5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:06.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f409806a280 0x7f40980c5a40 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f408801ab10 tx=0x7f4088019200 comp rx=0 tx=0).stop 2023-10-25T00:25:06.807 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 -- 172.21.15.107:0/1586635295 shutdown_connections 2023-10-25T00:25:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f4084065e90 0x7f4084068350 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4098150930 0x7f40980b8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f40980c1570 0x7f40981ec4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 --2- 172.21.15.107:0/1586635295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f409806a280 0x7f40980c5a40 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 -- 172.21.15.107:0/1586635295 >> 172.21.15.107:0/1586635295 conn(0x7f40980c65c0 msgr2=0x7f4098152c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 -- 172.21.15.107:0/1586635295 shutdown_connections 2023-10-25T00:25:06.809 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.804+0000 7f407e7fc640 1 -- 172.21.15.107:0/1586635295 wait complete. 2023-10-25T00:25:06.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-25T00:25:06.930 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.928+0000 7f748c39e640 1 Processor -- start 2023-10-25T00:25:06.931 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.928+0000 7f748c39e640 1 -- start start 2023-10-25T00:25:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.928+0000 7f748c39e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7484152040 0x7f7484158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.932 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.928+0000 7f748c39e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74840adb80 0x7f7484071a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748c39e640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7484152420 con 0x7f7484071fc0 2023-10-25T00:25:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748c39e640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74841556e0 con 0x7f7484152040 2023-10-25T00:25:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748c39e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7484154d40 con 0x7f74840adb80 2023-10-25T00:25:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748a93b640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f7484071fc0 0x7f748406e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50754/0 (socket says 172.21.15.107:50754) 2023-10-25T00:25:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748a93b640 1 -- 172.21.15.107:0/285799146 learned_addr learned my addr 172.21.15.107:0/285799146 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:25:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748a13a640 1 --2- 172.21.15.107:0/285799146 >> v2:172.21.15.110:3300/0 conn(0x7f7484152040 0x7f7484158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f7489939640 1 --2- 172.21.15.107:0/285799146 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74840adb80 0x7f7484071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f7489138640 1 -- 172.21.15.107:0/285799146 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307652327 0 0) 0x7f7484152420 con 0x7f7484071fc0 2023-10-25T00:25:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f7489138640 1 -- 172.21.15.107:0/285799146 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7470003530 con 0x7f7484071fc0 2023-10-25T00:25:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748a13a640 1 -- 172.21.15.107:0/285799146 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74840adb80 msgr2=0x7f7484071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f7489138640 1 -- 172.21.15.107:0/285799146 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1823218019 0 0) 0x7f7470003530 con 0x7f7484071fc0 2023-10-25T00:25:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748a13a640 1 --2- 172.21.15.107:0/285799146 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74840adb80 0x7f7484071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748a13a640 1 -- 172.21.15.107:0/285799146 >> v1:172.21.15.107:6789/0 conn(0x7f7484071fc0 legacy=0x7f748406e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:06.935 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748a13a640 1 -- 172.21.15.107:0/285799146 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7484155020 con 0x7f7484152040 2023-10-25T00:25:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748a13a640 1 --2- 172.21.15.107:0/285799146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7484152040 0x7f7484158940 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f74800097c0 tx=0x7f748001c2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eed1e7b1724094ea server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f7489138640 1 -- 172.21.15.107:0/285799146 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c001a00 con 0x7f7484152040 2023-10-25T00:25:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f7489138640 1 -- 172.21.15.107:0/285799146 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f74800a7050 con 0x7f7484152040 2023-10-25T00:25:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f7489138640 1 -- 172.21.15.107:0/285799146 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74800ab600 con 0x7f7484152040 2023-10-25T00:25:06.937 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748c39e640 1 -- 172.21.15.107:0/285799146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7484152040 msgr2=0x7f7484158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748c39e640 1 --2- 172.21.15.107:0/285799146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7484152040 0x7f7484158940 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f74800097c0 tx=0x7f748001c2f0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748c39e640 1 -- 172.21.15.107:0/285799146 shutdown_connections 2023-10-25T00:25:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748c39e640 1 --2- 172.21.15.107:0/285799146 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f74840adb80 0x7f7484071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.938 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748c39e640 1 --2- 172.21.15.107:0/285799146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7484152040 0x7f7484158940 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748c39e640 1 -- 172.21.15.107:0/285799146 >> 172.21.15.107:0/285799146 conn(0x7f7484070f60 msgr2=0x7f7484071370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748c39e640 1 -- 172.21.15.107:0/285799146 shutdown_connections 2023-10-25T00:25:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.932+0000 7f748c39e640 1 -- 172.21.15.107:0/285799146 wait complete. 2023-10-25T00:25:06.939 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748c39e640 1 Processor -- start 2023-10-25T00:25:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748c39e640 1 -- start start 2023-10-25T00:25:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748c39e640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7484071fc0 0x7f74841ed560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.940 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748c39e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74840adb80 0x7f7484156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.941 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748c39e640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7484152040 0x7f7484155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748c39e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74840be0a0 con 0x7f74840adb80 2023-10-25T00:25:06.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748c39e640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74840bd310 con 0x7f7484152040 2023-10-25T00:25:06.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748c39e640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74840bd780 con 0x7f7484071fc0 2023-10-25T00:25:06.942 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f7489939640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74840adb80 0x7f7484156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748a13a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7484071fc0 0x7f74841ed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.943 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f7489939640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74840adb80 0x7f7484156f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55334/0 (socket says 172.21.15.107:55334) 2023-10-25T00:25:06.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f7489939640 1 -- 172.21.15.107:0/612429488 learned_addr learned my addr 172.21.15.107:0/612429488 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:06.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748a93b640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7484152040 0x7f7484155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.945 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f7489939640 1 -- 172.21.15.107:0/612429488 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7484071fc0 msgr2=0x7f74841ed560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f7489939640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7484071fc0 0x7f74841ed560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f7489939640 1 -- 172.21.15.107:0/612429488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7484152040 msgr2=0x7f7484155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f7489939640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7484152040 0x7f7484155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f7489939640 1 -- 172.21.15.107:0/612429488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7480009420 con 0x7f74840adb80 2023-10-25T00:25:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f7489939640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74840adb80 0x7f7484156f70 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f747400e930 tx=0x7f747400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f747affd640 1 -- 172.21.15.107:0/612429488 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474009690 con 0x7f74840adb80 2023-10-25T00:25:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f747affd640 1 -- 172.21.15.107:0/612429488 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f747400ca00 con 0x7f74840adb80 2023-10-25T00:25:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f747affd640 1 -- 172.21.15.107:0/612429488 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747400cd90 con 0x7f74840adb80 2023-10-25T00:25:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748c39e640 1 -- 172.21.15.107:0/612429488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74840bda40 con 0x7f74840adb80 2023-10-25T00:25:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.936+0000 7f748c39e640 1 -- 172.21.15.107:0/612429488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74840bcfa0 con 0x7f74840adb80 2023-10-25T00:25:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.940+0000 7f747affd640 1 -- 172.21.15.107:0/612429488 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f7474013470 con 0x7f74840adb80 2023-10-25T00:25:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.940+0000 7f747affd640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7468065dd0 0x7f7468068290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:06.948 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.940+0000 7f747affd640 1 -- 172.21.15.107:0/612429488 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 10113+0+0 (secure 0 0 0) 0x7f7474084a50 con 0x7f74840adb80 2023-10-25T00:25:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.940+0000 7f748a13a640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7468065dd0 0x7f7468068290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:06.949 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.940+0000 7f7478ff9640 1 -- 172.21.15.107:0/612429488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74840bd130 con 0x7f74840adb80 2023-10-25T00:25:06.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.952+0000 7f747affd640 1 -- 172.21.15.107:0/612429488 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f7474017080 con 0x7f74840adb80 2023-10-25T00:25:06.955 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:06.952+0000 7f748a13a640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7468065dd0 0x7f7468068290 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f74800ad000 tx=0x7f74800a6000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:07.170 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.168+0000 7f7478ff9640 1 -- 172.21.15.107:0/612429488 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74840746e0 con 0x7f74840adb80 2023-10-25T00:25:07.172 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.172+0000 7f747affd640 1 -- 172.21.15.107:0/612429488 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+24077 (secure 0 0 0) 0x7f7474021030 con 0x7f74840adb80 2023-10-25T00:25:07.173 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:25:07.173 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":77,"fsid":"c963b8f9-0f03-467c-8a02-12930711a2e8","created":"2023-10-24T23:52:38.577215+0000","modified":"2023-10-25T00:25:05.294109+0000","last_up_change":"2023-10-24T23:52:49.974405+0000","last_in_change":"2023-10-24T23:52:40.855520+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-24T23:52:51.349622+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-24T23:52:57.086491+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_metadata","create_time":"2023-10-24T23:53:09.786122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"b_data","create_time":"2023-10-24T23:53:10.766645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1a640883-d481-424e-ab6c-f1e1d6a15ea5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":37540},{"type":"v1","addr":"172.21.15.107:6813","nonce":37540}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":37540},{"type":"v1","addr":"172.21.15.107:6819","nonce":37540}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":37540},{"type":"v1","addr":"172.21.15.107:6830","nonce":37540}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":37540},{"type":"v1","addr":"172.21.15.107:6825","nonce":37540}]},"public_addr":"172.21.15.107:6813/37540","cluster_addr":"172.21.15.107:6819/37540","heartbeat_back_addr":"172.21.15.107:6830/37540","heartbeat_front_addr":"172.21.15.107:6825/37540","state":["exists","up"]},{"osd":1,"uuid":"bcf419d2-e458-4108-8a67-e253c573ad97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":37537},{"type":"v1","addr":"172.21.15.107:6801","nonce":37537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":37537},{"type":"v1","addr":"172.21.15.107:6803","nonce":37537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":37537},{"type":"v1","addr":"172.21.15.107:6807","nonce":37537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":37537},{"type":"v1","addr":"172.21.15.107:6805","nonce":37537}]},"public_addr":"172.21.15.107:6801/37537","cluster_addr":"172.21.15.107:6803/37537","heartbeat_back_addr":"172.21.15.107:6807/37537","heartbeat_front_addr":"172.21.15.107:6805/37537","state":["exists","up"]},{"osd":2,"uuid":"3083fe01-9371-44a6-bc38-cb318ad3d8df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":37539},{"type":"v1","addr":"172.21.15.107:6809","nonce":37539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6811","nonce":37539},{"type":"v1","addr":"172.21.15.107:6814","nonce":37539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6823","nonce":37539},{"type":"v1","addr":"172.21.15.107:6826","nonce":37539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":37539},{"type":"v1","addr":"172.21.15.107:6820","nonce":37539}]},"public_addr":"172.21.15.107:6809/37539","cluster_addr":"172.21.15.107:6814/37539","heartbeat_back_addr":"172.21.15.107:6826/37539","heartbeat_front_addr":"172.21.15.107:6820/37539","state":["exists","up"]},{"osd":3,"uuid":"5b68afcb-5b5c-4723-a177-8d01a6ab2983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":37538},{"type":"v1","addr":"172.21.15.107:6815","nonce":37538}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":37538},{"type":"v1","addr":"172.21.15.107:6821","nonce":37538}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6829","nonce":37538},{"type":"v1","addr":"172.21.15.107:6831","nonce":37538}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":37538},{"type":"v1","addr":"172.21.15.107:6827","nonce":37538}]},"public_addr":"172.21.15.107:6815/37538","cluster_addr":"172.21.15.107:6821/37538","heartbeat_back_addr":"172.21.15.107:6831/37538","heartbeat_front_addr":"172.21.15.107:6827/37538","state":["exists","up"]},{"osd":4,"uuid":"906cdc85-4970-46df-8824-cbf545d165be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":37764},{"type":"v1","addr":"172.21.15.110:6803","nonce":37764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":37764},{"type":"v1","addr":"172.21.15.110:6807","nonce":37764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":37764},{"type":"v1","addr":"172.21.15.110:6815","nonce":37764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":37764},{"type":"v1","addr":"172.21.15.110:6811","nonce":37764}]},"public_addr":"172.21.15.110:6803/37764","cluster_addr":"172.21.15.110:6807/37764","heartbeat_back_addr":"172.21.15.110:6815/37764","heartbeat_front_addr":"172.21.15.110:6811/37764","state":["exists","up"]},{"osd":5,"uuid":"e0f462d0-c6ff-41aa-9b25-2f586762248a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":37763},{"type":"v1","addr":"172.21.15.110:6817","nonce":37763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6818","nonce":37763},{"type":"v1","addr":"172.21.15.110:6819","nonce":37763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6822","nonce":37763},{"type":"v1","addr":"172.21.15.110:6823","nonce":37763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6820","nonce":37763},{"type":"v1","addr":"172.21.15.110:6821","nonce":37763}]},"public_addr":"172.21.15.110:6817/37763","cluster_addr":"172.21.15.110:6819/37763","heartbeat_back_addr":"172.21.15.110:6823/37763","heartbeat_front_addr":"172.21.15.110:6821/37763","state":["exists","up"]},{"osd":6,"uuid":"a2702df0-c858-4ffd-ac62-6a15da158e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":37765},{"type":"v1","addr":"172.21.15.110:6802","nonce":37765}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":37765},{"type":"v1","addr":"172.21.15.110:6806","nonce":37765}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":37765},{"type":"v1","addr":"172.21.15.110:6814","nonce":37765}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":37765},{"type":"v1","addr":"172.21.15.110:6810","nonce":37765}]},"public_addr":"172.21.15.110:6802/37765","cluster_addr":"172.21.15.110:6806/37765","heartbeat_back_addr":"172.21.15.110:6814/37765","heartbeat_front_addr":"172.21.15.110:6810/37765","state":["exists","up"]},{"osd":7,"uuid":"d6cdb9b9-7f96-400e-afbd-522f3d3ff856","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6824","nonce":37766},{"type":"v1","addr":"172.21.15.110:6825","nonce":37766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6826","nonce":37766},{"type":"v1","addr":"172.21.15.110:6827","nonce":37766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6830","nonce":37766},{"type":"v1","addr":"172.21.15.110:6831","nonce":37766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6828","nonce":37766},{"type":"v1","addr":"172.21.15.110:6829","nonce":37766}]},"public_addr":"172.21.15.110:6825/37766","cluster_addr":"172.21.15.110:6827/37766","heartbeat_back_addr":"172.21.15.110:6831/37766","heartbeat_front_addr":"172.21.15.110:6829/37766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.442521+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.431189+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.532159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.529134+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.419266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.408324+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-24T23:52:46.489745+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.2b","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6835/2963448958":"2023-10-26T00:25:02.250896+0000","172.21.15.107:6834/2963448958":"2023-10-26T00:25:02.250896+0000","172.21.15.107:6837/3527905219":"2023-10-26T00:24:34.267980+0000","172.21.15.107:6836/3527905219":"2023-10-26T00:24:34.267980+0000","172.21.15.107:6835/1528689730":"2023-10-26T00:22:49.197964+0000","172.21.15.107:6834/1528689730":"2023-10-26T00:22:49.197964+0000","172.21.15.107:6835/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6837/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.110:6834/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.110:6832/3771564389":"2023-10-26T00:20:19.149072+0000","172.21.15.107:6835/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.107:6837/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.110:6834/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.110:6832/985135365":"2023-10-26T00:17:39.037219+0000","172.21.15.107:6835/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.107:6834/208713198":"2023-10-26T00:17:24.032902+0000","172.21.15.110:6833/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6836/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6836/418589140":"2023-10-26T00:18:49.118366+0000","172.21.15.107:6834/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6835/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6836/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.110:6834/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6835/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.110:6835/1619327197":"2023-10-26T00:15:22.673615+0000","172.21.15.107:6834/1381726420":"2023-10-26T00:01:28.521014+0000","172.21.15.107:6837/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.107:6837/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6832/1101018338":"2023-10-25T23:59:48.457105+0000","172.21.15.107:6837/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6833/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.110:6833/898258574":"2023-10-26T00:13:18.883076+0000","172.21.15.107:6834/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.110:6835/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6836/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6836/1555840329":"2023-10-26T00:05:18.610829+0000","172.21.15.110:6834/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.110:6835/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.107:6834/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.107:6834/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.110:6833/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.107:6837/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6834/2970408860":"2023-10-26T00:19:04.122659+0000","172.21.15.110:6833/2480841919":"2023-10-25T23:55:58.372958+0000","172.21.15.110:6834/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.110:6832/201284779":"2023-10-26T00:02:23.541454+0000","172.21.15.107:6836/838886697":"2023-10-25T23:56:18.381650+0000","172.21.15.107:6837/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.107:6837/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.110:6833/2348043270":"2023-10-25T23:58:53.436687+0000","172.21.15.107:6837/3370462806":"2023-10-26T00:12:33.862408+0000","172.21.15.107:6835/1220982369":"2023-10-25T23:55:23.361145+0000","172.21.15.107:6837/204494054":"2023-10-25T23:58:03.418411+0000","172.21.15.107:6837/175581297":"2023-10-26T00:15:18.921240+0000","172.21.15.107:6835/2856791739":"2023-10-26T00:13:48.893880+0000","172.21.15.110:6835/1031240534":"2023-10-25T23:57:23.404313+0000","172.21.15.110:6835/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.107:6836/657626630":"2023-10-26T00:21:19.167088+0000","172.21.15.107:6834/1004926771":"2023-10-25T23:57:28.406683+0000","172.21.15.110:6832/3721101226":"2023-10-25T23:58:03.418295+0000","172.21.15.107:6835/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6835/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.110:6832/2577776351":"2023-10-26T00:04:34.728765+0000","172.21.15.110:6833/326715537":"2023-10-26T00:11:03.824824+0000","172.21.15.107:6835/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6836/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6834/3474118649":"2023-10-26T00:11:18.831318+0000","172.21.15.107:6836/1292188015":"2023-10-26T00:00:48.479959+0000","172.21.15.110:6834/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6837/319918260":"2023-10-26T00:03:53.579302+0000","172.21.15.107:6836/3103569740":"2023-10-26T00:01:58.532147+0000","172.21.15.110:6835/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6834/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/3396644475":"2023-10-26T00:19:44.136579+0000","172.21.15.107:6835/4180698155":"2023-10-26T00:04:28.591981+0000","172.21.15.107:6834/2187059523":"2023-10-26T00:21:39.173084+0000","172.21.15.107:6835/1246928633":"2023-10-26T00:05:13.609231+0000","172.21.15.110:6832/174863522":"2023-10-26T00:05:53.622773+0000","172.21.15.107:6835/1508236506":"2023-10-25T23:58:58.439244+0000","172.21.15.107:6836/780830443":"2023-10-26T00:06:23.634894+0000","172.21.15.110:6833/2150827048":"2023-10-26T00:07:48.667043+0000","172.21.15.107:6835/913067382":"2023-10-26T00:08:08.675592+0000","172.21.15.107:6836/1317796403":"2023-10-26T00:08:23.682615+0000","172.21.15.107:6834/2276136116":"2023-10-26T00:09:18.706706+0000","172.21.15.107:6834/3232233949":"2023-10-26T00:03:28.568587+0000","172.21.15.107:6836/727042497":"2023-10-25T23:54:13.340839+0000","172.21.15.107:6836/1788268787":"2023-10-26T00:10:38.784519+0000","172.21.15.110:6835/3836115666":"2023-10-26T00:08:58.700067+0000","172.21.15.107:6837/1788268787":"2023-10-26T00:10:38.784519+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-25T00:25:07.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 -- 172.21.15.107:0/612429488 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7468065dd0 msgr2=0x7f7468068290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7468065dd0 0x7f7468068290 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f74800ad000 tx=0x7f74800a6000 comp rx=0 tx=0).stop 2023-10-25T00:25:07.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 -- 172.21.15.107:0/612429488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74840adb80 msgr2=0x7f7484156f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74840adb80 0x7f7484156f70 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f747400e930 tx=0x7f747400edf0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.176 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 -- 172.21.15.107:0/612429488 shutdown_connections 2023-10-25T00:25:07.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7484152040 0x7f7484155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74840adb80 0x7f7484156f70 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f7468065dd0 0x7f7468068290 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 --2- 172.21.15.107:0/612429488 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7484071fc0 0x7f74841ed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 -- 172.21.15.107:0/612429488 >> 172.21.15.107:0/612429488 conn(0x7f7484070f60 msgr2=0x7f748406c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:07.177 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 -- 172.21.15.107:0/612429488 shutdown_connections 2023-10-25T00:25:07.178 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.176+0000 7f7478ff9640 1 -- 172.21.15.107:0/612429488 wait complete. 2023-10-25T00:25:07.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail b 2023-10-25T00:25:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.316+0000 7f03d1373640 1 Processor -- start 2023-10-25T00:25:07.320 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.316+0000 7f03d1373640 1 -- start start 2023-10-25T00:25:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03d1373640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f03cc155990 0x7f03cc14e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03d1373640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc0721e0 con 0x7f03cc154ea0 2023-10-25T00:25:07.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03d1373640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc0709d0 con 0x7f03cc0714b0 2023-10-25T00:25:07.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03d1373640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03cc0abe50 con 0x7f03cc155990 2023-10-25T00:25:07.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03caffd640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f03cc154ea0 0x7f03cc0a9bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50768/0 (socket says 172.21.15.107:50768) 2023-10-25T00:25:07.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03caffd640 1 -- 172.21.15.107:0/2553590671 learned_addr learned my addr 172.21.15.107:0/2553590671 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:25:07.323 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733896824 0 0) 0x7f03cc0721e0 con 0x7f03cc154ea0 2023-10-25T00:25:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03cb7fe640 1 --2- 172.21.15.107:0/2553590671 >> v2:172.21.15.110:3300/0 conn(0x7f03cc155990 0x7f03cc14e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b4005740 con 0x7f03cc154ea0 2023-10-25T00:25:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 386490626 0 0) 0x7f03b4005740 con 0x7f03cc154ea0 2023-10-25T00:25:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 >> v2:172.21.15.110:3300/0 conn(0x7f03cc155990 msgr2=0x7f03cc14e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 --2- 172.21.15.107:0/2553590671 >> v2:172.21.15.110:3300/0 conn(0x7f03cc155990 0x7f03cc14e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 >> v1:172.21.15.110:6789/0 conn(0x7f03cc0714b0 legacy=0x7f03cc155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc070cb0 con 0x7f03cc154ea0 2023-10-25T00:25:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f03bc001680 con 0x7f03cc154ea0 2023-10-25T00:25:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 >> v1:172.21.15.107:6789/0 conn(0x7f03cc154ea0 legacy=0x7f03cc0a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03b4005b90 0x7f03b4005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.325 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03b40069e0 0x7f03b4006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b40074c0 0x7f03b400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b400e340 con 0x7f03b40074c0 2023-10-25T00:25:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b400e4b0 con 0x7f03b40069e0 2023-10-25T00:25:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b400e670 con 0x7f03b4005b90 2023-10-25T00:25:07.326 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03cb7fe640 1 --2- 172.21.15.107:0/2553590671 >> v2:172.21.15.110:3300/0 conn(0x7f03cc155990 0x7f03cc14e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:25:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03cb7fe640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b40074c0 0x7f03b400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03ca7fc640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03b40069e0 0x7f03b4006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03caffd640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03b4005b90 0x7f03b4005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.327 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03cb7fe640 1 -- 172.21.15.107:0/2553590671 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03b4005b90 msgr2=0x7f03b4005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03cb7fe640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03b4005b90 0x7f03b4005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03cb7fe640 1 -- 172.21.15.107:0/2553590671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03b40069e0 msgr2=0x7f03b4006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03cb7fe640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03b40069e0 0x7f03b4006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03cb7fe640 1 -- 172.21.15.107:0/2553590671 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b4005740 con 0x7f03b40074c0 2023-10-25T00:25:07.328 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.320+0000 7f03caffd640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03b4005b90 0x7f03b4005f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:07.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03cb7fe640 1 -- 172.21.15.107:0/2553590671 reap_dead start 2023-10-25T00:25:07.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03cb7fe640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b40074c0 0x7f03b400dba0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f03c40073b0 tx=0x7f03c401cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f9d267d0e272ab server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:07.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c40a5e00 con 0x7f03b40074c0 2023-10-25T00:25:07.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f03c40ad440 con 0x7f03b40074c0 2023-10-25T00:25:07.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03c9ffb640 1 -- 172.21.15.107:0/2553590671 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c40ad7f0 con 0x7f03b40074c0 2023-10-25T00:25:07.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 -- 172.21.15.107:0/2553590671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b40074c0 msgr2=0x7f03b400dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b40074c0 0x7f03b400dba0 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f03c40073b0 tx=0x7f03c401cf00 comp rx=0 tx=0).stop 2023-10-25T00:25:07.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 -- 172.21.15.107:0/2553590671 shutdown_connections 2023-10-25T00:25:07.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 --2- 172.21.15.107:0/2553590671 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03b40074c0 0x7f03b400dba0 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 -- 172.21.15.107:0/2553590671 >> 172.21.15.107:0/2553590671 conn(0x7f03cc0c4e30 msgr2=0x7f03cc0c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:07.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 -- 172.21.15.107:0/2553590671 shutdown_connections 2023-10-25T00:25:07.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 -- 172.21.15.107:0/2553590671 wait complete. 2023-10-25T00:25:07.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 Processor -- start 2023-10-25T00:25:07.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 -- start start 2023-10-25T00:25:07.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03b40074c0 0x7f03cc1e5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc13af00 0x7f03cc139780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03cc139cc0 0x7f03cc1436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc153a80 con 0x7f03cc13af00 2023-10-25T00:25:07.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc152a20 con 0x7f03cc139cc0 2023-10-25T00:25:07.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc152d00 con 0x7f03b40074c0 2023-10-25T00:25:07.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03ca7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc13af00 0x7f03cc139780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03ca7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc13af00 0x7f03cc139780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55352/0 (socket says 172.21.15.107:55352) 2023-10-25T00:25:07.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03ca7fc640 1 -- 172.21.15.107:0/818419469 learned_addr learned my addr 172.21.15.107:0/818419469 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:07.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03caffd640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03b40074c0 0x7f03cc1e5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03cb7fe640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03cc139cc0 0x7f03cc1436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03ca7fc640 1 -- 172.21.15.107:0/818419469 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03b40074c0 msgr2=0x7f03cc1e5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03ca7fc640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03b40074c0 0x7f03cc1e5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03ca7fc640 1 -- 172.21.15.107:0/818419469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03cc139cc0 msgr2=0x7f03cc1436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03ca7fc640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03cc139cc0 0x7f03cc1436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03ca7fc640 1 -- 172.21.15.107:0/818419469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc152fe0 con 0x7f03cc13af00 2023-10-25T00:25:07.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03caffd640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03b40074c0 0x7f03cc1e5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-25T00:25:07.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03ca7fc640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc13af00 0x7f03cc139780 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f03c000e490 tx=0x7f03c000e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:07.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03b37fe640 1 -- 172.21.15.107:0/818419469 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0017070 con 0x7f03cc13af00 2023-10-25T00:25:07.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03b37fe640 1 -- 172.21.15.107:0/818419469 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f03c0003a80 con 0x7f03cc13af00 2023-10-25T00:25:07.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 -- 172.21.15.107:0/818419469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc1532c0 con 0x7f03cc13af00 2023-10-25T00:25:07.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03b37fe640 1 -- 172.21.15.107:0/818419469 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0003c50 con 0x7f03cc13af00 2023-10-25T00:25:07.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.324+0000 7f03d1373640 1 -- 172.21.15.107:0/818419469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03cc13a170 con 0x7f03cc13af00 2023-10-25T00:25:07.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.328+0000 7f03d1373640 1 -- 172.21.15.107:0/818419469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03cc1433e0 con 0x7f03cc13af00 2023-10-25T00:25:07.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.328+0000 7f03b37fe640 1 -- 172.21.15.107:0/818419469 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f03c0004050 con 0x7f03cc13af00 2023-10-25T00:25:07.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.328+0000 7f03b37fe640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03b8065e90 0x7f03b8068350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.328+0000 7f03b37fe640 1 -- 172.21.15.107:0/818419469 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 10113+0+0 (secure 0 0 0) 0x7f03c0083c80 con 0x7f03cc13af00 2023-10-25T00:25:07.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.328+0000 7f03caffd640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03b8065e90 0x7f03b8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.340 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.328+0000 7f03caffd640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03b8065e90 0x7f03b8068350 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f03bc000f70 tx=0x7f03bc0015a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:07.341 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.336+0000 7f03b37fe640 1 -- 172.21.15.107:0/818419469 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f03cc1433e0 con 0x7f03cc13af00 2023-10-25T00:25:07.517 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.516+0000 7f03d1373640 1 -- 172.21.15.107:0/818419469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "b"} v 0) v1 -- 0x7f03cc1433e0 con 0x7f03cc13af00 2023-10-25T00:25:07.535 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.532+0000 7f03b37fe640 1 -- 172.21.15.107:0/818419469 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "b"}]=0 b marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v243) v1 ==== 151+0+0 (secure 0 0 0) 0x7f03cc1433e0 con 0x7f03cc13af00 2023-10-25T00:25:07.535 INFO:teuthology.orchestra.run.smithi107.stderr:b marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-25T00:25:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 -- 172.21.15.107:0/818419469 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03b8065e90 msgr2=0x7f03b8068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.537 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03b8065e90 0x7f03b8068350 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f03bc000f70 tx=0x7f03bc0015a0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 -- 172.21.15.107:0/818419469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc13af00 msgr2=0x7f03cc139780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc13af00 0x7f03cc139780 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f03c000e490 tx=0x7f03c000e950 comp rx=0 tx=0).stop 2023-10-25T00:25:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 -- 172.21.15.107:0/818419469 shutdown_connections 2023-10-25T00:25:07.538 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03cc139cc0 0x7f03cc1436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03cc13af00 0x7f03cc139780 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f03b8065e90 0x7f03b8068350 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 --2- 172.21.15.107:0/818419469 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f03b40074c0 0x7f03cc1e5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 -- 172.21.15.107:0/818419469 >> 172.21.15.107:0/818419469 conn(0x7f03cc0c4e30 msgr2=0x7f03cc14e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:07.539 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 -- 172.21.15.107:0/818419469 shutdown_connections 2023-10-25T00:25:07.540 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.536+0000 7f03d1373640 1 -- 172.21.15.107:0/818419469 wait complete. 2023-10-25T00:25:07.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm b --yes-i-really-mean-it 2023-10-25T00:25:07.574 INFO:tasks.ceph.mds.b.smithi110.stdout:starting mds.b at 2023-10-25T00:25:07.657 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87fc39b640 1 Processor -- start 2023-10-25T00:25:07.658 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87fc39b640 1 -- start start 2023-10-25T00:25:07.660 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87fc39b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f40abf70 0x7f87f406adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87fc39b640 1 -- --> v1:172.21.15.107:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f40ac380 con 0x7f87f41526d0 2023-10-25T00:25:07.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87fc39b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f4069d40 con 0x7f87f414c6b0 2023-10-25T00:25:07.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87fc39b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40a9b60 con 0x7f87f40abf70 2023-10-25T00:25:07.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9936640 1 --1- >> v1:172.21.15.107:6789/0 conn(0x7f87f41526d0 0x7f87f40ab860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.107:6789/0 says I am v1:172.21.15.107:50772/0 (socket says 172.21.15.107:50772) 2023-10-25T00:25:07.661 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9936640 1 -- 172.21.15.107:0/321828991 learned_addr learned my addr 172.21.15.107:0/321828991 (peer_addr_for_me v1:172.21.15.107:0/0) 2023-10-25T00:25:07.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87fa938640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f40abf70 0x7f87f406adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 <== mon.0 v1:172.21.15.107:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382403533 0 0) 0x7f87f40ac380 con 0x7f87f41526d0 2023-10-25T00:25:07.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 --> v1:172.21.15.107:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e0003530 con 0x7f87f41526d0 2023-10-25T00:25:07.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360362570 0 0) 0x7f87f4069d40 con 0x7f87f414c6b0 2023-10-25T00:25:07.662 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f40ac380 con 0x7f87f414c6b0 2023-10-25T00:25:07.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 <== mon.0 v1:172.21.15.107:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 197652564 0 0) 0x7f87e0003530 con 0x7f87f41526d0 2023-10-25T00:25:07.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f40abf70 msgr2=0x7f87f406adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:07.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f40abf70 0x7f87f406adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 >> v1:172.21.15.110:6789/0 conn(0x7f87f414c6b0 legacy=0x7f87f40a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.663 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 --> v1:172.21.15.107:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f406a000 con 0x7f87f41526d0 2023-10-25T00:25:07.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 <== mon.0 v1:172.21.15.107:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3639825684 0 0) 0x7f87e4001680 con 0x7f87f41526d0 2023-10-25T00:25:07.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 >> v1:172.21.15.107:6789/0 conn(0x7f87f41526d0 legacy=0x7f87f40ab860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 0x7f87e0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.664 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f40abf70 0x7f87e00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.656+0000 7f87f9135640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e0004d10 0x7f87e0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40a9b60 con 0x7f87e0003b40 2023-10-25T00:25:07.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e000ba90 con 0x7f87e0004d10 2023-10-25T00:25:07.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e000bc50 con 0x7f87f40abf70 2023-10-25T00:25:07.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fa137640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 0x7f87e0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.665 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fa938640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e0004d10 0x7f87e0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87f9936640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f40abf70 0x7f87e00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fa137640 1 -- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f40abf70 msgr2=0x7f87e00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:07.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fa137640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f40abf70 0x7f87e00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fa137640 1 -- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e0004d10 msgr2=0x7f87e0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:07.666 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fa137640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87e0004d10 0x7f87e0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fa137640 1 -- 172.21.15.107:0/321828991 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e0003530 con 0x7f87e0003b40 2023-10-25T00:25:07.667 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87f9936640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f40abf70 0x7f87e00047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:07.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fa137640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 0x7f87e0003f40 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f87f00018d0 tx=0x7f87f001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eafe45430af5118c server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:07.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f00052d0 con 0x7f87e0003b40 2023-10-25T00:25:07.670 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f87f0005450 con 0x7f87e0003b40 2023-10-25T00:25:07.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87f9135640 1 -- 172.21.15.107:0/321828991 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0003860 con 0x7f87e0003b40 2023-10-25T00:25:07.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fc39b640 1 -- 172.21.15.107:0/321828991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 msgr2=0x7f87e0003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fc39b640 1 --2- 172.21.15.107:0/321828991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 0x7f87e0003f40 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f87f00018d0 tx=0x7f87f001c530 comp rx=0 tx=0).stop 2023-10-25T00:25:07.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fa938640 1 -- 172.21.15.107:0/321828991 reap_dead start 2023-10-25T00:25:07.671 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fc39b640 1 -- 172.21.15.107:0/321828991 shutdown_connections 2023-10-25T00:25:07.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fc39b640 1 -- 172.21.15.107:0/321828991 >> 172.21.15.107:0/321828991 conn(0x7f87f40c4e30 msgr2=0x7f87f40c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:07.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fc39b640 1 -- 172.21.15.107:0/321828991 shutdown_connections 2023-10-25T00:25:07.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.660+0000 7f87fc39b640 1 -- 172.21.15.107:0/321828991 wait complete. 2023-10-25T00:25:07.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fc39b640 1 Processor -- start 2023-10-25T00:25:07.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fc39b640 1 -- start start 2023-10-25T00:25:07.672 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fc39b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 0x7f87f413de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fc39b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f413e350 0x7f87f413d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fc39b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87f40c2050 0x7f87f413d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fc39b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41eb010 con 0x7f87e0003b40 2023-10-25T00:25:07.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fc39b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f41403a0 con 0x7f87f40c2050 2023-10-25T00:25:07.673 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fc39b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4140680 con 0x7f87f413e350 2023-10-25T00:25:07.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fa137640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 0x7f87f413de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fa137640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 0x7f87f413de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55362/0 (socket says 172.21.15.107:55362) 2023-10-25T00:25:07.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fa137640 1 -- 172.21.15.107:0/4134006543 learned_addr learned my addr 172.21.15.107:0/4134006543 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:07.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87f9936640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f413e350 0x7f87f413d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.674 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fa938640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87f40c2050 0x7f87f413d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fa137640 1 -- 172.21.15.107:0/4134006543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f413e350 msgr2=0x7f87f413d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fa137640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f413e350 0x7f87f413d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fa137640 1 -- 172.21.15.107:0/4134006543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87f40c2050 msgr2=0x7f87f413d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:07.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fa137640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87f40c2050 0x7f87f413d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:07.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fa137640 1 -- 172.21.15.107:0/4134006543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4140960 con 0x7f87e0003b40 2023-10-25T00:25:07.675 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87f9936640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f413e350 0x7f87f413d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:07.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fa938640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87f40c2050 0x7f87f413d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:07.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fa137640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 0x7f87f413de10 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f87f00ae040 tx=0x7f87e0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:07.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87eaffd640 1 -- 172.21.15.107:0/4134006543 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f00050d0 con 0x7f87e0003b40 2023-10-25T00:25:07.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87eaffd640 1 -- 172.21.15.107:0/4134006543 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f87f0005230 con 0x7f87e0003b40 2023-10-25T00:25:07.676 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fc39b640 1 -- 172.21.15.107:0/4134006543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f4140c40 con 0x7f87e0003b40 2023-10-25T00:25:07.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87eaffd640 1 -- 172.21.15.107:0/4134006543 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0006a20 con 0x7f87e0003b40 2023-10-25T00:25:07.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fc39b640 1 -- 172.21.15.107:0/4134006543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f41eac30 con 0x7f87e0003b40 2023-10-25T00:25:07.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.664+0000 7f87fc39b640 1 -- 172.21.15.107:0/4134006543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f4140dd0 con 0x7f87e0003b40 2023-10-25T00:25:07.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.668+0000 7f87eaffd640 1 -- 172.21.15.107:0/4134006543 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f87f00b2040 con 0x7f87e0003b40 2023-10-25T00:25:07.677 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.668+0000 7f87eaffd640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87d8065e60 0x7f87d8068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.668+0000 7f87f9936640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87d8065e60 0x7f87d8068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.668+0000 7f87eaffd640 1 -- 172.21.15.107:0/4134006543 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 10199+0+0 (secure 0 0 0) 0x7f87f011e360 con 0x7f87e0003b40 2023-10-25T00:25:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.668+0000 7f87f9936640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87d8065e60 0x7f87d8068320 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f87e4000c10 tx=0x7f87e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:07.678 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.672+0000 7f87eaffd640 1 -- 172.21.15.107:0/4134006543 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f87f00eaa00 con 0x7f87e0003b40 2023-10-25T00:25:07.849 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:07.848+0000 7f87fc39b640 1 -- 172.21.15.107:0/4134006543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "b", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f87f4073dc0 con 0x7f87e0003b40 2023-10-25T00:25:08.550 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.548+0000 7f87eaffd640 1 -- 172.21.15.107:0/4134006543 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "b", "yes_i_really_mean_it": true}]=0 v244) v1 ==== 99+0+0 (secure 0 0 0) 0x7f87f00c00a0 con 0x7f87e0003b40 2023-10-25T00:25:08.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 -- 172.21.15.107:0/4134006543 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87d8065e60 msgr2=0x7f87d8068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:08.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87d8065e60 0x7f87d8068320 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f87e4000c10 tx=0x7f87e4009040 comp rx=0 tx=0).stop 2023-10-25T00:25:08.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 -- 172.21.15.107:0/4134006543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 msgr2=0x7f87f413de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:08.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 0x7f87f413de10 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f87f00ae040 tx=0x7f87e0005800 comp rx=0 tx=0).stop 2023-10-25T00:25:08.554 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 -- 172.21.15.107:0/4134006543 shutdown_connections 2023-10-25T00:25:08.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f87f40c2050 0x7f87f413d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:08.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f87d8065e60 0x7f87d8068320 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:08.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f87f413e350 0x7f87f413d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:08.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 --2- 172.21.15.107:0/4134006543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87e0003b40 0x7f87f413de10 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:08.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 -- 172.21.15.107:0/4134006543 >> 172.21.15.107:0/4134006543 conn(0x7f87f40c4e30 msgr2=0x7f87f40a9450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:08.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 -- 172.21.15.107:0/4134006543 shutdown_connections 2023-10-25T00:25:08.556 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.552+0000 7f87fc39b640 1 -- 172.21.15.107:0/4134006543 wait complete. 2023-10-25T00:25:08.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm b_metadata b_metadata --yes-i-really-really-mean-it 2023-10-25T00:25:08.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9088a83640 1 Processor -- start 2023-10-25T00:25:08.685 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9088a83640 1 -- start start 2023-10-25T00:25:08.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9088a83640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f90840ab2d0 0x7f90840a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:08.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9088a83640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f908406aac0 0x7f90840c0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:08.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9088a83640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f908406cc30 0x7f90840c0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:08.687 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9088a83640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90840c1390 con 0x7f908406aac0 2023-10-25T00:25:08.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9088a83640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9084156880 con 0x7f90840ab2d0 2023-10-25T00:25:08.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9088a83640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9084155f50 con 0x7f908406cc30 2023-10-25T00:25:08.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9081d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f908406aac0 0x7f90840c0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:08.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9081d9b640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f908406aac0 0x7f90840c0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55372/0 (socket says 172.21.15.107:55372) 2023-10-25T00:25:08.688 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9081d9b640 1 -- 172.21.15.107:0/625750898 learned_addr learned my addr 172.21.15.107:0/625750898 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:08.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f908259c640 1 --2- 172.21.15.107:0/625750898 >> v2:172.21.15.110:3300/0 conn(0x7f90840ab2d0 0x7f90840a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:08.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9082d9d640 1 --2- 172.21.15.107:0/625750898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f908406cc30 0x7f90840c0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:08.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9081d9b640 1 -- 172.21.15.107:0/625750898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f908406cc30 msgr2=0x7f90840c0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:08.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9081d9b640 1 --2- 172.21.15.107:0/625750898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f908406cc30 0x7f90840c0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:08.689 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9081d9b640 1 -- 172.21.15.107:0/625750898 >> v2:172.21.15.110:3300/0 conn(0x7f90840ab2d0 msgr2=0x7f90840a4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:08.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9081d9b640 1 --2- 172.21.15.107:0/625750898 >> v2:172.21.15.110:3300/0 conn(0x7f90840ab2d0 0x7f90840a4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:08.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9081d9b640 1 -- 172.21.15.107:0/625750898 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9084156230 con 0x7f908406aac0 2023-10-25T00:25:08.690 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.684+0000 7f9082d9d640 1 --2- 172.21.15.107:0/625750898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f908406cc30 0x7f90840c0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:08.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9081d9b640 1 --2- 172.21.15.107:0/625750898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f908406aac0 0x7f90840c0910 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f90840abef0 tx=0x7f907401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ec56d91ec58d8b server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:08.693 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f908159a640 1 -- 172.21.15.107:0/625750898 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90740ac630 con 0x7f908406aac0 2023-10-25T00:25:08.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f908159a640 1 -- 172.21.15.107:0/625750898 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f90740ac790 con 0x7f908406aac0 2023-10-25T00:25:08.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f908159a640 1 -- 172.21.15.107:0/625750898 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90740acb40 con 0x7f908406aac0 2023-10-25T00:25:08.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 -- 172.21.15.107:0/625750898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f908406aac0 msgr2=0x7f90840c0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:08.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 --2- 172.21.15.107:0/625750898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f908406aac0 0x7f90840c0910 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f90840abef0 tx=0x7f907401c500 comp rx=0 tx=0).stop 2023-10-25T00:25:08.694 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 -- 172.21.15.107:0/625750898 shutdown_connections 2023-10-25T00:25:08.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 --2- 172.21.15.107:0/625750898 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f908406cc30 0x7f90840c0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:08.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 --2- 172.21.15.107:0/625750898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f908406aac0 0x7f90840c0910 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:08.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 --2- 172.21.15.107:0/625750898 >> v2:172.21.15.110:3300/0 conn(0x7f90840ab2d0 0x7f90840a4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:08.695 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 -- 172.21.15.107:0/625750898 >> 172.21.15.107:0/625750898 conn(0x7f90840c27f0 msgr2=0x7f90840c2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 -- 172.21.15.107:0/625750898 shutdown_connections 2023-10-25T00:25:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 -- 172.21.15.107:0/625750898 wait complete. 2023-10-25T00:25:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 Processor -- start 2023-10-25T00:25:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 -- start start 2023-10-25T00:25:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f908406aac0 0x7f908414fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:08.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f908406cc30 0x7f908414ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:08.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90840ab2d0 0x7f908414a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:08.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908414b900 con 0x7f908406cc30 2023-10-25T00:25:08.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908414b060 con 0x7f908406aac0 2023-10-25T00:25:08.697 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.688+0000 7f9088a83640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908414b340 con 0x7f90840ab2d0 2023-10-25T00:25:08.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9081d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f908406cc30 0x7f908414ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:08.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9081d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f908406cc30 0x7f908414ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55382/0 (socket says 172.21.15.107:55382) 2023-10-25T00:25:08.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9081d9b640 1 -- 172.21.15.107:0/732438287 learned_addr learned my addr 172.21.15.107:0/732438287 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:08.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9082d9d640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90840ab2d0 0x7f908414a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:08.698 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f908259c640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f908406aac0 0x7f908414fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:08.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9081d9b640 1 -- 172.21.15.107:0/732438287 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90840ab2d0 msgr2=0x7f908414a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:08.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9081d9b640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90840ab2d0 0x7f908414a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:08.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9081d9b640 1 -- 172.21.15.107:0/732438287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f908406aac0 msgr2=0x7f908414fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:08.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9081d9b640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f908406aac0 0x7f908414fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:08.699 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9081d9b640 1 -- 172.21.15.107:0/732438287 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90740094d0 con 0x7f908406cc30 2023-10-25T00:25:08.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9081d9b640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f908406cc30 0x7f908414ddd0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f907401abb0 tx=0x7f90740192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:08.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9072ffd640 1 -- 172.21.15.107:0/732438287 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90740b5640 con 0x7f908406cc30 2023-10-25T00:25:08.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9072ffd640 1 -- 172.21.15.107:0/732438287 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f90740b57a0 con 0x7f908406cc30 2023-10-25T00:25:08.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9088a83640 1 -- 172.21.15.107:0/732438287 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908414b620 con 0x7f908406cc30 2023-10-25T00:25:08.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9072ffd640 1 -- 172.21.15.107:0/732438287 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90740b5b10 con 0x7f908406cc30 2023-10-25T00:25:08.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9088a83640 1 -- 172.21.15.107:0/732438287 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908414bef0 con 0x7f908406cc30 2023-10-25T00:25:08.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9088a83640 1 -- 172.21.15.107:0/732438287 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908414ba70 con 0x7f908406cc30 2023-10-25T00:25:08.701 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9072ffd640 1 -- 172.21.15.107:0/732438287 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f90740b3070 con 0x7f908406cc30 2023-10-25T00:25:08.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9072ffd640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9064065e60 0x7f9064068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:08.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f908259c640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9064065e60 0x7f9064068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:08.702 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f9072ffd640 1 -- 172.21.15.107:0/732438287 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 10199+0+0 (secure 0 0 0) 0x7f907411ea60 con 0x7f908406cc30 2023-10-25T00:25:08.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.692+0000 7f908259c640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9064065e60 0x7f9064068320 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f90840c12e0 tx=0x7f9078008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:08.703 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.696+0000 7f9072ffd640 1 -- 172.21.15.107:0/732438287 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f908414ba70 con 0x7f908406cc30 2023-10-25T00:25:08.842 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:08.840+0000 7f9088a83640 1 -- 172.21.15.107:0/732438287 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "b_metadata", "pool2": "b_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f908414ba70 con 0x7f908406cc30 2023-10-25T00:25:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.552+0000 7f9072ffd640 1 -- 172.21.15.107:0/732438287 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "b_metadata", "pool2": "b_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'b_metadata' removed v79) v1 ==== 166+0+0 (secure 0 0 0) 0x7f908414ba70 con 0x7f908406cc30 2023-10-25T00:25:09.555 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'b_metadata' removed 2023-10-25T00:25:09.558 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.556+0000 7f9088a83640 1 -- 172.21.15.107:0/732438287 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9064065e60 msgr2=0x7f9064068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.556+0000 7f9088a83640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9064065e60 0x7f9064068320 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f90840c12e0 tx=0x7f9078008040 comp rx=0 tx=0).stop 2023-10-25T00:25:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.556+0000 7f9088a83640 1 -- 172.21.15.107:0/732438287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f908406cc30 msgr2=0x7f908414ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:09.559 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.556+0000 7f9088a83640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f908406cc30 0x7f908414ddd0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f907401abb0 tx=0x7f90740192a0 comp rx=0 tx=0).stop 2023-10-25T00:25:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.564+0000 7f9088a83640 1 -- 172.21.15.107:0/732438287 shutdown_connections 2023-10-25T00:25:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.564+0000 7f9088a83640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f9064065e60 0x7f9064068320 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.564+0000 7f9088a83640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90840ab2d0 0x7f908414a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:09.564 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.564+0000 7f9088a83640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f908406cc30 0x7f908414ddd0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.564+0000 7f9088a83640 1 --2- 172.21.15.107:0/732438287 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f908406aac0 0x7f908414fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.564+0000 7f9088a83640 1 -- 172.21.15.107:0/732438287 >> 172.21.15.107:0/732438287 conn(0x7f90840c27f0 msgr2=0x7f908406f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:09.565 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.564+0000 7f9088a83640 1 -- 172.21.15.107:0/732438287 shutdown_connections 2023-10-25T00:25:09.566 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.564+0000 7f9088a83640 1 -- 172.21.15.107:0/732438287 wait complete. 2023-10-25T00:25:09.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm b_data b_data --yes-i-really-really-mean-it 2023-10-25T00:25:09.710 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.708+0000 7f27b8673640 1 Processor -- start 2023-10-25T00:25:09.711 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.708+0000 7f27b8673640 1 -- start start 2023-10-25T00:25:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.708+0000 7f27b8673640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f27b0070530 0x7f27b01553e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.708+0000 7f27b8673640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27b0071f40 0x7f27b00aae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.708+0000 7f27b8673640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b00aa990 con 0x7f27b013dbe0 2023-10-25T00:25:09.712 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.708+0000 7f27b8673640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27b00ac2d0 con 0x7f27b0070530 2023-10-25T00:25:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.708+0000 7f27b8673640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b00aa060 con 0x7f27b0071f40 2023-10-25T00:25:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.708+0000 7f27b5c0e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f27b0070530 0x7f27b01553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.708+0000 7f27b5c0e640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f27b0070530 0x7f27b01553e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55398/0 (socket says 172.21.15.107:55398) 2023-10-25T00:25:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.708+0000 7f27b5c0e640 1 -- 172.21.15.107:0/2000727686 learned_addr learned my addr 172.21.15.107:0/2000727686 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:09.713 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b6c10640 1 --2- 172.21.15.107:0/2000727686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27b0071f40 0x7f27b00aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b5c0e640 1 -- 172.21.15.107:0/2000727686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27b0071f40 msgr2=0x7f27b00aae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b5c0e640 1 --2- 172.21.15.107:0/2000727686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27b0071f40 0x7f27b00aae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b5c0e640 1 -- 172.21.15.107:0/2000727686 >> v1:172.21.15.110:6789/0 conn(0x7f27b013dbe0 legacy=0x7f27b006fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:09.714 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b5c0e640 1 -- 172.21.15.107:0/2000727686 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b00aa340 con 0x7f27b0070530 2023-10-25T00:25:09.715 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b5c0e640 1 --2- 172.21.15.107:0/2000727686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b0070530 0x7f27b01553e0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009870 tx=0x7f27a001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6cd698794643414 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b540d640 1 -- 172.21.15.107:0/2000727686 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a00ab600 con 0x7f27b0070530 2023-10-25T00:25:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b540d640 1 -- 172.21.15.107:0/2000727686 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f27a00a7050 con 0x7f27b0070530 2023-10-25T00:25:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b540d640 1 -- 172.21.15.107:0/2000727686 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a00ab9a0 con 0x7f27b0070530 2023-10-25T00:25:09.716 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b8673640 1 -- 172.21.15.107:0/2000727686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b0070530 msgr2=0x7f27b01553e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b8673640 1 --2- 172.21.15.107:0/2000727686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b0070530 0x7f27b01553e0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009870 tx=0x7f27a001c380 comp rx=0 tx=0).stop 2023-10-25T00:25:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b8673640 1 -- 172.21.15.107:0/2000727686 shutdown_connections 2023-10-25T00:25:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b8673640 1 --2- 172.21.15.107:0/2000727686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27b0071f40 0x7f27b00aae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b8673640 1 --2- 172.21.15.107:0/2000727686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b0070530 0x7f27b01553e0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:09.717 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b8673640 1 -- 172.21.15.107:0/2000727686 >> 172.21.15.107:0/2000727686 conn(0x7f27b00c4de0 msgr2=0x7f27b00c51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.712+0000 7f27b8673640 1 -- 172.21.15.107:0/2000727686 shutdown_connections 2023-10-25T00:25:09.718 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b8673640 1 -- 172.21.15.107:0/2000727686 wait complete. 2023-10-25T00:25:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b8673640 1 Processor -- start 2023-10-25T00:25:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b8673640 1 -- start start 2023-10-25T00:25:09.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b8673640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27b0070530 0x7f27b013b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b8673640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b0071f40 0x7f27b00c0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b8673640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27b013dbe0 0x7f27b01583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b8673640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b01f0d00 con 0x7f27b0071f40 2023-10-25T00:25:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b8673640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b01f0a20 con 0x7f27b0070530 2023-10-25T00:25:09.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b8673640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27b01f00f0 con 0x7f27b013dbe0 2023-10-25T00:25:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b5c0e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b0071f40 0x7f27b00c0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b5c0e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b0071f40 0x7f27b00c0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55410/0 (socket says 172.21.15.107:55410) 2023-10-25T00:25:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b5c0e640 1 -- 172.21.15.107:0/40216959 learned_addr learned my addr 172.21.15.107:0/40216959 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b6c10640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27b013dbe0 0x7f27b01583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:09.722 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b640f640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27b0070530 0x7f27b013b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b5c0e640 1 -- 172.21.15.107:0/40216959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27b013dbe0 msgr2=0x7f27b01583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b5c0e640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27b013dbe0 0x7f27b01583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b5c0e640 1 -- 172.21.15.107:0/40216959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27b0070530 msgr2=0x7f27b013b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b5c0e640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27b0070530 0x7f27b013b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:09.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b5c0e640 1 -- 172.21.15.107:0/40216959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a00094d0 con 0x7f27b0071f40 2023-10-25T00:25:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b5c0e640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b0071f40 0x7f27b00c0530 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f27a00ae040 tx=0x7f27a0019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27a6ffd640 1 -- 172.21.15.107:0/40216959 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a00ba070 con 0x7f27b0071f40 2023-10-25T00:25:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27a6ffd640 1 -- 172.21.15.107:0/40216959 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f27a00abd40 con 0x7f27b0071f40 2023-10-25T00:25:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27a6ffd640 1 -- 172.21.15.107:0/40216959 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a00b66d0 con 0x7f27b0071f40 2023-10-25T00:25:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b8673640 1 -- 172.21.15.107:0/40216959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b01f03d0 con 0x7f27b0071f40 2023-10-25T00:25:09.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.716+0000 7f27b8673640 1 -- 172.21.15.107:0/40216959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b013b8f0 con 0x7f27b0071f40 2023-10-25T00:25:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.720+0000 7f27b8673640 1 -- 172.21.15.107:0/40216959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b01f0e90 con 0x7f27b0071f40 2023-10-25T00:25:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.720+0000 7f27a6ffd640 1 -- 172.21.15.107:0/40216959 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f27a00b2070 con 0x7f27b0071f40 2023-10-25T00:25:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.720+0000 7f27a6ffd640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2794065e60 0x7f2794068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.720+0000 7f27a6ffd640 1 -- 172.21.15.107:0/40216959 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 9771+0+0 (secure 0 0 0) 0x7f27a011e1e0 con 0x7f27b0071f40 2023-10-25T00:25:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.720+0000 7f27b640f640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2794065e60 0x7f2794068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.720+0000 7f27b640f640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2794065e60 0x7f2794068320 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00b810 tx=0x7f27ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:09.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.724+0000 7f27a6ffd640 1 -- 172.21.15.107:0/40216959 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f27a00eab10 con 0x7f27b0071f40 2023-10-25T00:25:09.904 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:09.904+0000 7f27b8673640 1 -- 172.21.15.107:0/40216959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "b_data", "pool2": "b_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f27b00746e0 con 0x7f27b0071f40 2023-10-25T00:25:10.575 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.572+0000 7f27a6ffd640 1 -- 172.21.15.107:0/40216959 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "b_data", "pool2": "b_data", "yes_i_really_really_mean_it": true}]=0 pool 'b_data' removed v80) v1 ==== 154+0+0 (secure 0 0 0) 0x7f27a00ee130 con 0x7f27b0071f40 2023-10-25T00:25:10.579 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'b_data' removed 2023-10-25T00:25:10.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 -- 172.21.15.107:0/40216959 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2794065e60 msgr2=0x7f2794068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:10.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2794065e60 0x7f2794068320 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f27ac00b810 tx=0x7f27ac008040 comp rx=0 tx=0).stop 2023-10-25T00:25:10.579 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 -- 172.21.15.107:0/40216959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b0071f40 msgr2=0x7f27b00c0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:10.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b0071f40 0x7f27b00c0530 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f27a00ae040 tx=0x7f27a0019180 comp rx=0 tx=0).stop 2023-10-25T00:25:10.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 -- 172.21.15.107:0/40216959 shutdown_connections 2023-10-25T00:25:10.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f2794065e60 0x7f2794068320 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:10.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f27b013dbe0 0x7f27b01583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:10.580 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27b0071f40 0x7f27b00c0530 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:10.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 --2- 172.21.15.107:0/40216959 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27b0070530 0x7f27b013b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:10.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 -- 172.21.15.107:0/40216959 >> 172.21.15.107:0/40216959 conn(0x7f27b00c4de0 msgr2=0x7f27b0153060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:10.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 -- 172.21.15.107:0/40216959 shutdown_connections 2023-10-25T00:25:10.581 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:10.576+0000 7f27b8673640 1 -- 172.21.15.107:0/40216959 wait complete. 2023-10-25T00:25:10.623 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-25T00:25:10.623 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-25T00:25:10.623 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:10.666 INFO:tasks.ceph.mds.a:Stopped 2023-10-25T00:25:10.666 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-25T00:25:10.666 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:10.678 INFO:tasks.ceph.mds.c:Stopped 2023-10-25T00:25:10.678 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-25T00:25:10.679 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:10.690 INFO:tasks.ceph.mds.b:Stopped 2023-10-25T00:25:10.690 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-25T00:25:10.690 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:10.745 INFO:tasks.ceph.mds.d:Stopped 2023-10-25T00:25:10.746 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-25T00:25:10.746 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-25T00:25:10.747 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:16.849 INFO:tasks.ceph.osd.0:Stopped 2023-10-25T00:25:16.850 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-25T00:25:16.850 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:16.943 INFO:tasks.ceph.osd.1:Stopped 2023-10-25T00:25:16.943 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-25T00:25:16.944 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:17.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-25T00:25:23.046 INFO:tasks.ceph.osd.2:Stopped 2023-10-25T00:25:23.046 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-25T00:25:23.047 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:23.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-25T00:25:23.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-25T00:25:29.149 INFO:tasks.ceph.osd.3:Stopped 2023-10-25T00:25:29.149 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-25T00:25:29.150 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:29.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-10-25T00:25:29.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-25T00:25:35.252 INFO:tasks.ceph.osd.4:Stopped 2023-10-25T00:25:35.253 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-25T00:25:35.253 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:35.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-10-25T00:25:35.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-10-25T00:25:38.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-25T00:25:38.631 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.628+0000 7f76eaf69640 1 Processor -- start 2023-10-25T00:25:38.632 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.628+0000 7f76eaf69640 1 -- start start 2023-10-25T00:25:38.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76eaf69640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76e414fd40 0x7f76e40ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:38.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76eaf69640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76e40a9820 0x7f76e40aa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:38.633 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76eaf69640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f76e406ee40 0x7f76e40c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76eaf69640 1 -- --> v2:172.21.15.107:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e40aa750 con 0x7f76e40a9820 2023-10-25T00:25:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76eaf69640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e406f730 con 0x7f76e406ee40 2023-10-25T00:25:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76eaf69640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e4156a40 con 0x7f76e414fd40 2023-10-25T00:25:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76e40a9820 0x7f76e40aa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e3fff640 1 --2- >> v2:172.21.15.107:3300/0 conn(0x7f76e40a9820 0x7f76e40aa210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46274/0 (socket says 172.21.15.107:46274) 2023-10-25T00:25:38.634 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e3fff640 1 -- 172.21.15.107:0/53613537 learned_addr learned my addr 172.21.15.107:0/53613537 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:38.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e8d05640 1 --2- 172.21.15.107:0/53613537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76e414fd40 0x7f76e40ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:38.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e9506640 1 --2- 172.21.15.107:0/53613537 >> v2:172.21.15.110:3300/0 conn(0x7f76e406ee40 0x7f76e40c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:38.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e3fff640 1 -- 172.21.15.107:0/53613537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76e414fd40 msgr2=0x7f76e40ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:38.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e3fff640 1 --2- 172.21.15.107:0/53613537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76e414fd40 0x7f76e40ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.635 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e3fff640 1 -- 172.21.15.107:0/53613537 >> v2:172.21.15.110:3300/0 conn(0x7f76e406ee40 msgr2=0x7f76e40c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-25T00:25:38.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e3fff640 1 --2- 172.21.15.107:0/53613537 >> v2:172.21.15.110:3300/0 conn(0x7f76e406ee40 0x7f76e40c1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.636 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e3fff640 1 -- 172.21.15.107:0/53613537 --> v2:172.21.15.107:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e4156710 con 0x7f76e40a9820 2023-10-25T00:25:38.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e3fff640 1 --2- 172.21.15.107:0/53613537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76e40a9820 0x7f76e40aa210 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f76e406f260 tx=0x7f76d801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e92501335478e56 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:38.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e37fe640 1 -- 172.21.15.107:0/53613537 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d80ac640 con 0x7f76e40a9820 2023-10-25T00:25:38.639 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e37fe640 1 -- 172.21.15.107:0/53613537 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f76d80ac7a0 con 0x7f76e40a9820 2023-10-25T00:25:38.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76e37fe640 1 -- 172.21.15.107:0/53613537 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d80acb50 con 0x7f76e40a9820 2023-10-25T00:25:38.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76eaf69640 1 -- 172.21.15.107:0/53613537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76e40a9820 msgr2=0x7f76e40aa210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:38.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.632+0000 7f76eaf69640 1 --2- 172.21.15.107:0/53613537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76e40a9820 0x7f76e40aa210 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f76e406f260 tx=0x7f76d801c3b0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 -- 172.21.15.107:0/53613537 shutdown_connections 2023-10-25T00:25:38.640 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 --2- 172.21.15.107:0/53613537 >> v2:172.21.15.110:3300/0 conn(0x7f76e406ee40 0x7f76e40c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 --2- 172.21.15.107:0/53613537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76e40a9820 0x7f76e40aa210 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 --2- 172.21.15.107:0/53613537 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76e414fd40 0x7f76e40ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 -- 172.21.15.107:0/53613537 >> 172.21.15.107:0/53613537 conn(0x7f76e40c2950 msgr2=0x7f76e40c2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:38.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 -- 172.21.15.107:0/53613537 shutdown_connections 2023-10-25T00:25:38.641 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 -- 172.21.15.107:0/53613537 wait complete. 2023-10-25T00:25:38.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 Processor -- start 2023-10-25T00:25:38.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 -- start start 2023-10-25T00:25:38.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76e406ee40 0x7f76e4148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:38.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76e414fd40 0x7f76e41eae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:38.642 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76e41573a0 0x7f76e415b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e40c3980 con 0x7f76e406ee40 2023-10-25T00:25:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e41eb4c0 con 0x7f76e414fd40 2023-10-25T00:25:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e41eb7a0 con 0x7f76e41573a0 2023-10-25T00:25:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e8d05640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76e406ee40 0x7f76e4148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:38.643 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e8d05640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76e406ee40 0x7f76e4148b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46282/0 (socket says 172.21.15.107:46282) 2023-10-25T00:25:38.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e8d05640 1 -- 172.21.15.107:0/832557282 learned_addr learned my addr 172.21.15.107:0/832557282 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-10-25T00:25:38.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e9506640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76e41573a0 0x7f76e415b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:38.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e3fff640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76e414fd40 0x7f76e41eae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:38.644 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e8d05640 1 -- 172.21.15.107:0/832557282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76e41573a0 msgr2=0x7f76e415b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:38.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e8d05640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76e41573a0 0x7f76e415b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e8d05640 1 -- 172.21.15.107:0/832557282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76e414fd40 msgr2=0x7f76e41eae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:38.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e8d05640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76e414fd40 0x7f76e41eae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e8d05640 1 -- 172.21.15.107:0/832557282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d80094d0 con 0x7f76e406ee40 2023-10-25T00:25:38.645 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e3fff640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76e414fd40 0x7f76e41eae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-25T00:25:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e8d05640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76e406ee40 0x7f76e4148b20 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f76d400d5f0 tx=0x7f76d400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e17fa640 1 -- 172.21.15.107:0/832557282 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d40125f0 con 0x7f76e406ee40 2023-10-25T00:25:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e17fa640 1 -- 172.21.15.107:0/832557282 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f76d4012750 con 0x7f76e406ee40 2023-10-25T00:25:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 -- 172.21.15.107:0/832557282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e41eba80 con 0x7f76e406ee40 2023-10-25T00:25:38.646 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76e17fa640 1 -- 172.21.15.107:0/832557282 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d4012b00 con 0x7f76e406ee40 2023-10-25T00:25:38.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 -- 172.21.15.107:0/832557282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e4157850 con 0x7f76e406ee40 2023-10-25T00:25:38.647 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.636+0000 7f76eaf69640 1 -- 172.21.15.107:0/832557282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e40c24a0 con 0x7f76e406ee40 2023-10-25T00:25:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.640+0000 7f76e17fa640 1 -- 172.21.15.107:0/832557282 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 5) v1 ==== 85528+0+0 (secure 0 0 0) 0x7f76d4012c60 con 0x7f76e406ee40 2023-10-25T00:25:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.640+0000 7f76e17fa640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76d0065e60 0x7f76d0068320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-25T00:25:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.640+0000 7f76e17fa640 1 -- 172.21.15.107:0/832557282 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9338+0+0 (secure 0 0 0) 0x7f76d4084a70 con 0x7f76e406ee40 2023-10-25T00:25:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.640+0000 7f76e3fff640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76d0065e60 0x7f76d0068320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-25T00:25:38.648 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.640+0000 7f76e3fff640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76d0065e60 0x7f76d0068320 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f76d80ae000 tx=0x7f76d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-25T00:25:38.649 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.644+0000 7f76e17fa640 1 -- 172.21.15.107:0/832557282 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172073 (secure 0 0 0) 0x7f76d4051310 con 0x7f76e406ee40 2023-10-25T00:25:38.821 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.820+0000 7f76eaf69640 1 -- 172.21.15.107:0/832557282 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76e4070840 con 0x7f76e406ee40 2023-10-25T00:25:38.822 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.820+0000 7f76e17fa640 1 -- 172.21.15.107:0/832557282 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3726 (secure 0 0 0) 0x7f76d4007380 con 0x7f76e406ee40 2023-10-25T00:25:38.822 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-10-25T00:25:38.823 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":246,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6835/2437148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6833","nonce":2437148},{"type":"v1","addr":"172.21.15.110:6835","nonce":2437148}]},"join_fscid":-1,"laggy_since":"2023-10-25T00:25:24.293400+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":245},{"gid":7079,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.110:6834/1074947504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6832","nonce":1074947504},{"type":"v1","addr":"172.21.15.110:6834","nonce":1074947504}]},"join_fscid":-1,"laggy_since":"2023-10-25T00:25:24.293413+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":245},{"gid":9606,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/2839787144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":2839787144},{"type":"v1","addr":"172.21.15.107:6837","nonce":2839787144}]},"join_fscid":-1,"laggy_since":"2023-10-25T00:25:24.293425+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":245},{"gid":9660,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/434729603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":434729603},{"type":"v1","addr":"172.21.15.107:6835","nonce":434729603}]},"join_fscid":-1,"laggy_since":"2023-10-25T00:25:29.295743+0000","export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":246}],"filesystems":[]} 2023-10-25T00:25:38.823 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 246 2023-10-25T00:25:38.824 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 -- 172.21.15.107:0/832557282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76d0065e60 msgr2=0x7f76d0068320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76d0065e60 0x7f76d0068320 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f76d80ae000 tx=0x7f76d8008040 comp rx=0 tx=0).stop 2023-10-25T00:25:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 -- 172.21.15.107:0/832557282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76e406ee40 msgr2=0x7f76e4148b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-25T00:25:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76e406ee40 0x7f76e4148b20 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f76d400d5f0 tx=0x7f76d400dab0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.825 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 -- 172.21.15.107:0/832557282 shutdown_connections 2023-10-25T00:25:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.107:6832/37466,v1:172.21.15.107:6833/37466] conn(0x7f76d0065e60 0x7f76d0068320 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76e41573a0 0x7f76e415b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76e414fd40 0x7f76e41eae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 --2- 172.21.15.107:0/832557282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76e406ee40 0x7f76e4148b20 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-25T00:25:38.826 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 -- 172.21.15.107:0/832557282 >> 172.21.15.107:0/832557282 conn(0x7f76e40c2950 msgr2=0x7f76e414d390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-25T00:25:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 -- 172.21.15.107:0/832557282 shutdown_connections 2023-10-25T00:25:38.827 INFO:teuthology.orchestra.run.smithi107.stderr:2023-10-25T00:25:38.824+0000 7f76eaf69640 1 -- 172.21.15.107:0/832557282 wait complete. 2023-10-25T00:25:41.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-10-25T00:25:41.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-10-25T00:25:41.355 INFO:tasks.ceph.osd.5:Stopped 2023-10-25T00:25:41.356 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-25T00:25:41.356 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:46.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-10-25T00:25:46.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-10-25T00:25:47.458 INFO:tasks.ceph.osd.6:Stopped 2023-10-25T00:25:47.459 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-25T00:25:47.459 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:47.543 INFO:tasks.ceph.osd.7:Stopped 2023-10-25T00:25:47.543 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-25T00:25:47.544 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-25T00:25:47.544 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:47.574 INFO:tasks.ceph.mgr.y:Stopped 2023-10-25T00:25:47.574 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-25T00:25:47.574 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:47.601 INFO:tasks.ceph.mgr.x:Stopped 2023-10-25T00:25:47.602 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-25T00:25:47.602 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-25T00:25:47.602 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:47.638 INFO:tasks.ceph.mon.a:Stopped 2023-10-25T00:25:47.638 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-25T00:25:47.638 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:47.664 INFO:tasks.ceph.mon.b:Stopped 2023-10-25T00:25:47.665 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-25T00:25:47.665 INFO:teuthology.orchestra.run:waiting for 300 2023-10-25T00:25:47.721 INFO:tasks.ceph.mon.c:Stopped 2023-10-25T00:25:47.722 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-25T00:25:47.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-10-25T00:25:47.895 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi107.front.sepia.ceph.com 2023-10-25T00:25:47.895 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-25T00:25:48.168 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi107.front.sepia.ceph.com 2023-10-25T00:25:48.169 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-25T00:25:48.431 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi107.front.sepia.ceph.com 2023-10-25T00:25:48.431 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-25T00:25:48.641 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi107.front.sepia.ceph.com 2023-10-25T00:25:48.642 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-25T00:25:48.881 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-25T00:25:48.881 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-25T00:25:49.174 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-25T00:25:49.174 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-25T00:25:49.337 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-25T00:25:49.337 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-25T00:25:49.513 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi110.front.sepia.ceph.com 2023-10-25T00:25:49.513 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-25T00:25:49.682 INFO:tasks.ceph:Archiving mon data... 2023-10-25T00:25:49.683 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357/data/mon.a.tgz 2023-10-25T00:25:49.684 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2023-10-25T00:25:49.704 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.7JVREAgkby 2023-10-25T00:25:49.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7JVREAgkby 2023-10-25T00:25:49.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.7JVREAgkby 2023-10-25T00:25:49.971 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.7JVREAgkby is 459KB 2023-10-25T00:25:50.034 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.7JVREAgkby 2023-10-25T00:25:50.040 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357/data/mon.b.tgz 2023-10-25T00:25:50.040 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-10-25T00:25:50.052 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.1T99ytH7Tt 2023-10-25T00:25:50.052 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.1T99ytH7Tt 2023-10-25T00:25:50.265 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.1T99ytH7Tt 2023-10-25T00:25:50.351 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.1T99ytH7Tt is 461KB 2023-10-25T00:25:50.412 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.1T99ytH7Tt 2023-10-25T00:25:50.417 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357/data/mon.c.tgz 2023-10-25T00:25:50.417 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-10-25T00:25:50.466 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.brlivvrZ0j 2023-10-25T00:25:50.467 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.brlivvrZ0j 2023-10-25T00:25:50.615 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.brlivvrZ0j 2023-10-25T00:25:50.678 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.brlivvrZ0j is 461KB 2023-10-25T00:25:50.740 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.brlivvrZ0j 2023-10-25T00:25:50.747 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-25T00:25:50.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-25T00:25:50.751 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-25T00:25:50.807 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-25T00:25:50.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-25T00:25:50.810 DEBUG:teuthology.orchestra.run.smithi110:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-25T00:25:50.840 INFO:teuthology.orchestra.run.smithi107.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-25T00:25:50.871 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-25T00:25:50.875 INFO:tasks.ceph:Archiving crash dumps... 2023-10-25T00:25:50.877 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357/remote/smithi107/crash 2023-10-25T00:25:50.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-25T00:25:50.899 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357/remote/smithi110/crash 2023-10-25T00:25:50.900 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-25T00:25:50.930 INFO:tasks.ceph:Shutting down logrotate 2023-10-25T00:25:50.931 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/ceph.py", line 1951, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/ceph.py", line 256, in ceph_log logrotater.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/ceph.py", line 221, in end self.thread.get() File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/ceph.py", line 188, in invoke_logrotate run.wait(procs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-10-25T00:25:50.962 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-25T00:25:50.973 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/ceph.py", line 1951, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/ceph.py", line 256, in ceph_log logrotater.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/ceph.py", line 221, in end self.thread.get() File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_e652850897f60c32cb9cc05022cf9f93e3b00db1/qa/tasks/ceph.py", line 188, in invoke_logrotate run.wait(procs) File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_202b180cb047e798fb131047314a862593f45403/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi110 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-10-25T00:25:50.974 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-25T00:25:50.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-25T00:25:50.979 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-25T00:25:51.082 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-25T00:25:51.092 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-25T00:25:51.092 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-25T00:25:51.095 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-25T00:25:51.127 INFO:teuthology.orchestra.run.smithi107.stdout: remote refid st t when poll reach delay offset jitter 2023-10-25T00:25:51.127 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================== 2023-10-25T00:25:51.127 INFO:teuthology.orchestra.run.smithi107.stdout:*hv01.front.sepi 44.4.53.2 2 u 16 128 377 0.069 -3.662 0.502 2023-10-25T00:25:51.127 INFO:teuthology.orchestra.run.smithi107.stdout:+hv02.front.sepi 63.231.80.2 3 u 22 128 377 0.073 -1.514 0.411 2023-10-25T00:25:51.128 INFO:teuthology.orchestra.run.smithi107.stdout:+hv03.front.sepi 74.6.168.72 3 u 28 128 377 0.074 -3.777 0.479 2023-10-25T00:25:51.128 INFO:teuthology.orchestra.run.smithi107.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-25T00:25:51.131 INFO:teuthology.orchestra.run.smithi110.stdout: remote refid st t when poll reach delay offset jitter 2023-10-25T00:25:51.131 INFO:teuthology.orchestra.run.smithi110.stdout:============================================================================== 2023-10-25T00:25:51.131 INFO:teuthology.orchestra.run.smithi110.stdout:*hv01.front.sepi 44.4.53.2 2 u 7 64 377 0.074 -2.246 0.160 2023-10-25T00:25:51.132 INFO:teuthology.orchestra.run.smithi110.stdout:+hv02.front.sepi 63.231.80.2 3 u 20 64 377 0.081 -0.154 0.166 2023-10-25T00:25:51.132 INFO:teuthology.orchestra.run.smithi110.stdout:+hv03.front.sepi 74.6.168.72 3 u 7 64 377 0.074 -2.375 0.159 2023-10-25T00:25:51.132 INFO:teuthology.orchestra.run.smithi110.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-25T00:25:51.133 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-25T00:25:51.164 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-25T00:25:51.165 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-25T00:25:51.217 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-25T00:25:51.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-25T00:25:51.315 INFO:teuthology.task.internal:Duration was 2646.309610 seconds 2023-10-25T00:25:51.316 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-25T00:25:51.365 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-25T00:25:51.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-25T00:25:51.369 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-25T00:25:51.554 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-25T00:25:51.555 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2023-10-25T00:25:51.555 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-25T00:25:51.566 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2023-10-25T00:25:51.566 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-25T00:25:51.578 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-25T00:25:51.578 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-25T00:25:51.613 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-25T00:25:51.661 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-25T00:25:51.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-25T00:25:51.689 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-25T00:25:51.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-10-25T00:25:51.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-25T00:25:51.940 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-25T00:25:51.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-25T00:25:51.977 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-25T00:25:52.013 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-25T00:25:52.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-25T00:25:52.026 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-25T00:25:52.051 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2023-10-25T00:25:52.083 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2023-10-25T00:25:52.118 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-25T00:25:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-25T00:25:52.124 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-25T00:25:52.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-25T00:25:52.166 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-25T00:25:52.176 INFO:teuthology.task.internal:Transferring archived files... 2023-10-25T00:25:52.178 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357/remote/smithi107 2023-10-25T00:25:52.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-25T00:25:52.199 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357/remote/smithi110 2023-10-25T00:25:52.200 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-25T00:25:52.227 INFO:teuthology.task.internal:Removing archive directory... 2023-10-25T00:25:52.228 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-25T00:25:52.237 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-25T00:25:52.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-25T00:25:52.281 INFO:teuthology.task.internal:Not uploading archives. 2023-10-25T00:25:52.282 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-25T00:25:52.316 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-25T00:25:52.316 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-25T00:25:52.319 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-25T00:25:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: 658625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 25 00:25 /home/ubuntu/cephtest 2023-10-25T00:25:52.325 INFO:teuthology.orchestra.run.smithi110.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 25 00:25 /home/ubuntu/cephtest 2023-10-25T00:25:52.326 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-25T00:25:52.366 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-25T00:25:52.418 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi110 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-10-25T00:25:52.418 INFO:teuthology.nuke:Checking targets against current locks 2023-10-25T00:25:52.447 DEBUG:teuthology.nuke:shortname: smithi107 2023-10-25T00:25:52.448 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-25T00:25:52.472 DEBUG:teuthology.nuke:shortname: smithi110 2023-10-25T00:25:52.472 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-25T00:25:52.476 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-24 23:33:08.598976', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPL1FzpXP2qyGnObmjAsCdBxl4upbUplk4LyLEI0kr2ltoLMG8qjHtik+hMWxC3AgtLNcANRvfLwcYhciGTi1r8='} 2023-10-25T00:25:52.497 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435357', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-24 23:33:08.597388', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDF755M2c+AY0wPO/teNzumWbZarR33MHIzud+Kqg6Eo9+1OS5/JoqKUGpHGNoV1DVCGhd2kCArBv8Hzdp+dJjU='} 2023-10-25T00:25:52.533 INFO:teuthology.orchestra.console.smithi107:Power off 2023-10-25T00:25:52.534 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-25T00:25:52.559 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2023-10-25T00:25:52.560 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-25T00:25:52.573 INFO:teuthology.orchestra.console.smithi110:Power off 2023-10-25T00:25:52.574 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-25T00:25:52.583 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-10-25T00:25:52.598 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2023-10-25T00:25:52.598 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-25T00:25:52.622 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2023-10-25T00:25:56.585 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-25T00:25:56.623 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-25T00:25:56.697 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-10-25T00:25:56.733 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2023-10-25T00:26:00.698 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-25T00:26:00.734 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-25T00:26:00.808 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2023-10-25T00:26:00.845 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2023-10-25T00:26:00.946 INFO:teuthology.orchestra.console.smithi110:Power off completed 2023-10-25T00:26:04.810 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-25T00:26:04.921 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2023-10-25T00:26:05.022 INFO:teuthology.orchestra.console.smithi107:Power off completed 2023-10-25T00:26:05.126 INFO:teuthology.run:Summary data: description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} duration: 2646.309609889984 failure_reason: 'Command failed on smithi110 with status 3: ''sudo logrotate /etc/logrotate.d/ceph-test.conf''' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-10-25T00:26:05.127 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-25T00:26:05.207 INFO:teuthology.run:FAIL